2023-11-24T07:47:32.970 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-24T07:47:32.970 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582 --owner scheduled_rishabh@teuthology --name rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} --verbose -- /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582/orig.config.yaml 2023-11-24T07:47:33.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T07:47:33.101 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582 branch: rishabh-mgr-vol-misc description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} email: null first_in_suite: false job_id: '7465582' kernel: client: branch: testing flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-mgr-vol-misc ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-vol-tests sha1: 73843baf45f8a92d8e3fcbea3f024c97e73036da owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9764 sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c sleep_before_teardown: 0 suite: fs:volumes suite_branch: mgr-vol-tests suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 73843baf45f8a92d8e3fcbea3f024c97e73036da syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUWWHRQesnb0foUsW5bdD9/0e/M5daF7En+5E/TguR9kZ+kwLCXni0H9oxKVzMyoRSWtO4uh+NIzSB/tqpFRv8= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCd+WpaFe5MZUrwVrtno2fV8ntxSVfm4mTvahUZm6VEP8Jfebp4KLxUXw2rc9pNjqB960xGtkr+TIUiqqjxEqd0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-24_07:31:48 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-24T07:47:33.101 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa; will attempt to use it 2023-11-24T07:47:33.102 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks 2023-11-24T07:47:33.102 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-24T07:47:33.103 INFO:teuthology.task.internal:Checking packages... 2023-11-24T07:47:33.125 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a13ea5ff00cb4df7a9cc941166ddeb358881144c' 2023-11-24T07:47:33.126 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-24T07:47:33.126 INFO:teuthology.packaging:ref: None 2023-11-24T07:47:33.126 INFO:teuthology.packaging:tag: None 2023-11-24T07:47:33.126 INFO:teuthology.packaging:branch: rishabh-mgr-vol-misc 2023-11-24T07:47:33.126 INFO:teuthology.packaging:sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T07:47:33.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-mgr-vol-misc 2023-11-24T07:47:33.456 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7484.ga13ea5ff 2023-11-24T07:47:33.456 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-24T07:47:33.463 INFO:teuthology.task.internal:no buildpackages task found 2023-11-24T07:47:33.463 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-24T07:47:33.502 INFO:teuthology.task.internal:Saving configuration 2023-11-24T07:47:33.518 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-24T07:47:33.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T07:47:33.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 07:42:17.406660', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUWWHRQesnb0foUsW5bdD9/0e/M5daF7En+5E/TguR9kZ+kwLCXni0H9oxKVzMyoRSWtO4uh+NIzSB/tqpFRv8='} 2023-11-24T07:47:33.572 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 07:42:17.404787', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCd+WpaFe5MZUrwVrtno2fV8ntxSVfm4mTvahUZm6VEP8Jfebp4KLxUXw2rc9pNjqB960xGtkr+TIUiqqjxEqd0='} 2023-11-24T07:47:33.572 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-24T07:47:33.579 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-24T07:47:33.579 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-24T07:47:33.579 INFO:teuthology.run_tasks:Running task console_log... 2023-11-24T07:47:33.683 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc55b8f4b80>, signals=[15]) 2023-11-24T07:47:33.683 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-24T07:47:33.690 INFO:teuthology.task.internal:Opening connections... 2023-11-24T07:47:33.690 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T07:47:33.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:47:33.778 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T07:47:33.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:47:33.861 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-24T07:47:33.867 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-11-24T07:47:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-11-24T07:47:33.885 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-11-24T07:47:33.946 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2023-11-24T07:47:33.946 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2023-11-24T07:47:33.946 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2023-11-24T07:47:33.946 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2023-11-24T07:47:33.946 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-24T07:47:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-24T07:47:33.948 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-11-24T07:47:33.976 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2023-11-24T07:47:33.993 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2023-11-24T07:47:33.994 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2023-11-24T07:47:34.051 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2023-11-24T07:47:34.052 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2023-11-24T07:47:34.052 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2023-11-24T07:47:34.052 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2023-11-24T07:47:34.052 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2023-11-24T07:47:34.053 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2023-11-24T07:47:34.053 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-24T07:47:34.054 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-24T07:47:34.055 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2023-11-24T07:47:34.075 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-24T07:47:34.085 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-24T07:47:34.109 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-24T07:47:34.109 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-11-24T07:47:34.112 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2023-11-24T07:47:34.128 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-24T07:47:34.134 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-24T07:47:34.134 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-11-24T07:47:34.172 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2023-11-24T07:47:34.454 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-24T07:47:34.508 INFO:teuthology.run_tasks:Running task kernel... 2023-11-24T07:47:34.525 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-24T07:47:34.525 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-11-24T07:47:34.526 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-11-24T07:47:34.526 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T07:47:34.527 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T07:47:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T07:47:34.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-11-24T07:47:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T07:47:34.550 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-11-24T07:47:34.707 DEBUG:teuthology.task.kernel:sha1 for client.1 is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:47:34.707 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-24T07:47:34.708 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2023-11-24T07:47:34.724 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-361.el9.x86_64 2023-11-24T07:47:34.725 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:47:34.725 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-24T07:47:34.725 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum upgrade -y linux-firmware 2023-11-24T07:47:34.770 DEBUG:teuthology.task.kernel:sha1 for client.0 is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:47:34.770 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-24T07:47:34.771 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-11-24T07:47:34.787 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-361.el9.x86_64 2023-11-24T07:47:34.788 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:47:34.788 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-24T07:47:34.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum upgrade -y linux-firmware 2023-11-24T07:47:36.982 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-11-24T07:47:37.018 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-11-24T07:47:38.427 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2023-11-24T07:47:38.908 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 2.3 MB/s | 2.0 MB 00:00 2023-11-24T07:47:39.740 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-11-24T07:47:40.859 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 7.9 MB 00:02 2023-11-24T07:47:42.743 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2023-11-24T07:47:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 18 MB 00:01 2023-11-24T07:47:46.577 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 13 MB/s | 5.8 MB 00:00 2023-11-24T07:47:48.103 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2023-11-24T07:47:49.125 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 5.8 MB 00:02 2023-11-24T07:47:50.776 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-11-24T07:47:51.395 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 20 MB 00:03 2023-11-24T07:47:54.110 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 20 MB 00:03 2023-11-24T07:47:56.103 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2023-11-24T07:47:58.477 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-11-24T07:47:58.491 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 1 Package 2023-11-24T07:47:58.492 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:47:58.493 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 328 M 2023-11-24T07:47:58.493 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-11-24T07:47:58.910 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2023-11-24T07:48:01.277 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2023-11-24T07:48:01.287 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2023-11-24T07:48:01.287 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 1 Package 2023-11-24T07:48:01.288 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:48:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 328 M 2023-11-24T07:48:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2023-11-24T07:48:03.462 INFO:teuthology.orchestra.run.smithi107.stdout:linux-firmware-20231030-141.el9.noarch.rpm 66 MB/s | 328 MB 00:04 2023-11-24T07:48:03.463 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-11-24T07:48:03.463 INFO:teuthology.orchestra.run.smithi107.stdout:Total 66 MB/s | 328 MB 00:04 2023-11-24T07:48:03.464 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-11-24T07:48:03.562 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-11-24T07:48:03.562 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-11-24T07:48:05.734 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-11-24T07:48:05.734 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-11-24T07:48:06.345 INFO:teuthology.orchestra.run.smithi120.stdout:linux-firmware-20231030-141.el9.noarch.rpm 65 MB/s | 328 MB 00:05 2023-11-24T07:48:06.346 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2023-11-24T07:48:06.347 INFO:teuthology.orchestra.run.smithi120.stdout:Total 65 MB/s | 328 MB 00:05 2023-11-24T07:48:06.347 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2023-11-24T07:48:06.453 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2023-11-24T07:48:06.453 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2023-11-24T07:48:08.631 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2023-11-24T07:48:08.631 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2023-11-24T07:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-11-24T07:48:12.157 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-24T07:48:12.267 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:13.172 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:13.172 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-24T07:48:13.569 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:13.570 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:48:13.570 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2023-11-24T07:48:13.570 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-24T07:48:13.570 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:48:13.570 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-11-24T07:48:13.856 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.0... 2023-11-24T07:48:13.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:48:14.001 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-24T07:48:14.002 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-24T07:48:14.007 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi107.front.sepia.ceph.com to complete... 2023-11-24T07:48:14.425 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2023-11-24T07:48:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-24T07:48:14.903 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:15.935 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:15.935 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-24T07:48:16.324 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-24T07:48:16.325 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:48:16.325 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2023-11-24T07:48:16.326 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-24T07:48:16.326 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:48:16.327 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2023-11-24T07:48:16.631 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.1... 2023-11-24T07:48:16.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-24T07:48:16.852 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-24T07:48:16.853 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-24T07:48:16.856 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi120.front.sepia.ceph.com to complete... 2023-11-24T07:48:27.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24 07:48:27 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-24T07:48:27.430 INFO:teuthology.orchestra.run.smithi107.stderr:FINISHED --2023-11-24 07:48:27-- 2023-11-24T07:48:27.430 INFO:teuthology.orchestra.run.smithi107.stderr:Total wall clock time: 13s 2023-11-24T07:48:27.430 INFO:teuthology.orchestra.run.smithi107.stderr:Downloaded: 1 files, 306M in 13s (23.3 MB/s) 2023-11-24T07:48:27.431 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.0... 2023-11-24T07:48:27.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-24T07:48:29.277 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying... ######################################## 2023-11-24T07:48:29.336 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing... ######################################## 2023-11-24T07:48:29.416 INFO:teuthology.orchestra.run.smithi107.stdout:Updating / installing... 2023-11-24T07:48:36.472 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-24T07:48:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24 07:48:37 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-24T07:48:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:FINISHED --2023-11-24 07:48:37-- 2023-11-24T07:48:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:Total wall clock time: 21s 2023-11-24T07:48:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:Downloaded: 1 files, 306M in 21s (14.9 MB/s) 2023-11-24T07:48:37.924 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.1... 2023-11-24T07:48:37.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-24T07:48:39.757 INFO:teuthology.orchestra.run.smithi120.stdout:Verifying... ######################################## 2023-11-24T07:48:39.837 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing... ######################################## 2023-11-24T07:48:39.916 INFO:teuthology.orchestra.run.smithi120.stdout:Updating / installing... 2023-11-24T07:48:46.991 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-24T07:48:51.825 INFO:teuthology.orchestra.run.smithi107.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-24T07:49:02.670 INFO:teuthology.orchestra.run.smithi120.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-24T07:49:12.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-24T07:49:13.055 DEBUG:teuthology.orchestra.run.smithi107:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-24T07:49:13.137 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-24T07:49:13.138 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-24T07:49:13.139 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-24T07:49:13.140 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-24T07:49:13.141 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-24T07:49:13.142 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-24T07:49:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-24T07:49:13.144 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-24T07:49:13.145 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-24T07:49:13.146 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-24T07:49:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-24T07:49:13.148 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-24T07:49:13.149 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-24T07:49:13.150 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-24T07:49:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-24T07:49:13.152 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-24T07:49:13.153 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-24T07:49:13.154 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-24T07:49:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-24T07:49:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-24T07:49:13.157 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-24T07:49:13.158 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-24T07:49:13.159 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-24T07:49:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-24T07:49:13.161 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-24T07:49:13.162 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-24T07:49:13.163 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-24T07:49:13.164 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-24T07:49:13.165 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-24T07:49:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-24T07:49:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-24T07:49:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-24T07:49:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-24T07:49:13.170 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-24T07:49:13.171 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-24T07:49:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-24T07:49:13.173 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-24T07:49:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-24T07:49:13.175 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-24T07:49:13.176 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-24T07:49:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-24T07:49:13.178 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-24T07:49:13.179 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-24T07:49:13.180 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-24T07:49:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-24T07:49:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-24T07:49:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-24T07:49:13.184 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-24T07:49:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-24T07:49:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-24T07:49:13.187 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-24T07:49:13.188 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-24T07:49:13.189 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-24T07:49:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-24T07:49:13.191 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-24T07:49:13.192 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-24T07:49:13.193 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-24T07:49:13.194 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-24T07:49:13.195 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-24T07:49:13.196 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-24T07:49:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-24T07:49:13.198 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-24T07:49:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-24T07:49:13.200 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-24T07:49:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-24T07:49:13.202 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-24T07:49:13.203 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-24T07:49:13.204 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-24T07:49:13.205 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-24T07:49:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-24T07:49:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-24T07:49:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-24T07:49:13.209 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-24T07:49:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-24T07:49:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-24T07:49:13.212 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-24T07:49:13.213 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-24T07:49:13.214 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-24T07:49:13.215 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-24T07:49:13.216 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-24T07:49:13.217 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-24T07:49:13.218 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-24T07:49:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-24T07:49:13.220 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-24T07:49:13.221 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-24T07:49:13.222 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-24T07:49:13.223 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-24T07:49:13.224 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-24T07:49:13.225 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-24T07:49:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-24T07:49:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-24T07:49:13.228 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-24T07:49:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-24T07:49:13.230 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-24T07:49:13.231 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-24T07:49:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-24T07:49:13.233 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-24T07:49:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-24T07:49:13.235 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-24T07:49:13.236 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-24T07:49:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-24T07:49:13.238 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-24T07:49:13.239 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-24T07:49:13.240 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-24T07:49:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-24T07:49:13.242 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-24T07:49:13.243 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-24T07:49:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-24T07:49:13.245 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-24T07:49:13.246 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-24T07:49:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-24T07:49:13.248 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-24T07:49:13.249 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-24T07:49:13.250 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-24T07:49:13.251 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-24T07:49:13.252 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-24T07:49:13.253 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-24T07:49:13.254 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-24T07:49:13.255 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-24T07:49:13.256 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-24T07:49:13.257 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-24T07:49:13.258 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-24T07:49:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-24T07:49:13.260 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-24T07:49:13.261 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-24T07:49:13.262 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-24T07:49:13.263 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-24T07:49:13.264 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-24T07:49:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-24T07:49:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-24T07:49:13.267 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-24T07:49:13.269 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-24T07:49:13.269 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-24T07:49:13.269 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-24T07:49:13.269 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-24T07:49:13.270 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-24T07:49:13.271 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-24T07:49:13.272 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-24T07:49:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-24T07:49:13.274 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-24T07:49:13.275 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-24T07:49:13.276 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-24T07:49:13.277 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-24T07:49:13.278 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-24T07:49:13.279 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-24T07:49:13.280 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-24T07:49:13.281 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-24T07:49:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-24T07:49:13.283 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-24T07:49:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-24T07:49:13.285 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-24T07:49:13.286 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-24T07:49:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-24T07:49:13.288 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-24T07:49:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-24T07:49:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-24T07:49:13.291 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-24T07:49:13.292 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-24T07:49:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-24T07:49:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-24T07:49:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-24T07:49:13.296 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-24T07:49:13.296 DEBUG:teuthology.orchestra.run.smithi107:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:postinstall scriptlet (using /bin/sh): 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2023-11-24T07:49:13.319 INFO:teuthology.orchestra.run.smithi107.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-24T07:49:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2023-11-24T07:49:13.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2023-11-24T07:49:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-24T07:49:13.398 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-24T07:49:13.399 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-24T07:49:13.399 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2023-11-24T07:49:13.399 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T07:49:13.399 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2023-11-24T07:49:13.400 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-24T07:49:13.400 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 6.6.0-g8247bbb5b470 2023-11-24T07:49:13.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-24T07:49:13.977 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-11-24T07:49:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-24T07:49:15.422 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-11-24T07:49:15.424 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-11-24T07:49:15.439 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.XOMMyzPnPE 2023-11-24T07:49:15.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XOMMyzPnPE 2023-11-24T07:49:15.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.XOMMyzPnPE 2023-11-24T07:49:15.667 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.XOMMyzPnPE is 6KB 2023-11-24T07:49:15.686 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.XOMMyzPnPE 2023-11-24T07:49:15.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2023-11-24T07:49:15.765 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-24T07:49:15.765 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-24T07:49:15.765 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-24T07:49:15.765 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-24T07:49:15.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-24T07:49:15.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-11-24T07:49:23.865 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-24T07:49:23.865 DEBUG:teuthology.orchestra.run.smithi120:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-24T07:49:23.935 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-24T07:49:23.936 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-24T07:49:23.937 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-24T07:49:23.938 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-24T07:49:23.939 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-24T07:49:23.940 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-24T07:49:23.941 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-24T07:49:23.942 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-24T07:49:23.943 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-24T07:49:23.944 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-24T07:49:23.945 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-24T07:49:23.946 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-24T07:49:23.947 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-24T07:49:23.948 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-24T07:49:23.949 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-24T07:49:23.950 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-24T07:49:23.951 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-24T07:49:23.952 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-24T07:49:23.953 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-24T07:49:23.954 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-24T07:49:23.955 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-24T07:49:23.956 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-24T07:49:23.957 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-24T07:49:23.958 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-24T07:49:23.959 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-24T07:49:23.960 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-24T07:49:23.961 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-24T07:49:23.962 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-24T07:49:23.963 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-24T07:49:23.964 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-24T07:49:23.965 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-24T07:49:23.966 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-24T07:49:23.967 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-24T07:49:23.968 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-24T07:49:23.969 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-24T07:49:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-24T07:49:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-24T07:49:23.972 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-24T07:49:23.973 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-24T07:49:23.974 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-24T07:49:23.975 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-24T07:49:23.976 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-24T07:49:23.977 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-24T07:49:23.978 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-24T07:49:23.979 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-24T07:49:23.980 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-24T07:49:23.981 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-24T07:49:23.982 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-24T07:49:23.983 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-24T07:49:23.984 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-24T07:49:23.985 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-24T07:49:23.986 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-24T07:49:23.987 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-24T07:49:23.988 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-24T07:49:23.989 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-24T07:49:23.990 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-24T07:49:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-24T07:49:23.992 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-24T07:49:23.993 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-24T07:49:23.994 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-24T07:49:23.995 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-24T07:49:23.996 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-24T07:49:23.997 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-24T07:49:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-24T07:49:23.999 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-24T07:49:24.000 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-24T07:49:24.001 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-24T07:49:24.002 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-24T07:49:24.003 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-24T07:49:24.004 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-24T07:49:24.005 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-24T07:49:24.006 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-24T07:49:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-24T07:49:24.008 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-24T07:49:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-24T07:49:24.010 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-24T07:49:24.011 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-24T07:49:24.012 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-24T07:49:24.013 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-24T07:49:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-24T07:49:24.015 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-24T07:49:24.016 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-24T07:49:24.017 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-24T07:49:24.018 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-24T07:49:24.019 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-24T07:49:24.020 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-24T07:49:24.021 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-24T07:49:24.022 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-24T07:49:24.023 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-24T07:49:24.024 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-24T07:49:24.025 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-24T07:49:24.026 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-24T07:49:24.027 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-24T07:49:24.028 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-24T07:49:24.029 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-24T07:49:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-24T07:49:24.031 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-24T07:49:24.032 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-24T07:49:24.033 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-24T07:49:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-24T07:49:24.035 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-24T07:49:24.036 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-24T07:49:24.037 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-24T07:49:24.038 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-24T07:49:24.039 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-24T07:49:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-24T07:49:24.041 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-24T07:49:24.042 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-24T07:49:24.043 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-24T07:49:24.044 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-24T07:49:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-24T07:49:24.046 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-24T07:49:24.047 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-24T07:49:24.048 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-24T07:49:24.049 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-24T07:49:24.050 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-24T07:49:24.051 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-24T07:49:24.052 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-24T07:49:24.053 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-24T07:49:24.054 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-24T07:49:24.055 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-24T07:49:24.056 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-24T07:49:24.057 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-24T07:49:24.058 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-24T07:49:24.059 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-24T07:49:24.060 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-24T07:49:24.061 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-24T07:49:24.062 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-24T07:49:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-24T07:49:24.064 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-24T07:49:24.065 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-24T07:49:24.066 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-24T07:49:24.067 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-24T07:49:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-24T07:49:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-24T07:49:24.070 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-24T07:49:24.071 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-24T07:49:24.072 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-24T07:49:24.073 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-24T07:49:24.074 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-24T07:49:24.075 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-24T07:49:24.076 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-24T07:49:24.077 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-24T07:49:24.078 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-24T07:49:24.079 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-24T07:49:24.080 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-24T07:49:24.081 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-24T07:49:24.082 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-24T07:49:24.083 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-24T07:49:24.084 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-24T07:49:24.085 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-24T07:49:24.086 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-24T07:49:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-24T07:49:24.088 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-24T07:49:24.089 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-24T07:49:24.090 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-24T07:49:24.091 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-24T07:49:24.092 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-24T07:49:24.093 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-24T07:49:24.093 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-24T07:49:24.093 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-24T07:49:24.093 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-24T07:49:24.093 DEBUG:teuthology.orchestra.run.smithi120:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-24T07:49:24.115 INFO:teuthology.orchestra.run.smithi120.stdout:postinstall scriptlet (using /bin/sh): 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-24T07:49:24.116 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-24T07:49:24.117 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2023-11-24T07:49:24.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2023-11-24T07:49:24.192 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2023-11-24T07:49:24.192 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2023-11-24T07:49:24.192 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.06 2023-11-24T07:49:24.192 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 68.el9 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Group : Unspecified 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 8264254 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : builder@centos.org 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2023-11-24T07:49:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T07:49:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2023-11-24T07:49:24.195 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-24T07:49:24.195 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 6.6.0-g8247bbb5b470 2023-11-24T07:49:24.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-24T07:49:24.820 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2023-11-24T07:49:26.163 INFO:teuthology.orchestra.run.smithi120.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-24T07:49:26.198 INFO:teuthology.orchestra.run.smithi120.stderr:done 2023-11-24T07:49:26.200 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2023-11-24T07:49:26.215 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.h1SA92OAie 2023-11-24T07:49:26.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h1SA92OAie 2023-11-24T07:49:26.283 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.h1SA92OAie 2023-11-24T07:49:26.422 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.h1SA92OAie is 6KB 2023-11-24T07:49:26.470 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.h1SA92OAie 2023-11-24T07:49:26.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2023-11-24T07:49:26.553 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-24T07:49:26.553 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-24T07:49:26.553 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-24T07:49:26.553 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-24T07:49:26.555 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-24T07:49:26.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2023-11-24T07:49:45.958 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-11-24T07:49:45.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T07:49:45.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:49:56.734 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-11-24T07:49:56.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T07:49:56.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:50:04.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T07:50:13.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T07:50:13.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:50:15.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T07:50:16.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T07:50:24.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T07:50:24.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:50:27.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T07:50:28.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T07:50:28.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:50:39.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T07:50:39.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:51:11.254 DEBUG:teuthology.orchestra.run.smithi120:> true 2023-11-24T07:51:11.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T07:51:11.891 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-g8247bbb5b470"... 2023-11-24T07:51:11.891 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2023-11-24T07:51:11.942 INFO:teuthology.orchestra.run.smithi120.stdout:6.6.0-g8247bbb5b470 2023-11-24T07:51:11.942 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-24T07:51:11.942 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-24T07:51:11.942 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-24T07:51:12.943 DEBUG:teuthology.parallel:result is None 2023-11-24T07:51:28.459 DEBUG:teuthology.orchestra.remote:timed out 2023-11-24T07:51:43.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T07:51:43.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T07:51:43.996 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-11-24T07:51:44.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T07:51:44.600 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-g8247bbb5b470"... 2023-11-24T07:51:44.600 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-11-24T07:51:44.654 INFO:teuthology.orchestra.run.smithi107.stdout:6.6.0-g8247bbb5b470 2023-11-24T07:51:44.654 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-24T07:51:44.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-24T07:51:44.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-24T07:51:45.656 DEBUG:teuthology.parallel:result is None 2023-11-24T07:51:45.656 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-24T07:51:45.663 INFO:teuthology.task.internal:Creating test directory... 2023-11-24T07:51:45.664 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T07:51:45.674 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T07:51:45.737 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-24T07:51:45.745 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-24T07:51:45.770 INFO:teuthology.task.internal:Creating archive directory... 2023-11-24T07:51:45.770 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T07:51:45.779 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T07:51:46.005 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-24T07:51:46.012 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-24T07:51:46.012 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T07:51:46.021 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T07:51:46.157 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T07:51:46.196 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T07:51:46.225 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T07:51:46.265 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T07:51:46.270 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-24T07:51:46.277 INFO:teuthology.task.internal:Configuring sudo... 2023-11-24T07:51:46.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T07:51:46.286 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T07:51:46.386 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-24T07:51:46.395 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-24T07:51:46.396 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T07:51:46.401 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T07:51:46.458 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T07:51:46.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T07:51:46.646 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T07:51:46.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T07:51:46.826 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T07:51:46.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T07:51:46.919 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T07:51:47.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T07:51:47.125 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T07:51:47.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T07:51:47.316 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T07:51:47.316 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T07:51:47.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-11-24T07:51:47.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2023-11-24T07:51:47.539 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T07:51:47.582 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T07:51:47.957 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-24T07:51:47.966 INFO:teuthology.task.internal:Starting timer... 2023-11-24T07:51:47.966 INFO:teuthology.run_tasks:Running task pcp... 2023-11-24T07:51:48.007 INFO:teuthology.run_tasks:Running task selinux... 2023-11-24T07:51:48.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2023-11-24T07:51:48.157 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2023-11-24T07:51:48.161 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2023-11-24T07:51:48.282 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2023-11-24T07:51:48.287 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-24T07:51:48.287 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2023-11-24T07:51:48.356 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2023-11-24T07:51:48.357 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2023-11-24T07:51:48.419 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2023-11-24T07:51:48.420 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2023-11-24T07:51:48.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T07:51:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T07:51:48.519 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T07:51:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T07:51:48.616 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-24T07:51:48.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2023-11-24T07:51:48.788 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2023-11-24T07:51:48.926 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-24T07:51:48.936 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-24T07:51:48.936 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-24T07:51:48.954 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-24T07:51:48.955 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-24T08:01:44.058 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2023-11-24T08:01:44.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:01:44.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:01:44.229 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-11-24T08:01:44.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:01:44.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:01:44.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:01:44.526 DEBUG:teuthology.orchestra.run.smithi120:> true 2023-11-24T08:01:44.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:01:44.669 INFO:teuthology.run_tasks:Running task clock... 2023-11-24T08:01:44.679 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-24T08:01:44.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T08:01:44.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:01:44.688 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T08:01:44.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:01:44.787 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T08:01:44.792 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T08:01:44.838 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T08:01:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T08:01:44.936 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2023-11-24T08:01:44.939 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2023-11-24T08:01:44.977 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2023-11-24T08:01:44.982 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2023-11-24T08:01:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T08:01:45.034 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T08:01:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T08:01:45.089 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T08:01:45.213 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2023-11-24T08:01:45.222 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T08:01:45.222 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-11-24T08:01:45.223 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.223 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.223 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.223 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.223 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.232 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:01:45.233 INFO:teuthology.run_tasks:Running task install... 2023-11-24T08:01:45.242 DEBUG:teuthology.task.install:project ceph 2023-11-24T08:01:45.242 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}} 2023-11-24T08:01:45.242 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'} 2023-11-24T08:01:45.242 INFO:teuthology.task.install:Using flavor: default 2023-11-24T08:01:45.248 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-24T08:01:45.248 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-24T08:01:45.249 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T08:01:45.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T08:01:45.251 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T08:01:45.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T08:01:45.475 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/ 2023-11-24T08:01:45.475 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T08:01:45.486 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/ 2023-11-24T08:01:45.486 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T08:01:45.651 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T08:01:45.652 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:01:45.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T08:01:45.660 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T08:01:45.660 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:01:45.660 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T08:01:45.752 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T08:01:45.752 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T08:01:45.762 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T08:01:45.763 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T08:01:45.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T08:01:45.906 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T08:01:46.027 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T08:01:46.044 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T08:01:46.118 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2023-11-24T08:01:46.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2023-11-24T08:01:46.135 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2023-11-24T08:01:46.140 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2023-11-24T08:01:46.486 INFO:teuthology.orchestra.run.smithi107.stdout:73 files removed 2023-11-24T08:01:46.507 INFO:teuthology.orchestra.run.smithi120.stdout:73 files removed 2023-11-24T08:01:46.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T08:01:46.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T08:01:47.428 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 206 kB/s | 78 kB 00:00 2023-11-24T08:01:47.648 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 129 kB/s | 78 kB 00:00 2023-11-24T08:01:48.012 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 19 kB/s | 9.5 kB 00:00 2023-11-24T08:01:48.174 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2023-11-24T08:01:48.671 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2023-11-24T08:01:48.700 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2023-11-24T08:01:49.416 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.4 MB 00:00 2023-11-24T08:01:49.436 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.4 MB 00:00 2023-11-24T08:01:51.092 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2023-11-24T08:01:51.094 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2023-11-24T08:01:52.062 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-11-24T08:01:52.991 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 7.9 MB 00:01 2023-11-24T08:01:54.472 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 15 MB/s | 18 MB 00:01 2023-11-24T08:01:55.133 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 21 MB/s | 18 MB 00:00 2023-11-24T08:01:58.898 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 5.8 MB 00:00 2023-11-24T08:01:59.526 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 5.8 MB 00:00 2023-11-24T08:02:00.691 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-11-24T08:02:01.269 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2023-11-24T08:02:02.372 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-11-24T08:02:06.970 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 20 MB 00:05 2023-11-24T08:02:08.074 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-11-24T08:02:11.025 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-24T08:02:11.027 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-24T08:02:11.048 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-24T08:02:11.049 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-24T08:02:11.051 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-24T08:02:11.053 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-24T08:02:11.055 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-24T08:02:11.059 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-24T08:02:11.060 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-24T08:02:11.139 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-24T08:02:11.165 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 6.5 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 5.2 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 859 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 141 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 1.5 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 133 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 1.8 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 7.4 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 50 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 8.6 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 41 M 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 262 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 772 k 2023-11-24T08:02:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 217 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 18 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 31 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 719 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 126 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-24T08:02:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 162 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 319 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 298 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 99 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 85 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.0 M 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 170 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-24T08:02:11.168 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.3 M 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.0 M 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-24T08:02:11.169 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 19 M 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 28 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 2.1 M 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 243 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 4.4 M 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 18 M 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 15 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 24 k 2023-11-24T08:02:11.170 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 163 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-24T08:02:11.171 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 476 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 4.8 M 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-24T08:02:11.172 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 45 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 129 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-24T08:02:11.173 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-24T08:02:11.174 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-24T08:02:11.175 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-24T08:02:11.176 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:02:11.177 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout:Install 132 Packages 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 15 Packages 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 204 M 2023-11-24T08:02:11.178 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2023-11-24T08:02:12.059 INFO:teuthology.orchestra.run.smithi120.stdout:(1/147): ceph-18.0.0-7484.ga13ea5ff.el9.x86_64. 25 kB/s | 6.5 kB 00:00 2023-11-24T08:02:12.251 INFO:teuthology.orchestra.run.smithi120.stdout:(2/147): ceph-fuse-18.0.0-7484.ga13ea5ff.el9.x8 4.4 MB/s | 859 kB 00:00 2023-11-24T08:02:12.318 INFO:teuthology.orchestra.run.smithi120.stdout:(3/147): ceph-immutable-object-cache-18.0.0-748 2.1 MB/s | 141 kB 00:00 2023-11-24T08:02:12.485 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2023-11-24T08:02:12.551 INFO:teuthology.orchestra.run.smithi120.stdout:(4/147): ceph-mds-18.0.0-7484.ga13ea5ff.el9.x86 9.0 MB/s | 2.1 MB 00:00 2023-11-24T08:02:12.735 INFO:teuthology.orchestra.run.smithi120.stdout:(5/147): ceph-mgr-18.0.0-7484.ga13ea5ff.el9.x86 8.0 MB/s | 1.5 MB 00:00 2023-11-24T08:02:13.285 INFO:teuthology.orchestra.run.smithi120.stdout:(6/147): ceph-mon-18.0.0-7484.ga13ea5ff.el9.x86 8.0 MB/s | 4.4 MB 00:00 2023-11-24T08:02:13.394 INFO:teuthology.orchestra.run.smithi120.stdout:(7/147): ceph-base-18.0.0-7484.ga13ea5ff.el9.x8 3.3 MB/s | 5.2 MB 00:01 2023-11-24T08:02:13.652 INFO:teuthology.orchestra.run.smithi120.stdout:(8/147): ceph-common-18.0.0-7484.ga13ea5ff.el9. 10 MB/s | 19 MB 00:01 2023-11-24T08:02:13.711 INFO:teuthology.orchestra.run.smithi120.stdout:(9/147): ceph-selinux-18.0.0-7484.ga13ea5ff.el9 421 kB/s | 24 kB 00:00 2023-11-24T08:02:14.586 INFO:teuthology.orchestra.run.smithi120.stdout:(10/147): ceph-radosgw-18.0.0-7484.ga13ea5ff.el 7.3 MB/s | 8.6 MB 00:01 2023-11-24T08:02:14.686 INFO:teuthology.orchestra.run.smithi120.stdout:(11/147): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 2.1 MB/s | 217 kB 00:00 2023-11-24T08:02:14.770 INFO:teuthology.orchestra.run.smithi120.stdout:(12/147): libcephfs-devel-18.0.0-7484.ga13ea5ff 378 kB/s | 31 kB 00:00 2023-11-24T08:02:15.054 INFO:teuthology.orchestra.run.smithi120.stdout:(13/147): ceph-osd-18.0.0-7484.ga13ea5ff.el9.x8 10 MB/s | 18 MB 00:01 2023-11-24T08:02:15.106 INFO:teuthology.orchestra.run.smithi120.stdout:(14/147): libcephfs2-18.0.0-7484.ga13ea5ff.el9. 2.1 MB/s | 719 kB 00:00 2023-11-24T08:02:15.140 INFO:teuthology.orchestra.run.smithi120.stdout:(15/147): libcephsqlite-18.0.0-7484.ga13ea5ff.e 1.9 MB/s | 163 kB 00:00 2023-11-24T08:02:15.190 INFO:teuthology.orchestra.run.smithi120.stdout:(16/147): librados-devel-18.0.0-7484.ga13ea5ff. 1.5 MB/s | 126 kB 00:00 2023-11-24T08:02:15.248 INFO:teuthology.orchestra.run.smithi120.stdout:(17/147): libradosstriper1-18.0.0-7484.ga13ea5f 4.3 MB/s | 476 kB 00:00 2023-11-24T08:02:15.324 INFO:teuthology.orchestra.run.smithi120.stdout:(18/147): python3-ceph-argparse-18.0.0-7484.ga1 609 kB/s | 45 kB 00:00 2023-11-24T08:02:15.400 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-24T08:02:15.402 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-24T08:02:15.416 INFO:teuthology.orchestra.run.smithi120.stdout:(19/147): python3-ceph-common-18.0.0-7484.ga13e 1.4 MB/s | 129 kB 00:00 2023-11-24T08:02:15.424 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-24T08:02:15.425 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-24T08:02:15.426 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-24T08:02:15.428 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-24T08:02:15.430 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-24T08:02:15.434 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-24T08:02:15.436 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-24T08:02:15.500 INFO:teuthology.orchestra.run.smithi120.stdout:(20/147): python3-cephfs-18.0.0-7484.ga13ea5ff. 1.9 MB/s | 162 kB 00:00 2023-11-24T08:02:15.514 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-11-24T08:02:15.532 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-11-24T08:02:15.533 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-11-24T08:02:15.533 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-11-24T08:02:15.533 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-11-24T08:02:15.533 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-24T08:02:15.534 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 6.5 k 2023-11-24T08:02:15.534 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 5.2 M 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 859 k 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 141 k 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 1.5 M 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 133 k 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 1.8 M 2023-11-24T08:02:15.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 7.4 M 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 50 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 8.6 M 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 41 M 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 262 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 772 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 217 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 18 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-24T08:02:15.536 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 31 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 719 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 126 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 162 k 2023-11-24T08:02:15.537 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 319 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 298 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 99 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 85 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.0 M 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 170 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-24T08:02:15.538 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.3 M 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 3.0 M 2023-11-24T08:02:15.539 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 19 M 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 28 k 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 2.1 M 2023-11-24T08:02:15.540 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 243 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 4.4 M 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 18 M 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el9 ceph-noarch 15 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 24 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-24T08:02:15.541 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 163 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 476 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-24T08:02:15.542 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 4.8 M 2023-11-24T08:02:15.543 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-24T08:02:15.543 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-24T08:02:15.543 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-24T08:02:15.544 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 45 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el9 ceph 129 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-24T08:02:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-24T08:02:15.546 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-24T08:02:15.547 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-24T08:02:15.548 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-24T08:02:15.549 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-11-24T08:02:15.550 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-11-24T08:02:15.551 INFO:teuthology.orchestra.run.smithi107.stdout:Install 132 Packages 2023-11-24T08:02:15.551 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 15 Packages 2023-11-24T08:02:15.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:02:15.551 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 204 M 2023-11-24T08:02:15.551 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-11-24T08:02:15.591 INFO:teuthology.orchestra.run.smithi120.stdout:(21/147): python3-rados-18.0.0-7484.ga13ea5ff.e 3.4 MB/s | 319 kB 00:00 2023-11-24T08:02:15.683 INFO:teuthology.orchestra.run.smithi120.stdout:(22/147): python3-rbd-18.0.0-7484.ga13ea5ff.el9 3.2 MB/s | 298 kB 00:00 2023-11-24T08:02:15.785 INFO:teuthology.orchestra.run.smithi120.stdout:(23/147): librgw2-18.0.0-7484.ga13ea5ff.el9.x86 8.1 MB/s | 4.8 MB 00:00 2023-11-24T08:02:15.817 INFO:teuthology.orchestra.run.smithi120.stdout:(24/147): python3-rgw-18.0.0-7484.ga13ea5ff.el9 750 kB/s | 99 kB 00:00 2023-11-24T08:02:15.867 INFO:teuthology.orchestra.run.smithi120.stdout:(25/147): rbd-fuse-18.0.0-7484.ga13ea5ff.el9.x8 1.0 MB/s | 85 kB 00:00 2023-11-24T08:02:15.976 INFO:teuthology.orchestra.run.smithi120.stdout:(26/147): rbd-nbd-18.0.0-7484.ga13ea5ff.el9.x86 1.5 MB/s | 170 kB 00:00 2023-11-24T08:02:16.077 INFO:teuthology.orchestra.run.smithi120.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7484.g 283 kB/s | 28 kB 00:00 2023-11-24T08:02:16.152 INFO:teuthology.orchestra.run.smithi120.stdout:(28/147): rbd-mirror-18.0.0-7484.ga13ea5ff.el9. 9.0 MB/s | 3.0 MB 00:00 2023-11-24T08:02:16.177 INFO:teuthology.orchestra.run.smithi120.stdout:(29/147): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 1.3 MB/s | 133 kB 00:00 2023-11-24T08:02:16.261 INFO:teuthology.orchestra.run.smithi107.stdout:(1/147): ceph-18.0.0-7484.ga13ea5ff.el9.x86_64. 19 kB/s | 6.5 kB 00:00 2023-11-24T08:02:16.711 INFO:teuthology.orchestra.run.smithi120.stdout:(30/147): ceph-mgr-dashboard-18.0.0-7484.ga13ea 3.2 MB/s | 1.8 MB 00:00 2023-11-24T08:02:16.861 INFO:teuthology.orchestra.run.smithi120.stdout:(31/147): ceph-mgr-diskprediction-local-18.0.0- 11 MB/s | 7.4 MB 00:00 2023-11-24T08:02:16.895 INFO:teuthology.orchestra.run.smithi120.stdout:(32/147): ceph-mgr-modules-core-18.0.0-7484.ga1 1.3 MB/s | 243 kB 00:00 2023-11-24T08:02:16.903 INFO:teuthology.orchestra.run.smithi107.stdout:(2/147): ceph-fuse-18.0.0-7484.ga13ea5ff.el9.x8 1.3 MB/s | 859 kB 00:00 2023-11-24T08:02:16.912 INFO:teuthology.orchestra.run.smithi120.stdout:(33/147): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 1.0 MB/s | 50 kB 00:00 2023-11-24T08:02:16.986 INFO:teuthology.orchestra.run.smithi120.stdout:(34/147): ceph-prometheus-alerts-18.0.0-7484.ga 169 kB/s | 15 kB 00:00 2023-11-24T08:02:16.994 INFO:teuthology.orchestra.run.smithi107.stdout:(3/147): ceph-immutable-object-cache-18.0.0-748 1.5 MB/s | 141 kB 00:00 2023-11-24T08:02:17.021 INFO:teuthology.orchestra.run.smithi120.stdout:(35/147): ceph-volume-18.0.0-7484.ga13ea5ff.el9 2.4 MB/s | 262 kB 00:00 2023-11-24T08:02:17.403 INFO:teuthology.orchestra.run.smithi107.stdout:(4/147): ceph-mds-18.0.0-7484.ga13ea5ff.el9.x86 5.1 MB/s | 2.1 MB 00:00 2023-11-24T08:02:17.496 INFO:teuthology.orchestra.run.smithi120.stdout:(36/147): ceph-test-18.0.0-7484.ga13ea5ff.el9.x 11 MB/s | 41 MB 00:03 2023-11-24T08:02:17.522 INFO:teuthology.orchestra.run.smithi120.stdout:(37/147): cephfs-top-18.0.0-7484.ga13ea5ff.el9. 36 kB/s | 18 kB 00:00 2023-11-24T08:02:17.564 INFO:teuthology.orchestra.run.smithi120.stdout:(38/147): cephadm-18.0.0-7484.ga13ea5ff.el9.noa 1.3 MB/s | 772 kB 00:00 2023-11-24T08:02:17.603 INFO:teuthology.orchestra.run.smithi107.stdout:(5/147): ceph-mgr-18.0.0-7484.ga13ea5ff.el9.x86 7.3 MB/s | 1.5 MB 00:00 2023-11-24T08:02:17.847 INFO:teuthology.orchestra.run.smithi120.stdout:(39/147): boost-program-options-1.75.0-8.el9.x8 622 kB/s | 106 kB 00:00 2023-11-24T08:02:17.914 INFO:teuthology.orchestra.run.smithi120.stdout:(40/147): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 920 kB 00:00 2023-11-24T08:02:17.986 INFO:teuthology.orchestra.run.smithi107.stdout:(6/147): ceph-mon-18.0.0-7484.ga13ea5ff.el9.x86 11 MB/s | 4.4 MB 00:00 2023-11-24T08:02:18.023 INFO:teuthology.orchestra.run.smithi120.stdout:(41/147): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 662 kB 00:00 2023-11-24T08:02:18.048 INFO:teuthology.orchestra.run.smithi120.stdout:(42/147): elfutils-libelf-devel-0.190-1.el9.x86 110 kB/s | 21 kB 00:00 2023-11-24T08:02:18.074 INFO:teuthology.orchestra.run.smithi120.stdout:(43/147): flex-2.6.4-9.el9.x86_64.rpm 1.9 MB/s | 307 kB 00:00 2023-11-24T08:02:18.095 INFO:teuthology.orchestra.run.smithi107.stdout:(7/147): ceph-base-18.0.0-7484.ga13ea5ff.el9.x8 2.4 MB/s | 5.2 MB 00:02 2023-11-24T08:02:18.099 INFO:teuthology.orchestra.run.smithi120.stdout:(44/147): flexiblas-3.0.4-8.el9.x86_64.rpm 440 kB/s | 32 kB 00:00 2023-11-24T08:02:18.166 INFO:teuthology.orchestra.run.smithi120.stdout:(45/147): flexiblas-openblas-openmp-3.0.4-8.el9 190 kB/s | 17 kB 00:00 2023-11-24T08:02:18.291 INFO:teuthology.orchestra.run.smithi120.stdout:(46/147): jq-1.6-16.el9.x86_64.rpm 974 kB/s | 186 kB 00:00 2023-11-24T08:02:18.341 INFO:teuthology.orchestra.run.smithi120.stdout:(47/147): libaio-devel-0.3.111-13.el9.x86_64.rp 58 kB/s | 10 kB 00:00 2023-11-24T08:02:18.458 INFO:teuthology.orchestra.run.smithi120.stdout:(48/147): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.8 MB/s | 3.0 MB 00:00 2023-11-24T08:02:18.492 INFO:teuthology.orchestra.run.smithi120.stdout:(49/147): libpmemobj-1.12.1-1.el9.x86_64.rpm 814 kB/s | 159 kB 00:00 2023-11-24T08:02:18.517 INFO:teuthology.orchestra.run.smithi120.stdout:(50/147): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 44 kB 00:00 2023-11-24T08:02:18.568 INFO:teuthology.orchestra.run.smithi120.stdout:(51/147): librdkafka-1.6.1-102.el9.x86_64.rpm 6.1 MB/s | 661 kB 00:00 2023-11-24T08:02:18.593 INFO:teuthology.orchestra.run.smithi120.stdout:(52/147): libuuid-devel-2.37.4-15.el9.x86_64.rp 357 kB/s | 24 kB 00:00 2023-11-24T08:02:18.635 INFO:teuthology.orchestra.run.smithi120.stdout:(53/147): libtool-2.4.6-45.el9.x86_64.rpm 5.1 MB/s | 578 kB 00:00 2023-11-24T08:02:18.660 INFO:teuthology.orchestra.run.smithi120.stdout:(54/147): libzstd-devel-1.5.1-2.el9.x86_64.rpm 521 kB/s | 46 kB 00:00 2023-11-24T08:02:18.694 INFO:teuthology.orchestra.run.smithi120.stdout:(55/147): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 282 kB 00:00 2023-11-24T08:02:18.727 INFO:teuthology.orchestra.run.smithi120.stdout:(56/147): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-11-24T08:02:18.753 INFO:teuthology.orchestra.run.smithi120.stdout:(57/147): openblas-0.3.21-2.el9.x86_64.rpm 385 kB/s | 34 kB 00:00 2023-11-24T08:02:18.994 INFO:teuthology.orchestra.run.smithi120.stdout:(58/147): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2023-11-24T08:02:19.062 INFO:teuthology.orchestra.run.smithi120.stdout:(59/147): postgresql-13.11-1.el9.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2023-11-24T08:02:19.103 INFO:teuthology.orchestra.run.smithi120.stdout:(60/147): postgresql-contrib-13.11-1.el9.x86_64 2.3 MB/s | 811 kB 00:00 2023-11-24T08:02:19.120 INFO:teuthology.orchestra.run.smithi120.stdout:(61/147): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2023-11-24T08:02:19.271 INFO:teuthology.orchestra.run.smithi120.stdout:(62/147): python3-jinja2-2.11.3-4.el9.noarch.rp 1.5 MB/s | 228 kB 00:00 2023-11-24T08:02:19.446 INFO:teuthology.orchestra.run.smithi120.stdout:(63/147): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-11-24T08:02:19.453 INFO:teuthology.orchestra.run.smithi107.stdout:(8/147): ceph-osd-18.0.0-7484.ga13ea5ff.el9.x86 12 MB/s | 18 MB 00:01 2023-11-24T08:02:19.512 INFO:teuthology.orchestra.run.smithi107.stdout:(9/147): ceph-selinux-18.0.0-7484.ga13ea5ff.el9 421 kB/s | 24 kB 00:00 2023-11-24T08:02:19.563 INFO:teuthology.orchestra.run.smithi120.stdout:(64/147): postgresql-server-13.11-1.el9.x86_64. 12 MB/s | 5.7 MB 00:00 2023-11-24T08:02:19.580 INFO:teuthology.orchestra.run.smithi120.stdout:(65/147): python3-jsonpatch-1.21-16.el9.noarch. 80 kB/s | 24 kB 00:00 2023-11-24T08:02:19.605 INFO:teuthology.orchestra.run.smithi120.stdout:(66/147): python3-jsonpointer-2.0-4.el9.noarch. 115 kB/s | 18 kB 00:00 2023-11-24T08:02:19.622 INFO:teuthology.orchestra.run.smithi120.stdout:(67/147): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-11-24T08:02:19.647 INFO:teuthology.orchestra.run.smithi120.stdout:(68/147): python3-markupsafe-1.1.1-12.el9.x86_6 479 kB/s | 32 kB 00:00 2023-11-24T08:02:19.662 INFO:teuthology.orchestra.run.smithi107.stdout:(10/147): ceph-radosgw-18.0.0-7484.ga13ea5ff.el 5.5 MB/s | 8.6 MB 00:01 2023-11-24T08:02:19.746 INFO:teuthology.orchestra.run.smithi107.stdout:(11/147): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 2.6 MB/s | 217 kB 00:00 2023-11-24T08:02:19.790 INFO:teuthology.orchestra.run.smithi120.stdout:(69/147): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-11-24T08:02:19.823 INFO:teuthology.orchestra.run.smithi120.stdout:(70/147): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2023-11-24T08:02:19.848 INFO:teuthology.orchestra.run.smithi120.stdout:(71/147): python3-oauthlib-3.1.1-5.el9.noarch.r 938 kB/s | 174 kB 00:00 2023-11-24T08:02:19.874 INFO:teuthology.orchestra.run.smithi120.stdout:(72/147): python3-prettytable-0.7.2-27.el9.noar 516 kB/s | 41 kB 00:00 2023-11-24T08:02:19.907 INFO:teuthology.orchestra.run.smithi120.stdout:(73/147): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-11-24T08:02:19.932 INFO:teuthology.orchestra.run.smithi120.stdout:(74/147): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2023-11-24T08:02:19.958 INFO:teuthology.orchestra.run.smithi120.stdout:(75/147): python3-pytz-2021.1-5.el9.noarch.rpm 591 kB/s | 47 kB 00:00 2023-11-24T08:02:19.979 INFO:teuthology.orchestra.run.smithi107.stdout:(12/147): ceph-common-18.0.0-7484.ga13ea5ff.el9 4.7 MB/s | 19 MB 00:04 2023-11-24T08:02:19.991 INFO:teuthology.orchestra.run.smithi120.stdout:(76/147): python3-requests-oauthlib-1.3.0-12.el 749 kB/s | 43 kB 00:00 2023-11-24T08:02:20.005 INFO:teuthology.orchestra.run.smithi107.stdout:(13/147): libcephfs-devel-18.0.0-7484.ga13ea5ff 121 kB/s | 31 kB 00:00 2023-11-24T08:02:20.072 INFO:teuthology.orchestra.run.smithi107.stdout:(14/147): libcephsqlite-18.0.0-7484.ga13ea5ff.e 2.4 MB/s | 163 kB 00:00 2023-11-24T08:02:20.113 INFO:teuthology.orchestra.run.smithi107.stdout:(15/147): libcephfs2-18.0.0-7484.ga13ea5ff.el9. 5.3 MB/s | 719 kB 00:00 2023-11-24T08:02:20.172 INFO:teuthology.orchestra.run.smithi107.stdout:(16/147): librados-devel-18.0.0-7484.ga13ea5ff. 1.2 MB/s | 126 kB 00:00 2023-11-24T08:02:20.214 INFO:teuthology.orchestra.run.smithi107.stdout:(17/147): libradosstriper1-18.0.0-7484.ga13ea5f 4.7 MB/s | 476 kB 00:00 2023-11-24T08:02:20.273 INFO:teuthology.orchestra.run.smithi107.stdout:(18/147): python3-ceph-argparse-18.0.0-7484.ga1 787 kB/s | 45 kB 00:00 2023-11-24T08:02:20.334 INFO:teuthology.orchestra.run.smithi120.stdout:(77/147): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2023-11-24T08:02:20.358 INFO:teuthology.orchestra.run.smithi120.stdout:(78/147): python3-toml-0.10.2-6.el9.noarch.rpm 99 kB/s | 39 kB 00:00 2023-11-24T08:02:20.373 INFO:teuthology.orchestra.run.smithi107.stdout:(19/147): python3-ceph-common-18.0.0-7484.ga13e 1.3 MB/s | 129 kB 00:00 2023-11-24T08:02:20.400 INFO:teuthology.orchestra.run.smithi120.stdout:(79/147): socat-1.7.4.1-5.el9.x86_64.rpm 759 kB/s | 300 kB 00:00 2023-11-24T08:02:20.426 INFO:teuthology.orchestra.run.smithi120.stdout:(80/147): uuid-1.6.2-55.el9.x86_64.rpm 878 kB/s | 56 kB 00:00 2023-11-24T08:02:20.451 INFO:teuthology.orchestra.run.smithi120.stdout:(81/147): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 955 kB/s | 85 kB 00:00 2023-11-24T08:02:20.465 INFO:teuthology.orchestra.run.smithi107.stdout:(20/147): python3-cephfs-18.0.0-7484.ga13ea5ff. 1.8 MB/s | 162 kB 00:00 2023-11-24T08:02:20.476 INFO:teuthology.orchestra.run.smithi120.stdout:(82/147): xmlstarlet-1.6.1-20.el9.x86_64.rpm 852 kB/s | 63 kB 00:00 2023-11-24T08:02:20.502 INFO:teuthology.orchestra.run.smithi120.stdout:(83/147): zlib-devel-1.2.11-40.el9.x86_64.rpm 904 kB/s | 44 kB 00:00 2023-11-24T08:02:20.527 INFO:teuthology.orchestra.run.smithi120.stdout:(84/147): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2023-11-24T08:02:20.569 INFO:teuthology.orchestra.run.smithi120.stdout:(85/147): libgfortran-11.4.1-2.3.el9.x86_64.rpm 8.7 MB/s | 803 kB 00:00 2023-11-24T08:02:20.582 INFO:teuthology.orchestra.run.smithi107.stdout:(21/147): python3-rados-18.0.0-7484.ga13ea5ff.e 2.7 MB/s | 319 kB 00:00 2023-11-24T08:02:20.602 INFO:teuthology.orchestra.run.smithi120.stdout:(86/147): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.8 MB/s | 195 kB 00:00 2023-11-24T08:02:20.628 INFO:teuthology.orchestra.run.smithi120.stdout:(87/147): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 241 kB 00:00 2023-11-24T08:02:20.686 INFO:teuthology.orchestra.run.smithi120.stdout:(88/147): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2023-11-24T08:02:20.690 INFO:teuthology.orchestra.run.smithi107.stdout:(22/147): python3-rbd-18.0.0-7484.ga13ea5ff.el9 2.7 MB/s | 298 kB 00:00 2023-11-24T08:02:20.712 INFO:teuthology.orchestra.run.smithi120.stdout:(89/147): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2023-11-24T08:02:20.737 INFO:teuthology.orchestra.run.smithi120.stdout:(90/147): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2023-11-24T08:02:20.762 INFO:teuthology.orchestra.run.smithi120.stdout:(91/147): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-11-24T08:02:20.782 INFO:teuthology.orchestra.run.smithi107.stdout:(23/147): python3-rgw-18.0.0-7484.ga13ea5ff.el9 1.1 MB/s | 99 kB 00:00 2023-11-24T08:02:20.796 INFO:teuthology.orchestra.run.smithi120.stdout:(92/147): python3-urllib3-1.26.5-4.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2023-11-24T08:02:20.941 INFO:teuthology.orchestra.run.smithi107.stdout:(24/147): librgw2-18.0.0-7484.ga13ea5ff.el9.x86 6.2 MB/s | 4.8 MB 00:00 2023-11-24T08:02:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:(25/147): rbd-fuse-18.0.0-7484.ga13ea5ff.el9.x8 466 kB/s | 85 kB 00:00 2023-11-24T08:02:20.971 INFO:teuthology.orchestra.run.smithi120.stdout:(93/147): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2023-11-24T08:02:21.050 INFO:teuthology.orchestra.run.smithi107.stdout:(26/147): rbd-nbd-18.0.0-7484.ga13ea5ff.el9.x86 2.0 MB/s | 170 kB 00:00 2023-11-24T08:02:21.142 INFO:teuthology.orchestra.run.smithi107.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7484.g 308 kB/s | 28 kB 00:00 2023-11-24T08:02:21.273 INFO:teuthology.orchestra.run.smithi107.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 1.0 MB/s | 133 kB 00:00 2023-11-24T08:02:21.417 INFO:teuthology.orchestra.run.smithi107.stdout:(29/147): rbd-mirror-18.0.0-7484.ga13ea5ff.el9. 6.3 MB/s | 3.0 MB 00:00 2023-11-24T08:02:21.488 INFO:teuthology.orchestra.run.smithi120.stdout:(94/147): gperftools-libs-2.9.1-2.el9.x86_64.rp 447 kB/s | 309 kB 00:00 2023-11-24T08:02:21.597 INFO:teuthology.orchestra.run.smithi120.stdout:(95/147): libarrow-doc-9.0.0-9.el9.noarch.rpm 227 kB/s | 24 kB 00:00 2023-11-24T08:02:21.651 INFO:teuthology.orchestra.run.smithi107.stdout:(30/147): ceph-mgr-dashboard-18.0.0-7484.ga13ea 4.7 MB/s | 1.8 MB 00:00 2023-11-24T08:02:21.743 INFO:teuthology.orchestra.run.smithi107.stdout:(31/147): ceph-mgr-modules-core-18.0.0-7484.ga1 2.6 MB/s | 243 kB 00:00 2023-11-24T08:02:21.747 INFO:teuthology.orchestra.run.smithi120.stdout:(96/147): openssl-devel-3.0.7-25.el9.x86_64.rpm 4.1 MB/s | 4.1 MB 00:01 2023-11-24T08:02:21.773 INFO:teuthology.orchestra.run.smithi120.stdout:(97/147): liboath-2.6.7-2.el9.x86_64.rpm 279 kB/s | 49 kB 00:00 2023-11-24T08:02:21.873 INFO:teuthology.orchestra.run.smithi120.stdout:(98/147): libunwind-1.6.2-1.el9.x86_64.rpm 564 kB/s | 67 kB 00:00 2023-11-24T08:02:22.227 INFO:teuthology.orchestra.run.smithi107.stdout:(32/147): ceph-test-18.0.0-7484.ga13ea5ff.el9.x 15 MB/s | 41 MB 00:02 2023-11-24T08:02:22.252 INFO:teuthology.orchestra.run.smithi107.stdout:(33/147): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 99 kB/s | 50 kB 00:00 2023-11-24T08:02:22.257 INFO:teuthology.orchestra.run.smithi120.stdout:(99/147): libarrow-9.0.0-9.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2023-11-24T08:02:22.277 INFO:teuthology.orchestra.run.smithi107.stdout:(34/147): ceph-prometheus-alerts-18.0.0-7484.ga 309 kB/s | 15 kB 00:00 2023-11-24T08:02:22.307 INFO:teuthology.orchestra.run.smithi120.stdout:(100/147): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.5 MB/s | 837 kB 00:00 2023-11-24T08:02:22.310 INFO:teuthology.orchestra.run.smithi107.stdout:(35/147): ceph-volume-18.0.0-7484.ga13ea5ff.el9 4.5 MB/s | 262 kB 00:00 2023-11-24T08:02:22.341 INFO:teuthology.orchestra.run.smithi120.stdout:(101/147): python3-asyncssh-2.13.2-3.el9.noarch 1.2 MB/s | 548 kB 00:00 2023-11-24T08:02:22.361 INFO:teuthology.orchestra.run.smithi107.stdout:(36/147): cephfs-top-18.0.0-7484.ga13ea5ff.el9. 371 kB/s | 18 kB 00:00 2023-11-24T08:02:22.366 INFO:teuthology.orchestra.run.smithi120.stdout:(102/147): python3-bcrypt-3.2.2-1.el9.x86_64.rp 428 kB/s | 43 kB 00:00 2023-11-24T08:02:22.403 INFO:teuthology.orchestra.run.smithi107.stdout:(37/147): cephadm-18.0.0-7484.ga13ea5ff.el9.noa 6.1 MB/s | 772 kB 00:00 2023-11-24T08:02:22.449 INFO:teuthology.orchestra.run.smithi120.stdout:(103/147): python3-cachetools-4.2.4-1.el9.noarc 300 kB/s | 32 kB 00:00 2023-11-24T08:02:22.474 INFO:teuthology.orchestra.run.smithi120.stdout:(104/147): python3-certifi-2021.10.8-2.el9.noar 113 kB/s | 15 kB 00:00 2023-11-24T08:02:22.508 INFO:teuthology.orchestra.run.smithi120.stdout:(105/147): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2023-11-24T08:02:22.592 INFO:teuthology.orchestra.run.smithi120.stdout:(106/147): python3-cherrypy-18.6.1-2.el9.noarch 3.1 MB/s | 358 kB 00:00 2023-11-24T08:02:22.625 INFO:teuthology.orchestra.run.smithi120.stdout:(107/147): python3-google-auth-2.23.4-1.el9.noa 1.3 MB/s | 197 kB 00:00 2023-11-24T08:02:22.628 INFO:teuthology.orchestra.run.smithi107.stdout:(38/147): ceph-mgr-diskprediction-local-18.0.0- 6.1 MB/s | 7.4 MB 00:01 2023-11-24T08:02:22.651 INFO:teuthology.orchestra.run.smithi120.stdout:(108/147): python3-jaraco-8.2.1-3.el9.noarch.rp 75 kB/s | 11 kB 00:00 2023-11-24T08:02:22.692 INFO:teuthology.orchestra.run.smithi120.stdout:(109/147): python3-jaraco-classes-3.2.1-5.el9.n 180 kB/s | 18 kB 00:00 2023-11-24T08:02:22.753 INFO:teuthology.orchestra.run.smithi120.stdout:(110/147): python3-jaraco-collections-3.0.0-8.e 233 kB/s | 23 kB 00:00 2023-11-24T08:02:22.776 INFO:teuthology.orchestra.run.smithi120.stdout:(111/147): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2023-11-24T08:02:22.795 INFO:teuthology.orchestra.run.smithi107.stdout:(39/147): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 920 kB 00:00 2023-11-24T08:02:22.802 INFO:teuthology.orchestra.run.smithi120.stdout:(112/147): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2023-11-24T08:02:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:(40/147): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 662 kB 00:00 2023-11-24T08:02:22.877 INFO:teuthology.orchestra.run.smithi120.stdout:(113/147): python3-jwt+crypto-2.4.0-1.el9.noarc 91 kB/s | 9.0 kB 00:00 2023-11-24T08:02:22.895 INFO:teuthology.orchestra.run.smithi107.stdout:(41/147): boost-program-options-1.75.0-8.el9.x8 401 kB/s | 106 kB 00:00 2023-11-24T08:02:22.902 INFO:teuthology.orchestra.run.smithi120.stdout:(114/147): python3-jwt-2.4.0-1.el9.noarch.rpm 330 kB/s | 41 kB 00:00 2023-11-24T08:02:22.920 INFO:teuthology.orchestra.run.smithi107.stdout:(42/147): elfutils-libelf-devel-0.190-1.el9.x86 175 kB/s | 21 kB 00:00 2023-11-24T08:02:22.952 INFO:teuthology.orchestra.run.smithi120.stdout:(115/147): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2023-11-24T08:02:22.962 INFO:teuthology.orchestra.run.smithi107.stdout:(43/147): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2023-11-24T08:02:22.978 INFO:teuthology.orchestra.run.smithi120.stdout:(116/147): python3-logutils-0.3.5-21.el9.noarch 464 kB/s | 46 kB 00:00 2023-11-24T08:02:22.987 INFO:teuthology.orchestra.run.smithi107.stdout:(44/147): flexiblas-3.0.4-8.el9.x86_64.rpm 497 kB/s | 32 kB 00:00 2023-11-24T08:02:23.020 INFO:teuthology.orchestra.run.smithi120.stdout:(117/147): python3-more-itertools-8.12.0-2.el9. 734 kB/s | 79 kB 00:00 2023-11-24T08:02:23.053 INFO:teuthology.orchestra.run.smithi120.stdout:(118/147): python3-natsort-7.1.1-5.el9.noarch.r 576 kB/s | 58 kB 00:00 2023-11-24T08:02:23.118 INFO:teuthology.orchestra.run.smithi120.stdout:(119/147): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-11-24T08:02:23.129 INFO:teuthology.orchestra.run.smithi107.stdout:(45/147): flexiblas-openblas-openmp-3.0.4-8.el9 104 kB/s | 17 kB 00:00 2023-11-24T08:02:23.146 INFO:teuthology.orchestra.run.smithi120.stdout:(120/147): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2023-11-24T08:02:23.171 INFO:teuthology.orchestra.run.smithi120.stdout:(121/147): python3-pyOpenSSL-21.0.0-1.el9.noarc 770 kB/s | 90 kB 00:00 2023-11-24T08:02:23.213 INFO:teuthology.orchestra.run.smithi107.stdout:(46/147): jq-1.6-16.el9.x86_64.rpm 837 kB/s | 186 kB 00:00 2023-11-24T08:02:23.221 INFO:teuthology.orchestra.run.smithi120.stdout:(122/147): python3-repoze-lru-0.7-16.el9.noarch 300 kB/s | 31 kB 00:00 2023-11-24T08:02:23.255 INFO:teuthology.orchestra.run.smithi120.stdout:(123/147): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2023-11-24T08:02:23.280 INFO:teuthology.orchestra.run.smithi120.stdout:(124/147): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2023-11-24T08:02:23.297 INFO:teuthology.orchestra.run.smithi107.stdout:(47/147): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.0 MB/s | 3.0 MB 00:00 2023-11-24T08:02:23.322 INFO:teuthology.orchestra.run.smithi107.stdout:(48/147): libaio-devel-0.3.111-13.el9.x86_64.rp 54 kB/s | 10 kB 00:00 2023-11-24T08:02:23.330 INFO:teuthology.orchestra.run.smithi120.stdout:(125/147): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2023-11-24T08:02:23.355 INFO:teuthology.orchestra.run.smithi107.stdout:(49/147): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2023-11-24T08:02:23.364 INFO:teuthology.orchestra.run.smithi120.stdout:(126/147): python3-typing-extensions-4.4.0-2.el 473 kB/s | 51 kB 00:00 2023-11-24T08:02:23.374 INFO:teuthology.orchestra.run.smithi107.stdout:(50/147): librabbitmq-0.11.0-7.el9.x86_64.rpm 920 kB/s | 44 kB 00:00 2023-11-24T08:02:23.398 INFO:teuthology.orchestra.run.smithi120.stdout:(127/147): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-11-24T08:02:23.423 INFO:teuthology.orchestra.run.smithi107.stdout:(51/147): librdkafka-1.6.1-102.el9.x86_64.rpm 6.6 MB/s | 661 kB 00:00 2023-11-24T08:02:23.440 INFO:teuthology.orchestra.run.smithi120.stdout:(128/147): python3-websocket-client-1.2.3-2.el9 827 kB/s | 90 kB 00:00 2023-11-24T08:02:23.465 INFO:teuthology.orchestra.run.smithi107.stdout:(52/147): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 578 kB 00:00 2023-11-24T08:02:23.482 INFO:teuthology.orchestra.run.smithi107.stdout:(53/147): libuuid-devel-2.37.4-15.el9.x86_64.rp 230 kB/s | 24 kB 00:00 2023-11-24T08:02:23.490 INFO:teuthology.orchestra.run.smithi120.stdout:(129/147): python3-werkzeug-2.0.3-3.el9.1.noarc 3.4 MB/s | 427 kB 00:00 2023-11-24T08:02:23.499 INFO:teuthology.orchestra.run.smithi107.stdout:(54/147): libzstd-devel-1.5.1-2.el9.x86_64.rpm 663 kB/s | 46 kB 00:00 2023-11-24T08:02:23.516 INFO:teuthology.orchestra.run.smithi120.stdout:(130/147): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2023-11-24T08:02:23.549 INFO:teuthology.orchestra.run.smithi120.stdout:(131/147): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-11-24T08:02:23.565 INFO:teuthology.orchestra.run.smithi107.stdout:(55/147): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2023-11-24T08:02:23.590 INFO:teuthology.orchestra.run.smithi107.stdout:(56/147): openblas-0.3.21-2.el9.x86_64.rpm 388 kB/s | 34 kB 00:00 2023-11-24T08:02:23.625 INFO:teuthology.orchestra.run.smithi107.stdout:(57/147): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2023-11-24T08:02:23.674 INFO:teuthology.orchestra.run.smithi120.stdout:(132/147): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2023-11-24T08:02:23.725 INFO:teuthology.orchestra.run.smithi120.stdout:(133/147): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2023-11-24T08:02:23.833 INFO:teuthology.orchestra.run.smithi120.stdout:(134/147): NetworkManager-1.45.6-1.el9.x86_64.r 21 MB/s | 2.2 MB 00:00 2023-11-24T08:02:23.925 INFO:teuthology.orchestra.run.smithi120.stdout:(135/147): NetworkManager-libnm-1.45.6-1.el9.x8 19 MB/s | 1.8 MB 00:00 2023-11-24T08:02:23.926 INFO:teuthology.orchestra.run.smithi107.stdout:(58/147): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2023-11-24T08:02:23.983 INFO:teuthology.orchestra.run.smithi107.stdout:(59/147): postgresql-contrib-13.11-1.el9.x86_64 2.2 MB/s | 811 kB 00:00 2023-11-24T08:02:24.009 INFO:teuthology.orchestra.run.smithi120.stdout:(136/147): librados2-18.0.0-7484.ga13ea5ff.el9. 6.7 MB/s | 3.3 MB 00:00 2023-11-24T08:02:24.033 INFO:teuthology.orchestra.run.smithi107.stdout:(60/147): postgresql-13.11-1.el9.x86_64.rpm 3.5 MB/s | 1.5 MB 00:00 2023-11-24T08:02:24.034 INFO:teuthology.orchestra.run.smithi120.stdout:(137/147): NetworkManager-team-1.45.6-1.el9.x86 390 kB/s | 42 kB 00:00 2023-11-24T08:02:24.068 INFO:teuthology.orchestra.run.smithi120.stdout:(138/147): NetworkManager-tui-1.45.6-1.el9.x86_ 4.2 MB/s | 245 kB 00:00 2023-11-24T08:02:24.075 INFO:teuthology.orchestra.run.smithi107.stdout:(61/147): postgresql-private-libs-13.11-1.el9.x 936 kB/s | 134 kB 00:00 2023-11-24T08:02:24.094 INFO:teuthology.orchestra.run.smithi120.stdout:(139/147): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2023-11-24T08:02:24.110 INFO:teuthology.orchestra.run.smithi120.stdout:(140/147): elfutils-debuginfod-client-0.190-1.e 902 kB/s | 36 kB 00:00 2023-11-24T08:02:24.161 INFO:teuthology.orchestra.run.smithi120.stdout:(141/147): elfutils-libelf-0.190-1.el9.x86_64.r 2.9 MB/s | 192 kB 00:00 2023-11-24T08:02:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:(142/147): elfutils-libs-0.190-1.el9.x86_64.rpm 3.0 MB/s | 255 kB 00:00 2023-11-24T08:02:24.220 INFO:teuthology.orchestra.run.smithi120.stdout:(143/147): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 22 kB 00:00 2023-11-24T08:02:24.225 INFO:teuthology.orchestra.run.smithi107.stdout:(62/147): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2023-11-24T08:02:24.253 INFO:teuthology.orchestra.run.smithi120.stdout:(144/147): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2023-11-24T08:02:24.354 INFO:teuthology.orchestra.run.smithi120.stdout:(145/147): librbd1-18.0.0-7484.ga13ea5ff.el9.x8 3.8 MB/s | 3.0 MB 00:00 2023-11-24T08:02:24.367 INFO:teuthology.orchestra.run.smithi107.stdout:(63/147): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-11-24T08:02:24.393 INFO:teuthology.orchestra.run.smithi107.stdout:(64/147): python3-jinja2-2.11.3-4.el9.noarch.rp 755 kB/s | 228 kB 00:00 2023-11-24T08:02:24.417 INFO:teuthology.orchestra.run.smithi107.stdout:(65/147): python3-jsonpatch-1.21-16.el9.noarch. 137 kB/s | 24 kB 00:00 2023-11-24T08:02:24.459 INFO:teuthology.orchestra.run.smithi107.stdout:(66/147): python3-jsonpointer-2.0-4.el9.noarch. 449 kB/s | 18 kB 00:00 2023-11-24T08:02:24.485 INFO:teuthology.orchestra.run.smithi107.stdout:(67/147): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2023-11-24T08:02:24.496 INFO:teuthology.orchestra.run.smithi120.stdout:(146/147): openssl-3.0.7-25.el9.x86_64.rpm 4.5 MB/s | 1.2 MB 00:00 2023-11-24T08:02:24.510 INFO:teuthology.orchestra.run.smithi107.stdout:(68/147): python3-markupsafe-1.1.1-12.el9.x86_6 350 kB/s | 32 kB 00:00 2023-11-24T08:02:24.562 INFO:teuthology.orchestra.run.smithi120.stdout:(147/147): openssl-libs-3.0.7-25.el9.x86_64.rpm 7.0 MB/s | 2.2 MB 00:00 2023-11-24T08:02:24.576 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2023-11-24T08:02:24.576 INFO:teuthology.orchestra.run.smithi120.stdout:Total 15 MB/s | 204 MB 00:13 2023-11-24T08:02:24.668 INFO:teuthology.orchestra.run.smithi107.stdout:(69/147): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2023-11-24T08:02:24.702 INFO:teuthology.orchestra.run.smithi107.stdout:(70/147): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2023-11-24T08:02:24.727 INFO:teuthology.orchestra.run.smithi107.stdout:(71/147): python3-oauthlib-3.1.1-5.el9.noarch.r 869 kB/s | 174 kB 00:00 2023-11-24T08:02:24.752 INFO:teuthology.orchestra.run.smithi107.stdout:(72/147): python3-prettytable-0.7.2-27.el9.noar 495 kB/s | 41 kB 00:00 2023-11-24T08:02:24.777 INFO:teuthology.orchestra.run.smithi107.stdout:(73/147): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-11-24T08:02:24.796 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2023-11-24T08:02:24.803 INFO:teuthology.orchestra.run.smithi107.stdout:(74/147): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2023-11-24T08:02:24.828 INFO:teuthology.orchestra.run.smithi107.stdout:(75/147): python3-pytz-2021.1-5.el9.noarch.rpm 644 kB/s | 47 kB 00:00 2023-11-24T08:02:24.870 INFO:teuthology.orchestra.run.smithi107.stdout:(76/147): python3-requests-oauthlib-1.3.0-12.el 656 kB/s | 43 kB 00:00 2023-11-24T08:02:25.074 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2023-11-24T08:02:25.075 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2023-11-24T08:02:25.212 INFO:teuthology.orchestra.run.smithi107.stdout:(77/147): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2023-11-24T08:02:25.237 INFO:teuthology.orchestra.run.smithi107.stdout:(78/147): python3-toml-0.10.2-6.el9.noarch.rpm 98 kB/s | 39 kB 00:00 2023-11-24T08:02:25.271 INFO:teuthology.orchestra.run.smithi107.stdout:(79/147): socat-1.7.4.1-5.el9.x86_64.rpm 779 kB/s | 300 kB 00:00 2023-11-24T08:02:25.296 INFO:teuthology.orchestra.run.smithi107.stdout:(80/147): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2023-11-24T08:02:25.321 INFO:teuthology.orchestra.run.smithi107.stdout:(81/147): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.0 MB/s | 85 kB 00:00 2023-11-24T08:02:25.346 INFO:teuthology.orchestra.run.smithi107.stdout:(82/147): xmlstarlet-1.6.1-20.el9.x86_64.rpm 854 kB/s | 63 kB 00:00 2023-11-24T08:02:25.372 INFO:teuthology.orchestra.run.smithi107.stdout:(83/147): zlib-devel-1.2.11-40.el9.x86_64.rpm 906 kB/s | 44 kB 00:00 2023-11-24T08:02:25.430 INFO:teuthology.orchestra.run.smithi107.stdout:(84/147): fuse-2.9.9-15.el9.x86_64.rpm 737 kB/s | 79 kB 00:00 2023-11-24T08:02:25.472 INFO:teuthology.orchestra.run.smithi107.stdout:(85/147): libgfortran-11.4.1-2.3.el9.x86_64.rpm 6.4 MB/s | 803 kB 00:00 2023-11-24T08:02:25.514 INFO:teuthology.orchestra.run.smithi107.stdout:(86/147): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2023-11-24T08:02:25.539 INFO:teuthology.orchestra.run.smithi107.stdout:(87/147): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2023-11-24T08:02:25.598 INFO:teuthology.orchestra.run.smithi107.stdout:(88/147): python3-cryptography-36.0.1-4.el9.x86 9.5 MB/s | 1.1 MB 00:00 2023-11-24T08:02:25.623 INFO:teuthology.orchestra.run.smithi107.stdout:(89/147): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2023-11-24T08:02:25.648 INFO:teuthology.orchestra.run.smithi107.stdout:(90/147): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2023-11-24T08:02:25.674 INFO:teuthology.orchestra.run.smithi107.stdout:(91/147): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-11-24T08:02:25.707 INFO:teuthology.orchestra.run.smithi107.stdout:(92/147): python3-urllib3-1.26.5-4.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2023-11-24T08:02:26.024 INFO:teuthology.orchestra.run.smithi107.stdout:(93/147): python3-devel-3.9.17-2.el9.x86_64.rpm 706 kB/s | 245 kB 00:00 2023-11-24T08:02:26.225 INFO:teuthology.orchestra.run.smithi107.stdout:(94/147): gperftools-libs-2.9.1-2.el9.x86_64.rp 599 kB/s | 309 kB 00:00 2023-11-24T08:02:26.300 INFO:teuthology.orchestra.run.smithi107.stdout:(95/147): libarrow-doc-9.0.0-9.el9.noarch.rpm 329 kB/s | 24 kB 00:00 2023-11-24T08:02:26.391 INFO:teuthology.orchestra.run.smithi107.stdout:(96/147): liboath-2.6.7-2.el9.x86_64.rpm 535 kB/s | 49 kB 00:00 2023-11-24T08:02:26.533 INFO:teuthology.orchestra.run.smithi107.stdout:(97/147): openssl-devel-3.0.7-25.el9.x86_64.rpm 4.7 MB/s | 4.1 MB 00:00 2023-11-24T08:02:26.559 INFO:teuthology.orchestra.run.smithi107.stdout:(98/147): libunwind-1.6.2-1.el9.x86_64.rpm 406 kB/s | 67 kB 00:00 2023-11-24T08:02:26.767 INFO:teuthology.orchestra.run.smithi107.stdout:(99/147): python3-asyncssh-2.13.2-3.el9.noarch. 2.6 MB/s | 548 kB 00:00 2023-11-24T08:02:26.909 INFO:teuthology.orchestra.run.smithi107.stdout:(100/147): libarrow-9.0.0-9.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2023-11-24T08:02:26.959 INFO:teuthology.orchestra.run.smithi107.stdout:(101/147): python3-bcrypt-3.2.2-1.el9.x86_64.rp 228 kB/s | 43 kB 00:00 2023-11-24T08:02:27.010 INFO:teuthology.orchestra.run.smithi107.stdout:(102/147): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2023-11-24T08:02:27.060 INFO:teuthology.orchestra.run.smithi107.stdout:(103/147): python3-certifi-2021.10.8-2.el9.noar 151 kB/s | 15 kB 00:00 2023-11-24T08:02:27.093 INFO:teuthology.orchestra.run.smithi107.stdout:(104/147): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2023-11-24T08:02:27.144 INFO:teuthology.orchestra.run.smithi107.stdout:(105/147): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2023-11-24T08:02:27.177 INFO:teuthology.orchestra.run.smithi107.stdout:(106/147): python3-google-auth-2.23.4-1.el9.noa 2.3 MB/s | 197 kB 00:00 2023-11-24T08:02:27.236 INFO:teuthology.orchestra.run.smithi107.stdout:(107/147): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2023-11-24T08:02:27.295 INFO:teuthology.orchestra.run.smithi107.stdout:(108/147): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2023-11-24T08:02:27.326 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2023-11-24T08:02:27.326 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2023-11-24T08:02:27.337 INFO:teuthology.orchestra.run.smithi107.stdout:(109/147): python3-jaraco-collections-3.0.0-8.e 233 kB/s | 23 kB 00:00 2023-11-24T08:02:27.387 INFO:teuthology.orchestra.run.smithi107.stdout:(110/147): python3-jaraco-functools-3.5.0-2.el9 214 kB/s | 19 kB 00:00 2023-11-24T08:02:27.429 INFO:teuthology.orchestra.run.smithi107.stdout:(111/147): python3-jaraco-text-3.2.0-6.el9.noar 216 kB/s | 20 kB 00:00 2023-11-24T08:02:27.473 INFO:teuthology.orchestra.run.smithi107.stdout:(112/147): parquet-libs-9.0.0-9.el9.x86_64.rpm 895 kB/s | 837 kB 00:00 2023-11-24T08:02:27.513 INFO:teuthology.orchestra.run.smithi107.stdout:(113/147): python3-jwt+crypto-2.4.0-1.el9.noarc 72 kB/s | 9.0 kB 00:00 2023-11-24T08:02:27.555 INFO:teuthology.orchestra.run.smithi107.stdout:(114/147): python3-jwt-2.4.0-1.el9.noarch.rpm 328 kB/s | 41 kB 00:00 2023-11-24T08:02:27.646 INFO:teuthology.orchestra.run.smithi107.stdout:(115/147): python3-kubernetes-26.1.0-2.el9.noar 5.9 MB/s | 1.0 MB 00:00 2023-11-24T08:02:27.797 INFO:teuthology.orchestra.run.smithi107.stdout:(116/147): python3-logutils-0.3.5-21.el9.noarch 163 kB/s | 46 kB 00:00 2023-11-24T08:02:27.822 INFO:teuthology.orchestra.run.smithi107.stdout:(117/147): python3-more-itertools-8.12.0-2.el9. 296 kB/s | 79 kB 00:00 2023-11-24T08:02:27.864 INFO:teuthology.orchestra.run.smithi107.stdout:(118/147): python3-natsort-7.1.1-5.el9.noarch.r 265 kB/s | 58 kB 00:00 2023-11-24T08:02:27.915 INFO:teuthology.orchestra.run.smithi107.stdout:(119/147): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2023-11-24T08:02:27.940 INFO:teuthology.orchestra.run.smithi107.stdout:(120/147): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2023-11-24T08:02:28.008 INFO:teuthology.orchestra.run.smithi107.stdout:(121/147): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2023-11-24T08:02:28.057 INFO:teuthology.orchestra.run.smithi107.stdout:(122/147): python3-repoze-lru-0.7-16.el9.noarch 218 kB/s | 31 kB 00:00 2023-11-24T08:02:28.090 INFO:teuthology.orchestra.run.smithi107.stdout:(123/147): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2023-11-24T08:02:28.149 INFO:teuthology.orchestra.run.smithi107.stdout:(124/147): python3-rsa-4.9-2.el9.noarch.rpm 651 kB/s | 59 kB 00:00 2023-11-24T08:02:28.191 INFO:teuthology.orchestra.run.smithi107.stdout:(125/147): python3-tempora-5.0.0-2.el9.noarch.r 270 kB/s | 36 kB 00:00 2023-11-24T08:02:28.216 INFO:teuthology.orchestra.run.smithi107.stdout:(126/147): python3-typing-extensions-4.4.0-2.el 408 kB/s | 51 kB 00:00 2023-11-24T08:02:28.250 INFO:teuthology.orchestra.run.smithi107.stdout:(127/147): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2023-11-24T08:02:28.275 INFO:teuthology.orchestra.run.smithi107.stdout:(128/147): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2023-11-24T08:02:28.325 INFO:teuthology.orchestra.run.smithi107.stdout:(129/147): python3-zc-lockfile-2.0-10.el9.noarc 268 kB/s | 20 kB 00:00 2023-11-24T08:02:28.416 INFO:teuthology.orchestra.run.smithi107.stdout:(130/147): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2023-11-24T08:02:28.443 INFO:teuthology.orchestra.run.smithi107.stdout:(131/147): python3-werkzeug-2.0.3-3.el9.1.noarc 1.9 MB/s | 427 kB 00:00 2023-11-24T08:02:28.493 INFO:teuthology.orchestra.run.smithi107.stdout:(132/147): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2023-11-24T08:02:28.526 INFO:teuthology.orchestra.run.smithi107.stdout:(133/147): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-11-24T08:02:28.635 INFO:teuthology.orchestra.run.smithi107.stdout:(134/147): NetworkManager-1.45.6-1.el9.x86_64.r 21 MB/s | 2.2 MB 00:00 2023-11-24T08:02:28.719 INFO:teuthology.orchestra.run.smithi107.stdout:(135/147): NetworkManager-libnm-1.45.6-1.el9.x8 21 MB/s | 1.8 MB 00:00 2023-11-24T08:02:28.777 INFO:teuthology.orchestra.run.smithi107.stdout:(136/147): NetworkManager-team-1.45.6-1.el9.x86 733 kB/s | 42 kB 00:00 2023-11-24T08:02:28.869 INFO:teuthology.orchestra.run.smithi107.stdout:(137/147): librados2-18.0.0-7484.ga13ea5ff.el9. 6.8 MB/s | 3.3 MB 00:00 2023-11-24T08:02:28.903 INFO:teuthology.orchestra.run.smithi107.stdout:(138/147): NetworkManager-tui-1.45.6-1.el9.x86_ 1.9 MB/s | 245 kB 00:00 2023-11-24T08:02:28.928 INFO:teuthology.orchestra.run.smithi107.stdout:(139/147): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2023-11-24T08:02:28.945 INFO:teuthology.orchestra.run.smithi107.stdout:(140/147): elfutils-debuginfod-client-0.190-1.e 903 kB/s | 36 kB 00:00 2023-11-24T08:02:28.978 INFO:teuthology.orchestra.run.smithi107.stdout:(141/147): elfutils-libelf-0.190-1.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2023-11-24T08:02:29.012 INFO:teuthology.orchestra.run.smithi107.stdout:(142/147): elfutils-libs-0.190-1.el9.x86_64.rpm 3.8 MB/s | 255 kB 00:00 2023-11-24T08:02:29.029 INFO:teuthology.orchestra.run.smithi107.stdout:(143/147): libacl-2.3.1-4.el9.x86_64.rpm 465 kB/s | 22 kB 00:00 2023-11-24T08:02:29.063 INFO:teuthology.orchestra.run.smithi107.stdout:(144/147): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2023-11-24T08:02:29.163 INFO:teuthology.orchestra.run.smithi107.stdout:(145/147): librbd1-18.0.0-7484.ga13ea5ff.el9.x8 4.2 MB/s | 3.0 MB 00:00 2023-11-24T08:02:29.296 INFO:teuthology.orchestra.run.smithi107.stdout:(146/147): openssl-3.0.7-25.el9.x86_64.rpm 4.6 MB/s | 1.2 MB 00:00 2023-11-24T08:02:29.364 INFO:teuthology.orchestra.run.smithi107.stdout:(147/147): openssl-libs-3.0.7-25.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2023-11-24T08:02:29.378 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-11-24T08:02:29.378 INFO:teuthology.orchestra.run.smithi107.stdout:Total 15 MB/s | 204 MB 00:13 2023-11-24T08:02:29.601 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-11-24T08:02:29.879 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-11-24T08:02:29.879 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-11-24T08:02:32.215 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-11-24T08:02:32.215 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-11-24T08:02:33.433 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2023-11-24T08:02:33.605 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/162 2023-11-24T08:02:33.777 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/162 2023-11-24T08:02:33.927 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/162 2023-11-24T08:02:34.063 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/162 2023-11-24T08:02:34.097 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/162 2023-11-24T08:02:34.197 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/162 2023-11-24T08:02:34.608 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/162 2023-11-24T08:02:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:02:34.809 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:02:34.829 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:02:34.946 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:02:34.976 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 10/162 2023-11-24T08:02:35.033 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/162 2023-11-24T08:02:35.337 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/162 2023-11-24T08:02:35.424 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/162 2023-11-24T08:02:35.459 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 14/162 2023-11-24T08:02:35.621 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 14/162 2023-11-24T08:02:35.669 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 15/162 2023-11-24T08:02:35.752 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 15/162 2023-11-24T08:02:36.043 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/162 2023-11-24T08:02:36.200 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/162 2023-11-24T08:02:36.287 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/162 2023-11-24T08:02:36.477 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:36.947 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:36.983 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/162 2023-11-24T08:02:37.079 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/162 2023-11-24T08:02:37.155 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/162 2023-11-24T08:02:37.187 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/162 2023-11-24T08:02:37.240 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/162 2023-11-24T08:02:37.289 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/162 2023-11-24T08:02:37.392 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/162 2023-11-24T08:02:37.468 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/162 2023-11-24T08:02:37.500 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/162 2023-11-24T08:02:37.616 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/162 2023-11-24T08:02:37.654 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9. 29/162 2023-11-24T08:02:37.706 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 30/162 2023-11-24T08:02:37.810 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_6 31/162 2023-11-24T08:02:37.985 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/162 2023-11-24T08:02:38.048 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/162 2023-11-24T08:02:38.596 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-11-24T08:02:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/162 2023-11-24T08:02:38.812 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/162 2023-11-24T08:02:38.854 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/162 2023-11-24T08:02:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/162 2023-11-24T08:02:38.979 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/162 2023-11-24T08:02:39.052 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/162 2023-11-24T08:02:39.067 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/162 2023-11-24T08:02:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/162 2023-11-24T08:02:39.134 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/162 2023-11-24T08:02:39.189 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_6 38/162 2023-11-24T08:02:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/162 2023-11-24T08:02:39.271 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/162 2023-11-24T08:02:39.491 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/162 2023-11-24T08:02:39.564 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/162 2023-11-24T08:02:39.629 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/162 2023-11-24T08:02:39.685 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:02:39.712 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/162 2023-11-24T08:02:39.760 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 43/162 2023-11-24T08:02:39.836 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/162 2023-11-24T08:02:39.907 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:02:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/162 2023-11-24T08:02:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:02:39.968 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/162 2023-11-24T08:02:39.999 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/162 2023-11-24T08:02:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:02:40.065 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/162 2023-11-24T08:02:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 10/162 2023-11-24T08:02:40.123 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/162 2023-11-24T08:02:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/162 2023-11-24T08:02:40.182 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 50/162 2023-11-24T08:02:40.375 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 51/162 2023-11-24T08:02:40.417 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 52/162 2023-11-24T08:02:40.463 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 53/162 2023-11-24T08:02:40.505 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/162 2023-11-24T08:02:40.586 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/162 2023-11-24T08:02:40.590 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 54/162 2023-11-24T08:02:40.624 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 14/162 2023-11-24T08:02:40.643 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 55/162 2023-11-24T08:02:40.691 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 56/162 2023-11-24T08:02:40.725 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 57/162 2023-11-24T08:02:40.822 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 14/162 2023-11-24T08:02:40.858 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 15/162 2023-11-24T08:02:40.880 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 58/162 2023-11-24T08:02:40.929 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 59/162 2023-11-24T08:02:40.998 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 60/162 2023-11-24T08:02:41.009 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 15/162 2023-11-24T08:02:41.078 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 61/162 2023-11-24T08:02:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 62/162 2023-11-24T08:02:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/162 2023-11-24T08:02:41.392 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 63/162 2023-11-24T08:02:41.547 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/162 2023-11-24T08:02:41.564 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 64/162 2023-11-24T08:02:41.620 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 65/162 2023-11-24T08:02:41.645 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 66/162 2023-11-24T08:02:41.684 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 67/162 2023-11-24T08:02:41.694 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/162 2023-11-24T08:02:41.727 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 68/162 2023-11-24T08:02:41.764 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 69/162 2023-11-24T08:02:41.798 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 70/162 2023-11-24T08:02:41.845 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/162 2023-11-24T08:02:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:41.902 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/162 2023-11-24T08:02:41.949 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:41.956 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/162 2023-11-24T08:02:42.084 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/162 2023-11-24T08:02:42.127 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 75/162 2023-11-24T08:02:42.172 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/162 2023-11-24T08:02:42.236 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/162 2023-11-24T08:02:42.282 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/162 2023-11-24T08:02:42.428 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/162 2023-11-24T08:02:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/162 2023-11-24T08:02:42.472 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 80/162 2023-11-24T08:02:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/162 2023-11-24T08:02:42.614 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/162 2023-11-24T08:02:42.674 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/162 2023-11-24T08:02:42.687 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/162 2023-11-24T08:02:42.756 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/162 2023-11-24T08:02:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/162 2023-11-24T08:02:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/162 2023-11-24T08:02:42.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/162 2023-11-24T08:02:43.062 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/162 2023-11-24T08:02:43.116 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/162 2023-11-24T08:02:43.153 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/162 2023-11-24T08:02:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/162 2023-11-24T08:02:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9. 29/162 2023-11-24T08:02:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 30/162 2023-11-24T08:02:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_6 31/162 2023-11-24T08:02:43.662 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/162 2023-11-24T08:02:43.668 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/162 2023-11-24T08:02:43.719 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/162 2023-11-24T08:02:43.734 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/162 2023-11-24T08:02:43.755 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/162 2023-11-24T08:02:43.787 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/162 2023-11-24T08:02:44.028 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/162 2023-11-24T08:02:44.058 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/162 2023-11-24T08:02:44.156 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/162 2023-11-24T08:02:44.437 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/162 2023-11-24T08:02:44.542 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/162 2023-11-24T08:02:44.598 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/162 2023-11-24T08:02:44.677 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/162 2023-11-24T08:02:44.726 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_6 38/162 2023-11-24T08:02:44.808 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/162 2023-11-24T08:02:44.959 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/162 2023-11-24T08:02:45.058 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/162 2023-11-24T08:02:45.088 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/162 2023-11-24T08:02:45.143 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/162 2023-11-24T08:02:45.305 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/162 2023-11-24T08:02:45.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 43/162 2023-11-24T08:02:45.432 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/162 2023-11-24T08:02:45.537 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/162 2023-11-24T08:02:45.582 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/162 2023-11-24T08:02:45.627 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/162 2023-11-24T08:02:45.705 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/162 2023-11-24T08:02:45.769 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/162 2023-11-24T08:02:45.831 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 50/162 2023-11-24T08:02:46.040 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 51/162 2023-11-24T08:02:46.095 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 52/162 2023-11-24T08:02:46.145 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 53/162 2023-11-24T08:02:46.271 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 54/162 2023-11-24T08:02:46.322 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 55/162 2023-11-24T08:02:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 56/162 2023-11-24T08:02:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 57/162 2023-11-24T08:02:46.586 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 58/162 2023-11-24T08:02:46.633 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/162 2023-11-24T08:02:46.665 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 59/162 2023-11-24T08:02:46.736 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 60/162 2023-11-24T08:02:46.739 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/162 2023-11-24T08:02:46.770 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 94/162 2023-11-24T08:02:46.809 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 95/162 2023-11-24T08:02:46.817 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 61/162 2023-11-24T08:02:46.852 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/162 2023-11-24T08:02:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 97/162 2023-11-24T08:02:47.080 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 62/162 2023-11-24T08:02:47.144 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 63/162 2023-11-24T08:02:47.256 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/162 2023-11-24T08:02:47.289 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 99/162 2023-11-24T08:02:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 64/162 2023-11-24T08:02:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 65/162 2023-11-24T08:02:47.391 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 99/162 2023-11-24T08:02:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 66/162 2023-11-24T08:02:47.419 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 100/162 2023-11-24T08:02:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 67/162 2023-11-24T08:02:47.474 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/162 2023-11-24T08:02:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 68/162 2023-11-24T08:02:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 69/162 2023-11-24T08:02:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 70/162 2023-11-24T08:02:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/162 2023-11-24T08:02:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/162 2023-11-24T08:02:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/162 2023-11-24T08:02:47.949 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 102/162 2023-11-24T08:02:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/162 2023-11-24T08:02:47.983 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 103/162 2023-11-24T08:02:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 75/162 2023-11-24T08:02:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/162 2023-11-24T08:02:48.075 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 103/162 2023-11-24T08:02:48.103 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 104/162 2023-11-24T08:02:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/162 2023-11-24T08:02:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/162 2023-11-24T08:02:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/162 2023-11-24T08:02:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 80/162 2023-11-24T08:02:48.546 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/162 2023-11-24T08:02:48.621 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/162 2023-11-24T08:02:49.546 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/162 2023-11-24T08:02:49.622 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/162 2023-11-24T08:02:49.656 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/162 2023-11-24T08:02:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/162 2023-11-24T08:02:49.936 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/162 2023-11-24T08:02:49.977 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/162 2023-11-24T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/162 2023-11-24T08:02:50.559 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:50.591 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:50.909 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/162 2023-11-24T08:02:51.047 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/162 2023-11-24T08:02:51.212 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:51.253 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 106/162 2023-11-24T08:02:51.435 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 106/162 2023-11-24T08:02:51.436 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-24T08:02:51.436 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-24T08:02:51.436 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:02:51.480 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 107/162 2023-11-24T08:02:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/162 2023-11-24T08:02:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/162 2023-11-24T08:02:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 94/162 2023-11-24T08:02:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 95/162 2023-11-24T08:02:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/162 2023-11-24T08:02:52.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 97/162 2023-11-24T08:02:53.215 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/162 2023-11-24T08:02:53.287 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 99/162 2023-11-24T08:02:53.414 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 99/162 2023-11-24T08:02:53.447 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 100/162 2023-11-24T08:02:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/162 2023-11-24T08:02:54.044 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 102/162 2023-11-24T08:02:54.078 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 103/162 2023-11-24T08:02:54.169 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 103/162 2023-11-24T08:02:54.196 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 104/162 2023-11-24T08:02:56.670 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:56.726 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 105/162 2023-11-24T08:02:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 106/162 2023-11-24T08:02:57.562 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 106/162 2023-11-24T08:02:57.562 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-24T08:02:57.563 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-24T08:02:57.563 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:02:57.622 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 107/162 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 107/162 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2023-11-24T08:03:03.984 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2023-11-24T08:03:03.985 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2023-11-24T08:03:03.985 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:04.196 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 108/162 2023-11-24T08:03:04.248 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 108/162 2023-11-24T08:03:04.249 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:04.249 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-24T08:03:04.249 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:05.985 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 109/162 2023-11-24T08:03:06.070 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 109/162 2023-11-24T08:03:06.070 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:06.071 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-24T08:03:06.071 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-24T08:03:06.071 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-24T08:03:06.071 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:06.293 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 110/162 2023-11-24T08:03:06.369 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 110/162 2023-11-24T08:03:06.369 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:06.369 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-24T08:03:06.370 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-24T08:03:06.370 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-24T08:03:06.370 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:06.852 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 111/162 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 111/162 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-24T08:03:06.920 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:07.038 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 112/162 2023-11-24T08:03:07.670 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 113/162 2023-11-24T08:03:07.739 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 113/162 2023-11-24T08:03:07.786 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 114/162 2023-11-24T08:03:07.884 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 115/162 2023-11-24T08:03:07.910 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 116/162 2023-11-24T08:03:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 116/162 2023-11-24T08:03:08.261 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 117/162 2023-11-24T08:03:09.391 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 117/162 2023-11-24T08:03:09.420 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 118/162 2023-11-24T08:03:09.568 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 118/162 2023-11-24T08:03:09.746 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 119/162 2023-11-24T08:03:09.797 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 120/162 2023-11-24T08:03:09.867 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 120/162 2023-11-24T08:03:09.867 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:09.868 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-24T08:03:09.868 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-24T08:03:09.868 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-24T08:03:09.868 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:09.931 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 121/162 2023-11-24T08:03:09.971 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 121/162 2023-11-24T08:03:10.069 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 122/162 2023-11-24T08:03:10.297 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 107/162 2023-11-24T08:03:10.297 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2023-11-24T08:03:10.298 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 108/162 2023-11-24T08:03:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 108/162 2023-11-24T08:03:10.577 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:10.577 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-24T08:03:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:10.929 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 123/162 2023-11-24T08:03:10.958 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 124/162 2023-11-24T08:03:11.029 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 124/162 2023-11-24T08:03:11.029 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:11.030 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-24T08:03:11.030 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:03:11.030 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:03:11.030 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 125/162 2023-11-24T08:03:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 125/162 2023-11-24T08:03:11.166 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:11.166 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-24T08:03:11.166 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:11.235 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 126/162 2023-11-24T08:03:11.309 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 126/162 2023-11-24T08:03:11.310 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:11.310 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-24T08:03:11.310 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-24T08:03:11.310 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-24T08:03:11.310 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 127/162 2023-11-24T08:03:11.759 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 127/162 2023-11-24T08:03:11.759 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:11.759 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-24T08:03:11.759 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:03:11.760 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:03:11.760 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:12.332 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 109/162 2023-11-24T08:03:12.418 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 109/162 2023-11-24T08:03:12.419 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:12.419 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-24T08:03:12.419 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-24T08:03:12.419 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-24T08:03:12.419 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:12.684 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 110/162 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 110/162 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-24T08:03:12.760 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:13.257 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 111/162 2023-11-24T08:03:13.328 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 111/162 2023-11-24T08:03:13.328 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:13.329 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-24T08:03:13.329 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-24T08:03:13.329 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-24T08:03:13.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 112/162 2023-11-24T08:03:14.125 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 113/162 2023-11-24T08:03:14.185 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 113/162 2023-11-24T08:03:14.232 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 114/162 2023-11-24T08:03:14.338 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 115/162 2023-11-24T08:03:14.364 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 116/162 2023-11-24T08:03:14.687 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 116/162 2023-11-24T08:03:14.724 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 117/162 2023-11-24T08:03:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 117/162 2023-11-24T08:03:15.882 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 118/162 2023-11-24T08:03:15.900 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 128/162 2023-11-24T08:03:15.971 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 129/162 2023-11-24T08:03:16.025 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 118/162 2023-11-24T08:03:16.028 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 130/162 2023-11-24T08:03:16.066 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/162 2023-11-24T08:03:16.198 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 119/162 2023-11-24T08:03:16.234 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 120/162 2023-11-24T08:03:16.308 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 120/162 2023-11-24T08:03:16.308 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:16.308 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-24T08:03:16.308 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-24T08:03:16.309 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-24T08:03:16.309 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:16.355 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 132/162 2023-11-24T08:03:16.394 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 121/162 2023-11-24T08:03:16.433 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 121/162 2023-11-24T08:03:16.474 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 133/162 2023-11-24T08:03:16.504 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 134/162 2023-11-24T08:03:16.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 122/162 2023-11-24T08:03:16.537 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_ 135/162 2023-11-24T08:03:16.569 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 136/162 2023-11-24T08:03:16.613 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 137/162 2023-11-24T08:03:16.722 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 138/162 2023-11-24T08:03:16.801 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 139/162 2023-11-24T08:03:16.848 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 140/162 2023-11-24T08:03:17.390 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-45.el9.x86_64 123/162 2023-11-24T08:03:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 124/162 2023-11-24T08:03:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 124/162 2023-11-24T08:03:17.500 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:17.500 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-24T08:03:17.501 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:03:17.501 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:03:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:17.555 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 125/162 2023-11-24T08:03:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 125/162 2023-11-24T08:03:17.642 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:17.642 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-24T08:03:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 126/162 2023-11-24T08:03:17.792 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 126/162 2023-11-24T08:03:17.792 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:17.792 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-24T08:03:17.792 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-24T08:03:17.792 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-24T08:03:17.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:17.817 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:17.863 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:17.925 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 127/162 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 127/162 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:03:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:19.304 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 142/162 2023-11-24T08:03:19.435 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 143/162 2023-11-24T08:03:19.495 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 144/162 2023-11-24T08:03:19.564 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 145/162 2023-11-24T08:03:19.708 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 146/162 2023-11-24T08:03:19.741 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 147/162 2023-11-24T08:03:19.774 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 148/162 2023-11-24T08:03:19.808 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 149/162 2023-11-24T08:03:19.849 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 150/162 2023-11-24T08:03:19.850 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/162 2023-11-24T08:03:19.983 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/162 2023-11-24T08:03:20.050 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 152/162 2023-11-24T08:03:20.097 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 153/162 2023-11-24T08:03:20.129 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 154/162 2023-11-24T08:03:20.130 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/162 2023-11-24T08:03:20.258 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/162 2023-11-24T08:03:20.289 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 156/162 2023-11-24T08:03:20.330 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:20.331 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:20.663 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:20.663 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/162 2023-11-24T08:03:20.814 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/162 2023-11-24T08:03:20.859 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 159/162 2023-11-24T08:03:20.892 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 160/162 2023-11-24T08:03:20.927 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 161/162 2023-11-24T08:03:21.054 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 162/162 2023-11-24T08:03:21.337 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 162/162 2023-11-24T08:03:21.338 INFO:teuthology.orchestra.run.smithi120.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-24T08:03:21.338 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:22.369 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 128/162 2023-11-24T08:03:22.417 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 129/162 2023-11-24T08:03:22.480 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 130/162 2023-11-24T08:03:22.519 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/162 2023-11-24T08:03:22.801 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 132/162 2023-11-24T08:03:22.919 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 133/162 2023-11-24T08:03:22.948 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 134/162 2023-11-24T08:03:22.982 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_ 135/162 2023-11-24T08:03:23.041 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 136/162 2023-11-24T08:03:23.085 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 137/162 2023-11-24T08:03:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 138/162 2023-11-24T08:03:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 139/162 2023-11-24T08:03:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 140/162 2023-11-24T08:03:24.316 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:24.415 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 141/162 2023-11-24T08:03:25.737 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 142/162 2023-11-24T08:03:25.859 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 143/162 2023-11-24T08:03:25.930 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 144/162 2023-11-24T08:03:26.033 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 145/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 1/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 3/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 4/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 6/162 2023-11-24T08:03:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 7/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 10/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 11/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 12/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 13/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_ 14/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 15/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 16/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_6 17/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 18/162 2023-11-24T08:03:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 19/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9. 21/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_6 22/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 23/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 24/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 25/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 26/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 27/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 28/162 2023-11-24T08:03:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/162 2023-11-24T08:03:26.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 30/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 31/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 34/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 36/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 37/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 38/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/162 2023-11-24T08:03:26.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 47/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/162 2023-11-24T08:03:26.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 56/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/162 2023-11-24T08:03:26.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/162 2023-11-24T08:03:26.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 81/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/162 2023-11-24T08:03:26.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 85/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 86/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 91/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 92/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 93/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 94/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 95/162 2023-11-24T08:03:26.164 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 96/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 97/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 100/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 101/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 104/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 107/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/162 2023-11-24T08:03:26.165 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 113/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 114/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/162 2023-11-24T08:03:26.166 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 133/162 2023-11-24T08:03:26.167 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 135/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 137/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 138/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 139/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 141/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 143/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 145/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/162 2023-11-24T08:03:26.168 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/162 2023-11-24T08:03:26.169 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 159/162 2023-11-24T08:03:26.170 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 160/162 2023-11-24T08:03:26.170 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 161/162 2023-11-24T08:03:26.210 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 146/162 2023-11-24T08:03:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 147/162 2023-11-24T08:03:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 148/162 2023-11-24T08:03:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 149/162 2023-11-24T08:03:26.386 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 150/162 2023-11-24T08:03:26.387 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/162 2023-11-24T08:03:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/162 2023-11-24T08:03:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 152/162 2023-11-24T08:03:26.633 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 153/162 2023-11-24T08:03:26.665 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 154/162 2023-11-24T08:03:26.666 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/162 2023-11-24T08:03:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/162 2023-11-24T08:03:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 156/162 2023-11-24T08:03:26.931 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:26.932 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:27.267 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 157/162 2023-11-24T08:03:27.268 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/162 2023-11-24T08:03:27.409 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/162 2023-11-24T08:03:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 159/162 2023-11-24T08:03:27.504 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 160/162 2023-11-24T08:03:27.557 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 161/162 2023-11-24T08:03:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 162/162 2023-11-24T08:03:27.940 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 162/162 2023-11-24T08:03:27.941 INFO:teuthology.orchestra.run.smithi107.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-24T08:03:27.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 162/162 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:28.224 INFO:teuthology.orchestra.run.smithi120.stdout: acl-2.3.1-4.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2023-11-24T08:03:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2023-11-24T08:03:28.226 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2023-11-24T08:03:28.226 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-24T08:03:28.226 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.226 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.226 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.227 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.228 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.229 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-24T08:03:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.231 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.232 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.232 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-24T08:03:28.232 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-24T08:03:28.232 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-24T08:03:28.232 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-24T08:03:28.233 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-24T08:03:28.234 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.11-1.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-24T08:03:28.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.236 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.236 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.236 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-24T08:03:28.236 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-24T08:03:28.236 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-24T08:03:28.237 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-24T08:03:28.238 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-24T08:03:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-24T08:03:28.240 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-24T08:03:28.241 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-24T08:03:28.242 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-24T08:03:28.243 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-24T08:03:28.243 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:28.243 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2023-11-24T08:03:28.804 DEBUG:teuthology.parallel:result is None 2023-11-24T08:03:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 1/162 2023-11-24T08:03:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 3/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 4/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 6/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 7/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 8/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 9/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 10/162 2023-11-24T08:03:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 11/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 12/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 13/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_ 14/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 15/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 16/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_6 17/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86 18/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 19/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/162 2023-11-24T08:03:32.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9. 21/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_6 22/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 23/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 24/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 25/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 26/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 27/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 28/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noa 30/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.n 31/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/162 2023-11-24T08:03:32.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 34/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 36/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 37/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 38/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/162 2023-11-24T08:03:32.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 47/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/162 2023-11-24T08:03:32.691 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 56/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/162 2023-11-24T08:03:32.692 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/162 2023-11-24T08:03:32.693 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 81/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/162 2023-11-24T08:03:32.694 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 85/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 86/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 91/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 92/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 93/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 94/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 95/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 96/162 2023-11-24T08:03:32.695 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 97/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 100/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 101/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 104/162 2023-11-24T08:03:32.696 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 107/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 113/162 2023-11-24T08:03:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 114/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/162 2023-11-24T08:03:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/162 2023-11-24T08:03:32.699 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 133/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 135/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 137/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 138/162 2023-11-24T08:03:32.700 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 139/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 141/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 143/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 145/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/162 2023-11-24T08:03:32.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/162 2023-11-24T08:03:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/162 2023-11-24T08:03:32.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 159/162 2023-11-24T08:03:32.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 160/162 2023-11-24T08:03:32.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 161/162 2023-11-24T08:03:34.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 162/162 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: acl-2.3.1-4.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-24T08:03:34.895 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.897 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el9.noarch 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-24T08:03:34.898 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-24T08:03:34.899 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-24T08:03:34.900 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.11-1.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-24T08:03:34.901 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-24T08:03:34.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-24T08:03:34.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-24T08:03:34.904 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-24T08:03:34.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el9.x86_64 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-24T08:03:34.906 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:34.907 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-11-24T08:03:35.539 DEBUG:teuthology.parallel:result is None 2023-11-24T08:03:35.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T08:03:35.682 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T08:03:35.750 INFO:teuthology.orchestra.run.smithi107.stdout:18.0.0-7484.ga13ea5ff.el9 2023-11-24T08:03:35.750 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el9 2023-11-24T08:03:35.751 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T08:03:35.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T08:03:35.900 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T08:03:35.981 INFO:teuthology.orchestra.run.smithi120.stdout:18.0.0-7484.ga13ea5ff.el9 2023-11-24T08:03:35.981 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el9 2023-11-24T08:03:35.981 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T08:03:35.981 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-24T08:03:35.981 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:35.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T08:03:36.063 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:36.064 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T08:03:36.172 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-24T08:03:36.172 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:36.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T08:03:36.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T08:03:36.363 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:36.363 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T08:03:36.457 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T08:03:36.537 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-24T08:03:36.537 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:36.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T08:03:36.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T08:03:36.740 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:36.740 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T08:03:36.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T08:03:36.951 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-24T08:03:36.951 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:36.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T08:03:37.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T08:03:37.124 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:37.124 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T08:03:37.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T08:03:37.332 INFO:teuthology.run_tasks:Running task ceph... 2023-11-24T08:03:37.421 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-24T08:03:37.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2023-11-24T08:03:37.431 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2023-11-24T08:03:37.519 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-24T08:03:37.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T08:03:37.523 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T08:03:37.601 INFO:tasks.ceph:Creating extra log directories... 2023-11-24T08:03:37.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T08:03:37.607 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T08:03:37.691 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-24T08:03:37.692 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-24T08:03:37.692 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-24T08:03:37.692 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:37.692 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T08:03:37.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T08:03:37.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T08:03:38.019 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:38.019 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T08:03:38.079 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T08:03:38.349 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T08:03:38.457 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-24T08:03:38.457 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-24T08:03:38.457 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582', 'branch': 'rishabh-mgr-vol-misc', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}', 'email': None, 'first_in_suite': False, 'job_id': '7465582', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-mgr-vol-misc'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-vol-tests', 'sha1': '73843baf45f8a92d8e3fcbea3f024c97e73036da'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9764, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'mgr-vol-tests', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '73843baf45f8a92d8e3fcbea3f024c97e73036da', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUWWHRQesnb0foUsW5bdD9/0e/M5daF7En+5E/TguR9kZ+kwLCXni0H9oxKVzMyoRSWtO4uh+NIzSB/tqpFRv8=', 'smithi120.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCd+WpaFe5MZUrwVrtno2fV8ntxSVfm4mTvahUZm6VEP8Jfebp4KLxUXw2rc9pNjqB960xGtkr+TIUiqqjxEqd0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_07:31:48', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-24T08:03:38.458 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T08:03:38.463 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T08:03:38.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:03:38.547 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:03:38.651 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:38.651 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T08:03:38.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T08:03:38.693 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 631 Links: 1 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-11-24 08:03:28.462445433 +0000 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-11-24 08:00:56.012204807 +0000 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-11-24 08:00:56.012204807 +0000 2023-11-24T08:03:38.771 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-11-24 08:00:56.012204807 +0000 2023-11-24T08:03:38.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T08:03:38.880 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:03:38.881 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:03:38.881 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000366504 s, 1.4 MB/s 2023-11-24T08:03:38.883 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T08:03:38.924 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 650 Links: 1 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-11-24 08:03:28.462445433 +0000 2023-11-24T08:03:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-11-24 08:00:57.061187473 +0000 2023-11-24T08:03:39.006 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-11-24 08:00:57.061187473 +0000 2023-11-24T08:03:39.006 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-11-24 08:00:57.061187473 +0000 2023-11-24T08:03:39.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T08:03:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:03:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:03:39.115 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000259721 s, 2.0 MB/s 2023-11-24T08:03:39.117 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T08:03:39.158 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-11-24T08:03:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 670 Links: 1 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-11-24 08:03:28.463445417 +0000 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-11-24 08:00:58.066170866 +0000 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-11-24 08:00:58.066170866 +0000 2023-11-24T08:03:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-11-24 08:00:58.066170866 +0000 2023-11-24T08:03:39.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T08:03:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:03:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:03:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000341356 s, 1.5 MB/s 2023-11-24T08:03:39.348 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T08:03:39.391 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-11-24T08:03:39.475 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T08:03:39.475 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:39.475 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 689 Links: 1 2023-11-24T08:03:39.475 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:39.475 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:39.476 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-11-24 08:03:28.463445417 +0000 2023-11-24T08:03:39.476 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-11-24 08:00:59.109153632 +0000 2023-11-24T08:03:39.476 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-11-24 08:00:59.109153632 +0000 2023-11-24T08:03:39.476 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-11-24 08:00:59.109153632 +0000 2023-11-24T08:03:39.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T08:03:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:03:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:03:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000330112 s, 1.6 MB/s 2023-11-24T08:03:39.605 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T08:03:39.650 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:39.650 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:39.650 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T08:03:39.710 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T08:03:39.711 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2023-11-24T08:03:39.764 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T08:03:39.764 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:39.764 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 634 Links: 1 2023-11-24T08:03:39.764 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:39.765 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:39.765 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2023-11-24 08:03:21.780477790 +0000 2023-11-24T08:03:39.765 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2023-11-24 08:00:55.865150998 +0000 2023-11-24T08:03:39.765 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2023-11-24 08:00:55.865150998 +0000 2023-11-24T08:03:39.765 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2023-11-24 08:00:55.865150998 +0000 2023-11-24T08:03:39.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T08:03:39.857 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2023-11-24T08:03:39.857 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2023-11-24T08:03:39.857 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000356122 s, 1.4 MB/s 2023-11-24T08:03:39.860 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T08:03:39.918 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2023-11-24T08:03:39.970 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 650 Links: 1 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2023-11-24 08:03:21.781477767 +0000 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2023-11-24 08:00:56.888127898 +0000 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2023-11-24 08:00:56.888127898 +0000 2023-11-24T08:03:39.971 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2023-11-24 08:00:56.888127898 +0000 2023-11-24T08:03:39.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T08:03:40.051 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2023-11-24T08:03:40.052 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2023-11-24T08:03:40.052 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000300972 s, 1.7 MB/s 2023-11-24T08:03:40.054 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T08:03:40.109 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2023-11-24T08:03:40.156 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 669 Links: 1 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2023-11-24 08:03:21.781477767 +0000 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2023-11-24 08:00:57.919104617 +0000 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2023-11-24 08:00:57.919104617 +0000 2023-11-24T08:03:40.157 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2023-11-24 08:00:57.919104617 +0000 2023-11-24T08:03:40.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T08:03:40.241 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2023-11-24T08:03:40.241 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2023-11-24T08:03:40.241 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000326935 s, 1.6 MB/s 2023-11-24T08:03:40.244 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T08:03:40.296 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2023-11-24T08:03:40.376 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T08:03:40.376 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:03:40.376 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-11-24T08:03:40.376 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:03:40.376 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T08:03:40.377 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2023-11-24 08:03:21.782477744 +0000 2023-11-24T08:03:40.377 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2023-11-24 08:00:58.924081926 +0000 2023-11-24T08:03:40.377 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2023-11-24 08:00:58.924081926 +0000 2023-11-24T08:03:40.377 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2023-11-24 08:00:58.924081926 +0000 2023-11-24T08:03:40.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T08:03:40.463 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2023-11-24T08:03:40.463 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2023-11-24T08:03:40.463 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000330466 s, 1.5 MB/s 2023-11-24T08:03:40.467 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T08:03:40.524 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:40.524 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T08:03:40.524 INFO:tasks.ceph:Generating config... 2023-11-24T08:03:40.525 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-24T08:03:40.526 INFO:tasks.ceph:[client] debug client = 20 2023-11-24T08:03:40.526 INFO:tasks.ceph:[client] debug ms = 1 2023-11-24T08:03:40.526 INFO:tasks.ceph:[client] pid file = 2023-11-24T08:03:40.526 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-24T08:03:40.526 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-24T08:03:40.526 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-11-24T08:03:40.526 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-24T08:03:40.526 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-24T08:03:40.526 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mgr] debug client = 20 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-24T08:03:40.527 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-24T08:03:40.528 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-24T08:03:40.528 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-24T08:03:40.529 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-24T08:03:40.529 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-24T08:03:40.529 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-24T08:03:40.529 INFO:tasks.ceph:Setting up mon.a... 2023-11-24T08:03:40.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-24T08:03:40.676 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2023-11-24T08:03:40.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-24T08:03:40.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T08:03:40.893 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107'), ('mon.b', '172.21.15.120'), ('mon.c', '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]')] 2023-11-24T08:03:40.894 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.107,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-24T08:03:40.895 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:40.895 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:03:40.943 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.107 --add b 172.21.15.120 --addv c '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = pacific 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:fsid a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2023-11-24T08:03:41.087370+0000 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:created 2023-11-24T08:03:41.087370+0000 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 16 (pacific) 2023-11-24T08:03:41.088 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2023-11-24T08:03:41.089 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2023-11-24T08:03:41.089 INFO:teuthology.orchestra.run.smithi107.stdout:1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2023-11-24T08:03:41.089 INFO:teuthology.orchestra.run.smithi107.stdout:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2023-11-24T08:03:41.089 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-24T08:03:41.091 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:03:41.144 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d... 2023-11-24T08:03:41.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T08:03:41.150 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2023-11-24T08:03:41.327 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2023-11-24T08:03:41.328 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2023-11-24T08:03:41.329 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.107,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on skipped message = False 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug trim objects = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2023-11-24T08:03:41.331 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2023-11-24T08:03:41.332 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2023-11-24T08:03:41.333 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2023-11-24T08:03:41.334 INFO:teuthology.orchestra.run.smithi120.stdout: debug finisher = 20 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:03:41.335 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2023-11-24T08:03:41.336 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2023-11-24T08:03:41.337 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2023-11-24T08:03:41.338 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.340 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:03:41.341 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.107,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on skipped message = False 2023-11-24T08:03:41.342 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2023-11-24T08:03:41.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.344 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore allocator = bitmap 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1/20 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1/20 2023-11-24T08:03:41.345 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 4/10 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-11-24T08:03:41.346 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug client = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug finisher = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2023-11-24T08:03:41.347 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:03:41.348 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: mon op complaint time = 120 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:03:41.349 INFO:teuthology.orchestra.run.smithi107.stdout: client mount timeout = 600 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout: debug client = 20 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = "" 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-11-24T08:03:41.350 INFO:teuthology.orchestra.run.smithi107.stdout:[mds] 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds = 20 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds balancer = 20 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug frag = True 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug scatterstat = True 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: mds op complaint time = 180 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: mds verify scatter = True 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-11-24T08:03:41.351 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-11-24T08:03:41.374 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-24T08:03:41.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-24T08:03:41.513 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-24T08:03:41.513 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:41.513 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-24T08:03:41.563 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:41.563 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-24T08:03:41.661 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:03:41.661 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:41.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T08:03:41.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T08:03:41.815 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:03:41.816 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:41.918 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:03:41.918 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:41.918 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T08:03:41.918 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T08:03:42.077 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:03:42.077 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:42.176 INFO:tasks.ceph:Setting up mon nodes... 2023-11-24T08:03:42.176 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-24T08:03:42.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T08:03:42.356 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T08:03:42.363 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T08:03:42.562 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T08:03:42.573 INFO:tasks.ceph:Setting up mds nodes... 2023-11-24T08:03:42.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T08:03:42.745 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T08:03:42.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-24T08:03:42.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T08:03:42.978 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T08:03:42.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-24T08:03:43.065 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T08:03:43.251 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T08:03:43.260 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-24T08:03:43.357 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T08:03:43.530 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T08:03:43.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-24T08:03:43.634 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-24T08:03:43.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-24T08:03:43.747 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-24T08:03:43.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-24T08:03:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-24T08:03:43.975 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-24T08:03:44.110 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-24T08:03:44.168 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-24T08:03:44.297 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-24T08:03:44.353 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-24T08:03:44.354 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T08:03:44.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-24T08:03:44.455 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:44.455 INFO:tasks.ceph:role: osd.0 2023-11-24T08:03:44.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:03:44.455 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:03:44.553 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:03:45.072 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-11-24T08:03:45.078 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-11-24T08:03:45.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-24T08:03:45.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-24T08:03:45.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-24T08:03:45.480 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:45.480 INFO:tasks.ceph:role: osd.1 2023-11-24T08:03:45.480 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:03:45.480 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T08:03:45.575 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:03:45.575 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:03:45.575 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:03:45.575 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:03:45.575 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:03:45.576 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:03:45.576 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:03:45.576 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:03:45.576 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:03:45.576 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:03:46.055 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-11-24T08:03:46.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-11-24T08:03:46.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-24T08:03:46.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-24T08:03:46.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-24T08:03:46.411 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:46.411 INFO:tasks.ceph:role: osd.2 2023-11-24T08:03:46.411 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:03:46.411 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T08:03:46.508 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:03:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:03:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:03:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:03:46.509 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:03:47.030 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-11-24T08:03:47.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-11-24T08:03:47.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-24T08:03:47.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-24T08:03:47.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-24T08:03:47.437 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T08:03:47.437 INFO:tasks.ceph:role: osd.3 2023-11-24T08:03:47.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:03:47.438 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T08:03:47.538 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:03:47.539 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:03:47.997 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-11-24T08:03:48.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-11-24T08:03:48.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-24T08:03:48.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-24T08:03:48.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.392+0000 7f0e7d08a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-24T08:03:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.393+0000 7f0e7d08a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-24T08:03:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.393+0000 7f0e7d08a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T08:03:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.393+0000 7f0e7d08a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T08:03:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.393+0000 7f0e7d08a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T08:03:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:48.401+0000 7f0e7d08a500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-24T08:03:51.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-24T08:03:51.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:51.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.825+0000 7f28e4012500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-24T08:03:51.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.826+0000 7f28e4012500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-24T08:03:51.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.826+0000 7f28e4012500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T08:03:51.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.826+0000 7f28e4012500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T08:03:51.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.826+0000 7f28e4012500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T08:03:51.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:51.828+0000 7f28e4012500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-24T08:03:55.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-24T08:03:55.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.244+0000 7f34c76b6500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-24T08:03:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.244+0000 7f34c76b6500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-24T08:03:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.245+0000 7f34c76b6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T08:03:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.245+0000 7f34c76b6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T08:03:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.245+0000 7f34c76b6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T08:03:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:55.247+0000 7f34c76b6500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-24T08:03:58.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-24T08:03:58.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:03:58.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.420+0000 7f3ac4b94500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-24T08:03:58.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.420+0000 7f3ac4b94500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-24T08:03:58.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.420+0000 7f3ac4b94500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T08:03:58.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.421+0000 7f3ac4b94500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T08:03:58.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.421+0000 7f3ac4b94500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T08:03:58.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:03:58.423+0000 7f3ac4b94500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-24T08:04:01.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-24T08:04:01.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-24T08:04:01.801 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T08:04:01.801 INFO:tasks.ceph:role: osd.4 2023-11-24T08:04:01.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:04:01.802 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T08:04:01.902 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:04:01.902 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:04:01.902 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:04:01.902 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:04:01.903 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:04:02.409 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2023-11-24T08:04:02.416 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2023-11-24T08:04:02.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-24T08:04:02.613 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-24T08:04:02.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-24T08:04:02.806 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T08:04:02.807 INFO:tasks.ceph:role: osd.5 2023-11-24T08:04:02.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:04:02.807 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T08:04:02.901 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:04:02.901 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:04:02.901 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:04:02.902 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:04:03.373 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2023-11-24T08:04:03.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2023-11-24T08:04:03.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-24T08:04:03.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-24T08:04:03.585 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-24T08:04:03.665 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T08:04:03.665 INFO:tasks.ceph:role: osd.6 2023-11-24T08:04:03.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:04:03.665 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T08:04:03.745 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:04:03.745 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:04:03.746 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:04:03.746 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:04:03.746 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:04:03.746 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:04:03.746 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:04:03.747 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:04:03.747 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:04:03.747 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:04:04.265 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2023-11-24T08:04:04.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2023-11-24T08:04:04.270 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-24T08:04:04.444 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-24T08:04:04.548 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-24T08:04:04.644 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T08:04:04.644 INFO:tasks.ceph:role: osd.7 2023-11-24T08:04:04.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:04:04.644 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T08:04:04.744 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:04:04.744 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:04:04.744 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:04:04.745 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:04:05.208 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2023-11-24T08:04:05.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2023-11-24T08:04:05.214 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-24T08:04:05.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-24T08:04:05.442 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:05.590 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.587+0000 7f0cad416500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-24T08:04:05.594 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.592+0000 7f0cad416500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-24T08:04:05.594 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.592+0000 7f0cad416500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T08:04:05.594 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.592+0000 7f0cad416500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T08:04:05.594 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.592+0000 7f0cad416500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T08:04:05.599 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:05.597+0000 7f0cad416500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-24T08:04:08.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:04:08.468 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:04:08.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-24T08:04:08.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:09.028 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.027+0000 7fabd380e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-24T08:04:09.029 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.027+0000 7fabd380e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-24T08:04:09.029 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.027+0000 7fabd380e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T08:04:09.029 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.027+0000 7fabd380e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T08:04:09.029 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.027+0000 7fabd380e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T08:04:09.030 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:09.029+0000 7fabd380e500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-24T08:04:12.185 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-24T08:04:12.284 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:12.421 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.419+0000 7f4dfc20e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-24T08:04:12.421 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.419+0000 7f4dfc20e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-24T08:04:12.421 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.419+0000 7f4dfc20e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T08:04:12.421 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.419+0000 7f4dfc20e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T08:04:12.421 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.419+0000 7f4dfc20e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T08:04:12.423 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:12.422+0000 7f4dfc20e500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-24T08:04:15.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-24T08:04:15.693 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:15.823 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.822+0000 7f66dd40f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-24T08:04:15.824 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.823+0000 7f66dd40f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-24T08:04:15.824 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.823+0000 7f66dd40f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T08:04:15.824 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.823+0000 7f66dd40f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T08:04:15.824 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.823+0000 7f66dd40f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T08:04:15.826 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:15.825+0000 7f66dd40f500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-24T08:04:18.746 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-24T08:04:18.840 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-24T08:04:18.840 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:18.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-24T08:04:18.945 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:18.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-24T08:04:19.043 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:19.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-24T08:04:19.142 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:19.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-24T08:04:19.240 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:19.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-24T08:04:19.335 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:19.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-24T08:04:19.431 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:19.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-24T08:04:19.528 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:19.528 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-24T08:04:19.629 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:19.629 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-24T08:04:19.718 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:19.719 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-24T08:04:19.815 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:19.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-24T08:04:19.912 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:19.912 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-24T08:04:20.011 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:20.011 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-24T08:04:20.110 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:20.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-24T08:04:20.251 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:20.252 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-24T08:04:20.309 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:20.309 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-24T08:04:20.361 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:20.361 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-24T08:04:20.418 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:20.419 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-24T08:04:20.471 INFO:tasks.ceph:Adding keys to all mons... 2023-11-24T08:04:20.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T08:04:20.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T08:04:20.565 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2023-11-24T08:04:20.565 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g== 2023-11-24T08:04:20.565 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ== 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ== 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg== 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2023-11-24T08:04:20.566 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ== 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg== 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA== 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2023-11-24T08:04:20.567 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A== 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g== 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ== 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw== 2023-11-24T08:04:20.568 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg== 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg== 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA== 2023-11-24T08:04:20.569 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout:[client.2] 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg== 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g== 2023-11-24T08:04:20.570 INFO:teuthology.orchestra.run.smithi120.stdout:[client.3] 2023-11-24T08:04:20.571 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw== 2023-11-24T08:04:20.571 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2023-11-24T08:04:20.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g== 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ== 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ== 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg== 2023-11-24T08:04:20.572 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2023-11-24T08:04:20.573 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ== 2023-11-24T08:04:20.574 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2023-11-24T08:04:20.574 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg== 2023-11-24T08:04:20.574 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2023-11-24T08:04:20.574 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA== 2023-11-24T08:04:20.574 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A== 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g== 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ== 2023-11-24T08:04:20.575 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw== 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg== 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg== 2023-11-24T08:04:20.576 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA== 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout:[client.2] 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg== 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2023-11-24T08:04:20.577 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g== 2023-11-24T08:04:20.578 INFO:teuthology.orchestra.run.smithi107.stdout:[client.3] 2023-11-24T08:04:20.578 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw== 2023-11-24T08:04:20.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:04:20.586 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:04:20.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:20.776 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:20.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:20.934 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:21.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.235 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.385 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.532 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:21.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:04:21.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:04:21.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:21.821 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:21.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:21.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T08:04:22.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.120 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.420 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.569 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:04:22.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:22.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:22.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:22.872 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:23.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:23.025 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:23.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:23.180 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:04:23.330 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-24T08:04:23.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-24T08:04:23.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T08:04:23.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-24T08:04:23.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-24T08:04:23.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T08:04:23.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-24T08:04:24.009 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-24T08:04:24.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T08:04:24.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-24T08:04:24.342 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:24.352 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T08:04:24.406 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-24T08:04:24.407 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-24T08:04:24.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-24T08:04:24.413 INFO:tasks.ceph.mon.a:Started 2023-11-24T08:04:24.413 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-24T08:04:24.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-24T08:04:24.422 INFO:tasks.ceph.mon.b:Started 2023-11-24T08:04:24.423 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-24T08:04:24.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-24T08:04:24.431 INFO:tasks.ceph.mon.c:Started 2023-11-24T08:04:24.432 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-24T08:04:24.432 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-24T08:04:24.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-24T08:04:24.437 INFO:tasks.ceph.mgr.y:Started 2023-11-24T08:04:24.437 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-24T08:04:24.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-24T08:04:24.442 INFO:tasks.ceph.mgr.x:Started 2023-11-24T08:04:24.443 INFO:tasks.ceph:Setting crush tunables to default 2023-11-24T08:04:24.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2023-11-24T08:04:24.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.616+0000 7fd6b3c23640 1 Processor -- start 2023-11-24T08:04:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.617+0000 7fd6b3c23640 1 -- start start 2023-11-24T08:04:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b3c23640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b3c23640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b3c23640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac06e930 con 0x7fd6ac0b28d0 2023-11-24T08:04:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b3c23640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac0cf0c0 con 0x7fd6ac15d9b0 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b3c23640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac06f5c0 con 0x7fd6ac06eef0 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b11be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b21c0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd6ac0b28d0 0x7fd6ac06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:47706/0 (socket says 172.21.15.107:47706) 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b21c0640 1 -- 172.21.15.107:0/2348274314 learned_addr learned my addr 172.21.15.107:0/2348274314 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.618+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.619+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.619+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.619+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:04:24.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.819+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:24.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T08:04:24.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:24.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:24.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:24.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-24T08:04:24.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:24.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:24.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:24.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:24.820+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-24T08:04:25.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.221+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:25.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:25.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T08:04:25.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:25.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:25.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-24T08:04:25.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:25.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:25.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:25.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:25.222+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-24T08:04:26.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-24T08:04:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b19bf640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:26.024+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-24T08:04:27.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 msgr2=0x7fd6ac15e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6ac06eef0 0x7fd6ac15e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 msgr2=0x7fd6ac15e030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.107:3300/0 conn(0x7fd6ac15d9b0 0x7fd6ac15e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 -- 172.21.15.107:0/2348274314 >> v1:172.21.15.120:6789/0 conn(0x7fd6ac0b28d0 legacy=0x7fd6ac06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 -- 172.21.15.107:0/2348274314 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac15dd90 con 0x7fd6900036a0 2023-11-24T08:04:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.618+0000 7fd6a37fe640 1 -- 172.21.15.107:0/2348274314 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac06f5c0 con 0x7fd690007350 2023-11-24T08:04:27.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.619+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:27.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.620+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:27.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.620+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:27.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.620+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:27.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.620+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:04:27.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.821+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:27.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.821+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:27.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.821+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:27.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.821+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:27.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:27.821+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-24T08:04:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:28.223+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:28.223+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:28.223+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:28.223+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:28.223+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-24T08:04:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.025+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.025+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T08:04:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.025+0000 7fd6b11be640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T08:04:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.025+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T08:04:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.025+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-24T08:04:29.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.879+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281922952 0 0) 0x7fd6ac15dd90 con 0x7fd6900036a0 2023-11-24T08:04:29.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.880+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd698003520 con 0x7fd6900036a0 2023-11-24T08:04:29.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.880+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2100847359 0 0) 0x7fd698003520 con 0x7fd6900036a0 2023-11-24T08:04:29.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 msgr2=0x7fd690009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T08:04:29.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 --2- 172.21.15.107:0/2348274314 >> v2:172.21.15.120:3300/0 conn(0x7fd690007350 0x7fd690009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69000af10 con 0x7fd6900036a0 2023-11-24T08:04:29.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd6a800d580 con 0x7fd6900036a0 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 >> v1:172.21.15.107:6789/0 conn(0x7fd6900036a0 legacy=0x7fd690005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b21c0640 1 -- 172.21.15.107:0/2348274314 reap_dead start 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd698003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd69800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac06f5c0 con 0x7fd6980039c0 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800e450 con 0x7fd6980047c0 2023-11-24T08:04:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.881+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800e610 con 0x7fd698005380 2023-11-24T08:04:29.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.882+0000 7fd6b21c0640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd69800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.882+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd698003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b11be640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b21c0640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 msgr2=0x7fd698004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:29.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b21c0640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b21c0640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 msgr2=0x7fd698003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:29.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b21c0640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd698003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b21c0640 1 -- 172.21.15.107:0/2348274314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698003520 con 0x7fd698005380 2023-11-24T08:04:29.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.883+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd698003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:29.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.886+0000 7fd6b21c0640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd69800db90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0c3a10 tx=0x7fd6980057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5ada1a0de00392 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:29.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.887+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a402d620 con 0x7fd698005380 2023-11-24T08:04:29.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.887+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a402d780 con 0x7fd698005380 2023-11-24T08:04:29.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.887+0000 7fd6b09bd640 1 -- 172.21.15.107:0/2348274314 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a402db30 con 0x7fd698005380 2023-11-24T08:04:29.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 -- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 msgr2=0x7fd69800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:29.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd69800db90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0c3a10 tx=0x7fd6980057f0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 -- 172.21.15.107:0/2348274314 shutdown_connections 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd69800db90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/2348274314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd698003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 -- 172.21.15.107:0/2348274314 >> 172.21.15.107:0/2348274314 conn(0x7fd6ac0999d0 msgr2=0x7fd6ac0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:29.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 -- 172.21.15.107:0/2348274314 shutdown_connections 2023-11-24T08:04:29.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.889+0000 7fd6b3c23640 1 -- 172.21.15.107:0/2348274314 wait complete. 2023-11-24T08:04:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.894+0000 7fd6b3c23640 1 Processor -- start 2023-11-24T08:04:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.894+0000 7fd6b3c23640 1 -- start start 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.896+0000 7fd6b3c23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.896+0000 7fd6b3c23640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd6ac156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b3c23640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd6ac156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b3c23640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac15b400 con 0x7fd6980039c0 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b3c23640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac15b650 con 0x7fd6980047c0 2023-11-24T08:04:29.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b3c23640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac15b930 con 0x7fd698005380 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46804/0 (socket says 172.21.15.107:46804) 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 -- 172.21.15.107:0/3409018739 learned_addr learned my addr 172.21.15.107:0/3409018739 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b21c0640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd6ac156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b11be640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd6ac156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 -- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 msgr2=0x7fd6ac156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:29.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd6ac156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 -- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 msgr2=0x7fd6ac156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd6ac156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.897+0000 7fd6b19bf640 1 -- 172.21.15.107:0/3409018739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac15bc10 con 0x7fd6980039c0 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6b19bf640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800d730 tx=0x7fd6a8014720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8014e90 con 0x7fd6980039c0 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a801b070 con 0x7fd6980039c0 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8016e80 con 0x7fd6980039c0 2023-11-24T08:04:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac15bef0 con 0x7fd6980039c0 2023-11-24T08:04:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.898+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac0c3600 con 0x7fd6980039c0 2023-11-24T08:04:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.899+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd6a801d030 con 0x7fd6980039c0 2023-11-24T08:04:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.899+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6a8003c50 con 0x7fd6980039c0 2023-11-24T08:04:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.900+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac04e8b0 con 0x7fd6980039c0 2023-11-24T08:04:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.903+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd6a803fce0 con 0x7fd6980039c0 2023-11-24T08:04:29.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.976+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd6ac06e680 con 0x7fd6980039c0 2023-11-24T08:04:29.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.979+0000 7fd6a27fc640 1 -- 172.21.15.107:0/3409018739 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd6a802e020 con 0x7fd6980039c0 2023-11-24T08:04:29.981 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2023-11-24T08:04:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 msgr2=0x7fd6ac0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800d730 tx=0x7fd6a8014720 comp rx=0 tx=0).stop 2023-11-24T08:04:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 shutdown_connections 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd698005380 0x7fd6ac156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6980047c0 0x7fd6ac156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 --2- 172.21.15.107:0/3409018739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6980039c0 0x7fd6ac0c30c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 >> 172.21.15.107:0/3409018739 conn(0x7fd6ac0999d0 msgr2=0x7fd6ac0aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 shutdown_connections 2023-11-24T08:04:29.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:29.980+0000 7fd6b3c23640 1 -- 172.21.15.107:0/3409018739 wait complete. 2023-11-24T08:04:29.999 INFO:tasks.ceph:check_enable_crimson: False 2023-11-24T08:04:29.999 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-24T08:04:30.000 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:30.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-24T08:04:30.093 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:30.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-24T08:04:30.231 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:30.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-24T08:04:30.318 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:04:30.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-24T08:04:30.408 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:30.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-24T08:04:30.508 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:30.508 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-24T08:04:30.605 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:30.605 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-24T08:04:30.682 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:04:30.682 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-24T08:04:30.809 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new a391934b-b663-4d35-bf9a-98364789c9c3 0 2023-11-24T08:04:31.003 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.000+0000 7f04a8f4e640 1 Processor -- start 2023-11-24T08:04:31.003 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.001+0000 7f04a8f4e640 1 -- start start 2023-11-24T08:04:31.005 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a8f4e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04a406e8a0 0x7f04a406fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.005 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a8f4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.005 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a8f4e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a40702f0 con 0x7f04a40cbd30 2023-11-24T08:04:31.005 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a8f4e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a415feb0 con 0x7f04a406e8a0 2023-11-24T08:04:31.005 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a8f4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a4070e40 con 0x7f04a406ec80 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04a406e8a0 0x7f04a406fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04a406e8a0 0x7f04a406fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56904/0 (socket says 172.21.15.120:56904) 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f04a40cbd30 0x7f04a415bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:51922/0 (socket says 172.21.15.120:51922) 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f049bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.002+0000 7f04a37fe640 1 -- 172.21.15.120:0/3676673793 learned_addr learned my addr 172.21.15.120:0/3676673793 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.006 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f04a37fe640 1 -- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 msgr2=0x7f04a40cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:31.007 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f04a37fe640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.007 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f04a37fe640 1 -- 172.21.15.120:0/3676673793 >> v1:172.21.15.120:6789/0 conn(0x7f04a40cbd30 legacy=0x7f04a415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.007 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f04a37fe640 1 -- 172.21.15.120:0/3676673793 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a415f2a0 con 0x7f04a406e8a0 2023-11-24T08:04:31.007 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f04a2ffd640 1 -- 172.21.15.120:0/3676673793 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628047833 0 0) 0x7f04a40702f0 con 0x7f04a40cbd30 2023-11-24T08:04:31.007 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.003+0000 7f049bfff640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.004+0000 7f04a37fe640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a406e8a0 0x7f04a406fdb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0494009870 tx=0x7f049402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9182e8440f4b36 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.004+0000 7f04a2ffd640 1 -- 172.21.15.120:0/3676673793 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494030840 con 0x7f04a406e8a0 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.004+0000 7f04a2ffd640 1 -- 172.21.15.120:0/3676673793 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04940309a0 con 0x7f04a406e8a0 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.004+0000 7f04a2ffd640 1 -- 172.21.15.120:0/3676673793 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494030d30 con 0x7f04a406e8a0 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a406e8a0 msgr2=0x7f04a406fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a406e8a0 0x7f04a406fdb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0494009870 tx=0x7f049402b980 comp rx=0 tx=0).stop 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3676673793 shutdown_connections 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3676673793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a406e8a0 0x7f04a406fdb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3676673793 >> 172.21.15.120:0/3676673793 conn(0x7f04a40d10c0 msgr2=0x7f04a40d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3676673793 shutdown_connections 2023-11-24T08:04:31.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.005+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3676673793 wait complete. 2023-11-24T08:04:31.011 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.007+0000 7f04a8f4e640 1 Processor -- start 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.007+0000 7f04a8f4e640 1 -- start start 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a406e8a0 0x7f04a40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a40c91e0 con 0x7f04a406e8a0 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a40b5370 con 0x7f04a40cbd30 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a8f4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a40b5650 con 0x7f04a406ec80 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f049bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56908/0 (socket says 172.21.15.120:56908) 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 -- 172.21.15.120:0/3214347601 learned_addr learned my addr 172.21.15.120:0/3214347601 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 -- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 msgr2=0x7f04a40acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 -- 172.21.15.120:0/3214347601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a406e8a0 msgr2=0x7f04a40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a406e8a0 0x7f04a40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f04a3fff640 1 -- 172.21.15.120:0/3214347601 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c009400 con 0x7f04a40cbd30 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.008+0000 7f049bfff640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a3fff640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f049c00c7e0 tx=0x7f049c00cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.013 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00a640 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c00a7c0 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00ab70 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04940094d0 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a40b5c10 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f049c0070d0 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.009+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f049c012580 con 0x7f04a40cbd30 2023-11-24T08:04:31.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.010+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a40c9350 con 0x7f04a40cbd30 2023-11-24T08:04:31.015 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.013+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f049c018070 con 0x7f04a40cbd30 2023-11-24T08:04:31.092 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.090+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a391934b-b663-4d35-bf9a-98364789c9c3", "id": 0} v 0) v1 -- 0x7f04a404e840 con 0x7f04a40cbd30 2023-11-24T08:04:31.099 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.097+0000 7f04a0ff9640 1 -- 172.21.15.120:0/3214347601 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a391934b-b663-4d35-bf9a-98364789c9c3", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f049c007510 con 0x7f04a40cbd30 2023-11-24T08:04:31.100 INFO:teuthology.orchestra.run.smithi120.stdout:0 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 msgr2=0x7f04a40b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f049c00c7e0 tx=0x7f049c00cca0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 shutdown_connections 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40cbd30 0x7f04a40b4640 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a406ec80 0x7f04a40acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.102 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 --2- 172.21.15.120:0/3214347601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a406e8a0 0x7f04a40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.103 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.099+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 >> 172.21.15.120:0/3214347601 conn(0x7f04a40d10c0 msgr2=0x7f04a40d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.103 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.100+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 shutdown_connections 2023-11-24T08:04:31.103 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.100+0000 7f04a8f4e640 1 -- 172.21.15.120:0/3214347601 wait complete. 2023-11-24T08:04:31.119 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 428f478c-5a65-4cf2-9e5d-9ed1ece89179 1 2023-11-24T08:04:31.166 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:31.163+0000 7f37e4441040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T08:04:31.187 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:31.185+0000 7f45db62e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T08:04:31.230 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:31.228+0000 7f37e4441040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T08:04:31.249 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:31.247+0000 7f45db62e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T08:04:31.315 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.313+0000 7feb02248640 1 Processor -- start 2023-11-24T08:04:31.315 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.313+0000 7feb02248640 1 -- start start 2023-11-24T08:04:31.317 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feafc0ccbd0 0x7feafc1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.317 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feafc0ccfb0 0x7feafc162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc162540 0x7feafc0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feafc1655f0 con 0x7feafc0ccfb0 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feafc0cc380 con 0x7feafc0ccbd0 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feb02248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feafc162b30 con 0x7feafc162540 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feafc0ccbd0 0x7feafc1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feafc0ccbd0 0x7feafc1660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56924/0 (socket says 172.21.15.120:56924) 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 -- 172.21.15.120:0/2918159238 learned_addr learned my addr 172.21.15.120:0/2918159238 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafaffd640 1 --2- 172.21.15.120:0/2918159238 >> v2:172.21.15.107:3300/0 conn(0x7feafc0ccfb0 0x7feafc162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 -- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc162540 msgr2=0x7feafc0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:31.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 --2- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc162540 0x7feafc0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 -- 172.21.15.120:0/2918159238 >> v2:172.21.15.107:3300/0 conn(0x7feafc0ccfb0 msgr2=0x7feafc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:31.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 --2- 172.21.15.120:0/2918159238 >> v2:172.21.15.107:3300/0 conn(0x7feafc0ccfb0 0x7feafc162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.315+0000 7feafb7fe640 1 -- 172.21.15.120:0/2918159238 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc162e10 con 0x7feafc0ccbd0 2023-11-24T08:04:31.320 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.316+0000 7feafb7fe640 1 --2- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1660b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feafc06ea50 tx=0x7feaec02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff252415c2378836 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.320 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.317+0000 7feafa7fc640 1 -- 172.21.15.120:0/2918159238 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec031870 con 0x7feafc0ccbd0 2023-11-24T08:04:31.320 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.317+0000 7feafa7fc640 1 -- 172.21.15.120:0/2918159238 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec0319d0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.317+0000 7feafa7fc640 1 -- 172.21.15.120:0/2918159238 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec031d60 con 0x7feafc0ccbd0 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.317+0000 7feb02248640 1 -- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 msgr2=0x7feafc1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.317+0000 7feb02248640 1 --2- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1660b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feafc06ea50 tx=0x7feaec02b910 comp rx=0 tx=0).stop 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 -- 172.21.15.120:0/2918159238 shutdown_connections 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 --2- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc162540 0x7feafc0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 --2- 172.21.15.120:0/2918159238 >> v2:172.21.15.107:3300/0 conn(0x7feafc0ccfb0 0x7feafc162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 --2- 172.21.15.120:0/2918159238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1660b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 -- 172.21.15.120:0/2918159238 >> 172.21.15.120:0/2918159238 conn(0x7feafc0b0c50 msgr2=0x7feafc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 -- 172.21.15.120:0/2918159238 shutdown_connections 2023-11-24T08:04:31.321 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.318+0000 7feb02248640 1 -- 172.21.15.120:0/2918159238 wait complete. 2023-11-24T08:04:31.322 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.319+0000 7feb02248640 1 Processor -- start 2023-11-24T08:04:31.322 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.319+0000 7feb02248640 1 -- start start 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 0x7feafc152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feafc162540 0x7feafc160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc1654b0 con 0x7feafc162540 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc1f82a0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.320+0000 7feb02248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feafc164590 con 0x7feafc0ccfb0 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafaffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 0x7feafc152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.324 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafaffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 0x7feafc152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33710/0 (socket says 172.21.15.120:33710) 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafaffd640 1 -- 172.21.15.120:0/3724148975 learned_addr learned my addr 172.21.15.120:0/3724148975 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafbfff640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feafc162540 0x7feafc160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 -- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 msgr2=0x7feafc152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 0x7feafc152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 -- 172.21.15.120:0/3724148975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feafc162540 msgr2=0x7feafc160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.325 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feafc162540 0x7feafc160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 -- 172.21.15.120:0/3724148975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec0094d0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.321+0000 7feafb7fe640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1f7d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feaec033000 tx=0x7feaec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec03ad20 con 0x7feafc0ccbd0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec03ae80 con 0x7feafc0ccbd0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc164830 con 0x7feafc0ccbd0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec044630 con 0x7feafc0ccbd0 2023-11-24T08:04:31.326 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc164df0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.327 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feaec044a90 con 0x7feafc0ccbd0 2023-11-24T08:04:31.327 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.322+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7feaec02d040 con 0x7feafc0ccbd0 2023-11-24T08:04:31.327 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.323+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc04e8b0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.327 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.325+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7feaec035020 con 0x7feafc0ccbd0 2023-11-24T08:04:31.405 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.403+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "428f478c-5a65-4cf2-9e5d-9ed1ece89179", "id": 1} v 0) v1 -- 0x7feafc09cca0 con 0x7feafc0ccbd0 2023-11-24T08:04:31.415 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.413+0000 7feae3fff640 1 -- 172.21.15.120:0/3724148975 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "428f478c-5a65-4cf2-9e5d-9ed1ece89179", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7feaec04d400 con 0x7feafc0ccbd0 2023-11-24T08:04:31.415 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:04:31.416 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 msgr2=0x7feafc1f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.416 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1f7d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feaec033000 tx=0x7feaec0046c0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 shutdown_connections 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feafc162540 0x7feafc160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feafc0ccfb0 0x7feafc152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 --2- 172.21.15.120:0/3724148975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feafc0ccbd0 0x7feafc1f7d60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 >> 172.21.15.120:0/3724148975 conn(0x7feafc0b0c50 msgr2=0x7feafc0c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 shutdown_connections 2023-11-24T08:04:31.417 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.414+0000 7feb02248640 1 -- 172.21.15.120:0/3724148975 wait complete. 2023-11-24T08:04:31.433 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new c404c5f0-390b-4849-8786-3712dec22fdd 2 2023-11-24T08:04:31.621 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.618+0000 7f3c56dfa640 1 Processor -- start 2023-11-24T08:04:31.621 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.619+0000 7f3c56dfa640 1 -- start start 2023-11-24T08:04:31.622 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c56dfa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c56dfa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c500b41b0 con 0x7f3c500b2050 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c56dfa640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c500b4490 con 0x7f3c50160dd0 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c56dfa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c500b0090 con 0x7f3c501609f0 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c55df8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c55df8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33724/0 (socket says 172.21.15.120:33724) 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c55df8640 1 -- 172.21.15.120:0/901248033 learned_addr learned my addr 172.21.15.120:0/901248033 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.623 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.620+0000 7f3c555f7640 1 --1- 172.21.15.120:0/901248033 >> v1:172.21.15.120:6789/0 conn(0x7f3c50160dd0 0x7f3c500b0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:51934/0 (socket says 172.21.15.120:51934) 2023-11-24T08:04:31.624 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.621+0000 7f3c54df6640 1 -- 172.21.15.120:0/901248033 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244245515 0 0) 0x7f3c500b4490 con 0x7f3c50160dd0 2023-11-24T08:04:31.624 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.621+0000 7f3c55df8640 1 -- 172.21.15.120:0/901248033 >> v1:172.21.15.120:6789/0 conn(0x7f3c50160dd0 legacy=0x7f3c500b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.621+0000 7f3c55df8640 1 -- 172.21.15.120:0/901248033 >> v1:172.21.15.107:6789/0 conn(0x7f3c500b2050 legacy=0x7f3c500b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.621+0000 7f3c55df8640 1 -- 172.21.15.120:0/901248033 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c500b0370 con 0x7f3c501609f0 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.622+0000 7f3c55df8640 1 --2- 172.21.15.120:0/901248033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c0098a0 tx=0x7f3c4c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92137cbbb2786214 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.622+0000 7f3c54df6640 1 -- 172.21.15.120:0/901248033 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c02f830 con 0x7f3c501609f0 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.622+0000 7f3c54df6640 1 -- 172.21.15.120:0/901248033 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c02f990 con 0x7f3c501609f0 2023-11-24T08:04:31.625 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.622+0000 7f3c54df6640 1 -- 172.21.15.120:0/901248033 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c02fd20 con 0x7f3c501609f0 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 -- 172.21.15.120:0/901248033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 msgr2=0x7f3c500d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/901248033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c0098a0 tx=0x7f3c4c02ba10 comp rx=0 tx=0).stop 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 -- 172.21.15.120:0/901248033 shutdown_connections 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/901248033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c501609f0 0x7f3c500d12f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 -- 172.21.15.120:0/901248033 >> 172.21.15.120:0/901248033 conn(0x7f3c5009c5b0 msgr2=0x7f3c500d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 -- 172.21.15.120:0/901248033 shutdown_connections 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.623+0000 7f3c56dfa640 1 -- 172.21.15.120:0/901248033 wait complete. 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.625+0000 7f3c56dfa640 1 Processor -- start 2023-11-24T08:04:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.625+0000 7f3c56dfa640 1 -- start start 2023-11-24T08:04:31.629 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c500b2050 0x7f3c500d1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.630 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.630 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 0x7f3c501565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.630 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c501599a0 con 0x7f3c500b2050 2023-11-24T08:04:31.630 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c501f33e0 con 0x7f3c501609f0 2023-11-24T08:04:31.630 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c56dfa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50158e20 con 0x7f3c50160dd0 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c565f9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 0x7f3c501565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56942/0 (socket says 172.21.15.120:56942) 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 -- 172.21.15.120:0/2804227532 learned_addr learned my addr 172.21.15.120:0/2804227532 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c55df8640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c500b2050 0x7f3c500d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 -- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 msgr2=0x7f3c501565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 0x7f3c501565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 -- 172.21.15.120:0/2804227532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c500b2050 msgr2=0x7f3c500d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c500b2050 0x7f3c500d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.631 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c555f7640 1 -- 172.21.15.120:0/2804227532 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c0094d0 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.626+0000 7f3c565f9640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 0x7f3c501565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c555f7640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c3c0 tx=0x7f3c4000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c400074e0 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40007640 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c50159100 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40004a20 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c501596c0 con 0x7f3c501609f0 2023-11-24T08:04:31.632 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3c4001d3f0 con 0x7f3c501609f0 2023-11-24T08:04:31.633 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.627+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f3c400137e0 con 0x7f3c501609f0 2023-11-24T08:04:31.633 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.628+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c50067990 con 0x7f3c501609f0 2023-11-24T08:04:31.633 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.631+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3c400133c0 con 0x7f3c501609f0 2023-11-24T08:04:31.710 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.708+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c404c5f0-390b-4849-8786-3712dec22fdd", "id": 2} v 0) v1 -- 0x7f3c500b0f50 con 0x7f3c501609f0 2023-11-24T08:04:31.716 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.714+0000 7f3c467fc640 1 -- 172.21.15.120:0/2804227532 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c404c5f0-390b-4849-8786-3712dec22fdd", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c40013a80 con 0x7f3c501609f0 2023-11-24T08:04:31.716 INFO:teuthology.orchestra.run.smithi120.stdout:2 2023-11-24T08:04:31.717 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 msgr2=0x7f3c50158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c3c0 tx=0x7f3c4000c880 comp rx=0 tx=0).stop 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 shutdown_connections 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c50160dd0 0x7f3c501565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c501609f0 0x7f3c50158440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 --2- 172.21.15.120:0/2804227532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c500b2050 0x7f3c500d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 >> 172.21.15.120:0/2804227532 conn(0x7f3c5009c5b0 msgr2=0x7f3c5006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 shutdown_connections 2023-11-24T08:04:31.718 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.715+0000 7f3c56dfa640 1 -- 172.21.15.120:0/2804227532 wait complete. 2023-11-24T08:04:31.733 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 2b920ea1-1b8c-476e-a7be-cd01c78a93b5 3 2023-11-24T08:04:31.837 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:31.836+0000 7f37e4441040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T08:04:31.888 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:31.885+0000 7f45db62e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T08:04:31.899 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:31.897+0000 7f37e4441040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T08:04:31.916 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.914+0000 7f5074e83640 1 Processor -- start 2023-11-24T08:04:31.917 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.915+0000 7f5074e83640 1 -- start start 2023-11-24T08:04:31.918 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f5074e83640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50700b40d0 0x7f50700cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.918 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f5074e83640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5070070870 con 0x7f50700d0a00 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f5074e83640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5070070b50 con 0x7f50700cb850 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f5074e83640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5070070680 con 0x7f50700b40d0 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f506ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50700b40d0 0x7f50700cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f506e59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f50700cb850 0x7f50700cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:51936/0 (socket says 172.21.15.120:51936) 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f506ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50700b40d0 0x7f50700cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56946/0 (socket says 172.21.15.120:56946) 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.916+0000 7f506e59c640 1 -- 172.21.15.120:0/1095795790 learned_addr learned my addr 172.21.15.120:0/1095795790 (peer_addr_for_me v1:172.21.15.120:0/0) 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.917+0000 7f506ed9d640 1 -- 172.21.15.120:0/1095795790 >> v1:172.21.15.120:6789/0 conn(0x7f50700cb850 legacy=0x7f50700cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.917+0000 7f506ed9d640 1 -- 172.21.15.120:0/1095795790 >> v1:172.21.15.107:6789/0 conn(0x7f50700d0a00 legacy=0x7f507006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:31.919 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.917+0000 7f506ed9d640 1 -- 172.21.15.120:0/1095795790 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070070e30 con 0x7f50700b40d0 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f506ed9d640 1 --2- 172.21.15.120:0/1095795790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700b40d0 0x7f50700cf670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5068009d90 tx=0x7f506802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f0e273b985193 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f506d59a640 1 -- 172.21.15.120:0/1095795790 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506802cbd0 con 0x7f50700b40d0 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f506d59a640 1 -- 172.21.15.120:0/1095795790 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506802cd30 con 0x7f50700b40d0 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f506d59a640 1 -- 172.21.15.120:0/1095795790 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506802cea0 con 0x7f50700b40d0 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f5074e83640 1 -- 172.21.15.120:0/1095795790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700b40d0 msgr2=0x7f50700cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.918+0000 7f5074e83640 1 --2- 172.21.15.120:0/1095795790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700b40d0 0x7f50700cf670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5068009d90 tx=0x7f506802be20 comp rx=0 tx=0).stop 2023-11-24T08:04:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.919+0000 7f5074e83640 1 -- 172.21.15.120:0/1095795790 shutdown_connections 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.919+0000 7f5074e83640 1 --2- 172.21.15.120:0/1095795790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700b40d0 0x7f50700cf670 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.919+0000 7f5074e83640 1 -- 172.21.15.120:0/1095795790 >> 172.21.15.120:0/1095795790 conn(0x7f507006e7d0 msgr2=0x7f507006ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.919+0000 7f5074e83640 1 -- 172.21.15.120:0/1095795790 shutdown_connections 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.919+0000 7f5074e83640 1 -- 172.21.15.120:0/1095795790 wait complete. 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.920+0000 7f5074e83640 1 Processor -- start 2023-11-24T08:04:31.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 -- start start 2023-11-24T08:04:31.925 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.925 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 0x7f507015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.925 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 0x7f50700c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:31.925 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700ca700 con 0x7f50700d0a00 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700c85e0 con 0x7f50700cb850 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.921+0000 7f5074e83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700c9a80 con 0x7f50700b40d0 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 0x7f507015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 0x7f507015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56960/0 (socket says 172.21.15.120:56960) 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33742/0 (socket says 172.21.15.120:33742) 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506dd9b640 1 -- 172.21.15.120:0/2171708106 learned_addr learned my addr 172.21.15.120:0/2171708106 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506ed9d640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 0x7f50700c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 -- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 msgr2=0x7f507015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.926 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 0x7f507015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 -- 172.21.15.120:0/2171708106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 msgr2=0x7f50700c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 0x7f50700c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 -- 172.21.15.120:0/2171708106 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068009a70 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506ed9d640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 0x7f50700c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.922+0000 7f506e59c640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f506000c440 tx=0x7f506000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060005ce0 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5060005e40 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50700c9d60 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600072d0 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50700ca320 con 0x7f50700b40d0 2023-11-24T08:04:31.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f506001c440 con 0x7f50700b40d0 2023-11-24T08:04:31.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.923+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f506001cb80 con 0x7f50700b40d0 2023-11-24T08:04:31.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.924+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50700ca840 con 0x7f50700b40d0 2023-11-24T08:04:31.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:31.927+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f50700ca840 con 0x7f50700b40d0 2023-11-24T08:04:31.951 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:31.949+0000 7f37e4441040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T08:04:31.953 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:31.951+0000 7f45db62e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T08:04:32.004 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.002+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2b920ea1-1b8c-476e-a7be-cd01c78a93b5", "id": 3} v 0) v1 -- 0x7f507004e8b0 con 0x7f50700b40d0 2023-11-24T08:04:32.006 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.004+0000 7f45db62e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T08:04:32.012 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.010+0000 7f505effd640 1 -- 172.21.15.120:0/2171708106 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b920ea1-1b8c-476e-a7be-cd01c78a93b5", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5060021060 con 0x7f50700b40d0 2023-11-24T08:04:32.012 INFO:teuthology.orchestra.run.smithi120.stdout:3 2023-11-24T08:04:32.013 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.010+0000 7f37e4441040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 msgr2=0x7f50700cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f506000c440 tx=0x7f506000c900 comp rx=0 tx=0).stop 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 shutdown_connections 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50700d0a00 0x7f50700c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50700cb850 0x7f507015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.014 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 --2- 172.21.15.120:0/2171708106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50700b40d0 0x7f50700cab20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.015 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 >> 172.21.15.120:0/2171708106 conn(0x7f507006e7d0 msgr2=0x7f5070150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.015 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.011+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 shutdown_connections 2023-11-24T08:04:32.015 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.012+0000 7f5074e83640 1 -- 172.21.15.120:0/2171708106 wait complete. 2023-11-24T08:04:32.030 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 0a932040-69ef-4757-9e06-f3e1772c9426 4 2023-11-24T08:04:32.064 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.062+0000 7f37e4441040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T08:04:32.073 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.071+0000 7f45db62e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T08:04:32.126 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.124+0000 7f45db62e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T08:04:32.176 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.175+0000 7f37e4441040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T08:04:32.217 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.215+0000 7fdddf59e640 1 Processor -- start 2023-11-24T08:04:32.217 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.215+0000 7fdddf59e640 1 -- start start 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fdddf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 0x7fdde00d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fdddf59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde00d1ae0 con 0x7fdde0160a00 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fdddf59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde00d1dc0 con 0x7fdde00d3500 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fdddf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00d20a0 con 0x7fdde00d1550 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 0x7fdde00d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fddded9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdde00d3500 0x7fdde00ce660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:51940/0 (socket says 172.21.15.120:51940) 2023-11-24T08:04:32.220 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.217+0000 7fddded9d640 1 -- 172.21.15.120:0/646016833 learned_addr learned my addr 172.21.15.120:0/646016833 (peer_addr_for_me v1:172.21.15.120:0/0) 2023-11-24T08:04:32.220 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.218+0000 7fddd7fff640 1 -- 172.21.15.120:0/646016833 >> v1:172.21.15.120:6789/0 conn(0x7fdde00d3500 legacy=0x7fdde00ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.220 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.218+0000 7fddd7fff640 1 -- 172.21.15.120:0/646016833 >> v1:172.21.15.107:6789/0 conn(0x7fdde0160a00 legacy=0x7fdde00d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.220 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.218+0000 7fddd7fff640 1 -- 172.21.15.120:0/646016833 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00ce140 con 0x7fdde00d1550 2023-11-24T08:04:32.220 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.218+0000 7fddddd9b640 1 -- 172.21.15.120:0/646016833 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252267578 0 0) 0x7fdde00d1dc0 con 0x7fdde00d3500 2023-11-24T08:04:32.221 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.219+0000 7fddd7fff640 1 --2- 172.21.15.120:0/646016833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 0x7fdde00d2fc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097d0 tx=0x7fddc802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=266e9e5637007194 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.221 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.219+0000 7fddddd9b640 1 -- 172.21.15.120:0/646016833 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8031770 con 0x7fdde00d1550 2023-11-24T08:04:32.221 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.219+0000 7fddddd9b640 1 -- 172.21.15.120:0/646016833 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc80318d0 con 0x7fdde00d1550 2023-11-24T08:04:32.221 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.219+0000 7fddddd9b640 1 -- 172.21.15.120:0/646016833 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8031c60 con 0x7fdde00d1550 2023-11-24T08:04:32.222 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 -- 172.21.15.120:0/646016833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 msgr2=0x7fdde00d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.222 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 --2- 172.21.15.120:0/646016833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 0x7fdde00d2fc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097d0 tx=0x7fddc802b810 comp rx=0 tx=0).stop 2023-11-24T08:04:32.222 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 -- 172.21.15.120:0/646016833 shutdown_connections 2023-11-24T08:04:32.222 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 --2- 172.21.15.120:0/646016833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde00d1550 0x7fdde00d2fc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.222 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 -- 172.21.15.120:0/646016833 >> 172.21.15.120:0/646016833 conn(0x7fdde008cc90 msgr2=0x7fdde009c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.223 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 -- 172.21.15.120:0/646016833 shutdown_connections 2023-11-24T08:04:32.223 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.220+0000 7fdddf59e640 1 -- 172.21.15.120:0/646016833 wait complete. 2023-11-24T08:04:32.223 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.221+0000 7fdddf59e640 1 Processor -- start 2023-11-24T08:04:32.223 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 -- start start 2023-11-24T08:04:32.225 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdde00d1550 0x7fdde00b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.225 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 0x7fdde00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.225 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00b0d70 con 0x7fdde00d1550 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00afea0 con 0x7fdde00d3500 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.222+0000 7fdddf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00b00f0 con 0x7fdde0160a00 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 0x7fdde00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 0x7fdde00ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56966/0 (socket says 172.21.15.120:56966) 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33758/0 (socket says 172.21.15.120:33758) 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddd7fff640 1 -- 172.21.15.120:0/520904911 learned_addr learned my addr 172.21.15.120:0/520904911 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddde59c640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdde00d1550 0x7fdde00b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 -- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 msgr2=0x7fdde00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 0x7fdde00ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.226 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 -- 172.21.15.120:0/520904911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdde00d1550 msgr2=0x7fdde00b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdde00d1550 0x7fdde00b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.223+0000 7fddded9d640 1 -- 172.21.15.120:0/520904911 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8009400 con 0x7fdde0160a00 2023-11-24T08:04:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddded9d640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c750 tx=0x7fddd800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800ce60 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd800a640 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800a9d0 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde00b03d0 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde00b0990 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fddd80070d0 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.224+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fddd8007890 con 0x7fdde0160a00 2023-11-24T08:04:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.225+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00b0eb0 con 0x7fdde0160a00 2023-11-24T08:04:32.229 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.227+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fdde00b0eb0 con 0x7fdde0160a00 2023-11-24T08:04:32.246 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.244+0000 7f45db62e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T08:04:32.257 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.256+0000 7f37e4441040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T08:04:32.307 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.304+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0a932040-69ef-4757-9e06-f3e1772c9426", "id": 4} v 0) v1 -- 0x7fdde004e840 con 0x7fdde0160a00 2023-11-24T08:04:32.311 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.310+0000 7f37e4441040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T08:04:32.315 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.313+0000 7fddd6ffd640 1 -- 172.21.15.120:0/520904911 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a932040-69ef-4757-9e06-f3e1772c9426", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fddd8018070 con 0x7fdde0160a00 2023-11-24T08:04:32.316 INFO:teuthology.orchestra.run.smithi120.stdout:4 2023-11-24T08:04:32.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 msgr2=0x7fdde00b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c750 tx=0x7fddd800cc10 comp rx=0 tx=0).stop 2023-11-24T08:04:32.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 shutdown_connections 2023-11-24T08:04:32.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdde0160a00 0x7fdde00b88b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.318 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdde00d3500 0x7fdde00ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 --2- 172.21.15.120:0/520904911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdde00d1550 0x7fdde00b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 >> 172.21.15.120:0/520904911 conn(0x7fdde008cc90 msgr2=0x7fdde015f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 shutdown_connections 2023-11-24T08:04:32.319 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.315+0000 7fdddf59e640 1 -- 172.21.15.120:0/520904911 wait complete. 2023-11-24T08:04:32.331 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.329+0000 7f45db62e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T08:04:32.332 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new dba0f2ab-c900-4ee4-a501-ae62eb3b57ed 5 2023-11-24T08:04:32.372 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.370+0000 7f37e4441040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T08:04:32.386 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.384+0000 7f45db62e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T08:04:32.427 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.425+0000 7f37e4441040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T08:04:32.450 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.448+0000 7f45db62e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T08:04:32.506 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.504+0000 7f45db62e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T08:04:32.558 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.556+0000 7f46ca0f8640 1 Processor -- start 2023-11-24T08:04:32.559 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.556+0000 7f46ca0f8640 1 -- start start 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46ca0f8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46ca0f8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46c40b2050 0x7f46c40d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46ca0f8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c40b41b0 con 0x7f46c4160dd0 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46ca0f8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40b4490 con 0x7f46c40b2050 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46ca0f8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40b0090 con 0x7f46c41609f0 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46c90f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46c90f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33764/0 (socket says 172.21.15.120:33764) 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46c90f6640 1 -- 172.21.15.120:0/2451625260 learned_addr learned my addr 172.21.15.120:0/2451625260 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.558+0000 7f46c98f7640 1 --2- 172.21.15.120:0/2451625260 >> v2:172.21.15.107:3300/0 conn(0x7f46c40b2050 0x7f46c40d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.559+0000 7f46c90f6640 1 -- 172.21.15.120:0/2451625260 >> v2:172.21.15.107:3300/0 conn(0x7f46c40b2050 msgr2=0x7f46c40d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:32.561 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.559+0000 7f46c90f6640 1 --2- 172.21.15.120:0/2451625260 >> v2:172.21.15.107:3300/0 conn(0x7f46c40b2050 0x7f46c40d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.562 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.559+0000 7f46c90f6640 1 -- 172.21.15.120:0/2451625260 >> v1:172.21.15.107:6789/0 conn(0x7f46c4160dd0 legacy=0x7f46c40b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.562 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.559+0000 7f46c90f6640 1 -- 172.21.15.120:0/2451625260 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40b0370 con 0x7f46c41609f0 2023-11-24T08:04:32.563 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.560+0000 7f46c90f6640 1 --2- 172.21.15.120:0/2451625260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009870 tx=0x7f46c002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c494014a8f74d92a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.563 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.560+0000 7f46bbfff640 1 -- 172.21.15.120:0/2451625260 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0030840 con 0x7f46c41609f0 2023-11-24T08:04:32.563 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.560+0000 7f46bbfff640 1 -- 172.21.15.120:0/2451625260 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c00309a0 con 0x7f46c41609f0 2023-11-24T08:04:32.563 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.560+0000 7f46bbfff640 1 -- 172.21.15.120:0/2451625260 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0030d30 con 0x7f46c41609f0 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 -- 172.21.15.120:0/2451625260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 msgr2=0x7f46c40d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/2451625260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009870 tx=0x7f46c002b980 comp rx=0 tx=0).stop 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 -- 172.21.15.120:0/2451625260 shutdown_connections 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/2451625260 >> v2:172.21.15.107:3300/0 conn(0x7f46c40b2050 0x7f46c40d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/2451625260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c41609f0 0x7f46c40d12f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 -- 172.21.15.120:0/2451625260 >> 172.21.15.120:0/2451625260 conn(0x7f46c409c5b0 msgr2=0x7f46c40d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 -- 172.21.15.120:0/2451625260 shutdown_connections 2023-11-24T08:04:32.564 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.561+0000 7f46ca0f8640 1 -- 172.21.15.120:0/2451625260 wait complete. 2023-11-24T08:04:32.565 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.563+0000 7f46ca0f8640 1 Processor -- start 2023-11-24T08:04:32.565 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.564+0000 7f46ca0f8640 1 -- start start 2023-11-24T08:04:32.568 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.568 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46c41609f0 0x7f46c414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.568 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46c4160dd0 0x7f46c40c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.568 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c409f710 con 0x7f46c41609f0 2023-11-24T08:04:32.568 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c414fc20 con 0x7f46c4160dd0 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46ca0f8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c409eb80 con 0x7f46c40b2050 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46c90f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46c90f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33770/0 (socket says 172.21.15.120:33770) 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46c90f6640 1 -- 172.21.15.120:0/942615467 learned_addr learned my addr 172.21.15.120:0/942615467 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.565+0000 7f46c98f7640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46c4160dd0 0x7f46c40c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c88f5640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46c41609f0 0x7f46c414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 -- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46c4160dd0 msgr2=0x7f46c40c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46c4160dd0 0x7f46c40c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 -- 172.21.15.120:0/942615467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46c41609f0 msgr2=0x7f46c414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46c41609f0 0x7f46c414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 -- 172.21.15.120:0/942615467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00094d0 con 0x7f46c40b2050 2023-11-24T08:04:32.569 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46c90f6640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46c0000c00 tx=0x7f46c0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.570 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.566+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0039c90 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c0039df0 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0043640 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c409ee20 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c409f0c0 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f46c0043aa0 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.567+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f46c003e070 con 0x7f46c40b2050 2023-11-24T08:04:32.571 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.568+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c404e840 con 0x7f46c40b2050 2023-11-24T08:04:32.572 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.570+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f46c0034070 con 0x7f46c40b2050 2023-11-24T08:04:32.650 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.648+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "dba0f2ab-c900-4ee4-a501-ae62eb3b57ed", "id": 5} v 0) v1 -- 0x7f46c40cd510 con 0x7f46c40b2050 2023-11-24T08:04:32.658 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.656+0000 7f46b9ffb640 1 -- 172.21.15.120:0/942615467 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dba0f2ab-c900-4ee4-a501-ae62eb3b57ed", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f46c004d460 con 0x7f46c40b2050 2023-11-24T08:04:32.659 INFO:teuthology.orchestra.run.smithi120.stdout:5 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 msgr2=0x7f46c409d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46c0000c00 tx=0x7f46c0004730 comp rx=0 tx=0).stop 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 shutdown_connections 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46c4160dd0 0x7f46c40c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46c41609f0 0x7f46c414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.660 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 --2- 172.21.15.120:0/942615467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46c40b2050 0x7f46c409d020 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.661 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 >> 172.21.15.120:0/942615467 conn(0x7f46c409c5b0 msgr2=0x7f46c406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.661 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 shutdown_connections 2023-11-24T08:04:32.661 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.658+0000 7f46ca0f8640 1 -- 172.21.15.120:0/942615467 wait complete. 2023-11-24T08:04:32.675 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 307fd4c8-bea5-4c0f-a3f0-786e9e8728e8 6 2023-11-24T08:04:32.877 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:32.874+0000 7f37e4441040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T08:04:32.918 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.916+0000 7efced09b640 1 Processor -- start 2023-11-24T08:04:32.918 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.916+0000 7efced09b640 1 -- start start 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efced09b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efce80b1ed0 0x7efce81681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efced09b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce815db00 con 0x7efce815d720 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efced09b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce806d1a0 con 0x7efce80d3a40 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efced09b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce80d0ce0 con 0x7efce80b1ed0 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efcdffff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7efce80d3a40 0x7efce806ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:51950/0 (socket says 172.21.15.120:51950) 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.917+0000 7efcdffff640 1 -- 172.21.15.120:0/1296921057 learned_addr learned my addr 172.21.15.120:0/1296921057 (peer_addr_for_me v1:172.21.15.120:0/0) 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce759e640 1 --2- 172.21.15.120:0/1296921057 >> v2:172.21.15.107:3300/0 conn(0x7efce80b1ed0 0x7efce81681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce659c640 1 -- 172.21.15.120:0/1296921057 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570156811 0 0) 0x7efce806d1a0 con 0x7efce80d3a40 2023-11-24T08:04:32.920 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce659c640 1 -- 172.21.15.120:0/1296921057 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003520 con 0x7efce80d3a40 2023-11-24T08:04:32.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce759e640 1 -- 172.21.15.120:0/1296921057 >> v1:172.21.15.120:6789/0 conn(0x7efce80d3a40 legacy=0x7efce806ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce759e640 1 -- 172.21.15.120:0/1296921057 >> v1:172.21.15.107:6789/0 conn(0x7efce815d720 legacy=0x7efce80d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:32.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.918+0000 7efce759e640 1 -- 172.21.15.120:0/1296921057 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce806d400 con 0x7efce80b1ed0 2023-11-24T08:04:32.921 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.920+0000 7efce759e640 1 --2- 172.21.15.120:0/1296921057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce80b1ed0 0x7efce81681d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efce0009d90 tx=0x7efce002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118f96d20daf625e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.920+0000 7efce659c640 1 -- 172.21.15.120:0/1296921057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce002cc60 con 0x7efce80b1ed0 2023-11-24T08:04:32.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.920+0000 7efce659c640 1 -- 172.21.15.120:0/1296921057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce002cdc0 con 0x7efce80b1ed0 2023-11-24T08:04:32.922 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.920+0000 7efce659c640 1 -- 172.21.15.120:0/1296921057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0034660 con 0x7efce80b1ed0 2023-11-24T08:04:32.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 -- 172.21.15.120:0/1296921057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce80b1ed0 msgr2=0x7efce81681d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 --2- 172.21.15.120:0/1296921057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce80b1ed0 0x7efce81681d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efce0009d90 tx=0x7efce002beb0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 -- 172.21.15.120:0/1296921057 shutdown_connections 2023-11-24T08:04:32.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 --2- 172.21.15.120:0/1296921057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce80b1ed0 0x7efce81681d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.923 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 -- 172.21.15.120:0/1296921057 >> 172.21.15.120:0/1296921057 conn(0x7efce808af60 msgr2=0x7efce809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:32.924 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 -- 172.21.15.120:0/1296921057 shutdown_connections 2023-11-24T08:04:32.924 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.921+0000 7efced09b640 1 -- 172.21.15.120:0/1296921057 wait complete. 2023-11-24T08:04:32.924 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.922+0000 7efced09b640 1 Processor -- start 2023-11-24T08:04:32.924 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.923+0000 7efced09b640 1 -- start start 2023-11-24T08:04:32.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.923+0000 7efced09b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.923+0000 7efced09b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 0x7efce80ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efced09b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 0x7efce815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:32.927 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efced09b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce80caa30 con 0x7efce815d720 2023-11-24T08:04:32.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efced09b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce815ee80 con 0x7efce80d3a40 2023-11-24T08:04:32.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efced09b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce815f160 con 0x7efce80b1ed0 2023-11-24T08:04:32.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efcdffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 0x7efce80ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.928 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33778/0 (socket says 172.21.15.120:33778) 2023-11-24T08:04:32.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 -- 172.21.15.120:0/1197294167 learned_addr learned my addr 172.21.15.120:0/1197294167 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:32.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efcdffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 0x7efce80ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:56984/0 (socket says 172.21.15.120:56984) 2023-11-24T08:04:32.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce759e640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 0x7efce815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:32.929 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 -- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 msgr2=0x7efce80ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.930 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 0x7efce80ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.930 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 -- 172.21.15.120:0/1197294167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 msgr2=0x7efce815ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:32.930 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 0x7efce815ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:32.930 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce6d9d640 1 -- 172.21.15.120:0/1197294167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009a70 con 0x7efce80b1ed0 2023-11-24T08:04:32.930 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.924+0000 7efce759e640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 0x7efce815ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efce6d9d640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c450 tx=0x7efcd800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80052d0 con 0x7efce80b1ed0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd8005430 con 0x7efce80b1ed0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce815f440 con 0x7efce80b1ed0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8012870 con 0x7efce80b1ed0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce815fa00 con 0x7efce80b1ed0 2023-11-24T08:04:32.931 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efcd8012cd0 con 0x7efce80b1ed0 2023-11-24T08:04:32.932 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.925+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7efcd8011380 con 0x7efce80b1ed0 2023-11-24T08:04:32.932 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.926+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce804e8b0 con 0x7efce80b1ed0 2023-11-24T08:04:32.932 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:32.928+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7efcd80035f0 con 0x7efce80b1ed0 2023-11-24T08:04:32.983 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:32.981+0000 7f45db62e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T08:04:33.009 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.007+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "307fd4c8-bea5-4c0f-a3f0-786e9e8728e8", "id": 6} v 0) v1 -- 0x7efce815fbc0 con 0x7efce80b1ed0 2023-11-24T08:04:33.019 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.017+0000 7efcdf7fe640 1 -- 172.21.15.120:0/1197294167 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "307fd4c8-bea5-4c0f-a3f0-786e9e8728e8", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7efcd8003790 con 0x7efce80b1ed0 2023-11-24T08:04:33.020 INFO:teuthology.orchestra.run.smithi120.stdout:6 2023-11-24T08:04:33.020 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.018+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 msgr2=0x7efce8159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.020 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.018+0000 7efced09b640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c450 tx=0x7efcd800c910 comp rx=0 tx=0).stop 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.018+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 shutdown_connections 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce815d720 0x7efce815ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce80d3a40 0x7efce80ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 --2- 172.21.15.120:0/1197294167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce80b1ed0 0x7efce8159550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 >> 172.21.15.120:0/1197294167 conn(0x7efce808af60 msgr2=0x7efce80cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 shutdown_connections 2023-11-24T08:04:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.019+0000 7efced09b640 1 -- 172.21.15.120:0/1197294167 wait complete. 2023-11-24T08:04:33.038 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new e0953d37-b98c-44e9-9f1e-cad4370876fe 7 2023-11-24T08:04:33.039 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.037+0000 7f37e4441040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T08:04:33.150 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.148+0000 7f45db62e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T08:04:33.182 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.179+0000 7f37e4441040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T08:04:33.230 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.228+0000 7f6bebfff640 1 Processor -- start 2023-11-24T08:04:33.231 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.229+0000 7f6bebfff640 1 -- start start 2023-11-24T08:04:33.232 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.232 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6bebfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6bec160dd0 0x7f6bec0d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.232 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6bebfff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec0b41b0 con 0x7f6bec0b2050 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6bebfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bec0b4490 con 0x7f6bec160dd0 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6bebfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0b0090 con 0x7f6bec1609f0 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6beaffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6be3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6bec160dd0 0x7f6bec0d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6beaffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33788/0 (socket says 172.21.15.120:33788) 2023-11-24T08:04:33.233 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.230+0000 7f6beaffd640 1 -- 172.21.15.120:0/2128869012 learned_addr learned my addr 172.21.15.120:0/2128869012 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:33.234 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.231+0000 7f6beaffd640 1 -- 172.21.15.120:0/2128869012 >> v2:172.21.15.120:3300/0 conn(0x7f6bec160dd0 msgr2=0x7f6bec0d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:33.234 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.231+0000 7f6beaffd640 1 --2- 172.21.15.120:0/2128869012 >> v2:172.21.15.120:3300/0 conn(0x7f6bec160dd0 0x7f6bec0d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.234 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.231+0000 7f6beaffd640 1 -- 172.21.15.120:0/2128869012 >> v1:172.21.15.107:6789/0 conn(0x7f6bec0b2050 legacy=0x7f6bec0b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:33.234 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.231+0000 7f6beaffd640 1 -- 172.21.15.120:0/2128869012 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0b0370 con 0x7f6bec1609f0 2023-11-24T08:04:33.234 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.232+0000 7f6beaffd640 1 --2- 172.21.15.120:0/2128869012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009870 tx=0x7f6bdc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24dfbc6f264609c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:33.235 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.232+0000 7f6bea7fc640 1 -- 172.21.15.120:0/2128869012 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc031640 con 0x7f6bec1609f0 2023-11-24T08:04:33.235 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.232+0000 7f6bea7fc640 1 -- 172.21.15.120:0/2128869012 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc0317a0 con 0x7f6bec1609f0 2023-11-24T08:04:33.235 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.232+0000 7f6bea7fc640 1 -- 172.21.15.120:0/2128869012 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc031b30 con 0x7f6bec1609f0 2023-11-24T08:04:33.235 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.233+0000 7f37e4441040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T08:04:33.236 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 -- 172.21.15.120:0/2128869012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 msgr2=0x7f6bec0d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.236 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 --2- 172.21.15.120:0/2128869012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009870 tx=0x7f6bdc02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.236 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 -- 172.21.15.120:0/2128869012 shutdown_connections 2023-11-24T08:04:33.237 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 --2- 172.21.15.120:0/2128869012 >> v2:172.21.15.120:3300/0 conn(0x7f6bec160dd0 0x7f6bec0d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.237 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 --2- 172.21.15.120:0/2128869012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0d12f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.237 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 -- 172.21.15.120:0/2128869012 >> 172.21.15.120:0/2128869012 conn(0x7f6bec09c5b0 msgr2=0x7f6bec0d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:33.237 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 -- 172.21.15.120:0/2128869012 shutdown_connections 2023-11-24T08:04:33.237 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.233+0000 7f6bebfff640 1 -- 172.21.15.120:0/2128869012 wait complete. 2023-11-24T08:04:33.238 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.235+0000 7f6bebfff640 1 Processor -- start 2023-11-24T08:04:33.238 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.236+0000 7f6bebfff640 1 -- start start 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bec0b2050 0x7f6bec14f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bec160dd0 0x7f6bec0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0c6d40 con 0x7f6bec160dd0 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0ca0e0 con 0x7f6bec0b2050 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.237+0000 7f6bebfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0a07d0 con 0x7f6bec1609f0 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.241 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:33802/0 (socket says 172.21.15.120:33802) 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 -- 172.21.15.120:0/883954769 learned_addr learned my addr 172.21.15.120:0/883954769 (peer_addr_for_me v2:172.21.15.120:0/0) 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6beaffd640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bec0b2050 0x7f6bec14f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6beb7fe640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bec160dd0 0x7f6bec0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 -- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bec0b2050 msgr2=0x7f6bec14f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bec0b2050 0x7f6bec14f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 -- 172.21.15.120:0/883954769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bec160dd0 msgr2=0x7f6bec0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bec160dd0 0x7f6bec0cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 -- 172.21.15.120:0/883954769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0094d0 con 0x7f6bec1609f0 2023-11-24T08:04:33.242 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be3fff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400e840 tx=0x7f6bd400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.238+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40094f0 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd400c940 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd400ccd0 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec0a0ab0 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec0c69d0 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6bd4005c90 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.239+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f6bd400ce30 con 0x7f6bec1609f0 2023-11-24T08:04:33.243 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.240+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec0c6eb0 con 0x7f6bec1609f0 2023-11-24T08:04:33.244 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.242+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6bd4021060 con 0x7f6bec1609f0 2023-11-24T08:04:33.301 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.298+0000 7f45db62e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T08:04:33.322 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.320+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e0953d37-b98c-44e9-9f1e-cad4370876fe", "id": 7} v 0) v1 -- 0x7f6bec04e840 con 0x7f6bec1609f0 2023-11-24T08:04:33.332 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.330+0000 7f6be37fe640 1 -- 172.21.15.120:0/883954769 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e0953d37-b98c-44e9-9f1e-cad4370876fe", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6bd4029020 con 0x7f6bec1609f0 2023-11-24T08:04:33.332 INFO:teuthology.orchestra.run.smithi120.stdout:7 2023-11-24T08:04:33.333 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 msgr2=0x7f6bec0c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.333 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400e840 tx=0x7f6bd400ed00 comp rx=0 tx=0).stop 2023-11-24T08:04:33.333 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 shutdown_connections 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bec160dd0 0x7f6bec0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bec1609f0 0x7f6bec0c9ba0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 --2- 172.21.15.120:0/883954769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bec0b2050 0x7f6bec14f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 >> 172.21.15.120:0/883954769 conn(0x7f6bec09c5b0 msgr2=0x7f6bec06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 shutdown_connections 2023-11-24T08:04:33.334 INFO:teuthology.orchestra.run.smithi120.stderr:2023-11-24T08:04:33.331+0000 7f6bebfff640 1 -- 172.21.15.120:0/883954769 wait complete. 2023-11-24T08:04:33.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.343+0000 7f37e4441040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:04:33.350 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-24T08:04:33.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-24T08:04:33.357 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.355+0000 7f45db62e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T08:04:33.358 INFO:tasks.ceph.osd.0:Started 2023-11-24T08:04:33.359 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-24T08:04:33.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-24T08:04:33.367 INFO:tasks.ceph.osd.1:Started 2023-11-24T08:04:33.367 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-24T08:04:33.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-24T08:04:33.372 INFO:tasks.ceph.osd.2:Started 2023-11-24T08:04:33.372 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-24T08:04:33.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-24T08:04:33.378 INFO:tasks.ceph.osd.3:Started 2023-11-24T08:04:33.378 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-24T08:04:33.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-24T08:04:33.387 INFO:tasks.ceph.osd.4:Started 2023-11-24T08:04:33.388 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-24T08:04:33.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-24T08:04:33.396 INFO:tasks.ceph.osd.5:Started 2023-11-24T08:04:33.396 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-24T08:04:33.397 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-24T08:04:33.402 INFO:tasks.ceph.osd.6:Started 2023-11-24T08:04:33.402 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-24T08:04:33.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-24T08:04:33.409 INFO:tasks.ceph.osd.7:Started 2023-11-24T08:04:33.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:33.483 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.481+0000 7f37e4441040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T08:04:33.492 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.490+0000 7f45db62e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:04:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.617+0000 7f786290f640 1 Processor -- start 2023-11-24T08:04:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.618+0000 7f786290f640 1 -- start start 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 0x7f785c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 0x7f785c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0cdf90 con 0x7f785c0b2740 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0ce230 con 0x7f785c161120 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f786290f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0d2430 con 0x7f785c160d40 2023-11-24T08:04:33.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f7853fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 0x7f785c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f7860eac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 0x7f785c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40260/0 (socket says 172.21.15.107:40260) 2023-11-24T08:04:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f7853fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 0x7f785c0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46862/0 (socket says 172.21.15.107:46862) 2023-11-24T08:04:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 -- 172.21.15.107:0/671161057 learned_addr learned my addr 172.21.15.107:0/671161057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 -- 172.21.15.107:0/671161057 >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 msgr2=0x7f785c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 --2- 172.21.15.107:0/671161057 >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 0x7f785c0d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 -- 172.21.15.107:0/671161057 >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 msgr2=0x7f785c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 --2- 172.21.15.107:0/671161057 >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 0x7f785c0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f785bfff640 1 -- 172.21.15.107:0/671161057 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0d1b90 con 0x7f785c160d40 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.619+0000 7f7860eac640 1 --2- 172.21.15.107:0/671161057 >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 0x7f785c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f785bfff640 1 --2- 172.21.15.107:0/671161057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f785c0b2c20 tx=0x7f784c02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e84309247112126e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f785b7fe640 1 -- 172.21.15.107:0/671161057 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c031860 con 0x7f785c160d40 2023-11-24T08:04:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f785b7fe640 1 -- 172.21.15.107:0/671161057 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c0319c0 con 0x7f785c160d40 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f785b7fe640 1 -- 172.21.15.107:0/671161057 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c031d50 con 0x7f785c160d40 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f786290f640 1 -- 172.21.15.107:0/671161057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 msgr2=0x7f785c0b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.621+0000 7f786290f640 1 --2- 172.21.15.107:0/671161057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f785c0b2c20 tx=0x7f784c02b900 comp rx=0 tx=0).stop 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 -- 172.21.15.107:0/671161057 shutdown_connections 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 --2- 172.21.15.107:0/671161057 >> v2:172.21.15.120:3300/0 conn(0x7f785c161120 0x7f785c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 --2- 172.21.15.107:0/671161057 >> v2:172.21.15.107:3300/0 conn(0x7f785c0b2740 0x7f785c0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 --2- 172.21.15.107:0/671161057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c160d40 0x7f785c0b21d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 -- 172.21.15.107:0/671161057 >> 172.21.15.107:0/671161057 conn(0x7f785c08af60 msgr2=0x7f785c0d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 -- 172.21.15.107:0/671161057 shutdown_connections 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.622+0000 7f786290f640 1 -- 172.21.15.107:0/671161057 wait complete. 2023-11-24T08:04:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.623+0000 7f786290f640 1 Processor -- start 2023-11-24T08:04:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.623+0000 7f786290f640 1 -- start start 2023-11-24T08:04:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 0x7f785c0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f785c161120 0x7f785c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1554f0 con 0x7f785c160d40 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0c7750 con 0x7f785c161120 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.624+0000 7f786290f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0cd140 con 0x7f785c0b2740 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f7853fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 0x7f785c0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f7860eac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f785c161120 0x7f785c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40264/0 (socket says 172.21.15.107:40264) 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 -- 172.21.15.107:0/4120583393 learned_addr learned my addr 172.21.15.107:0/4120583393 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 -- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f785c161120 msgr2=0x7f785c15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f785c161120 0x7f785c15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 -- 172.21.15.107:0/4120583393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 msgr2=0x7f785c0c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 0x7f785c0c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 -- 172.21.15.107:0/4120583393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854009a70 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f7853fff640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 0x7f785c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f785bfff640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f784c02b940 tx=0x7f784c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.625+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c03ac60 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c03adc0 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c042630 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0094d0 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c0c73e0 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f784c042a90 con 0x7f785c0b2740 2023-11-24T08:04:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.626+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f784c030a80 con 0x7f785c0b2740 2023-11-24T08:04:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.627+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c0c6f60 con 0x7f785c0b2740 2023-11-24T08:04:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.630+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f785c0c6f60 con 0x7f785c0b2740 2023-11-24T08:04:33.637 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.635+0000 7f45db62e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T08:04:33.682 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.680+0000 7f37e4441040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:04:33.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.692+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f785c067470 con 0x7f785c0b2740 2023-11-24T08:04:33.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.694+0000 7f78597fa640 1 -- 172.21.15.107:0/4120583393 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f784c047070 con 0x7f785c0b2740 2023-11-24T08:04:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:33.696 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:33.323479+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 msgr2=0x7f785c154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f784c02b940 tx=0x7f784c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 shutdown_connections 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f785c161120 0x7f785c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f785c160d40 0x7f785c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 --2- 172.21.15.107:0/4120583393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f785c0b2740 0x7f785c154fb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 >> 172.21.15.107:0/4120583393 conn(0x7f785c08af60 msgr2=0x7f785c0cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 shutdown_connections 2023-11-24T08:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:33.695+0000 7f786290f640 1 -- 172.21.15.107:0/4120583393 wait complete. 2023-11-24T08:04:33.712 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-24T08:04:33.713 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-24T08:04:33.734 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.732+0000 7f37e4441040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T08:04:33.795 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.793+0000 7f45db62e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:04:33.845 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:33.843+0000 7f45db62e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T08:04:33.895 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:33.892+0000 7f37e4441040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T08:04:34.004 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:34.002+0000 7f45db62e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T08:04:34.072 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-11-24T08:04:34.070+0000 7f96ae412500 -1 Falling back to public interface 2023-11-24T08:04:34.075 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-11-24T08:04:34.073+0000 7fe94f80e500 -1 Falling back to public interface 2023-11-24T08:04:34.085 INFO:tasks.ceph.osd.4.smithi120.stderr:2023-11-24T08:04:34.083+0000 7fd409c8c500 -1 Falling back to public interface 2023-11-24T08:04:34.085 INFO:tasks.ceph.osd.5.smithi120.stderr:2023-11-24T08:04:34.083+0000 7f606b4c3500 -1 Falling back to public interface 2023-11-24T08:04:34.086 INFO:tasks.ceph.osd.5.smithi120.stderr:2023-11-24T08:04:34.083+0000 7f606a46a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-11-24T08:04:34.086 INFO:tasks.ceph.osd.4.smithi120.stderr:2023-11-24T08:04:34.084+0000 7fd408c35640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2023-11-24T08:04:34.099 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-11-24T08:04:34.097+0000 7f7def211500 -1 Falling back to public interface 2023-11-24T08:04:34.101 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-11-24T08:04:34.099+0000 7f23ae6be500 -1 Falling back to public interface 2023-11-24T08:04:34.115 INFO:tasks.ceph.osd.6.smithi120.stderr:2023-11-24T08:04:34.112+0000 7faf9b412500 -1 Falling back to public interface 2023-11-24T08:04:34.119 INFO:tasks.ceph.osd.7.smithi120.stderr:2023-11-24T08:04:34.117+0000 7f14fd3f9500 -1 Falling back to public interface 2023-11-24T08:04:34.517 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:34.678 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.676+0000 7ff906733640 1 Processor -- start 2023-11-24T08:04:34.678 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.676+0000 7ff906733640 1 -- start start 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff906733640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 0x7ff900161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff906733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff900146230 0x7ff9000cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff906733640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9000704e0 con 0x7ff9000b1f30 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff906733640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff900146f10 con 0x7ff900146230 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff906733640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9000b3020 con 0x7ff900145e50 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff900146230 0x7ff9000cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff900146230 0x7ff9000cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46878/0 (socket says 172.21.15.107:46878) 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 -- 172.21.15.107:0/3702884363 learned_addr learned my addr 172.21.15.107:0/3702884363 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:34.680 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8fffff640 1 --2- 172.21.15.107:0/3702884363 >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 0x7ff900161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:34.681 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 -- 172.21.15.107:0/3702884363 >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 msgr2=0x7ff900161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:34.681 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 --2- 172.21.15.107:0/3702884363 >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 0x7ff900161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.681 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 -- 172.21.15.107:0/3702884363 >> v1:172.21.15.107:6789/0 conn(0x7ff9000b1f30 legacy=0x7ff90006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:34.681 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.678+0000 7ff8f7fff640 1 -- 172.21.15.107:0/3702884363 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9000b24a0 con 0x7ff900146230 2023-11-24T08:04:34.681 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.679+0000 7ff8fffff640 1 --2- 172.21.15.107:0/3702884363 >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 0x7ff900161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:04:34.682 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff8f7fff640 1 --2- 172.21.15.107:0/3702884363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff900146230 0x7ff9000cec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff90006e850 tx=0x7ff8e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef8c3e5402878db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:34.682 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff8ff7fe640 1 -- 172.21.15.107:0/3702884363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8030860 con 0x7ff900146230 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff8ff7fe640 1 -- 172.21.15.107:0/3702884363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e80309c0 con 0x7ff900146230 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff8ff7fe640 1 -- 172.21.15.107:0/3702884363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8030d50 con 0x7ff900146230 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff906733640 1 -- 172.21.15.107:0/3702884363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff900146230 msgr2=0x7ff9000cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.680+0000 7ff906733640 1 --2- 172.21.15.107:0/3702884363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff900146230 0x7ff9000cec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff90006e850 tx=0x7ff8e802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 -- 172.21.15.107:0/3702884363 shutdown_connections 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 --2- 172.21.15.107:0/3702884363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff900146230 0x7ff9000cec00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 --2- 172.21.15.107:0/3702884363 >> v2:172.21.15.120:3300/0 conn(0x7ff900145e50 0x7ff900161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 -- 172.21.15.107:0/3702884363 >> 172.21.15.107:0/3702884363 conn(0x7ff90006a490 msgr2=0x7ff90006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 -- 172.21.15.107:0/3702884363 shutdown_connections 2023-11-24T08:04:34.683 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.681+0000 7ff906733640 1 -- 172.21.15.107:0/3702884363 wait complete. 2023-11-24T08:04:34.684 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.682+0000 7ff906733640 1 Processor -- start 2023-11-24T08:04:34.684 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 -- start start 2023-11-24T08:04:34.686 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:34.686 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff900145e50 0x7ff90013d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff900146230 0x7ff90013d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9001605e0 con 0x7ff9000b1f30 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015f960 con 0x7ff900145e50 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.683+0000 7ff906733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015fc40 con 0x7ff900146230 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46884/0 (socket says 172.21.15.107:46884) 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 -- 172.21.15.107:0/237775398 learned_addr learned my addr 172.21.15.107:0/237775398 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8f7fff640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff900145e50 0x7ff90013d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff904cd0640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff900146230 0x7ff90013d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 -- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff900146230 msgr2=0x7ff90013d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff900146230 0x7ff90013d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.687 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 -- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff900145e50 msgr2=0x7ff90013d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff900145e50 0x7ff90013d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 -- 172.21.15.107:0/237775398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e80094d0 con 0x7ff9000b1f30 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.684+0000 7ff8fffff640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000e730 tx=0x7ff8f000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004230 con 0x7ff9000b1f30 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0004390 con 0x7ff9000b1f30 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000c8f0 con 0x7ff9000b1f30 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90015ff20 con 0x7ff9000b1f30 2023-11-24T08:04:34.688 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90013add0 con 0x7ff9000b1f30 2023-11-24T08:04:34.689 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff8f000cd50 con 0x7ff9000b1f30 2023-11-24T08:04:34.689 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.685+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff8f000ee40 con 0x7ff9000b1f30 2023-11-24T08:04:34.689 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.686+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90013dd20 con 0x7ff9000b1f30 2023-11-24T08:04:34.690 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.689+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff8f0020060 con 0x7ff9000b1f30 2023-11-24T08:04:34.757 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.755+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff900160850 con 0x7ff9000b1f30 2023-11-24T08:04:34.758 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.756+0000 7ff8fd7fa640 1 -- 172.21.15.107:0/237775398 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7ff8f000f040 con 0x7ff9000b1f30 2023-11-24T08:04:34.758 INFO:teuthology.misc.health.smithi107.stdout: 2023-11-24T08:04:34.758 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:33.323479+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:34.759 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 msgr2=0x7ff90013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:34.759 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000e730 tx=0x7ff8f000ebf0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.759 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 shutdown_connections 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff900146230 0x7ff90013d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff900145e50 0x7ff90013d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 --2- 172.21.15.107:0/237775398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9000b1f30 0x7ff90013a890 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 >> 172.21.15.107:0/237775398 conn(0x7ff90006a490 msgr2=0x7ff9001424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.757+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 shutdown_connections 2023-11-24T08:04:34.760 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:34.758+0000 7ff906733640 1 -- 172.21.15.107:0/237775398 wait complete. 2023-11-24T08:04:34.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T08:04:34.931 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:34.930+0000 7f37e4441040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T08:04:35.065 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:35.063+0000 7f45db62e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T08:04:35.235 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:35.233+0000 7f37e4441040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T08:04:35.293 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:35.291+0000 7f37e4441040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T08:04:35.366 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:35.364+0000 7f45db62e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T08:04:35.405 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:35.403+0000 7f37e4441040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T08:04:35.427 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:35.425+0000 7f45db62e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T08:04:35.547 INFO:tasks.ceph.mgr.x.smithi120.stderr:2023-11-24T08:04:35.545+0000 7f45db62e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T08:04:36.446 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:36.444+0000 7f3785862640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-24T08:04:36.446 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:04:36.444+0000 7f3785862640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-24T08:04:37.593 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-11-24T08:04:37.592+0000 7f96ae412500 -1 osd.1 0 log_to_monitors true 2023-11-24T08:04:37.685 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-11-24T08:04:37.683+0000 7f23ae6be500 -1 osd.3 0 log_to_monitors true 2023-11-24T08:04:37.716 INFO:tasks.ceph.osd.5.smithi120.stderr:2023-11-24T08:04:37.714+0000 7f606b4c3500 -1 osd.5 0 log_to_monitors true 2023-11-24T08:04:37.717 INFO:tasks.ceph.osd.4.smithi120.stderr:2023-11-24T08:04:37.715+0000 7fd409c8c500 -1 osd.4 0 log_to_monitors true 2023-11-24T08:04:37.726 INFO:tasks.ceph.osd.7.smithi120.stderr:2023-11-24T08:04:37.724+0000 7f14fd3f9500 -1 osd.7 0 log_to_monitors true 2023-11-24T08:04:37.726 INFO:tasks.ceph.osd.6.smithi120.stderr:2023-11-24T08:04:37.725+0000 7faf9b412500 -1 osd.6 0 log_to_monitors true 2023-11-24T08:04:37.901 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-11-24T08:04:37.899+0000 7fe94f80e500 -1 osd.0 0 log_to_monitors true 2023-11-24T08:04:37.919 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-11-24T08:04:37.917+0000 7f7def211500 -1 osd.2 0 log_to_monitors true 2023-11-24T08:04:38.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:04:38.652 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:04:41.579 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:41.726 INFO:tasks.ceph.osd.5.smithi120.stderr:2023-11-24T08:04:41.723+0000 7f6066c2e640 -1 osd.5 0 waiting for initial osdmap 2023-11-24T08:04:41.735 INFO:tasks.ceph.osd.5.smithi120.stderr:2023-11-24T08:04:41.733+0000 7f605f267640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.741 INFO:tasks.ceph.osd.6.smithi120.stderr:2023-11-24T08:04:41.739+0000 7faf964bc640 -1 osd.6 0 waiting for initial osdmap 2023-11-24T08:04:41.757 INFO:tasks.ceph.osd.6.smithi120.stderr:2023-11-24T08:04:41.755+0000 7faf8eaf5640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.796 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-11-24T08:04:41.795+0000 7f7dea3bb640 -1 osd.2 0 waiting for initial osdmap 2023-11-24T08:04:41.802 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-11-24T08:04:41.800+0000 7f7de29f4640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.807 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-11-24T08:04:41.805+0000 7fe94a954640 -1 osd.0 0 waiting for initial osdmap 2023-11-24T08:04:41.811 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.809+0000 7f59fdfb0640 1 Processor -- start 2023-11-24T08:04:41.811 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.809+0000 7f59fdfb0640 1 -- start start 2023-11-24T08:04:41.812 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59fdfb0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59f80cb7b0 0x7f59f80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59fdfb0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f80b52e0 con 0x7f59f815b630 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59fdfb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f80b4a10 con 0x7f59f80cbc10 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59fdfb0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f80c53c0 con 0x7f59f80cb7b0 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59f77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f59f815b630 0x7f59f80c4cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54142/0 (socket says 172.21.15.107:54142) 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59f77fe640 1 -- 172.21.15.107:0/14324641 learned_addr learned my addr 172.21.15.107:0/14324641 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:41.813 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.811+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/14324641 >> v2:172.21.15.120:3300/0 conn(0x7f59f80cb7b0 0x7f59f80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:41.814 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.812+0000 7f59f67fc640 1 -- 172.21.15.107:0/14324641 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332633036 0 0) 0x7f59f80b52e0 con 0x7f59f815b630 2023-11-24T08:04:41.814 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.812+0000 7f59f67fc640 1 -- 172.21.15.107:0/14324641 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0003520 con 0x7f59f815b630 2023-11-24T08:04:41.814 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.812+0000 7f59f6ffd640 1 -- 172.21.15.107:0/14324641 >> v1:172.21.15.120:6789/0 conn(0x7f59f80cbc10 legacy=0x7f59f80b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:41.814 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.812+0000 7f59f6ffd640 1 -- 172.21.15.107:0/14324641 >> v1:172.21.15.107:6789/0 conn(0x7f59f815b630 legacy=0x7f59f80c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:41.814 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.812+0000 7f59f6ffd640 1 -- 172.21.15.107:0/14324641 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f80cc1e0 con 0x7f59f80cb7b0 2023-11-24T08:04:41.815 INFO:tasks.ceph.osd.7.smithi120.stderr:2023-11-24T08:04:41.813+0000 7f14f8b64640 -1 osd.7 0 waiting for initial osdmap 2023-11-24T08:04:41.815 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.814+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/14324641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f80cb7b0 0x7f59f80b2cc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59e80097a0 tx=0x7f59e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1bff9f17e688e02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:41.816 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.814+0000 7f59f67fc640 1 -- 172.21.15.107:0/14324641 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8030610 con 0x7f59f80cb7b0 2023-11-24T08:04:41.816 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.814+0000 7f59f67fc640 1 -- 172.21.15.107:0/14324641 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8030770 con 0x7f59f80cb7b0 2023-11-24T08:04:41.816 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.814+0000 7f59f67fc640 1 -- 172.21.15.107:0/14324641 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8030b00 con 0x7f59f80cb7b0 2023-11-24T08:04:41.817 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 -- 172.21.15.107:0/14324641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f80cb7b0 msgr2=0x7f59f80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:41.817 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/14324641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f80cb7b0 0x7f59f80b2cc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59e80097a0 tx=0x7f59e802b960 comp rx=0 tx=0).stop 2023-11-24T08:04:41.817 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 -- 172.21.15.107:0/14324641 shutdown_connections 2023-11-24T08:04:41.817 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/14324641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f80cb7b0 0x7f59f80b2cc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:41.817 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 -- 172.21.15.107:0/14324641 >> 172.21.15.107:0/14324641 conn(0x7f59f8070110 msgr2=0x7f59f80704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:41.818 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 -- 172.21.15.107:0/14324641 shutdown_connections 2023-11-24T08:04:41.818 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.815+0000 7f59fdfb0640 1 -- 172.21.15.107:0/14324641 wait complete. 2023-11-24T08:04:41.820 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.818+0000 7f59fdfb0640 1 Processor -- start 2023-11-24T08:04:41.820 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.818+0000 7f59fdfb0640 1 -- start start 2023-11-24T08:04:41.822 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:41.822 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 0x7f59f806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f815b630 0x7f59f8154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f806d0f0 con 0x7f59f80cbc10 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f806c4a0 con 0x7f59f815b630 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.820+0000 7f59fdfb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81549b0 con 0x7f59f80cb7b0 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:55268/0 (socket says 172.21.15.107:55268) 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 -- 172.21.15.107:0/954559171 learned_addr learned my addr 172.21.15.107:0/954559171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 0x7f59f806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:41.823 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f7fff640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f815b630 0x7f59f8154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:41.824 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 -- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f815b630 msgr2=0x7f59f8154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:41.824 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f815b630 0x7f59f8154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 -- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 msgr2=0x7f59f806bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 0x7f59f806bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f77fe640 1 -- 172.21.15.107:0/954559171 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8009400 con 0x7f59f80cb7b0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.821+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 0x7f59f806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59f77fe640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59f000cbc0 tx=0x7f59f000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000ad40 con 0x7f59f80cb7b0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f0012400 con 0x7f59f80cb7b0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000aea0 con 0x7f59f80cb7b0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f8156590 con 0x7f59f80cb7b0 2023-11-24T08:04:41.825 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.822+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f8156b50 con 0x7f59f80cb7b0 2023-11-24T08:04:41.826 INFO:tasks.ceph.osd.7.smithi120.stderr:2023-11-24T08:04:41.822+0000 7f14f119d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.826 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.823+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f80cd960 con 0x7f59f80cb7b0 2023-11-24T08:04:41.829 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-11-24T08:04:41.824+0000 7fe942f8d640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.829 INFO:tasks.ceph.osd.4.smithi120.stderr:2023-11-24T08:04:41.825+0000 7fd406c31640 -1 osd.4 0 waiting for initial osdmap 2023-11-24T08:04:41.830 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.826+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f59f0007070 con 0x7f59f80cb7b0 2023-11-24T08:04:41.830 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.827+0000 7f59dffff640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 0x7f59e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:41.830 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.827+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f59f0088700 con 0x7f59f80cb7b0 2023-11-24T08:04:41.830 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.828+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 0x7f59e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:41.830 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.828+0000 7f59f6ffd640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 0x7f59e406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59e8036000 tx=0x7f59e803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:41.833 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:41.832+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59f00546c0 con 0x7f59f80cb7b0 2023-11-24T08:04:41.853 INFO:tasks.ceph.osd.4.smithi120.stderr:2023-11-24T08:04:41.851+0000 7fd3fda32640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.868 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-11-24T08:04:41.866+0000 7f96a9554640 -1 osd.1 0 waiting for initial osdmap 2023-11-24T08:04:41.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-11-24T08:04:41.872+0000 7f96a1b8d640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:41.880 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-11-24T08:04:41.878+0000 7f23a9e29640 -1 osd.3 0 waiting for initial osdmap 2023-11-24T08:04:41.897 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-11-24T08:04:41.896+0000 7f23a2462640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:04:42.033 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.030+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59f80d1f30 con 0x7f59f80cb7b0 2023-11-24T08:04:42.034 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.032+0000 7f59dffff640 1 -- 172.21.15.107:0/954559171 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f59f0017090 con 0x7f59f80cb7b0 2023-11-24T08:04:42.034 INFO:teuthology.misc.health.smithi107.stdout: 2023-11-24T08:04:42.034 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":12,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:39.446511+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:42.036 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 msgr2=0x7f59e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:42.036 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 0x7f59e406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59e8036000 tx=0x7f59e803b040 comp rx=0 tx=0).stop 2023-11-24T08:04:42.036 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 msgr2=0x7f59f8157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:42.036 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59f000cbc0 tx=0x7f59f000a430 comp rx=0 tx=0).stop 2023-11-24T08:04:42.036 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 shutdown_connections 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f59e406adb0 0x7f59e406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59f815b630 0x7f59f8154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59f80cbc10 0x7f59f806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 --2- 172.21.15.107:0/954559171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59f80cb7b0 0x7f59f8157630 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 >> 172.21.15.107:0/954559171 conn(0x7f59f8070110 msgr2=0x7f59f80cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 shutdown_connections 2023-11-24T08:04:42.037 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:42.034+0000 7f59fdfb0640 1 -- 172.21.15.107:0/954559171 wait complete. 2023-11-24T08:04:42.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T08:04:48.856 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:49.042 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.041+0000 7f7e17548640 1 Processor -- start 2023-11-24T08:04:49.042 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.041+0000 7f7e17548640 1 -- start start 2023-11-24T08:04:49.044 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.042+0000 7f7e17548640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7e100703a0 0x7f7e10070830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.044 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.042+0000 7f7e17548640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e10162cf0 0x7f7e10163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.044 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.042+0000 7f7e17548640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100b3930 con 0x7f7e1014b440 2023-11-24T08:04:49.044 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.042+0000 7f7e17548640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10070d70 con 0x7f7e10162cf0 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e17548640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10161f70 con 0x7f7e100703a0 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e10162cf0 0x7f7e10163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e10162cf0 0x7f7e10163160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57290/0 (socket says 172.21.15.107:57290) 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 -- 172.21.15.107:0/1311955433 learned_addr learned my addr 172.21.15.107:0/1311955433 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e14ae3640 1 --2- 172.21.15.107:0/1311955433 >> v2:172.21.15.120:3300/0 conn(0x7f7e100703a0 0x7f7e10070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 -- 172.21.15.107:0/1311955433 >> v2:172.21.15.120:3300/0 conn(0x7f7e100703a0 msgr2=0x7f7e10070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 --2- 172.21.15.107:0/1311955433 >> v2:172.21.15.120:3300/0 conn(0x7f7e100703a0 0x7f7e10070830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.045 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 -- 172.21.15.107:0/1311955433 >> v1:172.21.15.120:6789/0 conn(0x7f7e1014b440 legacy=0x7f7e100b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:49.046 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.043+0000 7f7e15ae5640 1 -- 172.21.15.107:0/1311955433 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10162210 con 0x7f7e10162cf0 2023-11-24T08:04:49.047 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.044+0000 7f7e15ae5640 1 --2- 172.21.15.107:0/1311955433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e10162cf0 0x7f7e10163160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006bc90 tx=0x7f7e0802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6d18f50950921 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e07fff640 1 -- 172.21.15.107:0/1311955433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0802ca80 con 0x7f7e10162cf0 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e07fff640 1 -- 172.21.15.107:0/1311955433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0802cbe0 con 0x7f7e10162cf0 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e07fff640 1 -- 172.21.15.107:0/1311955433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e080343f0 con 0x7f7e10162cf0 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 -- 172.21.15.107:0/1311955433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e10162cf0 msgr2=0x7f7e10163160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 --2- 172.21.15.107:0/1311955433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e10162cf0 0x7f7e10163160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006bc90 tx=0x7f7e0802bee0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 -- 172.21.15.107:0/1311955433 shutdown_connections 2023-11-24T08:04:49.048 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 --2- 172.21.15.107:0/1311955433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e10162cf0 0x7f7e10163160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.049 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 --2- 172.21.15.107:0/1311955433 >> v2:172.21.15.120:3300/0 conn(0x7f7e100703a0 0x7f7e10070830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.049 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 -- 172.21.15.107:0/1311955433 >> 172.21.15.107:0/1311955433 conn(0x7f7e1006a490 msgr2=0x7f7e1006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:49.049 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 -- 172.21.15.107:0/1311955433 shutdown_connections 2023-11-24T08:04:49.049 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.045+0000 7f7e17548640 1 -- 172.21.15.107:0/1311955433 wait complete. 2023-11-24T08:04:49.052 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.047+0000 7f7e17548640 1 Processor -- start 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.047+0000 7f7e17548640 1 -- start start 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e100703a0 0x7f7e1013f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e10162cf0 0x7f7e10160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10160b40 con 0x7f7e1014b440 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1013ffa0 con 0x7f7e100703a0 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e17548640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10140280 con 0x7f7e10162cf0 2023-11-24T08:04:49.053 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e14ae3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e14ae3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57300/0 (socket says 172.21.15.107:57300) 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.048+0000 7f7e14ae3640 1 -- 172.21.15.107:0/2365316036 learned_addr learned my addr 172.21.15.107:0/2365316036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e15ae5640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e10162cf0 0x7f7e10160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 -- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e10162cf0 msgr2=0x7f7e10160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e10162cf0 0x7f7e10160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 -- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e100703a0 msgr2=0x7f7e1013f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e152e4640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e100703a0 0x7f7e1013f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e100703a0 0x7f7e1013f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 -- 172.21.15.107:0/2365316036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e08009b30 con 0x7f7e1014b440 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.049+0000 7f7e14ae3640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000d800 tx=0x7f7e0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.054 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.050+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000b6c0 con 0x7f7e1014b440 2023-11-24T08:04:49.055 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.050+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0000b820 con 0x7f7e1014b440 2023-11-24T08:04:49.055 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.050+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e10140560 con 0x7f7e1014b440 2023-11-24T08:04:49.055 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.050+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000bbb0 con 0x7f7e1014b440 2023-11-24T08:04:49.055 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.050+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10140b20 con 0x7f7e1014b440 2023-11-24T08:04:49.055 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.051+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10140ce0 con 0x7f7e1014b440 2023-11-24T08:04:49.056 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.052+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7e0000bd10 con 0x7f7e1014b440 2023-11-24T08:04:49.056 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.053+0000 7f7e05ffb640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 0x7f7df406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.056 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.053+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e00089900 con 0x7f7e1014b440 2023-11-24T08:04:49.056 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.053+0000 7f7e152e4640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 0x7f7df406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.056 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.054+0000 7f7e152e4640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 0x7f7df406d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0015d0 tx=0x7f7e0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.059 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.059+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e00054c60 con 0x7f7e1014b440 2023-11-24T08:04:49.252 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.251+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e100ae6f0 con 0x7f7e1014b440 2023-11-24T08:04:49.254 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.253+0000 7f7e05ffb640 1 -- 172.21.15.107:0/2365316036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f7e00058280 con 0x7f7e1014b440 2023-11-24T08:04:49.254 INFO:teuthology.misc.health.smithi107.stdout: 2023-11-24T08:04:49.255 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":16,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:45.777870+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:49.257 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 msgr2=0x7f7df406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 0x7f7df406d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0015d0 tx=0x7f7e0c009040 comp rx=0 tx=0).stop 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 msgr2=0x7f7e100ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000d800 tx=0x7f7e0000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 shutdown_connections 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7df406ae40 0x7f7df406d300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e10162cf0 0x7f7e10160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.258 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e1014b440 0x7f7e100ccd70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.259 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 --2- 172.21.15.107:0/2365316036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e100703a0 0x7f7e1013f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.259 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 >> 172.21.15.107:0/2365316036 conn(0x7f7e1006a490 msgr2=0x7f7e10164740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:49.259 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 shutdown_connections 2023-11-24T08:04:49.259 INFO:teuthology.misc.health.smithi107.stderr:2023-11-24T08:04:49.256+0000 7f7e17548640 1 -- 172.21.15.107:0/2365316036 wait complete. 2023-11-24T08:04:49.273 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-24T08:04:49.273 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-24T08:04:49.273 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:04:49.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:04:49.278 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:04:49.279 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:04:49.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:04:49.284 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:04:49.284 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:04:49.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:04:49.294 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:04:49.294 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:04:49.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:04:49.303 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:04:49.304 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-24T08:04:49.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:49.435 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:04:49.442 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:04:49.459 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:04:49.466 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:04:49.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.491+0000 7fe0019e5640 1 Processor -- start 2023-11-24T08:04:49.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.491+0000 7fe0019e5640 1 -- start start 2023-11-24T08:04:49.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fe0019e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fe0019e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdffc0d1040 0x7fdffc0d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fe0019e5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0d3da0 con 0x7fdffc15d780 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fe0019e5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdffc0ce4f0 con 0x7fdffc0d1040 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fe0019e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0d1d30 con 0x7fdffc0d3400 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdffc0d1040 0x7fdffc0d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffa7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffa7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:49800/0 (socket says 172.21.15.107:49800) 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdffc0d1040 0x7fdffc0d14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57348/0 (socket says 172.21.15.107:57348) 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffa7fc640 1 -- 172.21.15.107:0/1358572150 learned_addr learned my addr 172.21.15.107:0/1358572150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.492+0000 7fdffaffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdffc15d780 0x7fdffc0b3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:45370/0 (socket says 172.21.15.107:45370) 2023-11-24T08:04:49.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdffb7fe640 1 -- 172.21.15.107:0/1358572150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 msgr2=0x7fdffc0d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdffb7fe640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdffb7fe640 1 -- 172.21.15.107:0/1358572150 >> v1:172.21.15.120:6789/0 conn(0x7fdffc15d780 legacy=0x7fdffc0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdffb7fe640 1 -- 172.21.15.107:0/1358572150 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0d2010 con 0x7fdffc0d1040 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdffa7fc640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.493+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1358572150 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857093074 0 0) 0x7fdffc0d3da0 con 0x7fdffc15d780 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fdffb7fe640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d1040 0x7fdffc0d14d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009d90 tx=0x7fdff402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c901dc600e54c2f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1358572150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff402cb30 con 0x7fdffc0d1040 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1358572150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff402cc90 con 0x7fdffc0d1040 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1358572150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4034440 con 0x7fdffc0d1040 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fe0019e5640 1 -- 172.21.15.107:0/1358572150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d1040 msgr2=0x7fdffc0d14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fe0019e5640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d1040 0x7fdffc0d14d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009d90 tx=0x7fdff402bd80 comp rx=0 tx=0).stop 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fe0019e5640 1 -- 172.21.15.107:0/1358572150 shutdown_connections 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fe0019e5640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d1040 0x7fdffc0d14d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.494+0000 7fe0019e5640 1 --2- 172.21.15.107:0/1358572150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d3400 0x7fdffc0d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.495+0000 7fe0019e5640 1 -- 172.21.15.107:0/1358572150 >> 172.21.15.107:0/1358572150 conn(0x7fdffc08cec0 msgr2=0x7fdffc09c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.495+0000 7fe0019e5640 1 -- 172.21.15.107:0/1358572150 shutdown_connections 2023-11-24T08:04:49.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.495+0000 7fe0019e5640 1 -- 172.21.15.107:0/1358572150 wait complete. 2023-11-24T08:04:49.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.496+0000 7fe0019e5640 1 Processor -- start 2023-11-24T08:04:49.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.496+0000 7fe0019e5640 1 -- start start 2023-11-24T08:04:49.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 0x7fdffc1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 0x7fdffc1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc1f6710 con 0x7fdffc0d3400 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc1f3c50 con 0x7fdffc15d780 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fe0019e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc1f3f30 con 0x7fdffc0d1040 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.497+0000 7fdffa7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57360/0 (socket says 172.21.15.107:57360) 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 -- 172.21.15.107:0/3419474892 learned_addr learned my addr 172.21.15.107:0/3419474892 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffaffd640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 0x7fdffc1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffb7fe640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 0x7fdffc1f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 -- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 msgr2=0x7fdffc1f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 0x7fdffc1f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 -- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 msgr2=0x7fdffc1f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 0x7fdffc1f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 -- 172.21.15.107:0/3419474892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff00094d0 con 0x7fdffc0d3400 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffaffd640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 0x7fdffc1f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffb7fe640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 0x7fdffc1f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.498+0000 7fdffa7fc640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e9d0 tx=0x7fdff000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.499+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff000ca60 con 0x7fdffc0d3400 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.499+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff000cbc0 con 0x7fdffc0d3400 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.499+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0013470 con 0x7fdffc0d3400 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.499+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff4009a70 con 0x7fdffc0d3400 2023-11-24T08:04:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.499+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc1f6cb0 con 0x7fdffc0d3400 2023-11-24T08:04:49.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.500+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc067470 con 0x7fdffc0d3400 2023-11-24T08:04:49.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.501+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdff00135d0 con 0x7fdffc0d3400 2023-11-24T08:04:49.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.502+0000 7fdfe3fff640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 0x7fdfe806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.502+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdff0089580 con 0x7fdffc0d3400 2023-11-24T08:04:49.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.502+0000 7fdffaffd640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 0x7fdfe806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.503+0000 7fdffaffd640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 0x7fdfe806d300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfec000f60 tx=0x7fdfec007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.506+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdff0054990 con 0x7fdffc0d3400 2023-11-24T08:04:49.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.700+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdffc0b2f60 con 0x7fdffc0d3400 2023-11-24T08:04:49.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.702+0000 7fdfe3fff640 1 -- 172.21.15.107:0/3419474892 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fdff0057fb0 con 0x7fdffc0d3400 2023-11-24T08:04:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:49.703 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:45.777870+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.704+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 msgr2=0x7fdfe806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.704+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 0x7fdfe806d300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfec000f60 tx=0x7fdfec007d50 comp rx=0 tx=0).stop 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.704+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 msgr2=0x7fdffc15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.704+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e9d0 tx=0x7fdff000ee90 comp rx=0 tx=0).stop 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 shutdown_connections 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806ae40 0x7fdfe806d300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdffc15d780 0x7fdffc1f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdffc0d3400 0x7fdffc15e280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 --2- 172.21.15.107:0/3419474892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdffc0d1040 0x7fdffc1f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 >> 172.21.15.107:0/3419474892 conn(0x7fdffc08cec0 msgr2=0x7fdffc0d2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 shutdown_connections 2023-11-24T08:04:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.705+0000 7fe0019e5640 1 -- 172.21.15.107:0/3419474892 wait complete. 2023-11-24T08:04:49.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:04:49.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:04:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.930+0000 7fc30e73c640 1 Processor -- start 2023-11-24T08:04:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.930+0000 7fc30e73c640 1 -- start start 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc30e73c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc30815d720 0x7fc3080d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc30e73c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30815db00 con 0x7fc3080d3870 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc30e73c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080d0ce0 con 0x7fc30806ca90 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc30e73c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080d3d20 con 0x7fc30815d720 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc307fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc30815d720 0x7fc3080d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc2ff7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc3080d3870 0x7fc3080b0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34830/0 (socket says 172.21.15.107:34830) 2023-11-24T08:04:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.932+0000 7fc2ff7fe640 1 -- 172.21.15.107:0/1971351513 learned_addr learned my addr 172.21.15.107:0/1971351513 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:49.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.933+0000 7fc3077fe640 1 -- 172.21.15.107:0/1971351513 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024490961 0 0) 0x7fc30815db00 con 0x7fc3080d3870 2023-11-24T08:04:49.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.933+0000 7fc307fff640 1 -- 172.21.15.107:0/1971351513 >> v1:172.21.15.120:6789/0 conn(0x7fc30806ca90 legacy=0x7fc3080b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:49.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.933+0000 7fc307fff640 1 -- 172.21.15.107:0/1971351513 >> v1:172.21.15.107:6789/0 conn(0x7fc3080d3870 legacy=0x7fc3080b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.933+0000 7fc307fff640 1 -- 172.21.15.107:0/1971351513 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30806d400 con 0x7fc30815d720 2023-11-24T08:04:49.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc307fff640 1 --2- 172.21.15.107:0/1971351513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc3080d3330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009d90 tx=0x7fc2f802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4ae8ca16c957f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc3077fe640 1 -- 172.21.15.107:0/1971351513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f802cbd0 con 0x7fc30815d720 2023-11-24T08:04:49.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc3077fe640 1 -- 172.21.15.107:0/1971351513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f802cd30 con 0x7fc30815d720 2023-11-24T08:04:49.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc3077fe640 1 -- 172.21.15.107:0/1971351513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f802cea0 con 0x7fc30815d720 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc30e73c640 1 -- 172.21.15.107:0/1971351513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 msgr2=0x7fc3080d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.934+0000 7fc30e73c640 1 --2- 172.21.15.107:0/1971351513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc3080d3330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009d90 tx=0x7fc2f802be20 comp rx=0 tx=0).stop 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.935+0000 7fc30e73c640 1 -- 172.21.15.107:0/1971351513 shutdown_connections 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.935+0000 7fc30e73c640 1 --2- 172.21.15.107:0/1971351513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc3080d3330 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.935+0000 7fc30e73c640 1 -- 172.21.15.107:0/1971351513 >> 172.21.15.107:0/1971351513 conn(0x7fc30808af60 msgr2=0x7fc30809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.935+0000 7fc30e73c640 1 -- 172.21.15.107:0/1971351513 shutdown_connections 2023-11-24T08:04:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.935+0000 7fc30e73c640 1 -- 172.21.15.107:0/1971351513 wait complete. 2023-11-24T08:04:49.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.936+0000 7fc30e73c640 1 Processor -- start 2023-11-24T08:04:49.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.937+0000 7fc30e73c640 1 -- start start 2023-11-24T08:04:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc30806ca90 0x7fc3080b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3080d3870 0x7fc3080ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080c8070 con 0x7fc30815d720 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b9180 con 0x7fc3080d3870 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30e73c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080c73f0 con 0x7fc30806ca90 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57372/0 (socket says 172.21.15.107:57372) 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 -- 172.21.15.107:0/3401566880 learned_addr learned my addr 172.21.15.107:0/3401566880 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc307fff640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc30806ca90 0x7fc3080b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc2ff7fe640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3080d3870 0x7fc3080ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 -- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc30806ca90 msgr2=0x7fc3080b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc30806ca90 0x7fc3080b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 -- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3080d3870 msgr2=0x7fc3080ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3080d3870 0x7fc3080ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.938+0000 7fc30ccd9640 1 -- 172.21.15.107:0/3401566880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f8009a70 con 0x7fc30815d720 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc30ccd9640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c410 tx=0x7fc30000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011790 con 0x7fc30815d720 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3000118f0 con 0x7fc30815d720 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3080c76d0 con 0x7fc30815d720 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011ce0 con 0x7fc30815d720 2023-11-24T08:04:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.939+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3080c7c90 con 0x7fc30815d720 2023-11-24T08:04:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.940+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080c8410 con 0x7fc30815d720 2023-11-24T08:04:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.941+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc30001c460 con 0x7fc30815d720 2023-11-24T08:04:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.942+0000 7fc3057fa640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.942+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc300088fa0 con 0x7fc30815d720 2023-11-24T08:04:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.942+0000 7fc307fff640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.943+0000 7fc307fff640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 0x7fc2e806d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8036000 tx=0x7fc2f8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:49.947+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3080c8410 con 0x7fc30815d720 2023-11-24T08:04:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.134+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3080c8410 con 0x7fc30815d720 2023-11-24T08:04:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.135+0000 7fc3057fa640 1 -- 172.21.15.107:0/3401566880 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3080c8410 con 0x7fc30815d720 2023-11-24T08:04:50.136 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:04:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 msgr2=0x7fc2e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 0x7fc2e806d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8036000 tx=0x7fc2f8030000 comp rx=0 tx=0).stop 2023-11-24T08:04:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 msgr2=0x7fc308150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c410 tx=0x7fc30000c8d0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 shutdown_connections 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc30815d720 0x7fc308150ac0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3080d3870 0x7fc3080ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 --2- 172.21.15.107:0/3401566880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc30806ca90 0x7fc3080b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 >> 172.21.15.107:0/3401566880 conn(0x7fc30808af60 msgr2=0x7fc3080b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.137+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 shutdown_connections 2023-11-24T08:04:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.138+0000 7fc30e73c640 1 -- 172.21.15.107:0/3401566880 wait complete. 2023-11-24T08:04:50.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:04:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.360+0000 7f853b59e640 1 Processor -- start 2023-11-24T08:04:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.361+0000 7f853b59e640 1 -- start start 2023-11-24T08:04:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.362+0000 7f853b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f853c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.362+0000 7f853b59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f853c06d6c0 0x7f853c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.362+0000 7f853b59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c06e060 con 0x7f853c0b0a70 2023-11-24T08:04:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.362+0000 7f853b59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c13f9e0 con 0x7f853c06d6c0 2023-11-24T08:04:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.362+0000 7f853b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c13fcc0 con 0x7f853c13d120 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f853ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f853c0b0a70 0x7f853c0b0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34834/0 (socket says 172.21.15.107:34834) 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f853ad9d640 1 -- 172.21.15.107:0/283132224 learned_addr learned my addr 172.21.15.107:0/283132224 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f853a59c640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f853c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8533fff640 1 --2- 172.21.15.107:0/283132224 >> v2:172.21.15.120:3300/0 conn(0x7f853c06d6c0 0x7f853c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277308615 0 0) 0x7f853c06e060 con 0x7f853c0b0a70 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528005730 con 0x7f853c0b0a70 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 301404079 0 0) 0x7f8528005730 con 0x7f853c0b0a70 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 msgr2=0x7f853c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f853c15af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 >> v2:172.21.15.120:3300/0 conn(0x7f853c06d6c0 msgr2=0x7f853c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 --2- 172.21.15.107:0/283132224 >> v2:172.21.15.120:3300/0 conn(0x7f853c06d6c0 0x7f853c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0cec00 con 0x7f853c0b0a70 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.363+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f85340019f0 con 0x7f853c0b0a70 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 >> v1:172.21.15.107:6789/0 conn(0x7f853c0b0a70 legacy=0x7f853c0b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f8528005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8528006100 0x7f85280069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8528006f10 0x7f8528007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c13f9e0 con 0x7f8528006f10 2023-11-24T08:04:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c13fcc0 con 0x7f8528006100 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852800dbe0 con 0x7f853c13d120 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8528006f10 0x7f8528007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853a59c640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f8528005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f8533fff640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8528006100 0x7f85280069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 -- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 msgr2=0x7f8528005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c13d120 0x7f8528005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 -- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8528006100 msgr2=0x7f85280069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8528006100 0x7f85280069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.364+0000 7f853ad9d640 1 -- 172.21.15.107:0/283132224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8528005730 con 0x7f8528006f10 2023-11-24T08:04:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.365+0000 7f853ad9d640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8528006f10 0x7f8528007390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f853400a890 tx=0x7f8528007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4329c6efcb0cb1d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534034630 con 0x7f8528006f10 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853402f070 con 0x7f8528006f10 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f8539d9b640 1 -- 172.21.15.107:0/283132224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534034a70 con 0x7f8528006f10 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 -- 172.21.15.107:0/283132224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8528006f10 msgr2=0x7f8528007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 --2- 172.21.15.107:0/283132224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8528006f10 0x7f8528007390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f853400a890 tx=0x7f8528007a00 comp rx=0 tx=0).stop 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853ad9d640 1 -- 172.21.15.107:0/283132224 reap_dead start 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 -- 172.21.15.107:0/283132224 shutdown_connections 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 -- 172.21.15.107:0/283132224 >> 172.21.15.107:0/283132224 conn(0x7f853c0cb830 msgr2=0x7f853c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 -- 172.21.15.107:0/283132224 shutdown_connections 2023-11-24T08:04:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.366+0000 7f853b59e640 1 -- 172.21.15.107:0/283132224 wait complete. 2023-11-24T08:04:50.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.368+0000 7f853b59e640 1 Processor -- start 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.368+0000 7f853b59e640 1 -- start start 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853c139760 0x7f853c0b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c139b40 0x7f853c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15aeb0 con 0x7f853c1407a0 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15b190 con 0x7f853c139760 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.369+0000 7f853b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15c7e0 con 0x7f853c139b40 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57380/0 (socket says 172.21.15.107:57380) 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 -- 172.21.15.107:0/3781378993 learned_addr learned my addr 172.21.15.107:0/3781378993 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853a59c640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853c139760 0x7f853c0b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f8533fff640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c139b40 0x7f853c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 -- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c139b40 msgr2=0x7f853c159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c139b40 0x7f853c159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 -- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853c139760 msgr2=0x7f853c0b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853c139760 0x7f853c0b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 -- 172.21.15.107:0/3781378993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c15cac0 con 0x7f853c1407a0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.370+0000 7f853ad9d640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f853400a830 tx=0x7f8534001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.371+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534034840 con 0x7f853c1407a0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.371+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85340349a0 con 0x7f853c1407a0 2023-11-24T08:04:50.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.371+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534034d10 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.371+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c15cda0 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.371+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c140c20 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.372+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c04e840 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.373+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8534031030 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.374+0000 7f8532ffd640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 0x7f851c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.374+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85340aae70 con 0x7f853c1407a0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.374+0000 7f853a59c640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 0x7f851c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.375+0000 7f853a59c640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 0x7f851c06d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f852c009a10 tx=0x7f852c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.379+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8534076200 con 0x7f853c1407a0 2023-11-24T08:04:50.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.603+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f853c15b2d0 con 0x7f853c1407a0 2023-11-24T08:04:50.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.604+0000 7f8532ffd640 1 -- 172.21.15.107:0/3781378993 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3706 (secure 0 0 0) 0x7f8534045070 con 0x7f853c1407a0 2023-11-24T08:04:50.605 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:50.606 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-24T08:04:50.606 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2023-11-24T08:04:50.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.606+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 msgr2=0x7f851c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.606+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 0x7f851c06d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f852c009a10 tx=0x7f852c008040 comp rx=0 tx=0).stop 2023-11-24T08:04:50.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.606+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 msgr2=0x7f853c139ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.606+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f853400a830 tx=0x7f8534001d80 comp rx=0 tx=0).stop 2023-11-24T08:04:50.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 shutdown_connections 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f851c06ae40 0x7f851c06d300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853c1407a0 0x7f853c139ff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f853c139b40 0x7f853c159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 --2- 172.21.15.107:0/3781378993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853c139760 0x7f853c0b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 >> 172.21.15.107:0/3781378993 conn(0x7f853c0cb830 msgr2=0x7f853c13c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 shutdown_connections 2023-11-24T08:04:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.607+0000 7f853b59e640 1 -- 172.21.15.107:0/3781378993 wait complete. 2023-11-24T08:04:50.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:50.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.832+0000 7fcb9523d640 1 Processor -- start 2023-11-24T08:04:50.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.833+0000 7fcb9523d640 1 -- start start 2023-11-24T08:04:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.834+0000 7fcb9523d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb900ce290 0x7fcb900ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.834+0000 7fcb9523d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.834+0000 7fcb9523d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9015bcb0 con 0x7fcb900abec0 2023-11-24T08:04:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.834+0000 7fcb9523d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb900ce670 con 0x7fcb900ce290 2023-11-24T08:04:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.834+0000 7fcb9523d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9015bf90 con 0x7fcb9006c4f0 2023-11-24T08:04:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb8ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb900ce290 0x7fcb900ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:49860/0 (socket says 172.21.15.107:49860) 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 -- 172.21.15.107:0/1139836494 learned_addr learned my addr 172.21.15.107:0/1139836494 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 -- 172.21.15.107:0/1139836494 >> v2:172.21.15.120:3300/0 conn(0x7fcb900ce290 msgr2=0x7fcb900ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 --2- 172.21.15.107:0/1139836494 >> v2:172.21.15.120:3300/0 conn(0x7fcb900ce290 0x7fcb900ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.835+0000 7fcb87fff640 1 -- 172.21.15.107:0/1139836494 >> v1:172.21.15.120:6789/0 conn(0x7fcb900abec0 legacy=0x7fcb90070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.836+0000 7fcb87fff640 1 -- 172.21.15.107:0/1139836494 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9015c270 con 0x7fcb9006c4f0 2023-11-24T08:04:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.837+0000 7fcb87fff640 1 --2- 172.21.15.107:0/1139836494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009870 tx=0x7fcb7802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14e50538bbe00490 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb8e59c640 1 -- 172.21.15.107:0/1139836494 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78030840 con 0x7fcb9006c4f0 2023-11-24T08:04:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb8e59c640 1 -- 172.21.15.107:0/1139836494 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb780309a0 con 0x7fcb9006c4f0 2023-11-24T08:04:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb8e59c640 1 -- 172.21.15.107:0/1139836494 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78030d30 con 0x7fcb9006c4f0 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 -- 172.21.15.107:0/1139836494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 msgr2=0x7fcb900ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 --2- 172.21.15.107:0/1139836494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009870 tx=0x7fcb7802b980 comp rx=0 tx=0).stop 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 -- 172.21.15.107:0/1139836494 shutdown_connections 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 --2- 172.21.15.107:0/1139836494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9006c4f0 0x7fcb900ae5d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 --2- 172.21.15.107:0/1139836494 >> v2:172.21.15.120:3300/0 conn(0x7fcb900ce290 0x7fcb900ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 -- 172.21.15.107:0/1139836494 >> 172.21.15.107:0/1139836494 conn(0x7fcb900b0be0 msgr2=0x7fcb900b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 -- 172.21.15.107:0/1139836494 shutdown_connections 2023-11-24T08:04:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.838+0000 7fcb9523d640 1 -- 172.21.15.107:0/1139836494 wait complete. 2023-11-24T08:04:50.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.840+0000 7fcb9523d640 1 Processor -- start 2023-11-24T08:04:50.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.840+0000 7fcb9523d640 1 -- start start 2023-11-24T08:04:50.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9006c4f0 0x7fcb900b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb900ce290 0x7fcb900b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9014ed00 con 0x7fcb900abec0 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9014efe0 con 0x7fcb9006c4f0 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb9523d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9014f2c0 con 0x7fcb900ce290 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57388/0 (socket says 172.21.15.107:57388) 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.841+0000 7fcb87fff640 1 -- 172.21.15.107:0/19949713 learned_addr learned my addr 172.21.15.107:0/19949713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb8ed9d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9006c4f0 0x7fcb900b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb8f59e640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb900ce290 0x7fcb900b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 -- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb900ce290 msgr2=0x7fcb900b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb900ce290 0x7fcb900b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 -- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9006c4f0 msgr2=0x7fcb900b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9006c4f0 0x7fcb900b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 -- 172.21.15.107:0/19949713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780094d0 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb87fff640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78039c90 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78039df0 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9014f5a0 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.842+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78043640 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.843+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9014b050 con 0x7fcb900abec0 2023-11-24T08:04:50.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.844+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9004e840 con 0x7fcb900abec0 2023-11-24T08:04:50.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.845+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcb780437a0 con 0x7fcb900abec0 2023-11-24T08:04:50.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.845+0000 7fcb877fe640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 0x7fcb7006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:50.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.846+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcb7803e070 con 0x7fcb900abec0 2023-11-24T08:04:50.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.846+0000 7fcb8ed9d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 0x7fcb7006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:50.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.846+0000 7fcb8ed9d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 0x7fcb7006d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009a30 tx=0x7fcb80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:50.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:50.850+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb780776e0 con 0x7fcb900abec0 2023-11-24T08:04:51.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.029+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb900b1340 con 0x7fcb900abec0 2023-11-24T08:04:51.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.031+0000 7fcb877fe640 1 -- 172.21.15.107:0/19949713 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fcb7807ad00 con 0x7fcb900abec0 2023-11-24T08:04:51.032 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:51.032 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:45.777870+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 msgr2=0x7fcb7006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 0x7fcb7006d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009a30 tx=0x7fcb80008040 comp rx=0 tx=0).stop 2023-11-24T08:04:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 msgr2=0x7fcb9014ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb78004730 comp rx=0 tx=0).stop 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 shutdown_connections 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006ae70 0x7fcb7006d330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb900ce290 0x7fcb900b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb900abec0 0x7fcb9014ab10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 --2- 172.21.15.107:0/19949713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9006c4f0 0x7fcb900b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 >> 172.21.15.107:0/19949713 conn(0x7fcb900b0be0 msgr2=0x7fcb900710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 shutdown_connections 2023-11-24T08:04:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.033+0000 7fcb9523d640 1 -- 172.21.15.107:0/19949713 wait complete. 2023-11-24T08:04:51.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:04:51.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:04:51.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.268+0000 7f3203778640 1 Processor -- start 2023-11-24T08:04:51.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.268+0000 7f3203778640 1 -- start start 2023-11-24T08:04:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.270+0000 7f3203778640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.270+0000 7f3203778640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.270+0000 7f3203778640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc15eea0 con 0x7f31fc15b440 2023-11-24T08:04:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.270+0000 7f3203778640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31fc15b820 con 0x7f31fc0c57e0 2023-11-24T08:04:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.270+0000 7f3203778640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0abf30 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3201d15640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3200d13640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3201d15640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:48854/0 (socket says 172.21.15.107:48854) 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3201d15640 1 -- 172.21.15.107:0/3523803808 learned_addr learned my addr 172.21.15.107:0/3523803808 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3200d13640 1 -- 172.21.15.107:0/3523803808 >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 msgr2=0x7f31fc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3200d13640 1 --2- 172.21.15.107:0/3523803808 >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3200d13640 1 -- 172.21.15.107:0/3523803808 >> v1:172.21.15.120:6789/0 conn(0x7f31fc15b440 legacy=0x7f31fc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3200d13640 1 -- 172.21.15.107:0/3523803808 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc15f0f0 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.271+0000 7f3201d15640 1 --2- 172.21.15.107:0/3523803808 >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.273+0000 7f3200d13640 1 --2- 172.21.15.107:0/3523803808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c52a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0097a0 tx=0x7f31ec02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a61811f6542bde8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.273+0000 7f31f3fff640 1 -- 172.21.15.107:0/3523803808 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec031780 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.273+0000 7f31f3fff640 1 -- 172.21.15.107:0/3523803808 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec0318e0 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.273+0000 7f31f3fff640 1 -- 172.21.15.107:0/3523803808 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec031c70 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 -- 172.21.15.107:0/3523803808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 msgr2=0x7f31fc0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 --2- 172.21.15.107:0/3523803808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c52a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0097a0 tx=0x7f31ec02b820 comp rx=0 tx=0).stop 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 -- 172.21.15.107:0/3523803808 shutdown_connections 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 --2- 172.21.15.107:0/3523803808 >> v2:172.21.15.120:3300/0 conn(0x7f31fc0c57e0 0x7f31fc06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 --2- 172.21.15.107:0/3523803808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c52a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 -- 172.21.15.107:0/3523803808 >> 172.21.15.107:0/3523803808 conn(0x7f31fc0b0c50 msgr2=0x7f31fc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 -- 172.21.15.107:0/3523803808 shutdown_connections 2023-11-24T08:04:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.274+0000 7f3203778640 1 -- 172.21.15.107:0/3523803808 wait complete. 2023-11-24T08:04:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.276+0000 7f3203778640 1 Processor -- start 2023-11-24T08:04:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.276+0000 7f3203778640 1 -- start start 2023-11-24T08:04:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31fc15b440 0x7f31fc1f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0b4a00 con 0x7f31fc0c57e0 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0c9060 con 0x7f31fc15b440 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.277+0000 7f3203778640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc09a1e0 con 0x7f31fc0cc0f0 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57390/0 (socket says 172.21.15.107:57390) 2023-11-24T08:04:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 -- 172.21.15.107:0/3593821874 learned_addr learned my addr 172.21.15.107:0/3593821874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201d15640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31fc15b440 0x7f31fc1f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3200d13640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 -- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 msgr2=0x7f31fc0c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 -- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31fc15b440 msgr2=0x7f31fc1f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31fc15b440 0x7f31fc1f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.278+0000 7f3201514640 1 -- 172.21.15.107:0/3593821874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f40094d0 con 0x7f31fc0c57e0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f3201514640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c760 tx=0x7f31f800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8006d50 con 0x7f31fc0c57e0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f800ce70 con 0x7f31fc0c57e0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec009400 con 0x7f31fc0c57e0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8003b90 con 0x7f31fc0c57e0 2023-11-24T08:04:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.279+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc0c80d0 con 0x7f31fc0c57e0 2023-11-24T08:04:51.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.280+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc04e8b0 con 0x7f31fc0c57e0 2023-11-24T08:04:51.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.281+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f31f8003cf0 con 0x7f31fc0c57e0 2023-11-24T08:04:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.282+0000 7f31f1ffb640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 0x7f31e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.282+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f31f8088d70 con 0x7f31fc0c57e0 2023-11-24T08:04:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.283+0000 7f3200d13640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 0x7f31e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.283+0000 7f3200d13640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 0x7f31e006d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31ec036000 tx=0x7f31ec03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.287+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31f8057c90 con 0x7f31fc0c57e0 2023-11-24T08:04:51.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.465+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31fc06dc80 con 0x7f31fc0c57e0 2023-11-24T08:04:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.469+0000 7f31f1ffb640 1 -- 172.21.15.107:0/3593821874 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31f8057820 con 0x7f31fc0c57e0 2023-11-24T08:04:51.470 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:04:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 msgr2=0x7f31e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 0x7f31e006d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31ec036000 tx=0x7f31ec03d040 comp rx=0 tx=0).stop 2023-11-24T08:04:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 msgr2=0x7f31fc0b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c760 tx=0x7f31f800cc20 comp rx=0 tx=0).stop 2023-11-24T08:04:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 shutdown_connections 2023-11-24T08:04:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e006ae40 0x7f31e006d300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31fc15b440 0x7f31fc1f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31fc0cc0f0 0x7f31fc0c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 --2- 172.21.15.107:0/3593821874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31fc0c57e0 0x7f31fc0b44c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.471+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 >> 172.21.15.107:0/3593821874 conn(0x7f31fc0b0c50 msgr2=0x7f31fc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.472+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 shutdown_connections 2023-11-24T08:04:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.472+0000 7f3203778640 1 -- 172.21.15.107:0/3593821874 wait complete. 2023-11-24T08:04:51.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:04:51.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.689+0000 7f3794c51640 1 Processor -- start 2023-11-24T08:04:51.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.689+0000 7f3794c51640 1 -- start start 2023-11-24T08:04:51.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.690+0000 7f3794c51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f379015c7f0 0x7f379006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.690+0000 7f3794c51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cf450 0x7f37900cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.690+0000 7f3794c51640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900cece0 con 0x7f37900cdfb0 2023-11-24T08:04:51.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.690+0000 7f3794c51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3790156010 con 0x7f379015c7f0 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.690+0000 7f3794c51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379006ce30 con 0x7f37900cf450 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f379015c7f0 0x7f379006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37900cdfb0 0x7f37900ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34840/0 (socket says 172.21.15.107:34840) 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378ed9d640 1 -- 172.21.15.107:0/1614506803 learned_addr learned my addr 172.21.15.107:0/1614506803 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f3787fff640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cf450 0x7f37900cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378dd9b640 1 -- 172.21.15.107:0/1614506803 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799135633 0 0) 0x7f37900cece0 con 0x7f37900cdfb0 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378e59c640 1 -- 172.21.15.107:0/1614506803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cf450 msgr2=0x7f37900cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378e59c640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cf450 0x7f37900cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378e59c640 1 -- 172.21.15.107:0/1614506803 >> v1:172.21.15.107:6789/0 conn(0x7f37900cdfb0 legacy=0x7f37900ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.691+0000 7f378e59c640 1 -- 172.21.15.107:0/1614506803 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379006d040 con 0x7f379015c7f0 2023-11-24T08:04:51.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.692+0000 7f378e59c640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015c7f0 0x7f379006db20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839b8f4e254ea916 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f378dd9b640 1 -- 172.21.15.107:0/1614506803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780031830 con 0x7f379015c7f0 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f378dd9b640 1 -- 172.21.15.107:0/1614506803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780031990 con 0x7f379015c7f0 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f378dd9b640 1 -- 172.21.15.107:0/1614506803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780031d20 con 0x7f379015c7f0 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 -- 172.21.15.107:0/1614506803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015c7f0 msgr2=0x7f379006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015c7f0 0x7f379006db20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 -- 172.21.15.107:0/1614506803 shutdown_connections 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cf450 0x7f37900cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 --2- 172.21.15.107:0/1614506803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015c7f0 0x7f379006db20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 -- 172.21.15.107:0/1614506803 >> 172.21.15.107:0/1614506803 conn(0x7f379006a920 msgr2=0x7f379006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 -- 172.21.15.107:0/1614506803 shutdown_connections 2023-11-24T08:04:51.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.693+0000 7f3794c51640 1 -- 172.21.15.107:0/1614506803 wait complete. 2023-11-24T08:04:51.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.695+0000 7f3794c51640 1 Processor -- start 2023-11-24T08:04:51.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.695+0000 7f3794c51640 1 -- start start 2023-11-24T08:04:51.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cdfb0 0x7f37900b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379015c7f0 0x7f379015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790156150 con 0x7f37900cf450 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790156430 con 0x7f379015c7f0 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3794c51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3790156710 con 0x7f37900cdfb0 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3787fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3787fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57410/0 (socket says 172.21.15.107:57410) 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.696+0000 7f3787fff640 1 -- 172.21.15.107:0/524026495 learned_addr learned my addr 172.21.15.107:0/524026495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f378e59c640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cdfb0 0x7f37900b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 -- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cdfb0 msgr2=0x7f37900b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cdfb0 0x7f37900b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 -- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379015c7f0 msgr2=0x7f379015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379015c7f0 0x7f379015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 -- 172.21.15.107:0/524026495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780009400 con 0x7f37900cf450 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3787fff640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f377800d800 tx=0x7f377800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377800b6c0 con 0x7f37900cf450 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377800b820 con 0x7f37900cf450 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377800bbb0 con 0x7f37900cf450 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.697+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37901580d0 con 0x7f37900cf450 2023-11-24T08:04:51.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.698+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37900b3030 con 0x7f37900cf450 2023-11-24T08:04:51.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.699+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790067470 con 0x7f37900cf450 2023-11-24T08:04:51.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.699+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f377800bd10 con 0x7f37900cf450 2023-11-24T08:04:51.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.701+0000 7f3786ffd640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 0x7f377006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:51.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.701+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3778089870 con 0x7f37900cf450 2023-11-24T08:04:51.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.701+0000 7f378e59c640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 0x7f377006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:51.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.702+0000 7f378e59c640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 0x7f377006d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3780036000 tx=0x7f378003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:51.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.705+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3778054c60 con 0x7f37900cf450 2023-11-24T08:04:51.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.899+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f37900cd730 con 0x7f37900cf450 2023-11-24T08:04:51.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.900+0000 7f3786ffd640 1 -- 172.21.15.107:0/524026495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3778058280 con 0x7f37900cf450 2023-11-24T08:04:51.901 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:51.901 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:04:51.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:04:51.902 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:04:51.902 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:04:51.902 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:04:51.902 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:04:51.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 msgr2=0x7f377006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 0x7f377006d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3780036000 tx=0x7f378003c040 comp rx=0 tx=0).stop 2023-11-24T08:04:51.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 msgr2=0x7f37901588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:51.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f377800d800 tx=0x7f377800dcc0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 shutdown_connections 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006ae40 0x7f377006d300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379015c7f0 0x7f379015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37900cf450 0x7f37901588d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 --2- 172.21.15.107:0/524026495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37900cdfb0 0x7f37900b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 >> 172.21.15.107:0/524026495 conn(0x7f379006a920 msgr2=0x7f37900cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 shutdown_connections 2023-11-24T08:04:51.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:51.903+0000 7f3794c51640 1 -- 172.21.15.107:0/524026495 wait complete. 2023-11-24T08:04:51.923 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:04:51.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:04:52.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.125+0000 7fbd48ead640 1 Processor -- start 2023-11-24T08:04:52.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.126+0000 7fbd48ead640 1 -- start start 2023-11-24T08:04:52.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd48ead640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd44160b00 0x7fbd440d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd48ead640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4415ea30 con 0x7fbd440d12e0 2023-11-24T08:04:52.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd48ead640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4415de20 con 0x7fbd440d02f0 2023-11-24T08:04:52.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd48ead640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4415e100 con 0x7fbd44160b00 2023-11-24T08:04:52.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd42ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd440d12e0 0x7fbd440cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34848/0 (socket says 172.21.15.107:34848) 2023-11-24T08:04:52.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd42ffd640 1 -- 172.21.15.107:0/577404993 learned_addr learned my addr 172.21.15.107:0/577404993 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:52.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.127+0000 7fbd437fe640 1 --2- 172.21.15.107:0/577404993 >> v2:172.21.15.107:3300/0 conn(0x7fbd44160b00 0x7fbd440d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.128+0000 7fbd427fc640 1 -- 172.21.15.107:0/577404993 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706240232 0 0) 0x7fbd4415ea30 con 0x7fbd440d12e0 2023-11-24T08:04:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.128+0000 7fbd427fc640 1 -- 172.21.15.107:0/577404993 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd30003520 con 0x7fbd440d12e0 2023-11-24T08:04:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.128+0000 7fbd437fe640 1 -- 172.21.15.107:0/577404993 >> v1:172.21.15.120:6789/0 conn(0x7fbd440d02f0 legacy=0x7fbd4415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.128+0000 7fbd437fe640 1 -- 172.21.15.107:0/577404993 >> v1:172.21.15.107:6789/0 conn(0x7fbd440d12e0 legacy=0x7fbd440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.128+0000 7fbd437fe640 1 -- 172.21.15.107:0/577404993 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4415e3e0 con 0x7fbd44160b00 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.129+0000 7fbd437fe640 1 --2- 172.21.15.107:0/577404993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd440d0da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009d90 tx=0x7fbd3402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f18df0e778b83f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.129+0000 7fbd427fc640 1 -- 172.21.15.107:0/577404993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3402cc60 con 0x7fbd44160b00 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.129+0000 7fbd427fc640 1 -- 172.21.15.107:0/577404993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3402cdc0 con 0x7fbd44160b00 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.129+0000 7fbd427fc640 1 -- 172.21.15.107:0/577404993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd34034660 con 0x7fbd44160b00 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 -- 172.21.15.107:0/577404993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 msgr2=0x7fbd440d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 --2- 172.21.15.107:0/577404993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd440d0da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009d90 tx=0x7fbd3402beb0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 -- 172.21.15.107:0/577404993 shutdown_connections 2023-11-24T08:04:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 --2- 172.21.15.107:0/577404993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd440d0da0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 -- 172.21.15.107:0/577404993 >> 172.21.15.107:0/577404993 conn(0x7fbd4409c510 msgr2=0x7fbd440d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 -- 172.21.15.107:0/577404993 shutdown_connections 2023-11-24T08:04:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.130+0000 7fbd48ead640 1 -- 172.21.15.107:0/577404993 wait complete. 2023-11-24T08:04:52.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.131+0000 7fbd48ead640 1 Processor -- start 2023-11-24T08:04:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.132+0000 7fbd48ead640 1 -- start start 2023-11-24T08:04:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.132+0000 7fbd48ead640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd440d02f0 0x7fbd4414be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.132+0000 7fbd48ead640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 0x7fbd440b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd48ead640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd48ead640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440b8260 con 0x7fbd44160b00 2023-11-24T08:04:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd48ead640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4414b670 con 0x7fbd440d12e0 2023-11-24T08:04:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd48ead640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440b9060 con 0x7fbd440d02f0 2023-11-24T08:04:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57430/0 (socket says 172.21.15.107:57430) 2023-11-24T08:04:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 -- 172.21.15.107:0/2709212250 learned_addr learned my addr 172.21.15.107:0/2709212250 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd42ffd640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 0x7fbd440b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 -- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd440d02f0 msgr2=0x7fbd4414be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd440d02f0 0x7fbd4414be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 -- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 msgr2=0x7fbd440b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 0x7fbd440b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 -- 172.21.15.107:0/2709212250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34009a70 con 0x7fbd44160b00 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd42ffd640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 0x7fbd440b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.133+0000 7fbd43fff640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00c410 tx=0x7fbd3c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.134+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c011790 con 0x7fbd44160b00 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.134+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c0118f0 con 0x7fbd44160b00 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.134+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c011ce0 con 0x7fbd44160b00 2023-11-24T08:04:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.134+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd440b84e0 con 0x7fbd44160b00 2023-11-24T08:04:52.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.134+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd440b8aa0 con 0x7fbd44160b00 2023-11-24T08:04:52.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.135+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd441492c0 con 0x7fbd44160b00 2023-11-24T08:04:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.136+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbd3c01c460 con 0x7fbd44160b00 2023-11-24T08:04:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.137+0000 7fbd3bfff640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 0x7fbd2406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.137+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbd3c089070 con 0x7fbd44160b00 2023-11-24T08:04:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.137+0000 7fbd437fe640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 0x7fbd2406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.138+0000 7fbd437fe640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 0x7fbd2406d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbd34036000 tx=0x7fbd3402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.141+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd3c0545c0 con 0x7fbd44160b00 2023-11-24T08:04:52.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.315+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbd4404e840 con 0x7fbd44160b00 2023-11-24T08:04:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.480+0000 7fbd3bfff640 1 -- 172.21.15.107:0/2709212250 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbd3c057be0 con 0x7fbd44160b00 2023-11-24T08:04:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:04:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 msgr2=0x7fbd2406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 0x7fbd2406d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbd34036000 tx=0x7fbd3402e000 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 msgr2=0x7fbd4414b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00c410 tx=0x7fbd3c00c810 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 shutdown_connections 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd2406ae70 0x7fbd2406d330 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd44160b00 0x7fbd4414b130 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd440d12e0 0x7fbd440b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 --2- 172.21.15.107:0/2709212250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd440d02f0 0x7fbd4414be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.483+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 >> 172.21.15.107:0/2709212250 conn(0x7fbd4409c510 msgr2=0x7fbd440d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.484+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 shutdown_connections 2023-11-24T08:04:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.484+0000 7fbd48ead640 1 -- 172.21.15.107:0/2709212250 wait complete. 2023-11-24T08:04:52.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:04:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.709+0000 7f7890eee640 1 Processor -- start 2023-11-24T08:04:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.709+0000 7f7890eee640 1 -- start start 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f788c140520 0x7f788c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f788c161940 0x7f788c142d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0c6a70 con 0x7f788c161940 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c15de00 con 0x7f788c140520 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f7890eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c15e0e0 con 0x7f788c0c65f0 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.711+0000 7f788ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f788c161940 0x7f788c142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f788c161940 0x7f788c142d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57446/0 (socket says 172.21.15.107:57446) 2023-11-24T08:04:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 -- 172.21.15.107:0/315851033 learned_addr learned my addr 172.21.15.107:0/315851033 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788a59c640 1 --2- 172.21.15.107:0/315851033 >> v2:172.21.15.120:3300/0 conn(0x7f788c140520 0x7f788c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f7883fff640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 -- 172.21.15.107:0/315851033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 msgr2=0x7f788c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 -- 172.21.15.107:0/315851033 >> v2:172.21.15.120:3300/0 conn(0x7f788c140520 msgr2=0x7f788c0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 --2- 172.21.15.107:0/315851033 >> v2:172.21.15.120:3300/0 conn(0x7f788c140520 0x7f788c0c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f788ad9d640 1 -- 172.21.15.107:0/315851033 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c15e3c0 con 0x7f788c161940 2023-11-24T08:04:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.712+0000 7f7883fff640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:52.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.713+0000 7f788ad9d640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c161940 0x7f788c142d70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f788c06e0b0 tx=0x7f788402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f5848b1dcbc647 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.713+0000 7f7889d9b640 1 -- 172.21.15.107:0/315851033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788402ca70 con 0x7f788c161940 2023-11-24T08:04:52.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.713+0000 7f7889d9b640 1 -- 172.21.15.107:0/315851033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788402cbd0 con 0x7f788c161940 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.713+0000 7f7889d9b640 1 -- 172.21.15.107:0/315851033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78840343f0 con 0x7f788c161940 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 -- 172.21.15.107:0/315851033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c161940 msgr2=0x7f788c142d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c161940 0x7f788c142d70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f788c06e0b0 tx=0x7f788402bed0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 -- 172.21.15.107:0/315851033 shutdown_connections 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c161940 0x7f788c142d70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 --2- 172.21.15.107:0/315851033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 --2- 172.21.15.107:0/315851033 >> v2:172.21.15.120:3300/0 conn(0x7f788c140520 0x7f788c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 -- 172.21.15.107:0/315851033 >> 172.21.15.107:0/315851033 conn(0x7f788c0b2e40 msgr2=0x7f788c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:52.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 -- 172.21.15.107:0/315851033 shutdown_connections 2023-11-24T08:04:52.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.714+0000 7f7890eee640 1 -- 172.21.15.107:0/315851033 wait complete. 2023-11-24T08:04:52.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.716+0000 7f7890eee640 1 Processor -- start 2023-11-24T08:04:52.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.716+0000 7f7890eee640 1 -- start start 2023-11-24T08:04:52.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c15ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f788c161940 0x7f788c0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cfed0 con 0x7f788c140520 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cf5c0 con 0x7f788c161940 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7890eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c160ba0 con 0x7f788c0c65f0 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7883fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7883fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57452/0 (socket says 172.21.15.107:57452) 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f7883fff640 1 -- 172.21.15.107:0/3721009094 learned_addr learned my addr 172.21.15.107:0/3721009094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f788a59c640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c15ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.717+0000 7f788ad9d640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f788c161940 0x7f788c0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 -- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 msgr2=0x7f788c15ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c15ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 -- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f788c161940 msgr2=0x7f788c0c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f788c161940 0x7f788c0c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 -- 172.21.15.107:0/3721009094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884009b30 con 0x7f788c140520 2023-11-24T08:04:52.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7883fff640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f787400c7f0 tx=0x7f787400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874007820 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7874007980 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874007cf0 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c15f250 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.718+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c0cfaf0 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.719+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c0cfc80 con 0x7f788c140520 2023-11-24T08:04:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.721+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7874024480 con 0x7f788c140520 2023-11-24T08:04:52.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.722+0000 7f7882ffd640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 0x7f786c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:52.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.722+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7874089c10 con 0x7f788c140520 2023-11-24T08:04:52.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.722+0000 7f788a59c640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 0x7f786c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:52.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.723+0000 7f788a59c640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 0x7f786c06d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f787c006fd0 tx=0x7f787c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.725+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7874054e80 con 0x7f788c140520 2023-11-24T08:04:52.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:52.917+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f788c0d0010 con 0x7f788c140520 2023-11-24T08:04:53.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.486+0000 7f7882ffd640 1 -- 172.21.15.107:0/3721009094 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f787401f0d0 con 0x7f788c140520 2023-11-24T08:04:53.488 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:04:53.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 msgr2=0x7f786c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 0x7f786c06d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f787c006fd0 tx=0x7f787c008040 comp rx=0 tx=0).stop 2023-11-24T08:04:53.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 msgr2=0x7f788c153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f787400c7f0 tx=0x7f787400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 shutdown_connections 2023-11-24T08:04:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f786c06ae40 0x7f786c06d300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f788c161940 0x7f788c0c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f788c140520 0x7f788c153720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 --2- 172.21.15.107:0/3721009094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788c0c65f0 0x7f788c15ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.491+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 >> 172.21.15.107:0/3721009094 conn(0x7f788c0b2e40 msgr2=0x7f788c0705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.493+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 shutdown_connections 2023-11-24T08:04:53.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.493+0000 7f7890eee640 1 -- 172.21.15.107:0/3721009094 wait complete. 2023-11-24T08:04:53.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:04:53.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.713+0000 7f03d4133640 1 Processor -- start 2023-11-24T08:04:53.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.713+0000 7f03d4133640 1 -- start start 2023-11-24T08:04:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d4133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d4133640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 0x7f03cc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d4133640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0cc3a0 con 0x7f03cc0cbf20 2023-11-24T08:04:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d4133640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc15b820 con 0x7f03cc0c59b0 2023-11-24T08:04:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d4133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc15eea0 con 0x7f03cc15b440 2023-11-24T08:04:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.715+0000 7f03d1ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:49944/0 (socket says 172.21.15.107:49944) 2023-11-24T08:04:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 -- 172.21.15.107:0/3515765720 learned_addr learned my addr 172.21.15.107:0/3515765720 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d26d0640 1 --2- 172.21.15.107:0/3515765720 >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 0x7f03cc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 -- 172.21.15.107:0/3515765720 >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 msgr2=0x7f03cc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 --2- 172.21.15.107:0/3515765720 >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 0x7f03cc06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 -- 172.21.15.107:0/3515765720 >> v1:172.21.15.120:6789/0 conn(0x7f03cc0cbf20 legacy=0x7f03cc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.716+0000 7f03d1ecf640 1 -- 172.21.15.107:0/3515765720 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0abf30 con 0x7f03cc15b440 2023-11-24T08:04:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.717+0000 7f03d26d0640 1 --2- 172.21.15.107:0/3515765720 >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 0x7f03cc06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:04:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.718+0000 7f03d1ecf640 1 --2- 172.21.15.107:0/3515765720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009870 tx=0x7f03c802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a978aa4dbc131af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.718+0000 7f03d0ecd640 1 -- 172.21.15.107:0/3515765720 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8031640 con 0x7f03cc15b440 2023-11-24T08:04:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.719+0000 7f03d0ecd640 1 -- 172.21.15.107:0/3515765720 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c80317a0 con 0x7f03cc15b440 2023-11-24T08:04:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.719+0000 7f03d0ecd640 1 -- 172.21.15.107:0/3515765720 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8031b30 con 0x7f03cc15b440 2023-11-24T08:04:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.719+0000 7f03d4133640 1 -- 172.21.15.107:0/3515765720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 msgr2=0x7f03cc0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.719+0000 7f03d4133640 1 --2- 172.21.15.107:0/3515765720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009870 tx=0x7f03c802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 -- 172.21.15.107:0/3515765720 shutdown_connections 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 --2- 172.21.15.107:0/3515765720 >> v2:172.21.15.120:3300/0 conn(0x7f03cc0c59b0 0x7f03cc06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 --2- 172.21.15.107:0/3515765720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc15b440 0x7f03cc0cb9e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 -- 172.21.15.107:0/3515765720 >> 172.21.15.107:0/3515765720 conn(0x7f03cc0b0c50 msgr2=0x7f03cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 -- 172.21.15.107:0/3515765720 shutdown_connections 2023-11-24T08:04:53.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.720+0000 7f03d4133640 1 -- 172.21.15.107:0/3515765720 wait complete. 2023-11-24T08:04:53.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.722+0000 7f03d4133640 1 Processor -- start 2023-11-24T08:04:53.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.722+0000 7f03d4133640 1 -- start start 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc0c59b0 0x7f03cc156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0cbf20 0x7f03cc09c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0b2d60 con 0x7f03cc15b440 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc1f53d0 con 0x7f03cc0cbf20 2023-11-24T08:04:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d4133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc15c2f0 con 0x7f03cc0c59b0 2023-11-24T08:04:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d26d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d26d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57466/0 (socket says 172.21.15.107:57466) 2023-11-24T08:04:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d26d0640 1 -- 172.21.15.107:0/239750080 learned_addr learned my addr 172.21.15.107:0/239750080 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d1ecf640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc0c59b0 0x7f03cc156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.723+0000 7f03d16ce640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0cbf20 0x7f03cc09c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 -- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc0c59b0 msgr2=0x7f03cc156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc0c59b0 0x7f03cc156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 -- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0cbf20 msgr2=0x7f03cc09c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0cbf20 0x7f03cc09c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 -- 172.21.15.107:0/239750080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4009a70 con 0x7f03cc15b440 2023-11-24T08:04:53.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03d26d0640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009f00 tx=0x7f03c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.724+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4009d70 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.725+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c400f040 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.725+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4003bb0 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.725+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80094d0 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.725+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc1f4e10 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.726+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc04e8b0 con 0x7f03cc15b440 2023-11-24T08:04:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.727+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f03c4014480 con 0x7f03cc15b440 2023-11-24T08:04:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.728+0000 7f03c27fc640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 0x7f03b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.728+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f03c408a120 con 0x7f03cc15b440 2023-11-24T08:04:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.728+0000 7f03d1ecf640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 0x7f03b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:53.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.729+0000 7f03d1ecf640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 0x7f03b006d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03c8036000 tx=0x7f03c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:53.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.733+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03c40552d0 con 0x7f03cc15b440 2023-11-24T08:04:53.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:53.953+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f03cc15c430 con 0x7f03cc15b440 2023-11-24T08:04:54.538 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 2 and data pool 3 2023-11-24T08:04:54.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.533+0000 7f03c27fc640 1 -- 172.21.15.107:0/239750080 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f03c40588f0 con 0x7f03cc15b440 2023-11-24T08:04:54.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.541+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 msgr2=0x7f03b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.541+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 0x7f03b006d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03c8036000 tx=0x7f03c803c040 comp rx=0 tx=0).stop 2023-11-24T08:04:54.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.541+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 msgr2=0x7f03cc1f4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.541+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009f00 tx=0x7f03c400c490 comp rx=0 tx=0).stop 2023-11-24T08:04:54.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 shutdown_connections 2023-11-24T08:04:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006ae40 0x7f03b006d300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc15b440 0x7f03cc1f4450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0cbf20 0x7f03cc09c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 --2- 172.21.15.107:0/239750080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc0c59b0 0x7f03cc156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.542+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 >> 172.21.15.107:0/239750080 conn(0x7f03cc0b0c50 msgr2=0x7f03cc0c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:54.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.543+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 shutdown_connections 2023-11-24T08:04:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.543+0000 7f03d4133640 1 -- 172.21.15.107:0/239750080 wait complete. 2023-11-24T08:04:54.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:54.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.753+0000 7ff6e5f73640 1 Processor -- start 2023-11-24T08:04:54.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.753+0000 7ff6e5f73640 1 -- start start 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5f73640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6e006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5f73640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006e060 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5f73640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e013f9e0 con 0x7ff6e013d120 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5f73640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e013fcc0 con 0x7ff6e006d6c0 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5772640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff6e00cdbf0 0x7ff6e00ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34862/0 (socket says 172.21.15.107:34862) 2023-11-24T08:04:54.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6e5772640 1 -- 172.21.15.107:0/1254808221 learned_addr learned my addr 172.21.15.107:0/1254808221 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:54.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6dffff640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6e006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.755+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989578023 0 0) 0x7ff6e006e060 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d0005730 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953100724 0 0) 0x7ff6d0005730 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 msgr2=0x7ff6e006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6e006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 >> v1:172.21.15.120:6789/0 conn(0x7ff6e013d120 legacy=0x7ff6e015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e013ffa0 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff6d80019f0 con 0x7ff6e00cdbf0 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 >> v1:172.21.15.107:6789/0 conn(0x7ff6e00cdbf0 legacy=0x7ff6e00ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d0005bb0 0x7ff6d0005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6d00069d0 0x7ff6d0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6d00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e013fcc0 con 0x7ff6d00069d0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d000dcc0 con 0x7ff6d0005bb0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d000de80 con 0x7ff6e006d6c0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.756+0000 7ff6dffff640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6d00069d0 0x7ff6d0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6e5772640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6d00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6dffff640 1 -- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 msgr2=0x7ff6d00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6dffff640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e006d6c0 0x7ff6d00073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6e4f71640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d0005bb0 0x7ff6d0005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6dffff640 1 -- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d0005bb0 msgr2=0x7ff6d0005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6dffff640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d0005bb0 0x7ff6d0005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6dffff640 1 -- 172.21.15.107:0/1254808221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0005730 con 0x7ff6d00069d0 2023-11-24T08:04:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.757+0000 7ff6e4f71640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d0005bb0 0x7ff6d0005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:54.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.758+0000 7ff6dffff640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6d00069d0 0x7ff6d0006e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009ac0 tx=0x7ff6d0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db388d5eceb3ebd1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.758+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc02c6b0 con 0x7ff6d00069d0 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.758+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc02c810 con 0x7ff6d00069d0 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.758+0000 7ff6df7fe640 1 -- 172.21.15.107:0/1254808221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc02cba0 con 0x7ff6d00069d0 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 -- 172.21.15.107:0/1254808221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6d00069d0 msgr2=0x7ff6d0006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/1254808221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6d00069d0 0x7ff6d0006e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009ac0 tx=0x7ff6d0007a00 comp rx=0 tx=0).stop 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5772640 1 -- 172.21.15.107:0/1254808221 reap_dead start 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 -- 172.21.15.107:0/1254808221 shutdown_connections 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 -- 172.21.15.107:0/1254808221 >> 172.21.15.107:0/1254808221 conn(0x7ff6e00cb830 msgr2=0x7ff6e00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 -- 172.21.15.107:0/1254808221 shutdown_connections 2023-11-24T08:04:54.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.759+0000 7ff6e5f73640 1 -- 172.21.15.107:0/1254808221 wait complete. 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.760+0000 7ff6e5f73640 1 Processor -- start 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.761+0000 7ff6e5f73640 1 -- start start 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.761+0000 7ff6e5f73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d00069d0 0x7ff6e013d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5f73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5f73640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 0x7ff6e00af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5f73640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e00b3e40 con 0x7ff6e00b4e50 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5f73640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e012eb60 con 0x7ff6d00069d0 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5f73640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e00b4090 con 0x7ff6e012e6e0 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57482/0 (socket says 172.21.15.107:57482) 2023-11-24T08:04:54.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 -- 172.21.15.107:0/3983293324 learned_addr learned my addr 172.21.15.107:0/3983293324 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e4f71640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d00069d0 0x7ff6e013d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 -- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 msgr2=0x7ff6e00af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5772640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 0x7ff6e00af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 0x7ff6e00af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 -- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d00069d0 msgr2=0x7ff6e013d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d00069d0 0x7ff6e013d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6dffff640 1 -- 172.21.15.107:0/3983293324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e015a9d0 con 0x7ff6e00b4e50 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.762+0000 7ff6e5772640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 0x7ff6e00af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6dffff640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc02c250 tx=0x7ff6cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003810 con 0x7ff6e00b4e50 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc003970 con 0x7ff6e00b4e50 2023-11-24T08:04:54.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003ce0 con 0x7ff6e00b4e50 2023-11-24T08:04:54.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e00b4370 con 0x7ff6e00b4e50 2023-11-24T08:04:54.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.763+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e00b4930 con 0x7ff6e00b4e50 2023-11-24T08:04:54.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.764+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e00b4ac0 con 0x7ff6e00b4e50 2023-11-24T08:04:54.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.765+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff6cc034440 con 0x7ff6e00b4e50 2023-11-24T08:04:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.766+0000 7ff6dd7fa640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 0x7ff6c406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.767+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6cc0ab9f0 con 0x7ff6e00b4e50 2023-11-24T08:04:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.767+0000 7ff6e4f71640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 0x7ff6c406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.767+0000 7ff6e4f71640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 0x7ff6c406d300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001750 tx=0x7ff6d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:54.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.771+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6cc076a20 con 0x7ff6e00b4e50 2023-11-24T08:04:54.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.954+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e004e840 con 0x7ff6e00b4e50 2023-11-24T08:04:54.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.956+0000 7ff6dd7fa640 1 -- 172.21.15.107:0/3983293324 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7ff6cc039090 con 0x7ff6e00b4e50 2023-11-24T08:04:54.959 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:54.959 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:53.954036+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 msgr2=0x7ff6c406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 0x7ff6c406d300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001750 tx=0x7ff6d4009040 comp rx=0 tx=0).stop 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 msgr2=0x7ff6e012e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc02c250 tx=0x7ff6cc004730 comp rx=0 tx=0).stop 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 shutdown_connections 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff6c406ae40 0x7ff6c406d300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e012e6e0 0x7ff6e00af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00b4e50 0x7ff6e012e1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 --2- 172.21.15.107:0/3983293324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d00069d0 0x7ff6e013d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 >> 172.21.15.107:0/3983293324 conn(0x7ff6e00cb830 msgr2=0x7ff6e013ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 shutdown_connections 2023-11-24T08:04:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:54.959+0000 7ff6e5f73640 1 -- 172.21.15.107:0/3983293324 wait complete. 2023-11-24T08:04:54.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.177+0000 7f7ca9d1d640 1 Processor -- start 2023-11-24T08:04:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.177+0000 7f7ca9d1d640 1 -- start start 2023-11-24T08:04:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.178+0000 7f7ca9d1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ca40b2250 0x7f7ca415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca9d1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7ca40b2aa0 0x7f7ca415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca9d1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca415ab80 con 0x7f7ca40b2630 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca9d1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca415a990 con 0x7f7ca40b2250 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca9d1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca415e680 con 0x7f7ca40b2aa0 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ca40b2250 0x7f7ca415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ca40b2250 0x7f7ca415df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57490/0 (socket says 172.21.15.107:57490) 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 -- 172.21.15.107:0/4126972851 learned_addr learned my addr 172.21.15.107:0/4126972851 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca3fff640 1 --2- 172.21.15.107:0/4126972851 >> v2:172.21.15.120:3300/0 conn(0x7f7ca40b2aa0 0x7f7ca415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 -- 172.21.15.107:0/4126972851 >> v2:172.21.15.120:3300/0 conn(0x7f7ca40b2aa0 msgr2=0x7f7ca415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 --2- 172.21.15.107:0/4126972851 >> v2:172.21.15.120:3300/0 conn(0x7f7ca40b2aa0 0x7f7ca415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 -- 172.21.15.107:0/4126972851 >> v1:172.21.15.120:6789/0 conn(0x7f7ca40b2630 legacy=0x7f7ca40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.179+0000 7f7ca37fe640 1 -- 172.21.15.107:0/4126972851 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca415e960 con 0x7f7ca40b2250 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.180+0000 7f7ca37fe640 1 --2- 172.21.15.107:0/4126972851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca415df80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40b2fa0 tx=0x7f7c9402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b6e13d10220bdb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca27fc640 1 -- 172.21.15.107:0/4126972851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9402ee90 con 0x7f7ca40b2250 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca27fc640 1 -- 172.21.15.107:0/4126972851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94036440 con 0x7f7ca40b2250 2023-11-24T08:04:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca27fc640 1 -- 172.21.15.107:0/4126972851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c940367d0 con 0x7f7ca40b2250 2023-11-24T08:04:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/4126972851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 msgr2=0x7f7ca415df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/4126972851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca415df80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40b2fa0 tx=0x7f7c9402e0e0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/4126972851 shutdown_connections 2023-11-24T08:04:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/4126972851 >> v2:172.21.15.120:3300/0 conn(0x7f7ca40b2aa0 0x7f7ca415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/4126972851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca415df80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/4126972851 >> 172.21.15.107:0/4126972851 conn(0x7f7ca406a920 msgr2=0x7f7ca406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/4126972851 shutdown_connections 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.181+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/4126972851 wait complete. 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.183+0000 7f7ca9d1d640 1 Processor -- start 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.183+0000 7f7ca9d1d640 1 -- start start 2023-11-24T08:04:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca40b2630 0x7f7ca40ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca40b2aa0 0x7f7ca4155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca40b1360 con 0x7f7ca40b2250 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca415df10 con 0x7f7ca40b2aa0 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca9d1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca415e1d0 con 0x7f7ca40b2630 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57500/0 (socket says 172.21.15.107:57500) 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.184+0000 7f7ca37fe640 1 -- 172.21.15.107:0/3131785600 learned_addr learned my addr 172.21.15.107:0/3131785600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 -- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca40b2630 msgr2=0x7f7ca40ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca40b2630 0x7f7ca40ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 -- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca40b2aa0 msgr2=0x7f7ca4155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca3fff640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca40b2aa0 0x7f7ca4155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca40b2aa0 0x7f7ca4155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 -- 172.21.15.107:0/3131785600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9400b770 con 0x7f7ca40b2250 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca37fe640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7c9402e120 tx=0x7f7c940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c940348d0 con 0x7f7ca40b2250 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94034a50 con 0x7f7ca40b2250 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca40cd7c0 con 0x7f7ca40b2250 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94034e00 con 0x7f7ca40b2250 2023-11-24T08:04:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.185+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca40b18e0 con 0x7f7ca40b2250 2023-11-24T08:04:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.186+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca4067470 con 0x7f7ca40b2250 2023-11-24T08:04:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.187+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7c940403e0 con 0x7f7ca40b2250 2023-11-24T08:04:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.188+0000 7f7c8bfff640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 0x7f7c9006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.188+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c940ab940 con 0x7f7ca40b2250 2023-11-24T08:04:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.188+0000 7f7ca2ffd640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 0x7f7c9006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.189+0000 7f7ca2ffd640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 0x7f7c9006d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ca409d310 tx=0x7f7c980079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.193+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c94076950 con 0x7f7ca40b2250 2023-11-24T08:04:55.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.369+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ca40cb460 con 0x7f7ca40b2250 2023-11-24T08:04:55.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.370+0000 7f7c8bfff640 1 -- 172.21.15.107:0/3131785600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f7c9403b090 con 0x7f7ca40b2250 2023-11-24T08:04:55.373 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:53.954036+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 msgr2=0x7f7c9006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 0x7f7c9006d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ca409d310 tx=0x7f7c980079b0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 msgr2=0x7f7ca41541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7c9402e120 tx=0x7f7c940046c0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 shutdown_connections 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006ae70 0x7f7c9006d330 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca40b2aa0 0x7f7ca4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca40b2630 0x7f7ca40ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 --2- 172.21.15.107:0/3131785600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ca40b2250 0x7f7ca41541d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.373+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 >> 172.21.15.107:0/3131785600 conn(0x7f7ca406a920 msgr2=0x7f7ca40cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:55.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.374+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 shutdown_connections 2023-11-24T08:04:55.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.374+0000 7f7ca9d1d640 1 -- 172.21.15.107:0/3131785600 wait complete. 2023-11-24T08:04:55.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:04:55.468 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:04:55.467+0000 7f3df411b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:04:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.599+0000 7f06629c9640 1 Processor -- start 2023-11-24T08:04:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.599+0000 7f06629c9640 1 -- start start 2023-11-24T08:04:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.600+0000 7f06629c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f065c15b440 0x7f065c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.600+0000 7f06629c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.600+0000 7f06629c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c15eea0 con 0x7f065c0c57e0 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.600+0000 7f06629c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0cc3d0 con 0x7f065c15b440 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.600+0000 7f06629c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c15b820 con 0x7f065c0cbf20 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f065c15b440 0x7f065c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f0660f66640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f065c0c57e0 0x7f065c0c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34870/0 (socket says 172.21.15.107:34870) 2023-11-24T08:04:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f0660f66640 1 -- 172.21.15.107:0/2768478703 learned_addr learned my addr 172.21.15.107:0/2768478703 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065bfff640 1 -- 172.21.15.107:0/2768478703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 msgr2=0x7f065c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065bfff640 1 --2- 172.21.15.107:0/2768478703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065bfff640 1 -- 172.21.15.107:0/2768478703 >> v1:172.21.15.107:6789/0 conn(0x7f065c0c57e0 legacy=0x7f065c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065bfff640 1 -- 172.21.15.107:0/2768478703 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c06d600 con 0x7f065c15b440 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.601+0000 7f065b7fe640 1 -- 172.21.15.107:0/2768478703 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5270611 0 0) 0x7f065c15eea0 con 0x7f065c0c57e0 2023-11-24T08:04:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.602+0000 7f065bfff640 1 --2- 172.21.15.107:0/2768478703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0cb9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097a0 tx=0x7f064c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7267af7ba0c6a9cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.602+0000 7f065b7fe640 1 -- 172.21.15.107:0/2768478703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c02e070 con 0x7f065c15b440 2023-11-24T08:04:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f065b7fe640 1 -- 172.21.15.107:0/2768478703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c0315a0 con 0x7f065c15b440 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f065b7fe640 1 -- 172.21.15.107:0/2768478703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c031930 con 0x7f065c15b440 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 -- 172.21.15.107:0/2768478703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 msgr2=0x7f065c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 --2- 172.21.15.107:0/2768478703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0cb9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097a0 tx=0x7f064c02b840 comp rx=0 tx=0).stop 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 -- 172.21.15.107:0/2768478703 shutdown_connections 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 --2- 172.21.15.107:0/2768478703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 --2- 172.21.15.107:0/2768478703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0cb9e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 -- 172.21.15.107:0/2768478703 >> 172.21.15.107:0/2768478703 conn(0x7f065c0b0c50 msgr2=0x7f065c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 -- 172.21.15.107:0/2768478703 shutdown_connections 2023-11-24T08:04:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.603+0000 7f06629c9640 1 -- 172.21.15.107:0/2768478703 wait complete. 2023-11-24T08:04:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.605+0000 7f06629c9640 1 Processor -- start 2023-11-24T08:04:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.605+0000 7f06629c9640 1 -- start start 2023-11-24T08:04:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f065c0c57e0 0x7f065c0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1574a0 con 0x7f065c15b440 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0c6360 con 0x7f065c0c57e0 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f06629c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0b31f0 con 0x7f065c0cbf20 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f0660f66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f0660f66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57510/0 (socket says 172.21.15.107:57510) 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.606+0000 7f0660f66640 1 -- 172.21.15.107:0/602836347 learned_addr learned my addr 172.21.15.107:0/602836347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f065bfff640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f065c0c57e0 0x7f065c0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0653fff640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 -- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 msgr2=0x7f065c156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 -- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f065c0c57e0 msgr2=0x7f065c0b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f065c0c57e0 0x7f065c0b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 -- 172.21.15.107:0/602836347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c009400 con 0x7f065c15b440 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0653fff640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f0660f66640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f065400c7e0 tx=0x7f065400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400a640 con 0x7f065c15b440 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400a7c0 con 0x7f065c15b440 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c0b34d0 con 0x7f065c15b440 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400ab70 con 0x7f065c15b440 2023-11-24T08:04:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.607+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c0b3a90 con 0x7f065c15b440 2023-11-24T08:04:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.608+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c04e8b0 con 0x7f065c15b440 2023-11-24T08:04:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.609+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f065400acd0 con 0x7f065c15b440 2023-11-24T08:04:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.610+0000 7f06597fa640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 0x7f063c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.610+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0654088c40 con 0x7f065c15b440 2023-11-24T08:04:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.611+0000 7f065bfff640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 0x7f063c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.611+0000 7f065bfff640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 0x7f063c06d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f064c036000 tx=0x7f064c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.615+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0654053ce0 con 0x7f065c15b440 2023-11-24T08:04:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:55.828+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f065c06dc80 con 0x7f065c15b440 2023-11-24T08:04:56.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.718+0000 7f06597fa640 1 -- 172.21.15.107:0/602836347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0654018070 con 0x7f065c15b440 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 msgr2=0x7f063c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 0x7f063c06d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f064c036000 tx=0x7f064c03d040 comp rx=0 tx=0).stop 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 msgr2=0x7f065c0c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f065400c7e0 tx=0x7f065400cca0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 shutdown_connections 2023-11-24T08:04:56.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f063c06ae40 0x7f063c06d300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f065c15b440 0x7f065c0c8860 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f065c0cbf20 0x7f065c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 --2- 172.21.15.107:0/602836347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f065c0c57e0 0x7f065c0b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 >> 172.21.15.107:0/602836347 conn(0x7f065c0b0c50 msgr2=0x7f065c0cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 shutdown_connections 2023-11-24T08:04:56.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.723+0000 7f06629c9640 1 -- 172.21.15.107:0/602836347 wait complete. 2023-11-24T08:04:56.747 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:04:56.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:04:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.967+0000 7ff00f1dc640 1 Processor -- start 2023-11-24T08:04:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.967+0000 7ff00f1dc640 1 -- start start 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.968+0000 7ff00f1dc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff00815b440 0x7ff0080cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.968+0000 7ff00f1dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0080cbf20 0x7ff0080c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff00f1dc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00815eea0 con 0x7ff0080c57e0 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff00f1dc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0080cc3d0 con 0x7ff0080cbf20 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff00f1dc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00815b820 con 0x7ff00815b440 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff007fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0080cbf20 0x7ff0080c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff00d779640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff0080c57e0 0x7ff0080c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34878/0 (socket says 172.21.15.107:34878) 2023-11-24T08:04:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff00d779640 1 -- 172.21.15.107:0/2444131258 learned_addr learned my addr 172.21.15.107:0/2444131258 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff007fff640 1 -- 172.21.15.107:0/2444131258 >> v2:172.21.15.120:3300/0 conn(0x7ff00815b440 msgr2=0x7ff0080cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff007fff640 1 --2- 172.21.15.107:0/2444131258 >> v2:172.21.15.120:3300/0 conn(0x7ff00815b440 0x7ff0080cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff007fff640 1 -- 172.21.15.107:0/2444131258 >> v1:172.21.15.107:6789/0 conn(0x7ff0080c57e0 legacy=0x7ff0080c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.969+0000 7ff007fff640 1 -- 172.21.15.107:0/2444131258 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00806d600 con 0x7ff0080cbf20 2023-11-24T08:04:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.970+0000 7ff007fff640 1 --2- 172.21.15.107:0/2444131258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080c52a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff00806e770 tx=0x7feff402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a1d0b34294a5f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff0077fe640 1 -- 172.21.15.107:0/2444131258 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff402e070 con 0x7ff0080cbf20 2023-11-24T08:04:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff0077fe640 1 -- 172.21.15.107:0/2444131258 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff40315a0 con 0x7ff0080cbf20 2023-11-24T08:04:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff0077fe640 1 -- 172.21.15.107:0/2444131258 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4031930 con 0x7ff0080cbf20 2023-11-24T08:04:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 -- 172.21.15.107:0/2444131258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 msgr2=0x7ff0080c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/2444131258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080c52a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff00806e770 tx=0x7feff402b840 comp rx=0 tx=0).stop 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 -- 172.21.15.107:0/2444131258 shutdown_connections 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/2444131258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080c52a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/2444131258 >> v2:172.21.15.120:3300/0 conn(0x7ff00815b440 0x7ff0080cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.971+0000 7ff00f1dc640 1 -- 172.21.15.107:0/2444131258 >> 172.21.15.107:0/2444131258 conn(0x7ff0080b0c50 msgr2=0x7ff0080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.972+0000 7ff00f1dc640 1 -- 172.21.15.107:0/2444131258 shutdown_connections 2023-11-24T08:04:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.972+0000 7ff00f1dc640 1 -- 172.21.15.107:0/2444131258 wait complete. 2023-11-24T08:04:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.973+0000 7ff00f1dc640 1 Processor -- start 2023-11-24T08:04:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.973+0000 7ff00f1dc640 1 -- start start 2023-11-24T08:04:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0080c57e0 0x7ff008156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 0x7ff008153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff008099db0 con 0x7ff0080cbf20 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00809a090 con 0x7ff00815b440 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.974+0000 7ff00f1dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00809a370 con 0x7ff0080c57e0 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57524/0 (socket says 172.21.15.107:57524) 2023-11-24T08:04:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 -- 172.21.15.107:0/1586235580 learned_addr learned my addr 172.21.15.107:0/1586235580 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 -- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0080c57e0 msgr2=0x7ff008156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff00d779640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 0x7ff008153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff00cf78640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0080c57e0 0x7ff008156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0080c57e0 0x7ff008156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 -- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 msgr2=0x7ff008153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 0x7ff008153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 -- 172.21.15.107:0/1586235580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff4009400 con 0x7ff0080cbf20 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff00d779640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 0x7ff008153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.975+0000 7ff007fff640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.976+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4032840 con 0x7ff0080cbf20 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.976+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff40329c0 con 0x7ff0080cbf20 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.976+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4032d70 con 0x7ff0080cbf20 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.976+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00809a650 con 0x7ff0080cbf20 2023-11-24T08:04:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.976+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff008157340 con 0x7ff0080cbf20 2023-11-24T08:04:56.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.977+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00809ab70 con 0x7ff0080cbf20 2023-11-24T08:04:56.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.978+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feff403b3f0 con 0x7ff0080cbf20 2023-11-24T08:04:56.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.979+0000 7ff0057fa640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 0x7fefec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:56.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.979+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feff40ab6b0 con 0x7ff0080cbf20 2023-11-24T08:04:56.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.980+0000 7ff00cf78640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 0x7fefec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.980+0000 7ff00cf78640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 0x7fefec06d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feffc00a770 tx=0x7feffc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:56.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:56.984+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff00809ab70 con 0x7ff0080cbf20 2023-11-24T08:04:57.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.213+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff00809ab70 con 0x7ff0080cbf20 2023-11-24T08:04:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.728+0000 7ff0057fa640 1 -- 172.21.15.107:0/1586235580 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff00809ab70 con 0x7ff0080cbf20 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 msgr2=0x7fefec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 0x7fefec06d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feffc00a770 tx=0x7feffc008040 comp rx=0 tx=0).stop 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 msgr2=0x7ff0080b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff40046c0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 shutdown_connections 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefec06ae40 0x7fefec06d300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00815b440 0x7ff008153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0080cbf20 0x7ff0080b01d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 --2- 172.21.15.107:0/1586235580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0080c57e0 0x7ff008156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 >> 172.21.15.107:0/1586235580 conn(0x7ff0080b0c50 msgr2=0x7ff0080713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 shutdown_connections 2023-11-24T08:04:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.732+0000 7ff00f1dc640 1 -- 172.21.15.107:0/1586235580 wait complete. 2023-11-24T08:04:57.754 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:04:57.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:04:57.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.955+0000 7ff3cad33640 1 Processor -- start 2023-11-24T08:04:57.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.955+0000 7ff3cad33640 1 -- start start 2023-11-24T08:04:57.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3cad33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3c40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3cad33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c40cece0 con 0x7ff3c415c7f0 2023-11-24T08:04:57.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3cad33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c406d040 con 0x7ff3c40cdfb0 2023-11-24T08:04:57.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3cad33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4156010 con 0x7ff3c40cf450 2023-11-24T08:04:57.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3c8acf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff3c415c7f0 0x7ff3c406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59040/0 (socket says 172.21.15.107:59040) 2023-11-24T08:04:57.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3c8acf640 1 -- 172.21.15.107:0/4267082669 learned_addr learned my addr 172.21.15.107:0/4267082669 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.957+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3c40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362352471 0 0) 0x7ff3c40cece0 con 0x7ff3c415c7f0 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac003520 con 0x7ff3c415c7f0 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294042827 0 0) 0x7ff3ac003520 con 0x7ff3c415c7f0 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 msgr2=0x7ff3c40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3c40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 >> v1:172.21.15.120:6789/0 conn(0x7ff3c40cdfb0 legacy=0x7ff3c40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c406d320 con 0x7ff3c415c7f0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff3b8001670 con 0x7ff3c415c7f0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 >> v1:172.21.15.107:6789/0 conn(0x7ff3c415c7f0 legacy=0x7ff3c406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ac0047c0 0x7ff3ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4156010 con 0x7ff3ac0047c0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac00bae0 con 0x7ff3ac0039a0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac00bca0 con 0x7ff3c40cf450 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.958+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ac0047c0 0x7ff3ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c8acf640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c3fff640 1 -- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 msgr2=0x7ff3ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c92d0640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c3fff640 1 -- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0039a0 msgr2=0x7ff3ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c3fff640 1 -- 172.21.15.107:0/4267082669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac003520 con 0x7ff3ac0047c0 2023-11-24T08:04:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.959+0000 7ff3c92d0640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cf450 0x7ff3ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:57.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.960+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ac0047c0 0x7ff3ac004c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400ba30 tx=0x7ff3ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80297372421103f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:57.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.960+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4032650 con 0x7ff3ac0047c0 2023-11-24T08:04:57.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.960+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b40327b0 con 0x7ff3ac0047c0 2023-11-24T08:04:57.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.960+0000 7ff3c37fe640 1 -- 172.21.15.107:0/4267082669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4032b40 con 0x7ff3ac0047c0 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 -- 172.21.15.107:0/4267082669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ac0047c0 msgr2=0x7ff3ac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 --2- 172.21.15.107:0/4267082669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ac0047c0 0x7ff3ac004c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400ba30 tx=0x7ff3ac0057f0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3c92d0640 1 -- 172.21.15.107:0/4267082669 reap_dead start 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 -- 172.21.15.107:0/4267082669 shutdown_connections 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 -- 172.21.15.107:0/4267082669 >> 172.21.15.107:0/4267082669 conn(0x7ff3c406a920 msgr2=0x7ff3c406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 -- 172.21.15.107:0/4267082669 shutdown_connections 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.961+0000 7ff3cad33640 1 -- 172.21.15.107:0/4267082669 wait complete. 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.963+0000 7ff3cad33640 1 Processor -- start 2023-11-24T08:04:57.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.963+0000 7ff3cad33640 1 -- start start 2023-11-24T08:04:57.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 0x7ff3c41ef550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40ad550 0x7ff3c40b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40ad9d0 con 0x7ff3c40982a0 2023-11-24T08:04:57.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c41efa90 con 0x7ff3ac0047c0 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3cad33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c41f28a0 con 0x7ff3c40ad550 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44050/0 (socket says 172.21.15.107:44050) 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 -- 172.21.15.107:0/2384432257 learned_addr learned my addr 172.21.15.107:0/2384432257 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c8acf640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 0x7ff3c41ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c92d0640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40ad550 0x7ff3c40b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 -- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40ad550 msgr2=0x7ff3c40b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40ad550 0x7ff3c40b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 -- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 msgr2=0x7ff3c41ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 0x7ff3c41ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c3fff640 1 -- 172.21.15.107:0/2384432257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4149aa0 con 0x7ff3c40982a0 2023-11-24T08:04:57.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.964+0000 7ff3c8acf640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 0x7ff3c41ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3c3fff640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff3b402bed0 tx=0x7ff3b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b403b830 con 0x7ff3c40982a0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b403b990 con 0x7ff3c40982a0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b403bd40 con 0x7ff3c40982a0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c4148ed0 con 0x7ff3c40982a0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.965+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c4149450 con 0x7ff3c40982a0 2023-11-24T08:04:57.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.966+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4067470 con 0x7ff3c40982a0 2023-11-24T08:04:57.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.967+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff3b402e070 con 0x7ff3c40982a0 2023-11-24T08:04:57.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.968+0000 7ff3c17fa640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 0x7ff3b006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:57.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.969+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3b40ab9d0 con 0x7ff3c40982a0 2023-11-24T08:04:57.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.969+0000 7ff3c8acf640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 0x7ff3b006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:57.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.969+0000 7ff3c8acf640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 0x7ff3b006d2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8001eb0 tx=0x7ff3b8001d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:57.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:57.973+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3b4076950 con 0x7ff3c40982a0 2023-11-24T08:04:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.200+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3c40cd730 con 0x7ff3c40982a0 2023-11-24T08:04:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.201+0000 7ff3c17fa640 1 -- 172.21.15.107:0/2384432257 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7ff3b4036070 con 0x7ff3c40982a0 2023-11-24T08:04:58.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:58.203 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:04:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.203+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 msgr2=0x7ff3b006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.203+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 0x7ff3b006d2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8001eb0 tx=0x7ff3b8001d40 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.203+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 msgr2=0x7ff3c40ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.203+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff3b402bed0 tx=0x7ff3b4004730 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 shutdown_connections 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006ae30 0x7ff3b006d2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40ad550 0x7ff3c40b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c40982a0 0x7ff3c40ad010 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 --2- 172.21.15.107:0/2384432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3ac0047c0 0x7ff3c41ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 >> 172.21.15.107:0/2384432257 conn(0x7ff3c406a920 msgr2=0x7ff3c415b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 shutdown_connections 2023-11-24T08:04:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.204+0000 7ff3cad33640 1 -- 172.21.15.107:0/2384432257 wait complete. 2023-11-24T08:04:58.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:04:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.430+0000 7fe5dc9c8640 1 Processor -- start 2023-11-24T08:04:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.431+0000 7fe5dc9c8640 1 -- start start 2023-11-24T08:04:58.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5dc9c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 0x7fe5d80ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5dc9c8640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5d80cfbe0 0x7fe5d80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5dc9c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d80d1d20 con 0x7fe5d8161060 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5dc9c8640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d80d05a0 con 0x7fe5d80cfbe0 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5dc9c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b90c0 con 0x7fe5d80b8ce0 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5d759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5d8161060 0x7fe5d81614f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59052/0 (socket says 172.21.15.107:59052) 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.432+0000 7fe5d759e640 1 -- 172.21.15.107:0/3533241266 learned_addr learned my addr 172.21.15.107:0/3533241266 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 0x7fe5d80ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d659c640 1 --2- 172.21.15.107:0/3533241266 >> v2:172.21.15.120:3300/0 conn(0x7fe5d80cfbe0 0x7fe5d80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754375642 0 0) 0x7fe5d80d1d20 con 0x7fe5d8161060 2023-11-24T08:04:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c4003520 con 0x7fe5d8161060 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776600185 0 0) 0x7fe5c4003520 con 0x7fe5d8161060 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 msgr2=0x7fe5d80ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 0x7fe5d80ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 >> v2:172.21.15.120:3300/0 conn(0x7fe5d80cfbe0 msgr2=0x7fe5d80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 --2- 172.21.15.107:0/3533241266 >> v2:172.21.15.120:3300/0 conn(0x7fe5d80cfbe0 0x7fe5d80d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80cf2d0 con 0x7fe5d8161060 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe5d0001670 con 0x7fe5d8161060 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 >> v1:172.21.15.107:6789/0 conn(0x7fe5d8161060 legacy=0x7fe5d81614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 0x7fe5c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.433+0000 7fe5d5d9b640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5c4004d00 0x7fe5c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80d05a0 con 0x7fe5c4003970 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b90c0 con 0x7fe5c4004d00 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c400b980 con 0x7fe5d80b8ce0 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 msgr2=0x7fe5c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80b8ce0 0x7fe5c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5c4004d00 msgr2=0x7fe5c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5c4004d00 0x7fe5c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.434+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/3533241266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c4003520 con 0x7fe5c4003970 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.435+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5c4003d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009ac0 tx=0x7fe5c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca6fb041ab8f061 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.435+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c002c710 con 0x7fe5c4003970 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.435+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c002c870 con 0x7fe5c4003970 2023-11-24T08:04:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5d5d9b640 1 -- 172.21.15.107:0/3533241266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c002cc00 con 0x7fe5c4003970 2023-11-24T08:04:58.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/3533241266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 msgr2=0x7fe5c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/3533241266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5c4003d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009ac0 tx=0x7fe5c40057f0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5d759e640 1 -- 172.21.15.107:0/3533241266 reap_dead start 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/3533241266 shutdown_connections 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/3533241266 >> 172.21.15.107:0/3533241266 conn(0x7fe5d809c530 msgr2=0x7fe5d80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/3533241266 shutdown_connections 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.436+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/3533241266 wait complete. 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.438+0000 7fe5dc9c8640 1 Processor -- start 2023-11-24T08:04:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.438+0000 7fe5dc9c8640 1 -- start start 2023-11-24T08:04:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5d80c71d0 0x7fe5d80c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80cbf90 0x7fe5d80cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80c7b90 con 0x7fe5c4003970 2023-11-24T08:04:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80ccc80 con 0x7fe5d80c71d0 2023-11-24T08:04:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5dc9c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80ccf60 con 0x7fe5d80cbf90 2023-11-24T08:04:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44066/0 (socket says 172.21.15.107:44066) 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.439+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/1979767611 learned_addr learned my addr 172.21.15.107:0/1979767611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d759e640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80cbf90 0x7fe5d80cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80cbf90 msgr2=0x7fe5d80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d659c640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5d80c71d0 0x7fe5d80c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80cbf90 0x7fe5d80cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5d80c71d0 msgr2=0x7fe5d80c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5d80c71d0 0x7fe5d80c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 -- 172.21.15.107:0/1979767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8158010 con 0x7fe5c4003970 2023-11-24T08:04:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5d6d9d640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000c00 tx=0x7fe5c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0003810 con 0x7fe5c4003970 2023-11-24T08:04:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c002de90 con 0x7fe5c4003970 2023-11-24T08:04:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0003bb0 con 0x7fe5c4003970 2023-11-24T08:04:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.440+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d81582f0 con 0x7fe5c4003970 2023-11-24T08:04:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.441+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d80cc950 con 0x7fe5c4003970 2023-11-24T08:04:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.442+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d80727b0 con 0x7fe5c4003970 2023-11-24T08:04:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.442+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe5c0003d10 con 0x7fe5c4003970 2023-11-24T08:04:58.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.444+0000 7fe5cf7fe640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 0x7fe5b806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.444+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5c00ab5b0 con 0x7fe5c4003970 2023-11-24T08:04:58.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.444+0000 7fe5d659c640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 0x7fe5b806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.445+0000 7fe5d659c640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 0x7fe5b806d330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8157700 tx=0x7fe5c800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.448+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5c0076830 con 0x7fe5c4003970 2023-11-24T08:04:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.640+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5d804e840 con 0x7fe5c4003970 2023-11-24T08:04:58.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.642+0000 7fe5cf7fe640 1 -- 172.21.15.107:0/1979767611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fe5c0079e50 con 0x7fe5c4003970 2023-11-24T08:04:58.644 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:58.644 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.644+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 msgr2=0x7fe5b806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.644+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 0x7fe5b806d330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8157700 tx=0x7fe5c800b040 comp rx=0 tx=0).stop 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.644+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 msgr2=0x7fe5d8156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.644+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000c00 tx=0x7fe5c0004730 comp rx=0 tx=0).stop 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 shutdown_connections 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5b806ae70 0x7fe5b806d330 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5d80cbf90 0x7fe5d80cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5d80c71d0 0x7fe5d80c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 --2- 172.21.15.107:0/1979767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5c4003970 0x7fe5d8156630 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 >> 172.21.15.107:0/1979767611 conn(0x7fe5d809c530 msgr2=0x7fe5d80b7380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 shutdown_connections 2023-11-24T08:04:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.645+0000 7fe5dc9c8640 1 -- 172.21.15.107:0/1979767611 wait complete. 2023-11-24T08:04:58.666 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:04:58.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:04:58.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.878+0000 7f41084e7640 1 Processor -- start 2023-11-24T08:04:58.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.878+0000 7f41084e7640 1 -- start start 2023-11-24T08:04:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f41084e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f410015c7f0 0x7f410006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f41084e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 0x7f41000cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f41084e7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41000cece0 con 0x7f41000cdfb0 2023-11-24T08:04:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f41084e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4100156010 con 0x7f410015c7f0 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f41084e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410006ce30 con 0x7f41000cf450 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f410015c7f0 0x7f410006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f410015c7f0 0x7f410006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44072/0 (socket says 172.21.15.107:44072) 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 -- 172.21.15.107:0/2570693048 learned_addr learned my addr 172.21.15.107:0/2570693048 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4105a82640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 0x7f41000cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 -- 172.21.15.107:0/2570693048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 msgr2=0x7f41000cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 0x7f41000cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 -- 172.21.15.107:0/2570693048 >> v1:172.21.15.120:6789/0 conn(0x7f41000cdfb0 legacy=0x7f41000ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4106283640 1 -- 172.21.15.107:0/2570693048 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410006d040 con 0x7f410015c7f0 2023-11-24T08:04:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.880+0000 7f4105a82640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 0x7f41000cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f4106283640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f410015c7f0 0x7f410006db20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009870 tx=0x7f40fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e74251a1c905e96 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f4105281640 1 -- 172.21.15.107:0/2570693048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031640 con 0x7f410015c7f0 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f4105281640 1 -- 172.21.15.107:0/2570693048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc0317a0 con 0x7f410015c7f0 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f4105281640 1 -- 172.21.15.107:0/2570693048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031b30 con 0x7f410015c7f0 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f41084e7640 1 -- 172.21.15.107:0/2570693048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f410015c7f0 msgr2=0x7f410006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f41084e7640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f410015c7f0 0x7f410006db20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009870 tx=0x7f40fc02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f41084e7640 1 -- 172.21.15.107:0/2570693048 shutdown_connections 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.882+0000 7f41084e7640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cf450 0x7f41000cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.883+0000 7f41084e7640 1 --2- 172.21.15.107:0/2570693048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f410015c7f0 0x7f410006db20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.883+0000 7f41084e7640 1 -- 172.21.15.107:0/2570693048 >> 172.21.15.107:0/2570693048 conn(0x7f410006a920 msgr2=0x7f410006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.883+0000 7f41084e7640 1 -- 172.21.15.107:0/2570693048 shutdown_connections 2023-11-24T08:04:58.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.883+0000 7f41084e7640 1 -- 172.21.15.107:0/2570693048 wait complete. 2023-11-24T08:04:58.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.884+0000 7f41084e7640 1 Processor -- start 2023-11-24T08:04:58.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.884+0000 7f41084e7640 1 -- start start 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cdfb0 0x7f4100158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f410015c7f0 0x7f4100155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001f6d80 con 0x7f41000cf450 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001f6fb0 con 0x7f410015c7f0 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.885+0000 7f41084e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41001f7290 con 0x7f41000cdfb0 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44078/0 (socket says 172.21.15.107:44078) 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 -- 172.21.15.107:0/3144973032 learned_addr learned my addr 172.21.15.107:0/3144973032 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4106283640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cdfb0 0x7f4100158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4106a84640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f410015c7f0 0x7f4100155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 -- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cdfb0 msgr2=0x7f4100158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cdfb0 0x7f4100158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 -- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f410015c7f0 msgr2=0x7f4100155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f410015c7f0 0x7f4100155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 -- 172.21.15.107:0/3144973032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc0094d0 con 0x7f41000cf450 2023-11-24T08:04:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.886+0000 7f4105a82640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40f000c7f0 tx=0x7f40f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.887+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0007a30 con 0x7f41000cf450 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.887+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0007b90 con 0x7f41000cf450 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.887+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000ce40 con 0x7f41000cf450 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.887+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41001f7570 con 0x7f41000cf450 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.887+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4100155890 con 0x7f41000cf450 2023-11-24T08:04:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.888+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100156510 con 0x7f41000cf450 2023-11-24T08:04:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.889+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f40f00113f0 con 0x7f41000cf450 2023-11-24T08:04:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.890+0000 7f40f6ffd640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 0x7f40e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.890+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40f00898a0 con 0x7f41000cf450 2023-11-24T08:04:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.890+0000 7f4106283640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 0x7f40e406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.891+0000 7f4106283640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 0x7f40e406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40fc036000 tx=0x7f40fc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:58.894+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4100156510 con 0x7f41000cf450 2023-11-24T08:04:59.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.080+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41001f7400 con 0x7f40e406ae40 2023-11-24T08:04:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.101+0000 7f40f6ffd640 1 -- 172.21.15.107:0/3144973032 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252627 (secure 0 0 0) 0x7f41001f7400 con 0x7f40e406ae40 2023-11-24T08:04:59.106 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:59.108 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 msgr2=0x7f40e406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 0x7f40e406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40fc036000 tx=0x7f40fc03c040 comp rx=0 tx=0).stop 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 msgr2=0x7f41001f6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40f000c7f0 tx=0x7f40f000cbf0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 shutdown_connections 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f40e406ae40 0x7f40e406d300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f410015c7f0 0x7f4100155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41000cf450 0x7f41001f6840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 --2- 172.21.15.107:0/3144973032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41000cdfb0 0x7f4100158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 >> 172.21.15.107:0/3144973032 conn(0x7f410006a920 msgr2=0x7f410015eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 shutdown_connections 2023-11-24T08:04:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.106+0000 7f41084e7640 1 -- 172.21.15.107:0/3144973032 wait complete. 2023-11-24T08:04:59.127 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2023-11-24T08:04:57.432475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218632,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756088,"statfs":{"total":773094113280,"available":772870234112,"internally_reserved":0,"allocated":2531328,"data_stored":1593216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.950885"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.500043+0000","last_change":"2023-11-24T08:04:52.500043+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.500043+0000","last_undegraded":"2023-11-24T08:04:52.500043+0000","last_fullsized":"2023-11-24T08:04:52.500043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.501416+0000","last_change":"2023-11-24T08:04:52.501416+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.501416+0000","last_undegraded":"2023-11-24T08:04:52.501416+0000","last_fullsized":"2023-11-24T08:04:52.501416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.499925+0000","last_change":"2023-11-24T08:04:52.499925+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.499925+0000","last_undegraded":"2023-11-24T08:04:52.499925+0000","last_fullsized":"2023-11-24T08:04:52.499925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.497662+0000","last_change":"2023-11-24T08:04:52.497662+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.497662+0000","last_undegraded":"2023-11-24T08:04:52.497662+0000","last_fullsized":"2023-11-24T08:04:52.497662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.505535+0000","last_change":"2023-11-24T08:04:52.505535+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.505535+0000","last_undegraded":"2023-11-24T08:04:52.505535+0000","last_fullsized":"2023-11-24T08:04:52.505535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.498684+0000","last_change":"2023-11-24T08:04:52.498684+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.498684+0000","last_undegraded":"2023-11-24T08:04:52.498684+0000","last_fullsized":"2023-11-24T08:04:52.498684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.495597+0000","last_change":"2023-11-24T08:04:52.495597+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.495597+0000","last_undegraded":"2023-11-24T08:04:52.495597+0000","last_fullsized":"2023-11-24T08:04:52.495597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.502741+0000","last_change":"2023-11-24T08:04:52.502741+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.502741+0000","last_undegraded":"2023-11-24T08:04:52.502741+0000","last_fullsized":"2023-11-24T08:04:52.502741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.492476+0000","last_change":"2023-11-24T08:04:52.492476+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.492476+0000","last_undegraded":"2023-11-24T08:04:52.492476+0000","last_fullsized":"2023-11-24T08:04:52.492476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.500970+0000","last_change":"2023-11-24T08:04:52.500970+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.500970+0000","last_undegraded":"2023-11-24T08:04:52.500970+0000","last_fullsized":"2023-11-24T08:04:52.500970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.500821+0000","last_change":"2023-11-24T08:04:52.500821+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.500821+0000","last_undegraded":"2023-11-24T08:04:52.500821+0000","last_fullsized":"2023-11-24T08:04:52.500821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.504345+0000","last_change":"2023-11-24T08:04:52.504345+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.504345+0000","last_undegraded":"2023-11-24T08:04:52.504345+0000","last_fullsized":"2023-11-24T08:04:52.504345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.491590+0000","last_change":"2023-11-24T08:04:52.491590+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.491590+0000","last_undegraded":"2023-11-24T08:04:52.491590+0000","last_fullsized":"2023-11-24T08:04:52.491590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.505749+0000","last_change":"2023-11-24T08:04:52.505749+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.505749+0000","last_undegraded":"2023-11-24T08:04:52.505749+0000","last_fullsized":"2023-11-24T08:04:52.505749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.494379+0000","last_change":"2023-11-24T08:04:52.494379+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.494379+0000","last_undegraded":"2023-11-24T08:04:52.494379+0000","last_fullsized":"2023-11-24T08:04:52.494379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.503894+0000","last_change":"2023-11-24T08:04:52.503894+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.503894+0000","last_undegraded":"2023-11-24T08:04:52.503894+0000","last_fullsized":"2023-11-24T08:04:52.503894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-11-24T08:04:48.475654+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:04:48.475654+0000","last_peered":"2023-11-24T08:04:48.475654+0000","last_clean":"2023-11-24T08:04:48.475654+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:04:48.475654+0000","last_undegraded":"2023-11-24T08:04:48.475654+0000","last_fullsized":"2023-11-24T08:04:48.475654+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.497781+0000","last_change":"2023-11-24T08:04:52.497781+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.497781+0000","last_undegraded":"2023-11-24T08:04:52.497781+0000","last_fullsized":"2023-11-24T08:04:52.497781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.488882+0000","last_change":"2023-11-24T08:04:52.488882+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.488882+0000","last_undegraded":"2023-11-24T08:04:52.488882+0000","last_fullsized":"2023-11-24T08:04:52.488882+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.503523+0000","last_change":"2023-11-24T08:04:52.503523+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.503523+0000","last_undegraded":"2023-11-24T08:04:52.503523+0000","last_fullsized":"2023-11-24T08:04:52.503523+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.498604+0000","last_change":"2023-11-24T08:04:52.498604+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.498604+0000","last_undegraded":"2023-11-24T08:04:52.498604+0000","last_fullsized":"2023-11-24T08:04:52.498604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.493394+0000","last_change":"2023-11-24T08:04:52.493394+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.493394+0000","last_undegraded":"2023-11-24T08:04:52.493394+0000","last_fullsized":"2023-11-24T08:04:52.493394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.489123+0000","last_change":"2023-11-24T08:04:52.489123+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.489123+0000","last_undegraded":"2023-11-24T08:04:52.489123+0000","last_fullsized":"2023-11-24T08:04:52.489123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.492076+0000","last_change":"2023-11-24T08:04:52.492076+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.492076+0000","last_undegraded":"2023-11-24T08:04:52.492076+0000","last_fullsized":"2023-11-24T08:04:52.492076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.502910+0000","last_change":"2023-11-24T08:04:52.502910+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.502910+0000","last_undegraded":"2023-11-24T08:04:52.502910+0000","last_fullsized":"2023-11-24T08:04:52.502910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.491327+0000","last_change":"2023-11-24T08:04:52.491327+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.491327+0000","last_undegraded":"2023-11-24T08:04:52.491327+0000","last_fullsized":"2023-11-24T08:04:52.491327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.492074+0000","last_change":"2023-11-24T08:04:52.492074+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.492074+0000","last_undegraded":"2023-11-24T08:04:52.492074+0000","last_fullsized":"2023-11-24T08:04:52.492074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.493890+0000","last_change":"2023-11-24T08:04:52.493890+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.493890+0000","last_undegraded":"2023-11-24T08:04:52.493890+0000","last_fullsized":"2023-11-24T08:04:52.493890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.489642+0000","last_change":"2023-11-24T08:04:52.489642+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.489642+0000","last_undegraded":"2023-11-24T08:04:52.489642+0000","last_fullsized":"2023-11-24T08:04:52.489642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.489898+0000","last_change":"2023-11-24T08:04:52.489898+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.489898+0000","last_undegraded":"2023-11-24T08:04:52.489898+0000","last_fullsized":"2023-11-24T08:04:52.489898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.492106+0000","last_change":"2023-11-24T08:04:52.492106+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.492106+0000","last_undegraded":"2023-11-24T08:04:52.492106+0000","last_fullsized":"2023-11-24T08:04:52.492106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.499192+0000","last_change":"2023-11-24T08:04:52.499192+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.499192+0000","last_undegraded":"2023-11-24T08:04:52.499192+0000","last_fullsized":"2023-11-24T08:04:52.499192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:04:53.480238+0000","last_change":"2023-11-24T08:04:53.480238+0000","last_active":"2023-11-24T08:04:53.480238+0000","last_peered":"2023-11-24T08:04:53.480238+0000","last_clean":"2023-11-24T08:04:53.480238+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:53.480238+0000","last_undegraded":"2023-11-24T08:04:53.480238+0000","last_fullsized":"2023-11-24T08:04:53.480238+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2023-11-24T08:04:52.492792+0000","last_change":"2023-11-24T08:04:52.492792+0000","last_active":"2023-11-24T08:04:52.467483+0000","last_peered":"2023-11-24T08:04:52.467483+0000","last_clean":"2023-11-24T08:04:52.467483+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:04:52.492792+0000","last_undegraded":"2023-11-24T08:04:52.492792+0000","last_fullsized":"2023-11-24T08:04:52.492792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7989999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1509999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:04:59.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:04:59.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.381+0000 7f3090c75640 1 Processor -- start 2023-11-24T08:04:59.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.381+0000 7f3090c75640 1 -- start start 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f3090c75640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f308c15e390 0x7f308c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f3090c75640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c0ce0b0 con 0x7f308c157c70 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f3090c75640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c161960 con 0x7f308c0cfb60 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f3090c75640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c15e8d0 con 0x7f308c15e390 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f308a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f308c157c70 0x7f308c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59060/0 (socket says 172.21.15.107:59060) 2023-11-24T08:04:59.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f308a59c640 1 -- 172.21.15.107:0/732560968 learned_addr learned my addr 172.21.15.107:0/732560968 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f308ad9d640 1 --2- 172.21.15.107:0/732560968 >> v2:172.21.15.120:3300/0 conn(0x7f308c15e390 0x7f308c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.383+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972262077 0 0) 0x7f308c0ce0b0 con 0x7f308c157c70 2023-11-24T08:04:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3078003520 con 0x7f308c157c70 2023-11-24T08:04:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267529112 0 0) 0x7f3078003520 con 0x7f308c157c70 2023-11-24T08:04:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 >> v2:172.21.15.120:3300/0 conn(0x7f308c15e390 msgr2=0x7f308c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 --2- 172.21.15.107:0/732560968 >> v2:172.21.15.120:3300/0 conn(0x7f308c15e390 0x7f308c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 >> v1:172.21.15.120:6789/0 conn(0x7f308c0cfb60 legacy=0x7f308c15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c15ebb0 con 0x7f308c157c70 2023-11-24T08:04:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f307c0019f0 con 0x7f308c157c70 2023-11-24T08:04:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 >> v1:172.21.15.107:6789/0 conn(0x7f308c157c70 legacy=0x7f308c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3078003970 0x7f3078003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f3078004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30780052a0 0x7f307800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c15e8d0 con 0x7f30780047c0 2023-11-24T08:04:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307800c170 con 0x7f30780052a0 2023-11-24T08:04:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307800c330 con 0x7f3078003970 2023-11-24T08:04:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.384+0000 7f3081d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f3078004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f308ad9d640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30780052a0 0x7f307800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f308a59c640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3078003970 0x7f3078003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f3081d9b640 1 -- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3078003970 msgr2=0x7f3078003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f3081d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3078003970 0x7f3078003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f3081d9b640 1 -- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30780052a0 msgr2=0x7f307800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f3081d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30780052a0 0x7f307800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f3081d9b640 1 -- 172.21.15.107:0/732560968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078003520 con 0x7f30780047c0 2023-11-24T08:04:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.385+0000 7f308ad9d640 1 -- 172.21.15.107:0/732560968 reap_dead start 2023-11-24T08:04:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.386+0000 7f3081d9b640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f3078004c40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3074001720 tx=0x7f30780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a4cfe9900b48fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.386+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740052d0 con 0x7f30780047c0 2023-11-24T08:04:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.386+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307402e050 con 0x7f30780047c0 2023-11-24T08:04:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.386+0000 7f3089d9b640 1 -- 172.21.15.107:0/732560968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074003830 con 0x7f30780047c0 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 -- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 msgr2=0x7f3078004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f3078004c40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3074001720 tx=0x7f30780057f0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 -- 172.21.15.107:0/732560968 shutdown_connections 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 --2- 172.21.15.107:0/732560968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f3078004c40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 -- 172.21.15.107:0/732560968 >> 172.21.15.107:0/732560968 conn(0x7f308c06a920 msgr2=0x7f308c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 -- 172.21.15.107:0/732560968 shutdown_connections 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.387+0000 7f3090c75640 1 -- 172.21.15.107:0/732560968 wait complete. 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.389+0000 7f3090c75640 1 Processor -- start 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.389+0000 7f3090c75640 1 -- start start 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f308c156e10 0x7f308c0cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f308c0af1b0 0x7f308c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0afba0 con 0x7f30780047c0 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c098ef0 con 0x7f308c0af1b0 2023-11-24T08:04:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3090c75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c156980 con 0x7f308c156e10 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44098/0 (socket says 172.21.15.107:44098) 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 -- 172.21.15.107:0/1496994117 learned_addr learned my addr 172.21.15.107:0/1496994117 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f3081d9b640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f308c156e10 0x7f308c0cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 -- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f308c156e10 msgr2=0x7f308c0cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308ad9d640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f308c0af1b0 0x7f308c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f308c156e10 0x7f308c0cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 -- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f308c0af1b0 msgr2=0x7f308c0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f308c0af1b0 0x7f308c0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.390+0000 7f308a59c640 1 -- 172.21.15.107:0/1496994117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c155a90 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f308a59c640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f307c007a20 tx=0x7f307c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c00ee50 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c013420 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c155d70 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c0137d0 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.391+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c156330 con 0x7f30780047c0 2023-11-24T08:04:59.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.392+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c067470 con 0x7f30780047c0 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.393+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f307c013930 con 0x7f30780047c0 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.394+0000 7f30837fe640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 0x7f306c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.394+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f307c089df0 con 0x7f30780047c0 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.394+0000 7f3081d9b640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 0x7f306c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.395+0000 7f3081d9b640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 0x7f306c06d330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3074033000 tx=0x7f3074009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.398+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f307c058930 con 0x7f30780047c0 2023-11-24T08:04:59.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.588+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f308c158050 con 0x7f306c06ae70 2023-11-24T08:04:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.596+0000 7f30837fe640 1 -- 172.21.15.107:0/1496994117 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264382 (secure 0 0 0) 0x7f3074005430 con 0x7f306c06ae70 2023-11-24T08:04:59.600 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:04:59.603 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:04:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 msgr2=0x7f306c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 0x7f306c06d330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3074033000 tx=0x7f3074009030 comp rx=0 tx=0).stop 2023-11-24T08:04:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 msgr2=0x7f308c0989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f307c007a20 tx=0x7f307c00e530 comp rx=0 tx=0).stop 2023-11-24T08:04:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 shutdown_connections 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306c06ae70 0x7f306c06d330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f308c0af1b0 0x7f308c0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f308c156e10 0x7f308c0cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 --2- 172.21.15.107:0/1496994117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30780047c0 0x7f308c0989b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 >> 172.21.15.107:0/1496994117 conn(0x7f308c06a920 msgr2=0x7f308c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 shutdown_connections 2023-11-24T08:04:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.603+0000 7f3090c75640 1 -- 172.21.15.107:0/1496994117 wait complete. 2023-11-24T08:04:59.625 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-11-24T08:04:59.433025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001503"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008109+0000","last_change":"2023-11-24T08:04:54.558355+0000","last_active":"2023-11-24T08:04:57.008109+0000","last_peered":"2023-11-24T08:04:57.008109+0000","last_clean":"2023-11-24T08:04:57.008109+0000","last_became_active":"2023-11-24T08:04:54.558044+0000","last_became_peered":"2023-11-24T08:04:54.558044+0000","last_unstale":"2023-11-24T08:04:57.008109+0000","last_undegraded":"2023-11-24T08:04:57.008109+0000","last_fullsized":"2023-11-24T08:04:57.008109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536483+0000","last_change":"2023-11-24T08:04:54.555145+0000","last_active":"2023-11-24T08:04:55.536483+0000","last_peered":"2023-11-24T08:04:55.536483+0000","last_clean":"2023-11-24T08:04:55.536483+0000","last_became_active":"2023-11-24T08:04:54.554789+0000","last_became_peered":"2023-11-24T08:04:54.554789+0000","last_unstale":"2023-11-24T08:04:55.536483+0000","last_undegraded":"2023-11-24T08:04:55.536483+0000","last_fullsized":"2023-11-24T08:04:55.536483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111906+0000","last_change":"2023-11-24T08:04:54.548731+0000","last_active":"2023-11-24T08:04:56.111906+0000","last_peered":"2023-11-24T08:04:56.111906+0000","last_clean":"2023-11-24T08:04:56.111906+0000","last_became_active":"2023-11-24T08:04:54.547639+0000","last_became_peered":"2023-11-24T08:04:54.547639+0000","last_unstale":"2023-11-24T08:04:56.111906+0000","last_undegraded":"2023-11-24T08:04:56.111906+0000","last_fullsized":"2023-11-24T08:04:56.111906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005247+0000","last_change":"2023-11-24T08:04:54.563561+0000","last_active":"2023-11-24T08:04:57.005247+0000","last_peered":"2023-11-24T08:04:57.005247+0000","last_clean":"2023-11-24T08:04:57.005247+0000","last_became_active":"2023-11-24T08:04:54.563217+0000","last_became_peered":"2023-11-24T08:04:54.563217+0000","last_unstale":"2023-11-24T08:04:57.005247+0000","last_undegraded":"2023-11-24T08:04:57.005247+0000","last_fullsized":"2023-11-24T08:04:57.005247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009305+0000","last_change":"2023-11-24T08:04:54.545576+0000","last_active":"2023-11-24T08:04:57.009305+0000","last_peered":"2023-11-24T08:04:57.009305+0000","last_clean":"2023-11-24T08:04:57.009305+0000","last_became_active":"2023-11-24T08:04:54.543956+0000","last_became_peered":"2023-11-24T08:04:54.543956+0000","last_unstale":"2023-11-24T08:04:57.009305+0000","last_undegraded":"2023-11-24T08:04:57.009305+0000","last_fullsized":"2023-11-24T08:04:57.009305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005154+0000","last_change":"2023-11-24T08:04:54.562290+0000","last_active":"2023-11-24T08:04:57.005154+0000","last_peered":"2023-11-24T08:04:57.005154+0000","last_clean":"2023-11-24T08:04:57.005154+0000","last_became_active":"2023-11-24T08:04:54.561367+0000","last_became_peered":"2023-11-24T08:04:54.561367+0000","last_unstale":"2023-11-24T08:04:57.005154+0000","last_undegraded":"2023-11-24T08:04:57.005154+0000","last_fullsized":"2023-11-24T08:04:57.005154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008255+0000","last_change":"2023-11-24T08:04:54.557527+0000","last_active":"2023-11-24T08:04:57.008255+0000","last_peered":"2023-11-24T08:04:57.008255+0000","last_clean":"2023-11-24T08:04:57.008255+0000","last_became_active":"2023-11-24T08:04:54.554705+0000","last_became_peered":"2023-11-24T08:04:54.554705+0000","last_unstale":"2023-11-24T08:04:57.008255+0000","last_undegraded":"2023-11-24T08:04:57.008255+0000","last_fullsized":"2023-11-24T08:04:57.008255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697542+0000","last_change":"2023-11-24T08:04:54.549339+0000","last_active":"2023-11-24T08:04:55.697542+0000","last_peered":"2023-11-24T08:04:55.697542+0000","last_clean":"2023-11-24T08:04:55.697542+0000","last_became_active":"2023-11-24T08:04:54.548540+0000","last_became_peered":"2023-11-24T08:04:54.548540+0000","last_unstale":"2023-11-24T08:04:55.697542+0000","last_undegraded":"2023-11-24T08:04:55.697542+0000","last_fullsized":"2023-11-24T08:04:55.697542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008315+0000","last_change":"2023-11-24T08:04:54.556229+0000","last_active":"2023-11-24T08:04:57.008315+0000","last_peered":"2023-11-24T08:04:57.008315+0000","last_clean":"2023-11-24T08:04:57.008315+0000","last_became_active":"2023-11-24T08:04:54.554805+0000","last_became_peered":"2023-11-24T08:04:54.554805+0000","last_unstale":"2023-11-24T08:04:57.008315+0000","last_undegraded":"2023-11-24T08:04:57.008315+0000","last_fullsized":"2023-11-24T08:04:57.008315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006892+0000","last_change":"2023-11-24T08:04:54.544285+0000","last_active":"2023-11-24T08:04:57.006892+0000","last_peered":"2023-11-24T08:04:57.006892+0000","last_clean":"2023-11-24T08:04:57.006892+0000","last_became_active":"2023-11-24T08:04:54.543957+0000","last_became_peered":"2023-11-24T08:04:54.543957+0000","last_unstale":"2023-11-24T08:04:57.006892+0000","last_undegraded":"2023-11-24T08:04:57.006892+0000","last_fullsized":"2023-11-24T08:04:57.006892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005608+0000","last_change":"2023-11-24T08:04:54.539976+0000","last_active":"2023-11-24T08:04:57.005608+0000","last_peered":"2023-11-24T08:04:57.005608+0000","last_clean":"2023-11-24T08:04:57.005608+0000","last_became_active":"2023-11-24T08:04:54.539639+0000","last_became_peered":"2023-11-24T08:04:54.539639+0000","last_unstale":"2023-11-24T08:04:57.005608+0000","last_undegraded":"2023-11-24T08:04:57.005608+0000","last_fullsized":"2023-11-24T08:04:57.005608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538051+0000","last_change":"2023-11-24T08:04:54.546809+0000","last_active":"2023-11-24T08:04:55.538051+0000","last_peered":"2023-11-24T08:04:55.538051+0000","last_clean":"2023-11-24T08:04:55.538051+0000","last_became_active":"2023-11-24T08:04:54.546568+0000","last_became_peered":"2023-11-24T08:04:54.546568+0000","last_unstale":"2023-11-24T08:04:55.538051+0000","last_undegraded":"2023-11-24T08:04:55.538051+0000","last_fullsized":"2023-11-24T08:04:55.538051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:27.753563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698433+0000","last_change":"2023-11-24T08:04:54.563512+0000","last_active":"2023-11-24T08:04:55.698433+0000","last_peered":"2023-11-24T08:04:55.698433+0000","last_clean":"2023-11-24T08:04:55.698433+0000","last_became_active":"2023-11-24T08:04:54.562308+0000","last_became_peered":"2023-11-24T08:04:54.562308+0000","last_unstale":"2023-11-24T08:04:55.698433+0000","last_undegraded":"2023-11-24T08:04:55.698433+0000","last_fullsized":"2023-11-24T08:04:55.698433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111574+0000","last_change":"2023-11-24T08:04:54.552082+0000","last_active":"2023-11-24T08:04:56.111574+0000","last_peered":"2023-11-24T08:04:56.111574+0000","last_clean":"2023-11-24T08:04:56.111574+0000","last_became_active":"2023-11-24T08:04:54.542957+0000","last_became_peered":"2023-11-24T08:04:54.542957+0000","last_unstale":"2023-11-24T08:04:56.111574+0000","last_undegraded":"2023-11-24T08:04:56.111574+0000","last_fullsized":"2023-11-24T08:04:56.111574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:10.754583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535932+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.535932+0000","last_peered":"2023-11-24T08:04:55.535932+0000","last_clean":"2023-11-24T08:04:55.535932+0000","last_became_active":"2023-11-24T08:04:54.561664+0000","last_became_peered":"2023-11-24T08:04:54.561664+0000","last_unstale":"2023-11-24T08:04:55.535932+0000","last_undegraded":"2023-11-24T08:04:55.535932+0000","last_fullsized":"2023-11-24T08:04:55.535932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006259+0000","last_change":"2023-11-24T08:04:54.545602+0000","last_active":"2023-11-24T08:04:57.006259+0000","last_peered":"2023-11-24T08:04:57.006259+0000","last_clean":"2023-11-24T08:04:57.006259+0000","last_became_active":"2023-11-24T08:04:54.545279+0000","last_became_peered":"2023-11-24T08:04:54.545279+0000","last_unstale":"2023-11-24T08:04:57.006259+0000","last_undegraded":"2023-11-24T08:04:57.006259+0000","last_fullsized":"2023-11-24T08:04:57.006259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006979+0000","last_change":"2023-11-24T08:04:54.546983+0000","last_active":"2023-11-24T08:04:57.006979+0000","last_peered":"2023-11-24T08:04:57.006979+0000","last_clean":"2023-11-24T08:04:57.006979+0000","last_became_active":"2023-11-24T08:04:54.545361+0000","last_became_peered":"2023-11-24T08:04:54.545361+0000","last_unstale":"2023-11-24T08:04:57.006979+0000","last_undegraded":"2023-11-24T08:04:57.006979+0000","last_fullsized":"2023-11-24T08:04:57.006979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002164+0000","last_change":"2023-11-24T08:04:54.556787+0000","last_active":"2023-11-24T08:04:57.002164+0000","last_peered":"2023-11-24T08:04:57.002164+0000","last_clean":"2023-11-24T08:04:57.002164+0000","last_became_active":"2023-11-24T08:04:54.556507+0000","last_became_peered":"2023-11-24T08:04:54.556507+0000","last_unstale":"2023-11-24T08:04:57.002164+0000","last_undegraded":"2023-11-24T08:04:57.002164+0000","last_fullsized":"2023-11-24T08:04:57.002164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:23.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008934+0000","last_change":"2023-11-24T08:04:54.544572+0000","last_active":"2023-11-24T08:04:57.008934+0000","last_peered":"2023-11-24T08:04:57.008934+0000","last_clean":"2023-11-24T08:04:57.008934+0000","last_became_active":"2023-11-24T08:04:54.544277+0000","last_became_peered":"2023-11-24T08:04:54.544277+0000","last_unstale":"2023-11-24T08:04:57.008934+0000","last_undegraded":"2023-11-24T08:04:57.008934+0000","last_fullsized":"2023-11-24T08:04:57.008934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006337+0000","last_change":"2023-11-24T08:04:54.549562+0000","last_active":"2023-11-24T08:04:57.006337+0000","last_peered":"2023-11-24T08:04:57.006337+0000","last_clean":"2023-11-24T08:04:57.006337+0000","last_became_active":"2023-11-24T08:04:54.549054+0000","last_became_peered":"2023-11-24T08:04:54.549054+0000","last_unstale":"2023-11-24T08:04:57.006337+0000","last_undegraded":"2023-11-24T08:04:57.006337+0000","last_fullsized":"2023-11-24T08:04:57.006337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111982+0000","last_change":"2023-11-24T08:04:54.554392+0000","last_active":"2023-11-24T08:04:56.111982+0000","last_peered":"2023-11-24T08:04:56.111982+0000","last_clean":"2023-11-24T08:04:56.111982+0000","last_became_active":"2023-11-24T08:04:54.551225+0000","last_became_peered":"2023-11-24T08:04:54.551225+0000","last_unstale":"2023-11-24T08:04:56.111982+0000","last_undegraded":"2023-11-24T08:04:56.111982+0000","last_fullsized":"2023-11-24T08:04:56.111982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005328+0000","last_change":"2023-11-24T08:04:54.545848+0000","last_active":"2023-11-24T08:04:57.005328+0000","last_peered":"2023-11-24T08:04:57.005328+0000","last_clean":"2023-11-24T08:04:57.005328+0000","last_became_active":"2023-11-24T08:04:54.545564+0000","last_became_peered":"2023-11-24T08:04:54.545564+0000","last_unstale":"2023-11-24T08:04:57.005328+0000","last_undegraded":"2023-11-24T08:04:57.005328+0000","last_fullsized":"2023-11-24T08:04:57.005328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697541+0000","last_change":"2023-11-24T08:04:54.566061+0000","last_active":"2023-11-24T08:04:55.697541+0000","last_peered":"2023-11-24T08:04:55.697541+0000","last_clean":"2023-11-24T08:04:55.697541+0000","last_became_active":"2023-11-24T08:04:54.563794+0000","last_became_peered":"2023-11-24T08:04:54.563794+0000","last_unstale":"2023-11-24T08:04:55.697541+0000","last_undegraded":"2023-11-24T08:04:55.697541+0000","last_fullsized":"2023-11-24T08:04:55.697541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006549+0000","last_change":"2023-11-24T08:04:54.549393+0000","last_active":"2023-11-24T08:04:57.006549+0000","last_peered":"2023-11-24T08:04:57.006549+0000","last_clean":"2023-11-24T08:04:57.006549+0000","last_became_active":"2023-11-24T08:04:54.549057+0000","last_became_peered":"2023-11-24T08:04:54.549057+0000","last_unstale":"2023-11-24T08:04:57.006549+0000","last_undegraded":"2023-11-24T08:04:57.006549+0000","last_fullsized":"2023-11-24T08:04:57.006549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698534+0000","last_change":"2023-11-24T08:04:54.541713+0000","last_active":"2023-11-24T08:04:55.698534+0000","last_peered":"2023-11-24T08:04:55.698534+0000","last_clean":"2023-11-24T08:04:55.698534+0000","last_became_active":"2023-11-24T08:04:54.541443+0000","last_became_peered":"2023-11-24T08:04:54.541443+0000","last_unstale":"2023-11-24T08:04:55.698534+0000","last_undegraded":"2023-11-24T08:04:55.698534+0000","last_fullsized":"2023-11-24T08:04:55.698534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007090+0000","last_change":"2023-11-24T08:04:54.544917+0000","last_active":"2023-11-24T08:04:57.007090+0000","last_peered":"2023-11-24T08:04:57.007090+0000","last_clean":"2023-11-24T08:04:57.007090+0000","last_became_active":"2023-11-24T08:04:54.544595+0000","last_became_peered":"2023-11-24T08:04:54.544595+0000","last_unstale":"2023-11-24T08:04:57.007090+0000","last_undegraded":"2023-11-24T08:04:57.007090+0000","last_fullsized":"2023-11-24T08:04:57.007090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536471+0000","last_change":"2023-11-24T08:04:54.558219+0000","last_active":"2023-11-24T08:04:55.536471+0000","last_peered":"2023-11-24T08:04:55.536471+0000","last_clean":"2023-11-24T08:04:55.536471+0000","last_became_active":"2023-11-24T08:04:54.557796+0000","last_became_peered":"2023-11-24T08:04:54.557796+0000","last_unstale":"2023-11-24T08:04:55.536471+0000","last_undegraded":"2023-11-24T08:04:55.536471+0000","last_fullsized":"2023-11-24T08:04:55.536471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698163+0000","last_change":"2023-11-24T08:04:54.561235+0000","last_active":"2023-11-24T08:04:55.698163+0000","last_peered":"2023-11-24T08:04:55.698163+0000","last_clean":"2023-11-24T08:04:55.698163+0000","last_became_active":"2023-11-24T08:04:54.560941+0000","last_became_peered":"2023-11-24T08:04:54.560941+0000","last_unstale":"2023-11-24T08:04:55.698163+0000","last_undegraded":"2023-11-24T08:04:55.698163+0000","last_fullsized":"2023-11-24T08:04:55.698163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112074+0000","last_change":"2023-11-24T08:04:54.553956+0000","last_active":"2023-11-24T08:04:56.112074+0000","last_peered":"2023-11-24T08:04:56.112074+0000","last_clean":"2023-11-24T08:04:56.112074+0000","last_became_active":"2023-11-24T08:04:54.553685+0000","last_became_peered":"2023-11-24T08:04:54.553685+0000","last_unstale":"2023-11-24T08:04:56.112074+0000","last_undegraded":"2023-11-24T08:04:56.112074+0000","last_fullsized":"2023-11-24T08:04:56.112074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005534+0000","last_change":"2023-11-24T08:04:54.560091+0000","last_active":"2023-11-24T08:04:57.005534+0000","last_peered":"2023-11-24T08:04:57.005534+0000","last_clean":"2023-11-24T08:04:57.005534+0000","last_became_active":"2023-11-24T08:04:54.559784+0000","last_became_peered":"2023-11-24T08:04:54.559784+0000","last_unstale":"2023-11-24T08:04:57.005534+0000","last_undegraded":"2023-11-24T08:04:57.005534+0000","last_fullsized":"2023-11-24T08:04:57.005534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002234+0000","last_change":"2023-11-24T08:04:54.542929+0000","last_active":"2023-11-24T08:04:57.002234+0000","last_peered":"2023-11-24T08:04:57.002234+0000","last_clean":"2023-11-24T08:04:57.002234+0000","last_became_active":"2023-11-24T08:04:54.542574+0000","last_became_peered":"2023-11-24T08:04:54.542574+0000","last_unstale":"2023-11-24T08:04:57.002234+0000","last_undegraded":"2023-11-24T08:04:57.002234+0000","last_fullsized":"2023-11-24T08:04:57.002234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:51.449998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.004781+0000","last_change":"2023-11-24T08:04:54.547198+0000","last_active":"2023-11-24T08:04:57.004781+0000","last_peered":"2023-11-24T08:04:57.004781+0000","last_clean":"2023-11-24T08:04:57.004781+0000","last_became_active":"2023-11-24T08:04:54.546936+0000","last_became_peered":"2023-11-24T08:04:54.546936+0000","last_unstale":"2023-11-24T08:04:57.004781+0000","last_undegraded":"2023-11-24T08:04:57.004781+0000","last_fullsized":"2023-11-24T08:04:57.004781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537709+0000","last_change":"2023-11-24T08:04:54.542121+0000","last_active":"2023-11-24T08:04:55.537709+0000","last_peered":"2023-11-24T08:04:55.537709+0000","last_clean":"2023-11-24T08:04:55.537709+0000","last_became_active":"2023-11-24T08:04:54.541708+0000","last_became_peered":"2023-11-24T08:04:54.541708+0000","last_unstale":"2023-11-24T08:04:55.537709+0000","last_undegraded":"2023-11-24T08:04:55.537709+0000","last_fullsized":"2023-11-24T08:04:55.537709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:41.750215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563001+0000","last_change":"2023-11-24T08:04:53.530633+0000","last_active":"2023-11-24T08:04:54.563001+0000","last_peered":"2023-11-24T08:04:54.563001+0000","last_clean":"2023-11-24T08:04:54.563001+0000","last_became_active":"2023-11-24T08:04:53.530234+0000","last_became_peered":"2023-11-24T08:04:53.530234+0000","last_unstale":"2023-11-24T08:04:54.563001+0000","last_undegraded":"2023-11-24T08:04:54.563001+0000","last_fullsized":"2023-11-24T08:04:54.563001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112344+0000","last_change":"2023-11-24T08:04:54.563412+0000","last_active":"2023-11-24T08:04:56.112344+0000","last_peered":"2023-11-24T08:04:56.112344+0000","last_clean":"2023-11-24T08:04:56.112344+0000","last_became_active":"2023-11-24T08:04:54.561262+0000","last_became_peered":"2023-11-24T08:04:54.561262+0000","last_unstale":"2023-11-24T08:04:56.112344+0000","last_undegraded":"2023-11-24T08:04:56.112344+0000","last_fullsized":"2023-11-24T08:04:56.112344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.560535+0000","last_change":"2023-11-24T08:04:53.527156+0000","last_active":"2023-11-24T08:04:54.560535+0000","last_peered":"2023-11-24T08:04:54.560535+0000","last_clean":"2023-11-24T08:04:54.560535+0000","last_became_active":"2023-11-24T08:04:53.525686+0000","last_became_peered":"2023-11-24T08:04:53.525686+0000","last_unstale":"2023-11-24T08:04:54.560535+0000","last_undegraded":"2023-11-24T08:04:54.560535+0000","last_fullsized":"2023-11-24T08:04:54.560535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537878+0000","last_change":"2023-11-24T08:04:53.521369+0000","last_active":"2023-11-24T08:04:54.537878+0000","last_peered":"2023-11-24T08:04:54.537878+0000","last_clean":"2023-11-24T08:04:54.537878+0000","last_became_active":"2023-11-24T08:04:53.520852+0000","last_became_peered":"2023-11-24T08:04:53.520852+0000","last_unstale":"2023-11-24T08:04:54.537878+0000","last_undegraded":"2023-11-24T08:04:54.537878+0000","last_fullsized":"2023-11-24T08:04:54.537878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697652+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.697652+0000","last_peered":"2023-11-24T08:04:55.697652+0000","last_clean":"2023-11-24T08:04:55.697652+0000","last_became_active":"2023-11-24T08:04:54.561649+0000","last_became_peered":"2023-11-24T08:04:54.561649+0000","last_unstale":"2023-11-24T08:04:55.697652+0000","last_undegraded":"2023-11-24T08:04:55.697652+0000","last_fullsized":"2023-11-24T08:04:55.697652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.517832+0000","last_change":"2023-11-24T08:04:53.517986+0000","last_active":"2023-11-24T08:04:54.517832+0000","last_peered":"2023-11-24T08:04:54.517832+0000","last_clean":"2023-11-24T08:04:54.517832+0000","last_became_active":"2023-11-24T08:04:53.517686+0000","last_became_peered":"2023-11-24T08:04:53.517686+0000","last_unstale":"2023-11-24T08:04:54.517832+0000","last_undegraded":"2023-11-24T08:04:54.517832+0000","last_fullsized":"2023-11-24T08:04:54.517832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535862+0000","last_change":"2023-11-24T08:04:54.548962+0000","last_active":"2023-11-24T08:04:55.535862+0000","last_peered":"2023-11-24T08:04:55.535862+0000","last_clean":"2023-11-24T08:04:55.535862+0000","last_became_active":"2023-11-24T08:04:54.548697+0000","last_became_peered":"2023-11-24T08:04:54.548697+0000","last_unstale":"2023-11-24T08:04:55.535862+0000","last_undegraded":"2023-11-24T08:04:55.535862+0000","last_fullsized":"2023-11-24T08:04:55.535862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.533360+0000","last_change":"2023-11-24T08:04:53.529719+0000","last_active":"2023-11-24T08:04:54.533360+0000","last_peered":"2023-11-24T08:04:54.533360+0000","last_clean":"2023-11-24T08:04:54.533360+0000","last_became_active":"2023-11-24T08:04:53.529289+0000","last_became_peered":"2023-11-24T08:04:53.529289+0000","last_unstale":"2023-11-24T08:04:54.533360+0000","last_undegraded":"2023-11-24T08:04:54.533360+0000","last_fullsized":"2023-11-24T08:04:54.533360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005952+0000","last_change":"2023-11-24T08:04:54.546903+0000","last_active":"2023-11-24T08:04:57.005952+0000","last_peered":"2023-11-24T08:04:57.005952+0000","last_clean":"2023-11-24T08:04:57.005952+0000","last_became_active":"2023-11-24T08:04:54.545578+0000","last_became_peered":"2023-11-24T08:04:54.545578+0000","last_unstale":"2023-11-24T08:04:57.005952+0000","last_undegraded":"2023-11-24T08:04:57.005952+0000","last_fullsized":"2023-11-24T08:04:57.005952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521698+0000","last_change":"2023-11-24T08:04:53.533034+0000","last_active":"2023-11-24T08:04:54.521698+0000","last_peered":"2023-11-24T08:04:54.521698+0000","last_clean":"2023-11-24T08:04:54.521698+0000","last_became_active":"2023-11-24T08:04:53.532393+0000","last_became_peered":"2023-11-24T08:04:53.532393+0000","last_unstale":"2023-11-24T08:04:54.521698+0000","last_undegraded":"2023-11-24T08:04:54.521698+0000","last_fullsized":"2023-11-24T08:04:54.521698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112283+0000","last_change":"2023-11-24T08:04:54.547926+0000","last_active":"2023-11-24T08:04:56.112283+0000","last_peered":"2023-11-24T08:04:56.112283+0000","last_clean":"2023-11-24T08:04:56.112283+0000","last_became_active":"2023-11-24T08:04:54.542598+0000","last_became_peered":"2023-11-24T08:04:54.542598+0000","last_unstale":"2023-11-24T08:04:56.112283+0000","last_undegraded":"2023-11-24T08:04:56.112283+0000","last_fullsized":"2023-11-24T08:04:56.112283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565554+0000","last_change":"2023-11-24T08:04:53.525910+0000","last_active":"2023-11-24T08:04:54.565554+0000","last_peered":"2023-11-24T08:04:54.565554+0000","last_clean":"2023-11-24T08:04:54.565554+0000","last_became_active":"2023-11-24T08:04:53.525661+0000","last_became_peered":"2023-11-24T08:04:53.525661+0000","last_unstale":"2023-11-24T08:04:54.565554+0000","last_undegraded":"2023-11-24T08:04:54.565554+0000","last_fullsized":"2023-11-24T08:04:54.565554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538064+0000","last_change":"2023-11-24T08:04:54.545080+0000","last_active":"2023-11-24T08:04:55.538064+0000","last_peered":"2023-11-24T08:04:55.538064+0000","last_clean":"2023-11-24T08:04:55.538064+0000","last_became_active":"2023-11-24T08:04:54.544031+0000","last_became_peered":"2023-11-24T08:04:54.544031+0000","last_unstale":"2023-11-24T08:04:55.538064+0000","last_undegraded":"2023-11-24T08:04:55.538064+0000","last_fullsized":"2023-11-24T08:04:55.538064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:47.133354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571132+0000","last_change":"2023-11-24T08:04:53.528212+0000","last_active":"2023-11-24T08:04:54.571132+0000","last_peered":"2023-11-24T08:04:54.571132+0000","last_clean":"2023-11-24T08:04:54.571132+0000","last_became_active":"2023-11-24T08:04:53.527458+0000","last_became_peered":"2023-11-24T08:04:53.527458+0000","last_unstale":"2023-11-24T08:04:54.571132+0000","last_undegraded":"2023-11-24T08:04:54.571132+0000","last_fullsized":"2023-11-24T08:04:54.571132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007910+0000","last_change":"2023-11-24T08:04:54.559327+0000","last_active":"2023-11-24T08:04:57.007910+0000","last_peered":"2023-11-24T08:04:57.007910+0000","last_clean":"2023-11-24T08:04:57.007910+0000","last_became_active":"2023-11-24T08:04:54.559034+0000","last_became_peered":"2023-11-24T08:04:54.559034+0000","last_unstale":"2023-11-24T08:04:57.007910+0000","last_undegraded":"2023-11-24T08:04:57.007910+0000","last_fullsized":"2023-11-24T08:04:57.007910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571070+0000","last_change":"2023-11-24T08:04:53.514829+0000","last_active":"2023-11-24T08:04:54.571070+0000","last_peered":"2023-11-24T08:04:54.571070+0000","last_clean":"2023-11-24T08:04:54.571070+0000","last_became_active":"2023-11-24T08:04:53.514590+0000","last_became_peered":"2023-11-24T08:04:53.514590+0000","last_unstale":"2023-11-24T08:04:54.571070+0000","last_undegraded":"2023-11-24T08:04:54.571070+0000","last_fullsized":"2023-11-24T08:04:54.571070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111807+0000","last_change":"2023-11-24T08:04:54.554473+0000","last_active":"2023-11-24T08:04:56.111807+0000","last_peered":"2023-11-24T08:04:56.111807+0000","last_clean":"2023-11-24T08:04:56.111807+0000","last_became_active":"2023-11-24T08:04:54.554157+0000","last_became_peered":"2023-11-24T08:04:54.554157+0000","last_unstale":"2023-11-24T08:04:56.111807+0000","last_undegraded":"2023-11-24T08:04:56.111807+0000","last_fullsized":"2023-11-24T08:04:56.111807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:33.329459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.524663+0000","last_change":"2023-11-24T08:04:53.526561+0000","last_active":"2023-11-24T08:04:54.524663+0000","last_peered":"2023-11-24T08:04:54.524663+0000","last_clean":"2023-11-24T08:04:54.524663+0000","last_became_active":"2023-11-24T08:04:53.525788+0000","last_became_peered":"2023-11-24T08:04:53.525788+0000","last_unstale":"2023-11-24T08:04:54.524663+0000","last_undegraded":"2023-11-24T08:04:54.524663+0000","last_fullsized":"2023-11-24T08:04:54.524663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009421+0000","last_change":"2023-11-24T08:04:54.547419+0000","last_active":"2023-11-24T08:04:57.009421+0000","last_peered":"2023-11-24T08:04:57.009421+0000","last_clean":"2023-11-24T08:04:57.009421+0000","last_became_active":"2023-11-24T08:04:54.547136+0000","last_became_peered":"2023-11-24T08:04:54.547136+0000","last_unstale":"2023-11-24T08:04:57.009421+0000","last_undegraded":"2023-11-24T08:04:57.009421+0000","last_fullsized":"2023-11-24T08:04:57.009421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:07:38.219756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.529667+0000","last_change":"2023-11-24T08:04:53.522781+0000","last_active":"2023-11-24T08:04:54.529667+0000","last_peered":"2023-11-24T08:04:54.529667+0000","last_clean":"2023-11-24T08:04:54.529667+0000","last_became_active":"2023-11-24T08:04:53.522413+0000","last_became_peered":"2023-11-24T08:04:53.522413+0000","last_unstale":"2023-11-24T08:04:54.529667+0000","last_undegraded":"2023-11-24T08:04:54.529667+0000","last_fullsized":"2023-11-24T08:04:54.529667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005124+0000","last_change":"2023-11-24T08:04:54.550954+0000","last_active":"2023-11-24T08:04:57.005124+0000","last_peered":"2023-11-24T08:04:57.005124+0000","last_clean":"2023-11-24T08:04:57.005124+0000","last_became_active":"2023-11-24T08:04:54.550405+0000","last_became_peered":"2023-11-24T08:04:54.550405+0000","last_unstale":"2023-11-24T08:04:57.005124+0000","last_undegraded":"2023-11-24T08:04:57.005124+0000","last_fullsized":"2023-11-24T08:04:57.005124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.569402+0000","last_change":"2023-11-24T08:04:53.517811+0000","last_active":"2023-11-24T08:04:54.569402+0000","last_peered":"2023-11-24T08:04:54.569402+0000","last_clean":"2023-11-24T08:04:54.569402+0000","last_became_active":"2023-11-24T08:04:53.517449+0000","last_became_peered":"2023-11-24T08:04:53.517449+0000","last_unstale":"2023-11-24T08:04:54.569402+0000","last_undegraded":"2023-11-24T08:04:54.569402+0000","last_fullsized":"2023-11-24T08:04:54.569402+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:17.696987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008505+0000","last_change":"2023-11-24T08:04:54.561976+0000","last_active":"2023-11-24T08:04:57.008505+0000","last_peered":"2023-11-24T08:04:57.008505+0000","last_clean":"2023-11-24T08:04:57.008505+0000","last_became_active":"2023-11-24T08:04:54.561433+0000","last_became_peered":"2023-11-24T08:04:54.561433+0000","last_unstale":"2023-11-24T08:04:57.008505+0000","last_undegraded":"2023-11-24T08:04:57.008505+0000","last_fullsized":"2023-11-24T08:04:57.008505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571680+0000","last_change":"2023-11-24T08:04:53.512021+0000","last_active":"2023-11-24T08:04:54.571680+0000","last_peered":"2023-11-24T08:04:54.571680+0000","last_clean":"2023-11-24T08:04:54.571680+0000","last_became_active":"2023-11-24T08:04:53.511682+0000","last_became_peered":"2023-11-24T08:04:53.511682+0000","last_unstale":"2023-11-24T08:04:54.571680+0000","last_undegraded":"2023-11-24T08:04:54.571680+0000","last_fullsized":"2023-11-24T08:04:54.571680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007859+0000","last_change":"2023-11-24T08:04:54.561544+0000","last_active":"2023-11-24T08:04:57.007859+0000","last_peered":"2023-11-24T08:04:57.007859+0000","last_clean":"2023-11-24T08:04:57.007859+0000","last_became_active":"2023-11-24T08:04:54.561157+0000","last_became_peered":"2023-11-24T08:04:54.561157+0000","last_unstale":"2023-11-24T08:04:57.007859+0000","last_undegraded":"2023-11-24T08:04:57.007859+0000","last_fullsized":"2023-11-24T08:04:57.007859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565958+0000","last_change":"2023-11-24T08:04:53.510122+0000","last_active":"2023-11-24T08:04:54.565958+0000","last_peered":"2023-11-24T08:04:54.565958+0000","last_clean":"2023-11-24T08:04:54.565958+0000","last_became_active":"2023-11-24T08:04:53.509763+0000","last_became_peered":"2023-11-24T08:04:53.509763+0000","last_unstale":"2023-11-24T08:04:54.565958+0000","last_undegraded":"2023-11-24T08:04:54.565958+0000","last_fullsized":"2023-11-24T08:04:54.565958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008573+0000","last_change":"2023-11-24T08:04:54.560061+0000","last_active":"2023-11-24T08:04:57.008573+0000","last_peered":"2023-11-24T08:04:57.008573+0000","last_clean":"2023-11-24T08:04:57.008573+0000","last_became_active":"2023-11-24T08:04:54.559752+0000","last_became_peered":"2023-11-24T08:04:54.559752+0000","last_unstale":"2023-11-24T08:04:57.008573+0000","last_undegraded":"2023-11-24T08:04:57.008573+0000","last_fullsized":"2023-11-24T08:04:57.008573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565408+0000","last_change":"2023-11-24T08:04:53.522019+0000","last_active":"2023-11-24T08:04:54.565408+0000","last_peered":"2023-11-24T08:04:54.565408+0000","last_clean":"2023-11-24T08:04:54.565408+0000","last_became_active":"2023-11-24T08:04:53.521742+0000","last_became_peered":"2023-11-24T08:04:53.521742+0000","last_unstale":"2023-11-24T08:04:54.565408+0000","last_undegraded":"2023-11-24T08:04:54.565408+0000","last_fullsized":"2023-11-24T08:04:54.565408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005948+0000","last_change":"2023-11-24T08:04:54.563074+0000","last_active":"2023-11-24T08:04:57.005948+0000","last_peered":"2023-11-24T08:04:57.005948+0000","last_clean":"2023-11-24T08:04:57.005948+0000","last_became_active":"2023-11-24T08:04:54.562794+0000","last_became_peered":"2023-11-24T08:04:54.562794+0000","last_unstale":"2023-11-24T08:04:57.005948+0000","last_undegraded":"2023-11-24T08:04:57.005948+0000","last_fullsized":"2023-11-24T08:04:57.005948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.540207+0000","last_change":"2023-11-24T08:04:53.532053+0000","last_active":"2023-11-24T08:04:54.540207+0000","last_peered":"2023-11-24T08:04:54.540207+0000","last_clean":"2023-11-24T08:04:54.540207+0000","last_became_active":"2023-11-24T08:04:53.531648+0000","last_became_peered":"2023-11-24T08:04:53.531648+0000","last_unstale":"2023-11-24T08:04:54.540207+0000","last_undegraded":"2023-11-24T08:04:54.540207+0000","last_fullsized":"2023-11-24T08:04:54.540207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008303+0000","last_change":"2023-11-24T08:04:54.556574+0000","last_active":"2023-11-24T08:04:57.008303+0000","last_peered":"2023-11-24T08:04:57.008303+0000","last_clean":"2023-11-24T08:04:57.008303+0000","last_became_active":"2023-11-24T08:04:54.555243+0000","last_became_peered":"2023-11-24T08:04:54.555243+0000","last_unstale":"2023-11-24T08:04:57.008303+0000","last_undegraded":"2023-11-24T08:04:57.008303+0000","last_fullsized":"2023-11-24T08:04:57.008303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.531104+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:04:54.531104+0000","last_peered":"2023-11-24T08:04:54.531104+0000","last_clean":"2023-11-24T08:04:54.531104+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:04:54.531104+0000","last_undegraded":"2023-11-24T08:04:54.531104+0000","last_fullsized":"2023-11-24T08:04:54.531104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521262+0000","last_change":"2023-11-24T08:04:53.524229+0000","last_active":"2023-11-24T08:04:54.521262+0000","last_peered":"2023-11-24T08:04:54.521262+0000","last_clean":"2023-11-24T08:04:54.521262+0000","last_became_active":"2023-11-24T08:04:53.523839+0000","last_became_peered":"2023-11-24T08:04:53.523839+0000","last_unstale":"2023-11-24T08:04:54.521262+0000","last_undegraded":"2023-11-24T08:04:54.521262+0000","last_fullsized":"2023-11-24T08:04:54.521262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698503+0000","last_change":"2023-11-24T08:04:54.549976+0000","last_active":"2023-11-24T08:04:55.698503+0000","last_peered":"2023-11-24T08:04:55.698503+0000","last_clean":"2023-11-24T08:04:55.698503+0000","last_became_active":"2023-11-24T08:04:54.549680+0000","last_became_peered":"2023-11-24T08:04:54.549680+0000","last_unstale":"2023-11-24T08:04:55.698503+0000","last_undegraded":"2023-11-24T08:04:55.698503+0000","last_fullsized":"2023-11-24T08:04:55.698503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.536596+0000","last_change":"2023-11-24T08:04:53.528808+0000","last_active":"2023-11-24T08:04:54.536596+0000","last_peered":"2023-11-24T08:04:54.536596+0000","last_clean":"2023-11-24T08:04:54.536596+0000","last_became_active":"2023-11-24T08:04:53.528482+0000","last_became_peered":"2023-11-24T08:04:53.528482+0000","last_unstale":"2023-11-24T08:04:54.536596+0000","last_undegraded":"2023-11-24T08:04:54.536596+0000","last_fullsized":"2023-11-24T08:04:54.536596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111534+0000","last_change":"2023-11-24T08:04:54.543382+0000","last_active":"2023-11-24T08:04:56.111534+0000","last_peered":"2023-11-24T08:04:56.111534+0000","last_clean":"2023-11-24T08:04:56.111534+0000","last_became_active":"2023-11-24T08:04:54.541869+0000","last_became_peered":"2023-11-24T08:04:54.541869+0000","last_unstale":"2023-11-24T08:04:56.111534+0000","last_undegraded":"2023-11-24T08:04:56.111534+0000","last_fullsized":"2023-11-24T08:04:56.111534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:43.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.538981+0000","last_change":"2023-11-24T08:04:53.509173+0000","last_active":"2023-11-24T08:04:54.538981+0000","last_peered":"2023-11-24T08:04:54.538981+0000","last_clean":"2023-11-24T08:04:54.538981+0000","last_became_active":"2023-11-24T08:04:53.507954+0000","last_became_peered":"2023-11-24T08:04:53.507954+0000","last_unstale":"2023-11-24T08:04:54.538981+0000","last_undegraded":"2023-11-24T08:04:54.538981+0000","last_fullsized":"2023-11-24T08:04:54.538981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535690+0000","last_change":"2023-11-24T08:04:54.551554+0000","last_active":"2023-11-24T08:04:55.535690+0000","last_peered":"2023-11-24T08:04:55.535690+0000","last_clean":"2023-11-24T08:04:55.535690+0000","last_became_active":"2023-11-24T08:04:54.546942+0000","last_became_peered":"2023-11-24T08:04:54.546942+0000","last_unstale":"2023-11-24T08:04:55.535690+0000","last_undegraded":"2023-11-24T08:04:55.535690+0000","last_fullsized":"2023-11-24T08:04:55.535690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.525076+0000","last_change":"2023-11-24T08:04:53.526686+0000","last_active":"2023-11-24T08:04:54.525076+0000","last_peered":"2023-11-24T08:04:54.525076+0000","last_clean":"2023-11-24T08:04:54.525076+0000","last_became_active":"2023-11-24T08:04:53.526157+0000","last_became_peered":"2023-11-24T08:04:53.526157+0000","last_unstale":"2023-11-24T08:04:54.525076+0000","last_undegraded":"2023-11-24T08:04:54.525076+0000","last_fullsized":"2023-11-24T08:04:54.525076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:38.092773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537976+0000","last_change":"2023-11-24T08:04:54.556189+0000","last_active":"2023-11-24T08:04:55.537976+0000","last_peered":"2023-11-24T08:04:55.537976+0000","last_clean":"2023-11-24T08:04:55.537976+0000","last_became_active":"2023-11-24T08:04:54.555903+0000","last_became_peered":"2023-11-24T08:04:54.555903+0000","last_unstale":"2023-11-24T08:04:55.537976+0000","last_undegraded":"2023-11-24T08:04:55.537976+0000","last_fullsized":"2023-11-24T08:04:55.537976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:18.709743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532442+0000","last_change":"2023-11-24T08:04:53.516220+0000","last_active":"2023-11-24T08:04:54.532442+0000","last_peered":"2023-11-24T08:04:54.532442+0000","last_clean":"2023-11-24T08:04:54.532442+0000","last_became_active":"2023-11-24T08:04:53.515013+0000","last_became_peered":"2023-11-24T08:04:53.515013+0000","last_unstale":"2023-11-24T08:04:54.532442+0000","last_undegraded":"2023-11-24T08:04:54.532442+0000","last_fullsized":"2023-11-24T08:04:54.532442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006151+0000","last_change":"2023-11-24T08:04:54.563230+0000","last_active":"2023-11-24T08:04:57.006151+0000","last_peered":"2023-11-24T08:04:57.006151+0000","last_clean":"2023-11-24T08:04:57.006151+0000","last_became_active":"2023-11-24T08:04:54.562908+0000","last_became_peered":"2023-11-24T08:04:54.562908+0000","last_unstale":"2023-11-24T08:04:57.006151+0000","last_undegraded":"2023-11-24T08:04:57.006151+0000","last_fullsized":"2023-11-24T08:04:57.006151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537608+0000","last_change":"2023-11-24T08:04:53.529958+0000","last_active":"2023-11-24T08:04:54.537608+0000","last_peered":"2023-11-24T08:04:54.537608+0000","last_clean":"2023-11-24T08:04:54.537608+0000","last_became_active":"2023-11-24T08:04:53.529676+0000","last_became_peered":"2023-11-24T08:04:53.529676+0000","last_unstale":"2023-11-24T08:04:54.537608+0000","last_undegraded":"2023-11-24T08:04:54.537608+0000","last_fullsized":"2023-11-24T08:04:54.537608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535951+0000","last_change":"2023-11-24T08:04:54.545570+0000","last_active":"2023-11-24T08:04:55.535951+0000","last_peered":"2023-11-24T08:04:55.535951+0000","last_clean":"2023-11-24T08:04:55.535951+0000","last_became_active":"2023-11-24T08:04:54.545282+0000","last_became_peered":"2023-11-24T08:04:54.545282+0000","last_unstale":"2023-11-24T08:04:55.535951+0000","last_undegraded":"2023-11-24T08:04:55.535951+0000","last_fullsized":"2023-11-24T08:04:55.535951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.528509+0000","last_change":"2023-11-24T08:04:53.528071+0000","last_active":"2023-11-24T08:04:54.528509+0000","last_peered":"2023-11-24T08:04:54.528509+0000","last_clean":"2023-11-24T08:04:54.528509+0000","last_became_active":"2023-11-24T08:04:53.527761+0000","last_became_peered":"2023-11-24T08:04:53.527761+0000","last_unstale":"2023-11-24T08:04:54.528509+0000","last_undegraded":"2023-11-24T08:04:54.528509+0000","last_fullsized":"2023-11-24T08:04:54.528509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698850+0000","last_change":"2023-11-24T08:04:54.548899+0000","last_active":"2023-11-24T08:04:55.698850+0000","last_peered":"2023-11-24T08:04:55.698850+0000","last_clean":"2023-11-24T08:04:55.698850+0000","last_became_active":"2023-11-24T08:04:54.548630+0000","last_became_peered":"2023-11-24T08:04:54.548630+0000","last_unstale":"2023-11-24T08:04:55.698850+0000","last_undegraded":"2023-11-24T08:04:55.698850+0000","last_fullsized":"2023-11-24T08:04:55.698850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561594+0000","last_change":"2023-11-24T08:04:53.531179+0000","last_active":"2023-11-24T08:04:54.561594+0000","last_peered":"2023-11-24T08:04:54.561594+0000","last_clean":"2023-11-24T08:04:54.561594+0000","last_became_active":"2023-11-24T08:04:53.530771+0000","last_became_peered":"2023-11-24T08:04:53.530771+0000","last_unstale":"2023-11-24T08:04:54.561594+0000","last_undegraded":"2023-11-24T08:04:54.561594+0000","last_fullsized":"2023-11-24T08:04:54.561594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112025+0000","last_change":"2023-11-24T08:04:54.558549+0000","last_active":"2023-11-24T08:04:56.112025+0000","last_peered":"2023-11-24T08:04:56.112025+0000","last_clean":"2023-11-24T08:04:56.112025+0000","last_became_active":"2023-11-24T08:04:54.552869+0000","last_became_peered":"2023-11-24T08:04:54.552869+0000","last_unstale":"2023-11-24T08:04:56.112025+0000","last_undegraded":"2023-11-24T08:04:56.112025+0000","last_fullsized":"2023-11-24T08:04:56.112025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.530169+0000","last_change":"2023-11-24T08:04:53.530321+0000","last_active":"2023-11-24T08:04:54.530169+0000","last_peered":"2023-11-24T08:04:54.530169+0000","last_clean":"2023-11-24T08:04:54.530169+0000","last_became_active":"2023-11-24T08:04:53.529932+0000","last_became_peered":"2023-11-24T08:04:53.529932+0000","last_unstale":"2023-11-24T08:04:54.530169+0000","last_undegraded":"2023-11-24T08:04:54.530169+0000","last_fullsized":"2023-11-24T08:04:54.530169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537704+0000","last_change":"2023-11-24T08:04:54.553398+0000","last_active":"2023-11-24T08:04:55.537704+0000","last_peered":"2023-11-24T08:04:55.537704+0000","last_clean":"2023-11-24T08:04:55.537704+0000","last_became_active":"2023-11-24T08:04:54.552951+0000","last_became_peered":"2023-11-24T08:04:54.552951+0000","last_unstale":"2023-11-24T08:04:55.537704+0000","last_undegraded":"2023-11-24T08:04:55.537704+0000","last_fullsized":"2023-11-24T08:04:55.537704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:38.870187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.526579+0000","last_change":"2023-11-24T08:04:53.531280+0000","last_active":"2023-11-24T08:04:54.526579+0000","last_peered":"2023-11-24T08:04:54.526579+0000","last_clean":"2023-11-24T08:04:54.526579+0000","last_became_active":"2023-11-24T08:04:53.531012+0000","last_became_peered":"2023-11-24T08:04:53.531012+0000","last_unstale":"2023-11-24T08:04:54.526579+0000","last_undegraded":"2023-11-24T08:04:54.526579+0000","last_fullsized":"2023-11-24T08:04:54.526579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697931+0000","last_change":"2023-11-24T08:04:54.556299+0000","last_active":"2023-11-24T08:04:55.697931+0000","last_peered":"2023-11-24T08:04:55.697931+0000","last_clean":"2023-11-24T08:04:55.697931+0000","last_became_active":"2023-11-24T08:04:54.555882+0000","last_became_peered":"2023-11-24T08:04:54.555882+0000","last_unstale":"2023-11-24T08:04:55.697931+0000","last_undegraded":"2023-11-24T08:04:55.697931+0000","last_fullsized":"2023-11-24T08:04:55.697931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532819+0000","last_change":"2023-11-24T08:04:53.517295+0000","last_active":"2023-11-24T08:04:54.532819+0000","last_peered":"2023-11-24T08:04:54.532819+0000","last_clean":"2023-11-24T08:04:54.532819+0000","last_became_active":"2023-11-24T08:04:53.517017+0000","last_became_peered":"2023-11-24T08:04:53.517017+0000","last_unstale":"2023-11-24T08:04:54.532819+0000","last_undegraded":"2023-11-24T08:04:54.532819+0000","last_fullsized":"2023-11-24T08:04:54.532819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698757+0000","last_change":"2023-11-24T08:04:54.563699+0000","last_active":"2023-11-24T08:04:55.698757+0000","last_peered":"2023-11-24T08:04:55.698757+0000","last_clean":"2023-11-24T08:04:55.698757+0000","last_became_active":"2023-11-24T08:04:54.563266+0000","last_became_peered":"2023-11-24T08:04:54.563266+0000","last_unstale":"2023-11-24T08:04:55.698757+0000","last_undegraded":"2023-11-24T08:04:55.698757+0000","last_fullsized":"2023-11-24T08:04:55.698757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563981+0000","last_change":"2023-11-24T08:04:53.529030+0000","last_active":"2023-11-24T08:04:54.563981+0000","last_peered":"2023-11-24T08:04:54.563981+0000","last_clean":"2023-11-24T08:04:54.563981+0000","last_became_active":"2023-11-24T08:04:53.528620+0000","last_became_peered":"2023-11-24T08:04:53.528620+0000","last_unstale":"2023-11-24T08:04:54.563981+0000","last_undegraded":"2023-11-24T08:04:54.563981+0000","last_fullsized":"2023-11-24T08:04:54.563981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006725+0000","last_change":"2023-11-24T08:04:54.544336+0000","last_active":"2023-11-24T08:04:57.006725+0000","last_peered":"2023-11-24T08:04:57.006725+0000","last_clean":"2023-11-24T08:04:57.006725+0000","last_became_active":"2023-11-24T08:04:54.544020+0000","last_became_peered":"2023-11-24T08:04:54.544020+0000","last_unstale":"2023-11-24T08:04:57.006725+0000","last_undegraded":"2023-11-24T08:04:57.006725+0000","last_fullsized":"2023-11-24T08:04:57.006725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005795+0000","last_change":"2023-11-24T08:04:54.540666+0000","last_active":"2023-11-24T08:04:57.005795+0000","last_peered":"2023-11-24T08:04:57.005795+0000","last_clean":"2023-11-24T08:04:57.005795+0000","last_became_active":"2023-11-24T08:04:54.540347+0000","last_became_peered":"2023-11-24T08:04:54.540347+0000","last_unstale":"2023-11-24T08:04:57.005795+0000","last_undegraded":"2023-11-24T08:04:57.005795+0000","last_fullsized":"2023-11-24T08:04:57.005795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.559386+0000","last_change":"2023-11-24T08:04:53.533242+0000","last_active":"2023-11-24T08:04:54.559386+0000","last_peered":"2023-11-24T08:04:54.559386+0000","last_clean":"2023-11-24T08:04:54.559386+0000","last_became_active":"2023-11-24T08:04:53.532947+0000","last_became_peered":"2023-11-24T08:04:53.532947+0000","last_unstale":"2023-11-24T08:04:54.559386+0000","last_undegraded":"2023-11-24T08:04:54.559386+0000","last_fullsized":"2023-11-24T08:04:54.559386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:21.178905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005623+0000","last_change":"2023-11-24T08:04:54.551297+0000","last_active":"2023-11-24T08:04:57.005623+0000","last_peered":"2023-11-24T08:04:57.005623+0000","last_clean":"2023-11-24T08:04:57.005623+0000","last_became_active":"2023-11-24T08:04:54.548106+0000","last_became_peered":"2023-11-24T08:04:54.548106+0000","last_unstale":"2023-11-24T08:04:57.005623+0000","last_undegraded":"2023-11-24T08:04:57.005623+0000","last_fullsized":"2023-11-24T08:04:57.005623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561664+0000","last_change":"2023-11-24T08:04:53.528831+0000","last_active":"2023-11-24T08:04:54.561664+0000","last_peered":"2023-11-24T08:04:54.561664+0000","last_clean":"2023-11-24T08:04:54.561664+0000","last_became_active":"2023-11-24T08:04:53.528464+0000","last_became_peered":"2023-11-24T08:04:53.528464+0000","last_unstale":"2023-11-24T08:04:54.561664+0000","last_undegraded":"2023-11-24T08:04:54.561664+0000","last_fullsized":"2023-11-24T08:04:54.561664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:46.075298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112077+0000","last_change":"2023-11-24T08:04:54.551199+0000","last_active":"2023-11-24T08:04:56.112077+0000","last_peered":"2023-11-24T08:04:56.112077+0000","last_clean":"2023-11-24T08:04:56.112077+0000","last_became_active":"2023-11-24T08:04:54.548261+0000","last_became_peered":"2023-11-24T08:04:54.548261+0000","last_unstale":"2023-11-24T08:04:56.112077+0000","last_undegraded":"2023-11-24T08:04:56.112077+0000","last_fullsized":"2023-11-24T08:04:56.112077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.567550+0000","last_change":"2023-11-24T08:04:53.512889+0000","last_active":"2023-11-24T08:04:54.567550+0000","last_peered":"2023-11-24T08:04:54.567550+0000","last_clean":"2023-11-24T08:04:54.567550+0000","last_became_active":"2023-11-24T08:04:53.512580+0000","last_became_peered":"2023-11-24T08:04:53.512580+0000","last_unstale":"2023-11-24T08:04:54.567550+0000","last_undegraded":"2023-11-24T08:04:54.567550+0000","last_fullsized":"2023-11-24T08:04:54.567550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698368+0000","last_change":"2023-11-24T08:04:54.561224+0000","last_active":"2023-11-24T08:04:55.698368+0000","last_peered":"2023-11-24T08:04:55.698368+0000","last_clean":"2023-11-24T08:04:55.698368+0000","last_became_active":"2023-11-24T08:04:54.558658+0000","last_became_peered":"2023-11-24T08:04:54.558658+0000","last_unstale":"2023-11-24T08:04:55.698368+0000","last_undegraded":"2023-11-24T08:04:55.698368+0000","last_fullsized":"2023-11-24T08:04:55.698368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571974+0000","last_change":"2023-11-24T08:04:53.529967+0000","last_active":"2023-11-24T08:04:54.571974+0000","last_peered":"2023-11-24T08:04:54.571974+0000","last_clean":"2023-11-24T08:04:54.571974+0000","last_became_active":"2023-11-24T08:04:53.529738+0000","last_became_peered":"2023-11-24T08:04:53.529738+0000","last_unstale":"2023-11-24T08:04:54.571974+0000","last_undegraded":"2023-11-24T08:04:54.571974+0000","last_fullsized":"2023-11-24T08:04:54.571974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.819}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.141}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0950000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3439999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.732}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.786}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0670000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2330000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5390000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4369999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.669}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:04:59.629 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:04:59.629 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-24T08:04:59.630 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-24T08:04:59.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-24T08:04:59.630 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-24T08:04:59.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.885+0000 7f0894e94640 1 Processor -- start 2023-11-24T08:04:59.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.886+0000 7f0894e94640 1 -- start start 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0894e94640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f089015c7f0 0x7f089006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0894e94640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08900cece0 con 0x7f08900cf450 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0894e94640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089006d040 con 0x7f08900ce6c0 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0894e94640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08900ceaf0 con 0x7f089015c7f0 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0887fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f08900cf450 0x7f08900cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59074/0 (socket says 172.21.15.107:59074) 2023-11-24T08:04:59.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.887+0000 7f0887fff640 1 -- 172.21.15.107:0/584639283 learned_addr learned my addr 172.21.15.107:0/584639283 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088e59c640 1 --2- 172.21.15.107:0/584639283 >> v2:172.21.15.120:3300/0 conn(0x7f089015c7f0 0x7f089006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017868306 0 0) 0x7f08900cece0 con 0x7f08900cf450 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c003520 con 0x7f08900cf450 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1949017416 0 0) 0x7f087c003520 con 0x7f08900cf450 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 >> v2:172.21.15.120:3300/0 conn(0x7f089015c7f0 msgr2=0x7f089006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 --2- 172.21.15.107:0/584639283 >> v2:172.21.15.120:3300/0 conn(0x7f089015c7f0 0x7f089006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 >> v1:172.21.15.120:6789/0 conn(0x7f08900ce6c0 legacy=0x7f08900714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089006d320 con 0x7f08900cf450 2023-11-24T08:04:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0878001670 con 0x7f08900cf450 2023-11-24T08:04:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 >> v1:172.21.15.107:6789/0 conn(0x7f08900cf450 legacy=0x7f08900cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f087c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c0047c0 0x7f087c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f087c005180 0x7f087c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c00bff0 con 0x7f087c003970 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c00c160 con 0x7f087c0047c0 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.888+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c00c320 con 0x7f087c005180 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f087c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 -- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f087c005180 msgr2=0x7f087c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f087c005180 0x7f087c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 -- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c0047c0 msgr2=0x7f087c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c0047c0 0x7f087c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088e59c640 1 -- 172.21.15.107:0/584639283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c003520 con 0x7f087c003970 2023-11-24T08:04:59.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.889+0000 7f088ed9d640 1 -- 172.21.15.107:0/584639283 reap_dead start 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.890+0000 7f088e59c640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f087c003d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0880009ac0 tx=0x7f087c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fd26ab4ffa6d80 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.890+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088002c6d0 con 0x7f087c003970 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.890+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088002c830 con 0x7f087c003970 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.890+0000 7f088dd9b640 1 -- 172.21.15.107:0/584639283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088002cbc0 con 0x7f087c003970 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 -- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 msgr2=0x7f087c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f087c003d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0880009ac0 tx=0x7f087c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 -- 172.21.15.107:0/584639283 shutdown_connections 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 --2- 172.21.15.107:0/584639283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f087c003d90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 -- 172.21.15.107:0/584639283 >> 172.21.15.107:0/584639283 conn(0x7f089006a920 msgr2=0x7f089006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:04:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 -- 172.21.15.107:0/584639283 shutdown_connections 2023-11-24T08:04:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.891+0000 7f0894e94640 1 -- 172.21.15.107:0/584639283 wait complete. 2023-11-24T08:04:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.893+0000 7f0894e94640 1 Processor -- start 2023-11-24T08:04:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.893+0000 7f0894e94640 1 -- start start 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 0x7f0890155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 0x7f0890156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890153b30 con 0x7f087c003970 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08900b3160 con 0x7f08900b3a70 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f0894e94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08900b2550 con 0x7f08900b35f0 2023-11-24T08:04:59.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f088e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f088e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44108/0 (socket says 172.21.15.107:44108) 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.894+0000 7f088e59c640 1 -- 172.21.15.107:0/1052133776 learned_addr learned my addr 172.21.15.107:0/1052133776 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0887fff640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 0x7f0890155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 -- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 msgr2=0x7f0890155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088ed9d640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 0x7f0890156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 0x7f0890155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 -- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 msgr2=0x7f0890156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 0x7f0890156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 -- 172.21.15.107:0/1052133776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08900b2830 con 0x7f087c003970 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0887fff640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 0x7f0890155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088ed9d640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 0x7f0890156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:04:59.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f088e59c640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0880037040 tx=0x7f0880004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880003810 con 0x7f087c003970 2023-11-24T08:04:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880003970 con 0x7f087c003970 2023-11-24T08:04:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08900b2b10 con 0x7f087c003970 2023-11-24T08:04:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.895+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880003d00 con 0x7f087c003970 2023-11-24T08:04:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.896+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08900b20e0 con 0x7f087c003970 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.897+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890067470 con 0x7f087c003970 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.897+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0880040450 con 0x7f087c003970 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.898+0000 7f0886ffd640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 0x7f087006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.898+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08800ab660 con 0x7f087c003970 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.899+0000 7f0887fff640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 0x7f087006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:04:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.899+0000 7f0887fff640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 0x7f087006d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0878000ff0 tx=0x7f0878009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:04:59.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:04:59.903+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08800765f0 con 0x7f087c003970 2023-11-24T08:05:00.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.141+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0890154970 con 0x7f087c003970 2023-11-24T08:05:00.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.146+0000 7f0886ffd640 1 -- 172.21.15.107:0/1052133776 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171037 (secure 0 0 0) 0x7f08800044e0 con 0x7f087c003970 2023-11-24T08:05:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:00.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 msgr2=0x7f087006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 0x7f087006d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0878000ff0 tx=0x7f0878009040 comp rx=0 tx=0).stop 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 msgr2=0x7f08901535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0880037040 tx=0x7f0880004730 comp rx=0 tx=0).stop 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 shutdown_connections 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006ae40 0x7f087006d300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08900b3a70 0x7f0890156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08900b35f0 0x7f0890155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 --2- 172.21.15.107:0/1052133776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f087c003970 0x7f08901535f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.149+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 >> 172.21.15.107:0/1052133776 conn(0x7f089006a920 msgr2=0x7f08900b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.150+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 shutdown_connections 2023-11-24T08:05:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.150+0000 7f0894e94640 1 -- 172.21.15.107:0/1052133776 wait complete. 2023-11-24T08:05:00.171 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1315562075},{"type":"v1","addr":"172.21.15.107:6833","nonce":1315562075}]},"active_addr":"172.21.15.107:6833/1315562075","active_change":"2023-11-24T08:04:35.406567+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4109,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":77975759}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3820214535}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2398557609}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2188583966}]}]} 2023-11-24T08:05:00.174 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-24T08:05:00.174 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-24T08:05:00.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:00.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.385+0000 7fc49990f640 1 Processor -- start 2023-11-24T08:05:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.386+0000 7fc49990f640 1 -- start start 2023-11-24T08:05:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.387+0000 7fc49990f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc494157c70 0x7fc494161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.387+0000 7fc49990f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4940ce0b0 con 0x7fc49415e390 2023-11-24T08:05:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.387+0000 7fc49990f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc494161960 con 0x7fc4940cf450 2023-11-24T08:05:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.387+0000 7fc49990f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415e8d0 con 0x7fc494157c70 2023-11-24T08:05:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.388+0000 7fc4937fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc49415e390 0x7fc4940b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59086/0 (socket says 172.21.15.107:59086) 2023-11-24T08:05:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.388+0000 7fc4937fe640 1 -- 172.21.15.107:0/3219137973 learned_addr learned my addr 172.21.15.107:0/3219137973 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.388+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc494157c70 0x7fc494161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.388+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046502995 0 0) 0x7fc4940ce0b0 con 0x7fc49415e390 2023-11-24T08:05:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.388+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc480003520 con 0x7fc49415e390 2023-11-24T08:05:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2167581386 0 0) 0x7fc480003520 con 0x7fc49415e390 2023-11-24T08:05:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc494157c70 msgr2=0x7fc494161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc494157c70 0x7fc494161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 >> v1:172.21.15.120:6789/0 conn(0x7fc4940cf450 legacy=0x7fc49415dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49415ebb0 con 0x7fc49415e390 2023-11-24T08:05:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc48c0019f0 con 0x7fc49415e390 2023-11-24T08:05:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 >> v1:172.21.15.107:6789/0 conn(0x7fc49415e390 legacy=0x7fc4940b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4800039a0 0x7fc480003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4800047c0 0x7fc480004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800051d0 0x7fc48000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415e8d0 con 0x7fc4800039a0 2023-11-24T08:05:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48000c080 con 0x7fc4800047c0 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48000c240 con 0x7fc4800051d0 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc494157c70 0x7fc494161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4800039a0 0x7fc480003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.389+0000 7fc48bfff640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4800047c0 0x7fc480004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc4937fe640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800051d0 0x7fc48000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc492ffd640 1 -- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800051d0 msgr2=0x7fc48000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800051d0 0x7fc48000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc492ffd640 1 -- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4800047c0 msgr2=0x7fc480004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4800047c0 0x7fc480004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.390+0000 7fc492ffd640 1 -- 172.21.15.107:0/3219137973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480003520 con 0x7fc4800039a0 2023-11-24T08:05:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc492ffd640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4800039a0 0x7fc480003dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4940b52a0 tx=0x7fc4800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fecb2a959dad115 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484004030 con 0x7fc4800039a0 2023-11-24T08:05:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484004190 con 0x7fc4800039a0 2023-11-24T08:05:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc4927fc640 1 -- 172.21.15.107:0/3219137973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484004520 con 0x7fc4800039a0 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc49990f640 1 -- 172.21.15.107:0/3219137973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4800039a0 msgr2=0x7fc480003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc49990f640 1 --2- 172.21.15.107:0/3219137973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4800039a0 0x7fc480003dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4940b52a0 tx=0x7fc4800057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.391+0000 7fc4937fe640 1 -- 172.21.15.107:0/3219137973 reap_dead start 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.392+0000 7fc49990f640 1 -- 172.21.15.107:0/3219137973 shutdown_connections 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.392+0000 7fc49990f640 1 -- 172.21.15.107:0/3219137973 >> 172.21.15.107:0/3219137973 conn(0x7fc49406a920 msgr2=0x7fc49406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.392+0000 7fc49990f640 1 -- 172.21.15.107:0/3219137973 shutdown_connections 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.392+0000 7fc49990f640 1 -- 172.21.15.107:0/3219137973 wait complete. 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.393+0000 7fc49990f640 1 Processor -- start 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 -- start start 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800039a0 0x7fc4941f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4941f61c0 0x7fc494147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494147a30 con 0x7fc4941475b0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494147c20 con 0x7fc4941f61c0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.394+0000 7fc49990f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc494147f00 con 0x7fc4800039a0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44130/0 (socket says 172.21.15.107:44130) 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 -- 172.21.15.107:0/2029052817 learned_addr learned my addr 172.21.15.107:0/2029052817 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc492ffd640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800039a0 0x7fc4941f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc48bfff640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4941f61c0 0x7fc494147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 -- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800039a0 msgr2=0x7fc4941f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800039a0 0x7fc4941f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 -- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4941f61c0 msgr2=0x7fc494147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4941f61c0 0x7fc494147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.395+0000 7fc4937fe640 1 -- 172.21.15.107:0/2029052817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4941f68f0 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc4937fe640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc49415e770 tx=0x7fc48c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c00eb00 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c00ec80 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c013420 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4941f6bd0 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.396+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4941f6640 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.397+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4941f70f0 con 0x7fc4941475b0 2023-11-24T08:05:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.398+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc48c0135a0 con 0x7fc4941475b0 2023-11-24T08:05:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.400+0000 7fc490ff9640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 0x7fc47406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.400+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc48c0899d0 con 0x7fc4941475b0 2023-11-24T08:05:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.400+0000 7fc492ffd640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 0x7fc47406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.400+0000 7fc492ffd640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 0x7fc47406d330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc484035000 tx=0x7fc484030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.404+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4941f70f0 con 0x7fc4941475b0 2023-11-24T08:05:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.594+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4941f70f0 con 0x7fc4941475b0 2023-11-24T08:05:00.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.596+0000 7fc490ff9640 1 -- 172.21.15.107:0/2029052817 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fc4941f70f0 con 0x7fc4941475b0 2023-11-24T08:05:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:00.599 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 msgr2=0x7fc47406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 0x7fc47406d330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc484035000 tx=0x7fc484030000 comp rx=0 tx=0).stop 2023-11-24T08:05:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 msgr2=0x7fc494156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc49415e770 tx=0x7fc48c007bc0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 shutdown_connections 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc47406ae70 0x7fc47406d330 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4941475b0 0x7fc494156470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4941f61c0 0x7fc494147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 --2- 172.21.15.107:0/2029052817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4800039a0 0x7fc4941f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 >> 172.21.15.107:0/2029052817 conn(0x7fc49406a920 msgr2=0x7fc494070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 shutdown_connections 2023-11-24T08:05:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.599+0000 7fc49990f640 1 -- 172.21.15.107:0/2029052817 wait complete. 2023-11-24T08:05:00.622 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-24T08:05:00.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:00.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.823+0000 7f7b2b41c640 1 Processor -- start 2023-11-24T08:05:00.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.823+0000 7f7b2b41c640 1 -- start start 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b2b41c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b240cf450 0x7f7b240cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b2b41c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240cece0 con 0x7f7b2415c7f0 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b2b41c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2406d040 con 0x7f7b240cdfb0 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b2b41c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b240cfe40 con 0x7f7b240cf450 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b291b8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7b2415c7f0 0x7f7b2406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59090/0 (socket says 172.21.15.107:59090) 2023-11-24T08:05:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b291b8640 1 -- 172.21.15.107:0/2544881459 learned_addr learned my addr 172.21.15.107:0/2544881459 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.825+0000 7f7b289b7640 1 --2- 172.21.15.107:0/2544881459 >> v2:172.21.15.120:3300/0 conn(0x7f7b240cf450 0x7f7b240cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035551130 0 0) 0x7f7b240cece0 con 0x7f7b2415c7f0 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b10003520 con 0x7f7b2415c7f0 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337824205 0 0) 0x7f7b10003520 con 0x7f7b2415c7f0 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 >> v2:172.21.15.120:3300/0 conn(0x7f7b240cf450 msgr2=0x7f7b240cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 --2- 172.21.15.107:0/2544881459 >> v2:172.21.15.120:3300/0 conn(0x7f7b240cf450 0x7f7b240cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 >> v1:172.21.15.120:6789/0 conn(0x7f7b240cdfb0 legacy=0x7f7b240ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2406d320 con 0x7f7b2415c7f0 2023-11-24T08:05:00.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7b20001670 con 0x7f7b2415c7f0 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 >> v1:172.21.15.107:6789/0 conn(0x7f7b2415c7f0 legacy=0x7f7b2406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 0x7f7b10003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b100047c0 0x7f7b10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b10005180 0x7f7b1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240cfe40 con 0x7f7b100039c0 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000c080 con 0x7f7b10005180 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000c240 con 0x7f7b100047c0 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.826+0000 7f7b291b8640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 0x7f7b10003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b289b7640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b100047c0 0x7f7b10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b299b9640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b10005180 0x7f7b1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b291b8640 1 -- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b100047c0 msgr2=0x7f7b10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b291b8640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b100047c0 0x7f7b10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b291b8640 1 -- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b10005180 msgr2=0x7f7b1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b291b8640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b10005180 0x7f7b1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b291b8640 1 -- 172.21.15.107:0/2544881459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10003520 con 0x7f7b100039c0 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b299b9640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b10005180 0x7f7b1000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.827+0000 7f7b299b9640 1 -- 172.21.15.107:0/2544881459 reap_dead start 2023-11-24T08:05:00.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.828+0000 7f7b291b8640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 0x7f7b10003de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000c0e0 tx=0x7f7b100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67e7d395aa71572 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.828+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20005ad0 con 0x7f7b100039c0 2023-11-24T08:05:00.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.828+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20005c30 con 0x7f7b100039c0 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.828+0000 7f7b1bfff640 1 -- 172.21.15.107:0/2544881459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2002ca50 con 0x7f7b100039c0 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 -- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 msgr2=0x7f7b10003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 0x7f7b10003de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000c0e0 tx=0x7f7b100057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 -- 172.21.15.107:0/2544881459 shutdown_connections 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/2544881459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b100039c0 0x7f7b10003de0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 -- 172.21.15.107:0/2544881459 >> 172.21.15.107:0/2544881459 conn(0x7f7b2406a920 msgr2=0x7f7b2406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 -- 172.21.15.107:0/2544881459 shutdown_connections 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.829+0000 7f7b2b41c640 1 -- 172.21.15.107:0/2544881459 wait complete. 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.831+0000 7f7b2b41c640 1 Processor -- start 2023-11-24T08:05:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.831+0000 7f7b2b41c640 1 -- start start 2023-11-24T08:05:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 0x7f7b241483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 0x7f7b240ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240ade50 con 0x7f7b240c8d70 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c96f0 con 0x7f7b100039c0 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b2b41c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2415aa70 con 0x7f7b240ad490 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44152/0 (socket says 172.21.15.107:44152) 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 -- 172.21.15.107:0/520183481 learned_addr learned my addr 172.21.15.107:0/520183481 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b291b8640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 0x7f7b241483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b299b9640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 0x7f7b240ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 -- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 msgr2=0x7f7b240ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 0x7f7b240ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 -- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 msgr2=0x7f7b241483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 0x7f7b241483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b289b7640 1 -- 172.21.15.107:0/520183481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240ca490 con 0x7f7b240c8d70 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b291b8640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 0x7f7b241483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.832+0000 7f7b299b9640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 0x7f7b240ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b289b7640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b14011700 tx=0x7f7b14011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1400f9c0 con 0x7f7b240c8d70 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400fb20 con 0x7f7b240c8d70 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b241562a0 con 0x7f7b240c8d70 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1400fe90 con 0x7f7b240c8d70 2023-11-24T08:05:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.833+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b240c9bf0 con 0x7f7b240c8d70 2023-11-24T08:05:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.834+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b24067c10 con 0x7f7b240c8d70 2023-11-24T08:05:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.835+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b14015460 con 0x7f7b240c8d70 2023-11-24T08:05:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.836+0000 7f7b19ffb640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 0x7f7b0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.836+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b14089a80 con 0x7f7b240c8d70 2023-11-24T08:05:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.836+0000 7f7b291b8640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 0x7f7b0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:00.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.837+0000 7f7b291b8640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 0x7f7b0806d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b20034000 tx=0x7f7b20009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:00.840+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b14054b20 con 0x7f7b240c8d70 2023-11-24T08:05:01.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.034+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b24067470 con 0x7f7b240c8d70 2023-11-24T08:05:01.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.036+0000 7f7b19ffb640 1 -- 172.21.15.107:0/520183481 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f7b14022090 con 0x7f7b240c8d70 2023-11-24T08:05:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:01.039 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 msgr2=0x7f7b0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 0x7f7b0806d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b20034000 tx=0x7f7b20009030 comp rx=0 tx=0).stop 2023-11-24T08:05:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 msgr2=0x7f7b240c91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b14011700 tx=0x7f7b14011bc0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 shutdown_connections 2023-11-24T08:05:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0806ae70 0x7f7b0806d330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b240ad490 0x7f7b240ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b240c8d70 0x7f7b240c91b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 --2- 172.21.15.107:0/520183481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b100039c0 0x7f7b241483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 >> 172.21.15.107:0/520183481 conn(0x7f7b2406a920 msgr2=0x7f7b240cb420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.039+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 shutdown_connections 2023-11-24T08:05:01.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.040+0000 7f7b2b41c640 1 -- 172.21.15.107:0/520183481 wait complete. 2023-11-24T08:05:01.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-24T08:05:01.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-24T08:05:01.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-24T08:05:01.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-24T08:05:01.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-24T08:05:01.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-24T08:05:01.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-24T08:05:01.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-24T08:05:01.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.361+0000 7f454b583640 1 Processor -- start 2023-11-24T08:05:01.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.361+0000 7f454b583640 1 -- start start 2023-11-24T08:05:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.363+0000 7f454b583640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f453c0e3130 0x7f453c0e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.363+0000 7f454b583640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c007b40 con 0x7f453c053c80 2023-11-24T08:05:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.363+0000 7f454b583640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c0587e0 con 0x7f453c03c070 2023-11-24T08:05:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.363+0000 7f454b583640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f453c058ac0 con 0x7f453c0e3130 2023-11-24T08:05:01.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.364+0000 7f454a581640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f453c0e3130 0x7f453c0e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.364+0000 7f454a581640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f453c0e3130 0x7f453c0e7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58452/0 (socket says 172.21.15.107:58452) 2023-11-24T08:05:01.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.364+0000 7f454a581640 1 -- 172.21.15.107:0/2347627008 learned_addr learned my addr 172.21.15.107:0/2347627008 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.365+0000 7f454957f640 1 -- 172.21.15.107:0/2347627008 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852598398 0 0) 0x7f453c007b40 con 0x7f453c053c80 2023-11-24T08:05:01.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.365+0000 7f454a581640 1 -- 172.21.15.107:0/2347627008 >> v1:172.21.15.120:6789/0 conn(0x7f453c03c070 legacy=0x7f453c058020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.365+0000 7f454a581640 1 -- 172.21.15.107:0/2347627008 >> v1:172.21.15.107:6789/0 conn(0x7f453c053c80 legacy=0x7f453c0540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.365+0000 7f454a581640 1 -- 172.21.15.107:0/2347627008 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c006330 con 0x7f453c0e3130 2023-11-24T08:05:01.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.368+0000 7f454a581640 1 --2- 172.21.15.107:0/2347627008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0e7750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402ba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5480e3f6e1eaa7ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.368+0000 7f454957f640 1 -- 172.21.15.107:0/2347627008 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453402d840 con 0x7f453c0e3130 2023-11-24T08:05:01.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.368+0000 7f454957f640 1 -- 172.21.15.107:0/2347627008 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453402d9a0 con 0x7f453c0e3130 2023-11-24T08:05:01.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.368+0000 7f454957f640 1 -- 172.21.15.107:0/2347627008 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453402dd70 con 0x7f453c0e3130 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 -- 172.21.15.107:0/2347627008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 msgr2=0x7f453c0e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 --2- 172.21.15.107:0/2347627008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0e7750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402ba80 comp rx=0 tx=0).stop 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 -- 172.21.15.107:0/2347627008 shutdown_connections 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 --2- 172.21.15.107:0/2347627008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0e7750 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 -- 172.21.15.107:0/2347627008 >> 172.21.15.107:0/2347627008 conn(0x7f453c058eb0 msgr2=0x7f453c059290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 -- 172.21.15.107:0/2347627008 shutdown_connections 2023-11-24T08:05:01.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.369+0000 7f454b583640 1 -- 172.21.15.107:0/2347627008 wait complete. 2023-11-24T08:05:01.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.371+0000 7fbd97bdc640 1 Processor -- start 2023-11-24T08:05:01.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.372+0000 7f454b583640 1 Processor -- start 2023-11-24T08:05:01.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.372+0000 7f454b583640 1 -- start start 2023-11-24T08:05:01.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7ffaada36640 1 Processor -- start 2023-11-24T08:05:01.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 0x7f453c0e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f453c053c80 0x7f453c03c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0d4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c03df50 con 0x7f453c03c070 2023-11-24T08:05:01.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c03d340 con 0x7f453c0e3130 2023-11-24T08:05:01.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454b583640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453c03d620 con 0x7f453c053c80 2023-11-24T08:05:01.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7ffaada36640 1 -- start start 2023-11-24T08:05:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454a581640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 0x7f453c0e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.373+0000 7f454a581640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 0x7f453c0e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44156/0 (socket says 172.21.15.107:44156) 2023-11-24T08:05:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 -- 172.21.15.107:0/943866350 learned_addr learned my addr 172.21.15.107:0/943866350 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f4549d80640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f453c053c80 0x7f453c03c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.372+0000 7fbd97bdc640 1 -- start start 2023-11-24T08:05:01.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454ad82640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7fbd97bdc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbd900ccbd0 0x7fbd901660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 -- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f453c053c80 msgr2=0x7f453c03c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f453c053c80 0x7f453c03c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 -- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 msgr2=0x7f453c0d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f453c0e3130 0x7f453c0d4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7f454a581640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340094d0 con 0x7f453c03c070 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.374+0000 7fbd97bdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd900c9650 0x7fbd900c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.375+0000 7fbd97bdc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd901655f0 con 0x7fbd900ccfb0 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.375+0000 7fbd97bdc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900cc380 con 0x7fbd900ccbd0 2023-11-24T08:05:01.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.375+0000 7fbd97bdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd90162b30 con 0x7fbd900c9650 2023-11-24T08:05:01.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.375+0000 7f454a581640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 0x7f453c0e6e90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4534009990 tx=0x7f45340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453403b040 con 0x7f453c03c070 2023-11-24T08:05:01.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4534037cc0 con 0x7f453c03c070 2023-11-24T08:05:01.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c03d900 con 0x7f453c03c070 2023-11-24T08:05:01.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340426c0 con 0x7f453c03c070 2023-11-24T08:05:01.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7ffaada36640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e6de0 0x7ffaa00e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7fbd95177640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbd900ccfb0 0x7fbd90162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:51202/0 (socket says 172.21.15.107:51202) 2023-11-24T08:05:01.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7fbd95177640 1 -- 172.21.15.107:0/706507361 learned_addr learned my addr 172.21.15.107:0/706507361 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.376+0000 7ffaada36640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffaa0057ac0 0x7ffaa0057f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c0e73d0 con 0x7f453c03c070 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7ffaa6ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ffaa00e3180 0x7ffaa0006c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:51218/0 (socket says 172.21.15.107:51218) 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7ffaa6ffd640 1 -- 172.21.15.107:0/798076448 learned_addr learned my addr 172.21.15.107:0/798076448 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/798076448 >> v2:172.21.15.120:3300/0 conn(0x7ffaa0057ac0 0x7ffaa0057f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7ffaa67fc640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e6de0 0x7ffaa00e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.375+0000 7fbd96179640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd900c9650 0x7fbd900c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7fbd94976640 1 -- 172.21.15.107:0/706507361 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679650659 0 0) 0x7fbd901655f0 con 0x7fbd900ccfb0 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7fbd95978640 1 --2- 172.21.15.107:0/706507361 >> v2:172.21.15.120:3300/0 conn(0x7fbd900ccbd0 0x7fbd901660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.377+0000 7fbd94976640 1 -- 172.21.15.107:0/706507361 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c003520 con 0x7fbd900ccfb0 2023-11-24T08:05:01.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f453c0e3510 con 0x7f453c03c070 2023-11-24T08:05:01.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa0057630 con 0x7ffaa00e3180 2023-11-24T08:05:01.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa00570b0 con 0x7ffaa0057ac0 2023-11-24T08:05:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00e7760 con 0x7ffaa00e6de0 2023-11-24T08:05:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7fbd95978640 1 -- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd900c9650 msgr2=0x7fbd900c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7fbd95978640 1 --2- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd900c9650 0x7fbd900c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7fbd95978640 1 -- 172.21.15.107:0/706507361 >> v1:172.21.15.120:6789/0 conn(0x7fbd900ccfb0 legacy=0x7fbd90162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.378+0000 7fbd95978640 1 -- 172.21.15.107:0/706507361 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd90162e10 con 0x7fbd900ccbd0 2023-11-24T08:05:01.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa5ffb640 1 -- 172.21.15.107:0/798076448 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694511869 0 0) 0x7ffaa0057630 con 0x7ffaa00e3180 2023-11-24T08:05:01.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa5ffb640 1 -- 172.21.15.107:0/798076448 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa94003520 con 0x7ffaa00e3180 2023-11-24T08:05:01.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa77fe640 1 -- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e6de0 msgr2=0x7ffaa00e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e6de0 0x7ffaa00e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa77fe640 1 -- 172.21.15.107:0/798076448 >> v1:172.21.15.120:6789/0 conn(0x7ffaa00e3180 legacy=0x7ffaa0006c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.379+0000 7ffaa77fe640 1 -- 172.21.15.107:0/798076448 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa00076b0 con 0x7ffaa0057ac0 2023-11-24T08:05:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.380+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4534042820 con 0x7f453c03c070 2023-11-24T08:05:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.380+0000 7f9542101640 1 Processor -- start 2023-11-24T08:05:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7fbd95978640 1 --2- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccbd0 0x7fbd901660b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097a0 tx=0x7fbd8c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd3e239f00337844 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa0057ac0 0x7ffaa0057f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8000b60 tx=0x7ffaa80a6310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9bebf95ad30a39d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7f9542101640 1 -- start start 2023-11-24T08:05:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7f453affd640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f452806ae70 0x7f452806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7fbd94976640 1 -- 172.21.15.107:0/706507361 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02d040 con 0x7fbd900ccbd0 2023-11-24T08:05:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7ffaa5ffb640 1 -- 172.21.15.107:0/798076448 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80ae640 con 0x7ffaa0057ac0 2023-11-24T08:05:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4534032070 con 0x7f453c03c070 2023-11-24T08:05:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7fbd94976640 1 -- 172.21.15.107:0/706507361 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c031810 con 0x7fbd900ccbd0 2023-11-24T08:05:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.381+0000 7fbd94976640 1 -- 172.21.15.107:0/706507361 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031b80 con 0x7fbd900ccbd0 2023-11-24T08:05:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.382+0000 7ffaa5ffb640 1 -- 172.21.15.107:0/798076448 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa80ae7e0 con 0x7ffaa0057ac0 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.382+0000 7f4549d80640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f452806ae70 0x7f452806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.382+0000 7f453affd640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] conn(0x7f4528073090 0x7f45280754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.382+0000 7fbd97bdc640 1 -- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccbd0 msgr2=0x7fbd901660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.382+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccbd0 0x7fbd901660b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097a0 tx=0x7fbd8c02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fbd97bdc640 1 -- 172.21.15.107:0/706507361 shutdown_connections 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd900c9650 0x7fbd900c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/706507361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccbd0 0x7fbd901660b0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097a0 tx=0x7fbd8c02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fbd97bdc640 1 -- 172.21.15.107:0/706507361 >> 172.21.15.107:0/706507361 conn(0x7fbd900b0c50 msgr2=0x7fbd900b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4528075b60 con 0x7f4528073090 2023-11-24T08:05:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45340acfb0 con 0x7f453c03c070 2023-11-24T08:05:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa0057ac0 msgr2=0x7ffaa0057f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7ffaada36640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa0057ac0 0x7ffaa0057f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8000b60 tx=0x7ffaa80a6310 comp rx=0 tx=0).stop 2023-11-24T08:05:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f9542101640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f953c0ccbd0 0x7f953c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f9542101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f9542101640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953c1655f0 con 0x7f953c162540 2023-11-24T08:05:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f9542101640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953c0cc380 con 0x7f953c0ccbd0 2023-11-24T08:05:01.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f9542101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953c162b30 con 0x7f953c0ccfb0 2023-11-24T08:05:01.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fa77dddc640 1 Processor -- start 2023-11-24T08:05:01.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7fa77dddc640 1 -- start start 2023-11-24T08:05:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.383+0000 7f4549d80640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f452806ae70 0x7f452806d330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f453c0e2930 tx=0x7f4540009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 shutdown_connections 2023-11-24T08:05:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f953c0ccbd0 0x7f953c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f953c162540 0x7f953c0c9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:51232/0 (socket says 172.21.15.107:51232) 2023-11-24T08:05:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953bfff640 1 -- 172.21.15.107:0/2138377194 learned_addr learned my addr 172.21.15.107:0/2138377194 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7ffaada36640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa0057ac0 0x7ffaa0057f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7ffaada36640 1 --2- 172.21.15.107:0/798076448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e6de0 0x7ffaa00e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 >> 172.21.15.107:0/798076448 conn(0x7ffaa003a9f0 msgr2=0x7ffaa003ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953affd640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 shutdown_connections 2023-11-24T08:05:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953b7fe640 1 -- 172.21.15.107:0/2138377194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 msgr2=0x7f953c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953b7fe640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.384+0000 7f953b7fe640 1 -- 172.21.15.107:0/2138377194 >> v1:172.21.15.120:6789/0 conn(0x7f953c162540 legacy=0x7f953c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.385+0000 7f953b7fe640 1 -- 172.21.15.107:0/2138377194 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c162e10 con 0x7f953c0ccbd0 2023-11-24T08:05:01.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.385+0000 7ffaada36640 1 -- 172.21.15.107:0/798076448 wait complete. 2023-11-24T08:05:01.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.385+0000 7fbd97bdc640 1 -- 172.21.15.107:0/706507361 shutdown_connections 2023-11-24T08:05:01.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.385+0000 7fbd97bdc640 1 -- 172.21.15.107:0/706507361 wait complete. 2023-11-24T08:05:01.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.385+0000 7f454ad82640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] conn(0x7f4528073090 0x7f45280754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f3c6c378640 1 Processor -- start 2023-11-24T08:05:01.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7fa77dddc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa7780cc0f0 0x7fa7780c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f953b7fe640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c0ccbd0 0x7f953c1660b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9534009dc0 tx=0x7f953402bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd19776df1b216b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f953a7fc640 1 -- 172.21.15.107:0/2138377194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953402c9f0 con 0x7f953c0ccbd0 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f953a7fc640 1 -- 172.21.15.107:0/2138377194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953402cb50 con 0x7f953c0ccbd0 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f953a7fc640 1 -- 172.21.15.107:0/2138377194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9534034440 con 0x7f953c0ccbd0 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7fa77dddc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7fa77dddc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77815eea0 con 0x7fa77815b440 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7fa77dddc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77815b820 con 0x7fa7780cc0f0 2023-11-24T08:05:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7fa77dddc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780abf30 con 0x7fa7780c57e0 2023-11-24T08:05:01.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.387+0000 7f454ad82640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] conn(0x7f4528073090 0x7f45280754b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.387+0000 7fa777fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.387+0000 7fa777fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34308/0 (socket says 172.21.15.107:34308) 2023-11-24T08:05:01.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.387+0000 7fa777fff640 1 -- 172.21.15.107:0/1155744264 learned_addr learned my addr 172.21.15.107:0/1155744264 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fbd97bdc640 1 Processor -- start 2023-11-24T08:05:01.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.386+0000 7f3c6c378640 1 -- start start 2023-11-24T08:05:01.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fbd97bdc640 1 -- start start 2023-11-24T08:05:01.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7f9542101640 1 -- 172.21.15.107:0/2138377194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c0ccbd0 msgr2=0x7f953c1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7ffaada36640 1 Processor -- start 2023-11-24T08:05:01.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7f9542101640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c0ccbd0 0x7f953c1660b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9534009dc0 tx=0x7f953402bcc0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 -- 172.21.15.107:0/2138377194 shutdown_connections 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 --2- 172.21.15.107:0/2138377194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c0ccbd0 0x7f953c1660b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 -- 172.21.15.107:0/2138377194 >> 172.21.15.107:0/2138377194 conn(0x7f953c0b0c50 msgr2=0x7f953c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 -- 172.21.15.107:0/2138377194 shutdown_connections 2023-11-24T08:05:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f9542101640 1 -- 172.21.15.107:0/2138377194 wait complete. 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fa777fff640 1 -- 172.21.15.107:0/1155744264 >> v2:172.21.15.120:3300/0 conn(0x7fa7780cc0f0 msgr2=0x7fa7780c52a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fa777fff640 1 --2- 172.21.15.107:0/1155744264 >> v2:172.21.15.120:3300/0 conn(0x7fa7780cc0f0 0x7fa7780c52a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fa777fff640 1 -- 172.21.15.107:0/1155744264 >> v1:172.21.15.120:6789/0 conn(0x7fa77815b440 legacy=0x7fa7780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.388+0000 7fa777fff640 1 -- 172.21.15.107:0/1155744264 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77815f0f0 con 0x7fa7780c57e0 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f3c6c378640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 0x7f3c6406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f3c6c378640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3c640b1ed0 0x7f3c6406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== osd.0 v2:172.21.15.107:6807/1814653023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4528075b60 con 0x7f4528073090 2023-11-24T08:05:01.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.389+0000 7ffaada36640 1 -- start start 2023-11-24T08:05:01.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6c378640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6415db00 con 0x7f3c6415d720 2023-11-24T08:05:01.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6a114640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3c6415d720 0x7f3c640d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59118/0 (socket says 172.21.15.107:59118) 2023-11-24T08:05:01.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6a114640 1 -- 172.21.15.107:0/2179448255 learned_addr learned my addr 172.21.15.107:0/2179448255 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fa777fff640 1 --2- 172.21.15.107:0/1155744264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa770009e50 tx=0x7fa77002be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a25d3de325a1c05a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c69913640 1 --2- 172.21.15.107:0/2179448255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 0x7f3c6406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd900c9650 0x7fbd90159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c640d0ce0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6406d400 con 0x7f3c640d3a40 2023-11-24T08:05:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fa7767fc640 1 -- 172.21.15.107:0/1155744264 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77002ca10 con 0x7fa7780c57e0 2023-11-24T08:05:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7f3c6a915640 1 --2- 172.21.15.107:0/2179448255 >> v2:172.21.15.120:3300/0 conn(0x7f3c640b1ed0 0x7f3c6406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fa7767fc640 1 -- 172.21.15.107:0/1155744264 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77002cb70 con 0x7fa7780c57e0 2023-11-24T08:05:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fa7767fc640 1 -- 172.21.15.107:0/1155744264 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700343f0 con 0x7fa7780c57e0 2023-11-24T08:05:01.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccfb0 0x7fbd9014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9014e790 0x7fbd900ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9015bc40 con 0x7fbd900c9650 2023-11-24T08:05:01.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd90072640 con 0x7fbd900ccfb0 2023-11-24T08:05:01.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.390+0000 7fbd97bdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9014ec20 con 0x7fbd9014e790 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 0x7ffaa004e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 0x7ffaa0052b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa00e6de0 0x7ffaa0184f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00e6490 con 0x7ffaa0057ac0 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa004e500 con 0x7ffaa00e6de0 2023-11-24T08:05:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7ffaada36640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa004d8f0 con 0x7ffaa00e3180 2023-11-24T08:05:01.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7f3c69112640 1 -- 172.21.15.107:0/2179448255 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302697202 0 0) 0x7f3c6415db00 con 0x7f3c6415d720 2023-11-24T08:05:01.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7f9542101640 1 Processor -- start 2023-11-24T08:05:01.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7f9542101640 1 -- start start 2023-11-24T08:05:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fbd95978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd900c9650 0x7fbd90159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fbd96179640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9014e790 0x7fbd900ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fbd96179640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9014e790 0x7fbd900ac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34324/0 (socket says 172.21.15.107:34324) 2023-11-24T08:05:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fbd96179640 1 -- 172.21.15.107:0/1257382672 learned_addr learned my addr 172.21.15.107:0/1257382672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.391+0000 7fbd95177640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccfb0 0x7fbd9014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 0x7ffaa0052b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 0x7ffaa004e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7f3c69112640 1 -- 172.21.15.107:0/2179448255 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50003520 con 0x7f3c6415d720 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fa77dddc640 1 -- 172.21.15.107:0/1155744264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 msgr2=0x7fa77806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fa77dddc640 1 --2- 172.21.15.107:0/1155744264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa770009e50 tx=0x7fa77002be70 comp rx=0 tx=0).stop 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fbd95978640 1 -- 172.21.15.107:0/1257382672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9014e790 msgr2=0x7fbd900ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fbd95978640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9014e790 0x7fbd900ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fbd95978640 1 -- 172.21.15.107:0/1257382672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccfb0 msgr2=0x7fbd9014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fbd95978640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd900ccfb0 0x7fbd9014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7fbd95978640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c009400 con 0x7fbd900c9650 2023-11-24T08:05:01.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 0x7ffaa0052b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34334/0 (socket says 172.21.15.107:34334) 2023-11-24T08:05:01.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa67fc640 1 -- 172.21.15.107:0/774150072 learned_addr learned my addr 172.21.15.107:0/774150072 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 0x7ffaa004e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44188/0 (socket says 172.21.15.107:44188) 2023-11-24T08:05:01.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f953c0ccbd0 0x7f953c157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c164e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7f3c69913640 1 -- 172.21.15.107:0/2179448255 >> v2:172.21.15.120:3300/0 conn(0x7f3c640b1ed0 msgr2=0x7f3c6406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7f3c69913640 1 --2- 172.21.15.107:0/2179448255 >> v2:172.21.15.120:3300/0 conn(0x7f3c640b1ed0 0x7f3c6406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7f3c69913640 1 -- 172.21.15.107:0/2179448255 >> v1:172.21.15.107:6789/0 conn(0x7f3c6415d720 legacy=0x7f3c640d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7f3c69913640 1 -- 172.21.15.107:0/2179448255 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640b0100 con 0x7f3c640d3a40 2023-11-24T08:05:01.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c162540 0x7f953c163dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1653a0 con 0x7f953c162540 2023-11-24T08:05:01.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c0adbc0 con 0x7f953c0ccbd0 2023-11-24T08:05:01.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f9542101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1eb7f0 con 0x7f953c0ccfb0 2023-11-24T08:05:01.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.392+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa00e6de0 0x7ffaa0184f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7fbd95978640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd900c9650 0x7fbd90159220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c033000 tx=0x7fbd8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7fa77dddc640 1 -- 172.21.15.107:0/1155744264 shutdown_connections 2023-11-24T08:05:01.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7fa77dddc640 1 --2- 172.21.15.107:0/1155744264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7780c57e0 0x7fa77806c5f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7fa77dddc640 1 --2- 172.21.15.107:0/1155744264 >> v2:172.21.15.120:3300/0 conn(0x7fa7780cc0f0 0x7fa7780c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fa77dddc640 1 -- 172.21.15.107:0/1155744264 >> 172.21.15.107:0/1155744264 conn(0x7fa7780b0c50 msgr2=0x7fa7780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7f953affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7f953bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c162540 0x7f953c163dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7f953affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c164e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34340/0 (socket says 172.21.15.107:34340) 2023-11-24T08:05:01.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7f953affd640 1 -- 172.21.15.107:0/2860720159 learned_addr learned my addr 172.21.15.107:0/2860720159 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.393+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c003820 con 0x7fbd900c9650 2023-11-24T08:05:01.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fa77dddc640 1 -- 172.21.15.107:0/1155744264 shutdown_connections 2023-11-24T08:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c03f070 con 0x7fbd900c9650 2023-11-24T08:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c003bc0 con 0x7fbd900c9650 2023-11-24T08:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fa77dddc640 1 -- 172.21.15.107:0/1155744264 wait complete. 2023-11-24T08:05:01.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa67fc640 1 -- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa00e6de0 msgr2=0x7ffaa0184f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa67fc640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaa00e6de0 0x7ffaa0184f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa67fc640 1 -- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 msgr2=0x7ffaa004e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa67fc640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 0x7ffaa004e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa67fc640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa804f730 con 0x7ffaa00e3180 2023-11-24T08:05:01.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7f3c69913640 1 --2- 172.21.15.107:0/2179448255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 0x7f3c6406ba80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c600097a0 tx=0x7f3c6002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efbdab5f5b893028 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7f953b7fe640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f953c0ccbd0 0x7f953c157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7ffaa6ffd640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaa0057ac0 0x7ffaa004e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 -- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 msgr2=0x7f953c164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f953c0ccfb0 0x7f953c164e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 -- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f953c0ccbd0 msgr2=0x7f953c157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f953c0ccbd0 0x7f953c157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f3c69112640 1 -- 172.21.15.107:0/2179448255 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60031780 con 0x7f3c640d3a40 2023-11-24T08:05:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f3c69112640 1 -- 172.21.15.107:0/2179448255 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c600318e0 con 0x7f3c640d3a40 2023-11-24T08:05:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9534009a70 con 0x7f953c162540 2023-11-24T08:05:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7ffaa67fc640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 0x7ffaa0052b60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e870 tx=0x7ffa9800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.395+0000 7f953bfff640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c162540 0x7f953c163dc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f952400c500 tx=0x7f952400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9015afc0 con 0x7fbd900c9650 2023-11-24T08:05:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.394+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9015b580 con 0x7fbd900c9650 2023-11-24T08:05:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.396+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 msgr2=0x7f3c6406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.396+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2179448255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 0x7f3c6406ba80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c600097a0 tx=0x7f3c6002b820 comp rx=0 tx=0).stop 2023-11-24T08:05:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.396+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800ee60 con 0x7ffaa00e3180 2023-11-24T08:05:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.396+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa004dbd0 con 0x7ffaa00e3180 2023-11-24T08:05:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.396+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa004e190 con 0x7ffaa00e3180 2023-11-24T08:05:01.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524007620 con 0x7f953c162540 2023-11-24T08:05:01.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95240047e0 con 0x7f953c162540 2023-11-24T08:05:01.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524004bd0 con 0x7f953c162540 2023-11-24T08:05:01.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953c0ade60 con 0x7f953c162540 2023-11-24T08:05:01.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9800c940 con 0x7ffaa00e3180 2023-11-24T08:05:01.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800ccf0 con 0x7ffaa00e3180 2023-11-24T08:05:01.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7fa77dddc640 1 Processor -- start 2023-11-24T08:05:01.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7fa77dddc640 1 -- start start 2023-11-24T08:05:01.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7fb32a490640 1 Processor -- start 2023-11-24T08:05:01.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.397+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd900cba20 con 0x7fbd900c9650 2023-11-24T08:05:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 shutdown_connections 2023-11-24T08:05:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2179448255 >> v2:172.21.15.120:3300/0 conn(0x7f3c640b1ed0 0x7f3c6406bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2179448255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c640d3a40 0x7f3c6406ba80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 >> 172.21.15.107:0/2179448255 conn(0x7f3c6408af60 msgr2=0x7f3c6409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa8c000fc0 con 0x7ffaa00e3180 2023-11-24T08:05:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fa77dddc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7780c57e0 0x7fa77815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7f1c85bf5640 1 Processor -- start 2023-11-24T08:05:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 shutdown_connections 2023-11-24T08:05:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fa77dddc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7780cc0f0 0x7fa77814cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.398+0000 7fb32a490640 1 -- start start 2023-11-24T08:05:01.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb32a490640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb3240d3a40 0x7fb32406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb32a490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3240b1ed0 0x7fb32406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb32a490640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32415db00 con 0x7fb32415d720 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb32a490640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3240d0ce0 con 0x7fb3240d3a40 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb32a490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32406d400 con 0x7fb3240b1ed0 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb323fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb32415d720 0x7fb3240d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59120/0 (socket says 172.21.15.107:59120) 2023-11-24T08:05:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb323fff640 1 -- 172.21.15.107:0/103698027 learned_addr learned my addr 172.21.15.107:0/103698027 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fa77dddc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77815b440 0x7fa77814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7fb3237fe640 1 --2- 172.21.15.107:0/103698027 >> v2:172.21.15.107:3300/0 conn(0x7fb3240d3a40 0x7fb32406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa77dddc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77814a130 con 0x7fa7780cc0f0 2023-11-24T08:05:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953c164300 con 0x7f953c162540 2023-11-24T08:05:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa77dddc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa778149240 con 0x7fa7780c57e0 2023-11-24T08:05:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa776ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7780cc0f0 0x7fa77814cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fb328a2d640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3240b1ed0 0x7fb32406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa776ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7780cc0f0 0x7fa77814cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44214/0 (socket says 172.21.15.107:44214) 2023-11-24T08:05:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa776ffd640 1 -- 172.21.15.107:0/947361584 learned_addr learned my addr 172.21.15.107:0/947361584 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fa777fff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77815b440 0x7fa77814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fb3237fe640 1 -- 172.21.15.107:0/103698027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3240b1ed0 msgr2=0x7fb32406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fb3237fe640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3240b1ed0 0x7fb32406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.400+0000 7fb3237fe640 1 -- 172.21.15.107:0/103698027 >> v1:172.21.15.107:6789/0 conn(0x7fb32415d720 legacy=0x7fb3240d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.399+0000 7f1c85bf5640 1 -- start start 2023-11-24T08:05:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa778149520 con 0x7fa77815b440 2023-11-24T08:05:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7fa7777fe640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7780c57e0 0x7fa77815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7fb3237fe640 1 -- 172.21.15.107:0/103698027 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3240b0100 con 0x7fb3240d3a40 2023-11-24T08:05:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbd8c003d20 con 0x7fbd900c9650 2023-11-24T08:05:01.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c8015d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c80161260 0x7f1c801616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 0x7f1c800cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c800cf4e0 con 0x7f1c800ceb10 2023-11-24T08:05:01.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fa777fff640 1 -- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7780c57e0 msgr2=0x7fa77815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fa777fff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7780c57e0 0x7fa77815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fa777fff640 1 -- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7780cc0f0 msgr2=0x7fa77814cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fa777fff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7780cc0f0 0x7fa77814cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fa777fff640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa770009b30 con 0x7fa77815b440 2023-11-24T08:05:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8015db90 con 0x7f1c80161260 2023-11-24T08:05:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fbd867fc640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd7406ae40 0x7fbd7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.401+0000 7f1c85bf5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80161ce0 con 0x7f1c8015d270 2023-11-24T08:05:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f953c0cd410 con 0x7f953c162540 2023-11-24T08:05:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7fb3237fe640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32406ba80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3240d0e80 tx=0x7fb31402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee95d54d2379d61f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffa98013470 con 0x7ffaa00e3180 2023-11-24T08:05:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f3c6c378640 1 -- 172.21.15.107:0/2179448255 wait complete. 2023-11-24T08:05:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f1c7ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 0x7f1c800cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f1c7ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 0x7f1c800cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44224/0 (socket says 172.21.15.107:44224) 2023-11-24T08:05:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f1c7ffff640 1 -- 172.21.15.107:0/4165857756 learned_addr learned my addr 172.21.15.107:0/4165857756 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f1c77fff640 1 --2- 172.21.15.107:0/4165857756 >> v2:172.21.15.120:3300/0 conn(0x7f1c80161260 0x7f1c801616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.402+0000 7f1c7f7fe640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c8015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fb322ffd640 1 -- 172.21.15.107:0/103698027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314031780 con 0x7fb3240d3a40 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fb322ffd640 1 -- 172.21.15.107:0/103698027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3140318e0 con 0x7fb3240d3a40 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fb322ffd640 1 -- 172.21.15.107:0/103698027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314031c70 con 0x7fb3240d3a40 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd8c041070 con 0x7fbd900c9650 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fbd867fc640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] conn(0x7fbd74073060 0x7fbd74075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd74075b30 con 0x7fbd74073060 2023-11-24T08:05:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f1c77fff640 1 -- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 msgr2=0x7f1c8015d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f1c77fff640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c8015d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f1c77fff640 1 -- 172.21.15.107:0/4165857756 >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 msgr2=0x7f1c800cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f1c77fff640 1 --2- 172.21.15.107:0/4165857756 >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 0x7f1c800cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f1c77fff640 1 -- 172.21.15.107:0/4165857756 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c80161fc0 con 0x7f1c80161260 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 -- 172.21.15.107:0/103698027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 msgr2=0x7fb32406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32406ba80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3240d0e80 tx=0x7fb31402b820 comp rx=0 tx=0).stop 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 -- 172.21.15.107:0/103698027 shutdown_connections 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3240b1ed0 0x7fb32406bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 --2- 172.21.15.107:0/103698027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32406ba80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fb32a490640 1 -- 172.21.15.107:0/103698027 >> 172.21.15.107:0/103698027 conn(0x7fb32408af60 msgr2=0x7fb32409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.403+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9524004d30 con 0x7f953c162540 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f9533fff640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f951c06ae90 0x7f951c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9524088ce0 con 0x7f953c162540 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7f9533fff640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] conn(0x7f951c0730b0 0x7f951c0754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f951c075b80 con 0x7f951c0730b0 2023-11-24T08:05:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f952408b2c0 con 0x7f953c162540 2023-11-24T08:05:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd8c0abb90 con 0x7fbd900c9650 2023-11-24T08:05:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7fbd96179640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] conn(0x7fbd74073060 0x7fbd74075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7fbd95177640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd7406ae40 0x7fbd7406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7ffa9ffff640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa8806ae70 0x7ffa8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7fa777fff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77815b440 0x7fa77814d0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa770032000 tx=0x7fa770004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7f3c6c378640 1 Processor -- start 2023-11-24T08:05:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7f1c77fff640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c801616c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1c8008b850 tx=0x7f1c6802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eefe7c0d4f27ad4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa98089520 con 0x7ffaa00e3180 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7ffa9ffff640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] conn(0x7ffa88073090 0x7ffa880754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa88075b60 con 0x7ffa88073090 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.404+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa9808bb40 con 0x7ffaa00e3180 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] conn(0x7ffa88073090 0x7ffa880754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7ffaa6ffd640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa8806ae70 0x7ffa8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7f953b7fe640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f951c06ae90 0x7f951c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7f953affd640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] conn(0x7f951c0730b0 0x7f951c0754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77003e450 con 0x7fa77815b440 2023-11-24T08:05:01.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77003e5d0 con 0x7fa77815b440 2023-11-24T08:05:01.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77003e780 con 0x7fa77815b440 2023-11-24T08:05:01.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa778149800 con 0x7fa77815b440 2023-11-24T08:05:01.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7f1c7effd640 1 -- 172.21.15.107:0/4165857756 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c680317c0 con 0x7f1c80161260 2023-11-24T08:05:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7fb32a490640 1 -- 172.21.15.107:0/103698027 shutdown_connections 2023-11-24T08:05:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.405+0000 7fb32a490640 1 -- 172.21.15.107:0/103698027 wait complete. 2023-11-24T08:05:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa778149dc0 con 0x7fa77815b440 2023-11-24T08:05:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] conn(0x7ffa88073090 0x7ffa880754b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7f1c7effd640 1 -- 172.21.15.107:0/4165857756 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c68031920 con 0x7f1c80161260 2023-11-24T08:05:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.407+0000 7f1c7effd640 1 -- 172.21.15.107:0/4165857756 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68031cb0 con 0x7f1c80161260 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.406+0000 7f3c6c378640 1 -- start start 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c640b1ed0 0x7f3c6415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c640d3a40 0x7f3c6406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c6415d720 0x7f3c6406e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6415ce50 con 0x7f3c640b1ed0 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640c8970 con 0x7f3c640d3a40 2023-11-24T08:05:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f3c6c378640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640706f0 con 0x7f3c6415d720 2023-11-24T08:05:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa758000fc0 con 0x7fa77815b440 2023-11-24T08:05:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7f953b7fe640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f951c06ae90 0x7f951c06d350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9534036000 tx=0x7f9534030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7ffaa6ffd640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa8806ae70 0x7ffa8806d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa90000f60 tx=0x7ffa90007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7fbd95177640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd7406ae40 0x7fbd7406d300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd90158040 tx=0x7fbd80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c69913640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c640d3a40 0x7f3c6406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.408+0000 7fbd96179640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] conn(0x7fbd74073060 0x7fbd74075480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c640b1ed0 0x7f3c6415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7fb32a490640 1 Processor -- start 2023-11-24T08:05:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a915640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c6415d720 0x7f3c6406e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c69913640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c640d3a40 0x7f3c6406def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58534/0 (socket says 172.21.15.107:58534) 2023-11-24T08:05:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c69913640 1 -- 172.21.15.107:0/2895244812 learned_addr learned my addr 172.21.15.107:0/2895244812 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7fb32a490640 1 -- start start 2023-11-24T08:05:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f953affd640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] conn(0x7f951c0730b0 0x7f951c0754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 -- 172.21.15.107:0/2895244812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c6415d720 msgr2=0x7f3c6406e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c6415d720 0x7f3c6406e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 -- 172.21.15.107:0/2895244812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c640d3a40 msgr2=0x7f3c6406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c640d3a40 0x7f3c6406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f3c6a114640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c60009400 con 0x7f3c640b1ed0 2023-11-24T08:05:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f1c85bf5640 1 -- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 msgr2=0x7f1c801616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.409+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c801616c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1c8008b850 tx=0x7f1c6802b860 comp rx=0 tx=0).stop 2023-11-24T08:05:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 -- 172.21.15.107:0/4165857756 shutdown_connections 2023-11-24T08:05:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/4165857756 >> v2:172.21.15.107:3300/0 conn(0x7f1c800ceb10 0x7f1c800cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== osd.2 v2:172.21.15.107:6816/4260350828 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ffa88075b60 con 0x7ffa88073090 2023-11-24T08:05:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== osd.1 v2:172.21.15.107:6800/132161644 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbd74075b30 con 0x7fbd74073060 2023-11-24T08:05:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c801616c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/4165857756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c8015d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c6a114640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c640b1ed0 0x7f3c6415c910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c430 tx=0x7f3c5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 -- 172.21.15.107:0/4165857756 >> 172.21.15.107:0/4165857756 conn(0x7f1c8008af60 msgr2=0x7f1c8009c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54005ce0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c54005e40 con 0x7f3c640b1ed0 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c540072d0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64070940 con 0x7f3c640b1ed0 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 -- 172.21.15.107:0/4165857756 shutdown_connections 2023-11-24T08:05:01.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.412+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== osd.5 v2:172.21.15.120:6802/3047087772 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f951c075b80 con 0x7f951c0730b0 2023-11-24T08:05:01.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f1c85bf5640 1 -- 172.21.15.107:0/4165857756 wait complete. 2023-11-24T08:05:01.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.412+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa77003e8e0 con 0x7fa77815b440 2023-11-24T08:05:01.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.410+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64070f00 con 0x7f3c640b1ed0 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3240b1ed0 0x7fb32415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32415f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 0x7fb3240cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3240cb920 con 0x7fb3240d3a40 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32415ee80 con 0x7fb3240b1ed0 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7fb32a490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32415e270 con 0x7fb32415d720 2023-11-24T08:05:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7f1c85bf5640 1 Processor -- start 2023-11-24T08:05:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.413+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3c54007430 con 0x7f3c640b1ed0 2023-11-24T08:05:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fb328a2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 0x7fb3240cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7f1c85bf5640 1 -- start start 2023-11-24T08:05:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fa75ffff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406ae40 0x7fa76406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7700ac790 con 0x7fa77815b440 2023-11-24T08:05:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fb328a2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 0x7fb3240cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34372/0 (socket says 172.21.15.107:34372) 2023-11-24T08:05:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fb3237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fa75ffff640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] conn(0x7fa764072fe0 0x7fa764075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa764075ab0 con 0x7fa764072fe0 2023-11-24T08:05:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7700acbe0 con 0x7fa77815b440 2023-11-24T08:05:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.414+0000 7fb328a2d640 1 -- 172.21.15.107:0/2892056889 learned_addr learned my addr 172.21.15.107:0/2892056889 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c640b09a0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7f3c5affd640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c4806af10 0x7f3c4806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7fa7777fe640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406ae40 0x7fa76406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7fa776ffd640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] conn(0x7fa764072fe0 0x7fa764075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c54088ae0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f3c5affd640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] conn(0x7f3c480730b0 0x7f3c480754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c48075b80 con 0x7f3c480730b0 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c54053ae0 con 0x7f3c640b1ed0 2023-11-24T08:05:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f3c69913640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c4806af10 0x7f3c4806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f1c85bf5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 0x7f1c801589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7fa7777fe640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406ae40 0x7fa76406d300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa768006fd0 tx=0x7fa768008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f1c85bf5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c800c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.415+0000 7fb3237fe640 1 -- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 msgr2=0x7fb3240cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 0x7f1c801589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7fb3237fe640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 0x7fb3240cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7fb3237fe640 1 -- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3240b1ed0 msgr2=0x7fb32415dc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T08:05:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7fb3237fe640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3240b1ed0 0x7fb32415dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.416+0000 7fb3237fe640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb314009400 con 0x7fb3240d3a40 2023-11-24T08:05:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7fb328a2d640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb32415d720 0x7fb3240cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 0x7f1c801589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44246/0 (socket says 172.21.15.107:44246) 2023-11-24T08:05:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7fa776ffd640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] conn(0x7fa764072fe0 0x7fa764075400 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c7f7fe640 1 -- 172.21.15.107:0/549195118 learned_addr learned my addr 172.21.15.107:0/549195118 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c800b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80158f20 con 0x7f1c800ceb10 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80160c40 con 0x7f1c80161260 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c77fff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c800c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c801600c0 con 0x7f1c8015d270 2023-11-24T08:05:01.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.417+0000 7f3c6a915640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] conn(0x7f3c480730b0 0x7f3c480754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.418+0000 7f3c69913640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c4806af10 0x7f3c4806d3d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c60008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.418+0000 7f1c7ffff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c800b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.418+0000 7f3c6a915640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] conn(0x7f3c480730b0 0x7f3c480754d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.418+0000 7fb3237fe640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32415f310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb314033000 tx=0x7fb3140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7f7fe640 1 -- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 msgr2=0x7f1c800c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7f7fe640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c8015d270 0x7f1c800c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7f7fe640 1 -- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 msgr2=0x7f1c800b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7f7fe640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c800b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7f7fe640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c68009400 con 0x7f1c800ceb10 2023-11-24T08:05:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3140328e0 con 0x7fb3240d3a40 2023-11-24T08:05:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7f1c7ffff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c80161260 0x7f1c800b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32415e550 con 0x7fb3240d3a40 2023-11-24T08:05:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.420+0000 7f1c7f7fe640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 0x7f1c801589e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000d800 tx=0x7f1c7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32415eb10 con 0x7fb3240d3a40 2023-11-24T08:05:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314032a40 con 0x7fb3240d3a40 2023-11-24T08:05:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.419+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314032df0 con 0x7fb3240d3a40 2023-11-24T08:05:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.420+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== osd.4 v2:172.21.15.120:6800/239423732 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa764075ab0 con 0x7fa764072fe0 2023-11-24T08:05:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.420+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== osd.3 v2:172.21.15.107:6818/1213738659 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3c48075b80 con 0x7f3c480730b0 2023-11-24T08:05:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.421+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb304000fc0 con 0x7fb3240d3a40 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.421+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000b6c0 con 0x7f1c800ceb10 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.421+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c801603a0 con 0x7f1c800ceb10 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.421+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c80160960 con 0x7f1c800ceb10 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.422+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7000b820 con 0x7f1c800ceb10 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.422+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000b6c0 con 0x7f1c800ceb10 2023-11-24T08:05:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.423+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c64000fc0 con 0x7f1c800ceb10 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.423+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb31404e440 con 0x7fb3240d3a40 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.424+0000 7fb320ff9640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb31006ae70 0x7fb31006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.424+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3140ab7e0 con 0x7fb3240d3a40 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.424+0000 7fb320ff9640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] conn(0x7fb310073010 0x7fb310075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.424+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb310075ae0 con 0x7fb310073010 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.425+0000 7fb323fff640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb31006ae70 0x7fb31006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.426+0000 7fb323fff640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb31006ae70 0x7fb31006d330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0012e0 tx=0x7fb31c0015b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.426+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1c7000b990 con 0x7f1c800ceb10 2023-11-24T08:05:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.427+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb314076770 con 0x7fb3240d3a40 2023-11-24T08:05:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.427+0000 7f1c7cff9640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c6006aea0 0x7f1c6006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.427+0000 7fb328a2d640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] conn(0x7fb310073010 0x7fb310075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.427+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c7008a100 con 0x7f1c800ceb10 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.428+0000 7f1c7cff9640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] conn(0x7f1c600730c0 0x7f1c600754e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.428+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c60075b90 con 0x7f1c600730c0 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.428+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7008a4b0 con 0x7f1c800ceb10 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.428+0000 7f1c77fff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c6006aea0 0x7f1c6006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.429+0000 7f1c7ffff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] conn(0x7f1c600730c0 0x7f1c600754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.430+0000 7f1c7ffff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] conn(0x7f1c600730c0 0x7f1c600754e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.430+0000 7f1c77fff640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c6006aea0 0x7f1c6006d360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c68036000 tx=0x7f1c6803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.430+0000 7fb328a2d640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] conn(0x7fb310073010 0x7fb310075430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.432+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 --> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f453c03e090 con 0x7f4528073090 2023-11-24T08:05:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.432+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== osd.7 v2:172.21.15.120:6819/46675067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1c60075b90 con 0x7f1c600730c0 2023-11-24T08:05:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.432+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== osd.6 v2:172.21.15.120:6816/498313091 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb310075ae0 con 0x7fb310073010 2023-11-24T08:05:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.436+0000 7f453affd640 1 -- 172.21.15.107:0/943866350 <== osd.0 v2:172.21.15.107:6807/1814653023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f454404ee40 con 0x7f4528073090 2023-11-24T08:05:01.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] conn(0x7f4528073090 msgr2=0x7f45280754b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6807/1814653023,v1:172.21.15.107:6809/1814653023] conn(0x7f4528073090 0x7f45280754b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f452806ae70 msgr2=0x7f452806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f452806ae70 0x7f452806d330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f453c0e2930 tx=0x7f4540009040 comp rx=0 tx=0).stop 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 msgr2=0x7f453c0e6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454b583640 1 --2- 172.21.15.107:0/943866350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f453c03c070 0x7f453c0e6e90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4534009990 tx=0x7f45340046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.437+0000 7f454ad82640 1 -- 172.21.15.107:0/943866350 reap_dead start 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.438+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 shutdown_connections 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.438+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 >> 172.21.15.107:0/943866350 conn(0x7f453c058eb0 msgr2=0x7f453c0398f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.438+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 shutdown_connections 2023-11-24T08:05:01.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.438+0000 7f454b583640 1 -- 172.21.15.107:0/943866350 wait complete. 2023-11-24T08:05:01.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.443+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 --> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f953c165b90 con 0x7f951c0730b0 2023-11-24T08:05:01.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.443+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 --> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd900cd410 con 0x7fbd74073060 2023-11-24T08:05:01.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.444+0000 7f9533fff640 1 -- 172.21.15.107:0/2860720159 <== osd.5 v2:172.21.15.120:6802/3047087772 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f952c00a6c0 con 0x7f951c0730b0 2023-11-24T08:05:01.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.444+0000 7fbd867fc640 1 -- 172.21.15.107:0/1257382672 <== osd.1 v2:172.21.15.107:6800/132161644 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbd8800f040 con 0x7fbd74073060 2023-11-24T08:05:01.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.445+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] conn(0x7f951c0730b0 msgr2=0x7f951c0754d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.445+0000 7f9542101640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.120:6802/3047087772,v1:172.21.15.120:6803/3047087772] conn(0x7f951c0730b0 0x7f951c0754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.445+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] conn(0x7fbd74073060 msgr2=0x7fbd74075480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.445+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6800/132161644,v1:172.21.15.107:6801/132161644] conn(0x7fbd74073060 0x7fbd74075480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd7406ae40 msgr2=0x7fbd7406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd7406ae40 0x7fbd7406d300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd90158040 tx=0x7fbd80009040 comp rx=0 tx=0).stop 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f951c06ae90 msgr2=0x7f951c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd900c9650 msgr2=0x7fbd90159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 --2- 172.21.15.107:0/1257382672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd900c9650 0x7fbd90159220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c033000 tx=0x7fbd8c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd96179640 1 -- 172.21.15.107:0/1257382672 reap_dead start 2023-11-24T08:05:01.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7f9542101640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f951c06ae90 0x7f951c06d350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9534036000 tx=0x7f9534030000 comp rx=0 tx=0).stop 2023-11-24T08:05:01.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 shutdown_connections 2023-11-24T08:05:01.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c162540 msgr2=0x7f953c163dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f9542101640 1 --2- 172.21.15.107:0/2860720159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f953c162540 0x7f953c163dc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f952400c500 tx=0x7f952400c9c0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.446+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 >> 172.21.15.107:0/1257382672 conn(0x7fbd900b0c50 msgr2=0x7fbd900caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f953bfff640 1 -- 172.21.15.107:0/2860720159 reap_dead start 2023-11-24T08:05:01.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 shutdown_connections 2023-11-24T08:05:01.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 shutdown_connections 2023-11-24T08:05:01.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 >> 172.21.15.107:0/2860720159 conn(0x7f953c0b0c50 msgr2=0x7f953c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7fbd97bdc640 1 -- 172.21.15.107:0/1257382672 wait complete. 2023-11-24T08:05:01.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 shutdown_connections 2023-11-24T08:05:01.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.447+0000 7f9542101640 1 -- 172.21.15.107:0/2860720159 wait complete. 2023-11-24T08:05:01.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.450+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 --> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa8c002cb0 con 0x7ffa88073090 2023-11-24T08:05:01.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.451+0000 7ffa9ffff640 1 -- 172.21.15.107:0/774150072 <== osd.2 v2:172.21.15.107:6816/4260350828 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffa8c002cb0 con 0x7ffa88073090 2023-11-24T08:05:01.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] conn(0x7ffa88073090 msgr2=0x7ffa880754b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6816/4260350828,v1:172.21.15.107:6817/4260350828] conn(0x7ffa88073090 0x7ffa880754b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa8806ae70 msgr2=0x7ffa8806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa8806ae70 0x7ffa8806d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa90000f60 tx=0x7ffa90007d30 comp rx=0 tx=0).stop 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 msgr2=0x7ffaa0052b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 --2- 172.21.15.107:0/774150072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaa00e3180 0x7ffaa0052b60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e870 tx=0x7ffa9800ed30 comp rx=0 tx=0).stop 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaa77fe640 1 -- 172.21.15.107:0/774150072 reap_dead start 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 shutdown_connections 2023-11-24T08:05:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.452+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 >> 172.21.15.107:0/774150072 conn(0x7ffaa003a9f0 msgr2=0x7ffaa003d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.453+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 --> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3c640b06c0 con 0x7f3c480730b0 2023-11-24T08:05:01.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.454+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 shutdown_connections 2023-11-24T08:05:01.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.454+0000 7ffaada36640 1 -- 172.21.15.107:0/774150072 wait complete. 2023-11-24T08:05:01.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.454+0000 7f3c5affd640 1 -- 172.21.15.107:0/2895244812 <== osd.3 v2:172.21.15.107:6818/1213738659 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3c5c00f040 con 0x7f3c480730b0 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.455+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 --> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa758002cb0 con 0x7fa764072fe0 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.455+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] conn(0x7f3c480730b0 msgr2=0x7f3c480754d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.455+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6818/1213738659,v1:172.21.15.107:6819/1213738659] conn(0x7f3c480730b0 0x7f3c480754d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c4806af10 msgr2=0x7f3c4806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c4806af10 0x7f3c4806d3d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c60008030 comp rx=0 tx=0).stop 2023-11-24T08:05:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c640b1ed0 msgr2=0x7f3c6415c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 --2- 172.21.15.107:0/2895244812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c640b1ed0 0x7f3c6415c910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c430 tx=0x7f3c5400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6a915640 1 -- 172.21.15.107:0/2895244812 reap_dead start 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 shutdown_connections 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 >> 172.21.15.107:0/2895244812 conn(0x7f3c6408af60 msgr2=0x7f3c6415fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 shutdown_connections 2023-11-24T08:05:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.456+0000 7f3c6c378640 1 -- 172.21.15.107:0/2895244812 wait complete. 2023-11-24T08:05:01.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa75ffff640 1 -- 172.21.15.107:0/947361584 <== osd.4 v2:172.21.15.120:6800/239423732 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa758002cb0 con 0x7fa764072fe0 2023-11-24T08:05:01.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] conn(0x7fa764072fe0 msgr2=0x7fa764075400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:6800/239423732,v1:172.21.15.120:6801/239423732] conn(0x7fa764072fe0 0x7fa764075400 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406ae40 msgr2=0x7fa76406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406ae40 0x7fa76406d300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa768006fd0 tx=0x7fa768008040 comp rx=0 tx=0).stop 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77815b440 msgr2=0x7fa77814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.457+0000 7fa77dddc640 1 --2- 172.21.15.107:0/947361584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77815b440 0x7fa77814d0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa770032000 tx=0x7fa770004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.458+0000 7fa777fff640 1 -- 172.21.15.107:0/947361584 reap_dead start 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.458+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 shutdown_connections 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.458+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 >> 172.21.15.107:0/947361584 conn(0x7fa7780b0c50 msgr2=0x7fa7780cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.458+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 shutdown_connections 2023-11-24T08:05:01.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.459+0000 7fa77dddc640 1 -- 172.21.15.107:0/947361584 wait complete. 2023-11-24T08:05:01.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.463+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 --> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c64002cb0 con 0x7f1c600730c0 2023-11-24T08:05:01.517 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:01.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T08:05:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.465+0000 7f1c7cff9640 1 -- 172.21.15.107:0/549195118 <== osd.7 v2:172.21.15.120:6819/46675067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c64002cb0 con 0x7f1c600730c0 2023-11-24T08:05:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.465+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] conn(0x7f1c600730c0 msgr2=0x7f1c600754e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.465+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.120:6819/46675067,v1:172.21.15.120:6820/46675067] conn(0x7f1c600730c0 0x7f1c600754e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c6006aea0 msgr2=0x7f1c6006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c6006aea0 0x7f1c6006d360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c68036000 tx=0x7f1c6803d040 comp rx=0 tx=0).stop 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 msgr2=0x7f1c801589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 --2- 172.21.15.107:0/549195118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c800ceb10 0x7f1c801589e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000d800 tx=0x7f1c7000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c7ffff640 1 -- 172.21.15.107:0/549195118 reap_dead start 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 shutdown_connections 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.466+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 >> 172.21.15.107:0/549195118 conn(0x7f1c8008af60 msgr2=0x7f1c800cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.467+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 shutdown_connections 2023-11-24T08:05:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.467+0000 7f1c85bf5640 1 -- 172.21.15.107:0/549195118 wait complete. 2023-11-24T08:05:01.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.469+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 --> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb304002c20 con 0x7fb310073010 2023-11-24T08:05:01.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.470+0000 7fb320ff9640 1 -- 172.21.15.107:0/2892056889 <== osd.6 v2:172.21.15.120:6816/498313091 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb304002c20 con 0x7fb310073010 2023-11-24T08:05:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] conn(0x7fb310073010 msgr2=0x7fb310075430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.120:6816/498313091,v1:172.21.15.120:6817/498313091] conn(0x7fb310073010 0x7fb310075430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb31006ae70 msgr2=0x7fb31006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb31006ae70 0x7fb31006d330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0012e0 tx=0x7fb31c0015b0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 msgr2=0x7fb32415f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb32a490640 1 --2- 172.21.15.107:0/2892056889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3240d3a40 0x7fb32415f310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb314033000 tx=0x7fb3140046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.471+0000 7fb328a2d640 1 -- 172.21.15.107:0/2892056889 reap_dead start 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.472+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 shutdown_connections 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.472+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 >> 172.21.15.107:0/2892056889 conn(0x7fb32408af60 msgr2=0x7fb3240b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.472+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 shutdown_connections 2023-11-24T08:05:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.472+0000 7fb32a490640 1 -- 172.21.15.107:0/2892056889 wait complete. 2023-11-24T08:05:01.523 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:01.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T08:05:01.524 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:01.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T08:05:01.525 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2023-11-24T08:05:01.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T08:05:01.526 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:01.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T08:05:01.527 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-11-24T08:05:01.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T08:05:01.528 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:01.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T08:05:01.529 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:01.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T08:05:01.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.848+0000 7fe3c7fff640 1 Processor -- start 2023-11-24T08:05:01.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.849+0000 7fe3c7fff640 1 -- start start 2023-11-24T08:05:01.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.851+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.852+0000 7fe3c7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 0x7fe3c0038990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.852+0000 7fe3c7fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c0006860 con 0x7fe3c00df3e0 2023-11-24T08:05:01.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.853+0000 7fe3c7fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0007730 con 0x7fe3c0054230 2023-11-24T08:05:01.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.853+0000 7fe3c6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.853+0000 7fe3c6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34386/0 (socket says 172.21.15.107:34386) 2023-11-24T08:05:01.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.853+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/3333502317 learned_addr learned my addr 172.21.15.107:0/3333502317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.853+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0050370 con 0x7fe3c004fa50 2023-11-24T08:05:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.854+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365735693 0 0) 0x7fe3c0006860 con 0x7fe3c00df3e0 2023-11-24T08:05:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.854+0000 7fe3c77fe640 1 --2- 172.21.15.107:0/3333502317 >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 0x7fe3c0038990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.854+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b0003520 con 0x7fe3c00df3e0 2023-11-24T08:05:01.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.855+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/3333502317 >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 msgr2=0x7fe3c0038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.855+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1317540410 0 0) 0x7fe3b0003520 con 0x7fe3c00df3e0 2023-11-24T08:05:01.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.855+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/3333502317 >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 0x7fe3c0038990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.856+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/3333502317 >> v1:172.21.15.120:6789/0 conn(0x7fe3c00df3e0 legacy=0x7fe3c0038280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.856+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/3333502317 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0006ab0 con 0x7fe3c004fa50 2023-11-24T08:05:01.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.857+0000 7fe3c77fe640 1 --2- 172.21.15.107:0/3333502317 >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 0x7fe3c0038990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:01.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.858+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/3333502317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009870 tx=0x7fe3b802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20c6e8ba429718af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.859+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc001670 con 0x7fe3c004fa50 2023-11-24T08:05:01.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.859+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8031560 con 0x7fe3c004fa50 2023-11-24T08:05:01.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.861+0000 7fe3c5ffb640 1 -- 172.21.15.107:0/3333502317 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc001670 con 0x7fe3c004fa50 2023-11-24T08:05:01.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.861+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 msgr2=0x7fe3c004fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.861+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/3333502317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009870 tx=0x7fe3b802b880 comp rx=0 tx=0).stop 2023-11-24T08:05:01.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.863+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 shutdown_connections 2023-11-24T08:05:01.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.863+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/3333502317 >> v2:172.21.15.107:3300/0 conn(0x7fe3c0054230 0x7fe3c0038990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.863+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/3333502317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c004fe30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.863+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 >> 172.21.15.107:0/3333502317 conn(0x7fe3c0009f70 msgr2=0x7fe3c000a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.863+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 shutdown_connections 2023-11-24T08:05:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.864+0000 7fe3c7fff640 1 -- 172.21.15.107:0/3333502317 wait complete. 2023-11-24T08:05:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.864+0000 7fb8fb59e640 1 Processor -- start 2023-11-24T08:05:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.864+0000 7fb8fb59e640 1 -- start start 2023-11-24T08:05:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8fb59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8ec058980 0x7fb8ec0534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8fb59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8ec056a70 0x7fb8ec053a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8fb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec057d50 con 0x7fb8ec0e76d0 2023-11-24T08:05:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8fb59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec006220 con 0x7fb8ec056a70 2023-11-24T08:05:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8fb59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec006500 con 0x7fb8ec058980 2023-11-24T08:05:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8f933a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8ec0e76d0 0x7fb8ec04c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59128/0 (socket says 172.21.15.107:59128) 2023-11-24T08:05:01.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.866+0000 7fb8f933a640 1 -- 172.21.15.107:0/4288398354 learned_addr learned my addr 172.21.15.107:0/4288398354 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.867+0000 7fb8f9b3b640 1 --2- 172.21.15.107:0/4288398354 >> v2:172.21.15.107:3300/0 conn(0x7fb8ec056a70 0x7fb8ec053a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.867+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/4288398354 >> v2:172.21.15.120:3300/0 conn(0x7fb8ec058980 0x7fb8ec0534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.868+0000 7fb8f8b39640 1 -- 172.21.15.107:0/4288398354 >> v2:172.21.15.107:3300/0 conn(0x7fb8ec056a70 msgr2=0x7fb8ec053a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.868+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/4288398354 >> v2:172.21.15.107:3300/0 conn(0x7fb8ec056a70 0x7fb8ec053a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.868+0000 7fb8f8b39640 1 -- 172.21.15.107:0/4288398354 >> v1:172.21.15.107:6789/0 conn(0x7fb8ec0e76d0 legacy=0x7fb8ec04c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.868+0000 7fb8f8b39640 1 -- 172.21.15.107:0/4288398354 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0067e0 con 0x7fb8ec058980 2023-11-24T08:05:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.869+0000 7fe3c7fff640 1 Processor -- start 2023-11-24T08:05:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.869+0000 7fe3c7fff640 1 -- start start 2023-11-24T08:05:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/4288398354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec058980 0x7fb8ec0534f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00f8d0 tx=0x7fb8f002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2abc5315b8f4b3d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fb8ebfff640 1 -- 172.21.15.107:0/4288398354 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00317c0 con 0x7fb8ec058980 2023-11-24T08:05:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fb8ebfff640 1 -- 172.21.15.107:0/4288398354 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f0031920 con 0x7fb8ec058980 2023-11-24T08:05:01.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fb8ebfff640 1 -- 172.21.15.107:0/4288398354 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0031cf0 con 0x7fb8ec058980 2023-11-24T08:05:01.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c00d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.870+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44284/0 (socket says 172.21.15.107:44284) 2023-11-24T08:05:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c67fc640 1 -- 172.21.15.107:0/2544745871 learned_addr learned my addr 172.21.15.107:0/2544745871 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c00df3e0 0x7fe3c0053b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00e0f50 con 0x7fe3c0054230 2023-11-24T08:05:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.871+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00d64e0 con 0x7fe3c00df3e0 2023-11-24T08:05:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00e11a0 con 0x7fe3c004fa50 2023-11-24T08:05:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 -- 172.21.15.107:0/4288398354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec058980 msgr2=0x7fb8ec0534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/4288398354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec058980 0x7fb8ec0534f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00f8d0 tx=0x7fb8f002b860 comp rx=0 tx=0).stop 2023-11-24T08:05:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c00d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fe3c77fe640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c00df3e0 0x7fe3c0053b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 -- 172.21.15.107:0/4288398354 shutdown_connections 2023-11-24T08:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/4288398354 >> v2:172.21.15.107:3300/0 conn(0x7fb8ec056a70 0x7fb8ec053a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/4288398354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec058980 0x7fb8ec0534f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 -- 172.21.15.107:0/4288398354 >> 172.21.15.107:0/4288398354 conn(0x7fb8ec058ea0 msgr2=0x7fb8ec059280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 -- 172.21.15.107:0/4288398354 shutdown_connections 2023-11-24T08:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.872+0000 7fb8fb59e640 1 -- 172.21.15.107:0/4288398354 wait complete. 2023-11-24T08:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c00df3e0 msgr2=0x7fe3c0053b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c00df3e0 0x7fe3c0053b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 msgr2=0x7fe3c00535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80682a0 con 0x7fe3c004fa50 2023-11-24T08:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fe3c67fc640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fd2588c8640 1 Processor -- start 2023-11-24T08:05:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.874+0000 7fd2588c8640 1 -- start start 2023-11-24T08:05:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.875+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c00d5fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8033000 tx=0x7fe3b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.875+0000 7fb8fb59e640 1 Processor -- start 2023-11-24T08:05:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.875+0000 7fb8fb59e640 1 -- start start 2023-11-24T08:05:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8031ba0 con 0x7fe3c004fa50 2023-11-24T08:05:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd254159670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2540ae310 0x7fd2540b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25415d6b0 con 0x7fd25415d0f0 2023-11-24T08:05:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25415d990 con 0x7fd2540ae310 2023-11-24T08:05:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd2588c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25415dc70 con 0x7fd25406e640 2023-11-24T08:05:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7f33e46ee640 1 Processor -- start 2023-11-24T08:05:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd25359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd25359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44286/0 (socket says 172.21.15.107:44286) 2023-11-24T08:05:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b80094d0 con 0x7fe3c004fa50 2023-11-24T08:05:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.876+0000 7fd25359e640 1 -- 172.21.15.107:0/326974908 learned_addr learned my addr 172.21.15.107:0/326974908 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec056a70 0x7fb8ec052950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fd24a59c640 1 --2- 172.21.15.107:0/326974908 >> v2:172.21.15.120:3300/0 conn(0x7fd2540ae310 0x7fd2540b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0e76d0 0x7fb8ec050df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fd252d9d640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd254159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00e1760 con 0x7fe3c004fa50 2023-11-24T08:05:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b803f070 con 0x7fe3c004fa50 2023-11-24T08:05:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8003610 con 0x7fe3c004fa50 2023-11-24T08:05:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0d7860 con 0x7fb8ec058980 2023-11-24T08:05:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0518b0 con 0x7fb8ec056a70 2023-11-24T08:05:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.877+0000 7fb8fb59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec051b90 con 0x7fb8ec0e76d0 2023-11-24T08:05:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fb8f933a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec056a70 0x7fb8ec052950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fb8f8b39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fb8f8b39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44294/0 (socket says 172.21.15.107:44294) 2023-11-24T08:05:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fb8f8b39640 1 -- 172.21.15.107:0/3619761731 learned_addr learned my addr 172.21.15.107:0/3619761731 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fb8f9b3b640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0e76d0 0x7fb8ec050df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7f33e46ee640 1 -- start start 2023-11-24T08:05:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fd24a59c640 1 -- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 msgr2=0x7fd254159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fd24a59c640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd254159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fb8f8b39640 1 -- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0e76d0 msgr2=0x7fb8ec050df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0e76d0 0x7fb8ec050df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fb8f8b39640 1 -- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec056a70 msgr2=0x7fb8ec052950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec056a70 0x7fb8ec052950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fb8f8b39640 1 -- 172.21.15.107:0/3619761731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009400 con 0x7fb8ec058980 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 0x7f33d4053e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33d40543c0 0x7f33d403c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d403c610 0x7f33d403caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d4057d50 con 0x7f33d40543c0 2023-11-24T08:05:01.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d4058030 con 0x7f33d40e3370 2023-11-24T08:05:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e46ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4058310 con 0x7f33d403c610 2023-11-24T08:05:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fd24a59c640 1 -- 172.21.15.107:0/326974908 >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 msgr2=0x7fd2540b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fd24a59c640 1 --2- 172.21.15.107:0/326974908 >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.879+0000 7fd24a59c640 1 -- 172.21.15.107:0/326974908 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25415df50 con 0x7fd2540ae310 2023-11-24T08:05:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.878+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a8005250 con 0x7fe3c004fa50 2023-11-24T08:05:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e248a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 0x7f33d4053e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7fd25359e640 1 --2- 172.21.15.107:0/326974908 >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.881+0000 7fd24a59c640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2540ae310 0x7fd2540b0b40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2540b16e0 tx=0x7fd24402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f29f1e7f94138 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e2c8b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d403c610 0x7f33d403caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7f33e1c89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33d40543c0 0x7f33d403c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.881+0000 7f33e1c89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33d40543c0 0x7f33d403c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44296/0 (socket says 172.21.15.107:44296) 2023-11-24T08:05:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.881+0000 7f33e1c89640 1 -- 172.21.15.107:0/2247495826 learned_addr learned my addr 172.21.15.107:0/2247495826 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.881+0000 7fd25259c640 1 -- 172.21.15.107:0/326974908 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244031520 con 0x7fd2540ae310 2023-11-24T08:05:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.880+0000 7fb8f8b39640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0034040 tx=0x7fb8f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0032860 con 0x7fb8ec058980 2023-11-24T08:05:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f00329e0 con 0x7fb8ec058980 2023-11-24T08:05:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0032d90 con 0x7fb8ec058980 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e1c89640 1 -- 172.21.15.107:0/2247495826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d403c610 msgr2=0x7f33d403caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e1c89640 1 --2- 172.21.15.107:0/2247495826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d403c610 0x7f33d403caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e1c89640 1 -- 172.21.15.107:0/2247495826 >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 msgr2=0x7f33d4053e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e1c89640 1 --2- 172.21.15.107:0/2247495826 >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 0x7f33d4053e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e1c89640 1 -- 172.21.15.107:0/2247495826 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40585f0 con 0x7f33d40543c0 2023-11-24T08:05:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec051e70 con 0x7fb8ec058980 2023-11-24T08:05:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7f33e248a640 1 --2- 172.21.15.107:0/2247495826 >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 0x7f33d4053e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fd25259c640 1 -- 172.21.15.107:0/326974908 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd244031680 con 0x7fd2540ae310 2023-11-24T08:05:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec052430 con 0x7fb8ec058980 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.883+0000 7fd2588c8640 1 -- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2540ae310 msgr2=0x7fd2540b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.883+0000 7fd25259c640 1 -- 172.21.15.107:0/326974908 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244031a50 con 0x7fd2540ae310 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.883+0000 7fd2588c8640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2540ae310 0x7fd2540b0b40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2540b16e0 tx=0x7fd24402b7c0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 -- 172.21.15.107:0/326974908 shutdown_connections 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 --2- 172.21.15.107:0/326974908 >> v2:172.21.15.107:3300/0 conn(0x7fd25415d0f0 0x7fd2540b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2540ae310 0x7fd2540b0b40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 --2- 172.21.15.107:0/326974908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd254159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 -- 172.21.15.107:0/326974908 >> 172.21.15.107:0/326974908 conn(0x7fd2540cbb40 msgr2=0x7fd2540cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.882+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe3b8065d50 con 0x7fe3c004fa50 2023-11-24T08:05:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.883+0000 7fe3affff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b802f050 con 0x7fe3c004fa50 2023-11-24T08:05:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 -- 172.21.15.107:0/326974908 shutdown_connections 2023-11-24T08:05:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fd2588c8640 1 -- 172.21.15.107:0/326974908 wait complete. 2023-11-24T08:05:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7f33e1c89640 1 --2- 172.21.15.107:0/2247495826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40543c0 0x7f33d403c0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f33d4011990 tx=0x7f33cc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73ada2894a998f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fe3c67fc640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.884+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec052620 con 0x7fb8ec058980 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.885+0000 7f33e1488640 1 -- 172.21.15.107:0/2247495826 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031880 con 0x7f33d40543c0 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.885+0000 7f33e1488640 1 -- 172.21.15.107:0/2247495826 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc0319e0 con 0x7f33d40543c0 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.885+0000 7f33e1488640 1 -- 172.21.15.107:0/2247495826 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031d70 con 0x7f33d40543c0 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.885+0000 7fe3c67fc640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 0x7fe3b406d300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00e0f00 tx=0x7fe3bc000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.886+0000 7f33e46ee640 1 -- 172.21.15.107:0/2247495826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40543c0 msgr2=0x7f33d403c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.886+0000 7f33e46ee640 1 --2- 172.21.15.107:0/2247495826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40543c0 0x7f33d403c0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f33d4011990 tx=0x7f33cc02b920 comp rx=0 tx=0).stop 2023-11-24T08:05:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.887+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb8f003b3f0 con 0x7fb8ec058980 2023-11-24T08:05:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.887+0000 7fd2588c8640 1 Processor -- start 2023-11-24T08:05:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.887+0000 7fd2588c8640 1 -- start start 2023-11-24T08:05:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 -- 172.21.15.107:0/2247495826 shutdown_connections 2023-11-24T08:05:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 --2- 172.21.15.107:0/2247495826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d403c610 0x7f33d403caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 --2- 172.21.15.107:0/2247495826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40543c0 0x7f33d403c0d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 --2- 172.21.15.107:0/2247495826 >> v2:172.21.15.120:3300/0 conn(0x7f33d40e3370 0x7f33d4053e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 -- 172.21.15.107:0/2247495826 >> 172.21.15.107:0/2247495826 conn(0x7f33d403a990 msgr2=0x7f33d403ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 -- 172.21.15.107:0/2247495826 shutdown_connections 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7f33e46ee640 1 -- 172.21.15.107:0/2247495826 wait complete. 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.888+0000 7fb8e9ffb640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 0x7fb8d806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.889+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8f00aafa0 con 0x7fb8ec058980 2023-11-24T08:05:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.889+0000 7fd2588c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd2540c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.889+0000 7fd2588c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 0x7fd254159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.889+0000 7fb8f933a640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 0x7fb8d806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd2588c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd25415d0f0 0x7fd254159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd252d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd2540c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd2588c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25409a9c0 con 0x7fd2540ae310 2023-11-24T08:05:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd24a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 0x7fd254159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7f33e46ee640 1 Processor -- start 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7f33e46ee640 1 -- start start 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd252d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd2540c5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34436/0 (socket says 172.21.15.107:34436) 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd252d9d640 1 -- 172.21.15.107:0/901500954 learned_addr learned my addr 172.21.15.107:0/901500954 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2540b4600 con 0x7fd25415d0f0 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.890+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2540b4320 con 0x7fd25406e640 2023-11-24T08:05:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fb8f933a640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 0x7fb8d806d400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4074d90 tx=0x7fb8f406d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd25359e640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd25415d0f0 0x7fd254159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd24a59c640 1 -- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 msgr2=0x7fd2540c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7f33e46ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7f33e46ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d40543c0 0x7f33d40d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e46ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd24a59c640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd2540c5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd24a59c640 1 -- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd25415d0f0 msgr2=0x7fd254159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd24a59c640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd25415d0f0 0x7fd254159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.891+0000 7fd24a59c640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c00bd40 con 0x7fd2540ae310 2023-11-24T08:05:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e2c8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e2c8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44314/0 (socket says 172.21.15.107:44314) 2023-11-24T08:05:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e248a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e248a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58616/0 (socket says 172.21.15.107:58616) 2023-11-24T08:05:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f73174ea640 1 Processor -- start 2023-11-24T08:05:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7fd24a59c640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 0x7fd254159670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd244033000 tx=0x7fd2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f73174ea640 1 -- start start 2023-11-24T08:05:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e2c8b640 1 -- 172.21.15.107:0/1835368473 learned_addr learned my addr 172.21.15.107:0/1835368473 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d404c2d0 con 0x7f33d40e3370 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d404cc00 con 0x7f33d403c610 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.892+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4052db0 con 0x7f33d40543c0 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 -- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d40543c0 msgr2=0x7f33d40d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d40543c0 0x7f33d40d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 -- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 msgr2=0x7f33d404b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0094d0 con 0x7f33d40e3370 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e248a640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2023-11-24T08:05:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7f33e2c8b640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33dc079ba0 tx=0x7f33dc06b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7fb14bf1d640 1 Processor -- start 2023-11-24T08:05:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.893+0000 7fb14bf1d640 1 -- start start 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73174ea640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73101419c0 0x7f731006d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73174ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f731006dd50 0x7f73100cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73174ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731006ee10 con 0x7f73100cc0f0 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73174ea640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100cc580 con 0x7f73101419c0 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73174ea640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731006e290 con 0x7f731006dd50 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73164e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73101419c0 0x7f731006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73164e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73101419c0 0x7f731006d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44324/0 (socket says 172.21.15.107:44324) 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f7316ce9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f73100cc0f0 0x7f73100719a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59130/0 (socket says 172.21.15.107:59130) 2023-11-24T08:05:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.894+0000 7f73164e8640 1 -- 172.21.15.107:0/2247779972 learned_addr learned my addr 172.21.15.107:0/2247779972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f7315ce7640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f731006dd50 0x7f73100cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fe4e7b1d640 1 Processor -- start 2023-11-24T08:05:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0723c0 con 0x7f33d40e3370 2023-11-24T08:05:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc072520 con 0x7f33d40e3370 2023-11-24T08:05:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244031520 con 0x7fd2540ae310 2023-11-24T08:05:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd244009400 con 0x7fd2540ae310 2023-11-24T08:05:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd244033d90 con 0x7fd2540ae310 2023-11-24T08:05:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244031520 con 0x7fd2540ae310 2023-11-24T08:05:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc072890 con 0x7f33d40e3370 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f73164e8640 1 -- 172.21.15.107:0/2247779972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f731006dd50 msgr2=0x7f73100cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f73164e8640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f731006dd50 0x7f73100cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f73164e8640 1 -- 172.21.15.107:0/2247779972 >> v1:172.21.15.107:6789/0 conn(0x7f73100cc0f0 legacy=0x7f73100719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7f73164e8640 1 -- 172.21.15.107:0/2247779972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731006e570 con 0x7f73101419c0 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d404ba30 con 0x7f33d40e3370 2023-11-24T08:05:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d404bff0 con 0x7f33d40e3370 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fb14bf1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 0x7fb1440d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7fb14bf1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1440ce010 0x7fb1440ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7fb14bf1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1440d1070 con 0x7fb1441609e0 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7fb14bf1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440d3b50 con 0x7fb1440d31a0 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7fb14bf1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440cedc0 con 0x7fb1440ce010 2023-11-24T08:05:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.895+0000 7fe4e7b1d640 1 -- start start 2023-11-24T08:05:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.896+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25409a5e0 con 0x7fd2540ae310 2023-11-24T08:05:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb1494b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 0x7fb1440d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb14a4ba640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1440ce010 0x7fb1440ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb1494b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 0x7fb1440d3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44340/0 (socket says 172.21.15.107:44340) 2023-11-24T08:05:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb1494b8640 1 -- 172.21.15.107:0/4150230468 learned_addr learned my addr 172.21.15.107:0/4150230468 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d404c9d0 con 0x7f33d40e3370 2023-11-24T08:05:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8f00b0060 con 0x7fb8ec058980 2023-11-24T08:05:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb14a4ba640 1 -- 172.21.15.107:0/4150230468 >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 msgr2=0x7fb1440d3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb14a4ba640 1 --2- 172.21.15.107:0/4150230468 >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 0x7fb1440d3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb14a4ba640 1 -- 172.21.15.107:0/4150230468 >> v1:172.21.15.107:6789/0 conn(0x7fb1441609e0 legacy=0x7fb14406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.897+0000 7fb14a4ba640 1 -- 172.21.15.107:0/4150230468 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440cf0a0 con 0x7fb1440ce010 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fe4e7b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4e00cc470 0x7fe4e00abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fe4e7b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e00cfb60 0x7fe4e00b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fe4e7b1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e00ce270 con 0x7fe4e00ce640 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fe4e7b1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e006c560 con 0x7fe4e00cc470 2023-11-24T08:05:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7fe4e7b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e006d0a0 con 0x7fe4e00cfb60 2023-11-24T08:05:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fb14a4ba640 1 --2- 172.21.15.107:0/4150230468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1440ce010 0x7fb1440ce430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1440cddd0 tx=0x7fb13c02be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b15056aedd975ea8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fe4e58b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4e00cc470 0x7fe4e00abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3b8075ee0 con 0x7fe3c004fa50 2023-11-24T08:05:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd24403a040 con 0x7fd2540ae310 2023-11-24T08:05:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.898+0000 7f73164e8640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73101419c0 0x7f731006d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f730c009870 tx=0x7f730c02fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ba6a7d5863c653 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fe4e58b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4e00cc470 0x7fe4e00abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44342/0 (socket says 172.21.15.107:44342) 2023-11-24T08:05:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f33dc074940 con 0x7f33d40e3370 2023-11-24T08:05:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.899+0000 7fe4e58b9640 1 -- 172.21.15.107:0/1034370770 learned_addr learned my addr 172.21.15.107:0/1034370770 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fe4e60ba640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e00cfb60 0x7fe4e00b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fb148cb7640 1 -- 172.21.15.107:0/4150230468 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c02cbe0 con 0x7fb1440ce010 2023-11-24T08:05:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fb148cb7640 1 -- 172.21.15.107:0/4150230468 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c02cd40 con 0x7fb1440ce010 2023-11-24T08:05:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fb148cb7640 1 -- 172.21.15.107:0/4150230468 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c034660 con 0x7fb1440ce010 2023-11-24T08:05:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238005250 con 0x7fd2540ae310 2023-11-24T08:05:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.901+0000 7fd24bfff640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 0x7fd23406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.901+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2440ab000 con 0x7fd2540ae310 2023-11-24T08:05:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.901+0000 7f33d2ffd640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 0x7f33c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.901+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33dc103460 con 0x7f33d40e3370 2023-11-24T08:05:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fe4e58b9640 1 -- 172.21.15.107:0/1034370770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e00cfb60 msgr2=0x7fe4e00b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fe4e58b9640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e00cfb60 0x7fe4e00b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.902+0000 7fd252d9d640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 0x7fd23406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fe4e58b9640 1 -- 172.21.15.107:0/1034370770 >> v1:172.21.15.120:6789/0 conn(0x7fe4e00ce640 legacy=0x7fe4e00cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.900+0000 7fe4e58b9640 1 -- 172.21.15.107:0/1034370770 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00b4200 con 0x7fe4e00cc470 2023-11-24T08:05:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.902+0000 7fe4e58b9640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cc470 0x7fe4e00abfc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009e50 tx=0x7fe4d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303d89b56f4263bf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.902+0000 7fb14bf1d640 1 -- 172.21.15.107:0/4150230468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1440ce010 msgr2=0x7fb1440ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.902+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/4150230468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1440ce010 0x7fb1440ce430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1440cddd0 tx=0x7fb13c02be30 comp rx=0 tx=0).stop 2023-11-24T08:05:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 -- 172.21.15.107:0/4150230468 shutdown_connections 2023-11-24T08:05:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fd252d9d640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 0x7fd23406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006fd0 tx=0x7fd23c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/4150230468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1440ce010 0x7fb1440ce430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/4150230468 >> v2:172.21.15.107:3300/0 conn(0x7fb1440d31a0 0x7fb1440d3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 -- 172.21.15.107:0/4150230468 >> 172.21.15.107:0/4150230468 conn(0x7fb14408af60 msgr2=0x7fb14409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 -- 172.21.15.107:0/4150230468 shutdown_connections 2023-11-24T08:05:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7fb14bf1d640 1 -- 172.21.15.107:0/4150230468 wait complete. 2023-11-24T08:05:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.903+0000 7f33e248a640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 0x7f33c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.904+0000 7f33e248a640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 0x7f33c006d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006fd0 tx=0x7f33d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.904+0000 7effc8533640 1 Processor -- start 2023-11-24T08:05:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.905+0000 7effc8533640 1 -- start start 2023-11-24T08:05:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7fb14bf1d640 1 Processor -- start 2023-11-24T08:05:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc8533640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cf450 0x7effc00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc8533640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7effc00cdfb0 0x7effc00ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc8533640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00cece0 con 0x7effc015c7f0 2023-11-24T08:05:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc8533640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc0156010 con 0x7effc00cdfb0 2023-11-24T08:05:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc8533640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc006ce30 con 0x7effc00cf450 2023-11-24T08:05:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7fb14bf1d640 1 -- start start 2023-11-24T08:05:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc62cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7effc015c7f0 0x7effc006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59144/0 (socket says 172.21.15.107:59144) 2023-11-24T08:05:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc62cf640 1 -- 172.21.15.107:0/672136702 learned_addr learned my addr 172.21.15.107:0/672136702 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc5ace640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cf450 0x7effc00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7f73154e6640 1 -- 172.21.15.107:0/2247779972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c030b40 con 0x7f73101419c0 2023-11-24T08:05:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.907+0000 7effc6ad0640 1 --2- 172.21.15.107:0/672136702 >> v2:172.21.15.120:3300/0 conn(0x7effc00cdfb0 0x7effc00ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7fe4e48b7640 1 -- 172.21.15.107:0/1034370770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d802ca80 con 0x7fe4e00cc470 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33dc0ce520 con 0x7f33d40e3370 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7fb14bf1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 0x7fb14415a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb14bf1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 0x7fb1440b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb14bf1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1441609e0 0x7fb1440b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb14bf1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1441fb2e0 con 0x7fb1440d31a0 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb14bf1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1441fb580 con 0x7fb1441609e0 2023-11-24T08:05:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb14bf1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1441fb860 con 0x7fb1440ce010 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7effc6ad0640 1 -- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cf450 msgr2=0x7effc00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7effc6ad0640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cf450 0x7effc00cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7effc6ad0640 1 -- 172.21.15.107:0/672136702 >> v1:172.21.15.107:6789/0 conn(0x7effc015c7f0 legacy=0x7effc006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.908+0000 7effc6ad0640 1 -- 172.21.15.107:0/672136702 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc006d040 con 0x7effc00cdfb0 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.909+0000 7fb149cb9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 0x7fb14415a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7effc6ad0640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cdfb0 0x7effc00ce440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effb8009d90 tx=0x7effb802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d671014fd059bd94 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb14a4ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1441609e0 0x7fb1440b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb149cb9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 0x7fb14415a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34464/0 (socket says 172.21.15.107:34464) 2023-11-24T08:05:01.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb149cb9640 1 -- 172.21.15.107:0/1958378306 learned_addr learned my addr 172.21.15.107:0/1958378306 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb1494b8640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 0x7fb1440b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7effc52cd640 1 -- 172.21.15.107:0/672136702 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb802cc60 con 0x7effc00cdfb0 2023-11-24T08:05:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7effc52cd640 1 -- 172.21.15.107:0/672136702 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb802cdc0 con 0x7effc00cdfb0 2023-11-24T08:05:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7effc52cd640 1 -- 172.21.15.107:0/672136702 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8034660 con 0x7effc00cdfb0 2023-11-24T08:05:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb1494b8640 1 -- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 msgr2=0x7fb14415a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.910+0000 7fb1494b8640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 0x7fb14415a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7fb1494b8640 1 -- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1441609e0 msgr2=0x7fb1440b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7fb1494b8640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1441609e0 0x7fb1440b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7fb1494b8640 1 -- 172.21.15.107:0/1958378306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c009a70 con 0x7fb1440d31a0 2023-11-24T08:05:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7fb1494b8640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 0x7fb1440b48d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb14000e850 tx=0x7fb14000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 -- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cdfb0 msgr2=0x7effc00ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cdfb0 0x7effc00ce440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effb8009d90 tx=0x7effb802beb0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 -- 172.21.15.107:0/672136702 shutdown_connections 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cdfb0 0x7effc00ce440 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 --2- 172.21.15.107:0/672136702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cf450 0x7effc00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 -- 172.21.15.107:0/672136702 >> 172.21.15.107:0/672136702 conn(0x7effc006a920 msgr2=0x7effc006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.911+0000 7effc8533640 1 -- 172.21.15.107:0/672136702 shutdown_connections 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.912+0000 7effc8533640 1 -- 172.21.15.107:0/672136702 wait complete. 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.913+0000 7effc8533640 1 Processor -- start 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.913+0000 7effc8533640 1 -- start start 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cdfb0 0x7effc00b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cf450 0x7effc0158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0159d00 con 0x7effc015c7f0 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0156f40 con 0x7effc00cf450 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.914+0000 7effc8533640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc0143c30 con 0x7effc00cdfb0 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44364/0 (socket says 172.21.15.107:44364) 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 -- 172.21.15.107:0/3164036129 learned_addr learned my addr 172.21.15.107:0/3164036129 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc62cf640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cdfb0 0x7effc00b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 -- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cdfb0 msgr2=0x7effc00b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cdfb0 0x7effc00b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 -- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cf450 msgr2=0x7effc0158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cf450 0x7effc0158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.915+0000 7effc6ad0640 1 -- 172.21.15.107:0/3164036129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8009a70 con 0x7effc015c7f0 2023-11-24T08:05:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.916+0000 7effc6ad0640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7effb8009d90 tx=0x7effb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7f73154e6640 1 -- 172.21.15.107:0/2247779972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c030ca0 con 0x7f73101419c0 2023-11-24T08:05:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fe4e48b7640 1 -- 172.21.15.107:0/1034370770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d802cbe0 con 0x7fe4e00cc470 2023-11-24T08:05:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fe4e48b7640 1 -- 172.21.15.107:0/1034370770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80343f0 con 0x7fe4e00cc470 2023-11-24T08:05:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2440761a0 con 0x7fd2540ae310 2023-11-24T08:05:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7f73154e6640 1 -- 172.21.15.107:0/2247779972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c030b40 con 0x7f73101419c0 2023-11-24T08:05:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14000c940 con 0x7fb1440d31a0 2023-11-24T08:05:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14000caa0 con 0x7fb1440d31a0 2023-11-24T08:05:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14000ce10 con 0x7fb1440d31a0 2023-11-24T08:05:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8040650 con 0x7effc015c7f0 2023-11-24T08:05:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb80407b0 con 0x7effc015c7f0 2023-11-24T08:05:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8040b40 con 0x7effc015c7f0 2023-11-24T08:05:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc01596d0 con 0x7effc015c7f0 2023-11-24T08:05:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.918+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0159920 con 0x7effc015c7f0 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7f73174ea640 1 -- 172.21.15.107:0/2247779972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73101419c0 msgr2=0x7f731006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7f73174ea640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73101419c0 0x7f731006d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f730c009870 tx=0x7f730c02fd90 comp rx=0 tx=0).stop 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7f73174ea640 1 -- 172.21.15.107:0/2247779972 shutdown_connections 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7f73174ea640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f731006dd50 0x7f73100cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/1034370770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cc470 msgr2=0x7fe4e00abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cc470 0x7fe4e00abfc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009e50 tx=0x7fe4d802bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7f73174ea640 1 --2- 172.21.15.107:0/2247779972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73101419c0 0x7f731006d810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7f73174ea640 1 -- 172.21.15.107:0/2247779972 >> 172.21.15.107:0/2247779972 conn(0x7f73100b0bc0 msgr2=0x7f73100b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/1034370770 shutdown_connections 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e00cfb60 0x7fe4e00b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/1034370770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cc470 0x7fe4e00abfc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/1034370770 >> 172.21.15.107:0/1034370770 conn(0x7fe4e006a920 msgr2=0x7fe4e006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc0159e40 con 0x7effc015c7f0 2023-11-24T08:05:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1441fbb40 con 0x7fb1440d31a0 2023-11-24T08:05:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.919+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14415ae50 con 0x7fb1440d31a0 2023-11-24T08:05:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/1034370770 shutdown_connections 2023-11-24T08:05:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.920+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/1034370770 wait complete. 2023-11-24T08:05:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.921+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb144067470 con 0x7fb1440d31a0 2023-11-24T08:05:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.923+0000 7fe4e7b1d640 1 Processor -- start 2023-11-24T08:05:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.923+0000 7fe4e7b1d640 1 -- start start 2023-11-24T08:05:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.923+0000 7f73174ea640 1 -- 172.21.15.107:0/2247779972 shutdown_connections 2023-11-24T08:05:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.924+0000 7f73174ea640 1 -- 172.21.15.107:0/2247779972 wait complete. 2023-11-24T08:05:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.923+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7effb8040ca0 con 0x7effc015c7f0 2023-11-24T08:05:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.924+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb140013470 con 0x7fb1440d31a0 2023-11-24T08:05:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.924+0000 7effb6ffd640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 0x7effa406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.925+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effb80ac570 con 0x7effc015c7f0 2023-11-24T08:05:01.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.926+0000 7fb13a7fc640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 0x7fb12806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.925+0000 7effc62cf640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 0x7effa406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.926+0000 7f73174ea640 1 Processor -- start 2023-11-24T08:05:01.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7f73174ea640 1 -- start start 2023-11-24T08:05:01.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fb149cb9640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 0x7fb12806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cfb60 0x7fe4e01585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e0099140 0x7fe4e0159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e015a760 con 0x7fe4e00cfb60 2023-11-24T08:05:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e015aa40 con 0x7fe4e00ce640 2023-11-24T08:05:01.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e7b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e015ad20 con 0x7fe4e0099140 2023-11-24T08:05:01.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e58b9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e58b9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58658/0 (socket says 172.21.15.107:58658) 2023-11-24T08:05:01.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.927+0000 7fe4e58b9640 1 -- 172.21.15.107:0/3057509843 learned_addr learned my addr 172.21.15.107:0/3057509843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e60ba640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e0099140 0x7fe4e0159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e58b9640 1 -- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e0099140 msgr2=0x7fe4e0159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7effc62cf640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 0x7effa406d3d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7effbc007c30 tx=0x7effbc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e58b9640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e0099140 0x7fe4e0159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e58b9640 1 -- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cfb60 msgr2=0x7fe4e01585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e58b9640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cfb60 0x7fe4e01585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fe4e58b9640 1 -- 172.21.15.107:0/3057509843 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8009b30 con 0x7fe4e00ce640 2023-11-24T08:05:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.928+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb140058d60 con 0x7fb1440d31a0 2023-11-24T08:05:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7f73174ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7fb149cb9640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 0x7fb12806d350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1340099a0 tx=0x7fb1340090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7fe4e58b9640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4d802c470 tx=0x7fe4d8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7f73174ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73100cc0f0 0x7f73100aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8009510 con 0x7fe4e00ce640 2023-11-24T08:05:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7f73174ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73101419c0 0x7f73100aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d803e450 con 0x7fe4e00ce640 2023-11-24T08:05:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.929+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d803e800 con 0x7fe4e00ce640 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73164e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73164e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44382/0 (socket says 172.21.15.107:44382) 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73164e8640 1 -- 172.21.15.107:0/929797797 learned_addr learned my addr 172.21.15.107:0/929797797 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f7315ce7640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73100cc0f0 0x7f73100aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effb80777b0 con 0x7effc015c7f0 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7310200210 con 0x7f731006dd50 2023-11-24T08:05:01.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73101f32e0 con 0x7f73101419c0 2023-11-24T08:05:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73101f53a0 con 0x7f73100cc0f0 2023-11-24T08:05:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.930+0000 7f7316ce9640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73101419c0 0x7f73100aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e015b000 con 0x7fe4e00ce640 2023-11-24T08:05:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e0158b00 con 0x7fe4e00ce640 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7f73164e8640 1 -- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73100cc0f0 msgr2=0x7f73100aaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7f73164e8640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73100cc0f0 0x7f73100aaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7f73164e8640 1 -- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73101419c0 msgr2=0x7f73100aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7f73164e8640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73101419c0 0x7f73100aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.931+0000 7f73164e8640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0094d0 con 0x7f731006dd50 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.932+0000 7f73164e8640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f730c02fdd0 tx=0x7f730c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.933+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e015b520 con 0x7fe4e00ce640 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.935+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe4d802cd50 con 0x7fe4e00ce640 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.936+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c003c40 con 0x7f731006dd50 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.936+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73101f4790 con 0x7f731006dd50 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.936+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73101f4d50 con 0x7f731006dd50 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.937+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c003da0 con 0x7f731006dd50 2023-11-24T08:05:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.937+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c003c40 con 0x7f731006dd50 2023-11-24T08:05:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.937+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1400550a0 con 0x7fb1440d31a0 2023-11-24T08:05:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.937+0000 7fe4d27fc640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 0x7fe4c406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.937+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4d80ac7e0 con 0x7fe4e00ce640 2023-11-24T08:05:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.939+0000 7fe4e50b8640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 0x7fe4c406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.940+0000 7fe4e50b8640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 0x7fe4c406d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0160900 tx=0x7fe4dc0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.939+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8005250 con 0x7f731006dd50 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.941+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f730c038440 con 0x7f731006dd50 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.942+0000 7f7306ffd640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 0x7f72f406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.942+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f730c0ab660 con 0x7f731006dd50 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.944+0000 7f7315ce7640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 0x7f72f406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.945+0000 7f7315ce7640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 0x7f72f406d360 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7310140fc0 tx=0x7f7300008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.950+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f730c076690 con 0x7f731006dd50 2023-11-24T08:05:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:01.951+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d8077870 con 0x7fe4e00ce640 2023-11-24T08:05:02.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.302+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb8ec005350 con 0x7fb8ec058980 2023-11-24T08:05:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.303+0000 7fb8e9ffb640 1 -- 172.21.15.107:0/3619761731 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8f00b0020 con 0x7fb8ec058980 2023-11-24T08:05:02.306 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2023-11-24T08:05:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.307+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 msgr2=0x7fb8d806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.307+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 0x7fb8d806d400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4074d90 tx=0x7fb8f406d040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.307+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 msgr2=0x7fb8ec0d7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.307+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0034040 tx=0x7fb8f00046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe3a80057d0 con 0x7fe3c004fa50 2023-11-24T08:05:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 shutdown_connections 2023-11-24T08:05:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806af40 0x7fb8d806d400 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0e76d0 0x7fb8ec050df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec058980 0x7fb8ec0d7320 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.309+0000 7fe3affff640 1 -- 172.21.15.107:0/2544745871 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe3b8041070 con 0x7fe3c004fa50 2023-11-24T08:05:02.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 --2- 172.21.15.107:0/3619761731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec056a70 0x7fb8ec052950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.308+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 >> 172.21.15.107:0/3619761731 conn(0x7fb8ec058ea0 msgr2=0x7fb8ec0eb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.312 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:02.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.309+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 shutdown_connections 2023-11-24T08:05:02.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.309+0000 7fb8fb59e640 1 -- 172.21.15.107:0/3619761731 wait complete. 2023-11-24T08:05:02.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.312+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 msgr2=0x7fe3b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.313+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 0x7fe3b406d300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00e0f00 tx=0x7fe3bc000ed0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.313+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 msgr2=0x7fe3c00d5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.313+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c00d5fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8033000 tx=0x7fe3b80046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 shutdown_connections 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c00df3e0 0x7fe3c0053b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c0054230 0x7fe3c00535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2544745871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c004fa50 0x7fe3c00d5fa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.314+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 >> 172.21.15.107:0/2544745871 conn(0x7fe3c0009f70 msgr2=0x7fe3c003a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.315+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 shutdown_connections 2023-11-24T08:05:02.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.315+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2544745871 wait complete. 2023-11-24T08:05:02.329 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2023-11-24T08:05:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.324+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f33d404cd70 con 0x7f33d40e3370 2023-11-24T08:05:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.324+0000 7f33d2ffd640 1 -- 172.21.15.107:0/1835368473 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33dc0a1020 con 0x7f33d40e3370 2023-11-24T08:05:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 msgr2=0x7f33c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 0x7f33c006d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006fd0 tx=0x7f33d8008040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 msgr2=0x7f33d4039000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33dc079ba0 tx=0x7f33dc06b8e0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 shutdown_connections 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33c006ae40 0x7f33c006d300 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006fd0 tx=0x7f33d8008040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40e3370 0x7f33d4039000 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33d40543c0 0x7f33d40d2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 --2- 172.21.15.107:0/1835368473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33d403c610 0x7f33d404b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.328+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 >> 172.21.15.107:0/1835368473 conn(0x7f33d403a990 msgr2=0x7f33d4056a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.329+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 shutdown_connections 2023-11-24T08:05:02.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.329+0000 7f33e46ee640 1 -- 172.21.15.107:0/1835368473 wait complete. 2023-11-24T08:05:02.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.331+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd238005500 con 0x7fd2540ae310 2023-11-24T08:05:02.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.332+0000 7fd24bfff640 1 -- 172.21.15.107:0/901500954 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd244038020 con 0x7fd2540ae310 2023-11-24T08:05:02.334 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2023-11-24T08:05:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.335+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 msgr2=0x7fd23406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.335+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 0x7fd23406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006fd0 tx=0x7fd23c008040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.335+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 msgr2=0x7fd254159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.335+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 0x7fd254159670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd244033000 tx=0x7fd2440046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 shutdown_connections 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd23406af00 0x7fd23406d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd25415d0f0 0x7fd254159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2540ae310 0x7fd254159670 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 --2- 172.21.15.107:0/901500954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25406e640 0x7fd2540c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 >> 172.21.15.107:0/901500954 conn(0x7fd2540cbb40 msgr2=0x7fd2540cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 shutdown_connections 2023-11-24T08:05:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.336+0000 7fd2588c8640 1 -- 172.21.15.107:0/901500954 wait complete. 2023-11-24T08:05:02.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2023-11-24T08:05:02.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2023-11-24T08:05:02.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.342+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7effc0159e40 con 0x7effc015c7f0 2023-11-24T08:05:02.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.343+0000 7effb6ffd640 1 -- 172.21.15.107:0/3164036129 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7effb8039020 con 0x7effc015c7f0 2023-11-24T08:05:02.345 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.347+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 msgr2=0x7effa406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.347+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 0x7effa406d3d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7effbc007c30 tx=0x7effbc009040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.347+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 msgr2=0x7effc00af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.347+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7effb8009d90 tx=0x7effb8004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 shutdown_connections 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effa406af10 0x7effa406d3d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc015c7f0 0x7effc00af260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effc00cf450 0x7effc0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 --2- 172.21.15.107:0/3164036129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effc00cdfb0 0x7effc00b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 >> 172.21.15.107:0/3164036129 conn(0x7effc006a920 msgr2=0x7effc0153fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 shutdown_connections 2023-11-24T08:05:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.348+0000 7effc8533640 1 -- 172.21.15.107:0/3164036129 wait complete. 2023-11-24T08:05:02.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-11-24T08:05:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.353+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb1440b9000 con 0x7fb1440d31a0 2023-11-24T08:05:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.353+0000 7fb13a7fc640 1 -- 172.21.15.107:0/1958378306 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb140017090 con 0x7fb1440d31a0 2023-11-24T08:05:02.355 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:02.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 msgr2=0x7fb12806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 0x7fb12806d350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1340099a0 tx=0x7fb1340090e0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 msgr2=0x7fb1440b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 0x7fb1440b48d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb14000e850 tx=0x7fb14000ed10 comp rx=0 tx=0).stop 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 shutdown_connections 2023-11-24T08:05:02.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb12806ae90 0x7fb12806d350 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1441609e0 0x7fb1440b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1440d31a0 0x7fb1440b48d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.356+0000 7fb14bf1d640 1 --2- 172.21.15.107:0/1958378306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1440ce010 0x7fb14415a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.357+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 >> 172.21.15.107:0/1958378306 conn(0x7fb14408af60 msgr2=0x7fb1440ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.357+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 shutdown_connections 2023-11-24T08:05:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.357+0000 7fb14bf1d640 1 -- 172.21.15.107:0/1958378306 wait complete. 2023-11-24T08:05:02.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.363+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f72f8005500 con 0x7f731006dd50 2023-11-24T08:05:02.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.364+0000 7f7306ffd640 1 -- 172.21.15.107:0/929797797 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f730c0044e0 con 0x7f731006dd50 2023-11-24T08:05:02.366 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:02.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 msgr2=0x7f72f406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 0x7f72f406d360 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7310140fc0 tx=0x7f7300008040 comp rx=0 tx=0).stop 2023-11-24T08:05:02.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 msgr2=0x7f73101ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f730c02fdd0 tx=0x7f730c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 shutdown_connections 2023-11-24T08:05:02.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72f406aea0 0x7f72f406d360 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73101419c0 0x7f73100aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73100cc0f0 0x7f73100aaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 --2- 172.21.15.107:0/929797797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731006dd50 0x7f73101ffcd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.366+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 >> 172.21.15.107:0/929797797 conn(0x7f73100b0bc0 msgr2=0x7f73100b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.367+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 shutdown_connections 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.367+0000 7f73174ea640 1 -- 172.21.15.107:0/929797797 wait complete. 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.368+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe4e015b520 con 0x7fe4e00ce640 2023-11-24T08:05:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.369+0000 7fe4d27fc640 1 -- 172.21.15.107:0/3057509843 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4e015b520 con 0x7fe4e00ce640 2023-11-24T08:05:02.371 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2023-11-24T08:05:02.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 msgr2=0x7fe4c406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 0x7fe4c406d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0160900 tx=0x7fe4dc0079b0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 msgr2=0x7fe4e0158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4d802c470 tx=0x7fe4d8004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 shutdown_connections 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4c406ae70 0x7fe4c406d330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4e0099140 0x7fe4e0159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00cfb60 0x7fe4e01585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 --2- 172.21.15.107:0/3057509843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4e00ce640 0x7fe4e0158080 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:02.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 >> 172.21.15.107:0/3057509843 conn(0x7fe4e006a920 msgr2=0x7fe4e0160090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:02.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 shutdown_connections 2023-11-24T08:05:02.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:02.372+0000 7fe4e7b1d640 1 -- 172.21.15.107:0/3057509843 wait complete. 2023-11-24T08:05:02.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-11-24T08:05:02.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-11-24T08:05:02.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-11-24T08:05:03.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T08:05:03.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T08:05:03.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T08:05:03.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T08:05:03.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T08:05:03.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T08:05:03.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T08:05:03.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T08:05:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.632+0000 7f42cb26e640 1 Processor -- start 2023-11-24T08:05:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.633+0000 7f42cb26e640 1 -- start start 2023-11-24T08:05:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.635+0000 7f42cb26e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 0x7f42c415d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.636+0000 7f42cb26e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.636+0000 7f42cb26e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c40ae5c0 con 0x7f42c415d5c0 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.637+0000 7f42caa6d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.637+0000 7f42caa6d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34508/0 (socket says 172.21.15.107:34508) 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.637+0000 7f42caa6d640 1 -- 172.21.15.107:0/1377154629 learned_addr learned my addr 172.21.15.107:0/1377154629 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.636+0000 7f42ca26c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 0x7f42c415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.637+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c415dc20 con 0x7f42c4159700 2023-11-24T08:05:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.638+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40705f0 con 0x7f42c40b1250 2023-11-24T08:05:03.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.639+0000 7f42caa6d640 1 -- 172.21.15.107:0/1377154629 >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 msgr2=0x7f42c415d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.639+0000 7f42caa6d640 1 --2- 172.21.15.107:0/1377154629 >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 0x7f42c415d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.639+0000 7f42caa6d640 1 -- 172.21.15.107:0/1377154629 >> v1:172.21.15.107:6789/0 conn(0x7f42c415d5c0 legacy=0x7f42c40b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.639+0000 7f42caa6d640 1 -- 172.21.15.107:0/1377154629 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40708d0 con 0x7f42c40b1250 2023-11-24T08:05:03.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.640+0000 7f42ca26c640 1 --2- 172.21.15.107:0/1377154629 >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 0x7f42c415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:03.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.642+0000 7f42caa6d640 1 --2- 172.21.15.107:0/1377154629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c002db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a11348f56e002f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.643+0000 7f42c926a640 1 -- 172.21.15.107:0/1377154629 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c002e940 con 0x7f42c40b1250 2023-11-24T08:05:03.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.643+0000 7f42c926a640 1 -- 172.21.15.107:0/1377154629 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c002eaa0 con 0x7f42c40b1250 2023-11-24T08:05:03.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.643+0000 7f42c926a640 1 -- 172.21.15.107:0/1377154629 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c002ee30 con 0x7f42c40b1250 2023-11-24T08:05:03.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.644+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 msgr2=0x7f42c40710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.644+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1377154629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c002db90 comp rx=0 tx=0).stop 2023-11-24T08:05:03.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.644+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 shutdown_connections 2023-11-24T08:05:03.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.645+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1377154629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c40b1250 0x7f42c40710d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.645+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1377154629 >> v2:172.21.15.107:3300/0 conn(0x7f42c4159700 0x7f42c415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.645+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 >> 172.21.15.107:0/1377154629 conn(0x7f42c40cbb40 msgr2=0x7f42c40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.645+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 shutdown_connections 2023-11-24T08:05:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.646+0000 7f42cb26e640 1 -- 172.21.15.107:0/1377154629 wait complete. 2023-11-24T08:05:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.649+0000 7f42cb26e640 1 Processor -- start 2023-11-24T08:05:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.649+0000 7f42cb26e640 1 -- start start 2023-11-24T08:05:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.652+0000 7f42cb26e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 0x7f42c41f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.652+0000 7f42cb26e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 0x7f42c4146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.652+0000 7f42cb26e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 0x7f42c415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.653+0000 7f42ca26c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 0x7f42c41f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.653+0000 7f42cb26e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41473a0 con 0x7f42c40b1250 2023-11-24T08:05:03.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.653+0000 7f42caa6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 0x7f42c415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.654+0000 7f42caa6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 0x7f42c415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58666/0 (socket says 172.21.15.107:58666) 2023-11-24T08:05:03.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.654+0000 7f42caa6d640 1 -- 172.21.15.107:0/1865495846 learned_addr learned my addr 172.21.15.107:0/1865495846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.654+0000 7f42c9a6b640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 0x7f42c4146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.654+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40b2c90 con 0x7f42c415d5c0 2023-11-24T08:05:03.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.654+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c415c6f0 con 0x7f42c4159700 2023-11-24T08:05:03.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42c9a6b640 1 -- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 msgr2=0x7f42c415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42c9a6b640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 0x7f42c415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42c9a6b640 1 -- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 msgr2=0x7f42c41f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42c9a6b640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 0x7f42c41f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42c9a6b640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b400ae30 con 0x7f42c4159700 2023-11-24T08:05:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.655+0000 7f42ca26c640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 0x7f42c41f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.657+0000 7f42c9a6b640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 0x7f42c4146e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c700 tx=0x7f42bc00cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.657+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003b70 con 0x7f42c4159700 2023-11-24T08:05:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.657+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc003cd0 con 0x7f42c4159700 2023-11-24T08:05:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.657+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c00094d0 con 0x7f42c4159700 2023-11-24T08:05:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.658+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c415ccb0 con 0x7f42c4159700 2023-11-24T08:05:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.658+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003b70 con 0x7f42c4159700 2023-11-24T08:05:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.660+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c40b2eb0 con 0x7f42c4159700 2023-11-24T08:05:03.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.663+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f42bc0050d0 con 0x7f42c4159700 2023-11-24T08:05:03.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.665+0000 7f42baffd640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.665+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42bc088ee0 con 0x7f42c4159700 2023-11-24T08:05:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.666+0000 7f42ca26c640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.668+0000 7f42ca26c640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 0x7f42a806d360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f42b4004230 tx=0x7f42b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.678+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42bc053ef0 con 0x7f42c4159700 2023-11-24T08:05:03.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.687+0000 7f25c618e640 1 Processor -- start 2023-11-24T08:05:03.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.687+0000 7f25c618e640 1 -- start start 2023-11-24T08:05:03.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.689+0000 7f25c618e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25c015b690 0x7f25c00af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.689+0000 7f25c618e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25c00afba0 0x7f25c00b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.689+0000 7f25c618e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00cb960 con 0x7f25c00b1040 2023-11-24T08:05:03.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.689+0000 7f25c618e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00cca00 con 0x7f25c015b690 2023-11-24T08:05:03.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.689+0000 7f25c618e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00b0030 con 0x7f25c00afba0 2023-11-24T08:05:03.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.690+0000 7f25beffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25c00afba0 0x7f25c00b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.690+0000 7f25bffff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f25c00b1040 0x7f25c015f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:51266/0 (socket says 172.21.15.107:51266) 2023-11-24T08:05:03.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.690+0000 7f25bffff640 1 -- 172.21.15.107:0/3210588742 learned_addr learned my addr 172.21.15.107:0/3210588742 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:03.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25bf7fe640 1 --2- 172.21.15.107:0/3210588742 >> v2:172.21.15.107:3300/0 conn(0x7f25c015b690 0x7f25c00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25be7fc640 1 -- 172.21.15.107:0/3210588742 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048406236 0 0) 0x7f25c00cb960 con 0x7f25c00b1040 2023-11-24T08:05:03.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25be7fc640 1 -- 172.21.15.107:0/3210588742 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8003520 con 0x7f25c00b1040 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25beffd640 1 -- 172.21.15.107:0/3210588742 >> v2:172.21.15.107:3300/0 conn(0x7f25c015b690 msgr2=0x7f25c00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25beffd640 1 --2- 172.21.15.107:0/3210588742 >> v2:172.21.15.107:3300/0 conn(0x7f25c015b690 0x7f25c00af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25beffd640 1 -- 172.21.15.107:0/3210588742 >> v1:172.21.15.120:6789/0 conn(0x7f25c00b1040 legacy=0x7f25c015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f25beffd640 1 -- 172.21.15.107:0/3210588742 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00cbc40 con 0x7f25c00afba0 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.691+0000 7f83f3e0e640 1 Processor -- start 2023-11-24T08:05:03.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.692+0000 7f83f3e0e640 1 -- start start 2023-11-24T08:05:03.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.693+0000 7f25beffd640 1 --2- 172.21.15.107:0/3210588742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00afba0 0x7f25c00b0b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25c00b0920 tx=0x7f25b402b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0ed67b9e0daf5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.693+0000 7f25be7fc640 1 -- 172.21.15.107:0/3210588742 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4031740 con 0x7f25c00afba0 2023-11-24T08:05:03.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.693+0000 7f31a36d0640 1 Processor -- start 2023-11-24T08:05:03.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f25be7fc640 1 -- 172.21.15.107:0/3210588742 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b40318a0 con 0x7f25c00afba0 2023-11-24T08:05:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f25be7fc640 1 -- 172.21.15.107:0/3210588742 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4031740 con 0x7f25c00afba0 2023-11-24T08:05:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f31a36d0640 1 -- start start 2023-11-24T08:05:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7fbbca89b640 1 Processor -- start 2023-11-24T08:05:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f83f3e0e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83ec0d31a0 0x7f83ec0d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f83f3e0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec0ce010 0x7f83ec0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f83f3e0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ec0d1070 con 0x7f83ec1609e0 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f83f3e0e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ec0d3b50 con 0x7f83ec0d31a0 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.694+0000 7f83f3e0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0cedc0 con 0x7f83ec0ce010 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 -- 172.21.15.107:0/3210588742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00afba0 msgr2=0x7f25c00b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 --2- 172.21.15.107:0/3210588742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00afba0 0x7f25c00b0b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25c00b0920 tx=0x7f25b402b7e0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 -- 172.21.15.107:0/3210588742 shutdown_connections 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 --2- 172.21.15.107:0/3210588742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00afba0 0x7f25c00b0b00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 --2- 172.21.15.107:0/3210588742 >> v2:172.21.15.107:3300/0 conn(0x7f25c015b690 0x7f25c00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f25c618e640 1 -- 172.21.15.107:0/3210588742 >> 172.21.15.107:0/3210588742 conn(0x7f25c00b4500 msgr2=0x7f25c00b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83ec0d31a0 0x7f83ec0d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83ec0d31a0 0x7f83ec0d3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44416/0 (socket says 172.21.15.107:44416) 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 -- 172.21.15.107:0/3568336446 learned_addr learned my addr 172.21.15.107:0/3568336446 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f23ab640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec0ce010 0x7f83ec0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 -- 172.21.15.107:0/3568336446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec0ce010 msgr2=0x7f83ec0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec0ce010 0x7f83ec0ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 -- 172.21.15.107:0/3568336446 >> v1:172.21.15.120:6789/0 conn(0x7f83ec1609e0 legacy=0x7f83ec06bcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7f83f13a9640 1 -- 172.21.15.107:0/3568336446 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec0cf0a0 con 0x7f83ec0d31a0 2023-11-24T08:05:03.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7fbbca89b640 1 -- start start 2023-11-24T08:05:03.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.695+0000 7fa5d4f33640 1 Processor -- start 2023-11-24T08:05:03.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.696+0000 7f25c618e640 1 -- 172.21.15.107:0/3210588742 shutdown_connections 2023-11-24T08:05:03.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.696+0000 7f25c618e640 1 -- 172.21.15.107:0/3210588742 wait complete. 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.696+0000 7fa5d4f33640 1 -- start start 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.697+0000 7fbbca89b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40b1380 0x7fbbc40b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.697+0000 7fbbca89b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbbc40ae8d0 0x7fbbc40aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.697+0000 7fbbca89b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc40acbd0 con 0x7fbbc415d740 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.697+0000 7fbbca89b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc40abfc0 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.697+0000 7fbbca89b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40ac2a0 con 0x7fbbc40b1380 2023-11-24T08:05:03.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7f31a36d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f319c0d31a0 0x7f319c0d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fbbc3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbbc415d740 0x7fbbc40b0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:51274/0 (socket says 172.21.15.107:51274) 2023-11-24T08:05:03.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fa5d4f33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d00cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fa5d4f33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa5d00b0380 0x7fa5d00b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fbbbb7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40b1380 0x7fbbc40b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fa5d4f33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d00cc690 con 0x7fa5d00b1220 2023-11-24T08:05:03.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fa5d4f33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d00cc970 con 0x7fa5d00b0380 2023-11-24T08:05:03.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fa5d4f33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d00b2180 con 0x7fa5d00ce120 2023-11-24T08:05:03.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7fbbc3fff640 1 -- 172.21.15.107:0/1624417584 learned_addr learned my addr 172.21.15.107:0/1624417584 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:03.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.698+0000 7f31a146c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f319c1609e0 0x7f319c06bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59150/0 (socket says 172.21.15.107:59150) 2023-11-24T08:05:03.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc8e38640 1 --2- 172.21.15.107:0/1624417584 >> v2:172.21.15.107:3300/0 conn(0x7fbbc40ae8d0 0x7fbbc40aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f25c618e640 1 Processor -- start 2023-11-24T08:05:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d00cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a146c640 1 -- 172.21.15.107:0/2643758525 learned_addr learned my addr 172.21.15.107:0/2643758525 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a36d0640 1 --2- 172.21.15.107:0/2643758525 >> v2:172.21.15.120:3300/0 conn(0x7f319c0ce010 0x7f319c0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5c7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa5d00b0380 0x7fa5d00b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d00cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34546/0 (socket says 172.21.15.107:34546) 2023-11-24T08:05:03.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5ce59c640 1 -- 172.21.15.107:0/1824146837 learned_addr learned my addr 172.21.15.107:0/1824146837 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc37fe640 1 -- 172.21.15.107:0/1624417584 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273064313 0 0) 0x7fbbc40acbd0 con 0x7fbbc415d740 2023-11-24T08:05:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc37fe640 1 -- 172.21.15.107:0/1624417584 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb0003520 con 0x7fbbc415d740 2023-11-24T08:05:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1824146837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 msgr2=0x7fa5d00cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc8e38640 1 -- 172.21.15.107:0/1624417584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40b1380 msgr2=0x7fbbc40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc8e38640 1 --2- 172.21.15.107:0/1624417584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40b1380 0x7fbbc40b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fbbc8e38640 1 -- 172.21.15.107:0/1624417584 >> v1:172.21.15.120:6789/0 conn(0x7fbbc415d740 legacy=0x7fbbc40b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7fbbc8e38640 1 -- 172.21.15.107:0/1624417584 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc40ac580 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5c7fff640 1 --2- 172.21.15.107:0/1824146837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d00cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1824146837 >> v1:172.21.15.107:6789/0 conn(0x7fa5d00b1220 legacy=0x7fa5d00aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1824146837 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d00b2460 con 0x7fa5d00b0380 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319c0d1070 con 0x7f319c1609e0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c0d3b50 con 0x7f319c0d31a0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c0cedc0 con 0x7f319c0ce010 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f31a1c6d640 1 --2- 172.21.15.107:0/2643758525 >> v2:172.21.15.120:3300/0 conn(0x7f319c0ce010 0x7f319c0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/2643758525 >> v2:172.21.15.107:3300/0 conn(0x7f319c0d31a0 0x7f319c0d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f318ffff640 1 -- 172.21.15.107:0/2643758525 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166670496 0 0) 0x7f319c0d1070 con 0x7f319c1609e0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7f318ffff640 1 -- 172.21.15.107:0/2643758525 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3188003520 con 0x7f319c1609e0 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f31a0c6b640 1 -- 172.21.15.107:0/2643758525 >> v2:172.21.15.120:3300/0 conn(0x7f319c0ce010 msgr2=0x7f319c0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/2643758525 >> v2:172.21.15.120:3300/0 conn(0x7f319c0ce010 0x7f319c0ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f31a0c6b640 1 -- 172.21.15.107:0/2643758525 >> v1:172.21.15.107:6789/0 conn(0x7f319c1609e0 legacy=0x7f319c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f31a0c6b640 1 -- 172.21.15.107:0/2643758525 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0cf0a0 con 0x7f319c0d31a0 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.699+0000 7f25c618e640 1 -- start start 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7f25c618e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.700+0000 7f25c618e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 0x7f25c009b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f25c618e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c015b690 0x7f25c00b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f25c618e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0158420 con 0x7f25c00afba0 2023-11-24T08:05:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f25c618e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c009c730 con 0x7f25c00b1040 2023-11-24T08:05:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f25c618e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c009bbb0 con 0x7f25c015b690 2023-11-24T08:05:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f83f13a9640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0d3610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0d43c0 tx=0x7f83d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c3dbede228b7da server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.701+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44448/0 (socket says 172.21.15.107:44448) 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f25beffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 0x7f25c009b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f25beffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 0x7f25c009b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58690/0 (socket says 172.21.15.107:58690) 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f25beffd640 1 -- 172.21.15.107:0/2842078150 learned_addr learned my addr 172.21.15.107:0/2842078150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f25bffff640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c015b690 0x7f25c00b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f83f0ba8640 1 -- 172.21.15.107:0/3568336446 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8031640 con 0x7f83ec0d31a0 2023-11-24T08:05:03.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7fa5c7fff640 1 --2- 172.21.15.107:0/1824146837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d00b0ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097a0 tx=0x7fa5b802da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71590a835457752 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f83f0ba8640 1 -- 172.21.15.107:0/3568336446 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d80317a0 con 0x7f83ec0d31a0 2023-11-24T08:05:03.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.702+0000 7f83f0ba8640 1 -- 172.21.15.107:0/3568336446 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8031b30 con 0x7f83ec0d31a0 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 -- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c015b690 msgr2=0x7f25c00b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c015b690 0x7f25c00b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 -- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 msgr2=0x7f25c009b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 0x7f25c009b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 -- 172.21.15.107:0/2842078150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4009400 con 0x7f25c00afba0 2023-11-24T08:05:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7fa5cdd9b640 1 -- 172.21.15.107:0/1824146837 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8031850 con 0x7fa5d00b0380 2023-11-24T08:05:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7fa5cdd9b640 1 -- 172.21.15.107:0/1824146837 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b80319b0 con 0x7fa5d00b0380 2023-11-24T08:05:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7fa5cdd9b640 1 -- 172.21.15.107:0/1824146837 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8031d40 con 0x7fa5d00b0380 2023-11-24T08:05:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/2643758525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0d31a0 0x7f319c0d3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f319c0cddd0 tx=0x7f319802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bfa67831994376 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7fbbc8e38640 1 --2- 172.21.15.107:0/1624417584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40ae8d0 0x7fbbc40aed60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009d90 tx=0x7fbbbc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddb5f2f03204984 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f25bf7fe640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f25b000d520 tx=0x7f25b000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f83f3e0e640 1 -- 172.21.15.107:0/3568336446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0d31a0 msgr2=0x7f83ec0d3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.703+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0d3610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0d43c0 tx=0x7f83d802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f318ffff640 1 -- 172.21.15.107:0/2643758525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319802e070 con 0x7f319c0d31a0 2023-11-24T08:05:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f318ffff640 1 -- 172.21.15.107:0/2643758525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31980315a0 con 0x7f319c0d31a0 2023-11-24T08:05:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f318ffff640 1 -- 172.21.15.107:0/2643758525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198031930 con 0x7f319c0d31a0 2023-11-24T08:05:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7fbbc37fe640 1 -- 172.21.15.107:0/1624417584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc02cc50 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7fbbc37fe640 1 -- 172.21.15.107:0/1624417584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc02cdb0 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7fbbc37fe640 1 -- 172.21.15.107:0/1624417584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc034660 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b00125f0 con 0x7f25c00afba0 2023-11-24T08:05:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f83f3e0e640 1 -- 172.21.15.107:0/3568336446 shutdown_connections 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec0ce010 0x7f83ec0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/3568336446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0d3610 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f83f3e0e640 1 -- 172.21.15.107:0/3568336446 >> 172.21.15.107:0/3568336446 conn(0x7f83ec08af60 msgr2=0x7f83ec09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0012750 con 0x7f25c00afba0 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1824146837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 msgr2=0x7fa5d00b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1824146837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d00b0ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097a0 tx=0x7fa5b802da30 comp rx=0 tx=0).stop 2023-11-24T08:05:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b00125f0 con 0x7f25c00afba0 2023-11-24T08:05:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.704+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c009be90 con 0x7f25c00afba0 2023-11-24T08:05:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c009c450 con 0x7f25c00afba0 2023-11-24T08:05:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 -- 172.21.15.107:0/1624417584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40ae8d0 msgr2=0x7fbbc40aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 --2- 172.21.15.107:0/1624417584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40ae8d0 0x7fbbc40aed60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009d90 tx=0x7fbbbc02bea0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7f83f3e0e640 1 -- 172.21.15.107:0/3568336446 shutdown_connections 2023-11-24T08:05:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 -- 172.21.15.107:0/1624417584 shutdown_connections 2023-11-24T08:05:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 --2- 172.21.15.107:0/1624417584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40ae8d0 0x7fbbc40aed60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 --2- 172.21.15.107:0/1624417584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40b1380 0x7fbbc40b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 -- 172.21.15.107:0/1624417584 >> 172.21.15.107:0/1624417584 conn(0x7fbbc40b4500 msgr2=0x7fbbc40b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fbbca89b640 1 -- 172.21.15.107:0/1624417584 shutdown_connections 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1824146837 shutdown_connections 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1824146837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d00b0ce0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1824146837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d00cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.705+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1824146837 >> 172.21.15.107:0/1824146837 conn(0x7fa5d00b4500 msgr2=0x7fa5d00b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0d31a0 msgr2=0x7f319c0d3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7fbbca89b640 1 -- 172.21.15.107:0/1624417584 wait complete. 2023-11-24T08:05:03.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 --2- 172.21.15.107:0/2643758525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0d31a0 0x7f319c0d3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f319c0cddd0 tx=0x7f319802b840 comp rx=0 tx=0).stop 2023-11-24T08:05:03.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 shutdown_connections 2023-11-24T08:05:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 --2- 172.21.15.107:0/2643758525 >> v2:172.21.15.120:3300/0 conn(0x7f319c0ce010 0x7f319c0ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 --2- 172.21.15.107:0/2643758525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0d31a0 0x7f319c0d3610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 >> 172.21.15.107:0/2643758525 conn(0x7f319c08af60 msgr2=0x7f319c09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7fe06630f640 1 Processor -- start 2023-11-24T08:05:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c015ba70 con 0x7f25c00afba0 2023-11-24T08:05:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1824146837 shutdown_connections 2023-11-24T08:05:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 shutdown_connections 2023-11-24T08:05:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.706+0000 7f83f3e0e640 1 -- 172.21.15.107:0/3568336446 wait complete. 2023-11-24T08:05:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7f83f3e0e640 1 Processor -- start 2023-11-24T08:05:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7f83f3e0e640 1 -- start start 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.708+0000 7fbbca89b640 1 Processor -- start 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fbbca89b640 1 -- start start 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7fe06630f640 1 -- start start 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fe06630f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fe06630f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0600ce640 0x7fe0600b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fe06630f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0600ce270 con 0x7fe0600cf450 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fe06630f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06006c560 con 0x7fe0600cc470 2023-11-24T08:05:03.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fe06630f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06006d0a0 con 0x7fe0600ce640 2023-11-24T08:05:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1824146837 wait complete. 2023-11-24T08:05:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.709+0000 7fa5d4f33640 1 Processor -- start 2023-11-24T08:05:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.708+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f25b00128c0 con 0x7f25c00afba0 2023-11-24T08:05:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7f25a7fff640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 0x7f25ac06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7fa5d4f33640 1 -- start start 2023-11-24T08:05:03.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7fe0577fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0600ce640 0x7fe0600b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.707+0000 7f31a36d0640 1 -- 172.21.15.107:0/2643758525 wait complete. 2023-11-24T08:05:03.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7fe05ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7fe05ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44462/0 (socket says 172.21.15.107:44462) 2023-11-24T08:05:03.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7fe05ffff640 1 -- 172.21.15.107:0/2724714165 learned_addr learned my addr 172.21.15.107:0/2724714165 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 0x7fbbc409ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25b0058730 con 0x7f25c00afba0 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc415d740 0x7fbbc4095b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc409a370 con 0x7fbbc40b1380 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415b930 con 0x7fbbc415d740 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbca89b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc409b4f0 con 0x7fbbc40ae8d0 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbbb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 0x7fbbc409ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.710+0000 7f31a36d0640 1 Processor -- start 2023-11-24T08:05:03.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbbb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44476/0 (socket says 172.21.15.107:44476) 2023-11-24T08:05:03.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 0x7fbbc409ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34572/0 (socket says 172.21.15.107:34572) 2023-11-24T08:05:03.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fbbc3fff640 1 -- 172.21.15.107:0/3587975931 learned_addr learned my addr 172.21.15.107:0/3587975931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.711+0000 7fe05f7fe640 1 -- 172.21.15.107:0/2724714165 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090648342 0 0) 0x7fe0600ce270 con 0x7fe0600cf450 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7f83f3e0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7f25beffd640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 0x7f25ac06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbc8e38640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc415d740 0x7fbbc4095b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbbb7fe640 1 -- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 msgr2=0x7fbbc409ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbbb7fe640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 0x7fbbc409ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbbb7fe640 1 -- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc415d740 msgr2=0x7fbbc4095b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbbb7fe640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc415d740 0x7fbbc4095b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fbbbb7fe640 1 -- 172.21.15.107:0/3587975931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc009a70 con 0x7fbbc40b1380 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7fbbbb7fe640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00d5f0 tx=0x7fbbac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f25beffd640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 0x7f25ac06d360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25c0098e30 tx=0x7f25b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7f83f3e0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7f83f3e0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec1609e0 0x7f83ec0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f1baa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f1baa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44492/0 (socket says 172.21.15.107:44492) 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f1baa640 1 -- 172.21.15.107:0/4226897128 learned_addr learned my addr 172.21.15.107:0/4226897128 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec15af00 con 0x7f83ec0ce010 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f13a9640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0b4830 con 0x7f83ec0d31a0 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0b4b10 con 0x7f83ec1609e0 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7f31a36d0640 1 -- start start 2023-11-24T08:05:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe05f7fe640 1 -- 172.21.15.107:0/2724714165 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe040003520 con 0x7fe0600cf450 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe0577fe640 1 -- 172.21.15.107:0/2724714165 >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 msgr2=0x7fe0600abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe0577fe640 1 --2- 172.21.15.107:0/2724714165 >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe0577fe640 1 -- 172.21.15.107:0/2724714165 >> v1:172.21.15.120:6789/0 conn(0x7fe0600cf450 legacy=0x7fe06006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe0577fe640 1 -- 172.21.15.107:0/2724714165 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600b4200 con 0x7fe0600ce640 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.712+0000 7fe05ffff640 1 --2- 172.21.15.107:0/2724714165 >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:05:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.713+0000 7f83f23ab640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec1609e0 0x7f83ec0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 -- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec1609e0 msgr2=0x7f83ec0c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec1609e0 0x7f83ec0c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 -- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83ec0d31a0 msgr2=0x7f83ec0b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d80094d0 con 0x7f83ec0ce010 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d01efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01557c0 con 0x7fa5d00b0380 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d0156a20 con 0x7fa5d00b1220 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fa5d4f33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01ecc10 con 0x7fa5d00ce120 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7f83f1baa640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f83e000b3e0 tx=0x7f83e000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.714+0000 7fe0577fe640 1 --2- 172.21.15.107:0/2724714165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600ce640 0x7fe0600b3cc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe0600ae200 tx=0x7fe04802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7a4a3b7b57ad93 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fa5c7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fa5c7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58744/0 (socket says 172.21.15.107:58744) 2023-11-24T08:05:03.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1507507454 learned_addr learned my addr 172.21.15.107:0/1507507454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 0x7f319c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f319c1609e0 0x7f319c15db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c15fea0 con 0x7f319c0ce010 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c15d190 con 0x7f319c1609e0 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a36d0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0b7360 con 0x7f319c0d31a0 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fe05f7fe640 1 -- 172.21.15.107:0/2724714165 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048031830 con 0x7fe0600ce640 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fe05f7fe640 1 -- 172.21.15.107:0/2724714165 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe048031990 con 0x7fe0600ce640 2023-11-24T08:05:03.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fe05f7fe640 1 -- 172.21.15.107:0/2724714165 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048031d00 con 0x7fe0600ce640 2023-11-24T08:05:03.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a146c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7fa5ce59c640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d01efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a146c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44500/0 (socket says 172.21.15.107:44500) 2023-11-24T08:05:03.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a146c640 1 -- 172.21.15.107:0/1620085231 learned_addr learned my addr 172.21.15.107:0/1620085231 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a1c6d640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f319c1609e0 0x7f319c15db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.715+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 0x7f319c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5ced9d640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 -- 172.21.15.107:0/2724714165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600ce640 msgr2=0x7fe0600b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 --2- 172.21.15.107:0/2724714165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600ce640 0x7fe0600b3cc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe0600ae200 tx=0x7fe04802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 msgr2=0x7fa5d014cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5c7fff640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d014cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 msgr2=0x7fa5d01efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5c7fff640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d01efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fa5c7fff640 1 -- 172.21.15.107:0/1507507454 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b8009400 con 0x7fa5d00b1220 2023-11-24T08:05:03.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a146c640 1 -- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 msgr2=0x7f319c15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a146c640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 0x7f319c15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a146c640 1 -- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f319c1609e0 msgr2=0x7f319c15db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a146c640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f319c1609e0 0x7f319c15db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a146c640 1 -- 172.21.15.107:0/1620085231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198009400 con 0x7f319c0ce010 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 -- 172.21.15.107:0/2724714165 shutdown_connections 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 --2- 172.21.15.107:0/2724714165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600ce640 0x7fe0600b3cc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 --2- 172.21.15.107:0/2724714165 >> v2:172.21.15.107:3300/0 conn(0x7fe0600cc470 0x7fe0600abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 -- 172.21.15.107:0/2724714165 >> 172.21.15.107:0/2724714165 conn(0x7fe06006a920 msgr2=0x7fe06006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 0x7f319c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:03.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.716+0000 7fe06630f640 1 -- 172.21.15.107:0/2724714165 shutdown_connections 2023-11-24T08:05:03.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.717+0000 7f31a146c640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f319000c460 tx=0x7f319000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.717+0000 7fa5c7fff640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80098c0 tx=0x7fa5b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.717+0000 7fe06630f640 1 -- 172.21.15.107:0/2724714165 wait complete. 2023-11-24T08:05:03.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.718+0000 7fe06630f640 1 Processor -- start 2023-11-24T08:05:03.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.719+0000 7fe06630f640 1 -- start start 2023-11-24T08:05:03.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.718+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b803d070 con 0x7fa5d00b1220 2023-11-24T08:05:03.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.718+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8039940 con 0x7fa5d00b1220 2023-11-24T08:05:03.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.719+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d0155be0 con 0x7fa5d00b1220 2023-11-24T08:05:03.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.719+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8039cf0 con 0x7fa5d00b1220 2023-11-24T08:05:03.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.719+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d0154e20 con 0x7fa5d00b1220 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600cc470 0x7fe060096060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600cf450 0x7fe0600997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601619a0 con 0x7fe0600ce640 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600965a0 con 0x7fe0600cc470 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe06630f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06014d2a0 con 0x7fe0600cf450 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44518/0 (socket says 172.21.15.107:44518) 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe0577fe640 1 -- 172.21.15.107:0/1212073718 learned_addr learned my addr 172.21.15.107:0/1212073718 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.720+0000 7fe05ffff640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600cc470 0x7fe060096060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0648ac640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600cf450 0x7fe0600997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d0155900 con 0x7fa5d00b1220 2023-11-24T08:05:03.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0577fe640 1 -- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600cf450 msgr2=0x7fe0600997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0577fe640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600cf450 0x7fe0600997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0577fe640 1 -- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600cc470 msgr2=0x7fe060096060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0577fe640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600cc470 0x7fe060096060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.721+0000 7fe0577fe640 1 -- 172.21.15.107:0/1212073718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0500094d0 con 0x7fe0600ce640 2023-11-24T08:05:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.722+0000 7fe0577fe640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe048000c00 tx=0x7fe0480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.722+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa5b804c400 con 0x7fa5d00b1220 2023-11-24T08:05:03.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.723+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0125f0 con 0x7fbbc40b1380 2023-11-24T08:05:03.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.723+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc409b7d0 con 0x7fbbc40b1380 2023-11-24T08:05:03.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc409bd90 con 0x7fbbc40b1380 2023-11-24T08:05:03.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fa5c6ffd640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 0x7fa5b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25b0054b00 con 0x7f25c00afba0 2023-11-24T08:05:03.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5b80abdc0 con 0x7fa5d00b1220 2023-11-24T08:05:03.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31900074e0 con 0x7f319c0ce010 2023-11-24T08:05:03.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3190007640 con 0x7f319c0ce010 2023-11-24T08:05:03.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31900049e0 con 0x7f319c0ce010 2023-11-24T08:05:03.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c0b7640 con 0x7f319c0ce010 2023-11-24T08:05:03.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac012750 con 0x7fbbc40b1380 2023-11-24T08:05:03.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0125f0 con 0x7fbbc40b1380 2023-11-24T08:05:03.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480042d0 con 0x7fe0600ce640 2023-11-24T08:05:03.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0011830 con 0x7f83ec0ce010 2023-11-24T08:05:03.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c0b7c00 con 0x7f319c0ce010 2023-11-24T08:05:03.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.726+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba8005250 con 0x7fbbc40b1380 2023-11-24T08:05:03.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.726+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c067470 con 0x7f319c0ce010 2023-11-24T08:05:03.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.726+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe048004430 con 0x7fe0600ce640 2023-11-24T08:05:03.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.726+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048003850 con 0x7fe0600ce640 2023-11-24T08:05:03.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.728+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f319001b360 con 0x7f319c0ce010 2023-11-24T08:05:03.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.728+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbbac0128c0 con 0x7fbbc40b1380 2023-11-24T08:05:03.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.726+0000 7fa5ce59c640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 0x7fa5b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.729+0000 7fa5ce59c640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 0x7fa5b006d300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00099a0 tx=0x7fa5c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.728+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe048009400 con 0x7fe0600ce640 2023-11-24T08:05:03.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.729+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06014d860 con 0x7fe0600ce640 2023-11-24T08:05:03.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.724+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e00119b0 con 0x7f83ec0ce010 2023-11-24T08:05:03.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.728+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0011d60 con 0x7f83ec0ce010 2023-11-24T08:05:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.729+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec0b4df0 con 0x7f83ec0ce010 2023-11-24T08:05:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.729+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec0b53b0 con 0x7f83ec0ce010 2023-11-24T08:05:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.729+0000 7f318dffb640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 0x7f318006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.730+0000 7fbbc17fa640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 0x7fbba406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.730+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3190088f50 con 0x7f319c0ce010 2023-11-24T08:05:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.730+0000 7fbbc3fff640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 0x7fbba406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.730+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe060067470 con 0x7fe0600ce640 2023-11-24T08:05:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.731+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 0x7f318006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.731+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbac058ae0 con 0x7fbbc40b1380 2023-11-24T08:05:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.731+0000 7fbbc3fff640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 0x7fbba406d360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005880 tx=0x7fbbb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.732+0000 7f31a0c6b640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 0x7f318006d400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3198036000 tx=0x7f319803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.732+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ec0736d0 con 0x7f83ec0ce010 2023-11-24T08:05:03.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.735+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5b8076ee0 con 0x7fa5d00b1220 2023-11-24T08:05:03.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.737+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3190057bb0 con 0x7f319c0ce010 2023-11-24T08:05:03.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.744+0000 7feab4ba9640 1 Processor -- start 2023-11-24T08:05:03.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.745+0000 7feab4ba9640 1 -- start start 2023-11-24T08:05:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.746+0000 7feab4ba9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feab00ce8e0 0x7feab00cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.746+0000 7feab4ba9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b2d90 con 0x7feab00d1ed0 2023-11-24T08:05:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.746+0000 7feab4ba9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b45f0 con 0x7feab00d22b0 2023-11-24T08:05:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.746+0000 7feab4ba9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab00ced90 con 0x7feab00ce8e0 2023-11-24T08:05:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.746+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbbac054720 con 0x7fbbc40b1380 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.747+0000 7feaaed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feab00ce8e0 0x7feab00cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.747+0000 7feaaed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feab00ce8e0 0x7feab00cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:58758/0 (socket says 172.21.15.107:58758) 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.747+0000 7feaaed9d640 1 -- 172.21.15.107:0/1280720192 learned_addr learned my addr 172.21.15.107:0/1280720192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.747+0000 7feaaed9d640 1 -- 172.21.15.107:0/1280720192 >> v1:172.21.15.120:6789/0 conn(0x7feab00d22b0 legacy=0x7feab00ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.748+0000 7feaaed9d640 1 -- 172.21.15.107:0/1280720192 >> v1:172.21.15.107:6789/0 conn(0x7feab00d1ed0 legacy=0x7feab00d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.748+0000 7feaaed9d640 1 -- 172.21.15.107:0/1280720192 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00cf070 con 0x7feab00ce8e0 2023-11-24T08:05:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.748+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f83e0023430 con 0x7f83ec0ce010 2023-11-24T08:05:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.749+0000 7f83ea7fc640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 0x7f83d006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.749+0000 7feaaed9d640 1 --2- 172.21.15.107:0/1280720192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00ce8e0 0x7feab00cfdb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009e50 tx=0x7feaa802bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcdd8b6876649b38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.749+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe0480039b0 con 0x7fe0600ce640 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.750+0000 7f83f13a9640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 0x7f83d006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.750+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83e0089660 con 0x7f83ec0ce010 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.750+0000 7feaad59a640 1 -- 172.21.15.107:0/1280720192 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa802c840 con 0x7feab00ce8e0 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.750+0000 7feaad59a640 1 -- 172.21.15.107:0/1280720192 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa802c9a0 con 0x7feab00ce8e0 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.750+0000 7feaad59a640 1 -- 172.21.15.107:0/1280720192 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa802cd30 con 0x7feab00ce8e0 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.751+0000 7feab4ba9640 1 -- 172.21.15.107:0/1280720192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00ce8e0 msgr2=0x7feab00cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.751+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1280720192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00ce8e0 0x7feab00cfdb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009e50 tx=0x7feaa802bd20 comp rx=0 tx=0).stop 2023-11-24T08:05:03.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.751+0000 7f83f13a9640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 0x7f83d006d350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83d8036000 tx=0x7f83d803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7fe05d7fa640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 0x7fe04406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0480ab2f0 con 0x7fe0600ce640 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7feab4ba9640 1 -- 172.21.15.107:0/1280720192 shutdown_connections 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1280720192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00ce8e0 0x7feab00cfdb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7feab4ba9640 1 -- 172.21.15.107:0/1280720192 >> 172.21.15.107:0/1280720192 conn(0x7feab008af60 msgr2=0x7feab00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7feab4ba9640 1 -- 172.21.15.107:0/1280720192 shutdown_connections 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7feab4ba9640 1 -- 172.21.15.107:0/1280720192 wait complete. 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.753+0000 7fe05ffff640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 0x7fe04406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.754+0000 7fe05ffff640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 0x7fe04406d350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0600af770 tx=0x7fe050009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.756+0000 7feab4ba9640 1 Processor -- start 2023-11-24T08:05:03.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.756+0000 7feab4ba9640 1 -- start start 2023-11-24T08:05:03.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.757+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83e0054930 con 0x7f83ec0ce010 2023-11-24T08:05:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.758+0000 7feab4ba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 0x7feab00c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feab4ba9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 0x7feab01e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feab4ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feaae59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 0x7feab00c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feab4ba9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab01cf320 con 0x7feab00d22b0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feab4ba9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00b6740 con 0x7feab00d1ed0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.759+0000 7feab4ba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00b6180 con 0x7feab00ce8e0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.760+0000 7feaadd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 0x7feab01e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.760+0000 7feaaed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.760+0000 7feaaed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44532/0 (socket says 172.21.15.107:44532) 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.760+0000 7feaaed9d640 1 -- 172.21.15.107:0/1635934993 learned_addr learned my addr 172.21.15.107:0/1635934993 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaaed9d640 1 -- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 msgr2=0x7feab00c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaaed9d640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 0x7feab00c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaaed9d640 1 -- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 msgr2=0x7feab01e7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaaed9d640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 0x7feab01e7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaaed9d640 1 -- 172.21.15.107:0/1635934993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8009b30 con 0x7feab00d22b0 2023-11-24T08:05:03.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaadd9b640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 0x7feab01e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.761+0000 7feaae59c640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 0x7feab00c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.762+0000 7feaaed9d640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feaa8036040 tx=0x7feaa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.766+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8007930 con 0x7feab00d22b0 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.766+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8007ab0 con 0x7feab00d22b0 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.766+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8007e60 con 0x7feab00d22b0 2023-11-24T08:05:03.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.766+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab00b6460 con 0x7feab00d22b0 2023-11-24T08:05:03.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.767+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab01e78c0 con 0x7feab00d22b0 2023-11-24T08:05:03.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.766+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0480765c0 con 0x7fe0600ce640 2023-11-24T08:05:03.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.769+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab01e7440 con 0x7feab00d22b0 2023-11-24T08:05:03.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.778+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feaa803e450 con 0x7feab00d22b0 2023-11-24T08:05:03.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.779+0000 7fea9effd640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 0x7fea9006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:03.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.779+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaa80abfc0 con 0x7feab00d22b0 2023-11-24T08:05:03.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.783+0000 7feaae59c640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 0x7fea9006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:03.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.784+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feab01e7440 con 0x7feab00d22b0 2023-11-24T08:05:03.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:03.785+0000 7feaae59c640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 0x7fea9006d3d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaa0003eb0 tx=0x7feaa0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.088+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f42c404e840 con 0x7f42c4159700 2023-11-24T08:05:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.089+0000 7f42baffd640 1 -- 172.21.15.107:0/1865495846 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42bc057510 con 0x7f42c4159700 2023-11-24T08:05:04.091 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.093+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 msgr2=0x7f42a806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.093+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 0x7f42a806d360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f42b4004230 tx=0x7f42b4008040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.093+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 msgr2=0x7f42c4146e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 0x7f42c4146e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c700 tx=0x7f42bc00cb00 comp rx=0 tx=0).stop 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 shutdown_connections 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c415d5c0 0x7f42c415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42c4159700 0x7f42c4146e60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 --2- 172.21.15.107:0/1865495846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c40b1250 0x7f42c41f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.094+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 >> 172.21.15.107:0/1865495846 conn(0x7f42c40cbb40 msgr2=0x7f42c40b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.095+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 shutdown_connections 2023-11-24T08:05:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.095+0000 7f42cb26e640 1 -- 172.21.15.107:0/1865495846 wait complete. 2023-11-24T08:05:04.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-11-24T08:05:04.124 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.133+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa5d0155900 con 0x7fa5d00b1220 2023-11-24T08:05:04.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.133+0000 7fa5c6ffd640 1 -- 172.21.15.107:0/1507507454 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa5d0155900 con 0x7fa5d00b1220 2023-11-24T08:05:04.136 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:04.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 msgr2=0x7fa5b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 0x7fa5b006d300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00099a0 tx=0x7fa5c0008040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 msgr2=0x7fa5d01ec6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80098c0 tx=0x7fa5b80046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 shutdown_connections 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006ae40 0x7fa5b006d300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5d00ce120 0x7fa5d014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d00b1220 0x7fa5d01ec6d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 --2- 172.21.15.107:0/1507507454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d00b0380 0x7fa5d01efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 >> 172.21.15.107:0/1507507454 conn(0x7fa5d00b4500 msgr2=0x7fa5d014c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.136+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 shutdown_connections 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.137+0000 7fa5d4f33640 1 -- 172.21.15.107:0/1507507454 wait complete. 2023-11-24T08:05:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.137+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f319c0d2400 con 0x7f319c0ce010 2023-11-24T08:05:04.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.138+0000 7f318dffb640 1 -- 172.21.15.107:0/1620085231 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3190018020 con 0x7f319c0ce010 2023-11-24T08:05:04.140 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2023-11-24T08:05:04.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.141+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 msgr2=0x7f318006d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.141+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 0x7f318006d400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3198036000 tx=0x7f319803d040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.141+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 msgr2=0x7f319c0b6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.141+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f319000c460 tx=0x7f319000c920 comp rx=0 tx=0).stop 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 shutdown_connections 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f318006af40 0x7f318006d400 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f319c1609e0 0x7f319c15db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f319c0d31a0 0x7f319c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 --2- 172.21.15.107:0/1620085231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f319c0ce010 0x7f319c0b6c50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 >> 172.21.15.107:0/1620085231 conn(0x7f319c08af60 msgr2=0x7f319c0d1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 shutdown_connections 2023-11-24T08:05:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.142+0000 7f31a36d0640 1 -- 172.21.15.107:0/1620085231 wait complete. 2023-11-24T08:05:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.143+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f25c015ba70 con 0x7f25c00afba0 2023-11-24T08:05:04.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.143+0000 7f25a7fff640 1 -- 172.21.15.107:0/2842078150 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f25c015ba70 con 0x7f25c00afba0 2023-11-24T08:05:04.146 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:04.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 msgr2=0x7f25ac06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 0x7f25ac06d360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25c0098e30 tx=0x7f25b4008030 comp rx=0 tx=0).stop 2023-11-24T08:05:04.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 msgr2=0x7f25c0157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f25b000d520 tx=0x7f25b000d9e0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 shutdown_connections 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25ac06aea0 0x7f25ac06d360 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c015b690 0x7f25c00b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c00b1040 0x7f25c009b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 --2- 172.21.15.107:0/2842078150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c00afba0 0x7f25c0157ee0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.149+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 >> 172.21.15.107:0/2842078150 conn(0x7f25c00b4500 msgr2=0x7f25c015a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.150+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 shutdown_connections 2023-11-24T08:05:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.150+0000 7f25c618e640 1 -- 172.21.15.107:0/2842078150 wait complete. 2023-11-24T08:05:04.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.155+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbba8005500 con 0x7fbbc40b1380 2023-11-24T08:05:04.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-11-24T08:05:04.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.156+0000 7fbbc17fa640 1 -- 172.21.15.107:0/3587975931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbbac057d40 con 0x7fbbc40b1380 2023-11-24T08:05:04.158 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.159 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-11-24T08:05:04.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 msgr2=0x7fbba406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 0x7fbba406d360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005880 tx=0x7fbbb4009040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 msgr2=0x7fbbc415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00d5f0 tx=0x7fbbac00dab0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 shutdown_connections 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba406aea0 0x7fbba406d360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc415d740 0x7fbbc4095b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc40b1380 0x7fbbc415b3f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 --2- 172.21.15.107:0/3587975931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc40ae8d0 0x7fbbc409ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 >> 172.21.15.107:0/3587975931 conn(0x7fbbc40b4500 msgr2=0x7fbbc406db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 shutdown_connections 2023-11-24T08:05:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.159+0000 7fbbca89b640 1 -- 172.21.15.107:0/3587975931 wait complete. 2023-11-24T08:05:04.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2023-11-24T08:05:04.166 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.175+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe0600cd910 con 0x7fe0600ce640 2023-11-24T08:05:04.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.175+0000 7fe05d7fa640 1 -- 172.21.15.107:0/1212073718 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe048079be0 con 0x7fe0600ce640 2023-11-24T08:05:04.177 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-11-24T08:05:04.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-11-24T08:05:04.179 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 msgr2=0x7fe04406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 0x7fe04406d350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0600af770 tx=0x7fe050009240 comp rx=0 tx=0).stop 2023-11-24T08:05:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 msgr2=0x7fe060153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe048000c00 tx=0x7fe0480046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 shutdown_connections 2023-11-24T08:05:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406ae90 0x7fe04406d350 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600cf450 0x7fe0600997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600ce640 0x7fe060153fd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 --2- 172.21.15.107:0/1212073718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600cc470 0x7fe060096060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 >> 172.21.15.107:0/1212073718 conn(0x7fe06006a920 msgr2=0x7fe06006dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-11-24T08:05:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 shutdown_connections 2023-11-24T08:05:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.178+0000 7fe06630f640 1 -- 172.21.15.107:0/1212073718 wait complete. 2023-11-24T08:05:04.184 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.179+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f83ec067470 con 0x7f83ec0ce010 2023-11-24T08:05:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.180+0000 7f83ea7fc640 1 -- 172.21.15.107:0/4226897128 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f83e0015020 con 0x7f83ec0ce010 2023-11-24T08:05:04.185 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 msgr2=0x7f83d006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 0x7f83d006d350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83d8036000 tx=0x7f83d803c040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 msgr2=0x7f83ec14a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f83e000b3e0 tx=0x7f83e000b8a0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 shutdown_connections 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83d006ae90 0x7f83d006d350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83ec1609e0 0x7f83ec0c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83ec0d31a0 0x7f83ec0b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 --2- 172.21.15.107:0/4226897128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83ec0ce010 0x7f83ec14a910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 >> 172.21.15.107:0/4226897128 conn(0x7f83ec08af60 msgr2=0x7f83ec15e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 shutdown_connections 2023-11-24T08:05:04.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.183+0000 7f83f3e0e640 1 -- 172.21.15.107:0/4226897128 wait complete. 2023-11-24T08:05:04.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-11-24T08:05:04.197 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.201+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7feab01e7440 con 0x7feab00d22b0 2023-11-24T08:05:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.201+0000 7fea9effd640 1 -- 172.21.15.107:0/1635934993 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feab01e7440 con 0x7feab00d22b0 2023-11-24T08:05:04.203 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 msgr2=0x7fea9006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 0x7fea9006d3d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaa0003eb0 tx=0x7feaa0009040 comp rx=0 tx=0).stop 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 msgr2=0x7feab01cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feaa8036040 tx=0x7feaa8004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 shutdown_connections 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea9006af10 0x7fea9006d3d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feab00d22b0 0x7feab01cf590 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab00d1ed0 0x7feab01e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 --2- 172.21.15.107:0/1635934993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feab00ce8e0 0x7feab00c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 >> 172.21.15.107:0/1635934993 conn(0x7feab008af60 msgr2=0x7feab01e4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 shutdown_connections 2023-11-24T08:05:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.204+0000 7feab4ba9640 1 -- 172.21.15.107:0/1635934993 wait complete. 2023-11-24T08:05:04.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-11-24T08:05:04.208 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-11-24T08:05:04.222 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:04.222 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T08:05:04.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:05:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.463+0000 7f0016c6b640 1 Processor -- start 2023-11-24T08:05:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.464+0000 7f0016c6b640 1 -- start start 2023-11-24T08:05:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00101461d0 0x7f00100b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00100b4fe0 0x7f00100b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00100b3d00 con 0x7f00101461d0 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001006d180 con 0x7f00100b4fe0 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.465+0000 7f0016c6b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001006ba70 con 0x7f00100b3890 2023-11-24T08:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00101461d0 0x7f00100b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00101461d0 0x7f00100b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44546/0 (socket says 172.21.15.107:44546) 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 -- 172.21.15.107:0/2834848428 learned_addr learned my addr 172.21.15.107:0/2834848428 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 -- 172.21.15.107:0/2834848428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 msgr2=0x7f001015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 --2- 172.21.15.107:0/2834848428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 -- 172.21.15.107:0/2834848428 >> v2:172.21.15.120:3300/0 conn(0x7f00100b4fe0 msgr2=0x7f00100b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 --2- 172.21.15.107:0/2834848428 >> v2:172.21.15.120:3300/0 conn(0x7f00100b4fe0 0x7f00100b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.466+0000 7f0014a07640 1 -- 172.21.15.107:0/2834848428 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001006bd50 con 0x7f00101461d0 2023-11-24T08:05:04.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.467+0000 7f0014a07640 1 --2- 172.21.15.107:0/2834848428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f00100b4aa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0000009e80 tx=0x7f000002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27e60fa242bb0ba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.467+0000 7f000f7fe640 1 -- 172.21.15.107:0/2834848428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000002ca50 con 0x7f00101461d0 2023-11-24T08:05:04.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.467+0000 7f000f7fe640 1 -- 172.21.15.107:0/2834848428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000002cbb0 con 0x7f00101461d0 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.467+0000 7f000f7fe640 1 -- 172.21.15.107:0/2834848428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000343f0 con 0x7f00101461d0 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 -- 172.21.15.107:0/2834848428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 msgr2=0x7f00100b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 --2- 172.21.15.107:0/2834848428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f00100b4aa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0000009e80 tx=0x7f000002beb0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 -- 172.21.15.107:0/2834848428 shutdown_connections 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 --2- 172.21.15.107:0/2834848428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 --2- 172.21.15.107:0/2834848428 >> v2:172.21.15.120:3300/0 conn(0x7f00100b4fe0 0x7f00100b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 --2- 172.21.15.107:0/2834848428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f00100b4aa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 -- 172.21.15.107:0/2834848428 >> 172.21.15.107:0/2834848428 conn(0x7f0010061340 msgr2=0x7f0010061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 -- 172.21.15.107:0/2834848428 shutdown_connections 2023-11-24T08:05:04.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.468+0000 7f0016c6b640 1 -- 172.21.15.107:0/2834848428 wait complete. 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.470+0000 7f0016c6b640 1 Processor -- start 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.470+0000 7f0016c6b640 1 -- start start 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.470+0000 7f0016c6b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001013eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.470+0000 7f0016c6b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 0x7f0010142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0016c6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0016c6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100ced30 con 0x7f00101461d0 2023-11-24T08:05:04.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0016c6b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100ce420 con 0x7f00100b4fe0 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0016c6b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100ce6e0 con 0x7f00100b3890 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44554/0 (socket says 172.21.15.107:44554) 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 -- 172.21.15.107:0/503690028 learned_addr learned my addr 172.21.15.107:0/503690028 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f000ffff640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 0x7f0010142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 -- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 msgr2=0x7f001013eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0014a07640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001013eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001013eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 -- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 msgr2=0x7f0010142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 0x7f0010142ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 -- 172.21.15.107:0/503690028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000009b30 con 0x7f00101461d0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f000ffff640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 0x7f0010142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.471+0000 7f0015208640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f000800b410 tx=0x7f000800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.472+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008011830 con 0x7f00101461d0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.472+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00080119b0 con 0x7f00101461d0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.472+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008011d60 con 0x7f00101461d0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.472+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00100ce9c0 con 0x7f00101461d0 2023-11-24T08:05:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.472+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00101f6dd0 con 0x7f00101461d0 2023-11-24T08:05:04.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.473+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00100ceee0 con 0x7f00101461d0 2023-11-24T08:05:04.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.474+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f000801a400 con 0x7f00101461d0 2023-11-24T08:05:04.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.474+0000 7f000d7fa640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 0x7efff006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.475+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0008089700 con 0x7f00101461d0 2023-11-24T08:05:04.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.475+0000 7f0014a07640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 0x7efff006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.475+0000 7f0014a07640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 0x7efff006d330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0000035000 tx=0x7f0000030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.478+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00100ceee0 con 0x7f00101461d0 2023-11-24T08:05:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.652+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f001006c5f0 con 0x7efff006ae70 2023-11-24T08:05:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.663+0000 7f000d7fa640 1 -- 172.21.15.107:0/503690028 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264390 (secure 0 0 0) 0x7f0000009510 con 0x7efff006ae70 2023-11-24T08:05:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:04.670 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 msgr2=0x7efff006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 0x7efff006d330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0000035000 tx=0x7f0000030000 comp rx=0 tx=0).stop 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 msgr2=0x7f0010143010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f000800b410 tx=0x7f000800b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 shutdown_connections 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efff006ae70 0x7efff006d330 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00101461d0 0x7f0010143010 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00100b4fe0 0x7f0010142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 --2- 172.21.15.107:0/503690028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00100b3890 0x7f001013eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 >> 172.21.15.107:0/503690028 conn(0x7f0010061340 msgr2=0x7f0010142110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 shutdown_connections 2023-11-24T08:05:04.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.668+0000 7f0016c6b640 1 -- 172.21.15.107:0/503690028 wait complete. 2023-11-24T08:05:04.686 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-24T08:05:03.434041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.923023"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008109+0000","last_change":"2023-11-24T08:04:54.558355+0000","last_active":"2023-11-24T08:04:57.008109+0000","last_peered":"2023-11-24T08:04:57.008109+0000","last_clean":"2023-11-24T08:04:57.008109+0000","last_became_active":"2023-11-24T08:04:54.558044+0000","last_became_peered":"2023-11-24T08:04:54.558044+0000","last_unstale":"2023-11-24T08:04:57.008109+0000","last_undegraded":"2023-11-24T08:04:57.008109+0000","last_fullsized":"2023-11-24T08:04:57.008109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536483+0000","last_change":"2023-11-24T08:04:54.555145+0000","last_active":"2023-11-24T08:04:55.536483+0000","last_peered":"2023-11-24T08:04:55.536483+0000","last_clean":"2023-11-24T08:04:55.536483+0000","last_became_active":"2023-11-24T08:04:54.554789+0000","last_became_peered":"2023-11-24T08:04:54.554789+0000","last_unstale":"2023-11-24T08:04:55.536483+0000","last_undegraded":"2023-11-24T08:04:55.536483+0000","last_fullsized":"2023-11-24T08:04:55.536483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111906+0000","last_change":"2023-11-24T08:04:54.548731+0000","last_active":"2023-11-24T08:04:56.111906+0000","last_peered":"2023-11-24T08:04:56.111906+0000","last_clean":"2023-11-24T08:04:56.111906+0000","last_became_active":"2023-11-24T08:04:54.547639+0000","last_became_peered":"2023-11-24T08:04:54.547639+0000","last_unstale":"2023-11-24T08:04:56.111906+0000","last_undegraded":"2023-11-24T08:04:56.111906+0000","last_fullsized":"2023-11-24T08:04:56.111906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005247+0000","last_change":"2023-11-24T08:04:54.563561+0000","last_active":"2023-11-24T08:04:57.005247+0000","last_peered":"2023-11-24T08:04:57.005247+0000","last_clean":"2023-11-24T08:04:57.005247+0000","last_became_active":"2023-11-24T08:04:54.563217+0000","last_became_peered":"2023-11-24T08:04:54.563217+0000","last_unstale":"2023-11-24T08:04:57.005247+0000","last_undegraded":"2023-11-24T08:04:57.005247+0000","last_fullsized":"2023-11-24T08:04:57.005247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009305+0000","last_change":"2023-11-24T08:04:54.545576+0000","last_active":"2023-11-24T08:04:57.009305+0000","last_peered":"2023-11-24T08:04:57.009305+0000","last_clean":"2023-11-24T08:04:57.009305+0000","last_became_active":"2023-11-24T08:04:54.543956+0000","last_became_peered":"2023-11-24T08:04:54.543956+0000","last_unstale":"2023-11-24T08:04:57.009305+0000","last_undegraded":"2023-11-24T08:04:57.009305+0000","last_fullsized":"2023-11-24T08:04:57.009305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005154+0000","last_change":"2023-11-24T08:04:54.562290+0000","last_active":"2023-11-24T08:04:57.005154+0000","last_peered":"2023-11-24T08:04:57.005154+0000","last_clean":"2023-11-24T08:04:57.005154+0000","last_became_active":"2023-11-24T08:04:54.561367+0000","last_became_peered":"2023-11-24T08:04:54.561367+0000","last_unstale":"2023-11-24T08:04:57.005154+0000","last_undegraded":"2023-11-24T08:04:57.005154+0000","last_fullsized":"2023-11-24T08:04:57.005154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008255+0000","last_change":"2023-11-24T08:04:54.557527+0000","last_active":"2023-11-24T08:04:57.008255+0000","last_peered":"2023-11-24T08:04:57.008255+0000","last_clean":"2023-11-24T08:04:57.008255+0000","last_became_active":"2023-11-24T08:04:54.554705+0000","last_became_peered":"2023-11-24T08:04:54.554705+0000","last_unstale":"2023-11-24T08:04:57.008255+0000","last_undegraded":"2023-11-24T08:04:57.008255+0000","last_fullsized":"2023-11-24T08:04:57.008255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697542+0000","last_change":"2023-11-24T08:04:54.549339+0000","last_active":"2023-11-24T08:04:55.697542+0000","last_peered":"2023-11-24T08:04:55.697542+0000","last_clean":"2023-11-24T08:04:55.697542+0000","last_became_active":"2023-11-24T08:04:54.548540+0000","last_became_peered":"2023-11-24T08:04:54.548540+0000","last_unstale":"2023-11-24T08:04:55.697542+0000","last_undegraded":"2023-11-24T08:04:55.697542+0000","last_fullsized":"2023-11-24T08:04:55.697542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008315+0000","last_change":"2023-11-24T08:04:54.556229+0000","last_active":"2023-11-24T08:04:57.008315+0000","last_peered":"2023-11-24T08:04:57.008315+0000","last_clean":"2023-11-24T08:04:57.008315+0000","last_became_active":"2023-11-24T08:04:54.554805+0000","last_became_peered":"2023-11-24T08:04:54.554805+0000","last_unstale":"2023-11-24T08:04:57.008315+0000","last_undegraded":"2023-11-24T08:04:57.008315+0000","last_fullsized":"2023-11-24T08:04:57.008315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006892+0000","last_change":"2023-11-24T08:04:54.544285+0000","last_active":"2023-11-24T08:04:57.006892+0000","last_peered":"2023-11-24T08:04:57.006892+0000","last_clean":"2023-11-24T08:04:57.006892+0000","last_became_active":"2023-11-24T08:04:54.543957+0000","last_became_peered":"2023-11-24T08:04:54.543957+0000","last_unstale":"2023-11-24T08:04:57.006892+0000","last_undegraded":"2023-11-24T08:04:57.006892+0000","last_fullsized":"2023-11-24T08:04:57.006892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005608+0000","last_change":"2023-11-24T08:04:54.539976+0000","last_active":"2023-11-24T08:04:57.005608+0000","last_peered":"2023-11-24T08:04:57.005608+0000","last_clean":"2023-11-24T08:04:57.005608+0000","last_became_active":"2023-11-24T08:04:54.539639+0000","last_became_peered":"2023-11-24T08:04:54.539639+0000","last_unstale":"2023-11-24T08:04:57.005608+0000","last_undegraded":"2023-11-24T08:04:57.005608+0000","last_fullsized":"2023-11-24T08:04:57.005608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538051+0000","last_change":"2023-11-24T08:04:54.546809+0000","last_active":"2023-11-24T08:04:55.538051+0000","last_peered":"2023-11-24T08:04:55.538051+0000","last_clean":"2023-11-24T08:04:55.538051+0000","last_became_active":"2023-11-24T08:04:54.546568+0000","last_became_peered":"2023-11-24T08:04:54.546568+0000","last_unstale":"2023-11-24T08:04:55.538051+0000","last_undegraded":"2023-11-24T08:04:55.538051+0000","last_fullsized":"2023-11-24T08:04:55.538051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:27.753563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698433+0000","last_change":"2023-11-24T08:04:54.563512+0000","last_active":"2023-11-24T08:04:55.698433+0000","last_peered":"2023-11-24T08:04:55.698433+0000","last_clean":"2023-11-24T08:04:55.698433+0000","last_became_active":"2023-11-24T08:04:54.562308+0000","last_became_peered":"2023-11-24T08:04:54.562308+0000","last_unstale":"2023-11-24T08:04:55.698433+0000","last_undegraded":"2023-11-24T08:04:55.698433+0000","last_fullsized":"2023-11-24T08:04:55.698433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111574+0000","last_change":"2023-11-24T08:04:54.552082+0000","last_active":"2023-11-24T08:04:56.111574+0000","last_peered":"2023-11-24T08:04:56.111574+0000","last_clean":"2023-11-24T08:04:56.111574+0000","last_became_active":"2023-11-24T08:04:54.542957+0000","last_became_peered":"2023-11-24T08:04:54.542957+0000","last_unstale":"2023-11-24T08:04:56.111574+0000","last_undegraded":"2023-11-24T08:04:56.111574+0000","last_fullsized":"2023-11-24T08:04:56.111574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:10.754583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535932+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.535932+0000","last_peered":"2023-11-24T08:04:55.535932+0000","last_clean":"2023-11-24T08:04:55.535932+0000","last_became_active":"2023-11-24T08:04:54.561664+0000","last_became_peered":"2023-11-24T08:04:54.561664+0000","last_unstale":"2023-11-24T08:04:55.535932+0000","last_undegraded":"2023-11-24T08:04:55.535932+0000","last_fullsized":"2023-11-24T08:04:55.535932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006259+0000","last_change":"2023-11-24T08:04:54.545602+0000","last_active":"2023-11-24T08:04:57.006259+0000","last_peered":"2023-11-24T08:04:57.006259+0000","last_clean":"2023-11-24T08:04:57.006259+0000","last_became_active":"2023-11-24T08:04:54.545279+0000","last_became_peered":"2023-11-24T08:04:54.545279+0000","last_unstale":"2023-11-24T08:04:57.006259+0000","last_undegraded":"2023-11-24T08:04:57.006259+0000","last_fullsized":"2023-11-24T08:04:57.006259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006979+0000","last_change":"2023-11-24T08:04:54.546983+0000","last_active":"2023-11-24T08:04:57.006979+0000","last_peered":"2023-11-24T08:04:57.006979+0000","last_clean":"2023-11-24T08:04:57.006979+0000","last_became_active":"2023-11-24T08:04:54.545361+0000","last_became_peered":"2023-11-24T08:04:54.545361+0000","last_unstale":"2023-11-24T08:04:57.006979+0000","last_undegraded":"2023-11-24T08:04:57.006979+0000","last_fullsized":"2023-11-24T08:04:57.006979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002164+0000","last_change":"2023-11-24T08:04:54.556787+0000","last_active":"2023-11-24T08:04:57.002164+0000","last_peered":"2023-11-24T08:04:57.002164+0000","last_clean":"2023-11-24T08:04:57.002164+0000","last_became_active":"2023-11-24T08:04:54.556507+0000","last_became_peered":"2023-11-24T08:04:54.556507+0000","last_unstale":"2023-11-24T08:04:57.002164+0000","last_undegraded":"2023-11-24T08:04:57.002164+0000","last_fullsized":"2023-11-24T08:04:57.002164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:23.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008934+0000","last_change":"2023-11-24T08:04:54.544572+0000","last_active":"2023-11-24T08:04:57.008934+0000","last_peered":"2023-11-24T08:04:57.008934+0000","last_clean":"2023-11-24T08:04:57.008934+0000","last_became_active":"2023-11-24T08:04:54.544277+0000","last_became_peered":"2023-11-24T08:04:54.544277+0000","last_unstale":"2023-11-24T08:04:57.008934+0000","last_undegraded":"2023-11-24T08:04:57.008934+0000","last_fullsized":"2023-11-24T08:04:57.008934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006337+0000","last_change":"2023-11-24T08:04:54.549562+0000","last_active":"2023-11-24T08:04:57.006337+0000","last_peered":"2023-11-24T08:04:57.006337+0000","last_clean":"2023-11-24T08:04:57.006337+0000","last_became_active":"2023-11-24T08:04:54.549054+0000","last_became_peered":"2023-11-24T08:04:54.549054+0000","last_unstale":"2023-11-24T08:04:57.006337+0000","last_undegraded":"2023-11-24T08:04:57.006337+0000","last_fullsized":"2023-11-24T08:04:57.006337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111982+0000","last_change":"2023-11-24T08:04:54.554392+0000","last_active":"2023-11-24T08:04:56.111982+0000","last_peered":"2023-11-24T08:04:56.111982+0000","last_clean":"2023-11-24T08:04:56.111982+0000","last_became_active":"2023-11-24T08:04:54.551225+0000","last_became_peered":"2023-11-24T08:04:54.551225+0000","last_unstale":"2023-11-24T08:04:56.111982+0000","last_undegraded":"2023-11-24T08:04:56.111982+0000","last_fullsized":"2023-11-24T08:04:56.111982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005328+0000","last_change":"2023-11-24T08:04:54.545848+0000","last_active":"2023-11-24T08:04:57.005328+0000","last_peered":"2023-11-24T08:04:57.005328+0000","last_clean":"2023-11-24T08:04:57.005328+0000","last_became_active":"2023-11-24T08:04:54.545564+0000","last_became_peered":"2023-11-24T08:04:54.545564+0000","last_unstale":"2023-11-24T08:04:57.005328+0000","last_undegraded":"2023-11-24T08:04:57.005328+0000","last_fullsized":"2023-11-24T08:04:57.005328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697541+0000","last_change":"2023-11-24T08:04:54.566061+0000","last_active":"2023-11-24T08:04:55.697541+0000","last_peered":"2023-11-24T08:04:55.697541+0000","last_clean":"2023-11-24T08:04:55.697541+0000","last_became_active":"2023-11-24T08:04:54.563794+0000","last_became_peered":"2023-11-24T08:04:54.563794+0000","last_unstale":"2023-11-24T08:04:55.697541+0000","last_undegraded":"2023-11-24T08:04:55.697541+0000","last_fullsized":"2023-11-24T08:04:55.697541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006549+0000","last_change":"2023-11-24T08:04:54.549393+0000","last_active":"2023-11-24T08:04:57.006549+0000","last_peered":"2023-11-24T08:04:57.006549+0000","last_clean":"2023-11-24T08:04:57.006549+0000","last_became_active":"2023-11-24T08:04:54.549057+0000","last_became_peered":"2023-11-24T08:04:54.549057+0000","last_unstale":"2023-11-24T08:04:57.006549+0000","last_undegraded":"2023-11-24T08:04:57.006549+0000","last_fullsized":"2023-11-24T08:04:57.006549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698534+0000","last_change":"2023-11-24T08:04:54.541713+0000","last_active":"2023-11-24T08:04:55.698534+0000","last_peered":"2023-11-24T08:04:55.698534+0000","last_clean":"2023-11-24T08:04:55.698534+0000","last_became_active":"2023-11-24T08:04:54.541443+0000","last_became_peered":"2023-11-24T08:04:54.541443+0000","last_unstale":"2023-11-24T08:04:55.698534+0000","last_undegraded":"2023-11-24T08:04:55.698534+0000","last_fullsized":"2023-11-24T08:04:55.698534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007090+0000","last_change":"2023-11-24T08:04:54.544917+0000","last_active":"2023-11-24T08:04:57.007090+0000","last_peered":"2023-11-24T08:04:57.007090+0000","last_clean":"2023-11-24T08:04:57.007090+0000","last_became_active":"2023-11-24T08:04:54.544595+0000","last_became_peered":"2023-11-24T08:04:54.544595+0000","last_unstale":"2023-11-24T08:04:57.007090+0000","last_undegraded":"2023-11-24T08:04:57.007090+0000","last_fullsized":"2023-11-24T08:04:57.007090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536471+0000","last_change":"2023-11-24T08:04:54.558219+0000","last_active":"2023-11-24T08:04:55.536471+0000","last_peered":"2023-11-24T08:04:55.536471+0000","last_clean":"2023-11-24T08:04:55.536471+0000","last_became_active":"2023-11-24T08:04:54.557796+0000","last_became_peered":"2023-11-24T08:04:54.557796+0000","last_unstale":"2023-11-24T08:04:55.536471+0000","last_undegraded":"2023-11-24T08:04:55.536471+0000","last_fullsized":"2023-11-24T08:04:55.536471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698163+0000","last_change":"2023-11-24T08:04:54.561235+0000","last_active":"2023-11-24T08:04:55.698163+0000","last_peered":"2023-11-24T08:04:55.698163+0000","last_clean":"2023-11-24T08:04:55.698163+0000","last_became_active":"2023-11-24T08:04:54.560941+0000","last_became_peered":"2023-11-24T08:04:54.560941+0000","last_unstale":"2023-11-24T08:04:55.698163+0000","last_undegraded":"2023-11-24T08:04:55.698163+0000","last_fullsized":"2023-11-24T08:04:55.698163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112074+0000","last_change":"2023-11-24T08:04:54.553956+0000","last_active":"2023-11-24T08:04:56.112074+0000","last_peered":"2023-11-24T08:04:56.112074+0000","last_clean":"2023-11-24T08:04:56.112074+0000","last_became_active":"2023-11-24T08:04:54.553685+0000","last_became_peered":"2023-11-24T08:04:54.553685+0000","last_unstale":"2023-11-24T08:04:56.112074+0000","last_undegraded":"2023-11-24T08:04:56.112074+0000","last_fullsized":"2023-11-24T08:04:56.112074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005534+0000","last_change":"2023-11-24T08:04:54.560091+0000","last_active":"2023-11-24T08:04:57.005534+0000","last_peered":"2023-11-24T08:04:57.005534+0000","last_clean":"2023-11-24T08:04:57.005534+0000","last_became_active":"2023-11-24T08:04:54.559784+0000","last_became_peered":"2023-11-24T08:04:54.559784+0000","last_unstale":"2023-11-24T08:04:57.005534+0000","last_undegraded":"2023-11-24T08:04:57.005534+0000","last_fullsized":"2023-11-24T08:04:57.005534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002234+0000","last_change":"2023-11-24T08:04:54.542929+0000","last_active":"2023-11-24T08:04:57.002234+0000","last_peered":"2023-11-24T08:04:57.002234+0000","last_clean":"2023-11-24T08:04:57.002234+0000","last_became_active":"2023-11-24T08:04:54.542574+0000","last_became_peered":"2023-11-24T08:04:54.542574+0000","last_unstale":"2023-11-24T08:04:57.002234+0000","last_undegraded":"2023-11-24T08:04:57.002234+0000","last_fullsized":"2023-11-24T08:04:57.002234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:51.449998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.004781+0000","last_change":"2023-11-24T08:04:54.547198+0000","last_active":"2023-11-24T08:04:57.004781+0000","last_peered":"2023-11-24T08:04:57.004781+0000","last_clean":"2023-11-24T08:04:57.004781+0000","last_became_active":"2023-11-24T08:04:54.546936+0000","last_became_peered":"2023-11-24T08:04:54.546936+0000","last_unstale":"2023-11-24T08:04:57.004781+0000","last_undegraded":"2023-11-24T08:04:57.004781+0000","last_fullsized":"2023-11-24T08:04:57.004781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537709+0000","last_change":"2023-11-24T08:04:54.542121+0000","last_active":"2023-11-24T08:04:55.537709+0000","last_peered":"2023-11-24T08:04:55.537709+0000","last_clean":"2023-11-24T08:04:55.537709+0000","last_became_active":"2023-11-24T08:04:54.541708+0000","last_became_peered":"2023-11-24T08:04:54.541708+0000","last_unstale":"2023-11-24T08:04:55.537709+0000","last_undegraded":"2023-11-24T08:04:55.537709+0000","last_fullsized":"2023-11-24T08:04:55.537709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:41.750215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563001+0000","last_change":"2023-11-24T08:04:53.530633+0000","last_active":"2023-11-24T08:04:54.563001+0000","last_peered":"2023-11-24T08:04:54.563001+0000","last_clean":"2023-11-24T08:04:54.563001+0000","last_became_active":"2023-11-24T08:04:53.530234+0000","last_became_peered":"2023-11-24T08:04:53.530234+0000","last_unstale":"2023-11-24T08:04:54.563001+0000","last_undegraded":"2023-11-24T08:04:54.563001+0000","last_fullsized":"2023-11-24T08:04:54.563001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112344+0000","last_change":"2023-11-24T08:04:54.563412+0000","last_active":"2023-11-24T08:04:56.112344+0000","last_peered":"2023-11-24T08:04:56.112344+0000","last_clean":"2023-11-24T08:04:56.112344+0000","last_became_active":"2023-11-24T08:04:54.561262+0000","last_became_peered":"2023-11-24T08:04:54.561262+0000","last_unstale":"2023-11-24T08:04:56.112344+0000","last_undegraded":"2023-11-24T08:04:56.112344+0000","last_fullsized":"2023-11-24T08:04:56.112344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.560535+0000","last_change":"2023-11-24T08:04:53.527156+0000","last_active":"2023-11-24T08:04:54.560535+0000","last_peered":"2023-11-24T08:04:54.560535+0000","last_clean":"2023-11-24T08:04:54.560535+0000","last_became_active":"2023-11-24T08:04:53.525686+0000","last_became_peered":"2023-11-24T08:04:53.525686+0000","last_unstale":"2023-11-24T08:04:54.560535+0000","last_undegraded":"2023-11-24T08:04:54.560535+0000","last_fullsized":"2023-11-24T08:04:54.560535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537878+0000","last_change":"2023-11-24T08:04:53.521369+0000","last_active":"2023-11-24T08:04:54.537878+0000","last_peered":"2023-11-24T08:04:54.537878+0000","last_clean":"2023-11-24T08:04:54.537878+0000","last_became_active":"2023-11-24T08:04:53.520852+0000","last_became_peered":"2023-11-24T08:04:53.520852+0000","last_unstale":"2023-11-24T08:04:54.537878+0000","last_undegraded":"2023-11-24T08:04:54.537878+0000","last_fullsized":"2023-11-24T08:04:54.537878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697652+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.697652+0000","last_peered":"2023-11-24T08:04:55.697652+0000","last_clean":"2023-11-24T08:04:55.697652+0000","last_became_active":"2023-11-24T08:04:54.561649+0000","last_became_peered":"2023-11-24T08:04:54.561649+0000","last_unstale":"2023-11-24T08:04:55.697652+0000","last_undegraded":"2023-11-24T08:04:55.697652+0000","last_fullsized":"2023-11-24T08:04:55.697652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.517832+0000","last_change":"2023-11-24T08:04:53.517986+0000","last_active":"2023-11-24T08:04:54.517832+0000","last_peered":"2023-11-24T08:04:54.517832+0000","last_clean":"2023-11-24T08:04:54.517832+0000","last_became_active":"2023-11-24T08:04:53.517686+0000","last_became_peered":"2023-11-24T08:04:53.517686+0000","last_unstale":"2023-11-24T08:04:54.517832+0000","last_undegraded":"2023-11-24T08:04:54.517832+0000","last_fullsized":"2023-11-24T08:04:54.517832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535862+0000","last_change":"2023-11-24T08:04:54.548962+0000","last_active":"2023-11-24T08:04:55.535862+0000","last_peered":"2023-11-24T08:04:55.535862+0000","last_clean":"2023-11-24T08:04:55.535862+0000","last_became_active":"2023-11-24T08:04:54.548697+0000","last_became_peered":"2023-11-24T08:04:54.548697+0000","last_unstale":"2023-11-24T08:04:55.535862+0000","last_undegraded":"2023-11-24T08:04:55.535862+0000","last_fullsized":"2023-11-24T08:04:55.535862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.533360+0000","last_change":"2023-11-24T08:04:53.529719+0000","last_active":"2023-11-24T08:04:54.533360+0000","last_peered":"2023-11-24T08:04:54.533360+0000","last_clean":"2023-11-24T08:04:54.533360+0000","last_became_active":"2023-11-24T08:04:53.529289+0000","last_became_peered":"2023-11-24T08:04:53.529289+0000","last_unstale":"2023-11-24T08:04:54.533360+0000","last_undegraded":"2023-11-24T08:04:54.533360+0000","last_fullsized":"2023-11-24T08:04:54.533360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005952+0000","last_change":"2023-11-24T08:04:54.546903+0000","last_active":"2023-11-24T08:04:57.005952+0000","last_peered":"2023-11-24T08:04:57.005952+0000","last_clean":"2023-11-24T08:04:57.005952+0000","last_became_active":"2023-11-24T08:04:54.545578+0000","last_became_peered":"2023-11-24T08:04:54.545578+0000","last_unstale":"2023-11-24T08:04:57.005952+0000","last_undegraded":"2023-11-24T08:04:57.005952+0000","last_fullsized":"2023-11-24T08:04:57.005952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521698+0000","last_change":"2023-11-24T08:04:53.533034+0000","last_active":"2023-11-24T08:04:54.521698+0000","last_peered":"2023-11-24T08:04:54.521698+0000","last_clean":"2023-11-24T08:04:54.521698+0000","last_became_active":"2023-11-24T08:04:53.532393+0000","last_became_peered":"2023-11-24T08:04:53.532393+0000","last_unstale":"2023-11-24T08:04:54.521698+0000","last_undegraded":"2023-11-24T08:04:54.521698+0000","last_fullsized":"2023-11-24T08:04:54.521698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112283+0000","last_change":"2023-11-24T08:04:54.547926+0000","last_active":"2023-11-24T08:04:56.112283+0000","last_peered":"2023-11-24T08:04:56.112283+0000","last_clean":"2023-11-24T08:04:56.112283+0000","last_became_active":"2023-11-24T08:04:54.542598+0000","last_became_peered":"2023-11-24T08:04:54.542598+0000","last_unstale":"2023-11-24T08:04:56.112283+0000","last_undegraded":"2023-11-24T08:04:56.112283+0000","last_fullsized":"2023-11-24T08:04:56.112283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565554+0000","last_change":"2023-11-24T08:04:53.525910+0000","last_active":"2023-11-24T08:04:54.565554+0000","last_peered":"2023-11-24T08:04:54.565554+0000","last_clean":"2023-11-24T08:04:54.565554+0000","last_became_active":"2023-11-24T08:04:53.525661+0000","last_became_peered":"2023-11-24T08:04:53.525661+0000","last_unstale":"2023-11-24T08:04:54.565554+0000","last_undegraded":"2023-11-24T08:04:54.565554+0000","last_fullsized":"2023-11-24T08:04:54.565554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538064+0000","last_change":"2023-11-24T08:04:54.545080+0000","last_active":"2023-11-24T08:04:55.538064+0000","last_peered":"2023-11-24T08:04:55.538064+0000","last_clean":"2023-11-24T08:04:55.538064+0000","last_became_active":"2023-11-24T08:04:54.544031+0000","last_became_peered":"2023-11-24T08:04:54.544031+0000","last_unstale":"2023-11-24T08:04:55.538064+0000","last_undegraded":"2023-11-24T08:04:55.538064+0000","last_fullsized":"2023-11-24T08:04:55.538064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:47.133354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571132+0000","last_change":"2023-11-24T08:04:53.528212+0000","last_active":"2023-11-24T08:04:54.571132+0000","last_peered":"2023-11-24T08:04:54.571132+0000","last_clean":"2023-11-24T08:04:54.571132+0000","last_became_active":"2023-11-24T08:04:53.527458+0000","last_became_peered":"2023-11-24T08:04:53.527458+0000","last_unstale":"2023-11-24T08:04:54.571132+0000","last_undegraded":"2023-11-24T08:04:54.571132+0000","last_fullsized":"2023-11-24T08:04:54.571132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007910+0000","last_change":"2023-11-24T08:04:54.559327+0000","last_active":"2023-11-24T08:04:57.007910+0000","last_peered":"2023-11-24T08:04:57.007910+0000","last_clean":"2023-11-24T08:04:57.007910+0000","last_became_active":"2023-11-24T08:04:54.559034+0000","last_became_peered":"2023-11-24T08:04:54.559034+0000","last_unstale":"2023-11-24T08:04:57.007910+0000","last_undegraded":"2023-11-24T08:04:57.007910+0000","last_fullsized":"2023-11-24T08:04:57.007910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571070+0000","last_change":"2023-11-24T08:04:53.514829+0000","last_active":"2023-11-24T08:04:54.571070+0000","last_peered":"2023-11-24T08:04:54.571070+0000","last_clean":"2023-11-24T08:04:54.571070+0000","last_became_active":"2023-11-24T08:04:53.514590+0000","last_became_peered":"2023-11-24T08:04:53.514590+0000","last_unstale":"2023-11-24T08:04:54.571070+0000","last_undegraded":"2023-11-24T08:04:54.571070+0000","last_fullsized":"2023-11-24T08:04:54.571070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111807+0000","last_change":"2023-11-24T08:04:54.554473+0000","last_active":"2023-11-24T08:04:56.111807+0000","last_peered":"2023-11-24T08:04:56.111807+0000","last_clean":"2023-11-24T08:04:56.111807+0000","last_became_active":"2023-11-24T08:04:54.554157+0000","last_became_peered":"2023-11-24T08:04:54.554157+0000","last_unstale":"2023-11-24T08:04:56.111807+0000","last_undegraded":"2023-11-24T08:04:56.111807+0000","last_fullsized":"2023-11-24T08:04:56.111807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:33.329459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.524663+0000","last_change":"2023-11-24T08:04:53.526561+0000","last_active":"2023-11-24T08:04:54.524663+0000","last_peered":"2023-11-24T08:04:54.524663+0000","last_clean":"2023-11-24T08:04:54.524663+0000","last_became_active":"2023-11-24T08:04:53.525788+0000","last_became_peered":"2023-11-24T08:04:53.525788+0000","last_unstale":"2023-11-24T08:04:54.524663+0000","last_undegraded":"2023-11-24T08:04:54.524663+0000","last_fullsized":"2023-11-24T08:04:54.524663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009421+0000","last_change":"2023-11-24T08:04:54.547419+0000","last_active":"2023-11-24T08:04:57.009421+0000","last_peered":"2023-11-24T08:04:57.009421+0000","last_clean":"2023-11-24T08:04:57.009421+0000","last_became_active":"2023-11-24T08:04:54.547136+0000","last_became_peered":"2023-11-24T08:04:54.547136+0000","last_unstale":"2023-11-24T08:04:57.009421+0000","last_undegraded":"2023-11-24T08:04:57.009421+0000","last_fullsized":"2023-11-24T08:04:57.009421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:07:38.219756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.529667+0000","last_change":"2023-11-24T08:04:53.522781+0000","last_active":"2023-11-24T08:04:54.529667+0000","last_peered":"2023-11-24T08:04:54.529667+0000","last_clean":"2023-11-24T08:04:54.529667+0000","last_became_active":"2023-11-24T08:04:53.522413+0000","last_became_peered":"2023-11-24T08:04:53.522413+0000","last_unstale":"2023-11-24T08:04:54.529667+0000","last_undegraded":"2023-11-24T08:04:54.529667+0000","last_fullsized":"2023-11-24T08:04:54.529667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005124+0000","last_change":"2023-11-24T08:04:54.550954+0000","last_active":"2023-11-24T08:04:57.005124+0000","last_peered":"2023-11-24T08:04:57.005124+0000","last_clean":"2023-11-24T08:04:57.005124+0000","last_became_active":"2023-11-24T08:04:54.550405+0000","last_became_peered":"2023-11-24T08:04:54.550405+0000","last_unstale":"2023-11-24T08:04:57.005124+0000","last_undegraded":"2023-11-24T08:04:57.005124+0000","last_fullsized":"2023-11-24T08:04:57.005124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.569402+0000","last_change":"2023-11-24T08:04:53.517811+0000","last_active":"2023-11-24T08:04:54.569402+0000","last_peered":"2023-11-24T08:04:54.569402+0000","last_clean":"2023-11-24T08:04:54.569402+0000","last_became_active":"2023-11-24T08:04:53.517449+0000","last_became_peered":"2023-11-24T08:04:53.517449+0000","last_unstale":"2023-11-24T08:04:54.569402+0000","last_undegraded":"2023-11-24T08:04:54.569402+0000","last_fullsized":"2023-11-24T08:04:54.569402+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:17.696987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008505+0000","last_change":"2023-11-24T08:04:54.561976+0000","last_active":"2023-11-24T08:04:57.008505+0000","last_peered":"2023-11-24T08:04:57.008505+0000","last_clean":"2023-11-24T08:04:57.008505+0000","last_became_active":"2023-11-24T08:04:54.561433+0000","last_became_peered":"2023-11-24T08:04:54.561433+0000","last_unstale":"2023-11-24T08:04:57.008505+0000","last_undegraded":"2023-11-24T08:04:57.008505+0000","last_fullsized":"2023-11-24T08:04:57.008505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571680+0000","last_change":"2023-11-24T08:04:53.512021+0000","last_active":"2023-11-24T08:04:54.571680+0000","last_peered":"2023-11-24T08:04:54.571680+0000","last_clean":"2023-11-24T08:04:54.571680+0000","last_became_active":"2023-11-24T08:04:53.511682+0000","last_became_peered":"2023-11-24T08:04:53.511682+0000","last_unstale":"2023-11-24T08:04:54.571680+0000","last_undegraded":"2023-11-24T08:04:54.571680+0000","last_fullsized":"2023-11-24T08:04:54.571680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007859+0000","last_change":"2023-11-24T08:04:54.561544+0000","last_active":"2023-11-24T08:04:57.007859+0000","last_peered":"2023-11-24T08:04:57.007859+0000","last_clean":"2023-11-24T08:04:57.007859+0000","last_became_active":"2023-11-24T08:04:54.561157+0000","last_became_peered":"2023-11-24T08:04:54.561157+0000","last_unstale":"2023-11-24T08:04:57.007859+0000","last_undegraded":"2023-11-24T08:04:57.007859+0000","last_fullsized":"2023-11-24T08:04:57.007859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565958+0000","last_change":"2023-11-24T08:04:53.510122+0000","last_active":"2023-11-24T08:04:54.565958+0000","last_peered":"2023-11-24T08:04:54.565958+0000","last_clean":"2023-11-24T08:04:54.565958+0000","last_became_active":"2023-11-24T08:04:53.509763+0000","last_became_peered":"2023-11-24T08:04:53.509763+0000","last_unstale":"2023-11-24T08:04:54.565958+0000","last_undegraded":"2023-11-24T08:04:54.565958+0000","last_fullsized":"2023-11-24T08:04:54.565958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008573+0000","last_change":"2023-11-24T08:04:54.560061+0000","last_active":"2023-11-24T08:04:57.008573+0000","last_peered":"2023-11-24T08:04:57.008573+0000","last_clean":"2023-11-24T08:04:57.008573+0000","last_became_active":"2023-11-24T08:04:54.559752+0000","last_became_peered":"2023-11-24T08:04:54.559752+0000","last_unstale":"2023-11-24T08:04:57.008573+0000","last_undegraded":"2023-11-24T08:04:57.008573+0000","last_fullsized":"2023-11-24T08:04:57.008573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565408+0000","last_change":"2023-11-24T08:04:53.522019+0000","last_active":"2023-11-24T08:04:54.565408+0000","last_peered":"2023-11-24T08:04:54.565408+0000","last_clean":"2023-11-24T08:04:54.565408+0000","last_became_active":"2023-11-24T08:04:53.521742+0000","last_became_peered":"2023-11-24T08:04:53.521742+0000","last_unstale":"2023-11-24T08:04:54.565408+0000","last_undegraded":"2023-11-24T08:04:54.565408+0000","last_fullsized":"2023-11-24T08:04:54.565408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005948+0000","last_change":"2023-11-24T08:04:54.563074+0000","last_active":"2023-11-24T08:04:57.005948+0000","last_peered":"2023-11-24T08:04:57.005948+0000","last_clean":"2023-11-24T08:04:57.005948+0000","last_became_active":"2023-11-24T08:04:54.562794+0000","last_became_peered":"2023-11-24T08:04:54.562794+0000","last_unstale":"2023-11-24T08:04:57.005948+0000","last_undegraded":"2023-11-24T08:04:57.005948+0000","last_fullsized":"2023-11-24T08:04:57.005948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.540207+0000","last_change":"2023-11-24T08:04:53.532053+0000","last_active":"2023-11-24T08:04:54.540207+0000","last_peered":"2023-11-24T08:04:54.540207+0000","last_clean":"2023-11-24T08:04:54.540207+0000","last_became_active":"2023-11-24T08:04:53.531648+0000","last_became_peered":"2023-11-24T08:04:53.531648+0000","last_unstale":"2023-11-24T08:04:54.540207+0000","last_undegraded":"2023-11-24T08:04:54.540207+0000","last_fullsized":"2023-11-24T08:04:54.540207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008303+0000","last_change":"2023-11-24T08:04:54.556574+0000","last_active":"2023-11-24T08:04:57.008303+0000","last_peered":"2023-11-24T08:04:57.008303+0000","last_clean":"2023-11-24T08:04:57.008303+0000","last_became_active":"2023-11-24T08:04:54.555243+0000","last_became_peered":"2023-11-24T08:04:54.555243+0000","last_unstale":"2023-11-24T08:04:57.008303+0000","last_undegraded":"2023-11-24T08:04:57.008303+0000","last_fullsized":"2023-11-24T08:04:57.008303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.531104+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:04:54.531104+0000","last_peered":"2023-11-24T08:04:54.531104+0000","last_clean":"2023-11-24T08:04:54.531104+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:04:54.531104+0000","last_undegraded":"2023-11-24T08:04:54.531104+0000","last_fullsized":"2023-11-24T08:04:54.531104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521262+0000","last_change":"2023-11-24T08:04:53.524229+0000","last_active":"2023-11-24T08:04:54.521262+0000","last_peered":"2023-11-24T08:04:54.521262+0000","last_clean":"2023-11-24T08:04:54.521262+0000","last_became_active":"2023-11-24T08:04:53.523839+0000","last_became_peered":"2023-11-24T08:04:53.523839+0000","last_unstale":"2023-11-24T08:04:54.521262+0000","last_undegraded":"2023-11-24T08:04:54.521262+0000","last_fullsized":"2023-11-24T08:04:54.521262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698503+0000","last_change":"2023-11-24T08:04:54.549976+0000","last_active":"2023-11-24T08:04:55.698503+0000","last_peered":"2023-11-24T08:04:55.698503+0000","last_clean":"2023-11-24T08:04:55.698503+0000","last_became_active":"2023-11-24T08:04:54.549680+0000","last_became_peered":"2023-11-24T08:04:54.549680+0000","last_unstale":"2023-11-24T08:04:55.698503+0000","last_undegraded":"2023-11-24T08:04:55.698503+0000","last_fullsized":"2023-11-24T08:04:55.698503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.536596+0000","last_change":"2023-11-24T08:04:53.528808+0000","last_active":"2023-11-24T08:04:54.536596+0000","last_peered":"2023-11-24T08:04:54.536596+0000","last_clean":"2023-11-24T08:04:54.536596+0000","last_became_active":"2023-11-24T08:04:53.528482+0000","last_became_peered":"2023-11-24T08:04:53.528482+0000","last_unstale":"2023-11-24T08:04:54.536596+0000","last_undegraded":"2023-11-24T08:04:54.536596+0000","last_fullsized":"2023-11-24T08:04:54.536596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111534+0000","last_change":"2023-11-24T08:04:54.543382+0000","last_active":"2023-11-24T08:04:56.111534+0000","last_peered":"2023-11-24T08:04:56.111534+0000","last_clean":"2023-11-24T08:04:56.111534+0000","last_became_active":"2023-11-24T08:04:54.541869+0000","last_became_peered":"2023-11-24T08:04:54.541869+0000","last_unstale":"2023-11-24T08:04:56.111534+0000","last_undegraded":"2023-11-24T08:04:56.111534+0000","last_fullsized":"2023-11-24T08:04:56.111534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:43.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.538981+0000","last_change":"2023-11-24T08:04:53.509173+0000","last_active":"2023-11-24T08:04:54.538981+0000","last_peered":"2023-11-24T08:04:54.538981+0000","last_clean":"2023-11-24T08:04:54.538981+0000","last_became_active":"2023-11-24T08:04:53.507954+0000","last_became_peered":"2023-11-24T08:04:53.507954+0000","last_unstale":"2023-11-24T08:04:54.538981+0000","last_undegraded":"2023-11-24T08:04:54.538981+0000","last_fullsized":"2023-11-24T08:04:54.538981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535690+0000","last_change":"2023-11-24T08:04:54.551554+0000","last_active":"2023-11-24T08:04:55.535690+0000","last_peered":"2023-11-24T08:04:55.535690+0000","last_clean":"2023-11-24T08:04:55.535690+0000","last_became_active":"2023-11-24T08:04:54.546942+0000","last_became_peered":"2023-11-24T08:04:54.546942+0000","last_unstale":"2023-11-24T08:04:55.535690+0000","last_undegraded":"2023-11-24T08:04:55.535690+0000","last_fullsized":"2023-11-24T08:04:55.535690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.525076+0000","last_change":"2023-11-24T08:04:53.526686+0000","last_active":"2023-11-24T08:04:54.525076+0000","last_peered":"2023-11-24T08:04:54.525076+0000","last_clean":"2023-11-24T08:04:54.525076+0000","last_became_active":"2023-11-24T08:04:53.526157+0000","last_became_peered":"2023-11-24T08:04:53.526157+0000","last_unstale":"2023-11-24T08:04:54.525076+0000","last_undegraded":"2023-11-24T08:04:54.525076+0000","last_fullsized":"2023-11-24T08:04:54.525076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:38.092773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537976+0000","last_change":"2023-11-24T08:04:54.556189+0000","last_active":"2023-11-24T08:04:55.537976+0000","last_peered":"2023-11-24T08:04:55.537976+0000","last_clean":"2023-11-24T08:04:55.537976+0000","last_became_active":"2023-11-24T08:04:54.555903+0000","last_became_peered":"2023-11-24T08:04:54.555903+0000","last_unstale":"2023-11-24T08:04:55.537976+0000","last_undegraded":"2023-11-24T08:04:55.537976+0000","last_fullsized":"2023-11-24T08:04:55.537976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:18.709743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532442+0000","last_change":"2023-11-24T08:04:53.516220+0000","last_active":"2023-11-24T08:04:54.532442+0000","last_peered":"2023-11-24T08:04:54.532442+0000","last_clean":"2023-11-24T08:04:54.532442+0000","last_became_active":"2023-11-24T08:04:53.515013+0000","last_became_peered":"2023-11-24T08:04:53.515013+0000","last_unstale":"2023-11-24T08:04:54.532442+0000","last_undegraded":"2023-11-24T08:04:54.532442+0000","last_fullsized":"2023-11-24T08:04:54.532442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006151+0000","last_change":"2023-11-24T08:04:54.563230+0000","last_active":"2023-11-24T08:04:57.006151+0000","last_peered":"2023-11-24T08:04:57.006151+0000","last_clean":"2023-11-24T08:04:57.006151+0000","last_became_active":"2023-11-24T08:04:54.562908+0000","last_became_peered":"2023-11-24T08:04:54.562908+0000","last_unstale":"2023-11-24T08:04:57.006151+0000","last_undegraded":"2023-11-24T08:04:57.006151+0000","last_fullsized":"2023-11-24T08:04:57.006151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537608+0000","last_change":"2023-11-24T08:04:53.529958+0000","last_active":"2023-11-24T08:04:54.537608+0000","last_peered":"2023-11-24T08:04:54.537608+0000","last_clean":"2023-11-24T08:04:54.537608+0000","last_became_active":"2023-11-24T08:04:53.529676+0000","last_became_peered":"2023-11-24T08:04:53.529676+0000","last_unstale":"2023-11-24T08:04:54.537608+0000","last_undegraded":"2023-11-24T08:04:54.537608+0000","last_fullsized":"2023-11-24T08:04:54.537608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535951+0000","last_change":"2023-11-24T08:04:54.545570+0000","last_active":"2023-11-24T08:04:55.535951+0000","last_peered":"2023-11-24T08:04:55.535951+0000","last_clean":"2023-11-24T08:04:55.535951+0000","last_became_active":"2023-11-24T08:04:54.545282+0000","last_became_peered":"2023-11-24T08:04:54.545282+0000","last_unstale":"2023-11-24T08:04:55.535951+0000","last_undegraded":"2023-11-24T08:04:55.535951+0000","last_fullsized":"2023-11-24T08:04:55.535951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536611+0000","last_change":"2023-11-24T08:04:53.528071+0000","last_active":"2023-11-24T08:04:55.536611+0000","last_peered":"2023-11-24T08:04:55.536611+0000","last_clean":"2023-11-24T08:04:55.536611+0000","last_became_active":"2023-11-24T08:04:53.527761+0000","last_became_peered":"2023-11-24T08:04:53.527761+0000","last_unstale":"2023-11-24T08:04:55.536611+0000","last_undegraded":"2023-11-24T08:04:55.536611+0000","last_fullsized":"2023-11-24T08:04:55.536611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698850+0000","last_change":"2023-11-24T08:04:54.548899+0000","last_active":"2023-11-24T08:04:55.698850+0000","last_peered":"2023-11-24T08:04:55.698850+0000","last_clean":"2023-11-24T08:04:55.698850+0000","last_became_active":"2023-11-24T08:04:54.548630+0000","last_became_peered":"2023-11-24T08:04:54.548630+0000","last_unstale":"2023-11-24T08:04:55.698850+0000","last_undegraded":"2023-11-24T08:04:55.698850+0000","last_fullsized":"2023-11-24T08:04:55.698850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561594+0000","last_change":"2023-11-24T08:04:53.531179+0000","last_active":"2023-11-24T08:04:54.561594+0000","last_peered":"2023-11-24T08:04:54.561594+0000","last_clean":"2023-11-24T08:04:54.561594+0000","last_became_active":"2023-11-24T08:04:53.530771+0000","last_became_peered":"2023-11-24T08:04:53.530771+0000","last_unstale":"2023-11-24T08:04:54.561594+0000","last_undegraded":"2023-11-24T08:04:54.561594+0000","last_fullsized":"2023-11-24T08:04:54.561594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112025+0000","last_change":"2023-11-24T08:04:54.558549+0000","last_active":"2023-11-24T08:04:56.112025+0000","last_peered":"2023-11-24T08:04:56.112025+0000","last_clean":"2023-11-24T08:04:56.112025+0000","last_became_active":"2023-11-24T08:04:54.552869+0000","last_became_peered":"2023-11-24T08:04:54.552869+0000","last_unstale":"2023-11-24T08:04:56.112025+0000","last_undegraded":"2023-11-24T08:04:56.112025+0000","last_fullsized":"2023-11-24T08:04:56.112025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.530169+0000","last_change":"2023-11-24T08:04:53.530321+0000","last_active":"2023-11-24T08:04:54.530169+0000","last_peered":"2023-11-24T08:04:54.530169+0000","last_clean":"2023-11-24T08:04:54.530169+0000","last_became_active":"2023-11-24T08:04:53.529932+0000","last_became_peered":"2023-11-24T08:04:53.529932+0000","last_unstale":"2023-11-24T08:04:54.530169+0000","last_undegraded":"2023-11-24T08:04:54.530169+0000","last_fullsized":"2023-11-24T08:04:54.530169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537704+0000","last_change":"2023-11-24T08:04:54.553398+0000","last_active":"2023-11-24T08:04:55.537704+0000","last_peered":"2023-11-24T08:04:55.537704+0000","last_clean":"2023-11-24T08:04:55.537704+0000","last_became_active":"2023-11-24T08:04:54.552951+0000","last_became_peered":"2023-11-24T08:04:54.552951+0000","last_unstale":"2023-11-24T08:04:55.537704+0000","last_undegraded":"2023-11-24T08:04:55.537704+0000","last_fullsized":"2023-11-24T08:04:55.537704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:38.870187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.526579+0000","last_change":"2023-11-24T08:04:53.531280+0000","last_active":"2023-11-24T08:04:54.526579+0000","last_peered":"2023-11-24T08:04:54.526579+0000","last_clean":"2023-11-24T08:04:54.526579+0000","last_became_active":"2023-11-24T08:04:53.531012+0000","last_became_peered":"2023-11-24T08:04:53.531012+0000","last_unstale":"2023-11-24T08:04:54.526579+0000","last_undegraded":"2023-11-24T08:04:54.526579+0000","last_fullsized":"2023-11-24T08:04:54.526579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697931+0000","last_change":"2023-11-24T08:04:54.556299+0000","last_active":"2023-11-24T08:04:55.697931+0000","last_peered":"2023-11-24T08:04:55.697931+0000","last_clean":"2023-11-24T08:04:55.697931+0000","last_became_active":"2023-11-24T08:04:54.555882+0000","last_became_peered":"2023-11-24T08:04:54.555882+0000","last_unstale":"2023-11-24T08:04:55.697931+0000","last_undegraded":"2023-11-24T08:04:55.697931+0000","last_fullsized":"2023-11-24T08:04:55.697931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532819+0000","last_change":"2023-11-24T08:04:53.517295+0000","last_active":"2023-11-24T08:04:54.532819+0000","last_peered":"2023-11-24T08:04:54.532819+0000","last_clean":"2023-11-24T08:04:54.532819+0000","last_became_active":"2023-11-24T08:04:53.517017+0000","last_became_peered":"2023-11-24T08:04:53.517017+0000","last_unstale":"2023-11-24T08:04:54.532819+0000","last_undegraded":"2023-11-24T08:04:54.532819+0000","last_fullsized":"2023-11-24T08:04:54.532819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698757+0000","last_change":"2023-11-24T08:04:54.563699+0000","last_active":"2023-11-24T08:04:55.698757+0000","last_peered":"2023-11-24T08:04:55.698757+0000","last_clean":"2023-11-24T08:04:55.698757+0000","last_became_active":"2023-11-24T08:04:54.563266+0000","last_became_peered":"2023-11-24T08:04:54.563266+0000","last_unstale":"2023-11-24T08:04:55.698757+0000","last_undegraded":"2023-11-24T08:04:55.698757+0000","last_fullsized":"2023-11-24T08:04:55.698757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563981+0000","last_change":"2023-11-24T08:04:53.529030+0000","last_active":"2023-11-24T08:04:54.563981+0000","last_peered":"2023-11-24T08:04:54.563981+0000","last_clean":"2023-11-24T08:04:54.563981+0000","last_became_active":"2023-11-24T08:04:53.528620+0000","last_became_peered":"2023-11-24T08:04:53.528620+0000","last_unstale":"2023-11-24T08:04:54.563981+0000","last_undegraded":"2023-11-24T08:04:54.563981+0000","last_fullsized":"2023-11-24T08:04:54.563981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006725+0000","last_change":"2023-11-24T08:04:54.544336+0000","last_active":"2023-11-24T08:04:57.006725+0000","last_peered":"2023-11-24T08:04:57.006725+0000","last_clean":"2023-11-24T08:04:57.006725+0000","last_became_active":"2023-11-24T08:04:54.544020+0000","last_became_peered":"2023-11-24T08:04:54.544020+0000","last_unstale":"2023-11-24T08:04:57.006725+0000","last_undegraded":"2023-11-24T08:04:57.006725+0000","last_fullsized":"2023-11-24T08:04:57.006725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005795+0000","last_change":"2023-11-24T08:04:54.540666+0000","last_active":"2023-11-24T08:04:57.005795+0000","last_peered":"2023-11-24T08:04:57.005795+0000","last_clean":"2023-11-24T08:04:57.005795+0000","last_became_active":"2023-11-24T08:04:54.540347+0000","last_became_peered":"2023-11-24T08:04:54.540347+0000","last_unstale":"2023-11-24T08:04:57.005795+0000","last_undegraded":"2023-11-24T08:04:57.005795+0000","last_fullsized":"2023-11-24T08:04:57.005795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.559386+0000","last_change":"2023-11-24T08:04:53.533242+0000","last_active":"2023-11-24T08:04:54.559386+0000","last_peered":"2023-11-24T08:04:54.559386+0000","last_clean":"2023-11-24T08:04:54.559386+0000","last_became_active":"2023-11-24T08:04:53.532947+0000","last_became_peered":"2023-11-24T08:04:53.532947+0000","last_unstale":"2023-11-24T08:04:54.559386+0000","last_undegraded":"2023-11-24T08:04:54.559386+0000","last_fullsized":"2023-11-24T08:04:54.559386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:21.178905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005623+0000","last_change":"2023-11-24T08:04:54.551297+0000","last_active":"2023-11-24T08:04:57.005623+0000","last_peered":"2023-11-24T08:04:57.005623+0000","last_clean":"2023-11-24T08:04:57.005623+0000","last_became_active":"2023-11-24T08:04:54.548106+0000","last_became_peered":"2023-11-24T08:04:54.548106+0000","last_unstale":"2023-11-24T08:04:57.005623+0000","last_undegraded":"2023-11-24T08:04:57.005623+0000","last_fullsized":"2023-11-24T08:04:57.005623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561664+0000","last_change":"2023-11-24T08:04:53.528831+0000","last_active":"2023-11-24T08:04:54.561664+0000","last_peered":"2023-11-24T08:04:54.561664+0000","last_clean":"2023-11-24T08:04:54.561664+0000","last_became_active":"2023-11-24T08:04:53.528464+0000","last_became_peered":"2023-11-24T08:04:53.528464+0000","last_unstale":"2023-11-24T08:04:54.561664+0000","last_undegraded":"2023-11-24T08:04:54.561664+0000","last_fullsized":"2023-11-24T08:04:54.561664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:46.075298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112077+0000","last_change":"2023-11-24T08:04:54.551199+0000","last_active":"2023-11-24T08:04:56.112077+0000","last_peered":"2023-11-24T08:04:56.112077+0000","last_clean":"2023-11-24T08:04:56.112077+0000","last_became_active":"2023-11-24T08:04:54.548261+0000","last_became_peered":"2023-11-24T08:04:54.548261+0000","last_unstale":"2023-11-24T08:04:56.112077+0000","last_undegraded":"2023-11-24T08:04:56.112077+0000","last_fullsized":"2023-11-24T08:04:56.112077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.567550+0000","last_change":"2023-11-24T08:04:53.512889+0000","last_active":"2023-11-24T08:04:54.567550+0000","last_peered":"2023-11-24T08:04:54.567550+0000","last_clean":"2023-11-24T08:04:54.567550+0000","last_became_active":"2023-11-24T08:04:53.512580+0000","last_became_peered":"2023-11-24T08:04:53.512580+0000","last_unstale":"2023-11-24T08:04:54.567550+0000","last_undegraded":"2023-11-24T08:04:54.567550+0000","last_fullsized":"2023-11-24T08:04:54.567550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698368+0000","last_change":"2023-11-24T08:04:54.561224+0000","last_active":"2023-11-24T08:04:55.698368+0000","last_peered":"2023-11-24T08:04:55.698368+0000","last_clean":"2023-11-24T08:04:55.698368+0000","last_became_active":"2023-11-24T08:04:54.558658+0000","last_became_peered":"2023-11-24T08:04:54.558658+0000","last_unstale":"2023-11-24T08:04:55.698368+0000","last_undegraded":"2023-11-24T08:04:55.698368+0000","last_fullsized":"2023-11-24T08:04:55.698368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571974+0000","last_change":"2023-11-24T08:04:53.529967+0000","last_active":"2023-11-24T08:04:54.571974+0000","last_peered":"2023-11-24T08:04:54.571974+0000","last_clean":"2023-11-24T08:04:54.571974+0000","last_became_active":"2023-11-24T08:04:53.529738+0000","last_became_peered":"2023-11-24T08:04:53.529738+0000","last_unstale":"2023-11-24T08:04:54.571974+0000","last_undegraded":"2023-11-24T08:04:54.571974+0000","last_fullsized":"2023-11-24T08:04:54.571974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.141}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0950000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3439999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9510000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7160000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6869999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2349999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.929}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:05:04.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:05:04.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.934+0000 7fce7435a640 1 Processor -- start 2023-11-24T08:05:04.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.934+0000 7fce7435a640 1 -- start start 2023-11-24T08:05:04.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.935+0000 7fce7435a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce6c0b2250 0x7fce6c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce7435a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 0x7fce6c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce7435a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c15ab80 con 0x7fce6c0b2aa0 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce7435a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6c15a990 con 0x7fce6c0b2250 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce7435a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6c15e680 con 0x7fce6c0b2630 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce720f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce6c0b2250 0x7fce6c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce728f7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce6c0b2aa0 0x7fce6c15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59168/0 (socket says 172.21.15.107:59168) 2023-11-24T08:05:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce728f7640 1 -- 172.21.15.107:0/2629168126 learned_addr learned my addr 172.21.15.107:0/2629168126 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce718f5640 1 --2- 172.21.15.107:0/2629168126 >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 0x7fce6c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce720f6640 1 -- 172.21.15.107:0/2629168126 >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 msgr2=0x7fce6c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce720f6640 1 --2- 172.21.15.107:0/2629168126 >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 0x7fce6c0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce720f6640 1 -- 172.21.15.107:0/2629168126 >> v1:172.21.15.107:6789/0 conn(0x7fce6c0b2aa0 legacy=0x7fce6c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce720f6640 1 -- 172.21.15.107:0/2629168126 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c15e960 con 0x7fce6c0b2250 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce718f5640 1 --2- 172.21.15.107:0/2629168126 >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 0x7fce6c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.936+0000 7fce710f4640 1 -- 172.21.15.107:0/2629168126 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253370228 0 0) 0x7fce6c15ab80 con 0x7fce6c0b2aa0 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.937+0000 7fce720f6640 1 --2- 172.21.15.107:0/2629168126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2250 0x7fce6c15df80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0ae5c0 tx=0x7fce6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7de1d870f77f2c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce710f4640 1 -- 172.21.15.107:0/2629168126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68031830 con 0x7fce6c0b2250 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce710f4640 1 -- 172.21.15.107:0/2629168126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68031990 con 0x7fce6c0b2250 2023-11-24T08:05:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce710f4640 1 -- 172.21.15.107:0/2629168126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68031d20 con 0x7fce6c0b2250 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 -- 172.21.15.107:0/2629168126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2250 msgr2=0x7fce6c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 --2- 172.21.15.107:0/2629168126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2250 0x7fce6c15df80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0ae5c0 tx=0x7fce6802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 -- 172.21.15.107:0/2629168126 shutdown_connections 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 --2- 172.21.15.107:0/2629168126 >> v2:172.21.15.120:3300/0 conn(0x7fce6c0b2630 0x7fce6c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 --2- 172.21.15.107:0/2629168126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2250 0x7fce6c15df80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 -- 172.21.15.107:0/2629168126 >> 172.21.15.107:0/2629168126 conn(0x7fce6c06a920 msgr2=0x7fce6c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:04.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 -- 172.21.15.107:0/2629168126 shutdown_connections 2023-11-24T08:05:04.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.938+0000 7fce7435a640 1 -- 172.21.15.107:0/2629168126 wait complete. 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.940+0000 7fce7435a640 1 Processor -- start 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.940+0000 7fce7435a640 1 -- start start 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 0x7fce6c14b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce6c0b2aa0 0x7fce6c0ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c072050 con 0x7fce6c0b2630 2023-11-24T08:05:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c14b940 con 0x7fce6c0b2250 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce7435a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c071440 con 0x7fce6c0b2aa0 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44572/0 (socket says 172.21.15.107:44572) 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 -- 172.21.15.107:0/1469578368 learned_addr learned my addr 172.21.15.107:0/1469578368 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce720f6640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 0x7fce6c14b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 -- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce6c0b2aa0 msgr2=0x7fce6c0ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce728f7640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce6c0b2aa0 0x7fce6c0ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce6c0b2aa0 0x7fce6c0ac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 -- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 msgr2=0x7fce6c14b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 0x7fce6c14b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce718f5640 1 -- 172.21.15.107:0/1469578368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68009400 con 0x7fce6c0b2630 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.941+0000 7fce720f6640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 0x7fce6c14b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce718f5640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c7f0 tx=0x7fce5c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c007a30 con 0x7fce6c0b2630 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c007b90 con 0x7fce6c0b2630 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00ce40 con 0x7fce6c0b2630 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c071720 con 0x7fce6c0b2630 2023-11-24T08:05:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.942+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c071ce0 con 0x7fce6c0b2630 2023-11-24T08:05:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.943+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c072200 con 0x7fce6c0b2630 2023-11-24T08:05:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.944+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fce5c0113f0 con 0x7fce6c0b2630 2023-11-24T08:05:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.945+0000 7fce62ffd640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 0x7fce5006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.945+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce5c089800 con 0x7fce6c0b2630 2023-11-24T08:05:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.945+0000 7fce720f6640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 0x7fce5006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.946+0000 7fce720f6640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 0x7fce5006d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fce68036000 tx=0x7fce68008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:04.949+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce5c054810 con 0x7fce6c0b2630 2023-11-24T08:05:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.123+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce6c0cd470 con 0x7fce5006ae40 2023-11-24T08:05:05.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.144+0000 7fce62ffd640 1 -- 172.21.15.107:0/1469578368 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264390 (secure 0 0 0) 0x7fce68031630 con 0x7fce5006ae40 2023-11-24T08:05:05.148 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:05:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 msgr2=0x7fce5006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 0x7fce5006d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fce68036000 tx=0x7fce68008030 comp rx=0 tx=0).stop 2023-11-24T08:05:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 msgr2=0x7fce6c070a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c7f0 tx=0x7fce5c00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 shutdown_connections 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce5006ae40 0x7fce5006d300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce6c0b2aa0 0x7fce6c0ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6c0b2630 0x7fce6c070a60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 --2- 172.21.15.107:0/1469578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c0b2250 0x7fce6c14b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 >> 172.21.15.107:0/1469578368 conn(0x7fce6c06a920 msgr2=0x7fce6c0cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 shutdown_connections 2023-11-24T08:05:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.150+0000 7fce7435a640 1 -- 172.21.15.107:0/1469578368 wait complete. 2023-11-24T08:05:05.169 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-24T08:05:03.434041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.923023"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008109+0000","last_change":"2023-11-24T08:04:54.558355+0000","last_active":"2023-11-24T08:04:57.008109+0000","last_peered":"2023-11-24T08:04:57.008109+0000","last_clean":"2023-11-24T08:04:57.008109+0000","last_became_active":"2023-11-24T08:04:54.558044+0000","last_became_peered":"2023-11-24T08:04:54.558044+0000","last_unstale":"2023-11-24T08:04:57.008109+0000","last_undegraded":"2023-11-24T08:04:57.008109+0000","last_fullsized":"2023-11-24T08:04:57.008109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536483+0000","last_change":"2023-11-24T08:04:54.555145+0000","last_active":"2023-11-24T08:04:55.536483+0000","last_peered":"2023-11-24T08:04:55.536483+0000","last_clean":"2023-11-24T08:04:55.536483+0000","last_became_active":"2023-11-24T08:04:54.554789+0000","last_became_peered":"2023-11-24T08:04:54.554789+0000","last_unstale":"2023-11-24T08:04:55.536483+0000","last_undegraded":"2023-11-24T08:04:55.536483+0000","last_fullsized":"2023-11-24T08:04:55.536483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111906+0000","last_change":"2023-11-24T08:04:54.548731+0000","last_active":"2023-11-24T08:04:56.111906+0000","last_peered":"2023-11-24T08:04:56.111906+0000","last_clean":"2023-11-24T08:04:56.111906+0000","last_became_active":"2023-11-24T08:04:54.547639+0000","last_became_peered":"2023-11-24T08:04:54.547639+0000","last_unstale":"2023-11-24T08:04:56.111906+0000","last_undegraded":"2023-11-24T08:04:56.111906+0000","last_fullsized":"2023-11-24T08:04:56.111906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005247+0000","last_change":"2023-11-24T08:04:54.563561+0000","last_active":"2023-11-24T08:04:57.005247+0000","last_peered":"2023-11-24T08:04:57.005247+0000","last_clean":"2023-11-24T08:04:57.005247+0000","last_became_active":"2023-11-24T08:04:54.563217+0000","last_became_peered":"2023-11-24T08:04:54.563217+0000","last_unstale":"2023-11-24T08:04:57.005247+0000","last_undegraded":"2023-11-24T08:04:57.005247+0000","last_fullsized":"2023-11-24T08:04:57.005247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009305+0000","last_change":"2023-11-24T08:04:54.545576+0000","last_active":"2023-11-24T08:04:57.009305+0000","last_peered":"2023-11-24T08:04:57.009305+0000","last_clean":"2023-11-24T08:04:57.009305+0000","last_became_active":"2023-11-24T08:04:54.543956+0000","last_became_peered":"2023-11-24T08:04:54.543956+0000","last_unstale":"2023-11-24T08:04:57.009305+0000","last_undegraded":"2023-11-24T08:04:57.009305+0000","last_fullsized":"2023-11-24T08:04:57.009305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005154+0000","last_change":"2023-11-24T08:04:54.562290+0000","last_active":"2023-11-24T08:04:57.005154+0000","last_peered":"2023-11-24T08:04:57.005154+0000","last_clean":"2023-11-24T08:04:57.005154+0000","last_became_active":"2023-11-24T08:04:54.561367+0000","last_became_peered":"2023-11-24T08:04:54.561367+0000","last_unstale":"2023-11-24T08:04:57.005154+0000","last_undegraded":"2023-11-24T08:04:57.005154+0000","last_fullsized":"2023-11-24T08:04:57.005154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008255+0000","last_change":"2023-11-24T08:04:54.557527+0000","last_active":"2023-11-24T08:04:57.008255+0000","last_peered":"2023-11-24T08:04:57.008255+0000","last_clean":"2023-11-24T08:04:57.008255+0000","last_became_active":"2023-11-24T08:04:54.554705+0000","last_became_peered":"2023-11-24T08:04:54.554705+0000","last_unstale":"2023-11-24T08:04:57.008255+0000","last_undegraded":"2023-11-24T08:04:57.008255+0000","last_fullsized":"2023-11-24T08:04:57.008255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697542+0000","last_change":"2023-11-24T08:04:54.549339+0000","last_active":"2023-11-24T08:04:55.697542+0000","last_peered":"2023-11-24T08:04:55.697542+0000","last_clean":"2023-11-24T08:04:55.697542+0000","last_became_active":"2023-11-24T08:04:54.548540+0000","last_became_peered":"2023-11-24T08:04:54.548540+0000","last_unstale":"2023-11-24T08:04:55.697542+0000","last_undegraded":"2023-11-24T08:04:55.697542+0000","last_fullsized":"2023-11-24T08:04:55.697542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008315+0000","last_change":"2023-11-24T08:04:54.556229+0000","last_active":"2023-11-24T08:04:57.008315+0000","last_peered":"2023-11-24T08:04:57.008315+0000","last_clean":"2023-11-24T08:04:57.008315+0000","last_became_active":"2023-11-24T08:04:54.554805+0000","last_became_peered":"2023-11-24T08:04:54.554805+0000","last_unstale":"2023-11-24T08:04:57.008315+0000","last_undegraded":"2023-11-24T08:04:57.008315+0000","last_fullsized":"2023-11-24T08:04:57.008315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006892+0000","last_change":"2023-11-24T08:04:54.544285+0000","last_active":"2023-11-24T08:04:57.006892+0000","last_peered":"2023-11-24T08:04:57.006892+0000","last_clean":"2023-11-24T08:04:57.006892+0000","last_became_active":"2023-11-24T08:04:54.543957+0000","last_became_peered":"2023-11-24T08:04:54.543957+0000","last_unstale":"2023-11-24T08:04:57.006892+0000","last_undegraded":"2023-11-24T08:04:57.006892+0000","last_fullsized":"2023-11-24T08:04:57.006892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005608+0000","last_change":"2023-11-24T08:04:54.539976+0000","last_active":"2023-11-24T08:04:57.005608+0000","last_peered":"2023-11-24T08:04:57.005608+0000","last_clean":"2023-11-24T08:04:57.005608+0000","last_became_active":"2023-11-24T08:04:54.539639+0000","last_became_peered":"2023-11-24T08:04:54.539639+0000","last_unstale":"2023-11-24T08:04:57.005608+0000","last_undegraded":"2023-11-24T08:04:57.005608+0000","last_fullsized":"2023-11-24T08:04:57.005608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538051+0000","last_change":"2023-11-24T08:04:54.546809+0000","last_active":"2023-11-24T08:04:55.538051+0000","last_peered":"2023-11-24T08:04:55.538051+0000","last_clean":"2023-11-24T08:04:55.538051+0000","last_became_active":"2023-11-24T08:04:54.546568+0000","last_became_peered":"2023-11-24T08:04:54.546568+0000","last_unstale":"2023-11-24T08:04:55.538051+0000","last_undegraded":"2023-11-24T08:04:55.538051+0000","last_fullsized":"2023-11-24T08:04:55.538051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:27.753563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698433+0000","last_change":"2023-11-24T08:04:54.563512+0000","last_active":"2023-11-24T08:04:55.698433+0000","last_peered":"2023-11-24T08:04:55.698433+0000","last_clean":"2023-11-24T08:04:55.698433+0000","last_became_active":"2023-11-24T08:04:54.562308+0000","last_became_peered":"2023-11-24T08:04:54.562308+0000","last_unstale":"2023-11-24T08:04:55.698433+0000","last_undegraded":"2023-11-24T08:04:55.698433+0000","last_fullsized":"2023-11-24T08:04:55.698433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111574+0000","last_change":"2023-11-24T08:04:54.552082+0000","last_active":"2023-11-24T08:04:56.111574+0000","last_peered":"2023-11-24T08:04:56.111574+0000","last_clean":"2023-11-24T08:04:56.111574+0000","last_became_active":"2023-11-24T08:04:54.542957+0000","last_became_peered":"2023-11-24T08:04:54.542957+0000","last_unstale":"2023-11-24T08:04:56.111574+0000","last_undegraded":"2023-11-24T08:04:56.111574+0000","last_fullsized":"2023-11-24T08:04:56.111574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:10.754583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535932+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.535932+0000","last_peered":"2023-11-24T08:04:55.535932+0000","last_clean":"2023-11-24T08:04:55.535932+0000","last_became_active":"2023-11-24T08:04:54.561664+0000","last_became_peered":"2023-11-24T08:04:54.561664+0000","last_unstale":"2023-11-24T08:04:55.535932+0000","last_undegraded":"2023-11-24T08:04:55.535932+0000","last_fullsized":"2023-11-24T08:04:55.535932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006259+0000","last_change":"2023-11-24T08:04:54.545602+0000","last_active":"2023-11-24T08:04:57.006259+0000","last_peered":"2023-11-24T08:04:57.006259+0000","last_clean":"2023-11-24T08:04:57.006259+0000","last_became_active":"2023-11-24T08:04:54.545279+0000","last_became_peered":"2023-11-24T08:04:54.545279+0000","last_unstale":"2023-11-24T08:04:57.006259+0000","last_undegraded":"2023-11-24T08:04:57.006259+0000","last_fullsized":"2023-11-24T08:04:57.006259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006979+0000","last_change":"2023-11-24T08:04:54.546983+0000","last_active":"2023-11-24T08:04:57.006979+0000","last_peered":"2023-11-24T08:04:57.006979+0000","last_clean":"2023-11-24T08:04:57.006979+0000","last_became_active":"2023-11-24T08:04:54.545361+0000","last_became_peered":"2023-11-24T08:04:54.545361+0000","last_unstale":"2023-11-24T08:04:57.006979+0000","last_undegraded":"2023-11-24T08:04:57.006979+0000","last_fullsized":"2023-11-24T08:04:57.006979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:51.414779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002164+0000","last_change":"2023-11-24T08:04:54.556787+0000","last_active":"2023-11-24T08:04:57.002164+0000","last_peered":"2023-11-24T08:04:57.002164+0000","last_clean":"2023-11-24T08:04:57.002164+0000","last_became_active":"2023-11-24T08:04:54.556507+0000","last_became_peered":"2023-11-24T08:04:54.556507+0000","last_unstale":"2023-11-24T08:04:57.002164+0000","last_undegraded":"2023-11-24T08:04:57.002164+0000","last_fullsized":"2023-11-24T08:04:57.002164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:23.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008934+0000","last_change":"2023-11-24T08:04:54.544572+0000","last_active":"2023-11-24T08:04:57.008934+0000","last_peered":"2023-11-24T08:04:57.008934+0000","last_clean":"2023-11-24T08:04:57.008934+0000","last_became_active":"2023-11-24T08:04:54.544277+0000","last_became_peered":"2023-11-24T08:04:54.544277+0000","last_unstale":"2023-11-24T08:04:57.008934+0000","last_undegraded":"2023-11-24T08:04:57.008934+0000","last_fullsized":"2023-11-24T08:04:57.008934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006337+0000","last_change":"2023-11-24T08:04:54.549562+0000","last_active":"2023-11-24T08:04:57.006337+0000","last_peered":"2023-11-24T08:04:57.006337+0000","last_clean":"2023-11-24T08:04:57.006337+0000","last_became_active":"2023-11-24T08:04:54.549054+0000","last_became_peered":"2023-11-24T08:04:54.549054+0000","last_unstale":"2023-11-24T08:04:57.006337+0000","last_undegraded":"2023-11-24T08:04:57.006337+0000","last_fullsized":"2023-11-24T08:04:57.006337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111982+0000","last_change":"2023-11-24T08:04:54.554392+0000","last_active":"2023-11-24T08:04:56.111982+0000","last_peered":"2023-11-24T08:04:56.111982+0000","last_clean":"2023-11-24T08:04:56.111982+0000","last_became_active":"2023-11-24T08:04:54.551225+0000","last_became_peered":"2023-11-24T08:04:54.551225+0000","last_unstale":"2023-11-24T08:04:56.111982+0000","last_undegraded":"2023-11-24T08:04:56.111982+0000","last_fullsized":"2023-11-24T08:04:56.111982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005328+0000","last_change":"2023-11-24T08:04:54.545848+0000","last_active":"2023-11-24T08:04:57.005328+0000","last_peered":"2023-11-24T08:04:57.005328+0000","last_clean":"2023-11-24T08:04:57.005328+0000","last_became_active":"2023-11-24T08:04:54.545564+0000","last_became_peered":"2023-11-24T08:04:54.545564+0000","last_unstale":"2023-11-24T08:04:57.005328+0000","last_undegraded":"2023-11-24T08:04:57.005328+0000","last_fullsized":"2023-11-24T08:04:57.005328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697541+0000","last_change":"2023-11-24T08:04:54.566061+0000","last_active":"2023-11-24T08:04:55.697541+0000","last_peered":"2023-11-24T08:04:55.697541+0000","last_clean":"2023-11-24T08:04:55.697541+0000","last_became_active":"2023-11-24T08:04:54.563794+0000","last_became_peered":"2023-11-24T08:04:54.563794+0000","last_unstale":"2023-11-24T08:04:55.697541+0000","last_undegraded":"2023-11-24T08:04:55.697541+0000","last_fullsized":"2023-11-24T08:04:55.697541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006549+0000","last_change":"2023-11-24T08:04:54.549393+0000","last_active":"2023-11-24T08:04:57.006549+0000","last_peered":"2023-11-24T08:04:57.006549+0000","last_clean":"2023-11-24T08:04:57.006549+0000","last_became_active":"2023-11-24T08:04:54.549057+0000","last_became_peered":"2023-11-24T08:04:54.549057+0000","last_unstale":"2023-11-24T08:04:57.006549+0000","last_undegraded":"2023-11-24T08:04:57.006549+0000","last_fullsized":"2023-11-24T08:04:57.006549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:40.706380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698534+0000","last_change":"2023-11-24T08:04:54.541713+0000","last_active":"2023-11-24T08:04:55.698534+0000","last_peered":"2023-11-24T08:04:55.698534+0000","last_clean":"2023-11-24T08:04:55.698534+0000","last_became_active":"2023-11-24T08:04:54.541443+0000","last_became_peered":"2023-11-24T08:04:54.541443+0000","last_unstale":"2023-11-24T08:04:55.698534+0000","last_undegraded":"2023-11-24T08:04:55.698534+0000","last_fullsized":"2023-11-24T08:04:55.698534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007090+0000","last_change":"2023-11-24T08:04:54.544917+0000","last_active":"2023-11-24T08:04:57.007090+0000","last_peered":"2023-11-24T08:04:57.007090+0000","last_clean":"2023-11-24T08:04:57.007090+0000","last_became_active":"2023-11-24T08:04:54.544595+0000","last_became_peered":"2023-11-24T08:04:54.544595+0000","last_unstale":"2023-11-24T08:04:57.007090+0000","last_undegraded":"2023-11-24T08:04:57.007090+0000","last_fullsized":"2023-11-24T08:04:57.007090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536471+0000","last_change":"2023-11-24T08:04:54.558219+0000","last_active":"2023-11-24T08:04:55.536471+0000","last_peered":"2023-11-24T08:04:55.536471+0000","last_clean":"2023-11-24T08:04:55.536471+0000","last_became_active":"2023-11-24T08:04:54.557796+0000","last_became_peered":"2023-11-24T08:04:54.557796+0000","last_unstale":"2023-11-24T08:04:55.536471+0000","last_undegraded":"2023-11-24T08:04:55.536471+0000","last_fullsized":"2023-11-24T08:04:55.536471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698163+0000","last_change":"2023-11-24T08:04:54.561235+0000","last_active":"2023-11-24T08:04:55.698163+0000","last_peered":"2023-11-24T08:04:55.698163+0000","last_clean":"2023-11-24T08:04:55.698163+0000","last_became_active":"2023-11-24T08:04:54.560941+0000","last_became_peered":"2023-11-24T08:04:54.560941+0000","last_unstale":"2023-11-24T08:04:55.698163+0000","last_undegraded":"2023-11-24T08:04:55.698163+0000","last_fullsized":"2023-11-24T08:04:55.698163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112074+0000","last_change":"2023-11-24T08:04:54.553956+0000","last_active":"2023-11-24T08:04:56.112074+0000","last_peered":"2023-11-24T08:04:56.112074+0000","last_clean":"2023-11-24T08:04:56.112074+0000","last_became_active":"2023-11-24T08:04:54.553685+0000","last_became_peered":"2023-11-24T08:04:54.553685+0000","last_unstale":"2023-11-24T08:04:56.112074+0000","last_undegraded":"2023-11-24T08:04:56.112074+0000","last_fullsized":"2023-11-24T08:04:56.112074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005534+0000","last_change":"2023-11-24T08:04:54.560091+0000","last_active":"2023-11-24T08:04:57.005534+0000","last_peered":"2023-11-24T08:04:57.005534+0000","last_clean":"2023-11-24T08:04:57.005534+0000","last_became_active":"2023-11-24T08:04:54.559784+0000","last_became_peered":"2023-11-24T08:04:54.559784+0000","last_unstale":"2023-11-24T08:04:57.005534+0000","last_undegraded":"2023-11-24T08:04:57.005534+0000","last_fullsized":"2023-11-24T08:04:57.005534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.002234+0000","last_change":"2023-11-24T08:04:54.542929+0000","last_active":"2023-11-24T08:04:57.002234+0000","last_peered":"2023-11-24T08:04:57.002234+0000","last_clean":"2023-11-24T08:04:57.002234+0000","last_became_active":"2023-11-24T08:04:54.542574+0000","last_became_peered":"2023-11-24T08:04:54.542574+0000","last_unstale":"2023-11-24T08:04:57.002234+0000","last_undegraded":"2023-11-24T08:04:57.002234+0000","last_fullsized":"2023-11-24T08:04:57.002234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:51.449998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.004781+0000","last_change":"2023-11-24T08:04:54.547198+0000","last_active":"2023-11-24T08:04:57.004781+0000","last_peered":"2023-11-24T08:04:57.004781+0000","last_clean":"2023-11-24T08:04:57.004781+0000","last_became_active":"2023-11-24T08:04:54.546936+0000","last_became_peered":"2023-11-24T08:04:54.546936+0000","last_unstale":"2023-11-24T08:04:57.004781+0000","last_undegraded":"2023-11-24T08:04:57.004781+0000","last_fullsized":"2023-11-24T08:04:57.004781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537709+0000","last_change":"2023-11-24T08:04:54.542121+0000","last_active":"2023-11-24T08:04:55.537709+0000","last_peered":"2023-11-24T08:04:55.537709+0000","last_clean":"2023-11-24T08:04:55.537709+0000","last_became_active":"2023-11-24T08:04:54.541708+0000","last_became_peered":"2023-11-24T08:04:54.541708+0000","last_unstale":"2023-11-24T08:04:55.537709+0000","last_undegraded":"2023-11-24T08:04:55.537709+0000","last_fullsized":"2023-11-24T08:04:55.537709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:41.750215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563001+0000","last_change":"2023-11-24T08:04:53.530633+0000","last_active":"2023-11-24T08:04:54.563001+0000","last_peered":"2023-11-24T08:04:54.563001+0000","last_clean":"2023-11-24T08:04:54.563001+0000","last_became_active":"2023-11-24T08:04:53.530234+0000","last_became_peered":"2023-11-24T08:04:53.530234+0000","last_unstale":"2023-11-24T08:04:54.563001+0000","last_undegraded":"2023-11-24T08:04:54.563001+0000","last_fullsized":"2023-11-24T08:04:54.563001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112344+0000","last_change":"2023-11-24T08:04:54.563412+0000","last_active":"2023-11-24T08:04:56.112344+0000","last_peered":"2023-11-24T08:04:56.112344+0000","last_clean":"2023-11-24T08:04:56.112344+0000","last_became_active":"2023-11-24T08:04:54.561262+0000","last_became_peered":"2023-11-24T08:04:54.561262+0000","last_unstale":"2023-11-24T08:04:56.112344+0000","last_undegraded":"2023-11-24T08:04:56.112344+0000","last_fullsized":"2023-11-24T08:04:56.112344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:22.191660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.560535+0000","last_change":"2023-11-24T08:04:53.527156+0000","last_active":"2023-11-24T08:04:54.560535+0000","last_peered":"2023-11-24T08:04:54.560535+0000","last_clean":"2023-11-24T08:04:54.560535+0000","last_became_active":"2023-11-24T08:04:53.525686+0000","last_became_peered":"2023-11-24T08:04:53.525686+0000","last_unstale":"2023-11-24T08:04:54.560535+0000","last_undegraded":"2023-11-24T08:04:54.560535+0000","last_fullsized":"2023-11-24T08:04:54.560535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537878+0000","last_change":"2023-11-24T08:04:53.521369+0000","last_active":"2023-11-24T08:04:54.537878+0000","last_peered":"2023-11-24T08:04:54.537878+0000","last_clean":"2023-11-24T08:04:54.537878+0000","last_became_active":"2023-11-24T08:04:53.520852+0000","last_became_peered":"2023-11-24T08:04:53.520852+0000","last_unstale":"2023-11-24T08:04:54.537878+0000","last_undegraded":"2023-11-24T08:04:54.537878+0000","last_fullsized":"2023-11-24T08:04:54.537878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697652+0000","last_change":"2023-11-24T08:04:54.561993+0000","last_active":"2023-11-24T08:04:55.697652+0000","last_peered":"2023-11-24T08:04:55.697652+0000","last_clean":"2023-11-24T08:04:55.697652+0000","last_became_active":"2023-11-24T08:04:54.561649+0000","last_became_peered":"2023-11-24T08:04:54.561649+0000","last_unstale":"2023-11-24T08:04:55.697652+0000","last_undegraded":"2023-11-24T08:04:55.697652+0000","last_fullsized":"2023-11-24T08:04:55.697652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.517832+0000","last_change":"2023-11-24T08:04:53.517986+0000","last_active":"2023-11-24T08:04:54.517832+0000","last_peered":"2023-11-24T08:04:54.517832+0000","last_clean":"2023-11-24T08:04:54.517832+0000","last_became_active":"2023-11-24T08:04:53.517686+0000","last_became_peered":"2023-11-24T08:04:53.517686+0000","last_unstale":"2023-11-24T08:04:54.517832+0000","last_undegraded":"2023-11-24T08:04:54.517832+0000","last_fullsized":"2023-11-24T08:04:54.517832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535862+0000","last_change":"2023-11-24T08:04:54.548962+0000","last_active":"2023-11-24T08:04:55.535862+0000","last_peered":"2023-11-24T08:04:55.535862+0000","last_clean":"2023-11-24T08:04:55.535862+0000","last_became_active":"2023-11-24T08:04:54.548697+0000","last_became_peered":"2023-11-24T08:04:54.548697+0000","last_unstale":"2023-11-24T08:04:55.535862+0000","last_undegraded":"2023-11-24T08:04:55.535862+0000","last_fullsized":"2023-11-24T08:04:55.535862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:45.106981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.533360+0000","last_change":"2023-11-24T08:04:53.529719+0000","last_active":"2023-11-24T08:04:54.533360+0000","last_peered":"2023-11-24T08:04:54.533360+0000","last_clean":"2023-11-24T08:04:54.533360+0000","last_became_active":"2023-11-24T08:04:53.529289+0000","last_became_peered":"2023-11-24T08:04:53.529289+0000","last_unstale":"2023-11-24T08:04:54.533360+0000","last_undegraded":"2023-11-24T08:04:54.533360+0000","last_fullsized":"2023-11-24T08:04:54.533360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005952+0000","last_change":"2023-11-24T08:04:54.546903+0000","last_active":"2023-11-24T08:04:57.005952+0000","last_peered":"2023-11-24T08:04:57.005952+0000","last_clean":"2023-11-24T08:04:57.005952+0000","last_became_active":"2023-11-24T08:04:54.545578+0000","last_became_peered":"2023-11-24T08:04:54.545578+0000","last_unstale":"2023-11-24T08:04:57.005952+0000","last_undegraded":"2023-11-24T08:04:57.005952+0000","last_fullsized":"2023-11-24T08:04:57.005952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521698+0000","last_change":"2023-11-24T08:04:53.533034+0000","last_active":"2023-11-24T08:04:54.521698+0000","last_peered":"2023-11-24T08:04:54.521698+0000","last_clean":"2023-11-24T08:04:54.521698+0000","last_became_active":"2023-11-24T08:04:53.532393+0000","last_became_peered":"2023-11-24T08:04:53.532393+0000","last_unstale":"2023-11-24T08:04:54.521698+0000","last_undegraded":"2023-11-24T08:04:54.521698+0000","last_fullsized":"2023-11-24T08:04:54.521698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112283+0000","last_change":"2023-11-24T08:04:54.547926+0000","last_active":"2023-11-24T08:04:56.112283+0000","last_peered":"2023-11-24T08:04:56.112283+0000","last_clean":"2023-11-24T08:04:56.112283+0000","last_became_active":"2023-11-24T08:04:54.542598+0000","last_became_peered":"2023-11-24T08:04:54.542598+0000","last_unstale":"2023-11-24T08:04:56.112283+0000","last_undegraded":"2023-11-24T08:04:56.112283+0000","last_fullsized":"2023-11-24T08:04:56.112283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565554+0000","last_change":"2023-11-24T08:04:53.525910+0000","last_active":"2023-11-24T08:04:54.565554+0000","last_peered":"2023-11-24T08:04:54.565554+0000","last_clean":"2023-11-24T08:04:54.565554+0000","last_became_active":"2023-11-24T08:04:53.525661+0000","last_became_peered":"2023-11-24T08:04:53.525661+0000","last_unstale":"2023-11-24T08:04:54.565554+0000","last_undegraded":"2023-11-24T08:04:54.565554+0000","last_fullsized":"2023-11-24T08:04:54.565554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:11.104671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.538064+0000","last_change":"2023-11-24T08:04:54.545080+0000","last_active":"2023-11-24T08:04:55.538064+0000","last_peered":"2023-11-24T08:04:55.538064+0000","last_clean":"2023-11-24T08:04:55.538064+0000","last_became_active":"2023-11-24T08:04:54.544031+0000","last_became_peered":"2023-11-24T08:04:54.544031+0000","last_unstale":"2023-11-24T08:04:55.538064+0000","last_undegraded":"2023-11-24T08:04:55.538064+0000","last_fullsized":"2023-11-24T08:04:55.538064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:47.133354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571132+0000","last_change":"2023-11-24T08:04:53.528212+0000","last_active":"2023-11-24T08:04:54.571132+0000","last_peered":"2023-11-24T08:04:54.571132+0000","last_clean":"2023-11-24T08:04:54.571132+0000","last_became_active":"2023-11-24T08:04:53.527458+0000","last_became_peered":"2023-11-24T08:04:53.527458+0000","last_unstale":"2023-11-24T08:04:54.571132+0000","last_undegraded":"2023-11-24T08:04:54.571132+0000","last_fullsized":"2023-11-24T08:04:54.571132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007910+0000","last_change":"2023-11-24T08:04:54.559327+0000","last_active":"2023-11-24T08:04:57.007910+0000","last_peered":"2023-11-24T08:04:57.007910+0000","last_clean":"2023-11-24T08:04:57.007910+0000","last_became_active":"2023-11-24T08:04:54.559034+0000","last_became_peered":"2023-11-24T08:04:54.559034+0000","last_unstale":"2023-11-24T08:04:57.007910+0000","last_undegraded":"2023-11-24T08:04:57.007910+0000","last_fullsized":"2023-11-24T08:04:57.007910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:48.136389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571070+0000","last_change":"2023-11-24T08:04:53.514829+0000","last_active":"2023-11-24T08:04:54.571070+0000","last_peered":"2023-11-24T08:04:54.571070+0000","last_clean":"2023-11-24T08:04:54.571070+0000","last_became_active":"2023-11-24T08:04:53.514590+0000","last_became_peered":"2023-11-24T08:04:53.514590+0000","last_unstale":"2023-11-24T08:04:54.571070+0000","last_undegraded":"2023-11-24T08:04:54.571070+0000","last_fullsized":"2023-11-24T08:04:54.571070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111807+0000","last_change":"2023-11-24T08:04:54.554473+0000","last_active":"2023-11-24T08:04:56.111807+0000","last_peered":"2023-11-24T08:04:56.111807+0000","last_clean":"2023-11-24T08:04:56.111807+0000","last_became_active":"2023-11-24T08:04:54.554157+0000","last_became_peered":"2023-11-24T08:04:54.554157+0000","last_unstale":"2023-11-24T08:04:56.111807+0000","last_undegraded":"2023-11-24T08:04:56.111807+0000","last_fullsized":"2023-11-24T08:04:56.111807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:33.329459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.524663+0000","last_change":"2023-11-24T08:04:53.526561+0000","last_active":"2023-11-24T08:04:54.524663+0000","last_peered":"2023-11-24T08:04:54.524663+0000","last_clean":"2023-11-24T08:04:54.524663+0000","last_became_active":"2023-11-24T08:04:53.525788+0000","last_became_peered":"2023-11-24T08:04:53.525788+0000","last_unstale":"2023-11-24T08:04:54.524663+0000","last_undegraded":"2023-11-24T08:04:54.524663+0000","last_fullsized":"2023-11-24T08:04:54.524663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.009421+0000","last_change":"2023-11-24T08:04:54.547419+0000","last_active":"2023-11-24T08:04:57.009421+0000","last_peered":"2023-11-24T08:04:57.009421+0000","last_clean":"2023-11-24T08:04:57.009421+0000","last_became_active":"2023-11-24T08:04:54.547136+0000","last_became_peered":"2023-11-24T08:04:54.547136+0000","last_unstale":"2023-11-24T08:04:57.009421+0000","last_undegraded":"2023-11-24T08:04:57.009421+0000","last_fullsized":"2023-11-24T08:04:57.009421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:07:38.219756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.529667+0000","last_change":"2023-11-24T08:04:53.522781+0000","last_active":"2023-11-24T08:04:54.529667+0000","last_peered":"2023-11-24T08:04:54.529667+0000","last_clean":"2023-11-24T08:04:54.529667+0000","last_became_active":"2023-11-24T08:04:53.522413+0000","last_became_peered":"2023-11-24T08:04:53.522413+0000","last_unstale":"2023-11-24T08:04:54.529667+0000","last_undegraded":"2023-11-24T08:04:54.529667+0000","last_fullsized":"2023-11-24T08:04:54.529667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005124+0000","last_change":"2023-11-24T08:04:54.550954+0000","last_active":"2023-11-24T08:04:57.005124+0000","last_peered":"2023-11-24T08:04:57.005124+0000","last_clean":"2023-11-24T08:04:57.005124+0000","last_became_active":"2023-11-24T08:04:54.550405+0000","last_became_peered":"2023-11-24T08:04:54.550405+0000","last_unstale":"2023-11-24T08:04:57.005124+0000","last_undegraded":"2023-11-24T08:04:57.005124+0000","last_fullsized":"2023-11-24T08:04:57.005124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.569402+0000","last_change":"2023-11-24T08:04:53.517811+0000","last_active":"2023-11-24T08:04:54.569402+0000","last_peered":"2023-11-24T08:04:54.569402+0000","last_clean":"2023-11-24T08:04:54.569402+0000","last_became_active":"2023-11-24T08:04:53.517449+0000","last_became_peered":"2023-11-24T08:04:53.517449+0000","last_unstale":"2023-11-24T08:04:54.569402+0000","last_undegraded":"2023-11-24T08:04:54.569402+0000","last_fullsized":"2023-11-24T08:04:54.569402+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:17.696987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008505+0000","last_change":"2023-11-24T08:04:54.561976+0000","last_active":"2023-11-24T08:04:57.008505+0000","last_peered":"2023-11-24T08:04:57.008505+0000","last_clean":"2023-11-24T08:04:57.008505+0000","last_became_active":"2023-11-24T08:04:54.561433+0000","last_became_peered":"2023-11-24T08:04:54.561433+0000","last_unstale":"2023-11-24T08:04:57.008505+0000","last_undegraded":"2023-11-24T08:04:57.008505+0000","last_fullsized":"2023-11-24T08:04:57.008505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571680+0000","last_change":"2023-11-24T08:04:53.512021+0000","last_active":"2023-11-24T08:04:54.571680+0000","last_peered":"2023-11-24T08:04:54.571680+0000","last_clean":"2023-11-24T08:04:54.571680+0000","last_became_active":"2023-11-24T08:04:53.511682+0000","last_became_peered":"2023-11-24T08:04:53.511682+0000","last_unstale":"2023-11-24T08:04:54.571680+0000","last_undegraded":"2023-11-24T08:04:54.571680+0000","last_fullsized":"2023-11-24T08:04:54.571680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:41.799302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.007859+0000","last_change":"2023-11-24T08:04:54.561544+0000","last_active":"2023-11-24T08:04:57.007859+0000","last_peered":"2023-11-24T08:04:57.007859+0000","last_clean":"2023-11-24T08:04:57.007859+0000","last_became_active":"2023-11-24T08:04:54.561157+0000","last_became_peered":"2023-11-24T08:04:54.561157+0000","last_unstale":"2023-11-24T08:04:57.007859+0000","last_undegraded":"2023-11-24T08:04:57.007859+0000","last_fullsized":"2023-11-24T08:04:57.007859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565958+0000","last_change":"2023-11-24T08:04:53.510122+0000","last_active":"2023-11-24T08:04:54.565958+0000","last_peered":"2023-11-24T08:04:54.565958+0000","last_clean":"2023-11-24T08:04:54.565958+0000","last_became_active":"2023-11-24T08:04:53.509763+0000","last_became_peered":"2023-11-24T08:04:53.509763+0000","last_unstale":"2023-11-24T08:04:54.565958+0000","last_undegraded":"2023-11-24T08:04:54.565958+0000","last_fullsized":"2023-11-24T08:04:54.565958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008573+0000","last_change":"2023-11-24T08:04:54.560061+0000","last_active":"2023-11-24T08:04:57.008573+0000","last_peered":"2023-11-24T08:04:57.008573+0000","last_clean":"2023-11-24T08:04:57.008573+0000","last_became_active":"2023-11-24T08:04:54.559752+0000","last_became_peered":"2023-11-24T08:04:54.559752+0000","last_unstale":"2023-11-24T08:04:57.008573+0000","last_undegraded":"2023-11-24T08:04:57.008573+0000","last_fullsized":"2023-11-24T08:04:57.008573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:54.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.565408+0000","last_change":"2023-11-24T08:04:53.522019+0000","last_active":"2023-11-24T08:04:54.565408+0000","last_peered":"2023-11-24T08:04:54.565408+0000","last_clean":"2023-11-24T08:04:54.565408+0000","last_became_active":"2023-11-24T08:04:53.521742+0000","last_became_peered":"2023-11-24T08:04:53.521742+0000","last_unstale":"2023-11-24T08:04:54.565408+0000","last_undegraded":"2023-11-24T08:04:54.565408+0000","last_fullsized":"2023-11-24T08:04:54.565408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005948+0000","last_change":"2023-11-24T08:04:54.563074+0000","last_active":"2023-11-24T08:04:57.005948+0000","last_peered":"2023-11-24T08:04:57.005948+0000","last_clean":"2023-11-24T08:04:57.005948+0000","last_became_active":"2023-11-24T08:04:54.562794+0000","last_became_peered":"2023-11-24T08:04:54.562794+0000","last_unstale":"2023-11-24T08:04:57.005948+0000","last_undegraded":"2023-11-24T08:04:57.005948+0000","last_fullsized":"2023-11-24T08:04:57.005948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.540207+0000","last_change":"2023-11-24T08:04:53.532053+0000","last_active":"2023-11-24T08:04:54.540207+0000","last_peered":"2023-11-24T08:04:54.540207+0000","last_clean":"2023-11-24T08:04:54.540207+0000","last_became_active":"2023-11-24T08:04:53.531648+0000","last_became_peered":"2023-11-24T08:04:53.531648+0000","last_unstale":"2023-11-24T08:04:54.540207+0000","last_undegraded":"2023-11-24T08:04:54.540207+0000","last_fullsized":"2023-11-24T08:04:54.540207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.008303+0000","last_change":"2023-11-24T08:04:54.556574+0000","last_active":"2023-11-24T08:04:57.008303+0000","last_peered":"2023-11-24T08:04:57.008303+0000","last_clean":"2023-11-24T08:04:57.008303+0000","last_became_active":"2023-11-24T08:04:54.555243+0000","last_became_peered":"2023-11-24T08:04:54.555243+0000","last_unstale":"2023-11-24T08:04:57.008303+0000","last_undegraded":"2023-11-24T08:04:57.008303+0000","last_fullsized":"2023-11-24T08:04:57.008303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.531104+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:04:54.531104+0000","last_peered":"2023-11-24T08:04:54.531104+0000","last_clean":"2023-11-24T08:04:54.531104+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:04:54.531104+0000","last_undegraded":"2023-11-24T08:04:54.531104+0000","last_fullsized":"2023-11-24T08:04:54.531104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.521262+0000","last_change":"2023-11-24T08:04:53.524229+0000","last_active":"2023-11-24T08:04:54.521262+0000","last_peered":"2023-11-24T08:04:54.521262+0000","last_clean":"2023-11-24T08:04:54.521262+0000","last_became_active":"2023-11-24T08:04:53.523839+0000","last_became_peered":"2023-11-24T08:04:53.523839+0000","last_unstale":"2023-11-24T08:04:54.521262+0000","last_undegraded":"2023-11-24T08:04:54.521262+0000","last_fullsized":"2023-11-24T08:04:54.521262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:46.120599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698503+0000","last_change":"2023-11-24T08:04:54.549976+0000","last_active":"2023-11-24T08:04:55.698503+0000","last_peered":"2023-11-24T08:04:55.698503+0000","last_clean":"2023-11-24T08:04:55.698503+0000","last_became_active":"2023-11-24T08:04:54.549680+0000","last_became_peered":"2023-11-24T08:04:54.549680+0000","last_unstale":"2023-11-24T08:04:55.698503+0000","last_undegraded":"2023-11-24T08:04:55.698503+0000","last_fullsized":"2023-11-24T08:04:55.698503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.536596+0000","last_change":"2023-11-24T08:04:53.528808+0000","last_active":"2023-11-24T08:04:54.536596+0000","last_peered":"2023-11-24T08:04:54.536596+0000","last_clean":"2023-11-24T08:04:54.536596+0000","last_became_active":"2023-11-24T08:04:53.528482+0000","last_became_peered":"2023-11-24T08:04:53.528482+0000","last_unstale":"2023-11-24T08:04:54.536596+0000","last_undegraded":"2023-11-24T08:04:54.536596+0000","last_fullsized":"2023-11-24T08:04:54.536596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.111534+0000","last_change":"2023-11-24T08:04:54.543382+0000","last_active":"2023-11-24T08:04:56.111534+0000","last_peered":"2023-11-24T08:04:56.111534+0000","last_clean":"2023-11-24T08:04:56.111534+0000","last_became_active":"2023-11-24T08:04:54.541869+0000","last_became_peered":"2023-11-24T08:04:54.541869+0000","last_unstale":"2023-11-24T08:04:56.111534+0000","last_undegraded":"2023-11-24T08:04:56.111534+0000","last_fullsized":"2023-11-24T08:04:56.111534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:43.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.538981+0000","last_change":"2023-11-24T08:04:53.509173+0000","last_active":"2023-11-24T08:04:54.538981+0000","last_peered":"2023-11-24T08:04:54.538981+0000","last_clean":"2023-11-24T08:04:54.538981+0000","last_became_active":"2023-11-24T08:04:53.507954+0000","last_became_peered":"2023-11-24T08:04:53.507954+0000","last_unstale":"2023-11-24T08:04:54.538981+0000","last_undegraded":"2023-11-24T08:04:54.538981+0000","last_fullsized":"2023-11-24T08:04:54.538981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:40.737459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535690+0000","last_change":"2023-11-24T08:04:54.551554+0000","last_active":"2023-11-24T08:04:55.535690+0000","last_peered":"2023-11-24T08:04:55.535690+0000","last_clean":"2023-11-24T08:04:55.535690+0000","last_became_active":"2023-11-24T08:04:54.546942+0000","last_became_peered":"2023-11-24T08:04:54.546942+0000","last_unstale":"2023-11-24T08:04:55.535690+0000","last_undegraded":"2023-11-24T08:04:55.535690+0000","last_fullsized":"2023-11-24T08:04:55.535690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:47.088054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.525076+0000","last_change":"2023-11-24T08:04:53.526686+0000","last_active":"2023-11-24T08:04:54.525076+0000","last_peered":"2023-11-24T08:04:54.525076+0000","last_clean":"2023-11-24T08:04:54.525076+0000","last_became_active":"2023-11-24T08:04:53.526157+0000","last_became_peered":"2023-11-24T08:04:53.526157+0000","last_unstale":"2023-11-24T08:04:54.525076+0000","last_undegraded":"2023-11-24T08:04:54.525076+0000","last_fullsized":"2023-11-24T08:04:54.525076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:38.092773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537976+0000","last_change":"2023-11-24T08:04:54.556189+0000","last_active":"2023-11-24T08:04:55.537976+0000","last_peered":"2023-11-24T08:04:55.537976+0000","last_clean":"2023-11-24T08:04:55.537976+0000","last_became_active":"2023-11-24T08:04:54.555903+0000","last_became_peered":"2023-11-24T08:04:54.555903+0000","last_unstale":"2023-11-24T08:04:55.537976+0000","last_undegraded":"2023-11-24T08:04:55.537976+0000","last_fullsized":"2023-11-24T08:04:55.537976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:18.709743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532442+0000","last_change":"2023-11-24T08:04:53.516220+0000","last_active":"2023-11-24T08:04:54.532442+0000","last_peered":"2023-11-24T08:04:54.532442+0000","last_clean":"2023-11-24T08:04:54.532442+0000","last_became_active":"2023-11-24T08:04:53.515013+0000","last_became_peered":"2023-11-24T08:04:53.515013+0000","last_unstale":"2023-11-24T08:04:54.532442+0000","last_undegraded":"2023-11-24T08:04:54.532442+0000","last_fullsized":"2023-11-24T08:04:54.532442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:09.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006151+0000","last_change":"2023-11-24T08:04:54.563230+0000","last_active":"2023-11-24T08:04:57.006151+0000","last_peered":"2023-11-24T08:04:57.006151+0000","last_clean":"2023-11-24T08:04:57.006151+0000","last_became_active":"2023-11-24T08:04:54.562908+0000","last_became_peered":"2023-11-24T08:04:54.562908+0000","last_unstale":"2023-11-24T08:04:57.006151+0000","last_undegraded":"2023-11-24T08:04:57.006151+0000","last_fullsized":"2023-11-24T08:04:57.006151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:51.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.537608+0000","last_change":"2023-11-24T08:04:53.529958+0000","last_active":"2023-11-24T08:04:54.537608+0000","last_peered":"2023-11-24T08:04:54.537608+0000","last_clean":"2023-11-24T08:04:54.537608+0000","last_became_active":"2023-11-24T08:04:53.529676+0000","last_became_peered":"2023-11-24T08:04:53.529676+0000","last_unstale":"2023-11-24T08:04:54.537608+0000","last_undegraded":"2023-11-24T08:04:54.537608+0000","last_fullsized":"2023-11-24T08:04:54.537608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:27.498121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.535951+0000","last_change":"2023-11-24T08:04:54.545570+0000","last_active":"2023-11-24T08:04:55.535951+0000","last_peered":"2023-11-24T08:04:55.535951+0000","last_clean":"2023-11-24T08:04:55.535951+0000","last_became_active":"2023-11-24T08:04:54.545282+0000","last_became_peered":"2023-11-24T08:04:54.545282+0000","last_unstale":"2023-11-24T08:04:55.535951+0000","last_undegraded":"2023-11-24T08:04:55.535951+0000","last_fullsized":"2023-11-24T08:04:55.535951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:17:39.105528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.536611+0000","last_change":"2023-11-24T08:04:53.528071+0000","last_active":"2023-11-24T08:04:55.536611+0000","last_peered":"2023-11-24T08:04:55.536611+0000","last_clean":"2023-11-24T08:04:55.536611+0000","last_became_active":"2023-11-24T08:04:53.527761+0000","last_became_peered":"2023-11-24T08:04:53.527761+0000","last_unstale":"2023-11-24T08:04:55.536611+0000","last_undegraded":"2023-11-24T08:04:55.536611+0000","last_fullsized":"2023-11-24T08:04:55.536611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698850+0000","last_change":"2023-11-24T08:04:54.548899+0000","last_active":"2023-11-24T08:04:55.698850+0000","last_peered":"2023-11-24T08:04:55.698850+0000","last_clean":"2023-11-24T08:04:55.698850+0000","last_became_active":"2023-11-24T08:04:54.548630+0000","last_became_peered":"2023-11-24T08:04:54.548630+0000","last_unstale":"2023-11-24T08:04:55.698850+0000","last_undegraded":"2023-11-24T08:04:55.698850+0000","last_fullsized":"2023-11-24T08:04:55.698850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:15:58.733561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561594+0000","last_change":"2023-11-24T08:04:53.531179+0000","last_active":"2023-11-24T08:04:54.561594+0000","last_peered":"2023-11-24T08:04:54.561594+0000","last_clean":"2023-11-24T08:04:54.561594+0000","last_became_active":"2023-11-24T08:04:53.530771+0000","last_became_peered":"2023-11-24T08:04:53.530771+0000","last_unstale":"2023-11-24T08:04:54.561594+0000","last_undegraded":"2023-11-24T08:04:54.561594+0000","last_fullsized":"2023-11-24T08:04:54.561594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:42.431127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112025+0000","last_change":"2023-11-24T08:04:54.558549+0000","last_active":"2023-11-24T08:04:56.112025+0000","last_peered":"2023-11-24T08:04:56.112025+0000","last_clean":"2023-11-24T08:04:56.112025+0000","last_became_active":"2023-11-24T08:04:54.552869+0000","last_became_peered":"2023-11-24T08:04:54.552869+0000","last_unstale":"2023-11-24T08:04:56.112025+0000","last_undegraded":"2023-11-24T08:04:56.112025+0000","last_fullsized":"2023-11-24T08:04:56.112025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:28.510877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.530169+0000","last_change":"2023-11-24T08:04:53.530321+0000","last_active":"2023-11-24T08:04:54.530169+0000","last_peered":"2023-11-24T08:04:54.530169+0000","last_clean":"2023-11-24T08:04:54.530169+0000","last_became_active":"2023-11-24T08:04:53.529932+0000","last_became_peered":"2023-11-24T08:04:53.529932+0000","last_unstale":"2023-11-24T08:04:54.530169+0000","last_undegraded":"2023-11-24T08:04:54.530169+0000","last_fullsized":"2023-11-24T08:04:54.530169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.537704+0000","last_change":"2023-11-24T08:04:54.553398+0000","last_active":"2023-11-24T08:04:55.537704+0000","last_peered":"2023-11-24T08:04:55.537704+0000","last_clean":"2023-11-24T08:04:55.537704+0000","last_became_active":"2023-11-24T08:04:54.552951+0000","last_became_peered":"2023-11-24T08:04:54.552951+0000","last_unstale":"2023-11-24T08:04:55.537704+0000","last_undegraded":"2023-11-24T08:04:55.537704+0000","last_fullsized":"2023-11-24T08:04:55.537704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:38.870187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.526579+0000","last_change":"2023-11-24T08:04:53.531280+0000","last_active":"2023-11-24T08:04:54.526579+0000","last_peered":"2023-11-24T08:04:54.526579+0000","last_clean":"2023-11-24T08:04:54.526579+0000","last_became_active":"2023-11-24T08:04:53.531012+0000","last_became_peered":"2023-11-24T08:04:53.531012+0000","last_unstale":"2023-11-24T08:04:54.526579+0000","last_undegraded":"2023-11-24T08:04:54.526579+0000","last_fullsized":"2023-11-24T08:04:54.526579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:26.740808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.697931+0000","last_change":"2023-11-24T08:04:54.556299+0000","last_active":"2023-11-24T08:04:55.697931+0000","last_peered":"2023-11-24T08:04:55.697931+0000","last_clean":"2023-11-24T08:04:55.697931+0000","last_became_active":"2023-11-24T08:04:54.555882+0000","last_became_peered":"2023-11-24T08:04:54.555882+0000","last_unstale":"2023-11-24T08:04:55.697931+0000","last_undegraded":"2023-11-24T08:04:55.697931+0000","last_fullsized":"2023-11-24T08:04:55.697931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.532819+0000","last_change":"2023-11-24T08:04:53.517295+0000","last_active":"2023-11-24T08:04:54.532819+0000","last_peered":"2023-11-24T08:04:54.532819+0000","last_clean":"2023-11-24T08:04:54.532819+0000","last_became_active":"2023-11-24T08:04:53.517017+0000","last_became_peered":"2023-11-24T08:04:53.517017+0000","last_unstale":"2023-11-24T08:04:54.532819+0000","last_undegraded":"2023-11-24T08:04:54.532819+0000","last_fullsized":"2023-11-24T08:04:54.532819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698757+0000","last_change":"2023-11-24T08:04:54.563699+0000","last_active":"2023-11-24T08:04:55.698757+0000","last_peered":"2023-11-24T08:04:55.698757+0000","last_clean":"2023-11-24T08:04:55.698757+0000","last_became_active":"2023-11-24T08:04:54.563266+0000","last_became_peered":"2023-11-24T08:04:54.563266+0000","last_unstale":"2023-11-24T08:04:55.698757+0000","last_undegraded":"2023-11-24T08:04:55.698757+0000","last_fullsized":"2023-11-24T08:04:55.698757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:55.692040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.563981+0000","last_change":"2023-11-24T08:04:53.529030+0000","last_active":"2023-11-24T08:04:54.563981+0000","last_peered":"2023-11-24T08:04:54.563981+0000","last_clean":"2023-11-24T08:04:54.563981+0000","last_became_active":"2023-11-24T08:04:53.528620+0000","last_became_peered":"2023-11-24T08:04:53.528620+0000","last_unstale":"2023-11-24T08:04:54.563981+0000","last_undegraded":"2023-11-24T08:04:54.563981+0000","last_fullsized":"2023-11-24T08:04:54.563981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.006725+0000","last_change":"2023-11-24T08:04:54.544336+0000","last_active":"2023-11-24T08:04:57.006725+0000","last_peered":"2023-11-24T08:04:57.006725+0000","last_clean":"2023-11-24T08:04:57.006725+0000","last_became_active":"2023-11-24T08:04:54.544020+0000","last_became_peered":"2023-11-24T08:04:54.544020+0000","last_unstale":"2023-11-24T08:04:57.006725+0000","last_undegraded":"2023-11-24T08:04:57.006725+0000","last_fullsized":"2023-11-24T08:04:57.006725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:31.087845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005795+0000","last_change":"2023-11-24T08:04:54.540666+0000","last_active":"2023-11-24T08:04:57.005795+0000","last_peered":"2023-11-24T08:04:57.005795+0000","last_clean":"2023-11-24T08:04:57.005795+0000","last_became_active":"2023-11-24T08:04:54.540347+0000","last_became_peered":"2023-11-24T08:04:54.540347+0000","last_unstale":"2023-11-24T08:04:57.005795+0000","last_undegraded":"2023-11-24T08:04:57.005795+0000","last_fullsized":"2023-11-24T08:04:57.005795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:42:09.850992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.559386+0000","last_change":"2023-11-24T08:04:53.533242+0000","last_active":"2023-11-24T08:04:54.559386+0000","last_peered":"2023-11-24T08:04:54.559386+0000","last_clean":"2023-11-24T08:04:54.559386+0000","last_became_active":"2023-11-24T08:04:53.532947+0000","last_became_peered":"2023-11-24T08:04:53.532947+0000","last_unstale":"2023-11-24T08:04:54.559386+0000","last_undegraded":"2023-11-24T08:04:54.559386+0000","last_fullsized":"2023-11-24T08:04:54.559386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:21.178905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:57.005623+0000","last_change":"2023-11-24T08:04:54.551297+0000","last_active":"2023-11-24T08:04:57.005623+0000","last_peered":"2023-11-24T08:04:57.005623+0000","last_clean":"2023-11-24T08:04:57.005623+0000","last_became_active":"2023-11-24T08:04:54.548106+0000","last_became_peered":"2023-11-24T08:04:54.548106+0000","last_unstale":"2023-11-24T08:04:57.005623+0000","last_undegraded":"2023-11-24T08:04:57.005623+0000","last_fullsized":"2023-11-24T08:04:57.005623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:55.500424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.561664+0000","last_change":"2023-11-24T08:04:53.528831+0000","last_active":"2023-11-24T08:04:54.561664+0000","last_peered":"2023-11-24T08:04:54.561664+0000","last_clean":"2023-11-24T08:04:54.561664+0000","last_became_active":"2023-11-24T08:04:53.528464+0000","last_became_peered":"2023-11-24T08:04:53.528464+0000","last_unstale":"2023-11-24T08:04:54.561664+0000","last_undegraded":"2023-11-24T08:04:54.561664+0000","last_fullsized":"2023-11-24T08:04:54.561664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:46.075298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:56.112077+0000","last_change":"2023-11-24T08:04:54.551199+0000","last_active":"2023-11-24T08:04:56.112077+0000","last_peered":"2023-11-24T08:04:56.112077+0000","last_clean":"2023-11-24T08:04:56.112077+0000","last_became_active":"2023-11-24T08:04:54.548261+0000","last_became_peered":"2023-11-24T08:04:54.548261+0000","last_unstale":"2023-11-24T08:04:56.112077+0000","last_undegraded":"2023-11-24T08:04:56.112077+0000","last_fullsized":"2023-11-24T08:04:56.112077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:08.543496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.567550+0000","last_change":"2023-11-24T08:04:53.512889+0000","last_active":"2023-11-24T08:04:54.567550+0000","last_peered":"2023-11-24T08:04:54.567550+0000","last_clean":"2023-11-24T08:04:54.567550+0000","last_became_active":"2023-11-24T08:04:53.512580+0000","last_became_peered":"2023-11-24T08:04:53.512580+0000","last_unstale":"2023-11-24T08:04:54.567550+0000","last_undegraded":"2023-11-24T08:04:54.567550+0000","last_fullsized":"2023-11-24T08:04:54.567550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:32.316704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T08:04:55.698368+0000","last_change":"2023-11-24T08:04:54.561224+0000","last_active":"2023-11-24T08:04:55.698368+0000","last_peered":"2023-11-24T08:04:55.698368+0000","last_clean":"2023-11-24T08:04:55.698368+0000","last_became_active":"2023-11-24T08:04:54.558658+0000","last_became_peered":"2023-11-24T08:04:54.558658+0000","last_unstale":"2023-11-24T08:04:55.698368+0000","last_undegraded":"2023-11-24T08:04:55.698368+0000","last_fullsized":"2023-11-24T08:04:55.698368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:53.480238+0000","last_clean_scrub_stamp":"2023-11-24T08:04:53.480238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:07.578682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T08:04:54.571974+0000","last_change":"2023-11-24T08:04:53.529967+0000","last_active":"2023-11-24T08:04:54.571974+0000","last_peered":"2023-11-24T08:04:54.571974+0000","last_clean":"2023-11-24T08:04:54.571974+0000","last_became_active":"2023-11-24T08:04:53.529738+0000","last_became_peered":"2023-11-24T08:04:53.529738+0000","last_unstale":"2023-11-24T08:04:54.571974+0000","last_undegraded":"2023-11-24T08:04:54.571974+0000","last_fullsized":"2023-11-24T08:04:54.571974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:52.467483+0000","last_clean_scrub_stamp":"2023-11-24T08:04:52.467483+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:22.932967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.141}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0950000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3439999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9510000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7160000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6869999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2349999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.929}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:05:05.174 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-24T08:05:05.174 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-24T08:05:05.174 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-24T08:05:05.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-24T08:05:05.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.415+0000 7f82796cf640 1 Processor -- start 2023-11-24T08:05:05.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.416+0000 7f82796cf640 1 -- start start 2023-11-24T08:05:05.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82796cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82740cfb60 0x7f8274161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82796cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740ce0b0 con 0x7f827415dc80 2023-11-24T08:05:05.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82796cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8274161960 con 0x7f8274157c70 2023-11-24T08:05:05.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82796cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827415e8d0 con 0x7f82740cfb60 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82737fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f827415dc80 0x7f82740b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59176/0 (socket says 172.21.15.107:59176) 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82737fe640 1 -- 172.21.15.107:0/4166952040 learned_addr learned my addr 172.21.15.107:0/4166952040 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.417+0000 7f82727fc640 1 --2- 172.21.15.107:0/4166952040 >> v2:172.21.15.107:3300/0 conn(0x7f82740cfb60 0x7f8274161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.418+0000 7f82727fc640 1 -- 172.21.15.107:0/4166952040 >> v1:172.21.15.120:6789/0 conn(0x7f8274157c70 legacy=0x7f82740cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.418+0000 7f82727fc640 1 -- 172.21.15.107:0/4166952040 >> v1:172.21.15.107:6789/0 conn(0x7f827415dc80 legacy=0x7f82740b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.418+0000 7f8271ffb640 1 -- 172.21.15.107:0/4166952040 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935602704 0 0) 0x7f82740ce0b0 con 0x7f827415dc80 2023-11-24T08:05:05.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.418+0000 7f82727fc640 1 -- 172.21.15.107:0/4166952040 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827415ebb0 con 0x7f82740cfb60 2023-11-24T08:05:05.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.419+0000 7f82727fc640 1 --2- 172.21.15.107:0/4166952040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f8274161df0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f825c0097a0 tx=0x7f825c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80346743e02730f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.419+0000 7f8271ffb640 1 -- 172.21.15.107:0/4166952040 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c031830 con 0x7f82740cfb60 2023-11-24T08:05:05.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.419+0000 7f8271ffb640 1 -- 172.21.15.107:0/4166952040 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c031990 con 0x7f82740cfb60 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.419+0000 7f8271ffb640 1 -- 172.21.15.107:0/4166952040 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c031d20 con 0x7f82740cfb60 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 -- 172.21.15.107:0/4166952040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 msgr2=0x7f8274161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 --2- 172.21.15.107:0/4166952040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f8274161df0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f825c0097a0 tx=0x7f825c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 -- 172.21.15.107:0/4166952040 shutdown_connections 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 --2- 172.21.15.107:0/4166952040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f8274161df0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 -- 172.21.15.107:0/4166952040 >> 172.21.15.107:0/4166952040 conn(0x7f827406a920 msgr2=0x7f827406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 -- 172.21.15.107:0/4166952040 shutdown_connections 2023-11-24T08:05:05.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.420+0000 7f82796cf640 1 -- 172.21.15.107:0/4166952040 wait complete. 2023-11-24T08:05:05.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.422+0000 7f82796cf640 1 Processor -- start 2023-11-24T08:05:05.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.422+0000 7f82796cf640 1 -- start start 2023-11-24T08:05:05.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8274157c70 0x7f8274149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f827415dc80 0x7f8274149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cb160 con 0x7f82740cfb60 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274149eb0 con 0x7f8274157c70 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82796cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cb3b0 con 0x7f827415dc80 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44588/0 (socket says 172.21.15.107:44588) 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 -- 172.21.15.107:0/3196819359 learned_addr learned my addr 172.21.15.107:0/3196819359 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 -- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f827415dc80 msgr2=0x7f8274149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f827415dc80 0x7f8274149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f82727fc640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8274157c70 0x7f8274149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 -- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8274157c70 msgr2=0x7f8274149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8274157c70 0x7f8274149140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.423+0000 7f8272ffd640 1 -- 172.21.15.107:0/3196819359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c009400 con 0x7f82740cfb60 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f8272ffd640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f826400c870 tx=0x7f826400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264003b70 con 0x7f82740cfb60 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264003cf0 con 0x7f82740cfb60 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400aa90 con 0x7f82740cfb60 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82740cb690 con 0x7f82740cfb60 2023-11-24T08:05:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.424+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8274149bc0 con 0x7f82740cfb60 2023-11-24T08:05:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.425+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82740cbbb0 con 0x7f82740cfb60 2023-11-24T08:05:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.426+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f826400abf0 con 0x7f82740cfb60 2023-11-24T08:05:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.427+0000 7f826b7fe640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 0x7f825406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.427+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82640894f0 con 0x7f82740cfb60 2023-11-24T08:05:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.427+0000 7f82727fc640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 0x7f825406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.428+0000 7f82727fc640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 0x7f825406d300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f825c036000 tx=0x7f825c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.431+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82740cbbb0 con 0x7f82740cfb60 2023-11-24T08:05:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.645+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82740cbbb0 con 0x7f82740cfb60 2023-11-24T08:05:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.646+0000 7f826b7fe640 1 -- 172.21.15.107:0/3196819359 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8264003e60 con 0x7f82740cfb60 2023-11-24T08:05:05.649 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:05.649 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-24T08:05:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.648+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 msgr2=0x7f825406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.648+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 0x7f825406d300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f825c036000 tx=0x7f825c008030 comp rx=0 tx=0).stop 2023-11-24T08:05:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.648+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 msgr2=0x7f827409a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.648+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f826400c870 tx=0x7f826400cd30 comp rx=0 tx=0).stop 2023-11-24T08:05:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 shutdown_connections 2023-11-24T08:05:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f825406ae40 0x7f825406d300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f827415dc80 0x7f8274149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8274157c70 0x7f8274149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 --2- 172.21.15.107:0/3196819359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82740cfb60 0x7f827409a2c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 >> 172.21.15.107:0/3196819359 conn(0x7f827406a920 msgr2=0x7f8274157500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 shutdown_connections 2023-11-24T08:05:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.649+0000 7f82796cf640 1 -- 172.21.15.107:0/3196819359 wait complete. 2023-11-24T08:05:05.668 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-24T08:05:05.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:05.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.867+0000 7f4ea2de5640 1 Processor -- start 2023-11-24T08:05:05.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.867+0000 7f4ea2de5640 1 -- start start 2023-11-24T08:05:05.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.868+0000 7f4ea2de5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e9c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.868+0000 7f4ea2de5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e9c15e390 0x7f4e9c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.868+0000 7f4ea2de5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c0ce0b0 con 0x7f4e9c0cf450 2023-11-24T08:05:05.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.868+0000 7f4ea2de5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c0cfe10 con 0x7f4e9c15e390 2023-11-24T08:05:05.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.868+0000 7f4ea2de5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c161960 con 0x7f4e9c157c70 2023-11-24T08:05:05.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4e9c0cf450 0x7f4e9c15dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59192/0 (socket says 172.21.15.107:59192) 2023-11-24T08:05:05.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9bfff640 1 -- 172.21.15.107:0/494623942 learned_addr learned my addr 172.21.15.107:0/494623942 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:05.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4ea0b81640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e9c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042861091 0 0) 0x7f4e9c0ce0b0 con 0x7f4e9c0cf450 2023-11-24T08:05:05.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4ea1382640 1 --2- 172.21.15.107:0/494623942 >> v2:172.21.15.120:3300/0 conn(0x7f4e9c15e390 0x7f4e9c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c003520 con 0x7f4e9c0cf450 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035531774 0 0) 0x7f4e8c003520 con 0x7f4e9c0cf450 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 msgr2=0x7f4e9c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e9c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 >> v2:172.21.15.120:3300/0 conn(0x7f4e9c15e390 msgr2=0x7f4e9c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 --2- 172.21.15.107:0/494623942 >> v2:172.21.15.120:3300/0 conn(0x7f4e9c15e390 0x7f4e9c0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c15e8d0 con 0x7f4e9c0cf450 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.869+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4e88001670 con 0x7f4e9c0cf450 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 >> v1:172.21.15.107:6789/0 conn(0x7f4e9c0cf450 legacy=0x7f4e9c15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0cfe10 con 0x7f4e8c004d00 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c161960 con 0x7f4e8c003970 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c00b980 con 0x7f4e9c157c70 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4e9bfff640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea0b81640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 -- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 msgr2=0x7f4e8c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c157c70 0x7f4e8c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 -- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c003970 msgr2=0x7f4e8c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.870+0000 7f4ea1382640 1 -- 172.21.15.107:0/494623942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c003520 con 0x7f4e8c004d00 2023-11-24T08:05:05.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.871+0000 7f4ea1382640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009f90 tx=0x7f4e8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90560267bc1a0803 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9402cc20 con 0x7f4e8c004d00 2023-11-24T08:05:05.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400f040 con 0x7f4e8c004d00 2023-11-24T08:05:05.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4e9b7fe640 1 -- 172.21.15.107:0/494623942 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94034440 con 0x7f4e8c004d00 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 -- 172.21.15.107:0/494623942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 msgr2=0x7f4e8c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/494623942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009f90 tx=0x7f4e8c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea1382640 1 -- 172.21.15.107:0/494623942 reap_dead start 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 -- 172.21.15.107:0/494623942 shutdown_connections 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 -- 172.21.15.107:0/494623942 >> 172.21.15.107:0/494623942 conn(0x7f4e9c06a920 msgr2=0x7f4e9c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 -- 172.21.15.107:0/494623942 shutdown_connections 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.872+0000 7f4ea2de5640 1 -- 172.21.15.107:0/494623942 wait complete. 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.874+0000 7f4ea2de5640 1 Processor -- start 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.874+0000 7f4ea2de5640 1 -- start start 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c1f6570 0x7f4e9c0cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e9c1f3720 0x7f4e9c155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1f2f80 con 0x7f4e8c004d00 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1f69f0 con 0x7f4e9c1f3720 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea2de5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1f6cc0 con 0x7f4e9c1f6570 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea0b81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea0b81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44606/0 (socket says 172.21.15.107:44606) 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.875+0000 7f4ea0b81640 1 -- 172.21.15.107:0/558001138 learned_addr learned my addr 172.21.15.107:0/558001138 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4e9bfff640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c1f6570 0x7f4e9c0cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea1382640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e9c1f3720 0x7f4e9c155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 -- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c1f6570 msgr2=0x7f4e9c0cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c1f6570 0x7f4e9c0cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 -- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e9c1f3720 msgr2=0x7f4e9c155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e9c1f3720 0x7f4e9c155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 -- 172.21.15.107:0/558001138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c1f6fa0 con 0x7f4e8c004d00 2023-11-24T08:05:05.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea0b81640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4e90010860 tx=0x7f4e90010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90009610 con 0x7f4e8c004d00 2023-11-24T08:05:05.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9000eb60 con 0x7f4e8c004d00 2023-11-24T08:05:05.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9001c400 con 0x7f4e8c004d00 2023-11-24T08:05:05.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c1f7280 con 0x7f4e8c004d00 2023-11-24T08:05:05.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.876+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c0cc030 con 0x7f4e8c004d00 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.878+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c1f7470 con 0x7f4e8c004d00 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.879+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4e9001c560 con 0x7f4e8c004d00 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.879+0000 7f4e997fa640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 0x7f4e8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.880+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e90089b90 con 0x7f4e8c004d00 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.880+0000 7f4e9bfff640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 0x7f4e8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.880+0000 7f4e9bfff640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 0x7f4e8006d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c072a70 tx=0x7f4e88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:05.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:05.884+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e9c1f7470 con 0x7f4e8c004d00 2023-11-24T08:05:06.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.058+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e9c1f7470 con 0x7f4e8c004d00 2023-11-24T08:05:06.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.061+0000 7f4e997fa640 1 -- 172.21.15.107:0/558001138 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f4e9c1f7470 con 0x7f4e8c004d00 2023-11-24T08:05:06.064 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:06.064 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 msgr2=0x7f4e8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 0x7f4e8006d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c072a70 tx=0x7f4e88009040 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 msgr2=0x7f4e9c071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4e90010860 tx=0x7f4e90010d20 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 shutdown_connections 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8006ae40 0x7f4e8006d300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e9c1f3720 0x7f4e9c155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9c1f6570 0x7f4e9c0cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 --2- 172.21.15.107:0/558001138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c004d00 0x7f4e9c071a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.063+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 >> 172.21.15.107:0/558001138 conn(0x7f4e9c06a920 msgr2=0x7f4e9c0afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.064+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 shutdown_connections 2023-11-24T08:05:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.064+0000 7f4ea2de5640 1 -- 172.21.15.107:0/558001138 wait complete. 2023-11-24T08:05:06.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:06.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.283+0000 7f29a7fff640 1 Processor -- start 2023-11-24T08:05:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.283+0000 7f29a7fff640 1 -- start start 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f29a80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29a815ded0 0x7f29a815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a7fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a806d430 con 0x7f29a815a1c0 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a7fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a815ec50 con 0x7f29a815ded0 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a806cb00 con 0x7f29a80b0b40 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f29a815a1c0 0x7f29a806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59196/0 (socket says 172.21.15.107:59196) 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a6ffd640 1 -- 172.21.15.107:0/1395866009 learned_addr learned my addr 172.21.15.107:0/1395866009 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f29a77fe640 1 --2- 172.21.15.107:0/1395866009 >> v2:172.21.15.120:3300/0 conn(0x7f29a815ded0 0x7f29a815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.285+0000 7f299e7fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f29a80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272969304 0 0) 0x7f29a806d430 con 0x7f29a815a1c0 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988003520 con 0x7f29a815a1c0 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642178668 0 0) 0x7f2988003520 con 0x7f29a815a1c0 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 msgr2=0x7f29a80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f29a80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 >> v2:172.21.15.120:3300/0 conn(0x7f29a815ded0 msgr2=0x7f29a815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 --2- 172.21.15.107:0/1395866009 >> v2:172.21.15.120:3300/0 conn(0x7f29a815ded0 0x7f29a815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a80b14e0 con 0x7f29a815a1c0 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2998001670 con 0x7f29a815a1c0 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 >> v1:172.21.15.107:6789/0 conn(0x7f29a815a1c0 legacy=0x7f29a806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f2988003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2988003f40 0x7f29880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988004d00 0x7f2988005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a815ec50 con 0x7f2988003f40 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a806cb00 con 0x7f2988004d00 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298800b980 con 0x7f29a80b0b40 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.286+0000 7f299e7fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2988003f40 0x7f29880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f29a6ffd640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f2988003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f29a77fe640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988004d00 0x7f2988005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f299e7fc640 1 -- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 msgr2=0x7f2988003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f299e7fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a80b0b40 0x7f2988003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f299e7fc640 1 -- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988004d00 msgr2=0x7f2988005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f299e7fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988004d00 0x7f2988005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.287+0000 7f299e7fc640 1 -- 172.21.15.107:0/1395866009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988003520 con 0x7f2988003f40 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.288+0000 7f299e7fc640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2988003f40 0x7f29880047c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2990009af0 tx=0x7f29880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e401450a927c54 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.288+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299002c6b0 con 0x7f2988003f40 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.288+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299002c810 con 0x7f2988003f40 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.288+0000 7f29a67fc640 1 -- 172.21.15.107:0/1395866009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299002cba0 con 0x7f2988003f40 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 -- 172.21.15.107:0/1395866009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2988003f40 msgr2=0x7f29880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 --2- 172.21.15.107:0/1395866009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2988003f40 0x7f29880047c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2990009af0 tx=0x7f29880057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a77fe640 1 -- 172.21.15.107:0/1395866009 reap_dead start 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 -- 172.21.15.107:0/1395866009 shutdown_connections 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 -- 172.21.15.107:0/1395866009 >> 172.21.15.107:0/1395866009 conn(0x7f29a80cbb40 msgr2=0x7f29a80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 -- 172.21.15.107:0/1395866009 shutdown_connections 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.289+0000 7f29a7fff640 1 -- 172.21.15.107:0/1395866009 wait complete. 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.290+0000 7f29a7fff640 1 Processor -- start 2023-11-24T08:05:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.291+0000 7f29a7fff640 1 -- start start 2023-11-24T08:05:06.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.291+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988003f40 0x7f29a81f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a815c870 0x7f29a81f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f29a7fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80c8eb0 con 0x7f29a815c3f0 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f29a7fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80c6190 con 0x7f2988003f40 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f29a7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80c6470 con 0x7f29a815c870 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44614/0 (socket says 172.21.15.107:44614) 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 -- 172.21.15.107:0/4065327511 learned_addr learned my addr 172.21.15.107:0/4065327511 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 -- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a815c870 msgr2=0x7f29a81f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a815c870 0x7f29a81f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 -- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988003f40 msgr2=0x7f29a81f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988003f40 0x7f29a81f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 -- 172.21.15.107:0/4065327511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a815ea20 con 0x7f29a815c3f0 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.292+0000 7f299e7fc640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2990037040 tx=0x7f2990004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.293+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990003810 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.293+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2990003970 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.293+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990003ce0 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.293+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a80c6750 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.293+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a81f39b0 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.294+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a804e840 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.295+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2990034440 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.295+0000 7f299ffff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 0x7f298c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.296+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29900ab480 con 0x7f29a815c3f0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.296+0000 7f29a6ffd640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 0x7f298c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.296+0000 7f29a6ffd640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 0x7f298c06d300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2998001050 tx=0x7f2998000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.300+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2990076700 con 0x7f29a815c3f0 2023-11-24T08:05:06.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.490+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29a80c8eb0 con 0x7f29a815c3f0 2023-11-24T08:05:06.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.491+0000 7f299ffff640 1 -- 172.21.15.107:0/4065327511 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29900044e0 con 0x7f29a815c3f0 2023-11-24T08:05:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 msgr2=0x7f298c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 0x7f298c06d300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2998001050 tx=0x7f2998000ed0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 msgr2=0x7f29a80c8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2990037040 tx=0x7f2990004730 comp rx=0 tx=0).stop 2023-11-24T08:05:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 shutdown_connections 2023-11-24T08:05:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f298c06ae40 0x7f298c06d300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29a815c870 0x7f29a81f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29a815c3f0 0x7f29a80c8460 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 --2- 172.21.15.107:0/4065327511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988003f40 0x7f29a81f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 >> 172.21.15.107:0/4065327511 conn(0x7f29a80cbb40 msgr2=0x7f29a809a420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 shutdown_connections 2023-11-24T08:05:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.494+0000 7f29a7fff640 1 -- 172.21.15.107:0/4065327511 wait complete. 2023-11-24T08:05:06.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:06.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.718+0000 7fafd607e640 1 Processor -- start 2023-11-24T08:05:06.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.718+0000 7fafd607e640 1 -- start start 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd607e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fafd015d7d0 0x7fafd00cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd607e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00d15e0 con 0x7fafd00d1f40 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd607e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00d0280 con 0x7fafd015d3f0 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd607e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd00b0eb0 con 0x7fafd015d7d0 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd587d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fafd00d1f40 0x7fafd00ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59208/0 (socket says 172.21.15.107:59208) 2023-11-24T08:05:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd587d640 1 -- 172.21.15.107:0/3941774545 learned_addr learned my addr 172.21.15.107:0/3941774545 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafd487b640 1 --2- 172.21.15.107:0/3941774545 >> v2:172.21.15.120:3300/0 conn(0x7fafd015d7d0 0x7fafd00cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.720+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741574029 0 0) 0x7fafd00d15e0 con 0x7fafd00d1f40 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc003520 con 0x7fafd00d1f40 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199527713 0 0) 0x7fafbc003520 con 0x7fafd00d1f40 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 >> v2:172.21.15.120:3300/0 conn(0x7fafd015d7d0 msgr2=0x7fafd00cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 --2- 172.21.15.107:0/3941774545 >> v2:172.21.15.120:3300/0 conn(0x7fafd015d7d0 0x7fafd00cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 >> v1:172.21.15.120:6789/0 conn(0x7fafd015d3f0 legacy=0x7fafd00d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00d1300 con 0x7fafd00d1f40 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fafc8001670 con 0x7fafd00d1f40 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 >> v1:172.21.15.107:6789/0 conn(0x7fafd00d1f40 legacy=0x7fafd00ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafbc0039c0 0x7fafbc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafbc005180 0x7fafbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00b0eb0 con 0x7fafbc0047c0 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc00c080 con 0x7fafbc0039c0 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc00c240 con 0x7fafbc005180 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.721+0000 7fafd487b640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd487b640 1 -- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafbc005180 msgr2=0x7fafbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd487b640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafbc005180 0x7fafbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd507c640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafbc0039c0 0x7fafbc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd487b640 1 -- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafbc0039c0 msgr2=0x7fafbc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd487b640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafbc0039c0 0x7fafbc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd487b640 1 -- 172.21.15.107:0/3941774545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc003520 con 0x7fafbc0047c0 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd587d640 1 -- 172.21.15.107:0/3941774545 reap_dead start 2023-11-24T08:05:06.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.722+0000 7fafd507c640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafbc0039c0 0x7fafbc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.723+0000 7fafd487b640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafbc004c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009a90 tx=0x7fafbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae844b4e24e41a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.723+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc002c6b0 con 0x7fafbc0047c0 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.723+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc002c810 con 0x7fafbc0047c0 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.723+0000 7fafc7fff640 1 -- 172.21.15.107:0/3941774545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc002cba0 con 0x7fafbc0047c0 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 -- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 msgr2=0x7fafbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafbc004c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009a90 tx=0x7fafbc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 -- 172.21.15.107:0/3941774545 shutdown_connections 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 --2- 172.21.15.107:0/3941774545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafbc004c40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 -- 172.21.15.107:0/3941774545 >> 172.21.15.107:0/3941774545 conn(0x7fafd009c5b0 msgr2=0x7fafd00d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 -- 172.21.15.107:0/3941774545 shutdown_connections 2023-11-24T08:05:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.724+0000 7fafd607e640 1 -- 172.21.15.107:0/3941774545 wait complete. 2023-11-24T08:05:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.726+0000 7fafd607e640 1 Processor -- start 2023-11-24T08:05:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.726+0000 7fafd607e640 1 -- start start 2023-11-24T08:05:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 0x7fafd014d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafd00c8390 0x7fafd00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c8d50 con 0x7fafbc0047c0 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01f6080 con 0x7fafd00cce00 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd607e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd014d930 con 0x7fafd00c8390 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44634/0 (socket says 172.21.15.107:44634) 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 -- 172.21.15.107:0/3925792390 learned_addr learned my addr 172.21.15.107:0/3925792390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd487b640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 0x7fafd014d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd587d640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafd00c8390 0x7fafd00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 -- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafd00c8390 msgr2=0x7fafd00c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafd00c8390 0x7fafd00c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 -- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 msgr2=0x7fafd014d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 0x7fafd014d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.727+0000 7fafd507c640 1 -- 172.21.15.107:0/3925792390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd01473d0 con 0x7fafbc0047c0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafd487b640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 0x7fafd014d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafd507c640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00d830 tx=0x7fafcc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc003da0 con 0x7fafbc0047c0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc00b6b0 con 0x7fafbc0047c0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc00ba60 con 0x7fafbc0047c0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd00cd5c0 con 0x7fafbc0047c0 2023-11-24T08:05:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.728+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0147970 con 0x7fafbc0047c0 2023-11-24T08:05:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.729+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd00b2370 con 0x7fafbc0047c0 2023-11-24T08:05:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.730+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fafcc00bbc0 con 0x7fafbc0047c0 2023-11-24T08:05:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.731+0000 7fafc5ffb640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 0x7fafb406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.731+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafcc089820 con 0x7fafbc0047c0 2023-11-24T08:05:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.731+0000 7fafd487b640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 0x7fafb406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:06.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.732+0000 7fafd487b640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 0x7fafb406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fafc0036000 tx=0x7fafc0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.736+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafcc054990 con 0x7fafbc0047c0 2023-11-24T08:05:06.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.927+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fafd004e840 con 0x7fafbc0047c0 2023-11-24T08:05:06.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.927+0000 7fafc5ffb640 1 -- 172.21.15.107:0/3925792390 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fafcc089070 con 0x7fafbc0047c0 2023-11-24T08:05:06.930 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 msgr2=0x7fafb406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 0x7fafb406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fafc0036000 tx=0x7fafc0030000 comp rx=0 tx=0).stop 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 msgr2=0x7fafd00ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00d830 tx=0x7fafcc00dcf0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 shutdown_connections 2023-11-24T08:05:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafb406ae70 0x7fafb406d330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafd00c8390 0x7fafd00c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafd00cce00 0x7fafd014d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 --2- 172.21.15.107:0/3925792390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafbc0047c0 0x7fafd00ca9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 >> 172.21.15.107:0/3925792390 conn(0x7fafd009c5b0 msgr2=0x7fafd006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 shutdown_connections 2023-11-24T08:05:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:06.930+0000 7fafd607e640 1 -- 172.21.15.107:0/3925792390 wait complete. 2023-11-24T08:05:06.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.158+0000 7faf39f5c640 1 Processor -- start 2023-11-24T08:05:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.158+0000 7faf39f5c640 1 -- start start 2023-11-24T08:05:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.159+0000 7faf39f5c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 0x7faf340b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.159+0000 7faf39f5c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faf3415ded0 0x7faf3415e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.159+0000 7faf39f5c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3406d430 con 0x7faf3415a1c0 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.159+0000 7faf39f5c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3415ec50 con 0x7faf3415ded0 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.159+0000 7faf39f5c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3406cb00 con 0x7faf340b0b40 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf38f5a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faf3415a1c0 0x7faf3406d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59218/0 (socket says 172.21.15.107:59218) 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf38f5a640 1 -- 172.21.15.107:0/1010451187 learned_addr learned my addr 172.21.15.107:0/1010451187 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf3975b640 1 --2- 172.21.15.107:0/1010451187 >> v2:172.21.15.107:3300/0 conn(0x7faf3415ded0 0x7faf3415e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf2b7fe640 1 -- 172.21.15.107:0/1010451187 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693709232 0 0) 0x7faf3406d430 con 0x7faf3415a1c0 2023-11-24T08:05:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf2bfff640 1 --2- 172.21.15.107:0/1010451187 >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 0x7faf340b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf2b7fe640 1 -- 172.21.15.107:0/1010451187 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c005730 con 0x7faf3415a1c0 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf3975b640 1 -- 172.21.15.107:0/1010451187 >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 msgr2=0x7faf340b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf3975b640 1 --2- 172.21.15.107:0/1010451187 >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 0x7faf340b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf3975b640 1 -- 172.21.15.107:0/1010451187 >> v1:172.21.15.107:6789/0 conn(0x7faf3415a1c0 legacy=0x7faf3406d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf3975b640 1 -- 172.21.15.107:0/1010451187 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf340b14e0 con 0x7faf3415ded0 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.160+0000 7faf2bfff640 1 --2- 172.21.15.107:0/1010451187 >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 0x7faf340b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.161+0000 7faf3975b640 1 --2- 172.21.15.107:0/1010451187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf3415e360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf340cee20 tx=0x7faf2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8974f06692d449ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.161+0000 7faf2b7fe640 1 -- 172.21.15.107:0/1010451187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c02cc80 con 0x7faf3415ded0 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf2b7fe640 1 -- 172.21.15.107:0/1010451187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c02cde0 con 0x7faf3415ded0 2023-11-24T08:05:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf2b7fe640 1 -- 172.21.15.107:0/1010451187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c034660 con 0x7faf3415ded0 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 -- 172.21.15.107:0/1010451187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 msgr2=0x7faf3415e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 --2- 172.21.15.107:0/1010451187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf3415e360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf340cee20 tx=0x7faf2c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 -- 172.21.15.107:0/1010451187 shutdown_connections 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 --2- 172.21.15.107:0/1010451187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf3415e360 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 --2- 172.21.15.107:0/1010451187 >> v2:172.21.15.120:3300/0 conn(0x7faf340b0b40 0x7faf340b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 -- 172.21.15.107:0/1010451187 >> 172.21.15.107:0/1010451187 conn(0x7faf340cbb40 msgr2=0x7faf340cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 -- 172.21.15.107:0/1010451187 shutdown_connections 2023-11-24T08:05:07.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.162+0000 7faf39f5c640 1 -- 172.21.15.107:0/1010451187 wait complete. 2023-11-24T08:05:07.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.164+0000 7faf39f5c640 1 Processor -- start 2023-11-24T08:05:07.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.164+0000 7faf39f5c640 1 -- start start 2023-11-24T08:05:07.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf340b0b40 0x7faf3414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 0x7faf340c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340c7da0 con 0x7faf3415ded0 2023-11-24T08:05:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340c7910 con 0x7faf3415a1c0 2023-11-24T08:05:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf39f5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340c6d00 con 0x7faf340b0b40 2023-11-24T08:05:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.165+0000 7faf3975b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44654/0 (socket says 172.21.15.107:44654) 2023-11-24T08:05:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 -- 172.21.15.107:0/2441719538 learned_addr learned my addr 172.21.15.107:0/2441719538 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf38f5a640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf340b0b40 0x7faf3414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 -- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf340b0b40 msgr2=0x7faf3414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf340b0b40 0x7faf3414c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf2bfff640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 0x7faf340c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 -- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 msgr2=0x7faf340c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 0x7faf340c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 -- 172.21.15.107:0/2441719538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c009a70 con 0x7faf3415ded0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf2bfff640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 0x7faf340c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf3975b640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009e80 tx=0x7faf2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.166+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c040650 con 0x7faf3415ded0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.167+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c0407b0 con 0x7faf3415ded0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.167+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c040b60 con 0x7faf3415ded0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.167+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3415ea20 con 0x7faf3415ded0 2023-11-24T08:05:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.167+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf340c72c0 con 0x7faf3415ded0 2023-11-24T08:05:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.168+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf340c7aa0 con 0x7faf3415ded0 2023-11-24T08:05:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.169+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faf2c040cc0 con 0x7faf3415ded0 2023-11-24T08:05:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.170+0000 7faf297fa640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 0x7faf2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.170+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf2c0ac680 con 0x7faf3415ded0 2023-11-24T08:05:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.170+0000 7faf38f5a640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 0x7faf2006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.171+0000 7faf38f5a640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 0x7faf2006d300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf300011e0 tx=0x7faf30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.174+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf2c0777b0 con 0x7faf3415ded0 2023-11-24T08:05:07.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.364+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf3404e840 con 0x7faf3415ded0 2023-11-24T08:05:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.365+0000 7faf297fa640 1 -- 172.21.15.107:0/2441719538 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf2c009450 con 0x7faf3415ded0 2023-11-24T08:05:07.374 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 msgr2=0x7faf2006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 0x7faf2006d300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf300011e0 tx=0x7faf30009040 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 msgr2=0x7faf340c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009e80 tx=0x7faf2c004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 shutdown_connections 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006ae40 0x7faf2006d300 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf3415ded0 0x7faf340c66d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf3415a1c0 0x7faf340c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 --2- 172.21.15.107:0/2441719538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf340b0b40 0x7faf3414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 >> 172.21.15.107:0/2441719538 conn(0x7faf340cbb40 msgr2=0x7faf340cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 shutdown_connections 2023-11-24T08:05:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.368+0000 7faf39f5c640 1 -- 172.21.15.107:0/2441719538 wait complete. 2023-11-24T08:05:07.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.604+0000 7f471891d640 1 Processor -- start 2023-11-24T08:05:07.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.605+0000 7f471891d640 1 -- start start 2023-11-24T08:05:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f471891d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47141609e0 0x7f471406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f471891d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47140ce010 0x7f47140ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f471891d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471406c1f0 con 0x7f47140d31a0 2023-11-24T08:05:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f471891d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47140d1070 con 0x7f47140ce010 2023-11-24T08:05:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f471891d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47140cedc0 con 0x7f47141609e0 2023-11-24T08:05:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f470bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47140d31a0 0x7f47140d3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59222/0 (socket says 172.21.15.107:59222) 2023-11-24T08:05:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.606+0000 7f470bfff640 1 -- 172.21.15.107:0/3486743805 learned_addr learned my addr 172.21.15.107:0/3486743805 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4712d9d640 1 --2- 172.21.15.107:0/3486743805 >> v2:172.21.15.107:3300/0 conn(0x7f47140ce010 0x7f47140ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f471259c640 1 --2- 172.21.15.107:0/3486743805 >> v2:172.21.15.120:3300/0 conn(0x7f47141609e0 0x7f471406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4711d9b640 1 -- 172.21.15.107:0/3486743805 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590005708 0 0) 0x7f471406c1f0 con 0x7f47140d31a0 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4711d9b640 1 -- 172.21.15.107:0/3486743805 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4700003520 con 0x7f47140d31a0 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4712d9d640 1 -- 172.21.15.107:0/3486743805 >> v2:172.21.15.120:3300/0 conn(0x7f47141609e0 msgr2=0x7f471406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4712d9d640 1 --2- 172.21.15.107:0/3486743805 >> v2:172.21.15.120:3300/0 conn(0x7f47141609e0 0x7f471406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4712d9d640 1 -- 172.21.15.107:0/3486743805 >> v1:172.21.15.107:6789/0 conn(0x7f47140d31a0 legacy=0x7f47140d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.607+0000 7f4712d9d640 1 -- 172.21.15.107:0/3486743805 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47140cf0a0 con 0x7f47140ce010 2023-11-24T08:05:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.608+0000 7f4712d9d640 1 --2- 172.21.15.107:0/3486743805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140ce010 0x7f47140ce460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f471408b620 tx=0x7f470c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e48ec3b11f82303 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.608+0000 7f4711d9b640 1 -- 172.21.15.107:0/3486743805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c02cc80 con 0x7f47140ce010 2023-11-24T08:05:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f4711d9b640 1 -- 172.21.15.107:0/3486743805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c02cde0 con 0x7f47140ce010 2023-11-24T08:05:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f4711d9b640 1 -- 172.21.15.107:0/3486743805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c034660 con 0x7f47140ce010 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 -- 172.21.15.107:0/3486743805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140ce010 msgr2=0x7f47140ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 --2- 172.21.15.107:0/3486743805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140ce010 0x7f47140ce460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f471408b620 tx=0x7f470c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 -- 172.21.15.107:0/3486743805 shutdown_connections 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 --2- 172.21.15.107:0/3486743805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140ce010 0x7f47140ce460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 --2- 172.21.15.107:0/3486743805 >> v2:172.21.15.120:3300/0 conn(0x7f47141609e0 0x7f471406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 -- 172.21.15.107:0/3486743805 >> 172.21.15.107:0/3486743805 conn(0x7f471408af60 msgr2=0x7f471409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 -- 172.21.15.107:0/3486743805 shutdown_connections 2023-11-24T08:05:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.609+0000 7f471891d640 1 -- 172.21.15.107:0/3486743805 wait complete. 2023-11-24T08:05:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.611+0000 7f471891d640 1 Processor -- start 2023-11-24T08:05:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.611+0000 7f471891d640 1 -- start start 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47140ce010 0x7f47140b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47141609e0 0x7f47140b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47141f7930 con 0x7f47140d31a0 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471415c7f0 con 0x7f47140ce010 2023-11-24T08:05:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f471891d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47140700f0 con 0x7f47141609e0 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f470bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f470bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44678/0 (socket says 172.21.15.107:44678) 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.612+0000 7f470bfff640 1 -- 172.21.15.107:0/260483850 learned_addr learned my addr 172.21.15.107:0/260483850 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 -- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47141609e0 msgr2=0x7f47140b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47141609e0 0x7f47140b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 -- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47140ce010 msgr2=0x7f47140b74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47140ce010 0x7f47140b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 -- 172.21.15.107:0/260483850 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c009a70 con 0x7f47140d31a0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470bfff640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00c4d0 tx=0x7f46fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc005ce0 con 0x7f47140d31a0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc005e40 con 0x7f47140d31a0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc007270 con 0x7f47140d31a0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47140703d0 con 0x7f47140d31a0 2023-11-24T08:05:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.613+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47140b6cc0 con 0x7f47140d31a0 2023-11-24T08:05:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.615+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4714067b80 con 0x7f47140d31a0 2023-11-24T08:05:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.615+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f46fc007490 con 0x7f47140d31a0 2023-11-24T08:05:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.616+0000 7f470affd640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 0x7f46f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.616+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46fc088c20 con 0x7f47140d31a0 2023-11-24T08:05:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.617+0000 7f471259c640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 0x7f46f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.617+0000 7f471259c640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 0x7f46f406d300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4704006a90 tx=0x7f4704006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.621+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46fc053c50 con 0x7f47140d31a0 2023-11-24T08:05:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.856+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4714067470 con 0x7f47140d31a0 2023-11-24T08:05:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.858+0000 7f470affd640 1 -- 172.21.15.107:0/260483850 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f46fc017070 con 0x7f47140d31a0 2023-11-24T08:05:07.860 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:07.860 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 msgr2=0x7f46f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 0x7f46f406d300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4704006a90 tx=0x7f4704006a20 comp rx=0 tx=0).stop 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 msgr2=0x7f471415c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00c4d0 tx=0x7f46fc00c990 comp rx=0 tx=0).stop 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 shutdown_connections 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46f406ae40 0x7f46f406d300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47141609e0 0x7f47140b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47140d31a0 0x7f471415c2b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 --2- 172.21.15.107:0/260483850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47140ce010 0x7f47140b74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.860+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 >> 172.21.15.107:0/260483850 conn(0x7f471408af60 msgr2=0x7f47140cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.861+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 shutdown_connections 2023-11-24T08:05:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:07.861+0000 7f471891d640 1 -- 172.21.15.107:0/260483850 wait complete. 2023-11-24T08:05:07.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.089+0000 7f162cce7640 1 Processor -- start 2023-11-24T08:05:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.089+0000 7f162cce7640 1 -- start start 2023-11-24T08:05:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.090+0000 7f162cce7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f16280ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.090+0000 7f162cce7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f16280cd470 0x7f162815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.090+0000 7f162cce7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162815ab80 con 0x7f16280b2250 2023-11-24T08:05:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.090+0000 7f162cce7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162815a990 con 0x7f16280cd470 2023-11-24T08:05:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.090+0000 7f162cce7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162815e3c0 con 0x7f162815df80 2023-11-24T08:05:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.091+0000 7f1626d9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f16280cd470 0x7f162815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.091+0000 7f1626d9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f16280cd470 0x7f162815b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:55274/0 (socket says 172.21.15.107:55274) 2023-11-24T08:05:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.091+0000 7f1626d9d640 1 -- 172.21.15.107:0/3245248771 learned_addr learned my addr 172.21.15.107:0/3245248771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.091+0000 7f161ffff640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f16280ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.092+0000 7f1626d9d640 1 -- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 msgr2=0x7f16280ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.092+0000 7f1626d9d640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f16280ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.092+0000 7f1626d9d640 1 -- 172.21.15.107:0/3245248771 >> v1:172.21.15.120:6789/0 conn(0x7f16280b2250 legacy=0x7f16280b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.092+0000 7f1626d9d640 1 -- 172.21.15.107:0/3245248771 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16280b2d40 con 0x7f16280cd470 2023-11-24T08:05:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.093+0000 7f1626d9d640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280cd470 0x7f162815b080 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1620009e50 tx=0x7f162002e170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e70d822d33886725 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f1625d9b640 1 -- 172.21.15.107:0/3245248771 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162002ed10 con 0x7f16280cd470 2023-11-24T08:05:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f1625d9b640 1 -- 172.21.15.107:0/3245248771 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162002ee70 con 0x7f16280cd470 2023-11-24T08:05:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f1625d9b640 1 -- 172.21.15.107:0/3245248771 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16200365f0 con 0x7f16280cd470 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 -- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280cd470 msgr2=0x7f162815b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280cd470 0x7f162815b080 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1620009e50 tx=0x7f162002e170 comp rx=0 tx=0).stop 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 -- 172.21.15.107:0/3245248771 shutdown_connections 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280cd470 0x7f162815b080 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 --2- 172.21.15.107:0/3245248771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f16280ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 -- 172.21.15.107:0/3245248771 >> 172.21.15.107:0/3245248771 conn(0x7f162806a920 msgr2=0x7f162806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 -- 172.21.15.107:0/3245248771 shutdown_connections 2023-11-24T08:05:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.094+0000 7f162cce7640 1 -- 172.21.15.107:0/3245248771 wait complete. 2023-11-24T08:05:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.096+0000 7f162cce7640 1 Processor -- start 2023-11-24T08:05:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.097+0000 7f162cce7640 1 -- start start 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.097+0000 7f162cce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 0x7f16281494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162cce7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162cce7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f162809cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162cce7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1628149a00 con 0x7f16280cd470 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162cce7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162809a3f0 con 0x7f16280b2250 2023-11-24T08:05:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162cce7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16280aeec0 con 0x7f162815df80 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52246/0 (socket says 172.21.15.107:52246) 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 -- 172.21.15.107:0/4038151394 learned_addr learned my addr 172.21.15.107:0/4038151394 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f1626d9d640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f162809cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162659c640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 0x7f16281494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 -- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 msgr2=0x7f162809cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f162809cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 -- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 msgr2=0x7f16281494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 0x7f16281494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f161ffff640 1 -- 172.21.15.107:0/4038151394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620009b30 con 0x7f16280cd470 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f162659c640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 0x7f16281494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.098+0000 7f1626d9d640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f162809cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f161ffff640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f161000e870 tx=0x7f161000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16100094f0 con 0x7f16280cd470 2023-11-24T08:05:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161000c940 con 0x7f16280cd470 2023-11-24T08:05:08.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161000ccf0 con 0x7f16280cd470 2023-11-24T08:05:08.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16280af1a0 con 0x7f16280cd470 2023-11-24T08:05:08.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.099+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16280af760 con 0x7f16280cd470 2023-11-24T08:05:08.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.100+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16280ae3e0 con 0x7f16280cd470 2023-11-24T08:05:08.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.101+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1610013470 con 0x7f16280cd470 2023-11-24T08:05:08.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.102+0000 7f161effd640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 0x7f160806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.103+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1610089760 con 0x7f16280cd470 2023-11-24T08:05:08.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.103+0000 7f162659c640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 0x7f160806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.103+0000 7f162659c640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 0x7f160806d300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1618000f60 tx=0x7f1618009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.107+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1610054770 con 0x7f16280cd470 2023-11-24T08:05:08.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.296+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1628067470 con 0x7f16280cd470 2023-11-24T08:05:08.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.298+0000 7f161effd640 1 -- 172.21.15.107:0/4038151394 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1610057d90 con 0x7f16280cd470 2023-11-24T08:05:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:08.300 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 msgr2=0x7f160806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 0x7f160806d300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1618000f60 tx=0x7f1618009040 comp rx=0 tx=0).stop 2023-11-24T08:05:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 msgr2=0x7f16280aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f161000e870 tx=0x7f161000ed30 comp rx=0 tx=0).stop 2023-11-24T08:05:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 shutdown_connections 2023-11-24T08:05:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806ae40 0x7f160806d300 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f162815df80 0x7f162809cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16280cd470 0x7f16280aff60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 --2- 172.21.15.107:0/4038151394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280b2250 0x7f16281494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.300+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 >> 172.21.15.107:0/4038151394 conn(0x7f162806a920 msgr2=0x7f1628159b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:08.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.301+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 shutdown_connections 2023-11-24T08:05:08.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.301+0000 7f162cce7640 1 -- 172.21.15.107:0/4038151394 wait complete. 2023-11-24T08:05:08.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:08.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:08.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.526+0000 7fa6bf733640 1 Processor -- start 2023-11-24T08:05:08.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.526+0000 7fa6bf733640 1 -- start start 2023-11-24T08:05:08.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bf733640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa6b815b440 0x7fa6b80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bf733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6b80c59b0 0x7fa6b806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bf733640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b80c5dc0 con 0x7fa6b80cbf20 2023-11-24T08:05:08.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bf733640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b80cc3d0 con 0x7fa6b80c59b0 2023-11-24T08:05:08.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bf733640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b815b820 con 0x7fa6b815b440 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.528+0000 7fa6bdcd0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6b80c59b0 0x7fa6b806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6b80c59b0 0x7fa6b806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52252/0 (socket says 172.21.15.107:52252) 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 -- 172.21.15.107:0/1003541042 learned_addr learned my addr 172.21.15.107:0/1003541042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bd4cf640 1 --2- 172.21.15.107:0/1003541042 >> v2:172.21.15.120:3300/0 conn(0x7fa6b815b440 0x7fa6b80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 -- 172.21.15.107:0/1003541042 >> v2:172.21.15.120:3300/0 conn(0x7fa6b815b440 msgr2=0x7fa6b80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 --2- 172.21.15.107:0/1003541042 >> v2:172.21.15.120:3300/0 conn(0x7fa6b815b440 0x7fa6b80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 -- 172.21.15.107:0/1003541042 >> v1:172.21.15.120:6789/0 conn(0x7fa6b80cbf20 legacy=0x7fa6b80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.529+0000 7fa6bdcd0640 1 -- 172.21.15.107:0/1003541042 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b815eea0 con 0x7fa6b80c59b0 2023-11-24T08:05:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.530+0000 7fa6bdcd0640 1 --2- 172.21.15.107:0/1003541042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80c59b0 0x7fa6b806c5f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa6b806e770 tx=0x7fa6b002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd937289e07704a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.530+0000 7fa6affff640 1 -- 172.21.15.107:0/1003541042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b002c9f0 con 0x7fa6b80c59b0 2023-11-24T08:05:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.530+0000 7fa6affff640 1 -- 172.21.15.107:0/1003541042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b002cb50 con 0x7fa6b80c59b0 2023-11-24T08:05:08.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.530+0000 7fa6affff640 1 -- 172.21.15.107:0/1003541042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b00343f0 con 0x7fa6b80c59b0 2023-11-24T08:05:08.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 -- 172.21.15.107:0/1003541042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80c59b0 msgr2=0x7fa6b806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 --2- 172.21.15.107:0/1003541042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80c59b0 0x7fa6b806c5f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa6b806e770 tx=0x7fa6b002be50 comp rx=0 tx=0).stop 2023-11-24T08:05:08.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 -- 172.21.15.107:0/1003541042 shutdown_connections 2023-11-24T08:05:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 --2- 172.21.15.107:0/1003541042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80c59b0 0x7fa6b806c5f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 --2- 172.21.15.107:0/1003541042 >> v2:172.21.15.120:3300/0 conn(0x7fa6b815b440 0x7fa6b80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 -- 172.21.15.107:0/1003541042 >> 172.21.15.107:0/1003541042 conn(0x7fa6b80b0c50 msgr2=0x7fa6b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 -- 172.21.15.107:0/1003541042 shutdown_connections 2023-11-24T08:05:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.531+0000 7fa6bf733640 1 -- 172.21.15.107:0/1003541042 wait complete. 2023-11-24T08:05:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.533+0000 7fa6bf733640 1 Processor -- start 2023-11-24T08:05:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.533+0000 7fa6bf733640 1 -- start start 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b80c59b0 0x7fa6b814ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815b440 0x7fa6b8153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b814f260 con 0x7fa6b80cbf20 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81542b0 con 0x7fa6b80c59b0 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bf733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b8154590 con 0x7fa6b815b440 2023-11-24T08:05:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bccce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bccce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52264/0 (socket says 172.21.15.107:52264) 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.534+0000 7fa6bccce640 1 -- 172.21.15.107:0/3432371998 learned_addr learned my addr 172.21.15.107:0/3432371998 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bd4cf640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b80c59b0 0x7fa6b814ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 -- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815b440 msgr2=0x7fa6b8153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815b440 0x7fa6b8153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 -- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b80c59b0 msgr2=0x7fa6b814ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b80c59b0 0x7fa6b814ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 -- 172.21.15.107:0/3432371998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0009b30 con 0x7fa6b80cbf20 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bccce640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800c430 tx=0x7fa6a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8011790 con 0x7fa6b80cbf20 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a80118f0 con 0x7fa6b80cbf20 2023-11-24T08:05:08.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b8154870 con 0x7fa6b80cbf20 2023-11-24T08:05:08.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8011ce0 con 0x7fa6b80cbf20 2023-11-24T08:05:08.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.535+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b8155550 con 0x7fa6b80cbf20 2023-11-24T08:05:08.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.537+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b806dc80 con 0x7fa6b80cbf20 2023-11-24T08:05:08.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.537+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa6a801c460 con 0x7fa6b80cbf20 2023-11-24T08:05:08.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.538+0000 7fa6adffb640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 0x7fa69c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.538+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6a8089600 con 0x7fa6b80cbf20 2023-11-24T08:05:08.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.539+0000 7fa6bd4cf640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 0x7fa69c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.539+0000 7fa6bd4cf640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 0x7fa69c06d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.542+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6a8054610 con 0x7fa6b80cbf20 2023-11-24T08:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.733+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6b804e890 con 0x7fa6b80cbf20 2023-11-24T08:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.734+0000 7fa6adffb640 1 -- 172.21.15.107:0/3432371998 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6a8057c30 con 0x7fa6b80cbf20 2023-11-24T08:05:08.736 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.736+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 msgr2=0x7fa69c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.736+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 0x7fa69c06d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).stop 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 msgr2=0x7fa6b8153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800c430 tx=0x7fa6a800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 shutdown_connections 2023-11-24T08:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa69c06ae40 0x7fa69c06d300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815b440 0x7fa6b8153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6b80cbf20 0x7fa6b8153780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 --2- 172.21.15.107:0/3432371998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b80c59b0 0x7fa6b814ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 >> 172.21.15.107:0/3432371998 conn(0x7fa6b80b0c50 msgr2=0x7fa6b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 shutdown_connections 2023-11-24T08:05:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.737+0000 7fa6bf733640 1 -- 172.21.15.107:0/3432371998 wait complete. 2023-11-24T08:05:08.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:08.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:05:08.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:05:08.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.969+0000 7f475a1bc640 1 Processor -- start 2023-11-24T08:05:08.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.969+0000 7f475a1bc640 1 -- start start 2023-11-24T08:05:08.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.970+0000 7f475a1bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47540b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f475a1bc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475406d430 con 0x7f475415ded0 2023-11-24T08:05:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f475a1bc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540b14e0 con 0x7f475415a1c0 2023-11-24T08:05:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f475a1bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475415ea70 con 0x7f47540b0b40 2023-11-24T08:05:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f47599bb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f475415ded0 0x7f475415e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33860/0 (socket says 172.21.15.107:33860) 2023-11-24T08:05:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f47599bb640 1 -- 172.21.15.107:0/1154707420 learned_addr learned my addr 172.21.15.107:0/1154707420 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f47589b9640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47540b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057512470 0 0) 0x7f475406d430 con 0x7f475415ded0 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.971+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4740003520 con 0x7f475415ded0 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500738933 0 0) 0x7f4740003520 con 0x7f475415ded0 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 msgr2=0x7f47540b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47540b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 >> v1:172.21.15.120:6789/0 conn(0x7f475415a1c0 legacy=0x7f475406d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475415ec50 con 0x7f475415ded0 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f474c0019f0 con 0x7f475415ded0 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 >> v1:172.21.15.107:6789/0 conn(0x7f475415ded0 legacy=0x7f475415e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47400039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4740003ee0 0x7f47400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f4740005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475415ea70 con 0x7f4740004d00 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474000dc40 con 0x7f4740003ee0 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474000de00 con 0x7f47540b0b40 2023-11-24T08:05:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f47599bb640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f4740005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f47589b9640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4740003ee0 0x7f47400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.972+0000 7f47591ba640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47400039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.973+0000 7f47599bb640 1 -- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 msgr2=0x7f47400039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.973+0000 7f47599bb640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540b0b40 0x7f47400039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.973+0000 7f47599bb640 1 -- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4740003ee0 msgr2=0x7f47400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.973+0000 7f47599bb640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4740003ee0 0x7f47400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.973+0000 7f47599bb640 1 -- 172.21.15.107:0/1154707420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740003520 con 0x7f4740004d00 2023-11-24T08:05:08.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.974+0000 7f47599bb640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f4740005180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f474c00a810 tx=0x7f47400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40603163d51579e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.974+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c034630 con 0x7f4740004d00 2023-11-24T08:05:08.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.974+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c02f070 con 0x7f4740004d00 2023-11-24T08:05:08.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.974+0000 7f474bfff640 1 -- 172.21.15.107:0/1154707420 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c0349d0 con 0x7f4740004d00 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 -- 172.21.15.107:0/1154707420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 msgr2=0x7f4740005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 --2- 172.21.15.107:0/1154707420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f4740005180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f474c00a810 tx=0x7f47400057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f47599bb640 1 -- 172.21.15.107:0/1154707420 reap_dead start 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 -- 172.21.15.107:0/1154707420 shutdown_connections 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 -- 172.21.15.107:0/1154707420 >> 172.21.15.107:0/1154707420 conn(0x7f47540cbb40 msgr2=0x7f47540cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 -- 172.21.15.107:0/1154707420 shutdown_connections 2023-11-24T08:05:08.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.975+0000 7f475a1bc640 1 -- 172.21.15.107:0/1154707420 wait complete. 2023-11-24T08:05:08.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.977+0000 7f475a1bc640 1 Processor -- start 2023-11-24T08:05:08.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.977+0000 7f475a1bc640 1 -- start start 2023-11-24T08:05:08.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f475409b660 0x7f47540c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540715b0 0x7f4754071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540c5cd0 con 0x7f4740004d00 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4754071f70 con 0x7f475409b660 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f475a1bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475409c780 con 0x7f47540715b0 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f47591ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f47591ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52288/0 (socket says 172.21.15.107:52288) 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f47591ba640 1 -- 172.21.15.107:0/3953213109 learned_addr learned my addr 172.21.15.107:0/3953213109 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f47589b9640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f475409b660 0x7f47540c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.978+0000 7f47599bb640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540715b0 0x7f4754071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 -- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540715b0 msgr2=0x7f4754071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540715b0 0x7f4754071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 -- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f475409b660 msgr2=0x7f47540c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f475409b660 0x7f47540c2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 -- 172.21.15.107:0/3953213109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475409bae0 con 0x7f4740004d00 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f47591ba640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f475000e800 tx=0x7f475000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500050d0 con 0x7f4740004d00 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4750005230 con 0x7f4740004d00 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47540c3130 con 0x7f4740004d00 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475000c660 con 0x7f4740004d00 2023-11-24T08:05:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.979+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540c4f10 con 0x7f4740004d00 2023-11-24T08:05:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.981+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475404e840 con 0x7f4740004d00 2023-11-24T08:05:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.981+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f475000c7c0 con 0x7f4740004d00 2023-11-24T08:05:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.982+0000 7f4749ffb640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 0x7f473806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.982+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4750089410 con 0x7f4740004d00 2023-11-24T08:05:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.983+0000 7f47589b9640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 0x7f473806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.983+0000 7f47589b9640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 0x7f473806d330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4754157f70 tx=0x7f4744009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:08.987+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47500544b0 con 0x7f4740004d00 2023-11-24T08:05:09.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.176+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4754157fa0 con 0x7f4740004d00 2023-11-24T08:05:09.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.177+0000 7f4749ffb640 1 -- 172.21.15.107:0/3953213109 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47500012d0 con 0x7f4740004d00 2023-11-24T08:05:09.179 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 msgr2=0x7f473806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 0x7f473806d330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4754157f70 tx=0x7f4744009190 comp rx=0 tx=0).stop 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 msgr2=0x7f47540c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f475000e800 tx=0x7f475000ec00 comp rx=0 tx=0).stop 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 shutdown_connections 2023-11-24T08:05:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473806ae70 0x7f473806d330 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47540715b0 0x7f4754071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f475409b660 0x7f47540c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 --2- 172.21.15.107:0/3953213109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4740004d00 0x7f47540c45c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 >> 172.21.15.107:0/3953213109 conn(0x7f47540cbb40 msgr2=0x7f475406d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 shutdown_connections 2023-11-24T08:05:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.180+0000 7f475a1bc640 1 -- 172.21.15.107:0/3953213109 wait complete. 2023-11-24T08:05:09.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.415+0000 7f94d3fff640 1 Processor -- start 2023-11-24T08:05:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.415+0000 7f94d3fff640 1 -- start start 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94d40b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d415ed20 con 0x7f94d415b260 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d3fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d415f8a0 con 0x7f94d406c5b0 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d415b640 con 0x7f94d40b0b40 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94d415b260 0x7f94d406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33866/0 (socket says 172.21.15.107:33866) 2023-11-24T08:05:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d2ffd640 1 -- 172.21.15.107:0/2262286782 learned_addr learned my addr 172.21.15.107:0/2262286782 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94d40b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.417+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47045367 0 0) 0x7f94d415ed20 con 0x7f94d415b260 2023-11-24T08:05:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c0003520 con 0x7f94d415b260 2023-11-24T08:05:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4243269372 0 0) 0x7f94c0003520 con 0x7f94d415b260 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 msgr2=0x7f94d40b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94d40b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 >> v1:172.21.15.120:6789/0 conn(0x7f94d406c5b0 legacy=0x7f94d406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40b14c0 con 0x7f94d415b260 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f94c4001670 con 0x7f94d415b260 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 >> v1:172.21.15.107:6789/0 conn(0x7f94d415b260 legacy=0x7f94d406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00039a0 0x7f94c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c00047c0 0x7f94c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94c0005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d415b640 con 0x7f94c00047c0 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c000bb40 con 0x7f94c00039a0 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c000bd00 con 0x7f94d40b0b40 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c00047c0 0x7f94c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.418+0000 7f94d2ffd640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00039a0 0x7f94c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94d37fe640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94c0005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94ca7fc640 1 -- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 msgr2=0x7f94c0005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94c0005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94ca7fc640 1 -- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00039a0 msgr2=0x7f94c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00039a0 0x7f94c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94ca7fc640 1 -- 172.21.15.107:0/2262286782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c0003520 con 0x7f94c00047c0 2023-11-24T08:05:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.419+0000 7f94d37fe640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d40b0b40 0x7f94c0005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.420+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c00047c0 0x7f94c0004c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009fc0 tx=0x7f94c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4739976b0f22d596 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.420+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02cc20 con 0x7f94c00047c0 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.420+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc00f040 con 0x7f94c00047c0 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.420+0000 7f94d27fc640 1 -- 172.21.15.107:0/2262286782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc034440 con 0x7f94c00047c0 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 -- 172.21.15.107:0/2262286782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c00047c0 msgr2=0x7f94c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 --2- 172.21.15.107:0/2262286782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c00047c0 0x7f94c0004c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009fc0 tx=0x7f94c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d37fe640 1 -- 172.21.15.107:0/2262286782 reap_dead start 2023-11-24T08:05:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 -- 172.21.15.107:0/2262286782 shutdown_connections 2023-11-24T08:05:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 -- 172.21.15.107:0/2262286782 >> 172.21.15.107:0/2262286782 conn(0x7f94d40cbb40 msgr2=0x7f94d40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 -- 172.21.15.107:0/2262286782 shutdown_connections 2023-11-24T08:05:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.421+0000 7f94d3fff640 1 -- 172.21.15.107:0/2262286782 wait complete. 2023-11-24T08:05:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.423+0000 7f94d3fff640 1 Processor -- start 2023-11-24T08:05:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.423+0000 7f94d3fff640 1 -- start start 2023-11-24T08:05:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00047c0 0x7f94d4155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d415e430 0x7f94d40c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c5120 con 0x7f94d40c2bd0 2023-11-24T08:05:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d4156490 con 0x7f94c00047c0 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c3ed0 con 0x7f94d415e430 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52312/0 (socket says 172.21.15.107:52312) 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 -- 172.21.15.107:0/1897522757 learned_addr learned my addr 172.21.15.107:0/1897522757 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94ca7fc640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d415e430 0x7f94d40c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d2ffd640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00047c0 0x7f94d4155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 -- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d415e430 msgr2=0x7f94d40c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d415e430 0x7f94d40c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 -- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00047c0 msgr2=0x7f94d4155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00047c0 0x7f94d4155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 -- 172.21.15.107:0/1897522757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40c3910 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.424+0000 7f94d37fe640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f94d40b1750 tx=0x7f94cc00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.425+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00b070 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.425+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc007030 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.425+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0073e0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.425+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d40c3bf0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.425+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d40c5af0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.426+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d40c52d0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.427+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f94cc007540 con 0x7f94d40c2bd0 2023-11-24T08:05:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.428+0000 7f94d0ff9640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 0x7f94b406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.428+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94cc089100 con 0x7f94d40c2bd0 2023-11-24T08:05:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.428+0000 7f94d2ffd640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 0x7f94b406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.429+0000 7f94d2ffd640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 0x7f94b406d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f94c4001eb0 tx=0x7f94c4001d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.432+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94d40c52d0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.627+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f94d40c52d0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.628+0000 7f94d0ff9640 1 -- 172.21.15.107:0/1897522757 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f94d40c52d0 con 0x7f94d40c2bd0 2023-11-24T08:05:09.631 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 msgr2=0x7f94b406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 0x7f94b406d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f94c4001eb0 tx=0x7f94c4001d60 comp rx=0 tx=0).stop 2023-11-24T08:05:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 msgr2=0x7f94d40c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f94d40b1750 tx=0x7f94cc00d890 comp rx=0 tx=0).stop 2023-11-24T08:05:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 shutdown_connections 2023-11-24T08:05:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b406ae70 0x7f94b406d330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d40c2bd0 0x7f94d40c3080 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94d415e430 0x7f94d40c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 --2- 172.21.15.107:0/1897522757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c00047c0 0x7f94d4155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.630+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 >> 172.21.15.107:0/1897522757 conn(0x7f94d40cbb40 msgr2=0x7f94d40b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.631+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 shutdown_connections 2023-11-24T08:05:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.631+0000 7f94d3fff640 1 -- 172.21.15.107:0/1897522757 wait complete. 2023-11-24T08:05:09.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.857+0000 7fc58e02d640 1 Processor -- start 2023-11-24T08:05:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.857+0000 7fc58e02d640 1 -- start start 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc58e02d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc58815c7f0 0x7fc58806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc58e02d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 0x7fc5880cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc58e02d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880cece0 con 0x7fc5880cf450 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc58e02d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5880cfe40 con 0x7fc58815c7f0 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc58e02d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc588156010 con 0x7fc5880ce6c0 2023-11-24T08:05:09.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc586ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc5880cf450 0x7fc5880cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33872/0 (socket says 172.21.15.107:33872) 2023-11-24T08:05:09.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc586ffd640 1 -- 172.21.15.107:0/2033627531 learned_addr learned my addr 172.21.15.107:0/2033627531 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:09.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.859+0000 7fc5877fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc58815c7f0 0x7fc58806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc587fff640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 0x7fc5880cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc5877fe640 1 -- 172.21.15.107:0/2033627531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 msgr2=0x7fc5880cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc5877fe640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 0x7fc5880cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc5877fe640 1 -- 172.21.15.107:0/2033627531 >> v1:172.21.15.107:6789/0 conn(0x7fc5880cf450 legacy=0x7fc5880cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc5877fe640 1 -- 172.21.15.107:0/2033627531 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58806d040 con 0x7fc58815c7f0 2023-11-24T08:05:09.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc5867fc640 1 -- 172.21.15.107:0/2033627531 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175840892 0 0) 0x7fc5880cece0 con 0x7fc5880cf450 2023-11-24T08:05:09.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.860+0000 7fc587fff640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 0x7fc5880cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:09.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.861+0000 7fc5877fe640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58815c7f0 0x7fc58806db20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5780097a0 tx=0x7fc57802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bc41055bb22f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.861+0000 7fc5867fc640 1 -- 172.21.15.107:0/2033627531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578031830 con 0x7fc58815c7f0 2023-11-24T08:05:09.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.861+0000 7fc5867fc640 1 -- 172.21.15.107:0/2033627531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578031990 con 0x7fc58815c7f0 2023-11-24T08:05:09.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.861+0000 7fc5867fc640 1 -- 172.21.15.107:0/2033627531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578031d20 con 0x7fc58815c7f0 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 -- 172.21.15.107:0/2033627531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58815c7f0 msgr2=0x7fc58806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58815c7f0 0x7fc58806db20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5780097a0 tx=0x7fc57802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 -- 172.21.15.107:0/2033627531 shutdown_connections 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5880ce6c0 0x7fc5880cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 --2- 172.21.15.107:0/2033627531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58815c7f0 0x7fc58806db20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 -- 172.21.15.107:0/2033627531 >> 172.21.15.107:0/2033627531 conn(0x7fc58806a920 msgr2=0x7fc58806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 -- 172.21.15.107:0/2033627531 shutdown_connections 2023-11-24T08:05:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.862+0000 7fc58e02d640 1 -- 172.21.15.107:0/2033627531 wait complete. 2023-11-24T08:05:09.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.864+0000 7fc58e02d640 1 Processor -- start 2023-11-24T08:05:09.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.864+0000 7fc58e02d640 1 -- start start 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 0x7fc5880b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc58815c7f0 0x7fc5880b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58814d9d0 con 0x7fc5880ce6c0 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5881557a0 con 0x7fc5880cf450 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc58e02d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc588155a80 con 0x7fc58815c7f0 2023-11-24T08:05:09.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52328/0 (socket says 172.21.15.107:52328) 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 -- 172.21.15.107:0/3214155254 learned_addr learned my addr 172.21.15.107:0/3214155254 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc586ffd640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 0x7fc5880b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc587fff640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc58815c7f0 0x7fc5880b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 -- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc58815c7f0 msgr2=0x7fc5880b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc58815c7f0 0x7fc5880b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 -- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 msgr2=0x7fc5880b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 0x7fc5880b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.865+0000 7fc5877fe640 1 -- 172.21.15.107:0/3214155254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc578009400 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc586ffd640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 0x7fc5880b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc5877fe640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc578034040 tx=0x7fc5780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57803aa60 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57803abe0 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578044400 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5880c8400 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.866+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5880c89c0 con 0x7fc5880ce6c0 2023-11-24T08:05:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.867+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc588067470 con 0x7fc5880ce6c0 2023-11-24T08:05:09.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.868+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc57804d440 con 0x7fc5880ce6c0 2023-11-24T08:05:09.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.869+0000 7fc56ffff640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 0x7fc57406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.869+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5780ab8d0 con 0x7fc5880ce6c0 2023-11-24T08:05:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.869+0000 7fc586ffd640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 0x7fc57406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.870+0000 7fc586ffd640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 0x7fc57406d300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc588158f10 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:09.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:09.874+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc578076960 con 0x7fc5880ce6c0 2023-11-24T08:05:10.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.100+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58815b780 con 0x7fc5880ce6c0 2023-11-24T08:05:10.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.101+0000 7fc56ffff640 1 -- 172.21.15.107:0/3214155254 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7fc578035020 con 0x7fc5880ce6c0 2023-11-24T08:05:10.103 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:10.105 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 msgr2=0x7fc57406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 0x7fc57406d300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc588158f10 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 msgr2=0x7fc58814d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc578034040 tx=0x7fc5780046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 shutdown_connections 2023-11-24T08:05:10.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc57406ae40 0x7fc57406d300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc58815c7f0 0x7fc5880b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5880cf450 0x7fc5880b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 --2- 172.21.15.107:0/3214155254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5880ce6c0 0x7fc58814d490 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 >> 172.21.15.107:0/3214155254 conn(0x7fc58806a920 msgr2=0x7fc58815eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 shutdown_connections 2023-11-24T08:05:10.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.104+0000 7fc58e02d640 1 -- 172.21.15.107:0/3214155254 wait complete. 2023-11-24T08:05:10.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:10.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.335+0000 7f3af2623640 1 Processor -- start 2023-11-24T08:05:10.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.336+0000 7f3af2623640 1 -- start start 2023-11-24T08:05:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3af2623640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3aec15af40 0x7f3aec06d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3af2623640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15b320 0x7f3aec0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3af2623640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec15e8b0 con 0x7f3aec070c40 2023-11-24T08:05:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3af2623640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec15eb90 con 0x7f3aec15af40 2023-11-24T08:05:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3af2623640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15e6a0 con 0x7f3aec15b320 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3aec15af40 0x7f3aec06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3aec15af40 0x7f3aec06d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52330/0 (socket says 172.21.15.107:52330) 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 -- 172.21.15.107:0/4022391312 learned_addr learned my addr 172.21.15.107:0/4022391312 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3ae3fff640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15b320 0x7f3aec0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 -- 172.21.15.107:0/4022391312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15b320 msgr2=0x7f3aec0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15b320 0x7f3aec0ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 -- 172.21.15.107:0/4022391312 >> v1:172.21.15.120:6789/0 conn(0x7f3aec070c40 legacy=0x7f3aec161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:10.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.337+0000 7f3aebfff640 1 -- 172.21.15.107:0/4022391312 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec070180 con 0x7f3aec15af40 2023-11-24T08:05:10.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.338+0000 7f3aebfff640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec15af40 0x7f3aec06d220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3adc009870 tx=0x7f3adc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187df4678879f731 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3aeb7fe640 1 -- 172.21.15.107:0/4022391312 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc031640 con 0x7f3aec15af40 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3aeb7fe640 1 -- 172.21.15.107:0/4022391312 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc0317a0 con 0x7f3aec15af40 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3aeb7fe640 1 -- 172.21.15.107:0/4022391312 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc031b30 con 0x7f3aec15af40 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 -- 172.21.15.107:0/4022391312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec15af40 msgr2=0x7f3aec06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec15af40 0x7f3aec06d220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3adc009870 tx=0x7f3adc02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 -- 172.21.15.107:0/4022391312 shutdown_connections 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15b320 0x7f3aec0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 --2- 172.21.15.107:0/4022391312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec15af40 0x7f3aec06d220 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.339+0000 7f3af2623640 1 -- 172.21.15.107:0/4022391312 >> 172.21.15.107:0/4022391312 conn(0x7f3aec06e7f0 msgr2=0x7f3aec06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.340+0000 7f3af2623640 1 -- 172.21.15.107:0/4022391312 shutdown_connections 2023-11-24T08:05:10.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.340+0000 7f3af2623640 1 -- 172.21.15.107:0/4022391312 wait complete. 2023-11-24T08:05:10.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.341+0000 7f3af2623640 1 Processor -- start 2023-11-24T08:05:10.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.341+0000 7f3af2623640 1 -- start start 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15af40 0x7f3aec153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 0x7f3aec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec153da0 con 0x7f3aec070c40 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1561d0 con 0x7f3aec15b320 2023-11-24T08:05:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.342+0000 7f3af2623640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0c6740 con 0x7f3aec15af40 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52336/0 (socket says 172.21.15.107:52336) 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 -- 172.21.15.107:0/592364327 learned_addr learned my addr 172.21.15.107:0/592364327 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3af0bc0640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 0x7f3aec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 -- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15af40 msgr2=0x7f3aec153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3ae3fff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15af40 0x7f3aec153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15af40 0x7f3aec153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 -- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 msgr2=0x7f3aec155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 0x7f3aec155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 -- 172.21.15.107:0/592364327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc0094d0 con 0x7f3aec070c40 2023-11-24T08:05:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3af0bc0640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 0x7f3aec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:10.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.343+0000 7f3aebfff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3adc009870 tx=0x7f3adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.344+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc03ac60 con 0x7f3aec070c40 2023-11-24T08:05:10.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.344+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc03adc0 con 0x7f3aec070c40 2023-11-24T08:05:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.344+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc044630 con 0x7f3aec070c40 2023-11-24T08:05:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.344+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec1563e0 con 0x7f3aec070c40 2023-11-24T08:05:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.344+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec0c63d0 con 0x7f3aec070c40 2023-11-24T08:05:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.345+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec067470 con 0x7f3aec070c40 2023-11-24T08:05:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.346+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3adc02f050 con 0x7f3aec070c40 2023-11-24T08:05:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.347+0000 7f3ae97fa640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 0x7f3acc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.347+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3adc0ab400 con 0x7f3aec070c40 2023-11-24T08:05:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.347+0000 7f3ae3fff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 0x7f3acc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.348+0000 7f3ae3fff640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 0x7f3acc06d300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3aec06d630 tx=0x7f3ad4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.351+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3adc076540 con 0x7f3aec070c40 2023-11-24T08:05:10.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.542+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aec0c4890 con 0x7f3aec070c40 2023-11-24T08:05:10.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.546+0000 7f3ae97fa640 1 -- 172.21.15.107:0/592364327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f3adc031e60 con 0x7f3aec070c40 2023-11-24T08:05:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:10.549 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 msgr2=0x7f3acc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 0x7f3acc06d300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3aec06d630 tx=0x7f3ad4008040 comp rx=0 tx=0).stop 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 msgr2=0x7f3aec146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3adc009870 tx=0x7f3adc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 shutdown_connections 2023-11-24T08:05:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3acc06ae40 0x7f3acc06d300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aec15b320 0x7f3aec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aec15af40 0x7f3aec153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 --2- 172.21.15.107:0/592364327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aec070c40 0x7f3aec146120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.549+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 >> 172.21.15.107:0/592364327 conn(0x7f3aec06e7f0 msgr2=0x7f3aec15bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.550+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 shutdown_connections 2023-11-24T08:05:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.550+0000 7f3af2623640 1 -- 172.21.15.107:0/592364327 wait complete. 2023-11-24T08:05:10.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.780+0000 7f46ee451640 1 Processor -- start 2023-11-24T08:05:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.780+0000 7f46ee451640 1 -- start start 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.781+0000 7f46ee451640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46e80cf450 0x7f46e80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ee451640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e80cdfb0 0x7f46e80ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ee451640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80cece0 con 0x7f46e815c7f0 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ee451640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e80cfe40 con 0x7f46e80cdfb0 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ee451640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e8156010 con 0x7f46e80cf450 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e80cdfb0 0x7f46e80ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e80cdfb0 0x7f46e80ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52348/0 (socket says 172.21.15.107:52348) 2023-11-24T08:05:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 -- 172.21.15.107:0/982770145 learned_addr learned my addr 172.21.15.107:0/982770145 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46dffff640 1 --2- 172.21.15.107:0/982770145 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cf450 0x7f46e80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 -- 172.21.15.107:0/982770145 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cf450 msgr2=0x7f46e80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 --2- 172.21.15.107:0/982770145 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cf450 0x7f46e80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.782+0000 7f46ec9ee640 1 -- 172.21.15.107:0/982770145 >> v1:172.21.15.120:6789/0 conn(0x7f46e815c7f0 legacy=0x7f46e806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.783+0000 7f46ec9ee640 1 -- 172.21.15.107:0/982770145 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e806d040 con 0x7f46e80cdfb0 2023-11-24T08:05:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.784+0000 7f46ec9ee640 1 --2- 172.21.15.107:0/982770145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e80ce3e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46e8070410 tx=0x7f46e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108bf72467ed70ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.784+0000 7f46e77fe640 1 -- 172.21.15.107:0/982770145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e002ca80 con 0x7f46e80cdfb0 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.784+0000 7f46e77fe640 1 -- 172.21.15.107:0/982770145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e002cbe0 con 0x7f46e80cdfb0 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.784+0000 7f46e77fe640 1 -- 172.21.15.107:0/982770145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e00343f0 con 0x7f46e80cdfb0 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 -- 172.21.15.107:0/982770145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 msgr2=0x7f46e80ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 --2- 172.21.15.107:0/982770145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e80ce3e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46e8070410 tx=0x7f46e002bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 -- 172.21.15.107:0/982770145 shutdown_connections 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 --2- 172.21.15.107:0/982770145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e80ce3e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 --2- 172.21.15.107:0/982770145 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cf450 0x7f46e80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 -- 172.21.15.107:0/982770145 >> 172.21.15.107:0/982770145 conn(0x7f46e806a920 msgr2=0x7f46e806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 -- 172.21.15.107:0/982770145 shutdown_connections 2023-11-24T08:05:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.785+0000 7f46ee451640 1 -- 172.21.15.107:0/982770145 wait complete. 2023-11-24T08:05:10.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.787+0000 7f46ee451640 1 Processor -- start 2023-11-24T08:05:10.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.787+0000 7f46ee451640 1 -- start start 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80cf450 0x7f46e8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e815c7f0 0x7f46e8157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e815a9c0 con 0x7f46e80cdfb0 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e81576b0 con 0x7f46e815c7f0 2023-11-24T08:05:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ee451640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8157990 con 0x7f46e80cf450 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52350/0 (socket says 172.21.15.107:52350) 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 -- 172.21.15.107:0/3659453116 learned_addr learned my addr 172.21.15.107:0/3659453116 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46ec9ee640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e815c7f0 0x7f46e8157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46dffff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80cf450 0x7f46e8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 -- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80cf450 msgr2=0x7f46e8156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80cf450 0x7f46e8156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 -- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e815c7f0 msgr2=0x7f46e8157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e815c7f0 0x7f46e8157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.788+0000 7f46e7fff640 1 -- 172.21.15.107:0/3659453116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e0009b30 con 0x7f46e80cdfb0 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46e7fff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b3f0 tx=0x7f46d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8011830 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d8011990 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8011d40 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e8157c70 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.789+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e814d980 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.790+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e8067470 con 0x7f46e80cdfb0 2023-11-24T08:05:10.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.791+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f46d801a400 con 0x7f46e80cdfb0 2023-11-24T08:05:10.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.792+0000 7f46e57fa640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 0x7f46c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:10.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.792+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46d8089560 con 0x7f46e80cdfb0 2023-11-24T08:05:10.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.792+0000 7f46dffff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 0x7f46c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.793+0000 7f46dffff640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 0x7f46c806d300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46e80cfe10 tx=0x7f46d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:10.796+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46d80544e0 con 0x7f46e80cdfb0 2023-11-24T08:05:11.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.026+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46e80cd730 con 0x7f46e80cdfb0 2023-11-24T08:05:11.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.027+0000 7f46e57fa640 1 -- 172.21.15.107:0/3659453116 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f46d8057b00 con 0x7f46e80cdfb0 2023-11-24T08:05:11.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:11.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:11.029 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:11.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 msgr2=0x7f46c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 0x7f46c806d300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46e80cfe10 tx=0x7f46d0008040 comp rx=0 tx=0).stop 2023-11-24T08:05:11.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 msgr2=0x7f46e814d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b3f0 tx=0x7f46d800b8b0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 shutdown_connections 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46c806ae40 0x7f46c806d300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e815c7f0 0x7f46e8157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80cf450 0x7f46e8156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 --2- 172.21.15.107:0/3659453116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e80cdfb0 0x7f46e814d440 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.029+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 >> 172.21.15.107:0/3659453116 conn(0x7f46e806a920 msgr2=0x7f46e80b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.030+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 shutdown_connections 2023-11-24T08:05:11.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.030+0000 7f46ee451640 1 -- 172.21.15.107:0/3659453116 wait complete. 2023-11-24T08:05:11.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:04:53.954635+0000', 'modified': '2023-11-24T08:04:57.215293+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/1122137333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1122137333}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1122137333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:05:11.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:05:11.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:05:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.672+0000 7f1b888c8640 1 Processor -- start 2023-11-24T08:05:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.672+0000 7f1b888c8640 1 -- start start 2023-11-24T08:05:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b888c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b840b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b888c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b840ca0e0 con 0x7f1b840b1080 2023-11-24T08:05:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b888c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b84072800 con 0x7f1b8415b1d0 2023-11-24T08:05:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b888c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840b1510 con 0x7f1b84070550 2023-11-24T08:05:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b8359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1b840b1080 0x7f1b8415edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33882/0 (socket says 172.21.15.107:33882) 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b8359e640 1 -- 172.21.15.107:0/3052933926 learned_addr learned my addr 172.21.15.107:0/3052933926 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b840b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.674+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733655705 0 0) 0x7f1b840ca0e0 con 0x7f1b840b1080 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70003520 con 0x7f1b840b1080 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686312658 0 0) 0x7f1b70003520 con 0x7f1b840b1080 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 msgr2=0x7f1b840b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b840b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 >> v1:172.21.15.120:6789/0 conn(0x7f1b8415b1d0 legacy=0x7f1b84071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840c97b0 con 0x7f1b840b1080 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1b7c0019f0 con 0x7f1b840b1080 2023-11-24T08:05:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 >> v1:172.21.15.107:6789/0 conn(0x7f1b840b1080 legacy=0x7f1b8415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b700039a0 0x7f1b70003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840b1510 con 0x7f1b70004d00 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7000ba60 con 0x7f1b700039a0 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7000bc20 con 0x7f1b84070550 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.675+0000 7f1b8359e640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b700039a0 0x7f1b70003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b8359e640 1 -- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 msgr2=0x7f1b700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b8359e640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b8359e640 1 -- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b700039a0 msgr2=0x7f1b70003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b8359e640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b700039a0 0x7f1b70003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b8359e640 1 -- 172.21.15.107:0/3052933926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70003520 con 0x7f1b70004d00 2023-11-24T08:05:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.676+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84070550 0x7f1b700047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b8359e640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b70005180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c002230 tx=0x7f1b700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef2ed9d81d66e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c034630 con 0x7f1b70004d00 2023-11-24T08:05:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c02f070 con 0x7f1b70004d00 2023-11-24T08:05:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b8259c640 1 -- 172.21.15.107:0/3052933926 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c034a10 con 0x7f1b70004d00 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b888c8640 1 -- 172.21.15.107:0/3052933926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 msgr2=0x7f1b70005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.677+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3052933926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b70005180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c002230 tx=0x7f1b700057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.678+0000 7f1b8359e640 1 -- 172.21.15.107:0/3052933926 reap_dead start 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.678+0000 7f1b888c8640 1 -- 172.21.15.107:0/3052933926 shutdown_connections 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.678+0000 7f1b888c8640 1 -- 172.21.15.107:0/3052933926 >> 172.21.15.107:0/3052933926 conn(0x7f1b840cbb40 msgr2=0x7f1b840cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.678+0000 7f1b888c8640 1 -- 172.21.15.107:0/3052933926 shutdown_connections 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.678+0000 7f1b888c8640 1 -- 172.21.15.107:0/3052933926 wait complete. 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.679+0000 7f1b888c8640 1 Processor -- start 2023-11-24T08:05:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.680+0000 7f1b888c8640 1 -- start start 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.680+0000 7f1b888c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.680+0000 7f1b888c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84159e20 0x7f1b84159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b888c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b840c48d0 0x7f1b840c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b888c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c4450 con 0x7f1b70004d00 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b888c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c7760 con 0x7f1b840c48d0 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b888c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840cd060 con 0x7f1b84159e20 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52376/0 (socket says 172.21.15.107:52376) 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3282561397 learned_addr learned my addr 172.21.15.107:0/3282561397 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84159e20 msgr2=0x7f1b84159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84159e20 0x7f1b84159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b840c48d0 msgr2=0x7f1b840c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b8359e640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b840c48d0 0x7f1b840c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b840c48d0 0x7f1b840c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3282561397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840cd340 con 0x7f1b70004d00 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.681+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400e4d0 tx=0x7f1b7400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.682+0000 7f1b7b7fe640 1 -- 172.21.15.107:0/3282561397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b740053a0 con 0x7f1b70004d00 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.682+0000 7f1b7b7fe640 1 -- 172.21.15.107:0/3282561397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b740035f0 con 0x7f1b70004d00 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.682+0000 7f1b7b7fe640 1 -- 172.21.15.107:0/3282561397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74003980 con 0x7f1b70004d00 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.682+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840cd620 con 0x7f1b70004d00 2023-11-24T08:05:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.682+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8415a520 con 0x7f1b70004d00 2023-11-24T08:05:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.684+0000 7f1b7b7fe640 1 -- 172.21.15.107:0/3282561397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1b74003ae0 con 0x7f1b70004d00 2023-11-24T08:05:11.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.685+0000 7f1b7b7fe640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 0x7f1b6406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.686+0000 7f1b7b7fe640 1 -- 172.21.15.107:0/3282561397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b74089460 con 0x7f1b70004d00 2023-11-24T08:05:11.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.686+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 0x7f1b6406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.686+0000 7f1b8bc5d740 1 Processor -- start 2023-11-24T08:05:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.686+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 0x7f1b6406d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1b8415e8e0 tx=0x7f1b6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.686+0000 7f1b8bc5d740 1 -- start start 2023-11-24T08:05:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db43d70 0x55923db48270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db8c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923daa7880 con 0x55923db49070 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923dab6ae0 con 0x55923db487b0 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55923dab41d0 con 0x55923db43d70 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.687+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52386/0 (socket says 172.21.15.107:52386) 2023-11-24T08:05:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 -- 172.21.15.107:0/2010145364 learned_addr learned my addr 172.21.15.107:0/2010145364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db43d70 0x55923db48270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 -- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db43d70 msgr2=0x55923db48270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db43d70 0x55923db48270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 -- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 msgr2=0x55923db8c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db8c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 -- 172.21.15.107:0/2010145364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55923da77860 con 0x55923db49070 2023-11-24T08:05:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1b74087830 tx=0x7f1b74058030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430a087b833ef190 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b78ff9640 1 -- 172.21.15.107:0/2010145364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74058130 con 0x55923db49070 2023-11-24T08:05:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b78ff9640 1 -- 172.21.15.107:0/2010145364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7405c280 con 0x55923db49070 2023-11-24T08:05:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.688+0000 7f1b78ff9640 1 -- 172.21.15.107:0/2010145364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74060430 con 0x55923db49070 2023-11-24T08:05:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2010145364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 msgr2=0x55923db49450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1b74087830 tx=0x7f1b74058030 comp rx=0 tx=0).stop 2023-11-24T08:05:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2010145364 shutdown_connections 2023-11-24T08:05:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db8c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db43d70 0x55923db48270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2010145364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db49450 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2010145364 >> 172.21.15.107:0/2010145364 conn(0x55923db0ac70 msgr2=0x55923db8eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2010145364 shutdown_connections 2023-11-24T08:05:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.689+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2010145364 wait complete. 2023-11-24T08:05:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.690+0000 7f1b8bc5d740 1 -- start start 2023-11-24T08:05:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.690+0000 7f1b8bc5d740 1 Processor -- start 2023-11-24T08:05:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db93b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db49070 0x55923db58b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923daafac0 con 0x55923db43d70 2023-11-24T08:05:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923da4c2f0 con 0x55923db487b0 2023-11-24T08:05:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55923da69c70 con 0x55923db49070 2023-11-24T08:05:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b8bc5d740 10 client.0 _ll_register_callbacks cb 0x55923da6e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:05:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52398/0 (socket says 172.21.15.107:52398) 2023-11-24T08:05:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3583005650 learned_addr learned my addr 172.21.15.107:0/3583005650 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db49070 msgr2=0x55923db58b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db49070 0x55923db58b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 msgr2=0x55923db93b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db93b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.691+0000 7f1b82d9d640 1 -- 172.21.15.107:0/3583005650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8415a520 con 0x55923db43d70 2023-11-24T08:05:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1b740050a0 tx=0x7f1b74012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b049f2e4ce292c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:05:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74060430 con 0x55923db43d70 2023-11-24T08:05:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b74058130 con 0x55923db43d70 2023-11-24T08:05:11.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55923daa8aa0 con 0x55923db43d70 2023-11-24T08:05:11.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b74054840 con 0x55923db43d70 2023-11-24T08:05:11.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 10 client.4644 fetch_fsmap learned FSMap version 7 2023-11-24T08:05:11.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55923daae140 con 0x55923db43d70 2023-11-24T08:05:11.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55923daa8aa0 con 0x55923db43d70 2023-11-24T08:05:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1b74057620 con 0x55923db43d70 2023-11-24T08:05:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 10 client.4644 fetch_fsmap finished waiting for FSMap version 7 2023-11-24T08:05:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 10 client.4644 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:05:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 20 client.4644 populate_metadata read hostname 'smithi107' 2023-11-24T08:05:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.692+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db714a0 0x55923db71880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.693+0000 7f1b8bc5d740 4 client.4644 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:05:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.693+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 --> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55923daa87c0 con 0x55923db714a0 2023-11-24T08:05:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.693+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db714a0 0x55923db71880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.693+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db714a0 0x55923db71880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d2e3425740b991b server_cookie=abf71dd9a9b7aaf4 in_seq=0 out_seq=0 2023-11-24T08:05:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.693+0000 7f1b61ffb640 10 client.4644 ms_handle_connect on v2:172.21.15.107:6836/1122137333 2023-11-24T08:05:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b61ffb640 1 -- 172.21.15.107:0/3583005650 <== mds.0 v2:172.21.15.107:6836/1122137333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55923daa87c0 con 0x55923db714a0 2023-11-24T08:05:11.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b61ffb640 10 client.4644 handle_command_reply: tid=0 2023-11-24T08:05:11.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 client.4644 shutdown 2023-11-24T08:05:11.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 msgr2=0x55923db5b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1b740050a0 tx=0x7f1b74012000 comp rx=0 tx=0).stop 2023-11-24T08:05:11.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 shutdown_connections 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db714a0 0x55923db71880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db49070 0x55923db58b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db487b0 0x55923db93b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/3583005650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db43d70 0x55923db5b360 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b61ffb640 0 client.4644 ms_handle_reset on v2:172.21.15.107:6836/1122137333 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 >> 172.21.15.107:0/3583005650 conn(0x55923db0ac70 msgr2=0x55923db91ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 shutdown_connections 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.694+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/3583005650 wait complete. 2023-11-24T08:05:11.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.695+0000 7f1b8bc5d740 20 client.4644 trim_cache size 0 max 16384 2023-11-24T08:05:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.711+0000 7f1b8bc5d740 1 Processor -- start 2023-11-24T08:05:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.711+0000 7f1b8bc5d740 1 -- start start 2023-11-24T08:05:11.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923db77520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db42fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923d812f20 con 0x55923db49070 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923da33520 con 0x55923db43d70 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55923dbb1380 con 0x55923db487b0 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52404/0 (socket says 172.21.15.107:52404) 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.712+0000 7f1b8359e640 1 -- 172.21.15.107:0/179666675 learned_addr learned my addr 172.21.15.107:0/179666675 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db42fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 -- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 msgr2=0x55923db42fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db42fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923db77520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 -- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 msgr2=0x55923db77520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923db77520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 -- 172.21.15.107:0/179666675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55923da4bb80 con 0x55923db49070 2023-11-24T08:05:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db42fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b8359e640 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c02ca70 tx=0x7f1b7c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427ea18d85476635 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b797fa640 1 -- 172.21.15.107:0/179666675 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c001250 con 0x55923db49070 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b797fa640 1 -- 172.21.15.107:0/179666675 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c001730 con 0x55923db49070 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.713+0000 7f1b797fa640 1 -- 172.21.15.107:0/179666675 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c040460 con 0x55923db49070 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/179666675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 msgr2=0x55923dba0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c02ca70 tx=0x7f1b7c02e000 comp rx=0 tx=0).stop 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/179666675 shutdown_connections 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923dba0850 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db42fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/179666675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923db77520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/179666675 >> 172.21.15.107:0/179666675 conn(0x55923db0ac70 msgr2=0x55923db47960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/179666675 shutdown_connections 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.714+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/179666675 wait complete. 2023-11-24T08:05:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.715+0000 7f1b8bc5d740 1 -- start start 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.715+0000 7f1b8bc5d740 1 Processor -- start 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923dbf87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db6f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923db8b490 con 0x55923db49070 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923dba1a60 con 0x55923db43d70 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55923dbfa200 con 0x55923db487b0 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8bc5d740 10 client.0 _ll_register_callbacks cb 0x7f1b74057c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52406/0 (socket says 172.21.15.107:52406) 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 -- 172.21.15.107:0/2781060270 learned_addr learned my addr 172.21.15.107:0/2781060270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923dbf87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db6f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 -- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 msgr2=0x55923db6f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db6f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 -- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 msgr2=0x55923dbf87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923dbf87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b8359e640 1 -- 172.21.15.107:0/2781060270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c038040 con 0x55923db49070 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b7bfff640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db6f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.716+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923dbf87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b8359e640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c042040 tx=0x7f1b7c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d3e826f50f747a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c054470 con 0x55923db49070 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b7c0545d0 con 0x55923db49070 2023-11-24T08:05:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b7405c280 con 0x55923db49070 2023-11-24T08:05:11.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b7c054790 con 0x55923db49070 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b8bc5d740 10 client.4650 fetch_fsmap learned FSMap version 7 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55923da71170 con 0x55923db49070 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.717+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7405c280 con 0x55923db49070 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1b7c0548f0 con 0x55923db49070 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 10 client.4650 fetch_fsmap finished waiting for FSMap version 7 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 10 client.4650 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 20 client.4650 populate_metadata read hostname 'smithi107' 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db920d0 0x55923db924b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 4 client.4650 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 --> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55923d7b06c0 con 0x55923db920d0 2023-11-24T08:05:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db920d0 0x55923db924b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b82d9d640 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db920d0 0x55923db924b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d9ded78c04021e server_cookie=37cfbcab6f99089f in_seq=0 out_seq=0 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.718+0000 7f1b61ffb640 10 client.4650 ms_handle_connect on v2:172.21.15.107:6836/1122137333 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b61ffb640 1 -- 172.21.15.107:0/2781060270 <== mds.0 v2:172.21.15.107:6836/1122137333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55923d7b06c0 con 0x55923db920d0 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b61ffb640 10 client.4650 handle_command_reply: tid=0 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 client.4650 shutdown 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 msgr2=0x55923db6bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c042040 tx=0x7f1b7c02f000 comp rx=0 tx=0).stop 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 shutdown_connections 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:6836/1122137333,v1:172.21.15.107:6837/1122137333] conn(0x55923db920d0 0x55923db924b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55923db49070 0x55923db6bda0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b61ffb640 0 client.4650 ms_handle_reset on v2:172.21.15.107:6836/1122137333 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55923db487b0 0x55923db6f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 --2- 172.21.15.107:0/2781060270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55923db43d70 0x55923dbf87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 >> 172.21.15.107:0/2781060270 conn(0x55923db0ac70 msgr2=0x55923db44150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 shutdown_connections 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 1 -- 172.21.15.107:0/2781060270 wait complete. 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.719+0000 7f1b8bc5d740 20 client.4650 trim_cache size 0 max 16384 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4257, 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:05:11.726 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 17.186263284999999, 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 7, 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 20, 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 20, 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 22.273591954 2023-11-24T08:05:11.727 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 msgr2=0x7f1b6406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 0x7f1b6406d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1b8415e8e0 tx=0x7f1b6c008040 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 msgr2=0x7f1b8415b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400e4d0 tx=0x7f1b7400e990 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 shutdown_connections 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406ae70 0x7f1b6406d330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b840c48d0 0x7f1b840c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b84159e20 0x7f1b84159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 --2- 172.21.15.107:0/3282561397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b70004d00 0x7f1b8415b850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 >> 172.21.15.107:0/3282561397 conn(0x7f1b840cbb40 msgr2=0x7f1b8409b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 shutdown_connections 2023-11-24T08:05:11.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:11.720+0000 7f1b888c8640 1 -- 172.21.15.107:0/3282561397 wait complete. 2023-11-24T08:05:11.738 INFO:teuthology.run_tasks:Running task kclient... 2023-11-24T08:05:11.760 INFO:tasks.kclient:Mounting kernel clients... 2023-11-24T08:05:11.760 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-24T08:05:11.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:11.761 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:05:11.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:11.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-11-24T08:05:11.914 INFO:teuthology.orchestra.run.smithi107.stderr:Cannot find device "ceph-brx" 2023-11-24T08:05:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:05:11.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:11.917 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:05:11.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:11.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-11-24T08:05:12.066 INFO:teuthology.orchestra.run.smithi107.stderr:Cannot find device "ceph-brx" 2023-11-24T08:05:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:05:12.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.071 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:05:12.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.132 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2023-11-24T08:05:12.234 INFO:teuthology.orchestra.run.smithi120.stderr:Cannot find device "ceph-brx" 2023-11-24T08:05:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:05:12.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.239 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:05:12.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.293 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2023-11-24T08:05:12.392 INFO:teuthology.orchestra.run.smithi120.stderr:Cannot find device "ceph-brx" 2023-11-24T08:05:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:05:12.396 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-11-24T08:05:12.396 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T08:05:12.396 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:05:12.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:05:12.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.614+0000 7f7d69b33640 1 Processor -- start 2023-11-24T08:05:12.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.614+0000 7f7d69b33640 1 -- start start 2023-11-24T08:05:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d69b33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d6415c7f0 0x7f7d6406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d69b33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 0x7f7d640cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d69b33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640cece0 con 0x7f7d640cf450 2023-11-24T08:05:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d69b33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640cfe40 con 0x7f7d6415c7f0 2023-11-24T08:05:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d69b33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640ceaf0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d637fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d6415c7f0 0x7f7d6406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d5affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7d640cf450 0x7f7d640cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33896/0 (socket says 172.21.15.107:33896) 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.616+0000 7f7d5affd640 1 -- 172.21.15.107:0/116185581 learned_addr learned my addr 172.21.15.107:0/116185581 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d63fff640 1 --2- 172.21.15.107:0/116185581 >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 0x7f7d640cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d637fe640 1 -- 172.21.15.107:0/116185581 >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 msgr2=0x7f7d640cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d637fe640 1 --2- 172.21.15.107:0/116185581 >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 0x7f7d640cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d637fe640 1 -- 172.21.15.107:0/116185581 >> v1:172.21.15.107:6789/0 conn(0x7f7d640cf450 legacy=0x7f7d640cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d637fe640 1 -- 172.21.15.107:0/116185581 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6406d040 con 0x7f7d6415c7f0 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d62ffd640 1 -- 172.21.15.107:0/116185581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508066193 0 0) 0x7f7d640cece0 con 0x7f7d640cf450 2023-11-24T08:05:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.617+0000 7f7d63fff640 1 --2- 172.21.15.107:0/116185581 >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 0x7f7d640cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:12.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.618+0000 7f7d637fe640 1 --2- 172.21.15.107:0/116185581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d6415c7f0 0x7f7d6406db20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d640d00e0 tx=0x7f7d5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345aea8d284eacc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:12.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.618+0000 7f7d62ffd640 1 -- 172.21.15.107:0/116185581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031830 con 0x7f7d6415c7f0 2023-11-24T08:05:12.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.618+0000 7f7d62ffd640 1 -- 172.21.15.107:0/116185581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54031990 con 0x7f7d6415c7f0 2023-11-24T08:05:12.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.618+0000 7f7d62ffd640 1 -- 172.21.15.107:0/116185581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031d20 con 0x7f7d6415c7f0 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 -- 172.21.15.107:0/116185581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d6415c7f0 msgr2=0x7f7d6406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 --2- 172.21.15.107:0/116185581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d6415c7f0 0x7f7d6406db20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d640d00e0 tx=0x7f7d5402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 -- 172.21.15.107:0/116185581 shutdown_connections 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 --2- 172.21.15.107:0/116185581 >> v2:172.21.15.120:3300/0 conn(0x7f7d640ce6c0 0x7f7d640cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 --2- 172.21.15.107:0/116185581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d6415c7f0 0x7f7d6406db20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 -- 172.21.15.107:0/116185581 >> 172.21.15.107:0/116185581 conn(0x7f7d6406a920 msgr2=0x7f7d6406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 -- 172.21.15.107:0/116185581 shutdown_connections 2023-11-24T08:05:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.619+0000 7f7d69b33640 1 -- 172.21.15.107:0/116185581 wait complete. 2023-11-24T08:05:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.621+0000 7f7d69b33640 1 Processor -- start 2023-11-24T08:05:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.621+0000 7f7d69b33640 1 -- start start 2023-11-24T08:05:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d640cf450 0x7f7d640b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d6415c7f0 0x7f7d640ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640b41b0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641564e0 con 0x7f7d6415c7f0 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d69b33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641567c0 con 0x7f7d640cf450 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52422/0 (socket says 172.21.15.107:52422) 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 -- 172.21.15.107:0/1214346678 learned_addr learned my addr 172.21.15.107:0/1214346678 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 -- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d640cf450 msgr2=0x7f7d640b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d640cf450 0x7f7d640b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 -- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d6415c7f0 msgr2=0x7f7d640ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d6415c7f0 0x7f7d640ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.622+0000 7f7d637fe640 1 -- 172.21.15.107:0/1214346678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54009400 con 0x7f7d640ce6c0 2023-11-24T08:05:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d637fe640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5403aa60 con 0x7f7d640ce6c0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5403abe0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d540443f0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d64156aa0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.623+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d64157060 con 0x7f7d640ce6c0 2023-11-24T08:05:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.624+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d64061a50 con 0x7f7d640ce6c0 2023-11-24T08:05:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.625+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7d5404d450 con 0x7f7d640ce6c0 2023-11-24T08:05:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.626+0000 7f7d60ff9640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 0x7f7d4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.626+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d540ab860 con 0x7f7d640ce6c0 2023-11-24T08:05:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.626+0000 7f7d5affd640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 0x7f7d4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.627+0000 7f7d5affd640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 0x7f7d4806d300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c007fd0 tx=0x7f7d4c007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.630+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d54076910 con 0x7f7d640ce6c0 2023-11-24T08:05:12.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.853+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7d6415ffc0 con 0x7f7d640ce6c0 2023-11-24T08:05:12.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.854+0000 7f7d60ff9640 1 -- 172.21.15.107:0/1214346678 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7d54035070 con 0x7f7d640ce6c0 2023-11-24T08:05:12.856 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.856+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 msgr2=0x7f7d4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.856+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 0x7f7d4806d300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c007fd0 tx=0x7f7d4c007920 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.856+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 msgr2=0x7f7d64155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.856+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d540046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 shutdown_connections 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d4806ae40 0x7f7d4806d300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d6415c7f0 0x7f7d640ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d640cf450 0x7f7d640b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 --2- 172.21.15.107:0/1214346678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d640ce6c0 0x7f7d64155de0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 >> 172.21.15.107:0/1214346678 conn(0x7f7d6406a920 msgr2=0x7f7d640cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:12.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 shutdown_connections 2023-11-24T08:05:12.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:12.857+0000 7f7d69b33640 1 -- 172.21.15.107:0/1214346678 wait complete. 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:05:12.877 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:05:12.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.878 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:05:12.929 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:12.929 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:12.929 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:12.929 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:12.930 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:12.930 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:12.930 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:12.930 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:12.930 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:12.931 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:05:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42357sec preferred_lft 42357sec 2023-11-24T08:05:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:05:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:12.932 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T08:05:12.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2023-11-24T08:05:12.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T08:05:12.934 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:13.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:13.405 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:05:13.485 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:05:13.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:13.488 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:05:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:05:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:05:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:05:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:05:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:05:13.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:05:13.560 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:14.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:14.350 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:05:14.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:14.409 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2023-11-24T08:05:14.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:14.465 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:14.465 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:14.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:05:14.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-24T08:05:14.465 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:14.606 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-24T08:05:14.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-24T08:05:14.607 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:14.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:14.887 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:14.887 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:14.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:05:14.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T08:05:14.888 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:15.046 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:15.046 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:05:15.090 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:05:15.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:15.091 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:05:15.187 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:05:15.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:15.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:15.359 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:15.359 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:15.847 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:15.847 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:15.848 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:05:15.848 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:05:15.848 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:05:15.850 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:05:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:05:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:05:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:05:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:05:15.858 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:05:15.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:15.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:05:15.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:15.956 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:16.027 INFO:teuthology.orchestra.run.smithi107.stdout:client4421 2023-11-24T08:05:16.027 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:16.027 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:16.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.082 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:16.146 INFO:teuthology.orchestra.run.smithi107.stdout:client4421 2023-11-24T08:05:16.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.146 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:16.214 INFO:teuthology.orchestra.run.smithi107.stdout:client4421 2023-11-24T08:05:16.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.214 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.client4421/status') 2023-11-24T08:05:16.310 INFO:teuthology.orchestra.run.smithi107.stdout:instance: client.4421 (3)172.21.15.107:0/3173054686 2023-11-24T08:05:16.310 INFO:teuthology.orchestra.run.smithi107.stdout:blocklisted: false 2023-11-24T08:05:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records in 2023-11-24T08:05:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records out 2023-11-24T08:05:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:71 bytes copied, 6.5238e-05 s, 1.1 MB/s 2023-11-24T08:05:16.315 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-11-24T08:05:16.315 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T08:05:16.315 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T08:05:16.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:05:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.522+0000 7fb0b359e640 1 Processor -- start 2023-11-24T08:05:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.523+0000 7fb0b359e640 1 -- start start 2023-11-24T08:05:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.524+0000 7fb0b359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb0b415c410 0x7fb0b415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.524+0000 7fb0b359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b415bf80 con 0x7fb0b40b1de0 2023-11-24T08:05:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.524+0000 7fb0b359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b40b1560 con 0x7fb0b4144b30 2023-11-24T08:05:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.524+0000 7fb0b359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b40b1280 con 0x7fb0b415c410 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0b2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb0b40b1de0 0x7fb0b40b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33902/0 (socket says 172.21.15.107:33902) 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0abfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb0b415c410 0x7fb0b415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0b2d9d640 1 -- 172.21.15.107:0/291878014 learned_addr learned my addr 172.21.15.107:0/291878014 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0abfff640 1 -- 172.21.15.107:0/291878014 >> v1:172.21.15.120:6789/0 conn(0x7fb0b4144b30 legacy=0x7fb0b40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0abfff640 1 -- 172.21.15.107:0/291878014 >> v1:172.21.15.107:6789/0 conn(0x7fb0b40b1de0 legacy=0x7fb0b40b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.525+0000 7fb0abfff640 1 -- 172.21.15.107:0/291878014 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b415b930 con 0x7fb0b415c410 2023-11-24T08:05:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.526+0000 7fb0abfff640 1 --2- 172.21.15.107:0/291878014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b415c410 0x7fb0b415f920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0097a0 tx=0x7fb09c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae70fd0e3e5e608 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b1d9b640 1 -- 172.21.15.107:0/291878014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c031830 con 0x7fb0b415c410 2023-11-24T08:05:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b1d9b640 1 -- 172.21.15.107:0/291878014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c031990 con 0x7fb0b415c410 2023-11-24T08:05:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b1d9b640 1 -- 172.21.15.107:0/291878014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c031d20 con 0x7fb0b415c410 2023-11-24T08:05:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 -- 172.21.15.107:0/291878014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b415c410 msgr2=0x7fb0b415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 --2- 172.21.15.107:0/291878014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b415c410 0x7fb0b415f920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0097a0 tx=0x7fb09c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 -- 172.21.15.107:0/291878014 shutdown_connections 2023-11-24T08:05:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 --2- 172.21.15.107:0/291878014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b415c410 0x7fb0b415f920 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 -- 172.21.15.107:0/291878014 >> 172.21.15.107:0/291878014 conn(0x7fb0b40cb830 msgr2=0x7fb0b40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 -- 172.21.15.107:0/291878014 shutdown_connections 2023-11-24T08:05:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.527+0000 7fb0b359e640 1 -- 172.21.15.107:0/291878014 wait complete. 2023-11-24T08:05:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.529+0000 7fb0b359e640 1 Processor -- start 2023-11-24T08:05:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.529+0000 7fb0b359e640 1 -- start start 2023-11-24T08:05:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0b40b1de0 0x7fb0b413c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0b415c410 0x7fb0b413f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40ca670 con 0x7fb0b4144b30 2023-11-24T08:05:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c97e0 con 0x7fb0b40b1de0 2023-11-24T08:05:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0b359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c9a60 con 0x7fb0b415c410 2023-11-24T08:05:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52448/0 (socket says 172.21.15.107:52448) 2023-11-24T08:05:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.530+0000 7fb0abfff640 1 -- 172.21.15.107:0/593745925 learned_addr learned my addr 172.21.15.107:0/593745925 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0b2d9d640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0b415c410 0x7fb0b413f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 -- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0b415c410 msgr2=0x7fb0b413f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0b415c410 0x7fb0b413f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 -- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0b40b1de0 msgr2=0x7fb0b413c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0b40b1de0 0x7fb0b413c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 -- 172.21.15.107:0/593745925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c009400 con 0x7fb0b4144b30 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0abfff640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009890 tx=0x7fb09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c03baa0 con 0x7fb0b4144b30 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c03bc20 con 0x7fb0b4144b30 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0433f0 con 0x7fb0b4144b30 2023-11-24T08:05:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.531+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b40c9d40 con 0x7fb0b4144b30 2023-11-24T08:05:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.532+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b40ca300 con 0x7fb0b4144b30 2023-11-24T08:05:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.533+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b404e840 con 0x7fb0b4144b30 2023-11-24T08:05:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.534+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb09c030380 con 0x7fb0b4144b30 2023-11-24T08:05:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.535+0000 7fb0aaffd640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 0x7fb09406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.535+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb09c0ab300 con 0x7fb0b4144b30 2023-11-24T08:05:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.535+0000 7fb0b259c640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 0x7fb09406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.535+0000 7fb0b259c640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 0x7fb09406d300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4006fd0 tx=0x7fb0a4006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:16.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.540+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb09c076310 con 0x7fb0b4144b30 2023-11-24T08:05:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.766+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb0b406d860 con 0x7fb0b4144b30 2023-11-24T08:05:16.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.767+0000 7fb0aaffd640 1 -- 172.21.15.107:0/593745925 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb09c07ee60 con 0x7fb0b4144b30 2023-11-24T08:05:16.769 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:05:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 msgr2=0x7fb09406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 0x7fb09406d300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4006fd0 tx=0x7fb0a4006bc0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 msgr2=0x7fb0b4143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009890 tx=0x7fb09c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 shutdown_connections 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb09406ae40 0x7fb09406d300 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0b415c410 0x7fb0b413f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0b4144b30 0x7fb0b4143bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 --2- 172.21.15.107:0/593745925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0b40b1de0 0x7fb0b413c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 >> 172.21.15.107:0/593745925 conn(0x7fb0b40cb830 msgr2=0x7fb0b40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 shutdown_connections 2023-11-24T08:05:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:16.769+0000 7fb0b359e640 1 -- 172.21.15.107:0/593745925 wait complete. 2023-11-24T08:05:16.789 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T08:05:16.789 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:05:16.789 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:05:16.789 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-24T08:05:16.790 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T08:05:16.790 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T08:05:16.790 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T08:05:16.790 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:05:16.790 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-24T08:05:16.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.791 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:16.845 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42353sec preferred_lft 42353sec 2023-11-24T08:05:16.846 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.847 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:05:16.848 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:05:16.848 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:05:16.848 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:16.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.849 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:05:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:05:16.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.940 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:05:16.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:05:16.992 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:17.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.173 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:05:17.221 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:05:17.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.223 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2023-11-24T08:05:17.271 INFO:teuthology.orchestra.run.smithi107.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-24T08:05:17.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.272 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:17.272 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:17.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:05:17.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-24T08:05:17.272 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:17.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T08:05:17.491 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:eb:a5:16:5f:1b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T08:05:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::78eb:a5ff:fe16:5f1b/64 scope link 2023-11-24T08:05:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:eb:a5:16:5f:1b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.589 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::78eb:a5ff:fe16:5f1b/64 scope link 2023-11-24T08:05:17.590 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:17.593 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-24T08:05:17.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-24T08:05:17.593 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:17.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:17.854 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:05:17.855 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:05:17.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.1 up 2023-11-24T08:05:17.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T08:05:17.855 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:05:18.020 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:18.020 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-24T08:05:18.068 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-24T08:05:18.069 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:18.069 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-24T08:05:18.118 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-11-24T08:05:18.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:18.255 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:18.256 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:18.292 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:18.292 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:18.292 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:05:18.292 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 2@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:05:18.293 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:05:18.294 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2023-11-24T08:05:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:05:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:05:18.295 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:05:18.295 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:05:18.299 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:05:18.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-24T08:05:18.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.381 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:18.436 INFO:teuthology.orchestra.run.smithi107.stdout:client4543 2023-11-24T08:05:18.437 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:18.437 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:18.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.489 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:client4543 2023-11-24T08:05:18.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:18.598 INFO:teuthology.orchestra.run.smithi107.stdout:client4543 2023-11-24T08:05:18.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:18.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.client4543/status') 2023-11-24T08:05:18.681 INFO:teuthology.orchestra.run.smithi107.stdout:instance: client.4543 (3)172.21.15.107:0/2869368859 2023-11-24T08:05:18.681 INFO:teuthology.orchestra.run.smithi107.stdout:blocklisted: false 2023-11-24T08:05:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records in 2023-11-24T08:05:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records out 2023-11-24T08:05:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:71 bytes copied, 6.9636e-05 s, 1.0 MB/s 2023-11-24T08:05:18.685 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-11-24T08:05:18.685 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T08:05:18.685 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:05:18.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:05:18.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.889+0000 7fd2d7dce640 1 Processor -- start 2023-11-24T08:05:18.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.890+0000 7fd2d7dce640 1 -- start start 2023-11-24T08:05:18.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d7dce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2d006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d7dce640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00cece0 con 0x7fd2d00cf450 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d7dce640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d006d040 con 0x7fd2d00ce6c0 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d7dce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0156010 con 0x7fd2d015c7f0 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d5369640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd2d00cf450 0x7fd2d00cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54636/0 (socket says 172.21.15.107:54636) 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.891+0000 7fd2d5369640 1 -- 172.21.15.107:0/1858419781 learned_addr learned my addr 172.21.15.107:0/1858419781 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2d006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762035406 0 0) 0x7fd2d00cece0 con 0x7fd2d00cf450 2023-11-24T08:05:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc003520 con 0x7fd2d00cf450 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400462500 0 0) 0x7fd2bc003520 con 0x7fd2d00cf450 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 msgr2=0x7fd2d006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2d006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 >> v1:172.21.15.120:6789/0 conn(0x7fd2d00ce6c0 legacy=0x7fd2d00714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d006d320 con 0x7fd2d00cf450 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd2c4001670 con 0x7fd2d00cf450 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 >> v1:172.21.15.107:6789/0 conn(0x7fd2d00cf450 legacy=0x7fd2d00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc0039a0 0x7fd2bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0047c0 0x7fd2bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.892+0000 7fd2d4b68640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0156010 con 0x7fd2bc0039a0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00ba30 con 0x7fd2bc0047c0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00bbf0 con 0x7fd2d015c7f0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc0039a0 0x7fd2bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 -- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 msgr2=0x7fd2bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d636b640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5369640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0047c0 0x7fd2bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 -- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0047c0 msgr2=0x7fd2bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0047c0 0x7fd2bc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d5b6a640 1 -- 172.21.15.107:0/1858419781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc003520 con 0x7fd2bc0039a0 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.893+0000 7fd2d636b640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d015c7f0 0x7fd2bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.894+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc0039a0 0x7fd2bc003dc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009a90 tx=0x7fd2bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa9df65d46f9dde server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.894+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc02c660 con 0x7fd2bc0039a0 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.894+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc02c7c0 con 0x7fd2bc0039a0 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d4b68640 1 -- 172.21.15.107:0/1858419781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc02cb50 con 0x7fd2bc0039a0 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 -- 172.21.15.107:0/1858419781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc0039a0 msgr2=0x7fd2bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/1858419781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc0039a0 0x7fd2bc003dc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009a90 tx=0x7fd2bc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d636b640 1 -- 172.21.15.107:0/1858419781 reap_dead start 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 -- 172.21.15.107:0/1858419781 shutdown_connections 2023-11-24T08:05:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 -- 172.21.15.107:0/1858419781 >> 172.21.15.107:0/1858419781 conn(0x7fd2d006a920 msgr2=0x7fd2d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 -- 172.21.15.107:0/1858419781 shutdown_connections 2023-11-24T08:05:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.895+0000 7fd2d7dce640 1 -- 172.21.15.107:0/1858419781 wait complete. 2023-11-24T08:05:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.897+0000 7fd2d7dce640 1 Processor -- start 2023-11-24T08:05:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.897+0000 7fd2d7dce640 1 -- start start 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 0x7fd2d00b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d01565d0 0x7fd2d0158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00af220 con 0x7fd2d0156150 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00b1780 con 0x7fd2bc0039a0 2023-11-24T08:05:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.898+0000 7fd2d7dce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01f3410 con 0x7fd2d01565d0 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46886/0 (socket says 172.21.15.107:46886) 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 -- 172.21.15.107:0/3897809349 learned_addr learned my addr 172.21.15.107:0/3897809349 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 0x7fd2d00b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d636b640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d01565d0 0x7fd2d0158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 -- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d01565d0 msgr2=0x7fd2d0158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d01565d0 0x7fd2d0158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 -- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 msgr2=0x7fd2d00b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 0x7fd2d00b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 -- 172.21.15.107:0/3897809349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00aeed0 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 0x7fd2d00b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.899+0000 7fd2d5369640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400d830 tx=0x7fd2c400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.900+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40074e0 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.900+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4007640 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.900+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40049e0 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.900+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d01587d0 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.900+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d015ad80 con 0x7fd2d0156150 2023-11-24T08:05:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.901+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d00af360 con 0x7fd2d0156150 2023-11-24T08:05:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.902+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd2c4004b40 con 0x7fd2d0156150 2023-11-24T08:05:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.903+0000 7fd2c27fc640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 0x7fd2b406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.903+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2c4088fb0 con 0x7fd2d0156150 2023-11-24T08:05:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.903+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 0x7fd2b406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.904+0000 7fd2d5b6a640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 0x7fd2b406d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:18.907+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2d00af360 con 0x7fd2d0156150 2023-11-24T08:05:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.136+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd2d00af360 con 0x7fd2d0156150 2023-11-24T08:05:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.137+0000 7fd2c27fc640 1 -- 172.21.15.107:0/3897809349 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd2d00af360 con 0x7fd2d0156150 2023-11-24T08:05:19.139 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:05:19.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 msgr2=0x7fd2b406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 0x7fd2b406d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc030000 comp rx=0 tx=0).stop 2023-11-24T08:05:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 msgr2=0x7fd2d0157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400d830 tx=0x7fd2c400dcf0 comp rx=0 tx=0).stop 2023-11-24T08:05:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 shutdown_connections 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2b406ae70 0x7fd2b406d330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2d01565d0 0x7fd2d0158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2d0156150 0x7fd2d0157d50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 --2- 172.21.15.107:0/3897809349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc0039a0 0x7fd2d00b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 >> 172.21.15.107:0/3897809349 conn(0x7fd2d006a920 msgr2=0x7fd2d00b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 shutdown_connections 2023-11-24T08:05:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:19.140+0000 7fd2d7dce640 1 -- 172.21.15.107:0/3897809349 wait complete. 2023-11-24T08:05:19.160 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T08:05:19.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:05:19.160 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:05:19.161 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:05:19.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:19.161 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:05:19.223 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:19.223 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:19.223 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:19.223 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:19.224 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:05:19.225 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:19.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:05:19.225 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42350sec preferred_lft 42350sec 2023-11-24T08:05:19.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:05:19.225 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:19.226 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T08:05:19.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:19.226 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:19.226 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:19.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add name ceph-brx type bridge 2023-11-24T08:05:19.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr flush dev ceph-brx 2023-11-24T08:05:19.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set ceph-brx up 2023-11-24T08:05:19.227 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T08:05:19.227 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:19.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:19.692 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:05:19.771 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:05:19.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:19.775 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:05:19.829 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:05:19.829 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:05:19.829 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:05:19.829 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:05:19.830 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:05:19.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:05:19.831 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:20.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:20.658 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:05:20.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:20.715 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2023-11-24T08:05:20.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:20.770 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:20.770 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:20.770 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:05:20.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-24T08:05:20.771 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:20.916 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-24T08:05:20.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:20.916 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:20.916 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-24T08:05:20.917 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:21.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:21.194 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:21.194 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:21.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:05:21.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T08:05:21.195 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:21.360 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:21.360 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:05:21.402 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:05:21.403 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:21.403 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:05:21.500 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:05:21.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:21.500 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:21.676 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:21.676 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:22.141 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:22.141 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:22.141 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:05:22.141 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:05:22.141 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:05:22.143 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:05:22.144 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:05:22.144 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:05:22.144 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:05:22.144 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:05:22.151 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:05:22.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:22.151 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:05:22.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:22.242 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:22.300 INFO:teuthology.orchestra.run.smithi120.stdout:client4677 2023-11-24T08:05:22.300 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:22.300 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:22.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:22.358 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:22.418 INFO:teuthology.orchestra.run.smithi120.stdout:client4677 2023-11-24T08:05:22.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:22.418 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:22.480 INFO:teuthology.orchestra.run.smithi120.stdout:client4677 2023-11-24T08:05:22.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:22.481 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.client4677/status') 2023-11-24T08:05:22.573 INFO:teuthology.orchestra.run.smithi120.stdout:instance: client.4677 (3)172.21.15.120:0/1358335024 2023-11-24T08:05:22.574 INFO:teuthology.orchestra.run.smithi120.stdout:blocklisted: false 2023-11-24T08:05:22.574 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2023-11-24T08:05:22.574 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2023-11-24T08:05:22.574 INFO:teuthology.orchestra.run.smithi120.stderr:71 bytes copied, 6.7683e-05 s, 1.0 MB/s 2023-11-24T08:05:22.578 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-11-24T08:05:22.578 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T08:05:22.579 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T08:05:22.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:05:22.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.804+0000 7f25f6516640 1 Processor -- start 2023-11-24T08:05:22.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.804+0000 7f25f6516640 1 -- start start 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.805+0000 7f25f6516640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25f0144b30 0x7f25f015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.805+0000 7f25f6516640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f00b1de0 0x7f25f00b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.805+0000 7f25f6516640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f015bf80 con 0x7f25f015c410 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.805+0000 7f25f6516640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f00b27b0 con 0x7f25f0144b30 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.805+0000 7f25f6516640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00b1560 con 0x7f25f00b1de0 2023-11-24T08:05:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f4d13640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25f015c410 0x7f25f00b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54652/0 (socket says 172.21.15.107:54652) 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f4d13640 1 -- 172.21.15.107:0/1291641984 learned_addr learned my addr 172.21.15.107:0/1291641984 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5514640 1 --2- 172.21.15.107:0/1291641984 >> v2:172.21.15.107:3300/0 conn(0x7f25f0144b30 0x7f25f015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5d15640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f00b1de0 0x7f25f00b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25e7fff640 1 -- 172.21.15.107:0/1291641984 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722953313 0 0) 0x7f25f015bf80 con 0x7f25f015c410 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25e7fff640 1 -- 172.21.15.107:0/1291641984 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc003520 con 0x7f25f015c410 2023-11-24T08:05:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5514640 1 -- 172.21.15.107:0/1291641984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f00b1de0 msgr2=0x7f25f00b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5514640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f00b1de0 0x7f25f00b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5514640 1 -- 172.21.15.107:0/1291641984 >> v1:172.21.15.107:6789/0 conn(0x7f25f015c410 legacy=0x7f25f00b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.806+0000 7f25f5514640 1 -- 172.21.15.107:0/1291641984 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f00b1280 con 0x7f25f0144b30 2023-11-24T08:05:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.807+0000 7f25f5514640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f0144b30 0x7f25f015f920 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0097a0 tx=0x7f25ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc35566232a22e38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25e7fff640 1 -- 172.21.15.107:0/1291641984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec031830 con 0x7f25f0144b30 2023-11-24T08:05:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25e7fff640 1 -- 172.21.15.107:0/1291641984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec031990 con 0x7f25f0144b30 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25e7fff640 1 -- 172.21.15.107:0/1291641984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec031d20 con 0x7f25f0144b30 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 -- 172.21.15.107:0/1291641984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f0144b30 msgr2=0x7f25f015f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f0144b30 0x7f25f015f920 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0097a0 tx=0x7f25ec02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 -- 172.21.15.107:0/1291641984 shutdown_connections 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f00b1de0 0x7f25f00b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 --2- 172.21.15.107:0/1291641984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f0144b30 0x7f25f015f920 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.808+0000 7f25f6516640 1 -- 172.21.15.107:0/1291641984 >> 172.21.15.107:0/1291641984 conn(0x7f25f00cb830 msgr2=0x7f25f00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.809+0000 7f25f6516640 1 -- 172.21.15.107:0/1291641984 shutdown_connections 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.809+0000 7f25f6516640 1 -- 172.21.15.107:0/1291641984 wait complete. 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.810+0000 7f25f6516640 1 Processor -- start 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.810+0000 7f25f6516640 1 -- start start 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f00b1de0 0x7f25f0142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 0x7f25f00c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f013c260 con 0x7f25f015c410 2023-11-24T08:05:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00719b0 con 0x7f25f00b1de0 2023-11-24T08:05:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.811+0000 7f25f6516640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f0070da0 con 0x7f25f0144b30 2023-11-24T08:05:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46914/0 (socket says 172.21.15.107:46914) 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 -- 172.21.15.107:0/1413221985 learned_addr learned my addr 172.21.15.107:0/1413221985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f4d13640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 0x7f25f00c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5514640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f00b1de0 0x7f25f0142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 -- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 msgr2=0x7f25f00c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 0x7f25f00c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 -- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f00b1de0 msgr2=0x7f25f0142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f00b1de0 0x7f25f0142fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:22.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f5d15640 1 -- 172.21.15.107:0/1413221985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009400 con 0x7f25f015c410 2023-11-24T08:05:22.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.812+0000 7f25f4d13640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 0x7f25f00c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:22.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25f5d15640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25e8007e40 tx=0x7f25e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:22.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8014620 con 0x7f25f015c410 2023-11-24T08:05:22.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e800ee90 con 0x7f25f015c410 2023-11-24T08:05:22.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f0071080 con 0x7f25f015c410 2023-11-24T08:05:22.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e80149e0 con 0x7f25f015c410 2023-11-24T08:05:22.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.813+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f0071640 con 0x7f25f015c410 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.814+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f00b3610 con 0x7f25f015c410 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.815+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f25e8014b40 con 0x7f25f015c410 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.816+0000 7f25e5ffb640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 0x7f25d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.816+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25e808a060 con 0x7f25f015c410 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.816+0000 7f25f5514640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 0x7f25d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.817+0000 7f25f5514640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 0x7f25d406d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25ec036000 tx=0x7f25ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:22.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:22.821+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25e80bce90 con 0x7f25f015c410 2023-11-24T08:05:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.052+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f25f004e840 con 0x7f25f015c410 2023-11-24T08:05:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.052+0000 7f25e5ffb640 1 -- 172.21.15.107:0/1413221985 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f25f00b3610 con 0x7f25f015c410 2023-11-24T08:05:23.054 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:05:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 msgr2=0x7f25d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 0x7f25d406d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25ec036000 tx=0x7f25ec008030 comp rx=0 tx=0).stop 2023-11-24T08:05:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 msgr2=0x7f25f00c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25e8007e40 tx=0x7f25e800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 shutdown_connections 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d406ae70 0x7f25d406d330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f015c410 0x7f25f00c86a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f0144b30 0x7f25f00c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 --2- 172.21.15.107:0/1413221985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f00b1de0 0x7f25f0142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 >> 172.21.15.107:0/1413221985 conn(0x7f25f00cb830 msgr2=0x7f25f015dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.055+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 shutdown_connections 2023-11-24T08:05:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:23.056+0000 7f25f6516640 1 -- 172.21.15.107:0/1413221985 wait complete. 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:05:23.076 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-24T08:05:23.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.077 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:05:23.138 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:23.138 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:05:23.139 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42346sec preferred_lft 42346sec 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:05:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.141 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:05:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:05:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:05:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.142 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:05:23.231 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:05:23.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.238 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:05:23.292 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:05:23.292 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:05:23.292 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:05:23.292 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:05:23.292 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:05:23.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.293 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:23.294 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:23.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:05:23.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:05:23.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:05:23.294 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:23.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.480 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:05:23.530 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:05:23.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.531 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2023-11-24T08:05:23.581 INFO:teuthology.orchestra.run.smithi120.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-24T08:05:23.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.582 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:23.582 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:23.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:05:23.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-24T08:05:23.583 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:23.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.722 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.819 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:23.820 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 9a:ed:a9:fd:17:95 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T08:05:23.820 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T08:05:23.820 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.820 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::98ed:a9ff:fefd:1795/64 scope link 2023-11-24T08:05:23.820 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T08:05:23.921 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:05:23.921 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 9a:ed:a9:fd:17:95 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::98ed:a9ff:fefd:1795/64 scope link 2023-11-24T08:05:23.922 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:05:23.926 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-24T08:05:23.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:23.926 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:23.926 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:23.926 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-24T08:05:23.926 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T08:05:23.926 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-24T08:05:23.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-24T08:05:23.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-24T08:05:23.927 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:24.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.193 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:05:24.193 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:05:24.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.1 up 2023-11-24T08:05:24.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T08:05:24.193 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:05:24.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:24.363 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-24T08:05:24.411 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-24T08:05:24.411 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:05:24.411 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-24T08:05:24.457 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-11-24T08:05:24.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.458 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:24.631 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:24.631 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:24.668 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:05:24.668 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:05:24.668 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:05:24.668 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 3@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:05:24.668 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:05:24.671 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2023-11-24T08:05:24.672 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:05:24.672 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:05:24.672 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:05:24.672 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:05:24.677 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:05:24.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.678 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-24T08:05:24.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.773 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:24.831 INFO:teuthology.orchestra.run.smithi120.stdout:client4552 2023-11-24T08:05:24.832 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:24.832 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:24.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.884 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:24.942 INFO:teuthology.orchestra.run.smithi120.stdout:client4552 2023-11-24T08:05:24.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.942 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:05:24.998 INFO:teuthology.orchestra.run.smithi120.stdout:client4552 2023-11-24T08:05:24.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:24.999 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.client4552/status') 2023-11-24T08:05:25.090 INFO:teuthology.orchestra.run.smithi120.stdout:instance: client.4552 (3)192.168.0.2:0/2533364356 2023-11-24T08:05:25.091 INFO:teuthology.orchestra.run.smithi120.stdout:blocklisted: false 2023-11-24T08:05:25.091 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2023-11-24T08:05:25.091 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2023-11-24T08:05:25.091 INFO:teuthology.orchestra.run.smithi120.stderr:69 bytes copied, 7.4236e-05 s, 929 kB/s 2023-11-24T08:05:25.094 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-24T08:05:25.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.329+0000 7fd531407640 1 Processor -- start 2023-11-24T08:05:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.329+0000 7fd531407640 1 -- start start 2023-11-24T08:05:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd531407640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd52c06aa30 0x7fd52c0cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd531407640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd52c0b28d0 0x7fd52c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd531407640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52c0b2690 con 0x7fd52c0cf420 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd531407640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c0cc7f0 con 0x7fd52c06aa30 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd531407640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c06ed00 con 0x7fd52c0b28d0 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd52affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd52c06aa30 0x7fd52c0cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd52affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd52c06aa30 0x7fd52c0cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46922/0 (socket says 172.21.15.107:46922) 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.331+0000 7fd52affd640 1 -- 172.21.15.107:0/1097527668 learned_addr learned my addr 172.21.15.107:0/1097527668 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.332+0000 7fd52affd640 1 -- 172.21.15.107:0/1097527668 >> v2:172.21.15.120:3300/0 conn(0x7fd52c0b28d0 msgr2=0x7fd52c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.332+0000 7fd52affd640 1 --2- 172.21.15.107:0/1097527668 >> v2:172.21.15.120:3300/0 conn(0x7fd52c0b28d0 0x7fd52c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.332+0000 7fd52affd640 1 -- 172.21.15.107:0/1097527668 >> v1:172.21.15.120:6789/0 conn(0x7fd52c0cf420 legacy=0x7fd52c1616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.332+0000 7fd52affd640 1 -- 172.21.15.107:0/1097527668 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c06efe0 con 0x7fd52c06aa30 2023-11-24T08:05:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.333+0000 7fd52affd640 1 --2- 172.21.15.107:0/1097527668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0cc2b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0d0080 tx=0x7fd51c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c68c43fb2b9e35 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.333+0000 7fd529ffb640 1 -- 172.21.15.107:0/1097527668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c030860 con 0x7fd52c06aa30 2023-11-24T08:05:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.333+0000 7fd529ffb640 1 -- 172.21.15.107:0/1097527668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c0309c0 con 0x7fd52c06aa30 2023-11-24T08:05:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.333+0000 7fd529ffb640 1 -- 172.21.15.107:0/1097527668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c030d50 con 0x7fd52c06aa30 2023-11-24T08:05:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 -- 172.21.15.107:0/1097527668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 msgr2=0x7fd52c0cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 --2- 172.21.15.107:0/1097527668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0cc2b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0d0080 tx=0x7fd51c02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 -- 172.21.15.107:0/1097527668 shutdown_connections 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 --2- 172.21.15.107:0/1097527668 >> v2:172.21.15.120:3300/0 conn(0x7fd52c0b28d0 0x7fd52c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 --2- 172.21.15.107:0/1097527668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0cc2b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 -- 172.21.15.107:0/1097527668 >> 172.21.15.107:0/1097527668 conn(0x7fd52c0999d0 msgr2=0x7fd52c0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 -- 172.21.15.107:0/1097527668 shutdown_connections 2023-11-24T08:05:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.334+0000 7fd531407640 1 -- 172.21.15.107:0/1097527668 wait complete. 2023-11-24T08:05:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.336+0000 7fd531407640 1 Processor -- start 2023-11-24T08:05:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.336+0000 7fd531407640 1 -- start start 2023-11-24T08:05:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd52c0b28d0 0x7fd52c0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 0x7fd52c1f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c1f4620 con 0x7fd52c06aa30 2023-11-24T08:05:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0b3d60 con 0x7fd52c0cf420 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd531407640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0b4040 con 0x7fd52c0b28d0 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46932/0 (socket says 172.21.15.107:46932) 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 -- 172.21.15.107:0/1997777192 learned_addr learned my addr 172.21.15.107:0/1997777192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 -- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd52c0b28d0 msgr2=0x7fd52c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd52c0b28d0 0x7fd52c0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52affd640 1 -- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 msgr2=0x7fd52c1f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.337+0000 7fd52b7fe640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 0x7fd52c1f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd52affd640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 0x7fd52c1f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd52affd640 1 -- 172.21.15.107:0/1997777192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c0094d0 con 0x7fd52c06aa30 2023-11-24T08:05:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd52b7fe640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 0x7fd52c1f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd52affd640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd51c000c00 tx=0x7fd51c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c030810 con 0x7fd52c06aa30 2023-11-24T08:05:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c043400 con 0x7fd52c06aa30 2023-11-24T08:05:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0437b0 con 0x7fd52c06aa30 2023-11-24T08:05:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c0b4320 con 0x7fd52c06aa30 2023-11-24T08:05:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.338+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c0b50e0 con 0x7fd52c06aa30 2023-11-24T08:05:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.339+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c067470 con 0x7fd52c06aa30 2023-11-24T08:05:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.340+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd51c043910 con 0x7fd52c06aa30 2023-11-24T08:05:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.341+0000 7fd5137fe640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 0x7fd51806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.341+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd51c03e070 con 0x7fd52c06aa30 2023-11-24T08:05:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.342+0000 7fd52a7fc640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 0x7fd51806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.342+0000 7fd52a7fc640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 0x7fd51806d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0b4a80 tx=0x7fd520006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.346+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd51c077b90 con 0x7fd52c06aa30 2023-11-24T08:05:25.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.541+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd52c071940 con 0x7fd52c06aa30 2023-11-24T08:05:25.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.543+0000 7fd5137fe640 1 -- 172.21.15.107:0/1997777192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd51c07b1b0 con 0x7fd52c06aa30 2023-11-24T08:05:25.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:25.546 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:25.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 msgr2=0x7fd51806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 0x7fd51806d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0b4a80 tx=0x7fd520006c90 comp rx=0 tx=0).stop 2023-11-24T08:05:25.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 msgr2=0x7fd52c0b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd51c000c00 tx=0x7fd51c004730 comp rx=0 tx=0).stop 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 shutdown_connections 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806ae40 0x7fd51806d300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c0cf420 0x7fd52c1f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd52c0b28d0 0x7fd52c0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 --2- 172.21.15.107:0/1997777192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd52c06aa30 0x7fd52c0b3780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 >> 172.21.15.107:0/1997777192 conn(0x7fd52c0999d0 msgr2=0x7fd52c0af200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:25.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 shutdown_connections 2023-11-24T08:05:25.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.546+0000 7fd531407640 1 -- 172.21.15.107:0/1997777192 wait complete. 2023-11-24T08:05:25.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:25.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:25.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.783+0000 7f1a91341640 1 Processor -- start 2023-11-24T08:05:25.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.783+0000 7f1a91341640 1 -- start start 2023-11-24T08:05:25.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.785+0000 7f1a91341640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a8c140340 0x7f1a8c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.785+0000 7f1a91341640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c161600 con 0x7f1a8c140720 2023-11-24T08:05:25.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.785+0000 7f1a91341640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c1618e0 con 0x7f1a8c161080 2023-11-24T08:05:25.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.785+0000 7f1a91341640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c0cbf70 con 0x7f1a8c140340 2023-11-24T08:05:25.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.785+0000 7f1a83fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1a8c140720 0x7f1a8c0cb860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54666/0 (socket says 172.21.15.107:54666) 2023-11-24T08:05:25.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a83fff640 1 -- 172.21.15.107:0/3442883539 learned_addr learned my addr 172.21.15.107:0/3442883539 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:25.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894699171 0 0) 0x7f1a8c161600 con 0x7f1a8c140720 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a78003520 con 0x7f1a8c140720 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8affd640 1 --2- 172.21.15.107:0/3442883539 >> v2:172.21.15.120:3300/0 conn(0x7f1a8c140340 0x7f1a8c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584246378 0 0) 0x7f1a78003520 con 0x7f1a8c140720 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 >> v2:172.21.15.120:3300/0 conn(0x7f1a8c140340 msgr2=0x7f1a8c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 --2- 172.21.15.107:0/3442883539 >> v2:172.21.15.120:3300/0 conn(0x7f1a8c140340 0x7f1a8c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 >> v1:172.21.15.120:6789/0 conn(0x7f1a8c161080 legacy=0x7f1a8c0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c161bc0 con 0x7f1a8c140720 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1a74001670 con 0x7f1a8c140720 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 >> v1:172.21.15.107:6789/0 conn(0x7f1a8c140720 legacy=0x7f1a8c0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.786+0000 7f1a8a7fc640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780039c0 0x7f1a78003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8a7fc640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8a7fc640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a780051d0 0x7f1a7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0cbf70 con 0x7f1a780047c0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7800c120 con 0x7f1a780051d0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7800c2e0 con 0x7f1a780039c0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8b7fe640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a780051d0 0x7f1a7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 -- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780039c0 msgr2=0x7f1a78003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780039c0 0x7f1a78003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 -- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a780051d0 msgr2=0x7f1a7800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a780051d0 0x7f1a7800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a83fff640 1 -- 172.21.15.107:0/3442883539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a78003520 con 0x7f1a780047c0 2023-11-24T08:05:25.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8b7fe640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a780051d0 0x7f1a7800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.787+0000 7f1a8b7fe640 1 -- 172.21.15.107:0/3442883539 reap_dead start 2023-11-24T08:05:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.788+0000 7f1a83fff640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c0e0 tx=0x7f1a780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c497669c4f87c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74005ad0 con 0x7f1a780047c0 2023-11-24T08:05:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74005c30 con 0x7f1a780047c0 2023-11-24T08:05:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a8a7fc640 1 -- 172.21.15.107:0/3442883539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7402ca50 con 0x7f1a780047c0 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 -- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 msgr2=0x7f1a78004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c0e0 tx=0x7f1a780057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 -- 172.21.15.107:0/3442883539 shutdown_connections 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 --2- 172.21.15.107:0/3442883539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 -- 172.21.15.107:0/3442883539 >> 172.21.15.107:0/3442883539 conn(0x7f1a8c0b2e40 msgr2=0x7f1a8c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.789+0000 7f1a91341640 1 -- 172.21.15.107:0/3442883539 shutdown_connections 2023-11-24T08:05:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.790+0000 7f1a91341640 1 -- 172.21.15.107:0/3442883539 wait complete. 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.791+0000 7f1a91341640 1 Processor -- start 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.791+0000 7f1a91341640 1 -- start start 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780047c0 0x7f1a8c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8c15f240 0x7f1a8c1f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c15e870 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c1f4e70 con 0x7f1a8c15f240 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.792+0000 7f1a91341640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c15dc60 con 0x7f1a780047c0 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46942/0 (socket says 172.21.15.107:46942) 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 -- 172.21.15.107:0/3453709792 learned_addr learned my addr 172.21.15.107:0/3453709792 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 -- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780047c0 msgr2=0x7f1a8c15ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780047c0 0x7f1a8c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 -- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8c15f240 msgr2=0x7f1a8c1f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8c15f240 0x7f1a8c1f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 -- 172.21.15.107:0/3453709792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c15df40 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.793+0000 7f1a8b7fe640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c140b90 tx=0x7f1a8400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.794+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84003d80 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.794+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84005070 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.794+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84005420 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.794+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c15e220 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.794+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c0ca2c0 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.795+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c04e8b0 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.796+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1a84005580 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.797+0000 7f1a837fe640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 0x7f1a6c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.797+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a84088eb0 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.797+0000 7f1a8affd640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 0x7f1a6c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.798+0000 7f1a8affd640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 0x7f1a6c06d350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c15f6e0 tx=0x7f1a7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.802+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a84053f40 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.991+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a8c15e9b0 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.992+0000 7f1a837fe640 1 -- 172.21.15.107:0/3453709792 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a84018070 con 0x7f1a8c1f49e0 2023-11-24T08:05:25.994 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:25.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 msgr2=0x7f1a6c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 0x7f1a6c06d350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c15f6e0 tx=0x7f1a7c008040 comp rx=0 tx=0).stop 2023-11-24T08:05:25.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 msgr2=0x7f1a8c0c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:25.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c140b90 tx=0x7f1a8400d9b0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 shutdown_connections 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06ae90 0x7f1a6c06d350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8c1f49e0 0x7f1a8c0c9d80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8c15f240 0x7f1a8c1f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 --2- 172.21.15.107:0/3453709792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a780047c0 0x7f1a8c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 >> 172.21.15.107:0/3453709792 conn(0x7f1a8c0b2e40 msgr2=0x7f1a8c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.995+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 shutdown_connections 2023-11-24T08:05:25.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:25.996+0000 7f1a91341640 1 -- 172.21.15.107:0/3453709792 wait complete. 2023-11-24T08:05:26.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.225+0000 7f601a7e9640 1 Processor -- start 2023-11-24T08:05:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.226+0000 7f601a7e9640 1 -- start start 2023-11-24T08:05:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.227+0000 7f601a7e9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f601415b260 0x7f601406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.227+0000 7f601a7e9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f601406e580 0x7f60140b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.227+0000 7f601a7e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601406d5c0 con 0x7f60140b1080 2023-11-24T08:05:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.227+0000 7f601a7e9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f601415b640 con 0x7f601415b260 2023-11-24T08:05:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.227+0000 7f601a7e9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60140b14d0 con 0x7f601406e580 2023-11-24T08:05:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f60197e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f601415b260 0x7f601406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f6019fe8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f60140b1080 0x7f60140ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54674/0 (socket says 172.21.15.107:54674) 2023-11-24T08:05:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f6019fe8640 1 -- 172.21.15.107:0/1421196560 learned_addr learned my addr 172.21.15.107:0/1421196560 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f6018fe6640 1 --2- 172.21.15.107:0/1421196560 >> v2:172.21.15.120:3300/0 conn(0x7f601406e580 0x7f60140b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f600bfff640 1 -- 172.21.15.107:0/1421196560 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420942301 0 0) 0x7f601406d5c0 con 0x7f60140b1080 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f600bfff640 1 -- 172.21.15.107:0/1421196560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000003520 con 0x7f60140b1080 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f60197e7640 1 -- 172.21.15.107:0/1421196560 >> v2:172.21.15.120:3300/0 conn(0x7f601406e580 msgr2=0x7f60140b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f60197e7640 1 --2- 172.21.15.107:0/1421196560 >> v2:172.21.15.120:3300/0 conn(0x7f601406e580 0x7f60140b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f60197e7640 1 -- 172.21.15.107:0/1421196560 >> v1:172.21.15.107:6789/0 conn(0x7f60140b1080 legacy=0x7f60140ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.228+0000 7f60197e7640 1 -- 172.21.15.107:0/1421196560 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601415ed20 con 0x7f601415b260 2023-11-24T08:05:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.230+0000 7f60197e7640 1 --2- 172.21.15.107:0/1421196560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f601415b260 0x7f601406e040 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f60140ce370 tx=0x7f601002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44febb2f4b6c75f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.230+0000 7f600bfff640 1 -- 172.21.15.107:0/1421196560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010031710 con 0x7f601415b260 2023-11-24T08:05:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.230+0000 7f600bfff640 1 -- 172.21.15.107:0/1421196560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601002f050 con 0x7f601415b260 2023-11-24T08:05:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.230+0000 7f600bfff640 1 -- 172.21.15.107:0/1421196560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010031a90 con 0x7f601415b260 2023-11-24T08:05:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 -- 172.21.15.107:0/1421196560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f601415b260 msgr2=0x7f601406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 --2- 172.21.15.107:0/1421196560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f601415b260 0x7f601406e040 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f60140ce370 tx=0x7f601002b7b0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 -- 172.21.15.107:0/1421196560 shutdown_connections 2023-11-24T08:05:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 --2- 172.21.15.107:0/1421196560 >> v2:172.21.15.120:3300/0 conn(0x7f601406e580 0x7f60140b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 --2- 172.21.15.107:0/1421196560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f601415b260 0x7f601406e040 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 -- 172.21.15.107:0/1421196560 >> 172.21.15.107:0/1421196560 conn(0x7f60140cbb40 msgr2=0x7f60140cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 -- 172.21.15.107:0/1421196560 shutdown_connections 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.231+0000 7f601a7e9640 1 -- 172.21.15.107:0/1421196560 wait complete. 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.233+0000 7f601a7e9640 1 Processor -- start 2023-11-24T08:05:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.233+0000 7f601a7e9640 1 -- start start 2023-11-24T08:05:26.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f601406e580 0x7f6014149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f601415b260 0x7f6014159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601414a5d0 con 0x7f60140b1080 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601414a8b0 con 0x7f601415b260 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f601a7e9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601414ab90 con 0x7f601406e580 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f6018fe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f6018fe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46958/0 (socket says 172.21.15.107:46958) 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.234+0000 7f6018fe6640 1 -- 172.21.15.107:0/3205283607 learned_addr learned my addr 172.21.15.107:0/3205283607 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 -- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f601406e580 msgr2=0x7f6014149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f60197e7640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f601406e580 0x7f6014149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6019fe8640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f601415b260 0x7f6014159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f601406e580 0x7f6014149da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 -- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f601415b260 msgr2=0x7f6014159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f601415b260 0x7f6014159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 -- 172.21.15.107:0/3205283607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010009400 con 0x7f60140b1080 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6018fe6640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f600400d5f0 tx=0x7f600400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60040125f0 con 0x7f60140b1080 2023-11-24T08:05:26.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6004012750 con 0x7f60140b1080 2023-11-24T08:05:26.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004012b00 con 0x7f60140b1080 2023-11-24T08:05:26.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601414ae70 con 0x7f60140b1080 2023-11-24T08:05:26.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.235+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601414a2e0 con 0x7f60140b1080 2023-11-24T08:05:26.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.237+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601404e840 con 0x7f60140b1080 2023-11-24T08:05:26.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.238+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6004012c60 con 0x7f60140b1080 2023-11-24T08:05:26.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.238+0000 7f6009ffb640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 0x7f5ff806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.239+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6004089b00 con 0x7f60140b1080 2023-11-24T08:05:26.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.239+0000 7f60197e7640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 0x7f5ff806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.239+0000 7f60197e7640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 0x7f5ff806d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6010035000 tx=0x7f6010008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.243+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6004054bb0 con 0x7f60140b1080 2023-11-24T08:05:26.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.433+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f601414a710 con 0x7f60140b1080 2023-11-24T08:05:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.433+0000 7f6009ffb640 1 -- 172.21.15.107:0/3205283607 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f601414a710 con 0x7f60140b1080 2023-11-24T08:05:26.436 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 msgr2=0x7f5ff806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 0x7f5ff806d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6010035000 tx=0x7f6010008030 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 msgr2=0x7f601414b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f600400d5f0 tx=0x7f600400dab0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 shutdown_connections 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ff806ae40 0x7f5ff806d300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f601415b260 0x7f6014159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60140b1080 0x7f601414b720 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 --2- 172.21.15.107:0/3205283607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f601406e580 0x7f6014149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.436+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 >> 172.21.15.107:0/3205283607 conn(0x7f60140cbb40 msgr2=0x7f60140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.437+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 shutdown_connections 2023-11-24T08:05:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.437+0000 7f601a7e9640 1 -- 172.21.15.107:0/3205283607 wait complete. 2023-11-24T08:05:26.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:26.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.676+0000 7f824b083640 1 Processor -- start 2023-11-24T08:05:26.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.676+0000 7f824b083640 1 -- start start 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f824b083640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 0x7f824415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f824b083640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82440c49f0 0x7f82440705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f824b083640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824415ec90 con 0x7f8244072160 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f824b083640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440cd530 con 0x7f82440c49f0 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f824b083640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c4ed0 con 0x7f824406de10 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f8248e1f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8244072160 0x7f824406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54686/0 (socket says 172.21.15.107:54686) 2023-11-24T08:05:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.678+0000 7f8248e1f640 1 -- 172.21.15.107:0/204497224 learned_addr learned my addr 172.21.15.107:0/204497224 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f8249620640 1 --2- 172.21.15.107:0/204497224 >> v2:172.21.15.120:3300/0 conn(0x7f82440c49f0 0x7f82440705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f8243fff640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 0x7f824415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649279083 0 0) 0x7f8238001110 con 0x7f8244072160 2023-11-24T08:05:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234003520 con 0x7f8244072160 2023-11-24T08:05:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462152860 0 0) 0x7f8234003520 con 0x7f8244072160 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 msgr2=0x7f824415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 0x7f824415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 >> v2:172.21.15.120:3300/0 conn(0x7f82440c49f0 msgr2=0x7f82440705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 --2- 172.21.15.107:0/204497224 >> v2:172.21.15.120:3300/0 conn(0x7f82440c49f0 0x7f82440705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440c51b0 con 0x7f8244072160 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8238001800 con 0x7f8244072160 2023-11-24T08:05:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 >> v1:172.21.15.107:6789/0 conn(0x7f8244072160 legacy=0x7f824406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8234003970 0x7f8234003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82340047c0 0x7f8234004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 0x7f8234005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440cd530 con 0x7f8234003970 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c4ed0 con 0x7f82340047c0 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.679+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823400b980 con 0x7f824406de10 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8234003970 0x7f8234003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 -- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 msgr2=0x7f8234005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824406de10 0x7f8234005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 -- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82340047c0 msgr2=0x7f8234004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82340047c0 0x7f8234004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.680+0000 7f8248e1f640 1 -- 172.21.15.107:0/204497224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234003520 con 0x7f8234003970 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.681+0000 7f8248e1f640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8234003970 0x7f8234003d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f823800c110 tx=0x7f82340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457845d3208075d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.681+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238005ad0 con 0x7f8234003970 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.681+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238005c50 con 0x7f8234003970 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.681+0000 7f82437fe640 1 -- 172.21.15.107:0/204497224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823802c7f0 con 0x7f8234003970 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 -- 172.21.15.107:0/204497224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8234003970 msgr2=0x7f8234003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 --2- 172.21.15.107:0/204497224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8234003970 0x7f8234003d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f823800c110 tx=0x7f82340057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f8249620640 1 -- 172.21.15.107:0/204497224 reap_dead start 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 -- 172.21.15.107:0/204497224 shutdown_connections 2023-11-24T08:05:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 -- 172.21.15.107:0/204497224 >> 172.21.15.107:0/204497224 conn(0x7f82440b0c50 msgr2=0x7f82440b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 -- 172.21.15.107:0/204497224 shutdown_connections 2023-11-24T08:05:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.682+0000 7f824b083640 1 -- 172.21.15.107:0/204497224 wait complete. 2023-11-24T08:05:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.684+0000 7f824b083640 1 Processor -- start 2023-11-24T08:05:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.684+0000 7f824b083640 1 -- start start 2023-11-24T08:05:26.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8234003970 0x7f82441f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824409c380 0x7f824414ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c8b40 con 0x7f824414f1a0 2023-11-24T08:05:26.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82441f6640 con 0x7f824409c380 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f824b083640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c7b00 con 0x7f8234003970 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f8249620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f8249620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46968/0 (socket says 172.21.15.107:46968) 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.685+0000 7f8249620640 1 -- 172.21.15.107:0/12113306 learned_addr learned my addr 172.21.15.107:0/12113306 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8243fff640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824409c380 0x7f824414ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8248e1f640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8234003970 0x7f82441f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 -- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8234003970 msgr2=0x7f82441f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8234003970 0x7f82441f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 -- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824409c380 msgr2=0x7f824414ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824409c380 0x7f824414ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 -- 172.21.15.107:0/12113306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440c7d80 con 0x7f824414f1a0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f8249620640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82440b51a0 tx=0x7f823c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c010040 con 0x7f824414f1a0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c00bae0 con 0x7f824414f1a0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c00be70 con 0x7f824414f1a0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82440c8060 con 0x7f824414f1a0 2023-11-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.686+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82440c8620 con 0x7f824414f1a0 2023-11-24T08:05:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.688+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824404e8b0 con 0x7f824414f1a0 2023-11-24T08:05:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.689+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f823c01e410 con 0x7f824414f1a0 2023-11-24T08:05:26.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.689+0000 7f82417fa640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 0x7f822806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:26.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.690+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f823c089810 con 0x7f824414f1a0 2023-11-24T08:05:26.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.690+0000 7f8248e1f640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 0x7f822806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:26.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.690+0000 7f8248e1f640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 0x7f822806d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8238034000 tx=0x7f8238007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:26.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.694+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f823c058350 con 0x7f824414f1a0 2023-11-24T08:05:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.887+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f824415e720 con 0x7f824414f1a0 2023-11-24T08:05:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.888+0000 7f82417fa640 1 -- 172.21.15.107:0/12113306 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f823c057ee0 con 0x7f824414f1a0 2023-11-24T08:05:26.890 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.890+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 msgr2=0x7f822806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.890+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 0x7f822806d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8238034000 tx=0x7f8238007000 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.890+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 msgr2=0x7f824414f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.890+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82440b51a0 tx=0x7f823c00b480 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 shutdown_connections 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f822806ae70 0x7f822806d330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f824414f1a0 0x7f824414f620 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824409c380 0x7f824414ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 --2- 172.21.15.107:0/12113306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8234003970 0x7f82441f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 >> 172.21.15.107:0/12113306 conn(0x7f82440b0c50 msgr2=0x7f82440b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 shutdown_connections 2023-11-24T08:05:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:26.891+0000 7f824b083640 1 -- 172.21.15.107:0/12113306 wait complete. 2023-11-24T08:05:26.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.118+0000 7f5a4d2bb640 1 Processor -- start 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.118+0000 7f5a4d2bb640 1 -- start start 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4d2bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a48144b30 0x7f5a4815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4d2bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4815c410 0x7f5a480b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4d2bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4815bf80 con 0x7f5a480b1de0 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4d2bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a480b1560 con 0x7f5a48144b30 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4d2bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480b0fb0 con 0x7f5a4815c410 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a47fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a48144b30 0x7f5a4815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4caba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5a480b1de0 0x7f5a4806c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54700/0 (socket says 172.21.15.107:54700) 2023-11-24T08:05:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a4caba640 1 -- 172.21.15.107:0/491932796 learned_addr learned my addr 172.21.15.107:0/491932796 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.120+0000 7f5a477fe640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4815c410 0x7f5a480b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.121+0000 7f5a47fff640 1 -- 172.21.15.107:0/491932796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4815c410 msgr2=0x7f5a480b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.121+0000 7f5a47fff640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4815c410 0x7f5a480b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.121+0000 7f5a47fff640 1 -- 172.21.15.107:0/491932796 >> v1:172.21.15.107:6789/0 conn(0x7f5a480b1de0 legacy=0x7f5a4806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.121+0000 7f5a47fff640 1 -- 172.21.15.107:0/491932796 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a480b1280 con 0x7f5a48144b30 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.122+0000 7f5a47fff640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a4815f920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a380097a0 tx=0x7f5a3802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1692efd5b9a5a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.122+0000 7f5a46ffd640 1 -- 172.21.15.107:0/491932796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3802d040 con 0x7f5a48144b30 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.122+0000 7f5a46ffd640 1 -- 172.21.15.107:0/491932796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a38031810 con 0x7f5a48144b30 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.122+0000 7f5a46ffd640 1 -- 172.21.15.107:0/491932796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38031ba0 con 0x7f5a48144b30 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/491932796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 msgr2=0x7f5a4815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a4815f920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a380097a0 tx=0x7f5a3802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/491932796 shutdown_connections 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4815c410 0x7f5a480b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/491932796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a4815f920 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/491932796 >> 172.21.15.107:0/491932796 conn(0x7f5a480cb830 msgr2=0x7f5a480cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/491932796 shutdown_connections 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.123+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/491932796 wait complete. 2023-11-24T08:05:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.125+0000 7f5a4d2bb640 1 Processor -- start 2023-11-24T08:05:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.125+0000 7f5a4d2bb640 1 -- start start 2023-11-24T08:05:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 0x7f5a4813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a4815c410 0x7f5a480ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481418a0 con 0x7f5a48144b30 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48141b80 con 0x7f5a4815c410 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a4d2bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48141e60 con 0x7f5a480b1de0 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46982/0 (socket says 172.21.15.107:46982) 2023-11-24T08:05:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.126+0000 7f5a477fe640 1 -- 172.21.15.107:0/2103587122 learned_addr learned my addr 172.21.15.107:0/2103587122 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a4caba640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a4815c410 0x7f5a480ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a47fff640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 0x7f5a4813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 -- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 msgr2=0x7f5a4813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 0x7f5a4813e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 -- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a4815c410 msgr2=0x7f5a480ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a4815c410 0x7f5a480ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 -- 172.21.15.107:0/2103587122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38009400 con 0x7f5a48144b30 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a47fff640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 0x7f5a4813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a477fe640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e870 tx=0x7f5a3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0094f0 con 0x7f5a48144b30 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.127+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00c940 con 0x7f5a48144b30 2023-11-24T08:05:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.128+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a48142140 con 0x7f5a48144b30 2023-11-24T08:05:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.128+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00ccf0 con 0x7f5a48144b30 2023-11-24T08:05:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.128+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4813e710 con 0x7f5a48144b30 2023-11-24T08:05:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.129+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4804e840 con 0x7f5a48144b30 2023-11-24T08:05:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.130+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5a3c013470 con 0x7f5a48144b30 2023-11-24T08:05:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.131+0000 7f5a44ff9640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.132+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a3c0897d0 con 0x7f5a48144b30 2023-11-24T08:05:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.132+0000 7f5a47fff640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.132+0000 7f5a47fff640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 0x7f5a3406d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5a38036000 tx=0x7f5a3803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.137+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a3c0547e0 con 0x7f5a48144b30 2023-11-24T08:05:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.329+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a4806d860 con 0x7f5a48144b30 2023-11-24T08:05:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.333+0000 7f5a44ff9640 1 -- 172.21.15.107:0/2103587122 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f5a3c057e00 con 0x7f5a48144b30 2023-11-24T08:05:27.335 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:27.335 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 msgr2=0x7f5a3406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 0x7f5a3406d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5a38036000 tx=0x7f5a3803c040 comp rx=0 tx=0).stop 2023-11-24T08:05:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 msgr2=0x7f5a481f2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e870 tx=0x7f5a3c00ed30 comp rx=0 tx=0).stop 2023-11-24T08:05:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 shutdown_connections 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a4815c410 0x7f5a480ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a48144b30 0x7f5a481f2ef0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 --2- 172.21.15.107:0/2103587122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a480b1de0 0x7f5a4813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 >> 172.21.15.107:0/2103587122 conn(0x7f5a480cb830 msgr2=0x7f5a4815d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 shutdown_connections 2023-11-24T08:05:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.336+0000 7f5a4d2bb640 1 -- 172.21.15.107:0/2103587122 wait complete. 2023-11-24T08:05:27.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:27.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.565+0000 7f6f20856640 1 Processor -- start 2023-11-24T08:05:27.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.566+0000 7f6f20856640 1 -- start start 2023-11-24T08:05:27.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.567+0000 7f6f20856640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 0x7f6f1c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.567+0000 7f6f20856640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6f1c0c59b0 0x7f6f1c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.567+0000 7f6f20856640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1c0cc3a0 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.567+0000 7f6f20856640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1c15b820 con 0x7f6f1c0c59b0 2023-11-24T08:05:27.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.567+0000 7f6f20856640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c15eea0 con 0x7f6f1c15b440 2023-11-24T08:05:27.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f19d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f1c0cbf20 0x7f6f1c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54702/0 (socket says 172.21.15.107:54702) 2023-11-24T08:05:27.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f19d9b640 1 -- 172.21.15.107:0/500517781 learned_addr learned my addr 172.21.15.107:0/500517781 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:27.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f1ad9d640 1 --2- 172.21.15.107:0/500517781 >> v2:172.21.15.120:3300/0 conn(0x7f6f1c0c59b0 0x7f6f1c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 0x7f6f1c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298650896 0 0) 0x7f6f04001110 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.568+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f08003520 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578488305 0 0) 0x7f6f08003520 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 msgr2=0x7f6f1c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 0x7f6f1c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 >> v2:172.21.15.120:3300/0 conn(0x7f6f1c0c59b0 msgr2=0x7f6f1c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 --2- 172.21.15.107:0/500517781 >> v2:172.21.15.120:3300/0 conn(0x7f6f1c0c59b0 0x7f6f1c06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c0abf30 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6f04001800 con 0x7f6f1c0cbf20 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 >> v1:172.21.15.107:6789/0 conn(0x7f6f1c0cbf20 legacy=0x7f6f1c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 0x7f6f080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f08004d00 0x7f6f08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c15b820 con 0x7f6f08003970 2023-11-24T08:05:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c15eea0 con 0x7f6f08004d00 2023-11-24T08:05:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0800b980 con 0x7f6f1c15b440 2023-11-24T08:05:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1ad9d640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f08004d00 0x7f6f08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 -- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 msgr2=0x7f6f080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c15b440 0x7f6f080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 -- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f08004d00 msgr2=0x7f6f08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f08004d00 0x7f6f08005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.569+0000 7f6f1a59c640 1 -- 172.21.15.107:0/500517781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08003520 con 0x7f6f08003970 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.570+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f08003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009fc0 tx=0x7f6f080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100ae046b3e939fc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c02cc60 con 0x7f6f08003970 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c00f040 con 0x7f6f08003970 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f1959a640 1 -- 172.21.15.107:0/500517781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c034440 con 0x7f6f08003970 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 -- 172.21.15.107:0/500517781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 msgr2=0x7f6f08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 --2- 172.21.15.107:0/500517781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f08003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009fc0 tx=0x7f6f080057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f1ad9d640 1 -- 172.21.15.107:0/500517781 reap_dead start 2023-11-24T08:05:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 -- 172.21.15.107:0/500517781 shutdown_connections 2023-11-24T08:05:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 -- 172.21.15.107:0/500517781 >> 172.21.15.107:0/500517781 conn(0x7f6f1c0b0c50 msgr2=0x7f6f1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 -- 172.21.15.107:0/500517781 shutdown_connections 2023-11-24T08:05:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.571+0000 7f6f20856640 1 -- 172.21.15.107:0/500517781 wait complete. 2023-11-24T08:05:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.573+0000 7f6f20856640 1 Processor -- start 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.573+0000 7f6f20856640 1 -- start start 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f1c156520 0x7f6f1c1f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c156a70 0x7f6f1c0aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c0c60d0 con 0x7f6f08003970 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c0c63b0 con 0x7f6f1c156520 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.574+0000 7f6f20856640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c0c6690 con 0x7f6f1c156a70 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47006/0 (socket says 172.21.15.107:47006) 2023-11-24T08:05:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 -- 172.21.15.107:0/3003946233 learned_addr learned my addr 172.21.15.107:0/3003946233 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 -- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c156a70 msgr2=0x7f6f1c0aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c156a70 0x7f6f1c0aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 -- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f1c156520 msgr2=0x7f6f1c1f1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f1c156520 0x7f6f1c1f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 -- 172.21.15.107:0/3003946233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c0c6970 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f1a59c640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e230 tx=0x7f6f0c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.575+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c034bd0 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.576+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c034d30 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.576+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1c0c6c50 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.576+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c034ea0 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.576+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c14cf00 con 0x7f6f08003970 2023-11-24T08:05:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.577+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1c04e8b0 con 0x7f6f08003970 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.578+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6f0c040450 con 0x7f6f08003970 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.579+0000 7f6f12ffd640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 0x7f6efc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.579+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f0c0abd60 con 0x7f6f08003970 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.579+0000 7f6f19d9b640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 0x7f6efc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.580+0000 7f6f19d9b640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 0x7f6efc06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6f04000f60 tx=0x7f6f04001c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.583+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f0c076e00 con 0x7f6f08003970 2023-11-24T08:05:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.773+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f1c06dc80 con 0x7f6f08003970 2023-11-24T08:05:27.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.774+0000 7f6f12ffd640 1 -- 172.21.15.107:0/3003946233 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f0c07a420 con 0x7f6f08003970 2023-11-24T08:05:27.776 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:27.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 msgr2=0x7f6efc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 0x7f6efc06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6f04000f60 tx=0x7f6f04001c00 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 msgr2=0x7f6f1c14c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e230 tx=0x7f6f0c009750 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 shutdown_connections 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6efc06ae70 0x7f6efc06d330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f1c156a70 0x7f6f1c0aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f1c156520 0x7f6f1c1f1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 --2- 172.21.15.107:0/3003946233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f08003970 0x7f6f1c14c9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 >> 172.21.15.107:0/3003946233 conn(0x7f6f1c0b0c50 msgr2=0x7f6f1c06c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.777+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 shutdown_connections 2023-11-24T08:05:27.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:27.778+0000 7f6f20856640 1 -- 172.21.15.107:0/3003946233 wait complete. 2023-11-24T08:05:27.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:28.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.011+0000 7fbea333c640 1 Processor -- start 2023-11-24T08:05:28.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.011+0000 7fbea333c640 1 -- start start 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0c57e0 0x7fbe9c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0cc3d0 con 0x7fbe9c15b440 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c15b820 con 0x7fbe9c0c57e0 2023-11-24T08:05:28.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea333c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c15eea0 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea10d8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea10d8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52840/0 (socket says 172.21.15.107:52840) 2023-11-24T08:05:28.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.013+0000 7fbea10d8640 1 -- 172.21.15.107:0/1405565041 learned_addr learned my addr 172.21.15.107:0/1405565041 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:28.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea18d9640 1 --2- 172.21.15.107:0/1405565041 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0c57e0 0x7fbe9c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea08d7640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea10d8640 1 -- 172.21.15.107:0/1405565041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0cbf20 msgr2=0x7fbe9c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea10d8640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea10d8640 1 -- 172.21.15.107:0/1405565041 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0c57e0 msgr2=0x7fbe9c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea10d8640 1 --2- 172.21.15.107:0/1405565041 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0c57e0 0x7fbe9c0c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.014+0000 7fbea10d8640 1 -- 172.21.15.107:0/1405565041 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0abf30 con 0x7fbe9c15b440 2023-11-24T08:05:28.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.015+0000 7fbea10d8640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0cb910 tx=0x7fbe9802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5778c0581380102 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.016+0000 7fbe93fff640 1 -- 172.21.15.107:0/1405565041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9802c9e0 con 0x7fbe9c15b440 2023-11-24T08:05:28.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.016+0000 7fbe93fff640 1 -- 172.21.15.107:0/1405565041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9802cb40 con 0x7fbe9c15b440 2023-11-24T08:05:28.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.016+0000 7fbe93fff640 1 -- 172.21.15.107:0/1405565041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe980343f0 con 0x7fbe9c15b440 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.016+0000 7fbea333c640 1 -- 172.21.15.107:0/1405565041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c15b440 msgr2=0x7fbe9c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.016+0000 7fbea333c640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0cb910 tx=0x7fbe9802be40 comp rx=0 tx=0).stop 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 -- 172.21.15.107:0/1405565041 shutdown_connections 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 --2- 172.21.15.107:0/1405565041 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0c57e0 0x7fbe9c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 --2- 172.21.15.107:0/1405565041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c15b440 0x7fbe9c0cb9e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 -- 172.21.15.107:0/1405565041 >> 172.21.15.107:0/1405565041 conn(0x7fbe9c0b0c50 msgr2=0x7fbe9c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 -- 172.21.15.107:0/1405565041 shutdown_connections 2023-11-24T08:05:28.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.017+0000 7fbea333c640 1 -- 172.21.15.107:0/1405565041 wait complete. 2023-11-24T08:05:28.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.018+0000 7fbea333c640 1 Processor -- start 2023-11-24T08:05:28.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 -- start start 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0c57e0 0x7fbe9c0b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c15b440 0x7fbe9c0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c8620 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c15bb90 con 0x7fbe9c0c57e0 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.019+0000 7fbea333c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c15e6f0 con 0x7fbe9c15b440 2023-11-24T08:05:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52852/0 (socket says 172.21.15.107:52852) 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 -- 172.21.15.107:0/294738173 learned_addr learned my addr 172.21.15.107:0/294738173 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 -- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c15b440 msgr2=0x7fbe9c0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c15b440 0x7fbe9c0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 -- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0c57e0 msgr2=0x7fbe9c0b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0c57e0 0x7fbe9c0b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 -- 172.21.15.107:0/294738173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe98009b30 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.020+0000 7fbea08d7640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00d800 tx=0x7fbe8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.021+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c00b6c0 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.021+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00b820 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.021+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c00bbb0 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.021+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c0c7d80 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.021+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c0c8340 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.022+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c15b960 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.023+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbe8c00bd10 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.024+0000 7fbe91ffb640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 0x7fbe8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.024+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe8c089bd0 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.024+0000 7fbea10d8640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 0x7fbe8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.025+0000 7fbea10d8640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 0x7fbe8006d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbe98035000 tx=0x7fbe9802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.028+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe9c15b960 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.216+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe9c15b960 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.217+0000 7fbe91ffb640 1 -- 172.21.15.107:0/294738173 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe9c15b960 con 0x7fbe9c0cbf20 2023-11-24T08:05:28.219 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:28.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 msgr2=0x7fbe8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 0x7fbe8006d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbe98035000 tx=0x7fbe9802e000 comp rx=0 tx=0).stop 2023-11-24T08:05:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 msgr2=0x7fbe9c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00d800 tx=0x7fbe8c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 shutdown_connections 2023-11-24T08:05:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe8006ae40 0x7fbe8006d300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c15b440 0x7fbe9c0c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe9c0cbf20 0x7fbe9c0c8980 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 --2- 172.21.15.107:0/294738173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0c57e0 0x7fbe9c0b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.219+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 >> 172.21.15.107:0/294738173 conn(0x7fbe9c0b0c50 msgr2=0x7fbe9c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:28.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.220+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 shutdown_connections 2023-11-24T08:05:28.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.220+0000 7fbea333c640 1 -- 172.21.15.107:0/294738173 wait complete. 2023-11-24T08:05:28.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.451+0000 7f86fa7fa640 1 Processor -- start 2023-11-24T08:05:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.451+0000 7f86fa7fa640 1 -- start start 2023-11-24T08:05:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.452+0000 7f86fa7fa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86f40b0b40 0x7f86f40b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.452+0000 7f86fa7fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f406c5b0 0x7f86f406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.452+0000 7f86fa7fa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f415ed20 con 0x7f86f415b260 2023-11-24T08:05:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.452+0000 7f86fa7fa640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f415b640 con 0x7f86f40b0b40 2023-11-24T08:05:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.452+0000 7f86fa7fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b14f0 con 0x7f86f406c5b0 2023-11-24T08:05:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86f97f8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86f415b260 0x7f86f406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57608/0 (socket says 172.21.15.107:57608) 2023-11-24T08:05:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86f97f8640 1 -- 172.21.15.107:0/4121124369 learned_addr learned my addr 172.21.15.107:0/4121124369 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605515826 0 0) 0x7f86f415ed20 con 0x7f86f415b260 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/4121124369 >> v2:172.21.15.120:3300/0 conn(0x7f86f40b0b40 0x7f86f40b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e0003520 con 0x7f86f415b260 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86f9ff9640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f406c5b0 0x7f86f406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401786222 0 0) 0x7f86e0003520 con 0x7f86f415b260 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f406c5b0 msgr2=0x7f86f406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f406c5b0 0x7f86f406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 >> v2:172.21.15.120:3300/0 conn(0x7f86f40b0b40 msgr2=0x7f86f40b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 --2- 172.21.15.107:0/4121124369 >> v2:172.21.15.120:3300/0 conn(0x7f86f40b0b40 0x7f86f40b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f415f8a0 con 0x7f86f415b260 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.453+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f86e8001670 con 0x7f86f415b260 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 >> v1:172.21.15.107:6789/0 conn(0x7f86f415b260 legacy=0x7f86f406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e00039a0 0x7f86e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86e0005180 0x7f86e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f415b640 con 0x7f86e00047c0 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b14f0 con 0x7f86e00039a0 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e000c0a0 con 0x7f86e0005180 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f9ff9640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f406c5b0 0x7f86f406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f97f8640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e00039a0 0x7f86e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 -- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86e0005180 msgr2=0x7f86e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86e0005180 0x7f86e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 -- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e00039a0 msgr2=0x7f86e0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e00039a0 0x7f86e0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f8ff7640 1 -- 172.21.15.107:0/4121124369 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0003520 con 0x7f86e00047c0 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.454+0000 7f86f97f8640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e00039a0 0x7f86e0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:28.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.455+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86e0004c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009fc0 tx=0x7f86e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0eb297f7a3a069d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f002cc60 con 0x7f86e00047c0 2023-11-24T08:05:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f000f040 con 0x7f86e00047c0 2023-11-24T08:05:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86e7fff640 1 -- 172.21.15.107:0/4121124369 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0034440 con 0x7f86e00047c0 2023-11-24T08:05:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 -- 172.21.15.107:0/4121124369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 msgr2=0x7f86e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/4121124369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86e0004c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009fc0 tx=0x7f86e00057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86f9ff9640 1 -- 172.21.15.107:0/4121124369 reap_dead start 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 -- 172.21.15.107:0/4121124369 shutdown_connections 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 -- 172.21.15.107:0/4121124369 >> 172.21.15.107:0/4121124369 conn(0x7f86f40cbb40 msgr2=0x7f86f40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 -- 172.21.15.107:0/4121124369 shutdown_connections 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.456+0000 7f86fa7fa640 1 -- 172.21.15.107:0/4121124369 wait complete. 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.458+0000 7f86fa7fa640 1 Processor -- start 2023-11-24T08:05:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.458+0000 7f86fa7fa640 1 -- start start 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f40b4780 0x7f86f41e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f41e5e10 0x7f86f40afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b34f0 con 0x7f86e00047c0 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b3720 con 0x7f86f41e5e10 2023-11-24T08:05:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86fa7fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b39c0 con 0x7f86f40b4780 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86f97f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86f97f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52864/0 (socket says 172.21.15.107:52864) 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.459+0000 7f86f97f8640 1 -- 172.21.15.107:0/188967360 learned_addr learned my addr 172.21.15.107:0/188967360 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f9ff9640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f41e5e10 0x7f86f40afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 -- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f40b4780 msgr2=0x7f86f41e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f40b4780 0x7f86f41e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 -- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f41e5e10 msgr2=0x7f86f40afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f41e5e10 0x7f86f40afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 -- 172.21.15.107:0/188967360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40b3ca0 con 0x7f86e00047c0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86f97f8640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f86e800d800 tx=0x7f86e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e80074e0 con 0x7f86e00047c0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8007640 con 0x7f86e00047c0 2023-11-24T08:05:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.460+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e80049e0 con 0x7f86e00047c0 2023-11-24T08:05:28.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.461+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f40b3f80 con 0x7f86e00047c0 2023-11-24T08:05:28.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.461+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f41588e0 con 0x7f86e00047c0 2023-11-24T08:05:28.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.462+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f404e840 con 0x7f86e00047c0 2023-11-24T08:05:28.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.463+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f86e8004b40 con 0x7f86e00047c0 2023-11-24T08:05:28.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.463+0000 7f86e5ffb640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 0x7f86d806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.464+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86e8089080 con 0x7f86e00047c0 2023-11-24T08:05:28.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.464+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 0x7f86d806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.465+0000 7f86f8ff7640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 0x7f86d806d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f002e070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.468+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86e80bbe90 con 0x7f86e00047c0 2023-11-24T08:05:28.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.661+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86f4070d30 con 0x7f86e00047c0 2023-11-24T08:05:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.662+0000 7f86e5ffb640 1 -- 172.21.15.107:0/188967360 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86e80190a0 con 0x7f86e00047c0 2023-11-24T08:05:28.664 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 msgr2=0x7f86d806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 0x7f86d806d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f002e070 comp rx=0 tx=0).stop 2023-11-24T08:05:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 msgr2=0x7f86f41583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f86e800d800 tx=0x7f86e800dcc0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 shutdown_connections 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86d806ae70 0x7f86d806d330 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f41e5e10 0x7f86f40afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f40b4780 0x7f86f41e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 --2- 172.21.15.107:0/188967360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e00047c0 0x7f86f41583a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 >> 172.21.15.107:0/188967360 conn(0x7f86f40cbb40 msgr2=0x7f86f40aca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 shutdown_connections 2023-11-24T08:05:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.665+0000 7f86fa7fa640 1 -- 172.21.15.107:0/188967360 wait complete. 2023-11-24T08:05:28.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:28.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.897+0000 7f26d62d1640 1 Processor -- start 2023-11-24T08:05:28.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.897+0000 7f26d62d1640 1 -- start start 2023-11-24T08:05:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26d015d4e0 0x7f26d00d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 0x7f26d00d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00d4110 con 0x7f26d015d4e0 2023-11-24T08:05:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00d04e0 con 0x7f26d00d0070 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d62d1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d00ce990 con 0x7f26d00d38e0 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d52cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26d015d4e0 0x7f26d00d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d52cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26d015d4e0 0x7f26d00d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52866/0 (socket says 172.21.15.107:52866) 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.899+0000 7f26d52cf640 1 -- 172.21.15.107:0/3692116499 learned_addr learned my addr 172.21.15.107:0/3692116499 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d52cf640 1 -- 172.21.15.107:0/3692116499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 msgr2=0x7f26d00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3692116499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d52cf640 1 -- 172.21.15.107:0/3692116499 >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 msgr2=0x7f26d00d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d5ad0640 1 --2- 172.21.15.107:0/3692116499 >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 0x7f26d00d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3692116499 >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 0x7f26d00d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d52cf640 1 -- 172.21.15.107:0/3692116499 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d00cec70 con 0x7f26d015d4e0 2023-11-24T08:05:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.900+0000 7f26d5ad0640 1 --2- 172.21.15.107:0/3692116499 >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 0x7f26d00d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.901+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3692116499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00d1e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f26d00b2a00 tx=0x7f26cc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4601ef713a175812 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.901+0000 7f26c7fff640 1 -- 172.21.15.107:0/3692116499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc031860 con 0x7f26d015d4e0 2023-11-24T08:05:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.901+0000 7f26c7fff640 1 -- 172.21.15.107:0/3692116499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc0319c0 con 0x7f26d015d4e0 2023-11-24T08:05:28.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.901+0000 7f26c7fff640 1 -- 172.21.15.107:0/3692116499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc031d50 con 0x7f26d015d4e0 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 -- 172.21.15.107:0/3692116499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 msgr2=0x7f26d00d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3692116499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00d1e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f26d00b2a00 tx=0x7f26cc02b900 comp rx=0 tx=0).stop 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 -- 172.21.15.107:0/3692116499 shutdown_connections 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3692116499 >> v2:172.21.15.120:3300/0 conn(0x7f26d00d0070 0x7f26d00d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3692116499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3692116499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00d1e20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 -- 172.21.15.107:0/3692116499 >> 172.21.15.107:0/3692116499 conn(0x7f26d009c510 msgr2=0x7f26d00d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:28.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 -- 172.21.15.107:0/3692116499 shutdown_connections 2023-11-24T08:05:28.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.902+0000 7f26d62d1640 1 -- 172.21.15.107:0/3692116499 wait complete. 2023-11-24T08:05:28.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.904+0000 7f26d62d1640 1 Processor -- start 2023-11-24T08:05:28.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.904+0000 7f26d62d1640 1 -- start start 2023-11-24T08:05:28.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 0x7f26d01f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d015e7f0 con 0x7f26d015d4e0 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d015db70 con 0x7f26d00d0070 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d62d1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d015de50 con 0x7f26d00d38e0 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d5ad0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d5ad0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52878/0 (socket says 172.21.15.107:52878) 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.905+0000 7f26d5ad0640 1 -- 172.21.15.107:0/3787410615 learned_addr learned my addr 172.21.15.107:0/3787410615 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d4ace640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 0x7f26d01f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 -- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 msgr2=0x7f26d00b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 -- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 msgr2=0x7f26d01f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 0x7f26d01f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 -- 172.21.15.107:0/3787410615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc0094d0 con 0x7f26d015d4e0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d4ace640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 0x7f26d01f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d5ad0640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f26c800cef0 tx=0x7f26c800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c800d840 con 0x7f26d015d4e0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c800d9c0 con 0x7f26d015d4e0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c800dd70 con 0x7f26d015d4e0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.906+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d015e130 con 0x7f26d015d4e0 2023-11-24T08:05:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.907+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d01f2a40 con 0x7f26d015d4e0 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.908+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d015e930 con 0x7f26d015d4e0 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.909+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f26c801e3b0 con 0x7f26d015d4e0 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.910+0000 7f26c5ffb640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 0x7f26b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.910+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26c808a190 con 0x7f26d015d4e0 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.910+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 0x7f26b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.911+0000 7f26d52cf640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 0x7f26b406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f26cc036000 tx=0x7f26cc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:28.914+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26d015e930 con 0x7f26d015d4e0 2023-11-24T08:05:29.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.108+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26d015e930 con 0x7f26d015d4e0 2023-11-24T08:05:29.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.110+0000 7f26c5ffb640 1 -- 172.21.15.107:0/3787410615 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f26d015e930 con 0x7f26d015d4e0 2023-11-24T08:05:29.112 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:29.112 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 msgr2=0x7f26b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 0x7f26b406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f26cc036000 tx=0x7f26cc03c040 comp rx=0 tx=0).stop 2023-11-24T08:05:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 msgr2=0x7f26d00b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f26c800cef0 tx=0x7f26c800a710 comp rx=0 tx=0).stop 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 shutdown_connections 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b406ae40 0x7f26b406d300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d015d4e0 0x7f26d00b7380 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d00d38e0 0x7f26d00b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 --2- 172.21.15.107:0/3787410615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00d0070 0x7f26d01f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 >> 172.21.15.107:0/3787410615 conn(0x7f26d009c510 msgr2=0x7f26d00afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 shutdown_connections 2023-11-24T08:05:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.113+0000 7f26d62d1640 1 -- 172.21.15.107:0/3787410615 wait complete. 2023-11-24T08:05:29.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:29.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.353+0000 7f0e69790640 1 Processor -- start 2023-11-24T08:05:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.354+0000 7f0e69790640 1 -- start start 2023-11-24T08:05:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e69790640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e640cc0f0 0x7f0e640c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e69790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e6406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e69790640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6415eea0 con 0x7f0e6415b440 2023-11-24T08:05:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e69790640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6415b820 con 0x7f0e640cc0f0 2023-11-24T08:05:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e69790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640abf30 con 0x7f0e640c57e0 2023-11-24T08:05:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e62ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e6415b440 0x7f0e640cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57612/0 (socket says 172.21.15.107:57612) 2023-11-24T08:05:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.355+0000 7f0e62ffd640 1 -- 172.21.15.107:0/2506890749 learned_addr learned my addr 172.21.15.107:0/2506890749 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e627fc640 1 --2- 172.21.15.107:0/2506890749 >> v2:172.21.15.120:3300/0 conn(0x7f0e640cc0f0 0x7f0e640c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854574910 0 0) 0x7f0e6415eea0 con 0x7f0e6415b440 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e50003520 con 0x7f0e6415b440 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e637fe640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e6406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248419159 0 0) 0x7f0e50003520 con 0x7f0e6415b440 2023-11-24T08:05:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 msgr2=0x7f0e6406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e6406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 >> v2:172.21.15.120:3300/0 conn(0x7f0e640cc0f0 msgr2=0x7f0e640c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 --2- 172.21.15.107:0/2506890749 >> v2:172.21.15.120:3300/0 conn(0x7f0e640cc0f0 0x7f0e640c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6415f0f0 con 0x7f0e6415b440 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0e540019f0 con 0x7f0e6415b440 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.356+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 >> v1:172.21.15.107:6789/0 conn(0x7f0e6415b440 legacy=0x7f0e640cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e50003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50003ef0 0x7f0e500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e50004d00 0x7f0e50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6415b820 con 0x7f0e50004d00 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640abf30 con 0x7f0e50003ef0 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5000ba80 con 0x7f0e640c57e0 2023-11-24T08:05:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e50004d00 0x7f0e50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e62ffd640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e50003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e627fc640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50003ef0 0x7f0e500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 -- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 msgr2=0x7f0e50003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e640c57e0 0x7f0e50003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 -- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50003ef0 msgr2=0x7f0e500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50003ef0 0x7f0e500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.357+0000 7f0e637fe640 1 -- 172.21.15.107:0/2506890749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50003520 con 0x7f0e50004d00 2023-11-24T08:05:29.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e637fe640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e50004d00 0x7f0e50005180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00ba60 tx=0x7f0e500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a860d61fb992bd82 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c032630 con 0x7f0e50004d00 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c032790 con 0x7f0e50004d00 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e61ffb640 1 -- 172.21.15.107:0/2506890749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c032b20 con 0x7f0e50004d00 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e69790640 1 -- 172.21.15.107:0/2506890749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e50004d00 msgr2=0x7f0e50005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e69790640 1 --2- 172.21.15.107:0/2506890749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e50004d00 0x7f0e50005180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00ba60 tx=0x7f0e500057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e637fe640 1 -- 172.21.15.107:0/2506890749 reap_dead start 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e69790640 1 -- 172.21.15.107:0/2506890749 shutdown_connections 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.359+0000 7f0e69790640 1 -- 172.21.15.107:0/2506890749 >> 172.21.15.107:0/2506890749 conn(0x7f0e640b0c50 msgr2=0x7f0e640b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:29.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.360+0000 7f0e69790640 1 -- 172.21.15.107:0/2506890749 shutdown_connections 2023-11-24T08:05:29.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.360+0000 7f0e69790640 1 -- 172.21.15.107:0/2506890749 wait complete. 2023-11-24T08:05:29.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.361+0000 7f0e69790640 1 Processor -- start 2023-11-24T08:05:29.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.361+0000 7f0e69790640 1 -- start start 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50004d00 0x7f0e640ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e641541d0 0x7f0e64152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640add40 con 0x7f0e64152c90 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64153130 con 0x7f0e50004d00 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.362+0000 7f0e69790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64153410 con 0x7f0e641541d0 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52894/0 (socket says 172.21.15.107:52894) 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 -- 172.21.15.107:0/3424554655 learned_addr learned my addr 172.21.15.107:0/3424554655 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e62ffd640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50004d00 0x7f0e640ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e627fc640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e641541d0 0x7f0e64152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 -- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e641541d0 msgr2=0x7f0e64152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e641541d0 0x7f0e64152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 -- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50004d00 msgr2=0x7f0e640ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50004d00 0x7f0e640ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.363+0000 7f0e637fe640 1 -- 172.21.15.107:0/3424554655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e641536f0 con 0x7f0e64152c90 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e637fe640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c035040 tx=0x7f0e5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c004220 con 0x7f0e64152c90 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c0043a0 con 0x7f0e64152c90 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c03b930 con 0x7f0e64152c90 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e641539d0 con 0x7f0e64152c90 2023-11-24T08:05:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.364+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6414fb50 con 0x7f0e64152c90 2023-11-24T08:05:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.365+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6414fce0 con 0x7f0e64152c90 2023-11-24T08:05:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.366+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0e5c03ba90 con 0x7f0e64152c90 2023-11-24T08:05:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.367+0000 7f0e5b7fe640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 0x7f0e4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.367+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e5c0abbf0 con 0x7f0e64152c90 2023-11-24T08:05:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.367+0000 7f0e62ffd640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 0x7f0e4406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.368+0000 7f0e62ffd640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 0x7f0e4406d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e540013d0 tx=0x7f0e54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.372+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e5c076c00 con 0x7f0e64152c90 2023-11-24T08:05:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.561+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e64153ef0 con 0x7f0e64152c90 2023-11-24T08:05:29.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.562+0000 7f0e5b7fe640 1 -- 172.21.15.107:0/3424554655 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e5c07a220 con 0x7f0e64152c90 2023-11-24T08:05:29.564 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:29.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 msgr2=0x7f0e4406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 0x7f0e4406d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e540013d0 tx=0x7f0e54009040 comp rx=0 tx=0).stop 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 msgr2=0x7f0e6414ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c035040 tx=0x7f0e5c004730 comp rx=0 tx=0).stop 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 shutdown_connections 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e4406ae40 0x7f0e4406d300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e64152c90 0x7f0e6414ee60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e641541d0 0x7f0e64152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 --2- 172.21.15.107:0/3424554655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e50004d00 0x7f0e640ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 >> 172.21.15.107:0/3424554655 conn(0x7f0e640b0c50 msgr2=0x7f0e640cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 shutdown_connections 2023-11-24T08:05:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.565+0000 7f0e69790640 1 -- 172.21.15.107:0/3424554655 wait complete. 2023-11-24T08:05:29.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:29.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.799+0000 7fd39b43d640 1 Processor -- start 2023-11-24T08:05:29.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.799+0000 7fd39b43d640 1 -- start start 2023-11-24T08:05:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd39b43d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd394160a70 0x7fd3940cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd39b43d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 0x7fd3940d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd39b43d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3940b23f0 con 0x7fd394160e50 2023-11-24T08:05:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd39b43d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940d09c0 con 0x7fd394160a70 2023-11-24T08:05:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd39b43d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940b25c0 con 0x7fd3940d3b30 2023-11-24T08:05:29.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd3991d9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd394160a70 0x7fd3940cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd3991d9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd394160a70 0x7fd3940cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52896/0 (socket says 172.21.15.107:52896) 2023-11-24T08:05:29.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.801+0000 7fd3991d9640 1 -- 172.21.15.107:0/905853803 learned_addr learned my addr 172.21.15.107:0/905853803 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:29.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3999da640 1 --2- 172.21.15.107:0/905853803 >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 0x7fd3940d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3991d9640 1 -- 172.21.15.107:0/905853803 >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 msgr2=0x7fd3940d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3991d9640 1 --2- 172.21.15.107:0/905853803 >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 0x7fd3940d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3991d9640 1 -- 172.21.15.107:0/905853803 >> v1:172.21.15.120:6789/0 conn(0x7fd394160e50 legacy=0x7fd3940affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:29.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3991d9640 1 -- 172.21.15.107:0/905853803 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3940b2110 con 0x7fd394160a70 2023-11-24T08:05:29.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.802+0000 7fd3999da640 1 --2- 172.21.15.107:0/905853803 >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 0x7fd3940d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:29.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.803+0000 7fd3991d9640 1 --2- 172.21.15.107:0/905853803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940cfc60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3940c6da0 tx=0x7fd38802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26f689896bd09cd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.803+0000 7fd393fff640 1 -- 172.21.15.107:0/905853803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802c9f0 con 0x7fd394160a70 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.803+0000 7fd393fff640 1 -- 172.21.15.107:0/905853803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38802cb50 con 0x7fd394160a70 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.803+0000 7fd393fff640 1 -- 172.21.15.107:0/905853803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3880343f0 con 0x7fd394160a70 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 -- 172.21.15.107:0/905853803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 msgr2=0x7fd3940cfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 --2- 172.21.15.107:0/905853803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940cfc60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3940c6da0 tx=0x7fd38802be50 comp rx=0 tx=0).stop 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 -- 172.21.15.107:0/905853803 shutdown_connections 2023-11-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 --2- 172.21.15.107:0/905853803 >> v2:172.21.15.120:3300/0 conn(0x7fd3940d3b30 0x7fd3940d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 --2- 172.21.15.107:0/905853803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940cfc60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 -- 172.21.15.107:0/905853803 >> 172.21.15.107:0/905853803 conn(0x7fd39409c6d0 msgr2=0x7fd3940d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:29.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 -- 172.21.15.107:0/905853803 shutdown_connections 2023-11-24T08:05:29.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.804+0000 7fd39b43d640 1 -- 172.21.15.107:0/905853803 wait complete. 2023-11-24T08:05:29.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.806+0000 7fd39b43d640 1 Processor -- start 2023-11-24T08:05:29.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.806+0000 7fd39b43d640 1 -- start start 2023-11-24T08:05:29.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3940d3b30 0x7fd39409ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd394160e50 0x7fd3941f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394155420 con 0x7fd394160a70 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940b3550 con 0x7fd394160e50 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd39b43d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39409f470 con 0x7fd3940d3b30 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd3989d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd3989d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52912/0 (socket says 172.21.15.107:52912) 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.807+0000 7fd3989d8640 1 -- 172.21.15.107:0/3416324640 learned_addr learned my addr 172.21.15.107:0/3416324640 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 -- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3940d3b30 msgr2=0x7fd39409ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3940d3b30 0x7fd39409ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 -- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd394160e50 msgr2=0x7fd3941f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3999da640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd394160e50 0x7fd3941f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd394160e50 0x7fd3941f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 -- 172.21.15.107:0/3416324640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd388009b30 con 0x7fd394160a70 2023-11-24T08:05:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd3989d8640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd38000b3c0 tx=0x7fd38000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380011830 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd380011990 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39409f750 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.808+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380011d60 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.809+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39409fd10 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.810+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39408d0c0 con 0x7fd394160a70 2023-11-24T08:05:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.811+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd38001b460 con 0x7fd394160a70 2023-11-24T08:05:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.811+0000 7fd391ffb640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 0x7fd37806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.811+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd380089370 con 0x7fd394160a70 2023-11-24T08:05:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.812+0000 7fd3991d9640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 0x7fd37806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.812+0000 7fd3991d9640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 0x7fd37806d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd388035000 tx=0x7fd38802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:29.816+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3800544b0 con 0x7fd394160a70 2023-11-24T08:05:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.008+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd39404e840 con 0x7fd394160a70 2023-11-24T08:05:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.009+0000 7fd391ffb640 1 -- 172.21.15.107:0/3416324640 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd380057ad0 con 0x7fd394160a70 2023-11-24T08:05:30.011 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.011+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 msgr2=0x7fd37806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.011+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 0x7fd37806d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd388035000 tx=0x7fd38802e000 comp rx=0 tx=0).stop 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.011+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 msgr2=0x7fd3940b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.011+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd38000b3c0 tx=0x7fd38000b880 comp rx=0 tx=0).stop 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 shutdown_connections 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37806ae40 0x7fd37806d300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd394160e50 0x7fd3941f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd394160a70 0x7fd3940b3010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 --2- 172.21.15.107:0/3416324640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3940d3b30 0x7fd39409ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 >> 172.21.15.107:0/3416324640 conn(0x7fd39409c6d0 msgr2=0x7fd3940cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:30.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 shutdown_connections 2023-11-24T08:05:30.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.012+0000 7fd39b43d640 1 -- 172.21.15.107:0/3416324640 wait complete. 2023-11-24T08:05:30.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.250+0000 7fd4f845c640 1 Processor -- start 2023-11-24T08:05:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.250+0000 7fd4f845c640 1 -- start start 2023-11-24T08:05:30.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f845c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4f006de10 0x7fd4f015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f845c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 0x7fd4f00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f845c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f015ec90 con 0x7fd4f0072160 2023-11-24T08:05:30.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f845c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00cd530 con 0x7fd4f006de10 2023-11-24T08:05:30.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f845c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00c4ed0 con 0x7fd4f00c49f0 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f61f8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd4f0072160 0x7fd4f006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57628/0 (socket says 172.21.15.107:57628) 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f61f8640 1 -- 172.21.15.107:0/27970430 learned_addr learned my addr 172.21.15.107:0/27970430 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.252+0000 7fd4f59f7640 1 --2- 172.21.15.107:0/27970430 >> v2:172.21.15.107:3300/0 conn(0x7fd4f006de10 0x7fd4f015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f69f9640 1 --2- 172.21.15.107:0/27970430 >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 0x7fd4f00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f59f7640 1 -- 172.21.15.107:0/27970430 >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 msgr2=0x7fd4f00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f59f7640 1 --2- 172.21.15.107:0/27970430 >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 0x7fd4f00705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f59f7640 1 -- 172.21.15.107:0/27970430 >> v1:172.21.15.107:6789/0 conn(0x7fd4f0072160 legacy=0x7fd4f006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f59f7640 1 -- 172.21.15.107:0/27970430 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00c51b0 con 0x7fd4f006de10 2023-11-24T08:05:30.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f69f9640 1 --2- 172.21.15.107:0/27970430 >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 0x7fd4f00705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:30.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.253+0000 7fd4f51f6640 1 -- 172.21.15.107:0/27970430 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318732768 0 0) 0x7fd4ec001110 con 0x7fd4f0072160 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.254+0000 7fd4f59f7640 1 --2- 172.21.15.107:0/27970430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f015e750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0072a30 tx=0x7fd4e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b036a088e7b5f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.254+0000 7fd4f51f6640 1 -- 172.21.15.107:0/27970430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0031830 con 0x7fd4f006de10 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.254+0000 7fd4f51f6640 1 -- 172.21.15.107:0/27970430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0031990 con 0x7fd4f006de10 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.254+0000 7fd4f51f6640 1 -- 172.21.15.107:0/27970430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0031d20 con 0x7fd4f006de10 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 -- 172.21.15.107:0/27970430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 msgr2=0x7fd4f015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 --2- 172.21.15.107:0/27970430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f015e750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0072a30 tx=0x7fd4e002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 -- 172.21.15.107:0/27970430 shutdown_connections 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 --2- 172.21.15.107:0/27970430 >> v2:172.21.15.120:3300/0 conn(0x7fd4f00c49f0 0x7fd4f00705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 --2- 172.21.15.107:0/27970430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f015e750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 -- 172.21.15.107:0/27970430 >> 172.21.15.107:0/27970430 conn(0x7fd4f00b0c50 msgr2=0x7fd4f00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:30.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 -- 172.21.15.107:0/27970430 shutdown_connections 2023-11-24T08:05:30.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.255+0000 7fd4f845c640 1 -- 172.21.15.107:0/27970430 wait complete. 2023-11-24T08:05:30.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.257+0000 7fd4f845c640 1 Processor -- start 2023-11-24T08:05:30.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.257+0000 7fd4f845c640 1 -- start start 2023-11-24T08:05:30.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4f0072160 0x7fd4f00c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4f00c49f0 0x7fd4f00c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c7c30 con 0x7fd4f006de10 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c77a0 con 0x7fd4f00c49f0 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f845c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c6b90 con 0x7fd4f0072160 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52944/0 (socket says 172.21.15.107:52944) 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 -- 172.21.15.107:0/2134721192 learned_addr learned my addr 172.21.15.107:0/2134721192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f69f9640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4f00c49f0 0x7fd4f00c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 -- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4f0072160 msgr2=0x7fd4f00c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4f0072160 0x7fd4f00c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 -- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4f00c49f0 msgr2=0x7fd4f00c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4f00c49f0 0x7fd4f00c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.258+0000 7fd4f61f8640 1 -- 172.21.15.107:0/2134721192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e0009400 con 0x7fd4f006de10 2023-11-24T08:05:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4f61f8640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c780 tx=0x7fd4ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec0072d0 con 0x7fd4f006de10 2023-11-24T08:05:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec007430 con 0x7fd4f006de10 2023-11-24T08:05:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec0047e0 con 0x7fd4f006de10 2023-11-24T08:05:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f00c6e70 con 0x7fd4f006de10 2023-11-24T08:05:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.259+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f00c7430 con 0x7fd4f006de10 2023-11-24T08:05:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.260+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f004e8b0 con 0x7fd4f006de10 2023-11-24T08:05:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.261+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd4ec004940 con 0x7fd4f006de10 2023-11-24T08:05:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.262+0000 7fd4e6ffd640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 0x7fd4d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.262+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4ec088840 con 0x7fd4f006de10 2023-11-24T08:05:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.262+0000 7fd4f59f7640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 0x7fd4d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.263+0000 7fd4f59f7640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 0x7fd4d406d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0036000 tx=0x7fd4e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.267+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4ec053850 con 0x7fd4f006de10 2023-11-24T08:05:30.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.457+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4f015ec90 con 0x7fd4f006de10 2023-11-24T08:05:30.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.458+0000 7fd4e6ffd640 1 -- 172.21.15.107:0/2134721192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4f015ec90 con 0x7fd4f006de10 2023-11-24T08:05:30.460 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:30.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 msgr2=0x7fd4d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 0x7fd4d406d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0036000 tx=0x7fd4e0030000 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 msgr2=0x7fd4f014bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c780 tx=0x7fd4ec00cc40 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 shutdown_connections 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4d406ae40 0x7fd4d406d300 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4f00c49f0 0x7fd4f00c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4f0072160 0x7fd4f00c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 --2- 172.21.15.107:0/2134721192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4f006de10 0x7fd4f014bd80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 >> 172.21.15.107:0/2134721192 conn(0x7fd4f00b0c50 msgr2=0x7fd4f00ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.461+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 shutdown_connections 2023-11-24T08:05:30.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.462+0000 7fd4f845c640 1 -- 172.21.15.107:0/2134721192 wait complete. 2023-11-24T08:05:30.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.693+0000 7fb7c7535640 1 Processor -- start 2023-11-24T08:05:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.693+0000 7fb7c7535640 1 -- start start 2023-11-24T08:05:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.694+0000 7fb7c7535640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7c0159e80 0x7fb7c0071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.694+0000 7fb7c7535640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 0x7fb7c015a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c7535640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00ce870 con 0x7fb7c015a7a0 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c7535640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00b11f0 con 0x7fb7c0159e80 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c7535640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00cdc60 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7c0159e80 0x7fb7c0071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7c0159e80 0x7fb7c0071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52948/0 (socket says 172.21.15.107:52948) 2023-11-24T08:05:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 -- 172.21.15.107:0/3529168075 learned_addr learned my addr 172.21.15.107:0/3529168075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/3529168075 >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 0x7fb7c015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 -- 172.21.15.107:0/3529168075 >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 msgr2=0x7fb7c015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 --2- 172.21.15.107:0/3529168075 >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 0x7fb7c015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 -- 172.21.15.107:0/3529168075 >> v1:172.21.15.120:6789/0 conn(0x7fb7c015a7a0 legacy=0x7fb7c00b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c6533640 1 -- 172.21.15.107:0/3529168075 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00cdf40 con 0x7fb7c0159e80 2023-11-24T08:05:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.695+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/3529168075 >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 0x7fb7c015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.697+0000 7fb7c6533640 1 --2- 172.21.15.107:0/3529168075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c0159e80 0x7fb7c0071610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006e200 tx=0x7fb7bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9fff4839e5594e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.697+0000 7fb7c5531640 1 -- 172.21.15.107:0/3529168075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0317e0 con 0x7fb7c0159e80 2023-11-24T08:05:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.697+0000 7fb7c5531640 1 -- 172.21.15.107:0/3529168075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc031940 con 0x7fb7c0159e80 2023-11-24T08:05:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.697+0000 7fb7c5531640 1 -- 172.21.15.107:0/3529168075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc031cd0 con 0x7fb7c0159e80 2023-11-24T08:05:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 -- 172.21.15.107:0/3529168075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c0159e80 msgr2=0x7fb7c0071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 --2- 172.21.15.107:0/3529168075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c0159e80 0x7fb7c0071610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006e200 tx=0x7fb7bc02b880 comp rx=0 tx=0).stop 2023-11-24T08:05:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 -- 172.21.15.107:0/3529168075 shutdown_connections 2023-11-24T08:05:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 --2- 172.21.15.107:0/3529168075 >> v2:172.21.15.120:3300/0 conn(0x7fb7c00cd6a0 0x7fb7c015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 --2- 172.21.15.107:0/3529168075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c0159e80 0x7fb7c0071610 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 -- 172.21.15.107:0/3529168075 >> 172.21.15.107:0/3529168075 conn(0x7fb7c00cb830 msgr2=0x7fb7c00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 -- 172.21.15.107:0/3529168075 shutdown_connections 2023-11-24T08:05:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.698+0000 7fb7c7535640 1 -- 172.21.15.107:0/3529168075 wait complete. 2023-11-24T08:05:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.700+0000 7fb7c7535640 1 Processor -- start 2023-11-24T08:05:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.700+0000 7fb7c7535640 1 -- start start 2023-11-24T08:05:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 0x7fb7c0158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 0x7fb7c0158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c014c980 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c014cc60 con 0x7fb7c015a7a0 2023-11-24T08:05:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c7535640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c014cf40 con 0x7fb7c0159e80 2023-11-24T08:05:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c6533640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c6533640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52958/0 (socket says 172.21.15.107:52958) 2023-11-24T08:05:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c6533640 1 -- 172.21.15.107:0/2317439045 learned_addr learned my addr 172.21.15.107:0/2317439045 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c6d34640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 0x7fb7c0158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 0x7fb7c0158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.701+0000 7fb7c6533640 1 -- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 msgr2=0x7fb7c0158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6533640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 0x7fb7c0158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6533640 1 -- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 msgr2=0x7fb7c0158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6533640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 0x7fb7c0158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6533640 1 -- 172.21.15.107:0/2317439045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc0094d0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6d34640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 0x7fb7c0158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 0x7fb7c0158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c6533640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc02b8c0 tx=0x7fb7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0329c0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc032b40 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0453f0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c014d220 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.702+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c014c610 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.704+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c004e840 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.704+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb7bc04d3f0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.705+0000 7fb7b6ffd640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 0x7fb7a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.705+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7bc0ab5d0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.706+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 0x7fb7a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.706+0000 7fb7c5d32640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 0x7fb7a406d300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.710+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7bc076690 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.905+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c014d740 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.907+0000 7fb7b6ffd640 1 -- 172.21.15.107:0/2317439045 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fb7bc079cb0 con 0x7fb7c00cd6a0 2023-11-24T08:05:30.909 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:30.909 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 msgr2=0x7fb7a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 0x7fb7a406d300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0008040 comp rx=0 tx=0).stop 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 msgr2=0x7fb7c015b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc02b8c0 tx=0x7fb7bc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 shutdown_connections 2023-11-24T08:05:30.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7a406ae40 0x7fb7a406d300 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7c015a7a0 0x7fb7c0158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7c0159e80 0x7fb7c0158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 --2- 172.21.15.107:0/2317439045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c00cd6a0 0x7fb7c015b8c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 >> 172.21.15.107:0/2317439045 conn(0x7fb7c00cb830 msgr2=0x7fb7c0070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 shutdown_connections 2023-11-24T08:05:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:30.909+0000 7fb7c7535640 1 -- 172.21.15.107:0/2317439045 wait complete. 2023-11-24T08:05:30.932 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:30.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.147+0000 7f2fb2961640 1 Processor -- start 2023-11-24T08:05:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.147+0000 7f2fb2961640 1 -- start start 2023-11-24T08:05:31.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fb2961640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fac15b440 0x7f2fac0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fb2961640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2fac0c59b0 0x7f2fac06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fb2961640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0c5dc0 con 0x7f2fac0cbf20 2023-11-24T08:05:31.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fb2961640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0cc3d0 con 0x7f2fac15b440 2023-11-24T08:05:31.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fb2961640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac15b820 con 0x7f2fac0c59b0 2023-11-24T08:05:31.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fabfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fac15b440 0x7f2fac0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fabfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fac15b440 0x7f2fac0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52972/0 (socket says 172.21.15.107:52972) 2023-11-24T08:05:31.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.149+0000 7f2fabfff640 1 -- 172.21.15.107:0/3498594989 learned_addr learned my addr 172.21.15.107:0/3498594989 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:31.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.150+0000 7f2fb0efe640 1 --2- 172.21.15.107:0/3498594989 >> v2:172.21.15.120:3300/0 conn(0x7f2fac0c59b0 0x7f2fac06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.150+0000 7f2fabfff640 1 -- 172.21.15.107:0/3498594989 >> v2:172.21.15.120:3300/0 conn(0x7f2fac0c59b0 msgr2=0x7f2fac06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:31.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.150+0000 7f2fabfff640 1 --2- 172.21.15.107:0/3498594989 >> v2:172.21.15.120:3300/0 conn(0x7f2fac0c59b0 0x7f2fac06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.150+0000 7f2fabfff640 1 -- 172.21.15.107:0/3498594989 >> v1:172.21.15.120:6789/0 conn(0x7f2fac0cbf20 legacy=0x7f2fac0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.150+0000 7f2fabfff640 1 -- 172.21.15.107:0/3498594989 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac15eea0 con 0x7f2fac15b440 2023-11-24T08:05:31.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.151+0000 7f2fabfff640 1 --2- 172.21.15.107:0/3498594989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac15b440 0x7f2fac0cb9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0cb910 tx=0x7f2f9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11d77cc46d5097b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fab7fe640 1 -- 172.21.15.107:0/3498594989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c02ca80 con 0x7f2fac15b440 2023-11-24T08:05:31.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fab7fe640 1 -- 172.21.15.107:0/3498594989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c02cbe0 con 0x7f2fac15b440 2023-11-24T08:05:31.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fab7fe640 1 -- 172.21.15.107:0/3498594989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c0343f0 con 0x7f2fac15b440 2023-11-24T08:05:31.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 -- 172.21.15.107:0/3498594989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac15b440 msgr2=0x7f2fac0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 --2- 172.21.15.107:0/3498594989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac15b440 0x7f2fac0cb9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0cb910 tx=0x7f2f9c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 -- 172.21.15.107:0/3498594989 shutdown_connections 2023-11-24T08:05:31.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 --2- 172.21.15.107:0/3498594989 >> v2:172.21.15.120:3300/0 conn(0x7f2fac0c59b0 0x7f2fac06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 --2- 172.21.15.107:0/3498594989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac15b440 0x7f2fac0cb9e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 -- 172.21.15.107:0/3498594989 >> 172.21.15.107:0/3498594989 conn(0x7f2fac0b0c50 msgr2=0x7f2fac0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:31.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.152+0000 7f2fb2961640 1 -- 172.21.15.107:0/3498594989 shutdown_connections 2023-11-24T08:05:31.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.153+0000 7f2fb2961640 1 -- 172.21.15.107:0/3498594989 wait complete. 2023-11-24T08:05:31.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.154+0000 7f2fb2961640 1 Processor -- start 2023-11-24T08:05:31.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.154+0000 7f2fb2961640 1 -- start start 2023-11-24T08:05:31.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 0x7f2fac1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fac15b440 0x7f2fac0ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac157380 con 0x7f2fac0c59b0 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac158e60 con 0x7f2fac0cbf20 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.155+0000 7f2fb2961640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac159140 con 0x7f2fac15b440 2023-11-24T08:05:31.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52974/0 (socket says 172.21.15.107:52974) 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 -- 172.21.15.107:0/820474228 learned_addr learned my addr 172.21.15.107:0/820474228 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 -- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fac15b440 msgr2=0x7f2fac0ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fac15b440 0x7f2fac0ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 -- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 msgr2=0x7f2fac1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fa37fe640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 0x7f2fac1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 0x7f2fac1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fabfff640 1 -- 172.21.15.107:0/820474228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c009b30 con 0x7f2fac0c59b0 2023-11-24T08:05:31.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.156+0000 7f2fa37fe640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 0x7f2fac1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fabfff640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c000f80 tx=0x7f2f9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009510 con 0x7f2fac0c59b0 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c03e450 con 0x7f2fac0c59b0 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c03e800 con 0x7f2fac0c59b0 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac159420 con 0x7f2fac0c59b0 2023-11-24T08:05:31.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.157+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac1599e0 con 0x7f2fac0c59b0 2023-11-24T08:05:31.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.158+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac04e8b0 con 0x7f2fac0c59b0 2023-11-24T08:05:31.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.159+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2f9c02cd50 con 0x7f2fac0c59b0 2023-11-24T08:05:31.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.160+0000 7f2fa97fa640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 0x7f2f8c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.160+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f9c0ac170 con 0x7f2fac0c59b0 2023-11-24T08:05:31.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.160+0000 7f2fa37fe640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 0x7f2f8c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.161+0000 7f2fa37fe640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 0x7f2f8c06d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2fac159e50 tx=0x7f2f94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.164+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f9c077220 con 0x7f2fac0c59b0 2023-11-24T08:05:31.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.355+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fac15a600 con 0x7f2fac0c59b0 2023-11-24T08:05:31.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.356+0000 7f2fa97fa640 1 -- 172.21.15.107:0/820474228 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f9c07a840 con 0x7f2fac0c59b0 2023-11-24T08:05:31.358 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 msgr2=0x7f2f8c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 0x7f2f8c06d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2fac159e50 tx=0x7f2f94009040 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 msgr2=0x7f2fac0ad990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c000f80 tx=0x7f2f9c004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 shutdown_connections 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f8c06ae40 0x7f2f8c06d300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fac15b440 0x7f2fac0ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fac0cbf20 0x7f2fac1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 --2- 172.21.15.107:0/820474228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fac0c59b0 0x7f2fac0ad990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 >> 172.21.15.107:0/820474228 conn(0x7f2fac0b0c50 msgr2=0x7f2fac0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 shutdown_connections 2023-11-24T08:05:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.359+0000 7f2fb2961640 1 -- 172.21.15.107:0/820474228 wait complete. 2023-11-24T08:05:31.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.585+0000 7f2b17135640 1 Processor -- start 2023-11-24T08:05:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.585+0000 7f2b17135640 1 -- start start 2023-11-24T08:05:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b17135640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b17135640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b1015ded0 0x7f2b1015e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b17135640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1006d430 con 0x7f2b1015a1c0 2023-11-24T08:05:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b17135640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1015ec50 con 0x7f2b1015ded0 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b17135640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1006cb00 con 0x7f2b100b0b40 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b1015ded0 0x7f2b1015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b1015ded0 0x7f2b1015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52988/0 (socket says 172.21.15.107:52988) 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 -- 172.21.15.107:0/3561174910 learned_addr learned my addr 172.21.15.107:0/3561174910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b15932640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 -- 172.21.15.107:0/3561174910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 msgr2=0x7f2b100b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 -- 172.21.15.107:0/3561174910 >> v1:172.21.15.120:6789/0 conn(0x7f2b1015a1c0 legacy=0x7f2b1006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.587+0000 7f2b16934640 1 -- 172.21.15.107:0/3561174910 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100b14e0 con 0x7f2b1015ded0 2023-11-24T08:05:31.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b16934640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b1015e360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009e50 tx=0x7f2b0802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95eccaad022b8256 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b15131640 1 -- 172.21.15.107:0/3561174910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0802ca70 con 0x7f2b1015ded0 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b15131640 1 -- 172.21.15.107:0/3561174910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0802cbd0 con 0x7f2b1015ded0 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b15131640 1 -- 172.21.15.107:0/3561174910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b080343f0 con 0x7f2b1015ded0 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b17135640 1 -- 172.21.15.107:0/3561174910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 msgr2=0x7f2b1015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.589+0000 7f2b17135640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b1015e360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009e50 tx=0x7f2b0802bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 -- 172.21.15.107:0/3561174910 shutdown_connections 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b1015e360 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 --2- 172.21.15.107:0/3561174910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 -- 172.21.15.107:0/3561174910 >> 172.21.15.107:0/3561174910 conn(0x7f2b100cbb40 msgr2=0x7f2b100cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 -- 172.21.15.107:0/3561174910 shutdown_connections 2023-11-24T08:05:31.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.590+0000 7f2b17135640 1 -- 172.21.15.107:0/3561174910 wait complete. 2023-11-24T08:05:31.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.592+0000 7f2b17135640 1 Processor -- start 2023-11-24T08:05:31.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.592+0000 7f2b17135640 1 -- start start 2023-11-24T08:05:31.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b1015a1c0 0x7f2b100c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100ae490 con 0x7f2b1015ded0 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100ae730 con 0x7f2b1015a1c0 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b17135640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100aea10 con 0x7f2b100b0b40 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b16934640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b16934640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53002/0 (socket says 172.21.15.107:53002) 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.593+0000 7f2b16934640 1 -- 172.21.15.107:0/2520573075 learned_addr learned my addr 172.21.15.107:0/2520573075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b15932640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b1015a1c0 0x7f2b100c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16133640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 -- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 msgr2=0x7f2b100ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 -- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b1015a1c0 msgr2=0x7f2b100c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b1015a1c0 0x7f2b100c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 -- 172.21.15.107:0/2520573075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08009b30 con 0x7f2b1015ded0 2023-11-24T08:05:31.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b16934640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009e50 tx=0x7f2b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08040670 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b080407d0 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08040b80 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.594+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1015ea20 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.595+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b100acd90 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.596+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b100aef30 con 0x7f2b1015ded0 2023-11-24T08:05:31.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.596+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2b08040ce0 con 0x7f2b1015ded0 2023-11-24T08:05:31.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.597+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 0x7f2af406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:31.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.597+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b080ac700 con 0x7f2b1015ded0 2023-11-24T08:05:31.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.598+0000 7f2b16133640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 0x7f2af406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:31.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.598+0000 7f2b16133640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 0x7f2af406d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c003d90 tx=0x7f2b0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:31.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.601+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b100aef30 con 0x7f2b1015ded0 2023-11-24T08:05:31.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.792+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b100aef30 con 0x7f2b1015ded0 2023-11-24T08:05:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.793+0000 7f2b06ffd640 1 -- 172.21.15.107:0/2520573075 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b100aef30 con 0x7f2b1015ded0 2023-11-24T08:05:31.795 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 msgr2=0x7f2af406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 0x7f2af406d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c003d90 tx=0x7f2b0c009040 comp rx=0 tx=0).stop 2023-11-24T08:05:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 msgr2=0x7f2b10158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009e50 tx=0x7f2b08004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 shutdown_connections 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2af406ae40 0x7f2af406d300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b1015ded0 0x7f2b10158fd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b1015a1c0 0x7f2b100c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 --2- 172.21.15.107:0/2520573075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b100b0b40 0x7f2b100ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 >> 172.21.15.107:0/2520573075 conn(0x7f2b100cbb40 msgr2=0x7f2b10070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.796+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 shutdown_connections 2023-11-24T08:05:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:31.797+0000 7f2b17135640 1 -- 172.21.15.107:0/2520573075 wait complete. 2023-11-24T08:05:31.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.027+0000 7f55b809b640 1 Processor -- start 2023-11-24T08:05:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.027+0000 7f55b809b640 1 -- start start 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.028+0000 7f55b809b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55b006de10 0x7f55b015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b809b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b015ec90 con 0x7f55b0072160 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b809b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b00cd530 con 0x7f55b00c49f0 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b809b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b00c4ed0 con 0x7f55b006de10 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5e37640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f55b0072160 0x7f55b006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57640/0 (socket says 172.21.15.107:57640) 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5e37640 1 -- 172.21.15.107:0/4258798946 learned_addr learned my addr 172.21.15.107:0/4258798946 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5636640 1 --2- 172.21.15.107:0/4258798946 >> v2:172.21.15.107:3300/0 conn(0x7f55b006de10 0x7f55b015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b4e35640 1 -- 172.21.15.107:0/4258798946 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761096105 0 0) 0x7f55ac001110 con 0x7f55b0072160 2023-11-24T08:05:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b4e35640 1 -- 172.21.15.107:0/4258798946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c003520 con 0x7f55b0072160 2023-11-24T08:05:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5636640 1 -- 172.21.15.107:0/4258798946 >> v1:172.21.15.120:6789/0 conn(0x7f55b00c49f0 legacy=0x7f55b00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5636640 1 -- 172.21.15.107:0/4258798946 >> v1:172.21.15.107:6789/0 conn(0x7f55b0072160 legacy=0x7f55b006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.029+0000 7f55b5636640 1 -- 172.21.15.107:0/4258798946 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b00c51b0 con 0x7f55b006de10 2023-11-24T08:05:32.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b5636640 1 --2- 172.21.15.107:0/4258798946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b006de10 0x7f55b015e750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f55a00097a0 tx=0x7f55a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a969a89f09d6d5ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b4e35640 1 -- 172.21.15.107:0/4258798946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0031830 con 0x7f55b006de10 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b4e35640 1 -- 172.21.15.107:0/4258798946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a0031990 con 0x7f55b006de10 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b4e35640 1 -- 172.21.15.107:0/4258798946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0031d20 con 0x7f55b006de10 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b809b640 1 -- 172.21.15.107:0/4258798946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b006de10 msgr2=0x7f55b015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.031+0000 7f55b809b640 1 --2- 172.21.15.107:0/4258798946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b006de10 0x7f55b015e750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f55a00097a0 tx=0x7f55a002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.032+0000 7f55b809b640 1 -- 172.21.15.107:0/4258798946 shutdown_connections 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.032+0000 7f55b809b640 1 --2- 172.21.15.107:0/4258798946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b006de10 0x7f55b015e750 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.032+0000 7f55b809b640 1 -- 172.21.15.107:0/4258798946 >> 172.21.15.107:0/4258798946 conn(0x7f55b00b0c50 msgr2=0x7f55b00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.032+0000 7f55b809b640 1 -- 172.21.15.107:0/4258798946 shutdown_connections 2023-11-24T08:05:32.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.032+0000 7f55b809b640 1 -- 172.21.15.107:0/4258798946 wait complete. 2023-11-24T08:05:32.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.033+0000 7f55b809b640 1 Processor -- start 2023-11-24T08:05:32.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.033+0000 7f55b809b640 1 -- start start 2023-11-24T08:05:32.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55b006de10 0x7f55b015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 0x7f55b00acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b0152570 con 0x7f55b0072160 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b015ccb0 con 0x7f55b00c49f0 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.034+0000 7f55b809b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b01f7560 con 0x7f55b006de10 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53022/0 (socket says 172.21.15.107:53022) 2023-11-24T08:05:32.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 -- 172.21.15.107:0/3198423392 learned_addr learned my addr 172.21.15.107:0/3198423392 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:32.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5e37640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55b006de10 0x7f55b015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 -- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55b006de10 msgr2=0x7f55b015c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b6638640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 0x7f55b00acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55b006de10 0x7f55b015c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 -- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 msgr2=0x7f55b00acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 0x7f55b00acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b5636640 1 -- 172.21.15.107:0/3198423392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a0009400 con 0x7f55b0072160 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.035+0000 7f55b6638640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 0x7f55b00acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55b5636640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f55a00098c0 tx=0x7f55a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a003a6f0 con 0x7f55b0072160 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a003a870 con 0x7f55b0072160 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a003ac20 con 0x7f55b0072160 2023-11-24T08:05:32.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b01f79f0 con 0x7f55b0072160 2023-11-24T08:05:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.036+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b01f6f10 con 0x7f55b0072160 2023-11-24T08:05:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.037+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b004e8b0 con 0x7f55b0072160 2023-11-24T08:05:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.038+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f55a0030380 con 0x7f55b0072160 2023-11-24T08:05:32.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.039+0000 7f55a67fc640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 0x7f559406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.039+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55a00ab0e0 con 0x7f55b0072160 2023-11-24T08:05:32.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.039+0000 7f55b5e37640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 0x7f559406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.040+0000 7f55b5e37640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 0x7f559406d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f55ac001370 tx=0x7f55ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.043+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55a0076210 con 0x7f55b0072160 2023-11-24T08:05:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.232+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55b015cee0 con 0x7f55b0072160 2023-11-24T08:05:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.233+0000 7f55a67fc640 1 -- 172.21.15.107:0/3198423392 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f55a0079830 con 0x7f55b0072160 2023-11-24T08:05:32.235 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.235+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 msgr2=0x7f559406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.235+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 0x7f559406d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f55ac001370 tx=0x7f55ac009040 comp rx=0 tx=0).stop 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.235+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 msgr2=0x7f55b00ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.235+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f55a00098c0 tx=0x7f55a00046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 shutdown_connections 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f559406ae40 0x7f559406d300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55b00c49f0 0x7f55b00acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55b0072160 0x7f55b00ac840 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 --2- 172.21.15.107:0/3198423392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55b006de10 0x7f55b015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 >> 172.21.15.107:0/3198423392 conn(0x7f55b00b0c50 msgr2=0x7f55b00734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 shutdown_connections 2023-11-24T08:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.236+0000 7f55b809b640 1 -- 172.21.15.107:0/3198423392 wait complete. 2023-11-24T08:05:32.340 INFO:tasks.cephfs_test_runner:Starting test: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ensure failure status is shown when clone is in failed state and validate the reason 2023-11-24T08:05:32.340 INFO:tasks.cephfs_test_runner:test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:05:32.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.560+0000 7fdd527be640 1 Processor -- start 2023-11-24T08:05:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.560+0000 7fdd527be640 1 -- start start 2023-11-24T08:05:32.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.561+0000 7fdd527be640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdd4c0b2250 0x7fdd4c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.561+0000 7fdd527be640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd4c0b2630 0x7fdd4c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd527be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd527be640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c15a990 con 0x7fdd4c0b2250 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd527be640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c15ab80 con 0x7fdd4c0b2630 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd527be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c15e680 con 0x7fdd4c0b2aa0 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdd4c0b2250 0x7fdd4c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdd4c0b2250 0x7fdd4c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53036/0 (socket says 172.21.15.107:53036) 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 -- 172.21.15.107:0/310324417 learned_addr learned my addr 172.21.15.107:0/310324417 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:32.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4b7fe640 1 --2- 172.21.15.107:0/310324417 >> v2:172.21.15.120:3300/0 conn(0x7fdd4c0b2630 0x7fdd4c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 -- 172.21.15.107:0/310324417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 msgr2=0x7fdd4c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd50d5b640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c15b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 -- 172.21.15.107:0/310324417 >> v2:172.21.15.120:3300/0 conn(0x7fdd4c0b2630 msgr2=0x7fdd4c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/310324417 >> v2:172.21.15.120:3300/0 conn(0x7fdd4c0b2630 0x7fdd4c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.562+0000 7fdd4bfff640 1 -- 172.21.15.107:0/310324417 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c15e960 con 0x7fdd4c0b2250 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.563+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c15df80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0ae5c0 tx=0x7fdd3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd4969c9c881eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd4affd640 1 -- 172.21.15.107:0/310324417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c031870 con 0x7fdd4c0b2250 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd4affd640 1 -- 172.21.15.107:0/310324417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0319d0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd4affd640 1 -- 172.21.15.107:0/310324417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c031d60 con 0x7fdd4c0b2250 2023-11-24T08:05:32.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd527be640 1 -- 172.21.15.107:0/310324417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 msgr2=0x7fdd4c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd527be640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c15df80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0ae5c0 tx=0x7fdd3c02b910 comp rx=0 tx=0).stop 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd527be640 1 -- 172.21.15.107:0/310324417 shutdown_connections 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.564+0000 7fdd527be640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.565+0000 7fdd527be640 1 --2- 172.21.15.107:0/310324417 >> v2:172.21.15.120:3300/0 conn(0x7fdd4c0b2630 0x7fdd4c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.565+0000 7fdd527be640 1 --2- 172.21.15.107:0/310324417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c15df80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.565+0000 7fdd527be640 1 -- 172.21.15.107:0/310324417 >> 172.21.15.107:0/310324417 conn(0x7fdd4c06a920 msgr2=0x7fdd4c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.565+0000 7fdd527be640 1 -- 172.21.15.107:0/310324417 shutdown_connections 2023-11-24T08:05:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.565+0000 7fdd527be640 1 -- 172.21.15.107:0/310324417 wait complete. 2023-11-24T08:05:32.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.566+0000 7fdd527be640 1 Processor -- start 2023-11-24T08:05:32.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.566+0000 7fdd527be640 1 -- start start 2023-11-24T08:05:32.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd4c0b2630 0x7fdd4c0ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c0aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0b1560 con 0x7fdd4c0b2250 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0b17e0 con 0x7fdd4c0b2630 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.567+0000 7fdd527be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0b1ac0 con 0x7fdd4c0b2aa0 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53042/0 (socket says 172.21.15.107:53042) 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 -- 172.21.15.107:0/1199667934 learned_addr learned my addr 172.21.15.107:0/1199667934 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 -- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 msgr2=0x7fdd4c0aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c0aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4b7fe640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd4c0b2630 0x7fdd4c0ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 -- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd4c0b2630 msgr2=0x7fdd4c0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd4c0b2630 0x7fdd4c0ac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.568+0000 7fdd4bfff640 1 -- 172.21.15.107:0/1199667934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0094d0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd4bfff640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c02b950 tx=0x7fdd3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c031860 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0443f0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0447a0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c156a40 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.569+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c157000 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.570+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c1568a0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.571+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdd3c04d450 con 0x7fdd4c0b2250 2023-11-24T08:05:32.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.572+0000 7fdd48ff9640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 0x7fdd3806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:32.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.572+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd3c02d040 con 0x7fdd4c0b2250 2023-11-24T08:05:32.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.572+0000 7fdd4b7fe640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 0x7fdd3806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:32.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.573+0000 7fdd4b7fe640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 0x7fdd3806d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c154930 tx=0x7fdd40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:32.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.576+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd4c1568a0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.773+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd4c1568a0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.776+0000 7fdd48ff9640 1 -- 172.21.15.107:0/1199667934 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fdd4c1568a0 con 0x7fdd4c0b2250 2023-11-24T08:05:32.779 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:32.779 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.779+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 msgr2=0x7fdd3806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.779+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 0x7fdd3806d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c154930 tx=0x7fdd40008040 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.779+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 msgr2=0x7fdd4c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.779+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c02b950 tx=0x7fdd3c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 shutdown_connections 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3806ae40 0x7fdd3806d300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd4c0b2aa0 0x7fdd4c0aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd4c0b2630 0x7fdd4c0ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 --2- 172.21.15.107:0/1199667934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4c0b2250 0x7fdd4c0abfc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 >> 172.21.15.107:0/1199667934 conn(0x7fdd4c06a920 msgr2=0x7fdd4c158590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 shutdown_connections 2023-11-24T08:05:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:32.780+0000 7fdd527be640 1 -- 172.21.15.107:0/1199667934 wait complete. 2023-11-24T08:05:32.801 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:32.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:33.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.016+0000 7fbd3005d640 1 Processor -- start 2023-11-24T08:05:33.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.016+0000 7fbd3005d640 1 -- start start 2023-11-24T08:05:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd3005d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbd280cc0f0 0x7fbd280c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd3005d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd280c57e0 0x7fbd2806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd3005d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2815eea0 con 0x7fbd2815b440 2023-11-24T08:05:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd3005d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2815b820 con 0x7fbd280c57e0 2023-11-24T08:05:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd3005d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd280abf30 con 0x7fbd280cc0f0 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd2ddf9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd2815b440 0x7fbd280cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57642/0 (socket says 172.21.15.107:57642) 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd2ddf9640 1 -- 172.21.15.107:0/2539893638 learned_addr learned my addr 172.21.15.107:0/2539893638 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2539893638 >> v2:172.21.15.107:3300/0 conn(0x7fbd280c57e0 0x7fbd2806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.018+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2539893638 >> v2:172.21.15.120:3300/0 conn(0x7fbd280cc0f0 msgr2=0x7fbd280c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.019+0000 7fbd2d5f8640 1 --2- 172.21.15.107:0/2539893638 >> v2:172.21.15.120:3300/0 conn(0x7fbd280cc0f0 0x7fbd280c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.019+0000 7fbd2cdf7640 1 -- 172.21.15.107:0/2539893638 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204798072 0 0) 0x7fbd2815eea0 con 0x7fbd2815b440 2023-11-24T08:05:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.019+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2539893638 >> v2:172.21.15.120:3300/0 conn(0x7fbd280cc0f0 0x7fbd280c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.019+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2539893638 >> v1:172.21.15.107:6789/0 conn(0x7fbd2815b440 legacy=0x7fbd280cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.019+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2539893638 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2815f0f0 con 0x7fbd280c57e0 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2539893638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd280c57e0 0x7fbd2806c5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd2806e720 tx=0x7fbd2002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67fee5b17f62f48 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd2cdf7640 1 -- 172.21.15.107:0/2539893638 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2002cbf0 con 0x7fbd280c57e0 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd2cdf7640 1 -- 172.21.15.107:0/2539893638 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2002cd50 con 0x7fbd280c57e0 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd2cdf7640 1 -- 172.21.15.107:0/2539893638 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20034660 con 0x7fbd280c57e0 2023-11-24T08:05:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd3005d640 1 -- 172.21.15.107:0/2539893638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd280c57e0 msgr2=0x7fbd2806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.020+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2539893638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd280c57e0 0x7fbd2806c5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd2806e720 tx=0x7fbd2002be40 comp rx=0 tx=0).stop 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 -- 172.21.15.107:0/2539893638 shutdown_connections 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2539893638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd280c57e0 0x7fbd2806c5f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2539893638 >> v2:172.21.15.120:3300/0 conn(0x7fbd280cc0f0 0x7fbd280c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 -- 172.21.15.107:0/2539893638 >> 172.21.15.107:0/2539893638 conn(0x7fbd280b0c50 msgr2=0x7fbd280b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 -- 172.21.15.107:0/2539893638 shutdown_connections 2023-11-24T08:05:33.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.021+0000 7fbd3005d640 1 -- 172.21.15.107:0/2539893638 wait complete. 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.022+0000 7fbd3005d640 1 Processor -- start 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.023+0000 7fbd3005d640 1 -- start start 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.023+0000 7fbd3005d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 0x7fbd2815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.023+0000 7fbd3005d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 0x7fbd2815bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.023+0000 7fbd3005d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.023+0000 7fbd3005d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28158eb0 con 0x7fbd2815b440 2023-11-24T08:05:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd3005d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28155070 con 0x7fbd280c57e0 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd3005d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280acd10 con 0x7fbd280cc0f0 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53056/0 (socket says 172.21.15.107:53056) 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2943011558 learned_addr learned my addr 172.21.15.107:0/2943011558 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2ddf9640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 0x7fbd2815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2d5f8640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 0x7fbd2815bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 msgr2=0x7fbd2815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 0x7fbd2815bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 msgr2=0x7fbd2815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 0x7fbd2815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2e5fa640 1 -- 172.21.15.107:0/2943011558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20009a70 con 0x7fbd2815b440 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2ddf9640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 0x7fbd2815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.024+0000 7fbd2d5f8640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 0x7fbd2815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd2e5fa640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009e80 tx=0x7fbd20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20040650 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd200407d0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20040b80 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd280acff0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.025+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd28158ad0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.026+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd28158ff0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.027+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbd20040ce0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.028+0000 7fbd1e7fc640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 0x7fbd0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.028+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd200ac4c0 con 0x7fbd2815b440 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.028+0000 7fbd2ddf9640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 0x7fbd0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.029+0000 7fbd2ddf9640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 0x7fbd0c06d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f60 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.032+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd28158ff0 con 0x7fbd2815b440 2023-11-24T08:05:33.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.223+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd28158ff0 con 0x7fbd2815b440 2023-11-24T08:05:33.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.224+0000 7fbd1e7fc640 1 -- 172.21.15.107:0/2943011558 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd28158ff0 con 0x7fbd2815b440 2023-11-24T08:05:33.226 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:33.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 msgr2=0x7fbd0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 0x7fbd0c06d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f60 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2023-11-24T08:05:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 msgr2=0x7fbd280ac100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009e80 tx=0x7fbd20004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 shutdown_connections 2023-11-24T08:05:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.226+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd0c06ae40 0x7fbd0c06d300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd2815b440 0x7fbd280ac100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd280cc0f0 0x7fbd2815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 --2- 172.21.15.107:0/2943011558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd280c57e0 0x7fbd2815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 >> 172.21.15.107:0/2943011558 conn(0x7fbd280b0c50 msgr2=0x7fbd28070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 shutdown_connections 2023-11-24T08:05:33.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.227+0000 7fbd3005d640 1 -- 172.21.15.107:0/2943011558 wait complete. 2023-11-24T08:05:33.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:33.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.457+0000 7f15cffff640 1 Processor -- start 2023-11-24T08:05:33.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.457+0000 7f15cffff640 1 -- start start 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15d00cb580 0x7f15d00c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f15d00ca940 0x7f15d00c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d00cb960 con 0x7f15d00b0e90 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d00b19b0 con 0x7f15d00cb580 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d0161e40 con 0x7f15d00ca940 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15d00cb580 0x7f15d00c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15d00cb580 0x7f15d00c2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53068/0 (socket says 172.21.15.107:53068) 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 -- 172.21.15.107:0/3702989781 learned_addr learned my addr 172.21.15.107:0/3702989781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:33.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15cf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15d00b0e90 0x7f15d00b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57658/0 (socket says 172.21.15.107:57658) 2023-11-24T08:05:33.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 -- 172.21.15.107:0/3702989781 >> v2:172.21.15.120:3300/0 conn(0x7f15d00ca940 msgr2=0x7f15d00c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:33.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 --2- 172.21.15.107:0/3702989781 >> v2:172.21.15.120:3300/0 conn(0x7f15d00ca940 0x7f15d00c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ceffd640 1 -- 172.21.15.107:0/3702989781 >> v1:172.21.15.107:6789/0 conn(0x7f15d00b0e90 legacy=0x7f15d00b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.460+0000 7f15ceffd640 1 -- 172.21.15.107:0/3702989781 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d0162120 con 0x7f15d00cb580 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.459+0000 7f15ce7fc640 1 -- 172.21.15.107:0/3702989781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668890537 0 0) 0x7f15d00cb960 con 0x7f15d00b0e90 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15ceffd640 1 --2- 172.21.15.107:0/3702989781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00cb580 0x7f15d00c2090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f15d00d0160 tx=0x7f15c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7dd4654c101523 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15ce7fc640 1 -- 172.21.15.107:0/3702989781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0030860 con 0x7f15d00cb580 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15ce7fc640 1 -- 172.21.15.107:0/3702989781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c00309c0 con 0x7f15d00cb580 2023-11-24T08:05:33.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15ce7fc640 1 -- 172.21.15.107:0/3702989781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0030d50 con 0x7f15d00cb580 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15cffff640 1 -- 172.21.15.107:0/3702989781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00cb580 msgr2=0x7f15d00c2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15cffff640 1 --2- 172.21.15.107:0/3702989781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00cb580 0x7f15d00c2090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f15d00d0160 tx=0x7f15c002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.461+0000 7f15cffff640 1 -- 172.21.15.107:0/3702989781 shutdown_connections 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.462+0000 7f15cffff640 1 --2- 172.21.15.107:0/3702989781 >> v2:172.21.15.120:3300/0 conn(0x7f15d00ca940 0x7f15d00c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.462+0000 7f15cffff640 1 --2- 172.21.15.107:0/3702989781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00cb580 0x7f15d00c2090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.462+0000 7f15cffff640 1 -- 172.21.15.107:0/3702989781 >> 172.21.15.107:0/3702989781 conn(0x7f15d00cbb40 msgr2=0x7f15d00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:33.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.462+0000 7f15cffff640 1 -- 172.21.15.107:0/3702989781 shutdown_connections 2023-11-24T08:05:33.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.462+0000 7f15cffff640 1 -- 172.21.15.107:0/3702989781 wait complete. 2023-11-24T08:05:33.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.463+0000 7f15cffff640 1 Processor -- start 2023-11-24T08:05:33.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.463+0000 7f15cffff640 1 -- start start 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15d00b0e90 0x7f15d009b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15d00cb580 0x7f15d015b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c6190 con 0x7f15d00ca940 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c6470 con 0x7f15d00cb580 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.464+0000 7f15cffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c6750 con 0x7f15d00b0e90 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53082/0 (socket says 172.21.15.107:53082) 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 -- 172.21.15.107:0/575919197 learned_addr learned my addr 172.21.15.107:0/575919197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:33.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 -- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15d00b0e90 msgr2=0x7f15d009b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15d00b0e90 0x7f15d009b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 -- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15d00cb580 msgr2=0x7f15d015b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15d00cb580 0x7f15d015b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 -- 172.21.15.107:0/575919197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00094d0 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.465+0000 7f15c67fc640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f15b800b4a0 tx=0x7f15b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.466+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8010610 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.466+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b8010770 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.466+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d00c6a30 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.466+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8010b20 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.466+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d01fabf0 con 0x7f15d00ca940 2023-11-24T08:05:33.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.467+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d00c6f50 con 0x7f15d00ca940 2023-11-24T08:05:33.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.468+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f15b8010c80 con 0x7f15d00ca940 2023-11-24T08:05:33.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.469+0000 7f15c7fff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 0x7f15b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.469+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15b8089270 con 0x7f15d00ca940 2023-11-24T08:05:33.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.469+0000 7f15ceffd640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 0x7f15b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.470+0000 7f15ceffd640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 0x7f15b006d300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.473+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15d00c6f50 con 0x7f15d00ca940 2023-11-24T08:05:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.663+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15d00c6f50 con 0x7f15d00ca940 2023-11-24T08:05:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.664+0000 7f15c7fff640 1 -- 172.21.15.107:0/575919197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15d00c6f50 con 0x7f15d00ca940 2023-11-24T08:05:33.666 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 msgr2=0x7f15b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 0x7f15b006d300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c003b040 comp rx=0 tx=0).stop 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 msgr2=0x7f15d015b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f15b800b4a0 tx=0x7f15b800b960 comp rx=0 tx=0).stop 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 shutdown_connections 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15b006ae40 0x7f15b006d300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15d00cb580 0x7f15d015b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15d00ca940 0x7f15d015b420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 --2- 172.21.15.107:0/575919197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15d00b0e90 0x7f15d009b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 >> 172.21.15.107:0/575919197 conn(0x7f15d00cbb40 msgr2=0x7f15d00cf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 shutdown_connections 2023-11-24T08:05:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.667+0000 7f15cffff640 1 -- 172.21.15.107:0/575919197 wait complete. 2023-11-24T08:05:33.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.897+0000 7f0aa491d640 1 Processor -- start 2023-11-24T08:05:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.897+0000 7f0aa491d640 1 -- start start 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0aa015b440 0x7f0aa00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0aa00cbf20 0x7f0aa00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 0x7f0aa00c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa00cc3d0 con 0x7f0aa015b440 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa015b820 con 0x7f0aa00cbf20 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0aa491d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa015eea0 con 0x7f0aa00c57e0 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0a9e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0aa015b440 0x7f0aa00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0a9e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0aa015b440 0x7f0aa00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53088/0 (socket says 172.21.15.107:53088) 2023-11-24T08:05:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.899+0000 7f0a9e59c640 1 -- 172.21.15.107:0/2078418523 learned_addr learned my addr 172.21.15.107:0/2078418523 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:33.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9dd9b640 1 --2- 172.21.15.107:0/2078418523 >> v2:172.21.15.120:3300/0 conn(0x7f0aa00cbf20 0x7f0aa00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9ed9d640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 0x7f0aa00c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9e59c640 1 -- 172.21.15.107:0/2078418523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 msgr2=0x7f0aa00c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:33.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 0x7f0aa00c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9e59c640 1 -- 172.21.15.107:0/2078418523 >> v2:172.21.15.120:3300/0 conn(0x7f0aa00cbf20 msgr2=0x7f0aa00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2078418523 >> v2:172.21.15.120:3300/0 conn(0x7f0aa00cbf20 0x7f0aa00c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9e59c640 1 -- 172.21.15.107:0/2078418523 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00abf30 con 0x7f0aa015b440 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.900+0000 7f0a9ed9d640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 0x7f0aa00c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.901+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa015b440 0x7f0aa00cb9e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006e890 tx=0x7f0a9002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c19979237f83383 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.901+0000 7f0a9d59a640 1 -- 172.21.15.107:0/2078418523 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900317d0 con 0x7f0aa015b440 2023-11-24T08:05:33.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.901+0000 7f0a9d59a640 1 -- 172.21.15.107:0/2078418523 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90031930 con 0x7f0aa015b440 2023-11-24T08:05:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.901+0000 7f0a9d59a640 1 -- 172.21.15.107:0/2078418523 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90031cc0 con 0x7f0aa015b440 2023-11-24T08:05:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 -- 172.21.15.107:0/2078418523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa015b440 msgr2=0x7f0aa00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa015b440 0x7f0aa00cb9e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006e890 tx=0x7f0a9002b870 comp rx=0 tx=0).stop 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 -- 172.21.15.107:0/2078418523 shutdown_connections 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa00c57e0 0x7f0aa00c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2078418523 >> v2:172.21.15.120:3300/0 conn(0x7f0aa00cbf20 0x7f0aa00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2078418523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa015b440 0x7f0aa00cb9e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 -- 172.21.15.107:0/2078418523 >> 172.21.15.107:0/2078418523 conn(0x7f0aa00b0c50 msgr2=0x7f0aa00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 -- 172.21.15.107:0/2078418523 shutdown_connections 2023-11-24T08:05:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.902+0000 7f0aa491d640 1 -- 172.21.15.107:0/2078418523 wait complete. 2023-11-24T08:05:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.904+0000 7f0aa491d640 1 Processor -- start 2023-11-24T08:05:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.904+0000 7f0aa491d640 1 -- start start 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa00c57e0 0x7f0aa014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa015b440 0x7f0aa00b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0099ae0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0099dc0 con 0x7f0aa00c57e0 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0aa491d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa009a0a0 con 0x7f0aa015b440 2023-11-24T08:05:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0a9dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0a9dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53104/0 (socket says 172.21.15.107:53104) 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.905+0000 7f0a9dd9b640 1 -- 172.21.15.107:0/2515748877 learned_addr learned my addr 172.21.15.107:0/2515748877 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa00c57e0 0x7f0aa014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 -- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa015b440 msgr2=0x7f0aa00b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9ed9d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa015b440 0x7f0aa00b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa015b440 0x7f0aa00b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 -- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa00c57e0 msgr2=0x7f0aa014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa00c57e0 0x7f0aa014ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 -- 172.21.15.107:0/2515748877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900094d0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a9dd9b640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c7f0 tx=0x7f0a9400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94004220 con 0x7f0aa00cbf20 2023-11-24T08:05:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a940043a0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94007860 con 0x7f0aa00cbf20 2023-11-24T08:05:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.906+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa009a380 con 0x7f0aa00cbf20 2023-11-24T08:05:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.907+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa014cfa0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.908+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa00c5c30 con 0x7f0aa00cbf20 2023-11-24T08:05:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.909+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0a94007bd0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.910+0000 7f0a8effd640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 0x7f0a8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.910+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a940896e0 con 0x7f0aa00cbf20 2023-11-24T08:05:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.910+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 0x7f0a8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.911+0000 7f0a9e59c640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 0x7f0a8006d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0a90036000 tx=0x7f0a9003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:33.914+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a940546f0 con 0x7f0aa00cbf20 2023-11-24T08:05:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.105+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0aa004e8b0 con 0x7f0aa00cbf20 2023-11-24T08:05:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.106+0000 7f0a8effd640 1 -- 172.21.15.107:0/2515748877 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a94057d10 con 0x7f0aa00cbf20 2023-11-24T08:05:34.108 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 msgr2=0x7f0a8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 0x7f0a8006d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0a90036000 tx=0x7f0a9003d040 comp rx=0 tx=0).stop 2023-11-24T08:05:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 msgr2=0x7f0aa00b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c7f0 tx=0x7f0a9400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 shutdown_connections 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a8006ae40 0x7f0a8006d300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa015b440 0x7f0aa00b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa00cbf20 0x7f0aa00b42c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 --2- 172.21.15.107:0/2515748877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa00c57e0 0x7f0aa014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.108+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 >> 172.21.15.107:0/2515748877 conn(0x7f0aa00b0c50 msgr2=0x7f0aa00cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.109+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 shutdown_connections 2023-11-24T08:05:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.109+0000 7f0aa491d640 1 -- 172.21.15.107:0/2515748877 wait complete. 2023-11-24T08:05:34.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed' 2023-11-24T08:05:34.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.330+0000 7f38eaabb640 1 Processor -- start 2023-11-24T08:05:34.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.330+0000 7f38eaabb640 1 -- start start 2023-11-24T08:05:34.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e4144b30 0x7f38e415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 0x7f38e40b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38e40b1de0 0x7f38e40b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e40b27b0 con 0x7f38e40b1de0 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e415bf80 con 0x7f38e415c410 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.331+0000 7f38eaabb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40b1560 con 0x7f38e4144b30 2023-11-24T08:05:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38e40b1de0 0x7f38e40b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38e40b1de0 0x7f38e40b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53112/0 (socket says 172.21.15.107:53112) 2023-11-24T08:05:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 -- 172.21.15.107:0/2773623857 learned_addr learned my addr 172.21.15.107:0/2773623857 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38e92b8640 1 --2- 172.21.15.107:0/2773623857 >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 0x7f38e40b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38e9ab9640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e4144b30 0x7f38e415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 -- 172.21.15.107:0/2773623857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e4144b30 msgr2=0x7f38e415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e4144b30 0x7f38e415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 -- 172.21.15.107:0/2773623857 >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 msgr2=0x7f38e40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 --2- 172.21.15.107:0/2773623857 >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 0x7f38e40b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38ea2ba640 1 -- 172.21.15.107:0/2773623857 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e40b0fb0 con 0x7f38e40b1de0 2023-11-24T08:05:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.332+0000 7f38e92b8640 1 --2- 172.21.15.107:0/2773623857 >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 0x7f38e40b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.333+0000 7f38ea2ba640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e40b1de0 0x7f38e40b2270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f38e406e820 tx=0x7f38dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1187239f384afc97 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38e8ab7640 1 -- 172.21.15.107:0/2773623857 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc02ca70 con 0x7f38e40b1de0 2023-11-24T08:05:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38e8ab7640 1 -- 172.21.15.107:0/2773623857 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc02cbd0 con 0x7f38e40b1de0 2023-11-24T08:05:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38e8ab7640 1 -- 172.21.15.107:0/2773623857 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0343f0 con 0x7f38e40b1de0 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 -- 172.21.15.107:0/2773623857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e40b1de0 msgr2=0x7f38e40b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e40b1de0 0x7f38e40b2270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f38e406e820 tx=0x7f38dc02bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 -- 172.21.15.107:0/2773623857 shutdown_connections 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e40b1de0 0x7f38e40b2270 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 --2- 172.21.15.107:0/2773623857 >> v2:172.21.15.120:3300/0 conn(0x7f38e415c410 0x7f38e40b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 --2- 172.21.15.107:0/2773623857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e4144b30 0x7f38e415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 -- 172.21.15.107:0/2773623857 >> 172.21.15.107:0/2773623857 conn(0x7f38e40cb830 msgr2=0x7f38e40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 -- 172.21.15.107:0/2773623857 shutdown_connections 2023-11-24T08:05:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.334+0000 7f38eaabb640 1 -- 172.21.15.107:0/2773623857 wait complete. 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.336+0000 7f38eaabb640 1 Processor -- start 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.336+0000 7f38eaabb640 1 -- start start 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 0x7f38e415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e415c410 0x7f38e4142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e41372c0 con 0x7f38e4144b30 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e4137b20 con 0x7f38e40b1de0 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38eaabb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e415b0e0 con 0x7f38e415c410 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38e92b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38e92b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53114/0 (socket says 172.21.15.107:53114) 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.337+0000 7f38e92b8640 1 -- 172.21.15.107:0/1727988231 learned_addr learned my addr 172.21.15.107:0/1727988231 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e9ab9640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 0x7f38e415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38ea2ba640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e415c410 0x7f38e4142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 -- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e415c410 msgr2=0x7f38e4142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e415c410 0x7f38e4142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 -- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 msgr2=0x7f38e415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 0x7f38e415aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 -- 172.21.15.107:0/1727988231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc009b30 con 0x7f38e4144b30 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e9ab9640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 0x7f38e415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38e92b8640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38d400c7f0 tx=0x7f38d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4007820 con 0x7f38e4144b30 2023-11-24T08:05:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d4007980 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4007cf0 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.338+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e413a7b0 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.339+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e41374d0 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.340+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e40b3610 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.340+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f38d40113f0 con 0x7f38e4144b30 2023-11-24T08:05:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.341+0000 7f38da7fc640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 0x7f38c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.341+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38d40899f0 con 0x7f38e4144b30 2023-11-24T08:05:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.342+0000 7f38e9ab9640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 0x7f38c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.342+0000 7f38e9ab9640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 0x7f38c806d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006fd0 tx=0x7f38e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.346+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38e40b3610 con 0x7f38e4144b30 2023-11-24T08:05:34.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.612+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]} v 0) v1 -- 0x7f38e40b3610 con 0x7f38e4144b30 2023-11-24T08:05:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.784+0000 7f38da7fc640 1 -- 172.21.15.107:0/1727988231 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]}]=0 v61) v1 ==== 168+0+0 (secure 0 0 0) 0x7f38e40b3610 con 0x7f38e4144b30 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 msgr2=0x7f38c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 0x7f38c806d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006fd0 tx=0x7f38e0008040 comp rx=0 tx=0).stop 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 msgr2=0x7f38e413a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38d400c7f0 tx=0x7f38d400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 shutdown_connections 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38c806ae40 0x7f38c806d300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38e415c410 0x7f38e4142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38e4144b30 0x7f38e413a270 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 --2- 172.21.15.107:0/1727988231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e40b1de0 0x7f38e415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:34.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 >> 172.21.15.107:0/1727988231 conn(0x7f38e40cb830 msgr2=0x7f38e40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:34.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 shutdown_connections 2023-11-24T08:05:34.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:34.789+0000 7f38eaabb640 1 -- 172.21.15.107:0/1727988231 wait complete. 2023-11-24T08:05:34.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:05:35.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.015+0000 7f2cc9b1d640 1 Processor -- start 2023-11-24T08:05:35.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.016+0000 7f2cc9b1d640 1 -- start start 2023-11-24T08:05:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.017+0000 7f2cc9b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2cc40c59b0 0x7f2cc4166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.017+0000 7f2cc9b1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40c5dc0 con 0x7f2cc415b440 2023-11-24T08:05:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.017+0000 7f2cc9b1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc415eea0 con 0x7f2cc40cc0f0 2023-11-24T08:05:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.017+0000 7f2cc9b1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc415b820 con 0x7f2cc40c59b0 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.017+0000 7f2cc3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2cc40c59b0 0x7f2cc4166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.018+0000 7f2cc37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2cc415b440 0x7f2cc40cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57662/0 (socket says 172.21.15.107:57662) 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.018+0000 7f2cc37fe640 1 -- 172.21.15.107:0/4151302854 learned_addr learned my addr 172.21.15.107:0/4151302854 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.018+0000 7f2cc3fff640 1 -- 172.21.15.107:0/4151302854 >> v1:172.21.15.120:6789/0 conn(0x7f2cc40cc0f0 legacy=0x7f2cc40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.018+0000 7f2cc3fff640 1 -- 172.21.15.107:0/4151302854 >> v1:172.21.15.107:6789/0 conn(0x7f2cc415b440 legacy=0x7f2cc40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.018+0000 7f2cc3fff640 1 -- 172.21.15.107:0/4151302854 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc406d600 con 0x7f2cc40c59b0 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.019+0000 7f2cc3fff640 1 --2- 172.21.15.107:0/4151302854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc4166230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0097a0 tx=0x7f2cbc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aac03931b8e94af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.019+0000 7f2cc27fc640 1 -- 172.21.15.107:0/4151302854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc02e070 con 0x7f2cc40c59b0 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.019+0000 7f2cc27fc640 1 -- 172.21.15.107:0/4151302854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc0315a0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.019+0000 7f2cc27fc640 1 -- 172.21.15.107:0/4151302854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc031930 con 0x7f2cc40c59b0 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/4151302854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 msgr2=0x7f2cc4166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/4151302854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc4166230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0097a0 tx=0x7f2cbc02b840 comp rx=0 tx=0).stop 2023-11-24T08:05:35.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/4151302854 shutdown_connections 2023-11-24T08:05:35.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/4151302854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc4166230 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/4151302854 >> 172.21.15.107:0/4151302854 conn(0x7f2cc40b0c50 msgr2=0x7f2cc40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:35.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/4151302854 shutdown_connections 2023-11-24T08:05:35.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.020+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/4151302854 wait complete. 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.022+0000 7f2cc9b1d640 1 Processor -- start 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.022+0000 7f2cc9b1d640 1 -- start start 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2cc40cc0f0 0x7f2cc4156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cc415b440 0x7f2cc4157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40ad2c0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40ad5a0 con 0x7f2cc415b440 2023-11-24T08:05:35.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc9b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40ad880 con 0x7f2cc40cc0f0 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53130/0 (socket says 172.21.15.107:53130) 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 -- 172.21.15.107:0/3180398388 learned_addr learned my addr 172.21.15.107:0/3180398388 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 -- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2cc40cc0f0 msgr2=0x7f2cc4156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc2ffd640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2cc40cc0f0 0x7f2cc4156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2cc40cc0f0 0x7f2cc4156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 -- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cc415b440 msgr2=0x7f2cc4157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cc415b440 0x7f2cc4157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.023+0000 7f2cc37fe640 1 -- 172.21.15.107:0/3180398388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009400 con 0x7f2cc40c59b0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cc37fe640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400cbc0 tx=0x7f2cb400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400aca0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb400ae00 con 0x7f2cc40c59b0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4012600 con 0x7f2cc40c59b0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc40adb60 con 0x7f2cc40c59b0 2023-11-24T08:05:35.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.024+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40ae120 con 0x7f2cc40c59b0 2023-11-24T08:05:35.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.025+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc40c97e0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.026+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2cb4007070 con 0x7f2cc40c59b0 2023-11-24T08:05:35.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.027+0000 7f2cbbfff640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 0x7f2ca406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.027+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cb40894a0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.027+0000 7f2cc2ffd640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 0x7f2ca406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.028+0000 7f2cc2ffd640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 0x7f2ca406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2cc409c860 tx=0x7f2cac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.032+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cb4054560 con 0x7f2cc40c59b0 2023-11-24T08:05:35.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.222+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2cc404e8b0 con 0x7f2cc40c59b0 2023-11-24T08:05:35.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.228+0000 7f2cbbfff640 1 -- 172.21.15.107:0/3180398388 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2cb4057b80 con 0x7f2cc40c59b0 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 msgr2=0x7f2ca406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 0x7f2ca406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2cc409c860 tx=0x7f2cac009040 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 msgr2=0x7f2cc41541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400cbc0 tx=0x7f2cb400a060 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 shutdown_connections 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ca406ae40 0x7f2ca406d300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cc415b440 0x7f2cc4157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2cc40cc0f0 0x7f2cc4156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 --2- 172.21.15.107:0/3180398388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc40c59b0 0x7f2cc41541a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 >> 172.21.15.107:0/3180398388 conn(0x7f2cc40b0c50 msgr2=0x7f2cc40c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:35.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 shutdown_connections 2023-11-24T08:05:35.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.231+0000 7f2cc9b1d640 1 -- 172.21.15.107:0/3180398388 wait complete. 2023-11-24T08:05:35.251 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:05:35.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:05:35.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:05:35.490 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:05:35.490 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:35.490 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:35.550 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:35.551 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:35.602 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:35.602 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:35.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.914+0000 7f92bf5c5640 1 Processor -- start 2023-11-24T08:05:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.915+0000 7f92bf5c5640 1 -- start start 2023-11-24T08:05:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f92b815d5d0 0x7f92b80c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b815d9b0 0x7f92b806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f92b806ed10 0x7f92b80d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b815eac0 con 0x7f92b815d5d0 2023-11-24T08:05:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b815deb0 con 0x7f92b806ed10 2023-11-24T08:05:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.916+0000 7f92bf5c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b815e190 con 0x7f92b815d9b0 2023-11-24T08:05:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f92b815d5d0 0x7f92b80c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f92b815d5d0 0x7f92b80c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53146/0 (socket says 172.21.15.107:53146) 2023-11-24T08:05:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 -- 172.21.15.107:0/4285825320 learned_addr learned my addr 172.21.15.107:0/4285825320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 -- 172.21.15.107:0/4285825320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b815d9b0 msgr2=0x7f92b806e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 --2- 172.21.15.107:0/4285825320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b815d9b0 0x7f92b806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 -- 172.21.15.107:0/4285825320 >> v2:172.21.15.120:3300/0 conn(0x7f92b806ed10 msgr2=0x7f92b80d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 --2- 172.21.15.107:0/4285825320 >> v2:172.21.15.120:3300/0 conn(0x7f92b806ed10 0x7f92b80d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.917+0000 7f92bd361640 1 -- 172.21.15.107:0/4285825320 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b815e470 con 0x7f92b815d5d0 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.918+0000 7f92bd361640 1 --2- 172.21.15.107:0/4285825320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80c66a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f92b80d2270 tx=0x7f92ac02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822e7832e8f2e72c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92b7fff640 1 -- 172.21.15.107:0/4285825320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac02c960 con 0x7f92b815d5d0 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92b7fff640 1 -- 172.21.15.107:0/4285825320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac02cac0 con 0x7f92b815d5d0 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92b7fff640 1 -- 172.21.15.107:0/4285825320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac02ce50 con 0x7f92b815d5d0 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 -- 172.21.15.107:0/4285825320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 msgr2=0x7f92b80c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/4285825320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80c66a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f92b80d2270 tx=0x7f92ac02bdc0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 -- 172.21.15.107:0/4285825320 shutdown_connections 2023-11-24T08:05:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/4285825320 >> v2:172.21.15.120:3300/0 conn(0x7f92b806ed10 0x7f92b80d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/4285825320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b815d9b0 0x7f92b806e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/4285825320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80c66a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.919+0000 7f92bf5c5640 1 -- 172.21.15.107:0/4285825320 >> 172.21.15.107:0/4285825320 conn(0x7f92b80d12d0 msgr2=0x7f92b80d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.920+0000 7f92bf5c5640 1 -- 172.21.15.107:0/4285825320 shutdown_connections 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.920+0000 7f92bf5c5640 1 -- 172.21.15.107:0/4285825320 wait complete. 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.921+0000 7f92bf5c5640 1 Processor -- start 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.921+0000 7f92bf5c5640 1 -- start start 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 0x7f92b80cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 0x7f92b8153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80b5410 con 0x7f92b815d5d0 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b8153cc0 con 0x7f92b815d9b0 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.922+0000 7f92bf5c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80b5640 con 0x7f92b806ed10 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53150/0 (socket says 172.21.15.107:53150) 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 -- 172.21.15.107:0/979578008 learned_addr learned my addr 172.21.15.107:0/979578008 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bd361640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 0x7f92b80cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bdb62640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 0x7f92b8153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 -- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 msgr2=0x7f92b80cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 0x7f92b80cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 -- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 msgr2=0x7f92b8153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 0x7f92b8153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bcb60640 1 -- 172.21.15.107:0/979578008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac009b30 con 0x7f92b815d5d0 2023-11-24T08:05:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bdb62640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 0x7f92b8153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.923+0000 7f92bd361640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 0x7f92b80cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92bcb60640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92a400b4a0 tx=0x7f92a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4010610 con 0x7f92b815d5d0 2023-11-24T08:05:35.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4010770 con 0x7f92b815d5d0 2023-11-24T08:05:35.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4010b20 con 0x7f92b815d5d0 2023-11-24T08:05:35.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b80b5920 con 0x7f92b815d5d0 2023-11-24T08:05:35.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.924+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b80b5ee0 con 0x7f92b815d5d0 2023-11-24T08:05:35.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.925+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b806c840 con 0x7f92b815d5d0 2023-11-24T08:05:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.926+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92a4010c80 con 0x7f92b815d5d0 2023-11-24T08:05:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.927+0000 7f92b5ffb640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 0x7f929c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.927+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92a4089360 con 0x7f92b815d5d0 2023-11-24T08:05:35.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.928+0000 7f92bd361640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 0x7f929c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:35.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.928+0000 7f92bd361640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 0x7f929c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92ac035000 tx=0x7f92ac030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:35.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:35.932+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92a40542e0 con 0x7f92b815d5d0 2023-11-24T08:05:36.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.124+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b8067b80 con 0x7f92b815d5d0 2023-11-24T08:05:36.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.127+0000 7f92b5ffb640 1 -- 172.21.15.107:0/979578008 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f92a4014090 con 0x7f92b815d5d0 2023-11-24T08:05:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:36.130 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:04:55.527438+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:36.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.129+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 msgr2=0x7f929c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.129+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 0x7f929c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92ac035000 tx=0x7f92ac030000 comp rx=0 tx=0).stop 2023-11-24T08:05:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.129+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 msgr2=0x7f92b80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.129+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92a400b4a0 tx=0x7f92a400b960 comp rx=0 tx=0).stop 2023-11-24T08:05:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 shutdown_connections 2023-11-24T08:05:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f929c06ae40 0x7f929c06d300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92b815d9b0 0x7f92b8153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92b815d5d0 0x7f92b80b4ed0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 --2- 172.21.15.107:0/979578008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92b806ed10 0x7f92b80cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 >> 172.21.15.107:0/979578008 conn(0x7f92b80d12d0 msgr2=0x7f92b80b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:36.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 shutdown_connections 2023-11-24T08:05:36.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.130+0000 7f92bf5c5640 1 -- 172.21.15.107:0/979578008 wait complete. 2023-11-24T08:05:36.152 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:36.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.368+0000 7f5a5925c640 1 Processor -- start 2023-11-24T08:05:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.368+0000 7f5a5925c640 1 -- start start 2023-11-24T08:05:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.369+0000 7f5a5925c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a5415b440 0x7f5a540cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.369+0000 7f5a5925c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540cbf20 0x7f5a540c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.369+0000 7f5a5925c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5415eea0 con 0x7f5a540c57e0 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.369+0000 7f5a5925c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540cc3d0 con 0x7f5a5415b440 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.369+0000 7f5a5925c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5415b820 con 0x7f5a540cbf20 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a5415b440 0x7f5a540cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a5415b440 0x7f5a540cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53156/0 (socket says 172.21.15.107:53156) 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 -- 172.21.15.107:0/576095213 learned_addr learned my addr 172.21.15.107:0/576095213 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 -- 172.21.15.107:0/576095213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540cbf20 msgr2=0x7f5a540c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/576095213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540cbf20 0x7f5a540c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 -- 172.21.15.107:0/576095213 >> v1:172.21.15.120:6789/0 conn(0x7f5a540c57e0 legacy=0x7f5a540c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.370+0000 7f5a52d9d640 1 -- 172.21.15.107:0/576095213 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5406d600 con 0x7f5a5415b440 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.371+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/576095213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a540cb9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009870 tx=0x7f5a3c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c103a5dcacc913 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.371+0000 7f5a51d9b640 1 -- 172.21.15.107:0/576095213 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0317c0 con 0x7f5a5415b440 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.371+0000 7f5a51d9b640 1 -- 172.21.15.107:0/576095213 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c031920 con 0x7f5a5415b440 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.371+0000 7f5a51d9b640 1 -- 172.21.15.107:0/576095213 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c031cb0 con 0x7f5a5415b440 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 -- 172.21.15.107:0/576095213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 msgr2=0x7f5a540cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 --2- 172.21.15.107:0/576095213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a540cb9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009870 tx=0x7f5a3c02b860 comp rx=0 tx=0).stop 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 -- 172.21.15.107:0/576095213 shutdown_connections 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 --2- 172.21.15.107:0/576095213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540cbf20 0x7f5a540c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 --2- 172.21.15.107:0/576095213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a540cb9e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 -- 172.21.15.107:0/576095213 >> 172.21.15.107:0/576095213 conn(0x7f5a540b0c50 msgr2=0x7f5a540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:36.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 -- 172.21.15.107:0/576095213 shutdown_connections 2023-11-24T08:05:36.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.372+0000 7f5a5925c640 1 -- 172.21.15.107:0/576095213 wait complete. 2023-11-24T08:05:36.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.374+0000 7f5a5925c640 1 Processor -- start 2023-11-24T08:05:36.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.374+0000 7f5a5925c640 1 -- start start 2023-11-24T08:05:36.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 0x7f5a5409a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a540cbf20 0x7f5a540b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5409a000 con 0x7f5a5415b440 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540999d0 con 0x7f5a540cbf20 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5925c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a541f1af0 con 0x7f5a540c57e0 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53162/0 (socket says 172.21.15.107:53162) 2023-11-24T08:05:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.375+0000 7f5a5359e640 1 -- 172.21.15.107:0/2648778055 learned_addr learned my addr 172.21.15.107:0/2648778055 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 0x7f5a5409a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5259c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a540cbf20 0x7f5a540b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 -- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 msgr2=0x7f5a5409a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 0x7f5a5409a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 -- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a540cbf20 msgr2=0x7f5a540b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a540cbf20 0x7f5a540b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 -- 172.21.15.107:0/2648778055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0094d0 con 0x7f5a5415b440 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 0x7f5a5409a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5359e640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00c390 tx=0x7f5a4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0035f0 con 0x7f5a5415b440 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4c003750 con 0x7f5a5415b440 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a541f1dd0 con 0x7f5a5415b440 2023-11-24T08:05:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c003b00 con 0x7f5a5415b440 2023-11-24T08:05:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.376+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a540b3630 con 0x7f5a5415b440 2023-11-24T08:05:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.377+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5409a1b0 con 0x7f5a5415b440 2023-11-24T08:05:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.378+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5a4c003c60 con 0x7f5a5415b440 2023-11-24T08:05:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.379+0000 7f5a4b7fe640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 0x7f5a3406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.379+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5a4c088c50 con 0x7f5a5415b440 2023-11-24T08:05:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.379+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 0x7f5a3406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.380+0000 7f5a52d9d640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 0x7f5a3406d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c036000 tx=0x7f5a3c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.383+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a5409a1b0 con 0x7f5a5415b440 2023-11-24T08:05:36.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.574+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a5404e8b0 con 0x7f5a5415b440 2023-11-24T08:05:36.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.575+0000 7f5a4b7fe640 1 -- 172.21.15.107:0/2648778055 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a4c017020 con 0x7f5a5415b440 2023-11-24T08:05:36.577 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:36.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.577+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 msgr2=0x7f5a3406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.577+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 0x7f5a3406d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c036000 tx=0x7f5a3c03d040 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.577+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 msgr2=0x7f5a541f0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.577+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00c390 tx=0x7f5a4c00c850 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 shutdown_connections 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a3406ae70 0x7f5a3406d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a5415b440 0x7f5a541f0fc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a540cbf20 0x7f5a540b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 --2- 172.21.15.107:0/2648778055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a540c57e0 0x7f5a5409a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 >> 172.21.15.107:0/2648778055 conn(0x7f5a540b0c50 msgr2=0x7f5a540b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:36.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 shutdown_connections 2023-11-24T08:05:36.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.578+0000 7f5a5925c640 1 -- 172.21.15.107:0/2648778055 wait complete. 2023-11-24T08:05:36.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:36.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.812+0000 7f5455851640 1 Processor -- start 2023-11-24T08:05:36.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.813+0000 7f5455851640 1 -- start start 2023-11-24T08:05:36.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.814+0000 7f5455851640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f545015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.814+0000 7f5455851640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545015ec90 con 0x7f5450072160 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f5455851640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500cd530 con 0x7f545006de10 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f5455851640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54500c4ed0 con 0x7f54500c49f0 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f544effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5450072160 0x7f545006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57670/0 (socket says 172.21.15.107:57670) 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f544effd640 1 -- 172.21.15.107:0/4273619880 learned_addr learned my addr 172.21.15.107:0/4273619880 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986368872 0 0) 0x7f5440001490 con 0x7f5450072160 2023-11-24T08:05:36.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c005730 con 0x7f5450072160 2023-11-24T08:05:36.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.815+0000 7f544f7fe640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f545015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377038494 0 0) 0x7f543c005730 con 0x7f5450072160 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 msgr2=0x7f545015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f545015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 >> v1:172.21.15.120:6789/0 conn(0x7f545006de10 legacy=0x7f54500705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500c51b0 con 0x7f5450072160 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5440001b80 con 0x7f5450072160 2023-11-24T08:05:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 >> v1:172.21.15.107:6789/0 conn(0x7f5450072160 legacy=0x7f545006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f543c005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f543c0061e0 0x7f543c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c006f10 0x7f543c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500c4ed0 con 0x7f543c0061e0 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c00dd50 con 0x7f543c006f10 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c00df10 con 0x7f54500c49f0 2023-11-24T08:05:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544e7fc640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f543c0061e0 0x7f543c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544effd640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f543c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544f7fe640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c006f10 0x7f543c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544e7fc640 1 -- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 msgr2=0x7f543c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.816+0000 7f544e7fc640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54500c49f0 0x7f543c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.817+0000 7f544e7fc640 1 -- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c006f10 msgr2=0x7f543c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.817+0000 7f544e7fc640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c006f10 0x7f543c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.817+0000 7f544e7fc640 1 -- 172.21.15.107:0/4273619880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c005730 con 0x7f543c0061e0 2023-11-24T08:05:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.817+0000 7f544f7fe640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c006f10 0x7f543c007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f544e7fc640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f543c0061e0 0x7f543c0069d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5438001750 tx=0x7f543c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ee6c947a7dc2af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54380052d0 con 0x7f543c0061e0 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543802e050 con 0x7f543c0061e0 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f544dffb640 1 -- 172.21.15.107:0/4273619880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438003830 con 0x7f543c0061e0 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f5455851640 1 -- 172.21.15.107:0/4273619880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f543c0061e0 msgr2=0x7f543c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f5455851640 1 --2- 172.21.15.107:0/4273619880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f543c0061e0 0x7f543c0069d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5438001750 tx=0x7f543c007a00 comp rx=0 tx=0).stop 2023-11-24T08:05:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f544f7fe640 1 -- 172.21.15.107:0/4273619880 reap_dead start 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f5455851640 1 -- 172.21.15.107:0/4273619880 shutdown_connections 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.818+0000 7f5455851640 1 -- 172.21.15.107:0/4273619880 >> 172.21.15.107:0/4273619880 conn(0x7f54500b0c50 msgr2=0x7f54500b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.819+0000 7f5455851640 1 -- 172.21.15.107:0/4273619880 shutdown_connections 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.819+0000 7f5455851640 1 -- 172.21.15.107:0/4273619880 wait complete. 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.820+0000 7f5455851640 1 Processor -- start 2023-11-24T08:05:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.820+0000 7f5455851640 1 -- start start 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 0x7f5450154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f545015cb60 0x7f545015cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500ce2d0 con 0x7f5450070d20 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54501561a0 con 0x7f543c0061e0 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.821+0000 7f5455851640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54500716d0 con 0x7f545015cb60 2023-11-24T08:05:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53170/0 (socket says 172.21.15.107:53170) 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 -- 172.21.15.107:0/117769497 learned_addr learned my addr 172.21.15.107:0/117769497 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544effd640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 0x7f5450154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544e7fc640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f545015cb60 0x7f545015cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 -- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f545015cb60 msgr2=0x7f545015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f545015cb60 0x7f545015cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 -- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 msgr2=0x7f5450154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 0x7f5450154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 -- 172.21.15.107:0/117769497 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545015d4a0 con 0x7f5450070d20 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544effd640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 0x7f5450154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.822+0000 7f544f7fe640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f54500c4e00 tx=0x7f544800e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.823+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544800c070 con 0x7f5450070d20 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.823+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54480136d0 con 0x7f5450070d20 2023-11-24T08:05:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.823+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448013a80 con 0x7f5450070d20 2023-11-24T08:05:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.823+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54501563d0 con 0x7f5450070d20 2023-11-24T08:05:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.823+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450156990 con 0x7f5450070d20 2023-11-24T08:05:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.824+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545004e8b0 con 0x7f5450070d20 2023-11-24T08:05:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.825+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5448013be0 con 0x7f5450070d20 2023-11-24T08:05:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.826+0000 7f5447fff640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 0x7f543006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.826+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5448088d20 con 0x7f5450070d20 2023-11-24T08:05:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.826+0000 7f544effd640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 0x7f543006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.827+0000 7f544effd640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 0x7f543006d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f54400013d0 tx=0x7f5440001490 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:36.830+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5448053d50 con 0x7f5450070d20 2023-11-24T08:05:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.020+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f54500b4f60 con 0x7f5450070d20 2023-11-24T08:05:37.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.021+0000 7f5447fff640 1 -- 172.21.15.107:0/117769497 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5448057370 con 0x7f5450070d20 2023-11-24T08:05:37.023 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:37.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.023+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 msgr2=0x7f543006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.023+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 0x7f543006d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f54400013d0 tx=0x7f5440001490 comp rx=0 tx=0).stop 2023-11-24T08:05:37.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.023+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 msgr2=0x7f5450071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f54500c4e00 tx=0x7f544800e7f0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 shutdown_connections 2023-11-24T08:05:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f543006ae70 0x7f543006d330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5450070d20 0x7f5450071190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f545015cb60 0x7f545015cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 --2- 172.21.15.107:0/117769497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f543c0061e0 0x7f5450154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 >> 172.21.15.107:0/117769497 conn(0x7f54500b0c50 msgr2=0x7f545006c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 shutdown_connections 2023-11-24T08:05:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.024+0000 7f5455851640 1 -- 172.21.15.107:0/117769497 wait complete. 2023-11-24T08:05:37.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:37.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.258+0000 7f5fd4b1d640 1 Processor -- start 2023-11-24T08:05:37.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.258+0000 7f5fd4b1d640 1 -- start start 2023-11-24T08:05:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fd4b1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5fd006de10 0x7f5fd015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fd4b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fd4b1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd015ec90 con 0x7f5fd0072160 2023-11-24T08:05:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fd4b1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd00cd530 con 0x7f5fd006de10 2023-11-24T08:05:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fd4b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd00c4ed0 con 0x7f5fd00c49f0 2023-11-24T08:05:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.260+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:37134/0 (socket says 172.21.15.107:37134) 2023-11-24T08:05:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 -- 172.21.15.107:0/2971261609 learned_addr learned my addr 172.21.15.107:0/2971261609 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fcdd9b640 1 --2- 172.21.15.107:0/2971261609 >> v2:172.21.15.120:3300/0 conn(0x7f5fd006de10 0x7f5fd015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 -- 172.21.15.107:0/2971261609 >> v2:172.21.15.120:3300/0 conn(0x7f5fd006de10 msgr2=0x7f5fd015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 --2- 172.21.15.107:0/2971261609 >> v2:172.21.15.120:3300/0 conn(0x7f5fd006de10 0x7f5fd015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 -- 172.21.15.107:0/2971261609 >> v1:172.21.15.120:6789/0 conn(0x7f5fd0072160 legacy=0x7f5fd006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.261+0000 7f5fced9d640 1 -- 172.21.15.107:0/2971261609 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd00c51b0 con 0x7f5fd00c49f0 2023-11-24T08:05:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.262+0000 7f5fced9d640 1 --2- 172.21.15.107:0/2971261609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009e50 tx=0x7f5fc802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff891b25868a233d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.263+0000 7f5fcd59a640 1 -- 172.21.15.107:0/2971261609 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc802cb00 con 0x7f5fd00c49f0 2023-11-24T08:05:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.263+0000 7f5fcd59a640 1 -- 172.21.15.107:0/2971261609 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc802cc60 con 0x7f5fd00c49f0 2023-11-24T08:05:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.263+0000 7f5fcd59a640 1 -- 172.21.15.107:0/2971261609 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc80343f0 con 0x7f5fd00c49f0 2023-11-24T08:05:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.263+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/2971261609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 msgr2=0x7f5fd00705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.263+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/2971261609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009e50 tx=0x7f5fc802bf60 comp rx=0 tx=0).stop 2023-11-24T08:05:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/2971261609 shutdown_connections 2023-11-24T08:05:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/2971261609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00705c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/2971261609 >> v2:172.21.15.120:3300/0 conn(0x7f5fd006de10 0x7f5fd015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/2971261609 >> 172.21.15.107:0/2971261609 conn(0x7f5fd00b0c50 msgr2=0x7f5fd00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/2971261609 shutdown_connections 2023-11-24T08:05:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.264+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/2971261609 wait complete. 2023-11-24T08:05:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.265+0000 7f5fd4b1d640 1 Processor -- start 2023-11-24T08:05:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.265+0000 7f5fd4b1d640 1 -- start start 2023-11-24T08:05:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fd0072160 0x7f5fd01560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0156610 con 0x7f5fd006de10 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd00c68c0 con 0x7f5fd0072160 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.266+0000 7f5fd4b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd00c8650 con 0x7f5fd00c49f0 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53182/0 (socket says 172.21.15.107:53182) 2023-11-24T08:05:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 -- 172.21.15.107:0/225358187 learned_addr learned my addr 172.21.15.107:0/225358187 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fcdd9b640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fd0072160 0x7f5fd01560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fced9d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 -- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 msgr2=0x7f5fd00c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 -- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fd0072160 msgr2=0x7f5fd01560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fd0072160 0x7f5fd01560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 -- 172.21.15.107:0/225358187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc8009b30 con 0x7f5fd006de10 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.267+0000 7f5fce59c640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000c450 tx=0x7f5fc000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.268+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc00052d0 con 0x7f5fd006de10 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.268+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc0005430 con 0x7f5fd006de10 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.268+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0012870 con 0x7f5fd006de10 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.268+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd00c8930 con 0x7f5fd006de10 2023-11-24T08:05:37.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.268+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd00c6550 con 0x7f5fd006de10 2023-11-24T08:05:37.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.269+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd00c60b0 con 0x7f5fd006de10 2023-11-24T08:05:37.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.270+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5fc00129d0 con 0x7f5fd006de10 2023-11-24T08:05:37.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.271+0000 7f5fbeffd640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 0x7f5fb006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.271+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5fc00890e0 con 0x7f5fd006de10 2023-11-24T08:05:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.271+0000 7f5fcdd9b640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 0x7f5fb006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.272+0000 7f5fcdd9b640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 0x7f5fb006d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009ad0 tx=0x7f5fc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.275+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fc0054070 con 0x7f5fd006de10 2023-11-24T08:05:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.465+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5fd015ec90 con 0x7f5fd006de10 2023-11-24T08:05:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.466+0000 7f5fbeffd640 1 -- 172.21.15.107:0/225358187 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5fd015ec90 con 0x7f5fd006de10 2023-11-24T08:05:37.468 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.468+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 msgr2=0x7f5fb006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.468+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 0x7f5fb006d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009ad0 tx=0x7f5fc4008040 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.468+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 msgr2=0x7f5fd00c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.468+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000c450 tx=0x7f5fc000c910 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 shutdown_connections 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5fb006ae40 0x7f5fb006d300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fd00c49f0 0x7f5fd00c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fd0072160 0x7f5fd01560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 --2- 172.21.15.107:0/225358187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd006de10 0x7f5fd00c6d50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 >> 172.21.15.107:0/225358187 conn(0x7f5fd00b0c50 msgr2=0x7f5fd00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 shutdown_connections 2023-11-24T08:05:37.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.469+0000 7f5fd4b1d640 1 -- 172.21.15.107:0/225358187 wait complete. 2023-11-24T08:05:37.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:05:37.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.695+0000 7f4772d4c640 1 Processor -- start 2023-11-24T08:05:37.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.696+0000 7f4772d4c640 1 -- start start 2023-11-24T08:05:37.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.697+0000 7f4772d4c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f476c0d2000 0x7f476c0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.697+0000 7f4772d4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d0200 0x7f476c1605f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.697+0000 7f4772d4c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476c0ce410 con 0x7f476c0d1c20 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.697+0000 7f4772d4c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f476c06c270 con 0x7f476c0d2000 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.697+0000 7f4772d4c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476c0d06f0 con 0x7f476c0d0200 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f476c0d2000 0x7f476c0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f4770ae8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f476c0d1c20 0x7f476c06bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57682/0 (socket says 172.21.15.107:57682) 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f476c0d2000 0x7f476c0cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53194/0 (socket says 172.21.15.107:53194) 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f4770ae8640 1 -- 172.21.15.107:0/2852970748 learned_addr learned my addr 172.21.15.107:0/2852970748 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f47712e9640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d0200 0x7f476c1605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 -- 172.21.15.107:0/2852970748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d0200 msgr2=0x7f476c1605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d0200 0x7f476c1605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 -- 172.21.15.107:0/2852970748 >> v1:172.21.15.107:6789/0 conn(0x7f476c0d1c20 legacy=0x7f476c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476bfff640 1 -- 172.21.15.107:0/2852970748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c0d09d0 con 0x7f476c0d2000 2023-11-24T08:05:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.698+0000 7f476b7fe640 1 -- 172.21.15.107:0/2852970748 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789965231 0 0) 0x7f476c0ce410 con 0x7f476c0d1c20 2023-11-24T08:05:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.699+0000 7f476bfff640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d2000 0x7f476c0cfcc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47540097a0 tx=0x7f475402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1068ff937922ad9f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f476b7fe640 1 -- 172.21.15.107:0/2852970748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4754031830 con 0x7f476c0d2000 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f476b7fe640 1 -- 172.21.15.107:0/2852970748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4754031990 con 0x7f476c0d2000 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f476b7fe640 1 -- 172.21.15.107:0/2852970748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4754031d20 con 0x7f476c0d2000 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 -- 172.21.15.107:0/2852970748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d2000 msgr2=0x7f476c0cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d2000 0x7f476c0cfcc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47540097a0 tx=0x7f475402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 -- 172.21.15.107:0/2852970748 shutdown_connections 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d0200 0x7f476c1605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 --2- 172.21.15.107:0/2852970748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d2000 0x7f476c0cfcc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 -- 172.21.15.107:0/2852970748 >> 172.21.15.107:0/2852970748 conn(0x7f476c08cea0 msgr2=0x7f476c09c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.700+0000 7f4772d4c640 1 -- 172.21.15.107:0/2852970748 shutdown_connections 2023-11-24T08:05:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.701+0000 7f4772d4c640 1 -- 172.21.15.107:0/2852970748 wait complete. 2023-11-24T08:05:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.702+0000 7f4772d4c640 1 Processor -- start 2023-11-24T08:05:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.702+0000 7f4772d4c640 1 -- start start 2023-11-24T08:05:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 0x7f476c15a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d2000 0x7f476c15d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1f3ea0 con 0x7f476c0d1c20 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c0b8d20 con 0x7f476c0d0200 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.703+0000 7f4772d4c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476c0b6470 con 0x7f476c0d2000 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53196/0 (socket says 172.21.15.107:53196) 2023-11-24T08:05:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 -- 172.21.15.107:0/4270762977 learned_addr learned my addr 172.21.15.107:0/4270762977 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f4770ae8640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 0x7f476c15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f47712e9640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d2000 0x7f476c15d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 -- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d2000 msgr2=0x7f476c15d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d2000 0x7f476c15d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 -- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 msgr2=0x7f476c15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 0x7f476c15a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 -- 172.21.15.107:0/4270762977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754009400 con 0x7f476c0d1c20 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f4770ae8640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 0x7f476c15a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.704+0000 7f476bfff640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47540098c0 tx=0x7f47540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.705+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47540317d0 con 0x7f476c0d1c20 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.705+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475403aa60 con 0x7f476c0d1c20 2023-11-24T08:05:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.705+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475403ae10 con 0x7f476c0d1c20 2023-11-24T08:05:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.705+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476c15a9d0 con 0x7f476c0d1c20 2023-11-24T08:05:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.705+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476c0b8a40 con 0x7f476c0d1c20 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.706+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476c04e8b0 con 0x7f476c0d1c20 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.707+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4754030380 con 0x7f476c0d1c20 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.708+0000 7f47697fa640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 0x7f474c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.709+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f47540ab1a0 con 0x7f476c0d1c20 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.709+0000 7f4770ae8640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 0x7f474c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.709+0000 7f4770ae8640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 0x7f474c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f475c0012e0 tx=0x7f475c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:37.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.713+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47540761c0 con 0x7f476c0d1c20 2023-11-24T08:05:37.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.919+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f476c0b65b0 con 0x7f476c0d1c20 2023-11-24T08:05:37.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.920+0000 7f47697fa640 1 -- 172.21.15.107:0/4270762977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f47540797e0 con 0x7f476c0d1c20 2023-11-24T08:05:37.921 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:05:37.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 msgr2=0x7f474c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 0x7f474c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f475c0012e0 tx=0x7f475c009040 comp rx=0 tx=0).stop 2023-11-24T08:05:37.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 msgr2=0x7f476c15be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47540098c0 tx=0x7f47540046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 shutdown_connections 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f474c06ae40 0x7f474c06d300 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f476c0d2000 0x7f476c15d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f476c0d1c20 0x7f476c15be10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 --2- 172.21.15.107:0/4270762977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f476c0d0200 0x7f476c15a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.922+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 >> 172.21.15.107:0/4270762977 conn(0x7f476c08cea0 msgr2=0x7f476c0d3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.923+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 shutdown_connections 2023-11-24T08:05:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:37.923+0000 7f4772d4c640 1 -- 172.21.15.107:0/4270762977 wait complete. 2023-11-24T08:05:37.942 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:05:37.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:37.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-11-24T08:05:38.044 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:05:38.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:38.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:05:38.103 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:38.103 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:38.165 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:38.165 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:38.219 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:38.219 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:38.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.501+0000 7f838e2d4640 1 Processor -- start 2023-11-24T08:05:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.501+0000 7f838e2d4640 1 -- start start 2023-11-24T08:05:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838e2d4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 0x7f83880cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838e2d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83880cdfb0 0x7f83880ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838e2d4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880cece0 con 0x7f838815c7f0 2023-11-24T08:05:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838e2d4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83880cfe40 con 0x7f83880cdfb0 2023-11-24T08:05:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838e2d4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8388156010 con 0x7f83880cf450 2023-11-24T08:05:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838c871640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83880cdfb0 0x7f83880ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838c871640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83880cdfb0 0x7f83880ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40930/0 (socket says 172.21.15.107:40930) 2023-11-24T08:05:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.503+0000 7f838c871640 1 -- 172.21.15.107:0/57393847 learned_addr learned my addr 172.21.15.107:0/57393847 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f83877fe640 1 --2- 172.21.15.107:0/57393847 >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 0x7f83880cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f838c871640 1 -- 172.21.15.107:0/57393847 >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 msgr2=0x7f83880cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f838c871640 1 --2- 172.21.15.107:0/57393847 >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 0x7f83880cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f838c871640 1 -- 172.21.15.107:0/57393847 >> v1:172.21.15.120:6789/0 conn(0x7f838815c7f0 legacy=0x7f838806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:38.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f838c871640 1 -- 172.21.15.107:0/57393847 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838806d040 con 0x7f83880cdfb0 2023-11-24T08:05:38.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.504+0000 7f83877fe640 1 --2- 172.21.15.107:0/57393847 >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 0x7f83880cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:38.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.505+0000 7f838c871640 1 --2- 172.21.15.107:0/57393847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f83880ce3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8388070410 tx=0x7f838002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4c7857efb28f5d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.505+0000 7f8386ffd640 1 -- 172.21.15.107:0/57393847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838002ca80 con 0x7f83880cdfb0 2023-11-24T08:05:38.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.505+0000 7f8386ffd640 1 -- 172.21.15.107:0/57393847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838002cbe0 con 0x7f83880cdfb0 2023-11-24T08:05:38.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.505+0000 7f8386ffd640 1 -- 172.21.15.107:0/57393847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800343f0 con 0x7f83880cdfb0 2023-11-24T08:05:38.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 -- 172.21.15.107:0/57393847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 msgr2=0x7f83880ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 --2- 172.21.15.107:0/57393847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f83880ce3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8388070410 tx=0x7f838002bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 -- 172.21.15.107:0/57393847 shutdown_connections 2023-11-24T08:05:38.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 --2- 172.21.15.107:0/57393847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f83880ce3e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 --2- 172.21.15.107:0/57393847 >> v2:172.21.15.120:3300/0 conn(0x7f83880cf450 0x7f83880cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 -- 172.21.15.107:0/57393847 >> 172.21.15.107:0/57393847 conn(0x7f838806a920 msgr2=0x7f838806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:38.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 -- 172.21.15.107:0/57393847 shutdown_connections 2023-11-24T08:05:38.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.506+0000 7f838e2d4640 1 -- 172.21.15.107:0/57393847 wait complete. 2023-11-24T08:05:38.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.508+0000 7f838e2d4640 1 Processor -- start 2023-11-24T08:05:38.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.508+0000 7f838e2d4640 1 -- start start 2023-11-24T08:05:38.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880cf450 0x7f838815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 0x7f838815f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b2550 con 0x7f83880cdfb0 2023-11-24T08:05:38.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b27a0 con 0x7f838815c7f0 2023-11-24T08:05:38.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f838e2d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b2a80 con 0x7f83880cf450 2023-11-24T08:05:38.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f8387fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f8387fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40932/0 (socket says 172.21.15.107:40932) 2023-11-24T08:05:38.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.509+0000 7f8387fff640 1 -- 172.21.15.107:0/3695423175 learned_addr learned my addr 172.21.15.107:0/3695423175 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:38.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f838c871640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 0x7f838815f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 -- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880cf450 msgr2=0x7f838815eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f83877fe640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880cf450 0x7f838815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880cf450 0x7f838815eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 -- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 msgr2=0x7f838815f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 0x7f838815f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 -- 172.21.15.107:0/3695423175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8380009b30 con 0x7f83880cdfb0 2023-11-24T08:05:38.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f838c871640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 0x7f838815f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:38.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8387fff640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f837800b3f0 tx=0x7f837800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378011830 con 0x7f83880cdfb0 2023-11-24T08:05:38.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378011990 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880b2d60 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.510+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378011d40 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.511+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880b3320 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.512+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388067470 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.512+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f837801a400 con 0x7f83880cdfb0 2023-11-24T08:05:38.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.513+0000 7f8384ff9640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 0x7f837406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.513+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8378089490 con 0x7f83880cdfb0 2023-11-24T08:05:38.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.514+0000 7f83877fe640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 0x7f837406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.514+0000 7f83877fe640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 0x7f837406d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f838815af20 tx=0x7f837c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.518+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83780544c0 con 0x7f83880cdfb0 2023-11-24T08:05:38.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.709+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8388073b10 con 0x7f83880cdfb0 2023-11-24T08:05:38.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.710+0000 7f8384ff9640 1 -- 172.21.15.107:0/3695423175 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f8378057ae0 con 0x7f83880cdfb0 2023-11-24T08:05:38.713 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:38.713 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.713+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 msgr2=0x7f837406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.713+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 0x7f837406d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f838815af20 tx=0x7f837c008040 comp rx=0 tx=0).stop 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.713+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 msgr2=0x7f838815b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.713+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f837800b3f0 tx=0x7f837800b8b0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 shutdown_connections 2023-11-24T08:05:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f837406ae40 0x7f837406d300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f838815c7f0 0x7f838815f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880cf450 0x7f838815eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 --2- 172.21.15.107:0/3695423175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83880cdfb0 0x7f838815b190 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 >> 172.21.15.107:0/3695423175 conn(0x7f838806a920 msgr2=0x7f83880af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 shutdown_connections 2023-11-24T08:05:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.714+0000 7f838e2d4640 1 -- 172.21.15.107:0/3695423175 wait complete. 2023-11-24T08:05:38.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:38.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:38.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:05:38.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:05:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.951+0000 7f1e2f59e640 1 Processor -- start 2023-11-24T08:05:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.952+0000 7f1e2f59e640 1 -- start start 2023-11-24T08:05:38.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.953+0000 7f1e2f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e30157c30 0x7f1e300b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.953+0000 7f1e2f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30072320 0x7f1e3006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.953+0000 7f1e2f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3015dce0 con 0x7f1e30157850 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.953+0000 7f1e2f59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3015dad0 con 0x7f1e30157c30 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.953+0000 7f1e2f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300b3b70 con 0x7f1e30072320 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e30157c30 0x7f1e300b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e2e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e30157850 0x7f1e3015e4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59526/0 (socket says 172.21.15.107:59526) 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e30157c30 0x7f1e300b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40942/0 (socket says 172.21.15.107:40942) 2023-11-24T08:05:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e2e59c640 1 -- 172.21.15.107:0/3378344339 learned_addr learned my addr 172.21.15.107:0/3378344339 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e2ed9d640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30072320 0x7f1e3006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 -- 172.21.15.107:0/3378344339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30072320 msgr2=0x7f1e3006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30072320 0x7f1e3006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 -- 172.21.15.107:0/3378344339 >> v1:172.21.15.107:6789/0 conn(0x7f1e30157850 legacy=0x7f1e3015e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e27fff640 1 -- 172.21.15.107:0/3378344339 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300b35b0 con 0x7f1e30157c30 2023-11-24T08:05:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.954+0000 7f1e2dd9b640 1 -- 172.21.15.107:0/3378344339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308202850 0 0) 0x7f1e3015dce0 con 0x7f1e30157850 2023-11-24T08:05:38.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.956+0000 7f1e27fff640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30157c30 0x7f1e300b2ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1e180097a0 tx=0x7f1e1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2223a28d1a7bca13 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.956+0000 7f1e2dd9b640 1 -- 172.21.15.107:0/3378344339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18031830 con 0x7f1e30157c30 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.956+0000 7f1e2dd9b640 1 -- 172.21.15.107:0/3378344339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18031990 con 0x7f1e30157c30 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.956+0000 7f1e2dd9b640 1 -- 172.21.15.107:0/3378344339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18031d20 con 0x7f1e30157c30 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3378344339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30157c30 msgr2=0x7f1e300b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30157c30 0x7f1e300b2ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1e180097a0 tx=0x7f1e1802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3378344339 shutdown_connections 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30072320 0x7f1e3006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3378344339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30157c30 0x7f1e300b2ed0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3378344339 >> 172.21.15.107:0/3378344339 conn(0x7f1e300b1bd0 msgr2=0x7f1e300b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3378344339 shutdown_connections 2023-11-24T08:05:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.957+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3378344339 wait complete. 2023-11-24T08:05:38.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.959+0000 7f1e2f59e640 1 Processor -- start 2023-11-24T08:05:38.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.959+0000 7f1e2f59e640 1 -- start start 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30157850 0x7f1e300adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 0x7f1e30143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30145910 con 0x7f1e30072320 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30143270 con 0x7f1e30157c30 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30144110 con 0x7f1e30157850 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40948/0 (socket says 172.21.15.107:40948) 2023-11-24T08:05:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.960+0000 7f1e2e59c640 1 -- 172.21.15.107:0/3053253017 learned_addr learned my addr 172.21.15.107:0/3053253017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2ed9d640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 0x7f1e30143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 -- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30157850 msgr2=0x7f1e300adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e27fff640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30157850 0x7f1e300adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30157850 0x7f1e300adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 -- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 msgr2=0x7f1e30143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 0x7f1e30143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 -- 172.21.15.107:0/3053253017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18009400 con 0x7f1e30072320 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2ed9d640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 0x7f1e30143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e2e59c640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c4d0 tx=0x7f1e2000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.961+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e200074e0 con 0x7f1e30072320 2023-11-24T08:05:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.962+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e20007660 con 0x7f1e30072320 2023-11-24T08:05:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.962+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e20004a20 con 0x7f1e30072320 2023-11-24T08:05:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.962+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301443f0 con 0x7f1e30072320 2023-11-24T08:05:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.962+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e301449b0 con 0x7f1e30072320 2023-11-24T08:05:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.963+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3004e840 con 0x7f1e30072320 2023-11-24T08:05:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.964+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1e20004b80 con 0x7f1e30072320 2023-11-24T08:05:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.965+0000 7f1e26ffd640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 0x7f1e1006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.965+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1e200887f0 con 0x7f1e30072320 2023-11-24T08:05:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.965+0000 7f1e27fff640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 0x7f1e1006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.966+0000 7f1e27fff640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 0x7f1e1006d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e1803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:38.971+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e20053820 con 0x7f1e30072320 2023-11-24T08:05:39.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.162+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e30071970 con 0x7f1e30072320 2023-11-24T08:05:39.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.163+0000 7f1e26ffd640 1 -- 172.21.15.107:0/3053253017 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e20018020 con 0x7f1e30072320 2023-11-24T08:05:39.165 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 msgr2=0x7f1e1006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 0x7f1e1006d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e1803c040 comp rx=0 tx=0).stop 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 msgr2=0x7f1e300ad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c4d0 tx=0x7f1e2000c990 comp rx=0 tx=0).stop 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 shutdown_connections 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e1006ae70 0x7f1e1006d330 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e30157c30 0x7f1e30143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e30157850 0x7f1e300adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 --2- 172.21.15.107:0/3053253017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e30072320 0x7f1e300ad600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 >> 172.21.15.107:0/3053253017 conn(0x7f1e300b1bd0 msgr2=0x7f1e300ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 shutdown_connections 2023-11-24T08:05:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.166+0000 7f1e2f59e640 1 -- 172.21.15.107:0/3053253017 wait complete. 2023-11-24T08:05:39.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:39.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.400+0000 7fe0dffff640 1 Processor -- start 2023-11-24T08:05:39.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.400+0000 7fe0dffff640 1 -- start start 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0dffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0e00b0fb0 0x7fe0e006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0dffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e015ed20 con 0x7fe0e00b0b40 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0dffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e015f8a0 con 0x7fe0e015b260 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0dffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e015b640 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0de7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe0e00b0b40 0x7fe0e006c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59530/0 (socket says 172.21.15.107:59530) 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0de7fc640 1 -- 172.21.15.107:0/225476976 learned_addr learned my addr 172.21.15.107:0/225476976 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.402+0000 7fe0df7fe640 1 --2- 172.21.15.107:0/225476976 >> v2:172.21.15.107:3300/0 conn(0x7fe0e00b0fb0 0x7fe0e006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.403+0000 7fe0ddffb640 1 -- 172.21.15.107:0/225476976 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827169660 0 0) 0x7fe0e015ed20 con 0x7fe0e00b0b40 2023-11-24T08:05:39.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.403+0000 7fe0ddffb640 1 -- 172.21.15.107:0/225476976 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c8003520 con 0x7fe0e00b0b40 2023-11-24T08:05:39.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.403+0000 7fe0df7fe640 1 -- 172.21.15.107:0/225476976 >> v1:172.21.15.120:6789/0 conn(0x7fe0e015b260 legacy=0x7fe0e006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:39.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.403+0000 7fe0df7fe640 1 -- 172.21.15.107:0/225476976 >> v1:172.21.15.107:6789/0 conn(0x7fe0e00b0b40 legacy=0x7fe0e006c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.403+0000 7fe0df7fe640 1 -- 172.21.15.107:0/225476976 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e006e750 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.404+0000 7fe0df7fe640 1 --2- 172.21.15.107:0/225476976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0fb0 0x7fe0e006ccc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009d90 tx=0x7fe0d802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c2a7aa5c137d64 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.404+0000 7fe0ddffb640 1 -- 172.21.15.107:0/225476976 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d802cbd0 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.404+0000 7fe0ddffb640 1 -- 172.21.15.107:0/225476976 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d802cd30 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.404+0000 7fe0ddffb640 1 -- 172.21.15.107:0/225476976 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d802cea0 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 -- 172.21.15.107:0/225476976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0fb0 msgr2=0x7fe0e006ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 --2- 172.21.15.107:0/225476976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0fb0 0x7fe0e006ccc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009d90 tx=0x7fe0d802be20 comp rx=0 tx=0).stop 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 -- 172.21.15.107:0/225476976 shutdown_connections 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 --2- 172.21.15.107:0/225476976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0fb0 0x7fe0e006ccc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 -- 172.21.15.107:0/225476976 >> 172.21.15.107:0/225476976 conn(0x7fe0e00cbb40 msgr2=0x7fe0e00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:39.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 -- 172.21.15.107:0/225476976 shutdown_connections 2023-11-24T08:05:39.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.405+0000 7fe0dffff640 1 -- 172.21.15.107:0/225476976 wait complete. 2023-11-24T08:05:39.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.407+0000 7fe0dffff640 1 Processor -- start 2023-11-24T08:05:39.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.407+0000 7fe0dffff640 1 -- start start 2023-11-24T08:05:39.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0e00b0fb0 0x7fe0e00999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0e015b260 0x7fe0e01490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b4de0 con 0x7fe0e00b0b40 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e01495f0 con 0x7fe0e015b260 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0dffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e009c980 con 0x7fe0e00b0fb0 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40958/0 (socket says 172.21.15.107:40958) 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.408+0000 7fe0deffd640 1 -- 172.21.15.107:0/687227331 learned_addr learned my addr 172.21.15.107:0/687227331 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 -- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0e00b0fb0 msgr2=0x7fe0e00999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0e00b0fb0 0x7fe0e00999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 -- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0e015b260 msgr2=0x7fe0e01490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0df7fe640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0e015b260 0x7fe0e01490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0e015b260 0x7fe0e01490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 -- 172.21.15.107:0/687227331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8009a70 con 0x7fe0e00b0b40 2023-11-24T08:05:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0deffd640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000c430 tx=0x7fe0d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00052d0 con 0x7fe0e00b0b40 2023-11-24T08:05:39.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d0005430 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0003830 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e0149a50 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.409+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e009c330 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.411+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e004e840 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.411+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe0d0003990 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.412+0000 7fe0c77fe640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 0x7fe0cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.412+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe0d0088c50 con 0x7fe0e00b0b40 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.413+0000 7fe0de7fc640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 0x7fe0cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.413+0000 7fe0de7fc640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 0x7fe0cc06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00c5ce0 tx=0x7fe0d4007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.417+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0d0053c70 con 0x7fe0e00b0b40 2023-11-24T08:05:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.607+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0e00ce640 con 0x7fe0e00b0b40 2023-11-24T08:05:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.608+0000 7fe0c77fe640 1 -- 172.21.15.107:0/687227331 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0d0017020 con 0x7fe0e00b0b40 2023-11-24T08:05:39.610 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 msgr2=0x7fe0cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 0x7fe0cc06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00c5ce0 tx=0x7fe0d4007930 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 msgr2=0x7fe0e00b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000c430 tx=0x7fe0d000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 shutdown_connections 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe0cc06ae40 0x7fe0cc06d300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0e015b260 0x7fe0e01490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.610+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0e00b0fb0 0x7fe0e00999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.611+0000 7fe0dffff640 1 --2- 172.21.15.107:0/687227331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0e00b0b40 0x7fe0e00b47e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.611+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 >> 172.21.15.107:0/687227331 conn(0x7fe0e00cbb40 msgr2=0x7fe0e01593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:39.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.611+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 shutdown_connections 2023-11-24T08:05:39.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.611+0000 7fe0dffff640 1 -- 172.21.15.107:0/687227331 wait complete. 2023-11-24T08:05:39.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:39.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.843+0000 7f8cd4a94640 1 Processor -- start 2023-11-24T08:05:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.843+0000 7f8cd4a94640 1 -- start start 2023-11-24T08:05:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.844+0000 7f8cd4a94640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd015b440 0x7f8cd00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.844+0000 7f8cd4a94640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8cd00c59b0 0x7f8cd006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.844+0000 7f8cd4a94640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00c5dc0 con 0x7f8cd00cbf20 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.844+0000 7f8cd4a94640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00cc3d0 con 0x7f8cd015b440 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.844+0000 7f8cd4a94640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd015b820 con 0x7f8cd00c59b0 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd015b440 0x7f8cd00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd015b440 0x7f8cd00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40966/0 (socket says 172.21.15.107:40966) 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8ccdd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8cd00cbf20 0x7f8cd00c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59538/0 (socket says 172.21.15.107:59538) 2023-11-24T08:05:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 -- 172.21.15.107:0/3299683872 learned_addr learned my addr 172.21.15.107:0/3299683872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 -- 172.21.15.107:0/3299683872 >> v2:172.21.15.120:3300/0 conn(0x7f8cd00c59b0 msgr2=0x7f8cd006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 --2- 172.21.15.107:0/3299683872 >> v2:172.21.15.120:3300/0 conn(0x7f8cd00c59b0 0x7f8cd006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 -- 172.21.15.107:0/3299683872 >> v1:172.21.15.107:6789/0 conn(0x7f8cd00cbf20 legacy=0x7f8cd00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.845+0000 7f8cce59c640 1 -- 172.21.15.107:0/3299683872 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd015eea0 con 0x7f8cd015b440 2023-11-24T08:05:39.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.846+0000 7f8cce59c640 1 --2- 172.21.15.107:0/3299683872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd015b440 0x7f8cd00cb9e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00cb910 tx=0x7f8cc002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780a3b7bef676938 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.846+0000 7f8ccd59a640 1 -- 172.21.15.107:0/3299683872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0031880 con 0x7f8cd015b440 2023-11-24T08:05:39.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.846+0000 7f8ccd59a640 1 -- 172.21.15.107:0/3299683872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc00319e0 con 0x7f8cd015b440 2023-11-24T08:05:39.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.846+0000 7f8ccd59a640 1 -- 172.21.15.107:0/3299683872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0031d70 con 0x7f8cd015b440 2023-11-24T08:05:39.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 -- 172.21.15.107:0/3299683872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd015b440 msgr2=0x7f8cd00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/3299683872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd015b440 0x7f8cd00cb9e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00cb910 tx=0x7f8cc002b920 comp rx=0 tx=0).stop 2023-11-24T08:05:39.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 -- 172.21.15.107:0/3299683872 shutdown_connections 2023-11-24T08:05:39.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/3299683872 >> v2:172.21.15.120:3300/0 conn(0x7f8cd00c59b0 0x7f8cd006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/3299683872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd015b440 0x7f8cd00cb9e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 -- 172.21.15.107:0/3299683872 >> 172.21.15.107:0/3299683872 conn(0x7f8cd00b0c50 msgr2=0x7f8cd00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 -- 172.21.15.107:0/3299683872 shutdown_connections 2023-11-24T08:05:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.847+0000 7f8cd4a94640 1 -- 172.21.15.107:0/3299683872 wait complete. 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.849+0000 7f8cd4a94640 1 Processor -- start 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.849+0000 7f8cd4a94640 1 -- start start 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cd00cbf20 0x7f8cd01f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cd015b440 0x7f8cd015c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01f1810 con 0x7f8cd00c59b0 2023-11-24T08:05:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0099e60 con 0x7f8cd015b440 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cd4a94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd009a140 con 0x7f8cd00cbf20 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40982/0 (socket says 172.21.15.107:40982) 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.850+0000 7f8cce59c640 1 -- 172.21.15.107:0/4198497020 learned_addr learned my addr 172.21.15.107:0/4198497020 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 -- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cd00cbf20 msgr2=0x7f8cd01f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cd00cbf20 0x7f8cd01f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 -- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cd015b440 msgr2=0x7f8cd015c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cd015b440 0x7f8cd015c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 -- 172.21.15.107:0/4198497020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00094d0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cce59c640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000c00 tx=0x7f8cc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0031850 con 0x7f8cd00c59b0 2023-11-24T08:05:39.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.851+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc00443f0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.852+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd009a420 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.852+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00447a0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.852+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd009a9e0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.853+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd009abd0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.854+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8cc0044900 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.854+0000 7f8cbeffd640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 0x7f8cb006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.855+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8cc00abee0 con 0x7f8cd00c59b0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.855+0000 7f8ccdd9b640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 0x7f8cb006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.855+0000 7f8ccdd9b640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 0x7f8cb006d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8cd015df10 tx=0x7f8cc40079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:39.859+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cc0077030 con 0x7f8cd00c59b0 2023-11-24T08:05:40.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.049+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8cd004e8b0 con 0x7f8cd00c59b0 2023-11-24T08:05:40.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.049+0000 7f8cbeffd640 1 -- 172.21.15.107:0/4198497020 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8cc007a650 con 0x7f8cd00c59b0 2023-11-24T08:05:40.051 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:40.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 msgr2=0x7f8cb006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 0x7f8cb006d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8cd015df10 tx=0x7f8cc40079c0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 msgr2=0x7f8cd009af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000c00 tx=0x7f8cc00046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 shutdown_connections 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8cb006ae40 0x7f8cb006d300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cd015b440 0x7f8cd015c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cd00cbf20 0x7f8cd01f12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 --2- 172.21.15.107:0/4198497020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd00c59b0 0x7f8cd009af00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 >> 172.21.15.107:0/4198497020 conn(0x7f8cd00b0c50 msgr2=0x7f8cd0071800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 shutdown_connections 2023-11-24T08:05:40.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.052+0000 7f8cd4a94640 1 -- 172.21.15.107:0/4198497020 wait complete. 2023-11-24T08:05:40.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:05:40.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.280+0000 7efe98cdf640 1 Processor -- start 2023-11-24T08:05:40.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.281+0000 7efe98cdf640 1 -- start start 2023-11-24T08:05:40.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe98cdf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 0x7efe940cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe98cdf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe940d3940 0x7efe940d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe98cdf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe940cea30 con 0x7efe940b63d0 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe98cdf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe940d0450 con 0x7efe940d3940 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe98cdf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe940d0bf0 con 0x7efe9415d490 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.282+0000 7efe93fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe940d3940 0x7efe940d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe940d3940 0x7efe940d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40998/0 (socket says 172.21.15.107:40998) 2023-11-24T08:05:40.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 -- 172.21.15.107:0/2801032414 learned_addr learned my addr 172.21.15.107:0/2801032414 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 -- 172.21.15.107:0/2801032414 >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 msgr2=0x7efe940cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe937fe640 1 --2- 172.21.15.107:0/2801032414 >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 0x7efe940cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 --2- 172.21.15.107:0/2801032414 >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 0x7efe940cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 -- 172.21.15.107:0/2801032414 >> v1:172.21.15.120:6789/0 conn(0x7efe940b63d0 legacy=0x7efe940d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe93fff640 1 -- 172.21.15.107:0/2801032414 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe940ce750 con 0x7efe940d3940 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.283+0000 7efe937fe640 1 --2- 172.21.15.107:0/2801032414 >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 0x7efe940cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.284+0000 7efe93fff640 1 --2- 172.21.15.107:0/2801032414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe940d3940 0x7efe940d1250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efe940b52b0 tx=0x7efe8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cef46cbee8309f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:40.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.285+0000 7efe92ffd640 1 -- 172.21.15.107:0/2801032414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c031870 con 0x7efe940d3940 2023-11-24T08:05:40.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.285+0000 7efe92ffd640 1 -- 172.21.15.107:0/2801032414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c0319d0 con 0x7efe940d3940 2023-11-24T08:05:40.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.285+0000 7efe92ffd640 1 -- 172.21.15.107:0/2801032414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c031d60 con 0x7efe940d3940 2023-11-24T08:05:40.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.285+0000 7efe98cdf640 1 -- 172.21.15.107:0/2801032414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe940d3940 msgr2=0x7efe940d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.285+0000 7efe98cdf640 1 --2- 172.21.15.107:0/2801032414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe940d3940 0x7efe940d1250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efe940b52b0 tx=0x7efe8c02b910 comp rx=0 tx=0).stop 2023-11-24T08:05:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 -- 172.21.15.107:0/2801032414 shutdown_connections 2023-11-24T08:05:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 --2- 172.21.15.107:0/2801032414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe940d3940 0x7efe940d1250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 --2- 172.21.15.107:0/2801032414 >> v2:172.21.15.120:3300/0 conn(0x7efe9415d490 0x7efe940cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 -- 172.21.15.107:0/2801032414 >> 172.21.15.107:0/2801032414 conn(0x7efe94061340 msgr2=0x7efe94061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:40.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 -- 172.21.15.107:0/2801032414 shutdown_connections 2023-11-24T08:05:40.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.286+0000 7efe98cdf640 1 -- 172.21.15.107:0/2801032414 wait complete. 2023-11-24T08:05:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.287+0000 7efe98cdf640 1 Processor -- start 2023-11-24T08:05:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.288+0000 7efe98cdf640 1 -- start start 2023-11-24T08:05:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe940d3940 0x7efe940cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9415d490 0x7efe940c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9415d8e0 con 0x7efe941f3590 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe940ca370 con 0x7efe9415d490 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe98cdf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe941f3af0 con 0x7efe940d3940 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41004/0 (socket says 172.21.15.107:41004) 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 -- 172.21.15.107:0/533176932 learned_addr learned my addr 172.21.15.107:0/533176932 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe937fe640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe940d3940 0x7efe940cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe8affd640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9415d490 0x7efe940c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 -- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe940d3940 msgr2=0x7efe940cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe940d3940 0x7efe940cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 -- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9415d490 msgr2=0x7efe940c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9415d490 0x7efe940c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.289+0000 7efe93fff640 1 -- 172.21.15.107:0/533176932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0094d0 con 0x7efe941f3590 2023-11-24T08:05:40.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe93fff640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009960 tx=0x7efe8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:40.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c03ad70 con 0x7efe941f3590 2023-11-24T08:05:40.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c0443f0 con 0x7efe941f3590 2023-11-24T08:05:40.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c044760 con 0x7efe941f3590 2023-11-24T08:05:40.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe941f3dd0 con 0x7efe941f3590 2023-11-24T08:05:40.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.290+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe941f4390 con 0x7efe941f3590 2023-11-24T08:05:40.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.291+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe940ca530 con 0x7efe941f3590 2023-11-24T08:05:40.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.292+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe8c0448c0 con 0x7efe941f3590 2023-11-24T08:05:40.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.293+0000 7efe90ff9640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 0x7efe7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:40.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.293+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efe8c02d040 con 0x7efe941f3590 2023-11-24T08:05:40.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.293+0000 7efe937fe640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 0x7efe7406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:40.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.294+0000 7efe937fe640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 0x7efe7406d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe84008ee0 tx=0x7efe84008430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:40.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.297+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe8c0b1020 con 0x7efe941f3590 2023-11-24T08:05:40.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.499+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efe9404e840 con 0x7efe941f3590 2023-11-24T08:05:40.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.500+0000 7efe90ff9640 1 -- 172.21.15.107:0/533176932 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7efe8c035020 con 0x7efe941f3590 2023-11-24T08:05:40.502 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.502+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 msgr2=0x7efe7406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.502+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 0x7efe7406d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe84008ee0 tx=0x7efe84008430 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.502+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 msgr2=0x7efe940b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.502+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009960 tx=0x7efe8c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 shutdown_connections 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406ae40 0x7efe7406d300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe941f3590 0x7efe940b7db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9415d490 0x7efe940c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 --2- 172.21.15.107:0/533176932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe940d3940 0x7efe940cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 >> 172.21.15.107:0/533176932 conn(0x7efe94061340 msgr2=0x7efe940b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 shutdown_connections 2023-11-24T08:05:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:40.503+0000 7efe98cdf640 1 -- 172.21.15.107:0/533176932 wait complete. 2023-11-24T08:05:40.523 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:05:40.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:40.524 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-11-24T08:05:40.621 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:05:40.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:40.621 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:05:40.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:40.672 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:40.733 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:40.733 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:40.786 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:05:40.786 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:40.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:41.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.094+0000 7f9da45e3640 1 Processor -- start 2023-11-24T08:05:41.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.095+0000 7f9da45e3640 1 -- start start 2023-11-24T08:05:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da45e3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da45e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c0cbf80 0x7f9d9c06c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da45e3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0b3a40 con 0x7f9d9c159ce0 2023-11-24T08:05:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da45e3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0cc410 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da45e3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0cdfa0 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.096+0000 7f9da1b7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41016/0 (socket says 172.21.15.107:41016) 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 -- 172.21.15.107:0/1240442283 learned_addr learned my addr 172.21.15.107:0/1240442283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 -- 172.21.15.107:0/1240442283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c0cbf80 msgr2=0x7f9d9c06c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 --2- 172.21.15.107:0/1240442283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c0cbf80 0x7f9d9c06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 -- 172.21.15.107:0/1240442283 >> v1:172.21.15.120:6789/0 conn(0x7f9d9c159ce0 legacy=0x7f9d9c06de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.097+0000 7f9da1b7e640 1 -- 172.21.15.107:0/1240442283 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0cd9e0 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.098+0000 7f9da1b7e640 1 --2- 172.21.15.107:0/1240442283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009870 tx=0x7f9d8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6c300ebfe05dc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.098+0000 7f9da137d640 1 -- 172.21.15.107:0/1240442283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c031640 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.098+0000 7f9da137d640 1 -- 172.21.15.107:0/1240442283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c0317a0 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.098+0000 7f9da137d640 1 -- 172.21.15.107:0/1240442283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c031b30 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 -- 172.21.15.107:0/1240442283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0b21d0 msgr2=0x7f9d9c0cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 --2- 172.21.15.107:0/1240442283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009870 tx=0x7f9d8c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 -- 172.21.15.107:0/1240442283 shutdown_connections 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 --2- 172.21.15.107:0/1240442283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c0cbf80 0x7f9d9c06c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 --2- 172.21.15.107:0/1240442283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c0cba40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 -- 172.21.15.107:0/1240442283 >> 172.21.15.107:0/1240442283 conn(0x7f9d9c06a920 msgr2=0x7f9d9c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:41.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 -- 172.21.15.107:0/1240442283 shutdown_connections 2023-11-24T08:05:41.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.099+0000 7f9da45e3640 1 -- 172.21.15.107:0/1240442283 wait complete. 2023-11-24T08:05:41.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.101+0000 7f9da45e3640 1 Processor -- start 2023-11-24T08:05:41.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.101+0000 7f9da45e3640 1 -- start start 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c159ce0 0x7f9d9c0ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c155890 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1547f0 con 0x7f9d9c0b21d0 2023-11-24T08:05:41.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da45e3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c154ad0 con 0x7f9d9c159ce0 2023-11-24T08:05:41.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da1b7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da1b7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41022/0 (socket says 172.21.15.107:41022) 2023-11-24T08:05:41.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.102+0000 7f9da1b7e640 1 -- 172.21.15.107:0/3005074684 learned_addr learned my addr 172.21.15.107:0/3005074684 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da2b80640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c159ce0 0x7f9d9c0ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 -- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c159ce0 msgr2=0x7f9d9c0ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c159ce0 0x7f9d9c0ac1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 -- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d9c0b21d0 msgr2=0x7f9d9c14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 -- 172.21.15.107:0/3005074684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0094d0 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9da1b7e640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009870 tx=0x7f9d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c03ac60 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.103+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c03ade0 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.104+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c044630 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.104+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c154db0 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.104+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c155370 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.105+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c155560 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.106+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9d8c02f050 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.107+0000 7f9d92ffd640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 0x7f9d8006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.107+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d8c0ab930 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.107+0000 7f9da237f640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 0x7f9d8006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.108+0000 7f9da237f640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 0x7f9d8006d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9d98006fd0 tx=0x7f9d98006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.111+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d8c076830 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.303+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d9c067470 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.306+0000 7f9d92ffd640 1 -- 172.21.15.107:0/3005074684 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f9d8c079e50 con 0x7f9d9c0cbf80 2023-11-24T08:05:41.308 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:41.308 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:41.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 msgr2=0x7f9d8006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 0x7f9d8006d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9d98006fd0 tx=0x7f9d98006bc0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 msgr2=0x7f9d9c153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009870 tx=0x7f9d8c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 shutdown_connections 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d8006ae70 0x7f9d8006d330 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d9c159ce0 0x7f9d9c0ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d9c0cbf80 0x7f9d9c153d50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 --2- 172.21.15.107:0/3005074684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d9c0b21d0 0x7f9d9c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 >> 172.21.15.107:0/3005074684 conn(0x7f9d9c06a920 msgr2=0x7f9d9c0b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 shutdown_connections 2023-11-24T08:05:41.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.308+0000 7f9da45e3640 1 -- 172.21.15.107:0/3005074684 wait complete. 2023-11-24T08:05:41.329 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:41.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:41.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.539+0000 7f27cb59e640 1 Processor -- start 2023-11-24T08:05:41.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.539+0000 7f27cb59e640 1 -- start start 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.540+0000 7f27cb59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f27cc0b0b40 0x7f27cc0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27cb59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27cc06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27cb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc15ed20 con 0x7f27cc15b260 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27cb59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc15b640 con 0x7f27cc0b0b40 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27cb59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0b14f0 con 0x7f27cc06c5b0 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27ca59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f27cc15b260 0x7f27cc06e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59546/0 (socket says 172.21.15.107:59546) 2023-11-24T08:05:41.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27ca59c640 1 -- 172.21.15.107:0/4224962387 learned_addr learned my addr 172.21.15.107:0/4224962387 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27c9d9b640 1 --2- 172.21.15.107:0/4224962387 >> v2:172.21.15.120:3300/0 conn(0x7f27cc0b0b40 0x7f27cc0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27cad9d640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27cc06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898478862 0 0) 0x7f27cc15ed20 con 0x7f27cc15b260 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.541+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b4003520 con 0x7f27cc15b260 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1921121062 0 0) 0x7f27b4003520 con 0x7f27cc15b260 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 msgr2=0x7f27cc06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27cc06ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 >> v2:172.21.15.120:3300/0 conn(0x7f27cc0b0b40 msgr2=0x7f27cc0b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 --2- 172.21.15.107:0/4224962387 >> v2:172.21.15.120:3300/0 conn(0x7f27cc0b0b40 0x7f27cc0b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc15f8a0 con 0x7f27cc15b260 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f27bc001670 con 0x7f27cc15b260 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 >> v1:172.21.15.107:6789/0 conn(0x7f27cc15b260 legacy=0x7f27cc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27b40047c0 0x7f27b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc15b640 con 0x7f27b40039a0 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0b14f0 con 0x7f27b40047c0 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b400ba60 con 0x7f27cc06c5b0 2023-11-24T08:05:41.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27ca59c640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.542+0000 7f27cad9d640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27c9d9b640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27b40047c0 0x7f27b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27ca59c640 1 -- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 msgr2=0x7f27b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27ca59c640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06c5b0 0x7f27b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27ca59c640 1 -- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27b40047c0 msgr2=0x7f27b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27ca59c640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27b40047c0 0x7f27b4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.543+0000 7f27ca59c640 1 -- 172.21.15.107:0/4224962387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4003520 con 0x7f27b40039a0 2023-11-24T08:05:41.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27ca59c640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27b4003dc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c210 tx=0x7f27b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538f5d2c041fcf07 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc005ad0 con 0x7f27b40039a0 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc005c30 con 0x7f27b40039a0 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27c959a640 1 -- 172.21.15.107:0/4224962387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc02c870 con 0x7f27b40039a0 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27cb59e640 1 -- 172.21.15.107:0/4224962387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 msgr2=0x7f27b4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27cb59e640 1 --2- 172.21.15.107:0/4224962387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27b4003dc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c210 tx=0x7f27b40057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.544+0000 7f27cad9d640 1 -- 172.21.15.107:0/4224962387 reap_dead start 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.545+0000 7f27cb59e640 1 -- 172.21.15.107:0/4224962387 shutdown_connections 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.545+0000 7f27cb59e640 1 -- 172.21.15.107:0/4224962387 >> 172.21.15.107:0/4224962387 conn(0x7f27cc0cbb40 msgr2=0x7f27cc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.545+0000 7f27cb59e640 1 -- 172.21.15.107:0/4224962387 shutdown_connections 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.545+0000 7f27cb59e640 1 -- 172.21.15.107:0/4224962387 wait complete. 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.546+0000 7f27cb59e640 1 Processor -- start 2023-11-24T08:05:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.547+0000 7f27cb59e640 1 -- start start 2023-11-24T08:05:41.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.547+0000 7f27cb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.547+0000 7f27cb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc09c190 0x7f27cc09b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27cb59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06cdd0 0x7f27cc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27cb59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc1f53e0 con 0x7f27b40039a0 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27cb59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc06d790 con 0x7f27cc09c190 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27cb59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc09bd40 con 0x7f27cc06cdd0 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41042/0 (socket says 172.21.15.107:41042) 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 -- 172.21.15.107:0/2962181218 learned_addr learned my addr 172.21.15.107:0/2962181218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27c9d9b640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc09c190 0x7f27cc09b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27cad9d640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06cdd0 0x7f27cc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 -- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06cdd0 msgr2=0x7f27cc06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06cdd0 0x7f27cc06d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 -- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc09c190 msgr2=0x7f27cc09b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc09c190 0x7f27cc09b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.548+0000 7f27ca59c640 1 -- 172.21.15.107:0/2962181218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc09c6c0 con 0x7f27b40039a0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27ca59c640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f27bc02c410 tx=0x7f27bc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc039070 con 0x7f27b40039a0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc02ca70 con 0x7f27b40039a0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc03c040 con 0x7f27b40039a0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc1f4a40 con 0x7f27b40039a0 2023-11-24T08:05:41.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.549+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc1f5000 con 0x7f27b40039a0 2023-11-24T08:05:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.550+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc1f5650 con 0x7f27b40039a0 2023-11-24T08:05:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.551+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f27bc032070 con 0x7f27b40039a0 2023-11-24T08:05:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.552+0000 7f27baffd640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 0x7f27ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.552+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f27bc0aa850 con 0x7f27b40039a0 2023-11-24T08:05:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.552+0000 7f27c9d9b640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 0x7f27ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.553+0000 7f27c9d9b640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 0x7f27ac06d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f27cc15ee40 tx=0x7f27c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.556+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27bc075870 con 0x7f27b40039a0 2023-11-24T08:05:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.747+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27cc04e840 con 0x7f27b40039a0 2023-11-24T08:05:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.749+0000 7f27baffd640 1 -- 172.21.15.107:0/2962181218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27bc078e90 con 0x7f27b40039a0 2023-11-24T08:05:41.751 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 msgr2=0x7f27ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 0x7f27ac06d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f27cc15ee40 tx=0x7f27c0008040 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 msgr2=0x7f27cc155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f27bc02c410 tx=0x7f27bc001a00 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 shutdown_connections 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27ac06ae40 0x7f27ac06d300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27cc06cdd0 0x7f27cc06d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc09c190 0x7f27cc09b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 --2- 172.21.15.107:0/2962181218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b40039a0 0x7f27cc155870 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 >> 172.21.15.107:0/2962181218 conn(0x7f27cc0cbb40 msgr2=0x7f27cc071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.751+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 shutdown_connections 2023-11-24T08:05:41.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.752+0000 7f27cb59e640 1 -- 172.21.15.107:0/2962181218 wait complete. 2023-11-24T08:05:41.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.984+0000 7efea88ad640 1 Processor -- start 2023-11-24T08:05:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.984+0000 7efea88ad640 1 -- start start 2023-11-24T08:05:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.985+0000 7efea88ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efea415bee0 0x7efea40b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.985+0000 7efea88ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea406d7b0 con 0x7efea40709a0 2023-11-24T08:05:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.985+0000 7efea88ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea4162e30 con 0x7efea4070530 2023-11-24T08:05:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.985+0000 7efea88ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea41662e0 con 0x7efea415bee0 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efea40709a0 0x7efea4165bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59548/0 (socket says 172.21.15.107:59548) 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efea415bee0 0x7efea40b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea2d9d640 1 -- 172.21.15.107:0/2685182127 learned_addr learned my addr 172.21.15.107:0/2685182127 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea259c640 1 -- 172.21.15.107:0/2685182127 >> v1:172.21.15.120:6789/0 conn(0x7efea4070530 legacy=0x7efea41654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea259c640 1 -- 172.21.15.107:0/2685182127 >> v1:172.21.15.107:6789/0 conn(0x7efea40709a0 legacy=0x7efea4165bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.986+0000 7efea259c640 1 -- 172.21.15.107:0/2685182127 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea415c2c0 con 0x7efea415bee0 2023-11-24T08:05:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.987+0000 7efea259c640 1 --2- 172.21.15.107:0/2685182127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea415bee0 0x7efea40b1e70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efe940097a0 tx=0x7efe9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3839aeb945162b18 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.988+0000 7efea1d9b640 1 -- 172.21.15.107:0/2685182127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94031830 con 0x7efea415bee0 2023-11-24T08:05:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.988+0000 7efea1d9b640 1 -- 172.21.15.107:0/2685182127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe94031990 con 0x7efea415bee0 2023-11-24T08:05:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.988+0000 7efea1d9b640 1 -- 172.21.15.107:0/2685182127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94031d20 con 0x7efea415bee0 2023-11-24T08:05:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.988+0000 7efea88ad640 1 -- 172.21.15.107:0/2685182127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea415bee0 msgr2=0x7efea40b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.988+0000 7efea88ad640 1 --2- 172.21.15.107:0/2685182127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea415bee0 0x7efea40b1e70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efe940097a0 tx=0x7efe9402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.989+0000 7efea88ad640 1 -- 172.21.15.107:0/2685182127 shutdown_connections 2023-11-24T08:05:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.989+0000 7efea88ad640 1 --2- 172.21.15.107:0/2685182127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea415bee0 0x7efea40b1e70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.989+0000 7efea88ad640 1 -- 172.21.15.107:0/2685182127 >> 172.21.15.107:0/2685182127 conn(0x7efea40b0b70 msgr2=0x7efea40b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.989+0000 7efea88ad640 1 -- 172.21.15.107:0/2685182127 shutdown_connections 2023-11-24T08:05:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.989+0000 7efea88ad640 1 -- 172.21.15.107:0/2685182127 wait complete. 2023-11-24T08:05:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.990+0000 7efea88ad640 1 Processor -- start 2023-11-24T08:05:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.991+0000 7efea88ad640 1 -- start start 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efea40709a0 0x7efea41eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efea415bee0 0x7efea41eb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea4147880 con 0x7efea4070530 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea4165000 con 0x7efea415bee0 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea88ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea41652e0 con 0x7efea40709a0 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41062/0 (socket says 172.21.15.107:41062) 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 -- 172.21.15.107:0/326462216 learned_addr learned my addr 172.21.15.107:0/326462216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 -- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efea40709a0 msgr2=0x7efea41eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efea40709a0 0x7efea41eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 -- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efea415bee0 msgr2=0x7efea41eb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efea415bee0 0x7efea41eb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.992+0000 7efea259c640 1 -- 172.21.15.107:0/326462216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94009400 con 0x7efea4070530 2023-11-24T08:05:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efea259c640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efe94000c00 tx=0x7efe940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9403aa60 con 0x7efea4070530 2023-11-24T08:05:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9403abc0 con 0x7efea4070530 2023-11-24T08:05:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe940443f0 con 0x7efea4070530 2023-11-24T08:05:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea41655c0 con 0x7efea4070530 2023-11-24T08:05:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.993+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4144840 con 0x7efea4070530 2023-11-24T08:05:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.994+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea404e840 con 0x7efea4070530 2023-11-24T08:05:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.995+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe9404d450 con 0x7efea4070530 2023-11-24T08:05:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.996+0000 7efe9affd640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 0x7efe8406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.996+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efe940ab8e0 con 0x7efea4070530 2023-11-24T08:05:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.996+0000 7efe9bfff640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 0x7efe8406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:41.997+0000 7efe9bfff640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 0x7efe8406d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efea415aab0 tx=0x7efe8c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.000+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe94076910 con 0x7efea4070530 2023-11-24T08:05:42.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.192+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efea40ad0a0 con 0x7efea4070530 2023-11-24T08:05:42.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.193+0000 7efe9affd640 1 -- 172.21.15.107:0/326462216 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe94035020 con 0x7efea4070530 2023-11-24T08:05:42.195 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:42.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.195+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 msgr2=0x7efe8406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.195+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 0x7efe8406d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efea415aab0 tx=0x7efe8c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.195+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 msgr2=0x7efea41f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.195+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efe94000c00 tx=0x7efe940046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.195+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 shutdown_connections 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe8406ae40 0x7efe8406d300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efea415bee0 0x7efea41eb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efea40709a0 0x7efea41eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 --2- 172.21.15.107:0/326462216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea4070530 0x7efea41f7940 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 >> 172.21.15.107:0/326462216 conn(0x7efea40b0b70 msgr2=0x7efea415b000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 shutdown_connections 2023-11-24T08:05:42.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.196+0000 7efea88ad640 1 -- 172.21.15.107:0/326462216 wait complete. 2023-11-24T08:05:42.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.432+0000 7fd8697d5640 1 Processor -- start 2023-11-24T08:05:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.432+0000 7fd8697d5640 1 -- start start 2023-11-24T08:05:42.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.434+0000 7fd8697d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd86406de10 0x7fd86415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.434+0000 7fd8697d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8640c49f0 0x7fd8640705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.434+0000 7fd8697d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86415ec90 con 0x7fd864072160 2023-11-24T08:05:42.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.434+0000 7fd8697d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8640cd530 con 0x7fd8640c49f0 2023-11-24T08:05:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.434+0000 7fd8697d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8640c4ed0 con 0x7fd86406de10 2023-11-24T08:05:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8640c49f0 0x7fd8640705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8640c49f0 0x7fd8640705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41070/0 (socket says 172.21.15.107:41070) 2023-11-24T08:05:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 -- 172.21.15.107:0/3714997583 learned_addr learned my addr 172.21.15.107:0/3714997583 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 -- 172.21.15.107:0/3714997583 >> v2:172.21.15.120:3300/0 conn(0x7fd86406de10 msgr2=0x7fd86415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 --2- 172.21.15.107:0/3714997583 >> v2:172.21.15.120:3300/0 conn(0x7fd86406de10 0x7fd86415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 -- 172.21.15.107:0/3714997583 >> v1:172.21.15.120:6789/0 conn(0x7fd864072160 legacy=0x7fd86406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.435+0000 7fd8637fe640 1 -- 172.21.15.107:0/3714997583 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640c51b0 con 0x7fd8640c49f0 2023-11-24T08:05:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.436+0000 7fd8637fe640 1 --2- 172.21.15.107:0/3714997583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640c49f0 0x7fd8640705c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8640cef60 tx=0x7fd85c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f2480608e54177 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.437+0000 7fd8627fc640 1 -- 172.21.15.107:0/3714997583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c02ca80 con 0x7fd8640c49f0 2023-11-24T08:05:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.437+0000 7fd8627fc640 1 -- 172.21.15.107:0/3714997583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c02cbe0 con 0x7fd8640c49f0 2023-11-24T08:05:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.437+0000 7fd8627fc640 1 -- 172.21.15.107:0/3714997583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0343f0 con 0x7fd8640c49f0 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.437+0000 7fd8697d5640 1 -- 172.21.15.107:0/3714997583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640c49f0 msgr2=0x7fd8640705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.437+0000 7fd8697d5640 1 --2- 172.21.15.107:0/3714997583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640c49f0 0x7fd8640705c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8640cef60 tx=0x7fd85c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 -- 172.21.15.107:0/3714997583 shutdown_connections 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 --2- 172.21.15.107:0/3714997583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640c49f0 0x7fd8640705c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 --2- 172.21.15.107:0/3714997583 >> v2:172.21.15.120:3300/0 conn(0x7fd86406de10 0x7fd86415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 -- 172.21.15.107:0/3714997583 >> 172.21.15.107:0/3714997583 conn(0x7fd8640b0c50 msgr2=0x7fd8640b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 -- 172.21.15.107:0/3714997583 shutdown_connections 2023-11-24T08:05:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.438+0000 7fd8697d5640 1 -- 172.21.15.107:0/3714997583 wait complete. 2023-11-24T08:05:42.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.440+0000 7fd8697d5640 1 Processor -- start 2023-11-24T08:05:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.440+0000 7fd8697d5640 1 -- start start 2023-11-24T08:05:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 0x7fd8640b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 0x7fd86406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640b3980 con 0x7fd864072160 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641568a0 con 0x7fd8640c49f0 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.441+0000 7fd8697d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640b2c50 con 0x7fd86406de10 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41072/0 (socket says 172.21.15.107:41072) 2023-11-24T08:05:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 -- 172.21.15.107:0/692359180 learned_addr learned my addr 172.21.15.107:0/692359180 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd8637fe640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 0x7fd86406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 -- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 msgr2=0x7fd8640b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd862ffd640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 0x7fd8640b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 0x7fd8640b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 -- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 msgr2=0x7fd86406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 0x7fd86406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd85a7fc640 1 -- 172.21.15.107:0/692359180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c009b30 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd862ffd640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 0x7fd8640b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.442+0000 7fd8637fe640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 0x7fd86406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd85a7fc640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00b4d0 tx=0x7fd84c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c010870 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c0109d0 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c010d80 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8640b2f30 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.443+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86415bbe0 con 0x7fd864072160 2023-11-24T08:05:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.444+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86415be00 con 0x7fd864072160 2023-11-24T08:05:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.445+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd84c00f3c0 con 0x7fd864072160 2023-11-24T08:05:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.446+0000 7fd85bfff640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 0x7fd84406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.446+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd84c089650 con 0x7fd864072160 2023-11-24T08:05:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.446+0000 7fd862ffd640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 0x7fd84406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.447+0000 7fd862ffd640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 0x7fd84406d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd854003eb0 tx=0x7fd854009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.451+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd84c054680 con 0x7fd864072160 2023-11-24T08:05:42.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.648+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd864070ad0 con 0x7fd864072160 2023-11-24T08:05:42.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.648+0000 7fd85bfff640 1 -- 172.21.15.107:0/692359180 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd84c057ca0 con 0x7fd864072160 2023-11-24T08:05:42.650 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 msgr2=0x7fd84406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 0x7fd84406d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd854003eb0 tx=0x7fd854009040 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 msgr2=0x7fd864156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00b4d0 tx=0x7fd84c00b990 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 shutdown_connections 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd84406ae40 0x7fd84406d300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8640c49f0 0x7fd86406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd864072160 0x7fd864156360 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 --2- 172.21.15.107:0/692359180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd86406de10 0x7fd8640b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.651+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 >> 172.21.15.107:0/692359180 conn(0x7fd8640b0c50 msgr2=0x7fd8640cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.652+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 shutdown_connections 2023-11-24T08:05:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.652+0000 7fd8697d5640 1 -- 172.21.15.107:0/692359180 wait complete. 2023-11-24T08:05:42.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:05:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.881+0000 7fb863075640 1 Processor -- start 2023-11-24T08:05:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.881+0000 7fb863075640 1 -- start start 2023-11-24T08:05:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb863075640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb85c0c59b0 0x7fb85c166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb863075640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0c5dc0 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb863075640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c15eea0 con 0x7fb85c15b440 2023-11-24T08:05:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb863075640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c15b820 con 0x7fb85c0c59b0 2023-11-24T08:05:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb85bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb85c0cc0f0 0x7fb85c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59556/0 (socket says 172.21.15.107:59556) 2023-11-24T08:05:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb85bfff640 1 -- 172.21.15.107:0/3300570827 learned_addr learned my addr 172.21.15.107:0/3300570827 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.883+0000 7fb861612640 1 --2- 172.21.15.107:0/3300570827 >> v2:172.21.15.107:3300/0 conn(0x7fb85c0c59b0 0x7fb85c166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.884+0000 7fb85b7fe640 1 -- 172.21.15.107:0/3300570827 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008807975 0 0) 0x7fb85c0c5dc0 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.884+0000 7fb85b7fe640 1 -- 172.21.15.107:0/3300570827 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003520 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.884+0000 7fb861612640 1 -- 172.21.15.107:0/3300570827 >> v1:172.21.15.120:6789/0 conn(0x7fb85c15b440 legacy=0x7fb85c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.884+0000 7fb861612640 1 -- 172.21.15.107:0/3300570827 >> v1:172.21.15.107:6789/0 conn(0x7fb85c0cc0f0 legacy=0x7fb85c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.884+0000 7fb861612640 1 -- 172.21.15.107:0/3300570827 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c06d600 con 0x7fb85c0c59b0 2023-11-24T08:05:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.886+0000 7fb861612640 1 --2- 172.21.15.107:0/3300570827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0c59b0 0x7fb85c166230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb854009d90 tx=0x7fb85402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbe0e352e39546e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.886+0000 7fb85b7fe640 1 -- 172.21.15.107:0/3300570827 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85402cc60 con 0x7fb85c0c59b0 2023-11-24T08:05:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.886+0000 7fb85b7fe640 1 -- 172.21.15.107:0/3300570827 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85402cdc0 con 0x7fb85c0c59b0 2023-11-24T08:05:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.886+0000 7fb85b7fe640 1 -- 172.21.15.107:0/3300570827 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854034660 con 0x7fb85c0c59b0 2023-11-24T08:05:42.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 -- 172.21.15.107:0/3300570827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0c59b0 msgr2=0x7fb85c166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 --2- 172.21.15.107:0/3300570827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0c59b0 0x7fb85c166230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb854009d90 tx=0x7fb85402beb0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 -- 172.21.15.107:0/3300570827 shutdown_connections 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 --2- 172.21.15.107:0/3300570827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0c59b0 0x7fb85c166230 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 -- 172.21.15.107:0/3300570827 >> 172.21.15.107:0/3300570827 conn(0x7fb85c0b0c50 msgr2=0x7fb85c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 -- 172.21.15.107:0/3300570827 shutdown_connections 2023-11-24T08:05:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.887+0000 7fb863075640 1 -- 172.21.15.107:0/3300570827 wait complete. 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.889+0000 7fb863075640 1 Processor -- start 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.889+0000 7fb863075640 1 -- start start 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb85c0c59b0 0x7fb85c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb85c15b440 0x7fb85c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0cec60 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0ce290 con 0x7fb85c0c59b0 2023-11-24T08:05:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb863075640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c154cf0 con 0x7fb85c15b440 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb85bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb85bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41088/0 (socket says 172.21.15.107:41088) 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.890+0000 7fb85bfff640 1 -- 172.21.15.107:0/4023579104 learned_addr learned my addr 172.21.15.107:0/4023579104 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb860e11640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb85c0c59b0 0x7fb85c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb861612640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb85c15b440 0x7fb85c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 -- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb85c15b440 msgr2=0x7fb85c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb85c15b440 0x7fb85c1547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 -- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb85c0c59b0 msgr2=0x7fb85c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb85c0c59b0 0x7fb85c155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 -- 172.21.15.107:0/4023579104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854009a70 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb85bfff640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb84800c500 tx=0x7fb84800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848005ce0 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb848005e40 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848007330 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c0cd680 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.891+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c0cdc40 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.893+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c04e8b0 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.894+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb848007490 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.895+0000 7fb8597fa640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 0x7fb84006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.895+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb848088c40 con 0x7fb85c0cc0f0 2023-11-24T08:05:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.895+0000 7fb860e11640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 0x7fb84006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.895+0000 7fb860e11640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 0x7fb84006d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb850000f60 tx=0x7fb850009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:42.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:42.899+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb848053dc0 con 0x7fb85c0cc0f0 2023-11-24T08:05:43.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.104+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb85c06dc80 con 0x7fb85c0cc0f0 2023-11-24T08:05:43.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.105+0000 7fb8597fa640 1 -- 172.21.15.107:0/4023579104 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb848017020 con 0x7fb85c0cc0f0 2023-11-24T08:05:43.107 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:05:43.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 msgr2=0x7fb84006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 0x7fb84006d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb850000f60 tx=0x7fb850009040 comp rx=0 tx=0).stop 2023-11-24T08:05:43.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 msgr2=0x7fb85c0ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb84800c500 tx=0x7fb84800c9c0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 shutdown_connections 2023-11-24T08:05:43.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84006ae40 0x7fb84006d300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb85c15b440 0x7fb85c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb85c0cc0f0 0x7fb85c0ce720 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 --2- 172.21.15.107:0/4023579104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb85c0c59b0 0x7fb85c155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 >> 172.21.15.107:0/4023579104 conn(0x7fb85c0b0c50 msgr2=0x7fb85c0cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:43.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 shutdown_connections 2023-11-24T08:05:43.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.108+0000 7fb863075640 1 -- 172.21.15.107:0/4023579104 wait complete. 2023-11-24T08:05:43.128 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-11-24T08:05:43.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:43.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-11-24T08:05:43.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:05:43.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:43.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-24T08:05:43.282 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:43.282 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:43.343 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:43.344 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:43.397 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:05:43.397 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:05:43.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:43.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.670+0000 7f41a2d8a640 1 Processor -- start 2023-11-24T08:05:43.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.670+0000 7f41a2d8a640 1 -- start start 2023-11-24T08:05:43.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c15c7f0 0x7f419c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f419c0cf450 0x7f419c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f419c0cdfb0 0x7f419c0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c156010 con 0x7f419c0cf450 2023-11-24T08:05:43.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c06ce30 con 0x7f419c0cdfb0 2023-11-24T08:05:43.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a2d8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0cfe10 con 0x7f419c15c7f0 2023-11-24T08:05:43.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f4193fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f419c0cf450 0x7f419c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:43.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f4193fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f419c0cf450 0x7f419c0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41092/0 (socket says 172.21.15.107:41092) 2023-11-24T08:05:43.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f4193fff640 1 -- 172.21.15.107:0/1790082394 learned_addr learned my addr 172.21.15.107:0/1790082394 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a0b26640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c15c7f0 0x7f419c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.671+0000 7f41a1327640 1 --2- 172.21.15.107:0/1790082394 >> v2:172.21.15.120:3300/0 conn(0x7f419c0cdfb0 0x7f419c0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.672+0000 7f4193fff640 1 -- 172.21.15.107:0/1790082394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c15c7f0 msgr2=0x7f419c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.672+0000 7f4193fff640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c15c7f0 0x7f419c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.672+0000 7f4193fff640 1 -- 172.21.15.107:0/1790082394 >> v2:172.21.15.120:3300/0 conn(0x7f419c0cdfb0 msgr2=0x7f419c0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:43.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.672+0000 7f4193fff640 1 --2- 172.21.15.107:0/1790082394 >> v2:172.21.15.120:3300/0 conn(0x7f419c0cdfb0 0x7f419c0ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.672+0000 7f4193fff640 1 -- 172.21.15.107:0/1790082394 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0cece0 con 0x7f419c0cf450 2023-11-24T08:05:43.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.673+0000 7f4193fff640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cf450 0x7f419c0cf8d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f419c0aebc0 tx=0x7f418c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b5dfd738211470 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:43.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.673+0000 7f41937fe640 1 -- 172.21.15.107:0/1790082394 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c031870 con 0x7f419c0cf450 2023-11-24T08:05:43.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.673+0000 7f41937fe640 1 -- 172.21.15.107:0/1790082394 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c0319d0 con 0x7f419c0cf450 2023-11-24T08:05:43.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.673+0000 7f41937fe640 1 -- 172.21.15.107:0/1790082394 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c031d60 con 0x7f419c0cf450 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 -- 172.21.15.107:0/1790082394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cf450 msgr2=0x7f419c0cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cf450 0x7f419c0cf8d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f419c0aebc0 tx=0x7f418c02b910 comp rx=0 tx=0).stop 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 -- 172.21.15.107:0/1790082394 shutdown_connections 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/1790082394 >> v2:172.21.15.120:3300/0 conn(0x7f419c0cdfb0 0x7f419c0ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cf450 0x7f419c0cf8d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/1790082394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c15c7f0 0x7f419c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 -- 172.21.15.107:0/1790082394 >> 172.21.15.107:0/1790082394 conn(0x7f419c06a920 msgr2=0x7f419c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:43.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 -- 172.21.15.107:0/1790082394 shutdown_connections 2023-11-24T08:05:43.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.674+0000 7f41a2d8a640 1 -- 172.21.15.107:0/1790082394 wait complete. 2023-11-24T08:05:43.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.676+0000 7f41a2d8a640 1 Processor -- start 2023-11-24T08:05:43.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.676+0000 7f41a2d8a640 1 -- start start 2023-11-24T08:05:43.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.677+0000 7f41a2d8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a2d8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c0cf450 0x7f419c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a2d8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f419c15c7f0 0x7f419c158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a2d8a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c09bad0 con 0x7f419c0cdfb0 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a2d8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c09aec0 con 0x7f419c15c7f0 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a2d8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c09b1a0 con 0x7f419c0cf450 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41108/0 (socket says 172.21.15.107:41108) 2023-11-24T08:05:43.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 -- 172.21.15.107:0/3784656192 learned_addr learned my addr 172.21.15.107:0/3784656192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:43.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 -- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c0cf450 msgr2=0x7f419c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c0cf450 0x7f419c1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 -- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f419c15c7f0 msgr2=0x7f419c158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f419c15c7f0 0x7f419c158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.678+0000 7f41a0b26640 1 -- 172.21.15.107:0/3784656192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0094d0 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41a0b26640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4198007e40 tx=0x7f419800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198014620 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419800ee90 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c09b480 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41980149c0 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.679+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c154bf0 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.680+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c067470 con 0x7f419c0cdfb0 2023-11-24T08:05:43.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.681+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4198014b20 con 0x7f419c0cdfb0 2023-11-24T08:05:43.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.682+0000 7f41917fa640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:43.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.682+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f419808a090 con 0x7f419c0cdfb0 2023-11-24T08:05:43.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.682+0000 7f4193fff640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:43.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.683+0000 7f4193fff640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 0x7f418806d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f418c036000 tx=0x7f418c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:43.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.686+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4198055020 con 0x7f419c0cdfb0 2023-11-24T08:05:43.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.882+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f419c0cd730 con 0x7f419c0cdfb0 2023-11-24T08:05:43.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.885+0000 7f41917fa640 1 -- 172.21.15.107:0/3784656192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f4198058640 con 0x7f419c0cdfb0 2023-11-24T08:05:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:43.887 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:43.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 msgr2=0x7f418806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 0x7f418806d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f418c036000 tx=0x7f418c030000 comp rx=0 tx=0).stop 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 msgr2=0x7f419c1546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4198007e40 tx=0x7f419800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 shutdown_connections 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f419c15c7f0 0x7f419c158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419c0cf450 0x7f419c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 --2- 172.21.15.107:0/3784656192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f419c0cdfb0 0x7f419c1546b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:43.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 >> 172.21.15.107:0/3784656192 conn(0x7f419c06a920 msgr2=0x7f419c0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:43.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 shutdown_connections 2023-11-24T08:05:43.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:43.888+0000 7f41a2d8a640 1 -- 172.21.15.107:0/3784656192 wait complete. 2023-11-24T08:05:43.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:43.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.131+0000 7f34ee173640 1 Processor -- start 2023-11-24T08:05:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.131+0000 7f34ee173640 1 -- start start 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ee173640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34e815e5e0 0x7f34e80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ee173640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e806d430 con 0x7f34e815a1c0 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ee173640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80b14e0 con 0x7f34e80b0b40 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ee173640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e815ea70 con 0x7f34e815e5e0 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ed171640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f34e815a1c0 0x7f34e806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59564/0 (socket says 172.21.15.107:59564) 2023-11-24T08:05:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ed171640 1 -- 172.21.15.107:0/1504985132 learned_addr learned my addr 172.21.15.107:0/1504985132 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34ed972640 1 --2- 172.21.15.107:0/1504985132 >> v2:172.21.15.120:3300/0 conn(0x7f34e815e5e0 0x7f34e80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.133+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285437476 0 0) 0x7f34e806d430 con 0x7f34e815a1c0 2023-11-24T08:05:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d4003520 con 0x7f34e815a1c0 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619828399 0 0) 0x7f34d4003520 con 0x7f34e815a1c0 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 >> v2:172.21.15.120:3300/0 conn(0x7f34e815e5e0 msgr2=0x7f34e80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 --2- 172.21.15.107:0/1504985132 >> v2:172.21.15.120:3300/0 conn(0x7f34e815e5e0 0x7f34e80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 >> v1:172.21.15.120:6789/0 conn(0x7f34e80b0b40 legacy=0x7f34e815ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e815ec50 con 0x7f34e815a1c0 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f34e4001670 con 0x7f34e815a1c0 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 >> v1:172.21.15.107:6789/0 conn(0x7f34e815a1c0 legacy=0x7f34e806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 0x7f34d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d40047c0 0x7f34d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d4005180 0x7f34d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e815ea70 con 0x7f34d4003970 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d400c0d0 con 0x7f34d40047c0 2023-11-24T08:05:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d400c290 con 0x7f34d4005180 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.134+0000 7f34ed171640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 0x7f34d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ec970640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d40047c0 0x7f34d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed171640 1 -- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d4005180 msgr2=0x7f34d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed171640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d4005180 0x7f34d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed171640 1 -- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d40047c0 msgr2=0x7f34d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed171640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d40047c0 0x7f34d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed972640 1 -- 172.21.15.107:0/1504985132 reap_dead start 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ed171640 1 -- 172.21.15.107:0/1504985132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d4003520 con 0x7f34d4003970 2023-11-24T08:05:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.135+0000 7f34ec970640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d40047c0 0x7f34d4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.136+0000 7f34ed171640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 0x7f34d4003d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c240 tx=0x7f34d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5924a9358730ac00 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.136+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4005ad0 con 0x7f34d4003970 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.136+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4005c30 con 0x7f34d4003970 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.136+0000 7f34dffff640 1 -- 172.21.15.107:0/1504985132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e402caf0 con 0x7f34d4003970 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 -- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 msgr2=0x7f34d4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 0x7f34d4003d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c240 tx=0x7f34d40057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 -- 172.21.15.107:0/1504985132 shutdown_connections 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 --2- 172.21.15.107:0/1504985132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4003970 0x7f34d4003d90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 -- 172.21.15.107:0/1504985132 >> 172.21.15.107:0/1504985132 conn(0x7f34e80cbb40 msgr2=0x7f34e80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 -- 172.21.15.107:0/1504985132 shutdown_connections 2023-11-24T08:05:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.137+0000 7f34ee173640 1 -- 172.21.15.107:0/1504985132 wait complete. 2023-11-24T08:05:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.139+0000 7f34ee173640 1 Processor -- start 2023-11-24T08:05:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.139+0000 7f34ee173640 1 -- start start 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 0x7f34e8099e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34e81575f0 0x7f34e815f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8157aa0 con 0x7f34e80c6610 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e809c8b0 con 0x7f34d4003970 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.140+0000 7f34ee173640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8156950 con 0x7f34e81575f0 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41132/0 (socket says 172.21.15.107:41132) 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 -- 172.21.15.107:0/2163678672 learned_addr learned my addr 172.21.15.107:0/2163678672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ed972640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34e81575f0 0x7f34e815f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ed171640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 0x7f34e8099e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 -- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34e81575f0 msgr2=0x7f34e815f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34e81575f0 0x7f34e815f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 -- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 msgr2=0x7f34e8099e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 0x7f34e8099e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 -- 172.21.15.107:0/2163678672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e809bd30 con 0x7f34e80c6610 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ed171640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 0x7f34e8099e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.141+0000 7f34ec970640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f34d800d4a0 tx=0x7f34d800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.142+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005480 con 0x7f34e80c6610 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.142+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8007030 con 0x7f34e80c6610 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.142+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80073c0 con 0x7f34e80c6610 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.142+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e809c010 con 0x7f34e80c6610 2023-11-24T08:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.142+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e809c5d0 con 0x7f34e80c6610 2023-11-24T08:05:44.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.143+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e80cc150 con 0x7f34e80c6610 2023-11-24T08:05:44.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.144+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f34d8007520 con 0x7f34e80c6610 2023-11-24T08:05:44.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.145+0000 7f34ddffb640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 0x7f34cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.145+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34d8088a00 con 0x7f34e80c6610 2023-11-24T08:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.145+0000 7f34ed171640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 0x7f34cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.146+0000 7f34ed171640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 0x7f34cc06d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34e4034000 tx=0x7f34e4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.149+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34d8053ad0 con 0x7f34e80c6610 2023-11-24T08:05:44.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.338+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34e80b1100 con 0x7f34e80c6610 2023-11-24T08:05:44.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.339+0000 7f34ddffb640 1 -- 172.21.15.107:0/2163678672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34d8019020 con 0x7f34e80c6610 2023-11-24T08:05:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 msgr2=0x7f34cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 0x7f34cc06d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34e4034000 tx=0x7f34e4009030 comp rx=0 tx=0).stop 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 msgr2=0x7f34e81570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f34d800d4a0 tx=0x7f34d800d960 comp rx=0 tx=0).stop 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 shutdown_connections 2023-11-24T08:05:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34cc06ae40 0x7f34cc06d300 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34e81575f0 0x7f34e815f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34e80c6610 0x7f34e81570b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 --2- 172.21.15.107:0/2163678672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d4003970 0x7f34e8099e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 >> 172.21.15.107:0/2163678672 conn(0x7f34e80cbb40 msgr2=0x7f34e80ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 shutdown_connections 2023-11-24T08:05:44.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.342+0000 7f34ee173640 1 -- 172.21.15.107:0/2163678672 wait complete. 2023-11-24T08:05:44.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.575+0000 7fe22ffff640 1 Processor -- start 2023-11-24T08:05:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.575+0000 7fe22ffff640 1 -- start start 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe23015b260 0x7fe23006e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe2300b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2300b1080 0x7fe2300ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2300b1530 con 0x7fe2300b1080 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23015b640 con 0x7fe23015b260 2023-11-24T08:05:44.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.576+0000 7fe22ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23015ed20 con 0x7fe23006e580 2023-11-24T08:05:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2300b1080 0x7fe2300ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2300b1080 0x7fe2300ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41134/0 (socket says 172.21.15.107:41134) 2023-11-24T08:05:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 -- 172.21.15.107:0/37056176 learned_addr learned my addr 172.21.15.107:0/37056176 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe2267fc640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe2300b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 -- 172.21.15.107:0/37056176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 msgr2=0x7fe2300b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22effd640 1 --2- 172.21.15.107:0/37056176 >> v2:172.21.15.120:3300/0 conn(0x7fe23015b260 0x7fe23006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe2300b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 -- 172.21.15.107:0/37056176 >> v2:172.21.15.120:3300/0 conn(0x7fe23015b260 msgr2=0x7fe23006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 --2- 172.21.15.107:0/37056176 >> v2:172.21.15.120:3300/0 conn(0x7fe23015b260 0x7fe23006e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.577+0000 7fe22f7fe640 1 -- 172.21.15.107:0/37056176 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23015f8a0 con 0x7fe2300b1080 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.578+0000 7fe22f7fe640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2300ccc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe2300d0160 tx=0x7fe22802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b32067e35c95c16 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22e7fc640 1 -- 172.21.15.107:0/37056176 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22802c950 con 0x7fe2300b1080 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22e7fc640 1 -- 172.21.15.107:0/37056176 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22802cab0 con 0x7fe2300b1080 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22e7fc640 1 -- 172.21.15.107:0/37056176 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22802ce40 con 0x7fe2300b1080 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 -- 172.21.15.107:0/37056176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 msgr2=0x7fe2300ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2300ccc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe2300d0160 tx=0x7fe22802bdb0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 -- 172.21.15.107:0/37056176 shutdown_connections 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2300ccc90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 --2- 172.21.15.107:0/37056176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe2300b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 --2- 172.21.15.107:0/37056176 >> v2:172.21.15.120:3300/0 conn(0x7fe23015b260 0x7fe23006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 -- 172.21.15.107:0/37056176 >> 172.21.15.107:0/37056176 conn(0x7fe2300cbb40 msgr2=0x7fe2300cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 -- 172.21.15.107:0/37056176 shutdown_connections 2023-11-24T08:05:44.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.579+0000 7fe22ffff640 1 -- 172.21.15.107:0/37056176 wait complete. 2023-11-24T08:05:44.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.581+0000 7fe22ffff640 1 Processor -- start 2023-11-24T08:05:44.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.581+0000 7fe22ffff640 1 -- start start 2023-11-24T08:05:44.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe23014fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe23015b260 0x7fe2301f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2301511d0 con 0x7fe2300b1080 2023-11-24T08:05:44.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe230150550 con 0x7fe23015b260 2023-11-24T08:05:44.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe22ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe230150830 con 0x7fe23006e580 2023-11-24T08:05:44.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe2267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe2267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41150/0 (socket says 172.21.15.107:41150) 2023-11-24T08:05:44.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.582+0000 7fe2267fc640 1 -- 172.21.15.107:0/2635652528 learned_addr learned my addr 172.21.15.107:0/2635652528 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 -- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 msgr2=0x7fe23014fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe23014fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 -- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe23015b260 msgr2=0x7fe2301f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe23015b260 0x7fe2301f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 -- 172.21.15.107:0/2635652528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228009b30 con 0x7fe2300b1080 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe2267fc640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c7f0 tx=0x7fe21800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218004220 con 0x7fe2300b1080 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218004380 con 0x7fe2300b1080 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218007840 con 0x7fe2300b1080 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe230150b10 con 0x7fe2300b1080 2023-11-24T08:05:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.583+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe230150240 con 0x7fe2300b1080 2023-11-24T08:05:44.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.584+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23004e840 con 0x7fe2300b1080 2023-11-24T08:05:44.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.586+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe218007bb0 con 0x7fe2300b1080 2023-11-24T08:05:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.586+0000 7fe227fff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 0x7fe21006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.586+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe218089590 con 0x7fe2300b1080 2023-11-24T08:05:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.587+0000 7fe22effd640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 0x7fe21006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:44.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.587+0000 7fe22effd640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 0x7fe21006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe22000a770 tx=0x7fe220008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.591+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2180545b0 con 0x7fe2300b1080 2023-11-24T08:05:44.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.785+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2300ce640 con 0x7fe2300b1080 2023-11-24T08:05:44.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.785+0000 7fe227fff640 1 -- 172.21.15.107:0/2635652528 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe218007e60 con 0x7fe2300b1080 2023-11-24T08:05:44.786 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:44.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 msgr2=0x7fe21006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 0x7fe21006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe22000a770 tx=0x7fe220008040 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 msgr2=0x7fe2301f5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c7f0 tx=0x7fe21800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 shutdown_connections 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe21006ae40 0x7fe21006d300 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe23015b260 0x7fe2301f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2300b1080 0x7fe2301f5f00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 --2- 172.21.15.107:0/2635652528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe23006e580 0x7fe23014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.788+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 >> 172.21.15.107:0/2635652528 conn(0x7fe2300cbb40 msgr2=0x7fe2300ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.789+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 shutdown_connections 2023-11-24T08:05:44.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:44.789+0000 7fe22ffff640 1 -- 172.21.15.107:0/2635652528 wait complete. 2023-11-24T08:05:44.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.023+0000 7f8a849dc640 1 Processor -- start 2023-11-24T08:05:45.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.023+0000 7f8a849dc640 1 -- start start 2023-11-24T08:05:45.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.025+0000 7f8a849dc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a8015d580 0x7f8a800d4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.025+0000 7f8a849dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a800cfc80 0x7f8a800d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.025+0000 7f8a849dc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800d05e0 con 0x7f8a800d1e30 2023-11-24T08:05:45.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.025+0000 7f8a849dc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800cf300 con 0x7f8a8015d580 2023-11-24T08:05:45.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.025+0000 7f8a849dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800d1090 con 0x7f8a800cfc80 2023-11-24T08:05:45.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7ed9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a800cfc80 0x7f8a800d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a8015d580 0x7f8a800d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a8015d580 0x7f8a800d4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:44222/0 (socket says 172.21.15.107:44222) 2023-11-24T08:05:45.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 -- 172.21.15.107:0/707243432 learned_addr learned my addr 172.21.15.107:0/707243432 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:45.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 -- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a800cfc80 msgr2=0x7f8a800d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:45.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 --2- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a800cfc80 0x7f8a800d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 -- 172.21.15.107:0/707243432 >> v1:172.21.15.120:6789/0 conn(0x7f8a800d1e30 legacy=0x7f8a800cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.026+0000 7f8a7e59c640 1 -- 172.21.15.107:0/707243432 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800d1370 con 0x7f8a8015d580 2023-11-24T08:05:45.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.028+0000 7f8a7e59c640 1 --2- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8015d580 0x7f8a800d4010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009870 tx=0x7f8a7002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4b4d51b255520f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.028+0000 7f8a7dd9b640 1 -- 172.21.15.107:0/707243432 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70030840 con 0x7f8a8015d580 2023-11-24T08:05:45.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.028+0000 7f8a7dd9b640 1 -- 172.21.15.107:0/707243432 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a700309a0 con 0x7f8a8015d580 2023-11-24T08:05:45.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.028+0000 7f8a7dd9b640 1 -- 172.21.15.107:0/707243432 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70030d30 con 0x7f8a8015d580 2023-11-24T08:05:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 -- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8015d580 msgr2=0x7f8a800d4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 --2- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8015d580 0x7f8a800d4010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009870 tx=0x7f8a7002b980 comp rx=0 tx=0).stop 2023-11-24T08:05:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 -- 172.21.15.107:0/707243432 shutdown_connections 2023-11-24T08:05:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 --2- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a800cfc80 0x7f8a800d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 --2- 172.21.15.107:0/707243432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8015d580 0x7f8a800d4010 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 -- 172.21.15.107:0/707243432 >> 172.21.15.107:0/707243432 conn(0x7f8a8009c770 msgr2=0x7f8a800d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 -- 172.21.15.107:0/707243432 shutdown_connections 2023-11-24T08:05:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.029+0000 7f8a849dc640 1 -- 172.21.15.107:0/707243432 wait complete. 2023-11-24T08:05:45.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.031+0000 7f8a849dc640 1 Processor -- start 2023-11-24T08:05:45.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.032+0000 7f8a849dc640 1 -- start start 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800d1e30 0x7f8a800b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8015d580 0x7f8a8015a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b7650 con 0x7f8a800cfc80 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b7930 con 0x7f8a800d1e30 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a849dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b7c10 con 0x7f8a8015d580 2023-11-24T08:05:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41162/0 (socket says 172.21.15.107:41162) 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a7e59c640 1 -- 172.21.15.107:0/1183426173 learned_addr learned my addr 172.21.15.107:0/1183426173 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a77fff640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800d1e30 0x7f8a800b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.033+0000 7f8a7ed9d640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8015d580 0x7f8a8015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 -- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8015d580 msgr2=0x7f8a8015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8015d580 0x7f8a8015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 -- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800d1e30 msgr2=0x7f8a800b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800d1e30 0x7f8a800b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 -- 172.21.15.107:0/1183426173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700094d0 con 0x7f8a800cfc80 2023-11-24T08:05:45.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a7e59c640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8a70000c00 tx=0x7f8a70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70039c90 con 0x7f8a800cfc80 2023-11-24T08:05:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70039e10 con 0x7f8a800cfc80 2023-11-24T08:05:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70043640 con 0x7f8a800cfc80 2023-11-24T08:05:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800b7ef0 con 0x7f8a800cfc80 2023-11-24T08:05:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.034+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800ccdf0 con 0x7f8a800cfc80 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.036+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8004e8b0 con 0x7f8a800cfc80 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.036+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8a700437a0 con 0x7f8a800cfc80 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.037+0000 7f8a76ffd640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 0x7f8a6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.037+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8a7003e070 con 0x7f8a800cfc80 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.038+0000 7f8a77fff640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 0x7f8a6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.038+0000 7f8a77fff640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 0x7f8a6006d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc420 tx=0x7f8a68009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.042+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a700776a0 con 0x7f8a800cfc80 2023-11-24T08:05:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.230+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8a800c9de0 con 0x7f8a800cfc80 2023-11-24T08:05:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.231+0000 7f8a76ffd640 1 -- 172.21.15.107:0/1183426173 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8a7007acc0 con 0x7f8a800cfc80 2023-11-24T08:05:45.232 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 msgr2=0x7f8a6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 0x7f8a6006d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc420 tx=0x7f8a68009040 comp rx=0 tx=0).stop 2023-11-24T08:05:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 msgr2=0x7f8a800cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8a70000c00 tx=0x7f8a70004730 comp rx=0 tx=0).stop 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 shutdown_connections 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a6006ae40 0x7f8a6006d300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8015d580 0x7f8a8015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800d1e30 0x7f8a800b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 --2- 172.21.15.107:0/1183426173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800cfc80 0x7f8a800cc8b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 >> 172.21.15.107:0/1183426173 conn(0x7f8a8009c770 msgr2=0x7f8a800b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 shutdown_connections 2023-11-24T08:05:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.234+0000 7f8a849dc640 1 -- 172.21.15.107:0/1183426173 wait complete. 2023-11-24T08:05:45.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:05:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.475+0000 7fae50ebb640 1 Processor -- start 2023-11-24T08:05:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.475+0000 7fae50ebb640 1 -- start start 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae50ebb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae4c072160 0x7fae4c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae50ebb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae4c0c49f0 0x7fae4c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae50ebb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c15ec90 con 0x7fae4c06dc40 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae50ebb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c06e090 con 0x7fae4c072160 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae50ebb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0cd530 con 0x7fae4c0c49f0 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae4a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae4c072160 0x7fae4c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae4a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae4c072160 0x7fae4c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41164/0 (socket says 172.21.15.107:41164) 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae4a59c640 1 -- 172.21.15.107:0/1700849309 learned_addr learned my addr 172.21.15.107:0/1700849309 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.477+0000 7fae43fff640 1 --2- 172.21.15.107:0/1700849309 >> v2:172.21.15.120:3300/0 conn(0x7fae4c0c49f0 0x7fae4c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.478+0000 7fae4a59c640 1 -- 172.21.15.107:0/1700849309 >> v2:172.21.15.120:3300/0 conn(0x7fae4c0c49f0 msgr2=0x7fae4c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.478+0000 7fae4a59c640 1 --2- 172.21.15.107:0/1700849309 >> v2:172.21.15.120:3300/0 conn(0x7fae4c0c49f0 0x7fae4c06d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.478+0000 7fae4a59c640 1 -- 172.21.15.107:0/1700849309 >> v1:172.21.15.120:6789/0 conn(0x7fae4c06dc40 legacy=0x7fae4c0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.478+0000 7fae4a59c640 1 -- 172.21.15.107:0/1700849309 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0c4ed0 con 0x7fae4c072160 2023-11-24T08:05:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae4a59c640 1 --2- 172.21.15.107:0/1700849309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c072160 0x7fae4c15e750 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0cef60 tx=0x7fae3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e653517936f0ef5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae49d9b640 1 -- 172.21.15.107:0/1700849309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c031870 con 0x7fae4c072160 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae49d9b640 1 -- 172.21.15.107:0/1700849309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c0319d0 con 0x7fae4c072160 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae49d9b640 1 -- 172.21.15.107:0/1700849309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c031d60 con 0x7fae4c072160 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae50ebb640 1 -- 172.21.15.107:0/1700849309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c072160 msgr2=0x7fae4c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.479+0000 7fae50ebb640 1 --2- 172.21.15.107:0/1700849309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c072160 0x7fae4c15e750 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0cef60 tx=0x7fae3c02b910 comp rx=0 tx=0).stop 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 -- 172.21.15.107:0/1700849309 shutdown_connections 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 --2- 172.21.15.107:0/1700849309 >> v2:172.21.15.120:3300/0 conn(0x7fae4c0c49f0 0x7fae4c06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 --2- 172.21.15.107:0/1700849309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c072160 0x7fae4c15e750 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 -- 172.21.15.107:0/1700849309 >> 172.21.15.107:0/1700849309 conn(0x7fae4c0b0c50 msgr2=0x7fae4c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 -- 172.21.15.107:0/1700849309 shutdown_connections 2023-11-24T08:05:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.480+0000 7fae50ebb640 1 -- 172.21.15.107:0/1700849309 wait complete. 2023-11-24T08:05:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.481+0000 7fae50ebb640 1 Processor -- start 2023-11-24T08:05:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.481+0000 7fae50ebb640 1 -- start start 2023-11-24T08:05:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c06dc40 0x7fae4c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c072160 0x7fae4c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0acb70 con 0x7fae4c0c49f0 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0b3780 con 0x7fae4c06dc40 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.482+0000 7fae50ebb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0acdc0 con 0x7fae4c072160 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41178/0 (socket says 172.21.15.107:41178) 2023-11-24T08:05:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 -- 172.21.15.107:0/3384630536 learned_addr learned my addr 172.21.15.107:0/3384630536 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae43fff640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c072160 0x7fae4c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4a59c640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c06dc40 0x7fae4c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 -- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c072160 msgr2=0x7fae4c0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c072160 0x7fae4c0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 -- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c06dc40 msgr2=0x7fae4c0ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c06dc40 0x7fae4c0ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.483+0000 7fae4ad9d640 1 -- 172.21.15.107:0/3384630536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0094d0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fae4400b970 tx=0x7fae4400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4400cb40 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4400cca0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44014680 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c0ad0a0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.484+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c0ad660 con 0x7fae4c0c49f0 2023-11-24T08:05:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.485+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c04e8b0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.486+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fae4401d3b0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.487+0000 7fae42ffd640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 0x7fae2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:45.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.487+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae44089560 con 0x7fae4c0c49f0 2023-11-24T08:05:45.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.487+0000 7fae4a59c640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 0x7fae2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:45.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.488+0000 7fae4a59c640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 0x7fae2c06d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fae3c036000 tx=0x7fae3c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:45.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.491+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae440544f0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.693+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae4c15dab0 con 0x7fae4c0c49f0 2023-11-24T08:05:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.694+0000 7fae42ffd640 1 -- 172.21.15.107:0/3384630536 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fae44057b10 con 0x7fae4c0c49f0 2023-11-24T08:05:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:05:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 msgr2=0x7fae2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 0x7fae2c06d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fae3c036000 tx=0x7fae3c03c040 comp rx=0 tx=0).stop 2023-11-24T08:05:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 msgr2=0x7fae4c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fae4400b970 tx=0x7fae4400be30 comp rx=0 tx=0).stop 2023-11-24T08:05:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 shutdown_connections 2023-11-24T08:05:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae2c06ae40 0x7fae2c06d300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c0c49f0 0x7fae4c153ee0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c072160 0x7fae4c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 --2- 172.21.15.107:0/3384630536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c06dc40 0x7fae4c0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.696+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 >> 172.21.15.107:0/3384630536 conn(0x7fae4c0b0c50 msgr2=0x7fae4c071600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.697+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 shutdown_connections 2023-11-24T08:05:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:45.697+0000 7fae50ebb640 1 -- 172.21.15.107:0/3384630536 wait complete. 2023-11-24T08:05:45.718 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-11-24T08:05:45.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:45.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-11-24T08:05:45.830 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:05:45.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:05:45.831 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-24T08:05:45.881 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:05:45.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:46.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.104+0000 7fb066797640 1 Processor -- start 2023-11-24T08:05:46.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.104+0000 7fb066797640 1 -- start start 2023-11-24T08:05:46.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.105+0000 7fb066797640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb06015df80 0x7fb0600ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.105+0000 7fb066797640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0600cd470 0x7fb06015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.105+0000 7fb066797640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06015e3f0 con 0x7fb0600b2250 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb066797640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06015a990 con 0x7fb06015df80 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb066797640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06015ab80 con 0x7fb0600cd470 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb05ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb0600b2250 0x7fb0600b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59576/0 (socket says 172.21.15.107:59576) 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb05ffff640 1 -- 172.21.15.107:0/1149955022 learned_addr learned my addr 172.21.15.107:0/1149955022 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/1149955022 >> v2:172.21.15.107:3300/0 conn(0x7fb06015df80 0x7fb0600ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb064d34640 1 --2- 172.21.15.107:0/1149955022 >> v2:172.21.15.120:3300/0 conn(0x7fb0600cd470 0x7fb06015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb05effd640 1 -- 172.21.15.107:0/1149955022 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453995 0 0) 0x7fb06015e3f0 con 0x7fb0600b2250 2023-11-24T08:05:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.106+0000 7fb05effd640 1 -- 172.21.15.107:0/1149955022 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb048003520 con 0x7fb0600b2250 2023-11-24T08:05:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.107+0000 7fb05f7fe640 1 -- 172.21.15.107:0/1149955022 >> v2:172.21.15.120:3300/0 conn(0x7fb0600cd470 msgr2=0x7fb06015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.107+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/1149955022 >> v2:172.21.15.120:3300/0 conn(0x7fb0600cd470 0x7fb06015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.107+0000 7fb05f7fe640 1 -- 172.21.15.107:0/1149955022 >> v1:172.21.15.107:6789/0 conn(0x7fb0600b2250 legacy=0x7fb0600b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.107+0000 7fb05f7fe640 1 -- 172.21.15.107:0/1149955022 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600b2d40 con 0x7fb06015df80 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.108+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/1149955022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06015df80 0x7fb0600ccf30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb0600b2fa0 tx=0x7fb05402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f399baf61522080 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.108+0000 7fb05effd640 1 -- 172.21.15.107:0/1149955022 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054031830 con 0x7fb06015df80 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.108+0000 7fb05effd640 1 -- 172.21.15.107:0/1149955022 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb054031990 con 0x7fb06015df80 2023-11-24T08:05:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.108+0000 7fb05effd640 1 -- 172.21.15.107:0/1149955022 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054031d20 con 0x7fb06015df80 2023-11-24T08:05:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 -- 172.21.15.107:0/1149955022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06015df80 msgr2=0x7fb0600ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 --2- 172.21.15.107:0/1149955022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06015df80 0x7fb0600ccf30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb0600b2fa0 tx=0x7fb05402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 -- 172.21.15.107:0/1149955022 shutdown_connections 2023-11-24T08:05:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 --2- 172.21.15.107:0/1149955022 >> v2:172.21.15.120:3300/0 conn(0x7fb0600cd470 0x7fb06015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 --2- 172.21.15.107:0/1149955022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06015df80 0x7fb0600ccf30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 -- 172.21.15.107:0/1149955022 >> 172.21.15.107:0/1149955022 conn(0x7fb06006a920 msgr2=0x7fb06006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 -- 172.21.15.107:0/1149955022 shutdown_connections 2023-11-24T08:05:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.109+0000 7fb066797640 1 -- 172.21.15.107:0/1149955022 wait complete. 2023-11-24T08:05:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.111+0000 7fb066797640 1 Processor -- start 2023-11-24T08:05:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.111+0000 7fb066797640 1 -- start start 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0600b2250 0x7fb060141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb06015df80 0x7fb0600ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060140e70 con 0x7fb0600cd470 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060140260 con 0x7fb0600b2250 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb066797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb060140540 con 0x7fb06015df80 2023-11-24T08:05:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb05f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb05f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41192/0 (socket says 172.21.15.107:41192) 2023-11-24T08:05:46.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.112+0000 7fb05f7fe640 1 -- 172.21.15.107:0/329973340 learned_addr learned my addr 172.21.15.107:0/329973340 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05ffff640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0600b2250 0x7fb060141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb064d34640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb06015df80 0x7fb0600ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 -- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb06015df80 msgr2=0x7fb0600ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb06015df80 0x7fb0600ad980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 -- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0600b2250 msgr2=0x7fb060141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0600b2250 0x7fb060141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 -- 172.21.15.107:0/329973340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb054009400 con 0x7fb0600cd470 2023-11-24T08:05:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05f7fe640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb054000c00 tx=0x7fb0540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05403a8f0 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05403aa70 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.113+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05403ae20 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.114+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb060140820 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.114+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06013fe30 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.115+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060061a50 con 0x7fb0600cd470 2023-11-24T08:05:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.116+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb05404d450 con 0x7fb0600cd470 2023-11-24T08:05:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.116+0000 7fb05cff9640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 0x7fb04c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.117+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb0540ab5d0 con 0x7fb0600cd470 2023-11-24T08:05:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.117+0000 7fb05ffff640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 0x7fb04c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.118+0000 7fb05ffff640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 0x7fb04c06d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0500011e0 tx=0x7fb050009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.121+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb054076690 con 0x7fb0600cd470 2023-11-24T08:05:46.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.347+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06013ffc0 con 0x7fb0600cd470 2023-11-24T08:05:46.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.349+0000 7fb05cff9640 1 -- 172.21.15.107:0/329973340 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7fb054079cb0 con 0x7fb0600cd470 2023-11-24T08:05:46.350 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:46.350 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:46.351 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.351+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 msgr2=0x7fb04c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 0x7fb04c06d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0500011e0 tx=0x7fb050009040 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 msgr2=0x7fb06013f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb054000c00 tx=0x7fb0540046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 shutdown_connections 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb04c06ae70 0x7fb04c06d330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb06015df80 0x7fb0600ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0600cd470 0x7fb06013f8f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 --2- 172.21.15.107:0/329973340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0600b2250 0x7fb060141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 >> 172.21.15.107:0/329973340 conn(0x7fb06006a920 msgr2=0x7fb0601592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:46.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 shutdown_connections 2023-11-24T08:05:46.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.352+0000 7fb066797640 1 -- 172.21.15.107:0/329973340 wait complete. 2023-11-24T08:05:46.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.583+0000 7ff94a30f640 1 Processor -- start 2023-11-24T08:05:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.584+0000 7ff94a30f640 1 -- start start 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff94a30f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9440cf450 0x7ff9440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff94a30f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9440cdfb0 0x7ff9440ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff94a30f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9440cece0 con 0x7ff94415c7f0 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff94a30f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9440cfe40 con 0x7ff9440cf450 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff94a30f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff944156010 con 0x7ff9440cdfb0 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff943fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff94415c7f0 0x7ff94406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59586/0 (socket says 172.21.15.107:59586) 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff943fff640 1 -- 172.21.15.107:0/4202025831 learned_addr learned my addr 172.21.15.107:0/4202025831 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff93bfff640 1 --2- 172.21.15.107:0/4202025831 >> v2:172.21.15.107:3300/0 conn(0x7ff9440cf450 0x7ff9440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.585+0000 7ff9437fe640 1 -- 172.21.15.107:0/4202025831 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562211872 0 0) 0x7ff9440cece0 con 0x7ff94415c7f0 2023-11-24T08:05:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff9488ac640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9440cdfb0 0x7ff9440ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff9437fe640 1 -- 172.21.15.107:0/4202025831 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003520 con 0x7ff94415c7f0 2023-11-24T08:05:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff93bfff640 1 -- 172.21.15.107:0/4202025831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9440cdfb0 msgr2=0x7ff9440ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff93bfff640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9440cdfb0 0x7ff9440ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff93bfff640 1 -- 172.21.15.107:0/4202025831 >> v1:172.21.15.107:6789/0 conn(0x7ff94415c7f0 legacy=0x7ff94406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.586+0000 7ff93bfff640 1 -- 172.21.15.107:0/4202025831 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94406d040 con 0x7ff9440cf450 2023-11-24T08:05:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.587+0000 7ff93bfff640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cf450 0x7ff9440cf900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0097a0 tx=0x7ff92c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f10236a3ccb2465 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff9437fe640 1 -- 172.21.15.107:0/4202025831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c031830 con 0x7ff9440cf450 2023-11-24T08:05:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff9437fe640 1 -- 172.21.15.107:0/4202025831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c031990 con 0x7ff9440cf450 2023-11-24T08:05:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff9437fe640 1 -- 172.21.15.107:0/4202025831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c031d20 con 0x7ff9440cf450 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 -- 172.21.15.107:0/4202025831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cf450 msgr2=0x7ff9440cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cf450 0x7ff9440cf900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0097a0 tx=0x7ff92c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 -- 172.21.15.107:0/4202025831 shutdown_connections 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9440cdfb0 0x7ff9440ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 --2- 172.21.15.107:0/4202025831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cf450 0x7ff9440cf900 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 -- 172.21.15.107:0/4202025831 >> 172.21.15.107:0/4202025831 conn(0x7ff94406a920 msgr2=0x7ff94406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.588+0000 7ff94a30f640 1 -- 172.21.15.107:0/4202025831 shutdown_connections 2023-11-24T08:05:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.589+0000 7ff94a30f640 1 -- 172.21.15.107:0/4202025831 wait complete. 2023-11-24T08:05:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.590+0000 7ff94a30f640 1 Processor -- start 2023-11-24T08:05:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.590+0000 7ff94a30f640 1 -- start start 2023-11-24T08:05:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9440cf450 0x7ff944152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff94415c7f0 0x7ff944152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944157690 con 0x7ff9440cdfb0 2023-11-24T08:05:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94415a720 con 0x7ff9440cf450 2023-11-24T08:05:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff94a30f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94415a9c0 con 0x7ff94415c7f0 2023-11-24T08:05:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.591+0000 7ff943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41206/0 (socket says 172.21.15.107:41206) 2023-11-24T08:05:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 -- 172.21.15.107:0/1690228542 learned_addr learned my addr 172.21.15.107:0/1690228542 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 -- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff94415c7f0 msgr2=0x7ff944152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff94415c7f0 0x7ff944152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 -- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9440cf450 msgr2=0x7ff944152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9440cf450 0x7ff944152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 -- 172.21.15.107:0/1690228542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c009400 con 0x7ff9440cdfb0 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff943fff640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c410 tx=0x7ff93400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.592+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934005ce0 con 0x7ff9440cdfb0 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.593+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff934005e40 con 0x7ff9440cdfb0 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.593+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9340072f0 con 0x7ff9440cdfb0 2023-11-24T08:05:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.593+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94415aca0 con 0x7ff9440cdfb0 2023-11-24T08:05:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.593+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94415b260 con 0x7ff9440cdfb0 2023-11-24T08:05:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.594+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff944067470 con 0x7ff9440cdfb0 2023-11-24T08:05:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.595+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff934007510 con 0x7ff9440cdfb0 2023-11-24T08:05:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.595+0000 7ff9417fa640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 0x7ff92406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.596+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff934088de0 con 0x7ff9440cdfb0 2023-11-24T08:05:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.596+0000 7ff93bfff640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 0x7ff92406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.596+0000 7ff93bfff640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 0x7ff92406d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff92c036000 tx=0x7ff92c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.600+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff934053e30 con 0x7ff9440cdfb0 2023-11-24T08:05:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.790+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff94406cdd0 con 0x7ff9440cdfb0 2023-11-24T08:05:46.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.791+0000 7ff9417fa640 1 -- 172.21.15.107:0/1690228542 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7ff934017090 con 0x7ff9440cdfb0 2023-11-24T08:05:46.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:46.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 msgr2=0x7ff92406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 0x7ff92406d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff92c036000 tx=0x7ff92c030000 comp rx=0 tx=0).stop 2023-11-24T08:05:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 msgr2=0x7ff9441463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c410 tx=0x7ff93400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 shutdown_connections 2023-11-24T08:05:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff92406ae40 0x7ff92406d300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff94415c7f0 0x7ff944152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9440cf450 0x7ff944152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 --2- 172.21.15.107:0/1690228542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9440cdfb0 0x7ff9441463b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:46.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 >> 172.21.15.107:0/1690228542 conn(0x7ff94406a920 msgr2=0x7ff944155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:46.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.794+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 shutdown_connections 2023-11-24T08:05:46.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:46.795+0000 7ff94a30f640 1 -- 172.21.15.107:0/1690228542 wait complete. 2023-11-24T08:05:46.815 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:04:52.367413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:04:52.965711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:46.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:47.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.032+0000 7fcf39e51640 1 Processor -- start 2023-11-24T08:05:47.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.033+0000 7fcf39e51640 1 -- start start 2023-11-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.034+0000 7fcf39e51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf34072160 0x7fcf3415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.034+0000 7fcf39e51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf340c49f0 0x7fcf3406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.034+0000 7fcf39e51640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3415ec90 con 0x7fcf3406dc40 2023-11-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.034+0000 7fcf39e51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3406e0f0 con 0x7fcf34072160 2023-11-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.034+0000 7fcf39e51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340cd530 con 0x7fcf340c49f0 2023-11-24T08:05:47.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf33fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcf3406dc40 0x7fcf340cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59598/0 (socket says 172.21.15.107:59598) 2023-11-24T08:05:47.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf337fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf34072160 0x7fcf3415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf33fff640 1 -- 172.21.15.107:0/2159293117 learned_addr learned my addr 172.21.15.107:0/2159293117 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf32ffd640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf340c49f0 0x7fcf3406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf337fe640 1 -- 172.21.15.107:0/2159293117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf340c49f0 msgr2=0x7fcf3406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf337fe640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf340c49f0 0x7fcf3406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf337fe640 1 -- 172.21.15.107:0/2159293117 >> v1:172.21.15.107:6789/0 conn(0x7fcf3406dc40 legacy=0x7fcf340cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf337fe640 1 -- 172.21.15.107:0/2159293117 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340c4ed0 con 0x7fcf34072160 2023-11-24T08:05:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.035+0000 7fcf327fc640 1 -- 172.21.15.107:0/2159293117 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73535987 0 0) 0x7fcf2c001110 con 0x7fcf3406dc40 2023-11-24T08:05:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.036+0000 7fcf337fe640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf3415e750 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009d90 tx=0x7fcf2402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96a88127c6f87d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf327fc640 1 -- 172.21.15.107:0/2159293117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2402cc50 con 0x7fcf34072160 2023-11-24T08:05:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf327fc640 1 -- 172.21.15.107:0/2159293117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2402cdb0 con 0x7fcf34072160 2023-11-24T08:05:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf327fc640 1 -- 172.21.15.107:0/2159293117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24034660 con 0x7fcf34072160 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 -- 172.21.15.107:0/2159293117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 msgr2=0x7fcf3415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf3415e750 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009d90 tx=0x7fcf2402bea0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 -- 172.21.15.107:0/2159293117 shutdown_connections 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf340c49f0 0x7fcf3406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 --2- 172.21.15.107:0/2159293117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf3415e750 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 -- 172.21.15.107:0/2159293117 >> 172.21.15.107:0/2159293117 conn(0x7fcf340b0c50 msgr2=0x7fcf340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 -- 172.21.15.107:0/2159293117 shutdown_connections 2023-11-24T08:05:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.037+0000 7fcf39e51640 1 -- 172.21.15.107:0/2159293117 wait complete. 2023-11-24T08:05:47.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.039+0000 7fcf39e51640 1 Processor -- start 2023-11-24T08:05:47.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.039+0000 7fcf39e51640 1 -- start start 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3406dc40 0x7fcf34153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf340c49f0 0x7fcf34167860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340cb890 con 0x7fcf34072160 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34166ef0 con 0x7fcf340c49f0 2023-11-24T08:05:47.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf39e51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34167e90 con 0x7fcf3406dc40 2023-11-24T08:05:47.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.040+0000 7fcf32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41224/0 (socket says 172.21.15.107:41224) 2023-11-24T08:05:47.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 -- 172.21.15.107:0/204151241 learned_addr learned my addr 172.21.15.107:0/204151241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 -- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3406dc40 msgr2=0x7fcf34153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3406dc40 0x7fcf34153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 -- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf340c49f0 msgr2=0x7fcf34167860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf340c49f0 0x7fcf34167860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 -- 172.21.15.107:0/204151241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24009a70 con 0x7fcf34072160 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf32ffd640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00d5f0 tx=0x7fcf1c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c0125f0 con 0x7fcf34072160 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c012750 con 0x7fcf34072160 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.041+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c012b00 con 0x7fcf34072160 2023-11-24T08:05:47.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.042+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf341f3b00 con 0x7fcf34072160 2023-11-24T08:05:47.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.042+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf341f37b0 con 0x7fcf34072160 2023-11-24T08:05:47.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.043+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf341f3970 con 0x7fcf34072160 2023-11-24T08:05:47.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.044+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcf1c012c60 con 0x7fcf34072160 2023-11-24T08:05:47.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.045+0000 7fcf2bfff640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 0x7fcf1406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.045+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcf1c089b80 con 0x7fcf34072160 2023-11-24T08:05:47.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.045+0000 7fcf337fe640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 0x7fcf1406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.046+0000 7fcf337fe640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 0x7fcf1406d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcf24036000 tx=0x7fcf2402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.050+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf1c054bb0 con 0x7fcf34072160 2023-11-24T08:05:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.240+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf340b5220 con 0x7fcf34072160 2023-11-24T08:05:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.241+0000 7fcf2bfff640 1 -- 172.21.15.107:0/204151241 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf1c0581d0 con 0x7fcf34072160 2023-11-24T08:05:47.242 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:05:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 msgr2=0x7fcf1406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 0x7fcf1406d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcf24036000 tx=0x7fcf2402e000 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 msgr2=0x7fcf34167320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00d5f0 tx=0x7fcf1c00dab0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 shutdown_connections 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1406ae40 0x7fcf1406d300 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf340c49f0 0x7fcf34167860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf34072160 0x7fcf34167320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 --2- 172.21.15.107:0/204151241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3406dc40 0x7fcf34153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 >> 172.21.15.107:0/204151241 conn(0x7fcf340b0c50 msgr2=0x7fcf340ac450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.243+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 shutdown_connections 2023-11-24T08:05:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.244+0000 7fcf39e51640 1 -- 172.21.15.107:0/204151241 wait complete. 2023-11-24T08:05:47.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:05:47.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.470+0000 7feeaad53640 1 Processor -- start 2023-11-24T08:05:47.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.470+0000 7feeaad53640 1 -- start start 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feeaad53640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feea406e160 0x7feea40ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feeaad53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7feea4070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feeaad53640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea4161a10 con 0x7feea4146eb0 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feeaad53640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea40cd580 con 0x7feea406e160 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feeaad53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea4161cf0 con 0x7feea40cd110 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea8aef640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feea4146eb0 0x7feea4161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59608/0 (socket says 172.21.15.107:59608) 2023-11-24T08:05:47.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea8aef640 1 -- 172.21.15.107:0/759700507 learned_addr learned my addr 172.21.15.107:0/759700507 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea92f0640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7feea4070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea3fff640 1 --2- 172.21.15.107:0/759700507 >> v2:172.21.15.120:3300/0 conn(0x7feea406e160 0x7feea40ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262996372 0 0) 0x7feea4161a10 con 0x7feea4146eb0 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.472+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee94003520 con 0x7feea4146eb0 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087826348 0 0) 0x7fee94003520 con 0x7feea4146eb0 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 msgr2=0x7feea4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7feea4070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 >> v2:172.21.15.120:3300/0 conn(0x7feea406e160 msgr2=0x7feea40ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 --2- 172.21.15.107:0/759700507 >> v2:172.21.15.120:3300/0 conn(0x7feea406e160 0x7feea40ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4161fd0 con 0x7feea4146eb0 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fee98001690 con 0x7feea4146eb0 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 >> v1:172.21.15.107:6789/0 conn(0x7feea4146eb0 legacy=0x7feea4161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee94003970 0x7fee94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7fee94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7fee94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40cd580 con 0x7fee940047c0 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea4161cf0 con 0x7fee94003970 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9400ba00 con 0x7feea40cd110 2023-11-24T08:05:47.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.473+0000 7feea3fff640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7fee94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea3fff640 1 -- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 msgr2=0x7fee94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea92f0640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7fee94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea8aef640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee94003970 0x7fee94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea3fff640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40cd110 0x7fee94005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea3fff640 1 -- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee94003970 msgr2=0x7fee94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea3fff640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee94003970 0x7fee94003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.474+0000 7feea3fff640 1 -- 172.21.15.107:0/759700507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94003520 con 0x7fee940047c0 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.475+0000 7feea3fff640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7fee94004c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fee9000ba30 tx=0x7fee940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba66b99c72c5d89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.475+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90032680 con 0x7fee940047c0 2023-11-24T08:05:47.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.475+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee900327e0 con 0x7fee940047c0 2023-11-24T08:05:47.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.475+0000 7feea37fe640 1 -- 172.21.15.107:0/759700507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90032b70 con 0x7fee940047c0 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 -- 172.21.15.107:0/759700507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 msgr2=0x7fee94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 --2- 172.21.15.107:0/759700507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7fee94004c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fee9000ba30 tx=0x7fee940057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feea92f0640 1 -- 172.21.15.107:0/759700507 reap_dead start 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 -- 172.21.15.107:0/759700507 shutdown_connections 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 -- 172.21.15.107:0/759700507 >> 172.21.15.107:0/759700507 conn(0x7feea40b0c50 msgr2=0x7feea40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 -- 172.21.15.107:0/759700507 shutdown_connections 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.476+0000 7feeaad53640 1 -- 172.21.15.107:0/759700507 wait complete. 2023-11-24T08:05:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.477+0000 7feeaad53640 1 Processor -- start 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 -- start start 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40b1d40 0x7feea40b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feea4143090 0x7feea4143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40b3440 con 0x7fee940047c0 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea4143a50 con 0x7feea4143090 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.478+0000 7feeaad53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea40b27c0 con 0x7feea40b1d40 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41250/0 (socket says 172.21.15.107:41250) 2023-11-24T08:05:47.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 -- 172.21.15.107:0/2971056017 learned_addr learned my addr 172.21.15.107:0/2971056017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 -- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40b1d40 msgr2=0x7feea40b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40b1d40 0x7feea40b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 -- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feea4143090 msgr2=0x7feea4143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feea4143090 0x7feea4143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 -- 172.21.15.107:0/2971056017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40b2aa0 con 0x7fee940047c0 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.479+0000 7feea8aef640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee98000c00 tx=0x7fee9800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.480+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980072d0 con 0x7fee940047c0 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.480+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee98007430 con 0x7fee940047c0 2023-11-24T08:05:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.480+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980047e0 con 0x7fee940047c0 2023-11-24T08:05:47.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.480+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea40b2d80 con 0x7fee940047c0 2023-11-24T08:05:47.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.480+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea40b3860 con 0x7fee940047c0 2023-11-24T08:05:47.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.481+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea404e8b0 con 0x7fee940047c0 2023-11-24T08:05:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.482+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fee98004940 con 0x7fee940047c0 2023-11-24T08:05:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.483+0000 7feea17fa640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 0x7fee8806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.483+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fee98088f00 con 0x7fee940047c0 2023-11-24T08:05:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.483+0000 7feea3fff640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 0x7fee8806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.484+0000 7feea3fff640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 0x7fee8806d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fee9000ba00 tx=0x7fee90032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.487+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee98053f30 con 0x7fee940047c0 2023-11-24T08:05:47.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.662+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feea406ded0 con 0x7fee940047c0 2023-11-24T08:05:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.663+0000 7feea17fa640 1 -- 172.21.15.107:0/2971056017 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee980190a0 con 0x7fee940047c0 2023-11-24T08:05:47.664 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:05:47.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 msgr2=0x7fee8806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 0x7fee8806d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fee9000ba00 tx=0x7fee90032000 comp rx=0 tx=0).stop 2023-11-24T08:05:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 msgr2=0x7feea40b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee98000c00 tx=0x7fee9800db50 comp rx=0 tx=0).stop 2023-11-24T08:05:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 shutdown_connections 2023-11-24T08:05:47.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fee8806ae40 0x7fee8806d300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feea4143090 0x7feea4143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feea40b1d40 0x7feea40b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 --2- 172.21.15.107:0/2971056017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee940047c0 0x7feea40b4780 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 >> 172.21.15.107:0/2971056017 conn(0x7feea40b0c50 msgr2=0x7feea4145ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.665+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 shutdown_connections 2023-11-24T08:05:47.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.666+0000 7feeaad53640 1 -- 172.21.15.107:0/2971056017 wait complete. 2023-11-24T08:05:47.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:05:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.896+0000 7fb0d8cad640 1 Processor -- start 2023-11-24T08:05:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.896+0000 7fb0d8cad640 1 -- start start 2023-11-24T08:05:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.897+0000 7fb0d8cad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0d40b0b40 0x7fb0d40b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.897+0000 7fb0d8cad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 0x7fb0d40cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.897+0000 7fb0d8cad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d40b14d0 con 0x7fb0d415d310 2023-11-24T08:05:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.897+0000 7fb0d8cad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d406cd10 con 0x7fb0d40b0b40 2023-11-24T08:05:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.897+0000 7fb0d8cad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40ae5d0 con 0x7fb0d40cce40 2023-11-24T08:05:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0d37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb0d415d310 0x7fb0d40ca120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48620/0 (socket says 172.21.15.107:48620) 2023-11-24T08:05:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0d37fe640 1 -- 172.21.15.107:0/1704516137 learned_addr learned my addr 172.21.15.107:0/1704516137 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0cbfff640 1 --2- 172.21.15.107:0/1704516137 >> v2:172.21.15.120:3300/0 conn(0x7fb0d40b0b40 0x7fb0d40b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0d3fff640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 0x7fb0d40cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629537830 0 0) 0x7fb0c4001110 con 0x7fb0d415d310 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.898+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c0003520 con 0x7fb0d415d310 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749384625 0 0) 0x7fb0c0003520 con 0x7fb0d415d310 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 msgr2=0x7fb0d40cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 0x7fb0d40cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 >> v2:172.21.15.120:3300/0 conn(0x7fb0d40b0b40 msgr2=0x7fb0d40b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 --2- 172.21.15.107:0/1704516137 >> v2:172.21.15.120:3300/0 conn(0x7fb0d40b0b40 0x7fb0d40b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40ae8b0 con 0x7fb0d415d310 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb0c4001820 con 0x7fb0d415d310 2023-11-24T08:05:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 >> v1:172.21.15.107:6789/0 conn(0x7fb0d415d310 legacy=0x7fb0d40ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c00047c0 0x7fb0c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 0x7fb0c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d406cd10 con 0x7fb0c0003970 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40ae5d0 con 0x7fb0c00047c0 2023-11-24T08:05:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c000b980 con 0x7fb0d40cce40 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0cbfff640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c00047c0 0x7fb0c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 -- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 msgr2=0x7fb0c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d40cce40 0x7fb0c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 -- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c00047c0 msgr2=0x7fb0c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c00047c0 0x7fb0c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.899+0000 7fb0d37fe640 1 -- 172.21.15.107:0/1704516137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0003520 con 0x7fb0c0003970 2023-11-24T08:05:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.900+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0c0003d90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c160 tx=0x7fb0c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0744706620c111 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4005ad0 con 0x7fb0c0003970 2023-11-24T08:05:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c4005c30 con 0x7fb0c0003970 2023-11-24T08:05:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d2ffd640 1 -- 172.21.15.107:0/1704516137 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c402c810 con 0x7fb0c0003970 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 -- 172.21.15.107:0/1704516137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 msgr2=0x7fb0c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/1704516137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0c0003d90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c160 tx=0x7fb0c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d3fff640 1 -- 172.21.15.107:0/1704516137 reap_dead start 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 -- 172.21.15.107:0/1704516137 shutdown_connections 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 -- 172.21.15.107:0/1704516137 >> 172.21.15.107:0/1704516137 conn(0x7fb0d40cbb40 msgr2=0x7fb0d40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 -- 172.21.15.107:0/1704516137 shutdown_connections 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.901+0000 7fb0d8cad640 1 -- 172.21.15.107:0/1704516137 wait complete. 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.903+0000 7fb0d8cad640 1 Processor -- start 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.903+0000 7fb0d8cad640 1 -- start start 2023-11-24T08:05:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d415c500 0x7fb0d415fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 0x7fb0d4160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d409aa90 con 0x7fb0c0003970 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40b3420 con 0x7fb0d415c980 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d8cad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d409a7b0 con 0x7fb0d415c500 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41626/0 (socket says 172.21.15.107:41626) 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.904+0000 7fb0d37fe640 1 -- 172.21.15.107:0/2804707894 learned_addr learned my addr 172.21.15.107:0/2804707894 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d3fff640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 0x7fb0d4160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0cbfff640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d415c500 0x7fb0d415fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 -- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d415c500 msgr2=0x7fb0d415fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d415c500 0x7fb0d415fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 -- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 msgr2=0x7fb0d4160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 0x7fb0d4160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 -- 172.21.15.107:0/2804707894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40b2400 con 0x7fb0c0003970 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d3fff640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 0x7fb0d4160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d37fe640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c160 tx=0x7fb0c4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c402cc20 con 0x7fb0c0003970 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c402cd80 con 0x7fb0c0003970 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40072b0 con 0x7fb0c0003970 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d40b26e0 con 0x7fb0c0003970 2023-11-24T08:05:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.905+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d4099a40 con 0x7fb0c0003970 2023-11-24T08:05:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.906+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d40b2260 con 0x7fb0c0003970 2023-11-24T08:05:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.907+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb0c4032070 con 0x7fb0c0003970 2023-11-24T08:05:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.908+0000 7fb0d0ff9640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 0x7fb0b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.908+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb0c40aa9d0 con 0x7fb0c0003970 2023-11-24T08:05:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.908+0000 7fb0cbfff640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 0x7fb0b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.909+0000 7fb0cbfff640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 0x7fb0b406d300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0d409ca40 tx=0x7fb0bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:47.912+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0d40b2260 con 0x7fb0c0003970 2023-11-24T08:05:48.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.102+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0d40b14d0 con 0x7fb0c0003970 2023-11-24T08:05:48.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.103+0000 7fb0d0ff9640 1 -- 172.21.15.107:0/2804707894 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0d40b14d0 con 0x7fb0c0003970 2023-11-24T08:05:48.104 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:05:48.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.105+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 msgr2=0x7fb0b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.105+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 0x7fb0b406d300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0d409ca40 tx=0x7fb0bc008040 comp rx=0 tx=0).stop 2023-11-24T08:05:48.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.105+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 msgr2=0x7fb0d40b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.105+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c160 tx=0x7fb0c4001ae0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 shutdown_connections 2023-11-24T08:05:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0b406ae40 0x7fb0b406d300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0d415c980 0x7fb0d4160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0d415c500 0x7fb0d415fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 --2- 172.21.15.107:0/2804707894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0003970 0x7fb0d40b2ee0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 >> 172.21.15.107:0/2804707894 conn(0x7fb0d40cbb40 msgr2=0x7fb0d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:48.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 shutdown_connections 2023-11-24T08:05:48.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.106+0000 7fb0d8cad640 1 -- 172.21.15.107:0/2804707894 wait complete. 2023-11-24T08:05:48.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.332+0000 7fdf48dc5640 1 Processor -- start 2023-11-24T08:05:48.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.332+0000 7fdf48dc5640 1 -- start start 2023-11-24T08:05:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.333+0000 7fdf48dc5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf4415c7f0 0x7fdf4406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf48dc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf48dc5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf440cece0 con 0x7fdf440cf450 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf48dc5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf440cfe40 con 0x7fdf4415c7f0 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf48dc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44156010 con 0x7fdf440ce6c0 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf42d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf4259c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf4415c7f0 0x7fdf4406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf42d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34402/0 (socket says 172.21.15.107:34402) 2023-11-24T08:05:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.334+0000 7fdf42d9d640 1 -- 172.21.15.107:0/1872192971 learned_addr learned my addr 172.21.15.107:0/1872192971 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.335+0000 7fdf42d9d640 1 -- 172.21.15.107:0/1872192971 >> v2:172.21.15.120:3300/0 conn(0x7fdf4415c7f0 msgr2=0x7fdf4406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.335+0000 7fdf42d9d640 1 --2- 172.21.15.107:0/1872192971 >> v2:172.21.15.120:3300/0 conn(0x7fdf4415c7f0 0x7fdf4406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.335+0000 7fdf42d9d640 1 -- 172.21.15.107:0/1872192971 >> v1:172.21.15.120:6789/0 conn(0x7fdf440cf450 legacy=0x7fdf440cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.335+0000 7fdf42d9d640 1 -- 172.21.15.107:0/1872192971 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4406d040 con 0x7fdf440ce6c0 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.336+0000 7fdf42d9d640 1 --2- 172.21.15.107:0/1872192971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf3402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb1ad3755024a5d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf4159a640 1 -- 172.21.15.107:0/1872192971 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34030840 con 0x7fdf440ce6c0 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf4159a640 1 -- 172.21.15.107:0/1872192971 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf340309a0 con 0x7fdf440ce6c0 2023-11-24T08:05:48.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf4159a640 1 -- 172.21.15.107:0/1872192971 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34030d30 con 0x7fdf440ce6c0 2023-11-24T08:05:48.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf48dc5640 1 -- 172.21.15.107:0/1872192971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 msgr2=0x7fdf440cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/1872192971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf3402b980 comp rx=0 tx=0).stop 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf48dc5640 1 -- 172.21.15.107:0/1872192971 shutdown_connections 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.337+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/1872192971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440ce6c0 0x7fdf440cf900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.338+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/1872192971 >> v2:172.21.15.120:3300/0 conn(0x7fdf4415c7f0 0x7fdf4406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.338+0000 7fdf48dc5640 1 -- 172.21.15.107:0/1872192971 >> 172.21.15.107:0/1872192971 conn(0x7fdf4406a920 msgr2=0x7fdf4406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.338+0000 7fdf48dc5640 1 -- 172.21.15.107:0/1872192971 shutdown_connections 2023-11-24T08:05:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.338+0000 7fdf48dc5640 1 -- 172.21.15.107:0/1872192971 wait complete. 2023-11-24T08:05:48.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.340+0000 7fdf48dc5640 1 Processor -- start 2023-11-24T08:05:48.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.340+0000 7fdf48dc5640 1 -- start start 2023-11-24T08:05:48.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf440ce6c0 0x7fdf44145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440cf450 0x7fdf4415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441465b0 con 0x7fdf4415c7f0 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44146890 con 0x7fdf440ce6c0 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.341+0000 7fdf48dc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44146b70 con 0x7fdf440cf450 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41634/0 (socket says 172.21.15.107:41634) 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 -- 172.21.15.107:0/3858846736 learned_addr learned my addr 172.21.15.107:0/3858846736 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf4259c640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf440ce6c0 0x7fdf44145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf41d9b640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440cf450 0x7fdf4415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 -- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440cf450 msgr2=0x7fdf4415eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440cf450 0x7fdf4415eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 -- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf440ce6c0 msgr2=0x7fdf44145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf440ce6c0 0x7fdf44145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 -- 172.21.15.107:0/3858846736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340094d0 con 0x7fdf4415c7f0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.342+0000 7fdf42d9d640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.343+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34039c90 con 0x7fdf4415c7f0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.343+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34039e10 con 0x7fdf4415c7f0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.343+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34043640 con 0x7fdf4415c7f0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.343+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf44146e50 con 0x7fdf4415c7f0 2023-11-24T08:05:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.343+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf44146090 con 0x7fdf4415c7f0 2023-11-24T08:05:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.344+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf440afe50 con 0x7fdf4415c7f0 2023-11-24T08:05:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.345+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf340437a0 con 0x7fdf4415c7f0 2023-11-24T08:05:48.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.346+0000 7fdf3affd640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 0x7fdf2406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.346+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdf3403e070 con 0x7fdf4415c7f0 2023-11-24T08:05:48.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.346+0000 7fdf4259c640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 0x7fdf2406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.347+0000 7fdf4259c640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 0x7fdf2406d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0073f0 tx=0x7fdf3c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.352+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf34077600 con 0x7fdf4415c7f0 2023-11-24T08:05:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.579+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf441466f0 con 0x7fdf4415c7f0 2023-11-24T08:05:48.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.580+0000 7fdf3affd640 1 -- 172.21.15.107:0/3858846736 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7fdf441466f0 con 0x7fdf4415c7f0 2023-11-24T08:05:48.582 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:48.582 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.583+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 msgr2=0x7fdf2406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.583+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 0x7fdf2406d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0073f0 tx=0x7fdf3c00b040 comp rx=0 tx=0).stop 2023-11-24T08:05:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.583+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 msgr2=0x7fdf4415f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.583+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf34004730 comp rx=0 tx=0).stop 2023-11-24T08:05:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 shutdown_connections 2023-11-24T08:05:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf2406ae40 0x7fdf2406d300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf4415c7f0 0x7fdf4415f400 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf440cf450 0x7fdf4415eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 --2- 172.21.15.107:0/3858846736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf440ce6c0 0x7fdf44145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 >> 172.21.15.107:0/3858846736 conn(0x7fdf4406a920 msgr2=0x7fdf4406ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 shutdown_connections 2023-11-24T08:05:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.584+0000 7fdf48dc5640 1 -- 172.21.15.107:0/3858846736 wait complete. 2023-11-24T08:05:48.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.792+0000 7f6db47a9640 1 Processor -- start 2023-11-24T08:05:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.792+0000 7f6db47a9640 1 -- start start 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db47a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6dac15eaf0 0x7f6dac0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db47a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac06d180 con 0x7f6dac0b3890 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db47a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac06ba70 con 0x7f6dac1461d0 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db47a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac0b3d20 con 0x7f6dac15eaf0 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db2d46640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6dac0b3890 0x7f6dac167110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48628/0 (socket says 172.21.15.107:48628) 2023-11-24T08:05:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db2d46640 1 -- 172.21.15.107:0/1154003817 learned_addr learned my addr 172.21.15.107:0/1154003817 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db1d44640 1 --2- 172.21.15.107:0/1154003817 >> v2:172.21.15.120:3300/0 conn(0x7f6dac15eaf0 0x7f6dac0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.794+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321607263 0 0) 0x7f6dac06d180 con 0x7f6dac0b3890 2023-11-24T08:05:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94003520 con 0x7f6dac0b3890 2023-11-24T08:05:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088744535 0 0) 0x7f6d94003520 con 0x7f6dac0b3890 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 >> v2:172.21.15.120:3300/0 conn(0x7f6dac15eaf0 msgr2=0x7f6dac0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 --2- 172.21.15.107:0/1154003817 >> v2:172.21.15.120:3300/0 conn(0x7f6dac15eaf0 0x7f6dac0b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 >> v1:172.21.15.120:6789/0 conn(0x7f6dac1461d0 legacy=0x7f6dac0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac06bd50 con 0x7f6dac0b3890 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6da40019f0 con 0x7f6dac0b3890 2023-11-24T08:05:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 >> v1:172.21.15.107:6789/0 conn(0x7f6dac0b3890 legacy=0x7f6dac167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 0x7f6d94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d940047c0 0x7f6d94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d94005180 0x7f6d9400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0b3d20 con 0x7f6d940039c0 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9400c080 con 0x7f6d94005180 2023-11-24T08:05:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9400c240 con 0x7f6d940047c0 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.795+0000 7f6db2545640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 0x7f6d94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2d46640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d94005180 0x7f6d9400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2545640 1 -- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d940047c0 msgr2=0x7f6d94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2545640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d940047c0 0x7f6d94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2545640 1 -- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d94005180 msgr2=0x7f6d9400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2545640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d94005180 0x7f6d9400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2545640 1 -- 172.21.15.107:0/1154003817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94003520 con 0x7f6d940039c0 2023-11-24T08:05:48.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.796+0000 7f6db2d46640 1 -- 172.21.15.107:0/1154003817 reap_dead start 2023-11-24T08:05:48.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db2545640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 0x7f6d94003de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c001750 tx=0x7f6d940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2699797bb187fb17 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0052d0 con 0x7f6d940039c0 2023-11-24T08:05:48.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c02e050 con 0x7f6d940039c0 2023-11-24T08:05:48.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db1543640 1 -- 172.21.15.107:0/1154003817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c003830 con 0x7f6d940039c0 2023-11-24T08:05:48.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db47a9640 1 -- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 msgr2=0x7f6d94003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db47a9640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 0x7f6d94003de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c001750 tx=0x7f6d940057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db47a9640 1 -- 172.21.15.107:0/1154003817 shutdown_connections 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db47a9640 1 --2- 172.21.15.107:0/1154003817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d940039c0 0x7f6d94003de0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.797+0000 7f6db47a9640 1 -- 172.21.15.107:0/1154003817 >> 172.21.15.107:0/1154003817 conn(0x7f6dac061340 msgr2=0x7f6dac061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.798+0000 7f6db47a9640 1 -- 172.21.15.107:0/1154003817 shutdown_connections 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.798+0000 7f6db47a9640 1 -- 172.21.15.107:0/1154003817 wait complete. 2023-11-24T08:05:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.799+0000 7f6db47a9640 1 Processor -- start 2023-11-24T08:05:48.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.799+0000 7f6db47a9640 1 -- start start 2023-11-24T08:05:48.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d940039c0 0x7f6dac0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0cdfa0 0x7f6dac0ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0a9130 con 0x7f6dac0a8790 2023-11-24T08:05:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0ce990 con 0x7f6d940039c0 2023-11-24T08:05:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db47a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0b2330 con 0x7f6dac0cdfa0 2023-11-24T08:05:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db1d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.800+0000 7f6db1d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41648/0 (socket says 172.21.15.107:41648) 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 -- 172.21.15.107:0/3108208825 learned_addr learned my addr 172.21.15.107:0/3108208825 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 -- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0cdfa0 msgr2=0x7f6dac0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0cdfa0 0x7f6dac0ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 -- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d940039c0 msgr2=0x7f6dac0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d940039c0 0x7f6dac0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 -- 172.21.15.107:0/3108208825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0cec40 con 0x7f6dac0a8790 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db1d44640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6da800e770 tx=0x7f6da800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da800c730 con 0x7f6dac0a8790 2023-11-24T08:05:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da800c890 con 0x7f6dac0a8790 2023-11-24T08:05:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.801+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac0cef20 con 0x7f6dac0a8790 2023-11-24T08:05:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.802+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da800cc40 con 0x7f6dac0a8790 2023-11-24T08:05:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.802+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac132000 con 0x7f6dac0a8790 2023-11-24T08:05:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.803+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dac067470 con 0x7f6dac0a8790 2023-11-24T08:05:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.804+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6da8013470 con 0x7f6dac0a8790 2023-11-24T08:05:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.804+0000 7f6da2ffd640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 0x7f6d9806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.805+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6da80897d0 con 0x7f6dac0a8790 2023-11-24T08:05:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.805+0000 7f6db2545640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 0x7f6d9806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:48.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.805+0000 7f6db2545640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 0x7f6d9806d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c033000 tx=0x7f6d9c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:48.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:48.809+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6da8054800 con 0x7f6dac0a8790 2023-11-24T08:05:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.000+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dac0d2040 con 0x7f6dac0a8790 2023-11-24T08:05:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.003+0000 7f6da2ffd640 1 -- 172.21.15.107:0/3108208825 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f6da8017090 con 0x7f6dac0a8790 2023-11-24T08:05:49.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:49.005 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:49.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 msgr2=0x7f6d9806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 0x7f6d9806d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c033000 tx=0x7f6d9c02f000 comp rx=0 tx=0).stop 2023-11-24T08:05:49.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 msgr2=0x7f6dac0a8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6da800e770 tx=0x7f6da800ec30 comp rx=0 tx=0).stop 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 shutdown_connections 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6d9806ae70 0x7f6d9806d330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0cdfa0 0x7f6dac0ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dac0a8790 0x7f6dac0a8bf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 --2- 172.21.15.107:0/3108208825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d940039c0 0x7f6dac0b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.005+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 >> 172.21.15.107:0/3108208825 conn(0x7f6dac061340 msgr2=0x7f6dac143090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:49.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.006+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 shutdown_connections 2023-11-24T08:05:49.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.006+0000 7f6db47a9640 1 -- 172.21.15.107:0/3108208825 wait complete. 2023-11-24T08:05:49.025 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:05:49.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.232+0000 7f94a9d33640 1 Processor -- start 2023-11-24T08:05:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.232+0000 7f94a9d33640 1 -- start start 2023-11-24T08:05:49.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a9d33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94a415df80 0x7f94a40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a9d33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94a40cd470 0x7f94a415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a9d33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a415e3f0 con 0x7f94a40b2250 2023-11-24T08:05:49.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a9d33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a415a990 con 0x7f94a40cd470 2023-11-24T08:05:49.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a9d33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a415ab80 con 0x7f94a415df80 2023-11-24T08:05:49.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94a40b2250 0x7f94a40b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48630/0 (socket says 172.21.15.107:48630) 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a37fe640 1 -- 172.21.15.107:0/3492658969 learned_addr learned my addr 172.21.15.107:0/3492658969 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f94a3fff640 1 --2- 172.21.15.107:0/3492658969 >> v2:172.21.15.107:3300/0 conn(0x7f94a40cd470 0x7f94a415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.234+0000 7f949bfff640 1 --2- 172.21.15.107:0/3492658969 >> v2:172.21.15.120:3300/0 conn(0x7f94a415df80 0x7f94a40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a2ffd640 1 -- 172.21.15.107:0/3492658969 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785681780 0 0) 0x7f94a415e3f0 con 0x7f94a40b2250 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a2ffd640 1 -- 172.21.15.107:0/3492658969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003520 con 0x7f94a40b2250 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a3fff640 1 -- 172.21.15.107:0/3492658969 >> v2:172.21.15.120:3300/0 conn(0x7f94a415df80 msgr2=0x7f94a40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a3fff640 1 --2- 172.21.15.107:0/3492658969 >> v2:172.21.15.120:3300/0 conn(0x7f94a415df80 0x7f94a40ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a3fff640 1 -- 172.21.15.107:0/3492658969 >> v1:172.21.15.107:6789/0 conn(0x7f94a40b2250 legacy=0x7f94a40b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.235+0000 7f94a3fff640 1 -- 172.21.15.107:0/3492658969 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40b2d40 con 0x7f94a40cd470 2023-11-24T08:05:49.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.236+0000 7f94a3fff640 1 --2- 172.21.15.107:0/3492658969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40cd470 0x7f94a415b080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f94a40ae5c0 tx=0x7f949c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dd718d2f1ea1d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.236+0000 7f94a2ffd640 1 -- 172.21.15.107:0/3492658969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c02cc70 con 0x7f94a40cd470 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.236+0000 7f94a2ffd640 1 -- 172.21.15.107:0/3492658969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c02cdd0 con 0x7f94a40cd470 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.236+0000 7f94a2ffd640 1 -- 172.21.15.107:0/3492658969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c034640 con 0x7f94a40cd470 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 -- 172.21.15.107:0/3492658969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40cd470 msgr2=0x7f94a415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 --2- 172.21.15.107:0/3492658969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40cd470 0x7f94a415b080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f94a40ae5c0 tx=0x7f949c02bec0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 -- 172.21.15.107:0/3492658969 shutdown_connections 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 --2- 172.21.15.107:0/3492658969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40cd470 0x7f94a415b080 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 --2- 172.21.15.107:0/3492658969 >> v2:172.21.15.120:3300/0 conn(0x7f94a415df80 0x7f94a40ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 -- 172.21.15.107:0/3492658969 >> 172.21.15.107:0/3492658969 conn(0x7f94a406a920 msgr2=0x7f94a406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 -- 172.21.15.107:0/3492658969 shutdown_connections 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.237+0000 7f94a9d33640 1 -- 172.21.15.107:0/3492658969 wait complete. 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.239+0000 7f94a9d33640 1 Processor -- start 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.239+0000 7f94a9d33640 1 -- start start 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94a40cd470 0x7f94a41468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94a415df80 0x7f94a415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a41575a0 con 0x7f94a40b2250 2023-11-24T08:05:49.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4157880 con 0x7f94a40cd470 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a9d33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40aebf0 con 0x7f94a415df80 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41664/0 (socket says 172.21.15.107:41664) 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.240+0000 7f94a37fe640 1 -- 172.21.15.107:0/1119598550 learned_addr learned my addr 172.21.15.107:0/1119598550 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f949bfff640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94a40cd470 0x7f94a41468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a3fff640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94a415df80 0x7f94a415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 -- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94a415df80 msgr2=0x7f94a415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94a415df80 0x7f94a415c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 -- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94a40cd470 msgr2=0x7f94a41468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94a40cd470 0x7f94a41468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 -- 172.21.15.107:0/1119598550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009a70 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a37fe640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f949400c430 tx=0x7f949400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494005ce0 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9494005e40 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a40aeed0 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940072f0 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.241+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a4157200 con 0x7f94a40b2250 2023-11-24T08:05:49.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.243+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a4067470 con 0x7f94a40b2250 2023-11-24T08:05:49.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.243+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9494007450 con 0x7f94a40b2250 2023-11-24T08:05:49.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.244+0000 7f94a0ff9640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 0x7f948406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.244+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9494088d10 con 0x7f94a40b2250 2023-11-24T08:05:49.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.244+0000 7f949bfff640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 0x7f948406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.245+0000 7f949bfff640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 0x7f948406d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f948c006fd0 tx=0x7f948c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.249+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94940577d0 con 0x7f94a40b2250 2023-11-24T08:05:49.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.460+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a4166100 con 0x7f94a40b2250 2023-11-24T08:05:49.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.461+0000 7f94a0ff9640 1 -- 172.21.15.107:0/1119598550 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f9494017020 con 0x7f94a40b2250 2023-11-24T08:05:49.462 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:49.463 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:49.463 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 msgr2=0x7f948406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 0x7f948406d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f948c006fd0 tx=0x7f948c008040 comp rx=0 tx=0).stop 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 msgr2=0x7f94a4146390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f949400c430 tx=0x7f949400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 shutdown_connections 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948406ae40 0x7f948406d300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94a415df80 0x7f94a415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94a40cd470 0x7f94a41468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 --2- 172.21.15.107:0/1119598550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94a40b2250 0x7f94a4146390 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 >> 172.21.15.107:0/1119598550 conn(0x7f94a406a920 msgr2=0x7f94a4158fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:49.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 shutdown_connections 2023-11-24T08:05:49.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.464+0000 7f94a9d33640 1 -- 172.21.15.107:0/1119598550 wait complete. 2023-11-24T08:05:49.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:05:49.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.679+0000 7f60daf33640 1 Processor -- start 2023-11-24T08:05:49.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.679+0000 7f60daf33640 1 -- start start 2023-11-24T08:05:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60daf33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60d4159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60daf33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d4070a30 con 0x7f60d4159d60 2023-11-24T08:05:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60daf33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d40715b0 con 0x7f60d4073d30 2023-11-24T08:05:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60daf33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4071a40 con 0x7f60d40cda40 2023-11-24T08:05:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60d94d0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f60d4159d60 0x7f60d415a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48642/0 (socket says 172.21.15.107:48642) 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60d94d0640 1 -- 172.21.15.107:0/1159895710 learned_addr learned my addr 172.21.15.107:0/1159895710 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.681+0000 7f60d3fff640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60d4159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897483997 0 0) 0x7f60d4070a30 con 0x7f60d4159d60 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60bc003520 con 0x7f60d4159d60 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486412563 0 0) 0x7f60bc003520 con 0x7f60d4159d60 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 msgr2=0x7f60d4159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60d4159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 >> v1:172.21.15.120:6789/0 conn(0x7f60d4073d30 legacy=0x7f60d40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40cde60 con 0x7f60d4159d60 2023-11-24T08:05:49.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f60cc0019f0 con 0x7f60d4159d60 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 >> v1:172.21.15.107:6789/0 conn(0x7f60d4159d60 legacy=0x7f60d415a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60bc0039a0 0x7f60bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc004d00 0x7f60bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4071a40 con 0x7f60bc0039a0 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc00ba30 con 0x7f60bc004d00 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc00de00 con 0x7f60d40cda40 2023-11-24T08:05:49.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.682+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60bc0039a0 0x7f60bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d3fff640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d8ccf640 1 -- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 msgr2=0x7f60bc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d40cda40 0x7f60bc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d94d0640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc004d00 0x7f60bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d8ccf640 1 -- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc004d00 msgr2=0x7f60bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:49.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc004d00 0x7f60bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d8ccf640 1 -- 172.21.15.107:0/1159895710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc003520 con 0x7f60bc0039a0 2023-11-24T08:05:49.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.683+0000 7f60d94d0640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc004d00 0x7f60bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:49.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.684+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60bc0039a0 0x7f60bc003dc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60c8001720 tx=0x7f60bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63776b1edc923bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.684+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80052d0 con 0x7f60bc0039a0 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.684+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c802e050 con 0x7f60bc0039a0 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.684+0000 7f60d37fe640 1 -- 172.21.15.107:0/1159895710 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80035f0 con 0x7f60bc0039a0 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 -- 172.21.15.107:0/1159895710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60bc0039a0 msgr2=0x7f60bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 --2- 172.21.15.107:0/1159895710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60bc0039a0 0x7f60bc003dc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60c8001720 tx=0x7f60bc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60d94d0640 1 -- 172.21.15.107:0/1159895710 reap_dead start 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 -- 172.21.15.107:0/1159895710 shutdown_connections 2023-11-24T08:05:49.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 -- 172.21.15.107:0/1159895710 >> 172.21.15.107:0/1159895710 conn(0x7f60d40cf4a0 msgr2=0x7f60d40cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 -- 172.21.15.107:0/1159895710 shutdown_connections 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.685+0000 7f60daf33640 1 -- 172.21.15.107:0/1159895710 wait complete. 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.687+0000 7f60daf33640 1 Processor -- start 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.687+0000 7f60daf33640 1 -- start start 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc0039a0 0x7f60d414ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d415b540 0x7f60d415b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40c91c0 con 0x7f60d40c87f0 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4155ae0 con 0x7f60bc0039a0 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60daf33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4155dc0 con 0x7f60d415b540 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41686/0 (socket says 172.21.15.107:41686) 2023-11-24T08:05:49.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 -- 172.21.15.107:0/1957606374 learned_addr learned my addr 172.21.15.107:0/1957606374 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:49.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 -- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d415b540 msgr2=0x7f60d415b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d3fff640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d415b540 0x7f60d415b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d415b540 0x7f60d415b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc0039a0 0x7f60d414ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 -- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc0039a0 msgr2=0x7f60d414ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc0039a0 0x7f60d414ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.688+0000 7f60d94d0640 1 -- 172.21.15.107:0/1957606374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d41560a0 con 0x7f60d40c87f0 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60d94d0640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f60d415a140 tx=0x7f60cc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00bb80 con 0x7f60d40c87f0 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc00bce0 con 0x7f60d40c87f0 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0072b0 con 0x7f60d40c87f0 2023-11-24T08:05:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d4156380 con 0x7f60d40c87f0 2023-11-24T08:05:49.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.689+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4155700 con 0x7f60d40c87f0 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.690+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d41568a0 con 0x7f60d40c87f0 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.691+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f60cc007410 con 0x7f60d40c87f0 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.692+0000 7f60d17fa640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 0x7f60c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.693+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f60cc089bb0 con 0x7f60d40c87f0 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.693+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 0x7f60c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:49.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.693+0000 7f60d8ccf640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 0x7f60c006d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60c8033000 tx=0x7f60c8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:49.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.697+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60d41568a0 con 0x7f60d40c87f0 2023-11-24T08:05:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.925+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d41568a0 con 0x7f60d40c87f0 2023-11-24T08:05:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.926+0000 7f60d17fa640 1 -- 172.21.15.107:0/1957606374 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4919 (secure 0 0 0) 0x7f60d41568a0 con 0x7f60d40c87f0 2023-11-24T08:05:49.927 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:49.928 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4131445590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4131445590},{"type":"v1","addr":"172.21.15.120:6833","nonce":4131445590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2112296672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2112296672},{"type":"v1","addr":"172.21.15.120:6835","nonce":2112296672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2666607897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2666607897},{"type":"v1","addr":"172.21.15.107:6835","nonce":2666607897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:04:53.954635+0000","modified":"2023-11-24T08:04:57.215293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1122137333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1122137333},{"type":"v1","addr":"172.21.15.107:6837","nonce":1122137333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T08:05:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2023-11-24T08:05:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.928+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 msgr2=0x7f60c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.928+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 0x7f60c006d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60c8033000 tx=0x7f60c8009030 comp rx=0 tx=0).stop 2023-11-24T08:05:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.928+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 msgr2=0x7f60d40c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.928+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f60d415a140 tx=0x7f60cc00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 shutdown_connections 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60c006ae40 0x7f60c006d300 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d40c87f0 0x7f60d40c8c80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d415b540 0x7f60d415b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 --2- 172.21.15.107:0/1957606374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60bc0039a0 0x7f60d414ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 >> 172.21.15.107:0/1957606374 conn(0x7f60d40cf4a0 msgr2=0x7f60d40af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 shutdown_connections 2023-11-24T08:05:49.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:49.929+0000 7f60daf33640 1 -- 172.21.15.107:0/1957606374 wait complete. 2023-11-24T08:05:49.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:50.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.128+0000 7f25ae1a9640 1 Processor -- start 2023-11-24T08:05:50.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.128+0000 7f25ae1a9640 1 -- start start 2023-11-24T08:05:50.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.129+0000 7f25ae1a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25a80cf450 0x7f25a80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.129+0000 7f25ae1a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.129+0000 7f25ae1a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80cece0 con 0x7f25a815c7f0 2023-11-24T08:05:50.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.129+0000 7f25ae1a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a80cfe40 con 0x7f25a80cf450 2023-11-24T08:05:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.129+0000 7f25ae1a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8156010 con 0x7f25a80cdfb0 2023-11-24T08:05:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.130+0000 7f25a7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.130+0000 7f259ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25a80cf450 0x7f25a80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.130+0000 7f25a7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34460/0 (socket says 172.21.15.107:34460) 2023-11-24T08:05:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.130+0000 7f25a7fff640 1 -- 172.21.15.107:0/4090806508 learned_addr learned my addr 172.21.15.107:0/4090806508 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a6ffd640 1 -- 172.21.15.107:0/4090806508 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718965353 0 0) 0x7f25a80cece0 con 0x7f25a815c7f0 2023-11-24T08:05:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a6ffd640 1 -- 172.21.15.107:0/4090806508 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2594003520 con 0x7f25a815c7f0 2023-11-24T08:05:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a7fff640 1 -- 172.21.15.107:0/4090806508 >> v2:172.21.15.120:3300/0 conn(0x7f25a80cf450 msgr2=0x7f25a80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a7fff640 1 --2- 172.21.15.107:0/4090806508 >> v2:172.21.15.120:3300/0 conn(0x7f25a80cf450 0x7f25a80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a7fff640 1 -- 172.21.15.107:0/4090806508 >> v1:172.21.15.120:6789/0 conn(0x7f25a815c7f0 legacy=0x7f25a806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.131+0000 7f25a7fff640 1 -- 172.21.15.107:0/4090806508 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a806d040 con 0x7f25a80cdfb0 2023-11-24T08:05:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.132+0000 7f25a7fff640 1 --2- 172.21.15.107:0/4090806508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009e50 tx=0x7f25a002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10f4d7dfb13e1826 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25a6ffd640 1 -- 172.21.15.107:0/4090806508 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a002cb00 con 0x7f25a80cdfb0 2023-11-24T08:05:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25a6ffd640 1 -- 172.21.15.107:0/4090806508 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a002cc60 con 0x7f25a80cdfb0 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25a6ffd640 1 -- 172.21.15.107:0/4090806508 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00343f0 con 0x7f25a80cdfb0 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 -- 172.21.15.107:0/4090806508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 msgr2=0x7f25a80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/4090806508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009e50 tx=0x7f25a002bf60 comp rx=0 tx=0).stop 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 -- 172.21.15.107:0/4090806508 shutdown_connections 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/4090806508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a80ce420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/4090806508 >> v2:172.21.15.120:3300/0 conn(0x7f25a80cf450 0x7f25a80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 -- 172.21.15.107:0/4090806508 >> 172.21.15.107:0/4090806508 conn(0x7f25a806a920 msgr2=0x7f25a806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 -- 172.21.15.107:0/4090806508 shutdown_connections 2023-11-24T08:05:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.133+0000 7f25ae1a9640 1 -- 172.21.15.107:0/4090806508 wait complete. 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.135+0000 7f25ae1a9640 1 Processor -- start 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.135+0000 7f25ae1a9640 1 -- start start 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a81509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a80cf450 0x7f25a815f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80b3040 con 0x7f25a815c7f0 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8150ee0 con 0x7f25a80cf450 2023-11-24T08:05:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25ae1a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80b2280 con 0x7f25a80cdfb0 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41696/0 (socket says 172.21.15.107:41696) 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.136+0000 7f25a7fff640 1 -- 172.21.15.107:0/484561478 learned_addr learned my addr 172.21.15.107:0/484561478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a77fe640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a81509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 -- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 msgr2=0x7f25a81509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f259ffff640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a80cf450 0x7f25a815f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a81509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 -- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a80cf450 msgr2=0x7f25a815f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a80cf450 0x7f25a815f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 -- 172.21.15.107:0/484561478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009b30 con 0x7f25a815c7f0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a77fe640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a81509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a7fff640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f25a0036040 tx=0x7f25a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0007000 con 0x7f25a815c7f0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0009510 con 0x7f25a815c7f0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a003e690 con 0x7f25a815c7f0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.137+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a80b2560 con 0x7f25a815c7f0 2023-11-24T08:05:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.138+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a80b2b20 con 0x7f25a815c7f0 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.139+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a8067470 con 0x7f25a815c7f0 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.140+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f25a003e7f0 con 0x7f25a815c7f0 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.140+0000 7f25a4ff9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 0x7f258806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.140+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f25a00ac230 con 0x7f25a815c7f0 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.141+0000 7f25a77fe640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 0x7f258806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.141+0000 7f25a77fe640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 0x7f258806d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2598001610 tx=0x7f2598009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.145+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25a0077270 con 0x7f25a815c7f0 2023-11-24T08:05:50.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.328+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25a80cd730 con 0x7f25a815c7f0 2023-11-24T08:05:50.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.330+0000 7f25a4ff9640 1 -- 172.21.15.107:0/484561478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f25a007a890 con 0x7f25a815c7f0 2023-11-24T08:05:50.331 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:50.331 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:37.238141+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:04:52.367413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T08:04:52.965711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:50.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 msgr2=0x7f258806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 0x7f258806d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2598001610 tx=0x7f2598009040 comp rx=0 tx=0).stop 2023-11-24T08:05:50.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 msgr2=0x7f25a815f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f25a0036040 tx=0x7f25a0004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:50.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 shutdown_connections 2023-11-24T08:05:50.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258806ae40 0x7f258806d300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a815c7f0 0x7f25a815f7e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.332+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a80cf450 0x7f25a815f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.333+0000 7f25ae1a9640 1 --2- 172.21.15.107:0/484561478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a80cdfb0 0x7f25a81509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.333+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 >> 172.21.15.107:0/484561478 conn(0x7f25a806a920 msgr2=0x7f25a80af300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:50.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.333+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 shutdown_connections 2023-11-24T08:05:50.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.333+0000 7f25ae1a9640 1 -- 172.21.15.107:0/484561478 wait complete. 2023-11-24T08:05:50.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:05:50.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.541+0000 7f650ffff640 1 Processor -- start 2023-11-24T08:05:50.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.542+0000 7f650ffff640 1 -- start start 2023-11-24T08:05:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f65100b2f40 0x7f65100b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650ffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65100b4130 con 0x7f6510143a00 2023-11-24T08:05:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650ffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65100b3520 con 0x7f65100b2f40 2023-11-24T08:05:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65100b3800 con 0x7f65100b4b00 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650f7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650f7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:34480/0 (socket says 172.21.15.107:34480) 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f650f7fe640 1 -- 172.21.15.107:0/3002575822 learned_addr learned my addr 172.21.15.107:0/3002575822 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.544+0000 7f6507fff640 1 --2- 172.21.15.107:0/3002575822 >> v2:172.21.15.120:3300/0 conn(0x7f65100b2f40 0x7f65100b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.545+0000 7f650f7fe640 1 -- 172.21.15.107:0/3002575822 >> v2:172.21.15.120:3300/0 conn(0x7f65100b2f40 msgr2=0x7f65100b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.545+0000 7f650f7fe640 1 --2- 172.21.15.107:0/3002575822 >> v2:172.21.15.120:3300/0 conn(0x7f65100b2f40 0x7f65100b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.545+0000 7f650f7fe640 1 -- 172.21.15.107:0/3002575822 >> v1:172.21.15.120:6789/0 conn(0x7f6510143a00 legacy=0x7f651015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.545+0000 7f650f7fe640 1 -- 172.21.15.107:0/3002575822 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100b3ae0 con 0x7f65100b4b00 2023-11-24T08:05:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.547+0000 7f650f7fe640 1 --2- 172.21.15.107:0/3002575822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650802db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddcc9d3f51a066db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.547+0000 7f650e7fc640 1 -- 172.21.15.107:0/3002575822 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650802e940 con 0x7f65100b4b00 2023-11-24T08:05:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650e7fc640 1 -- 172.21.15.107:0/3002575822 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650802eaa0 con 0x7f65100b4b00 2023-11-24T08:05:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650e7fc640 1 -- 172.21.15.107:0/3002575822 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650802ee30 con 0x7f65100b4b00 2023-11-24T08:05:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 -- 172.21.15.107:0/3002575822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 msgr2=0x7f651006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 --2- 172.21.15.107:0/3002575822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650802db90 comp rx=0 tx=0).stop 2023-11-24T08:05:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 -- 172.21.15.107:0/3002575822 shutdown_connections 2023-11-24T08:05:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 --2- 172.21.15.107:0/3002575822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f651006d3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 --2- 172.21.15.107:0/3002575822 >> v2:172.21.15.120:3300/0 conn(0x7f65100b2f40 0x7f65100b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 -- 172.21.15.107:0/3002575822 >> 172.21.15.107:0/3002575822 conn(0x7f65100b1c40 msgr2=0x7f65100b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 -- 172.21.15.107:0/3002575822 shutdown_connections 2023-11-24T08:05:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.548+0000 7f650ffff640 1 -- 172.21.15.107:0/3002575822 wait complete. 2023-11-24T08:05:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.550+0000 7f650ffff640 1 Processor -- start 2023-11-24T08:05:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.550+0000 7f650ffff640 1 -- start start 2023-11-24T08:05:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f65101e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 0x7f65101e2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65101e34a0 con 0x7f65100b2f40 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651015c5e0 con 0x7f6510143a00 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.551+0000 7f650ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651015c300 con 0x7f65100b4b00 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41712/0 (socket says 172.21.15.107:41712) 2023-11-24T08:05:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 -- 172.21.15.107:0/135821705 learned_addr learned my addr 172.21.15.107:0/135821705 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f6507fff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f65101e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650f7fe640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 0x7f65101e2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 -- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 msgr2=0x7f65101e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f65101e6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 -- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 msgr2=0x7f65101e2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 0x7f65101e2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650effd640 1 -- 172.21.15.107:0/135821705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65080094d0 con 0x7f65100b2f40 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f6507fff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f65101e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.552+0000 7f650f7fe640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 0x7f65101e2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650effd640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f650000c870 tx=0x7f650000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500003b70 con 0x7f65100b2f40 2023-11-24T08:05:50.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6500003cf0 con 0x7f65100b2f40 2023-11-24T08:05:50.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65101e6340 con 0x7f65100b2f40 2023-11-24T08:05:50.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650000aa90 con 0x7f65100b2f40 2023-11-24T08:05:50.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.553+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65101e6900 con 0x7f65100b2f40 2023-11-24T08:05:50.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.554+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651004e840 con 0x7f65100b2f40 2023-11-24T08:05:50.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.555+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f650000abf0 con 0x7f65100b2f40 2023-11-24T08:05:50.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.556+0000 7f650cff9640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 0x7f64f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.556+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6500089550 con 0x7f65100b2f40 2023-11-24T08:05:50.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.556+0000 7f6507fff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 0x7f64f006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.557+0000 7f6507fff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 0x7f64f006d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f65101e6250 tx=0x7f64f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.561+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6500054580 con 0x7f65100b2f40 2023-11-24T08:05:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.782+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f651006e990 con 0x7f65100b2f40 2023-11-24T08:05:50.784 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:05:50.783+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:05:50.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.786+0000 7f650cff9640 1 -- 172.21.15.107:0/135821705 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6500003e60 con 0x7f65100b2f40 2023-11-24T08:05:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:05:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.788+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 msgr2=0x7f64f006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.788+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 0x7f64f006d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f65101e6250 tx=0x7f64f8008040 comp rx=0 tx=0).stop 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.788+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 msgr2=0x7f651015bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.788+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f650000c870 tx=0x7f650000cd30 comp rx=0 tx=0).stop 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 shutdown_connections 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f006ae40 0x7f64f006d300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510143a00 0x7f65101e2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100b4b00 0x7f65101e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 --2- 172.21.15.107:0/135821705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100b2f40 0x7f651015bc70 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f650000c870 tx=0x7f650000cd30 comp rx=0 tx=0).stop 2023-11-24T08:05:50.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 >> 172.21.15.107:0/135821705 conn(0x7f65100b1c40 msgr2=0x7f65100cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:50.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 shutdown_connections 2023-11-24T08:05:50.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.789+0000 7f650ffff640 1 -- 172.21.15.107:0/135821705 wait complete. 2023-11-24T08:05:50.801 INFO:tasks.ceph.mds.c.smithi107.stderr: -7491> 2023-11-24T08:04:55.467+0000 7f3df411b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:05:50.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:05:50.854 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:05:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.988+0000 7fcd4492c640 1 Processor -- start 2023-11-24T08:05:50.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.989+0000 7fcd4492c640 1 -- start start 2023-11-24T08:05:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd4492c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400d4010 0x7fcd400cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd4492c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd400cfc80 0x7fcd400d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd4492c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400d0610 con 0x7fcd4015d580 2023-11-24T08:05:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd4492c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd400cf300 con 0x7fcd400cfc80 2023-11-24T08:05:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd4492c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400d1090 con 0x7fcd400d4010 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.990+0000 7fcd3ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd400cfc80 0x7fcd400d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd37fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400d4010 0x7fcd400cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd400cfc80 0x7fcd400d00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:45986/0 (socket says 172.21.15.107:45986) 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/1970926983 learned_addr learned my addr 172.21.15.107:0/1970926983 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400d4010 msgr2=0x7fcd400cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 --2- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400d4010 0x7fcd400cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/1970926983 >> v1:172.21.15.120:6789/0 conn(0x7fcd4015d580 legacy=0x7fcd400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.991+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/1970926983 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400d1370 con 0x7fcd400cfc80 2023-11-24T08:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.993+0000 7fcd3ed9d640 1 --2- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400cfc80 0x7fcd400d00d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e50 tx=0x7fcd3802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cced2e5d1b4c6093 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd3dd9b640 1 -- 172.21.15.107:0/1970926983 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3802cb00 con 0x7fcd400cfc80 2023-11-24T08:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd3dd9b640 1 -- 172.21.15.107:0/1970926983 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3802cc60 con 0x7fcd400cfc80 2023-11-24T08:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd3dd9b640 1 -- 172.21.15.107:0/1970926983 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd380343f0 con 0x7fcd400cfc80 2023-11-24T08:05:50.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 -- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400cfc80 msgr2=0x7fcd400d00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:50.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 --2- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400cfc80 0x7fcd400d00d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e50 tx=0x7fcd3802bf60 comp rx=0 tx=0).stop 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 -- 172.21.15.107:0/1970926983 shutdown_connections 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 --2- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400cfc80 0x7fcd400d00d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 --2- 172.21.15.107:0/1970926983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400d4010 0x7fcd400cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.994+0000 7fcd4492c640 1 -- 172.21.15.107:0/1970926983 >> 172.21.15.107:0/1970926983 conn(0x7fcd4009c770 msgr2=0x7fcd400d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.995+0000 7fcd4492c640 1 -- 172.21.15.107:0/1970926983 shutdown_connections 2023-11-24T08:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.995+0000 7fcd4492c640 1 -- 172.21.15.107:0/1970926983 wait complete. 2023-11-24T08:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.996+0000 7fcd4492c640 1 Processor -- start 2023-11-24T08:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.997+0000 7fcd4492c640 1 -- start start 2023-11-24T08:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.997+0000 7fcd4492c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400cfc80 0x7fcd40158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.997+0000 7fcd4492c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 0x7fcd400b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.997+0000 7fcd4492c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.997+0000 7fcd4492c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400b83f0 con 0x7fcd4015d580 2023-11-24T08:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd4492c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40158e50 con 0x7fcd400d4010 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd4492c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400b6960 con 0x7fcd400cfc80 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41734/0 (socket says 172.21.15.107:41734) 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/3652743487 learned_addr learned my addr 172.21.15.107:0/3652743487 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd37fff640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 0x7fcd400b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3e59c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400cfc80 0x7fcd40158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400cfc80 msgr2=0x7fcd40158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400cfc80 0x7fcd40158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 msgr2=0x7fcd400b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 0x7fcd400b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd3ed9d640 1 -- 172.21.15.107:0/3652743487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38009b30 con 0x7fcd4015d580 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.998+0000 7fcd37fff640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 0x7fcd400b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd3ed9d640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd38036040 tx=0x7fcd38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38009510 con 0x7fcd4015d580 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3803e450 con 0x7fcd4015d580 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3803e820 con 0x7fcd4015d580 2023-11-24T08:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd400b6c40 con 0x7fcd4015d580 2023-11-24T08:05:51.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:50.999+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd400c77b0 con 0x7fcd4015d580 2023-11-24T08:05:51.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.000+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd400c7610 con 0x7fcd4015d580 2023-11-24T08:05:51.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.001+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcd3803e980 con 0x7fcd4015d580 2023-11-24T08:05:51.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.002+0000 7fcd36ffd640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 0x7fcd2006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:51.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.002+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fcd380ac440 con 0x7fcd4015d580 2023-11-24T08:05:51.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.003+0000 7fcd3e59c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 0x7fcd2006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:51.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.003+0000 7fcd3e59c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 0x7fcd2006d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd300015a0 tx=0x7fcd30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.007+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd400c7610 con 0x7fcd4015d580 2023-11-24T08:05:51.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.216+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcd400c7610 con 0x7fcd4015d580 2023-11-24T08:05:51.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.789+0000 7fcd36ffd640 1 -- 172.21.15.107:0/3652743487 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcd400c7610 con 0x7fcd4015d580 2023-11-24T08:05:51.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 msgr2=0x7fcd2006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:51.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 0x7fcd2006d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd300015a0 tx=0x7fcd30009040 comp rx=0 tx=0).stop 2023-11-24T08:05:51.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 msgr2=0x7fcd400b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:51.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd38036040 tx=0x7fcd38004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:51.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 shutdown_connections 2023-11-24T08:05:51.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd2006ae70 0x7fcd2006d330 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4015d580 0x7fcd400b1120 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd400d4010 0x7fcd400b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 --2- 172.21.15.107:0/3652743487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd400cfc80 0x7fcd40158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 >> 172.21.15.107:0/3652743487 conn(0x7fcd4009c770 msgr2=0x7fcd400b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 shutdown_connections 2023-11-24T08:05:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:51.793+0000 7fcd4492c640 1 -- 172.21.15.107:0/3652743487 wait complete. 2023-11-24T08:05:51.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:05:52.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.025+0000 7fcec1e3d640 1 Processor -- start 2023-11-24T08:05:52.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.025+0000 7fcec1e3d640 1 -- start start 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcec1e3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcec1e3d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0cf450 0x7fcebc0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcec1e3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0cece0 con 0x7fcebc0cdfb0 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcec1e3d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc156010 con 0x7fcebc0cf450 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcec1e3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc06ce30 con 0x7fcebc15c7f0 2023-11-24T08:05:52.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcebaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0cf450 0x7fcebc0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcebaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0cf450 0x7fcebc0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41742/0 (socket says 172.21.15.107:41742) 2023-11-24T08:05:52.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.027+0000 7fcebaffd640 1 -- 172.21.15.107:0/2414718889 learned_addr learned my addr 172.21.15.107:0/2414718889 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:52.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebaffd640 1 -- 172.21.15.107:0/2414718889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 msgr2=0x7fcebc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebaffd640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebaffd640 1 -- 172.21.15.107:0/2414718889 >> v1:172.21.15.120:6789/0 conn(0x7fcebc0cdfb0 legacy=0x7fcebc0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebaffd640 1 -- 172.21.15.107:0/2414718889 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc06d040 con 0x7fcebc0cf450 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.028+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.029+0000 7fcebaffd640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc0cf8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009870 tx=0x7fcea402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b97cd4827efdb9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:52.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.029+0000 7fceba7fc640 1 -- 172.21.15.107:0/2414718889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4031640 con 0x7fcebc0cf450 2023-11-24T08:05:52.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.029+0000 7fceba7fc640 1 -- 172.21.15.107:0/2414718889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea40317a0 con 0x7fcebc0cf450 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.029+0000 7fceba7fc640 1 -- 172.21.15.107:0/2414718889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4031b30 con 0x7fcebc0cf450 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 -- 172.21.15.107:0/2414718889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 msgr2=0x7fcebc0cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc0cf8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009870 tx=0x7fcea402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 -- 172.21.15.107:0/2414718889 shutdown_connections 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc0cf8d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/2414718889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 -- 172.21.15.107:0/2414718889 >> 172.21.15.107:0/2414718889 conn(0x7fcebc06a920 msgr2=0x7fcebc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:52.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 -- 172.21.15.107:0/2414718889 shutdown_connections 2023-11-24T08:05:52.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.030+0000 7fcec1e3d640 1 -- 172.21.15.107:0/2414718889 wait complete. 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.032+0000 7fcec1e3d640 1 Processor -- start 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.032+0000 7fcec1e3d640 1 -- start start 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 0x7fcebc0af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc14bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0abfc0 con 0x7fcebc0cf450 2023-11-24T08:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0ac2a0 con 0x7fcebc0cdfb0 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcec1e3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0ac580 con 0x7fcebc15c7f0 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41752/0 (socket says 172.21.15.107:41752) 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 -- 172.21.15.107:0/3338914378 learned_addr learned my addr 172.21.15.107:0/3338914378 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 0x7fcebc0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 -- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 msgr2=0x7fcebc14bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:52.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebbfff640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc14bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc14bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 -- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 msgr2=0x7fcebc0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 0x7fcebc0af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebaffd640 1 -- 172.21.15.107:0/3338914378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40094d0 con 0x7fcebc0cf450 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 0x7fcebc0af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.033+0000 7fcebbfff640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fcebaffd640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcea4000c00 tx=0x7fcea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea403ac60 con 0x7fcebc0cf450 2023-11-24T08:05:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea403ade0 con 0x7fcebc0cf450 2023-11-24T08:05:52.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4044630 con 0x7fcebc0cf450 2023-11-24T08:05:52.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc0ac860 con 0x7fcebc0cf450 2023-11-24T08:05:52.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.034+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc0afc80 con 0x7fcebc0cf450 2023-11-24T08:05:52.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.035+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0acd80 con 0x7fcebc0cf450 2023-11-24T08:05:52.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.036+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcea402f050 con 0x7fcebc0cf450 2023-11-24T08:05:52.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.037+0000 7fceb3fff640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 0x7fce9c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:52.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.038+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fcea40ab8c0 con 0x7fcebc0cf450 2023-11-24T08:05:52.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.038+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 0x7fce9c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:52.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.038+0000 7fcebb7fe640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 0x7fce9c06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fceac007f40 tx=0x7fceac007490 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:52.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.042+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcebc0acd80 con 0x7fcebc0cf450 2023-11-24T08:05:52.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.217+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcebc0acd80 con 0x7fcebc0cf450 2023-11-24T08:05:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.891+0000 7fceb3fff640 1 -- 172.21.15.107:0/3338914378 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcebc0acd80 con 0x7fcebc0cf450 2023-11-24T08:05:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:05:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 msgr2=0x7fce9c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 0x7fce9c06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fceac007f40 tx=0x7fceac007490 comp rx=0 tx=0).stop 2023-11-24T08:05:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 msgr2=0x7fcebc158830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcea4000c00 tx=0x7fcea40046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 shutdown_connections 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06ae40 0x7fce9c06d300 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc15c7f0 0x7fcebc14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc158830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 --2- 172.21.15.107:0/3338914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0cdfb0 0x7fcebc0af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 >> 172.21.15.107:0/3338914378 conn(0x7fcebc06a920 msgr2=0x7fcebc0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.896+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 shutdown_connections 2023-11-24T08:05:52.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:52.897+0000 7fcec1e3d640 1 -- 172.21.15.107:0/3338914378 wait complete. 2023-11-24T08:05:52.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:05:53.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.130+0000 7fc8eaab4640 1 Processor -- start 2023-11-24T08:05:53.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.130+0000 7fc8eaab4640 1 -- start start 2023-11-24T08:05:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.132+0000 7fc8eaab4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 0x7fc8e406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.132+0000 7fc8eaab4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 0x7fc8e40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.132+0000 7fc8eaab4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e40cece0 con 0x7fc8e40cf450 2023-11-24T08:05:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.132+0000 7fc8eaab4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e4156010 con 0x7fc8e40ce6c0 2023-11-24T08:05:53.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.132+0000 7fc8eaab4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e406ce30 con 0x7fc8e415c7f0 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e9051640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 0x7fc8e40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e8850640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 0x7fc8e406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e9051640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 0x7fc8e40cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:46022/0 (socket says 172.21.15.107:46022) 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e9051640 1 -- 172.21.15.107:0/1377591898 learned_addr learned my addr 172.21.15.107:0/1377591898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e8850640 1 -- 172.21.15.107:0/1377591898 >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 msgr2=0x7fc8e40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e8850640 1 --2- 172.21.15.107:0/1377591898 >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 0x7fc8e40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e8850640 1 -- 172.21.15.107:0/1377591898 >> v1:172.21.15.120:6789/0 conn(0x7fc8e40cf450 legacy=0x7fc8e40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.133+0000 7fc8e8850640 1 -- 172.21.15.107:0/1377591898 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e406d040 con 0x7fc8e415c7f0 2023-11-24T08:05:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.135+0000 7fc8e8850640 1 --2- 172.21.15.107:0/1377591898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 0x7fc8e406db20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80097a0 tx=0x7fc8d802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae309895382319ef server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.135+0000 7fc8e37fe640 1 -- 172.21.15.107:0/1377591898 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8031830 con 0x7fc8e415c7f0 2023-11-24T08:05:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.135+0000 7fc8e37fe640 1 -- 172.21.15.107:0/1377591898 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d8031990 con 0x7fc8e415c7f0 2023-11-24T08:05:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.135+0000 7fc8e37fe640 1 -- 172.21.15.107:0/1377591898 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8031d20 con 0x7fc8e415c7f0 2023-11-24T08:05:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 -- 172.21.15.107:0/1377591898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 msgr2=0x7fc8e406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/1377591898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 0x7fc8e406db20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80097a0 tx=0x7fc8d802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 -- 172.21.15.107:0/1377591898 shutdown_connections 2023-11-24T08:05:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/1377591898 >> v2:172.21.15.120:3300/0 conn(0x7fc8e40ce6c0 0x7fc8e40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/1377591898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e415c7f0 0x7fc8e406db20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 -- 172.21.15.107:0/1377591898 >> 172.21.15.107:0/1377591898 conn(0x7fc8e406a920 msgr2=0x7fc8e406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 -- 172.21.15.107:0/1377591898 shutdown_connections 2023-11-24T08:05:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.136+0000 7fc8eaab4640 1 -- 172.21.15.107:0/1377591898 wait complete. 2023-11-24T08:05:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.138+0000 7fc8eaab4640 1 Processor -- start 2023-11-24T08:05:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.138+0000 7fc8eaab4640 1 -- start start 2023-11-24T08:05:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8e40ce6c0 0x7fc8e4155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e40cf450 0x7fc8e415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40c8360 con 0x7fc8e415c7f0 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40b3e70 con 0x7fc8e40ce6c0 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8eaab4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40c8570 con 0x7fc8e40cf450 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41764/0 (socket says 172.21.15.107:41764) 2023-11-24T08:05:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 -- 172.21.15.107:0/2394467071 learned_addr learned my addr 172.21.15.107:0/2394467071 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e3fff640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e40cf450 0x7fc8e415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e8850640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8e40ce6c0 0x7fc8e4155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 -- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e40cf450 msgr2=0x7fc8e415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e40cf450 0x7fc8e415a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 -- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8e40ce6c0 msgr2=0x7fc8e4155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8e40ce6c0 0x7fc8e4155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.139+0000 7fc8e9051640 1 -- 172.21.15.107:0/2394467071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8009400 con 0x7fc8e415c7f0 2023-11-24T08:05:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8e9051640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00efc0 tx=0x7fc8dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00cd20 con 0x7fc8e415c7f0 2023-11-24T08:05:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc00f040 con 0x7fc8e415c7f0 2023-11-24T08:05:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e40c8850 con 0x7fc8e415c7f0 2023-11-24T08:05:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc003bb0 con 0x7fc8e415c7f0 2023-11-24T08:05:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.140+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e4155a50 con 0x7fc8e415c7f0 2023-11-24T08:05:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.141+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e40c8d70 con 0x7fc8e415c7f0 2023-11-24T08:05:53.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.142+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc8dc003d10 con 0x7fc8e415c7f0 2023-11-24T08:05:53.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.143+0000 7fc8e17fa640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:53.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.143+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fc8dc089ac0 con 0x7fc8e415c7f0 2023-11-24T08:05:53.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.144+0000 7fc8e8850640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:53.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.144+0000 7fc8e8850640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 0x7fc8c806d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8036000 tx=0x7fc8d803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:53.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.148+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8e40c8d70 con 0x7fc8e415c7f0 2023-11-24T08:05:53.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.336+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc8e40c8d70 con 0x7fc8e415c7f0 2023-11-24T08:05:53.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.903+0000 7fc8e17fa640 1 -- 172.21.15.107:0/2394467071 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc8e40c8d70 con 0x7fc8e415c7f0 2023-11-24T08:05:53.904 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 msgr2=0x7fc8c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 0x7fc8c806d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8036000 tx=0x7fc8d803c040 comp rx=0 tx=0).stop 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 msgr2=0x7fc8e415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00efc0 tx=0x7fc8dc00c490 comp rx=0 tx=0).stop 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 shutdown_connections 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e415c7f0 0x7fc8e415ac60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8e40cf450 0x7fc8e415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 --2- 172.21.15.107:0/2394467071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8e40ce6c0 0x7fc8e4155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:53.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 >> 172.21.15.107:0/2394467071 conn(0x7fc8e406a920 msgr2=0x7fc8e40afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:53.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 shutdown_connections 2023-11-24T08:05:53.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:53.907+0000 7fc8eaab4640 1 -- 172.21.15.107:0/2394467071 wait complete. 2023-11-24T08:05:53.933 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:05:53.933 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:05:53.933 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:05:53.933 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:05:53.934 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:05:53.934 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:05:53.934 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:05:53.934 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:05:53.934 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:05:53.935 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:05:53.935 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:05:53.935 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:05:53.935 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:05:53.935 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:05:53.935 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:05:53.936 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:05:53.952 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:05:53.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:05:53.954 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:05:53.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:05:53.957 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:05:53.958 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:05:53.959 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:05:53.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:05:54.001 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:05:54.002 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:54.006 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:05:54.006 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:54.009 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:05:54.009 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:54.013 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:05:54.013 DEBUG:teuthology.parallel:result is None 2023-11-24T08:05:54.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:05:54.170 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:05:54.178 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:05:54.188 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:05:54.191 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:05:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.214+0000 7f59479c5640 1 Processor -- start 2023-11-24T08:05:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.214+0000 7f59479c5640 1 -- start start 2023-11-24T08:05:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.215+0000 7f59479c5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f594015b440 0x7f59400cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.215+0000 7f59479c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400c59b0 0x7f594006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.215+0000 7f59479c5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594015eea0 con 0x7f59400cbf20 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.215+0000 7f59479c5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400c5e00 con 0x7f594015b440 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.215+0000 7f59479c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59400cc3d0 con 0x7f59400c59b0 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f594015b440 0x7f59400cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f594015b440 0x7f59400cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41812/0 (socket says 172.21.15.107:41812) 2023-11-24T08:05:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 -- 172.21.15.107:0/3881557266 learned_addr learned my addr 172.21.15.107:0/3881557266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945f62640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400c59b0 0x7f594006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 -- 172.21.15.107:0/3881557266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400c59b0 msgr2=0x7f594006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400c59b0 0x7f594006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 -- 172.21.15.107:0/3881557266 >> v1:172.21.15.120:6789/0 conn(0x7f59400cbf20 legacy=0x7f59400c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5937fff640 1 -- 172.21.15.107:0/3881557266 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593483140 0 0) 0x7f594015eea0 con 0x7f59400cbf20 2023-11-24T08:05:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.216+0000 7f5945761640 1 -- 172.21.15.107:0/3881557266 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400abf30 con 0x7f594015b440 2023-11-24T08:05:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.217+0000 7f5945761640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f59400cb9e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f593c009870 tx=0x7f593c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b218d3fc2fb5474 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.217+0000 7f5937fff640 1 -- 172.21.15.107:0/3881557266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c031640 con 0x7f594015b440 2023-11-24T08:05:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.217+0000 7f5937fff640 1 -- 172.21.15.107:0/3881557266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c0317a0 con 0x7f594015b440 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.217+0000 7f5937fff640 1 -- 172.21.15.107:0/3881557266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c031b30 con 0x7f594015b440 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 -- 172.21.15.107:0/3881557266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 msgr2=0x7f59400cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f59400cb9e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f593c009870 tx=0x7f593c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 -- 172.21.15.107:0/3881557266 shutdown_connections 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400c59b0 0x7f594006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 --2- 172.21.15.107:0/3881557266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f59400cb9e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 -- 172.21.15.107:0/3881557266 >> 172.21.15.107:0/3881557266 conn(0x7f59400b0c50 msgr2=0x7f59400b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 -- 172.21.15.107:0/3881557266 shutdown_connections 2023-11-24T08:05:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.218+0000 7f59479c5640 1 -- 172.21.15.107:0/3881557266 wait complete. 2023-11-24T08:05:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.220+0000 7f59479c5640 1 Processor -- start 2023-11-24T08:05:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.220+0000 7f59479c5640 1 -- start start 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 0x7f5940156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400cbf20 0x7f594009a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59400c70a0 con 0x7f594015b440 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59400c7380 con 0x7f59400c59b0 2023-11-24T08:05:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f59479c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59400c7660 con 0x7f59400cbf20 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f5945f62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f5945f62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41820/0 (socket says 172.21.15.107:41820) 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.221+0000 7f5945f62640 1 -- 172.21.15.107:0/1132021155 learned_addr learned my addr 172.21.15.107:0/1132021155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945761640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 0x7f5940156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5944f60640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400cbf20 0x7f594009a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 -- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400cbf20 msgr2=0x7f594009a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400cbf20 0x7f594009a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 -- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 msgr2=0x7f5940156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 0x7f5940156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 -- 172.21.15.107:0/1132021155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0094d0 con 0x7f594015b440 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945761640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 0x7f5940156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5945f62640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f593800efc0 tx=0x7f593800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938014480 con 0x7f594015b440 2023-11-24T08:05:54.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593800f040 con 0x7f594015b440 2023-11-24T08:05:54.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938014820 con 0x7f594015b440 2023-11-24T08:05:54.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.222+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59400c7940 con 0x7f594015b440 2023-11-24T08:05:54.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.223+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5940099c40 con 0x7f594015b440 2023-11-24T08:05:54.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.224+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5940156e60 con 0x7f594015b440 2023-11-24T08:05:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.225+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5938014980 con 0x7f594015b440 2023-11-24T08:05:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.226+0000 7f5935ffb640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 0x7f592406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.226+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5938089d40 con 0x7f594015b440 2023-11-24T08:05:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.226+0000 7f5945761640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 0x7f592406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.227+0000 7f5945761640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 0x7f592406d350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:54.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.230+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5938055260 con 0x7f594015b440 2023-11-24T08:05:54.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:54.432+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f594004e8b0 con 0x7f594015b440 2023-11-24T08:05:55.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.758+0000 7f5935ffb640 1 -- 172.21.15.107:0/1132021155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5938058880 con 0x7f594015b440 2023-11-24T08:05:55.761 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.762+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 msgr2=0x7f592406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.762+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 0x7f592406d350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c008030 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.762+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 msgr2=0x7f594009bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.762+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f593800efc0 tx=0x7f593800c490 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 shutdown_connections 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f592406ae90 0x7f592406d350 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f594015b440 0x7f594009bfe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59400cbf20 0x7f594009a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 --2- 172.21.15.107:0/1132021155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59400c59b0 0x7f5940156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 >> 172.21.15.107:0/1132021155 conn(0x7f59400b0c50 msgr2=0x7f594006c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:55.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 shutdown_connections 2023-11-24T08:05:55.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.763+0000 7f59479c5640 1 -- 172.21.15.107:0/1132021155 wait complete. 2023-11-24T08:05:55.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:05:55.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.990+0000 7f1f8eb83640 1 Processor -- start 2023-11-24T08:05:55.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.990+0000 7f1f8eb83640 1 -- start start 2023-11-24T08:05:55.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f8814b000 0x7f1f8806c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:55.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f88161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:55.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f881624e0 0x7f1f880b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8806d310 con 0x7f1f881624e0 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f88165440 con 0x7f1f8814b000 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.992+0000 7f1f8eb83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f88162cf0 con 0x7f1f8806cee0 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f881624e0 0x7f1f880b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f881624e0 0x7f1f880b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41832/0 (socket says 172.21.15.107:41832) 2023-11-24T08:05:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 -- 172.21.15.107:0/1597726949 learned_addr learned my addr 172.21.15.107:0/1597726949 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8d380640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f88161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8db81640 1 --2- 172.21.15.107:0/1597726949 >> v2:172.21.15.120:3300/0 conn(0x7f1f8814b000 0x7f1f8806c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 -- 172.21.15.107:0/1597726949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 msgr2=0x7f1f88161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f88161fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 -- 172.21.15.107:0/1597726949 >> v2:172.21.15.120:3300/0 conn(0x7f1f8814b000 msgr2=0x7f1f8806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 --2- 172.21.15.107:0/1597726949 >> v2:172.21.15.120:3300/0 conn(0x7f1f8814b000 0x7f1f8806c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.993+0000 7f1f8e382640 1 -- 172.21.15.107:0/1597726949 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f88165700 con 0x7f1f881624e0 2023-11-24T08:05:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.994+0000 7f1f8e382640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f880b0bc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1f8806e6d0 tx=0x7f1f8002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d7b01fcd7c140b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:55.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8cb7f640 1 -- 172.21.15.107:0/1597726949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8002ca70 con 0x7f1f881624e0 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8cb7f640 1 -- 172.21.15.107:0/1597726949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8002cbd0 con 0x7f1f881624e0 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8cb7f640 1 -- 172.21.15.107:0/1597726949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f800343f0 con 0x7f1f881624e0 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 -- 172.21.15.107:0/1597726949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 msgr2=0x7f1f880b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f880b0bc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1f8806e6d0 tx=0x7f1f8002bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 -- 172.21.15.107:0/1597726949 shutdown_connections 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f880b0bc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/1597726949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f88161fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.995+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/1597726949 >> v2:172.21.15.120:3300/0 conn(0x7f1f8814b000 0x7f1f8806c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.996+0000 7f1f8eb83640 1 -- 172.21.15.107:0/1597726949 >> 172.21.15.107:0/1597726949 conn(0x7f1f880b4630 msgr2=0x7f1f880b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.996+0000 7f1f8eb83640 1 -- 172.21.15.107:0/1597726949 shutdown_connections 2023-11-24T08:05:55.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.996+0000 7f1f8eb83640 1 -- 172.21.15.107:0/1597726949 wait complete. 2023-11-24T08:05:55.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.997+0000 7f1f8eb83640 1 Processor -- start 2023-11-24T08:05:55.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.997+0000 7f1f8eb83640 1 -- start start 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f881410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 0x7f1f88146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881415e0 con 0x7f1f881624e0 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f88145110 con 0x7f1f8814b000 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.998+0000 7f1f8eb83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881453f0 con 0x7f1f8806cee0 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41844/0 (socket says 172.21.15.107:41844) 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 -- 172.21.15.107:0/3221994092 learned_addr learned my addr 172.21.15.107:0/3221994092 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8db81640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f881410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8d380640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 0x7f1f88146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 -- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 msgr2=0x7f1f881410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f881410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 -- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 msgr2=0x7f1f88146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 0x7f1f88146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 -- 172.21.15.107:0/3221994092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80009b30 con 0x7f1f881624e0 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8d380640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 0x7f1f88146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:55.999+0000 7f1f8e382640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036040 tx=0x7f1f80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.000+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80009510 con 0x7f1f881624e0 2023-11-24T08:05:56.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.000+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8003e450 con 0x7f1f881624e0 2023-11-24T08:05:56.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.000+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8003e800 con 0x7f1f881624e0 2023-11-24T08:05:56.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.000+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f881456d0 con 0x7f1f881624e0 2023-11-24T08:05:56.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.000+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f88145c90 con 0x7f1f881624e0 2023-11-24T08:05:56.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.001+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8804e840 con 0x7f1f881624e0 2023-11-24T08:05:56.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.002+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1f8003e960 con 0x7f1f881624e0 2023-11-24T08:05:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.003+0000 7f1f7e7fc640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 0x7f1f6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.003+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1f800ac1a0 con 0x7f1f881624e0 2023-11-24T08:05:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.004+0000 7f1f8db81640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 0x7f1f6c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.004+0000 7f1f8db81640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 0x7f1f6c06d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:56.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.008+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f80077470 con 0x7f1f881624e0 2023-11-24T08:05:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.266+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1f88144d50 con 0x7f1f881624e0 2023-11-24T08:05:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.267+0000 7f1f7e7fc640 1 -- 172.21.15.107:0/3221994092 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f1f8007aa90 con 0x7f1f881624e0 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.270 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.271 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:05:56.271 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:05:56.271 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.272 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:05:56.272 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:05:56.272 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.272 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.273 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:05:56.273 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:05:56.273 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.273 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.274 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:05:56.274 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:05:56.274 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.275 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.275 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:05:56.275 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:05:56.275 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.276 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:05:56.276 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:05:56.277 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.277 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.277 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:05:56.277 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:05:56.277 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.278 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:05:56.278 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:05:56.280 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.283 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:05:56.286 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.287 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.288 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:05:56.290 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:05:56.291 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-mds", 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBlpLqgMhAAEV5Db6EYQ1Tk5RY2jYbYdQ==", 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-mds" 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-mgr", 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBl99igMhAAfFHUOMFHAlU1GQkC+QuEWg==", 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-osd", 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBlH/GgMhAACHUvfDo9PpzWs7+ipp8Nbw==", 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-osd" 2023-11-24T08:05:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-rbd", 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBlowmhMhAAi3Mtbnj7C9Dug8qRFpSG+g==", 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBlGyKhMhAASf5xTRVR56EXNpmaWio+7g==", 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.bootstrap-rgw", 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQANWWBlhTqhMhAAONqOySBcDReh/FEZDM78nQ==", 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-24T08:05:56.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:05:56.296 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:05:56.297 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:05:56.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 msgr2=0x7f1f6c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 0x7f1f6c06d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84008040 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 msgr2=0x7f1f88160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036040 tx=0x7f1f80004a40 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 shutdown_connections 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f6c06ae40 0x7f1f6c06d300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f881624e0 0x7f1f88160340 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f8814b000 0x7f1f88146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 --2- 172.21.15.107:0/3221994092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f8806cee0 0x7f1f881410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.270+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 >> 172.21.15.107:0/3221994092 conn(0x7f1f880b4630 msgr2=0x7f1f8814a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.271+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 shutdown_connections 2023-11-24T08:05:56.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.271+0000 7f1f8eb83640 1 -- 172.21.15.107:0/3221994092 wait complete. 2023-11-24T08:05:56.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-24T08:05:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.515+0000 7f060da64640 1 Processor -- start 2023-11-24T08:05:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.515+0000 7f060da64640 1 -- start start 2023-11-24T08:05:56.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.516+0000 7f060da64640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 0x7f0608161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.516+0000 7f060da64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06080cf450 0x7f06080cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.516+0000 7f060da64640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f060815dc80 0x7f060815e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.516+0000 7f060da64640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080cfe10 con 0x7f060815dc80 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f060da64640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080ce0b0 con 0x7f0608157c70 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f060da64640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0608161960 con 0x7f06080cf450 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f060815dc80 0x7f060815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f060815dc80 0x7f060815e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41856/0 (socket says 172.21.15.107:41856) 2023-11-24T08:05:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 -- 172.21.15.107:0/640654678 learned_addr learned my addr 172.21.15.107:0/640654678 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0606ffd640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06080cf450 0x7f06080cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f06077fe640 1 --2- 172.21.15.107:0/640654678 >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 0x7f0608161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 -- 172.21.15.107:0/640654678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06080cf450 msgr2=0x7f06080cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06080cf450 0x7f06080cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 -- 172.21.15.107:0/640654678 >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 msgr2=0x7f0608161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 --2- 172.21.15.107:0/640654678 >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 0x7f0608161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f0607fff640 1 -- 172.21.15.107:0/640654678 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060815e650 con 0x7f060815dc80 2023-11-24T08:05:56.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.517+0000 7f06077fe640 1 --2- 172.21.15.107:0/640654678 >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 0x7f0608161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.519+0000 7f0607fff640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f060815e110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f060806d5e0 tx=0x7f060002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13a89350f247ae4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.519+0000 7f06067fc640 1 -- 172.21.15.107:0/640654678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06000317e0 con 0x7f060815dc80 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.519+0000 7f06067fc640 1 -- 172.21.15.107:0/640654678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0600031940 con 0x7f060815dc80 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.519+0000 7f06067fc640 1 -- 172.21.15.107:0/640654678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600031cd0 con 0x7f060815dc80 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 -- 172.21.15.107:0/640654678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 msgr2=0x7f060815e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f060815e110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f060806d5e0 tx=0x7f060002b880 comp rx=0 tx=0).stop 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 -- 172.21.15.107:0/640654678 shutdown_connections 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f060815e110 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 --2- 172.21.15.107:0/640654678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06080cf450 0x7f06080cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 --2- 172.21.15.107:0/640654678 >> v2:172.21.15.120:3300/0 conn(0x7f0608157c70 0x7f0608161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 -- 172.21.15.107:0/640654678 >> 172.21.15.107:0/640654678 conn(0x7f060806a920 msgr2=0x7f060806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 -- 172.21.15.107:0/640654678 shutdown_connections 2023-11-24T08:05:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.520+0000 7f060da64640 1 -- 172.21.15.107:0/640654678 wait complete. 2023-11-24T08:05:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.522+0000 7f060da64640 1 Processor -- start 2023-11-24T08:05:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.522+0000 7f060da64640 1 -- start start 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 0x7f060809cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0608157c70 0x7f06081473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080aceb0 con 0x7f060815dc80 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060809d010 con 0x7f06080cf450 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f060da64640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080abfc0 con 0x7f0608157c70 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f0607fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f0607fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41862/0 (socket says 172.21.15.107:41862) 2023-11-24T08:05:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f0607fff640 1 -- 172.21.15.107:0/1305495379 learned_addr learned my addr 172.21.15.107:0/1305495379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f0606ffd640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0608157c70 0x7f06081473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.523+0000 7f06077fe640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 0x7f060809cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 -- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0608157c70 msgr2=0x7f06081473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0608157c70 0x7f06081473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 -- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 msgr2=0x7f060809cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 0x7f060809cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 -- 172.21.15.107:0/1305495379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000094d0 con 0x7f060815dc80 2023-11-24T08:05:56.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f06077fe640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 0x7f060809cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f0607fff640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f06000090b0 tx=0x7f06000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600032880 con 0x7f060815dc80 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06000329e0 con 0x7f060815dc80 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600032d90 con 0x7f060815dc80 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.524+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06080ac2a0 con 0x7f060815dc80 2023-11-24T08:05:56.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.525+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06080ac860 con 0x7f060815dc80 2023-11-24T08:05:56.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.526+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06080cdd60 con 0x7f060815dc80 2023-11-24T08:05:56.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.527+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f060004d3f0 con 0x7f060815dc80 2023-11-24T08:05:56.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.527+0000 7f05fffff640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 0x7f05e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:56.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.528+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f06000ab020 con 0x7f060815dc80 2023-11-24T08:05:56.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.528+0000 7f06077fe640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 0x7f05e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:56.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.528+0000 7f06077fe640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 0x7f05e806d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05f8007f70 tx=0x7f05f80074c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:56.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.532+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06000764d0 con 0x7f060815dc80 2023-11-24T08:05:56.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.789+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f0608071ab0 con 0x7f060815dc80 2023-11-24T08:05:56.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.794+0000 7f05fffff640 1 -- 172.21.15.107:0/1305495379 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0600079af0 con 0x7f060815dc80 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 msgr2=0x7f05e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 0x7f05e806d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05f8007f70 tx=0x7f05f80074c0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 msgr2=0x7f0608147ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f06000090b0 tx=0x7f06000046c0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 shutdown_connections 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05e806ae40 0x7f05e806d300 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f060815dc80 0x7f0608147ea0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0608157c70 0x7f06081473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 --2- 172.21.15.107:0/1305495379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06080cf450 0x7f060809cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:56.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 >> 172.21.15.107:0/1305495379 conn(0x7f060806a920 msgr2=0x7f06080b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:56.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 shutdown_connections 2023-11-24T08:05:56.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:56.797+0000 7f060da64640 1 -- 172.21.15.107:0/1305495379 wait complete. 2023-11-24T08:05:56.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-24T08:05:57.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.022+0000 7fa3eff68640 1 Processor -- start 2023-11-24T08:05:57.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.023+0000 7fa3eff68640 1 -- start start 2023-11-24T08:05:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3eff68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3e815c7f0 0x7fa3e806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3eff68640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa3e80ce6c0 0x7fa3e80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3eff68640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80cece0 con 0x7fa3e80cf450 2023-11-24T08:05:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3eff68640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e80cfe40 con 0x7fa3e815c7f0 2023-11-24T08:05:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3eff68640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e80ceaf0 con 0x7fa3e80ce6c0 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.024+0000 7fa3edd04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3e815c7f0 0x7fa3e806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3ed503640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3e80cf450 0x7fa3e80cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48664/0 (socket says 172.21.15.107:48664) 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3ed503640 1 -- 172.21.15.107:0/4292368282 learned_addr learned my addr 172.21.15.107:0/4292368282 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3edd04640 1 -- 172.21.15.107:0/4292368282 >> v2:172.21.15.120:3300/0 conn(0x7fa3e80ce6c0 msgr2=0x7fa3e80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3edd04640 1 --2- 172.21.15.107:0/4292368282 >> v2:172.21.15.120:3300/0 conn(0x7fa3e80ce6c0 0x7fa3e80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3edd04640 1 -- 172.21.15.107:0/4292368282 >> v1:172.21.15.107:6789/0 conn(0x7fa3e80cf450 legacy=0x7fa3e80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3edd04640 1 -- 172.21.15.107:0/4292368282 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e806d040 con 0x7fa3e815c7f0 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.025+0000 7fa3ecd02640 1 -- 172.21.15.107:0/4292368282 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578485271 0 0) 0x7fa3e80cece0 con 0x7fa3e80cf450 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.026+0000 7fa3edd04640 1 --2- 172.21.15.107:0/4292368282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e806db20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80d00e0 tx=0x7fa3d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209fc4ea2ff34e67 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.026+0000 7fa3ecd02640 1 -- 172.21.15.107:0/4292368282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d802cc80 con 0x7fa3e815c7f0 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.026+0000 7fa3ecd02640 1 -- 172.21.15.107:0/4292368282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d802cde0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.026+0000 7fa3ecd02640 1 -- 172.21.15.107:0/4292368282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8034660 con 0x7fa3e815c7f0 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 -- 172.21.15.107:0/4292368282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 msgr2=0x7fa3e806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 --2- 172.21.15.107:0/4292368282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e806db20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80d00e0 tx=0x7fa3d802bed0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 -- 172.21.15.107:0/4292368282 shutdown_connections 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 --2- 172.21.15.107:0/4292368282 >> v2:172.21.15.120:3300/0 conn(0x7fa3e80ce6c0 0x7fa3e80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 --2- 172.21.15.107:0/4292368282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e806db20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 -- 172.21.15.107:0/4292368282 >> 172.21.15.107:0/4292368282 conn(0x7fa3e806a920 msgr2=0x7fa3e806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 -- 172.21.15.107:0/4292368282 shutdown_connections 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.027+0000 7fa3eff68640 1 -- 172.21.15.107:0/4292368282 wait complete. 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.029+0000 7fa3eff68640 1 Processor -- start 2023-11-24T08:05:57.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.029+0000 7fa3eff68640 1 -- start start 2023-11-24T08:05:57.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e80ce6c0 0x7fa3e815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 0x7fa3e80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80cbee0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80ad010 con 0x7fa3e80cf450 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3eff68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80ad260 con 0x7fa3e80ce6c0 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3ee505640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3ee505640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41890/0 (socket says 172.21.15.107:41890) 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.030+0000 7fa3ee505640 1 -- 172.21.15.107:0/1311395439 learned_addr learned my addr 172.21.15.107:0/1311395439 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ed503640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 0x7fa3e80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3edd04640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e80ce6c0 0x7fa3e815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 -- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e80ce6c0 msgr2=0x7fa3e815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e80ce6c0 0x7fa3e815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 -- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 msgr2=0x7fa3e80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 0x7fa3e80cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 -- 172.21.15.107:0/1311395439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8009a70 con 0x7fa3e815c7f0 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ed503640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 0x7fa3e80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3ee505640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000b3e0 tx=0x7fa3e0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.031+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0011830 con 0x7fa3e815c7f0 2023-11-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.032+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e0011990 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.032+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e80ad540 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.032+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0011d20 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.032+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80adb00 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.033+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e80adcc0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.034+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa3e001a400 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.035+0000 7fa3e67fc640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 0x7fa3d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.035+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa3e00893f0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.035+0000 7fa3edd04640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 0x7fa3d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.035+0000 7fa3edd04640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 0x7fa3d406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8036000 tx=0x7fa3d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.039+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3e00547d0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.296+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fa3e80cece0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.300+0000 7fa3e67fc640 1 -- 172.21.15.107:0/1311395439 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa3e00150a0 con 0x7fa3e815c7f0 2023-11-24T08:05:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 msgr2=0x7fa3d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 0x7fa3d406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8036000 tx=0x7fa3d8030000 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 msgr2=0x7fa3e815adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000b3e0 tx=0x7fa3e0009f90 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 shutdown_connections 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.302+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3d406ae70 0x7fa3d406d330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e815c7f0 0x7fa3e815adf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e80cf450 0x7fa3e80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 --2- 172.21.15.107:0/1311395439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e80ce6c0 0x7fa3e815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 >> 172.21.15.107:0/1311395439 conn(0x7fa3e806a920 msgr2=0x7fa3e8071890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 shutdown_connections 2023-11-24T08:05:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.303+0000 7fa3eff68640 1 -- 172.21.15.107:0/1311395439 wait complete. 2023-11-24T08:05:57.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-24T08:05:57.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.541+0000 7fa78102b640 1 Processor -- start 2023-11-24T08:05:57.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.541+0000 7fa78102b640 1 -- start start 2023-11-24T08:05:57.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa78102b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa77c157c70 0x7fa77c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa78102b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 0x7fa77c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa78102b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0ce0b0 con 0x7fa77c15dc80 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa78102b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0cfe40 con 0x7fa77c157c70 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa78102b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c161960 con 0x7fa77c0cf450 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa77ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa77c157c70 0x7fa77c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa77b59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa77c15dc80 0x7fa77c15e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48670/0 (socket says 172.21.15.107:48670) 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.542+0000 7fa77b59e640 1 -- 172.21.15.107:0/2136626761 learned_addr learned my addr 172.21.15.107:0/2136626761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa77ad9d640 1 -- 172.21.15.107:0/2136626761 >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 msgr2=0x7fa77c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa773fff640 1 --2- 172.21.15.107:0/2136626761 >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 0x7fa77c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa77ad9d640 1 --2- 172.21.15.107:0/2136626761 >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 0x7fa77c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa77ad9d640 1 -- 172.21.15.107:0/2136626761 >> v1:172.21.15.107:6789/0 conn(0x7fa77c15dc80 legacy=0x7fa77c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa77ad9d640 1 -- 172.21.15.107:0/2136626761 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c15e8d0 con 0x7fa77c157c70 2023-11-24T08:05:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa773fff640 1 --2- 172.21.15.107:0/2136626761 >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 0x7fa77c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.543+0000 7fa77a59c640 1 -- 172.21.15.107:0/2136626761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588103569 0 0) 0x7fa77c0ce0b0 con 0x7fa77c15dc80 2023-11-24T08:05:57.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.544+0000 7fa77ad9d640 1 --2- 172.21.15.107:0/2136626761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c161df0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0b44b0 tx=0x7fa76c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53190bead6580ec4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.544+0000 7fa77a59c640 1 -- 172.21.15.107:0/2136626761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c02e070 con 0x7fa77c157c70 2023-11-24T08:05:57.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.544+0000 7fa77a59c640 1 -- 172.21.15.107:0/2136626761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c0315a0 con 0x7fa77c157c70 2023-11-24T08:05:57.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.544+0000 7fa77a59c640 1 -- 172.21.15.107:0/2136626761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c031930 con 0x7fa77c157c70 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 -- 172.21.15.107:0/2136626761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 msgr2=0x7fa77c161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 --2- 172.21.15.107:0/2136626761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c161df0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0b44b0 tx=0x7fa76c02b840 comp rx=0 tx=0).stop 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 -- 172.21.15.107:0/2136626761 shutdown_connections 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 --2- 172.21.15.107:0/2136626761 >> v2:172.21.15.120:3300/0 conn(0x7fa77c0cf450 0x7fa77c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 --2- 172.21.15.107:0/2136626761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c161df0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 -- 172.21.15.107:0/2136626761 >> 172.21.15.107:0/2136626761 conn(0x7fa77c06a920 msgr2=0x7fa77c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 -- 172.21.15.107:0/2136626761 shutdown_connections 2023-11-24T08:05:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.545+0000 7fa78102b640 1 -- 172.21.15.107:0/2136626761 wait complete. 2023-11-24T08:05:57.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.547+0000 7fa78102b640 1 Processor -- start 2023-11-24T08:05:57.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.547+0000 7fa78102b640 1 -- start start 2023-11-24T08:05:57.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77c0cf450 0x7fa77c14a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa77c15dc80 0x7fa77c0ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c14aa30 con 0x7fa77c157c70 2023-11-24T08:05:57.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c09bb10 con 0x7fa77c15dc80 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa78102b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c099140 con 0x7fa77c0cf450 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41900/0 (socket says 172.21.15.107:41900) 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 -- 172.21.15.107:0/1813070440 learned_addr learned my addr 172.21.15.107:0/1813070440 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 -- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77c0cf450 msgr2=0x7fa77c14a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77c0cf450 0x7fa77c14a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 -- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa77c15dc80 msgr2=0x7fa77c0ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa77c15dc80 0x7fa77c0ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.548+0000 7fa773fff640 1 -- 172.21.15.107:0/1813070440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009400 con 0x7fa77c157c70 2023-11-24T08:05:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa773fff640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c820 tx=0x7fa76400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764007a30 con 0x7fa77c157c70 2023-11-24T08:05:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa764007b90 con 0x7fa77c157c70 2023-11-24T08:05:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c1f3e50 con 0x7fa77c157c70 2023-11-24T08:05:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76400ce10 con 0x7fa77c157c70 2023-11-24T08:05:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.549+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c099670 con 0x7fa77c157c70 2023-11-24T08:05:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.550+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c067470 con 0x7fa77c157c70 2023-11-24T08:05:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.551+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa7640113f0 con 0x7fa77c157c70 2023-11-24T08:05:57.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.552+0000 7fa7737fe640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 0x7fa75c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:57.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.552+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa7640893b0 con 0x7fa77c157c70 2023-11-24T08:05:57.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.552+0000 7fa77ad9d640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 0x7fa75c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:57.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.553+0000 7fa77ad9d640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 0x7fa75c06d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa76c036000 tx=0x7fa76c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:57.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.556+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7640547b0 con 0x7fa77c157c70 2023-11-24T08:05:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.813+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fa77c0cd910 con 0x7fa77c157c70 2023-11-24T08:05:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.818+0000 7fa7737fe640 1 -- 172.21.15.107:0/1813070440 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa764057dd0 con 0x7fa77c157c70 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.820+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 msgr2=0x7fa75c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.820+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 0x7fa75c06d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa76c036000 tx=0x7fa76c030000 comp rx=0 tx=0).stop 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.820+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 msgr2=0x7fa77c0ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.820+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c820 tx=0x7fa76400cce0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 shutdown_connections 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75c06ae40 0x7fa75c06d300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa77c15dc80 0x7fa77c0ae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa77c157c70 0x7fa77c0ae460 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 --2- 172.21.15.107:0/1813070440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77c0cf450 0x7fa77c14a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:57.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 >> 172.21.15.107:0/1813070440 conn(0x7fa77c06a920 msgr2=0x7fa77c0af0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:57.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 shutdown_connections 2023-11-24T08:05:57.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:57.821+0000 7fa78102b640 1 -- 172.21.15.107:0/1813070440 wait complete. 2023-11-24T08:05:57.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-24T08:05:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.042+0000 7f5b7b08c640 1 Processor -- start 2023-11-24T08:05:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.042+0000 7f5b7b08c640 1 -- start start 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.043+0000 7f5b7b08c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b740cf480 0x7f5b7415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.043+0000 7f5b7b08c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5b7415d7b0 0x7f5b740b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.043+0000 7f5b7b08c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740b2850 con 0x7f5b740b2400 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.043+0000 7f5b7b08c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7415a700 con 0x7f5b740cf480 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.043+0000 7f5b7b08c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7413d140 con 0x7f5b7415d7b0 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b740cf480 0x7f5b7415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b740cf480 0x7f5b7415a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53750/0 (socket says 172.21.15.107:53750) 2023-11-24T08:05:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 -- 172.21.15.107:0/1679115373 learned_addr learned my addr 172.21.15.107:0/1679115373 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b79889640 1 --2- 172.21.15.107:0/1679115373 >> v2:172.21.15.120:3300/0 conn(0x7f5b7415d7b0 0x7f5b740b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 -- 172.21.15.107:0/1679115373 >> v2:172.21.15.120:3300/0 conn(0x7f5b7415d7b0 msgr2=0x7f5b740b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 --2- 172.21.15.107:0/1679115373 >> v2:172.21.15.120:3300/0 conn(0x7f5b7415d7b0 0x7f5b740b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 -- 172.21.15.107:0/1679115373 >> v1:172.21.15.120:6789/0 conn(0x7f5b740b2400 legacy=0x7f5b7413f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.044+0000 7f5b7a08a640 1 -- 172.21.15.107:0/1679115373 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7415b190 con 0x7f5b740cf480 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.045+0000 7f5b7a08a640 1 --2- 172.21.15.107:0/1679115373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7415a1c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b7406b2a0 tx=0x7f5b7002deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783fbcfbe023586f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b79088640 1 -- 172.21.15.107:0/1679115373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7002ec60 con 0x7f5b740cf480 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b79088640 1 -- 172.21.15.107:0/1679115373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b70032070 con 0x7f5b740cf480 2023-11-24T08:05:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b79088640 1 -- 172.21.15.107:0/1679115373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70038400 con 0x7f5b740cf480 2023-11-24T08:05:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b7b08c640 1 -- 172.21.15.107:0/1679115373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 msgr2=0x7f5b7415a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/1679115373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7415a1c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b7406b2a0 tx=0x7f5b7002deb0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b7b08c640 1 -- 172.21.15.107:0/1679115373 shutdown_connections 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/1679115373 >> v2:172.21.15.120:3300/0 conn(0x7f5b7415d7b0 0x7f5b740b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.046+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/1679115373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7415a1c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.047+0000 7f5b7b08c640 1 -- 172.21.15.107:0/1679115373 >> 172.21.15.107:0/1679115373 conn(0x7f5b7406a500 msgr2=0x7f5b7406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.047+0000 7f5b7b08c640 1 -- 172.21.15.107:0/1679115373 shutdown_connections 2023-11-24T08:05:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.047+0000 7f5b7b08c640 1 -- 172.21.15.107:0/1679115373 wait complete. 2023-11-24T08:05:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.048+0000 7f5b7b08c640 1 Processor -- start 2023-11-24T08:05:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.048+0000 7f5b7b08c640 1 -- start start 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b740b2400 0x7f5b74159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b7415d7b0 0x7f5b740cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b740ce9a0 con 0x7f5b740cf480 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7413ceb0 con 0x7f5b7415d7b0 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.049+0000 7f5b7b08c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b740cdd20 con 0x7f5b740b2400 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53756/0 (socket says 172.21.15.107:53756) 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 -- 172.21.15.107:0/2385366327 learned_addr learned my addr 172.21.15.107:0/2385366327 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 -- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b740b2400 msgr2=0x7f5b74159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b740b2400 0x7f5b74159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 -- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b7415d7b0 msgr2=0x7f5b740cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b7415d7b0 0x7f5b740cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 -- 172.21.15.107:0/2385366327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7000b760 con 0x7f5b740cf480 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b79889640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400c7f0 tx=0x7f5b6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.050+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64004220 con 0x7f5b740cf480 2023-11-24T08:05:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.051+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b64004380 con 0x7f5b740cf480 2023-11-24T08:05:58.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.051+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64007840 con 0x7f5b740cf480 2023-11-24T08:05:58.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.051+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b740ce000 con 0x7f5b740cf480 2023-11-24T08:05:58.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.051+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b740ce5c0 con 0x7f5b740cf480 2023-11-24T08:05:58.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.052+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7404e840 con 0x7f5b740cf480 2023-11-24T08:05:58.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.053+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5b64007bb0 con 0x7f5b740cf480 2023-11-24T08:05:58.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.054+0000 7f5b6affd640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 0x7f5b5806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.054+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b64089150 con 0x7f5b740cf480 2023-11-24T08:05:58.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.054+0000 7f5b7a08a640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 0x7f5b5806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.055+0000 7f5b7a08a640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 0x7f5b5806d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b70039000 tx=0x7f5b70030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.058+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b64054540 con 0x7f5b740cf480 2023-11-24T08:05:58.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.298+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f5b74072410 con 0x7f5b740cf480 2023-11-24T08:05:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.303+0000 7f5b6affd640 1 -- 172.21.15.107:0/2385366327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5b64007e60 con 0x7f5b740cf480 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 msgr2=0x7f5b5806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 0x7f5b5806d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b70039000 tx=0x7f5b70030000 comp rx=0 tx=0).stop 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 msgr2=0x7f5b7413c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400c7f0 tx=0x7f5b6400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 shutdown_connections 2023-11-24T08:05:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b5806ae40 0x7f5b5806d300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b7415d7b0 0x7f5b740cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b740cf480 0x7f5b7413c970 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 --2- 172.21.15.107:0/2385366327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b740b2400 0x7f5b74159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 >> 172.21.15.107:0/2385366327 conn(0x7f5b7406a500 msgr2=0x7f5b740ae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 shutdown_connections 2023-11-24T08:05:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.306+0000 7f5b7b08c640 1 -- 172.21.15.107:0/2385366327 wait complete. 2023-11-24T08:05:58.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-24T08:05:58.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.532+0000 7f9644b68640 1 Processor -- start 2023-11-24T08:05:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.533+0000 7f9644b68640 1 -- start start 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.534+0000 7f9644b68640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f964006e160 0x7f96400ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.534+0000 7f9644b68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96400cd110 0x7f9640070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.534+0000 7f9644b68640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9640161a10 con 0x7f9640146eb0 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.534+0000 7f9644b68640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96400cd580 con 0x7f96400cd110 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.534+0000 7f9644b68640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9640161cf0 con 0x7f964006e160 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96400cd110 0x7f9640070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96400cd110 0x7f9640070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53772/0 (socket says 172.21.15.107:53772) 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 -- 172.21.15.107:0/1629727775 learned_addr learned my addr 172.21.15.107:0/1629727775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f9637fff640 1 --2- 172.21.15.107:0/1629727775 >> v2:172.21.15.120:3300/0 conn(0x7f964006e160 0x7f96400ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 -- 172.21.15.107:0/1629727775 >> v2:172.21.15.120:3300/0 conn(0x7f964006e160 msgr2=0x7f96400ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 --2- 172.21.15.107:0/1629727775 >> v2:172.21.15.120:3300/0 conn(0x7f964006e160 0x7f96400ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 -- 172.21.15.107:0/1629727775 >> v1:172.21.15.120:6789/0 conn(0x7f9640146eb0 legacy=0x7f9640161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.535+0000 7f963ed9d640 1 -- 172.21.15.107:0/1629727775 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640161fd0 con 0x7f96400cd110 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.536+0000 7f963ed9d640 1 --2- 172.21.15.107:0/1629727775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f9640070e40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f964006eab0 tx=0x7f963802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5619a5d8034beb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f963dd9b640 1 -- 172.21.15.107:0/1629727775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963802ca80 con 0x7f96400cd110 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f963dd9b640 1 -- 172.21.15.107:0/1629727775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963802cbe0 con 0x7f96400cd110 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f963dd9b640 1 -- 172.21.15.107:0/1629727775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380343f0 con 0x7f96400cd110 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 -- 172.21.15.107:0/1629727775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 msgr2=0x7f9640070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 --2- 172.21.15.107:0/1629727775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f9640070e40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f964006eab0 tx=0x7f963802bee0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 -- 172.21.15.107:0/1629727775 shutdown_connections 2023-11-24T08:05:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 --2- 172.21.15.107:0/1629727775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f9640070e40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 --2- 172.21.15.107:0/1629727775 >> v2:172.21.15.120:3300/0 conn(0x7f964006e160 0x7f96400ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.537+0000 7f9644b68640 1 -- 172.21.15.107:0/1629727775 >> 172.21.15.107:0/1629727775 conn(0x7f96400b0c50 msgr2=0x7f96400b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.538+0000 7f9644b68640 1 -- 172.21.15.107:0/1629727775 shutdown_connections 2023-11-24T08:05:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.538+0000 7f9644b68640 1 -- 172.21.15.107:0/1629727775 wait complete. 2023-11-24T08:05:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.539+0000 7f9644b68640 1 Processor -- start 2023-11-24T08:05:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.539+0000 7f9644b68640 1 -- start start 2023-11-24T08:05:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f964006e160 0x7f9640143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9640146eb0 0x7f964009a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964015eec0 con 0x7f96400cd110 2023-11-24T08:05:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9640144350 con 0x7f9640146eb0 2023-11-24T08:05:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.540+0000 7f9644b68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96400b1eb0 con 0x7f964006e160 2023-11-24T08:05:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53778/0 (socket says 172.21.15.107:53778) 2023-11-24T08:05:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 -- 172.21.15.107:0/2116211323 learned_addr learned my addr 172.21.15.107:0/2116211323 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 -- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f964006e160 msgr2=0x7f9640143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f964006e160 0x7f9640143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 -- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9640146eb0 msgr2=0x7f964009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9640146eb0 0x7f964009a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 -- 172.21.15.107:0/2116211323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638009b30 con 0x7f96400cd110 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.541+0000 7f9637fff640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f962800c7f0 tx=0x7f962800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.542+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628004220 con 0x7f96400cd110 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.542+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96280043a0 con 0x7f96400cd110 2023-11-24T08:05:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.542+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628007860 con 0x7f96400cd110 2023-11-24T08:05:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.542+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96400b2190 con 0x7f96400cd110 2023-11-24T08:05:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.542+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96400b2750 con 0x7f96400cd110 2023-11-24T08:05:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.543+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96401607b0 con 0x7f96400cd110 2023-11-24T08:05:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.544+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9628007bd0 con 0x7f96400cd110 2023-11-24T08:05:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.545+0000 7f9636ffd640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 0x7f962006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.545+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9628089390 con 0x7f96400cd110 2023-11-24T08:05:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.545+0000 7f963e59c640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 0x7f962006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.546+0000 7f963e59c640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 0x7f962006d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9630009650 tx=0x7f9630006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.550+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96401607b0 con 0x7f96400cd110 2023-11-24T08:05:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.803+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f96401607b0 con 0x7f96400cd110 2023-11-24T08:05:58.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.808+0000 7f9636ffd640 1 -- 172.21.15.107:0/2116211323 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f96401607b0 con 0x7f96400cd110 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 msgr2=0x7f962006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 0x7f962006d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9630009650 tx=0x7f9630006ba0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 msgr2=0x7f964015e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f962800c7f0 tx=0x7f962800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 shutdown_connections 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f962006ae40 0x7f962006d300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9640146eb0 0x7f964009a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96400cd110 0x7f964015e550 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 --2- 172.21.15.107:0/2116211323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f964006e160 0x7f9640143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:58.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.810+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 >> 172.21.15.107:0/2116211323 conn(0x7f96400b0c50 msgr2=0x7f9640071740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:58.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.811+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 shutdown_connections 2023-11-24T08:05:58.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:58.811+0000 7f9644b68640 1 -- 172.21.15.107:0/2116211323 wait complete. 2023-11-24T08:05:58.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-24T08:05:59.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.036+0000 7f2b9fb5a640 1 Processor -- start 2023-11-24T08:05:59.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.036+0000 7f2b9fb5a640 1 -- start start 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9fb5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b9815c7f0 0x7f2b9806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9fb5a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 0x7f2b980cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9fb5a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980cece0 con 0x7f2b980cdfb0 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9fb5a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980cfe40 con 0x7f2b9815c7f0 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9fb5a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980ceaf0 con 0x7f2b980cf450 2023-11-24T08:05:59.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9d8f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b9815c7f0 0x7f2b9806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9d8f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b9815c7f0 0x7f2b9806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53786/0 (socket says 172.21.15.107:53786) 2023-11-24T08:05:59.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.038+0000 7f2b9d8f6640 1 -- 172.21.15.107:0/2564851973 learned_addr learned my addr 172.21.15.107:0/2564851973 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d0f5640 1 --2- 172.21.15.107:0/2564851973 >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 0x7f2b980cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d8f6640 1 -- 172.21.15.107:0/2564851973 >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 msgr2=0x7f2b980cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d8f6640 1 --2- 172.21.15.107:0/2564851973 >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 0x7f2b980cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d8f6640 1 -- 172.21.15.107:0/2564851973 >> v1:172.21.15.120:6789/0 conn(0x7f2b980cdfb0 legacy=0x7f2b980ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d8f6640 1 -- 172.21.15.107:0/2564851973 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9806d040 con 0x7f2b9815c7f0 2023-11-24T08:05:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.039+0000 7f2b9d0f5640 1 --2- 172.21.15.107:0/2564851973 >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 0x7f2b980cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:05:59.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.040+0000 7f2b9d8f6640 1 --2- 172.21.15.107:0/2564851973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b9806db20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2b98070410 tx=0x7f2b9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f494a00ac39f1b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.040+0000 7f2b9c8f4640 1 -- 172.21.15.107:0/2564851973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94031870 con 0x7f2b9815c7f0 2023-11-24T08:05:59.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.040+0000 7f2b9c8f4640 1 -- 172.21.15.107:0/2564851973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b940319d0 con 0x7f2b9815c7f0 2023-11-24T08:05:59.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.040+0000 7f2b9c8f4640 1 -- 172.21.15.107:0/2564851973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94031d60 con 0x7f2b9815c7f0 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/2564851973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 msgr2=0x7f2b9806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/2564851973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b9806db20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2b98070410 tx=0x7f2b9402b910 comp rx=0 tx=0).stop 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/2564851973 shutdown_connections 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/2564851973 >> v2:172.21.15.120:3300/0 conn(0x7f2b980cf450 0x7f2b980cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/2564851973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b9806db20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/2564851973 >> 172.21.15.107:0/2564851973 conn(0x7f2b9806a920 msgr2=0x7f2b9806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/2564851973 shutdown_connections 2023-11-24T08:05:59.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.041+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/2564851973 wait complete. 2023-11-24T08:05:59.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.043+0000 7f2b9fb5a640 1 Processor -- start 2023-11-24T08:05:59.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.043+0000 7f2b9fb5a640 1 -- start start 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b980cdfb0 0x7f2b98141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b980cf450 0x7f2b98098890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981430e0 con 0x7f2b9815c7f0 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981557c0 con 0x7f2b980cdfb0 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.044+0000 7f2b9fb5a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98142460 con 0x7f2b980cf450 2023-11-24T08:05:59.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53796/0 (socket says 172.21.15.107:53796) 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 -- 172.21.15.107:0/1337159257 learned_addr learned my addr 172.21.15.107:0/1337159257 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9d8f6640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b980cdfb0 0x7f2b98141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9d0f5640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b980cf450 0x7f2b98098890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 -- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b980cf450 msgr2=0x7f2b98098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b980cf450 0x7f2b98098890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 -- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b980cdfb0 msgr2=0x7f2b98141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b980cdfb0 0x7f2b98141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 -- 172.21.15.107:0/1337159257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b940094d0 con 0x7f2b9815c7f0 2023-11-24T08:05:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.045+0000 7f2b9e0f7640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c780 tx=0x7f2b9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.046+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90003d80 con 0x7f2b9815c7f0 2023-11-24T08:05:59.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.046+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9000ce30 con 0x7f2b9815c7f0 2023-11-24T08:05:59.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.046+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90005290 con 0x7f2b9815c7f0 2023-11-24T08:05:59.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.046+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b98142740 con 0x7f2b9815c7f0 2023-11-24T08:05:59.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.046+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b98142d00 con 0x7f2b9815c7f0 2023-11-24T08:05:59.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.047+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b98143220 con 0x7f2b9815c7f0 2023-11-24T08:05:59.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.048+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2b900053f0 con 0x7f2b9815c7f0 2023-11-24T08:05:59.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.048+0000 7f2b8e7fc640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 0x7f2b7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.049+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2b90088700 con 0x7f2b9815c7f0 2023-11-24T08:05:59.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.049+0000 7f2b9d8f6640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 0x7f2b7c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.049+0000 7f2b9d8f6640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 0x7f2b7c06d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2b94036000 tx=0x7f2b9403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.053+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b98143220 con 0x7f2b9815c7f0 2023-11-24T08:05:59.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.290+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f2b98143220 con 0x7f2b9815c7f0 2023-11-24T08:05:59.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.294+0000 7f2b8e7fc640 1 -- 172.21.15.107:0/1337159257 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2b98143220 con 0x7f2b9815c7f0 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 msgr2=0x7f2b7c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 0x7f2b7c06d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2b94036000 tx=0x7f2b9403c040 comp rx=0 tx=0).stop 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 msgr2=0x7f2b98143500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c780 tx=0x7f2b9000cc40 comp rx=0 tx=0).stop 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 shutdown_connections 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2b7c06ae40 0x7f2b7c06d300 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9815c7f0 0x7f2b98143500 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b980cf450 0x7f2b98098890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 --2- 172.21.15.107:0/1337159257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b980cdfb0 0x7f2b98141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 >> 172.21.15.107:0/1337159257 conn(0x7f2b9806a920 msgr2=0x7f2b980ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 shutdown_connections 2023-11-24T08:05:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.297+0000 7f2b9fb5a640 1 -- 172.21.15.107:0/1337159257 wait complete. 2023-11-24T08:05:59.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:05:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.524+0000 7fcba3333640 1 Processor -- start 2023-11-24T08:05:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.525+0000 7fcba3333640 1 -- start start 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba3333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 0x7fcb9c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba3333640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c0ce270 con 0x7fcb9c0cc470 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba3333640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c06d0a0 con 0x7fcb9c0ce640 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba3333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c06c560 con 0x7fcb9c0cfb60 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba10cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcb9c0cc470 0x7fcb9c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47966/0 (socket says 172.21.15.107:47966) 2023-11-24T08:05:59.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba10cf640 1 -- 172.21.15.107:0/1303413584 learned_addr learned my addr 172.21.15.107:0/1303413584 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971871701 0 0) 0x7fcb9c0ce270 con 0x7fcb9c0cc470 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcba08ce640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 0x7fcb9c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.526+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb88003520 con 0x7fcb9c0cc470 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2156776005 0 0) 0x7fcb88003520 con 0x7fcb9c0cc470 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 msgr2=0x7fcb9c0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 0x7fcb9c0abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 >> v1:172.21.15.120:6789/0 conn(0x7fcb9c0ce640 legacy=0x7fcb9c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c06c770 con 0x7fcb9c0cc470 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcb98001670 con 0x7fcb9c0cc470 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 >> v1:172.21.15.107:6789/0 conn(0x7fcb9c0cc470 legacy=0x7fcb9c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880047c0 0x7fcb88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 0x7fcb88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c06c560 con 0x7fcb880039a0 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8800ba30 con 0x7fcb880047c0 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8800bbf0 con 0x7fcb9c0cfb60 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.527+0000 7fcba10cf640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.528+0000 7fcba10cf640 1 -- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 msgr2=0x7fcb88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.528+0000 7fcba10cf640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c0cfb60 0x7fcb88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.528+0000 7fcba10cf640 1 -- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880047c0 msgr2=0x7fcb88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.528+0000 7fcba10cf640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880047c0 0x7fcb88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.528+0000 7fcba10cf640 1 -- 172.21.15.107:0/1303413584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88003520 con 0x7fcb880039a0 2023-11-24T08:05:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcba10cf640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb88003dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800c210 tx=0x7fcb880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66489f64ba9031b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98005ad0 con 0x7fcb880039a0 2023-11-24T08:05:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb98005c30 con 0x7fcb880039a0 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcb93fff640 1 -- 172.21.15.107:0/1303413584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9802c870 con 0x7fcb880039a0 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcba3333640 1 -- 172.21.15.107:0/1303413584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 msgr2=0x7fcb88003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcba3333640 1 --2- 172.21.15.107:0/1303413584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb88003dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800c210 tx=0x7fcb880057f0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.529+0000 7fcba18d0640 1 -- 172.21.15.107:0/1303413584 reap_dead start 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.530+0000 7fcba3333640 1 -- 172.21.15.107:0/1303413584 shutdown_connections 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.530+0000 7fcba3333640 1 -- 172.21.15.107:0/1303413584 >> 172.21.15.107:0/1303413584 conn(0x7fcb9c06a920 msgr2=0x7fcb9c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.530+0000 7fcba3333640 1 -- 172.21.15.107:0/1303413584 shutdown_connections 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.530+0000 7fcba3333640 1 -- 172.21.15.107:0/1303413584 wait complete. 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.532+0000 7fcba3333640 1 Processor -- start 2023-11-24T08:05:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.532+0000 7fcba3333640 1 -- start start 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c157be0 0x7fcb9c163090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c164bc0 0x7fcb9c165050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0afad0 con 0x7fcb880039a0 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0af130 con 0x7fcb9c164bc0 2023-11-24T08:05:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba3333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0af410 con 0x7fcb9c157be0 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53828/0 (socket says 172.21.15.107:53828) 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 -- 172.21.15.107:0/625465590 learned_addr learned my addr 172.21.15.107:0/625465590 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 -- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c157be0 msgr2=0x7fcb9c163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c157be0 0x7fcb9c163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 -- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c164bc0 msgr2=0x7fcb9c165050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c164bc0 0x7fcb9c165050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.533+0000 7fcba10cf640 1 -- 172.21.15.107:0/625465590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0af6f0 con 0x7fcb880039a0 2023-11-24T08:05:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcba10cf640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb98001930 tx=0x7fcb98001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98039070 con 0x7fcb880039a0 2023-11-24T08:05:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9802cc80 con 0x7fcb880039a0 2023-11-24T08:05:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9803c040 con 0x7fcb880039a0 2023-11-24T08:05:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c1564a0 con 0x7fcb880039a0 2023-11-24T08:05:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.534+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c1582c0 con 0x7fcb880039a0 2023-11-24T08:05:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.535+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c067470 con 0x7fcb880039a0 2023-11-24T08:05:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.536+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcb98032070 con 0x7fcb880039a0 2023-11-24T08:05:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.537+0000 7fcb91ffb640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 0x7fcb8006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.537+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcb980aa5e0 con 0x7fcb880039a0 2023-11-24T08:05:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.537+0000 7fcba08ce640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 0x7fcb8006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.538+0000 7fcba08ce640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 0x7fcb8006d2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c006e80 tx=0x7fcb8c006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.541+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb980759d0 con 0x7fcb880039a0 2023-11-24T08:05:59.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.734+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb9c0b1470 con 0x7fcb880039a0 2023-11-24T08:05:59.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.736+0000 7fcb91ffb640 1 -- 172.21.15.107:0/625465590 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fcb98046090 con 0x7fcb880039a0 2023-11-24T08:05:59.738 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:05:59.738 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:05:55.756289+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:05:59.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 msgr2=0x7fcb8006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 0x7fcb8006d2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c006e80 tx=0x7fcb8c006dd0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 msgr2=0x7fcb9c15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb98001930 tx=0x7fcb98001a00 comp rx=0 tx=0).stop 2023-11-24T08:05:59.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 shutdown_connections 2023-11-24T08:05:59.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb8006ae30 0x7fcb8006d2f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c164bc0 0x7fcb9c165050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb9c157be0 0x7fcb9c163090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 --2- 172.21.15.107:0/625465590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb880039a0 0x7fcb9c15a180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 >> 172.21.15.107:0/625465590 conn(0x7fcb9c06a920 msgr2=0x7fcb9c0722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 shutdown_connections 2023-11-24T08:05:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.739+0000 7fcba3333640 1 -- 172.21.15.107:0/625465590 wait complete. 2023-11-24T08:05:59.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:05:59.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:05:59.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.974+0000 7f37caae7640 1 Processor -- start 2023-11-24T08:05:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.975+0000 7f37caae7640 1 -- start start 2023-11-24T08:05:59.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.976+0000 7f37caae7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37c40d1c20 0x7f37c406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.976+0000 7f37caae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c40ce410 con 0x7f37c41605f0 2023-11-24T08:05:59.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.976+0000 7f37caae7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c40d06f0 con 0x7f37c40d2000 2023-11-24T08:05:59.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.976+0000 7f37caae7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c40d09d0 con 0x7f37c40d1c20 2023-11-24T08:05:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.976+0000 7f37c8883640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37c40d1c20 0x7f37c406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.977+0000 7f37c8883640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37c40d1c20 0x7f37c406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53838/0 (socket says 172.21.15.107:53838) 2023-11-24T08:05:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.977+0000 7f37c8883640 1 -- 172.21.15.107:0/2849080086 learned_addr learned my addr 172.21.15.107:0/2849080086 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.977+0000 7f37c8883640 1 -- 172.21.15.107:0/2849080086 >> v1:172.21.15.120:6789/0 conn(0x7f37c40d2000 legacy=0x7f37c40cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.977+0000 7f37c8883640 1 -- 172.21.15.107:0/2849080086 >> v1:172.21.15.107:6789/0 conn(0x7f37c41605f0 legacy=0x7f37c4160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:05:59.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.977+0000 7f37c8883640 1 -- 172.21.15.107:0/2849080086 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40ce0a0 con 0x7f37c40d1c20 2023-11-24T08:05:59.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.979+0000 7f37c8883640 1 --2- 172.21.15.107:0/2849080086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c406bb60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009e50 tx=0x7f37b402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295745381ba657e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.979+0000 7f37c37fe640 1 -- 172.21.15.107:0/2849080086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b402cbb0 con 0x7f37c40d1c20 2023-11-24T08:05:59.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.979+0000 7f37c37fe640 1 -- 172.21.15.107:0/2849080086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b402cd10 con 0x7f37c40d1c20 2023-11-24T08:05:59.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.979+0000 7f37c37fe640 1 -- 172.21.15.107:0/2849080086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4034610 con 0x7f37c40d1c20 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 -- 172.21.15.107:0/2849080086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 msgr2=0x7f37c406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 --2- 172.21.15.107:0/2849080086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c406bb60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009e50 tx=0x7f37b402c010 comp rx=0 tx=0).stop 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 -- 172.21.15.107:0/2849080086 shutdown_connections 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 --2- 172.21.15.107:0/2849080086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c406bb60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 -- 172.21.15.107:0/2849080086 >> 172.21.15.107:0/2849080086 conn(0x7f37c408cea0 msgr2=0x7f37c409c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:05:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 -- 172.21.15.107:0/2849080086 shutdown_connections 2023-11-24T08:05:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.980+0000 7f37caae7640 1 -- 172.21.15.107:0/2849080086 wait complete. 2023-11-24T08:05:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.982+0000 7f37caae7640 1 Processor -- start 2023-11-24T08:05:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.982+0000 7f37caae7640 1 -- start start 2023-11-24T08:05:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c40d2000 0x7f37c40b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c41605f0 0x7f37c40cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40b0120 con 0x7f37c40d1c20 2023-11-24T08:05:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40cced0 con 0x7f37c41605f0 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37caae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c409def0 con 0x7f37c40d2000 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53840/0 (socket says 172.21.15.107:53840) 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 -- 172.21.15.107:0/1907406689 learned_addr learned my addr 172.21.15.107:0/1907406689 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 -- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c40d2000 msgr2=0x7f37c40b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c40d2000 0x7f37c40b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 -- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c41605f0 msgr2=0x7f37c40cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:05:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c41605f0 0x7f37c40cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:05:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.983+0000 7f37c8883640 1 -- 172.21.15.107:0/1907406689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4009b30 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37c8883640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f37b4036040 tx=0x7f37b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b403e450 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b403e5f0 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b403e9a0 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c409e190 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.984+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c409e750 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.985+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c404e8b0 con 0x7f37c40d1c20 2023-11-24T08:05:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.986+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f37b403eb00 con 0x7f37c40d1c20 2023-11-24T08:05:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.987+0000 7f37c17fa640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 0x7f37a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:05:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.987+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f37b40ac1d0 con 0x7f37c40d1c20 2023-11-24T08:05:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.987+0000 7f37c3fff640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 0x7f37a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:05:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.988+0000 7f37c3fff640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 0x7f37a406d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37ac000f60 tx=0x7f37ac0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:05:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:05:59.992+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37b4077670 con 0x7f37c40d1c20 2023-11-24T08:06:00.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.181+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37c406b770 con 0x7f37c40d1c20 2023-11-24T08:06:00.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.182+0000 7f37c17fa640 1 -- 172.21.15.107:0/1907406689 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37b403edb0 con 0x7f37c40d1c20 2023-11-24T08:06:00.184 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:06:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 msgr2=0x7f37a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 0x7f37a406d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37ac000f60 tx=0x7f37ac0079b0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 msgr2=0x7f37c415e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f37b4036040 tx=0x7f37b4004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 shutdown_connections 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a406ae40 0x7f37a406d300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c41605f0 0x7f37c40cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c40d2000 0x7f37c40b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 --2- 172.21.15.107:0/1907406689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c40d1c20 0x7f37c415e920 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 >> 172.21.15.107:0/1907406689 conn(0x7f37c408cea0 msgr2=0x7f37c415fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 shutdown_connections 2023-11-24T08:06:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.185+0000 7f37caae7640 1 -- 172.21.15.107:0/1907406689 wait complete. 2023-11-24T08:06:00.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:06:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.415+0000 7fbec7694640 1 Processor -- start 2023-11-24T08:06:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.415+0000 7fbec7694640 1 -- start start 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec7694640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 0x7fbec00d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec7694640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbec00d22b0 0x7fbec00ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec7694640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00b2d90 con 0x7fbec00ce710 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec7694640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00ceb80 con 0x7fbec00d22b0 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec7694640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00d1540 con 0x7fbec00d1ed0 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec5c31640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbec00ce710 0x7fbec00cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47996/0 (socket says 172.21.15.107:47996) 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec5c31640 1 -- 172.21.15.107:0/400360092 learned_addr learned my addr 172.21.15.107:0/400360092 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec4c2f640 1 --2- 172.21.15.107:0/400360092 >> v2:172.21.15.107:3300/0 conn(0x7fbec00d22b0 0x7fbec00ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.417+0000 7fbec5430640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 0x7fbec00d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbeb7fff640 1 -- 172.21.15.107:0/400360092 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411510411 0 0) 0x7fbec00b2d90 con 0x7fbec00ce710 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbeb7fff640 1 -- 172.21.15.107:0/400360092 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac003520 con 0x7fbec00ce710 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbec4c2f640 1 -- 172.21.15.107:0/400360092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 msgr2=0x7fbec00d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbec4c2f640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 0x7fbec00d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbec4c2f640 1 -- 172.21.15.107:0/400360092 >> v1:172.21.15.107:6789/0 conn(0x7fbec00ce710 legacy=0x7fbec00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbec4c2f640 1 -- 172.21.15.107:0/400360092 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00b45f0 con 0x7fbec00d22b0 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.418+0000 7fbec5430640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 0x7fbec00d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:06:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.419+0000 7fbec4c2f640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec00ce1d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00097a0 tx=0x7fbeb002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173eb37acdf82c0d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.419+0000 7fbeb7fff640 1 -- 172.21.15.107:0/400360092 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb002e070 con 0x7fbec00d22b0 2023-11-24T08:06:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.419+0000 7fbeb7fff640 1 -- 172.21.15.107:0/400360092 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb00315a0 con 0x7fbec00d22b0 2023-11-24T08:06:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.419+0000 7fbeb7fff640 1 -- 172.21.15.107:0/400360092 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0031930 con 0x7fbec00d22b0 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 -- 172.21.15.107:0/400360092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 msgr2=0x7fbec00ce1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec00ce1d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00097a0 tx=0x7fbeb002b840 comp rx=0 tx=0).stop 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 -- 172.21.15.107:0/400360092 shutdown_connections 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec00ce1d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 --2- 172.21.15.107:0/400360092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00d1ed0 0x7fbec00d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 -- 172.21.15.107:0/400360092 >> 172.21.15.107:0/400360092 conn(0x7fbec008af60 msgr2=0x7fbec00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 -- 172.21.15.107:0/400360092 shutdown_connections 2023-11-24T08:06:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.420+0000 7fbec7694640 1 -- 172.21.15.107:0/400360092 wait complete. 2023-11-24T08:06:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.422+0000 7fbec7694640 1 Processor -- start 2023-11-24T08:06:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.422+0000 7fbec7694640 1 -- start start 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00ce710 0x7fbec006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbec00d1ed0 0x7fbec01e2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00ca730 con 0x7fbec00d22b0 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec01e2bc0 con 0x7fbec00d1ed0 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.423+0000 7fbec7694640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec01e2ea0 con 0x7fbec00ce710 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53844/0 (socket says 172.21.15.107:53844) 2023-11-24T08:06:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 -- 172.21.15.107:0/3421947900 learned_addr learned my addr 172.21.15.107:0/3421947900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5430640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00ce710 0x7fbec006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec4c2f640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbec00d1ed0 0x7fbec01e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 -- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00ce710 msgr2=0x7fbec006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00ce710 0x7fbec006c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 -- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbec00d1ed0 msgr2=0x7fbec01e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbec00d1ed0 0x7fbec01e2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.424+0000 7fbec5c31640 1 -- 172.21.15.107:0/3421947900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0009400 con 0x7fbec00d22b0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbec5c31640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800cb40 tx=0x7fbeb800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb800aca0 con 0x7fbec00d22b0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800ae00 con 0x7fbec00d22b0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec01e3180 con 0x7fbec00d22b0 2023-11-24T08:06:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8012620 con 0x7fbec00d22b0 2023-11-24T08:06:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.425+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec006cbe0 con 0x7fbec00d22b0 2023-11-24T08:06:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.426+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec01e33c0 con 0x7fbec00d22b0 2023-11-24T08:06:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.427+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbeb8007070 con 0x7fbec00d22b0 2023-11-24T08:06:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.428+0000 7fbeb5ffb640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 0x7fbea406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.428+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbeb8089320 con 0x7fbec00d22b0 2023-11-24T08:06:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.429+0000 7fbec5430640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 0x7fbea406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.429+0000 7fbec5430640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 0x7fbea406d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006fd0 tx=0x7fbebc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.433+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbec01e33c0 con 0x7fbec00d22b0 2023-11-24T08:06:00.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.627+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbec00ca500 con 0x7fbec00d22b0 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.628+0000 7fbeb5ffb640 1 -- 172.21.15.107:0/3421947900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbeb8017060 con 0x7fbec00d22b0 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:06:00.630 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:06:00.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:00.631 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:06:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 msgr2=0x7fbea406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 0x7fbea406d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006fd0 tx=0x7fbebc008040 comp rx=0 tx=0).stop 2023-11-24T08:06:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 msgr2=0x7fbec01e14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800cb40 tx=0x7fbeb800a060 comp rx=0 tx=0).stop 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 shutdown_connections 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbea406ae70 0x7fbea406d330 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec00d22b0 0x7fbec01e14e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbec00d1ed0 0x7fbec01e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 --2- 172.21.15.107:0/3421947900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbec00ce710 0x7fbec006c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.630+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 >> 172.21.15.107:0/3421947900 conn(0x7fbec008af60 msgr2=0x7fbec01e4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.631+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 shutdown_connections 2023-11-24T08:06:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.631+0000 7fbec7694640 1 -- 172.21.15.107:0/3421947900 wait complete. 2023-11-24T08:06:00.650 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:06:00.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:06:00.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.838+0000 7fc6b00d3640 1 Processor -- start 2023-11-24T08:06:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.838+0000 7fc6b00d3640 1 -- start start 2023-11-24T08:06:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6b00d3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc6a815c7f0 0x7fc6a806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6b00d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a80cf450 0x7fc6a80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6b00d3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a80cece0 con 0x7fc6a80cdfb0 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6b00d3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a8156010 con 0x7fc6a815c7f0 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6b00d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a806ce30 con 0x7fc6a80cf450 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6ade6f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc6a815c7f0 0x7fc6a806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6ade6f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc6a815c7f0 0x7fc6a806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53856/0 (socket says 172.21.15.107:53856) 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.839+0000 7fc6ade6f640 1 -- 172.21.15.107:0/3508605261 learned_addr learned my addr 172.21.15.107:0/3508605261 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.840+0000 7fc6ade6f640 1 -- 172.21.15.107:0/3508605261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a80cf450 msgr2=0x7fc6a80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.840+0000 7fc6ad66e640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a80cf450 0x7fc6a80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.840+0000 7fc6ade6f640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a80cf450 0x7fc6a80cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.840+0000 7fc6ade6f640 1 -- 172.21.15.107:0/3508605261 >> v1:172.21.15.120:6789/0 conn(0x7fc6a80cdfb0 legacy=0x7fc6a80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.840+0000 7fc6ade6f640 1 -- 172.21.15.107:0/3508605261 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a806d040 con 0x7fc6a815c7f0 2023-11-24T08:06:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.841+0000 7fc6ade6f640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a815c7f0 0x7fc6a806db20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009870 tx=0x7fc6a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ea48a65ae78f47 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.841+0000 7fc6ace6d640 1 -- 172.21.15.107:0/3508605261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031640 con 0x7fc6a815c7f0 2023-11-24T08:06:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.841+0000 7fc6ace6d640 1 -- 172.21.15.107:0/3508605261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a40317a0 con 0x7fc6a815c7f0 2023-11-24T08:06:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.841+0000 7fc6ace6d640 1 -- 172.21.15.107:0/3508605261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031b30 con 0x7fc6a815c7f0 2023-11-24T08:06:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 -- 172.21.15.107:0/3508605261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a815c7f0 msgr2=0x7fc6a806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a815c7f0 0x7fc6a806db20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009870 tx=0x7fc6a402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 -- 172.21.15.107:0/3508605261 shutdown_connections 2023-11-24T08:06:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a80cf450 0x7fc6a80cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/3508605261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a815c7f0 0x7fc6a806db20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 -- 172.21.15.107:0/3508605261 >> 172.21.15.107:0/3508605261 conn(0x7fc6a806a920 msgr2=0x7fc6a806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 -- 172.21.15.107:0/3508605261 shutdown_connections 2023-11-24T08:06:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.842+0000 7fc6b00d3640 1 -- 172.21.15.107:0/3508605261 wait complete. 2023-11-24T08:06:00.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.844+0000 7fc6b00d3640 1 Processor -- start 2023-11-24T08:06:00.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.844+0000 7fc6b00d3640 1 -- start start 2023-11-24T08:06:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a80cdfb0 0x7fc6a8157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a815c7f0 0x7fc6a815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8157ac0 con 0x7fc6a80cf450 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8156e40 con 0x7fc6a80cdfb0 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6b00d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8157120 con 0x7fc6a815c7f0 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6ad66e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6ad66e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53866/0 (socket says 172.21.15.107:53866) 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.845+0000 7fc6ad66e640 1 -- 172.21.15.107:0/2489925080 learned_addr learned my addr 172.21.15.107:0/2489925080 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ade6f640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a80cdfb0 0x7fc6a8157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 -- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a815c7f0 msgr2=0x7fc6a815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a815c7f0 0x7fc6a815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 -- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a80cdfb0 msgr2=0x7fc6a8157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:00.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a80cdfb0 0x7fc6a8157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:00.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 -- 172.21.15.107:0/2489925080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a40094d0 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6ad66e640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc69800c7f0 tx=0x7fc69800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698004220 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6980043a0 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a8157400 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.846+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698007860 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.847+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a814bdf0 con 0x7fc6a80cf450 2023-11-24T08:06:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.848+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8157c00 con 0x7fc6a80cf450 2023-11-24T08:06:00.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.850+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc698007bd0 con 0x7fc6a80cf450 2023-11-24T08:06:00.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.851+0000 7fc69e7fc640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 0x7fc68c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:00.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.851+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc698089180 con 0x7fc6a80cf450 2023-11-24T08:06:00.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.851+0000 7fc6ade6f640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 0x7fc68c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:00.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.852+0000 7fc6ade6f640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 0x7fc68c06d300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4036000 tx=0x7fc6a403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:00.856+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6a8157c00 con 0x7fc6a80cf450 2023-11-24T08:06:01.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.042+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc6a8157c00 con 0x7fc6a80cf450 2023-11-24T08:06:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.951+0000 7fc69e7fc640 1 -- 172.21.15.107:0/2489925080 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc6a8157c00 con 0x7fc6a80cf450 2023-11-24T08:06:01.953 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:06:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 msgr2=0x7fc68c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 0x7fc68c06d300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4036000 tx=0x7fc6a403c040 comp rx=0 tx=0).stop 2023-11-24T08:06:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 msgr2=0x7fc6a814b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:01.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc69800c7f0 tx=0x7fc69800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 shutdown_connections 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc68c06ae40 0x7fc68c06d300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a815c7f0 0x7fc6a815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6a80cf450 0x7fc6a814b8b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 --2- 172.21.15.107:0/2489925080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a80cdfb0 0x7fc6a8157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.954+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 >> 172.21.15.107:0/2489925080 conn(0x7fc6a806a920 msgr2=0x7fc6a80734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:01.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.955+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 shutdown_connections 2023-11-24T08:06:01.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:01.955+0000 7fc6b00d3640 1 -- 172.21.15.107:0/2489925080 wait complete. 2023-11-24T08:06:01.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:06:02.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.181+0000 7fcfea5ef640 1 Processor -- start 2023-11-24T08:06:02.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.181+0000 7fcfea5ef640 1 -- start start 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.182+0000 7fcfea5ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcfe415d490 0x7fcfe40cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfea5ef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfe40d3940 0x7fcfe40d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfea5ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe40cea30 con 0x7fcfe40b63d0 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfea5ef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe40d0450 con 0x7fcfe40d3940 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfea5ef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe40d0bf0 con 0x7fcfe415d490 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe9dee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfe40d3940 0x7fcfe40d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe8dec640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcfe40b63d0 0x7fcfe40d3230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48008/0 (socket says 172.21.15.107:48008) 2023-11-24T08:06:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe9dee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfe40d3940 0x7fcfe40d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53870/0 (socket says 172.21.15.107:53870) 2023-11-24T08:06:02.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe8dec640 1 -- 172.21.15.107:0/760087996 learned_addr learned my addr 172.21.15.107:0/760087996 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:02.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe95ed640 1 --2- 172.21.15.107:0/760087996 >> v2:172.21.15.120:3300/0 conn(0x7fcfe415d490 0x7fcfe40cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe9dee640 1 -- 172.21.15.107:0/760087996 >> v2:172.21.15.120:3300/0 conn(0x7fcfe415d490 msgr2=0x7fcfe40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe9dee640 1 --2- 172.21.15.107:0/760087996 >> v2:172.21.15.120:3300/0 conn(0x7fcfe415d490 0x7fcfe40cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfe9dee640 1 -- 172.21.15.107:0/760087996 >> v1:172.21.15.107:6789/0 conn(0x7fcfe40b63d0 legacy=0x7fcfe40d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.183+0000 7fcfdbfff640 1 -- 172.21.15.107:0/760087996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267291836 0 0) 0x7fcfe40cea30 con 0x7fcfe40b63d0 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.184+0000 7fcfe9dee640 1 -- 172.21.15.107:0/760087996 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40ce750 con 0x7fcfe40d3940 2023-11-24T08:06:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfe9dee640 1 --2- 172.21.15.107:0/760087996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe40d1250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40b52b0 tx=0x7fcfdc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c792d8b3d55ebaf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfdbfff640 1 -- 172.21.15.107:0/760087996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc02cc80 con 0x7fcfe40d3940 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfdbfff640 1 -- 172.21.15.107:0/760087996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfdc02cde0 con 0x7fcfe40d3940 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfdbfff640 1 -- 172.21.15.107:0/760087996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc034660 con 0x7fcfe40d3940 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfea5ef640 1 -- 172.21.15.107:0/760087996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 msgr2=0x7fcfe40d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.185+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/760087996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe40d1250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40b52b0 tx=0x7fcfdc02bed0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 -- 172.21.15.107:0/760087996 shutdown_connections 2023-11-24T08:06:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/760087996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe40d1250 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/760087996 >> v2:172.21.15.120:3300/0 conn(0x7fcfe415d490 0x7fcfe40cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 -- 172.21.15.107:0/760087996 >> 172.21.15.107:0/760087996 conn(0x7fcfe4061340 msgr2=0x7fcfe4061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 -- 172.21.15.107:0/760087996 shutdown_connections 2023-11-24T08:06:02.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.186+0000 7fcfea5ef640 1 -- 172.21.15.107:0/760087996 wait complete. 2023-11-24T08:06:02.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.187+0000 7fcfea5ef640 1 Processor -- start 2023-11-24T08:06:02.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.187+0000 7fcfea5ef640 1 -- start start 2023-11-24T08:06:02.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe415d490 0x7fcfe40c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40c9850 0x7fcfe40c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41600f0 con 0x7fcfe40d3940 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe415a9a0 con 0x7fcfe415d490 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.188+0000 7fcfea5ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe415f540 con 0x7fcfe40c9850 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53874/0 (socket says 172.21.15.107:53874) 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 -- 172.21.15.107:0/614834650 learned_addr learned my addr 172.21.15.107:0/614834650 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 -- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40c9850 msgr2=0x7fcfe40c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40c9850 0x7fcfe40c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 -- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe415d490 msgr2=0x7fcfe40c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe415d490 0x7fcfe40c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 -- 172.21.15.107:0/614834650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc009a70 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfe95ed640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000d620 tx=0x7fcfe000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.189+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe000b620 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.190+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe000b780 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.190+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe000bb30 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.190+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe4160580 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.190+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe415faa0 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.191+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe404e840 con 0x7fcfe40d3940 2023-11-24T08:06:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.192+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcfe000bc90 con 0x7fcfe40d3940 2023-11-24T08:06:02.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.193+0000 7fcfd9ffb640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 0x7fcfc806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:02.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.193+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fcfe00899a0 con 0x7fcfe40d3940 2023-11-24T08:06:02.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.193+0000 7fcfe8dec640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 0x7fcfc806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:02.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.194+0000 7fcfe8dec640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 0x7fcfc806d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4007fd0 tx=0x7fcfd4007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:02.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.198+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfe0054c20 con 0x7fcfe40d3940 2023-11-24T08:06:02.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.387+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcfe40b87d0 con 0x7fcfe40d3940 2023-11-24T08:06:02.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.968+0000 7fcfd9ffb640 1 -- 172.21.15.107:0/614834650 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcfe0058240 con 0x7fcfe40d3940 2023-11-24T08:06:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 msgr2=0x7fcfc806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 0x7fcfc806d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4007fd0 tx=0x7fcfd4007920 comp rx=0 tx=0).stop 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 msgr2=0x7fcfe415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000d620 tx=0x7fcfe000dae0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 shutdown_connections 2023-11-24T08:06:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc806ae40 0x7fcfc806d300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40c9850 0x7fcfe40c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe415d490 0x7fcfe40c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 --2- 172.21.15.107:0/614834650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe40d3940 0x7fcfe415a460 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 >> 172.21.15.107:0/614834650 conn(0x7fcfe4061340 msgr2=0x7fcfe40cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 shutdown_connections 2023-11-24T08:06:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:02.972+0000 7fcfea5ef640 1 -- 172.21.15.107:0/614834650 wait complete. 2023-11-24T08:06:03.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:06:03.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.205+0000 7f6e5c7dc640 1 Processor -- start 2023-11-24T08:06:03.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.205+0000 7f6e5c7dc640 1 -- start start 2023-11-24T08:06:03.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5c7dc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e540cc0f0 0x7f6e540c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5c7dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e5406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5c7dc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5415eea0 con 0x7f6e5415b440 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5c7dc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5415b820 con 0x7f6e540cc0f0 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5c7dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e540abf30 con 0x7f6e540c57e0 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5a578640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e5415b440 0x7f6e540cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48024/0 (socket says 172.21.15.107:48024) 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5a578640 1 -- 172.21.15.107:0/691972559 learned_addr learned my addr 172.21.15.107:0/691972559 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e59d77640 1 --2- 172.21.15.107:0/691972559 >> v2:172.21.15.120:3300/0 conn(0x7f6e540cc0f0 0x7f6e540c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.207+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e5406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637223419 0 0) 0x7f6e5415eea0 con 0x7f6e5415b440 2023-11-24T08:06:03.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40003520 con 0x7f6e5415b440 2023-11-24T08:06:03.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2131582222 0 0) 0x7f6e40003520 con 0x7f6e5415b440 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 msgr2=0x7f6e5406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e5406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 >> v2:172.21.15.120:3300/0 conn(0x7f6e540cc0f0 msgr2=0x7f6e540c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 --2- 172.21.15.107:0/691972559 >> v2:172.21.15.120:3300/0 conn(0x7f6e540cc0f0 0x7f6e540c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5415f0f0 con 0x7f6e5415b440 2023-11-24T08:06:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6e50001670 con 0x7f6e5415b440 2023-11-24T08:06:03.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 >> v1:172.21.15.107:6789/0 conn(0x7f6e5415b440 legacy=0x7f6e540cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e40003970 0x7f6e40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e40004d00 0x7f6e40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5415b820 con 0x7f6e40004d00 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e540abf30 con 0x7f6e40003970 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4000b980 con 0x7f6e540c57e0 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.208+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e40004d00 0x7f6e40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5a578640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e40003970 0x7f6e40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e59d77640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5ad79640 1 -- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 msgr2=0x7f6e400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5ad79640 1 -- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e40003970 msgr2=0x7f6e40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e40003970 0x7f6e40003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e5ad79640 1 -- 172.21.15.107:0/691972559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e40003520 con 0x7f6e40004d00 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.209+0000 7f6e59d77640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e540c57e0 0x7f6e400047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.210+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e40004d00 0x7f6e40005180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009fc0 tx=0x7f6e400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9ef705ea5dd378 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.210+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c02cc20 con 0x7f6e40004d00 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c00f040 con 0x7f6e40004d00 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e59576640 1 -- 172.21.15.107:0/691972559 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c034440 con 0x7f6e40004d00 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/691972559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e40004d00 msgr2=0x7f6e40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/691972559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e40004d00 0x7f6e40005180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009fc0 tx=0x7f6e400057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5ad79640 1 -- 172.21.15.107:0/691972559 reap_dead start 2023-11-24T08:06:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/691972559 shutdown_connections 2023-11-24T08:06:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/691972559 >> 172.21.15.107:0/691972559 conn(0x7f6e540b0c50 msgr2=0x7f6e540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/691972559 shutdown_connections 2023-11-24T08:06:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.211+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/691972559 wait complete. 2023-11-24T08:06:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.213+0000 7f6e5c7dc640 1 Processor -- start 2023-11-24T08:06:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.213+0000 7f6e5c7dc640 1 -- start start 2023-11-24T08:06:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e40004d00 0x7f6e541597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 0x7f6e540c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54158730 con 0x7f6e540c67f0 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54158a10 con 0x7f6e54158170 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5c7dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54158cf0 con 0x7f6e40004d00 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5ad79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.214+0000 7f6e5ad79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53900/0 (socket says 172.21.15.107:53900) 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 -- 172.21.15.107:0/3064149613 learned_addr learned my addr 172.21.15.107:0/3064149613 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e59d77640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 0x7f6e540c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5a578640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e40004d00 0x7f6e541597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 -- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e40004d00 msgr2=0x7f6e541597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e40004d00 0x7f6e541597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 -- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 msgr2=0x7f6e540c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 0x7f6e540c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 -- 172.21.15.107:0/3064149613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54158fd0 con 0x7f6e540c67f0 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e59d77640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 0x7f6e540c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.215+0000 7f6e5ad79640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c002230 tx=0x7f6e4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.216+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c043070 con 0x7f6e540c67f0 2023-11-24T08:06:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.216+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c034ad0 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.216+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c034e80 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.216+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e541592b0 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.216+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e54159ef0 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.217+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5415f670 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.218+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6e4c003990 con 0x7f6e540c67f0 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.220+0000 7f6e4affd640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 0x7f6e3806d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.220+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6e4c0ab9f0 con 0x7f6e540c67f0 2023-11-24T08:06:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.220+0000 7f6e5a578640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 0x7f6e3806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.221+0000 7f6e5a578640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 0x7f6e3806d290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6e50000f60 tx=0x7f6e50000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.224+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e4c076bf0 con 0x7f6e540c67f0 2023-11-24T08:06:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.444+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6e541594a0 con 0x7f6e540c67f0 2023-11-24T08:06:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.972+0000 7f6e4affd640 1 -- 172.21.15.107:0/3064149613 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6e4c039020 con 0x7f6e540c67f0 2023-11-24T08:06:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 4 and data pool 5 2023-11-24T08:06:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.977+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 msgr2=0x7f6e3806d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.977+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 0x7f6e3806d290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6e50000f60 tx=0x7f6e50000ed0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.977+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 msgr2=0x7f6e541455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.977+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c002230 tx=0x7f6e4c009750 comp rx=0 tx=0).stop 2023-11-24T08:06:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 shutdown_connections 2023-11-24T08:06:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e3806add0 0x7f6e3806d290 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e540c67f0 0x7f6e541455e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e54158170 0x7f6e540c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 --2- 172.21.15.107:0/3064149613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e40004d00 0x7f6e541597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.978+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 >> 172.21.15.107:0/3064149613 conn(0x7f6e540b0c50 msgr2=0x7f6e540cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:03.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.979+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 shutdown_connections 2023-11-24T08:06:03.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:03.980+0000 7f6e5c7dc640 1 -- 172.21.15.107:0/3064149613 wait complete. 2023-11-24T08:06:04.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:06:04.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.199+0000 7f325e42b640 1 Processor -- start 2023-11-24T08:06:04.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.200+0000 7f325e42b640 1 -- start start 2023-11-24T08:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f325e42b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f325815c7f0 0x7f325806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f325e42b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f32580cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f325e42b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32580cece0 con 0x7f32580cf450 2023-11-24T08:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f325e42b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32580cfe40 con 0x7f325815c7f0 2023-11-24T08:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f325e42b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3258156010 con 0x7f32580ce6c0 2023-11-24T08:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f3257fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f325815c7f0 0x7f325806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f3257fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f325815c7f0 0x7f325806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53914/0 (socket says 172.21.15.107:53914) 2023-11-24T08:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f3257fff640 1 -- 172.21.15.107:0/1056460246 learned_addr learned my addr 172.21.15.107:0/1056460246 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.201+0000 7f324ffff640 1 --1- 172.21.15.107:0/1056460246 >> v1:172.21.15.107:6789/0 conn(0x7f32580cf450 0x7f32580cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48038/0 (socket says 172.21.15.107:48038) 2023-11-24T08:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f325c9c8640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f32580cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f3257fff640 1 -- 172.21.15.107:0/1056460246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 msgr2=0x7f32580cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f3257fff640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f32580cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f3257fff640 1 -- 172.21.15.107:0/1056460246 >> v1:172.21.15.107:6789/0 conn(0x7f32580cf450 legacy=0x7f32580cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f3257fff640 1 -- 172.21.15.107:0/1056460246 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325806d040 con 0x7f325815c7f0 2023-11-24T08:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.202+0000 7f32577fe640 1 -- 172.21.15.107:0/1056460246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523207664 0 0) 0x7f32580cece0 con 0x7f32580cf450 2023-11-24T08:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.203+0000 7f3257fff640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f325806db20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3248009870 tx=0x7f324802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eafddfc3d6f7627 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.203+0000 7f32577fe640 1 -- 172.21.15.107:0/1056460246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248030840 con 0x7f325815c7f0 2023-11-24T08:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.203+0000 7f32577fe640 1 -- 172.21.15.107:0/1056460246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32480309a0 con 0x7f325815c7f0 2023-11-24T08:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.203+0000 7f32577fe640 1 -- 172.21.15.107:0/1056460246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248030d30 con 0x7f325815c7f0 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 -- 172.21.15.107:0/1056460246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 msgr2=0x7f325806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f325806db20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3248009870 tx=0x7f324802b980 comp rx=0 tx=0).stop 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 -- 172.21.15.107:0/1056460246 shutdown_connections 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f32580cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 --2- 172.21.15.107:0/1056460246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f325806db20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 -- 172.21.15.107:0/1056460246 >> 172.21.15.107:0/1056460246 conn(0x7f325806a920 msgr2=0x7f325806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 -- 172.21.15.107:0/1056460246 shutdown_connections 2023-11-24T08:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.204+0000 7f325e42b640 1 -- 172.21.15.107:0/1056460246 wait complete. 2023-11-24T08:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.206+0000 7f325e42b640 1 Processor -- start 2023-11-24T08:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.206+0000 7f325e42b640 1 -- start start 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f3258149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580cf450 0x7f32580c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258149ab0 con 0x7f325815c7f0 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32580abfc0 con 0x7f32580cf450 2023-11-24T08:06:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325e42b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32581552b0 con 0x7f32580ce6c0 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53916/0 (socket says 172.21.15.107:53916) 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 -- 172.21.15.107:0/2166855767 learned_addr learned my addr 172.21.15.107:0/2166855767 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 -- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 msgr2=0x7f3258149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f3258149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 -- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580cf450 msgr2=0x7f32580c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.207+0000 7f325c9c8640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580cf450 0x7f32580c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f325c9c8640 1 -- 172.21.15.107:0/2166855767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480094d0 con 0x7f325815c7f0 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f325c9c8640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3250007d90 tx=0x7f325000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250014420 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500145a0 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250014950 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325815b160 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.208+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32580ac1d0 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.209+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3258067470 con 0x7f325815c7f0 2023-11-24T08:06:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.210+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3250014ab0 con 0x7f325815c7f0 2023-11-24T08:06:04.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.212+0000 7f32557fa640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 0x7f323806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.212+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3250089e70 con 0x7f325815c7f0 2023-11-24T08:06:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.212+0000 7f3257fff640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 0x7f323806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.212+0000 7f3257fff640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 0x7f323806d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3248036000 tx=0x7f324803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.216+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3250054fb0 con 0x7f325815c7f0 2023-11-24T08:06:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.403+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32580cd730 con 0x7f325815c7f0 2023-11-24T08:06:04.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.406+0000 7f32557fa640 1 -- 172.21.15.107:0/2166855767 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f32500585d0 con 0x7f325815c7f0 2023-11-24T08:06:04.408 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:04.409 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:06:03.446459+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:06:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 msgr2=0x7f323806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 0x7f323806d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3248036000 tx=0x7f324803b040 comp rx=0 tx=0).stop 2023-11-24T08:06:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 msgr2=0x7f32580c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3250007d90 tx=0x7f325000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 shutdown_connections 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f323806ae70 0x7f323806d330 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815c7f0 0x7f32580c8d70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580cf450 0x7f32580c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 --2- 172.21.15.107:0/2166855767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580ce6c0 0x7f3258149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 >> 172.21.15.107:0/2166855767 conn(0x7f325806a920 msgr2=0x7f32580b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 shutdown_connections 2023-11-24T08:06:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.409+0000 7f325e42b640 1 -- 172.21.15.107:0/2166855767 wait complete. 2023-11-24T08:06:04.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:06:04.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.632+0000 7f142b91d640 1 Processor -- start 2023-11-24T08:06:04.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.632+0000 7f142b91d640 1 -- start start 2023-11-24T08:06:04.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.633+0000 7f142b91d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14240cf5d0 0x7f142406d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.633+0000 7f142b91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 0x7f142406b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.633+0000 7f142b91d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142415e8b0 con 0x7f142415af40 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.633+0000 7f142b91d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142415eb90 con 0x7f14240cf5d0 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.633+0000 7f142b91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142415e6a0 con 0x7f142415b320 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f14296b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f142415af40 0x7f14240ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48040/0 (socket says 172.21.15.107:48040) 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f14296b9640 1 -- 172.21.15.107:0/558977330 learned_addr learned my addr 172.21.15.107:0/558977330 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1428eb8640 1 --2- 172.21.15.107:0/558977330 >> v2:172.21.15.107:3300/0 conn(0x7f14240cf5d0 0x7f142406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1429eba640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 0x7f142406b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1428eb8640 1 -- 172.21.15.107:0/558977330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 msgr2=0x7f142406b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1428eb8640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 0x7f142406b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1428eb8640 1 -- 172.21.15.107:0/558977330 >> v1:172.21.15.107:6789/0 conn(0x7f142415af40 legacy=0x7f14240ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f141bfff640 1 -- 172.21.15.107:0/558977330 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887389266 0 0) 0x7f142415e8b0 con 0x7f142415af40 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1428eb8640 1 -- 172.21.15.107:0/558977330 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424070180 con 0x7f14240cf5d0 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.634+0000 7f1429eba640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 0x7f142406b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.635+0000 7f1428eb8640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142406d220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14140097a0 tx=0x7f141402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5656a565cbb060a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f141bfff640 1 -- 172.21.15.107:0/558977330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414031830 con 0x7f14240cf5d0 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f141bfff640 1 -- 172.21.15.107:0/558977330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414031990 con 0x7f14240cf5d0 2023-11-24T08:06:04.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f141bfff640 1 -- 172.21.15.107:0/558977330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414031d20 con 0x7f14240cf5d0 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 -- 172.21.15.107:0/558977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 msgr2=0x7f142406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142406d220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14140097a0 tx=0x7f141402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 -- 172.21.15.107:0/558977330 shutdown_connections 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415b320 0x7f142406b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 --2- 172.21.15.107:0/558977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142406d220 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 -- 172.21.15.107:0/558977330 >> 172.21.15.107:0/558977330 conn(0x7f142406e7f0 msgr2=0x7f142406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 -- 172.21.15.107:0/558977330 shutdown_connections 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.636+0000 7f142b91d640 1 -- 172.21.15.107:0/558977330 wait complete. 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.638+0000 7f142b91d640 1 Processor -- start 2023-11-24T08:06:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.638+0000 7f142b91d640 1 -- start start 2023-11-24T08:06:04.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415af40 0x7f14240b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f142415b320 0x7f14240c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142414d740 con 0x7f14240cf5d0 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424148020 con 0x7f142415b320 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f142b91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1424148300 con 0x7f142415af40 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f14296b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f14296b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53946/0 (socket says 172.21.15.107:53946) 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.639+0000 7f14296b9640 1 -- 172.21.15.107:0/1576084454 learned_addr learned my addr 172.21.15.107:0/1576084454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 -- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415af40 msgr2=0x7f14240b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415af40 0x7f14240b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 -- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f142415b320 msgr2=0x7f14240c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f142415b320 0x7f14240c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 -- 172.21.15.107:0/1576084454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009400 con 0x7f14240cf5d0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f14296b9640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f142000c4d0 tx=0x7f142000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14200074e0 con 0x7f14240cf5d0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1420007660 con 0x7f14240cf5d0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14241485e0 con 0x7f14240cf5d0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420004a20 con 0x7f14240cf5d0 2023-11-24T08:06:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.640+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1424148ba0 con 0x7f14240cf5d0 2023-11-24T08:06:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.641+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1424148d90 con 0x7f14240cf5d0 2023-11-24T08:06:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.642+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1420004b80 con 0x7f14240cf5d0 2023-11-24T08:06:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.643+0000 7f1419ffb640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 0x7f140806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.643+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1420088830 con 0x7f14240cf5d0 2023-11-24T08:06:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.643+0000 7f1428eb8640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 0x7f140806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:04.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.644+0000 7f1428eb8640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 0x7f140806d300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1414036000 tx=0x7f1414030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:04.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.647+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1420053850 con 0x7f14240cf5d0 2023-11-24T08:06:04.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.838+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1424067470 con 0x7f14240cf5d0 2023-11-24T08:06:04.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.839+0000 7f1419ffb640 1 -- 172.21.15.107:0/1576084454 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f1420018070 con 0x7f14240cf5d0 2023-11-24T08:06:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:04.842 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:06:03.446459+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:06:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 msgr2=0x7f140806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 0x7f140806d300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1414036000 tx=0x7f1414030000 comp rx=0 tx=0).stop 2023-11-24T08:06:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 msgr2=0x7f142414d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f142000c4d0 tx=0x7f142000c990 comp rx=0 tx=0).stop 2023-11-24T08:06:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 shutdown_connections 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f140806ae40 0x7f140806d300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f142415b320 0x7f14240c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f142415af40 0x7f14240b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 --2- 172.21.15.107:0/1576084454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14240cf5d0 0x7f142414d200 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 >> 172.21.15.107:0/1576084454 conn(0x7f142406e7f0 msgr2=0x7f142406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 shutdown_connections 2023-11-24T08:06:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:04.842+0000 7f142b91d640 1 -- 172.21.15.107:0/1576084454 wait complete. 2023-11-24T08:06:04.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:06:04.925 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:06:04.923+0000 7fe04a483640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:06:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.073+0000 7f01c5797640 1 Processor -- start 2023-11-24T08:06:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.073+0000 7f01c5797640 1 -- start start 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.074+0000 7f01c5797640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f01c00cfb90 0x7f01c0161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01c5797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c0161fa0 0x7f01c006ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01c5797640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c01612f0 con 0x7f01c00b2400 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01c5797640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c00cff70 con 0x7f01c00cfb90 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01c5797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00afb20 con 0x7f01c0161fa0 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01bf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f01c00b2400 0x7f01c00cc940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48044/0 (socket says 172.21.15.107:48044) 2023-11-24T08:06:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01bf7fe640 1 -- 172.21.15.107:0/2196612978 learned_addr learned my addr 172.21.15.107:0/2196612978 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01beffd640 1 --2- 172.21.15.107:0/2196612978 >> v2:172.21.15.120:3300/0 conn(0x7f01c00cfb90 0x7f01c0161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504808311 0 0) 0x7f01c01612f0 con 0x7f01c00b2400 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ac003520 con 0x7f01c00b2400 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.075+0000 7f01be7fc640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c0161fa0 0x7f01c006ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862083014 0 0) 0x7f01ac003520 con 0x7f01c00b2400 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c0161fa0 msgr2=0x7f01c006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c0161fa0 0x7f01c006ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 >> v2:172.21.15.120:3300/0 conn(0x7f01c00cfb90 msgr2=0x7f01c0161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 --2- 172.21.15.107:0/2196612978 >> v2:172.21.15.120:3300/0 conn(0x7f01c00cfb90 0x7f01c0161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c006a850 con 0x7f01c00b2400 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f01b8001670 con 0x7f01c00b2400 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 >> v1:172.21.15.107:6789/0 conn(0x7f01c00b2400 legacy=0x7f01c00cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01ac0047c0 0x7f01ac004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01ac005140 0x7f01ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00cff70 con 0x7f01ac0039a0 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00afb20 con 0x7f01ac005140 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac00bec0 con 0x7f01ac0047c0 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01beffd640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01be7fc640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01ac0047c0 0x7f01ac004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01beffd640 1 -- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01ac0047c0 msgr2=0x7f01ac004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.076+0000 7f01beffd640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01ac0047c0 0x7f01ac004c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.077+0000 7f01beffd640 1 -- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01ac005140 msgr2=0x7f01ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.077+0000 7f01beffd640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01ac005140 0x7f01ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.077+0000 7f01beffd640 1 -- 172.21.15.107:0/2196612978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac003520 con 0x7f01ac0039a0 2023-11-24T08:06:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01beffd640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f01b000e170 tx=0x7f01ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b672d43b83b900 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b002cc20 con 0x7f01ac0039a0 2023-11-24T08:06:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b000f040 con 0x7f01ac0039a0 2023-11-24T08:06:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01bdffb640 1 -- 172.21.15.107:0/2196612978 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0034440 con 0x7f01ac0039a0 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 -- 172.21.15.107:0/2196612978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 msgr2=0x7f01ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 --2- 172.21.15.107:0/2196612978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01ac003dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f01b000e170 tx=0x7f01ac0057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01bf7fe640 1 -- 172.21.15.107:0/2196612978 reap_dead start 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 -- 172.21.15.107:0/2196612978 shutdown_connections 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 -- 172.21.15.107:0/2196612978 >> 172.21.15.107:0/2196612978 conn(0x7f01c00999d0 msgr2=0x7f01c00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 -- 172.21.15.107:0/2196612978 shutdown_connections 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.078+0000 7f01c5797640 1 -- 172.21.15.107:0/2196612978 wait complete. 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.080+0000 7f01c5797640 1 Processor -- start 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.080+0000 7f01c5797640 1 -- start start 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c01596c0 0x7f01c00c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01c00c77c0 0x7f01c00b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c8cc0 con 0x7f01ac0039a0 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0159b40 con 0x7f01c00c77c0 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01c5797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c8040 con 0x7f01c01596c0 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53970/0 (socket says 172.21.15.107:53970) 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.081+0000 7f01beffd640 1 -- 172.21.15.107:0/1385989397 learned_addr learned my addr 172.21.15.107:0/1385989397 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 -- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c01596c0 msgr2=0x7f01c00c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c01596c0 0x7f01c00c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 -- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01c00c77c0 msgr2=0x7f01c00b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01c00c77c0 0x7f01c00b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 -- 172.21.15.107:0/1385989397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00c8320 con 0x7f01ac0039a0 2023-11-24T08:06:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01beffd640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f01b0037040 tx=0x7f01b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0040450 con 0x7f01ac0039a0 2023-11-24T08:06:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00405b0 con 0x7f01ac0039a0 2023-11-24T08:06:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c00c8600 con 0x7f01ac0039a0 2023-11-24T08:06:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0040960 con 0x7f01ac0039a0 2023-11-24T08:06:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.082+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c00c7c70 con 0x7f01ac0039a0 2023-11-24T08:06:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.083+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c00c8f30 con 0x7f01ac0039a0 2023-11-24T08:06:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.084+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f01b0040ac0 con 0x7f01ac0039a0 2023-11-24T08:06:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.085+0000 7f01b77fe640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 0x7f01a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.085+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01b00abee0 con 0x7f01ac0039a0 2023-11-24T08:06:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.085+0000 7f01be7fc640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 0x7f01a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.086+0000 7f01be7fc640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 0x7f01a006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006a90 tx=0x7f01a8006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.090+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01b0076ef0 con 0x7f01ac0039a0 2023-11-24T08:06:05.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:05.302+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f01c004e8b0 con 0x7f01ac0039a0 2023-11-24T08:06:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.015+0000 7f01b77fe640 1 -- 172.21.15.107:0/1385989397 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f01b007a510 con 0x7f01ac0039a0 2023-11-24T08:06:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 msgr2=0x7f01a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 0x7f01a006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006a90 tx=0x7f01a8006a20 comp rx=0 tx=0).stop 2023-11-24T08:06:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 msgr2=0x7f01c00c90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f01b0037040 tx=0x7f01b0009750 comp rx=0 tx=0).stop 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 shutdown_connections 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01a006ae70 0x7f01a006d330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01c00c77c0 0x7f01c00b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01c01596c0 0x7f01c00c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 --2- 172.21.15.107:0/1385989397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01ac0039a0 0x7f01c00c90e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 >> 172.21.15.107:0/1385989397 conn(0x7f01c00999d0 msgr2=0x7f01c015c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 shutdown_connections 2023-11-24T08:06:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.019+0000 7f01c5797640 1 -- 172.21.15.107:0/1385989397 wait complete. 2023-11-24T08:06:06.041 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:06:06.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:06:06.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.257+0000 7ff0ff59e640 1 Processor -- start 2023-11-24T08:06:06.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.258+0000 7ff0ff59e640 1 -- start start 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1001608f0 0x7ff1000d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce260 0x7ff1000d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1000ce710 0x7ff1000d36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000b8f60 con 0x7ff1001608f0 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff100160cd0 con 0x7ff1000ce710 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0ff59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000cdcb0 con 0x7ff1000ce260 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0fe59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1001608f0 0x7ff1000d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0fe59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1001608f0 0x7ff1000d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53972/0 (socket says 172.21.15.107:53972) 2023-11-24T08:06:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0fe59c640 1 -- 172.21.15.107:0/3009708230 learned_addr learned my addr 172.21.15.107:0/3009708230 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:06.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.259+0000 7ff0fdd9b640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce260 0x7ff1000d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fed9d640 1 --2- 172.21.15.107:0/3009708230 >> v2:172.21.15.120:3300/0 conn(0x7ff1000ce710 0x7ff1000d36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fe59c640 1 -- 172.21.15.107:0/3009708230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce260 msgr2=0x7ff1000d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce260 0x7ff1000d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fe59c640 1 -- 172.21.15.107:0/3009708230 >> v2:172.21.15.120:3300/0 conn(0x7ff1000ce710 msgr2=0x7ff1000d36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/3009708230 >> v2:172.21.15.120:3300/0 conn(0x7ff1000ce710 0x7ff1000d36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.260+0000 7ff0fe59c640 1 -- 172.21.15.107:0/3009708230 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000cdf90 con 0x7ff1001608f0 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.261+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000d1ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1000d3e60 tx=0x7ff0f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40074074381545ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0fd59a640 1 -- 172.21.15.107:0/3009708230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f002ca70 con 0x7ff1001608f0 2023-11-24T08:06:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0fd59a640 1 -- 172.21.15.107:0/3009708230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f002cbd0 con 0x7ff1001608f0 2023-11-24T08:06:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0fd59a640 1 -- 172.21.15.107:0/3009708230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00343f0 con 0x7ff1001608f0 2023-11-24T08:06:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 -- 172.21.15.107:0/3009708230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 msgr2=0x7ff1000d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000d1ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1000d3e60 tx=0x7ff0f002bed0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 -- 172.21.15.107:0/3009708230 shutdown_connections 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/3009708230 >> v2:172.21.15.120:3300/0 conn(0x7ff1000ce710 0x7ff1000d36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce260 0x7ff1000d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/3009708230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000d1ac0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 -- 172.21.15.107:0/3009708230 >> 172.21.15.107:0/3009708230 conn(0x7ff100061340 msgr2=0x7ff100061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 -- 172.21.15.107:0/3009708230 shutdown_connections 2023-11-24T08:06:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.262+0000 7ff0ff59e640 1 -- 172.21.15.107:0/3009708230 wait complete. 2023-11-24T08:06:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.264+0000 7ff0ff59e640 1 Processor -- start 2023-11-24T08:06:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.264+0000 7ff0ff59e640 1 -- start start 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 0x7ff10014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce710 0x7ff1000c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000c69c0 con 0x7ff1001608f0 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000c6ca0 con 0x7ff1000ce260 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.265+0000 7ff0ff59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000c6f80 con 0x7ff1000ce710 2023-11-24T08:06:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53988/0 (socket says 172.21.15.107:53988) 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 -- 172.21.15.107:0/1091165082 learned_addr learned my addr 172.21.15.107:0/1091165082 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fdd9b640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce710 0x7ff1000c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 0x7ff10014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 -- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce710 msgr2=0x7ff1000c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce710 0x7ff1000c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 -- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 msgr2=0x7ff10014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 0x7ff10014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 -- 172.21.15.107:0/1091165082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f0009b30 con 0x7ff1001608f0 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 0x7ff10014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.266+0000 7ff0fed9d640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800d800 tx=0x7ff0f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.267+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f800b6c0 con 0x7ff1001608f0 2023-11-24T08:06:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.267+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f800b840 con 0x7ff1001608f0 2023-11-24T08:06:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.267+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1000c7260 con 0x7ff1001608f0 2023-11-24T08:06:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.267+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f800bbf0 con 0x7ff1001608f0 2023-11-24T08:06:06.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.267+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1000c7820 con 0x7ff1001608f0 2023-11-24T08:06:06.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.268+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10004e840 con 0x7ff1001608f0 2023-11-24T08:06:06.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.269+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff0f800bd50 con 0x7ff1001608f0 2023-11-24T08:06:06.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.270+0000 7ff0f6ffd640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 0x7ff0e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:06.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.270+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0f8089b00 con 0x7ff1001608f0 2023-11-24T08:06:06.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.271+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 0x7ff0e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:06.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.271+0000 7ff0fe59c640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 0x7ff0e006d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0035000 tx=0x7ff0f0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:06.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.275+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0f8054c40 con 0x7ff1001608f0 2023-11-24T08:06:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:06.502+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff10015e100 con 0x7ff1001608f0 2023-11-24T08:06:07.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.019+0000 7ff0f6ffd640 1 -- 172.21.15.107:0/1091165082 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff0f8058260 con 0x7ff1001608f0 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 msgr2=0x7ff0e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 0x7ff0e006d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0035000 tx=0x7ff0f0030000 comp rx=0 tx=0).stop 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 msgr2=0x7ff1000cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800d800 tx=0x7ff0f800dcc0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 shutdown_connections 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0e006ae40 0x7ff0e006d300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1001608f0 0x7ff1000cc710 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1000ce710 0x7ff1000c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 --2- 172.21.15.107:0/1091165082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000ce260 0x7ff10014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 >> 172.21.15.107:0/1091165082 conn(0x7ff100061340 msgr2=0x7ff1000d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 shutdown_connections 2023-11-24T08:06:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.023+0000 7ff0ff59e640 1 -- 172.21.15.107:0/1091165082 wait complete. 2023-11-24T08:06:07.045 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:06:07.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:06:07.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.256+0000 7f21359ec640 1 Processor -- start 2023-11-24T08:06:07.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.256+0000 7f21359ec640 1 -- start start 2023-11-24T08:06:07.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f21359ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21300cfb60 0x7f21300abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f21359ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300ce640 0x7f21300b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f21359ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300ce270 con 0x7f21300cc470 2023-11-24T08:06:07.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f21359ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213006c560 con 0x7f21300cfb60 2023-11-24T08:06:07.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f21359ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213006d0a0 con 0x7f21300ce640 2023-11-24T08:06:07.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f2127fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21300cfb60 0x7f21300abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f2127fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21300cfb60 0x7f21300abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54004/0 (socket says 172.21.15.107:54004) 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.258+0000 7f2127fff640 1 -- 172.21.15.107:0/4277539054 learned_addr learned my addr 172.21.15.107:0/4277539054 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.259+0000 7f212f7fe640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300ce640 0x7f21300b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.259+0000 7f2127fff640 1 -- 172.21.15.107:0/4277539054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300ce640 msgr2=0x7f21300b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.259+0000 7f2127fff640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300ce640 0x7f21300b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.259+0000 7f2127fff640 1 -- 172.21.15.107:0/4277539054 >> v1:172.21.15.107:6789/0 conn(0x7f21300cc470 legacy=0x7f21300cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.259+0000 7f2127fff640 1 -- 172.21.15.107:0/4277539054 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21300b4200 con 0x7f21300cfb60 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.260+0000 7f2127fff640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f21300abfc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2118009870 tx=0x7f211802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bbaa069a43e621 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.260+0000 7f212e7fc640 1 -- 172.21.15.107:0/4277539054 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2118030840 con 0x7f21300cfb60 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.260+0000 7f212e7fc640 1 -- 172.21.15.107:0/4277539054 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21180309a0 con 0x7f21300cfb60 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.260+0000 7f212e7fc640 1 -- 172.21.15.107:0/4277539054 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2118030d30 con 0x7f21300cfb60 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 -- 172.21.15.107:0/4277539054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 msgr2=0x7f21300abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f21300abfc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2118009870 tx=0x7f211802b980 comp rx=0 tx=0).stop 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 -- 172.21.15.107:0/4277539054 shutdown_connections 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300ce640 0x7f21300b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 --2- 172.21.15.107:0/4277539054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f21300abfc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 -- 172.21.15.107:0/4277539054 >> 172.21.15.107:0/4277539054 conn(0x7f213006a920 msgr2=0x7f213006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 -- 172.21.15.107:0/4277539054 shutdown_connections 2023-11-24T08:06:07.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.261+0000 7f21359ec640 1 -- 172.21.15.107:0/4277539054 wait complete. 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.263+0000 7f21359ec640 1 Processor -- start 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.264+0000 7f21359ec640 1 -- start start 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300cc470 0x7f21301641f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21300ce640 0x7f2130164b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2130162af0 con 0x7f21300cfb60 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21301650c0 con 0x7f21300ce640 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f21359ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21301648f0 con 0x7f21300cc470 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54008/0 (socket says 172.21.15.107:54008) 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 -- 172.21.15.107:0/1788191741 learned_addr learned my addr 172.21.15.107:0/1788191741 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:07.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 -- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300cc470 msgr2=0x7f21301641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300cc470 0x7f21301641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 -- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21300ce640 msgr2=0x7f2130164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21300ce640 0x7f2130164b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.265+0000 7f212f7fe640 1 -- 172.21.15.107:0/1788191741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21180094d0 con 0x7f21300cfb60 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f212f7fe640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2128007dc0 tx=0x7f212800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128014420 con 0x7f21300cfb60 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21280145a0 con 0x7f21300cfb60 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2130162ce0 con 0x7f21300cfb60 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128014950 con 0x7f21300cfb60 2023-11-24T08:06:07.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.266+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2130162f80 con 0x7f21300cfb60 2023-11-24T08:06:07.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.267+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2130067470 con 0x7f21300cfb60 2023-11-24T08:06:07.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.268+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2128014ab0 con 0x7f21300cfb60 2023-11-24T08:06:07.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.269+0000 7f212cff9640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 0x7f211006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.269+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2128089fb0 con 0x7f21300cfb60 2023-11-24T08:06:07.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.269+0000 7f212effd640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 0x7f211006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.270+0000 7f212effd640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 0x7f211006d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2120000f60 tx=0x7f21200079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.273+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2128054fc0 con 0x7f21300cfb60 2023-11-24T08:06:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.497+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21300cd910 con 0x7f21300cfb60 2023-11-24T08:06:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.498+0000 7f212cff9640 1 -- 172.21.15.107:0/1788191741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f21280585e0 con 0x7f21300cfb60 2023-11-24T08:06:07.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:07.500 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:06:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 14 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.500+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 msgr2=0x7f211006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 0x7f211006d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2120000f60 tx=0x7f21200079b0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 msgr2=0x7f2130156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2128007dc0 tx=0x7f212800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 shutdown_connections 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f211006ae40 0x7f211006d300 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21300cfb60 0x7f2130156590 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21300ce640 0x7f2130164b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 --2- 172.21.15.107:0/1788191741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21300cc470 0x7f21301641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 >> 172.21.15.107:0/1788191741 conn(0x7f213006a920 msgr2=0x7f21300ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 shutdown_connections 2023-11-24T08:06:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.501+0000 7f21359ec640 1 -- 172.21.15.107:0/1788191741 wait complete. 2023-11-24T08:06:07.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:06:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.725+0000 7f2719161640 1 Processor -- start 2023-11-24T08:06:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.725+0000 7f2719161640 1 -- start start 2023-11-24T08:06:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.726+0000 7f2719161640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 0x7f271415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.726+0000 7f2719161640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27140b2a70 0x7f271415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.726+0000 7f2719161640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271415ab80 con 0x7f27140b2630 2023-11-24T08:06:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.726+0000 7f2719161640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271415a990 con 0x7f27140b2a70 2023-11-24T08:06:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.726+0000 7f2719161640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271415e680 con 0x7f27140b2250 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27140b2a70 0x7f271415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27140b2a70 0x7f271415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54014/0 (socket says 172.21.15.107:54014) 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 -- 172.21.15.107:0/3879134141 learned_addr learned my addr 172.21.15.107:0/3879134141 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f2712d9d640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 0x7f271415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 -- 172.21.15.107:0/3879134141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 msgr2=0x7f271415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 0x7f271415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 -- 172.21.15.107:0/3879134141 >> v1:172.21.15.120:6789/0 conn(0x7f27140b2630 legacy=0x7f27140ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f271359e640 1 -- 172.21.15.107:0/3879134141 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271415e960 con 0x7f27140b2a70 2023-11-24T08:06:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.727+0000 7f2712d9d640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 0x7f271415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.728+0000 7f271359e640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f271415b080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f270c009e50 tx=0x7f270c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa226231e6392dd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2711d9b640 1 -- 172.21.15.107:0/3879134141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c02ca70 con 0x7f27140b2a70 2023-11-24T08:06:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2711d9b640 1 -- 172.21.15.107:0/3879134141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c02cbd0 con 0x7f27140b2a70 2023-11-24T08:06:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2711d9b640 1 -- 172.21.15.107:0/3879134141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c0343f0 con 0x7f27140b2a70 2023-11-24T08:06:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 -- 172.21.15.107:0/3879134141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 msgr2=0x7f271415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f271415b080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f270c009e50 tx=0x7f270c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 -- 172.21.15.107:0/3879134141 shutdown_connections 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f271415b080 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 --2- 172.21.15.107:0/3879134141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2250 0x7f271415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 -- 172.21.15.107:0/3879134141 >> 172.21.15.107:0/3879134141 conn(0x7f271406a920 msgr2=0x7f271406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 -- 172.21.15.107:0/3879134141 shutdown_connections 2023-11-24T08:06:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.729+0000 7f2719161640 1 -- 172.21.15.107:0/3879134141 wait complete. 2023-11-24T08:06:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.731+0000 7f2719161640 1 Processor -- start 2023-11-24T08:06:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.731+0000 7f2719161640 1 -- start start 2023-11-24T08:06:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27140b2250 0x7f27140b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2630 0x7f2714072390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27140728d0 con 0x7f27140b2a70 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2714071610 con 0x7f27140b2250 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f2719161640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27140cd2f0 con 0x7f27140b2630 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f271359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f271359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54016/0 (socket says 172.21.15.107:54016) 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.732+0000 7f271359e640 1 -- 172.21.15.107:0/703348819 learned_addr learned my addr 172.21.15.107:0/703348819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 -- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2630 msgr2=0x7f2714072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2630 0x7f2714072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 -- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27140b2250 msgr2=0x7f27140b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27140b2250 0x7f27140b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 -- 172.21.15.107:0/703348819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c009b30 con 0x7f27140b2a70 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f271359e640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f270c036040 tx=0x7f270c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c009510 con 0x7f27140b2a70 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c03e450 con 0x7f27140b2a70 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c03e800 con 0x7f27140b2a70 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27140cd7c0 con 0x7f27140b2a70 2023-11-24T08:06:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.733+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2714071b90 con 0x7f27140b2a70 2023-11-24T08:06:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.735+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2714067470 con 0x7f27140b2a70 2023-11-24T08:06:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.735+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f270c03e960 con 0x7f27140b2a70 2023-11-24T08:06:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.736+0000 7f26fb7fe640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 0x7f270006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.736+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f270c0ac3d0 con 0x7f27140b2a70 2023-11-24T08:06:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.737+0000 7f2712d9d640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 0x7f270006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.737+0000 7f2712d9d640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 0x7f270006d300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2704006fd0 tx=0x7f2704008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.741+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f270c077510 con 0x7f27140b2a70 2023-11-24T08:06:07.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.919+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27140cea00 con 0x7f27140b2a70 2023-11-24T08:06:07.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.922+0000 7f26fb7fe640 1 -- 172.21.15.107:0/703348819 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f270c07ab30 con 0x7f27140b2a70 2023-11-24T08:06:07.924 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:07.924 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":34,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:06:04.988628+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:06:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 msgr2=0x7f270006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 0x7f270006d300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2704006fd0 tx=0x7f2704008040 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 msgr2=0x7f27140987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f270c036040 tx=0x7f270c004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 shutdown_connections 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270006ae40 0x7f270006d300 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27140b2a70 0x7f27140987f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27140b2630 0x7f2714072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 --2- 172.21.15.107:0/703348819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27140b2250 0x7f27140b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.924+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 >> 172.21.15.107:0/703348819 conn(0x7f271406a920 msgr2=0x7f271414ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.925+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 shutdown_connections 2023-11-24T08:06:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:07.925+0000 7f2719161640 1 -- 172.21.15.107:0/703348819 wait complete. 2023-11-24T08:06:07.945 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:06:07.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:06:08.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.132+0000 7ff248b7d640 1 Processor -- start 2023-11-24T08:06:08.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.132+0000 7ff248b7d640 1 -- start start 2023-11-24T08:06:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.133+0000 7ff248b7d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2440cfb60 0x7ff244161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.133+0000 7ff248b7d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff24415dc80 0x7ff24415e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.133+0000 7ff248b7d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2440ce0b0 con 0x7ff244157c70 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.133+0000 7ff248b7d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff244161960 con 0x7ff24415dc80 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff248b7d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24415e670 con 0x7ff2440cfb60 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff24259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff244157c70 0x7ff2440cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58870/0 (socket says 172.21.15.107:58870) 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff24259c640 1 -- 172.21.15.107:0/1426480668 learned_addr learned my addr 172.21.15.107:0/1426480668 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff242d9d640 1 --2- 172.21.15.107:0/1426480668 >> v2:172.21.15.107:3300/0 conn(0x7ff24415dc80 0x7ff24415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff241d9b640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2440cfb60 0x7ff244161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff24159a640 1 -- 172.21.15.107:0/1426480668 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523309652 0 0) 0x7ff2440ce0b0 con 0x7ff244157c70 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff24159a640 1 -- 172.21.15.107:0/1426480668 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22c003520 con 0x7ff244157c70 2023-11-24T08:06:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff242d9d640 1 -- 172.21.15.107:0/1426480668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2440cfb60 msgr2=0x7ff244161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff242d9d640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2440cfb60 0x7ff244161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff242d9d640 1 -- 172.21.15.107:0/1426480668 >> v1:172.21.15.107:6789/0 conn(0x7ff244157c70 legacy=0x7ff2440cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.134+0000 7ff242d9d640 1 -- 172.21.15.107:0/1426480668 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24415e8d0 con 0x7ff24415dc80 2023-11-24T08:06:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.136+0000 7ff242d9d640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff24415dc80 0x7ff24415e130 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009d90 tx=0x7ff23c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4597d5284ff8638b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.136+0000 7ff24159a640 1 -- 172.21.15.107:0/1426480668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c02cbc0 con 0x7ff24415dc80 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.136+0000 7ff24159a640 1 -- 172.21.15.107:0/1426480668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c02cd20 con 0x7ff24415dc80 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.136+0000 7ff24159a640 1 -- 172.21.15.107:0/1426480668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c034660 con 0x7ff24415dc80 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 -- 172.21.15.107:0/1426480668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff24415dc80 msgr2=0x7ff24415e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff24415dc80 0x7ff24415e130 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009d90 tx=0x7ff23c02be10 comp rx=0 tx=0).stop 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 -- 172.21.15.107:0/1426480668 shutdown_connections 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff24415dc80 0x7ff24415e130 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 --2- 172.21.15.107:0/1426480668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2440cfb60 0x7ff244161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 -- 172.21.15.107:0/1426480668 >> 172.21.15.107:0/1426480668 conn(0x7ff24406a920 msgr2=0x7ff24406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 -- 172.21.15.107:0/1426480668 shutdown_connections 2023-11-24T08:06:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.137+0000 7ff248b7d640 1 -- 172.21.15.107:0/1426480668 wait complete. 2023-11-24T08:06:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.139+0000 7ff248b7d640 1 Processor -- start 2023-11-24T08:06:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.139+0000 7ff248b7d640 1 -- start start 2023-11-24T08:06:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 0x7ff2440aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 0x7ff24409cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff244147ea0 con 0x7ff244157c70 2023-11-24T08:06:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2440abfc0 con 0x7ff2440cfb60 2023-11-24T08:06:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff248b7d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24409a2d0 con 0x7ff24415dc80 2023-11-24T08:06:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43322/0 (socket says 172.21.15.107:43322) 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 -- 172.21.15.107:0/3135087702 learned_addr learned my addr 172.21.15.107:0/3135087702 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff242d9d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 0x7ff24409cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff24259c640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 0x7ff2440aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 -- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 msgr2=0x7ff24409cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 0x7ff24409cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 -- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 msgr2=0x7ff2440aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 0x7ff2440aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff241d9b640 1 -- 172.21.15.107:0/3135087702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c009a70 con 0x7ff244157c70 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff242d9d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 0x7ff24409cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.140+0000 7ff24259c640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 0x7ff2440aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff241d9b640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff23800d5f0 tx=0x7ff23800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380125f0 con 0x7ff244157c70 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238012750 con 0x7ff244157c70 2023-11-24T08:06:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff244147990 con 0x7ff244157c70 2023-11-24T08:06:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238012b00 con 0x7ff244157c70 2023-11-24T08:06:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.141+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2440ac210 con 0x7ff244157c70 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.142+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff244067470 con 0x7ff244157c70 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.143+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff238012c60 con 0x7ff244157c70 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.144+0000 7ff232ffd640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 0x7ff22406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.144+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff238089a20 con 0x7ff244157c70 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.144+0000 7ff24259c640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 0x7ff22406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.145+0000 7ff24259c640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 0x7ff22406d300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2340011e0 tx=0x7ff234009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.148+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff238054b60 con 0x7ff244157c70 2023-11-24T08:06:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.333+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2441615f0 con 0x7ff22406ae40 2023-11-24T08:06:08.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.343+0000 7ff232ffd640 1 -- 172.21.15.107:0/3135087702 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281463 (secure 0 0 0) 0x7ff2340013f0 con 0x7ff22406ae40 2023-11-24T08:06:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:08.351 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:06:08.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 msgr2=0x7ff22406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 0x7ff22406d300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2340011e0 tx=0x7ff234009040 comp rx=0 tx=0).stop 2023-11-24T08:06:08.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 msgr2=0x7ff244147450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff23800d5f0 tx=0x7ff23800dab0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 shutdown_connections 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff22406ae40 0x7ff22406d300 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24415dc80 0x7ff24409cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff244157c70 0x7ff244147450 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 --2- 172.21.15.107:0/3135087702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2440cfb60 0x7ff2440aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.349+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 >> 172.21.15.107:0/3135087702 conn(0x7ff24406a920 msgr2=0x7ff2440cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.350+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 shutdown_connections 2023-11-24T08:06:08.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.350+0000 7ff248b7d640 1 -- 172.21.15.107:0/3135087702 wait complete. 2023-11-24T08:06:08.370 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-11-24T08:06:07.454840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3904,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3997696,"data_stored":2253726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.501171"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988645+0000","last_change":"2023-11-24T08:06:02.988645+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988645+0000","last_undegraded":"2023-11-24T08:06:02.988645+0000","last_fullsized":"2023-11-24T08:06:02.988645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990877+0000","last_change":"2023-11-24T08:06:02.990877+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990877+0000","last_undegraded":"2023-11-24T08:06:02.990877+0000","last_fullsized":"2023-11-24T08:06:02.990877+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991622+0000","last_change":"2023-11-24T08:06:02.991622+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991622+0000","last_undegraded":"2023-11-24T08:06:02.991622+0000","last_fullsized":"2023-11-24T08:06:02.991622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989545+0000","last_change":"2023-11-24T08:06:02.989545+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989545+0000","last_undegraded":"2023-11-24T08:06:02.989545+0000","last_fullsized":"2023-11-24T08:06:02.989545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987313+0000","last_change":"2023-11-24T08:06:02.987313+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987313+0000","last_undegraded":"2023-11-24T08:06:02.987313+0000","last_fullsized":"2023-11-24T08:06:02.987313+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997609+0000","last_change":"2023-11-24T08:06:02.997609+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997609+0000","last_undegraded":"2023-11-24T08:06:02.997609+0000","last_fullsized":"2023-11-24T08:06:02.997609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995334+0000","last_change":"2023-11-24T08:06:02.995334+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995334+0000","last_undegraded":"2023-11-24T08:06:02.995334+0000","last_fullsized":"2023-11-24T08:06:02.995334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.002111+0000","last_change":"2023-11-24T08:06:03.002111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.002111+0000","last_undegraded":"2023-11-24T08:06:03.002111+0000","last_fullsized":"2023-11-24T08:06:03.002111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978774+0000","last_change":"2023-11-24T08:06:02.978774+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978774+0000","last_undegraded":"2023-11-24T08:06:02.978774+0000","last_fullsized":"2023-11-24T08:06:02.978774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993312+0000","last_change":"2023-11-24T08:06:02.993312+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993312+0000","last_undegraded":"2023-11-24T08:06:02.993312+0000","last_fullsized":"2023-11-24T08:06:02.993312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000463+0000","last_change":"2023-11-24T08:06:03.000463+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000463+0000","last_undegraded":"2023-11-24T08:06:03.000463+0000","last_fullsized":"2023-11-24T08:06:03.000463+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990332+0000","last_change":"2023-11-24T08:06:02.990332+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990332+0000","last_undegraded":"2023-11-24T08:06:02.990332+0000","last_fullsized":"2023-11-24T08:06:02.990332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986467+0000","last_change":"2023-11-24T08:06:02.986467+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986467+0000","last_undegraded":"2023-11-24T08:06:02.986467+0000","last_fullsized":"2023-11-24T08:06:02.986467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991919+0000","last_change":"2023-11-24T08:06:02.991919+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991919+0000","last_undegraded":"2023-11-24T08:06:02.991919+0000","last_fullsized":"2023-11-24T08:06:02.991919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994432+0000","last_change":"2023-11-24T08:06:02.994432+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994432+0000","last_undegraded":"2023-11-24T08:06:02.994432+0000","last_fullsized":"2023-11-24T08:06:02.994432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995541+0000","last_change":"2023-11-24T08:06:02.995541+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995541+0000","last_undegraded":"2023-11-24T08:06:02.995541+0000","last_fullsized":"2023-11-24T08:06:02.995541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000531+0000","last_change":"2023-11-24T08:06:03.000531+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000531+0000","last_undegraded":"2023-11-24T08:06:03.000531+0000","last_fullsized":"2023-11-24T08:06:03.000531+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990434+0000","last_change":"2023-11-24T08:06:02.990434+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990434+0000","last_undegraded":"2023-11-24T08:06:02.990434+0000","last_fullsized":"2023-11-24T08:06:02.990434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989472+0000","last_change":"2023-11-24T08:06:02.989472+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989472+0000","last_undegraded":"2023-11-24T08:06:02.989472+0000","last_fullsized":"2023-11-24T08:06:02.989472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989920+0000","last_change":"2023-11-24T08:06:02.989920+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989920+0000","last_undegraded":"2023-11-24T08:06:02.989920+0000","last_fullsized":"2023-11-24T08:06:02.989920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982150+0000","last_change":"2023-11-24T08:06:02.982150+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982150+0000","last_undegraded":"2023-11-24T08:06:02.982150+0000","last_fullsized":"2023-11-24T08:06:02.982150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.977577+0000","last_change":"2023-11-24T08:06:02.977577+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.977577+0000","last_undegraded":"2023-11-24T08:06:02.977577+0000","last_fullsized":"2023-11-24T08:06:02.977577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982512+0000","last_change":"2023-11-24T08:06:02.982512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982512+0000","last_undegraded":"2023-11-24T08:06:02.982512+0000","last_fullsized":"2023-11-24T08:06:02.982512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989460+0000","last_change":"2023-11-24T08:06:02.989460+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989460+0000","last_undegraded":"2023-11-24T08:06:02.989460+0000","last_fullsized":"2023-11-24T08:06:02.989460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997244+0000","last_change":"2023-11-24T08:06:02.997244+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997244+0000","last_undegraded":"2023-11-24T08:06:02.997244+0000","last_fullsized":"2023-11-24T08:06:02.997244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989480+0000","last_change":"2023-11-24T08:06:02.989480+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989480+0000","last_undegraded":"2023-11-24T08:06:02.989480+0000","last_fullsized":"2023-11-24T08:06:02.989480+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993720+0000","last_change":"2023-11-24T08:06:02.993720+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993720+0000","last_undegraded":"2023-11-24T08:06:02.993720+0000","last_fullsized":"2023-11-24T08:06:02.993720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989000+0000","last_change":"2023-11-24T08:06:02.989000+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989000+0000","last_undegraded":"2023-11-24T08:06:02.989000+0000","last_fullsized":"2023-11-24T08:06:02.989000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981111+0000","last_change":"2023-11-24T08:06:02.981111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981111+0000","last_undegraded":"2023-11-24T08:06:02.981111+0000","last_fullsized":"2023-11-24T08:06:02.981111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.996512+0000","last_change":"2023-11-24T08:06:02.996512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.996512+0000","last_undegraded":"2023-11-24T08:06:02.996512+0000","last_fullsized":"2023-11-24T08:06:02.996512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988512+0000","last_change":"2023-11-24T08:06:02.988512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988512+0000","last_undegraded":"2023-11-24T08:06:02.988512+0000","last_fullsized":"2023-11-24T08:06:02.988512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000057+0000","last_change":"2023-11-24T08:06:03.000057+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000057+0000","last_undegraded":"2023-11-24T08:06:03.000057+0000","last_fullsized":"2023-11-24T08:06:03.000057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986477+0000","last_change":"2023-11-24T08:06:02.986477+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986477+0000","last_undegraded":"2023-11-24T08:06:02.986477+0000","last_fullsized":"2023-11-24T08:06:02.986477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011454+0000","last_change":"2023-11-24T08:06:03.008351+0000","last_active":"2023-11-24T08:06:03.011454+0000","last_peered":"2023-11-24T08:06:03.011454+0000","last_clean":"2023-11-24T08:06:03.011454+0000","last_became_active":"2023-11-24T08:06:03.008011+0000","last_became_peered":"2023-11-24T08:06:03.008011+0000","last_unstale":"2023-11-24T08:06:03.011454+0000","last_undegraded":"2023-11-24T08:06:03.011454+0000","last_fullsized":"2023-11-24T08:06:03.011454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986375+0000","last_change":"2023-11-24T08:06:02.986375+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986375+0000","last_undegraded":"2023-11-24T08:06:02.986375+0000","last_fullsized":"2023-11-24T08:06:02.986375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007823+0000","last_change":"2023-11-24T08:06:03.005602+0000","last_active":"2023-11-24T08:06:03.007823+0000","last_peered":"2023-11-24T08:06:03.007823+0000","last_clean":"2023-11-24T08:06:03.007823+0000","last_became_active":"2023-11-24T08:06:03.005346+0000","last_became_peered":"2023-11-24T08:06:03.005346+0000","last_unstale":"2023-11-24T08:06:03.007823+0000","last_undegraded":"2023-11-24T08:06:03.007823+0000","last_fullsized":"2023-11-24T08:06:03.007823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978376+0000","last_change":"2023-11-24T08:06:02.978376+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978376+0000","last_undegraded":"2023-11-24T08:06:02.978376+0000","last_fullsized":"2023-11-24T08:06:02.978376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004953+0000","last_change":"2023-11-24T08:06:02.997760+0000","last_active":"2023-11-24T08:06:03.004953+0000","last_peered":"2023-11-24T08:06:03.004953+0000","last_clean":"2023-11-24T08:06:03.004953+0000","last_became_active":"2023-11-24T08:06:02.997481+0000","last_became_peered":"2023-11-24T08:06:02.997481+0000","last_unstale":"2023-11-24T08:06:03.004953+0000","last_undegraded":"2023-11-24T08:06:03.004953+0000","last_fullsized":"2023-11-24T08:06:03.004953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992842+0000","last_change":"2023-11-24T08:06:02.992842+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992842+0000","last_undegraded":"2023-11-24T08:06:02.992842+0000","last_fullsized":"2023-11-24T08:06:02.992842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.999008+0000","last_change":"2023-11-24T08:06:02.997108+0000","last_active":"2023-11-24T08:06:02.999008+0000","last_peered":"2023-11-24T08:06:02.999008+0000","last_clean":"2023-11-24T08:06:02.999008+0000","last_became_active":"2023-11-24T08:06:02.996880+0000","last_became_peered":"2023-11-24T08:06:02.996880+0000","last_unstale":"2023-11-24T08:06:02.999008+0000","last_undegraded":"2023-11-24T08:06:02.999008+0000","last_fullsized":"2023-11-24T08:06:02.999008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995771+0000","last_change":"2023-11-24T08:06:02.995771+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995771+0000","last_undegraded":"2023-11-24T08:06:02.995771+0000","last_fullsized":"2023-11-24T08:06:02.995771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011211+0000","last_change":"2023-11-24T08:06:03.004832+0000","last_active":"2023-11-24T08:06:03.011211+0000","last_peered":"2023-11-24T08:06:03.011211+0000","last_clean":"2023-11-24T08:06:03.011211+0000","last_became_active":"2023-11-24T08:06:03.004536+0000","last_became_peered":"2023-11-24T08:06:03.004536+0000","last_unstale":"2023-11-24T08:06:03.011211+0000","last_undegraded":"2023-11-24T08:06:03.011211+0000","last_fullsized":"2023-11-24T08:06:03.011211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:06.344044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990140+0000","last_change":"2023-11-24T08:06:02.990140+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990140+0000","last_undegraded":"2023-11-24T08:06:02.990140+0000","last_fullsized":"2023-11-24T08:06:02.990140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005297+0000","last_change":"2023-11-24T08:06:02.993001+0000","last_active":"2023-11-24T08:06:03.005297+0000","last_peered":"2023-11-24T08:06:03.005297+0000","last_clean":"2023-11-24T08:06:03.005297+0000","last_became_active":"2023-11-24T08:06:02.992519+0000","last_became_peered":"2023-11-24T08:06:02.992519+0000","last_unstale":"2023-11-24T08:06:03.005297+0000","last_undegraded":"2023-11-24T08:06:03.005297+0000","last_fullsized":"2023-11-24T08:06:03.005297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:59.875793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.976833+0000","last_change":"2023-11-24T08:06:02.976833+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.976833+0000","last_undegraded":"2023-11-24T08:06:02.976833+0000","last_fullsized":"2023-11-24T08:06:02.976833+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.994102+0000","last_change":"2023-11-24T08:06:02.992653+0000","last_active":"2023-11-24T08:06:02.994102+0000","last_peered":"2023-11-24T08:06:02.994102+0000","last_clean":"2023-11-24T08:06:02.994102+0000","last_became_active":"2023-11-24T08:06:02.992420+0000","last_became_peered":"2023-11-24T08:06:02.992420+0000","last_unstale":"2023-11-24T08:06:02.994102+0000","last_undegraded":"2023-11-24T08:06:02.994102+0000","last_fullsized":"2023-11-24T08:06:02.994102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991684+0000","last_change":"2023-11-24T08:06:02.991684+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991684+0000","last_undegraded":"2023-11-24T08:06:02.991684+0000","last_fullsized":"2023-11-24T08:06:02.991684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005504+0000","last_change":"2023-11-24T08:06:02.998354+0000","last_active":"2023-11-24T08:06:03.005504+0000","last_peered":"2023-11-24T08:06:03.005504+0000","last_clean":"2023-11-24T08:06:03.005504+0000","last_became_active":"2023-11-24T08:06:02.998065+0000","last_became_peered":"2023-11-24T08:06:02.998065+0000","last_unstale":"2023-11-24T08:06:03.005504+0000","last_undegraded":"2023-11-24T08:06:03.005504+0000","last_fullsized":"2023-11-24T08:06:03.005504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:16:32.070719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993733+0000","last_change":"2023-11-24T08:06:02.993733+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993733+0000","last_undegraded":"2023-11-24T08:06:02.993733+0000","last_fullsized":"2023-11-24T08:06:02.993733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011199+0000","last_change":"2023-11-24T08:06:03.010255+0000","last_active":"2023-11-24T08:06:03.011199+0000","last_peered":"2023-11-24T08:06:03.011199+0000","last_clean":"2023-11-24T08:06:03.011199+0000","last_became_active":"2023-11-24T08:06:03.009943+0000","last_became_peered":"2023-11-24T08:06:03.009943+0000","last_unstale":"2023-11-24T08:06:03.011199+0000","last_undegraded":"2023-11-24T08:06:03.011199+0000","last_fullsized":"2023-11-24T08:06:03.011199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985887+0000","last_change":"2023-11-24T08:06:02.985887+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985887+0000","last_undegraded":"2023-11-24T08:06:02.985887+0000","last_fullsized":"2023-11-24T08:06:02.985887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006556+0000","last_change":"2023-11-24T08:06:03.004192+0000","last_active":"2023-11-24T08:06:03.006556+0000","last_peered":"2023-11-24T08:06:03.006556+0000","last_clean":"2023-11-24T08:06:03.006556+0000","last_became_active":"2023-11-24T08:06:03.003913+0000","last_became_peered":"2023-11-24T08:06:03.003913+0000","last_unstale":"2023-11-24T08:06:03.006556+0000","last_undegraded":"2023-11-24T08:06:03.006556+0000","last_fullsized":"2023-11-24T08:06:03.006556+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987547+0000","last_change":"2023-11-24T08:06:02.987547+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987547+0000","last_undegraded":"2023-11-24T08:06:02.987547+0000","last_fullsized":"2023-11-24T08:06:02.987547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011782+0000","last_change":"2023-11-24T08:06:03.009734+0000","last_active":"2023-11-24T08:06:03.011782+0000","last_peered":"2023-11-24T08:06:03.011782+0000","last_clean":"2023-11-24T08:06:03.011782+0000","last_became_active":"2023-11-24T08:06:03.009416+0000","last_became_peered":"2023-11-24T08:06:03.009416+0000","last_unstale":"2023-11-24T08:06:03.011782+0000","last_undegraded":"2023-11-24T08:06:03.011782+0000","last_fullsized":"2023-11-24T08:06:03.011782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992464+0000","last_change":"2023-11-24T08:06:02.992464+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992464+0000","last_undegraded":"2023-11-24T08:06:02.992464+0000","last_fullsized":"2023-11-24T08:06:02.992464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004874+0000","last_change":"2023-11-24T08:06:02.998808+0000","last_active":"2023-11-24T08:06:03.004874+0000","last_peered":"2023-11-24T08:06:03.004874+0000","last_clean":"2023-11-24T08:06:03.004874+0000","last_became_active":"2023-11-24T08:06:02.998600+0000","last_became_peered":"2023-11-24T08:06:02.998600+0000","last_unstale":"2023-11-24T08:06:03.004874+0000","last_undegraded":"2023-11-24T08:06:03.004874+0000","last_fullsized":"2023-11-24T08:06:03.004874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989274+0000","last_change":"2023-11-24T08:06:02.989274+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989274+0000","last_undegraded":"2023-11-24T08:06:02.989274+0000","last_fullsized":"2023-11-24T08:06:02.989274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010263+0000","last_change":"2023-11-24T08:06:03.004639+0000","last_active":"2023-11-24T08:06:03.010263+0000","last_peered":"2023-11-24T08:06:03.010263+0000","last_clean":"2023-11-24T08:06:03.010263+0000","last_became_active":"2023-11-24T08:06:03.004341+0000","last_became_peered":"2023-11-24T08:06:03.004341+0000","last_unstale":"2023-11-24T08:06:03.010263+0000","last_undegraded":"2023-11-24T08:06:03.010263+0000","last_fullsized":"2023-11-24T08:06:03.010263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:46.680770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986105+0000","last_change":"2023-11-24T08:06:02.986105+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986105+0000","last_undegraded":"2023-11-24T08:06:02.986105+0000","last_fullsized":"2023-11-24T08:06:02.986105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008069+0000","last_change":"2023-11-24T08:06:03.005631+0000","last_active":"2023-11-24T08:06:03.008069+0000","last_peered":"2023-11-24T08:06:03.008069+0000","last_clean":"2023-11-24T08:06:03.008069+0000","last_became_active":"2023-11-24T08:06:03.004131+0000","last_became_peered":"2023-11-24T08:06:03.004131+0000","last_unstale":"2023-11-24T08:06:03.008069+0000","last_undegraded":"2023-11-24T08:06:03.008069+0000","last_fullsized":"2023-11-24T08:06:03.008069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999457+0000","last_change":"2023-11-24T08:06:02.999457+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999457+0000","last_undegraded":"2023-11-24T08:06:02.999457+0000","last_fullsized":"2023-11-24T08:06:02.999457+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011168+0000","last_change":"2023-11-24T08:06:03.009985+0000","last_active":"2023-11-24T08:06:03.011168+0000","last_peered":"2023-11-24T08:06:03.011168+0000","last_clean":"2023-11-24T08:06:03.011168+0000","last_became_active":"2023-11-24T08:06:03.009676+0000","last_became_peered":"2023-11-24T08:06:03.009676+0000","last_unstale":"2023-11-24T08:06:03.011168+0000","last_undegraded":"2023-11-24T08:06:03.011168+0000","last_fullsized":"2023-11-24T08:06:03.011168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987482+0000","last_change":"2023-11-24T08:06:02.987482+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987482+0000","last_undegraded":"2023-11-24T08:06:02.987482+0000","last_fullsized":"2023-11-24T08:06:02.987482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004281+0000","last_change":"2023-11-24T08:06:02.996472+0000","last_active":"2023-11-24T08:06:03.004281+0000","last_peered":"2023-11-24T08:06:03.004281+0000","last_clean":"2023-11-24T08:06:03.004281+0000","last_became_active":"2023-11-24T08:06:02.996105+0000","last_became_peered":"2023-11-24T08:06:02.996105+0000","last_unstale":"2023-11-24T08:06:03.004281+0000","last_undegraded":"2023-11-24T08:06:03.004281+0000","last_fullsized":"2023-11-24T08:06:03.004281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:14.099330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980014+0000","last_change":"2023-11-24T08:06:02.980014+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980014+0000","last_undegraded":"2023-11-24T08:06:02.980014+0000","last_fullsized":"2023-11-24T08:06:02.980014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008394+0000","last_change":"2023-11-24T08:06:03.006947+0000","last_active":"2023-11-24T08:06:03.008394+0000","last_peered":"2023-11-24T08:06:03.008394+0000","last_clean":"2023-11-24T08:06:03.008394+0000","last_became_active":"2023-11-24T08:06:03.006640+0000","last_became_peered":"2023-11-24T08:06:03.006640+0000","last_unstale":"2023-11-24T08:06:03.008394+0000","last_undegraded":"2023-11-24T08:06:03.008394+0000","last_fullsized":"2023-11-24T08:06:03.008394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995208+0000","last_change":"2023-11-24T08:06:02.995208+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995208+0000","last_undegraded":"2023-11-24T08:06:02.995208+0000","last_fullsized":"2023-11-24T08:06:02.995208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007244+0000","last_change":"2023-11-24T08:06:03.003338+0000","last_active":"2023-11-24T08:06:03.007244+0000","last_peered":"2023-11-24T08:06:03.007244+0000","last_clean":"2023-11-24T08:06:03.007244+0000","last_became_active":"2023-11-24T08:06:02.998184+0000","last_became_peered":"2023-11-24T08:06:02.998184+0000","last_unstale":"2023-11-24T08:06:03.007244+0000","last_undegraded":"2023-11-24T08:06:03.007244+0000","last_fullsized":"2023-11-24T08:06:03.007244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989003+0000","last_change":"2023-11-24T08:06:02.989003+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989003+0000","last_undegraded":"2023-11-24T08:06:02.989003+0000","last_fullsized":"2023-11-24T08:06:02.989003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006462+0000","last_change":"2023-11-24T08:06:03.001694+0000","last_active":"2023-11-24T08:06:03.006462+0000","last_peered":"2023-11-24T08:06:03.006462+0000","last_clean":"2023-11-24T08:06:03.006462+0000","last_became_active":"2023-11-24T08:06:02.998616+0000","last_became_peered":"2023-11-24T08:06:02.998616+0000","last_unstale":"2023-11-24T08:06:03.006462+0000","last_undegraded":"2023-11-24T08:06:03.006462+0000","last_fullsized":"2023-11-24T08:06:03.006462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:30.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986501+0000","last_change":"2023-11-24T08:06:02.986501+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986501+0000","last_undegraded":"2023-11-24T08:06:02.986501+0000","last_fullsized":"2023-11-24T08:06:02.986501+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.003102+0000","last_change":"2023-11-24T08:06:02.993516+0000","last_active":"2023-11-24T08:06:03.003102+0000","last_peered":"2023-11-24T08:06:03.003102+0000","last_clean":"2023-11-24T08:06:03.003102+0000","last_became_active":"2023-11-24T08:06:02.992976+0000","last_became_peered":"2023-11-24T08:06:02.992976+0000","last_unstale":"2023-11-24T08:06:03.003102+0000","last_undegraded":"2023-11-24T08:06:03.003102+0000","last_fullsized":"2023-11-24T08:06:03.003102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:05:42.068467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985726+0000","last_change":"2023-11-24T08:06:02.985726+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985726+0000","last_undegraded":"2023-11-24T08:06:02.985726+0000","last_fullsized":"2023-11-24T08:06:02.985726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.000770+0000","last_change":"2023-11-24T08:06:02.998617+0000","last_active":"2023-11-24T08:06:03.000770+0000","last_peered":"2023-11-24T08:06:03.000770+0000","last_clean":"2023-11-24T08:06:03.000770+0000","last_became_active":"2023-11-24T08:06:02.998393+0000","last_became_peered":"2023-11-24T08:06:02.998393+0000","last_unstale":"2023-11-24T08:06:03.000770+0000","last_undegraded":"2023-11-24T08:06:03.000770+0000","last_fullsized":"2023-11-24T08:06:03.000770+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985538+0000","last_change":"2023-11-24T08:06:02.985538+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985538+0000","last_undegraded":"2023-11-24T08:06:02.985538+0000","last_fullsized":"2023-11-24T08:06:02.985538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010412+0000","last_change":"2023-11-24T08:06:03.005058+0000","last_active":"2023-11-24T08:06:03.010412+0000","last_peered":"2023-11-24T08:06:03.010412+0000","last_clean":"2023-11-24T08:06:03.010412+0000","last_became_active":"2023-11-24T08:06:03.004703+0000","last_became_peered":"2023-11-24T08:06:03.004703+0000","last_unstale":"2023-11-24T08:06:03.010412+0000","last_undegraded":"2023-11-24T08:06:03.010412+0000","last_fullsized":"2023-11-24T08:06:03.010412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985668+0000","last_change":"2023-11-24T08:06:02.985668+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985668+0000","last_undegraded":"2023-11-24T08:06:02.985668+0000","last_fullsized":"2023-11-24T08:06:02.985668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007000+0000","last_change":"2023-11-24T08:06:03.003128+0000","last_active":"2023-11-24T08:06:03.007000+0000","last_peered":"2023-11-24T08:06:03.007000+0000","last_clean":"2023-11-24T08:06:03.007000+0000","last_became_active":"2023-11-24T08:06:03.002791+0000","last_became_peered":"2023-11-24T08:06:03.002791+0000","last_unstale":"2023-11-24T08:06:03.007000+0000","last_undegraded":"2023-11-24T08:06:03.007000+0000","last_fullsized":"2023-11-24T08:06:03.007000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:49.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995875+0000","last_change":"2023-11-24T08:06:02.995875+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995875+0000","last_undegraded":"2023-11-24T08:06:02.995875+0000","last_fullsized":"2023-11-24T08:06:02.995875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011002+0000","last_change":"2023-11-24T08:06:03.006761+0000","last_active":"2023-11-24T08:06:03.011002+0000","last_peered":"2023-11-24T08:06:03.011002+0000","last_clean":"2023-11-24T08:06:03.011002+0000","last_became_active":"2023-11-24T08:06:03.006418+0000","last_became_peered":"2023-11-24T08:06:03.006418+0000","last_unstale":"2023-11-24T08:06:03.011002+0000","last_undegraded":"2023-11-24T08:06:03.011002+0000","last_fullsized":"2023-11-24T08:06:03.011002+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988366+0000","last_change":"2023-11-24T08:06:02.988366+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988366+0000","last_undegraded":"2023-11-24T08:06:02.988366+0000","last_fullsized":"2023-11-24T08:06:02.988366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010239+0000","last_change":"2023-11-24T08:06:03.008262+0000","last_active":"2023-11-24T08:06:03.010239+0000","last_peered":"2023-11-24T08:06:03.010239+0000","last_clean":"2023-11-24T08:06:03.010239+0000","last_became_active":"2023-11-24T08:06:03.007982+0000","last_became_peered":"2023-11-24T08:06:03.007982+0000","last_unstale":"2023-11-24T08:06:03.010239+0000","last_undegraded":"2023-11-24T08:06:03.010239+0000","last_fullsized":"2023-11-24T08:06:03.010239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994570+0000","last_change":"2023-11-24T08:06:02.994570+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994570+0000","last_undegraded":"2023-11-24T08:06:02.994570+0000","last_fullsized":"2023-11-24T08:06:02.994570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006539+0000","last_change":"2023-11-24T08:06:03.001269+0000","last_active":"2023-11-24T08:06:03.006539+0000","last_peered":"2023-11-24T08:06:03.006539+0000","last_clean":"2023-11-24T08:06:03.006539+0000","last_became_active":"2023-11-24T08:06:03.000568+0000","last_became_peered":"2023-11-24T08:06:03.000568+0000","last_unstale":"2023-11-24T08:06:03.006539+0000","last_undegraded":"2023-11-24T08:06:03.006539+0000","last_fullsized":"2023-11-24T08:06:03.006539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987028+0000","last_change":"2023-11-24T08:06:02.987028+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987028+0000","last_undegraded":"2023-11-24T08:06:02.987028+0000","last_fullsized":"2023-11-24T08:06:02.987028+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.998436+0000","last_change":"2023-11-24T08:06:02.995680+0000","last_active":"2023-11-24T08:06:02.998436+0000","last_peered":"2023-11-24T08:06:02.998436+0000","last_clean":"2023-11-24T08:06:02.998436+0000","last_became_active":"2023-11-24T08:06:02.995421+0000","last_became_peered":"2023-11-24T08:06:02.995421+0000","last_unstale":"2023-11-24T08:06:02.998436+0000","last_undegraded":"2023-11-24T08:06:02.998436+0000","last_fullsized":"2023-11-24T08:06:02.998436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985166+0000","last_change":"2023-11-24T08:06:02.985166+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985166+0000","last_undegraded":"2023-11-24T08:06:02.985166+0000","last_fullsized":"2023-11-24T08:06:02.985166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004405+0000","last_change":"2023-11-24T08:06:03.000694+0000","last_active":"2023-11-24T08:06:03.004405+0000","last_peered":"2023-11-24T08:06:03.004405+0000","last_clean":"2023-11-24T08:06:03.004405+0000","last_became_active":"2023-11-24T08:06:03.000285+0000","last_became_peered":"2023-11-24T08:06:03.000285+0000","last_unstale":"2023-11-24T08:06:03.004405+0000","last_undegraded":"2023-11-24T08:06:03.004405+0000","last_fullsized":"2023-11-24T08:06:03.004405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:44.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.983698+0000","last_change":"2023-11-24T08:06:02.983698+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.983698+0000","last_undegraded":"2023-11-24T08:06:02.983698+0000","last_fullsized":"2023-11-24T08:06:02.983698+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010558+0000","last_change":"2023-11-24T08:06:03.008992+0000","last_active":"2023-11-24T08:06:03.010558+0000","last_peered":"2023-11-24T08:06:03.010558+0000","last_clean":"2023-11-24T08:06:03.010558+0000","last_became_active":"2023-11-24T08:06:03.008648+0000","last_became_peered":"2023-11-24T08:06:03.008648+0000","last_unstale":"2023-11-24T08:06:03.010558+0000","last_undegraded":"2023-11-24T08:06:03.010558+0000","last_fullsized":"2023-11-24T08:06:03.010558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981885+0000","last_change":"2023-11-24T08:06:02.981885+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981885+0000","last_undegraded":"2023-11-24T08:06:02.981885+0000","last_fullsized":"2023-11-24T08:06:02.981885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007305+0000","last_change":"2023-11-24T08:06:03.003843+0000","last_active":"2023-11-24T08:06:03.007305+0000","last_peered":"2023-11-24T08:06:03.007305+0000","last_clean":"2023-11-24T08:06:03.007305+0000","last_became_active":"2023-11-24T08:06:03.002741+0000","last_became_peered":"2023-11-24T08:06:03.002741+0000","last_unstale":"2023-11-24T08:06:03.007305+0000","last_undegraded":"2023-11-24T08:06:03.007305+0000","last_fullsized":"2023-11-24T08:06:03.007305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:46:14.437116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980343+0000","last_change":"2023-11-24T08:06:02.980343+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980343+0000","last_undegraded":"2023-11-24T08:06:02.980343+0000","last_fullsized":"2023-11-24T08:06:02.980343+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011146+0000","last_change":"2023-11-24T08:06:03.009978+0000","last_active":"2023-11-24T08:06:03.011146+0000","last_peered":"2023-11-24T08:06:03.011146+0000","last_clean":"2023-11-24T08:06:03.011146+0000","last_became_active":"2023-11-24T08:06:03.009694+0000","last_became_peered":"2023-11-24T08:06:03.009694+0000","last_unstale":"2023-11-24T08:06:03.011146+0000","last_undegraded":"2023-11-24T08:06:03.011146+0000","last_fullsized":"2023-11-24T08:06:03.011146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999828+0000","last_change":"2023-11-24T08:06:02.999828+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999828+0000","last_undegraded":"2023-11-24T08:06:02.999828+0000","last_fullsized":"2023-11-24T08:06:02.999828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011405+0000","last_change":"2023-11-24T08:06:03.008297+0000","last_active":"2023-11-24T08:06:03.011405+0000","last_peered":"2023-11-24T08:06:03.011405+0000","last_clean":"2023-11-24T08:06:03.011405+0000","last_became_active":"2023-11-24T08:06:03.007985+0000","last_became_peered":"2023-11-24T08:06:03.007985+0000","last_unstale":"2023-11-24T08:06:03.011405+0000","last_undegraded":"2023-11-24T08:06:03.011405+0000","last_fullsized":"2023-11-24T08:06:03.011405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-24T08:06:01.972672+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:06:01.972672+0000","last_peered":"2023-11-24T08:06:01.972672+0000","last_clean":"2023-11-24T08:06:01.972672+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:06:01.972672+0000","last_undegraded":"2023-11-24T08:06:01.972672+0000","last_fullsized":"2023-11-24T08:06:01.972672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93100000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.048}]},{"osd":2,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.177},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":1.643}]},{"osd":3,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.486},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.365}]},{"osd":4,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.302},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.522},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.623}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.345},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.607}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":1.089}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.4370000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.661}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.639}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.585}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.90100000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.353}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.79200000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.286},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.321}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4019999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.127}]},{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.3879999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.4379999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.546},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.617}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.5899999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":1.335}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.504},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.5700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.2789999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.232},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.0509999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.623},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.3180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.359}]},{"osd":6,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.446},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.4099999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.496},{"interface":"front","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.536}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.83799999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.98599999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.2829999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.214}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.597,"15min":1.597},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3809999999999998,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.0940000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.87},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.93799999999999994}]},{"osd":2,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.2529999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":1.05}]},{"osd":6,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.9359999999999999,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3540000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.159}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.72899999999999998},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83499999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.877}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.0780000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.115},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.0920000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:06:08.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:06:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.617+0000 7f39bd389640 1 Processor -- start 2023-11-24T08:06:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.618+0000 7f39bd389640 1 -- start start 2023-11-24T08:06:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.619+0000 7f39bd389640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 0x7f39b80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.619+0000 7f39bd389640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.619+0000 7f39bd389640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b815e3f0 con 0x7f39b80b2250 2023-11-24T08:06:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.619+0000 7f39bd389640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b815a990 con 0x7f39b815df80 2023-11-24T08:06:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.619+0000 7f39bd389640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815ab80 con 0x7f39b80cd470 2023-11-24T08:06:08.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39affff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 0x7f39b80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40098/0 (socket says 172.21.15.107:40098) 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 -- 172.21.15.107:0/3159002843 learned_addr learned my addr 172.21.15.107:0/3159002843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 -- 172.21.15.107:0/3159002843 >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 msgr2=0x7f39b80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 --2- 172.21.15.107:0/3159002843 >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 0x7f39b80ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 -- 172.21.15.107:0/3159002843 >> v1:172.21.15.120:6789/0 conn(0x7f39b80b2250 legacy=0x7f39b80b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39b77fe640 1 -- 172.21.15.107:0/3159002843 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80b2d40 con 0x7f39b80cd470 2023-11-24T08:06:08.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.620+0000 7f39affff640 1 --2- 172.21.15.107:0/3159002843 >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 0x7f39b80ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:06:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.622+0000 7f39b77fe640 1 --2- 172.21.15.107:0/3159002843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d86698030e2971 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.622+0000 7f39b67fc640 1 -- 172.21.15.107:0/3159002843 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b002cb00 con 0x7f39b80cd470 2023-11-24T08:06:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.622+0000 7f39b67fc640 1 -- 172.21.15.107:0/3159002843 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b002cc60 con 0x7f39b80cd470 2023-11-24T08:06:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.622+0000 7f39b67fc640 1 -- 172.21.15.107:0/3159002843 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b00343f0 con 0x7f39b80cd470 2023-11-24T08:06:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 -- 172.21.15.107:0/3159002843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 msgr2=0x7f39b815b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 --2- 172.21.15.107:0/3159002843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b002bf60 comp rx=0 tx=0).stop 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 -- 172.21.15.107:0/3159002843 shutdown_connections 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 --2- 172.21.15.107:0/3159002843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80cd470 0x7f39b815b080 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 --2- 172.21.15.107:0/3159002843 >> v2:172.21.15.120:3300/0 conn(0x7f39b815df80 0x7f39b80ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 -- 172.21.15.107:0/3159002843 >> 172.21.15.107:0/3159002843 conn(0x7f39b806a920 msgr2=0x7f39b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 -- 172.21.15.107:0/3159002843 shutdown_connections 2023-11-24T08:06:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.623+0000 7f39bd389640 1 -- 172.21.15.107:0/3159002843 wait complete. 2023-11-24T08:06:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.625+0000 7f39bd389640 1 Processor -- start 2023-11-24T08:06:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.625+0000 7f39bd389640 1 -- start start 2023-11-24T08:06:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80b2250 0x7f39b8147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b80cd470 0x7f39b8147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815dae0 con 0x7f39b815df80 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8168a40 con 0x7f39b80cd470 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39bd389640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8168d20 con 0x7f39b80b2250 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43324/0 (socket says 172.21.15.107:43324) 2023-11-24T08:06:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b77fe640 1 -- 172.21.15.107:0/4246905931 learned_addr learned my addr 172.21.15.107:0/4246905931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b6ffd640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80b2250 0x7f39b8147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39affff640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b80cd470 0x7f39b8147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b77fe640 1 -- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80b2250 msgr2=0x7f39b8147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.626+0000 7f39b77fe640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80b2250 0x7f39b8147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39b77fe640 1 -- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b80cd470 msgr2=0x7f39b8147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39b77fe640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b80cd470 0x7f39b8147680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39b77fe640 1 -- 172.21.15.107:0/4246905931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a0009400 con 0x7f39b815df80 2023-11-24T08:06:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39b77fe640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0007ea0 con 0x7f39b815df80 2023-11-24T08:06:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b003e450 con 0x7f39b815df80 2023-11-24T08:06:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b003e800 con 0x7f39b815df80 2023-11-24T08:06:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0009b30 con 0x7f39b815df80 2023-11-24T08:06:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.627+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b81692e0 con 0x7f39b815df80 2023-11-24T08:06:08.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.628+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8067470 con 0x7f39b815df80 2023-11-24T08:06:08.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.629+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f39b003e960 con 0x7f39b815df80 2023-11-24T08:06:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.630+0000 7f39af7fe640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 0x7f399806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.630+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b00ac230 con 0x7f39b815df80 2023-11-24T08:06:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.630+0000 7f39b6ffd640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 0x7f399806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.631+0000 7f39b6ffd640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 0x7f399806d300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f39a8001640 tx=0x7f39a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:08.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.634+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39b00772d0 con 0x7f39b815df80 2023-11-24T08:06:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.822+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39b815ec40 con 0x7f399806ae40 2023-11-24T08:06:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.831+0000 7f39af7fe640 1 -- 172.21.15.107:0/4246905931 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281463 (secure 0 0 0) 0x7f39a80050d0 con 0x7f399806ae40 2023-11-24T08:06:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 msgr2=0x7f399806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 0x7f399806d300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f39a8001640 tx=0x7f39a8009040 comp rx=0 tx=0).stop 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 msgr2=0x7f39b814b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b0004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 shutdown_connections 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399806ae40 0x7f399806d300 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39b815df80 0x7f39b814b970 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b80cd470 0x7f39b8147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 --2- 172.21.15.107:0/4246905931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39b80b2250 0x7f39b8147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 >> 172.21.15.107:0/4246905931 conn(0x7f39b806a920 msgr2=0x7f39b814a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 shutdown_connections 2023-11-24T08:06:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:08.838+0000 7f39bd389640 1 -- 172.21.15.107:0/4246905931 wait complete. 2023-11-24T08:06:08.858 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-11-24T08:06:07.454840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3904,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3997696,"data_stored":2253726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.501171"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988645+0000","last_change":"2023-11-24T08:06:02.988645+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988645+0000","last_undegraded":"2023-11-24T08:06:02.988645+0000","last_fullsized":"2023-11-24T08:06:02.988645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990877+0000","last_change":"2023-11-24T08:06:02.990877+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990877+0000","last_undegraded":"2023-11-24T08:06:02.990877+0000","last_fullsized":"2023-11-24T08:06:02.990877+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991622+0000","last_change":"2023-11-24T08:06:02.991622+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991622+0000","last_undegraded":"2023-11-24T08:06:02.991622+0000","last_fullsized":"2023-11-24T08:06:02.991622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989545+0000","last_change":"2023-11-24T08:06:02.989545+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989545+0000","last_undegraded":"2023-11-24T08:06:02.989545+0000","last_fullsized":"2023-11-24T08:06:02.989545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987313+0000","last_change":"2023-11-24T08:06:02.987313+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987313+0000","last_undegraded":"2023-11-24T08:06:02.987313+0000","last_fullsized":"2023-11-24T08:06:02.987313+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997609+0000","last_change":"2023-11-24T08:06:02.997609+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997609+0000","last_undegraded":"2023-11-24T08:06:02.997609+0000","last_fullsized":"2023-11-24T08:06:02.997609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995334+0000","last_change":"2023-11-24T08:06:02.995334+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995334+0000","last_undegraded":"2023-11-24T08:06:02.995334+0000","last_fullsized":"2023-11-24T08:06:02.995334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.002111+0000","last_change":"2023-11-24T08:06:03.002111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.002111+0000","last_undegraded":"2023-11-24T08:06:03.002111+0000","last_fullsized":"2023-11-24T08:06:03.002111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978774+0000","last_change":"2023-11-24T08:06:02.978774+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978774+0000","last_undegraded":"2023-11-24T08:06:02.978774+0000","last_fullsized":"2023-11-24T08:06:02.978774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993312+0000","last_change":"2023-11-24T08:06:02.993312+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993312+0000","last_undegraded":"2023-11-24T08:06:02.993312+0000","last_fullsized":"2023-11-24T08:06:02.993312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000463+0000","last_change":"2023-11-24T08:06:03.000463+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000463+0000","last_undegraded":"2023-11-24T08:06:03.000463+0000","last_fullsized":"2023-11-24T08:06:03.000463+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990332+0000","last_change":"2023-11-24T08:06:02.990332+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990332+0000","last_undegraded":"2023-11-24T08:06:02.990332+0000","last_fullsized":"2023-11-24T08:06:02.990332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986467+0000","last_change":"2023-11-24T08:06:02.986467+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986467+0000","last_undegraded":"2023-11-24T08:06:02.986467+0000","last_fullsized":"2023-11-24T08:06:02.986467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991919+0000","last_change":"2023-11-24T08:06:02.991919+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991919+0000","last_undegraded":"2023-11-24T08:06:02.991919+0000","last_fullsized":"2023-11-24T08:06:02.991919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994432+0000","last_change":"2023-11-24T08:06:02.994432+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994432+0000","last_undegraded":"2023-11-24T08:06:02.994432+0000","last_fullsized":"2023-11-24T08:06:02.994432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995541+0000","last_change":"2023-11-24T08:06:02.995541+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995541+0000","last_undegraded":"2023-11-24T08:06:02.995541+0000","last_fullsized":"2023-11-24T08:06:02.995541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000531+0000","last_change":"2023-11-24T08:06:03.000531+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000531+0000","last_undegraded":"2023-11-24T08:06:03.000531+0000","last_fullsized":"2023-11-24T08:06:03.000531+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990434+0000","last_change":"2023-11-24T08:06:02.990434+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990434+0000","last_undegraded":"2023-11-24T08:06:02.990434+0000","last_fullsized":"2023-11-24T08:06:02.990434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989472+0000","last_change":"2023-11-24T08:06:02.989472+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989472+0000","last_undegraded":"2023-11-24T08:06:02.989472+0000","last_fullsized":"2023-11-24T08:06:02.989472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989920+0000","last_change":"2023-11-24T08:06:02.989920+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989920+0000","last_undegraded":"2023-11-24T08:06:02.989920+0000","last_fullsized":"2023-11-24T08:06:02.989920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982150+0000","last_change":"2023-11-24T08:06:02.982150+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982150+0000","last_undegraded":"2023-11-24T08:06:02.982150+0000","last_fullsized":"2023-11-24T08:06:02.982150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.977577+0000","last_change":"2023-11-24T08:06:02.977577+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.977577+0000","last_undegraded":"2023-11-24T08:06:02.977577+0000","last_fullsized":"2023-11-24T08:06:02.977577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982512+0000","last_change":"2023-11-24T08:06:02.982512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982512+0000","last_undegraded":"2023-11-24T08:06:02.982512+0000","last_fullsized":"2023-11-24T08:06:02.982512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989460+0000","last_change":"2023-11-24T08:06:02.989460+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989460+0000","last_undegraded":"2023-11-24T08:06:02.989460+0000","last_fullsized":"2023-11-24T08:06:02.989460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997244+0000","last_change":"2023-11-24T08:06:02.997244+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997244+0000","last_undegraded":"2023-11-24T08:06:02.997244+0000","last_fullsized":"2023-11-24T08:06:02.997244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989480+0000","last_change":"2023-11-24T08:06:02.989480+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989480+0000","last_undegraded":"2023-11-24T08:06:02.989480+0000","last_fullsized":"2023-11-24T08:06:02.989480+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993720+0000","last_change":"2023-11-24T08:06:02.993720+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993720+0000","last_undegraded":"2023-11-24T08:06:02.993720+0000","last_fullsized":"2023-11-24T08:06:02.993720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989000+0000","last_change":"2023-11-24T08:06:02.989000+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989000+0000","last_undegraded":"2023-11-24T08:06:02.989000+0000","last_fullsized":"2023-11-24T08:06:02.989000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981111+0000","last_change":"2023-11-24T08:06:02.981111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981111+0000","last_undegraded":"2023-11-24T08:06:02.981111+0000","last_fullsized":"2023-11-24T08:06:02.981111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.996512+0000","last_change":"2023-11-24T08:06:02.996512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.996512+0000","last_undegraded":"2023-11-24T08:06:02.996512+0000","last_fullsized":"2023-11-24T08:06:02.996512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988512+0000","last_change":"2023-11-24T08:06:02.988512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988512+0000","last_undegraded":"2023-11-24T08:06:02.988512+0000","last_fullsized":"2023-11-24T08:06:02.988512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000057+0000","last_change":"2023-11-24T08:06:03.000057+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000057+0000","last_undegraded":"2023-11-24T08:06:03.000057+0000","last_fullsized":"2023-11-24T08:06:03.000057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986477+0000","last_change":"2023-11-24T08:06:02.986477+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986477+0000","last_undegraded":"2023-11-24T08:06:02.986477+0000","last_fullsized":"2023-11-24T08:06:02.986477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011454+0000","last_change":"2023-11-24T08:06:03.008351+0000","last_active":"2023-11-24T08:06:03.011454+0000","last_peered":"2023-11-24T08:06:03.011454+0000","last_clean":"2023-11-24T08:06:03.011454+0000","last_became_active":"2023-11-24T08:06:03.008011+0000","last_became_peered":"2023-11-24T08:06:03.008011+0000","last_unstale":"2023-11-24T08:06:03.011454+0000","last_undegraded":"2023-11-24T08:06:03.011454+0000","last_fullsized":"2023-11-24T08:06:03.011454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986375+0000","last_change":"2023-11-24T08:06:02.986375+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986375+0000","last_undegraded":"2023-11-24T08:06:02.986375+0000","last_fullsized":"2023-11-24T08:06:02.986375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007823+0000","last_change":"2023-11-24T08:06:03.005602+0000","last_active":"2023-11-24T08:06:03.007823+0000","last_peered":"2023-11-24T08:06:03.007823+0000","last_clean":"2023-11-24T08:06:03.007823+0000","last_became_active":"2023-11-24T08:06:03.005346+0000","last_became_peered":"2023-11-24T08:06:03.005346+0000","last_unstale":"2023-11-24T08:06:03.007823+0000","last_undegraded":"2023-11-24T08:06:03.007823+0000","last_fullsized":"2023-11-24T08:06:03.007823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978376+0000","last_change":"2023-11-24T08:06:02.978376+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978376+0000","last_undegraded":"2023-11-24T08:06:02.978376+0000","last_fullsized":"2023-11-24T08:06:02.978376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004953+0000","last_change":"2023-11-24T08:06:02.997760+0000","last_active":"2023-11-24T08:06:03.004953+0000","last_peered":"2023-11-24T08:06:03.004953+0000","last_clean":"2023-11-24T08:06:03.004953+0000","last_became_active":"2023-11-24T08:06:02.997481+0000","last_became_peered":"2023-11-24T08:06:02.997481+0000","last_unstale":"2023-11-24T08:06:03.004953+0000","last_undegraded":"2023-11-24T08:06:03.004953+0000","last_fullsized":"2023-11-24T08:06:03.004953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992842+0000","last_change":"2023-11-24T08:06:02.992842+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992842+0000","last_undegraded":"2023-11-24T08:06:02.992842+0000","last_fullsized":"2023-11-24T08:06:02.992842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.999008+0000","last_change":"2023-11-24T08:06:02.997108+0000","last_active":"2023-11-24T08:06:02.999008+0000","last_peered":"2023-11-24T08:06:02.999008+0000","last_clean":"2023-11-24T08:06:02.999008+0000","last_became_active":"2023-11-24T08:06:02.996880+0000","last_became_peered":"2023-11-24T08:06:02.996880+0000","last_unstale":"2023-11-24T08:06:02.999008+0000","last_undegraded":"2023-11-24T08:06:02.999008+0000","last_fullsized":"2023-11-24T08:06:02.999008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995771+0000","last_change":"2023-11-24T08:06:02.995771+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995771+0000","last_undegraded":"2023-11-24T08:06:02.995771+0000","last_fullsized":"2023-11-24T08:06:02.995771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011211+0000","last_change":"2023-11-24T08:06:03.004832+0000","last_active":"2023-11-24T08:06:03.011211+0000","last_peered":"2023-11-24T08:06:03.011211+0000","last_clean":"2023-11-24T08:06:03.011211+0000","last_became_active":"2023-11-24T08:06:03.004536+0000","last_became_peered":"2023-11-24T08:06:03.004536+0000","last_unstale":"2023-11-24T08:06:03.011211+0000","last_undegraded":"2023-11-24T08:06:03.011211+0000","last_fullsized":"2023-11-24T08:06:03.011211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:06.344044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990140+0000","last_change":"2023-11-24T08:06:02.990140+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990140+0000","last_undegraded":"2023-11-24T08:06:02.990140+0000","last_fullsized":"2023-11-24T08:06:02.990140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005297+0000","last_change":"2023-11-24T08:06:02.993001+0000","last_active":"2023-11-24T08:06:03.005297+0000","last_peered":"2023-11-24T08:06:03.005297+0000","last_clean":"2023-11-24T08:06:03.005297+0000","last_became_active":"2023-11-24T08:06:02.992519+0000","last_became_peered":"2023-11-24T08:06:02.992519+0000","last_unstale":"2023-11-24T08:06:03.005297+0000","last_undegraded":"2023-11-24T08:06:03.005297+0000","last_fullsized":"2023-11-24T08:06:03.005297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:59.875793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.976833+0000","last_change":"2023-11-24T08:06:02.976833+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.976833+0000","last_undegraded":"2023-11-24T08:06:02.976833+0000","last_fullsized":"2023-11-24T08:06:02.976833+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.994102+0000","last_change":"2023-11-24T08:06:02.992653+0000","last_active":"2023-11-24T08:06:02.994102+0000","last_peered":"2023-11-24T08:06:02.994102+0000","last_clean":"2023-11-24T08:06:02.994102+0000","last_became_active":"2023-11-24T08:06:02.992420+0000","last_became_peered":"2023-11-24T08:06:02.992420+0000","last_unstale":"2023-11-24T08:06:02.994102+0000","last_undegraded":"2023-11-24T08:06:02.994102+0000","last_fullsized":"2023-11-24T08:06:02.994102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991684+0000","last_change":"2023-11-24T08:06:02.991684+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991684+0000","last_undegraded":"2023-11-24T08:06:02.991684+0000","last_fullsized":"2023-11-24T08:06:02.991684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005504+0000","last_change":"2023-11-24T08:06:02.998354+0000","last_active":"2023-11-24T08:06:03.005504+0000","last_peered":"2023-11-24T08:06:03.005504+0000","last_clean":"2023-11-24T08:06:03.005504+0000","last_became_active":"2023-11-24T08:06:02.998065+0000","last_became_peered":"2023-11-24T08:06:02.998065+0000","last_unstale":"2023-11-24T08:06:03.005504+0000","last_undegraded":"2023-11-24T08:06:03.005504+0000","last_fullsized":"2023-11-24T08:06:03.005504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:16:32.070719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993733+0000","last_change":"2023-11-24T08:06:02.993733+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993733+0000","last_undegraded":"2023-11-24T08:06:02.993733+0000","last_fullsized":"2023-11-24T08:06:02.993733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011199+0000","last_change":"2023-11-24T08:06:03.010255+0000","last_active":"2023-11-24T08:06:03.011199+0000","last_peered":"2023-11-24T08:06:03.011199+0000","last_clean":"2023-11-24T08:06:03.011199+0000","last_became_active":"2023-11-24T08:06:03.009943+0000","last_became_peered":"2023-11-24T08:06:03.009943+0000","last_unstale":"2023-11-24T08:06:03.011199+0000","last_undegraded":"2023-11-24T08:06:03.011199+0000","last_fullsized":"2023-11-24T08:06:03.011199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985887+0000","last_change":"2023-11-24T08:06:02.985887+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985887+0000","last_undegraded":"2023-11-24T08:06:02.985887+0000","last_fullsized":"2023-11-24T08:06:02.985887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006556+0000","last_change":"2023-11-24T08:06:03.004192+0000","last_active":"2023-11-24T08:06:03.006556+0000","last_peered":"2023-11-24T08:06:03.006556+0000","last_clean":"2023-11-24T08:06:03.006556+0000","last_became_active":"2023-11-24T08:06:03.003913+0000","last_became_peered":"2023-11-24T08:06:03.003913+0000","last_unstale":"2023-11-24T08:06:03.006556+0000","last_undegraded":"2023-11-24T08:06:03.006556+0000","last_fullsized":"2023-11-24T08:06:03.006556+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987547+0000","last_change":"2023-11-24T08:06:02.987547+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987547+0000","last_undegraded":"2023-11-24T08:06:02.987547+0000","last_fullsized":"2023-11-24T08:06:02.987547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011782+0000","last_change":"2023-11-24T08:06:03.009734+0000","last_active":"2023-11-24T08:06:03.011782+0000","last_peered":"2023-11-24T08:06:03.011782+0000","last_clean":"2023-11-24T08:06:03.011782+0000","last_became_active":"2023-11-24T08:06:03.009416+0000","last_became_peered":"2023-11-24T08:06:03.009416+0000","last_unstale":"2023-11-24T08:06:03.011782+0000","last_undegraded":"2023-11-24T08:06:03.011782+0000","last_fullsized":"2023-11-24T08:06:03.011782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992464+0000","last_change":"2023-11-24T08:06:02.992464+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992464+0000","last_undegraded":"2023-11-24T08:06:02.992464+0000","last_fullsized":"2023-11-24T08:06:02.992464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004874+0000","last_change":"2023-11-24T08:06:02.998808+0000","last_active":"2023-11-24T08:06:03.004874+0000","last_peered":"2023-11-24T08:06:03.004874+0000","last_clean":"2023-11-24T08:06:03.004874+0000","last_became_active":"2023-11-24T08:06:02.998600+0000","last_became_peered":"2023-11-24T08:06:02.998600+0000","last_unstale":"2023-11-24T08:06:03.004874+0000","last_undegraded":"2023-11-24T08:06:03.004874+0000","last_fullsized":"2023-11-24T08:06:03.004874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989274+0000","last_change":"2023-11-24T08:06:02.989274+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989274+0000","last_undegraded":"2023-11-24T08:06:02.989274+0000","last_fullsized":"2023-11-24T08:06:02.989274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010263+0000","last_change":"2023-11-24T08:06:03.004639+0000","last_active":"2023-11-24T08:06:03.010263+0000","last_peered":"2023-11-24T08:06:03.010263+0000","last_clean":"2023-11-24T08:06:03.010263+0000","last_became_active":"2023-11-24T08:06:03.004341+0000","last_became_peered":"2023-11-24T08:06:03.004341+0000","last_unstale":"2023-11-24T08:06:03.010263+0000","last_undegraded":"2023-11-24T08:06:03.010263+0000","last_fullsized":"2023-11-24T08:06:03.010263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:46.680770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986105+0000","last_change":"2023-11-24T08:06:02.986105+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986105+0000","last_undegraded":"2023-11-24T08:06:02.986105+0000","last_fullsized":"2023-11-24T08:06:02.986105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008069+0000","last_change":"2023-11-24T08:06:03.005631+0000","last_active":"2023-11-24T08:06:03.008069+0000","last_peered":"2023-11-24T08:06:03.008069+0000","last_clean":"2023-11-24T08:06:03.008069+0000","last_became_active":"2023-11-24T08:06:03.004131+0000","last_became_peered":"2023-11-24T08:06:03.004131+0000","last_unstale":"2023-11-24T08:06:03.008069+0000","last_undegraded":"2023-11-24T08:06:03.008069+0000","last_fullsized":"2023-11-24T08:06:03.008069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999457+0000","last_change":"2023-11-24T08:06:02.999457+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999457+0000","last_undegraded":"2023-11-24T08:06:02.999457+0000","last_fullsized":"2023-11-24T08:06:02.999457+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011168+0000","last_change":"2023-11-24T08:06:03.009985+0000","last_active":"2023-11-24T08:06:03.011168+0000","last_peered":"2023-11-24T08:06:03.011168+0000","last_clean":"2023-11-24T08:06:03.011168+0000","last_became_active":"2023-11-24T08:06:03.009676+0000","last_became_peered":"2023-11-24T08:06:03.009676+0000","last_unstale":"2023-11-24T08:06:03.011168+0000","last_undegraded":"2023-11-24T08:06:03.011168+0000","last_fullsized":"2023-11-24T08:06:03.011168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987482+0000","last_change":"2023-11-24T08:06:02.987482+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987482+0000","last_undegraded":"2023-11-24T08:06:02.987482+0000","last_fullsized":"2023-11-24T08:06:02.987482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004281+0000","last_change":"2023-11-24T08:06:02.996472+0000","last_active":"2023-11-24T08:06:03.004281+0000","last_peered":"2023-11-24T08:06:03.004281+0000","last_clean":"2023-11-24T08:06:03.004281+0000","last_became_active":"2023-11-24T08:06:02.996105+0000","last_became_peered":"2023-11-24T08:06:02.996105+0000","last_unstale":"2023-11-24T08:06:03.004281+0000","last_undegraded":"2023-11-24T08:06:03.004281+0000","last_fullsized":"2023-11-24T08:06:03.004281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:14.099330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980014+0000","last_change":"2023-11-24T08:06:02.980014+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980014+0000","last_undegraded":"2023-11-24T08:06:02.980014+0000","last_fullsized":"2023-11-24T08:06:02.980014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008394+0000","last_change":"2023-11-24T08:06:03.006947+0000","last_active":"2023-11-24T08:06:03.008394+0000","last_peered":"2023-11-24T08:06:03.008394+0000","last_clean":"2023-11-24T08:06:03.008394+0000","last_became_active":"2023-11-24T08:06:03.006640+0000","last_became_peered":"2023-11-24T08:06:03.006640+0000","last_unstale":"2023-11-24T08:06:03.008394+0000","last_undegraded":"2023-11-24T08:06:03.008394+0000","last_fullsized":"2023-11-24T08:06:03.008394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995208+0000","last_change":"2023-11-24T08:06:02.995208+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995208+0000","last_undegraded":"2023-11-24T08:06:02.995208+0000","last_fullsized":"2023-11-24T08:06:02.995208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007244+0000","last_change":"2023-11-24T08:06:03.003338+0000","last_active":"2023-11-24T08:06:03.007244+0000","last_peered":"2023-11-24T08:06:03.007244+0000","last_clean":"2023-11-24T08:06:03.007244+0000","last_became_active":"2023-11-24T08:06:02.998184+0000","last_became_peered":"2023-11-24T08:06:02.998184+0000","last_unstale":"2023-11-24T08:06:03.007244+0000","last_undegraded":"2023-11-24T08:06:03.007244+0000","last_fullsized":"2023-11-24T08:06:03.007244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989003+0000","last_change":"2023-11-24T08:06:02.989003+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989003+0000","last_undegraded":"2023-11-24T08:06:02.989003+0000","last_fullsized":"2023-11-24T08:06:02.989003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006462+0000","last_change":"2023-11-24T08:06:03.001694+0000","last_active":"2023-11-24T08:06:03.006462+0000","last_peered":"2023-11-24T08:06:03.006462+0000","last_clean":"2023-11-24T08:06:03.006462+0000","last_became_active":"2023-11-24T08:06:02.998616+0000","last_became_peered":"2023-11-24T08:06:02.998616+0000","last_unstale":"2023-11-24T08:06:03.006462+0000","last_undegraded":"2023-11-24T08:06:03.006462+0000","last_fullsized":"2023-11-24T08:06:03.006462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:30.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986501+0000","last_change":"2023-11-24T08:06:02.986501+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986501+0000","last_undegraded":"2023-11-24T08:06:02.986501+0000","last_fullsized":"2023-11-24T08:06:02.986501+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.003102+0000","last_change":"2023-11-24T08:06:02.993516+0000","last_active":"2023-11-24T08:06:03.003102+0000","last_peered":"2023-11-24T08:06:03.003102+0000","last_clean":"2023-11-24T08:06:03.003102+0000","last_became_active":"2023-11-24T08:06:02.992976+0000","last_became_peered":"2023-11-24T08:06:02.992976+0000","last_unstale":"2023-11-24T08:06:03.003102+0000","last_undegraded":"2023-11-24T08:06:03.003102+0000","last_fullsized":"2023-11-24T08:06:03.003102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:05:42.068467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985726+0000","last_change":"2023-11-24T08:06:02.985726+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985726+0000","last_undegraded":"2023-11-24T08:06:02.985726+0000","last_fullsized":"2023-11-24T08:06:02.985726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.000770+0000","last_change":"2023-11-24T08:06:02.998617+0000","last_active":"2023-11-24T08:06:03.000770+0000","last_peered":"2023-11-24T08:06:03.000770+0000","last_clean":"2023-11-24T08:06:03.000770+0000","last_became_active":"2023-11-24T08:06:02.998393+0000","last_became_peered":"2023-11-24T08:06:02.998393+0000","last_unstale":"2023-11-24T08:06:03.000770+0000","last_undegraded":"2023-11-24T08:06:03.000770+0000","last_fullsized":"2023-11-24T08:06:03.000770+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985538+0000","last_change":"2023-11-24T08:06:02.985538+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985538+0000","last_undegraded":"2023-11-24T08:06:02.985538+0000","last_fullsized":"2023-11-24T08:06:02.985538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010412+0000","last_change":"2023-11-24T08:06:03.005058+0000","last_active":"2023-11-24T08:06:03.010412+0000","last_peered":"2023-11-24T08:06:03.010412+0000","last_clean":"2023-11-24T08:06:03.010412+0000","last_became_active":"2023-11-24T08:06:03.004703+0000","last_became_peered":"2023-11-24T08:06:03.004703+0000","last_unstale":"2023-11-24T08:06:03.010412+0000","last_undegraded":"2023-11-24T08:06:03.010412+0000","last_fullsized":"2023-11-24T08:06:03.010412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985668+0000","last_change":"2023-11-24T08:06:02.985668+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985668+0000","last_undegraded":"2023-11-24T08:06:02.985668+0000","last_fullsized":"2023-11-24T08:06:02.985668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007000+0000","last_change":"2023-11-24T08:06:03.003128+0000","last_active":"2023-11-24T08:06:03.007000+0000","last_peered":"2023-11-24T08:06:03.007000+0000","last_clean":"2023-11-24T08:06:03.007000+0000","last_became_active":"2023-11-24T08:06:03.002791+0000","last_became_peered":"2023-11-24T08:06:03.002791+0000","last_unstale":"2023-11-24T08:06:03.007000+0000","last_undegraded":"2023-11-24T08:06:03.007000+0000","last_fullsized":"2023-11-24T08:06:03.007000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:49.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995875+0000","last_change":"2023-11-24T08:06:02.995875+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995875+0000","last_undegraded":"2023-11-24T08:06:02.995875+0000","last_fullsized":"2023-11-24T08:06:02.995875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011002+0000","last_change":"2023-11-24T08:06:03.006761+0000","last_active":"2023-11-24T08:06:03.011002+0000","last_peered":"2023-11-24T08:06:03.011002+0000","last_clean":"2023-11-24T08:06:03.011002+0000","last_became_active":"2023-11-24T08:06:03.006418+0000","last_became_peered":"2023-11-24T08:06:03.006418+0000","last_unstale":"2023-11-24T08:06:03.011002+0000","last_undegraded":"2023-11-24T08:06:03.011002+0000","last_fullsized":"2023-11-24T08:06:03.011002+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988366+0000","last_change":"2023-11-24T08:06:02.988366+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988366+0000","last_undegraded":"2023-11-24T08:06:02.988366+0000","last_fullsized":"2023-11-24T08:06:02.988366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010239+0000","last_change":"2023-11-24T08:06:03.008262+0000","last_active":"2023-11-24T08:06:03.010239+0000","last_peered":"2023-11-24T08:06:03.010239+0000","last_clean":"2023-11-24T08:06:03.010239+0000","last_became_active":"2023-11-24T08:06:03.007982+0000","last_became_peered":"2023-11-24T08:06:03.007982+0000","last_unstale":"2023-11-24T08:06:03.010239+0000","last_undegraded":"2023-11-24T08:06:03.010239+0000","last_fullsized":"2023-11-24T08:06:03.010239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994570+0000","last_change":"2023-11-24T08:06:02.994570+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994570+0000","last_undegraded":"2023-11-24T08:06:02.994570+0000","last_fullsized":"2023-11-24T08:06:02.994570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006539+0000","last_change":"2023-11-24T08:06:03.001269+0000","last_active":"2023-11-24T08:06:03.006539+0000","last_peered":"2023-11-24T08:06:03.006539+0000","last_clean":"2023-11-24T08:06:03.006539+0000","last_became_active":"2023-11-24T08:06:03.000568+0000","last_became_peered":"2023-11-24T08:06:03.000568+0000","last_unstale":"2023-11-24T08:06:03.006539+0000","last_undegraded":"2023-11-24T08:06:03.006539+0000","last_fullsized":"2023-11-24T08:06:03.006539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987028+0000","last_change":"2023-11-24T08:06:02.987028+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987028+0000","last_undegraded":"2023-11-24T08:06:02.987028+0000","last_fullsized":"2023-11-24T08:06:02.987028+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.998436+0000","last_change":"2023-11-24T08:06:02.995680+0000","last_active":"2023-11-24T08:06:02.998436+0000","last_peered":"2023-11-24T08:06:02.998436+0000","last_clean":"2023-11-24T08:06:02.998436+0000","last_became_active":"2023-11-24T08:06:02.995421+0000","last_became_peered":"2023-11-24T08:06:02.995421+0000","last_unstale":"2023-11-24T08:06:02.998436+0000","last_undegraded":"2023-11-24T08:06:02.998436+0000","last_fullsized":"2023-11-24T08:06:02.998436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985166+0000","last_change":"2023-11-24T08:06:02.985166+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985166+0000","last_undegraded":"2023-11-24T08:06:02.985166+0000","last_fullsized":"2023-11-24T08:06:02.985166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004405+0000","last_change":"2023-11-24T08:06:03.000694+0000","last_active":"2023-11-24T08:06:03.004405+0000","last_peered":"2023-11-24T08:06:03.004405+0000","last_clean":"2023-11-24T08:06:03.004405+0000","last_became_active":"2023-11-24T08:06:03.000285+0000","last_became_peered":"2023-11-24T08:06:03.000285+0000","last_unstale":"2023-11-24T08:06:03.004405+0000","last_undegraded":"2023-11-24T08:06:03.004405+0000","last_fullsized":"2023-11-24T08:06:03.004405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:44.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.983698+0000","last_change":"2023-11-24T08:06:02.983698+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.983698+0000","last_undegraded":"2023-11-24T08:06:02.983698+0000","last_fullsized":"2023-11-24T08:06:02.983698+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010558+0000","last_change":"2023-11-24T08:06:03.008992+0000","last_active":"2023-11-24T08:06:03.010558+0000","last_peered":"2023-11-24T08:06:03.010558+0000","last_clean":"2023-11-24T08:06:03.010558+0000","last_became_active":"2023-11-24T08:06:03.008648+0000","last_became_peered":"2023-11-24T08:06:03.008648+0000","last_unstale":"2023-11-24T08:06:03.010558+0000","last_undegraded":"2023-11-24T08:06:03.010558+0000","last_fullsized":"2023-11-24T08:06:03.010558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981885+0000","last_change":"2023-11-24T08:06:02.981885+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981885+0000","last_undegraded":"2023-11-24T08:06:02.981885+0000","last_fullsized":"2023-11-24T08:06:02.981885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007305+0000","last_change":"2023-11-24T08:06:03.003843+0000","last_active":"2023-11-24T08:06:03.007305+0000","last_peered":"2023-11-24T08:06:03.007305+0000","last_clean":"2023-11-24T08:06:03.007305+0000","last_became_active":"2023-11-24T08:06:03.002741+0000","last_became_peered":"2023-11-24T08:06:03.002741+0000","last_unstale":"2023-11-24T08:06:03.007305+0000","last_undegraded":"2023-11-24T08:06:03.007305+0000","last_fullsized":"2023-11-24T08:06:03.007305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:46:14.437116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980343+0000","last_change":"2023-11-24T08:06:02.980343+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980343+0000","last_undegraded":"2023-11-24T08:06:02.980343+0000","last_fullsized":"2023-11-24T08:06:02.980343+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011146+0000","last_change":"2023-11-24T08:06:03.009978+0000","last_active":"2023-11-24T08:06:03.011146+0000","last_peered":"2023-11-24T08:06:03.011146+0000","last_clean":"2023-11-24T08:06:03.011146+0000","last_became_active":"2023-11-24T08:06:03.009694+0000","last_became_peered":"2023-11-24T08:06:03.009694+0000","last_unstale":"2023-11-24T08:06:03.011146+0000","last_undegraded":"2023-11-24T08:06:03.011146+0000","last_fullsized":"2023-11-24T08:06:03.011146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999828+0000","last_change":"2023-11-24T08:06:02.999828+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999828+0000","last_undegraded":"2023-11-24T08:06:02.999828+0000","last_fullsized":"2023-11-24T08:06:02.999828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011405+0000","last_change":"2023-11-24T08:06:03.008297+0000","last_active":"2023-11-24T08:06:03.011405+0000","last_peered":"2023-11-24T08:06:03.011405+0000","last_clean":"2023-11-24T08:06:03.011405+0000","last_became_active":"2023-11-24T08:06:03.007985+0000","last_became_peered":"2023-11-24T08:06:03.007985+0000","last_unstale":"2023-11-24T08:06:03.011405+0000","last_undegraded":"2023-11-24T08:06:03.011405+0000","last_fullsized":"2023-11-24T08:06:03.011405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-24T08:06:01.972672+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:06:01.972672+0000","last_peered":"2023-11-24T08:06:01.972672+0000","last_clean":"2023-11-24T08:06:01.972672+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:06:01.972672+0000","last_undegraded":"2023-11-24T08:06:01.972672+0000","last_fullsized":"2023-11-24T08:06:01.972672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93100000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.048}]},{"osd":2,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.177},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":1.643}]},{"osd":3,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.486},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.365}]},{"osd":4,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.302},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.522},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.623}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.345},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.607}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":1.089}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.4370000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.661}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.639}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.585}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.90100000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.353}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.79200000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.286},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.321}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4019999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.127}]},{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.3879999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.4379999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.546},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.617}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.5899999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":1.335}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.504},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.5700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.2789999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.232},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.0509999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.623},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.3180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.359}]},{"osd":6,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.446},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.4099999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.496},{"interface":"front","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.536}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.83799999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.98599999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.2829999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.214}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.597,"15min":1.597},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3809999999999998,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.0940000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.87},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.93799999999999994}]},{"osd":2,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.2529999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":1.05}]},{"osd":6,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.9359999999999999,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3540000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.159}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.72899999999999998},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83499999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.877}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.0780000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.115},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.0920000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:06:08.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:06:09.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:06:09.087 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.106+0000 7fdc5ab27640 1 Processor -- start 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.106+0000 7fdc5ab27640 1 -- start start 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.107+0000 7fdc5ab27640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc540d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.107+0000 7fdc5ab27640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc540b2d00 con 0x7fdc5415d3e0 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.107+0000 7fdc5ab27640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc540b2a20 con 0x7fdc540d36c0 2023-11-24T08:06:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.107+0000 7fdc5ab27640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540d1370 con 0x7fdc5415d7c0 2023-11-24T08:06:09.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc59b25640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdc5415d3e0 0x7fdc540d2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58872/0 (socket says 172.21.15.107:58872) 2023-11-24T08:06:09.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc59b25640 1 -- 172.21.15.107:0/4161538309 learned_addr learned my addr 172.21.15.107:0/4161538309 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:09.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc59324640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc540d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333328741 0 0) 0x7fdc540b2d00 con 0x7fdc5415d3e0 2023-11-24T08:06:09.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc40003520 con 0x7fdc5415d3e0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070689331 0 0) 0x7fdc540b2a20 con 0x7fdc540d36c0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.108+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc540b2d00 con 0x7fdc540d36c0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587299744 0 0) 0x7fdc40003520 con 0x7fdc5415d3e0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 msgr2=0x7fdc540d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc540d0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 >> v1:172.21.15.120:6789/0 conn(0x7fdc540d36c0 legacy=0x7fdc540cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc540b3220 con 0x7fdc5415d3e0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdc50001670 con 0x7fdc5415d3e0 2023-11-24T08:06:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 >> v1:172.21.15.107:6789/0 conn(0x7fdc5415d3e0 legacy=0x7fdc540d2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc40003b30 0x7fdc40003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc400047c0 0x7fdc40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc400051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540d1370 con 0x7fdc40003b30 2023-11-24T08:06:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4000bb10 con 0x7fdc400047c0 2023-11-24T08:06:09.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4000bca0 con 0x7fdc5415d7c0 2023-11-24T08:06:09.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59b25640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc40003b30 0x7fdc40003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59324640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc400047c0 0x7fdc40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc5a326640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc400051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59b25640 1 -- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 msgr2=0x7fdc400051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59b25640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5415d7c0 0x7fdc400051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59b25640 1 -- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc400047c0 msgr2=0x7fdc40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.109+0000 7fdc59b25640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc400047c0 0x7fdc40004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.110+0000 7fdc59b25640 1 -- 172.21.15.107:0/4161538309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40003520 con 0x7fdc40003b30 2023-11-24T08:06:09.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.111+0000 7fdc59b25640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc40003b30 0x7fdc40003f30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdc50007fc0 tx=0x7fdc400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d221de2d127ea80a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.111+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50005ad0 con 0x7fdc40003b30 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.111+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc50005c30 con 0x7fdc40003b30 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.111+0000 7fdc58b23640 1 -- 172.21.15.107:0/4161538309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5002c7c0 con 0x7fdc40003b30 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 -- 172.21.15.107:0/4161538309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc40003b30 msgr2=0x7fdc40003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/4161538309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc40003b30 0x7fdc40003f30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdc50007fc0 tx=0x7fdc400057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5a326640 1 -- 172.21.15.107:0/4161538309 reap_dead start 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 -- 172.21.15.107:0/4161538309 shutdown_connections 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 -- 172.21.15.107:0/4161538309 >> 172.21.15.107:0/4161538309 conn(0x7fdc5408cc70 msgr2=0x7fdc5409c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 -- 172.21.15.107:0/4161538309 shutdown_connections 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.112+0000 7fdc5ab27640 1 -- 172.21.15.107:0/4161538309 wait complete. 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.114+0000 7fdc5ab27640 1 Processor -- start 2023-11-24T08:06:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.114+0000 7fdc5ab27640 1 -- start start 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc40003b30 0x7fdc54150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 0x7fdc541508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541f6df0 con 0x7fdc540c8450 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5415dba0 con 0x7fdc541f6970 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5ab27640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5415ddd0 con 0x7fdc40003b30 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc59324640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc59324640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43338/0 (socket says 172.21.15.107:43338) 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc59324640 1 -- 172.21.15.107:0/1444265489 learned_addr learned my addr 172.21.15.107:0/1444265489 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.115+0000 7fdc5a326640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 0x7fdc541508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59b25640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc40003b30 0x7fdc54150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 -- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc40003b30 msgr2=0x7fdc54150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc40003b30 0x7fdc54150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 -- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 msgr2=0x7fdc541508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 0x7fdc541508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 -- 172.21.15.107:0/1444265489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5415e0b0 con 0x7fdc540c8450 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc5a326640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 0x7fdc541508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:06:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc59324640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdc44011630 tx=0x7fdc44011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400f920 con 0x7fdc540c8450 2023-11-24T08:06:09.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4400fa80 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc540c7c50 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.116+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400fdf0 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.117+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc540ca5a0 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.118+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5404e840 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.119+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdc44015460 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.120+0000 7fdc4a7fc640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 0x7fdc3806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.120+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc44089b70 con 0x7fdc540c8450 2023-11-24T08:06:09.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.120+0000 7fdc59b25640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 0x7fdc3806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.121+0000 7fdc59b25640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 0x7fdc3806d300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc50034000 tx=0x7fdc50009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.124+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc44054b80 con 0x7fdc540c8450 2023-11-24T08:06:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.328+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc5415e2f0 con 0x7fdc3806ae40 2023-11-24T08:06:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.337+0000 7fdc4a7fc640 1 -- 172.21.15.107:0/1444265489 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281463 (secure 0 0 0) 0x7fdc5415e2f0 con 0x7fdc3806ae40 2023-11-24T08:06:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:06:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 msgr2=0x7fdc3806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 0x7fdc3806d300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc50034000 tx=0x7fdc50009030 comp rx=0 tx=0).stop 2023-11-24T08:06:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 msgr2=0x7fdc541f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdc44011630 tx=0x7fdc44011af0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 shutdown_connections 2023-11-24T08:06:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc3806ae40 0x7fdc3806d300 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc541f6970 0x7fdc541508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc540c8450 0x7fdc541f6430 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 --2- 172.21.15.107:0/1444265489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc40003b30 0x7fdc54150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 >> 172.21.15.107:0/1444265489 conn(0x7fdc5408cc70 msgr2=0x7fdc540b4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 shutdown_connections 2023-11-24T08:06:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.342+0000 7fdc5ab27640 1 -- 172.21.15.107:0/1444265489 wait complete. 2023-11-24T08:06:09.364 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-11-24T08:06:07.454840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3904,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3997696,"data_stored":2253726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.501171"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988645+0000","last_change":"2023-11-24T08:06:02.988645+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988645+0000","last_undegraded":"2023-11-24T08:06:02.988645+0000","last_fullsized":"2023-11-24T08:06:02.988645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990877+0000","last_change":"2023-11-24T08:06:02.990877+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990877+0000","last_undegraded":"2023-11-24T08:06:02.990877+0000","last_fullsized":"2023-11-24T08:06:02.990877+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991622+0000","last_change":"2023-11-24T08:06:02.991622+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991622+0000","last_undegraded":"2023-11-24T08:06:02.991622+0000","last_fullsized":"2023-11-24T08:06:02.991622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989545+0000","last_change":"2023-11-24T08:06:02.989545+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989545+0000","last_undegraded":"2023-11-24T08:06:02.989545+0000","last_fullsized":"2023-11-24T08:06:02.989545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987313+0000","last_change":"2023-11-24T08:06:02.987313+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987313+0000","last_undegraded":"2023-11-24T08:06:02.987313+0000","last_fullsized":"2023-11-24T08:06:02.987313+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997609+0000","last_change":"2023-11-24T08:06:02.997609+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997609+0000","last_undegraded":"2023-11-24T08:06:02.997609+0000","last_fullsized":"2023-11-24T08:06:02.997609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995334+0000","last_change":"2023-11-24T08:06:02.995334+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995334+0000","last_undegraded":"2023-11-24T08:06:02.995334+0000","last_fullsized":"2023-11-24T08:06:02.995334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.002111+0000","last_change":"2023-11-24T08:06:03.002111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.002111+0000","last_undegraded":"2023-11-24T08:06:03.002111+0000","last_fullsized":"2023-11-24T08:06:03.002111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978774+0000","last_change":"2023-11-24T08:06:02.978774+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978774+0000","last_undegraded":"2023-11-24T08:06:02.978774+0000","last_fullsized":"2023-11-24T08:06:02.978774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993312+0000","last_change":"2023-11-24T08:06:02.993312+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993312+0000","last_undegraded":"2023-11-24T08:06:02.993312+0000","last_fullsized":"2023-11-24T08:06:02.993312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000463+0000","last_change":"2023-11-24T08:06:03.000463+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000463+0000","last_undegraded":"2023-11-24T08:06:03.000463+0000","last_fullsized":"2023-11-24T08:06:03.000463+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990332+0000","last_change":"2023-11-24T08:06:02.990332+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990332+0000","last_undegraded":"2023-11-24T08:06:02.990332+0000","last_fullsized":"2023-11-24T08:06:02.990332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986467+0000","last_change":"2023-11-24T08:06:02.986467+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986467+0000","last_undegraded":"2023-11-24T08:06:02.986467+0000","last_fullsized":"2023-11-24T08:06:02.986467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991919+0000","last_change":"2023-11-24T08:06:02.991919+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991919+0000","last_undegraded":"2023-11-24T08:06:02.991919+0000","last_fullsized":"2023-11-24T08:06:02.991919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994432+0000","last_change":"2023-11-24T08:06:02.994432+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994432+0000","last_undegraded":"2023-11-24T08:06:02.994432+0000","last_fullsized":"2023-11-24T08:06:02.994432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995541+0000","last_change":"2023-11-24T08:06:02.995541+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995541+0000","last_undegraded":"2023-11-24T08:06:02.995541+0000","last_fullsized":"2023-11-24T08:06:02.995541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000531+0000","last_change":"2023-11-24T08:06:03.000531+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000531+0000","last_undegraded":"2023-11-24T08:06:03.000531+0000","last_fullsized":"2023-11-24T08:06:03.000531+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990434+0000","last_change":"2023-11-24T08:06:02.990434+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990434+0000","last_undegraded":"2023-11-24T08:06:02.990434+0000","last_fullsized":"2023-11-24T08:06:02.990434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989472+0000","last_change":"2023-11-24T08:06:02.989472+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989472+0000","last_undegraded":"2023-11-24T08:06:02.989472+0000","last_fullsized":"2023-11-24T08:06:02.989472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989920+0000","last_change":"2023-11-24T08:06:02.989920+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989920+0000","last_undegraded":"2023-11-24T08:06:02.989920+0000","last_fullsized":"2023-11-24T08:06:02.989920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982150+0000","last_change":"2023-11-24T08:06:02.982150+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982150+0000","last_undegraded":"2023-11-24T08:06:02.982150+0000","last_fullsized":"2023-11-24T08:06:02.982150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.977577+0000","last_change":"2023-11-24T08:06:02.977577+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.977577+0000","last_undegraded":"2023-11-24T08:06:02.977577+0000","last_fullsized":"2023-11-24T08:06:02.977577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.982512+0000","last_change":"2023-11-24T08:06:02.982512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.982512+0000","last_undegraded":"2023-11-24T08:06:02.982512+0000","last_fullsized":"2023-11-24T08:06:02.982512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989460+0000","last_change":"2023-11-24T08:06:02.989460+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989460+0000","last_undegraded":"2023-11-24T08:06:02.989460+0000","last_fullsized":"2023-11-24T08:06:02.989460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.997244+0000","last_change":"2023-11-24T08:06:02.997244+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.997244+0000","last_undegraded":"2023-11-24T08:06:02.997244+0000","last_fullsized":"2023-11-24T08:06:02.997244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989480+0000","last_change":"2023-11-24T08:06:02.989480+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989480+0000","last_undegraded":"2023-11-24T08:06:02.989480+0000","last_fullsized":"2023-11-24T08:06:02.989480+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993720+0000","last_change":"2023-11-24T08:06:02.993720+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993720+0000","last_undegraded":"2023-11-24T08:06:02.993720+0000","last_fullsized":"2023-11-24T08:06:02.993720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989000+0000","last_change":"2023-11-24T08:06:02.989000+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989000+0000","last_undegraded":"2023-11-24T08:06:02.989000+0000","last_fullsized":"2023-11-24T08:06:02.989000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981111+0000","last_change":"2023-11-24T08:06:02.981111+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981111+0000","last_undegraded":"2023-11-24T08:06:02.981111+0000","last_fullsized":"2023-11-24T08:06:02.981111+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.996512+0000","last_change":"2023-11-24T08:06:02.996512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.996512+0000","last_undegraded":"2023-11-24T08:06:02.996512+0000","last_fullsized":"2023-11-24T08:06:02.996512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988512+0000","last_change":"2023-11-24T08:06:02.988512+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988512+0000","last_undegraded":"2023-11-24T08:06:02.988512+0000","last_fullsized":"2023-11-24T08:06:02.988512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:03.000057+0000","last_change":"2023-11-24T08:06:03.000057+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:03.000057+0000","last_undegraded":"2023-11-24T08:06:03.000057+0000","last_fullsized":"2023-11-24T08:06:03.000057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986477+0000","last_change":"2023-11-24T08:06:02.986477+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986477+0000","last_undegraded":"2023-11-24T08:06:02.986477+0000","last_fullsized":"2023-11-24T08:06:02.986477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011454+0000","last_change":"2023-11-24T08:06:03.008351+0000","last_active":"2023-11-24T08:06:03.011454+0000","last_peered":"2023-11-24T08:06:03.011454+0000","last_clean":"2023-11-24T08:06:03.011454+0000","last_became_active":"2023-11-24T08:06:03.008011+0000","last_became_peered":"2023-11-24T08:06:03.008011+0000","last_unstale":"2023-11-24T08:06:03.011454+0000","last_undegraded":"2023-11-24T08:06:03.011454+0000","last_fullsized":"2023-11-24T08:06:03.011454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986375+0000","last_change":"2023-11-24T08:06:02.986375+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986375+0000","last_undegraded":"2023-11-24T08:06:02.986375+0000","last_fullsized":"2023-11-24T08:06:02.986375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007823+0000","last_change":"2023-11-24T08:06:03.005602+0000","last_active":"2023-11-24T08:06:03.007823+0000","last_peered":"2023-11-24T08:06:03.007823+0000","last_clean":"2023-11-24T08:06:03.007823+0000","last_became_active":"2023-11-24T08:06:03.005346+0000","last_became_peered":"2023-11-24T08:06:03.005346+0000","last_unstale":"2023-11-24T08:06:03.007823+0000","last_undegraded":"2023-11-24T08:06:03.007823+0000","last_fullsized":"2023-11-24T08:06:03.007823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.978376+0000","last_change":"2023-11-24T08:06:02.978376+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.978376+0000","last_undegraded":"2023-11-24T08:06:02.978376+0000","last_fullsized":"2023-11-24T08:06:02.978376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004953+0000","last_change":"2023-11-24T08:06:02.997760+0000","last_active":"2023-11-24T08:06:03.004953+0000","last_peered":"2023-11-24T08:06:03.004953+0000","last_clean":"2023-11-24T08:06:03.004953+0000","last_became_active":"2023-11-24T08:06:02.997481+0000","last_became_peered":"2023-11-24T08:06:02.997481+0000","last_unstale":"2023-11-24T08:06:03.004953+0000","last_undegraded":"2023-11-24T08:06:03.004953+0000","last_fullsized":"2023-11-24T08:06:03.004953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992842+0000","last_change":"2023-11-24T08:06:02.992842+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992842+0000","last_undegraded":"2023-11-24T08:06:02.992842+0000","last_fullsized":"2023-11-24T08:06:02.992842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.999008+0000","last_change":"2023-11-24T08:06:02.997108+0000","last_active":"2023-11-24T08:06:02.999008+0000","last_peered":"2023-11-24T08:06:02.999008+0000","last_clean":"2023-11-24T08:06:02.999008+0000","last_became_active":"2023-11-24T08:06:02.996880+0000","last_became_peered":"2023-11-24T08:06:02.996880+0000","last_unstale":"2023-11-24T08:06:02.999008+0000","last_undegraded":"2023-11-24T08:06:02.999008+0000","last_fullsized":"2023-11-24T08:06:02.999008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995771+0000","last_change":"2023-11-24T08:06:02.995771+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995771+0000","last_undegraded":"2023-11-24T08:06:02.995771+0000","last_fullsized":"2023-11-24T08:06:02.995771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011211+0000","last_change":"2023-11-24T08:06:03.004832+0000","last_active":"2023-11-24T08:06:03.011211+0000","last_peered":"2023-11-24T08:06:03.011211+0000","last_clean":"2023-11-24T08:06:03.011211+0000","last_became_active":"2023-11-24T08:06:03.004536+0000","last_became_peered":"2023-11-24T08:06:03.004536+0000","last_unstale":"2023-11-24T08:06:03.011211+0000","last_undegraded":"2023-11-24T08:06:03.011211+0000","last_fullsized":"2023-11-24T08:06:03.011211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:06.344044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.990140+0000","last_change":"2023-11-24T08:06:02.990140+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.990140+0000","last_undegraded":"2023-11-24T08:06:02.990140+0000","last_fullsized":"2023-11-24T08:06:02.990140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005297+0000","last_change":"2023-11-24T08:06:02.993001+0000","last_active":"2023-11-24T08:06:03.005297+0000","last_peered":"2023-11-24T08:06:03.005297+0000","last_clean":"2023-11-24T08:06:03.005297+0000","last_became_active":"2023-11-24T08:06:02.992519+0000","last_became_peered":"2023-11-24T08:06:02.992519+0000","last_unstale":"2023-11-24T08:06:03.005297+0000","last_undegraded":"2023-11-24T08:06:03.005297+0000","last_fullsized":"2023-11-24T08:06:03.005297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:59.875793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.976833+0000","last_change":"2023-11-24T08:06:02.976833+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.976833+0000","last_undegraded":"2023-11-24T08:06:02.976833+0000","last_fullsized":"2023-11-24T08:06:02.976833+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.994102+0000","last_change":"2023-11-24T08:06:02.992653+0000","last_active":"2023-11-24T08:06:02.994102+0000","last_peered":"2023-11-24T08:06:02.994102+0000","last_clean":"2023-11-24T08:06:02.994102+0000","last_became_active":"2023-11-24T08:06:02.992420+0000","last_became_peered":"2023-11-24T08:06:02.992420+0000","last_unstale":"2023-11-24T08:06:02.994102+0000","last_undegraded":"2023-11-24T08:06:02.994102+0000","last_fullsized":"2023-11-24T08:06:02.994102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.991684+0000","last_change":"2023-11-24T08:06:02.991684+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.991684+0000","last_undegraded":"2023-11-24T08:06:02.991684+0000","last_fullsized":"2023-11-24T08:06:02.991684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.005504+0000","last_change":"2023-11-24T08:06:02.998354+0000","last_active":"2023-11-24T08:06:03.005504+0000","last_peered":"2023-11-24T08:06:03.005504+0000","last_clean":"2023-11-24T08:06:03.005504+0000","last_became_active":"2023-11-24T08:06:02.998065+0000","last_became_peered":"2023-11-24T08:06:02.998065+0000","last_unstale":"2023-11-24T08:06:03.005504+0000","last_undegraded":"2023-11-24T08:06:03.005504+0000","last_fullsized":"2023-11-24T08:06:03.005504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:16:32.070719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.993733+0000","last_change":"2023-11-24T08:06:02.993733+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.993733+0000","last_undegraded":"2023-11-24T08:06:02.993733+0000","last_fullsized":"2023-11-24T08:06:02.993733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011199+0000","last_change":"2023-11-24T08:06:03.010255+0000","last_active":"2023-11-24T08:06:03.011199+0000","last_peered":"2023-11-24T08:06:03.011199+0000","last_clean":"2023-11-24T08:06:03.011199+0000","last_became_active":"2023-11-24T08:06:03.009943+0000","last_became_peered":"2023-11-24T08:06:03.009943+0000","last_unstale":"2023-11-24T08:06:03.011199+0000","last_undegraded":"2023-11-24T08:06:03.011199+0000","last_fullsized":"2023-11-24T08:06:03.011199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985887+0000","last_change":"2023-11-24T08:06:02.985887+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985887+0000","last_undegraded":"2023-11-24T08:06:02.985887+0000","last_fullsized":"2023-11-24T08:06:02.985887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006556+0000","last_change":"2023-11-24T08:06:03.004192+0000","last_active":"2023-11-24T08:06:03.006556+0000","last_peered":"2023-11-24T08:06:03.006556+0000","last_clean":"2023-11-24T08:06:03.006556+0000","last_became_active":"2023-11-24T08:06:03.003913+0000","last_became_peered":"2023-11-24T08:06:03.003913+0000","last_unstale":"2023-11-24T08:06:03.006556+0000","last_undegraded":"2023-11-24T08:06:03.006556+0000","last_fullsized":"2023-11-24T08:06:03.006556+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987547+0000","last_change":"2023-11-24T08:06:02.987547+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987547+0000","last_undegraded":"2023-11-24T08:06:02.987547+0000","last_fullsized":"2023-11-24T08:06:02.987547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011782+0000","last_change":"2023-11-24T08:06:03.009734+0000","last_active":"2023-11-24T08:06:03.011782+0000","last_peered":"2023-11-24T08:06:03.011782+0000","last_clean":"2023-11-24T08:06:03.011782+0000","last_became_active":"2023-11-24T08:06:03.009416+0000","last_became_peered":"2023-11-24T08:06:03.009416+0000","last_unstale":"2023-11-24T08:06:03.011782+0000","last_undegraded":"2023-11-24T08:06:03.011782+0000","last_fullsized":"2023-11-24T08:06:03.011782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.992464+0000","last_change":"2023-11-24T08:06:02.992464+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.992464+0000","last_undegraded":"2023-11-24T08:06:02.992464+0000","last_fullsized":"2023-11-24T08:06:02.992464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004874+0000","last_change":"2023-11-24T08:06:02.998808+0000","last_active":"2023-11-24T08:06:03.004874+0000","last_peered":"2023-11-24T08:06:03.004874+0000","last_clean":"2023-11-24T08:06:03.004874+0000","last_became_active":"2023-11-24T08:06:02.998600+0000","last_became_peered":"2023-11-24T08:06:02.998600+0000","last_unstale":"2023-11-24T08:06:03.004874+0000","last_undegraded":"2023-11-24T08:06:03.004874+0000","last_fullsized":"2023-11-24T08:06:03.004874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989274+0000","last_change":"2023-11-24T08:06:02.989274+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989274+0000","last_undegraded":"2023-11-24T08:06:02.989274+0000","last_fullsized":"2023-11-24T08:06:02.989274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010263+0000","last_change":"2023-11-24T08:06:03.004639+0000","last_active":"2023-11-24T08:06:03.010263+0000","last_peered":"2023-11-24T08:06:03.010263+0000","last_clean":"2023-11-24T08:06:03.010263+0000","last_became_active":"2023-11-24T08:06:03.004341+0000","last_became_peered":"2023-11-24T08:06:03.004341+0000","last_unstale":"2023-11-24T08:06:03.010263+0000","last_undegraded":"2023-11-24T08:06:03.010263+0000","last_fullsized":"2023-11-24T08:06:03.010263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:46.680770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986105+0000","last_change":"2023-11-24T08:06:02.986105+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986105+0000","last_undegraded":"2023-11-24T08:06:02.986105+0000","last_fullsized":"2023-11-24T08:06:02.986105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008069+0000","last_change":"2023-11-24T08:06:03.005631+0000","last_active":"2023-11-24T08:06:03.008069+0000","last_peered":"2023-11-24T08:06:03.008069+0000","last_clean":"2023-11-24T08:06:03.008069+0000","last_became_active":"2023-11-24T08:06:03.004131+0000","last_became_peered":"2023-11-24T08:06:03.004131+0000","last_unstale":"2023-11-24T08:06:03.008069+0000","last_undegraded":"2023-11-24T08:06:03.008069+0000","last_fullsized":"2023-11-24T08:06:03.008069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999457+0000","last_change":"2023-11-24T08:06:02.999457+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999457+0000","last_undegraded":"2023-11-24T08:06:02.999457+0000","last_fullsized":"2023-11-24T08:06:02.999457+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011168+0000","last_change":"2023-11-24T08:06:03.009985+0000","last_active":"2023-11-24T08:06:03.011168+0000","last_peered":"2023-11-24T08:06:03.011168+0000","last_clean":"2023-11-24T08:06:03.011168+0000","last_became_active":"2023-11-24T08:06:03.009676+0000","last_became_peered":"2023-11-24T08:06:03.009676+0000","last_unstale":"2023-11-24T08:06:03.011168+0000","last_undegraded":"2023-11-24T08:06:03.011168+0000","last_fullsized":"2023-11-24T08:06:03.011168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987482+0000","last_change":"2023-11-24T08:06:02.987482+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987482+0000","last_undegraded":"2023-11-24T08:06:02.987482+0000","last_fullsized":"2023-11-24T08:06:02.987482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004281+0000","last_change":"2023-11-24T08:06:02.996472+0000","last_active":"2023-11-24T08:06:03.004281+0000","last_peered":"2023-11-24T08:06:03.004281+0000","last_clean":"2023-11-24T08:06:03.004281+0000","last_became_active":"2023-11-24T08:06:02.996105+0000","last_became_peered":"2023-11-24T08:06:02.996105+0000","last_unstale":"2023-11-24T08:06:03.004281+0000","last_undegraded":"2023-11-24T08:06:03.004281+0000","last_fullsized":"2023-11-24T08:06:03.004281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:14.099330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980014+0000","last_change":"2023-11-24T08:06:02.980014+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980014+0000","last_undegraded":"2023-11-24T08:06:02.980014+0000","last_fullsized":"2023-11-24T08:06:02.980014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.008394+0000","last_change":"2023-11-24T08:06:03.006947+0000","last_active":"2023-11-24T08:06:03.008394+0000","last_peered":"2023-11-24T08:06:03.008394+0000","last_clean":"2023-11-24T08:06:03.008394+0000","last_became_active":"2023-11-24T08:06:03.006640+0000","last_became_peered":"2023-11-24T08:06:03.006640+0000","last_unstale":"2023-11-24T08:06:03.008394+0000","last_undegraded":"2023-11-24T08:06:03.008394+0000","last_fullsized":"2023-11-24T08:06:03.008394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995208+0000","last_change":"2023-11-24T08:06:02.995208+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995208+0000","last_undegraded":"2023-11-24T08:06:02.995208+0000","last_fullsized":"2023-11-24T08:06:02.995208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007244+0000","last_change":"2023-11-24T08:06:03.003338+0000","last_active":"2023-11-24T08:06:03.007244+0000","last_peered":"2023-11-24T08:06:03.007244+0000","last_clean":"2023-11-24T08:06:03.007244+0000","last_became_active":"2023-11-24T08:06:02.998184+0000","last_became_peered":"2023-11-24T08:06:02.998184+0000","last_unstale":"2023-11-24T08:06:03.007244+0000","last_undegraded":"2023-11-24T08:06:03.007244+0000","last_fullsized":"2023-11-24T08:06:03.007244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:49.207612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.989003+0000","last_change":"2023-11-24T08:06:02.989003+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.989003+0000","last_undegraded":"2023-11-24T08:06:02.989003+0000","last_fullsized":"2023-11-24T08:06:02.989003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006462+0000","last_change":"2023-11-24T08:06:03.001694+0000","last_active":"2023-11-24T08:06:03.006462+0000","last_peered":"2023-11-24T08:06:03.006462+0000","last_clean":"2023-11-24T08:06:03.006462+0000","last_became_active":"2023-11-24T08:06:02.998616+0000","last_became_peered":"2023-11-24T08:06:02.998616+0000","last_unstale":"2023-11-24T08:06:03.006462+0000","last_undegraded":"2023-11-24T08:06:03.006462+0000","last_fullsized":"2023-11-24T08:06:03.006462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:30.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.986501+0000","last_change":"2023-11-24T08:06:02.986501+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.986501+0000","last_undegraded":"2023-11-24T08:06:02.986501+0000","last_fullsized":"2023-11-24T08:06:02.986501+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.003102+0000","last_change":"2023-11-24T08:06:02.993516+0000","last_active":"2023-11-24T08:06:03.003102+0000","last_peered":"2023-11-24T08:06:03.003102+0000","last_clean":"2023-11-24T08:06:03.003102+0000","last_became_active":"2023-11-24T08:06:02.992976+0000","last_became_peered":"2023-11-24T08:06:02.992976+0000","last_unstale":"2023-11-24T08:06:03.003102+0000","last_undegraded":"2023-11-24T08:06:03.003102+0000","last_fullsized":"2023-11-24T08:06:03.003102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:05:42.068467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985726+0000","last_change":"2023-11-24T08:06:02.985726+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985726+0000","last_undegraded":"2023-11-24T08:06:02.985726+0000","last_fullsized":"2023-11-24T08:06:02.985726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.000770+0000","last_change":"2023-11-24T08:06:02.998617+0000","last_active":"2023-11-24T08:06:03.000770+0000","last_peered":"2023-11-24T08:06:03.000770+0000","last_clean":"2023-11-24T08:06:03.000770+0000","last_became_active":"2023-11-24T08:06:02.998393+0000","last_became_peered":"2023-11-24T08:06:02.998393+0000","last_unstale":"2023-11-24T08:06:03.000770+0000","last_undegraded":"2023-11-24T08:06:03.000770+0000","last_fullsized":"2023-11-24T08:06:03.000770+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985538+0000","last_change":"2023-11-24T08:06:02.985538+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985538+0000","last_undegraded":"2023-11-24T08:06:02.985538+0000","last_fullsized":"2023-11-24T08:06:02.985538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010412+0000","last_change":"2023-11-24T08:06:03.005058+0000","last_active":"2023-11-24T08:06:03.010412+0000","last_peered":"2023-11-24T08:06:03.010412+0000","last_clean":"2023-11-24T08:06:03.010412+0000","last_became_active":"2023-11-24T08:06:03.004703+0000","last_became_peered":"2023-11-24T08:06:03.004703+0000","last_unstale":"2023-11-24T08:06:03.010412+0000","last_undegraded":"2023-11-24T08:06:03.010412+0000","last_fullsized":"2023-11-24T08:06:03.010412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985668+0000","last_change":"2023-11-24T08:06:02.985668+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985668+0000","last_undegraded":"2023-11-24T08:06:02.985668+0000","last_fullsized":"2023-11-24T08:06:02.985668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007000+0000","last_change":"2023-11-24T08:06:03.003128+0000","last_active":"2023-11-24T08:06:03.007000+0000","last_peered":"2023-11-24T08:06:03.007000+0000","last_clean":"2023-11-24T08:06:03.007000+0000","last_became_active":"2023-11-24T08:06:03.002791+0000","last_became_peered":"2023-11-24T08:06:03.002791+0000","last_unstale":"2023-11-24T08:06:03.007000+0000","last_undegraded":"2023-11-24T08:06:03.007000+0000","last_fullsized":"2023-11-24T08:06:03.007000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:49.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.995875+0000","last_change":"2023-11-24T08:06:02.995875+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.995875+0000","last_undegraded":"2023-11-24T08:06:02.995875+0000","last_fullsized":"2023-11-24T08:06:02.995875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011002+0000","last_change":"2023-11-24T08:06:03.006761+0000","last_active":"2023-11-24T08:06:03.011002+0000","last_peered":"2023-11-24T08:06:03.011002+0000","last_clean":"2023-11-24T08:06:03.011002+0000","last_became_active":"2023-11-24T08:06:03.006418+0000","last_became_peered":"2023-11-24T08:06:03.006418+0000","last_unstale":"2023-11-24T08:06:03.011002+0000","last_undegraded":"2023-11-24T08:06:03.011002+0000","last_fullsized":"2023-11-24T08:06:03.011002+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.988366+0000","last_change":"2023-11-24T08:06:02.988366+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.988366+0000","last_undegraded":"2023-11-24T08:06:02.988366+0000","last_fullsized":"2023-11-24T08:06:02.988366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010239+0000","last_change":"2023-11-24T08:06:03.008262+0000","last_active":"2023-11-24T08:06:03.010239+0000","last_peered":"2023-11-24T08:06:03.010239+0000","last_clean":"2023-11-24T08:06:03.010239+0000","last_became_active":"2023-11-24T08:06:03.007982+0000","last_became_peered":"2023-11-24T08:06:03.007982+0000","last_unstale":"2023-11-24T08:06:03.010239+0000","last_undegraded":"2023-11-24T08:06:03.010239+0000","last_fullsized":"2023-11-24T08:06:03.010239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:21.588168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.994570+0000","last_change":"2023-11-24T08:06:02.994570+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.994570+0000","last_undegraded":"2023-11-24T08:06:02.994570+0000","last_fullsized":"2023-11-24T08:06:02.994570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.006539+0000","last_change":"2023-11-24T08:06:03.001269+0000","last_active":"2023-11-24T08:06:03.006539+0000","last_peered":"2023-11-24T08:06:03.006539+0000","last_clean":"2023-11-24T08:06:03.006539+0000","last_became_active":"2023-11-24T08:06:03.000568+0000","last_became_peered":"2023-11-24T08:06:03.000568+0000","last_unstale":"2023-11-24T08:06:03.006539+0000","last_undegraded":"2023-11-24T08:06:03.006539+0000","last_fullsized":"2023-11-24T08:06:03.006539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.987028+0000","last_change":"2023-11-24T08:06:02.987028+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.987028+0000","last_undegraded":"2023-11-24T08:06:02.987028+0000","last_fullsized":"2023-11-24T08:06:02.987028+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:02.998436+0000","last_change":"2023-11-24T08:06:02.995680+0000","last_active":"2023-11-24T08:06:02.998436+0000","last_peered":"2023-11-24T08:06:02.998436+0000","last_clean":"2023-11-24T08:06:02.998436+0000","last_became_active":"2023-11-24T08:06:02.995421+0000","last_became_peered":"2023-11-24T08:06:02.995421+0000","last_unstale":"2023-11-24T08:06:02.998436+0000","last_undegraded":"2023-11-24T08:06:02.998436+0000","last_fullsized":"2023-11-24T08:06:02.998436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.985166+0000","last_change":"2023-11-24T08:06:02.985166+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.985166+0000","last_undegraded":"2023-11-24T08:06:02.985166+0000","last_fullsized":"2023-11-24T08:06:02.985166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.004405+0000","last_change":"2023-11-24T08:06:03.000694+0000","last_active":"2023-11-24T08:06:03.004405+0000","last_peered":"2023-11-24T08:06:03.004405+0000","last_clean":"2023-11-24T08:06:03.004405+0000","last_became_active":"2023-11-24T08:06:03.000285+0000","last_became_peered":"2023-11-24T08:06:03.000285+0000","last_unstale":"2023-11-24T08:06:03.004405+0000","last_undegraded":"2023-11-24T08:06:03.004405+0000","last_fullsized":"2023-11-24T08:06:03.004405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:44.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.983698+0000","last_change":"2023-11-24T08:06:02.983698+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.983698+0000","last_undegraded":"2023-11-24T08:06:02.983698+0000","last_fullsized":"2023-11-24T08:06:02.983698+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.010558+0000","last_change":"2023-11-24T08:06:03.008992+0000","last_active":"2023-11-24T08:06:03.010558+0000","last_peered":"2023-11-24T08:06:03.010558+0000","last_clean":"2023-11-24T08:06:03.010558+0000","last_became_active":"2023-11-24T08:06:03.008648+0000","last_became_peered":"2023-11-24T08:06:03.008648+0000","last_unstale":"2023-11-24T08:06:03.010558+0000","last_undegraded":"2023-11-24T08:06:03.010558+0000","last_fullsized":"2023-11-24T08:06:03.010558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:04.313803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.981885+0000","last_change":"2023-11-24T08:06:02.981885+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.981885+0000","last_undegraded":"2023-11-24T08:06:02.981885+0000","last_fullsized":"2023-11-24T08:06:02.981885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.007305+0000","last_change":"2023-11-24T08:06:03.003843+0000","last_active":"2023-11-24T08:06:03.007305+0000","last_peered":"2023-11-24T08:06:03.007305+0000","last_clean":"2023-11-24T08:06:03.007305+0000","last_became_active":"2023-11-24T08:06:03.002741+0000","last_became_peered":"2023-11-24T08:06:03.002741+0000","last_unstale":"2023-11-24T08:06:03.007305+0000","last_undegraded":"2023-11-24T08:06:03.007305+0000","last_fullsized":"2023-11-24T08:06:03.007305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:46:14.437116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.980343+0000","last_change":"2023-11-24T08:06:02.980343+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.980343+0000","last_undegraded":"2023-11-24T08:06:02.980343+0000","last_fullsized":"2023-11-24T08:06:02.980343+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011146+0000","last_change":"2023-11-24T08:06:03.009978+0000","last_active":"2023-11-24T08:06:03.011146+0000","last_peered":"2023-11-24T08:06:03.011146+0000","last_clean":"2023-11-24T08:06:03.011146+0000","last_became_active":"2023-11-24T08:06:03.009694+0000","last_became_peered":"2023-11-24T08:06:03.009694+0000","last_unstale":"2023-11-24T08:06:03.011146+0000","last_undegraded":"2023-11-24T08:06:03.011146+0000","last_fullsized":"2023-11-24T08:06:03.011146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-24T08:06:02.999828+0000","last_change":"2023-11-24T08:06:02.999828+0000","last_active":"2023-11-24T08:06:02.953055+0000","last_peered":"2023-11-24T08:06:02.953055+0000","last_clean":"2023-11-24T08:06:02.953055+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:06:02.999828+0000","last_undegraded":"2023-11-24T08:06:02.999828+0000","last_fullsized":"2023-11-24T08:06:02.999828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.011405+0000","last_change":"2023-11-24T08:06:03.008297+0000","last_active":"2023-11-24T08:06:03.011405+0000","last_peered":"2023-11-24T08:06:03.011405+0000","last_clean":"2023-11-24T08:06:03.011405+0000","last_became_active":"2023-11-24T08:06:03.007985+0000","last_became_peered":"2023-11-24T08:06:03.007985+0000","last_unstale":"2023-11-24T08:06:03.011405+0000","last_undegraded":"2023-11-24T08:06:03.011405+0000","last_fullsized":"2023-11-24T08:06:03.011405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:22.477948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-24T08:06:01.972672+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:06:01.972672+0000","last_peered":"2023-11-24T08:06:01.972672+0000","last_clean":"2023-11-24T08:06:01.972672+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:06:01.972672+0000","last_undegraded":"2023-11-24T08:06:01.972672+0000","last_fullsized":"2023-11-24T08:06:01.972672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93100000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.048}]},{"osd":2,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.177},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":1.643}]},{"osd":3,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.486},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.365}]},{"osd":4,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.302},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.522},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.623}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.345},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.607}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":1.089}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.4370000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.661}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.639}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.585}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.90100000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.353}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.79200000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.286},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.321}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4019999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.127}]},{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.3879999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.4379999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.546},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.617}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.5899999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":1.335}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.504},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.5700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.2789999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.232},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.0509999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.623},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.3180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.359}]},{"osd":6,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.446},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.4099999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.496},{"interface":"front","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.536}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.83799999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.98599999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.2829999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.214}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.597,"15min":1.597},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3809999999999998,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.0940000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.87},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.93799999999999994}]},{"osd":2,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.2529999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":1.05}]},{"osd":6,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.9359999999999999,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3540000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.159}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.72899999999999998},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83499999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.877}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.0780000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.115},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.0920000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:06:09.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2023-11-24T08:06:09.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.988645+0000', 'last_change': '2023-11-24T08:06:02.988645+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.988645+0000', 'last_undegraded': '2023-11-24T08:06:02.988645+0000', 'last_fullsized': '2023-11-24T08:06:02.988645+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.990877+0000', 'last_change': '2023-11-24T08:06:02.990877+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.990877+0000', 'last_undegraded': '2023-11-24T08:06:02.990877+0000', 'last_fullsized': '2023-11-24T08:06:02.990877+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.991622+0000', 'last_change': '2023-11-24T08:06:02.991622+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.991622+0000', 'last_undegraded': '2023-11-24T08:06:02.991622+0000', 'last_fullsized': '2023-11-24T08:06:02.991622+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989545+0000', 'last_change': '2023-11-24T08:06:02.989545+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989545+0000', 'last_undegraded': '2023-11-24T08:06:02.989545+0000', 'last_fullsized': '2023-11-24T08:06:02.989545+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.987313+0000', 'last_change': '2023-11-24T08:06:02.987313+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.987313+0000', 'last_undegraded': '2023-11-24T08:06:02.987313+0000', 'last_fullsized': '2023-11-24T08:06:02.987313+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.997609+0000', 'last_change': '2023-11-24T08:06:02.997609+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.997609+0000', 'last_undegraded': '2023-11-24T08:06:02.997609+0000', 'last_fullsized': '2023-11-24T08:06:02.997609+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.995334+0000', 'last_change': '2023-11-24T08:06:02.995334+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.995334+0000', 'last_undegraded': '2023-11-24T08:06:02.995334+0000', 'last_fullsized': '2023-11-24T08:06:02.995334+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:03.002111+0000', 'last_change': '2023-11-24T08:06:03.002111+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:03.002111+0000', 'last_undegraded': '2023-11-24T08:06:03.002111+0000', 'last_fullsized': '2023-11-24T08:06:03.002111+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.978774+0000', 'last_change': '2023-11-24T08:06:02.978774+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.978774+0000', 'last_undegraded': '2023-11-24T08:06:02.978774+0000', 'last_fullsized': '2023-11-24T08:06:02.978774+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2023-11-24T08:06:09.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.993312+0000', 'last_change': '2023-11-24T08:06:02.993312+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.993312+0000', 'last_undegraded': '2023-11-24T08:06:02.993312+0000', 'last_fullsized': '2023-11-24T08:06:02.993312+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:03.000463+0000', 'last_change': '2023-11-24T08:06:03.000463+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:03.000463+0000', 'last_undegraded': '2023-11-24T08:06:03.000463+0000', 'last_fullsized': '2023-11-24T08:06:03.000463+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.990332+0000', 'last_change': '2023-11-24T08:06:02.990332+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.990332+0000', 'last_undegraded': '2023-11-24T08:06:02.990332+0000', 'last_fullsized': '2023-11-24T08:06:02.990332+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.986467+0000', 'last_change': '2023-11-24T08:06:02.986467+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.986467+0000', 'last_undegraded': '2023-11-24T08:06:02.986467+0000', 'last_fullsized': '2023-11-24T08:06:02.986467+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.991919+0000', 'last_change': '2023-11-24T08:06:02.991919+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.991919+0000', 'last_undegraded': '2023-11-24T08:06:02.991919+0000', 'last_fullsized': '2023-11-24T08:06:02.991919+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2023-11-24T08:06:09.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.994432+0000', 'last_change': '2023-11-24T08:06:02.994432+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.994432+0000', 'last_undegraded': '2023-11-24T08:06:02.994432+0000', 'last_fullsized': '2023-11-24T08:06:02.994432+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.995541+0000', 'last_change': '2023-11-24T08:06:02.995541+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.995541+0000', 'last_undegraded': '2023-11-24T08:06:02.995541+0000', 'last_fullsized': '2023-11-24T08:06:02.995541+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:03.000531+0000', 'last_change': '2023-11-24T08:06:03.000531+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:03.000531+0000', 'last_undegraded': '2023-11-24T08:06:03.000531+0000', 'last_fullsized': '2023-11-24T08:06:03.000531+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.990434+0000', 'last_change': '2023-11-24T08:06:02.990434+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.990434+0000', 'last_undegraded': '2023-11-24T08:06:02.990434+0000', 'last_fullsized': '2023-11-24T08:06:02.990434+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989472+0000', 'last_change': '2023-11-24T08:06:02.989472+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989472+0000', 'last_undegraded': '2023-11-24T08:06:02.989472+0000', 'last_fullsized': '2023-11-24T08:06:02.989472+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2023-11-24T08:06:09.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989920+0000', 'last_change': '2023-11-24T08:06:02.989920+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989920+0000', 'last_undegraded': '2023-11-24T08:06:02.989920+0000', 'last_fullsized': '2023-11-24T08:06:02.989920+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.982150+0000', 'last_change': '2023-11-24T08:06:02.982150+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.982150+0000', 'last_undegraded': '2023-11-24T08:06:02.982150+0000', 'last_fullsized': '2023-11-24T08:06:02.982150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.977577+0000', 'last_change': '2023-11-24T08:06:02.977577+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.977577+0000', 'last_undegraded': '2023-11-24T08:06:02.977577+0000', 'last_fullsized': '2023-11-24T08:06:02.977577+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.982512+0000', 'last_change': '2023-11-24T08:06:02.982512+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.982512+0000', 'last_undegraded': '2023-11-24T08:06:02.982512+0000', 'last_fullsized': '2023-11-24T08:06:02.982512+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989460+0000', 'last_change': '2023-11-24T08:06:02.989460+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989460+0000', 'last_undegraded': '2023-11-24T08:06:02.989460+0000', 'last_fullsized': '2023-11-24T08:06:02.989460+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2023-11-24T08:06:09.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.997244+0000', 'last_change': '2023-11-24T08:06:02.997244+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.997244+0000', 'last_undegraded': '2023-11-24T08:06:02.997244+0000', 'last_fullsized': '2023-11-24T08:06:02.997244+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989480+0000', 'last_change': '2023-11-24T08:06:02.989480+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989480+0000', 'last_undegraded': '2023-11-24T08:06:02.989480+0000', 'last_fullsized': '2023-11-24T08:06:02.989480+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.993720+0000', 'last_change': '2023-11-24T08:06:02.993720+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.993720+0000', 'last_undegraded': '2023-11-24T08:06:02.993720+0000', 'last_fullsized': '2023-11-24T08:06:02.993720+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989000+0000', 'last_change': '2023-11-24T08:06:02.989000+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989000+0000', 'last_undegraded': '2023-11-24T08:06:02.989000+0000', 'last_fullsized': '2023-11-24T08:06:02.989000+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.981111+0000', 'last_change': '2023-11-24T08:06:02.981111+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.981111+0000', 'last_undegraded': '2023-11-24T08:06:02.981111+0000', 'last_fullsized': '2023-11-24T08:06:02.981111+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.996512+0000', 'last_change': '2023-11-24T08:06:02.996512+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.996512+0000', 'last_undegraded': '2023-11-24T08:06:02.996512+0000', 'last_fullsized': '2023-11-24T08:06:02.996512+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.988512+0000', 'last_change': '2023-11-24T08:06:02.988512+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.988512+0000', 'last_undegraded': '2023-11-24T08:06:02.988512+0000', 'last_fullsized': '2023-11-24T08:06:02.988512+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:03.000057+0000', 'last_change': '2023-11-24T08:06:03.000057+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:03.000057+0000', 'last_undegraded': '2023-11-24T08:06:03.000057+0000', 'last_fullsized': '2023-11-24T08:06:03.000057+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.986477+0000', 'last_change': '2023-11-24T08:06:02.986477+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.986477+0000', 'last_undegraded': '2023-11-24T08:06:02.986477+0000', 'last_fullsized': '2023-11-24T08:06:02.986477+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T08:06:09.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.986375+0000', 'last_change': '2023-11-24T08:06:02.986375+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.986375+0000', 'last_undegraded': '2023-11-24T08:06:02.986375+0000', 'last_fullsized': '2023-11-24T08:06:02.986375+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.978376+0000', 'last_change': '2023-11-24T08:06:02.978376+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.978376+0000', 'last_undegraded': '2023-11-24T08:06:02.978376+0000', 'last_fullsized': '2023-11-24T08:06:02.978376+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.992842+0000', 'last_change': '2023-11-24T08:06:02.992842+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.992842+0000', 'last_undegraded': '2023-11-24T08:06:02.992842+0000', 'last_fullsized': '2023-11-24T08:06:02.992842+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.995771+0000', 'last_change': '2023-11-24T08:06:02.995771+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.995771+0000', 'last_undegraded': '2023-11-24T08:06:02.995771+0000', 'last_fullsized': '2023-11-24T08:06:02.995771+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.990140+0000', 'last_change': '2023-11-24T08:06:02.990140+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.990140+0000', 'last_undegraded': '2023-11-24T08:06:02.990140+0000', 'last_fullsized': '2023-11-24T08:06:02.990140+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.976833+0000', 'last_change': '2023-11-24T08:06:02.976833+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.976833+0000', 'last_undegraded': '2023-11-24T08:06:02.976833+0000', 'last_fullsized': '2023-11-24T08:06:02.976833+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.991684+0000', 'last_change': '2023-11-24T08:06:02.991684+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.991684+0000', 'last_undegraded': '2023-11-24T08:06:02.991684+0000', 'last_fullsized': '2023-11-24T08:06:02.991684+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.993733+0000', 'last_change': '2023-11-24T08:06:02.993733+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.993733+0000', 'last_undegraded': '2023-11-24T08:06:02.993733+0000', 'last_fullsized': '2023-11-24T08:06:02.993733+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.985887+0000', 'last_change': '2023-11-24T08:06:02.985887+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.985887+0000', 'last_undegraded': '2023-11-24T08:06:02.985887+0000', 'last_fullsized': '2023-11-24T08:06:02.985887+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-11-24T08:06:09.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.987547+0000', 'last_change': '2023-11-24T08:06:02.987547+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.987547+0000', 'last_undegraded': '2023-11-24T08:06:02.987547+0000', 'last_fullsized': '2023-11-24T08:06:02.987547+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2023-11-24T08:06:09.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.992464+0000', 'last_change': '2023-11-24T08:06:02.992464+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.992464+0000', 'last_undegraded': '2023-11-24T08:06:02.992464+0000', 'last_fullsized': '2023-11-24T08:06:02.992464+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2023-11-24T08:06:09.380 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989274+0000', 'last_change': '2023-11-24T08:06:02.989274+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989274+0000', 'last_undegraded': '2023-11-24T08:06:02.989274+0000', 'last_fullsized': '2023-11-24T08:06:02.989274+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.380 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2023-11-24T08:06:09.380 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.986105+0000', 'last_change': '2023-11-24T08:06:02.986105+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.986105+0000', 'last_undegraded': '2023-11-24T08:06:02.986105+0000', 'last_fullsized': '2023-11-24T08:06:02.986105+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.380 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.999457+0000', 'last_change': '2023-11-24T08:06:02.999457+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.999457+0000', 'last_undegraded': '2023-11-24T08:06:02.999457+0000', 'last_fullsized': '2023-11-24T08:06:02.999457+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.987482+0000', 'last_change': '2023-11-24T08:06:02.987482+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.987482+0000', 'last_undegraded': '2023-11-24T08:06:02.987482+0000', 'last_fullsized': '2023-11-24T08:06:02.987482+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.980014+0000', 'last_change': '2023-11-24T08:06:02.980014+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.980014+0000', 'last_undegraded': '2023-11-24T08:06:02.980014+0000', 'last_fullsized': '2023-11-24T08:06:02.980014+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.995208+0000', 'last_change': '2023-11-24T08:06:02.995208+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.995208+0000', 'last_undegraded': '2023-11-24T08:06:02.995208+0000', 'last_fullsized': '2023-11-24T08:06:02.995208+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.989003+0000', 'last_change': '2023-11-24T08:06:02.989003+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.989003+0000', 'last_undegraded': '2023-11-24T08:06:02.989003+0000', 'last_fullsized': '2023-11-24T08:06:02.989003+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.986501+0000', 'last_change': '2023-11-24T08:06:02.986501+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.986501+0000', 'last_undegraded': '2023-11-24T08:06:02.986501+0000', 'last_fullsized': '2023-11-24T08:06:02.986501+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.985726+0000', 'last_change': '2023-11-24T08:06:02.985726+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.985726+0000', 'last_undegraded': '2023-11-24T08:06:02.985726+0000', 'last_fullsized': '2023-11-24T08:06:02.985726+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2023-11-24T08:06:09.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.985538+0000', 'last_change': '2023-11-24T08:06:02.985538+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.985538+0000', 'last_undegraded': '2023-11-24T08:06:02.985538+0000', 'last_fullsized': '2023-11-24T08:06:02.985538+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.985668+0000', 'last_change': '2023-11-24T08:06:02.985668+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.985668+0000', 'last_undegraded': '2023-11-24T08:06:02.985668+0000', 'last_fullsized': '2023-11-24T08:06:02.985668+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.995875+0000', 'last_change': '2023-11-24T08:06:02.995875+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.995875+0000', 'last_undegraded': '2023-11-24T08:06:02.995875+0000', 'last_fullsized': '2023-11-24T08:06:02.995875+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.988366+0000', 'last_change': '2023-11-24T08:06:02.988366+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.988366+0000', 'last_undegraded': '2023-11-24T08:06:02.988366+0000', 'last_fullsized': '2023-11-24T08:06:02.988366+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.994570+0000', 'last_change': '2023-11-24T08:06:02.994570+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.994570+0000', 'last_undegraded': '2023-11-24T08:06:02.994570+0000', 'last_fullsized': '2023-11-24T08:06:02.994570+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-11-24T08:06:09.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.987028+0000', 'last_change': '2023-11-24T08:06:02.987028+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.987028+0000', 'last_undegraded': '2023-11-24T08:06:02.987028+0000', 'last_fullsized': '2023-11-24T08:06:02.987028+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.985166+0000', 'last_change': '2023-11-24T08:06:02.985166+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.985166+0000', 'last_undegraded': '2023-11-24T08:06:02.985166+0000', 'last_fullsized': '2023-11-24T08:06:02.985166+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.983698+0000', 'last_change': '2023-11-24T08:06:02.983698+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.983698+0000', 'last_undegraded': '2023-11-24T08:06:02.983698+0000', 'last_fullsized': '2023-11-24T08:06:02.983698+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.981885+0000', 'last_change': '2023-11-24T08:06:02.981885+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.981885+0000', 'last_undegraded': '2023-11-24T08:06:02.981885+0000', 'last_fullsized': '2023-11-24T08:06:02.981885+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-11-24T08:06:09.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.980343+0000', 'last_change': '2023-11-24T08:06:02.980343+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.980343+0000', 'last_undegraded': '2023-11-24T08:06:02.980343+0000', 'last_fullsized': '2023-11-24T08:06:02.980343+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T08:06:09.385 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-11-24T08:06:09.385 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-11-24T08:06:02.999828+0000', 'last_change': '2023-11-24T08:06:02.999828+0000', 'last_active': '2023-11-24T08:06:02.953055+0000', 'last_peered': '2023-11-24T08:06:02.953055+0000', 'last_clean': '2023-11-24T08:06:02.953055+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T08:06:02.999828+0000', 'last_undegraded': '2023-11-24T08:06:02.999828+0000', 'last_fullsized': '2023-11-24T08:06:02.999828+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'last_clean_scrub_stamp': '2023-11-24T08:06:02.953055+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T08:06:09.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-11-24T08:06:09.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.621+0000 7fae94989640 1 Processor -- start 2023-11-24T08:06:09.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.622+0000 7fae94989640 1 -- start start 2023-11-24T08:06:09.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.623+0000 7fae94989640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.623+0000 7fae94989640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae900d0240 0x7fae900b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.623+0000 7fae94989640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae900ce990 con 0x7fae900d38e0 2023-11-24T08:06:09.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.623+0000 7fae94989640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900d4110 con 0x7fae900d0240 2023-11-24T08:06:09.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.623+0000 7fae94989640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cec70 con 0x7fae9015d4e0 2023-11-24T08:06:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.624+0000 7fae8ed9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.624+0000 7fae8ed9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40144/0 (socket says 172.21.15.107:40144) 2023-11-24T08:06:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.624+0000 7fae8ed9d640 1 -- 172.21.15.107:0/1474856150 learned_addr learned my addr 172.21.15.107:0/1474856150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.624+0000 7fae8f59e640 1 --2- 172.21.15.107:0/1474856150 >> v2:172.21.15.120:3300/0 conn(0x7fae900d0240 0x7fae900b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.625+0000 7fae8ed9d640 1 -- 172.21.15.107:0/1474856150 >> v2:172.21.15.120:3300/0 conn(0x7fae900d0240 msgr2=0x7fae900b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.625+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1474856150 >> v2:172.21.15.120:3300/0 conn(0x7fae900d0240 0x7fae900b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.625+0000 7fae8ed9d640 1 -- 172.21.15.107:0/1474856150 >> v1:172.21.15.120:6789/0 conn(0x7fae900d38e0 legacy=0x7fae900cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.625+0000 7fae8ed9d640 1 -- 172.21.15.107:0/1474856150 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900cef50 con 0x7fae9015d4e0 2023-11-24T08:06:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.626+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1474856150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae80009870 tx=0x7fae8002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d07efcce4016258c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.626+0000 7fae8dd9b640 1 -- 172.21.15.107:0/1474856150 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80030840 con 0x7fae9015d4e0 2023-11-24T08:06:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.626+0000 7fae8dd9b640 1 -- 172.21.15.107:0/1474856150 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae800309a0 con 0x7fae9015d4e0 2023-11-24T08:06:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.626+0000 7fae8dd9b640 1 -- 172.21.15.107:0/1474856150 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80030d30 con 0x7fae9015d4e0 2023-11-24T08:06:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 -- 172.21.15.107:0/1474856150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 msgr2=0x7fae900d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 --2- 172.21.15.107:0/1474856150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae80009870 tx=0x7fae8002b980 comp rx=0 tx=0).stop 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 -- 172.21.15.107:0/1474856150 shutdown_connections 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 --2- 172.21.15.107:0/1474856150 >> v2:172.21.15.120:3300/0 conn(0x7fae900d0240 0x7fae900b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 --2- 172.21.15.107:0/1474856150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae9015d4e0 0x7fae900d1e20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 -- 172.21.15.107:0/1474856150 >> 172.21.15.107:0/1474856150 conn(0x7fae9009c510 msgr2=0x7fae900d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 -- 172.21.15.107:0/1474856150 shutdown_connections 2023-11-24T08:06:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.627+0000 7fae94989640 1 -- 172.21.15.107:0/1474856150 wait complete. 2023-11-24T08:06:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.629+0000 7fae94989640 1 Processor -- start 2023-11-24T08:06:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.629+0000 7fae94989640 1 -- start start 2023-11-24T08:06:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 0x7fae9015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 0x7fae90149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015e500 con 0x7fae9015d4e0 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c75d0 con 0x7fae900d38e0 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae94989640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c8f10 con 0x7fae900d0240 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43342/0 (socket says 172.21.15.107:43342) 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.630+0000 7fae8f59e640 1 -- 172.21.15.107:0/1409760593 learned_addr learned my addr 172.21.15.107:0/1409760593 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 0x7fae9015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8e59c640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 0x7fae90149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 -- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 msgr2=0x7fae9015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 0x7fae9015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 -- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 msgr2=0x7fae90149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 0x7fae90149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 -- 172.21.15.107:0/1409760593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800094d0 con 0x7fae9015d4e0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 0x7fae9015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8e59c640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 0x7fae90149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae8f59e640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fae8800efc0 tx=0x7fae8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88014480 con 0x7fae9015d4e0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8800f040 con 0x7fae9015d4e0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.631+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae900c9250 con 0x7fae9015d4e0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.632+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88014820 con 0x7fae9015d4e0 2023-11-24T08:06:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.632+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c9a90 con 0x7fae9015d4e0 2023-11-24T08:06:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.633+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900c9cb0 con 0x7fae9015d4e0 2023-11-24T08:06:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.634+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fae88014980 con 0x7fae9015d4e0 2023-11-24T08:06:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.634+0000 7fae777fe640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 0x7fae7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.635+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae8808a540 con 0x7fae9015d4e0 2023-11-24T08:06:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.635+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 0x7fae7c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.636+0000 7fae8ed9d640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 0x7fae7c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae80036000 tx=0x7fae8003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.639+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae880555e0 con 0x7fae9015d4e0 2023-11-24T08:06:09.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.898+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fae9004e840 con 0x7fae9015d4e0 2023-11-24T08:06:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.899+0000 7fae777fe640 1 -- 172.21.15.107:0/1409760593 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1033 (secure 0 0 0) 0x7fae88058c00 con 0x7fae9015d4e0 2023-11-24T08:06:09.901 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:09.901 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":100,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":34,"num_osds":8,"num_up_osds":8,"osd_up_since":1700813082,"num_in_osds":8,"osd_in_since":1700813073,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225349632,"bytes_avail":772868763648,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":14,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5070}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2023-11-24T08:05:51.450359+0000","services":{}},"progress_events":{}} 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 msgr2=0x7fae7c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 0x7fae7c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae80036000 tx=0x7fae8003b040 comp rx=0 tx=0).stop 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 msgr2=0x7fae9014c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fae8800efc0 tx=0x7fae8800c490 comp rx=0 tx=0).stop 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 shutdown_connections 2023-11-24T08:06:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae7c06ae40 0x7fae7c06d300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae9015d4e0 0x7fae9014c970 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae900d38e0 0x7fae90149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 --2- 172.21.15.107:0/1409760593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae900d0240 0x7fae9015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 >> 172.21.15.107:0/1409760593 conn(0x7fae9009c510 msgr2=0x7fae900cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 shutdown_connections 2023-11-24T08:06:09.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:09.902+0000 7fae94989640 1 -- 172.21.15.107:0/1409760593 wait complete. 2023-11-24T08:06:09.923 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-11-24T08:06:09.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:06:10.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.130+0000 7f1db4768640 1 Processor -- start 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.130+0000 7f1db4768640 1 -- start start 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db4768640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db4768640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1dac0ce010 0x7f1dac0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db4768640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac06c1f0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db4768640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0d1070 con 0x7f1dac0ce010 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db4768640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0cedc0 con 0x7f1dac1609e0 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db2504640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.132+0000 7f1db2504640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40162/0 (socket says 172.21.15.107:40162) 2023-11-24T08:06:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2504640 1 -- 172.21.15.107:0/825483713 learned_addr learned my addr 172.21.15.107:0/825483713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2d05640 1 --2- 172.21.15.107:0/825483713 >> v2:172.21.15.120:3300/0 conn(0x7f1dac0ce010 0x7f1dac0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2504640 1 -- 172.21.15.107:0/825483713 >> v2:172.21.15.120:3300/0 conn(0x7f1dac0ce010 msgr2=0x7f1dac0ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2504640 1 --2- 172.21.15.107:0/825483713 >> v2:172.21.15.120:3300/0 conn(0x7f1dac0ce010 0x7f1dac0ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2504640 1 -- 172.21.15.107:0/825483713 >> v1:172.21.15.120:6789/0 conn(0x7f1dac0d31a0 legacy=0x7f1dac0d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.133+0000 7f1db2504640 1 -- 172.21.15.107:0/825483713 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0cf0a0 con 0x7f1dac1609e0 2023-11-24T08:06:10.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.135+0000 7f1db2504640 1 --2- 172.21.15.107:0/825483713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009e50 tx=0x7f1d9c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d614bb10fdbb31ff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:10.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db1502640 1 -- 172.21.15.107:0/825483713 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c02cb00 con 0x7f1dac1609e0 2023-11-24T08:06:10.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db1502640 1 -- 172.21.15.107:0/825483713 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c02cc60 con 0x7f1dac1609e0 2023-11-24T08:06:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db1502640 1 -- 172.21.15.107:0/825483713 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c0343f0 con 0x7f1dac1609e0 2023-11-24T08:06:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 -- 172.21.15.107:0/825483713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 msgr2=0x7f1dac06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 --2- 172.21.15.107:0/825483713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009e50 tx=0x7f1d9c02bf60 comp rx=0 tx=0).stop 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 -- 172.21.15.107:0/825483713 shutdown_connections 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 --2- 172.21.15.107:0/825483713 >> v2:172.21.15.120:3300/0 conn(0x7f1dac0ce010 0x7f1dac0ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 --2- 172.21.15.107:0/825483713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac06bcb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.136+0000 7f1db4768640 1 -- 172.21.15.107:0/825483713 >> 172.21.15.107:0/825483713 conn(0x7f1dac08af60 msgr2=0x7f1dac09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.137+0000 7f1db4768640 1 -- 172.21.15.107:0/825483713 shutdown_connections 2023-11-24T08:06:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.137+0000 7f1db4768640 1 -- 172.21.15.107:0/825483713 wait complete. 2023-11-24T08:06:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.139+0000 7f1db4768640 1 Processor -- start 2023-11-24T08:06:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.139+0000 7f1db4768640 1 -- start start 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac0ce010 0x7f1dac0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0b8fd0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac15f4e0 con 0x7f1dac0ce010 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db4768640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0b78d0 con 0x7f1dac1609e0 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db1d03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db1d03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43346/0 (socket says 172.21.15.107:43346) 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db1d03640 1 -- 172.21.15.107:0/2898335909 learned_addr learned my addr 172.21.15.107:0/2898335909 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db2d05640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.140+0000 7f1db2504640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac0ce010 0x7f1dac0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 -- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 msgr2=0x7f1dac0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 -- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac0ce010 msgr2=0x7f1dac0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac0ce010 0x7f1dac0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 -- 172.21.15.107:0/2898335909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c009b30 con 0x7f1dac0d31a0 2023-11-24T08:06:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db1d03640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c500 tx=0x7f1da000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00052d0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0005430 con 0x7f1dac0d31a0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac0b6b10 con 0x7f1dac0d31a0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.141+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0012850 con 0x7f1dac0d31a0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.142+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac0b70d0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.143+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac067470 con 0x7f1dac0d31a0 2023-11-24T08:06:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.144+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1da00129b0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.145+0000 7f1daaffd640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 0x7f1d9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.145+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da00891c0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.145+0000 7f1db2504640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 0x7f1d9806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.146+0000 7f1db2504640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 0x7f1d9806d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c035000 tx=0x7f1d9c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:10.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.149+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1da00541e0 con 0x7f1dac0d31a0 2023-11-24T08:06:10.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.334+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1dac0cf380 con 0x7f1d9806ae40 2023-11-24T08:06:10.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.356+0000 7f1daaffd640 1 -- 172.21.15.107:0/2898335909 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286564 (secure 0 0 0) 0x7f1d9c044090 con 0x7f1d9806ae40 2023-11-24T08:06:10.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 msgr2=0x7f1d9806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 0x7f1d9806d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c035000 tx=0x7f1d9c030000 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 msgr2=0x7f1dac15efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c500 tx=0x7f1da000c9c0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 shutdown_connections 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9806ae40 0x7f1d9806d300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1609e0 0x7f1dac0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dac0d31a0 0x7f1dac15efa0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 --2- 172.21.15.107:0/2898335909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac0ce010 0x7f1dac0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 >> 172.21.15.107:0/2898335909 conn(0x7f1dac08af60 msgr2=0x7f1dac0d1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 shutdown_connections 2023-11-24T08:06:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:10.362+0000 7f1db4768640 1 -- 172.21.15.107:0/2898335909 wait complete. 2023-11-24T08:06:10.384 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-11-24T08:06:09.455358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220392,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754328,"statfs":{"total":773094113280,"available":772868431872,"internally_reserved":0,"allocated":4333568,"data_stored":2400748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.483339"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011371+0000","last_change":"2023-11-24T08:06:04.014820+0000","last_active":"2023-11-24T08:06:05.011371+0000","last_peered":"2023-11-24T08:06:05.011371+0000","last_clean":"2023-11-24T08:06:05.011371+0000","last_became_active":"2023-11-24T08:06:04.013653+0000","last_became_peered":"2023-11-24T08:06:04.013653+0000","last_unstale":"2023-11-24T08:06:05.011371+0000","last_undegraded":"2023-11-24T08:06:05.011371+0000","last_fullsized":"2023-11-24T08:06:05.011371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:45.842740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060248+0000","last_change":"2023-11-24T08:06:04.008376+0000","last_active":"2023-11-24T08:06:05.060248+0000","last_peered":"2023-11-24T08:06:05.060248+0000","last_clean":"2023-11-24T08:06:05.060248+0000","last_became_active":"2023-11-24T08:06:04.008093+0000","last_became_peered":"2023-11-24T08:06:04.008093+0000","last_unstale":"2023-11-24T08:06:05.060248+0000","last_undegraded":"2023-11-24T08:06:05.060248+0000","last_fullsized":"2023-11-24T08:06:05.060248+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:53:26.874992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062267+0000","last_change":"2023-11-24T08:06:04.019222+0000","last_active":"2023-11-24T08:06:05.062267+0000","last_peered":"2023-11-24T08:06:05.062267+0000","last_clean":"2023-11-24T08:06:05.062267+0000","last_became_active":"2023-11-24T08:06:04.018903+0000","last_became_peered":"2023-11-24T08:06:04.018903+0000","last_unstale":"2023-11-24T08:06:05.062267+0000","last_undegraded":"2023-11-24T08:06:05.062267+0000","last_fullsized":"2023-11-24T08:06:05.062267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060961+0000","last_change":"2023-11-24T08:06:04.019730+0000","last_active":"2023-11-24T08:06:05.060961+0000","last_peered":"2023-11-24T08:06:05.060961+0000","last_clean":"2023-11-24T08:06:05.060961+0000","last_became_active":"2023-11-24T08:06:04.019090+0000","last_became_peered":"2023-11-24T08:06:04.019090+0000","last_unstale":"2023-11-24T08:06:05.060961+0000","last_undegraded":"2023-11-24T08:06:05.060961+0000","last_fullsized":"2023-11-24T08:06:05.060961+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061427+0000","last_change":"2023-11-24T08:06:04.012201+0000","last_active":"2023-11-24T08:06:05.061427+0000","last_peered":"2023-11-24T08:06:05.061427+0000","last_clean":"2023-11-24T08:06:05.061427+0000","last_became_active":"2023-11-24T08:06:04.011833+0000","last_became_peered":"2023-11-24T08:06:04.011833+0000","last_unstale":"2023-11-24T08:06:05.061427+0000","last_undegraded":"2023-11-24T08:06:05.061427+0000","last_fullsized":"2023-11-24T08:06:05.061427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:15.111132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059707+0000","last_change":"2023-11-24T08:06:04.010695+0000","last_active":"2023-11-24T08:06:05.059707+0000","last_peered":"2023-11-24T08:06:05.059707+0000","last_clean":"2023-11-24T08:06:05.059707+0000","last_became_active":"2023-11-24T08:06:04.008919+0000","last_became_peered":"2023-11-24T08:06:04.008919+0000","last_unstale":"2023-11-24T08:06:05.059707+0000","last_undegraded":"2023-11-24T08:06:05.059707+0000","last_fullsized":"2023-11-24T08:06:05.059707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012957+0000","last_change":"2023-11-24T08:06:04.018935+0000","last_active":"2023-11-24T08:06:05.012957+0000","last_peered":"2023-11-24T08:06:05.012957+0000","last_clean":"2023-11-24T08:06:05.012957+0000","last_became_active":"2023-11-24T08:06:04.018621+0000","last_became_peered":"2023-11-24T08:06:04.018621+0000","last_unstale":"2023-11-24T08:06:05.012957+0000","last_undegraded":"2023-11-24T08:06:05.012957+0000","last_fullsized":"2023-11-24T08:06:05.012957+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.009666+0000","last_change":"2023-11-24T08:06:04.010972+0000","last_active":"2023-11-24T08:06:05.009666+0000","last_peered":"2023-11-24T08:06:05.009666+0000","last_clean":"2023-11-24T08:06:05.009666+0000","last_became_active":"2023-11-24T08:06:04.010652+0000","last_became_peered":"2023-11-24T08:06:04.010652+0000","last_unstale":"2023-11-24T08:06:05.009666+0000","last_undegraded":"2023-11-24T08:06:05.009666+0000","last_fullsized":"2023-11-24T08:06:05.009666+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:27:55.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063332+0000","last_change":"2023-11-24T08:06:04.006772+0000","last_active":"2023-11-24T08:06:05.063332+0000","last_peered":"2023-11-24T08:06:05.063332+0000","last_clean":"2023-11-24T08:06:05.063332+0000","last_became_active":"2023-11-24T08:06:04.006479+0000","last_became_peered":"2023-11-24T08:06:04.006479+0000","last_unstale":"2023-11-24T08:06:05.063332+0000","last_undegraded":"2023-11-24T08:06:05.063332+0000","last_fullsized":"2023-11-24T08:06:05.063332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:19.049650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062581+0000","last_change":"2023-11-24T08:06:04.020109+0000","last_active":"2023-11-24T08:06:05.062581+0000","last_peered":"2023-11-24T08:06:05.062581+0000","last_clean":"2023-11-24T08:06:05.062581+0000","last_became_active":"2023-11-24T08:06:04.019031+0000","last_became_peered":"2023-11-24T08:06:04.019031+0000","last_unstale":"2023-11-24T08:06:05.062581+0000","last_undegraded":"2023-11-24T08:06:05.062581+0000","last_fullsized":"2023-11-24T08:06:05.062581+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:41:49.179117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011522+0000","last_change":"2023-11-24T08:06:04.024972+0000","last_active":"2023-11-24T08:06:05.011522+0000","last_peered":"2023-11-24T08:06:05.011522+0000","last_clean":"2023-11-24T08:06:05.011522+0000","last_became_active":"2023-11-24T08:06:04.024709+0000","last_became_peered":"2023-11-24T08:06:04.024709+0000","last_unstale":"2023-11-24T08:06:05.011522+0000","last_undegraded":"2023-11-24T08:06:05.011522+0000","last_fullsized":"2023-11-24T08:06:05.011522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:55.954459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061390+0000","last_change":"2023-11-24T08:06:04.013179+0000","last_active":"2023-11-24T08:06:05.061390+0000","last_peered":"2023-11-24T08:06:05.061390+0000","last_clean":"2023-11-24T08:06:05.061390+0000","last_became_active":"2023-11-24T08:06:04.011994+0000","last_became_peered":"2023-11-24T08:06:04.011994+0000","last_unstale":"2023-11-24T08:06:05.061390+0000","last_undegraded":"2023-11-24T08:06:05.061390+0000","last_fullsized":"2023-11-24T08:06:05.061390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:53.581888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059534+0000","last_change":"2023-11-24T08:06:04.004867+0000","last_active":"2023-11-24T08:06:05.059534+0000","last_peered":"2023-11-24T08:06:05.059534+0000","last_clean":"2023-11-24T08:06:05.059534+0000","last_became_active":"2023-11-24T08:06:04.002888+0000","last_became_peered":"2023-11-24T08:06:04.002888+0000","last_unstale":"2023-11-24T08:06:05.059534+0000","last_undegraded":"2023-11-24T08:06:05.059534+0000","last_fullsized":"2023-11-24T08:06:05.059534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:39:14.226693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059825+0000","last_change":"2023-11-24T08:06:04.002250+0000","last_active":"2023-11-24T08:06:05.059825+0000","last_peered":"2023-11-24T08:06:05.059825+0000","last_clean":"2023-11-24T08:06:05.059825+0000","last_became_active":"2023-11-24T08:06:04.001900+0000","last_became_peered":"2023-11-24T08:06:04.001900+0000","last_unstale":"2023-11-24T08:06:05.059825+0000","last_undegraded":"2023-11-24T08:06:05.059825+0000","last_fullsized":"2023-11-24T08:06:05.059825+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062488+0000","last_change":"2023-11-24T08:06:04.017744+0000","last_active":"2023-11-24T08:06:05.062488+0000","last_peered":"2023-11-24T08:06:05.062488+0000","last_clean":"2023-11-24T08:06:05.062488+0000","last_became_active":"2023-11-24T08:06:04.017416+0000","last_became_peered":"2023-11-24T08:06:04.017416+0000","last_unstale":"2023-11-24T08:06:05.062488+0000","last_undegraded":"2023-11-24T08:06:05.062488+0000","last_fullsized":"2023-11-24T08:06:05.062488+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010338+0000","last_change":"2023-11-24T08:06:04.013352+0000","last_active":"2023-11-24T08:06:05.010338+0000","last_peered":"2023-11-24T08:06:05.010338+0000","last_clean":"2023-11-24T08:06:05.010338+0000","last_became_active":"2023-11-24T08:06:04.013063+0000","last_became_peered":"2023-11-24T08:06:04.013063+0000","last_unstale":"2023-11-24T08:06:05.010338+0000","last_undegraded":"2023-11-24T08:06:05.010338+0000","last_fullsized":"2023-11-24T08:06:05.010338+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:38.767269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.013170+0000","last_change":"2023-11-24T08:06:04.018129+0000","last_active":"2023-11-24T08:06:05.013170+0000","last_peered":"2023-11-24T08:06:05.013170+0000","last_clean":"2023-11-24T08:06:05.013170+0000","last_became_active":"2023-11-24T08:06:04.017828+0000","last_became_peered":"2023-11-24T08:06:04.017828+0000","last_unstale":"2023-11-24T08:06:05.013170+0000","last_undegraded":"2023-11-24T08:06:05.013170+0000","last_fullsized":"2023-11-24T08:06:05.013170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:38.225050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060311+0000","last_change":"2023-11-24T08:06:04.019286+0000","last_active":"2023-11-24T08:06:05.060311+0000","last_peered":"2023-11-24T08:06:05.060311+0000","last_clean":"2023-11-24T08:06:05.060311+0000","last_became_active":"2023-11-24T08:06:04.018989+0000","last_became_peered":"2023-11-24T08:06:04.018989+0000","last_unstale":"2023-11-24T08:06:05.060311+0000","last_undegraded":"2023-11-24T08:06:05.060311+0000","last_fullsized":"2023-11-24T08:06:05.060311+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:29:05.688382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059853+0000","last_change":"2023-11-24T08:06:04.010807+0000","last_active":"2023-11-24T08:06:05.059853+0000","last_peered":"2023-11-24T08:06:05.059853+0000","last_clean":"2023-11-24T08:06:05.059853+0000","last_became_active":"2023-11-24T08:06:04.010522+0000","last_became_peered":"2023-11-24T08:06:04.010522+0000","last_unstale":"2023-11-24T08:06:05.059853+0000","last_undegraded":"2023-11-24T08:06:05.059853+0000","last_fullsized":"2023-11-24T08:06:05.059853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061749+0000","last_change":"2023-11-24T08:06:04.012852+0000","last_active":"2023-11-24T08:06:05.061749+0000","last_peered":"2023-11-24T08:06:05.061749+0000","last_clean":"2023-11-24T08:06:05.061749+0000","last_became_active":"2023-11-24T08:06:04.012485+0000","last_became_peered":"2023-11-24T08:06:04.012485+0000","last_unstale":"2023-11-24T08:06:05.061749+0000","last_undegraded":"2023-11-24T08:06:05.061749+0000","last_fullsized":"2023-11-24T08:06:05.061749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:29.289950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061478+0000","last_change":"2023-11-24T08:06:03.997180+0000","last_active":"2023-11-24T08:06:05.061478+0000","last_peered":"2023-11-24T08:06:05.061478+0000","last_clean":"2023-11-24T08:06:05.061478+0000","last_became_active":"2023-11-24T08:06:03.996914+0000","last_became_peered":"2023-11-24T08:06:03.996914+0000","last_unstale":"2023-11-24T08:06:05.061478+0000","last_undegraded":"2023-11-24T08:06:05.061478+0000","last_fullsized":"2023-11-24T08:06:05.061478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063068+0000","last_change":"2023-11-24T08:06:04.003370+0000","last_active":"2023-11-24T08:06:05.063068+0000","last_peered":"2023-11-24T08:06:05.063068+0000","last_clean":"2023-11-24T08:06:05.063068+0000","last_became_active":"2023-11-24T08:06:04.002192+0000","last_became_peered":"2023-11-24T08:06:04.002192+0000","last_unstale":"2023-11-24T08:06:05.063068+0000","last_undegraded":"2023-11-24T08:06:05.063068+0000","last_fullsized":"2023-11-24T08:06:05.063068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:38.322203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026124+0000","last_change":"2023-11-24T08:06:04.005668+0000","last_active":"2023-11-24T08:06:05.026124+0000","last_peered":"2023-11-24T08:06:05.026124+0000","last_clean":"2023-11-24T08:06:05.026124+0000","last_became_active":"2023-11-24T08:06:04.004259+0000","last_became_peered":"2023-11-24T08:06:04.004259+0000","last_unstale":"2023-11-24T08:06:05.026124+0000","last_undegraded":"2023-11-24T08:06:05.026124+0000","last_fullsized":"2023-11-24T08:06:05.026124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:45.842740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060937+0000","last_change":"2023-11-24T08:06:04.016824+0000","last_active":"2023-11-24T08:06:05.060937+0000","last_peered":"2023-11-24T08:06:05.060937+0000","last_clean":"2023-11-24T08:06:05.060937+0000","last_became_active":"2023-11-24T08:06:04.014499+0000","last_became_peered":"2023-11-24T08:06:04.014499+0000","last_unstale":"2023-11-24T08:06:05.060937+0000","last_undegraded":"2023-11-24T08:06:05.060937+0000","last_fullsized":"2023-11-24T08:06:05.060937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:52:46.846590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062605+0000","last_change":"2023-11-24T08:06:04.014098+0000","last_active":"2023-11-24T08:06:05.062605+0000","last_peered":"2023-11-24T08:06:05.062605+0000","last_clean":"2023-11-24T08:06:05.062605+0000","last_became_active":"2023-11-24T08:06:04.013523+0000","last_became_peered":"2023-11-24T08:06:04.013523+0000","last_unstale":"2023-11-24T08:06:05.062605+0000","last_undegraded":"2023-11-24T08:06:05.062605+0000","last_fullsized":"2023-11-24T08:06:05.062605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:43.310056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060370+0000","last_change":"2023-11-24T08:06:04.016120+0000","last_active":"2023-11-24T08:06:05.060370+0000","last_peered":"2023-11-24T08:06:05.060370+0000","last_clean":"2023-11-24T08:06:05.060370+0000","last_became_active":"2023-11-24T08:06:04.012298+0000","last_became_peered":"2023-11-24T08:06:04.012298+0000","last_unstale":"2023-11-24T08:06:05.060370+0000","last_undegraded":"2023-11-24T08:06:05.060370+0000","last_fullsized":"2023-11-24T08:06:05.060370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:20:50.383127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062246+0000","last_change":"2023-11-24T08:06:04.021679+0000","last_active":"2023-11-24T08:06:05.062246+0000","last_peered":"2023-11-24T08:06:05.062246+0000","last_clean":"2023-11-24T08:06:05.062246+0000","last_became_active":"2023-11-24T08:06:04.019533+0000","last_became_peered":"2023-11-24T08:06:04.019533+0000","last_unstale":"2023-11-24T08:06:05.062246+0000","last_undegraded":"2023-11-24T08:06:05.062246+0000","last_fullsized":"2023-11-24T08:06:05.062246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:19.049650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061727+0000","last_change":"2023-11-24T08:06:04.012503+0000","last_active":"2023-11-24T08:06:05.061727+0000","last_peered":"2023-11-24T08:06:05.061727+0000","last_clean":"2023-11-24T08:06:05.061727+0000","last_became_active":"2023-11-24T08:06:04.011603+0000","last_became_peered":"2023-11-24T08:06:04.011603+0000","last_unstale":"2023-11-24T08:06:05.061727+0000","last_undegraded":"2023-11-24T08:06:05.061727+0000","last_fullsized":"2023-11-24T08:06:05.061727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:08.896877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061843+0000","last_change":"2023-11-24T08:06:04.006406+0000","last_active":"2023-11-24T08:06:05.061843+0000","last_peered":"2023-11-24T08:06:05.061843+0000","last_clean":"2023-11-24T08:06:05.061843+0000","last_became_active":"2023-11-24T08:06:04.006025+0000","last_became_peered":"2023-11-24T08:06:04.006025+0000","last_unstale":"2023-11-24T08:06:05.061843+0000","last_undegraded":"2023-11-24T08:06:05.061843+0000","last_fullsized":"2023-11-24T08:06:05.061843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012238+0000","last_change":"2023-11-24T08:06:04.011599+0000","last_active":"2023-11-24T08:06:05.012238+0000","last_peered":"2023-11-24T08:06:05.012238+0000","last_clean":"2023-11-24T08:06:05.012238+0000","last_became_active":"2023-11-24T08:06:04.011178+0000","last_became_peered":"2023-11-24T08:06:04.011178+0000","last_unstale":"2023-11-24T08:06:05.012238+0000","last_undegraded":"2023-11-24T08:06:05.012238+0000","last_fullsized":"2023-11-24T08:06:05.012238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011820+0000","last_change":"2023-11-24T08:06:04.008336+0000","last_active":"2023-11-24T08:06:05.011820+0000","last_peered":"2023-11-24T08:06:05.011820+0000","last_clean":"2023-11-24T08:06:05.011820+0000","last_became_active":"2023-11-24T08:06:04.008050+0000","last_became_peered":"2023-11-24T08:06:04.008050+0000","last_unstale":"2023-11-24T08:06:05.011820+0000","last_undegraded":"2023-11-24T08:06:05.011820+0000","last_fullsized":"2023-11-24T08:06:05.011820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.009646+0000","last_change":"2023-11-24T08:06:04.015110+0000","last_active":"2023-11-24T08:06:05.009646+0000","last_peered":"2023-11-24T08:06:05.009646+0000","last_clean":"2023-11-24T08:06:05.009646+0000","last_became_active":"2023-11-24T08:06:04.013150+0000","last_became_peered":"2023-11-24T08:06:04.013150+0000","last_unstale":"2023-11-24T08:06:05.009646+0000","last_undegraded":"2023-11-24T08:06:05.009646+0000","last_fullsized":"2023-11-24T08:06:05.009646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:52:46.846590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026917+0000","last_change":"2023-11-24T08:06:04.005018+0000","last_active":"2023-11-24T08:06:05.026917+0000","last_peered":"2023-11-24T08:06:05.026917+0000","last_clean":"2023-11-24T08:06:05.026917+0000","last_became_active":"2023-11-24T08:06:04.004687+0000","last_became_peered":"2023-11-24T08:06:04.004687+0000","last_unstale":"2023-11-24T08:06:05.026917+0000","last_undegraded":"2023-11-24T08:06:05.026917+0000","last_fullsized":"2023-11-24T08:06:05.026917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.019309+0000","last_change":"2023-11-24T08:06:03.008351+0000","last_active":"2023-11-24T08:06:04.019309+0000","last_peered":"2023-11-24T08:06:04.019309+0000","last_clean":"2023-11-24T08:06:04.019309+0000","last_became_active":"2023-11-24T08:06:03.008011+0000","last_became_peered":"2023-11-24T08:06:03.008011+0000","last_unstale":"2023-11-24T08:06:04.019309+0000","last_undegraded":"2023-11-24T08:06:04.019309+0000","last_fullsized":"2023-11-24T08:06:04.019309+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011765+0000","last_change":"2023-11-24T08:06:04.015529+0000","last_active":"2023-11-24T08:06:05.011765+0000","last_peered":"2023-11-24T08:06:05.011765+0000","last_clean":"2023-11-24T08:06:05.011765+0000","last_became_active":"2023-11-24T08:06:04.015237+0000","last_became_peered":"2023-11-24T08:06:04.015237+0000","last_unstale":"2023-11-24T08:06:05.011765+0000","last_undegraded":"2023-11-24T08:06:05.011765+0000","last_fullsized":"2023-11-24T08:06:05.011765+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.009961+0000","last_change":"2023-11-24T08:06:03.005602+0000","last_active":"2023-11-24T08:06:04.009961+0000","last_peered":"2023-11-24T08:06:04.009961+0000","last_clean":"2023-11-24T08:06:04.009961+0000","last_became_active":"2023-11-24T08:06:03.005346+0000","last_became_peered":"2023-11-24T08:06:03.005346+0000","last_unstale":"2023-11-24T08:06:04.009961+0000","last_undegraded":"2023-11-24T08:06:04.009961+0000","last_fullsized":"2023-11-24T08:06:04.009961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062141+0000","last_change":"2023-11-24T08:06:03.997421+0000","last_active":"2023-11-24T08:06:05.062141+0000","last_peered":"2023-11-24T08:06:05.062141+0000","last_clean":"2023-11-24T08:06:05.062141+0000","last_became_active":"2023-11-24T08:06:03.997145+0000","last_became_peered":"2023-11-24T08:06:03.997145+0000","last_unstale":"2023-11-24T08:06:05.062141+0000","last_undegraded":"2023-11-24T08:06:05.062141+0000","last_fullsized":"2023-11-24T08:06:05.062141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.023825+0000","last_change":"2023-11-24T08:06:02.997760+0000","last_active":"2023-11-24T08:06:04.023825+0000","last_peered":"2023-11-24T08:06:04.023825+0000","last_clean":"2023-11-24T08:06:04.023825+0000","last_became_active":"2023-11-24T08:06:02.997481+0000","last_became_peered":"2023-11-24T08:06:02.997481+0000","last_unstale":"2023-11-24T08:06:04.023825+0000","last_undegraded":"2023-11-24T08:06:04.023825+0000","last_fullsized":"2023-11-24T08:06:04.023825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:37.755467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062288+0000","last_change":"2023-11-24T08:06:04.019864+0000","last_active":"2023-11-24T08:06:05.062288+0000","last_peered":"2023-11-24T08:06:05.062288+0000","last_clean":"2023-11-24T08:06:05.062288+0000","last_became_active":"2023-11-24T08:06:04.018450+0000","last_became_peered":"2023-11-24T08:06:04.018450+0000","last_unstale":"2023-11-24T08:06:05.062288+0000","last_undegraded":"2023-11-24T08:06:05.062288+0000","last_fullsized":"2023-11-24T08:06:05.062288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:18.449398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.016510+0000","last_change":"2023-11-24T08:06:02.997108+0000","last_active":"2023-11-24T08:06:04.016510+0000","last_peered":"2023-11-24T08:06:04.016510+0000","last_clean":"2023-11-24T08:06:04.016510+0000","last_became_active":"2023-11-24T08:06:02.996880+0000","last_became_peered":"2023-11-24T08:06:02.996880+0000","last_unstale":"2023-11-24T08:06:04.016510+0000","last_undegraded":"2023-11-24T08:06:04.016510+0000","last_fullsized":"2023-11-24T08:06:04.016510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012659+0000","last_change":"2023-11-24T08:06:04.017101+0000","last_active":"2023-11-24T08:06:05.012659+0000","last_peered":"2023-11-24T08:06:05.012659+0000","last_clean":"2023-11-24T08:06:05.012659+0000","last_became_active":"2023-11-24T08:06:04.014260+0000","last_became_peered":"2023-11-24T08:06:04.014260+0000","last_unstale":"2023-11-24T08:06:05.012659+0000","last_undegraded":"2023-11-24T08:06:05.012659+0000","last_fullsized":"2023-11-24T08:06:05.012659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.012080+0000","last_change":"2023-11-24T08:06:03.004832+0000","last_active":"2023-11-24T08:06:04.012080+0000","last_peered":"2023-11-24T08:06:04.012080+0000","last_clean":"2023-11-24T08:06:04.012080+0000","last_became_active":"2023-11-24T08:06:03.004536+0000","last_became_peered":"2023-11-24T08:06:03.004536+0000","last_unstale":"2023-11-24T08:06:04.012080+0000","last_undegraded":"2023-11-24T08:06:04.012080+0000","last_fullsized":"2023-11-24T08:06:04.012080+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061318+0000","last_change":"2023-11-24T08:06:04.008187+0000","last_active":"2023-11-24T08:06:05.061318+0000","last_peered":"2023-11-24T08:06:05.061318+0000","last_clean":"2023-11-24T08:06:05.061318+0000","last_became_active":"2023-11-24T08:06:04.007852+0000","last_became_peered":"2023-11-24T08:06:04.007852+0000","last_unstale":"2023-11-24T08:06:05.061318+0000","last_undegraded":"2023-11-24T08:06:05.061318+0000","last_fullsized":"2023-11-24T08:06:05.061318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:27:55.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.011417+0000","last_change":"2023-11-24T08:06:02.993001+0000","last_active":"2023-11-24T08:06:04.011417+0000","last_peered":"2023-11-24T08:06:04.011417+0000","last_clean":"2023-11-24T08:06:04.011417+0000","last_became_active":"2023-11-24T08:06:02.992519+0000","last_became_peered":"2023-11-24T08:06:02.992519+0000","last_unstale":"2023-11-24T08:06:04.011417+0000","last_undegraded":"2023-11-24T08:06:04.011417+0000","last_fullsized":"2023-11-24T08:06:04.011417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061543+0000","last_change":"2023-11-24T08:06:04.008996+0000","last_active":"2023-11-24T08:06:05.061543+0000","last_peered":"2023-11-24T08:06:05.061543+0000","last_clean":"2023-11-24T08:06:05.061543+0000","last_became_active":"2023-11-24T08:06:04.008533+0000","last_became_peered":"2023-11-24T08:06:04.008533+0000","last_unstale":"2023-11-24T08:06:05.061543+0000","last_undegraded":"2023-11-24T08:06:05.061543+0000","last_fullsized":"2023-11-24T08:06:05.061543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.990855+0000","last_change":"2023-11-24T08:06:02.992653+0000","last_active":"2023-11-24T08:06:03.990855+0000","last_peered":"2023-11-24T08:06:03.990855+0000","last_clean":"2023-11-24T08:06:03.990855+0000","last_became_active":"2023-11-24T08:06:02.992420+0000","last_became_peered":"2023-11-24T08:06:02.992420+0000","last_unstale":"2023-11-24T08:06:03.990855+0000","last_undegraded":"2023-11-24T08:06:03.990855+0000","last_fullsized":"2023-11-24T08:06:03.990855+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:42.298254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060991+0000","last_change":"2023-11-24T08:06:04.014483+0000","last_active":"2023-11-24T08:06:05.060991+0000","last_peered":"2023-11-24T08:06:05.060991+0000","last_clean":"2023-11-24T08:06:05.060991+0000","last_became_active":"2023-11-24T08:06:04.014257+0000","last_became_peered":"2023-11-24T08:06:04.014257+0000","last_unstale":"2023-11-24T08:06:05.060991+0000","last_undegraded":"2023-11-24T08:06:05.060991+0000","last_fullsized":"2023-11-24T08:06:05.060991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:56.043697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.984304+0000","last_change":"2023-11-24T08:06:02.998354+0000","last_active":"2023-11-24T08:06:03.984304+0000","last_peered":"2023-11-24T08:06:03.984304+0000","last_clean":"2023-11-24T08:06:03.984304+0000","last_became_active":"2023-11-24T08:06:02.998065+0000","last_became_peered":"2023-11-24T08:06:02.998065+0000","last_unstale":"2023-11-24T08:06:03.984304+0000","last_undegraded":"2023-11-24T08:06:03.984304+0000","last_fullsized":"2023-11-24T08:06:03.984304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:57.508607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063429+0000","last_change":"2023-11-24T08:06:04.012119+0000","last_active":"2023-11-24T08:06:05.063429+0000","last_peered":"2023-11-24T08:06:05.063429+0000","last_clean":"2023-11-24T08:06:05.063429+0000","last_became_active":"2023-11-24T08:06:04.011694+0000","last_became_peered":"2023-11-24T08:06:04.011694+0000","last_unstale":"2023-11-24T08:06:05.063429+0000","last_undegraded":"2023-11-24T08:06:05.063429+0000","last_fullsized":"2023-11-24T08:06:05.063429+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994511+0000","last_change":"2023-11-24T08:06:03.010255+0000","last_active":"2023-11-24T08:06:03.994511+0000","last_peered":"2023-11-24T08:06:03.994511+0000","last_clean":"2023-11-24T08:06:03.994511+0000","last_became_active":"2023-11-24T08:06:03.009943+0000","last_became_peered":"2023-11-24T08:06:03.009943+0000","last_unstale":"2023-11-24T08:06:03.994511+0000","last_undegraded":"2023-11-24T08:06:03.994511+0000","last_fullsized":"2023-11-24T08:06:03.994511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:54.942656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060678+0000","last_change":"2023-11-24T08:06:04.011415+0000","last_active":"2023-11-24T08:06:05.060678+0000","last_peered":"2023-11-24T08:06:05.060678+0000","last_clean":"2023-11-24T08:06:05.060678+0000","last_became_active":"2023-11-24T08:06:04.011126+0000","last_became_peered":"2023-11-24T08:06:04.011126+0000","last_unstale":"2023-11-24T08:06:05.060678+0000","last_undegraded":"2023-11-24T08:06:05.060678+0000","last_fullsized":"2023-11-24T08:06:05.060678+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:34:53.211888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.980344+0000","last_change":"2023-11-24T08:06:03.004192+0000","last_active":"2023-11-24T08:06:03.980344+0000","last_peered":"2023-11-24T08:06:03.980344+0000","last_clean":"2023-11-24T08:06:03.980344+0000","last_became_active":"2023-11-24T08:06:03.003913+0000","last_became_peered":"2023-11-24T08:06:03.003913+0000","last_unstale":"2023-11-24T08:06:03.980344+0000","last_undegraded":"2023-11-24T08:06:03.980344+0000","last_fullsized":"2023-11-24T08:06:03.980344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061376+0000","last_change":"2023-11-24T08:06:04.010430+0000","last_active":"2023-11-24T08:06:05.061376+0000","last_peered":"2023-11-24T08:06:05.061376+0000","last_clean":"2023-11-24T08:06:05.061376+0000","last_became_active":"2023-11-24T08:06:04.010156+0000","last_became_peered":"2023-11-24T08:06:04.010156+0000","last_unstale":"2023-11-24T08:06:05.061376+0000","last_undegraded":"2023-11-24T08:06:05.061376+0000","last_fullsized":"2023-11-24T08:06:05.061376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:06:38.863342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.019405+0000","last_change":"2023-11-24T08:06:03.009734+0000","last_active":"2023-11-24T08:06:04.019405+0000","last_peered":"2023-11-24T08:06:04.019405+0000","last_clean":"2023-11-24T08:06:04.019405+0000","last_became_active":"2023-11-24T08:06:03.009416+0000","last_became_peered":"2023-11-24T08:06:03.009416+0000","last_unstale":"2023-11-24T08:06:04.019405+0000","last_undegraded":"2023-11-24T08:06:04.019405+0000","last_fullsized":"2023-11-24T08:06:04.019405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:41:48.167314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061927+0000","last_change":"2023-11-24T08:06:04.024388+0000","last_active":"2023-11-24T08:06:05.061927+0000","last_peered":"2023-11-24T08:06:05.061927+0000","last_clean":"2023-11-24T08:06:05.061927+0000","last_became_active":"2023-11-24T08:06:04.024132+0000","last_became_peered":"2023-11-24T08:06:04.024132+0000","last_unstale":"2023-11-24T08:06:05.061927+0000","last_undegraded":"2023-11-24T08:06:05.061927+0000","last_fullsized":"2023-11-24T08:06:05.061927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:38.767269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.031089+0000","last_change":"2023-11-24T08:06:02.998808+0000","last_active":"2023-11-24T08:06:04.031089+0000","last_peered":"2023-11-24T08:06:04.031089+0000","last_clean":"2023-11-24T08:06:04.031089+0000","last_became_active":"2023-11-24T08:06:02.998600+0000","last_became_peered":"2023-11-24T08:06:02.998600+0000","last_unstale":"2023-11-24T08:06:04.031089+0000","last_undegraded":"2023-11-24T08:06:04.031089+0000","last_fullsized":"2023-11-24T08:06:04.031089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:07.885075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060991+0000","last_change":"2023-11-24T08:06:04.014905+0000","last_active":"2023-11-24T08:06:05.060991+0000","last_peered":"2023-11-24T08:06:05.060991+0000","last_clean":"2023-11-24T08:06:05.060991+0000","last_became_active":"2023-11-24T08:06:04.014753+0000","last_became_peered":"2023-11-24T08:06:04.014753+0000","last_unstale":"2023-11-24T08:06:05.060991+0000","last_undegraded":"2023-11-24T08:06:05.060991+0000","last_fullsized":"2023-11-24T08:06:05.060991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:34.310112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.016651+0000","last_change":"2023-11-24T08:06:03.004639+0000","last_active":"2023-11-24T08:06:04.016651+0000","last_peered":"2023-11-24T08:06:04.016651+0000","last_clean":"2023-11-24T08:06:04.016651+0000","last_became_active":"2023-11-24T08:06:03.004341+0000","last_became_peered":"2023-11-24T08:06:03.004341+0000","last_unstale":"2023-11-24T08:06:04.016651+0000","last_undegraded":"2023-11-24T08:06:04.016651+0000","last_fullsized":"2023-11-24T08:06:04.016651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060166+0000","last_change":"2023-11-24T08:06:04.009125+0000","last_active":"2023-11-24T08:06:05.060166+0000","last_peered":"2023-11-24T08:06:05.060166+0000","last_clean":"2023-11-24T08:06:05.060166+0000","last_became_active":"2023-11-24T08:06:04.008706+0000","last_became_peered":"2023-11-24T08:06:04.008706+0000","last_unstale":"2023-11-24T08:06:05.060166+0000","last_undegraded":"2023-11-24T08:06:05.060166+0000","last_fullsized":"2023-11-24T08:06:05.060166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:38.225050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.989596+0000","last_change":"2023-11-24T08:06:03.005631+0000","last_active":"2023-11-24T08:06:03.989596+0000","last_peered":"2023-11-24T08:06:03.989596+0000","last_clean":"2023-11-24T08:06:03.989596+0000","last_became_active":"2023-11-24T08:06:03.004131+0000","last_became_peered":"2023-11-24T08:06:03.004131+0000","last_unstale":"2023-11-24T08:06:03.989596+0000","last_undegraded":"2023-11-24T08:06:03.989596+0000","last_fullsized":"2023-11-24T08:06:03.989596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010149+0000","last_change":"2023-11-24T08:06:04.015328+0000","last_active":"2023-11-24T08:06:05.010149+0000","last_peered":"2023-11-24T08:06:05.010149+0000","last_clean":"2023-11-24T08:06:05.010149+0000","last_became_active":"2023-11-24T08:06:04.014191+0000","last_became_peered":"2023-11-24T08:06:04.014191+0000","last_unstale":"2023-11-24T08:06:05.010149+0000","last_undegraded":"2023-11-24T08:06:05.010149+0000","last_fullsized":"2023-11-24T08:06:05.010149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:08.896877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.990756+0000","last_change":"2023-11-24T08:06:03.009985+0000","last_active":"2023-11-24T08:06:03.990756+0000","last_peered":"2023-11-24T08:06:03.990756+0000","last_clean":"2023-11-24T08:06:03.990756+0000","last_became_active":"2023-11-24T08:06:03.009676+0000","last_became_peered":"2023-11-24T08:06:03.009676+0000","last_unstale":"2023-11-24T08:06:03.990756+0000","last_undegraded":"2023-11-24T08:06:03.990756+0000","last_fullsized":"2023-11-24T08:06:03.990756+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:17.437596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012188+0000","last_change":"2023-11-24T08:06:04.017139+0000","last_active":"2023-11-24T08:06:05.012188+0000","last_peered":"2023-11-24T08:06:05.012188+0000","last_clean":"2023-11-24T08:06:05.012188+0000","last_became_active":"2023-11-24T08:06:04.016481+0000","last_became_peered":"2023-11-24T08:06:04.016481+0000","last_unstale":"2023-11-24T08:06:05.012188+0000","last_undegraded":"2023-11-24T08:06:05.012188+0000","last_fullsized":"2023-11-24T08:06:05.012188+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.026025+0000","last_change":"2023-11-24T08:06:02.996472+0000","last_active":"2023-11-24T08:06:04.026025+0000","last_peered":"2023-11-24T08:06:04.026025+0000","last_clean":"2023-11-24T08:06:04.026025+0000","last_became_active":"2023-11-24T08:06:02.996105+0000","last_became_peered":"2023-11-24T08:06:02.996105+0000","last_unstale":"2023-11-24T08:06:04.026025+0000","last_undegraded":"2023-11-24T08:06:04.026025+0000","last_fullsized":"2023-11-24T08:06:04.026025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:57.097374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063264+0000","last_change":"2023-11-24T08:06:04.011512+0000","last_active":"2023-11-24T08:06:05.063264+0000","last_peered":"2023-11-24T08:06:05.063264+0000","last_clean":"2023-11-24T08:06:05.063264+0000","last_became_active":"2023-11-24T08:06:04.011208+0000","last_became_peered":"2023-11-24T08:06:04.011208+0000","last_unstale":"2023-11-24T08:06:05.063264+0000","last_undegraded":"2023-11-24T08:06:05.063264+0000","last_fullsized":"2023-11-24T08:06:05.063264+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:56.043697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.023777+0000","last_change":"2023-11-24T08:06:03.006947+0000","last_active":"2023-11-24T08:06:04.023777+0000","last_peered":"2023-11-24T08:06:04.023777+0000","last_clean":"2023-11-24T08:06:04.023777+0000","last_became_active":"2023-11-24T08:06:03.006640+0000","last_became_peered":"2023-11-24T08:06:03.006640+0000","last_unstale":"2023-11-24T08:06:04.023777+0000","last_undegraded":"2023-11-24T08:06:04.023777+0000","last_fullsized":"2023-11-24T08:06:04.023777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062187+0000","last_change":"2023-11-24T08:06:04.015848+0000","last_active":"2023-11-24T08:06:05.062187+0000","last_peered":"2023-11-24T08:06:05.062187+0000","last_clean":"2023-11-24T08:06:05.062187+0000","last_became_active":"2023-11-24T08:06:04.015574+0000","last_became_peered":"2023-11-24T08:06:04.015574+0000","last_unstale":"2023-11-24T08:06:05.062187+0000","last_undegraded":"2023-11-24T08:06:05.062187+0000","last_fullsized":"2023-11-24T08:06:05.062187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:55.954459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.017612+0000","last_change":"2023-11-24T08:06:03.003338+0000","last_active":"2023-11-24T08:06:04.017612+0000","last_peered":"2023-11-24T08:06:04.017612+0000","last_clean":"2023-11-24T08:06:04.017612+0000","last_became_active":"2023-11-24T08:06:02.998184+0000","last_became_peered":"2023-11-24T08:06:02.998184+0000","last_unstale":"2023-11-24T08:06:04.017612+0000","last_undegraded":"2023-11-24T08:06:04.017612+0000","last_fullsized":"2023-11-24T08:06:04.017612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061039+0000","last_change":"2023-11-24T08:06:04.020991+0000","last_active":"2023-11-24T08:06:05.061039+0000","last_peered":"2023-11-24T08:06:05.061039+0000","last_clean":"2023-11-24T08:06:05.061039+0000","last_became_active":"2023-11-24T08:06:04.020725+0000","last_became_peered":"2023-11-24T08:06:04.020725+0000","last_unstale":"2023-11-24T08:06:05.061039+0000","last_undegraded":"2023-11-24T08:06:05.061039+0000","last_fullsized":"2023-11-24T08:06:05.061039+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:38.322203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.021082+0000","last_change":"2023-11-24T08:06:03.001694+0000","last_active":"2023-11-24T08:06:04.021082+0000","last_peered":"2023-11-24T08:06:04.021082+0000","last_clean":"2023-11-24T08:06:04.021082+0000","last_became_active":"2023-11-24T08:06:02.998616+0000","last_became_peered":"2023-11-24T08:06:02.998616+0000","last_unstale":"2023-11-24T08:06:04.021082+0000","last_undegraded":"2023-11-24T08:06:04.021082+0000","last_fullsized":"2023-11-24T08:06:04.021082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:14.099330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061929+0000","last_change":"2023-11-24T08:06:04.005239+0000","last_active":"2023-11-24T08:06:05.061929+0000","last_peered":"2023-11-24T08:06:05.061929+0000","last_clean":"2023-11-24T08:06:05.061929+0000","last_became_active":"2023-11-24T08:06:04.004973+0000","last_became_peered":"2023-11-24T08:06:04.004973+0000","last_unstale":"2023-11-24T08:06:05.061929+0000","last_undegraded":"2023-11-24T08:06:05.061929+0000","last_fullsized":"2023-11-24T08:06:05.061929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:36.128015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.985791+0000","last_change":"2023-11-24T08:06:02.993516+0000","last_active":"2023-11-24T08:06:03.985791+0000","last_peered":"2023-11-24T08:06:03.985791+0000","last_clean":"2023-11-24T08:06:03.985791+0000","last_became_active":"2023-11-24T08:06:02.992976+0000","last_became_peered":"2023-11-24T08:06:02.992976+0000","last_unstale":"2023-11-24T08:06:03.985791+0000","last_undegraded":"2023-11-24T08:06:03.985791+0000","last_fullsized":"2023-11-24T08:06:03.985791+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059342+0000","last_change":"2023-11-24T08:06:04.012243+0000","last_active":"2023-11-24T08:06:05.059342+0000","last_peered":"2023-11-24T08:06:05.059342+0000","last_clean":"2023-11-24T08:06:05.059342+0000","last_became_active":"2023-11-24T08:06:04.011797+0000","last_became_peered":"2023-11-24T08:06:04.011797+0000","last_unstale":"2023-11-24T08:06:05.059342+0000","last_undegraded":"2023-11-24T08:06:05.059342+0000","last_fullsized":"2023-11-24T08:06:05.059342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:23.489751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.024403+0000","last_change":"2023-11-24T08:06:02.998617+0000","last_active":"2023-11-24T08:06:04.024403+0000","last_peered":"2023-11-24T08:06:04.024403+0000","last_clean":"2023-11-24T08:06:04.024403+0000","last_became_active":"2023-11-24T08:06:02.998393+0000","last_became_peered":"2023-11-24T08:06:02.998393+0000","last_unstale":"2023-11-24T08:06:04.024403+0000","last_undegraded":"2023-11-24T08:06:04.024403+0000","last_fullsized":"2023-11-24T08:06:04.024403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:28.474060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059870+0000","last_change":"2023-11-24T08:06:03.997412+0000","last_active":"2023-11-24T08:06:05.059870+0000","last_peered":"2023-11-24T08:06:05.059870+0000","last_clean":"2023-11-24T08:06:05.059870+0000","last_became_active":"2023-11-24T08:06:03.996455+0000","last_became_peered":"2023-11-24T08:06:03.996455+0000","last_unstale":"2023-11-24T08:06:05.059870+0000","last_undegraded":"2023-11-24T08:06:05.059870+0000","last_fullsized":"2023-11-24T08:06:05.059870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:58.520409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994818+0000","last_change":"2023-11-24T08:06:03.005058+0000","last_active":"2023-11-24T08:06:03.994818+0000","last_peered":"2023-11-24T08:06:03.994818+0000","last_clean":"2023-11-24T08:06:03.994818+0000","last_became_active":"2023-11-24T08:06:03.004703+0000","last_became_peered":"2023-11-24T08:06:03.004703+0000","last_unstale":"2023-11-24T08:06:03.994818+0000","last_undegraded":"2023-11-24T08:06:03.994818+0000","last_fullsized":"2023-11-24T08:06:03.994818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059520+0000","last_change":"2023-11-24T08:06:03.994897+0000","last_active":"2023-11-24T08:06:05.059520+0000","last_peered":"2023-11-24T08:06:05.059520+0000","last_clean":"2023-11-24T08:06:05.059520+0000","last_became_active":"2023-11-24T08:06:03.994621+0000","last_became_peered":"2023-11-24T08:06:03.994621+0000","last_unstale":"2023-11-24T08:06:05.059520+0000","last_undegraded":"2023-11-24T08:06:05.059520+0000","last_fullsized":"2023-11-24T08:06:05.059520+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:29.485863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.982383+0000","last_change":"2023-11-24T08:06:03.003128+0000","last_active":"2023-11-24T08:06:03.982383+0000","last_peered":"2023-11-24T08:06:03.982383+0000","last_clean":"2023-11-24T08:06:03.982383+0000","last_became_active":"2023-11-24T08:06:03.002791+0000","last_became_peered":"2023-11-24T08:06:03.002791+0000","last_unstale":"2023-11-24T08:06:03.982383+0000","last_undegraded":"2023-11-24T08:06:03.982383+0000","last_fullsized":"2023-11-24T08:06:03.982383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010148+0000","last_change":"2023-11-24T08:06:04.017398+0000","last_active":"2023-11-24T08:06:05.010148+0000","last_peered":"2023-11-24T08:06:05.010148+0000","last_clean":"2023-11-24T08:06:05.010148+0000","last_became_active":"2023-11-24T08:06:04.017095+0000","last_became_peered":"2023-11-24T08:06:04.017095+0000","last_unstale":"2023-11-24T08:06:05.010148+0000","last_undegraded":"2023-11-24T08:06:05.010148+0000","last_fullsized":"2023-11-24T08:06:05.010148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:53.581888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.005909+0000","last_change":"2023-11-24T08:06:03.006761+0000","last_active":"2023-11-24T08:06:04.005909+0000","last_peered":"2023-11-24T08:06:04.005909+0000","last_clean":"2023-11-24T08:06:04.005909+0000","last_became_active":"2023-11-24T08:06:03.006418+0000","last_became_peered":"2023-11-24T08:06:03.006418+0000","last_unstale":"2023-11-24T08:06:04.005909+0000","last_undegraded":"2023-11-24T08:06:04.005909+0000","last_fullsized":"2023-11-24T08:06:04.005909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:42.298254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062156+0000","last_change":"2023-11-24T08:06:04.012846+0000","last_active":"2023-11-24T08:06:05.062156+0000","last_peered":"2023-11-24T08:06:05.062156+0000","last_clean":"2023-11-24T08:06:05.062156+0000","last_became_active":"2023-11-24T08:06:04.012509+0000","last_became_peered":"2023-11-24T08:06:04.012509+0000","last_unstale":"2023-11-24T08:06:05.062156+0000","last_undegraded":"2023-11-24T08:06:05.062156+0000","last_fullsized":"2023-11-24T08:06:05.062156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:43.822534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.021487+0000","last_change":"2023-11-24T08:06:03.008262+0000","last_active":"2023-11-24T08:06:04.021487+0000","last_peered":"2023-11-24T08:06:04.021487+0000","last_clean":"2023-11-24T08:06:04.021487+0000","last_became_active":"2023-11-24T08:06:03.007982+0000","last_became_peered":"2023-11-24T08:06:03.007982+0000","last_unstale":"2023-11-24T08:06:04.021487+0000","last_undegraded":"2023-11-24T08:06:04.021487+0000","last_fullsized":"2023-11-24T08:06:04.021487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062556+0000","last_change":"2023-11-24T08:06:04.010748+0000","last_active":"2023-11-24T08:06:05.062556+0000","last_peered":"2023-11-24T08:06:05.062556+0000","last_clean":"2023-11-24T08:06:05.062556+0000","last_became_active":"2023-11-24T08:06:04.010351+0000","last_became_peered":"2023-11-24T08:06:04.010351+0000","last_unstale":"2023-11-24T08:06:05.062556+0000","last_undegraded":"2023-11-24T08:06:05.062556+0000","last_fullsized":"2023-11-24T08:06:05.062556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.980472+0000","last_change":"2023-11-24T08:06:03.001269+0000","last_active":"2023-11-24T08:06:03.980472+0000","last_peered":"2023-11-24T08:06:03.980472+0000","last_clean":"2023-11-24T08:06:03.980472+0000","last_became_active":"2023-11-24T08:06:03.000568+0000","last_became_peered":"2023-11-24T08:06:03.000568+0000","last_unstale":"2023-11-24T08:06:03.980472+0000","last_undegraded":"2023-11-24T08:06:03.980472+0000","last_fullsized":"2023-11-24T08:06:03.980472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026681+0000","last_change":"2023-11-24T08:06:04.008832+0000","last_active":"2023-11-24T08:06:05.026681+0000","last_peered":"2023-11-24T08:06:05.026681+0000","last_clean":"2023-11-24T08:06:05.026681+0000","last_became_active":"2023-11-24T08:06:04.008583+0000","last_became_peered":"2023-11-24T08:06:04.008583+0000","last_unstale":"2023-11-24T08:06:05.026681+0000","last_undegraded":"2023-11-24T08:06:05.026681+0000","last_fullsized":"2023-11-24T08:06:05.026681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994141+0000","last_change":"2023-11-24T08:06:02.995680+0000","last_active":"2023-11-24T08:06:03.994141+0000","last_peered":"2023-11-24T08:06:03.994141+0000","last_clean":"2023-11-24T08:06:03.994141+0000","last_became_active":"2023-11-24T08:06:02.995421+0000","last_became_peered":"2023-11-24T08:06:02.995421+0000","last_unstale":"2023-11-24T08:06:03.994141+0000","last_undegraded":"2023-11-24T08:06:03.994141+0000","last_fullsized":"2023-11-24T08:06:03.994141+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:50.084260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011707+0000","last_change":"2023-11-24T08:06:04.003958+0000","last_active":"2023-11-24T08:06:05.011707+0000","last_peered":"2023-11-24T08:06:05.011707+0000","last_clean":"2023-11-24T08:06:05.011707+0000","last_became_active":"2023-11-24T08:06:04.003478+0000","last_became_peered":"2023-11-24T08:06:04.003478+0000","last_unstale":"2023-11-24T08:06:05.011707+0000","last_undegraded":"2023-11-24T08:06:05.011707+0000","last_fullsized":"2023-11-24T08:06:05.011707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.979841+0000","last_change":"2023-11-24T08:06:03.000694+0000","last_active":"2023-11-24T08:06:03.979841+0000","last_peered":"2023-11-24T08:06:03.979841+0000","last_clean":"2023-11-24T08:06:03.979841+0000","last_became_active":"2023-11-24T08:06:03.000285+0000","last_became_peered":"2023-11-24T08:06:03.000285+0000","last_unstale":"2023-11-24T08:06:03.979841+0000","last_undegraded":"2023-11-24T08:06:03.979841+0000","last_fullsized":"2023-11-24T08:06:03.979841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:57.508607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026651+0000","last_change":"2023-11-24T08:06:04.004758+0000","last_active":"2023-11-24T08:06:05.026651+0000","last_peered":"2023-11-24T08:06:05.026651+0000","last_clean":"2023-11-24T08:06:05.026651+0000","last_became_active":"2023-11-24T08:06:04.004484+0000","last_became_peered":"2023-11-24T08:06:04.004484+0000","last_unstale":"2023-11-24T08:06:05.026651+0000","last_undegraded":"2023-11-24T08:06:05.026651+0000","last_fullsized":"2023-11-24T08:06:05.026651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.989723+0000","last_change":"2023-11-24T08:06:03.008992+0000","last_active":"2023-11-24T08:06:03.989723+0000","last_peered":"2023-11-24T08:06:03.989723+0000","last_clean":"2023-11-24T08:06:03.989723+0000","last_became_active":"2023-11-24T08:06:03.008648+0000","last_became_peered":"2023-11-24T08:06:03.008648+0000","last_unstale":"2023-11-24T08:06:03.989723+0000","last_undegraded":"2023-11-24T08:06:03.989723+0000","last_fullsized":"2023-11-24T08:06:03.989723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:58.747325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059005+0000","last_change":"2023-11-24T08:06:04.011249+0000","last_active":"2023-11-24T08:06:05.059005+0000","last_peered":"2023-11-24T08:06:05.059005+0000","last_clean":"2023-11-24T08:06:05.059005+0000","last_became_active":"2023-11-24T08:06:04.009623+0000","last_became_peered":"2023-11-24T08:06:04.009623+0000","last_unstale":"2023-11-24T08:06:05.059005+0000","last_undegraded":"2023-11-24T08:06:05.059005+0000","last_fullsized":"2023-11-24T08:06:05.059005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.982606+0000","last_change":"2023-11-24T08:06:03.003843+0000","last_active":"2023-11-24T08:06:03.982606+0000","last_peered":"2023-11-24T08:06:03.982606+0000","last_clean":"2023-11-24T08:06:03.982606+0000","last_became_active":"2023-11-24T08:06:03.002741+0000","last_became_peered":"2023-11-24T08:06:03.002741+0000","last_unstale":"2023-11-24T08:06:03.982606+0000","last_undegraded":"2023-11-24T08:06:03.982606+0000","last_fullsized":"2023-11-24T08:06:03.982606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062696+0000","last_change":"2023-11-24T08:06:04.001276+0000","last_active":"2023-11-24T08:06:05.062696+0000","last_peered":"2023-11-24T08:06:05.062696+0000","last_clean":"2023-11-24T08:06:05.062696+0000","last_became_active":"2023-11-24T08:06:04.000935+0000","last_became_peered":"2023-11-24T08:06:04.000935+0000","last_unstale":"2023-11-24T08:06:05.062696+0000","last_undegraded":"2023-11-24T08:06:05.062696+0000","last_fullsized":"2023-11-24T08:06:05.062696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:34.310112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.988808+0000","last_change":"2023-11-24T08:06:03.009978+0000","last_active":"2023-11-24T08:06:03.988808+0000","last_peered":"2023-11-24T08:06:03.988808+0000","last_clean":"2023-11-24T08:06:03.988808+0000","last_became_active":"2023-11-24T08:06:03.009694+0000","last_became_peered":"2023-11-24T08:06:03.009694+0000","last_unstale":"2023-11-24T08:06:03.988808+0000","last_undegraded":"2023-11-24T08:06:03.988808+0000","last_fullsized":"2023-11-24T08:06:03.988808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:33.298310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011528+0000","last_change":"2023-11-24T08:06:04.006028+0000","last_active":"2023-11-24T08:06:05.011528+0000","last_peered":"2023-11-24T08:06:05.011528+0000","last_clean":"2023-11-24T08:06:05.011528+0000","last_became_active":"2023-11-24T08:06:04.004815+0000","last_became_peered":"2023-11-24T08:06:04.004815+0000","last_unstale":"2023-11-24T08:06:05.011528+0000","last_undegraded":"2023-11-24T08:06:05.011528+0000","last_fullsized":"2023-11-24T08:06:05.011528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.999191+0000","last_change":"2023-11-24T08:06:03.008297+0000","last_active":"2023-11-24T08:06:03.999191+0000","last_peered":"2023-11-24T08:06:03.999191+0000","last_clean":"2023-11-24T08:06:03.999191+0000","last_became_active":"2023-11-24T08:06:03.007985+0000","last_became_peered":"2023-11-24T08:06:03.007985+0000","last_unstale":"2023-11-24T08:06:03.999191+0000","last_undegraded":"2023-11-24T08:06:03.999191+0000","last_fullsized":"2023-11-24T08:06:03.999191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.997715+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:06:03.997715+0000","last_peered":"2023-11-24T08:06:03.997715+0000","last_clean":"2023-11-24T08:06:03.997715+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:06:03.997715+0000","last_undegraded":"2023-11-24T08:06:03.997715+0000","last_fullsized":"2023-11-24T08:06:03.997715+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":742357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.034},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.373}]},{"osd":2,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.099},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":1.776}]},{"osd":3,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.8640000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":2.206}]},{"osd":4,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.978},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":2.0600000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":2.4100000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":2.0990000000000002}]},{"osd":6,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":2.149}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.393}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.423},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.95299999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.349}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.3080000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.268}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.226}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.468},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.45}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.081}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.141},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.3720000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.878},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.218}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.018},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":1.3320000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.401}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.262}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4239999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.5369999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.0800000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.337}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.139},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3879999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.5899999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":1.47}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.605},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.29}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":153533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.0069999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.1850000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2629999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.109}]},{"osd":5,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.49},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.226}]},{"osd":6,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.54}]},{"osd":7,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.639}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.042}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.748},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.93600000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.131}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.298},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":1.6180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.464},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.794}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.835},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.526}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.597,"15min":1.597},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.742},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3809999999999998,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.76}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":153533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.99199999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.879}]},{"osd":4,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.1930000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.556},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":1.4730000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.9359999999999999,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.2150000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.28},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.3720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":742267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.3440000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.73199999999999998},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.629}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.98699999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.411},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.1100000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.306},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.2370000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.365},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.2130000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.4570000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:06:13.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:06:13.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.606+0000 7f41da8e5640 1 Processor -- start 2023-11-24T08:06:13.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.606+0000 7f41da8e5640 1 -- start start 2023-11-24T08:06:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.607+0000 7f41da8e5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 0x7f41d415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.607+0000 7f41da8e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41d40b2aa0 0x7f41d415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.607+0000 7f41da8e5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d415ab80 con 0x7f41d40b2630 2023-11-24T08:06:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.607+0000 7f41da8e5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d415a990 con 0x7f41d40b2aa0 2023-11-24T08:06:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.607+0000 7f41da8e5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d415e680 con 0x7f41d40b2250 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41d40b2aa0 0x7f41d415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41d40b2aa0 0x7f41d415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43348/0 (socket says 172.21.15.107:43348) 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 -- 172.21.15.107:0/3281572171 learned_addr learned my addr 172.21.15.107:0/3281572171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d3fff640 1 --2- 172.21.15.107:0/3281572171 >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 0x7f41d415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 -- 172.21.15.107:0/3281572171 >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 msgr2=0x7f41d415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 --2- 172.21.15.107:0/3281572171 >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 0x7f41d415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 -- 172.21.15.107:0/3281572171 >> v1:172.21.15.107:6789/0 conn(0x7f41d40b2630 legacy=0x7f41d40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d8e82640 1 -- 172.21.15.107:0/3281572171 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d415e960 con 0x7f41d40b2aa0 2023-11-24T08:06:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.608+0000 7f41d3fff640 1 --2- 172.21.15.107:0/3281572171 >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 0x7f41d415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:13.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.609+0000 7f41d8e82640 1 --2- 172.21.15.107:0/3281572171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d415b080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f41d4072a70 tx=0x7f41cc02e190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab6937f864441c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:13.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41d2ffd640 1 -- 172.21.15.107:0/3281572171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00b0d0 con 0x7f41d40b2aa0 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41d2ffd640 1 -- 172.21.15.107:0/3281572171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00b230 con 0x7f41d40b2aa0 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41d2ffd640 1 -- 172.21.15.107:0/3281572171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc009600 con 0x7f41d40b2aa0 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 -- 172.21.15.107:0/3281572171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 msgr2=0x7f41d415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 --2- 172.21.15.107:0/3281572171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d415b080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f41d4072a70 tx=0x7f41cc02e190 comp rx=0 tx=0).stop 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 -- 172.21.15.107:0/3281572171 shutdown_connections 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 --2- 172.21.15.107:0/3281572171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d415b080 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 --2- 172.21.15.107:0/3281572171 >> v2:172.21.15.120:3300/0 conn(0x7f41d40b2250 0x7f41d415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.610+0000 7f41da8e5640 1 -- 172.21.15.107:0/3281572171 >> 172.21.15.107:0/3281572171 conn(0x7f41d406a920 msgr2=0x7f41d406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.611+0000 7f41da8e5640 1 -- 172.21.15.107:0/3281572171 shutdown_connections 2023-11-24T08:06:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.611+0000 7f41da8e5640 1 -- 172.21.15.107:0/3281572171 wait complete. 2023-11-24T08:06:13.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.612+0000 7f41da8e5640 1 Processor -- start 2023-11-24T08:06:13.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.612+0000 7f41da8e5640 1 -- start start 2023-11-24T08:06:13.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d40b2250 0x7f41d40aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41d40b2630 0x7f41d4152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40af380 con 0x7f41d40b2aa0 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40b0d60 con 0x7f41d40b2250 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.613+0000 7f41da8e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40b1040 con 0x7f41d40b2630 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43350/0 (socket says 172.21.15.107:43350) 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 -- 172.21.15.107:0/1678232756 learned_addr learned my addr 172.21.15.107:0/1678232756 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 -- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41d40b2630 msgr2=0x7f41d4152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d37fe640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41d40b2630 0x7f41d4152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41d40b2630 0x7f41d4152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d3fff640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d40b2250 0x7f41d40aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 -- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d40b2250 msgr2=0x7f41d40aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d40b2250 0x7f41d40aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 -- 172.21.15.107:0/1678232756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc00bd40 con 0x7f41d40b2aa0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.614+0000 7f41d8e82640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f41cc038040 tx=0x7f41cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.615+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc036800 con 0x7f41d40b2aa0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.615+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc036980 con 0x7f41d40b2aa0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.615+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d40cd7c0 con 0x7f41d40b2aa0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.615+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc036d30 con 0x7f41d40b2aa0 2023-11-24T08:06:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.615+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d40af090 con 0x7f41d40b2aa0 2023-11-24T08:06:13.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.616+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d4067470 con 0x7f41d40b2aa0 2023-11-24T08:06:13.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.617+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f41cc04a460 con 0x7f41d40b2aa0 2023-11-24T08:06:13.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.618+0000 7f41d0ff9640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 0x7f41c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:13.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.618+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41cc0aca60 con 0x7f41d40b2aa0 2023-11-24T08:06:13.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.618+0000 7f41d3fff640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 0x7f41c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:13.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.619+0000 7f41d3fff640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 0x7f41c006d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f41c4008850 tx=0x7f41c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:13.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.622+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41cc077a70 con 0x7f41d40b2aa0 2023-11-24T08:06:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.810+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41d40cd640 con 0x7f41c006ae40 2023-11-24T08:06:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.825+0000 7f41d0ff9640 1 -- 172.21.15.107:0/1678232756 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286564 (secure 0 0 0) 0x7f41c400c040 con 0x7f41c006ae40 2023-11-24T08:06:13.831 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:13.832 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:06:13.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 msgr2=0x7f41c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 0x7f41c006d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f41c4008850 tx=0x7f41c4008040 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 msgr2=0x7f41d40cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f41cc038040 tx=0x7f41cc004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 shutdown_connections 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f41c006ae40 0x7f41c006d300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41d40b2aa0 0x7f41d40cba40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41d40b2630 0x7f41d4152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 --2- 172.21.15.107:0/1678232756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d40b2250 0x7f41d40aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 >> 172.21.15.107:0/1678232756 conn(0x7f41d406a920 msgr2=0x7f41d40ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 shutdown_connections 2023-11-24T08:06:13.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:13.831+0000 7f41da8e5640 1 -- 172.21.15.107:0/1678232756 wait complete. 2023-11-24T08:06:13.852 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2023-11-24T08:06:13.456595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220392,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754328,"statfs":{"total":773094113280,"available":772868431872,"internally_reserved":0,"allocated":4333568,"data_stored":2400748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":221236798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.480866"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011371+0000","last_change":"2023-11-24T08:06:04.014820+0000","last_active":"2023-11-24T08:06:05.011371+0000","last_peered":"2023-11-24T08:06:05.011371+0000","last_clean":"2023-11-24T08:06:05.011371+0000","last_became_active":"2023-11-24T08:06:04.013653+0000","last_became_peered":"2023-11-24T08:06:04.013653+0000","last_unstale":"2023-11-24T08:06:05.011371+0000","last_undegraded":"2023-11-24T08:06:05.011371+0000","last_fullsized":"2023-11-24T08:06:05.011371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:45.842740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060248+0000","last_change":"2023-11-24T08:06:04.008376+0000","last_active":"2023-11-24T08:06:05.060248+0000","last_peered":"2023-11-24T08:06:05.060248+0000","last_clean":"2023-11-24T08:06:05.060248+0000","last_became_active":"2023-11-24T08:06:04.008093+0000","last_became_peered":"2023-11-24T08:06:04.008093+0000","last_unstale":"2023-11-24T08:06:05.060248+0000","last_undegraded":"2023-11-24T08:06:05.060248+0000","last_fullsized":"2023-11-24T08:06:05.060248+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:53:26.874992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062267+0000","last_change":"2023-11-24T08:06:04.019222+0000","last_active":"2023-11-24T08:06:05.062267+0000","last_peered":"2023-11-24T08:06:05.062267+0000","last_clean":"2023-11-24T08:06:05.062267+0000","last_became_active":"2023-11-24T08:06:04.018903+0000","last_became_peered":"2023-11-24T08:06:04.018903+0000","last_unstale":"2023-11-24T08:06:05.062267+0000","last_undegraded":"2023-11-24T08:06:05.062267+0000","last_fullsized":"2023-11-24T08:06:05.062267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060961+0000","last_change":"2023-11-24T08:06:04.019730+0000","last_active":"2023-11-24T08:06:05.060961+0000","last_peered":"2023-11-24T08:06:05.060961+0000","last_clean":"2023-11-24T08:06:05.060961+0000","last_became_active":"2023-11-24T08:06:04.019090+0000","last_became_peered":"2023-11-24T08:06:04.019090+0000","last_unstale":"2023-11-24T08:06:05.060961+0000","last_undegraded":"2023-11-24T08:06:05.060961+0000","last_fullsized":"2023-11-24T08:06:05.060961+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061427+0000","last_change":"2023-11-24T08:06:04.012201+0000","last_active":"2023-11-24T08:06:05.061427+0000","last_peered":"2023-11-24T08:06:05.061427+0000","last_clean":"2023-11-24T08:06:05.061427+0000","last_became_active":"2023-11-24T08:06:04.011833+0000","last_became_peered":"2023-11-24T08:06:04.011833+0000","last_unstale":"2023-11-24T08:06:05.061427+0000","last_undegraded":"2023-11-24T08:06:05.061427+0000","last_fullsized":"2023-11-24T08:06:05.061427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:15.111132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059707+0000","last_change":"2023-11-24T08:06:04.010695+0000","last_active":"2023-11-24T08:06:05.059707+0000","last_peered":"2023-11-24T08:06:05.059707+0000","last_clean":"2023-11-24T08:06:05.059707+0000","last_became_active":"2023-11-24T08:06:04.008919+0000","last_became_peered":"2023-11-24T08:06:04.008919+0000","last_unstale":"2023-11-24T08:06:05.059707+0000","last_undegraded":"2023-11-24T08:06:05.059707+0000","last_fullsized":"2023-11-24T08:06:05.059707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012957+0000","last_change":"2023-11-24T08:06:04.018935+0000","last_active":"2023-11-24T08:06:05.012957+0000","last_peered":"2023-11-24T08:06:05.012957+0000","last_clean":"2023-11-24T08:06:05.012957+0000","last_became_active":"2023-11-24T08:06:04.018621+0000","last_became_peered":"2023-11-24T08:06:04.018621+0000","last_unstale":"2023-11-24T08:06:05.012957+0000","last_undegraded":"2023-11-24T08:06:05.012957+0000","last_fullsized":"2023-11-24T08:06:05.012957+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.009666+0000","last_change":"2023-11-24T08:06:04.010972+0000","last_active":"2023-11-24T08:06:05.009666+0000","last_peered":"2023-11-24T08:06:05.009666+0000","last_clean":"2023-11-24T08:06:05.009666+0000","last_became_active":"2023-11-24T08:06:04.010652+0000","last_became_peered":"2023-11-24T08:06:04.010652+0000","last_unstale":"2023-11-24T08:06:05.009666+0000","last_undegraded":"2023-11-24T08:06:05.009666+0000","last_fullsized":"2023-11-24T08:06:05.009666+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:27:55.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063332+0000","last_change":"2023-11-24T08:06:04.006772+0000","last_active":"2023-11-24T08:06:05.063332+0000","last_peered":"2023-11-24T08:06:05.063332+0000","last_clean":"2023-11-24T08:06:05.063332+0000","last_became_active":"2023-11-24T08:06:04.006479+0000","last_became_peered":"2023-11-24T08:06:04.006479+0000","last_unstale":"2023-11-24T08:06:05.063332+0000","last_undegraded":"2023-11-24T08:06:05.063332+0000","last_fullsized":"2023-11-24T08:06:05.063332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:19.049650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062581+0000","last_change":"2023-11-24T08:06:04.020109+0000","last_active":"2023-11-24T08:06:05.062581+0000","last_peered":"2023-11-24T08:06:05.062581+0000","last_clean":"2023-11-24T08:06:05.062581+0000","last_became_active":"2023-11-24T08:06:04.019031+0000","last_became_peered":"2023-11-24T08:06:04.019031+0000","last_unstale":"2023-11-24T08:06:05.062581+0000","last_undegraded":"2023-11-24T08:06:05.062581+0000","last_fullsized":"2023-11-24T08:06:05.062581+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:41:49.179117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011522+0000","last_change":"2023-11-24T08:06:04.024972+0000","last_active":"2023-11-24T08:06:05.011522+0000","last_peered":"2023-11-24T08:06:05.011522+0000","last_clean":"2023-11-24T08:06:05.011522+0000","last_became_active":"2023-11-24T08:06:04.024709+0000","last_became_peered":"2023-11-24T08:06:04.024709+0000","last_unstale":"2023-11-24T08:06:05.011522+0000","last_undegraded":"2023-11-24T08:06:05.011522+0000","last_fullsized":"2023-11-24T08:06:05.011522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:55.954459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061390+0000","last_change":"2023-11-24T08:06:04.013179+0000","last_active":"2023-11-24T08:06:05.061390+0000","last_peered":"2023-11-24T08:06:05.061390+0000","last_clean":"2023-11-24T08:06:05.061390+0000","last_became_active":"2023-11-24T08:06:04.011994+0000","last_became_peered":"2023-11-24T08:06:04.011994+0000","last_unstale":"2023-11-24T08:06:05.061390+0000","last_undegraded":"2023-11-24T08:06:05.061390+0000","last_fullsized":"2023-11-24T08:06:05.061390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:53.581888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059534+0000","last_change":"2023-11-24T08:06:04.004867+0000","last_active":"2023-11-24T08:06:05.059534+0000","last_peered":"2023-11-24T08:06:05.059534+0000","last_clean":"2023-11-24T08:06:05.059534+0000","last_became_active":"2023-11-24T08:06:04.002888+0000","last_became_peered":"2023-11-24T08:06:04.002888+0000","last_unstale":"2023-11-24T08:06:05.059534+0000","last_undegraded":"2023-11-24T08:06:05.059534+0000","last_fullsized":"2023-11-24T08:06:05.059534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:39:14.226693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059825+0000","last_change":"2023-11-24T08:06:04.002250+0000","last_active":"2023-11-24T08:06:05.059825+0000","last_peered":"2023-11-24T08:06:05.059825+0000","last_clean":"2023-11-24T08:06:05.059825+0000","last_became_active":"2023-11-24T08:06:04.001900+0000","last_became_peered":"2023-11-24T08:06:04.001900+0000","last_unstale":"2023-11-24T08:06:05.059825+0000","last_undegraded":"2023-11-24T08:06:05.059825+0000","last_fullsized":"2023-11-24T08:06:05.059825+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062488+0000","last_change":"2023-11-24T08:06:04.017744+0000","last_active":"2023-11-24T08:06:05.062488+0000","last_peered":"2023-11-24T08:06:05.062488+0000","last_clean":"2023-11-24T08:06:05.062488+0000","last_became_active":"2023-11-24T08:06:04.017416+0000","last_became_peered":"2023-11-24T08:06:04.017416+0000","last_unstale":"2023-11-24T08:06:05.062488+0000","last_undegraded":"2023-11-24T08:06:05.062488+0000","last_fullsized":"2023-11-24T08:06:05.062488+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010338+0000","last_change":"2023-11-24T08:06:04.013352+0000","last_active":"2023-11-24T08:06:05.010338+0000","last_peered":"2023-11-24T08:06:05.010338+0000","last_clean":"2023-11-24T08:06:05.010338+0000","last_became_active":"2023-11-24T08:06:04.013063+0000","last_became_peered":"2023-11-24T08:06:04.013063+0000","last_unstale":"2023-11-24T08:06:05.010338+0000","last_undegraded":"2023-11-24T08:06:05.010338+0000","last_fullsized":"2023-11-24T08:06:05.010338+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:38.767269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.013170+0000","last_change":"2023-11-24T08:06:04.018129+0000","last_active":"2023-11-24T08:06:05.013170+0000","last_peered":"2023-11-24T08:06:05.013170+0000","last_clean":"2023-11-24T08:06:05.013170+0000","last_became_active":"2023-11-24T08:06:04.017828+0000","last_became_peered":"2023-11-24T08:06:04.017828+0000","last_unstale":"2023-11-24T08:06:05.013170+0000","last_undegraded":"2023-11-24T08:06:05.013170+0000","last_fullsized":"2023-11-24T08:06:05.013170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:38.225050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060311+0000","last_change":"2023-11-24T08:06:04.019286+0000","last_active":"2023-11-24T08:06:05.060311+0000","last_peered":"2023-11-24T08:06:05.060311+0000","last_clean":"2023-11-24T08:06:05.060311+0000","last_became_active":"2023-11-24T08:06:04.018989+0000","last_became_peered":"2023-11-24T08:06:04.018989+0000","last_unstale":"2023-11-24T08:06:05.060311+0000","last_undegraded":"2023-11-24T08:06:05.060311+0000","last_fullsized":"2023-11-24T08:06:05.060311+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:29:05.688382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059853+0000","last_change":"2023-11-24T08:06:04.010807+0000","last_active":"2023-11-24T08:06:05.059853+0000","last_peered":"2023-11-24T08:06:05.059853+0000","last_clean":"2023-11-24T08:06:05.059853+0000","last_became_active":"2023-11-24T08:06:04.010522+0000","last_became_peered":"2023-11-24T08:06:04.010522+0000","last_unstale":"2023-11-24T08:06:05.059853+0000","last_undegraded":"2023-11-24T08:06:05.059853+0000","last_fullsized":"2023-11-24T08:06:05.059853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061749+0000","last_change":"2023-11-24T08:06:04.012852+0000","last_active":"2023-11-24T08:06:05.061749+0000","last_peered":"2023-11-24T08:06:05.061749+0000","last_clean":"2023-11-24T08:06:05.061749+0000","last_became_active":"2023-11-24T08:06:04.012485+0000","last_became_peered":"2023-11-24T08:06:04.012485+0000","last_unstale":"2023-11-24T08:06:05.061749+0000","last_undegraded":"2023-11-24T08:06:05.061749+0000","last_fullsized":"2023-11-24T08:06:05.061749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:29.289950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061478+0000","last_change":"2023-11-24T08:06:03.997180+0000","last_active":"2023-11-24T08:06:05.061478+0000","last_peered":"2023-11-24T08:06:05.061478+0000","last_clean":"2023-11-24T08:06:05.061478+0000","last_became_active":"2023-11-24T08:06:03.996914+0000","last_became_peered":"2023-11-24T08:06:03.996914+0000","last_unstale":"2023-11-24T08:06:05.061478+0000","last_undegraded":"2023-11-24T08:06:05.061478+0000","last_fullsized":"2023-11-24T08:06:05.061478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063068+0000","last_change":"2023-11-24T08:06:04.003370+0000","last_active":"2023-11-24T08:06:05.063068+0000","last_peered":"2023-11-24T08:06:05.063068+0000","last_clean":"2023-11-24T08:06:05.063068+0000","last_became_active":"2023-11-24T08:06:04.002192+0000","last_became_peered":"2023-11-24T08:06:04.002192+0000","last_unstale":"2023-11-24T08:06:05.063068+0000","last_undegraded":"2023-11-24T08:06:05.063068+0000","last_fullsized":"2023-11-24T08:06:05.063068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:38.322203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026124+0000","last_change":"2023-11-24T08:06:04.005668+0000","last_active":"2023-11-24T08:06:05.026124+0000","last_peered":"2023-11-24T08:06:05.026124+0000","last_clean":"2023-11-24T08:06:05.026124+0000","last_became_active":"2023-11-24T08:06:04.004259+0000","last_became_peered":"2023-11-24T08:06:04.004259+0000","last_unstale":"2023-11-24T08:06:05.026124+0000","last_undegraded":"2023-11-24T08:06:05.026124+0000","last_fullsized":"2023-11-24T08:06:05.026124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:45.842740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060937+0000","last_change":"2023-11-24T08:06:04.016824+0000","last_active":"2023-11-24T08:06:05.060937+0000","last_peered":"2023-11-24T08:06:05.060937+0000","last_clean":"2023-11-24T08:06:05.060937+0000","last_became_active":"2023-11-24T08:06:04.014499+0000","last_became_peered":"2023-11-24T08:06:04.014499+0000","last_unstale":"2023-11-24T08:06:05.060937+0000","last_undegraded":"2023-11-24T08:06:05.060937+0000","last_fullsized":"2023-11-24T08:06:05.060937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:52:46.846590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062605+0000","last_change":"2023-11-24T08:06:04.014098+0000","last_active":"2023-11-24T08:06:05.062605+0000","last_peered":"2023-11-24T08:06:05.062605+0000","last_clean":"2023-11-24T08:06:05.062605+0000","last_became_active":"2023-11-24T08:06:04.013523+0000","last_became_peered":"2023-11-24T08:06:04.013523+0000","last_unstale":"2023-11-24T08:06:05.062605+0000","last_undegraded":"2023-11-24T08:06:05.062605+0000","last_fullsized":"2023-11-24T08:06:05.062605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:43.310056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060370+0000","last_change":"2023-11-24T08:06:04.016120+0000","last_active":"2023-11-24T08:06:05.060370+0000","last_peered":"2023-11-24T08:06:05.060370+0000","last_clean":"2023-11-24T08:06:05.060370+0000","last_became_active":"2023-11-24T08:06:04.012298+0000","last_became_peered":"2023-11-24T08:06:04.012298+0000","last_unstale":"2023-11-24T08:06:05.060370+0000","last_undegraded":"2023-11-24T08:06:05.060370+0000","last_fullsized":"2023-11-24T08:06:05.060370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:20:50.383127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062246+0000","last_change":"2023-11-24T08:06:04.021679+0000","last_active":"2023-11-24T08:06:05.062246+0000","last_peered":"2023-11-24T08:06:05.062246+0000","last_clean":"2023-11-24T08:06:05.062246+0000","last_became_active":"2023-11-24T08:06:04.019533+0000","last_became_peered":"2023-11-24T08:06:04.019533+0000","last_unstale":"2023-11-24T08:06:05.062246+0000","last_undegraded":"2023-11-24T08:06:05.062246+0000","last_fullsized":"2023-11-24T08:06:05.062246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:19.049650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061727+0000","last_change":"2023-11-24T08:06:04.012503+0000","last_active":"2023-11-24T08:06:05.061727+0000","last_peered":"2023-11-24T08:06:05.061727+0000","last_clean":"2023-11-24T08:06:05.061727+0000","last_became_active":"2023-11-24T08:06:04.011603+0000","last_became_peered":"2023-11-24T08:06:04.011603+0000","last_unstale":"2023-11-24T08:06:05.061727+0000","last_undegraded":"2023-11-24T08:06:05.061727+0000","last_fullsized":"2023-11-24T08:06:05.061727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:08.896877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061843+0000","last_change":"2023-11-24T08:06:04.006406+0000","last_active":"2023-11-24T08:06:05.061843+0000","last_peered":"2023-11-24T08:06:05.061843+0000","last_clean":"2023-11-24T08:06:05.061843+0000","last_became_active":"2023-11-24T08:06:04.006025+0000","last_became_peered":"2023-11-24T08:06:04.006025+0000","last_unstale":"2023-11-24T08:06:05.061843+0000","last_undegraded":"2023-11-24T08:06:05.061843+0000","last_fullsized":"2023-11-24T08:06:05.061843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012238+0000","last_change":"2023-11-24T08:06:04.011599+0000","last_active":"2023-11-24T08:06:05.012238+0000","last_peered":"2023-11-24T08:06:05.012238+0000","last_clean":"2023-11-24T08:06:05.012238+0000","last_became_active":"2023-11-24T08:06:04.011178+0000","last_became_peered":"2023-11-24T08:06:04.011178+0000","last_unstale":"2023-11-24T08:06:05.012238+0000","last_undegraded":"2023-11-24T08:06:05.012238+0000","last_fullsized":"2023-11-24T08:06:05.012238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:11.692262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011820+0000","last_change":"2023-11-24T08:06:04.008336+0000","last_active":"2023-11-24T08:06:05.011820+0000","last_peered":"2023-11-24T08:06:05.011820+0000","last_clean":"2023-11-24T08:06:05.011820+0000","last_became_active":"2023-11-24T08:06:04.008050+0000","last_became_peered":"2023-11-24T08:06:04.008050+0000","last_unstale":"2023-11-24T08:06:05.011820+0000","last_undegraded":"2023-11-24T08:06:05.011820+0000","last_fullsized":"2023-11-24T08:06:05.011820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.009646+0000","last_change":"2023-11-24T08:06:04.015110+0000","last_active":"2023-11-24T08:06:05.009646+0000","last_peered":"2023-11-24T08:06:05.009646+0000","last_clean":"2023-11-24T08:06:05.009646+0000","last_became_active":"2023-11-24T08:06:04.013150+0000","last_became_peered":"2023-11-24T08:06:04.013150+0000","last_unstale":"2023-11-24T08:06:05.009646+0000","last_undegraded":"2023-11-24T08:06:05.009646+0000","last_fullsized":"2023-11-24T08:06:05.009646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:52:46.846590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026917+0000","last_change":"2023-11-24T08:06:04.005018+0000","last_active":"2023-11-24T08:06:05.026917+0000","last_peered":"2023-11-24T08:06:05.026917+0000","last_clean":"2023-11-24T08:06:05.026917+0000","last_became_active":"2023-11-24T08:06:04.004687+0000","last_became_peered":"2023-11-24T08:06:04.004687+0000","last_unstale":"2023-11-24T08:06:05.026917+0000","last_undegraded":"2023-11-24T08:06:05.026917+0000","last_fullsized":"2023-11-24T08:06:05.026917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:51.096062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.019309+0000","last_change":"2023-11-24T08:06:03.008351+0000","last_active":"2023-11-24T08:06:04.019309+0000","last_peered":"2023-11-24T08:06:04.019309+0000","last_clean":"2023-11-24T08:06:04.019309+0000","last_became_active":"2023-11-24T08:06:03.008011+0000","last_became_peered":"2023-11-24T08:06:03.008011+0000","last_unstale":"2023-11-24T08:06:04.019309+0000","last_undegraded":"2023-11-24T08:06:04.019309+0000","last_fullsized":"2023-11-24T08:06:04.019309+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011765+0000","last_change":"2023-11-24T08:06:04.015529+0000","last_active":"2023-11-24T08:06:05.011765+0000","last_peered":"2023-11-24T08:06:05.011765+0000","last_clean":"2023-11-24T08:06:05.011765+0000","last_became_active":"2023-11-24T08:06:04.015237+0000","last_became_peered":"2023-11-24T08:06:04.015237+0000","last_unstale":"2023-11-24T08:06:05.011765+0000","last_undegraded":"2023-11-24T08:06:05.011765+0000","last_fullsized":"2023-11-24T08:06:05.011765+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.009961+0000","last_change":"2023-11-24T08:06:03.005602+0000","last_active":"2023-11-24T08:06:04.009961+0000","last_peered":"2023-11-24T08:06:04.009961+0000","last_clean":"2023-11-24T08:06:04.009961+0000","last_became_active":"2023-11-24T08:06:03.005346+0000","last_became_peered":"2023-11-24T08:06:03.005346+0000","last_unstale":"2023-11-24T08:06:04.009961+0000","last_undegraded":"2023-11-24T08:06:04.009961+0000","last_fullsized":"2023-11-24T08:06:04.009961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062141+0000","last_change":"2023-11-24T08:06:03.997421+0000","last_active":"2023-11-24T08:06:05.062141+0000","last_peered":"2023-11-24T08:06:05.062141+0000","last_clean":"2023-11-24T08:06:05.062141+0000","last_became_active":"2023-11-24T08:06:03.997145+0000","last_became_peered":"2023-11-24T08:06:03.997145+0000","last_unstale":"2023-11-24T08:06:05.062141+0000","last_undegraded":"2023-11-24T08:06:05.062141+0000","last_fullsized":"2023-11-24T08:06:05.062141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.023825+0000","last_change":"2023-11-24T08:06:02.997760+0000","last_active":"2023-11-24T08:06:04.023825+0000","last_peered":"2023-11-24T08:06:04.023825+0000","last_clean":"2023-11-24T08:06:04.023825+0000","last_became_active":"2023-11-24T08:06:02.997481+0000","last_became_peered":"2023-11-24T08:06:02.997481+0000","last_unstale":"2023-11-24T08:06:04.023825+0000","last_undegraded":"2023-11-24T08:06:04.023825+0000","last_fullsized":"2023-11-24T08:06:04.023825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:37.755467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062288+0000","last_change":"2023-11-24T08:06:04.019864+0000","last_active":"2023-11-24T08:06:05.062288+0000","last_peered":"2023-11-24T08:06:05.062288+0000","last_clean":"2023-11-24T08:06:05.062288+0000","last_became_active":"2023-11-24T08:06:04.018450+0000","last_became_peered":"2023-11-24T08:06:04.018450+0000","last_unstale":"2023-11-24T08:06:05.062288+0000","last_undegraded":"2023-11-24T08:06:05.062288+0000","last_fullsized":"2023-11-24T08:06:05.062288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:18.449398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.016510+0000","last_change":"2023-11-24T08:06:02.997108+0000","last_active":"2023-11-24T08:06:04.016510+0000","last_peered":"2023-11-24T08:06:04.016510+0000","last_clean":"2023-11-24T08:06:04.016510+0000","last_became_active":"2023-11-24T08:06:02.996880+0000","last_became_peered":"2023-11-24T08:06:02.996880+0000","last_unstale":"2023-11-24T08:06:04.016510+0000","last_undegraded":"2023-11-24T08:06:04.016510+0000","last_fullsized":"2023-11-24T08:06:04.016510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012659+0000","last_change":"2023-11-24T08:06:04.017101+0000","last_active":"2023-11-24T08:06:05.012659+0000","last_peered":"2023-11-24T08:06:05.012659+0000","last_clean":"2023-11-24T08:06:05.012659+0000","last_became_active":"2023-11-24T08:06:04.014260+0000","last_became_peered":"2023-11-24T08:06:04.014260+0000","last_unstale":"2023-11-24T08:06:05.012659+0000","last_undegraded":"2023-11-24T08:06:05.012659+0000","last_fullsized":"2023-11-24T08:06:05.012659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.012080+0000","last_change":"2023-11-24T08:06:03.004832+0000","last_active":"2023-11-24T08:06:04.012080+0000","last_peered":"2023-11-24T08:06:04.012080+0000","last_clean":"2023-11-24T08:06:04.012080+0000","last_became_active":"2023-11-24T08:06:03.004536+0000","last_became_peered":"2023-11-24T08:06:03.004536+0000","last_unstale":"2023-11-24T08:06:04.012080+0000","last_undegraded":"2023-11-24T08:06:04.012080+0000","last_fullsized":"2023-11-24T08:06:04.012080+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061318+0000","last_change":"2023-11-24T08:06:04.008187+0000","last_active":"2023-11-24T08:06:05.061318+0000","last_peered":"2023-11-24T08:06:05.061318+0000","last_clean":"2023-11-24T08:06:05.061318+0000","last_became_active":"2023-11-24T08:06:04.007852+0000","last_became_peered":"2023-11-24T08:06:04.007852+0000","last_unstale":"2023-11-24T08:06:05.061318+0000","last_undegraded":"2023-11-24T08:06:05.061318+0000","last_fullsized":"2023-11-24T08:06:05.061318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:27:55.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.011417+0000","last_change":"2023-11-24T08:06:02.993001+0000","last_active":"2023-11-24T08:06:04.011417+0000","last_peered":"2023-11-24T08:06:04.011417+0000","last_clean":"2023-11-24T08:06:04.011417+0000","last_became_active":"2023-11-24T08:06:02.992519+0000","last_became_peered":"2023-11-24T08:06:02.992519+0000","last_unstale":"2023-11-24T08:06:04.011417+0000","last_undegraded":"2023-11-24T08:06:04.011417+0000","last_fullsized":"2023-11-24T08:06:04.011417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061543+0000","last_change":"2023-11-24T08:06:04.008996+0000","last_active":"2023-11-24T08:06:05.061543+0000","last_peered":"2023-11-24T08:06:05.061543+0000","last_clean":"2023-11-24T08:06:05.061543+0000","last_became_active":"2023-11-24T08:06:04.008533+0000","last_became_peered":"2023-11-24T08:06:04.008533+0000","last_unstale":"2023-11-24T08:06:05.061543+0000","last_undegraded":"2023-11-24T08:06:05.061543+0000","last_fullsized":"2023-11-24T08:06:05.061543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.990855+0000","last_change":"2023-11-24T08:06:02.992653+0000","last_active":"2023-11-24T08:06:03.990855+0000","last_peered":"2023-11-24T08:06:03.990855+0000","last_clean":"2023-11-24T08:06:03.990855+0000","last_became_active":"2023-11-24T08:06:02.992420+0000","last_became_peered":"2023-11-24T08:06:02.992420+0000","last_unstale":"2023-11-24T08:06:03.990855+0000","last_undegraded":"2023-11-24T08:06:03.990855+0000","last_fullsized":"2023-11-24T08:06:03.990855+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:42.298254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060991+0000","last_change":"2023-11-24T08:06:04.014483+0000","last_active":"2023-11-24T08:06:05.060991+0000","last_peered":"2023-11-24T08:06:05.060991+0000","last_clean":"2023-11-24T08:06:05.060991+0000","last_became_active":"2023-11-24T08:06:04.014257+0000","last_became_peered":"2023-11-24T08:06:04.014257+0000","last_unstale":"2023-11-24T08:06:05.060991+0000","last_undegraded":"2023-11-24T08:06:05.060991+0000","last_fullsized":"2023-11-24T08:06:05.060991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:56.043697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.984304+0000","last_change":"2023-11-24T08:06:02.998354+0000","last_active":"2023-11-24T08:06:03.984304+0000","last_peered":"2023-11-24T08:06:03.984304+0000","last_clean":"2023-11-24T08:06:03.984304+0000","last_became_active":"2023-11-24T08:06:02.998065+0000","last_became_peered":"2023-11-24T08:06:02.998065+0000","last_unstale":"2023-11-24T08:06:03.984304+0000","last_undegraded":"2023-11-24T08:06:03.984304+0000","last_fullsized":"2023-11-24T08:06:03.984304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:57.508607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063429+0000","last_change":"2023-11-24T08:06:04.012119+0000","last_active":"2023-11-24T08:06:05.063429+0000","last_peered":"2023-11-24T08:06:05.063429+0000","last_clean":"2023-11-24T08:06:05.063429+0000","last_became_active":"2023-11-24T08:06:04.011694+0000","last_became_peered":"2023-11-24T08:06:04.011694+0000","last_unstale":"2023-11-24T08:06:05.063429+0000","last_undegraded":"2023-11-24T08:06:05.063429+0000","last_fullsized":"2023-11-24T08:06:05.063429+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:10.121048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994511+0000","last_change":"2023-11-24T08:06:03.010255+0000","last_active":"2023-11-24T08:06:03.994511+0000","last_peered":"2023-11-24T08:06:03.994511+0000","last_clean":"2023-11-24T08:06:03.994511+0000","last_became_active":"2023-11-24T08:06:03.009943+0000","last_became_peered":"2023-11-24T08:06:03.009943+0000","last_unstale":"2023-11-24T08:06:03.994511+0000","last_undegraded":"2023-11-24T08:06:03.994511+0000","last_fullsized":"2023-11-24T08:06:03.994511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:54.942656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060678+0000","last_change":"2023-11-24T08:06:04.011415+0000","last_active":"2023-11-24T08:06:05.060678+0000","last_peered":"2023-11-24T08:06:05.060678+0000","last_clean":"2023-11-24T08:06:05.060678+0000","last_became_active":"2023-11-24T08:06:04.011126+0000","last_became_peered":"2023-11-24T08:06:04.011126+0000","last_unstale":"2023-11-24T08:06:05.060678+0000","last_undegraded":"2023-11-24T08:06:05.060678+0000","last_fullsized":"2023-11-24T08:06:05.060678+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:34:53.211888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.980344+0000","last_change":"2023-11-24T08:06:03.004192+0000","last_active":"2023-11-24T08:06:03.980344+0000","last_peered":"2023-11-24T08:06:03.980344+0000","last_clean":"2023-11-24T08:06:03.980344+0000","last_became_active":"2023-11-24T08:06:03.003913+0000","last_became_peered":"2023-11-24T08:06:03.003913+0000","last_unstale":"2023-11-24T08:06:03.980344+0000","last_undegraded":"2023-11-24T08:06:03.980344+0000","last_fullsized":"2023-11-24T08:06:03.980344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:42.810732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061376+0000","last_change":"2023-11-24T08:06:04.010430+0000","last_active":"2023-11-24T08:06:05.061376+0000","last_peered":"2023-11-24T08:06:05.061376+0000","last_clean":"2023-11-24T08:06:05.061376+0000","last_became_active":"2023-11-24T08:06:04.010156+0000","last_became_peered":"2023-11-24T08:06:04.010156+0000","last_unstale":"2023-11-24T08:06:05.061376+0000","last_undegraded":"2023-11-24T08:06:05.061376+0000","last_fullsized":"2023-11-24T08:06:05.061376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:06:38.863342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.019405+0000","last_change":"2023-11-24T08:06:03.009734+0000","last_active":"2023-11-24T08:06:04.019405+0000","last_peered":"2023-11-24T08:06:04.019405+0000","last_clean":"2023-11-24T08:06:04.019405+0000","last_became_active":"2023-11-24T08:06:03.009416+0000","last_became_peered":"2023-11-24T08:06:03.009416+0000","last_unstale":"2023-11-24T08:06:04.019405+0000","last_undegraded":"2023-11-24T08:06:04.019405+0000","last_fullsized":"2023-11-24T08:06:04.019405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:41:48.167314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061927+0000","last_change":"2023-11-24T08:06:04.024388+0000","last_active":"2023-11-24T08:06:05.061927+0000","last_peered":"2023-11-24T08:06:05.061927+0000","last_clean":"2023-11-24T08:06:05.061927+0000","last_became_active":"2023-11-24T08:06:04.024132+0000","last_became_peered":"2023-11-24T08:06:04.024132+0000","last_unstale":"2023-11-24T08:06:05.061927+0000","last_undegraded":"2023-11-24T08:06:05.061927+0000","last_fullsized":"2023-11-24T08:06:05.061927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:38.767269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.031089+0000","last_change":"2023-11-24T08:06:02.998808+0000","last_active":"2023-11-24T08:06:04.031089+0000","last_peered":"2023-11-24T08:06:04.031089+0000","last_clean":"2023-11-24T08:06:04.031089+0000","last_became_active":"2023-11-24T08:06:02.998600+0000","last_became_peered":"2023-11-24T08:06:02.998600+0000","last_unstale":"2023-11-24T08:06:04.031089+0000","last_undegraded":"2023-11-24T08:06:04.031089+0000","last_fullsized":"2023-11-24T08:06:04.031089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:07.885075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060991+0000","last_change":"2023-11-24T08:06:04.014905+0000","last_active":"2023-11-24T08:06:05.060991+0000","last_peered":"2023-11-24T08:06:05.060991+0000","last_clean":"2023-11-24T08:06:05.060991+0000","last_became_active":"2023-11-24T08:06:04.014753+0000","last_became_peered":"2023-11-24T08:06:04.014753+0000","last_unstale":"2023-11-24T08:06:05.060991+0000","last_undegraded":"2023-11-24T08:06:05.060991+0000","last_fullsized":"2023-11-24T08:06:05.060991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:34.310112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.016651+0000","last_change":"2023-11-24T08:06:03.004639+0000","last_active":"2023-11-24T08:06:04.016651+0000","last_peered":"2023-11-24T08:06:04.016651+0000","last_clean":"2023-11-24T08:06:04.016651+0000","last_became_active":"2023-11-24T08:06:03.004341+0000","last_became_peered":"2023-11-24T08:06:03.004341+0000","last_unstale":"2023-11-24T08:06:04.016651+0000","last_undegraded":"2023-11-24T08:06:04.016651+0000","last_fullsized":"2023-11-24T08:06:04.016651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.060166+0000","last_change":"2023-11-24T08:06:04.009125+0000","last_active":"2023-11-24T08:06:05.060166+0000","last_peered":"2023-11-24T08:06:05.060166+0000","last_clean":"2023-11-24T08:06:05.060166+0000","last_became_active":"2023-11-24T08:06:04.008706+0000","last_became_peered":"2023-11-24T08:06:04.008706+0000","last_unstale":"2023-11-24T08:06:05.060166+0000","last_undegraded":"2023-11-24T08:06:05.060166+0000","last_fullsized":"2023-11-24T08:06:05.060166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:38.225050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.989596+0000","last_change":"2023-11-24T08:06:03.005631+0000","last_active":"2023-11-24T08:06:03.989596+0000","last_peered":"2023-11-24T08:06:03.989596+0000","last_clean":"2023-11-24T08:06:03.989596+0000","last_became_active":"2023-11-24T08:06:03.004131+0000","last_became_peered":"2023-11-24T08:06:03.004131+0000","last_unstale":"2023-11-24T08:06:03.989596+0000","last_undegraded":"2023-11-24T08:06:03.989596+0000","last_fullsized":"2023-11-24T08:06:03.989596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:44.830937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010149+0000","last_change":"2023-11-24T08:06:04.015328+0000","last_active":"2023-11-24T08:06:05.010149+0000","last_peered":"2023-11-24T08:06:05.010149+0000","last_clean":"2023-11-24T08:06:05.010149+0000","last_became_active":"2023-11-24T08:06:04.014191+0000","last_became_peered":"2023-11-24T08:06:04.014191+0000","last_unstale":"2023-11-24T08:06:05.010149+0000","last_undegraded":"2023-11-24T08:06:05.010149+0000","last_fullsized":"2023-11-24T08:06:05.010149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:08.896877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.990756+0000","last_change":"2023-11-24T08:06:03.009985+0000","last_active":"2023-11-24T08:06:03.990756+0000","last_peered":"2023-11-24T08:06:03.990756+0000","last_clean":"2023-11-24T08:06:03.990756+0000","last_became_active":"2023-11-24T08:06:03.009676+0000","last_became_peered":"2023-11-24T08:06:03.009676+0000","last_unstale":"2023-11-24T08:06:03.990756+0000","last_undegraded":"2023-11-24T08:06:03.990756+0000","last_fullsized":"2023-11-24T08:06:03.990756+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:17.437596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.012188+0000","last_change":"2023-11-24T08:06:04.017139+0000","last_active":"2023-11-24T08:06:05.012188+0000","last_peered":"2023-11-24T08:06:05.012188+0000","last_clean":"2023-11-24T08:06:05.012188+0000","last_became_active":"2023-11-24T08:06:04.016481+0000","last_became_peered":"2023-11-24T08:06:04.016481+0000","last_unstale":"2023-11-24T08:06:05.012188+0000","last_undegraded":"2023-11-24T08:06:05.012188+0000","last_fullsized":"2023-11-24T08:06:05.012188+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:58.109176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.026025+0000","last_change":"2023-11-24T08:06:02.996472+0000","last_active":"2023-11-24T08:06:04.026025+0000","last_peered":"2023-11-24T08:06:04.026025+0000","last_clean":"2023-11-24T08:06:04.026025+0000","last_became_active":"2023-11-24T08:06:02.996105+0000","last_became_peered":"2023-11-24T08:06:02.996105+0000","last_unstale":"2023-11-24T08:06:04.026025+0000","last_undegraded":"2023-11-24T08:06:04.026025+0000","last_fullsized":"2023-11-24T08:06:04.026025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:57.097374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.063264+0000","last_change":"2023-11-24T08:06:04.011512+0000","last_active":"2023-11-24T08:06:05.063264+0000","last_peered":"2023-11-24T08:06:05.063264+0000","last_clean":"2023-11-24T08:06:05.063264+0000","last_became_active":"2023-11-24T08:06:04.011208+0000","last_became_peered":"2023-11-24T08:06:04.011208+0000","last_unstale":"2023-11-24T08:06:05.063264+0000","last_undegraded":"2023-11-24T08:06:05.063264+0000","last_fullsized":"2023-11-24T08:06:05.063264+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:56.043697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.023777+0000","last_change":"2023-11-24T08:06:03.006947+0000","last_active":"2023-11-24T08:06:04.023777+0000","last_peered":"2023-11-24T08:06:04.023777+0000","last_clean":"2023-11-24T08:06:04.023777+0000","last_became_active":"2023-11-24T08:06:03.006640+0000","last_became_peered":"2023-11-24T08:06:03.006640+0000","last_unstale":"2023-11-24T08:06:04.023777+0000","last_undegraded":"2023-11-24T08:06:04.023777+0000","last_fullsized":"2023-11-24T08:06:04.023777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062187+0000","last_change":"2023-11-24T08:06:04.015848+0000","last_active":"2023-11-24T08:06:05.062187+0000","last_peered":"2023-11-24T08:06:05.062187+0000","last_clean":"2023-11-24T08:06:05.062187+0000","last_became_active":"2023-11-24T08:06:04.015574+0000","last_became_peered":"2023-11-24T08:06:04.015574+0000","last_unstale":"2023-11-24T08:06:05.062187+0000","last_undegraded":"2023-11-24T08:06:05.062187+0000","last_fullsized":"2023-11-24T08:06:05.062187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:55.954459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.017612+0000","last_change":"2023-11-24T08:06:03.003338+0000","last_active":"2023-11-24T08:06:04.017612+0000","last_peered":"2023-11-24T08:06:04.017612+0000","last_clean":"2023-11-24T08:06:04.017612+0000","last_became_active":"2023-11-24T08:06:02.998184+0000","last_became_peered":"2023-11-24T08:06:02.998184+0000","last_unstale":"2023-11-24T08:06:04.017612+0000","last_undegraded":"2023-11-24T08:06:04.017612+0000","last_fullsized":"2023-11-24T08:06:04.017612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:28.278148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061039+0000","last_change":"2023-11-24T08:06:04.020991+0000","last_active":"2023-11-24T08:06:05.061039+0000","last_peered":"2023-11-24T08:06:05.061039+0000","last_clean":"2023-11-24T08:06:05.061039+0000","last_became_active":"2023-11-24T08:06:04.020725+0000","last_became_peered":"2023-11-24T08:06:04.020725+0000","last_unstale":"2023-11-24T08:06:05.061039+0000","last_undegraded":"2023-11-24T08:06:05.061039+0000","last_fullsized":"2023-11-24T08:06:05.061039+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:38.322203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.021082+0000","last_change":"2023-11-24T08:06:03.001694+0000","last_active":"2023-11-24T08:06:04.021082+0000","last_peered":"2023-11-24T08:06:04.021082+0000","last_clean":"2023-11-24T08:06:04.021082+0000","last_became_active":"2023-11-24T08:06:02.998616+0000","last_became_peered":"2023-11-24T08:06:02.998616+0000","last_unstale":"2023-11-24T08:06:04.021082+0000","last_undegraded":"2023-11-24T08:06:04.021082+0000","last_fullsized":"2023-11-24T08:06:04.021082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:14.099330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.061929+0000","last_change":"2023-11-24T08:06:04.005239+0000","last_active":"2023-11-24T08:06:05.061929+0000","last_peered":"2023-11-24T08:06:05.061929+0000","last_clean":"2023-11-24T08:06:05.061929+0000","last_became_active":"2023-11-24T08:06:04.004973+0000","last_became_peered":"2023-11-24T08:06:04.004973+0000","last_unstale":"2023-11-24T08:06:05.061929+0000","last_undegraded":"2023-11-24T08:06:05.061929+0000","last_fullsized":"2023-11-24T08:06:05.061929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:36.128015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.985791+0000","last_change":"2023-11-24T08:06:02.993516+0000","last_active":"2023-11-24T08:06:03.985791+0000","last_peered":"2023-11-24T08:06:03.985791+0000","last_clean":"2023-11-24T08:06:03.985791+0000","last_became_active":"2023-11-24T08:06:02.992976+0000","last_became_peered":"2023-11-24T08:06:02.992976+0000","last_unstale":"2023-11-24T08:06:03.985791+0000","last_undegraded":"2023-11-24T08:06:03.985791+0000","last_fullsized":"2023-11-24T08:06:03.985791+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059342+0000","last_change":"2023-11-24T08:06:04.012243+0000","last_active":"2023-11-24T08:06:05.059342+0000","last_peered":"2023-11-24T08:06:05.059342+0000","last_clean":"2023-11-24T08:06:05.059342+0000","last_became_active":"2023-11-24T08:06:04.011797+0000","last_became_peered":"2023-11-24T08:06:04.011797+0000","last_unstale":"2023-11-24T08:06:05.059342+0000","last_undegraded":"2023-11-24T08:06:05.059342+0000","last_fullsized":"2023-11-24T08:06:05.059342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:23.489751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.024403+0000","last_change":"2023-11-24T08:06:02.998617+0000","last_active":"2023-11-24T08:06:04.024403+0000","last_peered":"2023-11-24T08:06:04.024403+0000","last_clean":"2023-11-24T08:06:04.024403+0000","last_became_active":"2023-11-24T08:06:02.998393+0000","last_became_peered":"2023-11-24T08:06:02.998393+0000","last_unstale":"2023-11-24T08:06:04.024403+0000","last_undegraded":"2023-11-24T08:06:04.024403+0000","last_fullsized":"2023-11-24T08:06:04.024403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:28.474060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059870+0000","last_change":"2023-11-24T08:06:03.997412+0000","last_active":"2023-11-24T08:06:05.059870+0000","last_peered":"2023-11-24T08:06:05.059870+0000","last_clean":"2023-11-24T08:06:05.059870+0000","last_became_active":"2023-11-24T08:06:03.996455+0000","last_became_peered":"2023-11-24T08:06:03.996455+0000","last_unstale":"2023-11-24T08:06:05.059870+0000","last_undegraded":"2023-11-24T08:06:05.059870+0000","last_fullsized":"2023-11-24T08:06:05.059870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:58.520409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994818+0000","last_change":"2023-11-24T08:06:03.005058+0000","last_active":"2023-11-24T08:06:03.994818+0000","last_peered":"2023-11-24T08:06:03.994818+0000","last_clean":"2023-11-24T08:06:03.994818+0000","last_became_active":"2023-11-24T08:06:03.004703+0000","last_became_peered":"2023-11-24T08:06:03.004703+0000","last_unstale":"2023-11-24T08:06:03.994818+0000","last_undegraded":"2023-11-24T08:06:03.994818+0000","last_fullsized":"2023-11-24T08:06:03.994818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:35.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059520+0000","last_change":"2023-11-24T08:06:03.994897+0000","last_active":"2023-11-24T08:06:05.059520+0000","last_peered":"2023-11-24T08:06:05.059520+0000","last_clean":"2023-11-24T08:06:05.059520+0000","last_became_active":"2023-11-24T08:06:03.994621+0000","last_became_peered":"2023-11-24T08:06:03.994621+0000","last_unstale":"2023-11-24T08:06:05.059520+0000","last_undegraded":"2023-11-24T08:06:05.059520+0000","last_fullsized":"2023-11-24T08:06:05.059520+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:29.485863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.982383+0000","last_change":"2023-11-24T08:06:03.003128+0000","last_active":"2023-11-24T08:06:03.982383+0000","last_peered":"2023-11-24T08:06:03.982383+0000","last_clean":"2023-11-24T08:06:03.982383+0000","last_became_active":"2023-11-24T08:06:03.002791+0000","last_became_peered":"2023-11-24T08:06:03.002791+0000","last_unstale":"2023-11-24T08:06:03.982383+0000","last_undegraded":"2023-11-24T08:06:03.982383+0000","last_fullsized":"2023-11-24T08:06:03.982383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:42.628747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.010148+0000","last_change":"2023-11-24T08:06:04.017398+0000","last_active":"2023-11-24T08:06:05.010148+0000","last_peered":"2023-11-24T08:06:05.010148+0000","last_clean":"2023-11-24T08:06:05.010148+0000","last_became_active":"2023-11-24T08:06:04.017095+0000","last_became_peered":"2023-11-24T08:06:04.017095+0000","last_unstale":"2023-11-24T08:06:05.010148+0000","last_undegraded":"2023-11-24T08:06:05.010148+0000","last_fullsized":"2023-11-24T08:06:05.010148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:53.581888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.005909+0000","last_change":"2023-11-24T08:06:03.006761+0000","last_active":"2023-11-24T08:06:04.005909+0000","last_peered":"2023-11-24T08:06:04.005909+0000","last_clean":"2023-11-24T08:06:04.005909+0000","last_became_active":"2023-11-24T08:06:03.006418+0000","last_became_peered":"2023-11-24T08:06:03.006418+0000","last_unstale":"2023-11-24T08:06:04.005909+0000","last_undegraded":"2023-11-24T08:06:04.005909+0000","last_fullsized":"2023-11-24T08:06:04.005909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:42.298254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062156+0000","last_change":"2023-11-24T08:06:04.012846+0000","last_active":"2023-11-24T08:06:05.062156+0000","last_peered":"2023-11-24T08:06:05.062156+0000","last_clean":"2023-11-24T08:06:05.062156+0000","last_became_active":"2023-11-24T08:06:04.012509+0000","last_became_peered":"2023-11-24T08:06:04.012509+0000","last_unstale":"2023-11-24T08:06:05.062156+0000","last_undegraded":"2023-11-24T08:06:05.062156+0000","last_fullsized":"2023-11-24T08:06:05.062156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:43.822534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:04.021487+0000","last_change":"2023-11-24T08:06:03.008262+0000","last_active":"2023-11-24T08:06:04.021487+0000","last_peered":"2023-11-24T08:06:04.021487+0000","last_clean":"2023-11-24T08:06:04.021487+0000","last_became_active":"2023-11-24T08:06:03.007982+0000","last_became_peered":"2023-11-24T08:06:03.007982+0000","last_unstale":"2023-11-24T08:06:04.021487+0000","last_undegraded":"2023-11-24T08:06:04.021487+0000","last_fullsized":"2023-11-24T08:06:04.021487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062556+0000","last_change":"2023-11-24T08:06:04.010748+0000","last_active":"2023-11-24T08:06:05.062556+0000","last_peered":"2023-11-24T08:06:05.062556+0000","last_clean":"2023-11-24T08:06:05.062556+0000","last_became_active":"2023-11-24T08:06:04.010351+0000","last_became_peered":"2023-11-24T08:06:04.010351+0000","last_unstale":"2023-11-24T08:06:05.062556+0000","last_undegraded":"2023-11-24T08:06:05.062556+0000","last_fullsized":"2023-11-24T08:06:05.062556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:15.601354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.980472+0000","last_change":"2023-11-24T08:06:03.001269+0000","last_active":"2023-11-24T08:06:03.980472+0000","last_peered":"2023-11-24T08:06:03.980472+0000","last_clean":"2023-11-24T08:06:03.980472+0000","last_became_active":"2023-11-24T08:06:03.000568+0000","last_became_peered":"2023-11-24T08:06:03.000568+0000","last_unstale":"2023-11-24T08:06:03.980472+0000","last_undegraded":"2023-11-24T08:06:03.980472+0000","last_fullsized":"2023-11-24T08:06:03.980472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:43:42.376592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026681+0000","last_change":"2023-11-24T08:06:04.008832+0000","last_active":"2023-11-24T08:06:05.026681+0000","last_peered":"2023-11-24T08:06:05.026681+0000","last_clean":"2023-11-24T08:06:05.026681+0000","last_became_active":"2023-11-24T08:06:04.008583+0000","last_became_peered":"2023-11-24T08:06:04.008583+0000","last_unstale":"2023-11-24T08:06:05.026681+0000","last_undegraded":"2023-11-24T08:06:05.026681+0000","last_fullsized":"2023-11-24T08:06:05.026681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.994141+0000","last_change":"2023-11-24T08:06:02.995680+0000","last_active":"2023-11-24T08:06:03.994141+0000","last_peered":"2023-11-24T08:06:03.994141+0000","last_clean":"2023-11-24T08:06:03.994141+0000","last_became_active":"2023-11-24T08:06:02.995421+0000","last_became_peered":"2023-11-24T08:06:02.995421+0000","last_unstale":"2023-11-24T08:06:03.994141+0000","last_undegraded":"2023-11-24T08:06:03.994141+0000","last_fullsized":"2023-11-24T08:06:03.994141+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:50.084260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011707+0000","last_change":"2023-11-24T08:06:04.003958+0000","last_active":"2023-11-24T08:06:05.011707+0000","last_peered":"2023-11-24T08:06:05.011707+0000","last_clean":"2023-11-24T08:06:05.011707+0000","last_became_active":"2023-11-24T08:06:04.003478+0000","last_became_peered":"2023-11-24T08:06:04.003478+0000","last_unstale":"2023-11-24T08:06:05.011707+0000","last_undegraded":"2023-11-24T08:06:05.011707+0000","last_fullsized":"2023-11-24T08:06:05.011707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:59.759128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.979841+0000","last_change":"2023-11-24T08:06:03.000694+0000","last_active":"2023-11-24T08:06:03.979841+0000","last_peered":"2023-11-24T08:06:03.979841+0000","last_clean":"2023-11-24T08:06:03.979841+0000","last_became_active":"2023-11-24T08:06:03.000285+0000","last_became_peered":"2023-11-24T08:06:03.000285+0000","last_unstale":"2023-11-24T08:06:03.979841+0000","last_undegraded":"2023-11-24T08:06:03.979841+0000","last_fullsized":"2023-11-24T08:06:03.979841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:58:57.508607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.026651+0000","last_change":"2023-11-24T08:06:04.004758+0000","last_active":"2023-11-24T08:06:05.026651+0000","last_peered":"2023-11-24T08:06:05.026651+0000","last_clean":"2023-11-24T08:06:05.026651+0000","last_became_active":"2023-11-24T08:06:04.004484+0000","last_became_peered":"2023-11-24T08:06:04.004484+0000","last_unstale":"2023-11-24T08:06:05.026651+0000","last_undegraded":"2023-11-24T08:06:05.026651+0000","last_fullsized":"2023-11-24T08:06:05.026651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:32.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.989723+0000","last_change":"2023-11-24T08:06:03.008992+0000","last_active":"2023-11-24T08:06:03.989723+0000","last_peered":"2023-11-24T08:06:03.989723+0000","last_clean":"2023-11-24T08:06:03.989723+0000","last_became_active":"2023-11-24T08:06:03.008648+0000","last_became_peered":"2023-11-24T08:06:03.008648+0000","last_unstale":"2023-11-24T08:06:03.989723+0000","last_undegraded":"2023-11-24T08:06:03.989723+0000","last_fullsized":"2023-11-24T08:06:03.989723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:58.747325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.059005+0000","last_change":"2023-11-24T08:06:04.011249+0000","last_active":"2023-11-24T08:06:05.059005+0000","last_peered":"2023-11-24T08:06:05.059005+0000","last_clean":"2023-11-24T08:06:05.059005+0000","last_became_active":"2023-11-24T08:06:04.009623+0000","last_became_peered":"2023-11-24T08:06:04.009623+0000","last_unstale":"2023-11-24T08:06:05.059005+0000","last_undegraded":"2023-11-24T08:06:05.059005+0000","last_fullsized":"2023-11-24T08:06:05.059005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.982606+0000","last_change":"2023-11-24T08:06:03.003843+0000","last_active":"2023-11-24T08:06:03.982606+0000","last_peered":"2023-11-24T08:06:03.982606+0000","last_clean":"2023-11-24T08:06:03.982606+0000","last_became_active":"2023-11-24T08:06:03.002741+0000","last_became_peered":"2023-11-24T08:06:03.002741+0000","last_unstale":"2023-11-24T08:06:03.982606+0000","last_undegraded":"2023-11-24T08:06:03.982606+0000","last_fullsized":"2023-11-24T08:06:03.982606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:34.866245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.062696+0000","last_change":"2023-11-24T08:06:04.001276+0000","last_active":"2023-11-24T08:06:05.062696+0000","last_peered":"2023-11-24T08:06:05.062696+0000","last_clean":"2023-11-24T08:06:05.062696+0000","last_became_active":"2023-11-24T08:06:04.000935+0000","last_became_peered":"2023-11-24T08:06:04.000935+0000","last_unstale":"2023-11-24T08:06:05.062696+0000","last_undegraded":"2023-11-24T08:06:05.062696+0000","last_fullsized":"2023-11-24T08:06:05.062696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:34.310112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.988808+0000","last_change":"2023-11-24T08:06:03.009978+0000","last_active":"2023-11-24T08:06:03.988808+0000","last_peered":"2023-11-24T08:06:03.988808+0000","last_clean":"2023-11-24T08:06:03.988808+0000","last_became_active":"2023-11-24T08:06:03.009694+0000","last_became_peered":"2023-11-24T08:06:03.009694+0000","last_unstale":"2023-11-24T08:06:03.988808+0000","last_undegraded":"2023-11-24T08:06:03.988808+0000","last_fullsized":"2023-11-24T08:06:03.988808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:33.298310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T08:06:05.011528+0000","last_change":"2023-11-24T08:06:04.006028+0000","last_active":"2023-11-24T08:06:05.011528+0000","last_peered":"2023-11-24T08:06:05.011528+0000","last_clean":"2023-11-24T08:06:05.011528+0000","last_became_active":"2023-11-24T08:06:04.004815+0000","last_became_peered":"2023-11-24T08:06:04.004815+0000","last_unstale":"2023-11-24T08:06:05.011528+0000","last_undegraded":"2023-11-24T08:06:05.011528+0000","last_fullsized":"2023-11-24T08:06:05.011528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:02.953055+0000","last_clean_scrub_stamp":"2023-11-24T08:06:02.953055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:54:49.584352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.999191+0000","last_change":"2023-11-24T08:06:03.008297+0000","last_active":"2023-11-24T08:06:03.999191+0000","last_peered":"2023-11-24T08:06:03.999191+0000","last_clean":"2023-11-24T08:06:03.999191+0000","last_became_active":"2023-11-24T08:06:03.007985+0000","last_became_peered":"2023-11-24T08:06:03.007985+0000","last_unstale":"2023-11-24T08:06:03.999191+0000","last_undegraded":"2023-11-24T08:06:03.999191+0000","last_fullsized":"2023-11-24T08:06:03.999191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:06:01.941252+0000","last_clean_scrub_stamp":"2023-11-24T08:06:01.941252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:54:29.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T08:06:03.997715+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:06:03.997715+0000","last_peered":"2023-11-24T08:06:03.997715+0000","last_clean":"2023-11-24T08:06:03.997715+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:06:03.997715+0000","last_undegraded":"2023-11-24T08:06:03.997715+0000","last_fullsized":"2023-11-24T08:06:03.997715+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":742357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.034},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.373}]},{"osd":2,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.099},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":1.776}]},{"osd":3,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.8640000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":2.206}]},{"osd":4,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.978},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":2.0600000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":2.4100000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":2.0990000000000002}]},{"osd":6,"last update":"Fri Nov 24 08:05:42 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":2.149}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.393}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.423},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.95299999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.349}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.3080000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.268}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.629,"5min":2.629,"15min":2.629},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.226}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.468},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.45}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.081}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.141},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.3720000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.878},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.218}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.018},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":1.3320000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.401}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.262}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4239999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.5369999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.0800000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.337}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.139},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3879999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.5899999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":1.47}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.605},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.29}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":153533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.0069999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.1850000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2629999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.109}]},{"osd":5,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.49},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.226}]},{"osd":6,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.54}]},{"osd":7,"last update":"Fri Nov 24 08:05:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.639}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.042}]},{"osd":1,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.748},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.93600000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.131}]},{"osd":4,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.298},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":1.6180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.464},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.794}]},{"osd":6,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.835},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.526}]},{"osd":7,"last update":"Fri Nov 24 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.597,"15min":1.597},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.742},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3809999999999998,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.76}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":153533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.99199999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.879}]},{"osd":4,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.1930000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.556},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":1.4730000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.9359999999999999,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.2150000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.28},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.3720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":742267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.3440000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.73199999999999998},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.629}]},{"osd":3,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.98699999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.411},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.1100000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.306},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.2370000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.365},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.2130000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:05:44 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.4570000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:06:13.857 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:06:13.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:06:14.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.119+0000 7fe356a02640 1 Processor -- start 2023-11-24T08:06:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.120+0000 7fe356a02640 1 -- start start 2023-11-24T08:06:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe35015f530 0x7fe35015f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 0x7fe3500cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 0x7fe350140b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3500cf7b0 con 0x7fe35015f530 2023-11-24T08:06:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35013eb00 con 0x7fe35006ff70 2023-11-24T08:06:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe356a02640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35013ede0 con 0x7fe3500cf320 2023-11-24T08:06:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.121+0000 7fe34ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe35015f530 0x7fe35015f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe35015f530 0x7fe35015f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43358/0 (socket says 172.21.15.107:43358) 2023-11-24T08:06:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 -- 172.21.15.107:0/1514110984 learned_addr learned my addr 172.21.15.107:0/1514110984 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe354f9f640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 0x7fe350140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe347fff640 1 --2- 172.21.15.107:0/1514110984 >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 0x7fe3500cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 -- 172.21.15.107:0/1514110984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 msgr2=0x7fe350140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 0x7fe350140b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 -- 172.21.15.107:0/1514110984 >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 msgr2=0x7fe3500cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 --2- 172.21.15.107:0/1514110984 >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 0x7fe3500cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe34ffff640 1 -- 172.21.15.107:0/1514110984 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35015fe50 con 0x7fe35015f530 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe354f9f640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 0x7fe350140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.122+0000 7fe347fff640 1 --2- 172.21.15.107:0/1514110984 >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 0x7fe3500cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.123+0000 7fe34ffff640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35015f530 0x7fe35015f910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe35015ee60 tx=0x7fe34002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b2a12a864ce7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.123+0000 7fe34f7fe640 1 -- 172.21.15.107:0/1514110984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400317d0 con 0x7fe35015f530 2023-11-24T08:06:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe34f7fe640 1 -- 172.21.15.107:0/1514110984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe340031930 con 0x7fe35015f530 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe34f7fe640 1 -- 172.21.15.107:0/1514110984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340031cc0 con 0x7fe35015f530 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 -- 172.21.15.107:0/1514110984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35015f530 msgr2=0x7fe35015f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35015f530 0x7fe35015f910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe35015ee60 tx=0x7fe34002b870 comp rx=0 tx=0).stop 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 -- 172.21.15.107:0/1514110984 shutdown_connections 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3500cf320 0x7fe350140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 --2- 172.21.15.107:0/1514110984 >> v2:172.21.15.120:3300/0 conn(0x7fe35006ff70 0x7fe3500cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 --2- 172.21.15.107:0/1514110984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35015f530 0x7fe35015f910 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 -- 172.21.15.107:0/1514110984 >> 172.21.15.107:0/1514110984 conn(0x7fe3500b2e40 msgr2=0x7fe3500b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 -- 172.21.15.107:0/1514110984 shutdown_connections 2023-11-24T08:06:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.124+0000 7fe356a02640 1 -- 172.21.15.107:0/1514110984 wait complete. 2023-11-24T08:06:14.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.126+0000 7fe356a02640 1 Processor -- start 2023-11-24T08:06:14.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.126+0000 7fe356a02640 1 -- start start 2023-11-24T08:06:14.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe35006ff70 0x7fe3501435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35015f530 0x7fe3501f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe350144890 con 0x7fe3500cf320 2023-11-24T08:06:14.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3501f3fd0 con 0x7fe35015f530 2023-11-24T08:06:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe356a02640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe350143c80 con 0x7fe35006ff70 2023-11-24T08:06:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.127+0000 7fe347fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43374/0 (socket says 172.21.15.107:43374) 2023-11-24T08:06:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 -- 172.21.15.107:0/3199306796 learned_addr learned my addr 172.21.15.107:0/3199306796 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe34ffff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe35006ff70 0x7fe3501435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe354f9f640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35015f530 0x7fe3501f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 -- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe35006ff70 msgr2=0x7fe3501435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe35006ff70 0x7fe3501435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 -- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35015f530 msgr2=0x7fe3501f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35015f530 0x7fe3501f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 -- 172.21.15.107:0/3199306796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400094d0 con 0x7fe3500cf320 2023-11-24T08:06:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.128+0000 7fe347fff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c820 tx=0x7fe33800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.129+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338007a30 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.129+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338007b90 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.129+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe350143f60 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.129+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800ce10 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.129+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe350144520 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.130+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe350067e20 con 0x7fe3500cf320 2023-11-24T08:06:14.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.131+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe3380113f0 con 0x7fe3500cf320 2023-11-24T08:06:14.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.132+0000 7fe34d7fa640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 0x7fe33006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.132+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3380897d0 con 0x7fe3500cf320 2023-11-24T08:06:14.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.132+0000 7fe34ffff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 0x7fe33006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.133+0000 7fe34ffff640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 0x7fe33006d300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe340036000 tx=0x7fe34003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.136+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe338054760 con 0x7fe3500cf320 2023-11-24T08:06:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.397+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe3500c67a0 con 0x7fe3500cf320 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.402+0000 7fe34d7fa640 1 -- 172.21.15.107:0/3199306796 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe338057d80 con 0x7fe3500cf320 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:06:14.404 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:06:14.405 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 msgr2=0x7fe33006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 0x7fe33006d300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe340036000 tx=0x7fe34003d040 comp rx=0 tx=0).stop 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 msgr2=0x7fe3501f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c820 tx=0x7fe33800cce0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 shutdown_connections 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33006ae40 0x7fe33006d300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35015f530 0x7fe3501f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3500cf320 0x7fe3501f05e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 --2- 172.21.15.107:0/3199306796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe35006ff70 0x7fe3501435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 >> 172.21.15.107:0/3199306796 conn(0x7fe3500b2e40 msgr2=0x7fe3500c69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:14.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 shutdown_connections 2023-11-24T08:06:14.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.405+0000 7fe356a02640 1 -- 172.21.15.107:0/3199306796 wait complete. 2023-11-24T08:06:14.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:06:14.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.628+0000 7f75bab33640 1 Processor -- start 2023-11-24T08:06:14.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.628+0000 7f75bab33640 1 -- start start 2023-11-24T08:06:14.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75bab33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75b415c7f0 0x7f75b406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75bab33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b40cece0 con 0x7f75b40cf450 2023-11-24T08:06:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75bab33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b406d040 con 0x7f75b40ce6c0 2023-11-24T08:06:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75bab33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b40ceaf0 con 0x7f75b415c7f0 2023-11-24T08:06:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75abfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75b40cf450 0x7f75b40cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58894/0 (socket says 172.21.15.107:58894) 2023-11-24T08:06:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.630+0000 7f75abfff640 1 -- 172.21.15.107:0/3305454605 learned_addr learned my addr 172.21.15.107:0/3305454605 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75b88cf640 1 --2- 172.21.15.107:0/3305454605 >> v2:172.21.15.120:3300/0 conn(0x7f75b415c7f0 0x7f75b406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161053229 0 0) 0x7f75b40cece0 con 0x7f75b40cf450 2023-11-24T08:06:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c003520 con 0x7f75b40cf450 2023-11-24T08:06:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532392049 0 0) 0x7f759c003520 con 0x7f75b40cf450 2023-11-24T08:06:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 >> v2:172.21.15.120:3300/0 conn(0x7f75b415c7f0 msgr2=0x7f75b406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 --2- 172.21.15.107:0/3305454605 >> v2:172.21.15.120:3300/0 conn(0x7f75b415c7f0 0x7f75b406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 >> v1:172.21.15.120:6789/0 conn(0x7f75b40ce6c0 legacy=0x7f75b40714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b406d320 con 0x7f75b40cf450 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f75a4001670 con 0x7f75b40cf450 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 >> v1:172.21.15.107:6789/0 conn(0x7f75b40cf450 legacy=0x7f75b40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 0x7f759c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.631+0000 7f75ab7fe640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75ab7fe640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00bff0 con 0x7f759c003970 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00c160 con 0x7f759c005180 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00c320 con 0x7f759c0047c0 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 0x7f759c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b90d0640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75abfff640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 -- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c0047c0 msgr2=0x7f759c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 -- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759c005180 msgr2=0x7f759c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b88cf640 1 -- 172.21.15.107:0/3305454605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c003520 con 0x7f759c003970 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b90d0640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75abfff640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:14.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.632+0000 7f75b90d0640 1 -- 172.21.15.107:0/3305454605 reap_dead start 2023-11-24T08:06:14.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75b88cf640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 0x7f759c003d90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009ac0 tx=0x7f759c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb4cebda60fc08b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b002c6b0 con 0x7f759c003970 2023-11-24T08:06:14.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b002c810 con 0x7f759c003970 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75ab7fe640 1 -- 172.21.15.107:0/3305454605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b002cba0 con 0x7f759c003970 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75bab33640 1 -- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 msgr2=0x7f759c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.634+0000 7f75bab33640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 0x7f759c003d90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009ac0 tx=0x7f759c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.635+0000 7f75bab33640 1 -- 172.21.15.107:0/3305454605 shutdown_connections 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.635+0000 7f75bab33640 1 --2- 172.21.15.107:0/3305454605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f759c003970 0x7f759c003d90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.635+0000 7f75bab33640 1 -- 172.21.15.107:0/3305454605 >> 172.21.15.107:0/3305454605 conn(0x7f75b406a920 msgr2=0x7f75b406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.635+0000 7f75bab33640 1 -- 172.21.15.107:0/3305454605 shutdown_connections 2023-11-24T08:06:14.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.635+0000 7f75bab33640 1 -- 172.21.15.107:0/3305454605 wait complete. 2023-11-24T08:06:14.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.636+0000 7f75bab33640 1 Processor -- start 2023-11-24T08:06:14.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.637+0000 7f75bab33640 1 -- start start 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.637+0000 7f75bab33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c003970 0x7f75b4142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.637+0000 7f75bab33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75b4155de0 0x7f75b4140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75bab33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75bab33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b41431d0 con 0x7f75b41f08f0 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75bab33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4143470 con 0x7f75b4155de0 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75bab33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4143750 con 0x7f759c003970 2023-11-24T08:06:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43394/0 (socket says 172.21.15.107:43394) 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 -- 172.21.15.107:0/1695781943 learned_addr learned my addr 172.21.15.107:0/1695781943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b88cf640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c003970 0x7f75b4142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75abfff640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75b4155de0 0x7f75b4140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 -- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c003970 msgr2=0x7f75b4142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c003970 0x7f75b4142c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 -- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75b4155de0 msgr2=0x7f75b4140c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75b4155de0 0x7f75b4140c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 -- 172.21.15.107:0/1695781943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4143a30 con 0x7f75b41f08f0 2023-11-24T08:06:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.638+0000 7f75b90d0640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f75b40b5270 tx=0x7f75ac00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.639+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac003d80 con 0x7f75b41f08f0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.639+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac005070 con 0x7f75b41f08f0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.639+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0053e0 con 0x7f75b41f08f0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.639+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b4157cf0 con 0x7f75b41f08f0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.639+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b4141140 con 0x7f75b41f08f0 2023-11-24T08:06:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.640+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b4143bc0 con 0x7f75b41f08f0 2023-11-24T08:06:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.641+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f75ac005540 con 0x7f75b41f08f0 2023-11-24T08:06:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.642+0000 7f75a97fa640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 0x7f75a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.642+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75ac088d30 con 0x7f75b41f08f0 2023-11-24T08:06:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.642+0000 7f75b88cf640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 0x7f75a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:14.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.643+0000 7f75b88cf640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 0x7f75a006d330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75b0036000 tx=0x7f75b0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:14.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.646+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75ac053e80 con 0x7f75b41f08f0 2023-11-24T08:06:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.874+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b4067470 con 0x7f75b41f08f0 2023-11-24T08:06:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.875+0000 7f75a97fa640 1 -- 172.21.15.107:0/1695781943 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f75ac018070 con 0x7f75b41f08f0 2023-11-24T08:06:14.877 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:06:14.877 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:06:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 14 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.878+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 msgr2=0x7f75a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.878+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 0x7f75a006d330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75b0036000 tx=0x7f75b0030000 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.878+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 msgr2=0x7f75b4154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.878+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f75b40b5270 tx=0x7f75ac00d9d0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 shutdown_connections 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75a006ae70 0x7f75a006d330 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75b41f08f0 0x7f75b4154180 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75b4155de0 0x7f75b4140c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 --2- 172.21.15.107:0/1695781943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f759c003970 0x7f75b4142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 >> 172.21.15.107:0/1695781943 conn(0x7f75b406a920 msgr2=0x7f75b40b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 shutdown_connections 2023-11-24T08:06:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:14.879+0000 7f75bab33640 1 -- 172.21.15.107:0/1695781943 wait complete. 2023-11-24T08:06:15.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:06:03.446918+0000', 'modified': '2023-11-24T08:06:06.504388+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5070': {'gid': 5070, 'name': 'c', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.107:6837/4081761849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 4081761849}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 4081761849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:06:15.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:06:15.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:06:15.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.526+0000 7fdeb759e640 1 Processor -- start 2023-11-24T08:06:15.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.527+0000 7fdeb759e640 1 -- start start 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdeb8070c80 0x7fdeb806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb815b750 con 0x7fdeb815b370 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb80712d0 con 0x7fdeb806aa40 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb80715b0 con 0x7fdeb8070c80 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdeb815b370 0x7fdeb80aed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58896/0 (socket says 172.21.15.107:58896) 2023-11-24T08:06:15.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.528+0000 7fdeb659c640 1 -- 172.21.15.107:0/3227999761 learned_addr learned my addr 172.21.15.107:0/3227999761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3227999761 >> v2:172.21.15.107:3300/0 conn(0x7fdeb8070c80 0x7fdeb806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb559a640 1 -- 172.21.15.107:0/3227999761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376733144 0 0) 0x7fdeb815b750 con 0x7fdeb815b370 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb559a640 1 -- 172.21.15.107:0/3227999761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea4003520 con 0x7fdeb815b370 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3227999761 >> v1:172.21.15.120:6789/0 conn(0x7fdeb806aa40 legacy=0x7fdeb806e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3227999761 >> v1:172.21.15.107:6789/0 conn(0x7fdeb815b370 legacy=0x7fdeb80aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.529+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3227999761 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb8071890 con 0x7fdeb8070c80 2023-11-24T08:06:15.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.530+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3227999761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb8070c80 0x7fdeb806a500 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097a0 tx=0x7fdea002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a9d1d0777820c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb559a640 1 -- 172.21.15.107:0/3227999761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea002e070 con 0x7fdeb8070c80 2023-11-24T08:06:15.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb559a640 1 -- 172.21.15.107:0/3227999761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00315a0 con 0x7fdeb8070c80 2023-11-24T08:06:15.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb559a640 1 -- 172.21.15.107:0/3227999761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031930 con 0x7fdeb8070c80 2023-11-24T08:06:15.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb759e640 1 -- 172.21.15.107:0/3227999761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb8070c80 msgr2=0x7fdeb806a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb759e640 1 --2- 172.21.15.107:0/3227999761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb8070c80 0x7fdeb806a500 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097a0 tx=0x7fdea002b840 comp rx=0 tx=0).stop 2023-11-24T08:06:15.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb759e640 1 -- 172.21.15.107:0/3227999761 shutdown_connections 2023-11-24T08:06:15.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb759e640 1 --2- 172.21.15.107:0/3227999761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb8070c80 0x7fdeb806a500 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.531+0000 7fdeb759e640 1 -- 172.21.15.107:0/3227999761 >> 172.21.15.107:0/3227999761 conn(0x7fdeb80b14f0 msgr2=0x7fdeb80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.532+0000 7fdeb759e640 1 -- 172.21.15.107:0/3227999761 shutdown_connections 2023-11-24T08:06:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.532+0000 7fdeb759e640 1 -- 172.21.15.107:0/3227999761 wait complete. 2023-11-24T08:06:15.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.533+0000 7fdeb759e640 1 Processor -- start 2023-11-24T08:06:15.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.534+0000 7fdeb759e640 1 -- start start 2023-11-24T08:06:15.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.534+0000 7fdeb759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb806aa40 0x7fdeb8158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.534+0000 7fdeb759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 0x7fdeb80adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f3a90 con 0x7fdeb815b370 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f3d70 con 0x7fdeb8070c80 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f4050 con 0x7fdeb806aa40 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43410/0 (socket says 172.21.15.107:43410) 2023-11-24T08:06:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/261596627 learned_addr learned my addr 172.21.15.107:0/261596627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb659c640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb806aa40 0x7fdeb8158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb806aa40 msgr2=0x7fdeb8158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb806aa40 0x7fdeb8158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 0x7fdeb80adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 msgr2=0x7fdeb80adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 0x7fdeb80adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/261596627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0009400 con 0x7fdeb815b370 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.535+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 0x7fdeb80adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000b410 tx=0x7fdeb000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeaeffd640 1 -- 172.21.15.107:0/261596627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0004020 con 0x7fdeb815b370 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeaeffd640 1 -- 172.21.15.107:0/261596627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0004180 con 0x7fdeb815b370 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb81f4330 con 0x7fdeb815b370 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeaeffd640 1 -- 172.21.15.107:0/261596627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0011630 con 0x7fdeb815b370 2023-11-24T08:06:15.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.536+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb8159500 con 0x7fdeb815b370 2023-11-24T08:06:15.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.538+0000 7fdeaeffd640 1 -- 172.21.15.107:0/261596627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdeb0011790 con 0x7fdeb815b370 2023-11-24T08:06:15.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.539+0000 7fdeaeffd640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 0x7fde9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.539+0000 7fdeaeffd640 1 -- 172.21.15.107:0/261596627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdeb00892c0 con 0x7fdeb815b370 2023-11-24T08:06:15.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.539+0000 7fdeb659c640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 0x7fde9806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.540+0000 7fdebf61b740 1 Processor -- start 2023-11-24T08:06:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.540+0000 7fdeb659c640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 0x7fde9806d300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdea8001560 tx=0x7fdea8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.540+0000 7fdebf61b740 1 -- start start 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa4477a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 0x55fa44779270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa446d8880 con 0x55fa447797b0 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa446e7ae0 con 0x55fa44774d70 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa446e51d0 con 0x55fa4477a070 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43418/0 (socket says 172.21.15.107:43418) 2023-11-24T08:06:15.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.541+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3697812170 learned_addr learned my addr 172.21.15.107:0/3697812170 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa4477a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 0x55fa44779270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 msgr2=0x55fa4477a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa4477a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 msgr2=0x55fa44779270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 0x55fa44779270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3697812170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa446a8860 con 0x55fa447797b0 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 0x55fa44779270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0087790 tx=0x7fdeb0058030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95f2fbd2af43e8e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fde97fff640 1 -- 172.21.15.107:0/3697812170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00563d0 con 0x55fa447797b0 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fde97fff640 1 -- 172.21.15.107:0/3697812170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0054740 con 0x55fa447797b0 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.542+0000 7fde97fff640 1 -- 172.21.15.107:0/3697812170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0057760 con 0x55fa447797b0 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 -- 172.21.15.107:0/3697812170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 msgr2=0x55fa447bd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0087790 tx=0x7fdeb0058030 comp rx=0 tx=0).stop 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 -- 172.21.15.107:0/3697812170 shutdown_connections 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa447bd730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa44774d70 0x55fa44779270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3697812170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa4477a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 -- 172.21.15.107:0/3697812170 >> 172.21.15.107:0/3697812170 conn(0x55fa4473bc70 msgr2=0x55fa447bfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 -- 172.21.15.107:0/3697812170 shutdown_connections 2023-11-24T08:06:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.543+0000 7fdebf61b740 1 -- 172.21.15.107:0/3697812170 wait complete. 2023-11-24T08:06:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.544+0000 7fdebf61b740 1 -- start start 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.544+0000 7fdebf61b740 1 Processor -- start 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa447c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa446e0ac0 con 0x55fa4477a070 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa4467d2f0 con 0x55fa447797b0 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa4469ac70 con 0x55fa44774d70 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdebf61b740 10 client.0 _ll_register_callbacks cb 0x55fa4469f3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43430/0 (socket says 172.21.15.107:43430) 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.545+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3379090778 learned_addr learned my addr 172.21.15.107:0/3379090778 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa447c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 msgr2=0x55fa447d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447d3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 msgr2=0x55fa447c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa447c30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3379090778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb8159500 con 0x55fa4477a070 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00588e0 tx=0x7fdeb0089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a38d2d60861ee8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fde957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00548b0 con 0x55fa4477a070 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdeb005fea0 con 0x55fa4477a070 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.546+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa446d9aa0 con 0x55fa4477a070 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdeb0060080 con 0x55fa4477a070 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 10 client.5277 fetch_fsmap learned FSMap version 14 2023-11-24T08:06:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x55fa446df140 con 0x55fa4477a070 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fa446d9aa0 con 0x55fa4477a070 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 14) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdeb0087280 con 0x55fa4477a070 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 10 client.5277 fetch_fsmap finished waiting for FSMap version 14 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 10 client.5277 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 20 client.5277 populate_metadata read hostname 'smithi107' 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447a48a0 0x55fa447a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 4 client.5277 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 --> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa446d97c0 con 0x55fa447a48a0 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.547+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447a48a0 0x55fa447a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.548+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447a48a0 0x55fa447a4c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a2874a443634821 server_cookie=4b6c2e75290ca0c8 in_seq=0 out_seq=0 2023-11-24T08:06:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.548+0000 7fde957fa640 10 client.5277 ms_handle_connect on v2:172.21.15.107:6836/4081761849 2023-11-24T08:06:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.548+0000 7fde957fa640 1 -- 172.21.15.107:0/3379090778 <== mds.0 v2:172.21.15.107:6836/4081761849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fa446d97c0 con 0x55fa447a48a0 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.548+0000 7fde957fa640 10 client.5277 handle_command_reply: tid=0 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 client.5277 shutdown 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 msgr2=0x55fa447a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00588e0 tx=0x7fdeb0089040 comp rx=0 tx=0).stop 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 shutdown_connections 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447a48a0 0x55fa447a4c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa4477a070 0x55fa447a0d40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa447c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fde957fa640 0 client.5277 ms_handle_reset on v2:172.21.15.107:6836/4081761849 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3379090778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 >> 172.21.15.107:0/3379090778 conn(0x55fa4473bc70 msgr2=0x55fa447bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 shutdown_connections 2023-11-24T08:06:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 1 -- 172.21.15.107:0/3379090778 wait complete. 2023-11-24T08:06:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.549+0000 7fdebf61b740 20 client.5277 trim_cache size 0 max 16384 2023-11-24T08:06:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.566+0000 7fdebf61b740 1 Processor -- start 2023-11-24T08:06:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.566+0000 7fdebf61b740 1 -- start start 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa4477a070 0x55fa44774310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa447772d0 con 0x55fa447797b0 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa44443f20 con 0x55fa4477a070 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa447bf510 con 0x55fa44774d70 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdeb5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdeb5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43440/0 (socket says 172.21.15.107:43440) 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.567+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3753196343 learned_addr learned my addr 172.21.15.107:0/3753196343 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 msgr2=0x55fa447e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa4477a070 msgr2=0x55fa44774310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa4477a070 0x55fa44774310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/3753196343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa4467cb80 con 0x55fa447797b0 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097a0 tx=0x7fdea0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9123461301381a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeacff9640 1 -- 172.21.15.107:0/3753196343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031a10 con 0x55fa447797b0 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeacff9640 1 -- 172.21.15.107:0/3753196343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea0031b70 con 0x55fa447797b0 2023-11-24T08:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.568+0000 7fdeacff9640 1 -- 172.21.15.107:0/3753196343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031ee0 con 0x55fa447797b0 2023-11-24T08:06:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 -- 172.21.15.107:0/3753196343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 msgr2=0x55fa44773dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097a0 tx=0x7fdea0008030 comp rx=0 tx=0).stop 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 -- 172.21.15.107:0/3753196343 shutdown_connections 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa4477a070 0x55fa44774310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa447797b0 0x55fa44773dd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3753196343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa44774d70 0x55fa447e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 -- 172.21.15.107:0/3753196343 >> 172.21.15.107:0/3753196343 conn(0x55fa4473bc70 msgr2=0x55fa447a0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 -- 172.21.15.107:0/3753196343 shutdown_connections 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.569+0000 7fdebf61b740 1 -- 172.21.15.107:0/3753196343 wait complete. 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.570+0000 7fdebf61b740 1 -- start start 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.570+0000 7fdebf61b740 1 Processor -- start 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa44774310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa447bbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa447c0a10 con 0x55fa44774d70 2023-11-24T08:06:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa447bce50 con 0x55fa447797b0 2023-11-24T08:06:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa447c07c0 con 0x55fa4477a070 2023-11-24T08:06:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdebf61b740 10 client.0 _ll_register_callbacks cb 0x7fdeb0084070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43442/0 (socket says 172.21.15.107:43442) 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 -- 172.21.15.107:0/3751481011 learned_addr learned my addr 172.21.15.107:0/3751481011 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa447bbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa44774310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 -- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 msgr2=0x55fa447bbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa447bbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.571+0000 7fdeb659c640 1 -- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 msgr2=0x55fa44774310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa44774310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdeb659c640 1 -- 172.21.15.107:0/3751481011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb8071890 con 0x55fa44774d70 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa44774310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdea8007040 tx=0x7fdea8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8959890c853cbfa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fde957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80235c0 con 0x55fa44774d70 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdea8023720 con 0x55fa44774d70 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdeb0054740 con 0x55fa44774d70 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdea80238e0 con 0x55fa44774d70 2023-11-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdebf61b740 10 client.5283 fetch_fsmap learned FSMap version 14 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.572+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x55fa446a2170 con 0x55fa44774d70 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0054740 con 0x55fa44774d70 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 14) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdea802c440 con 0x55fa44774d70 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 10 client.5283 fetch_fsmap finished waiting for FSMap version 14 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 10 client.5283 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 20 client.5283 populate_metadata read hostname 'smithi107' 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447bc430 0x55fa4482c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 4 client.5283 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 --> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fa443e16c0 con 0x55fa447bc430 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.573+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447bc430 0x55fa4482c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdeb5d9b640 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447bc430 0x55fa4482c7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=541b257934075e7e server_cookie=6df9b4391168fc87 in_seq=0 out_seq=0 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fde957fa640 10 client.5283 ms_handle_connect on v2:172.21.15.107:6836/4081761849 2023-11-24T08:06:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fde957fa640 1 -- 172.21.15.107:0/3751481011 <== mds.0 v2:172.21.15.107:6836/4081761849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55fa443e16c0 con 0x55fa447bc430 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fde957fa640 10 client.5283 handle_command_reply: tid=0 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 client.5283 shutdown 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 msgr2=0x55fa44773dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdea8007040 tx=0x7fdea8001250 comp rx=0 tx=0).stop 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 shutdown_connections 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:6836/4081761849,v1:172.21.15.107:6837/4081761849] conn(0x55fa447bc430 0x55fa4482c7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fa4477a070 0x55fa447bbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa447797b0 0x55fa44774310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 --2- 172.21.15.107:0/3751481011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa44774d70 0x55fa44773dd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fde957fa640 0 client.5283 ms_handle_reset on v2:172.21.15.107:6836/4081761849 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 >> 172.21.15.107:0/3751481011 conn(0x55fa4473bc70 msgr2=0x55fa447a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 shutdown_connections 2023-11-24T08:06:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 1 -- 172.21.15.107:0/3751481011 wait complete. 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.574+0000 7fdebf61b740 20 client.5283 trim_cache size 0 max 16384 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 5070, 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:06:15.581 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 11.600016371000001, 2023-11-24T08:06:15.582 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 14, 2023-11-24T08:06:15.582 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 34, 2023-11-24T08:06:15.582 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 34, 2023-11-24T08:06:15.582 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 21.392992636999999 2023-11-24T08:06:15.582 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 msgr2=0x7fde9806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 0x7fde9806d300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdea8001560 tx=0x7fdea8009040 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 msgr2=0x7fdeb8148ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000b410 tx=0x7fdeb000b8d0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 shutdown_connections 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806ae40 0x7fde9806d300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb815b370 0x7fdeb8148ec0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb8070c80 0x7fdeb80adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 --2- 172.21.15.107:0/261596627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb806aa40 0x7fdeb8158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 >> 172.21.15.107:0/261596627 conn(0x7fdeb80b14f0 msgr2=0x7fdeb80cd640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 shutdown_connections 2023-11-24T08:06:15.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.575+0000 7fdeb759e640 1 -- 172.21.15.107:0/261596627 wait complete. 2023-11-24T08:06:15.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:06:15.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.796+0000 7fd9d8a90640 1 Processor -- start 2023-11-24T08:06:15.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.796+0000 7fd9d8a90640 1 -- start start 2023-11-24T08:06:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.797+0000 7fd9d8a90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9d40abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.797+0000 7fd9d8a90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40ce270 con 0x7fd9d40cc470 2023-11-24T08:06:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d8a90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d406d0a0 con 0x7fd9d40cfb60 2023-11-24T08:06:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d8a90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d406c560 con 0x7fd9d40ce640 2023-11-24T08:06:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9d40cc470 0x7fd9d40cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58910/0 (socket says 172.21.15.107:58910) 2023-11-24T08:06:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d259c640 1 -- 172.21.15.107:0/2996436430 learned_addr learned my addr 172.21.15.107:0/2996436430 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d2d9d640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9d40abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831716938 0 0) 0x7fd9d40ce270 con 0x7fd9d40cc470 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.798+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0003520 con 0x7fd9d40cc470 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746482871 0 0) 0x7fd9c0003520 con 0x7fd9d40cc470 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 msgr2=0x7fd9d40abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9d40abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 >> v1:172.21.15.120:6789/0 conn(0x7fd9d40cfb60 legacy=0x7fd9d406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d406c770 con 0x7fd9d40cc470 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd9c4001670 con 0x7fd9d40cc470 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 >> v1:172.21.15.107:6789/0 conn(0x7fd9d40cc470 legacy=0x7fd9d40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9c00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9c0003f20 0x7fd9c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0004d00 0x7fd9c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d406c560 con 0x7fd9c0003f20 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c000ba60 con 0x7fd9c0004d00 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c000bc20 con 0x7fd9d40ce640 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9c9d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9c0003f20 0x7fd9c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.799+0000 7fd9d259c640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9c00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9d2d9d640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0004d00 0x7fd9c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9c9d9b640 1 -- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 msgr2=0x7fd9c00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9c9d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40ce640 0x7fd9c00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9c9d9b640 1 -- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0004d00 msgr2=0x7fd9c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9c9d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0004d00 0x7fd9c0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.800+0000 7fd9c9d9b640 1 -- 172.21.15.107:0/2996436430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0003520 con 0x7fd9c0003f20 2023-11-24T08:06:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9c9d9b640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9c0003f20 0x7fd9c00047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001750 tx=0x7fd9c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7a7b0b885f24bf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0052d0 con 0x7fd9c0003f20 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc02e050 con 0x7fd9c0003f20 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d1d9b640 1 -- 172.21.15.107:0/2996436430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc003830 con 0x7fd9c0003f20 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d8a90640 1 -- 172.21.15.107:0/2996436430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9c0003f20 msgr2=0x7fd9c00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/2996436430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9c0003f20 0x7fd9c00047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001750 tx=0x7fd9c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d2d9d640 1 -- 172.21.15.107:0/2996436430 reap_dead start 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d8a90640 1 -- 172.21.15.107:0/2996436430 shutdown_connections 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.801+0000 7fd9d8a90640 1 -- 172.21.15.107:0/2996436430 >> 172.21.15.107:0/2996436430 conn(0x7fd9d406a920 msgr2=0x7fd9d406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.802+0000 7fd9d8a90640 1 -- 172.21.15.107:0/2996436430 shutdown_connections 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.802+0000 7fd9d8a90640 1 -- 172.21.15.107:0/2996436430 wait complete. 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.803+0000 7fd9d8a90640 1 Processor -- start 2023-11-24T08:06:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.803+0000 7fd9d8a90640 1 -- start start 2023-11-24T08:06:15.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0003f20 0x7fd9d4164710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40b15d0 0x7fd9d4142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41eb940 con 0x7fd9d4143420 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4163c30 con 0x7fd9c0003f20 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.804+0000 7fd9d8a90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4163f10 con 0x7fd9d40b15d0 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43456/0 (socket says 172.21.15.107:43456) 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 -- 172.21.15.107:0/1702965904 learned_addr learned my addr 172.21.15.107:0/1702965904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d259c640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0003f20 0x7fd9d4164710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9c9d9b640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40b15d0 0x7fd9d4142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 -- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40b15d0 msgr2=0x7fd9d4142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40b15d0 0x7fd9d4142ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 -- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0003f20 msgr2=0x7fd9d4164710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0003f20 0x7fd9d4164710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.805+0000 7fd9d2d9d640 1 -- 172.21.15.107:0/1702965904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d41641f0 con 0x7fd9d4143420 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9d2d9d640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4070570 tx=0x7fd9cc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc003990 con 0x7fd9d4143420 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc011040 con 0x7fd9d4143420 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc003d50 con 0x7fd9d4143420 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d40b1a50 con 0x7fd9d4143420 2023-11-24T08:06:15.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.806+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d41eb530 con 0x7fd9d4143420 2023-11-24T08:06:15.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.807+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d4067470 con 0x7fd9d4143420 2023-11-24T08:06:15.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.808+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd9cc003eb0 con 0x7fd9d4143420 2023-11-24T08:06:15.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.809+0000 7fd9cb7fe640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 0x7fd9b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:15.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.809+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9cc089b20 con 0x7fd9d4143420 2023-11-24T08:06:15.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.809+0000 7fd9d259c640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 0x7fd9b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:15.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.810+0000 7fd9d259c640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 0x7fd9b406d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4001c80 tx=0x7fd9c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:15.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:15.814+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9cc054b30 con 0x7fd9d4143420 2023-11-24T08:06:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.042+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd9d40ac390 con 0x7fd9d4143420 2023-11-24T08:06:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.043+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1702965904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd9cc058150 con 0x7fd9d4143420 2023-11-24T08:06:16.045 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:06:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 msgr2=0x7fd9b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 0x7fd9b406d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4001c80 tx=0x7fd9c4009040 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 msgr2=0x7fd9d41ed210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4070570 tx=0x7fd9cc00f4a0 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 shutdown_connections 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9b406ae40 0x7fd9b406d300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4143420 0x7fd9d41ed210 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9d40b15d0 0x7fd9d4142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 --2- 172.21.15.107:0/1702965904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c0003f20 0x7fd9d4164710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.045+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 >> 172.21.15.107:0/1702965904 conn(0x7fd9d406a920 msgr2=0x7fd9d40ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.046+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 shutdown_connections 2023-11-24T08:06:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:16.046+0000 7fd9d8a90640 1 -- 172.21.15.107:0/1702965904 wait complete. 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:06:16.066 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:06:16.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:06:16.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:06:16.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:06:16.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.067 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42294sec preferred_lft 42294sec 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:16.117 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:06:16.118 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:16.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.120 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:06:16.250 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:06:16.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.254 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:06:16.302 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:06:16.303 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:06:16.303 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:06:16.303 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:06:16.303 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:06:16.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.303 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:06:16.303 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:06:16.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:06:16.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:06:16.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:06:16.304 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:06:16.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.489 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:06:16.541 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:06:16.541 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:06:16.542 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:06:16.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:06:16.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:06:16.636 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:06:16.692 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:06:16.692 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:06:16.692 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:06:16.746 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:06:16.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:16.886 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:16.886 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:06:16.919 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:16.919 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:06:16.920 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:06:16.920 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:06:16.920 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:06:16.922 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:06:16.922 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:06:16.922 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:06:16.922 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:06:16.922 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:06:16.927 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:06:16.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:16.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:06:17.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:19.884 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:06:19.950 INFO:teuthology.orchestra.run.smithi107.stdout:client4628 2023-11-24T08:06:19.950 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:19.951 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:20.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:06:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.219+0000 7fd265e07640 1 Processor -- start 2023-11-24T08:06:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.219+0000 7fd265e07640 1 -- start start 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd265e07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd26015c7f0 0x7fd26006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd265e07640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 0x7fd2600cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd265e07640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2600cece0 con 0x7fd2600cdfb0 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd265e07640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2600cfe40 con 0x7fd26015c7f0 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd265e07640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2600ceaf0 con 0x7fd2600cf450 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd25f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd26015c7f0 0x7fd26006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd25f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd26015c7f0 0x7fd26006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52610/0 (socket says 172.21.15.107:52610) 2023-11-24T08:06:20.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.221+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2326834956 learned_addr learned my addr 172.21.15.107:0/2326834956 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:20.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd257fff640 1 --2- 172.21.15.107:0/2326834956 >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 0x7fd2600cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:20.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2326834956 >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 msgr2=0x7fd2600cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:20.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd25f7fe640 1 --2- 172.21.15.107:0/2326834956 >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 0x7fd2600cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2326834956 >> v1:172.21.15.120:6789/0 conn(0x7fd2600cdfb0 legacy=0x7fd2600ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2326834956 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26006d040 con 0x7fd26015c7f0 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.222+0000 7fd257fff640 1 --2- 172.21.15.107:0/2326834956 >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 0x7fd2600cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.223+0000 7fd25f7fe640 1 --2- 172.21.15.107:0/2326834956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd26015c7f0 0x7fd26006db20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd260070410 tx=0x7fd25002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e9820a8b0c40be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd25effd640 1 -- 172.21.15.107:0/2326834956 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250031870 con 0x7fd26015c7f0 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd25effd640 1 -- 172.21.15.107:0/2326834956 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2500319d0 con 0x7fd26015c7f0 2023-11-24T08:06:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd25effd640 1 -- 172.21.15.107:0/2326834956 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250031d60 con 0x7fd26015c7f0 2023-11-24T08:06:20.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 -- 172.21.15.107:0/2326834956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd26015c7f0 msgr2=0x7fd26006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:20.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 --2- 172.21.15.107:0/2326834956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd26015c7f0 0x7fd26006db20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd260070410 tx=0x7fd25002b910 comp rx=0 tx=0).stop 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 -- 172.21.15.107:0/2326834956 shutdown_connections 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 --2- 172.21.15.107:0/2326834956 >> v2:172.21.15.120:3300/0 conn(0x7fd2600cf450 0x7fd2600cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 --2- 172.21.15.107:0/2326834956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd26015c7f0 0x7fd26006db20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 -- 172.21.15.107:0/2326834956 >> 172.21.15.107:0/2326834956 conn(0x7fd26006a920 msgr2=0x7fd26006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 -- 172.21.15.107:0/2326834956 shutdown_connections 2023-11-24T08:06:20.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.224+0000 7fd265e07640 1 -- 172.21.15.107:0/2326834956 wait complete. 2023-11-24T08:06:20.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.226+0000 7fd265e07640 1 Processor -- start 2023-11-24T08:06:20.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.226+0000 7fd265e07640 1 -- start start 2023-11-24T08:06:20.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2600cf450 0x7fd2601f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd26015c7f0 0x7fd2601506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b24c0 con 0x7fd2600cdfb0 2023-11-24T08:06:20.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b27a0 con 0x7fd2600cf450 2023-11-24T08:06:20.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd265e07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b2a80 con 0x7fd26015c7f0 2023-11-24T08:06:20.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd25f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:20.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd25f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52624/0 (socket says 172.21.15.107:52624) 2023-11-24T08:06:20.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.227+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2912481740 learned_addr learned my addr 172.21.15.107:0/2912481740 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd26015c7f0 msgr2=0x7fd2601506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd26015c7f0 0x7fd2601506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2600cf450 msgr2=0x7fd2601f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2600cf450 0x7fd2601f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 -- 172.21.15.107:0/2912481740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500094d0 con 0x7fd2600cdfb0 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25f7fe640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd250000c00 tx=0x7fd2500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250031840 con 0x7fd2600cdfb0 2023-11-24T08:06:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2500443f0 con 0x7fd2600cdfb0 2023-11-24T08:06:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2600b2d60 con 0x7fd2600cdfb0 2023-11-24T08:06:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.228+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2500447c0 con 0x7fd2600cdfb0 2023-11-24T08:06:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.229+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2601f3970 con 0x7fd2600cdfb0 2023-11-24T08:06:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.230+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2600b3280 con 0x7fd2600cdfb0 2023-11-24T08:06:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.231+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd25004d450 con 0x7fd2600cdfb0 2023-11-24T08:06:20.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.231+0000 7fd25cff9640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 0x7fd24006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:20.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.232+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd25002d040 con 0x7fd2600cdfb0 2023-11-24T08:06:20.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.232+0000 7fd257fff640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 0x7fd24006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:20.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.233+0000 7fd257fff640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 0x7fd24006d330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd248006fd0 tx=0x7fd248008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:20.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.237+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2600b3280 con 0x7fd2600cdfb0 2023-11-24T08:06:20.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.463+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd2600b3280 con 0x7fd2600cdfb0 2023-11-24T08:06:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.464+0000 7fd25cff9640 1 -- 172.21.15.107:0/2912481740 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd2600b3280 con 0x7fd2600cdfb0 2023-11-24T08:06:20.466 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 msgr2=0x7fd24006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 0x7fd24006d330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd248006fd0 tx=0x7fd248008040 comp rx=0 tx=0).stop 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 msgr2=0x7fd26009caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd250000c00 tx=0x7fd2500046c0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 shutdown_connections 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd24006ae70 0x7fd24006d330 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd26015c7f0 0x7fd2601506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2600cf450 0x7fd2601f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 --2- 172.21.15.107:0/2912481740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2600cdfb0 0x7fd26009caf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.466+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 >> 172.21.15.107:0/2912481740 conn(0x7fd26006a920 msgr2=0x7fd2600ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.467+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 shutdown_connections 2023-11-24T08:06:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:20.467+0000 7fd265e07640 1 -- 172.21.15.107:0/2912481740 wait complete. 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:06:20.487 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:06:20.488 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:06:20.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.488 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.549 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42288sec preferred_lft 42288sec 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:06:20.551 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:06:20.552 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:06:20.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.553 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:06:20.641 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:06:20.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.646 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:06:20.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:06:20.700 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:06:20.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.879 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:06:20.929 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:06:20.929 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:06:20.930 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:06:20.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:20.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:06:21.026 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:06:21.027 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:06:21.082 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:06:21.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:06:21.083 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:06:21.133 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:06:21.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:21.133 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:21.266 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:21.266 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:06:21.303 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:06:21.304 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:06:21.304 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:06:21.304 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:06:21.304 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:06:21.306 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:06:21.306 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:06:21.307 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:06:21.307 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:06:21.307 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:06:21.311 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:06:21.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:21.311 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:06:21.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:21.403 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:06:21.465 INFO:teuthology.orchestra.run.smithi120.stdout:client5298 2023-11-24T08:06:21.466 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:06:21.466 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:21.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:06:21.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.737+0000 7f240d1b0640 1 Processor -- start 2023-11-24T08:06:21.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.738+0000 7f240d1b0640 1 -- start start 2023-11-24T08:06:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240d1b0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24080cf450 0x7f24080cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240d1b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24080cece0 con 0x7f24080cdfb0 2023-11-24T08:06:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240d1b0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240806d040 con 0x7f240815c7f0 2023-11-24T08:06:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240d1b0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24080cfe40 con 0x7f24080cf450 2023-11-24T08:06:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24080cdfb0 0x7f24080ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57832/0 (socket says 172.21.15.107:57832) 2023-11-24T08:06:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.739+0000 7f240759e640 1 -- 172.21.15.107:0/529813009 learned_addr learned my addr 172.21.15.107:0/529813009 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f23fffff640 1 --2- 172.21.15.107:0/529813009 >> v2:172.21.15.120:3300/0 conn(0x7f24080cf450 0x7f24080cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f240659c640 1 -- 172.21.15.107:0/529813009 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639131478 0 0) 0x7f24080cece0 con 0x7f24080cdfb0 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f240659c640 1 -- 172.21.15.107:0/529813009 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f4003520 con 0x7f24080cdfb0 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f23fffff640 1 -- 172.21.15.107:0/529813009 >> v1:172.21.15.120:6789/0 conn(0x7f240815c7f0 legacy=0x7f240806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f23fffff640 1 -- 172.21.15.107:0/529813009 >> v1:172.21.15.107:6789/0 conn(0x7f24080cdfb0 legacy=0x7f24080ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.740+0000 7f23fffff640 1 -- 172.21.15.107:0/529813009 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240806d320 con 0x7f24080cf450 2023-11-24T08:06:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.741+0000 7f23fffff640 1 --2- 172.21.15.107:0/529813009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24080cf450 0x7f24080cf900 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f23f00097a0 tx=0x7f23f002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b598cf37b051fa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.742+0000 7f240659c640 1 -- 172.21.15.107:0/529813009 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0031830 con 0x7f24080cf450 2023-11-24T08:06:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.742+0000 7f240659c640 1 -- 172.21.15.107:0/529813009 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0031990 con 0x7f24080cf450 2023-11-24T08:06:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.742+0000 7f240659c640 1 -- 172.21.15.107:0/529813009 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0031d00 con 0x7f24080cf450 2023-11-24T08:06:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.742+0000 7f240d1b0640 1 -- 172.21.15.107:0/529813009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24080cf450 msgr2=0x7f24080cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.742+0000 7f240d1b0640 1 --2- 172.21.15.107:0/529813009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24080cf450 0x7f24080cf900 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f23f00097a0 tx=0x7f23f002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:06:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.743+0000 7f240d1b0640 1 -- 172.21.15.107:0/529813009 shutdown_connections 2023-11-24T08:06:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.743+0000 7f240d1b0640 1 --2- 172.21.15.107:0/529813009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24080cf450 0x7f24080cf900 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.743+0000 7f240d1b0640 1 -- 172.21.15.107:0/529813009 >> 172.21.15.107:0/529813009 conn(0x7f240806a920 msgr2=0x7f240806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.743+0000 7f240d1b0640 1 -- 172.21.15.107:0/529813009 shutdown_connections 2023-11-24T08:06:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.743+0000 7f240d1b0640 1 -- 172.21.15.107:0/529813009 wait complete. 2023-11-24T08:06:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.745+0000 7f240d1b0640 1 Processor -- start 2023-11-24T08:06:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.745+0000 7f240d1b0640 1 -- start start 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24080cdfb0 0x7f2408149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f240815c7f0 0x7f24080b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240814a220 con 0x7f24080cf450 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24081f4580 con 0x7f240815c7f0 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.746+0000 7f240d1b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24081f4860 con 0x7f24080cdfb0 2023-11-24T08:06:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52632/0 (socket says 172.21.15.107:52632) 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 -- 172.21.15.107:0/1793005194 learned_addr learned my addr 172.21.15.107:0/1793005194 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f240759e640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f240815c7f0 0x7f24080b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f2406d9d640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24080cdfb0 0x7f2408149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 -- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24080cdfb0 msgr2=0x7f2408149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24080cdfb0 0x7f2408149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 -- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f240815c7f0 msgr2=0x7f24080b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f240815c7f0 0x7f24080b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.747+0000 7f23fffff640 1 -- 172.21.15.107:0/1793005194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f0009400 con 0x7f24080cf450 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f23fffff640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f23f002b910 tx=0x7f23f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f00317b0 con 0x7f24080cf450 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0032e10 con 0x7f24080cf450 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0003810 con 0x7f24080cf450 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24081f4b40 con 0x7f24080cf450 2023-11-24T08:06:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.748+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24081f5100 con 0x7f24080cf450 2023-11-24T08:06:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.749+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2408067470 con 0x7f24080cf450 2023-11-24T08:06:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.750+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f23f0003970 con 0x7f24080cf450 2023-11-24T08:06:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.751+0000 7f23ff7fe640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 0x7f23e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.751+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23f00aaf30 con 0x7f24080cf450 2023-11-24T08:06:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.751+0000 7f2406d9d640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 0x7f23e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.752+0000 7f2406d9d640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 0x7f23e806d300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23f8006fd0 tx=0x7f23f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.757+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23f00761a0 con 0x7f24080cf450 2023-11-24T08:06:21.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:21.982+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f24080cd730 con 0x7f24080cf450 2023-11-24T08:06:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.536+0000 7f23ff7fe640 1 -- 172.21.15.107:0/1793005194 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 136+0+0 (secure 0 0 0) 0x7f23f00350c0 con 0x7f24080cf450 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 msgr2=0x7f23e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 0x7f23e806d300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23f8006fd0 tx=0x7f23f8008040 comp rx=0 tx=0).stop 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 msgr2=0x7f24080b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f23f002b910 tx=0x7f23f00046c0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 shutdown_connections 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f23e806ae40 0x7f23e806d300 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f240815c7f0 0x7f24080b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24080cf450 0x7f24080b0cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 --2- 172.21.15.107:0/1793005194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24080cdfb0 0x7f2408149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 >> 172.21.15.107:0/1793005194 conn(0x7f240806a920 msgr2=0x7f24080b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 shutdown_connections 2023-11-24T08:06:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.540+0000 7f240d1b0640 1 -- 172.21.15.107:0/1793005194 wait complete. 2023-11-24T08:06:22.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:06:22.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.768+0000 7f20f283d640 1 Processor -- start 2023-11-24T08:06:22.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.769+0000 7f20f283d640 1 -- start start 2023-11-24T08:06:22.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.770+0000 7f20f283d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20ec146eb0 0x7f20ec161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.770+0000 7f20f283d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 0x7f20ec0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.770+0000 7f20f283d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec161a10 con 0x7f20ec0cd110 2023-11-24T08:06:22.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.770+0000 7f20f283d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec0cd580 con 0x7f20ec146eb0 2023-11-24T08:06:22.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.770+0000 7f20f283d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec161cf0 con 0x7f20ec06e160 2023-11-24T08:06:22.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20ec146eb0 0x7f20ec161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20ec146eb0 0x7f20ec161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52636/0 (socket says 172.21.15.107:52636) 2023-11-24T08:06:22.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 -- 172.21.15.107:0/2914605107 learned_addr learned my addr 172.21.15.107:0/2914605107 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 -- 172.21.15.107:0/2914605107 >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 msgr2=0x7f20ec0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20eb7fe640 1 --2- 172.21.15.107:0/2914605107 >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 0x7f20ec0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 --2- 172.21.15.107:0/2914605107 >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 0x7f20ec0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 -- 172.21.15.107:0/2914605107 >> v1:172.21.15.120:6789/0 conn(0x7f20ec0cd110 legacy=0x7f20ec06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20ebfff640 1 -- 172.21.15.107:0/2914605107 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec161fd0 con 0x7f20ec146eb0 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.771+0000 7f20eb7fe640 1 --2- 172.21.15.107:0/2914605107 >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 0x7f20ec0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:22.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.772+0000 7f20ebfff640 1 --2- 172.21.15.107:0/2914605107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec161150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f20ec06eab0 tx=0x7f20dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b6c878b96d4def server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:22.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20eaffd640 1 -- 172.21.15.107:0/2914605107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc02ca80 con 0x7f20ec146eb0 2023-11-24T08:06:22.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20eaffd640 1 -- 172.21.15.107:0/2914605107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc02cbe0 con 0x7f20ec146eb0 2023-11-24T08:06:22.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20eaffd640 1 -- 172.21.15.107:0/2914605107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0343f0 con 0x7f20ec146eb0 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 -- 172.21.15.107:0/2914605107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 msgr2=0x7f20ec161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 --2- 172.21.15.107:0/2914605107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec161150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f20ec06eab0 tx=0x7f20dc02bee0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 -- 172.21.15.107:0/2914605107 shutdown_connections 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 --2- 172.21.15.107:0/2914605107 >> v2:172.21.15.120:3300/0 conn(0x7f20ec06e160 0x7f20ec0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 --2- 172.21.15.107:0/2914605107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec161150 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.773+0000 7f20f283d640 1 -- 172.21.15.107:0/2914605107 >> 172.21.15.107:0/2914605107 conn(0x7f20ec0b0c50 msgr2=0x7f20ec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.774+0000 7f20f283d640 1 -- 172.21.15.107:0/2914605107 shutdown_connections 2023-11-24T08:06:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.774+0000 7f20f283d640 1 -- 172.21.15.107:0/2914605107 wait complete. 2023-11-24T08:06:22.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.776+0000 7f20f283d640 1 Processor -- start 2023-11-24T08:06:22.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.776+0000 7f20f283d640 1 -- start start 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20ec06e160 0x7f20ec15cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20ec0cd110 0x7f20ec15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec15eea0 con 0x7f20ec146eb0 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec15e4d0 con 0x7f20ec06e160 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.777+0000 7f20f283d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec09c050 con 0x7f20ec0cd110 2023-11-24T08:06:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52640/0 (socket says 172.21.15.107:52640) 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 -- 172.21.15.107:0/307842355 learned_addr learned my addr 172.21.15.107:0/307842355 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20eb7fe640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20ec0cd110 0x7f20ec15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20ebfff640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20ec06e160 0x7f20ec15cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 -- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20ec0cd110 msgr2=0x7f20ec15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20ec0cd110 0x7f20ec15e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 -- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20ec06e160 msgr2=0x7f20ec15cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20ec06e160 0x7f20ec15cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 -- 172.21.15.107:0/307842355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc009b30 con 0x7f20ec146eb0 2023-11-24T08:06:22.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.778+0000 7f20f0dda640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c390 tx=0x7f20e400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.779+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e40035f0 con 0x7f20ec146eb0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.779+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e4003750 con 0x7f20ec146eb0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.779+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4003b00 con 0x7f20ec146eb0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.779+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec15dba0 con 0x7f20ec146eb0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.779+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec15e160 con 0x7f20ec146eb0 2023-11-24T08:06:22.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.780+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec04e8b0 con 0x7f20ec146eb0 2023-11-24T08:06:22.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.781+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f20e4003c60 con 0x7f20ec146eb0 2023-11-24T08:06:22.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.782+0000 7f20e8ff9640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 0x7f20cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:22.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.783+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20e4088770 con 0x7f20ec146eb0 2023-11-24T08:06:22.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.783+0000 7f20ebfff640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 0x7f20cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:22.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.783+0000 7f20ebfff640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 0x7f20cc06d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f20dc035000 tx=0x7f20dc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:22.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.788+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20e4053810 con 0x7f20ec146eb0 2023-11-24T08:06:22.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.994+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f20ec161690 con 0x7f20cc06ae40 2023-11-24T08:06:22.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.996+0000 7f20e8ff9640 1 -- 172.21.15.107:0/307842355 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f20dc040450 con 0x7f20cc06ae40 2023-11-24T08:06:22.998 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:06:22.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:06:22.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:06:22.999 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:22.999 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 msgr2=0x7f20cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 0x7f20cc06d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f20dc035000 tx=0x7f20dc030000 comp rx=0 tx=0).stop 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 msgr2=0x7f20ec09b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c390 tx=0x7f20e400c850 comp rx=0 tx=0).stop 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 shutdown_connections 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20cc06ae40 0x7f20cc06d300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20ec146eb0 0x7f20ec09b7b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20ec0cd110 0x7f20ec15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 --2- 172.21.15.107:0/307842355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20ec06e160 0x7f20ec15cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 >> 172.21.15.107:0/307842355 conn(0x7f20ec0b0c50 msgr2=0x7f20ec145c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:23.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 shutdown_connections 2023-11-24T08:06:23.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:22.999+0000 7f20f283d640 1 -- 172.21.15.107:0/307842355 wait complete. 2023-11-24T08:06:23.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:06:23.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.225+0000 7fa9e60d1640 1 Processor -- start 2023-11-24T08:06:23.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.225+0000 7fa9e60d1640 1 -- start start 2023-11-24T08:06:23.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.226+0000 7fa9e60d1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9e00725b0 0x7fa9e0071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.226+0000 7fa9e60d1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e015e9b0 con 0x7fa9e015d810 2023-11-24T08:06:23.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.226+0000 7fa9e60d1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e015ebe0 con 0x7fa9e015de90 2023-11-24T08:06:23.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.226+0000 7fa9e60d1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e0070860 con 0x7fa9e00725b0 2023-11-24T08:06:23.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9e50cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa9e015d810 0x7fa9e0071ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57844/0 (socket says 172.21.15.107:57844) 2023-11-24T08:06:23.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9e50cf640 1 -- 172.21.15.107:0/136152923 learned_addr learned my addr 172.21.15.107:0/136152923 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000467562 0 0) 0x7fa9e015e9b0 con 0x7fa9e015d810 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc003520 con 0x7fa9e015d810 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/136152923 >> v2:172.21.15.120:3300/0 conn(0x7fa9e00725b0 0x7fa9e0071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170079965 0 0) 0x7fa9cc003520 con 0x7fa9e015d810 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 >> v2:172.21.15.120:3300/0 conn(0x7fa9e00725b0 msgr2=0x7fa9e0071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 --2- 172.21.15.107:0/136152923 >> v2:172.21.15.120:3300/0 conn(0x7fa9e00725b0 0x7fa9e0071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.227+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 >> v1:172.21.15.120:6789/0 conn(0x7fa9e015de90 legacy=0x7fa9e015e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0070b40 con 0x7fa9e015d810 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa9dc001670 con 0x7fa9e015d810 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 >> v1:172.21.15.107:6789/0 conn(0x7fa9e015d810 legacy=0x7fa9e0071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9cc0039c0 0x7fa9cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9cc0051d0 0x7fa9cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0070860 con 0x7fa9cc0047c0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc00c100 con 0x7fa9cc0051d0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc00c2c0 con 0x7fa9cc0039c0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e50cf640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9cc0039c0 0x7fa9cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e58d0640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9cc0051d0 0x7fa9cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 -- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9cc0039c0 msgr2=0x7fa9cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9cc0039c0 0x7fa9cc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 -- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9cc0051d0 msgr2=0x7fa9cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9cc0051d0 0x7fa9cc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e48ce640 1 -- 172.21.15.107:0/136152923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc003520 con 0x7fa9cc0047c0 2023-11-24T08:06:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.228+0000 7fa9e58d0640 1 -- 172.21.15.107:0/136152923 reap_dead start 2023-11-24T08:06:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.229+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9cc004c40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000ba00 tx=0x7fa9cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19b19f1b51fde45 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0032620 con 0x7fa9cc0047c0 2023-11-24T08:06:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0032780 con 0x7fa9cc0047c0 2023-11-24T08:06:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9d7fff640 1 -- 172.21.15.107:0/136152923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0032b10 con 0x7fa9cc0047c0 2023-11-24T08:06:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 -- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 msgr2=0x7fa9cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9cc004c40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000ba00 tx=0x7fa9cc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 -- 172.21.15.107:0/136152923 shutdown_connections 2023-11-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/136152923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9cc004c40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 -- 172.21.15.107:0/136152923 >> 172.21.15.107:0/136152923 conn(0x7fa9e006a500 msgr2=0x7fa9e006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 -- 172.21.15.107:0/136152923 shutdown_connections 2023-11-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.230+0000 7fa9e60d1640 1 -- 172.21.15.107:0/136152923 wait complete. 2023-11-24T08:06:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.232+0000 7fa9e60d1640 1 Processor -- start 2023-11-24T08:06:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.232+0000 7fa9e60d1640 1 -- start start 2023-11-24T08:06:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e00ce110 0x7fa9e00cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e00cf4b0 0x7fa9e0158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00cdcf0 con 0x7fa9cc0047c0 2023-11-24T08:06:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00cf9c0 con 0x7fa9e00cf4b0 2023-11-24T08:06:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e60d1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00cd350 con 0x7fa9e00ce110 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e50cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e50cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52660/0 (socket says 172.21.15.107:52660) 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.233+0000 7fa9e50cf640 1 -- 172.21.15.107:0/3688472006 learned_addr learned my addr 172.21.15.107:0/3688472006 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 -- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e00ce110 msgr2=0x7fa9e00cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e00ce110 0x7fa9e00cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 -- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e00cf4b0 msgr2=0x7fa9e0158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e00cf4b0 0x7fa9e0158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 -- 172.21.15.107:0/3688472006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00cd630 con 0x7fa9cc0047c0 2023-11-24T08:06:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e50cf640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc001200 tx=0x7fa9dc00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00c660 con 0x7fa9cc0047c0 2023-11-24T08:06:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc00c7c0 con 0x7fa9cc0047c0 2023-11-24T08:06:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00cb70 con 0x7fa9cc0047c0 2023-11-24T08:06:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.234+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e00cd910 con 0x7fa9cc0047c0 2023-11-24T08:06:23.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.235+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e01478a0 con 0x7fa9cc0047c0 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.236+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e00cfb80 con 0x7fa9cc0047c0 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.237+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa9dc013420 con 0x7fa9cc0047c0 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.238+0000 7fa9d5ffb640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 0x7fa9c406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.238+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9dc089550 con 0x7fa9cc0047c0 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.238+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 0x7fa9c406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.239+0000 7fa9e48ce640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 0x7fa9c406d330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0057a0 tx=0x7fa9d0032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.242+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9dc054560 con 0x7fa9cc0047c0 2023-11-24T08:06:23.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.432+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa9e004e840 con 0x7fa9cc0047c0 2023-11-24T08:06:23.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.433+0000 7fa9d5ffb640 1 -- 172.21.15.107:0/3688472006 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa9dc057b80 con 0x7fa9cc0047c0 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 msgr2=0x7fa9c406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 0x7fa9c406d330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0057a0 tx=0x7fa9d0032000 comp rx=0 tx=0).stop 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 msgr2=0x7fa9e0147360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc001200 tx=0x7fa9dc00eb90 comp rx=0 tx=0).stop 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 shutdown_connections 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa9c406ae70 0x7fa9c406d330 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e00cf4b0 0x7fa9e0158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e00ce110 0x7fa9e00cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 --2- 172.21.15.107:0/3688472006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9cc0047c0 0x7fa9e0147360 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 >> 172.21.15.107:0/3688472006 conn(0x7fa9e006a500 msgr2=0x7fa9e00b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 shutdown_connections 2023-11-24T08:06:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.435+0000 7fa9e60d1640 1 -- 172.21.15.107:0/3688472006 wait complete. 2023-11-24T08:06:23.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol7918 --mode=777 2023-11-24T08:06:23.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.672+0000 7f77dcd4e640 1 Processor -- start 2023-11-24T08:06:23.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.672+0000 7f77dcd4e640 1 -- start start 2023-11-24T08:06:23.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77dcd4e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d80b0390 0x7f77d80999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77dcd4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 0x7f77d8099f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77dcd4e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d80b3a30 con 0x7f77d806c4f0 2023-11-24T08:06:23.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77dcd4e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d80d0740 con 0x7f77d80b0390 2023-11-24T08:06:23.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77dcd4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80cf6a0 con 0x7f77d806b610 2023-11-24T08:06:23.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77d77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d80b0390 0x7f77d80999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.674+0000 7f77d77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d80b0390 0x7f77d80999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52668/0 (socket says 172.21.15.107:52668) 2023-11-24T08:06:23.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77d77fe640 1 -- 172.21.15.107:0/3500929913 learned_addr learned my addr 172.21.15.107:0/3500929913 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:23.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77ceffd640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 0x7f77d8099f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77d77fe640 1 -- 172.21.15.107:0/3500929913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 msgr2=0x7f77d8099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77d77fe640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 0x7f77d8099f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77d77fe640 1 -- 172.21.15.107:0/3500929913 >> v1:172.21.15.120:6789/0 conn(0x7f77d806c4f0 legacy=0x7f77d815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77d77fe640 1 -- 172.21.15.107:0/3500929913 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d80cf980 con 0x7f77d80b0390 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.675+0000 7f77ceffd640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 0x7f77d8099f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:23.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.677+0000 7f77d77fe640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d80b0390 0x7f77d80999d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009e50 tx=0x7f77c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e2bd72e696a79 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.677+0000 7f77d6ffd640 1 -- 172.21.15.107:0/3500929913 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c802c9e0 con 0x7f77d80b0390 2023-11-24T08:06:23.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.677+0000 7f77d6ffd640 1 -- 172.21.15.107:0/3500929913 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c802cb40 con 0x7f77d80b0390 2023-11-24T08:06:23.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.677+0000 7f77d6ffd640 1 -- 172.21.15.107:0/3500929913 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c80343f0 con 0x7f77d80b0390 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 -- 172.21.15.107:0/3500929913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d80b0390 msgr2=0x7f77d80999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d80b0390 0x7f77d80999d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009e50 tx=0x7f77c802be40 comp rx=0 tx=0).stop 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 -- 172.21.15.107:0/3500929913 shutdown_connections 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d806b610 0x7f77d8099f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/3500929913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d80b0390 0x7f77d80999d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 -- 172.21.15.107:0/3500929913 >> 172.21.15.107:0/3500929913 conn(0x7f77d80b2ae0 msgr2=0x7f77d80b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:23.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 -- 172.21.15.107:0/3500929913 shutdown_connections 2023-11-24T08:06:23.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.678+0000 7f77dcd4e640 1 -- 172.21.15.107:0/3500929913 wait complete. 2023-11-24T08:06:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.680+0000 7f77dcd4e640 1 Processor -- start 2023-11-24T08:06:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.680+0000 7f77dcd4e640 1 -- start start 2023-11-24T08:06:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77d806c4f0 0x7f77d80b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d80b0390 0x7f77d81e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81dc440 con 0x7f77d806b610 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80b5ac0 con 0x7f77d806c4f0 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77dcd4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81e5d50 con 0x7f77d80b0390 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.681+0000 7f77d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52670/0 (socket says 172.21.15.107:52670) 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 -- 172.21.15.107:0/775161531 learned_addr learned my addr 172.21.15.107:0/775161531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 -- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d80b0390 msgr2=0x7f77d81e6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d80b0390 0x7f77d81e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 -- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77d806c4f0 msgr2=0x7f77d80b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77d806c4f0 0x7f77d80b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 -- 172.21.15.107:0/775161531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c8009b30 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d77fe640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009e50 tx=0x7f77c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c803e450 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c803e5b0 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.682+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c803e960 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.683+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d81e6030 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.683+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d81e65f0 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.684+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d804e840 con 0x7f77d806b610 2023-11-24T08:06:23.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.685+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f77c803eac0 con 0x7f77d806b610 2023-11-24T08:06:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.686+0000 7f77d4ff9640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 0x7f77b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.686+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77c80ac490 con 0x7f77d806b610 2023-11-24T08:06:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.686+0000 7f77ceffd640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 0x7f77b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.687+0000 7f77ceffd640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 0x7f77b806d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f77c0006fd0 tx=0x7f77c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.690+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77c80775d0 con 0x7f77d806b610 2023-11-24T08:06:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.905+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol7918", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f77d806b110 con 0x7f77b806ae40 2023-11-24T08:06:23.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:23.913+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:23.916 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:23.913+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:23.916 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:23.913+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:23.916 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:23.914+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:23.916 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:23.914+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:23.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.952+0000 7f77d4ff9640 1 -- 172.21.15.107:0/775161531 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f77d806b110 con 0x7f77b806ae40 2023-11-24T08:06:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 msgr2=0x7f77b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 0x7f77b806d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f77c0006fd0 tx=0x7f77c0008040 comp rx=0 tx=0).stop 2023-11-24T08:06:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 msgr2=0x7f77d80b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009e50 tx=0x7f77c8004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 shutdown_connections 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f77b806ae40 0x7f77b806d300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77d80b0390 0x7f77d81e6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77d806c4f0 0x7f77d80b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 --2- 172.21.15.107:0/775161531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d806b610 0x7f77d80b5430 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 >> 172.21.15.107:0/775161531 conn(0x7f77d80b2ae0 msgr2=0x7f77d80b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 shutdown_connections 2023-11-24T08:06:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:23.955+0000 7f77dcd4e640 1 -- 172.21.15.107:0/775161531 wait complete. 2023-11-24T08:06:23.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol7918 2023-11-24T08:06:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.173+0000 7fb1e3397640 1 Processor -- start 2023-11-24T08:06:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.173+0000 7fb1e3397640 1 -- start start 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e3397640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1dc072160 0x7fb1dc15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e3397640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e3397640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc15ec90 con 0x7fb1dc06dc40 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e3397640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc06e0f0 con 0x7fb1dc072160 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e3397640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0cd530 con 0x7fb1dc0c49f0 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.175+0000 7fb1e1133640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1dc072160 0x7fb1dc15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1dc072160 0x7fb1dc15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52708/0 (socket says 172.21.15.107:52708) 2023-11-24T08:06:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 -- 172.21.15.107:0/1742177837 learned_addr learned my addr 172.21.15.107:0/1742177837 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e0932640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 -- 172.21.15.107:0/1742177837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 msgr2=0x7fb1dc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc06d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 -- 172.21.15.107:0/1742177837 >> v1:172.21.15.120:6789/0 conn(0x7fb1dc06dc40 legacy=0x7fb1dc0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.176+0000 7fb1e1133640 1 -- 172.21.15.107:0/1742177837 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0c4ed0 con 0x7fb1dc072160 2023-11-24T08:06:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.177+0000 7fb1e1133640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc072160 0x7fb1dc15e750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009870 tx=0x7fb1d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921abe1beb05a981 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.177+0000 7fb1d3fff640 1 -- 172.21.15.107:0/1742177837 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8030840 con 0x7fb1dc072160 2023-11-24T08:06:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.177+0000 7fb1d3fff640 1 -- 172.21.15.107:0/1742177837 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d80309a0 con 0x7fb1dc072160 2023-11-24T08:06:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.177+0000 7fb1d3fff640 1 -- 172.21.15.107:0/1742177837 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8030d30 con 0x7fb1dc072160 2023-11-24T08:06:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 -- 172.21.15.107:0/1742177837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc072160 msgr2=0x7fb1dc15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc072160 0x7fb1dc15e750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009870 tx=0x7fb1d802b980 comp rx=0 tx=0).stop 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 -- 172.21.15.107:0/1742177837 shutdown_connections 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 --2- 172.21.15.107:0/1742177837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc072160 0x7fb1dc15e750 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 -- 172.21.15.107:0/1742177837 >> 172.21.15.107:0/1742177837 conn(0x7fb1dc0b0c50 msgr2=0x7fb1dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 -- 172.21.15.107:0/1742177837 shutdown_connections 2023-11-24T08:06:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.178+0000 7fb1e3397640 1 -- 172.21.15.107:0/1742177837 wait complete. 2023-11-24T08:06:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.180+0000 7fb1e3397640 1 Processor -- start 2023-11-24T08:06:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.180+0000 7fb1e3397640 1 -- start start 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc072160 0x7fb1dc0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0ae320 con 0x7fb1dc06dc40 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc099fb0 con 0x7fb1dc072160 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e3397640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc09a290 con 0x7fb1dc0c49f0 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e1133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e1133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52716/0 (socket says 172.21.15.107:52716) 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.181+0000 7fb1e1133640 1 -- 172.21.15.107:0/3298046408 learned_addr learned my addr 172.21.15.107:0/3298046408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e0932640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc072160 0x7fb1dc0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 -- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 msgr2=0x7fb1dc151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 -- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc072160 msgr2=0x7fb1dc0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc072160 0x7fb1dc0b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 -- 172.21.15.107:0/3298046408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80094d0 con 0x7fb1dc06dc40 2023-11-24T08:06:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e1133640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8000c00 tx=0x7fb1d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8039c90 con 0x7fb1dc06dc40 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8039df0 con 0x7fb1dc06dc40 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc09a570 con 0x7fb1dc06dc40 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.182+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8043640 con 0x7fb1dc06dc40 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.183+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc152070 con 0x7fb1dc06dc40 2023-11-24T08:06:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.184+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc04e8b0 con 0x7fb1dc06dc40 2023-11-24T08:06:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.185+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb1d80437a0 con 0x7fb1dc06dc40 2023-11-24T08:06:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.186+0000 7fb1d1ffb640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 0x7fb1c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.186+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1d803e070 con 0x7fb1dc06dc40 2023-11-24T08:06:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.186+0000 7fb1e0932640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 0x7fb1c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.187+0000 7fb1e0932640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 0x7fb1c006d300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc006fd0 tx=0x7fb1cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.191+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1d80777b0 con 0x7fb1dc06dc40 2023-11-24T08:06:24.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.409+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol7918", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1dc0c5770 con 0x7fb1c006ae40 2023-11-24T08:06:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.417+0000 7fb1d1ffb640 1 -- 172.21.15.107:0/3298046408 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb1cc0094a0 con 0x7fb1c006ae40 2023-11-24T08:06:24.419 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456 2023-11-24T08:06:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.419+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 msgr2=0x7fb1c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.419+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 0x7fb1c006d300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc006fd0 tx=0x7fb1cc008040 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.419+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 msgr2=0x7fb1dc0999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.419+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8000c00 tx=0x7fb1d8004730 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 shutdown_connections 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1c006ae40 0x7fb1c006d300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1dc0c49f0 0x7fb1dc151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc072160 0x7fb1dc0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 --2- 172.21.15.107:0/3298046408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1dc06dc40 0x7fb1dc0999d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 >> 172.21.15.107:0/3298046408 conn(0x7fb1dc0b0c50 msgr2=0x7fb1dc0cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 shutdown_connections 2023-11-24T08:06:24.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:24.420+0000 7fb1e3397640 1 -- 172.21.15.107:0/3298046408 wait complete. 2023-11-24T08:06:24.439 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol7918 with 200 files each 1MB size under directory volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456 2023-11-24T08:06:24.439 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:24.439 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:24.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:24.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364128 s, 28.8 MB/s 2023-11-24T08:06:24.595 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:24.595 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:24.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:24.651 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332057 s, 31.6 MB/s 2023-11-24T08:06:24.741 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:24.741 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:24.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:24.795 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:24.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:24.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:24.924 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294266 s, 35.6 MB/s 2023-11-24T08:06:24.925 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:24.925 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:24.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:24.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207358 s, 50.6 MB/s 2023-11-24T08:06:25.059 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.059 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.205 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.205 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.205 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352509 s, 29.7 MB/s 2023-11-24T08:06:25.207 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.207 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.264 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265571 s, 39.5 MB/s 2023-11-24T08:06:25.347 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.347 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243769 s, 43.0 MB/s 2023-11-24T08:06:25.478 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.479 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.531 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.657 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.657 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253624 s, 41.3 MB/s 2023-11-24T08:06:25.658 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.658 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.716 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023652 s, 44.3 MB/s 2023-11-24T08:06:25.798 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.799 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:25.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:25.847 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284209 s, 36.9 MB/s 2023-11-24T08:06:25.976 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:25.977 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.028 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247629 s, 42.3 MB/s 2023-11-24T08:06:26.103 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.103 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.159 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193765 s, 54.1 MB/s 2023-11-24T08:06:26.231 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.231 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.287 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.359 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.359 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.359 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191643 s, 54.7 MB/s 2023-11-24T08:06:26.361 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.361 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.417 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.490 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204829 s, 51.2 MB/s 2023-11-24T08:06:26.492 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.492 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.639 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.639 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.639 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369101 s, 28.4 MB/s 2023-11-24T08:06:26.641 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.641 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.701 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.779 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.779 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.780 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247294 s, 42.4 MB/s 2023-11-24T08:06:26.781 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.781 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.842 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:26.918 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:26.918 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023173 s, 45.2 MB/s 2023-11-24T08:06:26.919 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:26.919 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:26.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:26.975 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.046 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.046 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.047 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019706 s, 53.2 MB/s 2023-11-24T08:06:27.048 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.048 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.105 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.181 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.181 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.181 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227484 s, 46.1 MB/s 2023-11-24T08:06:27.183 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.183 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.242 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287908 s, 36.4 MB/s 2023-11-24T08:06:27.326 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.326 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.383 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332916 s, 31.5 MB/s 2023-11-24T08:06:27.469 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.469 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.526 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201291 s, 52.1 MB/s 2023-11-24T08:06:27.600 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.601 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.651 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255268 s, 41.1 MB/s 2023-11-24T08:06:27.776 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.777 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.833 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160387 s, 65.4 MB/s 2023-11-24T08:06:27.904 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:27.904 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:27.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:27.958 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.080 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205188 s, 51.1 MB/s 2023-11-24T08:06:28.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.082 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.140 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.221 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273023 s, 38.4 MB/s 2023-11-24T08:06:28.223 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.223 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.277 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.356 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.356 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.357 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262277 s, 40.0 MB/s 2023-11-24T08:06:28.358 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.358 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250517 s, 41.9 MB/s 2023-11-24T08:06:28.495 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.495 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.671 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.671 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.671 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236291 s, 44.4 MB/s 2023-11-24T08:06:28.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.673 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.728 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.807 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.807 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.807 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280361 s, 37.4 MB/s 2023-11-24T08:06:28.809 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.809 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:28.937 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226956 s, 46.2 MB/s 2023-11-24T08:06:28.938 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:28.938 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:28.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:28.993 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236114 s, 44.4 MB/s 2023-11-24T08:06:29.069 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.070 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.126 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235069 s, 44.6 MB/s 2023-11-24T08:06:29.206 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.206 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.258 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.384 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.384 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.384 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262852 s, 39.9 MB/s 2023-11-24T08:06:29.386 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.386 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189 s, 55.5 MB/s 2023-11-24T08:06:29.516 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.516 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.570 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.647 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.647 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.648 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235715 s, 44.5 MB/s 2023-11-24T08:06:29.649 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.649 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.706 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.788 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259191 s, 40.5 MB/s 2023-11-24T08:06:29.789 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.789 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.844 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270099 s, 38.8 MB/s 2023-11-24T08:06:29.926 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:29.926 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:29.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:29.981 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.061 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.061 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.061 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244229 s, 42.9 MB/s 2023-11-24T08:06:30.062 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.063 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.119 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.195 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.195 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.195 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238298 s, 44.0 MB/s 2023-11-24T08:06:30.197 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.197 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.253 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.330 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.330 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.330 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249766 s, 42.0 MB/s 2023-11-24T08:06:30.332 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.332 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.390 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.473 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.473 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234797 s, 44.7 MB/s 2023-11-24T08:06:30.475 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.475 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.534 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242306 s, 43.3 MB/s 2023-11-24T08:06:30.615 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.615 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.673 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.750 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.751 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.751 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023755 s, 44.1 MB/s 2023-11-24T08:06:30.752 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.753 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.808 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:30.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:30.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:30.887 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220667 s, 47.5 MB/s 2023-11-24T08:06:30.888 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:30.888 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:30.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:30.945 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.024 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240114 s, 43.7 MB/s 2023-11-24T08:06:31.026 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.026 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.085 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.164 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238706 s, 43.9 MB/s 2023-11-24T08:06:31.165 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.165 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.304 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.305 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.305 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250836 s, 41.8 MB/s 2023-11-24T08:06:31.306 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.306 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.362 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027798 s, 37.7 MB/s 2023-11-24T08:06:31.443 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.443 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.502 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197899 s, 53.0 MB/s 2023-11-24T08:06:31.578 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.578 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279757 s, 37.5 MB/s 2023-11-24T08:06:31.715 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.715 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.771 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.849 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023328 s, 44.9 MB/s 2023-11-24T08:06:31.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:31.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:31.907 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213863 s, 49.0 MB/s 2023-11-24T08:06:31.987 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:31.987 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:32.123 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:32.123 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248711 s, 42.2 MB/s 2023-11-24T08:06:32.124 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:32.124 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237583 s, 44.1 MB/s 2023-11-24T08:06:32.256 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:32.256 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.309 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.386 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:32.387 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:32.387 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232305 s, 45.1 MB/s 2023-11-24T08:06:32.388 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:32.388 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.446 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.519 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:32.520 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:32.520 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197994 s, 53.0 MB/s 2023-11-24T08:06:32.521 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:32.521 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02609 s, 40.2 MB/s 2023-11-24T08:06:32.659 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:32.659 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:32.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:32.717 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:32.795 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223212 s, 47.0 MB/s 2023-11-24T08:06:33.021 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.021 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.084 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260938 s, 40.2 MB/s 2023-11-24T08:06:33.168 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.169 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266345 s, 39.4 MB/s 2023-11-24T08:06:33.308 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.308 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.365 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.444 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.444 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245503 s, 42.7 MB/s 2023-11-24T08:06:33.445 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.445 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.502 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190554 s, 55.0 MB/s 2023-11-24T08:06:33.573 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.573 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218311 s, 48.0 MB/s 2023-11-24T08:06:33.702 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.702 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.758 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.64 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.832 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245118 s, 42.8 MB/s 2023-11-24T08:06:33.833 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.834 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:33.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:33.890 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.65 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:33.968 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246744 s, 42.5 MB/s 2023-11-24T08:06:33.969 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:33.969 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.66 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187456 s, 55.9 MB/s 2023-11-24T08:06:34.105 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.105 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.161 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.67 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.238 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.238 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.238 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227243 s, 46.1 MB/s 2023-11-24T08:06:34.240 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.240 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.290 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.68 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186425 s, 56.2 MB/s 2023-11-24T08:06:34.362 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.362 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.69 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212715 s, 49.3 MB/s 2023-11-24T08:06:34.537 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.537 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.70 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194813 s, 53.8 MB/s 2023-11-24T08:06:34.669 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.670 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.71 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02071 s, 50.6 MB/s 2023-11-24T08:06:34.796 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.797 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.72 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241363 s, 43.4 MB/s 2023-11-24T08:06:34.930 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:34.930 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:34.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:34.983 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.73 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242243 s, 43.3 MB/s 2023-11-24T08:06:35.061 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.061 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.117 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.74 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.197 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.197 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.197 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269617 s, 38.9 MB/s 2023-11-24T08:06:35.199 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.199 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.75 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.375 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.376 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.376 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253033 s, 41.4 MB/s 2023-11-24T08:06:35.377 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.377 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.433 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.76 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023913 s, 43.8 MB/s 2023-11-24T08:06:35.510 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.510 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.563 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.77 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.643 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.644 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.644 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253147 s, 41.4 MB/s 2023-11-24T08:06:35.645 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.645 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.78 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.772 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191843 s, 54.7 MB/s 2023-11-24T08:06:35.774 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.774 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.831 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.79 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277017 s, 37.9 MB/s 2023-11-24T08:06:35.909 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:35.909 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:35.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:35.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.80 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223015 s, 47.0 MB/s 2023-11-24T08:06:36.046 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.046 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.102 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.81 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.177 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238331 s, 44.0 MB/s 2023-11-24T08:06:36.178 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.178 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.82 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023153 s, 45.3 MB/s 2023-11-24T08:06:36.316 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.316 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.83 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.452 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.452 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.452 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261353 s, 40.1 MB/s 2023-11-24T08:06:36.454 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.454 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.508 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.84 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.585 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.585 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.585 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216538 s, 48.4 MB/s 2023-11-24T08:06:36.587 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.587 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.643 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.85 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228686 s, 45.9 MB/s 2023-11-24T08:06:36.723 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.723 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.86 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.865 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.865 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.865 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290172 s, 36.1 MB/s 2023-11-24T08:06:36.866 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.867 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:36.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:36.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.87 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201595 s, 52.0 MB/s 2023-11-24T08:06:36.997 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:36.998 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.88 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232044 s, 45.2 MB/s 2023-11-24T08:06:37.128 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.129 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.185 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.89 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.260 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235172 s, 44.6 MB/s 2023-11-24T08:06:37.262 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.262 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.90 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223219 s, 47.0 MB/s 2023-11-24T08:06:37.396 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.396 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.91 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187537 s, 55.9 MB/s 2023-11-24T08:06:37.521 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.521 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.574 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.92 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214775 s, 48.8 MB/s 2023-11-24T08:06:37.651 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.651 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.93 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270264 s, 38.8 MB/s 2023-11-24T08:06:37.790 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.790 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.847 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.94 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248041 s, 42.3 MB/s 2023-11-24T08:06:37.929 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:37.929 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:37.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:37.986 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.95 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.058 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.058 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.058 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200681 s, 52.3 MB/s 2023-11-24T08:06:38.060 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.060 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.96 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.191 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.191 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.192 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202848 s, 51.7 MB/s 2023-11-24T08:06:38.193 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.194 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.97 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.335 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.335 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.335 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027248 s, 38.5 MB/s 2023-11-24T08:06:38.337 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.337 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.395 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.98 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258336 s, 40.6 MB/s 2023-11-24T08:06:38.479 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.479 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.99 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248477 s, 42.2 MB/s 2023-11-24T08:06:38.620 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.621 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.100 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203056 s, 51.6 MB/s 2023-11-24T08:06:38.799 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.799 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:38.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:38.858 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.101 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:38.940 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:38.940 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:38.941 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242538 s, 43.2 MB/s 2023-11-24T08:06:38.942 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:38.943 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.102 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223045 s, 47.0 MB/s 2023-11-24T08:06:39.076 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.077 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.134 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.103 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273922 s, 38.3 MB/s 2023-11-24T08:06:39.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:06:39.219 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.219 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:06:39.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.278 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.104 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.401 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264841 s, 39.6 MB/s 2023-11-24T08:06:39.403 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.403 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.461 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.105 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.583 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217984 s, 48.1 MB/s 2023-11-24T08:06:39.585 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.585 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.640 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.106 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210997 s, 49.7 MB/s 2023-11-24T08:06:39.716 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.716 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.770 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.107 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271994 s, 38.6 MB/s 2023-11-24T08:06:39.899 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:39.899 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:39.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:39.957 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.108 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206218 s, 50.8 MB/s 2023-11-24T08:06:40.036 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.036 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.093 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.109 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022943 s, 45.7 MB/s 2023-11-24T08:06:40.171 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.171 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.110 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.307 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.307 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.307 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266098 s, 39.4 MB/s 2023-11-24T08:06:40.309 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.309 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.368 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.111 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241998 s, 43.3 MB/s 2023-11-24T08:06:40.446 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.446 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.112 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.622 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.623 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.623 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232261 s, 45.1 MB/s 2023-11-24T08:06:40.624 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.625 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.683 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.113 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.758 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.758 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.759 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227291 s, 46.1 MB/s 2023-11-24T08:06:40.760 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.760 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.819 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.114 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:40.904 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:40.905 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:40.905 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254455 s, 41.2 MB/s 2023-11-24T08:06:40.906 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:40.906 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:40.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:40.965 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.115 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.044 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.045 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.045 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226363 s, 46.3 MB/s 2023-11-24T08:06:41.046 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.046 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.102 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.116 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.180 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.180 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.180 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239059 s, 43.9 MB/s 2023-11-24T08:06:41.182 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.182 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.236 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.117 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238229 s, 44.0 MB/s 2023-11-24T08:06:41.315 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.315 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.374 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.118 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243729 s, 43.0 MB/s 2023-11-24T08:06:41.458 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.458 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.517 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.119 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024293 s, 43.2 MB/s 2023-11-24T08:06:41.596 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.597 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.656 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.120 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.728 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.729 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.729 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206455 s, 50.8 MB/s 2023-11-24T08:06:41.730 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.730 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.121 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.858 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.859 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.859 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241726 s, 43.4 MB/s 2023-11-24T08:06:41.860 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:41.860 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:41.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:41.919 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.122 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221551 s, 47.3 MB/s 2023-11-24T08:06:42.000 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.000 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.123 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.133 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.133 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228087 s, 46.0 MB/s 2023-11-24T08:06:42.134 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.134 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.191 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.124 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022687 s, 46.2 MB/s 2023-11-24T08:06:42.267 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.268 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.125 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024814 s, 42.3 MB/s 2023-11-24T08:06:42.403 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.403 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.457 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.126 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.534 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.534 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235939 s, 44.4 MB/s 2023-11-24T08:06:42.535 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.535 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.586 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.127 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.660 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.661 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.661 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230282 s, 45.5 MB/s 2023-11-24T08:06:42.662 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.662 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.714 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.128 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217736 s, 48.2 MB/s 2023-11-24T08:06:42.786 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.786 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.845 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.129 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229823 s, 45.6 MB/s 2023-11-24T08:06:42.923 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:42.923 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:42.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:42.977 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.130 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208628 s, 50.3 MB/s 2023-11-24T08:06:43.050 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.050 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.131 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.227 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.228 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.228 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261449 s, 40.1 MB/s 2023-11-24T08:06:43.229 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.229 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.132 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.363 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.363 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.364 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021389 s, 49.0 MB/s 2023-11-24T08:06:43.365 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.365 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.421 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.133 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262748 s, 39.9 MB/s 2023-11-24T08:06:43.502 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.502 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.559 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.134 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200218 s, 52.4 MB/s 2023-11-24T08:06:43.629 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.629 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.135 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218069 s, 48.1 MB/s 2023-11-24T08:06:43.762 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.762 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.136 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:43.885 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:43.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:43.886 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157794 s, 66.5 MB/s 2023-11-24T08:06:43.887 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:43.887 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:43.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:43.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.137 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.026 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.027 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.027 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030501 s, 34.4 MB/s 2023-11-24T08:06:44.028 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.029 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.082 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.138 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254677 s, 41.2 MB/s 2023-11-24T08:06:44.160 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.161 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.217 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.139 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237532 s, 44.1 MB/s 2023-11-24T08:06:44.290 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.290 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.140 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.416 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.417 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.417 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216107 s, 48.5 MB/s 2023-11-24T08:06:44.418 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.418 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.141 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250717 s, 41.8 MB/s 2023-11-24T08:06:44.547 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.547 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.142 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.671 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200339 s, 52.3 MB/s 2023-11-24T08:06:44.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.672 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.726 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.143 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.801 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.801 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.802 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021996 s, 47.7 MB/s 2023-11-24T08:06:44.803 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.803 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.858 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.144 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198253 s, 52.9 MB/s 2023-11-24T08:06:44.928 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:44.929 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:44.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:44.980 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.145 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.107 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.107 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.107 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271576 s, 38.6 MB/s 2023-11-24T08:06:45.108 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.109 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.146 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236198 s, 44.4 MB/s 2023-11-24T08:06:45.247 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.247 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.303 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.147 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239093 s, 43.9 MB/s 2023-11-24T08:06:45.380 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.380 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.437 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.148 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.510 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.510 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.510 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220437 s, 47.6 MB/s 2023-11-24T08:06:45.511 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.512 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.567 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.149 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228147 s, 46.0 MB/s 2023-11-24T08:06:45.643 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.643 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.693 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.150 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214439 s, 48.9 MB/s 2023-11-24T08:06:45.818 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.819 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:45.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:45.872 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.151 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:45.945 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:45.945 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235854 s, 44.5 MB/s 2023-11-24T08:06:45.946 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:45.946 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.152 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217571 s, 48.2 MB/s 2023-11-24T08:06:46.084 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.084 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.153 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.210 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.211 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.211 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238325 s, 44.0 MB/s 2023-11-24T08:06:46.212 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.212 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.154 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.352 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.352 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.352 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265541 s, 39.5 MB/s 2023-11-24T08:06:46.353 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.353 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.155 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.495 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.495 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.495 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251062 s, 41.8 MB/s 2023-11-24T08:06:46.496 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.497 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.156 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268435 s, 39.1 MB/s 2023-11-24T08:06:46.678 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.678 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.157 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183436 s, 57.2 MB/s 2023-11-24T08:06:46.812 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.812 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:46.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:46.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.158 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:46.946 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:46.946 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:46.946 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258098 s, 40.6 MB/s 2023-11-24T08:06:46.948 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:46.948 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.006 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.159 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204698 s, 51.2 MB/s 2023-11-24T08:06:47.079 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.079 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.134 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.160 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.208 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023606 s, 44.4 MB/s 2023-11-24T08:06:47.210 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.210 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.161 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256936 s, 40.8 MB/s 2023-11-24T08:06:47.353 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.353 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.408 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.162 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.485 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.486 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227709 s, 46.0 MB/s 2023-11-24T08:06:47.487 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.487 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.163 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.615 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.616 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.616 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022697 s, 46.2 MB/s 2023-11-24T08:06:47.617 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.617 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.677 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.164 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228875 s, 45.8 MB/s 2023-11-24T08:06:47.758 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.758 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.165 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181372 s, 57.8 MB/s 2023-11-24T08:06:47.890 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:47.890 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:47.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:47.947 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.166 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.018 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.019 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.019 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201089 s, 52.1 MB/s 2023-11-24T08:06:48.020 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.020 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.167 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182826 s, 57.4 MB/s 2023-11-24T08:06:48.148 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.148 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.168 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147216 s, 71.2 MB/s 2023-11-24T08:06:48.273 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.273 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.327 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.169 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.405 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.405 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.405 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258598 s, 40.5 MB/s 2023-11-24T08:06:48.407 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.407 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.462 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.170 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205078 s, 51.1 MB/s 2023-11-24T08:06:48.536 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.537 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.592 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.171 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.669 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267278 s, 39.2 MB/s 2023-11-24T08:06:48.670 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.670 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.726 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.172 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206762 s, 50.7 MB/s 2023-11-24T08:06:48.800 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.800 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.173 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:48.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:48.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:48.933 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209044 s, 50.2 MB/s 2023-11-24T08:06:48.934 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:48.935 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:48.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:48.992 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.174 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.069 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208503 s, 50.3 MB/s 2023-11-24T08:06:49.070 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.070 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.129 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.175 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.208 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260072 s, 40.3 MB/s 2023-11-24T08:06:49.209 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.209 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.262 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.176 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.388 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.389 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.389 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274595 s, 38.2 MB/s 2023-11-24T08:06:49.390 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.390 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.446 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.177 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.521 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.522 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218274 s, 48.0 MB/s 2023-11-24T08:06:49.523 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.523 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.178 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.658 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.659 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.659 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253716 s, 41.3 MB/s 2023-11-24T08:06:49.660 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.660 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.719 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.179 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230814 s, 45.4 MB/s 2023-11-24T08:06:49.796 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.796 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.856 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.180 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:49.933 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232228 s, 45.2 MB/s 2023-11-24T08:06:49.935 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:49.935 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:49.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:49.989 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.181 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239466 s, 43.8 MB/s 2023-11-24T08:06:50.069 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.069 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.182 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.211 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250821 s, 41.8 MB/s 2023-11-24T08:06:50.213 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.213 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.272 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.183 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.347 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.347 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.347 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204197 s, 51.4 MB/s 2023-11-24T08:06:50.348 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.348 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.402 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.184 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255143 s, 41.1 MB/s 2023-11-24T08:06:50.531 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.532 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.588 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.185 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.665 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241209 s, 43.5 MB/s 2023-11-24T08:06:50.666 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.666 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.722 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.186 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.799 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023212 s, 45.2 MB/s 2023-11-24T08:06:50.801 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.802 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.860 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.187 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225669 s, 46.5 MB/s 2023-11-24T08:06:50.938 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:50.938 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:50.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:50.998 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.188 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201572 s, 52.0 MB/s 2023-11-24T08:06:51.073 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.073 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.130 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.189 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.207 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.207 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.207 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266158 s, 39.4 MB/s 2023-11-24T08:06:51.209 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.209 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.266 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.190 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249611 s, 42.0 MB/s 2023-11-24T08:06:51.348 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.348 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.407 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.191 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228756 s, 45.8 MB/s 2023-11-24T08:06:51.485 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.485 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.192 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.619 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.619 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.619 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211047 s, 49.7 MB/s 2023-11-24T08:06:51.621 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.621 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.677 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.193 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.750 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.750 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.750 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196007 s, 53.5 MB/s 2023-11-24T08:06:51.752 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.752 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.194 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:51.880 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:51.880 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:51.880 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263073 s, 39.9 MB/s 2023-11-24T08:06:51.881 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:51.882 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:51.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:51.934 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.195 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182391 s, 57.5 MB/s 2023-11-24T08:06:52.003 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:52.004 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:52.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:52.061 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.196 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:52.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:52.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:52.127 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184332 s, 56.9 MB/s 2023-11-24T08:06:52.128 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:52.128 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:52.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:52.176 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.197 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:52.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:52.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:52.293 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179986 s, 58.3 MB/s 2023-11-24T08:06:52.294 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:52.295 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:52.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:52.352 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.198 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243451 s, 43.1 MB/s 2023-11-24T08:06:52.434 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:06:52.434 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:06:52.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:52.492 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7918/d01545bb-17b4-4ba0-b948-a11a0e01e456/subvolume_file.199 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:06:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:06:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:06:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259493 s, 40.4 MB/s 2023-11-24T08:06:52.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol7918 subvol_snap1109 2023-11-24T08:06:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.794+0000 7f9ddd400640 1 Processor -- start 2023-11-24T08:06:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.795+0000 7f9ddd400640 1 -- start start 2023-11-24T08:06:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.796+0000 7f9ddd400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.796+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.796+0000 7f9ddd400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd80cece0 con 0x7f9dd815c7f0 2023-11-24T08:06:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.796+0000 7f9ddd400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd80cfe40 con 0x7f9dd80cf450 2023-11-24T08:06:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.796+0000 7f9ddd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8156010 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.797+0000 7f9dce7fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.797+0000 7f9dd77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.797+0000 7f9dd77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:41984/0 (socket says 172.21.15.107:41984) 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.797+0000 7f9dce7fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:37698/0 (socket says 172.21.15.107:37698) 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.797+0000 7f9dd77fe640 1 -- 172.21.15.107:0/2603651829 learned_addr learned my addr 172.21.15.107:0/2603651829 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.798+0000 7f9dd77fe640 1 -- 172.21.15.107:0/2603651829 >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 msgr2=0x7f9dd80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.798+0000 7f9dd77fe640 1 --2- 172.21.15.107:0/2603651829 >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.798+0000 7f9dd77fe640 1 -- 172.21.15.107:0/2603651829 >> v1:172.21.15.120:6789/0 conn(0x7f9dd815c7f0 legacy=0x7f9dd806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.798+0000 7f9dd77fe640 1 -- 172.21.15.107:0/2603651829 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd806d040 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.798+0000 7f9dce7fc640 1 --2- 172.21.15.107:0/2603651829 >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:06:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.800+0000 7f9dd77fe640 1 --2- 172.21.15.107:0/2603651829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf52e133573596f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.800+0000 7f9dd67fc640 1 -- 172.21.15.107:0/2603651829 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd002cb00 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.800+0000 7f9dd67fc640 1 -- 172.21.15.107:0/2603651829 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd002cc60 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.800+0000 7f9dd67fc640 1 -- 172.21.15.107:0/2603651829 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00343f0 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 -- 172.21.15.107:0/2603651829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 msgr2=0x7f9dd80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 --2- 172.21.15.107:0/2603651829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd002bf60 comp rx=0 tx=0).stop 2023-11-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 -- 172.21.15.107:0/2603651829 shutdown_connections 2023-11-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 --2- 172.21.15.107:0/2603651829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd80ce420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 --2- 172.21.15.107:0/2603651829 >> v2:172.21.15.120:3300/0 conn(0x7f9dd80cf450 0x7f9dd80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 -- 172.21.15.107:0/2603651829 >> 172.21.15.107:0/2603651829 conn(0x7f9dd806a920 msgr2=0x7f9dd806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 -- 172.21.15.107:0/2603651829 shutdown_connections 2023-11-24T08:06:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.801+0000 7f9ddd400640 1 -- 172.21.15.107:0/2603651829 wait complete. 2023-11-24T08:06:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.804+0000 7f9ddd400640 1 Processor -- start 2023-11-24T08:06:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.804+0000 7f9ddd400640 1 -- start start 2023-11-24T08:06:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd81f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd80cf450 0x7f9dd80c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81f78e0 con 0x7f9dd815c7f0 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81f3300 con 0x7f9dd80cf450 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9ddd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81f7320 con 0x7f9dd80cdfb0 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9dd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9dd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53366/0 (socket says 172.21.15.107:53366) 2023-11-24T08:06:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.805+0000 7f9dd77fe640 1 -- 172.21.15.107:0/3698262219 learned_addr learned my addr 172.21.15.107:0/3698262219 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd6ffd640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd81f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dce7fc640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd80cf450 0x7f9dd80c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 -- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 msgr2=0x7f9dd81f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd81f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 -- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd80cf450 msgr2=0x7f9dd80c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd80cf450 0x7f9dd80c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 -- 172.21.15.107:0/3698262219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0009400 con 0x7f9dd815c7f0 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.806+0000 7f9dd77fe640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.807+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0009510 con 0x7f9dd815c7f0 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.807+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd003e450 con 0x7f9dd815c7f0 2023-11-24T08:06:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.807+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd003e800 con 0x7f9dd815c7f0 2023-11-24T08:06:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.807+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0009b30 con 0x7f9dd815c7f0 2023-11-24T08:06:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.807+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd81f3560 con 0x7f9dd815c7f0 2023-11-24T08:06:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.808+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd8067470 con 0x7f9dd815c7f0 2023-11-24T08:06:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.809+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f9dd003e960 con 0x7f9dd815c7f0 2023-11-24T08:06:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.810+0000 7f9dcffff640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 0x7f9db806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.810+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dd00ac390 con 0x7f9dd815c7f0 2023-11-24T08:06:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.810+0000 7f9dd6ffd640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 0x7f9db806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.811+0000 7f9dd6ffd640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 0x7f9db806d390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80afe40 tx=0x7f9dc8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:52.815+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dd0077350 con 0x7f9dd815c7f0 2023-11-24T08:06:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.038+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol7918", "snap_name": "subvol_snap1109", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dd806ef40 con 0x7f9db806aed0 2023-11-24T08:06:53.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.093+0000 7f9dcffff640 1 -- 172.21.15.107:0/3698262219 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9dd806ef40 con 0x7f9db806aed0 2023-11-24T08:06:53.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 msgr2=0x7f9db806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 0x7f9db806d390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80afe40 tx=0x7f9dc8009040 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 msgr2=0x7f9dd81f6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd0004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 shutdown_connections 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9db806aed0 0x7f9db806d390 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd815c7f0 0x7f9dd81f6940 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd80cf450 0x7f9dd80c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 --2- 172.21.15.107:0/3698262219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dd80cdfb0 0x7f9dd81f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 >> 172.21.15.107:0/3698262219 conn(0x7f9dd806a920 msgr2=0x7f9dd806c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 shutdown_connections 2023-11-24T08:06:53.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.097+0000 7f9ddd400640 1 -- 172.21.15.107:0/3698262219 wait complete. 2023-11-24T08:06:53.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 5 2023-11-24T08:06:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.312+0000 7f67b759e640 1 Processor -- start 2023-11-24T08:06:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.312+0000 7f67b759e640 1 -- start start 2023-11-24T08:06:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.313+0000 7f67b759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f67b815b260 0x7f67b806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.313+0000 7f67b759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b815ed20 con 0x7f67b806e580 2023-11-24T08:06:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.313+0000 7f67b759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b815f8a0 con 0x7f67b80b1250 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.313+0000 7f67b759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b815b640 con 0x7f67b815b260 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f67b815b260 0x7f67b806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67add9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f67b806e580 0x7f67b80b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56628/0 (socket says 172.21.15.107:56628) 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67add9b640 1 -- 172.21.15.107:0/719961567 learned_addr learned my addr 172.21.15.107:0/719961567 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67b659c640 1 -- 172.21.15.107:0/719961567 >> v1:172.21.15.120:6789/0 conn(0x7f67b80b1250 legacy=0x7f67b806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67b659c640 1 -- 172.21.15.107:0/719961567 >> v1:172.21.15.107:6789/0 conn(0x7f67b806e580 legacy=0x7f67b80b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.314+0000 7f67b659c640 1 -- 172.21.15.107:0/719961567 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b80abe50 con 0x7f67b815b260 2023-11-24T08:06:53.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.315+0000 7f67b659c640 1 --2- 172.21.15.107:0/719961567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b815b260 0x7f67b806e040 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f67a80097a0 tx=0x7f67a802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83b77884f5e8330 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b5d9b640 1 -- 172.21.15.107:0/719961567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a802f050 con 0x7f67b815b260 2023-11-24T08:06:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b5d9b640 1 -- 172.21.15.107:0/719961567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8031700 con 0x7f67b815b260 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b5d9b640 1 -- 172.21.15.107:0/719961567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8031a90 con 0x7f67b815b260 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 -- 172.21.15.107:0/719961567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b815b260 msgr2=0x7f67b806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 --2- 172.21.15.107:0/719961567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b815b260 0x7f67b806e040 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f67a80097a0 tx=0x7f67a802b7a0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 -- 172.21.15.107:0/719961567 shutdown_connections 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 --2- 172.21.15.107:0/719961567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b815b260 0x7f67b806e040 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 -- 172.21.15.107:0/719961567 >> 172.21.15.107:0/719961567 conn(0x7f67b80cbb40 msgr2=0x7f67b80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:53.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 -- 172.21.15.107:0/719961567 shutdown_connections 2023-11-24T08:06:53.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.316+0000 7f67b759e640 1 -- 172.21.15.107:0/719961567 wait complete. 2023-11-24T08:06:53.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.318+0000 7f67b759e640 1 Processor -- start 2023-11-24T08:06:53.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.318+0000 7f67b759e640 1 -- start start 2023-11-24T08:06:53.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 0x7f67b80999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 0x7f67b80c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b809aea0 con 0x7f67b80b1250 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b809a220 con 0x7f67b815b260 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.319+0000 7f67b759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b809a500 con 0x7f67b806e580 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53396/0 (socket says 172.21.15.107:53396) 2023-11-24T08:06:53.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 -- 172.21.15.107:0/1147314692 learned_addr learned my addr 172.21.15.107:0/1147314692 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67b659c640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 0x7f67b80999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67b6d9d640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 0x7f67b80c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 -- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 msgr2=0x7f67b80999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 0x7f67b80999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 -- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 msgr2=0x7f67b80c2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 0x7f67b80c2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 -- 172.21.15.107:0/1147314692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a8009400 con 0x7f67b80b1250 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67b659c640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 0x7f67b80999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67b6d9d640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 0x7f67b80c2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.320+0000 7f67add9b640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c480 tx=0x7f67a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.321+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0005ce0 con 0x7f67b80b1250 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.321+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0005e40 con 0x7f67b80b1250 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.321+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00072f0 con 0x7f67b80b1250 2023-11-24T08:06:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.321+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b809a7e0 con 0x7f67b80b1250 2023-11-24T08:06:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.321+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b8099f10 con 0x7f67b80b1250 2023-11-24T08:06:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.322+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b804e840 con 0x7f67b80b1250 2023-11-24T08:06:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.323+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f67a0007510 con 0x7f67b80b1250 2023-11-24T08:06:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.324+0000 7f67af7fe640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 0x7f679806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.324+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67a0088bd0 con 0x7f67b80b1250 2023-11-24T08:06:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.324+0000 7f67b659c640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 0x7f679806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.325+0000 7f67b659c640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 0x7f679806d390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f67a8035000 tx=0x7f67a8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.328+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67a0053cd0 con 0x7f67b80b1250 2023-11-24T08:06:53.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.522+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f67b80ce640 con 0x7f67b80b1250 2023-11-24T08:06:53.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.528+0000 7f67af7fe640 1 -- 172.21.15.107:0/1147314692 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v11)=0 v11) v1 ==== 130+0+0 (secure 0 0 0) 0x7f67a0017020 con 0x7f67b80b1250 2023-11-24T08:06:53.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 msgr2=0x7f679806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 0x7f679806d390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f67a8035000 tx=0x7f67a8008030 comp rx=0 tx=0).stop 2023-11-24T08:06:53.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 msgr2=0x7f67b814d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c480 tx=0x7f67a000c940 comp rx=0 tx=0).stop 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 shutdown_connections 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f679806aed0 0x7f679806d390 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67b815b260 0x7f67b80c2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67b80b1250 0x7f67b814d410 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 --2- 172.21.15.107:0/1147314692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67b806e580 0x7f67b80999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 >> 172.21.15.107:0/1147314692 conn(0x7f67b80cbb40 msgr2=0x7f67b80af710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.531+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 shutdown_connections 2023-11-24T08:06:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.532+0000 7f67b759e640 1 -- 172.21.15.107:0/1147314692 wait complete. 2023-11-24T08:06:53.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol7918 subvol_snap1109 subvol_clone9616 2023-11-24T08:06:53.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.769+0000 7fe3fbde5640 1 Processor -- start 2023-11-24T08:06:53.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.769+0000 7fe3fbde5640 1 -- start start 2023-11-24T08:06:53.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fbde5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fbde5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3f40c57e0 0x7fe3f406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fbde5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f415eea0 con 0x7fe3f415b440 2023-11-24T08:06:53.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fbde5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f415b820 con 0x7fe3f40c57e0 2023-11-24T08:06:53.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fbde5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40abf30 con 0x7fe3f40cc0f0 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fa382640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3f40c57e0 0x7fe3f406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3f9b81640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3f415b440 0x7fe3f40cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56634/0 (socket says 172.21.15.107:56634) 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3f9b81640 1 -- 172.21.15.107:0/2740219299 learned_addr learned my addr 172.21.15.107:0/2740219299 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3f9380640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fa382640 1 -- 172.21.15.107:0/2740219299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 msgr2=0x7fe3f40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fa382640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f40c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.771+0000 7fe3fa382640 1 -- 172.21.15.107:0/2740219299 >> v1:172.21.15.107:6789/0 conn(0x7fe3f415b440 legacy=0x7fe3f40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.772+0000 7fe3fa382640 1 -- 172.21.15.107:0/2740219299 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f415f0f0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.772+0000 7fe3f8b7f640 1 -- 172.21.15.107:0/2740219299 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854349708 0 0) 0x7fe3f415eea0 con 0x7fe3f415b440 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fa382640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f406c5f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009d90 tx=0x7fe3ec02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb06d3552e2b384 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3f8b7f640 1 -- 172.21.15.107:0/2740219299 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec02cbc0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3f8b7f640 1 -- 172.21.15.107:0/2740219299 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec02cd20 con 0x7fe3f40c57e0 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3f8b7f640 1 -- 172.21.15.107:0/2740219299 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec034660 con 0x7fe3f40c57e0 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 -- 172.21.15.107:0/2740219299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 msgr2=0x7fe3f406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f406c5f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009d90 tx=0x7fe3ec02be10 comp rx=0 tx=0).stop 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 -- 172.21.15.107:0/2740219299 shutdown_connections 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f406c5f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/2740219299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f40c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.773+0000 7fe3fbde5640 1 -- 172.21.15.107:0/2740219299 >> 172.21.15.107:0/2740219299 conn(0x7fe3f40b0c50 msgr2=0x7fe3f40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.774+0000 7fe3fbde5640 1 -- 172.21.15.107:0/2740219299 shutdown_connections 2023-11-24T08:06:53.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.774+0000 7fe3fbde5640 1 -- 172.21.15.107:0/2740219299 wait complete. 2023-11-24T08:06:53.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.775+0000 7fe3fbde5640 1 Processor -- start 2023-11-24T08:06:53.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.775+0000 7fe3fbde5640 1 -- start start 2023-11-24T08:06:53.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f4156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3f415b440 0x7fe3f40b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40c8f40 con 0x7fe3f40c57e0 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40c7c40 con 0x7fe3f415b440 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.776+0000 7fe3fbde5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f4157140 con 0x7fe3f40cc0f0 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53410/0 (socket says 172.21.15.107:53410) 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 -- 172.21.15.107:0/4282693456 learned_addr learned my addr 172.21.15.107:0/4282693456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9380640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f4156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 -- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 msgr2=0x7fe3f4156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3fa382640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3f415b440 0x7fe3f40b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f4156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 -- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3f415b440 msgr2=0x7fe3f40b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3f415b440 0x7fe3f40b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 -- 172.21.15.107:0/4282693456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec009a70 con 0x7fe3f40c57e0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9380640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f4156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.777+0000 7fe3f9b81640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c430 tx=0x7fe3f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.778+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00074e0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.778+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0007660 con 0x7fe3f40c57e0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.778+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0004a20 con 0x7fe3f40c57e0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.778+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f40c7ea0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.778+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f40c8460 con 0x7fe3f40c57e0 2023-11-24T08:06:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.779+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f404e8b0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.780+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fe3f0004b80 con 0x7fe3f40c57e0 2023-11-24T08:06:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.781+0000 7fe3ea7fc640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 0x7fe3d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.781+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3f00895d0 con 0x7fe3f40c57e0 2023-11-24T08:06:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.781+0000 7fe3f9380640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 0x7fe3d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.782+0000 7fe3f9380640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 0x7fe3d806d390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40b2480 tx=0x7fe3e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:53.785+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3f0054590 con 0x7fe3f40c57e0 2023-11-24T08:06:54.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.009+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol7918", "snap_name": "subvol_snap1109", "target_sub_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3f40c9e90 con 0x7fe3d806aed0 2023-11-24T08:06:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.232+0000 7fe3ea7fc640 1 -- 172.21.15.107:0/4282693456 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe3f40c9e90 con 0x7fe3d806aed0 2023-11-24T08:06:54.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 msgr2=0x7fe3d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 0x7fe3d806d390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40b2480 tx=0x7fe3e4008040 comp rx=0 tx=0).stop 2023-11-24T08:06:54.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 msgr2=0x7fe3f40c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c430 tx=0x7fe3f000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 shutdown_connections 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3d806aed0 0x7fe3d806d390 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3f415b440 0x7fe3f40b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3f40cc0f0 0x7fe3f4156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.235+0000 7fe3fbde5640 1 --2- 172.21.15.107:0/4282693456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40c57e0 0x7fe3f40c7700 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.236+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 >> 172.21.15.107:0/4282693456 conn(0x7fe3f40b0c50 msgr2=0x7fe3f40cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:54.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.236+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 shutdown_connections 2023-11-24T08:06:54.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.236+0000 7fe3fbde5640 1 -- 172.21.15.107:0/4282693456 wait complete. 2023-11-24T08:06:54.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.239+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.239+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.239+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.239+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.241 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.239+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:06:54.259 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir ./volumes/_nogroup/subvol7918/.snap/subvol_snap1109) 2023-11-24T08:06:54.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.260+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.260+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.260+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.260+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:06:54.260+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:06:54.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:06:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.542+0000 7f9d01439640 1 Processor -- start 2023-11-24T08:06:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.542+0000 7f9d01439640 1 -- start start 2023-11-24T08:06:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.544+0000 7f9d01439640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.544+0000 7f9d01439640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9cfc0b0b40 0x7f9cfc0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.544+0000 7f9d01439640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc06d430 con 0x7f9cfc15ded0 2023-11-24T08:06:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.544+0000 7f9d01439640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc15ea70 con 0x7f9cfc0b0b40 2023-11-24T08:06:54.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.544+0000 7f9d01439640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc15ec50 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cf3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9cfc0b0b40 0x7f9cfc0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:42076/0 (socket says 172.21.15.107:42076) 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 -- 172.21.15.107:0/2043998699 learned_addr learned my addr 172.21.15.107:0/2043998699 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 -- 172.21.15.107:0/2043998699 >> v2:172.21.15.120:3300/0 conn(0x7f9cfc0b0b40 msgr2=0x7f9cfc0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 --2- 172.21.15.107:0/2043998699 >> v2:172.21.15.120:3300/0 conn(0x7f9cfc0b0b40 0x7f9cfc0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 -- 172.21.15.107:0/2043998699 >> v1:172.21.15.120:6789/0 conn(0x7f9cfc15ded0 legacy=0x7f9cfc15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.545+0000 7f9cfbfff640 1 -- 172.21.15.107:0/2043998699 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0b14e0 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.547+0000 7f9cfbfff640 1 --2- 172.21.15.107:0/2043998699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009870 tx=0x7f9cec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8380e2a3773d8032 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:54.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.547+0000 7f9cfb7fe640 1 -- 172.21.15.107:0/2043998699 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec030840 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.547+0000 7f9cfb7fe640 1 -- 172.21.15.107:0/2043998699 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec0309a0 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.547+0000 7f9cfb7fe640 1 -- 172.21.15.107:0/2043998699 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec030d30 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 -- 172.21.15.107:0/2043998699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 msgr2=0x7f9cfc06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 --2- 172.21.15.107:0/2043998699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009870 tx=0x7f9cec02b980 comp rx=0 tx=0).stop 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 -- 172.21.15.107:0/2043998699 shutdown_connections 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 --2- 172.21.15.107:0/2043998699 >> v2:172.21.15.120:3300/0 conn(0x7f9cfc0b0b40 0x7f9cfc0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 --2- 172.21.15.107:0/2043998699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc06d8c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 -- 172.21.15.107:0/2043998699 >> 172.21.15.107:0/2043998699 conn(0x7f9cfc0cbb40 msgr2=0x7f9cfc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:54.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 -- 172.21.15.107:0/2043998699 shutdown_connections 2023-11-24T08:06:54.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.548+0000 7f9d01439640 1 -- 172.21.15.107:0/2043998699 wait complete. 2023-11-24T08:06:54.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.550+0000 7f9d01439640 1 Processor -- start 2023-11-24T08:06:54.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.550+0000 7f9d01439640 1 -- start start 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cfc0b0b40 0x7f9cfc0c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c6310 con 0x7f9cfc15ded0 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c65d0 con 0x7f9cfc0b0b40 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d01439640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c68b0 con 0x7f9cfc15a1c0 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53452/0 (socket says 172.21.15.107:53452) 2023-11-24T08:06:54.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 -- 172.21.15.107:0/456265540 learned_addr learned my addr 172.21.15.107:0/456265540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9cf3fff640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9cfbfff640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cfc0b0b40 0x7f9cfc0c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 -- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 msgr2=0x7f9cfc154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 -- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cfc0b0b40 msgr2=0x7f9cfc0c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cfc0b0b40 0x7f9cfc0c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9d00c38640 1 -- 172.21.15.107:0/456265540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0094d0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.552+0000 7f9cf3fff640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9d00c38640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400c6b0 tx=0x7f9cf400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4003b70 con 0x7f9cfc15ded0 2023-11-24T08:06:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4003cf0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc0c6b90 con 0x7f9cfc15ded0 2023-11-24T08:06:54.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4003ea0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.553+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc154bd0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.555+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc0c70b0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.555+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f9cf4005310 con 0x7f9cfc15ded0 2023-11-24T08:06:54.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.556+0000 7f9cf97fa640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 0x7f9cdc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:54.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.556+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cf40894b0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.557+0000 7f9cfbfff640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 0x7f9cdc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:54.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.557+0000 7f9cfbfff640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 0x7f9cdc06d390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9cec036000 tx=0x7f9cec03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:54.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.561+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cfc0c70b0 con 0x7f9cfc15ded0 2023-11-24T08:06:54.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.791+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cfc06d0c0 con 0x7f9cdc06aed0 2023-11-24T08:06:54.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.801+0000 7f9cf97fa640 1 -- 172.21.15.107:0/456265540 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f9cec030b90 con 0x7f9cdc06aed0 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:54.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:54.804 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:06:54.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 msgr2=0x7f9cdc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 0x7f9cdc06d390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9cec036000 tx=0x7f9cec03b040 comp rx=0 tx=0).stop 2023-11-24T08:06:54.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 msgr2=0x7f9cfc158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:54.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400c6b0 tx=0x7f9cf400cb70 comp rx=0 tx=0).stop 2023-11-24T08:06:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 shutdown_connections 2023-11-24T08:06:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9cdc06aed0 0x7f9cdc06d390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cfc15ded0 0x7f9cfc158e60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cfc15a1c0 0x7f9cfc154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.804+0000 7f9d01439640 1 --2- 172.21.15.107:0/456265540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cfc0b0b40 0x7f9cfc0c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.805+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 >> 172.21.15.107:0/456265540 conn(0x7f9cfc0cbb40 msgr2=0x7f9cfc0b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.805+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 shutdown_connections 2023-11-24T08:06:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:54.805+0000 7f9d01439640 1 -- 172.21.15.107:0/456265540 wait complete. 2023-11-24T08:06:55.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:06:56.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.044+0000 7f743359e640 1 Processor -- start 2023-11-24T08:06:56.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.044+0000 7f743359e640 1 -- start start 2023-11-24T08:06:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74340b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74340cb960 con 0x7f74340cb580 2023-11-24T08:06:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7434161e40 con 0x7f74340ca940 2023-11-24T08:06:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7434162120 con 0x7f74340c27a0 2023-11-24T08:06:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f74340cb580 0x7f74340c2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56644/0 (socket says 172.21.15.107:56644) 2023-11-24T08:06:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.046+0000 7f743259c640 1 -- 172.21.15.107:0/3547576949 learned_addr learned my addr 172.21.15.107:0/3547576949 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f7431d9b640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74340b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515988166 0 0) 0x7f74340cb960 con 0x7f74340cb580 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7420003520 con 0x7f74340cb580 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330549559 0 0) 0x7f7420003520 con 0x7f74340cb580 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 msgr2=0x7f74340b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74340b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 >> v1:172.21.15.120:6789/0 conn(0x7f74340ca940 legacy=0x7f74340b13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7434162400 con 0x7f74340cb580 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7424001670 con 0x7f74340cb580 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 >> v1:172.21.15.107:6789/0 conn(0x7f74340cb580 legacy=0x7f74340c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74200039a0 0x7f7420003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7420004d00 0x7f7420005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7434162120 con 0x7f7420004d00 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742000bb10 con 0x7f74200039a0 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.047+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742000bcd0 con 0x7f74340c27a0 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7420004d00 0x7f7420005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f743259c640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74200039a0 0x7f7420003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7431d9b640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 -- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 msgr2=0x7f74200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340c27a0 0x7f74200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 -- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74200039a0 msgr2=0x7f7420003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:56.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74200039a0 0x7f7420003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.048+0000 7f7432d9d640 1 -- 172.21.15.107:0/3547576949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420003520 con 0x7f7420004d00 2023-11-24T08:06:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.050+0000 7f7432d9d640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7420004d00 0x7f7420005180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f742c001e50 tx=0x7f74200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c80ba427c9fd21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c003b70 con 0x7f7420004d00 2023-11-24T08:06:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c003cd0 con 0x7f7420004d00 2023-11-24T08:06:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f743159a640 1 -- 172.21.15.107:0/3547576949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c005310 con 0x7f7420004d00 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f743359e640 1 -- 172.21.15.107:0/3547576949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7420004d00 msgr2=0x7f7420005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f743359e640 1 --2- 172.21.15.107:0/3547576949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7420004d00 0x7f7420005180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f742c001e50 tx=0x7f74200057f0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.051+0000 7f7432d9d640 1 -- 172.21.15.107:0/3547576949 reap_dead start 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.052+0000 7f743359e640 1 -- 172.21.15.107:0/3547576949 shutdown_connections 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.052+0000 7f743359e640 1 -- 172.21.15.107:0/3547576949 >> 172.21.15.107:0/3547576949 conn(0x7f74340cbb40 msgr2=0x7f74340cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:56.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.052+0000 7f743359e640 1 -- 172.21.15.107:0/3547576949 shutdown_connections 2023-11-24T08:06:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.052+0000 7f743359e640 1 -- 172.21.15.107:0/3547576949 wait complete. 2023-11-24T08:06:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.054+0000 7f743359e640 1 Processor -- start 2023-11-24T08:06:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.054+0000 7f743359e640 1 -- start start 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340b38b0 0x7f74340c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74340b0800 0x7f74341581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340c7ac0 con 0x7f74340b3c90 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340c7da0 con 0x7f74340b0800 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f743359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74340c8080 con 0x7f74340b38b0 2023-11-24T08:06:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f7431d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f7431d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53470/0 (socket says 172.21.15.107:53470) 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.055+0000 7f7431d9b640 1 -- 172.21.15.107:0/3791129683 learned_addr learned my addr 172.21.15.107:0/3791129683 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f743259c640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340b38b0 0x7f74340c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 -- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340b38b0 msgr2=0x7f74340c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340b38b0 0x7f74340c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 -- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74340b0800 msgr2=0x7f74341581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74340b0800 0x7f74341581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 -- 172.21.15.107:0/3791129683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340c8360 con 0x7f74340b3c90 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f7431d9b640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f741c00f580 tx=0x7f741c00fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c014870 con 0x7f74340b3c90 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0149d0 con 0x7f74340b3c90 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c014d80 con 0x7f74340b3c90 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.056+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74340c7770 con 0x7f74340b3c90 2023-11-24T08:06:56.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.057+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74340c8dc0 con 0x7f74340b3c90 2023-11-24T08:06:56.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.058+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743404e840 con 0x7f74340b3c90 2023-11-24T08:06:56.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.059+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f741c015410 con 0x7f74340b3c90 2023-11-24T08:06:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.060+0000 7f742affd640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 0x7f741406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.060+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f741c08a350 con 0x7f74340b3c90 2023-11-24T08:06:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.060+0000 7f743259c640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 0x7f741406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.061+0000 7f743259c640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 0x7f741406d4e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7424001050 tx=0x7f7424001130 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.065+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f741c0550f0 con 0x7f74340b3c90 2023-11-24T08:06:56.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.293+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f74341629c0 con 0x7f741406b020 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.306+0000 7f742affd640 1 -- 172.21.15.107:0/3791129683 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f7424005e60 con 0x7f741406b020 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:06:56.308 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:06:56.309 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:06:56.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:56.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:56.309 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:06:56.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 msgr2=0x7f741406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 0x7f741406d4e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7424001050 tx=0x7f7424001130 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 msgr2=0x7f74340b02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f741c00f580 tx=0x7f741c00fa40 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 shutdown_connections 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f741406b020 0x7f741406d4e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74340b0800 0x7f74341581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74340b3c90 0x7f74340b02c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.308+0000 7f743359e640 1 --2- 172.21.15.107:0/3791129683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74340b38b0 0x7f74340c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.309+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 >> 172.21.15.107:0/3791129683 conn(0x7f74340cbb40 msgr2=0x7f743406e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.309+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 shutdown_connections 2023-11-24T08:06:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:56.309+0000 7f743359e640 1 -- 172.21.15.107:0/3791129683 wait complete. 2023-11-24T08:06:57.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:06:57.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.557+0000 7f32389f0640 1 Processor -- start 2023-11-24T08:06:57.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.557+0000 7f32389f0640 1 -- start start 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 0x7f323406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 0x7f32340b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32340b1080 0x7f32340ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340b1530 con 0x7f32340b1080 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323415b640 con 0x7f323406e580 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.559+0000 7f32389f0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323415ed20 con 0x7f323415b260 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32340b1080 0x7f32340ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32340b1080 0x7f32340ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53480/0 (socket says 172.21.15.107:53480) 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 -- 172.21.15.107:0/3545292775 learned_addr learned my addr 172.21.15.107:0/3545292775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f322bfff640 1 --2- 172.21.15.107:0/3545292775 >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 0x7f32340b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 -- 172.21.15.107:0/3545292775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 msgr2=0x7f323406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f3232d9d640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 0x7f323406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 0x7f323406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 -- 172.21.15.107:0/3545292775 >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 msgr2=0x7f32340b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 --2- 172.21.15.107:0/3545292775 >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 0x7f32340b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f323359e640 1 -- 172.21.15.107:0/3545292775 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323415f8a0 con 0x7f32340b1080 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f322bfff640 1 --2- 172.21.15.107:0/3545292775 >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 0x7f32340b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.560+0000 7f3232d9d640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 0x7f323406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:06:57.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.561+0000 7f323359e640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32340b1080 0x7f32340ccc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f32340cba30 tx=0x7f322c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17004481c95de9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:57.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.562+0000 7f323259c640 1 -- 172.21.15.107:0/3545292775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c02c950 con 0x7f32340b1080 2023-11-24T08:06:57.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.562+0000 7f323259c640 1 -- 172.21.15.107:0/3545292775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c02cab0 con 0x7f32340b1080 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.562+0000 7f323259c640 1 -- 172.21.15.107:0/3545292775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c02ce40 con 0x7f32340b1080 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.562+0000 7f32389f0640 1 -- 172.21.15.107:0/3545292775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32340b1080 msgr2=0x7f32340ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.562+0000 7f32389f0640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32340b1080 0x7f32340ccc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f32340cba30 tx=0x7f322c02bdb0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 -- 172.21.15.107:0/3545292775 shutdown_connections 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32340b1080 0x7f32340ccc90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 --2- 172.21.15.107:0/3545292775 >> v2:172.21.15.120:3300/0 conn(0x7f323406e580 0x7f32340b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 --2- 172.21.15.107:0/3545292775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323415b260 0x7f323406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 -- 172.21.15.107:0/3545292775 >> 172.21.15.107:0/3545292775 conn(0x7f32340cbb40 msgr2=0x7f32340cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 -- 172.21.15.107:0/3545292775 shutdown_connections 2023-11-24T08:06:57.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.563+0000 7f32389f0640 1 -- 172.21.15.107:0/3545292775 wait complete. 2023-11-24T08:06:57.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.565+0000 7f32389f0640 1 Processor -- start 2023-11-24T08:06:57.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.565+0000 7f32389f0640 1 -- start start 2023-11-24T08:06:57.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323406e580 0x7f323414b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32340b1080 0x7f32341f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414b750 con 0x7f323415b260 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414b9b0 con 0x7f32340b1080 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.566+0000 7f32389f0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323415d000 con 0x7f323406e580 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53488/0 (socket says 172.21.15.107:53488) 2023-11-24T08:06:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 -- 172.21.15.107:0/2567356673 learned_addr learned my addr 172.21.15.107:0/2567356673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:57.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f322bfff640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32340b1080 0x7f32341f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f3232d9d640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323406e580 0x7f323414b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 -- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323406e580 msgr2=0x7f323414b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323406e580 0x7f323414b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 -- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32340b1080 msgr2=0x7f32341f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32340b1080 0x7f32341f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.567+0000 7f323359e640 1 -- 172.21.15.107:0/2567356673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c009b30 con 0x7f323415b260 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f323359e640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f322c036040 tx=0x7f322c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c03e450 con 0x7f323415b260 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c03e5d0 con 0x7f323415b260 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c03e980 con 0x7f323415b260 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323414bc70 con 0x7f323415b260 2023-11-24T08:06:57.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.568+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323414c230 con 0x7f323415b260 2023-11-24T08:06:57.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.569+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3234099a90 con 0x7f323415b260 2023-11-24T08:06:57.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.570+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f322c03eae0 con 0x7f323415b260 2023-11-24T08:06:57.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.571+0000 7f322b7fe640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 0x7f321406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:57.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.571+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f322c0ac7e0 con 0x7f323415b260 2023-11-24T08:06:57.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.571+0000 7f3232d9d640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 0x7f321406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:57.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.572+0000 7f3232d9d640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 0x7f321406d4b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f322400a8d0 tx=0x7f3224008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:57.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.577+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f322c0776a0 con 0x7f323415b260 2023-11-24T08:06:57.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.803+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f3234072630 con 0x7f321406aff0 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.816+0000 7f322b7fe640 1 -- 172.21.15.107:0/2567356673 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f3234072630 con 0x7f321406aff0 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:06:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:06:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:57.819 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:06:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.818+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 msgr2=0x7f321406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.818+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 0x7f321406d4b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f322400a8d0 tx=0x7f3224008040 comp rx=0 tx=0).stop 2023-11-24T08:06:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.818+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 msgr2=0x7f32341f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.818+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f322c036040 tx=0x7f322c004a40 comp rx=0 tx=0).stop 2023-11-24T08:06:57.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 shutdown_connections 2023-11-24T08:06:57.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f321406aff0 0x7f321406d4b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f323415b260 0x7f32341f6850 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32340b1080 0x7f32341f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 --2- 172.21.15.107:0/2567356673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f323406e580 0x7f323414b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:57.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 >> 172.21.15.107:0/2567356673 conn(0x7f32340cbb40 msgr2=0x7f32340af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:57.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 shutdown_connections 2023-11-24T08:06:57.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:57.819+0000 7f32389f0640 1 -- 172.21.15.107:0/2567356673 wait complete. 2023-11-24T08:06:58.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:06:59.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.063+0000 7f22af00e640 1 Processor -- start 2023-11-24T08:06:59.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.063+0000 7f22af00e640 1 -- start start 2023-11-24T08:06:59.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.064+0000 7f22af00e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22a8072160 0x7f22a815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.064+0000 7f22af00e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a80c49f0 0x7f22a806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.064+0000 7f22af00e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a815ec90 con 0x7f22a806dc40 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.064+0000 7f22af00e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a806e0f0 con 0x7f22a8072160 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.064+0000 7f22af00e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80cd530 con 0x7f22a80c49f0 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22ad5ab640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f22a806dc40 0x7f22a80cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48264/0 (socket says 172.21.15.107:48264) 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22ad5ab640 1 -- 172.21.15.107:0/4274845667 learned_addr learned my addr 172.21.15.107:0/4274845667 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22acdaa640 1 --2- 172.21.15.107:0/4274845667 >> v2:172.21.15.107:3300/0 conn(0x7f22a8072160 0x7f22a815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22a7fff640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a80c49f0 0x7f22a806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22acdaa640 1 -- 172.21.15.107:0/4274845667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a80c49f0 msgr2=0x7f22a806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:06:59.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22acdaa640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a80c49f0 0x7f22a806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22acdaa640 1 -- 172.21.15.107:0/4274845667 >> v1:172.21.15.107:6789/0 conn(0x7f22a806dc40 legacy=0x7f22a80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.065+0000 7f22acdaa640 1 -- 172.21.15.107:0/4274845667 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80c4ed0 con 0x7f22a8072160 2023-11-24T08:06:59.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.066+0000 7f22acdaa640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a8072160 0x7f22a815e750 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2298009d90 tx=0x7f229802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f68a7d1222118ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:59.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.066+0000 7f22a77fe640 1 -- 172.21.15.107:0/4274845667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229802cc60 con 0x7f22a8072160 2023-11-24T08:06:59.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.066+0000 7f22a77fe640 1 -- 172.21.15.107:0/4274845667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229802cdc0 con 0x7f22a8072160 2023-11-24T08:06:59.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.066+0000 7f22a77fe640 1 -- 172.21.15.107:0/4274845667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298034660 con 0x7f22a8072160 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 -- 172.21.15.107:0/4274845667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a8072160 msgr2=0x7f22a815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a8072160 0x7f22a815e750 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2298009d90 tx=0x7f229802beb0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 -- 172.21.15.107:0/4274845667 shutdown_connections 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a80c49f0 0x7f22a806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 --2- 172.21.15.107:0/4274845667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a8072160 0x7f22a815e750 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 -- 172.21.15.107:0/4274845667 >> 172.21.15.107:0/4274845667 conn(0x7f22a80b0c50 msgr2=0x7f22a80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 -- 172.21.15.107:0/4274845667 shutdown_connections 2023-11-24T08:06:59.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.067+0000 7f22af00e640 1 -- 172.21.15.107:0/4274845667 wait complete. 2023-11-24T08:06:59.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.069+0000 7f22af00e640 1 Processor -- start 2023-11-24T08:06:59.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.069+0000 7f22af00e640 1 -- start start 2023-11-24T08:06:59.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a806dc40 0x7f22a8159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8072160 0x7f22a80c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a8159fd0 con 0x7f22a80c49f0 2023-11-24T08:06:59.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80c93c0 con 0x7f22a8072160 2023-11-24T08:06:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22af00e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80c7de0 con 0x7f22a806dc40 2023-11-24T08:06:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22ad5ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22ad5ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51238/0 (socket says 172.21.15.107:51238) 2023-11-24T08:06:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.070+0000 7f22ad5ab640 1 -- 172.21.15.107:0/458481084 learned_addr learned my addr 172.21.15.107:0/458481084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:06:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22acdaa640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a806dc40 0x7f22a8159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22a7fff640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8072160 0x7f22a80c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 -- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a806dc40 msgr2=0x7f22a8159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a806dc40 0x7f22a8159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 -- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8072160 msgr2=0x7f22a80c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8072160 0x7f22a80c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 -- 172.21.15.107:0/458481084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2298009a70 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22ad5ab640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f22a000c450 tx=0x7f22a000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0005ce0 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a0005e40 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a80c80c0 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.071+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0007270 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.072+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a80c8680 con 0x7f22a80c49f0 2023-11-24T08:06:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.073+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a804e8b0 con 0x7f22a80c49f0 2023-11-24T08:06:59.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.073+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f22a00073d0 con 0x7f22a80c49f0 2023-11-24T08:06:59.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.074+0000 7f22a57fa640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 0x7f228806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:06:59.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.074+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22a0088b80 con 0x7f22a80c49f0 2023-11-24T08:06:59.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.075+0000 7f22acdaa640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 0x7f228806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:06:59.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.075+0000 7f22acdaa640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 0x7f228806d4b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2298036000 tx=0x7f2298030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:06:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.079+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22a0053a60 con 0x7f22a80c49f0 2023-11-24T08:06:59.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.313+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f22a80c5770 con 0x7f228806aff0 2023-11-24T08:06:59.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.345+0000 7f22a57fa640 1 -- 172.21.15.107:0/458481084 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f22980440c0 con 0x7f228806aff0 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 msgr2=0x7f228806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 0x7f228806d4b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2298036000 tx=0x7f2298030000 comp rx=0 tx=0).stop 2023-11-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 msgr2=0x7f22a80b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f22a000c450 tx=0x7f22a000c910 comp rx=0 tx=0).stop 2023-11-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 shutdown_connections 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228806aff0 0x7f228806d4b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a80c49f0 0x7f22a80b4480 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8072160 0x7f22a80c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 --2- 172.21.15.107:0/458481084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a806dc40 0x7f22a8159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 >> 172.21.15.107:0/458481084 conn(0x7f22a80b0c50 msgr2=0x7f22a80ad500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 shutdown_connections 2023-11-24T08:06:59.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:06:59.348+0000 7f22af00e640 1 -- 172.21.15.107:0/458481084 wait complete. 2023-11-24T08:07:00.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:07:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.592+0000 7fd181f2e640 1 Processor -- start 2023-11-24T08:07:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.592+0000 7fd181f2e640 1 -- start start 2023-11-24T08:07:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd181f2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd181f2e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd181f2e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c06d5c0 con 0x7fd17c0b1080 2023-11-24T08:07:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd181f2e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c0b1530 con 0x7fd17c06e580 2023-11-24T08:07:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd181f2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c15b640 con 0x7fd17c15b260 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd17bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.594+0000 7fd180f2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:42626/0 (socket says 172.21.15.107:42626) 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 -- 172.21.15.107:0/2362826357 learned_addr learned my addr 172.21.15.107:0/2362826357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd17bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:52736/0 (socket says 172.21.15.107:52736) 2023-11-24T08:07:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd18172d640 1 --1- 172.21.15.107:0/2362826357 >> v1:172.21.15.120:6789/0 conn(0x7fd17c0b1080 0x7fd17c0ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:53176/0 (socket says 172.21.15.107:53176) 2023-11-24T08:07:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 -- 172.21.15.107:0/2362826357 >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 msgr2=0x7fd17c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 --2- 172.21.15.107:0/2362826357 >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 -- 172.21.15.107:0/2362826357 >> v1:172.21.15.120:6789/0 conn(0x7fd17c0b1080 legacy=0x7fd17c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd180f2c640 1 -- 172.21.15.107:0/2362826357 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c15ed20 con 0x7fd17c15b260 2023-11-24T08:07:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.595+0000 7fd17bfff640 1 --2- 172.21.15.107:0/2362826357 >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:07:00.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.597+0000 7fd180f2c640 1 --2- 172.21.15.107:0/2362826357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd170009870 tx=0x7fd17002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c4a5b45fa558d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd17b7fe640 1 -- 172.21.15.107:0/2362826357 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700317c0 con 0x7fd17c15b260 2023-11-24T08:07:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd17b7fe640 1 -- 172.21.15.107:0/2362826357 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170031920 con 0x7fd17c15b260 2023-11-24T08:07:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd17b7fe640 1 -- 172.21.15.107:0/2362826357 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170031cb0 con 0x7fd17c15b260 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 -- 172.21.15.107:0/2362826357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 msgr2=0x7fd17c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 --2- 172.21.15.107:0/2362826357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd170009870 tx=0x7fd17002b860 comp rx=0 tx=0).stop 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 -- 172.21.15.107:0/2362826357 shutdown_connections 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 --2- 172.21.15.107:0/2362826357 >> v2:172.21.15.120:3300/0 conn(0x7fd17c06e580 0x7fd17c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 --2- 172.21.15.107:0/2362826357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c15b260 0x7fd17c06e040 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.598+0000 7fd181f2e640 1 -- 172.21.15.107:0/2362826357 >> 172.21.15.107:0/2362826357 conn(0x7fd17c0cbb40 msgr2=0x7fd17c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.599+0000 7fd181f2e640 1 -- 172.21.15.107:0/2362826357 shutdown_connections 2023-11-24T08:07:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.599+0000 7fd181f2e640 1 -- 172.21.15.107:0/2362826357 wait complete. 2023-11-24T08:07:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.600+0000 7fd181f2e640 1 Processor -- start 2023-11-24T08:07:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.600+0000 7fd181f2e640 1 -- start start 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd17c06e580 0x7fd17c0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 0x7fd17c0b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c099df0 con 0x7fd17c15b260 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c4d10 con 0x7fd17c06e580 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.601+0000 7fd181f2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0b30a0 con 0x7fd17c0b1080 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51252/0 (socket says 172.21.15.107:51252) 2023-11-24T08:07:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 -- 172.21.15.107:0/3236142931 learned_addr learned my addr 172.21.15.107:0/3236142931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd180f2c640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd17c06e580 0x7fd17c0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd17bfff640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 0x7fd17c0b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 -- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 msgr2=0x7fd17c0b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 0x7fd17c0b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 -- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd17c06e580 msgr2=0x7fd17c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd17c06e580 0x7fd17c0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd18172d640 1 -- 172.21.15.107:0/3236142931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168009400 con 0x7fd17c15b260 2023-11-24T08:07:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.602+0000 7fd17bfff640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 0x7fd17c0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd18172d640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c730 tx=0x7fd17400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400ce40 con 0x7fd17c15b260 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17400a640 con 0x7fd17c15b260 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400a9f0 con 0x7fd17c15b260 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700094d0 con 0x7fd17c15b260 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.603+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c0b3660 con 0x7fd17c15b260 2023-11-24T08:07:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.604+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c099f60 con 0x7fd17c15b260 2023-11-24T08:07:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.605+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7fd17400ab50 con 0x7fd17c15b260 2023-11-24T08:07:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.606+0000 7fd1797fa640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 0x7fd16006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.606+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd174088fa0 con 0x7fd17c15b260 2023-11-24T08:07:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.606+0000 7fd180f2c640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 0x7fd16006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.607+0000 7fd180f2c640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 0x7fd16006d4b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd170036000 tx=0x7fd17003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:00.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.610+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd17c099f60 con 0x7fd17c15b260 2023-11-24T08:07:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:00.838+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7fd17c0ce3e0 con 0x7fd16006aff0 2023-11-24T08:07:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.214+0000 7fd1797fa640 1 -- 172.21.15.107:0/3236142931 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fd17c0ce3e0 con 0x7fd16006aff0 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:07:04.217 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:04.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:04.218 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.217+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 msgr2=0x7fd16006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.217+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 0x7fd16006d4b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd170036000 tx=0x7fd17003d040 comp rx=0 tx=0).stop 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.217+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 msgr2=0x7fd17c15d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.217+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c730 tx=0x7fd17400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 shutdown_connections 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd16006aff0 0x7fd16006d4b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd17c15b260 0x7fd17c15d950 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd17c0b1080 0x7fd17c0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 --2- 172.21.15.107:0/3236142931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd17c06e580 0x7fd17c0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 >> 172.21.15.107:0/3236142931 conn(0x7fd17c0cbb40 msgr2=0x7fd17c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 shutdown_connections 2023-11-24T08:07:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:04.218+0000 7fd181f2e640 1 -- 172.21.15.107:0/3236142931 wait complete. 2023-11-24T08:07:05.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:07:05.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.470+0000 7f37d359e640 1 Processor -- start 2023-11-24T08:07:05.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.470+0000 7f37d359e640 1 -- start start 2023-11-24T08:07:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.471+0000 7f37d359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37d406c7c0 0x7f37d4071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.471+0000 7f37d359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37d4071620 0x7f37d40ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.471+0000 7f37d359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4070860 con 0x7f37d415aa80 2023-11-24T08:07:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.471+0000 7f37d359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d4159f40 con 0x7f37d4071620 2023-11-24T08:07:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.471+0000 7f37d359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d415b020 con 0x7f37d406c7c0 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37cbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37d415aa80 0x7f37d406cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48268/0 (socket says 172.21.15.107:48268) 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37d4071620 0x7f37d40ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37cbfff640 1 -- 172.21.15.107:0/1915060267 learned_addr learned my addr 172.21.15.107:0/1915060267 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d259c640 1 --2- 172.21.15.107:0/1915060267 >> v2:172.21.15.120:3300/0 conn(0x7f37d406c7c0 0x7f37d4071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d2d9d640 1 -- 172.21.15.107:0/1915060267 >> v2:172.21.15.120:3300/0 conn(0x7f37d406c7c0 msgr2=0x7f37d4071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d2d9d640 1 --2- 172.21.15.107:0/1915060267 >> v2:172.21.15.120:3300/0 conn(0x7f37d406c7c0 0x7f37d4071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:05.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d2d9d640 1 -- 172.21.15.107:0/1915060267 >> v1:172.21.15.107:6789/0 conn(0x7f37d415aa80 legacy=0x7f37d406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:05.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.472+0000 7f37d2d9d640 1 -- 172.21.15.107:0/1915060267 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4070b40 con 0x7f37d4071620 2023-11-24T08:07:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.473+0000 7f37d2d9d640 1 --2- 172.21.15.107:0/1915060267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4071620 0x7f37d40ccc90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f37d40ccc40 tx=0x7f37cc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23324fc16b5b7830 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.473+0000 7f37d1d9b640 1 -- 172.21.15.107:0/1915060267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc02cd00 con 0x7f37d4071620 2023-11-24T08:07:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d1d9b640 1 -- 172.21.15.107:0/1915060267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc02ce60 con 0x7f37d4071620 2023-11-24T08:07:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d1d9b640 1 -- 172.21.15.107:0/1915060267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc034660 con 0x7f37d4071620 2023-11-24T08:07:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 -- 172.21.15.107:0/1915060267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4071620 msgr2=0x7f37d40ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 --2- 172.21.15.107:0/1915060267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4071620 0x7f37d40ccc90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f37d40ccc40 tx=0x7f37cc02bf50 comp rx=0 tx=0).stop 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 -- 172.21.15.107:0/1915060267 shutdown_connections 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 --2- 172.21.15.107:0/1915060267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4071620 0x7f37d40ccc90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 --2- 172.21.15.107:0/1915060267 >> v2:172.21.15.120:3300/0 conn(0x7f37d406c7c0 0x7f37d4071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 -- 172.21.15.107:0/1915060267 >> 172.21.15.107:0/1915060267 conn(0x7f37d40cbb40 msgr2=0x7f37d40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 -- 172.21.15.107:0/1915060267 shutdown_connections 2023-11-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.474+0000 7f37d359e640 1 -- 172.21.15.107:0/1915060267 wait complete. 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.476+0000 7f37d359e640 1 Processor -- start 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.476+0000 7f37d359e640 1 -- start start 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 0x7f37d414fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37d4071620 0x7f37d40b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40c5750 con 0x7f37d415aa80 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d414aaa0 con 0x7f37d406c7c0 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d414ec80 con 0x7f37d4071620 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51276/0 (socket says 172.21.15.107:51276) 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.477+0000 7f37d2d9d640 1 -- 172.21.15.107:0/3871108556 learned_addr learned my addr 172.21.15.107:0/3871108556 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d259c640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 0x7f37d414fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 -- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37d4071620 msgr2=0x7f37d40b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37d4071620 0x7f37d40b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 -- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 msgr2=0x7f37d414fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 0x7f37d414fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 -- 172.21.15.107:0/3871108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc009a70 con 0x7f37d415aa80 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d259c640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 0x7f37d414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d2d9d640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009e80 tx=0x7f37cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc03d600 con 0x7f37d415aa80 2023-11-24T08:07:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc03d780 con 0x7f37d415aa80 2023-11-24T08:07:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d414ef60 con 0x7f37d415aa80 2023-11-24T08:07:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.478+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc03db30 con 0x7f37d415aa80 2023-11-24T08:07:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.479+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d414f520 con 0x7f37d415aa80 2023-11-24T08:07:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.480+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d40ae480 con 0x7f37d415aa80 2023-11-24T08:07:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.480+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f37cc03b040 con 0x7f37d415aa80 2023-11-24T08:07:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.481+0000 7f37caffd640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 0x7f37b406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.481+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f37cc0ac170 con 0x7f37d415aa80 2023-11-24T08:07:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.482+0000 7f37d259c640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 0x7f37b406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.482+0000 7f37d259c640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 0x7f37b406d4b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f37c4006fd0 tx=0x7f37c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.486+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37cc076fe0 con 0x7f37d415aa80 2023-11-24T08:07:05.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:05.714+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f37d40ae8d0 con 0x7f37b406aff0 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.210+0000 7f37caffd640 1 -- 172.21.15.107:0/3871108556 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+269 (secure 0 0 0) 0x7f37c4009530 con 0x7f37b406aff0 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "failed", 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:07:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "failure": { 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "errno": "2", 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "error_msg": "snapshot 'subvol_snap1109' does not exist" 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:09.213 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:07:09.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 msgr2=0x7f37b406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 0x7f37b406d4b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f37c4006fd0 tx=0x7f37c4008040 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 msgr2=0x7f37d40c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009e80 tx=0x7f37cc004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 shutdown_connections 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37b406aff0 0x7f37b406d4b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d415aa80 0x7f37d40c6650 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37d4071620 0x7f37d40b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 --2- 172.21.15.107:0/3871108556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37d406c7c0 0x7f37d414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.213+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 >> 172.21.15.107:0/3871108556 conn(0x7f37d40cbb40 msgr2=0x7f37d40cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.214+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 shutdown_connections 2023-11-24T08:07:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.214+0000 7f37d359e640 1 -- 172.21.15.107:0/3871108556 wait complete. 2023-11-24T08:07:09.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2023-11-24T08:07:09.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:07:09.385 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:07:09.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.439+0000 7fb1abccf640 1 Processor -- start 2023-11-24T08:07:09.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.439+0000 7fb1abccf640 1 -- start start 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1abccf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1a415b440 0x7fb1a40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1abccf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c59b0 0x7fb1a406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1abccf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a415eea0 con 0x7fb1a40cbf20 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1abccf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a40c5e00 con 0x7fb1a415b440 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1abccf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40cc3d0 con 0x7fb1a40c59b0 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1a926a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb1a40cbf20 0x7fb1a40c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59836/0 (socket says 172.21.15.107:59836) 2023-11-24T08:07:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.440+0000 7fb1a926a640 1 -- 172.21.15.107:0/19507388 learned_addr learned my addr 172.21.15.107:0/19507388 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1aa26c640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c59b0 0x7fb1a406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/19507388 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415b440 0x7fb1a40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846412688 0 0) 0x7fb1a415eea0 con 0x7fb1a40cbf20 2023-11-24T08:07:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190003520 con 0x7fb1a40cbf20 2023-11-24T08:07:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048240910 0 0) 0x7fb190003520 con 0x7fb1a40cbf20 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c59b0 msgr2=0x7fb1a406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c59b0 0x7fb1a406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415b440 msgr2=0x7fb1a40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 --2- 172.21.15.107:0/19507388 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415b440 0x7fb1a40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40abf30 con 0x7fb1a40cbf20 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb194001670 con 0x7fb1a40cbf20 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 >> v1:172.21.15.107:6789/0 conn(0x7fb1a40cbf20 legacy=0x7fb1a40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb190003970 0x7fb190003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900047c0 0x7fb190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb190005180 0x7fb19000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40cc3d0 con 0x7fb190003970 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19000bec0 con 0x7fb1900047c0 2023-11-24T08:07:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19000c030 con 0x7fb190005180 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.441+0000 7fb1aa26c640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c59b0 0x7fb1a406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1aa26c640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb190005180 0x7fb19000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a926a640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900047c0 0x7fb190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb190003970 0x7fb190003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 -- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb190005180 msgr2=0x7fb19000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb190005180 0x7fb19000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 -- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900047c0 msgr2=0x7fb190004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900047c0 0x7fb190004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.442+0000 7fb1a9a6b640 1 -- 172.21.15.107:0/19507388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190003520 con 0x7fb190003970 2023-11-24T08:07:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb190003970 0x7fb190003d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009ac0 tx=0x7fb1900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0620d0209027a48 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a002c6d0 con 0x7fb190003970 2023-11-24T08:07:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a002c830 con 0x7fb190003970 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1a8a69640 1 -- 172.21.15.107:0/19507388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a002cbc0 con 0x7fb190003970 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1abccf640 1 -- 172.21.15.107:0/19507388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb190003970 msgr2=0x7fb190003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1abccf640 1 --2- 172.21.15.107:0/19507388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb190003970 0x7fb190003d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009ac0 tx=0x7fb1900057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1aa26c640 1 -- 172.21.15.107:0/19507388 reap_dead start 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1abccf640 1 -- 172.21.15.107:0/19507388 shutdown_connections 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.444+0000 7fb1abccf640 1 -- 172.21.15.107:0/19507388 >> 172.21.15.107:0/19507388 conn(0x7fb1a40b0c50 msgr2=0x7fb1a40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.445+0000 7fb1abccf640 1 -- 172.21.15.107:0/19507388 shutdown_connections 2023-11-24T08:07:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.445+0000 7fb1abccf640 1 -- 172.21.15.107:0/19507388 wait complete. 2023-11-24T08:07:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.446+0000 7fb1abccf640 1 Processor -- start 2023-11-24T08:07:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.446+0000 7fb1abccf640 1 -- start start 2023-11-24T08:07:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 0x7fb1a4156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c8c80 0x7fb1a40aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a415cc60 con 0x7fb1a415dd00 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a415cf40 con 0x7fb190003970 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.447+0000 7fb1abccf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a415d220 con 0x7fb1a40c8c80 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58300/0 (socket says 172.21.15.107:58300) 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 -- 172.21.15.107:0/1323814191 learned_addr learned my addr 172.21.15.107:0/1323814191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 0x7fb1a4156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1aa26c640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c8c80 0x7fb1a40aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 -- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c8c80 msgr2=0x7fb1a40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c8c80 0x7fb1a40aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 -- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 msgr2=0x7fb1a4156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 0x7fb1a4156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 -- 172.21.15.107:0/1323814191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a415d500 con 0x7fb1a415dd00 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 0x7fb1a4156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.448+0000 7fb1a926a640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb194007310 tx=0x7fb1940077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.449+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400f940 con 0x7fb1a415dd00 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.449+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19400fac0 con 0x7fb1a415dd00 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.449+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a415d7e0 con 0x7fb1a415dd00 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.449+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400fe70 con 0x7fb1a415dd00 2023-11-24T08:07:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.449+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a40b04a0 con 0x7fb1a415dd00 2023-11-24T08:07:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.450+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a406dc80 con 0x7fb1a415dd00 2023-11-24T08:07:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.451+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7fb19401d460 con 0x7fb1a415dd00 2023-11-24T08:07:09.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.452+0000 7fb19a7fc640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 0x7fb18806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.452+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb194089ae0 con 0x7fb1a415dd00 2023-11-24T08:07:09.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.453+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 0x7fb18806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.453+0000 7fb1a9a6b640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 0x7fb18806d4b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0036000 tx=0x7fb1a0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.457+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1940548c0 con 0x7fb1a415dd00 2023-11-24T08:07:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.687+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1a40c84a0 con 0x7fb18806aff0 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.701+0000 7fb19a7fc640 1 -- 172.21.15.107:0/1323814191 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+269 (secure 0 0 0) 0x7fb1a00040c0 con 0x7fb18806aff0 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "failed", 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7918", 2023-11-24T08:07:09.703 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1109" 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: "failure": { 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: "errno": "2", 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: "error_msg": "snapshot 'subvol_snap1109' does not exist" 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:09.704 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:07:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.703+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 msgr2=0x7fb18806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.703+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 0x7fb18806d4b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0036000 tx=0x7fb1a0030000 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.703+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 msgr2=0x7fb1a40c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.703+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb194007310 tx=0x7fb1940077d0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 shutdown_connections 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18806aff0 0x7fb18806d4b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40c8c80 0x7fb1a40aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415dd00 0x7fb1a40c8740 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 --2- 172.21.15.107:0/1323814191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb190003970 0x7fb1a4156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 >> 172.21.15.107:0/1323814191 conn(0x7fb1a40b0c50 msgr2=0x7fb1a40c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 shutdown_connections 2023-11-24T08:07:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.704+0000 7fb1abccf640 1 -- 172.21.15.107:0/1323814191 wait complete. 2023-11-24T08:07:09.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone9616 --force 2023-11-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.943+0000 7f6725a6e640 1 Processor -- start 2023-11-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.943+0000 7f6725a6e640 1 -- start start 2023-11-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.944+0000 7f6725a6e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.944+0000 7f6725a6e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67200b1e50 0x7f6720165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.944+0000 7f6725a6e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6720165970 con 0x7f672014acb0 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f6725a6e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67200b22e0 con 0x7f67200b1e50 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f6725a6e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6720165ba0 con 0x7f6720070ab0 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f671ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f672526d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67200b1e50 0x7f6720165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f671ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:42916/0 (socket says 172.21.15.107:42916) 2023-11-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.945+0000 7f671ffff640 1 -- 172.21.15.107:0/116684015 learned_addr learned my addr 172.21.15.107:0/116684015 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.946+0000 7f672526d640 1 -- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 msgr2=0x7f67200b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.946+0000 7f672526d640 1 --2- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.946+0000 7f672526d640 1 -- 172.21.15.107:0/116684015 >> v1:172.21.15.120:6789/0 conn(0x7f672014acb0 legacy=0x7f67200703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.946+0000 7f672526d640 1 -- 172.21.15.107:0/116684015 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6720165e80 con 0x7f67200b1e50 2023-11-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.947+0000 7f672526d640 1 --2- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67200b1e50 0x7f6720165430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6718009d90 tx=0x7f671802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2a23a048698cd68 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.948+0000 7f671f7fe640 1 -- 172.21.15.107:0/116684015 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671802cc60 con 0x7f67200b1e50 2023-11-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.948+0000 7f671f7fe640 1 -- 172.21.15.107:0/116684015 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671802cdc0 con 0x7f67200b1e50 2023-11-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.948+0000 7f671f7fe640 1 -- 172.21.15.107:0/116684015 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718034660 con 0x7f67200b1e50 2023-11-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.948+0000 7f6725a6e640 1 -- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67200b1e50 msgr2=0x7f6720165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 --2- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67200b1e50 0x7f6720165430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6718009d90 tx=0x7f671802beb0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 -- 172.21.15.107:0/116684015 shutdown_connections 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 --2- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67200b1e50 0x7f6720165430 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 --2- 172.21.15.107:0/116684015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 -- 172.21.15.107:0/116684015 >> 172.21.15.107:0/116684015 conn(0x7f67200cb8a0 msgr2=0x7f67200cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 -- 172.21.15.107:0/116684015 shutdown_connections 2023-11-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.949+0000 7f6725a6e640 1 -- 172.21.15.107:0/116684015 wait complete. 2023-11-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.950+0000 7f6725a6e640 1 Processor -- start 2023-11-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 -- start start 2023-11-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 0x7f6720164770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200a8970 con 0x7f67200b1e50 2023-11-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200a8bd0 con 0x7f672014acb0 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.951+0000 7f6725a6e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67200a8eb0 con 0x7f6720070ab0 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58316/0 (socket says 172.21.15.107:58316) 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 -- 172.21.15.107:0/2133481569 learned_addr learned my addr 172.21.15.107:0/2133481569 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f6724a6c640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f672526d640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 0x7f6720164770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 -- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 msgr2=0x7f67200b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 -- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 msgr2=0x7f6720164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 0x7f6720164770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f671ffff640 1 -- 172.21.15.107:0/2133481569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6718009a70 con 0x7f67200b1e50 2023-11-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.952+0000 7f672526d640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 0x7f6720164770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f671ffff640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f670800e8b0 tx=0x7f670800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708009600 con 0x7f67200b1e50 2023-11-24T08:07:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800c940 con 0x7f67200b1e50 2023-11-24T08:07:09.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67200a9190 con 0x7f67200b1e50 2023-11-24T08:07:09.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800ccf0 con 0x7f67200b1e50 2023-11-24T08:07:09.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.953+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67200a9750 con 0x7f67200b1e50 2023-11-24T08:07:09.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.954+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67200a98e0 con 0x7f67200b1e50 2023-11-24T08:07:09.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.955+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f6708013470 con 0x7f67200b1e50 2023-11-24T08:07:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.956+0000 7f671d7fa640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 0x7f670006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.956+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6708089e70 con 0x7f67200b1e50 2023-11-24T08:07:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.956+0000 7f6724a6c640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 0x7f670006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.957+0000 7f6724a6c640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 0x7f670006d4b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6710000f60 tx=0x7f6710009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:09.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:09.961+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6708054c70 con 0x7f67200b1e50 2023-11-24T08:07:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.179+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone9616", "force": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f67200a9020 con 0x7f670006aff0 2023-11-24T08:07:10.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.212+0000 7f671d7fa640 1 -- 172.21.15.107:0/2133481569 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f67200a9020 con 0x7f670006aff0 2023-11-24T08:07:10.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 msgr2=0x7f670006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 0x7f670006d4b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6710000f60 tx=0x7f6710009040 comp rx=0 tx=0).stop 2023-11-24T08:07:10.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 msgr2=0x7f67200a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f670800e8b0 tx=0x7f670800ed70 comp rx=0 tx=0).stop 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 shutdown_connections 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f670006aff0 0x7f670006d4b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f672014acb0 0x7f6720164770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67200b1e50 0x7f67200a8430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 --2- 172.21.15.107:0/2133481569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6720070ab0 0x7f67200b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 >> 172.21.15.107:0/2133481569 conn(0x7f67200cb8a0 msgr2=0x7f67200b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:10.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 shutdown_connections 2023-11-24T08:07:10.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.215+0000 7f6725a6e640 1 -- 172.21.15.107:0/2133481569 wait complete. 2023-11-24T08:07:10.220 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.219+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.221 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.219+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.221 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.219+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.221 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.219+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.221 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.219+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol7918 2023-11-24T08:07:10.239 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.237+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.239 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.237+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.237+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.237+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:10.237+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:10.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.425+0000 7f9001ba7640 1 Processor -- start 2023-11-24T08:07:10.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.426+0000 7f9001ba7640 1 -- start start 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9001ba7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8ffc06ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9001ba7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc15ed20 con 0x7f8ffc15b260 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9001ba7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc15f8a0 con 0x7f8ffc0b0b40 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9001ba7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc15b640 con 0x7f8ffc0b0fb0 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9000ba5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8ffc15b260 0x7f8ffc06e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59846/0 (socket says 172.21.15.107:59846) 2023-11-24T08:07:10.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.427+0000 7f9000ba5640 1 -- 172.21.15.107:0/3976058650 learned_addr learned my addr 172.21.15.107:0/3976058650 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f90013a6640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8ffc06ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287281151 0 0) 0x7f8ffc15ed20 con 0x7f8ffc15b260 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec003520 con 0x7f8ffc15b260 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772364211 0 0) 0x7f8fec003520 con 0x7f8ffc15b260 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 msgr2=0x7f8ffc06ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:10.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8ffc06ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 >> v1:172.21.15.120:6789/0 conn(0x7f8ffc0b0b40 legacy=0x7f8ffc06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc06e750 con 0x7f8ffc15b260 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8ff0001670 con 0x7f8ffc15b260 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 >> v1:172.21.15.107:6789/0 conn(0x7f8ffc15b260 legacy=0x7f8ffc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8fec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fec003f20 0x7f8fec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec004d00 0x7f8fec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc15b640 con 0x7f8fec003f20 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00bbc0 con 0x7f8fec004d00 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.428+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00bd80 con 0x7f8ffc0b0fb0 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fec003f20 0x7f8fec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f90013a6640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec004d00 0x7f8fec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f9000ba5640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8fec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 -- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 msgr2=0x7f8fec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8fec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 -- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec004d00 msgr2=0x7f8fec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec004d00 0x7f8fec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f8ffbfff640 1 -- 172.21.15.107:0/3976058650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec003520 con 0x7f8fec003f20 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f9000ba5640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b0fb0 0x7f8fec0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.429+0000 7f90013a6640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec004d00 0x7f8fec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:10.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.430+0000 7f8ffbfff640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fec003f20 0x7f8fec0047c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001750 tx=0x7f8fec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4079cf787861a96b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:10.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80052d0 con 0x7f8fec003f20 2023-11-24T08:07:10.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe802e050 con 0x7f8fec003f20 2023-11-24T08:07:10.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f8ffb7fe640 1 -- 172.21.15.107:0/3976058650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003830 con 0x7f8fec003f20 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 -- 172.21.15.107:0/3976058650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fec003f20 msgr2=0x7f8fec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 --2- 172.21.15.107:0/3976058650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fec003f20 0x7f8fec0047c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001750 tx=0x7f8fec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f90013a6640 1 -- 172.21.15.107:0/3976058650 reap_dead start 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 -- 172.21.15.107:0/3976058650 shutdown_connections 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 -- 172.21.15.107:0/3976058650 >> 172.21.15.107:0/3976058650 conn(0x7f8ffc0cbb40 msgr2=0x7f8ffc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 -- 172.21.15.107:0/3976058650 shutdown_connections 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.431+0000 7f9001ba7640 1 -- 172.21.15.107:0/3976058650 wait complete. 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.433+0000 7f9001ba7640 1 Processor -- start 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.433+0000 7f9001ba7640 1 -- start start 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec003f20 0x7f8ffc0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc09b570 0x7f8ffc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc156e50 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0b3d20 con 0x7f8fec003f20 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.434+0000 7f9001ba7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0b4000 con 0x7f8ffc09b570 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58368/0 (socket says 172.21.15.107:58368) 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 -- 172.21.15.107:0/2073436306 learned_addr learned my addr 172.21.15.107:0/2073436306 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f8ffbfff640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc09b570 0x7f8ffc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f9000ba5640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec003f20 0x7f8ffc0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 -- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc09b570 msgr2=0x7f8ffc1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc09b570 0x7f8ffc1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 -- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec003f20 msgr2=0x7f8ffc0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec003f20 0x7f8ffc0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 -- 172.21.15.107:0/2073436306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0b42e0 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f90013a6640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0cb9b0 tx=0x7f8ff400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.435+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4017400 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.436+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4011040 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.436+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0b45c0 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.436+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4017780 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.436+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc15bdb0 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.437+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc04e840 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.438+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 89384+0+0 (secure 0 0 0) 0x7f8ff40203b0 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.439+0000 7f8ff97fa640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 0x7f8fe006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.439+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8ff4089de0 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.439+0000 7f9000ba5640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 0x7f8fe006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.440+0000 7f9000ba5640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 0x7f8fe006d4b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc15a450 tx=0x7f8ff0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.443+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ff4054b70 con 0x7f8ffc09b9f0 2023-11-24T08:07:10.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.660+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol7918", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ffc06e5e0 con 0x7f8fe006aff0 2023-11-24T08:07:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.701+0000 7f8ff97fa640 1 -- 172.21.15.107:0/2073436306 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8ffc06e5e0 con 0x7f8fe006aff0 2023-11-24T08:07:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 msgr2=0x7f8fe006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 0x7f8fe006d4b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc15a450 tx=0x7f8ff0009040 comp rx=0 tx=0).stop 2023-11-24T08:07:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 msgr2=0x7f8ffc156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0cb9b0 tx=0x7f8ff400f4a0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 shutdown_connections 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe006aff0 0x7f8fe006d4b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ffc09b9f0 0x7f8ffc156910 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc09b570 0x7f8ffc1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 --2- 172.21.15.107:0/2073436306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fec003f20 0x7f8ffc0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 >> 172.21.15.107:0/2073436306 conn(0x7f8ffc0cbb40 msgr2=0x7f8ffc071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 shutdown_connections 2023-11-24T08:07:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:10.704+0000 7f9001ba7640 1 -- 172.21.15.107:0/2073436306 wait complete. 2023-11-24T08:07:10.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:10.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:07:10.777 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:07:15.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:15.779 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:07:15.843 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:07:15.845 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:07:15.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:07:15.977 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:07:16.099 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:07:16.099 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:16.099 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:16.158 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:16.159 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:16.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:16.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.422+0000 7f5c4c57d640 1 Processor -- start 2023-11-24T08:07:16.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.423+0000 7f5c4c57d640 1 -- start start 2023-11-24T08:07:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4c57d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c4415c7f0 0x7f5c4406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4c57d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c440ce6c0 0x7f5c440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4c57d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440cece0 con 0x7f5c440cf450 2023-11-24T08:07:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4c57d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c440cfe40 con 0x7f5c4415c7f0 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4c57d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c440ceaf0 con 0x7f5c440ce6c0 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.424+0000 7f5c4a319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c4415c7f0 0x7f5c4406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c4415c7f0 0x7f5c4406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58376/0 (socket says 172.21.15.107:58376) 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 -- 172.21.15.107:0/1405515979 learned_addr learned my addr 172.21.15.107:0/1405515979 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 -- 172.21.15.107:0/1405515979 >> v2:172.21.15.120:3300/0 conn(0x7f5c440ce6c0 msgr2=0x7f5c440cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 --2- 172.21.15.107:0/1405515979 >> v2:172.21.15.120:3300/0 conn(0x7f5c440ce6c0 0x7f5c440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 -- 172.21.15.107:0/1405515979 >> v1:172.21.15.120:6789/0 conn(0x7f5c440cf450 legacy=0x7f5c440cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.425+0000 7f5c4a319640 1 -- 172.21.15.107:0/1405515979 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4406d040 con 0x7f5c4415c7f0 2023-11-24T08:07:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.426+0000 7f5c4a319640 1 --2- 172.21.15.107:0/1405515979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c4415c7f0 0x7f5c4406db20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d00e0 tx=0x7f5c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79f4349737ac8e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.426+0000 7f5c49317640 1 -- 172.21.15.107:0/1405515979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40031870 con 0x7f5c4415c7f0 2023-11-24T08:07:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.426+0000 7f5c49317640 1 -- 172.21.15.107:0/1405515979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400319d0 con 0x7f5c4415c7f0 2023-11-24T08:07:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.426+0000 7f5c49317640 1 -- 172.21.15.107:0/1405515979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40031d60 con 0x7f5c4415c7f0 2023-11-24T08:07:16.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 -- 172.21.15.107:0/1405515979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c4415c7f0 msgr2=0x7f5c4406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/1405515979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c4415c7f0 0x7f5c4406db20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d00e0 tx=0x7f5c4002b910 comp rx=0 tx=0).stop 2023-11-24T08:07:16.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 -- 172.21.15.107:0/1405515979 shutdown_connections 2023-11-24T08:07:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/1405515979 >> v2:172.21.15.120:3300/0 conn(0x7f5c440ce6c0 0x7f5c440cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/1405515979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c4415c7f0 0x7f5c4406db20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 -- 172.21.15.107:0/1405515979 >> 172.21.15.107:0/1405515979 conn(0x7f5c4406a920 msgr2=0x7f5c4406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 -- 172.21.15.107:0/1405515979 shutdown_connections 2023-11-24T08:07:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.427+0000 7f5c4c57d640 1 -- 172.21.15.107:0/1405515979 wait complete. 2023-11-24T08:07:16.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.429+0000 7f5c4c57d640 1 Processor -- start 2023-11-24T08:07:16.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.429+0000 7f5c4c57d640 1 -- start start 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c440cf450 0x7f5c441e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4415c7f0 0x7f5c4415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441e7620 con 0x7f5c440ce6c0 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441e5930 con 0x7f5c440cf450 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.430+0000 7f5c4c57d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441e5c10 con 0x7f5c4415c7f0 2023-11-24T08:07:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58382/0 (socket says 172.21.15.107:58382) 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 -- 172.21.15.107:0/825784654 learned_addr learned my addr 172.21.15.107:0/825784654 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 -- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4415c7f0 msgr2=0x7f5c4415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4415c7f0 0x7f5c4415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 -- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c440cf450 msgr2=0x7f5c441e4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c440cf450 0x7f5c441e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 -- 172.21.15.107:0/825784654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c400094d0 con 0x7f5c440ce6c0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.431+0000 7f5c4a319640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5c40000c00 tx=0x7f5c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.432+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4003ad10 con 0x7f5c440ce6c0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.432+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4003ae70 con 0x7f5c440ce6c0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.432+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c441e5ef0 con 0x7f5c440ce6c0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.432+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40044610 con 0x7f5c440ce6c0 2023-11-24T08:07:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.432+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c44156920 con 0x7f5c440ce6c0 2023-11-24T08:07:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.433+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4406e040 con 0x7f5c440ce6c0 2023-11-24T08:07:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.434+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f5c4004d450 con 0x7f5c440ce6c0 2023-11-24T08:07:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.435+0000 7f5c3affd640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 0x7f5c2806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.435+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5c4002d040 con 0x7f5c440ce6c0 2023-11-24T08:07:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.435+0000 7f5c49b18640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 0x7f5c2806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.436+0000 7f5c49b18640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 0x7f5c2806d5d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c441f4b90 tx=0x7f5c34007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.441+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c4406e040 con 0x7f5c440ce6c0 2023-11-24T08:07:16.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.635+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c4406e040 con 0x7f5c440ce6c0 2023-11-24T08:07:16.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.637+0000 7f5c3affd640 1 -- 172.21.15.107:0/825784654 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f5c4406e040 con 0x7f5c440ce6c0 2023-11-24T08:07:16.639 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:16.639 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:09.751609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:16.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.639+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 msgr2=0x7f5c2806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.639+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 0x7f5c2806d5d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c441f4b90 tx=0x7f5c34007990 comp rx=0 tx=0).stop 2023-11-24T08:07:16.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.639+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 msgr2=0x7f5c441e70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.639+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5c40000c00 tx=0x7f5c400046c0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 shutdown_connections 2023-11-24T08:07:16.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2806b110 0x7f5c2806d5d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4415c7f0 0x7f5c4415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c440cf450 0x7f5c441e4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 --2- 172.21.15.107:0/825784654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c440ce6c0 0x7f5c441e70e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 >> 172.21.15.107:0/825784654 conn(0x7f5c4406a920 msgr2=0x7f5c4406c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:16.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 shutdown_connections 2023-11-24T08:07:16.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.640+0000 7f5c4c57d640 1 -- 172.21.15.107:0/825784654 wait complete. 2023-11-24T08:07:16.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:06:01.096943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:06:02.442773+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:07:16.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:07:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.873+0000 7f88fa875640 1 Processor -- start 2023-11-24T08:07:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.873+0000 7f88fa875640 1 -- start start 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88fa875640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88f415b440 0x7f88f40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88fa875640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40cbf20 0x7f88f40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88fa875640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f415eea0 con 0x7f88f40c57e0 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88fa875640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f40cc3d0 con 0x7f88f415b440 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88fa875640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f415b820 con 0x7f88f40cbf20 2023-11-24T08:07:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88f3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88f415b440 0x7f88f40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88f3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88f415b440 0x7f88f40cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58386/0 (socket says 172.21.15.107:58386) 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.875+0000 7f88f3fff640 1 -- 172.21.15.107:0/2464127709 learned_addr learned my addr 172.21.15.107:0/2464127709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.876+0000 7f88f3fff640 1 -- 172.21.15.107:0/2464127709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40cbf20 msgr2=0x7f88f40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.876+0000 7f88f3fff640 1 --2- 172.21.15.107:0/2464127709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40cbf20 0x7f88f40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.876+0000 7f88f3fff640 1 -- 172.21.15.107:0/2464127709 >> v1:172.21.15.120:6789/0 conn(0x7f88f40c57e0 legacy=0x7f88f40c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.876+0000 7f88f3fff640 1 -- 172.21.15.107:0/2464127709 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f406d600 con 0x7f88f415b440 2023-11-24T08:07:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.877+0000 7f88f3fff640 1 --2- 172.21.15.107:0/2464127709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f40cb9e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009870 tx=0x7f88e402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78277745ed730fba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.877+0000 7f88f37fe640 1 -- 172.21.15.107:0/2464127709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40317c0 con 0x7f88f415b440 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.877+0000 7f88f37fe640 1 -- 172.21.15.107:0/2464127709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4031920 con 0x7f88f415b440 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.877+0000 7f88f37fe640 1 -- 172.21.15.107:0/2464127709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4031cb0 con 0x7f88f415b440 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 -- 172.21.15.107:0/2464127709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 msgr2=0x7f88f40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 --2- 172.21.15.107:0/2464127709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f40cb9e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009870 tx=0x7f88e402b860 comp rx=0 tx=0).stop 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 -- 172.21.15.107:0/2464127709 shutdown_connections 2023-11-24T08:07:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 --2- 172.21.15.107:0/2464127709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40cbf20 0x7f88f40c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 --2- 172.21.15.107:0/2464127709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f40cb9e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 -- 172.21.15.107:0/2464127709 >> 172.21.15.107:0/2464127709 conn(0x7f88f40b0c50 msgr2=0x7f88f40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 -- 172.21.15.107:0/2464127709 shutdown_connections 2023-11-24T08:07:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.878+0000 7f88fa875640 1 -- 172.21.15.107:0/2464127709 wait complete. 2023-11-24T08:07:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.880+0000 7f88fa875640 1 Processor -- start 2023-11-24T08:07:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.880+0000 7f88fa875640 1 -- start start 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40c57e0 0x7f88f409a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f40cbf20 0x7f88f409b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f409a090 con 0x7f88f415b440 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f409a9f0 con 0x7f88f40cbf20 2023-11-24T08:07:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88fa875640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f409bde0 con 0x7f88f40c57e0 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88f8e12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88f8e12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58402/0 (socket says 172.21.15.107:58402) 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88f8e12640 1 -- 172.21.15.107:0/3148710943 learned_addr learned my addr 172.21.15.107:0/3148710943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88ebfff640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f40cbf20 0x7f88f409b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.881+0000 7f88f3fff640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40c57e0 0x7f88f409a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 -- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40c57e0 msgr2=0x7f88f409a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40c57e0 0x7f88f409a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 -- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f40cbf20 msgr2=0x7f88f409b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f40cbf20 0x7f88f409b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 -- 172.21.15.107:0/3148710943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40094d0 con 0x7f88f415b440 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f8e12640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c780 tx=0x7f88ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec003d80 con 0x7f88f415b440 2023-11-24T08:07:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec00ce90 con 0x7f88f415b440 2023-11-24T08:07:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0052b0 con 0x7f88f415b440 2023-11-24T08:07:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f409c0c0 con 0x7f88f415b440 2023-11-24T08:07:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.882+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f4099cb0 con 0x7f88f415b440 2023-11-24T08:07:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.883+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f409a1d0 con 0x7f88f415b440 2023-11-24T08:07:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.884+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f88ec005410 con 0x7f88f415b440 2023-11-24T08:07:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.885+0000 7f88f17fa640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 0x7f88d406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.885+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f88ec088f70 con 0x7f88f415b440 2023-11-24T08:07:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.885+0000 7f88f3fff640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 0x7f88d406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:16.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.886+0000 7f88f3fff640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 0x7f88d406d5d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88e4036000 tx=0x7f88e403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:16.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:16.890+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88f409a1d0 con 0x7f88f415b440 2023-11-24T08:07:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.085+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88f409a1d0 con 0x7f88f415b440 2023-11-24T08:07:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.085+0000 7f88f17fa640 1 -- 172.21.15.107:0/3148710943 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88f409a1d0 con 0x7f88f415b440 2023-11-24T08:07:17.088 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:07:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 msgr2=0x7f88d406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 0x7f88d406d5d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88e4036000 tx=0x7f88e403d040 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 msgr2=0x7f88f4155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c780 tx=0x7f88ec00cc40 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 shutdown_connections 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88d406b110 0x7f88d406d5d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88f415b440 0x7f88f4155db0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f40cbf20 0x7f88f409b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 --2- 172.21.15.107:0/3148710943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88f40c57e0 0x7f88f409a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.088+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 >> 172.21.15.107:0/3148710943 conn(0x7f88f40b0c50 msgr2=0x7f88f40b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.089+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 shutdown_connections 2023-11-24T08:07:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.089+0000 7f88fa875640 1 -- 172.21.15.107:0/3148710943 wait complete. 2023-11-24T08:07:17.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:07:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.324+0000 7fbf748d4640 1 Processor -- start 2023-11-24T08:07:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.324+0000 7fbf748d4640 1 -- start start 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf748d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf7015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf748d4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7015ec90 con 0x7fbf70072160 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf748d4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700cd530 con 0x7fbf700c49f0 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf748d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700c4ed0 con 0x7fbf7006de10 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf6e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbf70072160 0x7fbf7006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59850/0 (socket says 172.21.15.107:59850) 2023-11-24T08:07:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.326+0000 7fbf6e59c640 1 -- 172.21.15.107:0/1645893393 learned_addr learned my addr 172.21.15.107:0/1645893393 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:17.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf67fff640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf7015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952006883 0 0) 0x7fbf60001110 con 0x7fbf70072160 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c003520 con 0x7fbf70072160 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103703777 0 0) 0x7fbf5c003520 con 0x7fbf70072160 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 msgr2=0x7fbf7015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf7015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 >> v1:172.21.15.120:6789/0 conn(0x7fbf700c49f0 legacy=0x7fbf700705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700c51b0 con 0x7fbf70072160 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbf60001800 con 0x7fbf70072160 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 >> v1:172.21.15.107:6789/0 conn(0x7fbf70072160 legacy=0x7fbf7006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf5c0039a0 0x7fbf5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.327+0000 7fbf6dd9b640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6dd9b640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c004d00 0x7fbf5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700c4ed0 con 0x7fbf5c0039a0 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c00ba30 con 0x7fbf5c004d00 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c00bbf0 con 0x7fbf7006de10 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf5c0039a0 0x7fbf5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf67fff640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6ed9d640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c004d00 0x7fbf5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 -- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 msgr2=0x7fbf5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf7006de10 0x7fbf5c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 -- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c004d00 msgr2=0x7fbf5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c004d00 0x7fbf5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6e59c640 1 -- 172.21.15.107:0/1645893393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c003520 con 0x7fbf5c0039a0 2023-11-24T08:07:17.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.328+0000 7fbf6ed9d640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c004d00 0x7fbf5c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.329+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf5c0039a0 0x7fbf5c003dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c160 tx=0x7fbf5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99529ea39948cf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60005ad0 con 0x7fbf5c0039a0 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf60005c50 con 0x7fbf5c0039a0 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf6dd9b640 1 -- 172.21.15.107:0/1645893393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6002c7c0 con 0x7fbf5c0039a0 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 -- 172.21.15.107:0/1645893393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf5c0039a0 msgr2=0x7fbf5c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 --2- 172.21.15.107:0/1645893393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf5c0039a0 0x7fbf5c003dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c160 tx=0x7fbf5c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf6ed9d640 1 -- 172.21.15.107:0/1645893393 reap_dead start 2023-11-24T08:07:17.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 -- 172.21.15.107:0/1645893393 shutdown_connections 2023-11-24T08:07:17.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 -- 172.21.15.107:0/1645893393 >> 172.21.15.107:0/1645893393 conn(0x7fbf700b0c50 msgr2=0x7fbf700b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:17.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 -- 172.21.15.107:0/1645893393 shutdown_connections 2023-11-24T08:07:17.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.330+0000 7fbf748d4640 1 -- 172.21.15.107:0/1645893393 wait complete. 2023-11-24T08:07:17.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.332+0000 7fbf748d4640 1 Processor -- start 2023-11-24T08:07:17.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.332+0000 7fbf748d4640 1 -- start start 2023-11-24T08:07:17.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 0x7fbf70155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 0x7fbf7009c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70156420 con 0x7fbf7015c640 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015d000 con 0x7fbf5c0039a0 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf748d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700b4e40 con 0x7fbf701565b0 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.333+0000 7fbf6ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58418/0 (socket says 172.21.15.107:58418) 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 -- 172.21.15.107:0/2143225766 learned_addr learned my addr 172.21.15.107:0/2143225766 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 0x7fbf70155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf67fff640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 0x7fbf7009c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 -- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 msgr2=0x7fbf7009c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 0x7fbf7009c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 -- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 msgr2=0x7fbf70155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 0x7fbf70155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 -- 172.21.15.107:0/2143225766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7009c960 con 0x7fbf7015c640 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 0x7fbf70155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf67fff640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 0x7fbf7009c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.334+0000 7fbf6ed9d640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b51a0 tx=0x7fbf6800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.335+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68006d50 con 0x7fbf7015c640 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.335+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68003970 con 0x7fbf7015c640 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.335+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf700b4080 con 0x7fbf7015c640 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.335+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68003d60 con 0x7fbf7015c640 2023-11-24T08:07:17.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.335+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf700b4640 con 0x7fbf7015c640 2023-11-24T08:07:17.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.336+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7004e8b0 con 0x7fbf7015c640 2023-11-24T08:07:17.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.337+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fbf68005070 con 0x7fbf7015c640 2023-11-24T08:07:17.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.338+0000 7fbf66ffd640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 0x7fbf5006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.338+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbf68089a00 con 0x7fbf7015c640 2023-11-24T08:07:17.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.339+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 0x7fbf5006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.339+0000 7fbf6e59c640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 0x7fbf5006d600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf60034000 tx=0x7fbf60009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.343+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf68054630 con 0x7fbf7015c640 2023-11-24T08:07:17.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.535+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf7015dab0 con 0x7fbf7015c640 2023-11-24T08:07:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.535+0000 7fbf66ffd640 1 -- 172.21.15.107:0/2143225766 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf68057c50 con 0x7fbf7015c640 2023-11-24T08:07:17.537 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:07:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.538+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 msgr2=0x7fbf5006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.538+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 0x7fbf5006d600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf60034000 tx=0x7fbf60009030 comp rx=0 tx=0).stop 2023-11-24T08:07:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.538+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 msgr2=0x7fbf7015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.538+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b51a0 tx=0x7fbf6800dc00 comp rx=0 tx=0).stop 2023-11-24T08:07:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 shutdown_connections 2023-11-24T08:07:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf5006b140 0x7fbf5006d600 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7015c640 0x7fbf7015cac0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf701565b0 0x7fbf7009c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 --2- 172.21.15.107:0/2143225766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf5c0039a0 0x7fbf70155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 >> 172.21.15.107:0/2143225766 conn(0x7fbf700b0c50 msgr2=0x7fbf700cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 shutdown_connections 2023-11-24T08:07:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.539+0000 7fbf748d4640 1 -- 172.21.15.107:0/2143225766 wait complete. 2023-11-24T08:07:17.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:07:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.761+0000 7fc1425d5640 1 Processor -- start 2023-11-24T08:07:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.762+0000 7fc1425d5640 1 -- start start 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.763+0000 7fc1425d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.763+0000 7fc1425d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc13c0c59b0 0x7fc13c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.763+0000 7fc1425d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0cc3a0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.763+0000 7fc1425d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c15b820 con 0x7fc13c0c59b0 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.763+0000 7fc1425d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c15eea0 con 0x7fc13c15b440 2023-11-24T08:07:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc13c0c59b0 0x7fc13c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc13c0c59b0 0x7fc13c06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58434/0 (socket says 172.21.15.107:58434) 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 -- 172.21.15.107:0/990382344 learned_addr learned my addr 172.21.15.107:0/990382344 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc13bfff640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 -- 172.21.15.107:0/990382344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 msgr2=0x7fc13c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 -- 172.21.15.107:0/990382344 >> v1:172.21.15.120:6789/0 conn(0x7fc13c0cbf20 legacy=0x7fc13c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.764+0000 7fc140b72640 1 -- 172.21.15.107:0/990382344 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0abf30 con 0x7fc13c0c59b0 2023-11-24T08:07:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.765+0000 7fc140b72640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0c59b0 0x7fc13c06c5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc134009e50 tx=0x7fc13402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25bad307823ac41 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc13b7fe640 1 -- 172.21.15.107:0/990382344 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13402c9e0 con 0x7fc13c0c59b0 2023-11-24T08:07:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc13b7fe640 1 -- 172.21.15.107:0/990382344 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13402cb40 con 0x7fc13c0c59b0 2023-11-24T08:07:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc13b7fe640 1 -- 172.21.15.107:0/990382344 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1340343f0 con 0x7fc13c0c59b0 2023-11-24T08:07:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 -- 172.21.15.107:0/990382344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0c59b0 msgr2=0x7fc13c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0c59b0 0x7fc13c06c5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc134009e50 tx=0x7fc13402be40 comp rx=0 tx=0).stop 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 -- 172.21.15.107:0/990382344 shutdown_connections 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0c59b0 0x7fc13c06c5f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 --2- 172.21.15.107:0/990382344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.766+0000 7fc1425d5640 1 -- 172.21.15.107:0/990382344 >> 172.21.15.107:0/990382344 conn(0x7fc13c0b0c50 msgr2=0x7fc13c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.767+0000 7fc1425d5640 1 -- 172.21.15.107:0/990382344 shutdown_connections 2023-11-24T08:07:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.767+0000 7fc1425d5640 1 -- 172.21.15.107:0/990382344 wait complete. 2023-11-24T08:07:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.769+0000 7fc1425d5640 1 Processor -- start 2023-11-24T08:07:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.769+0000 7fc1425d5640 1 -- start start 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc13c0c59b0 0x7fc13c1f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b2940 con 0x7fc13c0cbf20 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1597c0 con 0x7fc13c0c59b0 2023-11-24T08:07:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc1425d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1f7660 con 0x7fc13c15b440 2023-11-24T08:07:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc133fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc133fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58448/0 (socket says 172.21.15.107:58448) 2023-11-24T08:07:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.770+0000 7fc133fff640 1 -- 172.21.15.107:0/3437606581 learned_addr learned my addr 172.21.15.107:0/3437606581 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc13bfff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc13c0c59b0 0x7fc13c1f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc140b72640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 -- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 msgr2=0x7fc13c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 -- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc13c0c59b0 msgr2=0x7fc13c1f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc13c0c59b0 0x7fc13c1f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 -- 172.21.15.107:0/3437606581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc134009b30 con 0x7fc13c0cbf20 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc140b72640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.771+0000 7fc133fff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c430 tx=0x7fc12400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.774+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011790 con 0x7fc13c0cbf20 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.774+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240118f0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.774+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011ce0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.774+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c1544e0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.774+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c15a420 con 0x7fc13c0cbf20 2023-11-24T08:07:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.776+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c0b2ab0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.777+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fc12401c460 con 0x7fc13c0cbf20 2023-11-24T08:07:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.778+0000 7fc1397fa640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 0x7fc11c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.778+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc1240899b0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.778+0000 7fc13bfff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 0x7fc11c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:17.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.779+0000 7fc13bfff640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 0x7fc11c06d5d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc12c006fd0 tx=0x7fc12c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.783+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc124054780 con 0x7fc13c0cbf20 2023-11-24T08:07:17.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.973+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc13c04e8b0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.974+0000 7fc1397fa640 1 -- 172.21.15.107:0/3437606581 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc124057da0 con 0x7fc13c0cbf20 2023-11-24T08:07:17.976 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 msgr2=0x7fc11c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 0x7fc11c06d5d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc12c006fd0 tx=0x7fc12c008040 comp rx=0 tx=0).stop 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 msgr2=0x7fc13c153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c430 tx=0x7fc12400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 shutdown_connections 2023-11-24T08:07:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11c06b110 0x7fc11c06d5d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13c15b440 0x7fc13c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c0cbf20 0x7fc13c153fa0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 --2- 172.21.15.107:0/3437606581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc13c0c59b0 0x7fc13c1f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 >> 172.21.15.107:0/3437606581 conn(0x7fc13c0b0c50 msgr2=0x7fc13c0c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 shutdown_connections 2023-11-24T08:07:17.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:17.977+0000 7fc1425d5640 1 -- 172.21.15.107:0/3437606581 wait complete. 2023-11-24T08:07:17.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:07:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.211+0000 7f8715487640 1 Processor -- start 2023-11-24T08:07:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.211+0000 7f8715487640 1 -- start start 2023-11-24T08:07:18.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f8715487640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f8715487640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87100c57e0 0x7f871006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f8715487640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871015eea0 con 0x7f871015b440 2023-11-24T08:07:18.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f8715487640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871015b820 con 0x7f87100c57e0 2023-11-24T08:07:18.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f8715487640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100abf30 con 0x7f87100cc0f0 2023-11-24T08:07:18.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f870f7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87100c57e0 0x7f871006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f870e7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f870e7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:55772/0 (socket says 172.21.15.107:55772) 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.213+0000 7f870e7fc640 1 -- 172.21.15.107:0/1189643212 learned_addr learned my addr 172.21.15.107:0/1189643212 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.214+0000 7f870e7fc640 1 -- 172.21.15.107:0/1189643212 >> v2:172.21.15.120:3300/0 conn(0x7f87100c57e0 msgr2=0x7f871006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.214+0000 7f870e7fc640 1 --2- 172.21.15.107:0/1189643212 >> v2:172.21.15.120:3300/0 conn(0x7f87100c57e0 0x7f871006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.214+0000 7f870e7fc640 1 -- 172.21.15.107:0/1189643212 >> v1:172.21.15.120:6789/0 conn(0x7f871015b440 legacy=0x7f87100cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.214+0000 7f870e7fc640 1 -- 172.21.15.107:0/1189643212 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871015f0f0 con 0x7f87100cc0f0 2023-11-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.216+0000 7f870e7fc640 1 --2- 172.21.15.107:0/1189643212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8704009870 tx=0x7f870402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc8041f61ed7034d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.216+0000 7f870dffb640 1 -- 172.21.15.107:0/1189643212 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704031640 con 0x7f87100cc0f0 2023-11-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.216+0000 7f870dffb640 1 -- 172.21.15.107:0/1189643212 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87040317a0 con 0x7f87100cc0f0 2023-11-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.216+0000 7f870dffb640 1 -- 172.21.15.107:0/1189643212 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704031b30 con 0x7f87100cc0f0 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 -- 172.21.15.107:0/1189643212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 msgr2=0x7f87100c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 --2- 172.21.15.107:0/1189643212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8704009870 tx=0x7f870402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 -- 172.21.15.107:0/1189643212 shutdown_connections 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 --2- 172.21.15.107:0/1189643212 >> v2:172.21.15.120:3300/0 conn(0x7f87100c57e0 0x7f871006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 --2- 172.21.15.107:0/1189643212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87100cc0f0 0x7f87100c52a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 -- 172.21.15.107:0/1189643212 >> 172.21.15.107:0/1189643212 conn(0x7f87100b0c50 msgr2=0x7f87100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 -- 172.21.15.107:0/1189643212 shutdown_connections 2023-11-24T08:07:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.217+0000 7f8715487640 1 -- 172.21.15.107:0/1189643212 wait complete. 2023-11-24T08:07:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.219+0000 7f8715487640 1 Processor -- start 2023-11-24T08:07:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.219+0000 7f8715487640 1 -- start start 2023-11-24T08:07:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87100cc0f0 0x7f87101f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f871015b440 0x7f87101f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f6ab0 con 0x7f87100c57e0 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871015cfd0 con 0x7f87100cc0f0 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f8715487640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f3ff0 con 0x7f871015b440 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f870effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f870effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46266/0 (socket says 172.21.15.107:46266) 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f870effd640 1 -- 172.21.15.107:0/2124255565 learned_addr learned my addr 172.21.15.107:0/2124255565 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f870f7fe640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f871015b440 0x7f87101f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.220+0000 7f870e7fc640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87100cc0f0 0x7f87101f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 -- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f871015b440 msgr2=0x7f87101f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f871015b440 0x7f87101f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 -- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87100cc0f0 msgr2=0x7f87101f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87100cc0f0 0x7f87101f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 -- 172.21.15.107:0/2124255565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040094d0 con 0x7f87100c57e0 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f870effd640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f870000c3c0 tx=0x7f870000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000052d0 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700005450 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700007260 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87101f42d0 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.221+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871015cc60 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.222+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871004e8b0 con 0x7f87100c57e0 2023-11-24T08:07:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.223+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f87000073c0 con 0x7f87100c57e0 2023-11-24T08:07:18.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.224+0000 7f86f77fe640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 0x7f86fc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:18.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.224+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8700088e70 con 0x7f87100c57e0 2023-11-24T08:07:18.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.224+0000 7f870e7fc640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 0x7f86fc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:18.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.225+0000 7f870e7fc640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 0x7f86fc06d600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8704036000 tx=0x7f870403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.229+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8700053b70 con 0x7f87100c57e0 2023-11-24T08:07:18.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.431+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f87101574f0 con 0x7f87100c57e0 2023-11-24T08:07:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.432+0000 7f86f77fe640 1 -- 172.21.15.107:0/2124255565 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8700057190 con 0x7f87100c57e0 2023-11-24T08:07:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.434+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 msgr2=0x7f86fc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.434+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 0x7f86fc06d600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8704036000 tx=0x7f870403c040 comp rx=0 tx=0).stop 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.434+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 msgr2=0x7f87101f6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.434+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f870000c3c0 tx=0x7f870000c880 comp rx=0 tx=0).stop 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 shutdown_connections 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f86fc06b140 0x7f86fc06d600 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f871015b440 0x7f87101f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87100cc0f0 0x7f87101f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 --2- 172.21.15.107:0/2124255565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87100c57e0 0x7f87101f6570 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 >> 172.21.15.107:0/2124255565 conn(0x7f87100b0c50 msgr2=0x7f87100cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 shutdown_connections 2023-11-24T08:07:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:18.435+0000 7f8715487640 1 -- 172.21.15.107:0/2124255565 wait complete. 2023-11-24T08:07:18.456 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:07:18.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:18.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:07:18.606 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:07:18.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:18.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:07:18.665 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:18.665 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:18.708 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:18.708 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:18.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.006+0000 7f4e500da640 1 Processor -- start 2023-11-24T08:07:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.006+0000 7f4e500da640 1 -- start start 2023-11-24T08:07:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e500da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e480cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e500da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e480cdfb0 0x7f4e480ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e500da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480cece0 con 0x7f4e4815c7f0 2023-11-24T08:07:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e500da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e48156010 con 0x7f4e480cdfb0 2023-11-24T08:07:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e500da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4806ce30 con 0x7f4e480cf450 2023-11-24T08:07:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.007+0000 7f4e4e677640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e480cdfb0 0x7f4e480ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e480cdfb0 0x7f4e480ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46276/0 (socket says 172.21.15.107:46276) 2023-11-24T08:07:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 -- 172.21.15.107:0/498623278 learned_addr learned my addr 172.21.15.107:0/498623278 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4d675640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e480cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 -- 172.21.15.107:0/498623278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 msgr2=0x7f4e480cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e480cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 -- 172.21.15.107:0/498623278 >> v1:172.21.15.107:6789/0 conn(0x7f4e4815c7f0 legacy=0x7f4e4806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4e677640 1 -- 172.21.15.107:0/498623278 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4806d040 con 0x7f4e480cdfb0 2023-11-24T08:07:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.008+0000 7f4e4ce74640 1 -- 172.21.15.107:0/498623278 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90796530 0 0) 0x7f4e480cece0 con 0x7f4e4815c7f0 2023-11-24T08:07:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.009+0000 7f4e4e677640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e480cdfb0 0x7f4e480ce440 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009e50 tx=0x7f4e4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf69e11282cb1d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e4ce74640 1 -- 172.21.15.107:0/498623278 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4002cb00 con 0x7f4e480cdfb0 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e4ce74640 1 -- 172.21.15.107:0/498623278 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4002cc60 con 0x7f4e480cdfb0 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e4ce74640 1 -- 172.21.15.107:0/498623278 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e400343f0 con 0x7f4e480cdfb0 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 -- 172.21.15.107:0/498623278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e480cdfb0 msgr2=0x7f4e480ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e480cdfb0 0x7f4e480ce440 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009e50 tx=0x7f4e4002bf60 comp rx=0 tx=0).stop 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 -- 172.21.15.107:0/498623278 shutdown_connections 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e480cdfb0 0x7f4e480ce440 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 --2- 172.21.15.107:0/498623278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e480cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 -- 172.21.15.107:0/498623278 >> 172.21.15.107:0/498623278 conn(0x7f4e4806a920 msgr2=0x7f4e4806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 -- 172.21.15.107:0/498623278 shutdown_connections 2023-11-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.010+0000 7f4e500da640 1 -- 172.21.15.107:0/498623278 wait complete. 2023-11-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.012+0000 7f4e500da640 1 Processor -- start 2023-11-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.012+0000 7f4e500da640 1 -- start start 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e480cdfb0 0x7f4e48150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e48158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4815e900 con 0x7f4e4815c7f0 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48151480 con 0x7f4e480cdfb0 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e500da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48150dd0 con 0x7f4e480cf450 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e4e677640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.013+0000 7f4e4e677640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46280/0 (socket says 172.21.15.107:46280) 2023-11-24T08:07:19.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 -- 172.21.15.107:0/4019076429 learned_addr learned my addr 172.21.15.107:0/4019076429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4d675640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e48158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 -- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 msgr2=0x7f4e48158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e48158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 -- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e480cdfb0 msgr2=0x7f4e48150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e480cdfb0 0x7f4e48150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 -- 172.21.15.107:0/4019076429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e40009b30 con 0x7f4e4815c7f0 2023-11-24T08:07:19.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e4e677640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036040 tx=0x7f4e40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.014+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40009510 con 0x7f4e4815c7f0 2023-11-24T08:07:19.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.015+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4003e450 con 0x7f4e4815c7f0 2023-11-24T08:07:19.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.015+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4003e800 con 0x7f4e4815c7f0 2023-11-24T08:07:19.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.015+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4815e340 con 0x7f4e4815c7f0 2023-11-24T08:07:19.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.015+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e48151110 con 0x7f4e4815c7f0 2023-11-24T08:07:19.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.016+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e48067470 con 0x7f4e4815c7f0 2023-11-24T08:07:19.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.017+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f4e4003e960 con 0x7f4e4815c7f0 2023-11-24T08:07:19.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.018+0000 7f4e3e7fc640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 0x7f4e2c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.018+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4e400ac800 con 0x7f4e4815c7f0 2023-11-24T08:07:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.018+0000 7f4e4de76640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 0x7f4e2c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.019+0000 7f4e4de76640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 0x7f4e2c06d600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4e44001560 tx=0x7f4e44007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.022+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e400774b0 con 0x7f4e4815c7f0 2023-11-24T08:07:19.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.215+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e480cd730 con 0x7f4e4815c7f0 2023-11-24T08:07:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.217+0000 7f4e3e7fc640 1 -- 172.21.15.107:0/4019076429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f4e4007aad0 con 0x7f4e4815c7f0 2023-11-24T08:07:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:19.220 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:09.751609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 msgr2=0x7f4e2c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 0x7f4e2c06d600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4e44001560 tx=0x7f4e44007930 comp rx=0 tx=0).stop 2023-11-24T08:07:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 msgr2=0x7f4e480aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036040 tx=0x7f4e40004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 shutdown_connections 2023-11-24T08:07:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e2c06b140 0x7f4e2c06d600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e4815c7f0 0x7f4e480aff40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e480cf450 0x7f4e48158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 --2- 172.21.15.107:0/4019076429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e480cdfb0 0x7f4e48150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 >> 172.21.15.107:0/4019076429 conn(0x7f4e4806a920 msgr2=0x7f4e480ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 shutdown_connections 2023-11-24T08:07:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.220+0000 7f4e500da640 1 -- 172.21.15.107:0/4019076429 wait complete. 2023-11-24T08:07:19.241 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:06:01.096943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:06:02.442773+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:07:19.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:07:19.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.444+0000 7f0313ba9640 1 Processor -- start 2023-11-24T08:07:19.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.444+0000 7f0313ba9640 1 -- start start 2023-11-24T08:07:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f030c0cfb90 0x7f030c161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f030c161fa0 0x7f030c06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0b2400 0x7f030c06f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c0cff70 con 0x7f030c161fa0 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c0afb20 con 0x7f030c0cfb90 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0313ba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c1612f0 con 0x7f030c0b2400 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0311144640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f030c161fa0 0x7f030c06ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0311144640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f030c161fa0 0x7f030c06ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46294/0 (socket says 172.21.15.107:46294) 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0311144640 1 -- 172.21.15.107:0/1627792714 learned_addr learned my addr 172.21.15.107:0/1627792714 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0312146640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0b2400 0x7f030c06f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.446+0000 7f0311945640 1 --2- 172.21.15.107:0/1627792714 >> v2:172.21.15.120:3300/0 conn(0x7f030c0cfb90 0x7f030c161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.447+0000 7f0311144640 1 -- 172.21.15.107:0/1627792714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0b2400 msgr2=0x7f030c06f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.447+0000 7f0311144640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0b2400 0x7f030c06f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.447+0000 7f0311144640 1 -- 172.21.15.107:0/1627792714 >> v2:172.21.15.120:3300/0 conn(0x7f030c0cfb90 msgr2=0x7f030c161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.447+0000 7f0311144640 1 --2- 172.21.15.107:0/1627792714 >> v2:172.21.15.120:3300/0 conn(0x7f030c0cfb90 0x7f030c161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.447+0000 7f0311144640 1 -- 172.21.15.107:0/1627792714 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c06a850 con 0x7f030c161fa0 2023-11-24T08:07:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.448+0000 7f0311144640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c161fa0 0x7f030c06ee50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f030c070d30 tx=0x7f030802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ef1f871244a108 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.448+0000 7f0310943640 1 -- 172.21.15.107:0/1627792714 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308030860 con 0x7f030c161fa0 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.448+0000 7f0310943640 1 -- 172.21.15.107:0/1627792714 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03080309c0 con 0x7f030c161fa0 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.448+0000 7f0310943640 1 -- 172.21.15.107:0/1627792714 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308030d50 con 0x7f030c161fa0 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 -- 172.21.15.107:0/1627792714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c161fa0 msgr2=0x7f030c06ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c161fa0 0x7f030c06ee50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f030c070d30 tx=0x7f030802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 -- 172.21.15.107:0/1627792714 shutdown_connections 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0b2400 0x7f030c06f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 --2- 172.21.15.107:0/1627792714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c161fa0 0x7f030c06ee50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 --2- 172.21.15.107:0/1627792714 >> v2:172.21.15.120:3300/0 conn(0x7f030c0cfb90 0x7f030c161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 -- 172.21.15.107:0/1627792714 >> 172.21.15.107:0/1627792714 conn(0x7f030c0999d0 msgr2=0x7f030c0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 -- 172.21.15.107:0/1627792714 shutdown_connections 2023-11-24T08:07:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.449+0000 7f0313ba9640 1 -- 172.21.15.107:0/1627792714 wait complete. 2023-11-24T08:07:19.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.451+0000 7f0313ba9640 1 Processor -- start 2023-11-24T08:07:19.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.451+0000 7f0313ba9640 1 -- start start 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 0x7f030c0c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0c8430 0x7f030c15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c9cb0 con 0x7f030c0cfb90 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c155f50 con 0x7f030c0b2400 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0313ba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c9820 con 0x7f030c0c8430 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46304/0 (socket says 172.21.15.107:46304) 2023-11-24T08:07:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 -- 172.21.15.107:0/3329078681 learned_addr learned my addr 172.21.15.107:0/3329078681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311945640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 0x7f030c0c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0312146640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0c8430 0x7f030c15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 -- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0c8430 msgr2=0x7f030c15e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0c8430 0x7f030c15e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 -- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 msgr2=0x7f030c0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 0x7f030c0c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311144640 1 -- 172.21.15.107:0/3329078681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080094d0 con 0x7f030c0cfb90 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.452+0000 7f0311945640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 0x7f030c0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f0311144640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f0308004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308039d60 con 0x7f030c0cfb90 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308044480 con 0x7f030c0cfb90 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308044830 con 0x7f030c0cfb90 2023-11-24T08:07:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c0c8c10 con 0x7f030c0cfb90 2023-11-24T08:07:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.453+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c0c91d0 con 0x7f030c0cfb90 2023-11-24T08:07:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.454+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c0c9990 con 0x7f030c0cfb90 2023-11-24T08:07:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.455+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f030804d3b0 con 0x7f030c0cfb90 2023-11-24T08:07:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.456+0000 7f03027fc640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 0x7f02f806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.456+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f030803e070 con 0x7f030c0cfb90 2023-11-24T08:07:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.456+0000 7f0311945640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 0x7f02f806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.457+0000 7f0311945640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 0x7f02f806d5d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02fc006fd0 tx=0x7f02fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.461+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0308077470 con 0x7f030c0cfb90 2023-11-24T08:07:19.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.652+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f030c04e8b0 con 0x7f030c0cfb90 2023-11-24T08:07:19.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.653+0000 7f03027fc640 1 -- 172.21.15.107:0/3329078681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03080ac020 con 0x7f030c0cfb90 2023-11-24T08:07:19.655 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:07:19.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 msgr2=0x7f02f806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 0x7f02f806d5d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02fc006fd0 tx=0x7f02fc008040 comp rx=0 tx=0).stop 2023-11-24T08:07:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 msgr2=0x7f030c0b5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f0308004730 comp rx=0 tx=0).stop 2023-11-24T08:07:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 shutdown_connections 2023-11-24T08:07:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02f806b110 0x7f02f806d5d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f030c0c8430 0x7f030c15e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f030c0cfb90 0x7f030c0b5a10 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 --2- 172.21.15.107:0/3329078681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f030c0b2400 0x7f030c0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.655+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 >> 172.21.15.107:0/3329078681 conn(0x7f030c0999d0 msgr2=0x7f030c0ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.656+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 shutdown_connections 2023-11-24T08:07:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.656+0000 7f0313ba9640 1 -- 172.21.15.107:0/3329078681 wait complete. 2023-11-24T08:07:19.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:07:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.890+0000 7fbc8213c640 1 Processor -- start 2023-11-24T08:07:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.890+0000 7fbc8213c640 1 -- start start 2023-11-24T08:07:19.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc8213c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc7c0cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc8213c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7c0cbd10 con 0x7fbc7c0aed70 2023-11-24T08:07:19.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc8213c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7c0cbff0 con 0x7fbc7c0ce8b0 2023-11-24T08:07:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc8213c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c15bce0 con 0x7fbc7c15c840 2023-11-24T08:07:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc7bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbc7c0aed70 0x7fbc7c15feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52762/0 (socket says 172.21.15.107:52762) 2023-11-24T08:07:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc7bfff640 1 -- 172.21.15.107:0/2773370072 learned_addr learned my addr 172.21.15.107:0/2773370072 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.892+0000 7fbc7affd640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc7c0cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051584191 0 0) 0x7fbc7c0cbd10 con 0x7fbc7c0aed70 2023-11-24T08:07:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64003520 con 0x7fbc7c0aed70 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2260216891 0 0) 0x7fbc64003520 con 0x7fbc7c0aed70 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 msgr2=0x7fbc7c0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc7c0cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 >> v1:172.21.15.120:6789/0 conn(0x7fbc7c0ce8b0 legacy=0x7fbc7c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0cc2d0 con 0x7fbc7c0aed70 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbc740019f0 con 0x7fbc7c0aed70 2023-11-24T08:07:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 >> v1:172.21.15.107:6789/0 conn(0x7fbc7c0aed70 legacy=0x7fbc7c15feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc640039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc64003ee0 0x7fbc640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6400ba30 con 0x7fbc64004d00 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6400bb70 con 0x7fbc64003ee0 2023-11-24T08:07:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.893+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6400bd30 con 0x7fbc7c15c840 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc640039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7affd640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc64003ee0 0x7fbc640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 -- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 msgr2=0x7fbc640039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc640039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 -- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc64003ee0 msgr2=0x7fbc640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc64003ee0 0x7fbc640047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7bfff640 1 -- 172.21.15.107:0/2773370072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64003520 con 0x7fbc64004d00 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.894+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c15c840 0x7fbc640039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.895+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc64005180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400a810 tx=0x7fbc640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3932e8b895ce6ccd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.895+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74034630 con 0x7fbc64004d00 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.895+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7402f070 con 0x7fbc64004d00 2023-11-24T08:07:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.895+0000 7fbc7a7fc640 1 -- 172.21.15.107:0/2773370072 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740349d0 con 0x7fbc64004d00 2023-11-24T08:07:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 -- 172.21.15.107:0/2773370072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 msgr2=0x7fbc64005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2773370072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc64005180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400a810 tx=0x7fbc640057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc7bfff640 1 -- 172.21.15.107:0/2773370072 reap_dead start 2023-11-24T08:07:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 -- 172.21.15.107:0/2773370072 shutdown_connections 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 -- 172.21.15.107:0/2773370072 >> 172.21.15.107:0/2773370072 conn(0x7fbc7c0cf900 msgr2=0x7fbc7c0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 -- 172.21.15.107:0/2773370072 shutdown_connections 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.896+0000 7fbc8213c640 1 -- 172.21.15.107:0/2773370072 wait complete. 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.898+0000 7fbc8213c640 1 Processor -- start 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.898+0000 7fbc8213c640 1 -- start start 2023-11-24T08:07:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c15cda0 0x7fbc7c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c149f10 0x7fbc7c14a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0b4720 con 0x7fbc64004d00 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c15d740 con 0x7fbc7c15cda0 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc8213c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c14a900 con 0x7fbc7c149f10 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46312/0 (socket says 172.21.15.107:46312) 2023-11-24T08:07:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/2713494376 learned_addr learned my addr 172.21.15.107:0/2713494376 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c149f10 msgr2=0x7fbc7c14a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c149f10 0x7fbc7c14a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c15cda0 msgr2=0x7fbc7c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c15cda0 0x7fbc7c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.899+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/2713494376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c15d980 con 0x7fbc64004d00 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d710 tx=0x7fbc6c00dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c005480 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00de20 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c003850 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7c15dc60 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.900+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7c15e220 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.901+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7c15b700 con 0x7fbc64004d00 2023-11-24T08:07:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.902+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fbc6c0039b0 con 0x7fbc64004d00 2023-11-24T08:07:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.903+0000 7fbc63fff640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.903+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbc6c089c40 con 0x7fbc64004d00 2023-11-24T08:07:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.903+0000 7fbc7affd640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.904+0000 7fbc7affd640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 0x7fbc6806d5d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cff70 tx=0x7fbc70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:19.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:19.908+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc7c15b700 con 0x7fbc64004d00 2023-11-24T08:07:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.084+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc7c15b700 con 0x7fbc64004d00 2023-11-24T08:07:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.085+0000 7fbc63fff640 1 -- 172.21.15.107:0/2713494376 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc7c15b700 con 0x7fbc64004d00 2023-11-24T08:07:20.087 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:07:20.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 msgr2=0x7fbc6806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 0x7fbc6806d5d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0cff70 tx=0x7fbc70008040 comp rx=0 tx=0).stop 2023-11-24T08:07:20.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 msgr2=0x7fbc7c0add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d710 tx=0x7fbc6c00dbd0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 shutdown_connections 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c149f10 0x7fbc7c14a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c15cda0 0x7fbc7c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 --2- 172.21.15.107:0/2713494376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc64004d00 0x7fbc7c0add60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 >> 172.21.15.107:0/2713494376 conn(0x7fbc7c0cf900 msgr2=0x7fbc7c15bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 shutdown_connections 2023-11-24T08:07:20.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.087+0000 7fbc8213c640 1 -- 172.21.15.107:0/2713494376 wait complete. 2023-11-24T08:07:20.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:07:20.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.317+0000 7f3a5ae2d640 1 Processor -- start 2023-11-24T08:07:20.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.317+0000 7f3a5ae2d640 1 -- start start 2023-11-24T08:07:20.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a5415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3a540c49f0 0x7f3a5406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a5406dc40 0x7f3a540cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5406e0f0 con 0x7f3a5406dc40 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5415ec90 con 0x7f3a540c49f0 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a5ae2d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540cd530 con 0x7f3a54072160 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.319+0000 7f3a593ca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a5406dc40 0x7f3a540cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a5406dc40 0x7f3a540cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46324/0 (socket says 172.21.15.107:46324) 2023-11-24T08:07:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 -- 172.21.15.107:0/1698312280 learned_addr learned my addr 172.21.15.107:0/1698312280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a53fff640 1 --2- 172.21.15.107:0/1698312280 >> v2:172.21.15.120:3300/0 conn(0x7f3a540c49f0 0x7f3a5406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a58bc9640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a5415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 -- 172.21.15.107:0/1698312280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 msgr2=0x7f3a5415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a5415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 -- 172.21.15.107:0/1698312280 >> v2:172.21.15.120:3300/0 conn(0x7f3a540c49f0 msgr2=0x7f3a5406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 --2- 172.21.15.107:0/1698312280 >> v2:172.21.15.120:3300/0 conn(0x7f3a540c49f0 0x7f3a5406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a593ca640 1 -- 172.21.15.107:0/1698312280 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540c4ed0 con 0x7f3a5406dc40 2023-11-24T08:07:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.320+0000 7f3a58bc9640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a5415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.322+0000 7f3a593ca640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a5406dc40 0x7f3a540cd750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a540ccb10 tx=0x7f3a4c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dcdb8f3632f298 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.322+0000 7f3a537fe640 1 -- 172.21.15.107:0/1698312280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c02ca70 con 0x7f3a5406dc40 2023-11-24T08:07:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.322+0000 7f3a537fe640 1 -- 172.21.15.107:0/1698312280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c02cbd0 con 0x7f3a5406dc40 2023-11-24T08:07:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.322+0000 7f3a537fe640 1 -- 172.21.15.107:0/1698312280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c0343f0 con 0x7f3a5406dc40 2023-11-24T08:07:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/1698312280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a5406dc40 msgr2=0x7f3a540cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a5406dc40 0x7f3a540cd750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a540ccb10 tx=0x7f3a4c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/1698312280 shutdown_connections 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a5406dc40 0x7f3a540cd750 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/1698312280 >> v2:172.21.15.120:3300/0 conn(0x7f3a540c49f0 0x7f3a5406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/1698312280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a5415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/1698312280 >> 172.21.15.107:0/1698312280 conn(0x7f3a540b0c50 msgr2=0x7f3a540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/1698312280 shutdown_connections 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.323+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/1698312280 wait complete. 2023-11-24T08:07:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.325+0000 7f3a5ae2d640 1 Processor -- start 2023-11-24T08:07:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.325+0000 7f3a5ae2d640 1 -- start start 2023-11-24T08:07:20.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5406dc40 0x7f3a54152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a540cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54152e80 con 0x7f3a540c49f0 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540cbc20 con 0x7f3a5406dc40 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a5ae2d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540cb010 con 0x7f3a54072160 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a593ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a593ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46336/0 (socket says 172.21.15.107:46336) 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a593ca640 1 -- 172.21.15.107:0/2064007230 learned_addr learned my addr 172.21.15.107:0/2064007230 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.326+0000 7f3a58bc9640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5406dc40 0x7f3a54152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a53fff640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a540cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 -- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 msgr2=0x7f3a540cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a540cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 -- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5406dc40 msgr2=0x7f3a54152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5406dc40 0x7f3a54152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 -- 172.21.15.107:0/2064007230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c009b30 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a593ca640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036040 tx=0x7f3a4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c009510 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c03e450 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c03e800 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.327+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a540cb2f0 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.328+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a540cb8b0 con 0x7f3a540c49f0 2023-11-24T08:07:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.329+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5404e8b0 con 0x7f3a540c49f0 2023-11-24T08:07:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.330+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f3a4c03e960 con 0x7f3a540c49f0 2023-11-24T08:07:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.331+0000 7f3a517fa640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 0x7f3a3806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.331+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3a4c0ac7d0 con 0x7f3a540c49f0 2023-11-24T08:07:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.331+0000 7f3a58bc9640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 0x7f3a3806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.332+0000 7f3a58bc9640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 0x7f3a3806d5d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3a540c9540 tx=0x7f3a48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.335+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a4c077680 con 0x7f3a540c49f0 2023-11-24T08:07:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.523+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a5415a500 con 0x7f3a540c49f0 2023-11-24T08:07:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.524+0000 7f3a517fa640 1 -- 172.21.15.107:0/2064007230 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a4c07aca0 con 0x7f3a540c49f0 2023-11-24T08:07:20.526 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 msgr2=0x7f3a3806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 0x7f3a3806d5d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3a540c9540 tx=0x7f3a48008040 comp rx=0 tx=0).stop 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 msgr2=0x7f3a540cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036040 tx=0x7f3a4c004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 shutdown_connections 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a3806b110 0x7f3a3806d5d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a540c49f0 0x7f3a540cc300 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a54072160 0x7f3a540cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 --2- 172.21.15.107:0/2064007230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5406dc40 0x7f3a54152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 >> 172.21.15.107:0/2064007230 conn(0x7f3a540b0c50 msgr2=0x7f3a540abff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 shutdown_connections 2023-11-24T08:07:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.527+0000 7f3a5ae2d640 1 -- 172.21.15.107:0/2064007230 wait complete. 2023-11-24T08:07:20.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:07:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.762+0000 7f71fdbe2640 1 Processor -- start 2023-11-24T08:07:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.762+0000 7f71fdbe2640 1 -- start start 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.763+0000 7f71fdbe2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71f815b1d0 0x7f71f8071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fdbe2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f71f80b1250 0x7f71f815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fdbe2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f80ca0e0 con 0x7f71f8070550 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fdbe2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f8072800 con 0x7f71f80b1250 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fdbe2640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f80c97b0 con 0x7f71f815b1d0 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fd3e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f71f80b1250 0x7f71f815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fd3e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f71f80b1250 0x7f71f815edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46352/0 (socket says 172.21.15.107:46352) 2023-11-24T08:07:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71f7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71f8070550 0x7f71f80b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52768/0 (socket says 172.21.15.107:52768) 2023-11-24T08:07:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fd3e1640 1 -- 172.21.15.107:0/2018347681 learned_addr learned my addr 172.21.15.107:0/2018347681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fcbe0640 1 --2- 172.21.15.107:0/2018347681 >> v2:172.21.15.120:3300/0 conn(0x7f71f815b1d0 0x7f71f8071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fd3e1640 1 -- 172.21.15.107:0/2018347681 >> v2:172.21.15.120:3300/0 conn(0x7f71f815b1d0 msgr2=0x7f71f8071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.764+0000 7f71fd3e1640 1 --2- 172.21.15.107:0/2018347681 >> v2:172.21.15.120:3300/0 conn(0x7f71f815b1d0 0x7f71f8071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.765+0000 7f71fd3e1640 1 -- 172.21.15.107:0/2018347681 >> v1:172.21.15.107:6789/0 conn(0x7f71f8070550 legacy=0x7f71f80b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.765+0000 7f71fd3e1640 1 -- 172.21.15.107:0/2018347681 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80c9a90 con 0x7f71f80b1250 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.765+0000 7f71f77fe640 1 -- 172.21.15.107:0/2018347681 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180973216 0 0) 0x7f71f80ca0e0 con 0x7f71f8070550 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71fd3e1640 1 --2- 172.21.15.107:0/2018347681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f80b1250 0x7f71f815edd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f71f806e7a0 tx=0x7f71f002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3dda3fce939e0a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71f77fe640 1 -- 172.21.15.107:0/2018347681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f002cb20 con 0x7f71f80b1250 2023-11-24T08:07:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71f77fe640 1 -- 172.21.15.107:0/2018347681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f002cc80 con 0x7f71f80b1250 2023-11-24T08:07:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71f77fe640 1 -- 172.21.15.107:0/2018347681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f00343f0 con 0x7f71f80b1250 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2018347681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f80b1250 msgr2=0x7f71f815edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.766+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2018347681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f80b1250 0x7f71f815edd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f71f806e7a0 tx=0x7f71f002bf80 comp rx=0 tx=0).stop 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2018347681 shutdown_connections 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2018347681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f80b1250 0x7f71f815edd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2018347681 >> v2:172.21.15.120:3300/0 conn(0x7f71f815b1d0 0x7f71f8071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2018347681 >> 172.21.15.107:0/2018347681 conn(0x7f71f80cbb40 msgr2=0x7f71f80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2018347681 shutdown_connections 2023-11-24T08:07:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.767+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2018347681 wait complete. 2023-11-24T08:07:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.768+0000 7f71fdbe2640 1 Processor -- start 2023-11-24T08:07:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.769+0000 7f71fdbe2640 1 -- start start 2023-11-24T08:07:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71f80b1250 0x7f71f80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71f815b1d0 0x7f71f8159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81ed410 con 0x7f71f8070550 2023-11-24T08:07:20.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80b2200 con 0x7f71f815b1d0 2023-11-24T08:07:20.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fdbe2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80b24a0 con 0x7f71f80b1250 2023-11-24T08:07:20.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46368/0 (socket says 172.21.15.107:46368) 2023-11-24T08:07:20.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 -- 172.21.15.107:0/2630943712 learned_addr learned my addr 172.21.15.107:0/2630943712 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:20.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 -- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71f80b1250 msgr2=0x7f71f80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71f80b1250 0x7f71f80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 -- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71f815b1d0 msgr2=0x7f71f8159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71f815b1d0 0x7f71f8159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.770+0000 7f71fcbe0640 1 -- 172.21.15.107:0/2630943712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f0009b30 con 0x7f71f8070550 2023-11-24T08:07:20.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71fcbe0640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d5d0 tx=0x7f71ec00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec0125f0 con 0x7f71f8070550 2023-11-24T08:07:20.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec012770 con 0x7f71f8070550 2023-11-24T08:07:20.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec012b20 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f80b2780 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.771+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f80b2d40 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.772+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f804e840 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.773+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f71ec012c80 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.774+0000 7f71f57fa640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 0x7f71dc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.774+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f71ec089fe0 con 0x7f71f8070550 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.774+0000 7f71f7fff640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 0x7f71dc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.775+0000 7f71f7fff640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 0x7f71dc06d5d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f71e4007fd0 tx=0x7f71e4007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.779+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71ec054ce0 con 0x7f71f8070550 2023-11-24T08:07:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.984+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f71f80c2800 con 0x7f71f8070550 2023-11-24T08:07:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.984+0000 7f71f57fa640 1 -- 172.21.15.107:0/2630943712 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f71ec058300 con 0x7f71f8070550 2023-11-24T08:07:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 msgr2=0x7f71dc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 0x7f71dc06d5d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f71e4007fd0 tx=0x7f71e4007920 comp rx=0 tx=0).stop 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 msgr2=0x7f71f81f2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d5d0 tx=0x7f71ec00da90 comp rx=0 tx=0).stop 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 shutdown_connections 2023-11-24T08:07:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71dc06b110 0x7f71dc06d5d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71f815b1d0 0x7f71f8159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71f80b1250 0x7f71f80b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 --2- 172.21.15.107:0/2630943712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71f8070550 0x7f71f81f2f00 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 >> 172.21.15.107:0/2630943712 conn(0x7f71f80cbb40 msgr2=0x7f71f809b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 shutdown_connections 2023-11-24T08:07:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:20.987+0000 7f71fdbe2640 1 -- 172.21.15.107:0/2630943712 wait complete. 2023-11-24T08:07:21.007 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:07:21.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:21.007 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:07:21.125 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:07:21.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:21.125 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:07:21.177 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:21.177 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:21.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:07:21.235 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:21.235 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:21.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:07:21.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:21.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.516+0000 7f40567a9640 1 Processor -- start 2023-11-24T08:07:21.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.516+0000 7f40567a9640 1 -- start start 2023-11-24T08:07:21.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f40567a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f405006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f40567a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500cece0 con 0x7f40500cf450 2023-11-24T08:07:21.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f40567a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405006d040 con 0x7f40500ce6c0 2023-11-24T08:07:21.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f40567a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4050156010 con 0x7f405015c7f0 2023-11-24T08:07:21.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f4047fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f40500cf450 0x7f40500cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52774/0 (socket says 172.21.15.107:52774) 2023-11-24T08:07:21.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.518+0000 7f4047fff640 1 -- 172.21.15.107:0/4006226625 learned_addr learned my addr 172.21.15.107:0/4006226625 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:21.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404ffff640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f405006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865720083 0 0) 0x7f40500cece0 con 0x7f40500cf450 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c003520 con 0x7f40500cf450 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597156078 0 0) 0x7f403c003520 con 0x7f40500cf450 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 msgr2=0x7f405006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f405006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 >> v1:172.21.15.120:6789/0 conn(0x7f40500ce6c0 legacy=0x7f40500714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405006d320 con 0x7f40500cf450 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4038001670 con 0x7f40500cf450 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 >> v1:172.21.15.107:6789/0 conn(0x7f40500cf450 legacy=0x7f40500cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f403c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c003f20 0x7f403c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f403c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4050156010 con 0x7f403c004d00 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c00fe70 con 0x7f403c003f20 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.519+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c010030 con 0x7f405015c7f0 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f403c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4047fff640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c003f20 0x7f403c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f404ffff640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f403c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 -- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 msgr2=0x7f403c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f403c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 -- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c003f20 msgr2=0x7f403c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c003f20 0x7f403c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f4054d46640 1 -- 172.21.15.107:0/4006226625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c003520 con 0x7f403c004d00 2023-11-24T08:07:21.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.520+0000 7f404ffff640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015c7f0 0x7f403c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:21.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.521+0000 7f4054d46640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f403c005180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4048001e50 tx=0x7f403c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c2130d47dce170 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048003b70 con 0x7f403c004d00 2023-11-24T08:07:21.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048003cd0 con 0x7f403c004d00 2023-11-24T08:07:21.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f404f7fe640 1 -- 172.21.15.107:0/4006226625 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048005310 con 0x7f403c004d00 2023-11-24T08:07:21.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 -- 172.21.15.107:0/4006226625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 msgr2=0x7f403c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 --2- 172.21.15.107:0/4006226625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f403c005180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4048001e50 tx=0x7f403c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f4054d46640 1 -- 172.21.15.107:0/4006226625 reap_dead start 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 -- 172.21.15.107:0/4006226625 shutdown_connections 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 -- 172.21.15.107:0/4006226625 >> 172.21.15.107:0/4006226625 conn(0x7f405006a920 msgr2=0x7f405006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 -- 172.21.15.107:0/4006226625 shutdown_connections 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.522+0000 7f40567a9640 1 -- 172.21.15.107:0/4006226625 wait complete. 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.524+0000 7f40567a9640 1 Processor -- start 2023-11-24T08:07:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.524+0000 7f40567a9640 1 -- start start 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015cc50 0x7f405015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40500adbe0 0x7f40500ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015d5f0 con 0x7f403c004d00 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500af990 con 0x7f40500adbe0 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f40567a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40500aed80 con 0x7f405015cc50 2023-11-24T08:07:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f404ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f404ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46394/0 (socket says 172.21.15.107:46394) 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.525+0000 7f404ffff640 1 -- 172.21.15.107:0/619908027 learned_addr learned my addr 172.21.15.107:0/619908027 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 -- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015cc50 msgr2=0x7f405015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015cc50 0x7f405015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 -- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40500adbe0 msgr2=0x7f40500ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40500adbe0 0x7f40500ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 -- 172.21.15.107:0/619908027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500af060 con 0x7f403c004d00 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404ffff640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f40400108b0 tx=0x7f4040010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404000eb60 con 0x7f403c004d00 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404000ecc0 con 0x7f403c004d00 2023-11-24T08:07:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40500af340 con 0x7f403c004d00 2023-11-24T08:07:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404001c400 con 0x7f403c004d00 2023-11-24T08:07:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.526+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40500ae5d0 con 0x7f403c004d00 2023-11-24T08:07:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.528+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405015de50 con 0x7f403c004d00 2023-11-24T08:07:21.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.529+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f404001c560 con 0x7f403c004d00 2023-11-24T08:07:21.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.530+0000 7f404d7fa640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 0x7f403006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.530+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f404008a040 con 0x7f403c004d00 2023-11-24T08:07:21.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.530+0000 7f4047fff640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 0x7f403006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.531+0000 7f4047fff640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 0x7f403006d600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4038001eb0 tx=0x7f4038001d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.534+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40400bce90 con 0x7f403c004d00 2023-11-24T08:07:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.751+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40500cdf30 con 0x7f403c004d00 2023-11-24T08:07:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.752+0000 7f404d7fa640 1 -- 172.21.15.107:0/619908027 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f4040054d40 con 0x7f403c004d00 2023-11-24T08:07:21.754 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:21.755 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:07:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 15 2023-11-24T08:07:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.754+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 msgr2=0x7f403006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.754+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 0x7f403006d600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4038001eb0 tx=0x7f4038001d60 comp rx=0 tx=0).stop 2023-11-24T08:07:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.754+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 msgr2=0x7f40501453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f40400108b0 tx=0x7f4040010d70 comp rx=0 tx=0).stop 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 shutdown_connections 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f403006b140 0x7f403006d600 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40500adbe0 0x7f40500ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f405015cc50 0x7f405015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 --2- 172.21.15.107:0/619908027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c004d00 0x7f40501453c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 >> 172.21.15.107:0/619908027 conn(0x7f405006a920 msgr2=0x7f4050159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 shutdown_connections 2023-11-24T08:07:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.755+0000 7f40567a9640 1 -- 172.21.15.107:0/619908027 wait complete. 2023-11-24T08:07:21.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.969+0000 7ff6410bb640 1 Processor -- start 2023-11-24T08:07:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.969+0000 7ff6410bb640 1 -- start start 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff6410bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff63c15c7f0 0x7ff63c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff6410bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff63c0ce6c0 0x7ff63c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff6410bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0cece0 con 0x7ff63c0cf450 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff6410bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c0cfe40 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff6410bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c0ceaf0 con 0x7ff63c15c7f0 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.970+0000 7ff63b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff63c0ce6c0 0x7ff63c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff63c0ce6c0 0x7ff63c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46408/0 (socket says 172.21.15.107:46408) 2023-11-24T08:07:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 -- 172.21.15.107:0/2801639733 learned_addr learned my addr 172.21.15.107:0/2801639733 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 -- 172.21.15.107:0/2801639733 >> v2:172.21.15.120:3300/0 conn(0x7ff63c15c7f0 msgr2=0x7ff63c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 --2- 172.21.15.107:0/2801639733 >> v2:172.21.15.120:3300/0 conn(0x7ff63c15c7f0 0x7ff63c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 -- 172.21.15.107:0/2801639733 >> v1:172.21.15.120:6789/0 conn(0x7ff63c0cf450 legacy=0x7ff63c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.971+0000 7ff63b59e640 1 -- 172.21.15.107:0/2801639733 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c06d040 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.972+0000 7ff63b59e640 1 --2- 172.21.15.107:0/2801639733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c0ce6c0 0x7ff63c0cf900 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0d0130 tx=0x7ff63402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bc1e6b19f3ebef server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.972+0000 7ff63a59c640 1 -- 172.21.15.107:0/2801639733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63402ca80 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.972+0000 7ff63a59c640 1 -- 172.21.15.107:0/2801639733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63402cbe0 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.972+0000 7ff63a59c640 1 -- 172.21.15.107:0/2801639733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340343f0 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 -- 172.21.15.107:0/2801639733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c0ce6c0 msgr2=0x7ff63c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 --2- 172.21.15.107:0/2801639733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c0ce6c0 0x7ff63c0cf900 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0d0130 tx=0x7ff63402bee0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 -- 172.21.15.107:0/2801639733 shutdown_connections 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 --2- 172.21.15.107:0/2801639733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c0ce6c0 0x7ff63c0cf900 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 --2- 172.21.15.107:0/2801639733 >> v2:172.21.15.120:3300/0 conn(0x7ff63c15c7f0 0x7ff63c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 -- 172.21.15.107:0/2801639733 >> 172.21.15.107:0/2801639733 conn(0x7ff63c06a920 msgr2=0x7ff63c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 -- 172.21.15.107:0/2801639733 shutdown_connections 2023-11-24T08:07:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.973+0000 7ff6410bb640 1 -- 172.21.15.107:0/2801639733 wait complete. 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.975+0000 7ff6410bb640 1 Processor -- start 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.975+0000 7ff6410bb640 1 -- start start 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c0ce6c0 0x7ff63c150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c0cf450 0x7ff63c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c151410 con 0x7ff63c15c7f0 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c159e40 con 0x7ff63c0cf450 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff6410bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0b3fd0 con 0x7ff63c0ce6c0 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46412/0 (socket says 172.21.15.107:46412) 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 -- 172.21.15.107:0/1963095924 learned_addr learned my addr 172.21.15.107:0/1963095924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 -- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c0ce6c0 msgr2=0x7ff63c150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c0ce6c0 0x7ff63c150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 -- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c0cf450 msgr2=0x7ff63c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c0cf450 0x7ff63c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.976+0000 7ff63b59e640 1 -- 172.21.15.107:0/1963095924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff634009b30 con 0x7ff63c15c7f0 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff63b59e640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff634036040 tx=0x7ff634004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634009510 con 0x7ff63c15c7f0 2023-11-24T08:07:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63403e450 con 0x7ff63c15c7f0 2023-11-24T08:07:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63403e7e0 con 0x7ff63c15c7f0 2023-11-24T08:07:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c159b60 con 0x7ff63c15c7f0 2023-11-24T08:07:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.977+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c0ad420 con 0x7ff63c15c7f0 2023-11-24T08:07:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.978+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c0ae6a0 con 0x7ff63c15c7f0 2023-11-24T08:07:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.979+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7ff63402cd50 con 0x7ff63c15c7f0 2023-11-24T08:07:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.980+0000 7ff6337fe640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 0x7ff62006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.980+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff6340ac410 con 0x7ff63c15c7f0 2023-11-24T08:07:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.980+0000 7ff63ad9d640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 0x7ff62006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.981+0000 7ff63ad9d640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 0x7ff62006d600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0ae0b0 tx=0x7ff62c006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:21.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:21.984+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6340770c0 con 0x7ff63c15c7f0 2023-11-24T08:07:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.161+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff63c067470 con 0x7ff63c15c7f0 2023-11-24T08:07:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.164+0000 7ff6337fe640 1 -- 172.21.15.107:0/1963095924 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7ff63407a6e0 con 0x7ff63c15c7f0 2023-11-24T08:07:22.166 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:22.167 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:09.751609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:22.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 msgr2=0x7ff62006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 0x7ff62006d600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0ae0b0 tx=0x7ff62c006bf0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 msgr2=0x7ff63c0af3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff634036040 tx=0x7ff634004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 shutdown_connections 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff62006b140 0x7ff62006d600 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff63c15c7f0 0x7ff63c0af3e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c0cf450 0x7ff63c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 --2- 172.21.15.107:0/1963095924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c0ce6c0 0x7ff63c150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 >> 172.21.15.107:0/1963095924 conn(0x7ff63c06a920 msgr2=0x7ff63c0cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 shutdown_connections 2023-11-24T08:07:22.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.167+0000 7ff6410bb640 1 -- 172.21.15.107:0/1963095924 wait complete. 2023-11-24T08:07:22.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:06:01.096943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:06:02.442773+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:07:22.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:07:22.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.393+0000 7fb381d6f640 1 Processor -- start 2023-11-24T08:07:22.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.394+0000 7fb381d6f640 1 -- start start 2023-11-24T08:07:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb381d6f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb381d6f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb37c0cbf20 0x7fb37c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb381d6f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37c0cc3a0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb381d6f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c15b820 con 0x7fb37c0cbf20 2023-11-24T08:07:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb381d6f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c15eea0 con 0x7fb37c15b440 2023-11-24T08:07:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.395+0000 7fb37affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb37c0cbf20 0x7fb37c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb37c0cbf20 0x7fb37c0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46416/0 (socket says 172.21.15.107:46416) 2023-11-24T08:07:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 -- 172.21.15.107:0/2488361411 learned_addr learned my addr 172.21.15.107:0/2488361411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37b7fe640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 -- 172.21.15.107:0/2488361411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 msgr2=0x7fb37c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 -- 172.21.15.107:0/2488361411 >> v1:172.21.15.107:6789/0 conn(0x7fb37c0c57e0 legacy=0x7fb37c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37affd640 1 -- 172.21.15.107:0/2488361411 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c06d600 con 0x7fb37c0cbf20 2023-11-24T08:07:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.396+0000 7fb37b7fe640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:22.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.397+0000 7fb37affd640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0cbf20 0x7fb37c0c52a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb364009870 tx=0x7fb36402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdbfcd1bff6f958 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb37a7fc640 1 -- 172.21.15.107:0/2488361411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364031640 con 0x7fb37c0cbf20 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb37a7fc640 1 -- 172.21.15.107:0/2488361411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640317a0 con 0x7fb37c0cbf20 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb37a7fc640 1 -- 172.21.15.107:0/2488361411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364031b30 con 0x7fb37c0cbf20 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 -- 172.21.15.107:0/2488361411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0cbf20 msgr2=0x7fb37c0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0cbf20 0x7fb37c0c52a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb364009870 tx=0x7fb36402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 -- 172.21.15.107:0/2488361411 shutdown_connections 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0cbf20 0x7fb37c0c52a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 --2- 172.21.15.107:0/2488361411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.398+0000 7fb381d6f640 1 -- 172.21.15.107:0/2488361411 >> 172.21.15.107:0/2488361411 conn(0x7fb37c0b0c50 msgr2=0x7fb37c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.399+0000 7fb381d6f640 1 -- 172.21.15.107:0/2488361411 shutdown_connections 2023-11-24T08:07:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.399+0000 7fb381d6f640 1 -- 172.21.15.107:0/2488361411 wait complete. 2023-11-24T08:07:22.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.400+0000 7fb381d6f640 1 Processor -- start 2023-11-24T08:07:22.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.400+0000 7fb381d6f640 1 -- start start 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c0cbf20 0x7fb37c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c1560f0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c156df0 con 0x7fb37c0cbf20 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.401+0000 7fb381d6f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c0c97d0 con 0x7fb37c15b440 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46426/0 (socket says 172.21.15.107:46426) 2023-11-24T08:07:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 -- 172.21.15.107:0/3781885735 learned_addr learned my addr 172.21.15.107:0/3781885735 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 -- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 msgr2=0x7fb37c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 -- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c0cbf20 msgr2=0x7fb37c153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c0cbf20 0x7fb37c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 -- 172.21.15.107:0/3781885735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640094d0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.402+0000 7fb37b7fe640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007ef0 tx=0x7fb36c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.403+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c003b90 con 0x7fb37c0c57e0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.403+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c00f040 con 0x7fb37c0c57e0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.403+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c00de80 con 0x7fb37c0c57e0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.403+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c156390 con 0x7fb37c0c57e0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.403+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c153c60 con 0x7fb37c0c57e0 2023-11-24T08:07:22.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.404+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c04e8b0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.405+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fb36c004030 con 0x7fb37c0c57e0 2023-11-24T08:07:22.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.406+0000 7fb373fff640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 0x7fb35c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.406+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb36c089df0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.406+0000 7fb37affd640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 0x7fb35c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.406+0000 7fb37affd640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 0x7fb35c06d600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb364036000 tx=0x7fb364030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.410+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb36c054a40 con 0x7fb37c0c57e0 2023-11-24T08:07:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.600+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb37c0b0a00 con 0x7fb37c0c57e0 2023-11-24T08:07:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.601+0000 7fb373fff640 1 -- 172.21.15.107:0/3781885735 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb36c01a0a0 con 0x7fb37c0c57e0 2023-11-24T08:07:22.603 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.604+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 msgr2=0x7fb35c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.604+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 0x7fb35c06d600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb364036000 tx=0x7fb364030000 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.604+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 msgr2=0x7fb37c1568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.604+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007ef0 tx=0x7fb36c00d4a0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 shutdown_connections 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb35c06b140 0x7fb35c06d600 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c15b440 0x7fb37c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c0cbf20 0x7fb37c153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 --2- 172.21.15.107:0/3781885735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb37c0c57e0 0x7fb37c1568b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 >> 172.21.15.107:0/3781885735 conn(0x7fb37c0b0c50 msgr2=0x7fb37c154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 shutdown_connections 2023-11-24T08:07:22.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.605+0000 7fb381d6f640 1 -- 172.21.15.107:0/3781885735 wait complete. 2023-11-24T08:07:22.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:07:22.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.830+0000 7ff987fff640 1 Processor -- start 2023-11-24T08:07:22.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.830+0000 7ff987fff640 1 -- start start 2023-11-24T08:07:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff987fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff98815d3f0 0x7ff9880d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff987fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 0x7ff9880d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff987fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880d15e0 con 0x7ff98815d7d0 2023-11-24T08:07:22.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff987fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9880d0280 con 0x7ff98815d3f0 2023-11-24T08:07:22.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff987fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880d1300 con 0x7ff9880d0e40 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff986ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff98815d3f0 0x7ff9880d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff9877fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 0x7ff9880d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff9877fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 0x7ff9880d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:55932/0 (socket says 172.21.15.107:55932) 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.832+0000 7ff9877fe640 1 -- 172.21.15.107:0/1697633602 learned_addr learned my addr 172.21.15.107:0/1697633602 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.833+0000 7ff986ffd640 1 -- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 msgr2=0x7ff9880d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.833+0000 7ff986ffd640 1 --2- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 0x7ff9880d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.833+0000 7ff986ffd640 1 -- 172.21.15.107:0/1697633602 >> v1:172.21.15.120:6789/0 conn(0x7ff98815d7d0 legacy=0x7ff9880afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.833+0000 7ff986ffd640 1 -- 172.21.15.107:0/1697633602 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880cfc30 con 0x7ff98815d3f0 2023-11-24T08:07:22.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.834+0000 7ff986ffd640 1 --2- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff98815d3f0 0x7ff9880d1830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff9780097a0 tx=0x7ff97802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b62bce201326bd4d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff9867fc640 1 -- 172.21.15.107:0/1697633602 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97802d040 con 0x7ff98815d3f0 2023-11-24T08:07:22.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff9867fc640 1 -- 172.21.15.107:0/1697633602 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff978031810 con 0x7ff98815d3f0 2023-11-24T08:07:22.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff9867fc640 1 -- 172.21.15.107:0/1697633602 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978031ba0 con 0x7ff98815d3f0 2023-11-24T08:07:22.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 -- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff98815d3f0 msgr2=0x7ff9880d1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 --2- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff98815d3f0 0x7ff9880d1830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff9780097a0 tx=0x7ff97802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 -- 172.21.15.107:0/1697633602 shutdown_connections 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 --2- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9880d0e40 0x7ff9880d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 --2- 172.21.15.107:0/1697633602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff98815d3f0 0x7ff9880d1830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.835+0000 7ff987fff640 1 -- 172.21.15.107:0/1697633602 >> 172.21.15.107:0/1697633602 conn(0x7ff98809c5b0 msgr2=0x7ff9880d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.836+0000 7ff987fff640 1 -- 172.21.15.107:0/1697633602 shutdown_connections 2023-11-24T08:07:22.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.836+0000 7ff987fff640 1 -- 172.21.15.107:0/1697633602 wait complete. 2023-11-24T08:07:22.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.837+0000 7ff987fff640 1 Processor -- start 2023-11-24T08:07:22.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.837+0000 7ff987fff640 1 -- start start 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9880d0e40 0x7ff9880b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff98815d3f0 0x7ff9880b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880b5580 con 0x7ff98815d7d0 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880c7e90 con 0x7ff9880d0e40 2023-11-24T08:07:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.838+0000 7ff987fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880c8130 con 0x7ff98815d3f0 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46438/0 (socket says 172.21.15.107:46438) 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 -- 172.21.15.107:0/2689232747 learned_addr learned my addr 172.21.15.107:0/2689232747 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff986ffd640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9880d0e40 0x7ff9880b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff97ffff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff98815d3f0 0x7ff9880b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 -- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff98815d3f0 msgr2=0x7ff9880b6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff98815d3f0 0x7ff9880b6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 -- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9880d0e40 msgr2=0x7ff9880b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9880d0e40 0x7ff9880b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:22.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 -- 172.21.15.107:0/2689232747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978009400 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.839+0000 7ff9877fe640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff98000efc0 tx=0x7ff98000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.840+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98000cd20 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.840+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98000f040 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.840+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9880c8410 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.840+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980003bd0 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.840+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9880b5800 con 0x7ff98815d7d0 2023-11-24T08:07:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.841+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9880d06f0 con 0x7ff98815d7d0 2023-11-24T08:07:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.842+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7ff980003d30 con 0x7ff98815d7d0 2023-11-24T08:07:22.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.843+0000 7ff97f7fe640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 0x7ff96806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:22.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.844+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff98008a1f0 con 0x7ff98815d7d0 2023-11-24T08:07:22.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.844+0000 7ff986ffd640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 0x7ff96806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:22.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.844+0000 7ff986ffd640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 0x7ff96806d5d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff978036000 tx=0x7ff97803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:22.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:22.848+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9880d06f0 con 0x7ff98815d7d0 2023-11-24T08:07:23.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.042+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9880c8930 con 0x7ff98815d7d0 2023-11-24T08:07:23.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.043+0000 7ff97f7fe640 1 -- 172.21.15.107:0/2689232747 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9880c8930 con 0x7ff98815d7d0 2023-11-24T08:07:23.046 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:07:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.046+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 msgr2=0x7ff96806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.046+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 0x7ff96806d5d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff978036000 tx=0x7ff97803c040 comp rx=0 tx=0).stop 2023-11-24T08:07:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.046+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 msgr2=0x7ff9880c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.046+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff98000efc0 tx=0x7ff98000c490 comp rx=0 tx=0).stop 2023-11-24T08:07:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 shutdown_connections 2023-11-24T08:07:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff96806b110 0x7ff96806d5d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff98815d7d0 0x7ff9880c8c10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff98815d3f0 0x7ff9880b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 --2- 172.21.15.107:0/2689232747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9880d0e40 0x7ff9880b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 >> 172.21.15.107:0/2689232747 conn(0x7ff98809c5b0 msgr2=0x7ff9880b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 shutdown_connections 2023-11-24T08:07:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.047+0000 7ff987fff640 1 -- 172.21.15.107:0/2689232747 wait complete. 2023-11-24T08:07:23.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:07:23.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.283+0000 7fa49911d640 1 Processor -- start 2023-11-24T08:07:23.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.283+0000 7fa49911d640 1 -- start start 2023-11-24T08:07:23.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa49911d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa494140520 0x7fa4940c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa49911d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4940c65f0 0x7fa494161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa49911d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49415de00 con 0x7fa494161940 2023-11-24T08:07:23.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa49911d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940c6a70 con 0x7fa4940c65f0 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa49911d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49415e0e0 con 0x7fa494140520 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa48bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4940c65f0 0x7fa494161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa48bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4940c65f0 0x7fa494161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46452/0 (socket says 172.21.15.107:46452) 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.285+0000 7fa48bfff640 1 -- 172.21.15.107:0/1801571538 learned_addr learned my addr 172.21.15.107:0/1801571538 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.286+0000 7fa492d9d640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa494140520 0x7fa4940c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.286+0000 7fa48bfff640 1 -- 172.21.15.107:0/1801571538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa494140520 msgr2=0x7fa4940c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:23.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.286+0000 7fa48bfff640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa494140520 0x7fa4940c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.286+0000 7fa48bfff640 1 -- 172.21.15.107:0/1801571538 >> v1:172.21.15.107:6789/0 conn(0x7fa494161940 legacy=0x7fa494142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.286+0000 7fa48bfff640 1 -- 172.21.15.107:0/1801571538 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49415e3c0 con 0x7fa4940c65f0 2023-11-24T08:07:23.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.287+0000 7fa48bfff640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4940c65f0 0x7fa494161400 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009870 tx=0x7fa47c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71dbcbe2e2dfd5a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.287+0000 7fa49259c640 1 -- 172.21.15.107:0/1801571538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c030840 con 0x7fa4940c65f0 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.287+0000 7fa49259c640 1 -- 172.21.15.107:0/1801571538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c0309a0 con 0x7fa4940c65f0 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.287+0000 7fa49259c640 1 -- 172.21.15.107:0/1801571538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c030d30 con 0x7fa4940c65f0 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 -- 172.21.15.107:0/1801571538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4940c65f0 msgr2=0x7fa494161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4940c65f0 0x7fa494161400 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009870 tx=0x7fa47c02b980 comp rx=0 tx=0).stop 2023-11-24T08:07:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 -- 172.21.15.107:0/1801571538 shutdown_connections 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4940c65f0 0x7fa494161400 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 --2- 172.21.15.107:0/1801571538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa494140520 0x7fa4940c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 -- 172.21.15.107:0/1801571538 >> 172.21.15.107:0/1801571538 conn(0x7fa4940b2e40 msgr2=0x7fa4940b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 -- 172.21.15.107:0/1801571538 shutdown_connections 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.288+0000 7fa49911d640 1 -- 172.21.15.107:0/1801571538 wait complete. 2023-11-24T08:07:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.290+0000 7fa49911d640 1 Processor -- start 2023-11-24T08:07:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.290+0000 7fa49911d640 1 -- start start 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c65f0 0x7fa49413c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 0x7fa4940d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa494153d20 con 0x7fa494140520 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49413aad0 con 0x7fa494161940 2023-11-24T08:07:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa49911d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49406cef0 con 0x7fa4940c65f0 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa48bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa48bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46468/0 (socket says 172.21.15.107:46468) 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa48bfff640 1 -- 172.21.15.107:0/1347424986 learned_addr learned my addr 172.21.15.107:0/1347424986 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.291+0000 7fa492d9d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c65f0 0x7fa49413c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa49359e640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 0x7fa4940d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 -- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c65f0 msgr2=0x7fa49413c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c65f0 0x7fa49413c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 -- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 msgr2=0x7fa4940d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 0x7fa4940d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 -- 172.21.15.107:0/1347424986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0094d0 con 0x7fa494140520 2023-11-24T08:07:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa49359e640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 0x7fa4940d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48bfff640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c039a90 con 0x7fa494140520 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c039bf0 con 0x7fa494140520 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c043410 con 0x7fa494140520 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4940d0a60 con 0x7fa494140520 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.292+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49406d4c0 con 0x7fa494140520 2023-11-24T08:07:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.294+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49406d680 con 0x7fa494140520 2023-11-24T08:07:23.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.294+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fa47c02f390 con 0x7fa494140520 2023-11-24T08:07:23.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.295+0000 7fa48b7fe640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 0x7fa47406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.295+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa47c0abd40 con 0x7fa494140520 2023-11-24T08:07:23.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.296+0000 7fa492d9d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 0x7fa47406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.296+0000 7fa492d9d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 0x7fa47406d5d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa484006fd0 tx=0x7fa484008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.300+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa49406d680 con 0x7fa494140520 2023-11-24T08:07:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.490+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4940cce40 con 0x7fa494140520 2023-11-24T08:07:23.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.491+0000 7fa48b7fe640 1 -- 172.21.15.107:0/1347424986 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa47c048020 con 0x7fa494140520 2023-11-24T08:07:23.493 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:07:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 msgr2=0x7fa47406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 0x7fa47406d5d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa484006fd0 tx=0x7fa484008040 comp rx=0 tx=0).stop 2023-11-24T08:07:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 msgr2=0x7fa49413a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c004730 comp rx=0 tx=0).stop 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 shutdown_connections 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47406b110 0x7fa47406d5d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494161940 0x7fa4940d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa494140520 0x7fa49413a590 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 --2- 172.21.15.107:0/1347424986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c65f0 0x7fa49413c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 >> 172.21.15.107:0/1347424986 conn(0x7fa4940b2e40 msgr2=0x7fa4940b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 shutdown_connections 2023-11-24T08:07:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.494+0000 7fa49911d640 1 -- 172.21.15.107:0/1347424986 wait complete. 2023-11-24T08:07:23.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.728+0000 7faede116640 1 Processor -- start 2023-11-24T08:07:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.728+0000 7faede116640 1 -- start start 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faed8144b30 0x7faed815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed815c410 0x7faed80b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80b1de0 0x7faed80b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed80b27b0 con 0x7faed80b1de0 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed815bf80 con 0x7faed8144b30 2023-11-24T08:07:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faede116640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed80b1560 con 0x7faed815c410 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faedd915640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80b1de0 0x7faed80b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faedd915640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80b1de0 0x7faed80b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46484/0 (socket says 172.21.15.107:46484) 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faedd915640 1 -- 172.21.15.107:0/2711702732 learned_addr learned my addr 172.21.15.107:0/2711702732 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faedc913640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed815c410 0x7faed80b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.730+0000 7faedd114640 1 --2- 172.21.15.107:0/2711702732 >> v2:172.21.15.120:3300/0 conn(0x7faed8144b30 0x7faed815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.731+0000 7faedd915640 1 -- 172.21.15.107:0/2711702732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed815c410 msgr2=0x7faed80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.731+0000 7faedd915640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed815c410 0x7faed80b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.731+0000 7faedd915640 1 -- 172.21.15.107:0/2711702732 >> v2:172.21.15.120:3300/0 conn(0x7faed8144b30 msgr2=0x7faed815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.731+0000 7faedd915640 1 --2- 172.21.15.107:0/2711702732 >> v2:172.21.15.120:3300/0 conn(0x7faed8144b30 0x7faed815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.731+0000 7faedd915640 1 -- 172.21.15.107:0/2711702732 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80b0fb0 con 0x7faed80b1de0 2023-11-24T08:07:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faedd915640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80b1de0 0x7faed80b2270 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faed806f690 tx=0x7faed002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b42842fcadfd806 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faecffff640 1 -- 172.21.15.107:0/2711702732 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed002ca70 con 0x7faed80b1de0 2023-11-24T08:07:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faecffff640 1 -- 172.21.15.107:0/2711702732 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed002cbd0 con 0x7faed80b1de0 2023-11-24T08:07:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faecffff640 1 -- 172.21.15.107:0/2711702732 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00343f0 con 0x7faed80b1de0 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faede116640 1 -- 172.21.15.107:0/2711702732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80b1de0 msgr2=0x7faed80b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.732+0000 7faede116640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80b1de0 0x7faed80b2270 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faed806f690 tx=0x7faed002bed0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 -- 172.21.15.107:0/2711702732 shutdown_connections 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80b1de0 0x7faed80b2270 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 --2- 172.21.15.107:0/2711702732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed815c410 0x7faed80b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 --2- 172.21.15.107:0/2711702732 >> v2:172.21.15.120:3300/0 conn(0x7faed8144b30 0x7faed815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 -- 172.21.15.107:0/2711702732 >> 172.21.15.107:0/2711702732 conn(0x7faed80cb830 msgr2=0x7faed80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 -- 172.21.15.107:0/2711702732 shutdown_connections 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.733+0000 7faede116640 1 -- 172.21.15.107:0/2711702732 wait complete. 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.734+0000 7faede116640 1 Processor -- start 2023-11-24T08:07:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.735+0000 7faede116640 1 -- start start 2023-11-24T08:07:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 0x7faed81413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed815c410 0x7faed813a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed813b5a0 con 0x7faed8144b30 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed815b070 con 0x7faed815c410 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faede116640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed81391a0 con 0x7faed80b1de0 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46486/0 (socket says 172.21.15.107:46486) 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 -- 172.21.15.107:0/377811108 learned_addr learned my addr 172.21.15.107:0/377811108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedd114640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 0x7faed81413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedd915640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed815c410 0x7faed813a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 -- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 msgr2=0x7faed81413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 0x7faed81413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 -- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed815c410 msgr2=0x7faed813a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed815c410 0x7faed813a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedc913640 1 -- 172.21.15.107:0/377811108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed0009b30 con 0x7faed8144b30 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.736+0000 7faedd114640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 0x7faed81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faedc913640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7faec800d5f0 tx=0x7faec800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80125f0 con 0x7faed8144b30 2023-11-24T08:07:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8012750 con 0x7faed8144b30 2023-11-24T08:07:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8012b00 con 0x7faed8144b30 2023-11-24T08:07:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faede116640 1 -- 172.21.15.107:0/377811108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed8139480 con 0x7faed8144b30 2023-11-24T08:07:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.737+0000 7faede116640 1 -- 172.21.15.107:0/377811108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed8136680 con 0x7faed8144b30 2023-11-24T08:07:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.738+0000 7faede116640 1 -- 172.21.15.107:0/377811108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed80b3610 con 0x7faed8144b30 2023-11-24T08:07:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.739+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7faec8012c60 con 0x7faed8144b30 2023-11-24T08:07:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.740+0000 7faecdffb640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 0x7faebc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.740+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faec8089f70 con 0x7faed8144b30 2023-11-24T08:07:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.740+0000 7faedd114640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 0x7faebc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.741+0000 7faedd114640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 0x7faebc06d5d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faed4006fd0 tx=0x7faed4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:23.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.744+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faed80b3610 con 0x7faed8144b30 2023-11-24T08:07:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.975+0000 7faede116640 1 -- 172.21.15.107:0/377811108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faed80b3610 con 0x7faed8144b30 2023-11-24T08:07:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.976+0000 7faecdffb640 1 -- 172.21.15.107:0/377811108 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4922 (secure 0 0 0) 0x7faed80b3610 con 0x7faed8144b30 2023-11-24T08:07:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:07:23.979 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 15 2023-11-24T08:07:23.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.979+0000 7faede116640 1 -- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 msgr2=0x7faebc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.979+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 0x7faebc06d5d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faed4006fd0 tx=0x7faed4008040 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.979+0000 7faede116640 1 -- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 msgr2=0x7faed815ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.979+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7faec800d5f0 tx=0x7faec800dab0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 -- 172.21.15.107:0/377811108 shutdown_connections 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faebc06b110 0x7faebc06d5d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed815c410 0x7faed813a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed8144b30 0x7faed815ab30 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 --2- 172.21.15.107:0/377811108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed80b1de0 0x7faed81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:23.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 -- 172.21.15.107:0/377811108 >> 172.21.15.107:0/377811108 conn(0x7faed80cb830 msgr2=0x7faed815db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:23.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 -- 172.21.15.107:0/377811108 shutdown_connections 2023-11-24T08:07:23.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:23.980+0000 7faede116640 1 -- 172.21.15.107:0/377811108 wait complete. 2023-11-24T08:07:24.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.202+0000 7fdecbebb640 1 Processor -- start 2023-11-24T08:07:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.203+0000 7fdecbebb640 1 -- start start 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.204+0000 7fdecbebb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec40d31a0 0x7fdec40d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.204+0000 7fdecbebb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40ceb40 con 0x7fdec40ce010 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.204+0000 7fdecbebb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40d1070 con 0x7fdec41609e0 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.204+0000 7fdecbebb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec40d3b50 con 0x7fdec40d31a0 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec40d31a0 0x7fdec40d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec40d31a0 0x7fdec40d3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46498/0 (socket says 172.21.15.107:46498) 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 -- 172.21.15.107:0/2571354905 learned_addr learned my addr 172.21.15.107:0/2571354905 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 -- 172.21.15.107:0/2571354905 >> v1:172.21.15.120:6789/0 conn(0x7fdec41609e0 legacy=0x7fdec406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 -- 172.21.15.107:0/2571354905 >> v1:172.21.15.107:6789/0 conn(0x7fdec40ce010 legacy=0x7fdec40ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.205+0000 7fdec9456640 1 -- 172.21.15.107:0/2571354905 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec40cedc0 con 0x7fdec40d31a0 2023-11-24T08:07:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.206+0000 7fdec9456640 1 --2- 172.21.15.107:0/2571354905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40d3610 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3c9abe20b54bc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.206+0000 7fdec8c55640 1 -- 172.21.15.107:0/2571354905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8030860 con 0x7fdec40d31a0 2023-11-24T08:07:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.206+0000 7fdec8c55640 1 -- 172.21.15.107:0/2571354905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb80309c0 con 0x7fdec40d31a0 2023-11-24T08:07:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.206+0000 7fdec8c55640 1 -- 172.21.15.107:0/2571354905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8030d50 con 0x7fdec40d31a0 2023-11-24T08:07:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 -- 172.21.15.107:0/2571354905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 msgr2=0x7fdec40d3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2571354905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40d3610 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 -- 172.21.15.107:0/2571354905 shutdown_connections 2023-11-24T08:07:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2571354905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40d3610 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 -- 172.21.15.107:0/2571354905 >> 172.21.15.107:0/2571354905 conn(0x7fdec408af60 msgr2=0x7fdec409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 -- 172.21.15.107:0/2571354905 shutdown_connections 2023-11-24T08:07:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.207+0000 7fdecbebb640 1 -- 172.21.15.107:0/2571354905 wait complete. 2023-11-24T08:07:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.209+0000 7fdecbebb640 1 Processor -- start 2023-11-24T08:07:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.209+0000 7fdecbebb640 1 -- start start 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 0x7fdec415f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdec41609e0 0x7fdec40b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40c8350 con 0x7fdec40d31a0 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415ff10 con 0x7fdec40ce010 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdecbebb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40c7ab0 con 0x7fdec41609e0 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46500/0 (socket says 172.21.15.107:46500) 2023-11-24T08:07:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 -- 172.21.15.107:0/2548022145 learned_addr learned my addr 172.21.15.107:0/2548022145 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 -- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdec41609e0 msgr2=0x7fdec40b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdec41609e0 0x7fdec40b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9456640 1 -- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 msgr2=0x7fdec415f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.210+0000 7fdec9c57640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 0x7fdec415f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdec9456640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 0x7fdec415f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdec9456640 1 -- 172.21.15.107:0/2548022145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80094d0 con 0x7fdec40d31a0 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdec9c57640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 0x7fdec415f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdec9456640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8030810 con 0x7fdec40d31a0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb8043400 con 0x7fdec40d31a0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec40c7d90 con 0x7fdec40d31a0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80437b0 con 0x7fdec40d31a0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.211+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec409e350 con 0x7fdec40d31a0 2023-11-24T08:07:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.213+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec409e190 con 0x7fdec40d31a0 2023-11-24T08:07:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.213+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fdeb8043910 con 0x7fdec40d31a0 2023-11-24T08:07:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.214+0000 7fdeb67fc640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 0x7fdea806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.214+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdeb803e070 con 0x7fdec40d31a0 2023-11-24T08:07:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.215+0000 7fdec9c57640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 0x7fdea806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.215+0000 7fdec9c57640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 0x7fdea806d5d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdec409de40 tx=0x7fdec0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.219+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdeb8077db0 con 0x7fdec40d31a0 2023-11-24T08:07:24.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.410+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdec409e540 con 0x7fdec40d31a0 2023-11-24T08:07:24.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.413+0000 7fdeb67fc640 1 -- 172.21.15.107:0/2548022145 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7fdeb8034090 con 0x7fdec40d31a0 2023-11-24T08:07:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:24.416 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:09.751609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 msgr2=0x7fdea806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 0x7fdea806d5d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdec409de40 tx=0x7fdec0009040 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 msgr2=0x7fdec40c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb8004730 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 shutdown_connections 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdea806b110 0x7fdea806d5d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdec41609e0 0x7fdec40b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec40d31a0 0x7fdec40c70a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 --2- 172.21.15.107:0/2548022145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdec40ce010 0x7fdec415f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.416+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 >> 172.21.15.107:0/2548022145 conn(0x7fdec408af60 msgr2=0x7fdec40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.417+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 shutdown_connections 2023-11-24T08:07:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.417+0000 7fdecbebb640 1 -- 172.21.15.107:0/2548022145 wait complete. 2023-11-24T08:07:24.438 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:07:24.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:24.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.653+0000 7fead1f8a640 1 Processor -- start 2023-11-24T08:07:24.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.653+0000 7fead1f8a640 1 -- start start 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead1f8a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feacc143ac0 0x7feacc070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead1f8a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc0715f0 con 0x7feacc1436e0 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead1f8a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc0718d0 con 0x7feacc0cf410 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead1f8a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc071bb0 con 0x7feacc143ac0 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead0f88640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feacc1436e0 0x7feacc0ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52810/0 (socket says 172.21.15.107:52810) 2023-11-24T08:07:24.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead0f88640 1 -- 172.21.15.107:0/3811320573 learned_addr learned my addr 172.21.15.107:0/3811320573 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7fead1789640 1 --2- 172.21.15.107:0/3811320573 >> v2:172.21.15.120:3300/0 conn(0x7feacc143ac0 0x7feacc070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051448379 0 0) 0x7feacc0715f0 con 0x7feacc1436e0 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.655+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feabc005730 con 0x7feacc1436e0 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405906410 0 0) 0x7feabc005730 con 0x7feacc1436e0 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 >> v2:172.21.15.120:3300/0 conn(0x7feacc143ac0 msgr2=0x7feacc070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 --2- 172.21.15.107:0/3811320573 >> v2:172.21.15.120:3300/0 conn(0x7feacc143ac0 0x7feacc070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 >> v1:172.21.15.120:6789/0 conn(0x7feacc0cf410 legacy=0x7feacc0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0cd300 con 0x7feacc1436e0 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7feac0001670 con 0x7feacc1436e0 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 >> v1:172.21.15.107:6789/0 conn(0x7feacc1436e0 legacy=0x7feacc0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feabc005c00 0x7feabc006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc0069d0 0x7feabc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 0x7feabc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc071bb0 con 0x7feabc007390 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc00e2c0 con 0x7feabc0069d0 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc00e480 con 0x7feabc005c00 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.656+0000 7fead1789640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 0x7feabc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead0f88640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feabc005c00 0x7feabc006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7feacbfff640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc0069d0 0x7feabc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead1789640 1 -- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feabc005c00 msgr2=0x7feabc006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead1789640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feabc005c00 0x7feabc006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead1789640 1 -- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc0069d0 msgr2=0x7feabc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead1789640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc0069d0 0x7feabc006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead1789640 1 -- 172.21.15.107:0/3811320573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc005730 con 0x7feabc007390 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7feacbfff640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc0069d0 0x7feabc006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.657+0000 7fead0f88640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feabc005c00 0x7feabc006020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:24.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.658+0000 7fead1789640 1 -- 172.21.15.107:0/3811320573 reap_dead start 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.658+0000 7fead1789640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 0x7feabc00db90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7feac4009f90 tx=0x7feabc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd57bbfbe0a1a4a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.658+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac402cc20 con 0x7feabc007390 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.658+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac400f040 con 0x7feabc007390 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.658+0000 7feacb7fe640 1 -- 172.21.15.107:0/3811320573 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4003990 con 0x7feabc007390 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 -- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 msgr2=0x7feabc00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 0x7feabc00db90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7feac4009f90 tx=0x7feabc007a00 comp rx=0 tx=0).stop 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 -- 172.21.15.107:0/3811320573 shutdown_connections 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 --2- 172.21.15.107:0/3811320573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feabc007390 0x7feabc00db90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 -- 172.21.15.107:0/3811320573 >> 172.21.15.107:0/3811320573 conn(0x7feacc0cba10 msgr2=0x7feacc0cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 -- 172.21.15.107:0/3811320573 shutdown_connections 2023-11-24T08:07:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.659+0000 7fead1f8a640 1 -- 172.21.15.107:0/3811320573 wait complete. 2023-11-24T08:07:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.660+0000 7fead1f8a640 1 Processor -- start 2023-11-24T08:07:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.661+0000 7fead1f8a640 1 -- start start 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.661+0000 7fead1f8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 0x7feacc140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1f8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1f8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 0x7feacc13c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1f8a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc1f2b20 con 0x7feacc1f1c30 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1f8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc1f25f0 con 0x7feabc007390 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1f8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc1363f0 con 0x7feacc13bbb0 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46524/0 (socket says 172.21.15.107:46524) 2023-11-24T08:07:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 -- 172.21.15.107:0/35630461 learned_addr learned my addr 172.21.15.107:0/35630461 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead0f88640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 0x7feacc140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1789640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 0x7feacc13c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 -- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 msgr2=0x7feacc13c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 0x7feacc13c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 -- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 msgr2=0x7feacc140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 0x7feacc140080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7feacbfff640 1 -- 172.21.15.107:0/35630461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc13c570 con 0x7feacc1f1c30 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead0f88640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 0x7feacc140080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.662+0000 7fead1789640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 0x7feacc13c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7feacbfff640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feab800d770 tx=0x7feab800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8005270 con 0x7feacc1f1c30 2023-11-24T08:07:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab80053d0 con 0x7feacc1f1c30 2023-11-24T08:07:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab800b8d0 con 0x7feacc1f1c30 2023-11-24T08:07:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc13c830 con 0x7feacc1f1c30 2023-11-24T08:07:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.663+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc13cdf0 con 0x7feacc1f1c30 2023-11-24T08:07:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.664+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc04e840 con 0x7feacc1f1c30 2023-11-24T08:07:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.665+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7feab800ba30 con 0x7feacc1f1c30 2023-11-24T08:07:24.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.666+0000 7feac97fa640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 0x7feab006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:24.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.667+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feab8089b70 con 0x7feacc1f1c30 2023-11-24T08:07:24.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.667+0000 7fead0f88640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 0x7feab006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:24.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.667+0000 7fead0f88640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 0x7feab006d600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feac00078d0 tx=0x7feac0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:24.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.671+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feab80547a0 con 0x7feacc1f1c30 2023-11-24T08:07:24.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.897+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feacc0ce7c0 con 0x7feacc1f1c30 2023-11-24T08:07:24.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.898+0000 7feac97fa640 1 -- 172.21.15.107:0/35630461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4922 (secure 0 0 0) 0x7feab8057dc0 con 0x7feacc1f1c30 2023-11-24T08:07:24.900 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:24.901 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:07:24.901 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 15 2023-11-24T08:07:24.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 msgr2=0x7feab006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 0x7feab006d600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feac00078d0 tx=0x7feac0009040 comp rx=0 tx=0).stop 2023-11-24T08:07:24.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 msgr2=0x7feacc1f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:24.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feab800d770 tx=0x7feab800dc30 comp rx=0 tx=0).stop 2023-11-24T08:07:24.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 shutdown_connections 2023-11-24T08:07:24.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feab006b140 0x7feab006d600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feacc13bbb0 0x7feacc13c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feacc1f1c30 0x7feacc1f20b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 --2- 172.21.15.107:0/35630461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feabc007390 0x7feacc140080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.901+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 >> 172.21.15.107:0/35630461 conn(0x7feacc0cba10 msgr2=0x7feacc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.902+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 shutdown_connections 2023-11-24T08:07:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:24.902+0000 7fead1f8a640 1 -- 172.21.15.107:0/35630461 wait complete. 2023-11-24T08:07:24.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.133+0000 7f327f1ac640 1 Processor -- start 2023-11-24T08:07:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.133+0000 7f327f1ac640 1 -- start start 2023-11-24T08:07:25.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.134+0000 7f327f1ac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 0x7f32780d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.134+0000 7f327f1ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32780d38e0 0x7f32780cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327f1ac640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780ce990 con 0x7f32780d0070 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327f1ac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780d4110 con 0x7f32780d38e0 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327f1ac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32780d04e0 con 0x7f327815d4e0 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32780d38e0 0x7f32780cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32780d38e0 0x7f32780cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46526/0 (socket says 172.21.15.107:46526) 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 -- 172.21.15.107:0/3921392891 learned_addr learned my addr 172.21.15.107:0/3921392891 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327e1aa640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 0x7f32780d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 -- 172.21.15.107:0/3921392891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 msgr2=0x7f32780d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 0x7f32780d1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 -- 172.21.15.107:0/3921392891 >> v1:172.21.15.120:6789/0 conn(0x7f32780d0070 legacy=0x7f32780d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327d9a9640 1 -- 172.21.15.107:0/3921392891 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780cec70 con 0x7f32780d38e0 2023-11-24T08:07:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.135+0000 7f327e1aa640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 0x7f32780d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.136+0000 7f327d9a9640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f32780cfb30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f327808d1a0 tx=0x7f326802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4620b08f0002588 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.137+0000 7f327d1a8640 1 -- 172.21.15.107:0/3921392891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268031830 con 0x7f32780d38e0 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.137+0000 7f327d1a8640 1 -- 172.21.15.107:0/3921392891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268031990 con 0x7f32780d38e0 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.137+0000 7f327d1a8640 1 -- 172.21.15.107:0/3921392891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268031d20 con 0x7f32780d38e0 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.137+0000 7f327f1ac640 1 -- 172.21.15.107:0/3921392891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 msgr2=0x7f32780cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.137+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f32780cfb30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f327808d1a0 tx=0x7f326802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 -- 172.21.15.107:0/3921392891 shutdown_connections 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f32780cfb30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3921392891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327815d4e0 0x7f32780d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 -- 172.21.15.107:0/3921392891 >> 172.21.15.107:0/3921392891 conn(0x7f327809c510 msgr2=0x7f32780d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 -- 172.21.15.107:0/3921392891 shutdown_connections 2023-11-24T08:07:25.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.138+0000 7f327f1ac640 1 -- 172.21.15.107:0/3921392891 wait complete. 2023-11-24T08:07:25.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.140+0000 7f327f1ac640 1 Processor -- start 2023-11-24T08:07:25.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.140+0000 7f327f1ac640 1 -- start start 2023-11-24T08:07:25.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32780d0070 0x7f3278160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327815d4e0 0x7f327815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327815ebc0 con 0x7f32780d38e0 2023-11-24T08:07:25.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327814fc00 con 0x7f327815d4e0 2023-11-24T08:07:25.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327f1ac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327815f520 con 0x7f32780d0070 2023-11-24T08:07:25.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327d9a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327d9a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46528/0 (socket says 172.21.15.107:46528) 2023-11-24T08:07:25.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.141+0000 7f327d9a9640 1 -- 172.21.15.107:0/3415684086 learned_addr learned my addr 172.21.15.107:0/3415684086 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:25.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327e9ab640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327815d4e0 0x7f327815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 -- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32780d0070 msgr2=0x7f3278160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327e1aa640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32780d0070 0x7f3278160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32780d0070 0x7f3278160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 -- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327815d4e0 msgr2=0x7f327815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327815d4e0 0x7f327815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 -- 172.21.15.107:0/3415684086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32680094d0 con 0x7f32780d38e0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f327d9a9640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f326802b910 tx=0x7f32680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326803aa60 con 0x7f32780d38e0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.142+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326803abc0 con 0x7f32780d38e0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.143+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327815df40 con 0x7f32780d38e0 2023-11-24T08:07:25.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.143+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32680443f0 con 0x7f32780d38e0 2023-11-24T08:07:25.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.143+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327815e500 con 0x7f32780d38e0 2023-11-24T08:07:25.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.144+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327804e840 con 0x7f32780d38e0 2023-11-24T08:07:25.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.145+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f3268044550 con 0x7f32780d38e0 2023-11-24T08:07:25.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.146+0000 7f326effd640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 0x7f325c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.146+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f32680abe20 con 0x7f32780d38e0 2023-11-24T08:07:25.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.146+0000 7f327e1aa640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 0x7f325c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.147+0000 7f327e1aa640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 0x7f325c06d5d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f327400b840 tx=0x7f3274008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.150+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3268076b70 con 0x7f32780d38e0 2023-11-24T08:07:25.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.377+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32780cd600 con 0x7f32780d38e0 2023-11-24T08:07:25.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.378+0000 7f326effd640 1 -- 172.21.15.107:0/3415684086 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f3268035020 con 0x7f32780d38e0 2023-11-24T08:07:25.380 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:25.381 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:06:03.446918+0000","modified":"2023-11-24T08:06:06.504388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6837/4081761849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4081761849},{"type":"v1","addr":"172.21.15.107:6837","nonce":4081761849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T08:07:25.381 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 15 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 msgr2=0x7f325c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 0x7f325c06d5d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f327400b840 tx=0x7f3274008040 comp rx=0 tx=0).stop 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 msgr2=0x7f327814f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f326802b910 tx=0x7f32680046c0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 shutdown_connections 2023-11-24T08:07:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b110 0x7f325c06d5d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327815d4e0 0x7f327815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32780d38e0 0x7f327814f6c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 --2- 172.21.15.107:0/3415684086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32780d0070 0x7f3278160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 >> 172.21.15.107:0/3415684086 conn(0x7f327809c510 msgr2=0x7f32780b1860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 shutdown_connections 2023-11-24T08:07:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.381+0000 7f327f1ac640 1 -- 172.21.15.107:0/3415684086 wait complete. 2023-11-24T08:07:25.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:25.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.609+0000 7f2d212ad640 1 Processor -- start 2023-11-24T08:07:25.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.609+0000 7f2d212ad640 1 -- start start 2023-11-24T08:07:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d212ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d1c06d6c0 0x7f2d1c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d212ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d212ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1c06e060 con 0x7f2d1c13d120 2023-11-24T08:07:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d212ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c13f9e0 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d212ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c13fcc0 con 0x7f2d1c0cdbf0 2023-11-24T08:07:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.610+0000 7f2d13fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d1c06d6c0 0x7f2d1c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d1c06d6c0 0x7f2d1c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46544/0 (socket says 172.21.15.107:46544) 2023-11-24T08:07:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 -- 172.21.15.107:0/1365878540 learned_addr learned my addr 172.21.15.107:0/1365878540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d20aac640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 -- 172.21.15.107:0/1365878540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 msgr2=0x7f2d1c0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 -- 172.21.15.107:0/1365878540 >> v1:172.21.15.120:6789/0 conn(0x7f2d1c13d120 legacy=0x7f2d1c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:25.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.611+0000 7f2d13fff640 1 -- 172.21.15.107:0/1365878540 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c13ffa0 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.612+0000 7f2d13fff640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c06db20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400bb10 tx=0x7f2d0402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2f03e06095cc90 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.612+0000 7f2d1b7fe640 1 -- 172.21.15.107:0/1365878540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0402ede0 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d1b7fe640 1 -- 172.21.15.107:0/1365878540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d04036440 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d1b7fe640 1 -- 172.21.15.107:0/1365878540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d040367d0 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 -- 172.21.15.107:0/1365878540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c06d6c0 msgr2=0x7f2d1c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c06db20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400bb10 tx=0x7f2d0402e030 comp rx=0 tx=0).stop 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 -- 172.21.15.107:0/1365878540 shutdown_connections 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1365878540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c06db20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 -- 172.21.15.107:0/1365878540 >> 172.21.15.107:0/1365878540 conn(0x7f2d1c0cb830 msgr2=0x7f2d1c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:25.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 -- 172.21.15.107:0/1365878540 shutdown_connections 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.613+0000 7f2d212ad640 1 -- 172.21.15.107:0/1365878540 wait complete. 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.615+0000 7f2d212ad640 1 Processor -- start 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.615+0000 7f2d212ad640 1 -- start start 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c13c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c15c9b0 con 0x7f2d1c13d120 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c13ccd0 con 0x7f2d1c06d6c0 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d212ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c15bda0 con 0x7f2d1c0cdbf0 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d20aac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.616+0000 7f2d20aac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46556/0 (socket says 172.21.15.107:46556) 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 -- 172.21.15.107:0/1685282760 learned_addr learned my addr 172.21.15.107:0/1685282760 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:25.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d1bfff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c13c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d13fff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 -- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 msgr2=0x7f2d1c0b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 -- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 msgr2=0x7f2d1c13c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c13c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 -- 172.21.15.107:0/1685282760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0400b770 con 0x7f2d1c13d120 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d1bfff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c13c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d13fff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d20aac640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400cda0 tx=0x7f2d1400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.617+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14003b90 con 0x7f2d1c13d120 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.618+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14003cf0 con 0x7f2d1c13d120 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.618+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14004240 con 0x7f2d1c13d120 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.618+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c15a9d0 con 0x7f2d1c13d120 2023-11-24T08:07:25.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.618+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c15c360 con 0x7f2d1c13d120 2023-11-24T08:07:25.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.619+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c15cb60 con 0x7f2d1c13d120 2023-11-24T08:07:25.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.620+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f2d140043a0 con 0x7f2d1c13d120 2023-11-24T08:07:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.621+0000 7f2d197fa640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 0x7f2cfc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.621+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2d14012070 con 0x7f2d1c13d120 2023-11-24T08:07:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.621+0000 7f2d1bfff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 0x7f2cfc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.622+0000 7f2d1bfff640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 0x7f2cfc06d5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00a830 tx=0x7f2d0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.625+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d1c15cb60 con 0x7f2d1c13d120 2023-11-24T08:07:25.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.814+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d1c15cb60 con 0x7f2d1c13d120 2023-11-24T08:07:25.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.817+0000 7f2d197fa640 1 -- 172.21.15.107:0/1685282760 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f2d1c15cb60 con 0x7f2d1c13d120 2023-11-24T08:07:25.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:25.819 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:09.751609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:06:01.096943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T08:06:02.442773+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.819+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 msgr2=0x7f2cfc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.819+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 0x7f2cfc06d5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00a830 tx=0x7f2d0c008040 comp rx=0 tx=0).stop 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.819+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 msgr2=0x7f2d1c155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.819+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400cda0 tx=0x7f2d1400a660 comp rx=0 tx=0).stop 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 shutdown_connections 2023-11-24T08:07:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2cfc06b110 0x7f2cfc06d5d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d1c13d120 0x7f2d1c155810 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d1c0cdbf0 0x7f2d1c0b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 --2- 172.21.15.107:0/1685282760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d1c06d6c0 0x7f2d1c13c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 >> 172.21.15.107:0/1685282760 conn(0x7f2d1c0cb830 msgr2=0x7f2d1c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 shutdown_connections 2023-11-24T08:07:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:25.820+0000 7f2d212ad640 1 -- 172.21.15.107:0/1685282760 wait complete. 2023-11-24T08:07:25.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:07:26.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.049+0000 7f62b70da640 1 Processor -- start 2023-11-24T08:07:26.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.049+0000 7f62b70da640 1 -- start start 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62b70da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62b0161060 0x7f62b0161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62b70da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62b015e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62b70da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b006dcf0 con 0x7f62b00b3330 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62b70da640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b006ec10 con 0x7f62b0161060 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62b70da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b0161980 con 0x7f62b006b6c0 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62affff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62b00b3330 0x7f62b015da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52826/0 (socket says 172.21.15.107:52826) 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.051+0000 7f62affff640 1 -- 172.21.15.107:0/1976049946 learned_addr learned my addr 172.21.15.107:0/1976049946 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:26.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62b4e76640 1 --2- 172.21.15.107:0/1976049946 >> v2:172.21.15.120:3300/0 conn(0x7f62b0161060 0x7f62b0161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551312971 0 0) 0x7f62b006dcf0 con 0x7f62b00b3330 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62b015e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a0003520 con 0x7f62b00b3330 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534811533 0 0) 0x7f62a0003520 con 0x7f62b00b3330 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 msgr2=0x7f62b015e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62b015e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 >> v2:172.21.15.120:3300/0 conn(0x7f62b0161060 msgr2=0x7f62b0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 --2- 172.21.15.107:0/1976049946 >> v2:172.21.15.120:3300/0 conn(0x7f62b0161060 0x7f62b0161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0161ec0 con 0x7f62b00b3330 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f629c0017f0 con 0x7f62b00b3330 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62b015e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 >> v1:172.21.15.107:6789/0 conn(0x7f62b00b3330 legacy=0x7f62b015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62a0003fb0 0x7f62a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62a0004d00 0x7f62a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b006ec10 con 0x7f62a0004d00 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b0161980 con 0x7f62a0003fb0 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a000bb40 con 0x7f62b006b6c0 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62b4e76640 1 --2- 172.21.15.107:0/1976049946 >> v2:172.21.15.120:3300/0 conn(0x7f62b0161060 0x7f62b0161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.052+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62a0004d00 0x7f62a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62affff640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62a0003fb0 0x7f62a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62b5677640 1 -- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 msgr2=0x7f62a0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62b006b6c0 0x7f62a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62b5677640 1 -- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62a0003fb0 msgr2=0x7f62a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62a0003fb0 0x7f62a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.053+0000 7f62b5677640 1 -- 172.21.15.107:0/1976049946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0003520 con 0x7f62a0004d00 2023-11-24T08:07:26.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.054+0000 7f62b5677640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62a0004d00 0x7f62a0005180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f62a80073e0 tx=0x7f62a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3670a4bb60de177 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.054+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a802cd70 con 0x7f62a0004d00 2023-11-24T08:07:26.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.054+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a8034440 con 0x7f62a0004d00 2023-11-24T08:07:26.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.054+0000 7f62af7fe640 1 -- 172.21.15.107:0/1976049946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a80347b0 con 0x7f62a0004d00 2023-11-24T08:07:26.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 -- 172.21.15.107:0/1976049946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62a0004d00 msgr2=0x7f62a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 --2- 172.21.15.107:0/1976049946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62a0004d00 0x7f62a0005180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f62a80073e0 tx=0x7f62a00057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b5677640 1 -- 172.21.15.107:0/1976049946 reap_dead start 2023-11-24T08:07:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 -- 172.21.15.107:0/1976049946 shutdown_connections 2023-11-24T08:07:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 -- 172.21.15.107:0/1976049946 >> 172.21.15.107:0/1976049946 conn(0x7f62b00cf900 msgr2=0x7f62b00cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 -- 172.21.15.107:0/1976049946 shutdown_connections 2023-11-24T08:07:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.055+0000 7f62b70da640 1 -- 172.21.15.107:0/1976049946 wait complete. 2023-11-24T08:07:26.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.056+0000 7f62b70da640 1 Processor -- start 2023-11-24T08:07:26.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.057+0000 7f62b70da640 1 -- start start 2023-11-24T08:07:26.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a0004d00 0x7f62b0152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b00add00 0x7f62b00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b00ae6c0 con 0x7f62b014c220 2023-11-24T08:07:26.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b01e6df0 con 0x7f62b00add00 2023-11-24T08:07:26.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b70da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b014cac0 con 0x7f62a0004d00 2023-11-24T08:07:26.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46580/0 (socket says 172.21.15.107:46580) 2023-11-24T08:07:26.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 -- 172.21.15.107:0/1785784611 learned_addr learned my addr 172.21.15.107:0/1785784611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62b5677640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b00add00 0x7f62b00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 -- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a0004d00 msgr2=0x7f62b0152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a0004d00 0x7f62b0152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 -- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b00add00 msgr2=0x7f62b00ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b00add00 0x7f62b00ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.058+0000 7f62affff640 1 -- 172.21.15.107:0/1785784611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b013f860 con 0x7f62b014c220 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62affff640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f629c00e7b0 tx=0x7f629c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c00eec0 con 0x7f62b014c220 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c0072e0 con 0x7f62b014c220 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b0152910 con 0x7f62b014c220 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c0057a0 con 0x7f62b014c220 2023-11-24T08:07:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.059+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b01e7040 con 0x7f62b014c220 2023-11-24T08:07:26.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.060+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b004e8b0 con 0x7f62b014c220 2023-11-24T08:07:26.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.061+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f629c005900 con 0x7f62b014c220 2023-11-24T08:07:26.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.062+0000 7f62ad7fa640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 0x7f629406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.062+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f629c089ad0 con 0x7f62b014c220 2023-11-24T08:07:26.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.062+0000 7f62b4e76640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 0x7f629406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.063+0000 7f62b4e76640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 0x7f629406d5d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62a4006fd0 tx=0x7f62a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.066+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f629c054860 con 0x7f62b014c220 2023-11-24T08:07:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.281+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f62b0073b10 con 0x7f62b014c220 2023-11-24T08:07:26.283 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:07:26.281+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:07:26.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.285+0000 7f62ad7fa640 1 -- 172.21.15.107:0/1785784611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f629c057e80 con 0x7f62b014c220 2023-11-24T08:07:26.287 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:07:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 msgr2=0x7f629406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 0x7f629406d5d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62a4006fd0 tx=0x7f62a4008040 comp rx=0 tx=0).stop 2023-11-24T08:07:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 msgr2=0x7f62b013f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f629c00e7b0 tx=0x7f629c00ec70 comp rx=0 tx=0).stop 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 shutdown_connections 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f629406b110 0x7f629406d5d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b00add00 0x7f62b00ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62b014c220 0x7f62b013f320 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 --2- 172.21.15.107:0/1785784611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a0004d00 0x7f62b0152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.288+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 >> 172.21.15.107:0/1785784611 conn(0x7f62b00cf900 msgr2=0x7f62b00b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.289+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 shutdown_connections 2023-11-24T08:07:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.289+0000 7f62b70da640 1 -- 172.21.15.107:0/1785784611 wait complete. 2023-11-24T08:07:26.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:07:26.365 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:07:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.506+0000 7ff0731fa640 1 Processor -- start 2023-11-24T08:07:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.506+0000 7ff0731fa640 1 -- start start 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0731fa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff06c0b1260 0x7ff06c0718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0731fa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff06c071e10 0x7ff06c142d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0731fa640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06c1450d0 con 0x7ff06c141b90 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0731fa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c1422b0 con 0x7ff06c0b1260 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0731fa640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c0725e0 con 0x7ff06c071e10 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff06c0b1260 0x7ff06c0718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff06c0b1260 0x7ff06c0718d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46614/0 (socket says 172.21.15.107:46614) 2023-11-24T08:07:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 -- 172.21.15.107:0/1190748411 learned_addr learned my addr 172.21.15.107:0/1190748411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 -- 172.21.15.107:0/1190748411 >> v2:172.21.15.120:3300/0 conn(0x7ff06c071e10 msgr2=0x7ff06c142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 --2- 172.21.15.107:0/1190748411 >> v2:172.21.15.120:3300/0 conn(0x7ff06c071e10 0x7ff06c142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 -- 172.21.15.107:0/1190748411 >> v1:172.21.15.120:6789/0 conn(0x7ff06c141b90 legacy=0x7ff06c0b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.508+0000 7ff0719f7640 1 -- 172.21.15.107:0/1190748411 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c142520 con 0x7ff06c0b1260 2023-11-24T08:07:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.509+0000 7ff0719f7640 1 --2- 172.21.15.107:0/1190748411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c0b1260 0x7ff06c0718d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff06c160120 tx=0x7ff05c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75b6e7dc9fb33b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.510+0000 7ff0711f6640 1 -- 172.21.15.107:0/1190748411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0317e0 con 0x7ff06c0b1260 2023-11-24T08:07:26.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.510+0000 7ff0711f6640 1 -- 172.21.15.107:0/1190748411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c031940 con 0x7ff06c0b1260 2023-11-24T08:07:26.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.510+0000 7ff0711f6640 1 -- 172.21.15.107:0/1190748411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c031cd0 con 0x7ff06c0b1260 2023-11-24T08:07:26.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.510+0000 7ff0731fa640 1 -- 172.21.15.107:0/1190748411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c0b1260 msgr2=0x7ff06c0718d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.510+0000 7ff0731fa640 1 --2- 172.21.15.107:0/1190748411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c0b1260 0x7ff06c0718d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff06c160120 tx=0x7ff05c02b880 comp rx=0 tx=0).stop 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 -- 172.21.15.107:0/1190748411 shutdown_connections 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 --2- 172.21.15.107:0/1190748411 >> v2:172.21.15.120:3300/0 conn(0x7ff06c071e10 0x7ff06c142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 --2- 172.21.15.107:0/1190748411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c0b1260 0x7ff06c0718d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 -- 172.21.15.107:0/1190748411 >> 172.21.15.107:0/1190748411 conn(0x7ff06c06a500 msgr2=0x7ff06c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 -- 172.21.15.107:0/1190748411 shutdown_connections 2023-11-24T08:07:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.511+0000 7ff0731fa640 1 -- 172.21.15.107:0/1190748411 wait complete. 2023-11-24T08:07:26.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.512+0000 7ff0731fa640 1 Processor -- start 2023-11-24T08:07:26.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.512+0000 7ff0731fa640 1 -- start start 2023-11-24T08:07:26.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c071e10 0x7ff06c150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff06c0b1260 0x7ff06c1f0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0cdc80 con 0x7ff06c141b90 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c1f0da0 con 0x7ff06c071e10 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.513+0000 7ff0731fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0ccd90 con 0x7ff06c0b1260 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46620/0 (socket says 172.21.15.107:46620) 2023-11-24T08:07:26.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 -- 172.21.15.107:0/2509129222 learned_addr learned my addr 172.21.15.107:0/2509129222 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:26.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0721f8640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c071e10 0x7ff06c150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 -- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff06c0b1260 msgr2=0x7ff06c1f0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff06c0b1260 0x7ff06c1f0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 -- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c071e10 msgr2=0x7ff06c150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:26.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c071e10 0x7ff06c150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 -- 172.21.15.107:0/2509129222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0094d0 con 0x7ff06c141b90 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.514+0000 7ff0729f9640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff06400b970 tx=0x7ff06400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.515+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06400cb40 con 0x7ff06c141b90 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.515+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06400ccc0 con 0x7ff06c141b90 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.515+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06c0cd070 con 0x7ff06c141b90 2023-11-24T08:07:26.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.515+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064014680 con 0x7ff06c141b90 2023-11-24T08:07:26.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.515+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06c0cd630 con 0x7ff06c141b90 2023-11-24T08:07:26.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.516+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06c0cddf0 con 0x7ff06c141b90 2023-11-24T08:07:26.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.517+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7ff06401d3b0 con 0x7ff06c141b90 2023-11-24T08:07:26.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.518+0000 7ff062ffd640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 0x7ff05006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:26.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.518+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff064089b80 con 0x7ff06c141b90 2023-11-24T08:07:26.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.519+0000 7ff0721f8640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 0x7ff05006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:26.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.519+0000 7ff0721f8640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 0x7ff05006d5d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff06800a750 tx=0x7ff068008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:26.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.523+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff064054760 con 0x7ff06c141b90 2023-11-24T08:07:26.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:26.750+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff06c04e840 con 0x7ff06c141b90 2023-11-24T08:07:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.288+0000 7ff062ffd640 1 -- 172.21.15.107:0/2509129222 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff0640190a0 con 0x7ff06c141b90 2023-11-24T08:07:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 msgr2=0x7ff05006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 0x7ff05006d5d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff06800a750 tx=0x7ff068008040 comp rx=0 tx=0).stop 2023-11-24T08:07:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 msgr2=0x7ff06c15c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff06400b970 tx=0x7ff06400be30 comp rx=0 tx=0).stop 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 shutdown_connections 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05006b110 0x7ff05006d5d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff06c141b90 0x7ff06c15c390 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff06c0b1260 0x7ff06c1f0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 --2- 172.21.15.107:0/2509129222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c071e10 0x7ff06c150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 >> 172.21.15.107:0/2509129222 conn(0x7ff06c06a500 msgr2=0x7ff06c144000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 shutdown_connections 2023-11-24T08:07:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.292+0000 7ff0731fa640 1 -- 172.21.15.107:0/2509129222 wait complete. 2023-11-24T08:07:27.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:07:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.576+0000 7fe360961640 1 Processor -- start 2023-11-24T08:07:27.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.576+0000 7fe360961640 1 -- start start 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe360961640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe35c0cfb60 0x7fe35c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe360961640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0ce0b0 con 0x7fe35c15dc80 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe360961640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c161960 con 0x7fe35c157c70 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe360961640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35c15e8d0 con 0x7fe35c0cfb60 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe35ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe35c15dc80 0x7fe35c0b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52854/0 (socket says 172.21.15.107:52854) 2023-11-24T08:07:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe35ad9d640 1 -- 172.21.15.107:0/3624461387 learned_addr learned my addr 172.21.15.107:0/3624461387 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.578+0000 7fe359d9b640 1 --2- 172.21.15.107:0/3624461387 >> v2:172.21.15.120:3300/0 conn(0x7fe35c0cfb60 0x7fe35c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760426337 0 0) 0x7fe35c0ce0b0 con 0x7fe35c15dc80 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344003520 con 0x7fe35c15dc80 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 258290662 0 0) 0x7fe344003520 con 0x7fe35c15dc80 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 >> v2:172.21.15.120:3300/0 conn(0x7fe35c0cfb60 msgr2=0x7fe35c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 --2- 172.21.15.107:0/3624461387 >> v2:172.21.15.120:3300/0 conn(0x7fe35c0cfb60 0x7fe35c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 >> v1:172.21.15.120:6789/0 conn(0x7fe35c157c70 legacy=0x7fe35c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c15ebb0 con 0x7fe35c15dc80 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe3540019f0 con 0x7fe35c15dc80 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 >> v1:172.21.15.107:6789/0 conn(0x7fe35c15dc80 legacy=0x7fe35c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3440039c0 0x7fe344003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 0x7fe344004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440051d0 0x7fe34400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c15e8d0 con 0x7fe3440047c0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34400c120 con 0x7fe3440039c0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34400c2e0 con 0x7fe3440051d0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.579+0000 7fe359d9b640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 0x7fe344004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe35a59c640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3440039c0 0x7fe344003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe359d9b640 1 -- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440051d0 msgr2=0x7fe34400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe359d9b640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440051d0 0x7fe34400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe359d9b640 1 -- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3440039c0 msgr2=0x7fe344003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe359d9b640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3440039c0 0x7fe344003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe359d9b640 1 -- 172.21.15.107:0/3624461387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe344003520 con 0x7fe3440047c0 2023-11-24T08:07:27.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.580+0000 7fe35ad9d640 1 -- 172.21.15.107:0/3624461387 reap_dead start 2023-11-24T08:07:27.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.581+0000 7fe359d9b640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 0x7fe344004c40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe350009a90 tx=0x7fe3440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f46e52511c1318c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.581+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35002c680 con 0x7fe3440047c0 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.581+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35002c7e0 con 0x7fe3440047c0 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.581+0000 7fe35959a640 1 -- 172.21.15.107:0/3624461387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35002cb70 con 0x7fe3440047c0 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 -- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 msgr2=0x7fe344004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 0x7fe344004c40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe350009a90 tx=0x7fe3440057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 -- 172.21.15.107:0/3624461387 shutdown_connections 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 --2- 172.21.15.107:0/3624461387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3440047c0 0x7fe344004c40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 -- 172.21.15.107:0/3624461387 >> 172.21.15.107:0/3624461387 conn(0x7fe35c06a920 msgr2=0x7fe35c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 -- 172.21.15.107:0/3624461387 shutdown_connections 2023-11-24T08:07:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.582+0000 7fe360961640 1 -- 172.21.15.107:0/3624461387 wait complete. 2023-11-24T08:07:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.584+0000 7fe360961640 1 Processor -- start 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.584+0000 7fe360961640 1 -- start start 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440047c0 0x7fe35c1e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35c0987b0 0x7fe35c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f7470 con 0x7fe35c0c9de0 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f7750 con 0x7fe35c0987b0 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe360961640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f7a30 con 0x7fe3440047c0 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46648/0 (socket says 172.21.15.107:46648) 2023-11-24T08:07:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 -- 172.21.15.107:0/2330386400 learned_addr learned my addr 172.21.15.107:0/2330386400 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 -- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440047c0 msgr2=0x7fe35c1e70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe359d9b640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35c0987b0 0x7fe35c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440047c0 0x7fe35c1e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 -- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35c0987b0 msgr2=0x7fe35c0c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35c0987b0 0x7fe35c0c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.585+0000 7fe35ad9d640 1 -- 172.21.15.107:0/2330386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c1f7d10 con 0x7fe35c0c9de0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe35ad9d640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe35c15e060 tx=0x7fe35400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400ce50 con 0x7fe35c0c9de0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400a610 con 0x7fe35c0c9de0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c1f7ff0 con 0x7fe35c0c9de0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400a9c0 con 0x7fe35c0c9de0 2023-11-24T08:07:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.586+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c098c30 con 0x7fe35c0c9de0 2023-11-24T08:07:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.587+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c098eb0 con 0x7fe35c0c9de0 2023-11-24T08:07:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.588+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fe35400ab20 con 0x7fe35c0c9de0 2023-11-24T08:07:27.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.589+0000 7fe34affd640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 0x7fe33c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:27.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.589+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe354089300 con 0x7fe35c0c9de0 2023-11-24T08:07:27.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.589+0000 7fe35a59c640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 0x7fe33c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.590+0000 7fe35a59c640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 0x7fe33c06d600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe35c064320 tx=0x7fe34c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:27.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.593+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe354053f20 con 0x7fe35c0c9de0 2023-11-24T08:07:27.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:27.782+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe35c1575f0 con 0x7fe35c0c9de0 2023-11-24T08:07:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.298+0000 7fe34affd640 1 -- 172.21.15.107:0/2330386400 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe35401a0d0 con 0x7fe35c0c9de0 2023-11-24T08:07:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:07:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 msgr2=0x7fe33c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 0x7fe33c06d600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe35c064320 tx=0x7fe34c009040 comp rx=0 tx=0).stop 2023-11-24T08:07:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 msgr2=0x7fe35c0ad4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe35c15e060 tx=0x7fe35400cc00 comp rx=0 tx=0).stop 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 shutdown_connections 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe33c06b140 0x7fe33c06d600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0c9de0 0x7fe35c0ad4b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe35c0987b0 0x7fe35c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 --2- 172.21.15.107:0/2330386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3440047c0 0x7fe35c1e70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 >> 172.21.15.107:0/2330386400 conn(0x7fe35c06a920 msgr2=0x7fe35c0b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 shutdown_connections 2023-11-24T08:07:28.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.302+0000 7fe360961640 1 -- 172.21.15.107:0/2330386400 wait complete. 2023-11-24T08:07:28.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:07:28.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.532+0000 7f321275a640 1 Processor -- start 2023-11-24T08:07:28.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.533+0000 7f321275a640 1 -- start start 2023-11-24T08:07:28.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.534+0000 7f321275a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f320c157c70 0x7f320c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.534+0000 7f321275a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c0cf450 0x7f320c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.534+0000 7f321275a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c0ce0b0 con 0x7f320c15dc80 2023-11-24T08:07:28.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.534+0000 7f321275a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c0cfe10 con 0x7f320c157c70 2023-11-24T08:07:28.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.534+0000 7f321275a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c161960 con 0x7f320c0cf450 2023-11-24T08:07:28.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.535+0000 7f3210cf7640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f320c15dc80 0x7f320c15e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:39434/0 (socket says 172.21.15.107:39434) 2023-11-24T08:07:28.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.535+0000 7f3210cf7640 1 -- 172.21.15.107:0/809440202 learned_addr learned my addr 172.21.15.107:0/809440202 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:28.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.535+0000 7f320bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f320c157c70 0x7f320c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.535+0000 7f3203fff640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c0cf450 0x7f320c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320b7fe640 1 -- 172.21.15.107:0/809440202 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135250563 0 0) 0x7f320c0ce0b0 con 0x7f320c15dc80 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320b7fe640 1 -- 172.21.15.107:0/809440202 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8003520 con 0x7f320c15dc80 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320bfff640 1 -- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c0cf450 msgr2=0x7f320c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320bfff640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c0cf450 0x7f320c0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320bfff640 1 -- 172.21.15.107:0/809440202 >> v1:172.21.15.120:6789/0 conn(0x7f320c15dc80 legacy=0x7f320c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.536+0000 7f320bfff640 1 -- 172.21.15.107:0/809440202 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c15e8d0 con 0x7f320c157c70 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.537+0000 7f320bfff640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c157c70 0x7f320c161df0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097a0 tx=0x7f31fc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=591ba41c66f967bd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.538+0000 7f320b7fe640 1 -- 172.21.15.107:0/809440202 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc031780 con 0x7f320c157c70 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.538+0000 7f320b7fe640 1 -- 172.21.15.107:0/809440202 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc0318e0 con 0x7f320c157c70 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.538+0000 7f320b7fe640 1 -- 172.21.15.107:0/809440202 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc031c70 con 0x7f320c157c70 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.538+0000 7f321275a640 1 -- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c157c70 msgr2=0x7f320c161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.538+0000 7f321275a640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c157c70 0x7f320c161df0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097a0 tx=0x7f31fc02b820 comp rx=0 tx=0).stop 2023-11-24T08:07:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 -- 172.21.15.107:0/809440202 shutdown_connections 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c0cf450 0x7f320c0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 --2- 172.21.15.107:0/809440202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c157c70 0x7f320c161df0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 -- 172.21.15.107:0/809440202 >> 172.21.15.107:0/809440202 conn(0x7f320c06a920 msgr2=0x7f320c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 -- 172.21.15.107:0/809440202 shutdown_connections 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.539+0000 7f321275a640 1 -- 172.21.15.107:0/809440202 wait complete. 2023-11-24T08:07:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.541+0000 7f321275a640 1 Processor -- start 2023-11-24T08:07:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.541+0000 7f321275a640 1 -- start start 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 0x7f320c154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c15dc80 0x7f320c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c9430 con 0x7f320c157c70 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c88b0 con 0x7f320c0cf450 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.542+0000 7f321275a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c8b90 con 0x7f320c15dc80 2023-11-24T08:07:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58186/0 (socket says 172.21.15.107:58186) 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 -- 172.21.15.107:0/1732919653 learned_addr learned my addr 172.21.15.107:0/1732919653 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f320bfff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 0x7f320c154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 -- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c15dc80 msgr2=0x7f320c0c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3210cf7640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c15dc80 0x7f320c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c15dc80 0x7f320c0c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 -- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 msgr2=0x7f320c154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 0x7f320c154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f3203fff640 1 -- 172.21.15.107:0/1732919653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc009400 con 0x7f320c157c70 2023-11-24T08:07:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.543+0000 7f320bfff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 0x7f320c154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:28.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f3203fff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e870 tx=0x7f31f400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:28.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f40094f0 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f400c940 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400ccf0 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c0c8e70 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.544+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c09c410 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.545+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c067470 con 0x7f320c157c70 2023-11-24T08:07:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.546+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f31f4013470 con 0x7f320c157c70 2023-11-24T08:07:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.547+0000 7f32097fa640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 0x7f31ec06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.547+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f31f4089a70 con 0x7f320c157c70 2023-11-24T08:07:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.547+0000 7f320bfff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 0x7f31ec06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.548+0000 7f320bfff640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 0x7f31ec06d5d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31fc036000 tx=0x7f31fc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:28.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.552+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31f40548e0 con 0x7f320c157c70 2023-11-24T08:07:28.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:28.732+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f320c0cd910 con 0x7f320c157c70 2023-11-24T08:07:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.311+0000 7f32097fa640 1 -- 172.21.15.107:0/1732919653 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31f40170d0 con 0x7f320c157c70 2023-11-24T08:07:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:07:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 msgr2=0x7f31ec06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 0x7f31ec06d5d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31fc036000 tx=0x7f31fc03d040 comp rx=0 tx=0).stop 2023-11-24T08:07:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 msgr2=0x7f320c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e870 tx=0x7f31f400ed30 comp rx=0 tx=0).stop 2023-11-24T08:07:29.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 shutdown_connections 2023-11-24T08:07:29.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31ec06b110 0x7f31ec06d5d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f320c15dc80 0x7f320c0c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320c157c70 0x7f320c09bed0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 --2- 172.21.15.107:0/1732919653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320c0cf450 0x7f320c154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.315+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 >> 172.21.15.107:0/1732919653 conn(0x7f320c06a920 msgr2=0x7f320c0b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:29.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.316+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 shutdown_connections 2023-11-24T08:07:29.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.316+0000 7f321275a640 1 -- 172.21.15.107:0/1732919653 wait complete. 2023-11-24T08:07:29.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:07:29.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.530+0000 7f508b2c2640 1 Processor -- start 2023-11-24T08:07:29.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.531+0000 7f508b2c2640 1 -- start start 2023-11-24T08:07:29.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.532+0000 7f508b2c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f508415b440 0x7f50840cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.532+0000 7f508b2c2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 0x7f50840c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.532+0000 7f508b2c2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508415eea0 con 0x7f50840c57e0 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.532+0000 7f508b2c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50840cc3d0 con 0x7f508415b440 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.532+0000 7f508b2c2640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f508415b820 con 0x7f50840cbf20 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f508415b440 0x7f50840cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f508415b440 0x7f50840cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58198/0 (socket says 172.21.15.107:58198) 2023-11-24T08:07:29.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 -- 172.21.15.107:0/1122931457 learned_addr learned my addr 172.21.15.107:0/1122931457 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508885d640 1 --2- 172.21.15.107:0/1122931457 >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 0x7f50840c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 -- 172.21.15.107:0/1122931457 >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 msgr2=0x7f50840c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 --2- 172.21.15.107:0/1122931457 >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 0x7f50840c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 -- 172.21.15.107:0/1122931457 >> v1:172.21.15.107:6789/0 conn(0x7f50840c57e0 legacy=0x7f50840c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508905e640 1 -- 172.21.15.107:0/1122931457 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508406d600 con 0x7f508415b440 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f508885d640 1 --2- 172.21.15.107:0/1122931457 >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 0x7f50840c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.533+0000 7f507bfff640 1 -- 172.21.15.107:0/1122931457 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217429012 0 0) 0x7f508415eea0 con 0x7f50840c57e0 2023-11-24T08:07:29.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f508905e640 1 --2- 172.21.15.107:0/1122931457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f508415b440 0x7f50840cb9e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f508406e890 tx=0x7f508002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ff9023d6a48e46 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f507bfff640 1 -- 172.21.15.107:0/1122931457 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080031870 con 0x7f508415b440 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f507bfff640 1 -- 172.21.15.107:0/1122931457 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50800319d0 con 0x7f508415b440 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f507bfff640 1 -- 172.21.15.107:0/1122931457 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080031d60 con 0x7f508415b440 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f508b2c2640 1 -- 172.21.15.107:0/1122931457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f508415b440 msgr2=0x7f50840cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.535+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1122931457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f508415b440 0x7f50840cb9e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f508406e890 tx=0x7f508002b910 comp rx=0 tx=0).stop 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 -- 172.21.15.107:0/1122931457 shutdown_connections 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1122931457 >> v2:172.21.15.120:3300/0 conn(0x7f50840cbf20 0x7f50840c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1122931457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f508415b440 0x7f50840cb9e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 -- 172.21.15.107:0/1122931457 >> 172.21.15.107:0/1122931457 conn(0x7f50840b0c50 msgr2=0x7f50840b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 -- 172.21.15.107:0/1122931457 shutdown_connections 2023-11-24T08:07:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.536+0000 7f508b2c2640 1 -- 172.21.15.107:0/1122931457 wait complete. 2023-11-24T08:07:29.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.537+0000 7f508b2c2640 1 Processor -- start 2023-11-24T08:07:29.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.537+0000 7f508b2c2640 1 -- start start 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50840cbf20 0x7f50841f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f508415b440 0x7f5084156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50841f7320 con 0x7f50840c57e0 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840c60d0 con 0x7f50840cbf20 2023-11-24T08:07:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.538+0000 7f508b2c2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50840b34d0 con 0x7f508415b440 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58210/0 (socket says 172.21.15.107:58210) 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 -- 172.21.15.107:0/1173715592 learned_addr learned my addr 172.21.15.107:0/1173715592 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 -- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f508415b440 msgr2=0x7f5084156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f508415b440 0x7f5084156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508885d640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50840cbf20 0x7f50841f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 -- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50840cbf20 msgr2=0x7f50841f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50840cbf20 0x7f50841f6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 -- 172.21.15.107:0/1173715592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800094d0 con 0x7f50840c57e0 2023-11-24T08:07:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.539+0000 7f508905e640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f508002b950 tx=0x7f50800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:29.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.540+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080031860 con 0x7f50840c57e0 2023-11-24T08:07:29.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.540+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50800443f0 con 0x7f50840c57e0 2023-11-24T08:07:29.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.540+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50840b37b0 con 0x7f50840c57e0 2023-11-24T08:07:29.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.540+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800447a0 con 0x7f50840c57e0 2023-11-24T08:07:29.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.540+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50840c6320 con 0x7f50840c57e0 2023-11-24T08:07:29.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.541+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508404e8b0 con 0x7f50840c57e0 2023-11-24T08:07:29.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.542+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f508004d450 con 0x7f50840c57e0 2023-11-24T08:07:29.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.543+0000 7f5079ffb640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 0x7f506806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.543+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f508002d040 con 0x7f50840c57e0 2023-11-24T08:07:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.543+0000 7f508885d640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 0x7f506806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.544+0000 7f508885d640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 0x7f506806d5d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f508406d5a0 tx=0x7f5074008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:29.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.547+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5080076f30 con 0x7f50840c57e0 2023-11-24T08:07:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.757+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f50840c6510 con 0x7f50840c57e0 2023-11-24T08:07:29.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.769+0000 7f5079ffb640 1 -- 172.21.15.107:0/1173715592 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v12) v1 ==== 104+0+0 (secure 0 0 0) 0x7f50800350a0 con 0x7f50840c57e0 2023-11-24T08:07:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 msgr2=0x7f506806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 0x7f506806d5d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f508406d5a0 tx=0x7f5074008040 comp rx=0 tx=0).stop 2023-11-24T08:07:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 msgr2=0x7f50840b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f508002b950 tx=0x7f50800046c0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 shutdown_connections 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f506806b110 0x7f506806d5d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f508415b440 0x7f5084156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50840cbf20 0x7f50841f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 --2- 172.21.15.107:0/1173715592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50840c57e0 0x7f50840b4080 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 >> 172.21.15.107:0/1173715592 conn(0x7f50840b0c50 msgr2=0x7f50840c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 shutdown_connections 2023-11-24T08:07:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.772+0000 7f508b2c2640 1 -- 172.21.15.107:0/1173715592 wait complete. 2023-11-24T08:07:29.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:07:29.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.996+0000 7f102759e640 1 Processor -- start 2023-11-24T08:07:29.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.997+0000 7f102759e640 1 -- start start 2023-11-24T08:07:30.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.998+0000 7f102759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10280b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.998+0000 7f102759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1028072960 con 0x7f10280b1ec0 2023-11-24T08:07:30.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.998+0000 7f102759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10280b2a60 con 0x7f102815e620 2023-11-24T08:07:30.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.998+0000 7f102759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102815dc10 con 0x7f10280b0b50 2023-11-24T08:07:30.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f1026d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10280b1ec0 0x7f10280b2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52024/0 (socket says 172.21.15.107:52024) 2023-11-24T08:07:30.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f1026d9d640 1 -- 172.21.15.107:0/1343550488 learned_addr learned my addr 172.21.15.107:0/1343550488 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:30.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f102659c640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10280b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133340402 0 0) 0x7f1028072960 con 0x7f10280b1ec0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1014003520 con 0x7f10280b1ec0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:29.999+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914909834 0 0) 0x7f1014003520 con 0x7f10280b1ec0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 msgr2=0x7f10280b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10280b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 >> v1:172.21.15.120:6789/0 conn(0x7f102815e620 legacy=0x7f10280ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280b1470 con 0x7f10280b1ec0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1020001670 con 0x7f10280b1ec0 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 >> v1:172.21.15.107:6789/0 conn(0x7f10280b1ec0 legacy=0x7f10280b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10140039a0 0x7f1014003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1014004d00 0x7f1014005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102815dc10 con 0x7f10140039a0 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101400ba60 con 0x7f1014004d00 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101400bc20 con 0x7f10280b0b50 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10140039a0 0x7f1014003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f1026d9d640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1014004d00 0x7f1014005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f1025d9b640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 -- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 msgr2=0x7f10140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10280b0b50 0x7f10140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 -- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1014004d00 msgr2=0x7f1014005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1014004d00 0x7f1014005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.000+0000 7f102659c640 1 -- 172.21.15.107:0/1343550488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014003520 con 0x7f10140039a0 2023-11-24T08:07:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102659c640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10140039a0 0x7f1014003dc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1010009a90 tx=0x7f10140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f5f4dd0b699be0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101002c680 con 0x7f10140039a0 2023-11-24T08:07:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101002c7e0 con 0x7f10140039a0 2023-11-24T08:07:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102559a640 1 -- 172.21.15.107:0/1343550488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101002cb70 con 0x7f10140039a0 2023-11-24T08:07:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102759e640 1 -- 172.21.15.107:0/1343550488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10140039a0 msgr2=0x7f1014003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f102759e640 1 --2- 172.21.15.107:0/1343550488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10140039a0 0x7f1014003dc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1010009a90 tx=0x7f10140057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.002+0000 7f1026d9d640 1 -- 172.21.15.107:0/1343550488 reap_dead start 2023-11-24T08:07:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.003+0000 7f102759e640 1 -- 172.21.15.107:0/1343550488 shutdown_connections 2023-11-24T08:07:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.003+0000 7f102759e640 1 -- 172.21.15.107:0/1343550488 >> 172.21.15.107:0/1343550488 conn(0x7f102806a500 msgr2=0x7f102806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.003+0000 7f102759e640 1 -- 172.21.15.107:0/1343550488 shutdown_connections 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.003+0000 7f102759e640 1 -- 172.21.15.107:0/1343550488 wait complete. 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.004+0000 7f102759e640 1 Processor -- start 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.004+0000 7f102759e640 1 -- start start 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 0x7f1028155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028150c70 0x7f1028143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028155620 con 0x7f1028152e20 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028154a10 con 0x7f10140039a0 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.005+0000 7f102759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1028154cf0 con 0x7f1028150c70 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58226/0 (socket says 172.21.15.107:58226) 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 -- 172.21.15.107:0/1176101507 learned_addr learned my addr 172.21.15.107:0/1176101507 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f102659c640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 0x7f1028155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 -- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028150c70 msgr2=0x7f1028143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1025d9b640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028150c70 0x7f1028143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028150c70 0x7f1028143a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 -- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 msgr2=0x7f1028155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 0x7f1028155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 -- 172.21.15.107:0/1176101507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028154fd0 con 0x7f1028152e20 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f102659c640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 0x7f1028155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.006+0000 7f1026d9d640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f10280cff30 tx=0x7f102000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.007+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000eed0 con 0x7f1028152e20 2023-11-24T08:07:30.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.007+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102000c630 con 0x7f1028152e20 2023-11-24T08:07:30.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.007+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000c9c0 con 0x7f1028152e20 2023-11-24T08:07:30.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.007+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10281552b0 con 0x7f1028152e20 2023-11-24T08:07:30.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.007+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10281532a0 con 0x7f1028152e20 2023-11-24T08:07:30.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.008+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1028067990 con 0x7f1028152e20 2023-11-24T08:07:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.009+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f102000cb20 con 0x7f1028152e20 2023-11-24T08:07:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.010+0000 7f101effd640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 0x7f100806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.010+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f10200893e0 con 0x7f1028152e20 2023-11-24T08:07:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.010+0000 7f102659c640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 0x7f100806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.011+0000 7f102659c640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 0x7f100806d5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1010036000 tx=0x7f1010030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.014+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1020054400 con 0x7f1028152e20 2023-11-24T08:07:30.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.204+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f1028153490 con 0x7f1028152e20 2023-11-24T08:07:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.209+0000 7f101effd640 1 -- 172.21.15.107:0/1176101507 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1020057a20 con 0x7f1028152e20 2023-11-24T08:07:30.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 msgr2=0x7f100806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 0x7f100806d5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1010036000 tx=0x7f1010030000 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 msgr2=0x7f10280af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f10280cff30 tx=0x7f102000eb90 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 shutdown_connections 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f100806b110 0x7f100806d5d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1028152e20 0x7f10280af140 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028150c70 0x7f1028143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 --2- 172.21.15.107:0/1176101507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10140039a0 0x7f1028155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 >> 172.21.15.107:0/1176101507 conn(0x7f102806a500 msgr2=0x7f10280ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.212+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 shutdown_connections 2023-11-24T08:07:30.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.213+0000 7f102759e640 1 -- 172.21.15.107:0/1176101507 wait complete. 2023-11-24T08:07:30.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed' 2023-11-24T08:07:30.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.431+0000 7f430ad4c640 1 Processor -- start 2023-11-24T08:07:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.432+0000 7f430ad4c640 1 -- start start 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f430ad4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 0x7f43040cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f430ad4c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43040cece0 con 0x7f43040cf450 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f430ad4c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430406d040 con 0x7f430415c7f0 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f430ad4c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43040cfe40 con 0x7f43040ce6c0 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f4303fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f43040cf450 0x7f43040cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52034/0 (socket says 172.21.15.107:52034) 2023-11-24T08:07:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.433+0000 7f4303fff640 1 -- 172.21.15.107:0/3571533467 learned_addr learned my addr 172.21.15.107:0/3571533467 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:30.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43092e9640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 0x7f43040cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840900712 0 0) 0x7f43040cece0 con 0x7f43040cf450 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f0003520 con 0x7f43040cf450 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 400439185 0 0) 0x7f42f0003520 con 0x7f43040cf450 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 msgr2=0x7f43040cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 0x7f43040cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 >> v1:172.21.15.120:6789/0 conn(0x7f430415c7f0 legacy=0x7f430406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430406d320 con 0x7f43040cf450 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f42ec001670 con 0x7f43040cf450 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 >> v1:172.21.15.107:6789/0 conn(0x7f43040cf450 legacy=0x7f43040cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f00039a0 0x7f42f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 0x7f42f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f0004d00 0x7f42f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040cfe40 con 0x7f42f0004d00 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f000bb10 con 0x7f42f00039a0 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.434+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f000bca0 con 0x7f43040ce6c0 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f0004d00 0x7f42f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 -- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 msgr2=0x7f42f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43040ce6c0 0x7f42f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 -- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f00039a0 msgr2=0x7f42f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f4308ae8640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f00039a0 0x7f42f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f00039a0 0x7f42f0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f43092e9640 1 -- 172.21.15.107:0/3571533467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0003520 con 0x7f42f0004d00 2023-11-24T08:07:30.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.435+0000 7f4308ae8640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f00039a0 0x7f42f0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:30.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.436+0000 7f43092e9640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f0004d00 0x7f42f0005180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f42fc007390 tx=0x7f42f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce42dcf7d66bb5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.436+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc02ccf0 con 0x7f42f0004d00 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.436+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc02ce50 con 0x7f42f0004d00 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.436+0000 7f43037fe640 1 -- 172.21.15.107:0/3571533467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc034660 con 0x7f42f0004d00 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 -- 172.21.15.107:0/3571533467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f0004d00 msgr2=0x7f42f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3571533467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f0004d00 0x7f42f0005180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f42fc007390 tx=0x7f42f00057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f43092e9640 1 -- 172.21.15.107:0/3571533467 reap_dead start 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 -- 172.21.15.107:0/3571533467 shutdown_connections 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 -- 172.21.15.107:0/3571533467 >> 172.21.15.107:0/3571533467 conn(0x7f430406a920 msgr2=0x7f430406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 -- 172.21.15.107:0/3571533467 shutdown_connections 2023-11-24T08:07:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.437+0000 7f430ad4c640 1 -- 172.21.15.107:0/3571533467 wait complete. 2023-11-24T08:07:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.439+0000 7f430ad4c640 1 Processor -- start 2023-11-24T08:07:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.439+0000 7f430ad4c640 1 -- start start 2023-11-24T08:07:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f0004d00 0x7f43041f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304154d50 0x7f430415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041f7d40 con 0x7f43041548d0 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041e5560 con 0x7f42f0004d00 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f430ad4c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43041e5280 con 0x7f4304154d50 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58250/0 (socket says 172.21.15.107:58250) 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 -- 172.21.15.107:0/3284033227 learned_addr learned my addr 172.21.15.107:0/3284033227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 -- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304154d50 msgr2=0x7f430415b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304154d50 0x7f430415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 -- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f0004d00 msgr2=0x7f43041f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f0004d00 0x7f43041f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 -- 172.21.15.107:0/3284033227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43041577d0 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.440+0000 7f4303fff640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c7a0 tx=0x7f42ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.441+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0074e0 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.441+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec007640 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.441+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4304157ab0 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.441+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0049e0 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.441+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4304156020 con 0x7f43041548d0 2023-11-24T08:07:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.442+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4304157fd0 con 0x7f43041548d0 2023-11-24T08:07:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.443+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f42ec004b40 con 0x7f43041548d0 2023-11-24T08:07:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.444+0000 7f43017fa640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 0x7f42e406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.444+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f42ec088d80 con 0x7f43041548d0 2023-11-24T08:07:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.444+0000 7f4308ae8640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 0x7f42e406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.444+0000 7f4308ae8640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 0x7f42e406d5d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42f4001650 tx=0x7f42f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.448+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42ec053d00 con 0x7f43041548d0 2023-11-24T08:07:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:30.686+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]} v 0) v1 -- 0x7f430406ef30 con 0x7f43041548d0 2023-11-24T08:07:31.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.324+0000 7f43017fa640 1 -- 172.21.15.107:0/3284033227 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]}]=0 v154) v1 ==== 165+0+0 (secure 0 0 0) 0x7f42ec018070 con 0x7f43041548d0 2023-11-24T08:07:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 msgr2=0x7f42e406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 0x7f42e406d5d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42f4001650 tx=0x7f42f4009040 comp rx=0 tx=0).stop 2023-11-24T08:07:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 msgr2=0x7f430415acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c7a0 tx=0x7f42ec00cc60 comp rx=0 tx=0).stop 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 shutdown_connections 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42e406b110 0x7f42e406d5d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304154d50 0x7f430415b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43041548d0 0x7f430415acc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 --2- 172.21.15.107:0/3284033227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42f0004d00 0x7f43041f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 >> 172.21.15.107:0/3284033227 conn(0x7f430406a920 msgr2=0x7f43040af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 shutdown_connections 2023-11-24T08:07:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.329+0000 7f430ad4c640 1 -- 172.21.15.107:0/3284033227 wait complete. 2023-11-24T08:07:31.353 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is in failed state and validate the reason ... ok 2023-11-24T08:07:31.354 INFO:tasks.cephfs_test_runner:Starting test: test_clone_failure_status_in_progress_cancelled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ensure failure status is shown when clone is cancelled during in-progress state and validate the reason 2023-11-24T08:07:31.354 INFO:tasks.cephfs_test_runner:test_clone_failure_status_in_progress_cancelled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:07:31.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:31.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.565+0000 7ff3a289b640 1 Processor -- start 2023-11-24T08:07:31.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.565+0000 7ff3a289b640 1 -- start start 2023-11-24T08:07:31.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff3a289b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff39c15c7f0 0x7ff39c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff3a289b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff39c0cf450 0x7ff39c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff3a289b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c0cece0 con 0x7ff39c0cdfb0 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff3a289b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39c0cfe40 con 0x7ff39c0cf450 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff3a289b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39c0ceaf0 con 0x7ff39c15c7f0 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff39c0cf450 0x7ff39c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff39c0cf450 0x7ff39c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58264/0 (socket says 172.21.15.107:58264) 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 -- 172.21.15.107:0/1658155914 learned_addr learned my addr 172.21.15.107:0/1658155914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff39bfff640 1 --2- 172.21.15.107:0/1658155914 >> v2:172.21.15.120:3300/0 conn(0x7ff39c15c7f0 0x7ff39c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 -- 172.21.15.107:0/1658155914 >> v2:172.21.15.120:3300/0 conn(0x7ff39c15c7f0 msgr2=0x7ff39c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 --2- 172.21.15.107:0/1658155914 >> v2:172.21.15.120:3300/0 conn(0x7ff39c15c7f0 0x7ff39c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 -- 172.21.15.107:0/1658155914 >> v1:172.21.15.120:6789/0 conn(0x7ff39c0cdfb0 legacy=0x7ff39c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:31.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.567+0000 7ff393fff640 1 -- 172.21.15.107:0/1658155914 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c06d040 con 0x7ff39c0cf450 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff393fff640 1 --2- 172.21.15.107:0/1658155914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c0cf450 0x7ff39c0cf900 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0d0130 tx=0x7ff38402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e3cd0623c5adb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff39b7fe640 1 -- 172.21.15.107:0/1658155914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384031870 con 0x7ff39c0cf450 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff39b7fe640 1 -- 172.21.15.107:0/1658155914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3840319d0 con 0x7ff39c0cf450 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff39b7fe640 1 -- 172.21.15.107:0/1658155914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384031d60 con 0x7ff39c0cf450 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff3a289b640 1 -- 172.21.15.107:0/1658155914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c0cf450 msgr2=0x7ff39c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.569+0000 7ff3a289b640 1 --2- 172.21.15.107:0/1658155914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c0cf450 0x7ff39c0cf900 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0d0130 tx=0x7ff38402b910 comp rx=0 tx=0).stop 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 -- 172.21.15.107:0/1658155914 shutdown_connections 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 --2- 172.21.15.107:0/1658155914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c0cf450 0x7ff39c0cf900 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 --2- 172.21.15.107:0/1658155914 >> v2:172.21.15.120:3300/0 conn(0x7ff39c15c7f0 0x7ff39c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 -- 172.21.15.107:0/1658155914 >> 172.21.15.107:0/1658155914 conn(0x7ff39c06a920 msgr2=0x7ff39c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:31.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 -- 172.21.15.107:0/1658155914 shutdown_connections 2023-11-24T08:07:31.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.570+0000 7ff3a289b640 1 -- 172.21.15.107:0/1658155914 wait complete. 2023-11-24T08:07:31.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.571+0000 7ff3a289b640 1 Processor -- start 2023-11-24T08:07:31.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.571+0000 7ff3a289b640 1 -- start start 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff39c0cdfb0 0x7ff39c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39c0cf450 0x7ff39c152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c09c140 con 0x7ff39c15c7f0 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c09b7e0 con 0x7ff39c0cf450 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.572+0000 7ff3a289b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c09ba80 con 0x7ff39c0cdfb0 2023-11-24T08:07:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58272/0 (socket says 172.21.15.107:58272) 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 -- 172.21.15.107:0/2295864669 learned_addr learned my addr 172.21.15.107:0/2295864669 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 -- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff39c0cdfb0 msgr2=0x7ff39c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff39c0cdfb0 0x7ff39c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff393fff640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39c0cf450 0x7ff39c152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 -- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39c0cf450 msgr2=0x7ff39c152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39c0cf450 0x7ff39c152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 -- 172.21.15.107:0/2295864669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840094d0 con 0x7ff39c15c7f0 2023-11-24T08:07:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.573+0000 7ff3a0e38640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c780 tx=0x7ff39400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.574+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff394003d80 con 0x7ff39c15c7f0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.574+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39400ce30 con 0x7ff39c15c7f0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.574+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff394005290 con 0x7ff39c15c7f0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.574+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c09bd60 con 0x7ff39c15c7f0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.574+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c0b2620 con 0x7ff39c15c7f0 2023-11-24T08:07:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.575+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c09c280 con 0x7ff39c15c7f0 2023-11-24T08:07:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.576+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7ff3940053f0 con 0x7ff39c15c7f0 2023-11-24T08:07:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.577+0000 7ff3997fa640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 0x7ff37c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.577+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff394088c20 con 0x7ff39c15c7f0 2023-11-24T08:07:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.577+0000 7ff39bfff640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 0x7ff37c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.578+0000 7ff39bfff640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 0x7ff37c06d5d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00a810 tx=0x7ff38c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:31.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.581+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff394053bf0 con 0x7ff39c15c7f0 2023-11-24T08:07:31.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.762+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff39c067470 con 0x7ff39c15c7f0 2023-11-24T08:07:31.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.764+0000 7ff3997fa640 1 -- 172.21.15.107:0/2295864669 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12988 (secure 0 0 0) 0x7ff394017020 con 0x7ff39c15c7f0 2023-11-24T08:07:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:31.768 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":39,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:29.300476+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/4081761849":"2023-11-25T08:07:26.282879+0000","172.21.15.107:6836/4081761849":"2023-11-25T08:07:26.282879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:31.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 msgr2=0x7ff37c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 0x7ff37c06d5d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00a810 tx=0x7ff38c008040 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 msgr2=0x7ff39c14a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c780 tx=0x7ff39400cc40 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 shutdown_connections 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff37c06b110 0x7ff37c06d5d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff39c15c7f0 0x7ff39c14a330 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39c0cf450 0x7ff39c152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 --2- 172.21.15.107:0/2295864669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff39c0cdfb0 0x7ff39c0b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 >> 172.21.15.107:0/2295864669 conn(0x7ff39c06a920 msgr2=0x7ff39c15b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 shutdown_connections 2023-11-24T08:07:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.767+0000 7ff3a289b640 1 -- 172.21.15.107:0/2295864669 wait complete. 2023-11-24T08:07:31.786 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:07:31.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:07:31.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.987+0000 7f5b04e72640 1 Processor -- start 2023-11-24T08:07:31.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.987+0000 7f5b04e72640 1 -- start start 2023-11-24T08:07:31.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5b04e72640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b000b0b40 0x7f5b000b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5b04e72640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5b0015ded0 0x7f5b0015e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5b04e72640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0006d430 con 0x7f5b0015a1c0 2023-11-24T08:07:31.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5b04e72640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0015ec50 con 0x7f5b000b0b40 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5b04e72640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0006cb00 con 0x7f5b0015ded0 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5af6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b000b0b40 0x7f5b000b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5af6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b000b0b40 0x7f5b000b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58280/0 (socket says 172.21.15.107:58280) 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.988+0000 7f5af6ffd640 1 -- 172.21.15.107:0/2500897648 learned_addr learned my addr 172.21.15.107:0/2500897648 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.989+0000 7f5af6ffd640 1 -- 172.21.15.107:0/2500897648 >> v2:172.21.15.120:3300/0 conn(0x7f5b0015ded0 msgr2=0x7f5b0015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.989+0000 7f5af6ffd640 1 --2- 172.21.15.107:0/2500897648 >> v2:172.21.15.120:3300/0 conn(0x7f5b0015ded0 0x7f5b0015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.989+0000 7f5af6ffd640 1 -- 172.21.15.107:0/2500897648 >> v1:172.21.15.107:6789/0 conn(0x7f5b0015a1c0 legacy=0x7f5b0006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.989+0000 7f5af6ffd640 1 -- 172.21.15.107:0/2500897648 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000b14e0 con 0x7f5b000b0b40 2023-11-24T08:07:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.990+0000 7f5af6ffd640 1 --2- 172.21.15.107:0/2500897648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b000b0b40 0x7f5b000b0fa0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5b000c9ad0 tx=0x7f5ae802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea3a630d95bede7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.990+0000 7f5afeffd640 1 -- 172.21.15.107:0/2500897648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8030880 con 0x7f5b000b0b40 2023-11-24T08:07:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.990+0000 7f5afeffd640 1 -- 172.21.15.107:0/2500897648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae80309e0 con 0x7f5b000b0b40 2023-11-24T08:07:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.990+0000 7f5afeffd640 1 -- 172.21.15.107:0/2500897648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8030d70 con 0x7f5b000b0b40 2023-11-24T08:07:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 -- 172.21.15.107:0/2500897648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b000b0b40 msgr2=0x7f5b000b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 --2- 172.21.15.107:0/2500897648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b000b0b40 0x7f5b000b0fa0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5b000c9ad0 tx=0x7f5ae802b9c0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 -- 172.21.15.107:0/2500897648 shutdown_connections 2023-11-24T08:07:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 --2- 172.21.15.107:0/2500897648 >> v2:172.21.15.120:3300/0 conn(0x7f5b0015ded0 0x7f5b0015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 --2- 172.21.15.107:0/2500897648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b000b0b40 0x7f5b000b0fa0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 -- 172.21.15.107:0/2500897648 >> 172.21.15.107:0/2500897648 conn(0x7f5b000cbb40 msgr2=0x7f5b000cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 -- 172.21.15.107:0/2500897648 shutdown_connections 2023-11-24T08:07:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.991+0000 7f5b04e72640 1 -- 172.21.15.107:0/2500897648 wait complete. 2023-11-24T08:07:31.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.993+0000 7f5b04e72640 1 Processor -- start 2023-11-24T08:07:31.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.993+0000 7f5b04e72640 1 -- start start 2023-11-24T08:07:31.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 0x7f5b00152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b0015ded0 0x7f5b000b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:31.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0015c870 con 0x7f5b0015a1c0 2023-11-24T08:07:31.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0015bc60 con 0x7f5b000b0b40 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5b04e72640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0015bf40 con 0x7f5b0015ded0 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58282/0 (socket says 172.21.15.107:58282) 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 -- 172.21.15.107:0/153372228 learned_addr learned my addr 172.21.15.107:0/153372228 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5aff7fe640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 0x7f5b00152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 -- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b0015ded0 msgr2=0x7f5b000b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b0015ded0 0x7f5b000b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 -- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 msgr2=0x7f5b00152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 0x7f5b00152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5af6ffd640 1 -- 172.21.15.107:0/153372228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80094d0 con 0x7f5b0015a1c0 2023-11-24T08:07:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.995+0000 7f5aff7fe640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 0x7f5b00152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:31.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5af6ffd640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8000c00 tx=0x7f5ae8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:31.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8030850 con 0x7f5b0015a1c0 2023-11-24T08:07:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8043400 con 0x7f5b0015a1c0 2023-11-24T08:07:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80437b0 con 0x7f5b0015a1c0 2023-11-24T08:07:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0015ea20 con 0x7f5b0015a1c0 2023-11-24T08:07:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.996+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0015c500 con 0x7f5b0015a1c0 2023-11-24T08:07:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.998+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0004e840 con 0x7f5b0015a1c0 2023-11-24T08:07:32.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.998+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f5ae8043910 con 0x7f5b0015a1c0 2023-11-24T08:07:32.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.999+0000 7f5afcff9640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 0x7f5ae006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.999+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ae803e070 con 0x7f5b0015a1c0 2023-11-24T08:07:32.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:31.999+0000 7f5aff7fe640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 0x7f5ae006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.000+0000 7f5aff7fe640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 0x7f5ae006d600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000f60 tx=0x7f5af0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.005+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ae8077c90 con 0x7f5b0015a1c0 2023-11-24T08:07:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.196+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b000ca020 con 0x7f5b0015a1c0 2023-11-24T08:07:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.197+0000 7f5afcff9640 1 -- 172.21.15.107:0/153372228 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ae807b2b0 con 0x7f5b0015a1c0 2023-11-24T08:07:32.199 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:07:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 msgr2=0x7f5ae006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 0x7f5ae006d600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000f60 tx=0x7f5af0009040 comp rx=0 tx=0).stop 2023-11-24T08:07:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 msgr2=0x7f5b001f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8000c00 tx=0x7f5ae8004730 comp rx=0 tx=0).stop 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 shutdown_connections 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ae006b140 0x7f5ae006d600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b0015ded0 0x7f5b000b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b0015a1c0 0x7f5b001f3370 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 --2- 172.21.15.107:0/153372228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b000b0b40 0x7f5b00152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 >> 172.21.15.107:0/153372228 conn(0x7f5b000cbb40 msgr2=0x7f5b00072080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 shutdown_connections 2023-11-24T08:07:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.200+0000 7f5b04e72640 1 -- 172.21.15.107:0/153372228 wait complete. 2023-11-24T08:07:32.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled' 2023-11-24T08:07:32.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.430+0000 7f8feaab4640 1 Processor -- start 2023-11-24T08:07:32.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.431+0000 7f8feaab4640 1 -- start start 2023-11-24T08:07:32.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8fe40b2250 0x7f8fe415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2630 0x7f8fe40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 0x7f8fe415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe415a990 con 0x7f8fe40b2250 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe415ab80 con 0x7f8fe40cd470 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8feaab4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe415e680 con 0x7f8fe40b2630 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.432+0000 7f8fe8850640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8fe40b2250 0x7f8fe415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8fe40b2250 0x7f8fe415df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58298/0 (socket says 172.21.15.107:58298) 2023-11-24T08:07:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 -- 172.21.15.107:0/960877014 learned_addr learned my addr 172.21.15.107:0/960877014 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe9051640 1 --2- 172.21.15.107:0/960877014 >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 0x7f8fe415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe3fff640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2630 0x7f8fe40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 -- 172.21.15.107:0/960877014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2630 msgr2=0x7f8fe40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2630 0x7f8fe40ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 -- 172.21.15.107:0/960877014 >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 msgr2=0x7f8fe415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 --2- 172.21.15.107:0/960877014 >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 0x7f8fe415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe8850640 1 -- 172.21.15.107:0/960877014 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe415e960 con 0x7f8fe40b2250 2023-11-24T08:07:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.433+0000 7f8fe9051640 1 --2- 172.21.15.107:0/960877014 >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 0x7f8fe415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.434+0000 7f8fe8850640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40b2250 0x7f8fe415df80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4158560 tx=0x7f8fd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415c03c1fa866412 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8fe37fe640 1 -- 172.21.15.107:0/960877014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8031870 con 0x7f8fe40b2250 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8fe37fe640 1 -- 172.21.15.107:0/960877014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd80319d0 con 0x7f8fe40b2250 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8fe37fe640 1 -- 172.21.15.107:0/960877014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8031d60 con 0x7f8fe40b2250 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 -- 172.21.15.107:0/960877014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40b2250 msgr2=0x7f8fe415df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40b2250 0x7f8fe415df80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4158560 tx=0x7f8fd802b910 comp rx=0 tx=0).stop 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 -- 172.21.15.107:0/960877014 shutdown_connections 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 --2- 172.21.15.107:0/960877014 >> v2:172.21.15.120:3300/0 conn(0x7f8fe40cd470 0x7f8fe415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2630 0x7f8fe40ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.435+0000 7f8feaab4640 1 --2- 172.21.15.107:0/960877014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40b2250 0x7f8fe415df80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.436+0000 7f8feaab4640 1 -- 172.21.15.107:0/960877014 >> 172.21.15.107:0/960877014 conn(0x7f8fe406a920 msgr2=0x7f8fe406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.436+0000 7f8feaab4640 1 -- 172.21.15.107:0/960877014 shutdown_connections 2023-11-24T08:07:32.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.436+0000 7f8feaab4640 1 -- 172.21.15.107:0/960877014 wait complete. 2023-11-24T08:07:32.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.437+0000 7f8feaab4640 1 Processor -- start 2023-11-24T08:07:32.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.438+0000 7f8feaab4640 1 -- start start 2023-11-24T08:07:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.438+0000 7f8feaab4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2250 0x7f8fe41f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.438+0000 7f8feaab4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe40b2630 0x7f8fe4150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.438+0000 7f8feaab4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.438+0000 7f8feaab4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4151e10 con 0x7f8fe40cd470 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8feaab4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe414e590 con 0x7f8fe40b2630 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8feaab4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4151570 con 0x7f8fe40b2250 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58310/0 (socket says 172.21.15.107:58310) 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 -- 172.21.15.107:0/2571006575 learned_addr learned my addr 172.21.15.107:0/2571006575 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe3fff640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe40b2630 0x7f8fe4150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 -- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2250 msgr2=0x7f8fe41f5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe8850640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2250 0x7f8fe41f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2250 0x7f8fe41f5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 -- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe40b2630 msgr2=0x7f8fe4150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:32.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe40b2630 0x7f8fe4150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:32.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.439+0000 7f8fe9051640 1 -- 172.21.15.107:0/2571006575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80094d0 con 0x7f8fe40cd470 2023-11-24T08:07:32.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8fe9051640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00cca0 tx=0x7f8fdc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00f680 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00f800 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe4151850 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00fbb0 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.440+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe40aec80 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.441+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe40aeea0 con 0x7f8fe40cd470 2023-11-24T08:07:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.442+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f8fdc00b040 con 0x7f8fe40cd470 2023-11-24T08:07:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.443+0000 7f8fe17fa640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 0x7f8fc806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.443+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8fdc08a110 con 0x7f8fe40cd470 2023-11-24T08:07:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.443+0000 7f8fe8850640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 0x7f8fc806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.444+0000 7f8fe8850640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 0x7f8fc806d5d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8036000 tx=0x7f8fd803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.447+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8fdc0550b0 con 0x7f8fe40cd470 2023-11-24T08:07:32.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:32.704+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled"]} v 0) v1 -- 0x7f8fe4067470 con 0x7f8fe40cd470 2023-11-24T08:07:33.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.336+0000 7f8fe17fa640 1 -- 172.21.15.107:0/2571006575 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled"]}]=0 v156) v1 ==== 183+0+0 (secure 0 0 0) 0x7f8fdc0586d0 con 0x7f8fe40cd470 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.339+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 msgr2=0x7f8fc806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.339+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 0x7f8fc806d5d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8036000 tx=0x7f8fd803c040 comp rx=0 tx=0).stop 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 msgr2=0x7f8fe4150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00cca0 tx=0x7f8fdc00a650 comp rx=0 tx=0).stop 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 shutdown_connections 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fc806b110 0x7f8fc806d5d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe40cd470 0x7f8fe4150f40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe40b2630 0x7f8fe4150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 --2- 172.21.15.107:0/2571006575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe40b2250 0x7f8fe41f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 >> 172.21.15.107:0/2571006575 conn(0x7f8fe406a920 msgr2=0x7f8fe4071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 shutdown_connections 2023-11-24T08:07:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.340+0000 7f8feaab4640 1 -- 172.21.15.107:0/2571006575 wait complete. 2023-11-24T08:07:33.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:07:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.577+0000 7f4c50fa9640 1 Processor -- start 2023-11-24T08:07:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.578+0000 7f4c50fa9640 1 -- start start 2023-11-24T08:07:33.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c50fa9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c4c160a70 0x7f4c4c0cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c50fa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c0d3b30 0x7f4c4c0d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c50fa9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c0b23f0 con 0x7f4c4c160e50 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c50fa9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0d09c0 con 0x7f4c4c160a70 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c50fa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b25c0 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c4a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c4c160a70 0x7f4c4c0cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c4a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c4c160a70 0x7f4c4c0cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58322/0 (socket says 172.21.15.107:58322) 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.579+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2305625770 learned_addr learned my addr 172.21.15.107:0/2305625770 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.580+0000 7f4c4ad9d640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c0d3b30 0x7f4c4c0d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.580+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2305625770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c0d3b30 msgr2=0x7f4c4c0d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.580+0000 7f4c4a59c640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c0d3b30 0x7f4c4c0d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.580+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2305625770 >> v1:172.21.15.120:6789/0 conn(0x7f4c4c160e50 legacy=0x7f4c4c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.580+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2305625770 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0b2110 con 0x7f4c4c160a70 2023-11-24T08:07:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.581+0000 7f4c4a59c640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c160a70 0x7f4c4c0cfc60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009e50 tx=0x7f4c3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bef2acef60cb854 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.582+0000 7f4c4959a640 1 -- 172.21.15.107:0/2305625770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c02ca70 con 0x7f4c4c160a70 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.582+0000 7f4c4959a640 1 -- 172.21.15.107:0/2305625770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c02cbd0 con 0x7f4c4c160a70 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.582+0000 7f4c4959a640 1 -- 172.21.15.107:0/2305625770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0343f0 con 0x7f4c4c160a70 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.582+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2305625770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c160a70 msgr2=0x7f4c4c0cfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.582+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c160a70 0x7f4c4c0cfc60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009e50 tx=0x7f4c3c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2305625770 shutdown_connections 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c0d3b30 0x7f4c4c0d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2305625770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c160a70 0x7f4c4c0cfc60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2305625770 >> 172.21.15.107:0/2305625770 conn(0x7f4c4c09c6d0 msgr2=0x7f4c4c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2305625770 shutdown_connections 2023-11-24T08:07:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.583+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2305625770 wait complete. 2023-11-24T08:07:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.584+0000 7f4c50fa9640 1 Processor -- start 2023-11-24T08:07:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.584+0000 7f4c50fa9640 1 -- start start 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c4c160a70 0x7f4c4c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c160e50 0x7f4c4c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c15ab30 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c15ad60 con 0x7f4c4c160a70 2023-11-24T08:07:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.585+0000 7f4c50fa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c156800 con 0x7f4c4c160e50 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58328/0 (socket says 172.21.15.107:58328) 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2723846016 learned_addr learned my addr 172.21.15.107:0/2723846016 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c160e50 msgr2=0x7f4c4c15a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c160e50 0x7f4c4c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c4c160a70 msgr2=0x7f4c4c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c4c160a70 0x7f4c4c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 -- 172.21.15.107:0/2723846016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c009b30 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.586+0000 7f4c4a59c640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f80 tx=0x7f4c3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:33.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.587+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c009510 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.587+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c03e450 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.587+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c03e800 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.587+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c156ae0 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.587+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c1570a0 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.588+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c156940 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.589+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f4c3c03e960 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.590+0000 7f4c42ffd640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 0x7f4c2c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.590+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4c3c0ac5b0 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.590+0000 7f4c49d9b640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 0x7f4c2c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.591+0000 7f4c49d9b640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 0x7f4c2c06d5d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0d05a0 tx=0x7f4c340079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:33.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.594+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c4c156940 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.783+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4c4c156940 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.789+0000 7f4c42ffd640 1 -- 172.21.15.107:0/2723846016 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c4c156940 con 0x7f4c4c0d3b30 2023-11-24T08:07:33.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.792+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 msgr2=0x7f4c2c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.792+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 0x7f4c2c06d5d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0d05a0 tx=0x7f4c340079b0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.792+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 msgr2=0x7f4c4c156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:33.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.792+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f80 tx=0x7f4c3c004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 shutdown_connections 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c2c06b110 0x7f4c2c06d5d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c4c160e50 0x7f4c4c15a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c4c160a70 0x7f4c4c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 --2- 172.21.15.107:0/2723846016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c4c0d3b30 0x7f4c4c156090 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 >> 172.21.15.107:0/2723846016 conn(0x7f4c4c09c6d0 msgr2=0x7f4c4c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 shutdown_connections 2023-11-24T08:07:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:33.793+0000 7f4c50fa9640 1 -- 172.21.15.107:0/2723846016 wait complete. 2023-11-24T08:07:33.814 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:07:33.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:07:33.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:07:34.048 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:07:34.048 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:34.049 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:34.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:07:34.109 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:34.109 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:34.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:07:34.167 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:34.167 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:34.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:07:34.225 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:34.226 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:34.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:07:34.286 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:07:34.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:34.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.505+0000 7f24e6dfa640 1 Processor -- start 2023-11-24T08:07:34.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.505+0000 7f24e6dfa640 1 -- start start 2023-11-24T08:07:34.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.506+0000 7f24e6dfa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24e006d6c0 0x7f24e006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.506+0000 7f24e6dfa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24e00ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.506+0000 7f24e6dfa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e006e060 con 0x7f24e013d120 2023-11-24T08:07:34.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.506+0000 7f24e6dfa640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e013f9e0 con 0x7f24e006d6c0 2023-11-24T08:07:34.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.506+0000 7f24e6dfa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e013fcc0 con 0x7f24e00cdbf0 2023-11-24T08:07:34.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e5df8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24e013d120 0x7f24e015af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52050/0 (socket says 172.21.15.107:52050) 2023-11-24T08:07:34.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e5df8640 1 -- 172.21.15.107:0/3679899060 learned_addr learned my addr 172.21.15.107:0/3679899060 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506865367 0 0) 0x7f24e006e060 con 0x7f24e013d120 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e65f9640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24e00ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc003520 con 0x7f24e013d120 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e55f7640 1 --2- 172.21.15.107:0/3679899060 >> v2:172.21.15.120:3300/0 conn(0x7f24e006d6c0 0x7f24e006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394873022 0 0) 0x7f24cc003520 con 0x7f24e013d120 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 msgr2=0x7f24e00ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24e00ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 >> v2:172.21.15.120:3300/0 conn(0x7f24e006d6c0 msgr2=0x7f24e006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 --2- 172.21.15.107:0/3679899060 >> v2:172.21.15.120:3300/0 conn(0x7f24e006d6c0 0x7f24e006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e013ffa0 con 0x7f24e013d120 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f24dc001670 con 0x7f24e013d120 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.507+0000 7f24e65f9640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24e00ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 >> v1:172.21.15.107:6789/0 conn(0x7f24e013d120 legacy=0x7f24e015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24cc003970 0x7f24cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc004d00 0x7f24cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e013f9e0 con 0x7f24cc003970 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e013fcc0 con 0x7f24cc004d00 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc00dcf0 con 0x7f24e00cdbf0 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24cc003970 0x7f24cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 -- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 msgr2=0x7f24cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e00cdbf0 0x7f24cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 -- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc004d00 msgr2=0x7f24cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc004d00 0x7f24cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.508+0000 7f24e5df8640 1 -- 172.21.15.107:0/3679899060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc003520 con 0x7f24cc003970 2023-11-24T08:07:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.509+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24cc003970 0x7f24cc003d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c0e0 tx=0x7f24cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6265a79af6282f55 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.509+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc005ad0 con 0x7f24cc003970 2023-11-24T08:07:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc005c50 con 0x7f24cc003970 2023-11-24T08:07:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e4df6640 1 -- 172.21.15.107:0/3679899060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc02c810 con 0x7f24cc003970 2023-11-24T08:07:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3679899060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24cc003970 msgr2=0x7f24cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3679899060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24cc003970 0x7f24cc003d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c0e0 tx=0x7f24cc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e65f9640 1 -- 172.21.15.107:0/3679899060 reap_dead start 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3679899060 shutdown_connections 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3679899060 >> 172.21.15.107:0/3679899060 conn(0x7f24e00cb830 msgr2=0x7f24e00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3679899060 shutdown_connections 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.510+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3679899060 wait complete. 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.512+0000 7f24e6dfa640 1 Processor -- start 2023-11-24T08:07:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.512+0000 7f24e6dfa640 1 -- start start 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc003970 0x7f24e013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e015bf00 0x7f24e015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e015c8d0 con 0x7f24e00b3430 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00b3ef0 con 0x7f24cc003970 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e6dfa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00b41d0 con 0x7f24e015bf00 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e55f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e55f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58356/0 (socket says 172.21.15.107:58356) 2023-11-24T08:07:34.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.513+0000 7f24e55f7640 1 -- 172.21.15.107:0/3312321906 learned_addr learned my addr 172.21.15.107:0/3312321906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc003970 0x7f24e013c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 -- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e015bf00 msgr2=0x7f24e015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e015bf00 0x7f24e015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 -- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc003970 msgr2=0x7f24e013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc003970 0x7f24e013c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 -- 172.21.15.107:0/3312321906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e015a9d0 con 0x7f24e00b3430 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e55f7640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d800 tx=0x7f24d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d000b6c0 con 0x7f24e00b3430 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d000b820 con 0x7f24e00b3430 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d000bbd0 con 0x7f24e00b3430 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e00b44b0 con 0x7f24e00b3430 2023-11-24T08:07:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.514+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e00b4a70 con 0x7f24e00b3430 2023-11-24T08:07:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.515+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e00b4c30 con 0x7f24e00b3430 2023-11-24T08:07:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.516+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f24d000bd30 con 0x7f24e00b3430 2023-11-24T08:07:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.517+0000 7f24d67fc640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 0x7f24c406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.517+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f24d0089b60 con 0x7f24e00b3430 2023-11-24T08:07:34.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.518+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 0x7f24c406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:34.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.518+0000 7f24e5df8640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 0x7f24c406d600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f24dc034000 tx=0x7f24dc007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.522+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24d0054c10 con 0x7f24e00b3430 2023-11-24T08:07:34.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.747+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24e00cef00 con 0x7f24e00b3430 2023-11-24T08:07:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.748+0000 7f24d67fc640 1 -- 172.21.15.107:0/3312321906 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f24d0058230 con 0x7f24e00b3430 2023-11-24T08:07:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:34.751 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4188072964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4188072964},{"type":"v1","addr":"172.21.15.107:6835","nonce":4188072964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/846888183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":846888183},{"type":"v1","addr":"172.21.15.120:6833","nonce":846888183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3812952531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3812952531},{"type":"v1","addr":"172.21.15.120:6835","nonce":3812952531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5565,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1471255325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1471255325},{"type":"v1","addr":"172.21.15.107:6837","nonce":1471255325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-11-24T08:07:34.751 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-11-24T08:07:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 msgr2=0x7f24c406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 0x7f24c406d600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f24dc034000 tx=0x7f24dc007000 comp rx=0 tx=0).stop 2023-11-24T08:07:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 msgr2=0x7f24e00b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d800 tx=0x7f24d000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 shutdown_connections 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c406b140 0x7f24c406d600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e015bf00 0x7f24e015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e00b3430 0x7f24e00b38b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 --2- 172.21.15.107:0/3312321906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24cc003970 0x7f24e013c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 >> 172.21.15.107:0/3312321906 conn(0x7f24e00cb830 msgr2=0x7f24e006ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 shutdown_connections 2023-11-24T08:07:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:34.751+0000 7f24e6dfa640 1 -- 172.21.15.107:0/3312321906 wait complete. 2023-11-24T08:07:34.773 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:07:34.811 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:07:34.811 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:07:34.812 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:07:34.812 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:07:34.812 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:07:34.812 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:07:34.812 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:07:34.812 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:07:34.813 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:07:34.813 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:07:34.813 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:07:34.813 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:07:34.813 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:07:34.813 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:07:34.814 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:07:34.833 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:07:34.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:07:34.834 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:07:34.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:07:34.835 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:07:34.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:07:34.837 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:07:34.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:07:34.883 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:07:34.883 DEBUG:teuthology.parallel:result is None 2023-11-24T08:07:34.885 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:07:34.885 DEBUG:teuthology.parallel:result is None 2023-11-24T08:07:34.891 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:07:34.891 DEBUG:teuthology.parallel:result is None 2023-11-24T08:07:34.892 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:07:34.893 DEBUG:teuthology.parallel:result is None 2023-11-24T08:07:34.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:07:35.045 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:07:35.047 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:07:35.048 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:07:35.048 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:07:35.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.081+0000 7fc450cda640 1 Processor -- start 2023-11-24T08:07:35.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.081+0000 7fc450cda640 1 -- start start 2023-11-24T08:07:35.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.082+0000 7fc450cda640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc44c146eb0 0x7fc44c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.082+0000 7fc450cda640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 0x7fc44c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.082+0000 7fc450cda640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44c161a10 con 0x7fc44c06e160 2023-11-24T08:07:35.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.082+0000 7fc450cda640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c161cf0 con 0x7fc44c146eb0 2023-11-24T08:07:35.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.082+0000 7fc450cda640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c161fd0 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc443fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc44c06e160 0x7fc44c0ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52076/0 (socket says 172.21.15.107:52076) 2023-11-24T08:07:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc443fff640 1 -- 172.21.15.107:0/1502224574 learned_addr learned my addr 172.21.15.107:0/1502224574 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44a59c640 1 --2- 172.21.15.107:0/1502224574 >> v2:172.21.15.120:3300/0 conn(0x7fc44c146eb0 0x7fc44c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/1502224574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 0x7fc44c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc449d9b640 1 -- 172.21.15.107:0/1502224574 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469170115 0 0) 0x7fc44c161a10 con 0x7fc44c06e160 2023-11-24T08:07:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44ad9d640 1 -- 172.21.15.107:0/1502224574 >> v2:172.21.15.120:3300/0 conn(0x7fc44c146eb0 msgr2=0x7fc44c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/1502224574 >> v2:172.21.15.120:3300/0 conn(0x7fc44c146eb0 0x7fc44c161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44ad9d640 1 -- 172.21.15.107:0/1502224574 >> v1:172.21.15.107:6789/0 conn(0x7fc44c06e160 legacy=0x7fc44c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.083+0000 7fc44ad9d640 1 -- 172.21.15.107:0/1502224574 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c070660 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.084+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/1502224574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 0x7fc44c070e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc444009d90 tx=0x7fc44402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8a96f4955285f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc449d9b640 1 -- 172.21.15.107:0/1502224574 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44402cc60 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc449d9b640 1 -- 172.21.15.107:0/1502224574 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44402cdc0 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc449d9b640 1 -- 172.21.15.107:0/1502224574 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444034660 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 -- 172.21.15.107:0/1502224574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 msgr2=0x7fc44c070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 --2- 172.21.15.107:0/1502224574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 0x7fc44c070e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc444009d90 tx=0x7fc44402beb0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 -- 172.21.15.107:0/1502224574 shutdown_connections 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 --2- 172.21.15.107:0/1502224574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c0cd2e0 0x7fc44c070e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 --2- 172.21.15.107:0/1502224574 >> v2:172.21.15.120:3300/0 conn(0x7fc44c146eb0 0x7fc44c161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 -- 172.21.15.107:0/1502224574 >> 172.21.15.107:0/1502224574 conn(0x7fc44c0b0c50 msgr2=0x7fc44c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 -- 172.21.15.107:0/1502224574 shutdown_connections 2023-11-24T08:07:35.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.085+0000 7fc450cda640 1 -- 172.21.15.107:0/1502224574 wait complete. 2023-11-24T08:07:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.087+0000 7fc450cda640 1 Processor -- start 2023-11-24T08:07:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.087+0000 7fc450cda640 1 -- start start 2023-11-24T08:07:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06e160 0x7fc44c15ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c0cd2e0 0x7fc44c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0abf30 con 0x7fc44c146eb0 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c15d390 con 0x7fc44c0cd2e0 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.088+0000 7fc450cda640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c09a2d0 con 0x7fc44c06e160 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58404/0 (socket says 172.21.15.107:58404) 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 -- 172.21.15.107:0/3630349705 learned_addr learned my addr 172.21.15.107:0/3630349705 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44a59c640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06e160 0x7fc44c15ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc443fff640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c0cd2e0 0x7fc44c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 -- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06e160 msgr2=0x7fc44c15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06e160 0x7fc44c15ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 -- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c0cd2e0 msgr2=0x7fc44c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c0cd2e0 0x7fc44c0999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 -- 172.21.15.107:0/3630349705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009a70 con 0x7fc44c146eb0 2023-11-24T08:07:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.089+0000 7fc44ad9d640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc444009eb0 tx=0x7fc444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:35.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.090+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444040650 con 0x7fc44c146eb0 2023-11-24T08:07:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.090+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4440407b0 con 0x7fc44c146eb0 2023-11-24T08:07:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.090+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c09a5b0 con 0x7fc44c146eb0 2023-11-24T08:07:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.090+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444040b60 con 0x7fc44c146eb0 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.090+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c0ac180 con 0x7fc44c146eb0 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.091+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c04e8b0 con 0x7fc44c146eb0 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.092+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fc444040cc0 con 0x7fc44c146eb0 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.092+0000 7fc442ffd640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 0x7fc42c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.093+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc4440ac4c0 con 0x7fc44c146eb0 2023-11-24T08:07:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.093+0000 7fc44a59c640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 0x7fc42c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:35.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.094+0000 7fc44a59c640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 0x7fc42c06d600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009a30 tx=0x7fc43c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:35.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.097+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4440774e0 con 0x7fc44c146eb0 2023-11-24T08:07:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:35.293+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc44c06ded0 con 0x7fc44c146eb0 2023-11-24T08:07:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.383+0000 7fc442ffd640 1 -- 172.21.15.107:0/3630349705 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc444009450 con 0x7fc44c146eb0 2023-11-24T08:07:36.385 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.386+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 msgr2=0x7fc42c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.386+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 0x7fc42c06d600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009a30 tx=0x7fc43c008040 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.386+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 msgr2=0x7fc44c15e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.386+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc444009eb0 tx=0x7fc444004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.386+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 shutdown_connections 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc42c06b140 0x7fc42c06d600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c146eb0 0x7fc44c15e6c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c0cd2e0 0x7fc44c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 --2- 172.21.15.107:0/3630349705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06e160 0x7fc44c15ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 >> 172.21.15.107:0/3630349705 conn(0x7fc44c0b0c50 msgr2=0x7fc44c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 shutdown_connections 2023-11-24T08:07:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.387+0000 7fc450cda640 1 -- 172.21.15.107:0/3630349705 wait complete. 2023-11-24T08:07:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:07:36.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.620+0000 7f0b4a075640 1 Processor -- start 2023-11-24T08:07:36.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.620+0000 7f0b4a075640 1 -- start start 2023-11-24T08:07:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b4a075640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b4415a4f0 0x7f0b440b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b4a075640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b440cdfb0 0x7f0b440ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b4a075640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b440b2c00 con 0x7f0b440b20c0 2023-11-24T08:07:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b4a075640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440ce980 con 0x7f0b4415a4f0 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b4a075640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440b3a40 con 0x7f0b440cdfb0 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.621+0000 7f0b437fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b4415a4f0 0x7f0b440b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b4415a4f0 0x7f0b440b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58416/0 (socket says 172.21.15.107:58416) 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 -- 172.21.15.107:0/3717784885 learned_addr learned my addr 172.21.15.107:0/3717784885 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 -- 172.21.15.107:0/3717784885 >> v2:172.21.15.120:3300/0 conn(0x7f0b440cdfb0 msgr2=0x7f0b440ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 --2- 172.21.15.107:0/3717784885 >> v2:172.21.15.120:3300/0 conn(0x7f0b440cdfb0 0x7f0b440ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 -- 172.21.15.107:0/3717784885 >> v1:172.21.15.120:6789/0 conn(0x7f0b440b20c0 legacy=0x7f0b440b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.622+0000 7f0b437fe640 1 -- 172.21.15.107:0/3717784885 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440b2e30 con 0x7f0b4415a4f0 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.623+0000 7f0b437fe640 1 --2- 172.21.15.107:0/3717784885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b440b3ed0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0b440aeb90 tx=0x7f0b3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce56c3308af9a1a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.623+0000 7f0b427fc640 1 -- 172.21.15.107:0/3717784885 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34031870 con 0x7f0b4415a4f0 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.623+0000 7f0b427fc640 1 -- 172.21.15.107:0/3717784885 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b340319d0 con 0x7f0b4415a4f0 2023-11-24T08:07:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.623+0000 7f0b427fc640 1 -- 172.21.15.107:0/3717784885 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34031d60 con 0x7f0b4415a4f0 2023-11-24T08:07:36.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 -- 172.21.15.107:0/3717784885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 msgr2=0x7f0b440b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 --2- 172.21.15.107:0/3717784885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b440b3ed0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0b440aeb90 tx=0x7f0b3402b910 comp rx=0 tx=0).stop 2023-11-24T08:07:36.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 -- 172.21.15.107:0/3717784885 shutdown_connections 2023-11-24T08:07:36.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 --2- 172.21.15.107:0/3717784885 >> v2:172.21.15.120:3300/0 conn(0x7f0b440cdfb0 0x7f0b440ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 --2- 172.21.15.107:0/3717784885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b440b3ed0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 -- 172.21.15.107:0/3717784885 >> 172.21.15.107:0/3717784885 conn(0x7f0b440b0c70 msgr2=0x7f0b440b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:36.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.624+0000 7f0b4a075640 1 -- 172.21.15.107:0/3717784885 shutdown_connections 2023-11-24T08:07:36.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.625+0000 7f0b4a075640 1 -- 172.21.15.107:0/3717784885 wait complete. 2023-11-24T08:07:36.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.626+0000 7f0b4a075640 1 Processor -- start 2023-11-24T08:07:36.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.626+0000 7f0b4a075640 1 -- start start 2023-11-24T08:07:36.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b440b20c0 0x7f0b441f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 0x7f0b441f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4415b580 con 0x7f0b4415a4f0 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4415a990 con 0x7f0b440cdfb0 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.627+0000 7f0b4a075640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4415ac50 con 0x7f0b440b20c0 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58432/0 (socket says 172.21.15.107:58432) 2023-11-24T08:07:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 -- 172.21.15.107:0/1702019096 learned_addr learned my addr 172.21.15.107:0/1702019096 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:36.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b42ffd640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 0x7f0b441f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:36.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b437fe640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b440b20c0 0x7f0b441f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:36.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 -- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b440b20c0 msgr2=0x7f0b441f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b440b20c0 0x7f0b441f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 -- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 msgr2=0x7f0b441f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 0x7f0b441f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 -- 172.21.15.107:0/1702019096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b340094d0 con 0x7f0b4415a4f0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b42ffd640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 0x7f0b441f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.628+0000 7f0b43fff640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c390 tx=0x7f0b3c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.629+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0035f0 con 0x7f0b4415a4f0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.629+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c003750 con 0x7f0b4415a4f0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.629+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c003b00 con 0x7f0b4415a4f0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.629+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4415af30 con 0x7f0b4415a4f0 2023-11-24T08:07:36.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.629+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b441f4670 con 0x7f0b4415a4f0 2023-11-24T08:07:36.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.630+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4415b730 con 0x7f0b4415a4f0 2023-11-24T08:07:36.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.631+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f0b3c003c60 con 0x7f0b4415a4f0 2023-11-24T08:07:36.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.632+0000 7f0b3bfff640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 0x7f0b2406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:36.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.632+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0b3c088880 con 0x7f0b4415a4f0 2023-11-24T08:07:36.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.632+0000 7f0b437fe640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 0x7f0b2406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.633+0000 7f0b437fe640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 0x7f0b2406d600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b34036000 tx=0x7f0b3403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:36.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.637+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b4415b730 con 0x7f0b4415a4f0 2023-11-24T08:07:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.874+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0b4404e8b0 con 0x7f0b4415a4f0 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.875+0000 7f0b3bfff640 1 -- 172.21.15.107:0/1702019096 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0b3c0538e0 con 0x7f0b4415a4f0 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:07:36.877 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.878 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:07:36.879 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.881 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:07:36.882 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:07:36.885 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:07:36.886 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.888 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:07:36.888 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:07:36.888 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.888 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.888 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.889 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.890 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.891 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:07:36.892 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:36.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:07:36.894 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 msgr2=0x7f0b2406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 0x7f0b2406d600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b34036000 tx=0x7f0b3403c040 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 msgr2=0x7f0b441effb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c390 tx=0x7f0b3c00c850 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 shutdown_connections 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b2406b140 0x7f0b2406d600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b4415a4f0 0x7f0b441effb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b440cdfb0 0x7f0b441f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 --2- 172.21.15.107:0/1702019096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b440b20c0 0x7f0b441f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 >> 172.21.15.107:0/1702019096 conn(0x7f0b440b0c70 msgr2=0x7f0b440cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 shutdown_connections 2023-11-24T08:07:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:36.877+0000 7f0b4a075640 1 -- 172.21.15.107:0/1702019096 wait complete. 2023-11-24T08:07:36.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.101+0000 7fc393fff640 1 Processor -- start 2023-11-24T08:07:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.102+0000 7fc393fff640 1 -- start start 2023-11-24T08:07:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.103+0000 7fc393fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc394159e80 0x7fc394071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.103+0000 7fc393fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc39415a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.103+0000 7fc393fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940ce870 con 0x7fc39415a7a0 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.103+0000 7fc393fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3940cdc60 con 0x7fc394159e80 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.103+0000 7fc393fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3940cdf40 con 0x7fc3940cd6a0 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc392ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc394159e80 0x7fc394071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc3937fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc39415a7a0 0x7fc39406c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52078/0 (socket says 172.21.15.107:52078) 2023-11-24T08:07:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc3937fe640 1 -- 172.21.15.107:0/931452395 learned_addr learned my addr 172.21.15.107:0/931452395 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc38bfff640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc39415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc3927fc640 1 -- 172.21.15.107:0/931452395 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239702575 0 0) 0x7fc3940ce870 con 0x7fc39415a7a0 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc3927fc640 1 -- 172.21.15.107:0/931452395 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380003520 con 0x7fc39415a7a0 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc392ffd640 1 -- 172.21.15.107:0/931452395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 msgr2=0x7fc39415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc392ffd640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc39415a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc392ffd640 1 -- 172.21.15.107:0/931452395 >> v1:172.21.15.107:6789/0 conn(0x7fc39415a7a0 legacy=0x7fc39406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.104+0000 7fc392ffd640 1 -- 172.21.15.107:0/931452395 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3940ce220 con 0x7fc394159e80 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.105+0000 7fc392ffd640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc394159e80 0x7fc394071610 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc3840097a0 tx=0x7fc38402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1790b5875cb9297e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc3927fc640 1 -- 172.21.15.107:0/931452395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384031780 con 0x7fc394159e80 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc3927fc640 1 -- 172.21.15.107:0/931452395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3840318e0 con 0x7fc394159e80 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc3927fc640 1 -- 172.21.15.107:0/931452395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384031c70 con 0x7fc394159e80 2023-11-24T08:07:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 -- 172.21.15.107:0/931452395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc394159e80 msgr2=0x7fc394071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc394159e80 0x7fc394071610 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc3840097a0 tx=0x7fc38402b820 comp rx=0 tx=0).stop 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 -- 172.21.15.107:0/931452395 shutdown_connections 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc39415a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 --2- 172.21.15.107:0/931452395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc394159e80 0x7fc394071610 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 -- 172.21.15.107:0/931452395 >> 172.21.15.107:0/931452395 conn(0x7fc3940cb830 msgr2=0x7fc3940cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 -- 172.21.15.107:0/931452395 shutdown_connections 2023-11-24T08:07:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.106+0000 7fc393fff640 1 -- 172.21.15.107:0/931452395 wait complete. 2023-11-24T08:07:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.108+0000 7fc393fff640 1 Processor -- start 2023-11-24T08:07:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.108+0000 7fc393fff640 1 -- start start 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc3941389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc394159e80 0x7fc394152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39414e480 con 0x7fc39415a7a0 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc394153760 con 0x7fc394159e80 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc393fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc394153a40 con 0x7fc3940cd6a0 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc3937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc3937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58456/0 (socket says 172.21.15.107:58456) 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.109+0000 7fc3937fe640 1 -- 172.21.15.107:0/3734838477 learned_addr learned my addr 172.21.15.107:0/3734838477 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc38bfff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc394159e80 0x7fc394152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc392ffd640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc3941389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 -- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 msgr2=0x7fc3941389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc3941389e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 -- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc394159e80 msgr2=0x7fc394152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc394159e80 0x7fc394152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 -- 172.21.15.107:0/3734838477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384009400 con 0x7fc39415a7a0 2023-11-24T08:07:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc3937fe640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00cb40 tx=0x7fc38c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00aca0 con 0x7fc39415a7a0 2023-11-24T08:07:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c00ae00 con 0x7fc39415a7a0 2023-11-24T08:07:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c012620 con 0x7fc39415a7a0 2023-11-24T08:07:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc394153d20 con 0x7fc39415a7a0 2023-11-24T08:07:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.110+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc394138f20 con 0x7fc39415a7a0 2023-11-24T08:07:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.111+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39414e5c0 con 0x7fc39415a7a0 2023-11-24T08:07:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.112+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fc38c007070 con 0x7fc39415a7a0 2023-11-24T08:07:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.113+0000 7fc38b7fe640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 0x7fc37406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.113+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc38c089790 con 0x7fc39415a7a0 2023-11-24T08:07:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.113+0000 7fc392ffd640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 0x7fc37406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.114+0000 7fc392ffd640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 0x7fc37406d5d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc384036000 tx=0x7fc38403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.117+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc39414e5c0 con 0x7fc39415a7a0 2023-11-24T08:07:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.308+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc39414e5c0 con 0x7fc39415a7a0 2023-11-24T08:07:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.309+0000 7fc38b7fe640 1 -- 172.21.15.107:0/3734838477 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fc39414e5c0 con 0x7fc39415a7a0 2023-11-24T08:07:37.311 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:37.312 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:36.380533+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.311+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 msgr2=0x7fc37406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.311+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 0x7fc37406d5d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc384036000 tx=0x7fc38403d040 comp rx=0 tx=0).stop 2023-11-24T08:07:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.311+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 msgr2=0x7fc394153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.311+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00cb40 tx=0x7fc38c00a060 comp rx=0 tx=0).stop 2023-11-24T08:07:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 shutdown_connections 2023-11-24T08:07:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc37406b110 0x7fc37406d5d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc39415a7a0 0x7fc394153150 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc394159e80 0x7fc394152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 --2- 172.21.15.107:0/3734838477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3940cd6a0 0x7fc3941389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 >> 172.21.15.107:0/3734838477 conn(0x7fc3940cb830 msgr2=0x7fc394159250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 shutdown_connections 2023-11-24T08:07:37.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.312+0000 7fc393fff640 1 -- 172.21.15.107:0/3734838477 wait complete. 2023-11-24T08:07:37.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:07:37.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:07:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.551+0000 7fbb0b548640 1 Processor -- start 2023-11-24T08:07:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.551+0000 7fbb0b548640 1 -- start start 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb04072160 0x7fbb0415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb040c49f0 0x7fbb0406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbb0406dc40 0x7fbb040cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0406e0f0 con 0x7fbb040c49f0 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0415ec90 con 0x7fbb0406dc40 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb0b548640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040cd530 con 0x7fbb04072160 2023-11-24T08:07:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb08ae3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb040c49f0 0x7fbb0406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb08ae3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb040c49f0 0x7fbb0406d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58460/0 (socket says 172.21.15.107:58460) 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.552+0000 7fbb08ae3640 1 -- 172.21.15.107:0/2264978521 learned_addr learned my addr 172.21.15.107:0/2264978521 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb09ae5640 1 --2- 172.21.15.107:0/2264978521 >> v2:172.21.15.120:3300/0 conn(0x7fbb0406dc40 0x7fbb040cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb092e4640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb04072160 0x7fbb0415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb08ae3640 1 -- 172.21.15.107:0/2264978521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb04072160 msgr2=0x7fbb0415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb08ae3640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb04072160 0x7fbb0415e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb08ae3640 1 -- 172.21.15.107:0/2264978521 >> v2:172.21.15.120:3300/0 conn(0x7fbb0406dc40 msgr2=0x7fbb040cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb08ae3640 1 --2- 172.21.15.107:0/2264978521 >> v2:172.21.15.120:3300/0 conn(0x7fbb0406dc40 0x7fbb040cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.553+0000 7fbb08ae3640 1 -- 172.21.15.107:0/2264978521 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040c4ed0 con 0x7fbb040c49f0 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.554+0000 7fbb08ae3640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb0406d700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbb04072a30 tx=0x7fbaf402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804e5af5af343a9e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.554+0000 7fbafbfff640 1 -- 172.21.15.107:0/2264978521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4031860 con 0x7fbb040c49f0 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.554+0000 7fbafbfff640 1 -- 172.21.15.107:0/2264978521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf40319c0 con 0x7fbb040c49f0 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.554+0000 7fbafbfff640 1 -- 172.21.15.107:0/2264978521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4031d50 con 0x7fbb040c49f0 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 -- 172.21.15.107:0/2264978521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 msgr2=0x7fbb0406d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb0406d700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbb04072a30 tx=0x7fbaf402b900 comp rx=0 tx=0).stop 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 -- 172.21.15.107:0/2264978521 shutdown_connections 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 --2- 172.21.15.107:0/2264978521 >> v2:172.21.15.120:3300/0 conn(0x7fbb0406dc40 0x7fbb040cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb0406d700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 --2- 172.21.15.107:0/2264978521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb04072160 0x7fbb0415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 -- 172.21.15.107:0/2264978521 >> 172.21.15.107:0/2264978521 conn(0x7fbb040b0c50 msgr2=0x7fbb040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 -- 172.21.15.107:0/2264978521 shutdown_connections 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.555+0000 7fbb0b548640 1 -- 172.21.15.107:0/2264978521 wait complete. 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.557+0000 7fbb0b548640 1 Processor -- start 2023-11-24T08:07:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.557+0000 7fbb0b548640 1 -- start start 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 0x7fbb040ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb04072160 0x7fbb04149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040caa70 con 0x7fbb040c49f0 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040cad50 con 0x7fbb04072160 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb0b548640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040cb030 con 0x7fbb0406dc40 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb09ae5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb09ae5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58466/0 (socket says 172.21.15.107:58466) 2023-11-24T08:07:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb09ae5640 1 -- 172.21.15.107:0/1698562938 learned_addr learned my addr 172.21.15.107:0/1698562938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.558+0000 7fbb08ae3640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb04072160 0x7fbb04149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb092e4640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 0x7fbb040ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 -- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 msgr2=0x7fbb040ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 0x7fbb040ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 -- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb04072160 msgr2=0x7fbb04149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb04072160 0x7fbb04149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 -- 172.21.15.107:0/1698562938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf40094d0 con 0x7fbb040c49f0 2023-11-24T08:07:37.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb092e4640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 0x7fbb040ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb09ae5640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00ccf0 tx=0x7fbafc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc003b90 con 0x7fbb040c49f0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc003cf0 con 0x7fbb040c49f0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb040cb310 con 0x7fbb040c49f0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc004240 con 0x7fbb040c49f0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.559+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb040ca780 con 0x7fbb040c49f0 2023-11-24T08:07:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.560+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb040cb830 con 0x7fbb040c49f0 2023-11-24T08:07:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.561+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fbafc0043a0 con 0x7fbb040c49f0 2023-11-24T08:07:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.562+0000 7fbaf9ffb640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 0x7fbae806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.562+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbafc012070 con 0x7fbb040c49f0 2023-11-24T08:07:37.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.563+0000 7fbb092e4640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 0x7fbae806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.564+0000 7fbb092e4640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 0x7fbae806d5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbb00008890 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.567+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb040cb830 con 0x7fbb040c49f0 2023-11-24T08:07:37.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.743+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb040cb830 con 0x7fbb040c49f0 2023-11-24T08:07:37.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.743+0000 7fbaf9ffb640 1 -- 172.21.15.107:0/1698562938 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb040cb830 con 0x7fbb040c49f0 2023-11-24T08:07:37.745 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:07:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 msgr2=0x7fbae806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 0x7fbae806d5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbb00008890 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2023-11-24T08:07:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 msgr2=0x7fbb040cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00ccf0 tx=0x7fbafc00a660 comp rx=0 tx=0).stop 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 shutdown_connections 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbae806b110 0x7fbae806d5d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb040c49f0 0x7fbb040cbb10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb04072160 0x7fbb04149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 --2- 172.21.15.107:0/1698562938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb0406dc40 0x7fbb040ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 >> 172.21.15.107:0/1698562938 conn(0x7fbb040b0c50 msgr2=0x7fbb040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 shutdown_connections 2023-11-24T08:07:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.746+0000 7fbb0b548640 1 -- 172.21.15.107:0/1698562938 wait complete. 2023-11-24T08:07:37.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:07:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.968+0000 7f8568941640 1 Processor -- start 2023-11-24T08:07:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.968+0000 7f8568941640 1 -- start start 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8568941640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f856415c7f0 0x7f856406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8568941640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 0x7f85640cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8568941640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640cece0 con 0x7f85640cf450 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8568941640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640cfe40 con 0x7f856415c7f0 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8568941640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640ceaf0 con 0x7f85640ce6c0 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f856259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f856415c7f0 0x7f856406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8559d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85640cf450 0x7f85640cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32928/0 (socket says 172.21.15.107:32928) 2023-11-24T08:07:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f856259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f856415c7f0 0x7f856406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42248/0 (socket says 172.21.15.107:42248) 2023-11-24T08:07:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.969+0000 7f8559d9b640 1 -- 172.21.15.107:0/4239407275 learned_addr learned my addr 172.21.15.107:0/4239407275 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f8562d9d640 1 --2- 172.21.15.107:0/4239407275 >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 0x7f85640cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f856259c640 1 -- 172.21.15.107:0/4239407275 >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 msgr2=0x7f85640cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f856259c640 1 --2- 172.21.15.107:0/4239407275 >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 0x7f85640cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f856259c640 1 -- 172.21.15.107:0/4239407275 >> v1:172.21.15.107:6789/0 conn(0x7f85640cf450 legacy=0x7f85640cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f856259c640 1 -- 172.21.15.107:0/4239407275 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856406d040 con 0x7f856415c7f0 2023-11-24T08:07:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f8561d9b640 1 -- 172.21.15.107:0/4239407275 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979532114 0 0) 0x7f85640cece0 con 0x7f85640cf450 2023-11-24T08:07:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.970+0000 7f8562d9d640 1 --2- 172.21.15.107:0/4239407275 >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 0x7f85640cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.971+0000 7f856259c640 1 --2- 172.21.15.107:0/4239407275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f856415c7f0 0x7f856406db20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f85640d00e0 tx=0x7f855402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249842177dd5ca21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.971+0000 7f8561d9b640 1 -- 172.21.15.107:0/4239407275 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031830 con 0x7f856415c7f0 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.971+0000 7f8561d9b640 1 -- 172.21.15.107:0/4239407275 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554031990 con 0x7f856415c7f0 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.971+0000 7f8561d9b640 1 -- 172.21.15.107:0/4239407275 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031d20 con 0x7f856415c7f0 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 -- 172.21.15.107:0/4239407275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f856415c7f0 msgr2=0x7f856406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 --2- 172.21.15.107:0/4239407275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f856415c7f0 0x7f856406db20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f85640d00e0 tx=0x7f855402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 -- 172.21.15.107:0/4239407275 shutdown_connections 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 --2- 172.21.15.107:0/4239407275 >> v2:172.21.15.120:3300/0 conn(0x7f85640ce6c0 0x7f85640cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 --2- 172.21.15.107:0/4239407275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f856415c7f0 0x7f856406db20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 -- 172.21.15.107:0/4239407275 >> 172.21.15.107:0/4239407275 conn(0x7f856406a920 msgr2=0x7f856406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 -- 172.21.15.107:0/4239407275 shutdown_connections 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.972+0000 7f8568941640 1 -- 172.21.15.107:0/4239407275 wait complete. 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.974+0000 7f8568941640 1 Processor -- start 2023-11-24T08:07:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.974+0000 7f8568941640 1 -- start start 2023-11-24T08:07:37.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85640ce6c0 0x7f8564155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f856415c7f0 0x7f85640ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85640b41b0 con 0x7f85640cf450 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85641564e0 con 0x7f856415c7f0 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8568941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85641567c0 con 0x7f85640ce6c0 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42258/0 (socket says 172.21.15.107:42258) 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 -- 172.21.15.107:0/1831068429 learned_addr learned my addr 172.21.15.107:0/1831068429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 -- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85640ce6c0 msgr2=0x7f8564155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85640ce6c0 0x7f8564155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 -- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f856415c7f0 msgr2=0x7f85640ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f856415c7f0 0x7f85640ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.975+0000 7f8559d9b640 1 -- 172.21.15.107:0/1831068429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554009400 con 0x7f85640cf450 2023-11-24T08:07:37.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f8559d9b640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f854c00c4d0 tx=0x7f854c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0074e0 con 0x7f85640cf450 2023-11-24T08:07:37.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c007660 con 0x7f85640cf450 2023-11-24T08:07:37.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8564156aa0 con 0x7f85640cf450 2023-11-24T08:07:37.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c004a20 con 0x7f85640cf450 2023-11-24T08:07:37.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.976+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8564157060 con 0x7f85640cf450 2023-11-24T08:07:37.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.977+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8564157280 con 0x7f85640cf450 2023-11-24T08:07:37.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.978+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f854c004b80 con 0x7f85640cf450 2023-11-24T08:07:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.979+0000 7f855b7fe640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 0x7f854806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.979+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f854c088d60 con 0x7f85640cf450 2023-11-24T08:07:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.979+0000 7f856259c640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 0x7f854806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.980+0000 7f856259c640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 0x7f854806d5d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f8554030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:37.983+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f854c053cf0 con 0x7f85640cf450 2023-11-24T08:07:38.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.176+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8564061a50 con 0x7f85640cf450 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.177+0000 7f855b7fe640 1 -- 172.21.15.107:0/1831068429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) v1 ==== 84+0+65 (secure 0 0 0) 0x7f854c018020 con 0x7f85640cf450 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:07:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:07:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:38.180 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 msgr2=0x7f854806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 0x7f854806d5d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f8554030000 comp rx=0 tx=0).stop 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 msgr2=0x7f85640b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f854c00c4d0 tx=0x7f854c00c990 comp rx=0 tx=0).stop 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 shutdown_connections 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f854806b110 0x7f854806d5d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f856415c7f0 0x7f85640ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85640cf450 0x7f85640b3c70 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 --2- 172.21.15.107:0/1831068429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85640ce6c0 0x7f8564155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 >> 172.21.15.107:0/1831068429 conn(0x7f856406a920 msgr2=0x7f85640cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 shutdown_connections 2023-11-24T08:07:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.180+0000 7f8568941640 1 -- 172.21.15.107:0/1831068429 wait complete. 2023-11-24T08:07:38.198 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:07:38.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:07:38.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.407+0000 7f1613e15640 1 Processor -- start 2023-11-24T08:07:38.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.408+0000 7f1613e15640 1 -- start start 2023-11-24T08:07:38.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1613e15640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f160c15c7f0 0x7f160c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1613e15640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f160c0ce6c0 0x7f160c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1613e15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0cece0 con 0x7f160c0cf450 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1613e15640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0cfe40 con 0x7f160c0ce6c0 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1613e15640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0ceaf0 con 0x7f160c15c7f0 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f16123b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f160c0ce6c0 0x7f160c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f16123b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f160c0ce6c0 0x7f160c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42270/0 (socket says 172.21.15.107:42270) 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f16123b2640 1 -- 172.21.15.107:0/1160356082 learned_addr learned my addr 172.21.15.107:0/1160356082 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f16113b0640 1 --1- 172.21.15.107:0/1160356082 >> v1:172.21.15.107:6789/0 conn(0x7f160c0cf450 0x7f160c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32940/0 (socket says 172.21.15.107:32940) 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.409+0000 7f1611bb1640 1 --2- 172.21.15.107:0/1160356082 >> v2:172.21.15.120:3300/0 conn(0x7f160c15c7f0 0x7f160c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.410+0000 7f16123b2640 1 -- 172.21.15.107:0/1160356082 >> v2:172.21.15.120:3300/0 conn(0x7f160c15c7f0 msgr2=0x7f160c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.410+0000 7f16123b2640 1 --2- 172.21.15.107:0/1160356082 >> v2:172.21.15.120:3300/0 conn(0x7f160c15c7f0 0x7f160c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.410+0000 7f16123b2640 1 -- 172.21.15.107:0/1160356082 >> v1:172.21.15.107:6789/0 conn(0x7f160c0cf450 legacy=0x7f160c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.410+0000 7f16123b2640 1 -- 172.21.15.107:0/1160356082 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c06d040 con 0x7f160c0ce6c0 2023-11-24T08:07:38.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.410+0000 7f1610baf640 1 -- 172.21.15.107:0/1160356082 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122855697 0 0) 0x7f160c0cece0 con 0x7f160c0cf450 2023-11-24T08:07:38.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f16123b2640 1 --2- 172.21.15.107:0/1160356082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c0ce6c0 0x7f160c0cf900 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f160c0d0130 tx=0x7f160402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d9f8c58952b25e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:38.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f1610baf640 1 -- 172.21.15.107:0/1160356082 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402cb20 con 0x7f160c0ce6c0 2023-11-24T08:07:38.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f1610baf640 1 -- 172.21.15.107:0/1160356082 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160402cc80 con 0x7f160c0ce6c0 2023-11-24T08:07:38.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f1610baf640 1 -- 172.21.15.107:0/1160356082 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16040343f0 con 0x7f160c0ce6c0 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f1613e15640 1 -- 172.21.15.107:0/1160356082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c0ce6c0 msgr2=0x7f160c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.411+0000 7f1613e15640 1 --2- 172.21.15.107:0/1160356082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c0ce6c0 0x7f160c0cf900 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f160c0d0130 tx=0x7f160402bf80 comp rx=0 tx=0).stop 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 -- 172.21.15.107:0/1160356082 shutdown_connections 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 --2- 172.21.15.107:0/1160356082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c0ce6c0 0x7f160c0cf900 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 --2- 172.21.15.107:0/1160356082 >> v2:172.21.15.120:3300/0 conn(0x7f160c15c7f0 0x7f160c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 -- 172.21.15.107:0/1160356082 >> 172.21.15.107:0/1160356082 conn(0x7f160c06a920 msgr2=0x7f160c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 -- 172.21.15.107:0/1160356082 shutdown_connections 2023-11-24T08:07:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.412+0000 7f1613e15640 1 -- 172.21.15.107:0/1160356082 wait complete. 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.413+0000 7f1613e15640 1 Processor -- start 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.414+0000 7f1613e15640 1 -- start start 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 0x7f160c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f160c0cf450 0x7f160c0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c15a8f0 con 0x7f160c15c7f0 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c15aeb0 con 0x7f160c0ce6c0 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1613e15640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c0b3480 con 0x7f160c0cf450 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42282/0 (socket says 172.21.15.107:42282) 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 -- 172.21.15.107:0/2621052080 learned_addr learned my addr 172.21.15.107:0/2621052080 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1611bb1640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 0x7f160c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16113b0640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f160c0cf450 0x7f160c0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 -- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f160c0cf450 msgr2=0x7f160c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f160c0cf450 0x7f160c0b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 -- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 msgr2=0x7f160c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 0x7f160c158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f16123b2640 1 -- 172.21.15.107:0/2621052080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1604009b30 con 0x7f160c15c7f0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.415+0000 7f1611bb1640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 0x7f160c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f16123b2640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1604036040 tx=0x7f1604004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604009510 con 0x7f160c15c7f0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160402c920 con 0x7f160c15c7f0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160403e690 con 0x7f160c15c7f0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c0b3760 con 0x7f160c15c7f0 2023-11-24T08:07:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.416+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c15b880 con 0x7f160c15c7f0 2023-11-24T08:07:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.417+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c15b020 con 0x7f160c15c7f0 2023-11-24T08:07:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.418+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f160403e7f0 con 0x7f160c15c7f0 2023-11-24T08:07:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.419+0000 7f16027fc640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 0x7f15f006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.419+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f16040ac280 con 0x7f160c15c7f0 2023-11-24T08:07:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.419+0000 7f1611bb1640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 0x7f15f006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.420+0000 7f1611bb1640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 0x7f15f006d5d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1608006fd0 tx=0x7f1608008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:38.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.424+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16040772a0 con 0x7f160c15c7f0 2023-11-24T08:07:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:38.613+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f160c067470 con 0x7f160c15c7f0 2023-11-24T08:07:39.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.410+0000 7f16027fc640 1 -- 172.21.15.107:0/2621052080 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) v1 ==== 138+0+0 (secure 0 0 0) 0x7f160407a8c0 con 0x7f160c15c7f0 2023-11-24T08:07:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 msgr2=0x7f15f006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 0x7f15f006d5d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1608006fd0 tx=0x7f1608008040 comp rx=0 tx=0).stop 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 msgr2=0x7f160c15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1604036040 tx=0x7f1604004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 shutdown_connections 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f15f006b110 0x7f15f006d5d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f160c15c7f0 0x7f160c15b340 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f160c0cf450 0x7f160c0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 --2- 172.21.15.107:0/2621052080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160c0ce6c0 0x7f160c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.414+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 >> 172.21.15.107:0/2621052080 conn(0x7f160c06a920 msgr2=0x7f160c15e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:39.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.415+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 shutdown_connections 2023-11-24T08:07:39.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.415+0000 7f1613e15640 1 -- 172.21.15.107:0/2621052080 wait complete. 2023-11-24T08:07:39.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:07:39.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:07:39.536 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:07:39.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.646+0000 7f7a3538e640 1 Processor -- start 2023-11-24T08:07:39.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.646+0000 7f7a3538e640 1 -- start start 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a3538e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a30072160 0x7f7a3015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a3538e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3015ec90 con 0x7f7a300c49f0 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a3538e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a300cd530 con 0x7f7a3006de10 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a3538e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a300c4ed0 con 0x7f7a30072160 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a267fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7a300c49f0 0x7f7a3006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32942/0 (socket says 172.21.15.107:32942) 2023-11-24T08:07:39.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.648+0000 7f7a267fc640 1 -- 172.21.15.107:0/3021064202 learned_addr learned my addr 172.21.15.107:0/3021064202 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:39.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3021064202 >> v2:172.21.15.120:3300/0 conn(0x7f7a30072160 0x7f7a3015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771718053 0 0) 0x7f7a18001110 con 0x7f7a300c49f0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1c005730 con 0x7f7a300c49f0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906979711 0 0) 0x7f7a1c005730 con 0x7f7a300c49f0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 >> v2:172.21.15.120:3300/0 conn(0x7f7a30072160 msgr2=0x7f7a3015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 --2- 172.21.15.107:0/3021064202 >> v2:172.21.15.120:3300/0 conn(0x7f7a30072160 0x7f7a3015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 >> v1:172.21.15.120:6789/0 conn(0x7f7a3006de10 legacy=0x7f7a300705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300c51b0 con 0x7f7a300c49f0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7a18001800 con 0x7f7a300c49f0 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 >> v1:172.21.15.107:6789/0 conn(0x7f7a300c49f0 legacy=0x7f7a3006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 0x7f7a1c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1c0069d0 0x7f7a1c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c007390 0x7f7a1c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300c4ed0 con 0x7f7a1c005bd0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c00e290 con 0x7f7a1c007390 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.649+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c00e450 con 0x7f7a1c0069d0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 0x7f7a1c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a267fc640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1c0069d0 0x7f7a1c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 -- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1c0069d0 msgr2=0x7f7a1c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1c0069d0 0x7f7a1c006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 -- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c007390 msgr2=0x7f7a1c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2f7fe640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c007390 0x7f7a1c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c007390 0x7f7a1c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2effd640 1 -- 172.21.15.107:0/3021064202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c005730 con 0x7f7a1c005bd0 2023-11-24T08:07:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2f7fe640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c007390 0x7f7a1c00db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:39.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.650+0000 7f7a2f7fe640 1 -- 172.21.15.107:0/3021064202 reap_dead start 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.651+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 0x7f7a1c005ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009ac0 tx=0x7f7a1c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74edbb3573b259b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.651+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2002c6b0 con 0x7f7a1c005bd0 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.651+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2002c810 con 0x7f7a1c005bd0 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.651+0000 7f7a2e7fc640 1 -- 172.21.15.107:0/3021064202 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2002cba0 con 0x7f7a1c005bd0 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 -- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 msgr2=0x7f7a1c005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 0x7f7a1c005ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009ac0 tx=0x7f7a1c007a00 comp rx=0 tx=0).stop 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 -- 172.21.15.107:0/3021064202 shutdown_connections 2023-11-24T08:07:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3021064202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a1c005bd0 0x7f7a1c005ff0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 -- 172.21.15.107:0/3021064202 >> 172.21.15.107:0/3021064202 conn(0x7f7a300b0c50 msgr2=0x7f7a300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 -- 172.21.15.107:0/3021064202 shutdown_connections 2023-11-24T08:07:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.652+0000 7f7a3538e640 1 -- 172.21.15.107:0/3021064202 wait complete. 2023-11-24T08:07:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.654+0000 7f7a3538e640 1 Processor -- start 2023-11-24T08:07:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.654+0000 7f7a3538e640 1 -- start start 2023-11-24T08:07:39.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c005bd0 0x7f7a30159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a3015aa90 0x7f7a3015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3015b480 con 0x7f7a3015c9c0 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3015ce40 con 0x7f7a1c005bd0 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a3538e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301f4130 con 0x7f7a3015aa90 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42294/0 (socket says 172.21.15.107:42294) 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.655+0000 7f7a267fc640 1 -- 172.21.15.107:0/3129474201 learned_addr learned my addr 172.21.15.107:0/3129474201 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c005bd0 0x7f7a30159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 -- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a3015aa90 msgr2=0x7f7a3015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a3015aa90 0x7f7a3015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 -- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c005bd0 msgr2=0x7f7a30159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c005bd0 0x7f7a30159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 -- 172.21.15.107:0/3129474201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a301f3660 con 0x7f7a3015c9c0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a267fc640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7a18001eb0 tx=0x7f7a1800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180074e0 con 0x7f7a3015c9c0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18007640 con 0x7f7a3015c9c0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180049e0 con 0x7f7a3015c9c0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a300cc880 con 0x7f7a3015c9c0 2023-11-24T08:07:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.656+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a301f3b70 con 0x7f7a3015c9c0 2023-11-24T08:07:39.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.658+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3004e8b0 con 0x7f7a3015c9c0 2023-11-24T08:07:39.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.659+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f7a18004b40 con 0x7f7a3015c9c0 2023-11-24T08:07:39.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.660+0000 7f7a27fff640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:39.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.660+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7a18089130 con 0x7f7a3015c9c0 2023-11-24T08:07:39.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.660+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:39.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.661+0000 7f7a2effd640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 0x7f7a1006d600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a20036000 tx=0x7f7a20030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:39.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.664+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a18053f20 con 0x7f7a3015c9c0 2023-11-24T08:07:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:39.855+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7a3015ec90 con 0x7f7a3015c9c0 2023-11-24T08:07:40.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.426+0000 7f7a27fff640 1 -- 172.21.15.107:0/3129474201 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7a3015ec90 con 0x7f7a3015c9c0 2023-11-24T08:07:40.429 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:07:40.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 msgr2=0x7f7a1006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 0x7f7a1006d600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a20036000 tx=0x7f7a20030000 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 msgr2=0x7f7a301f3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7a18001eb0 tx=0x7f7a1800dc00 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 shutdown_connections 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a3015aa90 0x7f7a3015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3015c9c0 0x7f7a301f3120 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 --2- 172.21.15.107:0/3129474201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1c005bd0 0x7f7a30159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 >> 172.21.15.107:0/3129474201 conn(0x7f7a300b0c50 msgr2=0x7f7a300cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.430+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 shutdown_connections 2023-11-24T08:07:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.431+0000 7f7a3538e640 1 -- 172.21.15.107:0/3129474201 wait complete. 2023-11-24T08:07:40.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:07:40.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.694+0000 7f34a90ec640 1 Processor -- start 2023-11-24T08:07:40.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.694+0000 7f34a90ec640 1 -- start start 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a90ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f34a415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a90ec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34a40c49f0 0x7f34a406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a90ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a415ec90 con 0x7f34a406dc40 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a90ec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a406e0f0 con 0x7f34a40c49f0 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a90ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40cd530 con 0x7f34a4072160 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f34a406dc40 0x7f34a40cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32954/0 (socket says 172.21.15.107:32954) 2023-11-24T08:07:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a359e640 1 -- 172.21.15.107:0/3239433417 learned_addr learned my addr 172.21.15.107:0/3239433417 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.696+0000 7f34a259c640 1 --2- 172.21.15.107:0/3239433417 >> v2:172.21.15.120:3300/0 conn(0x7f34a40c49f0 0x7f34a406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f34a415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090131494 0 0) 0x7f349c001490 con 0x7f34a406dc40 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348c003420 con 0x7f34a406dc40 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814705596 0 0) 0x7f348c003420 con 0x7f34a406dc40 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 msgr2=0x7f34a415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f34a415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 >> v2:172.21.15.120:3300/0 conn(0x7f34a40c49f0 msgr2=0x7f34a406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 --2- 172.21.15.107:0/3239433417 >> v2:172.21.15.120:3300/0 conn(0x7f34a40c49f0 0x7f34a406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40c4ed0 con 0x7f34a406dc40 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f349c001b80 con 0x7f34a406dc40 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 >> v1:172.21.15.107:6789/0 conn(0x7f34a406dc40 legacy=0x7f34a40cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f348c004a50 0x7f348c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f348c003660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c003ba0 0x7f348c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a406e0f0 con 0x7f348c003ba0 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40cd530 con 0x7f348c004a50 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.697+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c00fd00 con 0x7f34a4072160 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c003ba0 0x7f348c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a259c640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f348c003660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f348c004a50 0x7f348c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 -- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 msgr2=0x7f348c003660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34a4072160 0x7f348c003660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 -- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f348c004a50 msgr2=0x7f348c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f348c004a50 0x7f348c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a359e640 1 -- 172.21.15.107:0/3239433417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c003420 con 0x7f348c003ba0 2023-11-24T08:07:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.698+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f348c004a50 0x7f348c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.699+0000 7f34a359e640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c003ba0 0x7f348c004020 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f349c00a830 tx=0x7f348c0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3954aac2e604eb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.699+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034630 con 0x7f348c003ba0 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.699+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c02f070 con 0x7f348c003ba0 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.699+0000 7f34a1d9b640 1 -- 172.21.15.107:0/3239433417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0349f0 con 0x7f348c003ba0 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 -- 172.21.15.107:0/3239433417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c003ba0 msgr2=0x7f348c004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3239433417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c003ba0 0x7f348c004020 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f349c00a830 tx=0x7f348c0046e0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a359e640 1 -- 172.21.15.107:0/3239433417 reap_dead start 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 -- 172.21.15.107:0/3239433417 shutdown_connections 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 -- 172.21.15.107:0/3239433417 >> 172.21.15.107:0/3239433417 conn(0x7f34a40b0c50 msgr2=0x7f34a40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 -- 172.21.15.107:0/3239433417 shutdown_connections 2023-11-24T08:07:40.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.700+0000 7f34a90ec640 1 -- 172.21.15.107:0/3239433417 wait complete. 2023-11-24T08:07:40.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.702+0000 7f34a90ec640 1 Processor -- start 2023-11-24T08:07:40.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.702+0000 7f34a90ec640 1 -- start start 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f348c003ba0 0x7f34a40cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34a415c170 0x7f34a415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40b4490 con 0x7f34a40c80f0 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40b35e0 con 0x7f34a415c170 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a90ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40b3810 con 0x7f348c003ba0 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42314/0 (socket says 172.21.15.107:42314) 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.703+0000 7f34a359e640 1 -- 172.21.15.107:0/3204915708 learned_addr learned my addr 172.21.15.107:0/3204915708 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f348c003ba0 0x7f34a40cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a259c640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34a415c170 0x7f34a415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 -- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f348c003ba0 msgr2=0x7f34a40cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f348c003ba0 0x7f34a40cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 -- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34a415c170 msgr2=0x7f34a415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34a415c170 0x7f34a415ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 -- 172.21.15.107:0/3204915708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40b3af0 con 0x7f34a40c80f0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a359e640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f349c02cb80 tx=0x7f349c001f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034630 con 0x7f34a40c80f0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c034790 con 0x7f34a40c80f0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034b00 con 0x7f34a40c80f0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a40b3dd0 con 0x7f34a40c80f0 2023-11-24T08:07:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.704+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a40cc220 con 0x7f34a40c80f0 2023-11-24T08:07:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.706+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a40cc3e0 con 0x7f34a40c80f0 2023-11-24T08:07:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.707+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f349c031030 con 0x7f34a40c80f0 2023-11-24T08:07:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.707+0000 7f348b7fe640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 0x7f349006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.708+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f349c0ab790 con 0x7f34a40c80f0 2023-11-24T08:07:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.708+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 0x7f349006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.708+0000 7f34a2d9d640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 0x7f349006d5d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f34940099c0 tx=0x7f3494008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:40.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.713+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34a40cc3e0 con 0x7f34a40c80f0 2023-11-24T08:07:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:40.942+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f34a415ec90 con 0x7f34a40c80f0 2023-11-24T08:07:41.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.431+0000 7f348b7fe640 1 -- 172.21.15.107:0/3204915708 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 172+0+0 (secure 0 0 0) 0x7f34a415ec90 con 0x7f34a40c80f0 2023-11-24T08:07:41.433 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 6 and data pool 7 2023-11-24T08:07:41.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 msgr2=0x7f349006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 0x7f349006d5d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f34940099c0 tx=0x7f3494008040 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 msgr2=0x7f34a40c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f349c02cb80 tx=0x7f349c001f10 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 shutdown_connections 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f349006b110 0x7f349006d5d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40c80f0 0x7f34a40c85a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34a415c170 0x7f34a415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 --2- 172.21.15.107:0/3204915708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f348c003ba0 0x7f34a40cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.435+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 >> 172.21.15.107:0/3204915708 conn(0x7f34a40b0c50 msgr2=0x7f34a40cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:41.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.437+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 shutdown_connections 2023-11-24T08:07:41.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.437+0000 7f34a90ec640 1 -- 172.21.15.107:0/3204915708 wait complete. 2023-11-24T08:07:41.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.656+0000 7fdae64d4640 1 Processor -- start 2023-11-24T08:07:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.656+0000 7fdae64d4640 1 -- start start 2023-11-24T08:07:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.657+0000 7fdae64d4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdae00ce6c0 0x7fdae00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.657+0000 7fdae64d4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00cece0 con 0x7fdae00cf450 2023-11-24T08:07:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.657+0000 7fdae64d4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae006d040 con 0x7fdae015c7f0 2023-11-24T08:07:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.657+0000 7fdae64d4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae00cfe40 con 0x7fdae00ce6c0 2023-11-24T08:07:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdad7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdae00cf450 0x7fdae00cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32968/0 (socket says 172.21.15.107:32968) 2023-11-24T08:07:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdad7fff640 1 -- 172.21.15.107:0/742330722 learned_addr learned my addr 172.21.15.107:0/742330722 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024786358 0 0) 0x7fdae00cece0 con 0x7fdae00cf450 2023-11-24T08:07:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc003520 con 0x7fdae00cf450 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdae4a71640 1 --2- 172.21.15.107:0/742330722 >> v2:172.21.15.120:3300/0 conn(0x7fdae00ce6c0 0x7fdae00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845473575 0 0) 0x7fdacc003520 con 0x7fdae00cf450 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 >> v2:172.21.15.120:3300/0 conn(0x7fdae00ce6c0 msgr2=0x7fdae00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 --2- 172.21.15.107:0/742330722 >> v2:172.21.15.120:3300/0 conn(0x7fdae00ce6c0 0x7fdae00cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.658+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 >> v1:172.21.15.120:6789/0 conn(0x7fdae015c7f0 legacy=0x7fdae006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae006d320 con 0x7fdae00cf450 2023-11-24T08:07:41.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdac8001670 con 0x7fdae00cf450 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 >> v1:172.21.15.107:6789/0 conn(0x7fdae00cf450 legacy=0x7fdae00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 0x7fdacc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdacc0047c0 0x7fdacc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc005180 0x7fdacc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00cfe40 con 0x7fdacc003970 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00c0b0 con 0x7fdacc0047c0 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00c270 con 0x7fdacc005180 2023-11-24T08:07:41.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 0x7fdacc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 -- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc005180 msgr2=0x7fdacc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc005180 0x7fdacc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 -- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdacc0047c0 msgr2=0x7fdacc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdacc0047c0 0x7fdacc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdadffff640 1 -- 172.21.15.107:0/742330722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc003520 con 0x7fdacc003970 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.659+0000 7fdae4a71640 1 -- 172.21.15.107:0/742330722 reap_dead start 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.660+0000 7fdadffff640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 0x7fdacc003d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdad0003d80 tx=0x7fdacc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60e3be9a8c97bb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0032600 con 0x7fdacc003970 2023-11-24T08:07:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad002e050 con 0x7fdacc003970 2023-11-24T08:07:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdadf7fe640 1 -- 172.21.15.107:0/742330722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad00329e0 con 0x7fdacc003970 2023-11-24T08:07:41.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 -- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 msgr2=0x7fdacc003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 0x7fdacc003d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdad0003d80 tx=0x7fdacc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 -- 172.21.15.107:0/742330722 shutdown_connections 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 --2- 172.21.15.107:0/742330722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc003970 0x7fdacc003d70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 -- 172.21.15.107:0/742330722 >> 172.21.15.107:0/742330722 conn(0x7fdae006a920 msgr2=0x7fdae006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 -- 172.21.15.107:0/742330722 shutdown_connections 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.661+0000 7fdae64d4640 1 -- 172.21.15.107:0/742330722 wait complete. 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.663+0000 7fdae64d4640 1 Processor -- start 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.663+0000 7fdae64d4640 1 -- start start 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc003970 0x7fdae01474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae00c9de0 0x7fdae00ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae0147a20 con 0x7fdae00ad0a0 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00ca780 con 0x7fdae00c9de0 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae64d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00c8500 con 0x7fdacc003970 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae4a71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae4a71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42334/0 (socket says 172.21.15.107:42334) 2023-11-24T08:07:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.664+0000 7fdae4a71640 1 -- 172.21.15.107:0/3375057764 learned_addr learned my addr 172.21.15.107:0/3375057764 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 -- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc003970 msgr2=0x7fdae01474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc003970 0x7fdae01474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 -- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae00c9de0 msgr2=0x7fdae00ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae00c9de0 0x7fdae00ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 -- 172.21.15.107:0/3375057764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00c87a0 con 0x7fdae00ad0a0 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae4a71640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdae00ccd80 tx=0x7fdad800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad800ed90 con 0x7fdae00ad0a0 2023-11-24T08:07:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad8014420 con 0x7fdae00ad0a0 2023-11-24T08:07:41.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad80147d0 con 0x7fdae00ad0a0 2023-11-24T08:07:41.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.665+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae00c8a80 con 0x7fdae00ad0a0 2023-11-24T08:07:41.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.666+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae00c9040 con 0x7fdae00ad0a0 2023-11-24T08:07:41.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.666+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae0067470 con 0x7fdae00ad0a0 2023-11-24T08:07:41.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.667+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fdad8014930 con 0x7fdae00ad0a0 2023-11-24T08:07:41.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.669+0000 7fdadd7fa640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 0x7fdac006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:41.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.669+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdad8089e90 con 0x7fdae00ad0a0 2023-11-24T08:07:41.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.669+0000 7fdadffff640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 0x7fdac006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:41.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.670+0000 7fdadffff640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 0x7fdac006d600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdad0034000 tx=0x7fdad002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:41.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.673+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdad8054b20 con 0x7fdae00ad0a0 2023-11-24T08:07:41.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.853+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdae0070af0 con 0x7fdae00ad0a0 2023-11-24T08:07:41.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.856+0000 7fdadd7fa640 1 -- 172.21.15.107:0/3375057764 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16526 (secure 0 0 0) 0x7fdad808b090 con 0x7fdae00ad0a0 2023-11-24T08:07:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:41.859 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":47,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:40.944779+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 msgr2=0x7fdac006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 0x7fdac006d600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdad0034000 tx=0x7fdad002f000 comp rx=0 tx=0).stop 2023-11-24T08:07:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 msgr2=0x7fdae00ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdae00ccd80 tx=0x7fdad800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 shutdown_connections 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdac006b140 0x7fdac006d600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdae00ad0a0 0x7fdae00ad520 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae00c9de0 0x7fdae00ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 --2- 172.21.15.107:0/3375057764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdacc003970 0x7fdae01474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 >> 172.21.15.107:0/3375057764 conn(0x7fdae006a920 msgr2=0x7fdae00aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 shutdown_connections 2023-11-24T08:07:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:41.860+0000 7fdae64d4640 1 -- 172.21.15.107:0/3375057764 wait complete. 2023-11-24T08:07:41.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:42.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.073+0000 7fb7ec83d640 1 Processor -- start 2023-11-24T08:07:42.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.073+0000 7fb7ec83d640 1 -- start start 2023-11-24T08:07:42.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.074+0000 7fb7ec83d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7e815c7f0 0x7fb7e806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7ec83d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7e80cf450 0x7fb7e80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7ec83d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 0x7fb7e80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7ec83d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e80cfe40 con 0x7fb7e80cf450 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7ec83d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e8156010 con 0x7fb7e815c7f0 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7ec83d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e806ce30 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7e80cf450 0x7fb7e80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7e80cf450 0x7fb7e80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42342/0 (socket says 172.21.15.107:42342) 2023-11-24T08:07:42.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 -- 172.21.15.107:0/3581483746 learned_addr learned my addr 172.21.15.107:0/3581483746 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e659c640 1 --2- 172.21.15.107:0/3581483746 >> v2:172.21.15.120:3300/0 conn(0x7fb7e815c7f0 0x7fb7e806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 -- 172.21.15.107:0/3581483746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 msgr2=0x7fb7e80ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e6d9d640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 0x7fb7e80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 0x7fb7e80ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 -- 172.21.15.107:0/3581483746 >> v2:172.21.15.120:3300/0 conn(0x7fb7e815c7f0 msgr2=0x7fb7e806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.075+0000 7fb7e5d9b640 1 --2- 172.21.15.107:0/3581483746 >> v2:172.21.15.120:3300/0 conn(0x7fb7e815c7f0 0x7fb7e806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.076+0000 7fb7e5d9b640 1 -- 172.21.15.107:0/3581483746 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80cece0 con 0x7fb7e80cf450 2023-11-24T08:07:42.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.076+0000 7fb7e6d9d640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 0x7fb7e80ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7e5d9b640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cf450 0x7fb7e80cf900 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80d0130 tx=0x7fb7dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b81a8e08d212c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7e559a640 1 -- 172.21.15.107:0/3581483746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc031870 con 0x7fb7e80cf450 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7e559a640 1 -- 172.21.15.107:0/3581483746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0319d0 con 0x7fb7e80cf450 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7e559a640 1 -- 172.21.15.107:0/3581483746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc031d60 con 0x7fb7e80cf450 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7ec83d640 1 -- 172.21.15.107:0/3581483746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cf450 msgr2=0x7fb7e80cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.077+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cf450 0x7fb7e80cf900 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80d0130 tx=0x7fb7dc02b910 comp rx=0 tx=0).stop 2023-11-24T08:07:42.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 -- 172.21.15.107:0/3581483746 shutdown_connections 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e80cdfb0 0x7fb7e80ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/3581483746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cf450 0x7fb7e80cf900 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/3581483746 >> v2:172.21.15.120:3300/0 conn(0x7fb7e815c7f0 0x7fb7e806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 -- 172.21.15.107:0/3581483746 >> 172.21.15.107:0/3581483746 conn(0x7fb7e806a920 msgr2=0x7fb7e806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 -- 172.21.15.107:0/3581483746 shutdown_connections 2023-11-24T08:07:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.078+0000 7fb7ec83d640 1 -- 172.21.15.107:0/3581483746 wait complete. 2023-11-24T08:07:42.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.079+0000 7fb7ec83d640 1 Processor -- start 2023-11-24T08:07:42.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.079+0000 7fb7ec83d640 1 -- start start 2023-11-24T08:07:42.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e80cf450 0x7fb7e809ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e815c7f0 0x7fb7e80b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b3af0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b2c40 con 0x7fb7e80cf450 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.080+0000 7fb7ec83d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b2ee0 con 0x7fb7e815c7f0 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42346/0 (socket says 172.21.15.107:42346) 2023-11-24T08:07:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 -- 172.21.15.107:0/1595407256 learned_addr learned my addr 172.21.15.107:0/1595407256 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 -- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e815c7f0 msgr2=0x7fb7e80b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e815c7f0 0x7fb7e80b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 -- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e80cf450 msgr2=0x7fb7e809ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e80cf450 0x7fb7e809ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 -- 172.21.15.107:0/1595407256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0094d0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7e659c640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800d5f0 tx=0x7fb7d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.081+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80125f0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.082+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8012750 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.082+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8012ae0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.082+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e80b31c0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.082+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e80b3780 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.083+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e815be20 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.084+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fb7d8012c40 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.085+0000 7fb7d6ffd640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 0x7fb7c806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.085+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7d8089fe0 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.085+0000 7fb7e5d9b640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 0x7fb7c806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.086+0000 7fb7e5d9b640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 0x7fb7c806d600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc036000 tx=0x7fb7dc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.089+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7d8054c70 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.267+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7e8067470 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.270+0000 7fb7d6ffd640 1 -- 172.21.15.107:0/1595407256 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16526 (secure 0 0 0) 0x7fb7d8004640 con 0x7fb7e80cdfb0 2023-11-24T08:07:42.274 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:42.274 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":47,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:40.944779+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 msgr2=0x7fb7c806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 0x7fb7c806d600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc036000 tx=0x7fb7dc030000 comp rx=0 tx=0).stop 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 msgr2=0x7fb7e80abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800d5f0 tx=0x7fb7d800dab0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 shutdown_connections 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7c806b140 0x7fb7c806d600 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7e815c7f0 0x7fb7e80b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e80cf450 0x7fb7e809ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 --2- 172.21.15.107:0/1595407256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7e80cdfb0 0x7fb7e80abfc0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.272+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 >> 172.21.15.107:0/1595407256 conn(0x7fb7e806a920 msgr2=0x7fb7e806ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.273+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 shutdown_connections 2023-11-24T08:07:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.273+0000 7fb7ec83d640 1 -- 172.21.15.107:0/1595407256 wait complete. 2023-11-24T08:07:42.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:07:42.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.503+0000 7f368c4c6640 1 Processor -- start 2023-11-24T08:07:42.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.503+0000 7f368c4c6640 1 -- start start 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f368c4c6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f3684166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f368c4c6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368415eea0 con 0x7f36840cc0f0 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f368c4c6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368406d600 con 0x7f368415b440 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f368c4c6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36840c5e00 con 0x7f36840c59b0 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f3689a61640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f36840cc0f0 0x7f36840c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32984/0 (socket says 172.21.15.107:32984) 2023-11-24T08:07:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.505+0000 7f3689a61640 1 -- 172.21.15.107:0/3728714905 learned_addr learned my addr 172.21.15.107:0/3728714905 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115517590 0 0) 0x7f368415eea0 con 0x7f36840cc0f0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f368aa63640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f3684166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3670003520 con 0x7f36840cc0f0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324409112 0 0) 0x7f3670003520 con 0x7f36840cc0f0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 msgr2=0x7f3684166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f3684166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 >> v1:172.21.15.120:6789/0 conn(0x7f368415b440 legacy=0x7f36840cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840abf30 con 0x7f36840cc0f0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3674001670 con 0x7f36840cc0f0 2023-11-24T08:07:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 >> v1:172.21.15.107:6789/0 conn(0x7f36840cc0f0 legacy=0x7f36840c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f3670003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f36700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670004d00 0x7f3670005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367000bb30 con 0x7f36700039a0 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367000bca0 con 0x7f3670004d00 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367000be60 con 0x7f36840c59b0 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.506+0000 7f368a262640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f3670003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f3689a61640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f36700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f368a262640 1 -- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 msgr2=0x7f36700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f368a262640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840c59b0 0x7f36700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f368a262640 1 -- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670004d00 msgr2=0x7f3670005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:42.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f368a262640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670004d00 0x7f3670005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.507+0000 7f368a262640 1 -- 172.21.15.107:0/3728714905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670003520 con 0x7f36700039a0 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.508+0000 7f368a262640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f3670003dc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3680001750 tx=0x7f36700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9707837980759415 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.508+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800052d0 con 0x7f36700039a0 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.508+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368002e050 con 0x7f36700039a0 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.508+0000 7f3689260640 1 -- 172.21.15.107:0/3728714905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800035f0 con 0x7f36700039a0 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 -- 172.21.15.107:0/3728714905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 msgr2=0x7f3670003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3728714905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f3670003dc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3680001750 tx=0x7f36700057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368aa63640 1 -- 172.21.15.107:0/3728714905 reap_dead start 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 -- 172.21.15.107:0/3728714905 shutdown_connections 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 -- 172.21.15.107:0/3728714905 >> 172.21.15.107:0/3728714905 conn(0x7f36840b0c50 msgr2=0x7f36840b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 -- 172.21.15.107:0/3728714905 shutdown_connections 2023-11-24T08:07:42.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.509+0000 7f368c4c6640 1 -- 172.21.15.107:0/3728714905 wait complete. 2023-11-24T08:07:42.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.511+0000 7f368c4c6640 1 Processor -- start 2023-11-24T08:07:42.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.511+0000 7f368c4c6640 1 -- start start 2023-11-24T08:07:42.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f368415d7d0 0x7f368415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36841508b0 0x7f3684150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3684151280 con 0x7f36700039a0 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36840c6160 con 0x7f368415d7d0 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368c4c6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36840c6420 con 0x7f36841508b0 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42368/0 (socket says 172.21.15.107:42368) 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 -- 172.21.15.107:0/3524497739 learned_addr learned my addr 172.21.15.107:0/3524497739 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f3689a61640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f368415d7d0 0x7f368415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 -- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36841508b0 msgr2=0x7f3684150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36841508b0 0x7f3684150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 -- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f368415d7d0 msgr2=0x7f368415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f368415d7d0 0x7f368415dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.512+0000 7f368a262640 1 -- 172.21.15.107:0/3524497739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840c6700 con 0x7f36700039a0 2023-11-24T08:07:42.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f368a262640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3680008fd0 tx=0x7f3680001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800076c0 con 0x7f36700039a0 2023-11-24T08:07:42.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3680007a30 con 0x7f36700039a0 2023-11-24T08:07:42.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680006be0 con 0x7f36700039a0 2023-11-24T08:07:42.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36840c69e0 con 0x7f36700039a0 2023-11-24T08:07:42.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.513+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368415e960 con 0x7f36700039a0 2023-11-24T08:07:42.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.514+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368404e8b0 con 0x7f36700039a0 2023-11-24T08:07:42.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.515+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f3680006d40 con 0x7f36700039a0 2023-11-24T08:07:42.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.516+0000 7f367affd640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 0x7f366806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:42.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.516+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36800abce0 con 0x7f36700039a0 2023-11-24T08:07:42.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.516+0000 7f3689a61640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 0x7f366806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:42.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.517+0000 7f3689a61640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 0x7f366806d600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3674001eb0 tx=0x7f3674001d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:42.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.520+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3680076860 con 0x7f36700039a0 2023-11-24T08:07:42.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:42.751+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36840c6560 con 0x7f36700039a0 2023-11-24T08:07:43.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.471+0000 7f367affd640 1 -- 172.21.15.107:0/3524497739 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7f36840c6560 con 0x7f36700039a0 2023-11-24T08:07:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 msgr2=0x7f366806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 0x7f366806d600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3674001eb0 tx=0x7f3674001d40 comp rx=0 tx=0).stop 2023-11-24T08:07:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 msgr2=0x7f36841593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3680008fd0 tx=0x7f3680001120 comp rx=0 tx=0).stop 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 shutdown_connections 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f366806b140 0x7f366806d600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36841508b0 0x7f3684150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f368415d7d0 0x7f368415dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 --2- 172.21.15.107:0/3524497739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700039a0 0x7f36841593d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 >> 172.21.15.107:0/3524497739 conn(0x7f36840b0c50 msgr2=0x7f36840719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:43.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 shutdown_connections 2023-11-24T08:07:43.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.475+0000 7f368c4c6640 1 -- 172.21.15.107:0/3524497739 wait complete. 2023-11-24T08:07:43.497 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:07:43.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:07:43.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.713+0000 7fa70f49b640 1 Processor -- start 2023-11-24T08:07:43.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.713+0000 7fa70f49b640 1 -- start start 2023-11-24T08:07:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70f49b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa708146eb0 0x7fa708161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70f49b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa70806e160 0x7fa7080ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70f49b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa708161a10 con 0x7fa7080cd110 2023-11-24T08:07:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70f49b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7080cd580 con 0x7fa708146eb0 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70f49b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa708161cf0 con 0x7fa70806e160 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70d237640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa708146eb0 0x7fa708161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70d237640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa708146eb0 0x7fa708161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42370/0 (socket says 172.21.15.107:42370) 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.715+0000 7fa70d237640 1 -- 172.21.15.107:0/2060233388 learned_addr learned my addr 172.21.15.107:0/2060233388 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.716+0000 7fa70ca36640 1 --2- 172.21.15.107:0/2060233388 >> v2:172.21.15.120:3300/0 conn(0x7fa70806e160 0x7fa7080ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.716+0000 7fa70d237640 1 -- 172.21.15.107:0/2060233388 >> v2:172.21.15.120:3300/0 conn(0x7fa70806e160 msgr2=0x7fa7080ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.716+0000 7fa70d237640 1 --2- 172.21.15.107:0/2060233388 >> v2:172.21.15.120:3300/0 conn(0x7fa70806e160 0x7fa7080ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.716+0000 7fa70d237640 1 -- 172.21.15.107:0/2060233388 >> v1:172.21.15.120:6789/0 conn(0x7fa7080cd110 legacy=0x7fa70806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.716+0000 7fa70d237640 1 -- 172.21.15.107:0/2060233388 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708161fd0 con 0x7fa708146eb0 2023-11-24T08:07:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.717+0000 7fa70d237640 1 --2- 172.21.15.107:0/2060233388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa708161150 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa70806eab0 tx=0x7fa70402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a48b2222c65664 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa6fffff640 1 -- 172.21.15.107:0/2060233388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704031870 con 0x7fa708146eb0 2023-11-24T08:07:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa6fffff640 1 -- 172.21.15.107:0/2060233388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7040319d0 con 0x7fa708146eb0 2023-11-24T08:07:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa6fffff640 1 -- 172.21.15.107:0/2060233388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704031d60 con 0x7fa708146eb0 2023-11-24T08:07:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 -- 172.21.15.107:0/2060233388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 msgr2=0x7fa708161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2060233388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa708161150 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa70806eab0 tx=0x7fa70402b910 comp rx=0 tx=0).stop 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 -- 172.21.15.107:0/2060233388 shutdown_connections 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2060233388 >> v2:172.21.15.120:3300/0 conn(0x7fa70806e160 0x7fa7080ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2060233388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa708161150 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 -- 172.21.15.107:0/2060233388 >> 172.21.15.107:0/2060233388 conn(0x7fa7080b0c50 msgr2=0x7fa7080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 -- 172.21.15.107:0/2060233388 shutdown_connections 2023-11-24T08:07:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.718+0000 7fa70f49b640 1 -- 172.21.15.107:0/2060233388 wait complete. 2023-11-24T08:07:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.720+0000 7fa70f49b640 1 Processor -- start 2023-11-24T08:07:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.720+0000 7fa70f49b640 1 -- start start 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa70806e160 0x7fa70809b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7080cd110 0x7fa70809a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70809aab0 con 0x7fa708146eb0 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7080b1d10 con 0x7fa7080cd110 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70f49b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70809ba20 con 0x7fa70806e160 2023-11-24T08:07:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.721+0000 7fa70da38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42384/0 (socket says 172.21.15.107:42384) 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 -- 172.21.15.107:0/2488242686 learned_addr learned my addr 172.21.15.107:0/2488242686 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70ca36640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7080cd110 0x7fa70809a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70d237640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa70806e160 0x7fa70809b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 -- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa70806e160 msgr2=0x7fa70809b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa70806e160 0x7fa70809b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 -- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7080cd110 msgr2=0x7fa70809a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7080cd110 0x7fa70809a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 -- 172.21.15.107:0/2488242686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040094d0 con 0x7fa708146eb0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa70da38640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c780 tx=0x7fa70000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.722+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700004b60 con 0x7fa708146eb0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.723+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70809bd00 con 0x7fa708146eb0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.723+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70000ce90 con 0x7fa708146eb0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.723+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70000a850 con 0x7fa708146eb0 2023-11-24T08:07:43.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.723+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70809c2c0 con 0x7fa708146eb0 2023-11-24T08:07:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.724+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7080b1ec0 con 0x7fa708146eb0 2023-11-24T08:07:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.725+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fa70000a9b0 con 0x7fa708146eb0 2023-11-24T08:07:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.726+0000 7fa6fdffb640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 0x7fa6ec06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.726+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa700089f90 con 0x7fa708146eb0 2023-11-24T08:07:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.727+0000 7fa70d237640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 0x7fa6ec06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.727+0000 7fa70d237640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 0x7fa6ec06d5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa70403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.731+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7080b1ec0 con 0x7fa708146eb0 2023-11-24T08:07:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:43.958+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7080b1e80 con 0x7fa708146eb0 2023-11-24T08:07:44.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.475+0000 7fa6fdffb640 1 -- 172.21.15.107:0/2488242686 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa700054be0 con 0x7fa708146eb0 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 msgr2=0x7fa6ec06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 0x7fa6ec06d5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa70403c040 comp rx=0 tx=0).stop 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 msgr2=0x7fa70815cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c780 tx=0x7fa70000cc40 comp rx=0 tx=0).stop 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 shutdown_connections 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6ec06b110 0x7fa6ec06d5d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa708146eb0 0x7fa70815cf70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7080cd110 0x7fa70809a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 --2- 172.21.15.107:0/2488242686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa70806e160 0x7fa70809b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 >> 172.21.15.107:0/2488242686 conn(0x7fa7080b0c50 msgr2=0x7fa7080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:44.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 shutdown_connections 2023-11-24T08:07:44.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.478+0000 7fa70f49b640 1 -- 172.21.15.107:0/2488242686 wait complete. 2023-11-24T08:07:44.502 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:07:44.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:44.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.713+0000 7f6943fff640 1 Processor -- start 2023-11-24T08:07:44.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.713+0000 7f6943fff640 1 -- start start 2023-11-24T08:07:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f6943fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 0x7f694415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f6943fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694415bf80 con 0x7f69440b1de0 2023-11-24T08:07:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f6943fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440b1560 con 0x7f6944144b30 2023-11-24T08:07:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f6943fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69440b1280 con 0x7f694415c410 2023-11-24T08:07:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f69437fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69440b1de0 0x7f69440b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32994/0 (socket says 172.21.15.107:32994) 2023-11-24T08:07:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f69437fe640 1 -- 172.21.15.107:0/2222592905 learned_addr learned my addr 172.21.15.107:0/2222592905 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f693a7fc640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 0x7f694415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275097250 0 0) 0x7f694415bf80 con 0x7f69440b1de0 2023-11-24T08:07:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.715+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6930003520 con 0x7f69440b1de0 2023-11-24T08:07:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930461411 0 0) 0x7f6930003520 con 0x7f69440b1de0 2023-11-24T08:07:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 msgr2=0x7f694415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 0x7f694415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 >> v1:172.21.15.120:6789/0 conn(0x7f6944144b30 legacy=0x7f69440b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694415b930 con 0x7f69440b1de0 2023-11-24T08:07:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f693c0019f0 con 0x7f69440b1de0 2023-11-24T08:07:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 >> v1:172.21.15.107:6789/0 conn(0x7f69440b1de0 legacy=0x7f69440b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 0x7f69300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6930004d00 0x7f6930005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440b1280 con 0x7f69300039a0 2023-11-24T08:07:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693000ba30 con 0x7f6930004d00 2023-11-24T08:07:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693000bbf0 con 0x7f694415c410 2023-11-24T08:07:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.716+0000 7f6942ffd640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.717+0000 7f6942ffd640 1 -- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 msgr2=0x7f69300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.717+0000 7f6942ffd640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f694415c410 0x7f69300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.717+0000 7f6942ffd640 1 -- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6930004d00 msgr2=0x7f6930005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.717+0000 7f6942ffd640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6930004d00 0x7f6930005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.717+0000 7f6942ffd640 1 -- 172.21.15.107:0/2222592905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930003520 con 0x7f69300039a0 2023-11-24T08:07:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f6942ffd640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6934001720 tx=0x7f69300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218aa9aba6ea3f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69340052d0 con 0x7f69300039a0 2023-11-24T08:07:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693402e050 con 0x7f69300039a0 2023-11-24T08:07:44.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f69427fc640 1 -- 172.21.15.107:0/2222592905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934003830 con 0x7f69300039a0 2023-11-24T08:07:44.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f6943fff640 1 -- 172.21.15.107:0/2222592905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69300039a0 msgr2=0x7f6930003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f6943fff640 1 --2- 172.21.15.107:0/2222592905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6934001720 tx=0x7f69300057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.718+0000 7f69437fe640 1 -- 172.21.15.107:0/2222592905 reap_dead start 2023-11-24T08:07:44.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.719+0000 7f6943fff640 1 -- 172.21.15.107:0/2222592905 shutdown_connections 2023-11-24T08:07:44.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.719+0000 7f6943fff640 1 -- 172.21.15.107:0/2222592905 >> 172.21.15.107:0/2222592905 conn(0x7f69440cb830 msgr2=0x7f69440cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:44.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.719+0000 7f6943fff640 1 -- 172.21.15.107:0/2222592905 shutdown_connections 2023-11-24T08:07:44.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.719+0000 7f6943fff640 1 -- 172.21.15.107:0/2222592905 wait complete. 2023-11-24T08:07:44.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.720+0000 7f6943fff640 1 Processor -- start 2023-11-24T08:07:44.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.721+0000 7f6943fff640 1 -- start start 2023-11-24T08:07:44.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.721+0000 7f6943fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69300039a0 0x7f694415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.721+0000 7f6943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.721+0000 7f6943fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6944070d20 0x7f694415cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f6943fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694415afe0 con 0x7f69441e5d10 2023-11-24T08:07:44.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f6943fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441e61f0 con 0x7f69300039a0 2023-11-24T08:07:44.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f6943fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69441e64d0 con 0x7f6944070d20 2023-11-24T08:07:44.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42396/0 (socket says 172.21.15.107:42396) 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 -- 172.21.15.107:0/1064144775 learned_addr learned my addr 172.21.15.107:0/1064144775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f69437fe640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6944070d20 0x7f694415cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 -- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6944070d20 msgr2=0x7f694415cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6944070d20 0x7f694415cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 -- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69300039a0 msgr2=0x7f694415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f6942ffd640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69300039a0 0x7f694415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69300039a0 0x7f694415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.722+0000 7f693a7fc640 1 -- 172.21.15.107:0/1064144775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69441e67b0 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f693a7fc640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f692c00e7c0 tx=0x7f692c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c00eed0 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c00c940 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69441e6a90 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c00ccd0 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.723+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69441e77d0 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.724+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694404e840 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.725+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f692c013470 con 0x7f69441e5d10 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.726+0000 7f693bfff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 0x7f692406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:44.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.726+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f692c089b40 con 0x7f69441e5d10 2023-11-24T08:07:44.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.726+0000 7f6942ffd640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 0x7f692406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:44.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.727+0000 7f6942ffd640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 0x7f692406d5d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6934033000 tx=0x7f6934009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:44.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.731+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f692c054860 con 0x7f69441e5d10 2023-11-24T08:07:44.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.960+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f694406d860 con 0x7f69441e5d10 2023-11-24T08:07:44.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.961+0000 7f693bfff640 1 -- 172.21.15.107:0/1064144775 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f692c021090 con 0x7f69441e5d10 2023-11-24T08:07:44.963 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:44.964 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:07:44.964 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-11-24T08:07:44.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 msgr2=0x7f692406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 0x7f692406d5d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6934033000 tx=0x7f6934009030 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 msgr2=0x7f69441e7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f692c00e7c0 tx=0x7f692c00ec80 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 shutdown_connections 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f692406b110 0x7f692406d5d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6944070d20 0x7f694415cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69441e5d10 0x7f69441e7290 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 --2- 172.21.15.107:0/1064144775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69300039a0 0x7f694415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 >> 172.21.15.107:0/1064144775 conn(0x7f69440cb830 msgr2=0x7f69440cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 shutdown_connections 2023-11-24T08:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:44.964+0000 7f6943fff640 1 -- 172.21.15.107:0/1064144775 wait complete. 2023-11-24T08:07:44.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:07:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 Processor -- start 2023-11-24T08:07:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 -- start start 2023-11-24T08:07:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8de0144b30 0x7f8de015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8de015c410 0x7f8de00b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de00b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00b27b0 con 0x7f8de0144b30 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de015bf80 con 0x7f8de015c410 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.195+0000 7f8ddf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00b1560 con 0x7f8de00b1de0 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8de0144b30 0x7f8de015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8de0144b30 0x7f8de015f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42410/0 (socket says 172.21.15.107:42410) 2023-11-24T08:07:45.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 -- 172.21.15.107:0/2534782778 learned_addr learned my addr 172.21.15.107:0/2534782778 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dded9d640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de00b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dddd9b640 1 --2- 172.21.15.107:0/2534782778 >> v2:172.21.15.120:3300/0 conn(0x7f8de015c410 0x7f8de00b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 -- 172.21.15.107:0/2534782778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 msgr2=0x7f8de00b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de00b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 -- 172.21.15.107:0/2534782778 >> v2:172.21.15.120:3300/0 conn(0x7f8de015c410 msgr2=0x7f8de00b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2534782778 >> v2:172.21.15.120:3300/0 conn(0x7f8de015c410 0x7f8de00b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.196+0000 7f8dde59c640 1 -- 172.21.15.107:0/2534782778 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00b0fb0 con 0x7f8de0144b30 2023-11-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.197+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de0144b30 0x7f8de015f920 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8de00cc7a0 tx=0x7f8dd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74f963136df8808 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddd59a640 1 -- 172.21.15.107:0/2534782778 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd002ca70 con 0x7f8de0144b30 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddd59a640 1 -- 172.21.15.107:0/2534782778 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd002cbd0 con 0x7f8de0144b30 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddd59a640 1 -- 172.21.15.107:0/2534782778 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00343f0 con 0x7f8de0144b30 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2534782778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de0144b30 msgr2=0x7f8de015f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de0144b30 0x7f8de015f920 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8de00cc7a0 tx=0x7f8dd002bed0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2534782778 shutdown_connections 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de00b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2534782778 >> v2:172.21.15.120:3300/0 conn(0x7f8de015c410 0x7f8de00b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2534782778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de0144b30 0x7f8de015f920 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2534782778 >> 172.21.15.107:0/2534782778 conn(0x7f8de00cb830 msgr2=0x7f8de00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.198+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2534782778 shutdown_connections 2023-11-24T08:07:45.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.199+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2534782778 wait complete. 2023-11-24T08:07:45.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.200+0000 7f8ddf59e640 1 Processor -- start 2023-11-24T08:07:45.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.200+0000 7f8ddf59e640 1 -- start start 2023-11-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8de0144b30 0x7f8de0134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0134e90 con 0x7f8de015c410 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de015af10 con 0x7f8de0144b30 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.201+0000 7f8ddf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de015a300 con 0x7f8de00b1de0 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42412/0 (socket says 172.21.15.107:42412) 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 -- 172.21.15.107:0/2477982966 learned_addr learned my addr 172.21.15.107:0/2477982966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 -- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 msgr2=0x7f8de0159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dddd9b640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8de0144b30 0x7f8de0134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de0159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 -- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8de0144b30 msgr2=0x7f8de0134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8de0144b30 0x7f8de0134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 -- 172.21.15.107:0/2477982966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0009b30 con 0x7f8de015c410 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.202+0000 7f8dded9d640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800c570 tx=0x7f8dd800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.203+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8004220 con 0x7f8de015c410 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.203+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd8004380 con 0x7f8de015c410 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.203+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8003830 con 0x7f8de015c410 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.203+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de015a5e0 con 0x7f8de015c410 2023-11-24T08:07:45.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.203+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de015aba0 con 0x7f8de015c410 2023-11-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.204+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de015b0a0 con 0x7f8de015c410 2023-11-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.205+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f8dd8003990 con 0x7f8de015c410 2023-11-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.206+0000 7f8dd6ffd640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 0x7f8dc006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.206+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dd80898a0 con 0x7f8de015c410 2023-11-24T08:07:45.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.206+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 0x7f8dc006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.207+0000 7f8dde59c640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 0x7f8dc006d600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0035000 tx=0x7f8dd0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.211+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dd8054420 con 0x7f8de015c410 2023-11-24T08:07:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.401+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8de004e840 con 0x7f8de015c410 2023-11-24T08:07:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.403+0000 7f8dd6ffd640 1 -- 172.21.15.107:0/2477982966 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16513 (secure 0 0 0) 0x7f8dd8057a40 con 0x7f8de015c410 2023-11-24T08:07:45.405 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:45.405 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":48,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:07:42.447849+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:07:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 msgr2=0x7f8dc006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 0x7f8dc006d600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0035000 tx=0x7f8dd0030000 comp rx=0 tx=0).stop 2023-11-24T08:07:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 msgr2=0x7f8de00b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800c570 tx=0x7f8dd800ca30 comp rx=0 tx=0).stop 2023-11-24T08:07:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 shutdown_connections 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8dc006b140 0x7f8dc006d600 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8de015c410 0x7f8de00b3ea0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8de0144b30 0x7f8de0134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 --2- 172.21.15.107:0/2477982966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8de00b1de0 0x7f8de0159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 >> 172.21.15.107:0/2477982966 conn(0x7f8de00cb830 msgr2=0x7f8de015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 shutdown_connections 2023-11-24T08:07:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.406+0000 7f8ddf59e640 1 -- 172.21.15.107:0/2477982966 wait complete. 2023-11-24T08:07:45.428 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:07:45.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:07:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.641+0000 7f1fef71d640 1 Processor -- start 2023-11-24T08:07:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.641+0000 7f1fef71d640 1 -- start start 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fef71d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fe80b2250 0x7f1fe815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fef71d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe815ab80 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fef71d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe815e680 con 0x7f1fe80b2630 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fef71d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe815a990 con 0x7f1fe80b2250 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fed4b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fe80b2250 0x7f1fe815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fedcba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1fe80b2aa0 0x7f1fe815b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33010/0 (socket says 172.21.15.107:33010) 2023-11-24T08:07:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fedcba640 1 -- 172.21.15.107:0/136009274 learned_addr learned my addr 172.21.15.107:0/136009274 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fdffff640 1 -- 172.21.15.107:0/136009274 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793242645 0 0) 0x7f1fe815ab80 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fdffff640 1 -- 172.21.15.107:0/136009274 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd4003520 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fed4b9640 1 -- 172.21.15.107:0/136009274 >> v1:172.21.15.120:6789/0 conn(0x7f1fe80b2630 legacy=0x7f1fe80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fed4b9640 1 -- 172.21.15.107:0/136009274 >> v1:172.21.15.107:6789/0 conn(0x7f1fe80b2aa0 legacy=0x7f1fe815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.643+0000 7f1fed4b9640 1 -- 172.21.15.107:0/136009274 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe815e960 con 0x7f1fe80b2250 2023-11-24T08:07:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.645+0000 7f1fed4b9640 1 --2- 172.21.15.107:0/136009274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2250 0x7f1fe815df80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1fe815ad70 tx=0x7f1fe402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8905311143c7002 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fdffff640 1 -- 172.21.15.107:0/136009274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4030610 con 0x7f1fe80b2250 2023-11-24T08:07:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fdffff640 1 -- 172.21.15.107:0/136009274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4030770 con 0x7f1fe80b2250 2023-11-24T08:07:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fdffff640 1 -- 172.21.15.107:0/136009274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4030b00 con 0x7f1fe80b2250 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fef71d640 1 -- 172.21.15.107:0/136009274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2250 msgr2=0x7f1fe815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fef71d640 1 --2- 172.21.15.107:0/136009274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2250 0x7f1fe815df80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1fe815ad70 tx=0x7f1fe402b960 comp rx=0 tx=0).stop 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fef71d640 1 -- 172.21.15.107:0/136009274 shutdown_connections 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fef71d640 1 --2- 172.21.15.107:0/136009274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2250 0x7f1fe815df80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.646+0000 7f1fef71d640 1 -- 172.21.15.107:0/136009274 >> 172.21.15.107:0/136009274 conn(0x7f1fe806a920 msgr2=0x7f1fe806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.647+0000 7f1fef71d640 1 -- 172.21.15.107:0/136009274 shutdown_connections 2023-11-24T08:07:45.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.647+0000 7f1fef71d640 1 -- 172.21.15.107:0/136009274 wait complete. 2023-11-24T08:07:45.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.648+0000 7f1fef71d640 1 Processor -- start 2023-11-24T08:07:45.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.648+0000 7f1fef71d640 1 -- start start 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1fe80b2250 0x7f1fe8071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fe80b2630 0x7f1fe8071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe81f80b0 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe81f7430 con 0x7f1fe80b2630 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.649+0000 7f1fef71d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe81f7710 con 0x7f1fe80b2250 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42418/0 (socket says 172.21.15.107:42418) 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 -- 172.21.15.107:0/4240875087 learned_addr learned my addr 172.21.15.107:0/4240875087 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1feccb8640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fe80b2630 0x7f1fe8071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 -- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1fe80b2250 msgr2=0x7f1fe8071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1fe80b2250 0x7f1fe8071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 -- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fe80b2630 msgr2=0x7f1fe8071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fe80b2630 0x7f1fe8071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 -- 172.21.15.107:0/4240875087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe4009400 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.650+0000 7f1fedcba640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000c9f0 tx=0x7f1fe000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.651+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe000ab70 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.651+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe000acd0 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.651+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0012600 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.651+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe80cd7c0 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.651+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe81f7cd0 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.652+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe8067470 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.653+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f1fe0007070 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.654+0000 7f1fddffb640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 0x7f1fcc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.654+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fe0089880 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.654+0000 7f1fed4b9640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 0x7f1fcc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.655+0000 7f1fed4b9640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 0x7f1fcc06d5d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.658+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fe0054650 con 0x7f1fe80b2aa0 2023-11-24T08:07:45.784 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:07:45.783+0000 7f68bb71e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:07:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.845+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fe80cd640 con 0x7f1fcc06b110 2023-11-24T08:07:45.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.862+0000 7f1fddffb640 1 -- 172.21.15.107:0/4240875087 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286881 (secure 0 0 0) 0x7f1fe40308e0 con 0x7f1fcc06b110 2023-11-24T08:07:45.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:45.868 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.868+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 msgr2=0x7f1fcc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.868+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 0x7f1fcc06d5d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe403b040 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.868+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 msgr2=0x7f1fe80b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.868+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000c9f0 tx=0x7f1fe000ceb0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 shutdown_connections 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1fcc06b110 0x7f1fcc06d5d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe80b2aa0 0x7f1fe80b0980 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fe80b2630 0x7f1fe8071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 --2- 172.21.15.107:0/4240875087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1fe80b2250 0x7f1fe8071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:45.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 >> 172.21.15.107:0/4240875087 conn(0x7f1fe806a920 msgr2=0x7f1fe80ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:45.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 shutdown_connections 2023-11-24T08:07:45.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:45.869+0000 7f1fef71d640 1 -- 172.21.15.107:0/4240875087 wait complete. 2023-11-24T08:07:45.889 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2023-11-24T08:07:45.483773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242640,"kb_used_data":5640,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754732080,"statfs":{"total":773094113280,"available":772845649920,"internally_reserved":0,"allocated":5775360,"data_stored":3047596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":242535998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001139"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844874+0000","last_change":"2023-11-24T08:07:41.473827+0000","last_active":"2023-11-24T08:07:42.844874+0000","last_peered":"2023-11-24T08:07:42.844874+0000","last_clean":"2023-11-24T08:07:42.844874+0000","last_became_active":"2023-11-24T08:07:41.473511+0000","last_became_peered":"2023-11-24T08:07:41.473511+0000","last_unstale":"2023-11-24T08:07:42.844874+0000","last_undegraded":"2023-11-24T08:07:42.844874+0000","last_fullsized":"2023-11-24T08:07:42.844874+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:24.855235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841734+0000","last_change":"2023-11-24T08:07:41.479271+0000","last_active":"2023-11-24T08:07:42.841734+0000","last_peered":"2023-11-24T08:07:42.841734+0000","last_clean":"2023-11-24T08:07:42.841734+0000","last_became_active":"2023-11-24T08:07:41.478875+0000","last_became_peered":"2023-11-24T08:07:41.478875+0000","last_unstale":"2023-11-24T08:07:42.841734+0000","last_undegraded":"2023-11-24T08:07:42.841734+0000","last_fullsized":"2023-11-24T08:07:42.841734+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:39:44.765369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466243+0000","last_change":"2023-11-24T08:07:41.460984+0000","last_active":"2023-11-24T08:07:42.466243+0000","last_peered":"2023-11-24T08:07:42.466243+0000","last_clean":"2023-11-24T08:07:42.466243+0000","last_became_active":"2023-11-24T08:07:41.460668+0000","last_became_peered":"2023-11-24T08:07:41.460668+0000","last_unstale":"2023-11-24T08:07:42.466243+0000","last_undegraded":"2023-11-24T08:07:42.466243+0000","last_fullsized":"2023-11-24T08:07:42.466243+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:50.147256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840390+0000","last_change":"2023-11-24T08:07:41.468619+0000","last_active":"2023-11-24T08:07:42.840390+0000","last_peered":"2023-11-24T08:07:42.840390+0000","last_clean":"2023-11-24T08:07:42.840390+0000","last_became_active":"2023-11-24T08:07:41.465772+0000","last_became_peered":"2023-11-24T08:07:41.465772+0000","last_unstale":"2023-11-24T08:07:42.840390+0000","last_undegraded":"2023-11-24T08:07:42.840390+0000","last_fullsized":"2023-11-24T08:07:42.840390+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:30.418133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840644+0000","last_change":"2023-11-24T08:07:41.475378+0000","last_active":"2023-11-24T08:07:42.840644+0000","last_peered":"2023-11-24T08:07:42.840644+0000","last_clean":"2023-11-24T08:07:42.840644+0000","last_became_active":"2023-11-24T08:07:41.473509+0000","last_became_peered":"2023-11-24T08:07:41.473509+0000","last_unstale":"2023-11-24T08:07:42.840644+0000","last_undegraded":"2023-11-24T08:07:42.840644+0000","last_fullsized":"2023-11-24T08:07:42.840644+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:19.994298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843441+0000","last_change":"2023-11-24T08:07:41.472221+0000","last_active":"2023-11-24T08:07:42.843441+0000","last_peered":"2023-11-24T08:07:42.843441+0000","last_clean":"2023-11-24T08:07:42.843441+0000","last_became_active":"2023-11-24T08:07:41.468190+0000","last_became_peered":"2023-11-24T08:07:41.468190+0000","last_unstale":"2023-11-24T08:07:42.843441+0000","last_undegraded":"2023-11-24T08:07:42.843441+0000","last_fullsized":"2023-11-24T08:07:42.843441+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:47:45.395075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842924+0000","last_change":"2023-11-24T08:07:41.463573+0000","last_active":"2023-11-24T08:07:42.842924+0000","last_peered":"2023-11-24T08:07:42.842924+0000","last_clean":"2023-11-24T08:07:42.842924+0000","last_became_active":"2023-11-24T08:07:41.462861+0000","last_became_peered":"2023-11-24T08:07:41.462861+0000","last_unstale":"2023-11-24T08:07:42.842924+0000","last_undegraded":"2023-11-24T08:07:42.842924+0000","last_fullsized":"2023-11-24T08:07:42.842924+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:18.275648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840210+0000","last_change":"2023-11-24T08:07:41.466948+0000","last_active":"2023-11-24T08:07:42.840210+0000","last_peered":"2023-11-24T08:07:42.840210+0000","last_clean":"2023-11-24T08:07:42.840210+0000","last_became_active":"2023-11-24T08:07:41.466319+0000","last_became_peered":"2023-11-24T08:07:41.466319+0000","last_unstale":"2023-11-24T08:07:42.840210+0000","last_undegraded":"2023-11-24T08:07:42.840210+0000","last_fullsized":"2023-11-24T08:07:42.840210+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:16.113678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842688+0000","last_change":"2023-11-24T08:07:41.452006+0000","last_active":"2023-11-24T08:07:42.842688+0000","last_peered":"2023-11-24T08:07:42.842688+0000","last_clean":"2023-11-24T08:07:42.842688+0000","last_became_active":"2023-11-24T08:07:41.451738+0000","last_became_peered":"2023-11-24T08:07:41.451738+0000","last_unstale":"2023-11-24T08:07:42.842688+0000","last_undegraded":"2023-11-24T08:07:42.842688+0000","last_fullsized":"2023-11-24T08:07:42.842688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:04.261312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842534+0000","last_change":"2023-11-24T08:07:41.467923+0000","last_active":"2023-11-24T08:07:42.842534+0000","last_peered":"2023-11-24T08:07:42.842534+0000","last_clean":"2023-11-24T08:07:42.842534+0000","last_became_active":"2023-11-24T08:07:41.467550+0000","last_became_peered":"2023-11-24T08:07:41.467550+0000","last_unstale":"2023-11-24T08:07:42.842534+0000","last_undegraded":"2023-11-24T08:07:42.842534+0000","last_fullsized":"2023-11-24T08:07:42.842534+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:40.344933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843074+0000","last_change":"2023-11-24T08:07:41.458849+0000","last_active":"2023-11-24T08:07:42.843074+0000","last_peered":"2023-11-24T08:07:42.843074+0000","last_clean":"2023-11-24T08:07:42.843074+0000","last_became_active":"2023-11-24T08:07:41.458575+0000","last_became_peered":"2023-11-24T08:07:41.458575+0000","last_unstale":"2023-11-24T08:07:42.843074+0000","last_undegraded":"2023-11-24T08:07:42.843074+0000","last_fullsized":"2023-11-24T08:07:42.843074+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:19.994298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841278+0000","last_change":"2023-11-24T08:07:41.465343+0000","last_active":"2023-11-24T08:07:42.841278+0000","last_peered":"2023-11-24T08:07:42.841278+0000","last_clean":"2023-11-24T08:07:42.841278+0000","last_became_active":"2023-11-24T08:07:41.465002+0000","last_became_peered":"2023-11-24T08:07:41.465002+0000","last_unstale":"2023-11-24T08:07:42.841278+0000","last_undegraded":"2023-11-24T08:07:42.841278+0000","last_fullsized":"2023-11-24T08:07:42.841278+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:54.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842220+0000","last_change":"2023-11-24T08:07:41.456134+0000","last_active":"2023-11-24T08:07:42.842220+0000","last_peered":"2023-11-24T08:07:42.842220+0000","last_clean":"2023-11-24T08:07:42.842220+0000","last_became_active":"2023-11-24T08:07:41.455846+0000","last_became_peered":"2023-11-24T08:07:41.455846+0000","last_unstale":"2023-11-24T08:07:42.842220+0000","last_undegraded":"2023-11-24T08:07:42.842220+0000","last_fullsized":"2023-11-24T08:07:42.842220+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:13.922862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840496+0000","last_change":"2023-11-24T08:07:41.462455+0000","last_active":"2023-11-24T08:07:42.840496+0000","last_peered":"2023-11-24T08:07:42.840496+0000","last_clean":"2023-11-24T08:07:42.840496+0000","last_became_active":"2023-11-24T08:07:41.462174+0000","last_became_peered":"2023-11-24T08:07:41.462174+0000","last_unstale":"2023-11-24T08:07:42.840496+0000","last_undegraded":"2023-11-24T08:07:42.840496+0000","last_fullsized":"2023-11-24T08:07:42.840496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:15.445883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840035+0000","last_change":"2023-11-24T08:07:41.469977+0000","last_active":"2023-11-24T08:07:42.840035+0000","last_peered":"2023-11-24T08:07:42.840035+0000","last_clean":"2023-11-24T08:07:42.840035+0000","last_became_active":"2023-11-24T08:07:41.469707+0000","last_became_peered":"2023-11-24T08:07:41.469707+0000","last_unstale":"2023-11-24T08:07:42.840035+0000","last_undegraded":"2023-11-24T08:07:42.840035+0000","last_fullsized":"2023-11-24T08:07:42.840035+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:00.242666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842186+0000","last_change":"2023-11-24T08:07:41.471930+0000","last_active":"2023-11-24T08:07:42.842186+0000","last_peered":"2023-11-24T08:07:42.842186+0000","last_clean":"2023-11-24T08:07:42.842186+0000","last_became_active":"2023-11-24T08:07:41.471586+0000","last_became_peered":"2023-11-24T08:07:41.471586+0000","last_unstale":"2023-11-24T08:07:42.842186+0000","last_undegraded":"2023-11-24T08:07:42.842186+0000","last_fullsized":"2023-11-24T08:07:42.842186+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:20:44.155085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466560+0000","last_change":"2023-11-24T08:07:41.472539+0000","last_active":"2023-11-24T08:07:42.466560+0000","last_peered":"2023-11-24T08:07:42.466560+0000","last_clean":"2023-11-24T08:07:42.466560+0000","last_became_active":"2023-11-24T08:07:41.468323+0000","last_became_peered":"2023-11-24T08:07:41.468323+0000","last_unstale":"2023-11-24T08:07:42.466560+0000","last_undegraded":"2023-11-24T08:07:42.466560+0000","last_fullsized":"2023-11-24T08:07:42.466560+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:34.933490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467410+0000","last_change":"2023-11-24T08:07:41.471253+0000","last_active":"2023-11-24T08:07:42.467410+0000","last_peered":"2023-11-24T08:07:42.467410+0000","last_clean":"2023-11-24T08:07:42.467410+0000","last_became_active":"2023-11-24T08:07:41.470223+0000","last_became_peered":"2023-11-24T08:07:41.470223+0000","last_unstale":"2023-11-24T08:07:42.467410+0000","last_undegraded":"2023-11-24T08:07:42.467410+0000","last_fullsized":"2023-11-24T08:07:42.467410+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:40.344933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844557+0000","last_change":"2023-11-24T08:07:41.463419+0000","last_active":"2023-11-24T08:07:42.844557+0000","last_peered":"2023-11-24T08:07:42.844557+0000","last_clean":"2023-11-24T08:07:42.844557+0000","last_became_active":"2023-11-24T08:07:41.463051+0000","last_became_peered":"2023-11-24T08:07:41.463051+0000","last_unstale":"2023-11-24T08:07:42.844557+0000","last_undegraded":"2023-11-24T08:07:42.844557+0000","last_fullsized":"2023-11-24T08:07:42.844557+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:58.335836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842693+0000","last_change":"2023-11-24T08:07:41.464711+0000","last_active":"2023-11-24T08:07:42.842693+0000","last_peered":"2023-11-24T08:07:42.842693+0000","last_clean":"2023-11-24T08:07:42.842693+0000","last_became_active":"2023-11-24T08:07:41.464409+0000","last_became_peered":"2023-11-24T08:07:41.464409+0000","last_unstale":"2023-11-24T08:07:42.842693+0000","last_undegraded":"2023-11-24T08:07:42.842693+0000","last_fullsized":"2023-11-24T08:07:42.842693+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:20:44.155085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841576+0000","last_change":"2023-11-24T08:07:41.470935+0000","last_active":"2023-11-24T08:07:42.841576+0000","last_peered":"2023-11-24T08:07:42.841576+0000","last_clean":"2023-11-24T08:07:42.841576+0000","last_became_active":"2023-11-24T08:07:41.470625+0000","last_became_peered":"2023-11-24T08:07:41.470625+0000","last_unstale":"2023-11-24T08:07:42.841576+0000","last_undegraded":"2023-11-24T08:07:42.841576+0000","last_fullsized":"2023-11-24T08:07:42.841576+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:12:23.483181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840510+0000","last_change":"2023-11-24T08:07:41.471906+0000","last_active":"2023-11-24T08:07:42.840510+0000","last_peered":"2023-11-24T08:07:42.840510+0000","last_clean":"2023-11-24T08:07:42.840510+0000","last_became_active":"2023-11-24T08:07:41.471640+0000","last_became_peered":"2023-11-24T08:07:41.471640+0000","last_unstale":"2023-11-24T08:07:42.840510+0000","last_undegraded":"2023-11-24T08:07:42.840510+0000","last_fullsized":"2023-11-24T08:07:42.840510+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:19.186521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843057+0000","last_change":"2023-11-24T08:07:41.472346+0000","last_active":"2023-11-24T08:07:42.843057+0000","last_peered":"2023-11-24T08:07:42.843057+0000","last_clean":"2023-11-24T08:07:42.843057+0000","last_became_active":"2023-11-24T08:07:41.469021+0000","last_became_peered":"2023-11-24T08:07:41.469021+0000","last_unstale":"2023-11-24T08:07:42.843057+0000","last_undegraded":"2023-11-24T08:07:42.843057+0000","last_fullsized":"2023-11-24T08:07:42.843057+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:16.982235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842564+0000","last_change":"2023-11-24T08:07:41.473146+0000","last_active":"2023-11-24T08:07:42.842564+0000","last_peered":"2023-11-24T08:07:42.842564+0000","last_clean":"2023-11-24T08:07:42.842564+0000","last_became_active":"2023-11-24T08:07:41.470291+0000","last_became_peered":"2023-11-24T08:07:41.470291+0000","last_unstale":"2023-11-24T08:07:42.842564+0000","last_undegraded":"2023-11-24T08:07:42.842564+0000","last_fullsized":"2023-11-24T08:07:42.842564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:14.246888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844813+0000","last_change":"2023-11-24T08:07:41.464066+0000","last_active":"2023-11-24T08:07:42.844813+0000","last_peered":"2023-11-24T08:07:42.844813+0000","last_clean":"2023-11-24T08:07:42.844813+0000","last_became_active":"2023-11-24T08:07:41.463233+0000","last_became_peered":"2023-11-24T08:07:41.463233+0000","last_unstale":"2023-11-24T08:07:42.844813+0000","last_undegraded":"2023-11-24T08:07:42.844813+0000","last_fullsized":"2023-11-24T08:07:42.844813+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:04.924461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843109+0000","last_change":"2023-11-24T08:07:41.456768+0000","last_active":"2023-11-24T08:07:42.843109+0000","last_peered":"2023-11-24T08:07:42.843109+0000","last_clean":"2023-11-24T08:07:42.843109+0000","last_became_active":"2023-11-24T08:07:41.455041+0000","last_became_peered":"2023-11-24T08:07:41.455041+0000","last_unstale":"2023-11-24T08:07:42.843109+0000","last_undegraded":"2023-11-24T08:07:42.843109+0000","last_fullsized":"2023-11-24T08:07:42.843109+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:14.246888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466618+0000","last_change":"2023-11-24T08:07:41.464201+0000","last_active":"2023-11-24T08:07:42.466618+0000","last_peered":"2023-11-24T08:07:42.466618+0000","last_clean":"2023-11-24T08:07:42.466618+0000","last_became_active":"2023-11-24T08:07:41.463925+0000","last_became_peered":"2023-11-24T08:07:41.463925+0000","last_unstale":"2023-11-24T08:07:42.466618+0000","last_undegraded":"2023-11-24T08:07:42.466618+0000","last_fullsized":"2023-11-24T08:07:42.466618+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:56.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842615+0000","last_change":"2023-11-24T08:07:41.464869+0000","last_active":"2023-11-24T08:07:42.842615+0000","last_peered":"2023-11-24T08:07:42.842615+0000","last_clean":"2023-11-24T08:07:42.842615+0000","last_became_active":"2023-11-24T08:07:41.464571+0000","last_became_peered":"2023-11-24T08:07:41.464571+0000","last_unstale":"2023-11-24T08:07:42.842615+0000","last_undegraded":"2023-11-24T08:07:42.842615+0000","last_fullsized":"2023-11-24T08:07:42.842615+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:13:53.448314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841896+0000","last_change":"2023-11-24T08:07:41.462844+0000","last_active":"2023-11-24T08:07:42.841896+0000","last_peered":"2023-11-24T08:07:42.841896+0000","last_clean":"2023-11-24T08:07:42.841896+0000","last_became_active":"2023-11-24T08:07:41.460100+0000","last_became_peered":"2023-11-24T08:07:41.460100+0000","last_unstale":"2023-11-24T08:07:42.841896+0000","last_undegraded":"2023-11-24T08:07:42.841896+0000","last_fullsized":"2023-11-24T08:07:42.841896+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:13.922862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.465968+0000","last_change":"2023-11-24T08:07:41.467096+0000","last_active":"2023-11-24T08:07:42.465968+0000","last_peered":"2023-11-24T08:07:42.465968+0000","last_clean":"2023-11-24T08:07:42.465968+0000","last_became_active":"2023-11-24T08:07:41.466837+0000","last_became_peered":"2023-11-24T08:07:41.466837+0000","last_unstale":"2023-11-24T08:07:42.465968+0000","last_undegraded":"2023-11-24T08:07:42.465968+0000","last_fullsized":"2023-11-24T08:07:42.465968+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:15.445883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.839686+0000","last_change":"2023-11-24T08:07:41.462683+0000","last_active":"2023-11-24T08:07:42.839686+0000","last_peered":"2023-11-24T08:07:42.839686+0000","last_clean":"2023-11-24T08:07:42.839686+0000","last_became_active":"2023-11-24T08:07:41.462319+0000","last_became_peered":"2023-11-24T08:07:41.462319+0000","last_unstale":"2023-11-24T08:07:42.839686+0000","last_undegraded":"2023-11-24T08:07:42.839686+0000","last_fullsized":"2023-11-24T08:07:42.839686+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:31.395873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841125+0000","last_change":"2023-11-24T08:07:41.460653+0000","last_active":"2023-11-24T08:07:42.841125+0000","last_peered":"2023-11-24T08:07:42.841125+0000","last_clean":"2023-11-24T08:07:42.841125+0000","last_became_active":"2023-11-24T08:07:41.459508+0000","last_became_peered":"2023-11-24T08:07:41.459508+0000","last_unstale":"2023-11-24T08:07:42.841125+0000","last_undegraded":"2023-11-24T08:07:42.841125+0000","last_fullsized":"2023-11-24T08:07:42.841125+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:55:07.078915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462208+0000","last_change":"2023-11-24T08:07:41.456459+0000","last_active":"2023-11-24T08:07:42.462208+0000","last_peered":"2023-11-24T08:07:42.462208+0000","last_clean":"2023-11-24T08:07:42.462208+0000","last_became_active":"2023-11-24T08:07:41.456185+0000","last_became_peered":"2023-11-24T08:07:41.456185+0000","last_unstale":"2023-11-24T08:07:42.462208+0000","last_undegraded":"2023-11-24T08:07:42.462208+0000","last_fullsized":"2023-11-24T08:07:42.462208+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:30.418133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843651+0000","last_change":"2023-11-24T08:07:40.459884+0000","last_active":"2023-11-24T08:07:42.843651+0000","last_peered":"2023-11-24T08:07:42.843651+0000","last_clean":"2023-11-24T08:07:42.843651+0000","last_became_active":"2023-11-24T08:07:40.459629+0000","last_became_peered":"2023-11-24T08:07:40.459629+0000","last_unstale":"2023-11-24T08:07:42.843651+0000","last_undegraded":"2023-11-24T08:07:42.843651+0000","last_fullsized":"2023-11-24T08:07:42.843651+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:53.455094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844868+0000","last_change":"2023-11-24T08:07:41.476427+0000","last_active":"2023-11-24T08:07:42.844868+0000","last_peered":"2023-11-24T08:07:42.844868+0000","last_clean":"2023-11-24T08:07:42.844868+0000","last_became_active":"2023-11-24T08:07:41.476066+0000","last_became_peered":"2023-11-24T08:07:41.476066+0000","last_unstale":"2023-11-24T08:07:42.844868+0000","last_undegraded":"2023-11-24T08:07:42.844868+0000","last_fullsized":"2023-11-24T08:07:42.844868+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:29.029988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467676+0000","last_change":"2023-11-24T08:07:40.460230+0000","last_active":"2023-11-24T08:07:42.467676+0000","last_peered":"2023-11-24T08:07:42.467676+0000","last_clean":"2023-11-24T08:07:42.467676+0000","last_became_active":"2023-11-24T08:07:40.459906+0000","last_became_peered":"2023-11-24T08:07:40.459906+0000","last_unstale":"2023-11-24T08:07:42.467676+0000","last_undegraded":"2023-11-24T08:07:42.467676+0000","last_fullsized":"2023-11-24T08:07:42.467676+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:55.514109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842961+0000","last_change":"2023-11-24T08:07:41.475519+0000","last_active":"2023-11-24T08:07:42.842961+0000","last_peered":"2023-11-24T08:07:42.842961+0000","last_clean":"2023-11-24T08:07:42.842961+0000","last_became_active":"2023-11-24T08:07:41.475245+0000","last_became_peered":"2023-11-24T08:07:41.475245+0000","last_unstale":"2023-11-24T08:07:42.842961+0000","last_undegraded":"2023-11-24T08:07:42.842961+0000","last_fullsized":"2023-11-24T08:07:42.842961+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:21.137511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462351+0000","last_change":"2023-11-24T08:07:40.460426+0000","last_active":"2023-11-24T08:07:42.462351+0000","last_peered":"2023-11-24T08:07:42.462351+0000","last_clean":"2023-11-24T08:07:42.462351+0000","last_became_active":"2023-11-24T08:07:40.460191+0000","last_became_peered":"2023-11-24T08:07:40.460191+0000","last_unstale":"2023-11-24T08:07:42.462351+0000","last_undegraded":"2023-11-24T08:07:42.462351+0000","last_fullsized":"2023-11-24T08:07:42.462351+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:03.908377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462133+0000","last_change":"2023-11-24T08:07:41.464537+0000","last_active":"2023-11-24T08:07:42.462133+0000","last_peered":"2023-11-24T08:07:42.462133+0000","last_clean":"2023-11-24T08:07:42.462133+0000","last_became_active":"2023-11-24T08:07:41.464265+0000","last_became_peered":"2023-11-24T08:07:41.464265+0000","last_unstale":"2023-11-24T08:07:42.462133+0000","last_undegraded":"2023-11-24T08:07:42.462133+0000","last_fullsized":"2023-11-24T08:07:42.462133+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:56.530194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844279+0000","last_change":"2023-11-24T08:07:40.466384+0000","last_active":"2023-11-24T08:07:42.844279+0000","last_peered":"2023-11-24T08:07:42.844279+0000","last_clean":"2023-11-24T08:07:42.844279+0000","last_became_active":"2023-11-24T08:07:40.466070+0000","last_became_peered":"2023-11-24T08:07:40.466070+0000","last_unstale":"2023-11-24T08:07:42.844279+0000","last_undegraded":"2023-11-24T08:07:42.844279+0000","last_fullsized":"2023-11-24T08:07:42.844279+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:33:13.420060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466017+0000","last_change":"2023-11-24T08:07:41.464740+0000","last_active":"2023-11-24T08:07:42.466017+0000","last_peered":"2023-11-24T08:07:42.466017+0000","last_clean":"2023-11-24T08:07:42.466017+0000","last_became_active":"2023-11-24T08:07:41.464465+0000","last_became_peered":"2023-11-24T08:07:41.464465+0000","last_unstale":"2023-11-24T08:07:42.466017+0000","last_undegraded":"2023-11-24T08:07:42.466017+0000","last_fullsized":"2023-11-24T08:07:42.466017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:35.559462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842152+0000","last_change":"2023-11-24T08:07:40.457574+0000","last_active":"2023-11-24T08:07:42.842152+0000","last_peered":"2023-11-24T08:07:42.842152+0000","last_clean":"2023-11-24T08:07:42.842152+0000","last_became_active":"2023-11-24T08:07:40.457283+0000","last_became_peered":"2023-11-24T08:07:40.457283+0000","last_unstale":"2023-11-24T08:07:42.842152+0000","last_undegraded":"2023-11-24T08:07:42.842152+0000","last_fullsized":"2023-11-24T08:07:42.842152+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:53.955751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841127+0000","last_change":"2023-11-24T08:07:41.465589+0000","last_active":"2023-11-24T08:07:42.841127+0000","last_peered":"2023-11-24T08:07:42.841127+0000","last_clean":"2023-11-24T08:07:42.841127+0000","last_became_active":"2023-11-24T08:07:41.465308+0000","last_became_peered":"2023-11-24T08:07:41.465308+0000","last_unstale":"2023-11-24T08:07:42.841127+0000","last_undegraded":"2023-11-24T08:07:42.841127+0000","last_fullsized":"2023-11-24T08:07:42.841127+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:34.933490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466241+0000","last_change":"2023-11-24T08:07:40.459394+0000","last_active":"2023-11-24T08:07:42.466241+0000","last_peered":"2023-11-24T08:07:42.466241+0000","last_clean":"2023-11-24T08:07:42.466241+0000","last_became_active":"2023-11-24T08:07:40.457216+0000","last_became_peered":"2023-11-24T08:07:40.457216+0000","last_unstale":"2023-11-24T08:07:42.466241+0000","last_undegraded":"2023-11-24T08:07:42.466241+0000","last_fullsized":"2023-11-24T08:07:42.466241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:28.013904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840564+0000","last_change":"2023-11-24T08:07:41.467174+0000","last_active":"2023-11-24T08:07:42.840564+0000","last_peered":"2023-11-24T08:07:42.840564+0000","last_clean":"2023-11-24T08:07:42.840564+0000","last_became_active":"2023-11-24T08:07:41.466687+0000","last_became_peered":"2023-11-24T08:07:41.466687+0000","last_unstale":"2023-11-24T08:07:42.840564+0000","last_undegraded":"2023-11-24T08:07:42.840564+0000","last_fullsized":"2023-11-24T08:07:42.840564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:50.147256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840561+0000","last_change":"2023-11-24T08:07:40.458860+0000","last_active":"2023-11-24T08:07:42.840561+0000","last_peered":"2023-11-24T08:07:42.840561+0000","last_clean":"2023-11-24T08:07:42.840561+0000","last_became_active":"2023-11-24T08:07:40.457979+0000","last_became_peered":"2023-11-24T08:07:40.457979+0000","last_unstale":"2023-11-24T08:07:42.840561+0000","last_undegraded":"2023-11-24T08:07:42.840561+0000","last_fullsized":"2023-11-24T08:07:42.840561+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.781281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842923+0000","last_change":"2023-11-24T08:07:41.472828+0000","last_active":"2023-11-24T08:07:42.842923+0000","last_peered":"2023-11-24T08:07:42.842923+0000","last_clean":"2023-11-24T08:07:42.842923+0000","last_became_active":"2023-11-24T08:07:41.472547+0000","last_became_peered":"2023-11-24T08:07:41.472547+0000","last_unstale":"2023-11-24T08:07:42.842923+0000","last_undegraded":"2023-11-24T08:07:42.842923+0000","last_fullsized":"2023-11-24T08:07:42.842923+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:08:08.077023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840787+0000","last_change":"2023-11-24T08:07:40.453551+0000","last_active":"2023-11-24T08:07:42.840787+0000","last_peered":"2023-11-24T08:07:42.840787+0000","last_clean":"2023-11-24T08:07:42.840787+0000","last_became_active":"2023-11-24T08:07:40.453238+0000","last_became_peered":"2023-11-24T08:07:40.453238+0000","last_unstale":"2023-11-24T08:07:42.840787+0000","last_undegraded":"2023-11-24T08:07:42.840787+0000","last_fullsized":"2023-11-24T08:07:42.840787+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:48.178752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467608+0000","last_change":"2023-11-24T08:07:41.470675+0000","last_active":"2023-11-24T08:07:42.467608+0000","last_peered":"2023-11-24T08:07:42.467608+0000","last_clean":"2023-11-24T08:07:42.467608+0000","last_became_active":"2023-11-24T08:07:41.469620+0000","last_became_peered":"2023-11-24T08:07:41.469620+0000","last_unstale":"2023-11-24T08:07:42.467608+0000","last_undegraded":"2023-11-24T08:07:42.467608+0000","last_fullsized":"2023-11-24T08:07:42.467608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:31.037461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466774+0000","last_change":"2023-11-24T08:07:40.456573+0000","last_active":"2023-11-24T08:07:42.466774+0000","last_peered":"2023-11-24T08:07:42.466774+0000","last_clean":"2023-11-24T08:07:42.466774+0000","last_became_active":"2023-11-24T08:07:40.456259+0000","last_became_peered":"2023-11-24T08:07:40.456259+0000","last_unstale":"2023-11-24T08:07:42.466774+0000","last_undegraded":"2023-11-24T08:07:42.466774+0000","last_fullsized":"2023-11-24T08:07:42.466774+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:15.097594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.845067+0000","last_change":"2023-11-24T08:07:41.466779+0000","last_active":"2023-11-24T08:07:42.845067+0000","last_peered":"2023-11-24T08:07:42.845067+0000","last_clean":"2023-11-24T08:07:42.845067+0000","last_became_active":"2023-11-24T08:07:41.466433+0000","last_became_peered":"2023-11-24T08:07:41.466433+0000","last_unstale":"2023-11-24T08:07:42.845067+0000","last_undegraded":"2023-11-24T08:07:42.845067+0000","last_fullsized":"2023-11-24T08:07:42.845067+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:31.541468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842748+0000","last_change":"2023-11-24T08:07:40.460406+0000","last_active":"2023-11-24T08:07:42.842748+0000","last_peered":"2023-11-24T08:07:42.842748+0000","last_clean":"2023-11-24T08:07:42.842748+0000","last_became_active":"2023-11-24T08:07:40.459433+0000","last_became_peered":"2023-11-24T08:07:40.459433+0000","last_unstale":"2023-11-24T08:07:42.842748+0000","last_undegraded":"2023-11-24T08:07:42.842748+0000","last_fullsized":"2023-11-24T08:07:42.842748+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:55.617777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843305+0000","last_change":"2023-11-24T08:07:41.464595+0000","last_active":"2023-11-24T08:07:42.843305+0000","last_peered":"2023-11-24T08:07:42.843305+0000","last_clean":"2023-11-24T08:07:42.843305+0000","last_became_active":"2023-11-24T08:07:41.464303+0000","last_became_peered":"2023-11-24T08:07:41.464303+0000","last_unstale":"2023-11-24T08:07:42.843305+0000","last_undegraded":"2023-11-24T08:07:42.843305+0000","last_fullsized":"2023-11-24T08:07:42.843305+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:26.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844904+0000","last_change":"2023-11-24T08:07:40.465119+0000","last_active":"2023-11-24T08:07:42.844904+0000","last_peered":"2023-11-24T08:07:42.844904+0000","last_clean":"2023-11-24T08:07:42.844904+0000","last_became_active":"2023-11-24T08:07:40.464774+0000","last_became_peered":"2023-11-24T08:07:40.464774+0000","last_unstale":"2023-11-24T08:07:42.844904+0000","last_undegraded":"2023-11-24T08:07:42.844904+0000","last_fullsized":"2023-11-24T08:07:42.844904+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:13:52.432229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844653+0000","last_change":"2023-11-24T08:07:41.462187+0000","last_active":"2023-11-24T08:07:42.844653+0000","last_peered":"2023-11-24T08:07:42.844653+0000","last_clean":"2023-11-24T08:07:42.844653+0000","last_became_active":"2023-11-24T08:07:41.461922+0000","last_became_peered":"2023-11-24T08:07:41.461922+0000","last_unstale":"2023-11-24T08:07:42.844653+0000","last_undegraded":"2023-11-24T08:07:42.844653+0000","last_fullsized":"2023-11-24T08:07:42.844653+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:54.971836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":18,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843698+0000","last_change":"2023-11-24T08:07:40.453796+0000","last_active":"2023-11-24T08:07:42.843698+0000","last_peered":"2023-11-24T08:07:42.843698+0000","last_clean":"2023-11-24T08:07:42.843698+0000","last_became_active":"2023-11-24T08:07:40.453509+0000","last_became_peered":"2023-11-24T08:07:40.453509+0000","last_unstale":"2023-11-24T08:07:42.843698+0000","last_undegraded":"2023-11-24T08:07:42.843698+0000","last_fullsized":"2023-11-24T08:07:42.843698+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:19.431553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842690+0000","last_change":"2023-11-24T08:07:41.476259+0000","last_active":"2023-11-24T08:07:42.842690+0000","last_peered":"2023-11-24T08:07:42.842690+0000","last_clean":"2023-11-24T08:07:42.842690+0000","last_became_active":"2023-11-24T08:07:41.475274+0000","last_became_peered":"2023-11-24T08:07:41.475274+0000","last_unstale":"2023-11-24T08:07:42.842690+0000","last_undegraded":"2023-11-24T08:07:42.842690+0000","last_fullsized":"2023-11-24T08:07:42.842690+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:49.194836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842418+0000","last_change":"2023-11-24T08:07:40.454399+0000","last_active":"2023-11-24T08:07:42.842418+0000","last_peered":"2023-11-24T08:07:42.842418+0000","last_clean":"2023-11-24T08:07:42.842418+0000","last_became_active":"2023-11-24T08:07:40.453988+0000","last_became_peered":"2023-11-24T08:07:40.453988+0000","last_unstale":"2023-11-24T08:07:42.842418+0000","last_undegraded":"2023-11-24T08:07:42.842418+0000","last_fullsized":"2023-11-24T08:07:42.842418+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:27:22.254123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841365+0000","last_change":"2023-11-24T08:07:41.465201+0000","last_active":"2023-11-24T08:07:42.841365+0000","last_peered":"2023-11-24T08:07:42.841365+0000","last_clean":"2023-11-24T08:07:42.841365+0000","last_became_active":"2023-11-24T08:07:41.464932+0000","last_became_peered":"2023-11-24T08:07:41.464932+0000","last_unstale":"2023-11-24T08:07:42.841365+0000","last_undegraded":"2023-11-24T08:07:42.841365+0000","last_fullsized":"2023-11-24T08:07:42.841365+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:56.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843704+0000","last_change":"2023-11-24T08:07:40.461047+0000","last_active":"2023-11-24T08:07:42.843704+0000","last_peered":"2023-11-24T08:07:42.843704+0000","last_clean":"2023-11-24T08:07:42.843704+0000","last_became_active":"2023-11-24T08:07:40.460193+0000","last_became_peered":"2023-11-24T08:07:40.460193+0000","last_unstale":"2023-11-24T08:07:42.843704+0000","last_undegraded":"2023-11-24T08:07:42.843704+0000","last_fullsized":"2023-11-24T08:07:42.843704+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:33.917406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.839474+0000","last_change":"2023-11-24T08:07:41.454638+0000","last_active":"2023-11-24T08:07:42.839474+0000","last_peered":"2023-11-24T08:07:42.839474+0000","last_clean":"2023-11-24T08:07:42.839474+0000","last_became_active":"2023-11-24T08:07:41.454272+0000","last_became_peered":"2023-11-24T08:07:41.454272+0000","last_unstale":"2023-11-24T08:07:42.839474+0000","last_undegraded":"2023-11-24T08:07:42.839474+0000","last_fullsized":"2023-11-24T08:07:42.839474+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:47.782086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841676+0000","last_change":"2023-11-24T08:07:40.455492+0000","last_active":"2023-11-24T08:07:42.841676+0000","last_peered":"2023-11-24T08:07:42.841676+0000","last_clean":"2023-11-24T08:07:42.841676+0000","last_became_active":"2023-11-24T08:07:40.455215+0000","last_became_peered":"2023-11-24T08:07:40.455215+0000","last_unstale":"2023-11-24T08:07:42.841676+0000","last_undegraded":"2023-11-24T08:07:42.841676+0000","last_fullsized":"2023-11-24T08:07:42.841676+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:30.525384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843268+0000","last_change":"2023-11-24T08:07:41.463653+0000","last_active":"2023-11-24T08:07:42.843268+0000","last_peered":"2023-11-24T08:07:42.843268+0000","last_clean":"2023-11-24T08:07:42.843268+0000","last_became_active":"2023-11-24T08:07:41.463273+0000","last_became_peered":"2023-11-24T08:07:41.463273+0000","last_unstale":"2023-11-24T08:07:42.843268+0000","last_undegraded":"2023-11-24T08:07:42.843268+0000","last_fullsized":"2023-11-24T08:07:42.843268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:57.190796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":20,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841006+0000","last_change":"2023-11-24T08:07:40.462426+0000","last_active":"2023-11-24T08:07:42.841006+0000","last_peered":"2023-11-24T08:07:42.841006+0000","last_clean":"2023-11-24T08:07:42.841006+0000","last_became_active":"2023-11-24T08:07:40.461871+0000","last_became_peered":"2023-11-24T08:07:40.461871+0000","last_unstale":"2023-11-24T08:07:42.841006+0000","last_undegraded":"2023-11-24T08:07:42.841006+0000","last_fullsized":"2023-11-24T08:07:42.841006+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:08:07.060938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843492+0000","last_change":"2023-11-24T08:07:41.465724+0000","last_active":"2023-11-24T08:07:42.843492+0000","last_peered":"2023-11-24T08:07:42.843492+0000","last_clean":"2023-11-24T08:07:42.843492+0000","last_became_active":"2023-11-24T08:07:41.465477+0000","last_became_peered":"2023-11-24T08:07:41.465477+0000","last_unstale":"2023-11-24T08:07:42.843492+0000","last_undegraded":"2023-11-24T08:07:42.843492+0000","last_fullsized":"2023-11-24T08:07:42.843492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:49.194836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843161+0000","last_change":"2023-11-24T08:07:40.452058+0000","last_active":"2023-11-24T08:07:42.843161+0000","last_peered":"2023-11-24T08:07:42.843161+0000","last_clean":"2023-11-24T08:07:42.843161+0000","last_became_active":"2023-11-24T08:07:40.451765+0000","last_became_peered":"2023-11-24T08:07:40.451765+0000","last_unstale":"2023-11-24T08:07:42.843161+0000","last_undegraded":"2023-11-24T08:07:42.843161+0000","last_fullsized":"2023-11-24T08:07:42.843161+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:19:04.167570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843007+0000","last_change":"2023-11-24T08:07:41.458566+0000","last_active":"2023-11-24T08:07:42.843007+0000","last_peered":"2023-11-24T08:07:42.843007+0000","last_clean":"2023-11-24T08:07:42.843007+0000","last_became_active":"2023-11-24T08:07:41.458307+0000","last_became_peered":"2023-11-24T08:07:41.458307+0000","last_unstale":"2023-11-24T08:07:42.843007+0000","last_undegraded":"2023-11-24T08:07:42.843007+0000","last_fullsized":"2023-11-24T08:07:42.843007+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:33:14.436145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467804+0000","last_change":"2023-11-24T08:07:40.457373+0000","last_active":"2023-11-24T08:07:42.467804+0000","last_peered":"2023-11-24T08:07:42.467804+0000","last_clean":"2023-11-24T08:07:42.467804+0000","last_became_active":"2023-11-24T08:07:40.457073+0000","last_became_peered":"2023-11-24T08:07:40.457073+0000","last_unstale":"2023-11-24T08:07:42.467804+0000","last_undegraded":"2023-11-24T08:07:42.467804+0000","last_fullsized":"2023-11-24T08:07:42.467804+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:29.402048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844691+0000","last_change":"2023-11-24T08:07:41.473723+0000","last_active":"2023-11-24T08:07:42.844691+0000","last_peered":"2023-11-24T08:07:42.844691+0000","last_clean":"2023-11-24T08:07:42.844691+0000","last_became_active":"2023-11-24T08:07:41.473434+0000","last_became_peered":"2023-11-24T08:07:41.473434+0000","last_unstale":"2023-11-24T08:07:42.844691+0000","last_undegraded":"2023-11-24T08:07:42.844691+0000","last_fullsized":"2023-11-24T08:07:42.844691+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:52.892503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841664+0000","last_change":"2023-11-24T08:07:40.460830+0000","last_active":"2023-11-24T08:07:42.841664+0000","last_peered":"2023-11-24T08:07:42.841664+0000","last_clean":"2023-11-24T08:07:42.841664+0000","last_became_active":"2023-11-24T08:07:40.460057+0000","last_became_peered":"2023-11-24T08:07:40.460057+0000","last_unstale":"2023-11-24T08:07:42.841664+0000","last_undegraded":"2023-11-24T08:07:42.841664+0000","last_fullsized":"2023-11-24T08:07:42.841664+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:30.021377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844277+0000","last_change":"2023-11-24T08:07:41.465842+0000","last_active":"2023-11-24T08:07:42.844277+0000","last_peered":"2023-11-24T08:07:42.844277+0000","last_clean":"2023-11-24T08:07:42.844277+0000","last_became_active":"2023-11-24T08:07:41.465408+0000","last_became_peered":"2023-11-24T08:07:41.465408+0000","last_unstale":"2023-11-24T08:07:42.844277+0000","last_undegraded":"2023-11-24T08:07:42.844277+0000","last_fullsized":"2023-11-24T08:07:42.844277+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:03:36.624929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843505+0000","last_change":"2023-11-24T08:07:40.461173+0000","last_active":"2023-11-24T08:07:42.843505+0000","last_peered":"2023-11-24T08:07:42.843505+0000","last_clean":"2023-11-24T08:07:42.843505+0000","last_became_active":"2023-11-24T08:07:40.460769+0000","last_became_peered":"2023-11-24T08:07:40.460769+0000","last_unstale":"2023-11-24T08:07:42.843505+0000","last_undegraded":"2023-11-24T08:07:42.843505+0000","last_fullsized":"2023-11-24T08:07:42.843505+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:23.839151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842076+0000","last_change":"2023-11-24T08:07:41.459000+0000","last_active":"2023-11-24T08:07:42.842076+0000","last_peered":"2023-11-24T08:07:42.842076+0000","last_clean":"2023-11-24T08:07:42.842076+0000","last_became_active":"2023-11-24T08:07:41.458775+0000","last_became_peered":"2023-11-24T08:07:41.458775+0000","last_unstale":"2023-11-24T08:07:42.842076+0000","last_undegraded":"2023-11-24T08:07:42.842076+0000","last_fullsized":"2023-11-24T08:07:42.842076+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:50.248216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842709+0000","last_change":"2023-11-24T08:07:40.452291+0000","last_active":"2023-11-24T08:07:42.842709+0000","last_peered":"2023-11-24T08:07:42.842709+0000","last_clean":"2023-11-24T08:07:42.842709+0000","last_became_active":"2023-11-24T08:07:40.451953+0000","last_became_peered":"2023-11-24T08:07:40.451953+0000","last_unstale":"2023-11-24T08:07:42.842709+0000","last_undegraded":"2023-11-24T08:07:42.842709+0000","last_fullsized":"2023-11-24T08:07:42.842709+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:41.762383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462103+0000","last_change":"2023-11-24T08:07:41.465249+0000","last_active":"2023-11-24T08:07:42.462103+0000","last_peered":"2023-11-24T08:07:42.462103+0000","last_clean":"2023-11-24T08:07:42.462103+0000","last_became_active":"2023-11-24T08:07:41.464937+0000","last_became_peered":"2023-11-24T08:07:41.464937+0000","last_unstale":"2023-11-24T08:07:42.462103+0000","last_undegraded":"2023-11-24T08:07:42.462103+0000","last_fullsized":"2023-11-24T08:07:42.462103+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:16.113678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840375+0000","last_change":"2023-11-24T08:07:40.460390+0000","last_active":"2023-11-24T08:07:42.840375+0000","last_peered":"2023-11-24T08:07:42.840375+0000","last_clean":"2023-11-24T08:07:42.840375+0000","last_became_active":"2023-11-24T08:07:40.460111+0000","last_became_peered":"2023-11-24T08:07:40.460111+0000","last_unstale":"2023-11-24T08:07:42.840375+0000","last_undegraded":"2023-11-24T08:07:42.840375+0000","last_fullsized":"2023-11-24T08:07:42.840375+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:39.328848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840854+0000","last_change":"2023-11-24T08:07:41.461207+0000","last_active":"2023-11-24T08:07:42.840854+0000","last_peered":"2023-11-24T08:07:42.840854+0000","last_clean":"2023-11-24T08:07:42.840854+0000","last_became_active":"2023-11-24T08:07:41.460892+0000","last_became_peered":"2023-11-24T08:07:41.460892+0000","last_unstale":"2023-11-24T08:07:42.840854+0000","last_undegraded":"2023-11-24T08:07:42.840854+0000","last_fullsized":"2023-11-24T08:07:42.840854+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:31.506786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840019+0000","last_change":"2023-11-24T08:07:40.462075+0000","last_active":"2023-11-24T08:07:42.840019+0000","last_peered":"2023-11-24T08:07:42.840019+0000","last_clean":"2023-11-24T08:07:42.840019+0000","last_became_active":"2023-11-24T08:07:40.461523+0000","last_became_peered":"2023-11-24T08:07:40.461523+0000","last_unstale":"2023-11-24T08:07:42.840019+0000","last_undegraded":"2023-11-24T08:07:42.840019+0000","last_fullsized":"2023-11-24T08:07:42.840019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:19.431553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840233+0000","last_change":"2023-11-24T08:07:41.448976+0000","last_active":"2023-11-24T08:07:42.840233+0000","last_peered":"2023-11-24T08:07:42.840233+0000","last_clean":"2023-11-24T08:07:42.840233+0000","last_became_active":"2023-11-24T08:07:41.448718+0000","last_became_peered":"2023-11-24T08:07:41.448718+0000","last_unstale":"2023-11-24T08:07:42.840233+0000","last_undegraded":"2023-11-24T08:07:42.840233+0000","last_fullsized":"2023-11-24T08:07:42.840233+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:56.530194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841633+0000","last_change":"2023-11-24T08:07:40.459754+0000","last_active":"2023-11-24T08:07:42.841633+0000","last_peered":"2023-11-24T08:07:42.841633+0000","last_clean":"2023-11-24T08:07:42.841633+0000","last_became_active":"2023-11-24T08:07:40.459464+0000","last_became_peered":"2023-11-24T08:07:40.459464+0000","last_unstale":"2023-11-24T08:07:42.841633+0000","last_undegraded":"2023-11-24T08:07:42.841633+0000","last_fullsized":"2023-11-24T08:07:42.841633+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.781281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.845119+0000","last_change":"2023-11-24T08:07:41.469563+0000","last_active":"2023-11-24T08:07:42.845119+0000","last_peered":"2023-11-24T08:07:42.845119+0000","last_clean":"2023-11-24T08:07:42.845119+0000","last_became_active":"2023-11-24T08:07:41.469298+0000","last_became_peered":"2023-11-24T08:07:41.469298+0000","last_unstale":"2023-11-24T08:07:42.845119+0000","last_undegraded":"2023-11-24T08:07:42.845119+0000","last_fullsized":"2023-11-24T08:07:42.845119+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:56.633861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841121+0000","last_change":"2023-11-24T08:07:40.454062+0000","last_active":"2023-11-24T08:07:42.841121+0000","last_peered":"2023-11-24T08:07:42.841121+0000","last_clean":"2023-11-24T08:07:42.841121+0000","last_became_active":"2023-11-24T08:07:40.453786+0000","last_became_peered":"2023-11-24T08:07:40.453786+0000","last_unstale":"2023-11-24T08:07:42.841121+0000","last_undegraded":"2023-11-24T08:07:42.841121+0000","last_fullsized":"2023-11-24T08:07:42.841121+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840530+0000","last_change":"2023-11-24T08:07:41.467546+0000","last_active":"2023-11-24T08:07:42.840530+0000","last_peered":"2023-11-24T08:07:42.840530+0000","last_clean":"2023-11-24T08:07:42.840530+0000","last_became_active":"2023-11-24T08:07:41.467111+0000","last_became_peered":"2023-11-24T08:07:41.467111+0000","last_unstale":"2023-11-24T08:07:42.840530+0000","last_undegraded":"2023-11-24T08:07:42.840530+0000","last_fullsized":"2023-11-24T08:07:42.840530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:35.559462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844990+0000","last_change":"2023-11-24T08:07:40.466462+0000","last_active":"2023-11-24T08:07:42.844990+0000","last_peered":"2023-11-24T08:07:42.844990+0000","last_clean":"2023-11-24T08:07:42.844990+0000","last_became_active":"2023-11-24T08:07:40.466058+0000","last_became_peered":"2023-11-24T08:07:40.466058+0000","last_unstale":"2023-11-24T08:07:42.844990+0000","last_undegraded":"2023-11-24T08:07:42.844990+0000","last_fullsized":"2023-11-24T08:07:42.844990+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:20.121426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844860+0000","last_change":"2023-11-24T08:07:41.462712+0000","last_active":"2023-11-24T08:07:42.844860+0000","last_peered":"2023-11-24T08:07:42.844860+0000","last_clean":"2023-11-24T08:07:42.844860+0000","last_became_active":"2023-11-24T08:07:41.461818+0000","last_became_peered":"2023-11-24T08:07:41.461818+0000","last_unstale":"2023-11-24T08:07:42.844860+0000","last_undegraded":"2023-11-24T08:07:42.844860+0000","last_fullsized":"2023-11-24T08:07:42.844860+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:31:27.057881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.461841+0000","last_change":"2023-11-24T08:07:40.466286+0000","last_active":"2023-11-24T08:07:42.461841+0000","last_peered":"2023-11-24T08:07:42.461841+0000","last_clean":"2023-11-24T08:07:42.461841+0000","last_became_active":"2023-11-24T08:07:40.465986+0000","last_became_peered":"2023-11-24T08:07:40.465986+0000","last_unstale":"2023-11-24T08:07:42.461841+0000","last_undegraded":"2023-11-24T08:07:42.461841+0000","last_fullsized":"2023-11-24T08:07:42.461841+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:28.013904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842422+0000","last_change":"2023-11-24T08:07:41.471798+0000","last_active":"2023-11-24T08:07:42.842422+0000","last_peered":"2023-11-24T08:07:42.842422+0000","last_clean":"2023-11-24T08:07:42.842422+0000","last_became_active":"2023-11-24T08:07:41.468503+0000","last_became_peered":"2023-11-24T08:07:41.468503+0000","last_unstale":"2023-11-24T08:07:42.842422+0000","last_undegraded":"2023-11-24T08:07:42.842422+0000","last_fullsized":"2023-11-24T08:07:42.842422+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:58.516481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844289+0000","last_change":"2023-11-24T08:07:40.464140+0000","last_active":"2023-11-24T08:07:42.844289+0000","last_peered":"2023-11-24T08:07:42.844289+0000","last_clean":"2023-11-24T08:07:42.844289+0000","last_became_active":"2023-11-24T08:07:40.463475+0000","last_became_peered":"2023-11-24T08:07:40.463475+0000","last_unstale":"2023-11-24T08:07:42.844289+0000","last_undegraded":"2023-11-24T08:07:42.844289+0000","last_fullsized":"2023-11-24T08:07:42.844289+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:30.021377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841913+0000","last_change":"2023-11-24T08:07:41.464299+0000","last_active":"2023-11-24T08:07:42.841913+0000","last_peered":"2023-11-24T08:07:42.841913+0000","last_clean":"2023-11-24T08:07:42.841913+0000","last_became_active":"2023-11-24T08:07:41.463957+0000","last_became_peered":"2023-11-24T08:07:41.463957+0000","last_unstale":"2023-11-24T08:07:42.841913+0000","last_undegraded":"2023-11-24T08:07:42.841913+0000","last_fullsized":"2023-11-24T08:07:42.841913+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:47.513484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466308+0000","last_change":"2023-11-24T08:07:40.458933+0000","last_active":"2023-11-24T08:07:42.466308+0000","last_peered":"2023-11-24T08:07:42.466308+0000","last_clean":"2023-11-24T08:07:42.466308+0000","last_became_active":"2023-11-24T08:07:40.458662+0000","last_became_peered":"2023-11-24T08:07:40.458662+0000","last_unstale":"2023-11-24T08:07:42.466308+0000","last_undegraded":"2023-11-24T08:07:42.466308+0000","last_fullsized":"2023-11-24T08:07:42.466308+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:55.617777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840826+0000","last_change":"2023-11-24T08:07:41.462407+0000","last_active":"2023-11-24T08:07:42.840826+0000","last_peered":"2023-11-24T08:07:42.840826+0000","last_clean":"2023-11-24T08:07:42.840826+0000","last_became_active":"2023-11-24T08:07:41.462053+0000","last_became_peered":"2023-11-24T08:07:41.462053+0000","last_unstale":"2023-11-24T08:07:42.840826+0000","last_undegraded":"2023-11-24T08:07:42.840826+0000","last_fullsized":"2023-11-24T08:07:42.840826+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:15:02.354111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.461888+0000","last_change":"2023-11-24T08:07:40.458746+0000","last_active":"2023-11-24T08:07:42.461888+0000","last_peered":"2023-11-24T08:07:42.461888+0000","last_clean":"2023-11-24T08:07:42.461888+0000","last_became_active":"2023-11-24T08:07:40.458464+0000","last_became_peered":"2023-11-24T08:07:40.458464+0000","last_unstale":"2023-11-24T08:07:42.461888+0000","last_undegraded":"2023-11-24T08:07:42.461888+0000","last_fullsized":"2023-11-24T08:07:42.461888+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:51.876418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.465983+0000","last_change":"2023-11-24T08:07:41.456254+0000","last_active":"2023-11-24T08:07:42.465983+0000","last_peered":"2023-11-24T08:07:42.465983+0000","last_clean":"2023-11-24T08:07:42.465983+0000","last_became_active":"2023-11-24T08:07:41.455935+0000","last_became_peered":"2023-11-24T08:07:41.455935+0000","last_unstale":"2023-11-24T08:07:42.465983+0000","last_undegraded":"2023-11-24T08:07:42.465983+0000","last_fullsized":"2023-11-24T08:07:42.465983+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:19.186521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841245+0000","last_change":"2023-11-24T08:07:40.452981+0000","last_active":"2023-11-24T08:07:42.841245+0000","last_peered":"2023-11-24T08:07:42.841245+0000","last_clean":"2023-11-24T08:07:42.841245+0000","last_became_active":"2023-11-24T08:07:40.452073+0000","last_became_peered":"2023-11-24T08:07:40.452073+0000","last_unstale":"2023-11-24T08:07:42.841245+0000","last_undegraded":"2023-11-24T08:07:42.841245+0000","last_fullsized":"2023-11-24T08:07:42.841245+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:03:35.608845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842661+0000","last_change":"2023-11-24T08:07:41.465647+0000","last_active":"2023-11-24T08:07:42.842661+0000","last_peered":"2023-11-24T08:07:42.842661+0000","last_clean":"2023-11-24T08:07:42.842661+0000","last_became_active":"2023-11-24T08:07:41.465381+0000","last_became_peered":"2023-11-24T08:07:41.465381+0000","last_unstale":"2023-11-24T08:07:42.842661+0000","last_undegraded":"2023-11-24T08:07:42.842661+0000","last_fullsized":"2023-11-24T08:07:42.842661+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:31.037461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842398+0000","last_change":"2023-11-24T08:07:40.447806+0000","last_active":"2023-11-24T08:07:42.842398+0000","last_peered":"2023-11-24T08:07:42.842398+0000","last_clean":"2023-11-24T08:07:42.842398+0000","last_became_active":"2023-11-24T08:07:40.447411+0000","last_became_peered":"2023-11-24T08:07:40.447411+0000","last_unstale":"2023-11-24T08:07:42.842398+0000","last_undegraded":"2023-11-24T08:07:42.842398+0000","last_fullsized":"2023-11-24T08:07:42.842398+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.245227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-24T08:07:41.440774+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:07:41.440774+0000","last_peered":"2023-11-24T08:07:41.440774+0000","last_clean":"2023-11-24T08:07:41.440774+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:07:41.440774+0000","last_undegraded":"2023-11-24T08:07:41.440774+0000","last_fullsized":"2023-11-24T08:07:41.440774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1163264,"data_stored":823213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.014,"15min":0.98599999999999999},"min":{"1min":0.65300000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.68000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4470000000000001,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93500000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1479999999999999,"15min":1.1319999999999999},"min":{"1min":0.76000000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.105},{"interface":"front","average":{"1min":1.262,"5min":1.1879999999999999,"15min":1.1759999999999999},"min":{"1min":0.88700000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5389999999999999,"5min":1.639,"15min":1.639},"last":1.0549999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.2130000000000001,"15min":1.238},"min":{"1min":0.70799999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3140000000000001,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.278},{"interface":"front","average":{"1min":1.377,"5min":1.375,"15min":1.375},"min":{"1min":1.0549999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.776,"5min":4.633,"15min":4.633},"last":1.216}]},{"osd":3,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.1539999999999999,"15min":1.129},"min":{"1min":0.92000000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.361},"min":{"1min":1.1719999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.319}]},{"osd":4,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.333,"15min":1.3200000000000001},"min":{"1min":0.879,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":1.367},{"interface":"front","average":{"1min":1.337,"5min":1.3240000000000001,"15min":1.3220000000000001},"min":{"1min":1.077,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.3979999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.4630000000000001,"15min":1.4470000000000001},"min":{"1min":1.232,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.48},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4850000000000001,"15min":1.472},"min":{"1min":0.97899999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":1.5620000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.5429999999999999,"15min":1.526},"min":{"1min":1.2789999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.641},{"interface":"front","average":{"1min":1.587,"5min":1.45,"15min":1.427},"min":{"1min":1.212,"5min":1.03,"15min":1.03},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.736}]}]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":233427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.073,"15min":1.087},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.657},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1000000000000001,"15min":1.1140000000000001},"min":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.607,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.079}]},{"osd":1,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.28,"15min":1.3009999999999999},"min":{"1min":0.80700000000000005,"5min":0.752,"15min":0.752},"max":{"1min":1.512,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.3129999999999999,"15min":1.329},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.93600000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.1100000000000001,"15min":1.1299999999999999},"min":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.446,"5min":1.873,"15min":1.873},"last":1.393},{"interface":"front","average":{"1min":1.321,"5min":1.417,"15min":1.4330000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.1419999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4219999999999999,"15min":1.427},"min":{"1min":1,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1379999999999999,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.446},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.6659999999999999,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.6140000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.591,"15min":1.611},"min":{"1min":0.86299999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.595,"15min":1.599},"min":{"1min":1.157,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.585,"15min":1.611},"min":{"1min":0.91500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.004,"5min":2.629,"15min":2.629},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.629,"5min":1.734,"15min":1.752},"min":{"1min":1.1459999999999999,"5min":1.022,"15min":1.022},"max":{"1min":2.5430000000000001,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.7889999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.772,"15min":1.7889999999999999},"min":{"1min":1.0900000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.579},{"interface":"front","average":{"1min":1.623,"5min":1.72,"15min":1.736},"min":{"1min":1.1930000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.2269999999999999,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.718}]}]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":232825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.65100000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3660000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.022},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.198,"15min":1.2090000000000001},"min":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.4530000000000001,"5min":2.198,"15min":2.198},"last":0.65900000000000003}]},{"osd":1,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.417,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1399999999999999,"15min":1.1419999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2110000000000001,"15min":1.2190000000000001},"min":{"1min":0.81399999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.579,"5min":1.738,"15min":1.738},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.1830000000000001,"15min":1.1859999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6539999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.196,"15min":1.212},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.5369999999999999,"5min":1.611,"15min":1.611},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.357,"15min":1.3740000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.7230000000000001,"5min":1.99,"15min":1.99},"last":0.78400000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3460000000000001,"15min":1.3540000000000001},"min":{"1min":0.88300000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.819},"last":1.357},{"interface":"front","average":{"1min":1.466,"5min":1.577,"15min":1.595},"min":{"1min":1.1779999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.944,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.87,"5min":0.748,"15min":0.748},"max":{"1min":1.8680000000000001,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4530000000000001,"15min":1.4550000000000001},"min":{"1min":1.0009999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.8360000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.232}]},{"osd":7,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.623,"15min":1.653},"min":{"1min":1.0469999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.093,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.113},{"interface":"front","average":{"1min":1.54,"5min":1.619,"15min":1.6319999999999999},"min":{"1min":1.0860000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2,"5min":2,"15min":2},"last":1.3009999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":577536,"data_stored":234367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.153,"15min":1.179},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0960000000000001,"15min":1.1000000000000001},"min":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5369999999999999,"5min":2.222,"15min":2.222},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.141,"15min":1.159},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.446,"5min":1.627,"15min":1.627},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.179,"5min":1.1910000000000001,"15min":1.1930000000000001},"min":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.597,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.276}]},{"osd":2,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.175,"15min":1.2030000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.6100000000000001,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.3740000000000001,"15min":1.417},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7869999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.054}]},{"osd":3,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.452,"15min":1.478},"min":{"1min":0.83999999999999997,"5min":0.751,"15min":0.751},"max":{"1min":1.7629999999999999,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.149},{"interface":"front","average":{"1min":1.226,"5min":1.3380000000000001,"15min":1.357},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6870000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.5349999999999999},"min":{"1min":0.89800000000000002,"5min":0.878,"15min":0.878},"max":{"1min":1.833,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.417},{"interface":"front","average":{"1min":1.444,"5min":1.5760000000000001,"15min":1.5980000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7909999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.375}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.488,"15min":1.532},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5269999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":0.875},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.6759999999999999,"15min":1.7190000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.0190000000000001,"5min":3.391,"15min":3.391},"last":1.248}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.6579999999999999,"15min":1.6919999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0870000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.639,"15min":1.6839999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.1120000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":577536,"data_stored":233393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.77800000000000002,"15min":0.79200000000000004},"min":{"1min":0.39500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1419999999999999,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91200000000000003,"15min":0.90500000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":0.95599999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.85499999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.222,"5min":1.532,"15min":1.532},"last":1.224},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.93999999999999995,"15min":0.96399999999999997},"min":{"1min":0.45200000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3320000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.0189999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.98499999999999999,"15min":0.97799999999999998},"min":{"1min":0.47899999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.99199999999999999,"15min":1.016},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.302,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1379999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2809999999999999,"15min":1.27},"min":{"1min":0.90200000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":1.274},{"interface":"front","average":{"1min":1.081,"5min":1.325,"15min":1.3660000000000001},"min":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.593,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.397}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2669999999999999,"15min":1.2869999999999999},"min":{"1min":0.78200000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.647,"5min":2.48,"15min":2.48},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.345,"15min":1.345},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.3360000000000001,"15min":1.3680000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.8520000000000001,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.3640000000000001,"15min":1.3959999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7929999999999999,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.6719999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.4710000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":4.5060000000000002,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":1.337},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.482,"15min":1.53},"min":{"1min":0.78800000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6499999999999999,"5min":4.032,"15min":4.032},"last":1.823}]}]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":234389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.93300000000000005,"15min":0.95799999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.292,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.446},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.0249999999999999,"15min":1.0680000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3160000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.94199999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.1639999999999999,"15min":1.2090000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.2789999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.147,"15min":1.1910000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.508,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.1890000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.4490000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.2989999999999999,"15min":1.3460000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5609999999999999,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.079}]},{"osd":4,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.427,"15min":1.4610000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.639,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.431,"15min":1.448},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.996}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.482,"15min":1.5189999999999999},"min":{"1min":0.753,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.665,"5min":2.508,"15min":2.508},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.84599999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.794,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.5649999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.81599999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.2210000000000001,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.5369999999999999,"15min":1.581},"min":{"1min":0.80300000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.732,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.3839999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.5389999999999999,"15min":1.577},"min":{"1min":0.88400000000000001,"5min":0.755,"15min":0.755},"max":{"1min":1.766,"5min":2.859,"15min":2.859},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.516,"15min":1.548},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":3.2730000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.353}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":232825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.749,"15min":0.74299999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.2050000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67500000000000004,"15min":0.67100000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.01},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.93300000000000005,"15min":0.93400000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.431,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.175,"5min":1.464,"15min":1.464},"last":0.91900000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2230000000000001,"15min":1.2190000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6539999999999999,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.78300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.657,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.7629999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.175,"15min":1.165},"min":{"1min":0.80100000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.6359999999999999,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.657},{"interface":"front","average":{"1min":1.123,"5min":1.145,"15min":1.149},"min":{"1min":0.85799999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4730000000000001,"5min":3.117,"15min":3.117},"last":2.036}]},{"osd":6,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.181,"15min":1.1859999999999999},"min":{"1min":0.79100000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6220000000000001,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.921},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.157},"min":{"1min":0.76600000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6100000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.5289999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.2789999999999999},"min":{"1min":0.746,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.748,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.357},{"interface":"front","average":{"1min":1.224,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.82899999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.7010000000000001,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":2.149}]}]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1163264,"data_stored":823157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.89900000000000002,"15min":0.93100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.107,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.038,"15min":1.0740000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.484,"5min":1.885,"15min":1.885},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.0680000000000001,"15min":1.0860000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.847,"5min":1.865,"15min":1.865},"last":1.149},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.96499999999999997,"15min":0.98799999999999999},"min":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.6180000000000001,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2829999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.1419999999999999,"15min":1.165},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6619999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.879,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.681,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.212}]},{"osd":4,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4339999999999999,"15min":1.464},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.819,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.3999999999999999,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.427}]},{"osd":5,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3520000000000001,"15min":1.3839999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.744,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.52},{"interface":"front","average":{"1min":1.222,"5min":1.478,"15min":1.5209999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.651,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.591}]},{"osd":6,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.4710000000000001,"15min":1.5029999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7949999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.337},{"interface":"front","average":{"1min":1.268,"5min":1.484,"15min":1.52},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7749999999999999,"5min":2.738,"15min":2.738},"last":1.6419999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.5600000000000001,"15min":1.587},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.1509999999999998,"5min":2.669,"15min":2.669},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.556,"15min":1.587},"min":{"1min":0.999,"5min":0.875,"15min":0.875},"max":{"1min":1.9359999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:07:45.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:07:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.148+0000 7fd7c6051640 1 Processor -- start 2023-11-24T08:07:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.148+0000 7fd7c6051640 1 -- start start 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7c6051640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7c00b2250 0x7fd7c015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7c6051640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd7c00b2630 0x7fd7c00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7c6051640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c015ab80 con 0x7fd7c00b2aa0 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7c6051640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c015a990 con 0x7fd7c00b2630 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7c6051640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c015e680 con 0x7fd7c00b2250 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7b6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd7c00b2630 0x7fd7c00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7bffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd7c00b2aa0 0x7fd7c015b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33016/0 (socket says 172.21.15.107:33016) 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7bffff640 1 -- 172.21.15.107:0/3899121830 learned_addr learned my addr 172.21.15.107:0/3899121830 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7b6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd7c00b2630 0x7fd7c00ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42420/0 (socket says 172.21.15.107:42420) 2023-11-24T08:07:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.150+0000 7fd7bf7fe640 1 --2- 172.21.15.107:0/3899121830 >> v2:172.21.15.120:3300/0 conn(0x7fd7c00b2250 0x7fd7c015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.151+0000 7fd7b6ffd640 1 -- 172.21.15.107:0/3899121830 >> v2:172.21.15.120:3300/0 conn(0x7fd7c00b2250 msgr2=0x7fd7c015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.151+0000 7fd7b6ffd640 1 --2- 172.21.15.107:0/3899121830 >> v2:172.21.15.120:3300/0 conn(0x7fd7c00b2250 0x7fd7c015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.151+0000 7fd7b6ffd640 1 -- 172.21.15.107:0/3899121830 >> v1:172.21.15.107:6789/0 conn(0x7fd7c00b2aa0 legacy=0x7fd7c015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.151+0000 7fd7b6ffd640 1 -- 172.21.15.107:0/3899121830 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c015e960 con 0x7fd7c00b2630 2023-11-24T08:07:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.152+0000 7fd7b6ffd640 1 --2- 172.21.15.107:0/3899121830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2630 0x7fd7c00ccf30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0072a70 tx=0x7fd7a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb819bd566390fec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.152+0000 7fd7beffd640 1 -- 172.21.15.107:0/3899121830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8031830 con 0x7fd7c00b2630 2023-11-24T08:07:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.152+0000 7fd7beffd640 1 -- 172.21.15.107:0/3899121830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a8031990 con 0x7fd7c00b2630 2023-11-24T08:07:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.152+0000 7fd7beffd640 1 -- 172.21.15.107:0/3899121830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8031d20 con 0x7fd7c00b2630 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 -- 172.21.15.107:0/3899121830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2630 msgr2=0x7fd7c00ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 --2- 172.21.15.107:0/3899121830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2630 0x7fd7c00ccf30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0072a70 tx=0x7fd7a802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 -- 172.21.15.107:0/3899121830 shutdown_connections 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 --2- 172.21.15.107:0/3899121830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2630 0x7fd7c00ccf30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 --2- 172.21.15.107:0/3899121830 >> v2:172.21.15.120:3300/0 conn(0x7fd7c00b2250 0x7fd7c015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 -- 172.21.15.107:0/3899121830 >> 172.21.15.107:0/3899121830 conn(0x7fd7c006a920 msgr2=0x7fd7c006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 -- 172.21.15.107:0/3899121830 shutdown_connections 2023-11-24T08:07:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.153+0000 7fd7c6051640 1 -- 172.21.15.107:0/3899121830 wait complete. 2023-11-24T08:07:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.155+0000 7fd7c6051640 1 Processor -- start 2023-11-24T08:07:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.155+0000 7fd7c6051640 1 -- start start 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7c00b2630 0x7fd7c009ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7c00b2aa0 0x7fd7c00cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00b0710 con 0x7fd7c00b2250 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00af110 con 0x7fd7c00b2630 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7c6051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00af3f0 con 0x7fd7c00b2aa0 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42428/0 (socket says 172.21.15.107:42428) 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 -- 172.21.15.107:0/2793355985 learned_addr learned my addr 172.21.15.107:0/2793355985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7b6ffd640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7c00b2630 0x7fd7c009ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 -- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7c00b2aa0 msgr2=0x7fd7c00cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7c00b2aa0 0x7fd7c00cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 -- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7c00b2630 msgr2=0x7fd7c009ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.156+0000 7fd7bf7fe640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7c00b2630 0x7fd7c009ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7bf7fe640 1 -- 172.21.15.107:0/2793355985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8009400 con 0x7fd7c00b2250 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7bf7fe640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000d800 tx=0x7fd7b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b000b6c0 con 0x7fd7c00b2250 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b000b820 con 0x7fd7c00b2250 2023-11-24T08:07:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c00af6d0 con 0x7fd7c00b2250 2023-11-24T08:07:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b000bbb0 con 0x7fd7c00b2250 2023-11-24T08:07:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.157+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c009cfa0 con 0x7fd7c00b2250 2023-11-24T08:07:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.158+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c0067470 con 0x7fd7c00b2250 2023-11-24T08:07:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.159+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fd7b000bd10 con 0x7fd7c00b2250 2023-11-24T08:07:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.160+0000 7fd7bcff9640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 0x7fd7a006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.160+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7b0089f30 con 0x7fd7c00b2250 2023-11-24T08:07:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.160+0000 7fd7b6ffd640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 0x7fd7a006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.161+0000 7fd7b6ffd640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 0x7fd7a006d600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8036000 tx=0x7fd7a8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.164+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7b0054d20 con 0x7fd7c00b2250 2023-11-24T08:07:46.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.350+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7c0159b40 con 0x7fd7a006b140 2023-11-24T08:07:46.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.359+0000 7fd7bcff9640 1 -- 172.21.15.107:0/2793355985 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286881 (secure 0 0 0) 0x7fd7c0159b40 con 0x7fd7a006b140 2023-11-24T08:07:46.364 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:46.366 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:07:46.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 msgr2=0x7fd7a006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 0x7fd7a006d600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8036000 tx=0x7fd7a8008030 comp rx=0 tx=0).stop 2023-11-24T08:07:46.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 msgr2=0x7fd7c015d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000d800 tx=0x7fd7b000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 shutdown_connections 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7a006b140 0x7fd7a006d600 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7c00b2aa0 0x7fd7c00cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7c00b2630 0x7fd7c009ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 --2- 172.21.15.107:0/2793355985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7c00b2250 0x7fd7c015d690 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 >> 172.21.15.107:0/2793355985 conn(0x7fd7c006a920 msgr2=0x7fd7c00cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 shutdown_connections 2023-11-24T08:07:46.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.365+0000 7fd7c6051640 1 -- 172.21.15.107:0/2793355985 wait complete. 2023-11-24T08:07:46.384 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2023-11-24T08:07:45.483773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242640,"kb_used_data":5640,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754732080,"statfs":{"total":773094113280,"available":772845649920,"internally_reserved":0,"allocated":5775360,"data_stored":3047596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":242535998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001139"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844874+0000","last_change":"2023-11-24T08:07:41.473827+0000","last_active":"2023-11-24T08:07:42.844874+0000","last_peered":"2023-11-24T08:07:42.844874+0000","last_clean":"2023-11-24T08:07:42.844874+0000","last_became_active":"2023-11-24T08:07:41.473511+0000","last_became_peered":"2023-11-24T08:07:41.473511+0000","last_unstale":"2023-11-24T08:07:42.844874+0000","last_undegraded":"2023-11-24T08:07:42.844874+0000","last_fullsized":"2023-11-24T08:07:42.844874+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:24.855235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841734+0000","last_change":"2023-11-24T08:07:41.479271+0000","last_active":"2023-11-24T08:07:42.841734+0000","last_peered":"2023-11-24T08:07:42.841734+0000","last_clean":"2023-11-24T08:07:42.841734+0000","last_became_active":"2023-11-24T08:07:41.478875+0000","last_became_peered":"2023-11-24T08:07:41.478875+0000","last_unstale":"2023-11-24T08:07:42.841734+0000","last_undegraded":"2023-11-24T08:07:42.841734+0000","last_fullsized":"2023-11-24T08:07:42.841734+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:39:44.765369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466243+0000","last_change":"2023-11-24T08:07:41.460984+0000","last_active":"2023-11-24T08:07:42.466243+0000","last_peered":"2023-11-24T08:07:42.466243+0000","last_clean":"2023-11-24T08:07:42.466243+0000","last_became_active":"2023-11-24T08:07:41.460668+0000","last_became_peered":"2023-11-24T08:07:41.460668+0000","last_unstale":"2023-11-24T08:07:42.466243+0000","last_undegraded":"2023-11-24T08:07:42.466243+0000","last_fullsized":"2023-11-24T08:07:42.466243+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:50.147256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840390+0000","last_change":"2023-11-24T08:07:41.468619+0000","last_active":"2023-11-24T08:07:42.840390+0000","last_peered":"2023-11-24T08:07:42.840390+0000","last_clean":"2023-11-24T08:07:42.840390+0000","last_became_active":"2023-11-24T08:07:41.465772+0000","last_became_peered":"2023-11-24T08:07:41.465772+0000","last_unstale":"2023-11-24T08:07:42.840390+0000","last_undegraded":"2023-11-24T08:07:42.840390+0000","last_fullsized":"2023-11-24T08:07:42.840390+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:30.418133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840644+0000","last_change":"2023-11-24T08:07:41.475378+0000","last_active":"2023-11-24T08:07:42.840644+0000","last_peered":"2023-11-24T08:07:42.840644+0000","last_clean":"2023-11-24T08:07:42.840644+0000","last_became_active":"2023-11-24T08:07:41.473509+0000","last_became_peered":"2023-11-24T08:07:41.473509+0000","last_unstale":"2023-11-24T08:07:42.840644+0000","last_undegraded":"2023-11-24T08:07:42.840644+0000","last_fullsized":"2023-11-24T08:07:42.840644+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:19.994298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843441+0000","last_change":"2023-11-24T08:07:41.472221+0000","last_active":"2023-11-24T08:07:42.843441+0000","last_peered":"2023-11-24T08:07:42.843441+0000","last_clean":"2023-11-24T08:07:42.843441+0000","last_became_active":"2023-11-24T08:07:41.468190+0000","last_became_peered":"2023-11-24T08:07:41.468190+0000","last_unstale":"2023-11-24T08:07:42.843441+0000","last_undegraded":"2023-11-24T08:07:42.843441+0000","last_fullsized":"2023-11-24T08:07:42.843441+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:47:45.395075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842924+0000","last_change":"2023-11-24T08:07:41.463573+0000","last_active":"2023-11-24T08:07:42.842924+0000","last_peered":"2023-11-24T08:07:42.842924+0000","last_clean":"2023-11-24T08:07:42.842924+0000","last_became_active":"2023-11-24T08:07:41.462861+0000","last_became_peered":"2023-11-24T08:07:41.462861+0000","last_unstale":"2023-11-24T08:07:42.842924+0000","last_undegraded":"2023-11-24T08:07:42.842924+0000","last_fullsized":"2023-11-24T08:07:42.842924+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:18.275648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840210+0000","last_change":"2023-11-24T08:07:41.466948+0000","last_active":"2023-11-24T08:07:42.840210+0000","last_peered":"2023-11-24T08:07:42.840210+0000","last_clean":"2023-11-24T08:07:42.840210+0000","last_became_active":"2023-11-24T08:07:41.466319+0000","last_became_peered":"2023-11-24T08:07:41.466319+0000","last_unstale":"2023-11-24T08:07:42.840210+0000","last_undegraded":"2023-11-24T08:07:42.840210+0000","last_fullsized":"2023-11-24T08:07:42.840210+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:16.113678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842688+0000","last_change":"2023-11-24T08:07:41.452006+0000","last_active":"2023-11-24T08:07:42.842688+0000","last_peered":"2023-11-24T08:07:42.842688+0000","last_clean":"2023-11-24T08:07:42.842688+0000","last_became_active":"2023-11-24T08:07:41.451738+0000","last_became_peered":"2023-11-24T08:07:41.451738+0000","last_unstale":"2023-11-24T08:07:42.842688+0000","last_undegraded":"2023-11-24T08:07:42.842688+0000","last_fullsized":"2023-11-24T08:07:42.842688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:04.261312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842534+0000","last_change":"2023-11-24T08:07:41.467923+0000","last_active":"2023-11-24T08:07:42.842534+0000","last_peered":"2023-11-24T08:07:42.842534+0000","last_clean":"2023-11-24T08:07:42.842534+0000","last_became_active":"2023-11-24T08:07:41.467550+0000","last_became_peered":"2023-11-24T08:07:41.467550+0000","last_unstale":"2023-11-24T08:07:42.842534+0000","last_undegraded":"2023-11-24T08:07:42.842534+0000","last_fullsized":"2023-11-24T08:07:42.842534+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:40.344933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843074+0000","last_change":"2023-11-24T08:07:41.458849+0000","last_active":"2023-11-24T08:07:42.843074+0000","last_peered":"2023-11-24T08:07:42.843074+0000","last_clean":"2023-11-24T08:07:42.843074+0000","last_became_active":"2023-11-24T08:07:41.458575+0000","last_became_peered":"2023-11-24T08:07:41.458575+0000","last_unstale":"2023-11-24T08:07:42.843074+0000","last_undegraded":"2023-11-24T08:07:42.843074+0000","last_fullsized":"2023-11-24T08:07:42.843074+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:19.994298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841278+0000","last_change":"2023-11-24T08:07:41.465343+0000","last_active":"2023-11-24T08:07:42.841278+0000","last_peered":"2023-11-24T08:07:42.841278+0000","last_clean":"2023-11-24T08:07:42.841278+0000","last_became_active":"2023-11-24T08:07:41.465002+0000","last_became_peered":"2023-11-24T08:07:41.465002+0000","last_unstale":"2023-11-24T08:07:42.841278+0000","last_undegraded":"2023-11-24T08:07:42.841278+0000","last_fullsized":"2023-11-24T08:07:42.841278+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:54.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842220+0000","last_change":"2023-11-24T08:07:41.456134+0000","last_active":"2023-11-24T08:07:42.842220+0000","last_peered":"2023-11-24T08:07:42.842220+0000","last_clean":"2023-11-24T08:07:42.842220+0000","last_became_active":"2023-11-24T08:07:41.455846+0000","last_became_peered":"2023-11-24T08:07:41.455846+0000","last_unstale":"2023-11-24T08:07:42.842220+0000","last_undegraded":"2023-11-24T08:07:42.842220+0000","last_fullsized":"2023-11-24T08:07:42.842220+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:13.922862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840496+0000","last_change":"2023-11-24T08:07:41.462455+0000","last_active":"2023-11-24T08:07:42.840496+0000","last_peered":"2023-11-24T08:07:42.840496+0000","last_clean":"2023-11-24T08:07:42.840496+0000","last_became_active":"2023-11-24T08:07:41.462174+0000","last_became_peered":"2023-11-24T08:07:41.462174+0000","last_unstale":"2023-11-24T08:07:42.840496+0000","last_undegraded":"2023-11-24T08:07:42.840496+0000","last_fullsized":"2023-11-24T08:07:42.840496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:15.445883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840035+0000","last_change":"2023-11-24T08:07:41.469977+0000","last_active":"2023-11-24T08:07:42.840035+0000","last_peered":"2023-11-24T08:07:42.840035+0000","last_clean":"2023-11-24T08:07:42.840035+0000","last_became_active":"2023-11-24T08:07:41.469707+0000","last_became_peered":"2023-11-24T08:07:41.469707+0000","last_unstale":"2023-11-24T08:07:42.840035+0000","last_undegraded":"2023-11-24T08:07:42.840035+0000","last_fullsized":"2023-11-24T08:07:42.840035+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:00.242666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842186+0000","last_change":"2023-11-24T08:07:41.471930+0000","last_active":"2023-11-24T08:07:42.842186+0000","last_peered":"2023-11-24T08:07:42.842186+0000","last_clean":"2023-11-24T08:07:42.842186+0000","last_became_active":"2023-11-24T08:07:41.471586+0000","last_became_peered":"2023-11-24T08:07:41.471586+0000","last_unstale":"2023-11-24T08:07:42.842186+0000","last_undegraded":"2023-11-24T08:07:42.842186+0000","last_fullsized":"2023-11-24T08:07:42.842186+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:20:44.155085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466560+0000","last_change":"2023-11-24T08:07:41.472539+0000","last_active":"2023-11-24T08:07:42.466560+0000","last_peered":"2023-11-24T08:07:42.466560+0000","last_clean":"2023-11-24T08:07:42.466560+0000","last_became_active":"2023-11-24T08:07:41.468323+0000","last_became_peered":"2023-11-24T08:07:41.468323+0000","last_unstale":"2023-11-24T08:07:42.466560+0000","last_undegraded":"2023-11-24T08:07:42.466560+0000","last_fullsized":"2023-11-24T08:07:42.466560+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:34.933490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467410+0000","last_change":"2023-11-24T08:07:41.471253+0000","last_active":"2023-11-24T08:07:42.467410+0000","last_peered":"2023-11-24T08:07:42.467410+0000","last_clean":"2023-11-24T08:07:42.467410+0000","last_became_active":"2023-11-24T08:07:41.470223+0000","last_became_peered":"2023-11-24T08:07:41.470223+0000","last_unstale":"2023-11-24T08:07:42.467410+0000","last_undegraded":"2023-11-24T08:07:42.467410+0000","last_fullsized":"2023-11-24T08:07:42.467410+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:40.344933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844557+0000","last_change":"2023-11-24T08:07:41.463419+0000","last_active":"2023-11-24T08:07:42.844557+0000","last_peered":"2023-11-24T08:07:42.844557+0000","last_clean":"2023-11-24T08:07:42.844557+0000","last_became_active":"2023-11-24T08:07:41.463051+0000","last_became_peered":"2023-11-24T08:07:41.463051+0000","last_unstale":"2023-11-24T08:07:42.844557+0000","last_undegraded":"2023-11-24T08:07:42.844557+0000","last_fullsized":"2023-11-24T08:07:42.844557+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:36:58.335836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842693+0000","last_change":"2023-11-24T08:07:41.464711+0000","last_active":"2023-11-24T08:07:42.842693+0000","last_peered":"2023-11-24T08:07:42.842693+0000","last_clean":"2023-11-24T08:07:42.842693+0000","last_became_active":"2023-11-24T08:07:41.464409+0000","last_became_peered":"2023-11-24T08:07:41.464409+0000","last_unstale":"2023-11-24T08:07:42.842693+0000","last_undegraded":"2023-11-24T08:07:42.842693+0000","last_fullsized":"2023-11-24T08:07:42.842693+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:20:44.155085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841576+0000","last_change":"2023-11-24T08:07:41.470935+0000","last_active":"2023-11-24T08:07:42.841576+0000","last_peered":"2023-11-24T08:07:42.841576+0000","last_clean":"2023-11-24T08:07:42.841576+0000","last_became_active":"2023-11-24T08:07:41.470625+0000","last_became_peered":"2023-11-24T08:07:41.470625+0000","last_unstale":"2023-11-24T08:07:42.841576+0000","last_undegraded":"2023-11-24T08:07:42.841576+0000","last_fullsized":"2023-11-24T08:07:42.841576+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:12:23.483181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840510+0000","last_change":"2023-11-24T08:07:41.471906+0000","last_active":"2023-11-24T08:07:42.840510+0000","last_peered":"2023-11-24T08:07:42.840510+0000","last_clean":"2023-11-24T08:07:42.840510+0000","last_became_active":"2023-11-24T08:07:41.471640+0000","last_became_peered":"2023-11-24T08:07:41.471640+0000","last_unstale":"2023-11-24T08:07:42.840510+0000","last_undegraded":"2023-11-24T08:07:42.840510+0000","last_fullsized":"2023-11-24T08:07:42.840510+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:19.186521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843057+0000","last_change":"2023-11-24T08:07:41.472346+0000","last_active":"2023-11-24T08:07:42.843057+0000","last_peered":"2023-11-24T08:07:42.843057+0000","last_clean":"2023-11-24T08:07:42.843057+0000","last_became_active":"2023-11-24T08:07:41.469021+0000","last_became_peered":"2023-11-24T08:07:41.469021+0000","last_unstale":"2023-11-24T08:07:42.843057+0000","last_undegraded":"2023-11-24T08:07:42.843057+0000","last_fullsized":"2023-11-24T08:07:42.843057+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:16.982235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842564+0000","last_change":"2023-11-24T08:07:41.473146+0000","last_active":"2023-11-24T08:07:42.842564+0000","last_peered":"2023-11-24T08:07:42.842564+0000","last_clean":"2023-11-24T08:07:42.842564+0000","last_became_active":"2023-11-24T08:07:41.470291+0000","last_became_peered":"2023-11-24T08:07:41.470291+0000","last_unstale":"2023-11-24T08:07:42.842564+0000","last_undegraded":"2023-11-24T08:07:42.842564+0000","last_fullsized":"2023-11-24T08:07:42.842564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:14.246888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844813+0000","last_change":"2023-11-24T08:07:41.464066+0000","last_active":"2023-11-24T08:07:42.844813+0000","last_peered":"2023-11-24T08:07:42.844813+0000","last_clean":"2023-11-24T08:07:42.844813+0000","last_became_active":"2023-11-24T08:07:41.463233+0000","last_became_peered":"2023-11-24T08:07:41.463233+0000","last_unstale":"2023-11-24T08:07:42.844813+0000","last_undegraded":"2023-11-24T08:07:42.844813+0000","last_fullsized":"2023-11-24T08:07:42.844813+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:04.924461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843109+0000","last_change":"2023-11-24T08:07:41.456768+0000","last_active":"2023-11-24T08:07:42.843109+0000","last_peered":"2023-11-24T08:07:42.843109+0000","last_clean":"2023-11-24T08:07:42.843109+0000","last_became_active":"2023-11-24T08:07:41.455041+0000","last_became_peered":"2023-11-24T08:07:41.455041+0000","last_unstale":"2023-11-24T08:07:42.843109+0000","last_undegraded":"2023-11-24T08:07:42.843109+0000","last_fullsized":"2023-11-24T08:07:42.843109+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:14.246888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466618+0000","last_change":"2023-11-24T08:07:41.464201+0000","last_active":"2023-11-24T08:07:42.466618+0000","last_peered":"2023-11-24T08:07:42.466618+0000","last_clean":"2023-11-24T08:07:42.466618+0000","last_became_active":"2023-11-24T08:07:41.463925+0000","last_became_peered":"2023-11-24T08:07:41.463925+0000","last_unstale":"2023-11-24T08:07:42.466618+0000","last_undegraded":"2023-11-24T08:07:42.466618+0000","last_fullsized":"2023-11-24T08:07:42.466618+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:56.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842615+0000","last_change":"2023-11-24T08:07:41.464869+0000","last_active":"2023-11-24T08:07:42.842615+0000","last_peered":"2023-11-24T08:07:42.842615+0000","last_clean":"2023-11-24T08:07:42.842615+0000","last_became_active":"2023-11-24T08:07:41.464571+0000","last_became_peered":"2023-11-24T08:07:41.464571+0000","last_unstale":"2023-11-24T08:07:42.842615+0000","last_undegraded":"2023-11-24T08:07:42.842615+0000","last_fullsized":"2023-11-24T08:07:42.842615+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:13:53.448314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841896+0000","last_change":"2023-11-24T08:07:41.462844+0000","last_active":"2023-11-24T08:07:42.841896+0000","last_peered":"2023-11-24T08:07:42.841896+0000","last_clean":"2023-11-24T08:07:42.841896+0000","last_became_active":"2023-11-24T08:07:41.460100+0000","last_became_peered":"2023-11-24T08:07:41.460100+0000","last_unstale":"2023-11-24T08:07:42.841896+0000","last_undegraded":"2023-11-24T08:07:42.841896+0000","last_fullsized":"2023-11-24T08:07:42.841896+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:13.922862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.465968+0000","last_change":"2023-11-24T08:07:41.467096+0000","last_active":"2023-11-24T08:07:42.465968+0000","last_peered":"2023-11-24T08:07:42.465968+0000","last_clean":"2023-11-24T08:07:42.465968+0000","last_became_active":"2023-11-24T08:07:41.466837+0000","last_became_peered":"2023-11-24T08:07:41.466837+0000","last_unstale":"2023-11-24T08:07:42.465968+0000","last_undegraded":"2023-11-24T08:07:42.465968+0000","last_fullsized":"2023-11-24T08:07:42.465968+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:15.445883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.839686+0000","last_change":"2023-11-24T08:07:41.462683+0000","last_active":"2023-11-24T08:07:42.839686+0000","last_peered":"2023-11-24T08:07:42.839686+0000","last_clean":"2023-11-24T08:07:42.839686+0000","last_became_active":"2023-11-24T08:07:41.462319+0000","last_became_peered":"2023-11-24T08:07:41.462319+0000","last_unstale":"2023-11-24T08:07:42.839686+0000","last_undegraded":"2023-11-24T08:07:42.839686+0000","last_fullsized":"2023-11-24T08:07:42.839686+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:31.395873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841125+0000","last_change":"2023-11-24T08:07:41.460653+0000","last_active":"2023-11-24T08:07:42.841125+0000","last_peered":"2023-11-24T08:07:42.841125+0000","last_clean":"2023-11-24T08:07:42.841125+0000","last_became_active":"2023-11-24T08:07:41.459508+0000","last_became_peered":"2023-11-24T08:07:41.459508+0000","last_unstale":"2023-11-24T08:07:42.841125+0000","last_undegraded":"2023-11-24T08:07:42.841125+0000","last_fullsized":"2023-11-24T08:07:42.841125+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:55:07.078915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462208+0000","last_change":"2023-11-24T08:07:41.456459+0000","last_active":"2023-11-24T08:07:42.462208+0000","last_peered":"2023-11-24T08:07:42.462208+0000","last_clean":"2023-11-24T08:07:42.462208+0000","last_became_active":"2023-11-24T08:07:41.456185+0000","last_became_peered":"2023-11-24T08:07:41.456185+0000","last_unstale":"2023-11-24T08:07:42.462208+0000","last_undegraded":"2023-11-24T08:07:42.462208+0000","last_fullsized":"2023-11-24T08:07:42.462208+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:30.418133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843651+0000","last_change":"2023-11-24T08:07:40.459884+0000","last_active":"2023-11-24T08:07:42.843651+0000","last_peered":"2023-11-24T08:07:42.843651+0000","last_clean":"2023-11-24T08:07:42.843651+0000","last_became_active":"2023-11-24T08:07:40.459629+0000","last_became_peered":"2023-11-24T08:07:40.459629+0000","last_unstale":"2023-11-24T08:07:42.843651+0000","last_undegraded":"2023-11-24T08:07:42.843651+0000","last_fullsized":"2023-11-24T08:07:42.843651+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:53.455094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844868+0000","last_change":"2023-11-24T08:07:41.476427+0000","last_active":"2023-11-24T08:07:42.844868+0000","last_peered":"2023-11-24T08:07:42.844868+0000","last_clean":"2023-11-24T08:07:42.844868+0000","last_became_active":"2023-11-24T08:07:41.476066+0000","last_became_peered":"2023-11-24T08:07:41.476066+0000","last_unstale":"2023-11-24T08:07:42.844868+0000","last_undegraded":"2023-11-24T08:07:42.844868+0000","last_fullsized":"2023-11-24T08:07:42.844868+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:29.029988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467676+0000","last_change":"2023-11-24T08:07:40.460230+0000","last_active":"2023-11-24T08:07:42.467676+0000","last_peered":"2023-11-24T08:07:42.467676+0000","last_clean":"2023-11-24T08:07:42.467676+0000","last_became_active":"2023-11-24T08:07:40.459906+0000","last_became_peered":"2023-11-24T08:07:40.459906+0000","last_unstale":"2023-11-24T08:07:42.467676+0000","last_undegraded":"2023-11-24T08:07:42.467676+0000","last_fullsized":"2023-11-24T08:07:42.467676+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:55.514109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842961+0000","last_change":"2023-11-24T08:07:41.475519+0000","last_active":"2023-11-24T08:07:42.842961+0000","last_peered":"2023-11-24T08:07:42.842961+0000","last_clean":"2023-11-24T08:07:42.842961+0000","last_became_active":"2023-11-24T08:07:41.475245+0000","last_became_peered":"2023-11-24T08:07:41.475245+0000","last_unstale":"2023-11-24T08:07:42.842961+0000","last_undegraded":"2023-11-24T08:07:42.842961+0000","last_fullsized":"2023-11-24T08:07:42.842961+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:21.137511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462351+0000","last_change":"2023-11-24T08:07:40.460426+0000","last_active":"2023-11-24T08:07:42.462351+0000","last_peered":"2023-11-24T08:07:42.462351+0000","last_clean":"2023-11-24T08:07:42.462351+0000","last_became_active":"2023-11-24T08:07:40.460191+0000","last_became_peered":"2023-11-24T08:07:40.460191+0000","last_unstale":"2023-11-24T08:07:42.462351+0000","last_undegraded":"2023-11-24T08:07:42.462351+0000","last_fullsized":"2023-11-24T08:07:42.462351+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:03.908377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462133+0000","last_change":"2023-11-24T08:07:41.464537+0000","last_active":"2023-11-24T08:07:42.462133+0000","last_peered":"2023-11-24T08:07:42.462133+0000","last_clean":"2023-11-24T08:07:42.462133+0000","last_became_active":"2023-11-24T08:07:41.464265+0000","last_became_peered":"2023-11-24T08:07:41.464265+0000","last_unstale":"2023-11-24T08:07:42.462133+0000","last_undegraded":"2023-11-24T08:07:42.462133+0000","last_fullsized":"2023-11-24T08:07:42.462133+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:56.530194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844279+0000","last_change":"2023-11-24T08:07:40.466384+0000","last_active":"2023-11-24T08:07:42.844279+0000","last_peered":"2023-11-24T08:07:42.844279+0000","last_clean":"2023-11-24T08:07:42.844279+0000","last_became_active":"2023-11-24T08:07:40.466070+0000","last_became_peered":"2023-11-24T08:07:40.466070+0000","last_unstale":"2023-11-24T08:07:42.844279+0000","last_undegraded":"2023-11-24T08:07:42.844279+0000","last_fullsized":"2023-11-24T08:07:42.844279+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:33:13.420060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466017+0000","last_change":"2023-11-24T08:07:41.464740+0000","last_active":"2023-11-24T08:07:42.466017+0000","last_peered":"2023-11-24T08:07:42.466017+0000","last_clean":"2023-11-24T08:07:42.466017+0000","last_became_active":"2023-11-24T08:07:41.464465+0000","last_became_peered":"2023-11-24T08:07:41.464465+0000","last_unstale":"2023-11-24T08:07:42.466017+0000","last_undegraded":"2023-11-24T08:07:42.466017+0000","last_fullsized":"2023-11-24T08:07:42.466017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:35.559462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842152+0000","last_change":"2023-11-24T08:07:40.457574+0000","last_active":"2023-11-24T08:07:42.842152+0000","last_peered":"2023-11-24T08:07:42.842152+0000","last_clean":"2023-11-24T08:07:42.842152+0000","last_became_active":"2023-11-24T08:07:40.457283+0000","last_became_peered":"2023-11-24T08:07:40.457283+0000","last_unstale":"2023-11-24T08:07:42.842152+0000","last_undegraded":"2023-11-24T08:07:42.842152+0000","last_fullsized":"2023-11-24T08:07:42.842152+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:53.955751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841127+0000","last_change":"2023-11-24T08:07:41.465589+0000","last_active":"2023-11-24T08:07:42.841127+0000","last_peered":"2023-11-24T08:07:42.841127+0000","last_clean":"2023-11-24T08:07:42.841127+0000","last_became_active":"2023-11-24T08:07:41.465308+0000","last_became_peered":"2023-11-24T08:07:41.465308+0000","last_unstale":"2023-11-24T08:07:42.841127+0000","last_undegraded":"2023-11-24T08:07:42.841127+0000","last_fullsized":"2023-11-24T08:07:42.841127+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:34.933490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466241+0000","last_change":"2023-11-24T08:07:40.459394+0000","last_active":"2023-11-24T08:07:42.466241+0000","last_peered":"2023-11-24T08:07:42.466241+0000","last_clean":"2023-11-24T08:07:42.466241+0000","last_became_active":"2023-11-24T08:07:40.457216+0000","last_became_peered":"2023-11-24T08:07:40.457216+0000","last_unstale":"2023-11-24T08:07:42.466241+0000","last_undegraded":"2023-11-24T08:07:42.466241+0000","last_fullsized":"2023-11-24T08:07:42.466241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:28.013904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840564+0000","last_change":"2023-11-24T08:07:41.467174+0000","last_active":"2023-11-24T08:07:42.840564+0000","last_peered":"2023-11-24T08:07:42.840564+0000","last_clean":"2023-11-24T08:07:42.840564+0000","last_became_active":"2023-11-24T08:07:41.466687+0000","last_became_peered":"2023-11-24T08:07:41.466687+0000","last_unstale":"2023-11-24T08:07:42.840564+0000","last_undegraded":"2023-11-24T08:07:42.840564+0000","last_fullsized":"2023-11-24T08:07:42.840564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:50.147256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840561+0000","last_change":"2023-11-24T08:07:40.458860+0000","last_active":"2023-11-24T08:07:42.840561+0000","last_peered":"2023-11-24T08:07:42.840561+0000","last_clean":"2023-11-24T08:07:42.840561+0000","last_became_active":"2023-11-24T08:07:40.457979+0000","last_became_peered":"2023-11-24T08:07:40.457979+0000","last_unstale":"2023-11-24T08:07:42.840561+0000","last_undegraded":"2023-11-24T08:07:42.840561+0000","last_fullsized":"2023-11-24T08:07:42.840561+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.781281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842923+0000","last_change":"2023-11-24T08:07:41.472828+0000","last_active":"2023-11-24T08:07:42.842923+0000","last_peered":"2023-11-24T08:07:42.842923+0000","last_clean":"2023-11-24T08:07:42.842923+0000","last_became_active":"2023-11-24T08:07:41.472547+0000","last_became_peered":"2023-11-24T08:07:41.472547+0000","last_unstale":"2023-11-24T08:07:42.842923+0000","last_undegraded":"2023-11-24T08:07:42.842923+0000","last_fullsized":"2023-11-24T08:07:42.842923+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:08:08.077023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840787+0000","last_change":"2023-11-24T08:07:40.453551+0000","last_active":"2023-11-24T08:07:42.840787+0000","last_peered":"2023-11-24T08:07:42.840787+0000","last_clean":"2023-11-24T08:07:42.840787+0000","last_became_active":"2023-11-24T08:07:40.453238+0000","last_became_peered":"2023-11-24T08:07:40.453238+0000","last_unstale":"2023-11-24T08:07:42.840787+0000","last_undegraded":"2023-11-24T08:07:42.840787+0000","last_fullsized":"2023-11-24T08:07:42.840787+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:48.178752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467608+0000","last_change":"2023-11-24T08:07:41.470675+0000","last_active":"2023-11-24T08:07:42.467608+0000","last_peered":"2023-11-24T08:07:42.467608+0000","last_clean":"2023-11-24T08:07:42.467608+0000","last_became_active":"2023-11-24T08:07:41.469620+0000","last_became_peered":"2023-11-24T08:07:41.469620+0000","last_unstale":"2023-11-24T08:07:42.467608+0000","last_undegraded":"2023-11-24T08:07:42.467608+0000","last_fullsized":"2023-11-24T08:07:42.467608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:31.037461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466774+0000","last_change":"2023-11-24T08:07:40.456573+0000","last_active":"2023-11-24T08:07:42.466774+0000","last_peered":"2023-11-24T08:07:42.466774+0000","last_clean":"2023-11-24T08:07:42.466774+0000","last_became_active":"2023-11-24T08:07:40.456259+0000","last_became_peered":"2023-11-24T08:07:40.456259+0000","last_unstale":"2023-11-24T08:07:42.466774+0000","last_undegraded":"2023-11-24T08:07:42.466774+0000","last_fullsized":"2023-11-24T08:07:42.466774+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:15.097594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.845067+0000","last_change":"2023-11-24T08:07:41.466779+0000","last_active":"2023-11-24T08:07:42.845067+0000","last_peered":"2023-11-24T08:07:42.845067+0000","last_clean":"2023-11-24T08:07:42.845067+0000","last_became_active":"2023-11-24T08:07:41.466433+0000","last_became_peered":"2023-11-24T08:07:41.466433+0000","last_unstale":"2023-11-24T08:07:42.845067+0000","last_undegraded":"2023-11-24T08:07:42.845067+0000","last_fullsized":"2023-11-24T08:07:42.845067+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:31.541468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842748+0000","last_change":"2023-11-24T08:07:40.460406+0000","last_active":"2023-11-24T08:07:42.842748+0000","last_peered":"2023-11-24T08:07:42.842748+0000","last_clean":"2023-11-24T08:07:42.842748+0000","last_became_active":"2023-11-24T08:07:40.459433+0000","last_became_peered":"2023-11-24T08:07:40.459433+0000","last_unstale":"2023-11-24T08:07:42.842748+0000","last_undegraded":"2023-11-24T08:07:42.842748+0000","last_fullsized":"2023-11-24T08:07:42.842748+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:55.617777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843305+0000","last_change":"2023-11-24T08:07:41.464595+0000","last_active":"2023-11-24T08:07:42.843305+0000","last_peered":"2023-11-24T08:07:42.843305+0000","last_clean":"2023-11-24T08:07:42.843305+0000","last_became_active":"2023-11-24T08:07:41.464303+0000","last_became_peered":"2023-11-24T08:07:41.464303+0000","last_unstale":"2023-11-24T08:07:42.843305+0000","last_undegraded":"2023-11-24T08:07:42.843305+0000","last_fullsized":"2023-11-24T08:07:42.843305+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:26.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844904+0000","last_change":"2023-11-24T08:07:40.465119+0000","last_active":"2023-11-24T08:07:42.844904+0000","last_peered":"2023-11-24T08:07:42.844904+0000","last_clean":"2023-11-24T08:07:42.844904+0000","last_became_active":"2023-11-24T08:07:40.464774+0000","last_became_peered":"2023-11-24T08:07:40.464774+0000","last_unstale":"2023-11-24T08:07:42.844904+0000","last_undegraded":"2023-11-24T08:07:42.844904+0000","last_fullsized":"2023-11-24T08:07:42.844904+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:13:52.432229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844653+0000","last_change":"2023-11-24T08:07:41.462187+0000","last_active":"2023-11-24T08:07:42.844653+0000","last_peered":"2023-11-24T08:07:42.844653+0000","last_clean":"2023-11-24T08:07:42.844653+0000","last_became_active":"2023-11-24T08:07:41.461922+0000","last_became_peered":"2023-11-24T08:07:41.461922+0000","last_unstale":"2023-11-24T08:07:42.844653+0000","last_undegraded":"2023-11-24T08:07:42.844653+0000","last_fullsized":"2023-11-24T08:07:42.844653+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:54.971836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":18,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843698+0000","last_change":"2023-11-24T08:07:40.453796+0000","last_active":"2023-11-24T08:07:42.843698+0000","last_peered":"2023-11-24T08:07:42.843698+0000","last_clean":"2023-11-24T08:07:42.843698+0000","last_became_active":"2023-11-24T08:07:40.453509+0000","last_became_peered":"2023-11-24T08:07:40.453509+0000","last_unstale":"2023-11-24T08:07:42.843698+0000","last_undegraded":"2023-11-24T08:07:42.843698+0000","last_fullsized":"2023-11-24T08:07:42.843698+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:19.431553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842690+0000","last_change":"2023-11-24T08:07:41.476259+0000","last_active":"2023-11-24T08:07:42.842690+0000","last_peered":"2023-11-24T08:07:42.842690+0000","last_clean":"2023-11-24T08:07:42.842690+0000","last_became_active":"2023-11-24T08:07:41.475274+0000","last_became_peered":"2023-11-24T08:07:41.475274+0000","last_unstale":"2023-11-24T08:07:42.842690+0000","last_undegraded":"2023-11-24T08:07:42.842690+0000","last_fullsized":"2023-11-24T08:07:42.842690+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:49.194836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842418+0000","last_change":"2023-11-24T08:07:40.454399+0000","last_active":"2023-11-24T08:07:42.842418+0000","last_peered":"2023-11-24T08:07:42.842418+0000","last_clean":"2023-11-24T08:07:42.842418+0000","last_became_active":"2023-11-24T08:07:40.453988+0000","last_became_peered":"2023-11-24T08:07:40.453988+0000","last_unstale":"2023-11-24T08:07:42.842418+0000","last_undegraded":"2023-11-24T08:07:42.842418+0000","last_fullsized":"2023-11-24T08:07:42.842418+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:27:22.254123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841365+0000","last_change":"2023-11-24T08:07:41.465201+0000","last_active":"2023-11-24T08:07:42.841365+0000","last_peered":"2023-11-24T08:07:42.841365+0000","last_clean":"2023-11-24T08:07:42.841365+0000","last_became_active":"2023-11-24T08:07:41.464932+0000","last_became_peered":"2023-11-24T08:07:41.464932+0000","last_unstale":"2023-11-24T08:07:42.841365+0000","last_undegraded":"2023-11-24T08:07:42.841365+0000","last_fullsized":"2023-11-24T08:07:42.841365+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:56.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843704+0000","last_change":"2023-11-24T08:07:40.461047+0000","last_active":"2023-11-24T08:07:42.843704+0000","last_peered":"2023-11-24T08:07:42.843704+0000","last_clean":"2023-11-24T08:07:42.843704+0000","last_became_active":"2023-11-24T08:07:40.460193+0000","last_became_peered":"2023-11-24T08:07:40.460193+0000","last_unstale":"2023-11-24T08:07:42.843704+0000","last_undegraded":"2023-11-24T08:07:42.843704+0000","last_fullsized":"2023-11-24T08:07:42.843704+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:33.917406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.839474+0000","last_change":"2023-11-24T08:07:41.454638+0000","last_active":"2023-11-24T08:07:42.839474+0000","last_peered":"2023-11-24T08:07:42.839474+0000","last_clean":"2023-11-24T08:07:42.839474+0000","last_became_active":"2023-11-24T08:07:41.454272+0000","last_became_peered":"2023-11-24T08:07:41.454272+0000","last_unstale":"2023-11-24T08:07:42.839474+0000","last_undegraded":"2023-11-24T08:07:42.839474+0000","last_fullsized":"2023-11-24T08:07:42.839474+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:47.782086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841676+0000","last_change":"2023-11-24T08:07:40.455492+0000","last_active":"2023-11-24T08:07:42.841676+0000","last_peered":"2023-11-24T08:07:42.841676+0000","last_clean":"2023-11-24T08:07:42.841676+0000","last_became_active":"2023-11-24T08:07:40.455215+0000","last_became_peered":"2023-11-24T08:07:40.455215+0000","last_unstale":"2023-11-24T08:07:42.841676+0000","last_undegraded":"2023-11-24T08:07:42.841676+0000","last_fullsized":"2023-11-24T08:07:42.841676+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:30.525384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843268+0000","last_change":"2023-11-24T08:07:41.463653+0000","last_active":"2023-11-24T08:07:42.843268+0000","last_peered":"2023-11-24T08:07:42.843268+0000","last_clean":"2023-11-24T08:07:42.843268+0000","last_became_active":"2023-11-24T08:07:41.463273+0000","last_became_peered":"2023-11-24T08:07:41.463273+0000","last_unstale":"2023-11-24T08:07:42.843268+0000","last_undegraded":"2023-11-24T08:07:42.843268+0000","last_fullsized":"2023-11-24T08:07:42.843268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:57.190796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":20,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841006+0000","last_change":"2023-11-24T08:07:40.462426+0000","last_active":"2023-11-24T08:07:42.841006+0000","last_peered":"2023-11-24T08:07:42.841006+0000","last_clean":"2023-11-24T08:07:42.841006+0000","last_became_active":"2023-11-24T08:07:40.461871+0000","last_became_peered":"2023-11-24T08:07:40.461871+0000","last_unstale":"2023-11-24T08:07:42.841006+0000","last_undegraded":"2023-11-24T08:07:42.841006+0000","last_fullsized":"2023-11-24T08:07:42.841006+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:08:07.060938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843492+0000","last_change":"2023-11-24T08:07:41.465724+0000","last_active":"2023-11-24T08:07:42.843492+0000","last_peered":"2023-11-24T08:07:42.843492+0000","last_clean":"2023-11-24T08:07:42.843492+0000","last_became_active":"2023-11-24T08:07:41.465477+0000","last_became_peered":"2023-11-24T08:07:41.465477+0000","last_unstale":"2023-11-24T08:07:42.843492+0000","last_undegraded":"2023-11-24T08:07:42.843492+0000","last_fullsized":"2023-11-24T08:07:42.843492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:49.194836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843161+0000","last_change":"2023-11-24T08:07:40.452058+0000","last_active":"2023-11-24T08:07:42.843161+0000","last_peered":"2023-11-24T08:07:42.843161+0000","last_clean":"2023-11-24T08:07:42.843161+0000","last_became_active":"2023-11-24T08:07:40.451765+0000","last_became_peered":"2023-11-24T08:07:40.451765+0000","last_unstale":"2023-11-24T08:07:42.843161+0000","last_undegraded":"2023-11-24T08:07:42.843161+0000","last_fullsized":"2023-11-24T08:07:42.843161+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:19:04.167570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843007+0000","last_change":"2023-11-24T08:07:41.458566+0000","last_active":"2023-11-24T08:07:42.843007+0000","last_peered":"2023-11-24T08:07:42.843007+0000","last_clean":"2023-11-24T08:07:42.843007+0000","last_became_active":"2023-11-24T08:07:41.458307+0000","last_became_peered":"2023-11-24T08:07:41.458307+0000","last_unstale":"2023-11-24T08:07:42.843007+0000","last_undegraded":"2023-11-24T08:07:42.843007+0000","last_fullsized":"2023-11-24T08:07:42.843007+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:33:14.436145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.467804+0000","last_change":"2023-11-24T08:07:40.457373+0000","last_active":"2023-11-24T08:07:42.467804+0000","last_peered":"2023-11-24T08:07:42.467804+0000","last_clean":"2023-11-24T08:07:42.467804+0000","last_became_active":"2023-11-24T08:07:40.457073+0000","last_became_peered":"2023-11-24T08:07:40.457073+0000","last_unstale":"2023-11-24T08:07:42.467804+0000","last_undegraded":"2023-11-24T08:07:42.467804+0000","last_fullsized":"2023-11-24T08:07:42.467804+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:29.402048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844691+0000","last_change":"2023-11-24T08:07:41.473723+0000","last_active":"2023-11-24T08:07:42.844691+0000","last_peered":"2023-11-24T08:07:42.844691+0000","last_clean":"2023-11-24T08:07:42.844691+0000","last_became_active":"2023-11-24T08:07:41.473434+0000","last_became_peered":"2023-11-24T08:07:41.473434+0000","last_unstale":"2023-11-24T08:07:42.844691+0000","last_undegraded":"2023-11-24T08:07:42.844691+0000","last_fullsized":"2023-11-24T08:07:42.844691+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:52.892503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841664+0000","last_change":"2023-11-24T08:07:40.460830+0000","last_active":"2023-11-24T08:07:42.841664+0000","last_peered":"2023-11-24T08:07:42.841664+0000","last_clean":"2023-11-24T08:07:42.841664+0000","last_became_active":"2023-11-24T08:07:40.460057+0000","last_became_peered":"2023-11-24T08:07:40.460057+0000","last_unstale":"2023-11-24T08:07:42.841664+0000","last_undegraded":"2023-11-24T08:07:42.841664+0000","last_fullsized":"2023-11-24T08:07:42.841664+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:30.021377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844277+0000","last_change":"2023-11-24T08:07:41.465842+0000","last_active":"2023-11-24T08:07:42.844277+0000","last_peered":"2023-11-24T08:07:42.844277+0000","last_clean":"2023-11-24T08:07:42.844277+0000","last_became_active":"2023-11-24T08:07:41.465408+0000","last_became_peered":"2023-11-24T08:07:41.465408+0000","last_unstale":"2023-11-24T08:07:42.844277+0000","last_undegraded":"2023-11-24T08:07:42.844277+0000","last_fullsized":"2023-11-24T08:07:42.844277+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:03:36.624929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.843505+0000","last_change":"2023-11-24T08:07:40.461173+0000","last_active":"2023-11-24T08:07:42.843505+0000","last_peered":"2023-11-24T08:07:42.843505+0000","last_clean":"2023-11-24T08:07:42.843505+0000","last_became_active":"2023-11-24T08:07:40.460769+0000","last_became_peered":"2023-11-24T08:07:40.460769+0000","last_unstale":"2023-11-24T08:07:42.843505+0000","last_undegraded":"2023-11-24T08:07:42.843505+0000","last_fullsized":"2023-11-24T08:07:42.843505+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:23.839151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842076+0000","last_change":"2023-11-24T08:07:41.459000+0000","last_active":"2023-11-24T08:07:42.842076+0000","last_peered":"2023-11-24T08:07:42.842076+0000","last_clean":"2023-11-24T08:07:42.842076+0000","last_became_active":"2023-11-24T08:07:41.458775+0000","last_became_peered":"2023-11-24T08:07:41.458775+0000","last_unstale":"2023-11-24T08:07:42.842076+0000","last_undegraded":"2023-11-24T08:07:42.842076+0000","last_fullsized":"2023-11-24T08:07:42.842076+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:50.248216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842709+0000","last_change":"2023-11-24T08:07:40.452291+0000","last_active":"2023-11-24T08:07:42.842709+0000","last_peered":"2023-11-24T08:07:42.842709+0000","last_clean":"2023-11-24T08:07:42.842709+0000","last_became_active":"2023-11-24T08:07:40.451953+0000","last_became_peered":"2023-11-24T08:07:40.451953+0000","last_unstale":"2023-11-24T08:07:42.842709+0000","last_undegraded":"2023-11-24T08:07:42.842709+0000","last_fullsized":"2023-11-24T08:07:42.842709+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:41.762383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.462103+0000","last_change":"2023-11-24T08:07:41.465249+0000","last_active":"2023-11-24T08:07:42.462103+0000","last_peered":"2023-11-24T08:07:42.462103+0000","last_clean":"2023-11-24T08:07:42.462103+0000","last_became_active":"2023-11-24T08:07:41.464937+0000","last_became_peered":"2023-11-24T08:07:41.464937+0000","last_unstale":"2023-11-24T08:07:42.462103+0000","last_undegraded":"2023-11-24T08:07:42.462103+0000","last_fullsized":"2023-11-24T08:07:42.462103+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:16.113678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840375+0000","last_change":"2023-11-24T08:07:40.460390+0000","last_active":"2023-11-24T08:07:42.840375+0000","last_peered":"2023-11-24T08:07:42.840375+0000","last_clean":"2023-11-24T08:07:42.840375+0000","last_became_active":"2023-11-24T08:07:40.460111+0000","last_became_peered":"2023-11-24T08:07:40.460111+0000","last_unstale":"2023-11-24T08:07:42.840375+0000","last_undegraded":"2023-11-24T08:07:42.840375+0000","last_fullsized":"2023-11-24T08:07:42.840375+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:39.328848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840854+0000","last_change":"2023-11-24T08:07:41.461207+0000","last_active":"2023-11-24T08:07:42.840854+0000","last_peered":"2023-11-24T08:07:42.840854+0000","last_clean":"2023-11-24T08:07:42.840854+0000","last_became_active":"2023-11-24T08:07:41.460892+0000","last_became_peered":"2023-11-24T08:07:41.460892+0000","last_unstale":"2023-11-24T08:07:42.840854+0000","last_undegraded":"2023-11-24T08:07:42.840854+0000","last_fullsized":"2023-11-24T08:07:42.840854+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:31.506786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840019+0000","last_change":"2023-11-24T08:07:40.462075+0000","last_active":"2023-11-24T08:07:42.840019+0000","last_peered":"2023-11-24T08:07:42.840019+0000","last_clean":"2023-11-24T08:07:42.840019+0000","last_became_active":"2023-11-24T08:07:40.461523+0000","last_became_peered":"2023-11-24T08:07:40.461523+0000","last_unstale":"2023-11-24T08:07:42.840019+0000","last_undegraded":"2023-11-24T08:07:42.840019+0000","last_fullsized":"2023-11-24T08:07:42.840019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:19.431553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840233+0000","last_change":"2023-11-24T08:07:41.448976+0000","last_active":"2023-11-24T08:07:42.840233+0000","last_peered":"2023-11-24T08:07:42.840233+0000","last_clean":"2023-11-24T08:07:42.840233+0000","last_became_active":"2023-11-24T08:07:41.448718+0000","last_became_peered":"2023-11-24T08:07:41.448718+0000","last_unstale":"2023-11-24T08:07:42.840233+0000","last_undegraded":"2023-11-24T08:07:42.840233+0000","last_fullsized":"2023-11-24T08:07:42.840233+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:56.530194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841633+0000","last_change":"2023-11-24T08:07:40.459754+0000","last_active":"2023-11-24T08:07:42.841633+0000","last_peered":"2023-11-24T08:07:42.841633+0000","last_clean":"2023-11-24T08:07:42.841633+0000","last_became_active":"2023-11-24T08:07:40.459464+0000","last_became_peered":"2023-11-24T08:07:40.459464+0000","last_unstale":"2023-11-24T08:07:42.841633+0000","last_undegraded":"2023-11-24T08:07:42.841633+0000","last_fullsized":"2023-11-24T08:07:42.841633+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.781281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.845119+0000","last_change":"2023-11-24T08:07:41.469563+0000","last_active":"2023-11-24T08:07:42.845119+0000","last_peered":"2023-11-24T08:07:42.845119+0000","last_clean":"2023-11-24T08:07:42.845119+0000","last_became_active":"2023-11-24T08:07:41.469298+0000","last_became_peered":"2023-11-24T08:07:41.469298+0000","last_unstale":"2023-11-24T08:07:42.845119+0000","last_undegraded":"2023-11-24T08:07:42.845119+0000","last_fullsized":"2023-11-24T08:07:42.845119+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:56.633861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841121+0000","last_change":"2023-11-24T08:07:40.454062+0000","last_active":"2023-11-24T08:07:42.841121+0000","last_peered":"2023-11-24T08:07:42.841121+0000","last_clean":"2023-11-24T08:07:42.841121+0000","last_became_active":"2023-11-24T08:07:40.453786+0000","last_became_peered":"2023-11-24T08:07:40.453786+0000","last_unstale":"2023-11-24T08:07:42.841121+0000","last_undegraded":"2023-11-24T08:07:42.841121+0000","last_fullsized":"2023-11-24T08:07:42.841121+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840530+0000","last_change":"2023-11-24T08:07:41.467546+0000","last_active":"2023-11-24T08:07:42.840530+0000","last_peered":"2023-11-24T08:07:42.840530+0000","last_clean":"2023-11-24T08:07:42.840530+0000","last_became_active":"2023-11-24T08:07:41.467111+0000","last_became_peered":"2023-11-24T08:07:41.467111+0000","last_unstale":"2023-11-24T08:07:42.840530+0000","last_undegraded":"2023-11-24T08:07:42.840530+0000","last_fullsized":"2023-11-24T08:07:42.840530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:35.559462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844990+0000","last_change":"2023-11-24T08:07:40.466462+0000","last_active":"2023-11-24T08:07:42.844990+0000","last_peered":"2023-11-24T08:07:42.844990+0000","last_clean":"2023-11-24T08:07:42.844990+0000","last_became_active":"2023-11-24T08:07:40.466058+0000","last_became_peered":"2023-11-24T08:07:40.466058+0000","last_unstale":"2023-11-24T08:07:42.844990+0000","last_undegraded":"2023-11-24T08:07:42.844990+0000","last_fullsized":"2023-11-24T08:07:42.844990+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:20.121426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844860+0000","last_change":"2023-11-24T08:07:41.462712+0000","last_active":"2023-11-24T08:07:42.844860+0000","last_peered":"2023-11-24T08:07:42.844860+0000","last_clean":"2023-11-24T08:07:42.844860+0000","last_became_active":"2023-11-24T08:07:41.461818+0000","last_became_peered":"2023-11-24T08:07:41.461818+0000","last_unstale":"2023-11-24T08:07:42.844860+0000","last_undegraded":"2023-11-24T08:07:42.844860+0000","last_fullsized":"2023-11-24T08:07:42.844860+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:31:27.057881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.461841+0000","last_change":"2023-11-24T08:07:40.466286+0000","last_active":"2023-11-24T08:07:42.461841+0000","last_peered":"2023-11-24T08:07:42.461841+0000","last_clean":"2023-11-24T08:07:42.461841+0000","last_became_active":"2023-11-24T08:07:40.465986+0000","last_became_peered":"2023-11-24T08:07:40.465986+0000","last_unstale":"2023-11-24T08:07:42.461841+0000","last_undegraded":"2023-11-24T08:07:42.461841+0000","last_fullsized":"2023-11-24T08:07:42.461841+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:28.013904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842422+0000","last_change":"2023-11-24T08:07:41.471798+0000","last_active":"2023-11-24T08:07:42.842422+0000","last_peered":"2023-11-24T08:07:42.842422+0000","last_clean":"2023-11-24T08:07:42.842422+0000","last_became_active":"2023-11-24T08:07:41.468503+0000","last_became_peered":"2023-11-24T08:07:41.468503+0000","last_unstale":"2023-11-24T08:07:42.842422+0000","last_undegraded":"2023-11-24T08:07:42.842422+0000","last_fullsized":"2023-11-24T08:07:42.842422+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:58.516481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.844289+0000","last_change":"2023-11-24T08:07:40.464140+0000","last_active":"2023-11-24T08:07:42.844289+0000","last_peered":"2023-11-24T08:07:42.844289+0000","last_clean":"2023-11-24T08:07:42.844289+0000","last_became_active":"2023-11-24T08:07:40.463475+0000","last_became_peered":"2023-11-24T08:07:40.463475+0000","last_unstale":"2023-11-24T08:07:42.844289+0000","last_undegraded":"2023-11-24T08:07:42.844289+0000","last_fullsized":"2023-11-24T08:07:42.844289+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:30.021377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841913+0000","last_change":"2023-11-24T08:07:41.464299+0000","last_active":"2023-11-24T08:07:42.841913+0000","last_peered":"2023-11-24T08:07:42.841913+0000","last_clean":"2023-11-24T08:07:42.841913+0000","last_became_active":"2023-11-24T08:07:41.463957+0000","last_became_peered":"2023-11-24T08:07:41.463957+0000","last_unstale":"2023-11-24T08:07:42.841913+0000","last_undegraded":"2023-11-24T08:07:42.841913+0000","last_fullsized":"2023-11-24T08:07:42.841913+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:47.513484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.466308+0000","last_change":"2023-11-24T08:07:40.458933+0000","last_active":"2023-11-24T08:07:42.466308+0000","last_peered":"2023-11-24T08:07:42.466308+0000","last_clean":"2023-11-24T08:07:42.466308+0000","last_became_active":"2023-11-24T08:07:40.458662+0000","last_became_peered":"2023-11-24T08:07:40.458662+0000","last_unstale":"2023-11-24T08:07:42.466308+0000","last_undegraded":"2023-11-24T08:07:42.466308+0000","last_fullsized":"2023-11-24T08:07:42.466308+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:55.617777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.840826+0000","last_change":"2023-11-24T08:07:41.462407+0000","last_active":"2023-11-24T08:07:42.840826+0000","last_peered":"2023-11-24T08:07:42.840826+0000","last_clean":"2023-11-24T08:07:42.840826+0000","last_became_active":"2023-11-24T08:07:41.462053+0000","last_became_peered":"2023-11-24T08:07:41.462053+0000","last_unstale":"2023-11-24T08:07:42.840826+0000","last_undegraded":"2023-11-24T08:07:42.840826+0000","last_fullsized":"2023-11-24T08:07:42.840826+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:15:02.354111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.461888+0000","last_change":"2023-11-24T08:07:40.458746+0000","last_active":"2023-11-24T08:07:42.461888+0000","last_peered":"2023-11-24T08:07:42.461888+0000","last_clean":"2023-11-24T08:07:42.461888+0000","last_became_active":"2023-11-24T08:07:40.458464+0000","last_became_peered":"2023-11-24T08:07:40.458464+0000","last_unstale":"2023-11-24T08:07:42.461888+0000","last_undegraded":"2023-11-24T08:07:42.461888+0000","last_fullsized":"2023-11-24T08:07:42.461888+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:51.876418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.465983+0000","last_change":"2023-11-24T08:07:41.456254+0000","last_active":"2023-11-24T08:07:42.465983+0000","last_peered":"2023-11-24T08:07:42.465983+0000","last_clean":"2023-11-24T08:07:42.465983+0000","last_became_active":"2023-11-24T08:07:41.455935+0000","last_became_peered":"2023-11-24T08:07:41.455935+0000","last_unstale":"2023-11-24T08:07:42.465983+0000","last_undegraded":"2023-11-24T08:07:42.465983+0000","last_fullsized":"2023-11-24T08:07:42.465983+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:19.186521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.841245+0000","last_change":"2023-11-24T08:07:40.452981+0000","last_active":"2023-11-24T08:07:42.841245+0000","last_peered":"2023-11-24T08:07:42.841245+0000","last_clean":"2023-11-24T08:07:42.841245+0000","last_became_active":"2023-11-24T08:07:40.452073+0000","last_became_peered":"2023-11-24T08:07:40.452073+0000","last_unstale":"2023-11-24T08:07:42.841245+0000","last_undegraded":"2023-11-24T08:07:42.841245+0000","last_fullsized":"2023-11-24T08:07:42.841245+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:03:35.608845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842661+0000","last_change":"2023-11-24T08:07:41.465647+0000","last_active":"2023-11-24T08:07:42.842661+0000","last_peered":"2023-11-24T08:07:42.842661+0000","last_clean":"2023-11-24T08:07:42.842661+0000","last_became_active":"2023-11-24T08:07:41.465381+0000","last_became_peered":"2023-11-24T08:07:41.465381+0000","last_unstale":"2023-11-24T08:07:42.842661+0000","last_undegraded":"2023-11-24T08:07:42.842661+0000","last_fullsized":"2023-11-24T08:07:42.842661+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:40.412583+0000","last_clean_scrub_stamp":"2023-11-24T08:07:40.412583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:31.037461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-24T08:07:42.842398+0000","last_change":"2023-11-24T08:07:40.447806+0000","last_active":"2023-11-24T08:07:42.842398+0000","last_peered":"2023-11-24T08:07:42.842398+0000","last_clean":"2023-11-24T08:07:42.842398+0000","last_became_active":"2023-11-24T08:07:40.447411+0000","last_became_peered":"2023-11-24T08:07:40.447411+0000","last_unstale":"2023-11-24T08:07:42.842398+0000","last_undegraded":"2023-11-24T08:07:42.842398+0000","last_fullsized":"2023-11-24T08:07:42.842398+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:07:39.396498+0000","last_clean_scrub_stamp":"2023-11-24T08:07:39.396498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.245227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-24T08:07:41.440774+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:07:41.440774+0000","last_peered":"2023-11-24T08:07:41.440774+0000","last_clean":"2023-11-24T08:07:41.440774+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:07:41.440774+0000","last_undegraded":"2023-11-24T08:07:41.440774+0000","last_fullsized":"2023-11-24T08:07:41.440774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1163264,"data_stored":823213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.014,"15min":0.98599999999999999},"min":{"1min":0.65300000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.68000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4470000000000001,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93500000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1479999999999999,"15min":1.1319999999999999},"min":{"1min":0.76000000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.105},{"interface":"front","average":{"1min":1.262,"5min":1.1879999999999999,"15min":1.1759999999999999},"min":{"1min":0.88700000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5389999999999999,"5min":1.639,"15min":1.639},"last":1.0549999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.2130000000000001,"15min":1.238},"min":{"1min":0.70799999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3140000000000001,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.278},{"interface":"front","average":{"1min":1.377,"5min":1.375,"15min":1.375},"min":{"1min":1.0549999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.776,"5min":4.633,"15min":4.633},"last":1.216}]},{"osd":3,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.1539999999999999,"15min":1.129},"min":{"1min":0.92000000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.361},"min":{"1min":1.1719999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.319}]},{"osd":4,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.333,"15min":1.3200000000000001},"min":{"1min":0.879,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":1.367},{"interface":"front","average":{"1min":1.337,"5min":1.3240000000000001,"15min":1.3220000000000001},"min":{"1min":1.077,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.3979999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.4630000000000001,"15min":1.4470000000000001},"min":{"1min":1.232,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.48},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4850000000000001,"15min":1.472},"min":{"1min":0.97899999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":1.5620000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:06:45 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.5429999999999999,"15min":1.526},"min":{"1min":1.2789999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.641},{"interface":"front","average":{"1min":1.587,"5min":1.45,"15min":1.427},"min":{"1min":1.212,"5min":1.03,"15min":1.03},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.736}]}]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":233427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.073,"15min":1.087},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.657},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1000000000000001,"15min":1.1140000000000001},"min":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.607,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.079}]},{"osd":1,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.28,"15min":1.3009999999999999},"min":{"1min":0.80700000000000005,"5min":0.752,"15min":0.752},"max":{"1min":1.512,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.3129999999999999,"15min":1.329},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.93600000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.1100000000000001,"15min":1.1299999999999999},"min":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.446,"5min":1.873,"15min":1.873},"last":1.393},{"interface":"front","average":{"1min":1.321,"5min":1.417,"15min":1.4330000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.1419999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4219999999999999,"15min":1.427},"min":{"1min":1,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.1379999999999999,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.446},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.6659999999999999,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.6140000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.591,"15min":1.611},"min":{"1min":0.86299999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.595,"15min":1.599},"min":{"1min":1.157,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.585,"15min":1.611},"min":{"1min":0.91500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.004,"5min":2.629,"15min":2.629},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.629,"5min":1.734,"15min":1.752},"min":{"1min":1.1459999999999999,"5min":1.022,"15min":1.022},"max":{"1min":2.5430000000000001,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.7889999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:52 2023","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.772,"15min":1.7889999999999999},"min":{"1min":1.0900000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.579},{"interface":"front","average":{"1min":1.623,"5min":1.72,"15min":1.736},"min":{"1min":1.1930000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.2269999999999999,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.718}]}]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":232825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.65100000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3660000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.022},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.198,"15min":1.2090000000000001},"min":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.4530000000000001,"5min":2.198,"15min":2.198},"last":0.65900000000000003}]},{"osd":1,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.417,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1399999999999999,"15min":1.1419999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2110000000000001,"15min":1.2190000000000001},"min":{"1min":0.81399999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.579,"5min":1.738,"15min":1.738},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.1830000000000001,"15min":1.1859999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6539999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.196,"15min":1.212},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.5369999999999999,"5min":1.611,"15min":1.611},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.357,"15min":1.3740000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.7230000000000001,"5min":1.99,"15min":1.99},"last":0.78400000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3460000000000001,"15min":1.3540000000000001},"min":{"1min":0.88300000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.819},"last":1.357},{"interface":"front","average":{"1min":1.466,"5min":1.577,"15min":1.595},"min":{"1min":1.1779999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.944,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.87,"5min":0.748,"15min":0.748},"max":{"1min":1.8680000000000001,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4530000000000001,"15min":1.4550000000000001},"min":{"1min":1.0009999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.8360000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.232}]},{"osd":7,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.623,"15min":1.653},"min":{"1min":1.0469999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.093,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.113},{"interface":"front","average":{"1min":1.54,"5min":1.619,"15min":1.6319999999999999},"min":{"1min":1.0860000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2,"5min":2,"15min":2},"last":1.3009999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":577536,"data_stored":234367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.153,"15min":1.179},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0960000000000001,"15min":1.1000000000000001},"min":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5369999999999999,"5min":2.222,"15min":2.222},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.141,"15min":1.159},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.446,"5min":1.627,"15min":1.627},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.179,"5min":1.1910000000000001,"15min":1.1930000000000001},"min":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.597,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.276}]},{"osd":2,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.175,"15min":1.2030000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.6100000000000001,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.3740000000000001,"15min":1.417},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7869999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.054}]},{"osd":3,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.452,"15min":1.478},"min":{"1min":0.83999999999999997,"5min":0.751,"15min":0.751},"max":{"1min":1.7629999999999999,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.149},{"interface":"front","average":{"1min":1.226,"5min":1.3380000000000001,"15min":1.357},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6870000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.5349999999999999},"min":{"1min":0.89800000000000002,"5min":0.878,"15min":0.878},"max":{"1min":1.833,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.417},{"interface":"front","average":{"1min":1.444,"5min":1.5760000000000001,"15min":1.5980000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7909999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.375}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.488,"15min":1.532},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5269999999999999,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":0.875},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.6759999999999999,"15min":1.7190000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.0190000000000001,"5min":3.391,"15min":3.391},"last":1.248}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.6579999999999999,"15min":1.6919999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0870000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.639,"15min":1.6839999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.1120000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":577536,"data_stored":233393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.77800000000000002,"15min":0.79200000000000004},"min":{"1min":0.39500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1419999999999999,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91200000000000003,"15min":0.90500000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":0.95599999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.85499999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.222,"5min":1.532,"15min":1.532},"last":1.224},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.93999999999999995,"15min":0.96399999999999997},"min":{"1min":0.45200000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3320000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.0189999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.98499999999999999,"15min":0.97799999999999998},"min":{"1min":0.47899999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.99199999999999999,"15min":1.016},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.302,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1379999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2809999999999999,"15min":1.27},"min":{"1min":0.90200000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":1.274},{"interface":"front","average":{"1min":1.081,"5min":1.325,"15min":1.3660000000000001},"min":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.593,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.397}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2669999999999999,"15min":1.2869999999999999},"min":{"1min":0.78200000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.647,"5min":2.48,"15min":2.48},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.345,"15min":1.345},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.3360000000000001,"15min":1.3680000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.8520000000000001,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.3640000000000001,"15min":1.3959999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7929999999999999,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.6719999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.4710000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":4.5060000000000002,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":1.337},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.482,"15min":1.53},"min":{"1min":0.78800000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6499999999999999,"5min":4.032,"15min":4.032},"last":1.823}]}]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":234389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.93300000000000005,"15min":0.95799999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.292,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.446},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.0249999999999999,"15min":1.0680000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3160000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.94199999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.1639999999999999,"15min":1.2090000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.2789999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.147,"15min":1.1910000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.508,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.1890000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.4490000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.2989999999999999,"15min":1.3460000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5609999999999999,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.079}]},{"osd":4,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.427,"15min":1.4610000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.639,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.431,"15min":1.448},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.996}]},{"osd":5,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.482,"15min":1.5189999999999999},"min":{"1min":0.753,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.665,"5min":2.508,"15min":2.508},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.84599999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.794,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.5649999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.81599999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.2210000000000001,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.5369999999999999,"15min":1.581},"min":{"1min":0.80300000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.732,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.3839999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.5389999999999999,"15min":1.577},"min":{"1min":0.88400000000000001,"5min":0.755,"15min":0.755},"max":{"1min":1.766,"5min":2.859,"15min":2.859},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.516,"15min":1.548},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":3.2730000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.353}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":232825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.749,"15min":0.74299999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.2050000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67500000000000004,"15min":0.67100000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.998,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.01},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.93300000000000005,"15min":0.93400000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.431,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.175,"5min":1.464,"15min":1.464},"last":0.91900000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2230000000000001,"15min":1.2190000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6539999999999999,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.78300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.657,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.7629999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.175,"15min":1.165},"min":{"1min":0.80100000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.6359999999999999,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.657},{"interface":"front","average":{"1min":1.123,"5min":1.145,"15min":1.149},"min":{"1min":0.85799999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4730000000000001,"5min":3.117,"15min":3.117},"last":2.036}]},{"osd":6,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.181,"15min":1.1859999999999999},"min":{"1min":0.79100000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6220000000000001,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.921},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.157},"min":{"1min":0.76600000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6100000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.5289999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.2789999999999999},"min":{"1min":0.746,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.748,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.357},{"interface":"front","average":{"1min":1.224,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.82899999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.7010000000000001,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":2.149}]}]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1163264,"data_stored":823157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.89900000000000002,"15min":0.93100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.107,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.038,"15min":1.0740000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.484,"5min":1.885,"15min":1.885},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.0680000000000001,"15min":1.0860000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.847,"5min":1.865,"15min":1.865},"last":1.149},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.96499999999999997,"15min":0.98799999999999999},"min":{"1min":0.54200000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.6180000000000001,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2829999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.1419999999999999,"15min":1.165},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6619999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.879,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.681,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.212}]},{"osd":4,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4339999999999999,"15min":1.464},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.819,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.3999999999999999,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.427}]},{"osd":5,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3520000000000001,"15min":1.3839999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.744,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.52},{"interface":"front","average":{"1min":1.222,"5min":1.478,"15min":1.5209999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.651,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.591}]},{"osd":6,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.4710000000000001,"15min":1.5029999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7949999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.337},{"interface":"front","average":{"1min":1.268,"5min":1.484,"15min":1.52},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7749999999999999,"5min":2.738,"15min":2.738},"last":1.6419999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:06:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.5600000000000001,"15min":1.587},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.1509999999999998,"5min":2.669,"15min":2.669},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.556,"15min":1.587},"min":{"1min":0.999,"5min":0.875,"15min":0.875},"max":{"1min":1.9359999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:07:46.390 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:07:46.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:07:46.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.649+0000 7fbfabb41640 1 Processor -- start 2023-11-24T08:07:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.650+0000 7fbfabb41640 1 -- start start 2023-11-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.651+0000 7fbfabb41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbfa415b440 0x7fbfa40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.651+0000 7fbfabb41640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa40c5dc0 con 0x7fbfa40cbf20 2023-11-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.651+0000 7fbfabb41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa415eea0 con 0x7fbfa40c59b0 2023-11-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.651+0000 7fbfabb41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40cc3d0 con 0x7fbfa415b440 2023-11-24T08:07:46.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa90dc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbfa40cbf20 0x7fbfa40c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33024/0 (socket says 172.21.15.107:33024) 2023-11-24T08:07:46.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa90dc640 1 -- 172.21.15.107:0/2500859628 learned_addr learned my addr 172.21.15.107:0/2500859628 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:46.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa98dd640 1 --2- 172.21.15.107:0/2500859628 >> v2:172.21.15.107:3300/0 conn(0x7fbfa415b440 0x7fbfa40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa88db640 1 -- 172.21.15.107:0/2500859628 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110244757 0 0) 0x7fbfa40c5dc0 con 0x7fbfa40cbf20 2023-11-24T08:07:46.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa88db640 1 -- 172.21.15.107:0/2500859628 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90003520 con 0x7fbfa40cbf20 2023-11-24T08:07:46.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa98dd640 1 -- 172.21.15.107:0/2500859628 >> v1:172.21.15.120:6789/0 conn(0x7fbfa40c59b0 legacy=0x7fbfa406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:46.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa98dd640 1 -- 172.21.15.107:0/2500859628 >> v1:172.21.15.107:6789/0 conn(0x7fbfa40cbf20 legacy=0x7fbfa40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.652+0000 7fbfa98dd640 1 -- 172.21.15.107:0/2500859628 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa40abf30 con 0x7fbfa415b440 2023-11-24T08:07:46.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.653+0000 7fbfa98dd640 1 --2- 172.21.15.107:0/2500859628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40cb9e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4eb074890acfb39 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfa88db640 1 -- 172.21.15.107:0/2500859628 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0031830 con 0x7fbfa415b440 2023-11-24T08:07:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfa88db640 1 -- 172.21.15.107:0/2500859628 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0031990 con 0x7fbfa415b440 2023-11-24T08:07:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfa88db640 1 -- 172.21.15.107:0/2500859628 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0031d20 con 0x7fbfa415b440 2023-11-24T08:07:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 -- 172.21.15.107:0/2500859628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 msgr2=0x7fbfa40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 --2- 172.21.15.107:0/2500859628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40cb9e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 -- 172.21.15.107:0/2500859628 shutdown_connections 2023-11-24T08:07:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 --2- 172.21.15.107:0/2500859628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40cb9e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 -- 172.21.15.107:0/2500859628 >> 172.21.15.107:0/2500859628 conn(0x7fbfa40b0c50 msgr2=0x7fbfa40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.654+0000 7fbfabb41640 1 -- 172.21.15.107:0/2500859628 shutdown_connections 2023-11-24T08:07:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.655+0000 7fbfabb41640 1 -- 172.21.15.107:0/2500859628 wait complete. 2023-11-24T08:07:46.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.656+0000 7fbfabb41640 1 Processor -- start 2023-11-24T08:07:46.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.656+0000 7fbfabb41640 1 -- start start 2023-11-24T08:07:46.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfa40c59b0 0x7fbfa4156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfa40cbf20 0x7fbfa40c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40cdd50 con 0x7fbfa415b440 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40cd0d0 con 0x7fbfa40cbf20 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.657+0000 7fbfabb41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40cd3b0 con 0x7fbfa40c59b0 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42448/0 (socket says 172.21.15.107:42448) 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 -- 172.21.15.107:0/163746127 learned_addr learned my addr 172.21.15.107:0/163746127 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfa98dd640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfa40c59b0 0x7fbfa4156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 -- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfa40c59b0 msgr2=0x7fbfa4156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfa40c59b0 0x7fbfa4156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 -- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfa40cbf20 msgr2=0x7fbfa40c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfa40cbf20 0x7fbfa40c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 -- 172.21.15.107:0/163746127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0009400 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbfaa0de640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c7a0 tx=0x7fbf9c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.658+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c003b70 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.659+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c003cf0 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.659+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c003ea0 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.659+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa40cd690 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.659+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa41570e0 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.660+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa40cde90 con 0x7fbfa415b440 2023-11-24T08:07:46.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.661+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fbf9c00aa90 con 0x7fbfa415b440 2023-11-24T08:07:46.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.662+0000 7fbf9a7fc640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 0x7fbf8806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:46.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.662+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf9c089840 con 0x7fbfa415b440 2023-11-24T08:07:46.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.662+0000 7fbfa98dd640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 0x7fbf8806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:46.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.663+0000 7fbfa98dd640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 0x7fbf8806d5d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.666+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbfa40cde90 con 0x7fbfa415b440 2023-11-24T08:07:46.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.921+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbfa40cde90 con 0x7fbfa415b440 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.925+0000 7fbf9a7fc640 1 -- 172.21.15.107:0/163746127 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbfa40cde90 con 0x7fbfa415b440 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:07:46.927 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:07:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:07:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 msgr2=0x7fbf8806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 0x7fbf8806d5d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa003c040 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 msgr2=0x7fbfa40afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c7a0 tx=0x7fbf9c00cc60 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 shutdown_connections 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbf8806b110 0x7fbf8806d5d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfa415b440 0x7fbfa40afaf0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfa40cbf20 0x7fbfa40c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 --2- 172.21.15.107:0/163746127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfa40c59b0 0x7fbfa4156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 >> 172.21.15.107:0/163746127 conn(0x7fbfa40b0c50 msgr2=0x7fbfa40acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 shutdown_connections 2023-11-24T08:07:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:46.928+0000 7fbfabb41640 1 -- 172.21.15.107:0/163746127 wait complete. 2023-11-24T08:07:46.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:07:47.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.150+0000 7f3ae18d0640 1 Processor -- start 2023-11-24T08:07:47.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.150+0000 7f3ae18d0640 1 -- start start 2023-11-24T08:07:47.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.151+0000 7f3ae18d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3adc15c7f0 0x7f3adc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.151+0000 7f3ae18d0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3adc0cf450 0x7f3adc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.151+0000 7f3ae18d0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc0cece0 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.151+0000 7f3ae18d0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc0cfe40 con 0x7f3adc15c7f0 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3ae18d0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc0ceaf0 con 0x7f3adc0cf450 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3adc15c7f0 0x7f3adc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3adc15c7f0 0x7f3adc06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42462/0 (socket says 172.21.15.107:42462) 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 -- 172.21.15.107:0/1913335370 learned_addr learned my addr 172.21.15.107:0/1913335370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 -- 172.21.15.107:0/1913335370 >> v2:172.21.15.120:3300/0 conn(0x7f3adc0cf450 msgr2=0x7f3adc0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 --2- 172.21.15.107:0/1913335370 >> v2:172.21.15.120:3300/0 conn(0x7f3adc0cf450 0x7f3adc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 -- 172.21.15.107:0/1913335370 >> v1:172.21.15.120:6789/0 conn(0x7f3adc0cdfb0 legacy=0x7f3adc0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:47.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.152+0000 7f3adaffd640 1 -- 172.21.15.107:0/1913335370 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc06d040 con 0x7f3adc15c7f0 2023-11-24T08:07:47.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.153+0000 7f3adaffd640 1 --2- 172.21.15.107:0/1913335370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc15c7f0 0x7f3adc06db20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3adc070410 tx=0x7f3acc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2d570f62abeed0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:47.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ad9ffb640 1 -- 172.21.15.107:0/1913335370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc02ca80 con 0x7f3adc15c7f0 2023-11-24T08:07:47.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ad9ffb640 1 -- 172.21.15.107:0/1913335370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc02cbe0 con 0x7f3adc15c7f0 2023-11-24T08:07:47.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ad9ffb640 1 -- 172.21.15.107:0/1913335370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0343f0 con 0x7f3adc15c7f0 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 -- 172.21.15.107:0/1913335370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc15c7f0 msgr2=0x7f3adc06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/1913335370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc15c7f0 0x7f3adc06db20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3adc070410 tx=0x7f3acc02bee0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 -- 172.21.15.107:0/1913335370 shutdown_connections 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/1913335370 >> v2:172.21.15.120:3300/0 conn(0x7f3adc0cf450 0x7f3adc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/1913335370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc15c7f0 0x7f3adc06db20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 -- 172.21.15.107:0/1913335370 >> 172.21.15.107:0/1913335370 conn(0x7f3adc06a920 msgr2=0x7f3adc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:47.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.154+0000 7f3ae18d0640 1 -- 172.21.15.107:0/1913335370 shutdown_connections 2023-11-24T08:07:47.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.155+0000 7f3ae18d0640 1 -- 172.21.15.107:0/1913335370 wait complete. 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.156+0000 7f3ae18d0640 1 Processor -- start 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.156+0000 7f3ae18d0640 1 -- start start 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3adc0cf450 0x7f3adc1f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3adc15c7f0 0x7f3adc158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15b210 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc1f1ec0 con 0x7f3adc15c7f0 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.157+0000 7f3ae18d0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15a590 con 0x7f3adc0cf450 2023-11-24T08:07:47.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42468/0 (socket says 172.21.15.107:42468) 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 -- 172.21.15.107:0/820174006 learned_addr learned my addr 172.21.15.107:0/820174006 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3ada7fc640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3adc0cf450 0x7f3adc1f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 -- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3adc0cf450 msgr2=0x7f3adc1f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3adc0cf450 0x7f3adc1f06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 -- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3adc15c7f0 msgr2=0x7f3adc158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3adc15c7f0 0x7f3adc158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 -- 172.21.15.107:0/820174006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc009b30 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.158+0000 7f3adaffd640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3acc036040 tx=0x7f3acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.159+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc009510 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.159+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc03e450 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.159+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc03e800 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.159+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc15a870 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.159+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc15ae30 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.160+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc15afc0 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.161+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f3acc02cd50 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.161+0000 7f3ad3fff640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 0x7f3abc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:47.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.162+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3acc0ac810 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.162+0000 7f3ada7fc640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 0x7f3abc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:47.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.162+0000 7f3ada7fc640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 0x7f3abc06d5d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3adc1f4450 tx=0x7f3ac4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.166+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3acc077500 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.397+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3adc067470 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.397+0000 7f3ad3fff640 1 -- 172.21.15.107:0/820174006 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f3acc07ab20 con 0x7f3adc0cdfb0 2023-11-24T08:07:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:07:47.399 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:07:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-11-24T08:07:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 msgr2=0x7f3abc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 0x7f3abc06d5d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3adc1f4450 tx=0x7f3ac4008040 comp rx=0 tx=0).stop 2023-11-24T08:07:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 msgr2=0x7f3adc1f1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3acc036040 tx=0x7f3acc004a40 comp rx=0 tx=0).stop 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 shutdown_connections 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3abc06b110 0x7f3abc06d5d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3adc15c7f0 0x7f3adc158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3adc0cf450 0x7f3adc1f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 --2- 172.21.15.107:0/820174006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3adc0cdfb0 0x7f3adc1f1980 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.400+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 >> 172.21.15.107:0/820174006 conn(0x7f3adc06a920 msgr2=0x7f3adc0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.401+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 shutdown_connections 2023-11-24T08:07:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:47.401+0000 7f3ae18d0640 1 -- 172.21.15.107:0/820174006 wait complete. 2023-11-24T08:07:47.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:07:40.945269+0000', 'modified': '2023-11-24T08:07:43.960687+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5661}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5661': {'gid': 5661, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6836/1158168362', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 1158168362}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 1158168362}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:07:47.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:07:47.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:07:48.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.044+0000 7f02e01d5640 1 Processor -- start 2023-11-24T08:07:48.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.044+0000 7f02e01d5640 1 -- start start 2023-11-24T08:07:48.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02e01d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02d80cc0f0 0x7f02d80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02e01d5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d815eea0 con 0x7f02d815b440 2023-11-24T08:07:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02e01d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80abf30 con 0x7f02d80c57e0 2023-11-24T08:07:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02e01d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d815b820 con 0x7f02d80cc0f0 2023-11-24T08:07:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02ddf71640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02d815b440 0x7f02d80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55318/0 (socket says 172.21.15.107:55318) 2023-11-24T08:07:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02ddf71640 1 -- 172.21.15.107:0/3464231380 learned_addr learned my addr 172.21.15.107:0/3464231380 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507249786 0 0) 0x7f02d815eea0 con 0x7f02d815b440 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c4003520 con 0x7f02d815b440 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.046+0000 7f02dd770640 1 --2- 172.21.15.107:0/3464231380 >> v2:172.21.15.120:3300/0 conn(0x7f02d80cc0f0 0x7f02d80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452117920 0 0) 0x7f02c4003520 con 0x7f02d815b440 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 >> v2:172.21.15.120:3300/0 conn(0x7f02d80cc0f0 msgr2=0x7f02d80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 --2- 172.21.15.107:0/3464231380 >> v2:172.21.15.120:3300/0 conn(0x7f02d80cc0f0 0x7f02d80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 >> v1:172.21.15.120:6789/0 conn(0x7f02d80c57e0 legacy=0x7f02d806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d815f0f0 con 0x7f02d815b440 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f02d4001670 con 0x7f02d815b440 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 >> v1:172.21.15.107:6789/0 conn(0x7f02d815b440 legacy=0x7f02d80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 0x7f02c4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40047c0 0x7f02c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02c4005250 0x7f02c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815b820 con 0x7f02c40039c0 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c400c1a0 con 0x7f02c4005250 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c400c360 con 0x7f02c40047c0 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.047+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 0x7f02c4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02dd770640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40047c0 0x7f02c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02ddf71640 1 -- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40047c0 msgr2=0x7f02c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40047c0 0x7f02c4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02ddf71640 1 -- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02c4005250 msgr2=0x7f02c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02c4005250 0x7f02c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02ddf71640 1 -- 172.21.15.107:0/3464231380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4003520 con 0x7f02c40039c0 2023-11-24T08:07:48.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.048+0000 7f02de772640 1 -- 172.21.15.107:0/3464231380 reap_dead start 2023-11-24T08:07:48.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 0x7f02c4003dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c190 tx=0x7f02c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f46b4a558e2a2ba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4005ad0 con 0x7f02c40039c0 2023-11-24T08:07:48.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4005c30 con 0x7f02c40039c0 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02dcf6f640 1 -- 172.21.15.107:0/3464231380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d402c810 con 0x7f02c40039c0 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02e01d5640 1 -- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 msgr2=0x7f02c4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02e01d5640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 0x7f02c4003dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c190 tx=0x7f02c40057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02e01d5640 1 -- 172.21.15.107:0/3464231380 shutdown_connections 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02e01d5640 1 --2- 172.21.15.107:0/3464231380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02c40039c0 0x7f02c4003dc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.049+0000 7f02e01d5640 1 -- 172.21.15.107:0/3464231380 >> 172.21.15.107:0/3464231380 conn(0x7f02d80b0c50 msgr2=0x7f02d80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.050+0000 7f02e01d5640 1 -- 172.21.15.107:0/3464231380 shutdown_connections 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.050+0000 7f02e01d5640 1 -- 172.21.15.107:0/3464231380 wait complete. 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.051+0000 7f02e01d5640 1 Processor -- start 2023-11-24T08:07:48.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.051+0000 7f02e01d5640 1 -- start start 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40039c0 0x7f02d8156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d815c7c0 0x7f02d815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815c330 con 0x7f02d80c71a0 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b3760 con 0x7f02d815c7c0 2023-11-24T08:07:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.052+0000 7f02e01d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815ba00 con 0x7f02c40039c0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33126/0 (socket says 172.21.15.107:33126) 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 -- 172.21.15.107:0/2047323797 learned_addr learned my addr 172.21.15.107:0/2047323797 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 -- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40039c0 msgr2=0x7f02d8156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40039c0 0x7f02d8156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 -- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d815c7c0 msgr2=0x7f02d815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d815c7c0 0x7f02d815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 -- 172.21.15.107:0/2047323797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d815bce0 con 0x7f02d80c71a0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02dd770640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e560 tx=0x7f02c800ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.053+0000 7f02ce7fc640 1 -- 172.21.15.107:0/2047323797 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8013610 con 0x7f02d80c71a0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.054+0000 7f02ce7fc640 1 -- 172.21.15.107:0/2047323797 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8013770 con 0x7f02d80c71a0 2023-11-24T08:07:48.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.054+0000 7f02ce7fc640 1 -- 172.21.15.107:0/2047323797 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8013b00 con 0x7f02d80c71a0 2023-11-24T08:07:48.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.054+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d815bfc0 con 0x7f02d80c71a0 2023-11-24T08:07:48.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.054+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d80b45c0 con 0x7f02d80c71a0 2023-11-24T08:07:48.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.056+0000 7f02ce7fc640 1 -- 172.21.15.107:0/2047323797 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f02c8013c60 con 0x7f02d80c71a0 2023-11-24T08:07:48.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.057+0000 7f02ce7fc640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 0x7f02bc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.057+0000 7f02ce7fc640 1 -- 172.21.15.107:0/2047323797 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02c8089760 con 0x7f02d80c71a0 2023-11-24T08:07:48.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.057+0000 7f02ddf71640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 0x7f02bc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.058+0000 7f02e2d6c740 1 Processor -- start 2023-11-24T08:07:48.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.058+0000 7f02ddf71640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 0x7f02bc06d600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f02d4034000 tx=0x7f02d4039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.058+0000 7f02e2d6c740 1 -- start start 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61dbba0 0x55d3e61e00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e6224560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e613f880 con 0x55d3e61e0ea0 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e614eae0 con 0x55d3e61e05e0 2023-11-24T08:07:48.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d3e614c1d0 con 0x55d3e61dbba0 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33134/0 (socket says 172.21.15.107:33134) 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 -- 172.21.15.107:0/511458167 learned_addr learned my addr 172.21.15.107:0/511458167 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 -- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61dbba0 msgr2=0x55d3e61e00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61dbba0 0x55d3e61e00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 -- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 msgr2=0x55d3e6224560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e6224560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.059+0000 7f02ddf71640 1 -- 172.21.15.107:0/511458167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3e610f860 con 0x55d3e61e0ea0 2023-11-24T08:07:48.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02ddf71640 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f02d4001c60 tx=0x7f02d4007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a53c53cba27005b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02bb7fe640 1 -- 172.21.15.107:0/511458167 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4048070 con 0x55d3e61e0ea0 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02bb7fe640 1 -- 172.21.15.107:0/511458167 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d402cca0 con 0x55d3e61e0ea0 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02bb7fe640 1 -- 172.21.15.107:0/511458167 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d403c040 con 0x55d3e61e0ea0 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02e2d6c740 1 -- 172.21.15.107:0/511458167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 msgr2=0x55d3e61e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.060+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f02d4001c60 tx=0x7f02d4007000 comp rx=0 tx=0).stop 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 -- 172.21.15.107:0/511458167 shutdown_connections 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e6224560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61dbba0 0x55d3e61e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/511458167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e61e1280 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 -- 172.21.15.107:0/511458167 >> 172.21.15.107:0/511458167 conn(0x55d3e61a2c70 msgr2=0x55d3e6226990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 -- 172.21.15.107:0/511458167 shutdown_connections 2023-11-24T08:07:48.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 -- 172.21.15.107:0/511458167 wait complete. 2023-11-24T08:07:48.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.061+0000 7f02e2d6c740 1 -- start start 2023-11-24T08:07:48.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.062+0000 7f02e2d6c740 1 Processor -- start 2023-11-24T08:07:48.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e61f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e0ea0 0x55d3e62092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e6147ac0 con 0x55d3e61dbba0 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e60e42f0 con 0x55d3e61e05e0 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d3e6101c70 con 0x55d3e61e0ea0 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02e2d6c740 10 client.0 _ll_register_callbacks cb 0x55d3e61063b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33144/0 (socket says 172.21.15.107:33144) 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 -- 172.21.15.107:0/3192968029 learned_addr learned my addr 172.21.15.107:0/3192968029 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 -- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e0ea0 msgr2=0x55d3e62092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e0ea0 0x55d3e62092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 -- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 msgr2=0x55d3e61f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e61f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.063+0000 7f02ddf71640 1 -- 172.21.15.107:0/3192968029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c4003520 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02ddf71640 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c2b0 tx=0x7f02d402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b00cc0476c942e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d403c030 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02d402ce10 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3e6140aa0 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02d404d660 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 10 client.5799 fetch_fsmap learned FSMap version 23 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55d3e6146140 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d3e6140aa0 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f02d404d7c0 con 0x55d3e61dbba0 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 10 client.5799 fetch_fsmap finished waiting for FSMap version 23 2023-11-24T08:07:48.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 10 client.5799 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 20 client.5799 populate_metadata read hostname 'smithi107' 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e62068a0 0x55d3e61f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 4 client.5799 mds_command: new command op to 5661 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.064+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 --> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3e61407c0 con 0x55d3e62068a0 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.065+0000 7f02dd770640 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e62068a0 0x55d3e61f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.065+0000 7f02dd770640 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e62068a0 0x55d3e61f1980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30e22bb95dc14590 server_cookie=b9efc7db7298259 in_seq=0 out_seq=0 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.065+0000 7f02b8ff9640 10 client.5799 ms_handle_connect on v2:172.21.15.107:6834/1158168362 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02b8ff9640 1 -- 172.21.15.107:0/3192968029 <== mds.0 v2:172.21.15.107:6834/1158168362 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d3e61407c0 con 0x55d3e62068a0 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02b8ff9640 10 client.5799 handle_command_reply: tid=0 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 client.5799 shutdown 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 msgr2=0x55d3e6223540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c2b0 tx=0x7f02d402e070 comp rx=0 tx=0).stop 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 shutdown_connections 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e62068a0 0x55d3e61f1980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e0ea0 0x55d3e62092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e05e0 0x55d3e61f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/3192968029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e6223540 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02b8ff9640 0 client.5799 ms_handle_reset on v2:172.21.15.107:6834/1158168362 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 >> 172.21.15.107:0/3192968029 conn(0x55d3e61a2c70 msgr2=0x55d3e6223af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 shutdown_connections 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.066+0000 7f02e2d6c740 1 -- 172.21.15.107:0/3192968029 wait complete. 2023-11-24T08:07:48.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.067+0000 7f02e2d6c740 20 client.5799 trim_cache size 0 max 16384 2023-11-24T08:07:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.083+0000 7f02e2d6c740 1 Processor -- start 2023-11-24T08:07:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.083+0000 7f02e2d6c740 1 -- start start 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61dbba0 0x55d3e6240110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e6240650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e5eaaf20 con 0x55d3e61e0ea0 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e622eee0 con 0x55d3e61dbba0 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d3e6209470 con 0x55d3e61e05e0 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02de772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02de772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33156/0 (socket says 172.21.15.107:33156) 2023-11-24T08:07:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02de772640 1 -- 172.21.15.107:0/46703350 learned_addr learned my addr 172.21.15.107:0/46703350 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.084+0000 7f02dd770640 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e6240650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 -- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 msgr2=0x55d3e6240650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e6240650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 -- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61dbba0 msgr2=0x55d3e6240110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61dbba0 0x55d3e6240110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 -- 172.21.15.107:0/46703350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3e60e3b80 con 0x55d3e61e0ea0 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02de772640 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f02d806dc00 tx=0x7f02d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03028c5d2cc8146 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02bbfff640 1 -- 172.21.15.107:0/46703350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0004850 con 0x55d3e61e0ea0 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02bbfff640 1 -- 172.21.15.107:0/46703350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d000c610 con 0x55d3e61e0ea0 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.085+0000 7f02bbfff640 1 -- 172.21.15.107:0/46703350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000c9c0 con 0x55d3e61e0ea0 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- 172.21.15.107:0/46703350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 msgr2=0x55d3e62077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f02d806dc00 tx=0x7f02d0004600 comp rx=0 tx=0).stop 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- 172.21.15.107:0/46703350 shutdown_connections 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61e0ea0 0x55d3e62077d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e6240650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/46703350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61dbba0 0x55d3e6240110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- 172.21.15.107:0/46703350 >> 172.21.15.107:0/46703350 conn(0x55d3e61a2c70 msgr2=0x55d3e61d72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- 172.21.15.107:0/46703350 shutdown_connections 2023-11-24T08:07:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- 172.21.15.107:0/46703350 wait complete. 2023-11-24T08:07:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.086+0000 7f02e2d6c740 1 -- start start 2023-11-24T08:07:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.087+0000 7f02e2d6c740 1 Processor -- start 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e61f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e0ea0 0x55d3e6290970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e61dbf80 con 0x55d3e61dbba0 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3e6207820 con 0x55d3e61e0ea0 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d3e6225570 con 0x55d3e61e05e0 2023-11-24T08:07:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02e2d6c740 10 client.0 _ll_register_callbacks cb 0x7f02c400c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33166/0 (socket says 172.21.15.107:33166) 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 -- 172.21.15.107:0/147601030 learned_addr learned my addr 172.21.15.107:0/147601030 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 -- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 msgr2=0x55d3e61f1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e61f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 -- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e0ea0 msgr2=0x55d3e6290970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e0ea0 0x55d3e6290970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.088+0000 7f02ddf71640 1 -- 172.21.15.107:0/147601030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d0003ea0 con 0x55d3e61dbba0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02ddf71640 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f02d4001560 tx=0x7f02d4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9574c931f6f1cfa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d403c070 con 0x55d3e61dbba0 2023-11-24T08:07:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02d402ce10 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02d402cca0 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02d4001b70 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 10 client.5805 fetch_fsmap learned FSMap version 23 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55d3e6109170 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d402cca0 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f02d4041860 con 0x55d3e61dbba0 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 10 client.5805 fetch_fsmap finished waiting for FSMap version 23 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 10 client.5805 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 20 client.5805 populate_metadata read hostname 'smithi107' 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e6290eb0 0x55d3e6291290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 4 client.5805 mds_command: new command op to 5661 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.089+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 --> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d3e5e486c0 con 0x55d3e6290eb0 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02dd770640 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e6290eb0 0x55d3e6291290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02dd770640 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e6290eb0 0x55d3e6291290 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82da2a15769b7c7b server_cookie=f24299239ba13965 in_seq=0 out_seq=0 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02b8ff9640 10 client.5805 ms_handle_connect on v2:172.21.15.107:6834/1158168362 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02b8ff9640 1 -- 172.21.15.107:0/147601030 <== mds.0 v2:172.21.15.107:6834/1158168362 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d3e5e486c0 con 0x55d3e6290eb0 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02b8ff9640 10 client.5805 handle_command_reply: tid=0 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02e2d6c740 1 client.5805 shutdown 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 msgr2=0x55d3e62223c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.090+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f02d4001560 tx=0x7f02d4001930 comp rx=0 tx=0).stop 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 shutdown_connections 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:6834/1158168362,v1:172.21.15.107:6836/1158168362] conn(0x55d3e6290eb0 0x55d3e6291290 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3e61e0ea0 0x55d3e6290970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d3e61e05e0 0x55d3e61f1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02b8ff9640 0 client.5805 ms_handle_reset on v2:172.21.15.107:6834/1158168362 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 --2- 172.21.15.107:0/147601030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3e61dbba0 0x55d3e62223c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 >> 172.21.15.107:0/147601030 conn(0x55d3e61a2c70 msgr2=0x55d3e61f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 shutdown_connections 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 1 -- 172.21.15.107:0/147601030 wait complete. 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e2d6c740 20 client.5805 trim_cache size 0 max 16384 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:07:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 5661, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6572566340000003, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 23, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 48, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 48, 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.037927760000001 2023-11-24T08:07:48.097 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 msgr2=0x7f02bc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 0x7f02bc06d600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f02d4034000 tx=0x7f02d4039040 comp rx=0 tx=0).stop 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 msgr2=0x7f02d80c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e560 tx=0x7f02c800ea20 comp rx=0 tx=0).stop 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 shutdown_connections 2023-11-24T08:07:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f02bc06b140 0x7f02bc06d600 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d815c7c0 0x7f02d815cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02d80c71a0 0x7f02d80c7620 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 --2- 172.21.15.107:0/2047323797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02c40039c0 0x7f02d8156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.091+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 >> 172.21.15.107:0/2047323797 conn(0x7f02d80b0c50 msgr2=0x7f02d80cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.092+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 shutdown_connections 2023-11-24T08:07:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.092+0000 7f02e01d5640 1 -- 172.21.15.107:0/2047323797 wait complete. 2023-11-24T08:07:48.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:07:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.317+0000 7f3a27541640 1 Processor -- start 2023-11-24T08:07:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.317+0000 7f3a27541640 1 -- start start 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a27541640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3a200cc640 0x7f3a200cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a27541640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a200b56f0 0x7f3a200ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a27541640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200b3160 con 0x7f3a200cc260 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a27541640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a200cea00 con 0x7f3a200cc640 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a27541640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200cece0 con 0x7f3a200b56f0 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a252dd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3a200cc260 0x7f3a200b4fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55326/0 (socket says 172.21.15.107:55326) 2023-11-24T08:07:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a252dd640 1 -- 172.21.15.107:0/2014640805 learned_addr learned my addr 172.21.15.107:0/2014640805 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.319+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2014640805 >> v2:172.21.15.120:3300/0 conn(0x7f3a200cc640 0x7f3a200cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a25ade640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a200b56f0 0x7f3a200ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693673537 0 0) 0x7f3a200b3160 con 0x7f3a200cc260 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c005730 con 0x7f3a200cc260 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536832728 0 0) 0x7f3a0c005730 con 0x7f3a200cc260 2023-11-24T08:07:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a200b56f0 msgr2=0x7f3a200ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a200b56f0 0x7f3a200ce1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 >> v2:172.21.15.120:3300/0 conn(0x7f3a200cc640 msgr2=0x7f3a200cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 --2- 172.21.15.107:0/2014640805 >> v2:172.21.15.120:3300/0 conn(0x7f3a200cc640 0x7f3a200cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200b33d0 con 0x7f3a200cc260 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3a1c001670 con 0x7f3a200cc260 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 >> v1:172.21.15.107:6789/0 conn(0x7f3a200cc260 legacy=0x7f3a200b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a0c005b80 0x7f3a0c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a0c006570 0x7f3a0c006a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c007170 0x7f3a0c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200cea00 con 0x7f3a0c006570 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200cece0 con 0x7f3a0c007170 2023-11-24T08:07:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c00e100 con 0x7f3a0c005b80 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a25ade640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a200b56f0 0x7f3a200ce1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.320+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a0c006570 0x7f3a0c006a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a252dd640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a0c005b80 0x7f3a0c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a24adc640 1 -- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a0c005b80 msgr2=0x7f3a0c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a0c005b80 0x7f3a0c005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a24adc640 1 -- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c007170 msgr2=0x7f3a0c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c007170 0x7f3a0c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.321+0000 7f3a24adc640 1 -- 172.21.15.107:0/2014640805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c005730 con 0x7f3a0c006570 2023-11-24T08:07:48.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.322+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a0c006570 0x7f3a0c006a50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009ac0 tx=0x7f3a0c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd86a611efeaaaec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.322+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1002c710 con 0x7f3a0c006570 2023-11-24T08:07:48.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.322+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1002c870 con 0x7f3a0c006570 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.322+0000 7f3a17fff640 1 -- 172.21.15.107:0/2014640805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1002cc00 con 0x7f3a0c006570 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 -- 172.21.15.107:0/2014640805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a0c006570 msgr2=0x7f3a0c006a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 --2- 172.21.15.107:0/2014640805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a0c006570 0x7f3a0c006a50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009ac0 tx=0x7f3a0c007a00 comp rx=0 tx=0).stop 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a25ade640 1 -- 172.21.15.107:0/2014640805 reap_dead start 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 -- 172.21.15.107:0/2014640805 shutdown_connections 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 -- 172.21.15.107:0/2014640805 >> 172.21.15.107:0/2014640805 conn(0x7f3a20071d40 msgr2=0x7f3a2006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 -- 172.21.15.107:0/2014640805 shutdown_connections 2023-11-24T08:07:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.323+0000 7f3a27541640 1 -- 172.21.15.107:0/2014640805 wait complete. 2023-11-24T08:07:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.325+0000 7f3a27541640 1 Processor -- start 2023-11-24T08:07:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.325+0000 7f3a27541640 1 -- start start 2023-11-24T08:07:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 0x7f3a2014df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a2006da10 0x7f3a20152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a201efd90 con 0x7f3a2014d260 2023-11-24T08:07:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2006de90 con 0x7f3a0c006570 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a27541640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a201eee50 con 0x7f3a2006da10 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33190/0 (socket says 172.21.15.107:33190) 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 -- 172.21.15.107:0/2175901060 learned_addr learned my addr 172.21.15.107:0/2175901060 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a252dd640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 0x7f3a2014df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 -- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a2006da10 msgr2=0x7f3a20152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a2006da10 0x7f3a20152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 -- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 msgr2=0x7f3a2014df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 0x7f3a2014df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a24adc640 1 -- 172.21.15.107:0/2175901060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a201ef110 con 0x7f3a2014d260 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.326+0000 7f3a252dd640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 0x7f3a2014df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a24adc640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3a1002c2b0 tx=0x7f3a10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10003810 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1002de90 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10003b70 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a201ef3f0 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.327+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a201ef9b0 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.328+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a20151000 con 0x7f3a2014d260 2023-11-24T08:07:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.329+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f3a10003cd0 con 0x7f3a2014d260 2023-11-24T08:07:48.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.330+0000 7f3a15ffb640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 0x7f3a0406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:48.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.330+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a100ab900 con 0x7f3a2014d260 2023-11-24T08:07:48.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.330+0000 7f3a252dd640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 0x7f3a0406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.331+0000 7f3a252dd640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 0x7f3a0406d5d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a2006a620 tx=0x7f3a1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.335+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a10076660 con 0x7f3a2014d260 2023-11-24T08:07:48.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.569+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3a201efb70 con 0x7f3a2014d260 2023-11-24T08:07:48.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.569+0000 7f3a15ffb640 1 -- 172.21.15.107:0/2175901060 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3a10079c80 con 0x7f3a2014d260 2023-11-24T08:07:48.570 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:07:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 msgr2=0x7f3a0406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 0x7f3a0406d5d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a2006a620 tx=0x7f3a1c009040 comp rx=0 tx=0).stop 2023-11-24T08:07:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 msgr2=0x7f3a2006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3a1002c2b0 tx=0x7f3a10004730 comp rx=0 tx=0).stop 2023-11-24T08:07:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 shutdown_connections 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0406b110 0x7f3a0406d5d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a2006da10 0x7f3a20152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a2014d260 0x7f3a2006d4d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 --2- 172.21.15.107:0/2175901060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a0c006570 0x7f3a2014df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 >> 172.21.15.107:0/2175901060 conn(0x7f3a20071d40 msgr2=0x7f3a2014f790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 shutdown_connections 2023-11-24T08:07:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:48.572+0000 7f3a27541640 1 -- 172.21.15.107:0/2175901060 wait complete. 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:07:48.593 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:07:48.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:48.594 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:07:48.645 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:07:48.646 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42202sec preferred_lft 42202sec 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.647 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:48.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:48.650 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:07:48.741 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:07:48.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:48.745 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:07:48.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:07:48.802 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:07:48.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:48.979 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:07:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:07:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:07:49.033 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:07:49.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:49.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:07:49.129 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:07:49.129 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:07:49.181 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:07:49.182 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:07:49.182 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:07:49.229 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:07:49.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:49.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:49.419 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:49.419 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:07:49.453 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:49.454 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:07:49.454 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:07:49.454 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:07:49.454 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:07:49.457 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:07:49.457 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:07:49.457 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:07:49.457 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:07:49.457 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:07:49.461 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:07:49.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:49.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:07:49.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:49.547 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:07:49.615 INFO:teuthology.orchestra.run.smithi107.stdout:client4796 2023-11-24T08:07:49.616 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:49.616 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:49.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:07:49.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.870+0000 7efe234e7640 1 Processor -- start 2023-11-24T08:07:49.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.870+0000 7efe234e7640 1 -- start start 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.871+0000 7efe234e7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe1c15c7f0 0x7efe1c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.871+0000 7efe234e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1c0cf450 0x7efe1c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.871+0000 7efe234e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0cece0 con 0x7efe1c0cdfb0 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.871+0000 7efe234e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c0cfe40 con 0x7efe1c0cf450 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.871+0000 7efe234e7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c0ceaf0 con 0x7efe1c15c7f0 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1c0cf450 0x7efe1c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1c0cf450 0x7efe1c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33194/0 (socket says 172.21.15.107:33194) 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 -- 172.21.15.107:0/2243799442 learned_addr learned my addr 172.21.15.107:0/2243799442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe21a84640 1 --1- 172.21.15.107:0/2243799442 >> v1:172.21.15.107:6789/0 conn(0x7efe1c0cdfb0 0x7efe1c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55334/0 (socket says 172.21.15.107:55334) 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe21283640 1 --2- 172.21.15.107:0/2243799442 >> v2:172.21.15.120:3300/0 conn(0x7efe1c15c7f0 0x7efe1c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 -- 172.21.15.107:0/2243799442 >> v2:172.21.15.120:3300/0 conn(0x7efe1c15c7f0 msgr2=0x7efe1c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 --2- 172.21.15.107:0/2243799442 >> v2:172.21.15.120:3300/0 conn(0x7efe1c15c7f0 0x7efe1c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 -- 172.21.15.107:0/2243799442 >> v1:172.21.15.107:6789/0 conn(0x7efe1c0cdfb0 legacy=0x7efe1c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe13fff640 1 -- 172.21.15.107:0/2243799442 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239539640 0 0) 0x7efe1c0cece0 con 0x7efe1c0cdfb0 2023-11-24T08:07:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.872+0000 7efe20a82640 1 -- 172.21.15.107:0/2243799442 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c06d040 con 0x7efe1c0cf450 2023-11-24T08:07:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.873+0000 7efe20a82640 1 --2- 172.21.15.107:0/2243799442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c0cf450 0x7efe1c0cf900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0d0130 tx=0x7efe0c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e1c09422d0d3af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe13fff640 1 -- 172.21.15.107:0/2243799442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c030880 con 0x7efe1c0cf450 2023-11-24T08:07:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe13fff640 1 -- 172.21.15.107:0/2243799442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c0309e0 con 0x7efe1c0cf450 2023-11-24T08:07:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe13fff640 1 -- 172.21.15.107:0/2243799442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c030d70 con 0x7efe1c0cf450 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 -- 172.21.15.107:0/2243799442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c0cf450 msgr2=0x7efe1c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 --2- 172.21.15.107:0/2243799442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c0cf450 0x7efe1c0cf900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0d0130 tx=0x7efe0c02b9c0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 -- 172.21.15.107:0/2243799442 shutdown_connections 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 --2- 172.21.15.107:0/2243799442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c0cf450 0x7efe1c0cf900 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 --2- 172.21.15.107:0/2243799442 >> v2:172.21.15.120:3300/0 conn(0x7efe1c15c7f0 0x7efe1c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 -- 172.21.15.107:0/2243799442 >> 172.21.15.107:0/2243799442 conn(0x7efe1c06a920 msgr2=0x7efe1c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.874+0000 7efe234e7640 1 -- 172.21.15.107:0/2243799442 shutdown_connections 2023-11-24T08:07:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.875+0000 7efe234e7640 1 -- 172.21.15.107:0/2243799442 wait complete. 2023-11-24T08:07:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.876+0000 7efe234e7640 1 Processor -- start 2023-11-24T08:07:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.876+0000 7efe234e7640 1 -- start start 2023-11-24T08:07:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe1c0cdfb0 0x7efe1c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 0x7efe1c14b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c15eae0 con 0x7efe1c15c7f0 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0af810 con 0x7efe1c0cf450 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.877+0000 7efe234e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0af250 con 0x7efe1c0cdfb0 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33196/0 (socket says 172.21.15.107:33196) 2023-11-24T08:07:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 -- 172.21.15.107:0/100136747 learned_addr learned my addr 172.21.15.107:0/100136747 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21283640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe1c0cdfb0 0x7efe1c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe20a82640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 0x7efe1c14b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 -- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe1c0cdfb0 msgr2=0x7efe1c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe1c0cdfb0 0x7efe1c15e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 -- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 msgr2=0x7efe1c14b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 0x7efe1c14b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 -- 172.21.15.107:0/100136747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0094d0 con 0x7efe1c15c7f0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe20a82640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 0x7efe1c14b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.878+0000 7efe21a84640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c800 tx=0x7efe1400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.879+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14004d60 con 0x7efe1c15c7f0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.879+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400a640 con 0x7efe1c15c7f0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.879+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400aa30 con 0x7efe1c15c7f0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.879+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c0af530 con 0x7efe1c15c7f0 2023-11-24T08:07:49.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.879+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c159160 con 0x7efe1c15c7f0 2023-11-24T08:07:49.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.880+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c0b2450 con 0x7efe1c15c7f0 2023-11-24T08:07:49.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.881+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7efe1400ab90 con 0x7efe1c15c7f0 2023-11-24T08:07:49.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.882+0000 7efe11ffb640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 0x7efe0006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:49.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.882+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe14088f70 con 0x7efe1c15c7f0 2023-11-24T08:07:49.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.882+0000 7efe21283640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 0x7efe0006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:49.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.883+0000 7efe21283640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 0x7efe0006d5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efe18006fd0 tx=0x7efe18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:49.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:49.887+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe1c0b2450 con 0x7efe1c15c7f0 2023-11-24T08:07:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.116+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efe1c0b2450 con 0x7efe1c15c7f0 2023-11-24T08:07:50.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.117+0000 7efe11ffb640 1 -- 172.21.15.107:0/100136747 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7efe1c0b2450 con 0x7efe1c15c7f0 2023-11-24T08:07:50.118 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:07:50.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.119+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 msgr2=0x7efe0006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.119+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 0x7efe0006d5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efe18006fd0 tx=0x7efe18008040 comp rx=0 tx=0).stop 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.119+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 msgr2=0x7efe1c15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.119+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c800 tx=0x7efe1400ccc0 comp rx=0 tx=0).stop 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 shutdown_connections 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe0006b110 0x7efe0006d5d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1c15c7f0 0x7efe1c15a850 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe1c0cf450 0x7efe1c14b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:50.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 --2- 172.21.15.107:0/100136747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe1c0cdfb0 0x7efe1c15e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:50.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 >> 172.21.15.107:0/100136747 conn(0x7efe1c06a920 msgr2=0x7efe1c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:50.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 shutdown_connections 2023-11-24T08:07:50.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:50.120+0000 7efe234e7640 1 -- 172.21.15.107:0/100136747 wait complete. 2023-11-24T08:07:50.140 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:07:50.141 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:07:50.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.141 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:07:50.203 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:07:50.204 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42199sec preferred_lft 42199sec 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:07:50.205 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:07:50.206 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:07:50.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.207 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:07:50.300 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:07:50.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.304 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:07:50.357 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:07:50.357 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:07:50.357 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:07:50.358 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:07:50.358 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:07:50.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:07:50.359 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:07:50.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.545 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:07:50.598 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:07:50.598 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:07:50.598 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:07:50.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:07:50.693 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:07:50.694 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:07:50.750 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:07:50.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:07:50.750 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:07:50.802 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:07:50.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.802 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:50.937 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:50.938 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:07:50.971 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:07:50.971 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:07:50.972 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:07:50.972 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:07:50.972 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:07:50.973 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:07:50.973 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:07:50.974 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:07:50.974 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:07:50.974 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:07:50.977 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:07:50.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:50.978 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:07:51.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:51.064 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:07:51.123 INFO:teuthology.orchestra.run.smithi120.stdout:client4799 2023-11-24T08:07:51.124 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:07:51.124 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:51.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:07:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.401+0000 7fed8f38e640 1 Processor -- start 2023-11-24T08:07:51.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.402+0000 7fed8f38e640 1 -- start start 2023-11-24T08:07:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.403+0000 7fed8f38e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed880ce6c0 0x7fed880cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.403+0000 7fed8f38e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880cece0 con 0x7fed880cf450 2023-11-24T08:07:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.403+0000 7fed8f38e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8806d040 con 0x7fed8815c7f0 2023-11-24T08:07:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.403+0000 7fed8f38e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed880cfe40 con 0x7fed880ce6c0 2023-11-24T08:07:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8c929640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fed880cf450 0x7fed880cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55336/0 (socket says 172.21.15.107:55336) 2023-11-24T08:07:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8c929640 1 -- 172.21.15.107:0/1246522339 learned_addr learned my addr 172.21.15.107:0/1246522339 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8d92b640 1 --2- 172.21.15.107:0/1246522339 >> v2:172.21.15.107:3300/0 conn(0x7fed880ce6c0 0x7fed880cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8d92b640 1 -- 172.21.15.107:0/1246522339 >> v1:172.21.15.120:6789/0 conn(0x7fed8815c7f0 legacy=0x7fed8806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed7bfff640 1 -- 172.21.15.107:0/1246522339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647072019 0 0) 0x7fed880cece0 con 0x7fed880cf450 2023-11-24T08:07:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8d92b640 1 -- 172.21.15.107:0/1246522339 >> v1:172.21.15.107:6789/0 conn(0x7fed880cf450 legacy=0x7fed880cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.404+0000 7fed8d92b640 1 -- 172.21.15.107:0/1246522339 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8806d320 con 0x7fed880ce6c0 2023-11-24T08:07:51.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.405+0000 7fed8d92b640 1 --2- 172.21.15.107:0/1246522339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880ce6c0 0x7fed880cf900 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fed800097a0 tx=0x7fed8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9354c0c47526a8f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:51.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.405+0000 7fed7bfff640 1 -- 172.21.15.107:0/1246522339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80031830 con 0x7fed880ce6c0 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.405+0000 7fed7bfff640 1 -- 172.21.15.107:0/1246522339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80031990 con 0x7fed880ce6c0 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed7bfff640 1 -- 172.21.15.107:0/1246522339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80031d20 con 0x7fed880ce6c0 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 -- 172.21.15.107:0/1246522339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880ce6c0 msgr2=0x7fed880cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 --2- 172.21.15.107:0/1246522339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880ce6c0 0x7fed880cf900 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fed800097a0 tx=0x7fed8002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 -- 172.21.15.107:0/1246522339 shutdown_connections 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 --2- 172.21.15.107:0/1246522339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880ce6c0 0x7fed880cf900 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 -- 172.21.15.107:0/1246522339 >> 172.21.15.107:0/1246522339 conn(0x7fed8806a920 msgr2=0x7fed8806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 -- 172.21.15.107:0/1246522339 shutdown_connections 2023-11-24T08:07:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.406+0000 7fed8f38e640 1 -- 172.21.15.107:0/1246522339 wait complete. 2023-11-24T08:07:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.408+0000 7fed8f38e640 1 Processor -- start 2023-11-24T08:07:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.408+0000 7fed8f38e640 1 -- start start 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed880ce6c0 0x7fed8814b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed8815c7f0 0x7fed88154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8815a850 con 0x7fed880cf450 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88159c40 con 0x7fed8815c7f0 2023-11-24T08:07:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8f38e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed88159f20 con 0x7fed880ce6c0 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.409+0000 7fed8c929640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33210/0 (socket says 172.21.15.107:33210) 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 -- 172.21.15.107:0/631712775 learned_addr learned my addr 172.21.15.107:0/631712775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 -- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed880ce6c0 msgr2=0x7fed8814b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed880ce6c0 0x7fed8814b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 -- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed8815c7f0 msgr2=0x7fed88154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed8815c7f0 0x7fed88154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 -- 172.21.15.107:0/631712775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80009400 con 0x7fed880cf450 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed8c929640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c890 tx=0x7fed7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.410+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00a640 con 0x7fed880cf450 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.411+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c00a7c0 con 0x7fed880cf450 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.411+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8815a200 con 0x7fed880cf450 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.411+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00ab70 con 0x7fed880cf450 2023-11-24T08:07:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.411+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8814b940 con 0x7fed880cf450 2023-11-24T08:07:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.412+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88067470 con 0x7fed880cf450 2023-11-24T08:07:51.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.413+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fed7c00acd0 con 0x7fed880cf450 2023-11-24T08:07:51.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.413+0000 7fed79ffb640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 0x7fed6c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:51.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.414+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed7c089380 con 0x7fed880cf450 2023-11-24T08:07:51.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.414+0000 7fed8d12a640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 0x7fed6c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:51.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.414+0000 7fed8d12a640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 0x7fed6c06d600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed84006fd0 tx=0x7fed84006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:51.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.418+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed7c054000 con 0x7fed880cf450 2023-11-24T08:07:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:51.650+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fed880cd730 con 0x7fed880cf450 2023-11-24T08:07:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.513+0000 7fed79ffb640 1 -- 172.21.15.107:0/631712775 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 136+0+0 (secure 0 0 0) 0x7fed7c057620 con 0x7fed880cf450 2023-11-24T08:07:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.516+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 msgr2=0x7fed6c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.516+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 0x7fed6c06d600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed84006fd0 tx=0x7fed84006bc0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.516+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 msgr2=0x7fed8815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.516+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c890 tx=0x7fed7c00cd50 comp rx=0 tx=0).stop 2023-11-24T08:07:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 shutdown_connections 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed6c06b140 0x7fed6c06d600 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed8815c7f0 0x7fed88154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed880cf450 0x7fed8815b320 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 --2- 172.21.15.107:0/631712775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed880ce6c0 0x7fed8814b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 >> 172.21.15.107:0/631712775 conn(0x7fed8806a920 msgr2=0x7fed8815f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 shutdown_connections 2023-11-24T08:07:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.517+0000 7fed8f38e640 1 -- 172.21.15.107:0/631712775 wait complete. 2023-11-24T08:07:52.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:07:52.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.753+0000 7f7598848640 1 Processor -- start 2023-11-24T08:07:52.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.754+0000 7f7598848640 1 -- start start 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7598848640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75940cc0f0 0x7f75940c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7598848640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759415eea0 con 0x7f75940c57e0 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7598848640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940abf30 con 0x7f759415b440 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7598848640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759415b820 con 0x7f75940cc0f0 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7592d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75940c57e0 0x7f759406c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55340/0 (socket says 172.21.15.107:55340) 2023-11-24T08:07:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.755+0000 7f7592d9d640 1 -- 172.21.15.107:0/1573084493 learned_addr learned my addr 172.21.15.107:0/1573084493 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865396309 0 0) 0x7f759415eea0 con 0x7f75940c57e0 2023-11-24T08:07:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> v2:172.21.15.120:3300/0 conn(0x7f75940cc0f0 0x7f75940c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7580003520 con 0x7f75940c57e0 2023-11-24T08:07:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254455302 0 0) 0x7f7580003520 con 0x7f75940c57e0 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 >> v2:172.21.15.120:3300/0 conn(0x7f75940cc0f0 msgr2=0x7f75940c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 --2- 172.21.15.107:0/1573084493 >> v2:172.21.15.120:3300/0 conn(0x7f75940cc0f0 0x7f75940c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 >> v1:172.21.15.120:6789/0 conn(0x7f759415b440 legacy=0x7f75940cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759415f0f0 con 0x7f75940c57e0 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f758c001670 con 0x7f75940c57e0 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> v2:172.21.15.120:3300/0 conn(0x7f75940cc0f0 0x7f75940c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 >> v1:172.21.15.107:6789/0 conn(0x7f75940c57e0 legacy=0x7f759406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7580003970 0x7f7580003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 0x7f7580004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75800052a0 0x7f758000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759415b820 con 0x7f75800047c0 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758000c170 con 0x7f7580003970 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.756+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758000c330 con 0x7f75800052a0 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 0x7f7580004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 -- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75800052a0 msgr2=0x7f758000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75800052a0 0x7f758000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 -- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7580003970 msgr2=0x7f7580003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7580003970 0x7f7580003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7591d9b640 1 -- 172.21.15.107:0/1573084493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7580003520 con 0x7f75800047c0 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.757+0000 7f7592d9d640 1 -- 172.21.15.107:0/1573084493 reap_dead start 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.758+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 0x7f7580004c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f757c006fd0 tx=0x7f75800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7391648e4e7dbb7b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.758+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c004230 con 0x7f75800047c0 2023-11-24T08:07:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.758+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c004390 con 0x7f75800047c0 2023-11-24T08:07:52.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.758+0000 7f759159a640 1 -- 172.21.15.107:0/1573084493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c003830 con 0x7f75800047c0 2023-11-24T08:07:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 -- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 msgr2=0x7f7580004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 0x7f7580004c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f757c006fd0 tx=0x7f75800057f0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 -- 172.21.15.107:0/1573084493 shutdown_connections 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 --2- 172.21.15.107:0/1573084493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75800047c0 0x7f7580004c40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 -- 172.21.15.107:0/1573084493 >> 172.21.15.107:0/1573084493 conn(0x7f75940b0c50 msgr2=0x7f75940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 -- 172.21.15.107:0/1573084493 shutdown_connections 2023-11-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.759+0000 7f7598848640 1 -- 172.21.15.107:0/1573084493 wait complete. 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.761+0000 7f7598848640 1 Processor -- start 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.761+0000 7f7598848640 1 -- start start 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 0x7f75941f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75940b3090 0x7f75941568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75941f7950 con 0x7f75940c7120 2023-11-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594156e20 con 0x7f75800047c0 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7598848640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75940c78a0 con 0x7f75940b3090 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7592d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7592d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33222/0 (socket says 172.21.15.107:33222) 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.762+0000 7f7592d9d640 1 -- 172.21.15.107:0/1765579862 learned_addr learned my addr 172.21.15.107:0/1765579862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f759259c640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 0x7f75941f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7591d9b640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75940b3090 0x7f75941568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 -- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75940b3090 msgr2=0x7f75941568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75940b3090 0x7f75941568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 -- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 msgr2=0x7f75941f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 0x7f75941f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 -- 172.21.15.107:0/1765579862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940c7b80 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f759259c640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 0x7f75941f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7592d9d640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f759409d280 tx=0x7f758c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c004bf0 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c004d50 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c007270 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.763+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75940c7e60 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.764+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759415ba00 con 0x7f75940c7120 2023-11-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.765+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7594061a50 con 0x7f75940c7120 2023-11-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.766+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f758c0073d0 con 0x7f75940c7120 2023-11-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.766+0000 7f758affd640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 0x7f757406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.767+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758c088a80 con 0x7f75940c7120 2023-11-24T08:07:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.767+0000 7f759259c640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 0x7f757406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.767+0000 7f759259c640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 0x7f757406d600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7584001d60 tx=0x7f7584009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.771+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f758c08ae60 con 0x7f75940c7120 2023-11-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.984+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f759415f990 con 0x7f757406b140 2023-11-24T08:07:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.986+0000 7f758affd640 1 -- 172.21.15.107:0/1765579862 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7584006b50 con 0x7f757406b140 2023-11-24T08:07:52.987 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:07:52.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:07:52.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:07:52.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:07:52.988 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:07:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 msgr2=0x7f757406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 0x7f757406d600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7584001d60 tx=0x7f7584009040 comp rx=0 tx=0).stop 2023-11-24T08:07:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 msgr2=0x7f75940cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f759409d280 tx=0x7f758c00c8e0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 shutdown_connections 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757406b140 0x7f757406d600 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75940c7120 0x7f75940cd870 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75940b3090 0x7f75941568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 --2- 172.21.15.107:0/1765579862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75800047c0 0x7f75941f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 >> 172.21.15.107:0/1765579862 conn(0x7f75940b0c50 msgr2=0x7f75940afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 shutdown_connections 2023-11-24T08:07:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:52.989+0000 7f7598848640 1 -- 172.21.15.107:0/1765579862 wait complete. 2023-11-24T08:07:53.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:07:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.216+0000 7fea8632e640 1 Processor -- start 2023-11-24T08:07:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.216+0000 7fea8632e640 1 -- start start 2023-11-24T08:07:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.218+0000 7fea8632e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fea8006d080 0x7fea800ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.218+0000 7fea8632e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fea800ac840 0x7fea800ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.218+0000 7fea8632e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea800cdf70 con 0x7fea8015d5f0 2023-11-24T08:07:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.218+0000 7fea8632e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea800cb3d0 con 0x7fea800ac840 2023-11-24T08:07:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.218+0000 7fea8632e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea800cbfa0 con 0x7fea8006d080 2023-11-24T08:07:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea8532c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fea8015d5f0 0x7fea8006c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55354/0 (socket says 172.21.15.107:55354) 2023-11-24T08:07:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea8532c640 1 -- 172.21.15.107:0/3489302557 learned_addr learned my addr 172.21.15.107:0/3489302557 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea85b2d640 1 --2- 172.21.15.107:0/3489302557 >> v2:172.21.15.107:3300/0 conn(0x7fea800ac840 0x7fea800ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea84b2b640 1 --2- 172.21.15.107:0/3489302557 >> v2:172.21.15.120:3300/0 conn(0x7fea8006d080 0x7fea800ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea77fff640 1 -- 172.21.15.107:0/3489302557 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177978068 0 0) 0x7fea800cdf70 con 0x7fea8015d5f0 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea77fff640 1 -- 172.21.15.107:0/3489302557 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003520 con 0x7fea8015d5f0 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea85b2d640 1 -- 172.21.15.107:0/3489302557 >> v2:172.21.15.120:3300/0 conn(0x7fea8006d080 msgr2=0x7fea800ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea85b2d640 1 --2- 172.21.15.107:0/3489302557 >> v2:172.21.15.120:3300/0 conn(0x7fea8006d080 0x7fea800ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea85b2d640 1 -- 172.21.15.107:0/3489302557 >> v1:172.21.15.107:6789/0 conn(0x7fea8015d5f0 legacy=0x7fea8006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.219+0000 7fea85b2d640 1 -- 172.21.15.107:0/3489302557 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800cc280 con 0x7fea800ac840 2023-11-24T08:07:53.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.220+0000 7fea85b2d640 1 --2- 172.21.15.107:0/3489302557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea800ac840 0x7fea800ce9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fea8006b370 tx=0x7fea7802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b738248eb115eb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea77fff640 1 -- 172.21.15.107:0/3489302557 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7802cbe0 con 0x7fea800ac840 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea77fff640 1 -- 172.21.15.107:0/3489302557 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7802cd40 con 0x7fea800ac840 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea77fff640 1 -- 172.21.15.107:0/3489302557 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78034660 con 0x7fea800ac840 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 -- 172.21.15.107:0/3489302557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea800ac840 msgr2=0x7fea800ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 --2- 172.21.15.107:0/3489302557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea800ac840 0x7fea800ce9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fea8006b370 tx=0x7fea7802be30 comp rx=0 tx=0).stop 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 -- 172.21.15.107:0/3489302557 shutdown_connections 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 --2- 172.21.15.107:0/3489302557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea800ac840 0x7fea800ce9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 --2- 172.21.15.107:0/3489302557 >> v2:172.21.15.120:3300/0 conn(0x7fea8006d080 0x7fea800ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 -- 172.21.15.107:0/3489302557 >> 172.21.15.107:0/3489302557 conn(0x7fea8006a500 msgr2=0x7fea8006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 -- 172.21.15.107:0/3489302557 shutdown_connections 2023-11-24T08:07:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.221+0000 7fea8632e640 1 -- 172.21.15.107:0/3489302557 wait complete. 2023-11-24T08:07:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.223+0000 7fea8632e640 1 Processor -- start 2023-11-24T08:07:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.223+0000 7fea8632e640 1 -- start start 2023-11-24T08:07:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fea800ac840 0x7fea801f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea8015d5f0 0x7fea801f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8015b6b0 con 0x7fea8006d080 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea801f4320 con 0x7fea8015d5f0 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.224+0000 7fea8632e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea801f38b0 con 0x7fea800ac840 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33242/0 (socket says 172.21.15.107:33242) 2023-11-24T08:07:53.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 -- 172.21.15.107:0/126137741 learned_addr learned my addr 172.21.15.107:0/126137741 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 -- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fea800ac840 msgr2=0x7fea801f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fea800ac840 0x7fea801f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 -- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea8015d5f0 msgr2=0x7fea801f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea8015d5f0 0x7fea801f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 -- 172.21.15.107:0/126137741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78009a70 con 0x7fea8006d080 2023-11-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.225+0000 7fea8532c640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00c530 tx=0x7fea7c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.226+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c005ce0 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.226+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c005e40 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.226+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0072b0 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.226+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8015db00 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.226+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8015e0c0 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.227+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea80067990 con 0x7fea8006d080 2023-11-24T08:07:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.228+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fea7c007410 con 0x7fea8006d080 2023-11-24T08:07:53.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.229+0000 7fea75ffb640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 0x7fea6406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.229+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea7c0890e0 con 0x7fea8006d080 2023-11-24T08:07:53.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.229+0000 7fea84b2b640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 0x7fea6406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.230+0000 7fea84b2b640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 0x7fea6406d5d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea70006fd0 tx=0x7fea70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.233+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea7c053e70 con 0x7fea8006d080 2023-11-24T08:07:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.424+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fea8004e840 con 0x7fea8006d080 2023-11-24T08:07:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.425+0000 7fea75ffb640 1 -- 172.21.15.107:0/126137741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fea7c057490 con 0x7fea8006d080 2023-11-24T08:07:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 msgr2=0x7fea6406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 0x7fea6406d5d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea70006fd0 tx=0x7fea70008040 comp rx=0 tx=0).stop 2023-11-24T08:07:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 msgr2=0x7fea8015b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00c530 tx=0x7fea7c00c9f0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 shutdown_connections 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fea6406b110 0x7fea6406d5d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea8015d5f0 0x7fea801f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fea800ac840 0x7fea801f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 --2- 172.21.15.107:0/126137741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea8006d080 0x7fea8015b0b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 >> 172.21.15.107:0/126137741 conn(0x7fea8006a500 msgr2=0x7fea80159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 shutdown_connections 2023-11-24T08:07:53.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.428+0000 7fea8632e640 1 -- 172.21.15.107:0/126137741 wait complete. 2023-11-24T08:07:53.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol1626 --mode=777 2023-11-24T08:07:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.662+0000 7fce1582e640 1 Processor -- start 2023-11-24T08:07:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.663+0000 7fce1582e640 1 -- start start 2023-11-24T08:07:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce1582e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce100cfbe0 0x7fce100d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce1582e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fce1015ce20 0x7fce1015d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce1582e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1015ea30 con 0x7fce10160b00 2023-11-24T08:07:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce1582e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce100d0590 con 0x7fce100cfbe0 2023-11-24T08:07:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce1582e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1015de20 con 0x7fce1015ce20 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce0ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce100cfbe0 0x7fce100d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce0ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce100cfbe0 0x7fce100d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33248/0 (socket says 172.21.15.107:33248) 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.664+0000 7fce0ffff640 1 -- 172.21.15.107:0/2424960436 learned_addr learned my addr 172.21.15.107:0/2424960436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.665+0000 7fce0ffff640 1 -- 172.21.15.107:0/2424960436 >> v2:172.21.15.120:3300/0 conn(0x7fce1015ce20 msgr2=0x7fce1015d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.665+0000 7fce0ffff640 1 --2- 172.21.15.107:0/2424960436 >> v2:172.21.15.120:3300/0 conn(0x7fce1015ce20 0x7fce1015d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.665+0000 7fce0ffff640 1 -- 172.21.15.107:0/2424960436 >> v1:172.21.15.107:6789/0 conn(0x7fce10160b00 legacy=0x7fce100d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.665+0000 7fce0ffff640 1 -- 172.21.15.107:0/2424960436 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1015e100 con 0x7fce100cfbe0 2023-11-24T08:07:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.666+0000 7fce0ffff640 1 --2- 172.21.15.107:0/2424960436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce100cfbe0 0x7fce100d0050 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce101620e0 tx=0x7fcdf802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38e02cb527e5ad1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.666+0000 7fce0f7fe640 1 -- 172.21.15.107:0/2424960436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8030870 con 0x7fce100cfbe0 2023-11-24T08:07:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.666+0000 7fce0f7fe640 1 -- 172.21.15.107:0/2424960436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80309d0 con 0x7fce100cfbe0 2023-11-24T08:07:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.666+0000 7fce0f7fe640 1 -- 172.21.15.107:0/2424960436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8030d60 con 0x7fce100cfbe0 2023-11-24T08:07:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 -- 172.21.15.107:0/2424960436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce100cfbe0 msgr2=0x7fce100d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 --2- 172.21.15.107:0/2424960436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce100cfbe0 0x7fce100d0050 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce101620e0 tx=0x7fcdf802b9b0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 -- 172.21.15.107:0/2424960436 shutdown_connections 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 --2- 172.21.15.107:0/2424960436 >> v2:172.21.15.120:3300/0 conn(0x7fce1015ce20 0x7fce1015d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 --2- 172.21.15.107:0/2424960436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce100cfbe0 0x7fce100d0050 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 -- 172.21.15.107:0/2424960436 >> 172.21.15.107:0/2424960436 conn(0x7fce1009c510 msgr2=0x7fce100d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 -- 172.21.15.107:0/2424960436 shutdown_connections 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.667+0000 7fce1582e640 1 -- 172.21.15.107:0/2424960436 wait complete. 2023-11-24T08:07:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.669+0000 7fce1582e640 1 Processor -- start 2023-11-24T08:07:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.669+0000 7fce1582e640 1 -- start start 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 0x7fce101516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce10160b00 0x7fce1015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10151c00 con 0x7fce1015ce20 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce100b72f0 con 0x7fce10160b00 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce1582e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce100b75d0 con 0x7fce100cfbe0 2023-11-24T08:07:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33254/0 (socket says 172.21.15.107:33254) 2023-11-24T08:07:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.670+0000 7fce0ffff640 1 -- 172.21.15.107:0/2764181447 learned_addr learned my addr 172.21.15.107:0/2764181447 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce1482c640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 0x7fce101516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 -- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 msgr2=0x7fce101516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 0x7fce101516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 -- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce10160b00 msgr2=0x7fce1015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce10160b00 0x7fce1015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 -- 172.21.15.107:0/2764181447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80094d0 con 0x7fce1015ce20 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce1482c640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 0x7fce101516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0ffff640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8000c00 tx=0x7fcdf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8030840 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8039d90 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8043630 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce100b78b0 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.671+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce100b7e70 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.672+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1004e840 con 0x7fce1015ce20 2023-11-24T08:07:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.673+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7fcdf802f3c0 con 0x7fce1015ce20 2023-11-24T08:07:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.674+0000 7fce0d7fa640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 0x7fcdf006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.674+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdf80abf90 con 0x7fce1015ce20 2023-11-24T08:07:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.675+0000 7fce1482c640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 0x7fcdf006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:53.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.675+0000 7fce1482c640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 0x7fcdf006d5d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce1015db90 tx=0x7fce00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:53.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.679+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdf8076c70 con 0x7fce1015ce20 2023-11-24T08:07:53.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.900+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol1626", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fce1015e6c0 con 0x7fcdf006b110 2023-11-24T08:07:53.910 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:53.910+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:53.911 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:53.910+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:53.911 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:53.910+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:53.911 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:53.910+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:53.911 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:07:53.910+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:07:53.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.958+0000 7fce0d7fa640 1 -- 172.21.15.107:0/2764181447 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce00003990 con 0x7fcdf006b110 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 msgr2=0x7fcdf006d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 0x7fcdf006d5d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce1015db90 tx=0x7fce00009040 comp rx=0 tx=0).stop 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 msgr2=0x7fce100c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8000c00 tx=0x7fcdf8004730 comp rx=0 tx=0).stop 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 shutdown_connections 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdf006b110 0x7fcdf006d5d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce10160b00 0x7fce1015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1015ce20 0x7fce100c9530 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 --2- 172.21.15.107:0/2764181447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce100cfbe0 0x7fce101516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 >> 172.21.15.107:0/2764181447 conn(0x7fce1009c510 msgr2=0x7fce1015c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 shutdown_connections 2023-11-24T08:07:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:53.961+0000 7fce1582e640 1 -- 172.21.15.107:0/2764181447 wait complete. 2023-11-24T08:07:53.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1626 2023-11-24T08:07:54.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.172+0000 7f27edd2e640 1 Processor -- start 2023-11-24T08:07:54.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.173+0000 7f27edd2e640 1 -- start start 2023-11-24T08:07:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.174+0000 7f27edd2e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27e80b0b40 0x7f27e80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.174+0000 7f27edd2e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f27e815ded0 0x7f27e815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.174+0000 7f27edd2e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e806d430 con 0x7f27e815a1c0 2023-11-24T08:07:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.174+0000 7f27edd2e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e815ec50 con 0x7f27e80b0b40 2023-11-24T08:07:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.174+0000 7f27edd2e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e806cb00 con 0x7f27e815ded0 2023-11-24T08:07:54.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27ecd2c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f27e815a1c0 0x7f27e806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55374/0 (socket says 172.21.15.107:55374) 2023-11-24T08:07:54.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27ecd2c640 1 -- 172.21.15.107:0/3358802761 learned_addr learned my addr 172.21.15.107:0/3358802761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:07:54.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e7fff640 1 --2- 172.21.15.107:0/3358802761 >> v2:172.21.15.107:3300/0 conn(0x7f27e80b0b40 0x7f27e80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:54.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e77fe640 1 -- 172.21.15.107:0/3358802761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613496502 0 0) 0x7f27e806d430 con 0x7f27e815a1c0 2023-11-24T08:07:54.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e77fe640 1 -- 172.21.15.107:0/3358802761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d8003520 con 0x7f27e815a1c0 2023-11-24T08:07:54.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27ed52d640 1 --2- 172.21.15.107:0/3358802761 >> v2:172.21.15.120:3300/0 conn(0x7f27e815ded0 0x7f27e815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:54.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e7fff640 1 -- 172.21.15.107:0/3358802761 >> v2:172.21.15.120:3300/0 conn(0x7f27e815ded0 msgr2=0x7f27e815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:07:54.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e7fff640 1 --2- 172.21.15.107:0/3358802761 >> v2:172.21.15.120:3300/0 conn(0x7f27e815ded0 0x7f27e815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e7fff640 1 -- 172.21.15.107:0/3358802761 >> v1:172.21.15.107:6789/0 conn(0x7f27e815a1c0 legacy=0x7f27e806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.175+0000 7f27e7fff640 1 -- 172.21.15.107:0/3358802761 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e80b14e0 con 0x7f27e80b0b40 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.176+0000 7f27e7fff640 1 --2- 172.21.15.107:0/3358802761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e80b0b40 0x7f27e80b0fa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f27e80c9ad0 tx=0x7f27d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6f6c383a99b7ec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.177+0000 7f27e77fe640 1 -- 172.21.15.107:0/3358802761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4031830 con 0x7f27e80b0b40 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.177+0000 7f27e77fe640 1 -- 172.21.15.107:0/3358802761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d4031990 con 0x7f27e80b0b40 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.177+0000 7f27e77fe640 1 -- 172.21.15.107:0/3358802761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4031d20 con 0x7f27e80b0b40 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 -- 172.21.15.107:0/3358802761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e80b0b40 msgr2=0x7f27e80b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 --2- 172.21.15.107:0/3358802761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e80b0b40 0x7f27e80b0fa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f27e80c9ad0 tx=0x7f27d402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 -- 172.21.15.107:0/3358802761 shutdown_connections 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 --2- 172.21.15.107:0/3358802761 >> v2:172.21.15.120:3300/0 conn(0x7f27e815ded0 0x7f27e815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 --2- 172.21.15.107:0/3358802761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e80b0b40 0x7f27e80b0fa0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 -- 172.21.15.107:0/3358802761 >> 172.21.15.107:0/3358802761 conn(0x7f27e80cbb40 msgr2=0x7f27e80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:54.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 -- 172.21.15.107:0/3358802761 shutdown_connections 2023-11-24T08:07:54.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.178+0000 7f27edd2e640 1 -- 172.21.15.107:0/3358802761 wait complete. 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.179+0000 7f27edd2e640 1 Processor -- start 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.180+0000 7f27edd2e640 1 -- start start 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27e80b0b40 0x7f27e809b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27e815ded0 0x7f27e80ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80c69e0 con 0x7f27e815a1c0 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e8150ab0 con 0x7f27e80b0b40 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27edd2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80c6700 con 0x7f27e815ded0 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33290/0 (socket says 172.21.15.107:33290) 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 -- 172.21.15.107:0/2321198663 learned_addr learned my addr 172.21.15.107:0/2321198663 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 -- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27e815ded0 msgr2=0x7f27e80ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27ed52d640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27e815ded0 0x7f27e80ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27e815ded0 0x7f27e80ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 -- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27e80b0b40 msgr2=0x7f27e809b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27e80b0b40 0x7f27e809b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.181+0000 7f27e7fff640 1 -- 172.21.15.107:0/2321198663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d4009400 con 0x7f27e815a1c0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27e7fff640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009890 tx=0x7f27d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d403a6f0 con 0x7f27e815a1c0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d403a870 con 0x7f27e815a1c0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d403ac20 con 0x7f27e815a1c0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e815ea20 con 0x7f27e815a1c0 2023-11-24T08:07:54.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.182+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e80c8e00 con 0x7f27e815a1c0 2023-11-24T08:07:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.183+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e80c8980 con 0x7f27e815a1c0 2023-11-24T08:07:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.184+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 89486+0+0 (secure 0 0 0) 0x7f27d4043410 con 0x7f27e815a1c0 2023-11-24T08:07:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.185+0000 7f27e57fa640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 0x7f27cc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:07:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.185+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27d40ab6f0 con 0x7f27e815a1c0 2023-11-24T08:07:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.185+0000 7f27ecd2c640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 0x7f27cc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:07:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.186+0000 7f27ecd2c640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 0x7f27cc06d5d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f27dc001610 tx=0x7f27dc007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:07:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.189+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27e80c8980 con 0x7f27e815a1c0 2023-11-24T08:07:54.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.406+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1626", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e815e8a0 con 0x7f27cc06b110 2023-11-24T08:07:54.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.420+0000 7f27e57fa640 1 -- 172.21.15.107:0/2321198663 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f27dc0058b0 con 0x7f27cc06b110 2023-11-24T08:07:54.421 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3 2023-11-24T08:07:54.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 msgr2=0x7f27cc06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 0x7f27cc06d5d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f27dc001610 tx=0x7f27dc007930 comp rx=0 tx=0).stop 2023-11-24T08:07:54.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 msgr2=0x7f27e8150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009890 tx=0x7f27d40046c0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 shutdown_connections 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f27cc06b110 0x7f27cc06d5d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27e815ded0 0x7f27e80ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27e815a1c0 0x7f27e8150570 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 --2- 172.21.15.107:0/2321198663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27e80b0b40 0x7f27e809b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 >> 172.21.15.107:0/2321198663 conn(0x7f27e80cbb40 msgr2=0x7f27e80b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 shutdown_connections 2023-11-24T08:07:54.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:07:54.423+0000 7f27edd2e640 1 -- 172.21.15.107:0/2321198663 wait complete. 2023-11-24T08:07:54.443 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol1626 with 200 files each 1MB size under directory volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3 2023-11-24T08:07:54.443 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:54.444 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:54.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:54.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033254 s, 31.5 MB/s 2023-11-24T08:07:54.598 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:54.599 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:54.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:54.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:54.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:54.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:54.732 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027541 s, 38.1 MB/s 2023-11-24T08:07:54.734 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:54.734 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:54.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:54.793 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:54.877 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:54.877 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:54.877 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296062 s, 35.4 MB/s 2023-11-24T08:07:54.879 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:54.879 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:54.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:54.934 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198362 s, 52.9 MB/s 2023-11-24T08:07:55.008 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.008 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.142 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224462 s, 46.7 MB/s 2023-11-24T08:07:55.143 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.143 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.200 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271914 s, 38.6 MB/s 2023-11-24T08:07:55.392 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.393 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.453 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.535 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.536 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.536 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286709 s, 36.6 MB/s 2023-11-24T08:07:55.537 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.537 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.594 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.674 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.675 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.675 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269568 s, 38.9 MB/s 2023-11-24T08:07:55.676 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.676 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.734 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.814 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.814 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.814 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227712 s, 46.0 MB/s 2023-11-24T08:07:55.816 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.816 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:55.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:55.862 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:55.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:55.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:55.985 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023834 s, 44.0 MB/s 2023-11-24T08:07:55.987 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:55.987 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.116 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.116 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.116 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020196 s, 51.9 MB/s 2023-11-24T08:07:56.117 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.117 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.173 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.245 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.245 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.245 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207393 s, 50.6 MB/s 2023-11-24T08:07:56.247 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.247 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.301 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.374 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.375 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.376 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214 s, 49.0 MB/s 2023-11-24T08:07:56.376 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.376 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.433 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.506 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.506 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.506 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217563 s, 48.2 MB/s 2023-11-24T08:07:56.507 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.507 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242442 s, 43.3 MB/s 2023-11-24T08:07:56.637 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.637 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.695 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218613 s, 48.0 MB/s 2023-11-24T08:07:56.770 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.770 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.825 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214602 s, 48.9 MB/s 2023-11-24T08:07:56.905 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:56.905 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:56.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:56.961 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.040 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241198 s, 43.5 MB/s 2023-11-24T08:07:57.042 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.042 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246278 s, 42.6 MB/s 2023-11-24T08:07:57.178 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.179 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.236 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265262 s, 39.5 MB/s 2023-11-24T08:07:57.318 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.318 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.377 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.454 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.455 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.455 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245739 s, 42.7 MB/s 2023-11-24T08:07:57.456 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.456 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.512 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027573 s, 38.0 MB/s 2023-11-24T08:07:57.593 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.594 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236963 s, 44.3 MB/s 2023-11-24T08:07:57.734 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.734 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.855 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.855 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195392 s, 53.7 MB/s 2023-11-24T08:07:57.857 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.857 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:57.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:57.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:57.987 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:57.988 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:57.988 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238393 s, 44.0 MB/s 2023-11-24T08:07:57.989 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:57.989 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.046 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279581 s, 37.5 MB/s 2023-11-24T08:07:58.129 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.129 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.186 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022604 s, 46.4 MB/s 2023-11-24T08:07:58.263 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.263 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.316 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.440 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.440 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249904 s, 42.0 MB/s 2023-11-24T08:07:58.441 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.441 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.500 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272887 s, 38.4 MB/s 2023-11-24T08:07:58.583 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.583 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.640 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.718 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.718 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247852 s, 42.3 MB/s 2023-11-24T08:07:58.720 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.720 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199027 s, 52.7 MB/s 2023-11-24T08:07:58.854 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.855 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:58.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:58.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252691 s, 41.5 MB/s 2023-11-24T08:07:58.994 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:58.994 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.129 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.129 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.129 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02233 s, 47.0 MB/s 2023-11-24T08:07:59.131 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.131 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.183 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.302 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178922 s, 58.6 MB/s 2023-11-24T08:07:59.304 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.305 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236676 s, 44.3 MB/s 2023-11-24T08:07:59.439 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.439 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.498 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221167 s, 47.4 MB/s 2023-11-24T08:07:59.574 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.574 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.630 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026824 s, 39.1 MB/s 2023-11-24T08:07:59.713 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.713 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.772 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019929 s, 52.6 MB/s 2023-11-24T08:07:59.852 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.852 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:07:59.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:07:59.908 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:07:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:07:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:07:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253968 s, 41.3 MB/s 2023-11-24T08:07:59.984 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:07:59.984 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.042 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042837 s, 24.5 MB/s 2023-11-24T08:08:00.141 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.141 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.198 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195492 s, 53.6 MB/s 2023-11-24T08:08:00.271 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.271 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019658 s, 53.3 MB/s 2023-11-24T08:08:00.394 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.394 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.452 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.531 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02565 s, 40.9 MB/s 2023-11-24T08:08:00.533 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.533 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.590 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197906 s, 53.0 MB/s 2023-11-24T08:08:00.668 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.668 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329318 s, 31.8 MB/s 2023-11-24T08:08:00.815 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.815 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:00.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:00.870 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250311 s, 41.9 MB/s 2023-11-24T08:08:00.951 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:00.951 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.007 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.081 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209276 s, 50.1 MB/s 2023-11-24T08:08:01.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.082 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.140 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.223 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.223 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.224 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295009 s, 35.5 MB/s 2023-11-24T08:08:01.225 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.225 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.358 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179825 s, 58.3 MB/s 2023-11-24T08:08:01.359 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.359 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.413 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020641 s, 50.8 MB/s 2023-11-24T08:08:01.534 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.534 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.592 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251324 s, 41.7 MB/s 2023-11-24T08:08:01.671 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.728 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229127 s, 45.8 MB/s 2023-11-24T08:08:01.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.806 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.865 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156021 s, 67.2 MB/s 2023-11-24T08:08:01.934 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:01.934 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:01.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:01.984 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.103 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.103 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.104 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212934 s, 49.2 MB/s 2023-11-24T08:08:02.105 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.105 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.162 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.239 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.239 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.239 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022153 s, 47.3 MB/s 2023-11-24T08:08:02.240 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.240 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.299 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.370 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200068 s, 52.4 MB/s 2023-11-24T08:08:02.371 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.371 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.425 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.504 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.505 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.505 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253296 s, 41.4 MB/s 2023-11-24T08:08:02.506 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.506 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.566 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253895 s, 41.3 MB/s 2023-11-24T08:08:02.646 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.646 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.773 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02129 s, 49.3 MB/s 2023-11-24T08:08:02.774 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.774 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.831 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193544 s, 54.2 MB/s 2023-11-24T08:08:02.905 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:02.905 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:02.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:02.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246529 s, 42.5 MB/s 2023-11-24T08:08:03.045 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.045 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179163 s, 58.5 MB/s 2023-11-24T08:08:03.167 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.167 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.220 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.294 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019953 s, 52.6 MB/s 2023-11-24T08:08:03.295 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.295 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.355 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.430 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236597 s, 44.3 MB/s 2023-11-24T08:08:03.433 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.433 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.64 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.567 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.567 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.567 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249293 s, 42.1 MB/s 2023-11-24T08:08:03.569 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.569 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.65 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.697 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211244 s, 49.6 MB/s 2023-11-24T08:08:03.699 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.699 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.755 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.66 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:03.878 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229711 s, 45.6 MB/s 2023-11-24T08:08:03.880 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:03.880 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:03.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:03.934 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.67 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.007 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.008 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.008 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221355 s, 47.4 MB/s 2023-11-24T08:08:04.009 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.009 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.68 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.146 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023268 s, 45.1 MB/s 2023-11-24T08:08:04.147 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.148 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.69 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227534 s, 46.1 MB/s 2023-11-24T08:08:04.284 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.284 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.339 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.70 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.418 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268825 s, 39.0 MB/s 2023-11-24T08:08:04.420 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.420 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.477 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.71 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235295 s, 44.6 MB/s 2023-11-24T08:08:04.554 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.554 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.610 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.72 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206762 s, 50.7 MB/s 2023-11-24T08:08:04.682 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.682 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.73 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183897 s, 57.0 MB/s 2023-11-24T08:08:04.810 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.810 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.860 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.74 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:04.926 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:04.927 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:04.927 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192457 s, 54.5 MB/s 2023-11-24T08:08:04.928 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:04.928 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:04.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:04.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.75 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025931 s, 40.4 MB/s 2023-11-24T08:08:05.065 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.065 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.125 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.76 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221433 s, 47.4 MB/s 2023-11-24T08:08:05.202 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.202 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.255 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.77 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206375 s, 50.8 MB/s 2023-11-24T08:08:05.330 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.331 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.78 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251977 s, 41.6 MB/s 2023-11-24T08:08:05.469 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.469 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.519 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.79 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.640 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221766 s, 47.3 MB/s 2023-11-24T08:08:05.641 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.641 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.696 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.80 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.776 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.776 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.776 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262628 s, 39.9 MB/s 2023-11-24T08:08:05.777 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.777 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.836 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.81 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:05.913 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:05.913 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:05.913 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219775 s, 47.7 MB/s 2023-11-24T08:08:05.915 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:05.915 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:05.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:05.971 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.82 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243395 s, 43.1 MB/s 2023-11-24T08:08:06.050 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.050 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.107 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.83 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237828 s, 44.1 MB/s 2023-11-24T08:08:06.189 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.189 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.84 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206938 s, 50.7 MB/s 2023-11-24T08:08:06.314 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.314 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.85 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245338 s, 42.7 MB/s 2023-11-24T08:08:06.448 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.448 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.505 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.86 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303002 s, 34.6 MB/s 2023-11-24T08:08:06.591 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.591 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.646 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.87 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.721 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205638 s, 51.0 MB/s 2023-11-24T08:08:06.723 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.723 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.781 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.88 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.857 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230089 s, 45.6 MB/s 2023-11-24T08:08:06.858 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.858 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:06.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:06.913 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.89 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255649 s, 41.0 MB/s 2023-11-24T08:08:06.994 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:06.994 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.049 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.90 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.123 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.123 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.123 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238562 s, 44.0 MB/s 2023-11-24T08:08:07.125 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.125 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.182 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.91 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.254 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.255 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.255 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188585 s, 55.6 MB/s 2023-11-24T08:08:07.256 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.256 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.314 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.92 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019649 s, 53.4 MB/s 2023-11-24T08:08:07.384 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.384 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.93 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.532 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.532 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.533 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310996 s, 33.7 MB/s 2023-11-24T08:08:07.534 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.534 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.589 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.94 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.659 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.659 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.659 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190647 s, 55.0 MB/s 2023-11-24T08:08:07.660 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.660 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.718 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.95 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.792 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.793 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.793 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198901 s, 52.7 MB/s 2023-11-24T08:08:07.794 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.794 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.96 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:07.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:07.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:07.931 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263692 s, 39.8 MB/s 2023-11-24T08:08:07.933 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:07.933 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:07.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:07.988 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.97 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183629 s, 57.1 MB/s 2023-11-24T08:08:08.060 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.060 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.98 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218912 s, 47.9 MB/s 2023-11-24T08:08:08.192 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.192 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.99 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.334 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.334 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.334 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253196 s, 41.4 MB/s 2023-11-24T08:08:08.336 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.336 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.100 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211267 s, 49.6 MB/s 2023-11-24T08:08:08.468 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.468 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.523 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.101 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166703 s, 62.9 MB/s 2023-11-24T08:08:08.594 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.594 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.647 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.102 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196207 s, 53.4 MB/s 2023-11-24T08:08:08.769 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.769 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.103 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189654 s, 55.3 MB/s 2023-11-24T08:08:08.902 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:08.902 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:08.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:08.958 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.104 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.030 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.031 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.031 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218257 s, 48.0 MB/s 2023-11-24T08:08:09.032 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.032 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.090 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.105 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182856 s, 57.3 MB/s 2023-11-24T08:08:09.164 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.164 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.222 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.106 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.302 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.302 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.302 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233533 s, 44.9 MB/s 2023-11-24T08:08:09.303 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.303 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.107 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.437 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.437 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.437 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215972 s, 48.6 MB/s 2023-11-24T08:08:09.439 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.439 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.493 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.108 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223051 s, 47.0 MB/s 2023-11-24T08:08:09.566 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.567 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.622 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.109 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:08:09.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247196 s, 42.4 MB/s 2023-11-24T08:08:09.699 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.699 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.788 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.110 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:09.926 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:09.926 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:09.926 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260954 s, 40.2 MB/s 2023-11-24T08:08:09.927 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:09.927 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:09.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:09.987 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.111 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.063 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019897 s, 52.7 MB/s 2023-11-24T08:08:10.065 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.065 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.122 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.112 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.247 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.247 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.247 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248097 s, 42.3 MB/s 2023-11-24T08:08:10.249 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.249 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.113 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.383 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.384 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.384 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211308 s, 49.6 MB/s 2023-11-24T08:08:10.385 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.385 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.114 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200344 s, 52.3 MB/s 2023-11-24T08:08:10.519 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.520 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.579 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.115 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232845 s, 45.0 MB/s 2023-11-24T08:08:10.655 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.656 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.116 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207335 s, 50.6 MB/s 2023-11-24T08:08:10.788 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.788 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.844 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.117 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:10.910 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:10.910 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:10.910 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165911 s, 63.2 MB/s 2023-11-24T08:08:10.911 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:10.911 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:10.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:10.963 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.118 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253173 s, 41.4 MB/s 2023-11-24T08:08:11.044 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.044 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.102 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.119 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239216 s, 43.8 MB/s 2023-11-24T08:08:11.179 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.180 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.238 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.120 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.320 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.320 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.321 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241037 s, 43.5 MB/s 2023-11-24T08:08:11.322 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.322 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.380 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.121 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198845 s, 52.7 MB/s 2023-11-24T08:08:11.458 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.458 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.516 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.122 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027487 s, 38.1 MB/s 2023-11-24T08:08:11.598 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.598 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.654 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.123 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.730 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.730 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.731 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025275 s, 41.5 MB/s 2023-11-24T08:08:11.732 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.732 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.789 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.124 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:11.860 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:11.861 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:11.861 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203666 s, 51.5 MB/s 2023-11-24T08:08:11.862 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:11.863 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:11.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:11.914 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.125 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215524 s, 48.7 MB/s 2023-11-24T08:08:12.039 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.039 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.098 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.126 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.176 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.177 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023538 s, 44.5 MB/s 2023-11-24T08:08:12.178 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.178 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.232 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.127 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.359 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.359 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.359 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264027 s, 39.7 MB/s 2023-11-24T08:08:12.360 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.361 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.128 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.499 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.499 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.499 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231638 s, 45.3 MB/s 2023-11-24T08:08:12.500 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.500 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.556 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.129 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233776 s, 44.9 MB/s 2023-11-24T08:08:12.634 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.634 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.690 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.130 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.766 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.766 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.766 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209297 s, 50.1 MB/s 2023-11-24T08:08:12.767 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.768 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.825 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.131 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:12.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:12.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:12.899 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203227 s, 51.6 MB/s 2023-11-24T08:08:12.900 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:12.900 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:12.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:12.952 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.132 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.017 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.017 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.017 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152906 s, 68.6 MB/s 2023-11-24T08:08:13.018 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.018 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.067 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.133 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.193 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241349 s, 43.4 MB/s 2023-11-24T08:08:13.193 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.193 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.247 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.134 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204001 s, 51.4 MB/s 2023-11-24T08:08:13.369 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.369 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.425 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.135 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021151 s, 49.6 MB/s 2023-11-24T08:08:13.504 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.504 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.558 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.136 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.685 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.685 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.685 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278713 s, 37.6 MB/s 2023-11-24T08:08:13.687 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.687 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.746 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.137 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212936 s, 49.2 MB/s 2023-11-24T08:08:13.820 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.820 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:13.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:13.879 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.138 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201348 s, 52.1 MB/s 2023-11-24T08:08:13.955 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:13.955 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.011 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.139 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227476 s, 46.1 MB/s 2023-11-24T08:08:14.093 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.093 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.140 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222173 s, 47.2 MB/s 2023-11-24T08:08:14.232 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.233 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.290 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.141 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.365 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.365 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.365 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213537 s, 49.1 MB/s 2023-11-24T08:08:14.366 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.366 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.420 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.142 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.491 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206802 s, 50.7 MB/s 2023-11-24T08:08:14.493 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.493 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.143 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270336 s, 38.8 MB/s 2023-11-24T08:08:14.631 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.631 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.688 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.144 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.763 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.763 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.763 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207611 s, 50.5 MB/s 2023-11-24T08:08:14.764 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.764 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:14.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:14.819 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.145 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:14.946 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:14.946 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:14.946 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275733 s, 38.0 MB/s 2023-11-24T08:08:14.947 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:14.948 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.008 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.146 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.088 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.088 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.088 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219404 s, 47.8 MB/s 2023-11-24T08:08:15.089 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.089 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.144 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.147 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.208 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.208 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161785 s, 64.8 MB/s 2023-11-24T08:08:15.209 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.209 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.148 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202464 s, 51.8 MB/s 2023-11-24T08:08:15.380 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.380 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.437 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.149 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.515 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.515 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022348 s, 46.9 MB/s 2023-11-24T08:08:15.516 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.516 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.571 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.150 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.647 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.647 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.647 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02437 s, 43.0 MB/s 2023-11-24T08:08:15.649 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.649 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.705 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.151 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223643 s, 46.9 MB/s 2023-11-24T08:08:15.783 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.783 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.836 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.152 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189168 s, 55.4 MB/s 2023-11-24T08:08:15.906 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:15.906 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:15.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:15.953 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.153 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.079 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.079 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.079 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244116 s, 43.0 MB/s 2023-11-24T08:08:16.081 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.081 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.135 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.154 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.209 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.209 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.209 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220934 s, 47.5 MB/s 2023-11-24T08:08:16.211 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.211 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.155 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.341 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.341 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.341 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209767 s, 50.0 MB/s 2023-11-24T08:08:16.343 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.343 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.156 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.472 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.472 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.472 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021851 s, 48.0 MB/s 2023-11-24T08:08:16.474 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.474 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.157 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231739 s, 45.2 MB/s 2023-11-24T08:08:16.604 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.604 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.660 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.158 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.738 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.738 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.738 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233095 s, 45.0 MB/s 2023-11-24T08:08:16.740 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.740 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.796 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.159 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224017 s, 46.8 MB/s 2023-11-24T08:08:16.878 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:16.879 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:16.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:16.933 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.160 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.007 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.007 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.007 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205199 s, 51.1 MB/s 2023-11-24T08:08:17.009 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.009 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.161 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.140 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.140 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202629 s, 51.7 MB/s 2023-11-24T08:08:17.141 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.141 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.198 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.162 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.276 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.277 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.277 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247784 s, 42.3 MB/s 2023-11-24T08:08:17.278 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.278 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.163 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211329 s, 49.6 MB/s 2023-11-24T08:08:17.415 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.416 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.164 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260654 s, 40.2 MB/s 2023-11-24T08:08:17.551 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.551 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.165 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214604 s, 48.9 MB/s 2023-11-24T08:08:17.684 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.684 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.738 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.166 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.861 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.861 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.861 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231086 s, 45.4 MB/s 2023-11-24T08:08:17.863 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.863 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:17.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:17.916 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.167 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:17.987 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:17.987 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:17.987 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186553 s, 56.2 MB/s 2023-11-24T08:08:17.988 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:17.988 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.040 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.168 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.116 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.116 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.116 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228796 s, 45.8 MB/s 2023-11-24T08:08:18.118 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.118 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.175 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.169 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210457 s, 49.8 MB/s 2023-11-24T08:08:18.253 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.253 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.170 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.431 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.431 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.431 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215349 s, 48.7 MB/s 2023-11-24T08:08:18.433 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.433 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.489 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.171 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.566 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.566 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022548 s, 46.5 MB/s 2023-11-24T08:08:18.567 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.567 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.622 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.172 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.694 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.695 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.695 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216178 s, 48.5 MB/s 2023-11-24T08:08:18.696 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.696 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.752 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.173 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256742 s, 40.8 MB/s 2023-11-24T08:08:18.833 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.834 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:18.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:18.891 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.174 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:18.966 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:18.966 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:18.966 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02405 s, 43.6 MB/s 2023-11-24T08:08:18.968 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:18.968 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.020 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.175 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.094 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.094 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.094 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211592 s, 49.6 MB/s 2023-11-24T08:08:19.095 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.095 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.152 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.176 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143997 s, 72.8 MB/s 2023-11-24T08:08:19.216 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.217 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.268 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.177 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232656 s, 45.1 MB/s 2023-11-24T08:08:19.343 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.344 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.178 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.475 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023621 s, 44.4 MB/s 2023-11-24T08:08:19.478 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.478 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.539 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.179 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212702 s, 49.3 MB/s 2023-11-24T08:08:19.613 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.613 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.670 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.180 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025035 s, 41.9 MB/s 2023-11-24T08:08:19.750 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.750 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.807 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.181 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:19.883 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:19.884 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:19.884 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231794 s, 45.2 MB/s 2023-11-24T08:08:19.885 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:19.885 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:19.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:19.943 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.182 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200765 s, 52.2 MB/s 2023-11-24T08:08:20.018 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.018 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.183 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.147 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.147 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.147 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207434 s, 50.5 MB/s 2023-11-24T08:08:20.149 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.149 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.203 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.184 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020621 s, 50.8 MB/s 2023-11-24T08:08:20.325 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.325 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.381 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.185 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.452 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.453 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.453 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190096 s, 55.2 MB/s 2023-11-24T08:08:20.455 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.455 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.509 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.186 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241019 s, 43.5 MB/s 2023-11-24T08:08:20.583 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.583 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.640 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.187 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.719 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.720 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.720 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240471 s, 43.6 MB/s 2023-11-24T08:08:20.721 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.721 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.779 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.188 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:20.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236469 s, 44.3 MB/s 2023-11-24T08:08:20.869 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:20.869 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:20.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:20.927 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.189 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246077 s, 42.6 MB/s 2023-11-24T08:08:21.006 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.006 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.065 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.190 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175972 s, 59.6 MB/s 2023-11-24T08:08:21.138 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.138 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.194 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.191 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244321 s, 42.9 MB/s 2023-11-24T08:08:21.272 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.272 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.328 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.192 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.397 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.398 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.398 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203046 s, 51.6 MB/s 2023-11-24T08:08:21.401 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.401 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.193 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234899 s, 44.6 MB/s 2023-11-24T08:08:21.580 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.581 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.639 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.194 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.720 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.720 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.720 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268233 s, 39.1 MB/s 2023-11-24T08:08:21.721 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.721 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.195 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188278 s, 55.7 MB/s 2023-11-24T08:08:21.846 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:21.847 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:21.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:21.898 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.196 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218143 s, 48.1 MB/s 2023-11-24T08:08:22.022 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:22.022 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:22.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:22.080 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.197 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:22.158 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:22.159 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:22.159 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243066 s, 43.1 MB/s 2023-11-24T08:08:22.160 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:22.160 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:22.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:22.217 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.198 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224424 s, 46.7 MB/s 2023-11-24T08:08:22.296 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:22.296 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:22.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:22.352 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1626/12252aa6-e473-4e38-a961-e53c690dabb3/subvolume_file.199 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:08:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:08:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:08:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218175 s, 48.1 MB/s 2023-11-24T08:08:22.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol1626 subvol_snap5902 2023-11-24T08:08:22.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.654+0000 7f20a3fff640 1 Processor -- start 2023-11-24T08:08:22.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.654+0000 7f20a3fff640 1 -- start start 2023-11-24T08:08:22.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20a4144510 0x7f20a406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20a40b11f0 0x7f20a40b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a41448f0 con 0x7f20a415edd0 2023-11-24T08:08:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a3fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a4144d80 con 0x7f20a40b11f0 2023-11-24T08:08:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a3fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a415f2d0 con 0x7f20a4144510 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20a40b11f0 0x7f20a40b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20a40b11f0 0x7f20a40b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43686/0 (socket says 172.21.15.107:43686) 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a37fe640 1 -- 172.21.15.107:0/1574765069 learned_addr learned my addr 172.21.15.107:0/1574765069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.656+0000 7f20a2ffd640 1 --2- 172.21.15.107:0/1574765069 >> v2:172.21.15.120:3300/0 conn(0x7f20a4144510 0x7f20a406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.657+0000 7f20a37fe640 1 -- 172.21.15.107:0/1574765069 >> v2:172.21.15.120:3300/0 conn(0x7f20a4144510 msgr2=0x7f20a406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.657+0000 7f20a37fe640 1 --2- 172.21.15.107:0/1574765069 >> v2:172.21.15.120:3300/0 conn(0x7f20a4144510 0x7f20a406d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.657+0000 7f20a37fe640 1 -- 172.21.15.107:0/1574765069 >> v1:172.21.15.107:6789/0 conn(0x7f20a415edd0 legacy=0x7f20a40b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.657+0000 7f20a37fe640 1 -- 172.21.15.107:0/1574765069 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a415f5b0 con 0x7f20a40b11f0 2023-11-24T08:08:22.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.658+0000 7f20a37fe640 1 --2- 172.21.15.107:0/1574765069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a40b11f0 0x7f20a40b4180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f20a40b1730 tx=0x7f209c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dabbdbf15bf4ef3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.658+0000 7f20a1ffb640 1 -- 172.21.15.107:0/1574765069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c02ca80 con 0x7f20a40b11f0 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.658+0000 7f20a1ffb640 1 -- 172.21.15.107:0/1574765069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c02cbe0 con 0x7f20a40b11f0 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.658+0000 7f20a1ffb640 1 -- 172.21.15.107:0/1574765069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0343f0 con 0x7f20a40b11f0 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 -- 172.21.15.107:0/1574765069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a40b11f0 msgr2=0x7f20a40b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 --2- 172.21.15.107:0/1574765069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a40b11f0 0x7f20a40b4180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f20a40b1730 tx=0x7f209c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 -- 172.21.15.107:0/1574765069 shutdown_connections 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 --2- 172.21.15.107:0/1574765069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a40b11f0 0x7f20a40b4180 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 --2- 172.21.15.107:0/1574765069 >> v2:172.21.15.120:3300/0 conn(0x7f20a4144510 0x7f20a406d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 -- 172.21.15.107:0/1574765069 >> 172.21.15.107:0/1574765069 conn(0x7f20a40cb8a0 msgr2=0x7f20a40cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 -- 172.21.15.107:0/1574765069 shutdown_connections 2023-11-24T08:08:22.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.659+0000 7f20a3fff640 1 -- 172.21.15.107:0/1574765069 wait complete. 2023-11-24T08:08:22.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.660+0000 7f20a3fff640 1 Processor -- start 2023-11-24T08:08:22.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.661+0000 7f20a3fff640 1 -- start start 2023-11-24T08:08:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.661+0000 7f20a3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20a40b11f0 0x7f20a413a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.661+0000 7f20a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20a415edd0 0x7f20a40afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4070c20 con 0x7f20a4144510 2023-11-24T08:08:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40af4a0 con 0x7f20a40b11f0 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4070e70 con 0x7f20a415edd0 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43696/0 (socket says 172.21.15.107:43696) 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 -- 172.21.15.107:0/19647123 learned_addr learned my addr 172.21.15.107:0/19647123 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 -- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20a415edd0 msgr2=0x7f20a40afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20a415edd0 0x7f20a40afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a2ffd640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20a40b11f0 0x7f20a413a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 -- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20a40b11f0 msgr2=0x7f20a413a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20a40b11f0 0x7f20a413a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.662+0000 7f20a27fc640 1 -- 172.21.15.107:0/19647123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c009b30 con 0x7f20a4144510 2023-11-24T08:08:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f20a27fc640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f209800c3c0 tx=0x7f209800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980035f0 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098003750 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a4071150 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098003b40 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.663+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a40af120 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.664+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a4071670 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.665+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) v1 ==== 89537+0+0 (secure 0 0 0) 0x7f2098003ca0 con 0x7f20a4144510 2023-11-24T08:08:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.666+0000 7f208b7fe640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 0x7f209006d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.666+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2098089040 con 0x7f20a4144510 2023-11-24T08:08:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.666+0000 7f20a2ffd640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 0x7f209006d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.667+0000 7f20a2ffd640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 0x7f209006d660 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2094006fd0 tx=0x7f2094008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:22.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.671+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20980bbe90 con 0x7f20a4144510 2023-11-24T08:08:22.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.895+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol1626", "snap_name": "subvol_snap5902", "target": ["mon-mgr", ""]}) v1 -- 0x7f20a40b3c30 con 0x7f209006b1a0 2023-11-24T08:08:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.947+0000 7f208b7fe640 1 -- 172.21.15.107:0/19647123 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2094009d60 con 0x7f209006b1a0 2023-11-24T08:08:22.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 msgr2=0x7f209006d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 0x7f209006d660 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2094006fd0 tx=0x7f2094008040 comp rx=0 tx=0).stop 2023-11-24T08:08:22.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 msgr2=0x7f20a40af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:22.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f209800c3c0 tx=0x7f209800c880 comp rx=0 tx=0).stop 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 shutdown_connections 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f209006b1a0 0x7f209006d660 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20a415edd0 0x7f20a40afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a4144510 0x7f20a40af880 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 --2- 172.21.15.107:0/19647123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20a40b11f0 0x7f20a413a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.951+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 >> 172.21.15.107:0/19647123 conn(0x7f20a40cb8a0 msgr2=0x7f20a40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.952+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 shutdown_connections 2023-11-24T08:08:22.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:22.952+0000 7f20a3fff640 1 -- 172.21.15.107:0/19647123 wait complete. 2023-11-24T08:08:22.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 5 2023-11-24T08:08:23.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.162+0000 7f46dd7fa640 1 Processor -- start 2023-11-24T08:08:23.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.163+0000 7f46dd7fa640 1 -- start start 2023-11-24T08:08:23.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dd7fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d80b0b40 0x7f46d80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dd7fa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46d815ded0 0x7f46d815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dd7fa640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d806d430 con 0x7f46d815a1c0 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dd7fa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d815ec50 con 0x7f46d815ded0 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dd7fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d806cb00 con 0x7f46d80b0b40 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dcff9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46d815ded0 0x7f46d815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dcff9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46d815ded0 0x7f46d815e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43700/0 (socket says 172.21.15.107:43700) 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.164+0000 7f46dcff9640 1 -- 172.21.15.107:0/1104316248 learned_addr learned my addr 172.21.15.107:0/1104316248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:23.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.165+0000 7f46cffff640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d80b0b40 0x7f46d80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.165+0000 7f46dcff9640 1 -- 172.21.15.107:0/1104316248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d80b0b40 msgr2=0x7f46d80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.165+0000 7f46dcff9640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d80b0b40 0x7f46d80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.165+0000 7f46dcff9640 1 -- 172.21.15.107:0/1104316248 >> v1:172.21.15.120:6789/0 conn(0x7f46d815a1c0 legacy=0x7f46d806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.165+0000 7f46dcff9640 1 -- 172.21.15.107:0/1104316248 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80b14e0 con 0x7f46d815ded0 2023-11-24T08:08:23.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.166+0000 7f46dcff9640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d815ded0 0x7f46d815e360 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3957fb8299233957 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.166+0000 7f46d77fe640 1 -- 172.21.15.107:0/1104316248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d002caf0 con 0x7f46d815ded0 2023-11-24T08:08:23.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.166+0000 7f46d77fe640 1 -- 172.21.15.107:0/1104316248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d002cc50 con 0x7f46d815ded0 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.166+0000 7f46d77fe640 1 -- 172.21.15.107:0/1104316248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00343f0 con 0x7f46d815ded0 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1104316248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d815ded0 msgr2=0x7f46d815e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d815ded0 0x7f46d815e360 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d002bf50 comp rx=0 tx=0).stop 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1104316248 shutdown_connections 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d815ded0 0x7f46d815e360 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1104316248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d80b0b40 0x7f46d80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1104316248 >> 172.21.15.107:0/1104316248 conn(0x7f46d80cbb40 msgr2=0x7f46d80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1104316248 shutdown_connections 2023-11-24T08:08:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.167+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1104316248 wait complete. 2023-11-24T08:08:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.169+0000 7f46dd7fa640 1 Processor -- start 2023-11-24T08:08:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.169+0000 7f46dd7fa640 1 -- start start 2023-11-24T08:08:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46d815a1c0 0x7f46d815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d815ded0 0x7f46d81594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d815a7d0 con 0x7f46d80b0b40 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d815d770 con 0x7f46d815a1c0 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46dd7fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8159a20 con 0x7f46d815ded0 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43710/0 (socket says 172.21.15.107:43710) 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 -- 172.21.15.107:0/1711062802 learned_addr learned my addr 172.21.15.107:0/1711062802 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 -- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d815ded0 msgr2=0x7f46d81594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d815ded0 0x7f46d81594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46cffff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46d815a1c0 0x7f46d815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.170+0000 7f46d7fff640 1 -- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46d815a1c0 msgr2=0x7f46d815d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d7fff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46d815a1c0 0x7f46d815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d7fff640 1 -- 172.21.15.107:0/1711062802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0009b30 con 0x7f46d80b0b40 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d7fff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f46c800c450 tx=0x7f46c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80052d0 con 0x7f46d80b0b40 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8005430 con 0x7f46d80b0b40 2023-11-24T08:08:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8012870 con 0x7f46d80b0b40 2023-11-24T08:08:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d815ea20 con 0x7f46d80b0b40 2023-11-24T08:08:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.171+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d8158f20 con 0x7f46d80b0b40 2023-11-24T08:08:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.172+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d804e840 con 0x7f46d80b0b40 2023-11-24T08:08:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.173+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) v1 ==== 89537+0+0 (secure 0 0 0) 0x7f46c80129d0 con 0x7f46d80b0b40 2023-11-24T08:08:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.174+0000 7f46d57fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 0x7f46b806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.175+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46c8089730 con 0x7f46d80b0b40 2023-11-24T08:08:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.175+0000 7f46cffff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 0x7f46b806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.175+0000 7f46cffff640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 0x7f46b806d5d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.179+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46c8054340 con 0x7f46d80b0b40 2023-11-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.371+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f46d80ca020 con 0x7f46d80b0b40 2023-11-24T08:08:23.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.376+0000 7f46d57fa640 1 -- 172.21.15.107:0/1711062802 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 130+0+0 (secure 0 0 0) 0x7f46c8057960 con 0x7f46d80b0b40 2023-11-24T08:08:23.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 msgr2=0x7f46b806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 0x7f46b806d5d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0008040 comp rx=0 tx=0).stop 2023-11-24T08:08:23.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 msgr2=0x7f46d8157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f46c800c450 tx=0x7f46c800c910 comp rx=0 tx=0).stop 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 shutdown_connections 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46b806b110 0x7f46b806d5d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46d815ded0 0x7f46d81594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46d815a1c0 0x7f46d815d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 --2- 172.21.15.107:0/1711062802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d80b0b40 0x7f46d8157c30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 >> 172.21.15.107:0/1711062802 conn(0x7f46d80cbb40 msgr2=0x7f46d80ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 shutdown_connections 2023-11-24T08:08:23.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.379+0000 7f46dd7fa640 1 -- 172.21.15.107:0/1711062802 wait complete. 2023-11-24T08:08:23.397 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:23.395+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:23.397 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:23.395+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:23.397 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:23.395+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:23.398 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:23.395+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:23.398 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:23.395+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:23.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol1626 subvol_snap5902 subvol_clone2533 2023-11-24T08:08:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.594+0000 7ff4e04e7640 1 Processor -- start 2023-11-24T08:08:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.594+0000 7ff4e04e7640 1 -- start start 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4e04e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d815f530 0x7ff4d815f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4e04e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4d80cf4f0 0x7ff4d813f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4e04e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d813eb00 con 0x7ff4d806ff70 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4e04e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d813ede0 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4e04e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d815fe50 con 0x7ff4d815f530 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4dea84640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4d80cf4f0 0x7ff4d813f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4dda82640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff4d806ff70 0x7ff4d80cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50110/0 (socket says 172.21.15.107:50110) 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.596+0000 7ff4dda82640 1 -- 172.21.15.107:0/1435740500 learned_addr learned my addr 172.21.15.107:0/1435740500 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4de283640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d815f530 0x7ff4d815f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4dea84640 1 -- 172.21.15.107:0/1435740500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d815f530 msgr2=0x7ff4d815f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4dea84640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d815f530 0x7ff4d815f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4dea84640 1 -- 172.21.15.107:0/1435740500 >> v1:172.21.15.107:6789/0 conn(0x7ff4d806ff70 legacy=0x7ff4d80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4dea84640 1 -- 172.21.15.107:0/1435740500 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8070750 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.597+0000 7ff4dd281640 1 -- 172.21.15.107:0/1435740500 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671104262 0 0) 0x7ff4d813eb00 con 0x7ff4d806ff70 2023-11-24T08:08:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.598+0000 7ff4dea84640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cf4f0 0x7ff4d813f270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009d90 tx=0x7ff4d002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455b664051bf9c3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.598+0000 7ff4dd281640 1 -- 172.21.15.107:0/1435740500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d002cbc0 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.598+0000 7ff4dd281640 1 -- 172.21.15.107:0/1435740500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d002cd20 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.598+0000 7ff4dd281640 1 -- 172.21.15.107:0/1435740500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0034660 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1435740500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cf4f0 msgr2=0x7ff4d813f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cf4f0 0x7ff4d813f270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009d90 tx=0x7ff4d002be10 comp rx=0 tx=0).stop 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1435740500 shutdown_connections 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cf4f0 0x7ff4d813f270 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1435740500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d815f530 0x7ff4d815f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1435740500 >> 172.21.15.107:0/1435740500 conn(0x7ff4d80b2e40 msgr2=0x7ff4d80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1435740500 shutdown_connections 2023-11-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.599+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1435740500 wait complete. 2023-11-24T08:08:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.601+0000 7ff4e04e7640 1 Processor -- start 2023-11-24T08:08:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.601+0000 7ff4e04e7640 1 -- start start 2023-11-24T08:08:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d80cf4f0 0x7ff4d8135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d815f530 0x7ff4d81581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8135f90 con 0x7ff4d806ff70 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d80cc890 con 0x7ff4d815f530 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4e04e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d80ccb70 con 0x7ff4d80cf4f0 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4de283640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4de283640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43744/0 (socket says 172.21.15.107:43744) 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.602+0000 7ff4de283640 1 -- 172.21.15.107:0/1791555387 learned_addr learned my addr 172.21.15.107:0/1791555387 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4dda82640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d80cf4f0 0x7ff4d8135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 -- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d80cf4f0 msgr2=0x7ff4d8135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d80cf4f0 0x7ff4d8135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 -- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d815f530 msgr2=0x7ff4d81581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d815f530 0x7ff4d81581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 -- 172.21.15.107:0/1791555387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d0009a70 con 0x7ff4d806ff70 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4de283640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400d5f0 tx=0x7ff4d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40125f0 con 0x7ff4d806ff70 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4012750 con 0x7ff4d806ff70 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d80cce50 con 0x7ff4d806ff70 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.603+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4012b00 con 0x7ff4d806ff70 2023-11-24T08:08:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.604+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d80cd410 con 0x7ff4d806ff70 2023-11-24T08:08:23.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.605+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d804e8b0 con 0x7ff4d806ff70 2023-11-24T08:08:23.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.605+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) v1 ==== 89537+0+0 (secure 0 0 0) 0x7ff4d4012c60 con 0x7ff4d806ff70 2023-11-24T08:08:23.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.606+0000 7ff4ceffd640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 0x7ff4bc06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:23.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.606+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4d408a160 con 0x7ff4d806ff70 2023-11-24T08:08:23.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.607+0000 7ff4dda82640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 0x7ff4bc06d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:23.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.607+0000 7ff4dda82640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 0x7ff4bc06d690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007fd0 tx=0x7ff4c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:23.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.611+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4d4054d30 con 0x7ff4d806ff70 2023-11-24T08:08:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:23.837+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol1626", "snap_name": "subvol_snap5902", "target_sub_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d80703d0 con 0x7ff4bc06b1d0 2023-11-24T08:08:24.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.003+0000 7ff4ceffd640 1 -- 172.21.15.107:0/1791555387 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff4c8001c70 con 0x7ff4bc06b1d0 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.006+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 msgr2=0x7ff4bc06d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.006+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 0x7ff4bc06d690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007fd0 tx=0x7ff4c8009040 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.006+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 msgr2=0x7ff4d80cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.006+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400d5f0 tx=0x7ff4d400dab0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 shutdown_connections 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06b1d0 0x7ff4bc06d690 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d815f530 0x7ff4d81581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d80cf4f0 0x7ff4d8135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 --2- 172.21.15.107:0/1791555387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d806ff70 0x7ff4d80cd930 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 >> 172.21.15.107:0/1791555387 conn(0x7ff4d80b2e40 msgr2=0x7ff4d80c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:24.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 shutdown_connections 2023-11-24T08:08:24.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.007+0000 7ff4e04e7640 1 -- 172.21.15.107:0/1791555387 wait complete. 2023-11-24T08:08:24.012 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.010+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.012 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.010+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.012 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.010+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.012 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.010+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.012 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.010+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:24.037 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.035+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.037 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.035+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.037 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.035+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.037 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.035+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.037 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:24.035+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:24.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.220+0000 7f982bcb4640 1 Processor -- start 2023-11-24T08:08:24.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.220+0000 7f982bcb4640 1 -- start start 2023-11-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982bcb4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f982406d600 0x7f98240cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982bcb4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982415e400 con 0x7f98240c4220 2023-11-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982bcb4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240c46e0 con 0x7f982415d7f0 2023-11-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982bcb4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98240ce880 con 0x7f982406d600 2023-11-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982a251640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f982406d600 0x7f98240cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982a251640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f982406d600 0x7f98240cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43784/0 (socket says 172.21.15.107:43784) 2023-11-24T08:08:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.222+0000 7f982a251640 1 -- 172.21.15.107:0/666950714 learned_addr learned my addr 172.21.15.107:0/666950714 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.223+0000 7f982a251640 1 -- 172.21.15.107:0/666950714 >> v1:172.21.15.120:6789/0 conn(0x7f982415d7f0 legacy=0x7f98240c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.223+0000 7f982a251640 1 -- 172.21.15.107:0/666950714 >> v1:172.21.15.107:6789/0 conn(0x7f98240c4220 legacy=0x7f98240cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.223+0000 7f982a251640 1 -- 172.21.15.107:0/666950714 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240c49c0 con 0x7f982406d600 2023-11-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.224+0000 7f982a251640 1 --2- 172.21.15.107:0/666950714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982406d600 0x7f98240cb2c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f981c009e50 tx=0x7f981c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2e99ac281dcc14 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.224+0000 7f9828a4e640 1 -- 172.21.15.107:0/666950714 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c02cb20 con 0x7f982406d600 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.224+0000 7f9828a4e640 1 -- 172.21.15.107:0/666950714 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c02cc80 con 0x7f982406d600 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.224+0000 7f9828a4e640 1 -- 172.21.15.107:0/666950714 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0343f0 con 0x7f982406d600 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 -- 172.21.15.107:0/666950714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982406d600 msgr2=0x7f98240cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 --2- 172.21.15.107:0/666950714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982406d600 0x7f98240cb2c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f981c009e50 tx=0x7f981c02bf80 comp rx=0 tx=0).stop 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 -- 172.21.15.107:0/666950714 shutdown_connections 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 --2- 172.21.15.107:0/666950714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982406d600 0x7f98240cb2c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 -- 172.21.15.107:0/666950714 >> 172.21.15.107:0/666950714 conn(0x7f98240b0c50 msgr2=0x7f98240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 -- 172.21.15.107:0/666950714 shutdown_connections 2023-11-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.225+0000 7f982bcb4640 1 -- 172.21.15.107:0/666950714 wait complete. 2023-11-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.226+0000 7f982bcb4640 1 Processor -- start 2023-11-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.226+0000 7f982bcb4640 1 -- start start 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982406d600 0x7f98240b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98240c4220 0x7f982414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b2e90 con 0x7f982415d7f0 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240ac0b0 con 0x7f982406d600 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.227+0000 7f982bcb4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b4890 con 0x7f98240c4220 2023-11-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43794/0 (socket says 172.21.15.107:43794) 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 -- 172.21.15.107:0/3498179597 learned_addr learned my addr 172.21.15.107:0/3498179597 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 -- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98240c4220 msgr2=0x7f982414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f9829a50640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982406d600 0x7f98240b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98240c4220 0x7f982414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 -- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982406d600 msgr2=0x7f98240b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982406d600 0x7f98240b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 -- 172.21.15.107:0/3498179597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c009b30 con 0x7f982415d7f0 2023-11-24T08:08:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.228+0000 7f982a251640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f981c036040 tx=0x7f981c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.229+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c009510 con 0x7f982415d7f0 2023-11-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.229+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c007e50 con 0x7f982415d7f0 2023-11-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.229+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c03e690 con 0x7f982415d7f0 2023-11-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.229+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98240ac370 con 0x7f982415d7f0 2023-11-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.229+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982414f530 con 0x7f982415d7f0 2023-11-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.230+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98240ac890 con 0x7f982415d7f0 2023-11-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.231+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) v1 ==== 89537+0+0 (secure 0 0 0) 0x7f981c03e7f0 con 0x7f982415d7f0 2023-11-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.231+0000 7f98167fc640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 0x7f980806d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.232+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f981c0ac800 con 0x7f982415d7f0 2023-11-24T08:08:24.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.232+0000 7f9829a50640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 0x7f980806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:24.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.232+0000 7f9829a50640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 0x7f980806d660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f982000a750 tx=0x7f9820008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:24.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.236+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98240ac890 con 0x7f982415d7f0 2023-11-24T08:08:24.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.399+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 10) v1 ==== 89588+0+0 (secure 0 0 0) 0x7f981c0778c0 con 0x7f982415d7f0 2023-11-24T08:08:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.452+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7f98240c5260 con 0x7f980806b1a0 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.463+0000 7f98167fc640 1 -- 172.21.15.107:0/3498179597 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f982000a380 con 0x7f980806b1a0 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:24.466 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:24.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:24.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:24.466 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 msgr2=0x7f980806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 0x7f980806d660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f982000a750 tx=0x7f9820008040 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 msgr2=0x7f982414f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f981c036040 tx=0x7f981c004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 shutdown_connections 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980806b1a0 0x7f980806d660 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f982415d7f0 0x7f982414f8d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98240c4220 0x7f982414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 --2- 172.21.15.107:0/3498179597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982406d600 0x7f98240b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.465+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 >> 172.21.15.107:0/3498179597 conn(0x7f98240b0c50 msgr2=0x7f982415b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.466+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 shutdown_connections 2023-11-24T08:08:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:24.466+0000 7f982bcb4640 1 -- 172.21.15.107:0/3498179597 wait complete. 2023-11-24T08:08:25.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:25.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.708+0000 7fd5a347c640 1 Processor -- start 2023-11-24T08:08:25.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.708+0000 7fd5a347c640 1 -- start start 2023-11-24T08:08:25.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a347c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd59c146eb0 0x7fd59c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a347c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd59c0cd2e0 0x7fd59c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a347c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c161a10 con 0x7fd59c06e160 2023-11-24T08:08:25.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a347c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c161cf0 con 0x7fd59c146eb0 2023-11-24T08:08:25.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a347c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c161fd0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a1218640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd59c146eb0 0x7fd59c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:25.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a1218640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd59c146eb0 0x7fd59c161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43810/0 (socket says 172.21.15.107:43810) 2023-11-24T08:08:25.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.710+0000 7fd5a1218640 1 -- 172.21.15.107:0/759265609 learned_addr learned my addr 172.21.15.107:0/759265609 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:25.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.711+0000 7fd5a1218640 1 -- 172.21.15.107:0/759265609 >> v2:172.21.15.120:3300/0 conn(0x7fd59c0cd2e0 msgr2=0x7fd59c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:25.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.711+0000 7fd5a1218640 1 --2- 172.21.15.107:0/759265609 >> v2:172.21.15.120:3300/0 conn(0x7fd59c0cd2e0 0x7fd59c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.711+0000 7fd5a1218640 1 -- 172.21.15.107:0/759265609 >> v1:172.21.15.120:6789/0 conn(0x7fd59c06e160 legacy=0x7fd59c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:25.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.711+0000 7fd5a1218640 1 -- 172.21.15.107:0/759265609 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c070660 con 0x7fd59c146eb0 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.712+0000 7fd5a1218640 1 --2- 172.21.15.107:0/759265609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c146eb0 0x7fd59c161150 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0ae150 tx=0x7fd59802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353e1be32c3a342d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.712+0000 7fd593fff640 1 -- 172.21.15.107:0/759265609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598031870 con 0x7fd59c146eb0 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.712+0000 7fd593fff640 1 -- 172.21.15.107:0/759265609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5980319d0 con 0x7fd59c146eb0 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.712+0000 7fd593fff640 1 -- 172.21.15.107:0/759265609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598031d60 con 0x7fd59c146eb0 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 -- 172.21.15.107:0/759265609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c146eb0 msgr2=0x7fd59c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 --2- 172.21.15.107:0/759265609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c146eb0 0x7fd59c161150 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0ae150 tx=0x7fd59802b910 comp rx=0 tx=0).stop 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 -- 172.21.15.107:0/759265609 shutdown_connections 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 --2- 172.21.15.107:0/759265609 >> v2:172.21.15.120:3300/0 conn(0x7fd59c0cd2e0 0x7fd59c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 --2- 172.21.15.107:0/759265609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c146eb0 0x7fd59c161150 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 -- 172.21.15.107:0/759265609 >> 172.21.15.107:0/759265609 conn(0x7fd59c0b0c50 msgr2=0x7fd59c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 -- 172.21.15.107:0/759265609 shutdown_connections 2023-11-24T08:08:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.713+0000 7fd5a347c640 1 -- 172.21.15.107:0/759265609 wait complete. 2023-11-24T08:08:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.715+0000 7fd5a347c640 1 Processor -- start 2023-11-24T08:08:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.715+0000 7fd5a347c640 1 -- start start 2023-11-24T08:08:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c06e160 0x7fd59c143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c146eb0 0x7fd59c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c15f5d0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c15e720 con 0x7fd59c06e160 2023-11-24T08:08:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a347c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c15e950 con 0x7fd59c146eb0 2023-11-24T08:08:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.716+0000 7fd5a0a17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43824/0 (socket says 172.21.15.107:43824) 2023-11-24T08:08:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 -- 172.21.15.107:0/1163031213 learned_addr learned my addr 172.21.15.107:0/1163031213 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 -- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c146eb0 msgr2=0x7fd59c0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c146eb0 0x7fd59c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 -- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c06e160 msgr2=0x7fd59c143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c06e160 0x7fd59c143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 -- 172.21.15.107:0/1163031213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980094d0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd5a0a17640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c410 tx=0x7fd58c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c0035f0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.717+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c003750 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.718+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c15ec30 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.718+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c003b40 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.718+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c15f1f0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.719+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c04e8b0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.720+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 10) v1 ==== 89588+0+0 (secure 0 0 0) 0x7fd58c003ca0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.721+0000 7fd591ffb640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 0x7fd58006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.721+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd58c088ec0 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.721+0000 7fd5a1218640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 0x7fd58006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.722+0000 7fd5a1218640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 0x7fd58006d720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd598036000 tx=0x7fd598030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.726+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd58c053a20 con 0x7fd59c0cd2e0 2023-11-24T08:08:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.951+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7fd59c147290 con 0x7fd58006b260 2023-11-24T08:08:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.959+0000 7fd591ffb640 1 -- 172.21.15.107:0/1163031213 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7fd598031c40 con 0x7fd58006b260 2023-11-24T08:08:25.961 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:25.961 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:25.961 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:25.962 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 msgr2=0x7fd58006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 0x7fd58006d720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd598036000 tx=0x7fd598030000 comp rx=0 tx=0).stop 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 msgr2=0x7fd59c0b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c410 tx=0x7fd58c00c8d0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 shutdown_connections 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd58006b260 0x7fd58006d720 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c146eb0 0x7fd59c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd59c0cd2e0 0x7fd59c0b26f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 --2- 172.21.15.107:0/1163031213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c06e160 0x7fd59c143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:25.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 >> 172.21.15.107:0/1163031213 conn(0x7fd59c0b0c50 msgr2=0x7fd59c146090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:25.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 shutdown_connections 2023-11-24T08:08:25.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:25.962+0000 7fd5a347c640 1 -- 172.21.15.107:0/1163031213 wait complete. 2023-11-24T08:08:26.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:27.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.208+0000 7f9530ae7640 1 Processor -- start 2023-11-24T08:08:27.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.208+0000 7f9530ae7640 1 -- start start 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f9530ae7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f952c146eb0 0x7f952c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f9530ae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c161a10 con 0x7f952c0cd2e0 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f9530ae7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c161cf0 con 0x7f952c06e160 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f9530ae7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c161fd0 con 0x7f952c146eb0 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f952a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f952c146eb0 0x7f952c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f952a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f952c146eb0 0x7f952c161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43828/0 (socket says 172.21.15.107:43828) 2023-11-24T08:08:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f952a59c640 1 -- 172.21.15.107:0/3207601225 learned_addr learned my addr 172.21.15.107:0/3207601225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f952a59c640 1 -- 172.21.15.107:0/3207601225 >> v1:172.21.15.120:6789/0 conn(0x7f952c06e160 legacy=0x7f952c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.210+0000 7f952a59c640 1 -- 172.21.15.107:0/3207601225 >> v1:172.21.15.107:6789/0 conn(0x7f952c0cd2e0 legacy=0x7f952c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.211+0000 7f952a59c640 1 -- 172.21.15.107:0/3207601225 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c070660 con 0x7f952c146eb0 2023-11-24T08:08:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.211+0000 7f9529d9b640 1 -- 172.21.15.107:0/3207601225 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957073094 0 0) 0x7f952c161a10 con 0x7f952c0cd2e0 2023-11-24T08:08:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.212+0000 7f952a59c640 1 --2- 172.21.15.107:0/3207601225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c146eb0 0x7f952c161150 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f951c009870 tx=0x7f951c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecfd46581c41991 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.212+0000 7f9529d9b640 1 -- 172.21.15.107:0/3207601225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c030860 con 0x7f952c146eb0 2023-11-24T08:08:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.212+0000 7f9529d9b640 1 -- 172.21.15.107:0/3207601225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c0309c0 con 0x7f952c146eb0 2023-11-24T08:08:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.212+0000 7f9529d9b640 1 -- 172.21.15.107:0/3207601225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c030d50 con 0x7f952c146eb0 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 -- 172.21.15.107:0/3207601225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c146eb0 msgr2=0x7f952c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3207601225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c146eb0 0x7f952c161150 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f951c009870 tx=0x7f951c02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 -- 172.21.15.107:0/3207601225 shutdown_connections 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3207601225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c146eb0 0x7f952c161150 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 -- 172.21.15.107:0/3207601225 >> 172.21.15.107:0/3207601225 conn(0x7f952c0b0c50 msgr2=0x7f952c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 -- 172.21.15.107:0/3207601225 shutdown_connections 2023-11-24T08:08:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.213+0000 7f9530ae7640 1 -- 172.21.15.107:0/3207601225 wait complete. 2023-11-24T08:08:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.215+0000 7f9530ae7640 1 Processor -- start 2023-11-24T08:08:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.215+0000 7f9530ae7640 1 -- start start 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f952c0cd2e0 0x7f952c0acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f952c146eb0 0x7f952c0ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c15e220 con 0x7f952c06e160 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c160940 con 0x7f952c0cd2e0 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.216+0000 7f9530ae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c09bbe0 con 0x7f952c146eb0 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43836/0 (socket says 172.21.15.107:43836) 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 -- 172.21.15.107:0/3817473720 learned_addr learned my addr 172.21.15.107:0/3817473720 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 -- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f952c146eb0 msgr2=0x7f952c0ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f952c146eb0 0x7f952c0ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 -- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f952c0cd2e0 msgr2=0x7f952c0acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f952c0cd2e0 0x7f952c0acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 -- 172.21.15.107:0/3817473720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0094d0 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.217+0000 7f952a59c640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f951c000c00 tx=0x7f951c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.218+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c039b60 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.218+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c039cc0 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.218+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c043630 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.218+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c09bec0 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.218+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c15deb0 con 0x7f952c06e160 2023-11-24T08:08:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.219+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c15e070 con 0x7f952c06e160 2023-11-24T08:08:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.220+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7f951c02f3b0 con 0x7f952c06e160 2023-11-24T08:08:27.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.221+0000 7f95237fe640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 0x7f950c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:27.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.221+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f951c0ac2f0 con 0x7f952c06e160 2023-11-24T08:08:27.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.221+0000 7f9521d9b640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 0x7f950c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.222+0000 7f9521d9b640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 0x7f950c06d810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f952c1f6810 tx=0x7f95140079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.225+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f951c076c40 con 0x7f952c06e160 2023-11-24T08:08:27.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.452+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7f952c147290 con 0x7f950c06b350 2023-11-24T08:08:27.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.462+0000 7f95237fe640 1 -- 172.21.15.107:0/3817473720 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f9514003950 con 0x7f950c06b350 2023-11-24T08:08:27.464 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:27.465 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 msgr2=0x7f950c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 0x7f950c06d810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f952c1f6810 tx=0x7f95140079b0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 msgr2=0x7f952c143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f951c000c00 tx=0x7f951c004730 comp rx=0 tx=0).stop 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 shutdown_connections 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f950c06b350 0x7f950c06d810 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f952c146eb0 0x7f952c0ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f952c0cd2e0 0x7f952c0acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 --2- 172.21.15.107:0/3817473720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f952c06e160 0x7f952c143af0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 >> 172.21.15.107:0/3817473720 conn(0x7f952c0b0c50 msgr2=0x7f952c0cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 shutdown_connections 2023-11-24T08:08:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:27.465+0000 7f9530ae7640 1 -- 172.21.15.107:0/3817473720 wait complete. 2023-11-24T08:08:28.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:28.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.709+0000 7f04c73cf640 1 Processor -- start 2023-11-24T08:08:28.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.709+0000 7f04c73cf640 1 -- start start 2023-11-24T08:08:28.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.710+0000 7f04c73cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c00b0b40 0x7f04c00b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.710+0000 7f04c73cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04c006c5b0 0x7f04c006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.710+0000 7f04c73cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c015ed20 con 0x7f04c015b260 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.710+0000 7f04c73cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c015b640 con 0x7f04c00b0b40 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.710+0000 7f04c73cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c00b14f0 con 0x7f04c006c5b0 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c63cd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04c015b260 0x7f04c006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55886/0 (socket says 172.21.15.107:55886) 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c63cd640 1 -- 172.21.15.107:0/1202718412 learned_addr learned my addr 172.21.15.107:0/1202718412 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:28.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c5bcc640 1 --2- 172.21.15.107:0/1202718412 >> v2:172.21.15.107:3300/0 conn(0x7f04c00b0b40 0x7f04c00b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:28.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c53cb640 1 -- 172.21.15.107:0/1202718412 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647639688 0 0) 0x7f04c015ed20 con 0x7f04c015b260 2023-11-24T08:08:28.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c53cb640 1 -- 172.21.15.107:0/1202718412 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac003520 con 0x7f04c015b260 2023-11-24T08:08:28.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c6bce640 1 --2- 172.21.15.107:0/1202718412 >> v2:172.21.15.120:3300/0 conn(0x7f04c006c5b0 0x7f04c006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c5bcc640 1 -- 172.21.15.107:0/1202718412 >> v2:172.21.15.120:3300/0 conn(0x7f04c006c5b0 msgr2=0x7f04c006ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c5bcc640 1 --2- 172.21.15.107:0/1202718412 >> v2:172.21.15.120:3300/0 conn(0x7f04c006c5b0 0x7f04c006ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c5bcc640 1 -- 172.21.15.107:0/1202718412 >> v1:172.21.15.107:6789/0 conn(0x7f04c015b260 legacy=0x7f04c006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.711+0000 7f04c5bcc640 1 -- 172.21.15.107:0/1202718412 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c015f8a0 con 0x7f04c00b0b40 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.712+0000 7f04c5bcc640 1 --2- 172.21.15.107:0/1202718412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c00b0b40 0x7f04c00b0fb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f04c00ccac0 tx=0x7f04b402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e580426af923d96 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.713+0000 7f04c53cb640 1 -- 172.21.15.107:0/1202718412 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4031710 con 0x7f04c00b0b40 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.713+0000 7f04c53cb640 1 -- 172.21.15.107:0/1202718412 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b402f050 con 0x7f04c00b0b40 2023-11-24T08:08:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.713+0000 7f04c53cb640 1 -- 172.21.15.107:0/1202718412 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4031a90 con 0x7f04c00b0b40 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.713+0000 7f04c73cf640 1 -- 172.21.15.107:0/1202718412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c00b0b40 msgr2=0x7f04c00b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.713+0000 7f04c73cf640 1 --2- 172.21.15.107:0/1202718412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c00b0b40 0x7f04c00b0fb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f04c00ccac0 tx=0x7f04b402b7b0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 -- 172.21.15.107:0/1202718412 shutdown_connections 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 --2- 172.21.15.107:0/1202718412 >> v2:172.21.15.120:3300/0 conn(0x7f04c006c5b0 0x7f04c006ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 --2- 172.21.15.107:0/1202718412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c00b0b40 0x7f04c00b0fb0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 -- 172.21.15.107:0/1202718412 >> 172.21.15.107:0/1202718412 conn(0x7f04c00cbb40 msgr2=0x7f04c00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 -- 172.21.15.107:0/1202718412 shutdown_connections 2023-11-24T08:08:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.714+0000 7f04c73cf640 1 -- 172.21.15.107:0/1202718412 wait complete. 2023-11-24T08:08:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.715+0000 7f04c73cf640 1 Processor -- start 2023-11-24T08:08:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.715+0000 7f04c73cf640 1 -- start start 2023-11-24T08:08:28.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c00b0b40 0x7f04c00b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c015b260 0x7f04c00b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c009bdc0 con 0x7f04c006c5b0 2023-11-24T08:08:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00b4910 con 0x7f04c00b0b40 2023-11-24T08:08:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.716+0000 7f04c73cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c009b4d0 con 0x7f04c015b260 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44666/0 (socket says 172.21.15.107:44666) 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 -- 172.21.15.107:0/4122999958 learned_addr learned my addr 172.21.15.107:0/4122999958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 -- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c015b260 msgr2=0x7f04c00b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c015b260 0x7f04c00b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 -- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c00b0b40 msgr2=0x7f04c00b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c00b0b40 0x7f04c00b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 -- 172.21.15.107:0/4122999958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009400 con 0x7f04c006c5b0 2023-11-24T08:08:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.717+0000 7f04c63cd640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c410 tx=0x7f04bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.718+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc005ce0 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.718+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc005e40 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.718+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c009b770 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.718+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0072f0 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.718+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c00b2910 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.719+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c004e840 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.720+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7f04bc007450 con 0x7f04c006c5b0 2023-11-24T08:08:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.721+0000 7f04b2ffd640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 0x7f04a406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.721+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04bc089420 con 0x7f04c006c5b0 2023-11-24T08:08:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.721+0000 7f04c5bcc640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 0x7f04a406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.722+0000 7f04c5bcc640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 0x7f04a406d840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f04b4035000 tx=0x7f04b4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.725+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04bc057a20 con 0x7f04c006c5b0 2023-11-24T08:08:28.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.953+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c00cc4c0 con 0x7f04a406b380 2023-11-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.966+0000 7f04b2ffd640 1 -- 172.21.15.107:0/4122999958 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f04c00cc4c0 con 0x7f04a406b380 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 msgr2=0x7f04a406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 0x7f04a406d840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f04b4035000 tx=0x7f04b4030000 comp rx=0 tx=0).stop 2023-11-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 msgr2=0x7f04c00b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c410 tx=0x7f04bc00c8d0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 shutdown_connections 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406b380 0x7f04a406d840 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c015b260 0x7f04c00b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c00b0b40 0x7f04c00b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 --2- 172.21.15.107:0/4122999958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c006c5b0 0x7f04c00b2330 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 >> 172.21.15.107:0/4122999958 conn(0x7f04c00cbb40 msgr2=0x7f04c00ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 shutdown_connections 2023-11-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:28.969+0000 7f04c73cf640 1 -- 172.21.15.107:0/4122999958 wait complete. 2023-11-24T08:08:29.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:30.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.224+0000 7fa6cbb33640 1 Processor -- start 2023-11-24T08:08:30.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.224+0000 7fa6cbb33640 1 -- start start 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6c406b780 0x7fa6c4161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa6c406e950 0x7fa6c40b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c40b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c4162100 con 0x7fa6c406b780 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40ac840 con 0x7fa6c406e950 2023-11-24T08:08:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6cbb33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40ad6a0 con 0x7fa6c406bb60 2023-11-24T08:08:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6c98cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6c406b780 0x7fa6c4161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6c98cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6c406b780 0x7fa6c4161bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44670/0 (socket says 172.21.15.107:44670) 2023-11-24T08:08:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.226+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1136315110 learned_addr learned my addr 172.21.15.107:0/1136315110 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c90ce640 1 --2- 172.21.15.107:0/1136315110 >> v2:172.21.15.120:3300/0 conn(0x7fa6c406e950 0x7fa6c40b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1136315110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 msgr2=0x7fa6c40b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1136315110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c40b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1136315110 >> v2:172.21.15.120:3300/0 conn(0x7fa6c406e950 msgr2=0x7fa6c40b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1136315110 >> v2:172.21.15.120:3300/0 conn(0x7fa6c406e950 0x7fa6c40b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.227+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1136315110 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40aca90 con 0x7fa6c406b780 2023-11-24T08:08:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.228+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1136315110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4161bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4071e80 tx=0x7fa6c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d0a728ffdcb004 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.228+0000 7fa6c88cd640 1 -- 172.21.15.107:0/1136315110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0031870 con 0x7fa6c406b780 2023-11-24T08:08:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.228+0000 7fa6c88cd640 1 -- 172.21.15.107:0/1136315110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c00319d0 con 0x7fa6c406b780 2023-11-24T08:08:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.228+0000 7fa6c88cd640 1 -- 172.21.15.107:0/1136315110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0031d60 con 0x7fa6c406b780 2023-11-24T08:08:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1136315110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 msgr2=0x7fa6c4161bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1136315110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4161bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4071e80 tx=0x7fa6c002b910 comp rx=0 tx=0).stop 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1136315110 shutdown_connections 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1136315110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c40b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1136315110 >> v2:172.21.15.120:3300/0 conn(0x7fa6c406e950 0x7fa6c40b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1136315110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4161bc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1136315110 >> 172.21.15.107:0/1136315110 conn(0x7fa6c4070fa0 msgr2=0x7fa6c40713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.229+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1136315110 shutdown_connections 2023-11-24T08:08:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.230+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1136315110 wait complete. 2023-11-24T08:08:30.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.231+0000 7fa6cbb33640 1 Processor -- start 2023-11-24T08:08:30.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.232+0000 7fa6cbb33640 1 -- start start 2023-11-24T08:08:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.232+0000 7fa6cbb33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6cbb33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c4161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6cbb33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6c406e950 0x7fa6c415ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6cbb33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c41559b0 con 0x7fa6c406b780 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6cbb33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4160780 con 0x7fa6c406e950 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6cbb33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4160a60 con 0x7fa6c406bb60 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44676/0 (socket says 172.21.15.107:44676) 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1062953048 learned_addr learned my addr 172.21.15.107:0/1062953048 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c90ce640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c4161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 msgr2=0x7fa6c4161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c4161820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6c406e950 msgr2=0x7fa6c415ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6c406e950 0x7fa6c415ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.233+0000 7fa6c98cf640 1 -- 172.21.15.107:0/1062953048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00094d0 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6c98cf640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c003abe0 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c003ad40 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c4160d40 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0044610 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.234+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c4161300 con 0x7fa6c406b780 2023-11-24T08:08:30.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.235+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c40cc500 con 0x7fa6c406b780 2023-11-24T08:08:30.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.236+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7fa6c004d450 con 0x7fa6c406b780 2023-11-24T08:08:30.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.237+0000 7fa6ba7fc640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 0x7fa6a806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.238+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6c002d040 con 0x7fa6c406b780 2023-11-24T08:08:30.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.238+0000 7fa6c90ce640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 0x7fa6a806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.239+0000 7fa6c90ce640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 0x7fa6a806d810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.242+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6c0076f20 con 0x7fa6c406b780 2023-11-24T08:08:30.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.497+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6c40ad330 con 0x7fa6a806b350 2023-11-24T08:08:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.525+0000 7fa6ba7fc640 1 -- 172.21.15.107:0/1062953048 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fa6b4009480 con 0x7fa6a806b350 2023-11-24T08:08:30.528 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:30.528 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:30.528 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:08:30.528 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:30.528 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:30.529 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:30.529 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:30.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:30.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:30.530 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 msgr2=0x7fa6a806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 0x7fa6a806d810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).stop 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 msgr2=0x7fa6c4160130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c00046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 shutdown_connections 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6a806b350 0x7fa6a806d810 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6c406e950 0x7fa6c415ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6c406bb60 0x7fa6c4161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 --2- 172.21.15.107:0/1062953048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c406b780 0x7fa6c4160130 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 >> 172.21.15.107:0/1062953048 conn(0x7fa6c4070fa0 msgr2=0x7fa6c40b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:30.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 shutdown_connections 2023-11-24T08:08:30.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.528+0000 7fa6cbb33640 1 -- 172.21.15.107:0/1062953048 wait complete. 2023-11-24T08:08:30.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone2533 2023-11-24T08:08:30.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.769+0000 7fbdfa251640 1 Processor -- start 2023-11-24T08:08:30.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.769+0000 7fbdfa251640 1 -- start start 2023-11-24T08:08:30.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.770+0000 7fbdfa251640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.770+0000 7fbdfa251640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbdf40d0a00 0x7fbdf406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.770+0000 7fbdfa251640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40cc170 con 0x7fbdf40b40d0 2023-11-24T08:08:30.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.770+0000 7fbdfa251640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf4070870 con 0x7fbdf40d0a00 2023-11-24T08:08:30.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.770+0000 7fbdfa251640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4070b50 con 0x7fbdf40cb850 2023-11-24T08:08:30.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.771+0000 7fbdf37fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.771+0000 7fbdf37fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40654/0 (socket says 172.21.15.107:40654) 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.771+0000 7fbdf37fe640 1 -- 172.21.15.107:0/1526895081 learned_addr learned my addr 172.21.15.107:0/1526895081 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.771+0000 7fbdf2ffd640 1 --2- 172.21.15.107:0/1526895081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf40d0a00 0x7fbdf406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.772+0000 7fbdf37fe640 1 -- 172.21.15.107:0/1526895081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf40d0a00 msgr2=0x7fbdf406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.772+0000 7fbdf37fe640 1 --2- 172.21.15.107:0/1526895081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf40d0a00 0x7fbdf406ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.772+0000 7fbdf37fe640 1 -- 172.21.15.107:0/1526895081 >> v1:172.21.15.120:6789/0 conn(0x7fbdf40b40d0 legacy=0x7fbdf40cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.772+0000 7fbdf37fe640 1 -- 172.21.15.107:0/1526895081 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4070e30 con 0x7fbdf40cb850 2023-11-24T08:08:30.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.773+0000 7fbdf37fe640 1 --2- 172.21.15.107:0/1526895081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009870 tx=0x7fbde402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d2b298884d6ec9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.774+0000 7fbdf27fc640 1 -- 172.21.15.107:0/1526895081 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4031640 con 0x7fbdf40cb850 2023-11-24T08:08:30.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.774+0000 7fbdf27fc640 1 -- 172.21.15.107:0/1526895081 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde40317a0 con 0x7fbdf40cb850 2023-11-24T08:08:30.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.774+0000 7fbdf27fc640 1 -- 172.21.15.107:0/1526895081 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4031b30 con 0x7fbdf40cb850 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.774+0000 7fbdfa251640 1 -- 172.21.15.107:0/1526895081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 msgr2=0x7fbdf40cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.774+0000 7fbdfa251640 1 --2- 172.21.15.107:0/1526895081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009870 tx=0x7fbde402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 -- 172.21.15.107:0/1526895081 shutdown_connections 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 --2- 172.21.15.107:0/1526895081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf40d0a00 0x7fbdf406ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 --2- 172.21.15.107:0/1526895081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40cb850 0x7fbdf40cbc30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 -- 172.21.15.107:0/1526895081 >> 172.21.15.107:0/1526895081 conn(0x7fbdf406e7d0 msgr2=0x7fbdf406ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 -- 172.21.15.107:0/1526895081 shutdown_connections 2023-11-24T08:08:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.775+0000 7fbdfa251640 1 -- 172.21.15.107:0/1526895081 wait complete. 2023-11-24T08:08:30.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.777+0000 7fbdfa251640 1 Processor -- start 2023-11-24T08:08:30.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.777+0000 7fbdfa251640 1 -- start start 2023-11-24T08:08:30.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf40cb850 0x7fbdf40c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40d0a00 0x7fbdf40c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40caf30 con 0x7fbdf40b40d0 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf41f2df0 con 0x7fbdf40cb850 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdfa251640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40c9e90 con 0x7fbdf40d0a00 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44684/0 (socket says 172.21.15.107:44684) 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf37fe640 1 -- 172.21.15.107:0/422251696 learned_addr learned my addr 172.21.15.107:0/422251696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf3fff640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40d0a00 0x7fbdf40c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf2ffd640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf40cb850 0x7fbdf40c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.778+0000 7fbdf37fe640 1 -- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40d0a00 msgr2=0x7fbdf40c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdf37fe640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40d0a00 0x7fbdf40c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdf37fe640 1 -- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf40cb850 msgr2=0x7fbdf40c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdf37fe640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf40cb850 0x7fbdf40c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdf37fe640 1 -- 172.21.15.107:0/422251696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde40094d0 con 0x7fbdf40b40d0 2023-11-24T08:08:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdf37fe640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009870 tx=0x7fbde40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde403a930 con 0x7fbdf40b40d0 2023-11-24T08:08:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde403aa90 con 0x7fbdf40b40d0 2023-11-24T08:08:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde403ae40 con 0x7fbdf40b40d0 2023-11-24T08:08:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.779+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf40ca170 con 0x7fbdf40b40d0 2023-11-24T08:08:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.780+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf40ca730 con 0x7fbdf40b40d0 2023-11-24T08:08:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.781+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf404e8b0 con 0x7fbdf40b40d0 2023-11-24T08:08:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.782+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7fbde402f050 con 0x7fbdf40b40d0 2023-11-24T08:08:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.783+0000 7fbddbfff640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 0x7fbde006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.783+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde40abd80 con 0x7fbdf40b40d0 2023-11-24T08:08:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.783+0000 7fbdf2ffd640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 0x7fbde006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:30.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.784+0000 7fbdf2ffd640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 0x7fbde006d810 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40c4c80 tx=0x7fbde8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:30.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:30.787+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbde40767f0 con 0x7fbdf40b40d0 2023-11-24T08:08:31.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.057+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdf40cf150 con 0x7fbde006b350 2023-11-24T08:08:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.267+0000 7fbddbfff640 1 -- 172.21.15.107:0/422251696 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbde8009800 con 0x7fbde006b350 2023-11-24T08:08:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 msgr2=0x7fbde006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 0x7fbde006d810 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40c4c80 tx=0x7fbde8008040 comp rx=0 tx=0).stop 2023-11-24T08:08:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 msgr2=0x7fbdf40c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009870 tx=0x7fbde40046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 shutdown_connections 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbde006b350 0x7fbde006d810 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf40d0a00 0x7fbdf40c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf40cb850 0x7fbdf40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 --2- 172.21.15.107:0/422251696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdf40b40d0 0x7fbdf40c4fa0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 >> 172.21.15.107:0/422251696 conn(0x7fbdf406e7d0 msgr2=0x7fbdf40734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 shutdown_connections 2023-11-24T08:08:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.271+0000 7fbdfa251640 1 -- 172.21.15.107:0/422251696 wait complete. 2023-11-24T08:08:31.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2533 2023-11-24T08:08:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.501+0000 7f8ea3fff640 1 Processor -- start 2023-11-24T08:08:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.501+0000 7f8ea3fff640 1 -- start start 2023-11-24T08:08:31.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8ea415b260 0x7f8ea406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea415ed20 con 0x7f8ea40b1250 2023-11-24T08:08:31.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea3fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea415f8a0 con 0x7f8ea406e580 2023-11-24T08:08:31.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea3fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea415b640 con 0x7f8ea415b260 2023-11-24T08:08:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8ea40b1250 0x7f8ea406c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55902/0 (socket says 172.21.15.107:55902) 2023-11-24T08:08:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.502+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1909086458 learned_addr learned my addr 172.21.15.107:0/1909086458 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99099402 0 0) 0x7f8ea415ed20 con 0x7f8ea40b1250 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c003520 con 0x7f8ea40b1250 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/1909086458 >> v2:172.21.15.120:3300/0 conn(0x7f8ea415b260 0x7f8ea406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3810659230 0 0) 0x7f8e8c003520 con 0x7f8ea40b1250 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 >> v2:172.21.15.120:3300/0 conn(0x7f8ea415b260 msgr2=0x7f8ea406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 --2- 172.21.15.107:0/1909086458 >> v2:172.21.15.120:3300/0 conn(0x7f8ea415b260 0x7f8ea406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 >> v1:172.21.15.120:6789/0 conn(0x7f8ea406e580 legacy=0x7f8ea40b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40abe50 con 0x7f8ea40b1250 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8e9c0019f0 con 0x7f8ea40b1250 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 >> v1:172.21.15.107:6789/0 conn(0x7f8ea40b1250 legacy=0x7f8ea406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 0x7f8e8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e8c0047c0 0x7f8e8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c005380 0x7f8e8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea415b640 con 0x7f8e8c003970 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c00c210 con 0x7f8e8c0047c0 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.503+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c00c3d0 con 0x7f8e8c005380 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 0x7f8e8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea27fc640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e8c0047c0 0x7f8e8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c005380 0x7f8e8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c005380 msgr2=0x7f8e8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c005380 0x7f8e8c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e8c0047c0 msgr2=0x7f8e8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e8c0047c0 0x7f8e8c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1909086458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c003520 con 0x7f8e8c003970 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c005380 0x7f8e8c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.504+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1909086458 reap_dead start 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.505+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 0x7f8e8c003d90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ba30 tx=0x7f8e8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ef5ad210ac0eb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94032630 con 0x7f8e8c003970 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94032790 con 0x7f8e8c003970 2023-11-24T08:08:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea1ffb640 1 -- 172.21.15.107:0/1909086458 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94032b20 con 0x7f8e8c003970 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea3fff640 1 -- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 msgr2=0x7f8e8c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 0x7f8e8c003d90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ba30 tx=0x7f8e8c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea3fff640 1 -- 172.21.15.107:0/1909086458 shutdown_connections 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/1909086458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e8c003970 0x7f8e8c003d90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.506+0000 7f8ea3fff640 1 -- 172.21.15.107:0/1909086458 >> 172.21.15.107:0/1909086458 conn(0x7f8ea40cbb40 msgr2=0x7f8ea40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.507+0000 7f8ea3fff640 1 -- 172.21.15.107:0/1909086458 shutdown_connections 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.507+0000 7f8ea3fff640 1 -- 172.21.15.107:0/1909086458 wait complete. 2023-11-24T08:08:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.508+0000 7f8ea3fff640 1 Processor -- start 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.509+0000 7f8ea3fff640 1 -- start start 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c003970 0x7f8ea4157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40b4a30 0x7f8ea40b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea409c560 con 0x7f8ea409bba0 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40b27d0 con 0x7f8ea40b4a30 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40b2ab0 con 0x7f8e8c003970 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44692/0 (socket says 172.21.15.107:44692) 2023-11-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 -- 172.21.15.107:0/3568329358 learned_addr learned my addr 172.21.15.107:0/3568329358 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c003970 0x7f8ea4157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea27fc640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40b4a30 0x7f8ea40b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 -- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c003970 msgr2=0x7f8ea4157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c003970 0x7f8ea4157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 -- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40b4a30 msgr2=0x7f8ea40b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40b4a30 0x7f8ea40b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.510+0000 7f8ea37fe640 1 -- 172.21.15.107:0/3568329358 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea415b920 con 0x7f8ea409bba0 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c000f80 tx=0x7f8e9c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c004d60 con 0x7f8ea409bba0 2023-11-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00bb80 con 0x7f8ea409bba0 2023-11-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c007070 con 0x7f8ea409bba0 2023-11-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea415bc00 con 0x7f8ea409bba0 2023-11-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.511+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea40b24c0 con 0x7f8ea409bba0 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.513+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea415b780 con 0x7f8ea409bba0 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.513+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7f8e9c0071d0 con 0x7f8ea409bba0 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.514+0000 7f8e8bfff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 0x7f8e9006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.514+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e9c08a090 con 0x7f8ea409bba0 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.514+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 0x7f8e9006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.515+0000 7f8ea2ffd640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 0x7f8e9006d840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ba00 tx=0x7f8e9403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:31.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.518+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ea415b780 con 0x7f8ea409bba0 2023-11-24T08:08:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:31.745+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2533", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ea415f530 con 0x7f8e9006b380 2023-11-24T08:08:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.294+0000 7f8e8bfff640 1 -- 172.21.15.107:0/3568329358 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+262 (secure 0 0 0) 0x7f8e94032a00 con 0x7f8e9006b380 2023-11-24T08:08:32.296 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "canceled", 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1626", 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5902" 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "failure": { 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "errno": "4", 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: "error_msg": "user interrupted clone operation" 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:32.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:08:32.298 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.297+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 msgr2=0x7f8e9006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.297+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 0x7f8e9006d840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ba00 tx=0x7f8e9403d040 comp rx=0 tx=0).stop 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 msgr2=0x7f8ea409c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c000f80 tx=0x7f8e9c00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 shutdown_connections 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e9006b380 0x7f8e9006d840 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea409bba0 0x7f8ea409c020 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40b4a30 0x7f8ea40b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/3568329358 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e8c003970 0x7f8ea4157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 >> 172.21.15.107:0/3568329358 conn(0x7f8ea40cbb40 msgr2=0x7f8ea40ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 shutdown_connections 2023-11-24T08:08:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.298+0000 7f8ea3fff640 1 -- 172.21.15.107:0/3568329358 wait complete. 2023-11-24T08:08:32.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone2533 --force 2023-11-24T08:08:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.540+0000 7fb418a1d640 1 Processor -- start 2023-11-24T08:08:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.541+0000 7fb418a1d640 1 -- start start 2023-11-24T08:08:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb418a1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb414160b00 0x7fb4140d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb418a1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 0x7fb4140cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb418a1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41415ea30 con 0x7fb4140d0120 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb418a1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4140d05b0 con 0x7fb414160b00 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb418a1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41415de20 con 0x7fb4140d12e0 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb412d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb414160b00 0x7fb4140d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb412d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb414160b00 0x7fb4140d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44706/0 (socket says 172.21.15.107:44706) 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb412d9d640 1 -- 172.21.15.107:0/854982144 learned_addr learned my addr 172.21.15.107:0/854982144 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.542+0000 7fb41359e640 1 --1- 172.21.15.107:0/854982144 >> v1:172.21.15.107:6789/0 conn(0x7fb4140d0120 0x7fb41415ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55910/0 (socket says 172.21.15.107:55910) 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb41259c640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 0x7fb4140cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb412d9d640 1 -- 172.21.15.107:0/854982144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 msgr2=0x7fb4140cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb412d9d640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 0x7fb4140cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb412d9d640 1 -- 172.21.15.107:0/854982144 >> v1:172.21.15.107:6789/0 conn(0x7fb4140d0120 legacy=0x7fb41415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb412d9d640 1 -- 172.21.15.107:0/854982144 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41415e100 con 0x7fb414160b00 2023-11-24T08:08:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.543+0000 7fb41259c640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 0x7fb4140cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.544+0000 7fb412d9d640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb414160b00 0x7fb4140d0da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb404009e50 tx=0x7fb40402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1da17968329da9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.544+0000 7fb411d9b640 1 -- 172.21.15.107:0/854982144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40402cb00 con 0x7fb414160b00 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.544+0000 7fb411d9b640 1 -- 172.21.15.107:0/854982144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40402cc60 con 0x7fb414160b00 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.544+0000 7fb411d9b640 1 -- 172.21.15.107:0/854982144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4040343f0 con 0x7fb414160b00 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 -- 172.21.15.107:0/854982144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb414160b00 msgr2=0x7fb4140d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb414160b00 0x7fb4140d0da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb404009e50 tx=0x7fb40402bf60 comp rx=0 tx=0).stop 2023-11-24T08:08:32.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 -- 172.21.15.107:0/854982144 shutdown_connections 2023-11-24T08:08:32.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d12e0 0x7fb4140cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 --2- 172.21.15.107:0/854982144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb414160b00 0x7fb4140d0da0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 -- 172.21.15.107:0/854982144 >> 172.21.15.107:0/854982144 conn(0x7fb41409c510 msgr2=0x7fb4140d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:32.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 -- 172.21.15.107:0/854982144 shutdown_connections 2023-11-24T08:08:32.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.545+0000 7fb418a1d640 1 -- 172.21.15.107:0/854982144 wait complete. 2023-11-24T08:08:32.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.546+0000 7fb418a1d640 1 Processor -- start 2023-11-24T08:08:32.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.547+0000 7fb418a1d640 1 -- start start 2023-11-24T08:08:32.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.547+0000 7fb418a1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d0120 0x7fb414148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb418a1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb418a1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 0x7fb414160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb418a1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb414148a90 con 0x7fb4140d12e0 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb418a1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4140c9920 con 0x7fb414160b00 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb418a1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4140c9c00 con 0x7fb4140d0120 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44712/0 (socket says 172.21.15.107:44712) 2023-11-24T08:08:32.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 -- 172.21.15.107:0/1786850443 learned_addr learned my addr 172.21.15.107:0/1786850443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:32.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb412d9d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d0120 0x7fb414148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41359e640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 0x7fb414160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 -- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d0120 msgr2=0x7fb414148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d0120 0x7fb414148550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 -- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 msgr2=0x7fb414160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 0x7fb414160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41259c640 1 -- 172.21.15.107:0/1786850443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404009b30 con 0x7fb4140d12e0 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.548+0000 7fb41359e640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 0x7fb414160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb41259c640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c7f0 tx=0x7fb3fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc007a30 con 0x7fb4140d12e0 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc007b90 con 0x7fb4140d12e0 2023-11-24T08:08:32.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4140c9ee0 con 0x7fb4140d12e0 2023-11-24T08:08:32.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc00ce40 con 0x7fb4140d12e0 2023-11-24T08:08:32.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.549+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4140b68f0 con 0x7fb4140d12e0 2023-11-24T08:08:32.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.550+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4140b6ae0 con 0x7fb4140d12e0 2023-11-24T08:08:32.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.551+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7fb3fc0113f0 con 0x7fb4140d12e0 2023-11-24T08:08:32.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.552+0000 7fb40b7fe640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 0x7fb3f406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:32.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.552+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3fc08a1c0 con 0x7fb4140d12e0 2023-11-24T08:08:32.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.552+0000 7fb412d9d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 0x7fb3f406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:32.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.553+0000 7fb412d9d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 0x7fb3f406d810 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb404035000 tx=0x7fb404030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.556+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3fc054a80 con 0x7fb4140d12e0 2023-11-24T08:08:32.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.771+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone2533", "force": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb41415ee40 con 0x7fb3f406b350 2023-11-24T08:08:32.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.798+0000 7fb40b7fe640 1 -- 172.21.15.107:0/1786850443 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb404034560 con 0x7fb3f406b350 2023-11-24T08:08:32.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 msgr2=0x7fb3f406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 0x7fb3f406d810 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb404035000 tx=0x7fb404030000 comp rx=0 tx=0).stop 2023-11-24T08:08:32.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 msgr2=0x7fb4140c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:32.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c7f0 tx=0x7fb3fc00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 shutdown_connections 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3f406b350 0x7fb3f406d810 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb414160b00 0x7fb414160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4140d12e0 0x7fb4140c6d90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 --2- 172.21.15.107:0/1786850443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4140d0120 0x7fb414148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 >> 172.21.15.107:0/1786850443 conn(0x7fb41409c510 msgr2=0x7fb4140cebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 shutdown_connections 2023-11-24T08:08:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:32.801+0000 7fb418a1d640 1 -- 172.21.15.107:0/1786850443 wait complete. 2023-11-24T08:08:32.816 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:32.814+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:32.817 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:32.814+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:32.817 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:32.814+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:32.817 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:32.814+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:32.817 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:08:32.814+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:08:32.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol1626 subvol_snap5902 2023-11-24T08:08:33.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.029+0000 7fe4491b3640 1 Processor -- start 2023-11-24T08:08:33.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.029+0000 7fe4491b3640 1 -- start start 2023-11-24T08:08:33.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe4491b3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 0x7fe444142110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe4491b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe44406a500 0x7fe444072200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe4491b3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44406ed10 con 0x7fe4440cc8c0 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe4491b3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe444141dd0 con 0x7fe44406a500 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe4491b3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44406aa70 con 0x7fe4441419f0 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe43bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe44406a500 0x7fe444072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe43bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe44406a500 0x7fe444072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44734/0 (socket says 172.21.15.107:44734) 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe43bfff640 1 -- 172.21.15.107:0/1650407790 learned_addr learned my addr 172.21.15.107:0/1650407790 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe43bfff640 1 -- 172.21.15.107:0/1650407790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 msgr2=0x7fe444142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.031+0000 7fe443fff640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 0x7fe444142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.032+0000 7fe43bfff640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 0x7fe444142110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.032+0000 7fe43bfff640 1 -- 172.21.15.107:0/1650407790 >> v1:172.21.15.120:6789/0 conn(0x7fe4440cc8c0 legacy=0x7fe44406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.032+0000 7fe43bfff640 1 -- 172.21.15.107:0/1650407790 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44406ad50 con 0x7fe44406a500 2023-11-24T08:08:33.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.032+0000 7fe443fff640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 0x7fe444142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.033+0000 7fe43bfff640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe44406a500 0x7fe444072200 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a564ff8c805980 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.033+0000 7fe4437fe640 1 -- 172.21.15.107:0/1650407790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c031640 con 0x7fe44406a500 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.033+0000 7fe4437fe640 1 -- 172.21.15.107:0/1650407790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c0317a0 con 0x7fe44406a500 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.033+0000 7fe4437fe640 1 -- 172.21.15.107:0/1650407790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c031b30 con 0x7fe44406a500 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 -- 172.21.15.107:0/1650407790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe44406a500 msgr2=0x7fe444072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe44406a500 0x7fe444072200 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 -- 172.21.15.107:0/1650407790 shutdown_connections 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe44406a500 0x7fe444072200 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 --2- 172.21.15.107:0/1650407790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4441419f0 0x7fe444142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 -- 172.21.15.107:0/1650407790 >> 172.21.15.107:0/1650407790 conn(0x7fe4440cfed0 msgr2=0x7fe4440d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:33.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 -- 172.21.15.107:0/1650407790 shutdown_connections 2023-11-24T08:08:33.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.034+0000 7fe4491b3640 1 -- 172.21.15.107:0/1650407790 wait complete. 2023-11-24T08:08:33.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.036+0000 7fe4491b3640 1 Processor -- start 2023-11-24T08:08:33.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.036+0000 7fe4491b3640 1 -- start start 2023-11-24T08:08:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe44406a500 0x7fe4440c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 0x7fe444158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44415ad40 con 0x7fe4441419f0 2023-11-24T08:08:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440ca5b0 con 0x7fe4440cc8c0 2023-11-24T08:08:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4491b3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440ca890 con 0x7fe44406a500 2023-11-24T08:08:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4489b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4489b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44738/0 (socket says 172.21.15.107:44738) 2023-11-24T08:08:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.037+0000 7fe4489b2640 1 -- 172.21.15.107:0/3618982604 learned_addr learned my addr 172.21.15.107:0/3618982604 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe43bfff640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 0x7fe444158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe443fff640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe44406a500 0x7fe4440c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 -- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe44406a500 msgr2=0x7fe4440c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe44406a500 0x7fe4440c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 -- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 msgr2=0x7fe444158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 0x7fe444158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 -- 172.21.15.107:0/3618982604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0094d0 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe43bfff640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 0x7fe444158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4489b2640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00c780 tx=0x7fe43c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c004b60 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c00ce90 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4440cab70 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00a850 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.038+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4440ca210 con 0x7fe4441419f0 2023-11-24T08:08:33.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.039+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4440cdc40 con 0x7fe4441419f0 2023-11-24T08:08:33.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.040+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 89690+0+0 (secure 0 0 0) 0x7fe43c00a9b0 con 0x7fe4441419f0 2023-11-24T08:08:33.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.041+0000 7fe4417fa640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 0x7fe42406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.041+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe43c089fc0 con 0x7fe4441419f0 2023-11-24T08:08:33.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.042+0000 7fe443fff640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 0x7fe42406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.042+0000 7fe443fff640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 0x7fe42406d810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe43400b840 tx=0x7fe434008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.046+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe43c054a70 con 0x7fe4441419f0 2023-11-24T08:08:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.270+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol1626", "snap_name": "subvol_snap5902", "target": ["mon-mgr", ""]}) v1 -- 0x7fe444072740 con 0x7fe42406b350 2023-11-24T08:08:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.361+0000 7fe4417fa640 1 -- 172.21.15.107:0/3618982604 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe43400c040 con 0x7fe42406b350 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 msgr2=0x7fe42406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 0x7fe42406d810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe43400b840 tx=0x7fe434008040 comp rx=0 tx=0).stop 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 msgr2=0x7fe444159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00c780 tx=0x7fe43c00cc40 comp rx=0 tx=0).stop 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 shutdown_connections 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42406b350 0x7fe42406d810 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4441419f0 0x7fe444159160 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4440cc8c0 0x7fe444158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 --2- 172.21.15.107:0/3618982604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe44406a500 0x7fe4440c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 >> 172.21.15.107:0/3618982604 conn(0x7fe4440cfed0 msgr2=0x7fe44415e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 shutdown_connections 2023-11-24T08:08:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.365+0000 7fe4491b3640 1 -- 172.21.15.107:0/3618982604 wait complete. 2023-11-24T08:08:33.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol1626 2023-11-24T08:08:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.612+0000 7f1cbadba640 1 Processor -- start 2023-11-24T08:08:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.612+0000 7f1cbadba640 1 -- start start 2023-11-24T08:08:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.613+0000 7f1cbadba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 0x7f1cb406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.613+0000 7f1cbadba640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1cb415e5e0 0x7f1cb40b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.613+0000 7f1cbadba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb406d430 con 0x7f1cb40b0b40 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cbadba640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb415ea70 con 0x7f1cb415e5e0 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cbadba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb415ec50 con 0x7f1cb415a1c0 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cb9db8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 0x7f1cb406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cb95b7640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1cb40b0b40 0x7f1cb415ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:35914/0 (socket says 172.21.15.107:35914) 2023-11-24T08:08:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cb95b7640 1 -- 172.21.15.107:0/42004517 learned_addr learned my addr 172.21.15.107:0/42004517 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.614+0000 7f1cba5b9640 1 --2- 172.21.15.107:0/42004517 >> v2:172.21.15.120:3300/0 conn(0x7f1cb415e5e0 0x7f1cb40b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb8db6640 1 -- 172.21.15.107:0/42004517 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912728172 0 0) 0x7f1cb406d430 con 0x7f1cb40b0b40 2023-11-24T08:08:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb8db6640 1 -- 172.21.15.107:0/42004517 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca0003520 con 0x7f1cb40b0b40 2023-11-24T08:08:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb9db8640 1 -- 172.21.15.107:0/42004517 >> v2:172.21.15.120:3300/0 conn(0x7f1cb415e5e0 msgr2=0x7f1cb40b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/42004517 >> v2:172.21.15.120:3300/0 conn(0x7f1cb415e5e0 0x7f1cb40b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb9db8640 1 -- 172.21.15.107:0/42004517 >> v1:172.21.15.120:6789/0 conn(0x7f1cb40b0b40 legacy=0x7f1cb415ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.615+0000 7f1cb9db8640 1 -- 172.21.15.107:0/42004517 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb40b14e0 con 0x7f1cb415a1c0 2023-11-24T08:08:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.617+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/42004517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 0x7f1cb406d8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00097a0 tx=0x7f1cb002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8232e267ac92b983 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.617+0000 7f1cb8db6640 1 -- 172.21.15.107:0/42004517 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0031830 con 0x7f1cb415a1c0 2023-11-24T08:08:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.617+0000 7f1cb8db6640 1 -- 172.21.15.107:0/42004517 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0031990 con 0x7f1cb415a1c0 2023-11-24T08:08:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.617+0000 7f1cb8db6640 1 -- 172.21.15.107:0/42004517 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0031d00 con 0x7f1cb415a1c0 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 -- 172.21.15.107:0/42004517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 msgr2=0x7f1cb406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 --2- 172.21.15.107:0/42004517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 0x7f1cb406d8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00097a0 tx=0x7f1cb002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 -- 172.21.15.107:0/42004517 shutdown_connections 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 --2- 172.21.15.107:0/42004517 >> v2:172.21.15.120:3300/0 conn(0x7f1cb415e5e0 0x7f1cb40b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 --2- 172.21.15.107:0/42004517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb415a1c0 0x7f1cb406d8c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 -- 172.21.15.107:0/42004517 >> 172.21.15.107:0/42004517 conn(0x7f1cb40cbb40 msgr2=0x7f1cb40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 -- 172.21.15.107:0/42004517 shutdown_connections 2023-11-24T08:08:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.618+0000 7f1cbadba640 1 -- 172.21.15.107:0/42004517 wait complete. 2023-11-24T08:08:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.620+0000 7f1cbadba640 1 Processor -- start 2023-11-24T08:08:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.620+0000 7f1cbadba640 1 -- start start 2023-11-24T08:08:33.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 0x7f1cb4159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 0x7f1cb4158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb40c6f00 con 0x7f1cb415a1c0 2023-11-24T08:08:33.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb40c2b00 con 0x7f1cb415e5e0 2023-11-24T08:08:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cbadba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb40c2de0 con 0x7f1cb40b0b40 2023-11-24T08:08:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cb95b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.621+0000 7f1cb95b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44740/0 (socket says 172.21.15.107:44740) 2023-11-24T08:08:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 -- 172.21.15.107:0/1754679502 learned_addr learned my addr 172.21.15.107:0/1754679502 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cba5b9640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 0x7f1cb4158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 0x7f1cb4159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 -- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 msgr2=0x7f1cb4159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 0x7f1cb4159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 -- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 msgr2=0x7f1cb4158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 0x7f1cb4158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 -- 172.21.15.107:0/1754679502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0009400 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cba5b9640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 0x7f1cb4158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 0x7f1cb4159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.622+0000 7f1cb95b7640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400c430 tx=0x7f1ca400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.623+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4005ce0 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.623+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4005e40 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.623+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb40c30c0 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.623+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca40072f0 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.623+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb40c3680 con 0x7f1cb415a1c0 2023-11-24T08:08:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.624+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb404e840 con 0x7f1cb415a1c0 2023-11-24T08:08:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.625+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f1ca4007450 con 0x7f1cb415a1c0 2023-11-24T08:08:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.626+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 0x7f1c9806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.626+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ca4089680 con 0x7f1cb415a1c0 2023-11-24T08:08:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.626+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 0x7f1c9806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.627+0000 7f1cb9db8640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 0x7f1c9806d8a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0036000 tx=0x7f1cb0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.631+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ca40579e0 con 0x7f1cb415a1c0 2023-11-24T08:08:33.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.848+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol1626", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cb406d0c0 con 0x7f1c9806b3e0 2023-11-24T08:08:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.881+0000 7f1caa7fc640 1 -- 172.21.15.107:0/1754679502 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1cb0031b80 con 0x7f1c9806b3e0 2023-11-24T08:08:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 msgr2=0x7f1c9806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 0x7f1c9806d8a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0036000 tx=0x7f1cb0008030 comp rx=0 tx=0).stop 2023-11-24T08:08:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 msgr2=0x7f1cb40c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400c430 tx=0x7f1ca400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 shutdown_connections 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c9806b3e0 0x7f1c9806d8a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb415e5e0 0x7f1cb4158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb415a1c0 0x7f1cb40c69c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 --2- 172.21.15.107:0/1754679502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cb40b0b40 0x7f1cb4159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 >> 172.21.15.107:0/1754679502 conn(0x7f1cb40cbb40 msgr2=0x7f1cb40caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 shutdown_connections 2023-11-24T08:08:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:33.884+0000 7f1cbadba640 1 -- 172.21.15.107:0/1754679502 wait complete. 2023-11-24T08:08:33.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:33.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:08:33.962 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:08:38.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:38.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:08:39.029 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:08:39.030 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:08:39.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:08:39.155 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:08:39.275 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:08:39.275 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:39.275 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:39.332 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:39.333 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:39.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:39.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.627+0000 7fa3db056640 1 Processor -- start 2023-11-24T08:08:39.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.627+0000 7fa3db056640 1 -- start start 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3db056640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cf450 0x7fa3d40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3db056640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40cdfb0 0x7fa3d40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3db056640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40cece0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3db056640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d4156010 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3db056640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d406ce30 con 0x7fa3d40cf450 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40cdfb0 0x7fa3d40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40cdfb0 0x7fa3d40ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55174/0 (socket says 172.21.15.107:55174) 2023-11-24T08:08:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 -- 172.21.15.107:0/1549744 learned_addr learned my addr 172.21.15.107:0/1549744 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3cbfff640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cf450 0x7fa3d40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 -- 172.21.15.107:0/1549744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cf450 msgr2=0x7fa3d40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cf450 0x7fa3d40cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 -- 172.21.15.107:0/1549744 >> v1:172.21.15.120:6789/0 conn(0x7fa3d415c7f0 legacy=0x7fa3d406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.629+0000 7fa3d95f3640 1 -- 172.21.15.107:0/1549744 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d406d040 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.630+0000 7fa3d95f3640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40cdfb0 0x7fa3d40ce440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009870 tx=0x7fa3cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e181a458881627 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:39.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3cb7fe640 1 -- 172.21.15.107:0/1549744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc031640 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3cb7fe640 1 -- 172.21.15.107:0/1549744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc0317a0 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3cb7fe640 1 -- 172.21.15.107:0/1549744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc031b30 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 -- 172.21.15.107:0/1549744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40cdfb0 msgr2=0x7fa3d40ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40cdfb0 0x7fa3d40ce440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009870 tx=0x7fa3cc02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 -- 172.21.15.107:0/1549744 shutdown_connections 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40cdfb0 0x7fa3d40ce440 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 --2- 172.21.15.107:0/1549744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cf450 0x7fa3d40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.631+0000 7fa3db056640 1 -- 172.21.15.107:0/1549744 >> 172.21.15.107:0/1549744 conn(0x7fa3d406a920 msgr2=0x7fa3d406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.632+0000 7fa3db056640 1 -- 172.21.15.107:0/1549744 shutdown_connections 2023-11-24T08:08:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.632+0000 7fa3db056640 1 -- 172.21.15.107:0/1549744 wait complete. 2023-11-24T08:08:39.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.633+0000 7fa3db056640 1 Processor -- start 2023-11-24T08:08:39.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.633+0000 7fa3db056640 1 -- start start 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 0x7fa3d4158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 0x7fa3d40adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40b4140 con 0x7fa3d415c7f0 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ac9d0 con 0x7fa3d40cf450 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.634+0000 7fa3db056640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ad830 con 0x7fa3d40cdfb0 2023-11-24T08:08:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55176/0 (socket says 172.21.15.107:55176) 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 -- 172.21.15.107:0/2129381722 learned_addr learned my addr 172.21.15.107:0/2129381722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3cbfff640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 0x7fa3d40adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 -- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 msgr2=0x7fa3d4158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d8df2640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 0x7fa3d4158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 0x7fa3d4158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 -- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 msgr2=0x7fa3d40adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 0x7fa3d40adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d95f3640 1 -- 172.21.15.107:0/2129381722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0094d0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3cbfff640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 0x7fa3d40adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.635+0000 7fa3d8df2640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 0x7fa3d4158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3d95f3640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc02b930 tx=0x7fa3cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc03ac60 con 0x7fa3d415c7f0 2023-11-24T08:08:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc03ade0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d4159330 con 0x7fa3d415c7f0 2023-11-24T08:08:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc044630 con 0x7fa3d415c7f0 2023-11-24T08:08:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.636+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d40acf00 con 0x7fa3d415c7f0 2023-11-24T08:08:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.637+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d40ad9e0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.638+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fa3cc02f050 con 0x7fa3d415c7f0 2023-11-24T08:08:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.639+0000 7fa3c97fa640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 0x7fa3c006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.639+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa3cc0ac250 con 0x7fa3d415c7f0 2023-11-24T08:08:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.639+0000 7fa3d8df2640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 0x7fa3c006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.640+0000 7fa3d8df2640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 0x7fa3c006d8d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007e90 tx=0x7fa3d00073a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:39.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.643+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3d40ad9e0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:08:39.843 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:08:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.836+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3d40ad9a0 con 0x7fa3d415c7f0 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.839+0000 7fa3c97fa640 1 -- 172.21.15.107:0/2129381722 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16670 (secure 0 0 0) 0x7fa3cc076a40 con 0x7fa3d415c7f0 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 msgr2=0x7fa3c006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 0x7fa3c006d8d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007e90 tx=0x7fa3d00073a0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 msgr2=0x7fa3d40c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc02b930 tx=0x7fa3cc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 shutdown_connections 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006b410 0x7fa3c006d8d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d415c7f0 0x7fa3d40c9cf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40cf450 0x7fa3d40adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 --2- 172.21.15.107:0/2129381722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40cdfb0 0x7fa3d4158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.841+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 >> 172.21.15.107:0/2129381722 conn(0x7fa3d406a920 msgr2=0x7fa3d415de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:39.846 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:37.801647+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.842+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 shutdown_connections 2023-11-24T08:08:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:39.842+0000 7fa3db056640 1 -- 172.21.15.107:0/2129381722 wait complete. 2023-11-24T08:08:39.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:07:38.663153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:07:39.906724+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:08:39.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:08:40.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.110+0000 7f6258927640 1 Processor -- start 2023-11-24T08:08:40.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.110+0000 7f6258927640 1 -- start start 2023-11-24T08:08:40.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.111+0000 7f6258927640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f625415b260 0x7f625406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.111+0000 7f6258927640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625415ed20 con 0x7f62540b1250 2023-11-24T08:08:40.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.111+0000 7f6258927640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625415f8a0 con 0x7f625406e580 2023-11-24T08:08:40.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.111+0000 7f6258927640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625415b640 con 0x7f625415b260 2023-11-24T08:08:40.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.112+0000 7f6253fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62540b1250 0x7f625406c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53706/0 (socket says 172.21.15.107:53706) 2023-11-24T08:08:40.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.112+0000 7f6253fff640 1 -- 172.21.15.107:0/3163391849 learned_addr learned my addr 172.21.15.107:0/3163391849 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.112+0000 7f62537fe640 1 --2- 172.21.15.107:0/3163391849 >> v2:172.21.15.120:3300/0 conn(0x7f625415b260 0x7f625406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.112+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516848816 0 0) 0x7f625415ed20 con 0x7f62540b1250 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.112+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f623c003520 con 0x7f62540b1250 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863244091 0 0) 0x7f623c003520 con 0x7f62540b1250 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 >> v2:172.21.15.120:3300/0 conn(0x7f625415b260 msgr2=0x7f625406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 --2- 172.21.15.107:0/3163391849 >> v2:172.21.15.120:3300/0 conn(0x7f625415b260 0x7f625406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 >> v1:172.21.15.120:6789/0 conn(0x7f625406e580 legacy=0x7f62540b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540abe50 con 0x7f62540b1250 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f624c0019f0 con 0x7f62540b1250 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 >> v1:172.21.15.107:6789/0 conn(0x7f62540b1250 legacy=0x7f625406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c003970 0x7f623c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f623c0052a0 0x7f623c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625415b640 con 0x7f623c0047c0 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c00c170 con 0x7f623c003970 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c00c330 con 0x7f623c0052a0 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f62537fe640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c003970 0x7f623c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 -- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f623c0052a0 msgr2=0x7f623c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f623c0052a0 0x7f623c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 -- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c003970 msgr2=0x7f623c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c003970 0x7f623c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.113+0000 7f6252ffd640 1 -- 172.21.15.107:0/3163391849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c003520 con 0x7f623c0047c0 2023-11-24T08:08:40.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.114+0000 7f6253fff640 1 -- 172.21.15.107:0/3163391849 reap_dead start 2023-11-24T08:08:40.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.114+0000 7f6252ffd640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 0x7f623c004c40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6248001720 tx=0x7f623c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b21d13ac601636 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62480052d0 con 0x7f623c0047c0 2023-11-24T08:08:40.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624802e050 con 0x7f623c0047c0 2023-11-24T08:08:40.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f62527fc640 1 -- 172.21.15.107:0/3163391849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248003830 con 0x7f623c0047c0 2023-11-24T08:08:40.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 -- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 msgr2=0x7f623c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 0x7f623c004c40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6248001720 tx=0x7f623c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 -- 172.21.15.107:0/3163391849 shutdown_connections 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 --2- 172.21.15.107:0/3163391849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 -- 172.21.15.107:0/3163391849 >> 172.21.15.107:0/3163391849 conn(0x7f62540cbb40 msgr2=0x7f62540cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.115+0000 7f6258927640 1 -- 172.21.15.107:0/3163391849 shutdown_connections 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.116+0000 7f6258927640 1 -- 172.21.15.107:0/3163391849 wait complete. 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.117+0000 7f6258927640 1 Processor -- start 2023-11-24T08:08:40.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.117+0000 7f6258927640 1 -- start start 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c0047c0 0x7f6254153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62540b4250 0x7f62540b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6254153e50 con 0x7f62540c5890 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540b4c50 con 0x7f623c0047c0 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.118+0000 7f6258927640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62540c3930 con 0x7f62540b4250 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55184/0 (socket says 172.21.15.107:55184) 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 -- 172.21.15.107:0/1630272876 learned_addr learned my addr 172.21.15.107:0/1630272876 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 -- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62540b4250 msgr2=0x7f62540b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62540b4250 0x7f62540b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f62537fe640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c0047c0 0x7f6254153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 -- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c0047c0 msgr2=0x7f6254153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c0047c0 0x7f6254153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 -- 172.21.15.107:0/1630272876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540c3c10 con 0x7f62540c5890 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f6252ffd640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f624802bbb0 tx=0x7f6248001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.119+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62480076c0 con 0x7f62540c5890 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.120+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248007a30 con 0x7f62540c5890 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.120+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248006be0 con 0x7f62540c5890 2023-11-24T08:08:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.120+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62540c3ef0 con 0x7f62540c5890 2023-11-24T08:08:40.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.120+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62541562f0 con 0x7f62540c5890 2023-11-24T08:08:40.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.121+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62540c3a70 con 0x7f62540c5890 2023-11-24T08:08:40.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.122+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f6248006d40 con 0x7f62540c5890 2023-11-24T08:08:40.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.123+0000 7f623bfff640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 0x7f624006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.124+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f62480abd80 con 0x7f62540c5890 2023-11-24T08:08:40.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.124+0000 7f62537fe640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 0x7f624006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.124+0000 7f62537fe640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 0x7f624006d8d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f62440099d0 tx=0x7f6244008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.128+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62540c3a70 con 0x7f62540c5890 2023-11-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.318+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62540c3a70 con 0x7f62540c5890 2023-11-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.319+0000 7f623bfff640 1 -- 172.21.15.107:0/1630272876 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62540c3a70 con 0x7f62540c5890 2023-11-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.321+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 msgr2=0x7f624006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.321+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 0x7f624006d8d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f62440099d0 tx=0x7f6244008040 comp rx=0 tx=0).stop 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.321+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 msgr2=0x7f6254155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.321+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f624802bbb0 tx=0x7f6248001120 comp rx=0 tx=0).stop 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 shutdown_connections 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f624006b410 0x7f624006d8d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62540b4250 0x7f62540b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62540c5890 0x7f6254155db0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 --2- 172.21.15.107:0/1630272876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f623c0047c0 0x7f6254153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 >> 172.21.15.107:0/1630272876 conn(0x7f62540cbb40 msgr2=0x7f62540ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:40.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 shutdown_connections 2023-11-24T08:08:40.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.322+0000 7f6258927640 1 -- 172.21.15.107:0/1630272876 wait complete. 2023-11-24T08:08:40.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:08:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.552+0000 7f1bd0c1a640 1 Processor -- start 2023-11-24T08:08:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.552+0000 7f1bd0c1a640 1 -- start start 2023-11-24T08:08:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.553+0000 7f1bd0c1a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1bcc15b260 0x7f1bcc06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.553+0000 7f1bd0c1a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc15ed20 con 0x7f1bcc0b1250 2023-11-24T08:08:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.553+0000 7f1bd0c1a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc15f8a0 con 0x7f1bcc06e580 2023-11-24T08:08:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.553+0000 7f1bd0c1a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc15b640 con 0x7f1bcc15b260 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1bcc0b1250 0x7f1bcc06c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53716/0 (socket says 172.21.15.107:53716) 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcbfff640 1 -- 172.21.15.107:0/2438805686 learned_addr learned my addr 172.21.15.107:0/2438805686 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278112181 0 0) 0x7f1bcc15ed20 con 0x7f1bcc0b1250 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bac003520 con 0x7f1bcc0b1250 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcb7fe640 1 --2- 172.21.15.107:0/2438805686 >> v2:172.21.15.120:3300/0 conn(0x7f1bcc15b260 0x7f1bcc06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264043509 0 0) 0x7f1bac003520 con 0x7f1bcc0b1250 2023-11-24T08:08:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 >> v2:172.21.15.120:3300/0 conn(0x7f1bcc15b260 msgr2=0x7f1bcc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 --2- 172.21.15.107:0/2438805686 >> v2:172.21.15.120:3300/0 conn(0x7f1bcc15b260 0x7f1bcc06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 >> v1:172.21.15.120:6789/0 conn(0x7f1bcc06e580 legacy=0x7f1bcc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0abe50 con 0x7f1bcc0b1250 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1bc40019f0 con 0x7f1bcc0b1250 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.554+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 >> v1:172.21.15.107:6789/0 conn(0x7f1bcc0b1250 legacy=0x7f1bcc06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bac003970 0x7f1bac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bac005380 0x7f1bac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc15b640 con 0x7f1bac0047c0 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac00c210 con 0x7f1bac005380 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac00c3d0 con 0x7f1bac003970 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcb7fe640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bac003970 0x7f1bac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcbfff640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bac005380 0x7f1bac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 -- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bac003970 msgr2=0x7f1bac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bac003970 0x7f1bac003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 -- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bac005380 msgr2=0x7f1bac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bac005380 0x7f1bac00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bc3fff640 1 -- 172.21.15.107:0/2438805686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac003520 con 0x7f1bac0047c0 2023-11-24T08:08:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.555+0000 7f1bcbfff640 1 -- 172.21.15.107:0/2438805686 reap_dead start 2023-11-24T08:08:40.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.556+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bac004c40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4001750 tx=0x7f1bac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1219494cabd1a9d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40052d0 con 0x7f1bac0047c0 2023-11-24T08:08:40.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb402e050 con 0x7f1bac0047c0 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bcaffd640 1 -- 172.21.15.107:0/2438805686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4003830 con 0x7f1bac0047c0 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 msgr2=0x7f1bac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bac004c40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4001750 tx=0x7f1bac0057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/2438805686 shutdown_connections 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/2438805686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bac004c40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/2438805686 >> 172.21.15.107:0/2438805686 conn(0x7f1bcc0cbb40 msgr2=0x7f1bcc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/2438805686 shutdown_connections 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.557+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/2438805686 wait complete. 2023-11-24T08:08:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.559+0000 7f1bd0c1a640 1 Processor -- start 2023-11-24T08:08:40.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.559+0000 7f1bd0c1a640 1 -- start start 2023-11-24T08:08:40.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bcc1f1090 0x7f1bcc0b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bcc0b43a0 0x7f1bcc0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0b3810 con 0x7f1bac0047c0 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0c6f40 con 0x7f1bcc1f1090 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.560+0000 7f1bd0c1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0b4d60 con 0x7f1bcc0b43a0 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55208/0 (socket says 172.21.15.107:55208) 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 -- 172.21.15.107:0/3880565669 learned_addr learned my addr 172.21.15.107:0/3880565669 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 -- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bcc0b43a0 msgr2=0x7f1bcc0b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bcc0b43a0 0x7f1bcc0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 -- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bcc1f1090 msgr2=0x7f1bcc0b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bcc1f1090 0x7f1bcc0b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 -- 172.21.15.107:0/3880565669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0c6330 con 0x7f1bac0047c0 2023-11-24T08:08:40.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.561+0000 7f1bcb7fe640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00e540 tx=0x7f1bbc00ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.562+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc013610 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.562+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc013770 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.562+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc013ae0 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.562+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc0c6610 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.562+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc0c6bd0 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.563+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc04e840 con 0x7f1bac0047c0 2023-11-24T08:08:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.564+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f1bbc013c40 con 0x7f1bac0047c0 2023-11-24T08:08:40.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.565+0000 7f1bc8ff9640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 0x7f1bb006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:40.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.565+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1bbc089eb0 con 0x7f1bac0047c0 2023-11-24T08:08:40.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.565+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 0x7f1bb006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:40.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.566+0000 7f1bc3fff640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 0x7f1bb006d8a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4033000 tx=0x7f1bb402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:40.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.569+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bbc054580 con 0x7f1bac0047c0 2023-11-24T08:08:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.760+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1bcc0ce640 con 0x7f1bac0047c0 2023-11-24T08:08:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.761+0000 7f1bc8ff9640 1 -- 172.21.15.107:0/3880565669 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1bbc057ba0 con 0x7f1bac0047c0 2023-11-24T08:08:40.763 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 msgr2=0x7f1bb006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 0x7f1bb006d8a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4033000 tx=0x7f1bb402f000 comp rx=0 tx=0).stop 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 msgr2=0x7f1bcc0b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00e540 tx=0x7f1bbc00ea00 comp rx=0 tx=0).stop 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 shutdown_connections 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1bb006b3e0 0x7f1bb006d8a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bcc0b43a0 0x7f1bcc0b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bcc1f1090 0x7f1bcc0b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 --2- 172.21.15.107:0/3880565669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bac0047c0 0x7f1bcc0b0240 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 >> 172.21.15.107:0/3880565669 conn(0x7f1bcc0cbb40 msgr2=0x7f1bcc0ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 shutdown_connections 2023-11-24T08:08:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.764+0000 7f1bd0c1a640 1 -- 172.21.15.107:0/3880565669 wait complete. 2023-11-24T08:08:40.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:08:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.999+0000 7f820164b640 1 Processor -- start 2023-11-24T08:08:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:40.999+0000 7f820164b640 1 -- start start 2023-11-24T08:08:41.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.001+0000 7f820164b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81fc144e10 0x7f81fc15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.001+0000 7f820164b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f81fc0cf150 0x7f81fc06a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.001+0000 7f820164b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc06ae80 con 0x7f81fc06a500 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.001+0000 7f820164b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81fc15be30 con 0x7f81fc144e10 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.001+0000 7f820164b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81fc06b330 con 0x7f81fc0cf150 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81fc144e10 0x7f81fc15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81fc144e10 0x7f81fc15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55224/0 (socket says 172.21.15.107:55224) 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 -- 172.21.15.107:0/1630891811 learned_addr learned my addr 172.21.15.107:0/1630891811 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 -- 172.21.15.107:0/1630891811 >> v2:172.21.15.120:3300/0 conn(0x7f81fc0cf150 msgr2=0x7f81fc06a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 --2- 172.21.15.107:0/1630891811 >> v2:172.21.15.120:3300/0 conn(0x7f81fc0cf150 0x7f81fc06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 -- 172.21.15.107:0/1630891811 >> v1:172.21.15.120:6789/0 conn(0x7f81fc06a500 legacy=0x7f81fc0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.002+0000 7f81fbfff640 1 -- 172.21.15.107:0/1630891811 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0cf700 con 0x7f81fc144e10 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.003+0000 7f81fbfff640 1 --2- 172.21.15.107:0/1630891811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc144e10 0x7f81fc15b8f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0cc470 tx=0x7f81ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428dc4f6c8c65d9e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.003+0000 7f81fb7fe640 1 -- 172.21.15.107:0/1630891811 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec031870 con 0x7f81fc144e10 2023-11-24T08:08:41.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f81fb7fe640 1 -- 172.21.15.107:0/1630891811 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec0319d0 con 0x7f81fc144e10 2023-11-24T08:08:41.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f81fb7fe640 1 -- 172.21.15.107:0/1630891811 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec031d60 con 0x7f81fc144e10 2023-11-24T08:08:41.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 -- 172.21.15.107:0/1630891811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc144e10 msgr2=0x7f81fc15b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 --2- 172.21.15.107:0/1630891811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc144e10 0x7f81fc15b8f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0cc470 tx=0x7f81ec02b910 comp rx=0 tx=0).stop 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 -- 172.21.15.107:0/1630891811 shutdown_connections 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 --2- 172.21.15.107:0/1630891811 >> v2:172.21.15.120:3300/0 conn(0x7f81fc0cf150 0x7f81fc06a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 --2- 172.21.15.107:0/1630891811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc144e10 0x7f81fc15b8f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 -- 172.21.15.107:0/1630891811 >> 172.21.15.107:0/1630891811 conn(0x7f81fc0cff80 msgr2=0x7f81fc0d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 -- 172.21.15.107:0/1630891811 shutdown_connections 2023-11-24T08:08:41.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.004+0000 7f820164b640 1 -- 172.21.15.107:0/1630891811 wait complete. 2023-11-24T08:08:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.006+0000 7f820164b640 1 Processor -- start 2023-11-24T08:08:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.006+0000 7f820164b640 1 -- start start 2023-11-24T08:08:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81fc0cf150 0x7f81fc159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 0x7f81fc15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1f2ec0 con 0x7f81fc06a500 2023-11-24T08:08:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1f3e90 con 0x7f81fc0cf150 2023-11-24T08:08:41.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f820164b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1f3a00 con 0x7f81fc144e10 2023-11-24T08:08:41.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f81fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f81fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55230/0 (socket says 172.21.15.107:55230) 2023-11-24T08:08:41.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.007+0000 7f81fbfff640 1 -- 172.21.15.107:0/3827383458 learned_addr learned my addr 172.21.15.107:0/3827383458 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81f3fff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81fc0cf150 0x7f81fc159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f8200e4a640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 0x7f81fc15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 -- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 msgr2=0x7f81fc15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 0x7f81fc15a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 -- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81fc0cf150 msgr2=0x7f81fc159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81fc0cf150 0x7f81fc159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 -- 172.21.15.107:0/3827383458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0094d0 con 0x7f81fc06a500 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f8200e4a640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 0x7f81fc15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81fbfff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000c00 tx=0x7f81ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.008+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec03ad10 con 0x7f81fc06a500 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.009+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec03ae70 con 0x7f81fc06a500 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.009+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec044610 con 0x7f81fc06a500 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.009+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc1f30d0 con 0x7f81fc06a500 2023-11-24T08:08:41.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.009+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc1f3690 con 0x7f81fc06a500 2023-11-24T08:08:41.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.010+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc04e840 con 0x7f81fc06a500 2023-11-24T08:08:41.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.011+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f81ec04d450 con 0x7f81fc06a500 2023-11-24T08:08:41.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.012+0000 7f81f97fa640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 0x7f81dc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.012+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f81ec02d040 con 0x7f81fc06a500 2023-11-24T08:08:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.012+0000 7f81f3fff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 0x7f81dc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.013+0000 7f81f3fff640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 0x7f81dc06d8a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f81fc144020 tx=0x7f81e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.016+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81ec076f80 con 0x7f81fc06a500 2023-11-24T08:08:41.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.207+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f81fc0b08d0 con 0x7f81fc06a500 2023-11-24T08:08:41.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.207+0000 7f81f97fa640 1 -- 172.21.15.107:0/3827383458 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7f81ec07a5a0 con 0x7f81fc06a500 2023-11-24T08:08:41.209 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:08:41.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 msgr2=0x7f81dc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 0x7f81dc06d8a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f81fc144020 tx=0x7f81e4009040 comp rx=0 tx=0).stop 2023-11-24T08:08:41.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 msgr2=0x7f81fc158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000c00 tx=0x7f81ec0046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 shutdown_connections 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f81dc06b3e0 0x7f81dc06d8a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81fc144e10 0x7f81fc15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81fc0cf150 0x7f81fc159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 --2- 172.21.15.107:0/3827383458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81fc06a500 0x7f81fc158e30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 >> 172.21.15.107:0/3827383458 conn(0x7f81fc0cff80 msgr2=0x7f81fc0b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 shutdown_connections 2023-11-24T08:08:41.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.210+0000 7f820164b640 1 -- 172.21.15.107:0/3827383458 wait complete. 2023-11-24T08:08:41.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:08:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.444+0000 7ff200490640 1 Processor -- start 2023-11-24T08:08:41.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.445+0000 7ff200490640 1 -- start start 2023-11-24T08:08:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.446+0000 7ff200490640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1f80ccbd0 0x7ff1f81660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.446+0000 7ff200490640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f81655f0 con 0x7ff1f80ccfb0 2023-11-24T08:08:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.446+0000 7ff200490640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80cc380 con 0x7ff1f80c9650 2023-11-24T08:08:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.446+0000 7ff200490640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f8162b30 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fe22c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1f80ccbd0 0x7ff1f81660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fda2b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1f80ccfb0 0x7ff1f8162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53732/0 (socket says 172.21.15.107:53732) 2023-11-24T08:08:41.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fda2b640 1 -- 172.21.15.107:0/4212732127 learned_addr learned my addr 172.21.15.107:0/4212732127 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fd22a640 1 -- 172.21.15.107:0/4212732127 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665580402 0 0) 0x7ff1f81655f0 con 0x7ff1f80ccfb0 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fd22a640 1 -- 172.21.15.107:0/4212732127 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4003520 con 0x7ff1f80ccfb0 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fe22c640 1 -- 172.21.15.107:0/4212732127 >> v1:172.21.15.120:6789/0 conn(0x7ff1f80c9650 legacy=0x7ff1f80c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fe22c640 1 -- 172.21.15.107:0/4212732127 >> v1:172.21.15.107:6789/0 conn(0x7ff1f80ccfb0 legacy=0x7ff1f8162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.447+0000 7ff1fe22c640 1 -- 172.21.15.107:0/4212732127 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8162e10 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.449+0000 7ff1fe22c640 1 --2- 172.21.15.107:0/4212732127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80ccbd0 0x7ff1f81660b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eeb299ca6467449 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.449+0000 7ff1fd22a640 1 -- 172.21.15.107:0/4212732127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f402cd00 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.449+0000 7ff1fd22a640 1 -- 172.21.15.107:0/4212732127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f402ce60 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.449+0000 7ff1fd22a640 1 -- 172.21.15.107:0/4212732127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4034660 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 -- 172.21.15.107:0/4212732127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80ccbd0 msgr2=0x7ff1f81660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 --2- 172.21.15.107:0/4212732127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80ccbd0 0x7ff1f81660b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f402bf50 comp rx=0 tx=0).stop 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 -- 172.21.15.107:0/4212732127 shutdown_connections 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 --2- 172.21.15.107:0/4212732127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80ccbd0 0x7ff1f81660b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 -- 172.21.15.107:0/4212732127 >> 172.21.15.107:0/4212732127 conn(0x7ff1f80b0c50 msgr2=0x7ff1f80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 -- 172.21.15.107:0/4212732127 shutdown_connections 2023-11-24T08:08:41.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.450+0000 7ff200490640 1 -- 172.21.15.107:0/4212732127 wait complete. 2023-11-24T08:08:41.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.452+0000 7ff200490640 1 Processor -- start 2023-11-24T08:08:41.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.452+0000 7ff200490640 1 -- start start 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1f80ccbd0 0x7ff1f815c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1f80ccfb0 0x7ff1f814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f8163130 con 0x7ff1f80c9650 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80999d0 con 0x7ff1f80ccfb0 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff200490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f8161960 con 0x7ff1f80ccbd0 2023-11-24T08:08:41.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55260/0 (socket says 172.21.15.107:55260) 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 -- 172.21.15.107:0/1737577990 learned_addr learned my addr 172.21.15.107:0/1737577990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fda2b640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1f80ccbd0 0x7ff1f815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fea2d640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1f80ccfb0 0x7ff1f814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 -- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1f80ccbd0 msgr2=0x7ff1f815c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1f80ccbd0 0x7ff1f815c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 -- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1f80ccfb0 msgr2=0x7ff1f814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1f80ccfb0 0x7ff1f814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.453+0000 7ff1fe22c640 1 -- 172.21.15.107:0/1737577990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009a70 con 0x7ff1f80c9650 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff1fe22c640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f403d600 con 0x7ff1f80c9650 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f403d7a0 con 0x7ff1f80c9650 2023-11-24T08:08:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f815ecd0 con 0x7ff1f80c9650 2023-11-24T08:08:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f403db50 con 0x7ff1f80c9650 2023-11-24T08:08:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.454+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f815f290 con 0x7ff1f80c9650 2023-11-24T08:08:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.455+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f804e8b0 con 0x7ff1f80c9650 2023-11-24T08:08:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.456+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7ff1f403b040 con 0x7ff1f80c9650 2023-11-24T08:08:41.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.458+0000 7ff1eeffd640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 0x7ff1dc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:41.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.458+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff1f40ac2a0 con 0x7ff1f80c9650 2023-11-24T08:08:41.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.458+0000 7ff1fda2b640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 0x7ff1dc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:41.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.459+0000 7ff1fda2b640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 0x7ff1dc06d8a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80cb1d0 tx=0x7ff1e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:41.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.462+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1f4076bf0 con 0x7ff1f80c9650 2023-11-24T08:08:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.664+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1f812c1b0 con 0x7ff1f80c9650 2023-11-24T08:08:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.665+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1737577990 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v51) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff1f4039080 con 0x7ff1f80c9650 2023-11-24T08:08:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 msgr2=0x7ff1dc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 0x7ff1dc06d8a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80cb1d0 tx=0x7ff1e8009040 comp rx=0 tx=0).stop 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 msgr2=0x7ff1f81612d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f4004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 shutdown_connections 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1dc06b3e0 0x7ff1dc06d8a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1f80ccfb0 0x7ff1f814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1f80ccbd0 0x7ff1f815c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 --2- 172.21.15.107:0/1737577990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1f80c9650 0x7ff1f81612d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 >> 172.21.15.107:0/1737577990 conn(0x7ff1f80b0c50 msgr2=0x7ff1f806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 shutdown_connections 2023-11-24T08:08:41.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:41.668+0000 7ff200490640 1 -- 172.21.15.107:0/1737577990 wait complete. 2023-11-24T08:08:41.689 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:08:41.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:41.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:08:41.810 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:08:41.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:41.810 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:08:41.873 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:08:41.873 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:41.914 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:08:41.914 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:42.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:42.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.209+0000 7f81a7181640 1 Processor -- start 2023-11-24T08:08:42.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.209+0000 7f81a7181640 1 -- start start 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a7181640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81a015d2b0 0x7f81a00d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a7181640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f81a015d690 0x7f81a00b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a7181640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00d1700 con 0x7f81a00cedd0 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a7181640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00d19e0 con 0x7f81a015d2b0 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a7181640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00d1cc0 con 0x7f81a015d690 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a617f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81a015d2b0 0x7f81a00d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a617f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f81a015d2b0 0x7f81a00d0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55262/0 (socket says 172.21.15.107:55262) 2023-11-24T08:08:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a597e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f81a00cedd0 0x7f81a00d2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53736/0 (socket says 172.21.15.107:53736) 2023-11-24T08:08:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.211+0000 7f81a617f640 1 -- 172.21.15.107:0/3716864690 learned_addr learned my addr 172.21.15.107:0/3716864690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a6980640 1 --2- 172.21.15.107:0/3716864690 >> v2:172.21.15.120:3300/0 conn(0x7f81a015d690 0x7f81a00b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a617f640 1 -- 172.21.15.107:0/3716864690 >> v2:172.21.15.120:3300/0 conn(0x7f81a015d690 msgr2=0x7f81a00b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a617f640 1 --2- 172.21.15.107:0/3716864690 >> v2:172.21.15.120:3300/0 conn(0x7f81a015d690 0x7f81a00b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a617f640 1 -- 172.21.15.107:0/3716864690 >> v1:172.21.15.107:6789/0 conn(0x7f81a00cedd0 legacy=0x7f81a00d2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a517d640 1 -- 172.21.15.107:0/3716864690 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680509199 0 0) 0x7f81a00d1700 con 0x7f81a00cedd0 2023-11-24T08:08:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.212+0000 7f81a617f640 1 -- 172.21.15.107:0/3716864690 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a00d1fa0 con 0x7f81a015d2b0 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.213+0000 7f81a617f640 1 --2- 172.21.15.107:0/3716864690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a015d2b0 0x7f81a00d0d90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f81a00d3e00 tx=0x7f819c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae0006024f343ae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.213+0000 7f81a517d640 1 -- 172.21.15.107:0/3716864690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c030860 con 0x7f81a015d2b0 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.213+0000 7f81a517d640 1 -- 172.21.15.107:0/3716864690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c0309c0 con 0x7f81a015d2b0 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.213+0000 7f81a517d640 1 -- 172.21.15.107:0/3716864690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c030d50 con 0x7f81a015d2b0 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 -- 172.21.15.107:0/3716864690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a015d2b0 msgr2=0x7f81a00d0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 --2- 172.21.15.107:0/3716864690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a015d2b0 0x7f81a00d0d90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f81a00d3e00 tx=0x7f819c02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 -- 172.21.15.107:0/3716864690 shutdown_connections 2023-11-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 --2- 172.21.15.107:0/3716864690 >> v2:172.21.15.120:3300/0 conn(0x7f81a015d690 0x7f81a00b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 --2- 172.21.15.107:0/3716864690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a015d2b0 0x7f81a00d0d90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 -- 172.21.15.107:0/3716864690 >> 172.21.15.107:0/3716864690 conn(0x7f81a008cbe0 msgr2=0x7f81a009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 -- 172.21.15.107:0/3716864690 shutdown_connections 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.214+0000 7f81a7181640 1 -- 172.21.15.107:0/3716864690 wait complete. 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.216+0000 7f81a7181640 1 Processor -- start 2023-11-24T08:08:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.216+0000 7f81a7181640 1 -- start start 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81a015d2b0 0x7f81a00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81a015d690 0x7f81a00b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a015fd20 con 0x7f81a00cedd0 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0160360 con 0x7f81a015d690 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a7181640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00c9d60 con 0x7f81a015d2b0 2023-11-24T08:08:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55270/0 (socket says 172.21.15.107:55270) 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 -- 172.21.15.107:0/1235937895 learned_addr learned my addr 172.21.15.107:0/1235937895 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 -- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81a015d2b0 msgr2=0x7f81a00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81a015d2b0 0x7f81a00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 -- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81a015d690 msgr2=0x7f81a00b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81a015d690 0x7f81a00b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.217+0000 7f81a617f640 1 -- 172.21.15.107:0/1235937895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c0094d0 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f81a617f640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f819c000c00 tx=0x7f819c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c03e040 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c039cb0 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c004270 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a00c7d90 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.218+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a015fa40 con 0x7f81a00cedd0 2023-11-24T08:08:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.219+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a0067990 con 0x7f81a00cedd0 2023-11-24T08:08:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.220+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f819c0043d0 con 0x7f81a00cedd0 2023-11-24T08:08:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.221+0000 7f8196ffd640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 0x7f818406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.222+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f819c0acc00 con 0x7f81a00cedd0 2023-11-24T08:08:42.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.222+0000 7f81a597e640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 0x7f818406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.222+0000 7f81a597e640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 0x7f818406d8a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8190007fd0 tx=0x7f8190007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.226+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f819c0773b0 con 0x7f81a00cedd0 2023-11-24T08:08:42.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.417+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81a00cf620 con 0x7f81a00cedd0 2023-11-24T08:08:42.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.419+0000 7f8196ffd640 1 -- 172.21.15.107:0/1235937895 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16670 (secure 0 0 0) 0x7f819c07a9d0 con 0x7f81a00cedd0 2023-11-24T08:08:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:42.421 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:37.801647+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:42.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 msgr2=0x7f818406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 0x7f818406d8a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8190007fd0 tx=0x7f8190007920 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 msgr2=0x7f81a00c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f819c000c00 tx=0x7f819c004730 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 shutdown_connections 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f818406b3e0 0x7f818406d8a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81a015d690 0x7f81a00b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81a015d2b0 0x7f81a00c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 --2- 172.21.15.107:0/1235937895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a00cedd0 0x7f81a00c9820 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 >> 172.21.15.107:0/1235937895 conn(0x7f81a008cbe0 msgr2=0x7f81a00d3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 shutdown_connections 2023-11-24T08:08:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.422+0000 7f81a7181640 1 -- 172.21.15.107:0/1235937895 wait complete. 2023-11-24T08:08:42.442 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:07:38.663153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:07:39.906724+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:08:42.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:08:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.651+0000 7febaffff640 1 Processor -- start 2023-11-24T08:08:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.651+0000 7febaffff640 1 -- start start 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.652+0000 7febaffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7febb015a1c0 0x7febb006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7febb015e5e0 0x7febb00b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb006d430 con 0x7febb00b0b40 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb015ea70 con 0x7febb015e5e0 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb015ec50 con 0x7febb015a1c0 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7febb015e5e0 0x7febb00b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7febb015e5e0 0x7febb00b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55274/0 (socket says 172.21.15.107:55274) 2023-11-24T08:08:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 -- 172.21.15.107:0/553602251 learned_addr learned my addr 172.21.15.107:0/553602251 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaeffd640 1 --2- 172.21.15.107:0/553602251 >> v2:172.21.15.120:3300/0 conn(0x7febb015a1c0 0x7febb006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 -- 172.21.15.107:0/553602251 >> v2:172.21.15.120:3300/0 conn(0x7febb015a1c0 msgr2=0x7febb006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 --2- 172.21.15.107:0/553602251 >> v2:172.21.15.120:3300/0 conn(0x7febb015a1c0 0x7febb006d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 -- 172.21.15.107:0/553602251 >> v1:172.21.15.107:6789/0 conn(0x7febb00b0b40 legacy=0x7febb015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.653+0000 7febaf7fe640 1 -- 172.21.15.107:0/553602251 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00b14e0 con 0x7febb015e5e0 2023-11-24T08:08:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.655+0000 7febaf7fe640 1 --2- 172.21.15.107:0/553602251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015e5e0 0x7febb00b0fa0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7febb00cca80 tx=0x7feba802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a46d154b59812b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.655+0000 7febadffb640 1 -- 172.21.15.107:0/553602251 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba802cb10 con 0x7febb015e5e0 2023-11-24T08:08:42.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.655+0000 7febadffb640 1 -- 172.21.15.107:0/553602251 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba802cc70 con 0x7febb015e5e0 2023-11-24T08:08:42.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.655+0000 7febadffb640 1 -- 172.21.15.107:0/553602251 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80343f0 con 0x7febb015e5e0 2023-11-24T08:08:42.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.655+0000 7febaffff640 1 -- 172.21.15.107:0/553602251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015e5e0 msgr2=0x7febb00b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 --2- 172.21.15.107:0/553602251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015e5e0 0x7febb00b0fa0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7febb00cca80 tx=0x7feba802bf70 comp rx=0 tx=0).stop 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 -- 172.21.15.107:0/553602251 shutdown_connections 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 --2- 172.21.15.107:0/553602251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015e5e0 0x7febb00b0fa0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 --2- 172.21.15.107:0/553602251 >> v2:172.21.15.120:3300/0 conn(0x7febb015a1c0 0x7febb006d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 -- 172.21.15.107:0/553602251 >> 172.21.15.107:0/553602251 conn(0x7febb00cbb40 msgr2=0x7febb00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 -- 172.21.15.107:0/553602251 shutdown_connections 2023-11-24T08:08:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.656+0000 7febaffff640 1 -- 172.21.15.107:0/553602251 wait complete. 2023-11-24T08:08:42.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.658+0000 7febaffff640 1 Processor -- start 2023-11-24T08:08:42.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.658+0000 7febaffff640 1 -- start start 2023-11-24T08:08:42.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb00b0b40 0x7febb01560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 0x7febb00c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb015b590 con 0x7febb015a1c0 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb015b2b0 con 0x7febb00b0b40 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.659+0000 7febaffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb00c6230 con 0x7febb015e5e0 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55286/0 (socket says 172.21.15.107:55286) 2023-11-24T08:08:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 -- 172.21.15.107:0/1457491654 learned_addr learned my addr 172.21.15.107:0/1457491654 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febaeffd640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb00b0b40 0x7febb01560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febaf7fe640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 0x7febb00c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 -- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 msgr2=0x7febb00c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 0x7febb00c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 -- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb00b0b40 msgr2=0x7febb01560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb00b0b40 0x7febb01560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febae7fc640 1 -- 172.21.15.107:0/1457491654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8009b30 con 0x7febb015a1c0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.660+0000 7febaf7fe640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 0x7febb00c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7febae7fc640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feba400c400 tx=0x7feba400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40052d0 con 0x7febb015a1c0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba4005430 con 0x7febb015a1c0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb00c6510 con 0x7febb015a1c0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4003830 con 0x7febb015a1c0 2023-11-24T08:08:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.661+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb00b2670 con 0x7febb015a1c0 2023-11-24T08:08:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.662+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb004e840 con 0x7febb015a1c0 2023-11-24T08:08:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.663+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7feba4003990 con 0x7febb015a1c0 2023-11-24T08:08:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.664+0000 7feb97fff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 0x7feb9c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.664+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feba4089610 con 0x7febb015a1c0 2023-11-24T08:08:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.664+0000 7febaeffd640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 0x7feb9c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.665+0000 7febaeffd640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 0x7feb9c06d8a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feba0006fd0 tx=0x7feba0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.668+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feba4053e40 con 0x7febb015a1c0 2023-11-24T08:08:42.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.860+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febb015c1f0 con 0x7febb015a1c0 2023-11-24T08:08:42.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.861+0000 7feb97fff640 1 -- 172.21.15.107:0/1457491654 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7feba4057460 con 0x7febb015a1c0 2023-11-24T08:08:42.863 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:08:42.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 msgr2=0x7feb9c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 0x7feb9c06d8a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feba0006fd0 tx=0x7feba0008040 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 msgr2=0x7febb00c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feba400c400 tx=0x7feba400c8c0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 shutdown_connections 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9c06b3e0 0x7feb9c06d8a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febb015e5e0 0x7febb00c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febb015a1c0 0x7febb00c8610 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 --2- 172.21.15.107:0/1457491654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb00b0b40 0x7febb01560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 >> 172.21.15.107:0/1457491654 conn(0x7febb00cbb40 msgr2=0x7febb0071680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.863+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 shutdown_connections 2023-11-24T08:08:42.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:42.864+0000 7febaffff640 1 -- 172.21.15.107:0/1457491654 wait complete. 2023-11-24T08:08:42.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:08:43.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.078+0000 7fadf05ce640 1 Processor -- start 2023-11-24T08:08:43.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.078+0000 7fadf05ce640 1 -- start start 2023-11-24T08:08:43.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fade80c49f0 0x7fade806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fade806dc40 0x7fade80cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade806e0f0 con 0x7fade80c49f0 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade815ec90 con 0x7fade806dc40 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.079+0000 7fadf05ce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade80cd530 con 0x7fade8072160 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fade80c49f0 0x7fade806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fade80c49f0 0x7fade806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55294/0 (socket says 172.21.15.107:55294) 2023-11-24T08:08:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 -- 172.21.15.107:0/2142843224 learned_addr learned my addr 172.21.15.107:0/2142843224 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadeeb6b640 1 --2- 172.21.15.107:0/2142843224 >> v2:172.21.15.120:3300/0 conn(0x7fade806dc40 0x7fade80cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadee36a640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 -- 172.21.15.107:0/2142843224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 msgr2=0x7fade815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 -- 172.21.15.107:0/2142843224 >> v2:172.21.15.120:3300/0 conn(0x7fade806dc40 msgr2=0x7fade80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 --2- 172.21.15.107:0/2142843224 >> v2:172.21.15.120:3300/0 conn(0x7fade806dc40 0x7fade80cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.080+0000 7fadedb69640 1 -- 172.21.15.107:0/2142843224 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80c4ed0 con 0x7fade80c49f0 2023-11-24T08:08:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.081+0000 7fadedb69640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade80c49f0 0x7fade806d700 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fade8072a30 tx=0x7fadd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdc141a6bea2d0c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7faded368640 1 -- 172.21.15.107:0/2142843224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8031870 con 0x7fade80c49f0 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7faded368640 1 -- 172.21.15.107:0/2142843224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd80319d0 con 0x7fade80c49f0 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7faded368640 1 -- 172.21.15.107:0/2142843224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8031d60 con 0x7fade80c49f0 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 -- 172.21.15.107:0/2142843224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade80c49f0 msgr2=0x7fade806d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade80c49f0 0x7fade806d700 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fade8072a30 tx=0x7fadd802b910 comp rx=0 tx=0).stop 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 -- 172.21.15.107:0/2142843224 shutdown_connections 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2142843224 >> v2:172.21.15.120:3300/0 conn(0x7fade806dc40 0x7fade80cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade80c49f0 0x7fade806d700 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2142843224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 -- 172.21.15.107:0/2142843224 >> 172.21.15.107:0/2142843224 conn(0x7fade80b0c50 msgr2=0x7fade80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 -- 172.21.15.107:0/2142843224 shutdown_connections 2023-11-24T08:08:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.082+0000 7fadf05ce640 1 -- 172.21.15.107:0/2142843224 wait complete. 2023-11-24T08:08:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.084+0000 7fadf05ce640 1 Processor -- start 2023-11-24T08:08:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.084+0000 7fadf05ce640 1 -- start start 2023-11-24T08:08:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade8144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80c49f0 0x7fade80c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade8159960 con 0x7fade806dc40 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade8159450 con 0x7fade80c49f0 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadf05ce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade815c570 con 0x7fade8072160 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.085+0000 7fadee36a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55306/0 (socket says 172.21.15.107:55306) 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 -- 172.21.15.107:0/2470318788 learned_addr learned my addr 172.21.15.107:0/2470318788 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 -- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 msgr2=0x7fade8144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade8144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 -- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80c49f0 msgr2=0x7fade80c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80c49f0 0x7fade80c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 -- 172.21.15.107:0/2470318788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80094d0 con 0x7fade806dc40 2023-11-24T08:08:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadee36a640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fade400d830 tx=0x7fade400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400b6c0 con 0x7fade806dc40 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade400b820 con 0x7fade806dc40 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400bbb0 con 0x7fade806dc40 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.086+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade815cf40 con 0x7fade806dc40 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.087+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade815bc40 con 0x7fade806dc40 2023-11-24T08:08:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.088+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade804e8b0 con 0x7fade806dc40 2023-11-24T08:08:43.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.088+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fade401e460 con 0x7fade806dc40 2023-11-24T08:08:43.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.089+0000 7faddeffd640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 0x7fadcc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.089+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fade408a8a0 con 0x7fade806dc40 2023-11-24T08:08:43.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.090+0000 7fadedb69640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 0x7fadcc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.090+0000 7fadedb69640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 0x7fadcc06d8a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fadd8036000 tx=0x7fadd8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.093+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fade4055040 con 0x7fade806dc40 2023-11-24T08:08:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.284+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fade815dab0 con 0x7fade806dc40 2023-11-24T08:08:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.285+0000 7faddeffd640 1 -- 172.21.15.107:0/2470318788 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7fade4058660 con 0x7fade806dc40 2023-11-24T08:08:43.287 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 msgr2=0x7fadcc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 0x7fadcc06d8a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fadd8036000 tx=0x7fadd8030000 comp rx=0 tx=0).stop 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 msgr2=0x7fade815ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fade400d830 tx=0x7fade400dcf0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 shutdown_connections 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadcc06b3e0 0x7fadcc06d8a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80c49f0 0x7fade80c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8072160 0x7fade8144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 --2- 172.21.15.107:0/2470318788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade806dc40 0x7fade815ca00 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 >> 172.21.15.107:0/2470318788 conn(0x7fade80b0c50 msgr2=0x7fade80b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 shutdown_connections 2023-11-24T08:08:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.288+0000 7fadf05ce640 1 -- 172.21.15.107:0/2470318788 wait complete. 2023-11-24T08:08:43.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:08:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.504+0000 7f34f5648640 1 Processor -- start 2023-11-24T08:08:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.504+0000 7f34f5648640 1 -- start start 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.506+0000 7f34f5648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.506+0000 7f34f5648640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 0x7f34f006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.506+0000 7f34f5648640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f015eea0 con 0x7f34f015b440 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.506+0000 7f34f5648640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f015b820 con 0x7f34f00c57e0 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.506+0000 7f34f5648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00abf30 con 0x7f34f00cc0f0 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.507+0000 7f34e7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.507+0000 7f34e7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:57550/0 (socket says 172.21.15.107:57550) 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.507+0000 7f34e7fff640 1 -- 172.21.15.107:0/444140320 learned_addr learned my addr 172.21.15.107:0/444140320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.507+0000 7f34ef7fe640 1 --2- 172.21.15.107:0/444140320 >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 0x7f34f006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.508+0000 7f34e7fff640 1 -- 172.21.15.107:0/444140320 >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 msgr2=0x7f34f006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.508+0000 7f34e7fff640 1 --2- 172.21.15.107:0/444140320 >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 0x7f34f006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.508+0000 7f34e7fff640 1 -- 172.21.15.107:0/444140320 >> v1:172.21.15.120:6789/0 conn(0x7f34f015b440 legacy=0x7f34f00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.508+0000 7f34e7fff640 1 -- 172.21.15.107:0/444140320 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f015f0f0 con 0x7f34f00cc0f0 2023-11-24T08:08:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.508+0000 7f34ef7fe640 1 --2- 172.21.15.107:0/444140320 >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 0x7f34f006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:08:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.509+0000 7f34e7fff640 1 --2- 172.21.15.107:0/444140320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009870 tx=0x7f34d802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=672c680b0c156641 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.510+0000 7f34ee7fc640 1 -- 172.21.15.107:0/444140320 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8031640 con 0x7f34f00cc0f0 2023-11-24T08:08:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.510+0000 7f34ee7fc640 1 -- 172.21.15.107:0/444140320 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d80317a0 con 0x7f34f00cc0f0 2023-11-24T08:08:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.510+0000 7f34ee7fc640 1 -- 172.21.15.107:0/444140320 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8031b30 con 0x7f34f00cc0f0 2023-11-24T08:08:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.510+0000 7f34f5648640 1 -- 172.21.15.107:0/444140320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 msgr2=0x7f34f00c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.510+0000 7f34f5648640 1 --2- 172.21.15.107:0/444140320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009870 tx=0x7f34d802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 -- 172.21.15.107:0/444140320 shutdown_connections 2023-11-24T08:08:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 --2- 172.21.15.107:0/444140320 >> v2:172.21.15.120:3300/0 conn(0x7f34f00c57e0 0x7f34f006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 --2- 172.21.15.107:0/444140320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00cc0f0 0x7f34f00c52a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 -- 172.21.15.107:0/444140320 >> 172.21.15.107:0/444140320 conn(0x7f34f00b0c50 msgr2=0x7f34f00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 -- 172.21.15.107:0/444140320 shutdown_connections 2023-11-24T08:08:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.511+0000 7f34f5648640 1 -- 172.21.15.107:0/444140320 wait complete. 2023-11-24T08:08:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.512+0000 7f34f5648640 1 Processor -- start 2023-11-24T08:08:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.513+0000 7f34f5648640 1 -- start start 2023-11-24T08:08:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 0x7f34f00afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00cc0f0 0x7f34f00c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f009ac10 con 0x7f34f015b440 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0099f90 con 0x7f34f00cc0f0 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34f5648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f009a270 con 0x7f34f00c57e0 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55308/0 (socket says 172.21.15.107:55308) 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 -- 172.21.15.107:0/3574606752 learned_addr learned my addr 172.21.15.107:0/3574606752 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34eeffd640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 0x7f34f00afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 -- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 msgr2=0x7f34f00afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 0x7f34f00afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 -- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00cc0f0 msgr2=0x7f34f00c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34e7fff640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00cc0f0 0x7f34f00c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00cc0f0 0x7f34f00c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34ef7fe640 1 -- 172.21.15.107:0/3574606752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8009a70 con 0x7f34f015b440 2023-11-24T08:08:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.514+0000 7f34eeffd640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 0x7f34f00afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34ef7fe640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f34e800efc0 tx=0x7f34e8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e8009d70 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e800f040 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e8003990 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d80094d0 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.515+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f009a830 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.516+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f009ad50 con 0x7f34f015b440 2023-11-24T08:08:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.517+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f34e80144a0 con 0x7f34f015b440 2023-11-24T08:08:43.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.518+0000 7f34e77fe640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 0x7f34d006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.518+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34e808ae80 con 0x7f34f015b440 2023-11-24T08:08:43.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.518+0000 7f34eeffd640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 0x7f34d006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.519+0000 7f34eeffd640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 0x7f34d006d8a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f34e00015e0 tx=0x7f34e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.523+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34f009ad50 con 0x7f34f015b440 2023-11-24T08:08:43.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.700+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34f009ad50 con 0x7f34f015b440 2023-11-24T08:08:43.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.700+0000 7f34e77fe640 1 -- 172.21.15.107:0/3574606752 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34f009ad50 con 0x7f34f015b440 2023-11-24T08:08:43.703 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:08:43.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 msgr2=0x7f34d006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 0x7f34d006d8a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f34e00015e0 tx=0x7f34e0009040 comp rx=0 tx=0).stop 2023-11-24T08:08:43.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 msgr2=0x7f34f00c6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f34e800efc0 tx=0x7f34e8009f90 comp rx=0 tx=0).stop 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 shutdown_connections 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34d006b3e0 0x7f34d006d8a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34f015b440 0x7f34f00c6610 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00cc0f0 0x7f34f00c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 --2- 172.21.15.107:0/3574606752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f00c57e0 0x7f34f00afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 >> 172.21.15.107:0/3574606752 conn(0x7f34f00b0c50 msgr2=0x7f34f00adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 shutdown_connections 2023-11-24T08:08:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.703+0000 7f34f5648640 1 -- 172.21.15.107:0/3574606752 wait complete. 2023-11-24T08:08:43.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:08:43.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.932+0000 7fa290189640 1 Processor -- start 2023-11-24T08:08:43.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.932+0000 7fa290189640 1 -- start start 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa290189640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa288072160 0x7fa28815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa290189640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa28806de10 0x7fa2880705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa290189640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28815ec90 con 0x7fa2880c49f0 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa290189640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2880cd530 con 0x7fa288072160 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa290189640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2880c4ed0 con 0x7fa28806de10 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa28df25640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa288072160 0x7fa28815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa28df25640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa288072160 0x7fa28815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55322/0 (socket says 172.21.15.107:55322) 2023-11-24T08:08:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.934+0000 7fa28df25640 1 -- 172.21.15.107:0/3717076589 learned_addr learned my addr 172.21.15.107:0/3717076589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28e726640 1 --2- 172.21.15.107:0/3717076589 >> v2:172.21.15.120:3300/0 conn(0x7fa28806de10 0x7fa2880705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28df25640 1 -- 172.21.15.107:0/3717076589 >> v2:172.21.15.120:3300/0 conn(0x7fa28806de10 msgr2=0x7fa2880705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28df25640 1 --2- 172.21.15.107:0/3717076589 >> v2:172.21.15.120:3300/0 conn(0x7fa28806de10 0x7fa2880705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28df25640 1 -- 172.21.15.107:0/3717076589 >> v1:172.21.15.107:6789/0 conn(0x7fa2880c49f0 legacy=0x7fa28806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28cf23640 1 -- 172.21.15.107:0/3717076589 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364591823 0 0) 0x7fa278001110 con 0x7fa2880c49f0 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.935+0000 7fa28df25640 1 -- 172.21.15.107:0/3717076589 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2880c51b0 con 0x7fa288072160 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.936+0000 7fa28df25640 1 --2- 172.21.15.107:0/3717076589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa28815e750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa28806eab0 tx=0x7fa28402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14639c9cddeff2f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.936+0000 7fa28cf23640 1 -- 172.21.15.107:0/3717076589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284030860 con 0x7fa288072160 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa28cf23640 1 -- 172.21.15.107:0/3717076589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2840309c0 con 0x7fa288072160 2023-11-24T08:08:43.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa28cf23640 1 -- 172.21.15.107:0/3717076589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284030d50 con 0x7fa288072160 2023-11-24T08:08:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 -- 172.21.15.107:0/3717076589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 msgr2=0x7fa28815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 --2- 172.21.15.107:0/3717076589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa28815e750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa28806eab0 tx=0x7fa28402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 -- 172.21.15.107:0/3717076589 shutdown_connections 2023-11-24T08:08:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 --2- 172.21.15.107:0/3717076589 >> v2:172.21.15.120:3300/0 conn(0x7fa28806de10 0x7fa2880705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 --2- 172.21.15.107:0/3717076589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa28815e750 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 -- 172.21.15.107:0/3717076589 >> 172.21.15.107:0/3717076589 conn(0x7fa2880b0c50 msgr2=0x7fa2880b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 -- 172.21.15.107:0/3717076589 shutdown_connections 2023-11-24T08:08:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.937+0000 7fa290189640 1 -- 172.21.15.107:0/3717076589 wait complete. 2023-11-24T08:08:43.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.939+0000 7fa290189640 1 Processor -- start 2023-11-24T08:08:43.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.939+0000 7fa290189640 1 -- start start 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 0x7fa2880b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880c49f0 0x7fa288155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28806dc60 con 0x7fa288072160 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880b46c0 con 0x7fa28806de10 2023-11-24T08:08:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa290189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28806d400 con 0x7fa2880c49f0 2023-11-24T08:08:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.940+0000 7fa28d724640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55336/0 (socket says 172.21.15.107:55336) 2023-11-24T08:08:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 -- 172.21.15.107:0/1310217418 learned_addr learned my addr 172.21.15.107:0/1310217418 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28df25640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 0x7fa2880b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 -- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880c49f0 msgr2=0x7fa288155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880c49f0 0x7fa288155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 -- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 msgr2=0x7fa2880b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 0x7fa2880b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 -- 172.21.15.107:0/1310217418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2840094d0 con 0x7fa288072160 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28df25640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 0x7fa2880b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.941+0000 7fa28d724640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa27800c410 tx=0x7fa27800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.942+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2780035f0 con 0x7fa288072160 2023-11-24T08:08:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.942+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa278003770 con 0x7fa288072160 2023-11-24T08:08:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.942+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278003b20 con 0x7fa288072160 2023-11-24T08:08:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.942+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2880b4390 con 0x7fa288072160 2023-11-24T08:08:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.942+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28806d6a0 con 0x7fa288072160 2023-11-24T08:08:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.943+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2880b41f0 con 0x7fa288072160 2023-11-24T08:08:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.944+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fa278003c80 con 0x7fa288072160 2023-11-24T08:08:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.945+0000 7fa27e7fc640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 0x7fa26c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.945+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa278089590 con 0x7fa288072160 2023-11-24T08:08:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.945+0000 7fa28df25640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 0x7fa26c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.946+0000 7fa28df25640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 0x7fa26c06d8a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa284036000 tx=0x7fa28403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:43.949+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2880b41f0 con 0x7fa288072160 2023-11-24T08:08:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.154+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa28815ec90 con 0x7fa288072160 2023-11-24T08:08:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.154+0000 7fa27e7fc640 1 -- 172.21.15.107:0/1310217418 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v51) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa28815ec90 con 0x7fa288072160 2023-11-24T08:08:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 msgr2=0x7fa26c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 0x7fa26c06d8a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa284036000 tx=0x7fa28403b040 comp rx=0 tx=0).stop 2023-11-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 msgr2=0x7fa2880b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa27800c410 tx=0x7fa27800c8d0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 shutdown_connections 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa26c06b3e0 0x7fa26c06d8a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880c49f0 0x7fa288155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288072160 0x7fa2880b1d10 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 --2- 172.21.15.107:0/1310217418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa28806de10 0x7fa2880b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 >> 172.21.15.107:0/1310217418 conn(0x7fa2880b0c50 msgr2=0x7fa2880b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 shutdown_connections 2023-11-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.157+0000 7fa290189640 1 -- 172.21.15.107:0/1310217418 wait complete. 2023-11-24T08:08:44.179 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:08:44.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:44.180 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:08:44.295 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:08:44.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:08:44.295 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:08:44.349 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:44.349 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:44.408 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:08:44.409 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:08:44.409 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:44.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:08:44.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:08:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.686+0000 7f6b6369b640 1 Processor -- start 2023-11-24T08:08:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.686+0000 7f6b6369b640 1 -- start start 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.687+0000 7f6b6369b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b5c0cfb60 0x7f6b5c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.687+0000 7f6b6369b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0ce0b0 con 0x7f6b5c15dc80 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.687+0000 7f6b6369b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c161960 con 0x7f6b5c157c70 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.687+0000 7f6b6369b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c15e8d0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b60c36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b5c0cfb60 0x7f6b5c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b61c38640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6b5c15dc80 0x7f6b5c0b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53762/0 (socket says 172.21.15.107:53762) 2023-11-24T08:08:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b61c38640 1 -- 172.21.15.107:0/2809721188 learned_addr learned my addr 172.21.15.107:0/2809721188 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b60c36640 1 -- 172.21.15.107:0/2809721188 >> v1:172.21.15.120:6789/0 conn(0x7f6b5c157c70 legacy=0x7f6b5c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b60c36640 1 -- 172.21.15.107:0/2809721188 >> v1:172.21.15.107:6789/0 conn(0x7f6b5c15dc80 legacy=0x7f6b5c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b60c36640 1 -- 172.21.15.107:0/2809721188 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c15ebb0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.688+0000 7f6b53fff640 1 -- 172.21.15.107:0/2809721188 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283908967 0 0) 0x7f6b5c0ce0b0 con 0x7f6b5c15dc80 2023-11-24T08:08:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.689+0000 7f6b60c36640 1 --2- 172.21.15.107:0/2809721188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c161df0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0097a0 tx=0x7f6b4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea7ef07abdf5322 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b53fff640 1 -- 172.21.15.107:0/2809721188 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c02e070 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b53fff640 1 -- 172.21.15.107:0/2809721188 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c0315a0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b53fff640 1 -- 172.21.15.107:0/2809721188 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c031930 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 -- 172.21.15.107:0/2809721188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 msgr2=0x7f6b5c161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 --2- 172.21.15.107:0/2809721188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c161df0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0097a0 tx=0x7f6b4c02b840 comp rx=0 tx=0).stop 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 -- 172.21.15.107:0/2809721188 shutdown_connections 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 --2- 172.21.15.107:0/2809721188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c161df0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 -- 172.21.15.107:0/2809721188 >> 172.21.15.107:0/2809721188 conn(0x7f6b5c06a920 msgr2=0x7f6b5c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 -- 172.21.15.107:0/2809721188 shutdown_connections 2023-11-24T08:08:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.690+0000 7f6b6369b640 1 -- 172.21.15.107:0/2809721188 wait complete. 2023-11-24T08:08:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.692+0000 7f6b6369b640 1 Processor -- start 2023-11-24T08:08:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.692+0000 7f6b6369b640 1 -- start start 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b5c157c70 0x7f6b5c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b5c15dc80 0x7f6b5c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c155c90 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0cc060 con 0x7f6b5c157c70 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b6369b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1f4130 con 0x7f6b5c15dc80 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.693+0000 7f6b61437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55344/0 (socket says 172.21.15.107:55344) 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 -- 172.21.15.107:0/55278137 learned_addr learned my addr 172.21.15.107:0/55278137 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 -- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b5c15dc80 msgr2=0x7f6b5c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b5c15dc80 0x7f6b5c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 -- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b5c157c70 msgr2=0x7f6b5c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b5c157c70 0x7f6b5c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 -- 172.21.15.107:0/55278137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c009400 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b61437640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800c430 tx=0x7f6b5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58011790 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b580118f0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c1f4410 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.694+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58011ca0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.695+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c1f49d0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.696+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c067470 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.697+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f6b5801c460 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.697+0000 7f6b51ffb640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 0x7f6b4006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.698+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b5808a0d0 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.698+0000 7f6b60c36640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 0x7f6b4006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.699+0000 7f6b60c36640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 0x7f6b4006d8a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c036000 tx=0x7f6b4c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:44.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.702+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b58054870 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.933+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b5c0cd910 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.934+0000 7f6b51ffb640 1 -- 172.21.15.107:0/55278137 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f6b58057e90 con 0x7f6b5c0cfb60 2023-11-24T08:08:44.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:44.937 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:08:44.937 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2023-11-24T08:08:44.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 msgr2=0x7f6b4006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 0x7f6b4006d8a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c036000 tx=0x7f6b4c030000 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 msgr2=0x7f6b5c155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800c430 tx=0x7f6b5800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 shutdown_connections 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b4006b3e0 0x7f6b4006d8a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b5c15dc80 0x7f6b5c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b5c157c70 0x7f6b5c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 --2- 172.21.15.107:0/55278137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b5c0cfb60 0x7f6b5c155750 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 >> 172.21.15.107:0/55278137 conn(0x7f6b5c06a920 msgr2=0x7f6b5c0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 shutdown_connections 2023-11-24T08:08:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:44.937+0000 7f6b6369b640 1 -- 172.21.15.107:0/55278137 wait complete. 2023-11-24T08:08:44.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:45.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 Processor -- start 2023-11-24T08:08:45.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 -- start start 2023-11-24T08:08:45.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e80b2250 0x7ff5e815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 0x7ff5e80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e815ab80 con 0x7ff5e80b2aa0 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e815a990 con 0x7ff5e80b2250 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.165+0000 7ff5ef9e3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e815e680 con 0x7ff5e80b2630 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e80b2250 0x7ff5e815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e80b2250 0x7ff5e815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55352/0 (socket says 172.21.15.107:55352) 2023-11-24T08:08:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 -- 172.21.15.107:0/3007808036 learned_addr learned my addr 172.21.15.107:0/3007808036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 -- 172.21.15.107:0/3007808036 >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 msgr2=0x7ff5e80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ecf7e640 1 --2- 172.21.15.107:0/3007808036 >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 0x7ff5e80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 --2- 172.21.15.107:0/3007808036 >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 0x7ff5e80ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5dffff640 1 -- 172.21.15.107:0/3007808036 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819947543 0 0) 0x7ff5e815ab80 con 0x7ff5e80b2aa0 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 -- 172.21.15.107:0/3007808036 >> v1:172.21.15.107:6789/0 conn(0x7ff5e80b2aa0 legacy=0x7ff5e815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ed77f640 1 -- 172.21.15.107:0/3007808036 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e815e960 con 0x7ff5e80b2250 2023-11-24T08:08:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.166+0000 7ff5ecf7e640 1 --2- 172.21.15.107:0/3007808036 >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 0x7ff5e80ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:45.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.167+0000 7ff5ed77f640 1 --2- 172.21.15.107:0/3007808036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2250 0x7ff5e815df80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80ae5c0 tx=0x7ff5e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3745e218bce9f61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5dffff640 1 -- 172.21.15.107:0/3007808036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4030860 con 0x7ff5e80b2250 2023-11-24T08:08:45.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5dffff640 1 -- 172.21.15.107:0/3007808036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e40309c0 con 0x7ff5e80b2250 2023-11-24T08:08:45.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5dffff640 1 -- 172.21.15.107:0/3007808036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4030d50 con 0x7ff5e80b2250 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3007808036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2250 msgr2=0x7ff5e815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3007808036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2250 0x7ff5e815df80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80ae5c0 tx=0x7ff5e402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3007808036 shutdown_connections 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3007808036 >> v2:172.21.15.120:3300/0 conn(0x7ff5e80b2630 0x7ff5e80ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3007808036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2250 0x7ff5e815df80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3007808036 >> 172.21.15.107:0/3007808036 conn(0x7ff5e806a920 msgr2=0x7ff5e806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3007808036 shutdown_connections 2023-11-24T08:08:45.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.168+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3007808036 wait complete. 2023-11-24T08:08:45.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.170+0000 7ff5ef9e3640 1 Processor -- start 2023-11-24T08:08:45.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.170+0000 7ff5ef9e3640 1 -- start start 2023-11-24T08:08:45.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e80b2250 0x7ff5e81f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e80b2aa0 0x7ff5e8157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80b0be0 con 0x7ff5e80b2630 2023-11-24T08:08:45.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e81f41a0 con 0x7ff5e80b2250 2023-11-24T08:08:45.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ef9e3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80c87b0 con 0x7ff5e80b2aa0 2023-11-24T08:08:45.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ecf7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ecf7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55360/0 (socket says 172.21.15.107:55360) 2023-11-24T08:08:45.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.171+0000 7ff5ecf7e640 1 -- 172.21.15.107:0/3980044937 learned_addr learned my addr 172.21.15.107:0/3980044937 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 -- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e80b2aa0 msgr2=0x7ff5e8157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e80b2aa0 0x7ff5e8157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 -- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e80b2250 msgr2=0x7ff5e81f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ed77f640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e80b2250 0x7ff5e81f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e80b2250 0x7ff5e81f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 -- 172.21.15.107:0/3980044937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40094d0 con 0x7ff5e80b2630 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.172+0000 7ff5ecf7e640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c7f0 tx=0x7ff5d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.173+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8007a30 con 0x7ff5e80b2630 2023-11-24T08:08:45.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.173+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d8007b90 con 0x7ff5e80b2630 2023-11-24T08:08:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.173+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800ce40 con 0x7ff5e80b2630 2023-11-24T08:08:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.173+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e80c8af0 con 0x7ff5e80b2630 2023-11-24T08:08:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.173+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e80c9330 con 0x7ff5e80b2630 2023-11-24T08:08:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.174+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e8067470 con 0x7ff5e80b2630 2023-11-24T08:08:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.175+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7ff5d80113f0 con 0x7ff5e80b2630 2023-11-24T08:08:45.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.176+0000 7ff5ddffb640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 0x7ff5cc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.176+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff5d808a3d0 con 0x7ff5e80b2630 2023-11-24T08:08:45.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.176+0000 7ff5ed77f640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 0x7ff5cc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.177+0000 7ff5ed77f640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 0x7ff5cc06d8a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4036000 tx=0x7ff5e403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.180+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5d8054a60 con 0x7ff5e80b2630 2023-11-24T08:08:45.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.373+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5e80cea00 con 0x7ff5e80b2630 2023-11-24T08:08:45.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.375+0000 7ff5ddffb640 1 -- 172.21.15.107:0/3980044937 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16670 (secure 0 0 0) 0x7ff5d8058080 con 0x7ff5e80b2630 2023-11-24T08:08:45.378 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:45.378 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:44.788580+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 msgr2=0x7ff5cc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 0x7ff5cc06d8a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4036000 tx=0x7ff5e403b040 comp rx=0 tx=0).stop 2023-11-24T08:08:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 msgr2=0x7ff5e8071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c7f0 tx=0x7ff5d800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 shutdown_connections 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5cc06b3e0 0x7ff5cc06d8a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e80b2aa0 0x7ff5e8157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e80b2630 0x7ff5e8071880 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 --2- 172.21.15.107:0/3980044937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e80b2250 0x7ff5e81f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 >> 172.21.15.107:0/3980044937 conn(0x7ff5e806a920 msgr2=0x7ff5e80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 shutdown_connections 2023-11-24T08:08:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.378+0000 7ff5ef9e3640 1 -- 172.21.15.107:0/3980044937 wait complete. 2023-11-24T08:08:45.402 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:07:38.663153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:07:39.906724+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:08:45.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:08:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 Processor -- start 2023-11-24T08:08:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 -- start start 2023-11-24T08:08:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd59815a1c0 0x7fd59806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd5980b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59806d430 con 0x7fd5980b0b40 2023-11-24T08:08:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59815ea70 con 0x7fd59815a1c0 2023-11-24T08:08:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.618+0000 7fd59ddcf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59815ec50 con 0x7fd59815e5e0 2023-11-24T08:08:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd597fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd5980b0b40 0x7fd59815ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53770/0 (socket says 172.21.15.107:53770) 2023-11-24T08:08:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd597fff640 1 -- 172.21.15.107:0/2833511635 learned_addr learned my addr 172.21.15.107:0/2833511635 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/2833511635 >> v2:172.21.15.120:3300/0 conn(0x7fd59815a1c0 0x7fd59806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd5980b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627717562 0 0) 0x7fd59806d430 con 0x7fd5980b0b40 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.619+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd588003520 con 0x7fd5980b0b40 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877616421 0 0) 0x7fd588003520 con 0x7fd5980b0b40 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 msgr2=0x7fd5980b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd5980b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 >> v2:172.21.15.120:3300/0 conn(0x7fd59815a1c0 msgr2=0x7fd59806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 --2- 172.21.15.107:0/2833511635 >> v2:172.21.15.120:3300/0 conn(0x7fd59815a1c0 0x7fd59806d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980b14e0 con 0x7fd5980b0b40 2023-11-24T08:08:45.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd584001670 con 0x7fd5980b0b40 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd5980b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 >> v1:172.21.15.107:6789/0 conn(0x7fd5980b0b40 legacy=0x7fd59815ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd588003970 0x7fd588003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd588005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59815ea70 con 0x7fd588003970 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59815ec50 con 0x7fd5880047c0 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58800db90 con 0x7fd59815e5e0 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd588003970 0x7fd588003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd597fff640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd588005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59cdcd640 1 -- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 msgr2=0x7fd588005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:45.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59815e5e0 0x7fd588005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59cdcd640 1 -- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5880047c0 msgr2=0x7fd588004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:45.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.620+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5880047c0 0x7fd588004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.621+0000 7fd59cdcd640 1 -- 172.21.15.107:0/2833511635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588003520 con 0x7fd588003970 2023-11-24T08:08:45.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd588003970 0x7fd588003d90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009af0 tx=0x7fd5880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424a10918fe48927 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c02c710 con 0x7fd588003970 2023-11-24T08:08:45.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c02c870 con 0x7fd588003970 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd5977fe640 1 -- 172.21.15.107:0/2833511635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c02cc00 con 0x7fd588003970 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd59ddcf640 1 -- 172.21.15.107:0/2833511635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd588003970 msgr2=0x7fd588003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/2833511635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd588003970 0x7fd588003d90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009af0 tx=0x7fd5880057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.622+0000 7fd59d5ce640 1 -- 172.21.15.107:0/2833511635 reap_dead start 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.623+0000 7fd59ddcf640 1 -- 172.21.15.107:0/2833511635 shutdown_connections 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.623+0000 7fd59ddcf640 1 -- 172.21.15.107:0/2833511635 >> 172.21.15.107:0/2833511635 conn(0x7fd5980cbb40 msgr2=0x7fd5980cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.623+0000 7fd59ddcf640 1 -- 172.21.15.107:0/2833511635 shutdown_connections 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.623+0000 7fd59ddcf640 1 -- 172.21.15.107:0/2833511635 wait complete. 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.624+0000 7fd59ddcf640 1 Processor -- start 2023-11-24T08:08:45.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.625+0000 7fd59ddcf640 1 -- start start 2023-11-24T08:08:45.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.625+0000 7fd59ddcf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 0x7fd5980b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59ddcf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59815c060 0x7fd5980c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59ddcf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59ddcf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980b4a70 con 0x7fd5980c88a0 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59ddcf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5981f2e00 con 0x7fd59815c060 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59ddcf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980c8d20 con 0x7fd588003970 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55386/0 (socket says 172.21.15.107:55386) 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 -- 172.21.15.107:0/1852650926 learned_addr learned my addr 172.21.15.107:0/1852650926 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd597fff640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59815c060 0x7fd5980c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 0x7fd5980b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 -- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 msgr2=0x7fd5980b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 0x7fd5980b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 -- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59815c060 msgr2=0x7fd5980c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59815c060 0x7fd5980c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59d5ce640 1 -- 172.21.15.107:0/1852650926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5981f3010 con 0x7fd5980c88a0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.626+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 0x7fd5980b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd59d5ce640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd59815e9c0 tx=0x7fd5900046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590007cf0 con 0x7fd5980c88a0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd590007e90 con 0x7fd5980c88a0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5981f32f0 con 0x7fd5980c88a0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590014640 con 0x7fd5980c88a0 2023-11-24T08:08:45.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.627+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5981f38b0 con 0x7fd5980c88a0 2023-11-24T08:08:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.628+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59804e840 con 0x7fd5980c88a0 2023-11-24T08:08:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.629+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fd5900147a0 con 0x7fd5980c88a0 2023-11-24T08:08:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.630+0000 7fd5957fa640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 0x7fd57c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.630+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd590089cf0 con 0x7fd5980c88a0 2023-11-24T08:08:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.630+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 0x7fd57c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.631+0000 7fd59cdcd640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 0x7fd57c06d8a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.634+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd590054610 con 0x7fd5980c88a0 2023-11-24T08:08:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.826+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5980ca020 con 0x7fd5980c88a0 2023-11-24T08:08:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.827+0000 7fd5957fa640 1 -- 172.21.15.107:0/1852650926 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd590057c30 con 0x7fd5980c88a0 2023-11-24T08:08:45.829 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 msgr2=0x7fd57c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 0x7fd57c06d8a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c030000 comp rx=0 tx=0).stop 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 msgr2=0x7fd59814e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd59815e9c0 tx=0x7fd5900046e0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 shutdown_connections 2023-11-24T08:08:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57c06b3e0 0x7fd57c06d8a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5980c88a0 0x7fd59814e170 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59815c060 0x7fd5980c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 --2- 172.21.15.107:0/1852650926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd588003970 0x7fd5980b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 >> 172.21.15.107:0/1852650926 conn(0x7fd5980cbb40 msgr2=0x7fd59809c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 shutdown_connections 2023-11-24T08:08:45.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:45.830+0000 7fd59ddcf640 1 -- 172.21.15.107:0/1852650926 wait complete. 2023-11-24T08:08:45.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:08:46.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.065+0000 7f2994261640 1 Processor -- start 2023-11-24T08:08:46.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.065+0000 7f2994261640 1 -- start start 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.066+0000 7f2994261640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f298c0d1c20 0x7f298c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2994261640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c1605f0 0x7f298c160a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2994261640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c0ce410 con 0x7f298c0d2000 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2994261640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298c160f80 con 0x7f298c0d1c20 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2994261640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0d06f0 con 0x7f298c1605f0 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f298c0d1c20 0x7f298c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f298c0d1c20 0x7f298c06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55392/0 (socket says 172.21.15.107:55392) 2023-11-24T08:08:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 -- 172.21.15.107:0/120174785 learned_addr learned my addr 172.21.15.107:0/120174785 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f29927fe640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c1605f0 0x7f298c160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 -- 172.21.15.107:0/120174785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c1605f0 msgr2=0x7f298c160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c1605f0 0x7f298c160a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 -- 172.21.15.107:0/120174785 >> v1:172.21.15.120:6789/0 conn(0x7f298c0d2000 legacy=0x7f298c0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.067+0000 7f2991ffd640 1 -- 172.21.15.107:0/120174785 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c0d09d0 con 0x7f298c0d1c20 2023-11-24T08:08:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.068+0000 7f2991ffd640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c0d1c20 0x7f298c06bb60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3e38933ef0eb5f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.069+0000 7f2990ffb640 1 -- 172.21.15.107:0/120174785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2988031640 con 0x7f298c0d1c20 2023-11-24T08:08:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.069+0000 7f2990ffb640 1 -- 172.21.15.107:0/120174785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29880317a0 con 0x7f298c0d1c20 2023-11-24T08:08:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.069+0000 7f2990ffb640 1 -- 172.21.15.107:0/120174785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2988031b30 con 0x7f298c0d1c20 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.069+0000 7f2994261640 1 -- 172.21.15.107:0/120174785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c0d1c20 msgr2=0x7f298c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.069+0000 7f2994261640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c0d1c20 0x7f298c06bb60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 -- 172.21.15.107:0/120174785 shutdown_connections 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c1605f0 0x7f298c160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 --2- 172.21.15.107:0/120174785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c0d1c20 0x7f298c06bb60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 -- 172.21.15.107:0/120174785 >> 172.21.15.107:0/120174785 conn(0x7f298c08cea0 msgr2=0x7f298c09c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 -- 172.21.15.107:0/120174785 shutdown_connections 2023-11-24T08:08:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.070+0000 7f2994261640 1 -- 172.21.15.107:0/120174785 wait complete. 2023-11-24T08:08:46.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.071+0000 7f2994261640 1 Processor -- start 2023-11-24T08:08:46.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 -- start start 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 0x7f298c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 0x7f298c1f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c06fb40 con 0x7f298c1605f0 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c06fe20 con 0x7f298c0d2000 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.072+0000 7f2994261640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c070100 con 0x7f298c0d1c20 2023-11-24T08:08:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55396/0 (socket says 172.21.15.107:55396) 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 -- 172.21.15.107:0/2459978863 learned_addr learned my addr 172.21.15.107:0/2459978863 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f2991ffd640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 0x7f298c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 -- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 msgr2=0x7f298c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29917fc640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 0x7f298c1f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 0x7f298c070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 -- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 msgr2=0x7f298c1f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 0x7f298c1f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29927fe640 1 -- 172.21.15.107:0/2459978863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29880094d0 con 0x7f298c1605f0 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f2991ffd640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 0x7f298c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.073+0000 7f29917fc640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 0x7f298c1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f29927fe640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f298400cd70 tx=0x7f298400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984003b90 con 0x7f298c1605f0 2023-11-24T08:08:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2984003cf0 con 0x7f298c1605f0 2023-11-24T08:08:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984004240 con 0x7f298c1605f0 2023-11-24T08:08:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c0703e0 con 0x7f298c1605f0 2023-11-24T08:08:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.074+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c071120 con 0x7f298c1605f0 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.075+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c070900 con 0x7f298c1605f0 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.076+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f29840043a0 con 0x7f298c1605f0 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.077+0000 7f29827fc640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 0x7f297006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.077+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2984012070 con 0x7f298c1605f0 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.077+0000 7f2991ffd640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 0x7f297006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.078+0000 7f2991ffd640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 0x7f297006d8a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2988036000 tx=0x7f298803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.081+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f298c070900 con 0x7f298c1605f0 2023-11-24T08:08:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.270+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f298c070900 con 0x7f298c1605f0 2023-11-24T08:08:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.271+0000 7f29827fc640 1 -- 172.21.15.107:0/2459978863 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f298c070900 con 0x7f298c1605f0 2023-11-24T08:08:46.273 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:08:46.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.273+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 msgr2=0x7f297006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.273+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 0x7f297006d8a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2988036000 tx=0x7f298803c040 comp rx=0 tx=0).stop 2023-11-24T08:08:46.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.273+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 msgr2=0x7f298c0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.273+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f298400cd70 tx=0x7f298400a660 comp rx=0 tx=0).stop 2023-11-24T08:08:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 shutdown_connections 2023-11-24T08:08:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f297006b3e0 0x7f297006d8a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f298c1605f0 0x7f298c0b3240 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f298c0d2000 0x7f298c1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 --2- 172.21.15.107:0/2459978863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f298c0d1c20 0x7f298c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 >> 172.21.15.107:0/2459978863 conn(0x7f298c08cea0 msgr2=0x7f298c0c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:46.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 shutdown_connections 2023-11-24T08:08:46.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.274+0000 7f2994261640 1 -- 172.21.15.107:0/2459978863 wait complete. 2023-11-24T08:08:46.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:08:46.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.500+0000 7fc5b9d78640 1 Processor -- start 2023-11-24T08:08:46.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 -- start start 2023-11-24T08:08:46.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 0x7fc5b415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5b40c49f0 0x7fc5b406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b415ec90 con 0x7fc5b406dc40 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b406e0f0 con 0x7fc5b40c49f0 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.502+0000 7fc5b9d78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40cd530 con 0x7fc5b4072160 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5b40c49f0 0x7fc5b406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5b40c49f0 0x7fc5b406d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55402/0 (socket says 172.21.15.107:55402) 2023-11-24T08:08:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 -- 172.21.15.107:0/746879199 learned_addr learned my addr 172.21.15.107:0/746879199 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b37fe640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 0x7fc5b415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 -- 172.21.15.107:0/746879199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 msgr2=0x7fc5b415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 0x7fc5b415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 -- 172.21.15.107:0/746879199 >> v1:172.21.15.107:6789/0 conn(0x7fc5b406dc40 legacy=0x7fc5b40cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b2ffd640 1 -- 172.21.15.107:0/746879199 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b40c4ed0 con 0x7fc5b40c49f0 2023-11-24T08:08:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.503+0000 7fc5b37fe640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 0x7fc5b415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.504+0000 7fc5b2ffd640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b40c49f0 0x7fc5b406d700 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009870 tx=0x7fc59c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccca9412dc8f4d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b27fc640 1 -- 172.21.15.107:0/746879199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c030840 con 0x7fc5b40c49f0 2023-11-24T08:08:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b27fc640 1 -- 172.21.15.107:0/746879199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59c0309a0 con 0x7fc5b40c49f0 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b27fc640 1 -- 172.21.15.107:0/746879199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c030d30 con 0x7fc5b40c49f0 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 -- 172.21.15.107:0/746879199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b40c49f0 msgr2=0x7fc5b406d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b40c49f0 0x7fc5b406d700 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009870 tx=0x7fc59c02b980 comp rx=0 tx=0).stop 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 -- 172.21.15.107:0/746879199 shutdown_connections 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b40c49f0 0x7fc5b406d700 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/746879199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b4072160 0x7fc5b415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 -- 172.21.15.107:0/746879199 >> 172.21.15.107:0/746879199 conn(0x7fc5b40b0c50 msgr2=0x7fc5b40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.505+0000 7fc5b9d78640 1 -- 172.21.15.107:0/746879199 shutdown_connections 2023-11-24T08:08:46.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.506+0000 7fc5b9d78640 1 -- 172.21.15.107:0/746879199 wait complete. 2023-11-24T08:08:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.507+0000 7fc5b9d78640 1 Processor -- start 2023-11-24T08:08:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.508+0000 7fc5b9d78640 1 -- start start 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.508+0000 7fc5b9d78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.508+0000 7fc5b9d78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b4072160 0x7fc5b4071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b9d78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b40c49f0 0x7fc5b414e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b9d78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b414fbc0 con 0x7fc5b406dc40 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b9d78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40b3300 con 0x7fc5b4072160 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b9d78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b414f600 con 0x7fc5b40c49f0 2023-11-24T08:08:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55414/0 (socket says 172.21.15.107:55414) 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 -- 172.21.15.107:0/1051203556 learned_addr learned my addr 172.21.15.107:0/1051203556 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 -- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b40c49f0 msgr2=0x7fc5b414e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b40c49f0 0x7fc5b414e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 -- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b4072160 msgr2=0x7fc5b4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b4072160 0x7fc5b4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 -- 172.21.15.107:0/1051203556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c0094d0 con 0x7fc5b406dc40 2023-11-24T08:08:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.509+0000 7fc5b37fe640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400cf60 tx=0x7fc5a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.510+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400d840 con 0x7fc5b406dc40 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.510+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a400d9a0 con 0x7fc5b406dc40 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.510+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b414f8e0 con 0x7fc5b406dc40 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.510+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400dd30 con 0x7fc5b406dc40 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.510+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b40b3880 con 0x7fc5b406dc40 2023-11-24T08:08:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.511+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b40b3a70 con 0x7fc5b406dc40 2023-11-24T08:08:46.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.512+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fc5a4015440 con 0x7fc5b406dc40 2023-11-24T08:08:46.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.513+0000 7fc5abfff640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 0x7fc59406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.513+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc5a408aa70 con 0x7fc5b406dc40 2023-11-24T08:08:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.513+0000 7fc5b2ffd640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 0x7fc59406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.514+0000 7fc5b2ffd640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 0x7fc59406d8a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc59c036000 tx=0x7fc59c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.517+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5a4055300 con 0x7fc5b406dc40 2023-11-24T08:08:46.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.710+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5b404e8b0 con 0x7fc5b406dc40 2023-11-24T08:08:46.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.711+0000 7fc5abfff640 1 -- 172.21.15.107:0/1051203556 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5a401a0a0 con 0x7fc5b406dc40 2023-11-24T08:08:46.712 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 msgr2=0x7fc59406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 0x7fc59406d8a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc59c036000 tx=0x7fc59c03b040 comp rx=0 tx=0).stop 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 msgr2=0x7fc5b4070d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400cf60 tx=0x7fc5a400cf90 comp rx=0 tx=0).stop 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 shutdown_connections 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59406b3e0 0x7fc59406d8a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b40c49f0 0x7fc5b414e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b4072160 0x7fc5b4071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 --2- 172.21.15.107:0/1051203556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b406dc40 0x7fc5b4070d50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 >> 172.21.15.107:0/1051203556 conn(0x7fc5b40b0c50 msgr2=0x7fc5b40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:46.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 shutdown_connections 2023-11-24T08:08:46.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.714+0000 7fc5b9d78640 1 -- 172.21.15.107:0/1051203556 wait complete. 2023-11-24T08:08:46.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:08:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.950+0000 7f250587c640 1 Processor -- start 2023-11-24T08:08:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.950+0000 7f250587c640 1 -- start start 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.951+0000 7f250587c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250015c7f0 0x7f250006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.951+0000 7f250587c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25000ce6c0 0x7f25000cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f250587c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000cece0 con 0x7f25000cf450 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f250587c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2500156010 con 0x7f25000ce6c0 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f250587c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250006ce30 con 0x7f250015c7f0 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25000ce6c0 0x7f25000cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25000ce6c0 0x7f25000cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55424/0 (socket says 172.21.15.107:55424) 2023-11-24T08:08:46.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3256532685 learned_addr learned my addr 172.21.15.107:0/3256532685 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24feffd640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250015c7f0 0x7f250006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3256532685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250015c7f0 msgr2=0x7f250006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:46.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250015c7f0 0x7f250006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3256532685 >> v1:172.21.15.107:6789/0 conn(0x7f25000cf450 legacy=0x7f25000cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.952+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3256532685 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250006d040 con 0x7f25000ce6c0 2023-11-24T08:08:46.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.953+0000 7f24ff7fe640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25000ce6c0 0x7f25000cf8d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59008e7e03150c50 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f24fdffb640 1 -- 172.21.15.107:0/3256532685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f802caf0 con 0x7f25000ce6c0 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f24fdffb640 1 -- 172.21.15.107:0/3256532685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f802cc50 con 0x7f25000ce6c0 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f24fdffb640 1 -- 172.21.15.107:0/3256532685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80343f0 con 0x7f25000ce6c0 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 -- 172.21.15.107:0/3256532685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25000ce6c0 msgr2=0x7f25000cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25000ce6c0 0x7f25000cf8d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f802bf50 comp rx=0 tx=0).stop 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 -- 172.21.15.107:0/3256532685 shutdown_connections 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25000ce6c0 0x7f25000cf8d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 --2- 172.21.15.107:0/3256532685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250015c7f0 0x7f250006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 -- 172.21.15.107:0/3256532685 >> 172.21.15.107:0/3256532685 conn(0x7f250006a920 msgr2=0x7f250006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.954+0000 7f250587c640 1 -- 172.21.15.107:0/3256532685 shutdown_connections 2023-11-24T08:08:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.955+0000 7f250587c640 1 -- 172.21.15.107:0/3256532685 wait complete. 2023-11-24T08:08:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.957+0000 7f250587c640 1 Processor -- start 2023-11-24T08:08:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.957+0000 7f250587c640 1 -- start start 2023-11-24T08:08:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25000ce6c0 0x7f25000acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25000cf450 0x7f2500157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000ae160 con 0x7f250015c7f0 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2500157420 con 0x7f25000cf450 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f250587c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25000ad550 con 0x7f25000ce6c0 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f24ff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f24ff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55434/0 (socket says 172.21.15.107:55434) 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3003995463 learned_addr learned my addr 172.21.15.107:0/3003995463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.958+0000 7f24feffd640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25000ce6c0 0x7f25000acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24fe7fc640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25000cf450 0x7f2500157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25000ce6c0 msgr2=0x7f25000acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25000ce6c0 0x7f25000acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25000cf450 msgr2=0x7f2500157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25000cf450 0x7f2500157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 -- 172.21.15.107:0/3003995463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8009b30 con 0x7f250015c7f0 2023-11-24T08:08:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24ff7fe640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8009510 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f803e450 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f803e800 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.959+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25000ad830 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.960+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25000addf0 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.961+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25000ae310 con 0x7f250015c7f0 2023-11-24T08:08:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.962+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f24f803e960 con 0x7f250015c7f0 2023-11-24T08:08:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.963+0000 7f24e7fff640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 0x7f24ec06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.963+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f24f80ace40 con 0x7f250015c7f0 2023-11-24T08:08:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.963+0000 7f24feffd640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 0x7f24ec06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.963+0000 7f24feffd640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 0x7f24ec06d8a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f24f0006fd0 tx=0x7f24f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:46.967+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25000ae310 con 0x7f250015c7f0 2023-11-24T08:08:47.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.197+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25000ae310 con 0x7f250015c7f0 2023-11-24T08:08:47.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.197+0000 7f24e7fff640 1 -- 172.21.15.107:0/3003995463 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f25000ae310 con 0x7f250015c7f0 2023-11-24T08:08:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:47.199 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:08:47.199 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 msgr2=0x7f24ec06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 0x7f24ec06d8a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f24f0006fd0 tx=0x7f24f0008040 comp rx=0 tx=0).stop 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 msgr2=0x7f25001f2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f8004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 shutdown_connections 2023-11-24T08:08:47.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06b3e0 0x7f24ec06d8a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250015c7f0 0x7f25001f2af0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25000cf450 0x7f2500157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 --2- 172.21.15.107:0/3003995463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25000ce6c0 0x7f25000acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 >> 172.21.15.107:0/3003995463 conn(0x7f250006a920 msgr2=0x7f250015eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 shutdown_connections 2023-11-24T08:08:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.200+0000 7f250587c640 1 -- 172.21.15.107:0/3003995463 wait complete. 2023-11-24T08:08:47.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.431+0000 7ff8531e3640 1 Processor -- start 2023-11-24T08:08:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.431+0000 7ff8531e3640 1 -- start start 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff8531e3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff84c15b080 0x7ff84c15b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff8531e3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c15e3f0 con 0x7ff84c0b2250 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff8531e3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c15ab80 con 0x7ff84c15df80 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff8531e3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c15a990 con 0x7ff84c15b080 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff850f7f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff84c0b2250 0x7ff84c0b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53782/0 (socket says 172.21.15.107:53782) 2023-11-24T08:08:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff850f7f640 1 -- 172.21.15.107:0/1683295127 learned_addr learned my addr 172.21.15.107:0/1683295127 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219944972 0 0) 0x7ff84c15e3f0 con 0x7ff84c0b2250 2023-11-24T08:08:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.433+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c005730 con 0x7ff84c0b2250 2023-11-24T08:08:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff851780640 1 --2- 172.21.15.107:0/1683295127 >> v2:172.21.15.120:3300/0 conn(0x7ff84c15b080 0x7ff84c15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479233876 0 0) 0x7ff83c005730 con 0x7ff84c0b2250 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 >> v2:172.21.15.120:3300/0 conn(0x7ff84c15b080 msgr2=0x7ff84c15b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 --2- 172.21.15.107:0/1683295127 >> v2:172.21.15.120:3300/0 conn(0x7ff84c15b080 0x7ff84c15b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 >> v1:172.21.15.120:6789/0 conn(0x7ff84c15df80 legacy=0x7ff84c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c0b2d40 con 0x7ff84c0b2250 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff840001670 con 0x7ff84c0b2250 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 >> v1:172.21.15.107:6789/0 conn(0x7ff84c0b2250 legacy=0x7ff84c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83c005bd0 0x7ff83c005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 0x7ff83c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0073e0 0x7ff83c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c00e2e0 con 0x7ff83c0069d0 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c00e450 con 0x7ff83c005bd0 2023-11-24T08:08:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c00e610 con 0x7ff83c0073e0 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff84bfff640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 0x7ff83c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.434+0000 7ff851780640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0073e0 0x7ff83c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff850f7f640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83c005bd0 0x7ff83c005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff84bfff640 1 -- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0073e0 msgr2=0x7ff83c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff84bfff640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0073e0 0x7ff83c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff84bfff640 1 -- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83c005bd0 msgr2=0x7ff83c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff84bfff640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83c005bd0 0x7ff83c005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff84bfff640 1 -- 172.21.15.107:0/1683295127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c005730 con 0x7ff83c0069d0 2023-11-24T08:08:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff851780640 1 -- 172.21.15.107:0/1683295127 reap_dead start 2023-11-24T08:08:47.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.435+0000 7ff850f7f640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83c005bd0 0x7ff83c005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.436+0000 7ff84bfff640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 0x7ff83c006e50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff838003d80 tx=0x7ff83c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82581d4b447829ec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.436+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838032600 con 0x7ff83c0069d0 2023-11-24T08:08:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.436+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83802e050 con 0x7ff83c0069d0 2023-11-24T08:08:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.436+0000 7ff84b7fe640 1 -- 172.21.15.107:0/1683295127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380329e0 con 0x7ff83c0069d0 2023-11-24T08:08:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 -- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 msgr2=0x7ff83c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 0x7ff83c006e50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff838003d80 tx=0x7ff83c007a00 comp rx=0 tx=0).stop 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 -- 172.21.15.107:0/1683295127 shutdown_connections 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 --2- 172.21.15.107:0/1683295127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff83c0069d0 0x7ff83c006e50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 -- 172.21.15.107:0/1683295127 >> 172.21.15.107:0/1683295127 conn(0x7ff84c06a920 msgr2=0x7ff84c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 -- 172.21.15.107:0/1683295127 shutdown_connections 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.437+0000 7ff8531e3640 1 -- 172.21.15.107:0/1683295127 wait complete. 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.439+0000 7ff8531e3640 1 Processor -- start 2023-11-24T08:08:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.439+0000 7ff8531e3640 1 -- start start 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0069d0 0x7ff84c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff84c0adf30 0x7ff84c1539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c150e20 con 0x7ff84c1509c0 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c1561a0 con 0x7ff84c0adf30 2023-11-24T08:08:47.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff8531e3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c156420 con 0x7ff83c0069d0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55452/0 (socket says 172.21.15.107:55452) 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 -- 172.21.15.107:0/542309302 learned_addr learned my addr 172.21.15.107:0/542309302 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff850f7f640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0069d0 0x7ff84c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 -- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0069d0 msgr2=0x7ff84c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0069d0 0x7ff84c0accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff851780640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff84c0adf30 0x7ff84c1539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 -- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff84c0adf30 msgr2=0x7ff84c1539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff84c0adf30 0x7ff84c1539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.440+0000 7ff84bfff640 1 -- 172.21.15.107:0/542309302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c0ad450 con 0x7ff84c1509c0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff84bfff640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff838035040 tx=0x7ff8380036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83803c410 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83803c570 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c0ad730 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83803c8e0 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.441+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c156700 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.442+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c150e20 con 0x7ff84c1509c0 2023-11-24T08:08:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.443+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7ff838039050 con 0x7ff84c1509c0 2023-11-24T08:08:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.444+0000 7ff8497fa640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 0x7ff83006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.444+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8380ab600 con 0x7ff84c1509c0 2023-11-24T08:08:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.445+0000 7ff850f7f640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 0x7ff83006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.445+0000 7ff850f7f640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 0x7ff83006d8a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0681f0 tx=0x7ff840009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.448+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff84c150e20 con 0x7ff84c1509c0 2023-11-24T08:08:47.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.630+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff84c150e20 con 0x7ff84c1509c0 2023-11-24T08:08:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.633+0000 7ff8497fa640 1 -- 172.21.15.107:0/542309302 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16670 (secure 0 0 0) 0x7ff84c150e20 con 0x7ff84c1509c0 2023-11-24T08:08:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:47.635 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:44.788580+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 msgr2=0x7ff83006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 0x7ff83006d8a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0681f0 tx=0x7ff840009040 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 msgr2=0x7ff84c155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff838035040 tx=0x7ff8380036f0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 shutdown_connections 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff83006b3e0 0x7ff83006d8a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff84c0adf30 0x7ff84c1539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff84c1509c0 0x7ff84c155c60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 --2- 172.21.15.107:0/542309302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff83c0069d0 0x7ff84c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 >> 172.21.15.107:0/542309302 conn(0x7ff84c06a920 msgr2=0x7ff84c159660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 shutdown_connections 2023-11-24T08:08:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.636+0000 7ff8531e3640 1 -- 172.21.15.107:0/542309302 wait complete. 2023-11-24T08:08:47.656 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:08:47.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:08:47.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.858+0000 7fd29d1c9640 1 Processor -- start 2023-11-24T08:08:47.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.858+0000 7fd29d1c9640 1 -- start start 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd29d1c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd29815df80 0x7fd2980ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd29d1c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd29815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd29d1c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29815e3f0 con 0x7fd2980b2250 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd29d1c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29815a990 con 0x7fd29815df80 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd29d1c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29815ab80 con 0x7fd2980cd470 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd296d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd2980b2250 0x7fd2980b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53792/0 (socket says 172.21.15.107:53792) 2023-11-24T08:08:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.860+0000 7fd296d9d640 1 -- 172.21.15.107:0/1491217946 learned_addr learned my addr 172.21.15.107:0/1491217946 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:47.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd28ffff640 1 --2- 172.21.15.107:0/1491217946 >> v2:172.21.15.120:3300/0 conn(0x7fd29815df80 0x7fd2980ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29759e640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd29815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110037306 0 0) 0x7fd29815e3f0 con 0x7fd2980b2250 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284003520 con 0x7fd2980b2250 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488056841 0 0) 0x7fd284003520 con 0x7fd2980b2250 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 msgr2=0x7fd29815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd29815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 >> v2:172.21.15.120:3300/0 conn(0x7fd29815df80 msgr2=0x7fd2980ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 --2- 172.21.15.107:0/1491217946 >> v2:172.21.15.120:3300/0 conn(0x7fd29815df80 0x7fd2980ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980b2d40 con 0x7fd2980b2250 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd288001670 con 0x7fd2980b2250 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 >> v1:172.21.15.107:6789/0 conn(0x7fd2980b2250 legacy=0x7fd2980b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd284003970 0x7fd284003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2840047c0 0x7fd284004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd284005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29815ab80 con 0x7fd284003970 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28400b980 con 0x7fd2840047c0 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.861+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28400bac0 con 0x7fd2980cd470 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd284003970 0x7fd284003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd29759e640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd284005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd28ffff640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2840047c0 0x7fd284004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 -- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 msgr2=0x7fd284005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2980cd470 0x7fd284005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 -- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2840047c0 msgr2=0x7fd284004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2840047c0 0x7fd284004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.862+0000 7fd296d9d640 1 -- 172.21.15.107:0/1491217946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284003520 con 0x7fd284003970 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.863+0000 7fd296d9d640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd284003970 0x7fd284003d90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd28800c210 tx=0x7fd2840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7549309adb091e5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.863+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288005ad0 con 0x7fd284003970 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.863+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd288005c50 con 0x7fd284003970 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.863+0000 7fd29659c640 1 -- 172.21.15.107:0/1491217946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28802c8d0 con 0x7fd284003970 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 -- 172.21.15.107:0/1491217946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd284003970 msgr2=0x7fd284003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/1491217946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd284003970 0x7fd284003d90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd28800c210 tx=0x7fd2840057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29759e640 1 -- 172.21.15.107:0/1491217946 reap_dead start 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 -- 172.21.15.107:0/1491217946 shutdown_connections 2023-11-24T08:08:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 -- 172.21.15.107:0/1491217946 >> 172.21.15.107:0/1491217946 conn(0x7fd29806a920 msgr2=0x7fd29806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:47.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 -- 172.21.15.107:0/1491217946 shutdown_connections 2023-11-24T08:08:47.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.864+0000 7fd29d1c9640 1 -- 172.21.15.107:0/1491217946 wait complete. 2023-11-24T08:08:47.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.866+0000 7fd29d1c9640 1 Processor -- start 2023-11-24T08:08:47.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.866+0000 7fd29d1c9640 1 -- start start 2023-11-24T08:08:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 0x7fd298143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2981432c0 0x7fd298145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd298144260 con 0x7fd298147040 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd298144540 con 0x7fd284003970 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29d1c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd298144820 con 0x7fd2981432c0 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55466/0 (socket says 172.21.15.107:55466) 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.867+0000 7fd29759e640 1 -- 172.21.15.107:0/3757876833 learned_addr learned my addr 172.21.15.107:0/3757876833 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd296d9d640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 0x7fd298143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd28ffff640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2981432c0 0x7fd298145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 -- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2981432c0 msgr2=0x7fd298145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2981432c0 0x7fd298145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 -- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 msgr2=0x7fd298143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 0x7fd298143b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 -- 172.21.15.107:0/3757876833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298144b00 con 0x7fd298147040 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd296d9d640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 0x7fd298143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29759e640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd298158560 tx=0x7fd290009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290007c70 con 0x7fd298147040 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd290007dd0 con 0x7fd298147040 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd298144de0 con 0x7fd298147040 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290014640 con 0x7fd298147040 2023-11-24T08:08:47.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.868+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd298143720 con 0x7fd298147040 2023-11-24T08:08:47.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.869+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd298144fd0 con 0x7fd298147040 2023-11-24T08:08:47.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.871+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fd29001d3b0 con 0x7fd298147040 2023-11-24T08:08:47.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.871+0000 7fd28f7fe640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 0x7fd27806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:47.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.872+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd290089da0 con 0x7fd298147040 2023-11-24T08:08:47.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.872+0000 7fd296d9d640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 0x7fd27806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:47.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.872+0000 7fd296d9d640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 0x7fd27806d8d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd288034000 tx=0x7fd288007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:47.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:47.876+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2900544f0 con 0x7fd298147040 2023-11-24T08:08:48.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.105+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd298067470 con 0x7fd298147040 2023-11-24T08:08:48.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.106+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3757876833 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fd290057b10 con 0x7fd298147040 2023-11-24T08:08:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:08:48.108 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 msgr2=0x7fd27806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 0x7fd27806d8d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd288034000 tx=0x7fd288007000 comp rx=0 tx=0).stop 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 msgr2=0x7fd2981474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd298158560 tx=0x7fd290009cd0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 shutdown_connections 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27806b410 0x7fd27806d8d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd298147040 0x7fd2981474f0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2981432c0 0x7fd298145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 --2- 172.21.15.107:0/3757876833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284003970 0x7fd298143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 >> 172.21.15.107:0/3757876833 conn(0x7fd29806a920 msgr2=0x7fd2980734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 shutdown_connections 2023-11-24T08:08:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.109+0000 7fd29d1c9640 1 -- 172.21.15.107:0/3757876833 wait complete. 2023-11-24T08:08:48.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:08:48.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.339+0000 7f726b505640 1 Processor -- start 2023-11-24T08:08:48.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.340+0000 7f726b505640 1 -- start start 2023-11-24T08:08:48.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f726b505640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f726406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f726b505640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640cece0 con 0x7f72640cf450 2023-11-24T08:08:48.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f726b505640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726406d040 con 0x7f72640ce6c0 2023-11-24T08:08:48.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f726b505640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7264156010 con 0x7f726415c7f0 2023-11-24T08:08:48.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f7268aa0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72640cf450 0x7f72640cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56182/0 (socket says 172.21.15.107:56182) 2023-11-24T08:08:48.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.341+0000 7f7268aa0640 1 -- 172.21.15.107:0/3242774259 learned_addr learned my addr 172.21.15.107:0/3242774259 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303474499 0 0) 0x7f72640cece0 con 0x7f72640cf450 2023-11-24T08:08:48.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f72692a1640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f726406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7250003520 con 0x7f72640cf450 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946520335 0 0) 0x7f7250003520 con 0x7f72640cf450 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 msgr2=0x7f726406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f726406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 >> v1:172.21.15.120:6789/0 conn(0x7f72640ce6c0 legacy=0x7f72640714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726406d320 con 0x7f72640cf450 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7254001670 con 0x7f72640cf450 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 >> v1:172.21.15.107:6789/0 conn(0x7f72640cf450 legacy=0x7f72640cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500039a0 0x7f7250003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72500047c0 0x7f7250004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f7250005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264156010 con 0x7f72500047c0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725000bb10 con 0x7f72500039a0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.342+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725000bcd0 con 0x7f726415c7f0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72500047c0 0x7f7250004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f72692a1640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500039a0 0x7f7250003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7269aa2640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f7250005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 -- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 msgr2=0x7f7250005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f7250005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 -- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500039a0 msgr2=0x7f7250003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500039a0 0x7f7250003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7268aa0640 1 -- 172.21.15.107:0/3242774259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7250003520 con 0x7f72500047c0 2023-11-24T08:08:48.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f72692a1640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500039a0 0x7f7250003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:48.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.343+0000 7f7269aa2640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415c7f0 0x7f7250005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.344+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72500047c0 0x7f7250004c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f725400c0e0 tx=0x7f72500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42c4f424161bd52 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.344+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254005ad0 con 0x7f72500047c0 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.344+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254005c30 con 0x7f72500047c0 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.344+0000 7f725bfff640 1 -- 172.21.15.107:0/3242774259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725402c7f0 con 0x7f72500047c0 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 -- 172.21.15.107:0/3242774259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72500047c0 msgr2=0x7f7250004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 --2- 172.21.15.107:0/3242774259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72500047c0 0x7f7250004c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f725400c0e0 tx=0x7f72500057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f7269aa2640 1 -- 172.21.15.107:0/3242774259 reap_dead start 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 -- 172.21.15.107:0/3242774259 shutdown_connections 2023-11-24T08:08:48.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 -- 172.21.15.107:0/3242774259 >> 172.21.15.107:0/3242774259 conn(0x7f726406a920 msgr2=0x7f726406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:48.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 -- 172.21.15.107:0/3242774259 shutdown_connections 2023-11-24T08:08:48.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.345+0000 7f726b505640 1 -- 172.21.15.107:0/3242774259 wait complete. 2023-11-24T08:08:48.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.347+0000 7f726b505640 1 Processor -- start 2023-11-24T08:08:48.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.347+0000 7f726b505640 1 -- start start 2023-11-24T08:08:48.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500047c0 0x7f7264149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 0x7f72640aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726414a340 con 0x7f7264154320 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264154d10 con 0x7f72500047c0 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.348+0000 7f726b505640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72640b1240 con 0x7f726415b370 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44860/0 (socket says 172.21.15.107:44860) 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 -- 172.21.15.107:0/3195923556 learned_addr learned my addr 172.21.15.107:0/3195923556 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f72692a1640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500047c0 0x7f7264149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 0x7f72640aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 -- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 msgr2=0x7f72640aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 0x7f72640aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 -- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500047c0 msgr2=0x7f7264149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500047c0 0x7f7264149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 -- 172.21.15.107:0/3195923556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72640b3040 con 0x7f7264154320 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7268aa0640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 0x7f72640aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.349+0000 7f7269aa2640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f72640d0130 tx=0x7f725c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.350+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c003970 con 0x7f7264154320 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.350+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c003af0 con 0x7f7264154320 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.350+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c005070 con 0x7f7264154320 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.350+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72640b2430 con 0x7f7264154320 2023-11-24T08:08:48.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.350+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72640b29f0 con 0x7f7264154320 2023-11-24T08:08:48.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.351+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72640b31d0 con 0x7f7264154320 2023-11-24T08:08:48.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.352+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f725c0051d0 con 0x7f7264154320 2023-11-24T08:08:48.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.353+0000 7f7259ffb640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 0x7f724806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.353+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f725c089f00 con 0x7f7264154320 2023-11-24T08:08:48.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.353+0000 7f72692a1640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 0x7f724806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.354+0000 7f72692a1640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 0x7f724806d8d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7260006fd0 tx=0x7f7260008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.358+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f725c054540 con 0x7f7264154320 2023-11-24T08:08:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.584+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7264067470 con 0x7f7264154320 2023-11-24T08:08:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.586+0000 7f7259ffb640 1 -- 172.21.15.107:0/3195923556 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f725c057b60 con 0x7f7264154320 2023-11-24T08:08:48.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:48.587 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:07:40.945269+0000","modified":"2023-11-24T08:07:43.960687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5661":{"gid":5661,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/1158168362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1158168362},{"type":"v1","addr":"172.21.15.107:6836","nonce":1158168362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-24T08:08:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2023-11-24T08:08:48.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 msgr2=0x7f724806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 0x7f724806d8d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7260006fd0 tx=0x7f7260008040 comp rx=0 tx=0).stop 2023-11-24T08:08:48.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 msgr2=0x7f72641547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f72640d0130 tx=0x7f725c00d9d0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 shutdown_connections 2023-11-24T08:08:48.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f724806b410 0x7f724806d8d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7264154320 0x7f72641547d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f726415b370 0x7f72640aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 --2- 172.21.15.107:0/3195923556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72500047c0 0x7f7264149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 >> 172.21.15.107:0/3195923556 conn(0x7f726406a920 msgr2=0x7f726406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.588+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 shutdown_connections 2023-11-24T08:08:48.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.589+0000 7f726b505640 1 -- 172.21.15.107:0/3195923556 wait complete. 2023-11-24T08:08:48.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.819+0000 7f426562d640 1 Processor -- start 2023-11-24T08:08:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.819+0000 7f426562d640 1 -- start start 2023-11-24T08:08:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f426562d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42600b2250 0x7f426015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f426562d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 0x7f42600ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f426562d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426015ab80 con 0x7f42600b2aa0 2023-11-24T08:08:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f426562d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426015a990 con 0x7f42600b2250 2023-11-24T08:08:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f426562d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426015e680 con 0x7f42600b2630 2023-11-24T08:08:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f425effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42600b2250 0x7f426015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f425f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f42600b2aa0 0x7f426015b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56186/0 (socket says 172.21.15.107:56186) 2023-11-24T08:08:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.821+0000 7f425f7fe640 1 -- 172.21.15.107:0/3578488590 learned_addr learned my addr 172.21.15.107:0/3578488590 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f42567fc640 1 --2- 172.21.15.107:0/3578488590 >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 0x7f42600ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f425effd640 1 -- 172.21.15.107:0/3578488590 >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 msgr2=0x7f42600ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f425effd640 1 --2- 172.21.15.107:0/3578488590 >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 0x7f42600ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f425effd640 1 -- 172.21.15.107:0/3578488590 >> v1:172.21.15.107:6789/0 conn(0x7f42600b2aa0 legacy=0x7f426015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f425effd640 1 -- 172.21.15.107:0/3578488590 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426015e960 con 0x7f42600b2250 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f425e7fc640 1 -- 172.21.15.107:0/3578488590 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784011754 0 0) 0x7f426015ab80 con 0x7f42600b2aa0 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.822+0000 7f42567fc640 1 --2- 172.21.15.107:0/3578488590 >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 0x7f42600ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.823+0000 7f425effd640 1 --2- 172.21.15.107:0/3578488590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f426015df80 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f42600ae5c0 tx=0x7f425002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7830a06fa3f6a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.823+0000 7f425e7fc640 1 -- 172.21.15.107:0/3578488590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250031830 con 0x7f42600b2250 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.823+0000 7f425e7fc640 1 -- 172.21.15.107:0/3578488590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250031990 con 0x7f42600b2250 2023-11-24T08:08:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.823+0000 7f425e7fc640 1 -- 172.21.15.107:0/3578488590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250031d20 con 0x7f42600b2250 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 -- 172.21.15.107:0/3578488590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 msgr2=0x7f426015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 --2- 172.21.15.107:0/3578488590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f426015df80 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f42600ae5c0 tx=0x7f425002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 -- 172.21.15.107:0/3578488590 shutdown_connections 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 --2- 172.21.15.107:0/3578488590 >> v2:172.21.15.120:3300/0 conn(0x7f42600b2630 0x7f42600ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 --2- 172.21.15.107:0/3578488590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f426015df80 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 -- 172.21.15.107:0/3578488590 >> 172.21.15.107:0/3578488590 conn(0x7f426006a920 msgr2=0x7f426006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 -- 172.21.15.107:0/3578488590 shutdown_connections 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.824+0000 7f426562d640 1 -- 172.21.15.107:0/3578488590 wait complete. 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.826+0000 7f426562d640 1 Processor -- start 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.826+0000 7f426562d640 1 -- start start 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42600b2630 0x7f42600b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 0x7f42600b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260071070 con 0x7f42600b2250 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260071350 con 0x7f42600b2aa0 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f426562d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4260071630 con 0x7f42600b2630 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f425effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f425effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44868/0 (socket says 172.21.15.107:44868) 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.827+0000 7f425effd640 1 -- 172.21.15.107:0/1565435928 learned_addr learned my addr 172.21.15.107:0/1565435928 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 -- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42600b2630 msgr2=0x7f42600b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42600b2630 0x7f42600b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 -- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 msgr2=0x7f42600b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425f7fe640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 0x7f42600b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 0x7f42600b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 -- 172.21.15.107:0/1565435928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009400 con 0x7f42600b2250 2023-11-24T08:08:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425f7fe640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 0x7f42600b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f425effd640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4250009890 tx=0x7f42500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425003ba70 con 0x7f42600b2250 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425003bbf0 con 0x7f42600b2250 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250045400 con 0x7f42600b2250 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.828+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260071910 con 0x7f42600b2250 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.829+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42600af090 con 0x7f42600b2250 2023-11-24T08:08:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.830+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260071e30 con 0x7f42600b2250 2023-11-24T08:08:48.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.831+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f425004e440 con 0x7f42600b2250 2023-11-24T08:08:48.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.832+0000 7f4257fff640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 0x7f424006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:48.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.832+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f42500ac550 con 0x7f42600b2250 2023-11-24T08:08:48.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.832+0000 7f42567fc640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 0x7f424006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:48.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.833+0000 7f42567fc640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 0x7f424006d8a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42600b0470 tx=0x7f42480083d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:48.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:48.836+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4250076c20 con 0x7f42600b2250 2023-11-24T08:08:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.030+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4260067470 con 0x7f42600b2250 2023-11-24T08:08:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.033+0000 7f4257fff640 1 -- 172.21.15.107:0/1565435928 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16670 (secure 0 0 0) 0x7f425007a240 con 0x7f42600b2250 2023-11-24T08:08:49.034 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:49.034 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:44.788580+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:07:38.663153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-24T08:07:39.906724+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 msgr2=0x7f424006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 0x7f424006d8a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42600b0470 tx=0x7f42480083d0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 msgr2=0x7f42600aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4250009890 tx=0x7f42500046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 shutdown_connections 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f424006b3e0 0x7f424006d8a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42600b2aa0 0x7f42600b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42600b2630 0x7f42600b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 --2- 172.21.15.107:0/1565435928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42600b2250 0x7f42600aeb50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 >> 172.21.15.107:0/1565435928 conn(0x7f426006a920 msgr2=0x7f42600adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.035+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 shutdown_connections 2023-11-24T08:08:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.036+0000 7f426562d640 1 -- 172.21.15.107:0/1565435928 wait complete. 2023-11-24T08:08:49.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:08:49.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.261+0000 7fc7225a5640 1 Processor -- start 2023-11-24T08:08:49.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.262+0000 7fc7225a5640 1 -- start start 2023-11-24T08:08:49.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc7225a5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc71c06de10 0x7fc71c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc7225a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc71c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc7225a5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c15ec90 con 0x7fc71c072160 2023-11-24T08:08:49.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc7225a5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71c0cd530 con 0x7fc71c06de10 2023-11-24T08:08:49.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc7225a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c0c4ed0 con 0x7fc71c0c49f0 2023-11-24T08:08:49.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc71bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc71c072160 0x7fc71c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56190/0 (socket says 172.21.15.107:56190) 2023-11-24T08:08:49.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.263+0000 7fc71bfff640 1 -- 172.21.15.107:0/286719148 learned_addr learned my addr 172.21.15.107:0/286719148 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820277353 0 0) 0x7fc70c001110 con 0x7fc71c072160 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc708003520 con 0x7fc71c072160 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc713fff640 1 --2- 172.21.15.107:0/286719148 >> v2:172.21.15.120:3300/0 conn(0x7fc71c06de10 0x7fc71c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc720b42640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc71c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3176333124 0 0) 0x7fc708003520 con 0x7fc71c072160 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 msgr2=0x7fc71c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc71c0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 >> v2:172.21.15.120:3300/0 conn(0x7fc71c06de10 msgr2=0x7fc71c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/286719148 >> v2:172.21.15.120:3300/0 conn(0x7fc71c06de10 0x7fc71c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c0c51b0 con 0x7fc71c072160 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc70c001800 con 0x7fc71c072160 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.264+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 >> v1:172.21.15.107:6789/0 conn(0x7fc71c072160 legacy=0x7fc71c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc708003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc708003ff0 0x7fc7080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc708005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c0cd530 con 0x7fc708004d00 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c0c4ed0 con 0x7fc708003ff0 2023-11-24T08:08:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70800bae0 con 0x7fc71c0c49f0 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc708005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc71bfff640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc708003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc713fff640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc708003ff0 0x7fc7080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 -- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 msgr2=0x7fc708003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c0c49f0 0x7fc708003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 -- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc708003ff0 msgr2=0x7fc7080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc708003ff0 0x7fc7080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.265+0000 7fc720b42640 1 -- 172.21.15.107:0/286719148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708003520 con 0x7fc708004d00 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc720b42640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc708005180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc71c06ead0 tx=0x7fc7080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74239af24a96a0f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71402ee80 con 0x7fc708004d00 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc714036440 con 0x7fc708004d00 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc71b7fe640 1 -- 172.21.15.107:0/286719148 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7140367d0 con 0x7fc708004d00 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc7225a5640 1 -- 172.21.15.107:0/286719148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 msgr2=0x7fc708005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.267+0000 7fc7225a5640 1 --2- 172.21.15.107:0/286719148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc708005180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc71c06ead0 tx=0x7fc7080057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.268+0000 7fc720b42640 1 -- 172.21.15.107:0/286719148 reap_dead start 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.268+0000 7fc7225a5640 1 -- 172.21.15.107:0/286719148 shutdown_connections 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.268+0000 7fc7225a5640 1 -- 172.21.15.107:0/286719148 >> 172.21.15.107:0/286719148 conn(0x7fc71c0b0c50 msgr2=0x7fc71c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.268+0000 7fc7225a5640 1 -- 172.21.15.107:0/286719148 shutdown_connections 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.268+0000 7fc7225a5640 1 -- 172.21.15.107:0/286719148 wait complete. 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.270+0000 7fc7225a5640 1 Processor -- start 2023-11-24T08:08:49.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.270+0000 7fc7225a5640 1 -- start start 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c1e4550 0x7fc71c1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc71c15c790 0x7fc71c15cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c158ae0 con 0x7fc708004d00 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c15d150 con 0x7fc71c15c790 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc7225a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1f3340 con 0x7fc71c1e4550 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44890/0 (socket says 172.21.15.107:44890) 2023-11-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.271+0000 7fc71bfff640 1 -- 172.21.15.107:0/962974823 learned_addr learned my addr 172.21.15.107:0/962974823 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc713fff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c1e4550 0x7fc71c1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 -- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c1e4550 msgr2=0x7fc71c1f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc720b42640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc71c15c790 0x7fc71c15cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c1e4550 0x7fc71c1f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 -- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc71c15c790 msgr2=0x7fc71c15cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc71c15c790 0x7fc71c15cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 -- 172.21.15.107:0/962974823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c1e4aa0 con 0x7fc708004d00 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc71bfff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e8c0 tx=0x7fc70c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c00c8d0 con 0x7fc708004d00 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c00ca30 con 0x7fc708004d00 2023-11-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c00cde0 con 0x7fc708004d00 2023-11-24T08:08:49.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c1e4d80 con 0x7fc708004d00 2023-11-24T08:08:49.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.272+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c1e5340 con 0x7fc708004d00 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.273+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c04e8b0 con 0x7fc708004d00 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.274+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fc70c01d460 con 0x7fc708004d00 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.275+0000 7fc7197fa640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 0x7fc6fc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.275+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc70c08a0b0 con 0x7fc708004d00 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.275+0000 7fc713fff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 0x7fc6fc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.276+0000 7fc713fff640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 0x7fc6fc06d8a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc71c15d590 tx=0x7fc704008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.279+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc70c054850 con 0x7fc708004d00 2023-11-24T08:08:49.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.506+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc71c15ec90 con 0x7fc708004d00 2023-11-24T08:08:49.509 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:08:49.508+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:08:49.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.511+0000 7fc7197fa640 1 -- 172.21.15.107:0/962974823 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc71c15ec90 con 0x7fc708004d00 2023-11-24T08:08:49.513 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:08:49.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.515+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 msgr2=0x7fc6fc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 0x7fc6fc06d8a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc71c15d590 tx=0x7fc704008040 comp rx=0 tx=0).stop 2023-11-24T08:08:49.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 msgr2=0x7fc71c158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e8c0 tx=0x7fc70c00ed80 comp rx=0 tx=0).stop 2023-11-24T08:08:49.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 shutdown_connections 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6fc06b3e0 0x7fc6fc06d8a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc71c15c790 0x7fc71c15cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71c1e4550 0x7fc71c1f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 --2- 172.21.15.107:0/962974823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc708004d00 0x7fc71c158c70 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 >> 172.21.15.107:0/962974823 conn(0x7fc71c0b0c50 msgr2=0x7fc71c0cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 shutdown_connections 2023-11-24T08:08:49.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.516+0000 7fc7225a5640 1 -- 172.21.15.107:0/962974823 wait complete. 2023-11-24T08:08:49.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:08:49.595 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:08:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.739+0000 7fcb3e1de640 1 Processor -- start 2023-11-24T08:08:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.739+0000 7fcb3e1de640 1 -- start start 2023-11-24T08:08:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.740+0000 7fcb3e1de640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb3815b440 0x7fcb380cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.740+0000 7fcb3e1de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb380cbf20 0x7fcb380c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.740+0000 7fcb3e1de640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3815eea0 con 0x7fcb380c57e0 2023-11-24T08:08:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.740+0000 7fcb3e1de640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb380cc3d0 con 0x7fcb380cbf20 2023-11-24T08:08:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.740+0000 7fcb3e1de640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3815b820 con 0x7fcb3815b440 2023-11-24T08:08:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb380cbf20 0x7fcb380c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb380cbf20 0x7fcb380c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44918/0 (socket says 172.21.15.107:44918) 2023-11-24T08:08:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 -- 172.21.15.107:0/1710970141 learned_addr learned my addr 172.21.15.107:0/1710970141 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 -- 172.21.15.107:0/1710970141 >> v2:172.21.15.120:3300/0 conn(0x7fcb3815b440 msgr2=0x7fcb380cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 --2- 172.21.15.107:0/1710970141 >> v2:172.21.15.120:3300/0 conn(0x7fcb3815b440 0x7fcb380cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 -- 172.21.15.107:0/1710970141 >> v1:172.21.15.120:6789/0 conn(0x7fcb380c57e0 legacy=0x7fcb380c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.741+0000 7fcb2ffff640 1 -- 172.21.15.107:0/1710970141 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3806d600 con 0x7fcb380cbf20 2023-11-24T08:08:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.742+0000 7fcb2ffff640 1 --2- 172.21.15.107:0/1710970141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb380c52a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fcb3806e770 tx=0x7fcb2002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3147c18d76ca1140 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb36ffd640 1 -- 172.21.15.107:0/1710970141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200317e0 con 0x7fcb380cbf20 2023-11-24T08:08:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb36ffd640 1 -- 172.21.15.107:0/1710970141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20031940 con 0x7fcb380cbf20 2023-11-24T08:08:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb36ffd640 1 -- 172.21.15.107:0/1710970141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20031cd0 con 0x7fcb380cbf20 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 -- 172.21.15.107:0/1710970141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 msgr2=0x7fcb380c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/1710970141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb380c52a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fcb3806e770 tx=0x7fcb2002b880 comp rx=0 tx=0).stop 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 -- 172.21.15.107:0/1710970141 shutdown_connections 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/1710970141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb380c52a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/1710970141 >> v2:172.21.15.120:3300/0 conn(0x7fcb3815b440 0x7fcb380cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.743+0000 7fcb3e1de640 1 -- 172.21.15.107:0/1710970141 >> 172.21.15.107:0/1710970141 conn(0x7fcb380b0c50 msgr2=0x7fcb380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.744+0000 7fcb3e1de640 1 -- 172.21.15.107:0/1710970141 shutdown_connections 2023-11-24T08:08:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.744+0000 7fcb3e1de640 1 -- 172.21.15.107:0/1710970141 wait complete. 2023-11-24T08:08:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.745+0000 7fcb3e1de640 1 Processor -- start 2023-11-24T08:08:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.745+0000 7fcb3e1de640 1 -- start start 2023-11-24T08:08:49.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb380c57e0 0x7fcb380c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 0x7fcb38156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380c8320 con 0x7fcb380cbf20 2023-11-24T08:08:49.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380c8600 con 0x7fcb3815b440 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.746+0000 7fcb3e1de640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380c88e0 con 0x7fcb380c57e0 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44920/0 (socket says 172.21.15.107:44920) 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 -- 172.21.15.107:0/2839631796 learned_addr learned my addr 172.21.15.107:0/2839631796 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 -- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb380c57e0 msgr2=0x7fcb380c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb377fe640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb380c57e0 0x7fcb380c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb37fff640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 0x7fcb38156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb380c57e0 0x7fcb380c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 -- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 msgr2=0x7fcb38156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:49.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 0x7fcb38156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:49.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 -- 172.21.15.107:0/2839631796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200094d0 con 0x7fcb380cbf20 2023-11-24T08:08:49.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb37fff640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 0x7fcb38156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:49.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.747+0000 7fcb2ffff640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000c00 tx=0x7fcb200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.748+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20032880 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.748+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb200329e0 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.748+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3809ac10 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.748+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20032d90 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.748+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb38154a30 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.749+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3804e8b0 con 0x7fcb380cbf20 2023-11-24T08:08:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.750+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fcb2004d3f0 con 0x7fcb380cbf20 2023-11-24T08:08:49.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.751+0000 7fcb34ff9640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 0x7fcb1806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:49.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.751+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcb200ac120 con 0x7fcb380cbf20 2023-11-24T08:08:49.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.751+0000 7fcb377fe640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 0x7fcb1806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:49.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.752+0000 7fcb377fe640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 0x7fcb1806d8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800a770 tx=0x7fcb28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:49.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.755+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb200dee90 con 0x7fcb380cbf20 2023-11-24T08:08:49.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:49.988+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcb3806dc80 con 0x7fcb380cbf20 2023-11-24T08:08:50.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.515+0000 7fcb34ff9640 1 -- 172.21.15.107:0/2839631796 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcb20076780 con 0x7fcb380cbf20 2023-11-24T08:08:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 msgr2=0x7fcb1806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 0x7fcb1806d8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800a770 tx=0x7fcb28008040 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 msgr2=0x7fcb381544f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000c00 tx=0x7fcb200046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 shutdown_connections 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb1806b410 0x7fcb1806d8d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb3815b440 0x7fcb38156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb380cbf20 0x7fcb381544f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 --2- 172.21.15.107:0/2839631796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb380c57e0 0x7fcb380c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 >> 172.21.15.107:0/2839631796 conn(0x7fcb380b0c50 msgr2=0x7fcb380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.518+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 shutdown_connections 2023-11-24T08:08:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.519+0000 7fcb3e1de640 1 -- 172.21.15.107:0/2839631796 wait complete. 2023-11-24T08:08:50.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:08:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.754+0000 7f82ebf61640 1 Processor -- start 2023-11-24T08:08:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.754+0000 7f82ebf61640 1 -- start start 2023-11-24T08:08:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.755+0000 7f82ebf61640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82e415c7f0 0x7f82e406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.755+0000 7f82ebf61640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 0x7f82e40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.755+0000 7f82ebf61640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40cece0 con 0x7f82e40cdfb0 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.755+0000 7f82ebf61640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40cfe40 con 0x7f82e415c7f0 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.755+0000 7f82ebf61640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40ceaf0 con 0x7f82e40cf450 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82e415c7f0 0x7f82e406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82e415c7f0 0x7f82e406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44934/0 (socket says 172.21.15.107:44934) 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 -- 172.21.15.107:0/1691876177 learned_addr learned my addr 172.21.15.107:0/1691876177 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82ea4fe640 1 --1- 172.21.15.107:0/1691876177 >> v1:172.21.15.107:6789/0 conn(0x7f82e40cdfb0 0x7f82e40ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56202/0 (socket says 172.21.15.107:56202) 2023-11-24T08:08:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e94fc640 1 --2- 172.21.15.107:0/1691876177 >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 0x7f82e40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 -- 172.21.15.107:0/1691876177 >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 msgr2=0x7f82e40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 --2- 172.21.15.107:0/1691876177 >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 0x7f82e40cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 -- 172.21.15.107:0/1691876177 >> v1:172.21.15.107:6789/0 conn(0x7f82e40cdfb0 legacy=0x7f82e40ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e9cfd640 1 -- 172.21.15.107:0/1691876177 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e406d040 con 0x7f82e415c7f0 2023-11-24T08:08:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e8cfb640 1 -- 172.21.15.107:0/1691876177 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107047521 0 0) 0x7f82e40cece0 con 0x7f82e40cdfb0 2023-11-24T08:08:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.756+0000 7f82e94fc640 1 --2- 172.21.15.107:0/1691876177 >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 0x7f82e40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.757+0000 7f82e9cfd640 1 --2- 172.21.15.107:0/1691876177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e406db20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f82e4070410 tx=0x7f82e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7326dbb02d9885 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82e8cfb640 1 -- 172.21.15.107:0/1691876177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0030860 con 0x7f82e415c7f0 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82e8cfb640 1 -- 172.21.15.107:0/1691876177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00309c0 con 0x7f82e415c7f0 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82e8cfb640 1 -- 172.21.15.107:0/1691876177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0030d50 con 0x7f82e415c7f0 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 -- 172.21.15.107:0/1691876177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 msgr2=0x7f82e406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 --2- 172.21.15.107:0/1691876177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e406db20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f82e4070410 tx=0x7f82e002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 -- 172.21.15.107:0/1691876177 shutdown_connections 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 --2- 172.21.15.107:0/1691876177 >> v2:172.21.15.120:3300/0 conn(0x7f82e40cf450 0x7f82e40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 --2- 172.21.15.107:0/1691876177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e406db20 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.758+0000 7f82ebf61640 1 -- 172.21.15.107:0/1691876177 >> 172.21.15.107:0/1691876177 conn(0x7f82e406a920 msgr2=0x7f82e406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.759+0000 7f82ebf61640 1 -- 172.21.15.107:0/1691876177 shutdown_connections 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.759+0000 7f82ebf61640 1 -- 172.21.15.107:0/1691876177 wait complete. 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.760+0000 7f82ebf61640 1 Processor -- start 2023-11-24T08:08:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.761+0000 7f82ebf61640 1 -- start start 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.761+0000 7f82ebf61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82e40cdfb0 0x7f82e4150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.761+0000 7f82ebf61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 0x7f82e40adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.761+0000 7f82ebf61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ebf61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4150e40 con 0x7f82e415c7f0 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ebf61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e41524f0 con 0x7f82e40cf450 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ebf61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e41527d0 con 0x7f82e40cdfb0 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44948/0 (socket says 172.21.15.107:44948) 2023-11-24T08:08:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 -- 172.21.15.107:0/2797213705 learned_addr learned my addr 172.21.15.107:0/2797213705 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82e94fc640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 0x7f82e40adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82e9cfd640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82e40cdfb0 0x7f82e4150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 -- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82e40cdfb0 msgr2=0x7f82e4150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82e40cdfb0 0x7f82e4150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 -- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 msgr2=0x7f82e40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 0x7f82e40adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82ea4fe640 1 -- 172.21.15.107:0/2797213705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00094d0 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.762+0000 7f82e94fc640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 0x7f82e40adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82ea4fe640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f82dc002230 tx=0x7f82dc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00ce30 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc00a640 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e4152ab0 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00a9f0 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.763+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e4157200 con 0x7f82e415c7f0 2023-11-24T08:08:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.764+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e4157060 con 0x7f82e415c7f0 2023-11-24T08:08:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.765+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f82dc00ab50 con 0x7f82e415c7f0 2023-11-24T08:08:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.766+0000 7f82da7fc640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 0x7f82c806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.766+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f82dc0898b0 con 0x7f82e415c7f0 2023-11-24T08:08:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.766+0000 7f82e9cfd640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 0x7f82c806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.767+0000 7f82e9cfd640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 0x7f82c806d8a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f82e0036000 tx=0x7f82e003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.770+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82e4157060 con 0x7f82e415c7f0 2023-11-24T08:08:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:50.960+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f82e4157060 con 0x7f82e415c7f0 2023-11-24T08:08:51.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.911+0000 7f82da7fc640 1 -- 172.21.15.107:0/2797213705 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v54) v1 ==== 181+0+0 (secure 0 0 0) 0x7f82e4157060 con 0x7f82e415c7f0 2023-11-24T08:08:51.913 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:08:51.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.914+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 msgr2=0x7f82c806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:51.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.914+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 0x7f82c806d8a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f82e0036000 tx=0x7f82e003b040 comp rx=0 tx=0).stop 2023-11-24T08:08:51.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.914+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 msgr2=0x7f82e4153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:51.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.914+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f82dc002230 tx=0x7f82dc00cbe0 comp rx=0 tx=0).stop 2023-11-24T08:08:51.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 shutdown_connections 2023-11-24T08:08:51.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c806b3e0 0x7f82c806d8a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:51.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e415c7f0 0x7f82e4153e30 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:51.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e40cf450 0x7f82e40adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:51.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 --2- 172.21.15.107:0/2797213705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82e40cdfb0 0x7f82e4150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:51.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 >> 172.21.15.107:0/2797213705 conn(0x7f82e406a920 msgr2=0x7f82e4072120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:51.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 shutdown_connections 2023-11-24T08:08:51.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:51.915+0000 7f82ebf61640 1 -- 172.21.15.107:0/2797213705 wait complete. 2023-11-24T08:08:51.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:08:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.146+0000 7f7ac0ddb640 1 Processor -- start 2023-11-24T08:08:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.147+0000 7f7ac0ddb640 1 -- start start 2023-11-24T08:08:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.148+0000 7f7ac0ddb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ac0ddb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7abc070ab0 0x7f7abc165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ac0ddb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc0b22b0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ac0ddb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc165ba0 con 0x7f7abc070ab0 2023-11-24T08:08:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ac0ddb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc165e80 con 0x7f7abc14acb0 2023-11-24T08:08:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ab2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7abc0b1e50 0x7f7abc0703a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56208/0 (socket says 172.21.15.107:56208) 2023-11-24T08:08:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7ab2ffd640 1 -- 172.21.15.107:0/4227427693 learned_addr learned my addr 172.21.15.107:0/4227427693 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7abbfff640 1 --2- 172.21.15.107:0/4227427693 >> v2:172.21.15.107:3300/0 conn(0x7f7abc070ab0 0x7f7abc165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.149+0000 7f7abb7fe640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abaffd640 1 -- 172.21.15.107:0/4227427693 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434923839 0 0) 0x7f7abc0b22b0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abaffd640 1 -- 172.21.15.107:0/4227427693 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa8003520 con 0x7f7abc0b1e50 2023-11-24T08:08:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abbfff640 1 -- 172.21.15.107:0/4227427693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 msgr2=0x7f7abc0b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abbfff640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abbfff640 1 -- 172.21.15.107:0/4227427693 >> v1:172.21.15.107:6789/0 conn(0x7f7abc0b1e50 legacy=0x7f7abc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.150+0000 7f7abbfff640 1 -- 172.21.15.107:0/4227427693 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc165970 con 0x7f7abc070ab0 2023-11-24T08:08:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.152+0000 7f7abbfff640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc070ab0 0x7f7abc165430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009d90 tx=0x7f7ab402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568148992986e834 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.152+0000 7f7abaffd640 1 -- 172.21.15.107:0/4227427693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab402cc50 con 0x7f7abc070ab0 2023-11-24T08:08:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.152+0000 7f7abaffd640 1 -- 172.21.15.107:0/4227427693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab402cdb0 con 0x7f7abc070ab0 2023-11-24T08:08:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.152+0000 7f7abaffd640 1 -- 172.21.15.107:0/4227427693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4034660 con 0x7f7abc070ab0 2023-11-24T08:08:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4227427693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc070ab0 msgr2=0x7f7abc165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc070ab0 0x7f7abc165430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009d90 tx=0x7f7ab402bea0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4227427693 shutdown_connections 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc070ab0 0x7f7abc165430 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4227427693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4227427693 >> 172.21.15.107:0/4227427693 conn(0x7f7abc0cb8a0 msgr2=0x7f7abc0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4227427693 shutdown_connections 2023-11-24T08:08:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.153+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4227427693 wait complete. 2023-11-24T08:08:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.155+0000 7f7ac0ddb640 1 Processor -- start 2023-11-24T08:08:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.155+0000 7f7ac0ddb640 1 -- start start 2023-11-24T08:08:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 0x7f7abc164190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc1646d0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc1630f0 con 0x7f7abc070ab0 2023-11-24T08:08:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.156+0000 7f7ac0ddb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc1633d0 con 0x7f7abc14acb0 2023-11-24T08:08:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44968/0 (socket says 172.21.15.107:44968) 2023-11-24T08:08:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 -- 172.21.15.107:0/4222173987 learned_addr learned my addr 172.21.15.107:0/4222173987 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7abb7fe640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 0x7f7abc164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 -- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 msgr2=0x7f7abc0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 -- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 msgr2=0x7f7abc164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 0x7f7abc164190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 -- 172.21.15.107:0/4222173987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4009a70 con 0x7f7abc0b1e50 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7abb7fe640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 0x7f7abc164190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.157+0000 7f7ab2ffd640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4001260 tx=0x7f7aa400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.158+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4007480 con 0x7f7abc0b1e50 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.158+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40075e0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.158+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40059c0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.158+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc1636b0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.158+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc163c70 con 0x7f7abc0b1e50 2023-11-24T08:08:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.159+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc163e60 con 0x7f7abc0b1e50 2023-11-24T08:08:52.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.160+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f7aa4005b20 con 0x7f7abc0b1e50 2023-11-24T08:08:52.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.161+0000 7f7ab8ff9640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 0x7f7a9c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:52.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.161+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f7aa4089570 con 0x7f7abc0b1e50 2023-11-24T08:08:52.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.161+0000 7f7abb7fe640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 0x7f7a9c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.161+0000 7f7abb7fe640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 0x7f7a9c06d8a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009a20 tx=0x7f7aac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.165+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7aa4053ed0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.358+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7abc04e840 con 0x7f7abc0b1e50 2023-11-24T08:08:52.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.914+0000 7f7ab8ff9640 1 -- 172.21.15.107:0/4222173987 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v55) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7aa40170d0 con 0x7f7abc0b1e50 2023-11-24T08:08:52.916 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:08:52.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 msgr2=0x7f7a9c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 0x7f7a9c06d8a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009a20 tx=0x7f7aac008040 comp rx=0 tx=0).stop 2023-11-24T08:08:52.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 msgr2=0x7f7abc0aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4001260 tx=0x7f7aa400c960 comp rx=0 tx=0).stop 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 shutdown_connections 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a9c06b3e0 0x7f7a9c06d8a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc14acb0 0x7f7abc0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0b1e50 0x7f7abc0aaee0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 --2- 172.21.15.107:0/4222173987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc070ab0 0x7f7abc164190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 >> 172.21.15.107:0/4222173987 conn(0x7f7abc0cb8a0 msgr2=0x7f7abc070380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 shutdown_connections 2023-11-24T08:08:52.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:52.918+0000 7f7ac0ddb640 1 -- 172.21.15.107:0/4222173987 wait complete. 2023-11-24T08:08:52.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:08:53.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.169+0000 7f2301664640 1 Processor -- start 2023-11-24T08:08:53.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.170+0000 7f2301664640 1 -- start start 2023-11-24T08:08:53.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f2301664640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22fc0c59b0 0x7f22fc166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f2301664640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0c5dc0 con 0x7f22fc15b440 2023-11-24T08:08:53.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f2301664640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc15eea0 con 0x7f22fc0cc0f0 2023-11-24T08:08:53.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f2301664640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc15b820 con 0x7f22fc0c59b0 2023-11-24T08:08:53.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f22fb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22fc0c59b0 0x7f22fc166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f22faffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f22fc15b440 0x7f22fc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56224/0 (socket says 172.21.15.107:56224) 2023-11-24T08:08:53.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.171+0000 7f22faffd640 1 -- 172.21.15.107:0/3553357455 learned_addr learned my addr 172.21.15.107:0/3553357455 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:53.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.172+0000 7f22fb7fe640 1 -- 172.21.15.107:0/3553357455 >> v1:172.21.15.120:6789/0 conn(0x7f22fc0cc0f0 legacy=0x7f22fc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.172+0000 7f22fb7fe640 1 -- 172.21.15.107:0/3553357455 >> v1:172.21.15.107:6789/0 conn(0x7f22fc15b440 legacy=0x7f22fc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.172+0000 7f22fb7fe640 1 -- 172.21.15.107:0/3553357455 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc06d600 con 0x7f22fc0c59b0 2023-11-24T08:08:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.173+0000 7f22fb7fe640 1 --2- 172.21.15.107:0/3553357455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc0c59b0 0x7f22fc166230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba6f1edac118679 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.173+0000 7f22fa7fc640 1 -- 172.21.15.107:0/3553357455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f402cbd0 con 0x7f22fc0c59b0 2023-11-24T08:08:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.173+0000 7f22fa7fc640 1 -- 172.21.15.107:0/3553357455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f402cd30 con 0x7f22fc0c59b0 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.173+0000 7f22fa7fc640 1 -- 172.21.15.107:0/3553357455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f402cea0 con 0x7f22fc0c59b0 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 -- 172.21.15.107:0/3553357455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc0c59b0 msgr2=0x7f22fc166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 --2- 172.21.15.107:0/3553357455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc0c59b0 0x7f22fc166230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f402be20 comp rx=0 tx=0).stop 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 -- 172.21.15.107:0/3553357455 shutdown_connections 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 --2- 172.21.15.107:0/3553357455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc0c59b0 0x7f22fc166230 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 -- 172.21.15.107:0/3553357455 >> 172.21.15.107:0/3553357455 conn(0x7f22fc0b0c50 msgr2=0x7f22fc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:53.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 -- 172.21.15.107:0/3553357455 shutdown_connections 2023-11-24T08:08:53.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.174+0000 7f2301664640 1 -- 172.21.15.107:0/3553357455 wait complete. 2023-11-24T08:08:53.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.175+0000 7f2301664640 1 Processor -- start 2023-11-24T08:08:53.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.176+0000 7f2301664640 1 -- start start 2023-11-24T08:08:53.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.176+0000 7f2301664640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22fc0c59b0 0x7f22fc154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f2301664640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22fc0cc0f0 0x7f22fc156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f2301664640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f2301664640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc155c60 con 0x7f22fc15b440 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f2301664640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc154ee0 con 0x7f22fc0cc0f0 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f2301664640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc1555e0 con 0x7f22fc0c59b0 2023-11-24T08:08:53.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44986/0 (socket says 172.21.15.107:44986) 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 -- 172.21.15.107:0/843064105 learned_addr learned my addr 172.21.15.107:0/843064105 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 -- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22fc0c59b0 msgr2=0x7f22fc154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22fc0c59b0 0x7f22fc154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 -- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22fc0cc0f0 msgr2=0x7f22fc156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22fc0cc0f0 0x7f22fc156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.177+0000 7f22fb7fe640 1 -- 172.21.15.107:0/843064105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f4009a70 con 0x7f22fc15b440 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f22fb7fe640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f402cea0 con 0x7f22fc15b440 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f403e630 con 0x7f22fc15b440 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc0b41d0 con 0x7f22fc15b440 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f403e9e0 con 0x7f22fc15b440 2023-11-24T08:08:53.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.178+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc0b44b0 con 0x7f22fc15b440 2023-11-24T08:08:53.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.179+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc155da0 con 0x7f22fc15b440 2023-11-24T08:08:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.180+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f22f403b050 con 0x7f22fc15b440 2023-11-24T08:08:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.181+0000 7f22f3fff640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 0x7f22dc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.181+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f22f40ac610 con 0x7f22fc15b440 2023-11-24T08:08:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.181+0000 7f22faffd640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 0x7f22dc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.182+0000 7f22faffd640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 0x7f22dc06d8a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f22ec001550 tx=0x7f22ec007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.185+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22fc155da0 con 0x7f22fc15b440 2023-11-24T08:08:53.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.378+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f22fc155da0 con 0x7f22fc15b440 2023-11-24T08:08:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.384+0000 7f22f3fff640 1 -- 172.21.15.107:0/843064105 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7f22fc155da0 con 0x7f22fc15b440 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 msgr2=0x7f22dc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 0x7f22dc06d8a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f22ec001550 tx=0x7f22ec007a40 comp rx=0 tx=0).stop 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 msgr2=0x7f22fc0b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f4004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 shutdown_connections 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22dc06b3e0 0x7f22dc06d8a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22fc15b440 0x7f22fc0b3610 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22fc0cc0f0 0x7f22fc156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 --2- 172.21.15.107:0/843064105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22fc0c59b0 0x7f22fc154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 >> 172.21.15.107:0/843064105 conn(0x7f22fc0b0c50 msgr2=0x7f22fc0cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 shutdown_connections 2023-11-24T08:08:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.387+0000 7f2301664640 1 -- 172.21.15.107:0/843064105 wait complete. 2023-11-24T08:08:53.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:08:53.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.621+0000 7f8b5e327640 1 Processor -- start 2023-11-24T08:08:53.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.622+0000 7f8b5e327640 1 -- start start 2023-11-24T08:08:53.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.623+0000 7f8b5e327640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8b58072710 0x7f8b580ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.623+0000 7f8b5e327640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b580b3090 0x7f8b5815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.623+0000 7f8b5e327640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b580709c0 con 0x7f8b58154980 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.623+0000 7f8b5e327640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b580b0df0 con 0x7f8b58072710 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.623+0000 7f8b5e327640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5815af40 con 0x7f8b580b3090 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8b58072710 0x7f8b580ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8b58072710 0x7f8b580ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44992/0 (socket says 172.21.15.107:44992) 2023-11-24T08:08:53.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3562187954 learned_addr learned my addr 172.21.15.107:0/3562187954 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:53.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5db26640 1 --2- 172.21.15.107:0/3562187954 >> v2:172.21.15.120:3300/0 conn(0x7f8b580b3090 0x7f8b5815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3562187954 >> v2:172.21.15.120:3300/0 conn(0x7f8b580b3090 msgr2=0x7f8b5815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:53.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 --2- 172.21.15.107:0/3562187954 >> v2:172.21.15.120:3300/0 conn(0x7f8b580b3090 0x7f8b5815a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3562187954 >> v1:172.21.15.120:6789/0 conn(0x7f8b58154980 legacy=0x7f8b58072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:53.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.624+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3562187954 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5815b220 con 0x7f8b58072710 2023-11-24T08:08:53.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.625+0000 7f8b5cb24640 1 --2- 172.21.15.107:0/3562187954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b58072710 0x7f8b580ccfa0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8b580ccc60 tx=0x7f8b4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a379f434aa6882 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b4ffff640 1 -- 172.21.15.107:0/3562187954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48031870 con 0x7f8b58072710 2023-11-24T08:08:53.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b4ffff640 1 -- 172.21.15.107:0/3562187954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480319d0 con 0x7f8b58072710 2023-11-24T08:08:53.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b4ffff640 1 -- 172.21.15.107:0/3562187954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48031d60 con 0x7f8b58072710 2023-11-24T08:08:53.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 -- 172.21.15.107:0/3562187954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b58072710 msgr2=0x7f8b580ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3562187954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b58072710 0x7f8b580ccfa0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8b580ccc60 tx=0x7f8b4802b910 comp rx=0 tx=0).stop 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 -- 172.21.15.107:0/3562187954 shutdown_connections 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3562187954 >> v2:172.21.15.120:3300/0 conn(0x7f8b580b3090 0x7f8b5815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3562187954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b58072710 0x7f8b580ccfa0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.626+0000 7f8b5e327640 1 -- 172.21.15.107:0/3562187954 >> 172.21.15.107:0/3562187954 conn(0x7f8b5806a500 msgr2=0x7f8b5806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.627+0000 7f8b5e327640 1 -- 172.21.15.107:0/3562187954 shutdown_connections 2023-11-24T08:08:53.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.627+0000 7f8b5e327640 1 -- 172.21.15.107:0/3562187954 wait complete. 2023-11-24T08:08:53.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.628+0000 7f8b5e327640 1 Processor -- start 2023-11-24T08:08:53.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.628+0000 7f8b5e327640 1 -- start start 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b58072710 0x7f8b58154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b58154980 0x7f8b581f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b581567a0 con 0x7f8b580b3090 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5814dc70 con 0x7f8b58154980 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.629+0000 7f8b5e327640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5814d060 con 0x7f8b58072710 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45006/0 (socket says 172.21.15.107:45006) 2023-11-24T08:08:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3080823499 learned_addr learned my addr 172.21.15.107:0/3080823499 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b58072710 msgr2=0x7f8b58154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b58072710 0x7f8b58154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b58154980 msgr2=0x7f8b581f1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b58154980 0x7f8b581f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.630+0000 7f8b5cb24640 1 -- 172.21.15.107:0/3080823499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480094d0 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b5cb24640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c00 tx=0x7f8b480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48031840 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480443f0 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5814d340 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480447a0 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.631+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5814d900 con 0x7f8b580b3090 2023-11-24T08:08:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.632+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5804e840 con 0x7f8b580b3090 2023-11-24T08:08:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.633+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f8b4804d450 con 0x7f8b580b3090 2023-11-24T08:08:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.634+0000 7f8b4dffb640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 0x7f8b3c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.634+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8b4802d040 con 0x7f8b580b3090 2023-11-24T08:08:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.634+0000 7f8b5d325640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 0x7f8b3c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.635+0000 7f8b5d325640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 0x7f8b3c06d8a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b54006fd0 tx=0x7f8b54006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:53.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.639+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b480770c0 con 0x7f8b580b3090 2023-11-24T08:08:53.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.828+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f8b5814dde0 con 0x7f8b580b3090 2023-11-24T08:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.836+0000 7f8b4dffb640 1 -- 172.21.15.107:0/3080823499 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v17) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8b4807a6e0 con 0x7f8b580b3090 2023-11-24T08:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 msgr2=0x7f8b3c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 0x7f8b3c06d8a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b54006fd0 tx=0x7f8b54006bc0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 msgr2=0x7f8b580af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c00 tx=0x7f8b480046c0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 shutdown_connections 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b3c06b3e0 0x7f8b3c06d8a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b58154980 0x7f8b581f1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b580b3090 0x7f8b580af9f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 --2- 172.21.15.107:0/3080823499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b58072710 0x7f8b58154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 >> 172.21.15.107:0/3080823499 conn(0x7f8b5806a500 msgr2=0x7f8b580cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 shutdown_connections 2023-11-24T08:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:53.839+0000 7f8b5e327640 1 -- 172.21.15.107:0/3080823499 wait complete. 2023-11-24T08:08:53.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled' 2023-11-24T08:08:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.063+0000 7f54c3fff640 1 Processor -- start 2023-11-24T08:08:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.064+0000 7f54c3fff640 1 -- start start 2023-11-24T08:08:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.065+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54c40d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.065+0000 7f54c3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c40ce990 con 0x7f54c40d38e0 2023-11-24T08:08:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.065+0000 7f54c3fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c40cec70 con 0x7f54c40d0240 2023-11-24T08:08:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.065+0000 7f54c3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40d4110 con 0x7f54c415d4e0 2023-11-24T08:08:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f54c40d38e0 0x7f54c40cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56240/0 (socket says 172.21.15.107:56240) 2023-11-24T08:08:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c27fc640 1 -- 172.21.15.107:0/899664418 learned_addr learned my addr 172.21.15.107:0/899664418 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54c40d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232046246 0 0) 0x7f54c40ce990 con 0x7f54c40d38e0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b0003520 con 0x7f54c40d38e0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2115675132 0 0) 0x7f54b0003520 con 0x7f54c40d38e0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 msgr2=0x7f54c40d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54c40d1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 >> v1:172.21.15.120:6789/0 conn(0x7f54c40d0240 legacy=0x7f54c40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40cef50 con 0x7f54c40d38e0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f54b4001a10 con 0x7f54c40d38e0 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.066+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 >> v1:172.21.15.107:6789/0 conn(0x7f54c40d38e0 legacy=0x7f54c40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b00039a0 0x7f54b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b0004d00 0x7f54b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40d4110 con 0x7f54b0004d00 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b000ba30 con 0x7f54b00039a0 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b000bbf0 con 0x7f54c415d4e0 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b0004d00 0x7f54b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b00039a0 0x7f54b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c27fc640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 -- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 msgr2=0x7f54b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 -- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b00039a0 msgr2=0x7f54b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b00039a0 0x7f54b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c37fe640 1 -- 172.21.15.107:0/899664418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0003520 con 0x7f54b0004d00 2023-11-24T08:08:54.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c27fc640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c415d4e0 0x7f54b00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:54.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.067+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b00039a0 0x7f54b0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c37fe640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b0004d00 0x7f54b0005180 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54bc001720 tx=0x7f54b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d343d78ec8ede588 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0052d0 con 0x7f54b0004d00 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc02e050 con 0x7f54b0004d00 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c1ffb640 1 -- 172.21.15.107:0/899664418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0035f0 con 0x7f54b0004d00 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c3fff640 1 -- 172.21.15.107:0/899664418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b0004d00 msgr2=0x7f54b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c3fff640 1 --2- 172.21.15.107:0/899664418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b0004d00 0x7f54b0005180 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54bc001720 tx=0x7f54b00057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c37fe640 1 -- 172.21.15.107:0/899664418 reap_dead start 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c3fff640 1 -- 172.21.15.107:0/899664418 shutdown_connections 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.069+0000 7f54c3fff640 1 -- 172.21.15.107:0/899664418 >> 172.21.15.107:0/899664418 conn(0x7f54c409c510 msgr2=0x7f54c40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.070+0000 7f54c3fff640 1 -- 172.21.15.107:0/899664418 shutdown_connections 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.070+0000 7f54c3fff640 1 -- 172.21.15.107:0/899664418 wait complete. 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.071+0000 7f54c3fff640 1 Processor -- start 2023-11-24T08:08:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.071+0000 7f54c3fff640 1 -- start start 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0004d00 0x7f54c40ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c41526d0 0x7f54c40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40cb6e0 con 0x7f54c40cc240 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40cc6c0 con 0x7f54b0004d00 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.072+0000 7f54c3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40ca7e0 con 0x7f54c41526d0 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45036/0 (socket says 172.21.15.107:45036) 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 -- 172.21.15.107:0/351072961 learned_addr learned my addr 172.21.15.107:0/351072961 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:54.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c27fc640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c41526d0 0x7f54c40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 -- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c41526d0 msgr2=0x7f54c40cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0004d00 0x7f54c40ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c41526d0 0x7f54c40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 -- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0004d00 msgr2=0x7f54c40ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0004d00 0x7f54c40ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 -- 172.21.15.107:0/351072961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40caa60 con 0x7f54c40cc240 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.073+0000 7f54c37fe640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f54bc034040 tx=0x7f54bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.074+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc007a30 con 0x7f54c40cc240 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.074+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc0054a0 con 0x7f54c40cc240 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.074+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0069f0 con 0x7f54c40cc240 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.074+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c40cad40 con 0x7f54c40cc240 2023-11-24T08:08:54.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.074+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c40cb300 con 0x7f54c40cc240 2023-11-24T08:08:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.075+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c404e840 con 0x7f54c40cc240 2023-11-24T08:08:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.076+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f54bc006b50 con 0x7f54c40cc240 2023-11-24T08:08:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.077+0000 7f54bb7fe640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 0x7f54a406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.077+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f54bc0aac10 con 0x7f54c40cc240 2023-11-24T08:08:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.077+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 0x7f54a406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.078+0000 7f54c2ffd640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 0x7f54a406d8a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0099c0 tx=0x7f54ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.082+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54bc075640 con 0x7f54c40cc240 2023-11-24T08:08:54.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.347+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled"]} v 0) v1 -- 0x7f54c406c130 con 0x7f54c40cc240 2023-11-24T08:08:54.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.933+0000 7f54bb7fe640 1 -- 172.21.15.107:0/351072961 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_in_progress_cancelled"]}]=0 v221) v1 ==== 180+0+0 (secure 0 0 0) 0x7f54bc0430f0 con 0x7f54c40cc240 2023-11-24T08:08:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 msgr2=0x7f54a406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 0x7f54a406d8a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0099c0 tx=0x7f54ac008040 comp rx=0 tx=0).stop 2023-11-24T08:08:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 msgr2=0x7f54c40b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f54bc034040 tx=0x7f54bc001120 comp rx=0 tx=0).stop 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 shutdown_connections 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f54a406b3e0 0x7f54a406d8a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40cc240 0x7f54c40b6de0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54c41526d0 0x7f54c40cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 --2- 172.21.15.107:0/351072961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0004d00 0x7f54c40ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.937+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 >> 172.21.15.107:0/351072961 conn(0x7f54c409c510 msgr2=0x7f54c406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.938+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 shutdown_connections 2023-11-24T08:08:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:54.938+0000 7f54c3fff640 1 -- 172.21.15.107:0/351072961 wait complete. 2023-11-24T08:08:54.959 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is cancelled during in-progress state and validate the reason ... ok 2023-11-24T08:08:54.960 INFO:tasks.cephfs_test_runner:Starting test: test_clone_failure_status_pending_cancelled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ensure failure status is shown when clone is cancelled during pending state and validate the reason 2023-11-24T08:08:54.960 INFO:tasks.cephfs_test_runner:test_clone_failure_status_pending_cancelled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:08:54.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.174+0000 7f245494c640 1 Processor -- start 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.174+0000 7f245494c640 1 -- start start 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f245015c7f0 0x7f245006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24500cf450 0x7f24500cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500cdfb0 0x7f24500ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2450156010 con 0x7f24500cf450 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245006ce30 con 0x7f24500cdfb0 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.175+0000 7f245494c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24500cfe10 con 0x7f245015c7f0 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24500cf450 0x7f24500cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24500cf450 0x7f24500cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45042/0 (socket says 172.21.15.107:45042) 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 -- 172.21.15.107:0/4145749941 learned_addr learned my addr 172.21.15.107:0/4145749941 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f244e59c640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f245015c7f0 0x7f245006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f244ed9d640 1 --2- 172.21.15.107:0/4145749941 >> v2:172.21.15.120:3300/0 conn(0x7f24500cdfb0 0x7f24500ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 -- 172.21.15.107:0/4145749941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f245015c7f0 msgr2=0x7f245006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f245015c7f0 0x7f245006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 -- 172.21.15.107:0/4145749941 >> v2:172.21.15.120:3300/0 conn(0x7f24500cdfb0 msgr2=0x7f24500ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 --2- 172.21.15.107:0/4145749941 >> v2:172.21.15.120:3300/0 conn(0x7f24500cdfb0 0x7f24500ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.176+0000 7f2447fff640 1 -- 172.21.15.107:0/4145749941 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24500cece0 con 0x7f24500cf450 2023-11-24T08:08:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.178+0000 7f2447fff640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24500cf450 0x7f24500cf8d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f24500aebc0 tx=0x7f243802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c0d8b590a74f14 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.178+0000 7f244dd9b640 1 -- 172.21.15.107:0/4145749941 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438031870 con 0x7f24500cf450 2023-11-24T08:08:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.178+0000 7f244dd9b640 1 -- 172.21.15.107:0/4145749941 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24380319d0 con 0x7f24500cf450 2023-11-24T08:08:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.178+0000 7f244dd9b640 1 -- 172.21.15.107:0/4145749941 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438031d60 con 0x7f24500cf450 2023-11-24T08:08:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 -- 172.21.15.107:0/4145749941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24500cf450 msgr2=0x7f24500cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24500cf450 0x7f24500cf8d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f24500aebc0 tx=0x7f243802b910 comp rx=0 tx=0).stop 2023-11-24T08:08:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 -- 172.21.15.107:0/4145749941 shutdown_connections 2023-11-24T08:08:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 --2- 172.21.15.107:0/4145749941 >> v2:172.21.15.120:3300/0 conn(0x7f24500cdfb0 0x7f24500ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24500cf450 0x7f24500cf8d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 --2- 172.21.15.107:0/4145749941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f245015c7f0 0x7f245006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 -- 172.21.15.107:0/4145749941 >> 172.21.15.107:0/4145749941 conn(0x7f245006a920 msgr2=0x7f245006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 -- 172.21.15.107:0/4145749941 shutdown_connections 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.179+0000 7f245494c640 1 -- 172.21.15.107:0/4145749941 wait complete. 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.181+0000 7f245494c640 1 Processor -- start 2023-11-24T08:08:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.181+0000 7f245494c640 1 -- start start 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 0x7f24500b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24500cf450 0x7f24500b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24501f3410 con 0x7f245015c7f0 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24500af810 con 0x7f24500cdfb0 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f245494c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24501f3630 con 0x7f24500cf450 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f244ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f244ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45054/0 (socket says 172.21.15.107:45054) 2023-11-24T08:08:55.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.182+0000 7f244ed9d640 1 -- 172.21.15.107:0/3479154879 learned_addr learned my addr 172.21.15.107:0/3479154879 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f2447fff640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24500cf450 0x7f24500b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244e59c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 0x7f24500b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 -- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24500cf450 msgr2=0x7f24500b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24500cf450 0x7f24500b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 -- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 msgr2=0x7f24500b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 0x7f24500b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 -- 172.21.15.107:0/3479154879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380094d0 con 0x7f245015c7f0 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244e59c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 0x7f24500b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f244ed9d640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2448007d10 tx=0x7f244800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448014420 con 0x7f245015c7f0 2023-11-24T08:08:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24480145a0 con 0x7f245015c7f0 2023-11-24T08:08:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24501f3910 con 0x7f245015c7f0 2023-11-24T08:08:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.183+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448014950 con 0x7f245015c7f0 2023-11-24T08:08:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.184+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24501f3ed0 con 0x7f245015c7f0 2023-11-24T08:08:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.185+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245006b230 con 0x7f245015c7f0 2023-11-24T08:08:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.186+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f2448014ab0 con 0x7f245015c7f0 2023-11-24T08:08:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.187+0000 7f2446ffd640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 0x7f243006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.187+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f244808a590 con 0x7f245015c7f0 2023-11-24T08:08:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.187+0000 7f244e59c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 0x7f243006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.188+0000 7f244e59c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 0x7f243006d8d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.192+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2448055120 con 0x7f245015c7f0 2023-11-24T08:08:55.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.382+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2450067470 con 0x7f245015c7f0 2023-11-24T08:08:55.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.385+0000 7f2446ffd640 1 -- 172.21.15.107:0/3479154879 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12988 (secure 0 0 0) 0x7f2448058740 con 0x7f245015c7f0 2023-11-24T08:08:55.386 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":55,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:52.910836+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6836/1158168362":"2023-11-25T08:08:49.507890+0000","172.21.15.107:6834/1158168362":"2023-11-25T08:08:49.507890+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:08:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 msgr2=0x7f243006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 0x7f243006d8d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440008040 comp rx=0 tx=0).stop 2023-11-24T08:08:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 msgr2=0x7f2450099140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2448007d10 tx=0x7f244800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 shutdown_connections 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f243006b410 0x7f243006d8d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f245015c7f0 0x7f2450099140 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24500cf450 0x7f24500b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 --2- 172.21.15.107:0/3479154879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500cdfb0 0x7f24500b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 >> 172.21.15.107:0/3479154879 conn(0x7f245006a920 msgr2=0x7f2450071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.387+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 shutdown_connections 2023-11-24T08:08:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.388+0000 7f245494c640 1 -- 172.21.15.107:0/3479154879 wait complete. 2023-11-24T08:08:55.407 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:08:55.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:08:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.616+0000 7febfecb4640 1 Processor -- start 2023-11-24T08:08:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.616+0000 7febfecb4640 1 -- start start 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfecb4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7febf80b3330 0x7febf806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfecb4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf806dcf0 con 0x7febf8161060 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfecb4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf8161ec0 con 0x7febf80b3790 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfecb4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf806ec10 con 0x7febf80b3330 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfca50640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7febf8161060 0x7febf8161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56242/0 (socket says 172.21.15.107:56242) 2023-11-24T08:08:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.618+0000 7febfca50640 1 -- 172.21.15.107:0/975353764 learned_addr learned my addr 172.21.15.107:0/975353764 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:55.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810046375 0 0) 0x7febf806dcf0 con 0x7febf8161060 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe4003520 con 0x7febf8161060 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf7fff640 1 --2- 172.21.15.107:0/975353764 >> v2:172.21.15.120:3300/0 conn(0x7febf80b3330 0x7febf806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4227634615 0 0) 0x7febe4003520 con 0x7febf8161060 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 >> v2:172.21.15.120:3300/0 conn(0x7febf80b3330 msgr2=0x7febf806a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 --2- 172.21.15.107:0/975353764 >> v2:172.21.15.120:3300/0 conn(0x7febf80b3330 0x7febf806a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 >> v1:172.21.15.120:6789/0 conn(0x7febf80b3790 legacy=0x7febf806abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf806df90 con 0x7febf8161060 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7febe80019f0 con 0x7febf8161060 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 >> v1:172.21.15.107:6789/0 conn(0x7febf8161060 legacy=0x7febf8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe4003970 0x7febe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe40047c0 0x7febe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febe400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf806ec10 con 0x7febe4005180 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe400c120 con 0x7febe40047c0 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.619+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe400c2e0 con 0x7febe4003970 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febe400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfca50640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe4003970 0x7febe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febf7fff640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe40047c0 0x7febe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 -- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe4003970 msgr2=0x7febe4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe4003970 0x7febe4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 -- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe40047c0 msgr2=0x7febe4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe40047c0 0x7febe4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.620+0000 7febfd251640 1 -- 172.21.15.107:0/975353764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4003520 con 0x7febe4005180 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.621+0000 7febfd251640 1 -- 172.21.15.107:0/975353764 reap_dead start 2023-11-24T08:08:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.621+0000 7febfd251640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febe400b980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7febf0001750 tx=0x7febe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7a321c8cf9c076 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.621+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf00052d0 con 0x7febe4005180 2023-11-24T08:08:55.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.621+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf002e050 con 0x7febe4005180 2023-11-24T08:08:55.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.621+0000 7febf77fe640 1 -- 172.21.15.107:0/975353764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0003830 con 0x7febe4005180 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 -- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 msgr2=0x7febe400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febe400b980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7febf0001750 tx=0x7febe40057f0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 -- 172.21.15.107:0/975353764 shutdown_connections 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 --2- 172.21.15.107:0/975353764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febe400b980 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 -- 172.21.15.107:0/975353764 >> 172.21.15.107:0/975353764 conn(0x7febf80cf900 msgr2=0x7febf80cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 -- 172.21.15.107:0/975353764 shutdown_connections 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.622+0000 7febfecb4640 1 -- 172.21.15.107:0/975353764 wait complete. 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.624+0000 7febfecb4640 1 Processor -- start 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.624+0000 7febfecb4640 1 -- start start 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815f570 0x7febf80cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf80ad320 0x7febf80ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80adce0 con 0x7febe4005180 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81f3450 con 0x7febf815f570 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfecb4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80cc430 con 0x7febf80ad320 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfca50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfca50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45066/0 (socket says 172.21.15.107:45066) 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfca50640 1 -- 172.21.15.107:0/3893545985 learned_addr learned my addr 172.21.15.107:0/3893545985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfd251640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf80ad320 0x7febf80ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfca50640 1 -- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf80ad320 msgr2=0x7febf80ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.625+0000 7febfca50640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf80ad320 0x7febf80ad7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfca50640 1 -- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815f570 msgr2=0x7febf80cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfca50640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815f570 0x7febf80cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfca50640 1 -- 172.21.15.107:0/3893545985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80cc710 con 0x7febe4005180 2023-11-24T08:08:55.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfca50640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7febe8007a80 tx=0x7febe800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8013420 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8013580 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8013930 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf80cc9f0 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.626+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf8148880 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.627+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8067990 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.628+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7febe8013a90 con 0x7febe4005180 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.629+0000 7febf57fa640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 0x7febd806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:55.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.629+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7febe808a540 con 0x7febe4005180 2023-11-24T08:08:55.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.629+0000 7febf7fff640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 0x7febd806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:55.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.630+0000 7febf7fff640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 0x7febd806d8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7febf815fb10 tx=0x7febe0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:55.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.633+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febe8055150 con 0x7febe4005180 2023-11-24T08:08:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.824+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febf804e8b0 con 0x7febe4005180 2023-11-24T08:08:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.825+0000 7febf57fa640 1 -- 172.21.15.107:0/3893545985 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7febe8058770 con 0x7febe4005180 2023-11-24T08:08:55.826 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.827+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 msgr2=0x7febd806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.827+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 0x7febd806d8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7febf815fb10 tx=0x7febe0008040 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.827+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 msgr2=0x7febf8148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.827+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7febe8007a80 tx=0x7febe800e530 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 shutdown_connections 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7febd806b430 0x7febd806d8f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf80ad320 0x7febf80ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815f570 0x7febf80cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 --2- 172.21.15.107:0/3893545985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febe4005180 0x7febf8148340 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 >> 172.21.15.107:0/3893545985 conn(0x7febf80cf900 msgr2=0x7febf80ac840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 shutdown_connections 2023-11-24T08:08:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:55.828+0000 7febfecb4640 1 -- 172.21.15.107:0/3893545985 wait complete. 2023-11-24T08:08:55.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled' 2023-11-24T08:08:56.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.061+0000 7f286d5c9640 1 Processor -- start 2023-11-24T08:08:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.062+0000 7f286d5c9640 1 -- start start 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.063+0000 7f286d5c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f286815c7f0 0x7f286806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.063+0000 7f286d5c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28680cf450 0x7f28680cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.063+0000 7f286d5c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680cece0 con 0x7f28680cdfb0 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.063+0000 7f286d5c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2868156010 con 0x7f286815c7f0 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.063+0000 7f286d5c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286806ce30 con 0x7f28680cf450 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f286815c7f0 0x7f286806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f286815c7f0 0x7f286806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45078/0 (socket says 172.21.15.107:45078) 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 -- 172.21.15.107:0/825784566 learned_addr learned my addr 172.21.15.107:0/825784566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f28667fc640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28680cf450 0x7f28680cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 -- 172.21.15.107:0/825784566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28680cf450 msgr2=0x7f28680cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28680cf450 0x7f28680cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 -- 172.21.15.107:0/825784566 >> v1:172.21.15.107:6789/0 conn(0x7f28680cdfb0 legacy=0x7f28680ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2866ffd640 1 -- 172.21.15.107:0/825784566 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286806d040 con 0x7f286815c7f0 2023-11-24T08:08:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.064+0000 7f2865ffb640 1 -- 172.21.15.107:0/825784566 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318398827 0 0) 0x7f28680cece0 con 0x7f28680cdfb0 2023-11-24T08:08:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.065+0000 7f2866ffd640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f286815c7f0 0x7f286806db20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2858009e50 tx=0x7f285802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc1be45aa624424 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f2865ffb640 1 -- 172.21.15.107:0/825784566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285802cb00 con 0x7f286815c7f0 2023-11-24T08:08:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f2865ffb640 1 -- 172.21.15.107:0/825784566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285802cc60 con 0x7f286815c7f0 2023-11-24T08:08:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f2865ffb640 1 -- 172.21.15.107:0/825784566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28580343f0 con 0x7f286815c7f0 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 -- 172.21.15.107:0/825784566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f286815c7f0 msgr2=0x7f286806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f286815c7f0 0x7f286806db20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2858009e50 tx=0x7f285802bf60 comp rx=0 tx=0).stop 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 -- 172.21.15.107:0/825784566 shutdown_connections 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28680cf450 0x7f28680cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 --2- 172.21.15.107:0/825784566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f286815c7f0 0x7f286806db20 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.066+0000 7f286d5c9640 1 -- 172.21.15.107:0/825784566 >> 172.21.15.107:0/825784566 conn(0x7f286806a920 msgr2=0x7f286806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.067+0000 7f286d5c9640 1 -- 172.21.15.107:0/825784566 shutdown_connections 2023-11-24T08:08:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.067+0000 7f286d5c9640 1 -- 172.21.15.107:0/825784566 wait complete. 2023-11-24T08:08:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.068+0000 7f286d5c9640 1 Processor -- start 2023-11-24T08:08:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.068+0000 7f286d5c9640 1 -- start start 2023-11-24T08:08:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28680cf450 0x7f28681f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286815c7f0 0x7f286815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28681f5460 con 0x7f28680cdfb0 2023-11-24T08:08:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28681562f0 con 0x7f28680cf450 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.069+0000 7f286d5c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28681f4ea0 con 0x7f286815c7f0 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45088/0 (socket says 172.21.15.107:45088) 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 -- 172.21.15.107:0/1025726612 learned_addr learned my addr 172.21.15.107:0/1025726612 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 -- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286815c7f0 msgr2=0x7f286815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f28677fe640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286815c7f0 0x7f286815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286815c7f0 0x7f286815a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 -- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28680cf450 msgr2=0x7f28681f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28680cf450 0x7f28681f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 -- 172.21.15.107:0/1025726612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009b30 con 0x7f28680cdfb0 2023-11-24T08:08:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.070+0000 7f2866ffd640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2858000f80 tx=0x7f2858004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:56.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.071+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858009510 con 0x7f28680cdfb0 2023-11-24T08:08:56.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.071+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285803e450 con 0x7f28680cdfb0 2023-11-24T08:08:56.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.071+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285803e800 con 0x7f28680cdfb0 2023-11-24T08:08:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.071+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28681f5180 con 0x7f28680cdfb0 2023-11-24T08:08:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.071+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681f0d30 con 0x7f28680cdfb0 2023-11-24T08:08:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.072+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2868156150 con 0x7f28680cdfb0 2023-11-24T08:08:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.073+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f285803e960 con 0x7f28680cdfb0 2023-11-24T08:08:56.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.074+0000 7f285f7fe640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 0x7f284806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:56.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.074+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f28580aca70 con 0x7f28680cdfb0 2023-11-24T08:08:56.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.074+0000 7f28667fc640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 0x7f284806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:56.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.075+0000 7f28667fc640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 0x7f284806d8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2868153a30 tx=0x7f2850008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:56.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.078+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2868156150 con 0x7f28680cdfb0 2023-11-24T08:08:56.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.337+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled"]} v 0) v1 -- 0x7f2868156150 con 0x7f28680cdfb0 2023-11-24T08:08:56.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.948+0000 7f285f7fe640 1 -- 172.21.15.107:0/1025726612 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled"]}]=0 v223) v1 ==== 179+0+0 (secure 0 0 0) 0x7f2868156150 con 0x7f28680cdfb0 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.951+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 msgr2=0x7f284806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.951+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 0x7f284806d8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2868153a30 tx=0x7f2850008040 comp rx=0 tx=0).stop 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.951+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 msgr2=0x7f28681f07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.951+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2858000f80 tx=0x7f2858004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 shutdown_connections 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f284806b430 0x7f284806d8f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286815c7f0 0x7f286815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28680cf450 0x7f28681f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 --2- 172.21.15.107:0/1025726612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28680cdfb0 0x7f28681f07f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:56.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 >> 172.21.15.107:0/1025726612 conn(0x7f286806a920 msgr2=0x7f28680af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:56.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 shutdown_connections 2023-11-24T08:08:56.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:56.952+0000 7f286d5c9640 1 -- 172.21.15.107:0/1025726612 wait complete. 2023-11-24T08:08:56.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:08:57.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.190+0000 7f3805941640 1 Processor -- start 2023-11-24T08:08:57.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.190+0000 7f3805941640 1 -- start start 2023-11-24T08:08:57.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f3805941640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f380006de10 0x7f380015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f3805941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f38000705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f3805941640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380015ec90 con 0x7f3800072160 2023-11-24T08:08:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f3805941640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38000cd530 con 0x7f380006de10 2023-11-24T08:08:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f3805941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38000c4ed0 con 0x7f38000c49f0 2023-11-24T08:08:57.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f37f67fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f380006de10 0x7f380015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f37f67fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f380006de10 0x7f380015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45098/0 (socket says 172.21.15.107:45098) 2023-11-24T08:08:57.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f37f67fc640 1 -- 172.21.15.107:0/1068434623 learned_addr learned my addr 172.21.15.107:0/1068434623 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:57.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.192+0000 7f37ff7fe640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f38000705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.193+0000 7f37f67fc640 1 -- 172.21.15.107:0/1068434623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 msgr2=0x7f38000705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:57.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.193+0000 7f37f67fc640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f38000705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.193+0000 7f37f67fc640 1 -- 172.21.15.107:0/1068434623 >> v1:172.21.15.120:6789/0 conn(0x7f3800072160 legacy=0x7f380006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.193+0000 7f37f67fc640 1 -- 172.21.15.107:0/1068434623 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000c51b0 con 0x7f380006de10 2023-11-24T08:08:57.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.194+0000 7f37f67fc640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f380015e750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009e50 tx=0x7f37e802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6516e350eef1d85a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:57.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.194+0000 7f37fe7fc640 1 -- 172.21.15.107:0/1068434623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e802cb00 con 0x7f380006de10 2023-11-24T08:08:57.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.194+0000 7f37fe7fc640 1 -- 172.21.15.107:0/1068434623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e802cc60 con 0x7f380006de10 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.194+0000 7f37fe7fc640 1 -- 172.21.15.107:0/1068434623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e80343f0 con 0x7f380006de10 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 -- 172.21.15.107:0/1068434623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 msgr2=0x7f380015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f380015e750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009e50 tx=0x7f37e802bf60 comp rx=0 tx=0).stop 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 -- 172.21.15.107:0/1068434623 shutdown_connections 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f38000705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 --2- 172.21.15.107:0/1068434623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f380015e750 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 -- 172.21.15.107:0/1068434623 >> 172.21.15.107:0/1068434623 conn(0x7f38000b0c50 msgr2=0x7f38000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:57.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 -- 172.21.15.107:0/1068434623 shutdown_connections 2023-11-24T08:08:57.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.195+0000 7f3805941640 1 -- 172.21.15.107:0/1068434623 wait complete. 2023-11-24T08:08:57.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.197+0000 7f3805941640 1 Processor -- start 2023-11-24T08:08:57.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.197+0000 7f3805941640 1 -- start start 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 0x7f3800099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f380009a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000c8b70 con 0x7f380006de10 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380009c6e0 con 0x7f3800072160 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f3805941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380009bdf0 con 0x7f38000c49f0 2023-11-24T08:08:57.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f37feffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f37feffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45110/0 (socket says 172.21.15.107:45110) 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.198+0000 7f37feffd640 1 -- 172.21.15.107:0/2131337134 learned_addr learned my addr 172.21.15.107:0/2131337134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37f67fc640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 0x7f3800099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37ff7fe640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f380009a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 -- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 msgr2=0x7f380009a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f380009a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 -- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 msgr2=0x7f3800099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 0x7f3800099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 -- 172.21.15.107:0/2131337134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8009b30 con 0x7f380006de10 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37f67fc640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 0x7f3800099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37feffd640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c450 tx=0x7f37f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f00052d0 con 0x7f380006de10 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f0005430 con 0x7f380006de10 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0012870 con 0x7f380006de10 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380009c090 con 0x7f380006de10 2023-11-24T08:08:57.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.199+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38001531c0 con 0x7f380006de10 2023-11-24T08:08:57.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.201+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380004e8b0 con 0x7f380006de10 2023-11-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.202+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f37f00129d0 con 0x7f380006de10 2023-11-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.202+0000 7f37fcff9640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 0x7f37e006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.203+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f37f0089690 con 0x7f380006de10 2023-11-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.203+0000 7f37f67fc640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 0x7f37e006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.204+0000 7f37f67fc640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 0x7f37e006d8a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f37e8035000 tx=0x7f37e8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:57.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.207+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37f0054220 con 0x7f380006de10 2023-11-24T08:08:57.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.395+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f380015ec90 con 0x7f380006de10 2023-11-24T08:08:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.402+0000 7f37fcff9640 1 -- 172.21.15.107:0/2131337134 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f380015ec90 con 0x7f380006de10 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.404+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 msgr2=0x7f37e006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.404+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 0x7f37e006d8a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f37e8035000 tx=0x7f37e8030000 comp rx=0 tx=0).stop 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.404+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 msgr2=0x7f3800152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.404+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c450 tx=0x7f37f000c910 comp rx=0 tx=0).stop 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 shutdown_connections 2023-11-24T08:08:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37e006b3e0 0x7f37e006d8a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38000c49f0 0x7f380009a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3800072160 0x7f3800099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 --2- 172.21.15.107:0/2131337134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006de10 0x7f3800152c80 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 >> 172.21.15.107:0/2131337134 conn(0x7f38000b0c50 msgr2=0x7f38000c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 shutdown_connections 2023-11-24T08:08:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:57.405+0000 7f3805941640 1 -- 172.21.15.107:0/2131337134 wait complete. 2023-11-24T08:08:57.424 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:08:57.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:08:57.542 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:08:57.661 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:08:57.661 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:57.662 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:57.720 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:08:57.720 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:08:57.720 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:57.779 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:08:57.780 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:08:57.780 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:57.837 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:08:57.837 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:08:57.837 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:08:57.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:08:57.898 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:08:57.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:08:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.117+0000 7f106c25c640 1 Processor -- start 2023-11-24T08:08:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.117+0000 7f106c25c640 1 -- start start 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f106c25c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10640cfb30 0x7f10640cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f106c25c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10640b2690 con 0x7f106406aa30 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f106c25c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106406ed00 con 0x7f10640b28d0 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f106c25c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10640cc7f0 con 0x7f10640cfb30 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f1069ff8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f106406aa30 0x7f10640cc990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52384/0 (socket says 172.21.15.107:52384) 2023-11-24T08:08:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f1069ff8640 1 -- 172.21.15.107:0/2365535611 learned_addr learned my addr 172.21.15.107:0/2365535611 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:08:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.119+0000 7f106a7f9640 1 --2- 172.21.15.107:0/2365535611 >> v2:172.21.15.107:3300/0 conn(0x7f10640cfb30 0x7f10640cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.120+0000 7f106a7f9640 1 -- 172.21.15.107:0/2365535611 >> v1:172.21.15.120:6789/0 conn(0x7f10640b28d0 legacy=0x7f10640cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.120+0000 7f106a7f9640 1 -- 172.21.15.107:0/2365535611 >> v1:172.21.15.107:6789/0 conn(0x7f106406aa30 legacy=0x7f10640cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.120+0000 7f106a7f9640 1 -- 172.21.15.107:0/2365535611 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106406efe0 con 0x7f10640cfb30 2023-11-24T08:08:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.121+0000 7f106a7f9640 1 --2- 172.21.15.107:0/2365535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10640cc2b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f105c009d90 tx=0x7f105c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f02efa7f7b9251 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.121+0000 7f1068ff6640 1 -- 172.21.15.107:0/2365535611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c02ccf0 con 0x7f10640cfb30 2023-11-24T08:08:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.121+0000 7f1068ff6640 1 -- 172.21.15.107:0/2365535611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c02ce50 con 0x7f10640cfb30 2023-11-24T08:08:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.121+0000 7f1068ff6640 1 -- 172.21.15.107:0/2365535611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c034660 con 0x7f10640cfb30 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 -- 172.21.15.107:0/2365535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 msgr2=0x7f10640cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 --2- 172.21.15.107:0/2365535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10640cc2b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f105c009d90 tx=0x7f105c02bf40 comp rx=0 tx=0).stop 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 -- 172.21.15.107:0/2365535611 shutdown_connections 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 --2- 172.21.15.107:0/2365535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10640cc2b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 -- 172.21.15.107:0/2365535611 >> 172.21.15.107:0/2365535611 conn(0x7f10640999d0 msgr2=0x7f10640b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 -- 172.21.15.107:0/2365535611 shutdown_connections 2023-11-24T08:08:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.122+0000 7f106c25c640 1 -- 172.21.15.107:0/2365535611 wait complete. 2023-11-24T08:08:58.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.123+0000 7f106c25c640 1 Processor -- start 2023-11-24T08:08:58.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 -- start start 2023-11-24T08:08:58.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f106406aa30 0x7f10641f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10640b28d0 0x7f10641e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641f7b80 con 0x7f10640cfb30 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641f6bf0 con 0x7f10640b28d0 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.124+0000 7f106c25c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10641607b0 con 0x7f106406aa30 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33392/0 (socket says 172.21.15.107:33392) 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 -- 172.21.15.107:0/2056057893 learned_addr learned my addr 172.21.15.107:0/2056057893 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 -- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f106406aa30 msgr2=0x7f10641f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f106406aa30 0x7f10641f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 -- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10640b28d0 msgr2=0x7f10641e7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10640b28d0 0x7f10641e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 -- 172.21.15.107:0/2056057893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c009a70 con 0x7f10640cfb30 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.125+0000 7f106a7f9640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f105c02dfd0 tx=0x7f105c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.126+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c03d600 con 0x7f10640cfb30 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.126+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c03d760 con 0x7f10640cfb30 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.126+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064160a50 con 0x7f10640cfb30 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.126+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c03daf0 con 0x7f10640cfb30 2023-11-24T08:08:58.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.126+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641f7120 con 0x7f10640cfb30 2023-11-24T08:08:58.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.127+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10641f7310 con 0x7f10640cfb30 2023-11-24T08:08:58.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.128+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f105c03b040 con 0x7f10640cfb30 2023-11-24T08:08:58.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.129+0000 7f105a7fc640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 0x7f104806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.129+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f105c0abaf0 con 0x7f10640cfb30 2023-11-24T08:08:58.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.129+0000 7f1069ff8640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 0x7f104806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.129+0000 7f1069ff8640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 0x7f104806d8d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1060005880 tx=0x7f1060007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.133+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f105c076630 con 0x7f10640cfb30 2023-11-24T08:08:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.356+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1064067470 con 0x7f10640cfb30 2023-11-24T08:08:58.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.358+0000 7f105a7fc640 1 -- 172.21.15.107:0/2056057893 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f105c079c50 con 0x7f10640cfb30 2023-11-24T08:08:58.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:08:58.360 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1103645103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1103645103},{"type":"v1","addr":"172.21.15.120:6833","nonce":1103645103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2501728350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2501728350},{"type":"v1","addr":"172.21.15.120:6835","nonce":2501728350}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/250332471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":250332471},{"type":"v1","addr":"172.21.15.107:6837","nonce":250332471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":6138,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/3545788774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3545788774},{"type":"v1","addr":"172.21.15.107:6836","nonce":3545788774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[]} 2023-11-24T08:08:58.360 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 26 2023-11-24T08:08:58.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 msgr2=0x7f104806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 0x7f104806d8d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1060005880 tx=0x7f1060007a00 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 msgr2=0x7f10641e7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f105c02dfd0 tx=0x7f105c004a40 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 shutdown_connections 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f104806b410 0x7f104806d8d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10640cfb30 0x7f10641e7aa0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10640b28d0 0x7f10641e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 --2- 172.21.15.107:0/2056057893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f106406aa30 0x7f10641f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 >> 172.21.15.107:0/2056057893 conn(0x7f10640999d0 msgr2=0x7f106415d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 shutdown_connections 2023-11-24T08:08:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.360+0000 7f106c25c640 1 -- 172.21.15.107:0/2056057893 wait complete. 2023-11-24T08:08:58.381 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:08:58.381 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:08:58.382 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:08:58.382 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:08:58.382 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:08:58.382 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:08:58.382 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:08:58.382 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:08:58.383 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:08:58.383 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:08:58.383 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:08:58.383 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:08:58.383 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:08:58.383 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:08:58.384 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:08:58.384 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:08:58.399 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:08:58.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:08:58.399 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:08:58.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:08:58.412 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:08:58.413 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:08:58.415 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:08:58.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:08:58.447 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:08:58.447 DEBUG:teuthology.parallel:result is None 2023-11-24T08:08:58.454 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:08:58.454 DEBUG:teuthology.parallel:result is None 2023-11-24T08:08:58.463 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:08:58.463 DEBUG:teuthology.parallel:result is None 2023-11-24T08:08:58.471 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:08:58.471 DEBUG:teuthology.parallel:result is None 2023-11-24T08:08:58.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:08:58.603 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:08:58.604 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:08:58.629 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:08:58.632 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:08:58.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.654+0000 7f23377c4640 1 Processor -- start 2023-11-24T08:08:58.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.654+0000 7f23377c4640 1 -- start start 2023-11-24T08:08:58.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.655+0000 7f23377c4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f233015b440 0x7f23300cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.655+0000 7f23377c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f233006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.655+0000 7f23377c4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233015eea0 con 0x7f23300cbf20 2023-11-24T08:08:58.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.655+0000 7f23377c4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23300c5e00 con 0x7f233015b440 2023-11-24T08:08:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.655+0000 7f23377c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23300cc3d0 con 0x7f23300c59b0 2023-11-24T08:08:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f233015b440 0x7f23300cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f233015b440 0x7f23300cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33428/0 (socket says 172.21.15.107:33428) 2023-11-24T08:08:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 -- 172.21.15.107:0/791268212 learned_addr learned my addr 172.21.15.107:0/791268212 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335d61640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f233006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 -- 172.21.15.107:0/791268212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 msgr2=0x7f233006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:08:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f233006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 -- 172.21.15.107:0/791268212 >> v1:172.21.15.120:6789/0 conn(0x7f23300cbf20 legacy=0x7f23300c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.656+0000 7f2335560640 1 -- 172.21.15.107:0/791268212 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23300abf30 con 0x7f233015b440 2023-11-24T08:08:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.657+0000 7f2335560640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f23300cb9e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f232c009870 tx=0x7f232c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4087a17e0f058cc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f2327fff640 1 -- 172.21.15.107:0/791268212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c031640 con 0x7f233015b440 2023-11-24T08:08:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f2327fff640 1 -- 172.21.15.107:0/791268212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0317a0 con 0x7f233015b440 2023-11-24T08:08:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f2327fff640 1 -- 172.21.15.107:0/791268212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c031b30 con 0x7f233015b440 2023-11-24T08:08:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 -- 172.21.15.107:0/791268212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 msgr2=0x7f23300cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f23300cb9e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f232c009870 tx=0x7f232c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 -- 172.21.15.107:0/791268212 shutdown_connections 2023-11-24T08:08:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f233006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 --2- 172.21.15.107:0/791268212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f23300cb9e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 -- 172.21.15.107:0/791268212 >> 172.21.15.107:0/791268212 conn(0x7f23300b0c50 msgr2=0x7f23300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 -- 172.21.15.107:0/791268212 shutdown_connections 2023-11-24T08:08:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.658+0000 7f23377c4640 1 -- 172.21.15.107:0/791268212 wait complete. 2023-11-24T08:08:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.660+0000 7f23377c4640 1 Processor -- start 2023-11-24T08:08:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.660+0000 7f23377c4640 1 -- start start 2023-11-24T08:08:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f23301564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23300cbf20 0x7f23300cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23301569e0 con 0x7f233015b440 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233009b6e0 con 0x7f23300cbf20 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.661+0000 7f23377c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233009b9c0 con 0x7f23300c59b0 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2334d5f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23300cbf20 0x7f23300cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335560640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f23301564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33440/0 (socket says 172.21.15.107:33440) 2023-11-24T08:08:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 -- 172.21.15.107:0/3634558845 learned_addr learned my addr 172.21.15.107:0/3634558845 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 -- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 msgr2=0x7f23301564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f23301564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 -- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23300cbf20 msgr2=0x7f23300cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23300cbf20 0x7f23300cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 -- 172.21.15.107:0/3634558845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0094d0 con 0x7f233015b440 2023-11-24T08:08:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2335d61640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2328007e40 tx=0x7f232800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328014620 con 0x7f233015b440 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232800ee90 con 0x7f233015b440 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.662+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23280149e0 con 0x7f233015b440 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.663+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233009bca0 con 0x7f233015b440 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.663+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233009c260 con 0x7f233015b440 2023-11-24T08:08:58.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.664+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233004e8b0 con 0x7f233015b440 2023-11-24T08:08:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.665+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f2328014b40 con 0x7f233015b440 2023-11-24T08:08:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.666+0000 7f2325ffb640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 0x7f231406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:08:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.666+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f232808a590 con 0x7f233015b440 2023-11-24T08:08:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.666+0000 7f2335560640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 0x7f231406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:08:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.667+0000 7f2335560640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 0x7f231406d8a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f232c036000 tx=0x7f232c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:08:58.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.670+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2328055210 con 0x7f233015b440 2023-11-24T08:08:58.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:58.866+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f23300cbd00 con 0x7f233015b440 2023-11-24T08:08:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.991+0000 7f2325ffb640 1 -- 172.21.15.107:0/3634558845 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v60) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2328058830 con 0x7f233015b440 2023-11-24T08:08:59.994 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:08:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 msgr2=0x7f231406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 0x7f231406d8a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f232c036000 tx=0x7f232c03c040 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 msgr2=0x7f233015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2328007e40 tx=0x7f232800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 shutdown_connections 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f231406b3e0 0x7f231406d8a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233015b440 0x7f233015d0b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23300cbf20 0x7f23300cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 --2- 172.21.15.107:0/3634558845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23300c59b0 0x7f23301564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.995+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 >> 172.21.15.107:0/3634558845 conn(0x7f23300b0c50 msgr2=0x7f23300c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.996+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 shutdown_connections 2023-11-24T08:08:59.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:08:59.996+0000 7f23377c4640 1 -- 172.21.15.107:0/3634558845 wait complete. 2023-11-24T08:09:00.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:09:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.213+0000 7f8a4de5c640 1 Processor -- start 2023-11-24T08:09:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.213+0000 7f8a4de5c640 1 -- start start 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a4de5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a4806e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a4de5c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4806e5a0 con 0x7f8a4815d520 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a4de5c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4806c630 con 0x7f8a48071850 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a4de5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4806c910 con 0x7f8a4806dc00 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a477fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8a4815d520 0x7f8a480ce390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52404/0 (socket says 172.21.15.107:52404) 2023-11-24T08:09:00.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.215+0000 7f8a477fe640 1 -- 172.21.15.107:0/4082442581 learned_addr learned my addr 172.21.15.107:0/4082442581 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:00.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a3effd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a4806e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488759702 0 0) 0x7f8a4806e5a0 con 0x7f8a4815d520 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a34003520 con 0x7f8a4815d520 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079546661 0 0) 0x7f8a34003520 con 0x7f8a4815d520 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 msgr2=0x7f8a4806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a4806e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 >> v1:172.21.15.120:6789/0 conn(0x7f8a48071850 legacy=0x7f8a48071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4806cbf0 con 0x7f8a4815d520 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8a38001670 con 0x7f8a4815d520 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 >> v1:172.21.15.107:6789/0 conn(0x7f8a4815d520 legacy=0x7f8a480ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.216+0000 7f8a46ffd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a34003f20 0x7f8a340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a46ffd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a34004d00 0x7f8a34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4806c910 con 0x7f8a34003f20 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3400ba60 con 0x7f8a34004d00 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3400bc20 con 0x7f8a4806dc00 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a34003f20 0x7f8a340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a477fe640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 -- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 msgr2=0x7f8a340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4806dc00 0x7f8a340039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 -- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a34004d00 msgr2=0x7f8a34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a34004d00 0x7f8a34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.217+0000 7f8a3effd640 1 -- 172.21.15.107:0/4082442581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34003520 con 0x7f8a34003f20 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.218+0000 7f8a3effd640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a34003f20 0x7f8a340047c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009ac0 tx=0x7f8a340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea47617c3fc4246 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.218+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3002c6b0 con 0x7f8a34003f20 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.218+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3002c810 con 0x7f8a34003f20 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.218+0000 7f8a46ffd640 1 -- 172.21.15.107:0/4082442581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3002cba0 con 0x7f8a34003f20 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 -- 172.21.15.107:0/4082442581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a34003f20 msgr2=0x7f8a340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/4082442581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a34003f20 0x7f8a340047c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009ac0 tx=0x7f8a340057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a47fff640 1 -- 172.21.15.107:0/4082442581 reap_dead start 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 -- 172.21.15.107:0/4082442581 shutdown_connections 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 -- 172.21.15.107:0/4082442581 >> 172.21.15.107:0/4082442581 conn(0x7f8a480b0c70 msgr2=0x7f8a480b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 -- 172.21.15.107:0/4082442581 shutdown_connections 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.219+0000 7f8a4de5c640 1 -- 172.21.15.107:0/4082442581 wait complete. 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.221+0000 7f8a4de5c640 1 Processor -- start 2023-11-24T08:09:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.221+0000 7f8a4de5c640 1 -- start start 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a34003f20 0x7f8a48147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a48158220 0x7f8a4815f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a480b2d30 con 0x7f8a48157da0 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a48147990 con 0x7f8a48158220 2023-11-24T08:09:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a4de5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a480b23a0 con 0x7f8a34003f20 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33462/0 (socket says 172.21.15.107:33462) 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a3effd640 1 -- 172.21.15.107:0/532613300 learned_addr learned my addr 172.21.15.107:0/532613300 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.222+0000 7f8a477fe640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a34003f20 0x7f8a48147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 -- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a34003f20 msgr2=0x7f8a48147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a34003f20 0x7f8a48147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 -- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a48158220 msgr2=0x7f8a4815f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a48158220 0x7f8a4815f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 -- 172.21.15.107:0/532613300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a480b2640 con 0x7f8a48157da0 2023-11-24T08:09:00.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a3effd640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c250 tx=0x7f8a30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a30003810 con 0x7f8a48157da0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a30003970 con 0x7f8a48157da0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a30003ce0 con 0x7f8a48157da0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a480b2920 con 0x7f8a48157da0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.223+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a480b3220 con 0x7f8a48157da0 2023-11-24T08:09:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.224+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a48067990 con 0x7f8a48157da0 2023-11-24T08:09:00.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.225+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f8a30047450 con 0x7f8a48157da0 2023-11-24T08:09:00.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.226+0000 7f8a44ff9640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 0x7f8a2806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.226+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a300abf50 con 0x7f8a48157da0 2023-11-24T08:09:00.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.226+0000 7f8a477fe640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 0x7f8a2806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.227+0000 7f8a477fe640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 0x7f8a2806d8a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8a38001200 tx=0x7f8a38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.231+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a30076be0 con 0x7f8a48157da0 2023-11-24T08:09:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.491+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8a480b2ab0 con 0x7f8a48157da0 2023-11-24T08:09:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.492+0000 7f8a44ff9640 1 -- 172.21.15.107:0/532613300 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8a300044e0 con 0x7f8a48157da0 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:09:00.494 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:09:00.495 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.496 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.499 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:09:00.499 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:09:00.499 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.499 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.500 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.500 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.501 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:09:00.501 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:09:00.501 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.501 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.502 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.502 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:09:00.505 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.506 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:09:00.509 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.510 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:09:00.510 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:09:00.511 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.511 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.511 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:09:00.511 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:09:00.511 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.512 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.514 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:09:00.514 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:09:00.514 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.514 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:09:00.516 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:09:00.517 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:09:00.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.517 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:09:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:09:00.519 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:00.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:09:00.522 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 msgr2=0x7f8a2806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 0x7f8a2806d8a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8a38001200 tx=0x7f8a38009040 comp rx=0 tx=0).stop 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 msgr2=0x7f8a480b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c250 tx=0x7f8a30004730 comp rx=0 tx=0).stop 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 shutdown_connections 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a2806b3e0 0x7f8a2806d8a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a48158220 0x7f8a4815f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a48157da0 0x7f8a480b3740 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 --2- 172.21.15.107:0/532613300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a34003f20 0x7f8a48147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 >> 172.21.15.107:0/532613300 conn(0x7f8a480b0c70 msgr2=0x7f8a480afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 shutdown_connections 2023-11-24T08:09:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.494+0000 7f8a4de5c640 1 -- 172.21.15.107:0/532613300 wait complete. 2023-11-24T08:09:00.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:09:00.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.732+0000 7fe62e8df640 1 Processor -- start 2023-11-24T08:09:00.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.732+0000 7fe62e8df640 1 -- start start 2023-11-24T08:09:00.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.733+0000 7fe62e8df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe628144b30 0x7fe62815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.733+0000 7fe62e8df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6280b1de0 0x7fe6280b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.733+0000 7fe62e8df640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62815bf80 con 0x7fe62815c410 2023-11-24T08:09:00.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.733+0000 7fe62e8df640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6280b27b0 con 0x7fe6280b1de0 2023-11-24T08:09:00.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.733+0000 7fe62e8df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280b1560 con 0x7fe628144b30 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62d0dc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe62815c410 0x7fe6280b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52418/0 (socket says 172.21.15.107:52418) 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62d0dc640 1 -- 172.21.15.107:0/786489031 learned_addr learned my addr 172.21.15.107:0/786489031 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62e0de640 1 --2- 172.21.15.107:0/786489031 >> v2:172.21.15.107:3300/0 conn(0x7fe6280b1de0 0x7fe6280b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62d8dd640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe628144b30 0x7fe62815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62c8db640 1 -- 172.21.15.107:0/786489031 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400793019 0 0) 0x7fe62815bf80 con 0x7fe62815c410 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62c8db640 1 -- 172.21.15.107:0/786489031 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe614003520 con 0x7fe62815c410 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62e0de640 1 -- 172.21.15.107:0/786489031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe628144b30 msgr2=0x7fe62815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62e0de640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe628144b30 0x7fe62815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62e0de640 1 -- 172.21.15.107:0/786489031 >> v1:172.21.15.107:6789/0 conn(0x7fe62815c410 legacy=0x7fe6280b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.734+0000 7fe62e0de640 1 -- 172.21.15.107:0/786489031 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280b1280 con 0x7fe6280b1de0 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.735+0000 7fe62e0de640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6280b1de0 0x7fe6280b2270 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe620009d90 tx=0x7fe62002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e402c8415728a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62c8db640 1 -- 172.21.15.107:0/786489031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62002cc50 con 0x7fe6280b1de0 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62c8db640 1 -- 172.21.15.107:0/786489031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62002cdb0 con 0x7fe6280b1de0 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62c8db640 1 -- 172.21.15.107:0/786489031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620034660 con 0x7fe6280b1de0 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 -- 172.21.15.107:0/786489031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6280b1de0 msgr2=0x7fe6280b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6280b1de0 0x7fe6280b2270 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe620009d90 tx=0x7fe62002bea0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 -- 172.21.15.107:0/786489031 shutdown_connections 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6280b1de0 0x7fe6280b2270 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 --2- 172.21.15.107:0/786489031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe628144b30 0x7fe62815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 -- 172.21.15.107:0/786489031 >> 172.21.15.107:0/786489031 conn(0x7fe6280cb830 msgr2=0x7fe6280cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 -- 172.21.15.107:0/786489031 shutdown_connections 2023-11-24T08:09:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.736+0000 7fe62e8df640 1 -- 172.21.15.107:0/786489031 wait complete. 2023-11-24T08:09:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.738+0000 7fe62e8df640 1 Processor -- start 2023-11-24T08:09:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.738+0000 7fe62e8df640 1 -- start start 2023-11-24T08:09:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6280b1de0 0x7fe62815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe62815c410 0x7fe6280a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62815af90 con 0x7fe628144b30 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280c9450 con 0x7fe6280b1de0 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62e8df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe628131470 con 0x7fe62815c410 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62d0dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62d0dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33486/0 (socket says 172.21.15.107:33486) 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.739+0000 7fe62d0dc640 1 -- 172.21.15.107:0/2968632489 learned_addr learned my addr 172.21.15.107:0/2968632489 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 -- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe62815c410 msgr2=0x7fe6280a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe62815c410 0x7fe6280a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 -- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6280b1de0 msgr2=0x7fe62815aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6280b1de0 0x7fe62815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 -- 172.21.15.107:0/2968632489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620009a70 con 0x7fe628144b30 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62d0dc640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c410 tx=0x7fe61800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618005ce0 con 0x7fe628144b30 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe618005e40 con 0x7fe628144b30 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6180072f0 con 0x7fe628144b30 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe628131750 con 0x7fe628144b30 2023-11-24T08:09:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.740+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe628131d10 con 0x7fe628144b30 2023-11-24T08:09:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.741+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6280b3610 con 0x7fe628144b30 2023-11-24T08:09:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.742+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fe618007510 con 0x7fe628144b30 2023-11-24T08:09:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.744+0000 7fe61e7fc640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 0x7fe60c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:00.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.744+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe618089360 con 0x7fe628144b30 2023-11-24T08:09:00.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.744+0000 7fe62d8dd640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 0x7fe60c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.745+0000 7fe62d8dd640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 0x7fe60c06d8a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe6240098f0 tx=0x7fe624008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:00.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.748+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6280b3610 con 0x7fe628144b30 2023-11-24T08:09:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.938+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6280b3610 con 0x7fe628144b30 2023-11-24T08:09:00.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.940+0000 7fe61e7fc640 1 -- 172.21.15.107:0/2968632489 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fe6280b3610 con 0x7fe628144b30 2023-11-24T08:09:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:00.943 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:08:59.988914+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:09:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 msgr2=0x7fe60c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 0x7fe60c06d8a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe6240098f0 tx=0x7fe624008040 comp rx=0 tx=0).stop 2023-11-24T08:09:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 msgr2=0x7fe6280c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c410 tx=0x7fe61800c8d0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 shutdown_connections 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe60c06b3e0 0x7fe60c06d8a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe62815c410 0x7fe6280a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe628144b30 0x7fe6280c8f10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 --2- 172.21.15.107:0/2968632489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6280b1de0 0x7fe62815aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 >> 172.21.15.107:0/2968632489 conn(0x7fe6280cb830 msgr2=0x7fe62815d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 shutdown_connections 2023-11-24T08:09:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:00.943+0000 7fe62e8df640 1 -- 172.21.15.107:0/2968632489 wait complete. 2023-11-24T08:09:00.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:09:00.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.178+0000 7fe0bc97c640 1 Processor -- start 2023-11-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.178+0000 7fe0bc97c640 1 -- start start 2023-11-24T08:09:01.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0bc97c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0b8072160 0x7fe0b815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0bc97c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b815ec90 con 0x7fe0b806de10 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0bc97c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b80cd530 con 0x7fe0b80c49f0 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0bc97c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b80c4ed0 con 0x7fe0b8072160 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0b8072160 0x7fe0b815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0b6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe0b806de10 0x7fe0b80705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52422/0 (socket says 172.21.15.107:52422) 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.179+0000 7fe0b6d9d640 1 -- 172.21.15.107:0/106677698 learned_addr learned my addr 172.21.15.107:0/106677698 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.180+0000 7fe0b659c640 1 -- 172.21.15.107:0/106677698 >> v1:172.21.15.120:6789/0 conn(0x7fe0b80c49f0 legacy=0x7fe0b806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.180+0000 7fe0b659c640 1 -- 172.21.15.107:0/106677698 >> v1:172.21.15.107:6789/0 conn(0x7fe0b806de10 legacy=0x7fe0b80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.180+0000 7fe0b659c640 1 -- 172.21.15.107:0/106677698 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b80c51b0 con 0x7fe0b8072160 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.181+0000 7fe0b659c640 1 --2- 172.21.15.107:0/106677698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b8072160 0x7fe0b815e750 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800bfa0 tx=0x7fe0a802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf706f0a7f76f28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.181+0000 7fe0b559a640 1 -- 172.21.15.107:0/106677698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a802ea60 con 0x7fe0b8072160 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.181+0000 7fe0b559a640 1 -- 172.21.15.107:0/106677698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a802ebc0 con 0x7fe0b8072160 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.181+0000 7fe0b559a640 1 -- 172.21.15.107:0/106677698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8009540 con 0x7fe0b8072160 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 -- 172.21.15.107:0/106677698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b8072160 msgr2=0x7fe0b815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/106677698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b8072160 0x7fe0b815e750 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800bfa0 tx=0x7fe0a802e0c0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 -- 172.21.15.107:0/106677698 shutdown_connections 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/106677698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b8072160 0x7fe0b815e750 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 -- 172.21.15.107:0/106677698 >> 172.21.15.107:0/106677698 conn(0x7fe0b80b0c50 msgr2=0x7fe0b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 -- 172.21.15.107:0/106677698 shutdown_connections 2023-11-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.182+0000 7fe0bc97c640 1 -- 172.21.15.107:0/106677698 wait complete. 2023-11-24T08:09:01.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.183+0000 7fe0bc97c640 1 Processor -- start 2023-11-24T08:09:01.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.184+0000 7fe0bc97c640 1 -- start start 2023-11-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.184+0000 7fe0bc97c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0bc97c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b8072160 0x7fe0b80b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0bc97c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b80c49f0 0x7fe0b8159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0bc97c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b81f35a0 con 0x7fe0b806de10 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0bc97c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b81f3880 con 0x7fe0b80c49f0 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0bc97c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b81f3b60 con 0x7fe0b8072160 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33502/0 (socket says 172.21.15.107:33502) 2023-11-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 -- 172.21.15.107:0/1703198279 learned_addr learned my addr 172.21.15.107:0/1703198279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:01.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 -- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b8072160 msgr2=0x7fe0b80b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b8072160 0x7fe0b80b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 -- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b80c49f0 msgr2=0x7fe0b8159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b80c49f0 0x7fe0b8159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.185+0000 7fe0b659c640 1 -- 172.21.15.107:0/1703198279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a800bc80 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0b659c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000f80 tx=0x7fe0a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8036800 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a8036980 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b81f0c50 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8036d30 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.186+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b81f3e40 con 0x7fe0b806de10 2023-11-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.187+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b81f0e90 con 0x7fe0b806de10 2023-11-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.188+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fe0a804a460 con 0x7fe0b806de10 2023-11-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.189+0000 7fe0aeffd640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 0x7fe09806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.189+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe0a80ad290 con 0x7fe0b806de10 2023-11-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.189+0000 7fe0b5d9b640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 0x7fe09806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.190+0000 7fe0b5d9b640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 0x7fe09806d8a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe0b809d0b0 tx=0x7fe0a0006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.193+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0b81f0e90 con 0x7fe0b806de10 2023-11-24T08:09:01.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.382+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0b804e8b0 con 0x7fe0b806de10 2023-11-24T08:09:01.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.383+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1703198279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0a8077f40 con 0x7fe0b806de10 2023-11-24T08:09:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 msgr2=0x7fe09806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 0x7fe09806d8a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe0b809d0b0 tx=0x7fe0a0006ba0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 msgr2=0x7fe0b81f2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000f80 tx=0x7fe0a8004a40 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 shutdown_connections 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09806b3e0 0x7fe09806d8a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b80c49f0 0x7fe0b8159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b8072160 0x7fe0b80b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 --2- 172.21.15.107:0/1703198279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b806de10 0x7fe0b81f2f70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.385+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 >> 172.21.15.107:0/1703198279 conn(0x7fe0b80b0c50 msgr2=0x7fe0b80710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.386+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 shutdown_connections 2023-11-24T08:09:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.386+0000 7fe0bc97c640 1 -- 172.21.15.107:0/1703198279 wait complete. 2023-11-24T08:09:01.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:09:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.614+0000 7f3d7aa4b640 1 Processor -- start 2023-11-24T08:09:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.614+0000 7f3d7aa4b640 1 -- start start 2023-11-24T08:09:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.615+0000 7f3d7aa4b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 0x7f3d7415af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d7aa4b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740cec00 con 0x7f3d7406d6c0 2023-11-24T08:09:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d7aa4b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740b1890 con 0x7f3d7406db20 2023-11-24T08:09:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d7aa4b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7413f9e0 con 0x7f3d7413d120 2023-11-24T08:09:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d79248640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3d7406d6c0 0x7f3d740b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52424/0 (socket says 172.21.15.107:52424) 2023-11-24T08:09:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d79248640 1 -- 172.21.15.107:0/3464481905 learned_addr learned my addr 172.21.15.107:0/3464481905 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d79a49640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 0x7f3d7415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204731586 0 0) 0x7f3d740cec00 con 0x7f3d7406d6c0 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.616+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d60003520 con 0x7f3d7406d6c0 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527893607 0 0) 0x7f3d60003520 con 0x7f3d7406d6c0 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 msgr2=0x7f3d7415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 0x7f3d7415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 >> v1:172.21.15.120:6789/0 conn(0x7f3d7406db20 legacy=0x7f3d740b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7413fcc0 con 0x7f3d7406d6c0 2023-11-24T08:09:01.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3d64001670 con 0x7f3d7406d6c0 2023-11-24T08:09:01.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 >> v1:172.21.15.107:6789/0 conn(0x7f3d7406d6c0 legacy=0x7f3d740b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d600039a0 0x7f3d60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 0x7f3d600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7413f9e0 con 0x7f3d600047c0 2023-11-24T08:09:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6000dcc0 con 0x7f3d600039a0 2023-11-24T08:09:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6000de80 con 0x7f3d7413d120 2023-11-24T08:09:01.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.617+0000 7f3d79248640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79a49640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d600039a0 0x7f3d60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79248640 1 -- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 msgr2=0x7f3d600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79248640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d7413d120 0x7f3d600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79248640 1 -- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d600039a0 msgr2=0x7f3d60003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79248640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d600039a0 0x7f3d60003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.618+0000 7f3d79248640 1 -- 172.21.15.107:0/3464481905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d60003520 con 0x7f3d600047c0 2023-11-24T08:09:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.619+0000 7f3d79248640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d60004c40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400c240 tx=0x7f3d600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9046d68a74454312 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.619+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64005ad0 con 0x7f3d600047c0 2023-11-24T08:09:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.619+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64005c30 con 0x7f3d600047c0 2023-11-24T08:09:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.619+0000 7f3d78a47640 1 -- 172.21.15.107:0/3464481905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6402c8b0 con 0x7f3d600047c0 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/3464481905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 msgr2=0x7f3d60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/3464481905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d60004c40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400c240 tx=0x7f3d600057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7a24a640 1 -- 172.21.15.107:0/3464481905 reap_dead start 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/3464481905 shutdown_connections 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/3464481905 >> 172.21.15.107:0/3464481905 conn(0x7f3d740cb830 msgr2=0x7f3d740cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/3464481905 shutdown_connections 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.620+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/3464481905 wait complete. 2023-11-24T08:09:01.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.622+0000 7f3d7aa4b640 1 Processor -- start 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.622+0000 7f3d7aa4b640 1 -- start start 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d740b0360 0x7f3d740b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d740b2fa0 0x7f3d740ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740b5050 con 0x7f3d600047c0 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740b3420 con 0x7f3d740b0360 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d7aa4b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740ac970 con 0x7f3d740b2fa0 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33520/0 (socket says 172.21.15.107:33520) 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 -- 172.21.15.107:0/2193751019 learned_addr learned my addr 172.21.15.107:0/2193751019 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 -- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d740b2fa0 msgr2=0x7f3d740ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d740b2fa0 0x7f3d740ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79248640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d740b0360 0x7f3d740b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 -- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d740b0360 msgr2=0x7f3d740b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d740b0360 0x7f3d740b2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.623+0000 7f3d79a49640 1 -- 172.21.15.107:0/2193751019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74159890 con 0x7f3d600047c0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d79a49640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3d70011610 tx=0x7f3d70011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70004220 con 0x7f3d600047c0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70004380 con 0x7f3d600047c0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d74158ff0 con 0x7f3d600047c0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000f8f0 con 0x7f3d600047c0 2023-11-24T08:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.624+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d741595b0 con 0x7f3d600047c0 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.625+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7404e840 con 0x7f3d600047c0 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.626+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f3d7000fa50 con 0x7f3d600047c0 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.627+0000 7f3d6a7fc640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 0x7f3d5806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.627+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d70089f60 con 0x7f3d600047c0 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.627+0000 7f3d79248640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 0x7f3d5806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.628+0000 7f3d79248640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 0x7f3d5806d8a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d64034000 tx=0x7f3d64009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:01.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.631+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d70054ab0 con 0x7f3d600047c0 2023-11-24T08:09:01.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.825+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3d7415d170 con 0x7f3d600047c0 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.826+0000 7f3d6a7fc640 1 -- 172.21.15.107:0/2193751019 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v60) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3d70022090 con 0x7f3d600047c0 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:09:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:09:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:01.829 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 msgr2=0x7f3d5806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 0x7f3d5806d8a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d64034000 tx=0x7f3d64009030 comp rx=0 tx=0).stop 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 msgr2=0x7f3d74153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3d70011610 tx=0x7f3d70011ad0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 shutdown_connections 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5806b3e0 0x7f3d5806d8a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d740b2fa0 0x7f3d740ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d740b0360 0x7f3d740b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 --2- 172.21.15.107:0/2193751019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d600047c0 0x7f3d74153710 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 >> 172.21.15.107:0/2193751019 conn(0x7f3d740cb830 msgr2=0x7f3d74071630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 shutdown_connections 2023-11-24T08:09:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:01.830+0000 7f3d7aa4b640 1 -- 172.21.15.107:0/2193751019 wait complete. 2023-11-24T08:09:01.851 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:09:01.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:09:02.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.070+0000 7f3352fc9640 1 Processor -- start 2023-11-24T08:09:02.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.070+0000 7f3352fc9640 1 -- start start 2023-11-24T08:09:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3352fc9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f334c0cf450 0x7f334c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3352fc9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0cece0 con 0x7f334c0cdfb0 2023-11-24T08:09:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3352fc9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c06d040 con 0x7f334c15c7f0 2023-11-24T08:09:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3352fc9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c0cfe40 con 0x7f334c0cf450 2023-11-24T08:09:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3351566640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f334c0cdfb0 0x7f334c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52428/0 (socket says 172.21.15.107:52428) 2023-11-24T08:09:02.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f3351566640 1 -- 172.21.15.107:0/1560994987 learned_addr learned my addr 172.21.15.107:0/1560994987 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f334bfff640 1 --2- 172.21.15.107:0/1560994987 >> v2:172.21.15.120:3300/0 conn(0x7f334c0cf450 0x7f334c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.072+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571523621 0 0) 0x7f334c0cece0 con 0x7f334c0cdfb0 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3334003520 con 0x7f334c0cdfb0 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173817112 0 0) 0x7f3334003520 con 0x7f334c0cdfb0 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 >> v2:172.21.15.120:3300/0 conn(0x7f334c0cf450 msgr2=0x7f334c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 --2- 172.21.15.107:0/1560994987 >> v2:172.21.15.120:3300/0 conn(0x7f334c0cf450 0x7f334c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 >> v1:172.21.15.120:6789/0 conn(0x7f334c15c7f0 legacy=0x7f334c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c06d320 con 0x7f334c0cdfb0 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f33440019f0 con 0x7f334c0cdfb0 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 >> v1:172.21.15.107:6789/0 conn(0x7f334c0cdfb0 legacy=0x7f334c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33340039c0 0x7f3334003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 0x7f3334004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334005180 0x7f333400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0cfe40 con 0x7f33340047c0 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333400c080 con 0x7f3334005180 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333400c240 con 0x7f33340039c0 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.073+0000 7f334bfff640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 0x7f3334004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f3351566640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334005180 0x7f333400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f3350d65640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33340039c0 0x7f3334003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f334bfff640 1 -- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33340039c0 msgr2=0x7f3334003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f334bfff640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33340039c0 0x7f3334003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f334bfff640 1 -- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334005180 msgr2=0x7f333400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f334bfff640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334005180 0x7f333400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f334bfff640 1 -- 172.21.15.107:0/1560994987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334003520 con 0x7f33340047c0 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.074+0000 7f3351566640 1 -- 172.21.15.107:0/1560994987 reap_dead start 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.075+0000 7f334bfff640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 0x7f3334004c40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3340009ac0 tx=0x7f33340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3e5e2a723966ac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.075+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334002c6b0 con 0x7f33340047c0 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.075+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334002c810 con 0x7f33340047c0 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.075+0000 7f334b7fe640 1 -- 172.21.15.107:0/1560994987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334002cba0 con 0x7f33340047c0 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 -- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 msgr2=0x7f3334004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 0x7f3334004c40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3340009ac0 tx=0x7f33340057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 -- 172.21.15.107:0/1560994987 shutdown_connections 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1560994987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33340047c0 0x7f3334004c40 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 -- 172.21.15.107:0/1560994987 >> 172.21.15.107:0/1560994987 conn(0x7f334c06a920 msgr2=0x7f334c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 -- 172.21.15.107:0/1560994987 shutdown_connections 2023-11-24T08:09:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.076+0000 7f3352fc9640 1 -- 172.21.15.107:0/1560994987 wait complete. 2023-11-24T08:09:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.078+0000 7f3352fc9640 1 Processor -- start 2023-11-24T08:09:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.078+0000 7f3352fc9640 1 -- start start 2023-11-24T08:09:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33340047c0 0x7f334c14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f334c06cb20 0x7f334c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0b3a10 con 0x7f334c159c10 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0b2e00 con 0x7f33340047c0 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3352fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0b30e0 con 0x7f334c06cb20 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33538/0 (socket says 172.21.15.107:33538) 2023-11-24T08:09:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 -- 172.21.15.107:0/1113453507 learned_addr learned my addr 172.21.15.107:0/1113453507 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3350d65640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33340047c0 0x7f334c14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f334bfff640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f334c06cb20 0x7f334c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 -- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f334c06cb20 msgr2=0x7f334c1596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f334c06cb20 0x7f334c1596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 -- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33340047c0 msgr2=0x7f334c14c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.079+0000 7f3351566640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33340047c0 0x7f334c14c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f3351566640 1 -- 172.21.15.107:0/1113453507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0b33c0 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f3351566640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f334c0ccd80 tx=0x7f3344007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334400ea40 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334400eba0 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344013420 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c0b36a0 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.080+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c15aa10 con 0x7f334c159c10 2023-11-24T08:09:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.081+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c0b3830 con 0x7f334c159c10 2023-11-24T08:09:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.082+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f3344013580 con 0x7f334c159c10 2023-11-24T08:09:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.083+0000 7f33497fa640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 0x7f333806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.083+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f334408a170 con 0x7f334c159c10 2023-11-24T08:09:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.084+0000 7f3350d65640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 0x7f333806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.084+0000 7f3350d65640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 0x7f333806d8a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f334c0681f0 tx=0x7f333c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.087+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3344054c70 con 0x7f334c159c10 2023-11-24T08:09:02.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:02.279+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f334c067470 con 0x7f334c159c10 2023-11-24T08:09:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.019+0000 7f33497fa640 1 -- 172.21.15.107:0/1113453507 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v61) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3344058290 con 0x7f334c159c10 2023-11-24T08:09:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:09:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 msgr2=0x7f333806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 0x7f333806d8a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f334c0681f0 tx=0x7f333c009040 comp rx=0 tx=0).stop 2023-11-24T08:09:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 msgr2=0x7f334c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f334c0ccd80 tx=0x7f3344007af0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 shutdown_connections 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f333806b3e0 0x7f333806d8a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f334c159c10 0x7f334c15a090 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f334c06cb20 0x7f334c1596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 --2- 172.21.15.107:0/1113453507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33340047c0 0x7f334c14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 >> 172.21.15.107:0/1113453507 conn(0x7f334c06a920 msgr2=0x7f334c15eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.023+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 shutdown_connections 2023-11-24T08:09:03.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.024+0000 7f3352fc9640 1 -- 172.21.15.107:0/1113453507 wait complete. 2023-11-24T08:09:03.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:09:03.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.250+0000 7f8e506da640 1 Processor -- start 2023-11-24T08:09:03.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.250+0000 7f8e506da640 1 -- start start 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e506da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8e480d2000 0x7f8e480cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e506da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 0x7f8e481605f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e506da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e480ce410 con 0x7f8e480d1c20 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e506da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4806c270 con 0x7f8e480d2000 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e506da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480d06f0 con 0x7f8e480d0200 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e4e476640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8e480d1c20 0x7f8e4806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52432/0 (socket says 172.21.15.107:52432) 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e4e476640 1 -- 172.21.15.107:0/362314545 learned_addr learned my addr 172.21.15.107:0/362314545 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:03.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e4dc75640 1 --2- 172.21.15.107:0/362314545 >> v2:172.21.15.107:3300/0 conn(0x7f8e480d2000 0x7f8e480cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.252+0000 7f8e4ec77640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 0x7f8e481605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4d474640 1 -- 172.21.15.107:0/362314545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926218789 0 0) 0x7f8e480ce410 con 0x7f8e480d1c20 2023-11-24T08:09:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4dc75640 1 -- 172.21.15.107:0/362314545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 msgr2=0x7f8e481605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4dc75640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 0x7f8e481605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4dc75640 1 -- 172.21.15.107:0/362314545 >> v1:172.21.15.107:6789/0 conn(0x7f8e480d1c20 legacy=0x7f8e4806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4dc75640 1 -- 172.21.15.107:0/362314545 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e480d09d0 con 0x7f8e480d2000 2023-11-24T08:09:03.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.253+0000 7f8e4ec77640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 0x7f8e481605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:03.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.254+0000 7f8e4dc75640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d2000 0x7f8e480cfcc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8e380097a0 tx=0x7f8e3802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef07d2b6193cf3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.254+0000 7f8e4d474640 1 -- 172.21.15.107:0/362314545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3802d040 con 0x7f8e480d2000 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.254+0000 7f8e4d474640 1 -- 172.21.15.107:0/362314545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38031810 con 0x7f8e480d2000 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.254+0000 7f8e4d474640 1 -- 172.21.15.107:0/362314545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38031ba0 con 0x7f8e480d2000 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 -- 172.21.15.107:0/362314545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d2000 msgr2=0x7f8e480cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d2000 0x7f8e480cfcc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8e380097a0 tx=0x7f8e3802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 -- 172.21.15.107:0/362314545 shutdown_connections 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d0200 0x7f8e481605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 --2- 172.21.15.107:0/362314545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d2000 0x7f8e480cfcc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 -- 172.21.15.107:0/362314545 >> 172.21.15.107:0/362314545 conn(0x7f8e4808cea0 msgr2=0x7f8e4809c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:03.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 -- 172.21.15.107:0/362314545 shutdown_connections 2023-11-24T08:09:03.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.255+0000 7f8e506da640 1 -- 172.21.15.107:0/362314545 wait complete. 2023-11-24T08:09:03.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.256+0000 7f8e506da640 1 Processor -- start 2023-11-24T08:09:03.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.257+0000 7f8e506da640 1 -- start start 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480d1c20 0x7f8e480caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d2000 0x7f8e4815d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480cc150 con 0x7f8e480d0200 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480cb8b0 con 0x7f8e480d1c20 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e506da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480cbb90 con 0x7f8e480d2000 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33554/0 (socket says 172.21.15.107:33554) 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 -- 172.21.15.107:0/4293810285 learned_addr learned my addr 172.21.15.107:0/4293810285 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 -- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d2000 msgr2=0x7f8e4815d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d2000 0x7f8e4815d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 -- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480d1c20 msgr2=0x7f8e480caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480d1c20 0x7f8e480caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:03.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.258+0000 7f8e4e476640 1 -- 172.21.15.107:0/4293810285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38009400 con 0x7f8e480d0200 2023-11-24T08:09:03.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e4e476640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400c4d0 tx=0x7f8e4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:03.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440074e0 con 0x7f8e480d0200 2023-11-24T08:09:03.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e44007660 con 0x7f8e480d0200 2023-11-24T08:09:03.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44004a20 con 0x7f8e480d0200 2023-11-24T08:09:03.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e480cbe70 con 0x7f8e480d0200 2023-11-24T08:09:03.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.259+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e480cce50 con 0x7f8e480d0200 2023-11-24T08:09:03.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.260+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4804e8b0 con 0x7f8e480d0200 2023-11-24T08:09:03.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.261+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f8e4401b360 con 0x7f8e480d0200 2023-11-24T08:09:03.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.262+0000 7f8e3effd640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 0x7f8e2c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:03.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.262+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8e44089220 con 0x7f8e480d0200 2023-11-24T08:09:03.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.262+0000 7f8e4dc75640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 0x7f8e2c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:03.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.263+0000 7f8e4dc75640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 0x7f8e2c06d8a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8e38036000 tx=0x7f8e38030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:03.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.267+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e44053dd0 con 0x7f8e480d0200 2023-11-24T08:09:03.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:03.452+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8e4806b770 con 0x7f8e480d0200 2023-11-24T08:09:04.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.033+0000 7f8e3effd640 1 -- 172.21.15.107:0/4293810285 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v62) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8e440180d0 con 0x7f8e480d0200 2023-11-24T08:09:04.036 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 msgr2=0x7f8e2c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 0x7f8e2c06d8a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8e38036000 tx=0x7f8e38030000 comp rx=0 tx=0).stop 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 msgr2=0x7f8e481f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400c4d0 tx=0x7f8e4400c990 comp rx=0 tx=0).stop 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 shutdown_connections 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e2c06b3e0 0x7f8e2c06d8a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480d2000 0x7f8e4815d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480d1c20 0x7f8e480caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 --2- 172.21.15.107:0/4293810285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480d0200 0x7f8e481f2a90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 >> 172.21.15.107:0/4293810285 conn(0x7f8e4808cea0 msgr2=0x7f8e480d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 shutdown_connections 2023-11-24T08:09:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.037+0000 7f8e506da640 1 -- 172.21.15.107:0/4293810285 wait complete. 2023-11-24T08:09:04.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:09:04.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.250+0000 7fd6a3815640 1 Processor -- start 2023-11-24T08:09:04.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.250+0000 7fd6a3815640 1 -- start start 2023-11-24T08:09:04.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a3815640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd69c06de10 0x7fd69c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a3815640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd69c0c49f0 0x7fd69c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a3815640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c15ec90 con 0x7fd69c072160 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a3815640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c0cd530 con 0x7fd69c06de10 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a3815640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c0c4ed0 con 0x7fd69c0c49f0 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a0db0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd69c06de10 0x7fd69c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a0db0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd69c06de10 0x7fd69c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33566/0 (socket says 172.21.15.107:33566) 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.252+0000 7fd6a0db0640 1 -- 172.21.15.107:0/3785432356 learned_addr learned my addr 172.21.15.107:0/3785432356 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.253+0000 7fd6a1db2640 1 --2- 172.21.15.107:0/3785432356 >> v2:172.21.15.120:3300/0 conn(0x7fd69c0c49f0 0x7fd69c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.253+0000 7fd6a0db0640 1 -- 172.21.15.107:0/3785432356 >> v2:172.21.15.120:3300/0 conn(0x7fd69c0c49f0 msgr2=0x7fd69c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.253+0000 7fd6a0db0640 1 --2- 172.21.15.107:0/3785432356 >> v2:172.21.15.120:3300/0 conn(0x7fd69c0c49f0 0x7fd69c0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.253+0000 7fd6a0db0640 1 -- 172.21.15.107:0/3785432356 >> v1:172.21.15.120:6789/0 conn(0x7fd69c072160 legacy=0x7fd69c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.253+0000 7fd6a0db0640 1 -- 172.21.15.107:0/3785432356 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0c51b0 con 0x7fd69c06de10 2023-11-24T08:09:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.254+0000 7fd6a0db0640 1 --2- 172.21.15.107:0/3785432356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c06de10 0x7fd69c15e750 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd69c072a30 tx=0x7fd68c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa106177aaa06754 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.254+0000 7fd693fff640 1 -- 172.21.15.107:0/3785432356 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02ee90 con 0x7fd69c06de10 2023-11-24T08:09:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.254+0000 7fd693fff640 1 -- 172.21.15.107:0/3785432356 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c036440 con 0x7fd69c06de10 2023-11-24T08:09:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.254+0000 7fd693fff640 1 -- 172.21.15.107:0/3785432356 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0367d0 con 0x7fd69c06de10 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 -- 172.21.15.107:0/3785432356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c06de10 msgr2=0x7fd69c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3785432356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c06de10 0x7fd69c15e750 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd69c072a30 tx=0x7fd68c02e0e0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 -- 172.21.15.107:0/3785432356 shutdown_connections 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3785432356 >> v2:172.21.15.120:3300/0 conn(0x7fd69c0c49f0 0x7fd69c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3785432356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c06de10 0x7fd69c15e750 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 -- 172.21.15.107:0/3785432356 >> 172.21.15.107:0/3785432356 conn(0x7fd69c0b0c50 msgr2=0x7fd69c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 -- 172.21.15.107:0/3785432356 shutdown_connections 2023-11-24T08:09:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.255+0000 7fd6a3815640 1 -- 172.21.15.107:0/3785432356 wait complete. 2023-11-24T08:09:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.257+0000 7fd6a3815640 1 Processor -- start 2023-11-24T08:09:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.257+0000 7fd6a3815640 1 -- start start 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 0x7fd69c09c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 0x7fd69c09aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c09b740 con 0x7fd69c0c49f0 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c09ba20 con 0x7fd69c072160 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a3815640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1510c0 con 0x7fd69c06de10 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33576/0 (socket says 172.21.15.107:33576) 2023-11-24T08:09:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 -- 172.21.15.107:0/3631878424 learned_addr learned my addr 172.21.15.107:0/3631878424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a15b1640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 0x7fd69c09c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a0db0640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 0x7fd69c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 -- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 msgr2=0x7fd69c09c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 0x7fd69c09c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 -- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 msgr2=0x7fd69c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 0x7fd69c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a1db2640 1 -- 172.21.15.107:0/3631878424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c00b770 con 0x7fd69c0c49f0 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a15b1640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 0x7fd69c09c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.258+0000 7fd6a0db0640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 0x7fd69c09aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd6a1db2640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd69400cc00 tx=0x7fd69400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69400ce10 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694003990 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694003d20 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c09c930 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.259+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c0b47d0 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.260+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c150f20 con 0x7fd69c0c49f0 2023-11-24T08:09:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.261+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fd694003e80 con 0x7fd69c0c49f0 2023-11-24T08:09:04.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.263+0000 7fd691ffb640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 0x7fd68006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:04.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.263+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd69408a4f0 con 0x7fd69c0c49f0 2023-11-24T08:09:04.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.263+0000 7fd6a15b1640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 0x7fd68006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:04.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.263+0000 7fd6a15b1640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 0x7fd68006d8a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd698007fd0 tx=0x7fd698009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.267+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd69c150f20 con 0x7fd69c0c49f0 2023-11-24T08:09:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:04.479+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd69c150f20 con 0x7fd69c0c49f0 2023-11-24T08:09:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.036+0000 7fd691ffb640 1 -- 172.21.15.107:0/3631878424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v28) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd69c150f20 con 0x7fd69c0c49f0 2023-11-24T08:09:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 8 and data pool 9 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 msgr2=0x7fd68006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 0x7fd68006d8a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd698007fd0 tx=0x7fd698009040 comp rx=0 tx=0).stop 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 msgr2=0x7fd69c09b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd69400cc00 tx=0x7fd69400a5c0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 shutdown_connections 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd68006b3e0 0x7fd68006d8a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd69c0c49f0 0x7fd69c09b0e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69c072160 0x7fd69c09aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 --2- 172.21.15.107:0/3631878424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd69c06de10 0x7fd69c09c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 >> 172.21.15.107:0/3631878424 conn(0x7fd69c0b0c50 msgr2=0x7fd69c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 shutdown_connections 2023-11-24T08:09:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.041+0000 7fd6a3815640 1 -- 172.21.15.107:0/3631878424 wait complete. 2023-11-24T08:09:05.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:09:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.252+0000 7fa18ab7c640 1 Processor -- start 2023-11-24T08:09:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.252+0000 7fa18ab7c640 1 -- start start 2023-11-24T08:09:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.253+0000 7fa18ab7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.253+0000 7fa18ab7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 0x7fa1840cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.253+0000 7fa18ab7c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840cece0 con 0x7fa1840cdfb0 2023-11-24T08:09:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.253+0000 7fa18ab7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa184156010 con 0x7fa1840cf450 2023-11-24T08:09:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.253+0000 7fa18ab7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18406ce30 con 0x7fa18415c7f0 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.254+0000 7fa188918640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.254+0000 7fa183fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 0x7fa1840cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.254+0000 7fa188918640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56530/0 (socket says 172.21.15.107:56530) 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.254+0000 7fa188918640 1 -- 172.21.15.107:0/1474605770 learned_addr learned my addr 172.21.15.107:0/1474605770 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.255+0000 7fa188918640 1 -- 172.21.15.107:0/1474605770 >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 msgr2=0x7fa1840cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.255+0000 7fa188918640 1 --2- 172.21.15.107:0/1474605770 >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 0x7fa1840cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.255+0000 7fa188918640 1 -- 172.21.15.107:0/1474605770 >> v1:172.21.15.120:6789/0 conn(0x7fa1840cdfb0 legacy=0x7fa1840ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.255+0000 7fa188918640 1 -- 172.21.15.107:0/1474605770 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18406d040 con 0x7fa18415c7f0 2023-11-24T08:09:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.255+0000 7fa183fff640 1 --2- 172.21.15.107:0/1474605770 >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 0x7fa1840cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:09:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.256+0000 7fa188918640 1 --2- 172.21.15.107:0/1474605770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa178009870 tx=0x7fa17802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e753518100ce51b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa1837fe640 1 -- 172.21.15.107:0/1474605770 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178030840 con 0x7fa18415c7f0 2023-11-24T08:09:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa1837fe640 1 -- 172.21.15.107:0/1474605770 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1780309a0 con 0x7fa18415c7f0 2023-11-24T08:09:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa1837fe640 1 -- 172.21.15.107:0/1474605770 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178030d30 con 0x7fa18415c7f0 2023-11-24T08:09:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 -- 172.21.15.107:0/1474605770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 msgr2=0x7fa18406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/1474605770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa178009870 tx=0x7fa17802b980 comp rx=0 tx=0).stop 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 -- 172.21.15.107:0/1474605770 shutdown_connections 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/1474605770 >> v2:172.21.15.120:3300/0 conn(0x7fa1840cf450 0x7fa1840cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/1474605770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18406db20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 -- 172.21.15.107:0/1474605770 >> 172.21.15.107:0/1474605770 conn(0x7fa18406a920 msgr2=0x7fa18406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 -- 172.21.15.107:0/1474605770 shutdown_connections 2023-11-24T08:09:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.257+0000 7fa18ab7c640 1 -- 172.21.15.107:0/1474605770 wait complete. 2023-11-24T08:09:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.259+0000 7fa18ab7c640 1 Processor -- start 2023-11-24T08:09:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.260+0000 7fa18ab7c640 1 -- start start 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1840cdfb0 0x7fa1840ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18415b620 con 0x7fa1840cf450 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa184159770 con 0x7fa1840cdfb0 2023-11-24T08:09:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa18ab7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa184159a50 con 0x7fa18415c7f0 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33582/0 (socket says 172.21.15.107:33582) 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 -- 172.21.15.107:0/3093008716 learned_addr learned my addr 172.21.15.107:0/3093008716 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa188918640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1840cdfb0 0x7fa1840ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa189119640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 -- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 msgr2=0x7fa18415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 -- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1840cdfb0 msgr2=0x7fa1840ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1840cdfb0 0x7fa1840ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.261+0000 7fa183fff640 1 -- 172.21.15.107:0/3093008716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170009400 con 0x7fa1840cf450 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa189119640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa183fff640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa17000ebe0 tx=0x7fa17000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170009760 con 0x7fa1840cf450 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17000cc30 con 0x7fa1840cf450 2023-11-24T08:09:05.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17000eee0 con 0x7fa1840cf450 2023-11-24T08:09:05.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780094d0 con 0x7fa1840cf450 2023-11-24T08:09:05.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.262+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18415a010 con 0x7fa1840cf450 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.263+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa184067470 con 0x7fa1840cf450 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.264+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fa170013470 con 0x7fa1840cf450 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.266+0000 7fa1817fa640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 0x7fa16806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.266+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa17008a400 con 0x7fa1840cf450 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.266+0000 7fa188918640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 0x7fa16806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.267+0000 7fa188918640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 0x7fa16806d8a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa178036000 tx=0x7fa17803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.270+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa170054c00 con 0x7fa1840cf450 2023-11-24T08:09:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.459+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1840cd730 con 0x7fa1840cf450 2023-11-24T08:09:05.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.462+0000 7fa1817fa640 1 -- 172.21.15.107:0/3093008716 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16578 (secure 0 0 0) 0x7fa170058220 con 0x7fa1840cf450 2023-11-24T08:09:05.465 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:05.465 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":63,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:04.481317+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:09:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 msgr2=0x7fa16806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 0x7fa16806d8a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa178036000 tx=0x7fa17803b040 comp rx=0 tx=0).stop 2023-11-24T08:09:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 msgr2=0x7fa18415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa17000ebe0 tx=0x7fa17000c520 comp rx=0 tx=0).stop 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 shutdown_connections 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa16806b3e0 0x7fa16806d8a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa18415c7f0 0x7fa18415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1840cf450 0x7fa18415aab0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 --2- 172.21.15.107:0/3093008716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1840cdfb0 0x7fa1840ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 >> 172.21.15.107:0/3093008716 conn(0x7fa18406a920 msgr2=0x7fa1840ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 shutdown_connections 2023-11-24T08:09:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.465+0000 7fa18ab7c640 1 -- 172.21.15.107:0/3093008716 wait complete. 2023-11-24T08:09:05.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:09:05.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.696+0000 7f5a2f9be640 1 Processor -- start 2023-11-24T08:09:05.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.696+0000 7f5a2f9be640 1 -- start start 2023-11-24T08:09:05.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2f9be640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a280cf450 0x7f5a280cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2f9be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a280cece0 con 0x7f5a280cdfb0 2023-11-24T08:09:05.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2f9be640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2806d040 con 0x7f5a2815c7f0 2023-11-24T08:09:05.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2f9be640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a280cfe40 con 0x7f5a280cf450 2023-11-24T08:09:05.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2cf59640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a280cf450 0x7f5a280cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2cf59640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a280cf450 0x7f5a280cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33590/0 (socket says 172.21.15.107:33590) 2023-11-24T08:09:05.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2cf59640 1 -- 172.21.15.107:0/1599654199 learned_addr learned my addr 172.21.15.107:0/1599654199 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:05.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a2df5b640 1 --1- 172.21.15.107:0/1599654199 >> v1:172.21.15.107:6789/0 conn(0x7f5a280cdfb0 0x7f5a280ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52440/0 (socket says 172.21.15.107:52440) 2023-11-24T08:09:05.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.698+0000 7f5a1ffff640 1 -- 172.21.15.107:0/1599654199 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049377571 0 0) 0x7f5a280cece0 con 0x7f5a280cdfb0 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.699+0000 7f5a2cf59640 1 -- 172.21.15.107:0/1599654199 >> v1:172.21.15.120:6789/0 conn(0x7f5a2815c7f0 legacy=0x7f5a2806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.699+0000 7f5a2cf59640 1 -- 172.21.15.107:0/1599654199 >> v1:172.21.15.107:6789/0 conn(0x7f5a280cdfb0 legacy=0x7f5a280ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.699+0000 7f5a2cf59640 1 -- 172.21.15.107:0/1599654199 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2806d320 con 0x7f5a280cf450 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.700+0000 7f5a2cf59640 1 --2- 172.21.15.107:0/1599654199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a280cf900 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a4472689e89bbf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.700+0000 7f5a1ffff640 1 -- 172.21.15.107:0/1599654199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802f840 con 0x7f5a280cf450 2023-11-24T08:09:05.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.700+0000 7f5a1ffff640 1 -- 172.21.15.107:0/1599654199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1802f9a0 con 0x7f5a280cf450 2023-11-24T08:09:05.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.700+0000 7f5a1ffff640 1 -- 172.21.15.107:0/1599654199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802fd30 con 0x7f5a280cf450 2023-11-24T08:09:05.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 -- 172.21.15.107:0/1599654199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 msgr2=0x7f5a280cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/1599654199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a280cf900 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1802ba20 comp rx=0 tx=0).stop 2023-11-24T08:09:05.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 -- 172.21.15.107:0/1599654199 shutdown_connections 2023-11-24T08:09:05.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/1599654199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a280cf900 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 -- 172.21.15.107:0/1599654199 >> 172.21.15.107:0/1599654199 conn(0x7f5a2806a920 msgr2=0x7f5a2806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:05.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 -- 172.21.15.107:0/1599654199 shutdown_connections 2023-11-24T08:09:05.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.701+0000 7f5a2f9be640 1 -- 172.21.15.107:0/1599654199 wait complete. 2023-11-24T08:09:05.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.703+0000 7f5a2f9be640 1 Processor -- start 2023-11-24T08:09:05.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.703+0000 7f5a2f9be640 1 -- start start 2023-11-24T08:09:05.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a280cdfb0 0x7f5a281532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a2815c7f0 0x7f5a28154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28150510 con 0x7f5a280cf450 2023-11-24T08:09:05.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a281507f0 con 0x7f5a2815c7f0 2023-11-24T08:09:05.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2f9be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28150ad0 con 0x7f5a280cdfb0 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33602/0 (socket says 172.21.15.107:33602) 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 -- 172.21.15.107:0/4037225405 learned_addr learned my addr 172.21.15.107:0/4037225405 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 -- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a280cdfb0 msgr2=0x7f5a281532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a280cdfb0 0x7f5a281532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 -- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a2815c7f0 msgr2=0x7f5a28154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a2815c7f0 0x7f5a28154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.704+0000 7f5a2cf59640 1 -- 172.21.15.107:0/4037225405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180094d0 con 0x7f5a280cf450 2023-11-24T08:09:05.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a2cf59640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802f7f0 con 0x7f5a280cf450 2023-11-24T08:09:05.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a18041400 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a28150db0 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180417b0 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.705+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2814d630 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.706+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2814d880 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.707+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f5a18041910 con 0x7f5a280cf450 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.708+0000 7f5a1dffb640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 0x7f5a0c06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:05.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.708+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a1803c070 con 0x7f5a280cf450 2023-11-24T08:09:05.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.708+0000 7f5a2d75a640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 0x7f5a0c06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:05.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.709+0000 7f5a2d75a640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 0x7f5a0c06d8d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000f60 tx=0x7f5a240079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:05.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.713+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a18077850 con 0x7f5a280cf450 2023-11-24T08:09:05.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.907+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a281596d0 con 0x7f5a280cf450 2023-11-24T08:09:05.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.910+0000 7f5a1dffb640 1 -- 172.21.15.107:0/4037225405 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f5a1807ae70 con 0x7f5a280cf450 2023-11-24T08:09:05.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:05.913 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":64,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:05.801778+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 msgr2=0x7f5a0c06d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 0x7f5a0c06d8d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000f60 tx=0x7f5a240079b0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 msgr2=0x7f5a28143410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a18004730 comp rx=0 tx=0).stop 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 shutdown_connections 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a0c06b410 0x7f5a0c06d8d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a2815c7f0 0x7f5a28154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a280cf450 0x7f5a28143410 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 --2- 172.21.15.107:0/4037225405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a280cdfb0 0x7f5a281532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 >> 172.21.15.107:0/4037225405 conn(0x7f5a2806a920 msgr2=0x7f5a280b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.913+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 shutdown_connections 2023-11-24T08:09:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:05.914+0000 7f5a2f9be640 1 -- 172.21.15.107:0/4037225405 wait complete. 2023-11-24T08:09:05.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:09:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.148+0000 7f6e2e933640 1 Processor -- start 2023-11-24T08:09:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.148+0000 7f6e2e933640 1 -- start start 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2e933640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e28072160 0x7f6e2815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2e933640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2815ec90 con 0x7f6e2806de10 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2e933640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e280cd530 con 0x7f6e280c49f0 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2e933640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e280c4ed0 con 0x7f6e28072160 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2ced0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e2806de10 0x7f6e280705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52456/0 (socket says 172.21.15.107:52456) 2023-11-24T08:09:06.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.150+0000 7f6e2ced0640 1 -- 172.21.15.107:0/2463105925 learned_addr learned my addr 172.21.15.107:0/2463105925 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e27fff640 1 --2- 172.21.15.107:0/2463105925 >> v2:172.21.15.120:3300/0 conn(0x7f6e28072160 0x7f6e2815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338769028 0 0) 0x7f6e20001490 con 0x7f6e2806de10 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10005730 con 0x7f6e2806de10 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012083251 0 0) 0x7f6e10005730 con 0x7f6e2806de10 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 >> v2:172.21.15.120:3300/0 conn(0x7f6e28072160 msgr2=0x7f6e2815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 --2- 172.21.15.107:0/2463105925 >> v2:172.21.15.120:3300/0 conn(0x7f6e28072160 0x7f6e2815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 >> v1:172.21.15.120:6789/0 conn(0x7f6e280c49f0 legacy=0x7f6e2806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280c51b0 con 0x7f6e2806de10 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6e20001b80 con 0x7f6e2806de10 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 >> v1:172.21.15.107:6789/0 conn(0x7f6e2806de10 legacy=0x7f6e280705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 0x7f6e10005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e100069d0 0x7f6e10006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10007390 0x7f6e1000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280c4ed0 con 0x7f6e10005b80 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1000e230 con 0x7f6e100069d0 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1000e3f0 con 0x7f6e10007390 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.151+0000 7f6e27fff640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 0x7f6e10005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e27fff640 1 -- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10007390 msgr2=0x7f6e1000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e277fe640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e100069d0 0x7f6e10006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e27fff640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10007390 0x7f6e1000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e27fff640 1 -- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e100069d0 msgr2=0x7f6e10006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e27fff640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e100069d0 0x7f6e10006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e27fff640 1 -- 172.21.15.107:0/2463105925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e10005730 con 0x7f6e10005b80 2023-11-24T08:09:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.152+0000 7f6e2ced0640 1 -- 172.21.15.107:0/2463105925 reap_dead start 2023-11-24T08:09:06.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.153+0000 7f6e27fff640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 0x7f6e10005fa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009ac0 tx=0x7f6e10007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9a1c89572a0dfe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:06.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.153+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1802c6b0 con 0x7f6e10005b80 2023-11-24T08:09:06.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.153+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1802c810 con 0x7f6e10005b80 2023-11-24T08:09:06.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.153+0000 7f6e26ffd640 1 -- 172.21.15.107:0/2463105925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1802cba0 con 0x7f6e10005b80 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 -- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 msgr2=0x7f6e10005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 0x7f6e10005fa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009ac0 tx=0x7f6e10007a00 comp rx=0 tx=0).stop 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 -- 172.21.15.107:0/2463105925 shutdown_connections 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 --2- 172.21.15.107:0/2463105925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e10005b80 0x7f6e10005fa0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 -- 172.21.15.107:0/2463105925 >> 172.21.15.107:0/2463105925 conn(0x7f6e280b0c50 msgr2=0x7f6e280b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 -- 172.21.15.107:0/2463105925 shutdown_connections 2023-11-24T08:09:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.154+0000 7f6e2e933640 1 -- 172.21.15.107:0/2463105925 wait complete. 2023-11-24T08:09:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.156+0000 7f6e2e933640 1 Processor -- start 2023-11-24T08:09:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.156+0000 7f6e2e933640 1 -- start start 2023-11-24T08:09:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10005b80 0x7f6e281595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e28159be0 0x7f6e280c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280cbc90 con 0x7f6e280cb2d0 2023-11-24T08:09:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280c6f20 con 0x7f6e28159be0 2023-11-24T08:09:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2e933640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280b34d0 con 0x7f6e10005b80 2023-11-24T08:09:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2ced0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2ced0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33618/0 (socket says 172.21.15.107:33618) 2023-11-24T08:09:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e2ced0640 1 -- 172.21.15.107:0/1703608196 learned_addr learned my addr 172.21.15.107:0/1703608196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e277fe640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e28159be0 0x7f6e280c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.158+0000 7f6e27fff640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10005b80 0x7f6e281595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 -- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10005b80 msgr2=0x7f6e281595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10005b80 0x7f6e281595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 -- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e28159be0 msgr2=0x7f6e280c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e28159be0 0x7f6e280c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 -- 172.21.15.107:0/1703608196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280c7150 con 0x7f6e280cb2d0 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2ced0640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6e200012e0 tx=0x7f6e2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2000ceb0 con 0x7f6e280cb2d0 2023-11-24T08:09:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2000a610 con 0x7f6e280cb2d0 2023-11-24T08:09:06.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2000a9a0 con 0x7f6e280cb2d0 2023-11-24T08:09:06.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.159+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e280c7430 con 0x7f6e280cb2d0 2023-11-24T08:09:06.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.160+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e280c7710 con 0x7f6e280cb2d0 2023-11-24T08:09:06.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.161+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2804e8b0 con 0x7f6e280cb2d0 2023-11-24T08:09:06.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.162+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f6e2000ab00 con 0x7f6e280cb2d0 2023-11-24T08:09:06.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.163+0000 7f6e24ff9640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 0x7f6e1406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:06.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.163+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e20089690 con 0x7f6e280cb2d0 2023-11-24T08:09:06.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.163+0000 7f6e27fff640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 0x7f6e1406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:06.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.164+0000 7f6e27fff640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 0x7f6e1406d8a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e18036000 tx=0x7f6e18030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:06.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.167+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e20053f80 con 0x7f6e280cb2d0 2023-11-24T08:09:06.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:06.396+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6e2815ec90 con 0x7f6e280cb2d0 2023-11-24T08:09:07.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.064+0000 7f6e24ff9640 1 -- 172.21.15.107:0/1703608196 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6e2815ec90 con 0x7f6e280cb2d0 2023-11-24T08:09:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.067+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 msgr2=0x7f6e1406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.067+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 0x7f6e1406d8a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e18036000 tx=0x7f6e18030000 comp rx=0 tx=0).stop 2023-11-24T08:09:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.067+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 msgr2=0x7f6e280cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.067+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6e200012e0 tx=0x7f6e2000cc60 comp rx=0 tx=0).stop 2023-11-24T08:09:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 shutdown_connections 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e1406b3e0 0x7f6e1406d8a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e280cb2d0 0x7f6e280cb750 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e28159be0 0x7f6e280c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 --2- 172.21.15.107:0/1703608196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e10005b80 0x7f6e281595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 >> 172.21.15.107:0/1703608196 conn(0x7f6e280b0c50 msgr2=0x7f6e280ad430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 shutdown_connections 2023-11-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.068+0000 7f6e2e933640 1 -- 172.21.15.107:0/1703608196 wait complete. 2023-11-24T08:09:07.091 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:09:07.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:09:07.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.308+0000 7efccf4f1640 1 Processor -- start 2023-11-24T08:09:07.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.308+0000 7efccf4f1640 1 -- start start 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.309+0000 7efccf4f1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcc80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccf4f1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc806d430 con 0x7efcc815a1c0 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccf4f1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80b14e0 con 0x7efcc815ded0 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccf4f1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc815ea70 con 0x7efcc80b0b40 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efcce4ef640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efcc815a1c0 0x7efcc806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52468/0 (socket says 172.21.15.107:52468) 2023-11-24T08:09:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efcce4ef640 1 -- 172.21.15.107:0/856395096 learned_addr learned my addr 172.21.15.107:0/856395096 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccdcee640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcc80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055715958 0 0) 0x7efcc806d430 con 0x7efcc815a1c0 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.310+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb4005730 con 0x7efcc815a1c0 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193666147 0 0) 0x7efcb4005730 con 0x7efcc815a1c0 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 msgr2=0x7efcc80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcc80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 >> v1:172.21.15.120:6789/0 conn(0x7efcc815ded0 legacy=0x7efcc815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc815ec50 con 0x7efcc815a1c0 2023-11-24T08:09:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7efcc4001670 con 0x7efcc815a1c0 2023-11-24T08:09:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccdcee640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcc80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:09:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 >> v1:172.21.15.107:6789/0 conn(0x7efcc815a1c0 legacy=0x7efcc806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcb4005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb40069d0 0x7efcb4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcb4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc815ea70 con 0x7efcb4005bb0 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb400dc90 con 0x7efcb40069d0 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb400de50 con 0x7efcc80b0b40 2023-11-24T08:09:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efcce4ef640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcb4005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.311+0000 7efccdcee640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb40069d0 0x7efcb4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efcce4ef640 1 -- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 msgr2=0x7efcb4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efccecf0640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcb4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efcce4ef640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc80b0b40 0x7efcb4007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efcce4ef640 1 -- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb40069d0 msgr2=0x7efcb4006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efcce4ef640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb40069d0 0x7efcb4006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.312+0000 7efcce4ef640 1 -- 172.21.15.107:0/856395096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4005730 con 0x7efcb4005bb0 2023-11-24T08:09:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.313+0000 7efcce4ef640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcb4005fd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c240 tx=0x7efcb4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9705fa79b727090 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.313+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4005ad0 con 0x7efcb4005bb0 2023-11-24T08:09:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.313+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4005c30 con 0x7efcb4005bb0 2023-11-24T08:09:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.313+0000 7efccd4ed640 1 -- 172.21.15.107:0/856395096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc402c8b0 con 0x7efcb4005bb0 2023-11-24T08:09:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 -- 172.21.15.107:0/856395096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 msgr2=0x7efcb4005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 --2- 172.21.15.107:0/856395096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcb4005fd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c240 tx=0x7efcb4007a00 comp rx=0 tx=0).stop 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccecf0640 1 -- 172.21.15.107:0/856395096 reap_dead start 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 -- 172.21.15.107:0/856395096 shutdown_connections 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 -- 172.21.15.107:0/856395096 >> 172.21.15.107:0/856395096 conn(0x7efcc80cbb40 msgr2=0x7efcc80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 -- 172.21.15.107:0/856395096 shutdown_connections 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.314+0000 7efccf4f1640 1 -- 172.21.15.107:0/856395096 wait complete. 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.316+0000 7efccf4f1640 1 Processor -- start 2023-11-24T08:09:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.316+0000 7efccf4f1640 1 -- start start 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc814ec70 0x7efcc80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc80b1bf0 0x7efcc809bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80c7210 con 0x7efcb4005bb0 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc809ae20 con 0x7efcc80b1bf0 2023-11-24T08:09:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccf4f1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc809b100 con 0x7efcc814ec70 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33630/0 (socket says 172.21.15.107:33630) 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 -- 172.21.15.107:0/4140597087 learned_addr learned my addr 172.21.15.107:0/4140597087 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccecf0640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc80b1bf0 0x7efcc809bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 -- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc814ec70 msgr2=0x7efcc80c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efccdcee640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc814ec70 0x7efcc80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc814ec70 0x7efcc80c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 -- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc80b1bf0 msgr2=0x7efcc809bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc80b1bf0 0x7efcc809bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.317+0000 7efcce4ef640 1 -- 172.21.15.107:0/4140597087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc809b3e0 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efcce4ef640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7efcc402c450 tx=0x7efcc4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4039070 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc402dd50 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc403c040 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc809b6c0 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.318+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc814f0f0 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.319+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc80b5040 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.320+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7efcc4032070 con 0x7efcb4005bb0 2023-11-24T08:09:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.321+0000 7efcbeffd640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 0x7efcac06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.321+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcc40ab250 con 0x7efcb4005bb0 2023-11-24T08:09:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.321+0000 7efccdcee640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 0x7efcac06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.322+0000 7efccdcee640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 0x7efcac06d8a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efcc81f68e0 tx=0x7efcb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.325+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcc4075ae0 con 0x7efcb4005bb0 2023-11-24T08:09:07.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:07.558+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7efcc804e840 con 0x7efcb4005bb0 2023-11-24T08:09:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.076+0000 7efcbeffd640 1 -- 172.21.15.107:0/4140597087 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7efcc40460c0 con 0x7efcb4005bb0 2023-11-24T08:09:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 msgr2=0x7efcac06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 0x7efcac06d8a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efcc81f68e0 tx=0x7efcb8008040 comp rx=0 tx=0).stop 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 msgr2=0x7efcc809a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7efcc402c450 tx=0x7efcc4001a00 comp rx=0 tx=0).stop 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 shutdown_connections 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efcac06b3e0 0x7efcac06d8a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc80b1bf0 0x7efcc809bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcc814ec70 0x7efcc80c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 --2- 172.21.15.107:0/4140597087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb4005bb0 0x7efcc809a790 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.080+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 >> 172.21.15.107:0/4140597087 conn(0x7efcc80cbb40 msgr2=0x7efcc806d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.081+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 shutdown_connections 2023-11-24T08:09:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.081+0000 7efccf4f1640 1 -- 172.21.15.107:0/4140597087 wait complete. 2023-11-24T08:09:08.104 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:09:08.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:09:08.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.314+0000 7f88a2075640 1 Processor -- start 2023-11-24T08:09:08.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.315+0000 7f88a2075640 1 -- start start 2023-11-24T08:09:08.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f88a2075640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f88a2075640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f889c0cbf80 0x7f889c06c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f88a2075640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c0b3a40 con 0x7f889c159ce0 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f88a2075640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0cc410 con 0x7f889c0cbf80 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f88a2075640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0cdfa0 con 0x7f889c0b21d0 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.316+0000 7f889bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f889c0cbf80 0x7f889c06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f889c0cbf80 0x7f889c06c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37322/0 (socket says 172.21.15.107:37322) 2023-11-24T08:09:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 -- 172.21.15.107:0/263008882 learned_addr learned my addr 172.21.15.107:0/263008882 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889affd640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 -- 172.21.15.107:0/263008882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 msgr2=0x7f889c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 -- 172.21.15.107:0/263008882 >> v1:172.21.15.120:6789/0 conn(0x7f889c159ce0 legacy=0x7f889c06de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889bfff640 1 -- 172.21.15.107:0/263008882 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0cd9e0 con 0x7f889c0cbf80 2023-11-24T08:09:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.317+0000 7f889affd640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:09:08.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.318+0000 7f889bfff640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c0cbf80 0x7f889c06c560 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f889400bb00 tx=0x7f889402df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62febb9af7e622 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.319+0000 7f889a7fc640 1 -- 172.21.15.107:0/263008882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889402ece0 con 0x7f889c0cbf80 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.319+0000 7f889a7fc640 1 -- 172.21.15.107:0/263008882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8894032070 con 0x7f889c0cbf80 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.319+0000 7f889a7fc640 1 -- 172.21.15.107:0/263008882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894038620 con 0x7f889c0cbf80 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.319+0000 7f88a2075640 1 -- 172.21.15.107:0/263008882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c0cbf80 msgr2=0x7f889c06c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.319+0000 7f88a2075640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c0cbf80 0x7f889c06c560 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f889400bb00 tx=0x7f889402df30 comp rx=0 tx=0).stop 2023-11-24T08:09:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 -- 172.21.15.107:0/263008882 shutdown_connections 2023-11-24T08:09:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c0cbf80 0x7f889c06c560 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 --2- 172.21.15.107:0/263008882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 -- 172.21.15.107:0/263008882 >> 172.21.15.107:0/263008882 conn(0x7f889c06a920 msgr2=0x7f889c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 -- 172.21.15.107:0/263008882 shutdown_connections 2023-11-24T08:09:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.320+0000 7f88a2075640 1 -- 172.21.15.107:0/263008882 wait complete. 2023-11-24T08:09:08.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.322+0000 7f88a2075640 1 Processor -- start 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.322+0000 7f88a2075640 1 -- start start 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 0x7f889c148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0ad5b0 con 0x7f889c159ce0 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0ac9a0 con 0x7f889c0cbf80 2023-11-24T08:09:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f88a2075640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0acc80 con 0x7f889c0b21d0 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37330/0 (socket says 172.21.15.107:37330) 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 -- 172.21.15.107:0/527720193 learned_addr learned my addr 172.21.15.107:0/527720193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889b7fe640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889affd640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 0x7f889c148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 -- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 msgr2=0x7f889c0ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 -- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 msgr2=0x7f889c148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 0x7f889c148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889bfff640 1 -- 172.21.15.107:0/527720193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889400b760 con 0x7f889c159ce0 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889b7fe640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.323+0000 7f889affd640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 0x7f889c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f889bfff640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f889400bb00 tx=0x7f88940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894043470 con 0x7f889c159ce0 2023-11-24T08:09:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940435d0 con 0x7f889c159ce0 2023-11-24T08:09:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894043980 con 0x7f889c159ce0 2023-11-24T08:09:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c0acf60 con 0x7f889c159ce0 2023-11-24T08:09:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.324+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c148e60 con 0x7f889c159ce0 2023-11-24T08:09:08.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.325+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c0ad720 con 0x7f889c159ce0 2023-11-24T08:09:08.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.326+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f8894043ae0 con 0x7f889c159ce0 2023-11-24T08:09:08.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.327+0000 7f8893fff640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 0x7f887c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.327+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88940aed30 con 0x7f889c159ce0 2023-11-24T08:09:08.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.327+0000 7f889b7fe640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 0x7f887c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.328+0000 7f889b7fe640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 0x7f887c06d8a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f888c0012e0 tx=0x7f888c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.331+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8894079570 con 0x7f889c159ce0 2023-11-24T08:09:08.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.562+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f889c067470 con 0x7f889c159ce0 2023-11-24T08:09:08.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.563+0000 7f8893fff640 1 -- 172.21.15.107:0/527720193 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f8894047090 con 0x7f889c159ce0 2023-11-24T08:09:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:08.565 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:09:08.566 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-11-24T08:09:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 msgr2=0x7f887c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 0x7f887c06d8a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f888c0012e0 tx=0x7f888c009040 comp rx=0 tx=0).stop 2023-11-24T08:09:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 msgr2=0x7f889c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f889400bb00 tx=0x7f88940046c0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 shutdown_connections 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f887c06b3e0 0x7f887c06d8a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f889c159ce0 0x7f889c0abfc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c0cbf80 0x7f889c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.565+0000 7f88a2075640 1 --2- 172.21.15.107:0/527720193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f889c0b21d0 0x7f889c0ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.566+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 >> 172.21.15.107:0/527720193 conn(0x7f889c06a920 msgr2=0x7f889c1572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.566+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 shutdown_connections 2023-11-24T08:09:08.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.566+0000 7f88a2075640 1 -- 172.21.15.107:0/527720193 wait complete. 2023-11-24T08:09:08.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:09:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.797+0000 7f94cb91d640 1 Processor -- start 2023-11-24T08:09:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.797+0000 7f94cb91d640 1 -- start start 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.799+0000 7f94cb91d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.799+0000 7f94cb91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 0x7f94c415e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.799+0000 7f94cb91d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40ce0b0 con 0x7f94c4157c70 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.799+0000 7f94cb91d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c4161960 con 0x7f94c40cfb60 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.799+0000 7f94cb91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c415e670 con 0x7f94c415dc80 2023-11-24T08:09:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c8eb8640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c9eba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 0x7f94c415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c8eb8640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:52896/0 (socket says 172.21.15.107:52896) 2023-11-24T08:09:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c8eb8640 1 -- 172.21.15.107:0/3147123665 learned_addr learned my addr 172.21.15.107:0/3147123665 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c9eba640 1 -- 172.21.15.107:0/3147123665 >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 msgr2=0x7f94c4161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c9eba640 1 --2- 172.21.15.107:0/3147123665 >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c9eba640 1 -- 172.21.15.107:0/3147123665 >> v1:172.21.15.120:6789/0 conn(0x7f94c4157c70 legacy=0x7f94c40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.800+0000 7f94c9eba640 1 -- 172.21.15.107:0/3147123665 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c415e8d0 con 0x7f94c415dc80 2023-11-24T08:09:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.801+0000 7f94c8eb8640 1 --2- 172.21.15.107:0/3147123665 >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:09:08.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.802+0000 7f94c9eba640 1 --2- 172.21.15.107:0/3147123665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 0x7f94c415e130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009d90 tx=0x7f94bc02be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62b975a570f85a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.803+0000 7f94bbfff640 1 -- 172.21.15.107:0/3147123665 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02cbd0 con 0x7f94c415dc80 2023-11-24T08:09:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.803+0000 7f94bbfff640 1 -- 172.21.15.107:0/3147123665 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc02cd30 con 0x7f94c415dc80 2023-11-24T08:09:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.803+0000 7f94bbfff640 1 -- 172.21.15.107:0/3147123665 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02cea0 con 0x7f94c415dc80 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 -- 172.21.15.107:0/3147123665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 msgr2=0x7f94c415e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 --2- 172.21.15.107:0/3147123665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 0x7f94c415e130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009d90 tx=0x7f94bc02be20 comp rx=0 tx=0).stop 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 -- 172.21.15.107:0/3147123665 shutdown_connections 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 --2- 172.21.15.107:0/3147123665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c415dc80 0x7f94c415e130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 --2- 172.21.15.107:0/3147123665 >> v2:172.21.15.120:3300/0 conn(0x7f94c40cfb60 0x7f94c4161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 -- 172.21.15.107:0/3147123665 >> 172.21.15.107:0/3147123665 conn(0x7f94c406a920 msgr2=0x7f94c406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 -- 172.21.15.107:0/3147123665 shutdown_connections 2023-11-24T08:09:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.804+0000 7f94cb91d640 1 -- 172.21.15.107:0/3147123665 wait complete. 2023-11-24T08:09:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.806+0000 7f94cb91d640 1 Processor -- start 2023-11-24T08:09:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.806+0000 7f94cb91d640 1 -- start start 2023-11-24T08:09:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.807+0000 7f94cb91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c40cfb60 0x7f94c414a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94cb91d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94cb91d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c415dc80 0x7f94c40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94cb91d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41f7380 con 0x7f94c4157c70 2023-11-24T08:09:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94cb91d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41f77a0 con 0x7f94c415dc80 2023-11-24T08:09:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94cb91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41f6a00 con 0x7f94c40cfb60 2023-11-24T08:09:08.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37340/0 (socket says 172.21.15.107:37340) 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 -- 172.21.15.107:0/2794873320 learned_addr learned my addr 172.21.15.107:0/2794873320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c9eba640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c415dc80 0x7f94c40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c96b9640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c40cfb60 0x7f94c414a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 -- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c40cfb60 msgr2=0x7f94c414a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c40cfb60 0x7f94c414a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 -- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c415dc80 msgr2=0x7f94c40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c415dc80 0x7f94c40712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.808+0000 7f94c8eb8640 1 -- 172.21.15.107:0/2794873320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b40094d0 con 0x7f94c4157c70 2023-11-24T08:09:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94c8eb8640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f94b400e9b0 tx=0x7f94b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400c9a0 con 0x7f94c4157c70 2023-11-24T08:09:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b400cb00 con 0x7f94c4157c70 2023-11-24T08:09:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400ce70 con 0x7f94c4157c70 2023-11-24T08:09:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc009a70 con 0x7f94c4157c70 2023-11-24T08:09:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.809+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c41f6fa0 con 0x7f94c4157c70 2023-11-24T08:09:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.810+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c4067470 con 0x7f94c4157c70 2023-11-24T08:09:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.811+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f94b4013470 con 0x7f94c4157c70 2023-11-24T08:09:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.812+0000 7f94b9ffb640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 0x7f94a806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.812+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94b408a600 con 0x7f94c4157c70 2023-11-24T08:09:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.812+0000 7f94c96b9640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 0x7f94a806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.813+0000 7f94c96b9640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 0x7f94a806d8a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f94c0001640 tx=0x7f94c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:08.816+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94b4054e90 con 0x7f94c4157c70 2023-11-24T08:09:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.006+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94c40726a0 con 0x7f94c4157c70 2023-11-24T08:09:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.008+0000 7f94b9ffb640 1 -- 172.21.15.107:0/2794873320 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f94b40584b0 con 0x7f94c4157c70 2023-11-24T08:09:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:09.011 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":64,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:05.801778+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:09:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 msgr2=0x7f94a806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 0x7f94a806d8a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f94c0001640 tx=0x7f94c0009040 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 msgr2=0x7f94c4070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f94b400e9b0 tx=0x7f94b400ee70 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 shutdown_connections 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94a806b3e0 0x7f94a806d8a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c415dc80 0x7f94c40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c4157c70 0x7f94c4070d80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 --2- 172.21.15.107:0/2794873320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c40cfb60 0x7f94c414a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 >> 172.21.15.107:0/2794873320 conn(0x7f94c406a920 msgr2=0x7f94c40b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 shutdown_connections 2023-11-24T08:09:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.010+0000 7f94cb91d640 1 -- 172.21.15.107:0/2794873320 wait complete. 2023-11-24T08:09:09.032 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:09:09.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:09:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.232+0000 7f3ba0802640 1 Processor -- start 2023-11-24T08:09:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.232+0000 7f3ba0802640 1 -- start start 2023-11-24T08:09:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.233+0000 7f3ba0802640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b9c0b2250 0x7f3b9c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.233+0000 7f3ba0802640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.233+0000 7f3ba0802640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c15ab80 con 0x7f3b9c0cd470 2023-11-24T08:09:09.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.233+0000 7f3ba0802640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c15a990 con 0x7f3b9c0b2250 2023-11-24T08:09:09.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.234+0000 7f3ba0802640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c15e680 con 0x7f3b9c0b2630 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.234+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.234+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:50562/0 (socket says 172.21.15.107:50562) 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.234+0000 7f3b93fff640 1 -- 172.21.15.107:0/3800584612 learned_addr learned my addr 172.21.15.107:0/3800584612 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.234+0000 7f3b9a59c640 1 --2- 172.21.15.107:0/3800584612 >> v2:172.21.15.120:3300/0 conn(0x7f3b9c0b2250 0x7f3b9c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.235+0000 7f3b93fff640 1 -- 172.21.15.107:0/3800584612 >> v2:172.21.15.120:3300/0 conn(0x7f3b9c0b2250 msgr2=0x7f3b9c15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.235+0000 7f3b93fff640 1 --2- 172.21.15.107:0/3800584612 >> v2:172.21.15.120:3300/0 conn(0x7f3b9c0b2250 0x7f3b9c15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.235+0000 7f3b93fff640 1 -- 172.21.15.107:0/3800584612 >> v1:172.21.15.120:6789/0 conn(0x7f3b9c0cd470 legacy=0x7f3b9c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.235+0000 7f3b93fff640 1 -- 172.21.15.107:0/3800584612 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c15e960 con 0x7f3b9c0b2630 2023-11-24T08:09:09.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.236+0000 7f3b93fff640 1 --2- 172.21.15.107:0/3800584612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009870 tx=0x7f3b8402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f4d8c267b22fadc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3b99d9b640 1 -- 172.21.15.107:0/3800584612 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84030840 con 0x7f3b9c0b2630 2023-11-24T08:09:09.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3b99d9b640 1 -- 172.21.15.107:0/3800584612 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b840309a0 con 0x7f3b9c0b2630 2023-11-24T08:09:09.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3b99d9b640 1 -- 172.21.15.107:0/3800584612 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84030d30 con 0x7f3b9c0b2630 2023-11-24T08:09:09.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 -- 172.21.15.107:0/3800584612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 msgr2=0x7f3b9c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 --2- 172.21.15.107:0/3800584612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009870 tx=0x7f3b8402b980 comp rx=0 tx=0).stop 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 -- 172.21.15.107:0/3800584612 shutdown_connections 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 --2- 172.21.15.107:0/3800584612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2630 0x7f3b9c0ccf30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 --2- 172.21.15.107:0/3800584612 >> v2:172.21.15.120:3300/0 conn(0x7f3b9c0b2250 0x7f3b9c15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 -- 172.21.15.107:0/3800584612 >> 172.21.15.107:0/3800584612 conn(0x7f3b9c06a920 msgr2=0x7f3b9c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.237+0000 7f3ba0802640 1 -- 172.21.15.107:0/3800584612 shutdown_connections 2023-11-24T08:09:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.238+0000 7f3ba0802640 1 -- 172.21.15.107:0/3800584612 wait complete. 2023-11-24T08:09:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.239+0000 7f3ba0802640 1 Processor -- start 2023-11-24T08:09:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.239+0000 7f3ba0802640 1 -- start start 2023-11-24T08:09:09.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2250 0x7f3b9c14e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0b2630 0x7f3b9c0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c8cd0 con 0x7f3b9c0cd470 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c8fb0 con 0x7f3b9c0b2630 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3ba0802640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c9290 con 0x7f3b9c0b2250 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.240+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37344/0 (socket says 172.21.15.107:37344) 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 -- 172.21.15.107:0/2030801240 learned_addr learned my addr 172.21.15.107:0/2030801240 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9a59c640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2250 0x7f3b9c14e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b93fff640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0b2630 0x7f3b9c0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 -- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2250 msgr2=0x7f3b9c14e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2250 0x7f3b9c14e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 -- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0b2630 msgr2=0x7f3b9c0c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0b2630 0x7f3b9c0c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 -- 172.21.15.107:0/2030801240 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840094d0 con 0x7f3b9c0cd470 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b9ad9d640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c730 tx=0x7f3b9400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94006d50 con 0x7f3b9c0cd470 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94003970 con 0x7f3b9c0cd470 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c0c9570 con 0x7f3b9c0cd470 2023-11-24T08:09:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.241+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94003d20 con 0x7f3b9c0cd470 2023-11-24T08:09:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.242+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c14eb90 con 0x7f3b9c0cd470 2023-11-24T08:09:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.243+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c0ce7b0 con 0x7f3b9c0cd470 2023-11-24T08:09:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.243+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f3b940050d0 con 0x7f3b9c0cd470 2023-11-24T08:09:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.244+0000 7f3b92ffd640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 0x7f3b7c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.244+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b94089b00 con 0x7f3b9c0cd470 2023-11-24T08:09:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.245+0000 7f3b9a59c640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 0x7f3b7c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.245+0000 7f3b9a59c640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 0x7f3b7c06d8a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c09d310 tx=0x7f3b8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.249+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b94054390 con 0x7f3b9c0cd470 2023-11-24T08:09:09.420 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:09:09.418+0000 7f8244184640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:09:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.424+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b9c0b2a70 con 0x7f3b7c06b3e0 2023-11-24T08:09:09.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.436+0000 7f3b92ffd640 1 -- 172.21.15.107:0/2030801240 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274756 (secure 0 0 0) 0x7f3b8c0119a0 con 0x7f3b7c06b3e0 2023-11-24T08:09:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:09:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.442+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 msgr2=0x7f3b7c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.442+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 0x7f3b7c06d8a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c09d310 tx=0x7f3b8c008040 comp rx=0 tx=0).stop 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.442+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 msgr2=0x7f3b9c071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.442+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c730 tx=0x7f3b9400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 shutdown_connections 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b7c06b3e0 0x7f3b7c06d8a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b9c0cd470 0x7f3b9c071610 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0b2630 0x7f3b9c0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 --2- 172.21.15.107:0/2030801240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0b2250 0x7f3b9c14e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 >> 172.21.15.107:0/2030801240 conn(0x7f3b9c06a920 msgr2=0x7f3b9c158590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 shutdown_connections 2023-11-24T08:09:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.443+0000 7f3ba0802640 1 -- 172.21.15.107:0/2030801240 wait complete. 2023-11-24T08:09:09.463 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2023-11-24T08:09:07.509217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":256312,"kb_used_data":6864,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718408,"statfs":{"total":773094113280,"available":772831649792,"internally_reserved":0,"allocated":7028736,"data_stored":3621430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":255315518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.487748"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.032240+0000","last_change":"2023-11-24T08:09:03.032240+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.032240+0000","last_undegraded":"2023-11-24T08:09:03.032240+0000","last_fullsized":"2023-11-24T08:09:03.032240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.030232+0000","last_change":"2023-11-24T08:09:03.030232+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.030232+0000","last_undegraded":"2023-11-24T08:09:03.030232+0000","last_fullsized":"2023-11-24T08:09:03.030232+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.031073+0000","last_change":"2023-11-24T08:09:03.031073+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.031073+0000","last_undegraded":"2023-11-24T08:09:03.031073+0000","last_fullsized":"2023-11-24T08:09:03.031073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.036098+0000","last_change":"2023-11-24T08:09:03.036098+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.036098+0000","last_undegraded":"2023-11-24T08:09:03.036098+0000","last_fullsized":"2023-11-24T08:09:03.036098+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.026098+0000","last_change":"2023-11-24T08:09:03.026098+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.026098+0000","last_undegraded":"2023-11-24T08:09:03.026098+0000","last_fullsized":"2023-11-24T08:09:03.026098+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029470+0000","last_change":"2023-11-24T08:09:03.029470+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029470+0000","last_undegraded":"2023-11-24T08:09:03.029470+0000","last_fullsized":"2023-11-24T08:09:03.029470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.036209+0000","last_change":"2023-11-24T08:09:03.036209+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.036209+0000","last_undegraded":"2023-11-24T08:09:03.036209+0000","last_fullsized":"2023-11-24T08:09:03.036209+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.026145+0000","last_change":"2023-11-24T08:09:03.026145+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.026145+0000","last_undegraded":"2023-11-24T08:09:03.026145+0000","last_fullsized":"2023-11-24T08:09:03.026145+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.041208+0000","last_change":"2023-11-24T08:09:03.041208+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.041208+0000","last_undegraded":"2023-11-24T08:09:03.041208+0000","last_fullsized":"2023-11-24T08:09:03.041208+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029242+0000","last_change":"2023-11-24T08:09:03.029242+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029242+0000","last_undegraded":"2023-11-24T08:09:03.029242+0000","last_fullsized":"2023-11-24T08:09:03.029242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.040072+0000","last_change":"2023-11-24T08:09:03.040072+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.040072+0000","last_undegraded":"2023-11-24T08:09:03.040072+0000","last_fullsized":"2023-11-24T08:09:03.040072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.026345+0000","last_change":"2023-11-24T08:09:03.026345+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.026345+0000","last_undegraded":"2023-11-24T08:09:03.026345+0000","last_fullsized":"2023-11-24T08:09:03.026345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.030246+0000","last_change":"2023-11-24T08:09:03.030246+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.030246+0000","last_undegraded":"2023-11-24T08:09:03.030246+0000","last_fullsized":"2023-11-24T08:09:03.030246+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.036502+0000","last_change":"2023-11-24T08:09:03.036502+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.036502+0000","last_undegraded":"2023-11-24T08:09:03.036502+0000","last_fullsized":"2023-11-24T08:09:03.036502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034977+0000","last_change":"2023-11-24T08:09:03.034977+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034977+0000","last_undegraded":"2023-11-24T08:09:03.034977+0000","last_fullsized":"2023-11-24T08:09:03.034977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.037733+0000","last_change":"2023-11-24T08:09:03.037733+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.037733+0000","last_undegraded":"2023-11-24T08:09:03.037733+0000","last_fullsized":"2023-11-24T08:09:03.037733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029430+0000","last_change":"2023-11-24T08:09:03.029430+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029430+0000","last_undegraded":"2023-11-24T08:09:03.029430+0000","last_fullsized":"2023-11-24T08:09:03.029430+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.032147+0000","last_change":"2023-11-24T08:09:03.032147+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.032147+0000","last_undegraded":"2023-11-24T08:09:03.032147+0000","last_fullsized":"2023-11-24T08:09:03.032147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034896+0000","last_change":"2023-11-24T08:09:03.034896+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034896+0000","last_undegraded":"2023-11-24T08:09:03.034896+0000","last_fullsized":"2023-11-24T08:09:03.034896+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029916+0000","last_change":"2023-11-24T08:09:03.029916+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029916+0000","last_undegraded":"2023-11-24T08:09:03.029916+0000","last_fullsized":"2023-11-24T08:09:03.029916+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029266+0000","last_change":"2023-11-24T08:09:03.029266+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029266+0000","last_undegraded":"2023-11-24T08:09:03.029266+0000","last_fullsized":"2023-11-24T08:09:03.029266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034751+0000","last_change":"2023-11-24T08:09:03.034751+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034751+0000","last_undegraded":"2023-11-24T08:09:03.034751+0000","last_fullsized":"2023-11-24T08:09:03.034751+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034786+0000","last_change":"2023-11-24T08:09:03.034786+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034786+0000","last_undegraded":"2023-11-24T08:09:03.034786+0000","last_fullsized":"2023-11-24T08:09:03.034786+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034758+0000","last_change":"2023-11-24T08:09:03.034758+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034758+0000","last_undegraded":"2023-11-24T08:09:03.034758+0000","last_fullsized":"2023-11-24T08:09:03.034758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.031681+0000","last_change":"2023-11-24T08:09:03.031681+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.031681+0000","last_undegraded":"2023-11-24T08:09:03.031681+0000","last_fullsized":"2023-11-24T08:09:03.031681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.037768+0000","last_change":"2023-11-24T08:09:03.037768+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.037768+0000","last_undegraded":"2023-11-24T08:09:03.037768+0000","last_fullsized":"2023-11-24T08:09:03.037768+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034989+0000","last_change":"2023-11-24T08:09:03.034989+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034989+0000","last_undegraded":"2023-11-24T08:09:03.034989+0000","last_fullsized":"2023-11-24T08:09:03.034989+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034753+0000","last_change":"2023-11-24T08:09:03.034753+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034753+0000","last_undegraded":"2023-11-24T08:09:03.034753+0000","last_fullsized":"2023-11-24T08:09:03.034753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.026079+0000","last_change":"2023-11-24T08:09:03.026079+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.026079+0000","last_undegraded":"2023-11-24T08:09:03.026079+0000","last_fullsized":"2023-11-24T08:09:03.026079+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.034931+0000","last_change":"2023-11-24T08:09:03.034931+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.034931+0000","last_undegraded":"2023-11-24T08:09:03.034931+0000","last_fullsized":"2023-11-24T08:09:03.034931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.029252+0000","last_change":"2023-11-24T08:09:03.029252+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.029252+0000","last_undegraded":"2023-11-24T08:09:03.029252+0000","last_fullsized":"2023-11-24T08:09:03.029252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:09:04.020956+0000","last_change":"2023-11-24T08:09:04.020956+0000","last_active":"2023-11-24T08:09:04.020956+0000","last_peered":"2023-11-24T08:09:04.020956+0000","last_clean":"2023-11-24T08:09:04.020956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:04.020956+0000","last_undegraded":"2023-11-24T08:09:04.020956+0000","last_fullsized":"2023-11-24T08:09:04.020956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":61,"state":"creating+peering","last_fresh":"2023-11-24T08:09:03.037288+0000","last_change":"2023-11-24T08:09:03.037288+0000","last_active":"2023-11-24T08:09:03.005499+0000","last_peered":"2023-11-24T08:09:03.005499+0000","last_clean":"2023-11-24T08:09:03.005499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:09:03.037288+0000","last_undegraded":"2023-11-24T08:09:03.037288+0000","last_fullsized":"2023-11-24T08:09:03.037288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":200,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-11-24T08:09:00.748814+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:09:00.748814+0000","last_peered":"2023-11-24T08:09:00.748814+0000","last_clean":"2023-11-24T08:09:00.748814+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:09:00.748814+0000","last_undegraded":"2023-11-24T08:09:00.748814+0000","last_fullsized":"2023-11-24T08:09:00.748814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":896909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0389999999999999,"15min":0.995},"min":{"1min":0.626,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2969999999999999,"5min":1.621,"15min":1.621},"last":1.319},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.115,"15min":1.079},"min":{"1min":0.86399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.403,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.2889999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1439999999999999,"15min":1.131},"min":{"1min":0.629,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6739999999999999,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.107},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2090000000000001,"15min":1.1830000000000001},"min":{"1min":0.86099999999999999,"5min":0.628,"15min":0.628},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.165}]},{"osd":2,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1830000000000001,"15min":1.228},"min":{"1min":0.67000000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.333,"15min":1.361},"min":{"1min":0.79500000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7729999999999999,"5min":4.633,"15min":4.633},"last":1.234}]},{"osd":3,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1719999999999999,"15min":1.135},"min":{"1min":0.68700000000000006,"5min":0.504,"15min":0.504},"max":{"1min":1.653,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3600000000000001,"15min":1.357},"min":{"1min":0.86499999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":2.206,"15min":2.206},"last":1.494}]},{"osd":4,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.3340000000000001},"min":{"1min":1.0349999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.831,"5min":2.851,"15min":2.851},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.341,"5min":1.3540000000000001,"15min":1.3320000000000001},"min":{"1min":0.85999999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0529999999999999,"5min":3.0510000000000002,"15min":3.0510000000000002},"last":1.3779999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.482,"15min":1.454},"min":{"1min":0.95799999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8779999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.5249999999999999,"15min":1.4850000000000001},"min":{"1min":1.327,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.9410000000000001,"5min":2.629,"15min":2.629},"last":1.0800000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5900000000000001,"15min":1.542},"min":{"1min":1.272,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0179999999999998,"5min":2.746,"15min":2.746},"last":1.351},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4930000000000001,"15min":1.4410000000000001},"min":{"1min":1.0760000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8540000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4550000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":733184,"data_stored":306541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.093,"15min":1.0940000000000001},"min":{"1min":0.69199999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.534,"5min":1.726,"15min":1.726},"last":1.95},{"interface":"front","average":{"1min":1.044,"5min":1.083,"15min":1.109},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.659,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.8899999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2030000000000001,"15min":1.2749999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5009999999999999,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.542},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.76900000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.694,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.4019999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.119,"15min":1.133},"min":{"1min":0.752,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.601,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5409999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.833}]},{"osd":3,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.369,"15min":1.409},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0720000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3200000000000001,"15min":1.351},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8149999999999999,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.3120000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.476,"15min":1.5720000000000001},"min":{"1min":0.86499999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.629,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":2.1059999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.5429999999999999,"15min":1.5820000000000001},"min":{"1min":1.177,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.9790000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":2.242}]},{"osd":5,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.506,"15min":1.5840000000000001},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.2250000000000001,"5min":2.629,"15min":2.629},"last":1.722},{"interface":"front","average":{"1min":1.516,"5min":1.637,"15min":1.7190000000000001},"min":{"1min":1.274,"5min":1.022,"15min":1.022},"max":{"1min":1.7390000000000001,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":2.4249999999999998}]},{"osd":7,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.671,"15min":1.7549999999999999},"min":{"1min":1.2430000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1850000000000001,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.681},{"interface":"front","average":{"1min":1.556,"5min":1.649,"15min":1.7130000000000001},"min":{"1min":1.1519999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.29,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":2.0569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":733184,"data_stored":306541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0680000000000001,"15min":1.141},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6339999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.012},{"interface":"front","average":{"1min":1.077,"5min":1.1519999999999999,"15min":1.194},"min":{"1min":0.65600000000000003,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.597,"5min":2.198,"15min":2.198},"last":1.407}]},{"osd":1,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1220000000000001,"15min":1.179},"min":{"1min":0.71499999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.579,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1339999999999999,"15min":1.1399999999999999},"min":{"1min":0.72599999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.524,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.3680000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2050000000000001,"15min":1.2170000000000001},"min":{"1min":0.69499999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.169,"15min":1.181},"min":{"1min":0.69999999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.7370000000000001,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1779999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.1679999999999999,"15min":1.202},"min":{"1min":0.76800000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.282,"15min":1.349},"min":{"1min":0.79900000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.851,"5min":1.99,"15min":1.99},"last":1.5089999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3180000000000001,"15min":1.345},"min":{"1min":0.98899999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.296},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4990000000000001,"15min":1.569},"min":{"1min":1.004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.0579999999999998,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.603}]},{"osd":6,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3979999999999999,"15min":1.482},"min":{"1min":0.83699999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.1240000000000001,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.419,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9179999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.4810000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5429999999999999,"15min":1.627},"min":{"1min":0.92400000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.9950000000000001,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.45},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5880000000000001,"15min":1.6220000000000001},"min":{"1min":1.256,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.028,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.8779999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":720896,"data_stored":300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.115,"15min":1.167},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6890000000000001,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.085,"15min":1.0960000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.637,"5min":2.222,"15min":2.222},"last":1.0309999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.1699999999999999,"15min":1.169},"min":{"1min":0.93600000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6639999999999999,"5min":1.74,"15min":1.74},"last":1.486},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1890000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5609999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.258}]},{"osd":2,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.113,"15min":1.1830000000000001},"min":{"1min":0.72499999999999998,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4730000000000001,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3129999999999999,"15min":1.397},"min":{"1min":0.67700000000000005,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.649,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.292}]},{"osd":3,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.391,"15min":1.4570000000000001},"min":{"1min":0.79800000000000004,"5min":0.751,"15min":0.751},"max":{"1min":2.121,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.288,"15min":1.3400000000000001},"min":{"1min":0.89800000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.5229999999999999,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.1579999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5049999999999999,"15min":1.53},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.486,"15min":1.5680000000000001},"min":{"1min":0.88500000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.8640000000000001,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3620000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.355,"15min":1.488},"min":{"1min":0.95299999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4610000000000001,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.595,"15min":1.6919999999999999},"min":{"1min":1.2629999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.986,"5min":3.391,"15min":3.391},"last":1.601}]},{"osd":7,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5629999999999999,"15min":1.661},"min":{"1min":1.016,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.752,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4910000000000001,"15min":1.635},"min":{"1min":1.147,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.752,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.512}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":733184,"data_stored":306541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":0.82999999999999996},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2749999999999999,"5min":10.557,"15min":10.557},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.92600000000000005,"15min":0.91000000000000003},"min":{"1min":0.53100000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2509999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.79200000000000004,"15min":0.84099999999999997},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.2070000000000001,"5min":1.532,"15min":1.532},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.0549999999999999,"15min":1.0029999999999999},"min":{"1min":0.45700000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.28,"5min":13.090999999999999,"15min":13.090999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.96499999999999997,"15min":0.97099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2030000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.624},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.1419999999999999,"15min":1.0660000000000001},"min":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.343,"5min":13.460000000000001,"15min":13.460000000000001},"last":0.94099999999999995}]},{"osd":4,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.3080000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.623,"5min":4.367,"15min":4.367},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.2629999999999999,"15min":1.345},"min":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.607,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.0049999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.3660000000000001,"15min":1.3200000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5660000000000001,"5min":13.231,"15min":13.231},"last":1.155},{"interface":"front","average":{"1min":1.097,"5min":1.331,"15min":1.341},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.591,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":0.876}]},{"osd":6,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.4339999999999999,"15min":1.401},"min":{"1min":0.751,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.825,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.226},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.288,"15min":1.3700000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.7030000000000001,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.97599999999999998}]},{"osd":7,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.4059999999999999,"15min":1.454},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6779999999999999,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":1.107},{"interface":"front","average":{"1min":1.157,"5min":1.365,"15min":1.4910000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.764,"5min":4.032,"15min":4.032},"last":1.026}]}]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":733184,"data_stored":306541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.88300000000000001,"15min":0.94099999999999995},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.389,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.93200000000000005,"15min":1.0369999999999999},"min":{"1min":0.58399999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4370000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.80200000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0309999999999999,"15min":1.165},"min":{"1min":0.498,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3939999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0580000000000001,"15min":1.161},"min":{"1min":0.505,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.788,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.2030000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.92600000000000005,"15min":0.96999999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4890000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.1819999999999999,"15min":1.3069999999999999},"min":{"1min":0.56299999999999994,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.946,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.333}]},{"osd":4,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3520000000000001,"15min":1.4359999999999999},"min":{"1min":0.88300000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.893,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.367},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.361,"15min":1.425},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.857,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":1.1499999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.381,"15min":1.4850000000000001},"min":{"1min":0.80300000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0230000000000001,"5min":2.508,"15min":2.508},"last":1.246},{"interface":"front","average":{"1min":1.288,"5min":1.3660000000000001,"15min":1.4430000000000001},"min":{"1min":0.92500000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.7749999999999999,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.52}]},{"osd":6,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.302,"15min":1.3680000000000001},"min":{"1min":0.745,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.708,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.298},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.45,"15min":1.552},"min":{"1min":0.92400000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.1389999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.425}]},{"osd":7,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.5449999999999999},"min":{"1min":0.84299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.885,"5min":2.859,"15min":2.859},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4330000000000001,"15min":1.5209999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9410000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.581}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":720896,"data_stored":300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.42399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.2909999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84699999999999998,"15min":0.82599999999999996},"min":{"1min":0.53900000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.321,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.0289999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74099999999999999,"15min":0.69299999999999995},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2989999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.093},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88400000000000001,"15min":0.86899999999999999},"min":{"1min":0.438,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.90900000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.95699999999999996,"15min":0.94199999999999995},"min":{"1min":0.54500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3480000000000001,"5min":1.675,"15min":1.675},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.89000000000000001,"15min":0.873},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.425,"5min":1.464,"15min":1.464},"last":1.2969999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.256,"15min":1.23},"min":{"1min":0.94299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.5820000000000001,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.2509999999999999,"15min":1.252},"min":{"1min":0.84999999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.143}]},{"osd":5,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2370000000000001,"15min":1.1859999999999999},"min":{"1min":0.748,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.645,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.474},{"interface":"front","average":{"1min":1.228,"5min":1.216,"15min":1.1719999999999999},"min":{"1min":0.84599999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.742,"5min":3.286,"15min":3.286},"last":1.2689999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2330000000000001,"15min":1.2030000000000001},"min":{"1min":0.79500000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.897,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.421},{"interface":"front","average":{"1min":1.321,"5min":1.234,"15min":1.181},"min":{"1min":0.89200000000000002,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.847,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.401}]},{"osd":7,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2909999999999999,"15min":1.2869999999999999},"min":{"1min":0.82099999999999995,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.79,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.3049999999999999,"15min":1.252},"min":{"1min":0.871,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.972,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.3360000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":896909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.81299999999999994,"15min":0.90200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.996,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.93700000000000006,"15min":1.04},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.3029999999999999,"5min":1.885,"15min":1.885},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.04,"15min":1.0760000000000001},"min":{"1min":0.52900000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.8460000000000001,"5min":1.865,"15min":1.865},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.96699999999999997},"min":{"1min":0.502,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.51,"5min":1.847,"15min":1.847},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.075,"15min":1.143},"min":{"1min":0.52400000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5109999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.96899999999999997,"15min":1.0800000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.3759999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.083}]},{"osd":4,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.379,"15min":1.446},"min":{"1min":0.83199999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":3.7610000000000001,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.014,"5min":1.179,"15min":1.27},"min":{"1min":0.74099999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.2929999999999999,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.88300000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.2370000000000001,"15min":1.345},"min":{"1min":0.71799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.3939999999999999,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.333,"15min":1.4730000000000001},"min":{"1min":0.80800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7689999999999999,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.0349999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.351,"15min":1.4630000000000001},"min":{"1min":0.77700000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.9039999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.238,"5min":1.365,"15min":1.4810000000000001},"min":{"1min":0.84699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.5609999999999999,"5min":2.738,"15min":2.738},"last":1.2350000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.456,"15min":1.552},"min":{"1min":0.93799999999999994,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0379999999999998,"5min":2.669,"15min":2.669},"last":1.282},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.427,"15min":1.544},"min":{"1min":0.80100000000000005,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.871,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.0600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:09:09.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:09:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.718+0000 7effa52ee640 1 Processor -- start 2023-11-24T08:09:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.718+0000 7effa52ee640 1 -- start start 2023-11-24T08:09:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.719+0000 7effa52ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7effa015c7f0 0x7effa006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.719+0000 7effa52ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7effa52ee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa00cece0 con 0x7effa00cdfb0 2023-11-24T08:09:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7effa52ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa0156010 con 0x7effa015c7f0 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7effa52ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa006ce30 con 0x7effa00cf450 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7effa015c7f0 0x7effa006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7effa015c7f0 0x7effa006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37346/0 (socket says 172.21.15.107:37346) 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 -- 172.21.15.107:0/4042090459 learned_addr learned my addr 172.21.15.107:0/4042090459 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9f7fe640 1 --1- 172.21.15.107:0/4042090459 >> v1:172.21.15.107:6789/0 conn(0x7effa00cdfb0 0x7effa00ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44372/0 (socket says 172.21.15.107:44372) 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9e7fc640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 -- 172.21.15.107:0/4042090459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 msgr2=0x7effa00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 -- 172.21.15.107:0/4042090459 >> v1:172.21.15.107:6789/0 conn(0x7effa00cdfb0 legacy=0x7effa00ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.720+0000 7eff9effd640 1 -- 172.21.15.107:0/4042090459 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa006d040 con 0x7effa015c7f0 2023-11-24T08:09:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.721+0000 7eff9effd640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa015c7f0 0x7effa006db20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7eff90009870 tx=0x7eff9002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b31b9b2ce84e6d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7eff9dffb640 1 -- 172.21.15.107:0/4042090459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90030840 con 0x7effa015c7f0 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7eff9dffb640 1 -- 172.21.15.107:0/4042090459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900309a0 con 0x7effa015c7f0 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7eff9dffb640 1 -- 172.21.15.107:0/4042090459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90030d30 con 0x7effa015c7f0 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 -- 172.21.15.107:0/4042090459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa015c7f0 msgr2=0x7effa006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa015c7f0 0x7effa006db20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7eff90009870 tx=0x7eff9002b980 comp rx=0 tx=0).stop 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 -- 172.21.15.107:0/4042090459 shutdown_connections 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 --2- 172.21.15.107:0/4042090459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa015c7f0 0x7effa006db20 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 -- 172.21.15.107:0/4042090459 >> 172.21.15.107:0/4042090459 conn(0x7effa006a920 msgr2=0x7effa006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 -- 172.21.15.107:0/4042090459 shutdown_connections 2023-11-24T08:09:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.722+0000 7effa52ee640 1 -- 172.21.15.107:0/4042090459 wait complete. 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.724+0000 7effa52ee640 1 Processor -- start 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.724+0000 7effa52ee640 1 -- start start 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa015c7f0 0x7effa00b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa00b36b0 con 0x7effa00cdfb0 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa01448f0 con 0x7effa015c7f0 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.725+0000 7effa52ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa00c88a0 con 0x7effa00cf450 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37348/0 (socket says 172.21.15.107:37348) 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 -- 172.21.15.107:0/3321944352 learned_addr learned my addr 172.21.15.107:0/3321944352 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 -- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 msgr2=0x7effa00c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 -- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa015c7f0 msgr2=0x7effa00b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa015c7f0 0x7effa00b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 -- 172.21.15.107:0/3321944352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900094d0 con 0x7effa00cdfb0 2023-11-24T08:09:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff9effd640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7eff90000c00 tx=0x7eff90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.726+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90030810 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.727+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90039d90 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.727+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa00b2aa0 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.727+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90043670 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.727+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa00b3060 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.728+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa0067470 con 0x7effa00cdfb0 2023-11-24T08:09:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.729+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7eff9004c3b0 con 0x7effa00cdfb0 2023-11-24T08:09:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.730+0000 7eff877fe640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 0x7eff8c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.730+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff900ac690 con 0x7effa00cdfb0 2023-11-24T08:09:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.730+0000 7eff9e7fc640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 0x7eff8c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.731+0000 7eff9e7fc640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 0x7eff8c06d8a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:09.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.735+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff90076db0 con 0x7effa00cdfb0 2023-11-24T08:09:09.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.914+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effa006e060 con 0x7eff8c06b3e0 2023-11-24T08:09:09.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.923+0000 7eff877fe640 1 -- 172.21.15.107:0/3321944352 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286998 (secure 0 0 0) 0x7eff9400ae20 con 0x7eff8c06b3e0 2023-11-24T08:09:09.928 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:09.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:09:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 msgr2=0x7eff8c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 0x7eff8c06d8a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94008040 comp rx=0 tx=0).stop 2023-11-24T08:09:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 msgr2=0x7effa01443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7eff90000c00 tx=0x7eff90004730 comp rx=0 tx=0).stop 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 shutdown_connections 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff8c06b3e0 0x7eff8c06d8a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa015c7f0 0x7effa00b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effa00cf450 0x7effa00c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 --2- 172.21.15.107:0/3321944352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effa00cdfb0 0x7effa01443b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 >> 172.21.15.107:0/3321944352 conn(0x7effa006a920 msgr2=0x7effa00af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 shutdown_connections 2023-11-24T08:09:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:09.930+0000 7effa52ee640 1 -- 172.21.15.107:0/3321944352 wait complete. 2023-11-24T08:09:09.951 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2023-11-24T08:09:09.509783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256684,"kb_used_data":7228,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718036,"statfs":{"total":773094113280,"available":772831268864,"internally_reserved":0,"allocated":7401472,"data_stored":3786320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":255315518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001650"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184815+0000","last_change":"2023-11-24T08:09:04.829156+0000","last_active":"2023-11-24T08:09:08.184815+0000","last_peered":"2023-11-24T08:09:08.184815+0000","last_clean":"2023-11-24T08:09:08.184815+0000","last_became_active":"2023-11-24T08:09:04.828882+0000","last_became_peered":"2023-11-24T08:09:04.828882+0000","last_unstale":"2023-11-24T08:09:08.184815+0000","last_undegraded":"2023-11-24T08:09:08.184815+0000","last_fullsized":"2023-11-24T08:09:08.184815+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:11.121857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811912+0000","last_change":"2023-11-24T08:09:04.831666+0000","last_active":"2023-11-24T08:09:05.811912+0000","last_peered":"2023-11-24T08:09:05.811912+0000","last_clean":"2023-11-24T08:09:05.811912+0000","last_became_active":"2023-11-24T08:09:04.831356+0000","last_became_peered":"2023-11-24T08:09:04.831356+0000","last_unstale":"2023-11-24T08:09:05.811912+0000","last_undegraded":"2023-11-24T08:09:05.811912+0000","last_fullsized":"2023-11-24T08:09:05.811912+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:50.100283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512003+0000","last_change":"2023-11-24T08:09:04.836911+0000","last_active":"2023-11-24T08:09:07.512003+0000","last_peered":"2023-11-24T08:09:07.512003+0000","last_clean":"2023-11-24T08:09:07.512003+0000","last_became_active":"2023-11-24T08:09:04.836621+0000","last_became_peered":"2023-11-24T08:09:04.836621+0000","last_unstale":"2023-11-24T08:09:07.512003+0000","last_undegraded":"2023-11-24T08:09:07.512003+0000","last_fullsized":"2023-11-24T08:09:07.512003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:18.478087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999478+0000","last_change":"2023-11-24T08:09:04.837270+0000","last_active":"2023-11-24T08:09:07.999478+0000","last_peered":"2023-11-24T08:09:07.999478+0000","last_clean":"2023-11-24T08:09:07.999478+0000","last_became_active":"2023-11-24T08:09:04.836011+0000","last_became_peered":"2023-11-24T08:09:04.836011+0000","last_unstale":"2023-11-24T08:09:07.999478+0000","last_undegraded":"2023-11-24T08:09:07.999478+0000","last_fullsized":"2023-11-24T08:09:07.999478+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:51:48.143041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184549+0000","last_change":"2023-11-24T08:09:04.839644+0000","last_active":"2023-11-24T08:09:08.184549+0000","last_peered":"2023-11-24T08:09:08.184549+0000","last_clean":"2023-11-24T08:09:08.184549+0000","last_became_active":"2023-11-24T08:09:04.838528+0000","last_became_peered":"2023-11-24T08:09:04.838528+0000","last_unstale":"2023-11-24T08:09:08.184549+0000","last_undegraded":"2023-11-24T08:09:08.184549+0000","last_fullsized":"2023-11-24T08:09:08.184549+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:55.004246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159345+0000","last_change":"2023-11-24T08:09:04.830213+0000","last_active":"2023-11-24T08:09:06.159345+0000","last_peered":"2023-11-24T08:09:06.159345+0000","last_clean":"2023-11-24T08:09:06.159345+0000","last_became_active":"2023-11-24T08:09:04.828453+0000","last_became_peered":"2023-11-24T08:09:04.828453+0000","last_unstale":"2023-11-24T08:09:06.159345+0000","last_undegraded":"2023-11-24T08:09:06.159345+0000","last_fullsized":"2023-11-24T08:09:06.159345+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:51:48.143041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.814242+0000","last_change":"2023-11-24T08:09:04.845614+0000","last_active":"2023-11-24T08:09:05.814242+0000","last_peered":"2023-11-24T08:09:05.814242+0000","last_clean":"2023-11-24T08:09:05.814242+0000","last_became_active":"2023-11-24T08:09:04.845271+0000","last_became_peered":"2023-11-24T08:09:04.845271+0000","last_unstale":"2023-11-24T08:09:05.814242+0000","last_undegraded":"2023-11-24T08:09:05.814242+0000","last_fullsized":"2023-11-24T08:09:05.814242+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:18.784238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811973+0000","last_change":"2023-11-24T08:09:04.844438+0000","last_active":"2023-11-24T08:09:05.811973+0000","last_peered":"2023-11-24T08:09:05.811973+0000","last_clean":"2023-11-24T08:09:05.811973+0000","last_became_active":"2023-11-24T08:09:04.843994+0000","last_became_peered":"2023-11-24T08:09:04.843994+0000","last_unstale":"2023-11-24T08:09:05.811973+0000","last_undegraded":"2023-11-24T08:09:05.811973+0000","last_fullsized":"2023-11-24T08:09:05.811973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:02.057726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.814710+0000","last_change":"2023-11-24T08:09:04.843812+0000","last_active":"2023-11-24T08:09:05.814710+0000","last_peered":"2023-11-24T08:09:05.814710+0000","last_clean":"2023-11-24T08:09:05.814710+0000","last_became_active":"2023-11-24T08:09:04.843484+0000","last_became_peered":"2023-11-24T08:09:04.843484+0000","last_unstale":"2023-11-24T08:09:05.814710+0000","last_undegraded":"2023-11-24T08:09:05.814710+0000","last_fullsized":"2023-11-24T08:09:05.814710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:35:16.782130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.837086+0000","last_change":"2023-11-24T08:09:04.833340+0000","last_active":"2023-11-24T08:09:04.837086+0000","last_peered":"2023-11-24T08:09:04.837086+0000","last_clean":"2023-11-24T08:09:04.837086+0000","last_became_active":"2023-11-24T08:09:04.832410+0000","last_became_peered":"2023-11-24T08:09:04.832410+0000","last_unstale":"2023-11-24T08:09:04.837086+0000","last_undegraded":"2023-11-24T08:09:04.837086+0000","last_fullsized":"2023-11-24T08:09:04.837086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:05.815438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.837649+0000","last_change":"2023-11-24T08:09:04.833746+0000","last_active":"2023-11-24T08:09:04.837649+0000","last_peered":"2023-11-24T08:09:04.837649+0000","last_clean":"2023-11-24T08:09:04.837649+0000","last_became_active":"2023-11-24T08:09:04.833092+0000","last_became_peered":"2023-11-24T08:09:04.833092+0000","last_unstale":"2023-11-24T08:09:04.837649+0000","last_undegraded":"2023-11-24T08:09:04.837649+0000","last_fullsized":"2023-11-24T08:09:04.837649+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:25.388211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999827+0000","last_change":"2023-11-24T08:09:04.837024+0000","last_active":"2023-11-24T08:09:07.999827+0000","last_peered":"2023-11-24T08:09:07.999827+0000","last_clean":"2023-11-24T08:09:07.999827+0000","last_became_active":"2023-11-24T08:09:04.835672+0000","last_became_peered":"2023-11-24T08:09:04.835672+0000","last_unstale":"2023-11-24T08:09:07.999827+0000","last_undegraded":"2023-11-24T08:09:07.999827+0000","last_fullsized":"2023-11-24T08:09:07.999827+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:44.721027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.609620+0000","last_change":"2023-11-24T08:09:04.829838+0000","last_active":"2023-11-24T08:09:07.609620+0000","last_peered":"2023-11-24T08:09:07.609620+0000","last_clean":"2023-11-24T08:09:07.609620+0000","last_became_active":"2023-11-24T08:09:04.829429+0000","last_became_peered":"2023-11-24T08:09:04.829429+0000","last_unstale":"2023-11-24T08:09:07.609620+0000","last_undegraded":"2023-11-24T08:09:07.609620+0000","last_fullsized":"2023-11-24T08:09:07.609620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:57:24.214198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.815015+0000","last_change":"2023-11-24T08:09:04.842112+0000","last_active":"2023-11-24T08:09:05.815015+0000","last_peered":"2023-11-24T08:09:05.815015+0000","last_clean":"2023-11-24T08:09:05.815015+0000","last_became_active":"2023-11-24T08:09:04.841832+0000","last_became_peered":"2023-11-24T08:09:04.841832+0000","last_unstale":"2023-11-24T08:09:05.815015+0000","last_undegraded":"2023-11-24T08:09:05.815015+0000","last_fullsized":"2023-11-24T08:09:05.815015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:25.885119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.814429+0000","last_change":"2023-11-24T08:09:04.842108+0000","last_active":"2023-11-24T08:09:05.814429+0000","last_peered":"2023-11-24T08:09:05.814429+0000","last_clean":"2023-11-24T08:09:05.814429+0000","last_became_active":"2023-11-24T08:09:04.841487+0000","last_became_peered":"2023-11-24T08:09:04.841487+0000","last_unstale":"2023-11-24T08:09:05.814429+0000","last_undegraded":"2023-11-24T08:09:05.814429+0000","last_fullsized":"2023-11-24T08:09:05.814429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:29.725948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159292+0000","last_change":"2023-11-24T08:09:04.834353+0000","last_active":"2023-11-24T08:09:06.159292+0000","last_peered":"2023-11-24T08:09:06.159292+0000","last_clean":"2023-11-24T08:09:06.159292+0000","last_became_active":"2023-11-24T08:09:04.832652+0000","last_became_peered":"2023-11-24T08:09:04.832652+0000","last_unstale":"2023-11-24T08:09:06.159292+0000","last_undegraded":"2023-11-24T08:09:06.159292+0000","last_fullsized":"2023-11-24T08:09:06.159292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:33.778568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.812632+0000","last_change":"2023-11-24T08:09:04.837289+0000","last_active":"2023-11-24T08:09:05.812632+0000","last_peered":"2023-11-24T08:09:05.812632+0000","last_clean":"2023-11-24T08:09:05.812632+0000","last_became_active":"2023-11-24T08:09:04.837004+0000","last_became_peered":"2023-11-24T08:09:04.837004+0000","last_unstale":"2023-11-24T08:09:05.812632+0000","last_undegraded":"2023-11-24T08:09:05.812632+0000","last_fullsized":"2023-11-24T08:09:05.812632+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:45:14.922420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512226+0000","last_change":"2023-11-24T08:09:04.829052+0000","last_active":"2023-11-24T08:09:07.512226+0000","last_peered":"2023-11-24T08:09:07.512226+0000","last_clean":"2023-11-24T08:09:07.512226+0000","last_became_active":"2023-11-24T08:09:04.827042+0000","last_became_peered":"2023-11-24T08:09:04.827042+0000","last_unstale":"2023-11-24T08:09:07.512226+0000","last_undegraded":"2023-11-24T08:09:07.512226+0000","last_fullsized":"2023-11-24T08:09:07.512226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:41.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184607+0000","last_change":"2023-11-24T08:09:04.831497+0000","last_active":"2023-11-24T08:09:08.184607+0000","last_peered":"2023-11-24T08:09:08.184607+0000","last_clean":"2023-11-24T08:09:08.184607+0000","last_became_active":"2023-11-24T08:09:04.831209+0000","last_became_peered":"2023-11-24T08:09:04.831209+0000","last_unstale":"2023-11-24T08:09:08.184607+0000","last_undegraded":"2023-11-24T08:09:08.184607+0000","last_fullsized":"2023-11-24T08:09:08.184607+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:23.851039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811511+0000","last_change":"2023-11-24T08:09:04.834155+0000","last_active":"2023-11-24T08:09:05.811511+0000","last_peered":"2023-11-24T08:09:05.811511+0000","last_clean":"2023-11-24T08:09:05.811511+0000","last_became_active":"2023-11-24T08:09:04.833914+0000","last_became_peered":"2023-11-24T08:09:04.833914+0000","last_unstale":"2023-11-24T08:09:05.811511+0000","last_undegraded":"2023-11-24T08:09:05.811511+0000","last_fullsized":"2023-11-24T08:09:05.811511+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:20:13.144821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.511506+0000","last_change":"2023-11-24T08:09:04.838203+0000","last_active":"2023-11-24T08:09:07.511506+0000","last_peered":"2023-11-24T08:09:07.511506+0000","last_clean":"2023-11-24T08:09:07.511506+0000","last_became_active":"2023-11-24T08:09:04.837780+0000","last_became_peered":"2023-11-24T08:09:04.837780+0000","last_unstale":"2023-11-24T08:09:07.511506+0000","last_undegraded":"2023-11-24T08:09:07.511506+0000","last_fullsized":"2023-11-24T08:09:07.511506+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:33.778568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.812105+0000","last_change":"2023-11-24T08:09:04.840429+0000","last_active":"2023-11-24T08:09:05.812105+0000","last_peered":"2023-11-24T08:09:05.812105+0000","last_clean":"2023-11-24T08:09:05.812105+0000","last_became_active":"2023-11-24T08:09:04.838991+0000","last_became_peered":"2023-11-24T08:09:04.838991+0000","last_unstale":"2023-11-24T08:09:05.812105+0000","last_undegraded":"2023-11-24T08:09:05.812105+0000","last_fullsized":"2023-11-24T08:09:05.812105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:34.222348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.511569+0000","last_change":"2023-11-24T08:09:04.832507+0000","last_active":"2023-11-24T08:09:07.511569+0000","last_peered":"2023-11-24T08:09:07.511569+0000","last_clean":"2023-11-24T08:09:07.511569+0000","last_became_active":"2023-11-24T08:09:04.831908+0000","last_became_peered":"2023-11-24T08:09:04.831908+0000","last_unstale":"2023-11-24T08:09:07.511569+0000","last_undegraded":"2023-11-24T08:09:07.511569+0000","last_fullsized":"2023-11-24T08:09:07.511569+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:51:48.143041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.609687+0000","last_change":"2023-11-24T08:09:04.838473+0000","last_active":"2023-11-24T08:09:07.609687+0000","last_peered":"2023-11-24T08:09:07.609687+0000","last_clean":"2023-11-24T08:09:07.609687+0000","last_became_active":"2023-11-24T08:09:04.836912+0000","last_became_peered":"2023-11-24T08:09:04.836912+0000","last_unstale":"2023-11-24T08:09:07.609687+0000","last_undegraded":"2023-11-24T08:09:07.609687+0000","last_fullsized":"2023-11-24T08:09:07.609687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:23:00.598962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184858+0000","last_change":"2023-11-24T08:09:04.829289+0000","last_active":"2023-11-24T08:09:08.184858+0000","last_peered":"2023-11-24T08:09:08.184858+0000","last_clean":"2023-11-24T08:09:08.184858+0000","last_became_active":"2023-11-24T08:09:04.828960+0000","last_became_peered":"2023-11-24T08:09:04.828960+0000","last_unstale":"2023-11-24T08:09:08.184858+0000","last_undegraded":"2023-11-24T08:09:08.184858+0000","last_fullsized":"2023-11-24T08:09:08.184858+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:13.856589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999423+0000","last_change":"2023-11-24T08:09:04.836905+0000","last_active":"2023-11-24T08:09:07.999423+0000","last_peered":"2023-11-24T08:09:07.999423+0000","last_clean":"2023-11-24T08:09:07.999423+0000","last_became_active":"2023-11-24T08:09:04.836559+0000","last_became_peered":"2023-11-24T08:09:04.836559+0000","last_unstale":"2023-11-24T08:09:07.999423+0000","last_undegraded":"2023-11-24T08:09:07.999423+0000","last_fullsized":"2023-11-24T08:09:07.999423+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:58.284845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159864+0000","last_change":"2023-11-24T08:09:04.834177+0000","last_active":"2023-11-24T08:09:06.159864+0000","last_peered":"2023-11-24T08:09:06.159864+0000","last_clean":"2023-11-24T08:09:06.159864+0000","last_became_active":"2023-11-24T08:09:04.832900+0000","last_became_peered":"2023-11-24T08:09:04.832900+0000","last_unstale":"2023-11-24T08:09:06.159864+0000","last_undegraded":"2023-11-24T08:09:06.159864+0000","last_fullsized":"2023-11-24T08:09:06.159864+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:41.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184698+0000","last_change":"2023-11-24T08:09:04.839307+0000","last_active":"2023-11-24T08:09:08.184698+0000","last_peered":"2023-11-24T08:09:08.184698+0000","last_clean":"2023-11-24T08:09:08.184698+0000","last_became_active":"2023-11-24T08:09:04.838947+0000","last_became_peered":"2023-11-24T08:09:04.838947+0000","last_unstale":"2023-11-24T08:09:08.184698+0000","last_undegraded":"2023-11-24T08:09:08.184698+0000","last_fullsized":"2023-11-24T08:09:08.184698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:13:47.091554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811568+0000","last_change":"2023-11-24T08:09:04.841279+0000","last_active":"2023-11-24T08:09:05.811568+0000","last_peered":"2023-11-24T08:09:05.811568+0000","last_clean":"2023-11-24T08:09:05.811568+0000","last_became_active":"2023-11-24T08:09:04.840035+0000","last_became_peered":"2023-11-24T08:09:04.840035+0000","last_unstale":"2023-11-24T08:09:05.811568+0000","last_undegraded":"2023-11-24T08:09:05.811568+0000","last_fullsized":"2023-11-24T08:09:05.811568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:54:47.658471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.511548+0000","last_change":"2023-11-24T08:09:04.835611+0000","last_active":"2023-11-24T08:09:07.511548+0000","last_peered":"2023-11-24T08:09:07.511548+0000","last_clean":"2023-11-24T08:09:07.511548+0000","last_became_active":"2023-11-24T08:09:04.835278+0000","last_became_peered":"2023-11-24T08:09:04.835278+0000","last_unstale":"2023-11-24T08:09:07.511548+0000","last_undegraded":"2023-11-24T08:09:07.511548+0000","last_fullsized":"2023-11-24T08:09:07.511548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:58.284845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.184219+0000","last_change":"2023-11-24T08:09:04.828970+0000","last_active":"2023-11-24T08:09:08.184219+0000","last_peered":"2023-11-24T08:09:08.184219+0000","last_clean":"2023-11-24T08:09:08.184219+0000","last_became_active":"2023-11-24T08:09:04.828596+0000","last_became_peered":"2023-11-24T08:09:04.828596+0000","last_unstale":"2023-11-24T08:09:08.184219+0000","last_undegraded":"2023-11-24T08:09:08.184219+0000","last_fullsized":"2023-11-24T08:09:08.184219+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:11.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159364+0000","last_change":"2023-11-24T08:09:04.829728+0000","last_active":"2023-11-24T08:09:06.159364+0000","last_peered":"2023-11-24T08:09:06.159364+0000","last_clean":"2023-11-24T08:09:06.159364+0000","last_became_active":"2023-11-24T08:09:04.829454+0000","last_became_peered":"2023-11-24T08:09:04.829454+0000","last_unstale":"2023-11-24T08:09:06.159364+0000","last_undegraded":"2023-11-24T08:09:06.159364+0000","last_fullsized":"2023-11-24T08:09:06.159364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:35:48.733653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159940+0000","last_change":"2023-11-24T08:09:04.841671+0000","last_active":"2023-11-24T08:09:06.159940+0000","last_peered":"2023-11-24T08:09:06.159940+0000","last_clean":"2023-11-24T08:09:06.159940+0000","last_became_active":"2023-11-24T08:09:04.841381+0000","last_became_peered":"2023-11-24T08:09:04.841381+0000","last_unstale":"2023-11-24T08:09:06.159940+0000","last_undegraded":"2023-11-24T08:09:06.159940+0000","last_fullsized":"2023-11-24T08:09:06.159940+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:48:55.858287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"63'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.064862+0000","last_change":"2023-11-24T08:09:04.062624+0000","last_active":"2023-11-24T08:09:05.064862+0000","last_peered":"2023-11-24T08:09:05.064862+0000","last_clean":"2023-11-24T08:09:05.064862+0000","last_became_active":"2023-11-24T08:09:04.062319+0000","last_became_peered":"2023-11-24T08:09:04.062319+0000","last_unstale":"2023-11-24T08:09:05.064862+0000","last_undegraded":"2023-11-24T08:09:05.064862+0000","last_fullsized":"2023-11-24T08:09:05.064862+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:29.480023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.835639+0000","last_change":"2023-11-24T08:09:04.829967+0000","last_active":"2023-11-24T08:09:04.835639+0000","last_peered":"2023-11-24T08:09:04.835639+0000","last_clean":"2023-11-24T08:09:04.835639+0000","last_became_active":"2023-11-24T08:09:04.829670+0000","last_became_peered":"2023-11-24T08:09:04.829670+0000","last_unstale":"2023-11-24T08:09:04.835639+0000","last_undegraded":"2023-11-24T08:09:04.835639+0000","last_fullsized":"2023-11-24T08:09:04.835639+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:33:35.041540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.056581+0000","last_change":"2023-11-24T08:09:04.066323+0000","last_active":"2023-11-24T08:09:05.056581+0000","last_peered":"2023-11-24T08:09:05.056581+0000","last_clean":"2023-11-24T08:09:05.056581+0000","last_became_active":"2023-11-24T08:09:04.066068+0000","last_became_peered":"2023-11-24T08:09:04.066068+0000","last_unstale":"2023-11-24T08:09:05.056581+0000","last_undegraded":"2023-11-24T08:09:05.056581+0000","last_fullsized":"2023-11-24T08:09:05.056581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:16.255653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159740+0000","last_change":"2023-11-24T08:09:04.835858+0000","last_active":"2023-11-24T08:09:06.159740+0000","last_peered":"2023-11-24T08:09:06.159740+0000","last_clean":"2023-11-24T08:09:06.159740+0000","last_became_active":"2023-11-24T08:09:04.835468+0000","last_became_peered":"2023-11-24T08:09:04.835468+0000","last_unstale":"2023-11-24T08:09:06.159740+0000","last_undegraded":"2023-11-24T08:09:06.159740+0000","last_fullsized":"2023-11-24T08:09:06.159740+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:18.478087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.056170+0000","last_change":"2023-11-24T08:09:04.068488+0000","last_active":"2023-11-24T08:09:05.056170+0000","last_peered":"2023-11-24T08:09:05.056170+0000","last_clean":"2023-11-24T08:09:05.056170+0000","last_became_active":"2023-11-24T08:09:04.068196+0000","last_became_peered":"2023-11-24T08:09:04.068196+0000","last_unstale":"2023-11-24T08:09:05.056170+0000","last_undegraded":"2023-11-24T08:09:05.056170+0000","last_fullsized":"2023-11-24T08:09:05.056170+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:55:50.111876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159879+0000","last_change":"2023-11-24T08:09:04.839442+0000","last_active":"2023-11-24T08:09:06.159879+0000","last_peered":"2023-11-24T08:09:06.159879+0000","last_clean":"2023-11-24T08:09:06.159879+0000","last_became_active":"2023-11-24T08:09:04.839087+0000","last_became_peered":"2023-11-24T08:09:04.839087+0000","last_unstale":"2023-11-24T08:09:06.159879+0000","last_undegraded":"2023-11-24T08:09:06.159879+0000","last_fullsized":"2023-11-24T08:09:06.159879+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:33:55.449563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.068670+0000","last_change":"2023-11-24T08:09:04.066183+0000","last_active":"2023-11-24T08:09:05.068670+0000","last_peered":"2023-11-24T08:09:05.068670+0000","last_clean":"2023-11-24T08:09:05.068670+0000","last_became_active":"2023-11-24T08:09:04.064976+0000","last_became_peered":"2023-11-24T08:09:04.064976+0000","last_unstale":"2023-11-24T08:09:05.068670+0000","last_undegraded":"2023-11-24T08:09:05.068670+0000","last_fullsized":"2023-11-24T08:09:05.068670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:08:51.140227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.812208+0000","last_change":"2023-11-24T08:09:04.836459+0000","last_active":"2023-11-24T08:09:05.812208+0000","last_peered":"2023-11-24T08:09:05.812208+0000","last_clean":"2023-11-24T08:09:05.812208+0000","last_became_active":"2023-11-24T08:09:04.836103+0000","last_became_peered":"2023-11-24T08:09:04.836103+0000","last_unstale":"2023-11-24T08:09:05.812208+0000","last_undegraded":"2023-11-24T08:09:05.812208+0000","last_fullsized":"2023-11-24T08:09:05.812208+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:46.770423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.063100+0000","last_change":"2023-11-24T08:09:04.059292+0000","last_active":"2023-11-24T08:09:05.063100+0000","last_peered":"2023-11-24T08:09:05.063100+0000","last_clean":"2023-11-24T08:09:05.063100+0000","last_became_active":"2023-11-24T08:09:04.058879+0000","last_became_peered":"2023-11-24T08:09:04.058879+0000","last_unstale":"2023-11-24T08:09:05.063100+0000","last_undegraded":"2023-11-24T08:09:05.063100+0000","last_fullsized":"2023-11-24T08:09:05.063100+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:53:08.880428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811630+0000","last_change":"2023-11-24T08:09:04.835365+0000","last_active":"2023-11-24T08:09:05.811630+0000","last_peered":"2023-11-24T08:09:05.811630+0000","last_clean":"2023-11-24T08:09:05.811630+0000","last_became_active":"2023-11-24T08:09:04.835068+0000","last_became_peered":"2023-11-24T08:09:04.835068+0000","last_unstale":"2023-11-24T08:09:05.811630+0000","last_undegraded":"2023-11-24T08:09:05.811630+0000","last_fullsized":"2023-11-24T08:09:05.811630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:44:32.300114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.060019+0000","last_change":"2023-11-24T08:09:04.066020+0000","last_active":"2023-11-24T08:09:05.060019+0000","last_peered":"2023-11-24T08:09:05.060019+0000","last_clean":"2023-11-24T08:09:05.060019+0000","last_became_active":"2023-11-24T08:09:04.065723+0000","last_became_peered":"2023-11-24T08:09:04.065723+0000","last_unstale":"2023-11-24T08:09:05.060019+0000","last_undegraded":"2023-11-24T08:09:05.060019+0000","last_fullsized":"2023-11-24T08:09:05.060019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:24.869662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.609334+0000","last_change":"2023-11-24T08:09:04.840935+0000","last_active":"2023-11-24T08:09:07.609334+0000","last_peered":"2023-11-24T08:09:07.609334+0000","last_clean":"2023-11-24T08:09:07.609334+0000","last_became_active":"2023-11-24T08:09:04.840624+0000","last_became_peered":"2023-11-24T08:09:04.840624+0000","last_unstale":"2023-11-24T08:09:07.609334+0000","last_undegraded":"2023-11-24T08:09:07.609334+0000","last_fullsized":"2023-11-24T08:09:07.609334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:26:18.976061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.820789+0000","last_change":"2023-11-24T08:09:04.060916+0000","last_active":"2023-11-24T08:09:04.820789+0000","last_peered":"2023-11-24T08:09:04.820789+0000","last_clean":"2023-11-24T08:09:04.820789+0000","last_became_active":"2023-11-24T08:09:04.060425+0000","last_became_peered":"2023-11-24T08:09:04.060425+0000","last_unstale":"2023-11-24T08:09:04.820789+0000","last_undegraded":"2023-11-24T08:09:04.820789+0000","last_fullsized":"2023-11-24T08:09:04.820789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:43.528617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811623+0000","last_change":"2023-11-24T08:09:04.837947+0000","last_active":"2023-11-24T08:09:05.811623+0000","last_peered":"2023-11-24T08:09:05.811623+0000","last_clean":"2023-11-24T08:09:05.811623+0000","last_became_active":"2023-11-24T08:09:04.837689+0000","last_became_peered":"2023-11-24T08:09:04.837689+0000","last_unstale":"2023-11-24T08:09:05.811623+0000","last_undegraded":"2023-11-24T08:09:05.811623+0000","last_fullsized":"2023-11-24T08:09:05.811623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:21.831296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.835579+0000","last_change":"2023-11-24T08:09:04.058197+0000","last_active":"2023-11-24T08:09:04.835579+0000","last_peered":"2023-11-24T08:09:04.835579+0000","last_clean":"2023-11-24T08:09:04.835579+0000","last_became_active":"2023-11-24T08:09:04.057246+0000","last_became_peered":"2023-11-24T08:09:04.057246+0000","last_unstale":"2023-11-24T08:09:04.835579+0000","last_undegraded":"2023-11-24T08:09:04.835579+0000","last_fullsized":"2023-11-24T08:09:04.835579+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:05.271623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.815281+0000","last_change":"2023-11-24T08:09:04.843478+0000","last_active":"2023-11-24T08:09:05.815281+0000","last_peered":"2023-11-24T08:09:05.815281+0000","last_clean":"2023-11-24T08:09:05.815281+0000","last_became_active":"2023-11-24T08:09:04.843080+0000","last_became_peered":"2023-11-24T08:09:04.843080+0000","last_unstale":"2023-11-24T08:09:05.815281+0000","last_undegraded":"2023-11-24T08:09:05.815281+0000","last_fullsized":"2023-11-24T08:09:05.815281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:24:39.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.808836+0000","last_change":"2023-11-24T08:09:04.072945+0000","last_active":"2023-11-24T08:09:04.808836+0000","last_peered":"2023-11-24T08:09:04.808836+0000","last_clean":"2023-11-24T08:09:04.808836+0000","last_became_active":"2023-11-24T08:09:04.072702+0000","last_became_peered":"2023-11-24T08:09:04.072702+0000","last_unstale":"2023-11-24T08:09:04.808836+0000","last_undegraded":"2023-11-24T08:09:04.808836+0000","last_fullsized":"2023-11-24T08:09:04.808836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:59:01.709142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512472+0000","last_change":"2023-11-24T08:09:04.833645+0000","last_active":"2023-11-24T08:09:07.512472+0000","last_peered":"2023-11-24T08:09:07.512472+0000","last_clean":"2023-11-24T08:09:07.512472+0000","last_became_active":"2023-11-24T08:09:04.833145+0000","last_became_peered":"2023-11-24T08:09:04.833145+0000","last_unstale":"2023-11-24T08:09:07.512472+0000","last_undegraded":"2023-11-24T08:09:07.512472+0000","last_fullsized":"2023-11-24T08:09:07.512472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:01:16.863651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.823330+0000","last_change":"2023-11-24T08:09:04.064497+0000","last_active":"2023-11-24T08:09:04.823330+0000","last_peered":"2023-11-24T08:09:04.823330+0000","last_clean":"2023-11-24T08:09:04.823330+0000","last_became_active":"2023-11-24T08:09:04.064216+0000","last_became_peered":"2023-11-24T08:09:04.064216+0000","last_unstale":"2023-11-24T08:09:04.823330+0000","last_undegraded":"2023-11-24T08:09:04.823330+0000","last_fullsized":"2023-11-24T08:09:04.823330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:32:55.637692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.841488+0000","last_change":"2023-11-24T08:09:04.836709+0000","last_active":"2023-11-24T08:09:04.841488+0000","last_peered":"2023-11-24T08:09:04.841488+0000","last_clean":"2023-11-24T08:09:04.841488+0000","last_became_active":"2023-11-24T08:09:04.835564+0000","last_became_peered":"2023-11-24T08:09:04.835564+0000","last_unstale":"2023-11-24T08:09:04.841488+0000","last_undegraded":"2023-11-24T08:09:04.841488+0000","last_fullsized":"2023-11-24T08:09:04.841488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:50:44.416682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"63'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.066612+0000","last_change":"2023-11-24T08:09:04.070425+0000","last_active":"2023-11-24T08:09:05.066612+0000","last_peered":"2023-11-24T08:09:05.066612+0000","last_clean":"2023-11-24T08:09:05.066612+0000","last_became_active":"2023-11-24T08:09:04.070180+0000","last_became_peered":"2023-11-24T08:09:04.070180+0000","last_unstale":"2023-11-24T08:09:05.066612+0000","last_undegraded":"2023-11-24T08:09:05.066612+0000","last_fullsized":"2023-11-24T08:09:05.066612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:55:50.111876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512261+0000","last_change":"2023-11-24T08:09:04.828807+0000","last_active":"2023-11-24T08:09:07.512261+0000","last_peered":"2023-11-24T08:09:07.512261+0000","last_clean":"2023-11-24T08:09:07.512261+0000","last_became_active":"2023-11-24T08:09:04.826211+0000","last_became_peered":"2023-11-24T08:09:04.826211+0000","last_unstale":"2023-11-24T08:09:07.512261+0000","last_undegraded":"2023-11-24T08:09:07.512261+0000","last_fullsized":"2023-11-24T08:09:07.512261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:48:55.858287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"63'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.072286+0000","last_change":"2023-11-24T08:09:04.070207+0000","last_active":"2023-11-24T08:09:05.072286+0000","last_peered":"2023-11-24T08:09:05.072286+0000","last_clean":"2023-11-24T08:09:05.072286+0000","last_became_active":"2023-11-24T08:09:04.069882+0000","last_became_peered":"2023-11-24T08:09:04.069882+0000","last_unstale":"2023-11-24T08:09:05.072286+0000","last_undegraded":"2023-11-24T08:09:05.072286+0000","last_fullsized":"2023-11-24T08:09:05.072286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:25.326592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999185+0000","last_change":"2023-11-24T08:09:04.834348+0000","last_active":"2023-11-24T08:09:07.999185+0000","last_peered":"2023-11-24T08:09:07.999185+0000","last_clean":"2023-11-24T08:09:07.999185+0000","last_became_active":"2023-11-24T08:09:04.833594+0000","last_became_peered":"2023-11-24T08:09:04.833594+0000","last_unstale":"2023-11-24T08:09:07.999185+0000","last_undegraded":"2023-11-24T08:09:07.999185+0000","last_fullsized":"2023-11-24T08:09:07.999185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:32.289942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.058284+0000","last_change":"2023-11-24T08:09:04.062897+0000","last_active":"2023-11-24T08:09:05.058284+0000","last_peered":"2023-11-24T08:09:05.058284+0000","last_clean":"2023-11-24T08:09:05.058284+0000","last_became_active":"2023-11-24T08:09:04.062494+0000","last_became_peered":"2023-11-24T08:09:04.062494+0000","last_unstale":"2023-11-24T08:09:05.058284+0000","last_undegraded":"2023-11-24T08:09:05.058284+0000","last_fullsized":"2023-11-24T08:09:05.058284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:23.994655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.511899+0000","last_change":"2023-11-24T08:09:04.835839+0000","last_active":"2023-11-24T08:09:07.511899+0000","last_peered":"2023-11-24T08:09:07.511899+0000","last_clean":"2023-11-24T08:09:07.511899+0000","last_became_active":"2023-11-24T08:09:04.832031+0000","last_became_peered":"2023-11-24T08:09:04.832031+0000","last_unstale":"2023-11-24T08:09:07.511899+0000","last_undegraded":"2023-11-24T08:09:07.511899+0000","last_fullsized":"2023-11-24T08:09:07.511899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:35:48.733653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.819931+0000","last_change":"2023-11-24T08:09:04.069688+0000","last_active":"2023-11-24T08:09:04.819931+0000","last_peered":"2023-11-24T08:09:04.819931+0000","last_clean":"2023-11-24T08:09:04.819931+0000","last_became_active":"2023-11-24T08:09:04.069373+0000","last_became_peered":"2023-11-24T08:09:04.069373+0000","last_unstale":"2023-11-24T08:09:04.819931+0000","last_undegraded":"2023-11-24T08:09:04.819931+0000","last_fullsized":"2023-11-24T08:09:04.819931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:54:13.607818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.814234+0000","last_change":"2023-11-24T08:09:04.844311+0000","last_active":"2023-11-24T08:09:05.814234+0000","last_peered":"2023-11-24T08:09:05.814234+0000","last_clean":"2023-11-24T08:09:05.814234+0000","last_became_active":"2023-11-24T08:09:04.843118+0000","last_became_peered":"2023-11-24T08:09:04.843118+0000","last_unstale":"2023-11-24T08:09:05.814234+0000","last_undegraded":"2023-11-24T08:09:05.814234+0000","last_fullsized":"2023-11-24T08:09:05.814234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:30:33.711532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.822022+0000","last_change":"2023-11-24T08:09:04.065397+0000","last_active":"2023-11-24T08:09:04.822022+0000","last_peered":"2023-11-24T08:09:04.822022+0000","last_clean":"2023-11-24T08:09:04.822022+0000","last_became_active":"2023-11-24T08:09:04.064920+0000","last_became_peered":"2023-11-24T08:09:04.064920+0000","last_unstale":"2023-11-24T08:09:04.822022+0000","last_undegraded":"2023-11-24T08:09:04.822022+0000","last_fullsized":"2023-11-24T08:09:04.822022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:59:01.709142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512043+0000","last_change":"2023-11-24T08:09:04.832288+0000","last_active":"2023-11-24T08:09:07.512043+0000","last_peered":"2023-11-24T08:09:07.512043+0000","last_clean":"2023-11-24T08:09:07.512043+0000","last_became_active":"2023-11-24T08:09:04.831405+0000","last_became_peered":"2023-11-24T08:09:04.831405+0000","last_unstale":"2023-11-24T08:09:07.512043+0000","last_undegraded":"2023-11-24T08:09:07.512043+0000","last_fullsized":"2023-11-24T08:09:07.512043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:44.721027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"63'8","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.074567+0000","last_change":"2023-11-24T08:09:04.071957+0000","last_active":"2023-11-24T08:09:05.074567+0000","last_peered":"2023-11-24T08:09:05.074567+0000","last_clean":"2023-11-24T08:09:05.074567+0000","last_became_active":"2023-11-24T08:09:04.071715+0000","last_became_peered":"2023-11-24T08:09:04.071715+0000","last_unstale":"2023-11-24T08:09:05.074567+0000","last_undegraded":"2023-11-24T08:09:05.074567+0000","last_fullsized":"2023-11-24T08:09:05.074567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:30:32.696075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512625+0000","last_change":"2023-11-24T08:09:04.840538+0000","last_active":"2023-11-24T08:09:07.512625+0000","last_peered":"2023-11-24T08:09:07.512625+0000","last_clean":"2023-11-24T08:09:07.512625+0000","last_became_active":"2023-11-24T08:09:04.840243+0000","last_became_peered":"2023-11-24T08:09:04.840243+0000","last_unstale":"2023-11-24T08:09:07.512625+0000","last_undegraded":"2023-11-24T08:09:07.512625+0000","last_fullsized":"2023-11-24T08:09:07.512625+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:41:58.124391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.058293+0000","last_change":"2023-11-24T08:09:04.064751+0000","last_active":"2023-11-24T08:09:05.058293+0000","last_peered":"2023-11-24T08:09:05.058293+0000","last_clean":"2023-11-24T08:09:05.058293+0000","last_became_active":"2023-11-24T08:09:04.064503+0000","last_became_peered":"2023-11-24T08:09:04.064503+0000","last_unstale":"2023-11-24T08:09:05.058293+0000","last_undegraded":"2023-11-24T08:09:05.058293+0000","last_fullsized":"2023-11-24T08:09:05.058293+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:04.229482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999657+0000","last_change":"2023-11-24T08:09:04.835455+0000","last_active":"2023-11-24T08:09:07.999657+0000","last_peered":"2023-11-24T08:09:07.999657+0000","last_clean":"2023-11-24T08:09:07.999657+0000","last_became_active":"2023-11-24T08:09:04.834542+0000","last_became_peered":"2023-11-24T08:09:04.834542+0000","last_unstale":"2023-11-24T08:09:07.999657+0000","last_undegraded":"2023-11-24T08:09:07.999657+0000","last_fullsized":"2023-11-24T08:09:07.999657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:18.478087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.058325+0000","last_change":"2023-11-24T08:09:04.069613+0000","last_active":"2023-11-24T08:09:05.058325+0000","last_peered":"2023-11-24T08:09:05.058325+0000","last_clean":"2023-11-24T08:09:05.058325+0000","last_became_active":"2023-11-24T08:09:04.068645+0000","last_became_peered":"2023-11-24T08:09:04.068645+0000","last_unstale":"2023-11-24T08:09:05.058325+0000","last_undegraded":"2023-11-24T08:09:05.058325+0000","last_fullsized":"2023-11-24T08:09:05.058325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:21:04.895892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811633+0000","last_change":"2023-11-24T08:09:04.838132+0000","last_active":"2023-11-24T08:09:05.811633+0000","last_peered":"2023-11-24T08:09:05.811633+0000","last_clean":"2023-11-24T08:09:05.811633+0000","last_became_active":"2023-11-24T08:09:04.837867+0000","last_became_peered":"2023-11-24T08:09:04.837867+0000","last_unstale":"2023-11-24T08:09:05.811633+0000","last_undegraded":"2023-11-24T08:09:05.811633+0000","last_fullsized":"2023-11-24T08:09:05.811633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:34.764828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.068242+0000","last_change":"2023-11-24T08:09:04.067029+0000","last_active":"2023-11-24T08:09:05.068242+0000","last_peered":"2023-11-24T08:09:05.068242+0000","last_clean":"2023-11-24T08:09:05.068242+0000","last_became_active":"2023-11-24T08:09:04.066682+0000","last_became_peered":"2023-11-24T08:09:04.066682+0000","last_unstale":"2023-11-24T08:09:05.068242+0000","last_undegraded":"2023-11-24T08:09:05.068242+0000","last_fullsized":"2023-11-24T08:09:05.068242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:29.480023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512384+0000","last_change":"2023-11-24T08:09:04.828054+0000","last_active":"2023-11-24T08:09:07.512384+0000","last_peered":"2023-11-24T08:09:07.512384+0000","last_clean":"2023-11-24T08:09:07.512384+0000","last_became_active":"2023-11-24T08:09:04.826903+0000","last_became_peered":"2023-11-24T08:09:04.826903+0000","last_unstale":"2023-11-24T08:09:07.512384+0000","last_undegraded":"2023-11-24T08:09:07.512384+0000","last_fullsized":"2023-11-24T08:09:07.512384+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:21.015253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.822740+0000","last_change":"2023-11-24T08:09:04.059667+0000","last_active":"2023-11-24T08:09:04.822740+0000","last_peered":"2023-11-24T08:09:04.822740+0000","last_clean":"2023-11-24T08:09:04.822740+0000","last_became_active":"2023-11-24T08:09:04.059403+0000","last_became_peered":"2023-11-24T08:09:04.059403+0000","last_unstale":"2023-11-24T08:09:04.822740+0000","last_undegraded":"2023-11-24T08:09:04.822740+0000","last_fullsized":"2023-11-24T08:09:04.822740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:08:51.140227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999260+0000","last_change":"2023-11-24T08:09:04.834552+0000","last_active":"2023-11-24T08:09:07.999260+0000","last_peered":"2023-11-24T08:09:07.999260+0000","last_clean":"2023-11-24T08:09:07.999260+0000","last_became_active":"2023-11-24T08:09:04.834038+0000","last_became_peered":"2023-11-24T08:09:04.834038+0000","last_unstale":"2023-11-24T08:09:07.999260+0000","last_undegraded":"2023-11-24T08:09:07.999260+0000","last_fullsized":"2023-11-24T08:09:07.999260+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:18:33.778568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.056586+0000","last_change":"2023-11-24T08:09:04.058300+0000","last_active":"2023-11-24T08:09:05.056586+0000","last_peered":"2023-11-24T08:09:05.056586+0000","last_clean":"2023-11-24T08:09:05.056586+0000","last_became_active":"2023-11-24T08:09:04.057505+0000","last_became_peered":"2023-11-24T08:09:04.057505+0000","last_unstale":"2023-11-24T08:09:05.056586+0000","last_undegraded":"2023-11-24T08:09:05.056586+0000","last_fullsized":"2023-11-24T08:09:05.056586+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:42:44.752620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159765+0000","last_change":"2023-11-24T08:09:04.833001+0000","last_active":"2023-11-24T08:09:06.159765+0000","last_peered":"2023-11-24T08:09:06.159765+0000","last_clean":"2023-11-24T08:09:06.159765+0000","last_became_active":"2023-11-24T08:09:04.832695+0000","last_became_peered":"2023-11-24T08:09:04.832695+0000","last_unstale":"2023-11-24T08:09:06.159765+0000","last_undegraded":"2023-11-24T08:09:06.159765+0000","last_fullsized":"2023-11-24T08:09:06.159765+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:44.721027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.818007+0000","last_change":"2023-11-24T08:09:04.059741+0000","last_active":"2023-11-24T08:09:04.818007+0000","last_peered":"2023-11-24T08:09:04.818007+0000","last_clean":"2023-11-24T08:09:04.818007+0000","last_became_active":"2023-11-24T08:09:04.058901+0000","last_became_peered":"2023-11-24T08:09:04.058901+0000","last_unstale":"2023-11-24T08:09:04.818007+0000","last_undegraded":"2023-11-24T08:09:04.818007+0000","last_fullsized":"2023-11-24T08:09:04.818007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:21:30.776399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.159341+0000","last_change":"2023-11-24T08:09:04.840913+0000","last_active":"2023-11-24T08:09:06.159341+0000","last_peered":"2023-11-24T08:09:06.159341+0000","last_clean":"2023-11-24T08:09:06.159341+0000","last_became_active":"2023-11-24T08:09:04.840605+0000","last_became_peered":"2023-11-24T08:09:04.840605+0000","last_unstale":"2023-11-24T08:09:06.159341+0000","last_undegraded":"2023-11-24T08:09:06.159341+0000","last_fullsized":"2023-11-24T08:09:06.159341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:58.284845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.820540+0000","last_change":"2023-11-24T08:09:04.059744+0000","last_active":"2023-11-24T08:09:04.820540+0000","last_peered":"2023-11-24T08:09:04.820540+0000","last_clean":"2023-11-24T08:09:04.820540+0000","last_became_active":"2023-11-24T08:09:04.059058+0000","last_became_peered":"2023-11-24T08:09:04.059058+0000","last_unstale":"2023-11-24T08:09:04.820540+0000","last_undegraded":"2023-11-24T08:09:04.820540+0000","last_fullsized":"2023-11-24T08:09:04.820540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:27:10.744501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.999646+0000","last_change":"2023-11-24T08:09:04.834119+0000","last_active":"2023-11-24T08:09:07.999646+0000","last_peered":"2023-11-24T08:09:07.999646+0000","last_clean":"2023-11-24T08:09:07.999646+0000","last_became_active":"2023-11-24T08:09:04.833737+0000","last_became_peered":"2023-11-24T08:09:04.833737+0000","last_unstale":"2023-11-24T08:09:07.999646+0000","last_undegraded":"2023-11-24T08:09:07.999646+0000","last_fullsized":"2023-11-24T08:09:07.999646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:35:48.733653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"63'2","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.056984+0000","last_change":"2023-11-24T08:09:04.071379+0000","last_active":"2023-11-24T08:09:05.056984+0000","last_peered":"2023-11-24T08:09:05.056984+0000","last_clean":"2023-11-24T08:09:05.056984+0000","last_became_active":"2023-11-24T08:09:04.071052+0000","last_became_peered":"2023-11-24T08:09:04.071052+0000","last_unstale":"2023-11-24T08:09:05.056984+0000","last_undegraded":"2023-11-24T08:09:05.056984+0000","last_fullsized":"2023-11-24T08:09:05.056984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:54:13.607818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:08.000086+0000","last_change":"2023-11-24T08:09:04.835217+0000","last_active":"2023-11-24T08:09:08.000086+0000","last_peered":"2023-11-24T08:09:08.000086+0000","last_clean":"2023-11-24T08:09:08.000086+0000","last_became_active":"2023-11-24T08:09:04.833883+0000","last_became_peered":"2023-11-24T08:09:04.833883+0000","last_unstale":"2023-11-24T08:09:08.000086+0000","last_undegraded":"2023-11-24T08:09:08.000086+0000","last_fullsized":"2023-11-24T08:09:08.000086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:41.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.823175+0000","last_change":"2023-11-24T08:09:04.068367+0000","last_active":"2023-11-24T08:09:04.823175+0000","last_peered":"2023-11-24T08:09:04.823175+0000","last_clean":"2023-11-24T08:09:04.823175+0000","last_became_active":"2023-11-24T08:09:04.066342+0000","last_became_peered":"2023-11-24T08:09:04.066342+0000","last_unstale":"2023-11-24T08:09:04.823175+0000","last_undegraded":"2023-11-24T08:09:04.823175+0000","last_fullsized":"2023-11-24T08:09:04.823175+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:33:22.160791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.841280+0000","last_change":"2023-11-24T08:09:04.836088+0000","last_active":"2023-11-24T08:09:04.841280+0000","last_peered":"2023-11-24T08:09:04.841280+0000","last_clean":"2023-11-24T08:09:04.841280+0000","last_became_active":"2023-11-24T08:09:04.835830+0000","last_became_peered":"2023-11-24T08:09:04.835830+0000","last_unstale":"2023-11-24T08:09:04.841280+0000","last_undegraded":"2023-11-24T08:09:04.841280+0000","last_fullsized":"2023-11-24T08:09:04.841280+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:31.859527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.819826+0000","last_change":"2023-11-24T08:09:04.066944+0000","last_active":"2023-11-24T08:09:04.819826+0000","last_peered":"2023-11-24T08:09:04.819826+0000","last_clean":"2023-11-24T08:09:04.819826+0000","last_became_active":"2023-11-24T08:09:04.066106+0000","last_became_peered":"2023-11-24T08:09:04.066106+0000","last_unstale":"2023-11-24T08:09:04.819826+0000","last_undegraded":"2023-11-24T08:09:04.819826+0000","last_fullsized":"2023-11-24T08:09:04.819826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:04.799981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:06.160079+0000","last_change":"2023-11-24T08:09:04.837568+0000","last_active":"2023-11-24T08:09:06.160079+0000","last_peered":"2023-11-24T08:09:06.160079+0000","last_clean":"2023-11-24T08:09:06.160079+0000","last_became_active":"2023-11-24T08:09:04.837293+0000","last_became_peered":"2023-11-24T08:09:04.837293+0000","last_unstale":"2023-11-24T08:09:06.160079+0000","last_undegraded":"2023-11-24T08:09:06.160079+0000","last_fullsized":"2023-11-24T08:09:06.160079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:21.015253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.822777+0000","last_change":"2023-11-24T08:09:04.067323+0000","last_active":"2023-11-24T08:09:04.822777+0000","last_peered":"2023-11-24T08:09:04.822777+0000","last_clean":"2023-11-24T08:09:04.822777+0000","last_became_active":"2023-11-24T08:09:04.067031+0000","last_became_peered":"2023-11-24T08:09:04.067031+0000","last_unstale":"2023-11-24T08:09:04.822777+0000","last_undegraded":"2023-11-24T08:09:04.822777+0000","last_fullsized":"2023-11-24T08:09:04.822777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:21:04.895892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.812464+0000","last_change":"2023-11-24T08:09:04.831724+0000","last_active":"2023-11-24T08:09:05.812464+0000","last_peered":"2023-11-24T08:09:05.812464+0000","last_clean":"2023-11-24T08:09:05.812464+0000","last_became_active":"2023-11-24T08:09:04.831444+0000","last_became_peered":"2023-11-24T08:09:04.831444+0000","last_unstale":"2023-11-24T08:09:05.812464+0000","last_undegraded":"2023-11-24T08:09:05.812464+0000","last_fullsized":"2023-11-24T08:09:05.812464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:43.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.822372+0000","last_change":"2023-11-24T08:09:04.065248+0000","last_active":"2023-11-24T08:09:04.822372+0000","last_peered":"2023-11-24T08:09:04.822372+0000","last_clean":"2023-11-24T08:09:04.822372+0000","last_became_active":"2023-11-24T08:09:04.064941+0000","last_became_peered":"2023-11-24T08:09:04.064941+0000","last_unstale":"2023-11-24T08:09:04.822372+0000","last_undegraded":"2023-11-24T08:09:04.822372+0000","last_fullsized":"2023-11-24T08:09:04.822372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:51:10.163010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:07.512252+0000","last_change":"2023-11-24T08:09:04.837409+0000","last_active":"2023-11-24T08:09:07.512252+0000","last_peered":"2023-11-24T08:09:07.512252+0000","last_clean":"2023-11-24T08:09:07.512252+0000","last_became_active":"2023-11-24T08:09:04.831605+0000","last_became_peered":"2023-11-24T08:09:04.831605+0000","last_unstale":"2023-11-24T08:09:07.512252+0000","last_undegraded":"2023-11-24T08:09:07.512252+0000","last_fullsized":"2023-11-24T08:09:07.512252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:33:55.449563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.835841+0000","last_change":"2023-11-24T08:09:04.070273+0000","last_active":"2023-11-24T08:09:04.835841+0000","last_peered":"2023-11-24T08:09:04.835841+0000","last_clean":"2023-11-24T08:09:04.835841+0000","last_became_active":"2023-11-24T08:09:04.069977+0000","last_became_peered":"2023-11-24T08:09:04.069977+0000","last_unstale":"2023-11-24T08:09:04.835841+0000","last_undegraded":"2023-11-24T08:09:04.835841+0000","last_fullsized":"2023-11-24T08:09:04.835841+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:26.785491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.837394+0000","last_change":"2023-11-24T08:09:04.831921+0000","last_active":"2023-11-24T08:09:04.837394+0000","last_peered":"2023-11-24T08:09:04.837394+0000","last_clean":"2023-11-24T08:09:04.837394+0000","last_became_active":"2023-11-24T08:09:04.831373+0000","last_became_peered":"2023-11-24T08:09:04.831373+0000","last_unstale":"2023-11-24T08:09:04.837394+0000","last_undegraded":"2023-11-24T08:09:04.837394+0000","last_fullsized":"2023-11-24T08:09:04.837394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:09.003448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.818791+0000","last_change":"2023-11-24T08:09:04.072098+0000","last_active":"2023-11-24T08:09:04.818791+0000","last_peered":"2023-11-24T08:09:04.818791+0000","last_clean":"2023-11-24T08:09:04.818791+0000","last_became_active":"2023-11-24T08:09:04.071723+0000","last_became_peered":"2023-11-24T08:09:04.071723+0000","last_unstale":"2023-11-24T08:09:04.818791+0000","last_undegraded":"2023-11-24T08:09:04.818791+0000","last_fullsized":"2023-11-24T08:09:04.818791+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:43:48.221503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811987+0000","last_change":"2023-11-24T08:09:04.831991+0000","last_active":"2023-11-24T08:09:05.811987+0000","last_peered":"2023-11-24T08:09:05.811987+0000","last_clean":"2023-11-24T08:09:05.811987+0000","last_became_active":"2023-11-24T08:09:04.831705+0000","last_became_peered":"2023-11-24T08:09:04.831705+0000","last_unstale":"2023-11-24T08:09:05.811987+0000","last_undegraded":"2023-11-24T08:09:05.811987+0000","last_fullsized":"2023-11-24T08:09:05.811987+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:10:29.028717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.815884+0000","last_change":"2023-11-24T08:09:04.060855+0000","last_active":"2023-11-24T08:09:04.815884+0000","last_peered":"2023-11-24T08:09:04.815884+0000","last_clean":"2023-11-24T08:09:04.815884+0000","last_became_active":"2023-11-24T08:09:04.060262+0000","last_became_peered":"2023-11-24T08:09:04.060262+0000","last_unstale":"2023-11-24T08:09:04.815884+0000","last_undegraded":"2023-11-24T08:09:04.815884+0000","last_fullsized":"2023-11-24T08:09:04.815884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:19.999796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T08:09:05.811034+0000","last_change":"2023-11-24T08:09:04.835398+0000","last_active":"2023-11-24T08:09:05.811034+0000","last_peered":"2023-11-24T08:09:05.811034+0000","last_clean":"2023-11-24T08:09:05.811034+0000","last_became_active":"2023-11-24T08:09:04.835127+0000","last_became_peered":"2023-11-24T08:09:04.835127+0000","last_unstale":"2023-11-24T08:09:05.811034+0000","last_undegraded":"2023-11-24T08:09:05.811034+0000","last_fullsized":"2023-11-24T08:09:05.811034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:04.020956+0000","last_clean_scrub_stamp":"2023-11-24T08:09:04.020956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:22:55.356772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.817583+0000","last_change":"2023-11-24T08:09:04.068372+0000","last_active":"2023-11-24T08:09:04.817583+0000","last_peered":"2023-11-24T08:09:04.817583+0000","last_clean":"2023-11-24T08:09:04.817583+0000","last_became_active":"2023-11-24T08:09:04.068106+0000","last_became_peered":"2023-11-24T08:09:04.068106+0000","last_unstale":"2023-11-24T08:09:04.817583+0000","last_undegraded":"2023-11-24T08:09:04.817583+0000","last_fullsized":"2023-11-24T08:09:04.817583+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:09:03.005499+0000","last_clean_scrub_stamp":"2023-11-24T08:09:03.005499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:24.372754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T08:09:04.806948+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:09:04.806948+0000","last_peered":"2023-11-24T08:09:04.806948+0000","last_clean":"2023-11-24T08:09:04.806948+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:09:04.806948+0000","last_undegraded":"2023-11-24T08:09:04.806948+0000","last_fullsized":"2023-11-24T08:09:04.806948+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574905,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":916411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0389999999999999,"15min":0.995},"min":{"1min":0.626,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2969999999999999,"5min":1.621,"15min":1.621},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.115,"15min":1.079},"min":{"1min":0.86399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.403,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.4990000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1439999999999999,"15min":1.131},"min":{"1min":0.629,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6739999999999999,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2090000000000001,"15min":1.1830000000000001},"min":{"1min":0.86099999999999999,"5min":0.628,"15min":0.628},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.3859999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1830000000000001,"15min":1.228},"min":{"1min":0.67000000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.874},{"interface":"front","average":{"1min":1.248,"5min":1.333,"15min":1.361},"min":{"1min":0.79500000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7729999999999999,"5min":4.633,"15min":4.633},"last":1.532}]},{"osd":3,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1719999999999999,"15min":1.135},"min":{"1min":0.68700000000000006,"5min":0.504,"15min":0.504},"max":{"1min":1.653,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.3600000000000001,"15min":1.357},"min":{"1min":0.86499999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":2.206,"15min":2.206},"last":1.4450000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.3340000000000001},"min":{"1min":1.0349999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.831,"5min":2.851,"15min":2.851},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3540000000000001,"15min":1.3320000000000001},"min":{"1min":0.85999999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0529999999999999,"5min":3.0510000000000002,"15min":3.0510000000000002},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.482,"15min":1.454},"min":{"1min":0.95799999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8779999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.5249999999999999,"15min":1.4850000000000001},"min":{"1min":1.327,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.9410000000000001,"5min":2.629,"15min":2.629},"last":1.2829999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5900000000000001,"15min":1.542},"min":{"1min":1.272,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0179999999999998,"5min":2.746,"15min":2.746},"last":1.591},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4930000000000001,"15min":1.4410000000000001},"min":{"1min":1.0760000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8540000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.224}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31932,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":770048,"data_stored":325883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.093,"15min":1.0940000000000001},"min":{"1min":0.69199999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.534,"5min":1.726,"15min":1.726},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.044,"5min":1.083,"15min":1.109},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.659,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.2010000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2030000000000001,"15min":1.2749999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5009999999999999,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.133},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.76900000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.694,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.052}]},{"osd":2,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.119,"15min":1.133},"min":{"1min":0.752,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.601,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.415},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5409999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.996}]},{"osd":3,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.369,"15min":1.409},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0720000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3200000000000001,"15min":1.351},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8149999999999999,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.5409999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.476,"15min":1.5720000000000001},"min":{"1min":0.86499999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.629,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.5429999999999999,"15min":1.5820000000000001},"min":{"1min":1.177,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.9790000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.726}]},{"osd":5,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.506,"15min":1.5840000000000001},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.2250000000000001,"5min":2.629,"15min":2.629},"last":1.603},{"interface":"front","average":{"1min":1.516,"5min":1.637,"15min":1.7190000000000001},"min":{"1min":1.274,"5min":1.022,"15min":1.022},"max":{"1min":1.7390000000000001,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.4510000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:08:57 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.671,"15min":1.7549999999999999},"min":{"1min":1.2430000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1850000000000001,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.649,"15min":1.7130000000000001},"min":{"1min":1.1519999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.29,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.6759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":326535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0680000000000001,"15min":1.141},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6339999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.1519999999999999,"15min":1.194},"min":{"1min":0.65600000000000003,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.597,"5min":2.198,"15min":2.198},"last":1.6519999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1220000000000001,"15min":1.179},"min":{"1min":0.71499999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.579,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1339999999999999,"15min":1.1399999999999999},"min":{"1min":0.72599999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.524,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.635}]},{"osd":2,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2050000000000001,"15min":1.2170000000000001},"min":{"1min":0.69499999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":1.232},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.169,"15min":1.181},"min":{"1min":0.69999999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.7370000000000001,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.397}]},{"osd":3,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.1679999999999999,"15min":1.202},"min":{"1min":0.76800000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.282,"15min":1.349},"min":{"1min":0.79900000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.851,"5min":1.99,"15min":1.99},"last":1.3260000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3180000000000001,"15min":1.345},"min":{"1min":0.98899999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.478},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4990000000000001,"15min":1.569},"min":{"1min":1.004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.0579999999999998,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.8120000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3979999999999999,"15min":1.482},"min":{"1min":0.83699999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.1240000000000001,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.419,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9179999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.7470000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:08:54 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5429999999999999,"15min":1.627},"min":{"1min":0.92400000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.9950000000000001,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5880000000000001,"15min":1.6220000000000001},"min":{"1min":1.256,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.028,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.857}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":782336,"data_stored":327495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.115,"15min":1.167},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6890000000000001,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.085,"15min":1.0960000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.637,"5min":2.222,"15min":2.222},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.1699999999999999,"15min":1.169},"min":{"1min":0.93600000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6639999999999999,"5min":1.74,"15min":1.74},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1890000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5609999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.51900000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.113,"15min":1.1830000000000001},"min":{"1min":0.72499999999999998,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4730000000000001,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3129999999999999,"15min":1.397},"min":{"1min":0.67700000000000005,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.649,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.505}]},{"osd":3,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.391,"15min":1.4570000000000001},"min":{"1min":0.79800000000000004,"5min":0.751,"15min":0.751},"max":{"1min":2.121,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":0.62},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.288,"15min":1.3400000000000001},"min":{"1min":0.89800000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.5229999999999999,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5049999999999999,"15min":1.53},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.393,"5min":1.486,"15min":1.5680000000000001},"min":{"1min":0.88500000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.8640000000000001,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.355,"15min":1.488},"min":{"1min":0.95299999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4610000000000001,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":0.751},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.595,"15min":1.6919999999999999},"min":{"1min":1.2629999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.986,"5min":3.391,"15min":3.391},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5629999999999999,"15min":1.661},"min":{"1min":1.016,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.752,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4910000000000001,"15min":1.635},"min":{"1min":1.147,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.752,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.71799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":325939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":0.82999999999999996},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2749999999999999,"5min":10.557,"15min":10.557},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.92600000000000005,"15min":0.91000000000000003},"min":{"1min":0.53100000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2509999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.79200000000000004,"15min":0.84099999999999997},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.2070000000000001,"5min":1.532,"15min":1.532},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.0549999999999999,"15min":1.0029999999999999},"min":{"1min":0.45700000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.28,"5min":13.090999999999999,"15min":13.090999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.96499999999999997,"15min":0.97099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2030000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.435},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.1419999999999999,"15min":1.0660000000000001},"min":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.343,"5min":13.460000000000001,"15min":13.460000000000001},"last":0.621}]},{"osd":4,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.3080000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.623,"5min":4.367,"15min":4.367},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.141,"5min":1.2629999999999999,"15min":1.345},"min":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.607,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.3660000000000001,"15min":1.3200000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5660000000000001,"5min":13.231,"15min":13.231},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.097,"5min":1.331,"15min":1.341},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.591,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":0.71899999999999997}]},{"osd":6,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.4339999999999999,"15min":1.401},"min":{"1min":0.751,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.825,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.288,"15min":1.3700000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.7030000000000001,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.89800000000000002}]},{"osd":7,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.4059999999999999,"15min":1.454},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6779999999999999,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.157,"5min":1.365,"15min":1.4910000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.764,"5min":4.032,"15min":4.032},"last":0.80200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":325939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.88300000000000001,"15min":0.94099999999999995},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.389,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.93200000000000005,"15min":1.0369999999999999},"min":{"1min":0.58399999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4370000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.66700000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0309999999999999,"15min":1.165},"min":{"1min":0.498,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3939999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0580000000000001,"15min":1.161},"min":{"1min":0.505,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.788,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.73099999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.92600000000000005,"15min":0.96999999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4890000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.056,"5min":1.1819999999999999,"15min":1.3069999999999999},"min":{"1min":0.56299999999999994,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.946,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.097}]},{"osd":4,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3520000000000001,"15min":1.4359999999999999},"min":{"1min":0.88300000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.893,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.361,"15min":1.425},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.857,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":1.3420000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.381,"15min":1.4850000000000001},"min":{"1min":0.80300000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0230000000000001,"5min":2.508,"15min":2.508},"last":1.454},{"interface":"front","average":{"1min":1.288,"5min":1.3660000000000001,"15min":1.4430000000000001},"min":{"1min":0.92500000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.7749999999999999,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.284}]},{"osd":6,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.302,"15min":1.3680000000000001},"min":{"1min":0.745,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.708,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.45,"15min":1.552},"min":{"1min":0.92400000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.1389999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.6439999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:08:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.5449999999999999},"min":{"1min":0.84299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.885,"5min":2.859,"15min":2.859},"last":1.659},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4330000000000001,"15min":1.5209999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9410000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.369}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":319673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.42399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.2909999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84699999999999998,"15min":0.82599999999999996},"min":{"1min":0.53900000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.321,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.004}]},{"osd":2,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74099999999999999,"15min":0.69299999999999995},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2989999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88400000000000001,"15min":0.86899999999999999},"min":{"1min":0.438,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.85999999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.95699999999999996,"15min":0.94199999999999995},"min":{"1min":0.54500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3480000000000001,"5min":1.675,"15min":1.675},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.89000000000000001,"15min":0.873},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.425,"5min":1.464,"15min":1.464},"last":1.0549999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.256,"15min":1.23},"min":{"1min":0.94299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.5820000000000001,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.2509999999999999,"15min":1.252},"min":{"1min":0.84999999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.4159999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2370000000000001,"15min":1.1859999999999999},"min":{"1min":0.748,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.645,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.216,"15min":1.1719999999999999},"min":{"1min":0.84599999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.742,"5min":3.286,"15min":3.286},"last":1.577}]},{"osd":6,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2330000000000001,"15min":1.2030000000000001},"min":{"1min":0.79500000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.897,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":1.095},{"interface":"front","average":{"1min":1.321,"5min":1.234,"15min":1.181},"min":{"1min":0.89200000000000002,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.847,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.252}]},{"osd":7,"last update":"Fri Nov 24 08:08:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2909999999999999,"15min":1.2869999999999999},"min":{"1min":0.82099999999999995,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.79,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.129},{"interface":"front","average":{"1min":1.329,"5min":1.3049999999999999,"15min":1.252},"min":{"1min":0.871,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.972,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":1.502}]}]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":918445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.81299999999999994,"15min":0.90200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.996,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.93700000000000006,"15min":1.04},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.3029999999999999,"5min":1.885,"15min":1.885},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.04,"15min":1.0760000000000001},"min":{"1min":0.52900000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.8460000000000001,"5min":1.865,"15min":1.865},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.96699999999999997},"min":{"1min":0.502,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.51,"5min":1.847,"15min":1.847},"last":0.53100000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.075,"15min":1.143},"min":{"1min":0.52400000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5109999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.96899999999999997,"15min":1.0800000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.3759999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.379,"15min":1.446},"min":{"1min":0.83199999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":3.7610000000000001,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.179,"15min":1.27},"min":{"1min":0.74099999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.2929999999999999,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.2370000000000001,"15min":1.345},"min":{"1min":0.71799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.3939999999999999,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.198,"5min":1.333,"15min":1.4730000000000001},"min":{"1min":0.80800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7689999999999999,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.71699999999999997}]},{"osd":6,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.351,"15min":1.4630000000000001},"min":{"1min":0.77700000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.9039999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.365,"15min":1.4810000000000001},"min":{"1min":0.84699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.5609999999999999,"5min":2.738,"15min":2.738},"last":0.97099999999999997}]},{"osd":7,"last update":"Fri Nov 24 08:08:52 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.456,"15min":1.552},"min":{"1min":0.93799999999999994,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0379999999999998,"5min":2.669,"15min":2.669},"last":1.014},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.427,"15min":1.544},"min":{"1min":0.80100000000000005,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.871,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.88700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:09:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:09:09.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:09:09.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:09:10.007 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:09:10.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.192+0000 7fd921af1640 1 Processor -- start 2023-11-24T08:09:10.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.193+0000 7fd921af1640 1 -- start start 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd921af1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 0x7fd91c0ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd921af1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0cdf70 con 0x7fd91c0ac840 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd921af1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0cbfa0 con 0x7fd91c06d080 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd921af1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0cb3d0 con 0x7fd91c15d5f0 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd91bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd91c0ac840 0x7fd91c06c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44380/0 (socket says 172.21.15.107:44380) 2023-11-24T08:09:10.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.194+0000 7fd91bfff640 1 -- 172.21.15.107:0/3583951651 learned_addr learned my addr 172.21.15.107:0/3583951651 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:10.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd920aef640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 0x7fd91c0ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744999234 0 0) 0x7fd91c0cdf70 con 0x7fd91c0ac840 2023-11-24T08:09:10.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd908003520 con 0x7fd91c0ac840 2023-11-24T08:09:10.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 660943639 0 0) 0x7fd908003520 con 0x7fd91c0ac840 2023-11-24T08:09:10.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 msgr2=0x7fd91c0ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 0x7fd91c0ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 >> v1:172.21.15.120:6789/0 conn(0x7fd91c06d080 legacy=0x7fd91c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0cc280 con 0x7fd91c0ac840 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd904001670 con 0x7fd91c0ac840 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 >> v1:172.21.15.107:6789/0 conn(0x7fd91c0ac840 legacy=0x7fd91c06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd908003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.195+0000 7fd91b7fe640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9080047c0 0x7fd908004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd91b7fe640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 0x7fd9080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0cb3d0 con 0x7fd9080039a0 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90800bab0 con 0x7fd9080047c0 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90800bc70 con 0x7fd91c15d5f0 2023-11-24T08:09:10.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd908003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd91bfff640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9080047c0 0x7fd908004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 -- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 msgr2=0x7fd9080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c15d5f0 0x7fd9080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 -- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9080047c0 msgr2=0x7fd908004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9080047c0 0x7fd908004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.196+0000 7fd920aef640 1 -- 172.21.15.107:0/3583951651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908003520 con 0x7fd9080039a0 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.197+0000 7fd920aef640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd908003dc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009fc0 tx=0x7fd9080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee545d1e900d4ceb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.197+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c02cc60 con 0x7fd9080039a0 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.197+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c00f040 con 0x7fd9080039a0 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.197+0000 7fd91b7fe640 1 -- 172.21.15.107:0/3583951651 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c034440 con 0x7fd9080039a0 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 -- 172.21.15.107:0/3583951651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 msgr2=0x7fd908003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 --2- 172.21.15.107:0/3583951651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd908003dc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009fc0 tx=0x7fd9080057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd9212f0640 1 -- 172.21.15.107:0/3583951651 reap_dead start 2023-11-24T08:09:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 -- 172.21.15.107:0/3583951651 shutdown_connections 2023-11-24T08:09:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 -- 172.21.15.107:0/3583951651 >> 172.21.15.107:0/3583951651 conn(0x7fd91c06a500 msgr2=0x7fd91c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 -- 172.21.15.107:0/3583951651 shutdown_connections 2023-11-24T08:09:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.198+0000 7fd921af1640 1 -- 172.21.15.107:0/3583951651 wait complete. 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.200+0000 7fd921af1640 1 Processor -- start 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.200+0000 7fd921af1640 1 -- start start 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91c143b00 0x7fd91c156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c157190 0x7fd91c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c141760 con 0x7fd9080039a0 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c144a10 con 0x7fd91c143b00 2023-11-24T08:09:10.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd921af1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c15ea20 con 0x7fd91c157190 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd920aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd920aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37364/0 (socket says 172.21.15.107:37364) 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.201+0000 7fd920aef640 1 -- 172.21.15.107:0/1398433732 learned_addr learned my addr 172.21.15.107:0/1398433732 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 -- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c157190 msgr2=0x7fd91c154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c157190 0x7fd91c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 -- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91c143b00 msgr2=0x7fd91c156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91c143b00 0x7fd91c156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 -- 172.21.15.107:0/1398433732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c157630 con 0x7fd9080039a0 2023-11-24T08:09:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd920aef640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00e230 tx=0x7fd90c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c034bd0 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c034d30 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c157910 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c034ea0 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.202+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c15ec70 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.204+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c157e30 con 0x7fd9080039a0 2023-11-24T08:09:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.205+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fd90c040450 con 0x7fd9080039a0 2023-11-24T08:09:10.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.205+0000 7fd9197fa640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 0x7fd8fc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.206+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd90c0ac610 con 0x7fd9080039a0 2023-11-24T08:09:10.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.206+0000 7fd91bfff640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 0x7fd8fc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.206+0000 7fd91bfff640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 0x7fd8fc06d8a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9040017a0 tx=0x7fd904009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.210+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd90c077100 con 0x7fd9080039a0 2023-11-24T08:09:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.463+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd91c04e840 con 0x7fd9080039a0 2023-11-24T08:09:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.467+0000 7fd9197fa640 1 -- 172.21.15.107:0/1398433732 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd90c07a720 con 0x7fd9080039a0 2023-11-24T08:09:10.469 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:09:10.469 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:09:10.469 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:09:10.469 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:09:10.469 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:09:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:09:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 msgr2=0x7fd8fc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 0x7fd8fc06d8a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9040017a0 tx=0x7fd904009040 comp rx=0 tx=0).stop 2023-11-24T08:09:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 msgr2=0x7fd91c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00e230 tx=0x7fd90c009750 comp rx=0 tx=0).stop 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 shutdown_connections 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd8fc06b3e0 0x7fd8fc06d8a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91c157190 0x7fd91c154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91c143b00 0x7fd91c156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 --2- 172.21.15.107:0/1398433732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9080039a0 0x7fd91c141220 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 >> 172.21.15.107:0/1398433732 conn(0x7fd91c06a500 msgr2=0x7fd91c15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 shutdown_connections 2023-11-24T08:09:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.469+0000 7fd921af1640 1 -- 172.21.15.107:0/1398433732 wait complete. 2023-11-24T08:09:10.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:09:10.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.692+0000 7fe7d359e640 1 Processor -- start 2023-11-24T08:09:10.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.692+0000 7fe7d359e640 1 -- start start 2023-11-24T08:09:10.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.693+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.693+0000 7fe7d359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7d40d0070 0x7fe7d40b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.693+0000 7fe7d359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40d04b0 con 0x7fe7d415d4e0 2023-11-24T08:09:10.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.693+0000 7fe7d359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d40d4110 con 0x7fe7d40d0070 2023-11-24T08:09:10.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.693+0000 7fe7d359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40ce990 con 0x7fe7d40d1e20 2023-11-24T08:09:10.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.694+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.694+0000 7fe7d2d9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7d40d0070 0x7fe7d40b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.694+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:50608/0 (socket says 172.21.15.107:50608) 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.694+0000 7fe7cbfff640 1 -- 172.21.15.107:0/3814755178 learned_addr learned my addr 172.21.15.107:0/3814755178 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.695+0000 7fe7cbfff640 1 -- 172.21.15.107:0/3814755178 >> v2:172.21.15.120:3300/0 conn(0x7fe7d40d0070 msgr2=0x7fe7d40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.695+0000 7fe7cbfff640 1 --2- 172.21.15.107:0/3814755178 >> v2:172.21.15.120:3300/0 conn(0x7fe7d40d0070 0x7fe7d40b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.695+0000 7fe7cbfff640 1 -- 172.21.15.107:0/3814755178 >> v1:172.21.15.120:6789/0 conn(0x7fe7d415d4e0 legacy=0x7fe7d40d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.695+0000 7fe7cbfff640 1 -- 172.21.15.107:0/3814755178 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40cec70 con 0x7fe7d40d1e20 2023-11-24T08:09:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.696+0000 7fe7cbfff640 1 --2- 172.21.15.107:0/3814755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009870 tx=0x7fe7bc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d77cdf7b308861a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d1d9b640 1 -- 172.21.15.107:0/3814755178 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc030840 con 0x7fe7d40d1e20 2023-11-24T08:09:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d1d9b640 1 -- 172.21.15.107:0/3814755178 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc0309a0 con 0x7fe7d40d1e20 2023-11-24T08:09:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d1d9b640 1 -- 172.21.15.107:0/3814755178 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc030d30 con 0x7fe7d40d1e20 2023-11-24T08:09:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 -- 172.21.15.107:0/3814755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 msgr2=0x7fe7d40cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 --2- 172.21.15.107:0/3814755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009870 tx=0x7fe7bc02b980 comp rx=0 tx=0).stop 2023-11-24T08:09:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 -- 172.21.15.107:0/3814755178 shutdown_connections 2023-11-24T08:09:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 --2- 172.21.15.107:0/3814755178 >> v2:172.21.15.120:3300/0 conn(0x7fe7d40d0070 0x7fe7d40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 --2- 172.21.15.107:0/3814755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d1e20 0x7fe7d40cfb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 -- 172.21.15.107:0/3814755178 >> 172.21.15.107:0/3814755178 conn(0x7fe7d409c510 msgr2=0x7fe7d40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 -- 172.21.15.107:0/3814755178 shutdown_connections 2023-11-24T08:09:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.697+0000 7fe7d359e640 1 -- 172.21.15.107:0/3814755178 wait complete. 2023-11-24T08:09:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.699+0000 7fe7d359e640 1 Processor -- start 2023-11-24T08:09:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.699+0000 7fe7d359e640 1 -- start start 2023-11-24T08:09:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 0x7fe7d40ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 0x7fe7d40cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40cc540 con 0x7fe7d415d4e0 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40b7a90 con 0x7fe7d40d1e20 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40b7d70 con 0x7fe7d40d0070 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37378/0 (socket says 172.21.15.107:37378) 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.700+0000 7fe7d2d9d640 1 -- 172.21.15.107:0/697266375 learned_addr learned my addr 172.21.15.107:0/697266375 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d259c640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 0x7fe7d40ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7cbfff640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 0x7fe7d40cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 -- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 msgr2=0x7fe7d40ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 0x7fe7d40ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 -- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 msgr2=0x7fe7d40cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 0x7fe7d40cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 -- 172.21.15.107:0/697266375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0094d0 con 0x7fe7d415d4e0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d259c640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 0x7fe7d40ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7cbfff640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 0x7fe7d40cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d2d9d640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00efc0 tx=0x7fe7cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc014480 con 0x7fe7d415d4e0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00f040 con 0x7fe7d415d4e0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc014820 con 0x7fe7d415d4e0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.701+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d40b8050 con 0x7fe7d415d4e0 2023-11-24T08:09:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.702+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d40b8610 con 0x7fe7d415d4e0 2023-11-24T08:09:10.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.703+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d404e840 con 0x7fe7d415d4e0 2023-11-24T08:09:10.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.703+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fe7cc014980 con 0x7fe7d415d4e0 2023-11-24T08:09:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.704+0000 7fe7caffd640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 0x7fe7b406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.704+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7cc08b080 con 0x7fe7d415d4e0 2023-11-24T08:09:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.705+0000 7fe7d259c640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 0x7fe7b406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.705+0000 7fe7d259c640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 0x7fe7b406d8a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4000f60 tx=0x7fe7c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:10.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.709+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7cc055800 con 0x7fe7d415d4e0 2023-11-24T08:09:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.932+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7d415cb50 con 0x7fe7d415d4e0 2023-11-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.934+0000 7fe7caffd640 1 -- 172.21.15.107:0/697266375 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fe7cc018090 con 0x7fe7d415d4e0 2023-11-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-11-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 msgr2=0x7fe7b406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 0x7fe7b406d8a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4000f60 tx=0x7fe7c4009040 comp rx=0 tx=0).stop 2023-11-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 msgr2=0x7fe7d40cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00efc0 tx=0x7fe7cc00c490 comp rx=0 tx=0).stop 2023-11-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 shutdown_connections 2023-11-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7b406b3e0 0x7fe7b406d8a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7d415d4e0 0x7fe7d40cce30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7d40d1e20 0x7fe7d40cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 --2- 172.21.15.107:0/697266375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7d40d0070 0x7fe7d40ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 >> 172.21.15.107:0/697266375 conn(0x7fe7d409c510 msgr2=0x7fe7d40b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 shutdown_connections 2023-11-24T08:09:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:10.936+0000 7fe7d359e640 1 -- 172.21.15.107:0/697266375 wait complete. 2023-11-24T08:09:11.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:09:04.481642+0000', 'modified': '2023-11-24T08:09:07.560596+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6231}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6231': {'gid': 6231, 'name': 'c', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6836/3235590318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3235590318}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 3235590318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:09:11.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:09:11.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:09:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.579+0000 7f351ca01640 1 Processor -- start 2023-11-24T08:09:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.579+0000 7f351ca01640 1 -- start start 2023-11-24T08:09:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.580+0000 7f351ca01640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f351815a1c0 0x7f3518070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.580+0000 7f351ca01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f351806d640 0x7f351806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.580+0000 7f351ca01640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351806df90 con 0x7f35180b15e0 2023-11-24T08:09:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.580+0000 7f351ca01640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180cd920 con 0x7f351815a1c0 2023-11-24T08:09:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.580+0000 7f351ca01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180b1a50 con 0x7f351806d640 2023-11-24T08:09:11.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f3516d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f351815a1c0 0x7f3518070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f3516d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f351815a1c0 0x7f3518070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37382/0 (socket says 172.21.15.107:37382) 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f3516d9d640 1 -- 172.21.15.107:0/4166993334 learned_addr learned my addr 172.21.15.107:0/4166993334 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f350ffff640 1 --1- 172.21.15.107:0/4166993334 >> v1:172.21.15.107:6789/0 conn(0x7f35180b15e0 0x7f351815dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44396/0 (socket says 172.21.15.107:44396) 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f351759e640 1 --2- 172.21.15.107:0/4166993334 >> v2:172.21.15.120:3300/0 conn(0x7f351806d640 0x7f351806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f3516d9d640 1 -- 172.21.15.107:0/4166993334 >> v2:172.21.15.120:3300/0 conn(0x7f351806d640 msgr2=0x7f351806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f3516d9d640 1 --2- 172.21.15.107:0/4166993334 >> v2:172.21.15.120:3300/0 conn(0x7f351806d640 0x7f351806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.582+0000 7f3516d9d640 1 -- 172.21.15.107:0/4166993334 >> v1:172.21.15.107:6789/0 conn(0x7f35180b15e0 legacy=0x7f351815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.582+0000 7f3516d9d640 1 -- 172.21.15.107:0/4166993334 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351815e580 con 0x7f351815a1c0 2023-11-24T08:09:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.581+0000 7f351659c640 1 -- 172.21.15.107:0/4166993334 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152758990 0 0) 0x7f351806df90 con 0x7f35180b15e0 2023-11-24T08:09:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f3516d9d640 1 --2- 172.21.15.107:0/4166993334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351815a1c0 0x7f3518070ea0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f35180729e0 tx=0x7f350802e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e68c2c7e5f713d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f351659c640 1 -- 172.21.15.107:0/4166993334 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508036440 con 0x7f351815a1c0 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f351659c640 1 -- 172.21.15.107:0/4166993334 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35080365a0 con 0x7f351815a1c0 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f351659c640 1 -- 172.21.15.107:0/4166993334 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508036930 con 0x7f351815a1c0 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f351ca01640 1 -- 172.21.15.107:0/4166993334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351815a1c0 msgr2=0x7f3518070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.583+0000 7f351ca01640 1 --2- 172.21.15.107:0/4166993334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351815a1c0 0x7f3518070ea0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f35180729e0 tx=0x7f350802e200 comp rx=0 tx=0).stop 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 -- 172.21.15.107:0/4166993334 shutdown_connections 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 --2- 172.21.15.107:0/4166993334 >> v2:172.21.15.120:3300/0 conn(0x7f351806d640 0x7f351806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 --2- 172.21.15.107:0/4166993334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351815a1c0 0x7f3518070ea0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 -- 172.21.15.107:0/4166993334 >> 172.21.15.107:0/4166993334 conn(0x7f35180cbb40 msgr2=0x7f35180cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 -- 172.21.15.107:0/4166993334 shutdown_connections 2023-11-24T08:09:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.584+0000 7f351ca01640 1 -- 172.21.15.107:0/4166993334 wait complete. 2023-11-24T08:09:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.585+0000 7f351ca01640 1 Processor -- start 2023-11-24T08:09:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.585+0000 7f351ca01640 1 -- start start 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180b15e0 0x7f3518159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351815a1c0 0x7f351809bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351809a9a0 con 0x7f351806d640 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351815b330 con 0x7f35180b15e0 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f351ca01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3518099ff0 con 0x7f351815a1c0 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f3516d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.586+0000 7f3516d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37386/0 (socket says 172.21.15.107:37386) 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 -- 172.21.15.107:0/1357082000 learned_addr learned my addr 172.21.15.107:0/1357082000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 -- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351815a1c0 msgr2=0x7f351809bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351815a1c0 0x7f351809bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 -- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180b15e0 msgr2=0x7f3518159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180b15e0 0x7f3518159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 -- 172.21.15.107:0/1357082000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350800b770 con 0x7f351806d640 2023-11-24T08:09:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f350800bc00 tx=0x7f35080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f350f7fe640 1 -- 172.21.15.107:0/1357082000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508036440 con 0x7f351806d640 2023-11-24T08:09:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f350f7fe640 1 -- 172.21.15.107:0/1357082000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508004230 con 0x7f351806d640 2023-11-24T08:09:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351815b580 con 0x7f351806d640 2023-11-24T08:09:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.587+0000 7f350f7fe640 1 -- 172.21.15.107:0/1357082000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508034900 con 0x7f351806d640 2023-11-24T08:09:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.588+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3518099c80 con 0x7f351806d640 2023-11-24T08:09:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.590+0000 7f350f7fe640 1 -- 172.21.15.107:0/1357082000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f350803d070 con 0x7f351806d640 2023-11-24T08:09:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.591+0000 7f350f7fe640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 0x7f34f806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.591+0000 7f350f7fe640 1 -- 172.21.15.107:0/1357082000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35080ac5a0 con 0x7f351806d640 2023-11-24T08:09:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.591+0000 7f350ffff640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 0x7f34f806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.592+0000 7f351fd99740 1 Processor -- start 2023-11-24T08:09:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.592+0000 7f350ffff640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 0x7f34f806d8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3500007fd0 tx=0x7f3500007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.592+0000 7f351fd99740 1 -- start start 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac717450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac75a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac675880 con 0x55acac711d70 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac684ae0 con 0x55acac7167b0 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acac6821d0 con 0x55acac717070 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37388/0 (socket says 172.21.15.107:37388) 2023-11-24T08:09:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 -- 172.21.15.107:0/4054005495 learned_addr learned my addr 172.21.15.107:0/4054005495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f3516d9d640 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac717450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 -- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 msgr2=0x55acac717450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac717450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f351759e640 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac75a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 -- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 msgr2=0x55acac75a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac75a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.593+0000 7f350ffff640 1 -- 172.21.15.107:0/4054005495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acac645860 con 0x55acac711d70 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f350ffff640 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3500007520 tx=0x7f350000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89870b27dbc10a2d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f350cff9640 1 -- 172.21.15.107:0/4054005495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350001b070 con 0x55acac711d70 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f350cff9640 1 -- 172.21.15.107:0/4054005495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500007610 con 0x55acac711d70 2023-11-24T08:09:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f350cff9640 1 -- 172.21.15.107:0/4054005495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500005e80 con 0x55acac711d70 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 -- 172.21.15.107:0/4054005495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 msgr2=0x55acac716270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3500007520 tx=0x7f350000f000 comp rx=0 tx=0).stop 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 -- 172.21.15.107:0/4054005495 shutdown_connections 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac75a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac711d70 0x55acac716270 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 --2- 172.21.15.107:0/4054005495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac717450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 -- 172.21.15.107:0/4054005495 >> 172.21.15.107:0/4054005495 conn(0x55acac6d8c70 msgr2=0x55acac75cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.594+0000 7f351fd99740 1 -- 172.21.15.107:0/4054005495 shutdown_connections 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.595+0000 7f351fd99740 1 -- 172.21.15.107:0/4054005495 wait complete. 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.595+0000 7f351fd99740 1 -- start start 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.595+0000 7f351fd99740 1 Processor -- start 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac711d70 0x55acac73d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac7403d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac67dac0 con 0x55acac717070 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac61a2f0 con 0x55acac7167b0 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acac637c70 con 0x55acac711d70 2023-11-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.596+0000 7f351fd99740 10 client.0 _ll_register_callbacks cb 0x55acac63c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37392/0 (socket says 172.21.15.107:37392) 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 -- 172.21.15.107:0/349280280 learned_addr learned my addr 172.21.15.107:0/349280280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f350ffff640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac7403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f3516d9d640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac711d70 0x55acac73d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 -- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac711d70 msgr2=0x55acac73d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac711d70 0x55acac73d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 -- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 msgr2=0x55acac7403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac7403d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 -- 172.21.15.107:0/349280280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35000071d0 con 0x55acac717070 2023-11-24T08:09:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f351759e640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3510007980 tx=0x7f3510007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6a14389d03c7d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.597+0000 7f34f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000fc00 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f351000fd60 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acac676aa0 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f351001d420 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 10 client.6375 fetch_fsmap learned FSMap version 32 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55acac67c140 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55acac676aa0 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f351001d580 con 0x55acac717070 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 10 client.6375 fetch_fsmap finished waiting for FSMap version 32 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 10 client.6375 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:09:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 20 client.6375 populate_metadata read hostname 'smithi107' 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac761b50 0x55acac7703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 4 client.6375 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.598+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 --> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acac6767c0 con 0x55acac761b50 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.599+0000 7f3516d9d640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac761b50 0x55acac7703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.599+0000 7f3516d9d640 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac761b50 0x55acac7703d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a35dc40fb4714fbe server_cookie=55a1b87b0227a2fd in_seq=0 out_seq=0 2023-11-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.599+0000 7f34f5ffb640 10 client.6375 ms_handle_connect on v2:172.21.15.107:6834/3235590318 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f34f5ffb640 1 -- 172.21.15.107:0/349280280 <== mds.0 v2:172.21.15.107:6834/3235590318 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f35080a9a90 con 0x55acac761b50 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f34f5ffb640 10 client.6375 handle_command_reply: tid=0 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 client.6375 shutdown 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 msgr2=0x55acac761610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3510007980 tx=0x7f3510007e40 comp rx=0 tx=0).stop 2023-11-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 shutdown_connections 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac761b50 0x55acac7703d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac761610 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac7167b0 0x55acac7403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 --2- 172.21.15.107:0/349280280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac711d70 0x55acac73d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f34f5ffb640 0 client.6375 ms_handle_reset on v2:172.21.15.107:6834/3235590318 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 >> 172.21.15.107:0/349280280 conn(0x55acac6d8c70 msgr2=0x55acac717450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 shutdown_connections 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 1 -- 172.21.15.107:0/349280280 wait complete. 2023-11-24T08:09:11.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.600+0000 7f351fd99740 20 client.6375 trim_cache size 0 max 16384 2023-11-24T08:09:11.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.617+0000 7f351fd99740 1 Processor -- start 2023-11-24T08:09:11.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.617+0000 7f351fd99740 1 -- start start 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac759cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac711310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac3e0f20 con 0x55acac7167b0 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac631e30 con 0x55acac711d70 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acac740dc0 con 0x55acac717070 2023-11-24T08:09:11.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f350ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f350ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37404/0 (socket says 172.21.15.107:37404) 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.618+0000 7f350ffff640 1 -- 172.21.15.107:0/3005204985 learned_addr learned my addr 172.21.15.107:0/3005204985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 -- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 msgr2=0x55acac711310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac711310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 -- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 msgr2=0x55acac759cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac759cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 -- 172.21.15.107:0/3005204985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acac619b80 con 0x55acac7167b0 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350ffff640 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3500015000 tx=0x7f3500007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9425acce9a5402b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350d7fa640 1 -- 172.21.15.107:0/3005204985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500025070 con 0x55acac7167b0 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350d7fa640 1 -- 172.21.15.107:0/3005204985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500005c80 con 0x55acac7167b0 2023-11-24T08:09:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.619+0000 7f350d7fa640 1 -- 172.21.15.107:0/3005204985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500016950 con 0x55acac7167b0 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 -- 172.21.15.107:0/3005204985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 msgr2=0x55acac710dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3500015000 tx=0x7f3500007000 comp rx=0 tx=0).stop 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 -- 172.21.15.107:0/3005204985 shutdown_connections 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac717070 0x55acac711310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac7167b0 0x55acac710dd0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 --2- 172.21.15.107:0/3005204985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac759cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 -- 172.21.15.107:0/3005204985 >> 172.21.15.107:0/3005204985 conn(0x55acac6d8c70 msgr2=0x55acac763660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 -- 172.21.15.107:0/3005204985 shutdown_connections 2023-11-24T08:09:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.620+0000 7f351fd99740 1 -- 172.21.15.107:0/3005204985 wait complete. 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.621+0000 7f351fd99740 1 -- start start 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.621+0000 7f351fd99740 1 Processor -- start 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac7c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac7167b0 0x55acac76c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac73ece0 con 0x55acac717070 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acac760820 con 0x55acac711d70 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acac73ea90 con 0x55acac7167b0 2023-11-24T08:09:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351fd99740 10 client.0 _ll_register_callbacks cb 0x7f35100097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37418/0 (socket says 172.21.15.107:37418) 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 -- 172.21.15.107:0/1902200498 learned_addr learned my addr 172.21.15.107:0/1902200498 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f350ffff640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac7167b0 0x55acac76c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 -- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac7167b0 msgr2=0x55acac76c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac7c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac7167b0 0x55acac76c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 -- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 msgr2=0x55acac7c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac7c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.622+0000 7f351759e640 1 -- 172.21.15.107:0/1902200498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acac645860 con 0x55acac717070 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f351759e640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55acac770330 tx=0x7f3510009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff57cb11bc4ec142 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:09:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100099a0 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f351000f520 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3500007610 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f351000fa00 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f351fd99740 10 client.6381 fetch_fsmap learned FSMap version 32 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55acac63f170 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500007610 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.623+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f351000fb60 con 0x55acac717070 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 10 client.6381 fetch_fsmap finished waiting for FSMap version 32 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 10 client.6381 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 20 client.6381 populate_metadata read hostname 'smithi107' 2023-11-24T08:09:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac75fe90 0x55acac760270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 4 client.6381 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:09:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 --> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55acac37e6c0 con 0x55acac75fe90 2023-11-24T08:09:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac75fe90 0x55acac760270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f3516d9d640 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac75fe90 0x55acac760270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c46ac8ab6fecf server_cookie=401889291548e932 in_seq=0 out_seq=0 2023-11-24T08:09:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f34f5ffb640 10 client.6381 ms_handle_connect on v2:172.21.15.107:6834/3235590318 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f34f5ffb640 1 -- 172.21.15.107:0/1902200498 <== mds.0 v2:172.21.15.107:6834/3235590318 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55acac37e6c0 con 0x55acac75fe90 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f34f5ffb640 10 client.6381 handle_command_reply: tid=0 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.624+0000 7f351fd99740 1 client.6381 shutdown 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 msgr2=0x55acac73dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55acac770330 tx=0x7f3510009450 comp rx=0 tx=0).stop 2023-11-24T08:09:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 shutdown_connections 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:6834/3235590318,v1:172.21.15.107:6836/3235590318] conn(0x55acac75fe90 0x55acac760270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acac717070 0x55acac73dc30 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acac7167b0 0x55acac76c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 --2- 172.21.15.107:0/1902200498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acac711d70 0x55acac7c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f34f5ffb640 0 client.6381 ms_handle_reset on v2:172.21.15.107:6834/3235590318 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 >> 172.21.15.107:0/1902200498 conn(0x55acac6d8c70 msgr2=0x55acac740340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 shutdown_connections 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 1 -- 172.21.15.107:0/1902200498 wait complete. 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351fd99740 20 client.6381 trim_cache size 0 max 16384 2023-11-24T08:09:11.632 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 6231, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.5863038510000003, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 32, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 64, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 64, 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.019076768 2023-11-24T08:09:11.633 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:09:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 msgr2=0x7f34f806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 0x7f34f806d8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3500007fd0 tx=0x7f3500007920 comp rx=0 tx=0).stop 2023-11-24T08:09:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 msgr2=0x7f351809a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.625+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f350800bc00 tx=0x7f35080046c0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 shutdown_connections 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f806b410 0x7f34f806d8d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351815a1c0 0x7f351809bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180b15e0 0x7f3518159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 --2- 172.21.15.107:0/1357082000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351806d640 0x7f351809a3c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 >> 172.21.15.107:0/1357082000 conn(0x7f35180cbb40 msgr2=0x7f35180cf890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 shutdown_connections 2023-11-24T08:09:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.626+0000 7f351ca01640 1 -- 172.21.15.107:0/1357082000 wait complete. 2023-11-24T08:09:11.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:09:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.837+0000 7fd8b4f78640 1 Processor -- start 2023-11-24T08:09:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.837+0000 7fd8b4f78640 1 -- start start 2023-11-24T08:09:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.838+0000 7fd8b4f78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.838+0000 7fd8b4f78640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8b00cdfb0 0x7fd8b00ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.838+0000 7fd8b4f78640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00cece0 con 0x7fd8b015c7f0 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.838+0000 7fd8b4f78640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b0156010 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.838+0000 7fd8b4f78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b006ce30 con 0x7fd8b00cf450 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8b00cdfb0 0x7fd8b00ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8b00cdfb0 0x7fd8b00ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37430/0 (socket says 172.21.15.107:37430) 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1842897823 learned_addr learned my addr 172.21.15.107:0/1842897823 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8ae59c640 1 --1- 172.21.15.107:0/1842897823 >> v1:172.21.15.107:6789/0 conn(0x7fd8b015c7f0 0x7fd8b006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44400/0 (socket says 172.21.15.107:44400) 2023-11-24T08:09:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8add9b640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1842897823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 msgr2=0x7fd8b00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b00cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1842897823 >> v1:172.21.15.107:6789/0 conn(0x7fd8b015c7f0 legacy=0x7fd8b006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1842897823 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b006d040 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.839+0000 7fd8add9b640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.840+0000 7fd8aed9d640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b00ce440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009870 tx=0x7fd8a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e563f29a740ced98 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.840+0000 7fd8ad59a640 1 -- 172.21.15.107:0/1842897823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8030840 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.840+0000 7fd8ad59a640 1 -- 172.21.15.107:0/1842897823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a80309a0 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.840+0000 7fd8ad59a640 1 -- 172.21.15.107:0/1842897823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8030d30 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1842897823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00cdfb0 msgr2=0x7fd8b00ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b00ce440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009870 tx=0x7fd8a802b980 comp rx=0 tx=0).stop 2023-11-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1842897823 shutdown_connections 2023-11-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b00ce440 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1842897823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1842897823 >> 172.21.15.107:0/1842897823 conn(0x7fd8b006a920 msgr2=0x7fd8b006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1842897823 shutdown_connections 2023-11-24T08:09:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.841+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1842897823 wait complete. 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.843+0000 7fd8b4f78640 1 Processor -- start 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.843+0000 7fd8b4f78640 1 -- start start 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b01f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b0154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01f3790 con 0x7fd8b015c7f0 2023-11-24T08:09:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00ad010 con 0x7fd8b00cdfb0 2023-11-24T08:09:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8b4f78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b4030 con 0x7fd8b00cf450 2023-11-24T08:09:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37446/0 (socket says 172.21.15.107:37446) 2023-11-24T08:09:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.844+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1593359438 learned_addr learned my addr 172.21.15.107:0/1593359438 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8ae59c640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b01f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8add9b640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b0154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 msgr2=0x7fd8b0154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b0154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b00cdfb0 msgr2=0x7fd8b01f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b01f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 -- 172.21.15.107:0/1593359438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80094d0 con 0x7fd8b015c7f0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8aed9d640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009870 tx=0x7fd8a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8039d60 con 0x7fd8b015c7f0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a8043400 con 0x7fd8b015c7f0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80437b0 con 0x7fd8b015c7f0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.845+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00b3d50 con 0x7fd8b015c7f0 2023-11-24T08:09:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.846+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00acca0 con 0x7fd8b015c7f0 2023-11-24T08:09:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.847+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b00ad180 con 0x7fd8b015c7f0 2023-11-24T08:09:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.848+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fd8a8043910 con 0x7fd8b015c7f0 2023-11-24T08:09:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.848+0000 7fd89effd640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 0x7fd89006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.849+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8a803e070 con 0x7fd8b015c7f0 2023-11-24T08:09:11.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.849+0000 7fd8ae59c640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 0x7fd89006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:11.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.849+0000 7fd8ae59c640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 0x7fd89006d8a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00012e0 tx=0x7fd8a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:11.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:11.853+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8a80dfe90 con 0x7fd8b015c7f0 2023-11-24T08:09:12.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.082+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd8b0067470 con 0x7fd8b015c7f0 2023-11-24T08:09:12.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.083+0000 7fd89effd640 1 -- 172.21.15.107:0/1593359438 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd8a8077a80 con 0x7fd8b015c7f0 2023-11-24T08:09:12.085 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.085+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 msgr2=0x7fd89006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.085+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 0x7fd89006d8a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00012e0 tx=0x7fd8a0009040 comp rx=0 tx=0).stop 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.085+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 msgr2=0x7fd8b00af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.085+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009870 tx=0x7fd8a8004730 comp rx=0 tx=0).stop 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 shutdown_connections 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd89006b3e0 0x7fd89006d8a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b015c7f0 0x7fd8b00af590 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:12.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b00cf450 0x7fd8b0154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:12.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 --2- 172.21.15.107:0/1593359438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b00cdfb0 0x7fd8b01f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:12.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 >> 172.21.15.107:0/1593359438 conn(0x7fd8b006a920 msgr2=0x7fd8b00aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:12.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 shutdown_connections 2023-11-24T08:09:12.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:12.086+0000 7fd8b4f78640 1 -- 172.21.15.107:0/1593359438 wait complete. 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:09:12.108 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:09:12.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.109 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:09:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:09:12.160 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:09:12.160 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:09:12.160 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:12.161 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42118sec preferred_lft 42118sec 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:09:12.162 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:09:12.163 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:12.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.164 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:09:12.249 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:09:12.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.254 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:09:12.307 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:09:12.307 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:09:12.307 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:09:12.307 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:09:12.307 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:09:12.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:09:12.308 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:09:12.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.476 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:09:12.529 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:09:12.529 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:09:12.530 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:09:12.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:09:12.625 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:09:12.625 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:09:12.680 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:09:12.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:09:12.681 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:09:12.729 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:09:12.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:12.866 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:12.867 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:09:12.899 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:12.899 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:09:12.899 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:09:12.899 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:09:12.899 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:09:12.901 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:09:12.901 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:09:12.901 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:09:12.901 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:09:12.901 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:09:12.904 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:09:12.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:09:12.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:12.985 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:09:13.046 INFO:teuthology.orchestra.run.smithi107.stdout:client4949 2023-11-24T08:09:13.047 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:13.047 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:13.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:09:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.303+0000 7f728e259640 1 Processor -- start 2023-11-24T08:09:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.303+0000 7f728e259640 1 -- start start 2023-11-24T08:09:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.304+0000 7f728e259640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7288144b30 0x7f728815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.304+0000 7f728e259640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72880b1de0 0x7f72880b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.304+0000 7f728e259640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728815bf80 con 0x7f728815c410 2023-11-24T08:09:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.304+0000 7f728e259640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72880b27b0 con 0x7f72880b1de0 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.304+0000 7f728e259640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72880b1560 con 0x7f7288144b30 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72880b1de0 0x7f72880b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72880b1de0 0x7f72880b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37458/0 (socket says 172.21.15.107:37458) 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 -- 172.21.15.107:0/3364557943 learned_addr learned my addr 172.21.15.107:0/3364557943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728ca56640 1 --1- 172.21.15.107:0/3364557943 >> v1:172.21.15.107:6789/0 conn(0x7f728815c410 0x7f72880b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44402/0 (socket says 172.21.15.107:44402) 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728d257640 1 --2- 172.21.15.107:0/3364557943 >> v2:172.21.15.120:3300/0 conn(0x7f7288144b30 0x7f728815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 -- 172.21.15.107:0/3364557943 >> v2:172.21.15.120:3300/0 conn(0x7f7288144b30 msgr2=0x7f728815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 --2- 172.21.15.107:0/3364557943 >> v2:172.21.15.120:3300/0 conn(0x7f7288144b30 0x7f728815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 -- 172.21.15.107:0/3364557943 >> v1:172.21.15.107:6789/0 conn(0x7f728815c410 legacy=0x7f72880b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f728da58640 1 -- 172.21.15.107:0/3364557943 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72880b1280 con 0x7f72880b1de0 2023-11-24T08:09:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.305+0000 7f727bfff640 1 -- 172.21.15.107:0/3364557943 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101796927 0 0) 0x7f728815bf80 con 0x7f728815c410 2023-11-24T08:09:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.306+0000 7f728da58640 1 --2- 172.21.15.107:0/3364557943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f72880b2270 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f728806f690 tx=0x7f728002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346e85b3365e3978 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f727bfff640 1 -- 172.21.15.107:0/3364557943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728002cb20 con 0x7f72880b1de0 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f727bfff640 1 -- 172.21.15.107:0/3364557943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728002cc80 con 0x7f72880b1de0 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f727bfff640 1 -- 172.21.15.107:0/3364557943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72800343f0 con 0x7f72880b1de0 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 -- 172.21.15.107:0/3364557943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 msgr2=0x7f72880b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 --2- 172.21.15.107:0/3364557943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f72880b2270 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f728806f690 tx=0x7f728002bf80 comp rx=0 tx=0).stop 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 -- 172.21.15.107:0/3364557943 shutdown_connections 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 --2- 172.21.15.107:0/3364557943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f72880b2270 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 --2- 172.21.15.107:0/3364557943 >> v2:172.21.15.120:3300/0 conn(0x7f7288144b30 0x7f728815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 -- 172.21.15.107:0/3364557943 >> 172.21.15.107:0/3364557943 conn(0x7f72880cb830 msgr2=0x7f72880cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 -- 172.21.15.107:0/3364557943 shutdown_connections 2023-11-24T08:09:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.307+0000 7f728e259640 1 -- 172.21.15.107:0/3364557943 wait complete. 2023-11-24T08:09:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.309+0000 7f728e259640 1 Processor -- start 2023-11-24T08:09:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.310+0000 7f728e259640 1 -- start start 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 0x7f728815cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728815c410 0x7f72880b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288142530 con 0x7f72880b1de0 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72880a8930 con 0x7f7288144b30 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728e259640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288141e70 con 0x7f728815c410 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728d257640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728d257640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37468/0 (socket says 172.21.15.107:37468) 2023-11-24T08:09:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728d257640 1 -- 172.21.15.107:0/4264685061 learned_addr learned my addr 172.21.15.107:0/4264685061 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.311+0000 7f728da58640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728815c410 0x7f72880b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728ca56640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 0x7f728815cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 -- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728815c410 msgr2=0x7f72880b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728815c410 0x7f72880b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 -- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 msgr2=0x7f728815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 0x7f728815cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 -- 172.21.15.107:0/4264685061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280009b30 con 0x7f72880b1de0 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728ca56640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 0x7f728815cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728d257640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f728400d800 tx=0x7f728400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728400b6c0 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728400b820 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728400bbb0 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7288142150 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.312+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728815d0a0 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.313+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72881427a0 con 0x7f72880b1de0 2023-11-24T08:09:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.314+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f728400bd10 con 0x7f72880b1de0 2023-11-24T08:09:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.315+0000 7f7279ffb640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 0x7f726c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.315+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f728408a620 con 0x7f72880b1de0 2023-11-24T08:09:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.316+0000 7f728ca56640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 0x7f726c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.316+0000 7f728ca56640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 0x7f726c06d8a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f72880b4d90 tx=0x7f727c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.320+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7284054e60 con 0x7f72880b1de0 2023-11-24T08:09:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.550+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f728804e840 con 0x7f72880b1de0 2023-11-24T08:09:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.551+0000 7f7279ffb640 1 -- 172.21.15.107:0/4264685061 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7284058480 con 0x7f72880b1de0 2023-11-24T08:09:13.553 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 msgr2=0x7f726c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 0x7f726c06d8a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f72880b4d90 tx=0x7f727c009040 comp rx=0 tx=0).stop 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 msgr2=0x7f728813eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f728400d800 tx=0x7f728400dcc0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 shutdown_connections 2023-11-24T08:09:13.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f726c06b3e0 0x7f726c06d8a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728815c410 0x7f72880b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7288144b30 0x7f728815cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 --2- 172.21.15.107:0/4264685061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72880b1de0 0x7f728813eae0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 >> 172.21.15.107:0/4264685061 conn(0x7f72880cb830 msgr2=0x7f728813fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 shutdown_connections 2023-11-24T08:09:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:13.554+0000 7f728e259640 1 -- 172.21.15.107:0/4264685061 wait complete. 2023-11-24T08:09:13.577 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:09:13.577 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:09:13.577 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:09:13.577 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:09:13.578 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:09:13.578 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:09:13.578 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:09:13.578 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:09:13.578 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:09:13.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:13.578 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:09:13.633 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42115sec preferred_lft 42115sec 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:09:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:09:13.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:13.637 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:09:13.721 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:09:13.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:13.725 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:09:13.779 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:09:13.780 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:09:13.780 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:09:13.780 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:09:13.780 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:09:13.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:13.780 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:09:13.780 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:09:13.780 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:09:13.780 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:09:13.781 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:09:13.781 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:09:13.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:13.951 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:09:13.999 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:09:14.000 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:09:14.001 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:09:14.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:14.001 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:09:14.089 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:09:14.089 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:09:14.142 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:09:14.142 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:09:14.142 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:09:14.192 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:09:14.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:14.192 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:14.320 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:14.320 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:09:14.357 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:09:14.358 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:09:14.358 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:09:14.358 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:09:14.358 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:09:14.359 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:09:14.359 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:09:14.359 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:09:14.359 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:09:14.359 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:09:14.363 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:09:14.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:14.363 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:09:14.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:14.443 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:09:14.501 INFO:teuthology.orchestra.run.smithi120.stdout:client6402 2023-11-24T08:09:14.501 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:09:14.502 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:14.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:09:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.779+0000 7fa2f87de640 1 Processor -- start 2023-11-24T08:09:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.779+0000 7fa2f87de640 1 -- start start 2023-11-24T08:09:14.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f87de640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa2f00cf500 0x7fa2f00cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f87de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00afb60 con 0x7fa2f00b1790 2023-11-24T08:09:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f87de640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00b0120 con 0x7fa2f00af440 2023-11-24T08:09:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f87de640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f00b1b70 con 0x7fa2f00cf500 2023-11-24T08:09:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f657a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa2f00b1790 0x7fa2f006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44410/0 (socket says 172.21.15.107:44410) 2023-11-24T08:09:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f657a640 1 -- 172.21.15.107:0/4252265522 learned_addr learned my addr 172.21.15.107:0/4252265522 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/4252265522 >> v2:172.21.15.120:3300/0 conn(0x7fa2f00cf500 0x7fa2f00cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842160135 0 0) 0x7fa2f00afb60 con 0x7fa2f00b1790 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.781+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc003520 con 0x7fa2f00b1790 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496026344 0 0) 0x7fa2dc003520 con 0x7fa2f00b1790 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 >> v2:172.21.15.120:3300/0 conn(0x7fa2f00cf500 msgr2=0x7fa2f00cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 --2- 172.21.15.107:0/4252265522 >> v2:172.21.15.120:3300/0 conn(0x7fa2f00cf500 0x7fa2f00cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 >> v1:172.21.15.120:6789/0 conn(0x7fa2f00af440 legacy=0x7fa2f00b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f006ace0 con 0x7fa2f00b1790 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa2ec001670 con 0x7fa2f00b1790 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 >> v1:172.21.15.107:6789/0 conn(0x7fa2f00b1790 legacy=0x7fa2f006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0039c0 0x7fa2dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2dc0047c0 0x7fa2dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 0x7fa2dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f00b1b70 con 0x7fa2dc0051d0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc00c120 con 0x7fa2dc0047c0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc00c2e0 con 0x7fa2dc0039c0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.782+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 0x7fa2dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f657a640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0039c0 0x7fa2dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f5d79640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2dc0047c0 0x7fa2dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0039c0 msgr2=0x7fa2dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:14.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0039c0 0x7fa2dc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2dc0047c0 msgr2=0x7fa2dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:14.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2dc0047c0 0x7fa2dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/4252265522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc003520 con 0x7fa2dc0051d0 2023-11-24T08:09:14.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.783+0000 7fa2f5d79640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2dc0047c0 0x7fa2dc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:14.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.784+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/4252265522 reap_dead start 2023-11-24T08:09:14.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.784+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 0x7fa2dc00b980 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009f90 tx=0x7fa2dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8865938767dd531f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:14.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.784+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e802cc20 con 0x7fa2dc0051d0 2023-11-24T08:09:14.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.784+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e800f040 con 0x7fa2dc0051d0 2023-11-24T08:09:14.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.784+0000 7fa2f5578640 1 -- 172.21.15.107:0/4252265522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8003990 con 0x7fa2dc0051d0 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 -- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 msgr2=0x7fa2dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 0x7fa2dc00b980 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009f90 tx=0x7fa2dc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 -- 172.21.15.107:0/4252265522 shutdown_connections 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 --2- 172.21.15.107:0/4252265522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2dc0051d0 0x7fa2dc00b980 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 -- 172.21.15.107:0/4252265522 >> 172.21.15.107:0/4252265522 conn(0x7fa2f00b1cf0 msgr2=0x7fa2f00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 -- 172.21.15.107:0/4252265522 shutdown_connections 2023-11-24T08:09:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.785+0000 7fa2f87de640 1 -- 172.21.15.107:0/4252265522 wait complete. 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.787+0000 7fa2f87de640 1 Processor -- start 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.787+0000 7fa2f87de640 1 -- start start 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0051d0 0x7fa2f014b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f00c9fe0 0x7fa2f0160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f00b4500 con 0x7fa2f015df20 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0160ad0 con 0x7fa2f00c9fe0 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f87de640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f00b3eb0 con 0x7fa2dc0051d0 2023-11-24T08:09:14.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f6d7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f6d7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37478/0 (socket says 172.21.15.107:37478) 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.788+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/1642392446 learned_addr learned my addr 172.21.15.107:0/1642392446 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f657a640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0051d0 0x7fa2f014b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f5d79640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f00c9fe0 0x7fa2f0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0051d0 msgr2=0x7fa2f014b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0051d0 0x7fa2f014b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f00c9fe0 msgr2=0x7fa2f0160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f00c9fe0 0x7fa2f0160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 -- 172.21.15.107:0/1642392446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00b4190 con 0x7fa2f015df20 2023-11-24T08:09:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.789+0000 7fa2f6d7b640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa2e802c690 tx=0x7fa2e8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.790+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8003da0 con 0x7fa2f015df20 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.790+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e802de90 con 0x7fa2f015df20 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.790+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8035b70 con 0x7fa2f015df20 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.790+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01606a0 con 0x7fa2f015df20 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.790+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f014d940 con 0x7fa2f015df20 2023-11-24T08:09:14.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.791+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f00b4320 con 0x7fa2f015df20 2023-11-24T08:09:14.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.792+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fa2e8035cd0 con 0x7fa2f015df20 2023-11-24T08:09:14.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.793+0000 7fa2e6ffd640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 0x7fa2d406d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:14.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.793+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2e80accf0 con 0x7fa2f015df20 2023-11-24T08:09:14.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.793+0000 7fa2f657a640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 0x7fa2d406d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:14.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.794+0000 7fa2f657a640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 0x7fa2d406d8d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec001050 tx=0x7fa2ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:14.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:14.797+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2e8077610 con 0x7fa2f015df20 2023-11-24T08:09:15.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.025+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa2f0160890 con 0x7fa2f015df20 2023-11-24T08:09:15.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.814+0000 7fa2e6ffd640 1 -- 172.21.15.107:0/1642392446 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa2f0160890 con 0x7fa2f015df20 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.817+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 msgr2=0x7fa2d406d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.817+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 0x7fa2d406d8d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec001050 tx=0x7fa2ec009040 comp rx=0 tx=0).stop 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.817+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 msgr2=0x7fa2f014d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.817+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa2e802c690 tx=0x7fa2e8009750 comp rx=0 tx=0).stop 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 shutdown_connections 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa2d406b410 0x7fa2d406d8d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2f015df20 0x7fa2f014d220 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f00c9fe0 0x7fa2f0160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 --2- 172.21.15.107:0/1642392446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2dc0051d0 0x7fa2f014b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 >> 172.21.15.107:0/1642392446 conn(0x7fa2f00b1cf0 msgr2=0x7fa2f006d720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 shutdown_connections 2023-11-24T08:09:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:15.818+0000 7fa2f87de640 1 -- 172.21.15.107:0/1642392446 wait complete. 2023-11-24T08:09:15.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:09:16.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.051+0000 7fca34797640 1 Processor -- start 2023-11-24T08:09:16.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.052+0000 7fca34797640 1 -- start start 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca34797640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 0x7fca2c0d4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca34797640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca2c0d1e30 0x7fca2c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca34797640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0cf300 con 0x7fca2c0cfc80 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca34797640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0d1090 con 0x7fca2c0d1e30 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca34797640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0d1370 con 0x7fca2c15d580 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca31d32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca2c0d1e30 0x7fca2c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca31d32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca2c0d1e30 0x7fca2c0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37492/0 (socket says 172.21.15.107:37492) 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca31d32640 1 -- 172.21.15.107:0/3745205542 learned_addr learned my addr 172.21.15.107:0/3745205542 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.053+0000 7fca32d34640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fca2c0cfc80 0x7fca2c0d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44412/0 (socket says 172.21.15.107:44412) 2023-11-24T08:09:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca32533640 1 --2- 172.21.15.107:0/3745205542 >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 0x7fca2c0d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca31d32640 1 -- 172.21.15.107:0/3745205542 >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 msgr2=0x7fca2c0d4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca31d32640 1 --2- 172.21.15.107:0/3745205542 >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 0x7fca2c0d4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca31d32640 1 -- 172.21.15.107:0/3745205542 >> v1:172.21.15.107:6789/0 conn(0x7fca2c0cfc80 legacy=0x7fca2c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca31d32640 1 -- 172.21.15.107:0/3745205542 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0d1650 con 0x7fca2c0d1e30 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca31531640 1 -- 172.21.15.107:0/3745205542 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126165635 0 0) 0x7fca2c0cf300 con 0x7fca2c0cfc80 2023-11-24T08:09:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.054+0000 7fca32533640 1 --2- 172.21.15.107:0/3745205542 >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 0x7fca2c0d4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.055+0000 7fca31d32640 1 --2- 172.21.15.107:0/3745205542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c0d1e30 0x7fca2c0cead0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fca2c072b90 tx=0x7fca2002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26886908712e60c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.055+0000 7fca31531640 1 -- 172.21.15.107:0/3745205542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20030880 con 0x7fca2c0d1e30 2023-11-24T08:09:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.055+0000 7fca31531640 1 -- 172.21.15.107:0/3745205542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca200309e0 con 0x7fca2c0d1e30 2023-11-24T08:09:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.055+0000 7fca31531640 1 -- 172.21.15.107:0/3745205542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20030d70 con 0x7fca2c0d1e30 2023-11-24T08:09:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 -- 172.21.15.107:0/3745205542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c0d1e30 msgr2=0x7fca2c0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 --2- 172.21.15.107:0/3745205542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c0d1e30 0x7fca2c0cead0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fca2c072b90 tx=0x7fca2002b9c0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 -- 172.21.15.107:0/3745205542 shutdown_connections 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 --2- 172.21.15.107:0/3745205542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c0d1e30 0x7fca2c0cead0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 --2- 172.21.15.107:0/3745205542 >> v2:172.21.15.120:3300/0 conn(0x7fca2c15d580 0x7fca2c0d4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 -- 172.21.15.107:0/3745205542 >> 172.21.15.107:0/3745205542 conn(0x7fca2c09c770 msgr2=0x7fca2c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 -- 172.21.15.107:0/3745205542 shutdown_connections 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.056+0000 7fca34797640 1 -- 172.21.15.107:0/3745205542 wait complete. 2023-11-24T08:09:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.058+0000 7fca34797640 1 Processor -- start 2023-11-24T08:09:16.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.058+0000 7fca34797640 1 -- start start 2023-11-24T08:09:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0cfc80 0x7fca2c1f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c0d1e30 0x7fca2c1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0c89d0 con 0x7fca2c15d580 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0c7510 con 0x7fca2c0d1e30 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca34797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c149d10 con 0x7fca2c0cfc80 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca32d34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca32d34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37500/0 (socket says 172.21.15.107:37500) 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca32d34640 1 -- 172.21.15.107:0/3336726149 learned_addr learned my addr 172.21.15.107:0/3336726149 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca31d32640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c0d1e30 0x7fca2c1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.059+0000 7fca32533640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0cfc80 0x7fca2c1f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 -- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0cfc80 msgr2=0x7fca2c1f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0cfc80 0x7fca2c1f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 -- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c0d1e30 msgr2=0x7fca2c1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c0d1e30 0x7fca2c1497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 -- 172.21.15.107:0/3336726149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200094d0 con 0x7fca2c15d580 2023-11-24T08:09:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca32d34640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fca2400c7e0 tx=0x7fca2400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2400a640 con 0x7fca2c15d580 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2400a7c0 con 0x7fca2c15d580 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2400ab70 con 0x7fca2c15d580 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c0c7230 con 0x7fca2c15d580 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.060+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c0c8ee0 con 0x7fca2c15d580 2023-11-24T08:09:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.062+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c04e8b0 con 0x7fca2c15d580 2023-11-24T08:09:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.062+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fca2400acd0 con 0x7fca2c15d580 2023-11-24T08:09:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.063+0000 7fca2affd640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 0x7fca1806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.063+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca24089690 con 0x7fca2c15d580 2023-11-24T08:09:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.064+0000 7fca32533640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 0x7fca1806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.064+0000 7fca32533640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 0x7fca1806d8a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00a950 tx=0x7fca1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.068+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca24053ee0 con 0x7fca2c15d580 2023-11-24T08:09:16.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.281+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fca2c0d22c0 con 0x7fca1806b3e0 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.283+0000 7fca2affd640 1 -- 172.21.15.107:0/3336726149 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fca1c00c050 con 0x7fca1806b3e0 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:16.285 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 msgr2=0x7fca1806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 0x7fca1806d8a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00a950 tx=0x7fca1c008040 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 msgr2=0x7fca2c14c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fca2400c7e0 tx=0x7fca2400cca0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 shutdown_connections 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fca1806b3e0 0x7fca1806d8a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15d580 0x7fca2c14c200 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c0d1e30 0x7fca2c1497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 --2- 172.21.15.107:0/3336726149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0cfc80 0x7fca2c1f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.285+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 >> 172.21.15.107:0/3336726149 conn(0x7fca2c09c770 msgr2=0x7fca2c0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:16.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.286+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 shutdown_connections 2023-11-24T08:09:16.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.286+0000 7fca34797640 1 -- 172.21.15.107:0/3336726149 wait complete. 2023-11-24T08:09:16.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:09:16.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.520+0000 7fb9a28e0640 1 Processor -- start 2023-11-24T08:09:16.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.520+0000 7fb9a28e0640 1 -- start start 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb9a28e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c0d4010 0x7fb99c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb9a28e0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 0x7fb99c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb9a28e0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c0d05a0 con 0x7fb99c15d580 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb9a28e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0cf300 con 0x7fb99c0d4010 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb9a28e0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0d1090 con 0x7fb99c0cfc80 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb99b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c0d4010 0x7fb99c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb99b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c0d4010 0x7fb99c0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37508/0 (socket says 172.21.15.107:37508) 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.522+0000 7fb99b7fe640 1 -- 172.21.15.107:0/3553566771 learned_addr learned my addr 172.21.15.107:0/3553566771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb9a0e7d640 1 --2- 172.21.15.107:0/3553566771 >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 0x7fb99c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb99b7fe640 1 -- 172.21.15.107:0/3553566771 >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 msgr2=0x7fb99c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb99b7fe640 1 --2- 172.21.15.107:0/3553566771 >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 0x7fb99c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb99b7fe640 1 -- 172.21.15.107:0/3553566771 >> v1:172.21.15.120:6789/0 conn(0x7fb99c15d580 legacy=0x7fb99c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb99b7fe640 1 -- 172.21.15.107:0/3553566771 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0d1370 con 0x7fb99c0d4010 2023-11-24T08:09:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.523+0000 7fb9a0e7d640 1 --2- 172.21.15.107:0/3553566771 >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 0x7fb99c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.524+0000 7fb99b7fe640 1 --2- 172.21.15.107:0/3553566771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0d4010 0x7fb99c0cead0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0d09e0 tx=0x7fb98402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73af087581ee515f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.524+0000 7fb99affd640 1 -- 172.21.15.107:0/3553566771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984031870 con 0x7fb99c0d4010 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.524+0000 7fb99affd640 1 -- 172.21.15.107:0/3553566771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9840319d0 con 0x7fb99c0d4010 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.524+0000 7fb99affd640 1 -- 172.21.15.107:0/3553566771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984031d60 con 0x7fb99c0d4010 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 -- 172.21.15.107:0/3553566771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0d4010 msgr2=0x7fb99c0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/3553566771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0d4010 0x7fb99c0cead0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0d09e0 tx=0x7fb98402b910 comp rx=0 tx=0).stop 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 -- 172.21.15.107:0/3553566771 shutdown_connections 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/3553566771 >> v2:172.21.15.120:3300/0 conn(0x7fb99c0cfc80 0x7fb99c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/3553566771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0d4010 0x7fb99c0cead0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 -- 172.21.15.107:0/3553566771 >> 172.21.15.107:0/3553566771 conn(0x7fb99c09c770 msgr2=0x7fb99c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 -- 172.21.15.107:0/3553566771 shutdown_connections 2023-11-24T08:09:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.525+0000 7fb9a28e0640 1 -- 172.21.15.107:0/3553566771 wait complete. 2023-11-24T08:09:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.527+0000 7fb9a28e0640 1 Processor -- start 2023-11-24T08:09:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.527+0000 7fb9a28e0640 1 -- start start 2023-11-24T08:09:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb99c0cfc80 0x7fb99c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb99c0d4010 0x7fb99c158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c154770 con 0x7fb99c15d580 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0c6f00 con 0x7fb99c0cfc80 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a28e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0c7a40 con 0x7fb99c0d4010 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a0e7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a0e7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37512/0 (socket says 172.21.15.107:37512) 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb9a0e7d640 1 -- 172.21.15.107:0/1307686611 learned_addr learned my addr 172.21.15.107:0/1307686611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.528+0000 7fb99bfff640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb99c0cfc80 0x7fb99c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb99b7fe640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb99c0d4010 0x7fb99c158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 -- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb99c0d4010 msgr2=0x7fb99c158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb99c0d4010 0x7fb99c158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 -- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb99c0cfc80 msgr2=0x7fb99c154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb99c0cfc80 0x7fb99c154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 -- 172.21.15.107:0/1307686611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9840094d0 con 0x7fb99c15d580 2023-11-24T08:09:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a0e7d640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb99400c7c0 tx=0x7fb99400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994004230 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994004390 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994003850 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c158690 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.529+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c0c73f0 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.531+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c04e8b0 con 0x7fb99c15d580 2023-11-24T08:09:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.531+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fb9940039b0 con 0x7fb99c15d580 2023-11-24T08:09:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.532+0000 7fb998ff9640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 0x7fb97c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.532+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb99408a3d0 con 0x7fb99c15d580 2023-11-24T08:09:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.533+0000 7fb99bfff640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 0x7fb97c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.533+0000 7fb99bfff640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 0x7fb97c06d8a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb98c001db0 tx=0x7fb98c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.537+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb994054b70 con 0x7fb99c15d580 2023-11-24T08:09:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.726+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb99c0c7b80 con 0x7fb99c15d580 2023-11-24T08:09:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.727+0000 7fb998ff9640 1 -- 172.21.15.107:0/1307686611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb994058190 con 0x7fb99c15d580 2023-11-24T08:09:16.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 msgr2=0x7fb97c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 0x7fb97c06d8a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb98c001db0 tx=0x7fb98c009040 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 msgr2=0x7fb99c159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb99400c7c0 tx=0x7fb99400cc80 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 shutdown_connections 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97c06b3e0 0x7fb97c06d8a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c15d580 0x7fb99c159af0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb99c0d4010 0x7fb99c158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 --2- 172.21.15.107:0/1307686611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb99c0cfc80 0x7fb99c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.730+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 >> 172.21.15.107:0/1307686611 conn(0x7fb99c09c770 msgr2=0x7fb99c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.731+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 shutdown_connections 2023-11-24T08:09:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.731+0000 7fb9a28e0640 1 -- 172.21.15.107:0/1307686611 wait complete. 2023-11-24T08:09:16.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol2878 --mode=777 2023-11-24T08:09:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.965+0000 7f842359e640 1 Processor -- start 2023-11-24T08:09:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.965+0000 7f842359e640 1 -- start start 2023-11-24T08:09:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.966+0000 7f842359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84240b3600 0x7f84240cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.966+0000 7f842359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f842415ed60 0x7f842415f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.966+0000 7f842359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84240ccbe0 con 0x7f842415b400 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.966+0000 7f842359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84240b3a60 con 0x7f84240b3600 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.966+0000 7f842359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842415f730 con 0x7f842415ed60 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84240b3600 0x7f84240cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84240b3600 0x7f84240cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37514/0 (socket says 172.21.15.107:37514) 2023-11-24T08:09:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 -- 172.21.15.107:0/4095125488 learned_addr learned my addr 172.21.15.107:0/4095125488 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8422d9d640 1 --2- 172.21.15.107:0/4095125488 >> v2:172.21.15.120:3300/0 conn(0x7f842415ed60 0x7f842415f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 -- 172.21.15.107:0/4095125488 >> v2:172.21.15.120:3300/0 conn(0x7f842415ed60 msgr2=0x7f842415f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 --2- 172.21.15.107:0/4095125488 >> v2:172.21.15.120:3300/0 conn(0x7f842415ed60 0x7f842415f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 -- 172.21.15.107:0/4095125488 >> v1:172.21.15.107:6789/0 conn(0x7f842415b400 legacy=0x7f84240cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.967+0000 7f8421d9b640 1 -- 172.21.15.107:0/4095125488 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84240cebc0 con 0x7f84240b3600 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.968+0000 7f8421d9b640 1 --2- 172.21.15.107:0/4095125488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240b3600 0x7f84240cc6a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8424067cc0 tx=0x7f840c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1711287af2acfdde server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.968+0000 7f842159a640 1 -- 172.21.15.107:0/4095125488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c031870 con 0x7f84240b3600 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842159a640 1 -- 172.21.15.107:0/4095125488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840c0319d0 con 0x7f84240b3600 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842159a640 1 -- 172.21.15.107:0/4095125488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c031d60 con 0x7f84240b3600 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 -- 172.21.15.107:0/4095125488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240b3600 msgr2=0x7f84240cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 --2- 172.21.15.107:0/4095125488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240b3600 0x7f84240cc6a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8424067cc0 tx=0x7f840c02b910 comp rx=0 tx=0).stop 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 -- 172.21.15.107:0/4095125488 shutdown_connections 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 --2- 172.21.15.107:0/4095125488 >> v2:172.21.15.120:3300/0 conn(0x7f842415ed60 0x7f842415f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 --2- 172.21.15.107:0/4095125488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240b3600 0x7f84240cc6a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 -- 172.21.15.107:0/4095125488 >> 172.21.15.107:0/4095125488 conn(0x7f84240b2ae0 msgr2=0x7f84240b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 -- 172.21.15.107:0/4095125488 shutdown_connections 2023-11-24T08:09:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.969+0000 7f842359e640 1 -- 172.21.15.107:0/4095125488 wait complete. 2023-11-24T08:09:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.971+0000 7f842359e640 1 Processor -- start 2023-11-24T08:09:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.971+0000 7f842359e640 1 -- start start 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84240b3600 0x7f842415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f842415ed60 0x7f8424157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84240c9730 con 0x7f84240ca7d0 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84240c9a10 con 0x7f842415ed60 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f842359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84240c9cf0 con 0x7f84240b3600 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f8422d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f8422d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37530/0 (socket says 172.21.15.107:37530) 2023-11-24T08:09:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.972+0000 7f8422d9d640 1 -- 172.21.15.107:0/1598780317 learned_addr learned my addr 172.21.15.107:0/1598780317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8421d9b640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f842415ed60 0x7f8424157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f842259c640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84240b3600 0x7f842415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 -- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84240b3600 msgr2=0x7f842415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84240b3600 0x7f842415a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 -- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f842415ed60 msgr2=0x7f8424157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f842415ed60 0x7f8424157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 -- 172.21.15.107:0/1598780317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c0094d0 con 0x7f84240ca7d0 2023-11-24T08:09:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f8422d9d640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f841c00d800 tx=0x7f841c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c00b6c0 con 0x7f84240ca7d0 2023-11-24T08:09:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c00b840 con 0x7f84240ca7d0 2023-11-24T08:09:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c00bbf0 con 0x7f84240ca7d0 2023-11-24T08:09:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.973+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84240c9fd0 con 0x7f84240ca7d0 2023-11-24T08:09:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.974+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842406ce00 con 0x7f84240ca7d0 2023-11-24T08:09:16.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.975+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842406cf90 con 0x7f84240ca7d0 2023-11-24T08:09:16.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.976+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7f841c00bd50 con 0x7f84240ca7d0 2023-11-24T08:09:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.976+0000 7f841affd640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 0x7f840406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.977+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f841c08a650 con 0x7f84240ca7d0 2023-11-24T08:09:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.977+0000 7f842259c640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 0x7f840406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.977+0000 7f842259c640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 0x7f840406d8a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8414007fd0 tx=0x7f8414009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:16.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:16.981+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f841c054e50 con 0x7f84240ca7d0 2023-11-24T08:09:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.195+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol2878", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f84240cf0f0 con 0x7f840406b3e0 2023-11-24T08:09:17.207 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:17.205+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:17.207 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:17.205+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:17.207 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:17.205+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:17.207 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:17.205+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:17.208 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:17.205+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:17.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.259+0000 7f841affd640 1 -- 172.21.15.107:0/1598780317 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f84140019b0 con 0x7f840406b3e0 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.261+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 msgr2=0x7f840406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.261+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 0x7f840406d8a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8414007fd0 tx=0x7f8414009040 comp rx=0 tx=0).stop 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.261+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 msgr2=0x7f8424155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.261+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f841c00d800 tx=0x7f841c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 shutdown_connections 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840406b3e0 0x7f840406d8a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84240ca7d0 0x7f8424155f40 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f842415ed60 0x7f8424157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 --2- 172.21.15.107:0/1598780317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84240b3600 0x7f842415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 >> 172.21.15.107:0/1598780317 conn(0x7f84240b2ae0 msgr2=0x7f842406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:17.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 shutdown_connections 2023-11-24T08:09:17.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.262+0000 7f842359e640 1 -- 172.21.15.107:0/1598780317 wait complete. 2023-11-24T08:09:17.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol2878 2023-11-24T08:09:17.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.478+0000 7fe39bfff640 1 Processor -- start 2023-11-24T08:09:17.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.479+0000 7fe39bfff640 1 -- start start 2023-11-24T08:09:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.480+0000 7fe39bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe39c15d3f0 0x7fe39c0d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.480+0000 7fe39bfff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0d15e0 con 0x7fe39c15d7d0 2023-11-24T08:09:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.480+0000 7fe39bfff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0d0280 con 0x7fe39c0d0e40 2023-11-24T08:09:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.480+0000 7fe39bfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c0d1300 con 0x7fe39c15d3f0 2023-11-24T08:09:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.481+0000 7fe39a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe39c15d7d0 0x7fe39c0afea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44438/0 (socket says 172.21.15.107:44438) 2023-11-24T08:09:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.481+0000 7fe39a7fc640 1 -- 172.21.15.107:0/2587365283 learned_addr learned my addr 172.21.15.107:0/2587365283 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.481+0000 7fe39affd640 1 --2- 172.21.15.107:0/2587365283 >> v2:172.21.15.120:3300/0 conn(0x7fe39c15d3f0 0x7fe39c0d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.481+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167239252 0 0) 0x7fe39c0d15e0 con 0x7fe39c15d7d0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe388003520 con 0x7fe39c15d7d0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467237026 0 0) 0x7fe388003520 con 0x7fe39c15d7d0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 >> v2:172.21.15.120:3300/0 conn(0x7fe39c15d3f0 msgr2=0x7fe39c0d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 --2- 172.21.15.107:0/2587365283 >> v2:172.21.15.120:3300/0 conn(0x7fe39c15d3f0 0x7fe39c0d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 >> v1:172.21.15.120:6789/0 conn(0x7fe39c0d0e40 legacy=0x7fe39c0b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0cfc30 con 0x7fe39c15d7d0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe384001670 con 0x7fe39c15d7d0 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 >> v1:172.21.15.107:6789/0 conn(0x7fe39c15d7d0 legacy=0x7fe39c0afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 0x7fe388003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3880047c0 0x7fe388004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe388005180 0x7fe38800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0d1300 con 0x7fe388003970 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38800c000 con 0x7fe388005180 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38800c1c0 con 0x7fe3880047c0 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.482+0000 7fe39affd640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 0x7fe388003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39affd640 1 -- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3880047c0 msgr2=0x7fe388004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39affd640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3880047c0 0x7fe388004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39affd640 1 -- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe388005180 msgr2=0x7fe38800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39affd640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe388005180 0x7fe38800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39affd640 1 -- 172.21.15.107:0/2587365283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388003520 con 0x7fe388003970 2023-11-24T08:09:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.483+0000 7fe39b7fe640 1 -- 172.21.15.107:0/2587365283 reap_dead start 2023-11-24T08:09:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.484+0000 7fe39affd640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 0x7fe388003d70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009ac0 tx=0x7fe3880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175f80392038951a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.484+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c02c710 con 0x7fe388003970 2023-11-24T08:09:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.484+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38c02c870 con 0x7fe388003970 2023-11-24T08:09:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.484+0000 7fe399ffb640 1 -- 172.21.15.107:0/2587365283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c02cc00 con 0x7fe388003970 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 -- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 msgr2=0x7fe388003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 0x7fe388003d70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009ac0 tx=0x7fe3880057f0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 -- 172.21.15.107:0/2587365283 shutdown_connections 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 --2- 172.21.15.107:0/2587365283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe388003970 0x7fe388003d70 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 -- 172.21.15.107:0/2587365283 >> 172.21.15.107:0/2587365283 conn(0x7fe39c09c5b0 msgr2=0x7fe39c0d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 -- 172.21.15.107:0/2587365283 shutdown_connections 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.485+0000 7fe39bfff640 1 -- 172.21.15.107:0/2587365283 wait complete. 2023-11-24T08:09:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.486+0000 7fe39bfff640 1 Processor -- start 2023-11-24T08:09:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.487+0000 7fe39bfff640 1 -- start start 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.487+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388003970 0x7fe39c0c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.487+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe39c1494d0 0x7fe39c0b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0c9980 con 0x7fe39c0c9500 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39bfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c158cb0 con 0x7fe39c1494d0 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0c7850 con 0x7fe388003970 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37558/0 (socket says 172.21.15.107:37558) 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 -- 172.21.15.107:0/1640304704 learned_addr learned my addr 172.21.15.107:0/1640304704 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 -- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388003970 msgr2=0x7fe39c0c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388003970 0x7fe39c0c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 -- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe39c1494d0 msgr2=0x7fe39c0b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe39c1494d0 0x7fe39c0b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 -- 172.21.15.107:0/1640304704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0c7af0 con 0x7fe39c0c9500 2023-11-24T08:09:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.488+0000 7fe39a7fc640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe38400c730 tx=0x7fe38400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.489+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3840072d0 con 0x7fe39c0c9500 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.489+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe384007450 con 0x7fe39c0c9500 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.489+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3840047e0 con 0x7fe39c0c9500 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.489+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c0c7dd0 con 0x7fe39c0c9500 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.489+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c0c9b90 con 0x7fe39c0c9500 2023-11-24T08:09:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.490+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c04e840 con 0x7fe39c0c9500 2023-11-24T08:09:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.491+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 89741+0+0 (secure 0 0 0) 0x7fe384004b50 con 0x7fe39c0c9500 2023-11-24T08:09:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.492+0000 7fe3937fe640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 0x7fe37c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.492+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3840896a0 con 0x7fe39c0c9500 2023-11-24T08:09:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.492+0000 7fe39affd640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 0x7fe37c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.492+0000 7fe39affd640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 0x7fe37c06d8a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.496+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe384053f60 con 0x7fe39c0c9500 2023-11-24T08:09:17.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.717+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol2878", "target": ["mon-mgr", ""]}) v1 -- 0x7fe39c0d0710 con 0x7fe37c06b3e0 2023-11-24T08:09:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.724+0000 7fe3937fe640 1 -- 172.21.15.107:0/1640304704 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fe38c02c9e0 con 0x7fe37c06b3e0 2023-11-24T08:09:17.726 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773 2023-11-24T08:09:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 msgr2=0x7fe37c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 0x7fe37c06d8a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c02e000 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 msgr2=0x7fe39c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe38400c730 tx=0x7fe38400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 shutdown_connections 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37c06b3e0 0x7fe37c06d8a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe39c1494d0 0x7fe39c0b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39c0c9500 0x7fe39c158770 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 --2- 172.21.15.107:0/1640304704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388003970 0x7fe39c0c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.726+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 >> 172.21.15.107:0/1640304704 conn(0x7fe39c09c5b0 msgr2=0x7fe39c06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.727+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 shutdown_connections 2023-11-24T08:09:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:17.727+0000 7fe39bfff640 1 -- 172.21.15.107:0/1640304704 wait complete. 2023-11-24T08:09:17.748 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol2878 with 200 files each 1MB size under directory volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773 2023-11-24T08:09:17.748 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:17.748 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:17.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:17.798 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244766 s, 42.8 MB/s 2023-11-24T08:09:17.932 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:17.932 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:17.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:17.986 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.068 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032879 s, 31.9 MB/s 2023-11-24T08:09:18.069 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.069 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.123 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244763 s, 42.8 MB/s 2023-11-24T08:09:18.204 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.204 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.265 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318691 s, 32.9 MB/s 2023-11-24T08:09:18.353 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.353 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.409 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384068 s, 27.3 MB/s 2023-11-24T08:09:18.501 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.501 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.559 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.634 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.634 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.634 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224052 s, 46.8 MB/s 2023-11-24T08:09:18.635 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.635 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.686 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.812 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.812 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.813 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261382 s, 40.1 MB/s 2023-11-24T08:09:18.814 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.814 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:18.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:18.873 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:18.951 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:18.951 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:18.951 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249563 s, 42.0 MB/s 2023-11-24T08:09:18.952 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:18.953 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.006 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262932 s, 39.9 MB/s 2023-11-24T08:09:19.137 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.137 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243855 s, 43.0 MB/s 2023-11-24T08:09:19.275 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.275 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.331 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229196 s, 45.8 MB/s 2023-11-24T08:09:19.409 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.409 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.468 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209156 s, 50.1 MB/s 2023-11-24T08:09:19.544 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.544 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177981 s, 58.9 MB/s 2023-11-24T08:09:19.669 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.669 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.725 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.804 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.805 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.805 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250437 s, 41.9 MB/s 2023-11-24T08:09:19.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.806 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.859 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:19.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234442 s, 44.7 MB/s 2023-11-24T08:09:19.933 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:19.934 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:19.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:19.990 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271542 s, 38.6 MB/s 2023-11-24T08:09:20.069 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.069 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.123 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.193 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202855 s, 51.7 MB/s 2023-11-24T08:09:20.194 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.194 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.248 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.318 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185873 s, 56.4 MB/s 2023-11-24T08:09:20.320 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.320 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.487 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210092 s, 49.9 MB/s 2023-11-24T08:09:20.488 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.488 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.620 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.620 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.620 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245524 s, 42.7 MB/s 2023-11-24T08:09:20.621 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.621 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.674 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243321 s, 43.1 MB/s 2023-11-24T08:09:20.753 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.753 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.811 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222501 s, 47.1 MB/s 2023-11-24T08:09:20.891 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:20.891 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:20.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:20.946 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.022 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.022 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.023 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234543 s, 44.7 MB/s 2023-11-24T08:09:21.024 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.024 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.082 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197055 s, 53.2 MB/s 2023-11-24T08:09:21.153 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.153 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.210 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.285 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.285 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.286 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212114 s, 49.4 MB/s 2023-11-24T08:09:21.287 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.287 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.344 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.413 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.414 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.414 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155879 s, 67.3 MB/s 2023-11-24T08:09:21.415 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.415 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.548 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.548 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.548 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221458 s, 47.3 MB/s 2023-11-24T08:09:21.550 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.550 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.602 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171319 s, 61.2 MB/s 2023-11-24T08:09:21.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.673 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247783 s, 42.3 MB/s 2023-11-24T08:09:21.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:21.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:21.906 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:21.986 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:21.986 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:21.986 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282825 s, 37.1 MB/s 2023-11-24T08:09:21.988 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:21.988 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022553 s, 46.5 MB/s 2023-11-24T08:09:22.122 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.122 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.179 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257257 s, 40.8 MB/s 2023-11-24T08:09:22.261 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.261 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.324 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.395 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.395 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.395 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192169 s, 54.6 MB/s 2023-11-24T08:09:22.396 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.397 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.528 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237789 s, 44.1 MB/s 2023-11-24T08:09:22.529 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.529 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.587 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200781 s, 52.2 MB/s 2023-11-24T08:09:22.659 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.659 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.717 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.796 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.796 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.797 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266325 s, 39.4 MB/s 2023-11-24T08:09:22.798 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.798 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.855 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:22.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:22.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:22.923 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179635 s, 58.4 MB/s 2023-11-24T08:09:22.924 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:22.924 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:22.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:22.974 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.103 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263463 s, 39.8 MB/s 2023-11-24T08:09:23.103 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.103 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.163 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218279 s, 48.0 MB/s 2023-11-24T08:09:23.241 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.241 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.298 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.373 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.374 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.374 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232384 s, 45.1 MB/s 2023-11-24T08:09:23.375 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.375 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.428 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190009 s, 55.2 MB/s 2023-11-24T08:09:23.498 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.498 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.630 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.630 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.630 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246837 s, 42.5 MB/s 2023-11-24T08:09:23.632 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.632 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.686 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247266 s, 42.4 MB/s 2023-11-24T08:09:23.759 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.760 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.813 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:23.887 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:23.887 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:23.887 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211093 s, 49.7 MB/s 2023-11-24T08:09:23.889 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:23.889 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:23.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:23.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.014 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.014 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.014 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173743 s, 60.4 MB/s 2023-11-24T08:09:24.016 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.016 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.069 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176747 s, 59.3 MB/s 2023-11-24T08:09:24.140 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.140 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.266 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225917 s, 46.4 MB/s 2023-11-24T08:09:24.268 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.268 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.326 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.407 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.408 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.408 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239956 s, 43.7 MB/s 2023-11-24T08:09:24.409 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.409 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.464 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187327 s, 56.0 MB/s 2023-11-24T08:09:24.536 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.536 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.592 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187932 s, 55.8 MB/s 2023-11-24T08:09:24.666 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.666 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.808 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.808 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.808 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259116 s, 40.5 MB/s 2023-11-24T08:09:24.810 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.810 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:24.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:24.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:24.988 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:24.988 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:24.988 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245074 s, 42.8 MB/s 2023-11-24T08:09:24.990 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:24.990 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.160 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.160 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.160 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183387 s, 57.2 MB/s 2023-11-24T08:09:25.161 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.162 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.220 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.298 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.299 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.299 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239211 s, 43.8 MB/s 2023-11-24T08:09:25.300 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.300 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.429 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.429 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.429 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199678 s, 52.5 MB/s 2023-11-24T08:09:25.431 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.431 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.483 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.609 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.610 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.610 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235303 s, 44.6 MB/s 2023-11-24T08:09:25.611 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.611 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.669 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.746 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227615 s, 46.1 MB/s 2023-11-24T08:09:25.748 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.748 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.802 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:25.881 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:25.881 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:25.881 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247214 s, 42.4 MB/s 2023-11-24T08:09:25.882 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:25.882 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:25.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:25.939 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.015 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.015 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.015 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208142 s, 50.4 MB/s 2023-11-24T08:09:26.017 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.017 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.071 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237032 s, 44.2 MB/s 2023-11-24T08:09:26.148 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.148 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203157 s, 51.6 MB/s 2023-11-24T08:09:26.281 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.281 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250577 s, 41.8 MB/s 2023-11-24T08:09:26.418 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.418 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.475 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.549 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.549 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.549 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215334 s, 48.7 MB/s 2023-11-24T08:09:26.550 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.550 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.609 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203624 s, 51.5 MB/s 2023-11-24T08:09:26.683 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.683 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.734 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.64 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.811 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.812 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.812 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232191 s, 45.2 MB/s 2023-11-24T08:09:26.813 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.813 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:26.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:26.867 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.65 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253269 s, 41.4 MB/s 2023-11-24T08:09:26.994 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:26.995 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.052 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.66 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204145 s, 51.4 MB/s 2023-11-24T08:09:27.127 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.127 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.67 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.259 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022029 s, 47.6 MB/s 2023-11-24T08:09:27.261 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.261 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.319 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.68 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.395 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.395 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.395 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210713 s, 49.8 MB/s 2023-11-24T08:09:27.397 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.397 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.454 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.69 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244893 s, 42.8 MB/s 2023-11-24T08:09:27.530 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.531 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.586 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.70 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.663 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.664 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.664 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251883 s, 41.6 MB/s 2023-11-24T08:09:27.665 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.665 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.722 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.71 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.801 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.802 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.802 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256528 s, 40.9 MB/s 2023-11-24T08:09:27.803 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.803 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:27.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:27.861 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.72 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024001 s, 43.7 MB/s 2023-11-24T08:09:27.943 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:27.944 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.73 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277383 s, 37.8 MB/s 2023-11-24T08:09:28.085 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.086 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.74 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198316 s, 52.9 MB/s 2023-11-24T08:09:28.214 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.214 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.75 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.347 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269153 s, 39.0 MB/s 2023-11-24T08:09:28.348 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.348 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.76 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.480 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.480 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.480 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213234 s, 49.2 MB/s 2023-11-24T08:09:28.482 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.482 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.77 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234045 s, 44.8 MB/s 2023-11-24T08:09:28.618 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.618 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.671 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.78 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295291 s, 35.5 MB/s 2023-11-24T08:09:28.751 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.751 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.807 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.79 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189615 s, 55.3 MB/s 2023-11-24T08:09:28.883 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:28.883 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:28.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:28.937 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.80 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223423 s, 46.9 MB/s 2023-11-24T08:09:29.013 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.014 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.070 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.81 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.136 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.136 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.136 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017013 s, 61.6 MB/s 2023-11-24T08:09:29.137 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.137 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.194 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.82 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.265 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.265 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.266 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210215 s, 49.9 MB/s 2023-11-24T08:09:29.267 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.267 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.324 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.83 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.401 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223153 s, 47.0 MB/s 2023-11-24T08:09:29.403 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.403 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.459 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.84 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.534 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.534 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.534 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221334 s, 47.4 MB/s 2023-11-24T08:09:29.535 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.535 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.591 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.85 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02441 s, 43.0 MB/s 2023-11-24T08:09:29.671 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.728 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.86 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.799 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.799 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.799 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210148 s, 49.9 MB/s 2023-11-24T08:09:29.801 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.801 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.87 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:29.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:29.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:29.931 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02138 s, 49.0 MB/s 2023-11-24T08:09:29.932 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:29.932 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:29.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:29.988 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.88 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.067 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.068 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.068 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246018 s, 42.6 MB/s 2023-11-24T08:09:30.069 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.069 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.89 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245186 s, 42.8 MB/s 2023-11-24T08:09:30.208 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.209 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.90 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.350 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270169 s, 38.8 MB/s 2023-11-24T08:09:30.351 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.352 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.408 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.91 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206514 s, 50.8 MB/s 2023-11-24T08:09:30.485 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.485 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.539 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.92 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162036 s, 64.7 MB/s 2023-11-24T08:09:30.610 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.610 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.668 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.93 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245538 s, 42.7 MB/s 2023-11-24T08:09:30.748 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.748 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.802 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.94 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:30.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:30.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:30.879 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236612 s, 44.3 MB/s 2023-11-24T08:09:30.881 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:30.881 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:30.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:30.938 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.95 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181586 s, 57.7 MB/s 2023-11-24T08:09:31.011 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.012 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.065 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.96 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.139 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222004 s, 47.2 MB/s 2023-11-24T08:09:31.141 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.141 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.197 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.97 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195688 s, 53.6 MB/s 2023-11-24T08:09:31.271 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.271 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.328 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.98 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021477 s, 48.8 MB/s 2023-11-24T08:09:31.405 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.405 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.463 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.99 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253539 s, 41.4 MB/s 2023-11-24T08:09:31.543 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.544 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.100 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175662 s, 59.7 MB/s 2023-11-24T08:09:31.677 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.677 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.731 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.101 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249626 s, 42.0 MB/s 2023-11-24T08:09:31.812 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.812 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:31.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:31.870 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.102 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259 s, 40.5 MB/s 2023-11-24T08:09:31.953 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:31.953 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.011 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.103 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183671 s, 57.1 MB/s 2023-11-24T08:09:32.086 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.086 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.104 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278684 s, 37.6 MB/s 2023-11-24T08:09:32.222 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.222 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.280 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.105 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.355 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.356 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.356 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217201 s, 48.3 MB/s 2023-11-24T08:09:32.357 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.357 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.106 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206884 s, 50.7 MB/s 2023-11-24T08:09:32.489 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.489 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.107 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.618 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.619 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.619 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188453 s, 55.6 MB/s 2023-11-24T08:09:32.620 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.620 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.108 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.754 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.755 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.755 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026868 s, 39.0 MB/s 2023-11-24T08:09:32.756 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.756 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.812 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.109 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:32.885 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:32.885 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:32.885 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221649 s, 47.3 MB/s 2023-11-24T08:09:32.886 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:32.886 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:32.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:32.946 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.110 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.031 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.031 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.032 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232155 s, 45.2 MB/s 2023-11-24T08:09:33.033 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.033 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.092 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.111 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239554 s, 43.8 MB/s 2023-11-24T08:09:33.170 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.170 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.228 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.112 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.302 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.302 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.303 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202217 s, 51.9 MB/s 2023-11-24T08:09:33.304 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.304 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.113 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.435 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228573 s, 45.9 MB/s 2023-11-24T08:09:33.436 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.436 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.492 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.114 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.573 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.573 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.573 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219431 s, 47.8 MB/s 2023-11-24T08:09:33.575 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.575 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.636 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.115 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194538 s, 53.9 MB/s 2023-11-24T08:09:33.704 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.704 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.755 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.116 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241939 s, 43.3 MB/s 2023-11-24T08:09:33.835 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.835 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:33.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:33.889 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.117 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229068 s, 45.8 MB/s 2023-11-24T08:09:33.966 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:33.966 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.018 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.118 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262003 s, 40.0 MB/s 2023-11-24T08:09:34.098 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.098 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.119 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.218 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.218 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.218 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185713 s, 56.5 MB/s 2023-11-24T08:09:34.219 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.219 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.120 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242138 s, 43.3 MB/s 2023-11-24T08:09:34.353 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.354 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.409 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.121 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202056 s, 51.9 MB/s 2023-11-24T08:09:34.481 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.481 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.534 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.122 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.603 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182986 s, 57.3 MB/s 2023-11-24T08:09:34.604 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.604 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.123 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248061 s, 42.3 MB/s 2023-11-24T08:09:34.734 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.734 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.792 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.124 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:34.870 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:34.870 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:34.870 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241959 s, 43.3 MB/s 2023-11-24T08:09:34.871 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:34.871 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:34.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:34.924 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.125 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.050 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225143 s, 46.6 MB/s 2023-11-24T08:09:35.051 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.051 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.111 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.126 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.187 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.187 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.187 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202781 s, 51.7 MB/s 2023-11-24T08:09:35.188 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.188 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.127 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.318 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210683 s, 49.8 MB/s 2023-11-24T08:09:35.319 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.320 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.369 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.128 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240808 s, 43.5 MB/s 2023-11-24T08:09:35.496 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.497 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.547 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.129 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194408 s, 53.9 MB/s 2023-11-24T08:09:35.671 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.725 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.130 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.797 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.797 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.797 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224358 s, 46.7 MB/s 2023-11-24T08:09:35.798 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.798 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:35.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:35.848 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.131 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027534 s, 38.1 MB/s 2023-11-24T08:09:35.976 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:35.977 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.034 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.132 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220471 s, 47.6 MB/s 2023-11-24T08:09:36.114 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.114 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.170 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.133 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223787 s, 46.9 MB/s 2023-11-24T08:09:36.241 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.242 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.298 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.134 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018549 s, 56.5 MB/s 2023-11-24T08:09:36.369 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.369 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.427 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.135 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.498 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.498 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.498 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187281 s, 56.0 MB/s 2023-11-24T08:09:36.499 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.500 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.557 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.136 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244336 s, 42.9 MB/s 2023-11-24T08:09:36.636 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.637 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.693 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.137 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.768 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.768 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.768 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239903 s, 43.7 MB/s 2023-11-24T08:09:36.770 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.770 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.138 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:36.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:36.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:36.899 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211735 s, 49.5 MB/s 2023-11-24T08:09:36.901 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:36.901 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:36.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:36.956 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.139 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236254 s, 44.4 MB/s 2023-11-24T08:09:37.080 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.080 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.139 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.140 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.217 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023012 s, 45.6 MB/s 2023-11-24T08:09:37.219 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.219 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.141 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.345 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.345 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.345 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180994 s, 57.9 MB/s 2023-11-24T08:09:37.346 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.347 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.405 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.142 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244026 s, 43.0 MB/s 2023-11-24T08:09:37.488 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.488 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.548 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.143 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.626 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.627 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.627 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222651 s, 47.1 MB/s 2023-11-24T08:09:37.628 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.629 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.144 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256389 s, 40.9 MB/s 2023-11-24T08:09:37.771 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.771 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.829 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.145 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:37.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:37.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:37.912 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237042 s, 44.2 MB/s 2023-11-24T08:09:37.913 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:37.914 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:37.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:37.973 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.146 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229638 s, 45.7 MB/s 2023-11-24T08:09:38.051 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.051 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.108 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.147 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.185 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.185 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.185 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233829 s, 44.8 MB/s 2023-11-24T08:09:38.186 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.186 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.246 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.148 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.323 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.323 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.323 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024143 s, 43.4 MB/s 2023-11-24T08:09:38.325 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.325 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.383 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.149 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206141 s, 50.9 MB/s 2023-11-24T08:09:38.461 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.461 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.517 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.150 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206161 s, 50.9 MB/s 2023-11-24T08:09:38.591 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.591 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.644 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.151 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.706 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.706 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.706 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163365 s, 64.2 MB/s 2023-11-24T08:09:38.707 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.708 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.763 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.152 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283318 s, 37.0 MB/s 2023-11-24T08:09:38.843 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.843 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:38.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:38.896 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.153 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:38.968 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:38.969 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:38.969 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230563 s, 45.5 MB/s 2023-11-24T08:09:38.970 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:38.970 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.154 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192538 s, 54.5 MB/s 2023-11-24T08:09:39.102 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.102 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.159 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.155 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204788 s, 51.2 MB/s 2023-11-24T08:09:39.234 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.234 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.156 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.410 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.410 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.410 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240364 s, 43.6 MB/s 2023-11-24T08:09:39.411 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.412 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.157 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.545 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220191 s, 47.6 MB/s 2023-11-24T08:09:39.547 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.547 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.605 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.158 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215754 s, 48.6 MB/s 2023-11-24T08:09:39.681 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.681 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.735 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.159 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028827 s, 36.4 MB/s 2023-11-24T08:09:39.819 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.819 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:39.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:39.877 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.160 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232378 s, 45.1 MB/s 2023-11-24T08:09:39.957 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:39.957 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.010 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.161 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020691 s, 50.7 MB/s 2023-11-24T08:09:40.131 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.131 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.162 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.263 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.264 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.264 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218129 s, 48.1 MB/s 2023-11-24T08:09:40.266 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.266 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:09:40.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:09:40.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.320 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.163 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.425 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.426 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167112 s, 62.7 MB/s 2023-11-24T08:09:40.427 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.427 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.524 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.164 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.599 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207332 s, 50.6 MB/s 2023-11-24T08:09:40.600 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.600 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.165 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.722 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.723 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.723 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017528 s, 59.8 MB/s 2023-11-24T08:09:40.724 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.724 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.775 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.166 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:40.880 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:40.880 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:40.880 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203558 s, 51.5 MB/s 2023-11-24T08:09:40.881 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:40.881 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:40.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:40.928 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.167 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021366 s, 49.1 MB/s 2023-11-24T08:09:41.052 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.052 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.109 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.168 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.183 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.183 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.183 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240453 s, 43.6 MB/s 2023-11-24T08:09:41.185 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.185 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.169 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167605 s, 62.6 MB/s 2023-11-24T08:09:41.314 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.314 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.362 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.170 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198421 s, 52.8 MB/s 2023-11-24T08:09:41.485 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.485 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.540 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.171 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.615 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.616 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.616 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228267 s, 45.9 MB/s 2023-11-24T08:09:41.617 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.617 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.673 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.172 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265404 s, 39.5 MB/s 2023-11-24T08:09:41.755 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.755 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.812 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.173 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210852 s, 49.7 MB/s 2023-11-24T08:09:41.889 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:41.889 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:41.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:41.947 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.174 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.027 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279063 s, 37.6 MB/s 2023-11-24T08:09:42.029 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.029 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.084 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.175 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.163 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267075 s, 39.3 MB/s 2023-11-24T08:09:42.165 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.165 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.223 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.176 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.292 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.293 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199942 s, 52.4 MB/s 2023-11-24T08:09:42.294 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.294 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.177 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.414 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.415 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.415 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173202 s, 60.5 MB/s 2023-11-24T08:09:42.416 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.417 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.178 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169711 s, 61.8 MB/s 2023-11-24T08:09:42.544 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.545 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.179 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199895 s, 52.5 MB/s 2023-11-24T08:09:42.671 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.180 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246975 s, 42.5 MB/s 2023-11-24T08:09:42.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.806 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:42.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:42.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.181 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:42.942 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023632 s, 44.4 MB/s 2023-11-24T08:09:42.944 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:42.945 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.182 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212943 s, 49.2 MB/s 2023-11-24T08:09:43.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.078 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.134 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.183 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.216 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.216 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.216 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260632 s, 40.2 MB/s 2023-11-24T08:09:43.218 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.218 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.274 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.184 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.348 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.348 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.348 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019326 s, 54.3 MB/s 2023-11-24T08:09:43.349 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.349 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.401 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.185 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.474 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.474 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.474 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221809 s, 47.3 MB/s 2023-11-24T08:09:43.476 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.476 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.530 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.186 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.600 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198717 s, 52.8 MB/s 2023-11-24T08:09:43.601 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.601 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.187 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194186 s, 54.0 MB/s 2023-11-24T08:09:43.722 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.723 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.774 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.188 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243156 s, 43.1 MB/s 2023-11-24T08:09:43.901 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:43.901 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:43.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:43.958 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.189 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.037 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.037 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253185 s, 41.4 MB/s 2023-11-24T08:09:44.038 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.038 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.094 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.190 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.161 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.162 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.162 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018613 s, 56.3 MB/s 2023-11-24T08:09:44.163 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.164 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.219 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.191 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.289 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192901 s, 54.4 MB/s 2023-11-24T08:09:44.290 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.291 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.347 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.192 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257508 s, 40.7 MB/s 2023-11-24T08:09:44.428 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.428 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.485 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.193 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.555 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.555 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.555 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017713 s, 59.2 MB/s 2023-11-24T08:09:44.556 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.556 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.604 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.194 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215596 s, 48.6 MB/s 2023-11-24T08:09:44.726 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.726 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.786 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.195 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:44.862 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:44.863 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:44.863 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190138 s, 55.1 MB/s 2023-11-24T08:09:44.864 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:44.864 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:44.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:44.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.196 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:45.030 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:45.030 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:45.030 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176366 s, 59.5 MB/s 2023-11-24T08:09:45.031 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:45.032 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:45.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:45.087 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.197 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:45.168 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:45.169 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270114 s, 38.8 MB/s 2023-11-24T08:09:45.170 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:45.170 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:45.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:45.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.198 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:45.296 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:45.296 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:45.296 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189412 s, 55.4 MB/s 2023-11-24T08:09:45.297 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:09:45.297 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:09:45.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:45.350 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2878/98d8994d-ed5a-4c01-9027-9356a7090773/subvolume_file.199 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:09:45.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:09:45.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:09:45.426 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237889 s, 44.1 MB/s 2023-11-24T08:09:45.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol2878 subvol_snap4606 2023-11-24T08:09:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.649+0000 7fb13573c640 1 Processor -- start 2023-11-24T08:09:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.649+0000 7fb13573c640 1 -- start start 2023-11-24T08:09:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.650+0000 7fb13573c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb13006d2b0 0x7fb13015db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.650+0000 7fb13573c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 0x7fb13015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.650+0000 7fb13573c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1300cc930 con 0x7fb1301400a0 2023-11-24T08:09:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.650+0000 7fb13573c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13015e120 con 0x7fb13006d2b0 2023-11-24T08:09:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.650+0000 7fb13573c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13015e400 con 0x7fb13013d660 2023-11-24T08:09:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.651+0000 7fb134f3b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 0x7fb13015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.651+0000 7fb12ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb13006d2b0 0x7fb13015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.651+0000 7fb134f3b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 0x7fb13015ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:58014/0 (socket says 172.21.15.107:58014) 2023-11-24T08:09:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.651+0000 7fb134f3b640 1 -- 172.21.15.107:0/3632778624 learned_addr learned my addr 172.21.15.107:0/3632778624 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.652+0000 7fb12ffff640 1 -- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 msgr2=0x7fb13015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.652+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 0x7fb13015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.652+0000 7fb12ffff640 1 -- 172.21.15.107:0/3632778624 >> v1:172.21.15.120:6789/0 conn(0x7fb1301400a0 legacy=0x7fb13006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.652+0000 7fb12ffff640 1 -- 172.21.15.107:0/3632778624 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13015e6e0 con 0x7fb13006d2b0 2023-11-24T08:09:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.654+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb13006d2b0 0x7fb13015db00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1200097a0 tx=0x7fb12002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f968574e7fb9b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.654+0000 7fb12f7fe640 1 -- 172.21.15.107:0/3632778624 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12002d040 con 0x7fb13006d2b0 2023-11-24T08:09:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.654+0000 7fb12f7fe640 1 -- 172.21.15.107:0/3632778624 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb120031810 con 0x7fb13006d2b0 2023-11-24T08:09:45.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.654+0000 7fb12f7fe640 1 -- 172.21.15.107:0/3632778624 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120031ba0 con 0x7fb13006d2b0 2023-11-24T08:09:45.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 -- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb13006d2b0 msgr2=0x7fb13015db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 --2- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb13006d2b0 0x7fb13015db00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1200097a0 tx=0x7fb12002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 -- 172.21.15.107:0/3632778624 shutdown_connections 2023-11-24T08:09:45.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 --2- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13013d660 0x7fb13015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 --2- 172.21.15.107:0/3632778624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb13006d2b0 0x7fb13015db00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 -- 172.21.15.107:0/3632778624 >> 172.21.15.107:0/3632778624 conn(0x7fb1300b1c40 msgr2=0x7fb1300b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 -- 172.21.15.107:0/3632778624 shutdown_connections 2023-11-24T08:09:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.655+0000 7fb13573c640 1 -- 172.21.15.107:0/3632778624 wait complete. 2023-11-24T08:09:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.657+0000 7fb13573c640 1 Processor -- start 2023-11-24T08:09:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.657+0000 7fb13573c640 1 -- start start 2023-11-24T08:09:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.658+0000 7fb13573c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13006d2b0 0x7fb1300b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.658+0000 7fb13573c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb13573c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1301400a0 0x7fb13013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb13573c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13013c460 con 0x7fb13013d660 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb13573c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130154bb0 con 0x7fb1301400a0 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb13573c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300b45d0 con 0x7fb13006d2b0 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54332/0 (socket says 172.21.15.107:54332) 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 -- 172.21.15.107:0/1204876448 learned_addr learned my addr 172.21.15.107:0/1204876448 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb12ffff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13006d2b0 0x7fb1300b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb134f3b640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1301400a0 0x7fb13013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 -- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13006d2b0 msgr2=0x7fb1300b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13006d2b0 0x7fb1300b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 -- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1301400a0 msgr2=0x7fb13013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1301400a0 0x7fb13013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.659+0000 7fb127fff640 1 -- 172.21.15.107:0/1204876448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120009400 con 0x7fb13013d660 2023-11-24T08:09:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb127fff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c450 tx=0x7fb11800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1180052d0 con 0x7fb13013d660 2023-11-24T08:09:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118005430 con 0x7fb13013d660 2023-11-24T08:09:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118012870 con 0x7fb13013d660 2023-11-24T08:09:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1300b48b0 con 0x7fb13013d660 2023-11-24T08:09:45.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.660+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1301550e0 con 0x7fb13013d660 2023-11-24T08:09:45.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.661+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13004e840 con 0x7fb13013d660 2023-11-24T08:09:45.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.662+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 89792+0+0 (secure 0 0 0) 0x7fb1180129d0 con 0x7fb13013d660 2023-11-24T08:09:45.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.663+0000 7fb12d7fa640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 0x7fb11006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:45.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.663+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb118089c30 con 0x7fb13013d660 2023-11-24T08:09:45.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.663+0000 7fb12ffff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 0x7fb11006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:45.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.664+0000 7fb12ffff640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 0x7fb11006d930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb120036000 tx=0x7fb12003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:45.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.668+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1180543f0 con 0x7fb13013d660 2023-11-24T08:09:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.889+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol2878", "snap_name": "subvol_snap4606", "target": ["mon-mgr", ""]}) v1 -- 0x7fb13006b0e0 con 0x7fb11006b470 2023-11-24T08:09:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.926+0000 7fb12d7fa640 1 -- 172.21.15.107:0/1204876448 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb13006b0e0 con 0x7fb11006b470 2023-11-24T08:09:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 msgr2=0x7fb11006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 0x7fb11006d930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb120036000 tx=0x7fb12003c040 comp rx=0 tx=0).stop 2023-11-24T08:09:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 msgr2=0x7fb13013e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c450 tx=0x7fb11800c910 comp rx=0 tx=0).stop 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 shutdown_connections 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb11006b470 0x7fb11006d930 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1301400a0 0x7fb13013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb13013d660 0x7fb13013e800 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 --2- 172.21.15.107:0/1204876448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb13006d2b0 0x7fb1300b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 >> 172.21.15.107:0/1204876448 conn(0x7fb1300b1c40 msgr2=0x7fb130159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.929+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 shutdown_connections 2023-11-24T08:09:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:45.930+0000 7fb13573c640 1 -- 172.21.15.107:0/1204876448 wait complete. 2023-11-24T08:09:45.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 5 2023-11-24T08:09:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.161+0000 7f78d4e5d640 1 Processor -- start 2023-11-24T08:09:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.161+0000 7f78d4e5d640 1 -- start start 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.162+0000 7f78d4e5d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f78d00cfb90 0x7f78d0161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.162+0000 7f78d4e5d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78d00b2400 0x7f78d00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.162+0000 7f78d4e5d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d01612f0 con 0x7f78d0161fa0 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.162+0000 7f78d4e5d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00cff70 con 0x7f78d00b2400 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.162+0000 7f78d4e5d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00afb20 con 0x7f78d00cfb90 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78cdd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f78d0161fa0 0x7f78d006ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56686/0 (socket says 172.21.15.107:56686) 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78cdd9b640 1 -- 172.21.15.107:0/1765091417 learned_addr learned my addr 172.21.15.107:0/1765091417 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ced9d640 1 --2- 172.21.15.107:0/1765091417 >> v2:172.21.15.107:3300/0 conn(0x7f78d00b2400 0x7f78d00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ce59c640 1 --2- 172.21.15.107:0/1765091417 >> v2:172.21.15.120:3300/0 conn(0x7f78d00cfb90 0x7f78d0161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ced9d640 1 -- 172.21.15.107:0/1765091417 >> v2:172.21.15.120:3300/0 conn(0x7f78d00cfb90 msgr2=0x7f78d0161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ced9d640 1 --2- 172.21.15.107:0/1765091417 >> v2:172.21.15.120:3300/0 conn(0x7f78d00cfb90 0x7f78d0161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ced9d640 1 -- 172.21.15.107:0/1765091417 >> v1:172.21.15.107:6789/0 conn(0x7f78d0161fa0 legacy=0x7f78d006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.163+0000 7f78ced9d640 1 -- 172.21.15.107:0/1765091417 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d006a850 con 0x7f78d00b2400 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.164+0000 7f78ced9d640 1 --2- 172.21.15.107:0/1765091417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00b2400 0x7f78d00cc940 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f78d0070d30 tx=0x7f78c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247460f3174107ef server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78cd59a640 1 -- 172.21.15.107:0/1765091417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8030610 con 0x7f78d00b2400 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78cd59a640 1 -- 172.21.15.107:0/1765091417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c8030770 con 0x7f78d00b2400 2023-11-24T08:09:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78cd59a640 1 -- 172.21.15.107:0/1765091417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8030b00 con 0x7f78d00b2400 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 -- 172.21.15.107:0/1765091417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00b2400 msgr2=0x7f78d00cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/1765091417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00b2400 0x7f78d00cc940 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f78d0070d30 tx=0x7f78c802b960 comp rx=0 tx=0).stop 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 -- 172.21.15.107:0/1765091417 shutdown_connections 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/1765091417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00b2400 0x7f78d00cc940 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/1765091417 >> v2:172.21.15.120:3300/0 conn(0x7f78d00cfb90 0x7f78d0161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 -- 172.21.15.107:0/1765091417 >> 172.21.15.107:0/1765091417 conn(0x7f78d00999d0 msgr2=0x7f78d00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 -- 172.21.15.107:0/1765091417 shutdown_connections 2023-11-24T08:09:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.165+0000 7f78d4e5d640 1 -- 172.21.15.107:0/1765091417 wait complete. 2023-11-24T08:09:46.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.167+0000 7f78d4e5d640 1 Processor -- start 2023-11-24T08:09:46.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.167+0000 7f78d4e5d640 1 -- start start 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78d00b2400 0x7f78d0156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78d00cc930 0x7f78d006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0156ea0 con 0x7f78d00cfb90 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0157120 con 0x7f78d00cc930 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78d4e5d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0157400 con 0x7f78d00b2400 2023-11-24T08:09:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.168+0000 7f78cdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54352/0 (socket says 172.21.15.107:54352) 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 -- 172.21.15.107:0/718896156 learned_addr learned my addr 172.21.15.107:0/718896156 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78ce59c640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78d00b2400 0x7f78d0156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 -- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78d00b2400 msgr2=0x7f78d0156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78d00b2400 0x7f78d0156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 -- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78d00cc930 msgr2=0x7f78d006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78d00cc930 0x7f78d006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 -- 172.21.15.107:0/718896156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009400 con 0x7f78d00cfb90 2023-11-24T08:09:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.169+0000 7f78cdd9b640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f78b800cba0 tx=0x7f78b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.170+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b800ad40 con 0x7f78d00cfb90 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.170+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8012400 con 0x7f78d00cfb90 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.170+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b800aea0 con 0x7f78d00cfb90 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.170+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d015eea0 con 0x7f78d00cfb90 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.170+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d0148a80 con 0x7f78d00cfb90 2023-11-24T08:09:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.171+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d015f3c0 con 0x7f78d00cfb90 2023-11-24T08:09:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.172+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 89792+0+0 (secure 0 0 0) 0x7f78b8007070 con 0x7f78d00cfb90 2023-11-24T08:09:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.173+0000 7f78c6ffd640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 0x7f78b006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.173+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78b808a000 con 0x7f78d00cfb90 2023-11-24T08:09:46.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.173+0000 7f78ce59c640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 0x7f78b006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.174+0000 7f78ce59c640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 0x7f78b006d930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.177+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78d015f3c0 con 0x7f78d00cfb90 2023-11-24T08:09:46.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.367+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f78d015f3c0 con 0x7f78d00cfb90 2023-11-24T08:09:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.373+0000 7f78c6ffd640 1 -- 172.21.15.107:0/718896156 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v19)=0 v19) v1 ==== 130+0+0 (secure 0 0 0) 0x7f78d015f3c0 con 0x7f78d00cfb90 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.376+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 msgr2=0x7f78b006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.376+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 0x7f78b006d930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0008040 comp rx=0 tx=0).stop 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 msgr2=0x7f78d0148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f78b800cba0 tx=0x7f78b800a430 comp rx=0 tx=0).stop 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 shutdown_connections 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78b006b470 0x7f78b006d930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78d00cc930 0x7f78d006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d00cfb90 0x7f78d0148540 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 --2- 172.21.15.107:0/718896156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78d00b2400 0x7f78d0156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 >> 172.21.15.107:0/718896156 conn(0x7f78d00999d0 msgr2=0x7f78d00b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 shutdown_connections 2023-11-24T08:09:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.377+0000 7f78d4e5d640 1 -- 172.21.15.107:0/718896156 wait complete. 2023-11-24T08:09:46.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.391+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.391+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.391+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.391+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.391+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol2878 subvol_snap4606 subvol_clone9024 2023-11-24T08:09:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.588+0000 7f39a2a1c640 1 Processor -- start 2023-11-24T08:09:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.588+0000 7f39a2a1c640 1 -- start start 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f399c15b440 0x7f399c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f399c0c57e0 0x7f399c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c0cc3d0 con 0x7f399c15b440 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c15b820 con 0x7f399c0c57e0 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f39a2a1c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c15eea0 con 0x7f399c0cbf20 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f399bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f399c15b440 0x7f399c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f399bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f399c15b440 0x7f399c0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54392/0 (socket says 172.21.15.107:54392) 2023-11-24T08:09:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.590+0000 7f399bfff640 1 -- 172.21.15.107:0/2229813211 learned_addr learned my addr 172.21.15.107:0/2229813211 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f39a0fb9640 1 --2- 172.21.15.107:0/2229813211 >> v2:172.21.15.120:3300/0 conn(0x7f399c0c57e0 0x7f399c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f39937fe640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f399bfff640 1 -- 172.21.15.107:0/2229813211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 msgr2=0x7f399c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f399bfff640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f399bfff640 1 -- 172.21.15.107:0/2229813211 >> v2:172.21.15.120:3300/0 conn(0x7f399c0c57e0 msgr2=0x7f399c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f399bfff640 1 --2- 172.21.15.107:0/2229813211 >> v2:172.21.15.120:3300/0 conn(0x7f399c0c57e0 0x7f399c0c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f399bfff640 1 -- 172.21.15.107:0/2229813211 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0abf30 con 0x7f399c15b440 2023-11-24T08:09:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.591+0000 7f39937fe640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.592+0000 7f399bfff640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c15b440 0x7f399c0cb9e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f399c0cb910 tx=0x7f398c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b2b31b0d525387 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.592+0000 7f399b7fe640 1 -- 172.21.15.107:0/2229813211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02c9e0 con 0x7f399c15b440 2023-11-24T08:09:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.592+0000 7f399b7fe640 1 -- 172.21.15.107:0/2229813211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c02cb40 con 0x7f399c15b440 2023-11-24T08:09:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.592+0000 7f399b7fe640 1 -- 172.21.15.107:0/2229813211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0343f0 con 0x7f399c15b440 2023-11-24T08:09:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 -- 172.21.15.107:0/2229813211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c15b440 msgr2=0x7f399c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c15b440 0x7f399c0cb9e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f399c0cb910 tx=0x7f398c02be40 comp rx=0 tx=0).stop 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 -- 172.21.15.107:0/2229813211 shutdown_connections 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/2229813211 >> v2:172.21.15.120:3300/0 conn(0x7f399c0c57e0 0x7f399c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/2229813211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c15b440 0x7f399c0cb9e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 -- 172.21.15.107:0/2229813211 >> 172.21.15.107:0/2229813211 conn(0x7f399c0b0c50 msgr2=0x7f399c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 -- 172.21.15.107:0/2229813211 shutdown_connections 2023-11-24T08:09:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.593+0000 7f39a2a1c640 1 -- 172.21.15.107:0/2229813211 wait complete. 2023-11-24T08:09:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.595+0000 7f39a2a1c640 1 Processor -- start 2023-11-24T08:09:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.595+0000 7f39a2a1c640 1 -- start start 2023-11-24T08:09:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f399c15b440 0x7f399c15d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0b2dc0 con 0x7f399c0c57e0 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0b21b0 con 0x7f399c15b440 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f39a2a1c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0b2490 con 0x7f399c0cbf20 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f399bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f399bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54402/0 (socket says 172.21.15.107:54402) 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.596+0000 7f399bfff640 1 -- 172.21.15.107:0/3147310472 learned_addr learned my addr 172.21.15.107:0/3147310472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39a0fb9640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f399c15b440 0x7f399c15d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 -- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 msgr2=0x7f399c155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 -- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f399c15b440 msgr2=0x7f399c15d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f399c15b440 0x7f399c15d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 -- 172.21.15.107:0/3147310472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c009b30 con 0x7f399c0c57e0 2023-11-24T08:09:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f399bfff640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f398c000f80 tx=0x7f398c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c009510 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c03e450 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c0b2770 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c03e800 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.597+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c0b3790 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.599+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c0c9d10 con 0x7f399c0c57e0 2023-11-24T08:09:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.599+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 89792+0+0 (secure 0 0 0) 0x7f398c03e960 con 0x7f399c0c57e0 2023-11-24T08:09:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.600+0000 7f39997fa640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 0x7f397c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.600+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f398c0acfe0 con 0x7f399c0c57e0 2023-11-24T08:09:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.601+0000 7f39937fe640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 0x7f397c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:46.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.601+0000 7f39937fe640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 0x7f397c06d930 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:46.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.605+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f398c077710 con 0x7f399c0c57e0 2023-11-24T08:09:46.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.828+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol2878", "snap_name": "subvol_snap4606", "target_sub_name": "subvol_clone9024", "target": ["mon-mgr", ""]}) v1 -- 0x7f399c15f6b0 con 0x7f397c06b470 2023-11-24T08:09:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.964+0000 7f39997fa640 1 -- 172.21.15.107:0/3147310472 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f398400ae40 con 0x7f397c06b470 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.967+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 msgr2=0x7f397c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.967+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 0x7f397c06d930 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984008040 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.967+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 msgr2=0x7f399c0b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.967+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f398c000f80 tx=0x7f398c004a40 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 shutdown_connections 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f397c06b470 0x7f397c06d930 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f399c15b440 0x7f399c15d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f399c0cbf20 0x7f399c155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 --2- 172.21.15.107:0/3147310472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f399c0c57e0 0x7f399c0b3250 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 >> 172.21.15.107:0/3147310472 conn(0x7f399c0b0c50 msgr2=0x7f399c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 shutdown_connections 2023-11-24T08:09:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:46.968+0000 7f39a2a1c640 1 -- 172.21.15.107:0/3147310472 wait complete. 2023-11-24T08:09:46.975 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.973+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.975 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.973+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.976 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.973+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.976 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.973+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.976 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.973+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9024 2023-11-24T08:09:46.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.996+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.997+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.997+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.999 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.997+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:46.999 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:46.997+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.181+0000 7fe5ee451640 1 Processor -- start 2023-11-24T08:09:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.181+0000 7fe5ee451640 1 -- start start 2023-11-24T08:09:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ee451640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ee451640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5e80c59b0 0x7fe5e806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ee451640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80cc3a0 con 0x7fe5e80cbf20 2023-11-24T08:09:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ee451640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e815b820 con 0x7fe5e80c59b0 2023-11-24T08:09:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ee451640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e815eea0 con 0x7fe5e815b440 2023-11-24T08:09:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ec9ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5e80c59b0 0x7fe5e806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.183+0000 7fe5ec9ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5e80c59b0 0x7fe5e806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54450/0 (socket says 172.21.15.107:54450) 2023-11-24T08:09:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5ec9ee640 1 -- 172.21.15.107:0/484629368 learned_addr learned my addr 172.21.15.107:0/484629368 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5e7fff640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5ec9ee640 1 -- 172.21.15.107:0/484629368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 msgr2=0x7fe5e80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5ec9ee640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5ec9ee640 1 -- 172.21.15.107:0/484629368 >> v1:172.21.15.120:6789/0 conn(0x7fe5e80cbf20 legacy=0x7fe5e80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5ec9ee640 1 -- 172.21.15.107:0/484629368 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80abf30 con 0x7fe5e80c59b0 2023-11-24T08:09:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.184+0000 7fe5e7fff640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.185+0000 7fe5ec9ee640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e806c5f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009e50 tx=0x7fe5e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a1dc28fc76710c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5e77fe640 1 -- 172.21.15.107:0/484629368 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e002c9e0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5e77fe640 1 -- 172.21.15.107:0/484629368 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e002cb40 con 0x7fe5e80c59b0 2023-11-24T08:09:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5e77fe640 1 -- 172.21.15.107:0/484629368 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00343f0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 -- 172.21.15.107:0/484629368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 msgr2=0x7fe5e806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e806c5f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009e50 tx=0x7fe5e002be40 comp rx=0 tx=0).stop 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 -- 172.21.15.107:0/484629368 shutdown_connections 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e806c5f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 --2- 172.21.15.107:0/484629368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 -- 172.21.15.107:0/484629368 >> 172.21.15.107:0/484629368 conn(0x7fe5e80b0c50 msgr2=0x7fe5e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 -- 172.21.15.107:0/484629368 shutdown_connections 2023-11-24T08:09:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.186+0000 7fe5ee451640 1 -- 172.21.15.107:0/484629368 wait complete. 2023-11-24T08:09:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.188+0000 7fe5ee451640 1 Processor -- start 2023-11-24T08:09:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.188+0000 7fe5ee451640 1 -- start start 2023-11-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5e80cbf20 0x7fe5e8157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e8153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8156670 con 0x7fe5e80c59b0 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8157740 con 0x7fe5e80cbf20 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5ee451640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80c6420 con 0x7fe5e815b440 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54460/0 (socket says 172.21.15.107:54460) 2023-11-24T08:09:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.189+0000 7fe5e7fff640 1 -- 172.21.15.107:0/422543413 learned_addr learned my addr 172.21.15.107:0/422543413 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:47.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 -- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 msgr2=0x7fe5e8153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e8153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 -- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5e80cbf20 msgr2=0x7fe5e8157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5e80cbf20 0x7fe5e8157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 -- 172.21.15.107:0/422543413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0009b30 con 0x7fe5e80c59b0 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e7fff640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c7f0 tx=0x7fe5d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.190+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8007820 con 0x7fe5e80c59b0 2023-11-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.191+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8007980 con 0x7fe5e80c59b0 2023-11-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.191+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8007cf0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.191+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e80c6700 con 0x7fe5e80c59b0 2023-11-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.191+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e8155930 con 0x7fe5e80c59b0 2023-11-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.192+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e81554b0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.193+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 89792+0+0 (secure 0 0 0) 0x7fe5d80113f0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.194+0000 7fe5e57fa640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 0x7fe5c806d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:47.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.194+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5d808a5e0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.194+0000 7fe5dffff640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 0x7fe5c806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:47.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.195+0000 7fe5dffff640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 0x7fe5c806d980 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5e81f6970 tx=0x7fe5d00079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:47.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.198+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5e81554b0 con 0x7fe5e80c59b0 2023-11-24T08:09:47.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:47.431+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9024", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5e80b15d0 con 0x7fe5c806b4c0 2023-11-24T08:09:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:48.403+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 14) v1 ==== 89945+0+0 (secure 0 0 0) 0x7fe5d8055050 con 0x7fe5e80c59b0 2023-11-24T08:09:53.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.793+0000 7fe5e57fa640 1 -- 172.21.15.107:0/422543413 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe5e80b15d0 con 0x7fe5c806b4c0 2023-11-24T08:09:53.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 msgr2=0x7fe5c806d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 0x7fe5c806d980 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5e81f6970 tx=0x7fe5d00079b0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 msgr2=0x7fe5e8156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c7f0 tx=0x7fe5d800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 shutdown_connections 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe5c806b4c0 0x7fe5c806d980 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5e815b440 0x7fe5e8153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5e80cbf20 0x7fe5e8157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 --2- 172.21.15.107:0/422543413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e80c59b0 0x7fe5e8156130 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 >> 172.21.15.107:0/422543413 conn(0x7fe5e80b0c50 msgr2=0x7fe5e80c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.796+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 shutdown_connections 2023-11-24T08:09:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:53.797+0000 7fe5ee451640 1 -- 172.21.15.107:0/422543413 wait complete. 2023-11-24T08:09:53.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9024 2023-11-24T08:09:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.085+0000 7f0e1e547640 1 Processor -- start 2023-11-24T08:09:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.085+0000 7f0e1e547640 1 -- start start 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1e547640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e181608f0 0x7f0e180d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1e547640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e180d3890 0x7f0e180b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1e547640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e18160cd0 con 0x7f0e180ce260 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1e547640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180b8f60 con 0x7f0e180d3890 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1e547640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180cdcb0 con 0x7f0e181608f0 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1cd44640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e180ce260 0x7f0e180d3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33254/0 (socket says 172.21.15.107:33254) 2023-11-24T08:09:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1cd44640 1 -- 172.21.15.107:0/1497121854 learned_addr learned my addr 172.21.15.107:0/1497121854 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.087+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/1497121854 >> v2:172.21.15.107:3300/0 conn(0x7f0e180d3890 0x7f0e180b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e1d545640 1 --2- 172.21.15.107:0/1497121854 >> v2:172.21.15.120:3300/0 conn(0x7f0e181608f0 0x7f0e180d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e0bfff640 1 -- 172.21.15.107:0/1497121854 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887807064 0 0) 0x7f0e18160cd0 con 0x7f0e180ce260 2023-11-24T08:09:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e0bfff640 1 -- 172.21.15.107:0/1497121854 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e04003520 con 0x7f0e180ce260 2023-11-24T08:09:54.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e1dd46640 1 -- 172.21.15.107:0/1497121854 >> v2:172.21.15.120:3300/0 conn(0x7f0e181608f0 msgr2=0x7f0e180d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:54.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/1497121854 >> v2:172.21.15.120:3300/0 conn(0x7f0e181608f0 0x7f0e180d1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e1dd46640 1 -- 172.21.15.107:0/1497121854 >> v1:172.21.15.107:6789/0 conn(0x7f0e180ce260 legacy=0x7f0e180d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.088+0000 7f0e1dd46640 1 -- 172.21.15.107:0/1497121854 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180cdf90 con 0x7f0e180d3890 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.089+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/1497121854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e180d3890 0x7f0e180b63d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0e180b90a0 tx=0x7f0e1002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab9aacd3c13eb90 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.089+0000 7f0e0bfff640 1 -- 172.21.15.107:0/1497121854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1002cc60 con 0x7f0e180d3890 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.089+0000 7f0e0bfff640 1 -- 172.21.15.107:0/1497121854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1002cdc0 con 0x7f0e180d3890 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.089+0000 7f0e0bfff640 1 -- 172.21.15.107:0/1497121854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10034640 con 0x7f0e180d3890 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 -- 172.21.15.107:0/1497121854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e180d3890 msgr2=0x7f0e180b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 --2- 172.21.15.107:0/1497121854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e180d3890 0x7f0e180b63d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0e180b90a0 tx=0x7f0e1002beb0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 -- 172.21.15.107:0/1497121854 shutdown_connections 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 --2- 172.21.15.107:0/1497121854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e180d3890 0x7f0e180b63d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 --2- 172.21.15.107:0/1497121854 >> v2:172.21.15.120:3300/0 conn(0x7f0e181608f0 0x7f0e180d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 -- 172.21.15.107:0/1497121854 >> 172.21.15.107:0/1497121854 conn(0x7f0e18061340 msgr2=0x7f0e18061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 -- 172.21.15.107:0/1497121854 shutdown_connections 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.090+0000 7f0e1e547640 1 -- 172.21.15.107:0/1497121854 wait complete. 2023-11-24T08:09:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.092+0000 7f0e1e547640 1 Processor -- start 2023-11-24T08:09:54.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.092+0000 7f0e1e547640 1 -- start start 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180ce260 0x7f0e181583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180d3890 0x7f0e181557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180ca060 con 0x7f0e181608f0 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815b000 con 0x7f0e180d3890 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1e547640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815b2e0 con 0x7f0e180ce260 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1dd46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1dd46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57678/0 (socket says 172.21.15.107:57678) 2023-11-24T08:09:54.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.093+0000 7f0e1dd46640 1 -- 172.21.15.107:0/646053874 learned_addr learned my addr 172.21.15.107:0/646053874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1cd44640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180d3890 0x7f0e181557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1d545640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180ce260 0x7f0e181583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 -- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180ce260 msgr2=0x7f0e181583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180ce260 0x7f0e181583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 -- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180d3890 msgr2=0x7f0e181557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180d3890 0x7f0e181557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 -- 172.21.15.107:0/646053874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10009a70 con 0x7f0e181608f0 2023-11-24T08:09:54.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1dd46640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0e1002c4c0 tx=0x7f0e10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1003e610 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1003e770 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1003eb60 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.094+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1815b5c0 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.095+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180cb940 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.096+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1804e840 con 0x7f0e181608f0 2023-11-24T08:09:54.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.096+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 14) v1 ==== 89945+0+0 (secure 0 0 0) 0x7f0e1003b050 con 0x7f0e181608f0 2023-11-24T08:09:54.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.097+0000 7f0e09ffb640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 0x7f0dfc06dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:54.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.097+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e100a9f30 con 0x7f0e181608f0 2023-11-24T08:09:54.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.098+0000 7f0e1d545640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 0x7f0dfc06dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:54.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.098+0000 7f0e1d545640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 0x7f0dfc06dae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0e14006fd0 tx=0x7f0e14008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:54.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.102+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e10076be0 con 0x7f0e181608f0 2023-11-24T08:09:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.332+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9024", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e180d2ea0 con 0x7f0dfc06b620 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.819+0000 7f0e09ffb640 1 -- 172.21.15.107:0/646053874 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+262 (secure 0 0 0) 0x7f0e1400f860 con 0x7f0dfc06b620 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "canceled", 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:09:54.820 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol2878", 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap4606" 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: "failure": { 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: "errno": "4", 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: "error_msg": "user interrupted clone operation" 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:09:54.821 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:09:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 msgr2=0x7f0dfc06dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 0x7f0dfc06dae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0e14006fd0 tx=0x7f0e14008040 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 msgr2=0x7f0e180b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0e1002c4c0 tx=0x7f0e10004a40 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 shutdown_connections 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b620 0x7f0dfc06dae0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181608f0 0x7f0e180b41e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180d3890 0x7f0e181557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 --2- 172.21.15.107:0/646053874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180ce260 0x7f0e181583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.822+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 >> 172.21.15.107:0/646053874 conn(0x7f0e18061340 msgr2=0x7f0e180d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.823+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 shutdown_connections 2023-11-24T08:09:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:54.823+0000 7f0e1e547640 1 -- 172.21.15.107:0/646053874 wait complete. 2023-11-24T08:09:54.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone9024 --force 2023-11-24T08:09:55.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.062+0000 7fe042ae0640 1 Processor -- start 2023-11-24T08:09:55.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.062+0000 7fe042ae0640 1 -- start start 2023-11-24T08:09:55.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.063+0000 7fe042ae0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe03c0b2250 0x7fe03c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.063+0000 7fe042ae0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c15ab80 con 0x7fe03c0b2aa0 2023-11-24T08:09:55.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.063+0000 7fe042ae0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c15e680 con 0x7fe03c0b2630 2023-11-24T08:09:55.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.063+0000 7fe042ae0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03c15a990 con 0x7fe03c0b2250 2023-11-24T08:09:55.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04087c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe03c0b2250 0x7fe03c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04107d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe03c0b2aa0 0x7fe03c15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33256/0 (socket says 172.21.15.107:33256) 2023-11-24T08:09:55.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04107d640 1 -- 172.21.15.107:0/1928325679 learned_addr learned my addr 172.21.15.107:0/1928325679 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:09:55.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04087c640 1 -- 172.21.15.107:0/1928325679 >> v1:172.21.15.120:6789/0 conn(0x7fe03c0b2630 legacy=0x7fe03c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:55.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04087c640 1 -- 172.21.15.107:0/1928325679 >> v1:172.21.15.107:6789/0 conn(0x7fe03c0b2aa0 legacy=0x7fe03c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe04087c640 1 -- 172.21.15.107:0/1928325679 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c15e960 con 0x7fe03c0b2250 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.064+0000 7fe03b7fe640 1 -- 172.21.15.107:0/1928325679 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788198957 0 0) 0x7fe03c15ab80 con 0x7fe03c0b2aa0 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.065+0000 7fe04087c640 1 --2- 172.21.15.107:0/1928325679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2250 0x7fe03c15df80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe03c15ad70 tx=0x7fe02c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401c0fa0641cac9b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe03b7fe640 1 -- 172.21.15.107:0/1928325679 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c02cc50 con 0x7fe03c0b2250 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe03b7fe640 1 -- 172.21.15.107:0/1928325679 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c02cdb0 con 0x7fe03c0b2250 2023-11-24T08:09:55.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe03b7fe640 1 -- 172.21.15.107:0/1928325679 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c034660 con 0x7fe03c0b2250 2023-11-24T08:09:55.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 -- 172.21.15.107:0/1928325679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2250 msgr2=0x7fe03c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 --2- 172.21.15.107:0/1928325679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2250 0x7fe03c15df80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe03c15ad70 tx=0x7fe02c02bea0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 -- 172.21.15.107:0/1928325679 shutdown_connections 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 --2- 172.21.15.107:0/1928325679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2250 0x7fe03c15df80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 -- 172.21.15.107:0/1928325679 >> 172.21.15.107:0/1928325679 conn(0x7fe03c06a920 msgr2=0x7fe03c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 -- 172.21.15.107:0/1928325679 shutdown_connections 2023-11-24T08:09:55.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.066+0000 7fe042ae0640 1 -- 172.21.15.107:0/1928325679 wait complete. 2023-11-24T08:09:55.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.068+0000 7fe042ae0640 1 Processor -- start 2023-11-24T08:09:55.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.068+0000 7fe042ae0640 1 -- start start 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03c0b2250 0x7fe03c0993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe03c0b2aa0 0x7fe03c156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c157530 con 0x7fe03c0b2630 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c09b250 con 0x7fe03c0b2250 2023-11-24T08:09:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe042ae0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c157a30 con 0x7fe03c0b2aa0 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe03bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe03bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57690/0 (socket says 172.21.15.107:57690) 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.069+0000 7fe03bfff640 1 -- 172.21.15.107:0/905589566 learned_addr learned my addr 172.21.15.107:0/905589566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 -- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe03c0b2aa0 msgr2=0x7fe03c156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe03c0b2aa0 0x7fe03c156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 -- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03c0b2250 msgr2=0x7fe03c0993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03c0b2250 0x7fe03c0993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 -- 172.21.15.107:0/905589566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c009a70 con 0x7fe03c0b2630 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe03bfff640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c460 tx=0x7fe03000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030012830 con 0x7fe03c0b2630 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0300129d0 con 0x7fe03c0b2630 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.070+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030012d80 con 0x7fe03c0b2630 2023-11-24T08:09:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.071+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c0cd7c0 con 0x7fe03c0b2630 2023-11-24T08:09:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.071+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c09b480 con 0x7fe03c0b2630 2023-11-24T08:09:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.072+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c067470 con 0x7fe03c0b2630 2023-11-24T08:09:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.073+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 14) v1 ==== 89945+0+0 (secure 0 0 0) 0x7fe03001b400 con 0x7fe03c0b2630 2023-11-24T08:09:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.074+0000 7fe0397fa640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 0x7fe028071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.074+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe030087c30 con 0x7fe03c0b2630 2023-11-24T08:09:55.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.074+0000 7fe04087c640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 0x7fe028071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.074+0000 7fe04087c640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 0x7fe028071ef0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe02c036000 tx=0x7fe02c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.078+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0300548e0 con 0x7fe03c0b2630 2023-11-24T08:09:55.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.297+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone9024", "force": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe03c0cd640 con 0x7fe02806fa30 2023-11-24T08:09:55.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.329+0000 7fe0397fa640 1 -- 172.21.15.107:0/905589566 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe02c034530 con 0x7fe02806fa30 2023-11-24T08:09:55.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 msgr2=0x7fe028071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 0x7fe028071ef0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe02c036000 tx=0x7fe02c02e000 comp rx=0 tx=0).stop 2023-11-24T08:09:55.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 msgr2=0x7fe03c156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c460 tx=0x7fe03000c920 comp rx=0 tx=0).stop 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 shutdown_connections 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe02806fa30 0x7fe028071ef0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe03c0b2aa0 0x7fe03c156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe03c0b2630 0x7fe03c156ab0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 --2- 172.21.15.107:0/905589566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03c0b2250 0x7fe03c0993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 >> 172.21.15.107:0/905589566 conn(0x7fe03c06a920 msgr2=0x7fe03c15b080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.332+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 shutdown_connections 2023-11-24T08:09:55.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.333+0000 7fe042ae0640 1 -- 172.21.15.107:0/905589566 wait complete. 2023-11-24T08:09:55.347 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:55.346+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:55.347 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:55.346+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:55.348 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:55.346+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:55.348 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:55.346+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:55.348 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:09:55.346+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:09:55.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol2878 subvol_snap4606 2023-11-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.549+0000 7f61c23c9640 1 Processor -- start 2023-11-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.549+0000 7f61c23c9640 1 -- start start 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.550+0000 7f61c23c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61bc153ee0 0x7f61bc0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.550+0000 7f61c23c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc06a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61c23c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 0x7f61bc06aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61c23c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc0ce140 con 0x7f61bc153ee0 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61c23c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc15a580 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61c23c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc159c70 con 0x7f61bc15aa10 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61bc153ee0 0x7f61bc0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61bc153ee0 0x7f61bc0cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57710/0 (socket says 172.21.15.107:57710) 2023-11-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 -- 172.21.15.107:0/1774526252 learned_addr learned my addr 172.21.15.107:0/1774526252 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61b3fff640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc06a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61c0966640 1 --2- 172.21.15.107:0/1774526252 >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 0x7f61bc06aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 -- 172.21.15.107:0/1774526252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 msgr2=0x7f61bc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc06a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 -- 172.21.15.107:0/1774526252 >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 msgr2=0x7f61bc06aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:55.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.551+0000 7f61bbfff640 1 --2- 172.21.15.107:0/1774526252 >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 0x7f61bc06aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.552+0000 7f61bbfff640 1 -- 172.21.15.107:0/1774526252 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc0cde60 con 0x7f61bc153ee0 2023-11-24T08:09:55.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.552+0000 7f61c0966640 1 --2- 172.21.15.107:0/1774526252 >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 0x7f61bc06aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:55.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.553+0000 7f61bbfff640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc153ee0 0x7f61bc0cd3a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0cbde0 tx=0x7f61ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eacb35bfdb5b6c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.553+0000 7f61bb7fe640 1 -- 172.21.15.107:0/1774526252 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0317e0 con 0x7f61bc153ee0 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.553+0000 7f61bb7fe640 1 -- 172.21.15.107:0/1774526252 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac031940 con 0x7f61bc153ee0 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.553+0000 7f61bb7fe640 1 -- 172.21.15.107:0/1774526252 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac031cd0 con 0x7f61bc153ee0 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 -- 172.21.15.107:0/1774526252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc153ee0 msgr2=0x7f61bc0cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc153ee0 0x7f61bc0cd3a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0cbde0 tx=0x7f61ac02b880 comp rx=0 tx=0).stop 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 -- 172.21.15.107:0/1774526252 shutdown_connections 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 --2- 172.21.15.107:0/1774526252 >> v2:172.21.15.120:3300/0 conn(0x7f61bc0cd8e0 0x7f61bc06aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc06a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 --2- 172.21.15.107:0/1774526252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc153ee0 0x7f61bc0cd3a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 -- 172.21.15.107:0/1774526252 >> 172.21.15.107:0/1774526252 conn(0x7f61bc0cf4a0 msgr2=0x7f61bc0cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 -- 172.21.15.107:0/1774526252 shutdown_connections 2023-11-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.554+0000 7f61c23c9640 1 -- 172.21.15.107:0/1774526252 wait complete. 2023-11-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.556+0000 7f61c23c9640 1 Processor -- start 2023-11-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.556+0000 7f61c23c9640 1 -- start start 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61bc153ee0 0x7f61bc1f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc1f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc14cfc0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0b21a0 con 0x7f61bc153ee0 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61c23c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc14c300 con 0x7f61bc15aa10 2023-11-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57714/0 (socket says 172.21.15.107:57714) 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 -- 172.21.15.107:0/3881641589 learned_addr learned my addr 172.21.15.107:0/3881641589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61b3fff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61bc153ee0 0x7f61bc1f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 -- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 msgr2=0x7f61bc1f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc1f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 -- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61bc153ee0 msgr2=0x7f61bc1f6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.557+0000 7f61bbfff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61bc153ee0 0x7f61bc1f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61bbfff640 1 -- 172.21.15.107:0/3881641589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0094d0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61bbfff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02b8c0 tx=0x7f61ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac032880 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac032a00 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac032db0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc14d210 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.558+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc0b23f0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.559+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc14d730 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.560+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 14) v1 ==== 89945+0+0 (secure 0 0 0) 0x7f61ac04d3f0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.561+0000 7f61b97fa640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 0x7f619c06dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:55.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.561+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61ac0a9af0 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.561+0000 7f61b3fff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 0x7f619c06dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:55.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.562+0000 7f61b3fff640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 0x7f619c06dae0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006fd0 tx=0x7f61a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.566+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61ac076850 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.788+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol2878", "snap_name": "subvol_snap4606", "target": ["mon-mgr", ""]}) v1 -- 0x7f61bc071800 con 0x7f619c06b620 2023-11-24T08:09:55.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.845+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f61ac076c20 con 0x7f61bc0cd8e0 2023-11-24T08:09:55.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.858+0000 7f61b97fa640 1 -- 172.21.15.107:0/3881641589 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f61a4004030 con 0x7f619c06b620 2023-11-24T08:09:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 msgr2=0x7f619c06dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 0x7f619c06dae0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006fd0 tx=0x7f61a4008040 comp rx=0 tx=0).stop 2023-11-24T08:09:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 msgr2=0x7f61bc0cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02b8c0 tx=0x7f61ac0046c0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 shutdown_connections 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f619c06b620 0x7f619c06dae0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61bc15aa10 0x7f61bc1f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61bc153ee0 0x7f61bc1f6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 --2- 172.21.15.107:0/3881641589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61bc0cd8e0 0x7f61bc0cb280 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 >> 172.21.15.107:0/3881641589 conn(0x7f61bc0cf4a0 msgr2=0x7f61bc15b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 shutdown_connections 2023-11-24T08:09:55.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:55.861+0000 7f61c23c9640 1 -- 172.21.15.107:0/3881641589 wait complete. 2023-11-24T08:09:55.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol2878 2023-11-24T08:09:56.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.085+0000 7fe3ad875640 1 Processor -- start 2023-11-24T08:09:56.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.085+0000 7fe3ad875640 1 -- start start 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.087+0000 7fe3ad875640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3a8146eb0 0x7fe3a8161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.087+0000 7fe3ad875640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3a80cd2e0 0x7fe3a8070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.087+0000 7fe3ad875640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a8161a10 con 0x7fe3a806e160 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.087+0000 7fe3ad875640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a8161cf0 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.087+0000 7fe3ad875640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a8161fd0 con 0x7fe3a8146eb0 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3a80cd2e0 0x7fe3a8070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3a80cd2e0 0x7fe3a8070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57726/0 (socket says 172.21.15.107:57726) 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 -- 172.21.15.107:0/674403693 learned_addr learned my addr 172.21.15.107:0/674403693 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a6ffd640 1 --2- 172.21.15.107:0/674403693 >> v2:172.21.15.120:3300/0 conn(0x7fe3a8146eb0 0x7fe3a8161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 -- 172.21.15.107:0/674403693 >> v2:172.21.15.120:3300/0 conn(0x7fe3a8146eb0 msgr2=0x7fe3a8161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 --2- 172.21.15.107:0/674403693 >> v2:172.21.15.120:3300/0 conn(0x7fe3a8146eb0 0x7fe3a8161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 -- 172.21.15.107:0/674403693 >> v1:172.21.15.120:6789/0 conn(0x7fe3a806e160 legacy=0x7fe3a80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.088+0000 7fe3a77fe640 1 -- 172.21.15.107:0/674403693 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8070660 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.089+0000 7fe3a77fe640 1 --2- 172.21.15.107:0/674403693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8070e40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe3a806ea50 tx=0x7fe3a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336ea0917a25969d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.090+0000 7fe3a67fc640 1 -- 172.21.15.107:0/674403693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a002ca80 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.090+0000 7fe3a67fc640 1 -- 172.21.15.107:0/674403693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a002cbe0 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.090+0000 7fe3a67fc640 1 -- 172.21.15.107:0/674403693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00343f0 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.090+0000 7fe3ad875640 1 -- 172.21.15.107:0/674403693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a80cd2e0 msgr2=0x7fe3a8070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.090+0000 7fe3ad875640 1 --2- 172.21.15.107:0/674403693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8070e40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe3a806ea50 tx=0x7fe3a002bee0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 -- 172.21.15.107:0/674403693 shutdown_connections 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 --2- 172.21.15.107:0/674403693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8070e40 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 --2- 172.21.15.107:0/674403693 >> v2:172.21.15.120:3300/0 conn(0x7fe3a8146eb0 0x7fe3a8161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 -- 172.21.15.107:0/674403693 >> 172.21.15.107:0/674403693 conn(0x7fe3a80b0c50 msgr2=0x7fe3a80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 -- 172.21.15.107:0/674403693 shutdown_connections 2023-11-24T08:09:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.091+0000 7fe3ad875640 1 -- 172.21.15.107:0/674403693 wait complete. 2023-11-24T08:09:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.093+0000 7fe3ad875640 1 Processor -- start 2023-11-24T08:09:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.093+0000 7fe3ad875640 1 -- start start 2023-11-24T08:09:56.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 0x7fe3a80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a813a240 con 0x7fe3a806e160 2023-11-24T08:09:56.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a815e650 con 0x7fe3a80cd2e0 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3ad875640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a815e8f0 con 0x7fe3a8146eb0 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3a6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3a6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57732/0 (socket says 172.21.15.107:57732) 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3a6ffd640 1 -- 172.21.15.107:0/4114398471 learned_addr learned my addr 172.21.15.107:0/4114398471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe39e7fc640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.094+0000 7fe3a77fe640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 0x7fe3a80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 -- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 msgr2=0x7fe3a80b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 0x7fe3a80b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 -- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 msgr2=0x7fe3a8160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 -- 172.21.15.107:0/4114398471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0009b30 con 0x7fe3a806e160 2023-11-24T08:09:56.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe39e7fc640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a77fe640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 0x7fe3a80b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a6ffd640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe39800d5f0 tx=0x7fe39800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980125f0 con 0x7fe3a806e160 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398012750 con 0x7fe3a806e160 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a815ebd0 con 0x7fe3a806e160 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398012b00 con 0x7fe3a806e160 2023-11-24T08:09:56.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.095+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a815f190 con 0x7fe3a806e160 2023-11-24T08:09:56.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.097+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a8067680 con 0x7fe3a806e160 2023-11-24T08:09:56.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.097+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fe398012c60 con 0x7fe3a806e160 2023-11-24T08:09:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.099+0000 7fe3a4ff9640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 0x7fe38806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:09:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.099+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe398088530 con 0x7fe3a806e160 2023-11-24T08:09:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.099+0000 7fe39e7fc640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 0x7fe38806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:09:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.100+0000 7fe39e7fc640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 0x7fe38806db70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80b34b0 tx=0x7fe390009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:09:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.104+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe398054e60 con 0x7fe3a806e160 2023-11-24T08:09:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.323+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol2878", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3a8146a70 con 0x7fe38806b6b0 2023-11-24T08:09:56.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.362+0000 7fe3a4ff9640 1 -- 172.21.15.107:0/4114398471 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe3a8146a70 con 0x7fe38806b6b0 2023-11-24T08:09:56.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.364+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 msgr2=0x7fe38806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:56.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.364+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 0x7fe38806db70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80b34b0 tx=0x7fe390009040 comp rx=0 tx=0).stop 2023-11-24T08:09:56.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 msgr2=0x7fe3a8139d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:09:56.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe39800d5f0 tx=0x7fe39800dab0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 shutdown_connections 2023-11-24T08:09:56.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe38806b6b0 0x7fe38806db70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a8146eb0 0x7fe3a80b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a80cd2e0 0x7fe3a8160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 --2- 172.21.15.107:0/4114398471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a806e160 0x7fe3a8139d00 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:09:56.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 >> 172.21.15.107:0/4114398471 conn(0x7fe3a80b0c50 msgr2=0x7fe3a81454e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:09:56.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 shutdown_connections 2023-11-24T08:09:56.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:09:56.365+0000 7fe3ad875640 1 -- 172.21.15.107:0/4114398471 wait complete. 2023-11-24T08:09:56.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:09:56.387 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:09:56.438 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:10:01.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:01.439 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:10:01.503 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:10:01.504 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:10:01.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:10:01.626 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:10:01.751 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:10:01.751 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:01.751 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:01.812 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:01.812 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:01.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.077+0000 7fc5bf402640 1 Processor -- start 2023-11-24T08:10:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.077+0000 7fc5bf402640 1 -- start start 2023-11-24T08:10:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.078+0000 7fc5bf402640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5b815c7f0 0x7fc5b806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.078+0000 7fc5bf402640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc5b80cf450 0x7fc5b80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.078+0000 7fc5bf402640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cece0 con 0x7fc5b80cdfb0 2023-11-24T08:10:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.078+0000 7fc5bf402640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b80cfe40 con 0x7fc5b815c7f0 2023-11-24T08:10:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.078+0000 7fc5bf402640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b80ceaf0 con 0x7fc5b80cf450 2023-11-24T08:10:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.079+0000 7fc5bd19e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5b815c7f0 0x7fc5b806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.079+0000 7fc5bd99f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc5b80cdfb0 0x7fc5b80ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33952/0 (socket says 172.21.15.107:33952) 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.079+0000 7fc5bd99f640 1 -- 172.21.15.107:0/1720962029 learned_addr learned my addr 172.21.15.107:0/1720962029 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.079+0000 7fc5bc99d640 1 --2- 172.21.15.107:0/1720962029 >> v2:172.21.15.120:3300/0 conn(0x7fc5b80cf450 0x7fc5b80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.079+0000 7fc5bd19e640 1 -- 172.21.15.107:0/1720962029 >> v2:172.21.15.120:3300/0 conn(0x7fc5b80cf450 msgr2=0x7fc5b80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.080+0000 7fc5bd19e640 1 --2- 172.21.15.107:0/1720962029 >> v2:172.21.15.120:3300/0 conn(0x7fc5b80cf450 0x7fc5b80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.080+0000 7fc5bd19e640 1 -- 172.21.15.107:0/1720962029 >> v1:172.21.15.107:6789/0 conn(0x7fc5b80cdfb0 legacy=0x7fc5b80ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.080+0000 7fc5affff640 1 -- 172.21.15.107:0/1720962029 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933900313 0 0) 0x7fc5b80cece0 con 0x7fc5b80cdfb0 2023-11-24T08:10:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.080+0000 7fc5bd19e640 1 -- 172.21.15.107:0/1720962029 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b806d040 con 0x7fc5b815c7f0 2023-11-24T08:10:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.081+0000 7fc5bd19e640 1 --2- 172.21.15.107:0/1720962029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b806db20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8070410 tx=0x7fc5b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8447c7ce7800c06c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.081+0000 7fc5affff640 1 -- 172.21.15.107:0/1720962029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4031830 con 0x7fc5b815c7f0 2023-11-24T08:10:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.081+0000 7fc5affff640 1 -- 172.21.15.107:0/1720962029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4031990 con 0x7fc5b815c7f0 2023-11-24T08:10:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.081+0000 7fc5affff640 1 -- 172.21.15.107:0/1720962029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4031d20 con 0x7fc5b815c7f0 2023-11-24T08:10:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 -- 172.21.15.107:0/1720962029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 msgr2=0x7fc5b806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1720962029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b806db20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8070410 tx=0x7fc5b402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 -- 172.21.15.107:0/1720962029 shutdown_connections 2023-11-24T08:10:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1720962029 >> v2:172.21.15.120:3300/0 conn(0x7fc5b80cf450 0x7fc5b80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1720962029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b806db20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 -- 172.21.15.107:0/1720962029 >> 172.21.15.107:0/1720962029 conn(0x7fc5b806a920 msgr2=0x7fc5b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 -- 172.21.15.107:0/1720962029 shutdown_connections 2023-11-24T08:10:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.082+0000 7fc5bf402640 1 -- 172.21.15.107:0/1720962029 wait complete. 2023-11-24T08:10:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.084+0000 7fc5bf402640 1 Processor -- start 2023-11-24T08:10:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.084+0000 7fc5bf402640 1 -- start start 2023-11-24T08:10:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b80cdfb0 0x7fc5b80abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b80cf450 0x7fc5b815f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80cb220 con 0x7fc5b815c7f0 2023-11-24T08:10:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80cbd30 con 0x7fc5b80cdfb0 2023-11-24T08:10:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.085+0000 7fc5bf402640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80acb50 con 0x7fc5b80cf450 2023-11-24T08:10:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38198/0 (socket says 172.21.15.107:38198) 2023-11-24T08:10:02.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 -- 172.21.15.107:0/1400927706 learned_addr learned my addr 172.21.15.107:0/1400927706 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd19e640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b80cdfb0 0x7fc5b80abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bc99d640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b80cf450 0x7fc5b815f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 -- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b80cf450 msgr2=0x7fc5b815f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b80cf450 0x7fc5b815f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 -- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b80cdfb0 msgr2=0x7fc5b80abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b80cdfb0 0x7fc5b80abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 -- 172.21.15.107:0/1400927706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4009400 con 0x7fc5b815c7f0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.086+0000 7fc5bd99f640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0002230 tx=0x7fc5b000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.087+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000ce30 con 0x7fc5b815c7f0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.087+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000a640 con 0x7fc5b815c7f0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.087+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000a9f0 con 0x7fc5b815c7f0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.087+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80ace30 con 0x7fc5b815c7f0 2023-11-24T08:10:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.087+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b8150c30 con 0x7fc5b815c7f0 2023-11-24T08:10:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.088+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8067470 con 0x7fc5b815c7f0 2023-11-24T08:10:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.089+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fc5b000ab50 con 0x7fc5b815c7f0 2023-11-24T08:10:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.090+0000 7fc5adffb640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 0x7fc59c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.090+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fc5b0087590 con 0x7fc5b815c7f0 2023-11-24T08:10:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.090+0000 7fc5bd19e640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 0x7fc59c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.091+0000 7fc5bd19e640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 0x7fc59c06db70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4036000 tx=0x7fc5b403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.094+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5b0053ec0 con 0x7fc5b815c7f0 2023-11-24T08:10:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.284+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b8153ba0 con 0x7fc5b815c7f0 2023-11-24T08:10:02.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.286+0000 7fc5adffb640 1 -- 172.21.15.107:0/1400927706 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fc5b0018070 con 0x7fc5b815c7f0 2023-11-24T08:10:02.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:02.288 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:59.607764+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.288+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 msgr2=0x7fc59c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.288+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 0x7fc59c06db70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4036000 tx=0x7fc5b403c040 comp rx=0 tx=0).stop 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.288+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 msgr2=0x7fc5b815f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.288+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0002230 tx=0x7fc5b000cbe0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 shutdown_connections 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc59c06b6b0 0x7fc59c06db70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b815c7f0 0x7fc5b815f610 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b80cf450 0x7fc5b815f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 --2- 172.21.15.107:0/1400927706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b80cdfb0 0x7fc5b80abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 >> 172.21.15.107:0/1400927706 conn(0x7fc5b806a920 msgr2=0x7fc5b81547d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 shutdown_connections 2023-11-24T08:10:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.289+0000 7fc5bf402640 1 -- 172.21.15.107:0/1400927706 wait complete. 2023-11-24T08:10:02.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:09:02.329269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:09:03.506902+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:10:02.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:10:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.522+0000 7fb91b933640 1 Processor -- start 2023-11-24T08:10:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.522+0000 7fb91b933640 1 -- start start 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.523+0000 7fb91b933640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91415b440 0x7fb9140cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.523+0000 7fb91b933640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9140cbf20 0x7fb9140c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.523+0000 7fb91b933640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb9140c57e0 0x7fb9140c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.523+0000 7fb91b933640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91415b820 con 0x7fb9140cbf20 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb91b933640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140cc3a0 con 0x7fb9140c57e0 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb91b933640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91415eea0 con 0x7fb91415b440 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb918ece640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9140cbf20 0x7fb9140c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb918ece640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9140cbf20 0x7fb9140c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38204/0 (socket says 172.21.15.107:38204) 2023-11-24T08:10:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb918ece640 1 -- 172.21.15.107:0/1823946197 learned_addr learned my addr 172.21.15.107:0/1823946197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb919ed0640 1 --2- 172.21.15.107:0/1823946197 >> v2:172.21.15.120:3300/0 conn(0x7fb9140c57e0 0x7fb9140c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.524+0000 7fb9196cf640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91415b440 0x7fb9140cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.525+0000 7fb918ece640 1 -- 172.21.15.107:0/1823946197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91415b440 msgr2=0x7fb9140cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.525+0000 7fb918ece640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91415b440 0x7fb9140cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.525+0000 7fb918ece640 1 -- 172.21.15.107:0/1823946197 >> v2:172.21.15.120:3300/0 conn(0x7fb9140c57e0 msgr2=0x7fb9140c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.525+0000 7fb918ece640 1 --2- 172.21.15.107:0/1823946197 >> v2:172.21.15.120:3300/0 conn(0x7fb9140c57e0 0x7fb9140c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.525+0000 7fb918ece640 1 -- 172.21.15.107:0/1823946197 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140abf30 con 0x7fb9140cbf20 2023-11-24T08:10:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.526+0000 7fb918ece640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140cbf20 0x7fb9140c52a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb9140cb910 tx=0x7fb90402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d4b52c9e35f1a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb90bfff640 1 -- 172.21.15.107:0/1823946197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9040317e0 con 0x7fb9140cbf20 2023-11-24T08:10:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb90bfff640 1 -- 172.21.15.107:0/1823946197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb904031940 con 0x7fb9140cbf20 2023-11-24T08:10:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb90bfff640 1 -- 172.21.15.107:0/1823946197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904031cd0 con 0x7fb9140cbf20 2023-11-24T08:10:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 -- 172.21.15.107:0/1823946197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140cbf20 msgr2=0x7fb9140c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140cbf20 0x7fb9140c52a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb9140cb910 tx=0x7fb90402b880 comp rx=0 tx=0).stop 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 -- 172.21.15.107:0/1823946197 shutdown_connections 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 --2- 172.21.15.107:0/1823946197 >> v2:172.21.15.120:3300/0 conn(0x7fb9140c57e0 0x7fb9140c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140cbf20 0x7fb9140c52a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 --2- 172.21.15.107:0/1823946197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91415b440 0x7fb9140cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 -- 172.21.15.107:0/1823946197 >> 172.21.15.107:0/1823946197 conn(0x7fb9140b0c50 msgr2=0x7fb9140b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.527+0000 7fb91b933640 1 -- 172.21.15.107:0/1823946197 shutdown_connections 2023-11-24T08:10:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.528+0000 7fb91b933640 1 -- 172.21.15.107:0/1823946197 wait complete. 2023-11-24T08:10:02.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.529+0000 7fb91b933640 1 Processor -- start 2023-11-24T08:10:02.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.530+0000 7fb91b933640 1 -- start start 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 0x7fb9141524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 0x7fb91414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914152f20 con 0x7fb9140c57e0 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914153200 con 0x7fb91415b440 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb91b933640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141534e0 con 0x7fb9140cbf20 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38210/0 (socket says 172.21.15.107:38210) 2023-11-24T08:10:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 -- 172.21.15.107:0/2731848391 learned_addr learned my addr 172.21.15.107:0/2731848391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb919ed0640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 0x7fb91414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb918ece640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 0x7fb9141524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 -- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 msgr2=0x7fb9141524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 0x7fb9141524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 -- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 msgr2=0x7fb91414b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.531+0000 7fb9196cf640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 0x7fb91414b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb9196cf640 1 -- 172.21.15.107:0/2731848391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9040094d0 con 0x7fb9140c57e0 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb919ed0640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 0x7fb91414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb918ece640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 0x7fb9141524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb9196cf640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d5f0 tx=0x7fb91000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100125f0 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910012750 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910012b00 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9141537c0 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.532+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb914153d80 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.533+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140c5330 con 0x7fb9140c57e0 2023-11-24T08:10:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.534+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fb910012c60 con 0x7fb9140c57e0 2023-11-24T08:10:02.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.535+0000 7fb909ffb640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 0x7fb8f806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.535+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb9100885d0 con 0x7fb9140c57e0 2023-11-24T08:10:02.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.535+0000 7fb918ece640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 0x7fb8f806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.536+0000 7fb918ece640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 0x7fb8f806db70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb904036000 tx=0x7fb90403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.540+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb910054eb0 con 0x7fb9140c57e0 2023-11-24T08:10:02.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.730+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb91404e8b0 con 0x7fb9140c57e0 2023-11-24T08:10:02.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.731+0000 7fb909ffb640 1 -- 172.21.15.107:0/2731848391 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9100584d0 con 0x7fb9140c57e0 2023-11-24T08:10:02.733 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 msgr2=0x7fb8f806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 0x7fb8f806db70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb904036000 tx=0x7fb90403d040 comp rx=0 tx=0).stop 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 msgr2=0x7fb9141503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d5f0 tx=0x7fb91000dab0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 shutdown_connections 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8f806b6b0 0x7fb8f806db70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91415b440 0x7fb91414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9140cbf20 0x7fb9141524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 --2- 172.21.15.107:0/2731848391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9140c57e0 0x7fb9141503b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 >> 172.21.15.107:0/2731848391 conn(0x7fb9140b0c50 msgr2=0x7fb9140b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 shutdown_connections 2023-11-24T08:10:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.734+0000 7fb91b933640 1 -- 172.21.15.107:0/2731848391 wait complete. 2023-11-24T08:10:02.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:10:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.972+0000 7f28dba61640 1 Processor -- start 2023-11-24T08:10:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.972+0000 7f28dba61640 1 -- start start 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28dba61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 0x7f28d4166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28dba61640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d415eea0 con 0x7f28d40cc0f0 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28dba61640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d406d600 con 0x7f28d415b440 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28dba61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40c5e00 con 0x7f28d40c59b0 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28d8ffc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f28d40cc0f0 0x7f28d40c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33962/0 (socket says 172.21.15.107:33962) 2023-11-24T08:10:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.974+0000 7f28d8ffc640 1 -- 172.21.15.107:0/1623999415 learned_addr learned my addr 172.21.15.107:0/1623999415 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28d9ffe640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 0x7f28d4166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103661963 0 0) 0x7f28d415eea0 con 0x7f28d40cc0f0 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0003520 con 0x7f28d40cc0f0 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2153695300 0 0) 0x7f28c0003520 con 0x7f28d40cc0f0 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 msgr2=0x7f28d4166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 0x7f28d4166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 >> v1:172.21.15.120:6789/0 conn(0x7f28d415b440 legacy=0x7f28d40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40abf30 con 0x7f28d40cc0f0 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f28c4001670 con 0x7f28d40cc0f0 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 >> v1:172.21.15.107:6789/0 conn(0x7f28d40cc0f0 legacy=0x7f28d40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28c00039a0 0x7f28c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 0x7f28c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28c0004d00 0x7f28c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c000ba80 con 0x7f28c00039a0 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c000bbf0 con 0x7f28c0004d00 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.975+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c000bdb0 con 0x7f28d40c59b0 2023-11-24T08:10:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28c00039a0 0x7f28c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 -- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 msgr2=0x7f28c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40c59b0 0x7f28c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 -- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28c0004d00 msgr2=0x7f28c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28c0004d00 0x7f28c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.976+0000 7f28d97fd640 1 -- 172.21.15.107:0/1623999415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0003520 con 0x7f28c00039a0 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.977+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28c00039a0 0x7f28c0003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f28d0001750 tx=0x7f28c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4beb7d85737b4ac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.977+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00052d0 con 0x7f28c00039a0 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.977+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d002e050 con 0x7f28c00039a0 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.977+0000 7f28cbfff640 1 -- 172.21.15.107:0/1623999415 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00035f0 con 0x7f28c00039a0 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 -- 172.21.15.107:0/1623999415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28c00039a0 msgr2=0x7f28c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 --2- 172.21.15.107:0/1623999415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28c00039a0 0x7f28c0003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f28d0001750 tx=0x7f28c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28d9ffe640 1 -- 172.21.15.107:0/1623999415 reap_dead start 2023-11-24T08:10:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 -- 172.21.15.107:0/1623999415 shutdown_connections 2023-11-24T08:10:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 -- 172.21.15.107:0/1623999415 >> 172.21.15.107:0/1623999415 conn(0x7f28d40b0c50 msgr2=0x7f28d40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 -- 172.21.15.107:0/1623999415 shutdown_connections 2023-11-24T08:10:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.978+0000 7f28dba61640 1 -- 172.21.15.107:0/1623999415 wait complete. 2023-11-24T08:10:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.980+0000 7f28dba61640 1 Processor -- start 2023-11-24T08:10:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.980+0000 7f28dba61640 1 -- start start 2023-11-24T08:10:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28c00039a0 0x7f28d40c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 0x7f28d40cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d409a3a0 con 0x7f28d40b4850 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4156a30 con 0x7f28d40cce70 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28dba61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4099a70 con 0x7f28c00039a0 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38236/0 (socket says 172.21.15.107:38236) 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 -- 172.21.15.107:0/1464559259 learned_addr learned my addr 172.21.15.107:0/1464559259 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d9ffe640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 0x7f28d40cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 -- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28c00039a0 msgr2=0x7f28d40c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28c00039a0 0x7f28d40c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28c00039a0 0x7f28d40c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 -- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 msgr2=0x7f28d40cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 0x7f28d40cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.981+0000 7f28d8ffc640 1 -- 172.21.15.107:0/1464559259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4099ce0 con 0x7f28d40b4850 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28d9ffe640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 0x7f28d40cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:02.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28d8ffc640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c760 tx=0x7f28c400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c40074e0 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c4007680 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4004a00 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d4099fc0 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.982+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d40cd830 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.984+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d409a4e0 con 0x7f28d40b4850 2023-11-24T08:10:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.984+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f28c4004b60 con 0x7f28d40b4850 2023-11-24T08:10:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.986+0000 7f28c9ffb640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 0x7f28b806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.986+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f28c40874c0 con 0x7f28d40b4850 2023-11-24T08:10:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.986+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 0x7f28b806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.987+0000 7f28d97fd640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 0x7f28b806db70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f28d0033000 tx=0x7f28d0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:02.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:02.990+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28d409a4e0 con 0x7f28d40b4850 2023-11-24T08:10:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.181+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f28d409a4e0 con 0x7f28d40b4850 2023-11-24T08:10:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.182+0000 7f28c9ffb640 1 -- 172.21.15.107:0/1464559259 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) v1 ==== 104+0+11 (secure 0 0 0) 0x7f28d409a4e0 con 0x7f28d40b4850 2023-11-24T08:10:03.184 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 msgr2=0x7f28b806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 0x7f28b806db70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f28d0033000 tx=0x7f28d0009030 comp rx=0 tx=0).stop 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 msgr2=0x7f28d40c87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c760 tx=0x7f28c400cc20 comp rx=0 tx=0).stop 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 shutdown_connections 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f28b806b6b0 0x7f28b806db70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d40cce70 0x7f28d40cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28d40b4850 0x7f28d40c87e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 --2- 172.21.15.107:0/1464559259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28c00039a0 0x7f28d40c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 >> 172.21.15.107:0/1464559259 conn(0x7f28d40b0c50 msgr2=0x7f28d40cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 shutdown_connections 2023-11-24T08:10:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.185+0000 7f28dba61640 1 -- 172.21.15.107:0/1464559259 wait complete. 2023-11-24T08:10:03.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:10:03.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.419+0000 7f2c3b173640 1 Processor -- start 2023-11-24T08:10:03.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.419+0000 7f2c3b173640 1 -- start start 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3b173640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3415a1c0 0x7f2c3406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3b173640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c340b0b40 0x7f2c340b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3b173640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3406d430 con 0x7f2c3415ded0 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3b173640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3415ea70 con 0x7f2c3415a1c0 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3b173640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3415ec50 con 0x7f2c340b0b40 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3a171640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3415a1c0 0x7f2c3406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3a171640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3415a1c0 0x7f2c3406d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38242/0 (socket says 172.21.15.107:38242) 2023-11-24T08:10:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.421+0000 7f2c3a171640 1 -- 172.21.15.107:0/1303556595 learned_addr learned my addr 172.21.15.107:0/1303556595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.422+0000 7f2c3a171640 1 -- 172.21.15.107:0/1303556595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c340b0b40 msgr2=0x7f2c340b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.422+0000 7f2c3a171640 1 --2- 172.21.15.107:0/1303556595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c340b0b40 0x7f2c340b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.422+0000 7f2c3a171640 1 -- 172.21.15.107:0/1303556595 >> v1:172.21.15.120:6789/0 conn(0x7f2c3415ded0 legacy=0x7f2c3415e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.422+0000 7f2c3a171640 1 -- 172.21.15.107:0/1303556595 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340b14e0 con 0x7f2c3415a1c0 2023-11-24T08:10:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.423+0000 7f2c3a171640 1 --2- 172.21.15.107:0/1303556595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c3406d8c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d674848f9fcd8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.423+0000 7f2c3916f640 1 -- 172.21.15.107:0/1303556595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30031640 con 0x7f2c3415a1c0 2023-11-24T08:10:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.423+0000 7f2c3916f640 1 -- 172.21.15.107:0/1303556595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c300317a0 con 0x7f2c3415a1c0 2023-11-24T08:10:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.423+0000 7f2c3916f640 1 -- 172.21.15.107:0/1303556595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30031b30 con 0x7f2c3415a1c0 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 -- 172.21.15.107:0/1303556595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 msgr2=0x7f2c3406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 --2- 172.21.15.107:0/1303556595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c3406d8c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 -- 172.21.15.107:0/1303556595 shutdown_connections 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 --2- 172.21.15.107:0/1303556595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c340b0b40 0x7f2c340b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 --2- 172.21.15.107:0/1303556595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c3406d8c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 -- 172.21.15.107:0/1303556595 >> 172.21.15.107:0/1303556595 conn(0x7f2c340cbb40 msgr2=0x7f2c340cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 -- 172.21.15.107:0/1303556595 shutdown_connections 2023-11-24T08:10:03.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.424+0000 7f2c3b173640 1 -- 172.21.15.107:0/1303556595 wait complete. 2023-11-24T08:10:03.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.426+0000 7f2c3b173640 1 Processor -- start 2023-11-24T08:10:03.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.427+0000 7f2c3b173640 1 -- start start 2023-11-24T08:10:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c340b0b40 0x7f2c341f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 0x7f2c340c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c34150a50 con 0x7f2c3415a1c0 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340c7060 con 0x7f2c340b0b40 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c3b173640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340c7340 con 0x7f2c3415ded0 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c39970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c39970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38256/0 (socket says 172.21.15.107:38256) 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.428+0000 7f2c39970640 1 -- 172.21.15.107:0/876994148 learned_addr learned my addr 172.21.15.107:0/876994148 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c3a171640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c340b0b40 0x7f2c341f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c3a972640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 0x7f2c340c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 -- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 msgr2=0x7f2c340c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 0x7f2c340c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 -- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c340b0b40 msgr2=0x7f2c341f1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c340b0b40 0x7f2c341f1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 -- 172.21.15.107:0/876994148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300094d0 con 0x7f2c3415a1c0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c3a972640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 0x7f2c340c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c39970640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400b4a0 tx=0x7f2c2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24010610 con 0x7f2c3415a1c0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c24010770 con 0x7f2c3415a1c0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24010b20 con 0x7f2c3415a1c0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.429+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c340c7620 con 0x7f2c3415a1c0 2023-11-24T08:10:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.430+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c340c7be0 con 0x7f2c3415a1c0 2023-11-24T08:10:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.431+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c340cb730 con 0x7f2c3415a1c0 2023-11-24T08:10:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.432+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f2c24010c80 con 0x7f2c3415a1c0 2023-11-24T08:10:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.433+0000 7f2c2affd640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 0x7f2c1806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.433+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f2c24087ca0 con 0x7f2c3415a1c0 2023-11-24T08:10:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.433+0000 7f2c3a171640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 0x7f2c1806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.434+0000 7f2c3a171640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 0x7f2c1806db70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.438+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c24054630 con 0x7f2c3415a1c0 2023-11-24T08:10:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.636+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c3404e840 con 0x7f2c3415a1c0 2023-11-24T08:10:03.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.643+0000 7f2c2affd640 1 -- 172.21.15.107:0/876994148 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c24014020 con 0x7f2c3415a1c0 2023-11-24T08:10:03.645 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 msgr2=0x7f2c1806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 0x7f2c1806db70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003c040 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 msgr2=0x7f2c340c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400b4a0 tx=0x7f2c2400b960 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 shutdown_connections 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2c1806b6b0 0x7f2c1806db70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c3415ded0 0x7f2c340c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3415a1c0 0x7f2c340c8380 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 --2- 172.21.15.107:0/876994148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c340b0b40 0x7f2c341f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.646+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 >> 172.21.15.107:0/876994148 conn(0x7f2c340cbb40 msgr2=0x7f2c34070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.647+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 shutdown_connections 2023-11-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.647+0000 7f2c3b173640 1 -- 172.21.15.107:0/876994148 wait complete. 2023-11-24T08:10:03.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:10:03.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.874+0000 7ff1c48e1640 1 Processor -- start 2023-11-24T08:10:03.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.875+0000 7ff1c48e1640 1 -- start start 2023-11-24T08:10:03.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1c48e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1c00b63d0 0x7ff1c00d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1c48e1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1c00ce270 0x7ff1c00d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1c48e1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c00cea30 con 0x7ff1c015d490 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1c48e1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c00d0450 con 0x7ff1c00b63d0 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1c48e1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c00d0bf0 con 0x7ff1c00ce270 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1beffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1c00b63d0 0x7ff1c00d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.876+0000 7ff1beffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1c00b63d0 0x7ff1c00d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38268/0 (socket says 172.21.15.107:38268) 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.877+0000 7ff1beffd640 1 -- 172.21.15.107:0/117748872 learned_addr learned my addr 172.21.15.107:0/117748872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:03.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.877+0000 7ff1beffd640 1 -- 172.21.15.107:0/117748872 >> v2:172.21.15.120:3300/0 conn(0x7ff1c00ce270 msgr2=0x7ff1c00d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.877+0000 7ff1beffd640 1 --2- 172.21.15.107:0/117748872 >> v2:172.21.15.120:3300/0 conn(0x7ff1c00ce270 0x7ff1c00d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.877+0000 7ff1beffd640 1 -- 172.21.15.107:0/117748872 >> v1:172.21.15.120:6789/0 conn(0x7ff1c015d490 legacy=0x7ff1c00cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.877+0000 7ff1beffd640 1 -- 172.21.15.107:0/117748872 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c00d2260 con 0x7ff1c00b63d0 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.878+0000 7ff1beffd640 1 --2- 172.21.15.107:0/117748872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c00b63d0 0x7ff1c00d3230 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0161f30 tx=0x7ff1a802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f27d6bca4678db7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.878+0000 7ff1be7fc640 1 -- 172.21.15.107:0/117748872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8031890 con 0x7ff1c00b63d0 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.878+0000 7ff1be7fc640 1 -- 172.21.15.107:0/117748872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a80319f0 con 0x7ff1c00b63d0 2023-11-24T08:10:03.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.878+0000 7ff1be7fc640 1 -- 172.21.15.107:0/117748872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8031d80 con 0x7ff1c00b63d0 2023-11-24T08:10:03.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 -- 172.21.15.107:0/117748872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c00b63d0 msgr2=0x7ff1c00d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/117748872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c00b63d0 0x7ff1c00d3230 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0161f30 tx=0x7ff1a802b930 comp rx=0 tx=0).stop 2023-11-24T08:10:03.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 -- 172.21.15.107:0/117748872 shutdown_connections 2023-11-24T08:10:03.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/117748872 >> v2:172.21.15.120:3300/0 conn(0x7ff1c00ce270 0x7ff1c00d3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/117748872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c00b63d0 0x7ff1c00d3230 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 -- 172.21.15.107:0/117748872 >> 172.21.15.107:0/117748872 conn(0x7ff1c0061340 msgr2=0x7ff1c0061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:03.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 -- 172.21.15.107:0/117748872 shutdown_connections 2023-11-24T08:10:03.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.879+0000 7ff1c48e1640 1 -- 172.21.15.107:0/117748872 wait complete. 2023-11-24T08:10:03.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.881+0000 7ff1c48e1640 1 Processor -- start 2023-11-24T08:10:03.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.881+0000 7ff1c48e1640 1 -- start start 2023-11-24T08:10:03.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c00ce270 0x7ff1c0151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c00cc610 0x7ff1c01e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01e8440 con 0x7ff1c015d490 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c00cca90 con 0x7ff1c00cc610 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1c48e1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01e7df0 con 0x7ff1c00ce270 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.882+0000 7ff1beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38276/0 (socket says 172.21.15.107:38276) 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 -- 172.21.15.107:0/2700902160 learned_addr learned my addr 172.21.15.107:0/2700902160 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 -- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c00ce270 msgr2=0x7ff1c0151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c00ce270 0x7ff1c0151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 -- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c00cc610 msgr2=0x7ff1c01e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c00cc610 0x7ff1c01e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:03.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 -- 172.21.15.107:0/2700902160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a80094d0 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1beffd640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8000c00 tx=0x7ff1a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.883+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a803acb0 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.884+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a803ae10 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.884+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8044610 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.884+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c01e80d0 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.884+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c01e7a10 con 0x7ff1c015d490 2023-11-24T08:10:03.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.885+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c01e7c60 con 0x7ff1c015d490 2023-11-24T08:10:03.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.886+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7ff1a804d450 con 0x7ff1c015d490 2023-11-24T08:10:03.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.887+0000 7ff1b7fff640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 0x7ff1a006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:03.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.887+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7ff1a80aa890 con 0x7ff1c015d490 2023-11-24T08:10:03.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.887+0000 7ff1bf7fe640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 0x7ff1a006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:03.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.888+0000 7ff1bf7fe640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 0x7ff1a006db70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000abb0 tx=0x7ff1b0009250 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:03.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:03.891+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1a80771c0 con 0x7ff1c015d490 2023-11-24T08:10:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.094+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1c004e840 con 0x7ff1c015d490 2023-11-24T08:10:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.094+0000 7ff1b7fff640 1 -- 172.21.15.107:0/2700902160 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v65) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff1a807a7e0 con 0x7ff1c015d490 2023-11-24T08:10:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:10:04.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 msgr2=0x7ff1a006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 0x7ff1a006db70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000abb0 tx=0x7ff1b0009250 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 msgr2=0x7ff1c00cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8000c00 tx=0x7ff1a80046c0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 shutdown_connections 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff1a006b6b0 0x7ff1a006db70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c00cc610 0x7ff1c01e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c015d490 0x7ff1c00cc0d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 --2- 172.21.15.107:0/2700902160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c00ce270 0x7ff1c0151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 >> 172.21.15.107:0/2700902160 conn(0x7ff1c0061340 msgr2=0x7ff1c00cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.097+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 shutdown_connections 2023-11-24T08:10:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.098+0000 7ff1c48e1640 1 -- 172.21.15.107:0/2700902160 wait complete. 2023-11-24T08:10:04.118 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:10:04.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:04.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:10:04.253 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:10:04.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:04.253 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:10:04.311 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:04.311 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:04.355 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:04.355 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:04.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:04.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.650+0000 7fcc1ed8a640 1 Processor -- start 2023-11-24T08:10:04.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.650+0000 7fcc1ed8a640 1 -- start start 2023-11-24T08:10:04.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.651+0000 7fcc1ed8a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc1815d4e0 0x7fcc180d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.651+0000 7fcc1ed8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 0x7fcc180b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.651+0000 7fcc1ed8a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc180ce990 con 0x7fcc180d38e0 2023-11-24T08:10:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.651+0000 7fcc1ed8a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180d4110 con 0x7fcc1815d4e0 2023-11-24T08:10:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.651+0000 7fcc1ed8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180cec70 con 0x7fcc180d0240 2023-11-24T08:10:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1d587640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcc180d38e0 0x7fcc180cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33974/0 (socket says 172.21.15.107:33974) 2023-11-24T08:10:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1d587640 1 -- 172.21.15.107:0/1146350727 learned_addr learned my addr 172.21.15.107:0/1146350727 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1e589640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 0x7fcc180b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/1146350727 >> v2:172.21.15.120:3300/0 conn(0x7fcc1815d4e0 0x7fcc180d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342119161 0 0) 0x7fcc180ce990 con 0x7fcc180d38e0 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.652+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc04003520 con 0x7fcc180d38e0 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973502772 0 0) 0x7fcc04003520 con 0x7fcc180d38e0 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 msgr2=0x7fcc180b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 0x7fcc180b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 >> v2:172.21.15.120:3300/0 conn(0x7fcc1815d4e0 msgr2=0x7fcc180d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 --2- 172.21.15.107:0/1146350727 >> v2:172.21.15.120:3300/0 conn(0x7fcc1815d4e0 0x7fcc180d1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc180cef50 con 0x7fcc180d38e0 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcc0c001690 con 0x7fcc180d38e0 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 >> v1:172.21.15.107:6789/0 conn(0x7fcc180d38e0 legacy=0x7fcc180cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc04003970 0x7fcc04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 0x7fcc040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc04004d00 0x7fcc04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180d4110 con 0x7fcc04003970 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180cec70 con 0x7fcc04004d00 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0400b980 con 0x7fcc180d0240 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc04003970 0x7fcc04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 -- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 msgr2=0x7fcc040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc180d0240 0x7fcc040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 -- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc04004d00 msgr2=0x7fcc04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc04004d00 0x7fcc04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.653+0000 7fcc1dd88640 1 -- 172.21.15.107:0/1146350727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04003520 con 0x7fcc04003970 2023-11-24T08:10:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.654+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc04003970 0x7fcc04003d90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009ac0 tx=0x7fcc040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a29f4c8e898bcd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1402c6d0 con 0x7fcc04003970 2023-11-24T08:10:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1402c830 con 0x7fcc04003970 2023-11-24T08:10:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1cd86640 1 -- 172.21.15.107:0/1146350727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1402cbc0 con 0x7fcc04003970 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/1146350727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc04003970 msgr2=0x7fcc04003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/1146350727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc04003970 0x7fcc04003d90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009ac0 tx=0x7fcc040057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1e589640 1 -- 172.21.15.107:0/1146350727 reap_dead start 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/1146350727 shutdown_connections 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/1146350727 >> 172.21.15.107:0/1146350727 conn(0x7fcc1809c510 msgr2=0x7fcc180d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/1146350727 shutdown_connections 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.655+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/1146350727 wait complete. 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.657+0000 7fcc1ed8a640 1 Processor -- start 2023-11-24T08:10:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.657+0000 7fcc1ed8a640 1 -- start start 2023-11-24T08:10:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc04003970 0x7fcc18152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180b3240 0x7fcc180b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180b3c30 con 0x7fcc180b84a0 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180cb220 con 0x7fcc180b3240 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.658+0000 7fcc1ed8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180b7e40 con 0x7fcc04003970 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38300/0 (socket says 172.21.15.107:38300) 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 -- 172.21.15.107:0/287995434 learned_addr learned my addr 172.21.15.107:0/287995434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc04003970 0x7fcc18152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 -- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc04003970 msgr2=0x7fcc18152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc04003970 0x7fcc18152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 -- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180b3240 msgr2=0x7fcc180b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1e589640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180b3240 0x7fcc180b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180b3240 0x7fcc180b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 -- 172.21.15.107:0/287995434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18160740 con 0x7fcc180b84a0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.659+0000 7fcc1d587640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00c7a0 tx=0x7fcc0c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.660+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c007560 con 0x7fcc180b84a0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.660+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc180b7320 con 0x7fcc180b84a0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.660+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c0047e0 con 0x7fcc180b84a0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.660+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c004b50 con 0x7fcc180b84a0 2023-11-24T08:10:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.660+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc180b7880 con 0x7fcc180b84a0 2023-11-24T08:10:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.661+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1804e840 con 0x7fcc180b84a0 2023-11-24T08:10:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.662+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fcc0c00a640 con 0x7fcc180b84a0 2023-11-24T08:10:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.663+0000 7fcc0a7fc640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 0x7fcbfc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.663+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fcc0c0879e0 con 0x7fcc180b84a0 2023-11-24T08:10:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.663+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 0x7fcbfc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.664+0000 7fcc1dd88640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 0x7fcbfc06db70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036000 tx=0x7fcc14030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:04.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.667+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc0c054290 con 0x7fcc180b84a0 2023-11-24T08:10:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.858+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc1806c370 con 0x7fcc180b84a0 2023-11-24T08:10:04.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.861+0000 7fcc0a7fc640 1 -- 172.21.15.107:0/287995434 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fcc0c0578b0 con 0x7fcc180b84a0 2023-11-24T08:10:04.864 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:04.864 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:59.607764+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:04.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 msgr2=0x7fcbfc06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 0x7fcbfc06db70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036000 tx=0x7fcc14030000 comp rx=0 tx=0).stop 2023-11-24T08:10:04.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 msgr2=0x7fcc180b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:04.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00c7a0 tx=0x7fcc0c00cc60 comp rx=0 tx=0).stop 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 shutdown_connections 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06b6b0 0x7fcbfc06db70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180b3240 0x7fcc180b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180b84a0 0x7fcc180b6de0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 --2- 172.21.15.107:0/287995434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc04003970 0x7fcc18152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 >> 172.21.15.107:0/287995434 conn(0x7fcc1809c510 msgr2=0x7fcc1806c890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 shutdown_connections 2023-11-24T08:10:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:04.864+0000 7fcc1ed8a640 1 -- 172.21.15.107:0/287995434 wait complete. 2023-11-24T08:10:04.886 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:09:02.329269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:09:03.506902+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:10:04.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:10:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.102+0000 7f3bf7051640 1 Processor -- start 2023-11-24T08:10:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.102+0000 7f3bf7051640 1 -- start start 2023-11-24T08:10:05.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf7051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf00d3b50 0x7f3bf00d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf7051640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bf00cde70 0x7f3bf00ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf7051640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf006cf60 con 0x7f3bf0160ce0 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf7051640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf00d42f0 con 0x7f3bf00cde70 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf7051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00b2ca0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf55ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bf00cde70 0x7f3bf00ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.104+0000 7f3bf55ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bf00cde70 0x7f3bf00ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38304/0 (socket says 172.21.15.107:38304) 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3bf55ee640 1 -- 172.21.15.107:0/2643564843 learned_addr learned my addr 172.21.15.107:0/2643564843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3beffff640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf00d3b50 0x7f3bf00d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3bf55ee640 1 -- 172.21.15.107:0/2643564843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf00d3b50 msgr2=0x7f3bf00d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3bf55ee640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf00d3b50 0x7f3bf00d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3bf55ee640 1 -- 172.21.15.107:0/2643564843 >> v1:172.21.15.107:6789/0 conn(0x7f3bf0160ce0 legacy=0x7f3bf00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.105+0000 7f3bf55ee640 1 -- 172.21.15.107:0/2643564843 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf006c550 con 0x7f3bf00cde70 2023-11-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.106+0000 7f3bf55ee640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00cde70 0x7f3bf00ce2e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00d4430 tx=0x7f3be802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1696cdc632e467b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.106+0000 7f3bef7fe640 1 -- 172.21.15.107:0/2643564843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be802cae0 con 0x7f3bf00cde70 2023-11-24T08:10:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bef7fe640 1 -- 172.21.15.107:0/2643564843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be802cc40 con 0x7f3bf00cde70 2023-11-24T08:10:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bef7fe640 1 -- 172.21.15.107:0/2643564843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be80343f0 con 0x7f3bf00cde70 2023-11-24T08:10:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 -- 172.21.15.107:0/2643564843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00cde70 msgr2=0x7f3bf00ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00cde70 0x7f3bf00ce2e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00d4430 tx=0x7f3be802bf40 comp rx=0 tx=0).stop 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 -- 172.21.15.107:0/2643564843 shutdown_connections 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00cde70 0x7f3bf00ce2e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 --2- 172.21.15.107:0/2643564843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf00d3b50 0x7f3bf00d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 -- 172.21.15.107:0/2643564843 >> 172.21.15.107:0/2643564843 conn(0x7f3bf009c6f0 msgr2=0x7f3bf00d3440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 -- 172.21.15.107:0/2643564843 shutdown_connections 2023-11-24T08:10:05.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.107+0000 7f3bf7051640 1 -- 172.21.15.107:0/2643564843 wait complete. 2023-11-24T08:10:05.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.109+0000 7f3bf7051640 1 Processor -- start 2023-11-24T08:10:05.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.109+0000 7f3bf7051640 1 -- start start 2023-11-24T08:10:05.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bf00cde70 0x7f3bf00cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf0160ce0 0x7f3bf01f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf01f7790 con 0x7f3bf00d3b50 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf01f31d0 con 0x7f3bf00cde70 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.110+0000 7f3bf7051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf01f41a0 con 0x7f3bf0160ce0 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38308/0 (socket says 172.21.15.107:38308) 2023-11-24T08:10:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 -- 172.21.15.107:0/3065032086 learned_addr learned my addr 172.21.15.107:0/3065032086 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 -- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf0160ce0 msgr2=0x7f3bf01f07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3bf55ee640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf0160ce0 0x7f3bf01f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3bf4ded640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bf00cde70 0x7f3bf00cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf0160ce0 0x7f3bf01f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 -- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bf00cde70 msgr2=0x7f3bf00cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bf00cde70 0x7f3bf00cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 -- 172.21.15.107:0/3065032086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be8009b30 con 0x7f3bf00d3b50 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.111+0000 7f3beffff640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00e9e0 tx=0x7f3bdc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.112+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc009670 con 0x7f3bf00d3b50 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.112+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc00c9d0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.112+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc00cd80 con 0x7f3bf00d3b50 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.112+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf01f33e0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.112+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf01f39a0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.113+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf00cb670 con 0x7f3bf00d3b50 2023-11-24T08:10:05.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.114+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f3bdc013470 con 0x7f3bf00d3b50 2023-11-24T08:10:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.115+0000 7f3bed7fa640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 0x7f3bd406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.115+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f3bdc088270 con 0x7f3bf00d3b50 2023-11-24T08:10:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.115+0000 7f3bf4ded640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 0x7f3bd406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.116+0000 7f3bf4ded640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 0x7f3bd406db70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00d2550 tx=0x7f3be4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.119+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bdc054ba0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.312+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bf004e840 con 0x7f3bf00d3b50 2023-11-24T08:10:05.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.313+0000 7f3bed7fa640 1 -- 172.21.15.107:0/3065032086 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bdc0170a0 con 0x7f3bf00d3b50 2023-11-24T08:10:05.315 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 msgr2=0x7f3bd406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 0x7f3bd406db70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00d2550 tx=0x7f3be4009040 comp rx=0 tx=0).stop 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 msgr2=0x7f3bf01f02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00e9e0 tx=0x7f3bdc00eea0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 shutdown_connections 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3bd406b6b0 0x7f3bd406db70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bf0160ce0 0x7f3bf01f07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00d3b50 0x7f3bf01f02b0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 --2- 172.21.15.107:0/3065032086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bf00cde70 0x7f3bf00cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 >> 172.21.15.107:0/3065032086 conn(0x7f3bf009c6f0 msgr2=0x7f3bf00d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 shutdown_connections 2023-11-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.316+0000 7f3bf7051640 1 -- 172.21.15.107:0/3065032086 wait complete. 2023-11-24T08:10:05.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:10:05.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.554+0000 7fce37528640 1 Processor -- start 2023-11-24T08:10:05.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.554+0000 7fce37528640 1 -- start start 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce37528640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fce3015b690 0x7fce300ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce37528640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 0x7fce3006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce37528640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300cf380 con 0x7fce300cc8f0 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce37528640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce300cf660 con 0x7fce3015b690 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce37528640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce300cf940 con 0x7fce300cede0 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce36526640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce300cc8f0 0x7fce300cccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33990/0 (socket says 172.21.15.107:33990) 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.556+0000 7fce36526640 1 -- 172.21.15.107:0/1591399013 learned_addr learned my addr 172.21.15.107:0/1591399013 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35d25640 1 --2- 172.21.15.107:0/1591399013 >> v2:172.21.15.120:3300/0 conn(0x7fce3015b690 0x7fce300ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963467942 0 0) 0x7fce300cf380 con 0x7fce300cc8f0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1c003520 con 0x7fce300cc8f0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce36d27640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 0x7fce3006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637864173 0 0) 0x7fce1c003520 con 0x7fce300cc8f0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 msgr2=0x7fce3006eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 0x7fce3006eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 >> v2:172.21.15.120:3300/0 conn(0x7fce3015b690 msgr2=0x7fce300ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 --2- 172.21.15.107:0/1591399013 >> v2:172.21.15.120:3300/0 conn(0x7fce3015b690 0x7fce300ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300cfc20 con 0x7fce300cc8f0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fce2c001670 con 0x7fce300cc8f0 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 >> v1:172.21.15.107:6789/0 conn(0x7fce300cc8f0 legacy=0x7fce300cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 0x7fce1c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c003ef0 0x7fce1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c004d00 0x7fce1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300cf660 con 0x7fce1c004d00 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300cf940 con 0x7fce1c003ef0 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.557+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c00ba80 con 0x7fce300cede0 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c004d00 0x7fce1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 -- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 msgr2=0x7fce1c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300cede0 0x7fce1c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce35d25640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c003ef0 0x7fce1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 -- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c003ef0 msgr2=0x7fce1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c003ef0 0x7fce1c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.558+0000 7fce36d27640 1 -- 172.21.15.107:0/1591399013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c003520 con 0x7fce1c004d00 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.559+0000 7fce36d27640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c004d00 0x7fce1c005180 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fce28007390 tx=0x7fce1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc088aeab9923d97 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.559+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2802ccf0 con 0x7fce1c004d00 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.559+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2802ce50 con 0x7fce1c004d00 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.559+0000 7fce35524640 1 -- 172.21.15.107:0/1591399013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28034660 con 0x7fce1c004d00 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 -- 172.21.15.107:0/1591399013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c004d00 msgr2=0x7fce1c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 --2- 172.21.15.107:0/1591399013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c004d00 0x7fce1c005180 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fce28007390 tx=0x7fce1c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce36d27640 1 -- 172.21.15.107:0/1591399013 reap_dead start 2023-11-24T08:10:05.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 -- 172.21.15.107:0/1591399013 shutdown_connections 2023-11-24T08:10:05.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 -- 172.21.15.107:0/1591399013 >> 172.21.15.107:0/1591399013 conn(0x7fce300b1bd0 msgr2=0x7fce300b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:05.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 -- 172.21.15.107:0/1591399013 shutdown_connections 2023-11-24T08:10:05.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.560+0000 7fce37528640 1 -- 172.21.15.107:0/1591399013 wait complete. 2023-11-24T08:10:05.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.562+0000 7fce37528640 1 Processor -- start 2023-11-24T08:10:05.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.562+0000 7fce37528640 1 -- start start 2023-11-24T08:10:05.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c004d00 0x7fce3015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300b48d0 0x7fce300b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3015e450 con 0x7fce300b4d50 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3015e990 con 0x7fce1c004d00 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce37528640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3015e6b0 con 0x7fce300b48d0 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce36d27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce36d27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38322/0 (socket says 172.21.15.107:38322) 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.563+0000 7fce36d27640 1 -- 172.21.15.107:0/2964060786 learned_addr learned my addr 172.21.15.107:0/2964060786 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36526640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c004d00 0x7fce3015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce35d25640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300b48d0 0x7fce300b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 -- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300b48d0 msgr2=0x7fce300b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300b48d0 0x7fce300b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 -- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c004d00 msgr2=0x7fce3015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c004d00 0x7fce3015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 -- 172.21.15.107:0/2964060786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300b3070 con 0x7fce300b4d50 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce36d27640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce28007390 tx=0x7fce28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28045070 con 0x7fce300b4d50 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce28034e00 con 0x7fce300b4d50 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.564+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28003bd0 con 0x7fce300b4d50 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.565+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300b3350 con 0x7fce300b4d50 2023-11-24T08:10:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.565+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce300b4090 con 0x7fce300b4d50 2023-11-24T08:10:05.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.566+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce300b3870 con 0x7fce300b4d50 2023-11-24T08:10:05.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.567+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fce2803b040 con 0x7fce300b4d50 2023-11-24T08:10:05.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.568+0000 7fce26ffd640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 0x7fce1406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:05.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.568+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fce280a9410 con 0x7fce300b4d50 2023-11-24T08:10:05.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.568+0000 7fce36526640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 0x7fce1406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:05.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.569+0000 7fce36526640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 0x7fce1406db70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fce2c000ff0 tx=0x7fce2c001600 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:05.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.573+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce300b3870 con 0x7fce300b4d50 2023-11-24T08:10:05.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.765+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce300b3870 con 0x7fce300b4d50 2023-11-24T08:10:05.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.765+0000 7fce26ffd640 1 -- 172.21.15.107:0/2964060786 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce300b3870 con 0x7fce300b4d50 2023-11-24T08:10:05.768 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:10:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.768+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 msgr2=0x7fce1406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.768+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 0x7fce1406db70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fce2c000ff0 tx=0x7fce2c001600 comp rx=0 tx=0).stop 2023-11-24T08:10:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.768+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 msgr2=0x7fce30152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:05.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.768+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce28007390 tx=0x7fce28004ab0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 shutdown_connections 2023-11-24T08:10:05.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce1406b6b0 0x7fce1406db70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce300b4d50 0x7fce30152d30 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce300b48d0 0x7fce300b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 --2- 172.21.15.107:0/2964060786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c004d00 0x7fce3015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:05.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 >> 172.21.15.107:0/2964060786 conn(0x7fce300b1bd0 msgr2=0x7fce3006ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:05.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 shutdown_connections 2023-11-24T08:10:05.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:05.769+0000 7fce37528640 1 -- 172.21.15.107:0/2964060786 wait complete. 2023-11-24T08:10:05.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:10:06.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.002+0000 7f2f38a48640 1 Processor -- start 2023-11-24T08:10:06.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.002+0000 7f2f38a48640 1 -- start start 2023-11-24T08:10:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.003+0000 7f2f38a48640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f3415b440 0x7f2f340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.003+0000 7f2f38a48640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f340cbf20 0x7f2f340c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.003+0000 7f2f38a48640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3415eea0 con 0x7f2f340c57e0 2023-11-24T08:10:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.003+0000 7f2f38a48640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f340cc3d0 con 0x7f2f340cbf20 2023-11-24T08:10:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.003+0000 7f2f38a48640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3415b820 con 0x7f2f3415b440 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f340cbf20 0x7f2f340c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f340cbf20 0x7f2f340c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38324/0 (socket says 172.21.15.107:38324) 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 -- 172.21.15.107:0/1560543859 learned_addr learned my addr 172.21.15.107:0/1560543859 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f3259c640 1 --2- 172.21.15.107:0/1560543859 >> v2:172.21.15.120:3300/0 conn(0x7f2f3415b440 0x7f2f340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 -- 172.21.15.107:0/1560543859 >> v2:172.21.15.120:3300/0 conn(0x7f2f3415b440 msgr2=0x7f2f340cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 --2- 172.21.15.107:0/1560543859 >> v2:172.21.15.120:3300/0 conn(0x7f2f3415b440 0x7f2f340cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 -- 172.21.15.107:0/1560543859 >> v1:172.21.15.120:6789/0 conn(0x7f2f340c57e0 legacy=0x7f2f340c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:06.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.004+0000 7f2f31d9b640 1 -- 172.21.15.107:0/1560543859 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3406d600 con 0x7f2f340cbf20 2023-11-24T08:10:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.005+0000 7f2f31d9b640 1 --2- 172.21.15.107:0/1560543859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f340cbf20 0x7f2f340c52a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2f3406e770 tx=0x7f2f2802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a782b75e4fbb7fa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f3159a640 1 -- 172.21.15.107:0/1560543859 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f280317e0 con 0x7f2f340cbf20 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f3159a640 1 -- 172.21.15.107:0/1560543859 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f28031940 con 0x7f2f340cbf20 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f3159a640 1 -- 172.21.15.107:0/1560543859 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28031cd0 con 0x7f2f340cbf20 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 -- 172.21.15.107:0/1560543859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f340cbf20 msgr2=0x7f2f340c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 --2- 172.21.15.107:0/1560543859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f340cbf20 0x7f2f340c52a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2f3406e770 tx=0x7f2f2802b880 comp rx=0 tx=0).stop 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 -- 172.21.15.107:0/1560543859 shutdown_connections 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 --2- 172.21.15.107:0/1560543859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f340cbf20 0x7f2f340c52a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 --2- 172.21.15.107:0/1560543859 >> v2:172.21.15.120:3300/0 conn(0x7f2f3415b440 0x7f2f340cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 -- 172.21.15.107:0/1560543859 >> 172.21.15.107:0/1560543859 conn(0x7f2f340b0c50 msgr2=0x7f2f340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.006+0000 7f2f38a48640 1 -- 172.21.15.107:0/1560543859 shutdown_connections 2023-11-24T08:10:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.007+0000 7f2f38a48640 1 -- 172.21.15.107:0/1560543859 wait complete. 2023-11-24T08:10:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.008+0000 7f2f38a48640 1 Processor -- start 2023-11-24T08:10:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.008+0000 7f2f38a48640 1 -- start start 2023-11-24T08:10:06.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f340c57e0 0x7f2f34153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f340cbf20 0x7f2f340c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34155f60 con 0x7f2f3415b440 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34156240 con 0x7f2f340cbf20 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.009+0000 7f2f38a48640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34156520 con 0x7f2f340c57e0 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38340/0 (socket says 172.21.15.107:38340) 2023-11-24T08:10:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 -- 172.21.15.107:0/4011423995 learned_addr learned my addr 172.21.15.107:0/4011423995 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f3259c640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f340c57e0 0x7f2f34153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f31d9b640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f340cbf20 0x7f2f340c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 -- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f340c57e0 msgr2=0x7f2f34153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f340c57e0 0x7f2f34153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 -- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f340cbf20 msgr2=0x7f2f340c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f340cbf20 0x7f2f340c44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 -- 172.21.15.107:0/4011423995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280094d0 con 0x7f2f3415b440 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.010+0000 7f2f32d9d640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00c780 tx=0x7f2f2c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.011+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c004b60 con 0x7f2f3415b440 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.011+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c00ce90 con 0x7f2f3415b440 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.011+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00a850 con 0x7f2f3415b440 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.011+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f34156800 con 0x7f2f3415b440 2023-11-24T08:10:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.011+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f340b42e0 con 0x7f2f3415b440 2023-11-24T08:10:06.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.012+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f34156d20 con 0x7f2f3415b440 2023-11-24T08:10:06.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.013+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f2f2c00a9b0 con 0x7f2f3415b440 2023-11-24T08:10:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.014+0000 7f2f22ffd640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 0x7f2f1406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.014+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f2f2c0882f0 con 0x7f2f3415b440 2023-11-24T08:10:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.014+0000 7f2f3259c640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 0x7f2f1406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.015+0000 7f2f3259c640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 0x7f2f1406db70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2f24006fd0 tx=0x7f2f24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.018+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f34156d20 con 0x7f2f3415b440 2023-11-24T08:10:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.208+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f34156d20 con 0x7f2f3415b440 2023-11-24T08:10:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.209+0000 7f2f22ffd640 1 -- 172.21.15.107:0/4011423995 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f34156d20 con 0x7f2f3415b440 2023-11-24T08:10:06.211 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:10:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 msgr2=0x7f2f1406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 0x7f2f1406db70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2f24006fd0 tx=0x7f2f24008040 comp rx=0 tx=0).stop 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 msgr2=0x7f2f340c49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00c780 tx=0x7f2f2c00cc40 comp rx=0 tx=0).stop 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 shutdown_connections 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f1406b6b0 0x7f2f1406db70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3415b440 0x7f2f340c49e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f340cbf20 0x7f2f340c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 --2- 172.21.15.107:0/4011423995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f340c57e0 0x7f2f34153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.211+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 >> 172.21.15.107:0/4011423995 conn(0x7f2f340b0c50 msgr2=0x7f2f34070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:06.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.212+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 shutdown_connections 2023-11-24T08:10:06.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.212+0000 7f2f38a48640 1 -- 172.21.15.107:0/4011423995 wait complete. 2023-11-24T08:10:06.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:10:06.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.443+0000 7f458670f640 1 Processor -- start 2023-11-24T08:10:06.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.443+0000 7f458670f640 1 -- start start 2023-11-24T08:10:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.445+0000 7f458670f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.445+0000 7f458670f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45800c49f0 0x7f45800705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.445+0000 7f458670f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458015ec90 con 0x7f4580072160 2023-11-24T08:10:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.445+0000 7f458670f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800cd530 con 0x7f45800c49f0 2023-11-24T08:10:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.445+0000 7f458670f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c4ed0 con 0x7f458006de10 2023-11-24T08:10:06.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.446+0000 7f4584cac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45800c49f0 0x7f45800705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.446+0000 7f4577fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.446+0000 7f4584cac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45800c49f0 0x7f45800705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:38056/0 (socket says 172.21.15.107:38056) 2023-11-24T08:10:06.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.446+0000 7f4577fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:39532/0 (socket says 172.21.15.107:39532) 2023-11-24T08:10:06.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.446+0000 7f4584cac640 1 -- 172.21.15.107:0/1612314915 learned_addr learned my addr 172.21.15.107:0/1612314915 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.447+0000 7f4584cac640 1 -- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 msgr2=0x7f458015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.447+0000 7f4584cac640 1 --2- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.447+0000 7f4584cac640 1 -- 172.21.15.107:0/1612314915 >> v1:172.21.15.120:6789/0 conn(0x7f4580072160 legacy=0x7f458006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.447+0000 7f4584cac640 1 -- 172.21.15.107:0/1612314915 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45800c51b0 con 0x7f45800c49f0 2023-11-24T08:10:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.448+0000 7f4584cac640 1 --2- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f45800705c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4578009e50 tx=0x7f457802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2c29736dfa3f184 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.449+0000 7f457f7fe640 1 -- 172.21.15.107:0/1612314915 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457802caf0 con 0x7f45800c49f0 2023-11-24T08:10:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.449+0000 7f457f7fe640 1 -- 172.21.15.107:0/1612314915 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457802cc50 con 0x7f45800c49f0 2023-11-24T08:10:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.449+0000 7f457f7fe640 1 -- 172.21.15.107:0/1612314915 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45780343f0 con 0x7f45800c49f0 2023-11-24T08:10:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 -- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 msgr2=0x7f45800705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 --2- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f45800705c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4578009e50 tx=0x7f457802bf50 comp rx=0 tx=0).stop 2023-11-24T08:10:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 -- 172.21.15.107:0/1612314915 shutdown_connections 2023-11-24T08:10:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 --2- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f45800705c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 --2- 172.21.15.107:0/1612314915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 -- 172.21.15.107:0/1612314915 >> 172.21.15.107:0/1612314915 conn(0x7f45800b0c50 msgr2=0x7f45800b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 -- 172.21.15.107:0/1612314915 shutdown_connections 2023-11-24T08:10:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.450+0000 7f458670f640 1 -- 172.21.15.107:0/1612314915 wait complete. 2023-11-24T08:10:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.452+0000 7f458670f640 1 Processor -- start 2023-11-24T08:10:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.452+0000 7f458670f640 1 -- start start 2023-11-24T08:10:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f4580159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800b4850 con 0x7f4580072160 2023-11-24T08:10:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c82a0 con 0x7f45800c49f0 2023-11-24T08:10:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f458670f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c8580 con 0x7f458006de10 2023-11-24T08:10:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f4577fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f4577fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38344/0 (socket says 172.21.15.107:38344) 2023-11-24T08:10:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.453+0000 7f4577fff640 1 -- 172.21.15.107:0/3257247520 learned_addr learned my addr 172.21.15.107:0/3257247520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4584cac640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f4580159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f457ffff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 -- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 msgr2=0x7f458015cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 -- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 msgr2=0x7f4580159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f4580159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 -- 172.21.15.107:0/3257247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4578009b30 con 0x7f4580072160 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4584cac640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f4580159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.454+0000 7f4577fff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f456800e9b0 tx=0x7f456800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.455+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568009670 con 0x7f4580072160 2023-11-24T08:10:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.455+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456800c9a0 con 0x7f4580072160 2023-11-24T08:10:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.455+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45800c8860 con 0x7f4580072160 2023-11-24T08:10:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.455+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456800cd50 con 0x7f4580072160 2023-11-24T08:10:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.455+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45800b4d40 con 0x7f4580072160 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.456+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458004e8b0 con 0x7f4580072160 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.457+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f4568013470 con 0x7f4580072160 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.458+0000 7f457d7fa640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 0x7f456006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.458+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4568088350 con 0x7f4580072160 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.458+0000 7f457ffff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 0x7f456006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.458+0000 7f457ffff640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 0x7f456006db70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f45700015a0 tx=0x7f4570009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:06.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.462+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4568054c80 con 0x7f4580072160 2023-11-24T08:10:06.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.664+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f458009cee0 con 0x7f4580072160 2023-11-24T08:10:06.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.665+0000 7f457d7fa640 1 -- 172.21.15.107:0/3257247520 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v65) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4568017080 con 0x7f4580072160 2023-11-24T08:10:06.667 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 msgr2=0x7f456006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 0x7f456006db70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f45700015a0 tx=0x7f4570009040 comp rx=0 tx=0).stop 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 msgr2=0x7f458006c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f456800e9b0 tx=0x7f456800ee70 comp rx=0 tx=0).stop 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 shutdown_connections 2023-11-24T08:10:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456006b6b0 0x7f456006db70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45800c49f0 0x7f4580159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4580072160 0x7f458006c860 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 --2- 172.21.15.107:0/3257247520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458006de10 0x7f458015cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 >> 172.21.15.107:0/3257247520 conn(0x7f45800b0c50 msgr2=0x7f45800b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 shutdown_connections 2023-11-24T08:10:06.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:06.668+0000 7f458670f640 1 -- 172.21.15.107:0/3257247520 wait complete. 2023-11-24T08:10:06.688 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:10:06.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:06.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:10:06.799 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:10:06.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:06.800 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:10:06.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:06.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:06.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:10:06.910 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:06.910 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:06.969 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:10:06.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:07.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.190+0000 7f50167d5640 1 Processor -- start 2023-11-24T08:10:07.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.190+0000 7f50167d5640 1 -- start start 2023-11-24T08:10:07.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 0x7f501006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50100cf450 0x7f50100cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50100cdfb0 0x7f50100ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5010156010 con 0x7f50100cf450 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501006ce30 con 0x7f50100cdfb0 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.191+0000 7f50167d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50100cfe10 con 0x7f501015c7f0 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50100cf450 0x7f50100cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50100cf450 0x7f50100cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38350/0 (socket says 172.21.15.107:38350) 2023-11-24T08:10:07.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 -- 172.21.15.107:0/366516594 learned_addr learned my addr 172.21.15.107:0/366516594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5014d72640 1 --2- 172.21.15.107:0/366516594 >> v2:172.21.15.120:3300/0 conn(0x7f50100cdfb0 0x7f50100ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f500ffff640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 0x7f501006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 -- 172.21.15.107:0/366516594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 msgr2=0x7f501006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 0x7f501006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 -- 172.21.15.107:0/366516594 >> v2:172.21.15.120:3300/0 conn(0x7f50100cdfb0 msgr2=0x7f50100ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 --2- 172.21.15.107:0/366516594 >> v2:172.21.15.120:3300/0 conn(0x7f50100cdfb0 0x7f50100ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f5007fff640 1 -- 172.21.15.107:0/366516594 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100cece0 con 0x7f50100cf450 2023-11-24T08:10:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.192+0000 7f500ffff640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 0x7f501006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.194+0000 7f5007fff640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cf450 0x7f50100cf8d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f50100aebc0 tx=0x7f4ff802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9237eeb363533bea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.194+0000 7f500f7fe640 1 -- 172.21.15.107:0/366516594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8031880 con 0x7f50100cf450 2023-11-24T08:10:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.194+0000 7f500f7fe640 1 -- 172.21.15.107:0/366516594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff80319e0 con 0x7f50100cf450 2023-11-24T08:10:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.194+0000 7f500f7fe640 1 -- 172.21.15.107:0/366516594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8031d70 con 0x7f50100cf450 2023-11-24T08:10:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 -- 172.21.15.107:0/366516594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cf450 msgr2=0x7f50100cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cf450 0x7f50100cf8d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f50100aebc0 tx=0x7f4ff802b920 comp rx=0 tx=0).stop 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 -- 172.21.15.107:0/366516594 shutdown_connections 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 --2- 172.21.15.107:0/366516594 >> v2:172.21.15.120:3300/0 conn(0x7f50100cdfb0 0x7f50100ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cf450 0x7f50100cf8d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 --2- 172.21.15.107:0/366516594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f501015c7f0 0x7f501006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 -- 172.21.15.107:0/366516594 >> 172.21.15.107:0/366516594 conn(0x7f501006a920 msgr2=0x7f501006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 -- 172.21.15.107:0/366516594 shutdown_connections 2023-11-24T08:10:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.195+0000 7f50167d5640 1 -- 172.21.15.107:0/366516594 wait complete. 2023-11-24T08:10:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.197+0000 7f50167d5640 1 Processor -- start 2023-11-24T08:10:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.197+0000 7f50167d5640 1 -- start start 2023-11-24T08:10:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50100cf450 0x7f501009a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f501015c7f0 0x7f501014c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501009b860 con 0x7f50100cdfb0 2023-11-24T08:10:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501009abe0 con 0x7f501015c7f0 2023-11-24T08:10:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f50167d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501009aec0 con 0x7f50100cf450 2023-11-24T08:10:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f500ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.198+0000 7f500ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38362/0 (socket says 172.21.15.107:38362) 2023-11-24T08:10:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 -- 172.21.15.107:0/689706411 learned_addr learned my addr 172.21.15.107:0/689706411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 -- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50100cf450 msgr2=0x7f501009a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f5007fff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50100cf450 0x7f501009a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50100cf450 0x7f501009a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f5014d72640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f501015c7f0 0x7f501014c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 -- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f501015c7f0 msgr2=0x7f501014c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f501015c7f0 0x7f501014c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 -- 172.21.15.107:0/689706411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff80094d0 con 0x7f50100cdfb0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500ffff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f500000c7f0 tx=0x7f500000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.199+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000007820 con 0x7f50100cdfb0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.200+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000007980 con 0x7f50100cdfb0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.200+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000007cf0 con 0x7f50100cdfb0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.200+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501009b1a0 con 0x7f50100cdfb0 2023-11-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.200+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50100b3cd0 con 0x7f50100cdfb0 2023-11-24T08:10:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.201+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5010067470 con 0x7f50100cdfb0 2023-11-24T08:10:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.202+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f50000113f0 con 0x7f50100cdfb0 2023-11-24T08:10:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.203+0000 7f500d7fa640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 0x7f4ff006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.203+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5000088340 con 0x7f50100cdfb0 2023-11-24T08:10:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.203+0000 7f5007fff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 0x7f4ff006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.204+0000 7f5007fff640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 0x7f4ff006db70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8036000 tx=0x7f4ff803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.207+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5000054cd0 con 0x7f50100cdfb0 2023-11-24T08:10:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.435+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50100b51b0 con 0x7f50100cdfb0 2023-11-24T08:10:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.436+0000 7f500d7fa640 1 -- 172.21.15.107:0/689706411 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f500001f090 con 0x7f50100cdfb0 2023-11-24T08:10:07.438 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:07.438 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:10:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-11-24T08:10:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 msgr2=0x7f4ff006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 0x7f4ff006db70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8036000 tx=0x7f4ff803c040 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 msgr2=0x7f501015a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f500000c7f0 tx=0x7f500000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 shutdown_connections 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ff006b6b0 0x7f4ff006db70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f501015c7f0 0x7f501014c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50100cf450 0x7f501009a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 --2- 172.21.15.107:0/689706411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50100cdfb0 0x7f501015a7f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.439+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 >> 172.21.15.107:0/689706411 conn(0x7f501006a920 msgr2=0x7f50100aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.440+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 shutdown_connections 2023-11-24T08:10:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.440+0000 7f50167d5640 1 -- 172.21.15.107:0/689706411 wait complete. 2023-11-24T08:10:07.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.672+0000 7f4683661640 1 Processor -- start 2023-11-24T08:10:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.672+0000 7f4683661640 1 -- start start 2023-11-24T08:10:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4683661640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f467c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4683661640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c0cece0 con 0x7f467c0cdfb0 2023-11-24T08:10:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4683661640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c06d040 con 0x7f467c15c7f0 2023-11-24T08:10:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4683661640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c156010 con 0x7f467c0cf450 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4681bfe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f467c0cdfb0 0x7f467c0ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33998/0 (socket says 172.21.15.107:33998) 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4681bfe640 1 -- 172.21.15.107:0/1673846055 learned_addr learned my addr 172.21.15.107:0/1673846055 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.674+0000 7f4680bfc640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f467c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271213069 0 0) 0x7f467c0cece0 con 0x7f467c0cdfb0 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4668003520 con 0x7f467c0cdfb0 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284154873 0 0) 0x7f4668003520 con 0x7f467c0cdfb0 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 msgr2=0x7f467c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f467c0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 >> v1:172.21.15.120:6789/0 conn(0x7f467c15c7f0 legacy=0x7f467c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c06d320 con 0x7f467c0cdfb0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f46740019f0 con 0x7f467c0cdfb0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 >> v1:172.21.15.107:6789/0 conn(0x7f467c0cdfb0 legacy=0x7f467c0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46680039a0 0x7f4668003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f46680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f4668005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c156010 con 0x7f4668004d00 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466800ba90 con 0x7f46680039a0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466800bc50 con 0x7f467c0cf450 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.675+0000 7f4681bfe640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f4668005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f46813fd640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46680039a0 0x7f4668003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4680bfc640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f46680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4681bfe640 1 -- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 msgr2=0x7f46680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:07.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4681bfe640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c0cf450 0x7f46680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4681bfe640 1 -- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46680039a0 msgr2=0x7f4668003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:07.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4681bfe640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46680039a0 0x7f4668003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.676+0000 7f4681bfe640 1 -- 172.21.15.107:0/1673846055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668003520 con 0x7f4668004d00 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.677+0000 7f4681bfe640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f4668005180 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4674002230 tx=0x7f46680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f698f9310fbfea3f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.677+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674034630 con 0x7f4668004d00 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.677+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467402f070 con 0x7f4668004d00 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.677+0000 7f4673fff640 1 -- 172.21.15.107:0/1673846055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46740349f0 con 0x7f4668004d00 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 -- 172.21.15.107:0/1673846055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 msgr2=0x7f4668005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 --2- 172.21.15.107:0/1673846055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f4668005180 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4674002230 tx=0x7f46680057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4681bfe640 1 -- 172.21.15.107:0/1673846055 reap_dead start 2023-11-24T08:10:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 -- 172.21.15.107:0/1673846055 shutdown_connections 2023-11-24T08:10:07.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 -- 172.21.15.107:0/1673846055 >> 172.21.15.107:0/1673846055 conn(0x7f467c06a920 msgr2=0x7f467c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:07.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 -- 172.21.15.107:0/1673846055 shutdown_connections 2023-11-24T08:10:07.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.678+0000 7f4683661640 1 -- 172.21.15.107:0/1673846055 wait complete. 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.680+0000 7f4683661640 1 Processor -- start 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.680+0000 7f4683661640 1 -- start start 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c0ad010 0x7f467c0ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c1f43e0 0x7f467c1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c149f50 con 0x7f4668004d00 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0ad9d0 con 0x7f467c0ad010 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4683661640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c158d50 con 0x7f467c1f43e0 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f46813fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f46813fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38384/0 (socket says 172.21.15.107:38384) 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f46813fd640 1 -- 172.21.15.107:0/771119461 learned_addr learned my addr 172.21.15.107:0/771119461 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.681+0000 7f4680bfc640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c0ad010 0x7f467c0ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 -- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c1f43e0 msgr2=0x7f467c1f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c1f43e0 0x7f467c1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 -- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c0ad010 msgr2=0x7f467c0ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c0ad010 0x7f467c0ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 -- 172.21.15.107:0/771119461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c15d780 con 0x7f4668004d00 2023-11-24T08:10:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f46813fd640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f467800e4d0 tx=0x7f467800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46780052e0 con 0x7f4668004d00 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4678005440 con 0x7f4668004d00 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678003810 con 0x7f4668004d00 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c158140 con 0x7f4668004d00 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.682+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c158700 con 0x7f4668004d00 2023-11-24T08:10:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.683+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c150d70 con 0x7f4668004d00 2023-11-24T08:10:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.684+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f4678003970 con 0x7f4668004d00 2023-11-24T08:10:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.685+0000 7f4671ffb640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 0x7f466006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.685+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f467808a6e0 con 0x7f4668004d00 2023-11-24T08:10:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.685+0000 7f4680bfc640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 0x7f466006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.686+0000 7f4680bfc640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 0x7f466006db70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f467c0ae6e0 tx=0x7f466c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:07.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.689+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4678054720 con 0x7f4668004d00 2023-11-24T08:10:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.884+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467c067470 con 0x7f4668004d00 2023-11-24T08:10:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.886+0000 7f4671ffb640 1 -- 172.21.15.107:0/771119461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16673 (secure 0 0 0) 0x7f4678057d40 con 0x7f4668004d00 2023-11-24T08:10:07.889 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:07.889 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:09:59.607764+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 msgr2=0x7f466006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 0x7f466006db70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f467c0ae6e0 tx=0x7f466c008040 comp rx=0 tx=0).stop 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 msgr2=0x7f467c1545f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f467800e4d0 tx=0x7f467800e990 comp rx=0 tx=0).stop 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 shutdown_connections 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f466006b6b0 0x7f466006db70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c1f43e0 0x7f467c1f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c0ad010 0x7f467c0ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 --2- 172.21.15.107:0/771119461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668004d00 0x7f467c1545f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 >> 172.21.15.107:0/771119461 conn(0x7f467c06a920 msgr2=0x7f467c06da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 shutdown_connections 2023-11-24T08:10:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:07.889+0000 7f4683661640 1 -- 172.21.15.107:0/771119461 wait complete. 2023-11-24T08:10:07.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:09:02.329269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:09:03.506902+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:10:07.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:10:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.128+0000 7f0af4c90640 1 Processor -- start 2023-11-24T08:10:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.128+0000 7f0af4c90640 1 -- start start 2023-11-24T08:10:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.129+0000 7f0af4c90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0af0072160 0x7f0af015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.129+0000 7f0af4c90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af015ec90 con 0x7f0af006de10 2023-11-24T08:10:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.129+0000 7f0af4c90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af00cd530 con 0x7f0af00c49f0 2023-11-24T08:10:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.129+0000 7f0af4c90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0af00c4ed0 con 0x7f0af0072160 2023-11-24T08:10:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0af0072160 0x7f0af015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0af0072160 0x7f0af015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59036/0 (socket says 172.21.15.107:59036) 2023-11-24T08:10:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 -- 172.21.15.107:0/3670446584 learned_addr learned my addr 172.21.15.107:0/3670446584 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 -- 172.21.15.107:0/3670446584 >> v1:172.21.15.120:6789/0 conn(0x7f0af00c49f0 legacy=0x7f0af006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:08.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 -- 172.21.15.107:0/3670446584 >> v1:172.21.15.107:6789/0 conn(0x7f0af006de10 legacy=0x7f0af00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.130+0000 7f0aee59c640 1 -- 172.21.15.107:0/3670446584 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af00c51b0 con 0x7f0af0072160 2023-11-24T08:10:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.131+0000 7f0aee59c640 1 --2- 172.21.15.107:0/3670446584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af0072160 0x7f0af015e750 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009870 tx=0x7f0ae002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4c0d5282cfd04c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.131+0000 7f0aedd9b640 1 -- 172.21.15.107:0/3670446584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0030860 con 0x7f0af0072160 2023-11-24T08:10:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.131+0000 7f0aedd9b640 1 -- 172.21.15.107:0/3670446584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae00309c0 con 0x7f0af0072160 2023-11-24T08:10:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.131+0000 7f0aedd9b640 1 -- 172.21.15.107:0/3670446584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0030d50 con 0x7f0af0072160 2023-11-24T08:10:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 -- 172.21.15.107:0/3670446584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af0072160 msgr2=0x7f0af015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 --2- 172.21.15.107:0/3670446584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af0072160 0x7f0af015e750 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009870 tx=0x7f0ae002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 -- 172.21.15.107:0/3670446584 shutdown_connections 2023-11-24T08:10:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 --2- 172.21.15.107:0/3670446584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af0072160 0x7f0af015e750 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 -- 172.21.15.107:0/3670446584 >> 172.21.15.107:0/3670446584 conn(0x7f0af00b0c50 msgr2=0x7f0af00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 -- 172.21.15.107:0/3670446584 shutdown_connections 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.132+0000 7f0af4c90640 1 -- 172.21.15.107:0/3670446584 wait complete. 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.134+0000 7f0af4c90640 1 Processor -- start 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.134+0000 7f0af4c90640 1 -- start start 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 0x7f0af0099df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af00c49f0 0x7f0af015c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01468c0 con 0x7f0af006de10 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af00b1d10 con 0x7f0af0072160 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0af4c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af00b1f60 con 0x7f0af00c49f0 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0aee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0aee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59050/0 (socket says 172.21.15.107:59050) 2023-11-24T08:10:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.135+0000 7f0aee59c640 1 -- 172.21.15.107:0/1413337408 learned_addr learned my addr 172.21.15.107:0/1413337408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0ae7fff640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 0x7f0af0099df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 -- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af00c49f0 msgr2=0x7f0af015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aeed9d640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af00c49f0 0x7f0af015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af00c49f0 0x7f0af015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 -- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 msgr2=0x7f0af0099df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 0x7f0af0099df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 -- 172.21.15.107:0/1413337408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae00094d0 con 0x7f0af006de10 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0ae7fff640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 0x7f0af0099df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0aee59c640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009870 tx=0x7f0ae0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0039d60 con 0x7f0af006de10 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae0043400 con 0x7f0af006de10 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00437b0 con 0x7f0af006de10 2023-11-24T08:10:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.136+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af00b2240 con 0x7f0af006de10 2023-11-24T08:10:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.137+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af015c2e0 con 0x7f0af006de10 2023-11-24T08:10:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.138+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af004e8b0 con 0x7f0af006de10 2023-11-24T08:10:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.138+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f0ae0043910 con 0x7f0af006de10 2023-11-24T08:10:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.139+0000 7f0ae6ffd640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 0x7f0ad006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.139+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0ae003e070 con 0x7f0af006de10 2023-11-24T08:10:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.140+0000 7f0ae7fff640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 0x7f0ad006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.140+0000 7f0ae7fff640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 0x7f0ad006db70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0af00b1050 tx=0x7f0ad8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.144+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ae0077d80 con 0x7f0af006de10 2023-11-24T08:10:08.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.331+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0af015dab0 con 0x7f0af006de10 2023-11-24T08:10:08.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.332+0000 7f0ae6ffd640 1 -- 172.21.15.107:0/1413337408 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ae0034020 con 0x7f0af006de10 2023-11-24T08:10:08.334 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:10:08.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 msgr2=0x7f0ad006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 0x7f0ad006db70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0af00b1050 tx=0x7f0ad8008040 comp rx=0 tx=0).stop 2023-11-24T08:10:08.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 msgr2=0x7f0af015b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009870 tx=0x7f0ae0004730 comp rx=0 tx=0).stop 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 shutdown_connections 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ad006b6b0 0x7f0ad006db70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af00c49f0 0x7f0af015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af0072160 0x7f0af0099df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 --2- 172.21.15.107:0/1413337408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af006de10 0x7f0af015b960 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 >> 172.21.15.107:0/1413337408 conn(0x7f0af00b0c50 msgr2=0x7f0af015deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:08.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 shutdown_connections 2023-11-24T08:10:08.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.336+0000 7f0af4c90640 1 -- 172.21.15.107:0/1413337408 wait complete. 2023-11-24T08:10:08.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:10:08.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.573+0000 7fa6f06cf640 1 Processor -- start 2023-11-24T08:10:08.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.573+0000 7fa6f06cf640 1 -- start start 2023-11-24T08:10:08.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.575+0000 7fa6f06cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6e80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.575+0000 7fa6f06cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e80cc3a0 con 0x7fa6e80cbf20 2023-11-24T08:10:08.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.575+0000 7fa6f06cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e815eea0 con 0x7fa6e80c59b0 2023-11-24T08:10:08.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.575+0000 7fa6f06cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e815b820 con 0x7fa6e815b440 2023-11-24T08:10:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6edc6a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa6e80cbf20 0x7fa6e80c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33180/0 (socket says 172.21.15.107:33180) 2023-11-24T08:10:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6edc6a640 1 -- 172.21.15.107:0/3096166326 learned_addr learned my addr 172.21.15.107:0/3096166326 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6ee46b640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6e80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264906800 0 0) 0x7fa6d8001110 con 0x7fa6e80cbf20 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d4003520 con 0x7fa6e80cbf20 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.576+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219477910 0 0) 0x7fa6d4003520 con 0x7fa6e80cbf20 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 msgr2=0x7fa6e80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6e80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 >> v1:172.21.15.120:6789/0 conn(0x7fa6e80c59b0 legacy=0x7fa6e806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80abf30 con 0x7fa6e80cbf20 2023-11-24T08:10:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa6d8001800 con 0x7fa6e80cbf20 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 >> v1:172.21.15.107:6789/0 conn(0x7fa6e80cbf20 legacy=0x7fa6e80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6d40039a0 0x7fa6d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e815b820 con 0x7fa6d40047c0 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d400bab0 con 0x7fa6d40039a0 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d400bc70 con 0x7fa6e815b440 2023-11-24T08:10:08.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6ee46b640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6d40039a0 0x7fa6d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6eec6c640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6edc6a640 1 -- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 msgr2=0x7fa6d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e815b440 0x7fa6d40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6edc6a640 1 -- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6d40039a0 msgr2=0x7fa6d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.577+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6d40039a0 0x7fa6d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.578+0000 7fa6edc6a640 1 -- 172.21.15.107:0/3096166326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4003520 con 0x7fa6d40047c0 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6d4004c40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c240 tx=0x7fa6d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da05cf79fee27bb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8005ad0 con 0x7fa6d40047c0 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d8005c50 con 0x7fa6d40047c0 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6ed469640 1 -- 172.21.15.107:0/3096166326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d802c8b0 con 0x7fa6d40047c0 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6f06cf640 1 -- 172.21.15.107:0/3096166326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 msgr2=0x7fa6d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/3096166326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6d4004c40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c240 tx=0x7fa6d40057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.579+0000 7fa6eec6c640 1 -- 172.21.15.107:0/3096166326 reap_dead start 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.580+0000 7fa6f06cf640 1 -- 172.21.15.107:0/3096166326 shutdown_connections 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.580+0000 7fa6f06cf640 1 -- 172.21.15.107:0/3096166326 >> 172.21.15.107:0/3096166326 conn(0x7fa6e80b0c50 msgr2=0x7fa6e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.580+0000 7fa6f06cf640 1 -- 172.21.15.107:0/3096166326 shutdown_connections 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.580+0000 7fa6f06cf640 1 -- 172.21.15.107:0/3096166326 wait complete. 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.581+0000 7fa6f06cf640 1 Processor -- start 2023-11-24T08:10:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.581+0000 7fa6f06cf640 1 -- start start 2023-11-24T08:10:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80b4790 0x7fa6e80b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80b21f0 0x7fa6e80ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80b2680 con 0x7fa6d40047c0 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8152e70 con 0x7fa6e80b4790 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.582+0000 7fa6f06cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80cdff0 con 0x7fa6e80b21f0 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59066/0 (socket says 172.21.15.107:59066) 2023-11-24T08:10:08.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 -- 172.21.15.107:0/2160521568 learned_addr learned my addr 172.21.15.107:0/2160521568 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:08.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 -- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80b21f0 msgr2=0x7fa6e80ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80b21f0 0x7fa6e80ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 -- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80b4790 msgr2=0x7fa6e80b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80b4790 0x7fa6e80b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 -- 172.21.15.107:0/2160521568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80ce2d0 con 0x7fa6d40047c0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.583+0000 7fa6ee46b640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4011600 tx=0x7fa6e4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.584+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400f920 con 0x7fa6d40047c0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.584+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e400fa80 con 0x7fa6d40047c0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.584+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400fdf0 con 0x7fa6d40047c0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.584+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e80ce5b0 con 0x7fa6d40047c0 2023-11-24T08:10:08.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.584+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e80b4c10 con 0x7fa6d40047c0 2023-11-24T08:10:08.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.585+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e804e8b0 con 0x7fa6d40047c0 2023-11-24T08:10:08.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.586+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fa6e4015460 con 0x7fa6d40047c0 2023-11-24T08:10:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.587+0000 7fa6deffd640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 0x7fa6cc06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.587+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa6e4088540 con 0x7fa6d40047c0 2023-11-24T08:10:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.587+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 0x7fa6cc06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.588+0000 7fa6edc6a640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 0x7fa6cc06dba0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8034000 tx=0x7fa6d8007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:08.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.592+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6e4054e70 con 0x7fa6d40047c0 2023-11-24T08:10:08.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.781+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6e806dc80 con 0x7fa6d40047c0 2023-11-24T08:10:08.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.782+0000 7fa6deffd640 1 -- 172.21.15.107:0/2160521568 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa6e4022020 con 0x7fa6d40047c0 2023-11-24T08:10:08.784 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.784+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 msgr2=0x7fa6cc06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.784+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 0x7fa6cc06dba0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8034000 tx=0x7fa6d8007000 comp rx=0 tx=0).stop 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.784+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 msgr2=0x7fa6e814e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.784+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4011600 tx=0x7fa6e4011ac0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 shutdown_connections 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa6cc06b6e0 0x7fa6cc06dba0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80b21f0 0x7fa6e80ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80b4790 0x7fa6e80b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 --2- 172.21.15.107:0/2160521568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6d40047c0 0x7fa6e814e370 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:08.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 >> 172.21.15.107:0/2160521568 conn(0x7fa6e80b0c50 msgr2=0x7fa6e8071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:08.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 shutdown_connections 2023-11-24T08:10:08.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:08.785+0000 7fa6f06cf640 1 -- 172.21.15.107:0/2160521568 wait complete. 2023-11-24T08:10:08.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:10:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.011+0000 7f40429a5640 1 Processor -- start 2023-11-24T08:10:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.011+0000 7f40429a5640 1 -- start start 2023-11-24T08:10:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40429a5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 0x7f403c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40429a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40429a5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c15ec90 con 0x7f403c072160 2023-11-24T08:10:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40429a5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403c0cd530 con 0x7f403c06de10 2023-11-24T08:10:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40429a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0c4ed0 con 0x7f403c0c49f0 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f40337fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 0x7f403c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f4040f42640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f4040f42640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:52666/0 (socket says 172.21.15.107:52666) 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.013+0000 7f4040f42640 1 -- 172.21.15.107:0/1589881150 learned_addr learned my addr 172.21.15.107:0/1589881150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.014+0000 7f4040f42640 1 -- 172.21.15.107:0/1589881150 >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 msgr2=0x7f403c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.014+0000 7f4040f42640 1 --2- 172.21.15.107:0/1589881150 >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 0x7f403c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.014+0000 7f4040f42640 1 -- 172.21.15.107:0/1589881150 >> v1:172.21.15.120:6789/0 conn(0x7f403c072160 legacy=0x7f403c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.014+0000 7f4040f42640 1 -- 172.21.15.107:0/1589881150 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c0c51b0 con 0x7f403c0c49f0 2023-11-24T08:10:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.014+0000 7f40337fe640 1 --2- 172.21.15.107:0/1589881150 >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 0x7f403c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:10:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.016+0000 7f4040f42640 1 --2- 172.21.15.107:0/1589881150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f403400d150 tx=0x7f403402e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57790410b611403d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.017+0000 7f403b7fe640 1 -- 172.21.15.107:0/1589881150 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403400b0d0 con 0x7f403c0c49f0 2023-11-24T08:10:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.017+0000 7f403b7fe640 1 -- 172.21.15.107:0/1589881150 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403400b230 con 0x7f403c0c49f0 2023-11-24T08:10:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.017+0000 7f403b7fe640 1 -- 172.21.15.107:0/1589881150 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034009600 con 0x7f403c0c49f0 2023-11-24T08:10:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.017+0000 7f40429a5640 1 -- 172.21.15.107:0/1589881150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 msgr2=0x7f403c0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.017+0000 7f40429a5640 1 --2- 172.21.15.107:0/1589881150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f403400d150 tx=0x7f403402e160 comp rx=0 tx=0).stop 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 -- 172.21.15.107:0/1589881150 shutdown_connections 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 --2- 172.21.15.107:0/1589881150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c0c49f0 0x7f403c0705c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 --2- 172.21.15.107:0/1589881150 >> v2:172.21.15.120:3300/0 conn(0x7f403c06de10 0x7f403c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 -- 172.21.15.107:0/1589881150 >> 172.21.15.107:0/1589881150 conn(0x7f403c0b0c50 msgr2=0x7f403c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 -- 172.21.15.107:0/1589881150 shutdown_connections 2023-11-24T08:10:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.018+0000 7f40429a5640 1 -- 172.21.15.107:0/1589881150 wait complete. 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.019+0000 7f40429a5640 1 Processor -- start 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.020+0000 7f40429a5640 1 -- start start 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 0x7f403c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c0c49f0 0x7f403c14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c09a220 con 0x7f403c06de10 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c14e480 con 0x7f403c0c49f0 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40429a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c15c3b0 con 0x7f403c072160 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59078/0 (socket says 172.21.15.107:59078) 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 -- 172.21.15.107:0/2979122474 learned_addr learned my addr 172.21.15.107:0/2979122474 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40337fe640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 0x7f403c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f4040f42640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c0c49f0 0x7f403c14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 -- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 msgr2=0x7f403c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 0x7f403c0999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 -- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c0c49f0 msgr2=0x7f403c14e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c0c49f0 0x7f403c14e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f403bfff640 1 -- 172.21.15.107:0/2979122474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4024009400 con 0x7f403c06de10 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.021+0000 7f40337fe640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 0x7f403c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f403bfff640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c400 tx=0x7f402c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c0052d0 con 0x7f403c06de10 2023-11-24T08:10:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c005450 con 0x7f403c06de10 2023-11-24T08:10:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403400bd40 con 0x7f403c06de10 2023-11-24T08:10:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c007260 con 0x7f403c06de10 2023-11-24T08:10:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.022+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c15bd60 con 0x7f403c06de10 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.023+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c04e8b0 con 0x7f403c06de10 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.024+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f402c0073c0 con 0x7f403c06de10 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.025+0000 7f40397fa640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 0x7f401c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.025+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f402c087280 con 0x7f403c06de10 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.025+0000 7f40337fe640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 0x7f401c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.026+0000 7f40337fe640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 0x7f401c06dba0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f403c15b870 tx=0x7f4024008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.030+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f402c053b90 con 0x7f403c06de10 2023-11-24T08:10:09.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.220+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f403c15ec90 con 0x7f403c06de10 2023-11-24T08:10:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.221+0000 7f40397fa640 1 -- 172.21.15.107:0/2979122474 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) v1 ==== 100+0+11 (secure 0 0 0) 0x7f403c15ec90 con 0x7f403c06de10 2023-11-24T08:10:09.223 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 msgr2=0x7f401c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 0x7f401c06dba0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f403c15b870 tx=0x7f4024008040 comp rx=0 tx=0).stop 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 msgr2=0x7f403c14df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c400 tx=0x7f402c00c8c0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 shutdown_connections 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f401c06b6e0 0x7f401c06dba0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f403c0c49f0 0x7f403c14e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403c072160 0x7f403c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 --2- 172.21.15.107:0/2979122474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f403c06de10 0x7f403c14df40 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 >> 172.21.15.107:0/2979122474 conn(0x7f403c0b0c50 msgr2=0x7f403c0abf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:09.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 shutdown_connections 2023-11-24T08:10:09.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.224+0000 7f40429a5640 1 -- 172.21.15.107:0/2979122474 wait complete. 2023-11-24T08:10:09.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.441+0000 7fde13fff640 1 Processor -- start 2023-11-24T08:10:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.441+0000 7fde13fff640 1 -- start start 2023-11-24T08:10:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.442+0000 7fde13fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 0x7fde140ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde13fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde140b6040 0x7fde140b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde13fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140d1d20 con 0x7fde140b8ce0 2023-11-24T08:10:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde13fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde140b90c0 con 0x7fde140b6040 2023-11-24T08:10:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde13fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde140d1110 con 0x7fde140d02f0 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde12ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fde140b8ce0 0x7fde140cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33188/0 (socket says 172.21.15.107:33188) 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde12ffd640 1 -- 172.21.15.107:0/2535906719 learned_addr learned my addr 172.21.15.107:0/2535906719 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde137fe640 1 --2- 172.21.15.107:0/2535906719 >> v2:172.21.15.107:3300/0 conn(0x7fde140b6040 0x7fde140b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde127fc640 1 --2- 172.21.15.107:0/2535906719 >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 0x7fde140ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde137fe640 1 -- 172.21.15.107:0/2535906719 >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 msgr2=0x7fde140ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde137fe640 1 --2- 172.21.15.107:0/2535906719 >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 0x7fde140ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde137fe640 1 -- 172.21.15.107:0/2535906719 >> v1:172.21.15.107:6789/0 conn(0x7fde140b8ce0 legacy=0x7fde140cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde137fe640 1 -- 172.21.15.107:0/2535906719 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde140cf2d0 con 0x7fde140b6040 2023-11-24T08:10:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde127fc640 1 --2- 172.21.15.107:0/2535906719 >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 0x7fde140ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.443+0000 7fde11ffb640 1 -- 172.21.15.107:0/2535906719 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532012614 0 0) 0x7fde140d1d20 con 0x7fde140b8ce0 2023-11-24T08:10:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.444+0000 7fde137fe640 1 --2- 172.21.15.107:0/2535906719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140b64b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fde1409c960 tx=0x7fde0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c5a98e24e85fbc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde11ffb640 1 -- 172.21.15.107:0/2535906719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c031830 con 0x7fde140b6040 2023-11-24T08:10:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde11ffb640 1 -- 172.21.15.107:0/2535906719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c031990 con 0x7fde140b6040 2023-11-24T08:10:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde11ffb640 1 -- 172.21.15.107:0/2535906719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c031d20 con 0x7fde140b6040 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 -- 172.21.15.107:0/2535906719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 msgr2=0x7fde140b64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 --2- 172.21.15.107:0/2535906719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140b64b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fde1409c960 tx=0x7fde0c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 -- 172.21.15.107:0/2535906719 shutdown_connections 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 --2- 172.21.15.107:0/2535906719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140b64b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 --2- 172.21.15.107:0/2535906719 >> v2:172.21.15.120:3300/0 conn(0x7fde140d02f0 0x7fde140ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 -- 172.21.15.107:0/2535906719 >> 172.21.15.107:0/2535906719 conn(0x7fde1409c530 msgr2=0x7fde140d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 -- 172.21.15.107:0/2535906719 shutdown_connections 2023-11-24T08:10:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.445+0000 7fde13fff640 1 -- 172.21.15.107:0/2535906719 wait complete. 2023-11-24T08:10:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.447+0000 7fde13fff640 1 Processor -- start 2023-11-24T08:10:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.447+0000 7fde13fff640 1 -- start start 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde140b8ce0 0x7fde14148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde140d02f0 0x7fde140cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c7090 con 0x7fde140b6040 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde14149050 con 0x7fde140d02f0 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde13fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140ca460 con 0x7fde140b8ce0 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde12ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde12ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59096/0 (socket says 172.21.15.107:59096) 2023-11-24T08:10:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.448+0000 7fde12ffd640 1 -- 172.21.15.107:0/2003414687 learned_addr learned my addr 172.21.15.107:0/2003414687 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 -- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde140b8ce0 msgr2=0x7fde14148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde140b8ce0 0x7fde14148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 -- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde140d02f0 msgr2=0x7fde140cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde140d02f0 0x7fde140cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 -- 172.21.15.107:0/2003414687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c009400 con 0x7fde140b6040 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde12ffd640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c890 tx=0x7fde0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400a640 con 0x7fde140b6040 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400a7c0 con 0x7fde140b6040 2023-11-24T08:10:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde140ca740 con 0x7fde140b6040 2023-11-24T08:10:09.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400ab70 con 0x7fde140b6040 2023-11-24T08:10:09.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.449+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde140cad00 con 0x7fde140b6040 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.451+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde140d2190 con 0x7fde140b6040 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.452+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fde0400acd0 con 0x7fde140b6040 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.452+0000 7fde0b7fe640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 0x7fddf406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.452+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fde04087a10 con 0x7fde140b6040 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.453+0000 7fde127fc640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 0x7fddf406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.453+0000 7fde127fc640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 0x7fddf406db70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddfc006fd0 tx=0x7fddfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.457+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde140d2190 con 0x7fde140b6040 2023-11-24T08:10:09.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.689+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde140d2190 con 0x7fde140b6040 2023-11-24T08:10:09.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.690+0000 7fde0b7fe640 1 -- 172.21.15.107:0/2003414687 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fde140d2190 con 0x7fde140b6040 2023-11-24T08:10:09.693 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:10:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-11-24T08:10:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.693+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 msgr2=0x7fddf406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.693+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 0x7fddf406db70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddfc006fd0 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2023-11-24T08:10:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.693+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 msgr2=0x7fde140c6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c890 tx=0x7fde0400cd50 comp rx=0 tx=0).stop 2023-11-24T08:10:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 shutdown_connections 2023-11-24T08:10:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fddf406b6b0 0x7fddf406db70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde140d02f0 0x7fde140cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde140b8ce0 0x7fde14148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 --2- 172.21.15.107:0/2003414687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde140b6040 0x7fde140c6b50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 >> 172.21.15.107:0/2003414687 conn(0x7fde1409c530 msgr2=0x7fde1415df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 shutdown_connections 2023-11-24T08:10:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.694+0000 7fde13fff640 1 -- 172.21.15.107:0/2003414687 wait complete. 2023-11-24T08:10:09.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:09.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.924+0000 7ff4dfbc5640 1 Processor -- start 2023-11-24T08:10:09.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.925+0000 7ff4dfbc5640 1 -- start start 2023-11-24T08:10:09.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.926+0000 7ff4dfbc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4d806e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.926+0000 7ff4dfbc5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d806ef80 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.926+0000 7ff4dfbc5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80c7170 con 0x7ff4d806eb20 2023-11-24T08:10:09.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.926+0000 7ff4dfbc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81452a0 con 0x7ff4d8144ec0 2023-11-24T08:10:09.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4de162640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff4d80cf5b0 0x7ff4d8070090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33202/0 (socket says 172.21.15.107:33202) 2023-11-24T08:10:09.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4de162640 1 -- 172.21.15.107:0/4226393620 learned_addr learned my addr 172.21.15.107:0/4226393620 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:09.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dd961640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4d806e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763940298 0 0) 0x7ff4d806ef80 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c0003520 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556701490 0 0) 0x7ff4c0003520 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 msgr2=0x7ff4d806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4d806e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 >> v1:172.21.15.120:6789/0 conn(0x7ff4d806eb20 legacy=0x7ff4d80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80cfb60 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.927+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff4d0001670 con 0x7ff4d80cf5b0 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 >> v1:172.21.15.107:6789/0 conn(0x7ff4d80cf5b0 legacy=0x7ff4d8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c00039a0 0x7ff4c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c0004d00 0x7ff4c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81452a0 con 0x7ff4c00039a0 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c000ba30 con 0x7ff4c0004d00 2023-11-24T08:10:09.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c000bbf0 con 0x7ff4d8144ec0 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c00039a0 0x7ff4c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd160640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 -- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 msgr2=0x7ff4c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 -- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c0004d00 msgr2=0x7ff4c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c0004d00 0x7ff4c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd961640 1 -- 172.21.15.107:0/4226393620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c0003520 con 0x7ff4c00039a0 2023-11-24T08:10:09.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.928+0000 7ff4dd160640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8144ec0 0x7ff4c00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.929+0000 7ff4dd961640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c00039a0 0x7ff4c0003dc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009f90 tx=0x7ff4c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518c76fdf37fb9ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c802cbd0 con 0x7ff4c00039a0 2023-11-24T08:10:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c800f040 con 0x7ff4c00039a0 2023-11-24T08:10:09.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dc95f640 1 -- 172.21.15.107:0/4226393620 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8034440 con 0x7ff4c00039a0 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/4226393620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c00039a0 msgr2=0x7ff4c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/4226393620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c00039a0 0x7ff4c0003dc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009f90 tx=0x7ff4c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4de162640 1 -- 172.21.15.107:0/4226393620 reap_dead start 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/4226393620 shutdown_connections 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/4226393620 >> 172.21.15.107:0/4226393620 conn(0x7ff4d80d12d0 msgr2=0x7ff4d80d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/4226393620 shutdown_connections 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.930+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/4226393620 wait complete. 2023-11-24T08:10:09.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.932+0000 7ff4dfbc5640 1 Processor -- start 2023-11-24T08:10:09.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.932+0000 7ff4dfbc5640 1 -- start start 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c00039a0 0x7ff4d80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8156a00 0x7ff4d81584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8142700 con 0x7ff4d80cb830 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d80cbda0 con 0x7ff4c00039a0 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.933+0000 7ff4dfbc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d80cc080 con 0x7ff4d8156a00 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59110/0 (socket says 172.21.15.107:59110) 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 -- 172.21.15.107:0/1399030371 learned_addr learned my addr 172.21.15.107:0/1399030371 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:09.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 -- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8156a00 msgr2=0x7ff4d81584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4de162640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8156a00 0x7ff4d81584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd961640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c00039a0 0x7ff4d80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8156a00 0x7ff4d81584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 -- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c00039a0 msgr2=0x7ff4d80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c00039a0 0x7ff4d80cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:09.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 -- 172.21.15.107:0/1399030371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80cc360 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.934+0000 7ff4dd160640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00d7a0 tx=0x7ff4cc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.935+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc017070 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.935+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc005070 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.935+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc005420 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.935+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d80cc640 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.935+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d8156e80 con 0x7ff4d80cb830 2023-11-24T08:10:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.936+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d81570a0 con 0x7ff4d80cb830 2023-11-24T08:10:09.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.937+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7ff4cc007030 con 0x7ff4d80cb830 2023-11-24T08:10:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.938+0000 7ff4d67fc640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 0x7ff4c406dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.938+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff4cc087950 con 0x7ff4d80cb830 2023-11-24T08:10:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.938+0000 7ff4dd961640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 0x7ff4c406dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:09.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.939+0000 7ff4dd961640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 0x7ff4c406dbc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8036000 tx=0x7ff4c802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:09.943+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4cc0542c0 con 0x7ff4d80cb830 2023-11-24T08:10:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.137+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4d80c7920 con 0x7ff4d80cb830 2023-11-24T08:10:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.141+0000 7ff4d67fc640 1 -- 172.21.15.107:0/1399030371 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16630 (secure 0 0 0) 0x7ff4cc0578e0 con 0x7ff4d80cb830 2023-11-24T08:10:10.144 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:10.144 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":66,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:09.823967+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 msgr2=0x7ff4c406dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 0x7ff4c406dbc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8036000 tx=0x7ff4c802e000 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 msgr2=0x7ff4d81421c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00d7a0 tx=0x7ff4cc00dc60 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 shutdown_connections 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c406b700 0x7ff4c406dbc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4d8156a00 0x7ff4d81584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d80cb830 0x7ff4d81421c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 --2- 172.21.15.107:0/1399030371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c00039a0 0x7ff4d80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.144+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 >> 172.21.15.107:0/1399030371 conn(0x7ff4d80d12d0 msgr2=0x7ff4d80d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.145+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 shutdown_connections 2023-11-24T08:10:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.145+0000 7ff4dfbc5640 1 -- 172.21.15.107:0/1399030371 wait complete. 2023-11-24T08:10:10.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:10:10.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.368+0000 7fc7f349b640 1 Processor -- start 2023-11-24T08:10:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.368+0000 7fc7f349b640 1 -- start start 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f349b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc7ec0b2250 0x7fc7ec15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f349b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f349b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec15ab80 con 0x7fc7ec0b2aa0 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f349b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec15a990 con 0x7fc7ec0b2630 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f349b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec15e680 con 0x7fc7ec0b2250 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f0a36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f0a36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59114/0 (socket says 172.21.15.107:59114) 2023-11-24T08:10:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f0a36640 1 -- 172.21.15.107:0/265987918 learned_addr learned my addr 172.21.15.107:0/265987918 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f0a36640 1 -- 172.21.15.107:0/265987918 >> v2:172.21.15.120:3300/0 conn(0x7fc7ec0b2250 msgr2=0x7fc7ec15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.370+0000 7fc7f1237640 1 --2- 172.21.15.107:0/265987918 >> v2:172.21.15.120:3300/0 conn(0x7fc7ec0b2250 0x7fc7ec15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.371+0000 7fc7f0a36640 1 --2- 172.21.15.107:0/265987918 >> v2:172.21.15.120:3300/0 conn(0x7fc7ec0b2250 0x7fc7ec15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.371+0000 7fc7f0a36640 1 -- 172.21.15.107:0/265987918 >> v1:172.21.15.120:6789/0 conn(0x7fc7ec0b2aa0 legacy=0x7fc7ec15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.371+0000 7fc7f0a36640 1 -- 172.21.15.107:0/265987918 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec15e960 con 0x7fc7ec0b2630 2023-11-24T08:10:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7f0a36640 1 --2- 172.21.15.107:0/265987918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec072a70 tx=0x7fc7dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce0a1086b579de2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7e3fff640 1 -- 172.21.15.107:0/265987918 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031870 con 0x7fc7ec0b2630 2023-11-24T08:10:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7e3fff640 1 -- 172.21.15.107:0/265987918 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc0319d0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7e3fff640 1 -- 172.21.15.107:0/265987918 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031d60 con 0x7fc7ec0b2630 2023-11-24T08:10:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7f349b640 1 -- 172.21.15.107:0/265987918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 msgr2=0x7fc7ec0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.372+0000 7fc7f349b640 1 --2- 172.21.15.107:0/265987918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec072a70 tx=0x7fc7dc02b910 comp rx=0 tx=0).stop 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 -- 172.21.15.107:0/265987918 shutdown_connections 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 --2- 172.21.15.107:0/265987918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec0ccf30 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 --2- 172.21.15.107:0/265987918 >> v2:172.21.15.120:3300/0 conn(0x7fc7ec0b2250 0x7fc7ec15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 -- 172.21.15.107:0/265987918 >> 172.21.15.107:0/265987918 conn(0x7fc7ec06a920 msgr2=0x7fc7ec06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 -- 172.21.15.107:0/265987918 shutdown_connections 2023-11-24T08:10:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.373+0000 7fc7f349b640 1 -- 172.21.15.107:0/265987918 wait complete. 2023-11-24T08:10:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.374+0000 7fc7f349b640 1 Processor -- start 2023-11-24T08:10:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.375+0000 7fc7f349b640 1 -- start start 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.375+0000 7fc7f349b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ec0b2250 0x7fc7ec14b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.375+0000 7fc7f349b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f349b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ec0b2aa0 0x7fc7ec15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f349b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec0adce0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f349b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec14bb60 con 0x7fc7ec0b2250 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f349b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec1571f0 con 0x7fc7ec0b2aa0 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59128/0 (socket says 172.21.15.107:59128) 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 -- 172.21.15.107:0/4145077716 learned_addr learned my addr 172.21.15.107:0/4145077716 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 -- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ec0b2aa0 msgr2=0x7fc7ec15d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ec0b2aa0 0x7fc7ec15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f1237640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ec0b2250 0x7fc7ec14b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 -- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ec0b2250 msgr2=0x7fc7ec14b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ec0b2250 0x7fc7ec14b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.376+0000 7fc7f0a36640 1 -- 172.21.15.107:0/4145077716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0094d0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7f0a36640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc000c00 tx=0x7fc7dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031840 con 0x7fc7ec0b2630 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc0443f0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec1574d0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0447a0 con 0x7fc7ec0b2630 2023-11-24T08:10:10.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.377+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec157a90 con 0x7fc7ec0b2630 2023-11-24T08:10:10.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.378+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec061a50 con 0x7fc7ec0b2630 2023-11-24T08:10:10.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.379+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fc7dc04d450 con 0x7fc7ec0b2630 2023-11-24T08:10:10.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.380+0000 7fc7e1ffb640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 0x7fc7d006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.380+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc7dc02d040 con 0x7fc7ec0b2630 2023-11-24T08:10:10.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.381+0000 7fc7f1237640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 0x7fc7d006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.381+0000 7fc7f1237640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 0x7fc7d006db70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006fd0 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.385+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7dc0dfe90 con 0x7fc7ec0b2630 2023-11-24T08:10:10.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:10:10.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:10:10.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.617+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ec067470 con 0x7fc7ec0b2630 2023-11-24T08:10:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.618+0000 7fc7e1ffb640 1 -- 172.21.15.107:0/4145077716 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fc7dc035070 con 0x7fc7ec0b2630 2023-11-24T08:10:10.620 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:10.620 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:10:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 msgr2=0x7fc7d006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 0x7fc7d006db70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006fd0 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 msgr2=0x7fc7ec15cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc000c00 tx=0x7fc7dc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 shutdown_connections 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc7d006b6b0 0x7fc7d006db70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ec0b2aa0 0x7fc7ec15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ec0b2630 0x7fc7ec15cae0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 --2- 172.21.15.107:0/4145077716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ec0b2250 0x7fc7ec14b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 >> 172.21.15.107:0/4145077716 conn(0x7fc7ec06a920 msgr2=0x7fc7ec15ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 shutdown_connections 2023-11-24T08:10:10.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.621+0000 7fc7f349b640 1 -- 172.21.15.107:0/4145077716 wait complete. 2023-11-24T08:10:10.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:10.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.838+0000 7f9325ee7640 1 Processor -- start 2023-11-24T08:10:10.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.839+0000 7f9325ee7640 1 -- start start 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9325ee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f932015b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9325ee7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932015e3f0 con 0x7f932015df80 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9325ee7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932015ab80 con 0x7f93200b2250 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9325ee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932015a990 con 0x7f932015b080 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9317fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f932015df80 0x7f93200ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33212/0 (socket says 172.21.15.107:33212) 2023-11-24T08:10:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.840+0000 7f9317fff640 1 -- 172.21.15.107:0/2097057149 learned_addr learned my addr 172.21.15.107:0/2097057149 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931ffff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f932015b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524373451 0 0) 0x7f932015e3f0 con 0x7f932015df80 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c003520 con 0x7f932015df80 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334718422 0 0) 0x7f930c003520 con 0x7f932015df80 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 msgr2=0x7f932015b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f932015b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 >> v1:172.21.15.120:6789/0 conn(0x7f93200b2250 legacy=0x7f93200b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200b2d40 con 0x7f932015df80 2023-11-24T08:10:10.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9308001670 con 0x7f932015df80 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 >> v1:172.21.15.107:6789/0 conn(0x7f932015df80 legacy=0x7f93200ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f930c0039a0 0x7f930c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c0047c0 0x7f930c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f930c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c00ba80 con 0x7f930c0047c0 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c00bbf0 con 0x7f930c0039a0 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c00bdb0 con 0x7f932015b080 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.841+0000 7f9317fff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c0047c0 0x7f930c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f931f7fe640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f930c0039a0 0x7f930c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f931ffff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f930c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f9317fff640 1 -- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 msgr2=0x7f930c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f9317fff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f932015b080 0x7f930c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f9317fff640 1 -- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f930c0039a0 msgr2=0x7f930c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f9317fff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f930c0039a0 0x7f930c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.842+0000 7f9317fff640 1 -- 172.21.15.107:0/2097057149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c003520 con 0x7f930c0047c0 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.843+0000 7f9317fff640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c0047c0 0x7f930c004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f930800c240 tx=0x7f930c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15de764628d49bf7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.843+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9308005ad0 con 0x7f930c0047c0 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.843+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9308005c50 con 0x7f930c0047c0 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.843+0000 7f931effd640 1 -- 172.21.15.107:0/2097057149 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930802c8d0 con 0x7f930c0047c0 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 -- 172.21.15.107:0/2097057149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c0047c0 msgr2=0x7f930c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 --2- 172.21.15.107:0/2097057149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c0047c0 0x7f930c004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f930800c240 tx=0x7f930c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f931ffff640 1 -- 172.21.15.107:0/2097057149 reap_dead start 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 -- 172.21.15.107:0/2097057149 shutdown_connections 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 -- 172.21.15.107:0/2097057149 >> 172.21.15.107:0/2097057149 conn(0x7f932006a920 msgr2=0x7f932006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 -- 172.21.15.107:0/2097057149 shutdown_connections 2023-11-24T08:10:10.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.844+0000 7f9325ee7640 1 -- 172.21.15.107:0/2097057149 wait complete. 2023-11-24T08:10:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.846+0000 7f9325ee7640 1 Processor -- start 2023-11-24T08:10:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.846+0000 7f9325ee7640 1 -- start start 2023-11-24T08:10:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f930c0047c0 0x7f9320152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 0x7f93200af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932015b800 con 0x7f93200afa30 2023-11-24T08:10:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932015bae0 con 0x7f9320153040 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9325ee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932015bdc0 con 0x7f930c0047c0 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59134/0 (socket says 172.21.15.107:59134) 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 -- 172.21.15.107:0/853545239 learned_addr learned my addr 172.21.15.107:0/853545239 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931f7fe640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f930c0047c0 0x7f9320152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9317fff640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 0x7f93200af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 -- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f930c0047c0 msgr2=0x7f9320152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f930c0047c0 0x7f9320152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 -- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 msgr2=0x7f93200af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 0x7f93200af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f931ffff640 1 -- 172.21.15.107:0/853545239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932015c0a0 con 0x7f93200afa30 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.847+0000 7f9317fff640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 0x7f93200af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f931ffff640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f9320158560 tx=0x7f9318009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318007cf0 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931800f050 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932015c380 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318014650 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.848+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932015cde0 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.849+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9320073f60 con 0x7f93200afa30 2023-11-24T08:10:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.850+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f93180147b0 con 0x7f93200afa30 2023-11-24T08:10:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.851+0000 7f931cff9640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 0x7f930006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.851+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9318088570 con 0x7f93200afa30 2023-11-24T08:10:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.851+0000 7f931f7fe640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 0x7f930006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.852+0000 7f931f7fe640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 0x7f930006db70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9320153550 tx=0x7f9310009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:10.855+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9318054ea0 con 0x7f93200afa30 2023-11-24T08:10:11.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.085+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f932015c570 con 0x7f93200afa30 2023-11-24T08:10:11.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.086+0000 7f931cff9640 1 -- 172.21.15.107:0/853545239 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f93180584c0 con 0x7f93200afa30 2023-11-24T08:10:11.088 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:11.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:09:04.481642+0000","modified":"2023-11-24T08:09:07.560596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3235590318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3235590318},{"type":"v1","addr":"172.21.15.107:6836","nonce":3235590318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T08:10:11.089 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.088+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 msgr2=0x7f930006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.088+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 0x7f930006db70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9320153550 tx=0x7f9310009040 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.088+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 msgr2=0x7f932015c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.088+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f9320158560 tx=0x7f9318009cd0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 shutdown_connections 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006b6b0 0x7f930006db70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93200afa30 0x7f932015c8a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320153040 0x7f93200af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 --2- 172.21.15.107:0/853545239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f930c0047c0 0x7f9320152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 >> 172.21.15.107:0/853545239 conn(0x7f932006a920 msgr2=0x7f93200708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 shutdown_connections 2023-11-24T08:10:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.089+0000 7f9325ee7640 1 -- 172.21.15.107:0/853545239 wait complete. 2023-11-24T08:10:11.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:11.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.320+0000 7f1aa4fb0640 1 Processor -- start 2023-11-24T08:10:11.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.320+0000 7f1aa4fb0640 1 -- start start 2023-11-24T08:10:11.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.321+0000 7f1aa4fb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1aa00b2250 0x7f1aa015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1aa4fb0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 0x7f1aa00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1aa4fb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa015ab80 con 0x7f1aa00b2aa0 2023-11-24T08:10:11.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1aa4fb0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa015a990 con 0x7f1aa00b2250 2023-11-24T08:10:11.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1aa4fb0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa015e680 con 0x7f1aa00b2630 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1a9e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1aa00b2250 0x7f1aa015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1a9e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1aa00b2250 0x7f1aa015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59150/0 (socket says 172.21.15.107:59150) 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3734791405 learned_addr learned my addr 172.21.15.107:0/3734791405 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.322+0000 7f1a9dd9b640 1 --2- 172.21.15.107:0/3734791405 >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 0x7f1aa00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.323+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3734791405 >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 msgr2=0x7f1aa00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.323+0000 7f1a9e59c640 1 --2- 172.21.15.107:0/3734791405 >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 0x7f1aa00ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.323+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3734791405 >> v1:172.21.15.120:6789/0 conn(0x7f1aa00b2aa0 legacy=0x7f1aa015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.323+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3734791405 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa015e960 con 0x7f1aa00b2250 2023-11-24T08:10:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.323+0000 7f1a9dd9b640 1 --2- 172.21.15.107:0/3734791405 >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 0x7f1aa00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:10:11.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.324+0000 7f1a9e59c640 1 --2- 172.21.15.107:0/3734791405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa015df80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00ae5c0 tx=0x7f1a9002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df719f95bb8f17b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.325+0000 7f1a9d59a640 1 -- 172.21.15.107:0/3734791405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90030880 con 0x7f1aa00b2250 2023-11-24T08:10:11.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.325+0000 7f1a9d59a640 1 -- 172.21.15.107:0/3734791405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a900309e0 con 0x7f1aa00b2250 2023-11-24T08:10:11.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.325+0000 7f1a9d59a640 1 -- 172.21.15.107:0/3734791405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90030d70 con 0x7f1aa00b2250 2023-11-24T08:10:11.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.325+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3734791405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 msgr2=0x7f1aa015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.325+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3734791405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa015df80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00ae5c0 tx=0x7f1a9002b9c0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3734791405 shutdown_connections 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3734791405 >> v2:172.21.15.120:3300/0 conn(0x7f1aa00b2630 0x7f1aa00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3734791405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa015df80 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3734791405 >> 172.21.15.107:0/3734791405 conn(0x7f1aa006a920 msgr2=0x7f1aa006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3734791405 shutdown_connections 2023-11-24T08:10:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.326+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3734791405 wait complete. 2023-11-24T08:10:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.327+0000 7f1aa4fb0640 1 Processor -- start 2023-11-24T08:10:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.328+0000 7f1aa4fb0640 1 -- start start 2023-11-24T08:10:11.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa00b2630 0x7f1aa01f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1aa00b2aa0 0x7f1aa00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00b19b0 con 0x7f1aa00b2250 2023-11-24T08:10:11.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00b0d30 con 0x7f1aa00b2630 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1aa4fb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00b1010 con 0x7f1aa00b2aa0 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59154/0 (socket says 172.21.15.107:59154) 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3997376007 learned_addr learned my addr 172.21.15.107:0/3997376007 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1aa00b2aa0 msgr2=0x7f1aa00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9dd9b640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa00b2630 0x7f1aa01f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1aa00b2aa0 0x7f1aa00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa00b2630 msgr2=0x7f1aa01f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa00b2630 0x7f1aa01f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.329+0000 7f1a9e59c640 1 -- 172.21.15.107:0/3997376007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a900094d0 con 0x7f1aa00b2250 2023-11-24T08:10:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1a9e59c640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1a9002ba00 tx=0x7f1a90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90039d60 con 0x7f1aa00b2250 2023-11-24T08:10:11.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90043400 con 0x7f1aa00b2250 2023-11-24T08:10:11.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa00b12f0 con 0x7f1aa00b2250 2023-11-24T08:10:11.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a900437b0 con 0x7f1aa00b2250 2023-11-24T08:10:11.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.330+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa01f33d0 con 0x7f1aa00b2250 2023-11-24T08:10:11.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.331+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa0067470 con 0x7f1aa00b2250 2023-11-24T08:10:11.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.332+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f1a90043910 con 0x7f1aa00b2250 2023-11-24T08:10:11.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.333+0000 7f1a8effd640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 0x7f1a8006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.333+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1a9003e070 con 0x7f1aa00b2250 2023-11-24T08:10:11.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.333+0000 7f1a9dd9b640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 0x7f1a8006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.334+0000 7f1a9dd9b640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 0x7f1a8006db70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1aa01f6ab0 tx=0x7f1a94008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.337+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a90077bf0 con 0x7f1aa00b2250 2023-11-24T08:10:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.531+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aa006e360 con 0x7f1aa00b2250 2023-11-24T08:10:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.533+0000 7f1a8effd640 1 -- 172.21.15.107:0/3997376007 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f1a90034070 con 0x7f1aa00b2250 2023-11-24T08:10:11.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:11.535 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":66,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:09.823967+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:09:02.329269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T08:09:03.506902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 msgr2=0x7f1a8006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 0x7f1a8006db70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1aa01f6ab0 tx=0x7f1a94008040 comp rx=0 tx=0).stop 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 msgr2=0x7f1aa0144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1a9002ba00 tx=0x7f1a90004730 comp rx=0 tx=0).stop 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 shutdown_connections 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a8006b6b0 0x7f1a8006db70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1aa00b2aa0 0x7f1aa00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa00b2630 0x7f1aa01f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 --2- 172.21.15.107:0/3997376007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa00b2250 0x7f1aa0144b90 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 >> 172.21.15.107:0/3997376007 conn(0x7f1aa006a920 msgr2=0x7f1aa0159300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 shutdown_connections 2023-11-24T08:10:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.536+0000 7f1aa4fb0640 1 -- 172.21.15.107:0/3997376007 wait complete. 2023-11-24T08:10:11.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:10:11.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.771+0000 7fb70e5c9640 1 Processor -- start 2023-11-24T08:10:11.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.771+0000 7fb70e5c9640 1 -- start start 2023-11-24T08:10:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.772+0000 7fb70e5c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb70815b440 0x7fb7080cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.772+0000 7fb70e5c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 0x7fb7080c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.772+0000 7fb70e5c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70815eea0 con 0x7fb7080c57e0 2023-11-24T08:10:11.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.772+0000 7fb70e5c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080cc3d0 con 0x7fb70815b440 2023-11-24T08:10:11.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.772+0000 7fb70e5c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70815b820 con 0x7fb7080cbf20 2023-11-24T08:10:11.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.773+0000 7fb707fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb70815b440 0x7fb7080cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.773+0000 7fb707fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb70815b440 0x7fb7080cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:45748/0 (socket says 172.21.15.107:45748) 2023-11-24T08:10:11.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.773+0000 7fb707fff640 1 -- 172.21.15.107:0/2227123924 learned_addr learned my addr 172.21.15.107:0/2227123924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:11.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.773+0000 7fb7077fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 0x7fb7080c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.773+0000 7fb70cb66640 1 --1- 172.21.15.107:0/2227123924 >> v1:172.21.15.120:6789/0 conn(0x7fb7080c57e0 0x7fb7080c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:46682/0 (socket says 172.21.15.107:46682) 2023-11-24T08:10:11.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.774+0000 7fb707fff640 1 -- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 msgr2=0x7fb7080c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:11.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.774+0000 7fb707fff640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 0x7fb7080c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.774+0000 7fb707fff640 1 -- 172.21.15.107:0/2227123924 >> v1:172.21.15.120:6789/0 conn(0x7fb7080c57e0 legacy=0x7fb7080c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.774+0000 7fb707fff640 1 -- 172.21.15.107:0/2227123924 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70806d600 con 0x7fb70815b440 2023-11-24T08:10:11.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.774+0000 7fb7077fe640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 0x7fb7080c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.775+0000 7fb707fff640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70815b440 0x7fb7080cb9e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009870 tx=0x7fb6f802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55aaca9ca90e0535 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb706ffd640 1 -- 172.21.15.107:0/2227123924 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8031640 con 0x7fb70815b440 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb706ffd640 1 -- 172.21.15.107:0/2227123924 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f80317a0 con 0x7fb70815b440 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb706ffd640 1 -- 172.21.15.107:0/2227123924 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8031b30 con 0x7fb70815b440 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb70e5c9640 1 -- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70815b440 msgr2=0x7fb7080cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70815b440 0x7fb7080cb9e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009870 tx=0x7fb6f802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb70e5c9640 1 -- 172.21.15.107:0/2227123924 shutdown_connections 2023-11-24T08:10:11.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.776+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7080cbf20 0x7fb7080c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.777+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/2227123924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70815b440 0x7fb7080cb9e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.777+0000 7fb70e5c9640 1 -- 172.21.15.107:0/2227123924 >> 172.21.15.107:0/2227123924 conn(0x7fb7080b0c50 msgr2=0x7fb7080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.777+0000 7fb70e5c9640 1 -- 172.21.15.107:0/2227123924 shutdown_connections 2023-11-24T08:10:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.777+0000 7fb70e5c9640 1 -- 172.21.15.107:0/2227123924 wait complete. 2023-11-24T08:10:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.778+0000 7fb70e5c9640 1 Processor -- start 2023-11-24T08:10:11.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.778+0000 7fb70e5c9640 1 -- start start 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7080cbf20 0x7fb70809a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70815b440 0x7fb70809a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081573a0 con 0x7fb7080c57e0 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70809af50 con 0x7fb7080cbf20 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.779+0000 7fb70e5c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70809b230 con 0x7fb70815b440 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59168/0 (socket says 172.21.15.107:59168) 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 -- 172.21.15.107:0/1944092466 learned_addr learned my addr 172.21.15.107:0/1944092466 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 -- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70815b440 msgr2=0x7fb70809a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70815b440 0x7fb70809a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 -- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7080cbf20 msgr2=0x7fb70809a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7080cbf20 0x7fb70809a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 -- 172.21.15.107:0/1944092466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc009400 con 0x7fb7080c57e0 2023-11-24T08:10:11.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.780+0000 7fb707fff640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009870 tx=0x7fb6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.781+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f803a930 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.781+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f803aa90 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.781+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f80094d0 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.781+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f803ae40 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.781+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb708143ab0 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.782+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70804e8b0 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.783+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fb6f802f050 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.784+0000 7fb704ff9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 0x7fb6f406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.784+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb6f80a9f50 con 0x7fb7080c57e0 2023-11-24T08:10:11.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.784+0000 7fb7077fe640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 0x7fb6f406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:11.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.785+0000 7fb7077fe640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 0x7fb6f406db70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc006e80 tx=0x7fb6fc006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:11.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:11.788+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6f80768e0 con 0x7fb7080c57e0 2023-11-24T08:10:12.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.018+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb70806dc80 con 0x7fb7080c57e0 2023-11-24T08:10:12.021 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:10:12.020+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:10:12.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.023+0000 7fb704ff9640 1 -- 172.21.15.107:0/1944092466 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb6f8079f00 con 0x7fb7080c57e0 2023-11-24T08:10:12.025 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.026+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 msgr2=0x7fb6f406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.026+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 0x7fb6f406db70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc006e80 tx=0x7fb6fc006dd0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.026+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 msgr2=0x7fb708156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.026+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009870 tx=0x7fb6f80046c0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 shutdown_connections 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6f406b6b0 0x7fb6f406db70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70815b440 0x7fb70809a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7080cbf20 0x7fb70809a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 --2- 172.21.15.107:0/1944092466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7080c57e0 0x7fb708156830 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 >> 172.21.15.107:0/1944092466 conn(0x7fb7080b0c50 msgr2=0x7fb7080ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 shutdown_connections 2023-11-24T08:10:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.027+0000 7fb70e5c9640 1 -- 172.21.15.107:0/1944092466 wait complete. 2023-11-24T08:10:12.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:10:12.099 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:10:12.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.238+0000 7ffb6d83d640 1 Processor -- start 2023-11-24T08:10:12.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.238+0000 7ffb6d83d640 1 -- start start 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb68072160 0x7ffb6815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb680c49f0 0x7ffb6806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffb6806dc40 0x7ffb680cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6806e0f0 con 0x7ffb68072160 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6815ec90 con 0x7ffb6806dc40 2023-11-24T08:10:12.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb6d83d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cd530 con 0x7ffb680c49f0 2023-11-24T08:10:12.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb66ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb68072160 0x7ffb6815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb66ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb68072160 0x7ffb6815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59200/0 (socket says 172.21.15.107:59200) 2023-11-24T08:10:12.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.240+0000 7ffb66ffd640 1 -- 172.21.15.107:0/1190754061 learned_addr learned my addr 172.21.15.107:0/1190754061 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb5e7fc640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb680c49f0 0x7ffb6806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb66ffd640 1 -- 172.21.15.107:0/1190754061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb680c49f0 msgr2=0x7ffb6806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb680c49f0 0x7ffb6806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb66ffd640 1 -- 172.21.15.107:0/1190754061 >> v2:172.21.15.120:3300/0 conn(0x7ffb6806dc40 msgr2=0x7ffb680cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/1190754061 >> v2:172.21.15.120:3300/0 conn(0x7ffb6806dc40 0x7ffb680cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.241+0000 7ffb66ffd640 1 -- 172.21.15.107:0/1190754061 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680c4ed0 con 0x7ffb68072160 2023-11-24T08:10:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.242+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb68072160 0x7ffb6815e750 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffb68072a30 tx=0x7ffb5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb080149a8fc0a6e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.242+0000 7ffb667fc640 1 -- 172.21.15.107:0/1190754061 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5802ca70 con 0x7ffb68072160 2023-11-24T08:10:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.242+0000 7ffb667fc640 1 -- 172.21.15.107:0/1190754061 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5802cbd0 con 0x7ffb68072160 2023-11-24T08:10:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.242+0000 7ffb667fc640 1 -- 172.21.15.107:0/1190754061 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580343f0 con 0x7ffb68072160 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 -- 172.21.15.107:0/1190754061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb68072160 msgr2=0x7ffb6815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb68072160 0x7ffb6815e750 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffb68072a30 tx=0x7ffb5802bed0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 -- 172.21.15.107:0/1190754061 shutdown_connections 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/1190754061 >> v2:172.21.15.120:3300/0 conn(0x7ffb6806dc40 0x7ffb680cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb680c49f0 0x7ffb6806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/1190754061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb68072160 0x7ffb6815e750 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 -- 172.21.15.107:0/1190754061 >> 172.21.15.107:0/1190754061 conn(0x7ffb680b0c50 msgr2=0x7ffb680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 -- 172.21.15.107:0/1190754061 shutdown_connections 2023-11-24T08:10:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.243+0000 7ffb6d83d640 1 -- 172.21.15.107:0/1190754061 wait complete. 2023-11-24T08:10:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.245+0000 7ffb6d83d640 1 Processor -- start 2023-11-24T08:10:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.245+0000 7ffb6d83d640 1 -- start start 2023-11-24T08:10:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb6806dc40 0x7ffb681f7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 0x7ffb68158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f7880 con 0x7ffb680c49f0 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680b4710 con 0x7ffb6806dc40 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb6d83d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680b49f0 con 0x7ffb68072160 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59216/0 (socket says 172.21.15.107:59216) 2023-11-24T08:10:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb677fe640 1 -- 172.21.15.107:0/823167678 learned_addr learned my addr 172.21.15.107:0/823167678 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb5e7fc640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 0x7ffb68158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.246+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb6806dc40 0x7ffb681f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 -- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 msgr2=0x7ffb68158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 0x7ffb68158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 -- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb6806dc40 msgr2=0x7ffb681f7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb6806dc40 0x7ffb681f7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 -- 172.21.15.107:0/823167678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58009b30 con 0x7ffb680c49f0 2023-11-24T08:10:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb5e7fc640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 0x7ffb68158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb677fe640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009fd0 tx=0x7ffb6000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011830 con 0x7ffb680c49f0 2023-11-24T08:10:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb60011990 con 0x7ffb680c49f0 2023-11-24T08:10:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011d20 con 0x7ffb680c49f0 2023-11-24T08:10:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb680b4cd0 con 0x7ffb680c49f0 2023-11-24T08:10:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.247+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6809c7f0 con 0x7ffb680c49f0 2023-11-24T08:10:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.248+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804e8b0 con 0x7ffb680c49f0 2023-11-24T08:10:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.249+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7ffb6001b460 con 0x7ffb680c49f0 2023-11-24T08:10:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.250+0000 7ffb64ff9640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 0x7ffb4806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.250+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffb6008a6e0 con 0x7ffb680c49f0 2023-11-24T08:10:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.251+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 0x7ffb4806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.251+0000 7ffb66ffd640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 0x7ffb4806dba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb58035000 tx=0x7ffb58030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.255+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb60054790 con 0x7ffb680c49f0 2023-11-24T08:10:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:12.480+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffb6815dab0 con 0x7ffb680c49f0 2023-11-24T08:10:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.027+0000 7ffb64ff9640 1 -- 172.21.15.107:0/823167678 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffb60057db0 con 0x7ffb680c49f0 2023-11-24T08:10:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 msgr2=0x7ffb4806dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 0x7ffb4806dba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb58035000 tx=0x7ffb58030000 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 msgr2=0x7ffb6815c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009fd0 tx=0x7ffb6000b8e0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 shutdown_connections 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb4806b6e0 0x7ffb4806dba0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb680c49f0 0x7ffb6815c520 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb68072160 0x7ffb68158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 --2- 172.21.15.107:0/823167678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb6806dc40 0x7ffb681f7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.030+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 >> 172.21.15.107:0/823167678 conn(0x7ffb680b0c50 msgr2=0x7ffb68159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.031+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 shutdown_connections 2023-11-24T08:10:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.031+0000 7ffb6d83d640 1 -- 172.21.15.107:0/823167678 wait complete. 2023-11-24T08:10:13.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:10:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.263+0000 7f5abe133640 1 Processor -- start 2023-11-24T08:10:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.263+0000 7f5abe133640 1 -- start start 2023-11-24T08:10:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5abe133640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ab815c7f0 0x7f5ab806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5abe133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5ab80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5abe133640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80cece0 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5abe133640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab8156010 con 0x7f5ab815c7f0 2023-11-24T08:10:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5abe133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab806ce30 con 0x7f5ab80cf450 2023-11-24T08:10:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5ab7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5ab80cdfb0 0x7f5ab80ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33224/0 (socket says 172.21.15.107:33224) 2023-11-24T08:10:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5ab7fff640 1 -- 172.21.15.107:0/2987181968 learned_addr learned my addr 172.21.15.107:0/2987181968 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:13.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5aaffff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5ab80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5ab77fe640 1 --2- 172.21.15.107:0/2987181968 >> v2:172.21.15.120:3300/0 conn(0x7f5ab815c7f0 0x7f5ab806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214527459 0 0) 0x7f5ab80cece0 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.265+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa4003520 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699162311 0 0) 0x7f5aa4003520 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 msgr2=0x7f5ab80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5ab80cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 >> v2:172.21.15.120:3300/0 conn(0x7f5ab815c7f0 msgr2=0x7f5ab806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 --2- 172.21.15.107:0/2987181968 >> v2:172.21.15.120:3300/0 conn(0x7f5ab815c7f0 0x7f5ab806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab806d040 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5ab00019f0 con 0x7f5ab80cdfb0 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 >> v1:172.21.15.107:6789/0 conn(0x7f5ab80cdfb0 legacy=0x7f5ab80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5aa4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa4003f40 0x7f5aa40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5aa4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8156010 con 0x7f5aa4004d00 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab806ce30 con 0x7f5aa4003f40 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa400ba30 con 0x7f5ab80cf450 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.266+0000 7f5ab7fff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5aa4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab77fe640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5aa4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5aaffff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa4003f40 0x7f5aa40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab7fff640 1 -- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 msgr2=0x7f5aa4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab7fff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab80cf450 0x7f5aa4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab7fff640 1 -- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa4003f40 msgr2=0x7f5aa40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab7fff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa4003f40 0x7f5aa40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.267+0000 7f5ab7fff640 1 -- 172.21.15.107:0/2987181968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4003520 con 0x7f5aa4004d00 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.268+0000 7f5ab7fff640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5aa4005180 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000a790 tx=0x7f5aa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addbfc5053b6e04c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.268+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0034630 con 0x7f5aa4004d00 2023-11-24T08:10:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.268+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab002f070 con 0x7f5aa4004d00 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.268+0000 7f5ab6ffd640 1 -- 172.21.15.107:0/2987181968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0034a30 con 0x7f5aa4004d00 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 -- 172.21.15.107:0/2987181968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 msgr2=0x7f5aa4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 --2- 172.21.15.107:0/2987181968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5aa4005180 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000a790 tx=0x7f5aa40057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5ab7fff640 1 -- 172.21.15.107:0/2987181968 reap_dead start 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 -- 172.21.15.107:0/2987181968 shutdown_connections 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 -- 172.21.15.107:0/2987181968 >> 172.21.15.107:0/2987181968 conn(0x7f5ab806a920 msgr2=0x7f5ab806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 -- 172.21.15.107:0/2987181968 shutdown_connections 2023-11-24T08:10:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.269+0000 7f5abe133640 1 -- 172.21.15.107:0/2987181968 wait complete. 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.271+0000 7f5abe133640 1 Processor -- start 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.271+0000 7f5abe133640 1 -- start start 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab814c3f0 0x7f5ab8099470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab815ef30 0x7f5ab815f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab815f900 con 0x7f5aa4004d00 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab81f7540 con 0x7f5ab815ef30 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5abe133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8156480 con 0x7f5ab814c3f0 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59234/0 (socket says 172.21.15.107:59234) 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 -- 172.21.15.107:0/2783457247 learned_addr learned my addr 172.21.15.107:0/2783457247 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 -- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab814c3f0 msgr2=0x7f5ab8099470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab814c3f0 0x7f5ab8099470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 -- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab815ef30 msgr2=0x7f5ab815f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab7fff640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab815ef30 0x7f5ab815f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab815ef30 0x7f5ab815f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.272+0000 7f5ab77fe640 1 -- 172.21.15.107:0/2783457247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8156760 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5ab77fe640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80116d0 tx=0x7f5aa8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa800f9c0 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa800fb20 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa800fe90 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab8156a40 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.273+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab8157000 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.274+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab8067470 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.275+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f5aa8015460 con 0x7f5aa4004d00 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.276+0000 7f5ab4ff9640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 0x7f5a9806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.276+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5aa8088600 con 0x7f5aa4004d00 2023-11-24T08:10:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.276+0000 7f5aaffff640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 0x7f5a9806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.277+0000 7f5aaffff640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 0x7f5a9806db70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0006fd0 tx=0x7f5aa000d280 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.280+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5aa8054f90 con 0x7f5aa4004d00 2023-11-24T08:10:13.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:13.468+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5ab80cd730 con 0x7f5aa4004d00 2023-11-24T08:10:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.037+0000 7f5ab4ff9640 1 -- 172.21.15.107:0/2783457247 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5aa80220d0 con 0x7f5aa4004d00 2023-11-24T08:10:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:10:14.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 msgr2=0x7f5a9806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 0x7f5a9806db70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0006fd0 tx=0x7f5aa000d280 comp rx=0 tx=0).stop 2023-11-24T08:10:14.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 msgr2=0x7f5ab81f7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80116d0 tx=0x7f5aa8011b90 comp rx=0 tx=0).stop 2023-11-24T08:10:14.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 shutdown_connections 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9806b6b0 0x7f5a9806db70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab815ef30 0x7f5ab815f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab814c3f0 0x7f5ab8099470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 --2- 172.21.15.107:0/2783457247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa4004d00 0x7f5ab81f7000 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 >> 172.21.15.107:0/2783457247 conn(0x7f5ab806a920 msgr2=0x7f5ab806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 shutdown_connections 2023-11-24T08:10:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.041+0000 7f5abe133640 1 -- 172.21.15.107:0/2783457247 wait complete. 2023-11-24T08:10:14.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:10:14.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.279+0000 7f86ae4df640 1 Processor -- start 2023-11-24T08:10:14.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.279+0000 7f86ae4df640 1 -- start start 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ae4df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a8160b00 0x7f86a80d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ae4df640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86a80d12e0 0x7f86a80cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ae4df640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a815ea30 con 0x7f86a80d0120 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ae4df640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80d0590 con 0x7f86a8160b00 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ae4df640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a815de20 con 0x7f86a80d12e0 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ad4dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a8160b00 0x7f86a80d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ad4dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a8160b00 0x7f86a80d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59236/0 (socket says 172.21.15.107:59236) 2023-11-24T08:10:14.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.281+0000 7f86ad4dd640 1 -- 172.21.15.107:0/2852818340 learned_addr learned my addr 172.21.15.107:0/2852818340 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:14.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.282+0000 7f86accdc640 1 --2- 172.21.15.107:0/2852818340 >> v2:172.21.15.120:3300/0 conn(0x7f86a80d12e0 0x7f86a80cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:14.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.282+0000 7f86ad4dd640 1 -- 172.21.15.107:0/2852818340 >> v2:172.21.15.120:3300/0 conn(0x7f86a80d12e0 msgr2=0x7f86a80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:14.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.282+0000 7f86ad4dd640 1 --2- 172.21.15.107:0/2852818340 >> v2:172.21.15.120:3300/0 conn(0x7f86a80d12e0 0x7f86a80cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.282+0000 7f86ad4dd640 1 -- 172.21.15.107:0/2852818340 >> v1:172.21.15.107:6789/0 conn(0x7f86a80d0120 legacy=0x7f86a815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.282+0000 7f86ad4dd640 1 -- 172.21.15.107:0/2852818340 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a815e100 con 0x7f86a8160b00 2023-11-24T08:10:14.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.283+0000 7f86ad4dd640 1 --2- 172.21.15.107:0/2852818340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a8160b00 0x7f86a80d0da0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86a808bad0 tx=0x7f869c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c48f127807f57c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:14.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.283+0000 7f86a7fff640 1 -- 172.21.15.107:0/2852818340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c02cb20 con 0x7f86a8160b00 2023-11-24T08:10:14.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.283+0000 7f86a7fff640 1 -- 172.21.15.107:0/2852818340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c02cc80 con 0x7f86a8160b00 2023-11-24T08:10:14.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.283+0000 7f86a7fff640 1 -- 172.21.15.107:0/2852818340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0343f0 con 0x7f86a8160b00 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 -- 172.21.15.107:0/2852818340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a8160b00 msgr2=0x7f86a80d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 --2- 172.21.15.107:0/2852818340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a8160b00 0x7f86a80d0da0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86a808bad0 tx=0x7f869c02bf80 comp rx=0 tx=0).stop 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 -- 172.21.15.107:0/2852818340 shutdown_connections 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 --2- 172.21.15.107:0/2852818340 >> v2:172.21.15.120:3300/0 conn(0x7f86a80d12e0 0x7f86a80cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 --2- 172.21.15.107:0/2852818340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a8160b00 0x7f86a80d0da0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 -- 172.21.15.107:0/2852818340 >> 172.21.15.107:0/2852818340 conn(0x7f86a809c510 msgr2=0x7f86a80d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:14.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 -- 172.21.15.107:0/2852818340 shutdown_connections 2023-11-24T08:10:14.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.284+0000 7f86ae4df640 1 -- 172.21.15.107:0/2852818340 wait complete. 2023-11-24T08:10:14.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.286+0000 7f86ae4df640 1 Processor -- start 2023-11-24T08:10:14.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.286+0000 7f86ae4df640 1 -- start start 2023-11-24T08:10:14.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a80d0120 0x7f86a806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8160b00 0x7f86a80b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80b8a90 con 0x7f86a80d12e0 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80b7e80 con 0x7f86a80d0120 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86ae4df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80b8160 con 0x7f86a8160b00 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86accdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86accdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59248/0 (socket says 172.21.15.107:59248) 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.287+0000 7f86accdc640 1 -- 172.21.15.107:0/3538927015 learned_addr learned my addr 172.21.15.107:0/3538927015 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86ad4dd640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a80d0120 0x7f86a806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 -- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8160b00 msgr2=0x7f86a80b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8160b00 0x7f86a80b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 -- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a80d0120 msgr2=0x7f86a806bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:14.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a80d0120 0x7f86a806bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:14.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 -- 172.21.15.107:0/3538927015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c009b30 con 0x7f86a80d12e0 2023-11-24T08:10:14.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86accdc640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f869400c7f0 tx=0x7f869400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:14.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694004220 con 0x7f86a80d12e0 2023-11-24T08:10:14.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86940043a0 con 0x7f86a80d12e0 2023-11-24T08:10:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a80b8440 con 0x7f86a80d12e0 2023-11-24T08:10:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694007860 con 0x7f86a80d12e0 2023-11-24T08:10:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.288+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a815abb0 con 0x7f86a80d12e0 2023-11-24T08:10:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.289+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a80b8c40 con 0x7f86a80d12e0 2023-11-24T08:10:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.290+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f8694007bd0 con 0x7f86a80d12e0 2023-11-24T08:10:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.291+0000 7f86a5ffb640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 0x7f868c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.291+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f86940880d0 con 0x7f86a80d12e0 2023-11-24T08:10:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.292+0000 7f86ad4dd640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 0x7f868c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.292+0000 7f86ad4dd640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 0x7f868c06dba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f869c035000 tx=0x7f869c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:14.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.296+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86a80b8c40 con 0x7f86a80d12e0 2023-11-24T08:10:14.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:14.490+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86a804e840 con 0x7f86a80d12e0 2023-11-24T08:10:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.050+0000 7f86a5ffb640 1 -- 172.21.15.107:0/3538927015 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 169+0+0 (secure 0 0 0) 0x7f86940548f0 con 0x7f86a80d12e0 2023-11-24T08:10:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:10:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 msgr2=0x7f868c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 0x7f868c06dba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f869c035000 tx=0x7f869c030000 comp rx=0 tx=0).stop 2023-11-24T08:10:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 msgr2=0x7f86a80b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f869400c7f0 tx=0x7f869400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 shutdown_connections 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868c06b6e0 0x7f868c06dba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8160b00 0x7f86a80b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a80d12e0 0x7f86a80b72c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 --2- 172.21.15.107:0/3538927015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a80d0120 0x7f86a806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 >> 172.21.15.107:0/3538927015 conn(0x7f86a809c510 msgr2=0x7f86a815c610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 shutdown_connections 2023-11-24T08:10:15.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.054+0000 7f86ae4df640 1 -- 172.21.15.107:0/3538927015 wait complete. 2023-11-24T08:10:15.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:10:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.293+0000 7fc7a0a85640 1 Processor -- start 2023-11-24T08:10:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.293+0000 7fc7a0a85640 1 -- start start 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc7a0a85640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc79c0d1a00 0x7fc79c0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc7a0a85640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c0b41b0 con 0x7fc79c160dd0 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc7a0a85640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c0b4490 con 0x7fc79c1609f0 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc7a0a85640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c0b0090 con 0x7fc79c0d1a00 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc793fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc79c0d1a00 0x7fc79c0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc793fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc79c0d1a00 0x7fc79c0b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59264/0 (socket says 172.21.15.107:59264) 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc793fff640 1 -- 172.21.15.107:0/1344972817 learned_addr learned my addr 172.21.15.107:0/1344972817 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.294+0000 7fc79bfff640 1 --1- 172.21.15.107:0/1344972817 >> v1:172.21.15.107:6789/0 conn(0x7fc79c160dd0 0x7fc79c0b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33236/0 (socket says 172.21.15.107:33236) 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.295+0000 7fc793fff640 1 -- 172.21.15.107:0/1344972817 >> v1:172.21.15.120:6789/0 conn(0x7fc79c1609f0 legacy=0x7fc79c0d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.295+0000 7fc793fff640 1 -- 172.21.15.107:0/1344972817 >> v1:172.21.15.107:6789/0 conn(0x7fc79c160dd0 legacy=0x7fc79c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.295+0000 7fc793fff640 1 -- 172.21.15.107:0/1344972817 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0b0370 con 0x7fc79c0d1a00 2023-11-24T08:10:15.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.295+0000 7fc79affd640 1 -- 172.21.15.107:0/1344972817 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104978065 0 0) 0x7fc79c0b41b0 con 0x7fc79c160dd0 2023-11-24T08:10:15.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.296+0000 7fc793fff640 1 --2- 172.21.15.107:0/1344972817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c0b2be0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc784009870 tx=0x7fc78402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a3e26ae03face5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.296+0000 7fc79affd640 1 -- 172.21.15.107:0/1344972817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784030860 con 0x7fc79c0d1a00 2023-11-24T08:10:15.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.296+0000 7fc79affd640 1 -- 172.21.15.107:0/1344972817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7840309c0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.296+0000 7fc79affd640 1 -- 172.21.15.107:0/1344972817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784030d50 con 0x7fc79c0d1a00 2023-11-24T08:10:15.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 -- 172.21.15.107:0/1344972817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 msgr2=0x7fc79c0b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/1344972817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c0b2be0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc784009870 tx=0x7fc78402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 -- 172.21.15.107:0/1344972817 shutdown_connections 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/1344972817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c0b2be0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 -- 172.21.15.107:0/1344972817 >> 172.21.15.107:0/1344972817 conn(0x7fc79c09c5b0 msgr2=0x7fc79c0d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 -- 172.21.15.107:0/1344972817 shutdown_connections 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.297+0000 7fc7a0a85640 1 -- 172.21.15.107:0/1344972817 wait complete. 2023-11-24T08:10:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.299+0000 7fc7a0a85640 1 Processor -- start 2023-11-24T08:10:15.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.299+0000 7fc7a0a85640 1 -- start start 2023-11-24T08:10:15.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 0x7fc79c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc79c160dd0 0x7fc79c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0a0bb0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c150ec0 con 0x7fc79c1609f0 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc7a0a85640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c15d2b0 con 0x7fc79c160dd0 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc79b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc79b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59278/0 (socket says 172.21.15.107:59278) 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.300+0000 7fc79b7fe640 1 -- 172.21.15.107:0/952643534 learned_addr learned my addr 172.21.15.107:0/952643534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc793fff640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 0x7fc79c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79bfff640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc79c160dd0 0x7fc79c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 -- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc79c160dd0 msgr2=0x7fc79c0c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc79c160dd0 0x7fc79c0c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 -- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 msgr2=0x7fc79c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 0x7fc79c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 -- 172.21.15.107:0/952643534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7840094d0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc793fff640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 0x7fc79c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc79b7fe640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c430 tx=0x7fc78c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c011790 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c0118f0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c09ffa0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c011ce0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.301+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c0a0560 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.303+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c0a0d20 con 0x7fc79c0d1a00 2023-11-24T08:10:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.303+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fc78c01c460 con 0x7fc79c0d1a00 2023-11-24T08:10:15.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.304+0000 7fc798ff9640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 0x7fc77c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.304+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc78c088010 con 0x7fc79c0d1a00 2023-11-24T08:10:15.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.305+0000 7fc793fff640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 0x7fc77c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.305+0000 7fc793fff640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 0x7fc77c06db70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc784036000 tx=0x7fc78403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.309+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc78c054980 con 0x7fc79c0d1a00 2023-11-24T08:10:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.498+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc79c04e840 con 0x7fc79c0d1a00 2023-11-24T08:10:15.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.504+0000 7fc798ff9640 1 -- 172.21.15.107:0/952643534 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc78c057fa0 con 0x7fc79c0d1a00 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.506+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 msgr2=0x7fc77c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.506+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 0x7fc77c06db70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc784036000 tx=0x7fc78403b040 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.506+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 msgr2=0x7fc79c14d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.506+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c430 tx=0x7fc78c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 shutdown_connections 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc77c06b6b0 0x7fc77c06db70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc79c160dd0 0x7fc79c0c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc79c1609f0 0x7fc79c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 --2- 172.21.15.107:0/952643534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc79c0d1a00 0x7fc79c14d850 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 >> 172.21.15.107:0/952643534 conn(0x7fc79c09c5b0 msgr2=0x7fc79c0d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:15.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 shutdown_connections 2023-11-24T08:10:15.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.507+0000 7fc7a0a85640 1 -- 172.21.15.107:0/952643534 wait complete. 2023-11-24T08:10:15.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:10:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.739+0000 7f26f6c39640 1 Processor -- start 2023-11-24T08:10:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.739+0000 7f26f6c39640 1 -- start start 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f6c39640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f26f0141b90 0x7f26f00b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f6c39640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f00b14f0 con 0x7f26f00b1090 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f6c39640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f01450d0 con 0x7f26f0071fe0 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f6c39640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f01422b0 con 0x7f26f0141b90 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f5436640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f26f00b1090 0x7f26f00718d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33248/0 (socket says 172.21.15.107:33248) 2023-11-24T08:10:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.741+0000 7f26f5436640 1 -- 172.21.15.107:0/883884215 learned_addr learned my addr 172.21.15.107:0/883884215 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:15.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f5c37640 1 --2- 172.21.15.107:0/883884215 >> v2:172.21.15.120:3300/0 conn(0x7f26f0141b90 0x7f26f00b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794099378 0 0) 0x7f26f00b14f0 con 0x7f26f00b1090 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc003520 con 0x7f26f00b1090 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611957652 0 0) 0x7f26dc003520 con 0x7f26f00b1090 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 >> v2:172.21.15.120:3300/0 conn(0x7f26f0141b90 msgr2=0x7f26f00b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 --2- 172.21.15.107:0/883884215 >> v2:172.21.15.120:3300/0 conn(0x7f26f0141b90 0x7f26f00b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 >> v1:172.21.15.120:6789/0 conn(0x7f26f0071fe0 legacy=0x7f26f0142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00725e0 con 0x7f26f00b1090 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f26e0001670 con 0x7f26f00b1090 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 >> v1:172.21.15.107:6789/0 conn(0x7f26f00b1090 legacy=0x7f26f00718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc003970 0x7f26dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.742+0000 7f26f4c35640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26dc0047c0 0x7f26dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f4c35640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 0x7f26dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f01422b0 con 0x7f26dc005180 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc00c120 con 0x7f26dc003970 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc00c2e0 con 0x7f26dc0047c0 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 0x7f26dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f5c37640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc003970 0x7f26dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f5436640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26dc0047c0 0x7f26dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 -- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26dc0047c0 msgr2=0x7f26dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26dc0047c0 0x7f26dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 -- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc003970 msgr2=0x7f26dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc003970 0x7f26dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.743+0000 7f26f6438640 1 -- 172.21.15.107:0/883884215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc003520 con 0x7f26dc005180 2023-11-24T08:10:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.745+0000 7f26f6438640 1 -- 172.21.15.107:0/883884215 reap_dead start 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.745+0000 7f26f6438640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 0x7f26dc00b980 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f26e8001e50 tx=0x7f26dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850517478e64a528 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.745+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8003b70 con 0x7f26dc005180 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.745+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8003d10 con 0x7f26dc005180 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.745+0000 7f26f4c35640 1 -- 172.21.15.107:0/883884215 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8005310 con 0x7f26dc005180 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 -- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 msgr2=0x7f26dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 0x7f26dc00b980 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f26e8001e50 tx=0x7f26dc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 -- 172.21.15.107:0/883884215 shutdown_connections 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 --2- 172.21.15.107:0/883884215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26dc005180 0x7f26dc00b980 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 -- 172.21.15.107:0/883884215 >> 172.21.15.107:0/883884215 conn(0x7f26f006a500 msgr2=0x7f26f006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 -- 172.21.15.107:0/883884215 shutdown_connections 2023-11-24T08:10:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.746+0000 7f26f6c39640 1 -- 172.21.15.107:0/883884215 wait complete. 2023-11-24T08:10:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.748+0000 7f26f6c39640 1 Processor -- start 2023-11-24T08:10:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.748+0000 7f26f6c39640 1 -- start start 2023-11-24T08:10:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc005180 0x7f26f0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26f0140920 0x7f26f0154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f0157400 con 0x7f26f0156a40 2023-11-24T08:10:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f015c3c0 con 0x7f26dc005180 2023-11-24T08:10:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.749+0000 7f26f6c39640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f0152760 con 0x7f26f0140920 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59308/0 (socket says 172.21.15.107:59308) 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 -- 172.21.15.107:0/2903911901 learned_addr learned my addr 172.21.15.107:0/2903911901 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f5c37640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc005180 0x7f26f0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f5436640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26f0140920 0x7f26f0154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 -- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26f0140920 msgr2=0x7f26f0154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26f0140920 0x7f26f0154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 -- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc005180 msgr2=0x7f26f0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc005180 0x7f26f0152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.750+0000 7f26f6438640 1 -- 172.21.15.107:0/2903911901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f0152a40 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26f6438640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f26e8001f70 tx=0x7f26e80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8006d50 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8007030 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e80073a0 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f0152d20 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.751+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f0154970 con 0x7f26f0156a40 2023-11-24T08:10:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.752+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f015c220 con 0x7f26f0156a40 2023-11-24T08:10:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.753+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f26e8007500 con 0x7f26f0156a40 2023-11-24T08:10:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.754+0000 7f26e67fc640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 0x7f26d406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.754+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26e80a9b50 con 0x7f26f0156a40 2023-11-24T08:10:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.754+0000 7f26f5c37640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 0x7f26d406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.755+0000 7f26f5c37640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 0x7f26d406db70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0098d0 tx=0x7f26ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.758+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26f015c220 con 0x7f26f0156a40 2023-11-24T08:10:15.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.949+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f26f015c220 con 0x7f26f0156a40 2023-11-24T08:10:15.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.954+0000 7f26e67fc640 1 -- 172.21.15.107:0/2903911901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f26f015c220 con 0x7f26f0156a40 2023-11-24T08:10:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 msgr2=0x7f26d406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 0x7f26d406db70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0098d0 tx=0x7f26ec008040 comp rx=0 tx=0).stop 2023-11-24T08:10:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 msgr2=0x7f26f0156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f26e8001f70 tx=0x7f26e80014a0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 shutdown_connections 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406b6b0 0x7f26d406db70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f0156a40 0x7f26f0156ec0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26f0140920 0x7f26f0154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 --2- 172.21.15.107:0/2903911901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc005180 0x7f26f0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.957+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 >> 172.21.15.107:0/2903911901 conn(0x7f26f006a500 msgr2=0x7f26f0071890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.958+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 shutdown_connections 2023-11-24T08:10:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:15.958+0000 7f26f6c39640 1 -- 172.21.15.107:0/2903911901 wait complete. 2023-11-24T08:10:15.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled' 2023-11-24T08:10:16.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.176+0000 7f9b55b2c640 1 Processor -- start 2023-11-24T08:10:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.177+0000 7f9b55b2c640 1 -- start start 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b55b2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 0x7f9b500b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b55b2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b500711e0 con 0x7f9b5015b5e0 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b55b2c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b500cdc80 con 0x7f9b500abf50 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b55b2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500705d0 con 0x7f9b500cd5b0 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b4f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9b5015b5e0 0x7f9b500ccea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33262/0 (socket says 172.21.15.107:33262) 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.178+0000 7f9b4f7fe640 1 -- 172.21.15.107:0/153819311 learned_addr learned my addr 172.21.15.107:0/153819311 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4effd640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 0x7f9b500b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183071182 0 0) 0x7f9b500711e0 con 0x7f9b5015b5e0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c003520 con 0x7f9b5015b5e0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406065983 0 0) 0x7f9b3c003520 con 0x7f9b5015b5e0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 msgr2=0x7f9b500b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 0x7f9b500b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 >> v1:172.21.15.120:6789/0 conn(0x7f9b500abf50 legacy=0x7f9b50071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500708b0 con 0x7f9b5015b5e0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9b400019f0 con 0x7f9b5015b5e0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 >> v1:172.21.15.107:6789/0 conn(0x7f9b5015b5e0 legacy=0x7f9b500ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b3c0039a0 0x7f9b3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c0047c0 0x7f9b3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.179+0000 7f9b4e7fc640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 0x7f9b3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500705d0 con 0x7f9b3c0039a0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c00ba30 con 0x7f9b3c0047c0 2023-11-24T08:10:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c00bbf0 con 0x7f9b500cd5b0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b3c0039a0 0x7f9b3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4effd640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c0047c0 0x7f9b3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 -- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 msgr2=0x7f9b3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b500cd5b0 0x7f9b3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 -- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c0047c0 msgr2=0x7f9b3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c0047c0 0x7f9b3c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.180+0000 7f9b4f7fe640 1 -- 172.21.15.107:0/153819311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c003520 con 0x7f9b3c0039a0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.181+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b3c0039a0 0x7f9b3c003dc0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000a800 tx=0x7f9b3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c5f291e25bebdc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.181+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400072d0 con 0x7f9b3c0039a0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.181+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4002f070 con 0x7f9b3c0039a0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.181+0000 7f9b4e7fc640 1 -- 172.21.15.107:0/153819311 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400076b0 con 0x7f9b3c0039a0 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 -- 172.21.15.107:0/153819311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b3c0039a0 msgr2=0x7f9b3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/153819311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b3c0039a0 0x7f9b3c003dc0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000a800 tx=0x7f9b3c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b4ffff640 1 -- 172.21.15.107:0/153819311 reap_dead start 2023-11-24T08:10:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 -- 172.21.15.107:0/153819311 shutdown_connections 2023-11-24T08:10:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 -- 172.21.15.107:0/153819311 >> 172.21.15.107:0/153819311 conn(0x7f9b500b0c70 msgr2=0x7f9b500b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 -- 172.21.15.107:0/153819311 shutdown_connections 2023-11-24T08:10:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.182+0000 7f9b55b2c640 1 -- 172.21.15.107:0/153819311 wait complete. 2023-11-24T08:10:16.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.184+0000 7f9b55b2c640 1 Processor -- start 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.184+0000 7f9b55b2c640 1 -- start start 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c0039a0 0x7f9b501ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b501576c0 0x7f9b501f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5015a310 con 0x7f9b50159920 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5015d970 con 0x7f9b501576c0 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b55b2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b50157b80 con 0x7f9b3c0039a0 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59316/0 (socket says 172.21.15.107:59316) 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.185+0000 7f9b4ffff640 1 -- 172.21.15.107:0/1823956775 learned_addr learned my addr 172.21.15.107:0/1823956775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4effd640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b501576c0 0x7f9b501f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 -- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c0039a0 msgr2=0x7f9b501ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c0039a0 0x7f9b501ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 -- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b501576c0 msgr2=0x7f9b501f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b501576c0 0x7f9b501f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 -- 172.21.15.107:0/1823956775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b50157e60 con 0x7f9b50159920 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b4ffff640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9b500aea30 tx=0x7f9b4800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b480072e0 con 0x7f9b50159920 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b48007440 con 0x7f9b50159920 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48004020 con 0x7f9b50159920 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.186+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b50158140 con 0x7f9b50159920 2023-11-24T08:10:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.187+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b50158700 con 0x7f9b50159920 2023-11-24T08:10:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.188+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5004e8b0 con 0x7f9b50159920 2023-11-24T08:10:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.189+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f9b480041a0 con 0x7f9b50159920 2023-11-24T08:10:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.190+0000 7f9b47fff640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 0x7f9b3006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.190+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9b48087150 con 0x7f9b50159920 2023-11-24T08:10:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.190+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 0x7f9b3006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.191+0000 7f9b4f7fe640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 0x7f9b3006db70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b40034000 tx=0x7f9b4002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.194+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b48053ae0 con 0x7f9b50159920 2023-11-24T08:10:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:16.454+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled"]} v 0) v1 -- 0x7f9b500ae200 con 0x7f9b50159920 2023-11-24T08:10:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.056+0000 7f9b47fff640 1 -- 172.21.15.107:0/1823956775 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_cancelled"]}]=0 v284) v1 ==== 176+0+0 (secure 0 0 0) 0x7f9b48057100 con 0x7f9b50159920 2023-11-24T08:10:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 msgr2=0x7f9b3006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 0x7f9b3006db70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b40034000 tx=0x7f9b4002f000 comp rx=0 tx=0).stop 2023-11-24T08:10:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 msgr2=0x7f9b50159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9b500aea30 tx=0x7f9b4800c820 comp rx=0 tx=0).stop 2023-11-24T08:10:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 shutdown_connections 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9b3006b6b0 0x7f9b3006db70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b50159920 0x7f9b50159dd0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b501576c0 0x7f9b501f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 --2- 172.21.15.107:0/1823956775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c0039a0 0x7f9b501ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.059+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 >> 172.21.15.107:0/1823956775 conn(0x7f9b500b0c70 msgr2=0x7f9b50155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.060+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 shutdown_connections 2023-11-24T08:10:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.060+0000 7f9b55b2c640 1 -- 172.21.15.107:0/1823956775 wait complete. 2023-11-24T08:10:17.082 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is cancelled during pending state and validate the reason ... ok 2023-11-24T08:10:17.082 INFO:tasks.cephfs_test_runner:Starting test: test_clone_failure_status_pending_in_progress_complete (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ensure failure status is not shown when clone is not in failed/cancelled state 2023-11-24T08:10:17.082 INFO:tasks.cephfs_test_runner:test_clone_failure_status_pending_in_progress_complete (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:10:17.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:17.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.287+0000 7eff6569b640 1 Processor -- start 2023-11-24T08:10:17.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.287+0000 7eff6569b640 1 -- start start 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.289+0000 7eff6569b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff601461d0 0x7eff600b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.289+0000 7eff6569b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff600b3a60 0x7eff60167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff6569b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6006d180 con 0x7eff600b4fe0 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff6569b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6006ba70 con 0x7eff601461d0 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff6569b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6006bd50 con 0x7eff600b3a60 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff601461d0 0x7eff600b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff601461d0 0x7eff600b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59326/0 (socket says 172.21.15.107:59326) 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 -- 172.21.15.107:0/1572337376 learned_addr learned my addr 172.21.15.107:0/1572337376 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 -- 172.21.15.107:0/1572337376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff600b3a60 msgr2=0x7eff60167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 --2- 172.21.15.107:0/1572337376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff600b3a60 0x7eff60167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 -- 172.21.15.107:0/1572337376 >> v1:172.21.15.120:6789/0 conn(0x7eff600b4fe0 legacy=0x7eff600b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.290+0000 7eff5effd640 1 -- 172.21.15.107:0/1572337376 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6006c030 con 0x7eff601461d0 2023-11-24T08:10:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.291+0000 7eff5effd640 1 --2- 172.21.15.107:0/1572337376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff601461d0 0x7eff600b4aa0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7eff500098a0 tx=0x7eff5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119d6b71a57c7707 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff5e7fc640 1 -- 172.21.15.107:0/1572337376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50031640 con 0x7eff601461d0 2023-11-24T08:10:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff5e7fc640 1 -- 172.21.15.107:0/1572337376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff500317a0 con 0x7eff601461d0 2023-11-24T08:10:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff5e7fc640 1 -- 172.21.15.107:0/1572337376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50031b30 con 0x7eff601461d0 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 -- 172.21.15.107:0/1572337376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff601461d0 msgr2=0x7eff600b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 --2- 172.21.15.107:0/1572337376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff601461d0 0x7eff600b4aa0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7eff500098a0 tx=0x7eff5002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 -- 172.21.15.107:0/1572337376 shutdown_connections 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 --2- 172.21.15.107:0/1572337376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff600b3a60 0x7eff60167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 --2- 172.21.15.107:0/1572337376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff601461d0 0x7eff600b4aa0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 -- 172.21.15.107:0/1572337376 >> 172.21.15.107:0/1572337376 conn(0x7eff60061340 msgr2=0x7eff60061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 -- 172.21.15.107:0/1572337376 shutdown_connections 2023-11-24T08:10:17.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.292+0000 7eff6569b640 1 -- 172.21.15.107:0/1572337376 wait complete. 2023-11-24T08:10:17.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.294+0000 7eff6569b640 1 Processor -- start 2023-11-24T08:10:17.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.294+0000 7eff6569b640 1 -- start start 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff600b3a60 0x7eff600aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 0x7eff600cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff600aa790 con 0x7eff600b4fe0 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff600a91b0 con 0x7eff600b3a60 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.295+0000 7eff6569b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff600a9490 con 0x7eff601461d0 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59334/0 (socket says 172.21.15.107:59334) 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 -- 172.21.15.107:0/601541440 learned_addr learned my addr 172.21.15.107:0/601541440 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff5f7fe640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 0x7eff600cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff5effd640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff600b3a60 0x7eff600aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 -- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 msgr2=0x7eff600cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 0x7eff600cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 -- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff600b3a60 msgr2=0x7eff600aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff600b3a60 0x7eff600aa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff57fff640 1 -- 172.21.15.107:0/601541440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff500094d0 con 0x7eff600b4fe0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.296+0000 7eff5f7fe640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 0x7eff600cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff57fff640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c410 tx=0x7eff4800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff480035f0 con 0x7eff600b4fe0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff48003750 con 0x7eff600b4fe0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48003b40 con 0x7eff600b4fe0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff60146730 con 0x7eff600b4fe0 2023-11-24T08:10:17.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.297+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff600a9a50 con 0x7eff600b4fe0 2023-11-24T08:10:17.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.298+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff60067470 con 0x7eff600b4fe0 2023-11-24T08:10:17.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.299+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7eff48003ca0 con 0x7eff600b4fe0 2023-11-24T08:10:17.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.300+0000 7eff577fe640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 0x7eff4006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.300+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7eff480872d0 con 0x7eff600b4fe0 2023-11-24T08:10:17.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.300+0000 7eff5effd640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 0x7eff4006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.301+0000 7eff5effd640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 0x7eff4006dba0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7eff50036000 tx=0x7eff5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.304+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff48053c00 con 0x7eff600b4fe0 2023-11-24T08:10:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.493+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff6006f450 con 0x7eff600b4fe0 2023-11-24T08:10:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.495+0000 7eff577fe640 1 -- 172.21.15.107:0/601541440 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+12988 (secure 0 0 0) 0x7eff48057220 con 0x7eff600b4fe0 2023-11-24T08:10:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:17.498 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":69,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:15.038370+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6836/3235590318":"2023-11-25T08:10:12.020542+0000","172.21.15.107:6834/3235590318":"2023-11-25T08:10:12.020542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 msgr2=0x7eff4006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 0x7eff4006dba0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7eff50036000 tx=0x7eff5003c040 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 msgr2=0x7eff60141430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c410 tx=0x7eff4800c8d0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 shutdown_connections 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff4006b6e0 0x7eff4006dba0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff601461d0 0x7eff600cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff600b4fe0 0x7eff60141430 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 --2- 172.21.15.107:0/601541440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff600b3a60 0x7eff600aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.498+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 >> 172.21.15.107:0/601541440 conn(0x7eff60061340 msgr2=0x7eff6015e970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.499+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 shutdown_connections 2023-11-24T08:10:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.499+0000 7eff6569b640 1 -- 172.21.15.107:0/601541440 wait complete. 2023-11-24T08:10:17.520 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:10:17.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:10:17.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.720+0000 7f37cbb78640 1 Processor -- start 2023-11-24T08:10:17.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.720+0000 7f37cbb78640 1 -- start start 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37cbb78640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37c40cc0f0 0x7f37c40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37cbb78640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c415eea0 con 0x7f37c415b440 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37cbb78640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c40abf30 con 0x7f37c40c57e0 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37cbb78640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c415b820 con 0x7f37c40cc0f0 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37c9914640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37c415b440 0x7f37c40cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33276/0 (socket says 172.21.15.107:33276) 2023-11-24T08:10:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.722+0000 7f37c9914640 1 -- 172.21.15.107:0/684538428 learned_addr learned my addr 172.21.15.107:0/684538428 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c9113640 1 --2- 172.21.15.107:0/684538428 >> v2:172.21.15.120:3300/0 conn(0x7f37c40cc0f0 0x7f37c40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584045646 0 0) 0x7f37c415eea0 con 0x7f37c415b440 2023-11-24T08:10:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b0003520 con 0x7f37c415b440 2023-11-24T08:10:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30618287 0 0) 0x7f37b0003520 con 0x7f37c415b440 2023-11-24T08:10:17.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 >> v2:172.21.15.120:3300/0 conn(0x7f37c40cc0f0 msgr2=0x7f37c40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 --2- 172.21.15.107:0/684538428 >> v2:172.21.15.120:3300/0 conn(0x7f37c40cc0f0 0x7f37c40c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 >> v1:172.21.15.120:6789/0 conn(0x7f37c40c57e0 legacy=0x7f37c406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c415f0f0 con 0x7f37c415b440 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f37c0001670 con 0x7f37c415b440 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 >> v1:172.21.15.107:6789/0 conn(0x7f37c415b440 legacy=0x7f37c40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 0x7f37b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b00047c0 0x7f37b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0005180 0x7f37b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c415b820 con 0x7f37b0003970 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b000c120 con 0x7f37b0005180 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.723+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b000c2e0 con 0x7f37b00047c0 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 0x7f37b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37ca115640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0005180 0x7f37b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 -- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b00047c0 msgr2=0x7f37b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b00047c0 0x7f37b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 -- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0005180 msgr2=0x7f37b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0005180 0x7f37b000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37c9914640 1 -- 172.21.15.107:0/684538428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b0003520 con 0x7f37b0003970 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37ca115640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0005180 0x7f37b000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.724+0000 7f37ca115640 1 -- 172.21.15.107:0/684538428 reap_dead start 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.725+0000 7f37c9914640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 0x7f37b0003d90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f37c000c240 tx=0x7f37b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb0c19d7e2c5194 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.725+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0005ad0 con 0x7f37b0003970 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.725+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c0005c30 con 0x7f37b0003970 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.725+0000 7f37c8912640 1 -- 172.21.15.107:0/684538428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c002caf0 con 0x7f37b0003970 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 -- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 msgr2=0x7f37b0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 0x7f37b0003d90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f37c000c240 tx=0x7f37b00057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 -- 172.21.15.107:0/684538428 shutdown_connections 2023-11-24T08:10:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 --2- 172.21.15.107:0/684538428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b0003970 0x7f37b0003d90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 -- 172.21.15.107:0/684538428 >> 172.21.15.107:0/684538428 conn(0x7f37c40b0c50 msgr2=0x7f37c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 -- 172.21.15.107:0/684538428 shutdown_connections 2023-11-24T08:10:17.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.726+0000 7f37cbb78640 1 -- 172.21.15.107:0/684538428 wait complete. 2023-11-24T08:10:17.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.728+0000 7f37cbb78640 1 Processor -- start 2023-11-24T08:10:17.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.728+0000 7f37cbb78640 1 -- start start 2023-11-24T08:10:17.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0003970 0x7f37c4154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4154900 0x7f37c4154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c409ad30 con 0x7f37c409a370 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4154690 con 0x7f37b0003970 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37cbb78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c9b20 con 0x7f37c4154900 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37ca115640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37ca115640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59360/0 (socket says 172.21.15.107:59360) 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37ca115640 1 -- 172.21.15.107:0/3577267411 learned_addr learned my addr 172.21.15.107:0/3577267411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.729+0000 7f37c9914640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0003970 0x7f37c4154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 -- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4154900 msgr2=0x7f37c4154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4154900 0x7f37c4154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 -- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0003970 msgr2=0x7f37c4154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0003970 0x7f37c4154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 -- 172.21.15.107:0/3577267411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40ccd00 con 0x7f37c409a370 2023-11-24T08:10:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ca115640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f37c406e890 tx=0x7f37bc00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc003d80 con 0x7f37c409a370 2023-11-24T08:10:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc005070 con 0x7f37c409a370 2023-11-24T08:10:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc005420 con 0x7f37c409a370 2023-11-24T08:10:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c40b4ce0 con 0x7f37c409a370 2023-11-24T08:10:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.730+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c40c9840 con 0x7f37c409a370 2023-11-24T08:10:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.732+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c404e8b0 con 0x7f37c409a370 2023-11-24T08:10:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.733+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f37bc005580 con 0x7f37c409a370 2023-11-24T08:10:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.733+0000 7f37ba7fc640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 0x7f37a806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.734+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f37bc087950 con 0x7f37c409a370 2023-11-24T08:10:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.734+0000 7f37c9914640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 0x7f37a806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.734+0000 7f37c9914640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 0x7f37a806db70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f37c0034000 tx=0x7f37c0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.738+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37bc054280 con 0x7f37c409a370 2023-11-24T08:10:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.927+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37c406dc80 con 0x7f37c409a370 2023-11-24T08:10:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.928+0000 7f37ba7fc640 1 -- 172.21.15.107:0/3577267411 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37bc018020 con 0x7f37c409a370 2023-11-24T08:10:17.930 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 msgr2=0x7f37a806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 0x7f37a806db70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f37c0034000 tx=0x7f37c0009030 comp rx=0 tx=0).stop 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 msgr2=0x7f37c409a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f37c406e890 tx=0x7f37bc00d9d0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 shutdown_connections 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f37a806b6b0 0x7f37a806db70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37c409a370 0x7f37c409a7f0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4154900 0x7f37c4154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 --2- 172.21.15.107:0/3577267411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b0003970 0x7f37c4154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 >> 172.21.15.107:0/3577267411 conn(0x7f37c40b0c50 msgr2=0x7f37c40acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 shutdown_connections 2023-11-24T08:10:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:17.931+0000 7f37cbb78640 1 -- 172.21.15.107:0/3577267411 wait complete. 2023-11-24T08:10:17.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete' 2023-11-24T08:10:18.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.157+0000 7f8255390640 1 Processor -- start 2023-11-24T08:10:18.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.158+0000 7f8255390640 1 -- start start 2023-11-24T08:10:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.159+0000 7f8255390640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f825006ec40 0x7f8250071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.159+0000 7f8255390640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500cf070 con 0x7f82500af7d0 2023-11-24T08:10:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.159+0000 7f8255390640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500cae70 con 0x7f8250071640 2023-11-24T08:10:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.159+0000 7f8255390640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825006f120 con 0x7f825006ec40 2023-11-24T08:10:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.160+0000 7f824e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82500af7d0 0x7f82500cf500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59944/0 (socket says 172.21.15.107:59944) 2023-11-24T08:10:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.160+0000 7f824e7fc640 1 -- 172.21.15.107:0/772678138 learned_addr learned my addr 172.21.15.107:0/772678138 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.160+0000 7f824f7fe640 1 --2- 172.21.15.107:0/772678138 >> v2:172.21.15.120:3300/0 conn(0x7f825006ec40 0x7f8250071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.160+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244472761 0 0) 0x7f82500cf070 con 0x7f82500af7d0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.160+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003520 con 0x7f82500af7d0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571549607 0 0) 0x7f823c003520 con 0x7f82500af7d0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 >> v2:172.21.15.120:3300/0 conn(0x7f825006ec40 msgr2=0x7f8250071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 --2- 172.21.15.107:0/772678138 >> v2:172.21.15.120:3300/0 conn(0x7f825006ec40 0x7f8250071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 >> v1:172.21.15.120:6789/0 conn(0x7f8250071640 legacy=0x7f82500ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825006f400 con 0x7f82500af7d0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8238001670 con 0x7f82500af7d0 2023-11-24T08:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 >> v1:172.21.15.107:6789/0 conn(0x7f82500af7d0 legacy=0x7f82500cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c003970 0x7f823c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f823c0047c0 0x7f823c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 0x7f823c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825006f120 con 0x7f823c005180 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c00c020 con 0x7f823c003970 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c00c1e0 con 0x7f823c0047c0 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824f7fe640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 0x7f823c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824effd640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c003970 0x7f823c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824f7fe640 1 -- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f823c0047c0 msgr2=0x7f823c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824f7fe640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f823c0047c0 0x7f823c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824f7fe640 1 -- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c003970 msgr2=0x7f823c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.161+0000 7f824f7fe640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c003970 0x7f823c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.162+0000 7f824f7fe640 1 -- 172.21.15.107:0/772678138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c003520 con 0x7f823c005180 2023-11-24T08:10:18.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.162+0000 7f824f7fe640 1 -- 172.21.15.107:0/772678138 reap_dead start 2023-11-24T08:10:18.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f824f7fe640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 0x7f823c00b980 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f825006f700 tx=0x7f823c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580e73bb5967c79f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:18.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248004020 con 0x7f823c005180 2023-11-24T08:10:18.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8248004180 con 0x7f823c005180 2023-11-24T08:10:18.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f824dffb640 1 -- 172.21.15.107:0/772678138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824802c660 con 0x7f823c005180 2023-11-24T08:10:18.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f8255390640 1 -- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 msgr2=0x7f823c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:18.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f8255390640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 0x7f823c00b980 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f825006f700 tx=0x7f823c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f8255390640 1 -- 172.21.15.107:0/772678138 shutdown_connections 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f8255390640 1 --2- 172.21.15.107:0/772678138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f823c005180 0x7f823c00b980 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.163+0000 7f8255390640 1 -- 172.21.15.107:0/772678138 >> 172.21.15.107:0/772678138 conn(0x7f82500d0110 msgr2=0x7f82500d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.164+0000 7f8255390640 1 -- 172.21.15.107:0/772678138 shutdown_connections 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.164+0000 7f8255390640 1 -- 172.21.15.107:0/772678138 wait complete. 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.165+0000 7f8255390640 1 Processor -- start 2023-11-24T08:10:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.165+0000 7f8255390640 1 -- start start 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.166+0000 7f8255390640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c005180 0x7f82501551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.166+0000 7f8255390640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.166+0000 7f8255390640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82500b2d30 0x7f82500b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f8255390640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500b36f0 con 0x7f82501f6de0 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f8255390640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500c5a40 con 0x7f823c005180 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f8255390640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82500c4e30 con 0x7f82500b2d30 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60912/0 (socket says 172.21.15.107:60912) 2023-11-24T08:10:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 -- 172.21.15.107:0/1371214989 learned_addr learned my addr 172.21.15.107:0/1371214989 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 -- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82500b2d30 msgr2=0x7f82500b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82500b2d30 0x7f82500b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 -- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c005180 msgr2=0x7f82501551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c005180 0x7f82501551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 -- 172.21.15.107:0/1371214989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500c5110 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.167+0000 7f824e7fc640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f823800c730 tx=0x7f823800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.168+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380072d0 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.168+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238007430 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.168+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380047e0 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.168+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82500c53f0 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.168+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8250155720 con 0x7f82501f6de0 2023-11-24T08:10:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.169+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825004e8b0 con 0x7f82501f6de0 2023-11-24T08:10:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.170+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f8238004b50 con 0x7f82501f6de0 2023-11-24T08:10:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.171+0000 7f82477fe640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 0x7f823006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.171+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8238087700 con 0x7f82501f6de0 2023-11-24T08:10:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.172+0000 7f824effd640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 0x7f823006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:18.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.172+0000 7f824effd640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 0x7f823006dba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f82400013d0 tx=0x7f8240009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:18.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.176+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8238054030 con 0x7f82501f6de0 2023-11-24T08:10:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:18.427+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete"]} v 0) v1 -- 0x7f82500c5b80 con 0x7f82501f6de0 2023-11-24T08:10:19.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.067+0000 7f82477fe640 1 -- 172.21.15.107:0/1371214989 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete"]}]=0 v286) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8238018020 con 0x7f82501f6de0 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.071+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 msgr2=0x7f823006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.071+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 0x7f823006dba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f82400013d0 tx=0x7f8240009040 comp rx=0 tx=0).stop 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.071+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 msgr2=0x7f82500c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.071+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f823800c730 tx=0x7f823800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 shutdown_connections 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f823006b6e0 0x7f823006dba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82500b2d30 0x7f82500b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82501f6de0 0x7f82500c5ed0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 --2- 172.21.15.107:0/1371214989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f823c005180 0x7f82501551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 >> 172.21.15.107:0/1371214989 conn(0x7f82500d0110 msgr2=0x7f82500b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:19.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 shutdown_connections 2023-11-24T08:10:19.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.072+0000 7f8255390640 1 -- 172.21.15.107:0/1371214989 wait complete. 2023-11-24T08:10:19.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:10:19.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.314+0000 7fef70a83640 1 Processor -- start 2023-11-24T08:10:19.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.315+0000 7fef70a83640 1 -- start start 2023-11-24T08:10:19.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.316+0000 7fef70a83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef6c161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.316+0000 7fef70a83640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6c0ac840 con 0x7fef6c06e950 2023-11-24T08:10:19.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.316+0000 7fef70a83640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6c0ad6a0 con 0x7fef6c06b780 2023-11-24T08:10:19.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.316+0000 7fef70a83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c162100 con 0x7fef6c0ae240 2023-11-24T08:10:19.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.316+0000 7fef69d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fef6c06e950 0x7fef6c0adb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59960/0 (socket says 172.21.15.107:59960) 2023-11-24T08:10:19.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef69d9b640 1 -- 172.21.15.107:0/887484150 learned_addr learned my addr 172.21.15.107:0/887484150 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:19.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6ad9d640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef6c161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433293546 0 0) 0x7fef6c0ac840 con 0x7fef6c06e950 2023-11-24T08:10:19.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef58003520 con 0x7fef6c06e950 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400217513 0 0) 0x7fef58003520 con 0x7fef6c06e950 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 msgr2=0x7fef6c161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef6c161bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 >> v1:172.21.15.120:6789/0 conn(0x7fef6c06b780 legacy=0x7fef6c0b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.317+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c0aca90 con 0x7fef6c06e950 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fef54001670 con 0x7fef6c06e950 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 >> v1:172.21.15.107:6789/0 conn(0x7fef6c06e950 legacy=0x7fef6c0adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef580039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef58003f20 0x7fef580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58004d00 0x7fef58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c162100 con 0x7fef58003f20 2023-11-24T08:10:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5800ba60 con 0x7fef58004d00 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5800bc20 con 0x7fef6c0ae240 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef58003f20 0x7fef580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6a59c640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef580039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 -- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 msgr2=0x7fef580039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6ad9d640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58004d00 0x7fef58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef580039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 -- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58004d00 msgr2=0x7fef58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58004d00 0x7fef58005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef69d9b640 1 -- 172.21.15.107:0/887484150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef58003520 con 0x7fef58003f20 2023-11-24T08:10:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.318+0000 7fef6a59c640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0ae240 0x7fef580039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.319+0000 7fef69d9b640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef58003f20 0x7fef580047c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c210 tx=0x7fef580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c97050f6b63e2a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54005ad0 con 0x7fef58003f20 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54005c30 con 0x7fef58003f20 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef6959a640 1 -- 172.21.15.107:0/887484150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5402cad0 con 0x7fef58003f20 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 -- 172.21.15.107:0/887484150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef58003f20 msgr2=0x7fef580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 --2- 172.21.15.107:0/887484150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef58003f20 0x7fef580047c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c210 tx=0x7fef580057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef6ad9d640 1 -- 172.21.15.107:0/887484150 reap_dead start 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 -- 172.21.15.107:0/887484150 shutdown_connections 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 -- 172.21.15.107:0/887484150 >> 172.21.15.107:0/887484150 conn(0x7fef6c070fa0 msgr2=0x7fef6c0713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 -- 172.21.15.107:0/887484150 shutdown_connections 2023-11-24T08:10:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.320+0000 7fef70a83640 1 -- 172.21.15.107:0/887484150 wait complete. 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.322+0000 7fef70a83640 1 Processor -- start 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.322+0000 7fef70a83640 1 -- start start 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58003f20 0x7fef6c1567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0b13b0 0x7fef6c1f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c156ce0 con 0x7fef6c1616b0 2023-11-24T08:10:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c161220 con 0x7fef58003f20 2023-11-24T08:10:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.323+0000 7fef70a83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c1f3a80 con 0x7fef6c0b13b0 2023-11-24T08:10:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60924/0 (socket says 172.21.15.107:60924) 2023-11-24T08:10:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 -- 172.21.15.107:0/3371771101 learned_addr learned my addr 172.21.15.107:0/3371771101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6a59c640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58003f20 0x7fef6c1567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef69d9b640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0b13b0 0x7fef6c1f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 -- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0b13b0 msgr2=0x7fef6c1f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0b13b0 0x7fef6c1f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 -- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58003f20 msgr2=0x7fef6c1567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58003f20 0x7fef6c1567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 -- 172.21.15.107:0/3371771101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c1f3d60 con 0x7fef6c1616b0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.324+0000 7fef6ad9d640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ae670 tx=0x7fef64010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.325+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6400e8a0 con 0x7fef6c1616b0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.325+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6400ea00 con 0x7fef6c1616b0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.325+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6400edb0 con 0x7fef6c1616b0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.325+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6c1f4040 con 0x7fef6c1616b0 2023-11-24T08:10:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.325+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6c160df0 con 0x7fef6c1616b0 2023-11-24T08:10:19.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.326+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6c161010 con 0x7fef6c1616b0 2023-11-24T08:10:19.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.327+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fef64020410 con 0x7fef6c1616b0 2023-11-24T08:10:19.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.328+0000 7fef62ffd640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 0x7fef4c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:19.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.328+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fef640878f0 con 0x7fef6c1616b0 2023-11-24T08:10:19.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.328+0000 7fef6a59c640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 0x7fef4c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:19.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.329+0000 7fef6a59c640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 0x7fef4c06dba0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fef6c073580 tx=0x7fef5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.332+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef64054220 con 0x7fef6c1616b0 2023-11-24T08:10:19.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.522+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fef6c04e8b0 con 0x7fef6c1616b0 2023-11-24T08:10:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.530+0000 7fef62ffd640 1 -- 172.21.15.107:0/3371771101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef64057840 con 0x7fef6c1616b0 2023-11-24T08:10:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 msgr2=0x7fef4c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 0x7fef4c06dba0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fef6c073580 tx=0x7fef5c009040 comp rx=0 tx=0).stop 2023-11-24T08:10:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 msgr2=0x7fef6c161b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ae670 tx=0x7fef64010c20 comp rx=0 tx=0).stop 2023-11-24T08:10:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 shutdown_connections 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4c06b6e0 0x7fef4c06dba0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6c1616b0 0x7fef6c161b60 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6c0b13b0 0x7fef6c1f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 --2- 172.21.15.107:0/3371771101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef58003f20 0x7fef6c1567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 >> 172.21.15.107:0/3371771101 conn(0x7fef6c070fa0 msgr2=0x7fef6c0b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 shutdown_connections 2023-11-24T08:10:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:19.533+0000 7fef70a83640 1 -- 172.21.15.107:0/3371771101 wait complete. 2023-11-24T08:10:19.553 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:10:19.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:10:19.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:10:19.782 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:10:19.783 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:19.783 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:19.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:10:19.846 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:19.846 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:19.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:10:19.904 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:19.904 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:19.964 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:10:19.964 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:19.964 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:20.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:10:20.024 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:10:20.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:20.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.248+0000 7f85c29f4640 1 Processor -- start 2023-11-24T08:10:20.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.248+0000 7f85c29f4640 1 -- start start 2023-11-24T08:10:20.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 0x7f85bc071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85bc0cd6a0 0x7f85bc15a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0ce870 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0cdc60 con 0x7f85bc159e80 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.249+0000 7f85c29f4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0cdf40 con 0x7f85bc15a7a0 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85bc0cd6a0 0x7f85bc15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85bc0cd6a0 0x7f85bc15a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60926/0 (socket says 172.21.15.107:60926) 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 -- 172.21.15.107:0/1977373579 learned_addr learned my addr 172.21.15.107:0/1977373579 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c19f2640 1 --2- 172.21.15.107:0/1977373579 >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 0x7f85bc071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c21f3640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 -- 172.21.15.107:0/1977373579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 msgr2=0x7f85bc06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc06c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 -- 172.21.15.107:0/1977373579 >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 msgr2=0x7f85bc071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.250+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1977373579 >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 0x7f85bc071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.251+0000 7f85c11f1640 1 -- 172.21.15.107:0/1977373579 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc0ce220 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.251+0000 7f85c19f2640 1 --2- 172.21.15.107:0/1977373579 >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 0x7f85bc071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:10:20.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.252+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc15a260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f85bc072a70 tx=0x7f85ac02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ed3b5a1df3f425 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.252+0000 7f85c09f0640 1 -- 172.21.15.107:0/1977373579 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac02ede0 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.252+0000 7f85c09f0640 1 -- 172.21.15.107:0/1977373579 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac036440 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.252+0000 7f85c09f0640 1 -- 172.21.15.107:0/1977373579 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0367d0 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 -- 172.21.15.107:0/1977373579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 msgr2=0x7f85bc15a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc15a260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f85bc072a70 tx=0x7f85ac02e030 comp rx=0 tx=0).stop 2023-11-24T08:10:20.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 -- 172.21.15.107:0/1977373579 shutdown_connections 2023-11-24T08:10:20.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1977373579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc15a260 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1977373579 >> v2:172.21.15.120:3300/0 conn(0x7f85bc159e80 0x7f85bc071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 -- 172.21.15.107:0/1977373579 >> 172.21.15.107:0/1977373579 conn(0x7f85bc0cb830 msgr2=0x7f85bc0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 -- 172.21.15.107:0/1977373579 shutdown_connections 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.253+0000 7f85c29f4640 1 -- 172.21.15.107:0/1977373579 wait complete. 2023-11-24T08:10:20.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.255+0000 7f85c29f4640 1 Processor -- start 2023-11-24T08:10:20.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.255+0000 7f85c29f4640 1 -- start start 2023-11-24T08:10:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85bc159e80 0x7f85bc1e4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0acd70 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0acfa0 con 0x7f85bc159e80 2023-11-24T08:10:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c29f4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0ad280 con 0x7f85bc15a7a0 2023-11-24T08:10:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c19f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c19f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60932/0 (socket says 172.21.15.107:60932) 2023-11-24T08:10:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.256+0000 7f85c19f2640 1 -- 172.21.15.107:0/1464082364 learned_addr learned my addr 172.21.15.107:0/1464082364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 -- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 msgr2=0x7f85bc158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 -- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85bc159e80 msgr2=0x7f85bc1e4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85bc159e80 0x7f85bc1e4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85bc159e80 0x7f85bc1e4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 -- 172.21.15.107:0/1464082364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac00b770 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c19f2640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e7e0 tx=0x7f85b800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8004030 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8004190 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc0ad560 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8004520 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.257+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc0adb20 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.258+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc070940 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.259+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f85b800c6d0 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.260+0000 7f85b27fc640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 0x7f85a006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.260+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f85b808a370 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.261+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 0x7f85a006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.261+0000 7f85c11f1640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 0x7f85a006db70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f85ac038000 tx=0x7f85ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.265+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85b80546b0 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.494+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85bc04e840 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.495+0000 7f85b27fc640 1 -- 172.21.15.107:0/1464082364 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f85b8020e90 con 0x7f85bc0cd6a0 2023-11-24T08:10:20.497 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:20.497 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1959700544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1959700544},{"type":"v1","addr":"172.21.15.107:6837","nonce":1959700544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":4895,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2842522274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2842522274},{"type":"v1","addr":"172.21.15.120:6833","nonce":2842522274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/458455570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":458455570},{"type":"v1","addr":"172.21.15.120:6835","nonce":458455570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/3728774944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3728774944},{"type":"v1","addr":"172.21.15.107:6836","nonce":3728774944}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2023-11-24T08:10:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 35 2023-11-24T08:10:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 msgr2=0x7f85a006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 0x7f85a006db70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f85ac038000 tx=0x7f85ac008040 comp rx=0 tx=0).stop 2023-11-24T08:10:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 msgr2=0x7f85bc1446e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e7e0 tx=0x7f85b800eca0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 shutdown_connections 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85a006b6b0 0x7f85a006db70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85bc15a7a0 0x7f85bc158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85bc159e80 0x7f85bc1e4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 --2- 172.21.15.107:0/1464082364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc0cd6a0 0x7f85bc1446e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 >> 172.21.15.107:0/1464082364 conn(0x7f85bc0cb830 msgr2=0x7f85bc06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 shutdown_connections 2023-11-24T08:10:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.498+0000 7f85c29f4640 1 -- 172.21.15.107:0/1464082364 wait complete. 2023-11-24T08:10:20.519 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:10:20.519 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:10:20.519 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:10:20.519 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:10:20.519 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:10:20.519 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:10:20.520 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:10:20.520 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:10:20.520 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:10:20.520 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:10:20.520 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:10:20.520 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:10:20.520 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:10:20.520 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:10:20.521 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:10:20.521 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:10:20.535 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:10:20.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:10:20.537 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:10:20.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:10:20.545 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:10:20.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:10:20.551 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:10:20.551 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:10:20.582 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:10:20.582 DEBUG:teuthology.parallel:result is None 2023-11-24T08:10:20.589 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:10:20.589 DEBUG:teuthology.parallel:result is None 2023-11-24T08:10:20.599 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:10:20.599 DEBUG:teuthology.parallel:result is None 2023-11-24T08:10:20.606 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:10:20.606 DEBUG:teuthology.parallel:result is None 2023-11-24T08:10:20.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:10:20.741 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:10:20.750 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:10:20.765 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:10:20.774 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:10:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.791+0000 7f989183d640 1 Processor -- start 2023-11-24T08:10:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.791+0000 7f989183d640 1 -- start start 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f989183d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f988c072160 0x7f988c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f989183d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c06de10 0x7f988c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f989183d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c15ec90 con 0x7f988c0c49f0 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f989183d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0cd530 con 0x7f988c072160 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f989183d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0c4ed0 con 0x7f988c06de10 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f98827fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f988c0c49f0 0x7f988c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59964/0 (socket says 172.21.15.107:59964) 2023-11-24T08:10:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.793+0000 7f98827fc640 1 -- 172.21.15.107:0/2815352185 learned_addr learned my addr 172.21.15.107:0/2815352185 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988affd640 1 --2- 172.21.15.107:0/2815352185 >> v2:172.21.15.120:3300/0 conn(0x7f988c072160 0x7f988c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988b7fe640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c06de10 0x7f988c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988a7fc640 1 -- 172.21.15.107:0/2815352185 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239422175 0 0) 0x7f9874001110 con 0x7f988c0c49f0 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988a7fc640 1 -- 172.21.15.107:0/2815352185 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878005730 con 0x7f988c0c49f0 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988affd640 1 -- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c06de10 msgr2=0x7f988c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988affd640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c06de10 0x7f988c0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988affd640 1 -- 172.21.15.107:0/2815352185 >> v1:172.21.15.107:6789/0 conn(0x7f988c0c49f0 legacy=0x7f988c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.794+0000 7f988affd640 1 -- 172.21.15.107:0/2815352185 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0c51b0 con 0x7f988c072160 2023-11-24T08:10:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.795+0000 7f988affd640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c15e750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f987c0097a0 tx=0x7f987c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b9fe740afb3d38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.795+0000 7f988a7fc640 1 -- 172.21.15.107:0/2815352185 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c02d040 con 0x7f988c072160 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.795+0000 7f988a7fc640 1 -- 172.21.15.107:0/2815352185 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c031810 con 0x7f988c072160 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.795+0000 7f988a7fc640 1 -- 172.21.15.107:0/2815352185 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c031ba0 con 0x7f988c072160 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 -- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 msgr2=0x7f988c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c15e750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f987c0097a0 tx=0x7f987c02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 -- 172.21.15.107:0/2815352185 shutdown_connections 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c06de10 0x7f988c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 --2- 172.21.15.107:0/2815352185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c15e750 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 -- 172.21.15.107:0/2815352185 >> 172.21.15.107:0/2815352185 conn(0x7f988c0b0c50 msgr2=0x7f988c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 -- 172.21.15.107:0/2815352185 shutdown_connections 2023-11-24T08:10:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.796+0000 7f989183d640 1 -- 172.21.15.107:0/2815352185 wait complete. 2023-11-24T08:10:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.798+0000 7f989183d640 1 Processor -- start 2023-11-24T08:10:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.798+0000 7f989183d640 1 -- start start 2023-11-24T08:10:20.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 0x7f988c0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0b4950 con 0x7f988c06de10 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0b4c30 con 0x7f988c072160 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f989183d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0b4f10 con 0x7f988c0c49f0 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f988affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f988affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60952/0 (socket says 172.21.15.107:60952) 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f988affd640 1 -- 172.21.15.107:0/3699314680 learned_addr learned my addr 172.21.15.107:0/3699314680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f988b7fe640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 0x7f988c0ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.799+0000 7f98827fc640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 -- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 msgr2=0x7f988c0ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 0x7f988c0ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 -- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 msgr2=0x7f988c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 -- 172.21.15.107:0/3699314680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c009400 con 0x7f988c06de10 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988b7fe640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 0x7f988c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f988affd640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f987c0098c0 tx=0x7f987c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c03ad60 con 0x7f988c06de10 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c044400 con 0x7f988c06de10 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c15a550 con 0x7f988c06de10 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0445b0 con 0x7f988c06de10 2023-11-24T08:10:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.800+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c15a060 con 0x7f988c06de10 2023-11-24T08:10:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.802+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c04e8b0 con 0x7f988c06de10 2023-11-24T08:10:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.802+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f987c04d440 con 0x7f988c06de10 2023-11-24T08:10:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.804+0000 7f9888ff9640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 0x7f986c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.804+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f987c0aa150 con 0x7f988c06de10 2023-11-24T08:10:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.804+0000 7f98827fc640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 0x7f986c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.805+0000 7f98827fc640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 0x7f986c06db70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9874007fd0 tx=0x7f9874009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:20.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:20.808+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f987c076c50 con 0x7f988c06de10 2023-11-24T08:10:21.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:21.010+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f988c15dab0 con 0x7f988c06de10 2023-11-24T08:10:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.112+0000 7f9888ff9640 1 -- 172.21.15.107:0/3699314680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v74) v1 ==== 97+0+0 (secure 0 0 0) 0x7f987c07a270 con 0x7f988c06de10 2023-11-24T08:10:22.114 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:10:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 msgr2=0x7f986c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 0x7f986c06db70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9874007fd0 tx=0x7f9874009040 comp rx=0 tx=0).stop 2023-11-24T08:10:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 msgr2=0x7f988c159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f987c0098c0 tx=0x7f987c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 shutdown_connections 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f986c06b6b0 0x7f986c06db70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f988c0c49f0 0x7f988c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f988c072160 0x7f988c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 --2- 172.21.15.107:0/3699314680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f988c06de10 0x7f988c159b20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 >> 172.21.15.107:0/3699314680 conn(0x7f988c0b0c50 msgr2=0x7f988c0715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.115+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 shutdown_connections 2023-11-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.116+0000 7f989183d640 1 -- 172.21.15.107:0/3699314680 wait complete. 2023-11-24T08:10:22.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:10:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.352+0000 7f283171d640 1 Processor -- start 2023-11-24T08:10:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.352+0000 7f283171d640 1 -- start start 2023-11-24T08:10:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.353+0000 7f283171d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 0x7f282c0b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.353+0000 7f283171d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f283171d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0711e0 con 0x7f282c15b5e0 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f283171d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282c0b4540 con 0x7f282c0cd5b0 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f283171d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0cdc80 con 0x7f282c0abf50 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f282b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f28227fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 0x7f282c0b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f28227fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 0x7f282c0b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:45310/0 (socket says 172.21.15.107:45310) 2023-11-24T08:10:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f282b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:48398/0 (socket says 172.21.15.107:48398) 2023-11-24T08:10:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.354+0000 7f28227fc640 1 -- 172.21.15.107:0/1439513997 learned_addr learned my addr 172.21.15.107:0/1439513997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.355+0000 7f282b7fe640 1 -- 172.21.15.107:0/1439513997 >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 msgr2=0x7f282c0b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.355+0000 7f282b7fe640 1 --2- 172.21.15.107:0/1439513997 >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 0x7f282c0b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.355+0000 7f282b7fe640 1 -- 172.21.15.107:0/1439513997 >> v1:172.21.15.120:6789/0 conn(0x7f282c15b5e0 legacy=0x7f282c0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.355+0000 7f282b7fe640 1 -- 172.21.15.107:0/1439513997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0705d0 con 0x7f282c0abf50 2023-11-24T08:10:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.356+0000 7f282b7fe640 1 --2- 172.21.15.107:0/1439513997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28240097a0 tx=0x7f282402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10824b2ff2239e76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.357+0000 7f282a7fc640 1 -- 172.21.15.107:0/1439513997 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282402d040 con 0x7f282c0abf50 2023-11-24T08:10:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.357+0000 7f282a7fc640 1 -- 172.21.15.107:0/1439513997 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2824031810 con 0x7f282c0abf50 2023-11-24T08:10:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.357+0000 7f282a7fc640 1 -- 172.21.15.107:0/1439513997 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824031ba0 con 0x7f282c0abf50 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 -- 172.21.15.107:0/1439513997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 msgr2=0x7f282c0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 --2- 172.21.15.107:0/1439513997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28240097a0 tx=0x7f282402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 -- 172.21.15.107:0/1439513997 shutdown_connections 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 --2- 172.21.15.107:0/1439513997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c0b4000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 --2- 172.21.15.107:0/1439513997 >> v2:172.21.15.120:3300/0 conn(0x7f282c0cd5b0 0x7f282c0b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 -- 172.21.15.107:0/1439513997 >> 172.21.15.107:0/1439513997 conn(0x7f282c0b0c70 msgr2=0x7f282c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:22.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 -- 172.21.15.107:0/1439513997 shutdown_connections 2023-11-24T08:10:22.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.358+0000 7f283171d640 1 -- 172.21.15.107:0/1439513997 wait complete. 2023-11-24T08:10:22.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.360+0000 7f283171d640 1 Processor -- start 2023-11-24T08:10:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.360+0000 7f283171d640 1 -- start start 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c1492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 0x7f282c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c8fe0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c92c0 con 0x7f282c15b5e0 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f283171d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c95a0 con 0x7f282c0abf50 2023-11-24T08:10:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f28227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f28227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60964/0 (socket says 172.21.15.107:60964) 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f28227fc640 1 -- 172.21.15.107:0/2567932068 learned_addr learned my addr 172.21.15.107:0/2567932068 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f282b7fe640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 0x7f282c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.361+0000 7f282affd640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c1492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 -- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 msgr2=0x7f282c1492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c1492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 -- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 msgr2=0x7f282c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 0x7f282c15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 -- 172.21.15.107:0/2567932068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824009400 con 0x7f282c0cd5b0 2023-11-24T08:10:22.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f282b7fe640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 0x7f282c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f28227fc640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2814009fd0 tx=0x7f281400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400c940 con 0x7f282c0cd5b0 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281400caa0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400ce30 con 0x7f282c0cd5b0 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.362+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c0c9880 con 0x7f282c0cd5b0 2023-11-24T08:10:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.363+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282c1497f0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.364+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c04e8b0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.365+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f2814013470 con 0x7f282c0cd5b0 2023-11-24T08:10:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.365+0000 7f2823fff640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 0x7f280c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.366+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f28140881c0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.366+0000 7f282affd640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 0x7f280c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.366+0000 7f282affd640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 0x7f280c06db70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f281c001ca0 tx=0x7f281c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.370+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2814054b50 con 0x7f282c0cd5b0 2023-11-24T08:10:22.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.627+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f282c0cc9e0 con 0x7f282c0cd5b0 2023-11-24T08:10:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.628+0000 7f2823fff640 1 -- 172.21.15.107:0/2567932068 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2814058170 con 0x7f282c0cd5b0 2023-11-24T08:10:22.630 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.632 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:10:22.633 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.634 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:10:22.636 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.637 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.639 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:10:22.640 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:10:22.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:10:22.642 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.644 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:10:22.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:10:22.646 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 msgr2=0x7f280c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 0x7f280c06db70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f281c001ca0 tx=0x7f281c009040 comp rx=0 tx=0).stop 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 msgr2=0x7f282c159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2814009fd0 tx=0x7f281400ed40 comp rx=0 tx=0).stop 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 shutdown_connections 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f280c06b6b0 0x7f280c06db70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f282c15b5e0 0x7f282c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f282c0cd5b0 0x7f282c159b20 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 --2- 172.21.15.107:0/2567932068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f282c0abf50 0x7f282c1492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 >> 172.21.15.107:0/2567932068 conn(0x7f282c0b0c70 msgr2=0x7f282c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 shutdown_connections 2023-11-24T08:10:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.631+0000 7f283171d640 1 -- 172.21.15.107:0/2567932068 wait complete. 2023-11-24T08:10:22.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:22.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.858+0000 7f5e0ffff640 1 Processor -- start 2023-11-24T08:10:22.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.859+0000 7f5e0ffff640 1 -- start start 2023-11-24T08:10:22.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.860+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e10162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.860+0000 7f5e0ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e100cd160 0x7f5e100cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.860+0000 7f5e0ffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e10162eb0 con 0x7f5e1014b100 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e0ffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e101654a0 con 0x7f5e100cd160 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e0ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100ccf00 con 0x7f5e100703f0 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e067fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e10162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e0f7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e100cd160 0x7f5e100cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e067fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e10162970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:48412/0 (socket says 172.21.15.107:48412) 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.861+0000 7f5e067fc640 1 -- 172.21.15.107:0/814644977 learned_addr learned my addr 172.21.15.107:0/814644977 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.862+0000 7f5e0f7fe640 1 -- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 msgr2=0x7f5e10162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.862+0000 7f5e0f7fe640 1 --2- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e10162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.862+0000 7f5e0f7fe640 1 -- 172.21.15.107:0/814644977 >> v1:172.21.15.120:6789/0 conn(0x7f5e1014b100 legacy=0x7f5e10165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.862+0000 7f5e0f7fe640 1 -- 172.21.15.107:0/814644977 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10166320 con 0x7f5e100cd160 2023-11-24T08:10:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.863+0000 7f5e0f7fe640 1 --2- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e100cd160 0x7f5e100cd5f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5e080097a0 tx=0x7f5e0802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adec47932eefa915 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.864+0000 7f5e0e7fc640 1 -- 172.21.15.107:0/814644977 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0802e070 con 0x7f5e100cd160 2023-11-24T08:10:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.864+0000 7f5e0e7fc640 1 -- 172.21.15.107:0/814644977 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e080315a0 con 0x7f5e100cd160 2023-11-24T08:10:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.864+0000 7f5e0e7fc640 1 -- 172.21.15.107:0/814644977 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08031930 con 0x7f5e100cd160 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 -- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e100cd160 msgr2=0x7f5e100cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e100cd160 0x7f5e100cd5f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5e080097a0 tx=0x7f5e0802b840 comp rx=0 tx=0).stop 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 -- 172.21.15.107:0/814644977 shutdown_connections 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e100cd160 0x7f5e100cd5f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/814644977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e10162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 -- 172.21.15.107:0/814644977 >> 172.21.15.107:0/814644977 conn(0x7f5e100cb830 msgr2=0x7f5e100cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 -- 172.21.15.107:0/814644977 shutdown_connections 2023-11-24T08:10:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.865+0000 7f5e0ffff640 1 -- 172.21.15.107:0/814644977 wait complete. 2023-11-24T08:10:22.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.867+0000 7f5e0ffff640 1 Processor -- start 2023-11-24T08:10:22.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.867+0000 7f5e0ffff640 1 -- start start 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e101ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e1014b100 0x7f5e101607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100b1ca0 con 0x7f5e100cd160 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100b1e30 con 0x7f5e1014b100 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e101ec410 con 0x7f5e100703f0 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60980/0 (socket says 172.21.15.107:60980) 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 -- 172.21.15.107:0/2585900922 learned_addr learned my addr 172.21.15.107:0/2585900922 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0effd640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e101ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0f7fe640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e1014b100 0x7f5e101607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 -- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 msgr2=0x7f5e101ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e101ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 -- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e1014b100 msgr2=0x7f5e101607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e1014b100 0x7f5e101607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e067fc640 1 -- 172.21.15.107:0/2585900922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08009400 con 0x7f5e100cd160 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.868+0000 7f5e0effd640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e101ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e067fc640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5df800e840 tx=0x7f5df800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df800c8e0 con 0x7f5e100cd160 2023-11-24T08:10:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df800ca40 con 0x7f5e100cd160 2023-11-24T08:10:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df800cdf0 con 0x7f5e100cd160 2023-11-24T08:10:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e101ec6f0 con 0x7f5e100cd160 2023-11-24T08:10:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.869+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e101eccb0 con 0x7f5e100cd160 2023-11-24T08:10:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.870+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e100b1fa0 con 0x7f5e100cd160 2023-11-24T08:10:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.871+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f5df8013470 con 0x7f5e100cd160 2023-11-24T08:10:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.873+0000 7f5e07fff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 0x7f5df006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.873+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5df80880e0 con 0x7f5e100cd160 2023-11-24T08:10:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.873+0000 7f5e0effd640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 0x7f5df006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.874+0000 7f5e0effd640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 0x7f5df006db70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5e100cd800 tx=0x7f5e00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:22.877+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e100b1fa0 con 0x7f5e100cd160 2023-11-24T08:10:23.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.069+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e100b1fa0 con 0x7f5e100cd160 2023-11-24T08:10:23.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.071+0000 7f5e07fff640 1 -- 172.21.15.107:0/2585900922 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+12857 (secure 0 0 0) 0x7f5e100b1fa0 con 0x7f5e100cd160 2023-11-24T08:10:23.074 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:23.074 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":74,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:22.109351+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:23.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 msgr2=0x7f5df006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 0x7f5df006db70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5e100cd800 tx=0x7f5e00009040 comp rx=0 tx=0).stop 2023-11-24T08:10:23.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 msgr2=0x7f5e10144a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5df800e840 tx=0x7f5df800ed00 comp rx=0 tx=0).stop 2023-11-24T08:10:23.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 shutdown_connections 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006b6b0 0x7f5df006db70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e1014b100 0x7f5e101607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e100cd160 0x7f5e10144a70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 --2- 172.21.15.107:0/2585900922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e100703f0 0x7f5e101ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 >> 172.21.15.107:0/2585900922 conn(0x7f5e100cb830 msgr2=0x7f5e100ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 shutdown_connections 2023-11-24T08:10:23.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.074+0000 7f5e0ffff640 1 -- 172.21.15.107:0/2585900922 wait complete. 2023-11-24T08:10:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:10:23.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:10:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.301+0000 7f54bade4640 1 Processor -- start 2023-11-24T08:10:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.301+0000 7f54bade4640 1 -- start start 2023-11-24T08:10:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54bade4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 0x7f54b40b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54bade4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54bade4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b415ed20 con 0x7f54b415b260 2023-11-24T08:10:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54bade4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b415b640 con 0x7f54b40b0b40 2023-11-24T08:10:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54bade4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40b14f0 con 0x7f54b406c5b0 2023-11-24T08:10:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54b95e1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 0x7f54b40b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54ba5e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54b95e1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 0x7f54b40b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:45348/0 (socket says 172.21.15.107:45348) 2023-11-24T08:10:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54ba5e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:48426/0 (socket says 172.21.15.107:48426) 2023-11-24T08:10:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.303+0000 7f54b95e1640 1 -- 172.21.15.107:0/2772003806 learned_addr learned my addr 172.21.15.107:0/2772003806 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.304+0000 7f54ba5e3640 1 -- 172.21.15.107:0/2772003806 >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 msgr2=0x7f54b40b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.304+0000 7f54ba5e3640 1 --2- 172.21.15.107:0/2772003806 >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 0x7f54b40b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.304+0000 7f54ba5e3640 1 -- 172.21.15.107:0/2772003806 >> v1:172.21.15.120:6789/0 conn(0x7f54b415b260 legacy=0x7f54b406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.304+0000 7f54ba5e3640 1 -- 172.21.15.107:0/2772003806 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b415f8a0 con 0x7f54b406c5b0 2023-11-24T08:10:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.305+0000 7f54ba5e3640 1 --2- 172.21.15.107:0/2772003806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009d90 tx=0x7f54ac02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c46b99a6d6bdf798 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.306+0000 7f54b8de0640 1 -- 172.21.15.107:0/2772003806 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac02cb40 con 0x7f54b406c5b0 2023-11-24T08:10:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.306+0000 7f54b8de0640 1 -- 172.21.15.107:0/2772003806 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac02cca0 con 0x7f54b406c5b0 2023-11-24T08:10:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.306+0000 7f54b8de0640 1 -- 172.21.15.107:0/2772003806 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac034660 con 0x7f54b406c5b0 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.306+0000 7f54bade4640 1 -- 172.21.15.107:0/2772003806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 msgr2=0x7f54b406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.306+0000 7f54bade4640 1 --2- 172.21.15.107:0/2772003806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009d90 tx=0x7f54ac02bd90 comp rx=0 tx=0).stop 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 -- 172.21.15.107:0/2772003806 shutdown_connections 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 --2- 172.21.15.107:0/2772003806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b406c5b0 0x7f54b406ca40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 --2- 172.21.15.107:0/2772003806 >> v2:172.21.15.120:3300/0 conn(0x7f54b40b0b40 0x7f54b40b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 -- 172.21.15.107:0/2772003806 >> 172.21.15.107:0/2772003806 conn(0x7f54b40cbb40 msgr2=0x7f54b40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 -- 172.21.15.107:0/2772003806 shutdown_connections 2023-11-24T08:10:23.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.307+0000 7f54bade4640 1 -- 172.21.15.107:0/2772003806 wait complete. 2023-11-24T08:10:23.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.308+0000 7f54bade4640 1 Processor -- start 2023-11-24T08:10:23.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.309+0000 7f54bade4640 1 -- start start 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.309+0000 7f54bade4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b406c5b0 0x7f54b41599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54bade4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54bade4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b415b260 0x7f54b415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54bade4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40ca690 con 0x7f54b40b0b40 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54bade4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40c9a80 con 0x7f54b406c5b0 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54bade4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40c9d60 con 0x7f54b415b260 2023-11-24T08:10:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60994/0 (socket says 172.21.15.107:60994) 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 -- 172.21.15.107:0/3011706583 learned_addr learned my addr 172.21.15.107:0/3011706583 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b9de2640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b406c5b0 0x7f54b41599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54ba5e3640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b415b260 0x7f54b415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 -- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b415b260 msgr2=0x7f54b415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b415b260 0x7f54b415d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 -- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b406c5b0 msgr2=0x7f54b41599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b406c5b0 0x7f54b41599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.310+0000 7f54b95e1640 1 -- 172.21.15.107:0/3011706583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009a70 con 0x7f54b40b0b40 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54b95e1640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e840 tx=0x7f54a400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400c8e0 con 0x7f54b40b0b40 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a400ca40 con 0x7f54b40b0b40 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b40ca040 con 0x7f54b40b0b40 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400cdf0 con 0x7f54b40b0b40 2023-11-24T08:10:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.311+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b40c9710 con 0x7f54b40b0b40 2023-11-24T08:10:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.312+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b404e840 con 0x7f54b40b0b40 2023-11-24T08:10:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.314+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f54a4013470 con 0x7f54b40b0b40 2023-11-24T08:10:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.315+0000 7f54aa7fc640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 0x7f549806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.315+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54a4088180 con 0x7f54b40b0b40 2023-11-24T08:10:23.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.315+0000 7f54b9de2640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 0x7f549806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.315+0000 7f54b9de2640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 0x7f549806db70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f54b0007fd0 tx=0x7f54b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.319+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54a4054ab0 con 0x7f54b40b0b40 2023-11-24T08:10:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.511+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54b4073b10 con 0x7f54b40b0b40 2023-11-24T08:10:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.511+0000 7f54aa7fc640 1 -- 172.21.15.107:0/3011706583 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54a40580d0 con 0x7f54b40b0b40 2023-11-24T08:10:23.513 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.514+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 msgr2=0x7f549806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.514+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 0x7f549806db70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f54b0007fd0 tx=0x7f54b0009040 comp rx=0 tx=0).stop 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.514+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 msgr2=0x7f54b40cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.514+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e840 tx=0x7f54a400ed00 comp rx=0 tx=0).stop 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 shutdown_connections 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f549806b6b0 0x7f549806db70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b415b260 0x7f54b415d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b40b0b40 0x7f54b40cad80 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 --2- 172.21.15.107:0/3011706583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b406c5b0 0x7f54b41599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 >> 172.21.15.107:0/3011706583 conn(0x7f54b40cbb40 msgr2=0x7f54b40af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:23.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 shutdown_connections 2023-11-24T08:10:23.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.515+0000 7f54bade4640 1 -- 172.21.15.107:0/3011706583 wait complete. 2023-11-24T08:10:23.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.745+0000 7f63796bb640 1 Processor -- start 2023-11-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.745+0000 7f63796bb640 1 -- start start 2023-11-24T08:10:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.746+0000 7f63796bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f637406d6c0 0x7f637406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.746+0000 7f63796bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63740cdbf0 0x7f63740ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.746+0000 7f63796bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637406e060 con 0x7f637413d120 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.746+0000 7f63796bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637413f9e0 con 0x7f637406d6c0 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.746+0000 7f63796bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637413fcc0 con 0x7f63740cdbf0 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f6373fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f637413d120 0x7f637415af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59980/0 (socket says 172.21.15.107:59980) 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f6373fff640 1 -- 172.21.15.107:0/4226843880 learned_addr learned my addr 172.21.15.107:0/4226843880 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f63737fe640 1 --2- 172.21.15.107:0/4226843880 >> v2:172.21.15.107:3300/0 conn(0x7f637406d6c0 0x7f637406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f6378eba640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63740cdbf0 0x7f63740ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f6372ffd640 1 -- 172.21.15.107:0/4226843880 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849984906 0 0) 0x7f637406e060 con 0x7f637413d120 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f6372ffd640 1 -- 172.21.15.107:0/4226843880 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6360003520 con 0x7f637413d120 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f63737fe640 1 -- 172.21.15.107:0/4226843880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63740cdbf0 msgr2=0x7f63740ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f63737fe640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63740cdbf0 0x7f63740ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f63737fe640 1 -- 172.21.15.107:0/4226843880 >> v1:172.21.15.107:6789/0 conn(0x7f637413d120 legacy=0x7f637415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.747+0000 7f63737fe640 1 -- 172.21.15.107:0/4226843880 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637413ffa0 con 0x7f637406d6c0 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.748+0000 7f63737fe640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637406d6c0 0x7f637406db20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dfe054ee82be05 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f6372ffd640 1 -- 172.21.15.107:0/4226843880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c02cbc0 con 0x7f637406d6c0 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f6372ffd640 1 -- 172.21.15.107:0/4226843880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c02cd20 con 0x7f637406d6c0 2023-11-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f6372ffd640 1 -- 172.21.15.107:0/4226843880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c034660 con 0x7f637406d6c0 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 -- 172.21.15.107:0/4226843880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637406d6c0 msgr2=0x7f637406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637406d6c0 0x7f637406db20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c02be10 comp rx=0 tx=0).stop 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 -- 172.21.15.107:0/4226843880 shutdown_connections 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63740cdbf0 0x7f63740ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 --2- 172.21.15.107:0/4226843880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637406d6c0 0x7f637406db20 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.749+0000 7f63796bb640 1 -- 172.21.15.107:0/4226843880 >> 172.21.15.107:0/4226843880 conn(0x7f63740cb830 msgr2=0x7f63740cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.750+0000 7f63796bb640 1 -- 172.21.15.107:0/4226843880 shutdown_connections 2023-11-24T08:10:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.750+0000 7f63796bb640 1 -- 172.21.15.107:0/4226843880 wait complete. 2023-11-24T08:10:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.751+0000 7f63796bb640 1 Processor -- start 2023-11-24T08:10:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.751+0000 7f63796bb640 1 -- start start 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f637406d6c0 0x7f637413a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f637413d120 0x7f63740cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740ad450 con 0x7f63740cdbf0 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740ad730 con 0x7f637413d120 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63796bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63740ada10 con 0x7f637406d6c0 2023-11-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.752+0000 7f63737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32794/0 (socket says 172.21.15.107:32794) 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 -- 172.21.15.107:0/3772683213 learned_addr learned my addr 172.21.15.107:0/3772683213 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 -- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f637406d6c0 msgr2=0x7f637413a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f6373fff640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f637406d6c0 0x7f637413a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f6378eba640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f637413d120 0x7f63740cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f637406d6c0 0x7f637413a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 -- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f637413d120 msgr2=0x7f63740cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f637413d120 0x7f63740cf1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 -- 172.21.15.107:0/3772683213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c009a70 con 0x7f63740cdbf0 2023-11-24T08:10:23.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.753+0000 7f63737fe640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.754+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c040650 con 0x7f63740cdbf0 2023-11-24T08:10:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.754+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c02ce90 con 0x7f63740cdbf0 2023-11-24T08:10:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.754+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637415a9d0 con 0x7f63740cdbf0 2023-11-24T08:10:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.754+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0409f0 con 0x7f63740cdbf0 2023-11-24T08:10:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.754+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63740adfd0 con 0x7f63740cdbf0 2023-11-24T08:10:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.755+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6374067990 con 0x7f63740cdbf0 2023-11-24T08:10:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.756+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f635c040b50 con 0x7f63740cdbf0 2023-11-24T08:10:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.757+0000 7f6370ff9640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 0x7f635406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:23.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.757+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f635c0ad320 con 0x7f63740cdbf0 2023-11-24T08:10:23.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.757+0000 7f6373fff640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 0x7f635406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:23.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.757+0000 7f6373fff640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 0x7f635406db70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f63640011e0 tx=0x7f6364009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.761+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f635c077700 con 0x7f63740cdbf0 2023-11-24T08:10:23.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.952+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f63740af2f0 con 0x7f63740cdbf0 2023-11-24T08:10:23.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.952+0000 7f6370ff9640 1 -- 172.21.15.107:0/3772683213 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) v1 ==== 84+0+65 (secure 0 0 0) 0x7f635c07ad20 con 0x7f63740cdbf0 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:23.955 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:10:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 msgr2=0x7f635406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 0x7f635406db70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f63640011e0 tx=0x7f6364009040 comp rx=0 tx=0).stop 2023-11-24T08:10:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 msgr2=0x7f63740ac9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c004a40 comp rx=0 tx=0).stop 2023-11-24T08:10:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 shutdown_connections 2023-11-24T08:10:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f635406b6b0 0x7f635406db70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f637413d120 0x7f63740cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63740cdbf0 0x7f63740ac9d0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 --2- 172.21.15.107:0/3772683213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f637406d6c0 0x7f637413a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 >> 172.21.15.107:0/3772683213 conn(0x7f63740cb830 msgr2=0x7f6374159120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 shutdown_connections 2023-11-24T08:10:23.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:23.955+0000 7f63796bb640 1 -- 172.21.15.107:0/3772683213 wait complete. 2023-11-24T08:10:23.976 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:10:23.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:10:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.190+0000 7fdb6a67c640 1 Processor -- start 2023-11-24T08:10:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.191+0000 7fdb6a67c640 1 -- start start 2023-11-24T08:10:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb640d19f0 0x7fdb640d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640b2700 0x7fdb640cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdb640d0680 0x7fdb640ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb640d1230 con 0x7fdb640d19f0 2023-11-24T08:10:24.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb640d2310 con 0x7fdb640d0680 2023-11-24T08:10:24.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb6a67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640d3b40 con 0x7fdb640b2700 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb63fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb640d19f0 0x7fdb640d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb63fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb640d19f0 0x7fdb640d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32808/0 (socket says 172.21.15.107:32808) 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb63fff640 1 -- 172.21.15.107:0/424967746 learned_addr learned my addr 172.21.15.107:0/424967746 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb68c19640 1 --2- 172.21.15.107:0/424967746 >> v2:172.21.15.120:3300/0 conn(0x7fdb640d0680 0x7fdb640ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.192+0000 7fdb637fe640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640b2700 0x7fdb640cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.193+0000 7fdb63fff640 1 -- 172.21.15.107:0/424967746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640b2700 msgr2=0x7fdb640cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.193+0000 7fdb63fff640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640b2700 0x7fdb640cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.193+0000 7fdb63fff640 1 -- 172.21.15.107:0/424967746 >> v2:172.21.15.120:3300/0 conn(0x7fdb640d0680 msgr2=0x7fdb640ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:24.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.193+0000 7fdb63fff640 1 --2- 172.21.15.107:0/424967746 >> v2:172.21.15.120:3300/0 conn(0x7fdb640d0680 0x7fdb640ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.193+0000 7fdb63fff640 1 -- 172.21.15.107:0/424967746 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640d3e20 con 0x7fdb640d19f0 2023-11-24T08:10:24.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.194+0000 7fdb63fff640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640d19f0 0x7fdb640d1dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdb6408bdd0 tx=0x7fdb5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ef510a0fa7389e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:24.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.194+0000 7fdb62ffd640 1 -- 172.21.15.107:0/424967746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5402ca70 con 0x7fdb640d19f0 2023-11-24T08:10:24.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb62ffd640 1 -- 172.21.15.107:0/424967746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5402cbd0 con 0x7fdb640d19f0 2023-11-24T08:10:24.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb62ffd640 1 -- 172.21.15.107:0/424967746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540343f0 con 0x7fdb640d19f0 2023-11-24T08:10:24.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 -- 172.21.15.107:0/424967746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640d19f0 msgr2=0x7fdb640d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640d19f0 0x7fdb640d1dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdb6408bdd0 tx=0x7fdb5402bed0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 -- 172.21.15.107:0/424967746 shutdown_connections 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/424967746 >> v2:172.21.15.120:3300/0 conn(0x7fdb640d0680 0x7fdb640ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640b2700 0x7fdb640cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/424967746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640d19f0 0x7fdb640d1dd0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 -- 172.21.15.107:0/424967746 >> 172.21.15.107:0/424967746 conn(0x7fdb6408af60 msgr2=0x7fdb640d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 -- 172.21.15.107:0/424967746 shutdown_connections 2023-11-24T08:10:24.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.195+0000 7fdb6a67c640 1 -- 172.21.15.107:0/424967746 wait complete. 2023-11-24T08:10:24.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.197+0000 7fdb6a67c640 1 Processor -- start 2023-11-24T08:10:24.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.197+0000 7fdb6a67c640 1 -- start start 2023-11-24T08:10:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640d0680 0x7fdb640c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb640d19f0 0x7fdb640c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640b1260 con 0x7fdb640b2700 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640c7730 con 0x7fdb640d19f0 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb6a67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640b05e0 con 0x7fdb640d0680 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32812/0 (socket says 172.21.15.107:32812) 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.198+0000 7fdb63fff640 1 -- 172.21.15.107:0/2373388436 learned_addr learned my addr 172.21.15.107:0/2373388436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb68c19640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb640d19f0 0x7fdb640c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb637fe640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640d0680 0x7fdb640c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 -- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640d0680 msgr2=0x7fdb640c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640d0680 0x7fdb640c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 -- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb640d19f0 msgr2=0x7fdb640c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb640d19f0 0x7fdb640c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 -- 172.21.15.107:0/2373388436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009b30 con 0x7fdb640b2700 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb63fff640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fdb54000f80 tx=0x7fdb54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.199+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54009510 con 0x7fdb640b2700 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.200+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5403e450 con 0x7fdb640b2700 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.200+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5403e800 con 0x7fdb640b2700 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.200+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb640b08c0 con 0x7fdb640b2700 2023-11-24T08:10:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.200+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb640b0e80 con 0x7fdb640b2700 2023-11-24T08:10:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.201+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb64067470 con 0x7fdb640b2700 2023-11-24T08:10:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.202+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fdb5403e960 con 0x7fdb640b2700 2023-11-24T08:10:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.203+0000 7fdb60ff9640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 0x7fdb4406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.203+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdb540ad330 con 0x7fdb640b2700 2023-11-24T08:10:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.203+0000 7fdb637fe640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 0x7fdb4406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.204+0000 7fdb637fe640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 0x7fdb4406db70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdb641436c0 tx=0x7fdb4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.208+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb54077810 con 0x7fdb640b2700 2023-11-24T08:10:24.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:24.400+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb640d1fe0 con 0x7fdb640b2700 2023-11-24T08:10:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.128+0000 7fdb60ff9640 1 -- 172.21.15.107:0/2373388436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb5407ae30 con 0x7fdb640b2700 2023-11-24T08:10:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:10:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 msgr2=0x7fdb4406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 0x7fdb4406db70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdb641436c0 tx=0x7fdb4c008040 comp rx=0 tx=0).stop 2023-11-24T08:10:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 msgr2=0x7fdb640b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fdb54000f80 tx=0x7fdb54004a40 comp rx=0 tx=0).stop 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 shutdown_connections 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb4406b6b0 0x7fdb4406db70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb640d19f0 0x7fdb640c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb640d0680 0x7fdb640c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.131+0000 7fdb6a67c640 1 --2- 172.21.15.107:0/2373388436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640b2700 0x7fdb640b0000 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.132+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 >> 172.21.15.107:0/2373388436 conn(0x7fdb6408af60 msgr2=0x7fdb640cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.132+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 shutdown_connections 2023-11-24T08:10:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.132+0000 7fdb6a67c640 1 -- 172.21.15.107:0/2373388436 wait complete. 2023-11-24T08:10:25.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:10:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.361+0000 7f5d2fd53640 1 Processor -- start 2023-11-24T08:10:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.361+0000 7f5d2fd53640 1 -- start start 2023-11-24T08:10:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d2815b530 0x7f5d2806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2806aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 0x7f5d280ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d28072160 con 0x7f5d2815b530 2023-11-24T08:10:25.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2815b910 con 0x7f5d280999d0 2023-11-24T08:10:25.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2fd53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d280b3c80 con 0x7f5d280afa50 2023-11-24T08:10:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.363+0000 7f5d2daef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d2815b530 0x7f5d2806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d2815b530 0x7f5d2806a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32822/0 (socket says 172.21.15.107:32822) 2023-11-24T08:10:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 -- 172.21.15.107:0/1092207596 learned_addr learned my addr 172.21.15.107:0/1092207596 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:25.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2d2ee640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2806aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 -- 172.21.15.107:0/1092207596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 msgr2=0x7f5d2806aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2806aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2e2f0640 1 --2- 172.21.15.107:0/1092207596 >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 0x7f5d280ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 -- 172.21.15.107:0/1092207596 >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 msgr2=0x7f5d280ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 --2- 172.21.15.107:0/1092207596 >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 0x7f5d280ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2daef640 1 -- 172.21.15.107:0/1092207596 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d280b27b0 con 0x7f5d2815b530 2023-11-24T08:10:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.364+0000 7f5d2e2f0640 1 --2- 172.21.15.107:0/1092207596 >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 0x7f5d280ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.365+0000 7f5d2daef640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2806a500 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5d28072b40 tx=0x7f5d2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d002c15989d76c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.365+0000 7f5d2caed640 1 -- 172.21.15.107:0/1092207596 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24031870 con 0x7f5d2815b530 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2caed640 1 -- 172.21.15.107:0/1092207596 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d240319d0 con 0x7f5d2815b530 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2caed640 1 -- 172.21.15.107:0/1092207596 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24031d60 con 0x7f5d2815b530 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 -- 172.21.15.107:0/1092207596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 msgr2=0x7f5d2806a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2806a500 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5d28072b40 tx=0x7f5d2402b910 comp rx=0 tx=0).stop 2023-11-24T08:10:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 -- 172.21.15.107:0/1092207596 shutdown_connections 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/1092207596 >> v2:172.21.15.120:3300/0 conn(0x7f5d280999d0 0x7f5d280ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2806aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/1092207596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2806a500 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 -- 172.21.15.107:0/1092207596 >> 172.21.15.107:0/1092207596 conn(0x7f5d2806c090 msgr2=0x7f5d280b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 -- 172.21.15.107:0/1092207596 shutdown_connections 2023-11-24T08:10:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.366+0000 7f5d2fd53640 1 -- 172.21.15.107:0/1092207596 wait complete. 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.368+0000 7f5d2fd53640 1 Processor -- start 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.368+0000 7f5d2fd53640 1 -- start start 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d280999d0 0x7f5d2813dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2814e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28150a50 con 0x7f5d2815b530 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2814b0d0 con 0x7f5d280999d0 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2fd53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28150ca0 con 0x7f5d280afa50 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2e2f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2e2f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32838/0 (socket says 172.21.15.107:32838) 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.369+0000 7f5d2e2f0640 1 -- 172.21.15.107:0/306586121 learned_addr learned my addr 172.21.15.107:0/306586121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2daef640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d280999d0 0x7f5d2813dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2d2ee640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2814e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 -- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 msgr2=0x7f5d2814e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2814e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 -- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d280999d0 msgr2=0x7f5d2813dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d280999d0 0x7f5d2813dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 -- 172.21.15.107:0/306586121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240094d0 con 0x7f5d2815b530 2023-11-24T08:10:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2e2f0640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000cef0 tx=0x7f5d2000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000d840 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2000d9a0 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000dd50 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.370+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d28150f80 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.371+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28151540 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.372+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2804e8b0 con 0x7f5d2815b530 2023-11-24T08:10:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.373+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f5d2001e3b0 con 0x7f5d2815b530 2023-11-24T08:10:25.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.374+0000 7f5d1e7fc640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 0x7f5d0c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:25.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.374+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5d20088f90 con 0x7f5d2815b530 2023-11-24T08:10:25.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.374+0000 7f5d2daef640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 0x7f5d0c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:25.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.375+0000 7f5d2daef640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 0x7f5d0c06db70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d24036000 tx=0x7f5d2403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:25.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.378+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d200558c0 con 0x7f5d2815b530 2023-11-24T08:10:25.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:25.569+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5d280cc580 con 0x7f5d2815b530 2023-11-24T08:10:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.134+0000 7f5d1e7fc640 1 -- 172.21.15.107:0/306586121 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5d2001a070 con 0x7f5d2815b530 2023-11-24T08:10:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 msgr2=0x7f5d0c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 0x7f5d0c06db70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d24036000 tx=0x7f5d2403c040 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 msgr2=0x7f5d2814e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000cef0 tx=0x7f5d2000a710 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 shutdown_connections 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d0c06b6b0 0x7f5d0c06db70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d2815b530 0x7f5d2814e880 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.138+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d280afa50 0x7f5d2814e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.139+0000 7f5d2fd53640 1 --2- 172.21.15.107:0/306586121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d280999d0 0x7f5d2813dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.139+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 >> 172.21.15.107:0/306586121 conn(0x7f5d2806c090 msgr2=0x7f5d280b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:26.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.139+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 shutdown_connections 2023-11-24T08:10:26.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.139+0000 7f5d2fd53640 1 -- 172.21.15.107:0/306586121 wait complete. 2023-11-24T08:10:26.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:10:26.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.360+0000 7fec56a4b640 1 Processor -- start 2023-11-24T08:10:26.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.360+0000 7fec56a4b640 1 -- start start 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 0x7fec500c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec500ca940 0x7fec500c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 0x7fec500b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec500cb960 con 0x7fec500ca940 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec50161e40 con 0x7fec500b0e90 2023-11-24T08:10:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.361+0000 7fec56a4b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec50162120 con 0x7fec500cb580 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec500ca940 0x7fec500c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec500ca940 0x7fec500c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32848/0 (socket says 172.21.15.107:32848) 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 -- 172.21.15.107:0/3458396816 learned_addr learned my addr 172.21.15.107:0/3458396816 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec5624a640 1 --2- 172.21.15.107:0/3458396816 >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 0x7fec500b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55a49640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 0x7fec500c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 -- 172.21.15.107:0/3458396816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 msgr2=0x7fec500c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 0x7fec500c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 -- 172.21.15.107:0/3458396816 >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 msgr2=0x7fec500b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 --2- 172.21.15.107:0/3458396816 >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 0x7fec500b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55248640 1 -- 172.21.15.107:0/3458396816 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50162400 con 0x7fec500ca940 2023-11-24T08:10:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec55a49640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 0x7fec500c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:26.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.362+0000 7fec5624a640 1 --2- 172.21.15.107:0/3458396816 >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 0x7fec500b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.363+0000 7fec55248640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500ca940 0x7fec500c25d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fec50070310 tx=0x7fec4002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b797718c90e4819f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec54a47640 1 -- 172.21.15.107:0/3458396816 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40031880 con 0x7fec500ca940 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec54a47640 1 -- 172.21.15.107:0/3458396816 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec400319e0 con 0x7fec500ca940 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec54a47640 1 -- 172.21.15.107:0/3458396816 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40031d70 con 0x7fec500ca940 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 -- 172.21.15.107:0/3458396816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500ca940 msgr2=0x7fec500c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500ca940 0x7fec500c25d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fec50070310 tx=0x7fec4002b920 comp rx=0 tx=0).stop 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 -- 172.21.15.107:0/3458396816 shutdown_connections 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 --2- 172.21.15.107:0/3458396816 >> v2:172.21.15.120:3300/0 conn(0x7fec500b0e90 0x7fec500b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500ca940 0x7fec500c25d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 --2- 172.21.15.107:0/3458396816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500cb580 0x7fec500c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.364+0000 7fec56a4b640 1 -- 172.21.15.107:0/3458396816 >> 172.21.15.107:0/3458396816 conn(0x7fec500cbb40 msgr2=0x7fec500cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:26.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.365+0000 7fec56a4b640 1 -- 172.21.15.107:0/3458396816 shutdown_connections 2023-11-24T08:10:26.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.365+0000 7fec56a4b640 1 -- 172.21.15.107:0/3458396816 wait complete. 2023-11-24T08:10:26.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.366+0000 7fec56a4b640 1 Processor -- start 2023-11-24T08:10:26.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.366+0000 7fec56a4b640 1 -- start start 2023-11-24T08:10:26.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500ca940 0x7fec5015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec500cb580 0x7fec501f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5009c970 con 0x7fec500b0e90 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec500c6470 con 0x7fec500cb580 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.367+0000 7fec56a4b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5009c0d0 con 0x7fec500ca940 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32858/0 (socket says 172.21.15.107:32858) 2023-11-24T08:10:26.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 -- 172.21.15.107:0/258339075 learned_addr learned my addr 172.21.15.107:0/258339075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:26.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 -- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500ca940 msgr2=0x7fec5015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500ca940 0x7fec5015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 -- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec500cb580 msgr2=0x7fec501f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec5624a640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec500cb580 0x7fec501f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec500cb580 0x7fec501f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 -- 172.21.15.107:0/258339075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400094d0 con 0x7fec500b0e90 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.368+0000 7fec55a49640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c7f0 tx=0x7fec4c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.369+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c007820 con 0x7fec500b0e90 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.369+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c007980 con 0x7fec500b0e90 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.369+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c007cf0 con 0x7fec500b0e90 2023-11-24T08:10:26.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.369+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5009c3b0 con 0x7fec500b0e90 2023-11-24T08:10:26.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.369+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec50099b00 con 0x7fec500b0e90 2023-11-24T08:10:26.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.370+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5004e840 con 0x7fec500b0e90 2023-11-24T08:10:26.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.371+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fec4c0113f0 con 0x7fec500b0e90 2023-11-24T08:10:26.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.372+0000 7fec467fc640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 0x7fec3406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:26.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.372+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fec4c088360 con 0x7fec500b0e90 2023-11-24T08:10:26.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.372+0000 7fec55248640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 0x7fec3406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:26.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.373+0000 7fec55248640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 0x7fec3406db70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fec40036000 tx=0x7fec40030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:26.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.377+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec4c054cd0 con 0x7fec500b0e90 2023-11-24T08:10:26.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:26.603+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec500ca720 con 0x7fec500b0e90 2023-11-24T08:10:27.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.196+0000 7fec467fc640 1 -- 172.21.15.107:0/258339075 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7fec4c01f0c0 con 0x7fec500b0e90 2023-11-24T08:10:27.200 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 10 and data pool 11 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.202+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 msgr2=0x7fec3406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.202+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 0x7fec3406db70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fec40036000 tx=0x7fec40030000 comp rx=0 tx=0).stop 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.202+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 msgr2=0x7fec5009ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.202+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c7f0 tx=0x7fec4c00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 shutdown_connections 2023-11-24T08:10:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec3406b6b0 0x7fec3406db70 secure :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fec40036000 tx=0x7fec40030000 comp rx=0 tx=0).stop 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec500cb580 0x7fec501f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec500ca940 0x7fec5015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 --2- 172.21.15.107:0/258339075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500b0e90 0x7fec5009ba40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.203+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 >> 172.21.15.107:0/258339075 conn(0x7fec500cbb40 msgr2=0x7fec500ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.204+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 shutdown_connections 2023-11-24T08:10:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.204+0000 7fec56a4b640 1 -- 172.21.15.107:0/258339075 wait complete. 2023-11-24T08:10:27.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:27.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.417+0000 7f709b07e640 1 Processor -- start 2023-11-24T08:10:27.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.417+0000 7f709b07e640 1 -- start start 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709b07e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7094143870 0x7f7094146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709b07e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7094161510 con 0x7f70940b1180 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709b07e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70941617f0 con 0x7f7094161080 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709b07e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7094070460 con 0x7f7094143870 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709a87d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f70940b1180 0x7f70940cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59988/0 (socket says 172.21.15.107:59988) 2023-11-24T08:10:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709a87d640 1 -- 172.21.15.107:0/1917184881 learned_addr learned my addr 172.21.15.107:0/1917184881 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:27.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.419+0000 7f709a07c640 1 --2- 172.21.15.107:0/1917184881 >> v2:172.21.15.120:3300/0 conn(0x7f7094143870 0x7f7094146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014399476 0 0) 0x7f7094161510 con 0x7f70940b1180 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7080003520 con 0x7f70940b1180 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165445808 0 0) 0x7f7080003520 con 0x7f70940b1180 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 >> v2:172.21.15.120:3300/0 conn(0x7f7094143870 msgr2=0x7f7094146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 --2- 172.21.15.107:0/1917184881 >> v2:172.21.15.120:3300/0 conn(0x7f7094143870 0x7f7094146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 >> v1:172.21.15.120:6789/0 conn(0x7f7094161080 legacy=0x7f70940b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094161ad0 con 0x7f70940b1180 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f708c0019f0 con 0x7f70940b1180 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 >> v1:172.21.15.107:6789/0 conn(0x7f70940b1180 legacy=0x7f70940cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 0x7f7080003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800047c0 0x7f7080004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7080005180 0x7f708000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7094070460 con 0x7f70800039c0 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708000c080 con 0x7f70800047c0 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708000c240 con 0x7f7080005180 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.420+0000 7f709a07c640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 0x7f7080003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a87d640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7080005180 0x7f708000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a07c640 1 -- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7080005180 msgr2=0x7f708000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a07c640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7080005180 0x7f708000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709987b640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800047c0 0x7f7080004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a07c640 1 -- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800047c0 msgr2=0x7f7080004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:27.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a07c640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800047c0 0x7f7080004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a07c640 1 -- 172.21.15.107:0/1917184881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080003520 con 0x7f70800039c0 2023-11-24T08:10:27.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.421+0000 7f709a87d640 1 -- 172.21.15.107:0/1917184881 reap_dead start 2023-11-24T08:10:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.422+0000 7f709a07c640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 0x7f7080003de0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f7088009ac0 tx=0x7f70800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960e3ecd2a43b55e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.422+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708802c6b0 con 0x7f70800039c0 2023-11-24T08:10:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.422+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708802c810 con 0x7f70800039c0 2023-11-24T08:10:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.422+0000 7f709907a640 1 -- 172.21.15.107:0/1917184881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708802cba0 con 0x7f70800039c0 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 -- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 msgr2=0x7f7080003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 0x7f7080003de0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f7088009ac0 tx=0x7f70800057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 -- 172.21.15.107:0/1917184881 shutdown_connections 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 --2- 172.21.15.107:0/1917184881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70800039c0 0x7f7080003de0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 -- 172.21.15.107:0/1917184881 >> 172.21.15.107:0/1917184881 conn(0x7f70940cb830 msgr2=0x7f70940cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 -- 172.21.15.107:0/1917184881 shutdown_connections 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.423+0000 7f709b07e640 1 -- 172.21.15.107:0/1917184881 wait complete. 2023-11-24T08:10:27.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.425+0000 7f709b07e640 1 Processor -- start 2023-11-24T08:10:27.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.425+0000 7f709b07e640 1 -- start start 2023-11-24T08:10:27.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 0x7f70941344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709415ff70 0x7f70941603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70941447e0 con 0x7f70940b3230 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70941608f0 con 0x7f70800039c0 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709b07e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709413ed70 con 0x7f709415ff70 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709a87d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709a87d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32874/0 (socket says 172.21.15.107:32874) 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.426+0000 7f709a87d640 1 -- 172.21.15.107:0/2277291920 learned_addr learned my addr 172.21.15.107:0/2277291920 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:27.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a07c640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 0x7f70941344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709987b640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709415ff70 0x7f70941603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 -- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709415ff70 msgr2=0x7f70941603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709415ff70 0x7f70941603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 -- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 msgr2=0x7f70941344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 0x7f70941344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 -- 172.21.15.107:0/2277291920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094144a30 con 0x7f70940b3230 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a07c640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 0x7f70941344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709a87d640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f709406e250 tx=0x7f708c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00bdb0 con 0x7f70940b3230 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c004b60 con 0x7f70940b3230 2023-11-24T08:10:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c007070 con 0x7f70940b3230 2023-11-24T08:10:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7094144d10 con 0x7f70940b3230 2023-11-24T08:10:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.427+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70941452d0 con 0x7f70940b3230 2023-11-24T08:10:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.429+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709404e840 con 0x7f70940b3230 2023-11-24T08:10:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.430+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f708c0071d0 con 0x7f70940b3230 2023-11-24T08:10:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.431+0000 7f7086ffd640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 0x7f707806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.431+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f708c088380 con 0x7f70940b3230 2023-11-24T08:10:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.431+0000 7f709a07c640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 0x7f707806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.432+0000 7f709a07c640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 0x7f707806db70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7088036000 tx=0x7f7088030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.435+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f708c054dd0 con 0x7f70940b3230 2023-11-24T08:10:27.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.622+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f709406dd80 con 0x7f70940b3230 2023-11-24T08:10:27.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.625+0000 7f7086ffd640 1 -- 172.21.15.107:0/2277291920 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16581 (secure 0 0 0) 0x7f708c0583f0 con 0x7f70940b3230 2023-11-24T08:10:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:27.628 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":77,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:26.605609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:27.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.627+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 msgr2=0x7f707806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.627+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 0x7f707806db70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7088036000 tx=0x7f7088030000 comp rx=0 tx=0).stop 2023-11-24T08:10:27.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.627+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 msgr2=0x7f70940b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.627+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f709406e250 tx=0x7f708c00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 shutdown_connections 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707806b6b0 0x7f707806db70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70940b3230 0x7f70940b36b0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709415ff70 0x7f70941603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 --2- 172.21.15.107:0/2277291920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70800039c0 0x7f70941344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 >> 172.21.15.107:0/2277291920 conn(0x7f70940cb830 msgr2=0x7f70940afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:27.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 shutdown_connections 2023-11-24T08:10:27.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.628+0000 7f709b07e640 1 -- 172.21.15.107:0/2277291920 wait complete. 2023-11-24T08:10:27.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.844+0000 7f156247e640 1 Processor -- start 2023-11-24T08:10:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.844+0000 7f156247e640 1 -- start start 2023-11-24T08:10:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f156247e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f155c14b100 0x7f155c162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f156247e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f155c0cd160 0x7f155c0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f156247e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c162eb0 con 0x7f155c0703f0 2023-11-24T08:10:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f156247e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0cdb10 con 0x7f155c0cd160 2023-11-24T08:10:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f156247e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c1654a0 con 0x7f155c14b100 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f1561c7d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f155c0cd160 0x7f155c0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f1560c7b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f155c0703f0 0x7f155c165c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59992/0 (socket says 172.21.15.107:59992) 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.846+0000 7f1560c7b640 1 -- 172.21.15.107:0/3366290331 learned_addr learned my addr 172.21.15.107:0/3366290331 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f156147c640 1 --2- 172.21.15.107:0/3366290331 >> v2:172.21.15.120:3300/0 conn(0x7f155c14b100 0x7f155c162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f1561c7d640 1 -- 172.21.15.107:0/3366290331 >> v2:172.21.15.120:3300/0 conn(0x7f155c14b100 msgr2=0x7f155c162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f1561c7d640 1 --2- 172.21.15.107:0/3366290331 >> v2:172.21.15.120:3300/0 conn(0x7f155c14b100 0x7f155c162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f1561c7d640 1 -- 172.21.15.107:0/3366290331 >> v1:172.21.15.107:6789/0 conn(0x7f155c0703f0 legacy=0x7f155c165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f1553fff640 1 -- 172.21.15.107:0/3366290331 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291267595 0 0) 0x7f154c001110 con 0x7f155c0703f0 2023-11-24T08:10:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.847+0000 7f1561c7d640 1 -- 172.21.15.107:0/3366290331 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0ccf00 con 0x7f155c0cd160 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.848+0000 7f1561c7d640 1 --2- 172.21.15.107:0/3366290331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0cd160 0x7f155c0cd5d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f155c06e420 tx=0x7f155402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6da6c47a08eabeb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.848+0000 7f1553fff640 1 -- 172.21.15.107:0/3366290331 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155402cbf0 con 0x7f155c0cd160 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.848+0000 7f1553fff640 1 -- 172.21.15.107:0/3366290331 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155402cd50 con 0x7f155c0cd160 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.848+0000 7f1553fff640 1 -- 172.21.15.107:0/3366290331 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554034660 con 0x7f155c0cd160 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 -- 172.21.15.107:0/3366290331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0cd160 msgr2=0x7f155c0cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 --2- 172.21.15.107:0/3366290331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0cd160 0x7f155c0cd5d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f155c06e420 tx=0x7f155402be40 comp rx=0 tx=0).stop 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 -- 172.21.15.107:0/3366290331 shutdown_connections 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 --2- 172.21.15.107:0/3366290331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0cd160 0x7f155c0cd5d0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 --2- 172.21.15.107:0/3366290331 >> v2:172.21.15.120:3300/0 conn(0x7f155c14b100 0x7f155c162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 -- 172.21.15.107:0/3366290331 >> 172.21.15.107:0/3366290331 conn(0x7f155c0cb830 msgr2=0x7f155c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 -- 172.21.15.107:0/3366290331 shutdown_connections 2023-11-24T08:10:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.849+0000 7f156247e640 1 -- 172.21.15.107:0/3366290331 wait complete. 2023-11-24T08:10:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.851+0000 7f156247e640 1 Processor -- start 2023-11-24T08:10:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.851+0000 7f156247e640 1 -- start start 2023-11-24T08:10:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.851+0000 7f156247e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156247e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f155c0cd160 0x7f155c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156247e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f155c14b100 0x7f155c160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156247e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c139930 con 0x7f155c0703f0 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156247e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c139c10 con 0x7f155c14b100 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156247e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c139ef0 con 0x7f155c0cd160 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32902/0 (socket says 172.21.15.107:32902) 2023-11-24T08:10:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 -- 172.21.15.107:0/1912257057 learned_addr learned my addr 172.21.15.107:0/1912257057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 -- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f155c0cd160 msgr2=0x7f155c0ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f155c0cd160 0x7f155c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 -- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f155c14b100 msgr2=0x7f155c160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f155c14b100 0x7f155c160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 -- 172.21.15.107:0/1912257057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554009a70 con 0x7f155c0703f0 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.852+0000 7f156147c640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f155800d5f0 tx=0x7f155800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.853+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15580125f0 con 0x7f155c0703f0 2023-11-24T08:10:27.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.853+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1558012750 con 0x7f155c0703f0 2023-11-24T08:10:27.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.853+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558012b00 con 0x7f155c0703f0 2023-11-24T08:10:27.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.853+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c13a1d0 con 0x7f155c0703f0 2023-11-24T08:10:27.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.853+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c0ad7f0 con 0x7f155c0703f0 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.854+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c0ad9b0 con 0x7f155c0703f0 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.855+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f1558012c60 con 0x7f155c0703f0 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.856+0000 7f1551ffb640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 0x7f154006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.856+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1558088570 con 0x7f155c0703f0 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.856+0000 7f1560c7b640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 0x7f154006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.857+0000 7f1560c7b640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 0x7f154006db70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:27.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:27.860+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1558054ea0 con 0x7f155c0703f0 2023-11-24T08:10:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.053+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f155c04e840 con 0x7f155c0703f0 2023-11-24T08:10:28.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.056+0000 7f1551ffb640 1 -- 172.21.15.107:0/1912257057 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16581 (secure 0 0 0) 0x7f15580584c0 con 0x7f155c0703f0 2023-11-24T08:10:28.058 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:28.058 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":77,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:26.605609+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 msgr2=0x7f154006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 0x7f154006db70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c001310 comp rx=0 tx=0).stop 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 msgr2=0x7f155c0aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f155800d5f0 tx=0x7f155800dab0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 shutdown_connections 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f154006b6b0 0x7f154006db70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f155c14b100 0x7f155c160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f155c0cd160 0x7f155c0ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 --2- 172.21.15.107:0/1912257057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f155c0703f0 0x7f155c0aa620 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 >> 172.21.15.107:0/1912257057 conn(0x7f155c0cb830 msgr2=0x7f155c164cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:28.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 shutdown_connections 2023-11-24T08:10:28.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.059+0000 7f156247e640 1 -- 172.21.15.107:0/1912257057 wait complete. 2023-11-24T08:10:28.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:10:28.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.283+0000 7f0413961640 1 Processor -- start 2023-11-24T08:10:28.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.283+0000 7f0413961640 1 -- start start 2023-11-24T08:10:28.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.284+0000 7f0413961640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 0x7f040c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:28.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f0413961640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040c161a10 con 0x7f040c0cd2e0 2023-11-24T08:10:28.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f0413961640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040c161cf0 con 0x7f040c06e160 2023-11-24T08:10:28.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f0413961640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c161fd0 con 0x7f040c146eb0 2023-11-24T08:10:28.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f0411efe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f040c0cd2e0 0x7f040c070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35572/0 (socket says 172.21.15.107:35572) 2023-11-24T08:10:28.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f0411efe640 1 -- 172.21.15.107:0/118904123 learned_addr learned my addr 172.21.15.107:0/118904123 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:28.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.285+0000 7f04116fd640 1 --2- 172.21.15.107:0/118904123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 0x7f040c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539083432 0 0) 0x7f040c161a10 con 0x7f040c0cd2e0 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8003520 con 0x7f040c0cd2e0 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897624638 0 0) 0x7f040c161cf0 con 0x7f040c06e160 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f040c161a10 con 0x7f040c06e160 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595013773 0 0) 0x7f03f8003520 con 0x7f040c0cd2e0 2023-11-24T08:10:28.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f04116fd640 1 -- 172.21.15.107:0/118904123 >> v1:172.21.15.120:6789/0 conn(0x7f040c06e160 legacy=0x7f040c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f04116fd640 1 -- 172.21.15.107:0/118904123 >> v1:172.21.15.107:6789/0 conn(0x7f040c0cd2e0 legacy=0x7f040c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.286+0000 7f04116fd640 1 -- 172.21.15.107:0/118904123 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c070660 con 0x7f040c146eb0 2023-11-24T08:10:28.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.288+0000 7f04116fd640 1 --2- 172.21.15.107:0/118904123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 0x7f040c161150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0408009dc0 tx=0x7f040802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ec003f9a5af62e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:28.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.288+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404001670 con 0x7f040c146eb0 2023-11-24T08:10:28.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.288+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040802c910 con 0x7f040c146eb0 2023-11-24T08:10:28.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.288+0000 7f0403fff640 1 -- 172.21.15.107:0/118904123 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040802cca0 con 0x7f040c146eb0 2023-11-24T08:10:28.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 -- 172.21.15.107:0/118904123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 msgr2=0x7f040c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 --2- 172.21.15.107:0/118904123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 0x7f040c161150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0408009dc0 tx=0x7f040802bde0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 -- 172.21.15.107:0/118904123 shutdown_connections 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 --2- 172.21.15.107:0/118904123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c146eb0 0x7f040c161150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 -- 172.21.15.107:0/118904123 >> 172.21.15.107:0/118904123 conn(0x7f040c0b0c50 msgr2=0x7f040c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 -- 172.21.15.107:0/118904123 shutdown_connections 2023-11-24T08:10:28.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.289+0000 7f0413961640 1 -- 172.21.15.107:0/118904123 wait complete. 2023-11-24T08:10:28.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.290+0000 7f0413961640 1 Processor -- start 2023-11-24T08:10:28.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.290+0000 7f0413961640 1 -- start start 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c06e160 0x7f040c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f040c146eb0 0x7f040c0ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0b3d80 con 0x7f040c0cd2e0 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c15e190 con 0x7f040c146eb0 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.291+0000 7f0413961640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c15e470 con 0x7f040c06e160 2023-11-24T08:10:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54870/0 (socket says 172.21.15.107:54870) 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 -- 172.21.15.107:0/744884077 learned_addr learned my addr 172.21.15.107:0/744884077 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f04116fd640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c06e160 0x7f040c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 -- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c06e160 msgr2=0x7f040c15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c06e160 0x7f040c15db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 -- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f040c146eb0 msgr2=0x7f040c0ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f040c146eb0 0x7f040c0ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 -- 172.21.15.107:0/744884077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408009a70 con 0x7f040c0cd2e0 2023-11-24T08:10:28.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.292+0000 7f0410efc640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c450 tx=0x7f03fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.293+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc0052d0 con 0x7f040c0cd2e0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.293+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc005430 con 0x7f040c0cd2e0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.293+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c15e750 con 0x7f040c0cd2e0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.293+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc012870 con 0x7f040c0cd2e0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.293+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c15ed10 con 0x7f040c0cd2e0 2023-11-24T08:10:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.294+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c04e8b0 con 0x7f040c0cd2e0 2023-11-24T08:10:28.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.295+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f03fc0129d0 con 0x7f040c0cd2e0 2023-11-24T08:10:28.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.295+0000 7f0401ffb640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 0x7f03f006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:28.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.296+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03fc087c20 con 0x7f040c0cd2e0 2023-11-24T08:10:28.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.296+0000 7f04116fd640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 0x7f03f006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.296+0000 7f04116fd640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 0x7f03f006db70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0408036000 tx=0x7f0408030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:28.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.300+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03fc054610 con 0x7f040c0cd2e0 2023-11-24T08:10:28.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:28.511+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f040c09cf10 con 0x7f040c0cd2e0 2023-11-24T08:10:29.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.218+0000 7f0401ffb640 1 -- 172.21.15.107:0/744884077 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f03fc057c30 con 0x7f040c0cd2e0 2023-11-24T08:10:29.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.221+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 msgr2=0x7f03f006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.221+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 0x7f03f006db70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0408036000 tx=0x7f0408030000 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.221+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 msgr2=0x7f040c143400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.221+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c450 tx=0x7f03fc00c910 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 shutdown_connections 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006b6b0 0x7f03f006db70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f040c146eb0 0x7f040c0ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f040c0cd2e0 0x7f040c143400 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 --2- 172.21.15.107:0/744884077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040c06e160 0x7f040c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 >> 172.21.15.107:0/744884077 conn(0x7f040c0b0c50 msgr2=0x7f040c0cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 shutdown_connections 2023-11-24T08:10:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.222+0000 7f0413961640 1 -- 172.21.15.107:0/744884077 wait complete. 2023-11-24T08:10:29.244 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:10:29.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:10:29.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.454+0000 7fda8d9d5640 1 Processor -- start 2023-11-24T08:10:29.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.454+0000 7fda8d9d5640 1 -- start start 2023-11-24T08:10:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.456+0000 7fda8d9d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fda880cfb90 0x7fda88161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.456+0000 7fda8d9d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 0x7fda880cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.456+0000 7fda8d9d5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda881612f0 con 0x7fda88161fa0 2023-11-24T08:10:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.456+0000 7fda8d9d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda880cff70 con 0x7fda880cfb90 2023-11-24T08:10:29.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.456+0000 7fda8d9d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880afb20 con 0x7fda880b2400 2023-11-24T08:10:29.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda86ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fda880cfb90 0x7fda88161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda7e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fda88161fa0 0x7fda8806ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35578/0 (socket says 172.21.15.107:35578) 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda7e7fc640 1 -- 172.21.15.107:0/1084127982 learned_addr learned my addr 172.21.15.107:0/1084127982 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda86ffd640 1 -- 172.21.15.107:0/1084127982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 msgr2=0x7fda880cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda877fe640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 0x7fda880cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda86ffd640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 0x7fda880cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda867fc640 1 -- 172.21.15.107:0/1084127982 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663507903 0 0) 0x7fda881612f0 con 0x7fda88161fa0 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda86ffd640 1 -- 172.21.15.107:0/1084127982 >> v1:172.21.15.107:6789/0 conn(0x7fda88161fa0 legacy=0x7fda8806ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda86ffd640 1 -- 172.21.15.107:0/1084127982 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8806a850 con 0x7fda880cfb90 2023-11-24T08:10:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.457+0000 7fda877fe640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 0x7fda880cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.458+0000 7fda86ffd640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880cfb90 0x7fda88161a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fda780097a0 tx=0x7fda7802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561bf238b1088e67 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.458+0000 7fda867fc640 1 -- 172.21.15.107:0/1084127982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78030800 con 0x7fda880cfb90 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.458+0000 7fda867fc640 1 -- 172.21.15.107:0/1084127982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda78030960 con 0x7fda880cfb90 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.458+0000 7fda867fc640 1 -- 172.21.15.107:0/1084127982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78030cf0 con 0x7fda880cfb90 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 -- 172.21.15.107:0/1084127982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880cfb90 msgr2=0x7fda88161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880cfb90 0x7fda88161a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fda780097a0 tx=0x7fda7802b940 comp rx=0 tx=0).stop 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 -- 172.21.15.107:0/1084127982 shutdown_connections 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880b2400 0x7fda880cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/1084127982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880cfb90 0x7fda88161a60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 -- 172.21.15.107:0/1084127982 >> 172.21.15.107:0/1084127982 conn(0x7fda880999d0 msgr2=0x7fda880b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:29.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 -- 172.21.15.107:0/1084127982 shutdown_connections 2023-11-24T08:10:29.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.459+0000 7fda8d9d5640 1 -- 172.21.15.107:0/1084127982 wait complete. 2023-11-24T08:10:29.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.461+0000 7fda8d9d5640 1 Processor -- start 2023-11-24T08:10:29.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.461+0000 7fda8d9d5640 1 -- start start 2023-11-24T08:10:29.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880cfb90 0x7fda880c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda88158bf0 0x7fda880b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880c5160 con 0x7fda880b2400 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880c4550 con 0x7fda88158bf0 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda8d9d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880c4830 con 0x7fda880cfb90 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54896/0 (socket says 172.21.15.107:54896) 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.462+0000 7fda86ffd640 1 -- 172.21.15.107:0/3925820526 learned_addr learned my addr 172.21.15.107:0/3925820526 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 -- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880cfb90 msgr2=0x7fda880c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda7e7fc640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880cfb90 0x7fda880c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda877fe640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda88158bf0 0x7fda880b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880cfb90 0x7fda880c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 -- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda88158bf0 msgr2=0x7fda880b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda88158bf0 0x7fda880b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 -- 172.21.15.107:0/3925820526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78009400 con 0x7fda880b2400 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda86ffd640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fda7802bf50 tx=0x7fda78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7803e040 con 0x7fda880b2400 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda78039d00 con 0x7fda880b2400 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78044630 con 0x7fda880b2400 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda880c4b10 con 0x7fda880b2400 2023-11-24T08:10:29.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.463+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8814ba40 con 0x7fda880b2400 2023-11-24T08:10:29.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.465+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8804e8b0 con 0x7fda880b2400 2023-11-24T08:10:29.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.465+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fda78043390 con 0x7fda880b2400 2023-11-24T08:10:29.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.466+0000 7fda84ff9640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 0x7fda6806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:29.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.466+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda780aa380 con 0x7fda880b2400 2023-11-24T08:10:29.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.467+0000 7fda7e7fc640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 0x7fda6806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:29.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.467+0000 7fda7e7fc640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 0x7fda6806db70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca590 tx=0x7fda70009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:29.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.471+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda78076e50 con 0x7fda880b2400 2023-11-24T08:10:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:29.697+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fda880cd810 con 0x7fda880b2400 2023-11-24T08:10:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.227+0000 7fda84ff9640 1 -- 172.21.15.107:0/3925820526 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda7807a470 con 0x7fda880b2400 2023-11-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 msgr2=0x7fda6806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 0x7fda6806db70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca590 tx=0x7fda70009040 comp rx=0 tx=0).stop 2023-11-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 msgr2=0x7fda8814b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fda7802bf50 tx=0x7fda78004730 comp rx=0 tx=0).stop 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 shutdown_connections 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda6806b6b0 0x7fda6806db70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda88158bf0 0x7fda880b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda880cfb90 0x7fda880c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 --2- 172.21.15.107:0/3925820526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda880b2400 0x7fda8814b500 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 >> 172.21.15.107:0/3925820526 conn(0x7fda880999d0 msgr2=0x7fda8815d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.231+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 shutdown_connections 2023-11-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.232+0000 7fda8d9d5640 1 -- 172.21.15.107:0/3925820526 wait complete. 2023-11-24T08:10:30.253 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:10:30.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:30.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.466+0000 7f18dd4b4640 1 Processor -- start 2023-11-24T08:10:30.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.467+0000 7f18dd4b4640 1 -- start start 2023-11-24T08:10:30.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.468+0000 7f18dd4b4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f18d815c7f0 0x7f18d806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.468+0000 7f18dd4b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 0x7f18d80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.468+0000 7f18dd4b4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d80cece0 con 0x7f18d80cdfb0 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.468+0000 7f18dd4b4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d8156010 con 0x7f18d815c7f0 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.468+0000 7f18dd4b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d806ce30 con 0x7f18d80cf450 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f18d815c7f0 0x7f18d806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f18d80cdfb0 0x7f18d80ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35582/0 (socket says 172.21.15.107:35582) 2023-11-24T08:10:30.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d77fe640 1 -- 172.21.15.107:0/3388114280 learned_addr learned my addr 172.21.15.107:0/3388114280 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d67fc640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 0x7f18d80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d6ffd640 1 -- 172.21.15.107:0/3388114280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 msgr2=0x7f18d80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d6ffd640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 0x7f18d80cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d6ffd640 1 -- 172.21.15.107:0/3388114280 >> v1:172.21.15.107:6789/0 conn(0x7f18d80cdfb0 legacy=0x7f18d80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d5ffb640 1 -- 172.21.15.107:0/3388114280 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776665580 0 0) 0x7f18d80cece0 con 0x7f18d80cdfb0 2023-11-24T08:10:30.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d6ffd640 1 -- 172.21.15.107:0/3388114280 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d806d040 con 0x7f18d815c7f0 2023-11-24T08:10:30.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.469+0000 7f18d67fc640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 0x7f18d80cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:30.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.470+0000 7f18d6ffd640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d815c7f0 0x7f18d806db20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009d90 tx=0x7f18c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946611e220aa3517 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18d5ffb640 1 -- 172.21.15.107:0/3388114280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c802cc50 con 0x7f18d815c7f0 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18d5ffb640 1 -- 172.21.15.107:0/3388114280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c802cdb0 con 0x7f18d815c7f0 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18d5ffb640 1 -- 172.21.15.107:0/3388114280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8034660 con 0x7f18d815c7f0 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 -- 172.21.15.107:0/3388114280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d815c7f0 msgr2=0x7f18d806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d815c7f0 0x7f18d806db20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009d90 tx=0x7f18c802bea0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 -- 172.21.15.107:0/3388114280 shutdown_connections 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d80cf450 0x7f18d80cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/3388114280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d815c7f0 0x7f18d806db20 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 -- 172.21.15.107:0/3388114280 >> 172.21.15.107:0/3388114280 conn(0x7f18d806a920 msgr2=0x7f18d806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:30.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 -- 172.21.15.107:0/3388114280 shutdown_connections 2023-11-24T08:10:30.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.471+0000 7f18dd4b4640 1 -- 172.21.15.107:0/3388114280 wait complete. 2023-11-24T08:10:30.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.473+0000 7f18dd4b4640 1 Processor -- start 2023-11-24T08:10:30.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.473+0000 7f18dd4b4640 1 -- start start 2023-11-24T08:10:30.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18d80cdfb0 0x7f18d8151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d815c7f0 0x7f18d815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8151170 con 0x7f18d80cf450 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d815aa90 con 0x7f18d80cdfb0 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.474+0000 7f18dd4b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8159ba0 con 0x7f18d815c7f0 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54920/0 (socket says 172.21.15.107:54920) 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 -- 172.21.15.107:0/4093484071 learned_addr learned my addr 172.21.15.107:0/4093484071 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 -- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d815c7f0 msgr2=0x7f18d815af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d815c7f0 0x7f18d815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 -- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18d80cdfb0 msgr2=0x7f18d8151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18d80cdfb0 0x7f18d8151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 -- 172.21.15.107:0/4093484071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8009a70 con 0x7f18d80cf450 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18d67fc640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f18c000c7f0 tx=0x7f18c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.475+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0007a30 con 0x7f18d80cf450 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.476+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0007b90 con 0x7f18d80cf450 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.476+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c000ce40 con 0x7f18d80cf450 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.476+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d8159e80 con 0x7f18d80cf450 2023-11-24T08:10:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.476+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d815a440 con 0x7f18d80cf450 2023-11-24T08:10:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.477+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d8067470 con 0x7f18d80cf450 2023-11-24T08:10:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.478+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f18c00113f0 con 0x7f18d80cf450 2023-11-24T08:10:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.479+0000 7f18cf7fe640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 0x7f18b806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.479+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18c0088280 con 0x7f18d80cf450 2023-11-24T08:10:30.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.479+0000 7f18d6ffd640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 0x7f18b806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.480+0000 7f18d6ffd640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 0x7f18b806db70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f18c8036000 tx=0x7f18c802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.483+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18c0054bb0 con 0x7f18d80cf450 2023-11-24T08:10:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.693+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d815abd0 con 0x7f18d80cf450 2023-11-24T08:10:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.694+0000 7f18cf7fe640 1 -- 172.21.15.107:0/4093484071 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f18c001f090 con 0x7f18d80cf450 2023-11-24T08:10:30.696 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:30.697 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:10:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 41 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 msgr2=0x7f18b806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 0x7f18b806db70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f18c8036000 tx=0x7f18c802e000 comp rx=0 tx=0).stop 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 msgr2=0x7f18d8150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f18c000c7f0 tx=0x7f18c000cbf0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 shutdown_connections 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f18b806b6b0 0x7f18b806db70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18d815c7f0 0x7f18d815af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18d80cf450 0x7f18d8150c30 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 --2- 172.21.15.107:0/4093484071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18d80cdfb0 0x7f18d8151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.697+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 >> 172.21.15.107:0/4093484071 conn(0x7f18d806a920 msgr2=0x7f18d8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.698+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 shutdown_connections 2023-11-24T08:10:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.698+0000 7f18dd4b4640 1 -- 172.21.15.107:0/4093484071 wait complete. 2023-11-24T08:10:30.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:10:30.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.927+0000 7f67371ac640 1 Processor -- start 2023-11-24T08:10:30.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.927+0000 7f67371ac640 1 -- start start 2023-11-24T08:10:30.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.928+0000 7f67371ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f673013d120 0x7f673015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.928+0000 7f67371ac640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300cec00 con 0x7f673006d6c0 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67371ac640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300b1890 con 0x7f673006db20 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67371ac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673013f9e0 con 0x7f673013d120 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f673013d120 0x7f673015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f673013d120 0x7f673015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54922/0 (socket says 172.21.15.107:54922) 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 -- 172.21.15.107:0/4159929899 learned_addr learned my addr 172.21.15.107:0/4159929899 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67359a9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f673006d6c0 0x7f67300b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35592/0 (socket says 172.21.15.107:35592) 2023-11-24T08:10:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 -- 172.21.15.107:0/4159929899 >> v1:172.21.15.120:6789/0 conn(0x7f673006db20 legacy=0x7f67300b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 -- 172.21.15.107:0/4159929899 >> v1:172.21.15.107:6789/0 conn(0x7f673006d6c0 legacy=0x7f67300b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.929+0000 7f67361aa640 1 -- 172.21.15.107:0/4159929899 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673013fcc0 con 0x7f673013d120 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.930+0000 7f67361aa640 1 --2- 172.21.15.107:0/4159929899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f673015af40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f672c009870 tx=0x7f672c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2334861c0ec257 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67351a8640 1 -- 172.21.15.107:0/4159929899 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c031870 con 0x7f673013d120 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67351a8640 1 -- 172.21.15.107:0/4159929899 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c0319d0 con 0x7f673013d120 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67351a8640 1 -- 172.21.15.107:0/4159929899 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c031d60 con 0x7f673013d120 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67371ac640 1 -- 172.21.15.107:0/4159929899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 msgr2=0x7f673015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67371ac640 1 --2- 172.21.15.107:0/4159929899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f673015af40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f672c009870 tx=0x7f672c02b910 comp rx=0 tx=0).stop 2023-11-24T08:10:30.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67371ac640 1 -- 172.21.15.107:0/4159929899 shutdown_connections 2023-11-24T08:10:30.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67371ac640 1 --2- 172.21.15.107:0/4159929899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f673015af40 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.931+0000 7f67371ac640 1 -- 172.21.15.107:0/4159929899 >> 172.21.15.107:0/4159929899 conn(0x7f67300cb830 msgr2=0x7f67300cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:30.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.932+0000 7f67371ac640 1 -- 172.21.15.107:0/4159929899 shutdown_connections 2023-11-24T08:10:30.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.932+0000 7f67371ac640 1 -- 172.21.15.107:0/4159929899 wait complete. 2023-11-24T08:10:30.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.934+0000 7f67371ac640 1 Processor -- start 2023-11-24T08:10:30.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.934+0000 7f67371ac640 1 -- start start 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673006d6c0 0x7f6730158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 0x7f673015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301f4390 con 0x7f673013d120 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673015c370 con 0x7f673006db20 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67371ac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6730159490 con 0x7f673006d6c0 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67369ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67369ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54938/0 (socket says 172.21.15.107:54938) 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.935+0000 7f67369ab640 1 -- 172.21.15.107:0/937349090 learned_addr learned my addr 172.21.15.107:0/937349090 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67361aa640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673006d6c0 0x7f6730158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67359a9640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 0x7f673015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 -- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673006d6c0 msgr2=0x7f6730158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673006d6c0 0x7f6730158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 -- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 msgr2=0x7f673015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 0x7f673015be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 -- 172.21.15.107:0/937349090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0094d0 con 0x7f673013d120 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67359a9640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 0x7f673015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67369ab640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f672800c7b0 tx=0x7f672800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728003d80 con 0x7f673013d120 2023-11-24T08:10:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728005070 con 0x7f673013d120 2023-11-24T08:10:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728005420 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.936+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6730159770 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.937+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6730159d30 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.938+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67301f4540 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.939+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f6728005580 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.939+0000 7f6726ffd640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 0x7f671406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.940+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6728087550 con 0x7f673013d120 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.940+0000 7f67361aa640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 0x7f671406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.941+0000 7f67361aa640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 0x7f671406db70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f672c036000 tx=0x7f672c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:30.944+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67301f4540 con 0x7f673013d120 2023-11-24T08:10:31.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.123+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67301f4540 con 0x7f673013d120 2023-11-24T08:10:31.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.126+0000 7f6726ffd640 1 -- 172.21.15.107:0/937349090 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f67301f4540 con 0x7f673013d120 2023-11-24T08:10:31.128 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:31.129 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":78,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:10:28.195245+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:10:31.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.128+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 msgr2=0x7f671406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.128+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 0x7f671406db70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f672c036000 tx=0x7f672c03c040 comp rx=0 tx=0).stop 2023-11-24T08:10:31.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.128+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 msgr2=0x7f67300b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.128+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f672800c7b0 tx=0x7f672800cc70 comp rx=0 tx=0).stop 2023-11-24T08:10:31.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 shutdown_connections 2023-11-24T08:10:31.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f671406b6b0 0x7f671406db70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f673013d120 0x7f67300b0f00 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f673006db20 0x7f673015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 --2- 172.21.15.107:0/937349090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673006d6c0 0x7f6730158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 >> 172.21.15.107:0/937349090 conn(0x7f67300cb830 msgr2=0x7f67300cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 shutdown_connections 2023-11-24T08:10:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.129+0000 7f67371ac640 1 -- 172.21.15.107:0/937349090 wait complete. 2023-11-24T08:10:31.152 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:10:31.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:10:31.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.356+0000 7f76ccd5a640 1 Processor -- start 2023-11-24T08:10:31.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.357+0000 7f76ccd5a640 1 -- start start 2023-11-24T08:10:31.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76ccd5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76c81609e0 0x7f76c806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76ccd5a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f76c80d31a0 0x7f76c80d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76ccd5a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c806c1f0 con 0x7f76c80ce010 2023-11-24T08:10:31.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76ccd5a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c80ceb40 con 0x7f76c81609e0 2023-11-24T08:10:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76ccd5a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c80d1070 con 0x7f76c80d31a0 2023-11-24T08:10:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76c81609e0 0x7f76c806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76c81609e0 0x7f76c806bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54946/0 (socket says 172.21.15.107:54946) 2023-11-24T08:10:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.358+0000 7f76c659c640 1 -- 172.21.15.107:0/3700418696 learned_addr learned my addr 172.21.15.107:0/3700418696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.359+0000 7f76c659c640 1 -- 172.21.15.107:0/3700418696 >> v2:172.21.15.120:3300/0 conn(0x7f76c80d31a0 msgr2=0x7f76c80d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.359+0000 7f76c659c640 1 --2- 172.21.15.107:0/3700418696 >> v2:172.21.15.120:3300/0 conn(0x7f76c80d31a0 0x7f76c80d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.359+0000 7f76c659c640 1 -- 172.21.15.107:0/3700418696 >> v1:172.21.15.120:6789/0 conn(0x7f76c80ce010 legacy=0x7f76c80ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.359+0000 7f76c659c640 1 -- 172.21.15.107:0/3700418696 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c80d3b70 con 0x7f76c81609e0 2023-11-24T08:10:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.360+0000 7f76c659c640 1 --2- 172.21.15.107:0/3700418696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c81609e0 0x7f76c806bcb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f76c80d43c0 tx=0x7f76b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6c447cb25a312e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.360+0000 7f76c5d9b640 1 -- 172.21.15.107:0/3700418696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8031870 con 0x7f76c81609e0 2023-11-24T08:10:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.360+0000 7f76c5d9b640 1 -- 172.21.15.107:0/3700418696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b80319d0 con 0x7f76c81609e0 2023-11-24T08:10:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.360+0000 7f76c5d9b640 1 -- 172.21.15.107:0/3700418696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8031d60 con 0x7f76c81609e0 2023-11-24T08:10:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 -- 172.21.15.107:0/3700418696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c81609e0 msgr2=0x7f76c806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/3700418696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c81609e0 0x7f76c806bcb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f76c80d43c0 tx=0x7f76b802b910 comp rx=0 tx=0).stop 2023-11-24T08:10:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 -- 172.21.15.107:0/3700418696 shutdown_connections 2023-11-24T08:10:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/3700418696 >> v2:172.21.15.120:3300/0 conn(0x7f76c80d31a0 0x7f76c80d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/3700418696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c81609e0 0x7f76c806bcb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 -- 172.21.15.107:0/3700418696 >> 172.21.15.107:0/3700418696 conn(0x7f76c808af60 msgr2=0x7f76c809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 -- 172.21.15.107:0/3700418696 shutdown_connections 2023-11-24T08:10:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.361+0000 7f76ccd5a640 1 -- 172.21.15.107:0/3700418696 wait complete. 2023-11-24T08:10:31.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.363+0000 7f76ccd5a640 1 Processor -- start 2023-11-24T08:10:31.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.363+0000 7f76ccd5a640 1 -- start start 2023-11-24T08:10:31.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76c80ce010 0x7f76c809f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76c81609e0 0x7f76c80b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c80b9220 con 0x7f76c80d31a0 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c809f890 con 0x7f76c80ce010 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76ccd5a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c809fb70 con 0x7f76c81609e0 2023-11-24T08:10:31.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54954/0 (socket says 172.21.15.107:54954) 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76bffff640 1 -- 172.21.15.107:0/2201564643 learned_addr learned my addr 172.21.15.107:0/2201564643 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76c659c640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76c80ce010 0x7f76c809f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76c6d9d640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76c81609e0 0x7f76c80b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.364+0000 7f76bffff640 1 -- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76c81609e0 msgr2=0x7f76c80b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76bffff640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76c81609e0 0x7f76c80b82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76bffff640 1 -- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76c80ce010 msgr2=0x7f76c809f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76bffff640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76c80ce010 0x7f76c809f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76bffff640 1 -- 172.21.15.107:0/2201564643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b80094d0 con 0x7f76c80d31a0 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76bffff640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f76b000b4a0 tx=0x7f76b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0010610 con 0x7f76c80d31a0 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0010770 con 0x7f76c80d31a0 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c809fe50 con 0x7f76c80d31a0 2023-11-24T08:10:31.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0010b20 con 0x7f76c80d31a0 2023-11-24T08:10:31.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.365+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c80a0410 con 0x7f76c80d31a0 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.366+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c80a0630 con 0x7f76c80d31a0 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.367+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f76b0010c80 con 0x7f76c80d31a0 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.368+0000 7f76beffd640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 0x7f76a806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.368+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76b0087ca0 con 0x7f76c80d31a0 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.368+0000 7f76c659c640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 0x7f76a806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.369+0000 7f76c659c640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 0x7f76a806db70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036000 tx=0x7f76b803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.373+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76b0054630 con 0x7f76c80d31a0 2023-11-24T08:10:31.553 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:10:31.549+0000 7f38fd695640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:10:31.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.562+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76c809cd10 con 0x7f76a806b6b0 2023-11-24T08:10:31.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.574+0000 7f76beffd640 1 -- 172.21.15.107:0/2201564643 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287078 (secure 0 0 0) 0x7f76c809cd10 con 0x7f76a806b6b0 2023-11-24T08:10:31.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:10:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 msgr2=0x7f76a806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 0x7f76a806db70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036000 tx=0x7f76b803c040 comp rx=0 tx=0).stop 2023-11-24T08:10:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 msgr2=0x7f76c80b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f76b000b4a0 tx=0x7f76b000b960 comp rx=0 tx=0).stop 2023-11-24T08:10:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 shutdown_connections 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76a806b6b0 0x7f76a806db70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76c81609e0 0x7f76c80b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76c80d31a0 0x7f76c80b8ce0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 --2- 172.21.15.107:0/2201564643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76c80ce010 0x7f76c809f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 >> 172.21.15.107:0/2201564643 conn(0x7f76c808af60 msgr2=0x7f76c80ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.579+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 shutdown_connections 2023-11-24T08:10:31.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.580+0000 7f76ccd5a640 1 -- 172.21.15.107:0/2201564643 wait complete. 2023-11-24T08:10:31.601 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2023-11-24T08:10:31.537894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258000,"kb_used_data":8552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716720,"statfs":{"total":773094113280,"available":772829921280,"internally_reserved":0,"allocated":8757248,"data_stored":4389132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":255315518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002202"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211772+0000","last_change":"2023-11-24T08:10:27.225266+0000","last_active":"2023-11-24T08:10:28.211772+0000","last_peered":"2023-11-24T08:10:28.211772+0000","last_clean":"2023-11-24T08:10:28.211772+0000","last_became_active":"2023-11-24T08:10:27.224675+0000","last_became_peered":"2023-11-24T08:10:27.224675+0000","last_unstale":"2023-11-24T08:10:28.211772+0000","last_undegraded":"2023-11-24T08:10:28.211772+0000","last_fullsized":"2023-11-24T08:10:28.211772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:12:21.351644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214517+0000","last_change":"2023-11-24T08:10:27.212593+0000","last_active":"2023-11-24T08:10:28.214517+0000","last_peered":"2023-11-24T08:10:28.214517+0000","last_clean":"2023-11-24T08:10:28.214517+0000","last_became_active":"2023-11-24T08:10:27.212131+0000","last_became_peered":"2023-11-24T08:10:27.212131+0000","last_unstale":"2023-11-24T08:10:28.214517+0000","last_undegraded":"2023-11-24T08:10:28.214517+0000","last_fullsized":"2023-11-24T08:10:28.214517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:05.594017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226337+0000","last_change":"2023-11-24T08:10:27.215218+0000","last_active":"2023-11-24T08:10:27.226337+0000","last_peered":"2023-11-24T08:10:27.226337+0000","last_clean":"2023-11-24T08:10:27.226337+0000","last_became_active":"2023-11-24T08:10:27.214907+0000","last_became_peered":"2023-11-24T08:10:27.214907+0000","last_unstale":"2023-11-24T08:10:27.226337+0000","last_undegraded":"2023-11-24T08:10:27.226337+0000","last_fullsized":"2023-11-24T08:10:27.226337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:40:37.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.212926+0000","last_change":"2023-11-24T08:10:27.208149+0000","last_active":"2023-11-24T08:10:27.212926+0000","last_peered":"2023-11-24T08:10:27.212926+0000","last_clean":"2023-11-24T08:10:27.212926+0000","last_became_active":"2023-11-24T08:10:27.206429+0000","last_became_peered":"2023-11-24T08:10:27.206429+0000","last_unstale":"2023-11-24T08:10:27.212926+0000","last_undegraded":"2023-11-24T08:10:27.212926+0000","last_fullsized":"2023-11-24T08:10:27.212926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:02.751811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.223611+0000","last_change":"2023-11-24T08:10:27.213601+0000","last_active":"2023-11-24T08:10:27.223611+0000","last_peered":"2023-11-24T08:10:27.223611+0000","last_clean":"2023-11-24T08:10:27.223611+0000","last_became_active":"2023-11-24T08:10:27.213335+0000","last_became_peered":"2023-11-24T08:10:27.213335+0000","last_unstale":"2023-11-24T08:10:27.223611+0000","last_undegraded":"2023-11-24T08:10:27.223611+0000","last_fullsized":"2023-11-24T08:10:27.223611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:12:52.603647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214526+0000","last_change":"2023-11-24T08:10:27.214997+0000","last_active":"2023-11-24T08:10:28.214526+0000","last_peered":"2023-11-24T08:10:28.214526+0000","last_clean":"2023-11-24T08:10:28.214526+0000","last_became_active":"2023-11-24T08:10:27.214723+0000","last_became_peered":"2023-11-24T08:10:27.214723+0000","last_unstale":"2023-11-24T08:10:28.214526+0000","last_undegraded":"2023-11-24T08:10:28.214526+0000","last_fullsized":"2023-11-24T08:10:28.214526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:37.030588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.211346+0000","last_change":"2023-11-24T08:10:27.208106+0000","last_active":"2023-11-24T08:10:27.211346+0000","last_peered":"2023-11-24T08:10:27.211346+0000","last_clean":"2023-11-24T08:10:27.211346+0000","last_became_active":"2023-11-24T08:10:27.207812+0000","last_became_peered":"2023-11-24T08:10:27.207812+0000","last_unstale":"2023-11-24T08:10:27.211346+0000","last_undegraded":"2023-11-24T08:10:27.211346+0000","last_fullsized":"2023-11-24T08:10:27.211346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:37.873957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214122+0000","last_change":"2023-11-24T08:10:27.199753+0000","last_active":"2023-11-24T08:10:28.214122+0000","last_peered":"2023-11-24T08:10:28.214122+0000","last_clean":"2023-11-24T08:10:28.214122+0000","last_became_active":"2023-11-24T08:10:27.199213+0000","last_became_peered":"2023-11-24T08:10:27.199213+0000","last_unstale":"2023-11-24T08:10:28.214122+0000","last_undegraded":"2023-11-24T08:10:28.214122+0000","last_fullsized":"2023-11-24T08:10:28.214122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:56.330516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230048+0000","last_change":"2023-11-24T08:10:27.224075+0000","last_active":"2023-11-24T08:10:27.230048+0000","last_peered":"2023-11-24T08:10:27.230048+0000","last_clean":"2023-11-24T08:10:27.230048+0000","last_became_active":"2023-11-24T08:10:27.223797+0000","last_became_peered":"2023-11-24T08:10:27.223797+0000","last_unstale":"2023-11-24T08:10:27.230048+0000","last_undegraded":"2023-11-24T08:10:27.230048+0000","last_fullsized":"2023-11-24T08:10:27.230048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:55.886735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221727+0000","last_change":"2023-11-24T08:10:27.217542+0000","last_active":"2023-11-24T08:10:27.221727+0000","last_peered":"2023-11-24T08:10:27.221727+0000","last_clean":"2023-11-24T08:10:27.221727+0000","last_became_active":"2023-11-24T08:10:27.217132+0000","last_became_peered":"2023-11-24T08:10:27.217132+0000","last_unstale":"2023-11-24T08:10:27.221727+0000","last_undegraded":"2023-11-24T08:10:27.221727+0000","last_fullsized":"2023-11-24T08:10:27.221727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:24.165893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211169+0000","last_change":"2023-11-24T08:10:27.223066+0000","last_active":"2023-11-24T08:10:28.211169+0000","last_peered":"2023-11-24T08:10:28.211169+0000","last_clean":"2023-11-24T08:10:28.211169+0000","last_became_active":"2023-11-24T08:10:27.221960+0000","last_became_peered":"2023-11-24T08:10:27.221960+0000","last_unstale":"2023-11-24T08:10:28.211169+0000","last_undegraded":"2023-11-24T08:10:28.211169+0000","last_fullsized":"2023-11-24T08:10:28.211169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.223992+0000","last_change":"2023-11-24T08:10:27.212403+0000","last_active":"2023-11-24T08:10:27.223992+0000","last_peered":"2023-11-24T08:10:27.223992+0000","last_clean":"2023-11-24T08:10:27.223992+0000","last_became_active":"2023-11-24T08:10:27.207943+0000","last_became_peered":"2023-11-24T08:10:27.207943+0000","last_unstale":"2023-11-24T08:10:27.223992+0000","last_undegraded":"2023-11-24T08:10:27.223992+0000","last_fullsized":"2023-11-24T08:10:27.223992+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:35.135070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276490+0000","last_change":"2023-11-24T08:10:27.218397+0000","last_active":"2023-11-24T08:10:28.276490+0000","last_peered":"2023-11-24T08:10:28.276490+0000","last_clean":"2023-11-24T08:10:28.276490+0000","last_became_active":"2023-11-24T08:10:27.218062+0000","last_became_peered":"2023-11-24T08:10:27.218062+0000","last_unstale":"2023-11-24T08:10:28.276490+0000","last_undegraded":"2023-11-24T08:10:28.276490+0000","last_fullsized":"2023-11-24T08:10:28.276490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:23:15.270466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226882+0000","last_change":"2023-11-24T08:10:27.218853+0000","last_active":"2023-11-24T08:10:27.226882+0000","last_peered":"2023-11-24T08:10:27.226882+0000","last_clean":"2023-11-24T08:10:27.226882+0000","last_became_active":"2023-11-24T08:10:27.213120+0000","last_became_peered":"2023-11-24T08:10:27.213120+0000","last_unstale":"2023-11-24T08:10:27.226882+0000","last_undegraded":"2023-11-24T08:10:27.226882+0000","last_fullsized":"2023-11-24T08:10:27.226882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:14.263852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276496+0000","last_change":"2023-11-24T08:10:27.205790+0000","last_active":"2023-11-24T08:10:28.276496+0000","last_peered":"2023-11-24T08:10:28.276496+0000","last_clean":"2023-11-24T08:10:28.276496+0000","last_became_active":"2023-11-24T08:10:27.205501+0000","last_became_peered":"2023-11-24T08:10:27.205501+0000","last_unstale":"2023-11-24T08:10:28.276496+0000","last_undegraded":"2023-11-24T08:10:28.276496+0000","last_fullsized":"2023-11-24T08:10:28.276496+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:35:49.516231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222734+0000","last_change":"2023-11-24T08:10:27.218897+0000","last_active":"2023-11-24T08:10:27.222734+0000","last_peered":"2023-11-24T08:10:27.222734+0000","last_clean":"2023-11-24T08:10:27.222734+0000","last_became_active":"2023-11-24T08:10:27.218492+0000","last_became_peered":"2023-11-24T08:10:27.218492+0000","last_unstale":"2023-11-24T08:10:27.222734+0000","last_undegraded":"2023-11-24T08:10:27.222734+0000","last_fullsized":"2023-11-24T08:10:27.222734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:35.252989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.231518+0000","last_change":"2023-11-24T08:10:27.230806+0000","last_active":"2023-11-24T08:10:27.231518+0000","last_peered":"2023-11-24T08:10:27.231518+0000","last_clean":"2023-11-24T08:10:27.231518+0000","last_became_active":"2023-11-24T08:10:27.230466+0000","last_became_peered":"2023-11-24T08:10:27.230466+0000","last_unstale":"2023-11-24T08:10:27.231518+0000","last_undegraded":"2023-11-24T08:10:27.231518+0000","last_fullsized":"2023-11-24T08:10:27.231518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:54.398110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.210048+0000","last_change":"2023-11-24T08:10:27.206081+0000","last_active":"2023-11-24T08:10:27.210048+0000","last_peered":"2023-11-24T08:10:27.210048+0000","last_clean":"2023-11-24T08:10:27.210048+0000","last_became_active":"2023-11-24T08:10:27.202268+0000","last_became_peered":"2023-11-24T08:10:27.202268+0000","last_unstale":"2023-11-24T08:10:27.210048+0000","last_undegraded":"2023-11-24T08:10:27.210048+0000","last_fullsized":"2023-11-24T08:10:27.210048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:26.107488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214410+0000","last_change":"2023-11-24T08:10:27.201937+0000","last_active":"2023-11-24T08:10:28.214410+0000","last_peered":"2023-11-24T08:10:28.214410+0000","last_clean":"2023-11-24T08:10:28.214410+0000","last_became_active":"2023-11-24T08:10:27.198384+0000","last_became_peered":"2023-11-24T08:10:27.198384+0000","last_unstale":"2023-11-24T08:10:28.214410+0000","last_undegraded":"2023-11-24T08:10:28.214410+0000","last_fullsized":"2023-11-24T08:10:28.214410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:22:08.878591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276486+0000","last_change":"2023-11-24T08:10:27.217630+0000","last_active":"2023-11-24T08:10:28.276486+0000","last_peered":"2023-11-24T08:10:28.276486+0000","last_clean":"2023-11-24T08:10:28.276486+0000","last_became_active":"2023-11-24T08:10:27.217348+0000","last_became_peered":"2023-11-24T08:10:27.217348+0000","last_unstale":"2023-11-24T08:10:28.276486+0000","last_undegraded":"2023-11-24T08:10:28.276486+0000","last_fullsized":"2023-11-24T08:10:28.276486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:15.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.213631+0000","last_change":"2023-11-24T08:10:27.211125+0000","last_active":"2023-11-24T08:10:27.213631+0000","last_peered":"2023-11-24T08:10:27.213631+0000","last_clean":"2023-11-24T08:10:27.213631+0000","last_became_active":"2023-11-24T08:10:27.210802+0000","last_became_peered":"2023-11-24T08:10:27.210802+0000","last_unstale":"2023-11-24T08:10:27.213631+0000","last_undegraded":"2023-11-24T08:10:27.213631+0000","last_fullsized":"2023-11-24T08:10:27.213631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:54.408281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.224960+0000","last_change":"2023-11-24T08:10:27.221294+0000","last_active":"2023-11-24T08:10:27.224960+0000","last_peered":"2023-11-24T08:10:27.224960+0000","last_clean":"2023-11-24T08:10:27.224960+0000","last_became_active":"2023-11-24T08:10:27.221019+0000","last_became_peered":"2023-11-24T08:10:27.221019+0000","last_unstale":"2023-11-24T08:10:27.224960+0000","last_undegraded":"2023-11-24T08:10:27.224960+0000","last_fullsized":"2023-11-24T08:10:27.224960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:09.766638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210053+0000","last_change":"2023-11-24T08:10:27.213708+0000","last_active":"2023-11-24T08:10:28.210053+0000","last_peered":"2023-11-24T08:10:28.210053+0000","last_clean":"2023-11-24T08:10:28.210053+0000","last_became_active":"2023-11-24T08:10:27.213390+0000","last_became_peered":"2023-11-24T08:10:27.213390+0000","last_unstale":"2023-11-24T08:10:28.210053+0000","last_undegraded":"2023-11-24T08:10:28.210053+0000","last_fullsized":"2023-11-24T08:10:28.210053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:10:10.993270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.209935+0000","last_change":"2023-11-24T08:10:27.221241+0000","last_active":"2023-11-24T08:10:28.209935+0000","last_peered":"2023-11-24T08:10:28.209935+0000","last_clean":"2023-11-24T08:10:28.209935+0000","last_became_active":"2023-11-24T08:10:27.219668+0000","last_became_peered":"2023-11-24T08:10:27.219668+0000","last_unstale":"2023-11-24T08:10:28.209935+0000","last_undegraded":"2023-11-24T08:10:28.209935+0000","last_fullsized":"2023-11-24T08:10:28.209935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:47.553001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222635+0000","last_change":"2023-11-24T08:10:27.219465+0000","last_active":"2023-11-24T08:10:27.222635+0000","last_peered":"2023-11-24T08:10:27.222635+0000","last_clean":"2023-11-24T08:10:27.222635+0000","last_became_active":"2023-11-24T08:10:27.219164+0000","last_became_peered":"2023-11-24T08:10:27.219164+0000","last_unstale":"2023-11-24T08:10:27.222635+0000","last_undegraded":"2023-11-24T08:10:27.222635+0000","last_fullsized":"2023-11-24T08:10:27.222635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:51.136885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222213+0000","last_change":"2023-11-24T08:10:27.219238+0000","last_active":"2023-11-24T08:10:27.222213+0000","last_peered":"2023-11-24T08:10:27.222213+0000","last_clean":"2023-11-24T08:10:27.222213+0000","last_became_active":"2023-11-24T08:10:27.217909+0000","last_became_peered":"2023-11-24T08:10:27.217909+0000","last_unstale":"2023-11-24T08:10:27.222213+0000","last_undegraded":"2023-11-24T08:10:27.222213+0000","last_fullsized":"2023-11-24T08:10:27.222213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:54:46.487986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226357+0000","last_change":"2023-11-24T08:10:27.216997+0000","last_active":"2023-11-24T08:10:27.226357+0000","last_peered":"2023-11-24T08:10:27.226357+0000","last_clean":"2023-11-24T08:10:27.226357+0000","last_became_active":"2023-11-24T08:10:27.214936+0000","last_became_peered":"2023-11-24T08:10:27.214936+0000","last_unstale":"2023-11-24T08:10:27.226357+0000","last_undegraded":"2023-11-24T08:10:27.226357+0000","last_fullsized":"2023-11-24T08:10:27.226357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:34:02.186464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221619+0000","last_change":"2023-11-24T08:10:27.217610+0000","last_active":"2023-11-24T08:10:27.221619+0000","last_peered":"2023-11-24T08:10:27.221619+0000","last_clean":"2023-11-24T08:10:27.221619+0000","last_became_active":"2023-11-24T08:10:27.217092+0000","last_became_peered":"2023-11-24T08:10:27.217092+0000","last_unstale":"2023-11-24T08:10:27.221619+0000","last_undegraded":"2023-11-24T08:10:27.221619+0000","last_fullsized":"2023-11-24T08:10:27.221619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:56.872996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230605+0000","last_change":"2023-11-24T08:10:27.225938+0000","last_active":"2023-11-24T08:10:27.230605+0000","last_peered":"2023-11-24T08:10:27.230605+0000","last_clean":"2023-11-24T08:10:27.230605+0000","last_became_active":"2023-11-24T08:10:27.225615+0000","last_became_peered":"2023-11-24T08:10:27.225615+0000","last_unstale":"2023-11-24T08:10:27.230605+0000","last_undegraded":"2023-11-24T08:10:27.230605+0000","last_fullsized":"2023-11-24T08:10:27.230605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:57:57.406223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221388+0000","last_change":"2023-11-24T08:10:27.215414+0000","last_active":"2023-11-24T08:10:27.221388+0000","last_peered":"2023-11-24T08:10:27.221388+0000","last_clean":"2023-11-24T08:10:27.221388+0000","last_became_active":"2023-11-24T08:10:27.215143+0000","last_became_peered":"2023-11-24T08:10:27.215143+0000","last_unstale":"2023-11-24T08:10:27.221388+0000","last_undegraded":"2023-11-24T08:10:27.221388+0000","last_fullsized":"2023-11-24T08:10:27.221388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:22.971334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.211323+0000","last_change":"2023-11-24T08:10:27.207564+0000","last_active":"2023-11-24T08:10:27.211323+0000","last_peered":"2023-11-24T08:10:27.211323+0000","last_clean":"2023-11-24T08:10:27.211323+0000","last_became_active":"2023-11-24T08:10:27.207005+0000","last_became_peered":"2023-11-24T08:10:27.207005+0000","last_unstale":"2023-11-24T08:10:27.211323+0000","last_undegraded":"2023-11-24T08:10:27.211323+0000","last_fullsized":"2023-11-24T08:10:27.211323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:51.725014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213669+0000","last_change":"2023-11-24T08:10:27.218854+0000","last_active":"2023-11-24T08:10:28.213669+0000","last_peered":"2023-11-24T08:10:28.213669+0000","last_clean":"2023-11-24T08:10:28.213669+0000","last_became_active":"2023-11-24T08:10:27.218190+0000","last_became_peered":"2023-11-24T08:10:27.218190+0000","last_unstale":"2023-11-24T08:10:28.213669+0000","last_undegraded":"2023-11-24T08:10:28.213669+0000","last_fullsized":"2023-11-24T08:10:28.213669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:12.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211825+0000","last_change":"2023-11-24T08:10:27.221597+0000","last_active":"2023-11-24T08:10:28.211825+0000","last_peered":"2023-11-24T08:10:28.211825+0000","last_clean":"2023-11-24T08:10:28.211825+0000","last_became_active":"2023-11-24T08:10:27.221311+0000","last_became_peered":"2023-11-24T08:10:27.221311+0000","last_unstale":"2023-11-24T08:10:28.211825+0000","last_undegraded":"2023-11-24T08:10:28.211825+0000","last_fullsized":"2023-11-24T08:10:28.211825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:45.191425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"77'4","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.212139+0000","last_change":"2023-11-24T08:10:26.178357+0000","last_active":"2023-11-24T08:10:28.212139+0000","last_peered":"2023-11-24T08:10:28.212139+0000","last_clean":"2023-11-24T08:10:28.212139+0000","last_became_active":"2023-11-24T08:10:26.177974+0000","last_became_peered":"2023-11-24T08:10:26.177974+0000","last_unstale":"2023-11-24T08:10:28.212139+0000","last_undegraded":"2023-11-24T08:10:28.212139+0000","last_fullsized":"2023-11-24T08:10:28.212139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:08:44.981165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.224170+0000","last_change":"2023-11-24T08:10:27.215429+0000","last_active":"2023-11-24T08:10:27.224170+0000","last_peered":"2023-11-24T08:10:27.224170+0000","last_clean":"2023-11-24T08:10:27.224170+0000","last_became_active":"2023-11-24T08:10:27.215115+0000","last_became_peered":"2023-11-24T08:10:27.215115+0000","last_unstale":"2023-11-24T08:10:27.224170+0000","last_undegraded":"2023-11-24T08:10:27.224170+0000","last_fullsized":"2023-11-24T08:10:27.224170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:13.235501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276810+0000","last_change":"2023-11-24T08:10:26.156400+0000","last_active":"2023-11-24T08:10:28.276810+0000","last_peered":"2023-11-24T08:10:28.276810+0000","last_clean":"2023-11-24T08:10:28.276810+0000","last_became_active":"2023-11-24T08:10:26.154580+0000","last_became_peered":"2023-11-24T08:10:26.154580+0000","last_unstale":"2023-11-24T08:10:28.276810+0000","last_undegraded":"2023-11-24T08:10:28.276810+0000","last_fullsized":"2023-11-24T08:10:28.276810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:56.455782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214719+0000","last_change":"2023-11-24T08:10:27.207575+0000","last_active":"2023-11-24T08:10:28.214719+0000","last_peered":"2023-11-24T08:10:28.214719+0000","last_clean":"2023-11-24T08:10:28.214719+0000","last_became_active":"2023-11-24T08:10:27.207215+0000","last_became_peered":"2023-11-24T08:10:27.207215+0000","last_unstale":"2023-11-24T08:10:28.214719+0000","last_undegraded":"2023-11-24T08:10:28.214719+0000","last_fullsized":"2023-11-24T08:10:28.214719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:51.136885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.232293+0000","last_change":"2023-11-24T08:10:26.168479+0000","last_active":"2023-11-24T08:10:27.232293+0000","last_peered":"2023-11-24T08:10:27.232293+0000","last_clean":"2023-11-24T08:10:27.232293+0000","last_became_active":"2023-11-24T08:10:26.168155+0000","last_became_peered":"2023-11-24T08:10:26.168155+0000","last_unstale":"2023-11-24T08:10:27.232293+0000","last_undegraded":"2023-11-24T08:10:27.232293+0000","last_fullsized":"2023-11-24T08:10:27.232293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:55.326173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211413+0000","last_change":"2023-11-24T08:10:27.224724+0000","last_active":"2023-11-24T08:10:28.211413+0000","last_peered":"2023-11-24T08:10:28.211413+0000","last_clean":"2023-11-24T08:10:28.211413+0000","last_became_active":"2023-11-24T08:10:27.224458+0000","last_became_peered":"2023-11-24T08:10:27.224458+0000","last_unstale":"2023-11-24T08:10:28.211413+0000","last_undegraded":"2023-11-24T08:10:28.211413+0000","last_fullsized":"2023-11-24T08:10:28.211413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:35:42.452650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.215026+0000","last_change":"2023-11-24T08:10:26.171776+0000","last_active":"2023-11-24T08:10:28.215026+0000","last_peered":"2023-11-24T08:10:28.215026+0000","last_clean":"2023-11-24T08:10:28.215026+0000","last_became_active":"2023-11-24T08:10:26.171421+0000","last_became_peered":"2023-11-24T08:10:26.171421+0000","last_unstale":"2023-11-24T08:10:28.215026+0000","last_undegraded":"2023-11-24T08:10:28.215026+0000","last_fullsized":"2023-11-24T08:10:28.215026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:06.871902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221413+0000","last_change":"2023-11-24T08:10:27.217022+0000","last_active":"2023-11-24T08:10:27.221413+0000","last_peered":"2023-11-24T08:10:27.221413+0000","last_clean":"2023-11-24T08:10:27.221413+0000","last_became_active":"2023-11-24T08:10:27.216754+0000","last_became_peered":"2023-11-24T08:10:27.216754+0000","last_unstale":"2023-11-24T08:10:27.221413+0000","last_undegraded":"2023-11-24T08:10:27.221413+0000","last_fullsized":"2023-11-24T08:10:27.221413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:09.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"77'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236135+0000","last_change":"2023-11-24T08:10:26.175207+0000","last_active":"2023-11-24T08:10:27.236135+0000","last_peered":"2023-11-24T08:10:27.236135+0000","last_clean":"2023-11-24T08:10:27.236135+0000","last_became_active":"2023-11-24T08:10:26.174892+0000","last_became_peered":"2023-11-24T08:10:26.174892+0000","last_unstale":"2023-11-24T08:10:27.236135+0000","last_undegraded":"2023-11-24T08:10:27.236135+0000","last_fullsized":"2023-11-24T08:10:27.236135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:12:47.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.216635+0000","last_change":"2023-11-24T08:10:27.214536+0000","last_active":"2023-11-24T08:10:27.216635+0000","last_peered":"2023-11-24T08:10:27.216635+0000","last_clean":"2023-11-24T08:10:27.216635+0000","last_became_active":"2023-11-24T08:10:27.214154+0000","last_became_peered":"2023-11-24T08:10:27.214154+0000","last_unstale":"2023-11-24T08:10:27.216635+0000","last_undegraded":"2023-11-24T08:10:27.216635+0000","last_fullsized":"2023-11-24T08:10:27.216635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:43.939464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214735+0000","last_change":"2023-11-24T08:10:26.175012+0000","last_active":"2023-11-24T08:10:28.214735+0000","last_peered":"2023-11-24T08:10:28.214735+0000","last_clean":"2023-11-24T08:10:28.214735+0000","last_became_active":"2023-11-24T08:10:26.172944+0000","last_became_peered":"2023-11-24T08:10:26.172944+0000","last_unstale":"2023-11-24T08:10:28.214735+0000","last_undegraded":"2023-11-24T08:10:28.214735+0000","last_fullsized":"2023-11-24T08:10:28.214735+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:50.808137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225582+0000","last_change":"2023-11-24T08:10:27.216769+0000","last_active":"2023-11-24T08:10:27.225582+0000","last_peered":"2023-11-24T08:10:27.225582+0000","last_clean":"2023-11-24T08:10:27.225582+0000","last_became_active":"2023-11-24T08:10:27.216466+0000","last_became_peered":"2023-11-24T08:10:27.216466+0000","last_unstale":"2023-11-24T08:10:27.225582+0000","last_undegraded":"2023-11-24T08:10:27.225582+0000","last_fullsized":"2023-11-24T08:10:27.225582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:34:45.284416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.182558+0000","last_change":"2023-11-24T08:10:26.174299+0000","last_active":"2023-11-24T08:10:27.182558+0000","last_peered":"2023-11-24T08:10:27.182558+0000","last_clean":"2023-11-24T08:10:27.182558+0000","last_became_active":"2023-11-24T08:10:26.173991+0000","last_became_peered":"2023-11-24T08:10:26.173991+0000","last_unstale":"2023-11-24T08:10:27.182558+0000","last_undegraded":"2023-11-24T08:10:27.182558+0000","last_fullsized":"2023-11-24T08:10:27.182558+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:00:42.119077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276665+0000","last_change":"2023-11-24T08:10:27.212558+0000","last_active":"2023-11-24T08:10:28.276665+0000","last_peered":"2023-11-24T08:10:28.276665+0000","last_clean":"2023-11-24T08:10:28.276665+0000","last_became_active":"2023-11-24T08:10:27.211948+0000","last_became_peered":"2023-11-24T08:10:27.211948+0000","last_unstale":"2023-11-24T08:10:28.276665+0000","last_undegraded":"2023-11-24T08:10:28.276665+0000","last_fullsized":"2023-11-24T08:10:28.276665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:01.349793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.202099+0000","last_change":"2023-11-24T08:10:26.172236+0000","last_active":"2023-11-24T08:10:27.202099+0000","last_peered":"2023-11-24T08:10:27.202099+0000","last_clean":"2023-11-24T08:10:27.202099+0000","last_became_active":"2023-11-24T08:10:26.171907+0000","last_became_peered":"2023-11-24T08:10:26.171907+0000","last_unstale":"2023-11-24T08:10:27.202099+0000","last_undegraded":"2023-11-24T08:10:27.202099+0000","last_fullsized":"2023-11-24T08:10:27.202099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:19.388669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.220887+0000","last_change":"2023-11-24T08:10:27.217603+0000","last_active":"2023-11-24T08:10:27.220887+0000","last_peered":"2023-11-24T08:10:27.220887+0000","last_clean":"2023-11-24T08:10:27.220887+0000","last_became_active":"2023-11-24T08:10:27.215278+0000","last_became_peered":"2023-11-24T08:10:27.215278+0000","last_unstale":"2023-11-24T08:10:27.220887+0000","last_undegraded":"2023-11-24T08:10:27.220887+0000","last_fullsized":"2023-11-24T08:10:27.220887+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:38:48.633056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.209722+0000","last_change":"2023-11-24T08:10:26.169121+0000","last_active":"2023-11-24T08:10:27.209722+0000","last_peered":"2023-11-24T08:10:27.209722+0000","last_clean":"2023-11-24T08:10:27.209722+0000","last_became_active":"2023-11-24T08:10:26.168857+0000","last_became_peered":"2023-11-24T08:10:26.168857+0000","last_unstale":"2023-11-24T08:10:27.209722+0000","last_undegraded":"2023-11-24T08:10:27.209722+0000","last_fullsized":"2023-11-24T08:10:27.209722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:45.405096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211916+0000","last_change":"2023-11-24T08:10:27.223114+0000","last_active":"2023-11-24T08:10:28.211916+0000","last_peered":"2023-11-24T08:10:28.211916+0000","last_clean":"2023-11-24T08:10:28.211916+0000","last_became_active":"2023-11-24T08:10:27.222792+0000","last_became_peered":"2023-11-24T08:10:27.222792+0000","last_unstale":"2023-11-24T08:10:28.211916+0000","last_undegraded":"2023-11-24T08:10:28.211916+0000","last_fullsized":"2023-11-24T08:10:28.211916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:27.857377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214918+0000","last_change":"2023-11-24T08:10:26.169378+0000","last_active":"2023-11-24T08:10:28.214918+0000","last_peered":"2023-11-24T08:10:28.214918+0000","last_clean":"2023-11-24T08:10:28.214918+0000","last_became_active":"2023-11-24T08:10:26.168569+0000","last_became_peered":"2023-11-24T08:10:26.168569+0000","last_unstale":"2023-11-24T08:10:28.214918+0000","last_undegraded":"2023-11-24T08:10:28.214918+0000","last_fullsized":"2023-11-24T08:10:28.214918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:53.403938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214839+0000","last_change":"2023-11-24T08:10:27.214469+0000","last_active":"2023-11-24T08:10:28.214839+0000","last_peered":"2023-11-24T08:10:28.214839+0000","last_clean":"2023-11-24T08:10:28.214839+0000","last_became_active":"2023-11-24T08:10:27.214176+0000","last_became_peered":"2023-11-24T08:10:27.214176+0000","last_unstale":"2023-11-24T08:10:28.214839+0000","last_undegraded":"2023-11-24T08:10:28.214839+0000","last_fullsized":"2023-11-24T08:10:28.214839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:46.409439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"77'3","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236344+0000","last_change":"2023-11-24T08:10:26.166945+0000","last_active":"2023-11-24T08:10:27.236344+0000","last_peered":"2023-11-24T08:10:27.236344+0000","last_clean":"2023-11-24T08:10:27.236344+0000","last_became_active":"2023-11-24T08:10:26.166666+0000","last_became_peered":"2023-11-24T08:10:26.166666+0000","last_unstale":"2023-11-24T08:10:27.236344+0000","last_undegraded":"2023-11-24T08:10:27.236344+0000","last_fullsized":"2023-11-24T08:10:27.236344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:01:29.556715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.217192+0000","last_change":"2023-11-24T08:10:27.209127+0000","last_active":"2023-11-24T08:10:27.217192+0000","last_peered":"2023-11-24T08:10:27.217192+0000","last_clean":"2023-11-24T08:10:27.217192+0000","last_became_active":"2023-11-24T08:10:27.208802+0000","last_became_peered":"2023-11-24T08:10:27.208802+0000","last_unstale":"2023-11-24T08:10:27.217192+0000","last_undegraded":"2023-11-24T08:10:27.217192+0000","last_fullsized":"2023-11-24T08:10:27.217192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:28.019517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"77'6","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.237978+0000","last_change":"2023-11-24T08:10:26.169829+0000","last_active":"2023-11-24T08:10:27.237978+0000","last_peered":"2023-11-24T08:10:27.237978+0000","last_clean":"2023-11-24T08:10:27.237978+0000","last_became_active":"2023-11-24T08:10:26.169497+0000","last_became_peered":"2023-11-24T08:10:26.169497+0000","last_unstale":"2023-11-24T08:10:27.237978+0000","last_undegraded":"2023-11-24T08:10:27.237978+0000","last_fullsized":"2023-11-24T08:10:27.237978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:50.808137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210507+0000","last_change":"2023-11-24T08:10:27.213421+0000","last_active":"2023-11-24T08:10:28.210507+0000","last_peered":"2023-11-24T08:10:28.210507+0000","last_clean":"2023-11-24T08:10:28.210507+0000","last_became_active":"2023-11-24T08:10:27.212333+0000","last_became_peered":"2023-11-24T08:10:27.212333+0000","last_unstale":"2023-11-24T08:10:28.210507+0000","last_undegraded":"2023-11-24T08:10:28.210507+0000","last_fullsized":"2023-11-24T08:10:28.210507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:59:31.551186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230239+0000","last_change":"2023-11-24T08:10:26.175964+0000","last_active":"2023-11-24T08:10:27.230239+0000","last_peered":"2023-11-24T08:10:27.230239+0000","last_clean":"2023-11-24T08:10:27.230239+0000","last_became_active":"2023-11-24T08:10:26.175679+0000","last_became_peered":"2023-11-24T08:10:26.175679+0000","last_unstale":"2023-11-24T08:10:27.230239+0000","last_undegraded":"2023-11-24T08:10:27.230239+0000","last_fullsized":"2023-11-24T08:10:27.230239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:04.440151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.220698+0000","last_change":"2023-11-24T08:10:27.215063+0000","last_active":"2023-11-24T08:10:27.220698+0000","last_peered":"2023-11-24T08:10:27.220698+0000","last_clean":"2023-11-24T08:10:27.220698+0000","last_became_active":"2023-11-24T08:10:27.214700+0000","last_became_peered":"2023-11-24T08:10:27.214700+0000","last_unstale":"2023-11-24T08:10:27.220698+0000","last_undegraded":"2023-11-24T08:10:27.220698+0000","last_fullsized":"2023-11-24T08:10:27.220698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:16:22.949678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214931+0000","last_change":"2023-11-24T08:10:26.169175+0000","last_active":"2023-11-24T08:10:28.214931+0000","last_peered":"2023-11-24T08:10:28.214931+0000","last_clean":"2023-11-24T08:10:28.214931+0000","last_became_active":"2023-11-24T08:10:26.168866+0000","last_became_peered":"2023-11-24T08:10:26.168866+0000","last_unstale":"2023-11-24T08:10:28.214931+0000","last_undegraded":"2023-11-24T08:10:28.214931+0000","last_fullsized":"2023-11-24T08:10:28.214931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:17.756973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276628+0000","last_change":"2023-11-24T08:10:27.204593+0000","last_active":"2023-11-24T08:10:28.276628+0000","last_peered":"2023-11-24T08:10:28.276628+0000","last_clean":"2023-11-24T08:10:28.276628+0000","last_became_active":"2023-11-24T08:10:27.204006+0000","last_became_peered":"2023-11-24T08:10:27.204006+0000","last_unstale":"2023-11-24T08:10:28.276628+0000","last_undegraded":"2023-11-24T08:10:28.276628+0000","last_fullsized":"2023-11-24T08:10:28.276628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:08:45.985509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.196513+0000","last_change":"2023-11-24T08:10:26.175420+0000","last_active":"2023-11-24T08:10:27.196513+0000","last_peered":"2023-11-24T08:10:27.196513+0000","last_clean":"2023-11-24T08:10:27.196513+0000","last_became_active":"2023-11-24T08:10:26.173600+0000","last_became_peered":"2023-11-24T08:10:26.173600+0000","last_unstale":"2023-11-24T08:10:27.196513+0000","last_undegraded":"2023-11-24T08:10:27.196513+0000","last_fullsized":"2023-11-24T08:10:27.196513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:12.487792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226972+0000","last_change":"2023-11-24T08:10:27.219467+0000","last_active":"2023-11-24T08:10:27.226972+0000","last_peered":"2023-11-24T08:10:27.226972+0000","last_clean":"2023-11-24T08:10:27.226972+0000","last_became_active":"2023-11-24T08:10:27.219226+0000","last_became_peered":"2023-11-24T08:10:27.219226+0000","last_unstale":"2023-11-24T08:10:27.226972+0000","last_undegraded":"2023-11-24T08:10:27.226972+0000","last_fullsized":"2023-11-24T08:10:27.226972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:33.868126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236253+0000","last_change":"2023-11-24T08:10:26.164429+0000","last_active":"2023-11-24T08:10:27.236253+0000","last_peered":"2023-11-24T08:10:27.236253+0000","last_clean":"2023-11-24T08:10:27.236253+0000","last_became_active":"2023-11-24T08:10:26.164086+0000","last_became_peered":"2023-11-24T08:10:26.164086+0000","last_unstale":"2023-11-24T08:10:27.236253+0000","last_undegraded":"2023-11-24T08:10:27.236253+0000","last_fullsized":"2023-11-24T08:10:27.236253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:36.026244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221746+0000","last_change":"2023-11-24T08:10:27.218180+0000","last_active":"2023-11-24T08:10:27.221746+0000","last_peered":"2023-11-24T08:10:27.221746+0000","last_clean":"2023-11-24T08:10:27.221746+0000","last_became_active":"2023-11-24T08:10:27.217915+0000","last_became_peered":"2023-11-24T08:10:27.217915+0000","last_unstale":"2023-11-24T08:10:27.221746+0000","last_undegraded":"2023-11-24T08:10:27.221746+0000","last_fullsized":"2023-11-24T08:10:27.221746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:12.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214245+0000","last_change":"2023-11-24T08:10:26.171128+0000","last_active":"2023-11-24T08:10:28.214245+0000","last_peered":"2023-11-24T08:10:28.214245+0000","last_clean":"2023-11-24T08:10:28.214245+0000","last_became_active":"2023-11-24T08:10:26.170772+0000","last_became_peered":"2023-11-24T08:10:26.170772+0000","last_unstale":"2023-11-24T08:10:28.214245+0000","last_undegraded":"2023-11-24T08:10:28.214245+0000","last_fullsized":"2023-11-24T08:10:28.214245+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:34.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.212231+0000","last_change":"2023-11-24T08:10:27.225820+0000","last_active":"2023-11-24T08:10:28.212231+0000","last_peered":"2023-11-24T08:10:28.212231+0000","last_clean":"2023-11-24T08:10:28.212231+0000","last_became_active":"2023-11-24T08:10:27.225617+0000","last_became_peered":"2023-11-24T08:10:27.225617+0000","last_unstale":"2023-11-24T08:10:28.212231+0000","last_undegraded":"2023-11-24T08:10:28.212231+0000","last_fullsized":"2023-11-24T08:10:28.212231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:05.087800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.231754+0000","last_change":"2023-11-24T08:10:26.173050+0000","last_active":"2023-11-24T08:10:27.231754+0000","last_peered":"2023-11-24T08:10:27.231754+0000","last_clean":"2023-11-24T08:10:27.231754+0000","last_became_active":"2023-11-24T08:10:26.172416+0000","last_became_peered":"2023-11-24T08:10:26.172416+0000","last_unstale":"2023-11-24T08:10:27.231754+0000","last_undegraded":"2023-11-24T08:10:27.231754+0000","last_fullsized":"2023-11-24T08:10:27.231754+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:04.589673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.212945+0000","last_change":"2023-11-24T08:10:27.208563+0000","last_active":"2023-11-24T08:10:27.212945+0000","last_peered":"2023-11-24T08:10:27.212945+0000","last_clean":"2023-11-24T08:10:27.212945+0000","last_became_active":"2023-11-24T08:10:27.207818+0000","last_became_peered":"2023-11-24T08:10:27.207818+0000","last_unstale":"2023-11-24T08:10:27.212945+0000","last_undegraded":"2023-11-24T08:10:27.212945+0000","last_fullsized":"2023-11-24T08:10:27.212945+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:17.515094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214276+0000","last_change":"2023-11-24T08:10:26.165415+0000","last_active":"2023-11-24T08:10:28.214276+0000","last_peered":"2023-11-24T08:10:28.214276+0000","last_clean":"2023-11-24T08:10:28.214276+0000","last_became_active":"2023-11-24T08:10:26.165085+0000","last_became_peered":"2023-11-24T08:10:26.165085+0000","last_unstale":"2023-11-24T08:10:28.214276+0000","last_undegraded":"2023-11-24T08:10:28.214276+0000","last_fullsized":"2023-11-24T08:10:28.214276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:08.762295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225850+0000","last_change":"2023-11-24T08:10:27.217035+0000","last_active":"2023-11-24T08:10:27.225850+0000","last_peered":"2023-11-24T08:10:27.225850+0000","last_clean":"2023-11-24T08:10:27.225850+0000","last_became_active":"2023-11-24T08:10:27.216571+0000","last_became_peered":"2023-11-24T08:10:27.216571+0000","last_unstale":"2023-11-24T08:10:27.225850+0000","last_undegraded":"2023-11-24T08:10:27.225850+0000","last_fullsized":"2023-11-24T08:10:27.225850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:00:24.832767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210715+0000","last_change":"2023-11-24T08:10:26.178461+0000","last_active":"2023-11-24T08:10:28.210715+0000","last_peered":"2023-11-24T08:10:28.210715+0000","last_clean":"2023-11-24T08:10:28.210715+0000","last_became_active":"2023-11-24T08:10:26.178125+0000","last_became_peered":"2023-11-24T08:10:26.178125+0000","last_unstale":"2023-11-24T08:10:28.210715+0000","last_undegraded":"2023-11-24T08:10:28.210715+0000","last_fullsized":"2023-11-24T08:10:28.210715+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:48.852696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225828+0000","last_change":"2023-11-24T08:10:27.218809+0000","last_active":"2023-11-24T08:10:27.225828+0000","last_peered":"2023-11-24T08:10:27.225828+0000","last_clean":"2023-11-24T08:10:27.225828+0000","last_became_active":"2023-11-24T08:10:27.216100+0000","last_became_peered":"2023-11-24T08:10:27.216100+0000","last_unstale":"2023-11-24T08:10:27.225828+0000","last_undegraded":"2023-11-24T08:10:27.225828+0000","last_fullsized":"2023-11-24T08:10:27.225828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:01.358124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.215099+0000","last_change":"2023-11-24T08:10:26.170548+0000","last_active":"2023-11-24T08:10:28.215099+0000","last_peered":"2023-11-24T08:10:28.215099+0000","last_clean":"2023-11-24T08:10:28.215099+0000","last_became_active":"2023-11-24T08:10:26.170229+0000","last_became_peered":"2023-11-24T08:10:26.170229+0000","last_unstale":"2023-11-24T08:10:28.215099+0000","last_undegraded":"2023-11-24T08:10:28.215099+0000","last_fullsized":"2023-11-24T08:10:28.215099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:46.113937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.227424+0000","last_change":"2023-11-24T08:10:27.220307+0000","last_active":"2023-11-24T08:10:27.227424+0000","last_peered":"2023-11-24T08:10:27.227424+0000","last_clean":"2023-11-24T08:10:27.227424+0000","last_became_active":"2023-11-24T08:10:27.219987+0000","last_became_peered":"2023-11-24T08:10:27.219987+0000","last_unstale":"2023-11-24T08:10:27.227424+0000","last_undegraded":"2023-11-24T08:10:27.227424+0000","last_fullsized":"2023-11-24T08:10:27.227424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:10.043211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.196729+0000","last_change":"2023-11-24T08:10:26.166255+0000","last_active":"2023-11-24T08:10:27.196729+0000","last_peered":"2023-11-24T08:10:27.196729+0000","last_clean":"2023-11-24T08:10:27.196729+0000","last_became_active":"2023-11-24T08:10:26.165814+0000","last_became_peered":"2023-11-24T08:10:26.165814+0000","last_unstale":"2023-11-24T08:10:27.196729+0000","last_undegraded":"2023-11-24T08:10:27.196729+0000","last_fullsized":"2023-11-24T08:10:27.196729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:50.119768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226771+0000","last_change":"2023-11-24T08:10:27.220440+0000","last_active":"2023-11-24T08:10:27.226771+0000","last_peered":"2023-11-24T08:10:27.226771+0000","last_clean":"2023-11-24T08:10:27.226771+0000","last_became_active":"2023-11-24T08:10:27.220105+0000","last_became_peered":"2023-11-24T08:10:27.220105+0000","last_unstale":"2023-11-24T08:10:27.226771+0000","last_undegraded":"2023-11-24T08:10:27.226771+0000","last_fullsized":"2023-11-24T08:10:27.226771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:37.191476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276582+0000","last_change":"2023-11-24T08:10:26.155493+0000","last_active":"2023-11-24T08:10:28.276582+0000","last_peered":"2023-11-24T08:10:28.276582+0000","last_clean":"2023-11-24T08:10:28.276582+0000","last_became_active":"2023-11-24T08:10:26.154836+0000","last_became_peered":"2023-11-24T08:10:26.154836+0000","last_unstale":"2023-11-24T08:10:28.276582+0000","last_undegraded":"2023-11-24T08:10:28.276582+0000","last_fullsized":"2023-11-24T08:10:28.276582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:53:05.119965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276817+0000","last_change":"2023-11-24T08:10:27.212067+0000","last_active":"2023-11-24T08:10:28.276817+0000","last_peered":"2023-11-24T08:10:28.276817+0000","last_clean":"2023-11-24T08:10:28.276817+0000","last_became_active":"2023-11-24T08:10:27.211784+0000","last_became_peered":"2023-11-24T08:10:27.211784+0000","last_unstale":"2023-11-24T08:10:28.276817+0000","last_undegraded":"2023-11-24T08:10:28.276817+0000","last_fullsized":"2023-11-24T08:10:28.276817+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:47.553001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214570+0000","last_change":"2023-11-24T08:10:26.171117+0000","last_active":"2023-11-24T08:10:28.214570+0000","last_peered":"2023-11-24T08:10:28.214570+0000","last_clean":"2023-11-24T08:10:28.214570+0000","last_became_active":"2023-11-24T08:10:26.170860+0000","last_became_peered":"2023-11-24T08:10:26.170860+0000","last_unstale":"2023-11-24T08:10:28.214570+0000","last_undegraded":"2023-11-24T08:10:28.214570+0000","last_fullsized":"2023-11-24T08:10:28.214570+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:42.935121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.219878+0000","last_change":"2023-11-24T08:10:27.212153+0000","last_active":"2023-11-24T08:10:27.219878+0000","last_peered":"2023-11-24T08:10:27.219878+0000","last_clean":"2023-11-24T08:10:27.219878+0000","last_became_active":"2023-11-24T08:10:27.210605+0000","last_became_peered":"2023-11-24T08:10:27.210605+0000","last_unstale":"2023-11-24T08:10:27.219878+0000","last_undegraded":"2023-11-24T08:10:27.219878+0000","last_fullsized":"2023-11-24T08:10:27.219878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:12:42.719087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.197983+0000","last_change":"2023-11-24T08:10:26.169246+0000","last_active":"2023-11-24T08:10:27.197983+0000","last_peered":"2023-11-24T08:10:27.197983+0000","last_clean":"2023-11-24T08:10:27.197983+0000","last_became_active":"2023-11-24T08:10:26.167800+0000","last_became_peered":"2023-11-24T08:10:26.167800+0000","last_unstale":"2023-11-24T08:10:27.197983+0000","last_undegraded":"2023-11-24T08:10:27.197983+0000","last_fullsized":"2023-11-24T08:10:27.197983+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:48.852696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210662+0000","last_change":"2023-11-24T08:10:27.219616+0000","last_active":"2023-11-24T08:10:28.210662+0000","last_peered":"2023-11-24T08:10:28.210662+0000","last_clean":"2023-11-24T08:10:28.210662+0000","last_became_active":"2023-11-24T08:10:27.219343+0000","last_became_peered":"2023-11-24T08:10:27.219343+0000","last_unstale":"2023-11-24T08:10:28.210662+0000","last_undegraded":"2023-11-24T08:10:28.210662+0000","last_fullsized":"2023-11-24T08:10:28.210662+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:32:30.302468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.203370+0000","last_change":"2023-11-24T08:10:26.158707+0000","last_active":"2023-11-24T08:10:27.203370+0000","last_peered":"2023-11-24T08:10:27.203370+0000","last_clean":"2023-11-24T08:10:27.203370+0000","last_became_active":"2023-11-24T08:10:26.154743+0000","last_became_peered":"2023-11-24T08:10:26.154743+0000","last_unstale":"2023-11-24T08:10:27.203370+0000","last_undegraded":"2023-11-24T08:10:27.203370+0000","last_fullsized":"2023-11-24T08:10:27.203370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:09.837478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211547+0000","last_change":"2023-11-24T08:10:27.222902+0000","last_active":"2023-11-24T08:10:28.211547+0000","last_peered":"2023-11-24T08:10:28.211547+0000","last_clean":"2023-11-24T08:10:28.211547+0000","last_became_active":"2023-11-24T08:10:27.222579+0000","last_became_peered":"2023-11-24T08:10:27.222579+0000","last_unstale":"2023-11-24T08:10:28.211547+0000","last_undegraded":"2023-11-24T08:10:28.211547+0000","last_fullsized":"2023-11-24T08:10:28.211547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:50.582784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276607+0000","last_change":"2023-11-24T08:10:26.164937+0000","last_active":"2023-11-24T08:10:28.276607+0000","last_peered":"2023-11-24T08:10:28.276607+0000","last_clean":"2023-11-24T08:10:28.276607+0000","last_became_active":"2023-11-24T08:10:26.164635+0000","last_became_peered":"2023-11-24T08:10:26.164635+0000","last_unstale":"2023-11-24T08:10:28.276607+0000","last_undegraded":"2023-11-24T08:10:28.276607+0000","last_fullsized":"2023-11-24T08:10:28.276607+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:50:30.599798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226778+0000","last_change":"2023-11-24T08:10:27.218611+0000","last_active":"2023-11-24T08:10:27.226778+0000","last_peered":"2023-11-24T08:10:27.226778+0000","last_clean":"2023-11-24T08:10:27.226778+0000","last_became_active":"2023-11-24T08:10:27.218299+0000","last_became_peered":"2023-11-24T08:10:27.218299+0000","last_unstale":"2023-11-24T08:10:27.226778+0000","last_undegraded":"2023-11-24T08:10:27.226778+0000","last_fullsized":"2023-11-24T08:10:27.226778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:06.652241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.203059+0000","last_change":"2023-11-24T08:10:26.166584+0000","last_active":"2023-11-24T08:10:27.203059+0000","last_peered":"2023-11-24T08:10:27.203059+0000","last_clean":"2023-11-24T08:10:27.203059+0000","last_became_active":"2023-11-24T08:10:26.166203+0000","last_became_peered":"2023-11-24T08:10:26.166203+0000","last_unstale":"2023-11-24T08:10:27.203059+0000","last_undegraded":"2023-11-24T08:10:27.203059+0000","last_fullsized":"2023-11-24T08:10:27.203059+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:09.246865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213767+0000","last_change":"2023-11-24T08:10:27.216218+0000","last_active":"2023-11-24T08:10:28.213767+0000","last_peered":"2023-11-24T08:10:28.213767+0000","last_clean":"2023-11-24T08:10:28.213767+0000","last_became_active":"2023-11-24T08:10:27.215910+0000","last_became_peered":"2023-11-24T08:10:27.215910+0000","last_unstale":"2023-11-24T08:10:28.213767+0000","last_undegraded":"2023-11-24T08:10:28.213767+0000","last_fullsized":"2023-11-24T08:10:28.213767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:24.165893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.182619+0000","last_change":"2023-11-24T08:10:26.176750+0000","last_active":"2023-11-24T08:10:27.182619+0000","last_peered":"2023-11-24T08:10:27.182619+0000","last_clean":"2023-11-24T08:10:27.182619+0000","last_became_active":"2023-11-24T08:10:26.176467+0000","last_became_peered":"2023-11-24T08:10:26.176467+0000","last_unstale":"2023-11-24T08:10:27.182619+0000","last_undegraded":"2023-11-24T08:10:27.182619+0000","last_fullsized":"2023-11-24T08:10:27.182619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:37.967475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226360+0000","last_change":"2023-11-24T08:10:27.220915+0000","last_active":"2023-11-24T08:10:27.226360+0000","last_peered":"2023-11-24T08:10:27.226360+0000","last_clean":"2023-11-24T08:10:27.226360+0000","last_became_active":"2023-11-24T08:10:27.220636+0000","last_became_peered":"2023-11-24T08:10:27.220636+0000","last_unstale":"2023-11-24T08:10:27.226360+0000","last_undegraded":"2023-11-24T08:10:27.226360+0000","last_fullsized":"2023-11-24T08:10:27.226360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:20.232558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.208696+0000","last_change":"2023-11-24T08:10:26.170459+0000","last_active":"2023-11-24T08:10:27.208696+0000","last_peered":"2023-11-24T08:10:27.208696+0000","last_clean":"2023-11-24T08:10:27.208696+0000","last_became_active":"2023-11-24T08:10:26.170243+0000","last_became_peered":"2023-11-24T08:10:26.170243+0000","last_unstale":"2023-11-24T08:10:27.208696+0000","last_undegraded":"2023-11-24T08:10:27.208696+0000","last_fullsized":"2023-11-24T08:10:27.208696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:22:07.874248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213489+0000","last_change":"2023-11-24T08:10:27.219419+0000","last_active":"2023-11-24T08:10:28.213489+0000","last_peered":"2023-11-24T08:10:28.213489+0000","last_clean":"2023-11-24T08:10:28.213489+0000","last_became_active":"2023-11-24T08:10:27.218936+0000","last_became_peered":"2023-11-24T08:10:27.218936+0000","last_unstale":"2023-11-24T08:10:28.213489+0000","last_undegraded":"2023-11-24T08:10:28.213489+0000","last_fullsized":"2023-11-24T08:10:28.213489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:56.872996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213933+0000","last_change":"2023-11-24T08:10:26.170453+0000","last_active":"2023-11-24T08:10:28.213933+0000","last_peered":"2023-11-24T08:10:28.213933+0000","last_clean":"2023-11-24T08:10:28.213933+0000","last_became_active":"2023-11-24T08:10:26.169829+0000","last_became_peered":"2023-11-24T08:10:26.169829+0000","last_unstale":"2023-11-24T08:10:28.213933+0000","last_undegraded":"2023-11-24T08:10:28.213933+0000","last_fullsized":"2023-11-24T08:10:28.213933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:26.348763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.228839+0000","last_change":"2023-11-24T08:10:27.221363+0000","last_active":"2023-11-24T08:10:27.228839+0000","last_peered":"2023-11-24T08:10:27.228839+0000","last_clean":"2023-11-24T08:10:27.228839+0000","last_became_active":"2023-11-24T08:10:27.220947+0000","last_became_peered":"2023-11-24T08:10:27.220947+0000","last_unstale":"2023-11-24T08:10:27.228839+0000","last_undegraded":"2023-11-24T08:10:27.228839+0000","last_fullsized":"2023-11-24T08:10:27.228839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:21:25.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213984+0000","last_change":"2023-11-24T08:10:26.171855+0000","last_active":"2023-11-24T08:10:28.213984+0000","last_peered":"2023-11-24T08:10:28.213984+0000","last_clean":"2023-11-24T08:10:28.213984+0000","last_became_active":"2023-11-24T08:10:26.171560+0000","last_became_peered":"2023-11-24T08:10:26.171560+0000","last_unstale":"2023-11-24T08:10:28.213984+0000","last_undegraded":"2023-11-24T08:10:28.213984+0000","last_fullsized":"2023-11-24T08:10:28.213984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:24:16.460597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":214,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.205725+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:10:27.205725+0000","last_peered":"2023-11-24T08:10:27.205725+0000","last_clean":"2023-11-24T08:10:27.205725+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:10:27.205725+0000","last_undegraded":"2023-11-24T08:10:27.205725+0000","last_fullsized":"2023-11-24T08:10:27.205725+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":991431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.046,"15min":0.997},"min":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4179999999999999,"5min":1.621,"15min":1.621},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.137,"15min":1.0860000000000001},"min":{"1min":0.78600000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.548,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.2669999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1399999999999999,"15min":1.1299999999999999},"min":{"1min":0.72199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6200000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.206,"15min":1.1819999999999999},"min":{"1min":0.85399999999999998,"5min":0.628,"15min":0.628},"max":{"1min":1.5409999999999999,"5min":2.198,"15min":2.198},"last":1.151}]},{"osd":2,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.1459999999999999,"15min":1.216},"min":{"1min":0.71199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.502,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.196},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.298,"15min":1.349},"min":{"1min":0.85199999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.603,"5min":4.633,"15min":4.633},"last":1.3009999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.198,"15min":1.1439999999999999},"min":{"1min":0.83399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.5820000000000001,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.88100000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.855,"5min":2.206,"15min":2.206},"last":1.417}]},{"osd":4,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.371,"15min":1.333},"min":{"1min":0.96699999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7250000000000001,"5min":2.851,"15min":2.851},"last":1.391},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.3420000000000001,"15min":1.3280000000000001},"min":{"1min":0.90200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5700000000000001,"5min":3.0510000000000002,"15min":3.0510000000000002},"last":1.3360000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.458,"15min":1.446},"min":{"1min":1.03,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.712,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.105},{"interface":"front","average":{"1min":1.306,"5min":1.4930000000000001,"15min":1.4750000000000001},"min":{"1min":0.97299999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.677,"5min":2.629,"15min":2.629},"last":1.476}]},{"osd":6,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5800000000000001,"15min":1.538},"min":{"1min":1.0960000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.7589999999999999,"5min":2.746,"15min":2.746},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.49,"15min":1.4399999999999999},"min":{"1min":1.0640000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.677,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4450000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":946176,"data_stored":400539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.081,"15min":1.0900000000000001},"min":{"1min":0.71899999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.073},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.077,"15min":1.107},"min":{"1min":0.69899999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8899999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.0389999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1739999999999999,"15min":1.266},"min":{"1min":0.72699999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.319},{"interface":"front","average":{"1min":1.083,"5min":1.1890000000000001,"15min":1.288},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5980000000000001,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1080000000000001,"15min":1.129},"min":{"1min":0.78500000000000003,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4630000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.21,"5min":1.278,"15min":1.387},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.8999999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.341,"15min":1.3999999999999999},"min":{"1min":0.91300000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7869999999999999,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2989999999999999,"15min":1.3440000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.631,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.141}]},{"osd":4,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4279999999999999,"15min":1.556},"min":{"1min":0.92400000000000004,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1110000000000002,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.5169999999999999,"15min":1.573},"min":{"1min":1.143,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.242,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.278}]},{"osd":5,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.472,"15min":1.573},"min":{"1min":1.0680000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.7849999999999999,"5min":2.629,"15min":2.629},"last":1.24},{"interface":"front","average":{"1min":1.55,"5min":1.595,"15min":1.7050000000000001},"min":{"1min":1.101,"5min":1.022,"15min":1.022},"max":{"1min":2.4249999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.569}]},{"osd":7,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.6140000000000001,"15min":1.736},"min":{"1min":1.2330000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":1.8400000000000001,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.444},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6180000000000001,"15min":1.702},"min":{"1min":1.2629999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.1589999999999998,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.3640000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574920,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":400537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.044,"15min":1.133},"min":{"1min":0.73599999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.86499999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.71,"5min":2.198,"15min":2.198},"last":1.2689999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1220000000000001,"15min":1.179},"min":{"1min":0.88,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.7070000000000001,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.153,"15min":1.147},"min":{"1min":0.86199999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.482}]},{"osd":2,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.2,"15min":1.216},"min":{"1min":0.65100000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1479999999999999,"15min":1.1739999999999999},"min":{"1min":0.74299999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.397,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1579999999999999,"15min":1.1990000000000001},"min":{"1min":0.81799999999999995,"5min":0.432,"15min":0.432},"max":{"1min":1.6850000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.375,"5min":1.28,"15min":1.3480000000000001},"min":{"1min":0.871,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9730000000000001,"5min":1.99,"15min":1.99},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.325,"15min":1.347},"min":{"1min":0.84499999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.774,"5min":1.839,"15min":1.839},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.482,"15min":1.5640000000000001},"min":{"1min":0.96099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.1019999999999999,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.393}]},{"osd":6,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3620000000000001,"15min":1.47},"min":{"1min":0.90500000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.73,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4079999999999999,"15min":1.4399999999999999},"min":{"1min":0.98299999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7849999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.494,"15min":1.6100000000000001},"min":{"1min":1.024,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8149999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.45},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.581,"15min":1.619},"min":{"1min":1.284,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.5960000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":400517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.079,"15min":1.155},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.0640000000000001,"15min":1.089},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6000000000000001,"5min":2.222,"15min":2.222},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.72899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.516,"5min":1.74,"15min":1.74},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.1819999999999999,"15min":1.1899999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.27}]},{"osd":2,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.071,"15min":1.169},"min":{"1min":0.65100000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5449999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.347},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.256,"15min":1.3779999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.633,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.81100000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.3340000000000001,"15min":1.4379999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8029999999999999,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.234},{"interface":"front","average":{"1min":1.179,"5min":1.2509999999999999,"15min":1.3280000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.452,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.5880000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4650000000000001,"15min":1.5169999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.9350000000000001,"5min":2.23,"15min":2.23},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4370000000000001,"15min":1.5509999999999999},"min":{"1min":0.67000000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.579,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.4339999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.292,"15min":1.4670000000000001},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.671,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.073},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.534,"15min":1.6719999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8839999999999999,"5min":3.391,"15min":3.391},"last":1.4990000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.502,"15min":1.6399999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.96,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.534},{"interface":"front","average":{"1min":1.387,"5min":1.427,"15min":1.613},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7989999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.1950000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574921,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":946176,"data_stored":400559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.90000000000000002,"15min":0.83199999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.181,"5min":10.557,"15min":10.557},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2829999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":1.123}]},{"osd":1,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76500000000000001,"15min":0.83199999999999996},"min":{"1min":0.36699999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.2529999999999999,"5min":1.532,"15min":1.532},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.0429999999999999,"15min":0.999},"min":{"1min":0.52600000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":3.3260000000000001,"5min":13.090999999999999,"15min":13.090999999999999},"last":0.98099999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.91900000000000004,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2430000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":1.048},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.1399999999999999,"15min":1.0649999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.2480000000000002,"5min":13.460000000000001,"15min":13.460000000000001},"last":0.80100000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5620000000000001,"5min":4.367,"15min":4.367},"last":1.333},{"interface":"front","average":{"1min":1.141,"5min":1.214,"15min":1.329},"min":{"1min":0.71799999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5660000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.171}]},{"osd":5,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.355,"15min":1.3169999999999999},"min":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":3.4399999999999999,"5min":13.231,"15min":13.231},"last":1.294},{"interface":"front","average":{"1min":1.048,"5min":1.2709999999999999,"15min":1.321},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6060000000000001,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":1.2150000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.417,"15min":1.395},"min":{"1min":0.82399999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.5030000000000001,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.2290000000000001,"15min":1.351},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.498,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.3899999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.337,"15min":1.431},"min":{"1min":0.77900000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.4670000000000001,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.2929999999999999,"15min":1.4670000000000001},"min":{"1min":0.76400000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6619999999999999,"5min":4.032,"15min":4.032},"last":1.512}]}]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":402059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.80700000000000005,"15min":0.91600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1040000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.86699999999999999,"15min":1.0149999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3580000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.466}]},{"osd":1,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.92700000000000005,"15min":1.1299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1899999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.548},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.97599999999999998,"15min":1.1339999999999999},"min":{"1min":0.50900000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3759999999999999,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.637}]},{"osd":3,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.88100000000000001,"15min":0.95499999999999996},"min":{"1min":0.48799999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4710000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.075,"15min":1.2709999999999999},"min":{"1min":0.54100000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.333,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.8480000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.292,"15min":1.4159999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.544,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.3029999999999999,"15min":1.405},"min":{"1min":0.92900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.444,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":1.4039999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.294,"15min":1.456},"min":{"1min":0.86799999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.454,"5min":2.508,"15min":2.508},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.29,"15min":1.4179999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.52,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.821}]},{"osd":6,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.2350000000000001,"15min":1.345},"min":{"1min":0.82699999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6100000000000001,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.367,"15min":1.524},"min":{"1min":0.92100000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.6439999999999999,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.772}]},{"osd":7,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.363,"15min":1.5189999999999999},"min":{"1min":0.78700000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.659,"5min":2.859,"15min":2.859},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.343,"15min":1.4910000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.7250000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.591}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.752},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.97199999999999998,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.7749999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.86499999999999999,"15min":0.83199999999999996},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4730000000000001,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":2.1619999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.76800000000000002,"15min":0.70199999999999996},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.131,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.89200000000000002,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5700000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.8320000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.95599999999999996,"15min":0.94199999999999995},"min":{"1min":0.55200000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3660000000000001,"5min":1.675,"15min":1.675},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89900000000000002,"15min":0.876},"min":{"1min":0.48199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.302,"5min":1.464,"15min":1.464},"last":1.0529999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.256,"15min":1.23},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.655,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.25,"15min":1.252},"min":{"1min":0.81200000000000006,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.655,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":2.1080000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.254,"15min":1.1919999999999999},"min":{"1min":0.78300000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5660000000000001,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":2.02},{"interface":"front","average":{"1min":1.179,"5min":1.222,"15min":1.1739999999999999},"min":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7170000000000001,"5min":3.286,"15min":3.286},"last":1.6739999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.2669999999999999,"15min":1.2150000000000001},"min":{"1min":0.84299999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7430000000000001,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":4.2750000000000004},{"interface":"front","average":{"1min":1.244,"5min":1.252,"15min":1.1870000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.663,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":3.7040000000000002}]},{"osd":7,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2909999999999999,"15min":1.2869999999999999},"min":{"1min":0.76500000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5960000000000001,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":2.7280000000000002},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3149999999999999,"15min":1.2549999999999999},"min":{"1min":0.75600000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.607,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":3.653}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":990815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.1040000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89000000000000001,"15min":1.024},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.018,"15min":1.069},"min":{"1min":0.56100000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.647,"5min":1.865,"15min":1.865},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.872,"15min":0.95699999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.365,"5min":1.847,"15min":1.847},"last":0.628}]},{"osd":3,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.02,"15min":1.1240000000000001},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.79,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.91400000000000003,"15min":1.0620000000000001},"min":{"1min":0.47899999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.72299999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.3240000000000001,"15min":1.4279999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.3010000000000002,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.081,"5min":1.1319999999999999,"15min":1.254},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.615,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.874}]},{"osd":5,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.194,"15min":1.331},"min":{"1min":0.748,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.9730000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.25,"15min":1.4450000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.708,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.22}]},{"osd":6,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2909999999999999,"15min":1.4430000000000001},"min":{"1min":0.79400000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.986,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.3069999999999999,"15min":1.4610000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.9399999999999999,"5min":2.738,"15min":2.738},"last":1.2430000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.413,"15min":1.538},"min":{"1min":0.95799999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.097},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3620000000000001,"15min":1.522},"min":{"1min":0.83499999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8560000000000001,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.2869999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:10:31.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:10:31.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.854+0000 7f3e833c4640 1 Processor -- start 2023-11-24T08:10:31.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.854+0000 7f3e833c4640 1 -- start start 2023-11-24T08:10:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.855+0000 7f3e833c4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e7c15b080 0x7f3e7c15b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.855+0000 7f3e833c4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c15e3f0 con 0x7f3e7c15df80 2023-11-24T08:10:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.855+0000 7f3e833c4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c15ab80 con 0x7f3e7c0b2250 2023-11-24T08:10:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.855+0000 7f3e833c4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7c15a990 con 0x7f3e7c15b080 2023-11-24T08:10:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e8095f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3e7c15df80 0x7f3e7c0ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35596/0 (socket says 172.21.15.107:35596) 2023-11-24T08:10:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e81961640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e7c15b080 0x7f3e7c15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e8095f640 1 -- 172.21.15.107:0/2036149091 learned_addr learned my addr 172.21.15.107:0/2036149091 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e81961640 1 -- 172.21.15.107:0/2036149091 >> v1:172.21.15.120:6789/0 conn(0x7f3e7c0b2250 legacy=0x7f3e7c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e81961640 1 -- 172.21.15.107:0/2036149091 >> v1:172.21.15.107:6789/0 conn(0x7f3e7c15df80 legacy=0x7f3e7c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e81961640 1 -- 172.21.15.107:0/2036149091 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c0b2d40 con 0x7f3e7c15b080 2023-11-24T08:10:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.856+0000 7f3e73fff640 1 -- 172.21.15.107:0/2036149091 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357331355 0 0) 0x7f3e7c15e3f0 con 0x7f3e7c15df80 2023-11-24T08:10:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.857+0000 7f3e81961640 1 --2- 172.21.15.107:0/2036149091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15b080 0x7f3e7c15b490 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009d90 tx=0x7f3e7402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf539c91b779fc3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.857+0000 7f3e73fff640 1 -- 172.21.15.107:0/2036149091 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7402cc60 con 0x7f3e7c15b080 2023-11-24T08:10:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.857+0000 7f3e73fff640 1 -- 172.21.15.107:0/2036149091 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7402cdc0 con 0x7f3e7c15b080 2023-11-24T08:10:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.857+0000 7f3e73fff640 1 -- 172.21.15.107:0/2036149091 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74034660 con 0x7f3e7c15b080 2023-11-24T08:10:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 -- 172.21.15.107:0/2036149091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15b080 msgr2=0x7f3e7c15b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 --2- 172.21.15.107:0/2036149091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15b080 0x7f3e7c15b490 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009d90 tx=0x7f3e7402beb0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 -- 172.21.15.107:0/2036149091 shutdown_connections 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 --2- 172.21.15.107:0/2036149091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15b080 0x7f3e7c15b490 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 -- 172.21.15.107:0/2036149091 >> 172.21.15.107:0/2036149091 conn(0x7f3e7c06a920 msgr2=0x7f3e7c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 -- 172.21.15.107:0/2036149091 shutdown_connections 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.858+0000 7f3e833c4640 1 -- 172.21.15.107:0/2036149091 wait complete. 2023-11-24T08:10:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.860+0000 7f3e833c4640 1 Processor -- start 2023-11-24T08:10:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.860+0000 7f3e833c4640 1 -- start start 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e7c0b2250 0x7f3e7c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e7c15b080 0x7f3e7c071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c148da0 con 0x7f3e7c15df80 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c151be0 con 0x7f3e7c15b080 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e833c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0c92e0 con 0x7f3e7c0b2250 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54978/0 (socket says 172.21.15.107:54978) 2023-11-24T08:10:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 -- 172.21.15.107:0/1795082758 learned_addr learned my addr 172.21.15.107:0/1795082758 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81160640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e7c0b2250 0x7f3e7c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 -- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e7c0b2250 msgr2=0x7f3e7c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e8095f640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e7c15b080 0x7f3e7c071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e7c0b2250 0x7f3e7c071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 -- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e7c15b080 msgr2=0x7f3e7c071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.861+0000 7f3e81961640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e7c15b080 0x7f3e7c071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e81961640 1 -- 172.21.15.107:0/1795082758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74009a70 con 0x7f3e7c15df80 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e81961640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009d90 tx=0x7f3e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7403e610 con 0x7f3e7c15df80 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7403e770 con 0x7f3e7c15df80 2023-11-24T08:10:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c0c95c0 con 0x7f3e7c15df80 2023-11-24T08:10:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7403eb20 con 0x7f3e7c15df80 2023-11-24T08:10:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.862+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c151800 con 0x7f3e7c15df80 2023-11-24T08:10:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.863+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c1519c0 con 0x7f3e7c15df80 2023-11-24T08:10:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.864+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f3e7403b050 con 0x7f3e7c15df80 2023-11-24T08:10:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.866+0000 7f3e71ffb640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 0x7f3e6006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.866+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e740aaa20 con 0x7f3e7c15df80 2023-11-24T08:10:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.866+0000 7f3e81160640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 0x7f3e6006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.866+0000 7f3e81160640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 0x7f3e6006db70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e78003d90 tx=0x7f3e78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:31.870+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e74077410 con 0x7f3e7c15df80 2023-11-24T08:10:32.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.042+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e7c15e960 con 0x7f3e6006b6b0 2023-11-24T08:10:32.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.051+0000 7f3e71ffb640 1 -- 172.21.15.107:0/1795082758 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287078 (secure 0 0 0) 0x7f3e78003a30 con 0x7f3e6006b6b0 2023-11-24T08:10:32.057 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.056+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 msgr2=0x7f3e6006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.056+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 0x7f3e6006db70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e78003d90 tx=0x7f3e78009040 comp rx=0 tx=0).stop 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.056+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 msgr2=0x7f3e7c0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.056+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009d90 tx=0x7f3e74004a40 comp rx=0 tx=0).stop 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 shutdown_connections 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e6006b6b0 0x7f3e6006db70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7c15df80 0x7f3e7c0705c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e7c15b080 0x7f3e7c071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 --2- 172.21.15.107:0/1795082758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e7c0b2250 0x7f3e7c071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 >> 172.21.15.107:0/1795082758 conn(0x7f3e7c06a920 msgr2=0x7f3e7c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:32.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 shutdown_connections 2023-11-24T08:10:32.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.057+0000 7f3e833c4640 1 -- 172.21.15.107:0/1795082758 wait complete. 2023-11-24T08:10:32.076 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2023-11-24T08:10:31.537894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258000,"kb_used_data":8552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716720,"statfs":{"total":773094113280,"available":772829921280,"internally_reserved":0,"allocated":8757248,"data_stored":4389132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":255315518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002202"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211772+0000","last_change":"2023-11-24T08:10:27.225266+0000","last_active":"2023-11-24T08:10:28.211772+0000","last_peered":"2023-11-24T08:10:28.211772+0000","last_clean":"2023-11-24T08:10:28.211772+0000","last_became_active":"2023-11-24T08:10:27.224675+0000","last_became_peered":"2023-11-24T08:10:27.224675+0000","last_unstale":"2023-11-24T08:10:28.211772+0000","last_undegraded":"2023-11-24T08:10:28.211772+0000","last_fullsized":"2023-11-24T08:10:28.211772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:12:21.351644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214517+0000","last_change":"2023-11-24T08:10:27.212593+0000","last_active":"2023-11-24T08:10:28.214517+0000","last_peered":"2023-11-24T08:10:28.214517+0000","last_clean":"2023-11-24T08:10:28.214517+0000","last_became_active":"2023-11-24T08:10:27.212131+0000","last_became_peered":"2023-11-24T08:10:27.212131+0000","last_unstale":"2023-11-24T08:10:28.214517+0000","last_undegraded":"2023-11-24T08:10:28.214517+0000","last_fullsized":"2023-11-24T08:10:28.214517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:05.594017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226337+0000","last_change":"2023-11-24T08:10:27.215218+0000","last_active":"2023-11-24T08:10:27.226337+0000","last_peered":"2023-11-24T08:10:27.226337+0000","last_clean":"2023-11-24T08:10:27.226337+0000","last_became_active":"2023-11-24T08:10:27.214907+0000","last_became_peered":"2023-11-24T08:10:27.214907+0000","last_unstale":"2023-11-24T08:10:27.226337+0000","last_undegraded":"2023-11-24T08:10:27.226337+0000","last_fullsized":"2023-11-24T08:10:27.226337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:40:37.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.212926+0000","last_change":"2023-11-24T08:10:27.208149+0000","last_active":"2023-11-24T08:10:27.212926+0000","last_peered":"2023-11-24T08:10:27.212926+0000","last_clean":"2023-11-24T08:10:27.212926+0000","last_became_active":"2023-11-24T08:10:27.206429+0000","last_became_peered":"2023-11-24T08:10:27.206429+0000","last_unstale":"2023-11-24T08:10:27.212926+0000","last_undegraded":"2023-11-24T08:10:27.212926+0000","last_fullsized":"2023-11-24T08:10:27.212926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:19:02.751811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.223611+0000","last_change":"2023-11-24T08:10:27.213601+0000","last_active":"2023-11-24T08:10:27.223611+0000","last_peered":"2023-11-24T08:10:27.223611+0000","last_clean":"2023-11-24T08:10:27.223611+0000","last_became_active":"2023-11-24T08:10:27.213335+0000","last_became_peered":"2023-11-24T08:10:27.213335+0000","last_unstale":"2023-11-24T08:10:27.223611+0000","last_undegraded":"2023-11-24T08:10:27.223611+0000","last_fullsized":"2023-11-24T08:10:27.223611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:12:52.603647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214526+0000","last_change":"2023-11-24T08:10:27.214997+0000","last_active":"2023-11-24T08:10:28.214526+0000","last_peered":"2023-11-24T08:10:28.214526+0000","last_clean":"2023-11-24T08:10:28.214526+0000","last_became_active":"2023-11-24T08:10:27.214723+0000","last_became_peered":"2023-11-24T08:10:27.214723+0000","last_unstale":"2023-11-24T08:10:28.214526+0000","last_undegraded":"2023-11-24T08:10:28.214526+0000","last_fullsized":"2023-11-24T08:10:28.214526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:37.030588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.211346+0000","last_change":"2023-11-24T08:10:27.208106+0000","last_active":"2023-11-24T08:10:27.211346+0000","last_peered":"2023-11-24T08:10:27.211346+0000","last_clean":"2023-11-24T08:10:27.211346+0000","last_became_active":"2023-11-24T08:10:27.207812+0000","last_became_peered":"2023-11-24T08:10:27.207812+0000","last_unstale":"2023-11-24T08:10:27.211346+0000","last_undegraded":"2023-11-24T08:10:27.211346+0000","last_fullsized":"2023-11-24T08:10:27.211346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:44:37.873957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214122+0000","last_change":"2023-11-24T08:10:27.199753+0000","last_active":"2023-11-24T08:10:28.214122+0000","last_peered":"2023-11-24T08:10:28.214122+0000","last_clean":"2023-11-24T08:10:28.214122+0000","last_became_active":"2023-11-24T08:10:27.199213+0000","last_became_peered":"2023-11-24T08:10:27.199213+0000","last_unstale":"2023-11-24T08:10:28.214122+0000","last_undegraded":"2023-11-24T08:10:28.214122+0000","last_fullsized":"2023-11-24T08:10:28.214122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:56.330516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230048+0000","last_change":"2023-11-24T08:10:27.224075+0000","last_active":"2023-11-24T08:10:27.230048+0000","last_peered":"2023-11-24T08:10:27.230048+0000","last_clean":"2023-11-24T08:10:27.230048+0000","last_became_active":"2023-11-24T08:10:27.223797+0000","last_became_peered":"2023-11-24T08:10:27.223797+0000","last_unstale":"2023-11-24T08:10:27.230048+0000","last_undegraded":"2023-11-24T08:10:27.230048+0000","last_fullsized":"2023-11-24T08:10:27.230048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:55.886735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221727+0000","last_change":"2023-11-24T08:10:27.217542+0000","last_active":"2023-11-24T08:10:27.221727+0000","last_peered":"2023-11-24T08:10:27.221727+0000","last_clean":"2023-11-24T08:10:27.221727+0000","last_became_active":"2023-11-24T08:10:27.217132+0000","last_became_peered":"2023-11-24T08:10:27.217132+0000","last_unstale":"2023-11-24T08:10:27.221727+0000","last_undegraded":"2023-11-24T08:10:27.221727+0000","last_fullsized":"2023-11-24T08:10:27.221727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:24.165893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211169+0000","last_change":"2023-11-24T08:10:27.223066+0000","last_active":"2023-11-24T08:10:28.211169+0000","last_peered":"2023-11-24T08:10:28.211169+0000","last_clean":"2023-11-24T08:10:28.211169+0000","last_became_active":"2023-11-24T08:10:27.221960+0000","last_became_peered":"2023-11-24T08:10:27.221960+0000","last_unstale":"2023-11-24T08:10:28.211169+0000","last_undegraded":"2023-11-24T08:10:28.211169+0000","last_fullsized":"2023-11-24T08:10:28.211169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.223992+0000","last_change":"2023-11-24T08:10:27.212403+0000","last_active":"2023-11-24T08:10:27.223992+0000","last_peered":"2023-11-24T08:10:27.223992+0000","last_clean":"2023-11-24T08:10:27.223992+0000","last_became_active":"2023-11-24T08:10:27.207943+0000","last_became_peered":"2023-11-24T08:10:27.207943+0000","last_unstale":"2023-11-24T08:10:27.223992+0000","last_undegraded":"2023-11-24T08:10:27.223992+0000","last_fullsized":"2023-11-24T08:10:27.223992+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:35.135070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276490+0000","last_change":"2023-11-24T08:10:27.218397+0000","last_active":"2023-11-24T08:10:28.276490+0000","last_peered":"2023-11-24T08:10:28.276490+0000","last_clean":"2023-11-24T08:10:28.276490+0000","last_became_active":"2023-11-24T08:10:27.218062+0000","last_became_peered":"2023-11-24T08:10:27.218062+0000","last_unstale":"2023-11-24T08:10:28.276490+0000","last_undegraded":"2023-11-24T08:10:28.276490+0000","last_fullsized":"2023-11-24T08:10:28.276490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:23:15.270466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226882+0000","last_change":"2023-11-24T08:10:27.218853+0000","last_active":"2023-11-24T08:10:27.226882+0000","last_peered":"2023-11-24T08:10:27.226882+0000","last_clean":"2023-11-24T08:10:27.226882+0000","last_became_active":"2023-11-24T08:10:27.213120+0000","last_became_peered":"2023-11-24T08:10:27.213120+0000","last_unstale":"2023-11-24T08:10:27.226882+0000","last_undegraded":"2023-11-24T08:10:27.226882+0000","last_fullsized":"2023-11-24T08:10:27.226882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:14.263852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276496+0000","last_change":"2023-11-24T08:10:27.205790+0000","last_active":"2023-11-24T08:10:28.276496+0000","last_peered":"2023-11-24T08:10:28.276496+0000","last_clean":"2023-11-24T08:10:28.276496+0000","last_became_active":"2023-11-24T08:10:27.205501+0000","last_became_peered":"2023-11-24T08:10:27.205501+0000","last_unstale":"2023-11-24T08:10:28.276496+0000","last_undegraded":"2023-11-24T08:10:28.276496+0000","last_fullsized":"2023-11-24T08:10:28.276496+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:35:49.516231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222734+0000","last_change":"2023-11-24T08:10:27.218897+0000","last_active":"2023-11-24T08:10:27.222734+0000","last_peered":"2023-11-24T08:10:27.222734+0000","last_clean":"2023-11-24T08:10:27.222734+0000","last_became_active":"2023-11-24T08:10:27.218492+0000","last_became_peered":"2023-11-24T08:10:27.218492+0000","last_unstale":"2023-11-24T08:10:27.222734+0000","last_undegraded":"2023-11-24T08:10:27.222734+0000","last_fullsized":"2023-11-24T08:10:27.222734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:35.252989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.231518+0000","last_change":"2023-11-24T08:10:27.230806+0000","last_active":"2023-11-24T08:10:27.231518+0000","last_peered":"2023-11-24T08:10:27.231518+0000","last_clean":"2023-11-24T08:10:27.231518+0000","last_became_active":"2023-11-24T08:10:27.230466+0000","last_became_peered":"2023-11-24T08:10:27.230466+0000","last_unstale":"2023-11-24T08:10:27.231518+0000","last_undegraded":"2023-11-24T08:10:27.231518+0000","last_fullsized":"2023-11-24T08:10:27.231518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:54.398110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.210048+0000","last_change":"2023-11-24T08:10:27.206081+0000","last_active":"2023-11-24T08:10:27.210048+0000","last_peered":"2023-11-24T08:10:27.210048+0000","last_clean":"2023-11-24T08:10:27.210048+0000","last_became_active":"2023-11-24T08:10:27.202268+0000","last_became_peered":"2023-11-24T08:10:27.202268+0000","last_unstale":"2023-11-24T08:10:27.210048+0000","last_undegraded":"2023-11-24T08:10:27.210048+0000","last_fullsized":"2023-11-24T08:10:27.210048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:26.107488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214410+0000","last_change":"2023-11-24T08:10:27.201937+0000","last_active":"2023-11-24T08:10:28.214410+0000","last_peered":"2023-11-24T08:10:28.214410+0000","last_clean":"2023-11-24T08:10:28.214410+0000","last_became_active":"2023-11-24T08:10:27.198384+0000","last_became_peered":"2023-11-24T08:10:27.198384+0000","last_unstale":"2023-11-24T08:10:28.214410+0000","last_undegraded":"2023-11-24T08:10:28.214410+0000","last_fullsized":"2023-11-24T08:10:28.214410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:22:08.878591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276486+0000","last_change":"2023-11-24T08:10:27.217630+0000","last_active":"2023-11-24T08:10:28.276486+0000","last_peered":"2023-11-24T08:10:28.276486+0000","last_clean":"2023-11-24T08:10:28.276486+0000","last_became_active":"2023-11-24T08:10:27.217348+0000","last_became_peered":"2023-11-24T08:10:27.217348+0000","last_unstale":"2023-11-24T08:10:28.276486+0000","last_undegraded":"2023-11-24T08:10:28.276486+0000","last_fullsized":"2023-11-24T08:10:28.276486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:15.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.213631+0000","last_change":"2023-11-24T08:10:27.211125+0000","last_active":"2023-11-24T08:10:27.213631+0000","last_peered":"2023-11-24T08:10:27.213631+0000","last_clean":"2023-11-24T08:10:27.213631+0000","last_became_active":"2023-11-24T08:10:27.210802+0000","last_became_peered":"2023-11-24T08:10:27.210802+0000","last_unstale":"2023-11-24T08:10:27.213631+0000","last_undegraded":"2023-11-24T08:10:27.213631+0000","last_fullsized":"2023-11-24T08:10:27.213631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:54.408281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.224960+0000","last_change":"2023-11-24T08:10:27.221294+0000","last_active":"2023-11-24T08:10:27.224960+0000","last_peered":"2023-11-24T08:10:27.224960+0000","last_clean":"2023-11-24T08:10:27.224960+0000","last_became_active":"2023-11-24T08:10:27.221019+0000","last_became_peered":"2023-11-24T08:10:27.221019+0000","last_unstale":"2023-11-24T08:10:27.224960+0000","last_undegraded":"2023-11-24T08:10:27.224960+0000","last_fullsized":"2023-11-24T08:10:27.224960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:09.766638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210053+0000","last_change":"2023-11-24T08:10:27.213708+0000","last_active":"2023-11-24T08:10:28.210053+0000","last_peered":"2023-11-24T08:10:28.210053+0000","last_clean":"2023-11-24T08:10:28.210053+0000","last_became_active":"2023-11-24T08:10:27.213390+0000","last_became_peered":"2023-11-24T08:10:27.213390+0000","last_unstale":"2023-11-24T08:10:28.210053+0000","last_undegraded":"2023-11-24T08:10:28.210053+0000","last_fullsized":"2023-11-24T08:10:28.210053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:10:10.993270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.209935+0000","last_change":"2023-11-24T08:10:27.221241+0000","last_active":"2023-11-24T08:10:28.209935+0000","last_peered":"2023-11-24T08:10:28.209935+0000","last_clean":"2023-11-24T08:10:28.209935+0000","last_became_active":"2023-11-24T08:10:27.219668+0000","last_became_peered":"2023-11-24T08:10:27.219668+0000","last_unstale":"2023-11-24T08:10:28.209935+0000","last_undegraded":"2023-11-24T08:10:28.209935+0000","last_fullsized":"2023-11-24T08:10:28.209935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:47.553001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222635+0000","last_change":"2023-11-24T08:10:27.219465+0000","last_active":"2023-11-24T08:10:27.222635+0000","last_peered":"2023-11-24T08:10:27.222635+0000","last_clean":"2023-11-24T08:10:27.222635+0000","last_became_active":"2023-11-24T08:10:27.219164+0000","last_became_peered":"2023-11-24T08:10:27.219164+0000","last_unstale":"2023-11-24T08:10:27.222635+0000","last_undegraded":"2023-11-24T08:10:27.222635+0000","last_fullsized":"2023-11-24T08:10:27.222635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:51.136885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.222213+0000","last_change":"2023-11-24T08:10:27.219238+0000","last_active":"2023-11-24T08:10:27.222213+0000","last_peered":"2023-11-24T08:10:27.222213+0000","last_clean":"2023-11-24T08:10:27.222213+0000","last_became_active":"2023-11-24T08:10:27.217909+0000","last_became_peered":"2023-11-24T08:10:27.217909+0000","last_unstale":"2023-11-24T08:10:27.222213+0000","last_undegraded":"2023-11-24T08:10:27.222213+0000","last_fullsized":"2023-11-24T08:10:27.222213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:54:46.487986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226357+0000","last_change":"2023-11-24T08:10:27.216997+0000","last_active":"2023-11-24T08:10:27.226357+0000","last_peered":"2023-11-24T08:10:27.226357+0000","last_clean":"2023-11-24T08:10:27.226357+0000","last_became_active":"2023-11-24T08:10:27.214936+0000","last_became_peered":"2023-11-24T08:10:27.214936+0000","last_unstale":"2023-11-24T08:10:27.226357+0000","last_undegraded":"2023-11-24T08:10:27.226357+0000","last_fullsized":"2023-11-24T08:10:27.226357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:34:02.186464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221619+0000","last_change":"2023-11-24T08:10:27.217610+0000","last_active":"2023-11-24T08:10:27.221619+0000","last_peered":"2023-11-24T08:10:27.221619+0000","last_clean":"2023-11-24T08:10:27.221619+0000","last_became_active":"2023-11-24T08:10:27.217092+0000","last_became_peered":"2023-11-24T08:10:27.217092+0000","last_unstale":"2023-11-24T08:10:27.221619+0000","last_undegraded":"2023-11-24T08:10:27.221619+0000","last_fullsized":"2023-11-24T08:10:27.221619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:56.872996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230605+0000","last_change":"2023-11-24T08:10:27.225938+0000","last_active":"2023-11-24T08:10:27.230605+0000","last_peered":"2023-11-24T08:10:27.230605+0000","last_clean":"2023-11-24T08:10:27.230605+0000","last_became_active":"2023-11-24T08:10:27.225615+0000","last_became_peered":"2023-11-24T08:10:27.225615+0000","last_unstale":"2023-11-24T08:10:27.230605+0000","last_undegraded":"2023-11-24T08:10:27.230605+0000","last_fullsized":"2023-11-24T08:10:27.230605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:57:57.406223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221388+0000","last_change":"2023-11-24T08:10:27.215414+0000","last_active":"2023-11-24T08:10:27.221388+0000","last_peered":"2023-11-24T08:10:27.221388+0000","last_clean":"2023-11-24T08:10:27.221388+0000","last_became_active":"2023-11-24T08:10:27.215143+0000","last_became_peered":"2023-11-24T08:10:27.215143+0000","last_unstale":"2023-11-24T08:10:27.221388+0000","last_undegraded":"2023-11-24T08:10:27.221388+0000","last_fullsized":"2023-11-24T08:10:27.221388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:22.971334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.211323+0000","last_change":"2023-11-24T08:10:27.207564+0000","last_active":"2023-11-24T08:10:27.211323+0000","last_peered":"2023-11-24T08:10:27.211323+0000","last_clean":"2023-11-24T08:10:27.211323+0000","last_became_active":"2023-11-24T08:10:27.207005+0000","last_became_peered":"2023-11-24T08:10:27.207005+0000","last_unstale":"2023-11-24T08:10:27.211323+0000","last_undegraded":"2023-11-24T08:10:27.211323+0000","last_fullsized":"2023-11-24T08:10:27.211323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:51.725014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213669+0000","last_change":"2023-11-24T08:10:27.218854+0000","last_active":"2023-11-24T08:10:28.213669+0000","last_peered":"2023-11-24T08:10:28.213669+0000","last_clean":"2023-11-24T08:10:28.213669+0000","last_became_active":"2023-11-24T08:10:27.218190+0000","last_became_peered":"2023-11-24T08:10:27.218190+0000","last_unstale":"2023-11-24T08:10:28.213669+0000","last_undegraded":"2023-11-24T08:10:28.213669+0000","last_fullsized":"2023-11-24T08:10:28.213669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:12.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211825+0000","last_change":"2023-11-24T08:10:27.221597+0000","last_active":"2023-11-24T08:10:28.211825+0000","last_peered":"2023-11-24T08:10:28.211825+0000","last_clean":"2023-11-24T08:10:28.211825+0000","last_became_active":"2023-11-24T08:10:27.221311+0000","last_became_peered":"2023-11-24T08:10:27.221311+0000","last_unstale":"2023-11-24T08:10:28.211825+0000","last_undegraded":"2023-11-24T08:10:28.211825+0000","last_fullsized":"2023-11-24T08:10:28.211825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:45.191425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"77'4","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.212139+0000","last_change":"2023-11-24T08:10:26.178357+0000","last_active":"2023-11-24T08:10:28.212139+0000","last_peered":"2023-11-24T08:10:28.212139+0000","last_clean":"2023-11-24T08:10:28.212139+0000","last_became_active":"2023-11-24T08:10:26.177974+0000","last_became_peered":"2023-11-24T08:10:26.177974+0000","last_unstale":"2023-11-24T08:10:28.212139+0000","last_undegraded":"2023-11-24T08:10:28.212139+0000","last_fullsized":"2023-11-24T08:10:28.212139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:08:44.981165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.224170+0000","last_change":"2023-11-24T08:10:27.215429+0000","last_active":"2023-11-24T08:10:27.224170+0000","last_peered":"2023-11-24T08:10:27.224170+0000","last_clean":"2023-11-24T08:10:27.224170+0000","last_became_active":"2023-11-24T08:10:27.215115+0000","last_became_peered":"2023-11-24T08:10:27.215115+0000","last_unstale":"2023-11-24T08:10:27.224170+0000","last_undegraded":"2023-11-24T08:10:27.224170+0000","last_fullsized":"2023-11-24T08:10:27.224170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:13.235501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276810+0000","last_change":"2023-11-24T08:10:26.156400+0000","last_active":"2023-11-24T08:10:28.276810+0000","last_peered":"2023-11-24T08:10:28.276810+0000","last_clean":"2023-11-24T08:10:28.276810+0000","last_became_active":"2023-11-24T08:10:26.154580+0000","last_became_peered":"2023-11-24T08:10:26.154580+0000","last_unstale":"2023-11-24T08:10:28.276810+0000","last_undegraded":"2023-11-24T08:10:28.276810+0000","last_fullsized":"2023-11-24T08:10:28.276810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:56.455782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214719+0000","last_change":"2023-11-24T08:10:27.207575+0000","last_active":"2023-11-24T08:10:28.214719+0000","last_peered":"2023-11-24T08:10:28.214719+0000","last_clean":"2023-11-24T08:10:28.214719+0000","last_became_active":"2023-11-24T08:10:27.207215+0000","last_became_peered":"2023-11-24T08:10:27.207215+0000","last_unstale":"2023-11-24T08:10:28.214719+0000","last_undegraded":"2023-11-24T08:10:28.214719+0000","last_fullsized":"2023-11-24T08:10:28.214719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:51.136885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.232293+0000","last_change":"2023-11-24T08:10:26.168479+0000","last_active":"2023-11-24T08:10:27.232293+0000","last_peered":"2023-11-24T08:10:27.232293+0000","last_clean":"2023-11-24T08:10:27.232293+0000","last_became_active":"2023-11-24T08:10:26.168155+0000","last_became_peered":"2023-11-24T08:10:26.168155+0000","last_unstale":"2023-11-24T08:10:27.232293+0000","last_undegraded":"2023-11-24T08:10:27.232293+0000","last_fullsized":"2023-11-24T08:10:27.232293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:55.326173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211413+0000","last_change":"2023-11-24T08:10:27.224724+0000","last_active":"2023-11-24T08:10:28.211413+0000","last_peered":"2023-11-24T08:10:28.211413+0000","last_clean":"2023-11-24T08:10:28.211413+0000","last_became_active":"2023-11-24T08:10:27.224458+0000","last_became_peered":"2023-11-24T08:10:27.224458+0000","last_unstale":"2023-11-24T08:10:28.211413+0000","last_undegraded":"2023-11-24T08:10:28.211413+0000","last_fullsized":"2023-11-24T08:10:28.211413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:35:42.452650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.215026+0000","last_change":"2023-11-24T08:10:26.171776+0000","last_active":"2023-11-24T08:10:28.215026+0000","last_peered":"2023-11-24T08:10:28.215026+0000","last_clean":"2023-11-24T08:10:28.215026+0000","last_became_active":"2023-11-24T08:10:26.171421+0000","last_became_peered":"2023-11-24T08:10:26.171421+0000","last_unstale":"2023-11-24T08:10:28.215026+0000","last_undegraded":"2023-11-24T08:10:28.215026+0000","last_fullsized":"2023-11-24T08:10:28.215026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:06.871902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221413+0000","last_change":"2023-11-24T08:10:27.217022+0000","last_active":"2023-11-24T08:10:27.221413+0000","last_peered":"2023-11-24T08:10:27.221413+0000","last_clean":"2023-11-24T08:10:27.221413+0000","last_became_active":"2023-11-24T08:10:27.216754+0000","last_became_peered":"2023-11-24T08:10:27.216754+0000","last_unstale":"2023-11-24T08:10:27.221413+0000","last_undegraded":"2023-11-24T08:10:27.221413+0000","last_fullsized":"2023-11-24T08:10:27.221413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:09.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"77'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236135+0000","last_change":"2023-11-24T08:10:26.175207+0000","last_active":"2023-11-24T08:10:27.236135+0000","last_peered":"2023-11-24T08:10:27.236135+0000","last_clean":"2023-11-24T08:10:27.236135+0000","last_became_active":"2023-11-24T08:10:26.174892+0000","last_became_peered":"2023-11-24T08:10:26.174892+0000","last_unstale":"2023-11-24T08:10:27.236135+0000","last_undegraded":"2023-11-24T08:10:27.236135+0000","last_fullsized":"2023-11-24T08:10:27.236135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:12:47.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.216635+0000","last_change":"2023-11-24T08:10:27.214536+0000","last_active":"2023-11-24T08:10:27.216635+0000","last_peered":"2023-11-24T08:10:27.216635+0000","last_clean":"2023-11-24T08:10:27.216635+0000","last_became_active":"2023-11-24T08:10:27.214154+0000","last_became_peered":"2023-11-24T08:10:27.214154+0000","last_unstale":"2023-11-24T08:10:27.216635+0000","last_undegraded":"2023-11-24T08:10:27.216635+0000","last_fullsized":"2023-11-24T08:10:27.216635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:43.939464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214735+0000","last_change":"2023-11-24T08:10:26.175012+0000","last_active":"2023-11-24T08:10:28.214735+0000","last_peered":"2023-11-24T08:10:28.214735+0000","last_clean":"2023-11-24T08:10:28.214735+0000","last_became_active":"2023-11-24T08:10:26.172944+0000","last_became_peered":"2023-11-24T08:10:26.172944+0000","last_unstale":"2023-11-24T08:10:28.214735+0000","last_undegraded":"2023-11-24T08:10:28.214735+0000","last_fullsized":"2023-11-24T08:10:28.214735+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:50.808137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225582+0000","last_change":"2023-11-24T08:10:27.216769+0000","last_active":"2023-11-24T08:10:27.225582+0000","last_peered":"2023-11-24T08:10:27.225582+0000","last_clean":"2023-11-24T08:10:27.225582+0000","last_became_active":"2023-11-24T08:10:27.216466+0000","last_became_peered":"2023-11-24T08:10:27.216466+0000","last_unstale":"2023-11-24T08:10:27.225582+0000","last_undegraded":"2023-11-24T08:10:27.225582+0000","last_fullsized":"2023-11-24T08:10:27.225582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:34:45.284416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.182558+0000","last_change":"2023-11-24T08:10:26.174299+0000","last_active":"2023-11-24T08:10:27.182558+0000","last_peered":"2023-11-24T08:10:27.182558+0000","last_clean":"2023-11-24T08:10:27.182558+0000","last_became_active":"2023-11-24T08:10:26.173991+0000","last_became_peered":"2023-11-24T08:10:26.173991+0000","last_unstale":"2023-11-24T08:10:27.182558+0000","last_undegraded":"2023-11-24T08:10:27.182558+0000","last_fullsized":"2023-11-24T08:10:27.182558+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:00:42.119077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276665+0000","last_change":"2023-11-24T08:10:27.212558+0000","last_active":"2023-11-24T08:10:28.276665+0000","last_peered":"2023-11-24T08:10:28.276665+0000","last_clean":"2023-11-24T08:10:28.276665+0000","last_became_active":"2023-11-24T08:10:27.211948+0000","last_became_peered":"2023-11-24T08:10:27.211948+0000","last_unstale":"2023-11-24T08:10:28.276665+0000","last_undegraded":"2023-11-24T08:10:28.276665+0000","last_fullsized":"2023-11-24T08:10:28.276665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:01.349793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.202099+0000","last_change":"2023-11-24T08:10:26.172236+0000","last_active":"2023-11-24T08:10:27.202099+0000","last_peered":"2023-11-24T08:10:27.202099+0000","last_clean":"2023-11-24T08:10:27.202099+0000","last_became_active":"2023-11-24T08:10:26.171907+0000","last_became_peered":"2023-11-24T08:10:26.171907+0000","last_unstale":"2023-11-24T08:10:27.202099+0000","last_undegraded":"2023-11-24T08:10:27.202099+0000","last_fullsized":"2023-11-24T08:10:27.202099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:19.388669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.220887+0000","last_change":"2023-11-24T08:10:27.217603+0000","last_active":"2023-11-24T08:10:27.220887+0000","last_peered":"2023-11-24T08:10:27.220887+0000","last_clean":"2023-11-24T08:10:27.220887+0000","last_became_active":"2023-11-24T08:10:27.215278+0000","last_became_peered":"2023-11-24T08:10:27.215278+0000","last_unstale":"2023-11-24T08:10:27.220887+0000","last_undegraded":"2023-11-24T08:10:27.220887+0000","last_fullsized":"2023-11-24T08:10:27.220887+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:38:48.633056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.209722+0000","last_change":"2023-11-24T08:10:26.169121+0000","last_active":"2023-11-24T08:10:27.209722+0000","last_peered":"2023-11-24T08:10:27.209722+0000","last_clean":"2023-11-24T08:10:27.209722+0000","last_became_active":"2023-11-24T08:10:26.168857+0000","last_became_peered":"2023-11-24T08:10:26.168857+0000","last_unstale":"2023-11-24T08:10:27.209722+0000","last_undegraded":"2023-11-24T08:10:27.209722+0000","last_fullsized":"2023-11-24T08:10:27.209722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:45.405096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211916+0000","last_change":"2023-11-24T08:10:27.223114+0000","last_active":"2023-11-24T08:10:28.211916+0000","last_peered":"2023-11-24T08:10:28.211916+0000","last_clean":"2023-11-24T08:10:28.211916+0000","last_became_active":"2023-11-24T08:10:27.222792+0000","last_became_peered":"2023-11-24T08:10:27.222792+0000","last_unstale":"2023-11-24T08:10:28.211916+0000","last_undegraded":"2023-11-24T08:10:28.211916+0000","last_fullsized":"2023-11-24T08:10:28.211916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:27.857377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214918+0000","last_change":"2023-11-24T08:10:26.169378+0000","last_active":"2023-11-24T08:10:28.214918+0000","last_peered":"2023-11-24T08:10:28.214918+0000","last_clean":"2023-11-24T08:10:28.214918+0000","last_became_active":"2023-11-24T08:10:26.168569+0000","last_became_peered":"2023-11-24T08:10:26.168569+0000","last_unstale":"2023-11-24T08:10:28.214918+0000","last_undegraded":"2023-11-24T08:10:28.214918+0000","last_fullsized":"2023-11-24T08:10:28.214918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:53.403938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214839+0000","last_change":"2023-11-24T08:10:27.214469+0000","last_active":"2023-11-24T08:10:28.214839+0000","last_peered":"2023-11-24T08:10:28.214839+0000","last_clean":"2023-11-24T08:10:28.214839+0000","last_became_active":"2023-11-24T08:10:27.214176+0000","last_became_peered":"2023-11-24T08:10:27.214176+0000","last_unstale":"2023-11-24T08:10:28.214839+0000","last_undegraded":"2023-11-24T08:10:28.214839+0000","last_fullsized":"2023-11-24T08:10:28.214839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:46.409439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"77'3","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236344+0000","last_change":"2023-11-24T08:10:26.166945+0000","last_active":"2023-11-24T08:10:27.236344+0000","last_peered":"2023-11-24T08:10:27.236344+0000","last_clean":"2023-11-24T08:10:27.236344+0000","last_became_active":"2023-11-24T08:10:26.166666+0000","last_became_peered":"2023-11-24T08:10:26.166666+0000","last_unstale":"2023-11-24T08:10:27.236344+0000","last_undegraded":"2023-11-24T08:10:27.236344+0000","last_fullsized":"2023-11-24T08:10:27.236344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:01:29.556715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.217192+0000","last_change":"2023-11-24T08:10:27.209127+0000","last_active":"2023-11-24T08:10:27.217192+0000","last_peered":"2023-11-24T08:10:27.217192+0000","last_clean":"2023-11-24T08:10:27.217192+0000","last_became_active":"2023-11-24T08:10:27.208802+0000","last_became_peered":"2023-11-24T08:10:27.208802+0000","last_unstale":"2023-11-24T08:10:27.217192+0000","last_undegraded":"2023-11-24T08:10:27.217192+0000","last_fullsized":"2023-11-24T08:10:27.217192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:28.019517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"77'6","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.237978+0000","last_change":"2023-11-24T08:10:26.169829+0000","last_active":"2023-11-24T08:10:27.237978+0000","last_peered":"2023-11-24T08:10:27.237978+0000","last_clean":"2023-11-24T08:10:27.237978+0000","last_became_active":"2023-11-24T08:10:26.169497+0000","last_became_peered":"2023-11-24T08:10:26.169497+0000","last_unstale":"2023-11-24T08:10:27.237978+0000","last_undegraded":"2023-11-24T08:10:27.237978+0000","last_fullsized":"2023-11-24T08:10:27.237978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:50.808137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210507+0000","last_change":"2023-11-24T08:10:27.213421+0000","last_active":"2023-11-24T08:10:28.210507+0000","last_peered":"2023-11-24T08:10:28.210507+0000","last_clean":"2023-11-24T08:10:28.210507+0000","last_became_active":"2023-11-24T08:10:27.212333+0000","last_became_peered":"2023-11-24T08:10:27.212333+0000","last_unstale":"2023-11-24T08:10:28.210507+0000","last_undegraded":"2023-11-24T08:10:28.210507+0000","last_fullsized":"2023-11-24T08:10:28.210507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:59:31.551186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.230239+0000","last_change":"2023-11-24T08:10:26.175964+0000","last_active":"2023-11-24T08:10:27.230239+0000","last_peered":"2023-11-24T08:10:27.230239+0000","last_clean":"2023-11-24T08:10:27.230239+0000","last_became_active":"2023-11-24T08:10:26.175679+0000","last_became_peered":"2023-11-24T08:10:26.175679+0000","last_unstale":"2023-11-24T08:10:27.230239+0000","last_undegraded":"2023-11-24T08:10:27.230239+0000","last_fullsized":"2023-11-24T08:10:27.230239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:04.440151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.220698+0000","last_change":"2023-11-24T08:10:27.215063+0000","last_active":"2023-11-24T08:10:27.220698+0000","last_peered":"2023-11-24T08:10:27.220698+0000","last_clean":"2023-11-24T08:10:27.220698+0000","last_became_active":"2023-11-24T08:10:27.214700+0000","last_became_peered":"2023-11-24T08:10:27.214700+0000","last_unstale":"2023-11-24T08:10:27.220698+0000","last_undegraded":"2023-11-24T08:10:27.220698+0000","last_fullsized":"2023-11-24T08:10:27.220698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:16:22.949678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214931+0000","last_change":"2023-11-24T08:10:26.169175+0000","last_active":"2023-11-24T08:10:28.214931+0000","last_peered":"2023-11-24T08:10:28.214931+0000","last_clean":"2023-11-24T08:10:28.214931+0000","last_became_active":"2023-11-24T08:10:26.168866+0000","last_became_peered":"2023-11-24T08:10:26.168866+0000","last_unstale":"2023-11-24T08:10:28.214931+0000","last_undegraded":"2023-11-24T08:10:28.214931+0000","last_fullsized":"2023-11-24T08:10:28.214931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:17.756973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276628+0000","last_change":"2023-11-24T08:10:27.204593+0000","last_active":"2023-11-24T08:10:28.276628+0000","last_peered":"2023-11-24T08:10:28.276628+0000","last_clean":"2023-11-24T08:10:28.276628+0000","last_became_active":"2023-11-24T08:10:27.204006+0000","last_became_peered":"2023-11-24T08:10:27.204006+0000","last_unstale":"2023-11-24T08:10:28.276628+0000","last_undegraded":"2023-11-24T08:10:28.276628+0000","last_fullsized":"2023-11-24T08:10:28.276628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:08:45.985509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.196513+0000","last_change":"2023-11-24T08:10:26.175420+0000","last_active":"2023-11-24T08:10:27.196513+0000","last_peered":"2023-11-24T08:10:27.196513+0000","last_clean":"2023-11-24T08:10:27.196513+0000","last_became_active":"2023-11-24T08:10:26.173600+0000","last_became_peered":"2023-11-24T08:10:26.173600+0000","last_unstale":"2023-11-24T08:10:27.196513+0000","last_undegraded":"2023-11-24T08:10:27.196513+0000","last_fullsized":"2023-11-24T08:10:27.196513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:12.487792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226972+0000","last_change":"2023-11-24T08:10:27.219467+0000","last_active":"2023-11-24T08:10:27.226972+0000","last_peered":"2023-11-24T08:10:27.226972+0000","last_clean":"2023-11-24T08:10:27.226972+0000","last_became_active":"2023-11-24T08:10:27.219226+0000","last_became_peered":"2023-11-24T08:10:27.219226+0000","last_unstale":"2023-11-24T08:10:27.226972+0000","last_undegraded":"2023-11-24T08:10:27.226972+0000","last_fullsized":"2023-11-24T08:10:27.226972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:33.868126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.236253+0000","last_change":"2023-11-24T08:10:26.164429+0000","last_active":"2023-11-24T08:10:27.236253+0000","last_peered":"2023-11-24T08:10:27.236253+0000","last_clean":"2023-11-24T08:10:27.236253+0000","last_became_active":"2023-11-24T08:10:26.164086+0000","last_became_peered":"2023-11-24T08:10:26.164086+0000","last_unstale":"2023-11-24T08:10:27.236253+0000","last_undegraded":"2023-11-24T08:10:27.236253+0000","last_fullsized":"2023-11-24T08:10:27.236253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:36.026244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.221746+0000","last_change":"2023-11-24T08:10:27.218180+0000","last_active":"2023-11-24T08:10:27.221746+0000","last_peered":"2023-11-24T08:10:27.221746+0000","last_clean":"2023-11-24T08:10:27.221746+0000","last_became_active":"2023-11-24T08:10:27.217915+0000","last_became_peered":"2023-11-24T08:10:27.217915+0000","last_unstale":"2023-11-24T08:10:27.221746+0000","last_undegraded":"2023-11-24T08:10:27.221746+0000","last_fullsized":"2023-11-24T08:10:27.221746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:12.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214245+0000","last_change":"2023-11-24T08:10:26.171128+0000","last_active":"2023-11-24T08:10:28.214245+0000","last_peered":"2023-11-24T08:10:28.214245+0000","last_clean":"2023-11-24T08:10:28.214245+0000","last_became_active":"2023-11-24T08:10:26.170772+0000","last_became_peered":"2023-11-24T08:10:26.170772+0000","last_unstale":"2023-11-24T08:10:28.214245+0000","last_undegraded":"2023-11-24T08:10:28.214245+0000","last_fullsized":"2023-11-24T08:10:28.214245+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:34.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.212231+0000","last_change":"2023-11-24T08:10:27.225820+0000","last_active":"2023-11-24T08:10:28.212231+0000","last_peered":"2023-11-24T08:10:28.212231+0000","last_clean":"2023-11-24T08:10:28.212231+0000","last_became_active":"2023-11-24T08:10:27.225617+0000","last_became_peered":"2023-11-24T08:10:27.225617+0000","last_unstale":"2023-11-24T08:10:28.212231+0000","last_undegraded":"2023-11-24T08:10:28.212231+0000","last_fullsized":"2023-11-24T08:10:28.212231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:05.087800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"77'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.231754+0000","last_change":"2023-11-24T08:10:26.173050+0000","last_active":"2023-11-24T08:10:27.231754+0000","last_peered":"2023-11-24T08:10:27.231754+0000","last_clean":"2023-11-24T08:10:27.231754+0000","last_became_active":"2023-11-24T08:10:26.172416+0000","last_became_peered":"2023-11-24T08:10:26.172416+0000","last_unstale":"2023-11-24T08:10:27.231754+0000","last_undegraded":"2023-11-24T08:10:27.231754+0000","last_fullsized":"2023-11-24T08:10:27.231754+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:04.589673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.212945+0000","last_change":"2023-11-24T08:10:27.208563+0000","last_active":"2023-11-24T08:10:27.212945+0000","last_peered":"2023-11-24T08:10:27.212945+0000","last_clean":"2023-11-24T08:10:27.212945+0000","last_became_active":"2023-11-24T08:10:27.207818+0000","last_became_peered":"2023-11-24T08:10:27.207818+0000","last_unstale":"2023-11-24T08:10:27.212945+0000","last_undegraded":"2023-11-24T08:10:27.212945+0000","last_fullsized":"2023-11-24T08:10:27.212945+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:17.515094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"77'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214276+0000","last_change":"2023-11-24T08:10:26.165415+0000","last_active":"2023-11-24T08:10:28.214276+0000","last_peered":"2023-11-24T08:10:28.214276+0000","last_clean":"2023-11-24T08:10:28.214276+0000","last_became_active":"2023-11-24T08:10:26.165085+0000","last_became_peered":"2023-11-24T08:10:26.165085+0000","last_unstale":"2023-11-24T08:10:28.214276+0000","last_undegraded":"2023-11-24T08:10:28.214276+0000","last_fullsized":"2023-11-24T08:10:28.214276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:08.762295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225850+0000","last_change":"2023-11-24T08:10:27.217035+0000","last_active":"2023-11-24T08:10:27.225850+0000","last_peered":"2023-11-24T08:10:27.225850+0000","last_clean":"2023-11-24T08:10:27.225850+0000","last_became_active":"2023-11-24T08:10:27.216571+0000","last_became_peered":"2023-11-24T08:10:27.216571+0000","last_unstale":"2023-11-24T08:10:27.225850+0000","last_undegraded":"2023-11-24T08:10:27.225850+0000","last_fullsized":"2023-11-24T08:10:27.225850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:00:24.832767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210715+0000","last_change":"2023-11-24T08:10:26.178461+0000","last_active":"2023-11-24T08:10:28.210715+0000","last_peered":"2023-11-24T08:10:28.210715+0000","last_clean":"2023-11-24T08:10:28.210715+0000","last_became_active":"2023-11-24T08:10:26.178125+0000","last_became_peered":"2023-11-24T08:10:26.178125+0000","last_unstale":"2023-11-24T08:10:28.210715+0000","last_undegraded":"2023-11-24T08:10:28.210715+0000","last_fullsized":"2023-11-24T08:10:28.210715+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:48.852696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.225828+0000","last_change":"2023-11-24T08:10:27.218809+0000","last_active":"2023-11-24T08:10:27.225828+0000","last_peered":"2023-11-24T08:10:27.225828+0000","last_clean":"2023-11-24T08:10:27.225828+0000","last_became_active":"2023-11-24T08:10:27.216100+0000","last_became_peered":"2023-11-24T08:10:27.216100+0000","last_unstale":"2023-11-24T08:10:27.225828+0000","last_undegraded":"2023-11-24T08:10:27.225828+0000","last_fullsized":"2023-11-24T08:10:27.225828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:01.358124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.215099+0000","last_change":"2023-11-24T08:10:26.170548+0000","last_active":"2023-11-24T08:10:28.215099+0000","last_peered":"2023-11-24T08:10:28.215099+0000","last_clean":"2023-11-24T08:10:28.215099+0000","last_became_active":"2023-11-24T08:10:26.170229+0000","last_became_peered":"2023-11-24T08:10:26.170229+0000","last_unstale":"2023-11-24T08:10:28.215099+0000","last_undegraded":"2023-11-24T08:10:28.215099+0000","last_fullsized":"2023-11-24T08:10:28.215099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:46.113937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.227424+0000","last_change":"2023-11-24T08:10:27.220307+0000","last_active":"2023-11-24T08:10:27.227424+0000","last_peered":"2023-11-24T08:10:27.227424+0000","last_clean":"2023-11-24T08:10:27.227424+0000","last_became_active":"2023-11-24T08:10:27.219987+0000","last_became_peered":"2023-11-24T08:10:27.219987+0000","last_unstale":"2023-11-24T08:10:27.227424+0000","last_undegraded":"2023-11-24T08:10:27.227424+0000","last_fullsized":"2023-11-24T08:10:27.227424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:10.043211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.196729+0000","last_change":"2023-11-24T08:10:26.166255+0000","last_active":"2023-11-24T08:10:27.196729+0000","last_peered":"2023-11-24T08:10:27.196729+0000","last_clean":"2023-11-24T08:10:27.196729+0000","last_became_active":"2023-11-24T08:10:26.165814+0000","last_became_peered":"2023-11-24T08:10:26.165814+0000","last_unstale":"2023-11-24T08:10:27.196729+0000","last_undegraded":"2023-11-24T08:10:27.196729+0000","last_fullsized":"2023-11-24T08:10:27.196729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:50.119768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226771+0000","last_change":"2023-11-24T08:10:27.220440+0000","last_active":"2023-11-24T08:10:27.226771+0000","last_peered":"2023-11-24T08:10:27.226771+0000","last_clean":"2023-11-24T08:10:27.226771+0000","last_became_active":"2023-11-24T08:10:27.220105+0000","last_became_peered":"2023-11-24T08:10:27.220105+0000","last_unstale":"2023-11-24T08:10:27.226771+0000","last_undegraded":"2023-11-24T08:10:27.226771+0000","last_fullsized":"2023-11-24T08:10:27.226771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:37.191476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276582+0000","last_change":"2023-11-24T08:10:26.155493+0000","last_active":"2023-11-24T08:10:28.276582+0000","last_peered":"2023-11-24T08:10:28.276582+0000","last_clean":"2023-11-24T08:10:28.276582+0000","last_became_active":"2023-11-24T08:10:26.154836+0000","last_became_peered":"2023-11-24T08:10:26.154836+0000","last_unstale":"2023-11-24T08:10:28.276582+0000","last_undegraded":"2023-11-24T08:10:28.276582+0000","last_fullsized":"2023-11-24T08:10:28.276582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:53:05.119965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276817+0000","last_change":"2023-11-24T08:10:27.212067+0000","last_active":"2023-11-24T08:10:28.276817+0000","last_peered":"2023-11-24T08:10:28.276817+0000","last_clean":"2023-11-24T08:10:28.276817+0000","last_became_active":"2023-11-24T08:10:27.211784+0000","last_became_peered":"2023-11-24T08:10:27.211784+0000","last_unstale":"2023-11-24T08:10:28.276817+0000","last_undegraded":"2023-11-24T08:10:28.276817+0000","last_fullsized":"2023-11-24T08:10:28.276817+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:47.553001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.214570+0000","last_change":"2023-11-24T08:10:26.171117+0000","last_active":"2023-11-24T08:10:28.214570+0000","last_peered":"2023-11-24T08:10:28.214570+0000","last_clean":"2023-11-24T08:10:28.214570+0000","last_became_active":"2023-11-24T08:10:26.170860+0000","last_became_peered":"2023-11-24T08:10:26.170860+0000","last_unstale":"2023-11-24T08:10:28.214570+0000","last_undegraded":"2023-11-24T08:10:28.214570+0000","last_fullsized":"2023-11-24T08:10:28.214570+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:42.935121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.219878+0000","last_change":"2023-11-24T08:10:27.212153+0000","last_active":"2023-11-24T08:10:27.219878+0000","last_peered":"2023-11-24T08:10:27.219878+0000","last_clean":"2023-11-24T08:10:27.219878+0000","last_became_active":"2023-11-24T08:10:27.210605+0000","last_became_peered":"2023-11-24T08:10:27.210605+0000","last_unstale":"2023-11-24T08:10:27.219878+0000","last_undegraded":"2023-11-24T08:10:27.219878+0000","last_fullsized":"2023-11-24T08:10:27.219878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:12:42.719087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.197983+0000","last_change":"2023-11-24T08:10:26.169246+0000","last_active":"2023-11-24T08:10:27.197983+0000","last_peered":"2023-11-24T08:10:27.197983+0000","last_clean":"2023-11-24T08:10:27.197983+0000","last_became_active":"2023-11-24T08:10:26.167800+0000","last_became_peered":"2023-11-24T08:10:26.167800+0000","last_unstale":"2023-11-24T08:10:27.197983+0000","last_undegraded":"2023-11-24T08:10:27.197983+0000","last_fullsized":"2023-11-24T08:10:27.197983+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:48.852696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.210662+0000","last_change":"2023-11-24T08:10:27.219616+0000","last_active":"2023-11-24T08:10:28.210662+0000","last_peered":"2023-11-24T08:10:28.210662+0000","last_clean":"2023-11-24T08:10:28.210662+0000","last_became_active":"2023-11-24T08:10:27.219343+0000","last_became_peered":"2023-11-24T08:10:27.219343+0000","last_unstale":"2023-11-24T08:10:28.210662+0000","last_undegraded":"2023-11-24T08:10:28.210662+0000","last_fullsized":"2023-11-24T08:10:28.210662+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:32:30.302468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.203370+0000","last_change":"2023-11-24T08:10:26.158707+0000","last_active":"2023-11-24T08:10:27.203370+0000","last_peered":"2023-11-24T08:10:27.203370+0000","last_clean":"2023-11-24T08:10:27.203370+0000","last_became_active":"2023-11-24T08:10:26.154743+0000","last_became_peered":"2023-11-24T08:10:26.154743+0000","last_unstale":"2023-11-24T08:10:27.203370+0000","last_undegraded":"2023-11-24T08:10:27.203370+0000","last_fullsized":"2023-11-24T08:10:27.203370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:09.837478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.211547+0000","last_change":"2023-11-24T08:10:27.222902+0000","last_active":"2023-11-24T08:10:28.211547+0000","last_peered":"2023-11-24T08:10:28.211547+0000","last_clean":"2023-11-24T08:10:28.211547+0000","last_became_active":"2023-11-24T08:10:27.222579+0000","last_became_peered":"2023-11-24T08:10:27.222579+0000","last_unstale":"2023-11-24T08:10:28.211547+0000","last_undegraded":"2023-11-24T08:10:28.211547+0000","last_fullsized":"2023-11-24T08:10:28.211547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:50.582784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.276607+0000","last_change":"2023-11-24T08:10:26.164937+0000","last_active":"2023-11-24T08:10:28.276607+0000","last_peered":"2023-11-24T08:10:28.276607+0000","last_clean":"2023-11-24T08:10:28.276607+0000","last_became_active":"2023-11-24T08:10:26.164635+0000","last_became_peered":"2023-11-24T08:10:26.164635+0000","last_unstale":"2023-11-24T08:10:28.276607+0000","last_undegraded":"2023-11-24T08:10:28.276607+0000","last_fullsized":"2023-11-24T08:10:28.276607+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:50:30.599798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226778+0000","last_change":"2023-11-24T08:10:27.218611+0000","last_active":"2023-11-24T08:10:27.226778+0000","last_peered":"2023-11-24T08:10:27.226778+0000","last_clean":"2023-11-24T08:10:27.226778+0000","last_became_active":"2023-11-24T08:10:27.218299+0000","last_became_peered":"2023-11-24T08:10:27.218299+0000","last_unstale":"2023-11-24T08:10:27.226778+0000","last_undegraded":"2023-11-24T08:10:27.226778+0000","last_fullsized":"2023-11-24T08:10:27.226778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:06.652241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.203059+0000","last_change":"2023-11-24T08:10:26.166584+0000","last_active":"2023-11-24T08:10:27.203059+0000","last_peered":"2023-11-24T08:10:27.203059+0000","last_clean":"2023-11-24T08:10:27.203059+0000","last_became_active":"2023-11-24T08:10:26.166203+0000","last_became_peered":"2023-11-24T08:10:26.166203+0000","last_unstale":"2023-11-24T08:10:27.203059+0000","last_undegraded":"2023-11-24T08:10:27.203059+0000","last_fullsized":"2023-11-24T08:10:27.203059+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:09.246865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213767+0000","last_change":"2023-11-24T08:10:27.216218+0000","last_active":"2023-11-24T08:10:28.213767+0000","last_peered":"2023-11-24T08:10:28.213767+0000","last_clean":"2023-11-24T08:10:28.213767+0000","last_became_active":"2023-11-24T08:10:27.215910+0000","last_became_peered":"2023-11-24T08:10:27.215910+0000","last_unstale":"2023-11-24T08:10:28.213767+0000","last_undegraded":"2023-11-24T08:10:28.213767+0000","last_fullsized":"2023-11-24T08:10:28.213767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:24.165893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.182619+0000","last_change":"2023-11-24T08:10:26.176750+0000","last_active":"2023-11-24T08:10:27.182619+0000","last_peered":"2023-11-24T08:10:27.182619+0000","last_clean":"2023-11-24T08:10:27.182619+0000","last_became_active":"2023-11-24T08:10:26.176467+0000","last_became_peered":"2023-11-24T08:10:26.176467+0000","last_unstale":"2023-11-24T08:10:27.182619+0000","last_undegraded":"2023-11-24T08:10:27.182619+0000","last_fullsized":"2023-11-24T08:10:27.182619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:37.967475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.226360+0000","last_change":"2023-11-24T08:10:27.220915+0000","last_active":"2023-11-24T08:10:27.226360+0000","last_peered":"2023-11-24T08:10:27.226360+0000","last_clean":"2023-11-24T08:10:27.226360+0000","last_became_active":"2023-11-24T08:10:27.220636+0000","last_became_peered":"2023-11-24T08:10:27.220636+0000","last_unstale":"2023-11-24T08:10:27.226360+0000","last_undegraded":"2023-11-24T08:10:27.226360+0000","last_fullsized":"2023-11-24T08:10:27.226360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:20.232558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.208696+0000","last_change":"2023-11-24T08:10:26.170459+0000","last_active":"2023-11-24T08:10:27.208696+0000","last_peered":"2023-11-24T08:10:27.208696+0000","last_clean":"2023-11-24T08:10:27.208696+0000","last_became_active":"2023-11-24T08:10:26.170243+0000","last_became_peered":"2023-11-24T08:10:26.170243+0000","last_unstale":"2023-11-24T08:10:27.208696+0000","last_undegraded":"2023-11-24T08:10:27.208696+0000","last_fullsized":"2023-11-24T08:10:27.208696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:22:07.874248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213489+0000","last_change":"2023-11-24T08:10:27.219419+0000","last_active":"2023-11-24T08:10:28.213489+0000","last_peered":"2023-11-24T08:10:28.213489+0000","last_clean":"2023-11-24T08:10:28.213489+0000","last_became_active":"2023-11-24T08:10:27.218936+0000","last_became_peered":"2023-11-24T08:10:27.218936+0000","last_unstale":"2023-11-24T08:10:28.213489+0000","last_undegraded":"2023-11-24T08:10:28.213489+0000","last_fullsized":"2023-11-24T08:10:28.213489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:56.872996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213933+0000","last_change":"2023-11-24T08:10:26.170453+0000","last_active":"2023-11-24T08:10:28.213933+0000","last_peered":"2023-11-24T08:10:28.213933+0000","last_clean":"2023-11-24T08:10:28.213933+0000","last_became_active":"2023-11-24T08:10:26.169829+0000","last_became_peered":"2023-11-24T08:10:26.169829+0000","last_unstale":"2023-11-24T08:10:28.213933+0000","last_undegraded":"2023-11-24T08:10:28.213933+0000","last_fullsized":"2023-11-24T08:10:28.213933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:26.348763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.228839+0000","last_change":"2023-11-24T08:10:27.221363+0000","last_active":"2023-11-24T08:10:27.228839+0000","last_peered":"2023-11-24T08:10:27.228839+0000","last_clean":"2023-11-24T08:10:27.228839+0000","last_became_active":"2023-11-24T08:10:27.220947+0000","last_became_peered":"2023-11-24T08:10:27.220947+0000","last_unstale":"2023-11-24T08:10:27.228839+0000","last_undegraded":"2023-11-24T08:10:27.228839+0000","last_fullsized":"2023-11-24T08:10:27.228839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:26.129123+0000","last_clean_scrub_stamp":"2023-11-24T08:10:26.129123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:21:25.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"77'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-11-24T08:10:28.213984+0000","last_change":"2023-11-24T08:10:26.171855+0000","last_active":"2023-11-24T08:10:28.213984+0000","last_peered":"2023-11-24T08:10:28.213984+0000","last_clean":"2023-11-24T08:10:28.213984+0000","last_became_active":"2023-11-24T08:10:26.171560+0000","last_became_peered":"2023-11-24T08:10:26.171560+0000","last_unstale":"2023-11-24T08:10:28.213984+0000","last_undegraded":"2023-11-24T08:10:28.213984+0000","last_fullsized":"2023-11-24T08:10:28.213984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:10:25.124780+0000","last_clean_scrub_stamp":"2023-11-24T08:10:25.124780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:24:16.460597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":214,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-11-24T08:10:27.205725+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:10:27.205725+0000","last_peered":"2023-11-24T08:10:27.205725+0000","last_clean":"2023-11-24T08:10:27.205725+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:10:27.205725+0000","last_undegraded":"2023-11-24T08:10:27.205725+0000","last_fullsized":"2023-11-24T08:10:27.205725+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":991431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.046,"15min":0.997},"min":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4179999999999999,"5min":1.621,"15min":1.621},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.137,"15min":1.0860000000000001},"min":{"1min":0.78600000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.548,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.2669999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1399999999999999,"15min":1.1299999999999999},"min":{"1min":0.72199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6200000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.206,"15min":1.1819999999999999},"min":{"1min":0.85399999999999998,"5min":0.628,"15min":0.628},"max":{"1min":1.5409999999999999,"5min":2.198,"15min":2.198},"last":1.151}]},{"osd":2,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.1459999999999999,"15min":1.216},"min":{"1min":0.71199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.502,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.196},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.298,"15min":1.349},"min":{"1min":0.85199999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.603,"5min":4.633,"15min":4.633},"last":1.3009999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.198,"15min":1.1439999999999999},"min":{"1min":0.83399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.5820000000000001,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.88100000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.855,"5min":2.206,"15min":2.206},"last":1.417}]},{"osd":4,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.371,"15min":1.333},"min":{"1min":0.96699999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7250000000000001,"5min":2.851,"15min":2.851},"last":1.391},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.3420000000000001,"15min":1.3280000000000001},"min":{"1min":0.90200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5700000000000001,"5min":3.0510000000000002,"15min":3.0510000000000002},"last":1.3360000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.458,"15min":1.446},"min":{"1min":1.03,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.712,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.105},{"interface":"front","average":{"1min":1.306,"5min":1.4930000000000001,"15min":1.4750000000000001},"min":{"1min":0.97299999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.677,"5min":2.629,"15min":2.629},"last":1.476}]},{"osd":6,"last update":"Fri Nov 24 08:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5800000000000001,"15min":1.538},"min":{"1min":1.0960000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.7589999999999999,"5min":2.746,"15min":2.746},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.49,"15min":1.4399999999999999},"min":{"1min":1.0640000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.677,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4450000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":946176,"data_stored":400539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.081,"15min":1.0900000000000001},"min":{"1min":0.71899999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.073},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.077,"15min":1.107},"min":{"1min":0.69899999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8899999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.0389999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1739999999999999,"15min":1.266},"min":{"1min":0.72699999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.319},{"interface":"front","average":{"1min":1.083,"5min":1.1890000000000001,"15min":1.288},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5980000000000001,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1080000000000001,"15min":1.129},"min":{"1min":0.78500000000000003,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4630000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.21,"5min":1.278,"15min":1.387},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.8999999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.341,"15min":1.3999999999999999},"min":{"1min":0.91300000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7869999999999999,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2989999999999999,"15min":1.3440000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.631,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.141}]},{"osd":4,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4279999999999999,"15min":1.556},"min":{"1min":0.92400000000000004,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1110000000000002,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.5169999999999999,"15min":1.573},"min":{"1min":1.143,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.242,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.278}]},{"osd":5,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.472,"15min":1.573},"min":{"1min":1.0680000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.7849999999999999,"5min":2.629,"15min":2.629},"last":1.24},{"interface":"front","average":{"1min":1.55,"5min":1.595,"15min":1.7050000000000001},"min":{"1min":1.101,"5min":1.022,"15min":1.022},"max":{"1min":2.4249999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.569}]},{"osd":7,"last update":"Fri Nov 24 08:09:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.6140000000000001,"15min":1.736},"min":{"1min":1.2330000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":1.8400000000000001,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.444},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6180000000000001,"15min":1.702},"min":{"1min":1.2629999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.1589999999999998,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.3640000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574920,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":400537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.044,"15min":1.133},"min":{"1min":0.73599999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.86499999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.71,"5min":2.198,"15min":2.198},"last":1.2689999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1220000000000001,"15min":1.179},"min":{"1min":0.88,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.7070000000000001,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.153,"15min":1.147},"min":{"1min":0.86199999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.482}]},{"osd":2,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.2,"15min":1.216},"min":{"1min":0.65100000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1479999999999999,"15min":1.1739999999999999},"min":{"1min":0.74299999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.397,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1579999999999999,"15min":1.1990000000000001},"min":{"1min":0.81799999999999995,"5min":0.432,"15min":0.432},"max":{"1min":1.6850000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.375,"5min":1.28,"15min":1.3480000000000001},"min":{"1min":0.871,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9730000000000001,"5min":1.99,"15min":1.99},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.325,"15min":1.347},"min":{"1min":0.84499999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.774,"5min":1.839,"15min":1.839},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.482,"15min":1.5640000000000001},"min":{"1min":0.96099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.1019999999999999,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":1.393}]},{"osd":6,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3620000000000001,"15min":1.47},"min":{"1min":0.90500000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.73,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4079999999999999,"15min":1.4399999999999999},"min":{"1min":0.98299999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7849999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:09:57 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.494,"15min":1.6100000000000001},"min":{"1min":1.024,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8149999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.45},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.581,"15min":1.619},"min":{"1min":1.284,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.5960000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":400517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.079,"15min":1.155},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.0640000000000001,"15min":1.089},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6000000000000001,"5min":2.222,"15min":2.222},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.72899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.516,"5min":1.74,"15min":1.74},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.1819999999999999,"15min":1.1899999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.27}]},{"osd":2,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.071,"15min":1.169},"min":{"1min":0.65100000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5449999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.347},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.256,"15min":1.3779999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.633,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.81100000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.3340000000000001,"15min":1.4379999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8029999999999999,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":1.234},{"interface":"front","average":{"1min":1.179,"5min":1.2509999999999999,"15min":1.3280000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.452,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.5880000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4650000000000001,"15min":1.5169999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.9350000000000001,"5min":2.23,"15min":2.23},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4370000000000001,"15min":1.5509999999999999},"min":{"1min":0.67000000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.579,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.4339999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.292,"15min":1.4670000000000001},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.671,"5min":3.5899999999999999,"15min":3.5899999999999999},"last":1.073},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.534,"15min":1.6719999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8839999999999999,"5min":3.391,"15min":3.391},"last":1.4990000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.502,"15min":1.6399999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.96,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.534},{"interface":"front","average":{"1min":1.387,"5min":1.427,"15min":1.613},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7989999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":1.1950000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574921,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":946176,"data_stored":400559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.90000000000000002,"15min":0.83199999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.181,"5min":10.557,"15min":10.557},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2829999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":1.123}]},{"osd":1,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76500000000000001,"15min":0.83199999999999996},"min":{"1min":0.36699999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.2529999999999999,"5min":1.532,"15min":1.532},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.0429999999999999,"15min":0.999},"min":{"1min":0.52600000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":3.3260000000000001,"5min":13.090999999999999,"15min":13.090999999999999},"last":0.98099999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.91900000000000004,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2430000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":1.048},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.1399999999999999,"15min":1.0649999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.2480000000000002,"5min":13.460000000000001,"15min":13.460000000000001},"last":0.80100000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5620000000000001,"5min":4.367,"15min":4.367},"last":1.333},{"interface":"front","average":{"1min":1.141,"5min":1.214,"15min":1.329},"min":{"1min":0.71799999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5660000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.171}]},{"osd":5,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.355,"15min":1.3169999999999999},"min":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":3.4399999999999999,"5min":13.231,"15min":13.231},"last":1.294},{"interface":"front","average":{"1min":1.048,"5min":1.2709999999999999,"15min":1.321},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6060000000000001,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":1.2150000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.417,"15min":1.395},"min":{"1min":0.82399999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.5030000000000001,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.2290000000000001,"15min":1.351},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.498,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.3899999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.337,"15min":1.431},"min":{"1min":0.77900000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.4670000000000001,"5min":4.5060000000000002,"15min":4.5060000000000002},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.2929999999999999,"15min":1.4670000000000001},"min":{"1min":0.76400000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6619999999999999,"5min":4.032,"15min":4.032},"last":1.512}]}]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":402059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.80700000000000005,"15min":0.91600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1040000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.86699999999999999,"15min":1.0149999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3580000000000001,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.466}]},{"osd":1,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.92700000000000005,"15min":1.1299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1899999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.548},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.97599999999999998,"15min":1.1339999999999999},"min":{"1min":0.50900000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3759999999999999,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.637}]},{"osd":3,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.88100000000000001,"15min":0.95499999999999996},"min":{"1min":0.48799999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4710000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.075,"15min":1.2709999999999999},"min":{"1min":0.54100000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.333,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":1.8480000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.292,"15min":1.4159999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.544,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.3029999999999999,"15min":1.405},"min":{"1min":0.92900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.444,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":1.4039999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.294,"15min":1.456},"min":{"1min":0.86799999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.454,"5min":2.508,"15min":2.508},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.29,"15min":1.4179999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.52,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.821}]},{"osd":6,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.2350000000000001,"15min":1.345},"min":{"1min":0.82699999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6100000000000001,"5min":3.2210000000000001,"15min":3.2210000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.367,"15min":1.524},"min":{"1min":0.92100000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.6439999999999999,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.772}]},{"osd":7,"last update":"Fri Nov 24 08:09:54 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.363,"15min":1.5189999999999999},"min":{"1min":0.78700000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.659,"5min":2.859,"15min":2.859},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.343,"15min":1.4910000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.7250000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.591}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.752},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.97199999999999998,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.7749999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.86499999999999999,"15min":0.83199999999999996},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4730000000000001,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":2.1619999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.76800000000000002,"15min":0.70199999999999996},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.131,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.89200000000000002,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5700000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.8320000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.95599999999999996,"15min":0.94199999999999995},"min":{"1min":0.55200000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3660000000000001,"5min":1.675,"15min":1.675},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89900000000000002,"15min":0.876},"min":{"1min":0.48199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.302,"5min":1.464,"15min":1.464},"last":1.0529999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.256,"15min":1.23},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.655,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.25,"15min":1.252},"min":{"1min":0.81200000000000006,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.655,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":2.1080000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.254,"15min":1.1919999999999999},"min":{"1min":0.78300000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5660000000000001,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":2.02},{"interface":"front","average":{"1min":1.179,"5min":1.222,"15min":1.1739999999999999},"min":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7170000000000001,"5min":3.286,"15min":3.286},"last":1.6739999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.2669999999999999,"15min":1.2150000000000001},"min":{"1min":0.84299999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7430000000000001,"5min":2.9359999999999999,"15min":2.9359999999999999},"last":4.2750000000000004},{"interface":"front","average":{"1min":1.244,"5min":1.252,"15min":1.1870000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.663,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":3.7040000000000002}]},{"osd":7,"last update":"Fri Nov 24 08:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2909999999999999,"15min":1.2869999999999999},"min":{"1min":0.76500000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5960000000000001,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":2.7280000000000002},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3149999999999999,"15min":1.2549999999999999},"min":{"1min":0.75600000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.607,"5min":3.2559999999999998,"15min":3.2559999999999998},"last":3.653}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":990815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.1040000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89000000000000001,"15min":1.024},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.018,"15min":1.069},"min":{"1min":0.56100000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.647,"5min":1.865,"15min":1.865},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.872,"15min":0.95699999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.365,"5min":1.847,"15min":1.847},"last":0.628}]},{"osd":3,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.02,"15min":1.1240000000000001},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.79,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.91400000000000003,"15min":1.0620000000000001},"min":{"1min":0.47899999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.304,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.72299999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.3240000000000001,"15min":1.4279999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.3010000000000002,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.081,"5min":1.1319999999999999,"15min":1.254},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.615,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.874}]},{"osd":5,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.194,"15min":1.331},"min":{"1min":0.748,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.9730000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.25,"15min":1.4450000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.708,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.22}]},{"osd":6,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2909999999999999,"15min":1.4430000000000001},"min":{"1min":0.79400000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.986,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.3069999999999999,"15min":1.4610000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.9399999999999999,"5min":2.738,"15min":2.738},"last":1.2430000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.413,"15min":1.538},"min":{"1min":0.95799999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.097},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3620000000000001,"15min":1.522},"min":{"1min":0.83499999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8560000000000001,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.2869999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:10:32.082 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:10:32.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:10:32.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.333+0000 7f2da8148640 1 Processor -- start 2023-11-24T08:10:32.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.334+0000 7f2da8148640 1 -- start start 2023-11-24T08:10:32.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00cb850 0x7f2da00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2da00d0a00 0x7f2da006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2da00b40d0 0x7f2da00cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da00cc170 con 0x7f2da00b40d0 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da0070870 con 0x7f2da00d0a00 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.335+0000 7f2da8148640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da0070b50 con 0x7f2da00cb850 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2da00b40d0 0x7f2da00cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2da00b40d0 0x7f2da00cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54990/0 (socket says 172.21.15.107:54990) 2023-11-24T08:10:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 -- 172.21.15.107:0/260682800 learned_addr learned my addr 172.21.15.107:0/260682800 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 -- 172.21.15.107:0/260682800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00cb850 msgr2=0x7f2da00cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da5ee4640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00cb850 0x7f2da00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da56e3640 1 --2- 172.21.15.107:0/260682800 >> v2:172.21.15.120:3300/0 conn(0x7f2da00d0a00 0x7f2da006ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00cb850 0x7f2da00cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 -- 172.21.15.107:0/260682800 >> v2:172.21.15.120:3300/0 conn(0x7f2da00d0a00 msgr2=0x7f2da006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 --2- 172.21.15.107:0/260682800 >> v2:172.21.15.120:3300/0 conn(0x7f2da00d0a00 0x7f2da006ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.336+0000 7f2da66e5640 1 -- 172.21.15.107:0/260682800 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0070e30 con 0x7f2da00b40d0 2023-11-24T08:10:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.337+0000 7f2da66e5640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00b40d0 0x7f2da00cf670 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f2da006b4c0 tx=0x7f2d9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114d1923f42703ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.338+0000 7f2da4ee2640 1 -- 172.21.15.107:0/260682800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d980317e0 con 0x7f2da00b40d0 2023-11-24T08:10:32.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.338+0000 7f2da4ee2640 1 -- 172.21.15.107:0/260682800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98031940 con 0x7f2da00b40d0 2023-11-24T08:10:32.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.338+0000 7f2da4ee2640 1 -- 172.21.15.107:0/260682800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98031cd0 con 0x7f2da00b40d0 2023-11-24T08:10:32.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.338+0000 7f2da8148640 1 -- 172.21.15.107:0/260682800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00b40d0 msgr2=0x7f2da00cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.338+0000 7f2da8148640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00b40d0 0x7f2da00cf670 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f2da006b4c0 tx=0x7f2d9802b880 comp rx=0 tx=0).stop 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 -- 172.21.15.107:0/260682800 shutdown_connections 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00b40d0 0x7f2da00cf670 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 --2- 172.21.15.107:0/260682800 >> v2:172.21.15.120:3300/0 conn(0x7f2da00d0a00 0x7f2da006ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 --2- 172.21.15.107:0/260682800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00cb850 0x7f2da00cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 -- 172.21.15.107:0/260682800 >> 172.21.15.107:0/260682800 conn(0x7f2da006e7d0 msgr2=0x7f2da006ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 -- 172.21.15.107:0/260682800 shutdown_connections 2023-11-24T08:10:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.339+0000 7f2da8148640 1 -- 172.21.15.107:0/260682800 wait complete. 2023-11-24T08:10:32.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.340+0000 7f2da8148640 1 Processor -- start 2023-11-24T08:10:32.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.340+0000 7f2da8148640 1 -- start start 2023-11-24T08:10:32.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00b40d0 0x7f2da006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da00d0a00 0x7f2da015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da006c230 con 0x7f2da00cb850 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da006c480 con 0x7f2da00d0a00 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.341+0000 7f2da8148640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da006c760 con 0x7f2da00b40d0 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55002/0 (socket says 172.21.15.107:55002) 2023-11-24T08:10:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 -- 172.21.15.107:0/1426147738 learned_addr learned my addr 172.21.15.107:0/1426147738 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da5ee4640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00b40d0 0x7f2da006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 -- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00b40d0 msgr2=0x7f2da006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da66e5640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da00d0a00 0x7f2da015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00b40d0 0x7f2da006d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 -- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da00d0a00 msgr2=0x7f2da015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da00d0a00 0x7f2da015ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 -- 172.21.15.107:0/1426147738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980094d0 con 0x7f2da00cb850 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.342+0000 7f2da56e3640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00c7f0 tx=0x7f2d8c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.343+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c004220 con 0x7f2da00cb850 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.343+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c004380 con 0x7f2da00cb850 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.343+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c007840 con 0x7f2da00cb850 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.343+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da006ca40 con 0x7f2da00cb850 2023-11-24T08:10:32.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.343+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da01ed580 con 0x7f2da00cb850 2023-11-24T08:10:32.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.344+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da006cf60 con 0x7f2da00cb850 2023-11-24T08:10:32.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.345+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f2d8c007bb0 con 0x7f2da00cb850 2023-11-24T08:10:32.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.346+0000 7f2d9e7fc640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 0x7f2d8406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.346+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d8c087fb0 con 0x7f2da00cb850 2023-11-24T08:10:32.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.346+0000 7f2da5ee4640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 0x7f2d8406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.347+0000 7f2da5ee4640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 0x7f2d8406dba0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2da00c7990 tx=0x7f2d940083a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.350+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2da006cf60 con 0x7f2da00cb850 2023-11-24T08:10:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.599+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2da006cf60 con 0x7f2da00cb850 2023-11-24T08:10:32.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.604+0000 7f2d9e7fc640 1 -- 172.21.15.107:0/1426147738 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2da006cf60 con 0x7f2da00cb850 2023-11-24T08:10:32.606 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:10:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:10:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:10:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:10:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:10:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:10:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 msgr2=0x7f2d8406dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 0x7f2d8406dba0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2da00c7990 tx=0x7f2d940083a0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 msgr2=0x7f2da015c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00c7f0 tx=0x7f2d8c00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 shutdown_connections 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d8406b6e0 0x7f2d8406dba0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da00d0a00 0x7f2da015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2da00cb850 0x7f2da015c8d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 --2- 172.21.15.107:0/1426147738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2da00b40d0 0x7f2da006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 >> 172.21.15.107:0/1426147738 conn(0x7f2da006e7d0 msgr2=0x7f2da00cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.606+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 shutdown_connections 2023-11-24T08:10:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.607+0000 7f2da8148640 1 -- 172.21.15.107:0/1426147738 wait complete. 2023-11-24T08:10:32.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:10:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.828+0000 7f8679e6b640 1 Processor -- start 2023-11-24T08:10:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.828+0000 7f8679e6b640 1 -- start start 2023-11-24T08:10:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.829+0000 7f8679e6b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f867415c7f0 0x7f867406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.829+0000 7f8679e6b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86740ce6c0 0x7f86740cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.829+0000 7f8679e6b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86740cece0 con 0x7f86740cf450 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.829+0000 7f8679e6b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86740cfe40 con 0x7f86740ce6c0 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.829+0000 7f8679e6b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86740ceaf0 con 0x7f867415c7f0 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86740ce6c0 0x7f86740cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86740ce6c0 0x7f86740cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55014/0 (socket says 172.21.15.107:55014) 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 -- 172.21.15.107:0/884026001 learned_addr learned my addr 172.21.15.107:0/884026001 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f86737fe640 1 --2- 172.21.15.107:0/884026001 >> v2:172.21.15.120:3300/0 conn(0x7f867415c7f0 0x7f867406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 -- 172.21.15.107:0/884026001 >> v2:172.21.15.120:3300/0 conn(0x7f867415c7f0 msgr2=0x7f867406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 --2- 172.21.15.107:0/884026001 >> v2:172.21.15.120:3300/0 conn(0x7f867415c7f0 0x7f867406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 -- 172.21.15.107:0/884026001 >> v1:172.21.15.120:6789/0 conn(0x7f86740cf450 legacy=0x7f86740cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.830+0000 7f8673fff640 1 -- 172.21.15.107:0/884026001 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867406d040 con 0x7f86740ce6c0 2023-11-24T08:10:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.831+0000 7f8673fff640 1 --2- 172.21.15.107:0/884026001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86740ce6c0 0x7f86740cf900 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f86740d0130 tx=0x7f866c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c827c83e36faf9b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f86727fc640 1 -- 172.21.15.107:0/884026001 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c02ca80 con 0x7f86740ce6c0 2023-11-24T08:10:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f86727fc640 1 -- 172.21.15.107:0/884026001 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c02cbe0 con 0x7f86740ce6c0 2023-11-24T08:10:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f86727fc640 1 -- 172.21.15.107:0/884026001 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c0343f0 con 0x7f86740ce6c0 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 -- 172.21.15.107:0/884026001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86740ce6c0 msgr2=0x7f86740cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 --2- 172.21.15.107:0/884026001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86740ce6c0 0x7f86740cf900 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f86740d0130 tx=0x7f866c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 -- 172.21.15.107:0/884026001 shutdown_connections 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 --2- 172.21.15.107:0/884026001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86740ce6c0 0x7f86740cf900 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 --2- 172.21.15.107:0/884026001 >> v2:172.21.15.120:3300/0 conn(0x7f867415c7f0 0x7f867406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 -- 172.21.15.107:0/884026001 >> 172.21.15.107:0/884026001 conn(0x7f867406a920 msgr2=0x7f867406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.832+0000 7f8679e6b640 1 -- 172.21.15.107:0/884026001 shutdown_connections 2023-11-24T08:10:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.833+0000 7f8679e6b640 1 -- 172.21.15.107:0/884026001 wait complete. 2023-11-24T08:10:32.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.834+0000 7f8679e6b640 1 Processor -- start 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.834+0000 7f8679e6b640 1 -- start start 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86740ce6c0 0x7f867415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 0x7f86741f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740c83f0 con 0x7f867415c7f0 2023-11-24T08:10:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867415b530 con 0x7f86740cf450 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.835+0000 7f8679e6b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8674155f60 con 0x7f86740ce6c0 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55022/0 (socket says 172.21.15.107:55022) 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 -- 172.21.15.107:0/1055536711 learned_addr learned my addr 172.21.15.107:0/1055536711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f86737fe640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86740ce6c0 0x7f867415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8672ffd640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 0x7f86741f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 -- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86740ce6c0 msgr2=0x7f867415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86740ce6c0 0x7f867415aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 -- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 msgr2=0x7f86741f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 0x7f86741f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 -- 172.21.15.107:0/1055536711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c009b30 con 0x7f867415c7f0 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8672ffd640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 0x7f86741f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f8673fff640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f866c009f40 tx=0x7f866c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.836+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c009510 con 0x7f867415c7f0 2023-11-24T08:10:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.837+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c03e450 con 0x7f867415c7f0 2023-11-24T08:10:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.837+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c03e7e0 con 0x7f867415c7f0 2023-11-24T08:10:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.837+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86741f3c30 con 0x7f867415c7f0 2023-11-24T08:10:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.837+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8674156220 con 0x7f867415c7f0 2023-11-24T08:10:32.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.838+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8674067470 con 0x7f867415c7f0 2023-11-24T08:10:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.839+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f866c02cd50 con 0x7f867415c7f0 2023-11-24T08:10:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.839+0000 7f865bfff640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 0x7f866006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.840+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f866c0aaf20 con 0x7f867415c7f0 2023-11-24T08:10:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.840+0000 7f86737fe640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 0x7f866006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.840+0000 7f86737fe640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 0x7f866006db70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8664006fd0 tx=0x7f8664008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:32.844+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f866c077970 con 0x7f867415c7f0 2023-11-24T08:10:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.055+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8674155dc0 con 0x7f867415c7f0 2023-11-24T08:10:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.057+0000 7f865bfff640 1 -- 172.21.15.107:0/1055536711 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f8674155dc0 con 0x7f867415c7f0 2023-11-24T08:10:33.059 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:10:33.059 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:10:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 41 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 msgr2=0x7f866006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 0x7f866006db70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8664006fd0 tx=0x7f8664008040 comp rx=0 tx=0).stop 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 msgr2=0x7f86741f3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f866c009f40 tx=0x7f866c004a40 comp rx=0 tx=0).stop 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 shutdown_connections 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f866006b6b0 0x7f866006db70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867415c7f0 0x7f86741f3170 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86740cf450 0x7f86741f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 --2- 172.21.15.107:0/1055536711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86740ce6c0 0x7f867415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.059+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 >> 172.21.15.107:0/1055536711 conn(0x7f867406a920 msgr2=0x7f867406c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.060+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 shutdown_connections 2023-11-24T08:10:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.060+0000 7f8679e6b640 1 -- 172.21.15.107:0/1055536711 wait complete. 2023-11-24T08:10:33.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:10:26.606485+0000', 'modified': '2023-11-24T08:10:29.699709+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6735}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6735': {'gid': 6735, 'name': 'c', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/3083938901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 3083938901}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 3083938901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:10:33.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:10:33.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:10:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.701+0000 7f1197547640 1 Processor -- start 2023-11-24T08:10:33.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.701+0000 7f1197547640 1 -- start start 2023-11-24T08:10:33.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1197547640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f119015b1d0 0x7f1190071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1197547640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11900ca0e0 con 0x7f1190070550 2023-11-24T08:10:33.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1197547640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1190072800 con 0x7f11900b1250 2023-11-24T08:10:33.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1197547640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11900c97b0 con 0x7f119015b1d0 2023-11-24T08:10:33.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1195d44640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1190070550 0x7f11900b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35608/0 (socket says 172.21.15.107:35608) 2023-11-24T08:10:33.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.703+0000 7f1195d44640 1 -- 172.21.15.107:0/183319162 learned_addr learned my addr 172.21.15.107:0/183319162 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:33.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1196545640 1 --2- 172.21.15.107:0/183319162 >> v2:172.21.15.120:3300/0 conn(0x7f119015b1d0 0x7f1190071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625295210 0 0) 0x7f11900ca0e0 con 0x7f1190070550 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c003520 con 0x7f1190070550 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139284661 0 0) 0x7f117c003520 con 0x7f1190070550 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 >> v2:172.21.15.120:3300/0 conn(0x7f119015b1d0 msgr2=0x7f1190071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 --2- 172.21.15.107:0/183319162 >> v2:172.21.15.120:3300/0 conn(0x7f119015b1d0 0x7f1190071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 >> v1:172.21.15.120:6789/0 conn(0x7f11900b1250 legacy=0x7f119015edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900c9a90 con 0x7f1190070550 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1180001670 con 0x7f1190070550 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 >> v1:172.21.15.107:6789/0 conn(0x7f1190070550 legacy=0x7f11900b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c003970 0x7f117c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 0x7f117c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117c005180 0x7f117c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11900c97b0 con 0x7f117c0047c0 2023-11-24T08:10:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c020 con 0x7f117c005180 2023-11-24T08:10:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.704+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c1e0 con 0x7f117c003970 2023-11-24T08:10:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 0x7f117c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1196545640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c003970 0x7f117c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 -- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c003970 msgr2=0x7f117c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c003970 0x7f117c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 -- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117c005180 msgr2=0x7f117c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117c005180 0x7f117c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1195d44640 1 -- 172.21.15.107:0/183319162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c003520 con 0x7f117c0047c0 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.705+0000 7f1196d46640 1 -- 172.21.15.107:0/183319162 reap_dead start 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.706+0000 7f1195d44640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 0x7f117c004c40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f118000c240 tx=0x7f117c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67ad126588a951f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180005ad0 con 0x7f117c0047c0 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1180005c30 con 0x7f117c0047c0 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1195543640 1 -- 172.21.15.107:0/183319162 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118002caf0 con 0x7f117c0047c0 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 -- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 msgr2=0x7f117c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 0x7f117c004c40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f118000c240 tx=0x7f117c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 -- 172.21.15.107:0/183319162 shutdown_connections 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 --2- 172.21.15.107:0/183319162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f117c0047c0 0x7f117c004c40 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 -- 172.21.15.107:0/183319162 >> 172.21.15.107:0/183319162 conn(0x7f11900cbb40 msgr2=0x7f11900cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.707+0000 7f1197547640 1 -- 172.21.15.107:0/183319162 shutdown_connections 2023-11-24T08:10:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.708+0000 7f1197547640 1 -- 172.21.15.107:0/183319162 wait complete. 2023-11-24T08:10:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.709+0000 7f1197547640 1 Processor -- start 2023-11-24T08:10:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.709+0000 7f1197547640 1 -- start start 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c0047c0 0x7f11900b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f119009af60 0x7f119015c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119015d060 con 0x7f1190159b50 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119009b3e0 con 0x7f119009af60 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.710+0000 7f1197547640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900b2220 con 0x7f117c0047c0 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55036/0 (socket says 172.21.15.107:55036) 2023-11-24T08:10:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 -- 172.21.15.107:0/1480704757 learned_addr learned my addr 172.21.15.107:0/1480704757 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1196545640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c0047c0 0x7f11900b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1196d46640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f119009af60 0x7f119015c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 -- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c0047c0 msgr2=0x7f11900b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c0047c0 0x7f11900b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 -- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f119009af60 msgr2=0x7f119015c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f119009af60 0x7f119015c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.711+0000 7f1195d44640 1 -- 172.21.15.107:0/1480704757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900c7710 con 0x7f1190159b50 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1195d44640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1180001930 tx=0x7f118002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1186ffd640 1 -- 172.21.15.107:0/1480704757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118003c040 con 0x7f1190159b50 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1186ffd640 1 -- 172.21.15.107:0/1480704757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118002cd80 con 0x7f1190159b50 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1186ffd640 1 -- 172.21.15.107:0/1480704757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11800072b0 con 0x7f1190159b50 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11900c79f0 con 0x7f1190159b50 2023-11-24T08:10:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.712+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11900c7fb0 con 0x7f1190159b50 2023-11-24T08:10:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.714+0000 7f1186ffd640 1 -- 172.21.15.107:0/1480704757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f1180032070 con 0x7f1190159b50 2023-11-24T08:10:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.715+0000 7f1186ffd640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 0x7f117406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.715+0000 7f1186ffd640 1 -- 172.21.15.107:0/1480704757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11800a9440 con 0x7f1190159b50 2023-11-24T08:10:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.715+0000 7f1196545640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 0x7f117406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.716+0000 7f119b41b740 1 Processor -- start 2023-11-24T08:10:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.716+0000 7f119b41b740 1 -- start start 2023-11-24T08:10:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.716+0000 7f1196545640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 0x7f117406dba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f118c0099b0 tx=0x7f118c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f18270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f5c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999e77880 con 0x563999f19070 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999e86ae0 con 0x563999f187b0 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563999e841d0 con 0x563999f13d70 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55048/0 (socket says 172.21.15.107:55048) 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 -- 172.21.15.107:0/3195267346 learned_addr learned my addr 172.21.15.107:0/3195267346 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 -- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 msgr2=0x563999f18270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f18270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 -- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 msgr2=0x563999f5c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.717+0000 7f1196545640 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.718+0000 7f1196545640 1 -- 172.21.15.107:0/3195267346 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563999e47860 con 0x563999f19070 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.718+0000 7f1196545640 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f118c011d00 tx=0x7f118c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31011bcdbbb4a70 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.718+0000 7f1173fff640 1 -- 172.21.15.107:0/3195267346 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c011d30 con 0x563999f19070 2023-11-24T08:10:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.718+0000 7f1173fff640 1 -- 172.21.15.107:0/3195267346 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c011e90 con 0x563999f19070 2023-11-24T08:10:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.718+0000 7f1173fff640 1 -- 172.21.15.107:0/3195267346 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c021640 con 0x563999f19070 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 -- 172.21.15.107:0/3195267346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 msgr2=0x563999f19450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f118c011d00 tx=0x7f118c014000 comp rx=0 tx=0).stop 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 -- 172.21.15.107:0/3195267346 shutdown_connections 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f5c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f18270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 --2- 172.21.15.107:0/3195267346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f19450 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 -- 172.21.15.107:0/3195267346 >> 172.21.15.107:0/3195267346 conn(0x563999edac70 msgr2=0x563999f5eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 -- 172.21.15.107:0/3195267346 shutdown_connections 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.719+0000 7f119b41b740 1 -- 172.21.15.107:0/3195267346 wait complete. 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.720+0000 7f119b41b740 1 -- start start 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.720+0000 7f119b41b740 1 Processor -- start 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f42310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f42850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999e7fac0 con 0x563999f19070 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999e1c2f0 con 0x563999f187b0 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563999e39c70 con 0x563999f13d70 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f119b41b740 10 client.0 _ll_register_callbacks cb 0x563999e3e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f1196d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f1196d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55050/0 (socket says 172.21.15.107:55050) 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f1196d46640 1 -- 172.21.15.107:0/2447098171 learned_addr learned my addr 172.21.15.107:0/2447098171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.721+0000 7f1196545640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f42310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1195d44640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f42850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 -- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 msgr2=0x563999f42310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f42310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 -- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 msgr2=0x563999f42850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f42850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 -- 172.21.15.107:0/2447098171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c011980 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1195d44640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f42850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f1196d46640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f11900717d0 tx=0x7f118800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73cac24e3a36a6e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f11717fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118800b070 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1188005d40 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563999e78aa0 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f118800deb0 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f119b41b740 10 client.6873 fetch_fsmap learned FSMap version 41 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.722+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x563999e7e140 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563999e78aa0 con 0x563999f19070 2023-11-24T08:10:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 41) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f11880070d0 con 0x563999f19070 2023-11-24T08:10:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 10 client.6873 fetch_fsmap finished waiting for FSMap version 41 2023-11-24T08:10:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 10 client.6873 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:10:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 20 client.6873 populate_metadata read hostname 'smithi107' 2023-11-24T08:10:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f64310 0x563999f7bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 4 client.6873 mds_command: new command op to 6735 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:10:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 --> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563999e787c0 con 0x563999f64310 2023-11-24T08:10:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.723+0000 7f1196545640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f64310 0x563999f7bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.724+0000 7f1196545640 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f64310 0x563999f7bd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a05deca46b410517 server_cookie=c24b248b9d6f4a9c in_seq=0 out_seq=0 2023-11-24T08:10:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.724+0000 7f11717fa640 10 client.6873 ms_handle_connect on v2:172.21.15.107:6836/3083938901 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.724+0000 7f11717fa640 1 -- 172.21.15.107:0/2447098171 <== mds.0 v2:172.21.15.107:6836/3083938901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563999e787c0 con 0x563999f64310 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.724+0000 7f11717fa640 10 client.6873 handle_command_reply: tid=0 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 client.6873 shutdown 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 msgr2=0x563999f635a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f11900717d0 tx=0x7f118800d520 comp rx=0 tx=0).stop 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 shutdown_connections 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f64310 0x563999f7bd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999f635a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f187b0 0x563999f42850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 --2- 172.21.15.107:0/2447098171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f42310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 >> 172.21.15.107:0/2447098171 conn(0x563999edac70 msgr2=0x563999f61ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f11717fa640 0 client.6873 ms_handle_reset on v2:172.21.15.107:6836/3083938901 2023-11-24T08:10:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 shutdown_connections 2023-11-24T08:10:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 1 -- 172.21.15.107:0/2447098171 wait complete. 2023-11-24T08:10:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.725+0000 7f119b41b740 20 client.6873 trim_cache size 0 max 16384 2023-11-24T08:10:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.742+0000 7f119b41b740 1 Processor -- start 2023-11-24T08:10:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.742+0000 7f119b41b740 1 -- start start 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f12f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f19070 0x563999f42e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999be2f20 con 0x563999f187b0 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999f1a460 con 0x563999f19070 2023-11-24T08:10:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563999f0eb10 con 0x563999f13d70 2023-11-24T08:10:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f1195d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f1195d44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55060/0 (socket says 172.21.15.107:55060) 2023-11-24T08:10:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.743+0000 7f1195d44640 1 -- 172.21.15.107:0/1925195335 learned_addr learned my addr 172.21.15.107:0/1925195335 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 -- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 msgr2=0x563999f12f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f12f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 -- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f19070 msgr2=0x563999f42e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f19070 0x563999f42e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 -- 172.21.15.107:0/1925195335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563999e1bb80 con 0x563999f187b0 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1195d44640 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f118000c240 tx=0x7f1180079bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9182fe64183451f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1184ff9640 1 -- 172.21.15.107:0/1925195335 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180079cf0 con 0x563999f187b0 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1184ff9640 1 -- 172.21.15.107:0/1925195335 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118007de40 con 0x563999f187b0 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.744+0000 7f1184ff9640 1 -- 172.21.15.107:0/1925195335 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180082080 con 0x563999f187b0 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- 172.21.15.107:0/1925195335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 msgr2=0x563999f13480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f118000c240 tx=0x7f1180079bb0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- 172.21.15.107:0/1925195335 shutdown_connections 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f19070 0x563999f42e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f187b0 0x563999f13480 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 --2- 172.21.15.107:0/1925195335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f13d70 0x563999f12f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- 172.21.15.107:0/1925195335 >> 172.21.15.107:0/1925195335 conn(0x563999edac70 msgr2=0x563999f0f580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- 172.21.15.107:0/1925195335 shutdown_connections 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- 172.21.15.107:0/1925195335 wait complete. 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.745+0000 7f119b41b740 1 -- start start 2023-11-24T08:10:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.746+0000 7f119b41b740 1 Processor -- start 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f13d70 0x563999f46340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f187b0 0x563999f3cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999f7e480 con 0x563999f19070 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563999f646f0 con 0x563999f13d70 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563999fc9590 con 0x563999f187b0 2023-11-24T08:10:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f119b41b740 10 client.0 _ll_register_callbacks cb 0x7f117c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55072/0 (socket says 172.21.15.107:55072) 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 -- 172.21.15.107:0/1364606908 learned_addr learned my addr 172.21.15.107:0/1364606908 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 -- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f187b0 msgr2=0x563999f3cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f187b0 0x563999f3cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 -- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f13d70 msgr2=0x563999f46340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f13d70 0x563999f46340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.747+0000 7f1196d46640 1 -- 172.21.15.107:0/1364606908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11900c7fb0 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f1196d46640 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1188001330 tx=0x7f11880014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887544eff42f14e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188001dd0 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1188013470 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563999f71720 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1188007e80 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f119b41b740 10 client.6879 fetch_fsmap learned FSMap version 41 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x563999e41170 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563999f71720 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 41) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1188028020 con 0x563999f19070 2023-11-24T08:10:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.748+0000 7f119b41b740 10 client.6879 fetch_fsmap finished waiting for FSMap version 41 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f119b41b740 10 client.6879 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f119b41b740 20 client.6879 populate_metadata read hostname 'smithi107' 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f42e70 0x563999f43250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f119b41b740 4 client.6879 mds_command: new command op to 6735 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 --> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f118c011e90 con 0x563999f42e70 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f1196545640 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f42e70 0x563999f43250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f1196545640 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f42e70 0x563999f43250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe30ab6062002cf server_cookie=945dbdb1db2d4f2f in_seq=0 out_seq=0 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.749+0000 7f11717fa640 10 client.6879 ms_handle_connect on v2:172.21.15.107:6836/3083938901 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f11717fa640 1 -- 172.21.15.107:0/1364606908 <== mds.0 v2:172.21.15.107:6836/3083938901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f118c011e90 con 0x563999f42e70 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f11717fa640 10 client.6879 handle_command_reply: tid=0 2023-11-24T08:10:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 client.6879 shutdown 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 msgr2=0x563999fc8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1188001330 tx=0x7f11880014a0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 shutdown_connections 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:6836/3083938901,v1:172.21.15.107:6837/3083938901] conn(0x563999f42e70 0x563999f43250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563999f19070 0x563999fc8090 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563999f187b0 0x563999f3cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 --2- 172.21.15.107:0/1364606908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563999f13d70 0x563999f46340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 >> 172.21.15.107:0/1364606908 conn(0x563999edac70 msgr2=0x563999f65960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f11717fa640 0 client.6879 ms_handle_reset on v2:172.21.15.107:6836/3083938901 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 shutdown_connections 2023-11-24T08:10:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 1 -- 172.21.15.107:0/1364606908 wait complete. 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.750+0000 7f119b41b740 20 client.6879 trim_cache size 0 max 16384 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 6735, 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:10:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.5537144999999999, 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 41, 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 78, 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 78, 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.997291837000001 2023-11-24T08:10:33.759 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 msgr2=0x7f117406dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 0x7f117406dba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f118c0099b0 tx=0x7f118c008040 comp rx=0 tx=0).stop 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 msgr2=0x7f119009aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1180001930 tx=0x7f118002e070 comp rx=0 tx=0).stop 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 shutdown_connections 2023-11-24T08:10:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117406b6e0 0x7f117406dba0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f119009af60 0x7f119015c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1190159b50 0x7f119009aa20 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 --2- 172.21.15.107:0/1480704757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f117c0047c0 0x7f11900b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 >> 172.21.15.107:0/1480704757 conn(0x7f11900cbb40 msgr2=0x7f11900afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 shutdown_connections 2023-11-24T08:10:33.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.751+0000 7f1197547640 1 -- 172.21.15.107:0/1480704757 wait complete. 2023-11-24T08:10:33.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:10:33.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.973+0000 7f47f2797640 1 Processor -- start 2023-11-24T08:10:33.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.973+0000 7f47f2797640 1 -- start start 2023-11-24T08:10:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47f2797640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47ec06aa30 0x7f47ec0cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47f2797640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47ec0cf420 0x7f47ec0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47f2797640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec0b2690 con 0x7f47ec0b28d0 2023-11-24T08:10:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47f2797640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0cc7f0 con 0x7f47ec06aa30 2023-11-24T08:10:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47f2797640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0cfdb0 con 0x7f47ec0cf420 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47ec06aa30 0x7f47ec0cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.975+0000 7f47ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47ec06aa30 0x7f47ec0cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55086/0 (socket says 172.21.15.107:55086) 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47ebfff640 1 -- 172.21.15.107:0/222839355 learned_addr learned my addr 172.21.15.107:0/222839355 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47f0d34640 1 --2- 172.21.15.107:0/222839355 >> v2:172.21.15.120:3300/0 conn(0x7f47ec0cf420 0x7f47ec0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47ebfff640 1 -- 172.21.15.107:0/222839355 >> v2:172.21.15.120:3300/0 conn(0x7f47ec0cf420 msgr2=0x7f47ec0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47ebfff640 1 --2- 172.21.15.107:0/222839355 >> v2:172.21.15.120:3300/0 conn(0x7f47ec0cf420 0x7f47ec0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47ebfff640 1 -- 172.21.15.107:0/222839355 >> v1:172.21.15.107:6789/0 conn(0x7f47ec0b28d0 legacy=0x7f47ec0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.976+0000 7f47ebfff640 1 -- 172.21.15.107:0/222839355 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec06ed00 con 0x7f47ec06aa30 2023-11-24T08:10:33.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.977+0000 7f47ebfff640 1 --2- 172.21.15.107:0/222839355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec06aa30 0x7f47ec0cc2b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f47ec161230 tx=0x7f47dc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922228d9b8cb06c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.977+0000 7f47eb7fe640 1 -- 172.21.15.107:0/222839355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc02f850 con 0x7f47ec06aa30 2023-11-24T08:10:33.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.977+0000 7f47eb7fe640 1 -- 172.21.15.107:0/222839355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc02f9b0 con 0x7f47ec06aa30 2023-11-24T08:10:33.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.977+0000 7f47eb7fe640 1 -- 172.21.15.107:0/222839355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc02fd40 con 0x7f47ec06aa30 2023-11-24T08:10:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 -- 172.21.15.107:0/222839355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec06aa30 msgr2=0x7f47ec0cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 --2- 172.21.15.107:0/222839355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec06aa30 0x7f47ec0cc2b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f47ec161230 tx=0x7f47dc02ba30 comp rx=0 tx=0).stop 2023-11-24T08:10:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 -- 172.21.15.107:0/222839355 shutdown_connections 2023-11-24T08:10:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 --2- 172.21.15.107:0/222839355 >> v2:172.21.15.120:3300/0 conn(0x7f47ec0cf420 0x7f47ec0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 --2- 172.21.15.107:0/222839355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec06aa30 0x7f47ec0cc2b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 -- 172.21.15.107:0/222839355 >> 172.21.15.107:0/222839355 conn(0x7f47ec0999d0 msgr2=0x7f47ec0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:33.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 -- 172.21.15.107:0/222839355 shutdown_connections 2023-11-24T08:10:33.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.978+0000 7f47f2797640 1 -- 172.21.15.107:0/222839355 wait complete. 2023-11-24T08:10:33.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.980+0000 7f47f2797640 1 Processor -- start 2023-11-24T08:10:33.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.980+0000 7f47f2797640 1 -- start start 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47ec06aa30 0x7f47ec0b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47ec0b28d0 0x7f47ec0b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0b36a0 con 0x7f47ec0cf420 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0b5610 con 0x7f47ec06aa30 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f2797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0b2d70 con 0x7f47ec0b28d0 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f0d34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f0d34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55100/0 (socket says 172.21.15.107:55100) 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.981+0000 7f47f0d34640 1 -- 172.21.15.107:0/785318499 learned_addr learned my addr 172.21.15.107:0/785318499 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47ebfff640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47ec06aa30 0x7f47ec0b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 -- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47ec0b28d0 msgr2=0x7f47ec0b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47e3fff640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47ec0b28d0 0x7f47ec0b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47ec0b28d0 0x7f47ec0b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 -- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47ec06aa30 msgr2=0x7f47ec0b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47ec06aa30 0x7f47ec0b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 -- 172.21.15.107:0/785318499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0094d0 con 0x7f47ec0cf420 2023-11-24T08:10:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f0d34640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f47e4007e60 tx=0x7f47e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4014420 con 0x7f47ec0cf420 2023-11-24T08:10:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e40145a0 con 0x7f47ec0cf420 2023-11-24T08:10:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4014950 con 0x7f47ec0cf420 2023-11-24T08:10:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.982+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec0b3050 con 0x7f47ec0cf420 2023-11-24T08:10:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.983+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec153590 con 0x7f47ec0cf420 2023-11-24T08:10:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.984+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec0b3850 con 0x7f47ec0cf420 2023-11-24T08:10:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.985+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f47e4014ab0 con 0x7f47ec0cf420 2023-11-24T08:10:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.985+0000 7f47e97fa640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 0x7f47cc06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.986+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47e4088930 con 0x7f47ec0cf420 2023-11-24T08:10:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.986+0000 7f47ebfff640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 0x7f47cc06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.986+0000 7f47ebfff640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 0x7f47cc06dba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f47dc036000 tx=0x7f47dc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:33.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:33.990+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47ec0b3850 con 0x7f47ec0cf420 2023-11-24T08:10:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.202+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f47ec0b3850 con 0x7f47ec0cf420 2023-11-24T08:10:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.203+0000 7f47e97fa640 1 -- 172.21.15.107:0/785318499 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f47ec0b3850 con 0x7f47ec0cf420 2023-11-24T08:10:34.205 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.205+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 msgr2=0x7f47cc06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.205+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 0x7f47cc06dba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f47dc036000 tx=0x7f47dc008030 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.205+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 msgr2=0x7f47ec0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.205+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f47e4007e60 tx=0x7f47e400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 shutdown_connections 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47cc06b6e0 0x7f47cc06dba0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47ec0cf420 0x7f47ec0b4b70 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47ec0b28d0 0x7f47ec0b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 --2- 172.21.15.107:0/785318499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47ec06aa30 0x7f47ec0b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 >> 172.21.15.107:0/785318499 conn(0x7f47ec0999d0 msgr2=0x7f47ec0cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 shutdown_connections 2023-11-24T08:10:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:34.206+0000 7f47f2797640 1 -- 172.21.15.107:0/785318499 wait complete. 2023-11-24T08:10:34.226 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:10:34.226 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:10:34.226 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:10:34.227 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:10:34.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.227 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.281 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42036sec preferred_lft 42036sec 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:10:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:34.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.284 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:10:34.367 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:10:34.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.372 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:10:34.423 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:10:34.423 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:10:34.423 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:10:34.425 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:10:34.425 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:10:34.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.425 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:10:34.425 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:10:34.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:10:34.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:10:34.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:10:34.426 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:10:34.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.598 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:10:34.649 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:10:34.649 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:10:34.650 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:10:34.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:10:34.743 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:10:34.743 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:10:34.799 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:10:34.799 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:10:34.800 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:10:34.854 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:10:34.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:34.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:34.994 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:34.994 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:10:35.025 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:35.025 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:10:35.025 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:10:35.025 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:10:35.025 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:10:35.027 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:10:35.027 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:10:35.027 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:10:35.027 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:10:35.027 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:10:35.030 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:10:35.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:10:35.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.113 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:10:35.173 INFO:teuthology.orchestra.run.smithi107.stdout:client6891 2023-11-24T08:10:35.174 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:35.174 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:35.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:10:35.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.425+0000 7f308e4bb640 1 Processor -- start 2023-11-24T08:10:35.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.425+0000 7f308e4bb640 1 -- start start 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308e4bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f30880b2250 0x7f308815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308e4bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308815ab80 con 0x7f30880b2aa0 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308e4bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308815e680 con 0x7f30880b2630 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308e4bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308815a990 con 0x7f30880b2250 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308ca58640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f30880b2aa0 0x7f308815b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35626/0 (socket says 172.21.15.107:35626) 2023-11-24T08:10:35.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.426+0000 7f308ca58640 1 -- 172.21.15.107:0/191090535 learned_addr learned my addr 172.21.15.107:0/191090535 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:35.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3087fff640 1 --2- 172.21.15.107:0/191090535 >> v2:172.21.15.120:3300/0 conn(0x7f30880b2250 0x7f308815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041238486 0 0) 0x7f308815ab80 con 0x7f30880b2aa0 2023-11-24T08:10:35.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003520 con 0x7f30880b2aa0 2023-11-24T08:10:35.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562005805 0 0) 0x7f3070003520 con 0x7f30880b2aa0 2023-11-24T08:10:35.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 >> v2:172.21.15.120:3300/0 conn(0x7f30880b2250 msgr2=0x7f308815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 --2- 172.21.15.107:0/191090535 >> v2:172.21.15.120:3300/0 conn(0x7f30880b2250 0x7f308815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 >> v1:172.21.15.120:6789/0 conn(0x7f30880b2630 legacy=0x7f30880ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308815e960 con 0x7f30880b2aa0 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f30800019f0 con 0x7f30880b2aa0 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 >> v1:172.21.15.107:6789/0 conn(0x7f30880b2aa0 legacy=0x7f308815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30700039c0 0x7f3070003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700047c0 0x7f3070004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 0x7f307000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307000c080 con 0x7f3070005180 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307000c1f0 con 0x7f30700047c0 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.427+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307000c3b0 con 0x7f30700039c0 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 0x7f307000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f30877fe640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700047c0 0x7f3070004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 -- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30700039c0 msgr2=0x7f3070003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:35.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30700039c0 0x7f3070003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 -- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700047c0 msgr2=0x7f3070004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:35.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700047c0 0x7f3070004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.428+0000 7f308ca58640 1 -- 172.21.15.107:0/191090535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070003520 con 0x7f3070005180 2023-11-24T08:10:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.429+0000 7f308ca58640 1 -- 172.21.15.107:0/191090535 reap_dead start 2023-11-24T08:10:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.429+0000 7f308ca58640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 0x7f307000b980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f308000a370 tx=0x7f30700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60479188f42a95f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.429+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080036630 con 0x7f3070005180 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.429+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080031070 con 0x7f3070005180 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.429+0000 7f3086ffd640 1 -- 172.21.15.107:0/191090535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080036a10 con 0x7f3070005180 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 -- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 msgr2=0x7f307000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 0x7f307000b980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f308000a370 tx=0x7f30700057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 -- 172.21.15.107:0/191090535 shutdown_connections 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 --2- 172.21.15.107:0/191090535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3070005180 0x7f307000b980 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 -- 172.21.15.107:0/191090535 >> 172.21.15.107:0/191090535 conn(0x7f308806a920 msgr2=0x7f308806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 -- 172.21.15.107:0/191090535 shutdown_connections 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.430+0000 7f308e4bb640 1 -- 172.21.15.107:0/191090535 wait complete. 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.432+0000 7f308e4bb640 1 Processor -- start 2023-11-24T08:10:35.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.432+0000 7f308e4bb640 1 -- start start 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 0x7f3088158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30880ca580 0x7f30880c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308815ba30 con 0x7f30880ca9e0 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880cae60 con 0x7f3070005180 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308e4bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308815b090 con 0x7f30880ca580 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308ca58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308ca58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55116/0 (socket says 172.21.15.107:55116) 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.433+0000 7f308ca58640 1 -- 172.21.15.107:0/4228013288 learned_addr learned my addr 172.21.15.107:0/4228013288 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f3087fff640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 0x7f3088158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f30877fe640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30880ca580 0x7f30880c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 -- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30880ca580 msgr2=0x7f30880c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30880ca580 0x7f30880c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 -- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 msgr2=0x7f3088158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 0x7f3088158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 -- 172.21.15.107:0/4228013288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308815b370 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f3087fff640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 0x7f3088158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308ca58640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f308002ed90 tx=0x7f3080001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080001930 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080001250 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.434+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308815b650 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.435+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080004d80 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.435+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30880c8cf0 con 0x7f30880ca9e0 2023-11-24T08:10:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.436+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308815bb70 con 0x7f30880ca9e0 2023-11-24T08:10:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.437+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f3080033030 con 0x7f30880ca9e0 2023-11-24T08:10:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.438+0000 7f3084ff9640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 0x7f307406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.438+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30800a9020 con 0x7f30880ca9e0 2023-11-24T08:10:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.438+0000 7f3087fff640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 0x7f307406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.439+0000 7f3087fff640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 0x7f307406db70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3078009a20 tx=0x7f3078008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:35.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.443+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f308815bb70 con 0x7f30880ca9e0 2023-11-24T08:10:35.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.675+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f30880c8cf0 con 0x7f30880ca9e0 2023-11-24T08:10:35.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.675+0000 7f3084ff9640 1 -- 172.21.15.107:0/4228013288 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f30880c8cf0 con 0x7f30880ca9e0 2023-11-24T08:10:35.677 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:10:35.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 msgr2=0x7f307406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 0x7f307406db70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3078009a20 tx=0x7f3078008040 comp rx=0 tx=0).stop 2023-11-24T08:10:35.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 msgr2=0x7f308814c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:35.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f308002ed90 tx=0x7f3080001cb0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 shutdown_connections 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f307406b6b0 0x7f307406db70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30880ca9e0 0x7f308814c070 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30880ca580 0x7f30880c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 --2- 172.21.15.107:0/4228013288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3070005180 0x7f3088158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 >> 172.21.15.107:0/4228013288 conn(0x7f308806a920 msgr2=0x7f3088157d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 shutdown_connections 2023-11-24T08:10:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:35.678+0000 7f308e4bb640 1 -- 172.21.15.107:0/4228013288 wait complete. 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:10:35.699 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:10:35.700 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:10:35.700 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:10:35.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.700 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:10:35.762 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:10:35.762 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:10:35.762 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:10:35.762 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:10:35.763 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42033sec preferred_lft 42033sec 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.764 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:10:35.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.766 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:10:35.856 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:10:35.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.861 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:10:35.911 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:10:35.911 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:10:35.911 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:10:35.912 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:10:35.912 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:10:35.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:35.912 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:10:35.912 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:10:35.912 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:10:35.912 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:10:35.913 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:10:35.913 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:10:36.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:36.104 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:10:36.157 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:10:36.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:36.157 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:10:36.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:10:36.252 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:10:36.309 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:10:36.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:10:36.310 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:10:36.361 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:10:36.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:36.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:36.505 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:36.505 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:10:36.539 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:10:36.539 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:10:36.539 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:10:36.539 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:10:36.539 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:10:36.540 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:10:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:10:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:10:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:10:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:10:36.544 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:10:36.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:36.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:10:36.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:36.624 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:10:36.678 INFO:teuthology.orchestra.run.smithi120.stdout:client6900 2023-11-24T08:10:36.678 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:10:36.678 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:36.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:10:36.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.960+0000 7ff8a810f640 1 Processor -- start 2023-11-24T08:10:36.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.960+0000 7ff8a810f640 1 -- start start 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a810f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8a00cf450 0x7ff8a00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a810f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00cece0 con 0x7ff8a00cdfb0 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a810f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a006d040 con 0x7ff8a015c7f0 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a810f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a00cfe40 con 0x7ff8a00cf450 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a56aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8a00cf450 0x7ff8a00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:36.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a56aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8a00cf450 0x7ff8a00cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55122/0 (socket says 172.21.15.107:55122) 2023-11-24T08:10:36.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.962+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1822573801 learned_addr learned my addr 172.21.15.107:0/1822573801 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:36.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.963+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1822573801 >> v1:172.21.15.120:6789/0 conn(0x7ff8a015c7f0 legacy=0x7ff8a006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:36.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.963+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1822573801 >> v1:172.21.15.107:6789/0 conn(0x7ff8a00cdfb0 legacy=0x7ff8a00ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:36.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.963+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1822573801 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a006d320 con 0x7ff8a00cf450 2023-11-24T08:10:36.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.964+0000 7ff8a56aa640 1 --2- 172.21.15.107:0/1822573801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a00cf450 0x7ff8a00cf900 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff890009870 tx=0x7ff89002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f17224bd3441c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:36.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.964+0000 7ff8a4ea9640 1 -- 172.21.15.107:0/1822573801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890031870 con 0x7ff8a00cf450 2023-11-24T08:10:36.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.964+0000 7ff8a4ea9640 1 -- 172.21.15.107:0/1822573801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8900319d0 con 0x7ff8a00cf450 2023-11-24T08:10:36.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.964+0000 7ff8a4ea9640 1 -- 172.21.15.107:0/1822573801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890031d60 con 0x7ff8a00cf450 2023-11-24T08:10:36.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 -- 172.21.15.107:0/1822573801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a00cf450 msgr2=0x7ff8a00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:36.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1822573801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a00cf450 0x7ff8a00cf900 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff890009870 tx=0x7ff89002b910 comp rx=0 tx=0).stop 2023-11-24T08:10:36.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 -- 172.21.15.107:0/1822573801 shutdown_connections 2023-11-24T08:10:36.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1822573801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a00cf450 0x7ff8a00cf900 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:36.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 -- 172.21.15.107:0/1822573801 >> 172.21.15.107:0/1822573801 conn(0x7ff8a006a920 msgr2=0x7ff8a006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:36.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 -- 172.21.15.107:0/1822573801 shutdown_connections 2023-11-24T08:10:36.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.965+0000 7ff8a810f640 1 -- 172.21.15.107:0/1822573801 wait complete. 2023-11-24T08:10:36.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.967+0000 7ff8a810f640 1 Processor -- start 2023-11-24T08:10:36.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.967+0000 7ff8a810f640 1 -- start start 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8a00cdfb0 0x7ff8a015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a015c7f0 0x7ff8a00ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a01534b0 con 0x7ff8a015c7f0 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00ae470 con 0x7ff8a00cdfb0 2023-11-24T08:10:36.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.968+0000 7ff8a810f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015b920 con 0x7ff8a00cf450 2023-11-24T08:10:36.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.969+0000 7ff8a56aa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:36.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.969+0000 7ff8a5eab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8a00cdfb0 0x7ff8a015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:36.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.969+0000 7ff8a56aa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:50026/0 (socket says 172.21.15.107:50026) 2023-11-24T08:10:36.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.969+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1326330423 learned_addr learned my addr 172.21.15.107:0/1326330423 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:36.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.969+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8a00cdfb0 msgr2=0x7ff8a015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:36.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8a56aa640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8a00cdfb0 0x7ff8a015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:36.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a015c7f0 msgr2=0x7ff8a00ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8a56aa640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a015c7f0 0x7ff8a00ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8a56aa640 1 -- 172.21.15.107:0/1326330423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900094d0 con 0x7ff8a00cf450 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8a56aa640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff890009870 tx=0x7ff8900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89003ad30 con 0x7ff8a00cf450 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.970+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89003ae90 con 0x7ff8a00cf450 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.971+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890044630 con 0x7ff8a00cf450 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.971+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a0152b10 con 0x7ff8a00cf450 2023-11-24T08:10:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.971+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a01530d0 con 0x7ff8a00cf450 2023-11-24T08:10:36.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.972+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a01535f0 con 0x7ff8a00cf450 2023-11-24T08:10:36.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.974+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7ff89004d450 con 0x7ff8a00cf450 2023-11-24T08:10:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.975+0000 7ff8967fc640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 0x7ff88406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.975+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff89002d040 con 0x7ff8a00cf450 2023-11-24T08:10:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.975+0000 7ff8a5eab640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 0x7ff88406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.975+0000 7ff8a5eab640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 0x7ff88406db70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff89c006fd0 tx=0x7ff89c009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:36.988+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff890077690 con 0x7ff8a00cf450 2023-11-24T08:10:37.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.215+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff8a01535f0 con 0x7ff8a00cf450 2023-11-24T08:10:37.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.867+0000 7ff8967fc640 1 -- 172.21.15.107:0/1326330423 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff89007acb0 con 0x7ff8a00cf450 2023-11-24T08:10:37.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.871+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 msgr2=0x7ff88406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:37.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.871+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 0x7ff88406db70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff89c006fd0 tx=0x7ff89c009060 comp rx=0 tx=0).stop 2023-11-24T08:10:37.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.871+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 msgr2=0x7ff8a0144390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:37.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.871+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff890009870 tx=0x7ff8900046c0 comp rx=0 tx=0).stop 2023-11-24T08:10:37.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 shutdown_connections 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff88406b6b0 0x7ff88406db70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8a015c7f0 0x7ff8a00ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8a00cf450 0x7ff8a0144390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 --2- 172.21.15.107:0/1326330423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8a00cdfb0 0x7ff8a015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 >> 172.21.15.107:0/1326330423 conn(0x7ff8a006a920 msgr2=0x7ff8a00af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 shutdown_connections 2023-11-24T08:10:37.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:37.872+0000 7ff8a810f640 1 -- 172.21.15.107:0/1326330423 wait complete. 2023-11-24T08:10:37.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:10:38.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.105+0000 7fa49a71d640 1 Processor -- start 2023-11-24T08:10:38.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.105+0000 7fa49a71d640 1 -- start start 2023-11-24T08:10:38.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.107+0000 7fa49a71d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa49406de10 0x7fa49415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.107+0000 7fa49a71d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.107+0000 7fa49a71d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49415ec90 con 0x7fa494072160 2023-11-24T08:10:38.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.107+0000 7fa49a71d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940cd530 con 0x7fa49406de10 2023-11-24T08:10:38.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.107+0000 7fa49a71d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c4ed0 con 0x7fa4940c49f0 2023-11-24T08:10:38.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa4937fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa49406de10 0x7fa49415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa498cba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa498cba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:58760/0 (socket says 172.21.15.107:58760) 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa498cba640 1 -- 172.21.15.107:0/2205379437 learned_addr learned my addr 172.21.15.107:0/2205379437 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa4937fe640 1 -- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 msgr2=0x7fa4940705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa4937fe640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa4937fe640 1 -- 172.21.15.107:0/2205379437 >> v1:172.21.15.120:6789/0 conn(0x7fa494072160 legacy=0x7fa49406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa4937fe640 1 -- 172.21.15.107:0/2205379437 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940c51b0 con 0x7fa49406de10 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.108+0000 7fa498cba640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.110+0000 7fa4937fe640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49406de10 0x7fa49415e750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa4880097a0 tx=0x7fa48802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512eaf1d5b1c10e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.110+0000 7fa492ffd640 1 -- 172.21.15.107:0/2205379437 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48802d040 con 0x7fa49406de10 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa492ffd640 1 -- 172.21.15.107:0/2205379437 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa488031810 con 0x7fa49406de10 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa492ffd640 1 -- 172.21.15.107:0/2205379437 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488031ba0 con 0x7fa49406de10 2023-11-24T08:10:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 -- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49406de10 msgr2=0x7fa49415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49406de10 0x7fa49415e750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa4880097a0 tx=0x7fa48802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 -- 172.21.15.107:0/2205379437 shutdown_connections 2023-11-24T08:10:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa4940705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 --2- 172.21.15.107:0/2205379437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49406de10 0x7fa49415e750 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 -- 172.21.15.107:0/2205379437 >> 172.21.15.107:0/2205379437 conn(0x7fa4940b0c50 msgr2=0x7fa4940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 -- 172.21.15.107:0/2205379437 shutdown_connections 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.111+0000 7fa49a71d640 1 -- 172.21.15.107:0/2205379437 wait complete. 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.113+0000 7fa49a71d640 1 Processor -- start 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.113+0000 7fa49a71d640 1 -- start start 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494072160 0x7fa49415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa49415aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49415d2f0 con 0x7fa49406de10 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49415c6e0 con 0x7fa494072160 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.114+0000 7fa49a71d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49415c9c0 con 0x7fa4940c49f0 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59202/0 (socket says 172.21.15.107:59202) 2023-11-24T08:10:38.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 -- 172.21.15.107:0/787534720 learned_addr learned my addr 172.21.15.107:0/787534720 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa498cba640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa49415aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa4937fe640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494072160 0x7fa49415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 -- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 msgr2=0x7fa49415aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa49415aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 -- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494072160 msgr2=0x7fa49415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494072160 0x7fa49415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 -- 172.21.15.107:0/787534720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c009b30 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.115+0000 7fa493fff640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa48400c450 tx=0x7fa48400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.116+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4840052d0 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.116+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa484005430 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.116+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488009400 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.116+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484012870 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.116+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49415cf80 con 0x7fa49406de10 2023-11-24T08:10:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.117+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49404e8b0 con 0x7fa49406de10 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.118+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fa4840129d0 con 0x7fa49406de10 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.119+0000 7fa490ff9640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 0x7fa48006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.119+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa4840879f0 con 0x7fa49406de10 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.119+0000 7fa4937fe640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 0x7fa48006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.120+0000 7fa4937fe640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 0x7fa48006db70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.123+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa484054380 con 0x7fa49406de10 2023-11-24T08:10:38.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.336+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4940c5a50 con 0x7fa48006b6b0 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.338+0000 7fa490ff9640 1 -- 172.21.15.107:0/787534720 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa488031b00 con 0x7fa48006b6b0 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:10:38.340 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:10:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.340+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 msgr2=0x7fa48006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.340+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 0x7fa48006db70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).stop 2023-11-24T08:10:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.340+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 msgr2=0x7fa49415a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.340+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa48400c450 tx=0x7fa48400c910 comp rx=0 tx=0).stop 2023-11-24T08:10:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 shutdown_connections 2023-11-24T08:10:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa48006b6b0 0x7fa48006db70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4940c49f0 0x7fa49415aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa494072160 0x7fa49415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 --2- 172.21.15.107:0/787534720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa49406de10 0x7fa49415a470 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 >> 172.21.15.107:0/787534720 conn(0x7fa4940b0c50 msgr2=0x7fa4940c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 shutdown_connections 2023-11-24T08:10:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.341+0000 7fa49a71d640 1 -- 172.21.15.107:0/787534720 wait complete. 2023-11-24T08:10:38.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:10:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.580+0000 7fe53a86b640 1 Processor -- start 2023-11-24T08:10:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.580+0000 7fe53a86b640 1 -- start start 2023-11-24T08:10:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe53a86b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe534140520 0x7fe5340c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe53a86b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534161f60 con 0x7fe534161b10 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe53a86b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53415de00 con 0x7fe5340c65f0 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe53a86b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5340c6a70 con 0x7fe534140520 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe538e08640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe534161b10 0x7fe53406ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38298/0 (socket says 172.21.15.107:38298) 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.582+0000 7fe538e08640 1 -- 172.21.15.107:0/1355150177 learned_addr learned my addr 172.21.15.107:0/1355150177 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe533fff640 1 --2- 172.21.15.107:0/1355150177 >> v2:172.21.15.120:3300/0 conn(0x7fe534140520 0x7fe5340c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541762676 0 0) 0x7fe534161f60 con 0x7fe534161b10 2023-11-24T08:10:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c003520 con 0x7fe534161b10 2023-11-24T08:10:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172750136 0 0) 0x7fe51c003520 con 0x7fe534161b10 2023-11-24T08:10:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 >> v2:172.21.15.120:3300/0 conn(0x7fe534140520 msgr2=0x7fe5340c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 --2- 172.21.15.107:0/1355150177 >> v2:172.21.15.120:3300/0 conn(0x7fe534140520 0x7fe5340c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 >> v1:172.21.15.120:6789/0 conn(0x7fe5340c65f0 legacy=0x7fe534161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53415e0e0 con 0x7fe534161b10 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe52c0019f0 con 0x7fe534161b10 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 >> v1:172.21.15.107:6789/0 conn(0x7fe534161b10 legacy=0x7fe53406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe51c003970 0x7fe51c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe51c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe51c005180 0x7fe51c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.583+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340c6a70 con 0x7fe51c0047c0 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c00c020 con 0x7fe51c005180 2023-11-24T08:10:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c00c1e0 con 0x7fe51c003970 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe51c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe538e08640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe51c005180 0x7fe51c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe533fff640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe51c003970 0x7fe51c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 -- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe51c003970 msgr2=0x7fe51c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe51c003970 0x7fe51c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 -- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe51c005180 msgr2=0x7fe51c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe51c005180 0x7fe51c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe5337fe640 1 -- 172.21.15.107:0/1355150177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c003520 con 0x7fe51c0047c0 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.584+0000 7fe538e08640 1 -- 172.21.15.107:0/1355150177 reap_dead start 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.585+0000 7fe5337fe640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe51c004c40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe528001750 tx=0x7fe51c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971144d20137c134 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280052d0 con 0x7fe51c0047c0 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52802e050 con 0x7fe51c0047c0 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe532ffd640 1 -- 172.21.15.107:0/1355150177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528003830 con 0x7fe51c0047c0 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 -- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 msgr2=0x7fe51c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe51c004c40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe528001750 tx=0x7fe51c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 -- 172.21.15.107:0/1355150177 shutdown_connections 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 --2- 172.21.15.107:0/1355150177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe51c004c40 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 -- 172.21.15.107:0/1355150177 >> 172.21.15.107:0/1355150177 conn(0x7fe5340b2e40 msgr2=0x7fe5340b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 -- 172.21.15.107:0/1355150177 shutdown_connections 2023-11-24T08:10:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.586+0000 7fe53a86b640 1 -- 172.21.15.107:0/1355150177 wait complete. 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.588+0000 7fe53a86b640 1 Processor -- start 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.588+0000 7fe53a86b640 1 -- start start 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe534153510 0x7fe534160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe534160750 0x7fe5340ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340d0470 con 0x7fe51c0047c0 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe534160bd0 con 0x7fe534153510 2023-11-24T08:10:38.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.589+0000 7fe53a86b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340cf7f0 con 0x7fe534160750 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59214/0 (socket says 172.21.15.107:59214) 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 -- 172.21.15.107:0/3760737291 learned_addr learned my addr 172.21.15.107:0/3760737291 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 -- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe534160750 msgr2=0x7fe5340ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe534160750 0x7fe5340ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 -- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe534153510 msgr2=0x7fe534160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe534153510 0x7fe534160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 -- 172.21.15.107:0/3760737291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340cfad0 con 0x7fe51c0047c0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.590+0000 7fe533fff640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe5240114e0 tx=0x7fe5240119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.591+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524015870 con 0x7fe51c0047c0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.591+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240159d0 con 0x7fe51c0047c0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.591+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524015d40 con 0x7fe51c0047c0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.591+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5340cfdb0 con 0x7fe51c0047c0 2023-11-24T08:10:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.591+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe534153990 con 0x7fe51c0047c0 2023-11-24T08:10:38.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.592+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5340d06e0 con 0x7fe51c0047c0 2023-11-24T08:10:38.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.593+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fe524013400 con 0x7fe51c0047c0 2023-11-24T08:10:38.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.594+0000 7fe530ff9640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 0x7fe52006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:38.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.594+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe5240884a0 con 0x7fe51c0047c0 2023-11-24T08:10:38.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.594+0000 7fe5337fe640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 0x7fe52006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:38.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.595+0000 7fe5337fe640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 0x7fe52006db70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe528033000 tx=0x7fe52802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:38.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.598+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe524054e30 con 0x7fe51c0047c0 2023-11-24T08:10:38.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.791+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe53404e8b0 con 0x7fe51c0047c0 2023-11-24T08:10:38.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.792+0000 7fe530ff9640 1 -- 172.21.15.107:0/3760737291 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe5240220c0 con 0x7fe51c0047c0 2023-11-24T08:10:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 msgr2=0x7fe52006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 0x7fe52006db70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe528033000 tx=0x7fe52802f000 comp rx=0 tx=0).stop 2023-11-24T08:10:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 msgr2=0x7fe5340d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:38.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe5240114e0 tx=0x7fe5240119a0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 shutdown_connections 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe52006b6b0 0x7fe52006db70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe534160750 0x7fe5340ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe534153510 0x7fe534160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 --2- 172.21.15.107:0/3760737291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c0047c0 0x7fe5340d0890 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 >> 172.21.15.107:0/3760737291 conn(0x7fe5340b2e40 msgr2=0x7fe5340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.795+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 shutdown_connections 2023-11-24T08:10:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:38.796+0000 7fe53a86b640 1 -- 172.21.15.107:0/3760737291 wait complete. 2023-11-24T08:10:38.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol9055 --mode=777 2023-11-24T08:10:39.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.030+0000 7fc4e5815640 1 Processor -- start 2023-11-24T08:10:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.030+0000 7fc4e5815640 1 -- start start 2023-11-24T08:10:39.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4e5815640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4e015c7f0 0x7fc4e006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4e5815640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00cf450 0x7fc4e00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4e5815640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e00cece0 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4e5815640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e0156010 con 0x7fc4e015c7f0 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4e5815640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e006ce30 con 0x7fc4e00cf450 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.031+0000 7fc4deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4e015c7f0 0x7fc4e006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4e015c7f0 0x7fc4e006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59218/0 (socket says 172.21.15.107:59218) 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 -- 172.21.15.107:0/1951169570 learned_addr learned my addr 172.21.15.107:0/1951169570 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 -- 172.21.15.107:0/1951169570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00cf450 msgr2=0x7fc4e00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 --2- 172.21.15.107:0/1951169570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00cf450 0x7fc4e00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 -- 172.21.15.107:0/1951169570 >> v1:172.21.15.120:6789/0 conn(0x7fc4e00cdfb0 legacy=0x7fc4e00ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.032+0000 7fc4deffd640 1 -- 172.21.15.107:0/1951169570 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e006d040 con 0x7fc4e015c7f0 2023-11-24T08:10:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.033+0000 7fc4deffd640 1 --2- 172.21.15.107:0/1951169570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e015c7f0 0x7fc4e006db20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009e50 tx=0x7fc4d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcdc5d015b5a06a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.033+0000 7fc4ddffb640 1 -- 172.21.15.107:0/1951169570 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d002ca70 con 0x7fc4e015c7f0 2023-11-24T08:10:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.033+0000 7fc4ddffb640 1 -- 172.21.15.107:0/1951169570 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d002cbd0 con 0x7fc4e015c7f0 2023-11-24T08:10:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.033+0000 7fc4ddffb640 1 -- 172.21.15.107:0/1951169570 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00343f0 con 0x7fc4e015c7f0 2023-11-24T08:10:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 -- 172.21.15.107:0/1951169570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e015c7f0 msgr2=0x7fc4e006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 --2- 172.21.15.107:0/1951169570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e015c7f0 0x7fc4e006db20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009e50 tx=0x7fc4d002bed0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 -- 172.21.15.107:0/1951169570 shutdown_connections 2023-11-24T08:10:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 --2- 172.21.15.107:0/1951169570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00cf450 0x7fc4e00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 --2- 172.21.15.107:0/1951169570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e015c7f0 0x7fc4e006db20 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 -- 172.21.15.107:0/1951169570 >> 172.21.15.107:0/1951169570 conn(0x7fc4e006a920 msgr2=0x7fc4e006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 -- 172.21.15.107:0/1951169570 shutdown_connections 2023-11-24T08:10:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.034+0000 7fc4e5815640 1 -- 172.21.15.107:0/1951169570 wait complete. 2023-11-24T08:10:39.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.036+0000 7fc4e5815640 1 Processor -- start 2023-11-24T08:10:39.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.036+0000 7fc4e5815640 1 -- start start 2023-11-24T08:10:39.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e00cf450 0x7fc4e0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e015c7f0 0x7fc4e00adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00b3060 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0157b20 con 0x7fc4e00cf450 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4e5815640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00b24e0 con 0x7fc4e015c7f0 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.037+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59230/0 (socket says 172.21.15.107:59230) 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 -- 172.21.15.107:0/2187254894 learned_addr learned my addr 172.21.15.107:0/2187254894 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 -- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e015c7f0 msgr2=0x7fc4e00adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e015c7f0 0x7fc4e00adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 -- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e00cf450 msgr2=0x7fc4e0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e00cf450 0x7fc4e0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 -- 172.21.15.107:0/2187254894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0009b30 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4deffd640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009e50 tx=0x7fc4d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d003e450 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d003e5d0 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.038+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d003e980 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.039+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e00b27c0 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.039+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e00b2d80 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.040+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e00ac050 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.041+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7fc4d003eae0 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.042+0000 7fc4d7fff640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 0x7fc4c006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.042+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc4d00aaf60 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.042+0000 7fc4de7fc640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 0x7fc4c006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.042+0000 7fc4de7fc640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 0x7fc4c006db70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4e015c530 tx=0x7fc4c8006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.046+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4e00ac050 con 0x7fc4e00cdfb0 2023-11-24T08:10:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.251+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol9055", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4e006e060 con 0x7fc4c006b6b0 2023-11-24T08:10:39.263 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:10:39.261+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:10:39.263 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:10:39.261+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:10:39.263 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:10:39.261+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:10:39.263 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:10:39.261+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:10:39.264 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:10:39.261+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:10:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.297+0000 7fc4d7fff640 1 -- 172.21.15.107:0/2187254894 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc4c8013400 con 0x7fc4c006b6b0 2023-11-24T08:10:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 msgr2=0x7fc4c006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 0x7fc4c006db70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4e015c530 tx=0x7fc4c8006c90 comp rx=0 tx=0).stop 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 msgr2=0x7fc4e01575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009e50 tx=0x7fc4d0004a40 comp rx=0 tx=0).stop 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 shutdown_connections 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc4c006b6b0 0x7fc4c006db70 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e015c7f0 0x7fc4e00adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e00cf450 0x7fc4e0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 --2- 172.21.15.107:0/2187254894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4e00cdfb0 0x7fc4e01575e0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.299+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 >> 172.21.15.107:0/2187254894 conn(0x7fc4e006a920 msgr2=0x7fc4e00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.300+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 shutdown_connections 2023-11-24T08:10:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.300+0000 7fc4e5815640 1 -- 172.21.15.107:0/2187254894 wait complete. 2023-11-24T08:10:39.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol9055 2023-11-24T08:10:39.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.501+0000 7f05f766b640 1 Processor -- start 2023-11-24T08:10:39.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.501+0000 7f05f766b640 1 -- start start 2023-11-24T08:10:39.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 0x7f05f00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f05f00cbf20 0x7f05f00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05f00c57e0 0x7f05f00c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f015b820 con 0x7f05f00cbf20 2023-11-24T08:10:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00cc3a0 con 0x7f05f00c57e0 2023-11-24T08:10:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f766b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f015eea0 con 0x7f05f015b440 2023-11-24T08:10:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f4c06640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f05f00cbf20 0x7f05f00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f4c06640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f05f00cbf20 0x7f05f00c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59268/0 (socket says 172.21.15.107:59268) 2023-11-24T08:10:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.503+0000 7f05f4c06640 1 -- 172.21.15.107:0/1258717557 learned_addr learned my addr 172.21.15.107:0/1258717557 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f5407640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 0x7f05f00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f5c08640 1 --2- 172.21.15.107:0/1258717557 >> v2:172.21.15.120:3300/0 conn(0x7f05f00c57e0 0x7f05f00c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f4c06640 1 -- 172.21.15.107:0/1258717557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 msgr2=0x7f05f00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f4c06640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 0x7f05f00cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f4c06640 1 -- 172.21.15.107:0/1258717557 >> v2:172.21.15.120:3300/0 conn(0x7f05f00c57e0 msgr2=0x7f05f00c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:10:39.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f4c06640 1 --2- 172.21.15.107:0/1258717557 >> v2:172.21.15.120:3300/0 conn(0x7f05f00c57e0 0x7f05f00c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f4c06640 1 -- 172.21.15.107:0/1258717557 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f00abf30 con 0x7f05f00cbf20 2023-11-24T08:10:39.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.504+0000 7f05f5407640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 0x7f05f00cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:10:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.505+0000 7f05f4c06640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f00cbf20 0x7f05f00c52a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f05f00cb910 tx=0x7f05e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915515990ca7e9d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.505+0000 7f05e7fff640 1 -- 172.21.15.107:0/1258717557 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e00317e0 con 0x7f05f00cbf20 2023-11-24T08:10:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.505+0000 7f05e7fff640 1 -- 172.21.15.107:0/1258717557 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0031940 con 0x7f05f00cbf20 2023-11-24T08:10:39.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.505+0000 7f05e7fff640 1 -- 172.21.15.107:0/1258717557 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0031cd0 con 0x7f05f00cbf20 2023-11-24T08:10:39.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 -- 172.21.15.107:0/1258717557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f00cbf20 msgr2=0x7f05f00c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f00cbf20 0x7f05f00c52a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f05f00cb910 tx=0x7f05e002b880 comp rx=0 tx=0).stop 2023-11-24T08:10:39.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 -- 172.21.15.107:0/1258717557 shutdown_connections 2023-11-24T08:10:39.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 --2- 172.21.15.107:0/1258717557 >> v2:172.21.15.120:3300/0 conn(0x7f05f00c57e0 0x7f05f00c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f00cbf20 0x7f05f00c52a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 --2- 172.21.15.107:0/1258717557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f015b440 0x7f05f00cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 -- 172.21.15.107:0/1258717557 >> 172.21.15.107:0/1258717557 conn(0x7f05f00b0c50 msgr2=0x7f05f00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:39.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 -- 172.21.15.107:0/1258717557 shutdown_connections 2023-11-24T08:10:39.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.506+0000 7f05f766b640 1 -- 172.21.15.107:0/1258717557 wait complete. 2023-11-24T08:10:39.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.508+0000 7f05f766b640 1 Processor -- start 2023-11-24T08:10:39.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.508+0000 7f05f766b640 1 -- start start 2023-11-24T08:10:39.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 0x7f05f009b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f00cbf20 0x7f05f0155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0158db0 con 0x7f05f015b440 2023-11-24T08:10:39.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f009beb0 con 0x7f05f00c57e0 2023-11-24T08:10:39.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.509+0000 7f05f766b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0158130 con 0x7f05f00cbf20 2023-11-24T08:10:39.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59276/0 (socket says 172.21.15.107:59276) 2023-11-24T08:10:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 -- 172.21.15.107:0/3025981000 learned_addr learned my addr 172.21.15.107:0/3025981000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:10:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f4c06640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f00cbf20 0x7f05f0155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5407640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 0x7f05f009b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 -- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f00cbf20 msgr2=0x7f05f0155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f00cbf20 0x7f05f0155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 -- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 msgr2=0x7f05f009b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 0x7f05f009b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 -- 172.21.15.107:0/3025981000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e00094d0 con 0x7f05f015b440 2023-11-24T08:10:39.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5407640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 0x7f05f009b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:10:39.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.510+0000 7f05f5c08640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f05e8007e60 tx=0x7f05e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.511+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8014620 con 0x7f05f015b440 2023-11-24T08:10:39.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.511+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e800ee90 con 0x7f05f015b440 2023-11-24T08:10:39.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.511+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80149e0 con 0x7f05f015b440 2023-11-24T08:10:39.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.511+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f0158410 con 0x7f05f015b440 2023-11-24T08:10:39.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.511+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01589d0 con 0x7f05f015b440 2023-11-24T08:10:39.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.512+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f00cfe50 con 0x7f05f015b440 2023-11-24T08:10:39.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.513+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 89996+0+0 (secure 0 0 0) 0x7f05e8014b40 con 0x7f05f015b440 2023-11-24T08:10:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.514+0000 7f05e5ffb640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 0x7f05d406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:10:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.514+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05e80889f0 con 0x7f05f015b440 2023-11-24T08:10:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.514+0000 7f05f5407640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 0x7f05d406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:10:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.515+0000 7f05f5407640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 0x7f05d406db70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f05f009cc90 tx=0x7f05ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:10:39.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.518+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05e8055320 con 0x7f05f015b440 2023-11-24T08:10:39.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.734+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol9055", "target": ["mon-mgr", ""]}) v1 -- 0x7f05f015f6b0 con 0x7f05d406b6b0 2023-11-24T08:10:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.741+0000 7f05e5ffb640 1 -- 172.21.15.107:0/3025981000 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f05ec00a210 con 0x7f05d406b6b0 2023-11-24T08:10:39.744 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86 2023-11-24T08:10:39.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 msgr2=0x7f05d406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 0x7f05d406db70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f05f009cc90 tx=0x7f05ec008040 comp rx=0 tx=0).stop 2023-11-24T08:10:39.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 msgr2=0x7f05f00afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:10:39.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f05e8007e60 tx=0x7f05e800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 shutdown_connections 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f05d406b6b0 0x7f05d406db70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f015b440 0x7f05f00afa30 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f00cbf20 0x7f05f0155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 --2- 172.21.15.107:0/3025981000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f00c57e0 0x7f05f009b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 >> 172.21.15.107:0/3025981000 conn(0x7f05f00b0c50 msgr2=0x7f05f0071a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 shutdown_connections 2023-11-24T08:10:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:10:39.744+0000 7f05f766b640 1 -- 172.21.15.107:0/3025981000 wait complete. 2023-11-24T08:10:39.765 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol9055 with 200 files each 1MB size under directory volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86 2023-11-24T08:10:39.765 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:39.765 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:39.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:39.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:39.950 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:39.950 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:39.950 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294816 s, 35.6 MB/s 2023-11-24T08:10:39.952 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:39.952 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.005 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244406 s, 42.9 MB/s 2023-11-24T08:10:40.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.078 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.134 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234887 s, 44.6 MB/s 2023-11-24T08:10:40.214 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.214 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218609 s, 48.0 MB/s 2023-11-24T08:10:40.347 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.348 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.404 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.481 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.481 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.482 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234361 s, 44.7 MB/s 2023-11-24T08:10:40.483 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.483 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.540 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:10:40.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:10:40.620 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.621 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.621 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027012 s, 38.8 MB/s 2023-11-24T08:10:40.622 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.622 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.709 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.850 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209475 s, 50.1 MB/s 2023-11-24T08:10:40.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.852 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:40.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:40.906 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:40.978 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:40.978 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242201 s, 43.3 MB/s 2023-11-24T08:10:40.983 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:40.983 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.036 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275814 s, 38.0 MB/s 2023-11-24T08:10:41.117 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.117 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.173 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.259 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272856 s, 38.4 MB/s 2023-11-24T08:10:41.260 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.260 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219446 s, 47.8 MB/s 2023-11-24T08:10:41.395 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.395 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.453 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.522 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163019 s, 64.3 MB/s 2023-11-24T08:10:41.523 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.523 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.579 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.651 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.652 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.652 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203659 s, 51.5 MB/s 2023-11-24T08:10:41.653 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.653 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.780 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.781 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.781 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193336 s, 54.2 MB/s 2023-11-24T08:10:41.782 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.782 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.838 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:41.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:41.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:41.912 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193276 s, 54.3 MB/s 2023-11-24T08:10:41.913 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:41.913 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:41.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:41.963 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.033 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.033 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.033 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232743 s, 45.1 MB/s 2023-11-24T08:10:42.035 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.035 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.088 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219394 s, 47.8 MB/s 2023-11-24T08:10:42.213 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.213 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.265 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.337 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.337 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.337 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219007 s, 47.9 MB/s 2023-11-24T08:10:42.339 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.339 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.394 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.475 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.475 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.475 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237007 s, 44.2 MB/s 2023-11-24T08:10:42.477 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.477 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.535 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252452 s, 41.5 MB/s 2023-11-24T08:10:42.616 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.616 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.669 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244785 s, 42.8 MB/s 2023-11-24T08:10:42.748 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.748 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.800 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228345 s, 45.9 MB/s 2023-11-24T08:10:42.924 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:42.925 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:42.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:42.979 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028174 s, 37.2 MB/s 2023-11-24T08:10:43.062 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.062 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.118 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261156 s, 40.2 MB/s 2023-11-24T08:10:43.199 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.199 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.258 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233109 s, 45.0 MB/s 2023-11-24T08:10:43.337 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.337 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.394 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020231 s, 51.8 MB/s 2023-11-24T08:10:43.470 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.471 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.598 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.599 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194214 s, 54.0 MB/s 2023-11-24T08:10:43.600 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.600 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210661 s, 49.8 MB/s 2023-11-24T08:10:43.777 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.777 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.836 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:43.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:43.912 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:43.912 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202291 s, 51.8 MB/s 2023-11-24T08:10:43.913 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:43.913 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:43.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:43.968 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.049 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260374 s, 40.3 MB/s 2023-11-24T08:10:44.051 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.051 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.109 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.189 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.190 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.190 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233297 s, 44.9 MB/s 2023-11-24T08:10:44.191 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.191 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.273 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.361 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.361 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.361 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287544 s, 36.5 MB/s 2023-11-24T08:10:44.363 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.363 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.415 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.488 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.488 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.488 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208103 s, 50.4 MB/s 2023-11-24T08:10:44.489 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.489 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.625 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264138 s, 39.7 MB/s 2023-11-24T08:10:44.627 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.627 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202907 s, 51.7 MB/s 2023-11-24T08:10:44.755 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.755 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.808 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:44.932 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:44.932 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021452 s, 48.9 MB/s 2023-11-24T08:10:44.933 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:44.933 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:44.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:44.989 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.071 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.071 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.071 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273436 s, 38.3 MB/s 2023-11-24T08:10:45.073 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.073 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.200 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.200 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.200 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208623 s, 50.3 MB/s 2023-11-24T08:10:45.202 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.202 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.259 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205889 s, 50.9 MB/s 2023-11-24T08:10:45.329 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.329 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242528 s, 43.2 MB/s 2023-11-24T08:10:45.467 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.467 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.603 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.603 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.603 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199738 s, 52.5 MB/s 2023-11-24T08:10:45.605 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.605 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.658 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280038 s, 37.4 MB/s 2023-11-24T08:10:45.787 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.788 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:45.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:45.835 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220415 s, 47.6 MB/s 2023-11-24T08:10:45.957 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:45.957 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.011 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208365 s, 50.3 MB/s 2023-11-24T08:10:46.088 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.088 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.146 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239699 s, 43.7 MB/s 2023-11-24T08:10:46.225 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.225 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.357 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208209 s, 50.4 MB/s 2023-11-24T08:10:46.358 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.359 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026262 s, 39.9 MB/s 2023-11-24T08:10:46.494 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.495 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.553 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202575 s, 51.8 MB/s 2023-11-24T08:10:46.634 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.634 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.691 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.762 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.763 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.763 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194478 s, 53.9 MB/s 2023-11-24T08:10:46.764 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.764 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223088 s, 47.0 MB/s 2023-11-24T08:10:46.942 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:46.942 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:46.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:46.994 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.124 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.124 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.124 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272506 s, 38.5 MB/s 2023-11-24T08:10:47.126 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.126 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.185 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232764 s, 45.0 MB/s 2023-11-24T08:10:47.264 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.264 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.397 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208708 s, 50.2 MB/s 2023-11-24T08:10:47.399 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.399 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.456 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221075 s, 47.4 MB/s 2023-11-24T08:10:47.533 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.533 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.587 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.666 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234738 s, 44.7 MB/s 2023-11-24T08:10:47.668 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.669 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.725 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.808 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.809 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.809 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252008 s, 41.6 MB/s 2023-11-24T08:10:47.810 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.811 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:47.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:47.867 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207031 s, 50.6 MB/s 2023-11-24T08:10:47.943 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:47.944 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256255 s, 40.9 MB/s 2023-11-24T08:10:48.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.082 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222145 s, 47.2 MB/s 2023-11-24T08:10:48.219 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.219 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.357 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263386 s, 39.8 MB/s 2023-11-24T08:10:48.358 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.359 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.487 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.487 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166938 s, 62.8 MB/s 2023-11-24T08:10:48.488 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.488 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.539 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016441 s, 63.8 MB/s 2023-11-24T08:10:48.608 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.608 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.739 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.740 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.740 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232178 s, 45.2 MB/s 2023-11-24T08:10:48.741 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.741 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.797 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:48.873 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:48.874 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:48.874 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020033 s, 52.3 MB/s 2023-11-24T08:10:48.875 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:48.875 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:48.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:48.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.64 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230811 s, 45.4 MB/s 2023-11-24T08:10:49.006 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.006 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.065 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.65 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246281 s, 42.6 MB/s 2023-11-24T08:10:49.145 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.145 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.199 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.66 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.274 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021375 s, 49.1 MB/s 2023-11-24T08:10:49.275 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.276 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.331 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.67 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226939 s, 46.2 MB/s 2023-11-24T08:10:49.408 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.408 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.466 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.68 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.546 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.546 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237312 s, 44.2 MB/s 2023-11-24T08:10:49.547 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.547 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.69 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.681 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.682 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.682 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202408 s, 51.8 MB/s 2023-11-24T08:10:49.683 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.683 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.736 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.70 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.864 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.864 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.864 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264637 s, 39.6 MB/s 2023-11-24T08:10:49.866 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.866 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:49.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:49.924 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.71 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:49.994 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:49.994 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:49.994 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174069 s, 60.2 MB/s 2023-11-24T08:10:49.995 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:49.996 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.046 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.72 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295262 s, 35.5 MB/s 2023-11-24T08:10:50.179 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.180 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.73 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.316 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.316 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.317 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247472 s, 42.4 MB/s 2023-11-24T08:10:50.318 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.319 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.378 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.74 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242848 s, 43.2 MB/s 2023-11-24T08:10:50.460 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.460 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.520 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.75 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.593 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175677 s, 59.7 MB/s 2023-11-24T08:10:50.594 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.594 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.653 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.76 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206834 s, 50.7 MB/s 2023-11-24T08:10:50.726 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.726 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.77 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:50.843 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:50.843 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:50.844 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176483 s, 59.4 MB/s 2023-11-24T08:10:50.845 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:50.845 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:50.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:50.891 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.78 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.012 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212404 s, 49.4 MB/s 2023-11-24T08:10:51.014 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.014 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.79 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.141 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175362 s, 59.8 MB/s 2023-11-24T08:10:51.143 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.144 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.202 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.80 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.275 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.276 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.276 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200253 s, 52.4 MB/s 2023-11-24T08:10:51.277 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.277 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.332 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.81 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185244 s, 56.6 MB/s 2023-11-24T08:10:51.402 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.402 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.457 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.82 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.533 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.533 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224531 s, 46.7 MB/s 2023-11-24T08:10:51.534 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.535 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.591 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.83 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.669 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02327 s, 45.1 MB/s 2023-11-24T08:10:51.670 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.670 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.728 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.84 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.803 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.803 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.804 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196714 s, 53.3 MB/s 2023-11-24T08:10:51.805 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.806 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.85 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022364 s, 46.9 MB/s 2023-11-24T08:10:51.940 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:51.940 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:51.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:51.993 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.86 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245805 s, 42.7 MB/s 2023-11-24T08:10:52.121 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.121 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.87 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.256 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.257 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.257 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021453 s, 48.9 MB/s 2023-11-24T08:10:52.258 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.258 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.314 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.88 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.387 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.387 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.387 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216456 s, 48.4 MB/s 2023-11-24T08:10:52.389 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.389 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.89 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220333 s, 47.6 MB/s 2023-11-24T08:10:52.520 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.521 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.574 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.90 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221849 s, 47.3 MB/s 2023-11-24T08:10:52.649 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.649 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.704 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.91 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131004 s, 80.0 MB/s 2023-11-24T08:10:52.767 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.767 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.92 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:52.935 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:52.935 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:52.935 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170714 s, 61.4 MB/s 2023-11-24T08:10:52.937 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:52.937 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:52.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:52.986 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.93 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.110 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250333 s, 41.9 MB/s 2023-11-24T08:10:53.113 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.113 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.172 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.94 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.249 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.249 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.250 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237826 s, 44.1 MB/s 2023-11-24T08:10:53.251 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.251 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.310 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.95 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020628 s, 50.8 MB/s 2023-11-24T08:10:53.388 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.388 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.445 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.96 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.516 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.516 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.516 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182953 s, 57.3 MB/s 2023-11-24T08:10:53.518 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.518 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.573 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.97 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214737 s, 48.8 MB/s 2023-11-24T08:10:53.649 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.650 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.701 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.98 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.769 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.769 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.770 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207837 s, 50.5 MB/s 2023-11-24T08:10:53.771 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.771 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.99 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:53.898 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:53.898 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:53.899 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186356 s, 56.3 MB/s 2023-11-24T08:10:53.899 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:53.900 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:53.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:53.953 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.100 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262446 s, 40.0 MB/s 2023-11-24T08:10:54.035 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.035 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.092 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.101 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.168 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.168 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223133 s, 47.0 MB/s 2023-11-24T08:10:54.169 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.169 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.102 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197133 s, 53.2 MB/s 2023-11-24T08:10:54.301 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.301 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.358 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.103 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216541 s, 48.4 MB/s 2023-11-24T08:10:54.435 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.435 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.493 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.104 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.566 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.566 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.566 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205054 s, 51.1 MB/s 2023-11-24T08:10:54.567 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.567 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.624 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.105 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.704 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.704 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.704 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259295 s, 40.4 MB/s 2023-11-24T08:10:54.705 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.706 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.764 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.106 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.843 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252345 s, 41.6 MB/s 2023-11-24T08:10:54.844 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.844 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:54.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:54.896 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.107 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:54.969 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:54.970 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:54.970 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214808 s, 48.8 MB/s 2023-11-24T08:10:54.971 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:54.971 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.025 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.108 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.096 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.096 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.096 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183991 s, 57.0 MB/s 2023-11-24T08:10:55.098 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.098 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.109 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.234 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.234 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.234 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025078 s, 41.8 MB/s 2023-11-24T08:10:55.236 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.236 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.294 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.110 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.374 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.374 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.374 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025914 s, 40.5 MB/s 2023-11-24T08:10:55.376 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.376 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.434 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.111 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.508 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200911 s, 52.2 MB/s 2023-11-24T08:10:55.509 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.510 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.567 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.112 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230115 s, 45.6 MB/s 2023-11-24T08:10:55.643 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.643 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.113 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177812 s, 59.0 MB/s 2023-11-24T08:10:55.774 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.774 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:55.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:55.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.114 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226914 s, 46.2 MB/s 2023-11-24T08:10:55.946 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:55.946 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.115 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.080 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223303 s, 47.0 MB/s 2023-11-24T08:10:56.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.082 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.137 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.116 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.210 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.210 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.210 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193176 s, 54.3 MB/s 2023-11-24T08:10:56.211 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.212 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.117 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252789 s, 41.5 MB/s 2023-11-24T08:10:56.348 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.348 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.407 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.118 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168112 s, 62.4 MB/s 2023-11-24T08:10:56.479 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.479 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.533 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.119 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.604 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.605 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.605 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192765 s, 54.4 MB/s 2023-11-24T08:10:56.606 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.606 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.658 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.120 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.733 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219806 s, 47.7 MB/s 2023-11-24T08:10:56.734 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.734 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.792 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.121 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.869 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.869 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.869 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237757 s, 44.1 MB/s 2023-11-24T08:10:56.871 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.871 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:56.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:56.925 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.122 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:56.996 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:56.997 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:56.997 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187699 s, 55.9 MB/s 2023-11-24T08:10:56.998 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:56.998 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.123 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241177 s, 43.5 MB/s 2023-11-24T08:10:57.134 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.134 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.186 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.124 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233789 s, 44.9 MB/s 2023-11-24T08:10:57.267 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.267 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.125 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.392 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.392 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.392 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202157 s, 51.9 MB/s 2023-11-24T08:10:57.394 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.394 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.126 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227591 s, 46.1 MB/s 2023-11-24T08:10:57.524 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.524 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.127 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243754 s, 43.0 MB/s 2023-11-24T08:10:57.661 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.661 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.718 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.128 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.790 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.790 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.790 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174087 s, 60.2 MB/s 2023-11-24T08:10:57.791 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.791 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.844 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.129 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:57.916 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:57.916 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:57.916 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200589 s, 52.3 MB/s 2023-11-24T08:10:57.917 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:57.917 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:57.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:57.974 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.130 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253417 s, 41.4 MB/s 2023-11-24T08:10:58.056 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.057 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.114 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.131 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.194 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.194 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.194 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253409 s, 41.4 MB/s 2023-11-24T08:10:58.196 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.196 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.254 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.132 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169117 s, 62.0 MB/s 2023-11-24T08:10:58.322 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.323 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.373 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.133 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233404 s, 44.9 MB/s 2023-11-24T08:10:58.499 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.499 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.557 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.134 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.631 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.632 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.632 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209159 s, 50.1 MB/s 2023-11-24T08:10:58.634 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.634 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.690 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.135 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.767 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.767 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.767 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234699 s, 44.7 MB/s 2023-11-24T08:10:58.769 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.769 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.827 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.136 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255529 s, 41.0 MB/s 2023-11-24T08:10:58.908 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:58.908 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:58.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:58.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.137 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.042 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225765 s, 46.4 MB/s 2023-11-24T08:10:59.043 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.044 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.138 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.162 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.162 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.162 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150409 s, 69.7 MB/s 2023-11-24T08:10:59.164 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.164 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.217 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.139 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.340 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.341 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.341 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022653 s, 46.3 MB/s 2023-11-24T08:10:59.342 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.342 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.396 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.140 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.469 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.469 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.469 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227331 s, 46.1 MB/s 2023-11-24T08:10:59.470 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.471 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.529 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.141 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230183 s, 45.6 MB/s 2023-11-24T08:10:59.608 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.608 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.663 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.142 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.738 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.738 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.738 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224323 s, 46.7 MB/s 2023-11-24T08:10:59.740 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.740 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.797 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.143 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:10:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:10:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:10:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186281 s, 56.3 MB/s 2023-11-24T08:10:59.871 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:10:59.871 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:10:59.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:10:59.927 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.144 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230356 s, 45.5 MB/s 2023-11-24T08:11:00.006 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.006 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.065 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.145 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.147 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.147 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265035 s, 39.6 MB/s 2023-11-24T08:11:00.148 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.149 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.146 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.281 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.282 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.282 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235038 s, 44.6 MB/s 2023-11-24T08:11:00.284 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.284 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.147 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256127 s, 40.9 MB/s 2023-11-24T08:11:00.417 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.418 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.148 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.588 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.588 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.589 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174841 s, 60.0 MB/s 2023-11-24T08:11:00.590 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.590 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.647 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.149 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.724 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.724 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.724 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233858 s, 44.8 MB/s 2023-11-24T08:11:00.725 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.726 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.150 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.853 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.854 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.854 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194985 s, 53.8 MB/s 2023-11-24T08:11:00.855 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.855 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:00.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:00.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.151 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:00.973 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:00.973 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:00.973 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168469 s, 62.2 MB/s 2023-11-24T08:11:00.974 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:00.974 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.023 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.152 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231542 s, 45.3 MB/s 2023-11-24T08:11:01.148 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.148 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.203 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.153 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245064 s, 42.8 MB/s 2023-11-24T08:11:01.281 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.281 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.154 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.412 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.413 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206015 s, 50.9 MB/s 2023-11-24T08:11:01.414 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.414 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.155 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.542 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198436 s, 52.8 MB/s 2023-11-24T08:11:01.543 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.544 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.156 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237616 s, 44.1 MB/s 2023-11-24T08:11:01.673 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.674 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.157 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200469 s, 52.3 MB/s 2023-11-24T08:11:01.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.806 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:01.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:01.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.158 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:01.974 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:01.974 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:01.974 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206625 s, 50.7 MB/s 2023-11-24T08:11:01.975 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:01.975 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.031 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.159 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.108 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.109 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.109 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236602 s, 44.3 MB/s 2023-11-24T08:11:02.110 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.110 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.167 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.160 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.246 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.246 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.246 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250269 s, 41.9 MB/s 2023-11-24T08:11:02.248 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.248 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.308 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.161 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.385 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.386 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.386 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211289 s, 49.6 MB/s 2023-11-24T08:11:02.387 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.387 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.162 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205073 s, 51.1 MB/s 2023-11-24T08:11:02.520 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.520 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.575 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.163 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.645 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.645 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.645 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177681 s, 59.0 MB/s 2023-11-24T08:11:02.647 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.647 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.698 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.164 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.821 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.821 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.821 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204971 s, 51.2 MB/s 2023-11-24T08:11:02.823 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.823 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:02.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:02.882 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.165 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:02.954 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:02.955 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:02.955 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198972 s, 52.7 MB/s 2023-11-24T08:11:02.956 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:02.956 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.012 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.166 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.092 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.092 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.092 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251813 s, 41.6 MB/s 2023-11-24T08:11:03.093 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.094 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.167 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.235 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.235 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249444 s, 42.0 MB/s 2023-11-24T08:11:03.236 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.236 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.168 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217141 s, 48.3 MB/s 2023-11-24T08:11:03.371 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.371 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.428 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.169 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.507 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219812 s, 47.7 MB/s 2023-11-24T08:11:03.509 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.509 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.564 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.170 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175572 s, 59.7 MB/s 2023-11-24T08:11:03.635 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.635 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.171 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237563 s, 44.1 MB/s 2023-11-24T08:11:03.766 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.766 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.818 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.172 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183127 s, 57.3 MB/s 2023-11-24T08:11:03.939 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:03.939 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:03.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:03.995 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.173 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.077 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.077 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263653 s, 39.8 MB/s 2023-11-24T08:11:04.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.078 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.136 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.174 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212617 s, 49.3 MB/s 2023-11-24T08:11:04.214 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.214 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.272 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.175 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.350 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222425 s, 47.1 MB/s 2023-11-24T08:11:04.351 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.352 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.411 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.176 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244792 s, 42.8 MB/s 2023-11-24T08:11:04.492 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.492 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.177 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222905 s, 47.0 MB/s 2023-11-24T08:11:04.627 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.627 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.178 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.762 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.762 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.762 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221579 s, 47.3 MB/s 2023-11-24T08:11:04.764 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.764 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.821 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.179 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:04.896 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:04.897 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:04.897 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204841 s, 51.2 MB/s 2023-11-24T08:11:04.898 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:04.899 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:04.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:04.956 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.180 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226081 s, 46.4 MB/s 2023-11-24T08:11:05.032 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.033 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.087 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.181 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.167 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249944 s, 42.0 MB/s 2023-11-24T08:11:05.168 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.168 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.225 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.182 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137144 s, 76.5 MB/s 2023-11-24T08:11:05.293 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.293 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.343 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.183 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248485 s, 42.2 MB/s 2023-11-24T08:11:05.470 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.470 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.184 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197383 s, 53.1 MB/s 2023-11-24T08:11:05.599 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.599 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.185 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.729 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.729 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.729 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230572 s, 45.5 MB/s 2023-11-24T08:11:05.730 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.731 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.789 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.186 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.857 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233264 s, 45.0 MB/s 2023-11-24T08:11:05.858 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.858 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:05.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:05.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.187 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:05.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:05.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219048 s, 47.9 MB/s 2023-11-24T08:11:05.987 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:05.987 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.188 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.122 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244827 s, 42.8 MB/s 2023-11-24T08:11:06.124 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.124 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.189 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.257 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.258 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.258 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235602 s, 44.5 MB/s 2023-11-24T08:11:06.259 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.260 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.190 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.391 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.391 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.391 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189633 s, 55.3 MB/s 2023-11-24T08:11:06.392 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.393 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.191 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231728 s, 45.3 MB/s 2023-11-24T08:11:06.567 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.567 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.624 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.192 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186019 s, 56.4 MB/s 2023-11-24T08:11:06.697 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.697 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.754 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.193 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251707 s, 41.7 MB/s 2023-11-24T08:11:06.831 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:06.831 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:06.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:06.883 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.194 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237543 s, 44.1 MB/s 2023-11-24T08:11:07.008 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:07.008 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:07.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:07.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.195 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215708 s, 48.6 MB/s 2023-11-24T08:11:07.144 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:07.145 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:07.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:07.203 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.196 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232041 s, 45.2 MB/s 2023-11-24T08:11:07.283 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:07.283 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:07.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:07.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.197 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.413 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.413 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198644 s, 52.8 MB/s 2023-11-24T08:11:07.415 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:07.415 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:07.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:07.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.198 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176371 s, 59.5 MB/s 2023-11-24T08:11:07.546 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:07.546 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:07.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:07.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol9055/b87f05c1-2b31-4f72-8998-da8971b4ba86/subvolume_file.199 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:11:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:11:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:11:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229728 s, 45.6 MB/s 2023-11-24T08:11:07.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol9055 subvol_snap1401 2023-11-24T08:11:07.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.897+0000 7fc1ac953640 1 Processor -- start 2023-11-24T08:11:07.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.897+0000 7fc1ac953640 1 -- start start 2023-11-24T08:11:07.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1ac953640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 0x7fc1a80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1ac953640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a80cece0 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1ac953640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a806d040 con 0x7fc1a815c7f0 2023-11-24T08:11:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1ac953640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8156010 con 0x7fc1a80cf450 2023-11-24T08:11:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1a6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1a80cdfb0 0x7fc1a80ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49558/0 (socket says 172.21.15.107:49558) 2023-11-24T08:11:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.899+0000 7fc1a6d9d640 1 -- 172.21.15.107:0/1361335141 learned_addr learned my addr 172.21.15.107:0/1361335141 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc19ffff640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 0x7fc1a80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231238616 0 0) 0x7fc1a80cece0 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc194003520 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493171516 0 0) 0x7fc194003520 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 msgr2=0x7fc1a80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 0x7fc1a80cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 >> v1:172.21.15.120:6789/0 conn(0x7fc1a815c7f0 legacy=0x7fc1a806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a806d320 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc1a00019f0 con 0x7fc1a80cdfb0 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 >> v1:172.21.15.107:6789/0 conn(0x7fc1a80cdfb0 legacy=0x7fc1a80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1940039a0 0x7fc194003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc194004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.900+0000 7fc1a5d9b640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 0x7fc194005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8156010 con 0x7fc1940047c0 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19400ba90 con 0x7fc1940039a0 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19400bc50 con 0x7fc1a80cf450 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc194004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 -- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 msgr2=0x7fc194005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a80cf450 0x7fc194005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 -- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1940039a0 msgr2=0x7fc194003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1940039a0 0x7fc194003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.901+0000 7fc19ffff640 1 -- 172.21.15.107:0/1361335141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194003520 con 0x7fc1940047c0 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.902+0000 7fc19ffff640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc194004c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc190009a90 tx=0x7fc1940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed950bc5f239659 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19002c6b0 con 0x7fc1940047c0 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19002c810 con 0x7fc1940047c0 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1a5d9b640 1 -- 172.21.15.107:0/1361335141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19002cba0 con 0x7fc1940047c0 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1ac953640 1 -- 172.21.15.107:0/1361335141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 msgr2=0x7fc194004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:07.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1ac953640 1 --2- 172.21.15.107:0/1361335141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc194004c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc190009a90 tx=0x7fc1940057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1a6d9d640 1 -- 172.21.15.107:0/1361335141 reap_dead start 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1ac953640 1 -- 172.21.15.107:0/1361335141 shutdown_connections 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.903+0000 7fc1ac953640 1 -- 172.21.15.107:0/1361335141 >> 172.21.15.107:0/1361335141 conn(0x7fc1a806a920 msgr2=0x7fc1a806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.904+0000 7fc1ac953640 1 -- 172.21.15.107:0/1361335141 shutdown_connections 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.904+0000 7fc1ac953640 1 -- 172.21.15.107:0/1361335141 wait complete. 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.905+0000 7fc1ac953640 1 Processor -- start 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.905+0000 7fc1ac953640 1 -- start start 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a81561c0 0x7fc1a81f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a8141410 0x7fc1a8141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8157520 con 0x7fc1940047c0 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8156640 con 0x7fc1a8141410 2023-11-24T08:11:07.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.906+0000 7fc1ac953640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8159040 con 0x7fc1a81561c0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33912/0 (socket says 172.21.15.107:33912) 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 -- 172.21.15.107:0/3873919935 learned_addr learned my addr 172.21.15.107:0/3873919935 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 -- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a81561c0 msgr2=0x7fc1a81f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a81561c0 0x7fc1a81f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 -- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a8141410 msgr2=0x7fc1a8141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a8141410 0x7fc1a8141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 -- 172.21.15.107:0/3873919935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8158d60 con 0x7fc1940047c0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.907+0000 7fc1a659c640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c460 tx=0x7fc19800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.908+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980052d0 con 0x7fc1940047c0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.908+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198005470 con 0x7fc1940047c0 2023-11-24T08:11:07.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.908+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198007260 con 0x7fc1940047c0 2023-11-24T08:11:07.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.908+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a80acae0 con 0x7fc1940047c0 2023-11-24T08:11:07.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.908+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a80ad0a0 con 0x7fc1940047c0 2023-11-24T08:11:07.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.909+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a8067470 con 0x7fc1940047c0 2023-11-24T08:11:07.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.910+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 90047+0+0 (secure 0 0 0) 0x7fc1980073c0 con 0x7fc1940047c0 2023-11-24T08:11:07.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.911+0000 7fc19effd640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 0x7fc18806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:07.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.911+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc198087720 con 0x7fc1940047c0 2023-11-24T08:11:07.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.911+0000 7fc19ffff640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 0x7fc18806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:07.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.912+0000 7fc19ffff640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 0x7fc18806dc00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc190036000 tx=0x7fc19002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:07.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:07.916+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc198054220 con 0x7fc1940047c0 2023-11-24T08:11:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.136+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol9055", "snap_name": "subvol_snap1401", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1a80cd730 con 0x7fc18806b740 2023-11-24T08:11:08.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.182+0000 7fc19effd640 1 -- 172.21.15.107:0/3873919935 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc1a80cd730 con 0x7fc18806b740 2023-11-24T08:11:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 msgr2=0x7fc18806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 0x7fc18806dc00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc190036000 tx=0x7fc19002e000 comp rx=0 tx=0).stop 2023-11-24T08:11:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 msgr2=0x7fc1a80ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c460 tx=0x7fc19800c920 comp rx=0 tx=0).stop 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 shutdown_connections 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc18806b740 0x7fc18806dc00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a8141410 0x7fc1a8141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a81561c0 0x7fc1a81f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 --2- 172.21.15.107:0/3873919935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1940047c0 0x7fc1a80ad5c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 >> 172.21.15.107:0/3873919935 conn(0x7fc1a806a920 msgr2=0x7fc1a815df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 shutdown_connections 2023-11-24T08:11:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.186+0000 7fc1ac953640 1 -- 172.21.15.107:0/3873919935 wait complete. 2023-11-24T08:11:08.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 5 2023-11-24T08:11:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.415+0000 7f07ddfcb640 1 Processor -- start 2023-11-24T08:11:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.415+0000 7f07ddfcb640 1 -- start start 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.416+0000 7f07ddfcb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07d80b0b40 0x7f07d80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07ddfcb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07d815ded0 0x7f07d815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07ddfcb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d806d430 con 0x7f07d815a1c0 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07ddfcb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d815ec50 con 0x7f07d815ded0 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07ddfcb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d806cb00 con 0x7f07d80b0b40 2023-11-24T08:11:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dcfc9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f07d815a1c0 0x7f07d806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49568/0 (socket says 172.21.15.107:49568) 2023-11-24T08:11:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dcfc9640 1 -- 172.21.15.107:0/2411998631 learned_addr learned my addr 172.21.15.107:0/2411998631 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dd7ca640 1 --2- 172.21.15.107:0/2411998631 >> v2:172.21.15.107:3300/0 conn(0x7f07d815ded0 0x7f07d815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07d7fff640 1 --2- 172.21.15.107:0/2411998631 >> v2:172.21.15.120:3300/0 conn(0x7f07d80b0b40 0x7f07d80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dd7ca640 1 -- 172.21.15.107:0/2411998631 >> v2:172.21.15.120:3300/0 conn(0x7f07d80b0b40 msgr2=0x7f07d80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dd7ca640 1 --2- 172.21.15.107:0/2411998631 >> v2:172.21.15.120:3300/0 conn(0x7f07d80b0b40 0x7f07d80b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dd7ca640 1 -- 172.21.15.107:0/2411998631 >> v1:172.21.15.107:6789/0 conn(0x7f07d815a1c0 legacy=0x7f07d806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.417+0000 7f07dd7ca640 1 -- 172.21.15.107:0/2411998631 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80b14e0 con 0x7f07d815ded0 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.418+0000 7f07dd7ca640 1 --2- 172.21.15.107:0/2411998631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815ded0 0x7f07d815e360 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f07d80cee20 tx=0x7f07d002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d8a2467faabf0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07d77fe640 1 -- 172.21.15.107:0/2411998631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d002cc70 con 0x7f07d815ded0 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07d77fe640 1 -- 172.21.15.107:0/2411998631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d002cdd0 con 0x7f07d815ded0 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07d77fe640 1 -- 172.21.15.107:0/2411998631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0034660 con 0x7f07d815ded0 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2411998631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815ded0 msgr2=0x7f07d815e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2411998631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815ded0 0x7f07d815e360 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f07d80cee20 tx=0x7f07d002bec0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2411998631 shutdown_connections 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2411998631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815ded0 0x7f07d815e360 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2411998631 >> v2:172.21.15.120:3300/0 conn(0x7f07d80b0b40 0x7f07d80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.419+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2411998631 >> 172.21.15.107:0/2411998631 conn(0x7f07d80cbb40 msgr2=0x7f07d80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.420+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2411998631 shutdown_connections 2023-11-24T08:11:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.420+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2411998631 wait complete. 2023-11-24T08:11:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.421+0000 7f07ddfcb640 1 Processor -- start 2023-11-24T08:11:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.421+0000 7f07ddfcb640 1 -- start start 2023-11-24T08:11:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 0x7f07d815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07d815ded0 0x7f07d80c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b24a0 con 0x7f07d815a1c0 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8159460 con 0x7f07d80b0b40 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.422+0000 7f07ddfcb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d815c2a0 con 0x7f07d815ded0 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33928/0 (socket says 172.21.15.107:33928) 2023-11-24T08:11:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 -- 172.21.15.107:0/2575956933 learned_addr learned my addr 172.21.15.107:0/2575956933 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07dd7ca640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07d815ded0 0x7f07d80c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 -- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07d815ded0 msgr2=0x7f07d80c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07d815ded0 0x7f07d80c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07dcfc9640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 0x7f07d815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 -- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 msgr2=0x7f07d815b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 0x7f07d815b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 -- 172.21.15.107:0/2575956933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d0009a70 con 0x7f07d815a1c0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07dcfc9640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 0x7f07d815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.423+0000 7f07d7fff640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c7f0 tx=0x7f07c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.424+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4004220 con 0x7f07d815a1c0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.424+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c40043a0 con 0x7f07d815a1c0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.424+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4007860 con 0x7f07d815a1c0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.424+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d815ea20 con 0x7f07d815a1c0 2023-11-24T08:11:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.424+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80b1e50 con 0x7f07d815a1c0 2023-11-24T08:11:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.425+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d80b2610 con 0x7f07d815a1c0 2023-11-24T08:11:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.426+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 90047+0+0 (secure 0 0 0) 0x7f07c4007bd0 con 0x7f07d815a1c0 2023-11-24T08:11:08.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.427+0000 7f07d57fa640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 0x7f07bc06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.427+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f07c4087e70 con 0x7f07d815a1c0 2023-11-24T08:11:08.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.427+0000 7f07dcfc9640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 0x7f07bc06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.428+0000 7f07dcfc9640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 0x7f07bc06dc00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07cc007fd0 tx=0x7f07cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.431+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07c4054970 con 0x7f07d815a1c0 2023-11-24T08:11:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.622+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f07d804e840 con 0x7f07d815a1c0 2023-11-24T08:11:08.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.629+0000 7f07d57fa640 1 -- 172.21.15.107:0/2575956933 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v23)=0 v23) v1 ==== 130+0+0 (secure 0 0 0) 0x7f07c401e0c0 con 0x7f07d815a1c0 2023-11-24T08:11:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 msgr2=0x7f07bc06dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 0x7f07bc06dc00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07cc007fd0 tx=0x7f07cc009040 comp rx=0 tx=0).stop 2023-11-24T08:11:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 msgr2=0x7f07d815bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c7f0 tx=0x7f07c400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 shutdown_connections 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f07bc06b740 0x7f07bc06dc00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07d815ded0 0x7f07d80c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d815a1c0 0x7f07d815bc70 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 --2- 172.21.15.107:0/2575956933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07d80b0b40 0x7f07d815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 >> 172.21.15.107:0/2575956933 conn(0x7f07d80cbb40 msgr2=0x7f07d80cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 shutdown_connections 2023-11-24T08:11:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.632+0000 7f07ddfcb640 1 -- 172.21.15.107:0/2575956933 wait complete. 2023-11-24T08:11:08.650 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:08.648+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:08.651 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:08.648+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:08.651 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:08.648+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:08.651 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:08.648+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:08.651 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:08.648+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:08.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol9055 subvol_snap1401 subvol_clone6795 2023-11-24T08:11:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.842+0000 7f1db75ec640 1 Processor -- start 2023-11-24T08:11:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.843+0000 7f1db75ec640 1 -- start start 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.844+0000 7f1db75ec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1db00cfb90 0x7f1db0161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.844+0000 7f1db75ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1db00b2400 0x7f1db00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.844+0000 7f1db75ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db01612f0 con 0x7f1db0161fa0 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.844+0000 7f1db75ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db00cff70 con 0x7f1db00b2400 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.844+0000 7f1db75ec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db00afb20 con 0x7f1db00cfb90 2023-11-24T08:11:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db4b87640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1db0161fa0 0x7f1db006ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49578/0 (socket says 172.21.15.107:49578) 2023-11-24T08:11:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db4b87640 1 -- 172.21.15.107:0/4210407848 learned_addr learned my addr 172.21.15.107:0/4210407848 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5b89640 1 --2- 172.21.15.107:0/4210407848 >> v2:172.21.15.107:3300/0 conn(0x7f1db00b2400 0x7f1db00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5388640 1 --2- 172.21.15.107:0/4210407848 >> v2:172.21.15.120:3300/0 conn(0x7f1db00cfb90 0x7f1db0161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1da7fff640 1 -- 172.21.15.107:0/4210407848 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639358835 0 0) 0x7f1db01612f0 con 0x7f1db0161fa0 2023-11-24T08:11:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1da7fff640 1 -- 172.21.15.107:0/4210407848 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c003520 con 0x7f1db0161fa0 2023-11-24T08:11:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5b89640 1 -- 172.21.15.107:0/4210407848 >> v2:172.21.15.120:3300/0 conn(0x7f1db00cfb90 msgr2=0x7f1db0161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5b89640 1 --2- 172.21.15.107:0/4210407848 >> v2:172.21.15.120:3300/0 conn(0x7f1db00cfb90 0x7f1db0161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5b89640 1 -- 172.21.15.107:0/4210407848 >> v1:172.21.15.107:6789/0 conn(0x7f1db0161fa0 legacy=0x7f1db006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.845+0000 7f1db5b89640 1 -- 172.21.15.107:0/4210407848 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db006a850 con 0x7f1db00b2400 2023-11-24T08:11:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.846+0000 7f1db5b89640 1 --2- 172.21.15.107:0/4210407848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00b2400 0x7f1db00cc940 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1db0070d30 tx=0x7f1da802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895c9ebb9a022d24 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.847+0000 7f1da7fff640 1 -- 172.21.15.107:0/4210407848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da802cd00 con 0x7f1db00b2400 2023-11-24T08:11:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.847+0000 7f1da7fff640 1 -- 172.21.15.107:0/4210407848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da802ce60 con 0x7f1db00b2400 2023-11-24T08:11:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.847+0000 7f1da7fff640 1 -- 172.21.15.107:0/4210407848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8034640 con 0x7f1db00b2400 2023-11-24T08:11:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.847+0000 7f1db75ec640 1 -- 172.21.15.107:0/4210407848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00b2400 msgr2=0x7f1db00cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.847+0000 7f1db75ec640 1 --2- 172.21.15.107:0/4210407848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00b2400 0x7f1db00cc940 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1db0070d30 tx=0x7f1da802bf50 comp rx=0 tx=0).stop 2023-11-24T08:11:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 -- 172.21.15.107:0/4210407848 shutdown_connections 2023-11-24T08:11:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 --2- 172.21.15.107:0/4210407848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00b2400 0x7f1db00cc940 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 --2- 172.21.15.107:0/4210407848 >> v2:172.21.15.120:3300/0 conn(0x7f1db00cfb90 0x7f1db0161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 -- 172.21.15.107:0/4210407848 >> 172.21.15.107:0/4210407848 conn(0x7f1db00999d0 msgr2=0x7f1db00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 -- 172.21.15.107:0/4210407848 shutdown_connections 2023-11-24T08:11:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.848+0000 7f1db75ec640 1 -- 172.21.15.107:0/4210407848 wait complete. 2023-11-24T08:11:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.849+0000 7f1db75ec640 1 Processor -- start 2023-11-24T08:11:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 -- start start 2023-11-24T08:11:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1db00b2400 0x7f1db00c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1db00c9bf0 0x7f1db00c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c8800 con 0x7f1db00cfb90 2023-11-24T08:11:08.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00ca0b0 con 0x7f1db00c9bf0 2023-11-24T08:11:08.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.850+0000 7f1db75ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01f6500 con 0x7f1db00b2400 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33962/0 (socket says 172.21.15.107:33962) 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 -- 172.21.15.107:0/582610672 learned_addr learned my addr 172.21.15.107:0/582610672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 -- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1db00b2400 msgr2=0x7f1db00c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1db00b2400 0x7f1db00c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 -- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1db00c9bf0 msgr2=0x7f1db00c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:08.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1db00c9bf0 0x7f1db00c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:08.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 -- 172.21.15.107:0/582610672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da8009a70 con 0x7f1db00cfb90 2023-11-24T08:11:08.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.851+0000 7f1db4b87640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c430 tx=0x7f1da000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.852+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005ce0 con 0x7f1db00cfb90 2023-11-24T08:11:08.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.852+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0005e40 con 0x7f1db00cfb90 2023-11-24T08:11:08.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.852+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00072f0 con 0x7f1db00cfb90 2023-11-24T08:11:08.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.852+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db01f67e0 con 0x7f1db00cfb90 2023-11-24T08:11:08.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.852+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db00ca390 con 0x7f1db00cfb90 2023-11-24T08:11:08.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.853+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db01f6d00 con 0x7f1db00cfb90 2023-11-24T08:11:08.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.854+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 90047+0+0 (secure 0 0 0) 0x7f1da0007450 con 0x7f1db00cfb90 2023-11-24T08:11:08.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.855+0000 7f1da5ffb640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 0x7f1d9406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:08.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.855+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1da0087570 con 0x7f1db00cfb90 2023-11-24T08:11:08.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.855+0000 7f1db5388640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 0x7f1d9406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:08.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.856+0000 7f1db5388640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 0x7f1d9406dc00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098f0 tx=0x7f1dac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:08.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:08.859+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1db01f6d00 con 0x7f1db00cfb90 2023-11-24T08:11:09.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.085+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol9055", "snap_name": "subvol_snap1401", "target_sub_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f1db01f6670 con 0x7f1d9406b740 2023-11-24T08:11:09.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.229+0000 7f1da5ffb640 1 -- 172.21.15.107:0/582610672 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1db01f6670 con 0x7f1d9406b740 2023-11-24T08:11:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 msgr2=0x7f1d9406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 0x7f1d9406dc00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098f0 tx=0x7f1dac008040 comp rx=0 tx=0).stop 2023-11-24T08:11:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 msgr2=0x7f1db01f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c430 tx=0x7f1da000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 shutdown_connections 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1d9406b740 0x7f1d9406dc00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1db00c9bf0 0x7f1db00c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1db00cfb90 0x7f1db01f6fe0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 --2- 172.21.15.107:0/582610672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1db00b2400 0x7f1db00c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 >> 172.21.15.107:0/582610672 conn(0x7f1db00999d0 msgr2=0x7f1db00afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.232+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 shutdown_connections 2023-11-24T08:11:09.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.233+0000 7f1db75ec640 1 -- 172.21.15.107:0/582610672 wait complete. 2023-11-24T08:11:09.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.238+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.238+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.238+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.238+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.240 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.238+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:09.261 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.259+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.261 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.259+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.261 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.259+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.261 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.259+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.261 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:09.259+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.443+0000 7fedd0d0f640 1 Processor -- start 2023-11-24T08:11:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.443+0000 7fedd0d0f640 1 -- start start 2023-11-24T08:11:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.445+0000 7fedd0d0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 0x7fedcc0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.445+0000 7fedd0d0f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.445+0000 7fedd0d0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc15de00 con 0x7fedcc161940 2023-11-24T08:11:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.445+0000 7fedd0d0f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedcc0c6a70 con 0x7fedcc0c65f0 2023-11-24T08:11:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.445+0000 7fedd0d0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc15e0e0 con 0x7fedcc140520 2023-11-24T08:11:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedc1d9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedca59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 0x7fedcc0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedc1d9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:46696/0 (socket says 172.21.15.107:46696) 2023-11-24T08:11:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedc1d9b640 1 -- 172.21.15.107:0/2468977846 learned_addr learned my addr 172.21.15.107:0/2468977846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedca59c640 1 -- 172.21.15.107:0/2468977846 >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 msgr2=0x7fedcc161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedca59c640 1 --2- 172.21.15.107:0/2468977846 >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedca59c640 1 -- 172.21.15.107:0/2468977846 >> v1:172.21.15.120:6789/0 conn(0x7fedcc161940 legacy=0x7fedcc142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedca59c640 1 -- 172.21.15.107:0/2468977846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc15e3c0 con 0x7fedcc140520 2023-11-24T08:11:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.446+0000 7fedc1d9b640 1 --2- 172.21.15.107:0/2468977846 >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:11:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.448+0000 7fedca59c640 1 --2- 172.21.15.107:0/2468977846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 0x7fedcc0c60b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0097a0 tx=0x7fedbc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f45a788feddc5668 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.449+0000 7fedc9d9b640 1 -- 172.21.15.107:0/2468977846 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc02d040 con 0x7fedcc140520 2023-11-24T08:11:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.449+0000 7fedc9d9b640 1 -- 172.21.15.107:0/2468977846 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc031810 con 0x7fedcc140520 2023-11-24T08:11:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.449+0000 7fedc9d9b640 1 -- 172.21.15.107:0/2468977846 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc031ba0 con 0x7fedcc140520 2023-11-24T08:11:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.449+0000 7fedd0d0f640 1 -- 172.21.15.107:0/2468977846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 msgr2=0x7fedcc0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.449+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/2468977846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 0x7fedcc0c60b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0097a0 tx=0x7fedbc02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 -- 172.21.15.107:0/2468977846 shutdown_connections 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/2468977846 >> v2:172.21.15.120:3300/0 conn(0x7fedcc0c65f0 0x7fedcc161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/2468977846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc140520 0x7fedcc0c60b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 -- 172.21.15.107:0/2468977846 >> 172.21.15.107:0/2468977846 conn(0x7fedcc0b2e40 msgr2=0x7fedcc0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 -- 172.21.15.107:0/2468977846 shutdown_connections 2023-11-24T08:11:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.450+0000 7fedd0d0f640 1 -- 172.21.15.107:0/2468977846 wait complete. 2023-11-24T08:11:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.451+0000 7fedd0d0f640 1 Processor -- start 2023-11-24T08:11:09.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.452+0000 7fedd0d0f640 1 -- start start 2023-11-24T08:11:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.452+0000 7fedd0d0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 0x7fedcc0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedd0d0f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedcc140520 0x7fedcc1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedd0d0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedd0d0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0cf0f0 con 0x7fedcc161940 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedd0d0f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0cf330 con 0x7fedcc140520 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedd0d0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0cf610 con 0x7fedcc0c65f0 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34002/0 (socket says 172.21.15.107:34002) 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 -- 172.21.15.107:0/3285632539 learned_addr learned my addr 172.21.15.107:0/3285632539 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedc1d9b640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedcc140520 0x7fedcc1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedca59c640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 0x7fedcc0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 -- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 msgr2=0x7fedcc0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 0x7fedcc0cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 -- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedcc140520 msgr2=0x7fedcc1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedcc140520 0x7fedcc1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedcad9d640 1 -- 172.21.15.107:0/3285632539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40094d0 con 0x7fedcc161940 2023-11-24T08:11:09.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.453+0000 7fedca59c640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 0x7fedcc0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:09.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedcad9d640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c7a0 tx=0x7fedc400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4003b70 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc4003cf0 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4003ea0 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc009400 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.454+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc0b3fd0 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.455+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc04e8b0 con 0x7fedcc161940 2023-11-24T08:11:09.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.456+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 90047+0+0 (secure 0 0 0) 0x7fedc400aa90 con 0x7fedcc161940 2023-11-24T08:11:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.457+0000 7fedc37fe640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 0x7fedac06dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.457+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fedc4087d50 con 0x7fedcc161940 2023-11-24T08:11:09.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.458+0000 7fedca59c640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 0x7fedac06dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.458+0000 7fedca59c640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 0x7fedac06dc30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fedbc036000 tx=0x7fedbc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.461+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fedc4054810 con 0x7fedcc161940 2023-11-24T08:11:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.654+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fedc400ad40 con 0x7fedcc161940 2023-11-24T08:11:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.695+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7fedcc140900 con 0x7fedac06b770 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.711+0000 7fedc37fe640 1 -- 172.21.15.107:0/3285632539 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7fedbc031be0 con 0x7fedac06b770 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:09.714 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:09.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:09.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:09.714 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.713+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 msgr2=0x7fedac06dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.713+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 0x7fedac06dc30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fedbc036000 tx=0x7fedbc03c040 comp rx=0 tx=0).stop 2023-11-24T08:11:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.713+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 msgr2=0x7fedcc0c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.713+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c7a0 tx=0x7fedc400cc60 comp rx=0 tx=0).stop 2023-11-24T08:11:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 shutdown_connections 2023-11-24T08:11:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedac06b770 0x7fedac06dc30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedcc161940 0x7fedcc0c8fb0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedcc140520 0x7fedcc1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 --2- 172.21.15.107:0/3285632539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedcc0c65f0 0x7fedcc0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 >> 172.21.15.107:0/3285632539 conn(0x7fedcc0b2e40 msgr2=0x7fedcc0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 shutdown_connections 2023-11-24T08:11:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.714+0000 7fedd0d0f640 1 -- 172.21.15.107:0/3285632539 wait complete. 2023-11-24T08:11:09.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:09.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.947+0000 7fc041d8e640 1 Processor -- start 2023-11-24T08:11:09.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.947+0000 7fc041d8e640 1 -- start start 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc041d8e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc03c06de10 0x7fc03c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc041d8e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc03c0c49f0 0x7fc03c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc041d8e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c15ec90 con 0x7fc03c072160 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc041d8e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c0cd530 con 0x7fc03c06de10 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc041d8e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c0c4ed0 con 0x7fc03c0c49f0 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc03c06de10 0x7fc03c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc03c06de10 0x7fc03c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34008/0 (socket says 172.21.15.107:34008) 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 -- 172.21.15.107:0/1255151178 learned_addr learned my addr 172.21.15.107:0/1255151178 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03b7fe640 1 --1- 172.21.15.107:0/1255151178 >> v1:172.21.15.107:6789/0 conn(0x7fc03c072160 0x7fc03c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49590/0 (socket says 172.21.15.107:49590) 2023-11-24T08:11:09.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03bfff640 1 --2- 172.21.15.107:0/1255151178 >> v2:172.21.15.120:3300/0 conn(0x7fc03c0c49f0 0x7fc03c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 -- 172.21.15.107:0/1255151178 >> v2:172.21.15.120:3300/0 conn(0x7fc03c0c49f0 msgr2=0x7fc03c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 --2- 172.21.15.107:0/1255151178 >> v2:172.21.15.120:3300/0 conn(0x7fc03c0c49f0 0x7fc03c0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 -- 172.21.15.107:0/1255151178 >> v1:172.21.15.107:6789/0 conn(0x7fc03c072160 legacy=0x7fc03c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.949+0000 7fc03affd640 1 -- 172.21.15.107:0/1255151178 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0c51b0 con 0x7fc03c06de10 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc03affd640 1 --2- 172.21.15.107:0/1255151178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c06de10 0x7fc03c15e750 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc03c072a30 tx=0x7fc02402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacc0dceaa105bc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc03a7fc640 1 -- 172.21.15.107:0/1255151178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024030860 con 0x7fc03c06de10 2023-11-24T08:11:09.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc03a7fc640 1 -- 172.21.15.107:0/1255151178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0240309c0 con 0x7fc03c06de10 2023-11-24T08:11:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc03a7fc640 1 -- 172.21.15.107:0/1255151178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024030d50 con 0x7fc03c06de10 2023-11-24T08:11:09.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 -- 172.21.15.107:0/1255151178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c06de10 msgr2=0x7fc03c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 --2- 172.21.15.107:0/1255151178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c06de10 0x7fc03c15e750 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc03c072a30 tx=0x7fc02402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 -- 172.21.15.107:0/1255151178 shutdown_connections 2023-11-24T08:11:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 --2- 172.21.15.107:0/1255151178 >> v2:172.21.15.120:3300/0 conn(0x7fc03c0c49f0 0x7fc03c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 --2- 172.21.15.107:0/1255151178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c06de10 0x7fc03c15e750 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.951+0000 7fc041d8e640 1 -- 172.21.15.107:0/1255151178 >> 172.21.15.107:0/1255151178 conn(0x7fc03c0b0c50 msgr2=0x7fc03c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.952+0000 7fc041d8e640 1 -- 172.21.15.107:0/1255151178 shutdown_connections 2023-11-24T08:11:09.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.952+0000 7fc041d8e640 1 -- 172.21.15.107:0/1255151178 wait complete. 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.953+0000 7fc041d8e640 1 Processor -- start 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.953+0000 7fc041d8e640 1 -- start start 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc03c06de10 0x7fc03c0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc03c0c49f0 0x7fc03c152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0ad260 con 0x7fc03c072160 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0ad540 con 0x7fc03c06de10 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.954+0000 7fc041d8e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0c8d90 con 0x7fc03c0c49f0 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34022/0 (socket says 172.21.15.107:34022) 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 -- 172.21.15.107:0/2028313254 learned_addr learned my addr 172.21.15.107:0/2028313254 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 -- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc03c0c49f0 msgr2=0x7fc03c152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc03c0c49f0 0x7fc03c152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 -- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc03c06de10 msgr2=0x7fc03c0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:09.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc03c06de10 0x7fc03c0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 -- 172.21.15.107:0/2028313254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240094d0 con 0x7fc03c072160 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.955+0000 7fc03affd640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b9e0 tx=0x7fc024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.956+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024030850 con 0x7fc03c072160 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.956+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024043400 con 0x7fc03c072160 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.956+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c0c9070 con 0x7fc03c072160 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.956+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240437b0 con 0x7fc03c072160 2023-11-24T08:11:09.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.956+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c15cdf0 con 0x7fc03c072160 2023-11-24T08:11:09.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.957+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c0c9280 con 0x7fc03c072160 2023-11-24T08:11:09.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.958+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fc024043910 con 0x7fc03c072160 2023-11-24T08:11:09.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.959+0000 7fc033fff640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 0x7fc01c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:09.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.959+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc02403e070 con 0x7fc03c072160 2023-11-24T08:11:09.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.959+0000 7fc03b7fe640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 0x7fc01c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:09.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.960+0000 7fc03b7fe640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 0x7fc01c06ddb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007fd0 tx=0x7fc02c007d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:09.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:09.963+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0240b3020 con 0x7fc03c072160 2023-11-24T08:11:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.191+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7fc03c0c5a50 con 0x7fc01c06b8f0 2023-11-24T08:11:10.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.200+0000 7fc033fff640 1 -- 172.21.15.107:0/2028313254 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7fc02c004b60 con 0x7fc01c06b8f0 2023-11-24T08:11:10.202 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:10.202 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:10.202 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:10.203 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.202+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 msgr2=0x7fc01c06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.202+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 0x7fc01c06ddb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007fd0 tx=0x7fc02c007d40 comp rx=0 tx=0).stop 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.202+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 msgr2=0x7fc03c15c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.202+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b9e0 tx=0x7fc024004730 comp rx=0 tx=0).stop 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 shutdown_connections 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc01c06b8f0 0x7fc01c06ddb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc03c0c49f0 0x7fc03c152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc03c072160 0x7fc03c15c8b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:10.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 --2- 172.21.15.107:0/2028313254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc03c06de10 0x7fc03c0b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:10.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 >> 172.21.15.107:0/2028313254 conn(0x7fc03c0b0c50 msgr2=0x7fc03c15e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:10.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 shutdown_connections 2023-11-24T08:11:10.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:10.203+0000 7fc041d8e640 1 -- 172.21.15.107:0/2028313254 wait complete. 2023-11-24T08:11:10.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:11:10.748 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:11:11.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.449+0000 7fd3b36c2640 1 Processor -- start 2023-11-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.450+0000 7fd3b36c2640 1 -- start start 2023-11-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.451+0000 7fd3b36c2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac072160 0x7fd3ac15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.451+0000 7fd3b36c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3ac06de10 0x7fd3ac0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.451+0000 7fd3b36c2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac15ec90 con 0x7fd3ac0c49f0 2023-11-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.451+0000 7fd3b36c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac0cd530 con 0x7fd3ac06de10 2023-11-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.451+0000 7fd3b36c2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac0c4ed0 con 0x7fd3ac072160 2023-11-24T08:11:11.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3ac06de10 0x7fd3ac0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3ac06de10 0x7fd3ac0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34030/0 (socket says 172.21.15.107:34030) 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 -- 172.21.15.107:0/1669741512 learned_addr learned my addr 172.21.15.107:0/1669741512 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b145e640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac072160 0x7fd3ac15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 -- 172.21.15.107:0/1669741512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac072160 msgr2=0x7fd3ac15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac072160 0x7fd3ac15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 -- 172.21.15.107:0/1669741512 >> v1:172.21.15.120:6789/0 conn(0x7fd3ac0c49f0 legacy=0x7fd3ac06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.452+0000 7fd3b1c5f640 1 -- 172.21.15.107:0/1669741512 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0c51b0 con 0x7fd3ac06de10 2023-11-24T08:11:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.453+0000 7fd3b1c5f640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac06de10 0x7fd3ac0705c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009870 tx=0x7fd3a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f560a8e166dd86 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.454+0000 7fd3a3fff640 1 -- 172.21.15.107:0/1669741512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4030840 con 0x7fd3ac06de10 2023-11-24T08:11:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.454+0000 7fd3a3fff640 1 -- 172.21.15.107:0/1669741512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a40309a0 con 0x7fd3ac06de10 2023-11-24T08:11:11.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.454+0000 7fd3a3fff640 1 -- 172.21.15.107:0/1669741512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4030d30 con 0x7fd3ac06de10 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.454+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1669741512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac06de10 msgr2=0x7fd3ac0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.454+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac06de10 0x7fd3ac0705c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009870 tx=0x7fd3a402b980 comp rx=0 tx=0).stop 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1669741512 shutdown_connections 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac06de10 0x7fd3ac0705c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1669741512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac072160 0x7fd3ac15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1669741512 >> 172.21.15.107:0/1669741512 conn(0x7fd3ac0b0c50 msgr2=0x7fd3ac0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1669741512 shutdown_connections 2023-11-24T08:11:11.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.455+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1669741512 wait complete. 2023-11-24T08:11:11.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.456+0000 7fd3b36c2640 1 Processor -- start 2023-11-24T08:11:11.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 -- start start 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3ac06de10 0x7fd3ac14ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac0c49f0 0x7fd3ac14e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac158540 con 0x7fd3ac072160 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac15a7c0 con 0x7fd3ac06de10 2023-11-24T08:11:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.457+0000 7fd3b36c2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac09abc0 con 0x7fd3ac0c49f0 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34032/0 (socket says 172.21.15.107:34032) 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 -- 172.21.15.107:0/1912046370 learned_addr learned my addr 172.21.15.107:0/1912046370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b1c5f640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac0c49f0 0x7fd3ac14e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 -- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac0c49f0 msgr2=0x7fd3ac14e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac0c49f0 0x7fd3ac14e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b145e640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3ac06de10 0x7fd3ac14ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 -- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3ac06de10 msgr2=0x7fd3ac14ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3ac06de10 0x7fd3ac14ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.458+0000 7fd3b0c5d640 1 -- 172.21.15.107:0/1912046370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a40094d0 con 0x7fd3ac072160 2023-11-24T08:11:11.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3b0c5d640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c400 tx=0x7fd39c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:11.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0052d0 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c005430 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c003830 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac14ec00 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.459+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac157be0 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.460+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac158680 con 0x7fd3ac072160 2023-11-24T08:11:11.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.461+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fd39c003990 con 0x7fd3ac072160 2023-11-24T08:11:11.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.462+0000 7fd3a1ffb640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 0x7fd39006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:11.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.462+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd39c0875a0 con 0x7fd3ac072160 2023-11-24T08:11:11.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.462+0000 7fd3b145e640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 0x7fd39006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:11.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.463+0000 7fd3b145e640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 0x7fd39006ddb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009fd0 tx=0x7fd3a800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:11.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.466+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3ac158680 con 0x7fd3ac072160 2023-11-24T08:11:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.691+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3ac0c5a50 con 0x7fd39006b8f0 2023-11-24T08:11:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.704+0000 7fd3a1ffb640 1 -- 172.21.15.107:0/1912046370 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7fd3a8016420 con 0x7fd39006b8f0 2023-11-24T08:11:11.706 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:11:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:11.707 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.706+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 msgr2=0x7fd39006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.706+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 0x7fd39006ddb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009fd0 tx=0x7fd3a800b040 comp rx=0 tx=0).stop 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 msgr2=0x7fd3ac15b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c400 tx=0x7fd39c00c8c0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 shutdown_connections 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd39006b8f0 0x7fd39006ddb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3ac0c49f0 0x7fd3ac14e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ac072160 0x7fd3ac15b0a0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 --2- 172.21.15.107:0/1912046370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3ac06de10 0x7fd3ac14ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 >> 172.21.15.107:0/1912046370 conn(0x7fd3ac0b0c50 msgr2=0x7fd3ac0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 shutdown_connections 2023-11-24T08:11:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:11.707+0000 7fd3b36c2640 1 -- 172.21.15.107:0/1912046370 wait complete. 2023-11-24T08:11:12.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.946+0000 7f06aaee7640 1 Processor -- start 2023-11-24T08:11:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.946+0000 7f06aaee7640 1 -- start start 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06aaee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 0x7f06a4166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06aaee7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a415eea0 con 0x7f06a40cc0f0 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06aaee7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a406d600 con 0x7f06a415b440 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06aaee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40c5e00 con 0x7f06a40c59b0 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06a3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f06a40cc0f0 0x7f06a40c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49594/0 (socket says 172.21.15.107:49594) 2023-11-24T08:11:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.948+0000 7f06a3fff640 1 -- 172.21.15.107:0/642721202 learned_addr learned my addr 172.21.15.107:0/642721202 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a9484640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 0x7f06a4166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695993948 0 0) 0x7f06a415eea0 con 0x7f06a40cc0f0 2023-11-24T08:11:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0694003520 con 0x7f06a40cc0f0 2023-11-24T08:11:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039544379 0 0) 0x7f0694003520 con 0x7f06a40cc0f0 2023-11-24T08:11:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 msgr2=0x7f06a4166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 0x7f06a4166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 >> v1:172.21.15.120:6789/0 conn(0x7f06a415b440 legacy=0x7f06a40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a40abf30 con 0x7f06a40cc0f0 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0690001670 con 0x7f06a40cc0f0 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 >> v1:172.21.15.107:6789/0 conn(0x7f06a40cc0f0 legacy=0x7f06a40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06940039a0 0x7f0694003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06940047c0 0x7f0694004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 0x7f06940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069400ba80 con 0x7f06940039a0 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069400bbf0 con 0x7f06940047c0 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.949+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069400bdb0 con 0x7f06a40c59b0 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06940039a0 0x7f0694003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 -- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 msgr2=0x7f06940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06a40c59b0 0x7f06940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 -- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06940047c0 msgr2=0x7f0694004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06940047c0 0x7f0694004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.950+0000 7f06a8c83640 1 -- 172.21.15.107:0/642721202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694003520 con 0x7f06940039a0 2023-11-24T08:11:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.951+0000 7f06a8c83640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06940039a0 0x7f0694003dc0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0698001750 tx=0x7f06940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27a988d3382f39c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.951+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06980052d0 con 0x7f06940039a0 2023-11-24T08:11:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.951+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069802e050 con 0x7f06940039a0 2023-11-24T08:11:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.951+0000 7f06a37fe640 1 -- 172.21.15.107:0/642721202 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06980035f0 con 0x7f06940039a0 2023-11-24T08:11:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 -- 172.21.15.107:0/642721202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06940039a0 msgr2=0x7f0694003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 --2- 172.21.15.107:0/642721202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06940039a0 0x7f0694003dc0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0698001750 tx=0x7f06940057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06a9484640 1 -- 172.21.15.107:0/642721202 reap_dead start 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 -- 172.21.15.107:0/642721202 shutdown_connections 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 -- 172.21.15.107:0/642721202 >> 172.21.15.107:0/642721202 conn(0x7f06a40b0c50 msgr2=0x7f06a40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 -- 172.21.15.107:0/642721202 shutdown_connections 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.952+0000 7f06aaee7640 1 -- 172.21.15.107:0/642721202 wait complete. 2023-11-24T08:11:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.954+0000 7f06aaee7640 1 Processor -- start 2023-11-24T08:11:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.954+0000 7f06aaee7640 1 -- start start 2023-11-24T08:11:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06940039a0 0x7f06a4155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06a40b2fa0 0x7f06a415d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40b2b80 con 0x7f06a40b3420 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4156eb0 con 0x7f06a40b2fa0 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06aaee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40b1cb0 con 0x7f06940039a0 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34050/0 (socket says 172.21.15.107:34050) 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 -- 172.21.15.107:0/1466734627 learned_addr learned my addr 172.21.15.107:0/1466734627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a8c83640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06940039a0 0x7f06a4155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 -- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06940039a0 msgr2=0x7f06a4155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06940039a0 0x7f06a4155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 -- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06a40b2fa0 msgr2=0x7f06a415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06a40b2fa0 0x7f06a415d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.955+0000 7f06a9484640 1 -- 172.21.15.107:0/1466734627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a40b1f00 con 0x7f06a40b3420 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06a9484640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f06a406e890 tx=0x7f069c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c010040 con 0x7f06a40b3420 2023-11-24T08:11:12.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c00bce0 con 0x7f06a40b3420 2023-11-24T08:11:12.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c014640 con 0x7f06a40b3420 2023-11-24T08:11:12.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a40b21e0 con 0x7f06a40b3420 2023-11-24T08:11:12.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.956+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a40b27a0 con 0x7f06a40b3420 2023-11-24T08:11:12.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.957+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a40b2960 con 0x7f06a40b3420 2023-11-24T08:11:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.958+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f069c0147a0 con 0x7f06a40b3420 2023-11-24T08:11:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.960+0000 7f06a17fa640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 0x7f068806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.960+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f069c088250 con 0x7f06a40b3420 2023-11-24T08:11:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.960+0000 7f06a8c83640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 0x7f068806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.961+0000 7f06a8c83640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 0x7f068806dde0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0698033000 tx=0x7f0698009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:12.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:12.964+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f069c054d50 con 0x7f06a40b3420 2023-11-24T08:11:13.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.172+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a415f3d0 con 0x7f068806b920 2023-11-24T08:11:13.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.180+0000 7f06a17fa640 1 -- 172.21.15.107:0/1466734627 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f0698005350 con 0x7f068806b920 2023-11-24T08:11:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.182+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 msgr2=0x7f068806dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.182+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 0x7f068806dde0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0698033000 tx=0x7f0698009030 comp rx=0 tx=0).stop 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.182+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 msgr2=0x7f06a415dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.182+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f06a406e890 tx=0x7f069c00b480 comp rx=0 tx=0).stop 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 shutdown_connections 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f068806b920 0x7f068806dde0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40b3420 0x7f06a415dd30 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06a40b2fa0 0x7f06a415d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 --2- 172.21.15.107:0/1466734627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06940039a0 0x7f06a4155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 >> 172.21.15.107:0/1466734627 conn(0x7f06a40b0c50 msgr2=0x7f06a40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 shutdown_connections 2023-11-24T08:11:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:13.183+0000 7f06aaee7640 1 -- 172.21.15.107:0/1466734627 wait complete. 2023-11-24T08:11:14.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:14.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.432+0000 7f0e1fc87640 1 Processor -- start 2023-11-24T08:11:14.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.433+0000 7f0e1fc87640 1 -- start start 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1fc87640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e1806ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1fc87640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e180cc2b0 0x7f0e1806f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1fc87640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e181612f0 con 0x7f0e180cfb90 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1fc87640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180cff70 con 0x7f0e180cc2b0 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1fc87640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180afb20 con 0x7f0e180b2400 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1da23640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e180cfb90 0x7f0e18161a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49600/0 (socket says 172.21.15.107:49600) 2023-11-24T08:11:14.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.434+0000 7f0e1da23640 1 -- 172.21.15.107:0/3772317428 learned_addr learned my addr 172.21.15.107:0/3772317428 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:14.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1e224640 1 --2- 172.21.15.107:0/3772317428 >> v2:172.21.15.120:3300/0 conn(0x7f0e180cc2b0 0x7f0e1806f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1d222640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e1806ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189072892 0 0) 0x7f0e181612f0 con 0x7f0e180cfb90 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e04003520 con 0x7f0e180cfb90 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924848256 0 0) 0x7f0e04003520 con 0x7f0e180cfb90 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 msgr2=0x7f0e1806ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e1806ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 >> v2:172.21.15.120:3300/0 conn(0x7f0e180cc2b0 msgr2=0x7f0e1806f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 --2- 172.21.15.107:0/3772317428 >> v2:172.21.15.120:3300/0 conn(0x7f0e180cc2b0 0x7f0e1806f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18162170 con 0x7f0e180cfb90 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0e14001670 con 0x7f0e180cfb90 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 >> v1:172.21.15.107:6789/0 conn(0x7f0e180cfb90 legacy=0x7f0e18161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e040047c0 0x7f0e04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180cff70 con 0x7f0e040039a0 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180afb20 con 0x7f0e040047c0 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.435+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0400b9b0 con 0x7f0e180b2400 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1d222640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e040047c0 0x7f0e04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1e224640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 -- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 msgr2=0x7f0e04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 -- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e040047c0 msgr2=0x7f0e04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e040047c0 0x7f0e04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1da23640 1 -- 172.21.15.107:0/3772317428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04003520 con 0x7f0e040039a0 2023-11-24T08:11:14.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1e224640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180b2400 0x7f0e04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:14.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.436+0000 7f0e1d222640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e040047c0 0x7f0e04004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.437+0000 7f0e1da23640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e04003dc0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400c240 tx=0x7f0e040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8b2199f26cf73a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14005ad0 con 0x7f0e040039a0 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e14005c50 con 0x7f0e040039a0 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1ca21640 1 -- 172.21.15.107:0/3772317428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1402c8b0 con 0x7f0e040039a0 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 -- 172.21.15.107:0/3772317428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 msgr2=0x7f0e04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/3772317428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e04003dc0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400c240 tx=0x7f0e040057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1e224640 1 -- 172.21.15.107:0/3772317428 reap_dead start 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 -- 172.21.15.107:0/3772317428 shutdown_connections 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 -- 172.21.15.107:0/3772317428 >> 172.21.15.107:0/3772317428 conn(0x7f0e180999d0 msgr2=0x7f0e180b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 -- 172.21.15.107:0/3772317428 shutdown_connections 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.438+0000 7f0e1fc87640 1 -- 172.21.15.107:0/3772317428 wait complete. 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.440+0000 7f0e1fc87640 1 Processor -- start 2023-11-24T08:11:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.440+0000 7f0e1fc87640 1 -- start start 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180c6590 0x7f0e180c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180b51c0 0x7f0e180b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815e690 con 0x7f0e040039a0 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180c9d80 con 0x7f0e180b51c0 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.441+0000 7f0e1fc87640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180b3d40 con 0x7f0e180c6590 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34074/0 (socket says 172.21.15.107:34074) 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 -- 172.21.15.107:0/2827466744 learned_addr learned my addr 172.21.15.107:0/2827466744 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 -- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180c6590 msgr2=0x7f0e180c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180c6590 0x7f0e180c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 -- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180b51c0 msgr2=0x7f0e180b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180b51c0 0x7f0e180b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 -- 172.21.15.107:0/2827466744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1815dd60 con 0x7f0e040039a0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.442+0000 7f0e1da23640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400c240 tx=0x7f0e14001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.443+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14039070 con 0x7f0e040039a0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.443+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1402ccc0 con 0x7f0e040039a0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.443+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1403c040 con 0x7f0e040039a0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.443+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1815e040 con 0x7f0e040039a0 2023-11-24T08:11:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.443+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180c6a10 con 0x7f0e040039a0 2023-11-24T08:11:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.444+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1804e8b0 con 0x7f0e040039a0 2023-11-24T08:11:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.445+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f0e14032070 con 0x7f0e040039a0 2023-11-24T08:11:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.446+0000 7f0e0e7fc640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 0x7f0dfc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.446+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e140a94a0 con 0x7f0e040039a0 2023-11-24T08:11:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.446+0000 7f0e1d222640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 0x7f0dfc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.447+0000 7f0e1d222640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 0x7f0dfc06ddb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0e180c4600 tx=0x7f0e08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:14.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.450+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e14075dd0 con 0x7f0e040039a0 2023-11-24T08:11:14.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:14.679+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e1806ae10 con 0x7f0dfc06b8f0 2023-11-24T08:11:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.797+0000 7f0e0e7fc640 1 -- 172.21.15.107:0/2827466744 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f0e08009d80 con 0x7f0dfc06b8f0 2023-11-24T08:11:15.799 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:15.800 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 msgr2=0x7f0dfc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 0x7f0dfc06ddb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0e180c4600 tx=0x7f0e08008040 comp rx=0 tx=0).stop 2023-11-24T08:11:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 msgr2=0x7f0e180b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400c240 tx=0x7f0e14001a00 comp rx=0 tx=0).stop 2023-11-24T08:11:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 shutdown_connections 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06b8f0 0x7f0dfc06ddb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e180b51c0 0x7f0e180b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e180c6590 0x7f0e180c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 --2- 172.21.15.107:0/2827466744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e040039a0 0x7f0e180b36d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 >> 172.21.15.107:0/2827466744 conn(0x7f0e180999d0 msgr2=0x7f0e180ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 shutdown_connections 2023-11-24T08:11:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:15.802+0000 7f0e1fc87640 1 -- 172.21.15.107:0/2827466744 wait complete. 2023-11-24T08:11:15.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:16.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.057+0000 7f80ada9f640 1 Processor -- start 2023-11-24T08:11:16.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.057+0000 7f80ada9f640 1 -- start start 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80a80cd260 0x7f80a806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80d0be0 0x7f80a806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80a806b820 0x7f80a806f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a8070f50 con 0x7f80a806b820 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80702d0 con 0x7f80a80cd260 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ada9f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80705b0 con 0x7f80a80d0be0 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ad29e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80a806b820 0x7f80a806f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ad29e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80a806b820 0x7f80a806f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34080/0 (socket says 172.21.15.107:34080) 2023-11-24T08:11:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80ad29e640 1 -- 172.21.15.107:0/935251615 learned_addr learned my addr 172.21.15.107:0/935251615 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.059+0000 7f80a7fff640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80d0be0 0x7f80a806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80aca9d640 1 --2- 172.21.15.107:0/935251615 >> v2:172.21.15.120:3300/0 conn(0x7f80a80cd260 0x7f80a806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80ad29e640 1 -- 172.21.15.107:0/935251615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80d0be0 msgr2=0x7f80a806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80ad29e640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80d0be0 0x7f80a806eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80ad29e640 1 -- 172.21.15.107:0/935251615 >> v2:172.21.15.120:3300/0 conn(0x7f80a80cd260 msgr2=0x7f80a806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80ad29e640 1 --2- 172.21.15.107:0/935251615 >> v2:172.21.15.120:3300/0 conn(0x7f80a80cd260 0x7f80a806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.060+0000 7f80ad29e640 1 -- 172.21.15.107:0/935251615 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8070890 con 0x7f80a806b820 2023-11-24T08:11:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.061+0000 7f80ad29e640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a806b820 0x7f80a806f3f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f80a8068310 tx=0x7f80a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af370e03d22943a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.061+0000 7f80a77fe640 1 -- 172.21.15.107:0/935251615 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a002c9e0 con 0x7f80a806b820 2023-11-24T08:11:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80a77fe640 1 -- 172.21.15.107:0/935251615 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a002cb40 con 0x7f80a806b820 2023-11-24T08:11:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80a77fe640 1 -- 172.21.15.107:0/935251615 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a00343f0 con 0x7f80a806b820 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 -- 172.21.15.107:0/935251615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a806b820 msgr2=0x7f80a806f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a806b820 0x7f80a806f3f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f80a8068310 tx=0x7f80a002be40 comp rx=0 tx=0).stop 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 -- 172.21.15.107:0/935251615 shutdown_connections 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a806b820 0x7f80a806f3f0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 --2- 172.21.15.107:0/935251615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80d0be0 0x7f80a806eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 --2- 172.21.15.107:0/935251615 >> v2:172.21.15.120:3300/0 conn(0x7f80a80cd260 0x7f80a806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.062+0000 7f80ada9f640 1 -- 172.21.15.107:0/935251615 >> 172.21.15.107:0/935251615 conn(0x7f80a80d1060 msgr2=0x7f80a80d1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.063+0000 7f80ada9f640 1 -- 172.21.15.107:0/935251615 shutdown_connections 2023-11-24T08:11:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.063+0000 7f80ada9f640 1 -- 172.21.15.107:0/935251615 wait complete. 2023-11-24T08:11:16.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.065+0000 7f80ada9f640 1 Processor -- start 2023-11-24T08:11:16.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.065+0000 7f80ada9f640 1 -- start start 2023-11-24T08:11:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80a806b820 0x7f80a814f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80cd260 0x7f80a815a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80b4ae0 con 0x7f80a80d0be0 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a81f77a0 con 0x7f80a806b820 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.066+0000 7f80ada9f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80b4d30 con 0x7f80a80cd260 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34088/0 (socket says 172.21.15.107:34088) 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 -- 172.21.15.107:0/230797725 learned_addr learned my addr 172.21.15.107:0/230797725 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80aca9d640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80a806b820 0x7f80a814f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80a7fff640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80cd260 0x7f80a815a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 -- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80cd260 msgr2=0x7f80a815a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80cd260 0x7f80a815a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 -- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80a806b820 msgr2=0x7f80a814f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80a806b820 0x7f80a814f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.067+0000 7f80ad29e640 1 -- 172.21.15.107:0/230797725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0009b30 con 0x7f80a80d0be0 2023-11-24T08:11:16.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80ad29e640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f80a0036040 tx=0x7f80a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0009510 con 0x7f80a80d0be0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a003e450 con 0x7f80a80d0be0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a003e800 con 0x7f80a80d0be0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a80b5010 con 0x7f80a80d0be0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.068+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a80b55d0 con 0x7f80a80d0be0 2023-11-24T08:11:16.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.070+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a804e840 con 0x7f80a80d0be0 2023-11-24T08:11:16.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.070+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f80a003e960 con 0x7f80a80d0be0 2023-11-24T08:11:16.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.071+0000 7f80a57fa640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 0x7f808c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.072+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f80a00aacb0 con 0x7f80a80d0be0 2023-11-24T08:11:16.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.072+0000 7f80aca9d640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 0x7f808c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.072+0000 7f80aca9d640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 0x7f808c06dde0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f809c006fd0 tx=0x7f809c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.076+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80a00777f0 con 0x7f80a80d0be0 2023-11-24T08:11:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.351+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f80a806ff60 con 0x7f808c06b920 2023-11-24T08:11:16.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.361+0000 7f80a57fa640 1 -- 172.21.15.107:0/230797725 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f809c00b640 con 0x7f808c06b920 2023-11-24T08:11:16.363 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:16.363 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:16.364 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 msgr2=0x7f808c06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 0x7f808c06dde0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f809c006fd0 tx=0x7f809c008040 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 msgr2=0x7f80a815abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f80a0036040 tx=0x7f80a0004a40 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 shutdown_connections 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f808c06b920 0x7f808c06dde0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80a80d0be0 0x7f80a815abf0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80a80cd260 0x7f80a815a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.364+0000 7f80ada9f640 1 --2- 172.21.15.107:0/230797725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80a806b820 0x7f80a814f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.365+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 >> 172.21.15.107:0/230797725 conn(0x7f80a80d1060 msgr2=0x7f80a80cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.365+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 shutdown_connections 2023-11-24T08:11:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.365+0000 7f80ada9f640 1 -- 172.21.15.107:0/230797725 wait complete. 2023-11-24T08:11:16.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:16.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.602+0000 7fd6b983d640 1 Processor -- start 2023-11-24T08:11:16.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.602+0000 7fd6b983d640 1 -- start start 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 0x7fd6b40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6b40d0a00 0x7fd6b406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 0x7fd6b40cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b40cc170 con 0x7fd6b40d0a00 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b4070870 con 0x7fd6b40cb850 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b983d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4070b50 con 0x7fd6b40b40d0 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b27fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6b40d0a00 0x7fd6b406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b27fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6b40d0a00 0x7fd6b406ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34098/0 (socket says 172.21.15.107:34098) 2023-11-24T08:11:16.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.604+0000 7fd6b27fc640 1 -- 172.21.15.107:0/3529410966 learned_addr learned my addr 172.21.15.107:0/3529410966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:16.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b37fe640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 0x7fd6b40cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b27fc640 1 -- 172.21.15.107:0/3529410966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 msgr2=0x7fd6b40cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:16.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b2ffd640 1 --2- 172.21.15.107:0/3529410966 >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 0x7fd6b40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 0x7fd6b40cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b27fc640 1 -- 172.21.15.107:0/3529410966 >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 msgr2=0x7fd6b40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/3529410966 >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 0x7fd6b40cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b27fc640 1 -- 172.21.15.107:0/3529410966 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4070e30 con 0x7fd6b40d0a00 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b37fe640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 0x7fd6b40cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.605+0000 7fd6b2ffd640 1 --2- 172.21.15.107:0/3529410966 >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 0x7fd6b40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:16.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.606+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40d0a00 0x7fd6b406ff70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd6b406b440 tx=0x7fd6a802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d615c1eb087a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.606+0000 7fd6b1ffb640 1 -- 172.21.15.107:0/3529410966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a80317d0 con 0x7fd6b40d0a00 2023-11-24T08:11:16.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.606+0000 7fd6b1ffb640 1 -- 172.21.15.107:0/3529410966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a8031930 con 0x7fd6b40d0a00 2023-11-24T08:11:16.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.606+0000 7fd6b1ffb640 1 -- 172.21.15.107:0/3529410966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8031cc0 con 0x7fd6b40d0a00 2023-11-24T08:11:16.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 -- 172.21.15.107:0/3529410966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40d0a00 msgr2=0x7fd6b406ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40d0a00 0x7fd6b406ff70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd6b406b440 tx=0x7fd6a802b870 comp rx=0 tx=0).stop 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 -- 172.21.15.107:0/3529410966 shutdown_connections 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40b40d0 0x7fd6b40cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 --2- 172.21.15.107:0/3529410966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40d0a00 0x7fd6b406ff70 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 --2- 172.21.15.107:0/3529410966 >> v2:172.21.15.120:3300/0 conn(0x7fd6b40cb850 0x7fd6b40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.607+0000 7fd6b983d640 1 -- 172.21.15.107:0/3529410966 >> 172.21.15.107:0/3529410966 conn(0x7fd6b406e7d0 msgr2=0x7fd6b406ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.608+0000 7fd6b983d640 1 -- 172.21.15.107:0/3529410966 shutdown_connections 2023-11-24T08:11:16.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.608+0000 7fd6b983d640 1 -- 172.21.15.107:0/3529410966 wait complete. 2023-11-24T08:11:16.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.610+0000 7fd6b983d640 1 Processor -- start 2023-11-24T08:11:16.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.610+0000 7fd6b983d640 1 -- start start 2023-11-24T08:11:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.611+0000 7fd6b983d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.611+0000 7fd6b983d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6b40cb850 0x7fd6b40c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b983d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40d0a00 0x7fd6b40ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b983d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40ad220 con 0x7fd6b40b40d0 2023-11-24T08:11:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b983d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b415aa20 con 0x7fd6b40cb850 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b983d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40ca700 con 0x7fd6b40d0a00 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34114/0 (socket says 172.21.15.107:34114) 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 -- 172.21.15.107:0/2363718454 learned_addr learned my addr 172.21.15.107:0/2363718454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b37fe640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40d0a00 0x7fd6b40ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6b40cb850 0x7fd6b40c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 -- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40d0a00 msgr2=0x7fd6b40ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40d0a00 0x7fd6b40ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 -- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6b40cb850 msgr2=0x7fd6b40c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6b40cb850 0x7fd6b40c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.612+0000 7fd6b2ffd640 1 -- 172.21.15.107:0/2363718454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80094d0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd6b2ffd640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c7f0 tx=0x7fd6a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4007820 con 0x7fd6b40b40d0 2023-11-24T08:11:16.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4007980 con 0x7fd6b40b40d0 2023-11-24T08:11:16.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b40ac8f0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4007cf0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.613+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b40aceb0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.614+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b40cd680 con 0x7fd6b40b40d0 2023-11-24T08:11:16.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.616+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fd6a40113f0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.617+0000 7fd69bfff640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 0x7fd6a006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:16.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.617+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6a40882b0 con 0x7fd6b40b40d0 2023-11-24T08:11:16.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.617+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 0x7fd6a006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:16.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.618+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 0x7fd6a006ddb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8036000 tx=0x7fd6a803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:16.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.622+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a4054d20 con 0x7fd6b40b40d0 2023-11-24T08:11:16.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.896+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6b40cf150 con 0x7fd6a006b8f0 2023-11-24T08:11:16.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.909+0000 7fd69bfff640 1 -- 172.21.15.107:0/2363718454 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fd6a8031aa0 con 0x7fd6a006b8f0 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:16.912 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:16.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 msgr2=0x7fd6a006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 0x7fd6a006ddb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8036000 tx=0x7fd6a803d040 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 msgr2=0x7fd6b40c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c7f0 tx=0x7fd6a400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 shutdown_connections 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006b8f0 0x7fd6a006ddb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6b40d0a00 0x7fd6b40ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6b40cb850 0x7fd6b40c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 --2- 172.21.15.107:0/2363718454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6b40b40d0 0x7fd6b40c7170 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.912+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 >> 172.21.15.107:0/2363718454 conn(0x7fd6b406e7d0 msgr2=0x7fd6b40b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.913+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 shutdown_connections 2023-11-24T08:11:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:16.913+0000 7fd6b983d640 1 -- 172.21.15.107:0/2363718454 wait complete. 2023-11-24T08:11:17.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:18.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.165+0000 7feffd87c640 1 Processor -- start 2023-11-24T08:11:18.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.165+0000 7feffd87c640 1 -- start start 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feffd87c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feff806f360 0x7feff80cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feffd87c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806bb90 con 0x7feff815d380 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feffd87c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806be70 con 0x7feff80ce2c0 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feffd87c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff806b980 con 0x7feff806f360 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feff6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feff815d380 0x7feff80cdbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43484/0 (socket says 172.21.15.107:43484) 2023-11-24T08:11:18.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.167+0000 7feff6ffd640 1 -- 172.21.15.107:0/411878495 learned_addr learned my addr 172.21.15.107:0/411878495 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761330250 0 0) 0x7feff806bb90 con 0x7feff815d380 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe0003520 con 0x7feff815d380 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff67fc640 1 --2- 172.21.15.107:0/411878495 >> v2:172.21.15.120:3300/0 conn(0x7feff806f360 0x7feff80cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949916653 0 0) 0x7fefe0003520 con 0x7feff815d380 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 >> v2:172.21.15.120:3300/0 conn(0x7feff806f360 msgr2=0x7feff80cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 --2- 172.21.15.107:0/411878495 >> v2:172.21.15.120:3300/0 conn(0x7feff806f360 0x7feff80cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 >> v1:172.21.15.120:6789/0 conn(0x7feff80ce2c0 legacy=0x7feff8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff806c150 con 0x7feff815d380 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fefe8001670 con 0x7feff815d380 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 >> v1:172.21.15.107:6789/0 conn(0x7feff815d380 legacy=0x7feff80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe0003970 0x7fefe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7fefe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe0005180 0x7fefe000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff806b980 con 0x7fefe00047c0 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe000c0d0 con 0x7fefe0005180 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.168+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe000c290 con 0x7fefe0003970 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7fefe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff77fe640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe0005180 0x7fefe000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff6ffd640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe0003970 0x7fefe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 -- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe0003970 msgr2=0x7fefe0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe0003970 0x7fefe0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 -- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe0005180 msgr2=0x7fefe000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe0005180 0x7fefe000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff67fc640 1 -- 172.21.15.107:0/411878495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe0003520 con 0x7fefe00047c0 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff77fe640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe0005180 0x7fefe000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:11:18.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.169+0000 7feff77fe640 1 -- 172.21.15.107:0/411878495 reap_dead start 2023-11-24T08:11:18.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.172+0000 7feff67fc640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7fefe0004c40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ba30 tx=0x7fefe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b681bac94853 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:18.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.173+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec032640 con 0x7fefe00047c0 2023-11-24T08:11:18.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.173+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec0327a0 con 0x7fefe00047c0 2023-11-24T08:11:18.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.173+0000 7feff5ffb640 1 -- 172.21.15.107:0/411878495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec032b30 con 0x7fefe00047c0 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.173+0000 7feffd87c640 1 -- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 msgr2=0x7fefe0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.173+0000 7feffd87c640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7fefe0004c40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ba30 tx=0x7fefe00057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.174+0000 7feffd87c640 1 -- 172.21.15.107:0/411878495 shutdown_connections 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.174+0000 7feffd87c640 1 --2- 172.21.15.107:0/411878495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7fefe0004c40 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.174+0000 7feffd87c640 1 -- 172.21.15.107:0/411878495 >> 172.21.15.107:0/411878495 conn(0x7feff806e7d0 msgr2=0x7feff806ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.174+0000 7feffd87c640 1 -- 172.21.15.107:0/411878495 shutdown_connections 2023-11-24T08:11:18.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.174+0000 7feffd87c640 1 -- 172.21.15.107:0/411878495 wait complete. 2023-11-24T08:11:18.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.176+0000 7feffd87c640 1 Processor -- start 2023-11-24T08:11:18.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.176+0000 7feffd87c640 1 -- start start 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feff8157080 0x7feff815fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feff80c7070 0x7feff80c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8158020 con 0x7fefe00047c0 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8152d80 con 0x7feff8157080 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.177+0000 7feffd87c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff8157500 con 0x7feff80c7070 2023-11-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41192/0 (socket says 172.21.15.107:41192) 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 -- 172.21.15.107:0/3001891624 learned_addr learned my addr 172.21.15.107:0/3001891624 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 -- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feff80c7070 msgr2=0x7feff80c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feff80c7070 0x7feff80c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 -- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feff8157080 msgr2=0x7feff815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feff8157080 0x7feff815fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 -- 172.21.15.107:0/3001891624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8157780 con 0x7fefe00047c0 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.178+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fefe800e7b0 tx=0x7fefe800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.179+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800c850 con 0x7fefe00047c0 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.179+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe800c9d0 con 0x7fefe00047c0 2023-11-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.179+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800cd80 con 0x7fefe00047c0 2023-11-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.179+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff8157a60 con 0x7fefe00047c0 2023-11-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.179+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff81e7b80 con 0x7fefe00047c0 2023-11-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.180+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff81e79e0 con 0x7fefe00047c0 2023-11-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.181+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fefe801d460 con 0x7fefe00047c0 2023-11-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.182+0000 7fefdffff640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 0x7fefe406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.183+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fefe8087ee0 con 0x7fefe00047c0 2023-11-24T08:11:18.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.183+0000 7feff67fc640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 0x7fefe406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:18.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.183+0000 7feff67fc640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 0x7fefe406dde0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ba00 tx=0x7fefec032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.187+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feff81e79e0 con 0x7fefe00047c0 2023-11-24T08:11:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.469+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7feff806c710 con 0x7fefe406b920 2023-11-24T08:11:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.481+0000 7fefdffff640 1 -- 172.21.15.107:0/3001891624 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fefec032a10 con 0x7fefe406b920 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:18.484 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 msgr2=0x7fefe406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 0x7fefe406dde0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ba00 tx=0x7fefec032000 comp rx=0 tx=0).stop 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 msgr2=0x7feff8152500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fefe800e7b0 tx=0x7fefe800ec70 comp rx=0 tx=0).stop 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 shutdown_connections 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefe406b920 0x7fefe406dde0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feff80c7070 0x7feff80c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feff8157080 0x7feff815fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 --2- 172.21.15.107:0/3001891624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefe00047c0 0x7feff8152500 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.483+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 >> 172.21.15.107:0/3001891624 conn(0x7feff806e7d0 msgr2=0x7feff8161670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.484+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 shutdown_connections 2023-11-24T08:11:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:18.484+0000 7feffd87c640 1 -- 172.21.15.107:0/3001891624 wait complete. 2023-11-24T08:11:19.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.739+0000 7f2721c09640 1 Processor -- start 2023-11-24T08:11:19.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.740+0000 7f2721c09640 1 -- start start 2023-11-24T08:11:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f2721c09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 0x7f271c0d4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f2721c09640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0cf300 con 0x7f271c0d1e30 2023-11-24T08:11:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f2721c09640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0d1090 con 0x7f271c0cfc80 2023-11-24T08:11:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f2721c09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0d1370 con 0x7f271c15d580 2023-11-24T08:11:19.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f271affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f271c0d1e30 0x7f271c0cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43498/0 (socket says 172.21.15.107:43498) 2023-11-24T08:11:19.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.741+0000 7f271affd640 1 -- 172.21.15.107:0/3912767076 learned_addr learned my addr 172.21.15.107:0/3912767076 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:19.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271b7fe640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 0x7f271c0d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448771001 0 0) 0x7f271c0cf300 con 0x7f271c0d1e30 2023-11-24T08:11:19.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2704003520 con 0x7f271c0d1e30 2023-11-24T08:11:19.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775882316 0 0) 0x7f2704003520 con 0x7f271c0d1e30 2023-11-24T08:11:19.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 msgr2=0x7f271c0d4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:19.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 0x7f271c0d4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.742+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 >> v1:172.21.15.120:6789/0 conn(0x7f271c0cfc80 legacy=0x7f271c0d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c0d1650 con 0x7f271c0d1e30 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2710001670 con 0x7f271c0d1e30 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 >> v1:172.21.15.107:6789/0 conn(0x7f271c0d1e30 legacy=0x7f271c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27040039a0 0x7f2704003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f2704004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 0x7f2704005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0d1370 con 0x7f27040047c0 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270400ba60 con 0x7f27040039a0 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270400bc20 con 0x7f271c15d580 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271affd640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f2704004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271affd640 1 -- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 msgr2=0x7f2704005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271affd640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c15d580 0x7f2704005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271affd640 1 -- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27040039a0 msgr2=0x7f2704003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.743+0000 7f271affd640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27040039a0 0x7f2704003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.744+0000 7f271affd640 1 -- 172.21.15.107:0/3912767076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2704003520 con 0x7f27040047c0 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.745+0000 7f271affd640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f2704004c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2710001eb0 tx=0x7f27040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20efe26e466d43b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.745+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710005ad0 con 0x7f27040047c0 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.745+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710005c30 con 0x7f27040047c0 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.745+0000 7f271a7fc640 1 -- 172.21.15.107:0/3912767076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271002c7c0 con 0x7f27040047c0 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 -- 172.21.15.107:0/3912767076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 msgr2=0x7f2704004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 --2- 172.21.15.107:0/3912767076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f2704004c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2710001eb0 tx=0x7f27040057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f271bfff640 1 -- 172.21.15.107:0/3912767076 reap_dead start 2023-11-24T08:11:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 -- 172.21.15.107:0/3912767076 shutdown_connections 2023-11-24T08:11:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 -- 172.21.15.107:0/3912767076 >> 172.21.15.107:0/3912767076 conn(0x7f271c09c770 msgr2=0x7f271c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 -- 172.21.15.107:0/3912767076 shutdown_connections 2023-11-24T08:11:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.746+0000 7f2721c09640 1 -- 172.21.15.107:0/3912767076 wait complete. 2023-11-24T08:11:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.748+0000 7f2721c09640 1 Processor -- start 2023-11-24T08:11:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.748+0000 7f2721c09640 1 -- start start 2023-11-24T08:11:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c151240 0x7f271c0b7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c0b6810 0x7f271c0b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0b9340 con 0x7f27040047c0 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0b6380 con 0x7f271c0b6810 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.749+0000 7f2721c09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0b7c80 con 0x7f271c151240 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41200/0 (socket says 172.21.15.107:41200) 2023-11-24T08:11:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 -- 172.21.15.107:0/3963955900 learned_addr learned my addr 172.21.15.107:0/3963955900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271affd640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c151240 0x7f271c0b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 -- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c151240 msgr2=0x7f271c0b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c151240 0x7f271c0b7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 -- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c0b6810 msgr2=0x7f271c0b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271bfff640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c0b6810 0x7f271c0b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c0b6810 0x7f271c0b6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.750+0000 7f271b7fe640 1 -- 172.21.15.107:0/3963955900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c0b7f60 con 0x7f27040047c0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f271b7fe640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e8c0 tx=0x7f270c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c009680 con 0x7f27040047c0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c00c940 con 0x7f27040047c0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c00ccd0 con 0x7f27040047c0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c0b8240 con 0x7f27040047c0 2023-11-24T08:11:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.751+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c0b7990 con 0x7f27040047c0 2023-11-24T08:11:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.753+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c04e8b0 con 0x7f27040047c0 2023-11-24T08:11:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.754+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f270c013470 con 0x7f27040047c0 2023-11-24T08:11:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.755+0000 7f2703fff640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 0x7f270806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.755+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f270c0880d0 con 0x7f27040047c0 2023-11-24T08:11:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.755+0000 7f271affd640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 0x7f270806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.756+0000 7f271affd640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 0x7f270806ddb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2710034000 tx=0x7f2710009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:19.760+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f270c0586c0 con 0x7f27040047c0 2023-11-24T08:11:20.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.033+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f271c0b10e0 con 0x7f270806b8f0 2023-11-24T08:11:20.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.050+0000 7f2703fff640 1 -- 172.21.15.107:0/3963955900 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f271c0b10e0 con 0x7f270806b8f0 2023-11-24T08:11:20.052 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:20.053 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 msgr2=0x7f270806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 0x7f270806ddb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2710034000 tx=0x7f2710009030 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 msgr2=0x7f271c150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e8c0 tx=0x7f270c00ed80 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 shutdown_connections 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f270806b8f0 0x7f270806ddb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c0b6810 0x7f271c0b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c151240 0x7f271c0b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 --2- 172.21.15.107:0/3963955900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27040047c0 0x7f271c150bb0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.052+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 >> 172.21.15.107:0/3963955900 conn(0x7f271c09c770 msgr2=0x7f271c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:20.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.053+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 shutdown_connections 2023-11-24T08:11:20.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:20.053+0000 7f2721c09640 1 -- 172.21.15.107:0/3963955900 wait complete. 2023-11-24T08:11:21.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.311+0000 7f42b3fff640 1 Processor -- start 2023-11-24T08:11:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.311+0000 7f42b3fff640 1 -- start start 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42b40b2350 0x7f42b4165070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b406ef40 con 0x7f42b40af7d0 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b3fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b406f220 con 0x7f42b4143a90 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b3fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b40b5120 con 0x7f42b40b2350 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42b40b2350 0x7f42b4165070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f42b40af7d0 0x7f42b40b1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43506/0 (socket says 172.21.15.107:43506) 2023-11-24T08:11:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.313+0000 7f42b27fc640 1 -- 172.21.15.107:0/2643684373 learned_addr learned my addr 172.21.15.107:0/2643684373 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.314+0000 7f42b37fe640 1 -- 172.21.15.107:0/2643684373 >> v1:172.21.15.120:6789/0 conn(0x7f42b4143a90 legacy=0x7f42b40b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.314+0000 7f42b37fe640 1 -- 172.21.15.107:0/2643684373 >> v1:172.21.15.107:6789/0 conn(0x7f42b40af7d0 legacy=0x7f42b40b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.314+0000 7f42b37fe640 1 -- 172.21.15.107:0/2643684373 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b406f500 con 0x7f42b40b2350 2023-11-24T08:11:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.315+0000 7f42b37fe640 1 --2- 172.21.15.107:0/2643684373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b4165070 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009d90 tx=0x7f42ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bbf580caf9ca8a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.315+0000 7f42b1ffb640 1 -- 172.21.15.107:0/2643684373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac02cc60 con 0x7f42b40b2350 2023-11-24T08:11:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.315+0000 7f42b1ffb640 1 -- 172.21.15.107:0/2643684373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac02cdc0 con 0x7f42b40b2350 2023-11-24T08:11:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.315+0000 7f42b1ffb640 1 -- 172.21.15.107:0/2643684373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac034660 con 0x7f42b40b2350 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 -- 172.21.15.107:0/2643684373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 msgr2=0x7f42b4165070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2643684373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b4165070 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009d90 tx=0x7f42ac02beb0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 -- 172.21.15.107:0/2643684373 shutdown_connections 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2643684373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b4165070 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 -- 172.21.15.107:0/2643684373 >> 172.21.15.107:0/2643684373 conn(0x7f42b406d6c0 msgr2=0x7f42b406daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 -- 172.21.15.107:0/2643684373 shutdown_connections 2023-11-24T08:11:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.316+0000 7f42b3fff640 1 -- 172.21.15.107:0/2643684373 wait complete. 2023-11-24T08:11:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.318+0000 7f42b3fff640 1 Processor -- start 2023-11-24T08:11:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.318+0000 7f42b3fff640 1 -- start start 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 0x7f42b413c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42b4143a90 0x7f42b4141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b413cf90 con 0x7f42b40b2350 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b413b750 con 0x7f42b4143a90 2023-11-24T08:11:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b413ba30 con 0x7f42b40af7d0 2023-11-24T08:11:21.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:21.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41214/0 (socket says 172.21.15.107:41214) 2023-11-24T08:11:21.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.319+0000 7f42b27fc640 1 -- 172.21.15.107:0/2476912248 learned_addr learned my addr 172.21.15.107:0/2476912248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b2ffd640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 0x7f42b413c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 -- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 msgr2=0x7f42b413c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 0x7f42b413c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 -- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42b4143a90 msgr2=0x7f42b4141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42b4143a90 0x7f42b4141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 -- 172.21.15.107:0/2476912248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac009a70 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b2ffd640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 0x7f42b413c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.320+0000 7f42b27fc640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c500 tx=0x7f42a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.321+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8005ce0 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.321+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8005e40 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.321+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80072f0 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.321+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b413bd10 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.321+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b41422c0 con 0x7f42b40b2350 2023-11-24T08:11:21.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.322+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b413c230 con 0x7f42b40b2350 2023-11-24T08:11:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.323+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f42a8007510 con 0x7f42b40b2350 2023-11-24T08:11:21.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.324+0000 7f429b7fe640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 0x7f42a006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:21.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.324+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42a8087630 con 0x7f42b40b2350 2023-11-24T08:11:21.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.325+0000 7f42b2ffd640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 0x7f42a006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.325+0000 7f42b2ffd640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 0x7f42a006ddb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f42a400a830 tx=0x7f42a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.329+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42b413c230 con 0x7f42b40b2350 2023-11-24T08:11:21.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.592+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f42b4143e70 con 0x7f42a006b8f0 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.609+0000 7f429b7fe640 1 -- 172.21.15.107:0/2476912248 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f42a400adb0 con 0x7f42a006b8f0 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:21.611 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:21.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:21.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:21.612 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:21.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 msgr2=0x7f42a006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 0x7f42a006ddb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f42a400a830 tx=0x7f42a4008040 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 msgr2=0x7f42b413ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c500 tx=0x7f42a800c9c0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 shutdown_connections 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f42a006b8f0 0x7f42a006ddb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42b4143a90 0x7f42b4141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42b40b2350 0x7f42b413ca50 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 --2- 172.21.15.107:0/2476912248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42b40af7d0 0x7f42b413c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:21.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 >> 172.21.15.107:0/2476912248 conn(0x7f42b406d6c0 msgr2=0x7f42b40cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:21.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 shutdown_connections 2023-11-24T08:11:21.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:21.612+0000 7f42b3fff640 1 -- 172.21.15.107:0/2476912248 wait complete. 2023-11-24T08:11:22.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:22.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.861+0000 7f0689bbc640 1 Processor -- start 2023-11-24T08:11:22.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.862+0000 7f0689bbc640 1 -- start start 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0689bbc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f068415b260 0x7f068406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0689bbc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 0x7f068406c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0689bbc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068415ed20 con 0x7f068406e580 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0689bbc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068415b640 con 0x7f068415b260 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0689bbc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068415f8a0 con 0x7f06840b1250 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0688bba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f068415b260 0x7f068406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0688bba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f068415b260 0x7f068406e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41220/0 (socket says 172.21.15.107:41220) 2023-11-24T08:11:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.863+0000 7f0688bba640 1 -- 172.21.15.107:0/3935234564 learned_addr learned my addr 172.21.15.107:0/3935234564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f0688bba640 1 -- 172.21.15.107:0/3935234564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 msgr2=0x7f068406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f06893bb640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 0x7f068406c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f0688bba640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 0x7f068406c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f0688bba640 1 -- 172.21.15.107:0/3935234564 >> v1:172.21.15.107:6789/0 conn(0x7f068406e580 legacy=0x7f06840b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f0688bba640 1 -- 172.21.15.107:0/3935234564 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840abe50 con 0x7f068415b260 2023-11-24T08:11:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.864+0000 7f06893bb640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 0x7f068406c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:22.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.865+0000 7f0688bba640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068406e040 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc527e5626829d76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:22.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.866+0000 7f06837fe640 1 -- 172.21.15.107:0/3935234564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06740317c0 con 0x7f068415b260 2023-11-24T08:11:22.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.866+0000 7f06837fe640 1 -- 172.21.15.107:0/3935234564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674031920 con 0x7f068415b260 2023-11-24T08:11:22.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.866+0000 7f06837fe640 1 -- 172.21.15.107:0/3935234564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674031cb0 con 0x7f068415b260 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.866+0000 7f0689bbc640 1 -- 172.21.15.107:0/3935234564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 msgr2=0x7f068406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.866+0000 7f0689bbc640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068406e040 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b860 comp rx=0 tx=0).stop 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 -- 172.21.15.107:0/3935234564 shutdown_connections 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06840b1250 0x7f068406c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 --2- 172.21.15.107:0/3935234564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068406e040 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 -- 172.21.15.107:0/3935234564 >> 172.21.15.107:0/3935234564 conn(0x7f06840cbb40 msgr2=0x7f06840cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 -- 172.21.15.107:0/3935234564 shutdown_connections 2023-11-24T08:11:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.867+0000 7f0689bbc640 1 -- 172.21.15.107:0/3935234564 wait complete. 2023-11-24T08:11:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.869+0000 7f0689bbc640 1 Processor -- start 2023-11-24T08:11:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.869+0000 7f0689bbc640 1 -- start start 2023-11-24T08:11:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406e580 0x7f0684156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06840b1250 0x7f068415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068415c690 con 0x7f068415b260 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0684156ba0 con 0x7f06840b1250 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0689bbc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068415b7a0 con 0x7f068406e580 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f06893bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f06893bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41232/0 (socket says 172.21.15.107:41232) 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f06893bb640 1 -- 172.21.15.107:0/2909870098 learned_addr learned my addr 172.21.15.107:0/2909870098 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f0688bba640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406e580 0x7f0684156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.870+0000 7f0683fff640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06840b1250 0x7f068415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 -- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406e580 msgr2=0x7f0684156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406e580 0x7f0684156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 -- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06840b1250 msgr2=0x7f068415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06840b1250 0x7f068415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 -- 172.21.15.107:0/2909870098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740094d0 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06893bb640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f067c00ccf0 tx=0x7f067c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00f680 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c00f800 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.871+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00fbb0 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.872+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068415ba80 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.872+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068415c040 con 0x7f068415b260 2023-11-24T08:11:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.873+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068404e840 con 0x7f068415b260 2023-11-24T08:11:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.874+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f067c00b040 con 0x7f068415b260 2023-11-24T08:11:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.875+0000 7f06817fa640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 0x7f066406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.875+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f067c088b60 con 0x7f068415b260 2023-11-24T08:11:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.875+0000 7f0688bba640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 0x7f066406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.876+0000 7f0688bba640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 0x7f066406ddb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0674036000 tx=0x7f067403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:22.881+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f067c055490 con 0x7f068415b260 2023-11-24T08:11:23.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.145+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f068415f530 con 0x7f066406b8f0 2023-11-24T08:11:23.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.164+0000 7f06817fa640 1 -- 172.21.15.107:0/2909870098 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f0674031760 con 0x7f066406b8f0 2023-11-24T08:11:23.166 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol9055", 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap1401" 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:23.167 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 msgr2=0x7f066406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 0x7f066406ddb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0674036000 tx=0x7f067403d040 comp rx=0 tx=0).stop 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 msgr2=0x7f068415d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f067c00ccf0 tx=0x7f067c00a650 comp rx=0 tx=0).stop 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 shutdown_connections 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f066406b8f0 0x7f066406ddb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f068415b260 0x7f068415d640 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06840b1250 0x7f068415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 --2- 172.21.15.107:0/2909870098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406e580 0x7f0684156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 >> 172.21.15.107:0/2909870098 conn(0x7f06840cbb40 msgr2=0x7f068406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 shutdown_connections 2023-11-24T08:11:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:23.167+0000 7f0689bbc640 1 -- 172.21.15.107:0/2909870098 wait complete. 2023-11-24T08:11:24.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:24.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.411+0000 7f5840abb640 1 Processor -- start 2023-11-24T08:11:24.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.412+0000 7f5840abb640 1 -- start start 2023-11-24T08:11:24.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.413+0000 7f5840abb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f583c15b2f0 0x7f583c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.413+0000 7f5840abb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0b4eb0 con 0x7f583c15b6d0 2023-11-24T08:11:24.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.413+0000 7f5840abb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0b5620 con 0x7f583c06bc20 2023-11-24T08:11:24.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.413+0000 7f5840abb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c0cbf10 con 0x7f583c15b2f0 2023-11-24T08:11:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.414+0000 7f5833fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f583c15b6d0 0x7f583c06b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43512/0 (socket says 172.21.15.107:43512) 2023-11-24T08:11:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.414+0000 7f5833fff640 1 -- 172.21.15.107:0/347052022 learned_addr learned my addr 172.21.15.107:0/347052022 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.414+0000 7f583a59c640 1 --2- 172.21.15.107:0/347052022 >> v2:172.21.15.120:3300/0 conn(0x7f583c15b2f0 0x7f583c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.414+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518420423 0 0) 0x7f583c0b4eb0 con 0x7f583c15b6d0 2023-11-24T08:11:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.414+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5828003520 con 0x7f583c15b6d0 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385153694 0 0) 0x7f5828003520 con 0x7f583c15b6d0 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 >> v2:172.21.15.120:3300/0 conn(0x7f583c15b2f0 msgr2=0x7f583c0aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 --2- 172.21.15.107:0/347052022 >> v2:172.21.15.120:3300/0 conn(0x7f583c15b2f0 0x7f583c0aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 >> v1:172.21.15.120:6789/0 conn(0x7f583c06bc20 legacy=0x7f583c06d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0cc1f0 con 0x7f583c15b6d0 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5824001670 con 0x7f583c15b6d0 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 >> v1:172.21.15.107:6789/0 conn(0x7f583c15b6d0 legacy=0x7f583c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5828003970 0x7f5828003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 0x7f5828004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5828005180 0x7f582800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0cbf10 con 0x7f58280047c0 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582800c120 con 0x7f5828005180 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582800c2e0 con 0x7f5828003970 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 0x7f5828004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 -- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5828003970 msgr2=0x7f5828003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5828003970 0x7f5828003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 -- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5828005180 msgr2=0x7f582800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5828005180 0x7f582800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.415+0000 7f5833fff640 1 -- 172.21.15.107:0/347052022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5828003520 con 0x7f58280047c0 2023-11-24T08:11:24.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.416+0000 7f583ad9d640 1 -- 172.21.15.107:0/347052022 reap_dead start 2023-11-24T08:11:24.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5833fff640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 0x7f5828004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f582400c240 tx=0x7f58280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f369b105724a2b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:24.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5824005ad0 con 0x7f58280047c0 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5824005c30 con 0x7f58280047c0 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5839d9b640 1 -- 172.21.15.107:0/347052022 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582402caf0 con 0x7f58280047c0 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5840abb640 1 -- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 msgr2=0x7f5828004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.417+0000 7f5840abb640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 0x7f5828004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f582400c240 tx=0x7f58280057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.418+0000 7f5840abb640 1 -- 172.21.15.107:0/347052022 shutdown_connections 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.418+0000 7f5840abb640 1 --2- 172.21.15.107:0/347052022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58280047c0 0x7f5828004c40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.418+0000 7f5840abb640 1 -- 172.21.15.107:0/347052022 >> 172.21.15.107:0/347052022 conn(0x7f583c070fa0 msgr2=0x7f583c0713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.418+0000 7f5840abb640 1 -- 172.21.15.107:0/347052022 shutdown_connections 2023-11-24T08:11:24.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.418+0000 7f5840abb640 1 -- 172.21.15.107:0/347052022 wait complete. 2023-11-24T08:11:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.419+0000 7f5840abb640 1 Processor -- start 2023-11-24T08:11:24.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.420+0000 7f5840abb640 1 -- start start 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58280047c0 0x7f583c14d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 0x7f583c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c1f7280 con 0x7f583c1f6d30 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c1f7560 con 0x7f583c15bae0 2023-11-24T08:11:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5840abb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c1f7840 con 0x7f58280047c0 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41262/0 (socket says 172.21.15.107:41262) 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 -- 172.21.15.107:0/2365023450 learned_addr learned my addr 172.21.15.107:0/2365023450 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f583a59c640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58280047c0 0x7f583c14d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f583ad9d640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 0x7f583c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 -- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58280047c0 msgr2=0x7f583c14d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58280047c0 0x7f583c14d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 -- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 msgr2=0x7f583c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 0x7f583c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f5833fff640 1 -- 172.21.15.107:0/2365023450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c1f7b20 con 0x7f583c1f6d30 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.421+0000 7f583ad9d640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 0x7f583c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5833fff640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f582402c450 tx=0x7f582402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:24.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582403c040 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582402cd80 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5824007290 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c1596e0 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.422+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1550e0 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.423+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c1f7980 con 0x7f583c1f6d30 2023-11-24T08:11:24.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.424+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f5824032070 con 0x7f583c1f6d30 2023-11-24T08:11:24.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.425+0000 7f5832ffd640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 0x7f581c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:24.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.425+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58240a9780 con 0x7f583c1f6d30 2023-11-24T08:11:24.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.425+0000 7f583a59c640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 0x7f581c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:24.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.426+0000 7f583a59c640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 0x7f581c06ddb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f582c009ad0 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:24.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.430+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f583c1f7980 con 0x7f583c1f6d30 2023-11-24T08:11:24.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:24.657+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f583c0cd200 con 0x7f581c06b8f0 2023-11-24T08:11:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.339+0000 7f5832ffd640 1 -- 172.21.15.107:0/2365023450 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f582c009720 con 0x7f581c06b8f0 2023-11-24T08:11:25.341 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:25.342 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:25.342 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:11:25.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:25.342 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 msgr2=0x7f581c06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 0x7f581c06ddb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f582c009ad0 tx=0x7f582c008040 comp rx=0 tx=0).stop 2023-11-24T08:11:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 msgr2=0x7f583c154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f582402c450 tx=0x7f582402e070 comp rx=0 tx=0).stop 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 shutdown_connections 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f581c06b8f0 0x7f581c06ddb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f583c15bae0 0x7f583c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583c1f6d30 0x7f583c154ba0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 --2- 172.21.15.107:0/2365023450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58280047c0 0x7f583c14d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.342+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 >> 172.21.15.107:0/2365023450 conn(0x7f583c070fa0 msgr2=0x7f583c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.343+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 shutdown_connections 2023-11-24T08:11:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.343+0000 7f5840abb640 1 -- 172.21.15.107:0/2365023450 wait complete. 2023-11-24T08:11:25.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone6795 2023-11-24T08:11:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.578+0000 7fce57fdc640 1 Processor -- start 2023-11-24T08:11:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.579+0000 7fce57fdc640 1 -- start start 2023-11-24T08:11:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce57fdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 0x7fce5006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce57fdc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fce501605f0 0x7fce50160a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce57fdc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50160fc0 con 0x7fce500d2000 2023-11-24T08:11:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce57fdc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce500ce410 con 0x7fce501605f0 2023-11-24T08:11:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce57fdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500d06f0 con 0x7fce500d1c20 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce55577640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce500d2000 0x7fce500cfcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43528/0 (socket says 172.21.15.107:43528) 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.580+0000 7fce55577640 1 -- 172.21.15.107:0/3013743869 learned_addr learned my addr 172.21.15.107:0/3013743869 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce55d78640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 0x7fce5006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce56579640 1 --2- 172.21.15.107:0/3013743869 >> v2:172.21.15.120:3300/0 conn(0x7fce501605f0 0x7fce50160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788016262 0 0) 0x7fce3c001110 con 0x7fce500d2000 2023-11-24T08:11:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce40003520 con 0x7fce500d2000 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397521027 0 0) 0x7fce40003520 con 0x7fce500d2000 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 msgr2=0x7fce5006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 0x7fce5006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 >> v2:172.21.15.120:3300/0 conn(0x7fce501605f0 msgr2=0x7fce50160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 --2- 172.21.15.107:0/3013743869 >> v2:172.21.15.120:3300/0 conn(0x7fce501605f0 0x7fce50160a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500d09d0 con 0x7fce500d2000 2023-11-24T08:11:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fce3c001800 con 0x7fce500d2000 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 >> v1:172.21.15.107:6789/0 conn(0x7fce500d2000 legacy=0x7fce500cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 0x7fce40003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.581+0000 7fce54d76640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce40003ef0 0x7fce400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce54d76640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce40004d00 0x7fce40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500ce410 con 0x7fce40003ef0 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500d06f0 con 0x7fce40004d00 2023-11-24T08:11:25.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4000b980 con 0x7fce500d1c20 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce40003ef0 0x7fce400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 -- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 msgr2=0x7fce40003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500d1c20 0x7fce40003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 -- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce40004d00 msgr2=0x7fce40005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce40004d00 0x7fce40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.582+0000 7fce55577640 1 -- 172.21.15.107:0/3013743869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40003520 con 0x7fce40003ef0 2023-11-24T08:11:25.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.583+0000 7fce55577640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce40003ef0 0x7fce400047c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c240 tx=0x7fce400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cae8a0b7b3b3a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.583+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c005ad0 con 0x7fce40003ef0 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.583+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c005c30 con 0x7fce40003ef0 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.583+0000 7fce54d76640 1 -- 172.21.15.107:0/3013743869 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c02cae0 con 0x7fce40003ef0 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 -- 172.21.15.107:0/3013743869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce40003ef0 msgr2=0x7fce400047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3013743869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce40003ef0 0x7fce400047c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c240 tx=0x7fce400057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce56579640 1 -- 172.21.15.107:0/3013743869 reap_dead start 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 -- 172.21.15.107:0/3013743869 shutdown_connections 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 -- 172.21.15.107:0/3013743869 >> 172.21.15.107:0/3013743869 conn(0x7fce5008cea0 msgr2=0x7fce5009c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 -- 172.21.15.107:0/3013743869 shutdown_connections 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.584+0000 7fce57fdc640 1 -- 172.21.15.107:0/3013743869 wait complete. 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.586+0000 7fce57fdc640 1 Processor -- start 2023-11-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.586+0000 7fce57fdc640 1 -- start start 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 0x7fce500b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5009f170 0x7fce5009f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500b2e00 con 0x7fce500b39c0 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5009fb30 con 0x7fce5009f170 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce57fdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500b3e40 con 0x7fce40003ef0 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.587+0000 7fce55577640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41274/0 (socket says 172.21.15.107:41274) 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 -- 172.21.15.107:0/3372787366 learned_addr learned my addr 172.21.15.107:0/3372787366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55d78640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 0x7fce500b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 -- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 msgr2=0x7fce500b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce56579640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5009f170 0x7fce5009f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 0x7fce500b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 -- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5009f170 msgr2=0x7fce5009f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5009f170 0x7fce5009f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 -- 172.21.15.107:0/3372787366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500b2180 con 0x7fce500b39c0 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55d78640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 0x7fce500b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.588+0000 7fce55577640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c240 tx=0x7fce500ce550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.589+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c039070 con 0x7fce500b39c0 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.589+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c02dd60 con 0x7fce500b39c0 2023-11-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.589+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce500b2460 con 0x7fce500b39c0 2023-11-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.589+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c03c040 con 0x7fce500b39c0 2023-11-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.589+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce500b2a20 con 0x7fce500b39c0 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.590+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce500b2bb0 con 0x7fce500b39c0 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.591+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7fce3c032070 con 0x7fce500b39c0 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.592+0000 7fce4e7fc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 0x7fce3406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.592+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fce3c0a93f0 con 0x7fce500b39c0 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.592+0000 7fce55d78640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 0x7fce3406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.593+0000 7fce55d78640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 0x7fce3406ddb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fce44007340 tx=0x7fce4400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.597+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce3c075cc0 con 0x7fce500b39c0 2023-11-24T08:11:25.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.838+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7fce500d03d0 con 0x7fce3406b8f0 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.847+0000 7fce4e7fc640 1 -- 172.21.15.107:0/3372787366 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fce44011040 con 0x7fce3406b8f0 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:11:25.849 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.849+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 msgr2=0x7fce3406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.849+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 0x7fce3406ddb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fce44007340 tx=0x7fce4400b040 comp rx=0 tx=0).stop 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.849+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 msgr2=0x7fce5014f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.849+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c240 tx=0x7fce500ce550 comp rx=0 tx=0).stop 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 shutdown_connections 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce3406b8f0 0x7fce3406ddb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5009f170 0x7fce5009f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce500b39c0 0x7fce5014f9e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 --2- 172.21.15.107:0/3372787366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce40003ef0 0x7fce500b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:25.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 >> 172.21.15.107:0/3372787366 conn(0x7fce5008cea0 msgr2=0x7fce500b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:25.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 shutdown_connections 2023-11-24T08:11:25.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:25.850+0000 7fce57fdc640 1 -- 172.21.15.107:0/3372787366 wait complete. 2023-11-24T08:11:25.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol9055 subvol_snap1401 2023-11-24T08:11:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.078+0000 7f327b59e640 1 Processor -- start 2023-11-24T08:11:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.078+0000 7f327b59e640 1 -- start start 2023-11-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.079+0000 7f327b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c144510 0x7f327c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 0x7f327c0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327b59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c1448f0 con 0x7f327c0ccd00 2023-11-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327b59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c144d80 con 0x7f327c144510 2023-11-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c15f2d0 con 0x7f327c15edd0 2023-11-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c144510 0x7f327c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c144510 0x7f327c06d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41280/0 (socket says 172.21.15.107:41280) 2023-11-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 -- 172.21.15.107:0/1692781369 learned_addr learned my addr 172.21.15.107:0/1692781369 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f3279d9b640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 0x7f327c0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 -- 172.21.15.107:0/1692781369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 msgr2=0x7f327c0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 0x7f327c0b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 -- 172.21.15.107:0/1692781369 >> v1:172.21.15.120:6789/0 conn(0x7f327c0ccd00 legacy=0x7f327c0b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f327a59c640 1 -- 172.21.15.107:0/1692781369 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c15f5b0 con 0x7f327c144510 2023-11-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.080+0000 7f3279d9b640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 0x7f327c0b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327a59c640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c144510 0x7f327c06d7b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f326c009870 tx=0x7f326c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58567489c6cab6e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327959a640 1 -- 172.21.15.107:0/1692781369 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0317c0 con 0x7f327c144510 2023-11-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327959a640 1 -- 172.21.15.107:0/1692781369 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c031920 con 0x7f327c144510 2023-11-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327959a640 1 -- 172.21.15.107:0/1692781369 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c031cb0 con 0x7f327c144510 2023-11-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327b59e640 1 -- 172.21.15.107:0/1692781369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c144510 msgr2=0x7f327c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.082+0000 7f327b59e640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c144510 0x7f327c06d7b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f326c009870 tx=0x7f326c02b860 comp rx=0 tx=0).stop 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 -- 172.21.15.107:0/1692781369 shutdown_connections 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c15edd0 0x7f327c0b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 --2- 172.21.15.107:0/1692781369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c144510 0x7f327c06d7b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 -- 172.21.15.107:0/1692781369 >> 172.21.15.107:0/1692781369 conn(0x7f327c0cb8a0 msgr2=0x7f327c0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 -- 172.21.15.107:0/1692781369 shutdown_connections 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.083+0000 7f327b59e640 1 -- 172.21.15.107:0/1692781369 wait complete. 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.084+0000 7f327b59e640 1 Processor -- start 2023-11-24T08:11:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.085+0000 7f327b59e640 1 -- start start 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.085+0000 7f327b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 0x7f327c15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.085+0000 7f327b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c144510 0x7f327c13a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c15bc20 con 0x7f327c15edd0 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327b59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c15be50 con 0x7f327c0ccd00 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c15c130 con 0x7f327c144510 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41296/0 (socket says 172.21.15.107:41296) 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 -- 172.21.15.107:0/2809588097 learned_addr learned my addr 172.21.15.107:0/2809588097 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f3279d9b640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c144510 0x7f327c13a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327a59c640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 0x7f327c15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 -- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c144510 msgr2=0x7f327c13a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c144510 0x7f327c13a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 -- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 msgr2=0x7f327c15b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 0x7f327c15b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327ad9d640 1 -- 172.21.15.107:0/2809588097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0094d0 con 0x7f327c15edd0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.086+0000 7f327a59c640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 0x7f327c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f327ad9d640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f327400c780 tx=0x7f327400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274004b60 con 0x7f327c15edd0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327400ce90 con 0x7f327c15edd0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400a850 con 0x7f327c15edd0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c15c410 con 0x7f327c15edd0 2023-11-24T08:11:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.087+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c15c9d0 con 0x7f327c15edd0 2023-11-24T08:11:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.088+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c04e840 con 0x7f327c15edd0 2023-11-24T08:11:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.089+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f327400a9b0 con 0x7f327c15edd0 2023-11-24T08:11:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.090+0000 7f326affd640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 0x7f325c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.090+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3274088410 con 0x7f327c15edd0 2023-11-24T08:11:26.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.090+0000 7f327a59c640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 0x7f325c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.091+0000 7f327a59c640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 0x7f325c06ddb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f326c036000 tx=0x7f326c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.094+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3274054c50 con 0x7f327c15edd0 2023-11-24T08:11:26.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.331+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol9055", "snap_name": "subvol_snap1401", "target": ["mon-mgr", ""]}) v1 -- 0x7f327c0b3c30 con 0x7f325c06b8f0 2023-11-24T08:11:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.395+0000 7f326affd640 1 -- 172.21.15.107:0/2809588097 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f326c031b90 con 0x7f325c06b8f0 2023-11-24T08:11:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 msgr2=0x7f325c06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 0x7f325c06ddb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f326c036000 tx=0x7f326c03d040 comp rx=0 tx=0).stop 2023-11-24T08:11:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 msgr2=0x7f327c1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f327400c780 tx=0x7f327400cc40 comp rx=0 tx=0).stop 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 shutdown_connections 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f325c06b8f0 0x7f325c06ddb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c15edd0 0x7f327c1414b0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c144510 0x7f327c13a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 --2- 172.21.15.107:0/2809588097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c0ccd00 0x7f327c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.398+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 >> 172.21.15.107:0/2809588097 conn(0x7f327c0cb8a0 msgr2=0x7f327c0b3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.399+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 shutdown_connections 2023-11-24T08:11:26.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.399+0000 7f327b59e640 1 -- 172.21.15.107:0/2809588097 wait complete. 2023-11-24T08:11:26.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol9055 2023-11-24T08:11:26.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.621+0000 7f2bcdcbb640 1 Processor -- start 2023-11-24T08:11:26.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.621+0000 7f2bcdcbb640 1 -- start start 2023-11-24T08:11:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.622+0000 7f2bcdcbb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 0x7f2bc80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.622+0000 7f2bcdcbb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2bc815ded0 0x7f2bc815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.622+0000 7f2bcdcbb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc806d430 con 0x7f2bc815a1c0 2023-11-24T08:11:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.622+0000 7f2bcdcbb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc815ec50 con 0x7f2bc815ded0 2023-11-24T08:11:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.622+0000 7f2bcdcbb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc806cb00 con 0x7f2bc80b0b40 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2bc815ded0 0x7f2bc815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2bc815ded0 0x7f2bc815e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41306/0 (socket says 172.21.15.107:41306) 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 -- 172.21.15.107:0/1580998478 learned_addr learned my addr 172.21.15.107:0/1580998478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bbffff640 1 --2- 172.21.15.107:0/1580998478 >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 0x7f2bc80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 -- 172.21.15.107:0/1580998478 >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 msgr2=0x7f2bc80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 --2- 172.21.15.107:0/1580998478 >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 0x7f2bc80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 -- 172.21.15.107:0/1580998478 >> v1:172.21.15.120:6789/0 conn(0x7f2bc815a1c0 legacy=0x7f2bc806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bcd4ba640 1 -- 172.21.15.107:0/1580998478 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc80b14e0 con 0x7f2bc815ded0 2023-11-24T08:11:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.623+0000 7f2bbffff640 1 --2- 172.21.15.107:0/1580998478 >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 0x7f2bc80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.624+0000 7f2bcd4ba640 1 --2- 172.21.15.107:0/1580998478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc815ded0 0x7f2bc815e360 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2bc80cee20 tx=0x7f2bc002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26322c1737112e20 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.624+0000 7f2bbf7fe640 1 -- 172.21.15.107:0/1580998478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc002ca80 con 0x7f2bc815ded0 2023-11-24T08:11:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bbf7fe640 1 -- 172.21.15.107:0/1580998478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc002cbe0 con 0x7f2bc815ded0 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bbf7fe640 1 -- 172.21.15.107:0/1580998478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00343f0 con 0x7f2bc815ded0 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1580998478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc815ded0 msgr2=0x7f2bc815e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1580998478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc815ded0 0x7f2bc815e360 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2bc80cee20 tx=0x7f2bc002bee0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1580998478 shutdown_connections 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1580998478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc815ded0 0x7f2bc815e360 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1580998478 >> v2:172.21.15.120:3300/0 conn(0x7f2bc80b0b40 0x7f2bc80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1580998478 >> 172.21.15.107:0/1580998478 conn(0x7f2bc80cbb40 msgr2=0x7f2bc80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1580998478 shutdown_connections 2023-11-24T08:11:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.625+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1580998478 wait complete. 2023-11-24T08:11:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.627+0000 7f2bcdcbb640 1 Processor -- start 2023-11-24T08:11:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.627+0000 7f2bcdcbb640 1 -- start start 2023-11-24T08:11:26.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bc815a1c0 0x7f2bc80b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bc815ded0 0x7f2bc80b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c6d90 con 0x7f2bc80b0b40 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c7030 con 0x7f2bc815a1c0 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcdcbb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c7310 con 0x7f2bc815ded0 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcccb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcccb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41318/0 (socket says 172.21.15.107:41318) 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.628+0000 7f2bcccb9640 1 -- 172.21.15.107:0/1151433409 learned_addr learned my addr 172.21.15.107:0/1151433409 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 -- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bc815ded0 msgr2=0x7f2bc80b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bc815ded0 0x7f2bc80b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 -- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bc815a1c0 msgr2=0x7f2bc80b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bc815a1c0 0x7f2bc80b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 -- 172.21.15.107:0/1151433409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0009b30 con 0x7f2bc80b0b40 2023-11-24T08:11:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcccb9640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c410 tx=0x7f2bc400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40035f0 con 0x7f2bc80b0b40 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4003750 con 0x7f2bc80b0b40 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc815ea20 con 0x7f2bc80b0b40 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4003b40 con 0x7f2bc80b0b40 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.629+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc80c78d0 con 0x7f2bc80b0b40 2023-11-24T08:11:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.630+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc804e840 con 0x7f2bc80b0b40 2023-11-24T08:11:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.631+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f2bc4003ca0 con 0x7f2bc80b0b40 2023-11-24T08:11:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.632+0000 7f2bbd7fa640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 0x7f2bb406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.632+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2bc4087880 con 0x7f2bc80b0b40 2023-11-24T08:11:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.632+0000 7f2bbffff640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 0x7f2bb406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.633+0000 7f2bbffff640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 0x7f2bb406ddb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8006fd0 tx=0x7f2bb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:26.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.637+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bc40541b0 con 0x7f2bc80b0b40 2023-11-24T08:11:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.854+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol9055", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc806d0c0 con 0x7f2bb406b8f0 2023-11-24T08:11:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.893+0000 7f2bbd7fa640 1 -- 172.21.15.107:0/1151433409 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2bb800ae40 con 0x7f2bb406b8f0 2023-11-24T08:11:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 msgr2=0x7f2bb406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 0x7f2bb406ddb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8006fd0 tx=0x7f2bb8008040 comp rx=0 tx=0).stop 2023-11-24T08:11:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 msgr2=0x7f2bc80af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c410 tx=0x7f2bc400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 shutdown_connections 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bb406b8f0 0x7f2bb406ddb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bc815ded0 0x7f2bc80b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bc815a1c0 0x7f2bc80b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 --2- 172.21.15.107:0/1151433409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bc80b0b40 0x7f2bc80af4b0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 >> 172.21.15.107:0/1151433409 conn(0x7f2bc80cbb40 msgr2=0x7f2bc80ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 shutdown_connections 2023-11-24T08:11:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:26.896+0000 7f2bcdcbb640 1 -- 172.21.15.107:0/1151433409 wait complete. 2023-11-24T08:11:26.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:26.913+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:26.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:26.913+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:26.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:26.913+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:26.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:26.913+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:26.915 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:26.913+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:26.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone6795 2023-11-24T08:11:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.121+0000 7f0889367640 1 Processor -- start 2023-11-24T08:11:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.121+0000 7f0889367640 1 -- start start 2023-11-24T08:11:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0884146270 0x7f08840b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08841467f0 0x7f08840b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f08840b3520 0x7f0884160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840b39d0 con 0x7f08841467f0 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088406bae0 con 0x7f08840b3520 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f0889367640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088406c940 con 0x7f0884146270 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f08837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08841467f0 0x7f08840b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f08837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08841467f0 0x7f08840b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41350/0 (socket says 172.21.15.107:41350) 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.123+0000 7f08837fe640 1 -- 172.21.15.107:0/2279444400 learned_addr learned my addr 172.21.15.107:0/2279444400 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f0888b66640 1 --2- 172.21.15.107:0/2279444400 >> v2:172.21.15.120:3300/0 conn(0x7f08840b3520 0x7f0884160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f0883fff640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0884146270 0x7f08840b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f08837fe640 1 -- 172.21.15.107:0/2279444400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0884146270 msgr2=0x7f08840b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f08837fe640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0884146270 0x7f08840b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f08837fe640 1 -- 172.21.15.107:0/2279444400 >> v2:172.21.15.120:3300/0 conn(0x7f08840b3520 msgr2=0x7f0884160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f08837fe640 1 --2- 172.21.15.107:0/2279444400 >> v2:172.21.15.120:3300/0 conn(0x7f08840b3520 0x7f0884160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.124+0000 7f08837fe640 1 -- 172.21.15.107:0/2279444400 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840b4f80 con 0x7f08841467f0 2023-11-24T08:11:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.125+0000 7f08837fe640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08841467f0 0x7f08840b2fe0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f08840b3b40 tx=0x7f087802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56eb271af4fd7e8f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.125+0000 7f0882ffd640 1 -- 172.21.15.107:0/2279444400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878031870 con 0x7f08841467f0 2023-11-24T08:11:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.125+0000 7f0882ffd640 1 -- 172.21.15.107:0/2279444400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08780319d0 con 0x7f08841467f0 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.125+0000 7f0882ffd640 1 -- 172.21.15.107:0/2279444400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878031d60 con 0x7f08841467f0 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 -- 172.21.15.107:0/2279444400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08841467f0 msgr2=0x7f08840b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08841467f0 0x7f08840b2fe0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f08840b3b40 tx=0x7f087802b910 comp rx=0 tx=0).stop 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 -- 172.21.15.107:0/2279444400 shutdown_connections 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 --2- 172.21.15.107:0/2279444400 >> v2:172.21.15.120:3300/0 conn(0x7f08840b3520 0x7f0884160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08841467f0 0x7f08840b2fe0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 --2- 172.21.15.107:0/2279444400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0884146270 0x7f08840b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 -- 172.21.15.107:0/2279444400 >> 172.21.15.107:0/2279444400 conn(0x7f0884061100 msgr2=0x7f0884061510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 -- 172.21.15.107:0/2279444400 shutdown_connections 2023-11-24T08:11:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.126+0000 7f0889367640 1 -- 172.21.15.107:0/2279444400 wait complete. 2023-11-24T08:11:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.128+0000 7f0889367640 1 Processor -- start 2023-11-24T08:11:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.128+0000 7f0889367640 1 -- start start 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08840b3520 0x7f08840cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08841467f0 0x7f08841f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088415f590 con 0x7f0884146270 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840cc300 con 0x7f08840b3520 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f0889367640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08840cad20 con 0x7f08841467f0 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.129+0000 7f08837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41352/0 (socket says 172.21.15.107:41352) 2023-11-24T08:11:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 -- 172.21.15.107:0/2347777216 learned_addr learned my addr 172.21.15.107:0/2347777216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 -- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08841467f0 msgr2=0x7f08841f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08841467f0 0x7f08841f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 -- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08840b3520 msgr2=0x7f08840cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08840b3520 0x7f08840cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 -- 172.21.15.107:0/2347777216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780094d0 con 0x7f0884146270 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f08837fe640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f08780097c0 tx=0x7f08780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087803ac60 con 0x7f0884146270 2023-11-24T08:11:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087803adc0 con 0x7f0884146270 2023-11-24T08:11:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.130+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878044610 con 0x7f0884146270 2023-11-24T08:11:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.131+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08840cb000 con 0x7f0884146270 2023-11-24T08:11:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.131+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08840cb5c0 con 0x7f0884146270 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.132+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088404e840 con 0x7f0884146270 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.133+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 90200+0+0 (secure 0 0 0) 0x7f087804d450 con 0x7f0884146270 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.134+0000 7f0880ff9640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 0x7f087006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.135+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f087802d040 con 0x7f0884146270 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.135+0000 7f0883fff640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 0x7f087006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.135+0000 7f0883fff640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 0x7f087006ddb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f08740098f0 tx=0x7f0874008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.141+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0878077080 con 0x7f0884146270 2023-11-24T08:11:27.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.405+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone6795", "target": ["mon-mgr", ""]}) v1 -- 0x7f088406c010 con 0x7f087006b8f0 2023-11-24T08:11:27.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.428+0000 7f0880ff9640 1 -- 172.21.15.107:0/2347777216 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0874009e70 con 0x7f087006b8f0 2023-11-24T08:11:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 msgr2=0x7f087006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 0x7f087006ddb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f08740098f0 tx=0x7f0874008040 comp rx=0 tx=0).stop 2023-11-24T08:11:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 msgr2=0x7f08840ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f08780097c0 tx=0x7f08780046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 shutdown_connections 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f087006b8f0 0x7f087006ddb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08841467f0 0x7f08841f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884146270 0x7f08840ca3b0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 --2- 172.21.15.107:0/2347777216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08840b3520 0x7f08840cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 >> 172.21.15.107:0/2347777216 conn(0x7f0884061100 msgr2=0x7f08841427c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 shutdown_connections 2023-11-24T08:11:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:27.431+0000 7f0889367640 1 -- 172.21.15.107:0/2347777216 wait complete. 2023-11-24T08:11:27.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:27.453 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:11:27.660 INFO:teuthology.orchestra.run.smithi107.stdout:4 2023-11-24T08:11:27.672 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:27.671+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:27.673 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:27.671+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:27.673 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:27.671+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:27.673 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:27.671+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:27.673 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:11:27.671+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:11:32.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:32.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:11:32.726 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:11:32.727 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:11:32.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:11:32.841 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:11:32.967 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:11:32.967 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:32.968 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:33.027 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:33.027 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:33.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.293+0000 7f60945d5640 1 Processor -- start 2023-11-24T08:11:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.293+0000 7f60945d5640 1 -- start start 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f60945d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f608c157c70 0x7f608c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f60945d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f608c0cf450 0x7f608c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f60945d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c0ce0b0 con 0x7f608c15dc80 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f60945d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608c0cfe40 con 0x7f608c157c70 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f60945d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608c161960 con 0x7f608c0cf450 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f6092371640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f608c157c70 0x7f608c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f6092371640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f608c157c70 0x7f608c161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34422/0 (socket says 172.21.15.107:34422) 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.294+0000 7f6092371640 1 -- 172.21.15.107:0/289834510 learned_addr learned my addr 172.21.15.107:0/289834510 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.295+0000 7f6091b70640 1 --2- 172.21.15.107:0/289834510 >> v2:172.21.15.120:3300/0 conn(0x7f608c0cf450 0x7f608c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.295+0000 7f6092371640 1 -- 172.21.15.107:0/289834510 >> v2:172.21.15.120:3300/0 conn(0x7f608c0cf450 msgr2=0x7f608c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.295+0000 7f6092371640 1 --2- 172.21.15.107:0/289834510 >> v2:172.21.15.120:3300/0 conn(0x7f608c0cf450 0x7f608c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.295+0000 7f6092371640 1 -- 172.21.15.107:0/289834510 >> v1:172.21.15.120:6789/0 conn(0x7f608c15dc80 legacy=0x7f608c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:33.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.295+0000 7f6092371640 1 -- 172.21.15.107:0/289834510 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c15e8d0 con 0x7f608c157c70 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.296+0000 7f6092371640 1 --2- 172.21.15.107:0/289834510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c157c70 0x7f608c161df0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f608c0b44b0 tx=0x7f608802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a846e9baee28c6b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.296+0000 7f609136f640 1 -- 172.21.15.107:0/289834510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880317e0 con 0x7f608c157c70 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.296+0000 7f609136f640 1 -- 172.21.15.107:0/289834510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088031940 con 0x7f608c157c70 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.296+0000 7f609136f640 1 -- 172.21.15.107:0/289834510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088031cd0 con 0x7f608c157c70 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 -- 172.21.15.107:0/289834510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c157c70 msgr2=0x7f608c161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 --2- 172.21.15.107:0/289834510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c157c70 0x7f608c161df0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f608c0b44b0 tx=0x7f608802b880 comp rx=0 tx=0).stop 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 -- 172.21.15.107:0/289834510 shutdown_connections 2023-11-24T08:11:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 --2- 172.21.15.107:0/289834510 >> v2:172.21.15.120:3300/0 conn(0x7f608c0cf450 0x7f608c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 --2- 172.21.15.107:0/289834510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c157c70 0x7f608c161df0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 -- 172.21.15.107:0/289834510 >> 172.21.15.107:0/289834510 conn(0x7f608c06a920 msgr2=0x7f608c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:33.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 -- 172.21.15.107:0/289834510 shutdown_connections 2023-11-24T08:11:33.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.297+0000 7f60945d5640 1 -- 172.21.15.107:0/289834510 wait complete. 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.299+0000 7f60945d5640 1 Processor -- start 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.299+0000 7f60945d5640 1 -- start start 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 0x7f608c14ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f608c157c70 0x7f608c1453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c145900 con 0x7f608c15dc80 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0aed30 con 0x7f608c0cf450 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f60945d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c152620 con 0x7f608c157c70 2023-11-24T08:11:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f6092b72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f6092b72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34428/0 (socket says 172.21.15.107:34428) 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.300+0000 7f6092b72640 1 -- 172.21.15.107:0/1719230174 learned_addr learned my addr 172.21.15.107:0/1719230174 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092371640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 0x7f608c14ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6091b70640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f608c157c70 0x7f608c1453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 -- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f608c157c70 msgr2=0x7f608c1453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f608c157c70 0x7f608c1453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 -- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 msgr2=0x7f608c14ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 0x7f608c14ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 -- 172.21.15.107:0/1719230174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880094d0 con 0x7f608c15dc80 2023-11-24T08:11:33.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092371640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 0x7f608c14ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6092b72640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f608400c780 tx=0x7f608400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084003d80 con 0x7f608c15dc80 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608400ce30 con 0x7f608c15dc80 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.301+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084005290 con 0x7f608c15dc80 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.302+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c152900 con 0x7f608c15dc80 2023-11-24T08:11:33.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.302+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c1522f0 con 0x7f608c15dc80 2023-11-24T08:11:33.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.303+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c152e20 con 0x7f608c15dc80 2023-11-24T08:11:33.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.304+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f60840053f0 con 0x7f608c15dc80 2023-11-24T08:11:33.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.305+0000 7f6082ffd640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 0x7f607006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.305+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6084087600 con 0x7f608c15dc80 2023-11-24T08:11:33.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.305+0000 7f6092371640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 0x7f607006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.306+0000 7f6092371640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 0x7f607006ded0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6088036000 tx=0x7f608803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.309+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f608c152e20 con 0x7f608c15dc80 2023-11-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.499+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f608c152e20 con 0x7f608c15dc80 2023-11-24T08:11:33.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.501+0000 7f6082ffd640 1 -- 172.21.15.107:0/1719230174 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16776 (secure 0 0 0) 0x7f608c152e20 con 0x7f608c15dc80 2023-11-24T08:11:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:33.503 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:31.657077+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":81,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.1c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 msgr2=0x7f607006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 0x7f607006ded0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6088036000 tx=0x7f608803d040 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 msgr2=0x7f608c0ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f608400c780 tx=0x7f608400cc40 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 shutdown_connections 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f607006ba10 0x7f607006ded0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608c15dc80 0x7f608c0ac180 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f608c157c70 0x7f608c1453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 --2- 172.21.15.107:0/1719230174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f608c0cf450 0x7f608c14ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.503+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 >> 172.21.15.107:0/1719230174 conn(0x7f608c06a920 msgr2=0x7f608c0afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.504+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 shutdown_connections 2023-11-24T08:11:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.504+0000 7f60945d5640 1 -- 172.21.15.107:0/1719230174 wait complete. 2023-11-24T08:11:33.525 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:10:24.450447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:10:25.621972+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 81, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:11:33.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:11:33.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.737+0000 7f251d590640 1 Processor -- start 2023-11-24T08:11:33.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.738+0000 7f251d590640 1 -- start start 2023-11-24T08:11:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.739+0000 7f251d590640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25180c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f251d590640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251815eea0 con 0x7f25180c57e0 2023-11-24T08:11:33.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f251d590640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25180abf30 con 0x7f251815b440 2023-11-24T08:11:33.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f251d590640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251815b820 con 0x7f25180cc0f0 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f25177fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25180c57e0 0x7f251806c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47108/0 (socket says 172.21.15.107:47108) 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f25177fe640 1 -- 172.21.15.107:0/1034977116 learned_addr learned my addr 172.21.15.107:0/1034977116 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.740+0000 7f250ffff640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25180c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594278408 0 0) 0x7f251815eea0 con 0x7f25180c57e0 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2504003520 con 0x7f25180c57e0 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3352896497 0 0) 0x7f2504003520 con 0x7f25180c57e0 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 msgr2=0x7f25180c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25180c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 >> v1:172.21.15.120:6789/0 conn(0x7f251815b440 legacy=0x7f25180cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251815f0f0 con 0x7f25180c57e0 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f25100019f0 con 0x7f25180c57e0 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 >> v1:172.21.15.107:6789/0 conn(0x7f25180c57e0 legacy=0x7f251806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504003f20 0x7f25040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2504004d00 0x7f2504005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251815b820 con 0x7f2504004d00 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250400bb10 con 0x7f2504003f20 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.741+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250400bcd0 con 0x7f25180cc0f0 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2504004d00 0x7f2504005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f2516ffd640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 -- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 msgr2=0x7f25040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 -- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504003f20 msgr2=0x7f25040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504003f20 0x7f25040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f25177fe640 1 -- 172.21.15.107:0/1034977116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504003520 con 0x7f2504004d00 2023-11-24T08:11:33.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.742+0000 7f2516ffd640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25180cc0f0 0x7f25040039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.743+0000 7f25177fe640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2504004d00 0x7f2504005180 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f251000a830 tx=0x7f25040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f167a88053344067 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.743+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25100072d0 con 0x7f2504004d00 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.743+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251002f070 con 0x7f2504004d00 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.743+0000 7f25167fc640 1 -- 172.21.15.107:0/1034977116 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25100076b0 con 0x7f2504004d00 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 -- 172.21.15.107:0/1034977116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2504004d00 msgr2=0x7f2504005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 --2- 172.21.15.107:0/1034977116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2504004d00 0x7f2504005180 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f251000a830 tx=0x7f25040057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f25177fe640 1 -- 172.21.15.107:0/1034977116 reap_dead start 2023-11-24T08:11:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 -- 172.21.15.107:0/1034977116 shutdown_connections 2023-11-24T08:11:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 -- 172.21.15.107:0/1034977116 >> 172.21.15.107:0/1034977116 conn(0x7f25180b0c50 msgr2=0x7f25180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 -- 172.21.15.107:0/1034977116 shutdown_connections 2023-11-24T08:11:33.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.744+0000 7f251d590640 1 -- 172.21.15.107:0/1034977116 wait complete. 2023-11-24T08:11:33.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.746+0000 7f251d590640 1 Processor -- start 2023-11-24T08:11:33.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.746+0000 7f251d590640 1 -- start start 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 0x7f25180c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 0x7f25180c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25180c85f0 con 0x7f2518150bc0 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2518156400 con 0x7f2504004d00 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f251d590640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25180c8f50 con 0x7f2518155f80 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f250ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f250ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34456/0 (socket says 172.21.15.107:34456) 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.747+0000 7f250ffff640 1 -- 172.21.15.107:0/4150764441 learned_addr learned my addr 172.21.15.107:0/4150764441 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f25177fe640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 0x7f25180c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f2516ffd640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 0x7f25180c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 -- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 msgr2=0x7f25180c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 0x7f25180c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 -- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 msgr2=0x7f25180c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 0x7f25180c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 -- 172.21.15.107:0/4150764441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25180c6fc0 con 0x7f2518150bc0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f25177fe640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 0x7f25180c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f2516ffd640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 0x7f25180c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250ffff640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f250000e7c0 tx=0x7f250000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000eed0 con 0x7f2518150bc0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000c8c0 con 0x7f2518150bc0 2023-11-24T08:11:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25180c7c50 con 0x7f2518150bc0 2023-11-24T08:11:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000cc70 con 0x7f2518150bc0 2023-11-24T08:11:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.748+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25180c8210 con 0x7f2518150bc0 2023-11-24T08:11:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.749+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25180c8730 con 0x7f2518150bc0 2023-11-24T08:11:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.751+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f2500013470 con 0x7f2518150bc0 2023-11-24T08:11:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.751+0000 7f250f7fe640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 0x7f24f806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.752+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f25000882e0 con 0x7f2518150bc0 2023-11-24T08:11:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.752+0000 7f2516ffd640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 0x7f24f806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.753+0000 7f2516ffd640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 0x7f24f806df00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f25080016f0 tx=0x7f2508009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:33.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.756+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25180c8730 con 0x7f2518150bc0 2023-11-24T08:11:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.946+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25180c8730 con 0x7f2518150bc0 2023-11-24T08:11:33.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.947+0000 7f250f7fe640 1 -- 172.21.15.107:0/4150764441 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25180c8730 con 0x7f2518150bc0 2023-11-24T08:11:33.949 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 msgr2=0x7f24f806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 0x7f24f806df00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f25080016f0 tx=0x7f2508009040 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 msgr2=0x7f2518155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f250000e7c0 tx=0x7f250000ec80 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 shutdown_connections 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24f806ba40 0x7f24f806df00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2518155f80 0x7f25180c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2518150bc0 0x7f2518155a40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 --2- 172.21.15.107:0/4150764441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2504004d00 0x7f25180c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.949+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 >> 172.21.15.107:0/4150764441 conn(0x7f25180b0c50 msgr2=0x7f25180cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:33.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.950+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 shutdown_connections 2023-11-24T08:11:33.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:33.950+0000 7f251d590640 1 -- 172.21.15.107:0/4150764441 wait complete. 2023-11-24T08:11:33.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:11:34.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.168+0000 7f1200941640 1 Processor -- start 2023-11-24T08:11:34.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.168+0000 7f1200941640 1 -- start start 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f1200941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 0x7f11fc0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f1200941640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11fc0cd110 0x7f11fc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f1200941640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0cd510 con 0x7f11fc146eb0 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f1200941640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc161a10 con 0x7f11fc0cd110 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f1200941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc161cf0 con 0x7f11fc06e160 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11fc0cd110 0x7f11fc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fa59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f11fc146eb0 0x7f11fc161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47120/0 (socket says 172.21.15.107:47120) 2023-11-24T08:11:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fa59c640 1 -- 172.21.15.107:0/615871263 learned_addr learned my addr 172.21.15.107:0/615871263 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11f9d9b640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 0x7f11fc0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fad9d640 1 -- 172.21.15.107:0/615871263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 msgr2=0x7f11fc0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fad9d640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 0x7f11fc0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fad9d640 1 -- 172.21.15.107:0/615871263 >> v1:172.21.15.107:6789/0 conn(0x7f11fc146eb0 legacy=0x7f11fc161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11f959a640 1 -- 172.21.15.107:0/615871263 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871722240 0 0) 0x7f11fc0cd510 con 0x7f11fc146eb0 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.170+0000 7f11fad9d640 1 -- 172.21.15.107:0/615871263 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc161fd0 con 0x7f11fc0cd110 2023-11-24T08:11:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.171+0000 7f11f9d9b640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 0x7f11fc0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.172+0000 7f11fad9d640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc0cd110 0x7f11fc070e40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ca27d9bcd5851a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.172+0000 7f11f959a640 1 -- 172.21.15.107:0/615871263 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f402cc50 con 0x7f11fc0cd110 2023-11-24T08:11:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.172+0000 7f11f959a640 1 -- 172.21.15.107:0/615871263 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f402cdb0 con 0x7f11fc0cd110 2023-11-24T08:11:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.172+0000 7f11f959a640 1 -- 172.21.15.107:0/615871263 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4034660 con 0x7f11fc0cd110 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 -- 172.21.15.107:0/615871263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc0cd110 msgr2=0x7f11fc070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc0cd110 0x7f11fc070e40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f402bea0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 -- 172.21.15.107:0/615871263 shutdown_connections 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc0cd110 0x7f11fc070e40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 --2- 172.21.15.107:0/615871263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc06e160 0x7f11fc0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 -- 172.21.15.107:0/615871263 >> 172.21.15.107:0/615871263 conn(0x7f11fc0b0c50 msgr2=0x7f11fc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 -- 172.21.15.107:0/615871263 shutdown_connections 2023-11-24T08:11:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.173+0000 7f1200941640 1 -- 172.21.15.107:0/615871263 wait complete. 2023-11-24T08:11:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.175+0000 7f1200941640 1 Processor -- start 2023-11-24T08:11:34.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.175+0000 7f1200941640 1 -- start start 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11fc06e160 0x7f11fc142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc0cd110 0x7f11fc15d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0add80 con 0x7f11fc146eb0 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc15ced0 con 0x7f11fc06e160 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f1200941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc141aa0 con 0x7f11fc0cd110 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f11fad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f11fad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34464/0 (socket says 172.21.15.107:34464) 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f11fad9d640 1 -- 172.21.15.107:0/1999977692 learned_addr learned my addr 172.21.15.107:0/1999977692 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.176+0000 7f11fa59c640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11fc06e160 0x7f11fc142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11f9d9b640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc0cd110 0x7f11fc15d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 -- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc0cd110 msgr2=0x7f11fc15d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc0cd110 0x7f11fc15d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 -- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11fc06e160 msgr2=0x7f11fc142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11fc06e160 0x7f11fc142f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 -- 172.21.15.107:0/1999977692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4009a70 con 0x7f11fc146eb0 2023-11-24T08:11:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11fad9d640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4040650 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f40407d0 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4040b80 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc141d80 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.177+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc142340 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.179+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc04e8b0 con 0x7f11fc146eb0 2023-11-24T08:11:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.179+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f11f4040ce0 con 0x7f11fc146eb0 2023-11-24T08:11:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.180+0000 7f11eaffd640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 0x7f11dc06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.180+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f11f40aadb0 con 0x7f11fc146eb0 2023-11-24T08:11:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.181+0000 7f11fa59c640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 0x7f11dc06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.181+0000 7f11fa59c640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 0x7f11dc06df00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f11ec000f60 tx=0x7f11ec007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.184+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11f4077a30 con 0x7f11fc146eb0 2023-11-24T08:11:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.380+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11fc06ded0 con 0x7f11fc146eb0 2023-11-24T08:11:34.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.381+0000 7f11eaffd640 1 -- 172.21.15.107:0/1999977692 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11f407b050 con 0x7f11fc146eb0 2023-11-24T08:11:34.383 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:11:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.383+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 msgr2=0x7f11dc06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.383+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 0x7f11dc06df00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f11ec000f60 tx=0x7f11ec007c00 comp rx=0 tx=0).stop 2023-11-24T08:11:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.383+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 msgr2=0x7f11fc0ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.383+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f4004a40 comp rx=0 tx=0).stop 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 shutdown_connections 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f11dc06ba40 0x7f11dc06df00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11fc146eb0 0x7f11fc0ad840 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11fc0cd110 0x7f11fc15d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 --2- 172.21.15.107:0/1999977692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11fc06e160 0x7f11fc142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 >> 172.21.15.107:0/1999977692 conn(0x7f11fc0b0c50 msgr2=0x7f11fc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 shutdown_connections 2023-11-24T08:11:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.384+0000 7f1200941640 1 -- 172.21.15.107:0/1999977692 wait complete. 2023-11-24T08:11:34.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.615+0000 7f605222e640 1 Processor -- start 2023-11-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.615+0000 7f605222e640 1 -- start start 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f605222e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f604c15b260 0x7f604c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f605222e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c15ed20 con 0x7f604c06e580 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f605222e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c15f8a0 con 0x7f604c0b1250 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f605222e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c15b640 con 0x7f604c15b260 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f605122c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f604c15b260 0x7f604c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f6050a2b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f604c06e580 0x7f604c0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47124/0 (socket says 172.21.15.107:47124) 2023-11-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.617+0000 7f6050a2b640 1 -- 172.21.15.107:0/869198381 learned_addr learned my addr 172.21.15.107:0/869198381 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:34.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.618+0000 7f605122c640 1 -- 172.21.15.107:0/869198381 >> v1:172.21.15.120:6789/0 conn(0x7f604c0b1250 legacy=0x7f604c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:34.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.618+0000 7f605122c640 1 -- 172.21.15.107:0/869198381 >> v1:172.21.15.107:6789/0 conn(0x7f604c06e580 legacy=0x7f604c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.618+0000 7f605122c640 1 -- 172.21.15.107:0/869198381 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c0abe50 con 0x7f604c15b260 2023-11-24T08:11:34.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.618+0000 7f6043fff640 1 -- 172.21.15.107:0/869198381 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38876808 0 0) 0x7f604c15ed20 con 0x7f604c06e580 2023-11-24T08:11:34.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.619+0000 7f605122c640 1 --2- 172.21.15.107:0/869198381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c06e040 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f60480097a0 tx=0x7f604802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b8a0f9b7c13cd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.619+0000 7f6043fff640 1 -- 172.21.15.107:0/869198381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604802f050 con 0x7f604c15b260 2023-11-24T08:11:34.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.619+0000 7f6043fff640 1 -- 172.21.15.107:0/869198381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048031700 con 0x7f604c15b260 2023-11-24T08:11:34.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.619+0000 7f6043fff640 1 -- 172.21.15.107:0/869198381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048031a90 con 0x7f604c15b260 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.619+0000 7f605222e640 1 -- 172.21.15.107:0/869198381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 msgr2=0x7f604c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 --2- 172.21.15.107:0/869198381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c06e040 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f60480097a0 tx=0x7f604802b7a0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 -- 172.21.15.107:0/869198381 shutdown_connections 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 --2- 172.21.15.107:0/869198381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c06e040 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 -- 172.21.15.107:0/869198381 >> 172.21.15.107:0/869198381 conn(0x7f604c0cbb40 msgr2=0x7f604c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 -- 172.21.15.107:0/869198381 shutdown_connections 2023-11-24T08:11:34.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.620+0000 7f605222e640 1 -- 172.21.15.107:0/869198381 wait complete. 2023-11-24T08:11:34.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.622+0000 7f605222e640 1 Processor -- start 2023-11-24T08:11:34.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.622+0000 7f605222e640 1 -- start start 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f604c06e580 0x7f604c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f604c0b1250 0x7f604c15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c15ce10 con 0x7f604c15b260 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c09c6e0 con 0x7f604c06e580 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605222e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0b41a0 con 0x7f604c0b1250 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34472/0 (socket says 172.21.15.107:34472) 2023-11-24T08:11:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 -- 172.21.15.107:0/2610686434 learned_addr learned my addr 172.21.15.107:0/2610686434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f605122c640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f604c06e580 0x7f604c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6050a2b640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f604c0b1250 0x7f604c15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 -- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f604c0b1250 msgr2=0x7f604c15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f604c0b1250 0x7f604c15c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 -- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f604c06e580 msgr2=0x7f604c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f604c06e580 0x7f604c14e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.623+0000 7f6051a2d640 1 -- 172.21.15.107:0/2610686434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6048009400 con 0x7f604c15b260 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f6051a2d640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f604400ba10 tx=0x7f604400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400cb40 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604400cca0 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044014680 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c09c910 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.624+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c0b3b50 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.625+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c04e840 con 0x7f604c15b260 2023-11-24T08:11:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.626+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f604401d3b0 con 0x7f604c15b260 2023-11-24T08:11:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.627+0000 7f6041ffb640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 0x7f603006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.627+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6044088030 con 0x7f604c15b260 2023-11-24T08:11:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.627+0000 7f605122c640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 0x7f603006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:34.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.628+0000 7f605122c640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 0x7f603006ded0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6048035000 tx=0x7f6048008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.631+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60440548d0 con 0x7f604c15b260 2023-11-24T08:11:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.823+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f604c0b42e0 con 0x7f604c15b260 2023-11-24T08:11:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.824+0000 7f6041ffb640 1 -- 172.21.15.107:0/2610686434 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60440190a0 con 0x7f604c15b260 2023-11-24T08:11:34.826 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:11:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 msgr2=0x7f603006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 0x7f603006ded0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6048035000 tx=0x7f6048008030 comp rx=0 tx=0).stop 2023-11-24T08:11:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 msgr2=0x7f604c09c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f604400ba10 tx=0x7f604400bed0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 shutdown_connections 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f603006ba10 0x7f603006ded0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f604c15b260 0x7f604c09c1a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f604c0b1250 0x7f604c15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 --2- 172.21.15.107:0/2610686434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f604c06e580 0x7f604c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 >> 172.21.15.107:0/2610686434 conn(0x7f604c0cbb40 msgr2=0x7f604c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 shutdown_connections 2023-11-24T08:11:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:34.827+0000 7f605222e640 1 -- 172.21.15.107:0/2610686434 wait complete. 2023-11-24T08:11:34.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:11:35.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.048+0000 7fd9fa133640 1 Processor -- start 2023-11-24T08:11:35.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.048+0000 7fd9fa133640 1 -- start start 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.049+0000 7fd9fa133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9f415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9fa133640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd9f406de10 0x7fd9f40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9fa133640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f415ec90 con 0x7fd9f40c49f0 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9fa133640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f40cd530 con 0x7fd9f406de10 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9fa133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c4ed0 con 0x7fd9f4072160 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9f40c49f0 0x7fd9f406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47132/0 (socket says 172.21.15.107:47132) 2023-11-24T08:11:35.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f2ffd640 1 -- 172.21.15.107:0/2702684434 learned_addr learned my addr 172.21.15.107:0/2702684434 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/2702684434 >> v2:172.21.15.120:3300/0 conn(0x7fd9f406de10 0x7fd9f40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9f415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777829949 0 0) 0x7fd9e8001110 con 0x7fd9f40c49f0 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.050+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9dc003520 con 0x7fd9f40c49f0 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3666829074 0 0) 0x7fd9dc003520 con 0x7fd9f40c49f0 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 msgr2=0x7fd9f415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9f415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 >> v2:172.21.15.120:3300/0 conn(0x7fd9f406de10 msgr2=0x7fd9f40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 --2- 172.21.15.107:0/2702684434 >> v2:172.21.15.120:3300/0 conn(0x7fd9f406de10 0x7fd9f40705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40c51b0 con 0x7fd9f40c49f0 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd9e8001800 con 0x7fd9f40c49f0 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 >> v1:172.21.15.107:6789/0 conn(0x7fd9f40c49f0 legacy=0x7fd9f406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9dc003970 0x7fd9dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9dc004d00 0x7fd9dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40cd530 con 0x7fd9dc004d00 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c4ed0 con 0x7fd9dc003970 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00b980 con 0x7fd9f4072160 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.051+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9dc004d00 0x7fd9dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f3fff640 1 -- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 msgr2=0x7fd9dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9dc003970 0x7fd9dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f2ffd640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9f4072160 0x7fd9dc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f3fff640 1 -- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9dc003970 msgr2=0x7fd9dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9dc003970 0x7fd9dc003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.052+0000 7fd9f3fff640 1 -- 172.21.15.107:0/2702684434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc003520 con 0x7fd9dc004d00 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.053+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9dc004d00 0x7fd9dc005180 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40aeb90 tx=0x7fd9dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9218d87f71fa34c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.053+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec02cb90 con 0x7fd9dc004d00 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.053+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec00f040 con 0x7fd9dc004d00 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.053+0000 7fd9f27fc640 1 -- 172.21.15.107:0/2702684434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec034440 con 0x7fd9dc004d00 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 -- 172.21.15.107:0/2702684434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9dc004d00 msgr2=0x7fd9dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 --2- 172.21.15.107:0/2702684434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9dc004d00 0x7fd9dc005180 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40aeb90 tx=0x7fd9dc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9f3fff640 1 -- 172.21.15.107:0/2702684434 reap_dead start 2023-11-24T08:11:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 -- 172.21.15.107:0/2702684434 shutdown_connections 2023-11-24T08:11:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 -- 172.21.15.107:0/2702684434 >> 172.21.15.107:0/2702684434 conn(0x7fd9f40b0c50 msgr2=0x7fd9f40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 -- 172.21.15.107:0/2702684434 shutdown_connections 2023-11-24T08:11:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.054+0000 7fd9fa133640 1 -- 172.21.15.107:0/2702684434 wait complete. 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.056+0000 7fd9fa133640 1 Processor -- start 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.056+0000 7fd9fa133640 1 -- start start 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 0x7fd9f41592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9f40b4710 0x7fd9f40cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c8690 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f415d720 con 0x7fd9f40b4710 2023-11-24T08:11:35.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9fa133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c8f10 con 0x7fd9dc004d00 2023-11-24T08:11:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9f2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9f2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34492/0 (socket says 172.21.15.107:34492) 2023-11-24T08:11:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9f2ffd640 1 -- 172.21.15.107:0/499101990 learned_addr learned my addr 172.21.15.107:0/499101990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.057+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 0x7fd9f41592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f3fff640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9f40b4710 0x7fd9f40cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 -- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 msgr2=0x7fd9f41592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 0x7fd9f41592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 -- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9f40b4710 msgr2=0x7fd9f40cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9f40b4710 0x7fd9f40cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 -- 172.21.15.107:0/499101990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40c7750 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 0x7fd9f41592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9f2ffd640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d890 tx=0x7fd9e800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80074e0 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e8007640 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f40c7a10 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8004a20 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.058+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f40c7fd0 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.060+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f40c87d0 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.061+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fd9e8004b80 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.061+0000 7fd9dbfff640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 0x7fd9e006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.061+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd9e80878e0 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.062+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 0x7fd9e006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.062+0000 7fd9f37fe640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 0x7fd9e006ded0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40099e0 tx=0x7fd9e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.066+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9f40c87d0 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.263+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd9f415ec90 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.264+0000 7fd9dbfff640 1 -- 172.21.15.107:0/499101990 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd9f415ec90 con 0x7fd9f40c8ab0 2023-11-24T08:11:35.266 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.266+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 msgr2=0x7fd9e006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.266+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 0x7fd9e006ded0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40099e0 tx=0x7fd9e4008040 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.266+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 msgr2=0x7fd9f40c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.266+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d890 tx=0x7fd9e800dd50 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 shutdown_connections 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9e006ba10 0x7fd9e006ded0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9f40b4710 0x7fd9f40cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f40c8ab0 0x7fd9f40c7210 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 --2- 172.21.15.107:0/499101990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9dc004d00 0x7fd9f41592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 >> 172.21.15.107:0/499101990 conn(0x7fd9f40b0c50 msgr2=0x7fd9f40ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 shutdown_connections 2023-11-24T08:11:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.267+0000 7fd9fa133640 1 -- 172.21.15.107:0/499101990 wait complete. 2023-11-24T08:11:35.285 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:11:35.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:35.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:11:35.434 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:11:35.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:35.434 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:11:35.491 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:11:35.491 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:35.533 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:11:35.533 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:35.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:35.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.828+0000 7f50d29a9640 1 Processor -- start 2023-11-24T08:11:35.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.828+0000 7f50d29a9640 1 -- start start 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d29a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50cc0b2250 0x7f50cc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d29a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc15ab80 con 0x7f50cc0b2aa0 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d29a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc15e680 con 0x7f50cc0b2630 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d29a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc15a990 con 0x7f50cc0b2250 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d0f46640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f50cc0b2aa0 0x7f50cc15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47138/0 (socket says 172.21.15.107:47138) 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50d0f46640 1 -- 172.21.15.107:0/985522791 learned_addr learned my addr 172.21.15.107:0/985522791 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:35.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.830+0000 7f50cbfff640 1 --2- 172.21.15.107:0/985522791 >> v2:172.21.15.107:3300/0 conn(0x7f50cc0b2250 0x7f50cc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.831+0000 7f50cbfff640 1 -- 172.21.15.107:0/985522791 >> v1:172.21.15.120:6789/0 conn(0x7f50cc0b2630 legacy=0x7f50cc0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:35.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.831+0000 7f50cb7fe640 1 -- 172.21.15.107:0/985522791 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756579127 0 0) 0x7f50cc15ab80 con 0x7f50cc0b2aa0 2023-11-24T08:11:35.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.831+0000 7f50cbfff640 1 -- 172.21.15.107:0/985522791 >> v1:172.21.15.107:6789/0 conn(0x7f50cc0b2aa0 legacy=0x7f50cc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.831+0000 7f50cbfff640 1 -- 172.21.15.107:0/985522791 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc15e960 con 0x7f50cc0b2250 2023-11-24T08:11:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.832+0000 7f50cbfff640 1 --2- 172.21.15.107:0/985522791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc15df80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f50cc15ad70 tx=0x7f50bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef1cb308cdfe518 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.832+0000 7f50cb7fe640 1 -- 172.21.15.107:0/985522791 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc031830 con 0x7f50cc0b2250 2023-11-24T08:11:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.832+0000 7f50cb7fe640 1 -- 172.21.15.107:0/985522791 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc031990 con 0x7f50cc0b2250 2023-11-24T08:11:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.832+0000 7f50cb7fe640 1 -- 172.21.15.107:0/985522791 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc031d20 con 0x7f50cc0b2250 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 -- 172.21.15.107:0/985522791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 msgr2=0x7f50cc15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 --2- 172.21.15.107:0/985522791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc15df80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f50cc15ad70 tx=0x7f50bc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 -- 172.21.15.107:0/985522791 shutdown_connections 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 --2- 172.21.15.107:0/985522791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc15df80 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 -- 172.21.15.107:0/985522791 >> 172.21.15.107:0/985522791 conn(0x7f50cc06a920 msgr2=0x7f50cc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 -- 172.21.15.107:0/985522791 shutdown_connections 2023-11-24T08:11:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.833+0000 7f50d29a9640 1 -- 172.21.15.107:0/985522791 wait complete. 2023-11-24T08:11:35.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.834+0000 7f50d29a9640 1 Processor -- start 2023-11-24T08:11:35.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.835+0000 7f50d29a9640 1 -- start start 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.835+0000 7f50d29a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.835+0000 7f50d29a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50cc0b2630 0x7f50cc157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50d29a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50cc0b2aa0 0x7f50cc156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50d29a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0c9eb0 con 0x7f50cc0b2250 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50d29a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0c9a20 con 0x7f50cc0b2aa0 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50d29a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0c8e10 con 0x7f50cc0b2630 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34510/0 (socket says 172.21.15.107:34510) 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 -- 172.21.15.107:0/3200763096 learned_addr learned my addr 172.21.15.107:0/3200763096 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 -- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50cc0b2630 msgr2=0x7f50cc157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50cc0b2630 0x7f50cc157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 -- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50cc0b2aa0 msgr2=0x7f50cc156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50cc0b2aa0 0x7f50cc156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:35.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 -- 172.21.15.107:0/3200763096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc009400 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.836+0000 7f50cbfff640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f50bc02bee0 tx=0x7f50bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.837+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc03ca80 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.837+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc03cbe0 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.837+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc0cd7c0 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.837+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc004030 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.837+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc0c93d0 con 0x7f50cc0b2250 2023-11-24T08:11:35.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.838+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc067470 con 0x7f50cc0b2250 2023-11-24T08:11:35.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.839+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f50bc004190 con 0x7f50cc0b2250 2023-11-24T08:11:35.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.840+0000 7f50c97fa640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 0x7f50ac06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:35.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.840+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f50bc0aaa50 con 0x7f50cc0b2250 2023-11-24T08:11:35.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.840+0000 7f50c3fff640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 0x7f50ac06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:35.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.841+0000 7f50c3fff640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 0x7f50ac06ded0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f50cc09d310 tx=0x7f50b40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:35.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:35.844+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50bc0772f0 con 0x7f50cc0b2250 2023-11-24T08:11:36.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.033+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50cc067470 con 0x7f50cc0b2250 2023-11-24T08:11:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.035+0000 7f50c97fa640 1 -- 172.21.15.107:0/3200763096 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f50bc07a910 con 0x7f50cc0b2250 2023-11-24T08:11:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:36.038 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":82,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:34.860017+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":81,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.1c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 msgr2=0x7f50ac06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 0x7f50ac06ded0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f50cc09d310 tx=0x7f50b40079b0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 msgr2=0x7f50cc153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f50bc02bee0 tx=0x7f50bc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 shutdown_connections 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50ac06ba10 0x7f50ac06ded0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50cc0b2aa0 0x7f50cc156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50cc0b2630 0x7f50cc157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 --2- 172.21.15.107:0/3200763096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50cc0b2250 0x7f50cc153e10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.038+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 >> 172.21.15.107:0/3200763096 conn(0x7f50cc06a920 msgr2=0x7f50cc15b080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.039+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 shutdown_connections 2023-11-24T08:11:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.039+0000 7f50d29a9640 1 -- 172.21.15.107:0/3200763096 wait complete. 2023-11-24T08:11:36.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:10:24.450447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:10:25.621972+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 81, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:11:36.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:11:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.277+0000 7fda69a1e640 1 Processor -- start 2023-11-24T08:11:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.277+0000 7fda69a1e640 1 -- start start 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda69a1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fda6415b440 0x7fda640cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda69a1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda69a1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6415eea0 con 0x7fda640cbf20 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda69a1e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640c5e00 con 0x7fda6415b440 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda69a1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640cc3d0 con 0x7fda640c59b0 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda627fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fda640cbf20 0x7fda640c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47154/0 (socket says 172.21.15.107:47154) 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda627fc640 1 -- 172.21.15.107:0/556553063 learned_addr learned my addr 172.21.15.107:0/556553063 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.279+0000 7fda62ffd640 1 --2- 172.21.15.107:0/556553063 >> v2:172.21.15.107:3300/0 conn(0x7fda6415b440 0x7fda640cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda637fe640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda61ffb640 1 -- 172.21.15.107:0/556553063 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515426089 0 0) 0x7fda6415eea0 con 0x7fda640cbf20 2023-11-24T08:11:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda61ffb640 1 -- 172.21.15.107:0/556553063 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda4c003520 con 0x7fda640cbf20 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda62ffd640 1 -- 172.21.15.107:0/556553063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 msgr2=0x7fda6406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda62ffd640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda62ffd640 1 -- 172.21.15.107:0/556553063 >> v1:172.21.15.107:6789/0 conn(0x7fda640cbf20 legacy=0x7fda640c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.280+0000 7fda62ffd640 1 -- 172.21.15.107:0/556553063 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640abf30 con 0x7fda6415b440 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.281+0000 7fda62ffd640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6415b440 0x7fda640cb9e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5618b8bf8c3c62 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda61ffb640 1 -- 172.21.15.107:0/556553063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5402e070 con 0x7fda6415b440 2023-11-24T08:11:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda61ffb640 1 -- 172.21.15.107:0/556553063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda540315a0 con 0x7fda6415b440 2023-11-24T08:11:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda61ffb640 1 -- 172.21.15.107:0/556553063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54031910 con 0x7fda6415b440 2023-11-24T08:11:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 -- 172.21.15.107:0/556553063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6415b440 msgr2=0x7fda640cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6415b440 0x7fda640cb9e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b840 comp rx=0 tx=0).stop 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 -- 172.21.15.107:0/556553063 shutdown_connections 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 --2- 172.21.15.107:0/556553063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6415b440 0x7fda640cb9e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.282+0000 7fda69a1e640 1 -- 172.21.15.107:0/556553063 >> 172.21.15.107:0/556553063 conn(0x7fda640b0c50 msgr2=0x7fda640b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.283+0000 7fda69a1e640 1 -- 172.21.15.107:0/556553063 shutdown_connections 2023-11-24T08:11:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.283+0000 7fda69a1e640 1 -- 172.21.15.107:0/556553063 wait complete. 2023-11-24T08:11:36.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.284+0000 7fda69a1e640 1 Processor -- start 2023-11-24T08:11:36.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.284+0000 7fda69a1e640 1 -- start start 2023-11-24T08:11:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 0x7fda640c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6415ccd0 con 0x7fda640cbf20 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640c6c50 con 0x7fda6415b440 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.285+0000 7fda69a1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640c6f30 con 0x7fda640c59b0 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34526/0 (socket says 172.21.15.107:34526) 2023-11-24T08:11:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 -- 172.21.15.107:0/4229606531 learned_addr learned my addr 172.21.15.107:0/4229606531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:36.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda637fe640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 0x7fda640c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 -- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 msgr2=0x7fda6415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 -- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 msgr2=0x7fda640c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 0x7fda640c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 -- 172.21.15.107:0/4229606531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54009400 con 0x7fda640cbf20 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda637fe640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 0x7fda640c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.286+0000 7fda627fc640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fda5800c430 tx=0x7fda5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.287+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58005ce0 con 0x7fda640cbf20 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.287+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda58005e40 con 0x7fda640cbf20 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.287+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda640c7210 con 0x7fda640cbf20 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.287+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda580072f0 con 0x7fda640cbf20 2023-11-24T08:11:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.287+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6415dd00 con 0x7fda640cbf20 2023-11-24T08:11:36.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.288+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6404e8b0 con 0x7fda640cbf20 2023-11-24T08:11:36.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.289+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fda58007450 con 0x7fda640cbf20 2023-11-24T08:11:36.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.290+0000 7fda4bfff640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 0x7fda5006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.290+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fda580878e0 con 0x7fda640cbf20 2023-11-24T08:11:36.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.290+0000 7fda62ffd640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 0x7fda5006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.291+0000 7fda62ffd640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 0x7fda5006ded0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda54030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.294+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda58057c10 con 0x7fda640cbf20 2023-11-24T08:11:36.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.484+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda6406dc80 con 0x7fda640cbf20 2023-11-24T08:11:36.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.485+0000 7fda4bfff640 1 -- 172.21.15.107:0/4229606531 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda580170a0 con 0x7fda640cbf20 2023-11-24T08:11:36.487 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 msgr2=0x7fda5006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 0x7fda5006ded0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda54030000 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 msgr2=0x7fda640c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fda5800c430 tx=0x7fda5800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 shutdown_connections 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda5006ba10 0x7fda5006ded0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda6415b440 0x7fda640c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda640cbf20 0x7fda640c60d0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 --2- 172.21.15.107:0/4229606531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda640c59b0 0x7fda6415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 >> 172.21.15.107:0/4229606531 conn(0x7fda640b0c50 msgr2=0x7fda640716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 shutdown_connections 2023-11-24T08:11:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.488+0000 7fda69a1e640 1 -- 172.21.15.107:0/4229606531 wait complete. 2023-11-24T08:11:36.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:11:36.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.725+0000 7f4bd390f640 1 Processor -- start 2023-11-24T08:11:36.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.726+0000 7f4bd390f640 1 -- start start 2023-11-24T08:11:36.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.727+0000 7f4bd390f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bcc140520 0x7f4bcc0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.727+0000 7f4bd390f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc06ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.727+0000 7f4bd390f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bcc161fa0 con 0x7f4bcc0c65f0 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd390f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc0c6a70 con 0x7f4bcc140520 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd390f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc15de00 con 0x7f4bcc161b10 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bcc140520 0x7f4bcc0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bcc140520 0x7f4bcc0c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34538/0 (socket says 172.21.15.107:34538) 2023-11-24T08:11:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 -- 172.21.15.107:0/627236398 learned_addr learned my addr 172.21.15.107:0/627236398 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd1eac640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc06ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 -- 172.21.15.107:0/627236398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 msgr2=0x7f4bcc06ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc06ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 -- 172.21.15.107:0/627236398 >> v1:172.21.15.120:6789/0 conn(0x7f4bcc0c65f0 legacy=0x7f4bcc161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.728+0000 7f4bd16ab640 1 -- 172.21.15.107:0/627236398 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc15e0e0 con 0x7f4bcc140520 2023-11-24T08:11:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.729+0000 7f4bd16ab640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc0c60b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009e50 tx=0x7f4bc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4437d3684c2dfe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.730+0000 7f4bcbfff640 1 -- 172.21.15.107:0/627236398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc002ca70 con 0x7f4bcc140520 2023-11-24T08:11:36.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.730+0000 7f4bcbfff640 1 -- 172.21.15.107:0/627236398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc002cbd0 con 0x7f4bcc140520 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.730+0000 7f4bcbfff640 1 -- 172.21.15.107:0/627236398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00343f0 con 0x7f4bcc140520 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.730+0000 7f4bd390f640 1 -- 172.21.15.107:0/627236398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 msgr2=0x7f4bcc0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.730+0000 7f4bd390f640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc0c60b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009e50 tx=0x7f4bc002bed0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 -- 172.21.15.107:0/627236398 shutdown_connections 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc06ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 --2- 172.21.15.107:0/627236398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc0c60b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 -- 172.21.15.107:0/627236398 >> 172.21.15.107:0/627236398 conn(0x7f4bcc0b2e40 msgr2=0x7f4bcc0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 -- 172.21.15.107:0/627236398 shutdown_connections 2023-11-24T08:11:36.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.731+0000 7f4bd390f640 1 -- 172.21.15.107:0/627236398 wait complete. 2023-11-24T08:11:36.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.733+0000 7f4bd390f640 1 Processor -- start 2023-11-24T08:11:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.733+0000 7f4bd390f640 1 -- start start 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc0c65f0 0x7f4bcc13eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc13f3e0 con 0x7f4bcc140520 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc15d8d0 con 0x7f4bcc0c65f0 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.734+0000 7f4bd390f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0b1c30 con 0x7f4bcc161b10 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34550/0 (socket says 172.21.15.107:34550) 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 -- 172.21.15.107:0/1772136768 learned_addr learned my addr 172.21.15.107:0/1772136768 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd16ab640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc0c65f0 0x7f4bcc13eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 -- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 msgr2=0x7f4bcc1582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 -- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc0c65f0 msgr2=0x7f4bcc13eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc0c65f0 0x7f4bcc13eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 -- 172.21.15.107:0/1772136768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0009b30 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.735+0000 7f4bd0eaa640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800b420 tx=0x7f4bb800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.736+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011830 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.736+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8011990 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.736+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011d40 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.736+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc15d310 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.736+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc0b21c0 con 0x7f4bcc140520 2023-11-24T08:11:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.737+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc0b23e0 con 0x7f4bcc140520 2023-11-24T08:11:36.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.738+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f4bb801b460 con 0x7f4bcc140520 2023-11-24T08:11:36.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.740+0000 7f4bc9ffb640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 0x7f4bb006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.740+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4bb8087e50 con 0x7f4bcc140520 2023-11-24T08:11:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.740+0000 7f4bd16ab640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 0x7f4bb006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.741+0000 7f4bd16ab640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 0x7f4bb006ded0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0035000 tx=0x7f4bc0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:36.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.744+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bb80546f0 con 0x7f4bcc140520 2023-11-24T08:11:36.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.936+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4bcc04e8b0 con 0x7f4bcc140520 2023-11-24T08:11:36.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.937+0000 7f4bc9ffb640 1 -- 172.21.15.107:0/1772136768 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4bb8057d10 con 0x7f4bcc140520 2023-11-24T08:11:36.939 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 msgr2=0x7f4bb006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 0x7f4bb006ded0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0035000 tx=0x7f4bc0030000 comp rx=0 tx=0).stop 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 msgr2=0x7f4bcc15b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800b420 tx=0x7f4bb800b8e0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 shutdown_connections 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb006ba10 0x7f4bb006ded0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bcc161b10 0x7f4bcc1582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bcc140520 0x7f4bcc15b710 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 --2- 172.21.15.107:0/1772136768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc0c65f0 0x7f4bcc13eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:36.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 >> 172.21.15.107:0/1772136768 conn(0x7f4bcc0b2e40 msgr2=0x7f4bcc070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:36.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 shutdown_connections 2023-11-24T08:11:36.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:36.940+0000 7f4bd390f640 1 -- 172.21.15.107:0/1772136768 wait complete. 2023-11-24T08:11:36.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:11:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.168+0000 7f3a23ca6640 1 Processor -- start 2023-11-24T08:11:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.169+0000 7f3a23ca6640 1 -- start start 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.170+0000 7f3a23ca6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a1c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.170+0000 7f3a23ca6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c1655f0 con 0x7f3a1c0c9650 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a23ca6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c0cc380 con 0x7f3a1c0ccfb0 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a23ca6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c162b30 con 0x7f3a1c0ccbd0 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a22243640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3a1c0c9650 0x7f3a1c0c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47158/0 (socket says 172.21.15.107:47158) 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a22243640 1 -- 172.21.15.107:0/2147372196 learned_addr learned my addr 172.21.15.107:0/2147372196 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a21a42640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a1c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908841352 0 0) 0x7f3a1c1655f0 con 0x7f3a1c0c9650 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.171+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a04003520 con 0x7f3a1c0c9650 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899494088 0 0) 0x7f3a04003520 con 0x7f3a1c0c9650 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 msgr2=0x7f3a1c1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a1c1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 >> v1:172.21.15.120:6789/0 conn(0x7f3a1c0ccfb0 legacy=0x7f3a1c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c162e10 con 0x7f3a1c0c9650 2023-11-24T08:11:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3a14001670 con 0x7f3a1c0c9650 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 >> v1:172.21.15.107:6789/0 conn(0x7f3a1c0c9650 legacy=0x7f3a1c0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a04004d00 0x7f3a04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c162b30 con 0x7f3a040039a0 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0400ba30 con 0x7f3a04004d00 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0400bbf0 con 0x7f3a1c0ccbd0 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.172+0000 7f3a21a42640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a22243640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a04004d00 0x7f3a04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21241640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21a42640 1 -- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 msgr2=0x7f3a040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21a42640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21a42640 1 -- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a04004d00 msgr2=0x7f3a04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21a42640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a04004d00 0x7f3a04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21a42640 1 -- 172.21.15.107:0/2147372196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a04003520 con 0x7f3a040039a0 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a22243640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a04004d00 0x7f3a04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.173+0000 7f3a21241640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c0ccbd0 0x7f3a040047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.174+0000 7f3a21a42640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a04003dc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009f90 tx=0x7f3a040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb16d162c1c0fcb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.174+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c02cbd0 con 0x7f3a040039a0 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.174+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00f040 con 0x7f3a040039a0 2023-11-24T08:11:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.174+0000 7f3a20a40640 1 -- 172.21.15.107:0/2147372196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c034440 con 0x7f3a040039a0 2023-11-24T08:11:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 -- 172.21.15.107:0/2147372196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 msgr2=0x7f3a04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/2147372196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a04003dc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009f90 tx=0x7f3a040057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a22243640 1 -- 172.21.15.107:0/2147372196 reap_dead start 2023-11-24T08:11:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 -- 172.21.15.107:0/2147372196 shutdown_connections 2023-11-24T08:11:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 -- 172.21.15.107:0/2147372196 >> 172.21.15.107:0/2147372196 conn(0x7f3a1c0b0c50 msgr2=0x7f3a1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 -- 172.21.15.107:0/2147372196 shutdown_connections 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.175+0000 7f3a23ca6640 1 -- 172.21.15.107:0/2147372196 wait complete. 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.177+0000 7f3a23ca6640 1 Processor -- start 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.177+0000 7f3a23ca6640 1 -- start start 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 0x7f3a1c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a1c0b1d40 0x7f3a1c1640a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b4630 con 0x7f3a040039a0 2023-11-24T08:11:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0ac060 con 0x7f3a1c0b1d40 2023-11-24T08:11:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a23ca6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c158cb0 con 0x7f3a1c1f7630 2023-11-24T08:11:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34578/0 (socket says 172.21.15.107:34578) 2023-11-24T08:11:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 -- 172.21.15.107:0/1020447347 learned_addr learned my addr 172.21.15.107:0/1020447347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21241640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 0x7f3a1c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 -- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 msgr2=0x7f3a1c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a22243640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a1c0b1d40 0x7f3a1c1640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 0x7f3a1c158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 -- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a1c0b1d40 msgr2=0x7f3a1c1640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.178+0000 7f3a21a42640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a1c0b1d40 0x7f3a1c1640a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a21a42640 1 -- 172.21.15.107:0/1020447347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c1f7bd0 con 0x7f3a040039a0 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a21241640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 0x7f3a1c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a21a42640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c037040 tx=0x7f3a0c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c034d60 con 0x7f3a040039a0 2023-11-24T08:11:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c03e400 con 0x7f3a040039a0 2023-11-24T08:11:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c1f7eb0 con 0x7f3a040039a0 2023-11-24T08:11:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c03e7b0 con 0x7f3a040039a0 2023-11-24T08:11:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.179+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c1f8470 con 0x7f3a040039a0 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.181+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1c04e8b0 con 0x7f3a040039a0 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.181+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f3a0c03e910 con 0x7f3a040039a0 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.182+0000 7f3a1a7fc640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 0x7f3a0806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.182+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a0c0aa530 con 0x7f3a040039a0 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.183+0000 7f3a21241640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 0x7f3a0806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.183+0000 7f3a21241640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 0x7f3a0806df00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0caf20 tx=0x7f3a10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.187+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a0c0770e0 con 0x7f3a040039a0 2023-11-24T08:11:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.375+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a1c12c1b0 con 0x7f3a040039a0 2023-11-24T08:11:37.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.376+0000 7f3a1a7fc640 1 -- 172.21.15.107:0/1020447347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a0c04c020 con 0x7f3a040039a0 2023-11-24T08:11:37.378 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.378+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 msgr2=0x7f3a0806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.378+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 0x7f3a0806df00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0caf20 tx=0x7f3a10009040 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.378+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 msgr2=0x7f3a1c0b40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.378+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c037040 tx=0x7f3a0c009750 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 shutdown_connections 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a0806ba40 0x7f3a0806df00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a1c0b1d40 0x7f3a1c1640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a1c1f7630 0x7f3a1c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 --2- 172.21.15.107:0/1020447347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a040039a0 0x7f3a1c0b40f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 >> 172.21.15.107:0/1020447347 conn(0x7f3a1c0b0c50 msgr2=0x7f3a1c072080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 shutdown_connections 2023-11-24T08:11:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.379+0000 7f3a23ca6640 1 -- 172.21.15.107:0/1020447347 wait complete. 2023-11-24T08:11:37.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:11:37.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.607+0000 7fae1e7e4640 1 Processor -- start 2023-11-24T08:11:37.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.607+0000 7fae1e7e4640 1 -- start start 2023-11-24T08:11:37.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1e7e4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae1815a1c0 0x7fae18070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1e7e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1815e580 con 0x7fae180b15e0 2023-11-24T08:11:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1e7e4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1815e860 con 0x7fae1806d640 2023-11-24T08:11:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1e7e4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae180cd920 con 0x7fae1815a1c0 2023-11-24T08:11:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1cfe1640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fae180b15e0 0x7fae1815dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47162/0 (socket says 172.21.15.107:47162) 2023-11-24T08:11:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1cfe1640 1 -- 172.21.15.107:0/909052095 learned_addr learned my addr 172.21.15.107:0/909052095 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/909052095 >> v2:172.21.15.120:3300/0 conn(0x7fae1815a1c0 0x7fae18070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.609+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866192574 0 0) 0x7fae1815e580 con 0x7fae180b15e0 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae04003520 con 0x7fae180b15e0 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411897426 0 0) 0x7fae04003520 con 0x7fae180b15e0 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 >> v2:172.21.15.120:3300/0 conn(0x7fae1815a1c0 msgr2=0x7fae18070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 --2- 172.21.15.107:0/909052095 >> v2:172.21.15.120:3300/0 conn(0x7fae1815a1c0 0x7fae18070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 >> v1:172.21.15.120:6789/0 conn(0x7fae1806d640 legacy=0x7fae1806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1815eb40 con 0x7fae180b15e0 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fae08001670 con 0x7fae180b15e0 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 >> v1:172.21.15.107:6789/0 conn(0x7fae180b15e0 legacy=0x7fae1815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae04003970 0x7fae04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 0x7fae04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae04005180 0x7fae0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180cd920 con 0x7fae040047c0 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0400c030 con 0x7fae04005180 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0400c1f0 con 0x7fae04003970 2023-11-24T08:11:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.610+0000 7fae1cfe1640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 0x7fae04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae04003970 0x7fae04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1dfe3640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae04005180 0x7fae0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1cfe1640 1 -- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae04003970 msgr2=0x7fae04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1cfe1640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae04003970 0x7fae04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1cfe1640 1 -- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae04005180 msgr2=0x7fae0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1cfe1640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae04005180 0x7fae0400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1cfe1640 1 -- 172.21.15.107:0/909052095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae04003520 con 0x7fae040047c0 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1dfe3640 1 -- 172.21.15.107:0/909052095 reap_dead start 2023-11-24T08:11:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.611+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae04003970 0x7fae04003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.612+0000 7fae1cfe1640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 0x7fae04004c40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c210 tx=0x7fae040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb71ef4a9c65ac35 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.612+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08005ad0 con 0x7fae040047c0 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.612+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08005c50 con 0x7fae040047c0 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.612+0000 7fae0ffff640 1 -- 172.21.15.107:0/909052095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0802c890 con 0x7fae040047c0 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 -- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 msgr2=0x7fae04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 0x7fae04004c40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c210 tx=0x7fae040057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 -- 172.21.15.107:0/909052095 shutdown_connections 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/909052095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae040047c0 0x7fae04004c40 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 -- 172.21.15.107:0/909052095 >> 172.21.15.107:0/909052095 conn(0x7fae180cbb40 msgr2=0x7fae180cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:37.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 -- 172.21.15.107:0/909052095 shutdown_connections 2023-11-24T08:11:37.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.613+0000 7fae1e7e4640 1 -- 172.21.15.107:0/909052095 wait complete. 2023-11-24T08:11:37.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.615+0000 7fae1e7e4640 1 Processor -- start 2023-11-24T08:11:37.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.615+0000 7fae1e7e4640 1 -- start start 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae040047c0 0x7fae180c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae180b3e30 0x7fae180b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c6b50 con 0x7fae180c6190 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c92b0 con 0x7fae180b3e30 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1e7e4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1815b550 con 0x7fae040047c0 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1dfe3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1dfe3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34600/0 (socket says 172.21.15.107:34600) 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1dfe3640 1 -- 172.21.15.107:0/2816802271 learned_addr learned my addr 172.21.15.107:0/2816802271 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:37.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae040047c0 0x7fae180c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.616+0000 7fae1cfe1640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae180b3e30 0x7fae180b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 -- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae040047c0 msgr2=0x7fae180c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae040047c0 0x7fae180c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 -- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae180b3e30 msgr2=0x7fae180b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae180b3e30 0x7fae180b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 -- 172.21.15.107:0/2816802271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1815b830 con 0x7fae180c6190 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1dfe3640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fae180cbaa0 tx=0x7fae1000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10003b70 con 0x7fae180c6190 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae10003cd0 con 0x7fae180c6190 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae100052b0 con 0x7fae180c6190 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1815bb10 con 0x7fae180c6190 2023-11-24T08:11:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.617+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1815c0d0 con 0x7fae180c6190 2023-11-24T08:11:37.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.618+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1815c2f0 con 0x7fae180c6190 2023-11-24T08:11:37.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.619+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fae10005410 con 0x7fae180c6190 2023-11-24T08:11:37.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.620+0000 7fae0dffb640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 0x7fadfc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:37.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.620+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fae10087900 con 0x7fae180c6190 2023-11-24T08:11:37.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.621+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 0x7fadfc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:37.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.621+0000 7fae1d7e2640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 0x7fadfc06ded0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fae1809c7c0 tx=0x7fae14008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:37.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.624+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae100543c0 con 0x7fae180c6190 2023-11-24T08:11:37.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.832+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae1804e840 con 0x7fae180c6190 2023-11-24T08:11:37.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.833+0000 7fae0dffb640 1 -- 172.21.15.107:0/2816802271 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fae10018020 con 0x7fae180c6190 2023-11-24T08:11:37.835 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:11:37.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 msgr2=0x7fadfc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 0x7fadfc06ded0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fae1809c7c0 tx=0x7fae14008040 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 msgr2=0x7fae180c6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fae180cbaa0 tx=0x7fae1000dc00 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 shutdown_connections 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadfc06ba10 0x7fadfc06ded0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae180c6190 0x7fae180c6610 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae180b3e30 0x7fae180b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 --2- 172.21.15.107:0/2816802271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae040047c0 0x7fae180c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.835+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 >> 172.21.15.107:0/2816802271 conn(0x7fae180cbb40 msgr2=0x7fae180c9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.836+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 shutdown_connections 2023-11-24T08:11:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:37.836+0000 7fae1e7e4640 1 -- 172.21.15.107:0/2816802271 wait complete. 2023-11-24T08:11:37.855 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:11:37.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:37.855 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:11:37.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:11:37.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:11:37.974 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:11:38.028 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:38.028 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:38.088 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:11:38.088 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:11:38.088 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:38.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:11:38.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:11:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.369+0000 7fd99e0b4640 1 Processor -- start 2023-11-24T08:11:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.369+0000 7fd99e0b4640 1 -- start start 2023-11-24T08:11:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.370+0000 7fd99e0b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd998071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.370+0000 7fd99e0b4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980ce870 con 0x7fd99806d5c0 2023-11-24T08:11:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.370+0000 7fd99e0b4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980cdc60 con 0x7fd998159e80 2023-11-24T08:11:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.370+0000 7fd99e0b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980cdf40 con 0x7fd99815a970 2023-11-24T08:11:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.371+0000 7fd99c8b1640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd99806d5c0 0x7fd998161420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41812/0 (socket says 172.21.15.107:41812) 2023-11-24T08:11:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.371+0000 7fd99c8b1640 1 -- 172.21.15.107:0/971167407 learned_addr learned my addr 172.21.15.107:0/971167407 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.371+0000 7fd99d8b3640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd998071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.371+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209859116 0 0) 0x7fd9980ce870 con 0x7fd99806d5c0 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.371+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984003440 con 0x7fd99806d5c0 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329559875 0 0) 0x7fd984003440 con 0x7fd99806d5c0 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 msgr2=0x7fd998071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd998071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 >> v1:172.21.15.120:6789/0 conn(0x7fd998159e80 legacy=0x7fd99815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980ce220 con 0x7fd99806d5c0 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd988001670 con 0x7fd99806d5c0 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 >> v1:172.21.15.107:6789/0 conn(0x7fd99806d5c0 legacy=0x7fd998161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd984003a90 0x7fd984003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd98400a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd98400d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980cdf40 con 0x7fd98400b000 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98400dd40 con 0x7fd984003a90 2023-11-24T08:11:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98400df00 con 0x7fd99815a970 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd99d8b3640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd98400d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd99d0b2640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd984003a90 0x7fd984003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.372+0000 7fd99c8b1640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd98400a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.373+0000 7fd99d8b3640 1 -- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 msgr2=0x7fd98400a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.373+0000 7fd99d8b3640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd99815a970 0x7fd98400a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.373+0000 7fd99d8b3640 1 -- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd984003a90 msgr2=0x7fd984003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.373+0000 7fd99d8b3640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd984003a90 0x7fd984003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.373+0000 7fd99d8b3640 1 -- 172.21.15.107:0/971167407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984003440 con 0x7fd98400b000 2023-11-24T08:11:38.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd99d8b3640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd98400d490 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd990009fc0 tx=0x7fd9840036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603629d2071037a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99002cc20 con 0x7fd98400b000 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99000f040 con 0x7fd98400b000 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd98ffff640 1 -- 172.21.15.107:0/971167407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990034440 con 0x7fd98400b000 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd99e0b4640 1 -- 172.21.15.107:0/971167407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 msgr2=0x7fd98400d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.374+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/971167407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd98400d490 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd990009fc0 tx=0x7fd9840036f0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.375+0000 7fd99d8b3640 1 -- 172.21.15.107:0/971167407 reap_dead start 2023-11-24T08:11:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.375+0000 7fd99e0b4640 1 -- 172.21.15.107:0/971167407 shutdown_connections 2023-11-24T08:11:38.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.375+0000 7fd99e0b4640 1 -- 172.21.15.107:0/971167407 >> 172.21.15.107:0/971167407 conn(0x7fd9980cb830 msgr2=0x7fd9980cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:38.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.375+0000 7fd99e0b4640 1 -- 172.21.15.107:0/971167407 shutdown_connections 2023-11-24T08:11:38.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.375+0000 7fd99e0b4640 1 -- 172.21.15.107:0/971167407 wait complete. 2023-11-24T08:11:38.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.376+0000 7fd99e0b4640 1 Processor -- start 2023-11-24T08:11:38.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 -- start start 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd998140f10 0x7fd9981e31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998142550 0x7fd9981559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd998143ea0 con 0x7fd98400b000 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9981414b0 con 0x7fd998142550 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.377+0000 7fd99e0b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd998141790 con 0x7fd998140f10 2023-11-24T08:11:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36718/0 (socket says 172.21.15.107:36718) 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 -- 172.21.15.107:0/3588664366 learned_addr learned my addr 172.21.15.107:0/3588664366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 -- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd998140f10 msgr2=0x7fd9981e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd998140f10 0x7fd9981e31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 -- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998142550 msgr2=0x7fd9981559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998142550 0x7fd9981559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 -- 172.21.15.107:0/3588664366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd998141a70 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.378+0000 7fd99d0b2640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e550 tx=0x7fd99400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.379+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940052e0 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.379+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994005440 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.379+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994003810 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.379+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd998141d50 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.379+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9981e3710 con 0x7fd98400b000 2023-11-24T08:11:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.380+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99804e840 con 0x7fd98400b000 2023-11-24T08:11:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.381+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fd994003970 con 0x7fd98400b000 2023-11-24T08:11:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.382+0000 7fd98dffb640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 0x7fd97c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.382+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd994087f30 con 0x7fd98400b000 2023-11-24T08:11:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.382+0000 7fd99c8b1640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 0x7fd97c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.383+0000 7fd99c8b1640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 0x7fd97c06ded0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd988001050 tx=0x7fd988000f60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.386+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9940547d0 con 0x7fd98400b000 2023-11-24T08:11:38.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.612+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9981e38d0 con 0x7fd98400b000 2023-11-24T08:11:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.613+0000 7fd98dffb640 1 -- 172.21.15.107:0/3588664366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fd994057df0 con 0x7fd98400b000 2023-11-24T08:11:38.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:38.615 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:11:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-11-24T08:11:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.615+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 msgr2=0x7fd97c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.615+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 0x7fd97c06ded0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd988001050 tx=0x7fd988000f60 comp rx=0 tx=0).stop 2023-11-24T08:11:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 msgr2=0x7fd998143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e550 tx=0x7fd99400ea10 comp rx=0 tx=0).stop 2023-11-24T08:11:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 shutdown_connections 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97c06ba10 0x7fd97c06ded0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998142550 0x7fd9981559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd998140f10 0x7fd9981e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 --2- 172.21.15.107:0/3588664366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98400b000 0x7fd998143960 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 >> 172.21.15.107:0/3588664366 conn(0x7fd9980cb830 msgr2=0x7fd9980b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 shutdown_connections 2023-11-24T08:11:38.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.616+0000 7fd99e0b4640 1 -- 172.21.15.107:0/3588664366 wait complete. 2023-11-24T08:11:38.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.848+0000 7f7824f0f640 1 Processor -- start 2023-11-24T08:11:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.848+0000 7f7824f0f640 1 -- start start 2023-11-24T08:11:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.849+0000 7f7824f0f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7820157c70 0x7f7820161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.849+0000 7f7824f0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78200ce0b0 con 0x7f782015e390 2023-11-24T08:11:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.849+0000 7f7824f0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7820161960 con 0x7f78200cf450 2023-11-24T08:11:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.849+0000 7f7824f0f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782015e8d0 con 0x7f7820157c70 2023-11-24T08:11:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7820157c70 0x7f7820161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f782015e390 0x7f78200b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41814/0 (socket says 172.21.15.107:41814) 2023-11-24T08:11:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781ed9d640 1 -- 172.21.15.107:0/887480996 learned_addr learned my addr 172.21.15.107:0/887480996 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781dd9b640 1 -- 172.21.15.107:0/887480996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906364520 0 0) 0x7f78200ce0b0 con 0x7f782015e390 2023-11-24T08:11:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781dd9b640 1 -- 172.21.15.107:0/887480996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f780c003520 con 0x7f782015e390 2023-11-24T08:11:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781e59c640 1 -- 172.21.15.107:0/887480996 >> v1:172.21.15.120:6789/0 conn(0x7f78200cf450 legacy=0x7f782015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781e59c640 1 -- 172.21.15.107:0/887480996 >> v1:172.21.15.107:6789/0 conn(0x7f782015e390 legacy=0x7f78200b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.850+0000 7f781e59c640 1 -- 172.21.15.107:0/887480996 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782015ebb0 con 0x7f7820157c70 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.851+0000 7f781e59c640 1 --2- 172.21.15.107:0/887480996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7820157c70 0x7f7820161df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f78100097a0 tx=0x7f781002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7373db4728b47631 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f781dd9b640 1 -- 172.21.15.107:0/887480996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781002e070 con 0x7f7820157c70 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f781dd9b640 1 -- 172.21.15.107:0/887480996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78100315a0 con 0x7f7820157c70 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f781dd9b640 1 -- 172.21.15.107:0/887480996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810031930 con 0x7f7820157c70 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 -- 172.21.15.107:0/887480996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7820157c70 msgr2=0x7f7820161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 --2- 172.21.15.107:0/887480996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7820157c70 0x7f7820161df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f78100097a0 tx=0x7f781002b840 comp rx=0 tx=0).stop 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 -- 172.21.15.107:0/887480996 shutdown_connections 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 --2- 172.21.15.107:0/887480996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7820157c70 0x7f7820161df0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 -- 172.21.15.107:0/887480996 >> 172.21.15.107:0/887480996 conn(0x7f782006a920 msgr2=0x7f782006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 -- 172.21.15.107:0/887480996 shutdown_connections 2023-11-24T08:11:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.852+0000 7f7824f0f640 1 -- 172.21.15.107:0/887480996 wait complete. 2023-11-24T08:11:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.854+0000 7f7824f0f640 1 Processor -- start 2023-11-24T08:11:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.854+0000 7f7824f0f640 1 -- start start 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820157c70 0x7f7820099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 0x7f78200ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78200cabf0 con 0x7f78200cf450 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782009cea0 con 0x7f7820157c70 2023-11-24T08:11:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f7824f0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78200c8490 con 0x7f782015e390 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.855+0000 7f781e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36730/0 (socket says 172.21.15.107:36730) 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 -- 172.21.15.107:0/743189309 learned_addr learned my addr 172.21.15.107:0/743189309 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781ed9d640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 0x7f78200ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 -- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 msgr2=0x7f78200ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f7817fff640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820157c70 0x7f7820099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 0x7f78200ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 -- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820157c70 msgr2=0x7f7820099140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820157c70 0x7f7820099140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 -- 172.21.15.107:0/743189309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7810009400 con 0x7f78200cf450 2023-11-24T08:11:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781ed9d640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 0x7f78200ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.856+0000 7f781e59c640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f7810034040 tx=0x7f78100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.857+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810032840 con 0x7f78200cf450 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.857+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78100329c0 con 0x7f78200cf450 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.857+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78200c8770 con 0x7f78200cf450 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.857+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810032d70 con 0x7f78200cf450 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.857+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78200c8d30 con 0x7f78200cf450 2023-11-24T08:11:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.858+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78200cf8e0 con 0x7f78200cf450 2023-11-24T08:11:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.859+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f781003b3f0 con 0x7f78200cf450 2023-11-24T08:11:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.860+0000 7f7816ffd640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 0x7f780006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.860+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f78100aa110 con 0x7f78200cf450 2023-11-24T08:11:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.860+0000 7f7817fff640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 0x7f780006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.861+0000 7f7817fff640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 0x7f780006ded0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f78200ad960 tx=0x7f7808008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:38.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:38.864+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78200cf8e0 con 0x7f78200cf450 2023-11-24T08:11:39.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.056+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78200cf8e0 con 0x7f78200cf450 2023-11-24T08:11:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.058+0000 7f7816ffd640 1 -- 172.21.15.107:0/743189309 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f78200cf8e0 con 0x7f78200cf450 2023-11-24T08:11:39.061 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:39.061 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":82,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:34.860017+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":81,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.1c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 msgr2=0x7f780006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 0x7f780006ded0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f78200ad960 tx=0x7f7808008040 comp rx=0 tx=0).stop 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 msgr2=0x7f7820154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f7810034040 tx=0x7f78100046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 shutdown_connections 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f780006ba10 0x7f780006ded0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f782015e390 0x7f78200ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820157c70 0x7f7820099140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 --2- 172.21.15.107:0/743189309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78200cf450 0x7f7820154740 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.061+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 >> 172.21.15.107:0/743189309 conn(0x7f782006a920 msgr2=0x7f78200ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:39.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.062+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 shutdown_connections 2023-11-24T08:11:39.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.062+0000 7f7824f0f640 1 -- 172.21.15.107:0/743189309 wait complete. 2023-11-24T08:11:39.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:10:24.450447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:10:25.621972+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 81, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:11:39.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:11:39.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.294+0000 7fe6ee733640 1 Processor -- start 2023-11-24T08:11:39.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.294+0000 7fe6ee733640 1 -- start start 2023-11-24T08:11:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6ee733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6e80ccbd0 0x7fe6e81660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6ee733640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6e80ccfb0 0x7fe6e8162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6ee733640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e81655f0 con 0x7fe6e8162540 2023-11-24T08:11:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6ee733640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80cc380 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6ee733640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e8162b30 con 0x7fe6e80ccfb0 2023-11-24T08:11:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6e80ccbd0 0x7fe6e81660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6e80ccbd0 0x7fe6e81660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36740/0 (socket says 172.21.15.107:36740) 2023-11-24T08:11:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.296+0000 7fe6e7fff640 1 -- 172.21.15.107:0/803020752 learned_addr learned my addr 172.21.15.107:0/803020752 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.297+0000 7fe6e77fe640 1 --2- 172.21.15.107:0/803020752 >> v2:172.21.15.120:3300/0 conn(0x7fe6e80ccfb0 0x7fe6e8162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.297+0000 7fe6e7fff640 1 -- 172.21.15.107:0/803020752 >> v2:172.21.15.120:3300/0 conn(0x7fe6e80ccfb0 msgr2=0x7fe6e8162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.297+0000 7fe6e7fff640 1 --2- 172.21.15.107:0/803020752 >> v2:172.21.15.120:3300/0 conn(0x7fe6e80ccfb0 0x7fe6e8162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.297+0000 7fe6e7fff640 1 -- 172.21.15.107:0/803020752 >> v1:172.21.15.120:6789/0 conn(0x7fe6e8162540 legacy=0x7fe6e80c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.297+0000 7fe6e7fff640 1 -- 172.21.15.107:0/803020752 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8162e10 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.298+0000 7fe6e7fff640 1 --2- 172.21.15.107:0/803020752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e81660b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe6e806e700 tx=0x7fe6d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670d214081d15018 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.298+0000 7fe6e6ffd640 1 -- 172.21.15.107:0/803020752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8031870 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.298+0000 7fe6e6ffd640 1 -- 172.21.15.107:0/803020752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d80319d0 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.298+0000 7fe6e6ffd640 1 -- 172.21.15.107:0/803020752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8031d60 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 -- 172.21.15.107:0/803020752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e80ccbd0 msgr2=0x7fe6e81660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 --2- 172.21.15.107:0/803020752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e81660b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe6e806e700 tx=0x7fe6d802b910 comp rx=0 tx=0).stop 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 -- 172.21.15.107:0/803020752 shutdown_connections 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 --2- 172.21.15.107:0/803020752 >> v2:172.21.15.120:3300/0 conn(0x7fe6e80ccfb0 0x7fe6e8162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 --2- 172.21.15.107:0/803020752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e81660b0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 -- 172.21.15.107:0/803020752 >> 172.21.15.107:0/803020752 conn(0x7fe6e80b0c50 msgr2=0x7fe6e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 -- 172.21.15.107:0/803020752 shutdown_connections 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.299+0000 7fe6ee733640 1 -- 172.21.15.107:0/803020752 wait complete. 2023-11-24T08:11:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.301+0000 7fe6ee733640 1 Processor -- start 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.301+0000 7fe6ee733640 1 -- start start 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e80c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 0x7fe6e815f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8163050 con 0x7fe6e8162540 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81601b0 con 0x7fe6e80ccbd0 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6ee733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8160490 con 0x7fe6e80ccfb0 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6eccd0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6eccd0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36748/0 (socket says 172.21.15.107:36748) 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.302+0000 7fe6eccd0640 1 -- 172.21.15.107:0/3720728735 learned_addr learned my addr 172.21.15.107:0/3720728735 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6e77fe640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 0x7fe6e815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 -- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 msgr2=0x7fe6e815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 0x7fe6e815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 -- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6e80ccbd0 msgr2=0x7fe6e80c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e80c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 -- 172.21.15.107:0/3720728735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80094d0 con 0x7fe6e8162540 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6e77fe640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 0x7fe6e815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6eccd0640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000c780 tx=0x7fe6e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0003d80 con 0x7fe6e8162540 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e000ce90 con 0x7fe6e8162540 2023-11-24T08:11:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00052b0 con 0x7fe6e8162540 2023-11-24T08:11:39.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.303+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8160770 con 0x7fe6e8162540 2023-11-24T08:11:39.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.304+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e815fec0 con 0x7fe6e8162540 2023-11-24T08:11:39.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.305+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e804e8b0 con 0x7fe6e8162540 2023-11-24T08:11:39.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.306+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fe6e0005410 con 0x7fe6e8162540 2023-11-24T08:11:39.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.307+0000 7fe6e4ff9640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 0x7fe6d406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.307+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe6e0087630 con 0x7fe6e8162540 2023-11-24T08:11:39.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.307+0000 7fe6e7fff640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 0x7fe6d406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.308+0000 7fe6e7fff640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 0x7fe6d406ded0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.311+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6e0053ed0 con 0x7fe6e8162540 2023-11-24T08:11:39.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.494+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6e812c1b0 con 0x7fe6e8162540 2023-11-24T08:11:39.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.494+0000 7fe6e4ff9640 1 -- 172.21.15.107:0/3720728735 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6e00170a0 con 0x7fe6e8162540 2023-11-24T08:11:39.497 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.496+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 msgr2=0x7fe6d406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.496+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 0x7fe6d406ded0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d8030000 comp rx=0 tx=0).stop 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.496+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 msgr2=0x7fe6e80c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.496+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000c780 tx=0x7fe6e000cc40 comp rx=0 tx=0).stop 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 shutdown_connections 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6d406ba10 0x7fe6d406ded0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6e8162540 0x7fe6e80c7cd0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6e80ccfb0 0x7fe6e815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 --2- 172.21.15.107:0/3720728735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6e80ccbd0 0x7fe6e80c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 >> 172.21.15.107:0/3720728735 conn(0x7fe6e80b0c50 msgr2=0x7fe6e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:39.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 shutdown_connections 2023-11-24T08:11:39.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.497+0000 7fe6ee733640 1 -- 172.21.15.107:0/3720728735 wait complete. 2023-11-24T08:11:39.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:11:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.707+0000 7f1deed7c640 1 Processor -- start 2023-11-24T08:11:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.707+0000 7f1deed7c640 1 -- start start 2023-11-24T08:11:39.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1deed7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 0x7f1de806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1deed7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de81605f0 0x7f1de8160a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1deed7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80ce410 con 0x7f1de80d2000 2023-11-24T08:11:39.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1deed7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de8160f80 con 0x7f1de81605f0 2023-11-24T08:11:39.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1deed7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80d06f0 con 0x7f1de80d1c20 2023-11-24T08:11:39.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1ded319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de81605f0 0x7f1de8160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1ded319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de81605f0 0x7f1de8160a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36758/0 (socket says 172.21.15.107:36758) 2023-11-24T08:11:39.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.709+0000 7f1ded319640 1 -- 172.21.15.107:0/195327305 learned_addr learned my addr 172.21.15.107:0/195327305 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1decb18640 1 --2- 172.21.15.107:0/195327305 >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 0x7f1de806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1ded319640 1 -- 172.21.15.107:0/195327305 >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 msgr2=0x7f1de806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:39.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1ded319640 1 --2- 172.21.15.107:0/195327305 >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 0x7f1de806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1ded319640 1 -- 172.21.15.107:0/195327305 >> v1:172.21.15.107:6789/0 conn(0x7f1de80d2000 legacy=0x7f1de80cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1ded319640 1 -- 172.21.15.107:0/195327305 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80d09d0 con 0x7f1de81605f0 2023-11-24T08:11:39.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.710+0000 7f1decb18640 1 --2- 172.21.15.107:0/195327305 >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 0x7f1de806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:39.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.711+0000 7f1ded319640 1 --2- 172.21.15.107:0/195327305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de81605f0 0x7f1de8160a40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1de80b5ef0 tx=0x7f1de002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fbaf839d93b9db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.711+0000 7f1ddf7fe640 1 -- 172.21.15.107:0/195327305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0030860 con 0x7f1de81605f0 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.711+0000 7f1ddf7fe640 1 -- 172.21.15.107:0/195327305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de00309c0 con 0x7f1de81605f0 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.711+0000 7f1ddf7fe640 1 -- 172.21.15.107:0/195327305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0030d50 con 0x7f1de81605f0 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 -- 172.21.15.107:0/195327305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de81605f0 msgr2=0x7f1de8160a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 --2- 172.21.15.107:0/195327305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de81605f0 0x7f1de8160a40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1de80b5ef0 tx=0x7f1de002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 -- 172.21.15.107:0/195327305 shutdown_connections 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 --2- 172.21.15.107:0/195327305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de81605f0 0x7f1de8160a40 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 --2- 172.21.15.107:0/195327305 >> v2:172.21.15.120:3300/0 conn(0x7f1de80d1c20 0x7f1de806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 -- 172.21.15.107:0/195327305 >> 172.21.15.107:0/195327305 conn(0x7f1de808cea0 msgr2=0x7f1de809c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:39.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 -- 172.21.15.107:0/195327305 shutdown_connections 2023-11-24T08:11:39.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.712+0000 7f1deed7c640 1 -- 172.21.15.107:0/195327305 wait complete. 2023-11-24T08:11:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.714+0000 7f1deed7c640 1 Processor -- start 2023-11-24T08:11:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.714+0000 7f1deed7c640 1 -- start start 2023-11-24T08:11:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 0x7f1de814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81605f0 0x7f1de81f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80b8880 con 0x7f1de80d2000 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de814fad0 con 0x7f1de81605f0 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1deed7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80b7c70 con 0x7f1de80d1c20 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36766/0 (socket says 172.21.15.107:36766) 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 -- 172.21.15.107:0/3638654627 learned_addr learned my addr 172.21.15.107:0/3638654627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1decb18640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 0x7f1de814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ded319640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81605f0 0x7f1de81f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 -- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 msgr2=0x7f1de814f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 0x7f1de814f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 -- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81605f0 msgr2=0x7f1de81f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81605f0 0x7f1de81f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1ddffff640 1 -- 172.21.15.107:0/3638654627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00094d0 con 0x7f1de80d2000 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.716+0000 7f1decb18640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 0x7f1de814f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1ddffff640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c450 tx=0x7f1dd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80052d0 con 0x7f1de80d2000 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8005430 con 0x7f1de80d2000 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8012870 con 0x7f1de80d2000 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de80b7f50 con 0x7f1de80d2000 2023-11-24T08:11:39.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.717+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de80b8510 con 0x7f1de80d2000 2023-11-24T08:11:39.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.718+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de804e8b0 con 0x7f1de80d2000 2023-11-24T08:11:39.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.719+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f1dd80129d0 con 0x7f1de80d2000 2023-11-24T08:11:39.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.720+0000 7f1ddd7fa640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 0x7f1dd406df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:39.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.720+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1dd8087d20 con 0x7f1de80d2000 2023-11-24T08:11:39.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.720+0000 7f1decb18640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 0x7f1dd406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:39.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.721+0000 7f1decb18640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 0x7f1dd406df00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1de80cd0f0 tx=0x7f1de40074b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:39.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.727+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dd80545c0 con 0x7f1de80d2000 2023-11-24T08:11:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.924+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1de80b89c0 con 0x7f1de80d2000 2023-11-24T08:11:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.924+0000 7f1ddd7fa640 1 -- 172.21.15.107:0/3638654627 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1de80b89c0 con 0x7f1de80d2000 2023-11-24T08:11:39.927 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:11:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 msgr2=0x7f1dd406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 0x7f1dd406df00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1de80cd0f0 tx=0x7f1de40074b0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 msgr2=0x7f1de81f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c450 tx=0x7f1dd800c910 comp rx=0 tx=0).stop 2023-11-24T08:11:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 shutdown_connections 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dd406ba40 0x7f1dd406df00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81605f0 0x7f1de81f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80d2000 0x7f1de81f66e0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 --2- 172.21.15.107:0/3638654627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80d1c20 0x7f1de814f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 >> 172.21.15.107:0/3638654627 conn(0x7f1de808cea0 msgr2=0x7f1de80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 shutdown_connections 2023-11-24T08:11:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:39.927+0000 7f1deed7c640 1 -- 172.21.15.107:0/3638654627 wait complete. 2023-11-24T08:11:39.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:11:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.158+0000 7f5990e94640 1 Processor -- start 2023-11-24T08:11:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.159+0000 7f5990e94640 1 -- start start 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.160+0000 7f5990e94640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f598c072160 0x7f598c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.160+0000 7f5990e94640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f598c0c49f0 0x7f598c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.160+0000 7f5990e94640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c15ec90 con 0x7f598c06dc40 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.160+0000 7f5990e94640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c06e090 con 0x7f598c0c49f0 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.160+0000 7f5990e94640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0cd530 con 0x7f598c072160 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f5983fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f598c0c49f0 0x7f598c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f598ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f598c06dc40 0x7f598c0cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41832/0 (socket says 172.21.15.107:41832) 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f598ad9d640 1 -- 172.21.15.107:0/2168026700 learned_addr learned my addr 172.21.15.107:0/2168026700 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f598a59c640 1 --2- 172.21.15.107:0/2168026700 >> v2:172.21.15.120:3300/0 conn(0x7f598c072160 0x7f598c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f5983fff640 1 -- 172.21.15.107:0/2168026700 >> v2:172.21.15.120:3300/0 conn(0x7f598c072160 msgr2=0x7f598c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f5983fff640 1 --2- 172.21.15.107:0/2168026700 >> v2:172.21.15.120:3300/0 conn(0x7f598c072160 0x7f598c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f5983fff640 1 -- 172.21.15.107:0/2168026700 >> v1:172.21.15.107:6789/0 conn(0x7f598c06dc40 legacy=0x7f598c0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.161+0000 7f5983fff640 1 -- 172.21.15.107:0/2168026700 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0c4ed0 con 0x7f598c0c49f0 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.162+0000 7f5983fff640 1 --2- 172.21.15.107:0/2168026700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c0c49f0 0x7f598c06d700 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f598c0ccae0 tx=0x7f597402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c25b056cc389f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5989d9b640 1 -- 172.21.15.107:0/2168026700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974031830 con 0x7f598c0c49f0 2023-11-24T08:11:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5989d9b640 1 -- 172.21.15.107:0/2168026700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974031990 con 0x7f598c0c49f0 2023-11-24T08:11:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5989d9b640 1 -- 172.21.15.107:0/2168026700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974031d20 con 0x7f598c0c49f0 2023-11-24T08:11:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 -- 172.21.15.107:0/2168026700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c0c49f0 msgr2=0x7f598c06d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 --2- 172.21.15.107:0/2168026700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c0c49f0 0x7f598c06d700 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f598c0ccae0 tx=0x7f597402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 -- 172.21.15.107:0/2168026700 shutdown_connections 2023-11-24T08:11:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 --2- 172.21.15.107:0/2168026700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c0c49f0 0x7f598c06d700 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 --2- 172.21.15.107:0/2168026700 >> v2:172.21.15.120:3300/0 conn(0x7f598c072160 0x7f598c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 -- 172.21.15.107:0/2168026700 >> 172.21.15.107:0/2168026700 conn(0x7f598c0b0c50 msgr2=0x7f598c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:40.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.163+0000 7f5990e94640 1 -- 172.21.15.107:0/2168026700 shutdown_connections 2023-11-24T08:11:40.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.164+0000 7f5990e94640 1 -- 172.21.15.107:0/2168026700 wait complete. 2023-11-24T08:11:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.165+0000 7f5990e94640 1 Processor -- start 2023-11-24T08:11:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.165+0000 7f5990e94640 1 -- start start 2023-11-24T08:11:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 0x7f598c09c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f598c0c49f0 0x7f598c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c15ac10 con 0x7f598c072160 2023-11-24T08:11:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c15a650 con 0x7f598c0c49f0 2023-11-24T08:11:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.166+0000 7f5990e94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c15a930 con 0x7f598c06dc40 2023-11-24T08:11:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36792/0 (socket says 172.21.15.107:36792) 2023-11-24T08:11:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 -- 172.21.15.107:0/1136635833 learned_addr learned my addr 172.21.15.107:0/1136635833 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f598a59c640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 0x7f598c09c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f598ad9d640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f598c0c49f0 0x7f598c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 -- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 msgr2=0x7f598c09c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 0x7f598c09c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 -- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f598c0c49f0 msgr2=0x7f598c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f598c0c49f0 0x7f598c0999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 -- 172.21.15.107:0/1136635833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5974009400 con 0x7f598c072160 2023-11-24T08:11:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f598a59c640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 0x7f598c09c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.167+0000 7f5983fff640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5974009890 tx=0x7f59740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.168+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597403ba70 con 0x7f598c072160 2023-11-24T08:11:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.168+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597403bbf0 con 0x7f598c072160 2023-11-24T08:11:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.168+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974045400 con 0x7f598c072160 2023-11-24T08:11:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.168+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c15bb80 con 0x7f598c072160 2023-11-24T08:11:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.168+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c099f10 con 0x7f598c072160 2023-11-24T08:11:40.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.169+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c15c040 con 0x7f598c072160 2023-11-24T08:11:40.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.170+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f597404e440 con 0x7f598c072160 2023-11-24T08:11:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.171+0000 7f5982ffd640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 0x7f596c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.172+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f59740aa4c0 con 0x7f598c072160 2023-11-24T08:11:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.172+0000 7f598a59c640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 0x7f596c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.172+0000 7f598a59c640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 0x7f596c06ded0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f597c006fd0 tx=0x7f597c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.176+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5974076d60 con 0x7f598c072160 2023-11-24T08:11:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.367+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f598c04e8b0 con 0x7f598c072160 2023-11-24T08:11:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.367+0000 7f5982ffd640 1 -- 172.21.15.107:0/1136635833 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59740350a0 con 0x7f598c072160 2023-11-24T08:11:40.370 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:11:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 msgr2=0x7f596c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 0x7f596c06ded0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f597c006fd0 tx=0x7f597c008040 comp rx=0 tx=0).stop 2023-11-24T08:11:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 msgr2=0x7f598c156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5974009890 tx=0x7f59740046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 shutdown_connections 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f596c06ba10 0x7f596c06ded0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f598c0c49f0 0x7f598c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f598c072160 0x7f598c156c10 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 --2- 172.21.15.107:0/1136635833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598c06dc40 0x7f598c09c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.370+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 >> 172.21.15.107:0/1136635833 conn(0x7f598c0b0c50 msgr2=0x7f598c15e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.371+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 shutdown_connections 2023-11-24T08:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.371+0000 7f5990e94640 1 -- 172.21.15.107:0/1136635833 wait complete. 2023-11-24T08:11:40.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:11:40.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.599+0000 7f9d621a5640 1 Processor -- start 2023-11-24T08:11:40.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.599+0000 7f9d621a5640 1 -- start start 2023-11-24T08:11:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.600+0000 7f9d621a5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d5c0cf450 0x7f9d5c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.600+0000 7f9d621a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d5c0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.600+0000 7f9d621a5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c0cece0 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.600+0000 7f9d621a5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5c0cfe40 con 0x7f9d5c0cf450 2023-11-24T08:11:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.600+0000 7f9d621a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c156010 con 0x7f9d5c0cdfb0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d5b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9d5c15c7f0 0x7f9d5c06db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41846/0 (socket says 172.21.15.107:41846) 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d5b7fe640 1 -- 172.21.15.107:0/1279973647 learned_addr learned my addr 172.21.15.107:0/1279973647 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d5c0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/1279973647 >> v2:172.21.15.120:3300/0 conn(0x7f9d5c0cf450 0x7f9d5c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155475827 0 0) 0x7f9d5c0cece0 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.601+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48003520 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795180698 0 0) 0x7f9d48003520 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 msgr2=0x7f9d5c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d5c0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 >> v2:172.21.15.120:3300/0 conn(0x7f9d5c0cf450 msgr2=0x7f9d5c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 --2- 172.21.15.107:0/1279973647 >> v2:172.21.15.120:3300/0 conn(0x7f9d5c0cf450 0x7f9d5c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c06d040 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9d4c0019f0 con 0x7f9d5c15c7f0 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 >> v1:172.21.15.107:6789/0 conn(0x7f9d5c15c7f0 legacy=0x7f9d5c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d48003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48003f40 0x7f9d480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0cfe40 con 0x7f9d48004d00 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c156010 con 0x7f9d48003f40 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4800b980 con 0x7f9d5c0cdfb0 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.602+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48003f40 0x7f9d480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5bfff640 1 -- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 msgr2=0x7f9d48003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5b7fe640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d48003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c0cdfb0 0x7f9d48003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5bfff640 1 -- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48003f40 msgr2=0x7f9d480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48003f40 0x7f9d480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d5bfff640 1 -- 172.21.15.107:0/1279973647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48003520 con 0x7f9d48004d00 2023-11-24T08:11:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.603+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48003f40 0x7f9d480047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.604+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d48005180 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009af0 tx=0x7f9d480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c2b7909a3bcd71 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.604+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5402c6b0 con 0x7f9d48004d00 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.604+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5402c810 con 0x7f9d48004d00 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.604+0000 7f9d5affd640 1 -- 172.21.15.107:0/1279973647 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5402cba0 con 0x7f9d48004d00 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 -- 172.21.15.107:0/1279973647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 msgr2=0x7f9d48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 --2- 172.21.15.107:0/1279973647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d48005180 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009af0 tx=0x7f9d480057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d5bfff640 1 -- 172.21.15.107:0/1279973647 reap_dead start 2023-11-24T08:11:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 -- 172.21.15.107:0/1279973647 shutdown_connections 2023-11-24T08:11:40.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 -- 172.21.15.107:0/1279973647 >> 172.21.15.107:0/1279973647 conn(0x7f9d5c06a920 msgr2=0x7f9d5c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:40.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 -- 172.21.15.107:0/1279973647 shutdown_connections 2023-11-24T08:11:40.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.605+0000 7f9d621a5640 1 -- 172.21.15.107:0/1279973647 wait complete. 2023-11-24T08:11:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.607+0000 7f9d621a5640 1 Processor -- start 2023-11-24T08:11:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.607+0000 7f9d621a5640 1 -- start start 2023-11-24T08:11:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c14c1f0 0x7f9d5c1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d5c15ddd0 0x7f9d5c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c15e790 con 0x7f9d48004d00 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c1f6500 con 0x7f9d5c15ddd0 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d621a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c1f67e0 con 0x7f9d5c14c1f0 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36806/0 (socket says 172.21.15.107:36806) 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 -- 172.21.15.107:0/3880577266 learned_addr learned my addr 172.21.15.107:0/3880577266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c14c1f0 0x7f9d5c1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 -- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c14c1f0 msgr2=0x7f9d5c1f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c14c1f0 0x7f9d5c1f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5bfff640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d5c15ddd0 0x7f9d5c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 -- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d5c15ddd0 msgr2=0x7f9d5c15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d5c15ddd0 0x7f9d5c15e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.608+0000 7f9d5b7fe640 1 -- 172.21.15.107:0/3880577266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c1f6ac0 con 0x7f9d48004d00 2023-11-24T08:11:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d5b7fe640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c002140 tx=0x7f9d4c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c004d60 con 0x7f9d48004d00 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0070d0 con 0x7f9d48004d00 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c1f6da0 con 0x7f9d48004d00 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c007440 con 0x7f9d48004d00 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.609+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c14c9e0 con 0x7f9d48004d00 2023-11-24T08:11:40.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.610+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c1f72c0 con 0x7f9d48004d00 2023-11-24T08:11:40.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.611+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f9d4c0075a0 con 0x7f9d48004d00 2023-11-24T08:11:40.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.612+0000 7f9d58ff9640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 0x7f9d3c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:40.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.612+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9d4c088400 con 0x7f9d48004d00 2023-11-24T08:11:40.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.613+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 0x7f9d3c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:40.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.613+0000 7f9d52ffd640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 0x7f9d3c06ded0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0cba20 tx=0x7f9d44008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:40.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.617+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d4c0bde90 con 0x7f9d48004d00 2023-11-24T08:11:40.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.845+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d5c067470 con 0x7f9d48004d00 2023-11-24T08:11:40.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.846+0000 7f9d58ff9640 1 -- 172.21.15.107:0/3880577266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f9d5c1f72c0 con 0x7f9d48004d00 2023-11-24T08:11:40.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:40.849 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:11:40.849 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-11-24T08:11:40.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 msgr2=0x7f9d3c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 0x7f9d3c06ded0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0cba20 tx=0x7f9d44008040 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 msgr2=0x7f9d5c0cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c002140 tx=0x7f9d4c00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 shutdown_connections 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d3c06ba10 0x7f9d3c06ded0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d5c15ddd0 0x7f9d5c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d5c14c1f0 0x7f9d5c1f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 --2- 172.21.15.107:0/3880577266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d48004d00 0x7f9d5c0cadd0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 >> 172.21.15.107:0/3880577266 conn(0x7f9d5c06a920 msgr2=0x7f9d5c0af420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 shutdown_connections 2023-11-24T08:11:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:40.849+0000 7f9d621a5640 1 -- 172.21.15.107:0/3880577266 wait complete. 2023-11-24T08:11:40.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:40.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:11:40.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:11:41.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.073+0000 7f0fc7dbe640 1 Processor -- start 2023-11-24T08:11:41.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.073+0000 7f0fc7dbe640 1 -- start start 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc7dbe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0fc00cf450 0x7f0fc00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc7dbe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc00cdfb0 0x7f0fc00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc7dbe640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc00cece0 con 0x7f0fc015c7f0 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc7dbe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc00cfe40 con 0x7f0fc00cf450 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc7dbe640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0156010 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0fc00cf450 0x7f0fc00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0fc00cf450 0x7f0fc00cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36816/0 (socket says 172.21.15.107:36816) 2023-11-24T08:11:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 -- 172.21.15.107:0/1493770313 learned_addr learned my addr 172.21.15.107:0/1493770313 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 -- 172.21.15.107:0/1493770313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc00cdfb0 msgr2=0x7f0fc00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 --2- 172.21.15.107:0/1493770313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc00cdfb0 0x7f0fc00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.075+0000 7f0fc5359640 1 -- 172.21.15.107:0/1493770313 >> v1:172.21.15.120:6789/0 conn(0x7f0fc015c7f0 legacy=0x7f0fc006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.076+0000 7f0fc5359640 1 -- 172.21.15.107:0/1493770313 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc006d040 con 0x7f0fc00cf450 2023-11-24T08:11:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc5359640 1 --2- 172.21.15.107:0/1493770313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cf450 0x7f0fc00cf900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009870 tx=0x7f0fb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebfea51f8640fac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc4b58640 1 -- 172.21.15.107:0/1493770313 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0031640 con 0x7f0fc00cf450 2023-11-24T08:11:41.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc4b58640 1 -- 172.21.15.107:0/1493770313 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb00317a0 con 0x7f0fc00cf450 2023-11-24T08:11:41.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc4b58640 1 -- 172.21.15.107:0/1493770313 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0031b30 con 0x7f0fc00cf450 2023-11-24T08:11:41.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/1493770313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cf450 msgr2=0x7f0fc00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.077+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/1493770313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cf450 0x7f0fc00cf900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009870 tx=0x7f0fb002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/1493770313 shutdown_connections 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/1493770313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc00cdfb0 0x7f0fc00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/1493770313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cf450 0x7f0fc00cf900 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/1493770313 >> 172.21.15.107:0/1493770313 conn(0x7f0fc006a920 msgr2=0x7f0fc006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/1493770313 shutdown_connections 2023-11-24T08:11:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.078+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/1493770313 wait complete. 2023-11-24T08:11:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.079+0000 7f0fc7dbe640 1 Processor -- start 2023-11-24T08:11:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.080+0000 7f0fc7dbe640 1 -- start start 2023-11-24T08:11:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.080+0000 7f0fc7dbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.080+0000 7f0fc7dbe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00cf450 0x7f0fc00af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc7dbe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015c7f0 0x7f0fc00b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc7dbe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00ca820 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc7dbe640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00cb1f0 con 0x7f0fc00cf450 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc7dbe640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00c9c50 con 0x7f0fc015c7f0 2023-11-24T08:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36822/0 (socket says 172.21.15.107:36822) 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 -- 172.21.15.107:0/2415665103 learned_addr learned my addr 172.21.15.107:0/2415665103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 -- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015c7f0 msgr2=0x7f0fc00b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015c7f0 0x7f0fc00b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5359640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00cf450 0x7f0fc00af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 -- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00cf450 msgr2=0x7f0fc00af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00cf450 0x7f0fc00af3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 -- 172.21.15.107:0/2415665103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00094d0 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.081+0000 7f0fc5b5a640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00b3f0 tx=0x7f0fbc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.082+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc011830 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.082+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc011990 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.082+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc011d40 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.082+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc00c9ef0 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.082+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc00ca4b0 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.083+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc0067470 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.084+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f0fbc01a400 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.085+0000 7f0fb67fc640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 0x7f0fa406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.085+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0fbc0880b0 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.085+0000 7f0fc5359640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 0x7f0fa406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.086+0000 7f0fc5359640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 0x7f0fa406ded0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036000 tx=0x7f0fb003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.089+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fbc054950 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.280+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fc006cd80 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.282+0000 7f0fb67fc640 1 -- 172.21.15.107:0/2415665103 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f0fbc015020 con 0x7f0fc00cdfb0 2023-11-24T08:11:41.284 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:41.284 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":82,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:34.860017+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":81,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.1c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.284+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 msgr2=0x7f0fa406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.284+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 0x7f0fa406ded0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036000 tx=0x7f0fb003c040 comp rx=0 tx=0).stop 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.284+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 msgr2=0x7f0fc00cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.284+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00b3f0 tx=0x7f0fbc00b8b0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 shutdown_connections 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0fa406ba10 0x7f0fa406ded0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015c7f0 0x7f0fc00b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00cf450 0x7f0fc00af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 --2- 172.21.15.107:0/2415665103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc00cdfb0 0x7f0fc00cacb0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 >> 172.21.15.107:0/2415665103 conn(0x7f0fc006a920 msgr2=0x7f0fc006ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 shutdown_connections 2023-11-24T08:11:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.285+0000 7f0fc7dbe640 1 -- 172.21.15.107:0/2415665103 wait complete. 2023-11-24T08:11:41.307 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:11:41.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:11:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.514+0000 7fcc201b0640 1 Processor -- start 2023-11-24T08:11:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.514+0000 7fcc201b0640 1 -- start start 2023-11-24T08:11:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.515+0000 7fcc201b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc180ce6c0 0x7fcc180cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.515+0000 7fcc201b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc180cece0 con 0x7fcc1815c7f0 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.515+0000 7fcc201b0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1806d040 con 0x7fcc180cf450 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.515+0000 7fcc201b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180cfe40 con 0x7fcc180ce6c0 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc180ce6c0 0x7fcc180cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc180ce6c0 0x7fcc180cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36830/0 (socket says 172.21.15.107:36830) 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 -- 172.21.15.107:0/3171789900 learned_addr learned my addr 172.21.15.107:0/3171789900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1df4c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcc1815c7f0 0x7fcc1806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41858/0 (socket says 172.21.15.107:41858) 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 -- 172.21.15.107:0/3171789900 >> v1:172.21.15.120:6789/0 conn(0x7fcc180cf450 legacy=0x7fcc180cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 -- 172.21.15.107:0/3171789900 >> v1:172.21.15.107:6789/0 conn(0x7fcc1815c7f0 legacy=0x7fcc1806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1e74d640 1 -- 172.21.15.107:0/3171789900 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1806d320 con 0x7fcc180ce6c0 2023-11-24T08:11:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.516+0000 7fcc1cf4a640 1 -- 172.21.15.107:0/3171789900 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753028295 0 0) 0x7fcc180cece0 con 0x7fcc1815c7f0 2023-11-24T08:11:41.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.517+0000 7fcc1e74d640 1 --2- 172.21.15.107:0/3171789900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180ce6c0 0x7fcc180cf900 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009e50 tx=0x7fcc1002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1519d79bd2cd7e64 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc1cf4a640 1 -- 172.21.15.107:0/3171789900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1002cb20 con 0x7fcc180ce6c0 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc1cf4a640 1 -- 172.21.15.107:0/3171789900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1002cc80 con 0x7fcc180ce6c0 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc1cf4a640 1 -- 172.21.15.107:0/3171789900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc100343f0 con 0x7fcc180ce6c0 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 -- 172.21.15.107:0/3171789900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180ce6c0 msgr2=0x7fcc180cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 --2- 172.21.15.107:0/3171789900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180ce6c0 0x7fcc180cf900 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009e50 tx=0x7fcc1002bf80 comp rx=0 tx=0).stop 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 -- 172.21.15.107:0/3171789900 shutdown_connections 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 --2- 172.21.15.107:0/3171789900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180ce6c0 0x7fcc180cf900 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 -- 172.21.15.107:0/3171789900 >> 172.21.15.107:0/3171789900 conn(0x7fcc1806a920 msgr2=0x7fcc1806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 -- 172.21.15.107:0/3171789900 shutdown_connections 2023-11-24T08:11:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.518+0000 7fcc201b0640 1 -- 172.21.15.107:0/3171789900 wait complete. 2023-11-24T08:11:41.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.520+0000 7fcc201b0640 1 Processor -- start 2023-11-24T08:11:41.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.520+0000 7fcc201b0640 1 -- start start 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180ce6c0 0x7fcc1815acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc1815c7f0 0x7fcc181564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181f6920 con 0x7fcc180cf450 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18159bd0 con 0x7fcc180ce6c0 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc201b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18159eb0 con 0x7fcc1815c7f0 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc1d74b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc1d74b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36834/0 (socket says 172.21.15.107:36834) 2023-11-24T08:11:41.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.521+0000 7fcc1d74b640 1 -- 172.21.15.107:0/27474224 learned_addr learned my addr 172.21.15.107:0/27474224 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 -- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc1815c7f0 msgr2=0x7fcc181564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc1815c7f0 0x7fcc181564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 -- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180ce6c0 msgr2=0x7fcc1815acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180ce6c0 0x7fcc1815acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 -- 172.21.15.107:0/27474224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009b30 con 0x7fcc180cf450 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc1d74b640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c430 tx=0x7fcc0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08011790 con 0x7fcc180cf450 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc080118f0 con 0x7fcc180cf450 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08011ce0 con 0x7fcc180cf450 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.522+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1815a190 con 0x7fcc180cf450 2023-11-24T08:11:41.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.523+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1815b230 con 0x7fcc180cf450 2023-11-24T08:11:41.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.524+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc181f6a90 con 0x7fcc180cf450 2023-11-24T08:11:41.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.524+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fcc0801c460 con 0x7fcc180cf450 2023-11-24T08:11:41.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.525+0000 7fcc0e7fc640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 0x7fcbfc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.525+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc08088170 con 0x7fcc180cf450 2023-11-24T08:11:41.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.526+0000 7fcc1df4c640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 0x7fcbfc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.526+0000 7fcc1df4c640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 0x7fcbfc06ded0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcc14007fd0 tx=0x7fcc14007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.530+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc08054a50 con 0x7fcc180cf450 2023-11-24T08:11:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.756+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc18067470 con 0x7fcc180cf450 2023-11-24T08:11:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.757+0000 7fcc0e7fc640 1 -- 172.21.15.107:0/27474224 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fcc08058070 con 0x7fcc180cf450 2023-11-24T08:11:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:41.759 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:11:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-11-24T08:11:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 msgr2=0x7fcbfc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 0x7fcbfc06ded0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcc14007fd0 tx=0x7fcc14007960 comp rx=0 tx=0).stop 2023-11-24T08:11:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 msgr2=0x7fcc181554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c430 tx=0x7fcc0800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 shutdown_connections 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcbfc06ba10 0x7fcbfc06ded0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc1815c7f0 0x7fcc181564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc180cf450 0x7fcc181554a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 --2- 172.21.15.107:0/27474224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc180ce6c0 0x7fcc1815acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 >> 172.21.15.107:0/27474224 conn(0x7fcc1806a920 msgr2=0x7fcc180b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 shutdown_connections 2023-11-24T08:11:41.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.760+0000 7fcc201b0640 1 -- 172.21.15.107:0/27474224 wait complete. 2023-11-24T08:11:41.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:11:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.984+0000 7f9835767640 1 Processor -- start 2023-11-24T08:11:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.985+0000 7f9835767640 1 -- start start 2023-11-24T08:11:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.986+0000 7f9835767640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f983015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.986+0000 7f9835767640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983015bf80 con 0x7f9830144b30 2023-11-24T08:11:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.986+0000 7f9835767640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98300b1560 con 0x7f98300b1de0 2023-11-24T08:11:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.986+0000 7f9835767640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98300b1280 con 0x7f983015c410 2023-11-24T08:11:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9830144b30 0x7f98300b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41864/0 (socket says 172.21.15.107:41864) 2023-11-24T08:11:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982ffff640 1 -- 172.21.15.107:0/1229705499 learned_addr learned my addr 172.21.15.107:0/1229705499 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861909493 0 0) 0x7f983015bf80 con 0x7f9830144b30 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982f7fe640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f983015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818003520 con 0x7f9830144b30 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58572324 0 0) 0x7f9818003520 con 0x7f9830144b30 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 msgr2=0x7f983015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f983015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 >> v1:172.21.15.120:6789/0 conn(0x7f98300b1de0 legacy=0x7f98300b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983015b930 con 0x7f9830144b30 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.987+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9820001670 con 0x7f9830144b30 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 >> v1:172.21.15.107:6789/0 conn(0x7f9830144b30 legacy=0x7f98300b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f98180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818004000 0x7f98180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9818004d00 0x7f9818005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98300b1280 con 0x7f9818004d00 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981800bb70 con 0x7f9818004000 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981800bd30 con 0x7f983015c410 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9818004d00 0x7f9818005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982ffff640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f98180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 -- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 msgr2=0x7f98180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983015c410 0x7f98180039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 -- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818004000 msgr2=0x7f98180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f982f7fe640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818004000 0x7f98180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818004000 0x7f98180047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.988+0000 7f9834f66640 1 -- 172.21.15.107:0/1229705499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818003520 con 0x7f9818004d00 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.989+0000 7f9834f66640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9818004d00 0x7f9818005180 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f98280044b0 tx=0x7f98180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9146cbd338fd670 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828030620 con 0x7f9818004d00 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828030780 con 0x7f9818004d00 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f982effd640 1 -- 172.21.15.107:0/1229705499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828030b70 con 0x7f9818004d00 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 -- 172.21.15.107:0/1229705499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9818004d00 msgr2=0x7f9818005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 --2- 172.21.15.107:0/1229705499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9818004d00 0x7f9818005180 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f98280044b0 tx=0x7f98180057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9834f66640 1 -- 172.21.15.107:0/1229705499 reap_dead start 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 -- 172.21.15.107:0/1229705499 shutdown_connections 2023-11-24T08:11:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 -- 172.21.15.107:0/1229705499 >> 172.21.15.107:0/1229705499 conn(0x7f98300cb830 msgr2=0x7f98300cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 -- 172.21.15.107:0/1229705499 shutdown_connections 2023-11-24T08:11:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.990+0000 7f9835767640 1 -- 172.21.15.107:0/1229705499 wait complete. 2023-11-24T08:11:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.992+0000 7f9835767640 1 Processor -- start 2023-11-24T08:11:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.992+0000 7f9835767640 1 -- start start 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98300cdbb0 0x7f983015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98300cdf90 0x7f983015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983013dd80 con 0x7f98300c7050 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983015b080 con 0x7f98300cdf90 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f9835767640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983015eac0 con 0x7f98300cdbb0 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36844/0 (socket says 172.21.15.107:36844) 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.993+0000 7f982f7fe640 1 -- 172.21.15.107:0/2096913011 learned_addr learned my addr 172.21.15.107:0/2096913011 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982ffff640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98300cdbb0 0x7f983015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f9834f66640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98300cdf90 0x7f983015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 -- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98300cdbb0 msgr2=0x7f983015e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98300cdbb0 0x7f983015e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 -- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98300cdf90 msgr2=0x7f983015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98300cdf90 0x7f983015ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 -- 172.21.15.107:0/2096913011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300c3e80 con 0x7f98300c7050 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982f7fe640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f982400e810 tx=0x7f982400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400c670 con 0x7f98300c7050 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.994+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982400c7d0 con 0x7f98300c7050 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.995+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98300c75c0 con 0x7f98300c7050 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.995+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400cb60 con 0x7f98300c7050 2023-11-24T08:11:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.995+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98300c4120 con 0x7f98300c7050 2023-11-24T08:11:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.996+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983004e840 con 0x7f98300c7050 2023-11-24T08:11:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.997+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f982400ccc0 con 0x7f98300c7050 2023-11-24T08:11:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.998+0000 7f982cff9640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 0x7f981c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.998+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9824087f70 con 0x7f98300c7050 2023-11-24T08:11:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.998+0000 7f982ffff640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 0x7f981c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:41.999+0000 7f982ffff640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 0x7f981c06ded0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9820001eb0 tx=0x7f9820009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.002+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9824054810 con 0x7f98300c7050 2023-11-24T08:11:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.213+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f983006d860 con 0x7f98300c7050 2023-11-24T08:11:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.214+0000 7f982cff9640 1 -- 172.21.15.107:0/2096913011 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f9824021020 con 0x7f98300c7050 2023-11-24T08:11:42.217 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:42.217 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:10:26.606485+0000","modified":"2023-11-24T08:10:29.699709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6735":{"gid":6735,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/3083938901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3083938901},{"type":"v1","addr":"172.21.15.107:6837","nonce":3083938901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T08:11:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-11-24T08:11:42.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 msgr2=0x7f981c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 0x7f981c06ded0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9820001eb0 tx=0x7f9820009040 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 msgr2=0x7f983015a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f982400e810 tx=0x7f982400ecd0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 shutdown_connections 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981c06ba10 0x7f981c06ded0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98300cdf90 0x7f983015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98300c7050 0x7f983015a600 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 --2- 172.21.15.107:0/2096913011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98300cdbb0 0x7f983015e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 >> 172.21.15.107:0/2096913011 conn(0x7f98300cb830 msgr2=0x7f9830143dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.217+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 shutdown_connections 2023-11-24T08:11:42.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.218+0000 7f9835767640 1 -- 172.21.15.107:0/2096913011 wait complete. 2023-11-24T08:11:42.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.438+0000 7f5c279ec640 1 Processor -- start 2023-11-24T08:11:42.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.439+0000 7f5c279ec640 1 -- start start 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c279ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c2006de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c279ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200b3a40 con 0x7f5c200b21d0 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c279ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200cdfa0 con 0x7f5c200cc150 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c279ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200cd9e0 con 0x7f5c20159ce0 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c24f87640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c200b21d0 0x7f5c200cba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41870/0 (socket says 172.21.15.107:41870) 2023-11-24T08:11:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.440+0000 7f5c24f87640 1 -- 172.21.15.107:0/4063153244 learned_addr learned my addr 172.21.15.107:0/4063153244 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c25788640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c2006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904713168 0 0) 0x7f5c200b3a40 con 0x7f5c200b21d0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c004630 con 0x7f5c200b21d0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259594040 0 0) 0x7f5c0c004630 con 0x7f5c200b21d0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 msgr2=0x7f5c2006de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c2006de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 >> v1:172.21.15.120:6789/0 conn(0x7f5c200cc150 legacy=0x7f5c2006c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200cdcc0 con 0x7f5c200b21d0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5c10001670 con 0x7f5c200b21d0 2023-11-24T08:11:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 >> v1:172.21.15.107:6789/0 conn(0x7f5c200b21d0 legacy=0x7f5c200cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c0c004ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c0c005030 0x7f5c0c0058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c0c006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200cd9e0 con 0x7f5c0c005e10 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c00cb70 con 0x7f5c0c005030 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.441+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c00cd30 con 0x7f5c20159ce0 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c0c006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25788640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c0c004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c24f87640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c0c005030 0x7f5c0c0058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 -- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 msgr2=0x7f5c0c004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c0c004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 -- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c0c005030 msgr2=0x7f5c0c0058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c0c005030 0x7f5c0c0058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25f89640 1 -- 172.21.15.107:0/4063153244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c004630 con 0x7f5c0c005e10 2023-11-24T08:11:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.442+0000 7f5c25788640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20159ce0 0x7f5c0c004ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:11:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.443+0000 7f5c25f89640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c0c006290 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5c18001e20 tx=0x7f5c0c006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bd39d22b4c934e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18003b70 con 0x7f5c0c005e10 2023-11-24T08:11:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18003cd0 con 0x7f5c0c005e10 2023-11-24T08:11:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c17fff640 1 -- 172.21.15.107:0/4063153244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18005310 con 0x7f5c0c005e10 2023-11-24T08:11:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 -- 172.21.15.107:0/4063153244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 msgr2=0x7f5c0c006290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 --2- 172.21.15.107:0/4063153244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c0c006290 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5c18001e20 tx=0x7f5c0c006900 comp rx=0 tx=0).stop 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c25f89640 1 -- 172.21.15.107:0/4063153244 reap_dead start 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 -- 172.21.15.107:0/4063153244 shutdown_connections 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 -- 172.21.15.107:0/4063153244 >> 172.21.15.107:0/4063153244 conn(0x7f5c2006a920 msgr2=0x7f5c2006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 -- 172.21.15.107:0/4063153244 shutdown_connections 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.444+0000 7f5c279ec640 1 -- 172.21.15.107:0/4063153244 wait complete. 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.446+0000 7f5c279ec640 1 Processor -- start 2023-11-24T08:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.446+0000 7f5c279ec640 1 -- start start 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20155f70 0x7f5c2014c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c20099b50 0x7f5c2009a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c201570f0 con 0x7f5c0c005e10 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2014c970 con 0x7f5c20099b50 2023-11-24T08:11:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c279ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20156470 con 0x7f5c20155f70 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c25788640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c25788640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36862/0 (socket says 172.21.15.107:36862) 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.447+0000 7f5c25788640 1 -- 172.21.15.107:0/599051442 learned_addr learned my addr 172.21.15.107:0/599051442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c24f87640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20155f70 0x7f5c2014c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25f89640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c20099b50 0x7f5c2009a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 -- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20155f70 msgr2=0x7f5c2014c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20155f70 0x7f5c2014c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 -- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c20099b50 msgr2=0x7f5c2009a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c20099b50 0x7f5c2009a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 -- 172.21.15.107:0/599051442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20156750 con 0x7f5c0c005e10 2023-11-24T08:11:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c25788640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0116d0 tx=0x7f5c1c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00f9c0 con 0x7f5c0c005e10 2023-11-24T08:11:42.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c00fb20 con 0x7f5c0c005e10 2023-11-24T08:11:42.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c20156a30 con 0x7f5c0c005e10 2023-11-24T08:11:42.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.448+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00fed0 con 0x7f5c0c005e10 2023-11-24T08:11:42.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.449+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2009a000 con 0x7f5c0c005e10 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.450+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c20067470 con 0x7f5c0c005e10 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.451+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f5c1c015460 con 0x7f5c0c005e10 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.452+0000 7f5c15ffb640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 0x7f5c0406df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.452+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5c1c088a70 con 0x7f5c0c005e10 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.452+0000 7f5c24f87640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 0x7f5c0406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.452+0000 7f5c24f87640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 0x7f5c0406df00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c10001200 tx=0x7f5c10000f70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.456+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c1c055370 con 0x7f5c0c005e10 2023-11-24T08:11:42.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.647+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c200ce150 con 0x7f5c0c005e10 2023-11-24T08:11:42.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.649+0000 7f5c15ffb640 1 -- 172.21.15.107:0/599051442 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f5c1c058990 con 0x7f5c0c005e10 2023-11-24T08:11:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:42.651 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":82,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:34.860017+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:10:24.450447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T08:10:25.621972+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":81,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.1c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 msgr2=0x7f5c0406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 0x7f5c0406df00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c10001200 tx=0x7f5c10000f70 comp rx=0 tx=0).stop 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 msgr2=0x7f5c20157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0116d0 tx=0x7f5c1c011b90 comp rx=0 tx=0).stop 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 shutdown_connections 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c0406ba40 0x7f5c0406df00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c20099b50 0x7f5c2009a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c20155f70 0x7f5c2014c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 --2- 172.21.15.107:0/599051442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c0c005e10 0x7f5c20157510 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 >> 172.21.15.107:0/599051442 conn(0x7f5c2006a920 msgr2=0x7f5c200b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 shutdown_connections 2023-11-24T08:11:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.651+0000 7f5c279ec640 1 -- 172.21.15.107:0/599051442 wait complete. 2023-11-24T08:11:42.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:11:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.877+0000 7fe542475640 1 Processor -- start 2023-11-24T08:11:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.877+0000 7fe542475640 1 -- start start 2023-11-24T08:11:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.879+0000 7fe542475640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe53c072160 0x7fe53c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.879+0000 7fe542475640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c15ec90 con 0x7fe53c0c49f0 2023-11-24T08:11:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.879+0000 7fe542475640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0cd530 con 0x7fe53c06de10 2023-11-24T08:11:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.879+0000 7fe542475640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53c0c4ed0 con 0x7fe53c072160 2023-11-24T08:11:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe533fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe53c0c49f0 0x7fe53c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41872/0 (socket says 172.21.15.107:41872) 2023-11-24T08:11:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe533fff640 1 -- 172.21.15.107:0/1208448164 learned_addr learned my addr 172.21.15.107:0/1208448164 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53bfff640 1 --2- 172.21.15.107:0/1208448164 >> v2:172.21.15.120:3300/0 conn(0x7fe53c072160 0x7fe53c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766293950 0 0) 0x7fe524001110 con 0x7fe53c0c49f0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528003520 con 0x7fe53c0c49f0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179571766 0 0) 0x7fe528003520 con 0x7fe53c0c49f0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 >> v2:172.21.15.120:3300/0 conn(0x7fe53c072160 msgr2=0x7fe53c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.880+0000 7fe53b7fe640 1 --2- 172.21.15.107:0/1208448164 >> v2:172.21.15.120:3300/0 conn(0x7fe53c072160 0x7fe53c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 >> v1:172.21.15.120:6789/0 conn(0x7fe53c06de10 legacy=0x7fe53c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0c51b0 con 0x7fe53c0c49f0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe524001800 con 0x7fe53c0c49f0 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 >> v1:172.21.15.107:6789/0 conn(0x7fe53c0c49f0 legacy=0x7fe53c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe528003970 0x7fe528003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 0x7fe528004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe528005180 0x7fe52800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0c4ed0 con 0x7fe5280047c0 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52800e230 con 0x7fe528003970 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52800e3f0 con 0x7fe528005180 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 0x7fe528004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53bfff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe528003970 0x7fe528003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 -- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe528005180 msgr2=0x7fe52800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe528005180 0x7fe52800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 -- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe528003970 msgr2=0x7fe528003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe528003970 0x7fe528003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe533fff640 1 -- 172.21.15.107:0/1208448164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528003520 con 0x7fe5280047c0 2023-11-24T08:11:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.881+0000 7fe53bfff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe528003970 0x7fe528003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.882+0000 7fe540a12640 1 -- 172.21.15.107:0/1208448164 reap_dead start 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.883+0000 7fe533fff640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 0x7fe528004c40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c190 tx=0x7fe5280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad74ec2ef8dfe55 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.883+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524005ad0 con 0x7fe5280047c0 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524005c50 con 0x7fe5280047c0 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe53b7fe640 1 -- 172.21.15.107:0/1208448164 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52402c7e0 con 0x7fe5280047c0 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 -- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 msgr2=0x7fe528004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 0x7fe528004c40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c190 tx=0x7fe5280057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 -- 172.21.15.107:0/1208448164 shutdown_connections 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 --2- 172.21.15.107:0/1208448164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5280047c0 0x7fe528004c40 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 -- 172.21.15.107:0/1208448164 >> 172.21.15.107:0/1208448164 conn(0x7fe53c0b0c50 msgr2=0x7fe53c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:42.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 -- 172.21.15.107:0/1208448164 shutdown_connections 2023-11-24T08:11:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.884+0000 7fe542475640 1 -- 172.21.15.107:0/1208448164 wait complete. 2023-11-24T08:11:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.886+0000 7fe542475640 1 Processor -- start 2023-11-24T08:11:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.886+0000 7fe542475640 1 -- start start 2023-11-24T08:11:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 0x7fe53c0c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0c82f0 0x7fe53c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0c8770 con 0x7fe53c1596d0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0b4970 con 0x7fe53c0c82f0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe542475640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0b4c50 con 0x7fe5280047c0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.887+0000 7fe533fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36882/0 (socket says 172.21.15.107:36882) 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 -- 172.21.15.107:0/3729624713 learned_addr learned my addr 172.21.15.107:0/3729624713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe53bfff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 0x7fe53c0c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe540a12640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0c82f0 0x7fe53c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 -- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 msgr2=0x7fe53c0c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 0x7fe53c0c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 -- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0c82f0 msgr2=0x7fe53c15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:42.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0c82f0 0x7fe53c15c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 -- 172.21.15.107:0/3729624713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c15b560 con 0x7fe53c1596d0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe53bfff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 0x7fe53c0c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe533fff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe52402c380 tx=0x7fe52402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.888+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52403c040 con 0x7fe53c1596d0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.889+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52402cbf0 con 0x7fe53c1596d0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.889+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524007070 con 0x7fe53c1596d0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.889+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c15b840 con 0x7fe53c1596d0 2023-11-24T08:11:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.889+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c0b4640 con 0x7fe53c1596d0 2023-11-24T08:11:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.890+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c04e8b0 con 0x7fe53c1596d0 2023-11-24T08:11:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.891+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fe524032070 con 0x7fe53c1596d0 2023-11-24T08:11:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.892+0000 7fe5397fa640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 0x7fe51c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.892+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe5240a9480 con 0x7fe53c1596d0 2023-11-24T08:11:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.892+0000 7fe53bfff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 0x7fe51c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.893+0000 7fe53bfff640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 0x7fe51c06ded0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0681f0 tx=0x7fe52c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:42.896+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe524075d80 con 0x7fe53c1596d0 2023-11-24T08:11:43.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.123+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe53c15ec90 con 0x7fe53c1596d0 2023-11-24T08:11:43.127 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:11:43.125+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:11:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.129+0000 7fe5397fa640 1 -- 172.21.15.107:0/3729624713 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe53c15ec90 con 0x7fe53c1596d0 2023-11-24T08:11:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.132+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 msgr2=0x7fe51c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.132+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 0x7fe51c06ded0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0681f0 tx=0x7fe52c008040 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.132+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 msgr2=0x7fe53c0c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.132+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe52402c380 tx=0x7fe52402e070 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 shutdown_connections 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe51c06ba10 0x7fe51c06ded0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0c82f0 0x7fe53c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe53c1596d0 0x7fe53c0c7db0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 --2- 172.21.15.107:0/3729624713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5280047c0 0x7fe53c0c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 >> 172.21.15.107:0/3729624713 conn(0x7fe53c0b0c50 msgr2=0x7fe53c0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 shutdown_connections 2023-11-24T08:11:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.133+0000 7fe542475640 1 -- 172.21.15.107:0/3729624713 wait complete. 2023-11-24T08:11:43.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:11:43.256 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:11:43.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.362+0000 7f2b09456640 1 Processor -- start 2023-11-24T08:11:43.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.362+0000 7f2b09456640 1 -- start start 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.363+0000 7f2b09456640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 0x7f2b040c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b09456640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b04166510 con 0x7f2b041660b0 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b09456640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b041655f0 con 0x7f2b040ccbd0 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b09456640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040cc380 con 0x7f2b040c9650 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b027fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2b041660b0 0x7f2b04162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41888/0 (socket says 172.21.15.107:41888) 2023-11-24T08:11:43.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b027fc640 1 -- 172.21.15.107:0/2930354952 learned_addr learned my addr 172.21.15.107:0/2930354952 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b037fe640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 0x7f2b040c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216454036 0 0) 0x7f2aec001110 con 0x7f2b041660b0 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.364+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af0003520 con 0x7f2b041660b0 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866699340 0 0) 0x7f2af0003520 con 0x7f2b041660b0 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 msgr2=0x7f2b040c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 0x7f2b040c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 >> v1:172.21.15.120:6789/0 conn(0x7f2b040ccbd0 legacy=0x7f2b040ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04162b30 con 0x7f2b041660b0 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2aec001800 con 0x7f2b041660b0 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 >> v1:172.21.15.107:6789/0 conn(0x7f2b041660b0 legacy=0x7f2b04162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2af0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af00047c0 0x7f2af0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 0x7f2af0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040cc380 con 0x7f2af00039a0 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af000ba60 con 0x7f2af00047c0 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af000bc20 con 0x7f2b040c9650 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.365+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2af0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b02ffd640 1 -- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 msgr2=0x7f2af0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:43.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b040c9650 0x7f2af0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b02ffd640 1 -- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af00047c0 msgr2=0x7f2af0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b027fc640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af00047c0 0x7f2af0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af00047c0 0x7f2af0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b02ffd640 1 -- 172.21.15.107:0/2930354952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af0003520 con 0x7f2af00039a0 2023-11-24T08:11:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.366+0000 7f2b027fc640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af00047c0 0x7f2af0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:43.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2af0003dc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2af4001720 tx=0x7f2af00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38f73c1cfb7880c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:43.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40052d0 con 0x7f2af00039a0 2023-11-24T08:11:43.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af402e050 con 0x7f2af00039a0 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b01ffb640 1 -- 172.21.15.107:0/2930354952 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40035f0 con 0x7f2af00039a0 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b09456640 1 -- 172.21.15.107:0/2930354952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 msgr2=0x7f2af0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b09456640 1 --2- 172.21.15.107:0/2930354952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2af0003dc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2af4001720 tx=0x7f2af00057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.367+0000 7f2b037fe640 1 -- 172.21.15.107:0/2930354952 reap_dead start 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.368+0000 7f2b09456640 1 -- 172.21.15.107:0/2930354952 shutdown_connections 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.368+0000 7f2b09456640 1 -- 172.21.15.107:0/2930354952 >> 172.21.15.107:0/2930354952 conn(0x7f2b040b0c50 msgr2=0x7f2b040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.368+0000 7f2b09456640 1 -- 172.21.15.107:0/2930354952 shutdown_connections 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.368+0000 7f2b09456640 1 -- 172.21.15.107:0/2930354952 wait complete. 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.370+0000 7f2b09456640 1 Processor -- start 2023-11-24T08:11:43.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.370+0000 7f2b09456640 1 -- start start 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b04072220 0x7f2b041640a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b041645e0 0x7f2b041eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b04164a60 con 0x7f2af00039a0 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b04158c30 con 0x7f2b041645e0 2023-11-24T08:11:43.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b09456640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040714f0 con 0x7f2b04072220 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36928/0 (socket says 172.21.15.107:36928) 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 -- 172.21.15.107:0/3354554084 learned_addr learned my addr 172.21.15.107:0/3354554084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b027fc640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b04072220 0x7f2b041640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b037fe640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b041645e0 0x7f2b041eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 -- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b04072220 msgr2=0x7f2b041640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b04072220 0x7f2b041640a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 -- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b041645e0 msgr2=0x7f2b041eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b041645e0 0x7f2b041eb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.371+0000 7f2b02ffd640 1 -- 172.21.15.107:0/3354554084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b041edb80 con 0x7f2af00039a0 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2b02ffd640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2af4008fd0 tx=0x7f2af4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40076c0 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4007a30 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4006be0 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b040708e0 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.372+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b04070ea0 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.373+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b041649c0 con 0x7f2af00039a0 2023-11-24T08:11:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.374+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f2af4006d40 con 0x7f2af00039a0 2023-11-24T08:11:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.375+0000 7f2afb7fe640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 0x7f2ae406df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.375+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2af40aa330 con 0x7f2af00039a0 2023-11-24T08:11:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.376+0000 7f2b027fc640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 0x7f2ae406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.376+0000 7f2b027fc640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 0x7f2ae406df00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001eb0 tx=0x7f2aec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:43.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.380+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2af40dfe90 con 0x7f2af00039a0 2023-11-24T08:11:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:43.606+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2b0404e8b0 con 0x7f2af00039a0 2023-11-24T08:11:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.132+0000 7f2afb7fe640 1 -- 172.21.15.107:0/3354554084 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2af4076c30 con 0x7f2af00039a0 2023-11-24T08:11:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 msgr2=0x7f2ae406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 0x7f2ae406df00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001eb0 tx=0x7f2aec009040 comp rx=0 tx=0).stop 2023-11-24T08:11:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 msgr2=0x7f2b041582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2af4008fd0 tx=0x7f2af4001120 comp rx=0 tx=0).stop 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 shutdown_connections 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae406ba40 0x7f2ae406df00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b041645e0 0x7f2b041eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b04072220 0x7f2b041640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 --2- 172.21.15.107:0/3354554084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af00039a0 0x7f2b041582b0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 >> 172.21.15.107:0/3354554084 conn(0x7f2b040b0c50 msgr2=0x7f2b040adb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 shutdown_connections 2023-11-24T08:11:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.136+0000 7f2b09456640 1 -- 172.21.15.107:0/3354554084 wait complete. 2023-11-24T08:11:44.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:11:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.366+0000 7f3f84fd5640 1 Processor -- start 2023-11-24T08:11:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.366+0000 7f3f84fd5640 1 -- start start 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.367+0000 7f3f84fd5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f800b2250 0x7f3f8015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.367+0000 7f3f84fd5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f800b2aa0 0x7f3f8015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f84fd5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8015ab80 con 0x7f3f800b2630 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f84fd5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8015a990 con 0x7f3f800b2250 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f84fd5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8015e680 con 0x7f3f800b2aa0 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f800b2250 0x7f3f8015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f800b2250 0x7f3f8015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36932/0 (socket says 172.21.15.107:36932) 2023-11-24T08:11:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 -- 172.21.15.107:0/3971327027 learned_addr learned my addr 172.21.15.107:0/3971327027 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 -- 172.21.15.107:0/3971327027 >> v2:172.21.15.120:3300/0 conn(0x7f3f800b2aa0 msgr2=0x7f3f8015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 --2- 172.21.15.107:0/3971327027 >> v2:172.21.15.120:3300/0 conn(0x7f3f800b2aa0 0x7f3f8015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 -- 172.21.15.107:0/3971327027 >> v1:172.21.15.107:6789/0 conn(0x7f3f800b2630 legacy=0x7f3f800ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.368+0000 7f3f7e59c640 1 -- 172.21.15.107:0/3971327027 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8015e960 con 0x7f3f800b2250 2023-11-24T08:11:44.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.369+0000 7f3f7e59c640 1 --2- 172.21.15.107:0/3971327027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f8015df80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b2fa0 tx=0x7f3f7002e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ce3fb86e3b2202 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.370+0000 7f3f7dd9b640 1 -- 172.21.15.107:0/3971327027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70036440 con 0x7f3f800b2250 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.370+0000 7f3f7dd9b640 1 -- 172.21.15.107:0/3971327027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f700365a0 con 0x7f3f800b2250 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.370+0000 7f3f7dd9b640 1 -- 172.21.15.107:0/3971327027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70036930 con 0x7f3f800b2250 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.370+0000 7f3f84fd5640 1 -- 172.21.15.107:0/3971327027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 msgr2=0x7f3f8015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.370+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/3971327027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f8015df80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b2fa0 tx=0x7f3f7002e180 comp rx=0 tx=0).stop 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 -- 172.21.15.107:0/3971327027 shutdown_connections 2023-11-24T08:11:44.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/3971327027 >> v2:172.21.15.120:3300/0 conn(0x7f3f800b2aa0 0x7f3f8015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/3971327027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f8015df80 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 -- 172.21.15.107:0/3971327027 >> 172.21.15.107:0/3971327027 conn(0x7f3f8006a920 msgr2=0x7f3f8006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:44.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 -- 172.21.15.107:0/3971327027 shutdown_connections 2023-11-24T08:11:44.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.371+0000 7f3f84fd5640 1 -- 172.21.15.107:0/3971327027 wait complete. 2023-11-24T08:11:44.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.372+0000 7f3f84fd5640 1 Processor -- start 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.373+0000 7f3f84fd5640 1 -- start start 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.373+0000 7f3f84fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.373+0000 7f3f84fd5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f800b2630 0x7f3f801f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f84fd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f800b2aa0 0x7f3f8015b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f84fd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801f3c20 con 0x7f3f800b2250 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f84fd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800aeb50 con 0x7f3f800b2aa0 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f84fd5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800c9840 con 0x7f3f800b2630 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36944/0 (socket says 172.21.15.107:36944) 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 -- 172.21.15.107:0/1001843818 learned_addr learned my addr 172.21.15.107:0/1001843818 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:44.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 -- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f800b2630 msgr2=0x7f3f801f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f800b2630 0x7f3f801f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 -- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f800b2aa0 msgr2=0x7f3f8015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f800b2aa0 0x7f3f8015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 -- 172.21.15.107:0/1001843818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7000b770 con 0x7f3f800b2250 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.374+0000 7f3f7e59c640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3f7002e1c0 tx=0x7f3f700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.375+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70034b80 con 0x7f3f800b2250 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.375+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70034d00 con 0x7f3f800b2250 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.375+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700416a0 con 0x7f3f800b2250 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.375+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f800cd7c0 con 0x7f3f800b2250 2023-11-24T08:11:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.375+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f801f3e50 con 0x7f3f800b2250 2023-11-24T08:11:44.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.376+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80067470 con 0x7f3f800b2250 2023-11-24T08:11:44.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.377+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f3f70049360 con 0x7f3f800b2250 2023-11-24T08:11:44.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.378+0000 7f3f76ffd640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 0x7f3f6006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:44.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.378+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3f700aa760 con 0x7f3f800b2250 2023-11-24T08:11:44.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.379+0000 7f3f77fff640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 0x7f3f6006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:44.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.379+0000 7f3f77fff640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 0x7f3f6006ded0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3f68000f60 tx=0x7f3f680079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:44.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.383+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f70077060 con 0x7f3f800b2250 2023-11-24T08:11:44.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:44.572+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3f800ca860 con 0x7f3f800b2250 2023-11-24T08:11:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.144+0000 7f3f76ffd640 1 -- 172.21.15.107:0/1001843818 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3f7007a680 con 0x7f3f800b2250 2023-11-24T08:11:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:11:45.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 msgr2=0x7f3f6006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 0x7f3f6006ded0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3f68000f60 tx=0x7f3f680079b0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 msgr2=0x7f3f801f36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3f7002e1c0 tx=0x7f3f700046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 shutdown_connections 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f6006ba10 0x7f3f6006ded0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f800b2aa0 0x7f3f8015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f800b2630 0x7f3f801f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 --2- 172.21.15.107:0/1001843818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f800b2250 0x7f3f801f36e0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 >> 172.21.15.107:0/1001843818 conn(0x7f3f8006a920 msgr2=0x7f3f8015cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 shutdown_connections 2023-11-24T08:11:45.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.148+0000 7f3f84fd5640 1 -- 172.21.15.107:0/1001843818 wait complete. 2023-11-24T08:11:45.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.386+0000 7f6f91d1d640 1 Processor -- start 2023-11-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.387+0000 7f6f91d1d640 1 -- start start 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f91d1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f8c0b2250 0x7f6f8c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f91d1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c15ab80 con 0x7f6f8c0b2630 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f91d1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c15e680 con 0x7f6f8c0b2aa0 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f91d1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8c15a990 con 0x7f6f8c0b2250 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f8b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f8c0b2250 0x7f6f8c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f8b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f8c0b2250 0x7f6f8c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36952/0 (socket says 172.21.15.107:36952) 2023-11-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f83fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f8c0b2630 0x7f6f8c0ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41904/0 (socket says 172.21.15.107:41904) 2023-11-24T08:11:45.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.388+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/605157202 learned_addr learned my addr 172.21.15.107:0/605157202 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:45.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.389+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/605157202 >> v1:172.21.15.120:6789/0 conn(0x7f6f8c0b2aa0 legacy=0x7f6f8c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:45.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.389+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/605157202 >> v1:172.21.15.107:6789/0 conn(0x7f6f8c0b2630 legacy=0x7f6f8c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.389+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/605157202 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c15e960 con 0x7f6f8c0b2250 2023-11-24T08:11:45.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.390+0000 7f6f8b7fe640 1 --2- 172.21.15.107:0/605157202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c15df80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c15ad70 tx=0x7f6f7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef52b2aa7f63f4f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:45.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.390+0000 7f6f8affd640 1 -- 172.21.15.107:0/605157202 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c030840 con 0x7f6f8c0b2250 2023-11-24T08:11:45.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.390+0000 7f6f8affd640 1 -- 172.21.15.107:0/605157202 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c0309a0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.390+0000 7f6f8affd640 1 -- 172.21.15.107:0/605157202 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c030d30 con 0x7f6f8c0b2250 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 -- 172.21.15.107:0/605157202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 msgr2=0x7f6f8c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/605157202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c15df80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c15ad70 tx=0x7f6f7c02b980 comp rx=0 tx=0).stop 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 -- 172.21.15.107:0/605157202 shutdown_connections 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/605157202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c15df80 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 -- 172.21.15.107:0/605157202 >> 172.21.15.107:0/605157202 conn(0x7f6f8c06a920 msgr2=0x7f6f8c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 -- 172.21.15.107:0/605157202 shutdown_connections 2023-11-24T08:11:45.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.391+0000 7f6f91d1d640 1 -- 172.21.15.107:0/605157202 wait complete. 2023-11-24T08:11:45.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.393+0000 7f6f91d1d640 1 Processor -- start 2023-11-24T08:11:45.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.393+0000 7f6f91d1d640 1 -- start start 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c0b2630 0x7f6f8c142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f8c0b2aa0 0x7f6f8c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c1693c0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c142580 con 0x7f6f8c0b2630 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f91d1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c1690e0 con 0x7f6f8c0b2aa0 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36956/0 (socket says 172.21.15.107:36956) 2023-11-24T08:11:45.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/3903756330 learned_addr learned my addr 172.21.15.107:0/3903756330 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f83fff640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c0b2630 0x7f6f8c142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f8c0b2aa0 msgr2=0x7f6f8c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8bfff640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f8c0b2aa0 0x7f6f8c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f8c0b2aa0 0x7f6f8c1541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c0b2630 msgr2=0x7f6f8c142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c0b2630 0x7f6f8c142040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.394+0000 7f6f8b7fe640 1 -- 172.21.15.107:0/3903756330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0094d0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f8b7fe640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c039d40 con 0x7f6f8c0b2250 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c043400 con 0x7f6f8c0b2250 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c039ea0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c0cd7c0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.395+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c0ae140 con 0x7f6f8c0b2250 2023-11-24T08:11:45.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.396+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c0adcc0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.397+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f6f7c0437b0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.398+0000 7f6f88ff9640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 0x7f6f6c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:45.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.398+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f6f7c03e070 con 0x7f6f8c0b2250 2023-11-24T08:11:45.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.398+0000 7f6f83fff640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 0x7f6f6c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:45.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.399+0000 7f6f83fff640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 0x7f6f6c06df00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c14c800 tx=0x7f6f74007c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.403+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f8c0adcc0 con 0x7f6f8c0b2250 2023-11-24T08:11:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:45.589+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6f8c0adcc0 con 0x7f6f8c0b2250 2023-11-24T08:11:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.153+0000 7f6f88ff9640 1 -- 172.21.15.107:0/3903756330 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6f8c0adcc0 con 0x7f6f8c0b2250 2023-11-24T08:11:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:11:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 msgr2=0x7f6f6c06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 0x7f6f6c06df00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c14c800 tx=0x7f6f74007c80 comp rx=0 tx=0).stop 2023-11-24T08:11:46.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 msgr2=0x7f6f8c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c004730 comp rx=0 tx=0).stop 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 shutdown_connections 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f6c06ba40 0x7f6f6c06df00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f8c0b2aa0 0x7f6f8c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c0b2630 0x7f6f8c142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 --2- 172.21.15.107:0/3903756330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f8c0b2250 0x7f6f8c0abfc0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.161+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 >> 172.21.15.107:0/3903756330 conn(0x7f6f8c06a920 msgr2=0x7f6f8c0cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.165+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 shutdown_connections 2023-11-24T08:11:46.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.165+0000 7f6f91d1d640 1 -- 172.21.15.107:0/3903756330 wait complete. 2023-11-24T08:11:46.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:11:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.373+0000 7f8843e5c640 1 Processor -- start 2023-11-24T08:11:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.373+0000 7f8843e5c640 1 -- start start 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f8843e5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 0x7f883c166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f8843e5c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c15eea0 con 0x7f883c0cc0f0 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f8843e5c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c06d600 con 0x7f883c15b440 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f8843e5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c0c5e00 con 0x7f883c0c59b0 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f88413f7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f883c0cc0f0 0x7f883c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41906/0 (socket says 172.21.15.107:41906) 2023-11-24T08:11:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.375+0000 7f88413f7640 1 -- 172.21.15.107:0/1728671907 learned_addr learned my addr 172.21.15.107:0/1728671907 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f88423f9640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 0x7f883c166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452870599 0 0) 0x7f883c15eea0 con 0x7f883c0cc0f0 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882c003520 con 0x7f883c0cc0f0 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3678426347 0 0) 0x7f882c003520 con 0x7f883c0cc0f0 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 msgr2=0x7f883c166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 0x7f883c166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 >> v1:172.21.15.120:6789/0 conn(0x7f883c15b440 legacy=0x7f883c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c0abf30 con 0x7f883c0cc0f0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8828001670 con 0x7f883c0cc0f0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 >> v1:172.21.15.107:6789/0 conn(0x7f883c0cc0f0 legacy=0x7f883c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f882c0039a0 0x7f882c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 0x7f882c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882c004d00 0x7f882c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c00bb10 con 0x7f882c0039a0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c00bc50 con 0x7f882c004d00 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.376+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c00be10 con 0x7f883c0c59b0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f882c0039a0 0x7f882c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f88423f9640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882c004d00 0x7f882c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 -- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 msgr2=0x7f882c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f883c0c59b0 0x7f882c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 -- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882c004d00 msgr2=0x7f882c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882c004d00 0x7f882c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.377+0000 7f8841bf8640 1 -- 172.21.15.107:0/1728671907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c003520 con 0x7f882c0039a0 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.378+0000 7f8841bf8640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f882c0039a0 0x7f882c003dc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8830001e50 tx=0x7f882c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f838a004ba68466 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.378+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830003b70 con 0x7f882c0039a0 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.378+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8830003cd0 con 0x7f882c0039a0 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.378+0000 7f8840bf6640 1 -- 172.21.15.107:0/1728671907 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830005310 con 0x7f882c0039a0 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 -- 172.21.15.107:0/1728671907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f882c0039a0 msgr2=0x7f882c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 --2- 172.21.15.107:0/1728671907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f882c0039a0 0x7f882c003dc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8830001e50 tx=0x7f882c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f88423f9640 1 -- 172.21.15.107:0/1728671907 reap_dead start 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 -- 172.21.15.107:0/1728671907 shutdown_connections 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 -- 172.21.15.107:0/1728671907 >> 172.21.15.107:0/1728671907 conn(0x7f883c0b0c50 msgr2=0x7f883c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 -- 172.21.15.107:0/1728671907 shutdown_connections 2023-11-24T08:11:46.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.379+0000 7f8843e5c640 1 -- 172.21.15.107:0/1728671907 wait complete. 2023-11-24T08:11:46.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.381+0000 7f8843e5c640 1 Processor -- start 2023-11-24T08:11:46.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.381+0000 7f8843e5c640 1 -- start start 2023-11-24T08:11:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882c0039a0 0x7f883c1f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f883c145670 0x7f883c1eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c09b140 con 0x7f883c1ef780 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c09b6c0 con 0x7f883c145670 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f8843e5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c15e8e0 con 0x7f882c0039a0 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f88423f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f88423f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36972/0 (socket says 172.21.15.107:36972) 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f88423f9640 1 -- 172.21.15.107:0/2129061573 learned_addr learned my addr 172.21.15.107:0/2129061573 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.382+0000 7f88413f7640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f883c145670 0x7f883c1eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f8841bf8640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882c0039a0 0x7f883c1f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 -- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882c0039a0 msgr2=0x7f883c1f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882c0039a0 0x7f883c1f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 -- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f883c145670 msgr2=0x7f883c1eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f883c145670 0x7f883c1eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 -- 172.21.15.107:0/2129061573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c1f3090 con 0x7f883c1ef780 2023-11-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f88423f9640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f883c06e890 tx=0x7f8834010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88340148c0 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8834014a20 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c15e320 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834014dd0 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.383+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c09b350 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.385+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c09b870 con 0x7f883c1ef780 2023-11-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.386+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f8834024390 con 0x7f883c1ef780 2023-11-24T08:11:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.387+0000 7f883a7fc640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 0x7f882006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.387+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f88340882d0 con 0x7f883c1ef780 2023-11-24T08:11:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.387+0000 7f8841bf8640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 0x7f882006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.388+0000 7f8841bf8640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 0x7f882006df00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8830034000 tx=0x7f8830009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.391+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f883c09b870 con 0x7f883c1ef780 2023-11-24T08:11:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.579+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f883c09b870 con 0x7f883c1ef780 2023-11-24T08:11:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.586+0000 7f883a7fc640 1 -- 172.21.15.107:0/2129061573 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f883c09b870 con 0x7f883c1ef780 2023-11-24T08:11:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 msgr2=0x7f882006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 0x7f882006df00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8830034000 tx=0x7f8830009030 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 msgr2=0x7f883c1edd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f883c06e890 tx=0x7f8834010a10 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 shutdown_connections 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f882006ba40 0x7f882006df00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883c1ef780 0x7f883c1edd00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f883c145670 0x7f883c1eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 --2- 172.21.15.107:0/2129061573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882c0039a0 0x7f883c1f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 >> 172.21.15.107:0/2129061573 conn(0x7f883c0b0c50 msgr2=0x7f883c071800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 shutdown_connections 2023-11-24T08:11:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.588+0000 7f8843e5c640 1 -- 172.21.15.107:0/2129061573 wait complete. 2023-11-24T08:11:46.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:11:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.807+0000 7f64a0a48640 1 Processor -- start 2023-11-24T08:11:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.807+0000 7f64a0a48640 1 -- start start 2023-11-24T08:11:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.808+0000 7f64a0a48640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f649c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f64a0a48640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c15ec90 con 0x7f649c0c49f0 2023-11-24T08:11:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f64a0a48640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0cd530 con 0x7f649c06de10 2023-11-24T08:11:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f64a0a48640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0c4ed0 con 0x7f649c072160 2023-11-24T08:11:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f6499d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f649c0c49f0 0x7f649c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41910/0 (socket says 172.21.15.107:41910) 2023-11-24T08:11:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f6499d9b640 1 -- 172.21.15.107:0/2127943487 learned_addr learned my addr 172.21.15.107:0/2127943487 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f649a59c640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f649c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507587567 0 0) 0x7f6490001110 con 0x7f649c0c49f0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.809+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6484003520 con 0x7f649c0c49f0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406300611 0 0) 0x7f6484003520 con 0x7f649c0c49f0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 msgr2=0x7f649c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f649c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 >> v1:172.21.15.120:6789/0 conn(0x7f649c06de10 legacy=0x7f649c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0c51b0 con 0x7f649c0c49f0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6490001800 con 0x7f649c0c49f0 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 >> v1:172.21.15.107:6789/0 conn(0x7f649c0c49f0 legacy=0x7f649c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f64840039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f64840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6484004d00 0x7f6484005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0c4ed0 con 0x7f6484003f70 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648400bac0 con 0x7f6484004d00 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648400bc80 con 0x7f649c072160 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f6499d9b640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f64840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.810+0000 7f649a59c640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f64840039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f649ad9d640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6484004d00 0x7f6484005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f6499d9b640 1 -- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 msgr2=0x7f64840039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f6499d9b640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c072160 0x7f64840039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f6499d9b640 1 -- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6484004d00 msgr2=0x7f6484005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f6499d9b640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6484004d00 0x7f6484005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f6499d9b640 1 -- 172.21.15.107:0/2127943487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484003520 con 0x7f6484003f70 2023-11-24T08:11:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.811+0000 7f649ad9d640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6484004d00 0x7f6484005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.812+0000 7f6499d9b640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f64840047c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6490007d50 tx=0x7f64840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688c0a0d61a183e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.812+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649002eaf0 con 0x7f6484003f70 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.812+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649002ec70 con 0x7f6484003f70 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.812+0000 7f649959a640 1 -- 172.21.15.107:0/2127943487 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900058d0 con 0x7f6484003f70 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 -- 172.21.15.107:0/2127943487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 msgr2=0x7f64840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 --2- 172.21.15.107:0/2127943487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f64840047c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6490007d50 tx=0x7f64840057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f649ad9d640 1 -- 172.21.15.107:0/2127943487 reap_dead start 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 -- 172.21.15.107:0/2127943487 shutdown_connections 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 -- 172.21.15.107:0/2127943487 >> 172.21.15.107:0/2127943487 conn(0x7f649c0b0c50 msgr2=0x7f649c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 -- 172.21.15.107:0/2127943487 shutdown_connections 2023-11-24T08:11:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.813+0000 7f64a0a48640 1 -- 172.21.15.107:0/2127943487 wait complete. 2023-11-24T08:11:46.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.815+0000 7f64a0a48640 1 Processor -- start 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.815+0000 7f64a0a48640 1 -- start start 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649c0c60d0 0x7f649c0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c0c6550 0x7f649c0ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c159100 con 0x7f6484003f70 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0c7730 con 0x7f649c0c60d0 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f64a0a48640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0c6b20 con 0x7f649c0c6550 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36988/0 (socket says 172.21.15.107:36988) 2023-11-24T08:11:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 -- 172.21.15.107:0/1757114595 learned_addr learned my addr 172.21.15.107:0/1757114595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:46.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 -- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c0c6550 msgr2=0x7f649c0ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649ad9d640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c0c6550 0x7f649c0ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f6499d9b640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649c0c60d0 0x7f649c0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c0c6550 0x7f649c0ad990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 -- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649c0c60d0 msgr2=0x7f649c0ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649c0c60d0 0x7f649c0ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.816+0000 7f649a59c640 1 -- 172.21.15.107:0/1757114595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0c6e00 con 0x7f6484003f70 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f649a59c640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f648c011630 tx=0x7f648c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c00f920 con 0x7f6484003f70 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c00fa80 con 0x7f6484003f70 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c00fdf0 con 0x7f6484003f70 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c0c70e0 con 0x7f6484003f70 2023-11-24T08:11:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.817+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c1593e0 con 0x7f6484003f70 2023-11-24T08:11:46.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.818+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c06ef60 con 0x7f6484003f70 2023-11-24T08:11:46.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.819+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f648c015460 con 0x7f6484003f70 2023-11-24T08:11:46.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.820+0000 7f648affd640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 0x7f647c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:46.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.820+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f648c0886b0 con 0x7f6484003f70 2023-11-24T08:11:46.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.820+0000 7f6499d9b640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 0x7f647c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:46.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.821+0000 7f6499d9b640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 0x7f647c06df00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6490036000 tx=0x7f649003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:46.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:46.824+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f648c054fb0 con 0x7f6484003f70 2023-11-24T08:11:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.012+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f649c04e8b0 con 0x7f6484003f70 2023-11-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.018+0000 7f648affd640 1 -- 172.21.15.107:0/1757114595 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f648c0220b0 con 0x7f6484003f70 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.020+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 msgr2=0x7f647c06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.020+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 0x7f647c06df00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6490036000 tx=0x7f649003b040 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.020+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 msgr2=0x7f649c158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.020+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f648c011630 tx=0x7f648c011af0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 shutdown_connections 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647c06ba40 0x7f647c06df00 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649c0c6550 0x7f649c0ad990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649c0c60d0 0x7f649c0ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 --2- 172.21.15.107:0/1757114595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6484003f70 0x7f649c158bc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 >> 172.21.15.107:0/1757114595 conn(0x7f649c0b0c50 msgr2=0x7f649c0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 shutdown_connections 2023-11-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.021+0000 7f64a0a48640 1 -- 172.21.15.107:0/1757114595 wait complete. 2023-11-24T08:11:47.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete' 2023-11-24T08:11:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.235+0000 7f15b5c1a640 1 Processor -- start 2023-11-24T08:11:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.235+0000 7f15b5c1a640 1 -- start start 2023-11-24T08:11:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.236+0000 7f15b5c1a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15b0144b30 0x7f15b015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.236+0000 7f15b5c1a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f15b015c410 0x7f15b00b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.236+0000 7f15b5c1a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b015bf80 con 0x7f15b00b1de0 2023-11-24T08:11:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.236+0000 7f15b5c1a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b00b1560 con 0x7f15b0144b30 2023-11-24T08:11:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.236+0000 7f15b5c1a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b00b0fb0 con 0x7f15b015c410 2023-11-24T08:11:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b4c18640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15b0144b30 0x7f15b015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b5419640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15b00b1de0 0x7f15b006c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41926/0 (socket says 172.21.15.107:41926) 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b5419640 1 -- 172.21.15.107:0/4150671206 learned_addr learned my addr 172.21.15.107:0/4150671206 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b4c18640 1 -- 172.21.15.107:0/4150671206 >> v2:172.21.15.120:3300/0 conn(0x7f15b015c410 msgr2=0x7f15b00b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b4c18640 1 --2- 172.21.15.107:0/4150671206 >> v2:172.21.15.120:3300/0 conn(0x7f15b015c410 0x7f15b00b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b4c18640 1 -- 172.21.15.107:0/4150671206 >> v1:172.21.15.107:6789/0 conn(0x7f15b00b1de0 legacy=0x7f15b006c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15b4c18640 1 -- 172.21.15.107:0/4150671206 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00b1280 con 0x7f15b0144b30 2023-11-24T08:11:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.237+0000 7f15af7fe640 1 -- 172.21.15.107:0/4150671206 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115896128 0 0) 0x7f15b015bf80 con 0x7f15b00b1de0 2023-11-24T08:11:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.238+0000 7f15b4c18640 1 --2- 172.21.15.107:0/4150671206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b0144b30 0x7f15b015f920 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f15b00cc7a0 tx=0x7f15a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6725cf482c897b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.238+0000 7f15af7fe640 1 -- 172.21.15.107:0/4150671206 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4031830 con 0x7f15b0144b30 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15af7fe640 1 -- 172.21.15.107:0/4150671206 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a4031990 con 0x7f15b0144b30 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15af7fe640 1 -- 172.21.15.107:0/4150671206 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4031d20 con 0x7f15b0144b30 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 -- 172.21.15.107:0/4150671206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b0144b30 msgr2=0x7f15b015f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/4150671206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b0144b30 0x7f15b015f920 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f15b00cc7a0 tx=0x7f15a402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 -- 172.21.15.107:0/4150671206 shutdown_connections 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/4150671206 >> v2:172.21.15.120:3300/0 conn(0x7f15b015c410 0x7f15b00b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/4150671206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b0144b30 0x7f15b015f920 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 -- 172.21.15.107:0/4150671206 >> 172.21.15.107:0/4150671206 conn(0x7f15b00cb830 msgr2=0x7f15b00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 -- 172.21.15.107:0/4150671206 shutdown_connections 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.239+0000 7f15b5c1a640 1 -- 172.21.15.107:0/4150671206 wait complete. 2023-11-24T08:11:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.241+0000 7f15b5c1a640 1 Processor -- start 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.241+0000 7f15b5c1a640 1 -- start start 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b0144b30 0x7f15b01323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b015c410 0x7f15b00c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0130860 con 0x7f15b00b1de0 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0133360 con 0x7f15b015c410 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b5c1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0132da0 con 0x7f15b0144b30 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b4c18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b4c18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37014/0 (socket says 172.21.15.107:37014) 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.242+0000 7f15b4c18640 1 -- 172.21.15.107:0/1291061383 learned_addr learned my addr 172.21.15.107:0/1291061383 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 -- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b0144b30 msgr2=0x7f15b01323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b0144b30 0x7f15b01323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 -- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b015c410 msgr2=0x7f15b00c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b015c410 0x7f15b00c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 -- 172.21.15.107:0/1291061383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4009400 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b4c18640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009890 tx=0x7f15a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a403ba70 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a403bbd0 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b0133080 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40453f0 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.243+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b00b31a0 con 0x7f15b00b1de0 2023-11-24T08:11:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.244+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b004e840 con 0x7f15b00b1de0 2023-11-24T08:11:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.245+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f15a4045550 con 0x7f15b00b1de0 2023-11-24T08:11:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.246+0000 7f15ad7fa640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 0x7f159406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.246+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f15a40aa570 con 0x7f15b00b1de0 2023-11-24T08:11:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.246+0000 7f15affff640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 0x7f159406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.247+0000 7f15affff640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 0x7f159406ded0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f15b00cac50 tx=0x7f159c006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.250+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15a4076f30 con 0x7f15b00b1de0 2023-11-24T08:11:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:47.504+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete"]} v 0) v1 -- 0x7f15b006d860 con 0x7f15b00b1de0 2023-11-24T08:11:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.177+0000 7f15ad7fa640 1 -- 172.21.15.107:0/1291061383 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_pending_in_progress_complete"]}]=0 v356) v1 ==== 187+0+0 (secure 0 0 0) 0x7f15a407a550 con 0x7f15b00b1de0 2023-11-24T08:11:48.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 msgr2=0x7f159406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 0x7f159406ded0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f15b00cac50 tx=0x7f159c006bf0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 msgr2=0x7f15b00b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009890 tx=0x7f15a40046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 shutdown_connections 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f159406ba10 0x7f159406ded0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b015c410 0x7f15b00c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b0144b30 0x7f15b01323e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 --2- 172.21.15.107:0/1291061383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b00b1de0 0x7f15b00b2c60 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 >> 172.21.15.107:0/1291061383 conn(0x7f15b00cb830 msgr2=0x7f15b00722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 shutdown_connections 2023-11-24T08:11:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.180+0000 7f15b5c1a640 1 -- 172.21.15.107:0/1291061383 wait complete. 2023-11-24T08:11:48.203 INFO:tasks.cephfs_test_runner:ensure failure status is not shown when clone is not in failed/cancelled state ... ok 2023-11-24T08:11:48.203 INFO:tasks.cephfs_test_runner:Starting test: test_clone_subvolume_info (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:11:48.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:11:48.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.413+0000 7f126831d640 1 Processor -- start 2023-11-24T08:11:48.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.414+0000 7f126831d640 1 -- start start 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 0x7f12600af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12600721e0 0x7f126006d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f126006a8e0 0x7f12600ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1260071dc0 con 0x7f12600721e0 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1260072bb0 con 0x7f126006a500 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f126831d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126006ef10 con 0x7f126006a8e0 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.415+0000 7f12658b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12600721e0 0x7f126006d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12600721e0 0x7f126006d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49922/0 (socket says 172.21.15.107:49922) 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 -- 172.21.15.107:0/2550486058 learned_addr learned my addr 172.21.15.107:0/2550486058 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12668ba640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f126006a8e0 0x7f12600ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 -- 172.21.15.107:0/2550486058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f126006a8e0 msgr2=0x7f12600ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f126006a8e0 0x7f12600ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12660b9640 1 --2- 172.21.15.107:0/2550486058 >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 0x7f12600af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 -- 172.21.15.107:0/2550486058 >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 msgr2=0x7f12600af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 --2- 172.21.15.107:0/2550486058 >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 0x7f12600af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12658b8640 1 -- 172.21.15.107:0/2550486058 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126006f1b0 con 0x7f12600721e0 2023-11-24T08:11:48.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.416+0000 7f12660b9640 1 --2- 172.21.15.107:0/2550486058 >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 0x7f12600af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:48.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.417+0000 7f12658b8640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12600721e0 0x7f126006d6c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f12600683d0 tx=0x7f125002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcb5aaa6c2e3778 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.417+0000 7f12650b7640 1 -- 172.21.15.107:0/2550486058 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500317e0 con 0x7f12600721e0 2023-11-24T08:11:48.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.417+0000 7f12650b7640 1 -- 172.21.15.107:0/2550486058 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250031940 con 0x7f12600721e0 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.417+0000 7f12650b7640 1 -- 172.21.15.107:0/2550486058 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250031cd0 con 0x7f12600721e0 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 -- 172.21.15.107:0/2550486058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12600721e0 msgr2=0x7f126006d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12600721e0 0x7f126006d6c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f12600683d0 tx=0x7f125002b880 comp rx=0 tx=0).stop 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 -- 172.21.15.107:0/2550486058 shutdown_connections 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f126006a8e0 0x7f12600ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 --2- 172.21.15.107:0/2550486058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12600721e0 0x7f126006d6c0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 --2- 172.21.15.107:0/2550486058 >> v2:172.21.15.120:3300/0 conn(0x7f126006a500 0x7f12600af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 -- 172.21.15.107:0/2550486058 >> 172.21.15.107:0/2550486058 conn(0x7f12600d01b0 msgr2=0x7f12600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 -- 172.21.15.107:0/2550486058 shutdown_connections 2023-11-24T08:11:48.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.418+0000 7f126831d640 1 -- 172.21.15.107:0/2550486058 wait complete. 2023-11-24T08:11:48.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.420+0000 7f126831d640 1 Processor -- start 2023-11-24T08:11:48.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.420+0000 7f126831d640 1 -- start start 2023-11-24T08:11:48.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f126006a500 0x7f12601f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12600721e0 0x7f12601f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c77c0 con 0x7f126006a8e0 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c7aa0 con 0x7f126006a500 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f126831d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c7d80 con 0x7f12600721e0 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f12658b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f12658b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49934/0 (socket says 172.21.15.107:49934) 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f12658b8640 1 -- 172.21.15.107:0/590451436 learned_addr learned my addr 172.21.15.107:0/590451436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.421+0000 7f12660b9640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f126006a500 0x7f12601f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12668ba640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12600721e0 0x7f12601f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 -- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12600721e0 msgr2=0x7f12601f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12600721e0 0x7f12601f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 -- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f126006a500 msgr2=0x7f12601f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f126006a500 0x7f12601f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 -- 172.21.15.107:0/590451436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500094d0 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f12658b8640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f125002b8c0 tx=0x7f12500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500329c0 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250032b20 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500453f0 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.422+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12600c8060 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.423+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12600c7330 con 0x7f126006a8e0 2023-11-24T08:11:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.424+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12600b17b0 con 0x7f126006a8e0 2023-11-24T08:11:48.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.425+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f125004d3f0 con 0x7f126006a8e0 2023-11-24T08:11:48.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.425+0000 7f1256ffd640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 0x7f124406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.426+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f12500aa110 con 0x7f126006a8e0 2023-11-24T08:11:48.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.426+0000 7f12660b9640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 0x7f124406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.426+0000 7f12660b9640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 0x7f124406ded0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f125c006fd0 tx=0x7f125c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.430+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12500769b0 con 0x7f126006a8e0 2023-11-24T08:11:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.607+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12600c7520 con 0x7f126006a8e0 2023-11-24T08:11:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.608+0000 7f1256ffd640 1 -- 172.21.15.107:0/590451436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+12989 (secure 0 0 0) 0x7f1250079fd0 con 0x7f126006a8e0 2023-11-24T08:11:48.611 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:48.611 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:46.145058+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/3083938901":"2023-11-25T08:11:43.125845+0000","172.21.15.107:6836/3083938901":"2023-11-25T08:11:43.125845+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 msgr2=0x7f124406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 0x7f124406ded0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f125c006fd0 tx=0x7f125c008040 comp rx=0 tx=0).stop 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 msgr2=0x7f12601f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f125002b8c0 tx=0x7f12500046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 shutdown_connections 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f124406ba10 0x7f124406ded0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12600721e0 0x7f12601f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f126006a8e0 0x7f12601f6fa0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 --2- 172.21.15.107:0/590451436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f126006a500 0x7f12601f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 >> 172.21.15.107:0/590451436 conn(0x7f12600d01b0 msgr2=0x7f12600b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 shutdown_connections 2023-11-24T08:11:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.612+0000 7f126831d640 1 -- 172.21.15.107:0/590451436 wait complete. 2023-11-24T08:11:48.632 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:11:48.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:11:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.844+0000 7f6441141640 1 Processor -- start 2023-11-24T08:11:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.844+0000 7f6441141640 1 -- start start 2023-11-24T08:11:48.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.845+0000 7f6441141640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f643c0c59b0 0x7f643c166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f6441141640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0c5dc0 con 0x7f643c15b440 2023-11-24T08:11:48.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f6441141640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c15eea0 con 0x7f643c0cc0f0 2023-11-24T08:11:48.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f6441141640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c15b820 con 0x7f643c0c59b0 2023-11-24T08:11:48.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f643ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f643c15b440 0x7f643c0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32920/0 (socket says 172.21.15.107:32920) 2023-11-24T08:11:48.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f643ad9d640 1 -- 172.21.15.107:0/1010174829 learned_addr learned my addr 172.21.15.107:0/1010174829 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f643b59e640 1 --2- 172.21.15.107:0/1010174829 >> v2:172.21.15.120:3300/0 conn(0x7f643c0c59b0 0x7f643c166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.846+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382157130 0 0) 0x7f643c0c5dc0 con 0x7f643c15b440 2023-11-24T08:11:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6428003520 con 0x7f643c15b440 2023-11-24T08:11:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428466414 0 0) 0x7f6428003520 con 0x7f643c15b440 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 >> v2:172.21.15.120:3300/0 conn(0x7f643c0c59b0 msgr2=0x7f643c166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 --2- 172.21.15.107:0/1010174829 >> v2:172.21.15.120:3300/0 conn(0x7f643c0c59b0 0x7f643c166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 >> v1:172.21.15.120:6789/0 conn(0x7f643c0cc0f0 legacy=0x7f643c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c06d600 con 0x7f643c15b440 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f642c001670 con 0x7f643c15b440 2023-11-24T08:11:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 >> v1:172.21.15.107:6789/0 conn(0x7f643c15b440 legacy=0x7f643c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64280039c0 0x7f6428003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64280047c0 0x7f6428004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f642800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c15b820 con 0x7f64280051d0 2023-11-24T08:11:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642800c120 con 0x7f64280039c0 2023-11-24T08:11:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642800c2e0 con 0x7f64280047c0 2023-11-24T08:11:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.847+0000 7f643b59e640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f642800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643ad9d640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64280039c0 0x7f6428003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f6433fff640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64280047c0 0x7f6428004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643b59e640 1 -- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64280047c0 msgr2=0x7f6428004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643b59e640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64280047c0 0x7f6428004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643b59e640 1 -- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64280039c0 msgr2=0x7f6428003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643b59e640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64280039c0 0x7f6428003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643b59e640 1 -- 172.21.15.107:0/1010174829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6428003520 con 0x7f64280051d0 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f643ad9d640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64280039c0 0x7f6428003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.848+0000 7f6433fff640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64280047c0 0x7f6428004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f643b59e640 1 -- 172.21.15.107:0/1010174829 reap_dead start 2023-11-24T08:11:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f643b59e640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f642800b980 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f6434009f90 tx=0x7f64280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7519dd4c95b9ddf9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643402cbf0 con 0x7f64280051d0 2023-11-24T08:11:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643400f040 con 0x7f64280051d0 2023-11-24T08:11:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f643a59c640 1 -- 172.21.15.107:0/1010174829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434003990 con 0x7f64280051d0 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f6441141640 1 -- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 msgr2=0x7f642800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.849+0000 7f6441141640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f642800b980 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f6434009f90 tx=0x7f64280057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.850+0000 7f6441141640 1 -- 172.21.15.107:0/1010174829 shutdown_connections 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.850+0000 7f6441141640 1 --2- 172.21.15.107:0/1010174829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f642800b980 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.850+0000 7f6441141640 1 -- 172.21.15.107:0/1010174829 >> 172.21.15.107:0/1010174829 conn(0x7f643c0b0c50 msgr2=0x7f643c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:48.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.850+0000 7f6441141640 1 -- 172.21.15.107:0/1010174829 shutdown_connections 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.850+0000 7f6441141640 1 -- 172.21.15.107:0/1010174829 wait complete. 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.851+0000 7f6441141640 1 Processor -- start 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 -- start start 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f643c156e30 0x7f643c15b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f643c14a2e0 0x7f643c0ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c09bb80 con 0x7f64280051d0 2023-11-24T08:11:48.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c09af70 con 0x7f643c14a2e0 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.852+0000 7f6441141640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c09b250 con 0x7f643c156e30 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49946/0 (socket says 172.21.15.107:49946) 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 -- 172.21.15.107:0/3497492709 learned_addr learned my addr 172.21.15.107:0/3497492709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 -- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f643c156e30 msgr2=0x7f643c15b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f643c156e30 0x7f643c15b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 -- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f643c14a2e0 msgr2=0x7f643c0ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f643c14a2e0 0x7f643c0ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 -- 172.21.15.107:0/3497492709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c09b530 con 0x7f64280051d0 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.853+0000 7f643ad9d640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f642c00d6c0 tx=0x7f642c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.854+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c0072d0 con 0x7f64280051d0 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.854+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c007430 con 0x7f64280051d0 2023-11-24T08:11:48.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.854+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c0047e0 con 0x7f64280051d0 2023-11-24T08:11:48.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.854+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c09b810 con 0x7f64280051d0 2023-11-24T08:11:48.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.854+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c14a760 con 0x7f64280051d0 2023-11-24T08:11:48.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.855+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c09b9d0 con 0x7f64280051d0 2023-11-24T08:11:48.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.856+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f642c004940 con 0x7f64280051d0 2023-11-24T08:11:48.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.857+0000 7f64337fe640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 0x7f641c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:48.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.857+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f642c087940 con 0x7f64280051d0 2023-11-24T08:11:48.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.857+0000 7f6433fff640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 0x7f641c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:48.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.858+0000 7f6433fff640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 0x7f641c06ded0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f643c157620 tx=0x7f6424009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:48.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:48.861+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f642c0543b0 con 0x7f64280051d0 2023-11-24T08:11:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.050+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f643c14a9e0 con 0x7f64280051d0 2023-11-24T08:11:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.051+0000 7f64337fe640 1 -- 172.21.15.107:0/3497492709 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f642c019070 con 0x7f64280051d0 2023-11-24T08:11:49.053 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:11:49.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 msgr2=0x7f641c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:49.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 0x7f641c06ded0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f643c157620 tx=0x7f6424009040 comp rx=0 tx=0).stop 2023-11-24T08:11:49.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 msgr2=0x7f643c149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:49.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f642c00d6c0 tx=0x7f642c00db80 comp rx=0 tx=0).stop 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 shutdown_connections 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f641c06ba10 0x7f641c06ded0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f643c14a2e0 0x7f643c0ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f643c156e30 0x7f643c15b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 --2- 172.21.15.107:0/3497492709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64280051d0 0x7f643c149bc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 >> 172.21.15.107:0/3497492709 conn(0x7f643c0b0c50 msgr2=0x7f643c0cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 shutdown_connections 2023-11-24T08:11:49.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.054+0000 7f6441141640 1 -- 172.21.15.107:0/3497492709 wait complete. 2023-11-24T08:11:49.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info' 2023-11-24T08:11:49.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.284+0000 7f72ac861640 1 Processor -- start 2023-11-24T08:11:49.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.285+0000 7f72ac861640 1 -- start start 2023-11-24T08:11:49.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72ac861640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f72a80cf450 0x7f72a80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72ac861640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72a80cdfb0 0x7f72a80ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72ac861640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a80cece0 con 0x7f72a815c7f0 2023-11-24T08:11:49.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72ac861640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a80cfe40 con 0x7f72a80cdfb0 2023-11-24T08:11:49.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72ac861640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a8156010 con 0x7f72a80cf450 2023-11-24T08:11:49.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.286+0000 7f72a6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72a80cdfb0 0x7f72a80ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72a80cdfb0 0x7f72a80ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49950/0 (socket says 172.21.15.107:49950) 2023-11-24T08:11:49.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 -- 172.21.15.107:0/3281700248 learned_addr learned my addr 172.21.15.107:0/3281700248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:49.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a5d9b640 1 --2- 172.21.15.107:0/3281700248 >> v2:172.21.15.120:3300/0 conn(0x7f72a80cf450 0x7f72a80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 -- 172.21.15.107:0/3281700248 >> v2:172.21.15.120:3300/0 conn(0x7f72a80cf450 msgr2=0x7f72a80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:49.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 --2- 172.21.15.107:0/3281700248 >> v2:172.21.15.120:3300/0 conn(0x7f72a80cf450 0x7f72a80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 -- 172.21.15.107:0/3281700248 >> v1:172.21.15.120:6789/0 conn(0x7f72a815c7f0 legacy=0x7f72a806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:49.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.287+0000 7f72a6d9d640 1 -- 172.21.15.107:0/3281700248 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a806d040 con 0x7f72a80cdfb0 2023-11-24T08:11:49.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72a6d9d640 1 --2- 172.21.15.107:0/3281700248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a80cdfb0 0x7f72a80ce3e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f72a8070410 tx=0x7f729802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b4de7af6229bfd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72a559a640 1 -- 172.21.15.107:0/3281700248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298031890 con 0x7f72a80cdfb0 2023-11-24T08:11:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72a559a640 1 -- 172.21.15.107:0/3281700248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980319f0 con 0x7f72a80cdfb0 2023-11-24T08:11:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72a559a640 1 -- 172.21.15.107:0/3281700248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298031d80 con 0x7f72a80cdfb0 2023-11-24T08:11:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 -- 172.21.15.107:0/3281700248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a80cdfb0 msgr2=0x7f72a80ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 --2- 172.21.15.107:0/3281700248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a80cdfb0 0x7f72a80ce3e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f72a8070410 tx=0x7f729802b930 comp rx=0 tx=0).stop 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 -- 172.21.15.107:0/3281700248 shutdown_connections 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 --2- 172.21.15.107:0/3281700248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a80cdfb0 0x7f72a80ce3e0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 --2- 172.21.15.107:0/3281700248 >> v2:172.21.15.120:3300/0 conn(0x7f72a80cf450 0x7f72a80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 -- 172.21.15.107:0/3281700248 >> 172.21.15.107:0/3281700248 conn(0x7f72a806a920 msgr2=0x7f72a806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 -- 172.21.15.107:0/3281700248 shutdown_connections 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.289+0000 7f72ac861640 1 -- 172.21.15.107:0/3281700248 wait complete. 2023-11-24T08:11:49.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.291+0000 7f72ac861640 1 Processor -- start 2023-11-24T08:11:49.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.291+0000 7f72ac861640 1 -- start start 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72a80cdfb0 0x7f72a80c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 0x7f72a814b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a8158fe0 con 0x7f72a815c7f0 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a814bb60 con 0x7f72a80cf450 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72ac861640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a8158d00 con 0x7f72a80cdfb0 2023-11-24T08:11:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49954/0 (socket says 172.21.15.107:49954) 2023-11-24T08:11:49.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72a6d9d640 1 -- 172.21.15.107:0/1415376277 learned_addr learned my addr 172.21.15.107:0/1415376277 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:49.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72a659c640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72a80cdfb0 0x7f72a80c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.292+0000 7f72a5d9b640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 0x7f72a814b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 -- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72a80cdfb0 msgr2=0x7f72a80c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72a80cdfb0 0x7f72a80c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 -- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 msgr2=0x7f72a814b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 0x7f72a814b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 -- 172.21.15.107:0/1415376277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980094d0 con 0x7f72a815c7f0 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a5d9b640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 0x7f72a814b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72a6d9d640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f729802b970 tx=0x7f72980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:49.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729803adf0 con 0x7f72a815c7f0 2023-11-24T08:11:49.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980443f0 con 0x7f72a815c7f0 2023-11-24T08:11:49.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980447a0 con 0x7f72a815c7f0 2023-11-24T08:11:49.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a80b2280 con 0x7f72a815c7f0 2023-11-24T08:11:49.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.293+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a80b2840 con 0x7f72a815c7f0 2023-11-24T08:11:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.294+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a80b20e0 con 0x7f72a815c7f0 2023-11-24T08:11:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.295+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f7298044900 con 0x7f72a815c7f0 2023-11-24T08:11:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.297+0000 7f729effd640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 0x7f728806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.297+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f72980aa9d0 con 0x7f72a815c7f0 2023-11-24T08:11:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.297+0000 7f72a659c640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 0x7f728806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:49.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.298+0000 7f72a659c640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 0x7f728806ded0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006fd0 tx=0x7f72a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:49.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.301+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72a80b20e0 con 0x7f72a815c7f0 2023-11-24T08:11:49.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:49.558+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info"]} v 0) v1 -- 0x7f72a80b20e0 con 0x7f72a815c7f0 2023-11-24T08:11:50.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.190+0000 7f729effd640 1 -- 172.21.15.107:0/1415376277 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info"]}]=0 v358) v1 ==== 161+0+0 (secure 0 0 0) 0x7f72a80b20e0 con 0x7f72a815c7f0 2023-11-24T08:11:50.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 msgr2=0x7f728806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 0x7f728806ded0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006fd0 tx=0x7f72a0008040 comp rx=0 tx=0).stop 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 msgr2=0x7f72a8158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f729802b970 tx=0x7f72980046c0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 shutdown_connections 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f728806ba10 0x7f728806ded0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72a815c7f0 0x7f72a8158300 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a80cf450 0x7f72a814b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 --2- 172.21.15.107:0/1415376277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72a80cdfb0 0x7f72a80c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 >> 172.21.15.107:0/1415376277 conn(0x7f72a806a920 msgr2=0x7f72a8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:50.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.193+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 shutdown_connections 2023-11-24T08:11:50.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.194+0000 7f72ac861640 1 -- 172.21.15.107:0/1415376277 wait complete. 2023-11-24T08:11:50.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:11:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.430+0000 7fe55d251640 1 Processor -- start 2023-11-24T08:11:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.430+0000 7fe55d251640 1 -- start start 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe55d251640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe55815f530 0x7fe55815f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe55d251640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 0x7fe5580cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe55d251640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55813eb00 con 0x7fe5580cf320 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe55d251640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5580cf790 con 0x7fe55815f530 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe55d251640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55813ede0 con 0x7fe55806ff70 2023-11-24T08:11:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe556d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe55815f530 0x7fe55815f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe556d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe55815f530 0x7fe55815f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49956/0 (socket says 172.21.15.107:49956) 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.432+0000 7fe556d9d640 1 -- 172.21.15.107:0/3917719647 learned_addr learned my addr 172.21.15.107:0/3917719647 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe54e59c640 1 --2- 172.21.15.107:0/3917719647 >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 0x7fe5580cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe556d9d640 1 -- 172.21.15.107:0/3917719647 >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 msgr2=0x7fe5580cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe556d9d640 1 --2- 172.21.15.107:0/3917719647 >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 0x7fe5580cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe556d9d640 1 -- 172.21.15.107:0/3917719647 >> v1:172.21.15.120:6789/0 conn(0x7fe5580cf320 legacy=0x7fe558140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe556d9d640 1 -- 172.21.15.107:0/3917719647 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55815fe50 con 0x7fe55815f530 2023-11-24T08:11:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.433+0000 7fe54e59c640 1 --2- 172.21.15.107:0/3917719647 >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 0x7fe5580cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.434+0000 7fe556d9d640 1 --2- 172.21.15.107:0/3917719647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55815f530 0x7fe55815f910 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe55815ee60 tx=0x7fe54802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff0f034c69466fd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.434+0000 7fe55659c640 1 -- 172.21.15.107:0/3917719647 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54802c9f0 con 0x7fe55815f530 2023-11-24T08:11:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55659c640 1 -- 172.21.15.107:0/3917719647 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54802cb50 con 0x7fe55815f530 2023-11-24T08:11:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55659c640 1 -- 172.21.15.107:0/3917719647 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480343f0 con 0x7fe55815f530 2023-11-24T08:11:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 -- 172.21.15.107:0/3917719647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55815f530 msgr2=0x7fe55815f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 --2- 172.21.15.107:0/3917719647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55815f530 0x7fe55815f910 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe55815ee60 tx=0x7fe54802be50 comp rx=0 tx=0).stop 2023-11-24T08:11:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 -- 172.21.15.107:0/3917719647 shutdown_connections 2023-11-24T08:11:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 --2- 172.21.15.107:0/3917719647 >> v2:172.21.15.120:3300/0 conn(0x7fe55806ff70 0x7fe5580cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 --2- 172.21.15.107:0/3917719647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55815f530 0x7fe55815f910 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 -- 172.21.15.107:0/3917719647 >> 172.21.15.107:0/3917719647 conn(0x7fe5580b2e40 msgr2=0x7fe5580b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 -- 172.21.15.107:0/3917719647 shutdown_connections 2023-11-24T08:11:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.435+0000 7fe55d251640 1 -- 172.21.15.107:0/3917719647 wait complete. 2023-11-24T08:11:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.437+0000 7fe55d251640 1 Processor -- start 2023-11-24T08:11:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.437+0000 7fe55d251640 1 -- start start 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5580cf320 0x7fe55815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe55815f530 0x7fe55815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe558140aa0 con 0x7fe55806ff70 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55813e970 con 0x7fe55815f530 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe55d251640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581f1220 con 0x7fe5580cf320 2023-11-24T08:11:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe556d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe556d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49968/0 (socket says 172.21.15.107:49968) 2023-11-24T08:11:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.438+0000 7fe556d9d640 1 -- 172.21.15.107:0/1304843083 learned_addr learned my addr 172.21.15.107:0/1304843083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 -- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5580cf320 msgr2=0x7fe55815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe54e59c640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5580cf320 0x7fe55815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5580cf320 0x7fe55815d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 -- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe55815f530 msgr2=0x7fe55815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe55815f530 0x7fe55815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 -- 172.21.15.107:0/1304843083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548009b30 con 0x7fe55806ff70 2023-11-24T08:11:50.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe556d9d640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe548000f80 tx=0x7fe548004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54803e450 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54803e5f0 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5581f1500 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54803e9a0 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.439+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5581f1ac0 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.441+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55804e8b0 con 0x7fe55806ff70 2023-11-24T08:11:50.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.441+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fe54803eb00 con 0x7fe55806ff70 2023-11-24T08:11:50.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.442+0000 7fe54ffff640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 0x7fe53806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:50.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.442+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe5480ab070 con 0x7fe55806ff70 2023-11-24T08:11:50.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.443+0000 7fe54e59c640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 0x7fe53806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.443+0000 7fe54e59c640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 0x7fe53806ded0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe5580ce8b0 tx=0x7fe540008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.447+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe548077c90 con 0x7fe55806ff70 2023-11-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.638+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe5580b3630 con 0x7fe55806ff70 2023-11-24T08:11:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.644+0000 7fe54ffff640 1 -- 172.21.15.107:0/1304843083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe54803edb0 con 0x7fe55806ff70 2023-11-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 msgr2=0x7fe53806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 0x7fe53806ded0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe5580ce8b0 tx=0x7fe540008040 comp rx=0 tx=0).stop 2023-11-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 msgr2=0x7fe558149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe548000f80 tx=0x7fe548004a40 comp rx=0 tx=0).stop 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 shutdown_connections 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe53806ba10 0x7fe53806ded0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe55815f530 0x7fe55815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5580cf320 0x7fe55815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 --2- 172.21.15.107:0/1304843083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe55806ff70 0x7fe558149510 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.647+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 >> 172.21.15.107:0/1304843083 conn(0x7fe5580b2e40 msgr2=0x7fe5580b6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.648+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 shutdown_connections 2023-11-24T08:11:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:50.648+0000 7fe55d251640 1 -- 172.21.15.107:0/1304843083 wait complete. 2023-11-24T08:11:50.668 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:11:50.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:11:50.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:11:50.899 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:11:50.899 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:50.899 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:50.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:11:50.961 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:11:50.961 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:51.020 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:11:51.021 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:11:51.021 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:51.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:11:51.081 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:11:51.081 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:11:51.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:11:51.140 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:11:51.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:11:51.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.362+0000 7f041562b640 1 Processor -- start 2023-11-24T08:11:51.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.362+0000 7f041562b640 1 -- start start 2023-11-24T08:11:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 0x7f041015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 0x7f04100ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04100cd470 0x7f041015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041015a990 con 0x7f04100cd470 2023-11-24T08:11:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041015ab80 con 0x7f04100b2250 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f041562b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041015e680 con 0x7f04100b2630 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.364+0000 7f040f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04100cd470 0x7f041015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04100cd470 0x7f041015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49974/0 (socket says 172.21.15.107:49974) 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 -- 172.21.15.107:0/3577291890 learned_addr learned my addr 172.21.15.107:0/3577291890 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 -- 172.21.15.107:0/3577291890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 msgr2=0x7f04100ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040effd640 1 --2- 172.21.15.107:0/3577291890 >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 0x7f041015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040e7fc640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 0x7f04100ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 0x7f04100ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 -- 172.21.15.107:0/3577291890 >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 msgr2=0x7f041015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 --2- 172.21.15.107:0/3577291890 >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 0x7f041015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040f7fe640 1 -- 172.21.15.107:0/3577291890 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041015e960 con 0x7f04100cd470 2023-11-24T08:11:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040e7fc640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 0x7f04100ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.365+0000 7f040effd640 1 --2- 172.21.15.107:0/3577291890 >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 0x7f041015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.366+0000 7f040f7fe640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f041015b080 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0410072a70 tx=0x7f040802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259b9d6f808c3a9b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.366+0000 7f040dffb640 1 -- 172.21.15.107:0/3577291890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040802ca80 con 0x7f04100cd470 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.366+0000 7f040dffb640 1 -- 172.21.15.107:0/3577291890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040802cbe0 con 0x7f04100cd470 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.366+0000 7f040dffb640 1 -- 172.21.15.107:0/3577291890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080343f0 con 0x7f04100cd470 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 -- 172.21.15.107:0/3577291890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 msgr2=0x7f041015b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f041015b080 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0410072a70 tx=0x7f040802bee0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 -- 172.21.15.107:0/3577291890 shutdown_connections 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f041015b080 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 --2- 172.21.15.107:0/3577291890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2630 0x7f04100ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 --2- 172.21.15.107:0/3577291890 >> v2:172.21.15.120:3300/0 conn(0x7f04100b2250 0x7f041015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 -- 172.21.15.107:0/3577291890 >> 172.21.15.107:0/3577291890 conn(0x7f041006a920 msgr2=0x7f041006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 -- 172.21.15.107:0/3577291890 shutdown_connections 2023-11-24T08:11:51.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.367+0000 7f041562b640 1 -- 172.21.15.107:0/3577291890 wait complete. 2023-11-24T08:11:51.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.369+0000 7f041562b640 1 Processor -- start 2023-11-24T08:11:51.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.369+0000 7f041562b640 1 -- start start 2023-11-24T08:11:51.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 0x7f0410150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100b2630 0x7f0410155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410151300 con 0x7f04100cd470 2023-11-24T08:11:51.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04101506f0 con 0x7f04100b2630 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f041562b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04101509d0 con 0x7f04100b2250 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f040f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f040f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49976/0 (socket says 172.21.15.107:49976) 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.370+0000 7f040f7fe640 1 -- 172.21.15.107:0/132855154 learned_addr learned my addr 172.21.15.107:0/132855154 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040effd640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 0x7f0410150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 -- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 msgr2=0x7f0410150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 0x7f0410150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 -- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100b2630 msgr2=0x7f0410155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040e7fc640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100b2630 0x7f0410155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100b2630 0x7f0410155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 -- 172.21.15.107:0/132855154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408009b30 con 0x7f04100cd470 2023-11-24T08:11:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040effd640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 0x7f0410150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f040f7fe640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0408036040 tx=0x7f0408004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040803e450 con 0x7f04100cd470 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040803e5b0 con 0x7f04100cd470 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040803e960 con 0x7f04100cd470 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.371+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0410150cb0 con 0x7f04100cd470 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.372+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04100b0450 con 0x7f04100cd470 2023-11-24T08:11:51.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.373+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0410151470 con 0x7f04100cd470 2023-11-24T08:11:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.373+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f040802cd50 con 0x7f04100cd470 2023-11-24T08:11:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.374+0000 7f04077fe640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 0x7f03f006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.374+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f04080aaa60 con 0x7f04100cd470 2023-11-24T08:11:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.375+0000 7f040effd640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 0x7f03f006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:51.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.375+0000 7f040effd640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 0x7f03f006ded0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f04101598c0 tx=0x7f0400008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.378+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0408077670 con 0x7f04100cd470 2023-11-24T08:11:51.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.605+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0410067470 con 0x7f04100cd470 2023-11-24T08:11:51.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.606+0000 7f04077fe640 1 -- 172.21.15.107:0/132855154 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f040807ac90 con 0x7f04100cd470 2023-11-24T08:11:51.607 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:11:51.608 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1287439574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1287439574},{"type":"v1","addr":"172.21.15.120:6835","nonce":1287439574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3413994205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3413994205},{"type":"v1","addr":"172.21.15.120:6833","nonce":3413994205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4149865486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4149865486},{"type":"v1","addr":"172.21.15.107:6835","nonce":4149865486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":7245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2988142104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2988142104},{"type":"v1","addr":"172.21.15.107:6837","nonce":2988142104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2023-11-24T08:11:51.608 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 msgr2=0x7f03f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 0x7f03f006ded0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f04101598c0 tx=0x7f0400008040 comp rx=0 tx=0).stop 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 msgr2=0x7f04100aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0408036040 tx=0x7f0408004a40 comp rx=0 tx=0).stop 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 shutdown_connections 2023-11-24T08:11:51.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03f006ba10 0x7f03f006ded0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100cd470 0x7f04100aff10 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100b2630 0x7f0410155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 --2- 172.21.15.107:0/132855154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04100b2250 0x7f0410150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 >> 172.21.15.107:0/132855154 conn(0x7f041006a920 msgr2=0x7f04100cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 shutdown_connections 2023-11-24T08:11:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:51.609+0000 7f041562b640 1 -- 172.21.15.107:0/132855154 wait complete. 2023-11-24T08:11:51.629 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:11:51.630 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:11:51.630 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:11:51.630 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:11:51.630 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:11:51.630 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:11:51.630 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:11:51.631 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:11:51.631 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:11:51.631 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:11:51.631 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:11:51.631 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:11:51.631 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:11:51.632 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:11:51.632 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:11:51.632 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:11:51.649 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:11:51.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:11:51.650 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:11:51.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:11:51.654 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:11:51.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:11:51.698 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:11:51.698 DEBUG:teuthology.parallel:result is None 2023-11-24T08:11:51.703 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:11:51.703 DEBUG:teuthology.parallel:result is None 2023-11-24T08:11:51.704 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:11:51.704 DEBUG:teuthology.parallel:result is None 2023-11-24T08:11:51.855 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:11:51.859 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:11:51.885 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:11:57.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-24T08:11:57.655 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:11:57.656 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:11:57.666 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:11:57.666 DEBUG:teuthology.parallel:result is None 2023-11-24T08:11:57.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:11:57.849 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:11:57.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.889+0000 7f036d5db640 1 Processor -- start 2023-11-24T08:11:57.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.889+0000 7f036d5db640 1 -- start start 2023-11-24T08:11:57.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036d5db640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f036815a1c0 0x7f0368070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036d5db640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f036815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036d5db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680b1a20 con 0x7f036806d640 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036d5db640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680cd920 con 0x7f036815a1c0 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036d5db640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036815e2e0 con 0x7f03680b15e0 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036cdda640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f036806d640 0x7f03680ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32924/0 (socket says 172.21.15.107:32924) 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f036cdda640 1 -- 172.21.15.107:0/2826861329 learned_addr learned my addr 172.21.15.107:0/2826861329 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.891+0000 7f0367fff640 1 --2- 172.21.15.107:0/2826861329 >> v2:172.21.15.120:3300/0 conn(0x7f036815a1c0 0x7f0368070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f03677fe640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f036815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0366ffd640 1 -- 172.21.15.107:0/2826861329 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846875874 0 0) 0x7f0360001110 con 0x7f036806d640 2023-11-24T08:11:57.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0366ffd640 1 -- 172.21.15.107:0/2826861329 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0354003520 con 0x7f036806d640 2023-11-24T08:11:57.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0367fff640 1 -- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 msgr2=0x7f036815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0367fff640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f036815dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0367fff640 1 -- 172.21.15.107:0/2826861329 >> v1:172.21.15.107:6789/0 conn(0x7f036806d640 legacy=0x7f03680ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f0367fff640 1 -- 172.21.15.107:0/2826861329 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036815e580 con 0x7f036815a1c0 2023-11-24T08:11:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.892+0000 7f03677fe640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f036815dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:11:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.893+0000 7f0367fff640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036815a1c0 0x7f0368070ea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0358009d90 tx=0x7f035802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=901bf8fbf8814c35 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:57.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.894+0000 7f0366ffd640 1 -- 172.21.15.107:0/2826861329 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035802cce0 con 0x7f036815a1c0 2023-11-24T08:11:57.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.894+0000 7f0366ffd640 1 -- 172.21.15.107:0/2826861329 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035802ce40 con 0x7f036815a1c0 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.894+0000 7f0366ffd640 1 -- 172.21.15.107:0/2826861329 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358034640 con 0x7f036815a1c0 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 -- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036815a1c0 msgr2=0x7f0368070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036815a1c0 0x7f0368070ea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0358009d90 tx=0x7f035802bf30 comp rx=0 tx=0).stop 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 -- 172.21.15.107:0/2826861329 shutdown_connections 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f036815dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 --2- 172.21.15.107:0/2826861329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036815a1c0 0x7f0368070ea0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 -- 172.21.15.107:0/2826861329 >> 172.21.15.107:0/2826861329 conn(0x7f03680cbb40 msgr2=0x7f03680cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 -- 172.21.15.107:0/2826861329 shutdown_connections 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.895+0000 7f036d5db640 1 -- 172.21.15.107:0/2826861329 wait complete. 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.897+0000 7f036d5db640 1 Processor -- start 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.897+0000 7f036d5db640 1 -- start start 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036806d640 0x7f0368154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f03680c3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c4b70 con 0x7f036815a1c0 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c3ef0 con 0x7f036806d640 2023-11-24T08:11:57.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036d5db640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c41d0 con 0x7f03680b15e0 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036cdda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036cdda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60698/0 (socket says 172.21.15.107:60698) 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f036cdda640 1 -- 172.21.15.107:0/2218154899 learned_addr learned my addr 172.21.15.107:0/2218154899 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f03677fe640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f03680c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.898+0000 7f0367fff640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036806d640 0x7f0368154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 -- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 msgr2=0x7f03680c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f03680c3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 -- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036806d640 msgr2=0x7f0368154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036806d640 0x7f0368154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 -- 172.21.15.107:0/2218154899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350009400 con 0x7f036815a1c0 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f03677fe640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f03680c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036cdda640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f036000c700 tx=0x7f036000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360005ab0 con 0x7f036815a1c0 2023-11-24T08:11:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360005c10 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360001d50 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0358009a70 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.899+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03680c4790 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.900+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03680b22b0 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.901+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f0360007070 con 0x7f036815a1c0 2023-11-24T08:11:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.903+0000 7f0364ff9640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 0x7f034806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.903+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f03600875b0 con 0x7f036815a1c0 2023-11-24T08:11:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.903+0000 7f0367fff640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 0x7f034806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.904+0000 7f0367fff640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 0x7f034806ded0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0358036000 tx=0x7f0358030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:57.907+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0360053f00 con 0x7f036815a1c0 2023-11-24T08:11:58.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:58.111+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f03680b1a20 con 0x7f036815a1c0 2023-11-24T08:11:59.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.627+0000 7f0364ff9640 1 -- 172.21.15.107:0/2218154899 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) v1 ==== 97+0+0 (secure 0 0 0) 0x7f03680b1a20 con 0x7f036815a1c0 2023-11-24T08:11:59.628 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:11:59.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.630+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 msgr2=0x7f034806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.630+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 0x7f034806ded0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0358036000 tx=0x7f0358030000 comp rx=0 tx=0).stop 2023-11-24T08:11:59.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.630+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 msgr2=0x7f03680affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.630+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f036000c700 tx=0x7f036000cbc0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 shutdown_connections 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f034806ba10 0x7f034806ded0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f036815a1c0 0x7f03680affb0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680b15e0 0x7f03680c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 --2- 172.21.15.107:0/2218154899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f036806d640 0x7f0368154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 >> 172.21.15.107:0/2218154899 conn(0x7f03680cbb40 msgr2=0x7f03680cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:59.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 shutdown_connections 2023-11-24T08:11:59.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.631+0000 7f036d5db640 1 -- 172.21.15.107:0/2218154899 wait complete. 2023-11-24T08:11:59.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:11:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.863+0000 7f26edc75640 1 Processor -- start 2023-11-24T08:11:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.863+0000 7f26edc75640 1 -- start start 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26edc75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26e80b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26edc75640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80711e0 con 0x7f26e80cd5b0 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26edc75640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80cdc80 con 0x7f26e815b5e0 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26edc75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80705d0 con 0x7f26e80abf50 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26e6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f26e80cd5b0 0x7f26e8071670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59034/0 (socket says 172.21.15.107:59034) 2023-11-24T08:11:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26e6ffd640 1 -- 172.21.15.107:0/2240476836 learned_addr learned my addr 172.21.15.107:0/2240476836 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:11:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.865+0000 7f26e7fff640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26e80b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415605118 0 0) 0x7f26e80711e0 con 0x7f26e80cd5b0 2023-11-24T08:11:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d0003520 con 0x7f26e80cd5b0 2023-11-24T08:11:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2980310372 0 0) 0x7f26d0003520 con 0x7f26e80cd5b0 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 msgr2=0x7f26e80b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26e80b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 >> v1:172.21.15.120:6789/0 conn(0x7f26e815b5e0 legacy=0x7f26e80ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80708b0 con 0x7f26e80cd5b0 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f26dc001670 con 0x7f26e80cd5b0 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 >> v1:172.21.15.107:6789/0 conn(0x7f26e80cd5b0 legacy=0x7f26e8071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00039a0 0x7f26d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d0004d00 0x7f26d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80705d0 con 0x7f26d0004d00 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d000ba30 con 0x7f26d00039a0 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.866+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d000bbf0 con 0x7f26e80abf50 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d0004d00 0x7f26d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e77fe640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00039a0 0x7f26d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e6ffd640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 -- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 msgr2=0x7f26d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80abf50 0x7f26d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 -- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00039a0 msgr2=0x7f26d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26d00039a0 0x7f26d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.867+0000 7f26e7fff640 1 -- 172.21.15.107:0/2240476836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0003520 con 0x7f26d0004d00 2023-11-24T08:11:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.868+0000 7f26e7fff640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d0004d00 0x7f26d0005180 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009f90 tx=0x7f26d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc76459a7fd0fcaa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e002cbd0 con 0x7f26d0004d00 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000f040 con 0x7f26d0004d00 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26e67fc640 1 -- 172.21.15.107:0/2240476836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0034440 con 0x7f26d0004d00 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 -- 172.21.15.107:0/2240476836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d0004d00 msgr2=0x7f26d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 --2- 172.21.15.107:0/2240476836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26d0004d00 0x7f26d0005180 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009f90 tx=0x7f26d00057f0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26e7fff640 1 -- 172.21.15.107:0/2240476836 reap_dead start 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 -- 172.21.15.107:0/2240476836 shutdown_connections 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 -- 172.21.15.107:0/2240476836 >> 172.21.15.107:0/2240476836 conn(0x7f26e80b0c70 msgr2=0x7f26e80b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 -- 172.21.15.107:0/2240476836 shutdown_connections 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.870+0000 7f26edc75640 1 -- 172.21.15.107:0/2240476836 wait complete. 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.872+0000 7f26edc75640 1 Processor -- start 2023-11-24T08:11:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.872+0000 7f26edc75640 1 -- start start 2023-11-24T08:11:59.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.873+0000 7f26edc75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d0004d00 0x7f26e8155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26edc75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26edc75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 0x7f26e815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26edc75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e809d130 con 0x7f26e81483b0 2023-11-24T08:11:59.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26edc75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8099c80 con 0x7f26e815db60 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26edc75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e815e520 con 0x7f26d0004d00 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60712/0 (socket says 172.21.15.107:60712) 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 -- 172.21.15.107:0/53854543 learned_addr learned my addr 172.21.15.107:0/53854543 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e77fe640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d0004d00 0x7f26e8155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e7fff640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 0x7f26e815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 -- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d0004d00 msgr2=0x7f26e8155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d0004d00 0x7f26e8155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.874+0000 7f26e6ffd640 1 -- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 msgr2=0x7f26e815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26e6ffd640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 0x7f26e815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26e6ffd640 1 -- 172.21.15.107:0/53854543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80992e0 con 0x7f26e81483b0 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26e7fff640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 0x7f26e815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26e6ffd640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00e8c0 tx=0x7f26dc00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00c870 con 0x7f26e81483b0 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc00c9d0 con 0x7f26e81483b0 2023-11-24T08:11:59.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00cd80 con 0x7f26e81483b0 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e80995c0 con 0x7f26e81483b0 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.875+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e809ce20 con 0x7f26e81483b0 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.876+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e804e8b0 con 0x7f26e81483b0 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.877+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f26dc01d460 con 0x7f26e81483b0 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.878+0000 7f26cffff640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 0x7f26d406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:11:59.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.878+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26dc088220 con 0x7f26e81483b0 2023-11-24T08:11:59.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.879+0000 7f26e77fe640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 0x7f26d406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:11:59.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.879+0000 7f26e77fe640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 0x7f26d406ded0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f26d8007fd0 tx=0x7f26d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:11:59.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:11:59.883+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26dc054ac0 con 0x7f26e81483b0 2023-11-24T08:12:00.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.142+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f26e80cc9e0 con 0x7f26e81483b0 2023-11-24T08:12:00.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.142+0000 7f26cffff640 1 -- 172.21.15.107:0/53854543 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f26dc0580e0 con 0x7f26e81483b0 2023-11-24T08:12:00.143 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:00.146 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:12:00.149 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.151 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.152 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.153 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:12:00.155 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:00.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:12:00.159 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:00.161 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:12:00.162 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:12:00.162 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:00.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:00.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:12:00.165 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:00.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:00.165 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:12:00.166 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 msgr2=0x7f26d406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 0x7f26d406ded0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f26d8007fd0 tx=0x7f26d8009040 comp rx=0 tx=0).stop 2023-11-24T08:12:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 msgr2=0x7f26e809c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00e8c0 tx=0x7f26dc00ed80 comp rx=0 tx=0).stop 2023-11-24T08:12:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 shutdown_connections 2023-11-24T08:12:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26d406ba10 0x7f26d406ded0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e815db60 0x7f26e815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26e81483b0 0x7f26e809c8e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 --2- 172.21.15.107:0/53854543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26d0004d00 0x7f26e8155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 >> 172.21.15.107:0/53854543 conn(0x7f26e80b0c70 msgr2=0x7f26e80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 shutdown_connections 2023-11-24T08:12:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.145+0000 7f26edc75640 1 -- 172.21.15.107:0/53854543 wait complete. 2023-11-24T08:12:00.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:00.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.380+0000 7f7fc39a9640 1 Processor -- start 2023-11-24T08:12:00.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.380+0000 7f7fc39a9640 1 -- start start 2023-11-24T08:12:00.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.381+0000 7f7fc39a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cf450 0x7f7fbc0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.381+0000 7f7fc39a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7fbc0cdfb0 0x7f7fbc0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.381+0000 7f7fc39a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fbc0cece0 con 0x7f7fbc15c7f0 2023-11-24T08:12:00.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.381+0000 7f7fc39a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fbc156010 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.381+0000 7f7fc39a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc06ce30 con 0x7f7fbc0cf450 2023-11-24T08:12:00.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1745640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7fbc15c7f0 0x7f7fbc06db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59050/0 (socket says 172.21.15.107:59050) 2023-11-24T08:12:00.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1745640 1 -- 172.21.15.107:0/3049781920 learned_addr learned my addr 172.21.15.107:0/3049781920 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:00.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1f46640 1 --2- 172.21.15.107:0/3049781920 >> v2:172.21.15.107:3300/0 conn(0x7f7fbc0cdfb0 0x7f7fbc0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc0f44640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cf450 0x7f7fbc0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1f46640 1 -- 172.21.15.107:0/3049781920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cf450 msgr2=0x7f7fbc0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:00.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fb3fff640 1 -- 172.21.15.107:0/3049781920 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999040496 0 0) 0x7f7fbc0cece0 con 0x7f7fbc15c7f0 2023-11-24T08:12:00.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1f46640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cf450 0x7f7fbc0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1f46640 1 -- 172.21.15.107:0/3049781920 >> v1:172.21.15.107:6789/0 conn(0x7f7fbc15c7f0 legacy=0x7f7fbc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.382+0000 7f7fc1f46640 1 -- 172.21.15.107:0/3049781920 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc06d040 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.383+0000 7f7fc1f46640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0ce440 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009d90 tx=0x7f7fb402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41d1c7955ffe219 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.383+0000 7f7fb3fff640 1 -- 172.21.15.107:0/3049781920 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb402cc50 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fb3fff640 1 -- 172.21.15.107:0/3049781920 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb402cdb0 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fb3fff640 1 -- 172.21.15.107:0/3049781920 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4034660 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 -- 172.21.15.107:0/3049781920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cdfb0 msgr2=0x7f7fbc0ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0ce440 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009d90 tx=0x7f7fb402bea0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 -- 172.21.15.107:0/3049781920 shutdown_connections 2023-11-24T08:12:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0ce440 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/3049781920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cf450 0x7f7fbc0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 -- 172.21.15.107:0/3049781920 >> 172.21.15.107:0/3049781920 conn(0x7f7fbc06a920 msgr2=0x7f7fbc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:00.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 -- 172.21.15.107:0/3049781920 shutdown_connections 2023-11-24T08:12:00.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.384+0000 7f7fc39a9640 1 -- 172.21.15.107:0/3049781920 wait complete. 2023-11-24T08:12:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.386+0000 7f7fc39a9640 1 Processor -- start 2023-11-24T08:12:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.386+0000 7f7fc39a9640 1 -- start start 2023-11-24T08:12:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 0x7f7fbc15ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc159500 con 0x7f7fbc0cf450 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc098ef0 con 0x7f7fbc15c7f0 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.387+0000 7f7fc39a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc151940 con 0x7f7fbc0cdfb0 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60738/0 (socket says 172.21.15.107:60738) 2023-11-24T08:12:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 -- 172.21.15.107:0/4288417036 learned_addr learned my addr 172.21.15.107:0/4288417036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:00.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc1f46640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 0x7f7fbc15ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc1745640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 -- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 msgr2=0x7f7fbc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 -- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 msgr2=0x7f7fbc15ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 0x7f7fbc15ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 -- 172.21.15.107:0/4288417036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb4009a70 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc1745640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc1f46640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 0x7f7fbc15ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.388+0000 7f7fc0f44640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00d830 tx=0x7f7fac00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.389+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00b6c0 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.389+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac00b820 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.389+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00bbb0 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.389+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc159220 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.389+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fbc1515d0 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.390+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fbc151150 con 0x7f7fbc0cf450 2023-11-24T08:12:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.391+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f7fac01e460 con 0x7f7fbc0cf450 2023-11-24T08:12:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.392+0000 7f7fb1ffb640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 0x7f7fa006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.392+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7fac088a80 con 0x7f7fbc0cf450 2023-11-24T08:12:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.392+0000 7f7fc1745640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 0x7f7fa006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.392+0000 7f7fc1745640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 0x7f7fa006df00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f60 tx=0x7f7fb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.396+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fbc151150 con 0x7f7fbc0cf450 2023-11-24T08:12:00.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.587+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fbc151150 con 0x7f7fbc0cf450 2023-11-24T08:12:00.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.589+0000 7f7fb1ffb640 1 -- 172.21.15.107:0/4288417036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+12858 (secure 0 0 0) 0x7f7fbc151150 con 0x7f7fbc0cf450 2023-11-24T08:12:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:00.590 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:11:59.623415+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:00.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 msgr2=0x7f7fa006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 0x7f7fa006df00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f60 tx=0x7f7fb8009040 comp rx=0 tx=0).stop 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 msgr2=0x7f7fbc1589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00d830 tx=0x7f7fac00dcf0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 shutdown_connections 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7fa006ba40 0x7f7fa006df00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fbc15c7f0 0x7f7fbc15ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fbc0cf450 0x7f7fbc1589d0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 --2- 172.21.15.107:0/4288417036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fbc0cdfb0 0x7f7fbc0989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.591+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 >> 172.21.15.107:0/4288417036 conn(0x7f7fbc06a920 msgr2=0x7f7fbc1554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.592+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 shutdown_connections 2023-11-24T08:12:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.592+0000 7f7fc39a9640 1 -- 172.21.15.107:0/4288417036 wait complete. 2023-11-24T08:12:00.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:00.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.829+0000 7f41371a5640 1 Processor -- start 2023-11-24T08:12:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.829+0000 7f41371a5640 1 -- start start 2023-11-24T08:12:00.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f413006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41300b0b40 0x7f41300b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413015ded0 0x7f413015e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413006d430 con 0x7f413015ded0 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41300b14e0 con 0x7f41300b0b40 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41371a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413015ec50 con 0x7f413015a1c0 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41369a4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413015ded0 0x7f413015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41369a4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413015ded0 0x7f413015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60754/0 (socket says 172.21.15.107:60754) 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.831+0000 7f41369a4640 1 -- 172.21.15.107:0/1361909858 learned_addr learned my addr 172.21.15.107:0/1361909858 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41361a3640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f413006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41359a2640 1 --2- 172.21.15.107:0/1361909858 >> v2:172.21.15.120:3300/0 conn(0x7f41300b0b40 0x7f41300b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41369a4640 1 -- 172.21.15.107:0/1361909858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 msgr2=0x7f413006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41369a4640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f413006d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41369a4640 1 -- 172.21.15.107:0/1361909858 >> v2:172.21.15.120:3300/0 conn(0x7f41300b0b40 msgr2=0x7f41300b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41369a4640 1 --2- 172.21.15.107:0/1361909858 >> v2:172.21.15.120:3300/0 conn(0x7f41300b0b40 0x7f41300b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.832+0000 7f41369a4640 1 -- 172.21.15.107:0/1361909858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413006cb00 con 0x7f413015ded0 2023-11-24T08:12:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.833+0000 7f41369a4640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413015ded0 0x7f413015e360 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f41300ccb30 tx=0x7f412802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a97d3a46149564 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.833+0000 7f41351a1640 1 -- 172.21.15.107:0/1361909858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412802ca80 con 0x7f413015ded0 2023-11-24T08:12:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.833+0000 7f41351a1640 1 -- 172.21.15.107:0/1361909858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412802cbe0 con 0x7f413015ded0 2023-11-24T08:12:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.833+0000 7f41351a1640 1 -- 172.21.15.107:0/1361909858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280343f0 con 0x7f413015ded0 2023-11-24T08:12:00.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 -- 172.21.15.107:0/1361909858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413015ded0 msgr2=0x7f413015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413015ded0 0x7f413015e360 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f41300ccb30 tx=0x7f412802bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 -- 172.21.15.107:0/1361909858 shutdown_connections 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413015ded0 0x7f413015e360 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 --2- 172.21.15.107:0/1361909858 >> v2:172.21.15.120:3300/0 conn(0x7f41300b0b40 0x7f41300b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 --2- 172.21.15.107:0/1361909858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f413006d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 -- 172.21.15.107:0/1361909858 >> 172.21.15.107:0/1361909858 conn(0x7f41300cbb40 msgr2=0x7f41300cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:00.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 -- 172.21.15.107:0/1361909858 shutdown_connections 2023-11-24T08:12:00.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.834+0000 7f41371a5640 1 -- 172.21.15.107:0/1361909858 wait complete. 2023-11-24T08:12:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.836+0000 7f41371a5640 1 Processor -- start 2023-11-24T08:12:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.836+0000 7f41371a5640 1 -- start start 2023-11-24T08:12:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f41300b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413015ded0 0x7f41300b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301f67e0 con 0x7f41300b0b40 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301f6a30 con 0x7f413015ded0 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41371a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41301f6d10 con 0x7f413015a1c0 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41361a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41361a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60764/0 (socket says 172.21.15.107:60764) 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.837+0000 7f41361a3640 1 -- 172.21.15.107:0/2741596479 learned_addr learned my addr 172.21.15.107:0/2741596479 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 -- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 msgr2=0x7f41300b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f41300b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41369a4640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413015ded0 0x7f41300b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 -- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413015ded0 msgr2=0x7f41300b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413015ded0 0x7f41300b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 -- 172.21.15.107:0/2741596479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4128009b30 con 0x7f41300b0b40 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f41361a3640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f412c00d800 tx=0x7f412c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00b6c0 con 0x7f41300b0b40 2023-11-24T08:12:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.838+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c00b820 con 0x7f41300b0b40 2023-11-24T08:12:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.839+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00bbb0 con 0x7f41300b0b40 2023-11-24T08:12:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.839+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41301f6ff0 con 0x7f41300b0b40 2023-11-24T08:12:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.839+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4130159b40 con 0x7f41300b0b40 2023-11-24T08:12:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.840+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413004e840 con 0x7f41300b0b40 2023-11-24T08:12:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.841+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f412c00bd10 con 0x7f41300b0b40 2023-11-24T08:12:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.842+0000 7f4126ffd640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 0x7f411406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.842+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f412c0887b0 con 0x7f41300b0b40 2023-11-24T08:12:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.842+0000 7f41359a2640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 0x7f411406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.843+0000 7f41359a2640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 0x7f411406ded0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f41300cb8e0 tx=0x7f4120008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:00.846+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f412c055050 con 0x7f41300b0b40 2023-11-24T08:12:01.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.035+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41300ca020 con 0x7f41300b0b40 2023-11-24T08:12:01.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.036+0000 7f4126ffd640 1 -- 172.21.15.107:0/2741596479 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f412c058670 con 0x7f41300b0b40 2023-11-24T08:12:01.038 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.038+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 msgr2=0x7f411406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.038+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 0x7f411406ded0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f41300cb8e0 tx=0x7f4120008040 comp rx=0 tx=0).stop 2023-11-24T08:12:01.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.038+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 msgr2=0x7f4130159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.038+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f412c00d800 tx=0x7f412c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 shutdown_connections 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f411406ba10 0x7f411406ded0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413015ded0 0x7f41300b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413015a1c0 0x7f41300b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 --2- 172.21.15.107:0/2741596479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41300b0b40 0x7f4130159600 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 >> 172.21.15.107:0/2741596479 conn(0x7f41300cbb40 msgr2=0x7f413006df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 shutdown_connections 2023-11-24T08:12:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.039+0000 7f41371a5640 1 -- 172.21.15.107:0/2741596479 wait complete. 2023-11-24T08:12:01.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:12:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.275+0000 7f84e4def640 1 Processor -- start 2023-11-24T08:12:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.275+0000 7f84e4def640 1 -- start start 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84e4def640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84e015d4e0 0x7f84e00d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84e4def640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e00d38e0 0x7f84e00cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84e4def640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00ce990 con 0x7f84e00d0070 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84e4def640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e00d4110 con 0x7f84e015d4e0 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84e4def640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00d04e0 con 0x7f84e00d38e0 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84df7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84e015d4e0 0x7f84e00d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84df7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84e015d4e0 0x7f84e00d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60780/0 (socket says 172.21.15.107:60780) 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84df7fe640 1 -- 172.21.15.107:0/3683843709 learned_addr learned my addr 172.21.15.107:0/3683843709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84deffd640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e00d38e0 0x7f84e00cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84df7fe640 1 -- 172.21.15.107:0/3683843709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e00d38e0 msgr2=0x7f84e00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.277+0000 7f84df7fe640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e00d38e0 0x7f84e00cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.278+0000 7f84df7fe640 1 -- 172.21.15.107:0/3683843709 >> v1:172.21.15.120:6789/0 conn(0x7f84e00d0070 legacy=0x7f84e00d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.278+0000 7f84df7fe640 1 -- 172.21.15.107:0/3683843709 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00cec70 con 0x7f84e015d4e0 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.279+0000 7f84df7fe640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e015d4e0 0x7f84e00d1e20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84e008b800 tx=0x7f84d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca83b4f9efcdd3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.279+0000 7f84de7fc640 1 -- 172.21.15.107:0/3683843709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031830 con 0x7f84e015d4e0 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.279+0000 7f84de7fc640 1 -- 172.21.15.107:0/3683843709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0031990 con 0x7f84e015d4e0 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.279+0000 7f84de7fc640 1 -- 172.21.15.107:0/3683843709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031d20 con 0x7f84e015d4e0 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 -- 172.21.15.107:0/3683843709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e015d4e0 msgr2=0x7f84e00d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e015d4e0 0x7f84e00d1e20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84e008b800 tx=0x7f84d002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 -- 172.21.15.107:0/3683843709 shutdown_connections 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e00d38e0 0x7f84e00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 --2- 172.21.15.107:0/3683843709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e015d4e0 0x7f84e00d1e20 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 -- 172.21.15.107:0/3683843709 >> 172.21.15.107:0/3683843709 conn(0x7f84e009c510 msgr2=0x7f84e00d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 -- 172.21.15.107:0/3683843709 shutdown_connections 2023-11-24T08:12:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.280+0000 7f84e4def640 1 -- 172.21.15.107:0/3683843709 wait complete. 2023-11-24T08:12:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.282+0000 7f84e4def640 1 Processor -- start 2023-11-24T08:12:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.282+0000 7f84e4def640 1 -- start start 2023-11-24T08:12:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84e00d38e0 0x7f84e00cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e015d4e0 0x7f84e014e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0160330 con 0x7f84e00d0070 2023-11-24T08:12:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0160750 con 0x7f84e00d38e0 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84e4def640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e015f3d0 con 0x7f84e015d4e0 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60782/0 (socket says 172.21.15.107:60782) 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 -- 172.21.15.107:0/164533143 learned_addr learned my addr 172.21.15.107:0/164533143 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 -- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e015d4e0 msgr2=0x7f84e014e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e015d4e0 0x7f84e014e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 -- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84e00d38e0 msgr2=0x7f84e00cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84e00d38e0 0x7f84e00cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.283+0000 7f84df7fe640 1 -- 172.21.15.107:0/164533143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00094d0 con 0x7f84e00d0070 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84df7fe640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097f0 tx=0x7f84d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d003a830 con 0x7f84e00d0070 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d003a990 con 0x7f84e00d0070 2023-11-24T08:12:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d003ad40 con 0x7f84e00d0070 2023-11-24T08:12:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e015f6b0 con 0x7f84e00d0070 2023-11-24T08:12:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.284+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e015fc70 con 0x7f84e00d0070 2023-11-24T08:12:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.285+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e0160470 con 0x7f84e00d0070 2023-11-24T08:12:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.286+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f84d004d450 con 0x7f84e00d0070 2023-11-24T08:12:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.287+0000 7f84c7fff640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 0x7f84cc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.287+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f84d00aa290 con 0x7f84e00d0070 2023-11-24T08:12:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.287+0000 7f84deffd640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 0x7f84cc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.288+0000 7f84deffd640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 0x7f84cc06ded0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84e00cbf30 tx=0x7f84d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.291+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84e0160470 con 0x7f84e00d0070 2023-11-24T08:12:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.481+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f84e0160470 con 0x7f84e00d0070 2023-11-24T08:12:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.482+0000 7f84c7fff640 1 -- 172.21.15.107:0/164533143 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) v1 ==== 84+0+65 (secure 0 0 0) 0x7f84e0160470 con 0x7f84e00d0070 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:01.484 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 msgr2=0x7f84cc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 0x7f84cc06ded0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84e00cbf30 tx=0x7f84d4008040 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 msgr2=0x7f84e015e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097f0 tx=0x7f84d00046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 shutdown_connections 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f84cc06ba10 0x7f84cc06ded0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84e015d4e0 0x7f84e014e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84e00d38e0 0x7f84e00cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 --2- 172.21.15.107:0/164533143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e00d0070 0x7f84e015e420 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.484+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 >> 172.21.15.107:0/164533143 conn(0x7f84e009c510 msgr2=0x7f84e00b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.485+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 shutdown_connections 2023-11-24T08:12:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.485+0000 7f84e4def640 1 -- 172.21.15.107:0/164533143 wait complete. 2023-11-24T08:12:01.504 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:12:01.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:12:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.717+0000 7f853ed16640 1 Processor -- start 2023-11-24T08:12:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.717+0000 7f853ed16640 1 -- start start 2023-11-24T08:12:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.718+0000 7f853ed16640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85380cf450 0x7f85380cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.718+0000 7f853ed16640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380cece0 con 0x7f85380cdfb0 2023-11-24T08:12:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.718+0000 7f853ed16640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853806d040 con 0x7f853815c7f0 2023-11-24T08:12:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.718+0000 7f853ed16640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380cfe40 con 0x7f85380cf450 2023-11-24T08:12:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.719+0000 7f853d2b3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85380cdfb0 0x7f85380ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59052/0 (socket says 172.21.15.107:59052) 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.719+0000 7f853d2b3640 1 -- 172.21.15.107:0/1638537646 learned_addr learned my addr 172.21.15.107:0/1638537646 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.719+0000 7f852ffff640 1 --2- 172.21.15.107:0/1638537646 >> v2:172.21.15.120:3300/0 conn(0x7f85380cf450 0x7f85380cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.719+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746888541 0 0) 0x7f85380cece0 con 0x7f85380cdfb0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.719+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520003520 con 0x7f85380cdfb0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35786844 0 0) 0x7f8520003520 con 0x7f85380cdfb0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 >> v2:172.21.15.120:3300/0 conn(0x7f85380cf450 msgr2=0x7f85380cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 --2- 172.21.15.107:0/1638537646 >> v2:172.21.15.120:3300/0 conn(0x7f85380cf450 0x7f85380cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 >> v1:172.21.15.120:6789/0 conn(0x7f853815c7f0 legacy=0x7f853806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853806d320 con 0x7f85380cdfb0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f85300019f0 con 0x7f85380cdfb0 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 >> v1:172.21.15.107:6789/0 conn(0x7f85380cdfb0 legacy=0x7f85380ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 0x7f8520003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200047c0 0x7f8520004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8520005180 0x7f852000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380cfe40 con 0x7f85200039c0 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852000c080 con 0x7f8520005180 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852000c240 con 0x7f85200047c0 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 0x7f8520003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 -- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200047c0 msgr2=0x7f8520004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200047c0 0x7f8520004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 -- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8520005180 msgr2=0x7f852000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8520005180 0x7f852000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853cab2640 1 -- 172.21.15.107:0/1638537646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8520003520 con 0x7f85200039c0 2023-11-24T08:12:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.720+0000 7f853d2b3640 1 -- 172.21.15.107:0/1638537646 reap_dead start 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.721+0000 7f853cab2640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 0x7f8520003de0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8534001750 tx=0x7f85200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ebc91ce01896e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85340052d0 con 0x7f85200039c0 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853402e050 con 0x7f85200039c0 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f852f7fe640 1 -- 172.21.15.107:0/1638537646 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534003830 con 0x7f85200039c0 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 -- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 msgr2=0x7f8520003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 0x7f8520003de0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8534001750 tx=0x7f85200057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 -- 172.21.15.107:0/1638537646 shutdown_connections 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 --2- 172.21.15.107:0/1638537646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85200039c0 0x7f8520003de0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 -- 172.21.15.107:0/1638537646 >> 172.21.15.107:0/1638537646 conn(0x7f853806a920 msgr2=0x7f853806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 -- 172.21.15.107:0/1638537646 shutdown_connections 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.722+0000 7f853ed16640 1 -- 172.21.15.107:0/1638537646 wait complete. 2023-11-24T08:12:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.724+0000 7f853ed16640 1 Processor -- start 2023-11-24T08:12:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.724+0000 7f853ed16640 1 -- start start 2023-11-24T08:12:01.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200039c0 0x7f85380ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853814a240 0x7f85381f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381f5490 con 0x7f853814a7b0 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381f4880 con 0x7f853814a240 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f853ed16640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85381f4b60 con 0x7f85200039c0 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f852ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f852ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60796/0 (socket says 172.21.15.107:60796) 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.725+0000 7f852ffff640 1 -- 172.21.15.107:0/3436921876 learned_addr learned my addr 172.21.15.107:0/3436921876 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f853d2b3640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853814a240 0x7f85381f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 -- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200039c0 msgr2=0x7f85380ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200039c0 0x7f85380ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 -- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853814a240 msgr2=0x7f85381f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853814a240 0x7f85381f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 -- 172.21.15.107:0/3436921876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85381f4e40 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852ffff640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f852800e770 tx=0x7f852800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852800c730 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852800c890 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852800cc40 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.726+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85381f5120 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.727+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8538147670 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.727+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538067470 con 0x7f853814a7b0 2023-11-24T08:12:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.728+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f8528013470 con 0x7f853814a7b0 2023-11-24T08:12:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.729+0000 7f852d7fa640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 0x7f852406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.729+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85280883e0 con 0x7f853814a7b0 2023-11-24T08:12:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.730+0000 7f853cab2640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 0x7f852406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:01.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.730+0000 7f853cab2640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 0x7f852406ded0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8534033000 tx=0x7f853402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:01.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.733+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8528054c80 con 0x7f853814a7b0 2023-11-24T08:12:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:01.924+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8538155330 con 0x7f853814a7b0 2023-11-24T08:12:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.662+0000 7f852d7fa640 1 -- 172.21.15.107:0/3436921876 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7f85280210b0 con 0x7f853814a7b0 2023-11-24T08:12:02.665 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:12:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 msgr2=0x7f852406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 0x7f852406ded0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8534033000 tx=0x7f853402f000 comp rx=0 tx=0).stop 2023-11-24T08:12:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 msgr2=0x7f85380ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f852800e770 tx=0x7f852800ec30 comp rx=0 tx=0).stop 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 shutdown_connections 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f852406ba10 0x7f852406ded0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f853814a240 0x7f85381f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f853814a7b0 0x7f85380ad550 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 --2- 172.21.15.107:0/3436921876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85200039c0 0x7f85380ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 >> 172.21.15.107:0/3436921876 conn(0x7f853806a920 msgr2=0x7f853806c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:02.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 shutdown_connections 2023-11-24T08:12:02.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.666+0000 7f853ed16640 1 -- 172.21.15.107:0/3436921876 wait complete. 2023-11-24T08:12:02.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:12:02.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.890+0000 7fb4d52ff640 1 Processor -- start 2023-11-24T08:12:02.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.890+0000 7fb4d52ff640 1 -- start start 2023-11-24T08:12:02.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.891+0000 7fb4d52ff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb4d015b1d0 0x7fb4d0071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.891+0000 7fb4d52ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4d00b1250 0x7fb4d015edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.891+0000 7fb4d52ff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d00ca0e0 con 0x7fb4d0070550 2023-11-24T08:12:02.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.891+0000 7fb4d52ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d0072800 con 0x7fb4d00b1250 2023-11-24T08:12:02.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.891+0000 7fb4d52ff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d00c97b0 con 0x7fb4d015b1d0 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4d4afe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4d00b1250 0x7fb4d015edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4cf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb4d0070550 0x7fb4d00b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59068/0 (socket says 172.21.15.107:59068) 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4cf7fe640 1 -- 172.21.15.107:0/1541436420 learned_addr learned my addr 172.21.15.107:0/1541436420 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4cffff640 1 --2- 172.21.15.107:0/1541436420 >> v2:172.21.15.120:3300/0 conn(0x7fb4d015b1d0 0x7fb4d0071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4d4afe640 1 -- 172.21.15.107:0/1541436420 >> v2:172.21.15.120:3300/0 conn(0x7fb4d015b1d0 msgr2=0x7fb4d0071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4d4afe640 1 --2- 172.21.15.107:0/1541436420 >> v2:172.21.15.120:3300/0 conn(0x7fb4d015b1d0 0x7fb4d0071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4d4afe640 1 -- 172.21.15.107:0/1541436420 >> v1:172.21.15.107:6789/0 conn(0x7fb4d0070550 legacy=0x7fb4d00b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4d4afe640 1 -- 172.21.15.107:0/1541436420 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00c9a90 con 0x7fb4d00b1250 2023-11-24T08:12:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.892+0000 7fb4ceffd640 1 -- 172.21.15.107:0/1541436420 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095959595 0 0) 0x7fb4d00ca0e0 con 0x7fb4d0070550 2023-11-24T08:12:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.893+0000 7fb4d4afe640 1 --2- 172.21.15.107:0/1541436420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00b1250 0x7fb4d015edd0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006e7a0 tx=0x7fb4c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed47307f846908e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4ceffd640 1 -- 172.21.15.107:0/1541436420 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c802cc80 con 0x7fb4d00b1250 2023-11-24T08:12:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4ceffd640 1 -- 172.21.15.107:0/1541436420 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c802cde0 con 0x7fb4d00b1250 2023-11-24T08:12:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4ceffd640 1 -- 172.21.15.107:0/1541436420 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8034660 con 0x7fb4d00b1250 2023-11-24T08:12:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 -- 172.21.15.107:0/1541436420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00b1250 msgr2=0x7fb4d015edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/1541436420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00b1250 0x7fb4d015edd0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006e7a0 tx=0x7fb4c802bed0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 -- 172.21.15.107:0/1541436420 shutdown_connections 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/1541436420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00b1250 0x7fb4d015edd0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/1541436420 >> v2:172.21.15.120:3300/0 conn(0x7fb4d015b1d0 0x7fb4d0071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 -- 172.21.15.107:0/1541436420 >> 172.21.15.107:0/1541436420 conn(0x7fb4d00cbb40 msgr2=0x7fb4d00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 -- 172.21.15.107:0/1541436420 shutdown_connections 2023-11-24T08:12:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.894+0000 7fb4d52ff640 1 -- 172.21.15.107:0/1541436420 wait complete. 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.896+0000 7fb4d52ff640 1 Processor -- start 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.896+0000 7fb4d52ff640 1 -- start start 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4d00b1250 0x7fb4d015df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d015b1d0 0x7fb4d00c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d009a6b0 con 0x7fb4d0070550 2023-11-24T08:12:02.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015da90 con 0x7fb4d015b1d0 2023-11-24T08:12:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4d52ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015ce80 con 0x7fb4d00b1250 2023-11-24T08:12:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60808/0 (socket says 172.21.15.107:60808) 2023-11-24T08:12:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.897+0000 7fb4cffff640 1 -- 172.21.15.107:0/2726154433 learned_addr learned my addr 172.21.15.107:0/2726154433 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 -- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4d00b1250 msgr2=0x7fb4d015df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4d00b1250 0x7fb4d015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 -- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d015b1d0 msgr2=0x7fb4d00c4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d015b1d0 0x7fb4d00c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 -- 172.21.15.107:0/2726154433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8009a70 con 0x7fb4d0070550 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4cffff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000d5f0 tx=0x7fb4c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00125f0 con 0x7fb4d0070550 2023-11-24T08:12:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0012750 con 0x7fb4d0070550 2023-11-24T08:12:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0012b00 con 0x7fb4d0070550 2023-11-24T08:12:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.898+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d015d160 con 0x7fb4d0070550 2023-11-24T08:12:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.899+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d015d720 con 0x7fb4d0070550 2023-11-24T08:12:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.900+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d004e840 con 0x7fb4d0070550 2023-11-24T08:12:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.901+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fb4c0012c60 con 0x7fb4d0070550 2023-11-24T08:12:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.903+0000 7fb4ccff9640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 0x7fb4bc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.903+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb4c0089250 con 0x7fb4d0070550 2023-11-24T08:12:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.903+0000 7fb4cf7fe640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 0x7fb4bc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.903+0000 7fb4cf7fe640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 0x7fb4bc06ded0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007fd0 tx=0x7fb4c4007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:02.907+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4c0055af0 con 0x7fb4d0070550 2023-11-24T08:12:03.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.100+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb4d00c2800 con 0x7fb4d0070550 2023-11-24T08:12:03.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.680+0000 7fb4ccff9640 1 -- 172.21.15.107:0/2726154433 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb4c0059110 con 0x7fb4d0070550 2023-11-24T08:12:03.683 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 msgr2=0x7fb4bc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 0x7fb4bc06ded0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007fd0 tx=0x7fb4c4007920 comp rx=0 tx=0).stop 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 msgr2=0x7fb4d01576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000d5f0 tx=0x7fb4c000dab0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 shutdown_connections 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb4bc06ba10 0x7fb4bc06ded0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d015b1d0 0x7fb4d00c4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4d00b1250 0x7fb4d015df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 --2- 172.21.15.107:0/2726154433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d0070550 0x7fb4d01576c0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 >> 172.21.15.107:0/2726154433 conn(0x7fb4d00cbb40 msgr2=0x7fb4d00cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:03.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 shutdown_connections 2023-11-24T08:12:03.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.684+0000 7fb4d52ff640 1 -- 172.21.15.107:0/2726154433 wait complete. 2023-11-24T08:12:03.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:12:03.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.905+0000 7f6ee2eda640 1 Processor -- start 2023-11-24T08:12:03.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.905+0000 7f6ee2eda640 1 -- start start 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.906+0000 7f6ee2eda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6edc072160 0x7f6edc15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee2eda640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 0x7f6edc06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee2eda640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc15ec90 con 0x7f6edc06dc40 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee2eda640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc06e0f0 con 0x7f6edc072160 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee2eda640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0cd530 con 0x7f6edc0c49f0 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee0c76640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6edc072160 0x7f6edc15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee1477640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6edc06dc40 0x7f6edc0cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59072/0 (socket says 172.21.15.107:59072) 2023-11-24T08:12:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee1477640 1 -- 172.21.15.107:0/1348592047 learned_addr learned my addr 172.21.15.107:0/1348592047 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6edbfff640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 0x7f6edc06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee0c76640 1 -- 172.21.15.107:0/1348592047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 msgr2=0x7f6edc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee0c76640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 0x7f6edc06d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee0c76640 1 -- 172.21.15.107:0/1348592047 >> v1:172.21.15.107:6789/0 conn(0x7f6edc06dc40 legacy=0x7f6edc0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6ee0c76640 1 -- 172.21.15.107:0/1348592047 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0c4ed0 con 0x7f6edc072160 2023-11-24T08:12:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6edbfff640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 0x7f6edc06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.907+0000 7f6edb7fe640 1 -- 172.21.15.107:0/1348592047 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255577737 0 0) 0x7f6ed4001490 con 0x7f6edc06dc40 2023-11-24T08:12:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6ee0c76640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc15e750 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097a0 tx=0x7f6ed002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310ac10e2de6ada6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6edb7fe640 1 -- 172.21.15.107:0/1348592047 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed002d040 con 0x7f6edc072160 2023-11-24T08:12:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6edb7fe640 1 -- 172.21.15.107:0/1348592047 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed0031810 con 0x7f6edc072160 2023-11-24T08:12:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6edb7fe640 1 -- 172.21.15.107:0/1348592047 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0031ba0 con 0x7f6edc072160 2023-11-24T08:12:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1348592047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 msgr2=0x7f6edc15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.909+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc15e750 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097a0 tx=0x7f6ed002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1348592047 shutdown_connections 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc0c49f0 0x7f6edc06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1348592047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc15e750 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1348592047 >> 172.21.15.107:0/1348592047 conn(0x7f6edc0b0c50 msgr2=0x7f6edc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1348592047 shutdown_connections 2023-11-24T08:12:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.910+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1348592047 wait complete. 2023-11-24T08:12:03.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.911+0000 7f6ee2eda640 1 Processor -- start 2023-11-24T08:12:03.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 -- start start 2023-11-24T08:12:03.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc06dc40 0x7f6edc144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 0x7f6edc0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc159f00 con 0x7f6edc072160 2023-11-24T08:12:03.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c7cd0 con 0x7f6edc0c49f0 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.912+0000 7f6ee2eda640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc159380 con 0x7f6edc06dc40 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60826/0 (socket says 172.21.15.107:60826) 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 -- 172.21.15.107:0/1455801855 learned_addr learned my addr 172.21.15.107:0/1455801855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6ee0c76640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc06dc40 0x7f6edc144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6ee1477640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 0x7f6edc0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 -- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc06dc40 msgr2=0x7f6edc144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc06dc40 0x7f6edc144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 -- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 msgr2=0x7f6edc0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 0x7f6edc0ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:03.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6edbfff640 1 -- 172.21.15.107:0/1455801855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009400 con 0x7f6edc072160 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.913+0000 7f6ee1477640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 0x7f6edc0ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6edbfff640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800c7f0 tx=0x7f6ec800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8007820 con 0x7f6edc072160 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec8007980 con 0x7f6edc072160 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc159660 con 0x7f6edc072160 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8007cf0 con 0x7f6edc072160 2023-11-24T08:12:03.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.914+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc159c20 con 0x7f6edc072160 2023-11-24T08:12:03.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.915+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc06d990 con 0x7f6edc072160 2023-11-24T08:12:03.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.916+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f6ec80113f0 con 0x7f6edc072160 2023-11-24T08:12:03.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.917+0000 7f6ed97fa640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 0x7f6ec006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:03.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.917+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6ec80882d0 con 0x7f6edc072160 2023-11-24T08:12:03.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.917+0000 7f6ee0c76640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 0x7f6ec006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:03.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.918+0000 7f6ee0c76640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 0x7f6ec006ded0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0036000 tx=0x7f6ed003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:03.922+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ec8054e30 con 0x7f6edc072160 2023-11-24T08:12:04.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-24T08:12:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.147+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6edc04e8b0 con 0x7f6edc072160 2023-11-24T08:12:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.697+0000 7f6ed97fa640 1 -- 172.21.15.107:0/1455801855 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6ec8058450 con 0x7f6edc072160 2023-11-24T08:12:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 12 and data pool 13 2023-11-24T08:12:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.709+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 msgr2=0x7f6ec006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.709+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 0x7f6ec006ded0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0036000 tx=0x7f6ed003c040 comp rx=0 tx=0).stop 2023-11-24T08:12:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.709+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 msgr2=0x7f6edc158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.709+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800c7f0 tx=0x7f6ec800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 shutdown_connections 2023-11-24T08:12:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ec006ba10 0x7f6ec006ded0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6edc0c49f0 0x7f6edc0ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6edc072160 0x7f6edc158b30 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 --2- 172.21.15.107:0/1455801855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6edc06dc40 0x7f6edc144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 >> 172.21.15.107:0/1455801855 conn(0x7f6edc0b0c50 msgr2=0x7f6edc071110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.714+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 shutdown_connections 2023-11-24T08:12:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.715+0000 7f6ee2eda640 1 -- 172.21.15.107:0/1455801855 wait complete. 2023-11-24T08:12:04.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:04.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.921+0000 7f74e3116640 1 Processor -- start 2023-11-24T08:12:04.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.921+0000 7f74e3116640 1 -- start start 2023-11-24T08:12:04.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.922+0000 7f74e3116640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74dc0b8ce0 0x7f74dc0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:04.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.922+0000 7f74e3116640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0d1d20 con 0x7f74dc0d0060 2023-11-24T08:12:04.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.922+0000 7f74e3116640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0cf2d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.922+0000 7f74e3116640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc161e80 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2114640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74dc0b8ce0 0x7f74dc0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2915640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f74dc0d0060 0x7f74dc161770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59076/0 (socket says 172.21.15.107:59076) 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2915640 1 -- 172.21.15.107:0/2334169291 learned_addr learned my addr 172.21.15.107:0/2334169291 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2114640 1 -- 172.21.15.107:0/2334169291 >> v1:172.21.15.120:6789/0 conn(0x7f74dc0cfbe0 legacy=0x7f74dc161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2114640 1 -- 172.21.15.107:0/2334169291 >> v1:172.21.15.107:6789/0 conn(0x7f74dc0d0060 legacy=0x7f74dc161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.923+0000 7f74e2114640 1 -- 172.21.15.107:0/2334169291 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0b90c0 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.924+0000 7f74e2114640 1 --2- 172.21.15.107:0/2334169291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc0ced90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f74d80097a0 tx=0x7f74d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef4daaddeaffa5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:04.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.924+0000 7f74e1112640 1 -- 172.21.15.107:0/2334169291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8031830 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e1112640 1 -- 172.21.15.107:0/2334169291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8031990 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e1112640 1 -- 172.21.15.107:0/2334169291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8031d20 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 -- 172.21.15.107:0/2334169291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0b8ce0 msgr2=0x7f74dc0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 --2- 172.21.15.107:0/2334169291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc0ced90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f74d80097a0 tx=0x7f74d802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 -- 172.21.15.107:0/2334169291 shutdown_connections 2023-11-24T08:12:04.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 --2- 172.21.15.107:0/2334169291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc0ced90 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 -- 172.21.15.107:0/2334169291 >> 172.21.15.107:0/2334169291 conn(0x7f74dc09c530 msgr2=0x7f74dc0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:04.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 -- 172.21.15.107:0/2334169291 shutdown_connections 2023-11-24T08:12:04.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.925+0000 7f74e3116640 1 -- 172.21.15.107:0/2334169291 wait complete. 2023-11-24T08:12:04.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.927+0000 7f74e3116640 1 Processor -- start 2023-11-24T08:12:04.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.927+0000 7f74e3116640 1 -- start start 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74dc0d0060 0x7f74dc0cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0cb220 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc153d60 con 0x7f74dc0b8ce0 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e3116640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0ca610 con 0x7f74dc0d0060 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e1913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e1913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60844/0 (socket says 172.21.15.107:60844) 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.928+0000 7f74e1913640 1 -- 172.21.15.107:0/3591873913 learned_addr learned my addr 172.21.15.107:0/3591873913 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:04.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e2114640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 -- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74dc0d0060 msgr2=0x7f74dc0cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e2915640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74dc0d0060 0x7f74dc0cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74dc0d0060 0x7f74dc0cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 -- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74dc0b8ce0 msgr2=0x7f74dc153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 -- 172.21.15.107:0/3591873913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d8009400 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e1913640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c460 tx=0x7f74cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc012830 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc0129d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc012d80 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.929+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc0ca8f0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.930+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc0caeb0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.931+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc0cb3d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.932+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f74cc01a3b0 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.933+0000 7f74d2ffd640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 0x7f74c006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.933+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74cc087e80 con 0x7f74dc0cfbe0 2023-11-24T08:12:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.933+0000 7f74e2114640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 0x7f74c006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.934+0000 7f74e2114640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 0x7f74c006ded0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f74d8036000 tx=0x7f74d803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:04.937+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74dc0cb3d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.113+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74dc0cb3d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:05.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.116+0000 7f74d2ffd640 1 -- 172.21.15.107:0/3591873913 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f74dc0cb3d0 con 0x7f74dc0cfbe0 2023-11-24T08:12:05.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:05.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:04.149077+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 msgr2=0x7f74c006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 0x7f74c006ded0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f74d8036000 tx=0x7f74d803c040 comp rx=0 tx=0).stop 2023-11-24T08:12:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 msgr2=0x7f74dc09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c460 tx=0x7f74cc00c920 comp rx=0 tx=0).stop 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 shutdown_connections 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74c006ba10 0x7f74c006ded0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74dc0d0060 0x7f74dc0cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc0cfbe0 0x7f74dc09d9b0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 --2- 172.21.15.107:0/3591873913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74dc0b8ce0 0x7f74dc153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.119+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 >> 172.21.15.107:0/3591873913 conn(0x7f74dc09c530 msgr2=0x7f74dc06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.120+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 shutdown_connections 2023-11-24T08:12:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.120+0000 7f74e3116640 1 -- 172.21.15.107:0/3591873913 wait complete. 2023-11-24T08:12:05.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:05.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.343+0000 7f7b44ffb640 1 Processor -- start 2023-11-24T08:12:05.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.344+0000 7f7b44ffb640 1 -- start start 2023-11-24T08:12:05.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.345+0000 7f7b44ffb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b4006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.345+0000 7f7b44ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400ce6c0 0x7f7b400cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.345+0000 7f7b44ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400cece0 con 0x7f7b400cf450 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.345+0000 7f7b44ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b40156010 con 0x7f7b400ce6c0 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.345+0000 7f7b44ffb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4006ce30 con 0x7f7b4015c7f0 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400ce6c0 0x7f7b400cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b35d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7b400cf450 0x7f7b400cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59078/0 (socket says 172.21.15.107:59078) 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b35d9b640 1 -- 172.21.15.107:0/2338423301 learned_addr learned my addr 172.21.15.107:0/2338423301 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:05.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3e59c640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b4006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3ed9d640 1 -- 172.21.15.107:0/2338423301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 msgr2=0x7f7b4006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3ed9d640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b4006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3ed9d640 1 -- 172.21.15.107:0/2338423301 >> v1:172.21.15.107:6789/0 conn(0x7f7b400cf450 legacy=0x7f7b400cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3ed9d640 1 -- 172.21.15.107:0/2338423301 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4006d040 con 0x7f7b400ce6c0 2023-11-24T08:12:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.346+0000 7f7b3dd9b640 1 -- 172.21.15.107:0/2338423301 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101435167 0 0) 0x7f7b400cece0 con 0x7f7b400cf450 2023-11-24T08:12:05.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.347+0000 7f7b3ed9d640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400ce6c0 0x7f7b400cf8d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009d90 tx=0x7f7b3802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4ead89718ceb87 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.347+0000 7f7b3dd9b640 1 -- 172.21.15.107:0/2338423301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802cc60 con 0x7f7b400ce6c0 2023-11-24T08:12:05.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.347+0000 7f7b3dd9b640 1 -- 172.21.15.107:0/2338423301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3802cdc0 con 0x7f7b400ce6c0 2023-11-24T08:12:05.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.347+0000 7f7b3dd9b640 1 -- 172.21.15.107:0/2338423301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38034660 con 0x7f7b400ce6c0 2023-11-24T08:12:05.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 -- 172.21.15.107:0/2338423301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400ce6c0 msgr2=0x7f7b400cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400ce6c0 0x7f7b400cf8d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009d90 tx=0x7f7b3802beb0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 -- 172.21.15.107:0/2338423301 shutdown_connections 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400ce6c0 0x7f7b400cf8d0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/2338423301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b4006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 -- 172.21.15.107:0/2338423301 >> 172.21.15.107:0/2338423301 conn(0x7f7b4006a920 msgr2=0x7f7b4006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 -- 172.21.15.107:0/2338423301 shutdown_connections 2023-11-24T08:12:05.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.348+0000 7f7b44ffb640 1 -- 172.21.15.107:0/2338423301 wait complete. 2023-11-24T08:12:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.350+0000 7f7b44ffb640 1 Processor -- start 2023-11-24T08:12:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.350+0000 7f7b44ffb640 1 -- start start 2023-11-24T08:12:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400ce6c0 0x7f7b400b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b400ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400aded0 con 0x7f7b400cf450 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400b3280 con 0x7f7b400ce6c0 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.351+0000 7f7b44ffb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400adbf0 con 0x7f7b4015c7f0 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60856/0 (socket says 172.21.15.107:60856) 2023-11-24T08:12:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 -- 172.21.15.107:0/3154266123 learned_addr learned my addr 172.21.15.107:0/3154266123 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b3e59c640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400ce6c0 0x7f7b400b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 -- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 msgr2=0x7f7b400ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b400ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 -- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400ce6c0 msgr2=0x7f7b400b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400ce6c0 0x7f7b400b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 -- 172.21.15.107:0/3154266123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38009a70 con 0x7f7b400cf450 2023-11-24T08:12:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b35d9b640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c4d0 tx=0x7f7b2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.352+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b280074e0 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.353+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28007660 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.353+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004a20 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.353+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400b2260 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.353+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b400b2820 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.354+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400b2a10 con 0x7f7b400cf450 2023-11-24T08:12:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.355+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f7b28004b80 con 0x7f7b400cf450 2023-11-24T08:12:05.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.356+0000 7f7b377fe640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 0x7f7b2006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.356+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b28087440 con 0x7f7b400cf450 2023-11-24T08:12:05.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.356+0000 7f7b3e59c640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 0x7f7b2006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.357+0000 7f7b3e59c640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 0x7f7b2006ded0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.360+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b28053ce0 con 0x7f7b400cf450 2023-11-24T08:12:05.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.555+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b40067470 con 0x7f7b400cf450 2023-11-24T08:12:05.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.556+0000 7f7b377fe640 1 -- 172.21.15.107:0/3154266123 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f7b28018070 con 0x7f7b400cf450 2023-11-24T08:12:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:05.559 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:04.149077+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:05.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 msgr2=0x7f7b2006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 0x7f7b2006ded0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 msgr2=0x7f7b40152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c4d0 tx=0x7f7b2800c990 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 shutdown_connections 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2006ba10 0x7f7b2006ded0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015c7f0 0x7f7b400ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cf450 0x7f7b40152e90 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 --2- 172.21.15.107:0/3154266123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400ce6c0 0x7f7b400b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 >> 172.21.15.107:0/3154266123 conn(0x7f7b4006a920 msgr2=0x7f7b40073560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 shutdown_connections 2023-11-24T08:12:05.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.559+0000 7f7b44ffb640 1 -- 172.21.15.107:0/3154266123 wait complete. 2023-11-24T08:12:05.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:12:05.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.790+0000 7f5094be3640 1 Processor -- start 2023-11-24T08:12:05.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.791+0000 7f5094be3640 1 -- start start 2023-11-24T08:12:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f5094be3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50900d3a40 0x7f509006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f5094be3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50900b1ed0 0x7f509006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f5094be3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509015db00 con 0x7f509015d720 2023-11-24T08:12:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f5094be3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50900d0ce0 con 0x7f50900b1ed0 2023-11-24T08:12:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f5094be3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509006d400 con 0x7f50900d3a40 2023-11-24T08:12:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f508e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f509015d720 0x7f50900d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59080/0 (socket says 172.21.15.107:59080) 2023-11-24T08:12:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.792+0000 7f508ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50900b1ed0 0x7f509006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508e59c640 1 -- 172.21.15.107:0/1660641699 learned_addr learned my addr 172.21.15.107:0/1660641699 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508dd9b640 1 --2- 172.21.15.107:0/1660641699 >> v2:172.21.15.120:3300/0 conn(0x7f50900d3a40 0x7f509006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508ed9d640 1 -- 172.21.15.107:0/1660641699 >> v2:172.21.15.120:3300/0 conn(0x7f50900d3a40 msgr2=0x7f509006ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508ed9d640 1 --2- 172.21.15.107:0/1660641699 >> v2:172.21.15.120:3300/0 conn(0x7f50900d3a40 0x7f509006ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508ed9d640 1 -- 172.21.15.107:0/1660641699 >> v1:172.21.15.107:6789/0 conn(0x7f509015d720 legacy=0x7f50900d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.793+0000 7f508ed9d640 1 -- 172.21.15.107:0/1660641699 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50900b0100 con 0x7f50900b1ed0 2023-11-24T08:12:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.794+0000 7f508ed9d640 1 --2- 172.21.15.107:0/1660641699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50900b1ed0 0x7f509006bfc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f509008d3a0 tx=0x7f508802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49d22049e0e0bb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.794+0000 7f508d59a640 1 -- 172.21.15.107:0/1660641699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508802e070 con 0x7f50900b1ed0 2023-11-24T08:12:05.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.794+0000 7f508d59a640 1 -- 172.21.15.107:0/1660641699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50880315a0 con 0x7f50900b1ed0 2023-11-24T08:12:05.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.794+0000 7f508d59a640 1 -- 172.21.15.107:0/1660641699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088031930 con 0x7f50900b1ed0 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 -- 172.21.15.107:0/1660641699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50900b1ed0 msgr2=0x7f509006bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 --2- 172.21.15.107:0/1660641699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50900b1ed0 0x7f509006bfc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f509008d3a0 tx=0x7f508802b840 comp rx=0 tx=0).stop 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 -- 172.21.15.107:0/1660641699 shutdown_connections 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 --2- 172.21.15.107:0/1660641699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50900b1ed0 0x7f509006bfc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 --2- 172.21.15.107:0/1660641699 >> v2:172.21.15.120:3300/0 conn(0x7f50900d3a40 0x7f509006ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 -- 172.21.15.107:0/1660641699 >> 172.21.15.107:0/1660641699 conn(0x7f509008af60 msgr2=0x7f509009c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 -- 172.21.15.107:0/1660641699 shutdown_connections 2023-11-24T08:12:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.795+0000 7f5094be3640 1 -- 172.21.15.107:0/1660641699 wait complete. 2023-11-24T08:12:05.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.797+0000 7f5094be3640 1 Processor -- start 2023-11-24T08:12:05.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.797+0000 7f5094be3640 1 -- start start 2023-11-24T08:12:05.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50900b1ed0 0x7f50900c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50900d3a40 0x7f50900ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900cad20 con 0x7f509015d720 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900cb000 con 0x7f50900b1ed0 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f5094be3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50900cb2e0 con 0x7f50900d3a40 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.798+0000 7f508ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60868/0 (socket says 172.21.15.107:60868) 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 -- 172.21.15.107:0/2826458923 learned_addr learned my addr 172.21.15.107:0/2826458923 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508e59c640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50900b1ed0 0x7f50900c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 -- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50900d3a40 msgr2=0x7f50900ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50900d3a40 0x7f50900ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 -- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50900b1ed0 msgr2=0x7f50900c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50900b1ed0 0x7f50900c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 -- 172.21.15.107:0/2826458923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088009400 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f508ed9d640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5088009890 tx=0x7f50880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.799+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088032840 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.800+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50880329a0 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.800+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50900cb5c0 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.800+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088032d30 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.800+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50900caa30 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.801+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50900b2c20 con 0x7f509015d720 2023-11-24T08:12:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.802+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f508804e430 con 0x7f509015d720 2023-11-24T08:12:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.803+0000 7f5086ffd640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 0x7f507006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.803+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50880aa270 con 0x7f509015d720 2023-11-24T08:12:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.804+0000 7f508e59c640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 0x7f507006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.804+0000 7f508e59c640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 0x7f507006ded0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5080005c00 tx=0x7f5080009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:05.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:05.808+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5088076b10 con 0x7f509015d720 2023-11-24T08:12:06.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.037+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f50900d1f30 con 0x7f509015d720 2023-11-24T08:12:06.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.887+0000 7f5086ffd640 1 -- 172.21.15.107:0/2826458923 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f508807a130 con 0x7f509015d720 2023-11-24T08:12:06.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 msgr2=0x7f507006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:06.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 0x7f507006ded0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5080005c00 tx=0x7f5080009040 comp rx=0 tx=0).stop 2023-11-24T08:12:06.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 msgr2=0x7f50901f32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:06.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5088009890 tx=0x7f50880046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 shutdown_connections 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507006ba10 0x7f507006ded0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509015d720 0x7f50901f32e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50900d3a40 0x7f50900ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 --2- 172.21.15.107:0/2826458923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50900b1ed0 0x7f50900c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 >> 172.21.15.107:0/2826458923 conn(0x7f509008af60 msgr2=0x7f509015fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 shutdown_connections 2023-11-24T08:12:06.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:06.891+0000 7f5094be3640 1 -- 172.21.15.107:0/2826458923 wait complete. 2023-11-24T08:12:06.914 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:12:06.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:12:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.127+0000 7f217087c640 1 Processor -- start 2023-11-24T08:12:07.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.128+0000 7f217087c640 1 -- start start 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f217087c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f216c0c49f0 0x7f216c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f217087c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c15ec90 con 0x7f216c072160 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f217087c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c0cd530 con 0x7f216c06de10 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f217087c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c0c4ed0 con 0x7f216c0c49f0 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f216a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f216c072160 0x7f216c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59096/0 (socket says 172.21.15.107:59096) 2023-11-24T08:12:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.129+0000 7f216a59c640 1 -- 172.21.15.107:0/4236075320 learned_addr learned my addr 172.21.15.107:0/4236075320 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f216ad9d640 1 --2- 172.21.15.107:0/4236075320 >> v2:172.21.15.120:3300/0 conn(0x7f216c0c49f0 0x7f216c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541404429 0 0) 0x7f215c001110 con 0x7f216c072160 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2158005730 con 0x7f216c072160 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540980927 0 0) 0x7f2158005730 con 0x7f216c072160 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 >> v2:172.21.15.120:3300/0 conn(0x7f216c0c49f0 msgr2=0x7f216c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 --2- 172.21.15.107:0/4236075320 >> v2:172.21.15.120:3300/0 conn(0x7f216c0c49f0 0x7f216c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 >> v1:172.21.15.120:6789/0 conn(0x7f216c06de10 legacy=0x7f216c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c0c51b0 con 0x7f216c072160 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f215c001800 con 0x7f216c072160 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 >> v1:172.21.15.107:6789/0 conn(0x7f216c072160 legacy=0x7f216c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158005b80 0x7f2158005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21580069d0 0x7f2158006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 0x7f215800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0c4ed0 con 0x7f2158007390 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215800e2c0 con 0x7f2158005b80 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.130+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215800e480 con 0x7f21580069d0 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 0x7f215800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216a59c640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158005b80 0x7f2158005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f2163fff640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21580069d0 0x7f2158006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 -- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21580069d0 msgr2=0x7f2158006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21580069d0 0x7f2158006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 -- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158005b80 msgr2=0x7f2158005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158005b80 0x7f2158005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.131+0000 7f216ad9d640 1 -- 172.21.15.107:0/4236075320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158005730 con 0x7f2158007390 2023-11-24T08:12:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.132+0000 7f216ad9d640 1 -- 172.21.15.107:0/4236075320 reap_dead start 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.132+0000 7f216ad9d640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 0x7f215800db90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f216c0c4e00 tx=0x7f2158007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0b24054ca8f43d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.132+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216402cca0 con 0x7f2158007390 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.132+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216402ce00 con 0x7f2158007390 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f2169d9b640 1 -- 172.21.15.107:0/4236075320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164003bd0 con 0x7f2158007390 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 -- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 msgr2=0x7f215800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 0x7f215800db90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f216c0c4e00 tx=0x7f2158007a00 comp rx=0 tx=0).stop 2023-11-24T08:12:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 -- 172.21.15.107:0/4236075320 shutdown_connections 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 --2- 172.21.15.107:0/4236075320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2158007390 0x7f215800db90 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 -- 172.21.15.107:0/4236075320 >> 172.21.15.107:0/4236075320 conn(0x7f216c0b0c50 msgr2=0x7f216c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 -- 172.21.15.107:0/4236075320 shutdown_connections 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.133+0000 7f217087c640 1 -- 172.21.15.107:0/4236075320 wait complete. 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.135+0000 7f217087c640 1 Processor -- start 2023-11-24T08:12:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.135+0000 7f217087c640 1 -- start start 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2158007390 0x7f216c0b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f216c157f90 0x7f216c149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c09b800 con 0x7f216c14a3f0 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c09ac10 con 0x7f216c157f90 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f217087c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0b21c0 con 0x7f2158007390 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.136+0000 7f216ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60890/0 (socket says 172.21.15.107:60890) 2023-11-24T08:12:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 -- 172.21.15.107:0/3745667857 learned_addr learned my addr 172.21.15.107:0/3745667857 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216a59c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2158007390 0x7f216c0b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f2163fff640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f216c157f90 0x7f216c149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 -- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2158007390 msgr2=0x7f216c0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2158007390 0x7f216c0b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 -- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f216c157f90 msgr2=0x7f216c149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f216c157f90 0x7f216c149aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 -- 172.21.15.107:0/3745667857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c149fe0 con 0x7f216c14a3f0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.137+0000 7f216ad9d640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f216402c710 tx=0x7f2164004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.138+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164039070 con 0x7f216c14a3f0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.138+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2164003e10 con 0x7f216c14a3f0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.138+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164004490 con 0x7f216c14a3f0 2023-11-24T08:12:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.138+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c09ae60 con 0x7f216c14a3f0 2023-11-24T08:12:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.138+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c09b420 con 0x7f216c14a3f0 2023-11-24T08:12:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.139+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c09b940 con 0x7f216c14a3f0 2023-11-24T08:12:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.140+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f216403d070 con 0x7f216c14a3f0 2023-11-24T08:12:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.141+0000 7f2162ffd640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 0x7f214c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.141+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21640a92d0 con 0x7f216c14a3f0 2023-11-24T08:12:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.141+0000 7f216a59c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 0x7f214c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.142+0000 7f216a59c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 0x7f214c06df00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f215c001eb0 tx=0x7f215c000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.145+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f216c09b940 con 0x7f216c14a3f0 2023-11-24T08:12:07.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.366+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f216c09b940 con 0x7f216c14a3f0 2023-11-24T08:12:07.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.899+0000 7f2162ffd640 1 -- 172.21.15.107:0/3745667857 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f216c09b940 con 0x7f216c14a3f0 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.903+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 msgr2=0x7f214c06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.903+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 0x7f214c06df00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f215c001eb0 tx=0x7f215c000ef0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.903+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 msgr2=0x7f216c0ad4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.903+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f216402c710 tx=0x7f2164004ab0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 shutdown_connections 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f214c06ba40 0x7f214c06df00 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f216c14a3f0 0x7f216c0ad4b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f216c157f90 0x7f216c149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 --2- 172.21.15.107:0/3745667857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2158007390 0x7f216c0b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 >> 172.21.15.107:0/3745667857 conn(0x7f216c0b0c50 msgr2=0x7f216c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 shutdown_connections 2023-11-24T08:12:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:07.904+0000 7f217087c640 1 -- 172.21.15.107:0/3745667857 wait complete. 2023-11-24T08:12:07.927 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:12:07.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.131+0000 7f39774da640 1 Processor -- start 2023-11-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.131+0000 7f39774da640 1 -- start start 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f39774da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f397015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f39774da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f397006d6c0 0x7f397006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f39774da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397006e060 con 0x7f39700b0a70 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f39774da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397013f9e0 con 0x7f397006d6c0 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f39774da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397013fcc0 con 0x7f397013d120 2023-11-24T08:12:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f3975cd7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f397006d6c0 0x7f397006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f3976cd9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f39700b0a70 0x7f39700b0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39182/0 (socket says 172.21.15.107:39182) 2023-11-24T08:12:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.133+0000 7f3976cd9640 1 -- 172.21.15.107:0/3370047199 learned_addr learned my addr 172.21.15.107:0/3370047199 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f39764d8640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f397015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f3975cd7640 1 -- 172.21.15.107:0/3370047199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 msgr2=0x7f397015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f3975cd7640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f397015af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f3975cd7640 1 -- 172.21.15.107:0/3370047199 >> v1:172.21.15.107:6789/0 conn(0x7f39700b0a70 legacy=0x7f39700b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f39754d6640 1 -- 172.21.15.107:0/3370047199 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545064858 0 0) 0x7f397006e060 con 0x7f39700b0a70 2023-11-24T08:12:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.134+0000 7f3975cd7640 1 -- 172.21.15.107:0/3370047199 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700cec00 con 0x7f397006d6c0 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.135+0000 7f3975cd7640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f397006db20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f39600097a0 tx=0x7f396002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794acc0d560ba910 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.135+0000 7f39754d6640 1 -- 172.21.15.107:0/3370047199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396002e070 con 0x7f397006d6c0 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.135+0000 7f39754d6640 1 -- 172.21.15.107:0/3370047199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39600315a0 con 0x7f397006d6c0 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.135+0000 7f39754d6640 1 -- 172.21.15.107:0/3370047199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960031930 con 0x7f397006d6c0 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 -- 172.21.15.107:0/3370047199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 msgr2=0x7f397006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f397006db20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f39600097a0 tx=0x7f396002b840 comp rx=0 tx=0).stop 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 -- 172.21.15.107:0/3370047199 shutdown_connections 2023-11-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f397006db20 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 --2- 172.21.15.107:0/3370047199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f397015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 -- 172.21.15.107:0/3370047199 >> 172.21.15.107:0/3370047199 conn(0x7f39700cb830 msgr2=0x7f39700cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 -- 172.21.15.107:0/3370047199 shutdown_connections 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.136+0000 7f39774da640 1 -- 172.21.15.107:0/3370047199 wait complete. 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.138+0000 7f39774da640 1 Processor -- start 2023-11-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.138+0000 7f39774da640 1 -- start start 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 0x7f3970138850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f39701f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3970138d90 con 0x7f397006d6c0 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39701f3ee0 con 0x7f39700b0a70 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39774da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39701f3510 con 0x7f397013d120 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50940/0 (socket says 172.21.15.107:50940) 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 -- 172.21.15.107:0/743133055 learned_addr learned my addr 172.21.15.107:0/743133055 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f3975cd7640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 0x7f3970138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 -- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 msgr2=0x7f39701f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f39701f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 -- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 msgr2=0x7f3970138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 0x7f3970138850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f39764d8640 1 -- 172.21.15.107:0/743133055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960009400 con 0x7f397006d6c0 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.139+0000 7f3975cd7640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 0x7f3970138850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f39764d8640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d800 tx=0x7f396c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00b6c0 con 0x7f397006d6c0 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c00b820 con 0x7f397006d6c0 2023-11-24T08:12:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3970143470 con 0x7f397006d6c0 2023-11-24T08:12:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00bbb0 con 0x7f397006d6c0 2023-11-24T08:12:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.140+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3970159bf0 con 0x7f397006d6c0 2023-11-24T08:12:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.141+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3970067990 con 0x7f397006d6c0 2023-11-24T08:12:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.142+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f396c00bd10 con 0x7f397006d6c0 2023-11-24T08:12:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.143+0000 7f3966ffd640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 0x7f395406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.143+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f396c0887f0 con 0x7f397006d6c0 2023-11-24T08:12:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.143+0000 7f3975cd7640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 0x7f395406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.144+0000 7f3975cd7640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 0x7f395406ded0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3960036000 tx=0x7f396003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.148+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f396c055090 con 0x7f397006d6c0 2023-11-24T08:12:08.234 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-11-24T08:12:08.232+0000 7f118769c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:12:08.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.360+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397015b400 con 0x7f397006d6c0 2023-11-24T08:12:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.362+0000 7f3966ffd640 1 -- 172.21.15.107:0/743133055 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f396c0586b0 con 0x7f397006d6c0 2023-11-24T08:12:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:08.364 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 51 2023-11-24T08:12:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.364+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 msgr2=0x7f395406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.364+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 0x7f395406ded0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3960036000 tx=0x7f396003d040 comp rx=0 tx=0).stop 2023-11-24T08:12:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.364+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 msgr2=0x7f39700b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.364+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d800 tx=0x7f396c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.364+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 shutdown_connections 2023-11-24T08:12:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f395406ba10 0x7f395406ded0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f397013d120 0x7f39701f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39700b0a70 0x7f3970138850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 --2- 172.21.15.107:0/743133055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397006d6c0 0x7f39700b4d80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 >> 172.21.15.107:0/743133055 conn(0x7f39700cb830 msgr2=0x7f397013e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 shutdown_connections 2023-11-24T08:12:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.365+0000 7f39774da640 1 -- 172.21.15.107:0/743133055 wait complete. 2023-11-24T08:12:08.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:08.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.594+0000 7f1a57701640 1 Processor -- start 2023-11-24T08:12:08.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.594+0000 7f1a57701640 1 -- start start 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a57701640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a501461d0 0x7f1a500b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a57701640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 0x7f1a500b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a57701640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5006d180 con 0x7f1a500b3890 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a57701640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a500b3d00 con 0x7f1a501461d0 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a57701640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5006ba70 con 0x7f1a500b4fe0 2023-11-24T08:12:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a5549d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a501461d0 0x7f1a500b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a5549d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a501461d0 0x7f1a500b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50950/0 (socket says 172.21.15.107:50950) 2023-11-24T08:12:08.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.596+0000 7f1a5549d640 1 -- 172.21.15.107:0/1638587574 learned_addr learned my addr 172.21.15.107:0/1638587574 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:08.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a54c9c640 1 --2- 172.21.15.107:0/1638587574 >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 0x7f1a500b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a5549d640 1 -- 172.21.15.107:0/1638587574 >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 msgr2=0x7f1a500b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a5549d640 1 --2- 172.21.15.107:0/1638587574 >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 0x7f1a500b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a5549d640 1 -- 172.21.15.107:0/1638587574 >> v1:172.21.15.120:6789/0 conn(0x7f1a500b3890 legacy=0x7f1a5015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a5549d640 1 -- 172.21.15.107:0/1638587574 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5006bd50 con 0x7f1a501461d0 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.597+0000 7f1a54c9c640 1 --2- 172.21.15.107:0/1638587574 >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 0x7f1a500b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:08.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.598+0000 7f1a5549d640 1 --2- 172.21.15.107:0/1638587574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a501461d0 0x7f1a500b4aa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009870 tx=0x7f1a4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe787106b0ab182f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.598+0000 7f1a47fff640 1 -- 172.21.15.107:0/1638587574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c031870 con 0x7f1a501461d0 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.598+0000 7f1a47fff640 1 -- 172.21.15.107:0/1638587574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0319d0 con 0x7f1a501461d0 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.598+0000 7f1a47fff640 1 -- 172.21.15.107:0/1638587574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c031d60 con 0x7f1a501461d0 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 -- 172.21.15.107:0/1638587574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a501461d0 msgr2=0x7f1a500b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 --2- 172.21.15.107:0/1638587574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a501461d0 0x7f1a500b4aa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009870 tx=0x7f1a4c02b910 comp rx=0 tx=0).stop 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 -- 172.21.15.107:0/1638587574 shutdown_connections 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 --2- 172.21.15.107:0/1638587574 >> v2:172.21.15.120:3300/0 conn(0x7f1a500b4fe0 0x7f1a500b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 --2- 172.21.15.107:0/1638587574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a501461d0 0x7f1a500b4aa0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 -- 172.21.15.107:0/1638587574 >> 172.21.15.107:0/1638587574 conn(0x7f1a50061340 msgr2=0x7f1a50061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 -- 172.21.15.107:0/1638587574 shutdown_connections 2023-11-24T08:12:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.599+0000 7f1a57701640 1 -- 172.21.15.107:0/1638587574 wait complete. 2023-11-24T08:12:08.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.601+0000 7f1a57701640 1 Processor -- start 2023-11-24T08:12:08.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.601+0000 7f1a57701640 1 -- start start 2023-11-24T08:12:08.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500b4fe0 0x7f1a5015fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a501461d0 0x7f1a5015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a50158f70 con 0x7f1a500b3890 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a501602a0 con 0x7f1a501461d0 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a57701640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a50158660 con 0x7f1a500b4fe0 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50956/0 (socket says 172.21.15.107:50956) 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 -- 172.21.15.107:0/1090514963 learned_addr learned my addr 172.21.15.107:0/1090514963 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 -- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500b4fe0 msgr2=0x7f1a5015fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500b4fe0 0x7f1a5015fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 -- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a501461d0 msgr2=0x7f1a5015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a501461d0 0x7f1a5015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.602+0000 7f1a5549d640 1 -- 172.21.15.107:0/1090514963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0094d0 con 0x7f1a500b3890 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a5549d640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009870 tx=0x7f1a4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c031820 con 0x7f1a500b3890 2023-11-24T08:12:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0443f0 con 0x7f1a500b3890 2023-11-24T08:12:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c0447a0 con 0x7f1a500b3890 2023-11-24T08:12:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a500caaa0 con 0x7f1a500b3890 2023-11-24T08:12:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.603+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a50158b90 con 0x7f1a500b3890 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.604+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a501590b0 con 0x7f1a500b3890 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.605+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f1a4c04d450 con 0x7f1a500b3890 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.606+0000 7f1a45ffb640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 0x7f1a3406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.607+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a4c02d040 con 0x7f1a500b3890 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.607+0000 7f1a54c9c640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 0x7f1a3406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.607+0000 7f1a54c9c640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 0x7f1a3406ded0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1a40006fd0 tx=0x7f1a40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.611+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a501590b0 con 0x7f1a500b3890 2023-11-24T08:12:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.800+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a501590b0 con 0x7f1a500b3890 2023-11-24T08:12:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.803+0000 7f1a45ffb640 1 -- 172.21.15.107:0/1090514963 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f1a501590b0 con 0x7f1a500b3890 2023-11-24T08:12:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:08.806 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:05.691679+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.805+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 msgr2=0x7f1a3406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.805+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 0x7f1a3406ded0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1a40006fd0 tx=0x7f1a40008040 comp rx=0 tx=0).stop 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.805+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 msgr2=0x7f1a5013eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.805+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009870 tx=0x7f1a4c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 shutdown_connections 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3406ba10 0x7f1a3406ded0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a501461d0 0x7f1a5015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500b4fe0 0x7f1a5015fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 --2- 172.21.15.107:0/1090514963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a500b3890 0x7f1a5013eec0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 >> 172.21.15.107:0/1090514963 conn(0x7f1a50061340 msgr2=0x7f1a500b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 shutdown_connections 2023-11-24T08:12:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:08.806+0000 7f1a57701640 1 -- 172.21.15.107:0/1090514963 wait complete. 2023-11-24T08:12:08.825 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:12:08.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:12:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.037+0000 7f10be8cf640 1 Processor -- start 2023-11-24T08:12:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.037+0000 7f10be8cf640 1 -- start start 2023-11-24T08:12:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.038+0000 7f10be8cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80b2250 0x7f10b815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.038+0000 7f10be8cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f10b80b2630 0x7f10b80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.038+0000 7f10be8cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b815ab80 con 0x7f10b80cd470 2023-11-24T08:12:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.038+0000 7f10be8cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b815a990 con 0x7f10b80b2630 2023-11-24T08:12:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.038+0000 7f10be8cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b815e680 con 0x7f10b80b2250 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.039+0000 7f10affff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f10b80b2630 0x7f10b80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.039+0000 7f10b7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80b2250 0x7f10b815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.039+0000 7f10affff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f10b80b2630 0x7f10b80ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:57788/0 (socket says 172.21.15.107:57788) 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.039+0000 7f10affff640 1 -- 172.21.15.107:0/4287134611 learned_addr learned my addr 172.21.15.107:0/4287134611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.040+0000 7f10affff640 1 -- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80b2250 msgr2=0x7f10b815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.040+0000 7f10affff640 1 --2- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80b2250 0x7f10b815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.040+0000 7f10affff640 1 -- 172.21.15.107:0/4287134611 >> v1:172.21.15.120:6789/0 conn(0x7f10b80cd470 legacy=0x7f10b815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.040+0000 7f10affff640 1 -- 172.21.15.107:0/4287134611 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b815e960 con 0x7f10b80b2630 2023-11-24T08:12:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.041+0000 7f10affff640 1 --2- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2630 0x7f10b80ccf30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009870 tx=0x7f10a002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd2efdf45bc281b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10b77fe640 1 -- 172.21.15.107:0/4287134611 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0030840 con 0x7f10b80b2630 2023-11-24T08:12:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10b77fe640 1 -- 172.21.15.107:0/4287134611 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a00309a0 con 0x7f10b80b2630 2023-11-24T08:12:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10b77fe640 1 -- 172.21.15.107:0/4287134611 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0030d30 con 0x7f10b80b2630 2023-11-24T08:12:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 -- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2630 msgr2=0x7f10b80ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 --2- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2630 0x7f10b80ccf30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009870 tx=0x7f10a002b980 comp rx=0 tx=0).stop 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 -- 172.21.15.107:0/4287134611 shutdown_connections 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 --2- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2630 0x7f10b80ccf30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 --2- 172.21.15.107:0/4287134611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80b2250 0x7f10b815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.042+0000 7f10be8cf640 1 -- 172.21.15.107:0/4287134611 >> 172.21.15.107:0/4287134611 conn(0x7f10b806a920 msgr2=0x7f10b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.043+0000 7f10be8cf640 1 -- 172.21.15.107:0/4287134611 shutdown_connections 2023-11-24T08:12:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.043+0000 7f10be8cf640 1 -- 172.21.15.107:0/4287134611 wait complete. 2023-11-24T08:12:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.044+0000 7f10be8cf640 1 Processor -- start 2023-11-24T08:12:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.044+0000 7f10be8cf640 1 -- start start 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2250 0x7f10b8149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80cd470 0x7f10b8153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80ae060 con 0x7f10b80b2630 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cab90 con 0x7f10b80b2250 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.045+0000 7f10be8cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b81541c0 con 0x7f10b80cd470 2023-11-24T08:12:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50960/0 (socket says 172.21.15.107:50960) 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 -- 172.21.15.107:0/1656391668 learned_addr learned my addr 172.21.15.107:0/1656391668 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10b7fff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2250 0x7f10b8149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10bce6c640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80cd470 0x7f10b8153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 -- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80cd470 msgr2=0x7f10b8153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80cd470 0x7f10b8153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 -- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2250 msgr2=0x7f10b8149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2250 0x7f10b8149740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.046+0000 7f10affff640 1 -- 172.21.15.107:0/1656391668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00094d0 con 0x7f10b80b2630 2023-11-24T08:12:09.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10affff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f10a0000c00 tx=0x7f10a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0039c90 con 0x7f10b80b2630 2023-11-24T08:12:09.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a0039df0 con 0x7f10b80b2630 2023-11-24T08:12:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0043640 con 0x7f10b80b2630 2023-11-24T08:12:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8154460 con 0x7f10b80b2630 2023-11-24T08:12:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.047+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b80ca820 con 0x7f10b80b2630 2023-11-24T08:12:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.049+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b80cad40 con 0x7f10b80b2630 2023-11-24T08:12:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.049+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f10a00437a0 con 0x7f10b80b2630 2023-11-24T08:12:09.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.051+0000 7f10b57fa640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 0x7f109806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.051+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a003e070 con 0x7f10b80b2630 2023-11-24T08:12:09.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.051+0000 7f10b7fff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 0x7f109806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.052+0000 7f10b7fff640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 0x7f109806df20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f10b81f6980 tx=0x7f10a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.055+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10b80cad40 con 0x7f10b80b2630 2023-11-24T08:12:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.244+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10b80b2a70 con 0x7f109806ba60 2023-11-24T08:12:09.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.265+0000 7f10b57fa640 1 -- 172.21.15.107:0/1656391668 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+275590 (secure 0 0 0) 0x7f10a80119a0 con 0x7f109806ba60 2023-11-24T08:12:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.271+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 msgr2=0x7f109806df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.271+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 0x7f109806df20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f10b81f6980 tx=0x7f10a8008040 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.271+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 msgr2=0x7f10b81490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.271+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f10a0000c00 tx=0x7f10a0004730 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 shutdown_connections 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f109806ba60 0x7f109806df20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10b80cd470 0x7f10b8153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10b80b2630 0x7f10b81490b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 --2- 172.21.15.107:0/1656391668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10b80b2250 0x7f10b8149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 >> 172.21.15.107:0/1656391668 conn(0x7f10b806a920 msgr2=0x7f10b8158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 shutdown_connections 2023-11-24T08:12:09.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.272+0000 7f10be8cf640 1 -- 172.21.15.107:0/1656391668 wait complete. 2023-11-24T08:12:09.294 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2023-11-24T08:12:07.568288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267636,"kb_used_data":9872,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754707084,"statfs":{"total":773094113280,"available":772820054016,"internally_reserved":0,"allocated":10108928,"data_stored":5013318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":263835198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.903449"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.679553+0000","last_change":"2023-11-24T08:12:02.679553+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.679553+0000","last_undegraded":"2023-11-24T08:12:02.679553+0000","last_fullsized":"2023-11-24T08:12:02.679553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.672066+0000","last_change":"2023-11-24T08:12:02.672066+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.672066+0000","last_undegraded":"2023-11-24T08:12:02.672066+0000","last_fullsized":"2023-11-24T08:12:02.672066+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.681204+0000","last_change":"2023-11-24T08:12:02.681204+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.681204+0000","last_undegraded":"2023-11-24T08:12:02.681204+0000","last_fullsized":"2023-11-24T08:12:02.681204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.680847+0000","last_change":"2023-11-24T08:12:02.680847+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.680847+0000","last_undegraded":"2023-11-24T08:12:02.680847+0000","last_fullsized":"2023-11-24T08:12:02.680847+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.671853+0000","last_change":"2023-11-24T08:12:02.671853+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.671853+0000","last_undegraded":"2023-11-24T08:12:02.671853+0000","last_fullsized":"2023-11-24T08:12:02.671853+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.672335+0000","last_change":"2023-11-24T08:12:02.672335+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.672335+0000","last_undegraded":"2023-11-24T08:12:02.672335+0000","last_fullsized":"2023-11-24T08:12:02.672335+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677837+0000","last_change":"2023-11-24T08:12:02.677837+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677837+0000","last_undegraded":"2023-11-24T08:12:02.677837+0000","last_fullsized":"2023-11-24T08:12:02.677837+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.673839+0000","last_change":"2023-11-24T08:12:02.673839+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.673839+0000","last_undegraded":"2023-11-24T08:12:02.673839+0000","last_fullsized":"2023-11-24T08:12:02.673839+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677814+0000","last_change":"2023-11-24T08:12:02.677814+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677814+0000","last_undegraded":"2023-11-24T08:12:02.677814+0000","last_fullsized":"2023-11-24T08:12:02.677814+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.676276+0000","last_change":"2023-11-24T08:12:02.676276+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.676276+0000","last_undegraded":"2023-11-24T08:12:02.676276+0000","last_fullsized":"2023-11-24T08:12:02.676276+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677621+0000","last_change":"2023-11-24T08:12:02.677621+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677621+0000","last_undegraded":"2023-11-24T08:12:02.677621+0000","last_fullsized":"2023-11-24T08:12:02.677621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.683913+0000","last_change":"2023-11-24T08:12:02.683913+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.683913+0000","last_undegraded":"2023-11-24T08:12:02.683913+0000","last_fullsized":"2023-11-24T08:12:02.683913+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.673815+0000","last_change":"2023-11-24T08:12:02.673815+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.673815+0000","last_undegraded":"2023-11-24T08:12:02.673815+0000","last_fullsized":"2023-11-24T08:12:02.673815+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.675834+0000","last_change":"2023-11-24T08:12:02.675834+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.675834+0000","last_undegraded":"2023-11-24T08:12:02.675834+0000","last_fullsized":"2023-11-24T08:12:02.675834+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.683935+0000","last_change":"2023-11-24T08:12:02.683935+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.683935+0000","last_undegraded":"2023-11-24T08:12:02.683935+0000","last_fullsized":"2023-11-24T08:12:02.683935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677640+0000","last_change":"2023-11-24T08:12:02.677640+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677640+0000","last_undegraded":"2023-11-24T08:12:02.677640+0000","last_fullsized":"2023-11-24T08:12:02.677640+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.676342+0000","last_change":"2023-11-24T08:12:02.676342+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.676342+0000","last_undegraded":"2023-11-24T08:12:02.676342+0000","last_fullsized":"2023-11-24T08:12:02.676342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677800+0000","last_change":"2023-11-24T08:12:02.677800+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677800+0000","last_undegraded":"2023-11-24T08:12:02.677800+0000","last_fullsized":"2023-11-24T08:12:02.677800+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677764+0000","last_change":"2023-11-24T08:12:02.677764+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677764+0000","last_undegraded":"2023-11-24T08:12:02.677764+0000","last_fullsized":"2023-11-24T08:12:02.677764+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.675841+0000","last_change":"2023-11-24T08:12:02.675841+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.675841+0000","last_undegraded":"2023-11-24T08:12:02.675841+0000","last_fullsized":"2023-11-24T08:12:02.675841+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.680156+0000","last_change":"2023-11-24T08:12:02.680156+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.680156+0000","last_undegraded":"2023-11-24T08:12:02.680156+0000","last_fullsized":"2023-11-24T08:12:02.680156+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.680526+0000","last_change":"2023-11-24T08:12:02.680526+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.680526+0000","last_undegraded":"2023-11-24T08:12:02.680526+0000","last_fullsized":"2023-11-24T08:12:02.680526+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.675882+0000","last_change":"2023-11-24T08:12:02.675882+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.675882+0000","last_undegraded":"2023-11-24T08:12:02.675882+0000","last_fullsized":"2023-11-24T08:12:02.675882+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.674007+0000","last_change":"2023-11-24T08:12:02.674007+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.674007+0000","last_undegraded":"2023-11-24T08:12:02.674007+0000","last_fullsized":"2023-11-24T08:12:02.674007+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.673917+0000","last_change":"2023-11-24T08:12:02.673917+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.673917+0000","last_undegraded":"2023-11-24T08:12:02.673917+0000","last_fullsized":"2023-11-24T08:12:02.673917+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.676282+0000","last_change":"2023-11-24T08:12:02.676282+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.676282+0000","last_undegraded":"2023-11-24T08:12:02.676282+0000","last_fullsized":"2023-11-24T08:12:02.676282+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.676165+0000","last_change":"2023-11-24T08:12:02.676165+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.676165+0000","last_undegraded":"2023-11-24T08:12:02.676165+0000","last_fullsized":"2023-11-24T08:12:02.676165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.677776+0000","last_change":"2023-11-24T08:12:02.677776+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.677776+0000","last_undegraded":"2023-11-24T08:12:02.677776+0000","last_fullsized":"2023-11-24T08:12:02.677776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.671181+0000","last_change":"2023-11-24T08:12:02.671181+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.671181+0000","last_undegraded":"2023-11-24T08:12:02.671181+0000","last_fullsized":"2023-11-24T08:12:02.671181+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.676364+0000","last_change":"2023-11-24T08:12:02.676364+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.676364+0000","last_undegraded":"2023-11-24T08:12:02.676364+0000","last_fullsized":"2023-11-24T08:12:02.676364+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.670565+0000","last_change":"2023-11-24T08:12:02.670565+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.670565+0000","last_undegraded":"2023-11-24T08:12:02.670565+0000","last_fullsized":"2023-11-24T08:12:02.670565+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T08:12:03.664703+0000","last_change":"2023-11-24T08:12:03.664703+0000","last_active":"2023-11-24T08:12:03.664703+0000","last_peered":"2023-11-24T08:12:03.664703+0000","last_clean":"2023-11-24T08:12:03.664703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:03.664703+0000","last_undegraded":"2023-11-24T08:12:03.664703+0000","last_fullsized":"2023-11-24T08:12:03.664703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2023-11-24T08:12:02.670633+0000","last_change":"2023-11-24T08:12:02.670633+0000","last_active":"2023-11-24T08:12:02.648370+0000","last_peered":"2023-11-24T08:12:02.648370+0000","last_clean":"2023-11-24T08:12:02.648370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:02.670633+0000","last_undegraded":"2023-11-24T08:12:02.670633+0000","last_fullsized":"2023-11-24T08:12:02.670633+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-11-24T08:11:59.690614+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:11:59.690614+0000","last_peered":"2023-11-24T08:11:59.690614+0000","last_clean":"2023-11-24T08:11:59.690614+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:11:59.690614+0000","last_undegraded":"2023-11-24T08:11:59.690614+0000","last_fullsized":"2023-11-24T08:11:59.690614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1712128,"data_stored":1070895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0669999999999999,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.532,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1359999999999999,"15min":1.0860000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.52900000000000003},"max":{"1min":1.794,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.6419999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.161,"15min":1.1439999999999999},"min":{"1min":0.64700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6120000000000001,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5},{"interface":"front","average":{"1min":1.089,"5min":1.202,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.6859999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.2669999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.151,"15min":1.2050000000000001},"min":{"1min":0.71699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8859999999999999,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3340000000000001,"15min":1.361},"min":{"1min":1.0649999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7410000000000001,"5min":4.0650000000000004,"15min":4.633},"last":1.2090000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.216,"15min":1.1619999999999999},"min":{"1min":0.82299999999999995,"5min":0.504,"15min":0.504},"max":{"1min":1.619,"5min":3.133,"15min":3.133},"last":1.071},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4039999999999999,"15min":1.377},"min":{"1min":1.0740000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8999999999999999,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.1220000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.355,"15min":1.3340000000000001},"min":{"1min":0.96199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.9610000000000001,"5min":2.851,"15min":2.851},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.3819999999999999,"15min":1.3420000000000001},"min":{"1min":0.96499999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.788,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.23}]},{"osd":5,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.5049999999999999,"15min":1.4690000000000001},"min":{"1min":1.1140000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0089999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4810000000000001},"min":{"1min":1.0649999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8560000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.554}]},{"osd":6,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5680000000000001,"15min":1.5429999999999999},"min":{"1min":1.1020000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0499999999999998,"5min":2.992,"15min":2.992},"last":1.413},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5149999999999999,"15min":1.46},"min":{"1min":1.022,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8069999999999999,"5min":3.798,"15min":3.798},"last":1.171}]}]},{"osd":6,"up_from":13,"seq":55834574939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":480527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.0880000000000001,"15min":1.085},"min":{"1min":0.64700000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.631,"5min":1.95,"15min":1.95},"last":1.004},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0720000000000001,"15min":1.0980000000000001},"min":{"1min":0.68200000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5669999999999999,"5min":1.986,"15min":2.2069999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1539999999999999,"15min":1.248},"min":{"1min":0.85799999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6799999999999999,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.155},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1319999999999999,"15min":1.2609999999999999},"min":{"1min":0.93500000000000005,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5429999999999999,"5min":1.694,"15min":2.4039999999999999},"last":0.93500000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0980000000000001,"15min":1.1160000000000001},"min":{"1min":0.72499999999999998,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.6779999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.226,"15min":1.361},"min":{"1min":0.92800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.8220000000000001,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.288}]},{"osd":3,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.333,"15min":1.395},"min":{"1min":0.96099999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.325,"15min":1.3460000000000001},"min":{"1min":1.0489999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.4750000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.375,"15min":1.5289999999999999},"min":{"1min":0.85399999999999998,"5min":0.79700000000000004,"15min":0.76500000000000001},"max":{"1min":1.829,"5min":2.895,"15min":2.895},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4510000000000001,"15min":1.5489999999999999},"min":{"1min":1.101,"5min":0.97999999999999998,"15min":0.90800000000000003},"max":{"1min":1.901,"5min":2.2770000000000001,"15min":2.8820000000000001},"last":1.3220000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4059999999999999,"15min":1.538},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":1.744,"5min":2.2250000000000001,"15min":2.629},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5169999999999999,"15min":1.671},"min":{"1min":1.119,"5min":1.0820000000000001,"15min":1.022},"max":{"1min":2.032,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.423}]},{"osd":7,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5389999999999999,"15min":1.7030000000000001},"min":{"1min":1.234,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":1.931,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5429999999999999,"15min":1.669},"min":{"1min":1.077,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.0299999999999998,"5min":2.3660000000000001,"15min":2.6890000000000001},"last":1.077}]}]},{"osd":5,"up_from":13,"seq":55834574939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":474710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.018,"15min":1.1240000000000001},"min":{"1min":0.58599999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.694,"5min":1.694,"15min":3.1179999999999999},"last":1.04},{"interface":"front","average":{"1min":1.107,"5min":1.125,"15min":1.1850000000000001},"min":{"1min":0.65100000000000002,"5min":0.64500000000000002,"15min":0.60999999999999999},"max":{"1min":1.859,"5min":1.859,"15min":2.198},"last":1.377}]},{"osd":1,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.093,"15min":1.169},"min":{"1min":0.73699999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.3939999999999999,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.155,"15min":1.147},"min":{"1min":0.80800000000000005,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.548,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.331}]},{"osd":2,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1919999999999999,"15min":1.2130000000000001},"min":{"1min":0.71099999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.768,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.1339999999999999,"15min":1.1699999999999999},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6839999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1339999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.117,"15min":1.1850000000000001},"min":{"1min":0.72399999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.2989999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.228},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2589999999999999,"15min":1.341},"min":{"1min":0.80900000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.748,"5min":1.9730000000000001,"15min":1.99},"last":1.472}]},{"osd":4,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3180000000000001,"15min":1.345},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4410000000000001,"15min":1.55},"min":{"1min":0.75600000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.956,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.534}]},{"osd":6,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3100000000000001,"15min":1.4530000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3939999999999999,"15min":1.4359999999999999},"min":{"1min":0.69299999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.79,"5min":1.9179999999999999,"15min":2.1099999999999999},"last":1.4319999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:10:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.446,"15min":1.5940000000000001},"min":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8420000000000001,"5min":2.093,"15min":3.4420000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.609,"5min":1.575,"15min":1.617},"min":{"1min":0.78900000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.5629999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":480527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0720000000000001,"15min":1.139},"min":{"1min":0.71899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5760000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.129,"5min":1.0569999999999999,"15min":1.085},"min":{"1min":0.68300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5489999999999999,"5min":1.7150000000000001,"15min":2.222},"last":0.86799999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.627,"5min":1.74,"15min":1.74},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.1639999999999999,"15min":1.1830000000000001},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5589999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.26}]},{"osd":2,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0620000000000001,"15min":1.1519999999999999},"min":{"1min":0.874,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5780000000000001,"5min":1.5840000000000001,"15min":2.1419999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.248,"5min":1.23,"15min":1.347},"min":{"1min":0.85499999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.649,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":1.1539999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.292,"15min":1.4119999999999999},"min":{"1min":0.95699999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.7230000000000001,"5min":2.121,"15min":2.9100000000000001},"last":1.028},{"interface":"front","average":{"1min":1.276,"5min":1.242,"15min":1.3160000000000001},"min":{"1min":1.0249999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5049999999999999,"5min":1.778,"15min":1.8280000000000001},"last":1.3720000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4510000000000001,"15min":1.5049999999999999},"min":{"1min":1.0009999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.8440000000000001,"5min":2.23,"15min":2.23},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.395,"15min":1.526},"min":{"1min":1.1699999999999999,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.6220000000000001,"5min":2.1890000000000001,"15min":2.3879999999999999},"last":1.484}]},{"osd":6,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2190000000000001,"15min":1.4199999999999999},"min":{"1min":0.88700000000000001,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.7030000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.097},{"interface":"front","average":{"1min":1.476,"5min":1.5049999999999999,"15min":1.6399999999999999},"min":{"1min":1.159,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.794,"5min":2.1230000000000002,"15min":3.391},"last":1.3180000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4570000000000001,"15min":1.6080000000000001},"min":{"1min":0.98999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.6870000000000001,"5min":2.0750000000000002,"15min":3.1230000000000002},"last":1.544},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.5680000000000001},"min":{"1min":1.056,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6659999999999999,"5min":1.9419999999999999,"15min":3.4849999999999999},"last":1.204}]}]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":480527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.93799999999999994,"15min":0.83799999999999997},"min":{"1min":0.44500000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.1000000000000001,"5min":10.557,"15min":10.557},"last":1.306},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.363,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":1.075}]},{"osd":1,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78800000000000003,"15min":0.82899999999999996},"min":{"1min":0.45500000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.006,"5min":1.0720000000000001,"15min":0.996},"min":{"1min":0.58499999999999996,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":2.1909999999999998,"5min":13.090999999999999,"15min":13.090999999999999},"last":1.121}]},{"osd":2,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.91000000000000003,"15min":0.95699999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.3700000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.1950000000000001,"15min":1.0720000000000001},"min":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.9690000000000001,"5min":13.460000000000001,"15min":13.460000000000001},"last":1.3839999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2689999999999999,"15min":1.272},"min":{"1min":0.71199999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.639,"5min":4.367,"15min":4.367},"last":1.413},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2210000000000001,"15min":1.3109999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.2569999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3779999999999999,"15min":1.3140000000000001},"min":{"1min":0.745,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2869999999999999,"5min":13.231,"15min":13.231},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2450000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8340000000000001,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":1.2330000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4550000000000001,"15min":1.3919999999999999},"min":{"1min":0.81599999999999995,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":3.1120000000000001,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.53},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.222,"15min":1.333},"min":{"1min":0.73799999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.5580000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3140000000000001,"15min":1.4159999999999999},"min":{"1min":0.82599999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.101,"5min":4.266,"15min":4.5060000000000002},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.2889999999999999,"15min":1.4410000000000001},"min":{"1min":0.76900000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.331,"5min":3.331,"15min":4.032},"last":1.639}]}]},{"osd":2,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36432,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335408,"statfs":{"total":96636764160,"available":96599457792,"internally_reserved":0,"allocated":1118208,"data_stored":480527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.76800000000000002,"15min":0.89000000000000001},"min":{"1min":0.47599999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.86299999999999999,"15min":0.99299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":2.5779999999999998},"last":1.179}]},{"osd":1,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.873,"15min":1.089},"min":{"1min":0.49399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.51,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.93300000000000005,"15min":1.0980000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5920000000000001,"5min":1.8360000000000001,"15min":2.6240000000000001},"last":1.3069999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.84599999999999997,"15min":0.93600000000000005},"min":{"1min":0.56499999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.552,"5min":1.552,"15min":2.2090000000000001},"last":0.874},{"interface":"front","average":{"1min":1.089,"5min":1.0069999999999999,"15min":1.2250000000000001},"min":{"1min":0.627,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.631,"5min":1.946,"15min":2.9950000000000001},"last":1.339}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.27,"15min":1.391},"min":{"1min":0.93400000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7829999999999999,"5min":1.9730000000000001,"15min":2.7160000000000002},"last":1.6659999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.2549999999999999,"15min":1.381},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8779999999999999,"5min":1.929,"15min":3.3279999999999998},"last":1.6319999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.25,"15min":1.423},"min":{"1min":0.88500000000000001,"5min":0.75800000000000001,"15min":0.73399999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.552},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2390000000000001,"15min":1.385},"min":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":1.516}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1930000000000001,"15min":1.3220000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.323,"15min":1.4870000000000001},"min":{"1min":0.98199999999999998,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.6060000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3180000000000001,"15min":1.484},"min":{"1min":0.77700000000000002,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.1070000000000002,"5min":2.1299999999999999,"15min":2.859},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.2669999999999999,"15min":1.4490000000000001},"min":{"1min":0.88600000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.9930000000000001,"5min":2.02,"15min":3.3809999999999998},"last":1.446}]}]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36432,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335408,"statfs":{"total":96636764160,"available":96599457792,"internally_reserved":0,"allocated":1118208,"data_stored":480527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.81000000000000005,"15min":0.76600000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3400000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.84599999999999997},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":1.6970000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81999999999999995,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.369,"5min":1.861,"15min":1.861},"last":0.622},{"interface":"front","average":{"1min":0.878,"5min":0.92700000000000005,"15min":0.88300000000000001},"min":{"1min":0.46500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.46500000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0009999999999999,"15min":0.95599999999999996},"min":{"1min":0.53300000000000003,"5min":0.497,"15min":0.44500000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.94299999999999995,"15min":0.88600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.24},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.028,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.276,"15min":1.2549999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.016,"5min":2.8300000000000001,"15min":3.0409999999999999},"last":1.034}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2869999999999999,"15min":1.208},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.9610000000000001,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.2470000000000001,"15min":1.181},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8759999999999999,"5min":3.286,"15min":3.286},"last":0.97599999999999998}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3460000000000001,"15min":1.238},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.5270000000000001,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3149999999999999,"15min":1.2110000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9179999999999999,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.0089999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3500000000000001,"15min":1.3009999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.3959999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.377,"15min":1.276},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0649999999999999,"5min":3.653,"15min":3.653},"last":1.0529999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1699840,"data_stored":1065078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72299999999999998,"15min":0.85599999999999998},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0189999999999999,"5min":1.425,"15min":1.6020000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.83599999999999997,"15min":0.98899999999999999},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.181,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.006,"15min":1.056},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.411,"5min":1.891,"15min":1.891},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.93999999999999995},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96599999999999997,"5min":1.847,"15min":1.847},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98599999999999999,"15min":1.101},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.724,"5min":1.79,"15min":2.0670000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88900000000000001,"15min":1.036},"min":{"1min":0.53400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.494,"5min":1.494,"15min":2.1789999999999998},"last":0.69499999999999995}]},{"osd":4,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.256,"15min":1.3899999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.528,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":1.014},{"interface":"front","average":{"1min":1.151,"5min":1.0980000000000001,"15min":1.2290000000000001},"min":{"1min":0.76100000000000001,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6240000000000001,"5min":1.6890000000000001,"15min":2.2839999999999998},"last":0.88100000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.141,"15min":1.298},"min":{"1min":0.72799999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6519999999999999,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.177,"15min":1.399},"min":{"1min":0.83699999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.47,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":0.83699999999999997}]},{"osd":6,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.26,"15min":1.417},"min":{"1min":0.71099999999999997,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.649,"5min":1.986,"15min":2.4369999999999998},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.2569999999999999,"15min":1.427},"min":{"1min":0.86399999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.766,"5min":1.9399999999999999,"15min":2.738},"last":1.0409999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.355,"15min":1.5049999999999999},"min":{"1min":0.879,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.3319999999999999,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.073},{"interface":"front","average":{"1min":1.385,"5min":1.3200000000000001,"15min":1.4930000000000001},"min":{"1min":0.996,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.0549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:12:09.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:12:09.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.551+0000 7f1280768640 1 Processor -- start 2023-11-24T08:12:09.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.551+0000 7f1280768640 1 -- start start 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f1280768640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f12780d04f0 0x7f12780b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f1280768640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12780b2740 0x7f12780ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f1280768640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12780cdf90 con 0x7f1278160d40 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f1280768640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12780ce230 con 0x7f12780b2740 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f1280768640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12780d2430 con 0x7f12780d04f0 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12780b2740 0x7f12780ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f12780b2740 0x7f12780ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50972/0 (socket says 172.21.15.107:50972) 2023-11-24T08:12:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 -- 172.21.15.107:0/2750194306 learned_addr learned my addr 172.21.15.107:0/2750194306 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 -- 172.21.15.107:0/2750194306 >> v2:172.21.15.120:3300/0 conn(0x7f12780d04f0 msgr2=0x7f12780b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 --2- 172.21.15.107:0/2750194306 >> v2:172.21.15.120:3300/0 conn(0x7f12780d04f0 0x7f12780b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 -- 172.21.15.107:0/2750194306 >> v1:172.21.15.120:6789/0 conn(0x7f1278160d40 legacy=0x7f12780cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.553+0000 7f127ed05640 1 -- 172.21.15.107:0/2750194306 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780d1b90 con 0x7f12780b2740 2023-11-24T08:12:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f127ed05640 1 --2- 172.21.15.107:0/2750194306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780b2740 0x7f12780ce710 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f127808d450 tx=0x7f127002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dba4e642e10b28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f127d502640 1 -- 172.21.15.107:0/2750194306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270031870 con 0x7f12780b2740 2023-11-24T08:12:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f127d502640 1 -- 172.21.15.107:0/2750194306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12700319d0 con 0x7f12780b2740 2023-11-24T08:12:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f127d502640 1 -- 172.21.15.107:0/2750194306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270031d60 con 0x7f12780b2740 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f1280768640 1 -- 172.21.15.107:0/2750194306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780b2740 msgr2=0x7f12780ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.555+0000 7f1280768640 1 --2- 172.21.15.107:0/2750194306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780b2740 0x7f12780ce710 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f127808d450 tx=0x7f127002b910 comp rx=0 tx=0).stop 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 -- 172.21.15.107:0/2750194306 shutdown_connections 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 --2- 172.21.15.107:0/2750194306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780b2740 0x7f12780ce710 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 --2- 172.21.15.107:0/2750194306 >> v2:172.21.15.120:3300/0 conn(0x7f12780d04f0 0x7f12780b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 -- 172.21.15.107:0/2750194306 >> 172.21.15.107:0/2750194306 conn(0x7f127808af60 msgr2=0x7f12780d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 -- 172.21.15.107:0/2750194306 shutdown_connections 2023-11-24T08:12:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.556+0000 7f1280768640 1 -- 172.21.15.107:0/2750194306 wait complete. 2023-11-24T08:12:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.557+0000 7f1280768640 1 Processor -- start 2023-11-24T08:12:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.558+0000 7f1280768640 1 -- start start 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.558+0000 7f1280768640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 0x7f127814ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.558+0000 7f1280768640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.558+0000 7f1280768640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1278160d40 0x7f12780b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.558+0000 7f1280768640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780b8960 con 0x7f12780d04f0 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f1280768640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780cbe80 con 0x7f12780b2740 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f1280768640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127814f160 con 0x7f1278160d40 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50978/0 (socket says 172.21.15.107:50978) 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 -- 172.21.15.107:0/3043542333 learned_addr learned my addr 172.21.15.107:0/3043542333 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127e504640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 0x7f127814ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127ed05640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1278160d40 0x7f12780b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 -- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1278160d40 msgr2=0x7f12780b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1278160d40 0x7f12780b7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 -- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 msgr2=0x7f127814ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 0x7f127814ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127dd03640 1 -- 172.21.15.107:0/3043542333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700094d0 con 0x7f12780d04f0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.559+0000 7f127e504640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 0x7f127814ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f127dd03640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f126c00ba50 tx=0x7f126c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c00cbe0 con 0x7f12780d04f0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c00cd60 con 0x7f12780d04f0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c014680 con 0x7f12780d04f0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12780b8030 con 0x7f12780d04f0 2023-11-24T08:12:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.560+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12780b85f0 con 0x7f12780d04f0 2023-11-24T08:12:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.561+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12780b8ad0 con 0x7f12780d04f0 2023-11-24T08:12:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.562+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f126c01d3b0 con 0x7f12780d04f0 2023-11-24T08:12:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.563+0000 7f126affd640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 0x7f125c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.563+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f126c0881e0 con 0x7f12780d04f0 2023-11-24T08:12:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.563+0000 7f127e504640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 0x7f125c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:09.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.564+0000 7f127e504640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 0x7f125c06ded0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f12780c9510 tx=0x7f1274009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:09.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.567+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f126c054ae0 con 0x7f12780d04f0 2023-11-24T08:12:09.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.755+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12780d2150 con 0x7f125c06ba10 2023-11-24T08:12:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.772+0000 7f126affd640 1 -- 172.21.15.107:0/3043542333 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287469 (secure 0 0 0) 0x7f1274005050 con 0x7f125c06ba10 2023-11-24T08:12:09.778 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:12:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 msgr2=0x7f125c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 0x7f125c06ded0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f12780c9510 tx=0x7f1274009040 comp rx=0 tx=0).stop 2023-11-24T08:12:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 msgr2=0x7f12780b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f126c00ba50 tx=0x7f126c00bf10 comp rx=0 tx=0).stop 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 shutdown_connections 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f125c06ba10 0x7f125c06ded0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1278160d40 0x7f12780b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12780d04f0 0x7f12780b72d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 --2- 172.21.15.107:0/3043542333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12780b2740 0x7f127814ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 >> 172.21.15.107:0/3043542333 conn(0x7f127808af60 msgr2=0x7f12780ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 shutdown_connections 2023-11-24T08:12:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:09.779+0000 7f1280768640 1 -- 172.21.15.107:0/3043542333 wait complete. 2023-11-24T08:12:09.801 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2023-11-24T08:12:09.568929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268036,"kb_used_data":10248,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754706684,"statfs":{"total":773094113280,"available":772819644416,"internally_reserved":0,"allocated":10493952,"data_stored":5184572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":263835198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001965"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703159+0000","last_change":"2023-11-24T08:12:04.717389+0000","last_active":"2023-11-24T08:12:05.703159+0000","last_peered":"2023-11-24T08:12:05.703159+0000","last_clean":"2023-11-24T08:12:05.703159+0000","last_became_active":"2023-11-24T08:12:04.716626+0000","last_became_peered":"2023-11-24T08:12:04.716626+0000","last_unstale":"2023-11-24T08:12:05.703159+0000","last_undegraded":"2023-11-24T08:12:05.703159+0000","last_fullsized":"2023-11-24T08:12:05.703159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:09.489727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548481+0000","last_change":"2023-11-24T08:12:04.714891+0000","last_active":"2023-11-24T08:12:06.548481+0000","last_peered":"2023-11-24T08:12:06.548481+0000","last_clean":"2023-11-24T08:12:06.548481+0000","last_became_active":"2023-11-24T08:12:04.714575+0000","last_became_peered":"2023-11-24T08:12:04.714575+0000","last_unstale":"2023-11-24T08:12:06.548481+0000","last_undegraded":"2023-11-24T08:12:06.548481+0000","last_fullsized":"2023-11-24T08:12:06.548481+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:46.267989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.053320+0000","last_change":"2023-11-24T08:12:04.706275+0000","last_active":"2023-11-24T08:12:06.053320+0000","last_peered":"2023-11-24T08:12:06.053320+0000","last_clean":"2023-11-24T08:12:06.053320+0000","last_became_active":"2023-11-24T08:12:04.705085+0000","last_became_peered":"2023-11-24T08:12:04.705085+0000","last_unstale":"2023-11-24T08:12:06.053320+0000","last_undegraded":"2023-11-24T08:12:06.053320+0000","last_fullsized":"2023-11-24T08:12:06.053320+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:30:26.489507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544651+0000","last_change":"2023-11-24T08:12:04.704302+0000","last_active":"2023-11-24T08:12:06.544651+0000","last_peered":"2023-11-24T08:12:06.544651+0000","last_clean":"2023-11-24T08:12:06.544651+0000","last_became_active":"2023-11-24T08:12:04.703850+0000","last_became_peered":"2023-11-24T08:12:04.703850+0000","last_unstale":"2023-11-24T08:12:06.544651+0000","last_undegraded":"2023-11-24T08:12:06.544651+0000","last_fullsized":"2023-11-24T08:12:06.544651+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:41:05.392957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.135904+0000","last_change":"2023-11-24T08:12:04.712768+0000","last_active":"2023-11-24T08:12:06.135904+0000","last_peered":"2023-11-24T08:12:06.135904+0000","last_clean":"2023-11-24T08:12:06.135904+0000","last_became_active":"2023-11-24T08:12:04.712404+0000","last_became_peered":"2023-11-24T08:12:04.712404+0000","last_unstale":"2023-11-24T08:12:06.135904+0000","last_undegraded":"2023-11-24T08:12:06.135904+0000","last_fullsized":"2023-11-24T08:12:06.135904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:02:54.298540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.701291+0000","last_change":"2023-11-24T08:12:04.707732+0000","last_active":"2023-11-24T08:12:05.701291+0000","last_peered":"2023-11-24T08:12:05.701291+0000","last_clean":"2023-11-24T08:12:05.701291+0000","last_became_active":"2023-11-24T08:12:04.707374+0000","last_became_peered":"2023-11-24T08:12:04.707374+0000","last_unstale":"2023-11-24T08:12:05.701291+0000","last_undegraded":"2023-11-24T08:12:05.701291+0000","last_fullsized":"2023-11-24T08:12:05.701291+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:10.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.543141+0000","last_change":"2023-11-24T08:12:04.713633+0000","last_active":"2023-11-24T08:12:06.543141+0000","last_peered":"2023-11-24T08:12:06.543141+0000","last_clean":"2023-11-24T08:12:06.543141+0000","last_became_active":"2023-11-24T08:12:04.713197+0000","last_became_peered":"2023-11-24T08:12:04.713197+0000","last_unstale":"2023-11-24T08:12:06.543141+0000","last_undegraded":"2023-11-24T08:12:06.543141+0000","last_fullsized":"2023-11-24T08:12:06.543141+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:13:58.887224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.700227+0000","last_change":"2023-11-24T08:12:04.714706+0000","last_active":"2023-11-24T08:12:05.700227+0000","last_peered":"2023-11-24T08:12:05.700227+0000","last_clean":"2023-11-24T08:12:05.700227+0000","last_became_active":"2023-11-24T08:12:04.714427+0000","last_became_peered":"2023-11-24T08:12:04.714427+0000","last_unstale":"2023-11-24T08:12:05.700227+0000","last_undegraded":"2023-11-24T08:12:05.700227+0000","last_fullsized":"2023-11-24T08:12:05.700227+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:59.757400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703873+0000","last_change":"2023-11-24T08:12:04.712240+0000","last_active":"2023-11-24T08:12:05.703873+0000","last_peered":"2023-11-24T08:12:05.703873+0000","last_clean":"2023-11-24T08:12:05.703873+0000","last_became_active":"2023-11-24T08:12:04.711940+0000","last_became_peered":"2023-11-24T08:12:04.711940+0000","last_unstale":"2023-11-24T08:12:05.703873+0000","last_undegraded":"2023-11-24T08:12:05.703873+0000","last_fullsized":"2023-11-24T08:12:05.703873+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:46.877093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703066+0000","last_change":"2023-11-24T08:12:04.713795+0000","last_active":"2023-11-24T08:12:05.703066+0000","last_peered":"2023-11-24T08:12:05.703066+0000","last_clean":"2023-11-24T08:12:05.703066+0000","last_became_active":"2023-11-24T08:12:04.713467+0000","last_became_peered":"2023-11-24T08:12:04.713467+0000","last_unstale":"2023-11-24T08:12:05.703066+0000","last_undegraded":"2023-11-24T08:12:05.703066+0000","last_fullsized":"2023-11-24T08:12:05.703066+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:55.296434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548337+0000","last_change":"2023-11-24T08:12:04.712342+0000","last_active":"2023-11-24T08:12:06.548337+0000","last_peered":"2023-11-24T08:12:06.548337+0000","last_clean":"2023-11-24T08:12:06.548337+0000","last_became_active":"2023-11-24T08:12:04.712016+0000","last_became_peered":"2023-11-24T08:12:04.712016+0000","last_unstale":"2023-11-24T08:12:06.548337+0000","last_undegraded":"2023-11-24T08:12:06.548337+0000","last_fullsized":"2023-11-24T08:12:06.548337+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:14:25.706147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.702336+0000","last_change":"2023-11-24T08:12:04.711891+0000","last_active":"2023-11-24T08:12:05.702336+0000","last_peered":"2023-11-24T08:12:05.702336+0000","last_clean":"2023-11-24T08:12:05.702336+0000","last_became_active":"2023-11-24T08:12:04.710798+0000","last_became_peered":"2023-11-24T08:12:04.710798+0000","last_unstale":"2023-11-24T08:12:05.702336+0000","last_undegraded":"2023-11-24T08:12:05.702336+0000","last_fullsized":"2023-11-24T08:12:05.702336+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:20.067454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.052778+0000","last_change":"2023-11-24T08:12:04.706559+0000","last_active":"2023-11-24T08:12:06.052778+0000","last_peered":"2023-11-24T08:12:06.052778+0000","last_clean":"2023-11-24T08:12:06.052778+0000","last_became_active":"2023-11-24T08:12:04.706233+0000","last_became_peered":"2023-11-24T08:12:04.706233+0000","last_unstale":"2023-11-24T08:12:06.052778+0000","last_undegraded":"2023-11-24T08:12:06.052778+0000","last_fullsized":"2023-11-24T08:12:06.052778+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:42.075184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544364+0000","last_change":"2023-11-24T08:12:04.713857+0000","last_active":"2023-11-24T08:12:06.544364+0000","last_peered":"2023-11-24T08:12:06.544364+0000","last_clean":"2023-11-24T08:12:06.544364+0000","last_became_active":"2023-11-24T08:12:04.713558+0000","last_became_peered":"2023-11-24T08:12:04.713558+0000","last_unstale":"2023-11-24T08:12:06.544364+0000","last_undegraded":"2023-11-24T08:12:06.544364+0000","last_fullsized":"2023-11-24T08:12:06.544364+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:53:22.727005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.542841+0000","last_change":"2023-11-24T08:12:04.705515+0000","last_active":"2023-11-24T08:12:06.542841+0000","last_peered":"2023-11-24T08:12:06.542841+0000","last_clean":"2023-11-24T08:12:06.542841+0000","last_became_active":"2023-11-24T08:12:04.702759+0000","last_became_peered":"2023-11-24T08:12:04.702759+0000","last_unstale":"2023-11-24T08:12:06.542841+0000","last_undegraded":"2023-11-24T08:12:06.542841+0000","last_fullsized":"2023-11-24T08:12:06.542841+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:19:27.541302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.135540+0000","last_change":"2023-11-24T08:12:04.709939+0000","last_active":"2023-11-24T08:12:06.135540+0000","last_peered":"2023-11-24T08:12:06.135540+0000","last_clean":"2023-11-24T08:12:06.135540+0000","last_became_active":"2023-11-24T08:12:04.708954+0000","last_became_peered":"2023-11-24T08:12:04.708954+0000","last_unstale":"2023-11-24T08:12:06.135540+0000","last_undegraded":"2023-11-24T08:12:06.135540+0000","last_fullsized":"2023-11-24T08:12:06.135540+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:25:00.439880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136500+0000","last_change":"2023-11-24T08:12:04.710335+0000","last_active":"2023-11-24T08:12:06.136500+0000","last_peered":"2023-11-24T08:12:06.136500+0000","last_clean":"2023-11-24T08:12:06.136500+0000","last_became_active":"2023-11-24T08:12:04.707914+0000","last_became_peered":"2023-11-24T08:12:04.707914+0000","last_unstale":"2023-11-24T08:12:06.136500+0000","last_undegraded":"2023-11-24T08:12:06.136500+0000","last_fullsized":"2023-11-24T08:12:06.136500+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:10.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544781+0000","last_change":"2023-11-24T08:12:04.715693+0000","last_active":"2023-11-24T08:12:06.544781+0000","last_peered":"2023-11-24T08:12:06.544781+0000","last_clean":"2023-11-24T08:12:06.544781+0000","last_became_active":"2023-11-24T08:12:04.715361+0000","last_became_peered":"2023-11-24T08:12:04.715361+0000","last_unstale":"2023-11-24T08:12:06.544781+0000","last_undegraded":"2023-11-24T08:12:06.544781+0000","last_fullsized":"2023-11-24T08:12:06.544781+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:19.569216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.545826+0000","last_change":"2023-11-24T08:12:04.703692+0000","last_active":"2023-11-24T08:12:06.545826+0000","last_peered":"2023-11-24T08:12:06.545826+0000","last_clean":"2023-11-24T08:12:06.545826+0000","last_became_active":"2023-11-24T08:12:04.703424+0000","last_became_peered":"2023-11-24T08:12:04.703424+0000","last_unstale":"2023-11-24T08:12:06.545826+0000","last_undegraded":"2023-11-24T08:12:06.545826+0000","last_fullsized":"2023-11-24T08:12:06.545826+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:01:26.067951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.545018+0000","last_change":"2023-11-24T08:12:04.704108+0000","last_active":"2023-11-24T08:12:06.545018+0000","last_peered":"2023-11-24T08:12:06.545018+0000","last_clean":"2023-11-24T08:12:06.545018+0000","last_became_active":"2023-11-24T08:12:04.703549+0000","last_became_peered":"2023-11-24T08:12:04.703549+0000","last_unstale":"2023-11-24T08:12:06.545018+0000","last_undegraded":"2023-11-24T08:12:06.545018+0000","last_fullsized":"2023-11-24T08:12:06.545018+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:21.982900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.545580+0000","last_change":"2023-11-24T08:12:04.701691+0000","last_active":"2023-11-24T08:12:06.545580+0000","last_peered":"2023-11-24T08:12:06.545580+0000","last_clean":"2023-11-24T08:12:06.545580+0000","last_became_active":"2023-11-24T08:12:04.701273+0000","last_became_peered":"2023-11-24T08:12:04.701273+0000","last_unstale":"2023-11-24T08:12:06.545580+0000","last_undegraded":"2023-11-24T08:12:06.545580+0000","last_fullsized":"2023-11-24T08:12:06.545580+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:02.982242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.546512+0000","last_change":"2023-11-24T08:12:04.713129+0000","last_active":"2023-11-24T08:12:06.546512+0000","last_peered":"2023-11-24T08:12:06.546512+0000","last_clean":"2023-11-24T08:12:06.546512+0000","last_became_active":"2023-11-24T08:12:04.712822+0000","last_became_peered":"2023-11-24T08:12:04.712822+0000","last_unstale":"2023-11-24T08:12:06.546512+0000","last_undegraded":"2023-11-24T08:12:06.546512+0000","last_fullsized":"2023-11-24T08:12:06.546512+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:38.893704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.702257+0000","last_change":"2023-11-24T08:12:04.707835+0000","last_active":"2023-11-24T08:12:05.702257+0000","last_peered":"2023-11-24T08:12:05.702257+0000","last_clean":"2023-11-24T08:12:05.702257+0000","last_became_active":"2023-11-24T08:12:04.707289+0000","last_became_peered":"2023-11-24T08:12:04.707289+0000","last_unstale":"2023-11-24T08:12:05.702257+0000","last_undegraded":"2023-11-24T08:12:05.702257+0000","last_fullsized":"2023-11-24T08:12:05.702257+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:59.757400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.542735+0000","last_change":"2023-11-24T08:12:04.709722+0000","last_active":"2023-11-24T08:12:06.542735+0000","last_peered":"2023-11-24T08:12:06.542735+0000","last_clean":"2023-11-24T08:12:06.542735+0000","last_became_active":"2023-11-24T08:12:04.708951+0000","last_became_peered":"2023-11-24T08:12:04.708951+0000","last_unstale":"2023-11-24T08:12:06.542735+0000","last_undegraded":"2023-11-24T08:12:06.542735+0000","last_fullsized":"2023-11-24T08:12:06.542735+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:19.988230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548456+0000","last_change":"2023-11-24T08:12:04.726755+0000","last_active":"2023-11-24T08:12:06.548456+0000","last_peered":"2023-11-24T08:12:06.548456+0000","last_clean":"2023-11-24T08:12:06.548456+0000","last_became_active":"2023-11-24T08:12:04.725174+0000","last_became_peered":"2023-11-24T08:12:04.725174+0000","last_unstale":"2023-11-24T08:12:06.548456+0000","last_undegraded":"2023-11-24T08:12:06.548456+0000","last_fullsized":"2023-11-24T08:12:06.548456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:08.096638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.053086+0000","last_change":"2023-11-24T08:12:04.715587+0000","last_active":"2023-11-24T08:12:06.053086+0000","last_peered":"2023-11-24T08:12:06.053086+0000","last_clean":"2023-11-24T08:12:06.053086+0000","last_became_active":"2023-11-24T08:12:04.713824+0000","last_became_peered":"2023-11-24T08:12:04.713824+0000","last_unstale":"2023-11-24T08:12:06.053086+0000","last_undegraded":"2023-11-24T08:12:06.053086+0000","last_fullsized":"2023-11-24T08:12:06.053086+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:37:14.698597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136251+0000","last_change":"2023-11-24T08:12:04.701746+0000","last_active":"2023-11-24T08:12:06.136251+0000","last_peered":"2023-11-24T08:12:06.136251+0000","last_clean":"2023-11-24T08:12:06.136251+0000","last_became_active":"2023-11-24T08:12:04.701118+0000","last_became_peered":"2023-11-24T08:12:04.701118+0000","last_unstale":"2023-11-24T08:12:06.136251+0000","last_undegraded":"2023-11-24T08:12:06.136251+0000","last_fullsized":"2023-11-24T08:12:06.136251+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:59.757400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.702704+0000","last_change":"2023-11-24T08:12:04.712599+0000","last_active":"2023-11-24T08:12:05.702704+0000","last_peered":"2023-11-24T08:12:05.702704+0000","last_clean":"2023-11-24T08:12:05.702704+0000","last_became_active":"2023-11-24T08:12:04.712299+0000","last_became_peered":"2023-11-24T08:12:04.712299+0000","last_unstale":"2023-11-24T08:12:05.702704+0000","last_undegraded":"2023-11-24T08:12:05.702704+0000","last_fullsized":"2023-11-24T08:12:05.702704+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:31:38.212627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548037+0000","last_change":"2023-11-24T08:12:04.709586+0000","last_active":"2023-11-24T08:12:06.548037+0000","last_peered":"2023-11-24T08:12:06.548037+0000","last_clean":"2023-11-24T08:12:06.548037+0000","last_became_active":"2023-11-24T08:12:04.709297+0000","last_became_peered":"2023-11-24T08:12:04.709297+0000","last_unstale":"2023-11-24T08:12:06.548037+0000","last_undegraded":"2023-11-24T08:12:06.548037+0000","last_fullsized":"2023-11-24T08:12:06.548037+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:42:14.937007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.546831+0000","last_change":"2023-11-24T08:12:04.722671+0000","last_active":"2023-11-24T08:12:06.546831+0000","last_peered":"2023-11-24T08:12:06.546831+0000","last_clean":"2023-11-24T08:12:06.546831+0000","last_became_active":"2023-11-24T08:12:04.722413+0000","last_became_peered":"2023-11-24T08:12:04.722413+0000","last_unstale":"2023-11-24T08:12:06.546831+0000","last_undegraded":"2023-11-24T08:12:06.546831+0000","last_fullsized":"2023-11-24T08:12:06.546831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:34.941802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.135758+0000","last_change":"2023-11-24T08:12:04.700307+0000","last_active":"2023-11-24T08:12:06.135758+0000","last_peered":"2023-11-24T08:12:06.135758+0000","last_clean":"2023-11-24T08:12:06.135758+0000","last_became_active":"2023-11-24T08:12:04.699753+0000","last_became_peered":"2023-11-24T08:12:04.699753+0000","last_unstale":"2023-11-24T08:12:06.135758+0000","last_undegraded":"2023-11-24T08:12:06.135758+0000","last_fullsized":"2023-11-24T08:12:06.135758+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:07.342366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.701424+0000","last_change":"2023-11-24T08:12:04.715001+0000","last_active":"2023-11-24T08:12:05.701424+0000","last_peered":"2023-11-24T08:12:05.701424+0000","last_clean":"2023-11-24T08:12:05.701424+0000","last_became_active":"2023-11-24T08:12:04.714735+0000","last_became_peered":"2023-11-24T08:12:04.714735+0000","last_unstale":"2023-11-24T08:12:05.701424+0000","last_undegraded":"2023-11-24T08:12:05.701424+0000","last_fullsized":"2023-11-24T08:12:05.701424+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:07.342366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136432+0000","last_change":"2023-11-24T08:12:04.713060+0000","last_active":"2023-11-24T08:12:06.136432+0000","last_peered":"2023-11-24T08:12:06.136432+0000","last_clean":"2023-11-24T08:12:06.136432+0000","last_became_active":"2023-11-24T08:12:04.711362+0000","last_became_peered":"2023-11-24T08:12:04.711362+0000","last_unstale":"2023-11-24T08:12:06.136432+0000","last_undegraded":"2023-11-24T08:12:06.136432+0000","last_fullsized":"2023-11-24T08:12:06.136432+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:31:38.212627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.729485+0000","last_change":"2023-11-24T08:12:03.718143+0000","last_active":"2023-11-24T08:12:04.729485+0000","last_peered":"2023-11-24T08:12:04.729485+0000","last_clean":"2023-11-24T08:12:04.729485+0000","last_became_active":"2023-11-24T08:12:03.717487+0000","last_became_peered":"2023-11-24T08:12:03.717487+0000","last_unstale":"2023-11-24T08:12:04.729485+0000","last_undegraded":"2023-11-24T08:12:04.729485+0000","last_fullsized":"2023-11-24T08:12:04.729485+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:15:31.359238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703391+0000","last_change":"2023-11-24T08:12:04.717673+0000","last_active":"2023-11-24T08:12:05.703391+0000","last_peered":"2023-11-24T08:12:05.703391+0000","last_clean":"2023-11-24T08:12:05.703391+0000","last_became_active":"2023-11-24T08:12:04.717389+0000","last_became_peered":"2023-11-24T08:12:04.717389+0000","last_unstale":"2023-11-24T08:12:05.703391+0000","last_undegraded":"2023-11-24T08:12:05.703391+0000","last_fullsized":"2023-11-24T08:12:05.703391+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:46.877093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.730235+0000","last_change":"2023-11-24T08:12:03.707017+0000","last_active":"2023-11-24T08:12:04.730235+0000","last_peered":"2023-11-24T08:12:04.730235+0000","last_clean":"2023-11-24T08:12:04.730235+0000","last_became_active":"2023-11-24T08:12:03.706726+0000","last_became_peered":"2023-11-24T08:12:03.706726+0000","last_unstale":"2023-11-24T08:12:04.730235+0000","last_undegraded":"2023-11-24T08:12:04.730235+0000","last_fullsized":"2023-11-24T08:12:04.730235+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:50.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703030+0000","last_change":"2023-11-24T08:12:04.715637+0000","last_active":"2023-11-24T08:12:05.703030+0000","last_peered":"2023-11-24T08:12:05.703030+0000","last_clean":"2023-11-24T08:12:05.703030+0000","last_became_active":"2023-11-24T08:12:04.714599+0000","last_became_peered":"2023-11-24T08:12:04.714599+0000","last_unstale":"2023-11-24T08:12:05.703030+0000","last_undegraded":"2023-11-24T08:12:05.703030+0000","last_fullsized":"2023-11-24T08:12:05.703030+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:10.819773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.732940+0000","last_change":"2023-11-24T08:12:03.713322+0000","last_active":"2023-11-24T08:12:04.732940+0000","last_peered":"2023-11-24T08:12:04.732940+0000","last_clean":"2023-11-24T08:12:04.732940+0000","last_became_active":"2023-11-24T08:12:03.712992+0000","last_became_peered":"2023-11-24T08:12:03.712992+0000","last_unstale":"2023-11-24T08:12:04.732940+0000","last_undegraded":"2023-11-24T08:12:04.732940+0000","last_fullsized":"2023-11-24T08:12:04.732940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:54.280102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.053403+0000","last_change":"2023-11-24T08:12:04.715934+0000","last_active":"2023-11-24T08:12:06.053403+0000","last_peered":"2023-11-24T08:12:06.053403+0000","last_clean":"2023-11-24T08:12:06.053403+0000","last_became_active":"2023-11-24T08:12:04.714638+0000","last_became_peered":"2023-11-24T08:12:04.714638+0000","last_unstale":"2023-11-24T08:12:06.053403+0000","last_undegraded":"2023-11-24T08:12:06.053403+0000","last_fullsized":"2023-11-24T08:12:06.053403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:23.173372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.732838+0000","last_change":"2023-11-24T08:12:03.717727+0000","last_active":"2023-11-24T08:12:04.732838+0000","last_peered":"2023-11-24T08:12:04.732838+0000","last_clean":"2023-11-24T08:12:04.732838+0000","last_became_active":"2023-11-24T08:12:03.717310+0000","last_became_peered":"2023-11-24T08:12:03.717310+0000","last_unstale":"2023-11-24T08:12:04.732838+0000","last_undegraded":"2023-11-24T08:12:04.732838+0000","last_fullsized":"2023-11-24T08:12:04.732838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:58.741067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.547145+0000","last_change":"2023-11-24T08:12:04.719733+0000","last_active":"2023-11-24T08:12:06.547145+0000","last_peered":"2023-11-24T08:12:06.547145+0000","last_clean":"2023-11-24T08:12:06.547145+0000","last_became_active":"2023-11-24T08:12:04.717691+0000","last_became_peered":"2023-11-24T08:12:04.717691+0000","last_unstale":"2023-11-24T08:12:06.547145+0000","last_undegraded":"2023-11-24T08:12:06.547145+0000","last_fullsized":"2023-11-24T08:12:06.547145+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:39.722043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.732252+0000","last_change":"2023-11-24T08:12:03.714063+0000","last_active":"2023-11-24T08:12:04.732252+0000","last_peered":"2023-11-24T08:12:04.732252+0000","last_clean":"2023-11-24T08:12:04.732252+0000","last_became_active":"2023-11-24T08:12:03.713622+0000","last_became_peered":"2023-11-24T08:12:03.713622+0000","last_unstale":"2023-11-24T08:12:04.732252+0000","last_undegraded":"2023-11-24T08:12:04.732252+0000","last_fullsized":"2023-11-24T08:12:04.732252+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:25.242371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136365+0000","last_change":"2023-11-24T08:12:04.706144+0000","last_active":"2023-11-24T08:12:06.136365+0000","last_peered":"2023-11-24T08:12:06.136365+0000","last_clean":"2023-11-24T08:12:06.136365+0000","last_became_active":"2023-11-24T08:12:04.705512+0000","last_became_peered":"2023-11-24T08:12:04.705512+0000","last_unstale":"2023-11-24T08:12:06.136365+0000","last_undegraded":"2023-11-24T08:12:06.136365+0000","last_fullsized":"2023-11-24T08:12:06.136365+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:20.067454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.729591+0000","last_change":"2023-11-24T08:12:03.709262+0000","last_active":"2023-11-24T08:12:04.729591+0000","last_peered":"2023-11-24T08:12:04.729591+0000","last_clean":"2023-11-24T08:12:04.729591+0000","last_became_active":"2023-11-24T08:12:03.708121+0000","last_became_peered":"2023-11-24T08:12:03.708121+0000","last_unstale":"2023-11-24T08:12:04.729591+0000","last_undegraded":"2023-11-24T08:12:04.729591+0000","last_fullsized":"2023-11-24T08:12:04.729591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:04.608770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.052803+0000","last_change":"2023-11-24T08:12:04.709586+0000","last_active":"2023-11-24T08:12:06.052803+0000","last_peered":"2023-11-24T08:12:06.052803+0000","last_clean":"2023-11-24T08:12:06.052803+0000","last_became_active":"2023-11-24T08:12:04.709264+0000","last_became_peered":"2023-11-24T08:12:04.709264+0000","last_unstale":"2023-11-24T08:12:06.052803+0000","last_undegraded":"2023-11-24T08:12:06.052803+0000","last_fullsized":"2023-11-24T08:12:06.052803+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:47:08.564872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.678272+0000","last_change":"2023-11-24T08:12:03.717095+0000","last_active":"2023-11-24T08:12:04.678272+0000","last_peered":"2023-11-24T08:12:04.678272+0000","last_clean":"2023-11-24T08:12:04.678272+0000","last_became_active":"2023-11-24T08:12:03.716795+0000","last_became_peered":"2023-11-24T08:12:03.716795+0000","last_unstale":"2023-11-24T08:12:04.678272+0000","last_undegraded":"2023-11-24T08:12:04.678272+0000","last_fullsized":"2023-11-24T08:12:04.678272+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:02:53.282207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.543176+0000","last_change":"2023-11-24T08:12:04.705129+0000","last_active":"2023-11-24T08:12:06.543176+0000","last_peered":"2023-11-24T08:12:06.543176+0000","last_clean":"2023-11-24T08:12:06.543176+0000","last_became_active":"2023-11-24T08:12:04.704861+0000","last_became_peered":"2023-11-24T08:12:04.704861+0000","last_unstale":"2023-11-24T08:12:06.543176+0000","last_undegraded":"2023-11-24T08:12:06.543176+0000","last_fullsized":"2023-11-24T08:12:06.543176+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:13.247336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.707606+0000","last_change":"2023-11-24T08:12:03.712202+0000","last_active":"2023-11-24T08:12:04.707606+0000","last_peered":"2023-11-24T08:12:04.707606+0000","last_clean":"2023-11-24T08:12:04.707606+0000","last_became_active":"2023-11-24T08:12:03.711913+0000","last_became_peered":"2023-11-24T08:12:03.711913+0000","last_unstale":"2023-11-24T08:12:04.707606+0000","last_undegraded":"2023-11-24T08:12:04.707606+0000","last_fullsized":"2023-11-24T08:12:04.707606+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:41.607047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.547466+0000","last_change":"2023-11-24T08:12:04.714080+0000","last_active":"2023-11-24T08:12:06.547466+0000","last_peered":"2023-11-24T08:12:06.547466+0000","last_clean":"2023-11-24T08:12:06.547466+0000","last_became_active":"2023-11-24T08:12:04.713747+0000","last_became_peered":"2023-11-24T08:12:04.713747+0000","last_unstale":"2023-11-24T08:12:06.547466+0000","last_undegraded":"2023-11-24T08:12:06.547466+0000","last_fullsized":"2023-11-24T08:12:06.547466+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:42:05.705736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.709481+0000","last_change":"2023-11-24T08:12:03.717367+0000","last_active":"2023-11-24T08:12:04.709481+0000","last_peered":"2023-11-24T08:12:04.709481+0000","last_clean":"2023-11-24T08:12:04.709481+0000","last_became_active":"2023-11-24T08:12:03.716985+0000","last_became_peered":"2023-11-24T08:12:03.716985+0000","last_unstale":"2023-11-24T08:12:04.709481+0000","last_undegraded":"2023-11-24T08:12:04.709481+0000","last_fullsized":"2023-11-24T08:12:04.709481+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:33.235756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703638+0000","last_change":"2023-11-24T08:12:04.708268+0000","last_active":"2023-11-24T08:12:05.703638+0000","last_peered":"2023-11-24T08:12:05.703638+0000","last_clean":"2023-11-24T08:12:05.703638+0000","last_became_active":"2023-11-24T08:12:04.708002+0000","last_became_peered":"2023-11-24T08:12:04.708002+0000","last_unstale":"2023-11-24T08:12:05.703638+0000","last_undegraded":"2023-11-24T08:12:05.703638+0000","last_fullsized":"2023-11-24T08:12:05.703638+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:09.489727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.708923+0000","last_change":"2023-11-24T08:12:03.717295+0000","last_active":"2023-11-24T08:12:04.708923+0000","last_peered":"2023-11-24T08:12:04.708923+0000","last_clean":"2023-11-24T08:12:04.708923+0000","last_became_active":"2023-11-24T08:12:03.716167+0000","last_became_peered":"2023-11-24T08:12:03.716167+0000","last_unstale":"2023-11-24T08:12:04.708923+0000","last_undegraded":"2023-11-24T08:12:04.708923+0000","last_fullsized":"2023-11-24T08:12:04.708923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:02:08.782115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544612+0000","last_change":"2023-11-24T08:12:04.708139+0000","last_active":"2023-11-24T08:12:06.544612+0000","last_peered":"2023-11-24T08:12:06.544612+0000","last_clean":"2023-11-24T08:12:06.544612+0000","last_became_active":"2023-11-24T08:12:04.707885+0000","last_became_peered":"2023-11-24T08:12:04.707885+0000","last_unstale":"2023-11-24T08:12:06.544612+0000","last_undegraded":"2023-11-24T08:12:06.544612+0000","last_fullsized":"2023-11-24T08:12:06.544612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:58.685286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.731124+0000","last_change":"2023-11-24T08:12:03.717070+0000","last_active":"2023-11-24T08:12:04.731124+0000","last_peered":"2023-11-24T08:12:04.731124+0000","last_clean":"2023-11-24T08:12:04.731124+0000","last_became_active":"2023-11-24T08:12:03.716609+0000","last_became_peered":"2023-11-24T08:12:03.716609+0000","last_unstale":"2023-11-24T08:12:04.731124+0000","last_undegraded":"2023-11-24T08:12:04.731124+0000","last_fullsized":"2023-11-24T08:12:04.731124+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:20.464871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136646+0000","last_change":"2023-11-24T08:12:04.708216+0000","last_active":"2023-11-24T08:12:06.136646+0000","last_peered":"2023-11-24T08:12:06.136646+0000","last_clean":"2023-11-24T08:12:06.136646+0000","last_became_active":"2023-11-24T08:12:04.707931+0000","last_became_peered":"2023-11-24T08:12:04.707931+0000","last_unstale":"2023-11-24T08:12:06.136646+0000","last_undegraded":"2023-11-24T08:12:06.136646+0000","last_fullsized":"2023-11-24T08:12:06.136646+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:09.489727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.734607+0000","last_change":"2023-11-24T08:12:03.719347+0000","last_active":"2023-11-24T08:12:04.734607+0000","last_peered":"2023-11-24T08:12:04.734607+0000","last_clean":"2023-11-24T08:12:04.734607+0000","last_became_active":"2023-11-24T08:12:03.718422+0000","last_became_peered":"2023-11-24T08:12:03.718422+0000","last_unstale":"2023-11-24T08:12:04.734607+0000","last_undegraded":"2023-11-24T08:12:04.734607+0000","last_fullsized":"2023-11-24T08:12:04.734607+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:02:53.282207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544125+0000","last_change":"2023-11-24T08:12:04.697452+0000","last_active":"2023-11-24T08:12:06.544125+0000","last_peered":"2023-11-24T08:12:06.544125+0000","last_clean":"2023-11-24T08:12:06.544125+0000","last_became_active":"2023-11-24T08:12:04.697142+0000","last_became_peered":"2023-11-24T08:12:04.697142+0000","last_unstale":"2023-11-24T08:12:06.544125+0000","last_undegraded":"2023-11-24T08:12:06.544125+0000","last_fullsized":"2023-11-24T08:12:06.544125+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:08.918557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.733482+0000","last_change":"2023-11-24T08:12:03.713899+0000","last_active":"2023-11-24T08:12:04.733482+0000","last_peered":"2023-11-24T08:12:04.733482+0000","last_clean":"2023-11-24T08:12:04.733482+0000","last_became_active":"2023-11-24T08:12:03.713529+0000","last_became_peered":"2023-11-24T08:12:03.713529+0000","last_unstale":"2023-11-24T08:12:04.733482+0000","last_undegraded":"2023-11-24T08:12:04.733482+0000","last_fullsized":"2023-11-24T08:12:04.733482+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:20.966568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.052974+0000","last_change":"2023-11-24T08:12:04.709161+0000","last_active":"2023-11-24T08:12:06.052974+0000","last_peered":"2023-11-24T08:12:06.052974+0000","last_clean":"2023-11-24T08:12:06.052974+0000","last_became_active":"2023-11-24T08:12:04.708600+0000","last_became_peered":"2023-11-24T08:12:04.708600+0000","last_unstale":"2023-11-24T08:12:06.052974+0000","last_undegraded":"2023-11-24T08:12:06.052974+0000","last_fullsized":"2023-11-24T08:12:06.052974+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:57:03.895506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.690324+0000","last_change":"2023-11-24T08:12:03.713813+0000","last_active":"2023-11-24T08:12:04.690324+0000","last_peered":"2023-11-24T08:12:04.690324+0000","last_clean":"2023-11-24T08:12:04.690324+0000","last_became_active":"2023-11-24T08:12:03.713539+0000","last_became_peered":"2023-11-24T08:12:03.713539+0000","last_unstale":"2023-11-24T08:12:04.690324+0000","last_undegraded":"2023-11-24T08:12:04.690324+0000","last_fullsized":"2023-11-24T08:12:04.690324+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:13:21.642354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544388+0000","last_change":"2023-11-24T08:12:04.721036+0000","last_active":"2023-11-24T08:12:06.544388+0000","last_peered":"2023-11-24T08:12:06.544388+0000","last_clean":"2023-11-24T08:12:06.544388+0000","last_became_active":"2023-11-24T08:12:04.720515+0000","last_became_peered":"2023-11-24T08:12:04.720515+0000","last_unstale":"2023-11-24T08:12:06.544388+0000","last_undegraded":"2023-11-24T08:12:06.544388+0000","last_fullsized":"2023-11-24T08:12:06.544388+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:47.578791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.695008+0000","last_change":"2023-11-24T08:12:03.716116+0000","last_active":"2023-11-24T08:12:04.695008+0000","last_peered":"2023-11-24T08:12:04.695008+0000","last_clean":"2023-11-24T08:12:04.695008+0000","last_became_active":"2023-11-24T08:12:03.715257+0000","last_became_peered":"2023-11-24T08:12:03.715257+0000","last_unstale":"2023-11-24T08:12:04.695008+0000","last_undegraded":"2023-11-24T08:12:04.695008+0000","last_fullsized":"2023-11-24T08:12:04.695008+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:24:59.423548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.545199+0000","last_change":"2023-11-24T08:12:04.721460+0000","last_active":"2023-11-24T08:12:06.545199+0000","last_peered":"2023-11-24T08:12:06.545199+0000","last_clean":"2023-11-24T08:12:06.545199+0000","last_became_active":"2023-11-24T08:12:04.721163+0000","last_became_peered":"2023-11-24T08:12:04.721163+0000","last_unstale":"2023-11-24T08:12:06.545199+0000","last_undegraded":"2023-11-24T08:12:06.545199+0000","last_fullsized":"2023-11-24T08:12:06.545199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:08:14.727056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.734805+0000","last_change":"2023-11-24T08:12:03.714210+0000","last_active":"2023-11-24T08:12:04.734805+0000","last_peered":"2023-11-24T08:12:04.734805+0000","last_clean":"2023-11-24T08:12:04.734805+0000","last_became_active":"2023-11-24T08:12:03.713954+0000","last_became_peered":"2023-11-24T08:12:03.713954+0000","last_unstale":"2023-11-24T08:12:04.734805+0000","last_undegraded":"2023-11-24T08:12:04.734805+0000","last_fullsized":"2023-11-24T08:12:04.734805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:06.326033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.053410+0000","last_change":"2023-11-24T08:12:04.711431+0000","last_active":"2023-11-24T08:12:06.053410+0000","last_peered":"2023-11-24T08:12:06.053410+0000","last_clean":"2023-11-24T08:12:06.053410+0000","last_became_active":"2023-11-24T08:12:04.710229+0000","last_became_peered":"2023-11-24T08:12:04.710229+0000","last_unstale":"2023-11-24T08:12:06.053410+0000","last_undegraded":"2023-11-24T08:12:06.053410+0000","last_fullsized":"2023-11-24T08:12:06.053410+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:44.306008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.724689+0000","last_change":"2023-11-24T08:12:03.718342+0000","last_active":"2023-11-24T08:12:04.724689+0000","last_peered":"2023-11-24T08:12:04.724689+0000","last_clean":"2023-11-24T08:12:04.724689+0000","last_became_active":"2023-11-24T08:12:03.718045+0000","last_became_peered":"2023-11-24T08:12:03.718045+0000","last_unstale":"2023-11-24T08:12:04.724689+0000","last_undegraded":"2023-11-24T08:12:04.724689+0000","last_fullsized":"2023-11-24T08:12:04.724689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:50.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703485+0000","last_change":"2023-11-24T08:12:04.713817+0000","last_active":"2023-11-24T08:12:05.703485+0000","last_peered":"2023-11-24T08:12:05.703485+0000","last_clean":"2023-11-24T08:12:05.703485+0000","last_became_active":"2023-11-24T08:12:04.713475+0000","last_became_peered":"2023-11-24T08:12:04.713475+0000","last_unstale":"2023-11-24T08:12:05.703485+0000","last_undegraded":"2023-11-24T08:12:05.703485+0000","last_fullsized":"2023-11-24T08:12:05.703485+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:20.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.731951+0000","last_change":"2023-11-24T08:12:03.709039+0000","last_active":"2023-11-24T08:12:04.731951+0000","last_peered":"2023-11-24T08:12:04.731951+0000","last_clean":"2023-11-24T08:12:04.731951+0000","last_became_active":"2023-11-24T08:12:03.708766+0000","last_became_peered":"2023-11-24T08:12:03.708766+0000","last_unstale":"2023-11-24T08:12:04.731951+0000","last_undegraded":"2023-11-24T08:12:04.731951+0000","last_fullsized":"2023-11-24T08:12:04.731951+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:19.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136949+0000","last_change":"2023-11-24T08:12:04.715407+0000","last_active":"2023-11-24T08:12:06.136949+0000","last_peered":"2023-11-24T08:12:06.136949+0000","last_clean":"2023-11-24T08:12:06.136949+0000","last_became_active":"2023-11-24T08:12:04.715061+0000","last_became_peered":"2023-11-24T08:12:04.715061+0000","last_unstale":"2023-11-24T08:12:06.136949+0000","last_undegraded":"2023-11-24T08:12:06.136949+0000","last_fullsized":"2023-11-24T08:12:06.136949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:18:54.253351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.678821+0000","last_change":"2023-11-24T08:12:03.718342+0000","last_active":"2023-11-24T08:12:04.678821+0000","last_peered":"2023-11-24T08:12:04.678821+0000","last_clean":"2023-11-24T08:12:04.678821+0000","last_became_active":"2023-11-24T08:12:03.718045+0000","last_became_peered":"2023-11-24T08:12:03.718045+0000","last_unstale":"2023-11-24T08:12:04.678821+0000","last_undegraded":"2023-11-24T08:12:04.678821+0000","last_fullsized":"2023-11-24T08:12:04.678821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:19.051121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.544726+0000","last_change":"2023-11-24T08:12:04.708085+0000","last_active":"2023-11-24T08:12:06.544726+0000","last_peered":"2023-11-24T08:12:06.544726+0000","last_clean":"2023-11-24T08:12:06.544726+0000","last_became_active":"2023-11-24T08:12:04.707774+0000","last_became_peered":"2023-11-24T08:12:04.707774+0000","last_unstale":"2023-11-24T08:12:06.544726+0000","last_undegraded":"2023-11-24T08:12:06.544726+0000","last_fullsized":"2023-11-24T08:12:06.544726+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:42.623379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.731444+0000","last_change":"2023-11-24T08:12:03.704922+0000","last_active":"2023-11-24T08:12:04.731444+0000","last_peered":"2023-11-24T08:12:04.731444+0000","last_clean":"2023-11-24T08:12:04.731444+0000","last_became_active":"2023-11-24T08:12:03.704588+0000","last_became_peered":"2023-11-24T08:12:03.704588+0000","last_unstale":"2023-11-24T08:12:04.731444+0000","last_undegraded":"2023-11-24T08:12:04.731444+0000","last_fullsized":"2023-11-24T08:12:04.731444+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:57:41.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.702024+0000","last_change":"2023-11-24T08:12:04.709746+0000","last_active":"2023-11-24T08:12:05.702024+0000","last_peered":"2023-11-24T08:12:05.702024+0000","last_clean":"2023-11-24T08:12:05.702024+0000","last_became_active":"2023-11-24T08:12:04.709479+0000","last_became_peered":"2023-11-24T08:12:04.709479+0000","last_unstale":"2023-11-24T08:12:05.702024+0000","last_undegraded":"2023-11-24T08:12:05.702024+0000","last_fullsized":"2023-11-24T08:12:05.702024+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:55.296434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.689796+0000","last_change":"2023-11-24T08:12:03.714924+0000","last_active":"2023-11-24T08:12:04.689796+0000","last_peered":"2023-11-24T08:12:04.689796+0000","last_clean":"2023-11-24T08:12:04.689796+0000","last_became_active":"2023-11-24T08:12:03.714663+0000","last_became_peered":"2023-11-24T08:12:03.714663+0000","last_unstale":"2023-11-24T08:12:04.689796+0000","last_undegraded":"2023-11-24T08:12:04.689796+0000","last_fullsized":"2023-11-24T08:12:04.689796+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:01:08.070433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.543064+0000","last_change":"2023-11-24T08:12:04.716996+0000","last_active":"2023-11-24T08:12:06.543064+0000","last_peered":"2023-11-24T08:12:06.543064+0000","last_clean":"2023-11-24T08:12:06.543064+0000","last_became_active":"2023-11-24T08:12:04.716681+0000","last_became_peered":"2023-11-24T08:12:04.716681+0000","last_unstale":"2023-11-24T08:12:06.543064+0000","last_undegraded":"2023-11-24T08:12:06.543064+0000","last_fullsized":"2023-11-24T08:12:06.543064+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:12:28.118364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.691959+0000","last_change":"2023-11-24T08:12:03.713832+0000","last_active":"2023-11-24T08:12:04.691959+0000","last_peered":"2023-11-24T08:12:04.691959+0000","last_clean":"2023-11-24T08:12:04.691959+0000","last_became_active":"2023-11-24T08:12:03.713561+0000","last_became_peered":"2023-11-24T08:12:03.713561+0000","last_unstale":"2023-11-24T08:12:04.691959+0000","last_undegraded":"2023-11-24T08:12:04.691959+0000","last_fullsized":"2023-11-24T08:12:04.691959+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:30:26.977411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.700403+0000","last_change":"2023-11-24T08:12:04.704248+0000","last_active":"2023-11-24T08:12:05.700403+0000","last_peered":"2023-11-24T08:12:05.700403+0000","last_clean":"2023-11-24T08:12:05.700403+0000","last_became_active":"2023-11-24T08:12:04.703916+0000","last_became_peered":"2023-11-24T08:12:04.703916+0000","last_unstale":"2023-11-24T08:12:05.700403+0000","last_undegraded":"2023-11-24T08:12:05.700403+0000","last_fullsized":"2023-11-24T08:12:05.700403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:20.067454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.689682+0000","last_change":"2023-11-24T08:12:03.705669+0000","last_active":"2023-11-24T08:12:04.689682+0000","last_peered":"2023-11-24T08:12:04.689682+0000","last_clean":"2023-11-24T08:12:04.689682+0000","last_became_active":"2023-11-24T08:12:03.704614+0000","last_became_peered":"2023-11-24T08:12:03.704614+0000","last_unstale":"2023-11-24T08:12:04.689682+0000","last_undegraded":"2023-11-24T08:12:04.689682+0000","last_fullsized":"2023-11-24T08:12:04.689682+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:17:30.010603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136523+0000","last_change":"2023-11-24T08:12:04.706518+0000","last_active":"2023-11-24T08:12:06.136523+0000","last_peered":"2023-11-24T08:12:06.136523+0000","last_clean":"2023-11-24T08:12:06.136523+0000","last_became_active":"2023-11-24T08:12:04.706215+0000","last_became_peered":"2023-11-24T08:12:04.706215+0000","last_unstale":"2023-11-24T08:12:06.136523+0000","last_undegraded":"2023-11-24T08:12:06.136523+0000","last_fullsized":"2023-11-24T08:12:06.136523+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:55.296434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.731301+0000","last_change":"2023-11-24T08:12:03.711233+0000","last_active":"2023-11-24T08:12:04.731301+0000","last_peered":"2023-11-24T08:12:04.731301+0000","last_clean":"2023-11-24T08:12:04.731301+0000","last_became_active":"2023-11-24T08:12:03.710906+0000","last_became_peered":"2023-11-24T08:12:03.710906+0000","last_unstale":"2023-11-24T08:12:04.731301+0000","last_undegraded":"2023-11-24T08:12:04.731301+0000","last_fullsized":"2023-11-24T08:12:04.731301+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:18.552883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.053382+0000","last_change":"2023-11-24T08:12:04.717880+0000","last_active":"2023-11-24T08:12:06.053382+0000","last_peered":"2023-11-24T08:12:06.053382+0000","last_clean":"2023-11-24T08:12:06.053382+0000","last_became_active":"2023-11-24T08:12:04.716975+0000","last_became_peered":"2023-11-24T08:12:04.716975+0000","last_unstale":"2023-11-24T08:12:06.053382+0000","last_undegraded":"2023-11-24T08:12:06.053382+0000","last_fullsized":"2023-11-24T08:12:06.053382+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:45.286002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"94'1","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548594+0000","last_change":"2023-11-24T08:12:03.714287+0000","last_active":"2023-11-24T08:12:06.548594+0000","last_peered":"2023-11-24T08:12:06.548594+0000","last_clean":"2023-11-24T08:12:06.548594+0000","last_became_active":"2023-11-24T08:12:03.713122+0000","last_became_peered":"2023-11-24T08:12:03.713122+0000","last_unstale":"2023-11-24T08:12:06.548594+0000","last_undegraded":"2023-11-24T08:12:06.548594+0000","last_fullsized":"2023-11-24T08:12:06.548594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:01.965910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.703382+0000","last_change":"2023-11-24T08:12:04.709100+0000","last_active":"2023-11-24T08:12:05.703382+0000","last_peered":"2023-11-24T08:12:05.703382+0000","last_clean":"2023-11-24T08:12:05.703382+0000","last_became_active":"2023-11-24T08:12:04.708576+0000","last_became_peered":"2023-11-24T08:12:04.708576+0000","last_unstale":"2023-11-24T08:12:05.703382+0000","last_undegraded":"2023-11-24T08:12:05.703382+0000","last_fullsized":"2023-11-24T08:12:05.703382+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:18:54.253351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.709640+0000","last_change":"2023-11-24T08:12:03.713683+0000","last_active":"2023-11-24T08:12:04.709640+0000","last_peered":"2023-11-24T08:12:04.709640+0000","last_clean":"2023-11-24T08:12:04.709640+0000","last_became_active":"2023-11-24T08:12:03.713260+0000","last_became_peered":"2023-11-24T08:12:03.713260+0000","last_unstale":"2023-11-24T08:12:04.709640+0000","last_undegraded":"2023-11-24T08:12:04.709640+0000","last_fullsized":"2023-11-24T08:12:04.709640+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:34.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.136837+0000","last_change":"2023-11-24T08:12:04.701789+0000","last_active":"2023-11-24T08:12:06.136837+0000","last_peered":"2023-11-24T08:12:06.136837+0000","last_clean":"2023-11-24T08:12:06.136837+0000","last_became_active":"2023-11-24T08:12:04.701196+0000","last_became_peered":"2023-11-24T08:12:04.701196+0000","last_unstale":"2023-11-24T08:12:06.136837+0000","last_undegraded":"2023-11-24T08:12:06.136837+0000","last_fullsized":"2023-11-24T08:12:06.136837+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:06:46.877093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.708956+0000","last_change":"2023-11-24T08:12:03.712142+0000","last_active":"2023-11-24T08:12:04.708956+0000","last_peered":"2023-11-24T08:12:04.708956+0000","last_clean":"2023-11-24T08:12:04.708956+0000","last_became_active":"2023-11-24T08:12:03.711876+0000","last_became_peered":"2023-11-24T08:12:03.711876+0000","last_unstale":"2023-11-24T08:12:04.708956+0000","last_undegraded":"2023-11-24T08:12:04.708956+0000","last_fullsized":"2023-11-24T08:12:04.708956+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:25.242371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.701285+0000","last_change":"2023-11-24T08:12:04.705276+0000","last_active":"2023-11-24T08:12:05.701285+0000","last_peered":"2023-11-24T08:12:05.701285+0000","last_clean":"2023-11-24T08:12:05.701285+0000","last_became_active":"2023-11-24T08:12:04.704995+0000","last_became_peered":"2023-11-24T08:12:04.704995+0000","last_unstale":"2023-11-24T08:12:05.701285+0000","last_undegraded":"2023-11-24T08:12:05.701285+0000","last_fullsized":"2023-11-24T08:12:05.701285+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:31:38.212627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.679580+0000","last_change":"2023-11-24T08:12:03.711249+0000","last_active":"2023-11-24T08:12:04.679580+0000","last_peered":"2023-11-24T08:12:04.679580+0000","last_clean":"2023-11-24T08:12:04.679580+0000","last_became_active":"2023-11-24T08:12:03.710962+0000","last_became_peered":"2023-11-24T08:12:03.710962+0000","last_unstale":"2023-11-24T08:12:04.679580+0000","last_undegraded":"2023-11-24T08:12:04.679580+0000","last_fullsized":"2023-11-24T08:12:04.679580+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:18:53.237019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.545009+0000","last_change":"2023-11-24T08:12:04.699211+0000","last_active":"2023-11-24T08:12:06.545009+0000","last_peered":"2023-11-24T08:12:06.545009+0000","last_clean":"2023-11-24T08:12:06.545009+0000","last_became_active":"2023-11-24T08:12:04.698310+0000","last_became_peered":"2023-11-24T08:12:04.698310+0000","last_unstale":"2023-11-24T08:12:06.545009+0000","last_undegraded":"2023-11-24T08:12:06.545009+0000","last_fullsized":"2023-11-24T08:12:06.545009+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:03.365297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.685708+0000","last_change":"2023-11-24T08:12:03.709914+0000","last_active":"2023-11-24T08:12:04.685708+0000","last_peered":"2023-11-24T08:12:04.685708+0000","last_clean":"2023-11-24T08:12:04.685708+0000","last_became_active":"2023-11-24T08:12:03.709304+0000","last_became_peered":"2023-11-24T08:12:03.709304+0000","last_unstale":"2023-11-24T08:12:04.685708+0000","last_undegraded":"2023-11-24T08:12:04.685708+0000","last_fullsized":"2023-11-24T08:12:04.685708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:27:00.913350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.052989+0000","last_change":"2023-11-24T08:12:04.716088+0000","last_active":"2023-11-24T08:12:06.052989+0000","last_peered":"2023-11-24T08:12:06.052989+0000","last_clean":"2023-11-24T08:12:06.052989+0000","last_became_active":"2023-11-24T08:12:04.715819+0000","last_became_peered":"2023-11-24T08:12:04.715819+0000","last_unstale":"2023-11-24T08:12:06.052989+0000","last_undegraded":"2023-11-24T08:12:06.052989+0000","last_fullsized":"2023-11-24T08:12:06.052989+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:54.462524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.702673+0000","last_change":"2023-11-24T08:12:03.713448+0000","last_active":"2023-11-24T08:12:04.702673+0000","last_peered":"2023-11-24T08:12:04.702673+0000","last_clean":"2023-11-24T08:12:04.702673+0000","last_became_active":"2023-11-24T08:12:03.712713+0000","last_became_peered":"2023-11-24T08:12:03.712713+0000","last_unstale":"2023-11-24T08:12:04.702673+0000","last_undegraded":"2023-11-24T08:12:04.702673+0000","last_fullsized":"2023-11-24T08:12:04.702673+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:16.049203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:05.702254+0000","last_change":"2023-11-24T08:12:04.709205+0000","last_active":"2023-11-24T08:12:05.702254+0000","last_peered":"2023-11-24T08:12:05.702254+0000","last_clean":"2023-11-24T08:12:05.702254+0000","last_became_active":"2023-11-24T08:12:04.707824+0000","last_became_peered":"2023-11-24T08:12:04.707824+0000","last_unstale":"2023-11-24T08:12:05.702254+0000","last_undegraded":"2023-11-24T08:12:05.702254+0000","last_fullsized":"2023-11-24T08:12:05.702254+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:02:54.298540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.683387+0000","last_change":"2023-11-24T08:12:03.716827+0000","last_active":"2023-11-24T08:12:04.683387+0000","last_peered":"2023-11-24T08:12:04.683387+0000","last_clean":"2023-11-24T08:12:04.683387+0000","last_became_active":"2023-11-24T08:12:03.715722+0000","last_became_peered":"2023-11-24T08:12:03.715722+0000","last_unstale":"2023-11-24T08:12:04.683387+0000","last_undegraded":"2023-11-24T08:12:04.683387+0000","last_fullsized":"2023-11-24T08:12:04.683387+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:28:56.192818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-11-24T08:12:06.548001+0000","last_change":"2023-11-24T08:12:04.712837+0000","last_active":"2023-11-24T08:12:06.548001+0000","last_peered":"2023-11-24T08:12:06.548001+0000","last_clean":"2023-11-24T08:12:06.548001+0000","last_became_active":"2023-11-24T08:12:04.712479+0000","last_became_peered":"2023-11-24T08:12:04.712479+0000","last_unstale":"2023-11-24T08:12:06.548001+0000","last_undegraded":"2023-11-24T08:12:06.548001+0000","last_fullsized":"2023-11-24T08:12:06.548001+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:03.664703+0000","last_clean_scrub_stamp":"2023-11-24T08:12:03.664703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:23:02.621600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.730730+0000","last_change":"2023-11-24T08:12:03.711823+0000","last_active":"2023-11-24T08:12:04.730730+0000","last_peered":"2023-11-24T08:12:04.730730+0000","last_clean":"2023-11-24T08:12:04.730730+0000","last_became_active":"2023-11-24T08:12:03.710636+0000","last_became_peered":"2023-11-24T08:12:03.710636+0000","last_unstale":"2023-11-24T08:12:04.730730+0000","last_undegraded":"2023-11-24T08:12:04.730730+0000","last_fullsized":"2023-11-24T08:12:04.730730+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:02.648370+0000","last_clean_scrub_stamp":"2023-11-24T08:12:02.648370+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:07.902225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":230,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-24T08:12:04.691331+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:12:04.691331+0000","last_peered":"2023-11-24T08:12:04.691331+0000","last_clean":"2023-11-24T08:12:04.691331+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:12:04.691331+0000","last_undegraded":"2023-11-24T08:12:04.691331+0000","last_fullsized":"2023-11-24T08:12:04.691331+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1753088,"data_stored":1090315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0669999999999999,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.532,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1359999999999999,"15min":1.0860000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.52900000000000003},"max":{"1min":1.794,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.107}]},{"osd":1,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.161,"15min":1.1439999999999999},"min":{"1min":0.64700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6120000000000001,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.202,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.6859999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":0.97999999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.151,"15min":1.2050000000000001},"min":{"1min":0.71699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8859999999999999,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3340000000000001,"15min":1.361},"min":{"1min":1.0649999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7410000000000001,"5min":4.0650000000000004,"15min":4.633},"last":1.349}]},{"osd":3,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.216,"15min":1.1619999999999999},"min":{"1min":0.82299999999999995,"5min":0.504,"15min":0.504},"max":{"1min":1.619,"5min":3.133,"15min":3.133},"last":1.306},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4039999999999999,"15min":1.377},"min":{"1min":1.0740000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8999999999999999,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.4399999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.355,"15min":1.3340000000000001},"min":{"1min":0.96199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.9610000000000001,"5min":2.851,"15min":2.851},"last":1.395},{"interface":"front","average":{"1min":1.393,"5min":1.3819999999999999,"15min":1.3420000000000001},"min":{"1min":0.96499999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.788,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.1919999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.5049999999999999,"15min":1.4690000000000001},"min":{"1min":1.1140000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0089999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4810000000000001},"min":{"1min":1.0649999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8560000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.4870000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:11:50 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5680000000000001,"15min":1.5429999999999999},"min":{"1min":1.1020000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0499999999999998,"5min":2.992,"15min":2.992},"last":1.611},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5149999999999999,"15min":1.46},"min":{"1min":1.022,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8069999999999999,"5min":3.798,"15min":3.798},"last":1.5569999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574940,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":500571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.0880000000000001,"15min":1.085},"min":{"1min":0.64700000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.631,"5min":1.95,"15min":1.95},"last":1.593},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0720000000000001,"15min":1.0980000000000001},"min":{"1min":0.68200000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5669999999999999,"5min":1.986,"15min":2.2069999999999999},"last":1.143}]},{"osd":1,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1539999999999999,"15min":1.248},"min":{"1min":0.85799999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6799999999999999,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1319999999999999,"15min":1.2609999999999999},"min":{"1min":0.93500000000000005,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5429999999999999,"5min":1.694,"15min":2.4039999999999999},"last":0.93700000000000006}]},{"osd":2,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0980000000000001,"15min":1.1160000000000001},"min":{"1min":0.72499999999999998,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.6779999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.226,"15min":1.361},"min":{"1min":0.92800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.8220000000000001,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.2030000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.333,"15min":1.395},"min":{"1min":0.96099999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.325,"15min":1.3460000000000001},"min":{"1min":1.0489999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.2709999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.375,"15min":1.5289999999999999},"min":{"1min":0.85399999999999998,"5min":0.79700000000000004,"15min":0.76500000000000001},"max":{"1min":1.829,"5min":2.895,"15min":2.895},"last":1.446},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4510000000000001,"15min":1.5489999999999999},"min":{"1min":1.101,"5min":0.97999999999999998,"15min":0.90800000000000003},"max":{"1min":1.901,"5min":2.2770000000000001,"15min":2.8820000000000001},"last":1.419}]},{"osd":5,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4059999999999999,"15min":1.538},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":1.744,"5min":2.2250000000000001,"15min":2.629},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5169999999999999,"15min":1.671},"min":{"1min":1.119,"5min":1.0820000000000001,"15min":1.022},"max":{"1min":2.032,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.728}]},{"osd":7,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5389999999999999,"15min":1.7030000000000001},"min":{"1min":1.234,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":1.931,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.647},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5429999999999999,"15min":1.669},"min":{"1min":1.077,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.0299999999999998,"5min":2.3660000000000001,"15min":2.6890000000000001},"last":1.7649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1171456,"data_stored":501511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0349999999999999,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.419,"5min":1.694,"15min":3.1179999999999999},"last":1.046},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.131,"15min":1.181},"min":{"1min":0.85899999999999999,"5min":0.65100000000000002,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":1.859,"15min":2.198},"last":1.627}]},{"osd":1,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1200000000000001,"15min":1.1639999999999999},"min":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.66700000000000004},"max":{"1min":1.536,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.1719999999999999,"15min":1.1519999999999999},"min":{"1min":0.73399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.55,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.4770000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.218,"15min":1.2170000000000001},"min":{"1min":0.87,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.619,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1379999999999999,"15min":1.169},"min":{"1min":0.84099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5529999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.431}]},{"osd":3,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1339999999999999,"15min":1.1830000000000001},"min":{"1min":0.78300000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.4810000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.272,"15min":1.337},"min":{"1min":0.79800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.893,"5min":1.9730000000000001,"15min":1.99},"last":1.381}]},{"osd":4,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3100000000000001,"15min":1.3380000000000001},"min":{"1min":0.871,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7310000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4550000000000001,"15min":1.5449999999999999},"min":{"1min":1.2949999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.0009999999999999,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.6850000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.302,"15min":1.4359999999999999},"min":{"1min":0.91100000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6539999999999999,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.393,"15min":1.4339999999999999},"min":{"1min":1.02,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":1.734}]},{"osd":7,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4419999999999999,"15min":1.5780000000000001},"min":{"1min":0.91700000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.129,"5min":2.129,"15min":3.4420000000000002},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5780000000000001,"15min":1.6120000000000001},"min":{"1min":1.2150000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.849}]}]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":499877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0720000000000001,"15min":1.139},"min":{"1min":0.71899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5760000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.179},{"interface":"front","average":{"1min":1.129,"5min":1.0569999999999999,"15min":1.085},"min":{"1min":0.68300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5489999999999999,"5min":1.7150000000000001,"15min":2.222},"last":1.4810000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.627,"5min":1.74,"15min":1.74},"last":1.05},{"interface":"front","average":{"1min":1.135,"5min":1.1639999999999999,"15min":1.1830000000000001},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5589999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.4059999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0620000000000001,"15min":1.1519999999999999},"min":{"1min":0.874,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5780000000000001,"5min":1.5840000000000001,"15min":2.1419999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.248,"5min":1.23,"15min":1.347},"min":{"1min":0.85499999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.649,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":1.361}]},{"osd":3,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.292,"15min":1.4119999999999999},"min":{"1min":0.95699999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.7230000000000001,"5min":2.121,"15min":2.9100000000000001},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.242,"15min":1.3160000000000001},"min":{"1min":1.0249999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5049999999999999,"5min":1.778,"15min":1.8280000000000001},"last":1.1279999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4510000000000001,"15min":1.5049999999999999},"min":{"1min":1.0009999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.8440000000000001,"5min":2.23,"15min":2.23},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.395,"15min":1.526},"min":{"1min":1.1699999999999999,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.6220000000000001,"5min":2.1890000000000001,"15min":2.3879999999999999},"last":1.2490000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2190000000000001,"15min":1.4199999999999999},"min":{"1min":0.88700000000000001,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.7030000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.5049999999999999,"15min":1.6399999999999999},"min":{"1min":1.159,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.794,"5min":2.1230000000000002,"15min":3.391},"last":1.2709999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4570000000000001,"15min":1.6080000000000001},"min":{"1min":0.98999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.6870000000000001,"5min":2.0750000000000002,"15min":3.1230000000000002},"last":1.3},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.5680000000000001},"min":{"1min":1.056,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6659999999999999,"5min":1.9419999999999999,"15min":3.4849999999999999},"last":1.446}]}]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1167360,"data_stored":501489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.93799999999999994,"15min":0.83799999999999997},"min":{"1min":0.44500000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.1000000000000001,"5min":10.557,"15min":10.557},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.363,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78800000000000003,"15min":0.82899999999999996},"min":{"1min":0.45500000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.006,"5min":1.0720000000000001,"15min":0.996},"min":{"1min":0.58499999999999996,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":2.1909999999999998,"5min":13.090999999999999,"15min":13.090999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.91000000000000003,"15min":0.95699999999999996},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.3700000000000001,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1950000000000001,"15min":1.0720000000000001},"min":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.9690000000000001,"5min":13.460000000000001,"15min":13.460000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2689999999999999,"15min":1.272},"min":{"1min":0.71199999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.639,"5min":4.367,"15min":4.367},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2210000000000001,"15min":1.3109999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":0.878}]},{"osd":5,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3779999999999999,"15min":1.3140000000000001},"min":{"1min":0.745,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2869999999999999,"5min":13.231,"15min":13.231},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2450000000000001,"15min":1.3120000000000001},"min":{"1min":0.752,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8340000000000001,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":0.81899999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4550000000000001,"15min":1.3919999999999999},"min":{"1min":0.81599999999999995,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":3.1120000000000001,"5min":13.359999999999999,"15min":13.359999999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.222,"15min":1.333},"min":{"1min":0.73799999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":0.83599999999999997}]},{"osd":7,"last update":"Fri Nov 24 08:11:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3140000000000001,"15min":1.4159999999999999},"min":{"1min":0.82599999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.101,"5min":4.266,"15min":4.5060000000000002},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.2889999999999999,"15min":1.4410000000000001},"min":{"1min":0.76900000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.331,"5min":3.331,"15min":4.032},"last":1.0649999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574939,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36476,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335364,"statfs":{"total":96636764160,"available":96599412736,"internally_reserved":0,"allocated":1159168,"data_stored":500459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.76800000000000002,"15min":0.89000000000000001},"min":{"1min":0.47599999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.86299999999999999,"15min":0.99299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":2.5779999999999998},"last":1.3129999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.873,"15min":1.089},"min":{"1min":0.49399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.51,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":1.123},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.93300000000000005,"15min":1.0980000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5920000000000001,"5min":1.8360000000000001,"15min":2.6240000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.84599999999999997,"15min":0.93600000000000005},"min":{"1min":0.56499999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.552,"5min":1.552,"15min":2.2090000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.0069999999999999,"15min":1.2250000000000001},"min":{"1min":0.627,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.631,"5min":1.946,"15min":2.9950000000000001},"last":1.226}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.27,"15min":1.391},"min":{"1min":0.93400000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7829999999999999,"5min":1.9730000000000001,"15min":2.7160000000000002},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.2549999999999999,"15min":1.381},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8779999999999999,"5min":1.929,"15min":3.3279999999999998},"last":1.079}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.25,"15min":1.423},"min":{"1min":0.88500000000000001,"5min":0.75800000000000001,"15min":0.73399999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2390000000000001,"15min":1.385},"min":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":1.034}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1930000000000001,"15min":1.3220000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.323,"15min":1.4870000000000001},"min":{"1min":0.98199999999999998,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.2969999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3180000000000001,"15min":1.484},"min":{"1min":0.77700000000000002,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.1070000000000002,"5min":2.1299999999999999,"15min":2.859},"last":1.181},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.2669999999999999,"15min":1.4490000000000001},"min":{"1min":0.88600000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.9930000000000001,"5min":2.02,"15min":3.3809999999999998},"last":1.2769999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":500071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.81000000000000005,"15min":0.76600000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3400000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.84599999999999997},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":1.6970000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.81000000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81999999999999995,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.369,"5min":1.861,"15min":1.861},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.92700000000000005,"15min":0.88300000000000001},"min":{"1min":0.46500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.4039999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0009999999999999,"15min":0.95599999999999996},"min":{"1min":0.53300000000000003,"5min":0.497,"15min":0.44500000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.94299999999999995,"15min":0.88600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.88500000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.24},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.028,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.276,"15min":1.2549999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.016,"5min":2.8300000000000001,"15min":3.0409999999999999},"last":1.2669999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2869999999999999,"15min":1.208},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.9610000000000001,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.2470000000000001,"15min":1.181},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8759999999999999,"5min":3.286,"15min":3.286},"last":1.633}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3460000000000001,"15min":1.238},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.5270000000000001,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.012},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3149999999999999,"15min":1.2110000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9179999999999999,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.514}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3500000000000001,"15min":1.3009999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.3959999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.222},{"interface":"front","average":{"1min":1.266,"5min":1.377,"15min":1.276},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0649999999999999,"5min":3.653,"15min":3.653},"last":1.718}]}]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1753088,"data_stored":1090279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72299999999999998,"15min":0.85599999999999998},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0189999999999999,"5min":1.425,"15min":1.6020000000000001},"last":1.054},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.83599999999999997,"15min":0.98899999999999999},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.181,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.79800000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.006,"15min":1.056},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.411,"5min":1.891,"15min":1.891},"last":1.002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.93999999999999995},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96599999999999997,"5min":1.847,"15min":1.847},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98599999999999999,"15min":1.101},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.724,"5min":1.79,"15min":2.0670000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88900000000000001,"15min":1.036},"min":{"1min":0.53400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.494,"5min":1.494,"15min":2.1789999999999998},"last":1.2270000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.256,"15min":1.3899999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.528,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":1.101},{"interface":"front","average":{"1min":1.151,"5min":1.0980000000000001,"15min":1.2290000000000001},"min":{"1min":0.76100000000000001,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6240000000000001,"5min":1.6890000000000001,"15min":2.2839999999999998},"last":1.3400000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.141,"15min":1.298},"min":{"1min":0.72799999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6519999999999999,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.177,"15min":1.399},"min":{"1min":0.83699999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.47,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":1.298}]},{"osd":6,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.26,"15min":1.417},"min":{"1min":0.71099999999999997,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.649,"5min":1.986,"15min":2.4369999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.2569999999999999,"15min":1.427},"min":{"1min":0.86399999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.766,"5min":1.9399999999999999,"15min":2.738},"last":1.266}]},{"osd":7,"last update":"Fri Nov 24 08:11:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.355,"15min":1.5049999999999999},"min":{"1min":0.879,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.3319999999999999,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.165},{"interface":"front","average":{"1min":1.385,"5min":1.3200000000000001,"15min":1.4930000000000001},"min":{"1min":0.996,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.4410000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:12:09.806 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:12:09.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:12:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.070+0000 7fc2765cf640 1 Processor -- start 2023-11-24T08:12:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.070+0000 7fc2765cf640 1 -- start start 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc270141b30 0x7fc27006c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc27015ef00 0x7fc270071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc270071c40 0x7fc2700cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2700ccdf0 con 0x7fc270071c40 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27015f470 con 0x7fc270141b30 2023-11-24T08:12:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc2765cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27015f750 con 0x7fc27015ef00 2023-11-24T08:12:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc275dce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc270071c40 0x7fc2700cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc275dce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc270071c40 0x7fc2700cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50982/0 (socket says 172.21.15.107:50982) 2023-11-24T08:12:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.072+0000 7fc275dce640 1 -- 172.21.15.107:0/301306884 learned_addr learned my addr 172.21.15.107:0/301306884 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc2755cd640 1 --2- 172.21.15.107:0/301306884 >> v2:172.21.15.120:3300/0 conn(0x7fc270141b30 0x7fc27006c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc274dcc640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc27015ef00 0x7fc270071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc275dce640 1 -- 172.21.15.107:0/301306884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc27015ef00 msgr2=0x7fc270071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc275dce640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc27015ef00 0x7fc270071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc275dce640 1 -- 172.21.15.107:0/301306884 >> v2:172.21.15.120:3300/0 conn(0x7fc270141b30 msgr2=0x7fc27006c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc275dce640 1 --2- 172.21.15.107:0/301306884 >> v2:172.21.15.120:3300/0 conn(0x7fc270141b30 0x7fc27006c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.073+0000 7fc275dce640 1 -- 172.21.15.107:0/301306884 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27015fa30 con 0x7fc270071c40 2023-11-24T08:12:10.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.074+0000 7fc275dce640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270071c40 0x7fc2700cede0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc2700b1c10 tx=0x7fc26802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421ffafa7aae0656 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.074+0000 7fc267fff640 1 -- 172.21.15.107:0/301306884 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268031740 con 0x7fc270071c40 2023-11-24T08:12:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.074+0000 7fc267fff640 1 -- 172.21.15.107:0/301306884 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2680318a0 con 0x7fc270071c40 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.074+0000 7fc267fff640 1 -- 172.21.15.107:0/301306884 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268031c30 con 0x7fc270071c40 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 -- 172.21.15.107:0/301306884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270071c40 msgr2=0x7fc2700cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270071c40 0x7fc2700cede0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc2700b1c10 tx=0x7fc26802b7e0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 -- 172.21.15.107:0/301306884 shutdown_connections 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270071c40 0x7fc2700cede0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 --2- 172.21.15.107:0/301306884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc27015ef00 0x7fc270071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 --2- 172.21.15.107:0/301306884 >> v2:172.21.15.120:3300/0 conn(0x7fc270141b30 0x7fc27006c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 -- 172.21.15.107:0/301306884 >> 172.21.15.107:0/301306884 conn(0x7fc27006a500 msgr2=0x7fc27006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 -- 172.21.15.107:0/301306884 shutdown_connections 2023-11-24T08:12:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.075+0000 7fc2765cf640 1 -- 172.21.15.107:0/301306884 wait complete. 2023-11-24T08:12:10.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.078+0000 7fc2765cf640 1 Processor -- start 2023-11-24T08:12:10.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.078+0000 7fc2765cf640 1 -- start start 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc270071c40 0x7fc2700cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc270141b30 0x7fc270141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270142990 con 0x7fc27015ef00 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270142c70 con 0x7fc270141b30 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.079+0000 7fc2765cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc270142f50 con 0x7fc270071c40 2023-11-24T08:12:10.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50990/0 (socket says 172.21.15.107:50990) 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 -- 172.21.15.107:0/1306417986 learned_addr learned my addr 172.21.15.107:0/1306417986 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc274dcc640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc270141b30 0x7fc270141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc2755cd640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc270071c40 0x7fc2700cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 -- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc270071c40 msgr2=0x7fc2700cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc270071c40 0x7fc2700cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 -- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc270141b30 msgr2=0x7fc270141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc270141b30 0x7fc270141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.080+0000 7fc275dce640 1 -- 172.21.15.107:0/1306417986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680094d0 con 0x7fc27015ef00 2023-11-24T08:12:10.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc275dce640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc2680090b0 tx=0x7fc2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268033d60 con 0x7fc27015ef00 2023-11-24T08:12:10.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26803c400 con 0x7fc27015ef00 2023-11-24T08:12:10.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26803c7b0 con 0x7fc27015ef00 2023-11-24T08:12:10.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc270143230 con 0x7fc27015ef00 2023-11-24T08:12:10.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.081+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27015e410 con 0x7fc27015ef00 2023-11-24T08:12:10.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.083+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc270143750 con 0x7fc27015ef00 2023-11-24T08:12:10.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.083+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fc26803a040 con 0x7fc27015ef00 2023-11-24T08:12:10.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.084+0000 7fc265ffb640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 0x7fc25406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.084+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2680a9e10 con 0x7fc27015ef00 2023-11-24T08:12:10.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.084+0000 7fc2755cd640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 0x7fc25406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.085+0000 7fc2755cd640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 0x7fc25406ded0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c007560 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.089+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc270143750 con 0x7fc27015ef00 2023-11-24T08:12:10.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.352+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc270143750 con 0x7fc27015ef00 2023-11-24T08:12:10.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.357+0000 7fc265ffb640 1 -- 172.21.15.107:0/1306417986 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc270143750 con 0x7fc27015ef00 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:12:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:12:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 msgr2=0x7fc25406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 0x7fc25406ded0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c007560 comp rx=0 tx=0).stop 2023-11-24T08:12:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 msgr2=0x7fc270159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc2680090b0 tx=0x7fc2680046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 shutdown_connections 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc25406ba10 0x7fc25406ded0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc27015ef00 0x7fc270159db0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc270141b30 0x7fc270141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 --2- 172.21.15.107:0/1306417986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc270071c40 0x7fc2700cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 >> 172.21.15.107:0/1306417986 conn(0x7fc27006a500 msgr2=0x7fc27006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.360+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 shutdown_connections 2023-11-24T08:12:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.361+0000 7fc2765cf640 1 -- 172.21.15.107:0/1306417986 wait complete. 2023-11-24T08:12:10.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:10.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.595+0000 7fa9affff640 1 Processor -- start 2023-11-24T08:12:10.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.595+0000 7fa9affff640 1 -- start start 2023-11-24T08:12:10.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.596+0000 7fa9affff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9affff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9b006db20 0x7fa9b00b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9affff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00cec00 con 0x7fa9b006d6c0 2023-11-24T08:12:10.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9affff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b013f9e0 con 0x7fa9b006db20 2023-11-24T08:12:10.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9affff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b013fcc0 con 0x7fa9b013d120 2023-11-24T08:12:10.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9af7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9b006db20 0x7fa9b00b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9aeffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9af7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9b006db20 0x7fa9b00b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:57822/0 (socket says 172.21.15.107:57822) 2023-11-24T08:12:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.597+0000 7fa9af7fe640 1 -- 172.21.15.107:0/2702775930 learned_addr learned my addr 172.21.15.107:0/2702775930 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.598+0000 7fa9af7fe640 1 -- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 msgr2=0x7fa9b015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.598+0000 7fa9af7fe640 1 --2- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b015af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.598+0000 7fa9af7fe640 1 -- 172.21.15.107:0/2702775930 >> v1:172.21.15.120:6789/0 conn(0x7fa9b006d6c0 legacy=0x7fa9b00b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.598+0000 7fa9af7fe640 1 -- 172.21.15.107:0/2702775930 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b013ffa0 con 0x7fa9b006db20 2023-11-24T08:12:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.599+0000 7fa9af7fe640 1 --2- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006db20 0x7fa9b00b1180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009870 tx=0x7fa9a802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c83d48b93e027e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9ae7fc640 1 -- 172.21.15.107:0/2702775930 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8031640 con 0x7fa9b006db20 2023-11-24T08:12:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9ae7fc640 1 -- 172.21.15.107:0/2702775930 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80317a0 con 0x7fa9b006db20 2023-11-24T08:12:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9ae7fc640 1 -- 172.21.15.107:0/2702775930 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8031b30 con 0x7fa9b006db20 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 -- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006db20 msgr2=0x7fa9b00b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 --2- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006db20 0x7fa9b00b1180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009870 tx=0x7fa9a802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 -- 172.21.15.107:0/2702775930 shutdown_connections 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 --2- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006db20 0x7fa9b00b1180 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 --2- 172.21.15.107:0/2702775930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.600+0000 7fa9affff640 1 -- 172.21.15.107:0/2702775930 >> 172.21.15.107:0/2702775930 conn(0x7fa9b00cb830 msgr2=0x7fa9b00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.601+0000 7fa9affff640 1 -- 172.21.15.107:0/2702775930 shutdown_connections 2023-11-24T08:12:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.601+0000 7fa9affff640 1 -- 172.21.15.107:0/2702775930 wait complete. 2023-11-24T08:12:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.602+0000 7fa9affff640 1 Processor -- start 2023-11-24T08:12:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.602+0000 7fa9affff640 1 -- start start 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006d6c0 0x7fa9b00b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b0143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b013ccd0 con 0x7fa9b006db20 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00ada10 con 0x7fa9b006d6c0 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.603+0000 7fa9affff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00ace00 con 0x7fa9b013d120 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50996/0 (socket says 172.21.15.107:50996) 2023-11-24T08:12:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 -- 172.21.15.107:0/2403878845 learned_addr learned my addr 172.21.15.107:0/2403878845 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9aeffd640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006d6c0 0x7fa9b00b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9af7fe640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b0143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 -- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 msgr2=0x7fa9b0143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b0143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 -- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006d6c0 msgr2=0x7fa9b00b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006d6c0 0x7fa9b00b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 -- 172.21.15.107:0/2403878845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a80094d0 con 0x7fa9b006db20 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.604+0000 7fa9a67fc640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa99800c870 tx=0x7fa99800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.605+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998003b70 con 0x7fa9b006db20 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.605+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998003cf0 con 0x7fa9b006db20 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.605+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b00ad0e0 con 0x7fa9b006db20 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.605+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99800aa90 con 0x7fa9b006db20 2023-11-24T08:12:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.605+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b00ad6a0 con 0x7fa9b006db20 2023-11-24T08:12:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.606+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b00ad830 con 0x7fa9b006db20 2023-11-24T08:12:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.607+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fa99800abf0 con 0x7fa9b006db20 2023-11-24T08:12:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.608+0000 7fa9a7fff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 0x7fa99006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.608+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa998088120 con 0x7fa9b006db20 2023-11-24T08:12:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.608+0000 7fa9aeffd640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 0x7fa99006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.609+0000 7fa9aeffd640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 0x7fa99006ded0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009a30 tx=0x7fa9a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:10.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.612+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9980549c0 con 0x7fa9b006db20 2023-11-24T08:12:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.840+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b0067d20 con 0x7fa9b006db20 2023-11-24T08:12:10.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.841+0000 7fa9a7fff640 1 -- 172.21.15.107:0/2403878845 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fa998003e60 con 0x7fa9b006db20 2023-11-24T08:12:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:10.843 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:10.843 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 51 2023-11-24T08:12:10.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 msgr2=0x7fa99006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 0x7fa99006ded0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009a30 tx=0x7fa9a0008040 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 msgr2=0x7fa9b0142e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa99800c870 tx=0x7fa99800cd30 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 shutdown_connections 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa99006ba10 0x7fa99006ded0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9b013d120 0x7fa9b0143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b006db20 0x7fa9b0142e20 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 --2- 172.21.15.107:0/2403878845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9b006d6c0 0x7fa9b00b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.843+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 >> 172.21.15.107:0/2403878845 conn(0x7fa9b00cb830 msgr2=0x7fa9b006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.844+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 shutdown_connections 2023-11-24T08:12:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:10.844+0000 7fa9affff640 1 -- 172.21.15.107:0/2403878845 wait complete. 2023-11-24T08:12:11.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:12:11.086 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:12:11.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:12:04.149897+0000', 'modified': '2023-11-24T08:12:07.368471+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7332}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7332': {'gid': 7332, 'name': 'a', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.107:6837/1534091061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1534091061}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1534091061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:12:11.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:12:11.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-24T08:12:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.491+0000 7f5913101640 1 Processor -- start 2023-11-24T08:12:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.491+0000 7f5913101640 1 -- start start 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f5913101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f590c0aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f5913101640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590c15aef0 con 0x7f590c0b3c00 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f5913101640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590c159d70 con 0x7f590c06e6c0 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f5913101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c06e2a0 con 0x7f590c0cf430 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f591169e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f590c0b3c00 0x7f590c06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39198/0 (socket says 172.21.15.107:39198) 2023-11-24T08:12:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.493+0000 7f591169e640 1 -- 172.21.15.107:0/1868434260 learned_addr learned my addr 172.21.15.107:0/1868434260 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098305326 0 0) 0x7f590c15aef0 con 0x7f590c0b3c00 2023-11-24T08:12:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f590c0aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f8003440 con 0x7f590c0b3c00 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466403917 0 0) 0x7f58f8003440 con 0x7f590c0b3c00 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 msgr2=0x7f590c0aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f590c0aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 >> v1:172.21.15.120:6789/0 conn(0x7f590c06e6c0 legacy=0x7f590c0b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c06d900 con 0x7f590c0b3c00 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5904001670 con 0x7f590c0b3c00 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f590c0aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.494+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 >> v1:172.21.15.107:6789/0 conn(0x7f590c0b3c00 legacy=0x7f590c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58f8003a90 0x7f58f8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f58f800a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58f800b0c0 0x7f58f800d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c06e2a0 con 0x7f58f8003a90 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f800ddd0 con 0x7f58f800b0c0 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f800df90 con 0x7f590c0cf430 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58f8003a90 0x7f58f8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f590bfff640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f58f800a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 -- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 msgr2=0x7f58f800a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f590c0cf430 0x7f58f800a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 -- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58f800b0c0 msgr2=0x7f58f800d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f591169e640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58f800b0c0 0x7f58f800d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58f800b0c0 0x7f58f800d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.495+0000 7f5910e9d640 1 -- 172.21.15.107:0/1868434260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f8003440 con 0x7f58f8003a90 2023-11-24T08:12:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.496+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58f8003a90 0x7f58f8003eb0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0073e0 tx=0x7f58f80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c546fe525d284a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc02cd30 con 0x7f58f8003a90 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc02ce90 con 0x7f58f8003a90 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f590b7fe640 1 -- 172.21.15.107:0/1868434260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc034640 con 0x7f58f8003a90 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f5913101640 1 -- 172.21.15.107:0/1868434260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58f8003a90 msgr2=0x7f58f8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f5913101640 1 --2- 172.21.15.107:0/1868434260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58f8003a90 0x7f58f8003eb0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0073e0 tx=0x7f58f80036f0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.497+0000 7f591169e640 1 -- 172.21.15.107:0/1868434260 reap_dead start 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.498+0000 7f5913101640 1 -- 172.21.15.107:0/1868434260 shutdown_connections 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.498+0000 7f5913101640 1 -- 172.21.15.107:0/1868434260 >> 172.21.15.107:0/1868434260 conn(0x7f590c0cf900 msgr2=0x7f590c0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.498+0000 7f5913101640 1 -- 172.21.15.107:0/1868434260 shutdown_connections 2023-11-24T08:12:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.498+0000 7f5913101640 1 -- 172.21.15.107:0/1868434260 wait complete. 2023-11-24T08:12:11.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.499+0000 7f5913101640 1 Processor -- start 2023-11-24T08:12:11.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 -- start start 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58f8003a90 0x7f590c0aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f590c0acf60 0x7f590c1e65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c147b90 con 0x7f590c146590 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0ad3e0 con 0x7f590c0acf60 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.500+0000 7f5913101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c146f10 con 0x7f58f8003a90 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51028/0 (socket says 172.21.15.107:51028) 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 -- 172.21.15.107:0/1537340466 learned_addr learned my addr 172.21.15.107:0/1537340466 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58f8003a90 0x7f590c0aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f591169e640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f590c0acf60 0x7f590c1e65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 -- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58f8003a90 msgr2=0x7f590c0aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58f8003a90 0x7f590c0aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 -- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f590c0acf60 msgr2=0x7f590c1e65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f590c0acf60 0x7f590c1e65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.501+0000 7f590bfff640 1 -- 172.21.15.107:0/1537340466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c1471f0 con 0x7f590c146590 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f590bfff640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d570 tx=0x7f58f400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f59097fa640 1 -- 172.21.15.107:0/1537340466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4005480 con 0x7f590c146590 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f59097fa640 1 -- 172.21.15.107:0/1537340466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4003660 con 0x7f590c146590 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590c1474d0 con 0x7f590c146590 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f59097fa640 1 -- 172.21.15.107:0/1537340466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40039d0 con 0x7f590c146590 2023-11-24T08:12:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.502+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c146a10 con 0x7f590c146590 2023-11-24T08:12:11.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.504+0000 7f59097fa640 1 -- 172.21.15.107:0/1537340466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f58f4003b30 con 0x7f590c146590 2023-11-24T08:12:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.505+0000 7f59097fa640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 0x7f58ec06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.505+0000 7f59097fa640 1 -- 172.21.15.107:0/1537340466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f4087e90 con 0x7f590c146590 2023-11-24T08:12:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.505+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 0x7f58ec06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.506+0000 7f5915d6a740 1 Processor -- start 2023-11-24T08:12:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.506+0000 7f5915d6a740 1 -- start start 2023-11-24T08:12:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.506+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 0x7f58ec06df00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f58fc036000 tx=0x7f58fc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b94280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08b930a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08af2880 con 0x555e08b935e0 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08b01ae0 con 0x555e08b8eba0 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e08aff1d0 con 0x555e08b93ea0 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f591169e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f591169e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51030/0 (socket says 172.21.15.107:51030) 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.507+0000 7f591169e640 1 -- 172.21.15.107:0/1748635245 learned_addr learned my addr 172.21.15.107:0/1748635245 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f5910e9d640 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b94280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f590bfff640 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08b930a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 -- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 msgr2=0x555e08b94280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b94280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 -- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 msgr2=0x555e08b930a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08b930a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 -- 172.21.15.107:0/1748635245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e08ac2860 con 0x555e08b935e0 2023-11-24T08:12:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f591169e640 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f590c15d430 tx=0x7f5904007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2183db63a961faf0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f59027fc640 1 -- 172.21.15.107:0/1748635245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904007230 con 0x555e08b935e0 2023-11-24T08:12:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f59027fc640 1 -- 172.21.15.107:0/1748635245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590400d040 con 0x555e08b935e0 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.508+0000 7f59027fc640 1 -- 172.21.15.107:0/1748635245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59040075b0 con 0x555e08b935e0 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 -- 172.21.15.107:0/1748635245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 msgr2=0x555e08bd7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f590c15d430 tx=0x7f5904007be0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 -- 172.21.15.107:0/1748635245 shutdown_connections 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bd7560 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08b930a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 --2- 172.21.15.107:0/1748635245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b94280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 -- 172.21.15.107:0/1748635245 >> 172.21.15.107:0/1748635245 conn(0x555e08b55c70 msgr2=0x555e08bd9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 -- 172.21.15.107:0/1748635245 shutdown_connections 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.509+0000 7f5915d6a740 1 -- 172.21.15.107:0/1748635245 wait complete. 2023-11-24T08:12:11.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.510+0000 7f5915d6a740 1 -- start start 2023-11-24T08:12:11.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.510+0000 7f5915d6a740 1 Processor -- start 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b8eba0 0x555e08bd6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b93ea0 0x555e08bbed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08afaac0 con 0x555e08b935e0 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08a972f0 con 0x555e08b93ea0 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e08ab4c70 con 0x555e08b8eba0 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f5915d6a740 10 client.0 _ll_register_callbacks cb 0x555e08ab93b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f590bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f590bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51044/0 (socket says 172.21.15.107:51044) 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.511+0000 7f590bfff640 1 -- 172.21.15.107:0/2552510284 learned_addr learned my addr 172.21.15.107:0/2552510284 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 -- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b8eba0 msgr2=0x555e08bd6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b8eba0 0x555e08bd6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 -- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b93ea0 msgr2=0x555e08bbed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b93ea0 0x555e08bbed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 -- 172.21.15.107:0/2552510284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c06d900 con 0x555e08b935e0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f590bfff640 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f58f400df30 tx=0x7f58f4058650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33521b2bd9dbfe1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f58eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40606c0 con 0x555e08b935e0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f4060950 con 0x555e08b935e0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e08af3aa0 con 0x555e08b935e0 2023-11-24T08:12:11.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58f4054440 con 0x555e08b935e0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f5915d6a740 10 client.7476 fetch_fsmap learned FSMap version 51 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.512+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x555e08af9140 con 0x555e08b935e0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555e08af3aa0 con 0x555e08b935e0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f58f4078050 con 0x555e08b935e0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 10 client.7476 fetch_fsmap finished waiting for FSMap version 51 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 10 client.7476 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 20 client.7476 populate_metadata read hostname 'smithi107' 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bbaee0 0x555e08bc0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 4 client.7476 mds_command: new command op to 7332 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 --> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e08af37c0 con 0x555e08bbaee0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.513+0000 7f5910e9d640 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bbaee0 0x555e08bc0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.514+0000 7f5910e9d640 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bbaee0 0x555e08bc0e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a70a46ed0b46800a server_cookie=9525a10c5e4c57a5 in_seq=0 out_seq=0 2023-11-24T08:12:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.514+0000 7f58eb7fe640 10 client.7476 ms_handle_connect on v2:172.21.15.107:6836/1534091061 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f58eb7fe640 1 -- 172.21.15.107:0/2552510284 <== mds.0 v2:172.21.15.107:6836/1534091061 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e08af37c0 con 0x555e08bbaee0 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f58eb7fe640 10 client.7476 handle_command_reply: tid=0 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 client.7476 shutdown 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 msgr2=0x555e08bbe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f58f400df30 tx=0x7f58f4058650 comp rx=0 tx=0).stop 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 shutdown_connections 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bbaee0 0x555e08bc0e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b93ea0 0x555e08bbed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b935e0 0x555e08bbe800 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f58eb7fe640 0 client.7476 ms_handle_reset on v2:172.21.15.107:6836/1534091061 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 --2- 172.21.15.107:0/2552510284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b8eba0 0x555e08bd6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 >> 172.21.15.107:0/2552510284 conn(0x555e08b55c70 msgr2=0x555e08bdcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 shutdown_connections 2023-11-24T08:12:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 1 -- 172.21.15.107:0/2552510284 wait complete. 2023-11-24T08:12:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.515+0000 7f5915d6a740 20 client.7476 trim_cache size 0 max 16384 2023-11-24T08:12:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.532+0000 7f5915d6a740 1 Processor -- start 2023-11-24T08:12:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.532+0000 7f5915d6a740 1 -- start start 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08bdd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b935e0 0x555e08bd5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0885df20 con 0x555e08b93ea0 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08bbf1f0 con 0x555e08b8eba0 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e08bdcea0 con 0x555e08b935e0 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.533+0000 7f591169e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51050/0 (socket says 172.21.15.107:51050) 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 -- 172.21.15.107:0/4277578152 learned_addr learned my addr 172.21.15.107:0/4277578152 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 -- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b935e0 msgr2=0x555e08bd5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b935e0 0x555e08bd5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 -- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 msgr2=0x555e08bdd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08bdd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 -- 172.21.15.107:0/4277578152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e08a96b80 con 0x555e08b93ea0 2023-11-24T08:12:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f591169e640 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f590400c000 tx=0x7f5904011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21889ef81ad78162 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.534+0000 7f5902ffd640 1 -- 172.21.15.107:0/4277578152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904007240 con 0x555e08b93ea0 2023-11-24T08:12:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5902ffd640 1 -- 172.21.15.107:0/4277578152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59040073e0 con 0x555e08b93ea0 2023-11-24T08:12:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5902ffd640 1 -- 172.21.15.107:0/4277578152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59040017a0 con 0x555e08b93ea0 2023-11-24T08:12:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 -- 172.21.15.107:0/4277578152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 msgr2=0x555e08be1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f590400c000 tx=0x7f5904011000 comp rx=0 tx=0).stop 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 -- 172.21.15.107:0/4277578152 shutdown_connections 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b93ea0 0x555e08be1100 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b935e0 0x555e08bd5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 --2- 172.21.15.107:0/4277578152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b8eba0 0x555e08bdd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 -- 172.21.15.107:0/4277578152 >> 172.21.15.107:0/4277578152 conn(0x555e08b55c70 msgr2=0x555e08bbf390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 -- 172.21.15.107:0/4277578152 shutdown_connections 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.535+0000 7f5915d6a740 1 -- 172.21.15.107:0/4277578152 wait complete. 2023-11-24T08:12:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.536+0000 7f5915d6a740 1 -- start start 2023-11-24T08:12:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.536+0000 7f5915d6a740 1 Processor -- start 2023-11-24T08:12:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b935e0 0x555e08be1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b8db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08bf9fc0 con 0x555e08b8eba0 2023-11-24T08:12:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e08beb930 con 0x555e08b935e0 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e08bec420 con 0x555e08b93ea0 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5915d6a740 10 client.0 _ll_register_callbacks cb 0x7f58f4001300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5910e9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5910e9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51060/0 (socket says 172.21.15.107:51060) 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.537+0000 7f5910e9d640 1 -- 172.21.15.107:0/3627381101 learned_addr learned my addr 172.21.15.107:0/3627381101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 -- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 msgr2=0x555e08b8db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b8db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 -- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b935e0 msgr2=0x555e08be1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b935e0 0x555e08be1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 -- 172.21.15.107:0/3627381101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e08ac2860 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5910e9d640 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f58fc02c6d0 tx=0x7f58fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa9d7c5be852bd6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f58eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc050070 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58fc03fda0 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.538+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f590400d040 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58fc051b00 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 10 client.7482 fetch_fsmap learned FSMap version 51 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x555e08abc170 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590400d040 con 0x555e08b8eba0 2023-11-24T08:12:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f58fc051c60 con 0x555e08b8eba0 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 10 client.7482 fetch_fsmap finished waiting for FSMap version 51 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 10 client.7482 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 20 client.7482 populate_metadata read hostname 'smithi107' 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bf2d20 0x555e08bf3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 4 client.7482 mds_command: new command op to 7332 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 --> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555e087fb6c0 con 0x555e08bf2d20 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.539+0000 7f590bfff640 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bf2d20 0x555e08bf3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f590bfff640 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bf2d20 0x555e08bf3100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d0216745456bac server_cookie=883e848bdfe84bc4 in_seq=0 out_seq=0 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f58eb7fe640 10 client.7482 ms_handle_connect on v2:172.21.15.107:6836/1534091061 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f58eb7fe640 1 -- 172.21.15.107:0/3627381101 <== mds.0 v2:172.21.15.107:6836/1534091061 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x555e087fb6c0 con 0x555e08bf2d20 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f58eb7fe640 10 client.7482 handle_command_reply: tid=0 2023-11-24T08:12:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 client.7482 shutdown 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 msgr2=0x555e08c46af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f58fc02c6d0 tx=0x7f58fc004ab0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 shutdown_connections 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:6836/1534091061,v1:172.21.15.107:6837/1534091061] conn(0x555e08bf2d20 0x555e08bf3100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e08b93ea0 0x555e08b8db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e08b935e0 0x555e08be1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 --2- 172.21.15.107:0/3627381101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e08b8eba0 0x555e08c46af0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f58eb7fe640 0 client.7482 ms_handle_reset on v2:172.21.15.107:6836/1534091061 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 >> 172.21.15.107:0/3627381101 conn(0x555e08b55c70 msgr2=0x555e08bd55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 shutdown_connections 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.540+0000 7f5915d6a740 1 -- 172.21.15.107:0/3627381101 wait complete. 2023-11-24T08:12:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5915d6a740 20 client.7482 trim_cache size 0 max 16384 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 7332, 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.8467855780000004, 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 51, 2023-11-24T08:12:11.548 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 94, 2023-11-24T08:12:11.549 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 94, 2023-11-24T08:12:11.549 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 19.678051266000001 2023-11-24T08:12:11.549 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 msgr2=0x7f58ec06df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 0x7f58ec06df00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f58fc036000 tx=0x7f58fc030000 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 msgr2=0x7f590c0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d570 tx=0x7f58f400da30 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 shutdown_connections 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f58ec06ba40 0x7f58ec06df00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f590c0acf60 0x7f590c1e65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f590c146590 0x7f590c0aca20 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 --2- 172.21.15.107:0/1537340466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58f8003a90 0x7f590c0aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 >> 172.21.15.107:0/1537340466 conn(0x7f590c0cf900 msgr2=0x7f590c0b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 shutdown_connections 2023-11-24T08:12:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.541+0000 7f5913101640 1 -- 172.21.15.107:0/1537340466 wait complete. 2023-11-24T08:12:11.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:12:11.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.759+0000 7f11114cf640 1 Processor -- start 2023-11-24T08:12:11.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.760+0000 7f11114cf640 1 -- start start 2023-11-24T08:12:11.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.761+0000 7f11114cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f110c15d780 0x7f110c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.761+0000 7f11114cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d1750 0x7f110c0d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.761+0000 7f11114cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c0d3da0 con 0x7f110c0d3400 2023-11-24T08:12:11.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.761+0000 7f11114cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110c0ce4f0 con 0x7f110c15d780 2023-11-24T08:12:11.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.761+0000 7f11114cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0b36c0 con 0x7f110c0d1750 2023-11-24T08:12:11.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f110c15d780 0x7f110c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f110c15d780 0x7f110c0b3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51066/0 (socket says 172.21.15.107:51066) 2023-11-24T08:12:11.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 -- 172.21.15.107:0/166737175 learned_addr learned my addr 172.21.15.107:0/166737175 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110b7fe640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d1750 0x7f110c0d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 -- 172.21.15.107:0/166737175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d1750 msgr2=0x7f110c0d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d1750 0x7f110c0d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 -- 172.21.15.107:0/166737175 >> v1:172.21.15.120:6789/0 conn(0x7f110c0d3400 legacy=0x7f110c0d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:11.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.762+0000 7f110affd640 1 -- 172.21.15.107:0/166737175 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c0d1d30 con 0x7f110c15d780 2023-11-24T08:12:11.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.763+0000 7f110affd640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c15d780 0x7f110c0b3180 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009870 tx=0x7f10fc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3f4408a01b57bd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.764+0000 7f110a7fc640 1 -- 172.21.15.107:0/166737175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc031740 con 0x7f110c15d780 2023-11-24T08:12:11.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.764+0000 7f110a7fc640 1 -- 172.21.15.107:0/166737175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc0318a0 con 0x7f110c15d780 2023-11-24T08:12:11.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.764+0000 7f110a7fc640 1 -- 172.21.15.107:0/166737175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc031c30 con 0x7f110c15d780 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.764+0000 7f11114cf640 1 -- 172.21.15.107:0/166737175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c15d780 msgr2=0x7f110c0b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.764+0000 7f11114cf640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c15d780 0x7f110c0b3180 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009870 tx=0x7f10fc02b7e0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 -- 172.21.15.107:0/166737175 shutdown_connections 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d1750 0x7f110c0d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 --2- 172.21.15.107:0/166737175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c15d780 0x7f110c0b3180 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 -- 172.21.15.107:0/166737175 >> 172.21.15.107:0/166737175 conn(0x7f110c08cec0 msgr2=0x7f110c09c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 -- 172.21.15.107:0/166737175 shutdown_connections 2023-11-24T08:12:11.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.765+0000 7f11114cf640 1 -- 172.21.15.107:0/166737175 wait complete. 2023-11-24T08:12:11.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.766+0000 7f11114cf640 1 Processor -- start 2023-11-24T08:12:11.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.767+0000 7f11114cf640 1 -- start start 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d3400 0x7f110c0b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f110c15d780 0x7f110c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c15ef00 con 0x7f110c0d1750 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c160690 con 0x7f110c15d780 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f11114cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c15e6f0 con 0x7f110c0d3400 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51070/0 (socket says 172.21.15.107:51070) 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 -- 172.21.15.107:0/2752454320 learned_addr learned my addr 172.21.15.107:0/2752454320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 -- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d3400 msgr2=0x7f110c0b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d3400 0x7f110c0b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 -- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f110c15d780 msgr2=0x7f110c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f110c15d780 0x7f110c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.768+0000 7f110affd640 1 -- 172.21.15.107:0/2752454320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0094d0 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f110affd640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009870 tx=0x7f10fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc033c70 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc033dd0 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc03c620 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c0b8a60 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.769+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c15e940 con 0x7f110c0d1750 2023-11-24T08:12:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.770+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c067470 con 0x7f110c0d1750 2023-11-24T08:12:11.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.771+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f10fc03a040 con 0x7f110c0d1750 2023-11-24T08:12:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.772+0000 7f1103fff640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 0x7f10ec06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.772+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10fc0a9c90 con 0x7f110c0d1750 2023-11-24T08:12:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.772+0000 7f11027fc640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 0x7f10ec06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:11.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.773+0000 7f11027fc640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 0x7f10ec06ded0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:11.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.776+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10fc076590 con 0x7f110c0d1750 2023-11-24T08:12:12.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:11.999+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f110c0b2f60 con 0x7f110c0d1750 2023-11-24T08:12:12.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.000+0000 7f1103fff640 1 -- 172.21.15.107:0/2752454320 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f10fc02f070 con 0x7f110c0d1750 2023-11-24T08:12:12.002 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.002+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 msgr2=0x7f10ec06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.002+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 0x7f10ec06ded0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006bc0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 msgr2=0x7f110c1f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009870 tx=0x7f10fc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 shutdown_connections 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f10ec06ba10 0x7f10ec06ded0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f110c15d780 0x7f110c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f110c0d3400 0x7f110c0b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 --2- 172.21.15.107:0/2752454320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f110c0d1750 0x7f110c1f7580 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:12.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 >> 172.21.15.107:0/2752454320 conn(0x7f110c08cec0 msgr2=0x7f110c15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:12.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 shutdown_connections 2023-11-24T08:12:12.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:12.003+0000 7f11114cf640 1 -- 172.21.15.107:0/2752454320 wait complete. 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:12:12.023 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:12:12.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.024 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:12.075 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41938sec preferred_lft 41938sec 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.076 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:12:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:12.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.079 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:12:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:12:12.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.162 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:12:12.215 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:12:12.215 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:12:12.215 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:12:12.215 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:12:12.215 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:12:12.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:12:12.216 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:12:12.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.397 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:12:12.447 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:12:12.448 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:12:12.449 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:12:12.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:12:12.537 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:12:12.537 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:12:12.592 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:12:12.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:12:12.593 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:12:12.642 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:12:12.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:12.780 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:12.781 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:12:12.814 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:12.815 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:12:12.815 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:12:12.815 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:12:12.815 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:12:12.816 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:12:12.816 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:12:12.817 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:12:12.817 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:12:12.817 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:12:12.820 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:12:12.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:12:12.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:12.904 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:12:12.966 INFO:teuthology.orchestra.run.smithi107.stdout:client5770 2023-11-24T08:12:12.966 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:12.967 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:13.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:12:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.229+0000 7f6d22705640 1 Processor -- start 2023-11-24T08:12:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.229+0000 7f6d22705640 1 -- start start 2023-11-24T08:12:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.230+0000 7f6d22705640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6d1c15c7f0 0x7f6d1c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.230+0000 7f6d22705640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6d1c0cf450 0x7f6d1c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.230+0000 7f6d22705640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c0cece0 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.230+0000 7f6d22705640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0cfe40 con 0x7f6d1c0cf450 2023-11-24T08:12:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.230+0000 7f6d22705640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0ceaf0 con 0x7f6d1c15c7f0 2023-11-24T08:12:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6d1c0cf450 0x7f6d1c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6d1c0cf450 0x7f6d1c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51086/0 (socket says 172.21.15.107:51086) 2023-11-24T08:12:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 -- 172.21.15.107:0/1385035042 learned_addr learned my addr 172.21.15.107:0/1385035042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 -- 172.21.15.107:0/1385035042 >> v2:172.21.15.120:3300/0 conn(0x7f6d1c15c7f0 msgr2=0x7f6d1c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 --2- 172.21.15.107:0/1385035042 >> v2:172.21.15.120:3300/0 conn(0x7f6d1c15c7f0 0x7f6d1c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 -- 172.21.15.107:0/1385035042 >> v1:172.21.15.120:6789/0 conn(0x7f6d1c0cdfb0 legacy=0x7f6d1c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.231+0000 7f6d13fff640 1 -- 172.21.15.107:0/1385035042 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c06d040 con 0x7f6d1c0cf450 2023-11-24T08:12:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.232+0000 7f6d13fff640 1 --2- 172.21.15.107:0/1385035042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cf450 0x7f6d1c0cf900 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0d0130 tx=0x7f6d0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6613d683a882fd76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d1b7fe640 1 -- 172.21.15.107:0/1385035042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04031870 con 0x7f6d1c0cf450 2023-11-24T08:12:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d1b7fe640 1 -- 172.21.15.107:0/1385035042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d040319d0 con 0x7f6d1c0cf450 2023-11-24T08:12:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d1b7fe640 1 -- 172.21.15.107:0/1385035042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04031d60 con 0x7f6d1c0cf450 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 -- 172.21.15.107:0/1385035042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cf450 msgr2=0x7f6d1c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 --2- 172.21.15.107:0/1385035042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cf450 0x7f6d1c0cf900 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0d0130 tx=0x7f6d0402b910 comp rx=0 tx=0).stop 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 -- 172.21.15.107:0/1385035042 shutdown_connections 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 --2- 172.21.15.107:0/1385035042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cf450 0x7f6d1c0cf900 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 --2- 172.21.15.107:0/1385035042 >> v2:172.21.15.120:3300/0 conn(0x7f6d1c15c7f0 0x7f6d1c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 -- 172.21.15.107:0/1385035042 >> 172.21.15.107:0/1385035042 conn(0x7f6d1c06a920 msgr2=0x7f6d1c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.233+0000 7f6d22705640 1 -- 172.21.15.107:0/1385035042 shutdown_connections 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.234+0000 7f6d22705640 1 -- 172.21.15.107:0/1385035042 wait complete. 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.235+0000 7f6d22705640 1 Processor -- start 2023-11-24T08:12:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.236+0000 7f6d22705640 1 -- start start 2023-11-24T08:12:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d1c0cf450 0x7f6d1c1443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d1c15c7f0 0x7f6d1c154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c154000 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0ae0e0 con 0x7f6d1c15c7f0 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d22705640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0abfc0 con 0x7f6d1c0cf450 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51092/0 (socket says 172.21.15.107:51092) 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.237+0000 7f6d1bfff640 1 -- 172.21.15.107:0/3544659564 learned_addr learned my addr 172.21.15.107:0/3544659564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d20ca2640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d1c15c7f0 0x7f6d1c154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 -- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d1c0cf450 msgr2=0x7f6d1c1443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d1c0cf450 0x7f6d1c1443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 -- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d1c15c7f0 msgr2=0x7f6d1c154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d1c15c7f0 0x7f6d1c154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 -- 172.21.15.107:0/3544659564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040094d0 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d1bfff640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b410 tx=0x7f6d0c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011830 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c0119b0 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011d60 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.238+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c0ac2a0 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.239+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c0ac860 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.240+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c15b540 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.241+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f6d0c01a400 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.242+0000 7f6d197fa640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 0x7f6cfc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.242+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d0c0883e0 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.242+0000 7f6d13fff640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 0x7f6cfc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.243+0000 7f6d13fff640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 0x7f6cfc06ded0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6d04036000 tx=0x7f6d04030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.248+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d0c054c80 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.478+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6d1c067470 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.479+0000 7f6d197fa640 1 -- 172.21.15.107:0/3544659564 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6d0c015070 con 0x7f6d1c0cdfb0 2023-11-24T08:12:13.481 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:12:13.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 msgr2=0x7f6cfc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:13.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 0x7f6cfc06ded0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6d04036000 tx=0x7f6d04030000 comp rx=0 tx=0).stop 2023-11-24T08:12:13.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 msgr2=0x7f6d1c0ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b410 tx=0x7f6d0c00b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 shutdown_connections 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6cfc06ba10 0x7f6cfc06ded0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d1c15c7f0 0x7f6d1c154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d1c0cf450 0x7f6d1c1443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 --2- 172.21.15.107:0/3544659564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d1c0cdfb0 0x7f6d1c0ad9e0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 >> 172.21.15.107:0/3544659564 conn(0x7f6d1c06a920 msgr2=0x7f6d1c0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 shutdown_connections 2023-11-24T08:12:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:13.482+0000 7f6d22705640 1 -- 172.21.15.107:0/3544659564 wait complete. 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:12:13.503 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:12:13.504 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:12:13.504 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:12:13.504 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:12:13.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.504 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:12:13.566 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:12:13.566 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:13.567 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41935sec preferred_lft 41935sec 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.568 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:12:13.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.569 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:12:13.660 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:12:13.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.665 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:12:13.721 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:12:13.722 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:12:13.722 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:12:13.722 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:12:13.722 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:12:13.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.722 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:12:13.723 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:12:13.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:12:13.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:12:13.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:12:13.723 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:12:13.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.901 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:12:13.951 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:12:13.951 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:12:13.951 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:12:13.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:13.952 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:12:14.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:12:14.040 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:12:14.096 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:12:14.096 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:12:14.096 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:12:14.149 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:12:14.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:14.150 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:14.288 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:14.288 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:12:14.323 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:12:14.323 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:12:14.324 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:12:14.324 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:12:14.324 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:12:14.326 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:12:14.326 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:12:14.327 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:12:14.327 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:12:14.327 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:12:14.331 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:12:14.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:14.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:12:14.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:14.414 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:12:14.469 INFO:teuthology.orchestra.run.smithi120.stdout:client5294 2023-11-24T08:12:14.469 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:14.469 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:14.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:12:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.742+0000 7f98f5b89640 1 Processor -- start 2023-11-24T08:12:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.743+0000 7f98f5b89640 1 -- start start 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98f5b89640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f98f00d19f0 0x7f98f00d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98f5b89640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f00b1010 con 0x7f98f00d0680 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98f5b89640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f00d1230 con 0x7f98f00b2700 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98f5b89640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f00d2310 con 0x7f98f00d19f0 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98effff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98f00d0680 0x7f98f00ce650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39202/0 (socket says 172.21.15.107:39202) 2023-11-24T08:12:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.744+0000 7f98effff640 1 -- 172.21.15.107:0/3730640929 learned_addr learned my addr 172.21.15.107:0/3730640929 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450819830 0 0) 0x7f98f00b1010 con 0x7f98f00d0680 2023-11-24T08:12:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d8003520 con 0x7f98f00d0680 2023-11-24T08:12:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ef7fe640 1 --2- 172.21.15.107:0/3730640929 >> v2:172.21.15.120:3300/0 conn(0x7f98f00d19f0 0x7f98f00d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602145858 0 0) 0x7f98d8003520 con 0x7f98f00d0680 2023-11-24T08:12:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 >> v2:172.21.15.120:3300/0 conn(0x7f98f00d19f0 msgr2=0x7f98f00d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 --2- 172.21.15.107:0/3730640929 >> v2:172.21.15.120:3300/0 conn(0x7f98f00d19f0 0x7f98f00d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 >> v1:172.21.15.120:6789/0 conn(0x7f98f00b2700 legacy=0x7f98f00cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f00d3b40 con 0x7f98f00d0680 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f98e8001670 con 0x7f98f00d0680 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 >> v1:172.21.15.107:6789/0 conn(0x7f98f00d0680 legacy=0x7f98f00ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8003970 0x7f98d8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 0x7f98d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8005180 0x7f98d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00d2310 con 0x7f98d80047c0 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d800c0b0 con 0x7f98d8005180 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.745+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d800c270 con 0x7f98d8003970 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 0x7f98d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98effff640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8005180 0x7f98d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 -- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8003970 msgr2=0x7f98d8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8003970 0x7f98d8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 -- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8005180 msgr2=0x7f98d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8005180 0x7f98d800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98eeffd640 1 -- 172.21.15.107:0/3730640929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8003520 con 0x7f98d80047c0 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98effff640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8005180 0x7f98d800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.746+0000 7f98effff640 1 -- 172.21.15.107:0/3730640929 reap_dead start 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.747+0000 7f98eeffd640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 0x7f98d8004c40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f98e4003d80 tx=0x7f98d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9039a7a63176c777 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.747+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4032600 con 0x7f98d80047c0 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.747+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e402e050 con 0x7f98d80047c0 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.747+0000 7f98ee7fc640 1 -- 172.21.15.107:0/3730640929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e40329e0 con 0x7f98d80047c0 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 -- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 msgr2=0x7f98d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:14.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 0x7f98d8004c40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f98e4003d80 tx=0x7f98d80057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 -- 172.21.15.107:0/3730640929 shutdown_connections 2023-11-24T08:12:14.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 --2- 172.21.15.107:0/3730640929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d80047c0 0x7f98d8004c40 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 -- 172.21.15.107:0/3730640929 >> 172.21.15.107:0/3730640929 conn(0x7f98f008af60 msgr2=0x7f98f00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:14.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 -- 172.21.15.107:0/3730640929 shutdown_connections 2023-11-24T08:12:14.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.748+0000 7f98f5b89640 1 -- 172.21.15.107:0/3730640929 wait complete. 2023-11-24T08:12:14.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.750+0000 7f98f5b89640 1 Processor -- start 2023-11-24T08:12:14.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.750+0000 7f98f5b89640 1 -- start start 2023-11-24T08:12:14.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d80047c0 0x7f98f015f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 0x7f98f01f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00c6cb0 con 0x7f98f00b2e30 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01f6c20 con 0x7f98d80047c0 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98f5b89640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01f6e80 con 0x7f98f015b9c0 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51102/0 (socket says 172.21.15.107:51102) 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.751+0000 7f98effff640 1 -- 172.21.15.107:0/1277688039 learned_addr learned my addr 172.21.15.107:0/1277688039 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98ef7fe640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d80047c0 0x7f98f015f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98eeffd640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 0x7f98f01f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 -- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 msgr2=0x7f98f01f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 0x7f98f01f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 -- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d80047c0 msgr2=0x7f98f015f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d80047c0 0x7f98f015f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 -- 172.21.15.107:0/1277688039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f01f7160 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98eeffd640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 0x7f98f01f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98effff640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f98f008d360 tx=0x7f98e800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8007230 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8007390 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80047e0 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.752+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f01f7440 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.753+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f00b32b0 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.754+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f00c7140 con 0x7f98f00b2e30 2023-11-24T08:12:14.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.755+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f98e8004940 con 0x7f98f00b2e30 2023-11-24T08:12:14.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.755+0000 7f98d7fff640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 0x7f98dc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:14.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.756+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98e8087d40 con 0x7f98f00b2e30 2023-11-24T08:12:14.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.756+0000 7f98ef7fe640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 0x7f98dc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:14.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.756+0000 7f98ef7fe640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 0x7f98dc06ded0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f98e0004b30 tx=0x7f98e000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:14.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.760+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98f00c7140 con 0x7f98f00b2e30 2023-11-24T08:12:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:14.989+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f98f00c7140 con 0x7f98f00b2e30 2023-11-24T08:12:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.889+0000 7f98d7fff640 1 -- 172.21.15.107:0/1277688039 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 136+0+0 (secure 0 0 0) 0x7f98f00c7140 con 0x7f98f00b2e30 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 msgr2=0x7f98dc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 0x7f98dc06ded0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f98e0004b30 tx=0x7f98e000b040 comp rx=0 tx=0).stop 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 msgr2=0x7f98f00b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f98f008d360 tx=0x7f98e800dc00 comp rx=0 tx=0).stop 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 shutdown_connections 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98dc06ba10 0x7f98dc06ded0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f00b2e30 0x7f98f00b8220 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98f015b9c0 0x7f98f01f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 --2- 172.21.15.107:0/1277688039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d80047c0 0x7f98f015f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.893+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 >> 172.21.15.107:0/1277688039 conn(0x7f98f008af60 msgr2=0x7f98f00cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.894+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 shutdown_connections 2023-11-24T08:12:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:15.894+0000 7f98f5b89640 1 -- 172.21.15.107:0/1277688039 wait complete. 2023-11-24T08:12:15.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:12:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.125+0000 7f0a4b2ee640 1 Processor -- start 2023-11-24T08:12:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.126+0000 7f0a4b2ee640 1 -- start start 2023-11-24T08:12:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4b2ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a4415d720 0x7f0a440d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4b2ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a440b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4b2ee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4415db00 con 0x7f0a440b1400 2023-11-24T08:12:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4b2ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440d3d20 con 0x7f0a4415d720 2023-11-24T08:12:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4b2ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440d0ce0 con 0x7f0a440d3870 2023-11-24T08:12:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4908a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a4415d720 0x7f0a440d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4908a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a4415d720 0x7f0a440d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51112/0 (socket says 172.21.15.107:51112) 2023-11-24T08:12:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.127+0000 7f0a4908a640 1 -- 172.21.15.107:0/2568968891 learned_addr learned my addr 172.21.15.107:0/2568968891 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.128+0000 7f0a48889640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a440b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.128+0000 7f0a4908a640 1 -- 172.21.15.107:0/2568968891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 msgr2=0x7f0a440b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.128+0000 7f0a4908a640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a440b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.128+0000 7f0a4908a640 1 -- 172.21.15.107:0/2568968891 >> v1:172.21.15.120:6789/0 conn(0x7f0a440b1400 legacy=0x7f0a440b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.128+0000 7f0a4908a640 1 -- 172.21.15.107:0/2568968891 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4406d400 con 0x7f0a4415d720 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.129+0000 7f0a4908a640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a4415d720 0x7f0a440d3330 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009e50 tx=0x7f0a3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179e5150d7604e4b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a43fff640 1 -- 172.21.15.107:0/2568968891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3802ca70 con 0x7f0a4415d720 2023-11-24T08:12:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a43fff640 1 -- 172.21.15.107:0/2568968891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3802cbd0 con 0x7f0a4415d720 2023-11-24T08:12:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a43fff640 1 -- 172.21.15.107:0/2568968891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a380343f0 con 0x7f0a4415d720 2023-11-24T08:12:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/2568968891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a4415d720 msgr2=0x7f0a440d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a4415d720 0x7f0a440d3330 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009e50 tx=0x7f0a3802bed0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/2568968891 shutdown_connections 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a440b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/2568968891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a4415d720 0x7f0a440d3330 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/2568968891 >> 172.21.15.107:0/2568968891 conn(0x7f0a4408af60 msgr2=0x7f0a4409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.130+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/2568968891 shutdown_connections 2023-11-24T08:12:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.131+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/2568968891 wait complete. 2023-11-24T08:12:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.132+0000 7f0a4b2ee640 1 Processor -- start 2023-11-24T08:12:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.132+0000 7f0a4b2ee640 1 -- start start 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a4415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a4415d720 0x7f0a440b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440cbc40 con 0x7f0a440b1400 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440cd3a0 con 0x7f0a4415d720 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.133+0000 7f0a4b2ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440cb030 con 0x7f0a440d3870 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51120/0 (socket says 172.21.15.107:51120) 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 -- 172.21.15.107:0/604331216 learned_addr learned my addr 172.21.15.107:0/604331216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 -- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 msgr2=0x7f0a4415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a4415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 -- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a4415d720 msgr2=0x7f0a440b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a4415d720 0x7f0a440b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 -- 172.21.15.107:0/604331216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38009b30 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.134+0000 7f0a4908a640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f0a38036040 tx=0x7f0a38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.135+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38009510 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.135+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3803e450 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.135+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3803e800 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.135+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a440cb310 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.135+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a440cb8d0 con 0x7f0a440b1400 2023-11-24T08:12:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.136+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a440b2c20 con 0x7f0a440b1400 2023-11-24T08:12:16.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.137+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f0a3803e960 con 0x7f0a440b1400 2023-11-24T08:12:16.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.138+0000 7f0a41ffb640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 0x7f0a2806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.138+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a380aadc0 con 0x7f0a440b1400 2023-11-24T08:12:16.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.138+0000 7f0a48889640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 0x7f0a2806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.139+0000 7f0a48889640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 0x7f0a2806ded0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0a30006fd0 tx=0x7f0a30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.143+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a380779e0 con 0x7f0a440b1400 2023-11-24T08:12:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.350+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a440b06c0 con 0x7f0a2806ba10 2023-11-24T08:12:16.354 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:12:16.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:16.355 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:12:16.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:16.355 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:12:16.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.353+0000 7f0a41ffb640 1 -- 172.21.15.107:0/604331216 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0a3000be20 con 0x7f0a2806ba10 2023-11-24T08:12:16.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 msgr2=0x7f0a2806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 0x7f0a2806ded0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0a30006fd0 tx=0x7f0a30008040 comp rx=0 tx=0).stop 2023-11-24T08:12:16.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 msgr2=0x7f0a440cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f0a38036040 tx=0x7f0a38004a40 comp rx=0 tx=0).stop 2023-11-24T08:12:16.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 shutdown_connections 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a2806ba10 0x7f0a2806ded0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a4415d720 0x7f0a440b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a440d3870 0x7f0a4415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 --2- 172.21.15.107:0/604331216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a440b1400 0x7f0a440cce60 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 >> 172.21.15.107:0/604331216 conn(0x7f0a4408af60 msgr2=0x7f0a440b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 shutdown_connections 2023-11-24T08:12:16.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.355+0000 7f0a4b2ee640 1 -- 172.21.15.107:0/604331216 wait complete. 2023-11-24T08:12:16.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:12:16.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.580+0000 7f811da09640 1 Processor -- start 2023-11-24T08:12:16.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.580+0000 7f811da09640 1 -- start start 2023-11-24T08:12:16.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.582+0000 7f811da09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118141b30 0x7f811806c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.582+0000 7f811da09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f811815ef00 0x7f8118071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.582+0000 7f811da09640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81180ccdf0 con 0x7f8118071c40 2023-11-24T08:12:16.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.582+0000 7f811da09640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811815f470 con 0x7f811815ef00 2023-11-24T08:12:16.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.582+0000 7f811da09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811815f750 con 0x7f8118141b30 2023-11-24T08:12:16.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f811815ef00 0x7f8118071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f811815ef00 0x7f8118071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51132/0 (socket says 172.21.15.107:51132) 2023-11-24T08:12:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 -- 172.21.15.107:0/2546654211 learned_addr learned my addr 172.21.15.107:0/2546654211 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f811ca07640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118141b30 0x7f811806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 -- 172.21.15.107:0/2546654211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118141b30 msgr2=0x7f811806c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118141b30 0x7f811806c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 -- 172.21.15.107:0/2546654211 >> v1:172.21.15.120:6789/0 conn(0x7f8118071c40 legacy=0x7f81180cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:16.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.583+0000 7f8117fff640 1 -- 172.21.15.107:0/2546654211 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811815fa30 con 0x7f811815ef00 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.584+0000 7f8117fff640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f8118071700 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8104009870 tx=0x7f810402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33624b7aa70a38ea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.585+0000 7f81177fe640 1 -- 172.21.15.107:0/2546654211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104031740 con 0x7f811815ef00 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.585+0000 7f81177fe640 1 -- 172.21.15.107:0/2546654211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81040318a0 con 0x7f811815ef00 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.585+0000 7f81177fe640 1 -- 172.21.15.107:0/2546654211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104031c30 con 0x7f811815ef00 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.585+0000 7f811da09640 1 -- 172.21.15.107:0/2546654211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 msgr2=0x7f8118071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.585+0000 7f811da09640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f8118071700 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8104009870 tx=0x7f810402b7e0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 -- 172.21.15.107:0/2546654211 shutdown_connections 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f8118071700 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 --2- 172.21.15.107:0/2546654211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118141b30 0x7f811806c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 -- 172.21.15.107:0/2546654211 >> 172.21.15.107:0/2546654211 conn(0x7f811806a500 msgr2=0x7f811806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 -- 172.21.15.107:0/2546654211 shutdown_connections 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.586+0000 7f811da09640 1 -- 172.21.15.107:0/2546654211 wait complete. 2023-11-24T08:12:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.587+0000 7f811da09640 1 Processor -- start 2023-11-24T08:12:16.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 -- start start 2023-11-24T08:12:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118071c40 0x7f81180a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8118141b30 0x7f81181f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811815c0f0 con 0x7f811815ef00 2023-11-24T08:12:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811813fef0 con 0x7f8118141b30 2023-11-24T08:12:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.588+0000 7f811da09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81181401d0 con 0x7f8118071c40 2023-11-24T08:12:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51148/0 (socket says 172.21.15.107:51148) 2023-11-24T08:12:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 -- 172.21.15.107:0/4099904543 learned_addr learned my addr 172.21.15.107:0/4099904543 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 -- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118071c40 msgr2=0x7f81180a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118071c40 0x7f81180a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 -- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8118141b30 msgr2=0x7f81181f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8118141b30 0x7f81181f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 -- 172.21.15.107:0/4099904543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040094d0 con 0x7f811815ef00 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.589+0000 7f811d208640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f811000c700 tx=0x7f811000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.590+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811000aa50 con 0x7f811815ef00 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.590+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811000abb0 con 0x7f811815ef00 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.590+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811000ce10 con 0x7f811815ef00 2023-11-24T08:12:16.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.590+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81181404b0 con 0x7f811815ef00 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.590+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8118140a70 con 0x7f811815ef00 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.591+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81180aa400 con 0x7f811815ef00 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.592+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7f8110012420 con 0x7f811815ef00 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.593+0000 7f81157fa640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 0x7f80fc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.593+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81100886a0 con 0x7f811815ef00 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.593+0000 7f811ca07640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 0x7f80fc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:16.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.594+0000 7f811ca07640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 0x7f80fc06ded0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098f0 tx=0x7f810c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.597+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8110054fa0 con 0x7f811815ef00 2023-11-24T08:12:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.787+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f811804e840 con 0x7f811815ef00 2023-11-24T08:12:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.787+0000 7f81157fa640 1 -- 172.21.15.107:0/4099904543 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8110003eb0 con 0x7f811815ef00 2023-11-24T08:12:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 msgr2=0x7f80fc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 0x7f80fc06ded0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098f0 tx=0x7f810c008040 comp rx=0 tx=0).stop 2023-11-24T08:12:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 msgr2=0x7f81181f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f811000c700 tx=0x7f811000cbc0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 shutdown_connections 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f80fc06ba10 0x7f80fc06ded0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811815ef00 0x7f81181f2740 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8118141b30 0x7f81181f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 --2- 172.21.15.107:0/4099904543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8118071c40 0x7f81180a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 >> 172.21.15.107:0/4099904543 conn(0x7f811806a500 msgr2=0x7f81180cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.790+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 shutdown_connections 2023-11-24T08:12:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:16.791+0000 7f811da09640 1 -- 172.21.15.107:0/4099904543 wait complete. 2023-11-24T08:12:16.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol1183 --mode=777 2023-11-24T08:12:17.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.031+0000 7fc16cd47640 1 Processor -- start 2023-11-24T08:12:17.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.032+0000 7fc16cd47640 1 -- start start 2023-11-24T08:12:17.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc16cd47640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1680b2ad0 0x7fc16806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc16cd47640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1680b2f60 0x7fc16806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc16cd47640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc168143c10 con 0x7fc1681431f0 2023-11-24T08:12:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc16cd47640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680b4fd0 con 0x7fc1680b2f60 2023-11-24T08:12:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc16cd47640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680705b0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.033+0000 7fc167fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1680b2f60 0x7fc16806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1680b2f60 0x7fc16806dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51154/0 (socket says 172.21.15.107:51154) 2023-11-24T08:12:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 -- 172.21.15.107:0/1613563276 learned_addr learned my addr 172.21.15.107:0/1613563276 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 -- 172.21.15.107:0/1613563276 >> v2:172.21.15.120:3300/0 conn(0x7fc1680b2ad0 msgr2=0x7fc16806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 --2- 172.21.15.107:0/1613563276 >> v2:172.21.15.120:3300/0 conn(0x7fc1680b2ad0 0x7fc16806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 -- 172.21.15.107:0/1613563276 >> v1:172.21.15.120:6789/0 conn(0x7fc1681431f0 legacy=0x7fc16815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.034+0000 7fc167fff640 1 -- 172.21.15.107:0/1613563276 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168070890 con 0x7fc1680b2f60 2023-11-24T08:12:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.035+0000 7fc167fff640 1 --2- 172.21.15.107:0/1613563276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2f60 0x7fc16806dbe0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc1680cd7e0 tx=0x7fc16002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d45b79fa32374f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.035+0000 7fc166ffd640 1 -- 172.21.15.107:0/1613563276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16002c9f0 con 0x7fc1680b2f60 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.035+0000 7fc166ffd640 1 -- 172.21.15.107:0/1613563276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16002cb50 con 0x7fc1680b2f60 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.035+0000 7fc166ffd640 1 -- 172.21.15.107:0/1613563276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1600343f0 con 0x7fc1680b2f60 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 -- 172.21.15.107:0/1613563276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2f60 msgr2=0x7fc16806dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 --2- 172.21.15.107:0/1613563276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2f60 0x7fc16806dbe0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc1680cd7e0 tx=0x7fc16002be50 comp rx=0 tx=0).stop 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 -- 172.21.15.107:0/1613563276 shutdown_connections 2023-11-24T08:12:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 --2- 172.21.15.107:0/1613563276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2f60 0x7fc16806dbe0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 --2- 172.21.15.107:0/1613563276 >> v2:172.21.15.120:3300/0 conn(0x7fc1680b2ad0 0x7fc16806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 -- 172.21.15.107:0/1613563276 >> 172.21.15.107:0/1613563276 conn(0x7fc1680cb8a0 msgr2=0x7fc1680cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 -- 172.21.15.107:0/1613563276 shutdown_connections 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.036+0000 7fc16cd47640 1 -- 172.21.15.107:0/1613563276 wait complete. 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.038+0000 7fc16cd47640 1 Processor -- start 2023-11-24T08:12:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.038+0000 7fc16cd47640 1 -- start start 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1680b2f60 0x7fc168152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1681431f0 0x7fc16813c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16813ce20 con 0x7fc1680b2ad0 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681f1120 con 0x7fc1680b2f60 2023-11-24T08:12:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc16cd47640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681f13c0 con 0x7fc1681431f0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51156/0 (socket says 172.21.15.107:51156) 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 -- 172.21.15.107:0/2766921019 learned_addr learned my addr 172.21.15.107:0/2766921019 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 -- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1681431f0 msgr2=0x7fc16813c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1681431f0 0x7fc16813c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 -- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1680b2f60 msgr2=0x7fc168152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc15ffff640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1680b2f60 0x7fc168152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1680b2f60 0x7fc168152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.039+0000 7fc1677fe640 1 -- 172.21.15.107:0/2766921019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160009b30 con 0x7fc1680b2ad0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc1677fe640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc15800b3e0 tx=0x7fc15800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158011830 con 0x7fc1680b2ad0 2023-11-24T08:12:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1580119b0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1681f16a0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158011d60 con 0x7fc1680b2ad0 2023-11-24T08:12:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.040+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1681534f0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.041+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1681f1bc0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.042+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fc15801a400 con 0x7fc1680b2ad0 2023-11-24T08:12:17.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.043+0000 7fc164ff9640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 0x7fc14806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.043+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc158088810 con 0x7fc1680b2ad0 2023-11-24T08:12:17.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.044+0000 7fc15ffff640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 0x7fc14806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.044+0000 7fc15ffff640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 0x7fc14806ded0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc1680cfd60 tx=0x7fc150008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.047+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1580550f0 con 0x7fc1680b2ad0 2023-11-24T08:12:17.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.246+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol1183", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fc168071130 con 0x7fc14806ba10 2023-11-24T08:12:17.259 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:17.257+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:17.259 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:17.257+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:17.259 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:17.257+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:17.259 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:17.257+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:17.259 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:17.257+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.304+0000 7fc164ff9640 1 -- 172.21.15.107:0/2766921019 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc15000adc0 con 0x7fc14806ba10 2023-11-24T08:12:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.307+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 msgr2=0x7fc14806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.307+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 0x7fc14806ded0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc1680cfd60 tx=0x7fc150008040 comp rx=0 tx=0).stop 2023-11-24T08:12:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 msgr2=0x7fc168155580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc15800b3e0 tx=0x7fc15800b8a0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 shutdown_connections 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14806ba10 0x7fc14806ded0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1681431f0 0x7fc16813c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1680b2f60 0x7fc168152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 --2- 172.21.15.107:0/2766921019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b2ad0 0x7fc168155580 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 >> 172.21.15.107:0/2766921019 conn(0x7fc1680cb8a0 msgr2=0x7fc168159b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 shutdown_connections 2023-11-24T08:12:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.308+0000 7fc16cd47640 1 -- 172.21.15.107:0/2766921019 wait complete. 2023-11-24T08:12:17.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1183 2023-11-24T08:12:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.527+0000 7fa88391d640 1 Processor -- start 2023-11-24T08:12:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.527+0000 7fa88391d640 1 -- start start 2023-11-24T08:12:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.528+0000 7fa88391d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa87c15b440 0x7fa87c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.528+0000 7fa88391d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa87c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.528+0000 7fa88391d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87c15eea0 con 0x7fa87c0c57e0 2023-11-24T08:12:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.528+0000 7fa88391d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87c0cc3d0 con 0x7fa87c15b440 2023-11-24T08:12:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.528+0000 7fa88391d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c15b820 con 0x7fa87c0cbf20 2023-11-24T08:12:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa881eba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa87c0c57e0 0x7fa87c0c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39212/0 (socket says 172.21.15.107:39212) 2023-11-24T08:12:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa881eba640 1 -- 172.21.15.107:0/2650017874 learned_addr learned my addr 172.21.15.107:0/2650017874 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112870101 0 0) 0x7fa87c15eea0 con 0x7fa87c0c57e0 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa880eb8640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa87c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2650017874 >> v2:172.21.15.120:3300/0 conn(0x7fa87c15b440 0x7fa87c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa86c003520 con 0x7fa87c0c57e0 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.529+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218929086 0 0) 0x7fa86c003520 con 0x7fa87c0c57e0 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 msgr2=0x7fa87c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa87c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 >> v2:172.21.15.120:3300/0 conn(0x7fa87c15b440 msgr2=0x7fa87c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 --2- 172.21.15.107:0/2650017874 >> v2:172.21.15.120:3300/0 conn(0x7fa87c15b440 0x7fa87c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c06d600 con 0x7fa87c0c57e0 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa874001670 con 0x7fa87c0c57e0 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa880eb8640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa87c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 >> v1:172.21.15.107:6789/0 conn(0x7fa87c0c57e0 legacy=0x7fa87c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa86c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa86c003f60 0x7fa86c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa86c004d00 0x7fa86c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0cc3d0 con 0x7fa86c004d00 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c15b820 con 0x7fa86c003f60 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c00bae0 con 0x7fa87c0cbf20 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2650017874 >> v2:172.21.15.120:3300/0 conn(0x7fa87c15b440 0x7fa87c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa881eba640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa86c004d00 0x7fa86c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa880eb8640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa86c003f60 0x7fa86c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa86c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa881eba640 1 -- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 msgr2=0x7fa86c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa881eba640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa87c0cbf20 0x7fa86c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa881eba640 1 -- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa86c003f60 msgr2=0x7fa86c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.530+0000 7fa881eba640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa86c003f60 0x7fa86c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.531+0000 7fa881eba640 1 -- 172.21.15.107:0/2650017874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c003520 con 0x7fa86c004d00 2023-11-24T08:12:17.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa881eba640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa86c004d00 0x7fa86c005180 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c210 tx=0x7fa86c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cf11582bbe18de server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87402cae0 con 0x7fa86c004d00 2023-11-24T08:12:17.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87402cc40 con 0x7fa86c004d00 2023-11-24T08:12:17.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa87bfff640 1 -- 172.21.15.107:0/2650017874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa874005af0 con 0x7fa86c004d00 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa88391d640 1 -- 172.21.15.107:0/2650017874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa86c004d00 msgr2=0x7fa86c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa88391d640 1 --2- 172.21.15.107:0/2650017874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa86c004d00 0x7fa86c005180 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c210 tx=0x7fa86c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.532+0000 7fa881eba640 1 -- 172.21.15.107:0/2650017874 reap_dead start 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.533+0000 7fa88391d640 1 -- 172.21.15.107:0/2650017874 shutdown_connections 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.533+0000 7fa88391d640 1 -- 172.21.15.107:0/2650017874 >> 172.21.15.107:0/2650017874 conn(0x7fa87c0b0c50 msgr2=0x7fa87c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.533+0000 7fa88391d640 1 -- 172.21.15.107:0/2650017874 shutdown_connections 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.533+0000 7fa88391d640 1 -- 172.21.15.107:0/2650017874 wait complete. 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.534+0000 7fa88391d640 1 Processor -- start 2023-11-24T08:12:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.535+0000 7fa88391d640 1 -- start start 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.535+0000 7fa88391d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa86c004d00 0x7fa87c1f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.535+0000 7fa88391d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa87c156c00 0x7fa87c1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa88391d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa88391d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c1f1d30 con 0x7fa87c1f48f0 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa88391d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c1eea00 con 0x7fa87c156c00 2023-11-24T08:12:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa88391d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c09cd10 con 0x7fa86c004d00 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51200/0 (socket says 172.21.15.107:51200) 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 -- 172.21.15.107:0/2631844779 learned_addr learned my addr 172.21.15.107:0/2631844779 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa86c004d00 0x7fa87c1f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa880eb8640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa87c156c00 0x7fa87c1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 -- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa86c004d00 msgr2=0x7fa87c1f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa86c004d00 0x7fa87c1f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 -- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa87c156c00 msgr2=0x7fa87c1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa87c156c00 0x7fa87c1f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.536+0000 7fa881eba640 1 -- 172.21.15.107:0/2631844779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c1f1f80 con 0x7fa87c1f48f0 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa881eba640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa874001200 tx=0x7fa874001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87403c040 con 0x7fa87c1f48f0 2023-11-24T08:12:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa874001630 con 0x7fa87c1f48f0 2023-11-24T08:12:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c1f2240 con 0x7fa87c1f48f0 2023-11-24T08:12:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa874007270 con 0x7fa87c1f48f0 2023-11-24T08:12:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.537+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c1f2800 con 0x7fa87c1f48f0 2023-11-24T08:12:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.538+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c04e8b0 con 0x7fa87c1f48f0 2023-11-24T08:12:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.539+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) v1 ==== 90302+0+0 (secure 0 0 0) 0x7fa874032070 con 0x7fa87c1f48f0 2023-11-24T08:12:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.541+0000 7fa879ffb640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 0x7fa86006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.541+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8740a9510 con 0x7fa87c1f48f0 2023-11-24T08:12:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.541+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 0x7fa86006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.542+0000 7fa8816b9640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 0x7fa86006ded0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa870009ac0 tx=0x7fa87000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.545+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa874075db0 con 0x7fa87c1f48f0 2023-11-24T08:12:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.762+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1183", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87c09ce50 con 0x7fa86006ba10 2023-11-24T08:12:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.770+0000 7fa879ffb640 1 -- 172.21.15.107:0/2631844779 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fa87c09ce50 con 0x7fa86006ba10 2023-11-24T08:12:17.772 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol1183/fc363f32-c50a-41cf-9a0a-aa62fd77879a 2023-11-24T08:12:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.772+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 msgr2=0x7fa86006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.772+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 0x7fa86006ded0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa870009ac0 tx=0x7fa87000b040 comp rx=0 tx=0).stop 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.772+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 msgr2=0x7fa87c1f4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.772+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa874001200 tx=0x7fa874001a00 comp rx=0 tx=0).stop 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 shutdown_connections 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa86006ba10 0x7fa86006ded0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa87c1f48f0 0x7fa87c1f4d70 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa87c156c00 0x7fa87c1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 --2- 172.21.15.107:0/2631844779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa86c004d00 0x7fa87c1f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 >> 172.21.15.107:0/2631844779 conn(0x7fa87c0b0c50 msgr2=0x7fa87c071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 shutdown_connections 2023-11-24T08:12:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:17.773+0000 7fa88391d640 1 -- 172.21.15.107:0/2631844779 wait complete. 2023-11-24T08:12:17.792 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol1183 with 1 files each 1MB size under directory volumes/_nogroup/subvol1183/fc363f32-c50a-41cf-9a0a-aa62fd77879a 2023-11-24T08:12:17.792 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:17.792 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:17.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:17.843 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1183/fc363f32-c50a-41cf-9a0a-aa62fd77879a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:12:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:12:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:12:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293233 s, 35.8 MB/s 2023-11-24T08:12:17.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol1183 subvol_snap2260 2023-11-24T08:12:18.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.196+0000 7fa04d25c640 1 Processor -- start 2023-11-24T08:12:18.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.196+0000 7fa04d25c640 1 -- start start 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04d25c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa04815d700 0x7fa0480cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04d25c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa048070070 0x7fa0480704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04d25c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa048070ef0 con 0x7fa04806cfc0 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04d25c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa048070a20 con 0x7fa048070070 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04d25c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04806d400 con 0x7fa04815d700 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa048070070 0x7fa0480704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa048070070 0x7fa0480704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57630/0 (socket says 172.21.15.107:57630) 2023-11-24T08:12:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 -- 172.21.15.107:0/689546706 learned_addr learned my addr 172.21.15.107:0/689546706 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:18.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa046d9d640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa04815d700 0x7fa0480cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:18.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 -- 172.21.15.107:0/689546706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa04815d700 msgr2=0x7fa0480cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:18.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa04815d700 0x7fa0480cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 -- 172.21.15.107:0/689546706 >> v1:172.21.15.120:6789/0 conn(0x7fa04806cfc0 legacy=0x7fa0480d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:18.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.198+0000 7fa04759e640 1 -- 172.21.15.107:0/689546706 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04806d650 con 0x7fa048070070 2023-11-24T08:12:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa04759e640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa048070070 0x7fa0480704e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa040009870 tx=0x7fa04002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a51a4e4c2fcf0f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa045d9b640 1 -- 172.21.15.107:0/689546706 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040031640 con 0x7fa048070070 2023-11-24T08:12:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa045d9b640 1 -- 172.21.15.107:0/689546706 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0400317a0 con 0x7fa048070070 2023-11-24T08:12:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa045d9b640 1 -- 172.21.15.107:0/689546706 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040031b30 con 0x7fa048070070 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa04d25c640 1 -- 172.21.15.107:0/689546706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa048070070 msgr2=0x7fa0480704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.200+0000 7fa04d25c640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa048070070 0x7fa0480704e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa040009870 tx=0x7fa04002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 -- 172.21.15.107:0/689546706 shutdown_connections 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa048070070 0x7fa0480704e0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 --2- 172.21.15.107:0/689546706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa04815d700 0x7fa0480cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 -- 172.21.15.107:0/689546706 >> 172.21.15.107:0/689546706 conn(0x7fa04806e8a0 msgr2=0x7fa04806ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 -- 172.21.15.107:0/689546706 shutdown_connections 2023-11-24T08:12:18.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.201+0000 7fa04d25c640 1 -- 172.21.15.107:0/689546706 wait complete. 2023-11-24T08:12:18.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.202+0000 7fa04d25c640 1 Processor -- start 2023-11-24T08:12:18.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.202+0000 7fa04d25c640 1 -- start start 2023-11-24T08:12:18.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa04806cfc0 0x7fa0481e46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa048070070 0x7fa0480b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0481e3410 con 0x7fa04815d700 2023-11-24T08:12:18.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0481572e0 con 0x7fa04806cfc0 2023-11-24T08:12:18.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.203+0000 7fa04d25c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0481e3620 con 0x7fa048070070 2023-11-24T08:12:18.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:18.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57642/0 (socket says 172.21.15.107:57642) 2023-11-24T08:12:18.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 -- 172.21.15.107:0/429600952 learned_addr learned my addr 172.21.15.107:0/429600952 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 -- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa048070070 msgr2=0x7fa0480b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa048070070 0x7fa0480b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 -- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa04806cfc0 msgr2=0x7fa0481e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa046d9d640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa04806cfc0 0x7fa0481e46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa04806cfc0 0x7fa0481e46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 -- 172.21.15.107:0/429600952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0400094d0 con 0x7fa04815d700 2023-11-24T08:12:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.204+0000 7fa04759e640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fa040009870 tx=0x7fa0400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.205+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04003ab10 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.205+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04003ac90 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.205+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0481e3900 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.205+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0400443f0 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.205+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0481e41a0 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.206+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0481e4390 con 0x7fa04815d700 2023-11-24T08:12:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.207+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) v1 ==== 90353+0+0 (secure 0 0 0) 0x7fa04002f050 con 0x7fa04815d700 2023-11-24T08:12:18.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.208+0000 7fa03f7fe640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 0x7fa02806df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.208+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0400aa4c0 con 0x7fa04815d700 2023-11-24T08:12:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.208+0000 7fa046d9d640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 0x7fa02806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.209+0000 7fa046d9d640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 0x7fa02806df60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa038008e50 tx=0x7fa0380083a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:18.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.213+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa040076e60 con 0x7fa04815d700 2023-11-24T08:12:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.434+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol1183", "snap_name": "subvol_snap2260", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0480d0e30 con 0x7fa02806baa0 2023-11-24T08:12:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.467+0000 7fa03f7fe640 1 -- 172.21.15.107:0/429600952 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa03800ddf0 con 0x7fa02806baa0 2023-11-24T08:12:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 msgr2=0x7fa02806df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 0x7fa02806df60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa038008e50 tx=0x7fa0380083a0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 msgr2=0x7fa0481f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fa040009870 tx=0x7fa0400046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 shutdown_connections 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa02806baa0 0x7fa02806df60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04815d700 0x7fa0481f2e80 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa048070070 0x7fa0480b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 --2- 172.21.15.107:0/429600952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa04806cfc0 0x7fa0481e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 >> 172.21.15.107:0/429600952 conn(0x7fa04806e8a0 msgr2=0x7fa04806b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 shutdown_connections 2023-11-24T08:12:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.470+0000 7fa04d25c640 1 -- 172.21.15.107:0/429600952 wait complete. 2023-11-24T08:12:18.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol1183 subvol_snap2260 subvol_clone7064 2023-11-24T08:12:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.698+0000 7f740cf90640 1 Processor -- start 2023-11-24T08:12:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.698+0000 7f740cf90640 1 -- start start 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.699+0000 7f740cf90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74080cb850 0x7f74080cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.699+0000 7f740cf90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.699+0000 7f740cf90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080cc170 con 0x7f74080b40d0 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.699+0000 7f740cf90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7408070870 con 0x7f74080cb850 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.699+0000 7f740cf90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7408070b50 con 0x7f74080d0a00 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f740659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74080cb850 0x7f74080cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f7406d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f74080b40d0 0x7f74080cf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42328/0 (socket says 172.21.15.107:42328) 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f7406d9d640 1 -- 172.21.15.107:0/1237110245 learned_addr learned my addr 172.21.15.107:0/1237110245 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f73fffff640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f740659c640 1 -- 172.21.15.107:0/1237110245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 msgr2=0x7f740806ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f740659c640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f740659c640 1 -- 172.21.15.107:0/1237110245 >> v1:172.21.15.107:6789/0 conn(0x7f74080b40d0 legacy=0x7f74080cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f740659c640 1 -- 172.21.15.107:0/1237110245 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408070e30 con 0x7f74080cb850 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.700+0000 7f73fffff640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.701+0000 7f740659c640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74080cbc30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f73f80097a0 tx=0x7f73f802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d53806814ed295 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.702+0000 7f7405d9b640 1 -- 172.21.15.107:0/1237110245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8031780 con 0x7f74080cb850 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.702+0000 7f7405d9b640 1 -- 172.21.15.107:0/1237110245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80318e0 con 0x7f74080cb850 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.702+0000 7f7405d9b640 1 -- 172.21.15.107:0/1237110245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8031c70 con 0x7f74080cb850 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.702+0000 7f740cf90640 1 -- 172.21.15.107:0/1237110245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 msgr2=0x7f74080cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.702+0000 7f740cf90640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74080cbc30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f73f80097a0 tx=0x7f73f802b820 comp rx=0 tx=0).stop 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 -- 172.21.15.107:0/1237110245 shutdown_connections 2023-11-24T08:12:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 --2- 172.21.15.107:0/1237110245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74080cbc30 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 -- 172.21.15.107:0/1237110245 >> 172.21.15.107:0/1237110245 conn(0x7f740806e7d0 msgr2=0x7f740806ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 -- 172.21.15.107:0/1237110245 shutdown_connections 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.703+0000 7f740cf90640 1 -- 172.21.15.107:0/1237110245 wait complete. 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.705+0000 7f740cf90640 1 Processor -- start 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.705+0000 7f740cf90640 1 -- start start 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74080b40d0 0x7f74080c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080c6d10 con 0x7f74080cb850 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080c76e0 con 0x7f74080b40d0 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.706+0000 7f740cf90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74080c6100 con 0x7f74080d0a00 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57652/0 (socket says 172.21.15.107:57652) 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 -- 172.21.15.107:0/2045140919 learned_addr learned my addr 172.21.15.107:0/2045140919 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f7406d9d640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f740659c640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74080b40d0 0x7f74080c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 -- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 msgr2=0x7f740806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 -- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74080b40d0 msgr2=0x7f74080c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74080b40d0 0x7f74080c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.707+0000 7f73fffff640 1 -- 172.21.15.107:0/2045140919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f8009400 con 0x7f74080cb850 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f73fffff640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f73f000c7f0 tx=0x7f73f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000cea0 con 0x7f74080cb850 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f0007a30 con 0x7f74080cb850 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0007dc0 con 0x7f74080cb850 2023-11-24T08:12:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74080c63e0 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.708+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74080c69a0 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.709+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74080c6ec0 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.710+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) v1 ==== 90353+0+0 (secure 0 0 0) 0x7f73f00113f0 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.711+0000 7f73feffd640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 0x7f73e806df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.711+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73f0088400 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.712+0000 7f740659c640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 0x7f73e806df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.712+0000 7f740659c640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 0x7f73e806df90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73f8036000 tx=0x7f73f803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.716+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73f00bde90 con 0x7f74080cb850 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:18.942+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol1183", "snap_name": "subvol_snap2260", "target_sub_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7f74080cf150 con 0x7f73e806bad0 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.078+0000 7f73feffd640 1 -- 172.21.15.107:0/2045140919 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f73f8031c80 con 0x7f73e806bad0 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 msgr2=0x7f73e806df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 0x7f73e806df90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73f8036000 tx=0x7f73f803d040 comp rx=0 tx=0).stop 2023-11-24T08:12:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 msgr2=0x7f74081508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f73f000c7f0 tx=0x7f73f000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 shutdown_connections 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73e806bad0 0x7f73e806df90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74080d0a00 0x7f740806be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74080cb850 0x7f74081508b0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 --2- 172.21.15.107:0/2045140919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74080b40d0 0x7f74080c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 >> 172.21.15.107:0/2045140919 conn(0x7f740806e7d0 msgr2=0x7f74080b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 shutdown_connections 2023-11-24T08:12:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:19.081+0000 7f740cf90640 1 -- 172.21.15.107:0/2045140919 wait complete. 2023-11-24T08:12:23.505 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.091+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.505 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.091+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.505 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.091+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.505 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.091+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.505 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.091+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7064 2023-11-24T08:12:23.506 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.112+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.506 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.112+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.506 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.112+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.507 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.112+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.507 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:19.112+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.733+0000 7f6e66c21640 1 Processor -- start 2023-11-24T08:12:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.734+0000 7f6e66c21640 1 -- start start 2023-11-24T08:12:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66c21640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6e6015d3f0 0x7f6e600d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66c21640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e6015d7d0 0x7f6e600d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66c21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600d15e0 con 0x7f6e600d0e40 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66c21640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600d0280 con 0x7f6e6015d3f0 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66c21640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600d1300 con 0x7f6e6015d7d0 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e65c1f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6e6015d3f0 0x7f6e600d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66420640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e600d0e40 0x7f6e600afea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42334/0 (socket says 172.21.15.107:42334) 2023-11-24T08:12:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.735+0000 7f6e66420640 1 -- 172.21.15.107:0/1355749862 learned_addr learned my addr 172.21.15.107:0/1355749862 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e6541e640 1 --2- 172.21.15.107:0/1355749862 >> v2:172.21.15.120:3300/0 conn(0x7f6e6015d7d0 0x7f6e600d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e65c1f640 1 -- 172.21.15.107:0/1355749862 >> v2:172.21.15.120:3300/0 conn(0x7f6e6015d7d0 msgr2=0x7f6e600d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e65c1f640 1 --2- 172.21.15.107:0/1355749862 >> v2:172.21.15.120:3300/0 conn(0x7f6e6015d7d0 0x7f6e600d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e65c1f640 1 -- 172.21.15.107:0/1355749862 >> v1:172.21.15.107:6789/0 conn(0x7f6e600d0e40 legacy=0x7f6e600afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e64c1d640 1 -- 172.21.15.107:0/1355749862 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753797793 0 0) 0x7f6e600d15e0 con 0x7f6e600d0e40 2023-11-24T08:12:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.736+0000 7f6e65c1f640 1 -- 172.21.15.107:0/1355749862 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600cfc30 con 0x7f6e6015d3f0 2023-11-24T08:12:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.737+0000 7f6e65c1f640 1 --2- 172.21.15.107:0/1355749862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6015d3f0 0x7f6e600d1830 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f6e60067bd0 tx=0x7f6e5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9f7b1b781c09ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.737+0000 7f6e64c1d640 1 -- 172.21.15.107:0/1355749862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5402cc80 con 0x7f6e6015d3f0 2023-11-24T08:12:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.737+0000 7f6e64c1d640 1 -- 172.21.15.107:0/1355749862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5402cde0 con 0x7f6e6015d3f0 2023-11-24T08:12:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.737+0000 7f6e64c1d640 1 -- 172.21.15.107:0/1355749862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54034660 con 0x7f6e6015d3f0 2023-11-24T08:12:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 -- 172.21.15.107:0/1355749862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6015d3f0 msgr2=0x7f6e600d1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 --2- 172.21.15.107:0/1355749862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6015d3f0 0x7f6e600d1830 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f6e60067bd0 tx=0x7f6e5402bed0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 -- 172.21.15.107:0/1355749862 shutdown_connections 2023-11-24T08:12:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 --2- 172.21.15.107:0/1355749862 >> v2:172.21.15.120:3300/0 conn(0x7f6e6015d7d0 0x7f6e600d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 --2- 172.21.15.107:0/1355749862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6015d3f0 0x7f6e600d1830 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 -- 172.21.15.107:0/1355749862 >> 172.21.15.107:0/1355749862 conn(0x7f6e6009c5b0 msgr2=0x7f6e600d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 -- 172.21.15.107:0/1355749862 shutdown_connections 2023-11-24T08:12:23.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.738+0000 7f6e66c21640 1 -- 172.21.15.107:0/1355749862 wait complete. 2023-11-24T08:12:23.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.740+0000 7f6e66c21640 1 Processor -- start 2023-11-24T08:12:23.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.740+0000 7f6e66c21640 1 -- start start 2023-11-24T08:12:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6015d3f0 0x7f6e600c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6015d7d0 0x7f6e600cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600c8460 con 0x7f6e600d0e40 2023-11-24T08:12:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600c8740 con 0x7f6e6015d7d0 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e66c21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600c8a20 con 0x7f6e6015d3f0 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e65c1f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e65c1f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57702/0 (socket says 172.21.15.107:57702) 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.741+0000 7f6e65c1f640 1 -- 172.21.15.107:0/2106087232 learned_addr learned my addr 172.21.15.107:0/2106087232 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 -- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6015d3f0 msgr2=0x7f6e600c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6015d3f0 0x7f6e600c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 -- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6015d7d0 msgr2=0x7f6e600cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6015d7d0 0x7f6e600cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 -- 172.21.15.107:0/2106087232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54009a70 con 0x7f6e600d0e40 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e65c1f640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009e80 tx=0x7f6e54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.742+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54003990 con 0x7f6e600d0e40 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.743+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54003af0 con 0x7f6e600d0e40 2023-11-24T08:12:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.743+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54003e60 con 0x7f6e600d0e40 2023-11-24T08:12:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.743+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e600c8d00 con 0x7f6e600d0e40 2023-11-24T08:12:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.743+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e600c8130 con 0x7f6e600d0e40 2023-11-24T08:12:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.744+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e600c9220 con 0x7f6e600d0e40 2023-11-24T08:12:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.745+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7f6e54004000 con 0x7f6e600d0e40 2023-11-24T08:12:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.745+0000 7f6e5e7fc640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 0x7f6e4406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.746+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e540ab000 con 0x7f6e600d0e40 2023-11-24T08:12:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.746+0000 7f6e6541e640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 0x7f6e4406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.747+0000 7f6e6541e640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 0x7f6e4406e080 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006fd0 tx=0x7f6e4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.750+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e600c9220 con 0x7f6e600d0e40 2023-11-24T08:12:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.976+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e600d0710 con 0x7f6e4406bbc0 2023-11-24T08:12:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.986+0000 7f6e5e7fc640 1 -- 172.21.15.107:0/2106087232 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f6e4c00be20 con 0x7f6e4406bbc0 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1183", 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap2260" 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:23.989 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 msgr2=0x7f6e4406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 0x7f6e4406e080 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006fd0 tx=0x7f6e4c008040 comp rx=0 tx=0).stop 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 msgr2=0x7f6e600b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009e80 tx=0x7f6e54004a40 comp rx=0 tx=0).stop 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 shutdown_connections 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e4406bbc0 0x7f6e4406e080 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6015d7d0 0x7f6e600cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6015d3f0 0x7f6e600c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 --2- 172.21.15.107:0/2106087232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e600d0e40 0x7f6e600b4db0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 >> 172.21.15.107:0/2106087232 conn(0x7f6e6009c5b0 msgr2=0x7f6e6006c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 shutdown_connections 2023-11-24T08:12:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:23.989+0000 7f6e66c21640 1 -- 172.21.15.107:0/2106087232 wait complete. 2023-11-24T08:12:25.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7064 2023-11-24T08:12:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.229+0000 7f3c3d72c640 1 Processor -- start 2023-11-24T08:12:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.229+0000 7f3c3d72c640 1 -- start start 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.230+0000 7f3c3d72c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 0x7f3c380cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c3d72c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3c380c59b0 0x7f3c3806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c3d72c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380c5dc0 con 0x7f3c380cbf20 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c3d72c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c380cc3d0 con 0x7f3c380c59b0 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c3d72c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3815b820 con 0x7f3c3815b440 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3c380c59b0 0x7f3c3806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3c380c59b0 0x7f3c3806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57712/0 (socket says 172.21.15.107:57712) 2023-11-24T08:12:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 -- 172.21.15.107:0/2001126497 learned_addr learned my addr 172.21.15.107:0/2001126497 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c36ffd640 1 --2- 172.21.15.107:0/2001126497 >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 0x7f3c380cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 -- 172.21.15.107:0/2001126497 >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 msgr2=0x7f3c380cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 --2- 172.21.15.107:0/2001126497 >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 0x7f3c380cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 -- 172.21.15.107:0/2001126497 >> v1:172.21.15.107:6789/0 conn(0x7f3c380cbf20 legacy=0x7f3c380c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c377fe640 1 -- 172.21.15.107:0/2001126497 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3815eea0 con 0x7f3c380c59b0 2023-11-24T08:12:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.231+0000 7f3c36ffd640 1 --2- 172.21.15.107:0/2001126497 >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 0x7f3c380cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.232+0000 7f3c377fe640 1 --2- 172.21.15.107:0/2001126497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c3806c5f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f3c3806e770 tx=0x7f3c3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6938951b5ea9e0a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.233+0000 7f3c35ffb640 1 -- 172.21.15.107:0/2001126497 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3002ca80 con 0x7f3c380c59b0 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.233+0000 7f3c35ffb640 1 -- 172.21.15.107:0/2001126497 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3002cbe0 con 0x7f3c380c59b0 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.233+0000 7f3c35ffb640 1 -- 172.21.15.107:0/2001126497 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300343f0 con 0x7f3c380c59b0 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.233+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2001126497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 msgr2=0x7f3c3806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.233+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2001126497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c3806c5f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f3c3806e770 tx=0x7f3c3002bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2001126497 shutdown_connections 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2001126497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c3806c5f0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2001126497 >> v2:172.21.15.120:3300/0 conn(0x7f3c3815b440 0x7f3c380cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2001126497 >> 172.21.15.107:0/2001126497 conn(0x7f3c380b0c50 msgr2=0x7f3c380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2001126497 shutdown_connections 2023-11-24T08:12:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.234+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2001126497 wait complete. 2023-11-24T08:12:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.235+0000 7f3c3d72c640 1 Processor -- start 2023-11-24T08:12:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.236+0000 7f3c3d72c640 1 -- start start 2023-11-24T08:12:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.236+0000 7f3c3d72c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.236+0000 7f3c3d72c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c380cbf20 0x7f3c38146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c3d72c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c3815b440 0x7f3c380af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c3d72c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38155e70 con 0x7f3c380c59b0 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c3d72c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38147410 con 0x7f3c3815b440 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c3d72c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38156cb0 con 0x7f3c380cbf20 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57720/0 (socket says 172.21.15.107:57720) 2023-11-24T08:12:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 -- 172.21.15.107:0/2592108650 learned_addr learned my addr 172.21.15.107:0/2592108650 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 -- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c380cbf20 msgr2=0x7f3c38146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c380cbf20 0x7f3c38146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 -- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c3815b440 msgr2=0x7f3c380af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c3815b440 0x7f3c380af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 -- 172.21.15.107:0/2592108650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30009b30 con 0x7f3c380c59b0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.237+0000 7f3c36ffd640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c7f0 tx=0x7f3c2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.238+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28007820 con 0x7f3c380c59b0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.238+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c28007980 con 0x7f3c380c59b0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.238+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28007cf0 con 0x7f3c380c59b0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.238+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c381560a0 con 0x7f3c380c59b0 2023-11-24T08:12:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.238+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c38156660 con 0x7f3c380c59b0 2023-11-24T08:12:25.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.239+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3804e8b0 con 0x7f3c380c59b0 2023-11-24T08:12:25.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.240+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7f3c280113f0 con 0x7f3c380c59b0 2023-11-24T08:12:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.241+0000 7f3c1f7fe640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 0x7f3c2406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.241+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c28088720 con 0x7f3c380c59b0 2023-11-24T08:12:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.241+0000 7f3c367fc640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 0x7f3c2406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.242+0000 7f3c367fc640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 0x7f3c2406e0b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c000f60 tx=0x7f3c2c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.246+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c28054ed0 con 0x7f3c380c59b0 2023-11-24T08:12:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:25.473+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c3815f3d0 con 0x7f3c2406bbf0 2023-11-24T08:12:26.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.903+0000 7f3c1f7fe640 1 -- 172.21.15.107:0/2592108650 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f3c2c0012b0 con 0x7f3c2406bbf0 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1183", 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap2260" 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:26.906 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:26.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 msgr2=0x7f3c2406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 0x7f3c2406e0b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c000f60 tx=0x7f3c2c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 msgr2=0x7f3c38157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c7f0 tx=0x7f3c2800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 shutdown_connections 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3c2406bbf0 0x7f3c2406e0b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c3815b440 0x7f3c380af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c380cbf20 0x7f3c38146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 --2- 172.21.15.107:0/2592108650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c380c59b0 0x7f3c38157140 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 >> 172.21.15.107:0/2592108650 conn(0x7f3c380b0c50 msgr2=0x7f3c380c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 shutdown_connections 2023-11-24T08:12:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:26.907+0000 7f3c3d72c640 1 -- 172.21.15.107:0/2592108650 wait complete. 2023-11-24T08:12:27.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7064 2023-11-24T08:12:28.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.158+0000 7faac4f0f640 1 Processor -- start 2023-11-24T08:12:28.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.158+0000 7faac4f0f640 1 -- start start 2023-11-24T08:12:28.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.159+0000 7faac4f0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.159+0000 7faac4f0f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faac00cbf20 0x7faac00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faac4f0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac00cc3a0 con 0x7faac00c57e0 2023-11-24T08:12:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faac4f0f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac015b820 con 0x7faac00cbf20 2023-11-24T08:12:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faac4f0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac015eea0 con 0x7faac015b440 2023-11-24T08:12:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faabe59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faab7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faac00cbf20 0x7faac00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faabed9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7faac00c57e0 0x7faac00c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:42650/0 (socket says 172.21.15.107:42650) 2023-11-24T08:12:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faabe59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:52196/0 (socket says 172.21.15.107:52196) 2023-11-24T08:12:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.160+0000 7faabed9d640 1 -- 172.21.15.107:0/535643847 learned_addr learned my addr 172.21.15.107:0/535643847 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.161+0000 7faab7fff640 1 -- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 msgr2=0x7faac00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.161+0000 7faab7fff640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:28.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.161+0000 7faab7fff640 1 -- 172.21.15.107:0/535643847 >> v1:172.21.15.120:6789/0 conn(0x7faac00c57e0 legacy=0x7faac00c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:28.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.161+0000 7faab7fff640 1 -- 172.21.15.107:0/535643847 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac006d600 con 0x7faac00cbf20 2023-11-24T08:12:28.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.161+0000 7faabe59c640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:12:28.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.162+0000 7faab7fff640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac00cbf20 0x7faac00c52a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f7f55b943af2345 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:28.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.163+0000 7faabdd9b640 1 -- 172.21.15.107:0/535643847 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8031780 con 0x7faac00cbf20 2023-11-24T08:12:28.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.163+0000 7faabdd9b640 1 -- 172.21.15.107:0/535643847 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa80318e0 con 0x7faac00cbf20 2023-11-24T08:12:28.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.163+0000 7faabdd9b640 1 -- 172.21.15.107:0/535643847 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8031c70 con 0x7faac00cbf20 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 -- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac00cbf20 msgr2=0x7faac00c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac00cbf20 0x7faac00c52a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802b820 comp rx=0 tx=0).stop 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 -- 172.21.15.107:0/535643847 shutdown_connections 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac00cbf20 0x7faac00c52a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 --2- 172.21.15.107:0/535643847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac015b440 0x7faac00cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 -- 172.21.15.107:0/535643847 >> 172.21.15.107:0/535643847 conn(0x7faac00b0c50 msgr2=0x7faac00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 -- 172.21.15.107:0/535643847 shutdown_connections 2023-11-24T08:12:28.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.164+0000 7faac4f0f640 1 -- 172.21.15.107:0/535643847 wait complete. 2023-11-24T08:12:28.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.166+0000 7faac4f0f640 1 Processor -- start 2023-11-24T08:12:28.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.166+0000 7faac4f0f640 1 -- start start 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.167+0000 7faac4f0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.167+0000 7faac4f0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac00cbf20 0x7faac0070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.167+0000 7faac4f0f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac015b440 0x7faac0071110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faac4f0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac0152a70 con 0x7faac00c57e0 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faac4f0f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac015a490 con 0x7faac015b440 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faac4f0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac015a770 con 0x7faac00cbf20 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51208/0 (socket says 172.21.15.107:51208) 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 -- 172.21.15.107:0/2706134880 learned_addr learned my addr 172.21.15.107:0/2706134880 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabed9d640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac015b440 0x7faac0071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faab7fff640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac00cbf20 0x7faac0070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 -- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac00cbf20 msgr2=0x7faac0070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac00cbf20 0x7faac0070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 -- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac015b440 msgr2=0x7faac0071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac015b440 0x7faac0071110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.168+0000 7faabe59c640 1 -- 172.21.15.107:0/2706134880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab0009400 con 0x7faac00c57e0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faabe59c640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7faab000e960 tx=0x7faab000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab000ca00 con 0x7faac00c57e0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab000cb60 con 0x7faac00c57e0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa8009400 con 0x7faac00c57e0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0013470 con 0x7faac00c57e0 2023-11-24T08:12:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.169+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac0147330 con 0x7faac00c57e0 2023-11-24T08:12:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.170+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac015ac90 con 0x7faac00c57e0 2023-11-24T08:12:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.171+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7faab00135d0 con 0x7faac00c57e0 2023-11-24T08:12:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.173+0000 7faab6ffd640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 0x7faaa006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.173+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faab0088600 con 0x7faac00c57e0 2023-11-24T08:12:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.173+0000 7faab7fff640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 0x7faaa006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.174+0000 7faab7fff640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 0x7faaa006e080 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faaa8036000 tx=0x7faaa803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.177+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faac015ac90 con 0x7faac00c57e0 2023-11-24T08:12:28.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:28.402+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7faac00c52a0 con 0x7faaa006bbc0 2023-11-24T08:12:31.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.889+0000 7faab6ffd640 1 -- 172.21.15.107:0/2706134880 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7faac00c52a0 con 0x7faaa006bbc0 2023-11-24T08:12:31.891 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:31.891 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:12:31.891 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:12:31.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:31.892 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 msgr2=0x7faaa006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 0x7faaa006e080 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faaa8036000 tx=0x7faaa803d040 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 msgr2=0x7faac0146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7faab000e960 tx=0x7faab000ee20 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 shutdown_connections 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faaa006bbc0 0x7faaa006e080 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faac015b440 0x7faac0071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faac00cbf20 0x7faac0070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 --2- 172.21.15.107:0/2706134880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faac00c57e0 0x7faac0146df0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 >> 172.21.15.107:0/2706134880 conn(0x7faac00b0c50 msgr2=0x7faac00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.892+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 shutdown_connections 2023-11-24T08:12:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:31.893+0000 7faac4f0f640 1 -- 172.21.15.107:0/2706134880 wait complete. 2023-11-24T08:12:31.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol1183 subvol_snap2260 2023-11-24T08:12:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.134+0000 7f70839b0640 1 Processor -- start 2023-11-24T08:12:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.134+0000 7f70839b0640 1 -- start start 2023-11-24T08:12:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f70839b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f707c15c7f0 0x7f707c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f70839b0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f707c0cf450 0x7f707c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f70839b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707c0cece0 con 0x7f707c0cdfb0 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f70839b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c0cfe40 con 0x7f707c15c7f0 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f70839b0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c0ceaf0 con 0x7f707c0cf450 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f708174c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f707c15c7f0 0x7f707c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f7081f4d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f707c0cdfb0 0x7f707c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33650/0 (socket says 172.21.15.107:33650) 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.136+0000 7f7081f4d640 1 -- 172.21.15.107:0/1803537593 learned_addr learned my addr 172.21.15.107:0/1803537593 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.137+0000 7f7080f4b640 1 --2- 172.21.15.107:0/1803537593 >> v2:172.21.15.120:3300/0 conn(0x7f707c0cf450 0x7f707c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.137+0000 7f708174c640 1 -- 172.21.15.107:0/1803537593 >> v2:172.21.15.120:3300/0 conn(0x7f707c0cf450 msgr2=0x7f707c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.137+0000 7f708174c640 1 --2- 172.21.15.107:0/1803537593 >> v2:172.21.15.120:3300/0 conn(0x7f707c0cf450 0x7f707c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.137+0000 7f708174c640 1 -- 172.21.15.107:0/1803537593 >> v1:172.21.15.107:6789/0 conn(0x7f707c0cdfb0 legacy=0x7f707c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.137+0000 7f708174c640 1 -- 172.21.15.107:0/1803537593 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c06d040 con 0x7f707c15c7f0 2023-11-24T08:12:32.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.138+0000 7f708174c640 1 --2- 172.21.15.107:0/1803537593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c15c7f0 0x7f707c06db20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f707c070410 tx=0x7f706c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e2a23ad8a1216 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.138+0000 7f707bfff640 1 -- 172.21.15.107:0/1803537593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c02cc70 con 0x7f707c15c7f0 2023-11-24T08:12:32.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.138+0000 7f707bfff640 1 -- 172.21.15.107:0/1803537593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c02cdd0 con 0x7f707c15c7f0 2023-11-24T08:12:32.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.138+0000 7f707bfff640 1 -- 172.21.15.107:0/1803537593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c034660 con 0x7f707c15c7f0 2023-11-24T08:12:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 -- 172.21.15.107:0/1803537593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c15c7f0 msgr2=0x7f707c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 --2- 172.21.15.107:0/1803537593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c15c7f0 0x7f707c06db20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f707c070410 tx=0x7f706c02bec0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 -- 172.21.15.107:0/1803537593 shutdown_connections 2023-11-24T08:12:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 --2- 172.21.15.107:0/1803537593 >> v2:172.21.15.120:3300/0 conn(0x7f707c0cf450 0x7f707c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 --2- 172.21.15.107:0/1803537593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c15c7f0 0x7f707c06db20 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 -- 172.21.15.107:0/1803537593 >> 172.21.15.107:0/1803537593 conn(0x7f707c06a920 msgr2=0x7f707c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 -- 172.21.15.107:0/1803537593 shutdown_connections 2023-11-24T08:12:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.139+0000 7f70839b0640 1 -- 172.21.15.107:0/1803537593 wait complete. 2023-11-24T08:12:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.141+0000 7f70839b0640 1 Processor -- start 2023-11-24T08:12:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.141+0000 7f70839b0640 1 -- start start 2023-11-24T08:12:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f707c0cf450 0x7f707c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707c15c7f0 0x7f707c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c1479e0 con 0x7f707c0cdfb0 2023-11-24T08:12:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c158770 con 0x7f707c0cf450 2023-11-24T08:12:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f70839b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c158a50 con 0x7f707c15c7f0 2023-11-24T08:12:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f708174c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f708174c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51212/0 (socket says 172.21.15.107:51212) 2023-11-24T08:12:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.142+0000 7f708174c640 1 -- 172.21.15.107:0/3207409597 learned_addr learned my addr 172.21.15.107:0/3207409597 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 -- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707c15c7f0 msgr2=0x7f707c157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707c15c7f0 0x7f707c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 -- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f707c0cf450 msgr2=0x7f707c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f7080f4b640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f707c0cf450 0x7f707c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f707c0cf450 0x7f707c155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 -- 172.21.15.107:0/3207409597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c009a70 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f708174c640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f706c009e80 tx=0x7f706c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c03e610 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c03e790 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c03eb40 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.143+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c158d30 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.144+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c1592f0 con 0x7f707c0cdfb0 2023-11-24T08:12:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.145+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c067470 con 0x7f707c0cdfb0 2023-11-24T08:12:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.146+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7f706c03b050 con 0x7f707c0cdfb0 2023-11-24T08:12:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.147+0000 7f7079ffb640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 0x7f706806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.147+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f706c0aa4f0 con 0x7f707c0cdfb0 2023-11-24T08:12:32.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.147+0000 7f7080f4b640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 0x7f706806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.148+0000 7f7080f4b640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 0x7f706806e080 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f707c06e1e0 tx=0x7f7070008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.151+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f706c076da0 con 0x7f707c0cdfb0 2023-11-24T08:12:32.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.373+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol1183", "snap_name": "subvol_snap2260", "target": ["mon-mgr", ""]}) v1 -- 0x7f707c06e060 con 0x7f706806bbc0 2023-11-24T08:12:32.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.441+0000 7f7079ffb640 1 -- 172.21.15.107:0/3207409597 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7070009d60 con 0x7f706806bbc0 2023-11-24T08:12:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 msgr2=0x7f706806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 0x7f706806e080 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f707c06e1e0 tx=0x7f7070008040 comp rx=0 tx=0).stop 2023-11-24T08:12:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 msgr2=0x7f707c1474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f706c009e80 tx=0x7f706c004a40 comp rx=0 tx=0).stop 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 shutdown_connections 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f706806bbc0 0x7f706806e080 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707c15c7f0 0x7f707c157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f707c0cf450 0x7f707c155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 --2- 172.21.15.107:0/3207409597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707c0cdfb0 0x7f707c1474a0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 >> 172.21.15.107:0/3207409597 conn(0x7f707c06a920 msgr2=0x7f707c071890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 shutdown_connections 2023-11-24T08:12:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.446+0000 7f70839b0640 1 -- 172.21.15.107:0/3207409597 wait complete. 2023-11-24T08:12:32.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone7064 2023-11-24T08:12:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.677+0000 7fb7fca1a640 1 Processor -- start 2023-11-24T08:12:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.677+0000 7fb7fca1a640 1 -- start start 2023-11-24T08:12:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f81608f0 0x7fb7f80d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 0x7fb7f80d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce710 0x7fb7f80d36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f80b8f60 con 0x7fb7f81608f0 2023-11-24T08:12:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f8160cd0 con 0x7fb7f80ce260 2023-11-24T08:12:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7fca1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80cdcb0 con 0x7fb7f80ce710 2023-11-24T08:12:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7f77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f81608f0 0x7fb7f80d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7f77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f81608f0 0x7fb7f80d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51216/0 (socket says 172.21.15.107:51216) 2023-11-24T08:12:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7f77fe640 1 -- 172.21.15.107:0/3188088547 learned_addr learned my addr 172.21.15.107:0/3188088547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7eeffd640 1 --2- 172.21.15.107:0/3188088547 >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 0x7fb7f80d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.679+0000 7fb7f7fff640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce710 0x7fb7f80d36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7f77fe640 1 -- 172.21.15.107:0/3188088547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce710 msgr2=0x7fb7f80d36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce710 0x7fb7f80d36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7f77fe640 1 -- 172.21.15.107:0/3188088547 >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 msgr2=0x7fb7f80d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/3188088547 >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 0x7fb7f80d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7f77fe640 1 -- 172.21.15.107:0/3188088547 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80cdf90 con 0x7fb7f81608f0 2023-11-24T08:12:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.680+0000 7fb7eeffd640 1 --2- 172.21.15.107:0/3188088547 >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 0x7fb7f80d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:32.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.681+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f80d1ac0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80d3e90 tx=0x7fb7e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf324af6a0e933c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.681+0000 7fb7f6ffd640 1 -- 172.21.15.107:0/3188088547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8031870 con 0x7fb7f81608f0 2023-11-24T08:12:32.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7f6ffd640 1 -- 172.21.15.107:0/3188088547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e80319d0 con 0x7fb7f81608f0 2023-11-24T08:12:32.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7f6ffd640 1 -- 172.21.15.107:0/3188088547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8031d60 con 0x7fb7f81608f0 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 -- 172.21.15.107:0/3188088547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 msgr2=0x7fb7f80d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f80d1ac0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80d3e90 tx=0x7fb7e802b910 comp rx=0 tx=0).stop 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 -- 172.21.15.107:0/3188088547 shutdown_connections 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce710 0x7fb7f80d36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/3188088547 >> v2:172.21.15.120:3300/0 conn(0x7fb7f80ce260 0x7fb7f80d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/3188088547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f80d1ac0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 -- 172.21.15.107:0/3188088547 >> 172.21.15.107:0/3188088547 conn(0x7fb7f8061340 msgr2=0x7fb7f8061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 -- 172.21.15.107:0/3188088547 shutdown_connections 2023-11-24T08:12:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.682+0000 7fb7fca1a640 1 -- 172.21.15.107:0/3188088547 wait complete. 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.684+0000 7fb7fca1a640 1 Processor -- start 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.684+0000 7fb7fca1a640 1 -- start start 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce260 0x7fb7f80c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f80ce710 0x7fb7f81f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c7080 con 0x7fb7f81608f0 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f81f3d30 con 0x7fb7f80ce710 2023-11-24T08:12:32.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.685+0000 7fb7fca1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8156a20 con 0x7fb7f80ce260 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51218/0 (socket says 172.21.15.107:51218) 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 -- 172.21.15.107:0/876496525 learned_addr learned my addr 172.21.15.107:0/876496525 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce260 0x7fb7f80c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 -- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce260 msgr2=0x7fb7f80c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce260 0x7fb7f80c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 -- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f80ce710 msgr2=0x7fb7f81f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f80ce710 0x7fb7f81f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 -- 172.21.15.107:0/876496525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80094d0 con 0x7fb7f81608f0 2023-11-24T08:12:32.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f7fff640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007e40 tx=0x7fb7f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0014620 con 0x7fb7f81608f0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.686+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000ee90 con 0x7fb7f81608f0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.687+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00149e0 con 0x7fb7f81608f0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.687+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f80c8790 con 0x7fb7f81608f0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.687+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f80c8d50 con 0x7fb7f81608f0 2023-11-24T08:12:32.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.688+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80617d0 con 0x7fb7f81608f0 2023-11-24T08:12:32.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.689+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7fb7f0014b40 con 0x7fb7f81608f0 2023-11-24T08:12:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.690+0000 7fb7f4ff9640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 0x7fb7d806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.690+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7f0088c90 con 0x7fb7f81608f0 2023-11-24T08:12:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.691+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 0x7fb7d806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:32.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.691+0000 7fb7f77fe640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 0x7fb7d806e0b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:32.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.695+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7f0055540 con 0x7fb7f81608f0 2023-11-24T08:12:32.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.916+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7f80cecb0 con 0x7fb7d806bbf0 2023-11-24T08:12:32.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.927+0000 7fb7f4ff9640 1 -- 172.21.15.107:0/876496525 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+707 (secure 0 0 0) 0x7fb7f80cecb0 con 0x7fb7d806bbf0 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:12:17", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 1048576, 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:12:18", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:12:26", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:12:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:12:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:12:17", 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone7064/079490e9-e699-426d-9804-7a9ba3738b4a", 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:12:32.932 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 msgr2=0x7fb7d806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 0x7fb7d806e0b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e803c040 comp rx=0 tx=0).stop 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 msgr2=0x7fb7f81564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007e40 tx=0x7fb7f000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 shutdown_connections 2023-11-24T08:12:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7d806bbf0 0x7fb7d806e0b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f81608f0 0x7fb7f81564e0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f80ce710 0x7fb7f81f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 --2- 172.21.15.107:0/876496525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7f80ce260 0x7fb7f80c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.930+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 >> 172.21.15.107:0/876496525 conn(0x7fb7f8061340 msgr2=0x7fb7f80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.931+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 shutdown_connections 2023-11-24T08:12:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:32.931+0000 7fb7fca1a640 1 -- 172.21.15.107:0/876496525 wait complete. 2023-11-24T08:12:32.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol1183 2023-11-24T08:12:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.165+0000 7f17517c1640 1 Processor -- start 2023-11-24T08:12:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.165+0000 7f17517c1640 1 -- start start 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f17517c1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f174c15a1c0 0x7f174c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f17517c1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c06d430 con 0x7f174c0b0b40 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f17517c1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c15ec50 con 0x7f174c15e5e0 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f17517c1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c15ea70 con 0x7f174c15a1c0 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f174bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f174c15a1c0 0x7f174c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f174b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f174c0b0b40 0x7f174c15ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33658/0 (socket says 172.21.15.107:33658) 2023-11-24T08:12:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.167+0000 7f174b7fe640 1 -- 172.21.15.107:0/839124214 learned_addr learned my addr 172.21.15.107:0/839124214 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.168+0000 7f174bfff640 1 -- 172.21.15.107:0/839124214 >> v1:172.21.15.120:6789/0 conn(0x7f174c15e5e0 legacy=0x7f174c070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.168+0000 7f174bfff640 1 -- 172.21.15.107:0/839124214 >> v1:172.21.15.107:6789/0 conn(0x7f174c0b0b40 legacy=0x7f174c15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.168+0000 7f174bfff640 1 -- 172.21.15.107:0/839124214 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c06cb00 con 0x7f174c15a1c0 2023-11-24T08:12:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.168+0000 7f174affd640 1 -- 172.21.15.107:0/839124214 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421525582 0 0) 0x7f174c06d430 con 0x7f174c0b0b40 2023-11-24T08:12:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f174bfff640 1 --2- 172.21.15.107:0/839124214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c15a1c0 0x7f174c06d8c0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f173c0097a0 tx=0x7f173c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb900b27cd5c33ae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f174affd640 1 -- 172.21.15.107:0/839124214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c031830 con 0x7f174c15a1c0 2023-11-24T08:12:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f174affd640 1 -- 172.21.15.107:0/839124214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c031990 con 0x7f174c15a1c0 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f174affd640 1 -- 172.21.15.107:0/839124214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c031d20 con 0x7f174c15a1c0 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f17517c1640 1 -- 172.21.15.107:0/839124214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c15a1c0 msgr2=0x7f174c06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.169+0000 7f17517c1640 1 --2- 172.21.15.107:0/839124214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c15a1c0 0x7f174c06d8c0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f173c0097a0 tx=0x7f173c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.170+0000 7f17517c1640 1 -- 172.21.15.107:0/839124214 shutdown_connections 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.170+0000 7f17517c1640 1 --2- 172.21.15.107:0/839124214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c15a1c0 0x7f174c06d8c0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.170+0000 7f17517c1640 1 -- 172.21.15.107:0/839124214 >> 172.21.15.107:0/839124214 conn(0x7f174c0cbb40 msgr2=0x7f174c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.170+0000 7f17517c1640 1 -- 172.21.15.107:0/839124214 shutdown_connections 2023-11-24T08:12:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.170+0000 7f17517c1640 1 -- 172.21.15.107:0/839124214 wait complete. 2023-11-24T08:12:33.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.171+0000 7f17517c1640 1 Processor -- start 2023-11-24T08:12:33.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.172+0000 7f17517c1640 1 -- start start 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 0x7f174c0b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f174c15e5e0 0x7f174c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0b2930 con 0x7f174c0b0b40 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0b3300 con 0x7f174c15a1c0 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f17517c1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0b1d20 con 0x7f174c15e5e0 2023-11-24T08:12:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51228/0 (socket says 172.21.15.107:51228) 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 -- 172.21.15.107:0/2384828967 learned_addr learned my addr 172.21.15.107:0/2384828967 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 -- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f174c15e5e0 msgr2=0x7f174c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f174c15e5e0 0x7f174c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174b7fe640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 0x7f174c0b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 -- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 msgr2=0x7f174c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 0x7f174c0b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174bfff640 1 -- 172.21.15.107:0/2384828967 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c009400 con 0x7f174c0b0b40 2023-11-24T08:12:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.173+0000 7f174b7fe640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 0x7f174c0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:33.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f174bfff640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098c0 tx=0x7f173c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c03bbf0 con 0x7f174c0b0b40 2023-11-24T08:12:33.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c03bd50 con 0x7f174c0b0b40 2023-11-24T08:12:33.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c045630 con 0x7f174c0b0b40 2023-11-24T08:12:33.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c0b2000 con 0x7f174c0b0b40 2023-11-24T08:12:33.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.174+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c0b25c0 con 0x7f174c0b0b40 2023-11-24T08:12:33.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.175+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174c04e840 con 0x7f174c0b0b40 2023-11-24T08:12:33.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.176+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7f173c04e450 con 0x7f174c0b0b40 2023-11-24T08:12:33.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.177+0000 7f1748ff9640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 0x7f173806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.177+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f173c0aa6a0 con 0x7f174c0b0b40 2023-11-24T08:12:33.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.177+0000 7f174b7fe640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 0x7f173806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.178+0000 7f174b7fe640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 0x7f173806e080 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c9490 tx=0x7f1740009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.181+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f173c076e90 con 0x7f174c0b0b40 2023-11-24T08:12:33.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.404+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol1183", "target": ["mon-mgr", ""]}) v1 -- 0x7f174c15dba0 con 0x7f173806bbc0 2023-11-24T08:12:33.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.437+0000 7f1748ff9640 1 -- 172.21.15.107:0/2384828967 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f174c15dba0 con 0x7f173806bbc0 2023-11-24T08:12:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.439+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 msgr2=0x7f173806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.439+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 0x7f173806e080 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c9490 tx=0x7f1740009040 comp rx=0 tx=0).stop 2023-11-24T08:12:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.439+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 msgr2=0x7f174c09bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.439+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098c0 tx=0x7f173c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 shutdown_connections 2023-11-24T08:12:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f173806bbc0 0x7f173806e080 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f174c15e5e0 0x7f174c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f174c15a1c0 0x7f174c0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 --2- 172.21.15.107:0/2384828967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c0b0b40 0x7f174c09bd20 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 >> 172.21.15.107:0/2384828967 conn(0x7f174c0cbb40 msgr2=0x7f174c15ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 shutdown_connections 2023-11-24T08:12:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.440+0000 7f17517c1640 1 -- 172.21.15.107:0/2384828967 wait complete. 2023-11-24T08:12:33.445 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.443+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.445 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.443+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.445 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.443+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.445 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.443+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.446 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.443+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone7064 2023-11-24T08:12:33.474 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.472+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.474 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.472+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.474 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.472+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.474 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.472+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.474 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:12:33.472+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:12:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.656+0000 7f3e32a07640 1 Processor -- start 2023-11-24T08:12:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.656+0000 7f3e32a07640 1 -- start start 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.657+0000 7f3e32a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 0x7f3e2c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.657+0000 7f3e32a07640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3e2c0ccfb0 0x7f3e2c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e32a07640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c1655f0 con 0x7f3e2c162540 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e32a07640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c0cc380 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e32a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c162b30 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 0x7f3e2c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e2b7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3e2c0ccfb0 0x7f3e2c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 0x7f3e2c1660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:52286/0 (socket says 172.21.15.107:52286) 2023-11-24T08:12:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.658+0000 7f3e2bfff640 1 -- 172.21.15.107:0/4262810570 learned_addr learned my addr 172.21.15.107:0/4262810570 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.659+0000 7f3e2b7fe640 1 -- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 msgr2=0x7f3e2c1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.659+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 0x7f3e2c1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.659+0000 7f3e2b7fe640 1 -- 172.21.15.107:0/4262810570 >> v1:172.21.15.120:6789/0 conn(0x7f3e2c162540 legacy=0x7f3e2c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.659+0000 7f3e2b7fe640 1 -- 172.21.15.107:0/4262810570 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c162e10 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.660+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c162000 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e200097d0 tx=0x7f3e2002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14dbca6e001f24d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.661+0000 7f3e2affd640 1 -- 172.21.15.107:0/4262810570 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031800 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.661+0000 7f3e2affd640 1 -- 172.21.15.107:0/4262810570 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e20031960 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.661+0000 7f3e2affd640 1 -- 172.21.15.107:0/4262810570 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031cf0 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.661+0000 7f3e32a07640 1 -- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 msgr2=0x7f3e2c162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.661+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c162000 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e200097d0 tx=0x7f3e2002b8a0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 -- 172.21.15.107:0/4262810570 shutdown_connections 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c162000 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4262810570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c0ccbd0 0x7f3e2c1660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 -- 172.21.15.107:0/4262810570 >> 172.21.15.107:0/4262810570 conn(0x7f3e2c0b0c50 msgr2=0x7f3e2c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 -- 172.21.15.107:0/4262810570 shutdown_connections 2023-11-24T08:12:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.662+0000 7f3e32a07640 1 -- 172.21.15.107:0/4262810570 wait complete. 2023-11-24T08:12:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.663+0000 7f3e32a07640 1 Processor -- start 2023-11-24T08:12:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.664+0000 7f3e32a07640 1 -- start start 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 0x7f3e2c1f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1faba0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c158cb0 con 0x7f3e2c0ccfb0 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e32a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1f7e20 con 0x7f3e2c162540 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51264/0 (socket says 172.21.15.107:51264) 2023-11-24T08:12:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e2bfff640 1 -- 172.21.15.107:0/4057852255 learned_addr learned my addr 172.21.15.107:0/4057852255 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.665+0000 7f3e30fa4640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 0x7f3e2c1f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 -- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 msgr2=0x7f3e2c1f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 0x7f3e2c1f75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 -- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 msgr2=0x7f3e2c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 -- 172.21.15.107:0/4057852255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20009400 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e30fa4640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 0x7f3e2c1f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e2bfff640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e840 tx=0x7f3e1c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00c8e0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c00ca40 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00cdf0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c1f8100 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.666+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c1fadf0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.668+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c04e8b0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.669+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) v1 ==== 90455+0+0 (secure 0 0 0) 0x7f3e1c013470 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.669+0000 7f3e28ff9640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 0x7f3e1806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.670+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e1c0883d0 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.670+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 0x7f3e1806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.670+0000 7f3e2b7fe640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 0x7f3e1806e0d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e20036000 tx=0x7f3e20008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.674+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e1c054c20 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.681+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3e1c087f70 con 0x7f3e2c0ccbd0 2023-11-24T08:12:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.894+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone7064", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e2c0cd410 con 0x7f3e1806bc10 2023-11-24T08:12:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.935+0000 7f3e28ff9640 1 -- 172.21.15.107:0/4057852255 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3e20031b50 con 0x7f3e1806bc10 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 msgr2=0x7f3e1806e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 0x7f3e1806e0d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e20036000 tx=0x7f3e20008030 comp rx=0 tx=0).stop 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 msgr2=0x7f3e2c0ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e840 tx=0x7f3e1c00ed00 comp rx=0 tx=0).stop 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 shutdown_connections 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e1806bc10 0x7f3e1806e0d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e2c162540 0x7f3e2c1f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e2c0ccfb0 0x7f3e2c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 --2- 172.21.15.107:0/4057852255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2c0ccbd0 0x7f3e2c0ade70 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 >> 172.21.15.107:0/4057852255 conn(0x7f3e2c0b0c50 msgr2=0x7f3e2c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 shutdown_connections 2023-11-24T08:12:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:33.938+0000 7f3e32a07640 1 -- 172.21.15.107:0/4057852255 wait complete. 2023-11-24T08:12:33.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:33.959 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:12:34.064 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:12:34.065 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:12:34.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:12:34.177 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:12:34.297 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:12:34.297 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:34.297 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:34.359 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:34.359 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:34.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.620+0000 7f699a4ca640 1 Processor -- start 2023-11-24T08:12:34.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.621+0000 7f699a4ca640 1 -- start start 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.622+0000 7f699a4ca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6994143870 0x7f6994146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.622+0000 7f699a4ca640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994161080 0x7f69940b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.622+0000 7f699a4ca640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6994161510 con 0x7f6994146eb0 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.622+0000 7f699a4ca640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6994070460 con 0x7f6994143870 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.622+0000 7f699a4ca640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69941617f0 con 0x7f6994161080 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6994143870 0x7f6994146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:34.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6994143870 0x7f6994146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51266/0 (socket says 172.21.15.107:51266) 2023-11-24T08:12:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 -- 172.21.15.107:0/357168390 learned_addr learned my addr 172.21.15.107:0/357168390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 -- 172.21.15.107:0/357168390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994161080 msgr2=0x7f69940b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 --2- 172.21.15.107:0/357168390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994161080 0x7f69940b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 -- 172.21.15.107:0/357168390 >> v1:172.21.15.107:6789/0 conn(0x7f6994146eb0 legacy=0x7f699406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:34.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.623+0000 7f69994c8640 1 -- 172.21.15.107:0/357168390 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994161ad0 con 0x7f6994143870 2023-11-24T08:12:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.624+0000 7f69994c8640 1 --2- 172.21.15.107:0/357168390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6994143870 0x7f6994146970 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f6990009870 tx=0x7f699002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d4d7d587fa604 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:34.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.624+0000 7f698bfff640 1 -- 172.21.15.107:0/357168390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990030840 con 0x7f6994143870 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f698bfff640 1 -- 172.21.15.107:0/357168390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69900309a0 con 0x7f6994143870 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f698bfff640 1 -- 172.21.15.107:0/357168390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990030d30 con 0x7f6994143870 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 -- 172.21.15.107:0/357168390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6994143870 msgr2=0x7f6994146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 --2- 172.21.15.107:0/357168390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6994143870 0x7f6994146970 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f6990009870 tx=0x7f699002b980 comp rx=0 tx=0).stop 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 -- 172.21.15.107:0/357168390 shutdown_connections 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 --2- 172.21.15.107:0/357168390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994161080 0x7f69940b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 --2- 172.21.15.107:0/357168390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6994143870 0x7f6994146970 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 -- 172.21.15.107:0/357168390 >> 172.21.15.107:0/357168390 conn(0x7f69940cb830 msgr2=0x7f69940cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 -- 172.21.15.107:0/357168390 shutdown_connections 2023-11-24T08:12:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.625+0000 7f699a4ca640 1 -- 172.21.15.107:0/357168390 wait complete. 2023-11-24T08:12:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.627+0000 7f699a4ca640 1 Processor -- start 2023-11-24T08:12:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.627+0000 7f699a4ca640 1 -- start start 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 0x7f699415f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994146eb0 0x7f699415db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6994160750 con 0x7f699415c410 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699415cf80 con 0x7f6994143870 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f699a4ca640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699415fad0 con 0x7f6994146eb0 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f6999cc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f6999cc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51276/0 (socket says 172.21.15.107:51276) 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.628+0000 7f6999cc9640 1 -- 172.21.15.107:0/3919923831 learned_addr learned my addr 172.21.15.107:0/3919923831 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f69994c8640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 0x7f699415f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6998cc7640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994146eb0 0x7f699415db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 -- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994146eb0 msgr2=0x7f699415db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994146eb0 0x7f699415db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 -- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 msgr2=0x7f699415f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 0x7f699415f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 -- 172.21.15.107:0/3919923831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69900094d0 con 0x7f699415c410 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f69994c8640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 0x7f699415f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6999cc9640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f698c00c760 tx=0x7f698c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c003b70 con 0x7f699415c410 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c003cd0 con 0x7f699415c410 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699415fdb0 con 0x7f699415c410 2023-11-24T08:12:34.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.629+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c00aa90 con 0x7f699415c410 2023-11-24T08:12:34.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.630+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6994160370 con 0x7f699415c410 2023-11-24T08:12:34.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.631+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6994160890 con 0x7f699415c410 2023-11-24T08:12:34.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.632+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f698c00abf0 con 0x7f699415c410 2023-11-24T08:12:34.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.632+0000 7f6989ffb640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 0x7f697806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:34.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.632+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f698c087e80 con 0x7f699415c410 2023-11-24T08:12:34.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.633+0000 7f69994c8640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 0x7f697806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:34.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.634+0000 7f69994c8640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 0x7f697806e1a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6990036000 tx=0x7f699003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:34.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.637+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6994160890 con 0x7f699415c410 2023-11-24T08:12:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.830+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6994160890 con 0x7f699415c410 2023-11-24T08:12:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.832+0000 7f6989ffb640 1 -- 172.21.15.107:0/3919923831 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f6994160890 con 0x7f699415c410 2023-11-24T08:12:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:34.835 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:05.691679+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 msgr2=0x7f697806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 0x7f697806e1a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6990036000 tx=0x7f699003b040 comp rx=0 tx=0).stop 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 msgr2=0x7f69940ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f698c00c760 tx=0x7f698c00cc20 comp rx=0 tx=0).stop 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 shutdown_connections 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f697806bce0 0x7f697806e1a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f699415c410 0x7f69940ced70 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6994146eb0 0x7f699415db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 --2- 172.21.15.107:0/3919923831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6994143870 0x7f699415f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:34.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 >> 172.21.15.107:0/3919923831 conn(0x7f69940cb830 msgr2=0x7f6994072300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:34.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 shutdown_connections 2023-11-24T08:12:34.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:34.835+0000 7f699a4ca640 1 -- 172.21.15.107:0/3919923831 wait complete. 2023-11-24T08:12:34.856 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:01.979497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:03.154712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:34.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:35.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.062+0000 7fda3092b640 1 Processor -- start 2023-11-24T08:12:35.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.062+0000 7fda3092b640 1 -- start start 2023-11-24T08:12:35.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.063+0000 7fda3092b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c15b260 0x7fda2c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.063+0000 7fda3092b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fda2c06e580 0x7fda2c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda3092b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c06d5c0 con 0x7fda2c0b1080 2023-11-24T08:12:35.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda3092b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c0b1530 con 0x7fda2c06e580 2023-11-24T08:12:35.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda3092b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c15b640 con 0x7fda2c15b260 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fda2c06e580 0x7fda2c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2b59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fda2c0b1080 0x7fda2c0ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33678/0 (socket says 172.21.15.107:33678) 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2b59e640 1 -- 172.21.15.107:0/2413229858 learned_addr learned my addr 172.21.15.107:0/2413229858 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2ad9d640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c15b260 0x7fda2c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2a59c640 1 -- 172.21.15.107:0/2413229858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c15b260 msgr2=0x7fda2c06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2a59c640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c15b260 0x7fda2c06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2a59c640 1 -- 172.21.15.107:0/2413229858 >> v1:172.21.15.107:6789/0 conn(0x7fda2c0b1080 legacy=0x7fda2c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.064+0000 7fda2a59c640 1 -- 172.21.15.107:0/2413229858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c15ed20 con 0x7fda2c06e580 2023-11-24T08:12:35.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda2a59c640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c06e580 0x7fda2c0b0b40 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fda200097a0 tx=0x7fda2002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d56f23f393ef97 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda29d9b640 1 -- 172.21.15.107:0/2413229858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200316f0 con 0x7fda2c06e580 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda29d9b640 1 -- 172.21.15.107:0/2413229858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20031850 con 0x7fda2c06e580 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda29d9b640 1 -- 172.21.15.107:0/2413229858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20031be0 con 0x7fda2c06e580 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda3092b640 1 -- 172.21.15.107:0/2413229858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c06e580 msgr2=0x7fda2c0b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.066+0000 7fda3092b640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c06e580 0x7fda2c0b0b40 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fda200097a0 tx=0x7fda2002b790 comp rx=0 tx=0).stop 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 -- 172.21.15.107:0/2413229858 shutdown_connections 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c06e580 0x7fda2c0b0b40 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 --2- 172.21.15.107:0/2413229858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c15b260 0x7fda2c06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 -- 172.21.15.107:0/2413229858 >> 172.21.15.107:0/2413229858 conn(0x7fda2c0cbb40 msgr2=0x7fda2c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 -- 172.21.15.107:0/2413229858 shutdown_connections 2023-11-24T08:12:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.067+0000 7fda3092b640 1 -- 172.21.15.107:0/2413229858 wait complete. 2023-11-24T08:12:35.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.068+0000 7fda3092b640 1 Processor -- start 2023-11-24T08:12:35.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 -- start start 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c06e580 0x7fda2c09b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda2c15b260 0x7fda2c1f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c15bdb0 con 0x7fda2c0b1080 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c158b60 con 0x7fda2c15b260 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.069+0000 7fda3092b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c14bc80 con 0x7fda2c06e580 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51288/0 (socket says 172.21.15.107:51288) 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 -- 172.21.15.107:0/253292083 learned_addr learned my addr 172.21.15.107:0/253292083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 -- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c06e580 msgr2=0x7fda2c09b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c06e580 0x7fda2c09b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 -- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda2c15b260 msgr2=0x7fda2c1f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda2c15b260 0x7fda2c1f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 -- 172.21.15.107:0/253292083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20009400 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.070+0000 7fda2a59c640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fda2002bda0 tx=0x7fda200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.071+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20031f00 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.071+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20034690 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.071+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20034a20 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.071+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c15b780 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.071+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c09c280 con 0x7fda2c0b1080 2023-11-24T08:12:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.072+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c15bef0 con 0x7fda2c0b1080 2023-11-24T08:12:35.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.073+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fda2003b040 con 0x7fda2c0b1080 2023-11-24T08:12:35.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.074+0000 7fda137fe640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 0x7fda1806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.074+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda200a9c40 con 0x7fda2c0b1080 2023-11-24T08:12:35.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.074+0000 7fda2ad9d640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 0x7fda1806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.075+0000 7fda2ad9d640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 0x7fda1806e1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0098f0 tx=0x7fda1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.079+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda2c15bef0 con 0x7fda2c0b1080 2023-11-24T08:12:35.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.271+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda2c15bef0 con 0x7fda2c0b1080 2023-11-24T08:12:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.272+0000 7fda137fe640 1 -- 172.21.15.107:0/253292083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda2c15bef0 con 0x7fda2c0b1080 2023-11-24T08:12:35.274 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 msgr2=0x7fda1806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 0x7fda1806e1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0098f0 tx=0x7fda1c008040 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 msgr2=0x7fda2c14b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fda2002bda0 tx=0x7fda200046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 shutdown_connections 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fda1806bce0 0x7fda1806e1a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda2c15b260 0x7fda2c1f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda2c0b1080 0x7fda2c14b740 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 --2- 172.21.15.107:0/253292083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda2c06e580 0x7fda2c09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.275+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 >> 172.21.15.107:0/253292083 conn(0x7fda2c0cbb40 msgr2=0x7fda2c0adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.276+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 shutdown_connections 2023-11-24T08:12:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.276+0000 7fda3092b640 1 -- 172.21.15.107:0/253292083 wait complete. 2023-11-24T08:12:35.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:12:35.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.508+0000 7f123e661640 1 Processor -- start 2023-11-24T08:12:35.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.509+0000 7f123e661640 1 -- start start 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.511+0000 7f123e661640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1238072160 0x7f123815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.511+0000 7f123e661640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f123806de10 0x7f12380705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123e661640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123815ec90 con 0x7f12380c49f0 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123e661640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380cd530 con 0x7f123806de10 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123e661640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380c4ed0 con 0x7f1238072160 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f123806de10 0x7f12380705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f123806de10 0x7f12380705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51294/0 (socket says 172.21.15.107:51294) 2023-11-24T08:12:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 -- 172.21.15.107:0/2526438586 learned_addr learned my addr 172.21.15.107:0/2526438586 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:35.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 -- 172.21.15.107:0/2526438586 >> v2:172.21.15.120:3300/0 conn(0x7f1238072160 msgr2=0x7f123815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:35.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 --2- 172.21.15.107:0/2526438586 >> v2:172.21.15.120:3300/0 conn(0x7f1238072160 0x7f123815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 -- 172.21.15.107:0/2526438586 >> v1:172.21.15.120:6789/0 conn(0x7f12380c49f0 legacy=0x7f123806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.512+0000 7f123cbfe640 1 -- 172.21.15.107:0/2526438586 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380c51b0 con 0x7f123806de10 2023-11-24T08:12:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.514+0000 7f123cbfe640 1 --2- 172.21.15.107:0/2526438586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f12380705c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f12380ccae0 tx=0x7f123002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f796f13bcf173f72 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.514+0000 7f12377fe640 1 -- 172.21.15.107:0/2526438586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123002ca80 con 0x7f123806de10 2023-11-24T08:12:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.514+0000 7f12377fe640 1 -- 172.21.15.107:0/2526438586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123002cbe0 con 0x7f123806de10 2023-11-24T08:12:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.514+0000 7f12377fe640 1 -- 172.21.15.107:0/2526438586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12300343f0 con 0x7f123806de10 2023-11-24T08:12:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 -- 172.21.15.107:0/2526438586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 msgr2=0x7f12380705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 --2- 172.21.15.107:0/2526438586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f12380705c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f12380ccae0 tx=0x7f123002bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 -- 172.21.15.107:0/2526438586 shutdown_connections 2023-11-24T08:12:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 --2- 172.21.15.107:0/2526438586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f12380705c0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 --2- 172.21.15.107:0/2526438586 >> v2:172.21.15.120:3300/0 conn(0x7f1238072160 0x7f123815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 -- 172.21.15.107:0/2526438586 >> 172.21.15.107:0/2526438586 conn(0x7f12380b0c50 msgr2=0x7f12380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 -- 172.21.15.107:0/2526438586 shutdown_connections 2023-11-24T08:12:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.515+0000 7f123e661640 1 -- 172.21.15.107:0/2526438586 wait complete. 2023-11-24T08:12:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.517+0000 7f123e661640 1 Processor -- start 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.517+0000 7f123e661640 1 -- start start 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1238072160 0x7f123815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 0x7f123814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123815b3c0 con 0x7f123806de10 2023-11-24T08:12:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123815c580 con 0x7f12380c49f0 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123e661640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123815d290 con 0x7f1238072160 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51300/0 (socket says 172.21.15.107:51300) 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 -- 172.21.15.107:0/3154300478 learned_addr learned my addr 172.21.15.107:0/3154300478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123cbfe640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 0x7f123814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f122ffff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1238072160 0x7f123815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 -- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1238072160 msgr2=0x7f123815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1238072160 0x7f123815cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 -- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 msgr2=0x7f123814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 0x7f123814bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f1237fff640 1 -- 172.21.15.107:0/3154300478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1230009b30 con 0x7f123806de10 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.518+0000 7f123cbfe640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 0x7f123814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f1237fff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f122800b4a0 tx=0x7f122800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228010610 con 0x7f123806de10 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228010770 con 0x7f123806de10 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228010b20 con 0x7f123806de10 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123815b020 con 0x7f123806de10 2023-11-24T08:12:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.519+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123815ca70 con 0x7f123806de10 2023-11-24T08:12:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.520+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123815ae80 con 0x7f123806de10 2023-11-24T08:12:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.521+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f1228010c80 con 0x7f123806de10 2023-11-24T08:12:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.522+0000 7f12357fa640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 0x7f121806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.522+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1228087f90 con 0x7f123806de10 2023-11-24T08:12:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.522+0000 7f122ffff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 0x7f121806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.523+0000 7f122ffff640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 0x7f121806e1a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f12380713b0 tx=0x7f1220008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.526+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f123815ae80 con 0x7f123806de10 2023-11-24T08:12:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.721+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f123815ae80 con 0x7f123806de10 2023-11-24T08:12:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.721+0000 7f12357fa640 1 -- 172.21.15.107:0/3154300478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f122801e020 con 0x7f123806de10 2023-11-24T08:12:35.724 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:12:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 msgr2=0x7f121806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 0x7f121806e1a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f12380713b0 tx=0x7f1220008040 comp rx=0 tx=0).stop 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 msgr2=0x7f1238158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f122800b4a0 tx=0x7f122800b960 comp rx=0 tx=0).stop 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 shutdown_connections 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f121806bce0 0x7f121806e1a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380c49f0 0x7f123814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1238072160 0x7f123815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 --2- 172.21.15.107:0/3154300478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123806de10 0x7f1238158e60 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 >> 172.21.15.107:0/3154300478 conn(0x7f12380b0c50 msgr2=0x7f12380abf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 shutdown_connections 2023-11-24T08:12:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.724+0000 7f123e661640 1 -- 172.21.15.107:0/3154300478 wait complete. 2023-11-24T08:12:35.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:12:35.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.947+0000 7f7709600640 1 Processor -- start 2023-11-24T08:12:35.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.947+0000 7f7709600640 1 -- start start 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.948+0000 7f7709600640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f770415b290 0x7f770406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.948+0000 7f7709600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f770406e470 0x7f77040ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.948+0000 7f7709600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770406e8d0 con 0x7f77040ad3c0 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.948+0000 7f7709600640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770415b670 con 0x7f770415b290 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.948+0000 7f7709600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77040adbd0 con 0x7f770406e470 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f770415b290 0x7f770406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f770415b290 0x7f770406df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51310/0 (socket says 172.21.15.107:51310) 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 -- 172.21.15.107:0/3794635345 learned_addr learned my addr 172.21.15.107:0/3794635345 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:35.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 -- 172.21.15.107:0/3794635345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f770406e470 msgr2=0x7f77040ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:35.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 --2- 172.21.15.107:0/3794635345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f770406e470 0x7f77040ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 -- 172.21.15.107:0/3794635345 >> v1:172.21.15.120:6789/0 conn(0x7f77040ad3c0 legacy=0x7f770415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:35.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.949+0000 7f7702ffd640 1 -- 172.21.15.107:0/3794635345 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040adeb0 con 0x7f770415b290 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.950+0000 7f7702ffd640 1 --2- 172.21.15.107:0/3794635345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770406df30 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009870 tx=0x7f76f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cc2b4e9d337a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f77027fc640 1 -- 172.21.15.107:0/3794635345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4031640 con 0x7f770415b290 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f77027fc640 1 -- 172.21.15.107:0/3794635345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f40317a0 con 0x7f770415b290 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f77027fc640 1 -- 172.21.15.107:0/3794635345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4031b30 con 0x7f770415b290 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 -- 172.21.15.107:0/3794635345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 msgr2=0x7f770406df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 --2- 172.21.15.107:0/3794635345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770406df30 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009870 tx=0x7f76f402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 -- 172.21.15.107:0/3794635345 shutdown_connections 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 --2- 172.21.15.107:0/3794635345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f770406e470 0x7f77040ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 --2- 172.21.15.107:0/3794635345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770406df30 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 -- 172.21.15.107:0/3794635345 >> 172.21.15.107:0/3794635345 conn(0x7f77040b0b70 msgr2=0x7f77040b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 -- 172.21.15.107:0/3794635345 shutdown_connections 2023-11-24T08:12:35.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.951+0000 7f7709600640 1 -- 172.21.15.107:0/3794635345 wait complete. 2023-11-24T08:12:35.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.953+0000 7f7709600640 1 Processor -- start 2023-11-24T08:12:35.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.953+0000 7f7709600640 1 -- start start 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 0x7f7704154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77040ad3c0 0x7f77041517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7704154a50 con 0x7f770415b290 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770414f800 con 0x7f770406e470 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f7709600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7704157f30 con 0x7f77040ad3c0 2023-11-24T08:12:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f77037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f77037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51312/0 (socket says 172.21.15.107:51312) 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.954+0000 7f77037fe640 1 -- 172.21.15.107:0/1453178955 learned_addr learned my addr 172.21.15.107:0/1453178955 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f7702ffd640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 0x7f7704154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 -- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77040ad3c0 msgr2=0x7f77041517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77040ad3c0 0x7f77041517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 -- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 msgr2=0x7f7704154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 0x7f7704154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 -- 172.21.15.107:0/1453178955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40094d0 con 0x7f770415b290 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f7702ffd640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 0x7f7704154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f77037fe640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00c7b0 tx=0x7f76fc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc003d80 con 0x7f770415b290 2023-11-24T08:12:35.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc005070 con 0x7f770415b290 2023-11-24T08:12:35.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc005420 con 0x7f770415b290 2023-11-24T08:12:35.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770414eed0 con 0x7f770415b290 2023-11-24T08:12:35.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.955+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770414f490 con 0x7f770415b290 2023-11-24T08:12:35.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.956+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7704067f90 con 0x7f770415b290 2023-11-24T08:12:35.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.957+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f76fc005580 con 0x7f770415b290 2023-11-24T08:12:35.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.958+0000 7f76fb7fe640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 0x7f76e406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:35.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.958+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76fc087780 con 0x7f770415b290 2023-11-24T08:12:35.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.958+0000 7f7702ffd640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 0x7f76e406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:35.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.959+0000 7f7702ffd640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 0x7f76e406e1d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f76f4036000 tx=0x7f76f403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:35.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:35.962+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76fc053fa0 con 0x7f770415b290 2023-11-24T08:12:36.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.151+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f770404e840 con 0x7f770415b290 2023-11-24T08:12:36.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.151+0000 7f76fb7fe640 1 -- 172.21.15.107:0/1453178955 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f76fc0575c0 con 0x7f770415b290 2023-11-24T08:12:36.154 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:12:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 msgr2=0x7f76e406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 0x7f76e406e1d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f76f4036000 tx=0x7f76f403c040 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 msgr2=0x7f770414fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00c7b0 tx=0x7f76fc00cc70 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 shutdown_connections 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406bd10 0x7f76e406e1d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f770415b290 0x7f770414fc90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77040ad3c0 0x7f77041517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 --2- 172.21.15.107:0/1453178955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f770406e470 0x7f7704154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.154+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 >> 172.21.15.107:0/1453178955 conn(0x7f77040b0b70 msgr2=0x7f770415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.155+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 shutdown_connections 2023-11-24T08:12:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.155+0000 7f7709600640 1 -- 172.21.15.107:0/1453178955 wait complete. 2023-11-24T08:12:36.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:12:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.375+0000 7f7e8b275640 1 Processor -- start 2023-11-24T08:12:36.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.375+0000 7f7e8b275640 1 -- start start 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e8b275640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e840c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e8b275640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8415eea0 con 0x7f7e840c57e0 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e8b275640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840abf30 con 0x7f7e8415b440 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e8b275640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8415b820 con 0x7f7e840cc0f0 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e89812640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7e840c57e0 0x7f7e8406c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33680/0 (socket says 172.21.15.107:33680) 2023-11-24T08:12:36.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.377+0000 7f7e89812640 1 -- 172.21.15.107:0/3202276690 learned_addr learned my addr 172.21.15.107:0/3202276690 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e840c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948683541 0 0) 0x7f7e8415eea0 con 0x7f7e840c57e0 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e70003520 con 0x7f7e840c57e0 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879677825 0 0) 0x7f7e70003520 con 0x7f7e840c57e0 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 msgr2=0x7f7e840c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e840c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 >> v1:172.21.15.120:6789/0 conn(0x7f7e8415b440 legacy=0x7f7e840cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8415f0f0 con 0x7f7e840c57e0 2023-11-24T08:12:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7e7c0019f0 con 0x7f7e840c57e0 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e840c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 >> v1:172.21.15.107:6789/0 conn(0x7f7e840c57e0 legacy=0x7f7e8406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e700039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e70003fe0 0x7f7e700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e70004d00 0x7f7e70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8415b820 con 0x7f7e70003fe0 2023-11-24T08:12:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7000bbc0 con 0x7f7e70004d00 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.378+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7000bd80 con 0x7f7e840cc0f0 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e70003fe0 0x7f7e700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e89011640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e700039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e89812640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e70004d00 0x7f7e70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 -- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 msgr2=0x7f7e700039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e840cc0f0 0x7f7e700039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 -- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e70004d00 msgr2=0x7f7e70005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e70004d00 0x7f7e70005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e88810640 1 -- 172.21.15.107:0/3202276690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70003520 con 0x7f7e70003fe0 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.379+0000 7f7e89812640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e70004d00 0x7f7e70005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.380+0000 7f7e88810640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e70003fe0 0x7f7e700047c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7e740098a0 tx=0x7f7e700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36bec4a771f21db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.380+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74004230 con 0x7f7e70003fe0 2023-11-24T08:12:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.380+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e74004390 con 0x7f7e70003fe0 2023-11-24T08:12:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.380+0000 7f7e7bfff640 1 -- 172.21.15.107:0/3202276690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74003830 con 0x7f7e70003fe0 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 -- 172.21.15.107:0/3202276690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e70003fe0 msgr2=0x7f7e700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 --2- 172.21.15.107:0/3202276690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e70003fe0 0x7f7e700047c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7e740098a0 tx=0x7f7e700057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e89812640 1 -- 172.21.15.107:0/3202276690 reap_dead start 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 -- 172.21.15.107:0/3202276690 shutdown_connections 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 -- 172.21.15.107:0/3202276690 >> 172.21.15.107:0/3202276690 conn(0x7f7e840b0c50 msgr2=0x7f7e840b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 -- 172.21.15.107:0/3202276690 shutdown_connections 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.381+0000 7f7e8b275640 1 -- 172.21.15.107:0/3202276690 wait complete. 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.383+0000 7f7e8b275640 1 Processor -- start 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.383+0000 7f7e8b275640 1 -- start start 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e70003fe0 0x7f7e841f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 0x7f7e841ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f2eb0 con 0x7f7e841ee770 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840c67a0 con 0x7f7e84157530 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e8b275640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84157a60 con 0x7f7e70003fe0 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e89812640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e89812640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51330/0 (socket says 172.21.15.107:51330) 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e89812640 1 -- 172.21.15.107:0/129390108 learned_addr learned my addr 172.21.15.107:0/129390108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e89011640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e70003fe0 0x7f7e841f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.384+0000 7f7e88810640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 0x7f7e841ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 -- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e70003fe0 msgr2=0x7f7e841f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e70003fe0 0x7f7e841f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 -- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 msgr2=0x7f7e841ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 0x7f7e841ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 -- 172.21.15.107:0/129390108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84157f50 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e88810640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 0x7f7e841ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e89812640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7e8409d280 tx=0x7f7e7c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00bb20 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c00bc80 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e840c61e0 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c007110 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.385+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e841eec00 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.386+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e841eedf0 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.387+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f7e7c007290 con 0x7f7e841ee770 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.388+0000 7f7e79ffb640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 0x7f7e6806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.388+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e7c088810 con 0x7f7e841ee770 2023-11-24T08:12:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.388+0000 7f7e89011640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 0x7f7e6806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.389+0000 7f7e89011640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 0x7f7e6806e1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7e80007fd0 tx=0x7f7e80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.393+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e7c055090 con 0x7f7e841ee770 2023-11-24T08:12:36.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.594+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7e8406dc80 con 0x7f7e841ee770 2023-11-24T08:12:36.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.595+0000 7f7e79ffb640 1 -- 172.21.15.107:0/129390108 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v94) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7e7c0586b0 con 0x7f7e841ee770 2023-11-24T08:12:36.597 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:12:36.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 msgr2=0x7f7e6806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 0x7f7e6806e1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7e80007fd0 tx=0x7f7e80009040 comp rx=0 tx=0).stop 2023-11-24T08:12:36.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 msgr2=0x7f7e841ef120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7e8409d280 tx=0x7f7e7c00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 shutdown_connections 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e6806bce0 0x7f7e6806e1a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e841ee770 0x7f7e841ef120 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e84157530 0x7f7e841ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 --2- 172.21.15.107:0/129390108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e70003fe0 0x7f7e841f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 >> 172.21.15.107:0/129390108 conn(0x7f7e840b0c50 msgr2=0x7f7e840cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 shutdown_connections 2023-11-24T08:12:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:36.598+0000 7f7e8b275640 1 -- 172.21.15.107:0/129390108 wait complete. 2023-11-24T08:12:36.618 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:12:36.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:36.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:12:36.710 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:12:36.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:36.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:12:36.762 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:36.762 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:36.823 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:36.823 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:36.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.094+0000 7f6f6a413640 1 Processor -- start 2023-11-24T08:12:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.094+0000 7f6f6a413640 1 -- start start 2023-11-24T08:12:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.095+0000 7f6f6a413640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f6415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.095+0000 7f6f6a413640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6415bf80 con 0x7f6f64144b30 2023-11-24T08:12:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.095+0000 7f6f6a413640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f640b1560 con 0x7f6f640b1de0 2023-11-24T08:12:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.095+0000 7f6f6a413640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640b1280 con 0x7f6f6415c410 2023-11-24T08:12:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f69411640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f64144b30 0x7f6f640b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33686/0 (socket says 172.21.15.107:33686) 2023-11-24T08:12:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f69411640 1 -- 172.21.15.107:0/3656747523 learned_addr learned my addr 172.21.15.107:0/3656747523 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926668458 0 0) 0x7f6f6415bf80 con 0x7f6f64144b30 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f50003520 con 0x7f6f64144b30 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f68c10640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f6415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666140756 0 0) 0x7f6f50003520 con 0x7f6f64144b30 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 msgr2=0x7f6f6415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f6415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 >> v1:172.21.15.120:6789/0 conn(0x7f6f640b1de0 legacy=0x7f6f640b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6415b930 con 0x7f6f64144b30 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.096+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6f60001670 con 0x7f6f64144b30 2023-11-24T08:12:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 >> v1:172.21.15.107:6789/0 conn(0x7f6f64144b30 legacy=0x7f6f640b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500039a0 0x7f6f50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500047c0 0x7f6f50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f50005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640b1280 con 0x7f6f500047c0 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5000bb10 con 0x7f6f500039a0 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5000bcd0 con 0x7f6f6415c410 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500047c0 0x7f6f50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f69411640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500039a0 0x7f6f50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f69c12640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f50005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 -- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 msgr2=0x7f6f50005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f50005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 -- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500039a0 msgr2=0x7f6f50003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500039a0 0x7f6f50003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f68c10640 1 -- 172.21.15.107:0/3656747523 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50003520 con 0x7f6f500047c0 2023-11-24T08:12:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f69c12640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415c410 0x7f6f50005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.097+0000 7f6f69411640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500039a0 0x7f6f50003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.098+0000 7f6f68c10640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500047c0 0x7f6f50004c40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400ba30 tx=0x7f6f500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a574ef4f2a4550be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54032640 con 0x7f6f500047c0 2023-11-24T08:12:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f540327a0 con 0x7f6f500047c0 2023-11-24T08:12:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f5bfff640 1 -- 172.21.15.107:0/3656747523 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54032b30 con 0x7f6f500047c0 2023-11-24T08:12:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 -- 172.21.15.107:0/3656747523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500047c0 msgr2=0x7f6f50004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 --2- 172.21.15.107:0/3656747523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500047c0 0x7f6f50004c40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400ba30 tx=0x7f6f500057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f69c12640 1 -- 172.21.15.107:0/3656747523 reap_dead start 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 -- 172.21.15.107:0/3656747523 shutdown_connections 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 -- 172.21.15.107:0/3656747523 >> 172.21.15.107:0/3656747523 conn(0x7f6f640cb830 msgr2=0x7f6f640cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 -- 172.21.15.107:0/3656747523 shutdown_connections 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.099+0000 7f6f6a413640 1 -- 172.21.15.107:0/3656747523 wait complete. 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.101+0000 7f6f6a413640 1 Processor -- start 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.101+0000 7f6f6a413640 1 -- start start 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500047c0 0x7f6f64070e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415a910 0x7f6f641528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64070390 con 0x7f6f64071610 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64152df0 con 0x7f6f500047c0 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f6a413640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64071f10 con 0x7f6f6415a910 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.102+0000 7f6f69c12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51350/0 (socket says 172.21.15.107:51350) 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 -- 172.21.15.107:0/76211429 learned_addr learned my addr 172.21.15.107:0/76211429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 -- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415a910 msgr2=0x7f6f641528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415a910 0x7f6f641528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 -- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500047c0 msgr2=0x7f6f64070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500047c0 0x7f6f64070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 -- 172.21.15.107:0/76211429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f64070630 con 0x7f6f64071610 2023-11-24T08:12:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f69c12640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c000f80 tx=0x7f6f5c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c006d50 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c003970 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.103+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c003d20 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.104+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f64070910 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.104+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f64071ac0 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.105+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f64071cb0 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.106+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f6f5c005070 con 0x7f6f64071610 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.106+0000 7f6f59ffb640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 0x7f6f4806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.107+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f5c088070 con 0x7f6f64071610 2023-11-24T08:12:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.107+0000 7f6f69411640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 0x7f6f4806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.107+0000 7f6f69411640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 0x7f6f4806e1a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.111+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f5c0548f0 con 0x7f6f64071610 2023-11-24T08:12:37.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.303+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f64070b00 con 0x7f6f64071610 2023-11-24T08:12:37.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.305+0000 7f6f59ffb640 1 -- 172.21.15.107:0/76211429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f6f5c01ce80 con 0x7f6f64071610 2023-11-24T08:12:37.307 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:37.308 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:05.691679+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 msgr2=0x7f6f4806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 0x7f6f4806e1a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f60009040 comp rx=0 tx=0).stop 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 msgr2=0x7f6f6415dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c000f80 tx=0x7f6f5c00dc00 comp rx=0 tx=0).stop 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 shutdown_connections 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f4806bce0 0x7f6f4806e1a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f64071610 0x7f6f6415dde0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f6415a910 0x7f6f641528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 --2- 172.21.15.107:0/76211429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500047c0 0x7f6f64070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 >> 172.21.15.107:0/76211429 conn(0x7f6f640cb830 msgr2=0x7f6f64143e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 shutdown_connections 2023-11-24T08:12:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.308+0000 7f6f6a413640 1 -- 172.21.15.107:0/76211429 wait complete. 2023-11-24T08:12:37.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:01.979497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:03.154712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:37.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:37.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.544+0000 7f1b2700e640 1 Processor -- start 2023-11-24T08:12:37.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.545+0000 7f1b2700e640 1 -- start start 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 0x7f1b20161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b2006e160 0x7f1b200ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b2006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b200cd580 con 0x7f1b2006e160 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b20161a10 con 0x7f1b20146eb0 2023-11-24T08:12:37.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.546+0000 7f1b2700e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b20161cf0 con 0x7f1b200cd110 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b2006e160 0x7f1b200ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b2006e160 0x7f1b200ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51366/0 (socket says 172.21.15.107:51366) 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 -- 172.21.15.107:0/1331200999 learned_addr learned my addr 172.21.15.107:0/1331200999 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b24daa640 1 --2- 172.21.15.107:0/1331200999 >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 0x7f1b20161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b255ab640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b2006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 -- 172.21.15.107:0/1331200999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 msgr2=0x7f1b2006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b2006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 -- 172.21.15.107:0/1331200999 >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 msgr2=0x7f1b20161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 --2- 172.21.15.107:0/1331200999 >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 0x7f1b20161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b1ffff640 1 -- 172.21.15.107:0/1331200999 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20161fd0 con 0x7f1b2006e160 2023-11-24T08:12:37.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.547+0000 7f1b24daa640 1 --2- 172.21.15.107:0/1331200999 >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 0x7f1b20161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.548+0000 7f1b1ffff640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2006e160 0x7f1b200ccbd0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f1b2006ea50 tx=0x7f1b0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959e3e2ab9905e72 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b1f7fe640 1 -- 172.21.15.107:0/1331200999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08031870 con 0x7f1b2006e160 2023-11-24T08:12:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b1f7fe640 1 -- 172.21.15.107:0/1331200999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b080319d0 con 0x7f1b2006e160 2023-11-24T08:12:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b1f7fe640 1 -- 172.21.15.107:0/1331200999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08031d60 con 0x7f1b2006e160 2023-11-24T08:12:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 -- 172.21.15.107:0/1331200999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2006e160 msgr2=0x7f1b200ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2006e160 0x7f1b200ccbd0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f1b2006ea50 tx=0x7f1b0802b910 comp rx=0 tx=0).stop 2023-11-24T08:12:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 -- 172.21.15.107:0/1331200999 shutdown_connections 2023-11-24T08:12:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b2006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 --2- 172.21.15.107:0/1331200999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2006e160 0x7f1b200ccbd0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 --2- 172.21.15.107:0/1331200999 >> v2:172.21.15.120:3300/0 conn(0x7f1b20146eb0 0x7f1b20161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 -- 172.21.15.107:0/1331200999 >> 172.21.15.107:0/1331200999 conn(0x7f1b200b0c50 msgr2=0x7f1b200b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 -- 172.21.15.107:0/1331200999 shutdown_connections 2023-11-24T08:12:37.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.549+0000 7f1b2700e640 1 -- 172.21.15.107:0/1331200999 wait complete. 2023-11-24T08:12:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.551+0000 7f1b2700e640 1 Processor -- start 2023-11-24T08:12:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.551+0000 7f1b2700e640 1 -- start start 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b2006e160 0x7f1b2015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b201f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201f4190 con 0x7f1b20146eb0 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2015f320 con 0x7f1b2006e160 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.552+0000 7f1b2700e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201f3890 con 0x7f1b200cd110 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51374/0 (socket says 172.21.15.107:51374) 2023-11-24T08:12:37.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 -- 172.21.15.107:0/2458401931 learned_addr learned my addr 172.21.15.107:0/2458401931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b24daa640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b2006e160 0x7f1b2015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b1ffff640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b201f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 -- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 msgr2=0x7f1b201f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b201f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 -- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b2006e160 msgr2=0x7f1b2015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b2006e160 0x7f1b2015cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 -- 172.21.15.107:0/2458401931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b080094d0 con 0x7f1b20146eb0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.553+0000 7f1b255ab640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1b1800c570 tx=0x7f1b1800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.554+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b18004220 con 0x7f1b20146eb0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.554+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b18004380 con 0x7f1b20146eb0 2023-11-24T08:12:37.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.554+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b201f3b70 con 0x7f1b20146eb0 2023-11-24T08:12:37.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.554+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b18003830 con 0x7f1b20146eb0 2023-11-24T08:12:37.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.554+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2015f5c0 con 0x7f1b20146eb0 2023-11-24T08:12:37.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.555+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2004e8b0 con 0x7f1b20146eb0 2023-11-24T08:12:37.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.556+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f1b18003990 con 0x7f1b20146eb0 2023-11-24T08:12:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.557+0000 7f1b1d7fa640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 0x7f1b0006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.558+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b18088000 con 0x7f1b20146eb0 2023-11-24T08:12:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.558+0000 7f1b24daa640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 0x7f1b0006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.558+0000 7f1b24daa640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 0x7f1b0006e1a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1b20143d60 tx=0x7f1b100084f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.562+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b18054880 con 0x7f1b20146eb0 2023-11-24T08:12:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.769+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b2006ded0 con 0x7f1b20146eb0 2023-11-24T08:12:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.770+0000 7f1b1d7fa640 1 -- 172.21.15.107:0/2458401931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b18057ea0 con 0x7f1b20146eb0 2023-11-24T08:12:37.772 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.772+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 msgr2=0x7f1b0006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.772+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 0x7f1b0006e1a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1b20143d60 tx=0x7f1b100084f0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.772+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 msgr2=0x7f1b200acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.772+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1b1800c570 tx=0x7f1b1800ca30 comp rx=0 tx=0).stop 2023-11-24T08:12:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 shutdown_connections 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b0006bce0 0x7f1b0006e1a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b20146eb0 0x7f1b200acf80 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b200cd110 0x7f1b201f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 --2- 172.21.15.107:0/2458401931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b2006e160 0x7f1b2015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 >> 172.21.15.107:0/2458401931 conn(0x7f1b200b0c50 msgr2=0x7f1b20146120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 shutdown_connections 2023-11-24T08:12:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.773+0000 7f1b2700e640 1 -- 172.21.15.107:0/2458401931 wait complete. 2023-11-24T08:12:37.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:12:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.987+0000 7f307aed1640 1 Processor -- start 2023-11-24T08:12:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.987+0000 7f307aed1640 1 -- start start 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.988+0000 7f307aed1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f307415d3f0 0x7f30740d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.988+0000 7f307aed1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30740d0e40 0x7f30740d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.988+0000 7f307aed1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30740d15e0 con 0x7f307415d7d0 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.988+0000 7f307aed1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30740d0280 con 0x7f30740d0e40 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.988+0000 7f307aed1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30740d1300 con 0x7f307415d3f0 2023-11-24T08:12:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30740d0e40 0x7f30740d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30740d0e40 0x7f30740d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36810/0 (socket says 172.21.15.107:36810) 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 -- 172.21.15.107:0/1544026567 learned_addr learned my addr 172.21.15.107:0/1544026567 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f3079ecf640 1 --2- 172.21.15.107:0/1544026567 >> v2:172.21.15.120:3300/0 conn(0x7f307415d3f0 0x7f30740d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 -- 172.21.15.107:0/1544026567 >> v2:172.21.15.120:3300/0 conn(0x7f307415d3f0 msgr2=0x7f30740d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 --2- 172.21.15.107:0/1544026567 >> v2:172.21.15.120:3300/0 conn(0x7f307415d3f0 0x7f30740d1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 -- 172.21.15.107:0/1544026567 >> v1:172.21.15.120:6789/0 conn(0x7f307415d7d0 legacy=0x7f30740afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.989+0000 7f307a6d0640 1 -- 172.21.15.107:0/1544026567 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30740cfc30 con 0x7f30740d0e40 2023-11-24T08:12:37.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.990+0000 7f307a6d0640 1 --2- 172.21.15.107:0/1544026567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740d1d70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3074067c60 tx=0x7f306c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e647b613439b4c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f3078ecd640 1 -- 172.21.15.107:0/1544026567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c031870 con 0x7f30740d0e40 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f3078ecd640 1 -- 172.21.15.107:0/1544026567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c0319d0 con 0x7f30740d0e40 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f3078ecd640 1 -- 172.21.15.107:0/1544026567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c031d60 con 0x7f30740d0e40 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 -- 172.21.15.107:0/1544026567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 msgr2=0x7f30740d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 --2- 172.21.15.107:0/1544026567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740d1d70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3074067c60 tx=0x7f306c02b910 comp rx=0 tx=0).stop 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 -- 172.21.15.107:0/1544026567 shutdown_connections 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 --2- 172.21.15.107:0/1544026567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740d1d70 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 --2- 172.21.15.107:0/1544026567 >> v2:172.21.15.120:3300/0 conn(0x7f307415d3f0 0x7f30740d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.991+0000 7f307aed1640 1 -- 172.21.15.107:0/1544026567 >> 172.21.15.107:0/1544026567 conn(0x7f307409c5b0 msgr2=0x7f30740d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.992+0000 7f307aed1640 1 -- 172.21.15.107:0/1544026567 shutdown_connections 2023-11-24T08:12:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.992+0000 7f307aed1640 1 -- 172.21.15.107:0/1544026567 wait complete. 2023-11-24T08:12:37.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.993+0000 7f307aed1640 1 Processor -- start 2023-11-24T08:12:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.993+0000 7f307aed1640 1 -- start start 2023-11-24T08:12:37.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307415d3f0 0x7f30740b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307415d7d0 0x7f30740b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b5370 con 0x7f30740d0e40 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b4440 con 0x7f307415d3f0 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.994+0000 7f307aed1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b46f0 con 0x7f307415d7d0 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36824/0 (socket says 172.21.15.107:36824) 2023-11-24T08:12:37.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 -- 172.21.15.107:0/267394821 learned_addr learned my addr 172.21.15.107:0/267394821 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 -- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307415d7d0 msgr2=0x7f30740b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307415d7d0 0x7f30740b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 -- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307415d3f0 msgr2=0x7f30740b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307415d3f0 0x7f30740b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 -- 172.21.15.107:0/267394821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c0094d0 con 0x7f30740d0e40 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f3079ecf640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f306400ce10 tx=0x7f306400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.995+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064003b90 con 0x7f30740d0e40 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.996+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064003cf0 con 0x7f30740d0e40 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.996+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064003ea0 con 0x7f30740d0e40 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.996+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30740b49d0 con 0x7f30740d0e40 2023-11-24T08:12:37.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.996+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30740b4f90 con 0x7f30740d0e40 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.997+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307404e840 con 0x7f30740d0e40 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.998+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3064004240 con 0x7f30740d0e40 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.999+0000 7f30727fc640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 0x7f306006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.999+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3064012070 con 0x7f30740d0e40 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:37.999+0000 7f30796ce640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 0x7f306006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.000+0000 7f30796ce640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 0x7f306006e1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f30740b8a50 tx=0x7f3068006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.003+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3064055650 con 0x7f30740d0e40 2023-11-24T08:12:38.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.191+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f30740b27c0 con 0x7f30740d0e40 2023-11-24T08:12:38.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.191+0000 7f30727fc640 1 -- 172.21.15.107:0/267394821 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3064058c70 con 0x7f30740d0e40 2023-11-24T08:12:38.193 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:12:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 msgr2=0x7f306006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 0x7f306006e1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f30740b8a50 tx=0x7f3068006ba0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 msgr2=0x7f30740c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f306400ce10 tx=0x7f306400a660 comp rx=0 tx=0).stop 2023-11-24T08:12:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 shutdown_connections 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f306006bce0 0x7f306006e1a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307415d7d0 0x7f30740b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307415d3f0 0x7f30740b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 --2- 172.21.15.107:0/267394821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30740d0e40 0x7f30740c9370 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 >> 172.21.15.107:0/267394821 conn(0x7f307409c5b0 msgr2=0x7f307415c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 shutdown_connections 2023-11-24T08:12:38.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.194+0000 7f307aed1640 1 -- 172.21.15.107:0/267394821 wait complete. 2023-11-24T08:12:38.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:12:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.425+0000 7f96b230c640 1 Processor -- start 2023-11-24T08:12:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.425+0000 7f96b230c640 1 -- start start 2023-11-24T08:12:38.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96b230c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96ac06ff70 0x7f96ac0cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96b230c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f96ac0cf320 0x7f96ac13f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96b230c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac13eb00 con 0x7f96ac15f530 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96b230c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac0cf7b0 con 0x7f96ac06ff70 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96b230c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac13ede0 con 0x7f96ac0cf320 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96abfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f96ac15f530 0x7f96ac15f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40448/0 (socket says 172.21.15.107:40448) 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.427+0000 7f96abfff640 1 -- 172.21.15.107:0/120420911 learned_addr learned my addr 172.21.15.107:0/120420911 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:38.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96a3fff640 1 --2- 172.21.15.107:0/120420911 >> v2:172.21.15.107:3300/0 conn(0x7f96ac06ff70 0x7f96ac0cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96b08a9640 1 --2- 172.21.15.107:0/120420911 >> v2:172.21.15.120:3300/0 conn(0x7f96ac0cf320 0x7f96ac13f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96ab7fe640 1 -- 172.21.15.107:0/120420911 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262919487 0 0) 0x7f96ac13eb00 con 0x7f96ac15f530 2023-11-24T08:12:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96ab7fe640 1 -- 172.21.15.107:0/120420911 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698003520 con 0x7f96ac15f530 2023-11-24T08:12:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96a3fff640 1 -- 172.21.15.107:0/120420911 >> v2:172.21.15.120:3300/0 conn(0x7f96ac0cf320 msgr2=0x7f96ac13f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96a3fff640 1 --2- 172.21.15.107:0/120420911 >> v2:172.21.15.120:3300/0 conn(0x7f96ac0cf320 0x7f96ac13f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96a3fff640 1 -- 172.21.15.107:0/120420911 >> v1:172.21.15.107:6789/0 conn(0x7f96ac15f530 legacy=0x7f96ac15f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.428+0000 7f96a3fff640 1 -- 172.21.15.107:0/120420911 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac070750 con 0x7f96ac06ff70 2023-11-24T08:12:38.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.429+0000 7f96a3fff640 1 --2- 172.21.15.107:0/120420911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac0cede0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0b0d80 tx=0x7f969402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712e16b54cf32d4e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96ab7fe640 1 -- 172.21.15.107:0/120420911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969402e070 con 0x7f96ac06ff70 2023-11-24T08:12:38.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96ab7fe640 1 -- 172.21.15.107:0/120420911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96940315a0 con 0x7f96ac06ff70 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96ab7fe640 1 -- 172.21.15.107:0/120420911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694031930 con 0x7f96ac06ff70 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 -- 172.21.15.107:0/120420911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 msgr2=0x7f96ac0cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 --2- 172.21.15.107:0/120420911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac0cede0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0b0d80 tx=0x7f969402b840 comp rx=0 tx=0).stop 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 -- 172.21.15.107:0/120420911 shutdown_connections 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 --2- 172.21.15.107:0/120420911 >> v2:172.21.15.120:3300/0 conn(0x7f96ac0cf320 0x7f96ac13f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 --2- 172.21.15.107:0/120420911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac0cede0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 -- 172.21.15.107:0/120420911 >> 172.21.15.107:0/120420911 conn(0x7f96ac0b2e40 msgr2=0x7f96ac0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:38.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 -- 172.21.15.107:0/120420911 shutdown_connections 2023-11-24T08:12:38.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.430+0000 7f96b230c640 1 -- 172.21.15.107:0/120420911 wait complete. 2023-11-24T08:12:38.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.432+0000 7f96b230c640 1 Processor -- start 2023-11-24T08:12:38.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.432+0000 7f96b230c640 1 -- start start 2023-11-24T08:12:38.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 0x7f96ac0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96ac15f530 0x7f96ac0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0cd510 con 0x7f96ac06ff70 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0cc890 con 0x7f96ac0cf320 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.433+0000 7f96b230c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0ccb70 con 0x7f96ac15f530 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36848/0 (socket says 172.21.15.107:36848) 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 -- 172.21.15.107:0/1072816479 learned_addr learned my addr 172.21.15.107:0/1072816479 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:38.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96a3fff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 0x7f96ac0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96b08a9640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96ac15f530 0x7f96ac0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 -- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96ac15f530 msgr2=0x7f96ac0b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96ac15f530 0x7f96ac0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 -- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 msgr2=0x7f96ac0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 0x7f96ac0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 -- 172.21.15.107:0/1072816479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9694009400 con 0x7f96ac06ff70 2023-11-24T08:12:38.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96a3fff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 0x7f96ac0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:38.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.434+0000 7f96abfff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f969c00c410 tx=0x7f969c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.435+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c005ce0 con 0x7f96ac06ff70 2023-11-24T08:12:38.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.435+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c005e40 con 0x7f96ac06ff70 2023-11-24T08:12:38.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.435+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0072f0 con 0x7f96ac06ff70 2023-11-24T08:12:38.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.435+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac0cce50 con 0x7f96ac06ff70 2023-11-24T08:12:38.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.435+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac0ce060 con 0x7f96ac06ff70 2023-11-24T08:12:38.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.436+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac073e50 con 0x7f96ac06ff70 2023-11-24T08:12:38.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.437+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f969c007450 con 0x7f96ac06ff70 2023-11-24T08:12:38.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.438+0000 7f96a97fa640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 0x7f968c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.438+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f969c0879f0 con 0x7f96ac06ff70 2023-11-24T08:12:38.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.438+0000 7f96a3fff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 0x7f968c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.439+0000 7f96a3fff640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 0x7f968c06e1a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9694036000 tx=0x7f969403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.442+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96ac073e50 con 0x7f96ac06ff70 2023-11-24T08:12:38.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.624+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f96ac073e50 con 0x7f96ac06ff70 2023-11-24T08:12:38.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.625+0000 7f96a97fa640 1 -- 172.21.15.107:0/1072816479 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f969c017020 con 0x7f96ac06ff70 2023-11-24T08:12:38.627 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:12:38.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 msgr2=0x7f968c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 0x7f968c06e1a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9694036000 tx=0x7f969403d040 comp rx=0 tx=0).stop 2023-11-24T08:12:38.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 msgr2=0x7f96ac13df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f969c00c410 tx=0x7f969c00c8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 shutdown_connections 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f968c06bce0 0x7f968c06e1a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96ac15f530 0x7f96ac0b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac0cf320 0x7f96ac0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 --2- 172.21.15.107:0/1072816479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96ac06ff70 0x7f96ac13df90 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 >> 172.21.15.107:0/1072816479 conn(0x7f96ac0b2e40 msgr2=0x7f96ac0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 shutdown_connections 2023-11-24T08:12:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.627+0000 7f96b230c640 1 -- 172.21.15.107:0/1072816479 wait complete. 2023-11-24T08:12:38.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:12:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.843+0000 7efd49b1d640 1 Processor -- start 2023-11-24T08:12:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.843+0000 7efd49b1d640 1 -- start start 2023-11-24T08:12:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd49b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd441660b0 0x7efd44162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd49b1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efd44162540 0x7efd440c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd49b1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd44166510 con 0x7efd440ccbd0 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd49b1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd441655f0 con 0x7efd441660b0 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd49b1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd440cc380 con 0x7efd44162540 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd3affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd441660b0 0x7efd44162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd3affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd441660b0 0x7efd44162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36864/0 (socket says 172.21.15.107:36864) 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.845+0000 7efd3affd640 1 -- 172.21.15.107:0/76449812 learned_addr learned my addr 172.21.15.107:0/76449812 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.846+0000 7efd3affd640 1 -- 172.21.15.107:0/76449812 >> v2:172.21.15.120:3300/0 conn(0x7efd44162540 msgr2=0x7efd440c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.846+0000 7efd3affd640 1 --2- 172.21.15.107:0/76449812 >> v2:172.21.15.120:3300/0 conn(0x7efd44162540 0x7efd440c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.846+0000 7efd3affd640 1 -- 172.21.15.107:0/76449812 >> v1:172.21.15.120:6789/0 conn(0x7efd440ccbd0 legacy=0x7efd440ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.846+0000 7efd3affd640 1 -- 172.21.15.107:0/76449812 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd44162b30 con 0x7efd441660b0 2023-11-24T08:12:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.847+0000 7efd3affd640 1 --2- 172.21.15.107:0/76449812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd44162000 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7efd440ae430 tx=0x7efd2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846b25211afaf1e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.847+0000 7efd42ffd640 1 -- 172.21.15.107:0/76449812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c02ca80 con 0x7efd441660b0 2023-11-24T08:12:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.847+0000 7efd42ffd640 1 -- 172.21.15.107:0/76449812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c02cbe0 con 0x7efd441660b0 2023-11-24T08:12:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.847+0000 7efd42ffd640 1 -- 172.21.15.107:0/76449812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0343f0 con 0x7efd441660b0 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 -- 172.21.15.107:0/76449812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 msgr2=0x7efd44162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 --2- 172.21.15.107:0/76449812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd44162000 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7efd440ae430 tx=0x7efd2c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 -- 172.21.15.107:0/76449812 shutdown_connections 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 --2- 172.21.15.107:0/76449812 >> v2:172.21.15.120:3300/0 conn(0x7efd44162540 0x7efd440c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 --2- 172.21.15.107:0/76449812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd44162000 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 -- 172.21.15.107:0/76449812 >> 172.21.15.107:0/76449812 conn(0x7efd440b0c50 msgr2=0x7efd440b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:38.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 -- 172.21.15.107:0/76449812 shutdown_connections 2023-11-24T08:12:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.848+0000 7efd49b1d640 1 -- 172.21.15.107:0/76449812 wait complete. 2023-11-24T08:12:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.850+0000 7efd49b1d640 1 Processor -- start 2023-11-24T08:12:38.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.850+0000 7efd49b1d640 1 -- start start 2023-11-24T08:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd440ccbd0 0x7efd44160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd44162540 0x7efd441612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440704c0 con 0x7efd441660b0 2023-11-24T08:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440706a0 con 0x7efd44162540 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd49b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd44070980 con 0x7efd440ccbd0 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36872/0 (socket says 172.21.15.107:36872) 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 -- 172.21.15.107:0/2254821554 learned_addr learned my addr 172.21.15.107:0/2254821554 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 -- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd440ccbd0 msgr2=0x7efd44160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd440ccbd0 0x7efd44160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 -- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd44162540 msgr2=0x7efd441612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.851+0000 7efd43fff640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd44162540 0x7efd441612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd43fff640 1 -- 172.21.15.107:0/2254821554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c009b30 con 0x7efd441660b0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd43fff640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fd0 tx=0x7efd3c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c011830 con 0x7efd441660b0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3c011990 con 0x7efd441660b0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c011d20 con 0x7efd441660b0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd44070c60 con 0x7efd441660b0 2023-11-24T08:12:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.852+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd440c6610 con 0x7efd441660b0 2023-11-24T08:12:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.853+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd44071180 con 0x7efd441660b0 2023-11-24T08:12:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.854+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7efd3c01b460 con 0x7efd441660b0 2023-11-24T08:12:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.856+0000 7efd40ff9640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 0x7efd2406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.856+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efd3c088280 con 0x7efd441660b0 2023-11-24T08:12:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.856+0000 7efd437fe640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 0x7efd2406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.857+0000 7efd437fe640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 0x7efd2406e1a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd34001560 tx=0x7efd34007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:38.860+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd44071180 con 0x7efd441660b0 2023-11-24T08:12:39.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.062+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efd44071180 con 0x7efd441660b0 2023-11-24T08:12:39.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.063+0000 7efd40ff9640 1 -- 172.21.15.107:0/2254821554 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) v1 ==== 80+0+0 (secure 0 0 0) 0x7efd44071180 con 0x7efd441660b0 2023-11-24T08:12:39.065 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.065+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 msgr2=0x7efd2406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.065+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 0x7efd2406e1a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd34001560 tx=0x7efd34007990 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.065+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 msgr2=0x7efd440c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.065+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fd0 tx=0x7efd3c00b8e0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 shutdown_connections 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd2406bce0 0x7efd2406e1a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd441660b0 0x7efd440c60d0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd44162540 0x7efd441612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 --2- 172.21.15.107:0/2254821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd440ccbd0 0x7efd44160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 >> 172.21.15.107:0/2254821554 conn(0x7efd440b0c50 msgr2=0x7efd440c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:39.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 shutdown_connections 2023-11-24T08:12:39.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.066+0000 7efd49b1d640 1 -- 172.21.15.107:0/2254821554 wait complete. 2023-11-24T08:12:39.086 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:12:39.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:39.087 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:12:39.200 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:12:39.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:12:39.201 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:12:39.251 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:39.252 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:39.310 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:12:39.310 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:39.311 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:39.370 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:12:39.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:39.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.589+0000 7f24948d9640 1 Processor -- start 2023-11-24T08:12:39.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.589+0000 7f24948d9640 1 -- start start 2023-11-24T08:12:39.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f24948d9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f249006d6c0 0x7f249006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f24948d9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 0x7f24900ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f24948d9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249006e060 con 0x7f249013d120 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f24948d9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249013f9e0 con 0x7f249006d6c0 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f24948d9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249013fcc0 con 0x7f24900cdbf0 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f248ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f249013d120 0x7f249015af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40450/0 (socket says 172.21.15.107:40450) 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.590+0000 7f248ed9d640 1 -- 172.21.15.107:0/478655367 learned_addr learned my addr 172.21.15.107:0/478655367 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590032230 0 0) 0x7f249006e060 con 0x7f249013d120 2023-11-24T08:12:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c003520 con 0x7f249013d120 2023-11-24T08:12:39.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579144595 0 0) 0x7f247c003520 con 0x7f249013d120 2023-11-24T08:12:39.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 msgr2=0x7f24900ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:39.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248f59e640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 0x7f24900ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 0x7f24900ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 >> v2:172.21.15.120:3300/0 conn(0x7f249006d6c0 msgr2=0x7f249006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> v2:172.21.15.120:3300/0 conn(0x7f249006d6c0 0x7f249006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 --2- 172.21.15.107:0/478655367 >> v2:172.21.15.120:3300/0 conn(0x7f249006d6c0 0x7f249006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249013ffa0 con 0x7f249013d120 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2480001670 con 0x7f249013d120 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 >> v1:172.21.15.107:6789/0 conn(0x7f249013d120 legacy=0x7f249015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247c0039a0 0x7f247c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247c0047c0 0x7f247c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 0x7f247c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249013f9e0 con 0x7f247c0047c0 2023-11-24T08:12:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249013fcc0 con 0x7f247c0039a0 2023-11-24T08:12:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c005640 con 0x7f24900cdbf0 2023-11-24T08:12:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.591+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> v2:172.21.15.120:3300/0 conn(0x7f249006d6c0 0x7f249006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247c0047c0 0x7f247c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f248ed9d640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247c0039a0 0x7f247c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 -- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 msgr2=0x7f247c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24900cdbf0 0x7f247c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 -- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247c0039a0 msgr2=0x7f247c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247c0039a0 0x7f247c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f2487fff640 1 -- 172.21.15.107:0/478655367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c003520 con 0x7f247c0047c0 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.592+0000 7f248ed9d640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247c0039a0 0x7f247c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.593+0000 7f2487fff640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247c0047c0 0x7f247c004c40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f247800aa70 tx=0x7f247c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd5c480cf5bf037 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.593+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478004220 con 0x7f247c0047c0 2023-11-24T08:12:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.593+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2478004380 con 0x7f247c0047c0 2023-11-24T08:12:39.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.593+0000 7f248e59c640 1 -- 172.21.15.107:0/478655367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478003830 con 0x7f247c0047c0 2023-11-24T08:12:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 -- 172.21.15.107:0/478655367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247c0047c0 msgr2=0x7f247c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 --2- 172.21.15.107:0/478655367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247c0047c0 0x7f247c004c40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f247800aa70 tx=0x7f247c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f248f59e640 1 -- 172.21.15.107:0/478655367 reap_dead start 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 -- 172.21.15.107:0/478655367 shutdown_connections 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 -- 172.21.15.107:0/478655367 >> 172.21.15.107:0/478655367 conn(0x7f24900cb830 msgr2=0x7f24900cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 -- 172.21.15.107:0/478655367 shutdown_connections 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.594+0000 7f24948d9640 1 -- 172.21.15.107:0/478655367 wait complete. 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.596+0000 7f24948d9640 1 Processor -- start 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.596+0000 7f24948d9640 1 -- start start 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 0x7f2490158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2490140ce0 0x7f249015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249015c870 con 0x7f249015b210 2023-11-24T08:12:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249015bc60 con 0x7f2490140ce0 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f24948d9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249015bf40 con 0x7f247c0047c0 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f2487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f2487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36898/0 (socket says 172.21.15.107:36898) 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f2487fff640 1 -- 172.21.15.107:0/2628396215 learned_addr learned my addr 172.21.15.107:0/2628396215 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f248ed9d640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 0x7f2490158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.597+0000 7f248f59e640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2490140ce0 0x7f249015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 -- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 msgr2=0x7f2490158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 0x7f2490158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 -- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2490140ce0 msgr2=0x7f249015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2490140ce0 0x7f249015cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 -- 172.21.15.107:0/2628396215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249015a9d0 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f248ed9d640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 0x7f2490158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f2487fff640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f247800a120 tx=0x7f2478004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478037810 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247803d040 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249015c220 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478037bb0 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.598+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24900af340 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.599+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24900af500 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.600+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f2478037d10 con 0x7f249015b210 2023-11-24T08:12:39.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.601+0000 7f24877fe640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 0x7f247006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:39.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.601+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f24780a98f0 con 0x7f249015b210 2023-11-24T08:12:39.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.602+0000 7f248ed9d640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 0x7f247006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:39.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.602+0000 7f248ed9d640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 0x7f247006e1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2480001eb0 tx=0x7f2480009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:39.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.606+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2478076450 con 0x7f249015b210 2023-11-24T08:12:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.836+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f249004e840 con 0x7f249015b210 2023-11-24T08:12:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.837+0000 7f24877fe640 1 -- 172.21.15.107:0/2628396215 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f2478079a70 con 0x7f249015b210 2023-11-24T08:12:39.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:39.840 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 52 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.839+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 msgr2=0x7f247006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.839+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 0x7f247006e1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2480001eb0 tx=0x7f2480009040 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.839+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 msgr2=0x7f24901407a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.839+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f247800a120 tx=0x7f2478004730 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 shutdown_connections 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f247006bce0 0x7f247006e1a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2490140ce0 0x7f249015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249015b210 0x7f24901407a0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 --2- 172.21.15.107:0/2628396215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f247c0047c0 0x7f2490158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 >> 172.21.15.107:0/2628396215 conn(0x7f24900cb830 msgr2=0x7f24900cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 shutdown_connections 2023-11-24T08:12:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:39.840+0000 7f24948d9640 1 -- 172.21.15.107:0/2628396215 wait complete. 2023-11-24T08:12:39.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:40.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.070+0000 7f0135333640 1 Processor -- start 2023-11-24T08:12:40.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.070+0000 7f0135333640 1 -- start start 2023-11-24T08:12:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.071+0000 7f0135333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f013015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f0135333640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013015ab80 con 0x7f01300b2630 2023-11-24T08:12:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f0135333640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013015e680 con 0x7f01300b2a70 2023-11-24T08:12:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f0135333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013015a990 con 0x7f01300b2250 2023-11-24T08:12:40.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f012e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f01300b2630 0x7f01300ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40464/0 (socket says 172.21.15.107:40464) 2023-11-24T08:12:40.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f012e7fc640 1 -- 172.21.15.107:0/2258244459 learned_addr learned my addr 172.21.15.107:0/2258244459 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:40.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476611369 0 0) 0x7f013015ab80 con 0x7f01300b2630 2023-11-24T08:12:40.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118005730 con 0x7f01300b2630 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.072+0000 7f012effd640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f013015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600192786 0 0) 0x7f0118005730 con 0x7f01300b2630 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 msgr2=0x7f013015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f013015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 >> v1:172.21.15.120:6789/0 conn(0x7f01300b2a70 legacy=0x7f013015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013015e960 con 0x7f01300b2630 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0124001670 con 0x7f01300b2630 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 >> v1:172.21.15.107:6789/0 conn(0x7f01300b2630 legacy=0x7f01300ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0118005bb0 0x7f0118005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01180069d0 0x7f0118006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f0118007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011800dd20 con 0x7f01180069d0 2023-11-24T08:12:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011800de60 con 0x7f0118005bb0 2023-11-24T08:12:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011800e020 con 0x7f01300b2250 2023-11-24T08:12:40.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.073+0000 7f012e7fc640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01180069d0 0x7f0118006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012effd640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0118005bb0 0x7f0118005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012f7fe640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f0118007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012e7fc640 1 -- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 msgr2=0x7f0118007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012e7fc640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f0118007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012e7fc640 1 -- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0118005bb0 msgr2=0x7f0118005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012e7fc640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0118005bb0 0x7f0118005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012e7fc640 1 -- 172.21.15.107:0/2258244459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118005730 con 0x7f01180069d0 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012effd640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0118005bb0 0x7f0118005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.074+0000 7f012f7fe640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300b2250 0x7f0118007420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f012e7fc640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01180069d0 0x7f0118006e50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f012400c0e0 tx=0x7f0118007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b01a851479dd06b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124005ad0 con 0x7f01180069d0 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124005c30 con 0x7f01180069d0 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f012dffb640 1 -- 172.21.15.107:0/2258244459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012402c7f0 con 0x7f01180069d0 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f0135333640 1 -- 172.21.15.107:0/2258244459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01180069d0 msgr2=0x7f0118006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f0135333640 1 --2- 172.21.15.107:0/2258244459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01180069d0 0x7f0118006e50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f012400c0e0 tx=0x7f0118007a00 comp rx=0 tx=0).stop 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.075+0000 7f012f7fe640 1 -- 172.21.15.107:0/2258244459 reap_dead start 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.076+0000 7f0135333640 1 -- 172.21.15.107:0/2258244459 shutdown_connections 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.076+0000 7f0135333640 1 -- 172.21.15.107:0/2258244459 >> 172.21.15.107:0/2258244459 conn(0x7f013006a920 msgr2=0x7f013006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.076+0000 7f0135333640 1 -- 172.21.15.107:0/2258244459 shutdown_connections 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.076+0000 7f0135333640 1 -- 172.21.15.107:0/2258244459 wait complete. 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.077+0000 7f0135333640 1 Processor -- start 2023-11-24T08:12:40.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.078+0000 7f0135333640 1 -- start start 2023-11-24T08:12:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.078+0000 7f0135333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01180069d0 0x7f01300722a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.078+0000 7f0135333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300aeb50 0x7f01301537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f0135333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f0135333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300705c0 con 0x7f0130148390 2023-11-24T08:12:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f0135333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300cbfe0 con 0x7f01180069d0 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f0135333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300cc280 con 0x7f01300aeb50 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36920/0 (socket says 172.21.15.107:36920) 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 -- 172.21.15.107:0/3133971056 learned_addr learned my addr 172.21.15.107:0/3133971056 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012e7fc640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300aeb50 0x7f01301537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 -- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300aeb50 msgr2=0x7f01301537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300aeb50 0x7f01301537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 -- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01180069d0 msgr2=0x7f01300722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01180069d0 0x7f01300722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 -- 172.21.15.107:0/3133971056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300cd7c0 con 0x7f0130148390 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.079+0000 7f012f7fe640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f01300b2e50 tx=0x7f012800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.080+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128003970 con 0x7f0130148390 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.080+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0128003ad0 con 0x7f0130148390 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.080+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128003e80 con 0x7f0130148390 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.080+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0130070810 con 0x7f0130148390 2023-11-24T08:12:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.080+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0130148a90 con 0x7f0130148390 2023-11-24T08:12:40.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.081+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130070d30 con 0x7f0130148390 2023-11-24T08:12:40.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.082+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f0128005070 con 0x7f0130148390 2023-11-24T08:12:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.083+0000 7f01177fe640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 0x7f011c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.083+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0128088040 con 0x7f0130148390 2023-11-24T08:12:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.083+0000 7f012effd640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 0x7f011c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.084+0000 7f012effd640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 0x7f011c06e1d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0120006fd0 tx=0x7f0120009340 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.088+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0130070d30 con 0x7f0130148390 2023-11-24T08:12:40.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.280+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0130070d30 con 0x7f0130148390 2023-11-24T08:12:40.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.282+0000 7f01177fe640 1 -- 172.21.15.107:0/3133971056 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16724 (secure 0 0 0) 0x7f0130070d30 con 0x7f0130148390 2023-11-24T08:12:40.285 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:40.285 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:38.692986+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.23","mappings":[{"from":1,"to":5}]},{"pgid":"13.30","mappings":[{"from":6,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 msgr2=0x7f011c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 0x7f011c06e1d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0120006fd0 tx=0x7f0120009340 comp rx=0 tx=0).stop 2023-11-24T08:12:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 msgr2=0x7f013014d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f01300b2e50 tx=0x7f012800d9d0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 shutdown_connections 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f011c06bd10 0x7f011c06e1d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0130148390 0x7f013014d760 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01300aeb50 0x7f01301537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 --2- 172.21.15.107:0/3133971056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01180069d0 0x7f01300722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 >> 172.21.15.107:0/3133971056 conn(0x7f013006a920 msgr2=0x7f01300ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 shutdown_connections 2023-11-24T08:12:40.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.285+0000 7f0135333640 1 -- 172.21.15.107:0/3133971056 wait complete. 2023-11-24T08:12:40.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:01.979497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:03.154712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:40.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:40.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.520+0000 7f72e2948640 1 Processor -- start 2023-11-24T08:12:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.521+0000 7f72e2948640 1 -- start start 2023-11-24T08:12:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.522+0000 7f72e2948640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f72dc1660b0 0x7f72dc162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.522+0000 7f72e2948640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f72dc162540 0x7f72dc0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.522+0000 7f72e2948640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc166510 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.522+0000 7f72e2948640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72dc1655f0 con 0x7f72dc162540 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.522+0000 7f72e2948640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72dc0cc380 con 0x7f72dc1660b0 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72dbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72dc0ccbd0 0x7f72dc0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40478/0 (socket says 172.21.15.107:40478) 2023-11-24T08:12:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72dbfff640 1 -- 172.21.15.107:0/2355525529 learned_addr learned my addr 172.21.15.107:0/2355525529 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72e0ee5640 1 --2- 172.21.15.107:0/2355525529 >> v2:172.21.15.107:3300/0 conn(0x7f72dc162540 0x7f72dc0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72d3fff640 1 --2- 172.21.15.107:0/2355525529 >> v2:172.21.15.120:3300/0 conn(0x7f72dc1660b0 0x7f72dc162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72db7fe640 1 -- 172.21.15.107:0/2355525529 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652552003 0 0) 0x7f72cc001110 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72db7fe640 1 -- 172.21.15.107:0/2355525529 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c8003520 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72e0ee5640 1 -- 172.21.15.107:0/2355525529 >> v2:172.21.15.120:3300/0 conn(0x7f72dc1660b0 msgr2=0x7f72dc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72e0ee5640 1 --2- 172.21.15.107:0/2355525529 >> v2:172.21.15.120:3300/0 conn(0x7f72dc1660b0 0x7f72dc162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72e0ee5640 1 -- 172.21.15.107:0/2355525529 >> v1:172.21.15.107:6789/0 conn(0x7f72dc0ccbd0 legacy=0x7f72dc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.523+0000 7f72e0ee5640 1 -- 172.21.15.107:0/2355525529 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc162b30 con 0x7f72dc162540 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.524+0000 7f72e0ee5640 1 --2- 172.21.15.107:0/2355525529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc162540 0x7f72dc0c9650 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f72dc06e700 tx=0x7f72d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e8e537f89ea3c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.525+0000 7f72db7fe640 1 -- 172.21.15.107:0/2355525529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d402cc80 con 0x7f72dc162540 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.525+0000 7f72db7fe640 1 -- 172.21.15.107:0/2355525529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d402cde0 con 0x7f72dc162540 2023-11-24T08:12:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.525+0000 7f72db7fe640 1 -- 172.21.15.107:0/2355525529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4034660 con 0x7f72dc162540 2023-11-24T08:12:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.525+0000 7f72e2948640 1 -- 172.21.15.107:0/2355525529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc162540 msgr2=0x7f72dc0c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.525+0000 7f72e2948640 1 --2- 172.21.15.107:0/2355525529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc162540 0x7f72dc0c9650 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f72dc06e700 tx=0x7f72d402bed0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 -- 172.21.15.107:0/2355525529 shutdown_connections 2023-11-24T08:12:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 --2- 172.21.15.107:0/2355525529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc162540 0x7f72dc0c9650 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 --2- 172.21.15.107:0/2355525529 >> v2:172.21.15.120:3300/0 conn(0x7f72dc1660b0 0x7f72dc162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 -- 172.21.15.107:0/2355525529 >> 172.21.15.107:0/2355525529 conn(0x7f72dc0b0c50 msgr2=0x7f72dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:40.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 -- 172.21.15.107:0/2355525529 shutdown_connections 2023-11-24T08:12:40.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.526+0000 7f72e2948640 1 -- 172.21.15.107:0/2355525529 wait complete. 2023-11-24T08:12:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.527+0000 7f72e2948640 1 Processor -- start 2023-11-24T08:12:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 -- start start 2023-11-24T08:12:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc162540 0x7f72dc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72dc1660b0 0x7f72dc14a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc071380 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc1558f0 con 0x7f72dc162540 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.528+0000 7f72e2948640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc155b40 con 0x7f72dc1660b0 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36946/0 (socket says 172.21.15.107:36946) 2023-11-24T08:12:40.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 -- 172.21.15.107:0/1551810044 learned_addr learned my addr 172.21.15.107:0/1551810044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:40.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 -- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72dc1660b0 msgr2=0x7f72dc14a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72dc1660b0 0x7f72dc14a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 -- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc162540 msgr2=0x7f72dc070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc162540 0x7f72dc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 -- 172.21.15.107:0/1551810044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d4009a70 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.529+0000 7f72dbfff640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00c700 tx=0x7f72cc00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.530+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc005ce0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.530+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc005e40 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.530+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc007290 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.530+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc155e20 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.530+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc1563e0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.531+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc1565d0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.532+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f72cc0074b0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.533+0000 7f72d97fa640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 0x7f72bc06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.533+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f72cc0877d0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.533+0000 7f72d3fff640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 0x7f72bc06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.534+0000 7f72d3fff640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 0x7f72bc06e1a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006fd0 tx=0x7f72c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.537+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72cc057ae0 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.729+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72dc166510 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.730+0000 7f72d97fa640 1 -- 172.21.15.107:0/1551810044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72dc166510 con 0x7f72dc0ccbd0 2023-11-24T08:12:40.732 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 msgr2=0x7f72bc06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 0x7f72bc06e1a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006fd0 tx=0x7f72c4008040 comp rx=0 tx=0).stop 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 msgr2=0x7f72dc156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00c700 tx=0x7f72cc00cb00 comp rx=0 tx=0).stop 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 shutdown_connections 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f72bc06bce0 0x7f72bc06e1a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72dc1660b0 0x7f72dc14a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc162540 0x7f72dc070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 --2- 172.21.15.107:0/1551810044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc0ccbd0 0x7f72dc156ca0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.732+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 >> 172.21.15.107:0/1551810044 conn(0x7f72dc0b0c50 msgr2=0x7f72dc0af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:40.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.733+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 shutdown_connections 2023-11-24T08:12:40.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.733+0000 7f72e2948640 1 -- 172.21.15.107:0/1551810044 wait complete. 2023-11-24T08:12:40.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:12:40.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.971+0000 7f79124ad640 1 Processor -- start 2023-11-24T08:12:40.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.972+0000 7f79124ad640 1 -- start start 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 0x7f790c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f790c0c57e0 0x7f790c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c15b820 con 0x7f790c0c57e0 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c0cc3a0 con 0x7f790c0cbf20 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f79124ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c15eea0 con 0x7f790c15b440 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f7910a4a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f790c0c57e0 0x7f790c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f7910a4a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f790c0c57e0 0x7f790c0c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36960/0 (socket says 172.21.15.107:36960) 2023-11-24T08:12:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.973+0000 7f7910a4a640 1 -- 172.21.15.107:0/1213902723 learned_addr learned my addr 172.21.15.107:0/1213902723 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f7910a4a640 1 -- 172.21.15.107:0/1213902723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 msgr2=0x7f790c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f790bfff640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f790b7fe640 1 --2- 172.21.15.107:0/1213902723 >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 0x7f790c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f7910a4a640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f7910a4a640 1 -- 172.21.15.107:0/1213902723 >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 msgr2=0x7f790c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f7910a4a640 1 --2- 172.21.15.107:0/1213902723 >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 0x7f790c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f7910a4a640 1 -- 172.21.15.107:0/1213902723 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0abf30 con 0x7f790c0c57e0 2023-11-24T08:12:40.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.974+0000 7f790b7fe640 1 --2- 172.21.15.107:0/1213902723 >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 0x7f790c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:40.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.975+0000 7f7910a4a640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c0c3ec0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f790c0cee60 tx=0x7f790402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b03dff36ab9064f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.975+0000 7f790affd640 1 -- 172.21.15.107:0/1213902723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79040317e0 con 0x7f790c0c57e0 2023-11-24T08:12:40.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.975+0000 7f790affd640 1 -- 172.21.15.107:0/1213902723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7904031940 con 0x7f790c0c57e0 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f790affd640 1 -- 172.21.15.107:0/1213902723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904031cd0 con 0x7f790c0c57e0 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 -- 172.21.15.107:0/1213902723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 msgr2=0x7f790c0c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c0c3ec0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f790c0cee60 tx=0x7f790402b880 comp rx=0 tx=0).stop 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 -- 172.21.15.107:0/1213902723 shutdown_connections 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c0c3ec0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 --2- 172.21.15.107:0/1213902723 >> v2:172.21.15.120:3300/0 conn(0x7f790c0cbf20 0x7f790c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 --2- 172.21.15.107:0/1213902723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 -- 172.21.15.107:0/1213902723 >> 172.21.15.107:0/1213902723 conn(0x7f790c0b0c50 msgr2=0x7f790c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 -- 172.21.15.107:0/1213902723 shutdown_connections 2023-11-24T08:12:40.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.976+0000 7f79124ad640 1 -- 172.21.15.107:0/1213902723 wait complete. 2023-11-24T08:12:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.978+0000 7f79124ad640 1 Processor -- start 2023-11-24T08:12:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.978+0000 7f79124ad640 1 -- start start 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f790c0cbf20 0x7f790c09be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c1f4800 con 0x7f790c0c57e0 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0c6270 con 0x7f790c0cbf20 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f79124ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0c6550 con 0x7f790c15b440 2023-11-24T08:12:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.979+0000 7f790bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36966/0 (socket says 172.21.15.107:36966) 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 -- 172.21.15.107:0/3419247613 learned_addr learned my addr 172.21.15.107:0/3419247613 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f7910a4a640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790b7fe640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f790c0cbf20 0x7f790c09be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 -- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 msgr2=0x7f790c0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 -- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f790c0cbf20 msgr2=0x7f790c09be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f790c0cbf20 0x7f790c09be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 -- 172.21.15.107:0/3419247613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040094d0 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.980+0000 7f790bfff640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00ccd0 tx=0x7f78fc00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.981+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc003990 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.981+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc003af0 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.981+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc004020 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.981+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c0c6830 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.981+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c0c6df0 con 0x7f790c0c57e0 2023-11-24T08:12:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.982+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c0c7010 con 0x7f790c0c57e0 2023-11-24T08:12:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.983+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f78fc004180 con 0x7f790c0c57e0 2023-11-24T08:12:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.984+0000 7f7908ff9640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 0x7f78ec06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.984+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f78fc088520 con 0x7f790c0c57e0 2023-11-24T08:12:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.985+0000 7f790b7fe640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 0x7f78ec06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.985+0000 7f790b7fe640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 0x7f78ec06e1a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f790c0ae2d0 tx=0x7f78f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:40.989+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78fc054fc0 con 0x7f790c0c57e0 2023-11-24T08:12:41.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.180+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f790c04e8b0 con 0x7f790c0c57e0 2023-11-24T08:12:41.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.180+0000 7f7908ff9640 1 -- 172.21.15.107:0/3419247613 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78fc01d0a0 con 0x7f790c0c57e0 2023-11-24T08:12:41.182 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 msgr2=0x7f78ec06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 0x7f78ec06e1a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f790c0ae2d0 tx=0x7f78f4008040 comp rx=0 tx=0).stop 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 msgr2=0x7f790c09b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00ccd0 tx=0x7f78fc00a5c0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 shutdown_connections 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78ec06bce0 0x7f78ec06e1a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f790c15b440 0x7f790c0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f790c0cbf20 0x7f790c09be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 --2- 172.21.15.107:0/3419247613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790c0c57e0 0x7f790c09b900 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 >> 172.21.15.107:0/3419247613 conn(0x7f790c0b0c50 msgr2=0x7f790c071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 shutdown_connections 2023-11-24T08:12:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.183+0000 7f79124ad640 1 -- 172.21.15.107:0/3419247613 wait complete. 2023-11-24T08:12:41.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:12:41.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:12:41.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:12:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.397+0000 7f4f2ee3d640 1 Processor -- start 2023-11-24T08:12:41.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.398+0000 7f4f2ee3d640 1 -- start start 2023-11-24T08:12:41.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f2ee3d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f2815b440 0x7f4f280cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f2ee3d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4f280c59b0 0x7f4f2806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f2ee3d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280c5dc0 con 0x7f4f280cbf20 2023-11-24T08:12:41.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f2ee3d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f280cc3d0 con 0x7f4f2815b440 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f2ee3d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2815b820 con 0x7f4f280c59b0 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.399+0000 7f4f27fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4f280cbf20 0x7f4f280c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40482/0 (socket says 172.21.15.107:40482) 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f27fff640 1 -- 172.21.15.107:0/3716196740 learned_addr learned my addr 172.21.15.107:0/3716196740 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/3716196740 >> v2:172.21.15.107:3300/0 conn(0x7f4f2815b440 0x7f4f280cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2d3da640 1 --2- 172.21.15.107:0/3716196740 >> v2:172.21.15.120:3300/0 conn(0x7f4f280c59b0 0x7f4f2806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f277fe640 1 -- 172.21.15.107:0/3716196740 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940423435 0 0) 0x7f4f280c5dc0 con 0x7f4f280cbf20 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f277fe640 1 -- 172.21.15.107:0/3716196740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f18003520 con 0x7f4f280cbf20 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/3716196740 >> v2:172.21.15.120:3300/0 conn(0x7f4f280c59b0 msgr2=0x7f4f2806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/3716196740 >> v2:172.21.15.120:3300/0 conn(0x7f4f280c59b0 0x7f4f2806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/3716196740 >> v1:172.21.15.107:6789/0 conn(0x7f4f280cbf20 legacy=0x7f4f280c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.400+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/3716196740 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2815eea0 con 0x7f4f2815b440 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.401+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/3716196740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f2815b440 0x7f4f280cb9e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4f280cb910 tx=0x7f4f1c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2406c76ae201cc89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.401+0000 7f4f277fe640 1 -- 172.21.15.107:0/3716196740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c02e070 con 0x7f4f2815b440 2023-11-24T08:12:41.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f277fe640 1 -- 172.21.15.107:0/3716196740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c0315a0 con 0x7f4f2815b440 2023-11-24T08:12:41.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f277fe640 1 -- 172.21.15.107:0/3716196740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c031930 con 0x7f4f2815b440 2023-11-24T08:12:41.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/3716196740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f2815b440 msgr2=0x7f4f280cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/3716196740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f2815b440 0x7f4f280cb9e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4f280cb910 tx=0x7f4f1c02b840 comp rx=0 tx=0).stop 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/3716196740 shutdown_connections 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/3716196740 >> v2:172.21.15.120:3300/0 conn(0x7f4f280c59b0 0x7f4f2806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/3716196740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f2815b440 0x7f4f280cb9e0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/3716196740 >> 172.21.15.107:0/3716196740 conn(0x7f4f280b0c50 msgr2=0x7f4f280b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/3716196740 shutdown_connections 2023-11-24T08:12:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.402+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/3716196740 wait complete. 2023-11-24T08:12:41.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.404+0000 7f4f2ee3d640 1 Processor -- start 2023-11-24T08:12:41.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.404+0000 7f4f2ee3d640 1 -- start start 2023-11-24T08:12:41.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f280cbf20 0x7f4f280b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 0x7f4f2815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2815de10 con 0x7f4f280c59b0 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f280b3e70 con 0x7f4f280cbf20 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2ee3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f280b4150 con 0x7f4f2815b440 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.405+0000 7f4f2cbd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36990/0 (socket says 172.21.15.107:36990) 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/1575721059 learned_addr learned my addr 172.21.15.107:0/1575721059 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:41.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f27fff640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f280cbf20 0x7f4f280b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2d3da640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 0x7f4f2815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 msgr2=0x7f4f2815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 0x7f4f2815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f280cbf20 msgr2=0x7f4f280b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f280cbf20 0x7f4f280b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 -- 172.21.15.107:0/1575721059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c009400 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2d3da640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 0x7f4f2815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.406+0000 7f4f2cbd9640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009890 tx=0x7f4f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.407+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c032840 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.407+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c031f00 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.407+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c032be0 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.407+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f280b4430 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.407+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2809a200 con 0x7f4f280c59b0 2023-11-24T08:12:41.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.408+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2804e8b0 con 0x7f4f280c59b0 2023-11-24T08:12:41.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.409+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f4f1c032d40 con 0x7f4f280c59b0 2023-11-24T08:12:41.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.410+0000 7f4f257fa640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 0x7f4f0c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.411+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4f1c0aa260 con 0x7f4f280c59b0 2023-11-24T08:12:41.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.411+0000 7f4f27fff640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 0x7f4f0c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.411+0000 7f4f27fff640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 0x7f4f0c06e1a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4f280c5960 tx=0x7f4f14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.415+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f1c076ae0 con 0x7f4f280c59b0 2023-11-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.606+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4f2806dc80 con 0x7f4f280c59b0 2023-11-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.607+0000 7f4f257fa640 1 -- 172.21.15.107:0/1575721059 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4f1c07a100 con 0x7f4f280c59b0 2023-11-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.609+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 msgr2=0x7f4f0c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 0x7f4f0c06e1a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4f280c5960 tx=0x7f4f14009040 comp rx=0 tx=0).stop 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 msgr2=0x7f4f28156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009890 tx=0x7f4f1c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 shutdown_connections 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0c06bce0 0x7f4f0c06e1a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f2815b440 0x7f4f2815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f280cbf20 0x7f4f280b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 --2- 172.21.15.107:0/1575721059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f280c59b0 0x7f4f28156ff0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 >> 172.21.15.107:0/1575721059 conn(0x7f4f280b0c50 msgr2=0x7f4f280710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:41.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 shutdown_connections 2023-11-24T08:12:41.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.610+0000 7f4f2ee3d640 1 -- 172.21.15.107:0/1575721059 wait complete. 2023-11-24T08:12:41.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:41.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.882+0000 7fdbd22da640 1 Processor -- start 2023-11-24T08:12:41.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.882+0000 7fdbd22da640 1 -- start start 2023-11-24T08:12:41.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.883+0000 7fdbd22da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbcc15df80 0x7fdbcc0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.883+0000 7fdbd22da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbcc15e3f0 con 0x7fdbcc0cd470 2023-11-24T08:12:41.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.883+0000 7fdbd22da640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbcc15ab80 con 0x7fdbcc0b2250 2023-11-24T08:12:41.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.883+0000 7fdbd22da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbcc15a990 con 0x7fdbcc15df80 2023-11-24T08:12:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbcb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbcc15df80 0x7fdbcc0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbd0877640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdbcc0cd470 0x7fdbcc15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40490/0 (socket says 172.21.15.107:40490) 2023-11-24T08:12:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbd0877640 1 -- 172.21.15.107:0/1138662112 learned_addr learned my addr 172.21.15.107:0/1138662112 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbcb7fe640 1 -- 172.21.15.107:0/1138662112 >> v1:172.21.15.120:6789/0 conn(0x7fdbcc0b2250 legacy=0x7fdbcc0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbcb7fe640 1 -- 172.21.15.107:0/1138662112 >> v1:172.21.15.107:6789/0 conn(0x7fdbcc0cd470 legacy=0x7fdbcc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.884+0000 7fdbcb7fe640 1 -- 172.21.15.107:0/1138662112 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc0b2d40 con 0x7fdbcc15df80 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.885+0000 7fdbcb7fe640 1 --2- 172.21.15.107:0/1138662112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0ccf30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0097a0 tx=0x7fdbbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b549c09e24e54ed1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.885+0000 7fdbcaffd640 1 -- 172.21.15.107:0/1138662112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc031830 con 0x7fdbcc15df80 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.885+0000 7fdbcaffd640 1 -- 172.21.15.107:0/1138662112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc031990 con 0x7fdbcc15df80 2023-11-24T08:12:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.885+0000 7fdbcaffd640 1 -- 172.21.15.107:0/1138662112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc031d20 con 0x7fdbcc15df80 2023-11-24T08:12:41.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 -- 172.21.15.107:0/1138662112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 msgr2=0x7fdbcc0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 --2- 172.21.15.107:0/1138662112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0ccf30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0097a0 tx=0x7fdbbc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 -- 172.21.15.107:0/1138662112 shutdown_connections 2023-11-24T08:12:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 --2- 172.21.15.107:0/1138662112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0ccf30 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 -- 172.21.15.107:0/1138662112 >> 172.21.15.107:0/1138662112 conn(0x7fdbcc06a920 msgr2=0x7fdbcc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 -- 172.21.15.107:0/1138662112 shutdown_connections 2023-11-24T08:12:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.886+0000 7fdbd22da640 1 -- 172.21.15.107:0/1138662112 wait complete. 2023-11-24T08:12:41.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.888+0000 7fdbd22da640 1 Processor -- start 2023-11-24T08:12:41.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.888+0000 7fdbd22da640 1 -- start start 2023-11-24T08:12:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbcc0b2250 0x7fdbcc0af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbcc0cd470 0x7fdbcc098ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0b0a10 con 0x7fdbcc15df80 2023-11-24T08:12:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0b0cf0 con 0x7fdbcc0cd470 2023-11-24T08:12:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd22da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0b0fd0 con 0x7fdbcc0b2250 2023-11-24T08:12:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd0877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd0877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37010/0 (socket says 172.21.15.107:37010) 2023-11-24T08:12:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.889+0000 7fdbd0877640 1 -- 172.21.15.107:0/2627185931 learned_addr learned my addr 172.21.15.107:0/2627185931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 -- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbcc0b2250 msgr2=0x7fdbcc0af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbcc0b2250 0x7fdbcc0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 -- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbcc0cd470 msgr2=0x7fdbcc098ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbcc0cd470 0x7fdbcc098ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 -- 172.21.15.107:0/2627185931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc009400 con 0x7fdbcc15df80 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd0877640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400ca70 tx=0x7fdbc400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400abd0 con 0x7fdbcc15df80 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc400ad30 con 0x7fdbcc15df80 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4012600 con 0x7fdbcc15df80 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.890+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc0b12b0 con 0x7fdbcc15df80 2023-11-24T08:12:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.891+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc0b1ab0 con 0x7fdbcc15df80 2023-11-24T08:12:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.892+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc0b17d0 con 0x7fdbcc15df80 2023-11-24T08:12:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.893+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fdbc4007070 con 0x7fdbcc15df80 2023-11-24T08:12:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.894+0000 7fdbc8ff9640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 0x7fdbac06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.894+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdbc40881a0 con 0x7fdbcc15df80 2023-11-24T08:12:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.894+0000 7fdbcbfff640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 0x7fdbac06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.895+0000 7fdbcbfff640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 0x7fdbac06e1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4003cc0 tx=0x7fdbb40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:41.898+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbcc0b17d0 con 0x7fdbcc15df80 2023-11-24T08:12:42.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.125+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbcc0b17d0 con 0x7fdbcc15df80 2023-11-24T08:12:42.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.126+0000 7fdbc8ff9640 1 -- 172.21.15.107:0/2627185931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fdbcc0b17d0 con 0x7fdbcc15df80 2023-11-24T08:12:42.128 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:42.128 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 52 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 msgr2=0x7fdbac06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 0x7fdbac06e1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4003cc0 tx=0x7fdbb40079b0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 msgr2=0x7fdbcc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400ca70 tx=0x7fdbc400cf30 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 shutdown_connections 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdbac06bce0 0x7fdbac06e1a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc15df80 0x7fdbcc0b0000 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbcc0cd470 0x7fdbcc098ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 --2- 172.21.15.107:0/2627185931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbcc0b2250 0x7fdbcc0af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 >> 172.21.15.107:0/2627185931 conn(0x7fdbcc06a920 msgr2=0x7fdbcc15d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 shutdown_connections 2023-11-24T08:12:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.130+0000 7fdbd22da640 1 -- 172.21.15.107:0/2627185931 wait complete. 2023-11-24T08:12:42.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.360+0000 7fb99b3be640 1 Processor -- start 2023-11-24T08:12:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.360+0000 7fb99b3be640 1 -- start start 2023-11-24T08:12:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99b3be640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb9940ce6c0 0x7fb9940cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99b3be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9940cece0 con 0x7fb99415c7f0 2023-11-24T08:12:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99b3be640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99406d040 con 0x7fb9940cf450 2023-11-24T08:12:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99b3be640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9940cfe40 con 0x7fb9940ce6c0 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99915a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb99415c7f0 0x7fb99406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40502/0 (socket says 172.21.15.107:40502) 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99915a640 1 -- 172.21.15.107:0/1744825440 learned_addr learned my addr 172.21.15.107:0/1744825440 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.362+0000 7fb99995b640 1 --2- 172.21.15.107:0/1744825440 >> v2:172.21.15.120:3300/0 conn(0x7fb9940ce6c0 0x7fb9940cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286618629 0 0) 0x7fb9940cece0 con 0x7fb99415c7f0 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb980003520 con 0x7fb99415c7f0 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020124502 0 0) 0x7fb980003520 con 0x7fb99415c7f0 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 >> v2:172.21.15.120:3300/0 conn(0x7fb9940ce6c0 msgr2=0x7fb9940cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 --2- 172.21.15.107:0/1744825440 >> v2:172.21.15.120:3300/0 conn(0x7fb9940ce6c0 0x7fb9940cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 >> v1:172.21.15.120:6789/0 conn(0x7fb9940cf450 legacy=0x7fb9940cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99406d320 con 0x7fb99415c7f0 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb9900019f0 con 0x7fb99415c7f0 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 >> v1:172.21.15.107:6789/0 conn(0x7fb99415c7f0 legacy=0x7fb99406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb980003970 0x7fb980003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 0x7fb980004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb980005180 0x7fb98000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9940cfe40 con 0x7fb9800047c0 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98000c020 con 0x7fb980003970 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98000c1e0 con 0x7fb980005180 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.363+0000 7fb998959640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 0x7fb980004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb99915a640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb980003970 0x7fb980003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb998959640 1 -- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb980005180 msgr2=0x7fb98000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb998959640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb980005180 0x7fb98000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb998959640 1 -- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb980003970 msgr2=0x7fb980003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb998959640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb980003970 0x7fb980003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb998959640 1 -- 172.21.15.107:0/1744825440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb980003520 con 0x7fb9800047c0 2023-11-24T08:12:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.364+0000 7fb99995b640 1 -- 172.21.15.107:0/1744825440 reap_dead start 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.365+0000 7fb998959640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 0x7fb980004c40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb984001750 tx=0x7fb9800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4bdabdfcdd28bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.365+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840052d0 con 0x7fb9800047c0 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.365+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98402e050 con 0x7fb9800047c0 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.365+0000 7fb98bfff640 1 -- 172.21.15.107:0/1744825440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984003830 con 0x7fb9800047c0 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 -- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 msgr2=0x7fb980004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 0x7fb980004c40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb984001750 tx=0x7fb9800057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 -- 172.21.15.107:0/1744825440 shutdown_connections 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 --2- 172.21.15.107:0/1744825440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9800047c0 0x7fb980004c40 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 -- 172.21.15.107:0/1744825440 >> 172.21.15.107:0/1744825440 conn(0x7fb99406a920 msgr2=0x7fb99406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 -- 172.21.15.107:0/1744825440 shutdown_connections 2023-11-24T08:12:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.366+0000 7fb99b3be640 1 -- 172.21.15.107:0/1744825440 wait complete. 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.368+0000 7fb99b3be640 1 Processor -- start 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.368+0000 7fb99b3be640 1 -- start start 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9800047c0 0x7fb99415abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb994157a30 0x7fb9940b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb994152dd0 con 0x7fb994151bd0 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb994152590 con 0x7fb9800047c0 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99b3be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb994158210 con 0x7fb994157a30 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37030/0 (socket says 172.21.15.107:37030) 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 -- 172.21.15.107:0/401184533 learned_addr learned my addr 172.21.15.107:0/401184533 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 -- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb994157a30 msgr2=0x7fb9940b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb994157a30 0x7fb9940b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 -- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9800047c0 msgr2=0x7fb99415abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99915a640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9800047c0 0x7fb99415abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9800047c0 0x7fb99415abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.369+0000 7fb99995b640 1 -- 172.21.15.107:0/401184533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9941584f0 con 0x7fb994151bd0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb99995b640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb9940ccd80 tx=0x7fb98c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c013680 con 0x7fb994151bd0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c0137e0 con 0x7fb994151bd0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9941587d0 con 0x7fb994151bd0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c013b90 con 0x7fb994151bd0 2023-11-24T08:12:42.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.370+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb994157eb0 con 0x7fb994151bd0 2023-11-24T08:12:42.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.371+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb994152f40 con 0x7fb994151bd0 2023-11-24T08:12:42.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.372+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fb98c013cf0 con 0x7fb994151bd0 2023-11-24T08:12:42.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.373+0000 7fb989ffb640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 0x7fb97806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.373+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb98c087d80 con 0x7fb994151bd0 2023-11-24T08:12:42.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.373+0000 7fb99915a640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 0x7fb97806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.374+0000 7fb99915a640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 0x7fb97806e1a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb990001580 tx=0x7fb990001250 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.377+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb98c054880 con 0x7fb994151bd0 2023-11-24T08:12:42.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.567+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb994067470 con 0x7fb994151bd0 2023-11-24T08:12:42.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.570+0000 7fb989ffb640 1 -- 172.21.15.107:0/401184533 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16724 (secure 0 0 0) 0x7fb98c057ea0 con 0x7fb994151bd0 2023-11-24T08:12:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:42.572 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:38.692986+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.23","mappings":[{"from":1,"to":5}]},{"pgid":"13.30","mappings":[{"from":6,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:42.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 msgr2=0x7fb97806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 0x7fb97806e1a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb990001580 tx=0x7fb990001250 comp rx=0 tx=0).stop 2023-11-24T08:12:42.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 msgr2=0x7fb994152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb9940ccd80 tx=0x7fb98c00e960 comp rx=0 tx=0).stop 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 shutdown_connections 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb97806bce0 0x7fb97806e1a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb994151bd0 0x7fb994152050 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb994157a30 0x7fb9940b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 --2- 172.21.15.107:0/401184533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9800047c0 0x7fb99415abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.572+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 >> 172.21.15.107:0/401184533 conn(0x7fb99406a920 msgr2=0x7fb9940cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.573+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 shutdown_connections 2023-11-24T08:12:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.573+0000 7fb99b3be640 1 -- 172.21.15.107:0/401184533 wait complete. 2023-11-24T08:12:42.593 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:12:42.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:42.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.806+0000 7fdebcfd5640 1 Processor -- start 2023-11-24T08:12:42.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.806+0000 7fdebcfd5640 1 -- start start 2023-11-24T08:12:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.807+0000 7fdebcfd5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdeb80b2250 0x7fdeb815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.807+0000 7fdebcfd5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdeb80b2630 0x7fdeb80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.807+0000 7fdebcfd5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb815ab80 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.807+0000 7fdebcfd5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb815a990 con 0x7fdeb80b2250 2023-11-24T08:12:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.807+0000 7fdebcfd5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb815e680 con 0x7fdeb80b2630 2023-11-24T08:12:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdeb80b2aa0 0x7fdeb815b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40512/0 (socket says 172.21.15.107:40512) 2023-11-24T08:12:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/2826575280 learned_addr learned my addr 172.21.15.107:0/2826575280 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb659c640 1 --2- 172.21.15.107:0/2826575280 >> v2:172.21.15.107:3300/0 conn(0x7fdeb80b2250 0x7fdeb815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeaffff640 1 --2- 172.21.15.107:0/2826575280 >> v2:172.21.15.120:3300/0 conn(0x7fdeb80b2630 0x7fdeb80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/2826575280 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148406728 0 0) 0x7fdeb815ab80 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/2826575280 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea4003520 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb659c640 1 -- 172.21.15.107:0/2826575280 >> v2:172.21.15.120:3300/0 conn(0x7fdeb80b2630 msgr2=0x7fdeb80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb659c640 1 --2- 172.21.15.107:0/2826575280 >> v2:172.21.15.120:3300/0 conn(0x7fdeb80b2630 0x7fdeb80ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb659c640 1 -- 172.21.15.107:0/2826575280 >> v1:172.21.15.107:6789/0 conn(0x7fdeb80b2aa0 legacy=0x7fdeb815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.808+0000 7fdeb659c640 1 -- 172.21.15.107:0/2826575280 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb815e960 con 0x7fdeb80b2250 2023-11-24T08:12:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.809+0000 7fdeb659c640 1 --2- 172.21.15.107:0/2826575280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2250 0x7fdeb815df80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80ae5c0 tx=0x7fdea802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b25770004baca17 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/2826575280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8031830 con 0x7fdeb80b2250 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/2826575280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8031990 con 0x7fdeb80b2250 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdeb5d9b640 1 -- 172.21.15.107:0/2826575280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8031d20 con 0x7fdeb80b2250 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 -- 172.21.15.107:0/2826575280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2250 msgr2=0x7fdeb815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/2826575280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2250 0x7fdeb815df80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80ae5c0 tx=0x7fdea802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 -- 172.21.15.107:0/2826575280 shutdown_connections 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/2826575280 >> v2:172.21.15.120:3300/0 conn(0x7fdeb80b2630 0x7fdeb80ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/2826575280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2250 0x7fdeb815df80 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.810+0000 7fdebcfd5640 1 -- 172.21.15.107:0/2826575280 >> 172.21.15.107:0/2826575280 conn(0x7fdeb806a920 msgr2=0x7fdeb806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.811+0000 7fdebcfd5640 1 -- 172.21.15.107:0/2826575280 shutdown_connections 2023-11-24T08:12:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.811+0000 7fdebcfd5640 1 -- 172.21.15.107:0/2826575280 wait complete. 2023-11-24T08:12:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.812+0000 7fdebcfd5640 1 Processor -- start 2023-11-24T08:12:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.812+0000 7fdebcfd5640 1 -- start start 2023-11-24T08:12:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb80b2250 0x7fdeb8141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb80b2630 0x7fdeb809cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8152920 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8144bd0 con 0x7fdeb80b2250 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.813+0000 7fdebcfd5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb809d150 con 0x7fdeb80b2630 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37054/0 (socket says 172.21.15.107:37054) 2023-11-24T08:12:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3954070847 learned_addr learned my addr 172.21.15.107:0/3954070847 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeaffff640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb80b2630 0x7fdeb809cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb80b2250 0x7fdeb8141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb80b2630 msgr2=0x7fdeb809cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb80b2630 0x7fdeb809cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb80b2250 msgr2=0x7fdeb8141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb80b2250 0x7fdeb8141d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 -- 172.21.15.107:0/3954070847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8009400 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.814+0000 7fdeb6d9d640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ca20 tx=0x7fdeb000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.815+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000abd0 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.815+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb000ad30 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.815+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000aea0 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.815+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb81553d0 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.815+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80b01f0 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.816+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb80b0050 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.817+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fdeb0007070 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.818+0000 7fdeaeffd640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 0x7fde9806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.818+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdeb0088270 con 0x7fdeb80b2aa0 2023-11-24T08:12:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.818+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 0x7fde9806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.819+0000 7fdeb659c640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 0x7fde9806e1a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdea8036000 tx=0x7fdea8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:42.822+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdeb80b0050 con 0x7fdeb80b2aa0 2023-11-24T08:12:43.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.052+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeb80b0050 con 0x7fdeb80b2aa0 2023-11-24T08:12:43.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.053+0000 7fdeaeffd640 1 -- 172.21.15.107:0/3954070847 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fdeb80b0050 con 0x7fdeb80b2aa0 2023-11-24T08:12:43.055 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:43.055 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:43.055 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 52 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 msgr2=0x7fde9806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 0x7fde9806e1a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdea8036000 tx=0x7fdea8008030 comp rx=0 tx=0).stop 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 msgr2=0x7fdeb8154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ca20 tx=0x7fdeb000cee0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 shutdown_connections 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde9806bce0 0x7fde9806e1a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeb80b2aa0 0x7fdeb8154e90 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdeb80b2630 0x7fdeb809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 --2- 172.21.15.107:0/3954070847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdeb80b2250 0x7fdeb8141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 >> 172.21.15.107:0/3954070847 conn(0x7fdeb806a920 msgr2=0x7fdeb80734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 shutdown_connections 2023-11-24T08:12:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.056+0000 7fdebcfd5640 1 -- 172.21.15.107:0/3954070847 wait complete. 2023-11-24T08:12:43.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.282+0000 7faeb37c5640 1 Processor -- start 2023-11-24T08:12:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.283+0000 7faeb37c5640 1 -- start start 2023-11-24T08:12:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb37c5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faeac15c7f0 0x7faeac06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb37c5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0cece0 con 0x7faeac0cf450 2023-11-24T08:12:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb37c5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac06d040 con 0x7faeac0ce6c0 2023-11-24T08:12:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb37c5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeac0ceaf0 con 0x7faeac15c7f0 2023-11-24T08:12:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb0d60640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faeac0cf450 0x7faeac0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40518/0 (socket says 172.21.15.107:40518) 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.284+0000 7faeb0d60640 1 -- 172.21.15.107:0/1073165317 learned_addr learned my addr 172.21.15.107:0/1073165317 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faeb1561640 1 --2- 172.21.15.107:0/1073165317 >> v2:172.21.15.120:3300/0 conn(0x7faeac15c7f0 0x7faeac06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285788278 0 0) 0x7faeac0cece0 con 0x7faeac0cf450 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae98003520 con 0x7faeac0cf450 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746693998 0 0) 0x7fae98003520 con 0x7faeac0cf450 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 >> v2:172.21.15.120:3300/0 conn(0x7faeac15c7f0 msgr2=0x7faeac06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 --2- 172.21.15.107:0/1073165317 >> v2:172.21.15.120:3300/0 conn(0x7faeac15c7f0 0x7faeac06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 >> v1:172.21.15.120:6789/0 conn(0x7faeac0ce6c0 legacy=0x7faeac0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac06d320 con 0x7faeac0cf450 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fae9c001670 con 0x7faeac0cf450 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 >> v1:172.21.15.107:6789/0 conn(0x7faeac0cf450 legacy=0x7faeac0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae98003970 0x7fae98003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 0x7fae98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae98005180 0x7fae9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9800c080 con 0x7fae980047c0 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9800c1f0 con 0x7fae98005180 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.285+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9800c3b0 con 0x7fae98003970 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 0x7fae98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 -- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae98003970 msgr2=0x7fae98003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb1561640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae98003970 0x7fae98003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae98003970 0x7fae98003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 -- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae98005180 msgr2=0x7fae9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb1d62640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae98005180 0x7fae9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae98005180 0x7fae9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb0d60640 1 -- 172.21.15.107:0/1073165317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98003520 con 0x7fae980047c0 2023-11-24T08:12:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb1d62640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae98005180 0x7fae9800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.286+0000 7faeb1d62640 1 -- 172.21.15.107:0/1073165317 reap_dead start 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.287+0000 7faeb0d60640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 0x7fae98004c40 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001eb0 tx=0x7fae980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b01113afa21e20f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.287+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c005ad0 con 0x7fae980047c0 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.287+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c005c50 con 0x7fae980047c0 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.287+0000 7faea3fff640 1 -- 172.21.15.107:0/1073165317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c02c890 con 0x7fae980047c0 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 -- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 msgr2=0x7fae98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 0x7fae98004c40 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001eb0 tx=0x7fae980057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 -- 172.21.15.107:0/1073165317 shutdown_connections 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 --2- 172.21.15.107:0/1073165317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae980047c0 0x7fae98004c40 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 -- 172.21.15.107:0/1073165317 >> 172.21.15.107:0/1073165317 conn(0x7faeac06a920 msgr2=0x7faeac06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 -- 172.21.15.107:0/1073165317 shutdown_connections 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.288+0000 7faeb37c5640 1 -- 172.21.15.107:0/1073165317 wait complete. 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.290+0000 7faeb37c5640 1 Processor -- start 2023-11-24T08:12:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.290+0000 7faeb37c5640 1 -- start start 2023-11-24T08:12:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae980047c0 0x7faeac0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 0x7faeac1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac156f80 con 0x7faeac156160 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac154ef0 con 0x7faeac154550 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb37c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeac157190 con 0x7fae980047c0 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb1d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb1d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37070/0 (socket says 172.21.15.107:37070) 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.291+0000 7faeb1d62640 1 -- 172.21.15.107:0/3980369550 learned_addr learned my addr 172.21.15.107:0/3980369550 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 -- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae980047c0 msgr2=0x7faeac0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb0d60640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 0x7faeac1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1561640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae980047c0 0x7faeac0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae980047c0 0x7faeac0adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 -- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 msgr2=0x7faeac1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 0x7faeac1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 -- 172.21.15.107:0/3980369550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac157470 con 0x7faeac156160 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb0d60640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 0x7faeac1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.292+0000 7faeb1d62640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7faeac0ccd80 tx=0x7faea400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.293+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4017070 con 0x7faeac156160 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.293+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea4007020 con 0x7faeac156160 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.293+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac157750 con 0x7faeac156160 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.293+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea40073b0 con 0x7faeac156160 2023-11-24T08:12:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.293+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac156b20 con 0x7faeac156160 2023-11-24T08:12:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.294+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac1572d0 con 0x7faeac156160 2023-11-24T08:12:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.295+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7faea4007510 con 0x7faeac156160 2023-11-24T08:12:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.296+0000 7faea1ffb640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 0x7fae9006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.296+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7faea4087720 con 0x7faeac156160 2023-11-24T08:12:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.296+0000 7faeb1561640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 0x7fae9006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.297+0000 7faeb1561640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 0x7fae9006e1d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faea8009e70 tx=0x7faea800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.300+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faeac1572d0 con 0x7faeac156160 2023-11-24T08:12:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.526+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeac157c70 con 0x7faeac156160 2023-11-24T08:12:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.527+0000 7faea1ffb640 1 -- 172.21.15.107:0/3980369550 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4924 (secure 0 0 0) 0x7faeac157c70 con 0x7faeac156160 2023-11-24T08:12:43.529 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:43.529 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:04.149897+0000","modified":"2023-11-24T08:12:07.368471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7332":{"gid":7332,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/1534091061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1534091061},{"type":"v1","addr":"172.21.15.107:6837","nonce":1534091061}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-24T08:12:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 52 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 msgr2=0x7fae9006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 0x7fae9006e1d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faea8009e70 tx=0x7faea800b040 comp rx=0 tx=0).stop 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 msgr2=0x7faeac1565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7faeac0ccd80 tx=0x7faea400d890 comp rx=0 tx=0).stop 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 shutdown_connections 2023-11-24T08:12:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae9006bd10 0x7fae9006e1d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faeac156160 0x7faeac1565e0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faeac154550 0x7faeac1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 --2- 172.21.15.107:0/3980369550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae980047c0 0x7faeac0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 >> 172.21.15.107:0/3980369550 conn(0x7faeac06a920 msgr2=0x7faeac06ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 shutdown_connections 2023-11-24T08:12:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.530+0000 7faeb37c5640 1 -- 172.21.15.107:0/3980369550 wait complete. 2023-11-24T08:12:43.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:43.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.767+0000 7f78a05ea640 1 Processor -- start 2023-11-24T08:12:43.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.767+0000 7f78a05ea640 1 -- start start 2023-11-24T08:12:43.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f78a05ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f78980ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f78a05ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789815ab80 con 0x7f78980cd470 2023-11-24T08:12:43.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f78a05ea640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789815e3c0 con 0x7f78980b2250 2023-11-24T08:12:43.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f78a05ea640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789815a990 con 0x7f789815df80 2023-11-24T08:12:43.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f789eb87640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f78980cd470 0x7f789815b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40520/0 (socket says 172.21.15.107:40520) 2023-11-24T08:12:43.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f789eb87640 1 -- 172.21.15.107:0/3689600078 learned_addr learned my addr 172.21.15.107:0/3689600078 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f789db85640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f78980ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104143346 0 0) 0x7f789815ab80 con 0x7f78980cd470 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.769+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7884003520 con 0x7f78980cd470 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085242259 0 0) 0x7f7884003520 con 0x7f78980cd470 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 msgr2=0x7f78980ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f78980ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 >> v1:172.21.15.120:6789/0 conn(0x7f78980b2250 legacy=0x7f78980b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78980b2d40 con 0x7f78980cd470 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7890001670 con 0x7f78980cd470 2023-11-24T08:12:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 >> v1:172.21.15.107:6789/0 conn(0x7f78980cd470 legacy=0x7f789815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840039a0 0x7f7884003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7884004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f7884005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788400dd20 con 0x7f78840047c0 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788400de90 con 0x7f78840039a0 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788400e050 con 0x7f789815df80 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789db85640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7884004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.770+0000 7f789e386640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840039a0 0x7f7884003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789db85640 1 -- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 msgr2=0x7f7884005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789eb87640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f7884005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789db85640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f789815df80 0x7f7884005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789db85640 1 -- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840039a0 msgr2=0x7f7884003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789db85640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840039a0 0x7f7884003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789db85640 1 -- 172.21.15.107:0/3689600078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884003520 con 0x7f78840047c0 2023-11-24T08:12:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.771+0000 7f789e386640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840039a0 0x7f7884003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f789db85640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7884004c40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f788800ba30 tx=0x7f78840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781a12f20f3d1364 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888032640 con 0x7f78840047c0 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78880327a0 con 0x7f78840047c0 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f789d384640 1 -- 172.21.15.107:0/3689600078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888032b30 con 0x7f78840047c0 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f78a05ea640 1 -- 172.21.15.107:0/3689600078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 msgr2=0x7f7884004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f78a05ea640 1 --2- 172.21.15.107:0/3689600078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7884004c40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f788800ba30 tx=0x7f78840057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.772+0000 7f789eb87640 1 -- 172.21.15.107:0/3689600078 reap_dead start 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.773+0000 7f78a05ea640 1 -- 172.21.15.107:0/3689600078 shutdown_connections 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.773+0000 7f78a05ea640 1 -- 172.21.15.107:0/3689600078 >> 172.21.15.107:0/3689600078 conn(0x7f789806a920 msgr2=0x7f789806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.773+0000 7f78a05ea640 1 -- 172.21.15.107:0/3689600078 shutdown_connections 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.773+0000 7f78a05ea640 1 -- 172.21.15.107:0/3689600078 wait complete. 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.774+0000 7f78a05ea640 1 Processor -- start 2023-11-24T08:12:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.775+0000 7f78a05ea640 1 -- start start 2023-11-24T08:12:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.775+0000 7f78a05ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.775+0000 7f78a05ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 0x7f7898155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.775+0000 7f78a05ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7898072580 0x7f7898155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f78a05ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78980c9c60 con 0x7f78840047c0 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f78a05ea640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78980c9f40 con 0x7f7898072100 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f78a05ea640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78980ca220 con 0x7f7898072580 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37084/0 (socket says 172.21.15.107:37084) 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 -- 172.21.15.107:0/1828248319 learned_addr learned my addr 172.21.15.107:0/1828248319 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789db85640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 0x7f7898155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 -- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7898072580 msgr2=0x7f7898155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7898072580 0x7f7898155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 -- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 msgr2=0x7f7898155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 0x7f7898155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 -- 172.21.15.107:0/1828248319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78980ca500 con 0x7f78840047c0 2023-11-24T08:12:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789db85640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 0x7f7898155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.776+0000 7f789e386640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f789400dae0 tx=0x7f7894002070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.777+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894003da0 con 0x7f78840047c0 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.777+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789400b710 con 0x7f78840047c0 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.777+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400bb00 con 0x7f78840047c0 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.777+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78980ca7e0 con 0x7f78840047c0 2023-11-24T08:12:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.777+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78980c9880 con 0x7f78840047c0 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.778+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7898067470 con 0x7f78840047c0 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.779+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f789400bc60 con 0x7f78840047c0 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.780+0000 7f788effd640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 0x7f787c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.780+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7894088640 con 0x7f78840047c0 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.780+0000 7f789db85640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 0x7f787c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.781+0000 7f789db85640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 0x7f787c06e1a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f788800ba00 tx=0x7f788803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:43.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.784+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7894054ec0 con 0x7f78840047c0 2023-11-24T08:12:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.977+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78980cb7a0 con 0x7f78840047c0 2023-11-24T08:12:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.979+0000 7f788effd640 1 -- 172.21.15.107:0/1828248319 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16724 (secure 0 0 0) 0x7f78940584e0 con 0x7f78840047c0 2023-11-24T08:12:43.982 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:38.692986+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:01.979497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:03.154712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.23","mappings":[{"from":1,"to":5}]},{"pgid":"13.30","mappings":[{"from":6,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 msgr2=0x7f787c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 0x7f787c06e1a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f788800ba00 tx=0x7f788803d040 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 msgr2=0x7f7898150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f789400dae0 tx=0x7f7894002070 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 shutdown_connections 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f787c06bce0 0x7f787c06e1a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7898072580 0x7f7898155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7898072100 0x7f7898155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 --2- 172.21.15.107:0/1828248319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78840047c0 0x7f7898150700 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.982+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 >> 172.21.15.107:0/1828248319 conn(0x7f789806a920 msgr2=0x7f78981585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.983+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 shutdown_connections 2023-11-24T08:12:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:43.983+0000 7f78a05ea640 1 -- 172.21.15.107:0/1828248319 wait complete. 2023-11-24T08:12:44.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:12:44.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.210+0000 7fc12fb43640 1 Processor -- start 2023-11-24T08:12:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.211+0000 7fc12fb43640 1 -- start start 2023-11-24T08:12:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12fb43640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 0x7fc12806ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12fb43640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1280cc2b0 0x7fc12806f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12fb43640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1281612f0 con 0x7fc1280cfb90 2023-11-24T08:12:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12fb43640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280cff70 con 0x7fc1280cc2b0 2023-11-24T08:12:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12fb43640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280afb20 con 0x7fc1280b2400 2023-11-24T08:12:44.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12d8df640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1280cfb90 0x7fc128161a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40528/0 (socket says 172.21.15.107:40528) 2023-11-24T08:12:44.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.212+0000 7fc12d8df640 1 -- 172.21.15.107:0/3380929623 learned_addr learned my addr 172.21.15.107:0/3380929623 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:44.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/3380929623 >> v2:172.21.15.107:3300/0 conn(0x7fc1280cc2b0 0x7fc12806f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12d0de640 1 --2- 172.21.15.107:0/3380929623 >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 0x7fc12806ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12c8dd640 1 -- 172.21.15.107:0/3380929623 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152464728 0 0) 0x7fc1281612f0 con 0x7fc1280cfb90 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12c8dd640 1 -- 172.21.15.107:0/3380929623 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110003520 con 0x7fc1280cfb90 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12e0e0640 1 -- 172.21.15.107:0/3380929623 >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 msgr2=0x7fc12806ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/3380929623 >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 0x7fc12806ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12e0e0640 1 -- 172.21.15.107:0/3380929623 >> v1:172.21.15.107:6789/0 conn(0x7fc1280cfb90 legacy=0x7fc128161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12e0e0640 1 -- 172.21.15.107:0/3380929623 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128162170 con 0x7fc1280cc2b0 2023-11-24T08:12:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.213+0000 7fc12d0de640 1 --2- 172.21.15.107:0/3380929623 >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 0x7fc12806ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:12:44.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.214+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/3380929623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cc2b0 0x7fc12806f390 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc1280cdb80 tx=0x7fc12002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94d9989a6037fb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12c8dd640 1 -- 172.21.15.107:0/3380929623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12002cd00 con 0x7fc1280cc2b0 2023-11-24T08:12:44.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12c8dd640 1 -- 172.21.15.107:0/3380929623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12002ce60 con 0x7fc1280cc2b0 2023-11-24T08:12:44.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12c8dd640 1 -- 172.21.15.107:0/3380929623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120034660 con 0x7fc1280cc2b0 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 -- 172.21.15.107:0/3380929623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cc2b0 msgr2=0x7fc12806f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 --2- 172.21.15.107:0/3380929623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cc2b0 0x7fc12806f390 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc1280cdb80 tx=0x7fc12002bf50 comp rx=0 tx=0).stop 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 -- 172.21.15.107:0/3380929623 shutdown_connections 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 --2- 172.21.15.107:0/3380929623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cc2b0 0x7fc12806f390 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 --2- 172.21.15.107:0/3380929623 >> v2:172.21.15.120:3300/0 conn(0x7fc1280b2400 0x7fc12806ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 -- 172.21.15.107:0/3380929623 >> 172.21.15.107:0/3380929623 conn(0x7fc1280999d0 msgr2=0x7fc1280b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 -- 172.21.15.107:0/3380929623 shutdown_connections 2023-11-24T08:12:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.215+0000 7fc12fb43640 1 -- 172.21.15.107:0/3380929623 wait complete. 2023-11-24T08:12:44.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.217+0000 7fc12fb43640 1 Processor -- start 2023-11-24T08:12:44.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.217+0000 7fc12fb43640 1 -- start start 2023-11-24T08:12:44.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1280b2400 0x7fc12815aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cc2b0 0x7fc1280b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1281f7260 con 0x7fc1280cfb90 2023-11-24T08:12:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815b530 con 0x7fc1280cc2b0 2023-11-24T08:12:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.218+0000 7fc12fb43640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1281f65e0 con 0x7fc1280b2400 2023-11-24T08:12:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37106/0 (socket says 172.21.15.107:37106) 2023-11-24T08:12:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 -- 172.21.15.107:0/2326209990 learned_addr learned my addr 172.21.15.107:0/2326209990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:44.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12d8df640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1280b2400 0x7fc12815aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 -- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1280b2400 msgr2=0x7fc12815aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1280b2400 0x7fc12815aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12d0de640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cc2b0 0x7fc1280b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 -- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cc2b0 msgr2=0x7fc1280b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cc2b0 0x7fc1280b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.219+0000 7fc12e0e0640 1 -- 172.21.15.107:0/2326209990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009a70 con 0x7fc1280cfb90 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc12e0e0640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc120009d90 tx=0x7fc120004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12003d600 con 0x7fc1280cfb90 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12003d780 con 0x7fc1280cfb90 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1281f68c0 con 0x7fc1280cfb90 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12003db30 con 0x7fc1280cfb90 2023-11-24T08:12:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.220+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281f6e80 con 0x7fc1280cfb90 2023-11-24T08:12:44.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.221+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12806ab10 con 0x7fc1280cfb90 2023-11-24T08:12:44.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.222+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fc12003b040 con 0x7fc1280cfb90 2023-11-24T08:12:44.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.223+0000 7fc1267fc640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 0x7fc11406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.223+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc1200aa8e0 con 0x7fc1280cfb90 2023-11-24T08:12:44.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.223+0000 7fc12d8df640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 0x7fc11406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.224+0000 7fc12d8df640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 0x7fc11406e1a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1180011e0 tx=0x7fc118009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.227+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc12806ab10 con 0x7fc1280cfb90 2023-11-24T08:12:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.454+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc12806ab10 con 0x7fc1280cfb90 2023-11-24T08:12:44.457 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:12:44.456+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:12:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.459+0000 7fc1267fc640 1 -- 172.21.15.107:0/2326209990 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc12806ab10 con 0x7fc1280cfb90 2023-11-24T08:12:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 msgr2=0x7fc11406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 0x7fc11406e1a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1180011e0 tx=0x7fc118009040 comp rx=0 tx=0).stop 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 msgr2=0x7fc1280ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc120009d90 tx=0x7fc120004a40 comp rx=0 tx=0).stop 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 shutdown_connections 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc11406bce0 0x7fc11406e1a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1280cfb90 0x7fc1280ae4d0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cc2b0 0x7fc1280b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 --2- 172.21.15.107:0/2326209990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1280b2400 0x7fc12815aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 >> 172.21.15.107:0/2326209990 conn(0x7fc1280999d0 msgr2=0x7fc1280b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 shutdown_connections 2023-11-24T08:12:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.463+0000 7fc12fb43640 1 -- 172.21.15.107:0/2326209990 wait complete. 2023-11-24T08:12:44.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:12:44.532 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:12:44.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.675+0000 7f6bbd45d640 1 Processor -- start 2023-11-24T08:12:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.675+0000 7f6bbd45d640 1 -- start start 2023-11-24T08:12:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.677+0000 7f6bbd45d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6bb80cc0f0 0x7f6bb80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.677+0000 7f6bbd45d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb815eea0 con 0x7f6bb80c57e0 2023-11-24T08:12:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.677+0000 7f6bbd45d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb80abf30 con 0x7f6bb815b440 2023-11-24T08:12:44.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.677+0000 7f6bbd45d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb815b820 con 0x7f6bb80cc0f0 2023-11-24T08:12:44.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.678+0000 7f6bb77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6bb80c57e0 0x7f6bb806c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40540/0 (socket says 172.21.15.107:40540) 2023-11-24T08:12:44.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.678+0000 7f6bb77fe640 1 -- 172.21.15.107:0/1506118364 learned_addr learned my addr 172.21.15.107:0/1506118364 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.678+0000 7f6baffff640 1 --2- 172.21.15.107:0/1506118364 >> v2:172.21.15.120:3300/0 conn(0x7f6bb80cc0f0 0x7f6bb80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.678+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171234832 0 0) 0x7f6bb815eea0 con 0x7f6bb80c57e0 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.678+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba4003520 con 0x7f6bb80c57e0 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095932945 0 0) 0x7f6ba4003520 con 0x7f6bb80c57e0 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 >> v2:172.21.15.120:3300/0 conn(0x7f6bb80cc0f0 msgr2=0x7f6bb80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 --2- 172.21.15.107:0/1506118364 >> v2:172.21.15.120:3300/0 conn(0x7f6bb80cc0f0 0x7f6bb80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 >> v1:172.21.15.120:6789/0 conn(0x7f6bb815b440 legacy=0x7f6bb80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb815f0f0 con 0x7f6bb80c57e0 2023-11-24T08:12:44.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6bb00019f0 con 0x7f6bb80c57e0 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 >> v1:172.21.15.107:6789/0 conn(0x7f6bb80c57e0 legacy=0x7f6bb806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ba4003970 0x7f6ba4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba40047c0 0x7f6ba4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 0x7f6ba400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb815b820 con 0x7f6ba4005180 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba400c120 con 0x7f6ba40047c0 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba400c2e0 con 0x7f6ba4003970 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 0x7f6ba400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.679+0000 7f6baffff640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba40047c0 0x7f6ba4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb6ffd640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ba4003970 0x7f6ba4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb77fe640 1 -- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ba4003970 msgr2=0x7f6ba4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ba4003970 0x7f6ba4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb77fe640 1 -- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba40047c0 msgr2=0x7f6ba4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:44.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba40047c0 0x7f6ba4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.680+0000 7f6bb77fe640 1 -- 172.21.15.107:0/1506118364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba4003520 con 0x7f6ba4005180 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.681+0000 7f6bb77fe640 1 -- 172.21.15.107:0/1506118364 reap_dead start 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.681+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 0x7f6ba400b980 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000a830 tx=0x7f6ba40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fb6d39a392a961 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.681+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00072d0 con 0x7f6ba4005180 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.681+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb002f070 con 0x7f6ba4005180 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.681+0000 7f6bb67fc640 1 -- 172.21.15.107:0/1506118364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0007690 con 0x7f6ba4005180 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 -- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 msgr2=0x7f6ba400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 0x7f6ba400b980 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000a830 tx=0x7f6ba40057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 -- 172.21.15.107:0/1506118364 shutdown_connections 2023-11-24T08:12:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/1506118364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ba4005180 0x7f6ba400b980 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 -- 172.21.15.107:0/1506118364 >> 172.21.15.107:0/1506118364 conn(0x7f6bb80b0c50 msgr2=0x7f6bb80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:44.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 -- 172.21.15.107:0/1506118364 shutdown_connections 2023-11-24T08:12:44.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.682+0000 7f6bbd45d640 1 -- 172.21.15.107:0/1506118364 wait complete. 2023-11-24T08:12:44.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.684+0000 7f6bbd45d640 1 Processor -- start 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.684+0000 7f6bbd45d640 1 -- start start 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba4005180 0x7f6bb80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 0x7f6bb815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80c7fd0 con 0x7f6bb8156630 2023-11-24T08:12:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80c7350 con 0x7f6ba4005180 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bbd45d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80c7630 con 0x7f6bb80ce8b0 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37140/0 (socket says 172.21.15.107:37140) 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 -- 172.21.15.107:0/2968719610 learned_addr learned my addr 172.21.15.107:0/2968719610 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6baffff640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 0x7f6bb815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb6ffd640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba4005180 0x7f6bb80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 -- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 msgr2=0x7f6bb815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 0x7f6bb815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 -- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba4005180 msgr2=0x7f6bb80c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.685+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba4005180 0x7f6bb80c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6bb77fe640 1 -- 172.21.15.107:0/2968719610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb80c7910 con 0x7f6bb8156630 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6baffff640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 0x7f6bb815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6bb77fe640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0000f80 tx=0x7f6bb0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0004d60 con 0x7f6bb8156630 2023-11-24T08:12:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb00070d0 con 0x7f6bb8156630 2023-11-24T08:12:44.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb80c7bf0 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0007440 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.686+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb815bea0 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.687+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb804e8b0 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.688+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f6bb0031030 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.689+0000 7f6baf7fe640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 0x7f6b9806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.689+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6bb00a9fd0 con 0x7f6bb8156630 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.689+0000 7f6bb6ffd640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 0x7f6b9806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.690+0000 7f6bb6ffd640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 0x7f6b9806e1a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8007fd0 tx=0x7f6ba8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.693+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6bb0076850 con 0x7f6bb8156630 2023-11-24T08:12:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:44.926+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6bb804e8b0 con 0x7f6bb8156630 2023-11-24T08:12:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.463+0000 7f6baf7fe640 1 -- 172.21.15.107:0/2968719610 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6bb0079e70 con 0x7f6bb8156630 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.466+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 msgr2=0x7f6b9806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.466+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 0x7f6b9806e1a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8007fd0 tx=0x7f6ba8009040 comp rx=0 tx=0).stop 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.466+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 msgr2=0x7f6bb809b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0000f80 tx=0x7f6bb0001d80 comp rx=0 tx=0).stop 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 shutdown_connections 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6b9806bce0 0x7f6b9806e1a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb8156630 0x7f6bb809b840 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bb80ce8b0 0x7f6bb815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 --2- 172.21.15.107:0/2968719610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ba4005180 0x7f6bb80c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 >> 172.21.15.107:0/2968719610 conn(0x7f6bb80b0c50 msgr2=0x7f6bb8071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 shutdown_connections 2023-11-24T08:12:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.467+0000 7f6bbd45d640 1 -- 172.21.15.107:0/2968719610 wait complete. 2023-11-24T08:12:45.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:12:45.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.751+0000 7f5c79c1e640 1 Processor -- start 2023-11-24T08:12:45.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.752+0000 7f5c79c1e640 1 -- start start 2023-11-24T08:12:45.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c79c1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c740ce640 0x7f5c740abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:45.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c79c1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c740ce270 con 0x7f5c740cc470 2023-11-24T08:12:45.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c79c1e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7406d0a0 con 0x7f5c740cfb60 2023-11-24T08:12:45.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c79c1e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7406c560 con 0x7f5c740ce640 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c73fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c740ce640 0x7f5c740abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c737fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c740cc470 0x7f5c740cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40548/0 (socket says 172.21.15.107:40548) 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c73fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c740ce640 0x7f5c740abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37150/0 (socket says 172.21.15.107:37150) 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.753+0000 7f5c737fe640 1 -- 172.21.15.107:0/4111002774 learned_addr learned my addr 172.21.15.107:0/4111002774 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.754+0000 7f5c73fff640 1 -- 172.21.15.107:0/4111002774 >> v1:172.21.15.120:6789/0 conn(0x7f5c740cfb60 legacy=0x7f5c7406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.754+0000 7f5c73fff640 1 -- 172.21.15.107:0/4111002774 >> v1:172.21.15.107:6789/0 conn(0x7f5c740cc470 legacy=0x7f5c740cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.754+0000 7f5c73fff640 1 -- 172.21.15.107:0/4111002774 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7406c770 con 0x7f5c740ce640 2023-11-24T08:12:45.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.755+0000 7f5c73fff640 1 --2- 172.21.15.107:0/4111002774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740ce640 0x7f5c740abfc0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0097a0 tx=0x7f5c6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23497f80cf7e4423 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:45.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.755+0000 7f5c727fc640 1 -- 172.21.15.107:0/4111002774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c031830 con 0x7f5c740ce640 2023-11-24T08:12:45.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.755+0000 7f5c727fc640 1 -- 172.21.15.107:0/4111002774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c031990 con 0x7f5c740ce640 2023-11-24T08:12:45.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.755+0000 7f5c727fc640 1 -- 172.21.15.107:0/4111002774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c031d20 con 0x7f5c740ce640 2023-11-24T08:12:45.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 -- 172.21.15.107:0/4111002774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740ce640 msgr2=0x7f5c740abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/4111002774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740ce640 0x7f5c740abfc0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0097a0 tx=0x7f5c6c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 -- 172.21.15.107:0/4111002774 shutdown_connections 2023-11-24T08:12:45.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/4111002774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740ce640 0x7f5c740abfc0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 -- 172.21.15.107:0/4111002774 >> 172.21.15.107:0/4111002774 conn(0x7f5c7406a920 msgr2=0x7f5c7406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:45.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 -- 172.21.15.107:0/4111002774 shutdown_connections 2023-11-24T08:12:45.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.756+0000 7f5c79c1e640 1 -- 172.21.15.107:0/4111002774 wait complete. 2023-11-24T08:12:45.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.758+0000 7f5c79c1e640 1 Processor -- start 2023-11-24T08:12:45.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.758+0000 7f5c79c1e640 1 -- start start 2023-11-24T08:12:45.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:45.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c740ce640 0x7f5c74157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c740cfb60 0x7f5c741630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74157d20 con 0x7f5c740cc470 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7415ba60 con 0x7f5c740ce640 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c79c1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7415bd40 con 0x7f5c740cfb60 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37164/0 (socket says 172.21.15.107:37164) 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c737fe640 1 -- 172.21.15.107:0/652208207 learned_addr learned my addr 172.21.15.107:0/652208207 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c737fe640 1 -- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c740cfb60 msgr2=0x7f5c741630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.759+0000 7f5c737fe640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c740cfb60 0x7f5c741630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c737fe640 1 -- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c740ce640 msgr2=0x7f5c74157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c737fe640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c740ce640 0x7f5c74157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c737fe640 1 -- 172.21.15.107:0/652208207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c009400 con 0x7f5c740cc470 2023-11-24T08:12:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c737fe640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400cba0 tx=0x7f5c6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400ad40 con 0x7f5c740cc470 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64012400 con 0x7f5c740cc470 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7415c020 con 0x7f5c740cc470 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400aea0 con 0x7f5c740cc470 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.760+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c74158460 con 0x7f5c740cc470 2023-11-24T08:12:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.761+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7415c540 con 0x7f5c740cc470 2023-11-24T08:12:45.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.762+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f5c64007070 con 0x7f5c740cc470 2023-11-24T08:12:45.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.763+0000 7f5c6bfff640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 0x7f5c5406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:45.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.764+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f5c64088250 con 0x7f5c740cc470 2023-11-24T08:12:45.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.764+0000 7f5c72ffd640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 0x7f5c5406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:45.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.764+0000 7f5c72ffd640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 0x7f5c5406e1a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5c741fac30 tx=0x7f5c5c006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:45.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.768+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c7415c540 con 0x7f5c740cc470 2023-11-24T08:12:45.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:45.963+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5c7415c540 con 0x7f5c740cc470 2023-11-24T08:12:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.903+0000 7f5c6bfff640 1 -- 172.21.15.107:0/652208207 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5c7415c540 con 0x7f5c740cc470 2023-11-24T08:12:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:12:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.907+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 msgr2=0x7f5c5406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.907+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 0x7f5c5406e1a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5c741fac30 tx=0x7f5c5c006ba0 comp rx=0 tx=0).stop 2023-11-24T08:12:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.907+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 msgr2=0x7f5c741577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.907+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400cba0 tx=0x7f5c6400a430 comp rx=0 tx=0).stop 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 shutdown_connections 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5406bce0 0x7f5c5406e1a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c740cfb60 0x7f5c741630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c740ce640 0x7f5c74157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 --2- 172.21.15.107:0/652208207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c740cc470 0x7f5c741577e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 >> 172.21.15.107:0/652208207 conn(0x7f5c7406a920 msgr2=0x7f5c740ac5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 shutdown_connections 2023-11-24T08:12:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:46.909+0000 7f5c79c1e640 1 -- 172.21.15.107:0/652208207 wait complete. 2023-11-24T08:12:46.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:12:47.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.135+0000 7f141b733640 1 Processor -- start 2023-11-24T08:12:47.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.136+0000 7f141b733640 1 -- start start 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f141b733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f141406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f141b733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14140cf450 0x7f14140cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f141b733640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14140cece0 con 0x7f14140cdfb0 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f141b733640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1414156010 con 0x7f14140cf450 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f141b733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141406ce30 con 0x7f141415c7f0 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f1419cd0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f14140cdfb0 0x7f14140ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40562/0 (socket says 172.21.15.107:40562) 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f1419cd0640 1 -- 172.21.15.107:0/2410552950 learned_addr learned my addr 172.21.15.107:0/2410552950 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.137+0000 7f1418cce640 1 --2- 172.21.15.107:0/2410552950 >> v2:172.21.15.107:3300/0 conn(0x7f14140cf450 0x7f14140cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f14194cf640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f141406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1407fff640 1 -- 172.21.15.107:0/2410552950 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518868780 0 0) 0x7f14140cece0 con 0x7f14140cdfb0 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1407fff640 1 -- 172.21.15.107:0/2410552950 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1400003520 con 0x7f14140cdfb0 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1418cce640 1 -- 172.21.15.107:0/2410552950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 msgr2=0x7f141406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1418cce640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f141406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1418cce640 1 -- 172.21.15.107:0/2410552950 >> v1:172.21.15.107:6789/0 conn(0x7f14140cdfb0 legacy=0x7f14140ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.138+0000 7f1418cce640 1 -- 172.21.15.107:0/2410552950 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141406d040 con 0x7f14140cf450 2023-11-24T08:12:47.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.139+0000 7f1418cce640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14140cf8d0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f14080097a0 tx=0x7f140802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f66ea7a1de6537 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:47.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.139+0000 7f1407fff640 1 -- 172.21.15.107:0/2410552950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140802d040 con 0x7f14140cf450 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.139+0000 7f1407fff640 1 -- 172.21.15.107:0/2410552950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1408031810 con 0x7f14140cf450 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.139+0000 7f1407fff640 1 -- 172.21.15.107:0/2410552950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408031ba0 con 0x7f14140cf450 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 -- 172.21.15.107:0/2410552950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 msgr2=0x7f14140cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14140cf8d0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f14080097a0 tx=0x7f140802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 -- 172.21.15.107:0/2410552950 shutdown_connections 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14140cf8d0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 --2- 172.21.15.107:0/2410552950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f141406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 -- 172.21.15.107:0/2410552950 >> 172.21.15.107:0/2410552950 conn(0x7f141406a920 msgr2=0x7f141406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 -- 172.21.15.107:0/2410552950 shutdown_connections 2023-11-24T08:12:47.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.140+0000 7f141b733640 1 -- 172.21.15.107:0/2410552950 wait complete. 2023-11-24T08:12:47.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.142+0000 7f141b733640 1 Processor -- start 2023-11-24T08:12:47.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.142+0000 7f141b733640 1 -- start start 2023-11-24T08:12:47.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14140cdfb0 0x7f1414153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f1414158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1414153a50 con 0x7f14140cf450 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14141f3690 con 0x7f14140cdfb0 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f141b733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1414158e80 con 0x7f141415c7f0 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.143+0000 7f1418cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37184/0 (socket says 172.21.15.107:37184) 2023-11-24T08:12:47.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 -- 172.21.15.107:0/255245588 learned_addr learned my addr 172.21.15.107:0/255245588 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:47.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f14194cf640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14140cdfb0 0x7f1414153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:47.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 -- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 msgr2=0x7f1414158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f1414158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 -- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14140cdfb0 msgr2=0x7f1414153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14140cdfb0 0x7f1414153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 -- 172.21.15.107:0/255245588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408009400 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.144+0000 7f1418cce640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f140802bec0 tx=0x7f14080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.145+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140803ad60 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.145+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14080423f0 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.145+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1414159160 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.145+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080427a0 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.145+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14140b2180 con 0x7f14140cf450 2023-11-24T08:12:47.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.146+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141415ff20 con 0x7f14140cf450 2023-11-24T08:12:47.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.147+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f1408042900 con 0x7f14140cf450 2023-11-24T08:12:47.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.148+0000 7f1405ffb640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 0x7f13f806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:47.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.148+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f14080aa3e0 con 0x7f14140cf450 2023-11-24T08:12:47.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.148+0000 7f14194cf640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 0x7f13f806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:47.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.149+0000 7f14194cf640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 0x7f13f806e1a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f1410008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:47.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.152+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1408076c00 con 0x7f14140cf450 2023-11-24T08:12:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.346+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1414067470 con 0x7f14140cf450 2023-11-24T08:12:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.917+0000 7f1405ffb640 1 -- 172.21.15.107:0/255245588 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7f140807a220 con 0x7f14140cf450 2023-11-24T08:12:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:12:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 msgr2=0x7f13f806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 0x7f13f806e1a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f1410008040 comp rx=0 tx=0).stop 2023-11-24T08:12:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 msgr2=0x7f14141f3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f140802bec0 tx=0x7f14080046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 shutdown_connections 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13f806bce0 0x7f13f806e1a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f141415c7f0 0x7f1414158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14140cf450 0x7f14141f3150 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 --2- 172.21.15.107:0/255245588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14140cdfb0 0x7f1414153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 >> 172.21.15.107:0/255245588 conn(0x7f141406a920 msgr2=0x7f1414155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 shutdown_connections 2023-11-24T08:12:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:47.921+0000 7f141b733640 1 -- 172.21.15.107:0/255245588 wait complete. 2023-11-24T08:12:47.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:12:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.149+0000 7f7b3255a640 1 Processor -- start 2023-11-24T08:12:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.149+0000 7f7b3255a640 1 -- start start 2023-11-24T08:12:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b3255a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b2c0c49f0 0x7f7b2c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b3255a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c15ec90 con 0x7f7b2c06de10 2023-11-24T08:12:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b3255a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0cd530 con 0x7f7b2c072160 2023-11-24T08:12:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b3255a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c0c4ed0 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b30af7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b2c0c49f0 0x7f7b2c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b237fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7b2c06de10 0x7f7b2c0705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54940/0 (socket says 172.21.15.107:54940) 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.151+0000 7f7b237fe640 1 -- 172.21.15.107:0/3770985607 learned_addr learned my addr 172.21.15.107:0/3770985607 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.152+0000 7f7b30af7640 1 -- 172.21.15.107:0/3770985607 >> v1:172.21.15.120:6789/0 conn(0x7f7b2c072160 legacy=0x7f7b2c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.152+0000 7f7b30af7640 1 -- 172.21.15.107:0/3770985607 >> v1:172.21.15.107:6789/0 conn(0x7f7b2c06de10 legacy=0x7f7b2c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.152+0000 7f7b30af7640 1 -- 172.21.15.107:0/3770985607 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0c51b0 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.152+0000 7f7b2b7fe640 1 -- 172.21.15.107:0/3770985607 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164889235 0 0) 0x7f7b14001110 con 0x7f7b2c06de10 2023-11-24T08:12:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.153+0000 7f7b30af7640 1 --2- 172.21.15.107:0/3770985607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c0c49f0 0x7f7b2c15e750 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009d90 tx=0x7f7b2402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25676cf0bcdad14 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.153+0000 7f7b2b7fe640 1 -- 172.21.15.107:0/3770985607 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2402cc60 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.153+0000 7f7b2b7fe640 1 -- 172.21.15.107:0/3770985607 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2402cdc0 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.153+0000 7f7b2b7fe640 1 -- 172.21.15.107:0/3770985607 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24034660 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 -- 172.21.15.107:0/3770985607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c0c49f0 msgr2=0x7f7b2c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 --2- 172.21.15.107:0/3770985607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c0c49f0 0x7f7b2c15e750 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009d90 tx=0x7f7b2402beb0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 -- 172.21.15.107:0/3770985607 shutdown_connections 2023-11-24T08:12:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 --2- 172.21.15.107:0/3770985607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c0c49f0 0x7f7b2c15e750 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 -- 172.21.15.107:0/3770985607 >> 172.21.15.107:0/3770985607 conn(0x7f7b2c0b0c50 msgr2=0x7f7b2c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:48.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 -- 172.21.15.107:0/3770985607 shutdown_connections 2023-11-24T08:12:48.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.154+0000 7f7b3255a640 1 -- 172.21.15.107:0/3770985607 wait complete. 2023-11-24T08:12:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.156+0000 7f7b3255a640 1 Processor -- start 2023-11-24T08:12:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.156+0000 7f7b3255a640 1 -- start start 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2c072160 0x7f7b2c14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2c0c49f0 0x7f7b2c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c09cb30 con 0x7f7b2c06de10 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c15cb00 con 0x7f7b2c072160 2023-11-24T08:12:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b3255a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14c610 con 0x7f7b2c0c49f0 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38608/0 (socket says 172.21.15.107:38608) 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.157+0000 7f7b2bfff640 1 -- 172.21.15.107:0/2291645543 learned_addr learned my addr 172.21.15.107:0/2291645543 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 -- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2c0c49f0 msgr2=0x7f7b2c15b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2c0c49f0 0x7f7b2c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 -- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2c072160 msgr2=0x7f7b2c14c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2c072160 0x7f7b2c14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 -- 172.21.15.107:0/2291645543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24009a70 con 0x7f7b2c06de10 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b2bfff640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c430 tx=0x7f7b1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c011790 con 0x7f7b2c06de10 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c0118f0 con 0x7f7b2c06de10 2023-11-24T08:12:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c011ce0 con 0x7f7b2c06de10 2023-11-24T08:12:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.158+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c15bc30 con 0x7f7b2c06de10 2023-11-24T08:12:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.159+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c15c1d0 con 0x7f7b2c06de10 2023-11-24T08:12:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.160+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c04e8b0 con 0x7f7b2c06de10 2023-11-24T08:12:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.160+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f7b1c01c460 con 0x7f7b2c06de10 2023-11-24T08:12:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.161+0000 7f7b297fa640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 0x7f7b0c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.162+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7b1c0883d0 con 0x7f7b2c06de10 2023-11-24T08:12:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.162+0000 7f7b237fe640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 0x7f7b0c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.163+0000 7f7b237fe640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 0x7f7b0c06e1a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b14007fd0 tx=0x7f7b14001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.166+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b1c054bf0 con 0x7f7b2c06de10 2023-11-24T08:12:48.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.358+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7b2c15dab0 con 0x7f7b2c06de10 2023-11-24T08:12:48.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.365+0000 7f7b297fa640 1 -- 172.21.15.107:0/2291645543 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7b1c058210 con 0x7f7b2c06de10 2023-11-24T08:12:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 msgr2=0x7f7b0c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 0x7f7b0c06e1a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b14007fd0 tx=0x7f7b14001310 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 msgr2=0x7f7b2c159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c430 tx=0x7f7b1c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 shutdown_connections 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06bce0 0x7f7b0c06e1a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2c0c49f0 0x7f7b2c15b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2c072160 0x7f7b2c14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 --2- 172.21.15.107:0/2291645543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2c06de10 0x7f7b2c159df0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 >> 172.21.15.107:0/2291645543 conn(0x7f7b2c0b0c50 msgr2=0x7f7b2c0acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 shutdown_connections 2023-11-24T08:12:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.368+0000 7f7b3255a640 1 -- 172.21.15.107:0/2291645543 wait complete. 2023-11-24T08:12:48.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info' 2023-11-24T08:12:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.600+0000 7f69d7f78640 1 Processor -- start 2023-11-24T08:12:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.600+0000 7f69d7f78640 1 -- start start 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d7f78640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69d00ce6c0 0x7f69d00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d7f78640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d00cece0 con 0x7f69d015c7f0 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d7f78640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d006d040 con 0x7f69d00cf450 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d7f78640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d00cfe40 con 0x7f69d00ce6c0 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d5d14640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69d015c7f0 0x7f69d006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54954/0 (socket says 172.21.15.107:54954) 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d5d14640 1 -- 172.21.15.107:0/3496438134 learned_addr learned my addr 172.21.15.107:0/3496438134 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.602+0000 7f69d6515640 1 --2- 172.21.15.107:0/3496438134 >> v2:172.21.15.107:3300/0 conn(0x7f69d00ce6c0 0x7f69d00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.603+0000 7f69d6515640 1 -- 172.21.15.107:0/3496438134 >> v1:172.21.15.120:6789/0 conn(0x7f69d00cf450 legacy=0x7f69d00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.603+0000 7f69d6515640 1 -- 172.21.15.107:0/3496438134 >> v1:172.21.15.107:6789/0 conn(0x7f69d015c7f0 legacy=0x7f69d006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.603+0000 7f69d6515640 1 -- 172.21.15.107:0/3496438134 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d006d320 con 0x7f69d00ce6c0 2023-11-24T08:12:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.604+0000 7f69d6515640 1 --2- 172.21.15.107:0/3496438134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00ce6c0 0x7f69d00cf900 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009d90 tx=0x7f69c802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62715cb67737dd7a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.604+0000 7f69d4d12640 1 -- 172.21.15.107:0/3496438134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c802cc60 con 0x7f69d00ce6c0 2023-11-24T08:12:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.604+0000 7f69d4d12640 1 -- 172.21.15.107:0/3496438134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c802cdc0 con 0x7f69d00ce6c0 2023-11-24T08:12:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.604+0000 7f69d4d12640 1 -- 172.21.15.107:0/3496438134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8034660 con 0x7f69d00ce6c0 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 -- 172.21.15.107:0/3496438134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00ce6c0 msgr2=0x7f69d00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 --2- 172.21.15.107:0/3496438134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00ce6c0 0x7f69d00cf900 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009d90 tx=0x7f69c802beb0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 -- 172.21.15.107:0/3496438134 shutdown_connections 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 --2- 172.21.15.107:0/3496438134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00ce6c0 0x7f69d00cf900 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 -- 172.21.15.107:0/3496438134 >> 172.21.15.107:0/3496438134 conn(0x7f69d006a920 msgr2=0x7f69d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 -- 172.21.15.107:0/3496438134 shutdown_connections 2023-11-24T08:12:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.605+0000 7f69d7f78640 1 -- 172.21.15.107:0/3496438134 wait complete. 2023-11-24T08:12:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.607+0000 7f69d7f78640 1 Processor -- start 2023-11-24T08:12:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.607+0000 7f69d7f78640 1 -- start start 2023-11-24T08:12:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69d00ce6c0 0x7f69d0159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69d015c7f0 0x7f69d01596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d0098ed0 con 0x7f69d00cf450 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d015ab10 con 0x7f69d00ce6c0 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d7f78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d0159f00 con 0x7f69d015c7f0 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d5513640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d5513640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38626/0 (socket says 172.21.15.107:38626) 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d5513640 1 -- 172.21.15.107:0/568782771 learned_addr learned my addr 172.21.15.107:0/568782771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d5d14640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69d00ce6c0 0x7f69d0159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.608+0000 7f69d6515640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69d015c7f0 0x7f69d01596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 -- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69d015c7f0 msgr2=0x7f69d01596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69d015c7f0 0x7f69d01596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 -- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69d00ce6c0 msgr2=0x7f69d0159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69d00ce6c0 0x7f69d0159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 -- 172.21.15.107:0/568782771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8009a70 con 0x7f69d00cf450 2023-11-24T08:12:48.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d5513640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f69c000c430 tx=0x7f69c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0011790 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c00118f0 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0011ce0 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d015a1e0 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.609+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d015a7a0 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.611+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d015a930 con 0x7f69d00cf450 2023-11-24T08:12:48.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.611+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f69c001c460 con 0x7f69d00cf450 2023-11-24T08:12:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.612+0000 7f69c67fc640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 0x7f69b406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.612+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69c0088390 con 0x7f69d00cf450 2023-11-24T08:12:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.613+0000 7f69d5d14640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 0x7f69b406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.613+0000 7f69d5d14640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 0x7f69b406e1d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f69cc005880 tx=0x7f69cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:48.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.617+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69c0054b80 con 0x7f69d00cf450 2023-11-24T08:12:48.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.890+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info"]} v 0) v1 -- 0x7f69d006ef30 con 0x7f69d00cf450 2023-11-24T08:12:48.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.919+0000 7f69c67fc640 1 -- 172.21.15.107:0/568782771 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_subvolume_info"]}]=0 v409) v1 ==== 158+0+0 (secure 0 0 0) 0x7f69c00581a0 con 0x7f69d00cf450 2023-11-24T08:12:48.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 msgr2=0x7f69b406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 0x7f69b406e1d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f69cc005880 tx=0x7f69cc009040 comp rx=0 tx=0).stop 2023-11-24T08:12:48.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 msgr2=0x7f69d0098990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f69c000c430 tx=0x7f69c000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 shutdown_connections 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69b406bd10 0x7f69b406e1d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69d015c7f0 0x7f69d01596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69d00cf450 0x7f69d0098990 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 --2- 172.21.15.107:0/568782771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69d00ce6c0 0x7f69d0159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 >> 172.21.15.107:0/568782771 conn(0x7f69d006a920 msgr2=0x7f69d0154a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 shutdown_connections 2023-11-24T08:12:48.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:48.923+0000 7f69d7f78640 1 -- 172.21.15.107:0/568782771 wait complete. 2023-11-24T08:12:48.944 INFO:tasks.cephfs_test_runner:test_clone_subvolume_info (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:12:48.944 INFO:tasks.cephfs_test_runner:Starting test: test_non_clone_status (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:12:48.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.152+0000 7fe2a5290640 1 Processor -- start 2023-11-24T08:12:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.152+0000 7fe2a5290640 1 -- start start 2023-11-24T08:12:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.153+0000 7fe2a5290640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe2a015c7f0 0x7fe2a006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe2a5290640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe2a5290640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a00cece0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe2a5290640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a0156010 con 0x7fe2a015c7f0 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe2a5290640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a006ce30 con 0x7fe2a00cf450 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe29e7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe29effd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe2a015c7f0 0x7fe2a006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe29e7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:38758/0 (socket says 172.21.15.107:38758) 2023-11-24T08:12:49.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.154+0000 7fe29e7fc640 1 -- 172.21.15.107:0/2867384429 learned_addr learned my addr 172.21.15.107:0/2867384429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:49.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.155+0000 7fe29e7fc640 1 -- 172.21.15.107:0/2867384429 >> v2:172.21.15.120:3300/0 conn(0x7fe2a015c7f0 msgr2=0x7fe2a006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:49.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.155+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/2867384429 >> v2:172.21.15.120:3300/0 conn(0x7fe2a015c7f0 0x7fe2a006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.155+0000 7fe29e7fc640 1 -- 172.21.15.107:0/2867384429 >> v1:172.21.15.120:6789/0 conn(0x7fe2a00cdfb0 legacy=0x7fe2a00ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.155+0000 7fe29e7fc640 1 -- 172.21.15.107:0/2867384429 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a006d040 con 0x7fe2a00cf450 2023-11-24T08:12:49.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.156+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/2867384429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe294009870 tx=0x7fe29402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd5ef2a0539fe58 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.157+0000 7fe29dffb640 1 -- 172.21.15.107:0/2867384429 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294030840 con 0x7fe2a00cf450 2023-11-24T08:12:49.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.157+0000 7fe29dffb640 1 -- 172.21.15.107:0/2867384429 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2940309a0 con 0x7fe2a00cf450 2023-11-24T08:12:49.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.157+0000 7fe29dffb640 1 -- 172.21.15.107:0/2867384429 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294030d30 con 0x7fe2a00cf450 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 -- 172.21.15.107:0/2867384429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 msgr2=0x7fe2a00cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 --2- 172.21.15.107:0/2867384429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe294009870 tx=0x7fe29402b980 comp rx=0 tx=0).stop 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 -- 172.21.15.107:0/2867384429 shutdown_connections 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 --2- 172.21.15.107:0/2867384429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a00cf450 0x7fe2a00cf8d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 --2- 172.21.15.107:0/2867384429 >> v2:172.21.15.120:3300/0 conn(0x7fe2a015c7f0 0x7fe2a006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 -- 172.21.15.107:0/2867384429 >> 172.21.15.107:0/2867384429 conn(0x7fe2a006a920 msgr2=0x7fe2a006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 -- 172.21.15.107:0/2867384429 shutdown_connections 2023-11-24T08:12:49.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.158+0000 7fe2a5290640 1 -- 172.21.15.107:0/2867384429 wait complete. 2023-11-24T08:12:49.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.159+0000 7fe2a5290640 1 Processor -- start 2023-11-24T08:12:49.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.160+0000 7fe2a5290640 1 -- start start 2023-11-24T08:12:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 0x7fe2a0154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a015c7f0 0x7fe2a0157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a01585a0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a01597e0 con 0x7fe2a00cf450 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe2a5290640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a0159ac0 con 0x7fe2a015c7f0 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38634/0 (socket says 172.21.15.107:38634) 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 -- 172.21.15.107:0/1713551172 learned_addr learned my addr 172.21.15.107:0/1713551172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 0x7fe2a0154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 -- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a015c7f0 msgr2=0x7fe2a0157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a015c7f0 0x7fe2a0157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 -- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 msgr2=0x7fe2a0154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 0x7fe2a0154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29effd640 1 -- 172.21.15.107:0/1713551172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2940094d0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.161+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 0x7fe2a0154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:49.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe29effd640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe29000e870 tx=0x7fe29000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2900094f0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29000c940 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a0159da0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29000ccf0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.162+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a01557e0 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.163+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a0067470 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.164+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fe290013470 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.165+0000 7fe2877fe640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 0x7fe28c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.165+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe290088240 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.165+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 0x7fe28c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.166+0000 7fe29e7fc640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 0x7fe28c06e1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe294036000 tx=0x7fe29403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.169+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe290054d10 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.359+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2a00cd730 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.361+0000 7fe2877fe640 1 -- 172.21.15.107:0/1713551172 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+12990 (secure 0 0 0) 0x7fe290021090 con 0x7fe2a00cdfb0 2023-11-24T08:12:49.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:49.363 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":100,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:47.904934+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/1534091061":"2023-11-25T08:12:44.456870+0000","172.21.15.107:6836/1534091061":"2023-11-25T08:12:44.456870+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:49.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 msgr2=0x7fe28c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 0x7fe28c06e1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe294036000 tx=0x7fe29403b040 comp rx=0 tx=0).stop 2023-11-24T08:12:49.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 msgr2=0x7fe2a015a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe29000e870 tx=0x7fe29000ed30 comp rx=0 tx=0).stop 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 shutdown_connections 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe28c06bce0 0x7fe28c06e1a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2a015c7f0 0x7fe2a0157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00cf450 0x7fe2a0154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 --2- 172.21.15.107:0/1713551172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a00cdfb0 0x7fe2a015a2c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.363+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 >> 172.21.15.107:0/1713551172 conn(0x7fe2a006a920 msgr2=0x7fe2a00ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.364+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 shutdown_connections 2023-11-24T08:12:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.364+0000 7fe2a5290640 1 -- 172.21.15.107:0/1713551172 wait complete. 2023-11-24T08:12:49.384 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:49.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:49.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.602+0000 7efff8778640 1 Processor -- start 2023-11-24T08:12:49.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.602+0000 7efff8778640 1 -- start start 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff8778640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7efff0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff8778640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff0166510 con 0x7efff0162540 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff8778640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff01655f0 con 0x7efff00ccbd0 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff8778640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff00cc380 con 0x7efff01660b0 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff6d15640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efff0162540 0x7efff00c9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54966/0 (socket says 172.21.15.107:54966) 2023-11-24T08:12:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.604+0000 7efff6d15640 1 -- 172.21.15.107:0/3576469520 learned_addr learned my addr 172.21.15.107:0/3576469520 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5d13640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7efff0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832960310 0 0) 0x7effe8001490 con 0x7efff0162540 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effdc003520 con 0x7efff0162540 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706682387 0 0) 0x7effdc003520 con 0x7efff0162540 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 msgr2=0x7efff0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7efff0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 >> v1:172.21.15.120:6789/0 conn(0x7efff00ccbd0 legacy=0x7efff00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0162b30 con 0x7efff0162540 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7effe8001b80 con 0x7efff0162540 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 >> v1:172.21.15.107:6789/0 conn(0x7efff0162540 legacy=0x7efff00c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effdc0039a0 0x7effdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc0047c0 0x7effdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7effdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00cc380 con 0x7effdc0039a0 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc00ba80 con 0x7effdc0047c0 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.605+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc00bc40 con 0x7efff01660b0 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effdc0039a0 0x7effdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6d15640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7effdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 -- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 msgr2=0x7effdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff5d13640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc0047c0 0x7effdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7effdc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 -- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc0047c0 msgr2=0x7effdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc0047c0 0x7effdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6514640 1 -- 172.21.15.107:0/3576469520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc003520 con 0x7effdc0039a0 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff6d15640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff01660b0 0x7effdc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.606+0000 7efff5d13640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc0047c0 0x7effdc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.607+0000 7efff6514640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effdc0039a0 0x7effdc003de0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7effec001750 tx=0x7effdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757a0753e707db07 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.607+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec0052d0 con 0x7effdc0039a0 2023-11-24T08:12:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.607+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec02e050 con 0x7effdc0039a0 2023-11-24T08:12:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.607+0000 7efff5512640 1 -- 172.21.15.107:0/3576469520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec0035f0 con 0x7effdc0039a0 2023-11-24T08:12:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 -- 172.21.15.107:0/3576469520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effdc0039a0 msgr2=0x7effdc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 --2- 172.21.15.107:0/3576469520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effdc0039a0 0x7effdc003de0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7effec001750 tx=0x7effdc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff6d15640 1 -- 172.21.15.107:0/3576469520 reap_dead start 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 -- 172.21.15.107:0/3576469520 shutdown_connections 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 -- 172.21.15.107:0/3576469520 >> 172.21.15.107:0/3576469520 conn(0x7efff00b0c50 msgr2=0x7efff00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 -- 172.21.15.107:0/3576469520 shutdown_connections 2023-11-24T08:12:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.608+0000 7efff8778640 1 -- 172.21.15.107:0/3576469520 wait complete. 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.610+0000 7efff8778640 1 Processor -- start 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.610+0000 7efff8778640 1 -- start start 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effdc0039a0 0x7efff0157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 0x7efff0164630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff0164b70 con 0x7efff01ecaf0 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff01ec620 con 0x7efff01641b0 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff8778640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff01eb580 con 0x7effdc0039a0 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff6d15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff6d15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38650/0 (socket says 172.21.15.107:38650) 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.611+0000 7efff6d15640 1 -- 172.21.15.107:0/3631563605 learned_addr learned my addr 172.21.15.107:0/3631563605 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff5d13640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 0x7efff0164630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6514640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effdc0039a0 0x7efff0157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 -- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effdc0039a0 msgr2=0x7efff0157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effdc0039a0 0x7efff0157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 -- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 msgr2=0x7efff0164630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 0x7efff0164630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 -- 172.21.15.107:0/3631563605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff01eb860 con 0x7efff01ecaf0 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff5d13640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 0x7efff0164630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff6d15640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7effe8001710 tx=0x7effe800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8004d60 con 0x7efff01ecaf0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe800bb80 con 0x7efff01ecaf0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff01ebb40 con 0x7efff01ecaf0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.612+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8007110 con 0x7efff01ecaf0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.613+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff01ec100 con 0x7efff01ecaf0 2023-11-24T08:12:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.614+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0162000 con 0x7efff01ecaf0 2023-11-24T08:12:49.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.615+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7effe8007270 con 0x7efff01ecaf0 2023-11-24T08:12:49.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.616+0000 7effe6ffd640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 0x7effd406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:49.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.616+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7effe8088300 con 0x7efff01ecaf0 2023-11-24T08:12:49.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.616+0000 7efff6514640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 0x7effd406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:49.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.617+0000 7efff6514640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 0x7effd406e1d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7effec033000 tx=0x7effec009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:49.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.620+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effe808ce60 con 0x7efff01ecaf0 2023-11-24T08:12:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.796+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efff0166510 con 0x7efff01ecaf0 2023-11-24T08:12:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.798+0000 7effe6ffd640 1 -- 172.21.15.107:0/3631563605 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7efff0166510 con 0x7efff01ecaf0 2023-11-24T08:12:49.800 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:49.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 msgr2=0x7effd406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 0x7effd406e1d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7effec033000 tx=0x7effec009030 comp rx=0 tx=0).stop 2023-11-24T08:12:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 msgr2=0x7efff01ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7effe8001710 tx=0x7effe800b4a0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 shutdown_connections 2023-11-24T08:12:49.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7effd406bd10 0x7effd406e1d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff01ecaf0 0x7efff01ecfa0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff01641b0 0x7efff0164630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 --2- 172.21.15.107:0/3631563605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7effdc0039a0 0x7efff0157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:49.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 >> 172.21.15.107:0/3631563605 conn(0x7efff00b0c50 msgr2=0x7efff00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:49.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 shutdown_connections 2023-11-24T08:12:49.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:49.800+0000 7efff8778640 1 -- 172.21.15.107:0/3631563605 wait complete. 2023-11-24T08:12:49.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status' 2023-11-24T08:12:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.017+0000 7f3796f2c640 1 Processor -- start 2023-11-24T08:12:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.017+0000 7f3796f2c640 1 -- start start 2023-11-24T08:12:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.018+0000 7f3796f2c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37900d0480 0x7f37900cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.018+0000 7f3796f2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900ce0a0 con 0x7f379015da20 2023-11-24T08:12:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.018+0000 7f3796f2c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900d0d70 con 0x7f37900cf2f0 2023-11-24T08:12:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.018+0000 7f3796f2c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37900d0a90 con 0x7f37900d0480 2023-11-24T08:12:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f3794cc8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f379015da20 0x7f37900d1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54980/0 (socket says 172.21.15.107:54980) 2023-11-24T08:12:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f3794cc8640 1 -- 172.21.15.107:0/1370412717 learned_addr learned my addr 172.21.15.107:0/1370412717 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378ffff640 1 --2- 172.21.15.107:0/1370412717 >> v2:172.21.15.120:3300/0 conn(0x7f37900d0480 0x7f37900cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448257407 0 0) 0x7f37900ce0a0 con 0x7f379015da20 2023-11-24T08:12:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c003520 con 0x7f379015da20 2023-11-24T08:12:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 808478269 0 0) 0x7f377c003520 con 0x7f379015da20 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 >> v2:172.21.15.120:3300/0 conn(0x7f37900d0480 msgr2=0x7f37900cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 --2- 172.21.15.107:0/1370412717 >> v2:172.21.15.120:3300/0 conn(0x7f37900d0480 0x7f37900cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 >> v1:172.21.15.120:6789/0 conn(0x7f37900cf2f0 legacy=0x7f379006ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900ce2d0 con 0x7f379015da20 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f37800019f0 con 0x7f379015da20 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 >> v1:172.21.15.107:6789/0 conn(0x7f379015da20 legacy=0x7f37900d1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0039c0 0x7f377c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.019+0000 7f378f7fe640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 0x7f377c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378f7fe640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f377c005180 0x7f377c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900d0a90 con 0x7f377c0047c0 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c00c050 con 0x7f377c005180 2023-11-24T08:12:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c00c210 con 0x7f377c0039c0 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 0x7f377c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 -- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0039c0 msgr2=0x7f377c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0039c0 0x7f377c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 -- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f377c005180 msgr2=0x7f377c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f377c005180 0x7f377c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f378ffff640 1 -- 172.21.15.107:0/1370412717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c003520 con 0x7f377c0047c0 2023-11-24T08:12:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.020+0000 7f37954c9640 1 -- 172.21.15.107:0/1370412717 reap_dead start 2023-11-24T08:12:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.021+0000 7f378ffff640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 0x7f377c004c40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3778009ac0 tx=0x7f377c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8020d7a1d8c9d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.021+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377802c710 con 0x7f377c0047c0 2023-11-24T08:12:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377802c870 con 0x7f377c0047c0 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f378f7fe640 1 -- 172.21.15.107:0/1370412717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377802cc00 con 0x7f377c0047c0 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 -- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 msgr2=0x7f377c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 0x7f377c004c40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3778009ac0 tx=0x7f377c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 -- 172.21.15.107:0/1370412717 shutdown_connections 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 --2- 172.21.15.107:0/1370412717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f377c0047c0 0x7f377c004c40 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 -- 172.21.15.107:0/1370412717 >> 172.21.15.107:0/1370412717 conn(0x7f379009c7e0 msgr2=0x7f379008af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 -- 172.21.15.107:0/1370412717 shutdown_connections 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.022+0000 7f3796f2c640 1 -- 172.21.15.107:0/1370412717 wait complete. 2023-11-24T08:12:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.024+0000 7f3796f2c640 1 Processor -- start 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.024+0000 7f3796f2c640 1 -- start start 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0047c0 0x7f379015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379009edb0 0x7f37900b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900c8010 con 0x7f379015cb50 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379009f780 con 0x7f379009edb0 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f3796f2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379009fa60 con 0x7f377c0047c0 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38674/0 (socket says 172.21.15.107:38674) 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 -- 172.21.15.107:0/124006970 learned_addr learned my addr 172.21.15.107:0/124006970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 -- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0047c0 msgr2=0x7f379015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0047c0 0x7f379015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 -- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379009edb0 msgr2=0x7f37900b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379009edb0 0x7f37900b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.025+0000 7f378ffff640 1 -- 172.21.15.107:0/124006970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900c8260 con 0x7f379015cb50 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f378ffff640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f377802c2b0 tx=0x7f3778004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778003810 con 0x7f379015cb50 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377802de90 con 0x7f379015cb50 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37900c8540 con 0x7f379015cb50 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778003b70 con 0x7f379015cb50 2023-11-24T08:12:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.026+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379009f410 con 0x7f379015cb50 2023-11-24T08:12:50.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.027+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790067b80 con 0x7f379015cb50 2023-11-24T08:12:50.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.028+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3778003cd0 con 0x7f379015cb50 2023-11-24T08:12:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.029+0000 7f378d7fa640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 0x7f377006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.029+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f37780aa260 con 0x7f379015cb50 2023-11-24T08:12:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.029+0000 7f3794cc8640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 0x7f377006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.030+0000 7f3794cc8640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 0x7f377006e1d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f37901605b0 tx=0x7f3780001980 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.034+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3778076a80 con 0x7f379015cb50 2023-11-24T08:12:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.282+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status"]} v 0) v1 -- 0x7f37900d4340 con 0x7f379015cb50 2023-11-24T08:12:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.933+0000 7f378d7fa640 1 -- 172.21.15.107:0/124006970 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status"]}]=0 v411) v1 ==== 157+0+0 (secure 0 0 0) 0x7f377807a0a0 con 0x7f379015cb50 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 msgr2=0x7f377006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 0x7f377006e1d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f37901605b0 tx=0x7f3780001980 comp rx=0 tx=0).stop 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 msgr2=0x7f37900c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f377802c2b0 tx=0x7f3778004730 comp rx=0 tx=0).stop 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 shutdown_connections 2023-11-24T08:12:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f377006bd10 0x7f377006e1d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f379009edb0 0x7f37900b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379015cb50 0x7f37900c8a60 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 --2- 172.21.15.107:0/124006970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f377c0047c0 0x7f379015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 >> 172.21.15.107:0/124006970 conn(0x7f379009c7e0 msgr2=0x7f37901609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 shutdown_connections 2023-11-24T08:12:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:50.938+0000 7f3796f2c640 1 -- 172.21.15.107:0/124006970 wait complete. 2023-11-24T08:12:50.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:12:51.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.167+0000 7f325e07c640 1 Processor -- start 2023-11-24T08:12:51.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.167+0000 7f325e07c640 1 -- start start 2023-11-24T08:12:51.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f325e07c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32580c59b0 0x7f3258166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:51.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f325e07c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32580c5dc0 con 0x7f32580cc0f0 2023-11-24T08:12:51.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f325e07c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325815eea0 con 0x7f325815b440 2023-11-24T08:12:51.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f325e07c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325815b820 con 0x7f32580c59b0 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f3256ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f32580cc0f0 0x7f32580c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54992/0 (socket says 172.21.15.107:54992) 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f3256ffd640 1 -- 172.21.15.107:0/3015090433 learned_addr learned my addr 172.21.15.107:0/3015090433 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.169+0000 7f3257fff640 1 --2- 172.21.15.107:0/3015090433 >> v2:172.21.15.107:3300/0 conn(0x7f32580c59b0 0x7f3258166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.170+0000 7f32567fc640 1 -- 172.21.15.107:0/3015090433 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334936891 0 0) 0x7f32580c5dc0 con 0x7f32580cc0f0 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.170+0000 7f32567fc640 1 -- 172.21.15.107:0/3015090433 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3240003520 con 0x7f32580cc0f0 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.170+0000 7f3257fff640 1 -- 172.21.15.107:0/3015090433 >> v1:172.21.15.120:6789/0 conn(0x7f325815b440 legacy=0x7f32580cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.170+0000 7f3257fff640 1 -- 172.21.15.107:0/3015090433 >> v1:172.21.15.107:6789/0 conn(0x7f32580cc0f0 legacy=0x7f32580c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.170+0000 7f3257fff640 1 -- 172.21.15.107:0/3015090433 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325806d600 con 0x7f32580c59b0 2023-11-24T08:12:51.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.171+0000 7f3257fff640 1 --2- 172.21.15.107:0/3015090433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32580c59b0 0x7f3258166230 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f325002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ac97b9e38bd2c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:51.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.171+0000 7f32567fc640 1 -- 172.21.15.107:0/3015090433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325002cc60 con 0x7f32580c59b0 2023-11-24T08:12:51.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.171+0000 7f32567fc640 1 -- 172.21.15.107:0/3015090433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325002cdc0 con 0x7f32580c59b0 2023-11-24T08:12:51.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f32567fc640 1 -- 172.21.15.107:0/3015090433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250034660 con 0x7f32580c59b0 2023-11-24T08:12:51.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 -- 172.21.15.107:0/3015090433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32580c59b0 msgr2=0x7f3258166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 --2- 172.21.15.107:0/3015090433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32580c59b0 0x7f3258166230 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f325002beb0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 -- 172.21.15.107:0/3015090433 shutdown_connections 2023-11-24T08:12:51.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 --2- 172.21.15.107:0/3015090433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32580c59b0 0x7f3258166230 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 -- 172.21.15.107:0/3015090433 >> 172.21.15.107:0/3015090433 conn(0x7f32580b0c50 msgr2=0x7f32580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:51.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 -- 172.21.15.107:0/3015090433 shutdown_connections 2023-11-24T08:12:51.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.172+0000 7f325e07c640 1 -- 172.21.15.107:0/3015090433 wait complete. 2023-11-24T08:12:51.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.174+0000 7f325e07c640 1 Processor -- start 2023-11-24T08:12:51.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.174+0000 7f325e07c640 1 -- start start 2023-11-24T08:12:51.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580c59b0 0x7f32581f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:51.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580cc0f0 0x7f32580acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:51.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32580ae210 con 0x7f325815b440 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32580ad170 con 0x7f32580c59b0 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f325e07c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32580ad450 con 0x7f32580cc0f0 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f3257fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f3257fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38696/0 (socket says 172.21.15.107:38696) 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.175+0000 7f3257fff640 1 -- 172.21.15.107:0/2316125057 learned_addr learned my addr 172.21.15.107:0/2316125057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f32577fe640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580c59b0 0x7f32581f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 -- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580cc0f0 msgr2=0x7f32580acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580cc0f0 0x7f32580acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 -- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580c59b0 msgr2=0x7f32581f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580c59b0 0x7f32581f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 -- 172.21.15.107:0/2316125057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250009a70 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f3257fff640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f3250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250040650 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500407b0 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250040b60 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.176+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32580ad730 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.177+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32580adcf0 con 0x7f325815b440 2023-11-24T08:12:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.178+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32580adee0 con 0x7f325815b440 2023-11-24T08:12:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.179+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3250040cc0 con 0x7f325815b440 2023-11-24T08:12:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.179+0000 7f323ffff640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 0x7f324406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.180+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f32500aae50 con 0x7f325815b440 2023-11-24T08:12:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.180+0000 7f32577fe640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 0x7f324406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:51.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.180+0000 7f32577fe640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 0x7f324406e1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3248000f60 tx=0x7f3248009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.184+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32500779b0 con 0x7f325815b440 2023-11-24T08:12:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.376+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f325804e8b0 con 0x7f325815b440 2023-11-24T08:12:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.381+0000 7f323ffff640 1 -- 172.21.15.107:0/2316125057 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3250009450 con 0x7f325815b440 2023-11-24T08:12:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.383+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 msgr2=0x7f324406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.383+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 0x7f324406e1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3248000f60 tx=0x7f3248009040 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.383+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 msgr2=0x7f32580c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.383+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f3250004a40 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 shutdown_connections 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f324406bce0 0x7f324406e1a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f325815b440 0x7f32580c7c80 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32580cc0f0 0x7f32580acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 --2- 172.21.15.107:0/2316125057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32580c59b0 0x7f32581f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 >> 172.21.15.107:0/2316125057 conn(0x7f32580b0c50 msgr2=0x7f32580cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 shutdown_connections 2023-11-24T08:12:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:51.384+0000 7f325e07c640 1 -- 172.21.15.107:0/2316125057 wait complete. 2023-11-24T08:12:51.402 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:12:51.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:12:51.523 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:12:51.647 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:12:51.647 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:51.647 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:51.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:12:51.709 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:51.709 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:51.770 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:12:51.771 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:12:51.771 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:51.831 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:12:51.831 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:12:51.831 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:12:51.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:12:51.890 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:12:51.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:12:52.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.107+0000 7f9271659640 1 Processor -- start 2023-11-24T08:12:52.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.107+0000 7f9271659640 1 -- start start 2023-11-24T08:12:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.108+0000 7f9271659640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f926c159e80 0x7f926c071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.108+0000 7f9271659640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f926c06d5c0 0x7f926c163430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.108+0000 7f9271659640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0ce870 con 0x7f926c0cd6a0 2023-11-24T08:12:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f9271659640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0cdc60 con 0x7f926c159e80 2023-11-24T08:12:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f9271659640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0cdf40 con 0x7f926c06d5c0 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f926bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f926c159e80 0x7f926c071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f9263fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f926c0cd6a0 0x7f926c15a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55004/0 (socket says 172.21.15.107:55004) 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f9263fff640 1 -- 172.21.15.107:0/528859921 learned_addr learned my addr 172.21.15.107:0/528859921 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f926bfff640 1 -- 172.21.15.107:0/528859921 >> v2:172.21.15.120:3300/0 conn(0x7f926c06d5c0 msgr2=0x7f926c163430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f926bfff640 1 --2- 172.21.15.107:0/528859921 >> v2:172.21.15.120:3300/0 conn(0x7f926c06d5c0 0x7f926c163430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f926bfff640 1 -- 172.21.15.107:0/528859921 >> v1:172.21.15.107:6789/0 conn(0x7f926c0cd6a0 legacy=0x7f926c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.109+0000 7f926bfff640 1 -- 172.21.15.107:0/528859921 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0ce220 con 0x7f926c159e80 2023-11-24T08:12:52.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.110+0000 7f926bfff640 1 --2- 172.21.15.107:0/528859921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c071610 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f926c0ae9f0 tx=0x7f925c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137da61f6e547940 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f926b7fe640 1 -- 172.21.15.107:0/528859921 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c02e070 con 0x7f926c159e80 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f926b7fe640 1 -- 172.21.15.107:0/528859921 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c0315a0 con 0x7f926c159e80 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f926b7fe640 1 -- 172.21.15.107:0/528859921 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c031930 con 0x7f926c159e80 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 -- 172.21.15.107:0/528859921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 msgr2=0x7f926c071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 --2- 172.21.15.107:0/528859921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c071610 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f926c0ae9f0 tx=0x7f925c02b840 comp rx=0 tx=0).stop 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 -- 172.21.15.107:0/528859921 shutdown_connections 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 --2- 172.21.15.107:0/528859921 >> v2:172.21.15.120:3300/0 conn(0x7f926c06d5c0 0x7f926c163430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 --2- 172.21.15.107:0/528859921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c071610 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.111+0000 7f9271659640 1 -- 172.21.15.107:0/528859921 >> 172.21.15.107:0/528859921 conn(0x7f926c0cb830 msgr2=0x7f926c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:52.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.112+0000 7f9271659640 1 -- 172.21.15.107:0/528859921 shutdown_connections 2023-11-24T08:12:52.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.112+0000 7f9271659640 1 -- 172.21.15.107:0/528859921 wait complete. 2023-11-24T08:12:52.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.113+0000 7f9271659640 1 Processor -- start 2023-11-24T08:12:52.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.113+0000 7f9271659640 1 -- start start 2023-11-24T08:12:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c06d5c0 0x7f926c139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c0cd6a0 0x7f926c139b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c13cb20 con 0x7f926c159e80 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c139850 con 0x7f926c0cd6a0 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.114+0000 7f9271659640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c13c560 con 0x7f926c06d5c0 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38714/0 (socket says 172.21.15.107:38714) 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 -- 172.21.15.107:0/2631136082 learned_addr learned my addr 172.21.15.107:0/2631136082 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 -- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c06d5c0 msgr2=0x7f926c139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c06d5c0 0x7f926c139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 -- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c0cd6a0 msgr2=0x7f926c139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c0cd6a0 0x7f926c139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 -- 172.21.15.107:0/2631136082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c009400 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.115+0000 7f9270e58640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f926400b9a0 tx=0x7f926400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.116+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400cba0 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.116+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926400cd20 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.116+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264014680 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.116+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c13c840 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.116+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c151ae0 con 0x7f926c159e80 2023-11-24T08:12:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.117+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c151d00 con 0x7f926c159e80 2023-11-24T08:12:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.118+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f926401d3b0 con 0x7f926c159e80 2023-11-24T08:12:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.119+0000 7f92697fa640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 0x7f924c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.119+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9264088420 con 0x7f926c159e80 2023-11-24T08:12:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.119+0000 7f926bfff640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 0x7f924c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.119+0000 7f926bfff640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 0x7f924c06e1a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f925c036000 tx=0x7f925c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.123+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9264054bb0 con 0x7f926c159e80 2023-11-24T08:12:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.352+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f926c04e840 con 0x7f926c159e80 2023-11-24T08:12:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.354+0000 7f92697fa640 1 -- 172.21.15.107:0/2631136082 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f9264019070 con 0x7f926c159e80 2023-11-24T08:12:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:52.355 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/262397224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":262397224},{"type":"v1","addr":"172.21.15.120:6835","nonce":262397224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2770184873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2770184873},{"type":"v1","addr":"172.21.15.120:6833","nonce":2770184873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3756032619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3756032619},{"type":"v1","addr":"172.21.15.107:6835","nonce":3756032619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/783603630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":783603630},{"type":"v1","addr":"172.21.15.107:6837","nonce":783603630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[]} 2023-11-24T08:12:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 54 2023-11-24T08:12:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 msgr2=0x7f924c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 0x7f924c06e1a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f925c036000 tx=0x7f925c030000 comp rx=0 tx=0).stop 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 msgr2=0x7f926c13bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f926400b9a0 tx=0x7f926400be60 comp rx=0 tx=0).stop 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 shutdown_connections 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f924c06bce0 0x7f924c06e1a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f926c159e80 0x7f926c13bbc0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c0cd6a0 0x7f926c139b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 --2- 172.21.15.107:0/2631136082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c06d5c0 0x7f926c139310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 >> 172.21.15.107:0/2631136082 conn(0x7f926c0cb830 msgr2=0x7f926c14b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 shutdown_connections 2023-11-24T08:12:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.357+0000 7f9271659640 1 -- 172.21.15.107:0/2631136082 wait complete. 2023-11-24T08:12:52.377 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:12:52.377 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:12:52.377 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:12:52.377 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:12:52.377 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:12:52.378 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:12:52.378 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:12:52.378 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:12:52.378 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:12:52.379 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:12:52.379 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:12:52.379 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:12:52.379 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:12:52.380 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:12:52.380 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:12:52.380 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:12:52.393 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:12:52.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:12:52.397 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:12:52.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:12:52.404 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:12:52.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:12:52.408 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:12:52.408 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:12:52.444 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:12:52.445 DEBUG:teuthology.parallel:result is None 2023-11-24T08:12:52.449 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:12:52.449 DEBUG:teuthology.parallel:result is None 2023-11-24T08:12:52.457 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:12:52.458 DEBUG:teuthology.parallel:result is None 2023-11-24T08:12:52.465 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:12:52.465 DEBUG:teuthology.parallel:result is None 2023-11-24T08:12:52.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:12:52.600 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:12:52.600 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:12:52.621 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:12:52.622 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:12:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.654+0000 7f105c8da640 1 Processor -- start 2023-11-24T08:12:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.654+0000 7f105c8da640 1 -- start start 2023-11-24T08:12:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105c8da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f105815b440 0x7f10580cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105c8da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10580cbf20 0x7f10580c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105c8da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105815eea0 con 0x7f10580c57e0 2023-11-24T08:12:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105c8da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580cc3d0 con 0x7f10580cbf20 2023-11-24T08:12:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105c8da640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105815b820 con 0x7f105815b440 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f1056d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10580c57e0 0x7f10580c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55032/0 (socket says 172.21.15.107:55032) 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f1055d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10580cbf20 0x7f10580c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f1056d9d640 1 -- 172.21.15.107:0/2821390034 learned_addr learned my addr 172.21.15.107:0/2821390034 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.655+0000 7f105659c640 1 --2- 172.21.15.107:0/2821390034 >> v2:172.21.15.120:3300/0 conn(0x7f105815b440 0x7f10580cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.656+0000 7f1055d9b640 1 -- 172.21.15.107:0/2821390034 >> v2:172.21.15.120:3300/0 conn(0x7f105815b440 msgr2=0x7f10580cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.656+0000 7f1055d9b640 1 --2- 172.21.15.107:0/2821390034 >> v2:172.21.15.120:3300/0 conn(0x7f105815b440 0x7f10580cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.656+0000 7f1055d9b640 1 -- 172.21.15.107:0/2821390034 >> v1:172.21.15.107:6789/0 conn(0x7f10580c57e0 legacy=0x7f10580c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.656+0000 7f1055d9b640 1 -- 172.21.15.107:0/2821390034 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105806d600 con 0x7f10580cbf20 2023-11-24T08:12:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.656+0000 7f105559a640 1 -- 172.21.15.107:0/2821390034 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252579467 0 0) 0x7f105815eea0 con 0x7f10580c57e0 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.657+0000 7f1055d9b640 1 --2- 172.21.15.107:0/2821390034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10580cbf20 0x7f10580c52a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f105806e770 tx=0x7f104c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ad54dce08a1565 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.657+0000 7f105559a640 1 -- 172.21.15.107:0/2821390034 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c02e070 con 0x7f10580cbf20 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.657+0000 7f105559a640 1 -- 172.21.15.107:0/2821390034 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c0315a0 con 0x7f10580cbf20 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.657+0000 7f105559a640 1 -- 172.21.15.107:0/2821390034 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c031930 con 0x7f10580cbf20 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 -- 172.21.15.107:0/2821390034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10580cbf20 msgr2=0x7f10580c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 --2- 172.21.15.107:0/2821390034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10580cbf20 0x7f10580c52a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f105806e770 tx=0x7f104c02b840 comp rx=0 tx=0).stop 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 -- 172.21.15.107:0/2821390034 shutdown_connections 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 --2- 172.21.15.107:0/2821390034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10580cbf20 0x7f10580c52a0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 --2- 172.21.15.107:0/2821390034 >> v2:172.21.15.120:3300/0 conn(0x7f105815b440 0x7f10580cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 -- 172.21.15.107:0/2821390034 >> 172.21.15.107:0/2821390034 conn(0x7f10580b0c50 msgr2=0x7f10580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 -- 172.21.15.107:0/2821390034 shutdown_connections 2023-11-24T08:12:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.658+0000 7f105c8da640 1 -- 172.21.15.107:0/2821390034 wait complete. 2023-11-24T08:12:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.660+0000 7f105c8da640 1 Processor -- start 2023-11-24T08:12:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.660+0000 7f105c8da640 1 -- start start 2023-11-24T08:12:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10580c57e0 0x7f10580ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 0x7f1058154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580b3790 con 0x7f105815b440 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580cd1b0 con 0x7f10580cbf20 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105c8da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580cd490 con 0x7f10580c57e0 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38744/0 (socket says 172.21.15.107:38744) 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 -- 172.21.15.107:0/1312460168 learned_addr learned my addr 172.21.15.107:0/1312460168 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f105659c640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10580c57e0 0x7f10580ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1055d9b640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 0x7f1058154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 -- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10580c57e0 msgr2=0x7f10580ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10580c57e0 0x7f10580ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 -- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 msgr2=0x7f1058154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 0x7f1058154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:52.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1056d9d640 1 -- 172.21.15.107:0/1312460168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c009400 con 0x7f105815b440 2023-11-24T08:12:52.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.661+0000 7f1055d9b640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 0x7f1058154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:52.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f1056d9d640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f105000c7a0 tx=0x7f105000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000a640 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105000a7a0 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000ab90 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580cd770 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.662+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10581550c0 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.663+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10580b38d0 con 0x7f105815b440 2023-11-24T08:12:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.664+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f105000acf0 con 0x7f105815b440 2023-11-24T08:12:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.665+0000 7f1046ffd640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 0x7f103806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.665+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1050087940 con 0x7f105815b440 2023-11-24T08:12:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.665+0000 7f105659c640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 0x7f103806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:52.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.666+0000 7f105659c640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 0x7f103806e1a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1048009a30 tx=0x7f1048008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.670+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10580b38d0 con 0x7f105815b440 2023-11-24T08:12:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:52.875+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f10580b38d0 con 0x7f105815b440 2023-11-24T08:12:53.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.979+0000 7f1046ffd640 1 -- 172.21.15.107:0/1312460168 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) v1 ==== 97+0+0 (secure 0 0 0) 0x7f10580b38d0 con 0x7f105815b440 2023-11-24T08:12:53.981 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.983+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 msgr2=0x7f103806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.983+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 0x7f103806e1a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1048009a30 tx=0x7f1048008040 comp rx=0 tx=0).stop 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 msgr2=0x7f10580b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f105000c7a0 tx=0x7f105000cc60 comp rx=0 tx=0).stop 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 shutdown_connections 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103806bce0 0x7f103806e1a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f105815b440 0x7f10580b0470 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10580cbf20 0x7f1058154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:53.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 --2- 172.21.15.107:0/1312460168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10580c57e0 0x7f10580ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:53.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 >> 172.21.15.107:0/1312460168 conn(0x7f10580b0c50 msgr2=0x7f1058070980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:53.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 shutdown_connections 2023-11-24T08:12:53.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:53.984+0000 7f105c8da640 1 -- 172.21.15.107:0/1312460168 wait complete. 2023-11-24T08:12:54.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:12:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.212+0000 7f0cc194c640 1 Processor -- start 2023-11-24T08:12:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.212+0000 7f0cc194c640 1 -- start start 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 0x7f0cbc0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 0x7f0cbc06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cbc06e140 0x7f0cbc0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc06e5d0 con 0x7f0cbc06e140 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc06c630 con 0x7f0cbc0ce8d0 2023-11-24T08:12:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cc194c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc06c910 con 0x7f0cbc15d520 2023-11-24T08:12:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cbb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cbc06e140 0x7f0cbc0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cbb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cbc06e140 0x7f0cbc0abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38754/0 (socket says 172.21.15.107:38754) 2023-11-24T08:12:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cbb7fe640 1 -- 172.21.15.107:0/1500222578 learned_addr learned my addr 172.21.15.107:0/1500222578 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.214+0000 7f0cba7fc640 1 --2- 172.21.15.107:0/1500222578 >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 0x7f0cbc06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbaffd640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 0x7f0cbc0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbb7fe640 1 -- 172.21.15.107:0/1500222578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 msgr2=0x7f0cbc0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbb7fe640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 0x7f0cbc0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbb7fe640 1 -- 172.21.15.107:0/1500222578 >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 msgr2=0x7f0cbc06dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbb7fe640 1 --2- 172.21.15.107:0/1500222578 >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 0x7f0cbc06dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbb7fe640 1 -- 172.21.15.107:0/1500222578 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc06cbf0 con 0x7f0cbc06e140 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cba7fc640 1 --2- 172.21.15.107:0/1500222578 >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 0x7f0cbc06dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.215+0000 7f0cbaffd640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 0x7f0cbc0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.216+0000 7f0cbb7fe640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc0abf50 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc06f980 tx=0x7f0cb402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7ac0583fe95469 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.216+0000 7f0cb9ffb640 1 -- 172.21.15.107:0/1500222578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb402ca80 con 0x7f0cbc06e140 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.216+0000 7f0cb9ffb640 1 -- 172.21.15.107:0/1500222578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb402cbe0 con 0x7f0cbc06e140 2023-11-24T08:12:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.216+0000 7f0cb9ffb640 1 -- 172.21.15.107:0/1500222578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40343f0 con 0x7f0cbc06e140 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 -- 172.21.15.107:0/1500222578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 msgr2=0x7f0cbc0abf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc0abf50 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc06f980 tx=0x7f0cb402bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 -- 172.21.15.107:0/1500222578 shutdown_connections 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc0abf50 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 --2- 172.21.15.107:0/1500222578 >> v2:172.21.15.120:3300/0 conn(0x7f0cbc0ce8d0 0x7f0cbc06dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 --2- 172.21.15.107:0/1500222578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc15d520 0x7f0cbc0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 -- 172.21.15.107:0/1500222578 >> 172.21.15.107:0/1500222578 conn(0x7f0cbc0b0c70 msgr2=0x7f0cbc0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 -- 172.21.15.107:0/1500222578 shutdown_connections 2023-11-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.217+0000 7f0cc194c640 1 -- 172.21.15.107:0/1500222578 wait complete. 2023-11-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.219+0000 7f0cc194c640 1 Processor -- start 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.219+0000 7f0cc194c640 1 -- start start 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc0ce8d0 0x7f0cbc14ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cbc15d520 0x7f0cbc150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc15e8e0 con 0x7f0cbc06e140 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1552d0 con 0x7f0cbc15d520 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cc194c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1555b0 con 0x7f0cbc0ce8d0 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38762/0 (socket says 172.21.15.107:38762) 2023-11-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 -- 172.21.15.107:0/3941660830 learned_addr learned my addr 172.21.15.107:0/3941660830 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbb7fe640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cbc15d520 0x7f0cbc150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 -- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc0ce8d0 msgr2=0x7f0cbc14ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc0ce8d0 0x7f0cbc14ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 -- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cbc15d520 msgr2=0x7f0cbc150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cbc15d520 0x7f0cbc150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.220+0000 7f0cbaffd640 1 -- 172.21.15.107:0/3941660830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4009b30 con 0x7f0cbc06e140 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0cbaffd640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c7f0 tx=0x7f0cac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac007a30 con 0x7f0cbc06e140 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac007b90 con 0x7f0cbc06e140 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00ce40 con 0x7f0cbc06e140 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc155890 con 0x7f0cbc06e140 2023-11-24T08:12:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.221+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc14aac0 con 0x7f0cbc06e140 2023-11-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.222+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc15ea50 con 0x7f0cbc06e140 2023-11-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.223+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f0cac0113f0 con 0x7f0cbc06e140 2023-11-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.224+0000 7f0ca3fff640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 0x7f0ca806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.224+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cac0885a0 con 0x7f0cbc06e140 2023-11-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.224+0000 7f0cba7fc640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 0x7f0ca806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.225+0000 7f0cba7fc640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 0x7f0ca806e1a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0006fd0 tx=0x7f0cb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.231+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cac054d60 con 0x7f0cbc06e140 2023-11-24T08:12:54.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.488+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0cbc155db0 con 0x7f0cbc06e140 2023-11-24T08:12:54.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.489+0000 7f0ca3fff640 1 -- 172.21.15.107:0/3941660830 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0cac01f090 con 0x7f0cbc06e140 2023-11-24T08:12:54.490 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.492 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:12:54.493 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.494 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:12:54.496 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.497 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.499 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.500 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:54.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:12:54.503 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:54.504 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:54.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:12:54.506 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 msgr2=0x7f0ca806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 0x7f0ca806e1a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0006fd0 tx=0x7f0cb0008040 comp rx=0 tx=0).stop 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 msgr2=0x7f0cbc14a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c7f0 tx=0x7f0cac00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 shutdown_connections 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ca806bce0 0x7f0ca806e1a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cbc15d520 0x7f0cbc150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cbc0ce8d0 0x7f0cbc14ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 --2- 172.21.15.107:0/3941660830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cbc06e140 0x7f0cbc14a580 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 >> 172.21.15.107:0/3941660830 conn(0x7f0cbc0b0c70 msgr2=0x7f0cbc0ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:54.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 shutdown_connections 2023-11-24T08:12:54.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.492+0000 7f0cc194c640 1 -- 172.21.15.107:0/3941660830 wait complete. 2023-11-24T08:12:54.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.717+0000 7f242f997640 1 Processor -- start 2023-11-24T08:12:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.717+0000 7f242f997640 1 -- start start 2023-11-24T08:12:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f242815c7f0 0x7f242806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 0x7f24280cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280cfe40 con 0x7f242815c7f0 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2428156010 con 0x7f24280cf450 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242f997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242806ce30 con 0x7f24280cdfb0 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242d733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f242815c7f0 0x7f242806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242d733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f242815c7f0 0x7f242806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38778/0 (socket says 172.21.15.107:38778) 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242d733640 1 -- 172.21.15.107:0/3607788496 learned_addr learned my addr 172.21.15.107:0/3607788496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242df34640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.719+0000 7f242cf32640 1 --2- 172.21.15.107:0/3607788496 >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 0x7f24280cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242d733640 1 -- 172.21.15.107:0/3607788496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 msgr2=0x7f24280ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242d733640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242d733640 1 -- 172.21.15.107:0/3607788496 >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 msgr2=0x7f24280cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242d733640 1 --2- 172.21.15.107:0/3607788496 >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 0x7f24280cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242d733640 1 -- 172.21.15.107:0/3607788496 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280cece0 con 0x7f242815c7f0 2023-11-24T08:12:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.720+0000 7f242cf32640 1 --2- 172.21.15.107:0/3607788496 >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 0x7f24280cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.721+0000 7f242d733640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f242806db20 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2428070410 tx=0x7f242402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fa24e19b2fcef1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.721+0000 7f241ffff640 1 -- 172.21.15.107:0/3607788496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424031860 con 0x7f242815c7f0 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.721+0000 7f241ffff640 1 -- 172.21.15.107:0/3607788496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24240319c0 con 0x7f242815c7f0 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.721+0000 7f241ffff640 1 -- 172.21.15.107:0/3607788496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424031d50 con 0x7f242815c7f0 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 -- 172.21.15.107:0/3607788496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 msgr2=0x7f242806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f242806db20 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2428070410 tx=0x7f242402b900 comp rx=0 tx=0).stop 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 -- 172.21.15.107:0/3607788496 shutdown_connections 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 --2- 172.21.15.107:0/3607788496 >> v2:172.21.15.120:3300/0 conn(0x7f24280cf450 0x7f24280cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 --2- 172.21.15.107:0/3607788496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f242806db20 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 -- 172.21.15.107:0/3607788496 >> 172.21.15.107:0/3607788496 conn(0x7f242806a920 msgr2=0x7f242806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 -- 172.21.15.107:0/3607788496 shutdown_connections 2023-11-24T08:12:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.722+0000 7f242f997640 1 -- 172.21.15.107:0/3607788496 wait complete. 2023-11-24T08:12:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.724+0000 7f242f997640 1 Processor -- start 2023-11-24T08:12:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.724+0000 7f242f997640 1 -- start start 2023-11-24T08:12:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 0x7f24281552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280ace70 con 0x7f242815c7f0 2023-11-24T08:12:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280ad150 con 0x7f24280cf450 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242f997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280ad430 con 0x7f24280cdfb0 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38788/0 (socket says 172.21.15.107:38788) 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 -- 172.21.15.107:0/3428246810 learned_addr learned my addr 172.21.15.107:0/3428246810 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242d733640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242cf32640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 0x7f24281552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 -- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 msgr2=0x7f24280adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 -- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 msgr2=0x7f24281552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 0x7f24281552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.725+0000 7f242df34640 1 -- 172.21.15.107:0/3428246810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240094d0 con 0x7f242815c7f0 2023-11-24T08:12:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f242cf32640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 0x7f24281552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f242df34640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2420007d10 tx=0x7f242000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420014420 con 0x7f242815c7f0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24200145a0 con 0x7f242815c7f0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420014950 con 0x7f242815c7f0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24280ad710 con 0x7f242815c7f0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.726+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24280ae450 con 0x7f242815c7f0 2023-11-24T08:12:54.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.727+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24280adc30 con 0x7f242815c7f0 2023-11-24T08:12:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.728+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f2420014ab0 con 0x7f242815c7f0 2023-11-24T08:12:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.729+0000 7f241dffb640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 0x7f240c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.729+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2420088c30 con 0x7f242815c7f0 2023-11-24T08:12:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.729+0000 7f242d733640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 0x7f240c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.730+0000 7f242d733640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 0x7f240c06e1a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:54.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.734+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24280adc30 con 0x7f242815c7f0 2023-11-24T08:12:54.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.922+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24280adc30 con 0x7f242815c7f0 2023-11-24T08:12:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.924+0000 7f241dffb640 1 -- 172.21.15.107:0/3428246810 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+12859 (secure 0 0 0) 0x7f24280adc30 con 0x7f242815c7f0 2023-11-24T08:12:54.925 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:54.925 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:53.976093+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.926+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 msgr2=0x7f240c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.926+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 0x7f240c06e1a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242403c040 comp rx=0 tx=0).stop 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.926+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 msgr2=0x7f24280c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.926+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2420007d10 tx=0x7f242000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 shutdown_connections 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f240c06bce0 0x7f240c06e1a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f242815c7f0 0x7f24280c9480 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24280cf450 0x7f24281552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 --2- 172.21.15.107:0/3428246810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24280cdfb0 0x7f24280adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:54.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 >> 172.21.15.107:0/3428246810 conn(0x7f242806a920 msgr2=0x7f242806c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:54.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 shutdown_connections 2023-11-24T08:12:54.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:54.927+0000 7f242f997640 1 -- 172.21.15.107:0/3428246810 wait complete. 2023-11-24T08:12:54.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:12:54.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:12:55.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.164+0000 7efdbefa9640 1 Processor -- start 2023-11-24T08:12:55.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.164+0000 7efdbefa9640 1 -- start start 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbefa9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdb8072160 0x7efdb815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbefa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb80c49f0 0x7efdb806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbefa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb815ec90 con 0x7efdb806dc40 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbefa9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb806e0f0 con 0x7efdb8072160 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbefa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80cd530 con 0x7efdb80c49f0 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbcd45640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdb8072160 0x7efdb815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbd546640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efdb806dc40 0x7efdb80cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55046/0 (socket says 172.21.15.107:55046) 2023-11-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.166+0000 7efdbd546640 1 -- 172.21.15.107:0/180866039 learned_addr learned my addr 172.21.15.107:0/180866039 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:55.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdaffff640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb80c49f0 0x7efdb806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdaf7fe640 1 -- 172.21.15.107:0/180866039 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551014206 0 0) 0x7efdb0001490 con 0x7efdb806dc40 2023-11-24T08:12:55.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdaf7fe640 1 -- 172.21.15.107:0/180866039 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0003520 con 0x7efdb806dc40 2023-11-24T08:12:55.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdbcd45640 1 -- 172.21.15.107:0/180866039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb80c49f0 msgr2=0x7efdb806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:55.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdbcd45640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb80c49f0 0x7efdb806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdbcd45640 1 -- 172.21.15.107:0/180866039 >> v1:172.21.15.107:6789/0 conn(0x7efdb806dc40 legacy=0x7efdb80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.167+0000 7efdbcd45640 1 -- 172.21.15.107:0/180866039 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb80c4ed0 con 0x7efdb8072160 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.168+0000 7efdbcd45640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb8072160 0x7efdb815e750 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99e5321ac52401 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.168+0000 7efdaf7fe640 1 -- 172.21.15.107:0/180866039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4031830 con 0x7efdb8072160 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.168+0000 7efdaf7fe640 1 -- 172.21.15.107:0/180866039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb4031990 con 0x7efdb8072160 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.168+0000 7efdaf7fe640 1 -- 172.21.15.107:0/180866039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4031d20 con 0x7efdb8072160 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 -- 172.21.15.107:0/180866039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb8072160 msgr2=0x7efdb815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb8072160 0x7efdb815e750 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 -- 172.21.15.107:0/180866039 shutdown_connections 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb80c49f0 0x7efdb806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 --2- 172.21.15.107:0/180866039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb8072160 0x7efdb815e750 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 -- 172.21.15.107:0/180866039 >> 172.21.15.107:0/180866039 conn(0x7efdb80b0c50 msgr2=0x7efdb80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 -- 172.21.15.107:0/180866039 shutdown_connections 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.169+0000 7efdbefa9640 1 -- 172.21.15.107:0/180866039 wait complete. 2023-11-24T08:12:55.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.171+0000 7efdbefa9640 1 Processor -- start 2023-11-24T08:12:55.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.171+0000 7efdbefa9640 1 -- start start 2023-11-24T08:12:55.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb8072160 0x7efdb81f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80c49f0 0x7efdb80aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f1740 con 0x7efdb806dc40 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f1a20 con 0x7efdb80c49f0 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbefa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f1d00 con 0x7efdb8072160 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbcd45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbcd45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38804/0 (socket says 172.21.15.107:38804) 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.172+0000 7efdbcd45640 1 -- 172.21.15.107:0/250571021 learned_addr learned my addr 172.21.15.107:0/250571021 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 -- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb8072160 msgr2=0x7efdb81f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb8072160 0x7efdb81f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 -- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80c49f0 msgr2=0x7efdb80aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80c49f0 0x7efdb80aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 -- 172.21.15.107:0/250571021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb4009400 con 0x7efdb806dc40 2023-11-24T08:12:55.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbcd45640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7efdb402bee0 tx=0x7efdb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb403a6f0 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb403a870 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb403ac20 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb81f1fe0 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.173+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb81f13f0 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.174+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb804e8b0 con 0x7efdb806dc40 2023-11-24T08:12:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.175+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7efdb4030380 con 0x7efdb806dc40 2023-11-24T08:12:55.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.176+0000 7efdad7fa640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 0x7efda406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.176+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efdb40a9cf0 con 0x7efdb806dc40 2023-11-24T08:12:55.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.176+0000 7efdaffff640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 0x7efda406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.177+0000 7efdaffff640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 0x7efda406e1d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.181+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdb4076510 con 0x7efdb806dc40 2023-11-24T08:12:55.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.368+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efdb809cc10 con 0x7efdb806dc40 2023-11-24T08:12:55.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.369+0000 7efdad7fa640 1 -- 172.21.15.107:0/250571021 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7efdb40350a0 con 0x7efdb806dc40 2023-11-24T08:12:55.370 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 msgr2=0x7efda406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 0x7efda406e1d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8008040 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 msgr2=0x7efdb814b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7efdb402bee0 tx=0x7efdb40046c0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 shutdown_connections 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda406bd10 0x7efda406e1d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80c49f0 0x7efdb80aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb8072160 0x7efdb81f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 --2- 172.21.15.107:0/250571021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb806dc40 0x7efdb814b460 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 >> 172.21.15.107:0/250571021 conn(0x7efdb80b0c50 msgr2=0x7efdb80acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.371+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 shutdown_connections 2023-11-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.372+0000 7efdbefa9640 1 -- 172.21.15.107:0/250571021 wait complete. 2023-11-24T08:12:55.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:12:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.580+0000 7f3441da9640 1 Processor -- start 2023-11-24T08:12:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.580+0000 7f3441da9640 1 -- start start 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3441da9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f343c15b320 0x7f343c06d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3441da9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c15e8b0 con 0x7f343c0cf5d0 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3441da9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c070180 con 0x7f343c15af40 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3441da9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c15eb90 con 0x7f343c15b320 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3432ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f343c0cf5d0 0x7f343c06b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55052/0 (socket says 172.21.15.107:55052) 2023-11-24T08:12:55.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.582+0000 7f3432ffd640 1 -- 172.21.15.107:0/2336302006 learned_addr learned my addr 172.21.15.107:0/2336302006 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:55.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343bfff640 1 --2- 172.21.15.107:0/2336302006 >> v2:172.21.15.120:3300/0 conn(0x7f343c15b320 0x7f343c06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237663354 0 0) 0x7f343c15e8b0 con 0x7f343c0cf5d0 2023-11-24T08:12:55.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3428003520 con 0x7f343c0cf5d0 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008142091 0 0) 0x7f3428003520 con 0x7f343c0cf5d0 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 >> v2:172.21.15.120:3300/0 conn(0x7f343c15b320 msgr2=0x7f343c06d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 --2- 172.21.15.107:0/2336302006 >> v2:172.21.15.120:3300/0 conn(0x7f343c15b320 0x7f343c06d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 >> v1:172.21.15.120:6789/0 conn(0x7f343c15af40 legacy=0x7f343c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c070460 con 0x7f343c0cf5d0 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3424001670 con 0x7f343c0cf5d0 2023-11-24T08:12:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 >> v1:172.21.15.107:6789/0 conn(0x7f343c0cf5d0 legacy=0x7f343c06b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280039c0 0x7f3428003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 0x7f3428004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3428005180 0x7f342800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c15eb90 con 0x7f34280047c0 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342800c080 con 0x7f34280039c0 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.583+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342800c240 con 0x7f3428005180 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 0x7f3428004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f343b7fe640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280039c0 0x7f3428003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 -- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3428005180 msgr2=0x7f342800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3428005180 0x7f342800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 -- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280039c0 msgr2=0x7f3428003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:55.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280039c0 0x7f3428003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f3432ffd640 1 -- 172.21.15.107:0/2336302006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428003520 con 0x7f34280047c0 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.584+0000 7f343bfff640 1 -- 172.21.15.107:0/2336302006 reap_dead start 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.585+0000 7f3432ffd640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 0x7f3428004c40 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f343c071310 tx=0x7f34280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bb887c950dd277 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.585+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424005ad0 con 0x7f34280047c0 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.585+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424005c30 con 0x7f34280047c0 2023-11-24T08:12:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.585+0000 7f343affd640 1 -- 172.21.15.107:0/2336302006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240072b0 con 0x7f34280047c0 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 -- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 msgr2=0x7f3428004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 0x7f3428004c40 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f343c071310 tx=0x7f34280057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 -- 172.21.15.107:0/2336302006 shutdown_connections 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 --2- 172.21.15.107:0/2336302006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34280047c0 0x7f3428004c40 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 -- 172.21.15.107:0/2336302006 >> 172.21.15.107:0/2336302006 conn(0x7f343c06e7f0 msgr2=0x7f343c06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 -- 172.21.15.107:0/2336302006 shutdown_connections 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.586+0000 7f3441da9640 1 -- 172.21.15.107:0/2336302006 wait complete. 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.587+0000 7f3441da9640 1 Processor -- start 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 -- start start 2023-11-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280047c0 0x7f343c156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f343c157fe0 0x7f343c158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0afb90 con 0x7f343c1589a0 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c156d90 con 0x7f34280047c0 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.588+0000 7f3441da9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c158e20 con 0x7f343c157fe0 2023-11-24T08:12:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38832/0 (socket says 172.21.15.107:38832) 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 -- 172.21.15.107:0/3076225728 learned_addr learned my addr 172.21.15.107:0/3076225728 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 -- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f343c157fe0 msgr2=0x7f343c158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f343c157fe0 0x7f343c158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 -- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280047c0 msgr2=0x7f343c156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280047c0 0x7f343c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 -- 172.21.15.107:0/3076225728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0c5470 con 0x7f343c1589a0 2023-11-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.589+0000 7f343bfff640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f343400ee50 tx=0x7f343400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.590+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434010040 con 0x7f343c1589a0 2023-11-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.590+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400bbf0 con 0x7f343c1589a0 2023-11-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.590+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434014400 con 0x7f343c1589a0 2023-11-24T08:12:55.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.590+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c0c5710 con 0x7f343c1589a0 2023-11-24T08:12:55.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.590+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0c5cd0 con 0x7f343c1589a0 2023-11-24T08:12:55.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.591+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c0c5ef0 con 0x7f343c1589a0 2023-11-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.592+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3434014560 con 0x7f343c1589a0 2023-11-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.593+0000 7f3438ff9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 0x7f341c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.593+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3434088730 con 0x7f343c1589a0 2023-11-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.593+0000 7f343b7fe640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 0x7f341c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.594+0000 7f343b7fe640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 0x7f341c06e1d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f342c007fd0 tx=0x7f342c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:55.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.597+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3434054fb0 con 0x7f343c1589a0 2023-11-24T08:12:55.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.791+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f343c067470 con 0x7f343c1589a0 2023-11-24T08:12:55.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.792+0000 7f3438ff9640 1 -- 172.21.15.107:0/3076225728 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) v1 ==== 84+0+65 (secure 0 0 0) 0x7f34340585d0 con 0x7f343c1589a0 2023-11-24T08:12:55.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:12:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 msgr2=0x7f341c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 0x7f341c06e1d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f342c007fd0 tx=0x7f342c009040 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 msgr2=0x7f343c0c61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f343400ee50 tx=0x7f343400b480 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 shutdown_connections 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341c06bd10 0x7f341c06e1d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c1589a0 0x7f343c0c61f0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f343c157fe0 0x7f343c158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 --2- 172.21.15.107:0/3076225728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280047c0 0x7f343c156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.795+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 >> 172.21.15.107:0/3076225728 conn(0x7f343c06e7f0 msgr2=0x7f343c0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.796+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 shutdown_connections 2023-11-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:55.796+0000 7f3441da9640 1 -- 172.21.15.107:0/3076225728 wait complete. 2023-11-24T08:12:55.815 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:12:55.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:12:56.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.035+0000 7f9e0706b640 1 Processor -- start 2023-11-24T08:12:56.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.036+0000 7f9e0706b640 1 -- start start 2023-11-24T08:12:56.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e0706b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9e0015c7f0 0x7f9e0006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e0706b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 0x7f9e000cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e0706b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000cece0 con 0x7f9e000cdfb0 2023-11-24T08:12:56.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e0706b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000cfe40 con 0x7f9e0015c7f0 2023-11-24T08:12:56.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e0706b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000ceaf0 con 0x7f9e000cf450 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e04e07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9e0015c7f0 0x7f9e0006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.037+0000 7f9e05608640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9e000cdfb0 0x7f9e000ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55062/0 (socket says 172.21.15.107:55062) 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9e05608640 1 -- 172.21.15.107:0/1441025748 learned_addr learned my addr 172.21.15.107:0/1441025748 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9dfffff640 1 --2- 172.21.15.107:0/1441025748 >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 0x7f9e000cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9e04e07640 1 -- 172.21.15.107:0/1441025748 >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 msgr2=0x7f9e000cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9e04e07640 1 --2- 172.21.15.107:0/1441025748 >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 0x7f9e000cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9e04e07640 1 -- 172.21.15.107:0/1441025748 >> v1:172.21.15.107:6789/0 conn(0x7f9e000cdfb0 legacy=0x7f9e000ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9e04e07640 1 -- 172.21.15.107:0/1441025748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0006d040 con 0x7f9e0015c7f0 2023-11-24T08:12:56.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.038+0000 7f9dfffff640 1 --2- 172.21.15.107:0/1441025748 >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 0x7f9e000cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:56.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.039+0000 7f9e04e07640 1 --2- 172.21.15.107:0/1441025748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0006db20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f9e00070410 tx=0x7f9df402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f059654538b1366 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:56.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9dff7fe640 1 -- 172.21.15.107:0/1441025748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4031830 con 0x7f9e0015c7f0 2023-11-24T08:12:56.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9dff7fe640 1 -- 172.21.15.107:0/1441025748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4031990 con 0x7f9e0015c7f0 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9dff7fe640 1 -- 172.21.15.107:0/1441025748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4031d20 con 0x7f9e0015c7f0 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 -- 172.21.15.107:0/1441025748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 msgr2=0x7f9e0006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 --2- 172.21.15.107:0/1441025748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0006db20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f9e00070410 tx=0x7f9df402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 -- 172.21.15.107:0/1441025748 shutdown_connections 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 --2- 172.21.15.107:0/1441025748 >> v2:172.21.15.120:3300/0 conn(0x7f9e000cf450 0x7f9e000cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 --2- 172.21.15.107:0/1441025748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0006db20 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 -- 172.21.15.107:0/1441025748 >> 172.21.15.107:0/1441025748 conn(0x7f9e0006a920 msgr2=0x7f9e0006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.040+0000 7f9e0706b640 1 -- 172.21.15.107:0/1441025748 shutdown_connections 2023-11-24T08:12:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.041+0000 7f9e0706b640 1 -- 172.21.15.107:0/1441025748 wait complete. 2023-11-24T08:12:56.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.042+0000 7f9e0706b640 1 Processor -- start 2023-11-24T08:12:56.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.042+0000 7f9e0706b640 1 -- start start 2023-11-24T08:12:56.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 0x7f9e00157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 0x7f9e000ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e001f4770 con 0x7f9e0015c7f0 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e001f4a50 con 0x7f9e000cdfb0 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.043+0000 7f9e0706b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e001f4d30 con 0x7f9e000cf450 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38858/0 (socket says 172.21.15.107:38858) 2023-11-24T08:12:56.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 -- 172.21.15.107:0/4268306513 learned_addr learned my addr 172.21.15.107:0/4268306513 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e04e07640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 0x7f9e00157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9dfffff640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 0x7f9e000ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 -- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 msgr2=0x7f9e000ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 0x7f9e000ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 -- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 msgr2=0x7f9e00157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 0x7f9e00157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 -- 172.21.15.107:0/4268306513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4009400 con 0x7f9e0015c7f0 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9dfffff640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 0x7f9e000ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e04e07640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 0x7f9e00157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.044+0000 7f9e05608640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f9df800ca70 tx=0x7f9df800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:56.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.045+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df800abd0 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.045+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800ad30 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.045+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e001f5010 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.045+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8012600 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.045+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e001f55d0 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.046+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00067470 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.047+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f9df8007070 con 0x7f9e0015c7f0 2023-11-24T08:12:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.048+0000 7f9dfd7fa640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 0x7f9de406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:56.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.049+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9df8088120 con 0x7f9e0015c7f0 2023-11-24T08:12:56.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.049+0000 7f9e04e07640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 0x7f9de406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:56.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.049+0000 7f9e04e07640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 0x7f9de406e1a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9df4036000 tx=0x7f9df403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.053+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9df8054a30 con 0x7f9e0015c7f0 2023-11-24T08:12:56.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:56.244+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9e000cd730 con 0x7f9e0015c7f0 2023-11-24T08:12:57.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.010+0000 7f9dfd7fa640 1 -- 172.21.15.107:0/4268306513 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9df8017020 con 0x7f9e0015c7f0 2023-11-24T08:12:57.012 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 msgr2=0x7f9de406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 0x7f9de406e1a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9df4036000 tx=0x7f9df403c040 comp rx=0 tx=0).stop 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 msgr2=0x7f9e0009a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f9df800ca70 tx=0x7f9df800cf30 comp rx=0 tx=0).stop 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 shutdown_connections 2023-11-24T08:12:57.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9de406bce0 0x7f9de406e1a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e0015c7f0 0x7f9e0009a180 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e000cf450 0x7f9e000ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 --2- 172.21.15.107:0/4268306513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e000cdfb0 0x7f9e00157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 >> 172.21.15.107:0/4268306513 conn(0x7f9e0006a920 msgr2=0x7f9e0006c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:57.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.014+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 shutdown_connections 2023-11-24T08:12:57.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.015+0000 7f9e0706b640 1 -- 172.21.15.107:0/4268306513 wait complete. 2023-11-24T08:12:57.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:12:57.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.248+0000 7fd6fd1c9640 1 Processor -- start 2023-11-24T08:12:57.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.249+0000 7fd6fd1c9640 1 -- start start 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8140340 0x7fd6f80ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6f8140720 0x7fd6f80cb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6f8161080 0x7fd6f80c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f8161600 con 0x7fd6f8161080 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f81618e0 con 0x7fd6f8140720 2023-11-24T08:12:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.250+0000 7fd6fd1c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8161bc0 con 0x7fd6f8140340 2023-11-24T08:12:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6f8161080 0x7fd6f80c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6f8161080 0x7fd6f80c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38872/0 (socket says 172.21.15.107:38872) 2023-11-24T08:12:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 -- 172.21.15.107:0/2892731925 learned_addr learned my addr 172.21.15.107:0/2892731925 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 -- 172.21.15.107:0/2892731925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8140340 msgr2=0x7fd6f80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 --2- 172.21.15.107:0/2892731925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8140340 0x7fd6f80ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 -- 172.21.15.107:0/2892731925 >> v2:172.21.15.120:3300/0 conn(0x7fd6f8140720 msgr2=0x7fd6f80cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 --2- 172.21.15.107:0/2892731925 >> v2:172.21.15.120:3300/0 conn(0x7fd6f8140720 0x7fd6f80cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.251+0000 7fd6f759e640 1 -- 172.21.15.107:0/2892731925 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8161ea0 con 0x7fd6f8161080 2023-11-24T08:12:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.252+0000 7fd6f759e640 1 --2- 172.21.15.107:0/2892731925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8161080 0x7fd6f80c60b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8068130 tx=0x7fd6f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bf51ac501c5d84 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6f5d9b640 1 -- 172.21.15.107:0/2892731925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f002ca80 con 0x7fd6f8161080 2023-11-24T08:12:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6f5d9b640 1 -- 172.21.15.107:0/2892731925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f002cbe0 con 0x7fd6f8161080 2023-11-24T08:12:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6f5d9b640 1 -- 172.21.15.107:0/2892731925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f00343f0 con 0x7fd6f8161080 2023-11-24T08:12:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/2892731925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8161080 msgr2=0x7fd6f80c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/2892731925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8161080 0x7fd6f80c60b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8068130 tx=0x7fd6f002bee0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/2892731925 shutdown_connections 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/2892731925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8161080 0x7fd6f80c60b0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/2892731925 >> v2:172.21.15.120:3300/0 conn(0x7fd6f8140720 0x7fd6f80cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/2892731925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8140340 0x7fd6f80ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/2892731925 >> 172.21.15.107:0/2892731925 conn(0x7fd6f80b2e40 msgr2=0x7fd6f80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/2892731925 shutdown_connections 2023-11-24T08:12:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.253+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/2892731925 wait complete. 2023-11-24T08:12:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.255+0000 7fd6fd1c9640 1 Processor -- start 2023-11-24T08:12:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.255+0000 7fd6fd1c9640 1 -- start start 2023-11-24T08:12:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f8140720 0x7fd6f8158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8161080 0x7fd6f80d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f815dec0 con 0x7fd6f8140340 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8159430 con 0x7fd6f8140720 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.256+0000 7fd6fd1c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80c9730 con 0x7fd6f8161080 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38886/0 (socket says 172.21.15.107:38886) 2023-11-24T08:12:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 -- 172.21.15.107:0/1363671843 learned_addr learned my addr 172.21.15.107:0/1363671843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 -- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8161080 msgr2=0x7fd6f80d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8161080 0x7fd6f80d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 -- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f8140720 msgr2=0x7fd6f8158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f8140720 0x7fd6f8158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 -- 172.21.15.107:0/1363671843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f0009b30 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.257+0000 7fd6f6d9d640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800b4a0 tx=0x7fd6e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.258+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8010610 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.258+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8010770 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.258+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f80c9a10 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.258+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8010b20 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.258+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f80c9fd0 con 0x7fd6f8140340 2023-11-24T08:12:57.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.259+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f80ca1f0 con 0x7fd6f8140340 2023-11-24T08:12:57.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.261+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fd6e8010c80 con 0x7fd6f8140340 2023-11-24T08:12:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.261+0000 7fd6ef7fe640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 0x7fd6d806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.261+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd6e80884c0 con 0x7fd6f8140340 2023-11-24T08:12:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.262+0000 7fd6f659c640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 0x7fd6d806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.262+0000 7fd6f659c640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 0x7fd6d806e1a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80d1f50 tx=0x7fd6e0006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.265+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6e8054d20 con 0x7fd6f8140340 2023-11-24T08:12:57.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:57.456+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd6f804e8b0 con 0x7fd6f8140340 2023-11-24T08:12:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.026+0000 7fd6ef7fe640 1 -- 172.21.15.107:0/1363671843 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd6e801e020 con 0x7fd6f8140340 2023-11-24T08:12:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:12:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.029+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 msgr2=0x7fd6d806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.029+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 0x7fd6d806e1a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80d1f50 tx=0x7fd6e0006c90 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.029+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 msgr2=0x7fd6f8132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.029+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800b4a0 tx=0x7fd6e800b960 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 shutdown_connections 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6d806bce0 0x7fd6d806e1a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f8161080 0x7fd6f80d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f8140720 0x7fd6f8158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 --2- 172.21.15.107:0/1363671843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6f8140340 0x7fd6f8132590 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 >> 172.21.15.107:0/1363671843 conn(0x7fd6f80b2e40 msgr2=0x7fd6f80cc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 shutdown_connections 2023-11-24T08:12:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.030+0000 7fd6fd1c9640 1 -- 172.21.15.107:0/1363671843 wait complete. 2023-11-24T08:12:58.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:12:58.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.249+0000 7f39aa2df640 1 Processor -- start 2023-11-24T08:12:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.249+0000 7f39aa2df640 1 -- start start 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39aa2df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 0x7f39a40cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39aa2df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39a40b63d0 0x7f39a40d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39aa2df640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40cea30 con 0x7f39a40ce270 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39aa2df640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40d0450 con 0x7f39a40b63d0 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39aa2df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40d0bf0 con 0x7f39a415d490 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39a8adc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39a40b63d0 0x7f39a40d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39a8adc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39a40b63d0 0x7f39a40d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45120/0 (socket says 172.21.15.107:45120) 2023-11-24T08:12:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.251+0000 7f39a8adc640 1 -- 172.21.15.107:0/3994725862 learned_addr learned my addr 172.21.15.107:0/3994725862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a92dd640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 0x7f39a40cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a8adc640 1 -- 172.21.15.107:0/3994725862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 msgr2=0x7f39a40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a8adc640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 0x7f39a40cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a8adc640 1 -- 172.21.15.107:0/3994725862 >> v1:172.21.15.120:6789/0 conn(0x7f39a40ce270 legacy=0x7f39a4160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a8adc640 1 -- 172.21.15.107:0/3994725862 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40d2260 con 0x7f39a40b63d0 2023-11-24T08:12:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.252+0000 7f39a92dd640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 0x7f39a40cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.253+0000 7f39a8adc640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40b63d0 0x7f39a40d3230 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f3998009870 tx=0x7f399802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a6125331652245 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.253+0000 7f39a3fff640 1 -- 172.21.15.107:0/3994725862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998031640 con 0x7f39a40b63d0 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.253+0000 7f39a3fff640 1 -- 172.21.15.107:0/3994725862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39980317a0 con 0x7f39a40b63d0 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.253+0000 7f39a3fff640 1 -- 172.21.15.107:0/3994725862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998031b30 con 0x7f39a40b63d0 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 -- 172.21.15.107:0/3994725862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40b63d0 msgr2=0x7f39a40d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40b63d0 0x7f39a40d3230 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f3998009870 tx=0x7f399802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 -- 172.21.15.107:0/3994725862 shutdown_connections 2023-11-24T08:12:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40b63d0 0x7f39a40d3230 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 --2- 172.21.15.107:0/3994725862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a415d490 0x7f39a40cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 -- 172.21.15.107:0/3994725862 >> 172.21.15.107:0/3994725862 conn(0x7f39a4061340 msgr2=0x7f39a4061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 -- 172.21.15.107:0/3994725862 shutdown_connections 2023-11-24T08:12:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.254+0000 7f39aa2df640 1 -- 172.21.15.107:0/3994725862 wait complete. 2023-11-24T08:12:58.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.256+0000 7f39aa2df640 1 Processor -- start 2023-11-24T08:12:58.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.256+0000 7f39aa2df640 1 -- start start 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39a415d490 0x7f39a40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a4148970 0x7f39a4148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4149330 con 0x7f39a40ce270 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a414c550 con 0x7f39a415d490 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.257+0000 7f39aa2df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40cb6f0 con 0x7f39a4148970 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45130/0 (socket says 172.21.15.107:45130) 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 -- 172.21.15.107:0/413198495 learned_addr learned my addr 172.21.15.107:0/413198495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 -- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a4148970 msgr2=0x7f39a4148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a4148970 0x7f39a4148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 -- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39a415d490 msgr2=0x7f39a40b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a8adc640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39a415d490 0x7f39a40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39a415d490 0x7f39a40b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 -- 172.21.15.107:0/413198495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980094d0 con 0x7f39a40ce270 2023-11-24T08:12:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.258+0000 7f39a92dd640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f399400ce00 tx=0x7f399400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.259+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399400f890 con 0x7f39a40ce270 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.259+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399400f9f0 con 0x7f39a40ce270 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.259+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399400fdc0 con 0x7f39a40ce270 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.259+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a40cb9d0 con 0x7f39a40ce270 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.259+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a40cbf90 con 0x7f39a40ce270 2023-11-24T08:12:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.260+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a404e840 con 0x7f39a40ce270 2023-11-24T08:12:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.261+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f399400b040 con 0x7f39a40ce270 2023-11-24T08:12:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.262+0000 7f39a1ffb640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 0x7f399006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.262+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3994088ac0 con 0x7f39a40ce270 2023-11-24T08:12:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.262+0000 7f39a8adc640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 0x7f399006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.263+0000 7f39a8adc640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 0x7f399006e1a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f399803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.267+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3994055620 con 0x7f39a40ce270 2023-11-24T08:12:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:58.489+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f39a40b2ec0 con 0x7f39a40ce270 2023-11-24T08:12:59.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.047+0000 7f39a1ffb640 1 -- 172.21.15.107:0/413198495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7f399401d020 con 0x7f39a40ce270 2023-11-24T08:12:59.049 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 14 and data pool 15 2023-11-24T08:12:59.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.055+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 msgr2=0x7f399006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.055+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 0x7f399006e1a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f399803c040 comp rx=0 tx=0).stop 2023-11-24T08:12:59.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.055+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 msgr2=0x7f39a414c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.055+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f399400ce00 tx=0x7f399400a650 comp rx=0 tx=0).stop 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 shutdown_connections 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f399006bce0 0x7f399006e1a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39a4148970 0x7f39a4148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39a415d490 0x7f39a40b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 --2- 172.21.15.107:0/413198495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ce270 0x7f39a414c010 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.056+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 >> 172.21.15.107:0/413198495 conn(0x7f39a4061340 msgr2=0x7f39a40d3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:59.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.058+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 shutdown_connections 2023-11-24T08:12:59.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.058+0000 7f39aa2df640 1 -- 172.21.15.107:0/413198495 wait complete. 2023-11-24T08:12:59.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.272+0000 7f79a9bec640 1 Processor -- start 2023-11-24T08:12:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.272+0000 7f79a9bec640 1 -- start start 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a9bec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f79a406d1c0 0x7f79a4165ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a9bec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a415b710 con 0x7f79a40c9ef0 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a9bec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a40c4c70 con 0x7f79a406a500 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a9bec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a406bd40 con 0x7f79a406d1c0 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79a40c9ef0 0x7f79a40c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52694/0 (socket says 172.21.15.107:52694) 2023-11-24T08:12:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a2ffd640 1 -- 172.21.15.107:0/4140775451 learned_addr learned my addr 172.21.15.107:0/4140775451 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.274+0000 7f79a3fff640 1 --2- 172.21.15.107:0/4140775451 >> v2:172.21.15.120:3300/0 conn(0x7f79a406d1c0 0x7f79a4165ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709903752 0 0) 0x7f79a415b710 con 0x7f79a40c9ef0 2023-11-24T08:12:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7990003520 con 0x7f79a40c9ef0 2023-11-24T08:12:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108642368 0 0) 0x7f7990003520 con 0x7f79a40c9ef0 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 >> v2:172.21.15.120:3300/0 conn(0x7f79a406d1c0 msgr2=0x7f79a4165ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 --2- 172.21.15.107:0/4140775451 >> v2:172.21.15.120:3300/0 conn(0x7f79a406d1c0 0x7f79a4165ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 >> v1:172.21.15.120:6789/0 conn(0x7f79a406a500 legacy=0x7f79a40c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a40c4f30 con 0x7f79a40c9ef0 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f798c001670 con 0x7f79a40c9ef0 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 >> v1:172.21.15.107:6789/0 conn(0x7f79a40c9ef0 legacy=0x7f79a40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f7990003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79900047c0 0x7f7990004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7990005180 0x7f799000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a406bd40 con 0x7f7990003970 2023-11-24T08:12:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799000c020 con 0x7f79900047c0 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799000c1e0 con 0x7f7990005180 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.275+0000 7f79a37fe640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f7990003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a2ffd640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79900047c0 0x7f7990004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a3fff640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7990005180 0x7f799000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a37fe640 1 -- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7990005180 msgr2=0x7f799000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a37fe640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7990005180 0x7f799000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a37fe640 1 -- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79900047c0 msgr2=0x7f7990004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a37fe640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79900047c0 0x7f7990004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a37fe640 1 -- 172.21.15.107:0/4140775451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7990003520 con 0x7f7990003970 2023-11-24T08:12:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.276+0000 7f79a3fff640 1 -- 172.21.15.107:0/4140775451 reap_dead start 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.277+0000 7f79a37fe640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f7990003d90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7994001750 tx=0x7f79900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b379bd572d30270 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.277+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79940052d0 con 0x7f7990003970 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.277+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799402e050 con 0x7f7990003970 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.277+0000 7f79a27fc640 1 -- 172.21.15.107:0/4140775451 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994003830 con 0x7f7990003970 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 -- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 msgr2=0x7f7990003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f7990003d90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7994001750 tx=0x7f79900057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 -- 172.21.15.107:0/4140775451 shutdown_connections 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 --2- 172.21.15.107:0/4140775451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f7990003d90 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 -- 172.21.15.107:0/4140775451 >> 172.21.15.107:0/4140775451 conn(0x7f79a40cc320 msgr2=0x7f79a40cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 -- 172.21.15.107:0/4140775451 shutdown_connections 2023-11-24T08:12:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.278+0000 7f79a9bec640 1 -- 172.21.15.107:0/4140775451 wait complete. 2023-11-24T08:12:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.280+0000 7f79a9bec640 1 Processor -- start 2023-11-24T08:12:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.280+0000 7f79a9bec640 1 -- start start 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79a41f0660 0x7f79a40ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 0x7f79a40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a4151c90 con 0x7f7990003970 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a40ced40 con 0x7f79a40ce370 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a9bec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a40ad090 con 0x7f79a41f0660 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45138/0 (socket says 172.21.15.107:45138) 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.281+0000 7f79a37fe640 1 -- 172.21.15.107:0/3320662078 learned_addr learned my addr 172.21.15.107:0/3320662078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a3fff640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 0x7f79a40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a2ffd640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79a41f0660 0x7f79a40ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 -- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79a41f0660 msgr2=0x7f79a40ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79a41f0660 0x7f79a40ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 -- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 msgr2=0x7f79a40ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 0x7f79a40ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 -- 172.21.15.107:0/3320662078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a40ad370 con 0x7f7990003970 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a3fff640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 0x7f79a40ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a37fe640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7994034040 tx=0x7f7994001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994007a30 con 0x7f7990003970 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79940069e0 con 0x7f7990003970 2023-11-24T08:12:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a40ad650 con 0x7f7990003970 2023-11-24T08:12:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994006d70 con 0x7f7990003970 2023-11-24T08:12:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.282+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a40adc10 con 0x7f7990003970 2023-11-24T08:12:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.283+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a4067b80 con 0x7f7990003970 2023-11-24T08:12:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.284+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f79940035f0 con 0x7f7990003970 2023-11-24T08:12:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.285+0000 7f799bfff640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 0x7f798406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.285+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79940aa0c0 con 0x7f7990003970 2023-11-24T08:12:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.286+0000 7f79a2ffd640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 0x7f798406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.286+0000 7f79a2ffd640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 0x7f798406e1d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f798c001c90 tx=0x7f798c000ff0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.290+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7994076c20 con 0x7f7990003970 2023-11-24T08:12:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.479+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79a415ebf0 con 0x7f7990003970 2023-11-24T08:12:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.481+0000 7f799bfff640 1 -- 172.21.15.107:0/3320662078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f799404b090 con 0x7f7990003970 2023-11-24T08:12:59.482 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:59.482 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:58.490534+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 msgr2=0x7f798406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 0x7f798406e1d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f798c001c90 tx=0x7f798c000ff0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 msgr2=0x7f79a41f2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7994034040 tx=0x7f7994001120 comp rx=0 tx=0).stop 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 shutdown_connections 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f798406bd10 0x7f798406e1d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a40ce370 0x7f79a40ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79a41f0660 0x7f79a40ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 --2- 172.21.15.107:0/3320662078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7990003970 0x7f79a41f2fa0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 >> 172.21.15.107:0/3320662078 conn(0x7f79a40cc320 msgr2=0x7f79a40af440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 shutdown_connections 2023-11-24T08:12:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.484+0000 7f79a9bec640 1 -- 172.21.15.107:0/3320662078 wait complete. 2023-11-24T08:12:59.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:12:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.704+0000 7f03d3741640 1 Processor -- start 2023-11-24T08:12:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.704+0000 7f03d3741640 1 -- start start 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d3741640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 0x7f03cc06a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d3741640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0d0000 con 0x7f03cc15b2b0 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d3741640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc15ec80 con 0x7f03cc06f290 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d3741640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc06aeb0 con 0x7f03cc06a500 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d14dd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03cc15b2b0 0x7f03cc161c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52702/0 (socket says 172.21.15.107:52702) 2023-11-24T08:12:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.706+0000 7f03d14dd640 1 -- 172.21.15.107:0/4100552566 learned_addr learned my addr 172.21.15.107:0/4100552566 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:12:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03d1cde640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 0x7f03cc06a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457162560 0 0) 0x7f03cc0d0000 con 0x7f03cc15b2b0 2023-11-24T08:12:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b8003520 con 0x7f03cc15b2b0 2023-11-24T08:12:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153758785 0 0) 0x7f03b8003520 con 0x7f03cc15b2b0 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 msgr2=0x7f03cc06a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 0x7f03cc06a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 >> v1:172.21.15.120:6789/0 conn(0x7f03cc06f290 legacy=0x7f03cc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc15e3f0 con 0x7f03cc15b2b0 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f03c80019f0 con 0x7f03cc15b2b0 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 >> v1:172.21.15.107:6789/0 conn(0x7f03cc15b2b0 legacy=0x7f03cc161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03b80039a0 0x7f03b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 0x7f03b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b8004d00 0x7f03b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc06aeb0 con 0x7f03b8004d00 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b800bab0 con 0x7f03b80039a0 2023-11-24T08:12:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.707+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b800bc70 con 0x7f03cc06a500 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b8004d00 0x7f03b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d14dd640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03b80039a0 0x7f03b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 -- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 msgr2=0x7f03b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03cc06a500 0x7f03b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 -- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03b80039a0 msgr2=0x7f03b8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03b80039a0 0x7f03b8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.708+0000 7f03d1cde640 1 -- 172.21.15.107:0/4100552566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b8003520 con 0x7f03b8004d00 2023-11-24T08:12:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.709+0000 7f03d1cde640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b8004d00 0x7f03b8005180 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009ac0 tx=0x7f03b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708eac128b4cf08c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.709+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c402c710 con 0x7f03b8004d00 2023-11-24T08:12:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.709+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c402c870 con 0x7f03b8004d00 2023-11-24T08:12:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.709+0000 7f03c3fff640 1 -- 172.21.15.107:0/4100552566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c402cc00 con 0x7f03b8004d00 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 -- 172.21.15.107:0/4100552566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b8004d00 msgr2=0x7f03b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 --2- 172.21.15.107:0/4100552566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b8004d00 0x7f03b8005180 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009ac0 tx=0x7f03b80057f0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d1cde640 1 -- 172.21.15.107:0/4100552566 reap_dead start 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 -- 172.21.15.107:0/4100552566 shutdown_connections 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 -- 172.21.15.107:0/4100552566 >> 172.21.15.107:0/4100552566 conn(0x7f03cc0d01b0 msgr2=0x7f03cc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 -- 172.21.15.107:0/4100552566 shutdown_connections 2023-11-24T08:12:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.710+0000 7f03d3741640 1 -- 172.21.15.107:0/4100552566 wait complete. 2023-11-24T08:12:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.712+0000 7f03d3741640 1 Processor -- start 2023-11-24T08:12:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.712+0000 7f03d3741640 1 -- start start 2023-11-24T08:12:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 0x7f03cc15f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 0x7f03cc0c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0c4fa0 con 0x7f03cc0c47b0 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0c5280 con 0x7f03cc0afc10 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d3741640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0adeb0 con 0x7f03b8004d00 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d1cde640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d1cde640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45152/0 (socket says 172.21.15.107:45152) 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.713+0000 7f03d1cde640 1 -- 172.21.15.107:0/1166694129 learned_addr learned my addr 172.21.15.107:0/1166694129 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:12:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d14dd640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 0x7f03cc15f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d0cdc640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 0x7f03cc0c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 -- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 msgr2=0x7f03cc15f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 0x7f03cc15f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 -- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 msgr2=0x7f03cc0c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 0x7f03cc0c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 -- 172.21.15.107:0/1166694129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0ae190 con 0x7f03cc0c47b0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d14dd640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 0x7f03cc15f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d0cdc640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 0x7f03cc0c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.714+0000 7f03d1cde640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f03c4037040 tx=0x7f03c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.715+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c402c710 con 0x7f03cc0c47b0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.715+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c402de90 con 0x7f03cc0c47b0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.715+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4003a30 con 0x7f03cc0c47b0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.715+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc0c5560 con 0x7f03cc0c47b0 2023-11-24T08:12:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.715+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc15fb80 con 0x7f03cc0c47b0 2023-11-24T08:12:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.716+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc15fd70 con 0x7f03cc0c47b0 2023-11-24T08:12:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.717+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f03c4003b90 con 0x7f03cc0c47b0 2023-11-24T08:12:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.718+0000 7f03c1ffb640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 0x7f03b006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:12:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.718+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03c40a92e0 con 0x7f03cc0c47b0 2023-11-24T08:12:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.718+0000 7f03d14dd640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 0x7f03b006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:12:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.719+0000 7f03d14dd640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 0x7f03b006e1d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f03c8007c40 tx=0x7f03c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:12:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.722+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03c4075b60 con 0x7f03cc0c47b0 2023-11-24T08:12:59.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.913+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03cc067b80 con 0x7f03cc0c47b0 2023-11-24T08:12:59.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.914+0000 7f03c1ffb640 1 -- 172.21.15.107:0/1166694129 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f03c4079180 con 0x7f03cc0c47b0 2023-11-24T08:12:59.916 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:12:59.916 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:12:58.490534+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:12:59.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 msgr2=0x7f03b006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 0x7f03b006e1d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f03c8007c40 tx=0x7f03c8009040 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 msgr2=0x7f03cc157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f03c4037040 tx=0x7f03c4004730 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 shutdown_connections 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b006bd10 0x7f03b006e1d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc0c47b0 0x7f03cc157fd0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc0afc10 0x7f03cc0c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 --2- 172.21.15.107:0/1166694129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03b8004d00 0x7f03cc15f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 >> 172.21.15.107:0/1166694129 conn(0x7f03cc0d01b0 msgr2=0x7f03cc0ca7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 shutdown_connections 2023-11-24T08:12:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:12:59.917+0000 7f03d3741640 1 -- 172.21.15.107:0/1166694129 wait complete. 2023-11-24T08:12:59.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:13:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.147+0000 7f2ac0806640 1 Processor -- start 2023-11-24T08:13:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.147+0000 7f2ac0806640 1 -- start start 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ac0806640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2abc15b260 0x7f2abc06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ac0806640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc15ed20 con 0x7f2abc06e580 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ac0806640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc15f8a0 con 0x7f2abc0b1250 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ac0806640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc15b640 con 0x7f2abc15b260 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ab3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2abc06e580 0x7f2abc0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52710/0 (socket says 172.21.15.107:52710) 2023-11-24T08:13:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.149+0000 7f2ab3fff640 1 -- 172.21.15.107:0/409078378 learned_addr learned my addr 172.21.15.107:0/409078378 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:00.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983658306 0 0) 0x7f2abc15ed20 con 0x7f2abc06e580 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2abad9d640 1 --2- 172.21.15.107:0/409078378 >> v2:172.21.15.120:3300/0 conn(0x7f2abc15b260 0x7f2abc06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8003520 con 0x7f2abc06e580 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4260450903 0 0) 0x7f2aa8003520 con 0x7f2abc06e580 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 >> v2:172.21.15.120:3300/0 conn(0x7f2abc15b260 msgr2=0x7f2abc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 --2- 172.21.15.107:0/409078378 >> v2:172.21.15.120:3300/0 conn(0x7f2abc15b260 0x7f2abc06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 >> v1:172.21.15.120:6789/0 conn(0x7f2abc0b1250 legacy=0x7f2abc06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0abe50 con 0x7f2abc06e580 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2aa4001670 con 0x7f2abc06e580 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 >> v1:172.21.15.107:6789/0 conn(0x7f2abc06e580 legacy=0x7f2abc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2aa80039c0 0x7f2aa8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2aa80047c0 0x7f2aa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2aa800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc15b640 con 0x7f2aa8005380 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa800c240 con 0x7f2aa80039c0 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa800c400 con 0x7f2aa80047c0 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.150+0000 7f2abb59e640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2aa800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2ab3fff640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2aa80047c0 0x7f2aa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abad9d640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2aa80039c0 0x7f2aa8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abb59e640 1 -- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2aa80047c0 msgr2=0x7f2aa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abb59e640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2aa80047c0 0x7f2aa8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abb59e640 1 -- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2aa80039c0 msgr2=0x7f2aa8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abb59e640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2aa80039c0 0x7f2aa8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abb59e640 1 -- 172.21.15.107:0/409078378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8003520 con 0x7f2aa8005380 2023-11-24T08:13:00.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.151+0000 7f2abad9d640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2aa80039c0 0x7f2aa8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:00.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.152+0000 7f2abb59e640 1 -- 172.21.15.107:0/409078378 reap_dead start 2023-11-24T08:13:00.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.152+0000 7f2abb59e640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2aa800b980 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4004480 tx=0x7f2aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476ac845495a82cc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.152+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4006b50 con 0x7f2aa8005380 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.152+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab4006cd0 con 0x7f2aa8005380 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.152+0000 7f2aba59c640 1 -- 172.21.15.107:0/409078378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4007260 con 0x7f2aa8005380 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 -- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 msgr2=0x7f2aa800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2aa800b980 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4004480 tx=0x7f2aa80057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 -- 172.21.15.107:0/409078378 shutdown_connections 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 --2- 172.21.15.107:0/409078378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2aa800b980 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 -- 172.21.15.107:0/409078378 >> 172.21.15.107:0/409078378 conn(0x7f2abc0cbb40 msgr2=0x7f2abc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 -- 172.21.15.107:0/409078378 shutdown_connections 2023-11-24T08:13:00.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.153+0000 7f2ac0806640 1 -- 172.21.15.107:0/409078378 wait complete. 2023-11-24T08:13:00.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.155+0000 7f2ac0806640 1 Processor -- start 2023-11-24T08:13:00.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.155+0000 7f2ac0806640 1 -- start start 2023-11-24T08:13:00.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc0b3480 0x7f2abc0b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2abc15d4a0 0x7f2abc1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc0b38c0 con 0x7f2aa8005380 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc0b23e0 con 0x7f2abc0b3480 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2ac0806640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc0b26c0 con 0x7f2abc15d4a0 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45168/0 (socket says 172.21.15.107:45168) 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 -- 172.21.15.107:0/2218859746 learned_addr learned my addr 172.21.15.107:0/2218859746 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 -- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2abc15d4a0 msgr2=0x7f2abc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2abc15d4a0 0x7f2abc1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 -- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc0b3480 msgr2=0x7f2abc0b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc0b3480 0x7f2abc0b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 -- 172.21.15.107:0/2218859746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0b29a0 con 0x7f2aa8005380 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.156+0000 7f2abad9d640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00e7e0 tx=0x7f2aac00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.157+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00c6d0 con 0x7f2aa8005380 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.157+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac00c830 con 0x7f2aa8005380 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.157+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00cba0 con 0x7f2aa8005380 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.157+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc0b2c80 con 0x7f2aa8005380 2023-11-24T08:13:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.157+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc15d920 con 0x7f2aa8005380 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.158+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc15cb00 con 0x7f2aa8005380 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.159+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f2aac00cd00 con 0x7f2aa8005380 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.160+0000 7f2ab37fe640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 0x7f2a9c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.160+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aac088200 con 0x7f2aa8005380 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.160+0000 7f2ab3fff640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 0x7f2a9c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.161+0000 7f2ab3fff640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 0x7f2a9c06e1a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4001eb0 tx=0x7f2aa4001020 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:00.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.164+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2abc15cb00 con 0x7f2aa8005380 2023-11-24T08:13:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:00.394+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2abc15cb00 con 0x7f2aa8005380 2023-11-24T08:13:01.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.064+0000 7f2ab37fe640 1 -- 172.21.15.107:0/2218859746 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2abc15cb00 con 0x7f2aa8005380 2023-11-24T08:13:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 msgr2=0x7f2a9c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 0x7f2a9c06e1a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4001eb0 tx=0x7f2aa4001020 comp rx=0 tx=0).stop 2023-11-24T08:13:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 msgr2=0x7f2abc0b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00e7e0 tx=0x7f2aac00eca0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 shutdown_connections 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2a9c06bce0 0x7f2a9c06e1a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2abc15d4a0 0x7f2abc1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc0b3480 0x7f2abc0b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 --2- 172.21.15.107:0/2218859746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aa8005380 0x7f2abc0b49e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 >> 172.21.15.107:0/2218859746 conn(0x7f2abc0cbb40 msgr2=0x7f2abc0ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 shutdown_connections 2023-11-24T08:13:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.068+0000 7f2ac0806640 1 -- 172.21.15.107:0/2218859746 wait complete. 2023-11-24T08:13:01.091 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:13:01.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:13:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.302+0000 7fb870c51640 1 Processor -- start 2023-11-24T08:13:01.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.303+0000 7fb870c51640 1 -- start start 2023-11-24T08:13:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.304+0000 7fb870c51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb86c146eb0 0x7fb86c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.304+0000 7fb870c51640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb86c06e160 0x7fb86c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.304+0000 7fb870c51640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c161a10 con 0x7fb86c0cd110 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.304+0000 7fb870c51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c0cd580 con 0x7fb86c146eb0 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.304+0000 7fb870c51640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c161cf0 con 0x7fb86c06e160 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb86ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb86c0cd110 0x7fb86c06c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52724/0 (socket says 172.21.15.107:52724) 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb86ad9d640 1 -- 172.21.15.107:0/227547686 learned_addr learned my addr 172.21.15.107:0/227547686 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb86a59c640 1 --2- 172.21.15.107:0/227547686 >> v2:172.21.15.107:3300/0 conn(0x7fb86c146eb0 0x7fb86c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb863fff640 1 --2- 172.21.15.107:0/227547686 >> v2:172.21.15.120:3300/0 conn(0x7fb86c06e160 0x7fb86c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb869d9b640 1 -- 172.21.15.107:0/227547686 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268945594 0 0) 0x7fb86c161a10 con 0x7fb86c0cd110 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.305+0000 7fb869d9b640 1 -- 172.21.15.107:0/227547686 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858003520 con 0x7fb86c0cd110 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.306+0000 7fb86a59c640 1 -- 172.21.15.107:0/227547686 >> v2:172.21.15.120:3300/0 conn(0x7fb86c06e160 msgr2=0x7fb86c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.306+0000 7fb86a59c640 1 --2- 172.21.15.107:0/227547686 >> v2:172.21.15.120:3300/0 conn(0x7fb86c06e160 0x7fb86c0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.306+0000 7fb86a59c640 1 -- 172.21.15.107:0/227547686 >> v1:172.21.15.107:6789/0 conn(0x7fb86c0cd110 legacy=0x7fb86c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.306+0000 7fb86a59c640 1 -- 172.21.15.107:0/227547686 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c161fd0 con 0x7fb86c146eb0 2023-11-24T08:13:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.307+0000 7fb86a59c640 1 --2- 172.21.15.107:0/227547686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c146eb0 0x7fb86c161150 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fb86c06eab0 tx=0x7fb85c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120a665b09627b76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.307+0000 7fb869d9b640 1 -- 172.21.15.107:0/227547686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c031830 con 0x7fb86c146eb0 2023-11-24T08:13:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.307+0000 7fb869d9b640 1 -- 172.21.15.107:0/227547686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c031990 con 0x7fb86c146eb0 2023-11-24T08:13:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.307+0000 7fb869d9b640 1 -- 172.21.15.107:0/227547686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c031d00 con 0x7fb86c146eb0 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 -- 172.21.15.107:0/227547686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c146eb0 msgr2=0x7fb86c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 --2- 172.21.15.107:0/227547686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c146eb0 0x7fb86c161150 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fb86c06eab0 tx=0x7fb85c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 -- 172.21.15.107:0/227547686 shutdown_connections 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 --2- 172.21.15.107:0/227547686 >> v2:172.21.15.120:3300/0 conn(0x7fb86c06e160 0x7fb86c0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 --2- 172.21.15.107:0/227547686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c146eb0 0x7fb86c161150 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 -- 172.21.15.107:0/227547686 >> 172.21.15.107:0/227547686 conn(0x7fb86c0b0c50 msgr2=0x7fb86c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 -- 172.21.15.107:0/227547686 shutdown_connections 2023-11-24T08:13:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.308+0000 7fb870c51640 1 -- 172.21.15.107:0/227547686 wait complete. 2023-11-24T08:13:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.310+0000 7fb870c51640 1 Processor -- start 2023-11-24T08:13:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.310+0000 7fb870c51640 1 -- start start 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 0x7fb86c143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c146eb0 0x7fb86c1435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c1f65d0 con 0x7fb86c06e160 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c1f6870 con 0x7fb86c0cd110 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb870c51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c1f6b50 con 0x7fb86c146eb0 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb86a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb86a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45180/0 (socket says 172.21.15.107:45180) 2023-11-24T08:13:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.311+0000 7fb86a59c640 1 -- 172.21.15.107:0/131280656 learned_addr learned my addr 172.21.15.107:0/131280656 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86ad9d640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c146eb0 0x7fb86c1435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb863fff640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 0x7fb86c143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 -- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c146eb0 msgr2=0x7fb86c1435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c146eb0 0x7fb86c1435d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 -- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 msgr2=0x7fb86c143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 0x7fb86c143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 -- 172.21.15.107:0/131280656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c009400 con 0x7fb86c06e160 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb863fff640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 0x7fb86c143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb86a59c640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fb85c02b910 tx=0x7fb85c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c03a830 con 0x7fb86c06e160 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c03a990 con 0x7fb86c06e160 2023-11-24T08:13:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c03ad20 con 0x7fb86c06e160 2023-11-24T08:13:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.312+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c1f6e30 con 0x7fb86c06e160 2023-11-24T08:13:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.313+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c0ad730 con 0x7fb86c06e160 2023-11-24T08:13:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.314+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c1f7350 con 0x7fb86c06e160 2023-11-24T08:13:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.314+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fb85c04d440 con 0x7fb86c06e160 2023-11-24T08:13:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.315+0000 7fb862ffd640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 0x7fb84c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.316+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb85c0aa0e0 con 0x7fb86c06e160 2023-11-24T08:13:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.316+0000 7fb863fff640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 0x7fb84c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.316+0000 7fb863fff640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 0x7fb84c06e1a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb86c13e1c0 tx=0x7fb854008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.321+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb86c1f7350 con 0x7fb86c06e160 2023-11-24T08:13:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:01.554+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb86c1f7350 con 0x7fb86c06e160 2023-11-24T08:13:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.077+0000 7fb862ffd640 1 -- 172.21.15.107:0/131280656 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb86c1f7350 con 0x7fb86c06e160 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.081+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 msgr2=0x7fb84c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.081+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 0x7fb84c06e1a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb86c13e1c0 tx=0x7fb854008040 comp rx=0 tx=0).stop 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 msgr2=0x7fb86c0ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fb85c02b910 tx=0x7fb85c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 shutdown_connections 2023-11-24T08:13:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb84c06bce0 0x7fb84c06e1a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c146eb0 0x7fb86c1435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0cd110 0x7fb86c143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 --2- 172.21.15.107:0/131280656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c06e160 0x7fb86c0ad1f0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 >> 172.21.15.107:0/131280656 conn(0x7fb86c0b0c50 msgr2=0x7fb86c1456b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 shutdown_connections 2023-11-24T08:13:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.082+0000 7fb870c51640 1 -- 172.21.15.107:0/131280656 wait complete. 2023-11-24T08:13:02.104 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:13:02.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.314+0000 7f86ac683640 1 Processor -- start 2023-11-24T08:13:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.314+0000 7f86ac683640 1 -- start start 2023-11-24T08:13:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.315+0000 7f86ac683640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a415df80 0x7f86a40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86ac683640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86ac683640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a415e3f0 con 0x7f86a40b2250 2023-11-24T08:13:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86ac683640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a415a990 con 0x7f86a415df80 2023-11-24T08:13:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86ac683640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a415ab80 con 0x7f86a40cd470 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a415df80 0x7f86a40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a415df80 0x7f86a40ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45186/0 (socket says 172.21.15.107:45186) 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 -- 172.21.15.107:0/3407867057 learned_addr learned my addr 172.21.15.107:0/3407867057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86aac20640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 -- 172.21.15.107:0/3407867057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 msgr2=0x7f86a415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 -- 172.21.15.107:0/3407867057 >> v1:172.21.15.120:6789/0 conn(0x7f86a40b2250 legacy=0x7f86a40b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86a9c1e640 1 -- 172.21.15.107:0/3407867057 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40b2d40 con 0x7f86a415df80 2023-11-24T08:13:02.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.316+0000 7f86aac20640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.318+0000 7f86a9c1e640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a415df80 0x7f86a40ccf30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f8694009870 tx=0x7f869402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd18a417451a4473 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.318+0000 7f86a941d640 1 -- 172.21.15.107:0/3407867057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694031640 con 0x7f86a415df80 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.318+0000 7f86a941d640 1 -- 172.21.15.107:0/3407867057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86940317a0 con 0x7f86a415df80 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.318+0000 7f86a941d640 1 -- 172.21.15.107:0/3407867057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694031b30 con 0x7f86a415df80 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 -- 172.21.15.107:0/3407867057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a415df80 msgr2=0x7f86a40ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a415df80 0x7f86a40ccf30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f8694009870 tx=0x7f869402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 -- 172.21.15.107:0/3407867057 shutdown_connections 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 --2- 172.21.15.107:0/3407867057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a415df80 0x7f86a40ccf30 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 -- 172.21.15.107:0/3407867057 >> 172.21.15.107:0/3407867057 conn(0x7f86a406a920 msgr2=0x7f86a406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:02.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 -- 172.21.15.107:0/3407867057 shutdown_connections 2023-11-24T08:13:02.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.319+0000 7f86ac683640 1 -- 172.21.15.107:0/3407867057 wait complete. 2023-11-24T08:13:02.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.321+0000 7f86ac683640 1 Processor -- start 2023-11-24T08:13:02.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.321+0000 7f86ac683640 1 -- start start 2023-11-24T08:13:02.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a40b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a415df80 0x7f86a40afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4140de0 con 0x7f86a40b2250 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40b17b0 con 0x7f86a415df80 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86ac683640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41407b0 con 0x7f86a40cd470 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45188/0 (socket says 172.21.15.107:45188) 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 -- 172.21.15.107:0/624200749 learned_addr learned my addr 172.21.15.107:0/624200749 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 -- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 msgr2=0x7f86a40b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a40b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 -- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a415df80 msgr2=0x7f86a40afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a415df80 0x7f86a40afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.322+0000 7f86aa41f640 1 -- 172.21.15.107:0/624200749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86940094d0 con 0x7f86a40b2250 2023-11-24T08:13:02.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f86aa41f640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f86a000c430 tx=0x7f86a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0011790 con 0x7f86a40b2250 2023-11-24T08:13:02.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a00118f0 con 0x7f86a40b2250 2023-11-24T08:13:02.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0011ce0 con 0x7f86a40b2250 2023-11-24T08:13:02.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a4154150 con 0x7f86a40b2250 2023-11-24T08:13:02.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.323+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a41581c0 con 0x7f86a40b2250 2023-11-24T08:13:02.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.324+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a4140f20 con 0x7f86a40b2250 2023-11-24T08:13:02.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.325+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f86a001c460 con 0x7f86a40b2250 2023-11-24T08:13:02.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.326+0000 7f869affd640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 0x7f868806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.326+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86a0088390 con 0x7f86a40b2250 2023-11-24T08:13:02.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.326+0000 7f86a9c1e640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 0x7f868806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.327+0000 7f86a9c1e640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 0x7f868806e1a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8694036000 tx=0x7f8694030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.331+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86a4140f20 con 0x7f86a40b2250 2023-11-24T08:13:02.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.557+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a4158350 con 0x7f86a40b2250 2023-11-24T08:13:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.558+0000 7f869affd640 1 -- 172.21.15.107:0/624200749 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f86a0054bb0 con 0x7f86a40b2250 2023-11-24T08:13:02.560 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:02.560 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:02.561 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 59 2023-11-24T08:13:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 msgr2=0x7f868806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 0x7f868806e1a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8694036000 tx=0x7f8694030000 comp rx=0 tx=0).stop 2023-11-24T08:13:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 msgr2=0x7f86a4099140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f86a000c430 tx=0x7f86a000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 shutdown_connections 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f868806bce0 0x7f868806e1a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a415df80 0x7f86a40afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40cd470 0x7f86a40b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 --2- 172.21.15.107:0/624200749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40b2250 0x7f86a4099140 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 >> 172.21.15.107:0/624200749 conn(0x7f86a406a920 msgr2=0x7f86a4159540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 shutdown_connections 2023-11-24T08:13:02.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.561+0000 7f86ac683640 1 -- 172.21.15.107:0/624200749 wait complete. 2023-11-24T08:13:02.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.781+0000 7fe21c8ad640 1 Processor -- start 2023-11-24T08:13:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.782+0000 7fe21c8ad640 1 -- start start 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe21c8ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe21806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe21c8ad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe2180cdbf0 0x7fe2180ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe21c8ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21806e060 con 0x7fe21813d120 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe21c8ad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21813f9e0 con 0x7fe2180cdbf0 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe21c8ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21813fcc0 con 0x7fe21806d6c0 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe2177fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe21813d120 0x7fe21815af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52740/0 (socket says 172.21.15.107:52740) 2023-11-24T08:13:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.783+0000 7fe2177fe640 1 -- 172.21.15.107:0/3468719372 learned_addr learned my addr 172.21.15.107:0/3468719372 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860678114 0 0) 0x7fe21806e060 con 0x7fe21813d120 2023-11-24T08:13:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe204005730 con 0x7fe21813d120 2023-11-24T08:13:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe21806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe217fff640 1 --2- 172.21.15.107:0/3468719372 >> v2:172.21.15.120:3300/0 conn(0x7fe2180cdbf0 0x7fe2180ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168184092 0 0) 0x7fe204005730 con 0x7fe21813d120 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 msgr2=0x7fe21806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe21806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 >> v2:172.21.15.120:3300/0 conn(0x7fe2180cdbf0 msgr2=0x7fe2180ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 --2- 172.21.15.107:0/3468719372 >> v2:172.21.15.120:3300/0 conn(0x7fe2180cdbf0 0x7fe2180ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21813ffa0 con 0x7fe21813d120 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe208001670 con 0x7fe21813d120 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 >> v1:172.21.15.107:6789/0 conn(0x7fe21813d120 legacy=0x7fe21815af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe204005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2040061b0 0x7fe2040069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe204006f10 0x7fe204007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21813f9e0 con 0x7fe204006f10 2023-11-24T08:13:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21813fcc0 con 0x7fe2040061b0 2023-11-24T08:13:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20400dd50 con 0x7fe21806d6c0 2023-11-24T08:13:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.784+0000 7fe217fff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe204006f10 0x7fe204007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe204005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2040061b0 0x7fe2040069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe217fff640 1 -- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 msgr2=0x7fe204005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe217fff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe204005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe217fff640 1 -- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2040061b0 msgr2=0x7fe2040069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe217fff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2040061b0 0x7fe2040069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe217fff640 1 -- 172.21.15.107:0/3468719372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204005730 con 0x7fe204006f10 2023-11-24T08:13:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2040061b0 0x7fe2040069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:02.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.785+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21806d6c0 0x7fe204005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.786+0000 7fe217fff640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe204006f10 0x7fe204007390 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fe210007360 tx=0x7fe204007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef03833fe36a242 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.786+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21002ccf0 con 0x7fe204006f10 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.786+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21002ce50 con 0x7fe204006f10 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.786+0000 7fe216ffd640 1 -- 172.21.15.107:0/3468719372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210034660 con 0x7fe204006f10 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3468719372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe204006f10 msgr2=0x7fe204007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3468719372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe204006f10 0x7fe204007390 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fe210007360 tx=0x7fe204007a00 comp rx=0 tx=0).stop 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe217fff640 1 -- 172.21.15.107:0/3468719372 reap_dead start 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3468719372 shutdown_connections 2023-11-24T08:13:02.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3468719372 >> 172.21.15.107:0/3468719372 conn(0x7fe2180cb830 msgr2=0x7fe2180cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3468719372 shutdown_connections 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.787+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3468719372 wait complete. 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.789+0000 7fe21c8ad640 1 Processor -- start 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.789+0000 7fe21c8ad640 1 -- start start 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 0x7fe21815c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21813c230 0x7fe21812b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21815aeb0 con 0x7fe21815c6a0 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21815bcd0 con 0x7fe204006f10 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe21c8ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21815b0c0 con 0x7fe21813c230 2023-11-24T08:13:02.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45206/0 (socket says 172.21.15.107:45206) 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe20ffff640 1 -- 172.21.15.107:0/3472372515 learned_addr learned my addr 172.21.15.107:0/3472372515 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.790+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 0x7fe21815c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe217fff640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21813c230 0x7fe21812b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 -- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21813c230 msgr2=0x7fe21812b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21813c230 0x7fe21812b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 -- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 msgr2=0x7fe21815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 0x7fe21815c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 -- 172.21.15.107:0/3472372515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21815a9d0 con 0x7fe21815c6a0 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 0x7fe21815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe20ffff640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e7e0 tx=0x7fe20000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20000c6d0 con 0x7fe21815c6a0 2023-11-24T08:13:02.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20000c830 con 0x7fe21815c6a0 2023-11-24T08:13:02.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20000cba0 con 0x7fe21815c6a0 2023-11-24T08:13:02.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21815b3a0 con 0x7fe21815c6a0 2023-11-24T08:13:02.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.791+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21815b960 con 0x7fe21815c6a0 2023-11-24T08:13:02.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.792+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21804e840 con 0x7fe21815c6a0 2023-11-24T08:13:02.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.793+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fe20000cd00 con 0x7fe21815c6a0 2023-11-24T08:13:02.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.795+0000 7fe214ff9640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 0x7fe1f806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:02.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.795+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe200088230 con 0x7fe21815c6a0 2023-11-24T08:13:02.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.795+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 0x7fe1f806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:02.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.796+0000 7fe2177fe640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 0x7fe1f806e1a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe208007af0 tx=0x7fe208009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:02.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.799+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe200054a50 con 0x7fe21815c6a0 2023-11-24T08:13:02.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.990+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe21815be10 con 0x7fe21815c6a0 2023-11-24T08:13:02.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.992+0000 7fe214ff9640 1 -- 172.21.15.107:0/3472372515 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7fe200058070 con 0x7fe21815c6a0 2023-11-24T08:13:02.995 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:02.995 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:02.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 msgr2=0x7fe1f806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 0x7fe1f806e1a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe208007af0 tx=0x7fe208009040 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 msgr2=0x7fe21812b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e7e0 tx=0x7fe20000eca0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 shutdown_connections 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe1f806bce0 0x7fe1f806e1a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe21813c230 0x7fe21812b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21815c6a0 0x7fe21812b150 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 --2- 172.21.15.107:0/3472372515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe204006f10 0x7fe21815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.995+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 >> 172.21.15.107:0/3472372515 conn(0x7fe2180cb830 msgr2=0x7fe218158230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.996+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 shutdown_connections 2023-11-24T08:13:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:02.996+0000 7fe21c8ad640 1 -- 172.21.15.107:0/3472372515 wait complete. 2023-11-24T08:13:03.016 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:13:03.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:13:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.218+0000 7f22a6eee640 1 Processor -- start 2023-11-24T08:13:03.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.218+0000 7f22a6eee640 1 -- start start 2023-11-24T08:13:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.219+0000 7f22a6eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f22a0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a6eee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f22a00cf450 0x7f22a00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a6eee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a00ce0b0 con 0x7f22a015dc80 2023-11-24T08:13:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a6eee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a00cfe10 con 0x7f22a00cf450 2023-11-24T08:13:03.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a6eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0161960 con 0x7f22a0157c70 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a548b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f22a015dc80 0x7f22a015e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52750/0 (socket says 172.21.15.107:52750) 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a548b640 1 -- 172.21.15.107:0/1186257794 learned_addr learned my addr 172.21.15.107:0/1186257794 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f229ffff640 1 --2- 172.21.15.107:0/1186257794 >> v2:172.21.15.120:3300/0 conn(0x7f22a00cf450 0x7f22a00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f22a4c8a640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f22a0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.220+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776952876 0 0) 0x7f22a00ce0b0 con 0x7f22a015dc80 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290003520 con 0x7f22a015dc80 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085960250 0 0) 0x7f2290003520 con 0x7f22a015dc80 2023-11-24T08:13:03.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 msgr2=0x7f22a0161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f22a0161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 >> v2:172.21.15.120:3300/0 conn(0x7f22a00cf450 msgr2=0x7f22a00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 --2- 172.21.15.107:0/1186257794 >> v2:172.21.15.120:3300/0 conn(0x7f22a00cf450 0x7f22a00cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a015e8d0 con 0x7f22a015dc80 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f22980019f0 con 0x7f22a015dc80 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 >> v1:172.21.15.107:6789/0 conn(0x7f22a015dc80 legacy=0x7f22a015e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f2290003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2290003f40 0x7f22900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2290004d00 0x7f2290005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00cfe10 con 0x7f2290003f40 2023-11-24T08:13:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0161960 con 0x7f2290004d00 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.221+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229000ba30 con 0x7f22a0157c70 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2290003f40 0x7f22900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f22a4c8a640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f2290003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 -- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 msgr2=0x7f2290003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f22a548b640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2290004d00 0x7f2290005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22a0157c70 0x7f2290003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 -- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2290004d00 msgr2=0x7f2290005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2290004d00 0x7f2290005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.222+0000 7f229ffff640 1 -- 172.21.15.107:0/1186257794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290003520 con 0x7f2290003f40 2023-11-24T08:13:03.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.223+0000 7f229ffff640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2290003f40 0x7f22900047c0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f228c009ac0 tx=0x7f22900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d8a1a5f5570512 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.223+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c02c710 con 0x7f2290003f40 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.223+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c02c870 con 0x7f2290003f40 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.223+0000 7f229f7fe640 1 -- 172.21.15.107:0/1186257794 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c02cc00 con 0x7f2290003f40 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 -- 172.21.15.107:0/1186257794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2290003f40 msgr2=0x7f22900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 --2- 172.21.15.107:0/1186257794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2290003f40 0x7f22900047c0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f228c009ac0 tx=0x7f22900057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a548b640 1 -- 172.21.15.107:0/1186257794 reap_dead start 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 -- 172.21.15.107:0/1186257794 shutdown_connections 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 -- 172.21.15.107:0/1186257794 >> 172.21.15.107:0/1186257794 conn(0x7f22a006a920 msgr2=0x7f22a006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 -- 172.21.15.107:0/1186257794 shutdown_connections 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.224+0000 7f22a6eee640 1 -- 172.21.15.107:0/1186257794 wait complete. 2023-11-24T08:13:03.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.226+0000 7f22a6eee640 1 Processor -- start 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.226+0000 7f22a6eee640 1 -- start start 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2290003f40 0x7f22a009c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a014a3f0 0x7f22a00c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00ca750 con 0x7f22a00ca140 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00cc020 con 0x7f22a014a3f0 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.227+0000 7f22a6eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a009b440 con 0x7f2290003f40 2023-11-24T08:13:03.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45220/0 (socket says 172.21.15.107:45220) 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 -- 172.21.15.107:0/2427791931 learned_addr learned my addr 172.21.15.107:0/2427791931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a4c8a640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2290003f40 0x7f22a009c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f229ffff640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a014a3f0 0x7f22a00c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 -- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2290003f40 msgr2=0x7f22a009c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2290003f40 0x7f22a009c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 -- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a014a3f0 msgr2=0x7f22a00c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a014a3f0 0x7f22a00c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 -- 172.21.15.107:0/2427791931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a009b720 con 0x7f22a00ca140 2023-11-24T08:13:03.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.228+0000 7f22a548b640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f22a006b4d0 tx=0x7f229800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.229+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298004d60 con 0x7f22a00ca140 2023-11-24T08:13:03.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.229+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229800bb80 con 0x7f22a00ca140 2023-11-24T08:13:03.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.229+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a009ba00 con 0x7f22a00ca140 2023-11-24T08:13:03.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.229+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298007110 con 0x7f22a00ca140 2023-11-24T08:13:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.229+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a014a850 con 0x7f22a00ca140 2023-11-24T08:13:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.230+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a0067470 con 0x7f22a00ca140 2023-11-24T08:13:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.231+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f2298007270 con 0x7f22a00ca140 2023-11-24T08:13:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.233+0000 7f229d7fa640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 0x7f228406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.233+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2298088780 con 0x7f22a00ca140 2023-11-24T08:13:03.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.233+0000 7f22a4c8a640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 0x7f228406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.233+0000 7f22a4c8a640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 0x7f228406e1d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2294009930 tx=0x7f2294008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.237+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2298054fa0 con 0x7f22a00ca140 2023-11-24T08:13:03.400 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:13:03.398+0000 7f282fd3a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:13:03.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.422+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22a009b5b0 con 0x7f228406bd10 2023-11-24T08:13:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.434+0000 7f229d7fa640 1 -- 172.21.15.107:0/2427791931 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282336 (secure 0 0 0) 0x7f22a009b5b0 con 0x7f228406bd10 2023-11-24T08:13:03.441 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 msgr2=0x7f228406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 0x7f228406e1d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2294009930 tx=0x7f2294008040 comp rx=0 tx=0).stop 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 msgr2=0x7f22a0070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f22a006b4d0 tx=0x7f229800b4a0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 shutdown_connections 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f228406bd10 0x7f228406e1d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22a00ca140 0x7f22a0070650 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a014a3f0 0x7f22a00c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 --2- 172.21.15.107:0/2427791931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2290003f40 0x7f22a009c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 >> 172.21.15.107:0/2427791931 conn(0x7f22a006a920 msgr2=0x7f22a0071470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 shutdown_connections 2023-11-24T08:13:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.440+0000 7f22a6eee640 1 -- 172.21.15.107:0/2427791931 wait complete. 2023-11-24T08:13:03.459 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-11-24T08:13:01.585079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":273376,"kb_used_data":11448,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754701344,"statfs":{"total":773094113280,"available":772814176256,"internally_reserved":0,"allocated":11722752,"data_stored":5748556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":268095038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.572829"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.041312+0000","last_change":"2023-11-24T08:12:58.041312+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.041312+0000","last_undegraded":"2023-11-24T08:12:58.041312+0000","last_fullsized":"2023-11-24T08:12:58.041312+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.051599+0000","last_change":"2023-11-24T08:12:58.051599+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.051599+0000","last_undegraded":"2023-11-24T08:12:58.051599+0000","last_fullsized":"2023-11-24T08:12:58.051599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.043253+0000","last_change":"2023-11-24T08:12:58.043253+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.043253+0000","last_undegraded":"2023-11-24T08:12:58.043253+0000","last_fullsized":"2023-11-24T08:12:58.043253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.040210+0000","last_change":"2023-11-24T08:12:58.040210+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.040210+0000","last_undegraded":"2023-11-24T08:12:58.040210+0000","last_fullsized":"2023-11-24T08:12:58.040210+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.055719+0000","last_change":"2023-11-24T08:12:58.055719+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.055719+0000","last_undegraded":"2023-11-24T08:12:58.055719+0000","last_fullsized":"2023-11-24T08:12:58.055719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.045006+0000","last_change":"2023-11-24T08:12:58.045006+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.045006+0000","last_undegraded":"2023-11-24T08:12:58.045006+0000","last_fullsized":"2023-11-24T08:12:58.045006+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.047850+0000","last_change":"2023-11-24T08:12:58.047850+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.047850+0000","last_undegraded":"2023-11-24T08:12:58.047850+0000","last_fullsized":"2023-11-24T08:12:58.047850+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050889+0000","last_change":"2023-11-24T08:12:58.050889+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050889+0000","last_undegraded":"2023-11-24T08:12:58.050889+0000","last_fullsized":"2023-11-24T08:12:58.050889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.041751+0000","last_change":"2023-11-24T08:12:58.041751+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.041751+0000","last_undegraded":"2023-11-24T08:12:58.041751+0000","last_fullsized":"2023-11-24T08:12:58.041751+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.056090+0000","last_change":"2023-11-24T08:12:58.056090+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.056090+0000","last_undegraded":"2023-11-24T08:12:58.056090+0000","last_fullsized":"2023-11-24T08:12:58.056090+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050750+0000","last_change":"2023-11-24T08:12:58.050750+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050750+0000","last_undegraded":"2023-11-24T08:12:58.050750+0000","last_fullsized":"2023-11-24T08:12:58.050750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.042321+0000","last_change":"2023-11-24T08:12:58.042321+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.042321+0000","last_undegraded":"2023-11-24T08:12:58.042321+0000","last_fullsized":"2023-11-24T08:12:58.042321+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.039728+0000","last_change":"2023-11-24T08:12:58.039728+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.039728+0000","last_undegraded":"2023-11-24T08:12:58.039728+0000","last_fullsized":"2023-11-24T08:12:58.039728+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050969+0000","last_change":"2023-11-24T08:12:58.050969+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050969+0000","last_undegraded":"2023-11-24T08:12:58.050969+0000","last_fullsized":"2023-11-24T08:12:58.050969+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.041232+0000","last_change":"2023-11-24T08:12:58.041232+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.041232+0000","last_undegraded":"2023-11-24T08:12:58.041232+0000","last_fullsized":"2023-11-24T08:12:58.041232+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.044144+0000","last_change":"2023-11-24T08:12:58.044144+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.044144+0000","last_undegraded":"2023-11-24T08:12:58.044144+0000","last_fullsized":"2023-11-24T08:12:58.044144+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050809+0000","last_change":"2023-11-24T08:12:58.050809+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050809+0000","last_undegraded":"2023-11-24T08:12:58.050809+0000","last_fullsized":"2023-11-24T08:12:58.050809+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.039082+0000","last_change":"2023-11-24T08:12:58.039082+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.039082+0000","last_undegraded":"2023-11-24T08:12:58.039082+0000","last_fullsized":"2023-11-24T08:12:58.039082+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.048377+0000","last_change":"2023-11-24T08:12:58.048377+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.048377+0000","last_undegraded":"2023-11-24T08:12:58.048377+0000","last_fullsized":"2023-11-24T08:12:58.048377+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.041175+0000","last_change":"2023-11-24T08:12:58.041175+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.041175+0000","last_undegraded":"2023-11-24T08:12:58.041175+0000","last_fullsized":"2023-11-24T08:12:58.041175+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.047729+0000","last_change":"2023-11-24T08:12:58.047729+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.047729+0000","last_undegraded":"2023-11-24T08:12:58.047729+0000","last_fullsized":"2023-11-24T08:12:58.047729+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.051148+0000","last_change":"2023-11-24T08:12:58.051148+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.051148+0000","last_undegraded":"2023-11-24T08:12:58.051148+0000","last_fullsized":"2023-11-24T08:12:58.051148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050226+0000","last_change":"2023-11-24T08:12:58.050226+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050226+0000","last_undegraded":"2023-11-24T08:12:58.050226+0000","last_fullsized":"2023-11-24T08:12:58.050226+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.041623+0000","last_change":"2023-11-24T08:12:58.041623+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.041623+0000","last_undegraded":"2023-11-24T08:12:58.041623+0000","last_fullsized":"2023-11-24T08:12:58.041623+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.059183+0000","last_change":"2023-11-24T08:12:58.059183+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.059183+0000","last_undegraded":"2023-11-24T08:12:58.059183+0000","last_fullsized":"2023-11-24T08:12:58.059183+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.049904+0000","last_change":"2023-11-24T08:12:58.049904+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.049904+0000","last_undegraded":"2023-11-24T08:12:58.049904+0000","last_fullsized":"2023-11-24T08:12:58.049904+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.049802+0000","last_change":"2023-11-24T08:12:58.049802+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.049802+0000","last_undegraded":"2023-11-24T08:12:58.049802+0000","last_fullsized":"2023-11-24T08:12:58.049802+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.058270+0000","last_change":"2023-11-24T08:12:58.058270+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.058270+0000","last_undegraded":"2023-11-24T08:12:58.058270+0000","last_fullsized":"2023-11-24T08:12:58.058270+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.049232+0000","last_change":"2023-11-24T08:12:58.049232+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.049232+0000","last_undegraded":"2023-11-24T08:12:58.049232+0000","last_fullsized":"2023-11-24T08:12:58.049232+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.050594+0000","last_change":"2023-11-24T08:12:58.050594+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.050594+0000","last_undegraded":"2023-11-24T08:12:58.050594+0000","last_fullsized":"2023-11-24T08:12:58.050594+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.039894+0000","last_change":"2023-11-24T08:12:58.039894+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.039894+0000","last_undegraded":"2023-11-24T08:12:58.039894+0000","last_fullsized":"2023-11-24T08:12:58.039894+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.038979+0000","last_change":"2023-11-24T08:12:58.038979+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.038979+0000","last_undegraded":"2023-11-24T08:12:58.038979+0000","last_fullsized":"2023-11-24T08:12:58.038979+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.044611+0000","last_change":"2023-11-24T08:12:58.044611+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.044611+0000","last_undegraded":"2023-11-24T08:12:58.044611+0000","last_fullsized":"2023-11-24T08:12:58.044611+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.069232+0000","last_change":"2023-11-24T08:12:58.064302+0000","last_active":"2023-11-24T08:12:58.069232+0000","last_peered":"2023-11-24T08:12:58.069232+0000","last_clean":"2023-11-24T08:12:58.069232+0000","last_became_active":"2023-11-24T08:12:58.063387+0000","last_became_peered":"2023-11-24T08:12:58.063387+0000","last_unstale":"2023-11-24T08:12:58.069232+0000","last_undegraded":"2023-11-24T08:12:58.069232+0000","last_fullsized":"2023-11-24T08:12:58.069232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:19:01.598008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.037998+0000","last_change":"2023-11-24T08:12:58.037998+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.037998+0000","last_undegraded":"2023-11-24T08:12:58.037998+0000","last_fullsized":"2023-11-24T08:12:58.037998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.065235+0000","last_change":"2023-11-24T08:12:58.063990+0000","last_active":"2023-11-24T08:12:58.065235+0000","last_peered":"2023-11-24T08:12:58.065235+0000","last_clean":"2023-11-24T08:12:58.065235+0000","last_became_active":"2023-11-24T08:12:58.063698+0000","last_became_peered":"2023-11-24T08:12:58.063698+0000","last_unstale":"2023-11-24T08:12:58.065235+0000","last_undegraded":"2023-11-24T08:12:58.065235+0000","last_fullsized":"2023-11-24T08:12:58.065235+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:15.184470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.049553+0000","last_change":"2023-11-24T08:12:58.049553+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.049553+0000","last_undegraded":"2023-11-24T08:12:58.049553+0000","last_fullsized":"2023-11-24T08:12:58.049553+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.064466+0000","last_change":"2023-11-24T08:12:58.062652+0000","last_active":"2023-11-24T08:12:58.064466+0000","last_peered":"2023-11-24T08:12:58.064466+0000","last_clean":"2023-11-24T08:12:58.064466+0000","last_became_active":"2023-11-24T08:12:58.062391+0000","last_became_peered":"2023-11-24T08:12:58.062391+0000","last_unstale":"2023-11-24T08:12:58.064466+0000","last_undegraded":"2023-11-24T08:12:58.064466+0000","last_fullsized":"2023-11-24T08:12:58.064466+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:05:41.938496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.040299+0000","last_change":"2023-11-24T08:12:58.040299+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.040299+0000","last_undegraded":"2023-11-24T08:12:58.040299+0000","last_fullsized":"2023-11-24T08:12:58.040299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.059428+0000","last_change":"2023-11-24T08:12:58.052352+0000","last_active":"2023-11-24T08:12:58.059428+0000","last_peered":"2023-11-24T08:12:58.059428+0000","last_clean":"2023-11-24T08:12:58.059428+0000","last_became_active":"2023-11-24T08:12:58.052069+0000","last_became_peered":"2023-11-24T08:12:58.052069+0000","last_unstale":"2023-11-24T08:12:58.059428+0000","last_undegraded":"2023-11-24T08:12:58.059428+0000","last_fullsized":"2023-11-24T08:12:58.059428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:15.988776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.039680+0000","last_change":"2023-11-24T08:12:58.039680+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.039680+0000","last_undegraded":"2023-11-24T08:12:58.039680+0000","last_fullsized":"2023-11-24T08:12:58.039680+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.064929+0000","last_change":"2023-11-24T08:12:58.060560+0000","last_active":"2023-11-24T08:12:58.064929+0000","last_peered":"2023-11-24T08:12:58.064929+0000","last_clean":"2023-11-24T08:12:58.064929+0000","last_became_active":"2023-11-24T08:12:58.060019+0000","last_became_peered":"2023-11-24T08:12:58.060019+0000","last_unstale":"2023-11-24T08:12:58.064929+0000","last_undegraded":"2023-11-24T08:12:58.064929+0000","last_fullsized":"2023-11-24T08:12:58.064929+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:07:40.207182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.047217+0000","last_change":"2023-11-24T08:12:58.047217+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.047217+0000","last_undegraded":"2023-11-24T08:12:58.047217+0000","last_fullsized":"2023-11-24T08:12:58.047217+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.063379+0000","last_change":"2023-11-24T08:12:58.061363+0000","last_active":"2023-11-24T08:12:58.063379+0000","last_peered":"2023-11-24T08:12:58.063379+0000","last_clean":"2023-11-24T08:12:58.063379+0000","last_became_active":"2023-11-24T08:12:58.060216+0000","last_became_peered":"2023-11-24T08:12:58.060216+0000","last_unstale":"2023-11-24T08:12:58.063379+0000","last_undegraded":"2023-11-24T08:12:58.063379+0000","last_fullsized":"2023-11-24T08:12:58.063379+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:24.357024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.038439+0000","last_change":"2023-11-24T08:12:58.038439+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.038439+0000","last_undegraded":"2023-11-24T08:12:58.038439+0000","last_fullsized":"2023-11-24T08:12:58.038439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.064771+0000","last_change":"2023-11-24T08:12:58.062225+0000","last_active":"2023-11-24T08:12:58.064771+0000","last_peered":"2023-11-24T08:12:58.064771+0000","last_clean":"2023-11-24T08:12:58.064771+0000","last_became_active":"2023-11-24T08:12:58.061728+0000","last_became_peered":"2023-11-24T08:12:58.061728+0000","last_unstale":"2023-11-24T08:12:58.064771+0000","last_undegraded":"2023-11-24T08:12:58.064771+0000","last_fullsized":"2023-11-24T08:12:58.064771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:36.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.035535+0000","last_change":"2023-11-24T08:12:58.035535+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.035535+0000","last_undegraded":"2023-11-24T08:12:58.035535+0000","last_fullsized":"2023-11-24T08:12:58.035535+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.058892+0000","last_change":"2023-11-24T08:12:58.054354+0000","last_active":"2023-11-24T08:12:58.058892+0000","last_peered":"2023-11-24T08:12:58.058892+0000","last_clean":"2023-11-24T08:12:58.058892+0000","last_became_active":"2023-11-24T08:12:58.050353+0000","last_became_peered":"2023-11-24T08:12:58.050353+0000","last_unstale":"2023-11-24T08:12:58.058892+0000","last_undegraded":"2023-11-24T08:12:58.058892+0000","last_fullsized":"2023-11-24T08:12:58.058892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:08:48.626523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.048816+0000","last_change":"2023-11-24T08:12:58.048816+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.048816+0000","last_undegraded":"2023-11-24T08:12:58.048816+0000","last_fullsized":"2023-11-24T08:12:58.048816+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.056244+0000","last_change":"2023-11-24T08:12:58.052036+0000","last_active":"2023-11-24T08:12:58.056244+0000","last_peered":"2023-11-24T08:12:58.056244+0000","last_clean":"2023-11-24T08:12:58.056244+0000","last_became_active":"2023-11-24T08:12:58.048824+0000","last_became_peered":"2023-11-24T08:12:58.048824+0000","last_unstale":"2023-11-24T08:12:58.056244+0000","last_undegraded":"2023-11-24T08:12:58.056244+0000","last_fullsized":"2023-11-24T08:12:58.056244+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:20:02.819816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.040559+0000","last_change":"2023-11-24T08:12:58.040559+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.040559+0000","last_undegraded":"2023-11-24T08:12:58.040559+0000","last_fullsized":"2023-11-24T08:12:58.040559+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.058951+0000","last_change":"2023-11-24T08:12:58.053338+0000","last_active":"2023-11-24T08:12:58.058951+0000","last_peered":"2023-11-24T08:12:58.058951+0000","last_clean":"2023-11-24T08:12:58.058951+0000","last_became_active":"2023-11-24T08:12:58.052335+0000","last_became_peered":"2023-11-24T08:12:58.052335+0000","last_unstale":"2023-11-24T08:12:58.058951+0000","last_undegraded":"2023-11-24T08:12:58.058951+0000","last_fullsized":"2023-11-24T08:12:58.058951+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:33.242217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.055521+0000","last_change":"2023-11-24T08:12:58.055521+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.055521+0000","last_undegraded":"2023-11-24T08:12:58.055521+0000","last_fullsized":"2023-11-24T08:12:58.055521+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.063142+0000","last_change":"2023-11-24T08:12:58.059636+0000","last_active":"2023-11-24T08:12:58.063142+0000","last_peered":"2023-11-24T08:12:58.063142+0000","last_clean":"2023-11-24T08:12:58.063142+0000","last_became_active":"2023-11-24T08:12:58.058671+0000","last_became_peered":"2023-11-24T08:12:58.058671+0000","last_unstale":"2023-11-24T08:12:58.063142+0000","last_undegraded":"2023-11-24T08:12:58.063142+0000","last_fullsized":"2023-11-24T08:12:58.063142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:13:59.389883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.048851+0000","last_change":"2023-11-24T08:12:58.048851+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.048851+0000","last_undegraded":"2023-11-24T08:12:58.048851+0000","last_fullsized":"2023-11-24T08:12:58.048851+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.050882+0000","last_change":"2023-11-24T08:12:58.047421+0000","last_active":"2023-11-24T08:12:58.050882+0000","last_peered":"2023-11-24T08:12:58.050882+0000","last_clean":"2023-11-24T08:12:58.050882+0000","last_became_active":"2023-11-24T08:12:58.044741+0000","last_became_peered":"2023-11-24T08:12:58.044741+0000","last_unstale":"2023-11-24T08:12:58.050882+0000","last_undegraded":"2023-11-24T08:12:58.050882+0000","last_fullsized":"2023-11-24T08:12:58.050882+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:25:37.659021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.037476+0000","last_change":"2023-11-24T08:12:58.037476+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.037476+0000","last_undegraded":"2023-11-24T08:12:58.037476+0000","last_fullsized":"2023-11-24T08:12:58.037476+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.062119+0000","last_change":"2023-11-24T08:12:58.058054+0000","last_active":"2023-11-24T08:12:58.062119+0000","last_peered":"2023-11-24T08:12:58.062119+0000","last_clean":"2023-11-24T08:12:58.062119+0000","last_became_active":"2023-11-24T08:12:58.057427+0000","last_became_peered":"2023-11-24T08:12:58.057427+0000","last_unstale":"2023-11-24T08:12:58.062119+0000","last_undegraded":"2023-11-24T08:12:58.062119+0000","last_fullsized":"2023-11-24T08:12:58.062119+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:46:19.588792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.042040+0000","last_change":"2023-11-24T08:12:58.042040+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.042040+0000","last_undegraded":"2023-11-24T08:12:58.042040+0000","last_fullsized":"2023-11-24T08:12:58.042040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.069428+0000","last_change":"2023-11-24T08:12:58.064914+0000","last_active":"2023-11-24T08:12:58.069428+0000","last_peered":"2023-11-24T08:12:58.069428+0000","last_clean":"2023-11-24T08:12:58.069428+0000","last_became_active":"2023-11-24T08:12:58.064310+0000","last_became_peered":"2023-11-24T08:12:58.064310+0000","last_unstale":"2023-11-24T08:12:58.069428+0000","last_undegraded":"2023-11-24T08:12:58.069428+0000","last_fullsized":"2023-11-24T08:12:58.069428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:16:25.705659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.035022+0000","last_change":"2023-11-24T08:12:58.035022+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.035022+0000","last_undegraded":"2023-11-24T08:12:58.035022+0000","last_fullsized":"2023-11-24T08:12:58.035022+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.063506+0000","last_change":"2023-11-24T08:12:58.060340+0000","last_active":"2023-11-24T08:12:58.063506+0000","last_peered":"2023-11-24T08:12:58.063506+0000","last_clean":"2023-11-24T08:12:58.063506+0000","last_became_active":"2023-11-24T08:12:58.060022+0000","last_became_peered":"2023-11-24T08:12:58.060022+0000","last_unstale":"2023-11-24T08:12:58.063506+0000","last_undegraded":"2023-11-24T08:12:58.063506+0000","last_fullsized":"2023-11-24T08:12:58.063506+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:36.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.046963+0000","last_change":"2023-11-24T08:12:58.046963+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.046963+0000","last_undegraded":"2023-11-24T08:12:58.046963+0000","last_fullsized":"2023-11-24T08:12:58.046963+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.062785+0000","last_change":"2023-11-24T08:12:58.057389+0000","last_active":"2023-11-24T08:12:58.062785+0000","last_peered":"2023-11-24T08:12:58.062785+0000","last_clean":"2023-11-24T08:12:58.062785+0000","last_became_active":"2023-11-24T08:12:58.057073+0000","last_became_peered":"2023-11-24T08:12:58.057073+0000","last_unstale":"2023-11-24T08:12:58.062785+0000","last_undegraded":"2023-11-24T08:12:58.062785+0000","last_fullsized":"2023-11-24T08:12:58.062785+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:44:04.149861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.056467+0000","last_change":"2023-11-24T08:12:58.056467+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.056467+0000","last_undegraded":"2023-11-24T08:12:58.056467+0000","last_fullsized":"2023-11-24T08:12:58.056467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.062932+0000","last_change":"2023-11-24T08:12:58.060210+0000","last_active":"2023-11-24T08:12:58.062932+0000","last_peered":"2023-11-24T08:12:58.062932+0000","last_clean":"2023-11-24T08:12:58.062932+0000","last_became_active":"2023-11-24T08:12:58.059914+0000","last_became_peered":"2023-11-24T08:12:58.059914+0000","last_unstale":"2023-11-24T08:12:58.062932+0000","last_undegraded":"2023-11-24T08:12:58.062932+0000","last_fullsized":"2023-11-24T08:12:58.062932+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:15.988776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.039566+0000","last_change":"2023-11-24T08:12:58.039566+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.039566+0000","last_undegraded":"2023-11-24T08:12:58.039566+0000","last_fullsized":"2023-11-24T08:12:58.039566+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.063125+0000","last_change":"2023-11-24T08:12:58.061395+0000","last_active":"2023-11-24T08:12:58.063125+0000","last_peered":"2023-11-24T08:12:58.063125+0000","last_clean":"2023-11-24T08:12:58.063125+0000","last_became_active":"2023-11-24T08:12:58.060964+0000","last_became_peered":"2023-11-24T08:12:58.060964+0000","last_unstale":"2023-11-24T08:12:58.063125+0000","last_undegraded":"2023-11-24T08:12:58.063125+0000","last_fullsized":"2023-11-24T08:12:58.063125+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:05:41.938496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.045804+0000","last_change":"2023-11-24T08:12:58.045804+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.045804+0000","last_undegraded":"2023-11-24T08:12:58.045804+0000","last_fullsized":"2023-11-24T08:12:58.045804+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.060137+0000","last_change":"2023-11-24T08:12:58.053951+0000","last_active":"2023-11-24T08:12:58.060137+0000","last_peered":"2023-11-24T08:12:58.060137+0000","last_clean":"2023-11-24T08:12:58.060137+0000","last_became_active":"2023-11-24T08:12:58.053678+0000","last_became_peered":"2023-11-24T08:12:58.053678+0000","last_unstale":"2023-11-24T08:12:58.060137+0000","last_undegraded":"2023-11-24T08:12:58.060137+0000","last_fullsized":"2023-11-24T08:12:58.060137+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:26.125225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.046610+0000","last_change":"2023-11-24T08:12:58.046610+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.046610+0000","last_undegraded":"2023-11-24T08:12:58.046610+0000","last_fullsized":"2023-11-24T08:12:58.046610+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.064640+0000","last_change":"2023-11-24T08:12:58.062100+0000","last_active":"2023-11-24T08:12:58.064640+0000","last_peered":"2023-11-24T08:12:58.064640+0000","last_clean":"2023-11-24T08:12:58.064640+0000","last_became_active":"2023-11-24T08:12:58.061851+0000","last_became_peered":"2023-11-24T08:12:58.061851+0000","last_unstale":"2023-11-24T08:12:58.064640+0000","last_undegraded":"2023-11-24T08:12:58.064640+0000","last_fullsized":"2023-11-24T08:12:58.064640+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:36.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.047019+0000","last_change":"2023-11-24T08:12:58.047019+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.047019+0000","last_undegraded":"2023-11-24T08:12:58.047019+0000","last_fullsized":"2023-11-24T08:12:58.047019+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.052374+0000","last_change":"2023-11-24T08:12:58.050820+0000","last_active":"2023-11-24T08:12:58.052374+0000","last_peered":"2023-11-24T08:12:58.052374+0000","last_clean":"2023-11-24T08:12:58.052374+0000","last_became_active":"2023-11-24T08:12:58.050539+0000","last_became_peered":"2023-11-24T08:12:58.050539+0000","last_unstale":"2023-11-24T08:12:58.052374+0000","last_undegraded":"2023-11-24T08:12:58.052374+0000","last_fullsized":"2023-11-24T08:12:58.052374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:54:30.503971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.045742+0000","last_change":"2023-11-24T08:12:58.045742+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.045742+0000","last_undegraded":"2023-11-24T08:12:58.045742+0000","last_fullsized":"2023-11-24T08:12:58.045742+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.049189+0000","last_change":"2023-11-24T08:12:58.046227+0000","last_active":"2023-11-24T08:12:58.049189+0000","last_peered":"2023-11-24T08:12:58.049189+0000","last_clean":"2023-11-24T08:12:58.049189+0000","last_became_active":"2023-11-24T08:12:58.045672+0000","last_became_peered":"2023-11-24T08:12:58.045672+0000","last_unstale":"2023-11-24T08:12:58.049189+0000","last_undegraded":"2023-11-24T08:12:58.049189+0000","last_fullsized":"2023-11-24T08:12:58.049189+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:36.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.054581+0000","last_change":"2023-11-24T08:12:58.054581+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.054581+0000","last_undegraded":"2023-11-24T08:12:58.054581+0000","last_fullsized":"2023-11-24T08:12:58.054581+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.062358+0000","last_change":"2023-11-24T08:12:58.060251+0000","last_active":"2023-11-24T08:12:58.062358+0000","last_peered":"2023-11-24T08:12:58.062358+0000","last_clean":"2023-11-24T08:12:58.062358+0000","last_became_active":"2023-11-24T08:12:58.059968+0000","last_became_peered":"2023-11-24T08:12:58.059968+0000","last_unstale":"2023-11-24T08:12:58.062358+0000","last_undegraded":"2023-11-24T08:12:58.062358+0000","last_fullsized":"2023-11-24T08:12:58.062358+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:06.577424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.048084+0000","last_change":"2023-11-24T08:12:58.048084+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.048084+0000","last_undegraded":"2023-11-24T08:12:58.048084+0000","last_fullsized":"2023-11-24T08:12:58.048084+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.055373+0000","last_change":"2023-11-24T08:12:58.052871+0000","last_active":"2023-11-24T08:12:58.055373+0000","last_peered":"2023-11-24T08:12:58.055373+0000","last_clean":"2023-11-24T08:12:58.055373+0000","last_became_active":"2023-11-24T08:12:58.051678+0000","last_became_peered":"2023-11-24T08:12:58.051678+0000","last_unstale":"2023-11-24T08:12:58.055373+0000","last_undegraded":"2023-11-24T08:12:58.055373+0000","last_fullsized":"2023-11-24T08:12:58.055373+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:24.357024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.040678+0000","last_change":"2023-11-24T08:12:58.040678+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.040678+0000","last_undegraded":"2023-11-24T08:12:58.040678+0000","last_fullsized":"2023-11-24T08:12:58.040678+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.069292+0000","last_change":"2023-11-24T08:12:58.065160+0000","last_active":"2023-11-24T08:12:58.069292+0000","last_peered":"2023-11-24T08:12:58.069292+0000","last_clean":"2023-11-24T08:12:58.069292+0000","last_became_active":"2023-11-24T08:12:58.064875+0000","last_became_peered":"2023-11-24T08:12:58.064875+0000","last_unstale":"2023-11-24T08:12:58.069292+0000","last_undegraded":"2023-11-24T08:12:58.069292+0000","last_fullsized":"2023-11-24T08:12:58.069292+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:42:59.035824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.042686+0000","last_change":"2023-11-24T08:12:58.042686+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.042686+0000","last_undegraded":"2023-11-24T08:12:58.042686+0000","last_fullsized":"2023-11-24T08:12:58.042686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.060679+0000","last_change":"2023-11-24T08:12:58.057913+0000","last_active":"2023-11-24T08:12:58.060679+0000","last_peered":"2023-11-24T08:12:58.060679+0000","last_clean":"2023-11-24T08:12:58.060679+0000","last_became_active":"2023-11-24T08:12:58.057642+0000","last_became_peered":"2023-11-24T08:12:58.057642+0000","last_unstale":"2023-11-24T08:12:58.060679+0000","last_undegraded":"2023-11-24T08:12:58.060679+0000","last_fullsized":"2023-11-24T08:12:58.060679+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:20:20.871391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.055621+0000","last_change":"2023-11-24T08:12:58.055621+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.055621+0000","last_undegraded":"2023-11-24T08:12:58.055621+0000","last_fullsized":"2023-11-24T08:12:58.055621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.064795+0000","last_change":"2023-11-24T08:12:58.062251+0000","last_active":"2023-11-24T08:12:58.064795+0000","last_peered":"2023-11-24T08:12:58.064795+0000","last_clean":"2023-11-24T08:12:58.064795+0000","last_became_active":"2023-11-24T08:12:58.061964+0000","last_became_peered":"2023-11-24T08:12:58.061964+0000","last_unstale":"2023-11-24T08:12:58.064795+0000","last_undegraded":"2023-11-24T08:12:58.064795+0000","last_fullsized":"2023-11-24T08:12:58.064795+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:25:37.659021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.045838+0000","last_change":"2023-11-24T08:12:58.045838+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.045838+0000","last_undegraded":"2023-11-24T08:12:58.045838+0000","last_fullsized":"2023-11-24T08:12:58.045838+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.059496+0000","last_change":"2023-11-24T08:12:58.052615+0000","last_active":"2023-11-24T08:12:58.059496+0000","last_peered":"2023-11-24T08:12:58.059496+0000","last_clean":"2023-11-24T08:12:58.059496+0000","last_became_active":"2023-11-24T08:12:58.052319+0000","last_became_peered":"2023-11-24T08:12:58.052319+0000","last_unstale":"2023-11-24T08:12:58.059496+0000","last_undegraded":"2023-11-24T08:12:58.059496+0000","last_fullsized":"2023-11-24T08:12:58.059496+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:25:37.659021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.038036+0000","last_change":"2023-11-24T08:12:58.038036+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.038036+0000","last_undegraded":"2023-11-24T08:12:58.038036+0000","last_fullsized":"2023-11-24T08:12:58.038036+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.065167+0000","last_change":"2023-11-24T08:12:58.063685+0000","last_active":"2023-11-24T08:12:58.065167+0000","last_peered":"2023-11-24T08:12:58.065167+0000","last_clean":"2023-11-24T08:12:58.065167+0000","last_became_active":"2023-11-24T08:12:58.063263+0000","last_became_peered":"2023-11-24T08:12:58.063263+0000","last_unstale":"2023-11-24T08:12:58.065167+0000","last_undegraded":"2023-11-24T08:12:58.065167+0000","last_fullsized":"2023-11-24T08:12:58.065167+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:15.988776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.044539+0000","last_change":"2023-11-24T08:12:58.044539+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.044539+0000","last_undegraded":"2023-11-24T08:12:58.044539+0000","last_fullsized":"2023-11-24T08:12:58.044539+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.066469+0000","last_change":"2023-11-24T08:12:58.059428+0000","last_active":"2023-11-24T08:12:58.066469+0000","last_peered":"2023-11-24T08:12:58.066469+0000","last_clean":"2023-11-24T08:12:58.066469+0000","last_became_active":"2023-11-24T08:12:58.059111+0000","last_became_peered":"2023-11-24T08:12:58.059111+0000","last_unstale":"2023-11-24T08:12:58.066469+0000","last_undegraded":"2023-11-24T08:12:58.066469+0000","last_fullsized":"2023-11-24T08:12:58.066469+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:02.960505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.054046+0000","last_change":"2023-11-24T08:12:58.054046+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.054046+0000","last_undegraded":"2023-11-24T08:12:58.054046+0000","last_fullsized":"2023-11-24T08:12:58.054046+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.066821+0000","last_change":"2023-11-24T08:12:58.061502+0000","last_active":"2023-11-24T08:12:58.066821+0000","last_peered":"2023-11-24T08:12:58.066821+0000","last_clean":"2023-11-24T08:12:58.066821+0000","last_became_active":"2023-11-24T08:12:58.061225+0000","last_became_peered":"2023-11-24T08:12:58.061225+0000","last_unstale":"2023-11-24T08:12:58.066821+0000","last_undegraded":"2023-11-24T08:12:58.066821+0000","last_fullsized":"2023-11-24T08:12:58.066821+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:03:03.128536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2023-11-24T08:12:58.043897+0000","last_change":"2023-11-24T08:12:58.043897+0000","last_active":"2023-11-24T08:12:58.011574+0000","last_peered":"2023-11-24T08:12:58.011574+0000","last_clean":"2023-11-24T08:12:58.011574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:12:58.043897+0000","last_undegraded":"2023-11-24T08:12:58.043897+0000","last_fullsized":"2023-11-24T08:12:58.043897+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-24T08:12:58.063180+0000","last_change":"2023-11-24T08:12:58.059478+0000","last_active":"2023-11-24T08:12:58.063180+0000","last_peered":"2023-11-24T08:12:58.063180+0000","last_clean":"2023-11-24T08:12:58.063180+0000","last_became_active":"2023-11-24T08:12:58.059140+0000","last_became_peered":"2023-11-24T08:12:58.059140+0000","last_unstale":"2023-11-24T08:12:58.063180+0000","last_undegraded":"2023-11-24T08:12:58.063180+0000","last_fullsized":"2023-11-24T08:12:58.063180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:07.402307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":242,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-24T08:12:57.028665+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:12:57.028665+0000","last_peered":"2023-11-24T08:12:57.028665+0000","last_clean":"2023-11-24T08:12:57.028665+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:12:57.028665+0000","last_undegraded":"2023-11-24T08:12:57.028665+0000","last_fullsized":"2023-11-24T08:12:57.028665+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":1164483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0880000000000001,"15min":1.0289999999999999},"min":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.516,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.119,"5min":1.1279999999999999,"15min":1.0900000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.6419999999999999,"5min":2.403,"15min":2.7109999999999999},"last":0.84799999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.161,"15min":1.145},"min":{"1min":0.77600000000000002,"5min":0.629,"15min":0.46300000000000002},"max":{"1min":1.579,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.81499999999999995,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.6639999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.444}]},{"osd":2,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1319999999999999,"15min":1.196},"min":{"1min":0.75,"5min":0.67000000000000004,"15min":0.58699999999999997},"max":{"1min":1.4410000000000001,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.232,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7669999999999999,"5min":4.0650000000000004,"15min":4.633},"last":1.28}]},{"osd":3,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.79500000000000004,"5min":0.68700000000000006,"15min":0.504},"max":{"1min":1.6040000000000001,"5min":3.133,"15min":3.133},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.397,"15min":1.3740000000000001},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.706,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.3979999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.331,"15min":1.3340000000000001},"min":{"1min":0.84699999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.635,"5min":2.661,"15min":2.851},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.335},"min":{"1min":0.67700000000000005,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.7929999999999999,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.343}]},{"osd":5,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4910000000000001,"15min":1.4670000000000001},"min":{"1min":0.74099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.841,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.48,"15min":1.48},"min":{"1min":0.89100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9770000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.827}]},{"osd":6,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.556,"15min":1.546},"min":{"1min":0.77300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0529999999999999,"5min":2.992,"15min":2.992},"last":1.754},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4870000000000001,"15min":1.46},"min":{"1min":0.93200000000000005,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.702,"5min":3.798,"15min":3.798},"last":1.716}]}]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":567840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.0880000000000001,"15min":1.085},"min":{"1min":0.64700000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.631,"5min":1.95,"15min":1.95},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0720000000000001,"15min":1.0980000000000001},"min":{"1min":0.68200000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5669999999999999,"5min":1.986,"15min":2.2069999999999999},"last":1.0209999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1539999999999999,"15min":1.248},"min":{"1min":0.85799999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6799999999999999,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.079},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1319999999999999,"15min":1.2609999999999999},"min":{"1min":0.93500000000000005,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5429999999999999,"5min":1.694,"15min":2.4039999999999999},"last":1.1519999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0980000000000001,"15min":1.1160000000000001},"min":{"1min":0.72499999999999998,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.6779999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.226,"15min":1.361},"min":{"1min":0.92800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.8220000000000001,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.3899999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.333,"15min":1.395},"min":{"1min":0.96099999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.262},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.325,"15min":1.3460000000000001},"min":{"1min":1.0489999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.327}]},{"osd":4,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.375,"15min":1.5289999999999999},"min":{"1min":0.85399999999999998,"5min":0.79700000000000004,"15min":0.76500000000000001},"max":{"1min":1.829,"5min":2.895,"15min":2.895},"last":1.53},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4510000000000001,"15min":1.5489999999999999},"min":{"1min":1.101,"5min":0.97999999999999998,"15min":0.90800000000000003},"max":{"1min":1.901,"5min":2.2770000000000001,"15min":2.8820000000000001},"last":1.4370000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4059999999999999,"15min":1.538},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":1.744,"5min":2.2250000000000001,"15min":2.629},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5169999999999999,"15min":1.671},"min":{"1min":1.119,"5min":1.0820000000000001,"15min":1.022},"max":{"1min":2.032,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.7250000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5389999999999999,"15min":1.7030000000000001},"min":{"1min":1.234,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":1.931,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.607},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5429999999999999,"15min":1.669},"min":{"1min":1.077,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.0299999999999998,"5min":2.3660000000000001,"15min":2.6890000000000001},"last":1.6619999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":574115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0349999999999999,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.419,"5min":1.694,"15min":3.1179999999999999},"last":1.161},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.131,"15min":1.181},"min":{"1min":0.85899999999999999,"5min":0.65100000000000002,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":1.859,"15min":2.198},"last":1.444}]},{"osd":1,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1200000000000001,"15min":1.1639999999999999},"min":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.66700000000000004},"max":{"1min":1.536,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.1719999999999999,"15min":1.1519999999999999},"min":{"1min":0.73399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.55,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.218,"15min":1.2170000000000001},"min":{"1min":0.87,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.619,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1379999999999999,"15min":1.169},"min":{"1min":0.84099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5529999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.84999999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1339999999999999,"15min":1.1830000000000001},"min":{"1min":0.78300000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.4810000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.071},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.272,"15min":1.337},"min":{"1min":0.79800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.893,"5min":1.9730000000000001,"15min":1.99},"last":1.3939999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3100000000000001,"15min":1.3380000000000001},"min":{"1min":0.871,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7310000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4550000000000001,"15min":1.5449999999999999},"min":{"1min":1.2949999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.0009999999999999,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.49}]},{"osd":6,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.302,"15min":1.4359999999999999},"min":{"1min":0.91100000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6539999999999999,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":1.26},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.393,"15min":1.4339999999999999},"min":{"1min":1.02,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":1.835}]},{"osd":7,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4419999999999999,"15min":1.5780000000000001},"min":{"1min":0.91700000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.129,"5min":2.129,"15min":3.4420000000000002},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.5780000000000001,"15min":1.6120000000000001},"min":{"1min":1.2150000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.8999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1310720,"data_stored":567840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0720000000000001,"15min":1.139},"min":{"1min":0.71899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5760000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.248},{"interface":"front","average":{"1min":1.129,"5min":1.0569999999999999,"15min":1.085},"min":{"1min":0.68300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5489999999999999,"5min":1.7150000000000001,"15min":2.222},"last":1.2969999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.627,"5min":1.74,"15min":1.74},"last":1.379},{"interface":"front","average":{"1min":1.135,"5min":1.1639999999999999,"15min":1.1830000000000001},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5589999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0620000000000001,"15min":1.1519999999999999},"min":{"1min":0.874,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5780000000000001,"5min":1.5840000000000001,"15min":2.1419999999999999},"last":1.127},{"interface":"front","average":{"1min":1.248,"5min":1.23,"15min":1.347},"min":{"1min":0.85499999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.649,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.292,"15min":1.4119999999999999},"min":{"1min":0.95699999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.7230000000000001,"5min":2.121,"15min":2.9100000000000001},"last":1.472},{"interface":"front","average":{"1min":1.276,"5min":1.242,"15min":1.3160000000000001},"min":{"1min":1.0249999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5049999999999999,"5min":1.778,"15min":1.8280000000000001},"last":1.206}]},{"osd":5,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4510000000000001,"15min":1.5049999999999999},"min":{"1min":1.0009999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.8440000000000001,"5min":2.23,"15min":2.23},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.395,"15min":1.526},"min":{"1min":1.1699999999999999,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.6220000000000001,"5min":2.1890000000000001,"15min":2.3879999999999999},"last":1.544}]},{"osd":6,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2190000000000001,"15min":1.4199999999999999},"min":{"1min":0.88700000000000001,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.7030000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.5049999999999999,"15min":1.6399999999999999},"min":{"1min":1.159,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.794,"5min":2.1230000000000002,"15min":3.391},"last":1.659}]},{"osd":7,"last update":"Fri Nov 24 08:12:00 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4570000000000001,"15min":1.6080000000000001},"min":{"1min":0.98999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.6870000000000001,"5min":2.0750000000000002,"15min":3.1230000000000002},"last":1.429},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.5680000000000001},"min":{"1min":1.056,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6659999999999999,"5min":1.9419999999999999,"15min":3.4849999999999999},"last":1.345}]}]},{"osd":3,"up_from":13,"seq":55834574951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":574115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.55700000000000005,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.829,"5min":3.181,"15min":10.557},"last":0.751},{"interface":"front","average":{"1min":1.093,"5min":0.88800000000000001,"15min":0.91500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":1.2609999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.80200000000000005,"15min":0.83199999999999996},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.496,"5min":2.101,"15min":2.101},"last":1.496},{"interface":"front","average":{"1min":1.054,"5min":0.94399999999999995,"15min":1.0009999999999999},"min":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":1.5660000000000001,"5min":3.3260000000000001,"15min":13.090999999999999},"last":1.5660000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88700000000000001,"15min":0.95899999999999996},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5780000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.081,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.649,"5min":3.2480000000000002,"15min":13.460000000000001},"last":1.018}]},{"osd":4,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.72399999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.79,"5min":1.79,"15min":4.367},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.2110000000000001,"15min":1.3029999999999999},"min":{"1min":0.878,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.5720000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.3440000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.244,"15min":1.3169999999999999},"min":{"1min":0.748,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.0609999999999999,"5min":3.4399999999999999,"15min":13.231},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.1930000000000001,"15min":1.306},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.76,"5min":1.8340000000000001,"15min":5.1180000000000003},"last":1.393}]},{"osd":6,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.331,"15min":1.3979999999999999},"min":{"1min":0.95999999999999996,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":1.9590000000000001,"5min":3.5030000000000001,"15min":13.359999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.242,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0150000000000001,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.6799999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.2989999999999999,"15min":1.4159999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":2.101,"15min":4.5060000000000002},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3029999999999999,"15min":1.4339999999999999},"min":{"1min":0.76400000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.887,"5min":3.331,"15min":4.032},"last":1.607}]}]},{"osd":2,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1310720,"data_stored":567840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.76800000000000002,"15min":0.89000000000000001},"min":{"1min":0.47599999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.86299999999999999,"15min":0.99299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":2.5779999999999998},"last":1.244}]},{"osd":1,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.873,"15min":1.089},"min":{"1min":0.49399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.51,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.93300000000000005,"15min":1.0980000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5920000000000001,"5min":1.8360000000000001,"15min":2.6240000000000001},"last":1.0069999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.84599999999999997,"15min":0.93600000000000005},"min":{"1min":0.56499999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.552,"5min":1.552,"15min":2.2090000000000001},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0069999999999999,"15min":1.2250000000000001},"min":{"1min":0.627,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.631,"5min":1.946,"15min":2.9950000000000001},"last":1.0600000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.27,"15min":1.391},"min":{"1min":0.93400000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7829999999999999,"5min":1.9730000000000001,"15min":2.7160000000000002},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.2549999999999999,"15min":1.381},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8779999999999999,"5min":1.929,"15min":3.3279999999999998},"last":1.3100000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.25,"15min":1.423},"min":{"1min":0.88500000000000001,"5min":0.75800000000000001,"15min":0.73399999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2390000000000001,"15min":1.385},"min":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":1.587}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1930000000000001,"15min":1.3220000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.395},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.323,"15min":1.4870000000000001},"min":{"1min":0.98199999999999998,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.3520000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3180000000000001,"15min":1.484},"min":{"1min":0.77700000000000002,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.1070000000000002,"5min":2.1299999999999999,"15min":2.859},"last":1.675},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.2669999999999999,"15min":1.4490000000000001},"min":{"1min":0.88600000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.9930000000000001,"5min":2.02,"15min":3.3809999999999998},"last":1.6279999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":574115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.81000000000000005,"15min":0.76600000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3400000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.84599999999999997},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":1.6970000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.86099999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81999999999999995,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.369,"5min":1.861,"15min":1.861},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.92700000000000005,"15min":0.88300000000000001},"min":{"1min":0.46500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.84299999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0009999999999999,"15min":0.95599999999999996},"min":{"1min":0.53300000000000003,"5min":0.497,"15min":0.44500000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.044},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.94299999999999995,"15min":0.88600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.006}]},{"osd":4,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.24},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.028,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.236},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.276,"15min":1.2549999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.016,"5min":2.8300000000000001,"15min":3.0409999999999999},"last":1.554}]},{"osd":5,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2869999999999999,"15min":1.208},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.9610000000000001,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.357},{"interface":"front","average":{"1min":1.163,"5min":1.2470000000000001,"15min":1.181},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8759999999999999,"5min":3.286,"15min":3.286},"last":1.252}]},{"osd":6,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3460000000000001,"15min":1.238},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.5270000000000001,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3149999999999999,"15min":1.2110000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9179999999999999,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.5880000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:11:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3500000000000001,"15min":1.3009999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.3959999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.7509999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.377,"15min":1.276},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0649999999999999,"5min":3.653,"15min":3.653},"last":1.157}]}]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1158208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.84299999999999997},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.27,"5min":1.425,"15min":1.6020000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.97199999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.452,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0229999999999999,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.833,"5min":1.891,"15min":1.891},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.88300000000000001,"15min":0.93899999999999995},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.505}]},{"osd":3,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98499999999999999,"15min":1.0920000000000001},"min":{"1min":0.55600000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.5820000000000001,"5min":1.79,"15min":2.0670000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90300000000000002,"15min":1.022},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.641,"5min":1.641,"15min":2.1789999999999998},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2350000000000001,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.619,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":1.133},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0960000000000001,"15min":1.2170000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.569,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":0.82299999999999995}]},{"osd":5,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1439999999999999,"15min":1.282},"min":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.69699999999999995},"max":{"1min":1.7210000000000001,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1879999999999999,"15min":1.377},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5609999999999999,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":1.0580000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.288,"15min":1.405},"min":{"1min":0.84599999999999997,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.792,"5min":1.986,"15min":2.4369999999999998},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2609999999999999,"15min":1.4079999999999999},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.998,"5min":1.998,"15min":2.738},"last":0.96899999999999997}]},{"osd":7,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.4910000000000001},"min":{"1min":0.96899999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.1349999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.4750000000000001},"min":{"1min":0.86799999999999999,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":2.0499999999999998,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":0.90300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:13:03.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:13:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.704+0000 7f0a23e3d640 1 Processor -- start 2023-11-24T08:13:03.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.704+0000 7f0a23e3d640 1 -- start start 2023-11-24T08:13:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.706+0000 7f0a23e3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c0b4140 0x7f0a1c0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.706+0000 7f0a23e3d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0a1c15e0e0 0x7f0a1c15e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.706+0000 7f0a23e3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c06c2c0 con 0x7f0a1c146ec0 2023-11-24T08:13:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.706+0000 7f0a23e3d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c15dcc0 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.706+0000 7f0a23e3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0b4b00 con 0x7f0a1c0b4140 2023-11-24T08:13:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a213d8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c0b4140 0x7f0a1c0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0a1c15e0e0 0x7f0a1c15e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0a1c15e0e0 0x7f0a1c15e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:41178/0 (socket says 172.21.15.107:41178) 2023-11-24T08:13:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a21bd9640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0a1c146ec0 0x7f0a1c06bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58036/0 (socket says 172.21.15.107:58036) 2023-11-24T08:13:03.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 -- 172.21.15.107:0/449188013 learned_addr learned my addr 172.21.15.107:0/449188013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 -- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c0b4140 msgr2=0x7f0a1c0b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 --2- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c0b4140 0x7f0a1c0b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 -- 172.21.15.107:0/449188013 >> v1:172.21.15.120:6789/0 conn(0x7f0a1c146ec0 legacy=0x7f0a1c06bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.707+0000 7f0a223da640 1 -- 172.21.15.107:0/449188013 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c06c590 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.709+0000 7f0a223da640 1 --2- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c15e0e0 0x7f0a1c15e550 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a1402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93cb3502b4e03f03 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.709+0000 7f0a20bd7640 1 -- 172.21.15.107:0/449188013 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1402cb00 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.709+0000 7f0a20bd7640 1 -- 172.21.15.107:0/449188013 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1402cc60 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.709+0000 7f0a20bd7640 1 -- 172.21.15.107:0/449188013 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a140343f0 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 -- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c15e0e0 msgr2=0x7f0a1c15e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c15e0e0 0x7f0a1c15e550 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a1402bf60 comp rx=0 tx=0).stop 2023-11-24T08:13:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 -- 172.21.15.107:0/449188013 shutdown_connections 2023-11-24T08:13:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c15e0e0 0x7f0a1c15e550 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/449188013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c0b4140 0x7f0a1c0b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 -- 172.21.15.107:0/449188013 >> 172.21.15.107:0/449188013 conn(0x7f0a1c06e880 msgr2=0x7f0a1c06ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 -- 172.21.15.107:0/449188013 shutdown_connections 2023-11-24T08:13:03.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.710+0000 7f0a23e3d640 1 -- 172.21.15.107:0/449188013 wait complete. 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.712+0000 7f0a23e3d640 1 Processor -- start 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.713+0000 7f0a23e3d640 1 -- start start 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c0b4140 0x7f0a1c15cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c15e0e0 0x7f0a1c0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c141950 con 0x7f0a1c146ec0 2023-11-24T08:13:03.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c140cd0 con 0x7f0a1c0b4140 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a23e3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c140fb0 con 0x7f0a1c15e0e0 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45226/0 (socket says 172.21.15.107:45226) 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 -- 172.21.15.107:0/3817798864 learned_addr learned my addr 172.21.15.107:0/3817798864 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a21bd9640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c0b4140 0x7f0a1c15cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 -- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c15e0e0 msgr2=0x7f0a1c0cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c15e0e0 0x7f0a1c0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 -- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c0b4140 msgr2=0x7f0a1c15cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c0b4140 0x7f0a1c15cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.714+0000 7f0a213d8640 1 -- 172.21.15.107:0/3817798864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14009b30 con 0x7f0a1c146ec0 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a213d8640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00e870 tx=0x7f0a0c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0094f0 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c00c940 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c141290 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c00ccf0 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.715+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c15d440 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.716+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c141a90 con 0x7f0a1c146ec0 2023-11-24T08:13:03.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.717+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f0a0c013470 con 0x7f0a1c146ec0 2023-11-24T08:13:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.718+0000 7f0a127fc640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 0x7f0a0006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.718+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a0c088210 con 0x7f0a1c146ec0 2023-11-24T08:13:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.719+0000 7f0a21bd9640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 0x7f0a0006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.719+0000 7f0a21bd9640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 0x7f0a0006e1a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0a18000f60 tx=0x7f0a18007c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.722+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a1c141a90 con 0x7f0a1c146ec0 2023-11-24T08:13:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.907+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a1c06ce30 con 0x7f0a0006bce0 2023-11-24T08:13:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.929+0000 7f0a127fc640 1 -- 172.21.15.107:0/3817798864 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287662 (secure 0 0 0) 0x7f0a18005a50 con 0x7f0a0006bce0 2023-11-24T08:13:03.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:13:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 msgr2=0x7f0a0006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 0x7f0a0006e1a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0a18000f60 tx=0x7f0a18007c80 comp rx=0 tx=0).stop 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 msgr2=0x7f0a1c142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00e870 tx=0x7f0a0c00ed30 comp rx=0 tx=0).stop 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 shutdown_connections 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a0006bce0 0x7f0a0006e1a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a1c15e0e0 0x7f0a1c0cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c146ec0 0x7f0a1c142970 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 --2- 172.21.15.107:0/3817798864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a1c0b4140 0x7f0a1c15cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 >> 172.21.15.107:0/3817798864 conn(0x7f0a1c06e880 msgr2=0x7f0a1c0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 shutdown_connections 2023-11-24T08:13:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:03.936+0000 7f0a23e3d640 1 -- 172.21.15.107:0/3817798864 wait complete. 2023-11-24T08:13:03.957 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2023-11-24T08:13:03.585622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277844,"kb_used_data":11752,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754696876,"statfs":{"total":773094113280,"available":772809601024,"internally_reserved":0,"allocated":12034048,"data_stored":5883076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":272354878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.605567+0000","last_change":"2023-11-24T08:12:59.063832+0000","last_active":"2023-11-24T08:13:00.605567+0000","last_peered":"2023-11-24T08:13:00.605567+0000","last_clean":"2023-11-24T08:13:00.605567+0000","last_became_active":"2023-11-24T08:12:59.063583+0000","last_became_peered":"2023-11-24T08:12:59.063583+0000","last_unstale":"2023-11-24T08:13:00.605567+0000","last_undegraded":"2023-11-24T08:13:00.605567+0000","last_fullsized":"2023-11-24T08:13:00.605567+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:38:09.045468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730514+0000","last_change":"2023-11-24T08:12:59.069433+0000","last_active":"2023-11-24T08:13:01.730514+0000","last_peered":"2023-11-24T08:13:01.730514+0000","last_clean":"2023-11-24T08:13:01.730514+0000","last_became_active":"2023-11-24T08:12:59.069208+0000","last_became_peered":"2023-11-24T08:12:59.069208+0000","last_unstale":"2023-11-24T08:13:01.730514+0000","last_undegraded":"2023-11-24T08:13:01.730514+0000","last_fullsized":"2023-11-24T08:13:01.730514+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:24.512892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.605918+0000","last_change":"2023-11-24T08:12:59.070486+0000","last_active":"2023-11-24T08:13:00.605918+0000","last_peered":"2023-11-24T08:13:00.605918+0000","last_clean":"2023-11-24T08:13:00.605918+0000","last_became_active":"2023-11-24T08:12:59.068890+0000","last_became_peered":"2023-11-24T08:12:59.068890+0000","last_unstale":"2023-11-24T08:13:00.605918+0000","last_undegraded":"2023-11-24T08:13:00.605918+0000","last_fullsized":"2023-11-24T08:13:00.605918+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:39.632873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.404448+0000","last_change":"2023-11-24T08:12:59.071951+0000","last_active":"2023-11-24T08:13:01.404448+0000","last_peered":"2023-11-24T08:13:01.404448+0000","last_clean":"2023-11-24T08:13:01.404448+0000","last_became_active":"2023-11-24T08:12:59.071644+0000","last_became_peered":"2023-11-24T08:12:59.071644+0000","last_unstale":"2023-11-24T08:13:01.404448+0000","last_undegraded":"2023-11-24T08:13:01.404448+0000","last_fullsized":"2023-11-24T08:13:01.404448+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:08:27.446555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058203+0000","last_change":"2023-11-24T08:12:59.071565+0000","last_active":"2023-11-24T08:13:00.058203+0000","last_peered":"2023-11-24T08:13:00.058203+0000","last_clean":"2023-11-24T08:13:00.058203+0000","last_became_active":"2023-11-24T08:12:59.069894+0000","last_became_peered":"2023-11-24T08:12:59.069894+0000","last_unstale":"2023-11-24T08:13:00.058203+0000","last_undegraded":"2023-11-24T08:13:00.058203+0000","last_fullsized":"2023-11-24T08:13:00.058203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:16.201252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731933+0000","last_change":"2023-11-24T08:12:59.067176+0000","last_active":"2023-11-24T08:13:01.731933+0000","last_peered":"2023-11-24T08:13:01.731933+0000","last_clean":"2023-11-24T08:13:01.731933+0000","last_became_active":"2023-11-24T08:12:59.066901+0000","last_became_peered":"2023-11-24T08:12:59.066901+0000","last_unstale":"2023-11-24T08:13:01.731933+0000","last_undegraded":"2023-11-24T08:13:01.731933+0000","last_fullsized":"2023-11-24T08:13:01.731933+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:41.439421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732045+0000","last_change":"2023-11-24T08:12:59.069042+0000","last_active":"2023-11-24T08:13:01.732045+0000","last_peered":"2023-11-24T08:13:01.732045+0000","last_clean":"2023-11-24T08:13:01.732045+0000","last_became_active":"2023-11-24T08:12:59.068677+0000","last_became_peered":"2023-11-24T08:12:59.068677+0000","last_unstale":"2023-11-24T08:13:01.732045+0000","last_undegraded":"2023-11-24T08:13:01.732045+0000","last_fullsized":"2023-11-24T08:13:01.732045+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:36.393068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730007+0000","last_change":"2023-11-24T08:12:59.068810+0000","last_active":"2023-11-24T08:13:01.730007+0000","last_peered":"2023-11-24T08:13:01.730007+0000","last_clean":"2023-11-24T08:13:01.730007+0000","last_became_active":"2023-11-24T08:12:59.068449+0000","last_became_peered":"2023-11-24T08:12:59.068449+0000","last_unstale":"2023-11-24T08:13:01.730007+0000","last_undegraded":"2023-11-24T08:13:01.730007+0000","last_fullsized":"2023-11-24T08:13:01.730007+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:47:05.518928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731216+0000","last_change":"2023-11-24T08:12:59.069273+0000","last_active":"2023-11-24T08:13:01.731216+0000","last_peered":"2023-11-24T08:13:01.731216+0000","last_clean":"2023-11-24T08:13:01.731216+0000","last_became_active":"2023-11-24T08:12:59.068926+0000","last_became_peered":"2023-11-24T08:12:59.068926+0000","last_unstale":"2023-11-24T08:13:01.731216+0000","last_undegraded":"2023-11-24T08:13:01.731216+0000","last_fullsized":"2023-11-24T08:13:01.731216+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:53.376428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058521+0000","last_change":"2023-11-24T08:12:59.075243+0000","last_active":"2023-11-24T08:13:00.058521+0000","last_peered":"2023-11-24T08:13:00.058521+0000","last_clean":"2023-11-24T08:13:00.058521+0000","last_became_active":"2023-11-24T08:12:59.074939+0000","last_became_peered":"2023-11-24T08:12:59.074939+0000","last_unstale":"2023-11-24T08:13:00.058521+0000","last_undegraded":"2023-11-24T08:13:00.058521+0000","last_fullsized":"2023-11-24T08:13:00.058521+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:00:56.842537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731306+0000","last_change":"2023-11-24T08:12:59.064130+0000","last_active":"2023-11-24T08:13:01.731306+0000","last_peered":"2023-11-24T08:13:01.731306+0000","last_clean":"2023-11-24T08:13:01.731306+0000","last_became_active":"2023-11-24T08:12:59.063103+0000","last_became_peered":"2023-11-24T08:12:59.063103+0000","last_unstale":"2023-11-24T08:13:01.731306+0000","last_undegraded":"2023-11-24T08:13:01.731306+0000","last_fullsized":"2023-11-24T08:13:01.731306+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:34.095524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731932+0000","last_change":"2023-11-24T08:12:59.065068+0000","last_active":"2023-11-24T08:13:01.731932+0000","last_peered":"2023-11-24T08:13:01.731932+0000","last_clean":"2023-11-24T08:13:01.731932+0000","last_became_active":"2023-11-24T08:12:59.064782+0000","last_became_peered":"2023-11-24T08:12:59.064782+0000","last_unstale":"2023-11-24T08:13:01.731932+0000","last_undegraded":"2023-11-24T08:13:01.731932+0000","last_fullsized":"2023-11-24T08:13:01.731932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:08.419089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.404094+0000","last_change":"2023-11-24T08:12:59.057231+0000","last_active":"2023-11-24T08:13:01.404094+0000","last_peered":"2023-11-24T08:13:01.404094+0000","last_clean":"2023-11-24T08:13:01.404094+0000","last_became_active":"2023-11-24T08:12:59.056953+0000","last_became_peered":"2023-11-24T08:12:59.056953+0000","last_unstale":"2023-11-24T08:13:01.404094+0000","last_undegraded":"2023-11-24T08:13:01.404094+0000","last_fullsized":"2023-11-24T08:13:01.404094+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:09.579877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053718+0000","last_change":"2023-11-24T08:12:59.067163+0000","last_active":"2023-11-24T08:13:00.053718+0000","last_peered":"2023-11-24T08:13:00.053718+0000","last_clean":"2023-11-24T08:13:00.053718+0000","last_became_active":"2023-11-24T08:12:59.066855+0000","last_became_peered":"2023-11-24T08:12:59.066855+0000","last_unstale":"2023-11-24T08:13:00.053718+0000","last_undegraded":"2023-11-24T08:13:00.053718+0000","last_fullsized":"2023-11-24T08:13:00.053718+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:20.414822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.403722+0000","last_change":"2023-11-24T08:12:59.074576+0000","last_active":"2023-11-24T08:13:01.403722+0000","last_peered":"2023-11-24T08:13:01.403722+0000","last_clean":"2023-11-24T08:13:01.403722+0000","last_became_active":"2023-11-24T08:12:59.074211+0000","last_became_peered":"2023-11-24T08:12:59.074211+0000","last_unstale":"2023-11-24T08:13:01.403722+0000","last_undegraded":"2023-11-24T08:13:01.403722+0000","last_fullsized":"2023-11-24T08:13:01.403722+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:32.973123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730658+0000","last_change":"2023-11-24T08:12:59.072253+0000","last_active":"2023-11-24T08:13:01.730658+0000","last_peered":"2023-11-24T08:13:01.730658+0000","last_clean":"2023-11-24T08:13:01.730658+0000","last_became_active":"2023-11-24T08:12:59.070046+0000","last_became_peered":"2023-11-24T08:12:59.070046+0000","last_unstale":"2023-11-24T08:13:01.730658+0000","last_undegraded":"2023-11-24T08:13:01.730658+0000","last_fullsized":"2023-11-24T08:13:01.730658+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:24.512892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730814+0000","last_change":"2023-11-24T08:12:59.076036+0000","last_active":"2023-11-24T08:13:01.730814+0000","last_peered":"2023-11-24T08:13:01.730814+0000","last_clean":"2023-11-24T08:13:01.730814+0000","last_became_active":"2023-11-24T08:12:59.075757+0000","last_became_peered":"2023-11-24T08:12:59.075757+0000","last_unstale":"2023-11-24T08:13:01.730814+0000","last_undegraded":"2023-11-24T08:13:01.730814+0000","last_fullsized":"2023-11-24T08:13:01.730814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:34.095524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.404481+0000","last_change":"2023-11-24T08:12:59.065175+0000","last_active":"2023-11-24T08:13:01.404481+0000","last_peered":"2023-11-24T08:13:01.404481+0000","last_clean":"2023-11-24T08:13:01.404481+0000","last_became_active":"2023-11-24T08:12:59.064924+0000","last_became_peered":"2023-11-24T08:12:59.064924+0000","last_unstale":"2023-11-24T08:13:01.404481+0000","last_undegraded":"2023-11-24T08:13:01.404481+0000","last_fullsized":"2023-11-24T08:13:01.404481+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:27.000977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054467+0000","last_change":"2023-11-24T08:12:59.059962+0000","last_active":"2023-11-24T08:13:00.054467+0000","last_peered":"2023-11-24T08:13:00.054467+0000","last_clean":"2023-11-24T08:13:00.054467+0000","last_became_active":"2023-11-24T08:12:59.059561+0000","last_became_peered":"2023-11-24T08:12:59.059561+0000","last_unstale":"2023-11-24T08:13:00.054467+0000","last_undegraded":"2023-11-24T08:13:00.054467+0000","last_fullsized":"2023-11-24T08:13:00.054467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:36.422054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732293+0000","last_change":"2023-11-24T08:12:59.075090+0000","last_active":"2023-11-24T08:13:01.732293+0000","last_peered":"2023-11-24T08:13:01.732293+0000","last_clean":"2023-11-24T08:13:01.732293+0000","last_became_active":"2023-11-24T08:12:59.074290+0000","last_became_peered":"2023-11-24T08:12:59.074290+0000","last_unstale":"2023-11-24T08:13:01.732293+0000","last_undegraded":"2023-11-24T08:13:01.732293+0000","last_fullsized":"2023-11-24T08:13:01.732293+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:16.201252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054069+0000","last_change":"2023-11-24T08:12:59.058948+0000","last_active":"2023-11-24T08:13:00.054069+0000","last_peered":"2023-11-24T08:13:00.054069+0000","last_clean":"2023-11-24T08:13:00.054069+0000","last_became_active":"2023-11-24T08:12:59.057943+0000","last_became_peered":"2023-11-24T08:12:59.057943+0000","last_unstale":"2023-11-24T08:13:00.054069+0000","last_undegraded":"2023-11-24T08:13:00.054069+0000","last_fullsized":"2023-11-24T08:13:00.054069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:00.951324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730368+0000","last_change":"2023-11-24T08:12:59.064511+0000","last_active":"2023-11-24T08:13:01.730368+0000","last_peered":"2023-11-24T08:13:01.730368+0000","last_clean":"2023-11-24T08:13:01.730368+0000","last_became_active":"2023-11-24T08:12:59.064190+0000","last_became_peered":"2023-11-24T08:12:59.064190+0000","last_unstale":"2023-11-24T08:13:01.730368+0000","last_undegraded":"2023-11-24T08:13:01.730368+0000","last_fullsized":"2023-11-24T08:13:01.730368+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:48.432388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730052+0000","last_change":"2023-11-24T08:12:59.066744+0000","last_active":"2023-11-24T08:13:01.730052+0000","last_peered":"2023-11-24T08:13:01.730052+0000","last_clean":"2023-11-24T08:13:01.730052+0000","last_became_active":"2023-11-24T08:12:59.066410+0000","last_became_peered":"2023-11-24T08:12:59.066410+0000","last_unstale":"2023-11-24T08:13:01.730052+0000","last_undegraded":"2023-11-24T08:13:01.730052+0000","last_fullsized":"2023-11-24T08:13:01.730052+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:47.666744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730932+0000","last_change":"2023-11-24T08:12:59.074101+0000","last_active":"2023-11-24T08:13:01.730932+0000","last_peered":"2023-11-24T08:13:01.730932+0000","last_clean":"2023-11-24T08:13:01.730932+0000","last_became_active":"2023-11-24T08:12:59.073839+0000","last_became_peered":"2023-11-24T08:12:59.073839+0000","last_unstale":"2023-11-24T08:13:01.730932+0000","last_undegraded":"2023-11-24T08:13:01.730932+0000","last_fullsized":"2023-11-24T08:13:01.730932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:09:12.308987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058790+0000","last_change":"2023-11-24T08:12:59.079236+0000","last_active":"2023-11-24T08:13:00.058790+0000","last_peered":"2023-11-24T08:13:00.058790+0000","last_clean":"2023-11-24T08:13:00.058790+0000","last_became_active":"2023-11-24T08:12:59.078883+0000","last_became_peered":"2023-11-24T08:12:59.078883+0000","last_unstale":"2023-11-24T08:13:00.058790+0000","last_undegraded":"2023-11-24T08:13:00.058790+0000","last_fullsized":"2023-11-24T08:13:00.058790+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:27.197173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730688+0000","last_change":"2023-11-24T08:12:59.079896+0000","last_active":"2023-11-24T08:13:01.730688+0000","last_peered":"2023-11-24T08:13:01.730688+0000","last_clean":"2023-11-24T08:13:01.730688+0000","last_became_active":"2023-11-24T08:12:59.078743+0000","last_became_peered":"2023-11-24T08:12:59.078743+0000","last_unstale":"2023-11-24T08:13:01.730688+0000","last_undegraded":"2023-11-24T08:13:01.730688+0000","last_fullsized":"2023-11-24T08:13:01.730688+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:09:12.308987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731716+0000","last_change":"2023-11-24T08:12:59.066382+0000","last_active":"2023-11-24T08:13:01.731716+0000","last_peered":"2023-11-24T08:13:01.731716+0000","last_clean":"2023-11-24T08:13:01.731716+0000","last_became_active":"2023-11-24T08:12:59.066039+0000","last_became_peered":"2023-11-24T08:12:59.066039+0000","last_unstale":"2023-11-24T08:13:01.731716+0000","last_undegraded":"2023-11-24T08:13:01.731716+0000","last_fullsized":"2023-11-24T08:13:01.731716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:34.651481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058291+0000","last_change":"2023-11-24T08:12:59.080599+0000","last_active":"2023-11-24T08:13:00.058291+0000","last_peered":"2023-11-24T08:13:00.058291+0000","last_clean":"2023-11-24T08:13:00.058291+0000","last_became_active":"2023-11-24T08:12:59.080232+0000","last_became_peered":"2023-11-24T08:12:59.080232+0000","last_unstale":"2023-11-24T08:13:00.058291+0000","last_undegraded":"2023-11-24T08:13:00.058291+0000","last_fullsized":"2023-11-24T08:13:00.058291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:50.040369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730594+0000","last_change":"2023-11-24T08:12:59.076060+0000","last_active":"2023-11-24T08:13:01.730594+0000","last_peered":"2023-11-24T08:13:01.730594+0000","last_clean":"2023-11-24T08:13:01.730594+0000","last_became_active":"2023-11-24T08:12:59.075717+0000","last_became_peered":"2023-11-24T08:12:59.075717+0000","last_unstale":"2023-11-24T08:13:01.730594+0000","last_undegraded":"2023-11-24T08:13:01.730594+0000","last_fullsized":"2023-11-24T08:13:01.730594+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:32.610468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053590+0000","last_change":"2023-11-24T08:12:59.060950+0000","last_active":"2023-11-24T08:13:00.053590+0000","last_peered":"2023-11-24T08:13:00.053590+0000","last_clean":"2023-11-24T08:13:00.053590+0000","last_became_active":"2023-11-24T08:12:59.060670+0000","last_became_peered":"2023-11-24T08:12:59.060670+0000","last_unstale":"2023-11-24T08:13:00.053590+0000","last_undegraded":"2023-11-24T08:13:00.053590+0000","last_fullsized":"2023-11-24T08:13:00.053590+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:50:15.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.403671+0000","last_change":"2023-11-24T08:12:59.069169+0000","last_active":"2023-11-24T08:13:01.403671+0000","last_peered":"2023-11-24T08:13:01.403671+0000","last_clean":"2023-11-24T08:13:01.403671+0000","last_became_active":"2023-11-24T08:12:59.068782+0000","last_became_peered":"2023-11-24T08:12:59.068782+0000","last_unstale":"2023-11-24T08:13:01.403671+0000","last_undegraded":"2023-11-24T08:13:01.403671+0000","last_fullsized":"2023-11-24T08:13:01.403671+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:02:38.698720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.605099+0000","last_change":"2023-11-24T08:12:59.070610+0000","last_active":"2023-11-24T08:13:00.605099+0000","last_peered":"2023-11-24T08:13:00.605099+0000","last_clean":"2023-11-24T08:13:00.605099+0000","last_became_active":"2023-11-24T08:12:59.070320+0000","last_became_peered":"2023-11-24T08:12:59.070320+0000","last_unstale":"2023-11-24T08:13:00.605099+0000","last_undegraded":"2023-11-24T08:13:00.605099+0000","last_fullsized":"2023-11-24T08:13:00.605099+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:48.809395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054550+0000","last_change":"2023-11-24T08:12:59.064853+0000","last_active":"2023-11-24T08:13:00.054550+0000","last_peered":"2023-11-24T08:13:00.054550+0000","last_clean":"2023-11-24T08:13:00.054550+0000","last_became_active":"2023-11-24T08:12:59.064556+0000","last_became_peered":"2023-11-24T08:12:59.064556+0000","last_unstale":"2023-11-24T08:13:00.054550+0000","last_undegraded":"2023-11-24T08:13:00.054550+0000","last_fullsized":"2023-11-24T08:13:00.054550+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:48:02.911743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"108'4","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.091212+0000","last_change":"2023-11-24T08:12:58.064302+0000","last_active":"2023-11-24T08:12:59.091212+0000","last_peered":"2023-11-24T08:12:59.091212+0000","last_clean":"2023-11-24T08:12:59.091212+0000","last_became_active":"2023-11-24T08:12:58.063387+0000","last_became_peered":"2023-11-24T08:12:58.063387+0000","last_unstale":"2023-11-24T08:12:59.091212+0000","last_undegraded":"2023-11-24T08:12:59.091212+0000","last_fullsized":"2023-11-24T08:12:59.091212+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:41:37.139893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.404741+0000","last_change":"2023-11-24T08:12:59.072713+0000","last_active":"2023-11-24T08:13:01.404741+0000","last_peered":"2023-11-24T08:13:01.404741+0000","last_clean":"2023-11-24T08:13:01.404741+0000","last_became_active":"2023-11-24T08:12:59.072405+0000","last_became_peered":"2023-11-24T08:12:59.072405+0000","last_unstale":"2023-11-24T08:13:01.404741+0000","last_undegraded":"2023-11-24T08:13:01.404741+0000","last_fullsized":"2023-11-24T08:13:01.404741+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:21.177168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.084733+0000","last_change":"2023-11-24T08:12:58.063990+0000","last_active":"2023-11-24T08:12:59.084733+0000","last_peered":"2023-11-24T08:12:59.084733+0000","last_clean":"2023-11-24T08:12:59.084733+0000","last_became_active":"2023-11-24T08:12:58.063698+0000","last_became_peered":"2023-11-24T08:12:58.063698+0000","last_unstale":"2023-11-24T08:12:59.084733+0000","last_undegraded":"2023-11-24T08:12:59.084733+0000","last_fullsized":"2023-11-24T08:12:59.084733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:16:46.649962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.606043+0000","last_change":"2023-11-24T08:12:59.070696+0000","last_active":"2023-11-24T08:13:00.606043+0000","last_peered":"2023-11-24T08:13:00.606043+0000","last_clean":"2023-11-24T08:13:00.606043+0000","last_became_active":"2023-11-24T08:12:59.070474+0000","last_became_peered":"2023-11-24T08:12:59.070474+0000","last_unstale":"2023-11-24T08:13:00.606043+0000","last_undegraded":"2023-11-24T08:13:00.606043+0000","last_fullsized":"2023-11-24T08:13:00.606043+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:38.652879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.091460+0000","last_change":"2023-11-24T08:12:58.062652+0000","last_active":"2023-11-24T08:12:59.091460+0000","last_peered":"2023-11-24T08:12:59.091460+0000","last_clean":"2023-11-24T08:12:59.091460+0000","last_became_active":"2023-11-24T08:12:58.062391+0000","last_became_peered":"2023-11-24T08:12:58.062391+0000","last_unstale":"2023-11-24T08:12:59.091460+0000","last_undegraded":"2023-11-24T08:12:59.091460+0000","last_fullsized":"2023-11-24T08:12:59.091460+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:02.960505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732511+0000","last_change":"2023-11-24T08:12:59.060447+0000","last_active":"2023-11-24T08:13:01.732511+0000","last_peered":"2023-11-24T08:13:01.732511+0000","last_clean":"2023-11-24T08:13:01.732511+0000","last_became_active":"2023-11-24T08:12:59.060131+0000","last_became_peered":"2023-11-24T08:12:59.060131+0000","last_unstale":"2023-11-24T08:13:01.732511+0000","last_undegraded":"2023-11-24T08:13:01.732511+0000","last_fullsized":"2023-11-24T08:13:01.732511+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:19.915355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.091067+0000","last_change":"2023-11-24T08:12:58.052352+0000","last_active":"2023-11-24T08:12:59.091067+0000","last_peered":"2023-11-24T08:12:59.091067+0000","last_clean":"2023-11-24T08:12:59.091067+0000","last_became_active":"2023-11-24T08:12:58.052069+0000","last_became_peered":"2023-11-24T08:12:58.052069+0000","last_unstale":"2023-11-24T08:12:59.091067+0000","last_undegraded":"2023-11-24T08:12:59.091067+0000","last_fullsized":"2023-11-24T08:12:59.091067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:18.898572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730945+0000","last_change":"2023-11-24T08:12:59.060053+0000","last_active":"2023-11-24T08:13:01.730945+0000","last_peered":"2023-11-24T08:13:01.730945+0000","last_clean":"2023-11-24T08:13:01.730945+0000","last_became_active":"2023-11-24T08:12:59.059681+0000","last_became_peered":"2023-11-24T08:12:59.059681+0000","last_unstale":"2023-11-24T08:13:01.730945+0000","last_undegraded":"2023-11-24T08:13:01.730945+0000","last_fullsized":"2023-11-24T08:13:01.730945+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:27.142007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.085413+0000","last_change":"2023-11-24T08:12:58.060560+0000","last_active":"2023-11-24T08:12:59.085413+0000","last_peered":"2023-11-24T08:12:59.085413+0000","last_clean":"2023-11-24T08:12:59.085413+0000","last_became_active":"2023-11-24T08:12:58.060019+0000","last_became_peered":"2023-11-24T08:12:58.060019+0000","last_unstale":"2023-11-24T08:12:59.085413+0000","last_undegraded":"2023-11-24T08:12:59.085413+0000","last_fullsized":"2023-11-24T08:12:59.085413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:35.405272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730903+0000","last_change":"2023-11-24T08:12:59.072387+0000","last_active":"2023-11-24T08:13:01.730903+0000","last_peered":"2023-11-24T08:13:01.730903+0000","last_clean":"2023-11-24T08:13:01.730903+0000","last_became_active":"2023-11-24T08:12:59.071239+0000","last_became_peered":"2023-11-24T08:12:59.071239+0000","last_unstale":"2023-11-24T08:13:01.730903+0000","last_undegraded":"2023-11-24T08:13:01.730903+0000","last_fullsized":"2023-11-24T08:13:01.730903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:32.610468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.091203+0000","last_change":"2023-11-24T08:12:58.061363+0000","last_active":"2023-11-24T08:12:59.091203+0000","last_peered":"2023-11-24T08:12:59.091203+0000","last_clean":"2023-11-24T08:12:59.091203+0000","last_became_active":"2023-11-24T08:12:58.060216+0000","last_became_peered":"2023-11-24T08:12:58.060216+0000","last_unstale":"2023-11-24T08:12:59.091203+0000","last_undegraded":"2023-11-24T08:12:59.091203+0000","last_fullsized":"2023-11-24T08:12:59.091203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:49.023587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.403632+0000","last_change":"2023-11-24T08:12:59.074625+0000","last_active":"2023-11-24T08:13:01.403632+0000","last_peered":"2023-11-24T08:13:01.403632+0000","last_clean":"2023-11-24T08:13:01.403632+0000","last_became_active":"2023-11-24T08:12:59.073909+0000","last_became_peered":"2023-11-24T08:12:59.073909+0000","last_unstale":"2023-11-24T08:13:01.403632+0000","last_undegraded":"2023-11-24T08:13:01.403632+0000","last_fullsized":"2023-11-24T08:13:01.403632+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:36.393068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.065329+0000","last_change":"2023-11-24T08:12:58.062225+0000","last_active":"2023-11-24T08:12:59.065329+0000","last_peered":"2023-11-24T08:12:59.065329+0000","last_clean":"2023-11-24T08:12:59.065329+0000","last_became_active":"2023-11-24T08:12:58.061728+0000","last_became_peered":"2023-11-24T08:12:58.061728+0000","last_unstale":"2023-11-24T08:12:59.065329+0000","last_undegraded":"2023-11-24T08:12:59.065329+0000","last_fullsized":"2023-11-24T08:12:59.065329+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:52.128862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.729756+0000","last_change":"2023-11-24T08:12:59.065887+0000","last_active":"2023-11-24T08:13:01.729756+0000","last_peered":"2023-11-24T08:13:01.729756+0000","last_clean":"2023-11-24T08:13:01.729756+0000","last_became_active":"2023-11-24T08:12:59.065617+0000","last_became_peered":"2023-11-24T08:12:59.065617+0000","last_unstale":"2023-11-24T08:13:01.729756+0000","last_undegraded":"2023-11-24T08:13:01.729756+0000","last_fullsized":"2023-11-24T08:13:01.729756+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:09.608864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.051221+0000","last_change":"2023-11-24T08:12:58.054354+0000","last_active":"2023-11-24T08:12:59.051221+0000","last_peered":"2023-11-24T08:12:59.051221+0000","last_clean":"2023-11-24T08:12:59.051221+0000","last_became_active":"2023-11-24T08:12:58.050353+0000","last_became_peered":"2023-11-24T08:12:58.050353+0000","last_unstale":"2023-11-24T08:12:59.051221+0000","last_undegraded":"2023-11-24T08:12:59.051221+0000","last_fullsized":"2023-11-24T08:12:59.051221+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:19:59.934542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730870+0000","last_change":"2023-11-24T08:12:59.080507+0000","last_active":"2023-11-24T08:13:01.730870+0000","last_peered":"2023-11-24T08:13:01.730870+0000","last_clean":"2023-11-24T08:13:01.730870+0000","last_became_active":"2023-11-24T08:12:59.079351+0000","last_became_peered":"2023-11-24T08:12:59.079351+0000","last_unstale":"2023-11-24T08:13:01.730870+0000","last_undegraded":"2023-11-24T08:13:01.730870+0000","last_fullsized":"2023-11-24T08:13:01.730870+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:34.651481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.063412+0000","last_change":"2023-11-24T08:12:58.052036+0000","last_active":"2023-11-24T08:12:59.063412+0000","last_peered":"2023-11-24T08:12:59.063412+0000","last_clean":"2023-11-24T08:12:59.063412+0000","last_became_active":"2023-11-24T08:12:58.048824+0000","last_became_peered":"2023-11-24T08:12:58.048824+0000","last_unstale":"2023-11-24T08:12:59.063412+0000","last_undegraded":"2023-11-24T08:12:59.063412+0000","last_fullsized":"2023-11-24T08:12:59.063412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:48:01.894961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732798+0000","last_change":"2023-11-24T08:12:59.062918+0000","last_active":"2023-11-24T08:13:01.732798+0000","last_peered":"2023-11-24T08:13:01.732798+0000","last_clean":"2023-11-24T08:13:01.732798+0000","last_became_active":"2023-11-24T08:12:59.062646+0000","last_became_peered":"2023-11-24T08:12:59.062646+0000","last_unstale":"2023-11-24T08:13:01.732798+0000","last_undegraded":"2023-11-24T08:13:01.732798+0000","last_fullsized":"2023-11-24T08:13:01.732798+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:50.040369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.054965+0000","last_change":"2023-11-24T08:12:58.053338+0000","last_active":"2023-11-24T08:12:59.054965+0000","last_peered":"2023-11-24T08:12:59.054965+0000","last_clean":"2023-11-24T08:12:59.054965+0000","last_became_active":"2023-11-24T08:12:58.052335+0000","last_became_peered":"2023-11-24T08:12:58.052335+0000","last_unstale":"2023-11-24T08:12:59.054965+0000","last_undegraded":"2023-11-24T08:12:59.054965+0000","last_fullsized":"2023-11-24T08:12:59.054965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:23.496109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.059035+0000","last_change":"2023-11-24T08:12:59.071213+0000","last_active":"2023-11-24T08:13:00.059035+0000","last_peered":"2023-11-24T08:13:00.059035+0000","last_clean":"2023-11-24T08:13:00.059035+0000","last_became_active":"2023-11-24T08:12:59.070674+0000","last_became_peered":"2023-11-24T08:12:59.070674+0000","last_unstale":"2023-11-24T08:13:00.059035+0000","last_undegraded":"2023-11-24T08:13:00.059035+0000","last_fullsized":"2023-11-24T08:13:00.059035+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:34.258999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"108'3","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.082071+0000","last_change":"2023-11-24T08:12:58.059636+0000","last_active":"2023-11-24T08:12:59.082071+0000","last_peered":"2023-11-24T08:12:59.082071+0000","last_clean":"2023-11-24T08:12:59.082071+0000","last_became_active":"2023-11-24T08:12:58.058671+0000","last_became_peered":"2023-11-24T08:12:58.058671+0000","last_unstale":"2023-11-24T08:12:59.082071+0000","last_undegraded":"2023-11-24T08:12:59.082071+0000","last_fullsized":"2023-11-24T08:12:59.082071+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:31:07.531281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732049+0000","last_change":"2023-11-24T08:12:59.064014+0000","last_active":"2023-11-24T08:13:01.732049+0000","last_peered":"2023-11-24T08:13:01.732049+0000","last_clean":"2023-11-24T08:13:01.732049+0000","last_became_active":"2023-11-24T08:12:59.062921+0000","last_became_peered":"2023-11-24T08:12:59.062921+0000","last_unstale":"2023-11-24T08:13:01.732049+0000","last_undegraded":"2023-11-24T08:13:01.732049+0000","last_fullsized":"2023-11-24T08:13:01.732049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:02:38.698720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"108'6","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.093673+0000","last_change":"2023-11-24T08:12:58.047421+0000","last_active":"2023-11-24T08:12:59.093673+0000","last_peered":"2023-11-24T08:12:59.093673+0000","last_clean":"2023-11-24T08:12:59.093673+0000","last_became_active":"2023-11-24T08:12:58.044741+0000","last_became_peered":"2023-11-24T08:12:58.044741+0000","last_unstale":"2023-11-24T08:12:59.093673+0000","last_undegraded":"2023-11-24T08:12:59.093673+0000","last_fullsized":"2023-11-24T08:12:59.093673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:51.468224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.403940+0000","last_change":"2023-11-24T08:12:59.066793+0000","last_active":"2023-11-24T08:13:01.403940+0000","last_peered":"2023-11-24T08:13:01.403940+0000","last_clean":"2023-11-24T08:13:01.403940+0000","last_became_active":"2023-11-24T08:12:59.066553+0000","last_became_peered":"2023-11-24T08:12:59.066553+0000","last_unstale":"2023-11-24T08:13:01.403940+0000","last_undegraded":"2023-11-24T08:13:01.403940+0000","last_fullsized":"2023-11-24T08:13:01.403940+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:23:31.866927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.088531+0000","last_change":"2023-11-24T08:12:58.058054+0000","last_active":"2023-11-24T08:12:59.088531+0000","last_peered":"2023-11-24T08:12:59.088531+0000","last_clean":"2023-11-24T08:12:59.088531+0000","last_became_active":"2023-11-24T08:12:58.057427+0000","last_became_peered":"2023-11-24T08:12:58.057427+0000","last_unstale":"2023-11-24T08:12:59.088531+0000","last_undegraded":"2023-11-24T08:12:59.088531+0000","last_fullsized":"2023-11-24T08:12:59.088531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:03:47.628628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.605742+0000","last_change":"2023-11-24T08:12:59.069923+0000","last_active":"2023-11-24T08:13:00.605742+0000","last_peered":"2023-11-24T08:13:00.605742+0000","last_clean":"2023-11-24T08:13:00.605742+0000","last_became_active":"2023-11-24T08:12:59.069606+0000","last_became_peered":"2023-11-24T08:12:59.069606+0000","last_unstale":"2023-11-24T08:13:00.605742+0000","last_undegraded":"2023-11-24T08:13:00.605742+0000","last_fullsized":"2023-11-24T08:13:00.605742+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:20.836378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.039414+0000","last_change":"2023-11-24T08:12:58.064914+0000","last_active":"2023-11-24T08:12:59.039414+0000","last_peered":"2023-11-24T08:12:59.039414+0000","last_clean":"2023-11-24T08:12:59.039414+0000","last_became_active":"2023-11-24T08:12:58.064310+0000","last_became_peered":"2023-11-24T08:12:58.064310+0000","last_unstale":"2023-11-24T08:12:59.039414+0000","last_undegraded":"2023-11-24T08:12:59.039414+0000","last_fullsized":"2023-11-24T08:12:59.039414+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:07:40.207182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.729775+0000","last_change":"2023-11-24T08:12:59.060335+0000","last_active":"2023-11-24T08:13:01.729775+0000","last_peered":"2023-11-24T08:13:01.729775+0000","last_clean":"2023-11-24T08:13:01.729775+0000","last_became_active":"2023-11-24T08:12:59.059997+0000","last_became_peered":"2023-11-24T08:12:59.059997+0000","last_unstale":"2023-11-24T08:13:01.729775+0000","last_undegraded":"2023-11-24T08:13:01.729775+0000","last_fullsized":"2023-11-24T08:13:01.729775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:48:57.385175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.046043+0000","last_change":"2023-11-24T08:12:58.060340+0000","last_active":"2023-11-24T08:12:59.046043+0000","last_peered":"2023-11-24T08:12:59.046043+0000","last_clean":"2023-11-24T08:12:59.046043+0000","last_became_active":"2023-11-24T08:12:58.060022+0000","last_became_peered":"2023-11-24T08:12:58.060022+0000","last_unstale":"2023-11-24T08:12:59.046043+0000","last_undegraded":"2023-11-24T08:12:59.046043+0000","last_fullsized":"2023-11-24T08:12:59.046043+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:30:30.181601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053698+0000","last_change":"2023-11-24T08:12:59.054001+0000","last_active":"2023-11-24T08:13:00.053698+0000","last_peered":"2023-11-24T08:13:00.053698+0000","last_clean":"2023-11-24T08:13:00.053698+0000","last_became_active":"2023-11-24T08:12:59.053656+0000","last_became_peered":"2023-11-24T08:12:59.053656+0000","last_unstale":"2023-11-24T08:13:00.053698+0000","last_undegraded":"2023-11-24T08:13:00.053698+0000","last_fullsized":"2023-11-24T08:13:00.053698+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:57.712168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"108'8","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.088757+0000","last_change":"2023-11-24T08:12:58.057389+0000","last_active":"2023-11-24T08:12:59.088757+0000","last_peered":"2023-11-24T08:12:59.088757+0000","last_clean":"2023-11-24T08:12:59.088757+0000","last_became_active":"2023-11-24T08:12:58.057073+0000","last_became_peered":"2023-11-24T08:12:58.057073+0000","last_unstale":"2023-11-24T08:12:59.088757+0000","last_undegraded":"2023-11-24T08:12:59.088757+0000","last_fullsized":"2023-11-24T08:12:59.088757+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:58.960635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058985+0000","last_change":"2023-11-24T08:12:59.072577+0000","last_active":"2023-11-24T08:13:00.058985+0000","last_peered":"2023-11-24T08:13:00.058985+0000","last_clean":"2023-11-24T08:13:00.058985+0000","last_became_active":"2023-11-24T08:12:59.072275+0000","last_became_peered":"2023-11-24T08:12:59.072275+0000","last_unstale":"2023-11-24T08:13:00.058985+0000","last_undegraded":"2023-11-24T08:13:00.058985+0000","last_fullsized":"2023-11-24T08:13:00.058985+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:03.977288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.089265+0000","last_change":"2023-11-24T08:12:58.060210+0000","last_active":"2023-11-24T08:12:59.089265+0000","last_peered":"2023-11-24T08:12:59.089265+0000","last_clean":"2023-11-24T08:12:59.089265+0000","last_became_active":"2023-11-24T08:12:58.059914+0000","last_became_peered":"2023-11-24T08:12:58.059914+0000","last_unstale":"2023-11-24T08:12:59.089265+0000","last_undegraded":"2023-11-24T08:12:59.089265+0000","last_fullsized":"2023-11-24T08:12:59.089265+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:52.128862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730547+0000","last_change":"2023-11-24T08:12:59.066269+0000","last_active":"2023-11-24T08:13:01.730547+0000","last_peered":"2023-11-24T08:13:01.730547+0000","last_clean":"2023-11-24T08:13:01.730547+0000","last_became_active":"2023-11-24T08:12:59.065984+0000","last_became_peered":"2023-11-24T08:12:59.065984+0000","last_unstale":"2023-11-24T08:13:01.730547+0000","last_undegraded":"2023-11-24T08:13:01.730547+0000","last_fullsized":"2023-11-24T08:13:01.730547+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:43:00.052606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.082512+0000","last_change":"2023-11-24T08:12:58.061395+0000","last_active":"2023-11-24T08:12:59.082512+0000","last_peered":"2023-11-24T08:12:59.082512+0000","last_clean":"2023-11-24T08:12:59.082512+0000","last_became_active":"2023-11-24T08:12:58.060964+0000","last_became_peered":"2023-11-24T08:12:58.060964+0000","last_unstale":"2023-11-24T08:12:59.082512+0000","last_undegraded":"2023-11-24T08:12:59.082512+0000","last_fullsized":"2023-11-24T08:12:59.082512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:52.359646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.732376+0000","last_change":"2023-11-24T08:12:59.061859+0000","last_active":"2023-11-24T08:13:01.732376+0000","last_peered":"2023-11-24T08:13:01.732376+0000","last_clean":"2023-11-24T08:13:01.732376+0000","last_became_active":"2023-11-24T08:12:59.061012+0000","last_became_peered":"2023-11-24T08:12:59.061012+0000","last_unstale":"2023-11-24T08:13:01.732376+0000","last_undegraded":"2023-11-24T08:13:01.732376+0000","last_fullsized":"2023-11-24T08:13:01.732376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:32.973123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.063674+0000","last_change":"2023-11-24T08:12:58.053951+0000","last_active":"2023-11-24T08:12:59.063674+0000","last_peered":"2023-11-24T08:12:59.063674+0000","last_clean":"2023-11-24T08:12:59.063674+0000","last_became_active":"2023-11-24T08:12:58.053678+0000","last_became_peered":"2023-11-24T08:12:58.053678+0000","last_unstale":"2023-11-24T08:12:59.063674+0000","last_undegraded":"2023-11-24T08:12:59.063674+0000","last_fullsized":"2023-11-24T08:12:59.063674+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:26.180391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053797+0000","last_change":"2023-11-24T08:12:59.061977+0000","last_active":"2023-11-24T08:13:00.053797+0000","last_peered":"2023-11-24T08:13:00.053797+0000","last_clean":"2023-11-24T08:13:00.053797+0000","last_became_active":"2023-11-24T08:12:59.061518+0000","last_became_peered":"2023-11-24T08:12:59.061518+0000","last_unstale":"2023-11-24T08:13:00.053797+0000","last_undegraded":"2023-11-24T08:13:00.053797+0000","last_fullsized":"2023-11-24T08:13:00.053797+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:46.576584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.088932+0000","last_change":"2023-11-24T08:12:58.062100+0000","last_active":"2023-11-24T08:12:59.088932+0000","last_peered":"2023-11-24T08:12:59.088932+0000","last_clean":"2023-11-24T08:12:59.088932+0000","last_became_active":"2023-11-24T08:12:58.061851+0000","last_became_peered":"2023-11-24T08:12:58.061851+0000","last_unstale":"2023-11-24T08:12:59.088932+0000","last_undegraded":"2023-11-24T08:12:59.088932+0000","last_fullsized":"2023-11-24T08:12:59.088932+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:07.402307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054340+0000","last_change":"2023-11-24T08:12:59.059606+0000","last_active":"2023-11-24T08:13:00.054340+0000","last_peered":"2023-11-24T08:13:00.054340+0000","last_clean":"2023-11-24T08:13:00.054340+0000","last_became_active":"2023-11-24T08:12:59.058904+0000","last_became_peered":"2023-11-24T08:12:59.058904+0000","last_unstale":"2023-11-24T08:13:00.054340+0000","last_undegraded":"2023-11-24T08:13:00.054340+0000","last_fullsized":"2023-11-24T08:13:00.054340+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:59.977417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.063731+0000","last_change":"2023-11-24T08:12:58.050820+0000","last_active":"2023-11-24T08:12:59.063731+0000","last_peered":"2023-11-24T08:12:59.063731+0000","last_clean":"2023-11-24T08:12:59.063731+0000","last_became_active":"2023-11-24T08:12:58.050539+0000","last_became_peered":"2023-11-24T08:12:58.050539+0000","last_unstale":"2023-11-24T08:12:59.063731+0000","last_undegraded":"2023-11-24T08:12:59.063731+0000","last_fullsized":"2023-11-24T08:12:59.063731+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:06.577424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053590+0000","last_change":"2023-11-24T08:12:59.062070+0000","last_active":"2023-11-24T08:13:00.053590+0000","last_peered":"2023-11-24T08:13:00.053590+0000","last_clean":"2023-11-24T08:13:00.053590+0000","last_became_active":"2023-11-24T08:12:59.061784+0000","last_became_peered":"2023-11-24T08:12:59.061784+0000","last_unstale":"2023-11-24T08:13:00.053590+0000","last_undegraded":"2023-11-24T08:13:00.053590+0000","last_fullsized":"2023-11-24T08:13:00.053590+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:10:03.265428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.062341+0000","last_change":"2023-11-24T08:12:58.046227+0000","last_active":"2023-11-24T08:12:59.062341+0000","last_peered":"2023-11-24T08:12:59.062341+0000","last_clean":"2023-11-24T08:12:59.062341+0000","last_became_active":"2023-11-24T08:12:58.045672+0000","last_became_peered":"2023-11-24T08:12:58.045672+0000","last_unstale":"2023-11-24T08:12:59.062341+0000","last_undegraded":"2023-11-24T08:12:59.062341+0000","last_fullsized":"2023-11-24T08:12:59.062341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:05:41.938496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.057626+0000","last_change":"2023-11-24T08:12:59.073379+0000","last_active":"2023-11-24T08:13:00.057626+0000","last_peered":"2023-11-24T08:13:00.057626+0000","last_clean":"2023-11-24T08:13:00.057626+0000","last_became_active":"2023-11-24T08:12:59.073111+0000","last_became_peered":"2023-11-24T08:12:59.073111+0000","last_unstale":"2023-11-24T08:13:00.057626+0000","last_undegraded":"2023-11-24T08:13:00.057626+0000","last_fullsized":"2023-11-24T08:13:00.057626+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:00.406665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.051977+0000","last_change":"2023-11-24T08:12:58.060251+0000","last_active":"2023-11-24T08:12:59.051977+0000","last_peered":"2023-11-24T08:12:59.051977+0000","last_clean":"2023-11-24T08:12:59.051977+0000","last_became_active":"2023-11-24T08:12:58.059968+0000","last_became_peered":"2023-11-24T08:12:58.059968+0000","last_unstale":"2023-11-24T08:12:59.051977+0000","last_undegraded":"2023-11-24T08:12:59.051977+0000","last_fullsized":"2023-11-24T08:12:59.051977+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:52.128862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.730952+0000","last_change":"2023-11-24T08:12:59.069784+0000","last_active":"2023-11-24T08:13:01.730952+0000","last_peered":"2023-11-24T08:13:01.730952+0000","last_clean":"2023-11-24T08:13:01.730952+0000","last_became_active":"2023-11-24T08:12:59.069496+0000","last_became_peered":"2023-11-24T08:12:59.069496+0000","last_unstale":"2023-11-24T08:13:01.730952+0000","last_undegraded":"2023-11-24T08:13:01.730952+0000","last_fullsized":"2023-11-24T08:13:01.730952+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:41.439421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.078965+0000","last_change":"2023-11-24T08:12:58.052871+0000","last_active":"2023-11-24T08:12:59.078965+0000","last_peered":"2023-11-24T08:12:59.078965+0000","last_clean":"2023-11-24T08:12:59.078965+0000","last_became_active":"2023-11-24T08:12:58.051678+0000","last_became_peered":"2023-11-24T08:12:58.051678+0000","last_unstale":"2023-11-24T08:12:59.078965+0000","last_undegraded":"2023-11-24T08:12:59.078965+0000","last_fullsized":"2023-11-24T08:12:59.078965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:20:20.871391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.605785+0000","last_change":"2023-11-24T08:12:59.067291+0000","last_active":"2023-11-24T08:13:00.605785+0000","last_peered":"2023-11-24T08:13:00.605785+0000","last_clean":"2023-11-24T08:13:00.605785+0000","last_became_active":"2023-11-24T08:12:59.066953+0000","last_became_peered":"2023-11-24T08:12:59.066953+0000","last_unstale":"2023-11-24T08:13:00.605785+0000","last_undegraded":"2023-11-24T08:13:00.605785+0000","last_fullsized":"2023-11-24T08:13:00.605785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:17.520243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731136+0000","last_change":"2023-11-24T08:12:58.065160+0000","last_active":"2023-11-24T08:13:01.731136+0000","last_peered":"2023-11-24T08:13:01.731136+0000","last_clean":"2023-11-24T08:13:01.731136+0000","last_became_active":"2023-11-24T08:12:58.064875+0000","last_became_peered":"2023-11-24T08:12:58.064875+0000","last_unstale":"2023-11-24T08:13:01.731136+0000","last_undegraded":"2023-11-24T08:13:01.731136+0000","last_fullsized":"2023-11-24T08:13:01.731136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:31:07.531281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.606289+0000","last_change":"2023-11-24T08:12:59.065947+0000","last_active":"2023-11-24T08:13:00.606289+0000","last_peered":"2023-11-24T08:13:00.606289+0000","last_clean":"2023-11-24T08:13:00.606289+0000","last_became_active":"2023-11-24T08:12:59.065618+0000","last_became_peered":"2023-11-24T08:12:59.065618+0000","last_unstale":"2023-11-24T08:13:00.606289+0000","last_undegraded":"2023-11-24T08:13:00.606289+0000","last_fullsized":"2023-11-24T08:13:00.606289+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:12:05.293413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.068103+0000","last_change":"2023-11-24T08:12:58.057913+0000","last_active":"2023-11-24T08:12:59.068103+0000","last_peered":"2023-11-24T08:12:59.068103+0000","last_clean":"2023-11-24T08:12:59.068103+0000","last_became_active":"2023-11-24T08:12:58.057642+0000","last_became_peered":"2023-11-24T08:12:58.057642+0000","last_unstale":"2023-11-24T08:12:59.068103+0000","last_undegraded":"2023-11-24T08:12:59.068103+0000","last_fullsized":"2023-11-24T08:12:59.068103+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:33.242217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.058225+0000","last_change":"2023-11-24T08:12:59.080756+0000","last_active":"2023-11-24T08:13:00.058225+0000","last_peered":"2023-11-24T08:13:00.058225+0000","last_clean":"2023-11-24T08:13:00.058225+0000","last_became_active":"2023-11-24T08:12:59.080403+0000","last_became_peered":"2023-11-24T08:12:59.080403+0000","last_unstale":"2023-11-24T08:13:00.058225+0000","last_undegraded":"2023-11-24T08:13:00.058225+0000","last_fullsized":"2023-11-24T08:13:00.058225+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:19.915355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.066236+0000","last_change":"2023-11-24T08:12:58.062251+0000","last_active":"2023-11-24T08:12:59.066236+0000","last_peered":"2023-11-24T08:12:59.066236+0000","last_clean":"2023-11-24T08:12:59.066236+0000","last_became_active":"2023-11-24T08:12:58.061964+0000","last_became_peered":"2023-11-24T08:12:58.061964+0000","last_unstale":"2023-11-24T08:12:59.066236+0000","last_undegraded":"2023-11-24T08:12:59.066236+0000","last_fullsized":"2023-11-24T08:12:59.066236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:48:56.368393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054011+0000","last_change":"2023-11-24T08:12:59.060123+0000","last_active":"2023-11-24T08:13:00.054011+0000","last_peered":"2023-11-24T08:13:00.054011+0000","last_clean":"2023-11-24T08:13:00.054011+0000","last_became_active":"2023-11-24T08:12:59.058488+0000","last_became_peered":"2023-11-24T08:12:59.058488+0000","last_unstale":"2023-11-24T08:13:00.054011+0000","last_undegraded":"2023-11-24T08:13:00.054011+0000","last_fullsized":"2023-11-24T08:13:00.054011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:56:29.456776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.061102+0000","last_change":"2023-11-24T08:12:58.052615+0000","last_active":"2023-11-24T08:12:59.061102+0000","last_peered":"2023-11-24T08:12:59.061102+0000","last_clean":"2023-11-24T08:12:59.061102+0000","last_became_active":"2023-11-24T08:12:58.052319+0000","last_became_peered":"2023-11-24T08:12:58.052319+0000","last_unstale":"2023-11-24T08:12:59.061102+0000","last_undegraded":"2023-11-24T08:12:59.061102+0000","last_fullsized":"2023-11-24T08:12:59.061102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:15.184470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:01.731375+0000","last_change":"2023-11-24T08:12:59.069429+0000","last_active":"2023-11-24T08:13:01.731375+0000","last_peered":"2023-11-24T08:13:01.731375+0000","last_clean":"2023-11-24T08:13:01.731375+0000","last_became_active":"2023-11-24T08:12:59.069048+0000","last_became_peered":"2023-11-24T08:12:59.069048+0000","last_unstale":"2023-11-24T08:13:01.731375+0000","last_undegraded":"2023-11-24T08:13:01.731375+0000","last_fullsized":"2023-11-24T08:13:01.731375+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:00.406665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.065407+0000","last_change":"2023-11-24T08:12:58.063685+0000","last_active":"2023-11-24T08:12:59.065407+0000","last_peered":"2023-11-24T08:12:59.065407+0000","last_clean":"2023-11-24T08:12:59.065407+0000","last_became_active":"2023-11-24T08:12:58.063263+0000","last_became_peered":"2023-11-24T08:12:58.063263+0000","last_unstale":"2023-11-24T08:12:59.065407+0000","last_undegraded":"2023-11-24T08:12:59.065407+0000","last_fullsized":"2023-11-24T08:12:59.065407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:08.592082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.053699+0000","last_change":"2023-11-24T08:12:59.048213+0000","last_active":"2023-11-24T08:13:00.053699+0000","last_peered":"2023-11-24T08:13:00.053699+0000","last_clean":"2023-11-24T08:13:00.053699+0000","last_became_active":"2023-11-24T08:12:59.047275+0000","last_became_peered":"2023-11-24T08:12:59.047275+0000","last_unstale":"2023-11-24T08:13:00.053699+0000","last_undegraded":"2023-11-24T08:13:00.053699+0000","last_fullsized":"2023-11-24T08:13:00.053699+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:53.032157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.054667+0000","last_change":"2023-11-24T08:12:58.059428+0000","last_active":"2023-11-24T08:12:59.054667+0000","last_peered":"2023-11-24T08:12:59.054667+0000","last_clean":"2023-11-24T08:12:59.054667+0000","last_became_active":"2023-11-24T08:12:58.059111+0000","last_became_peered":"2023-11-24T08:12:58.059111+0000","last_unstale":"2023-11-24T08:12:59.054667+0000","last_undegraded":"2023-11-24T08:12:59.054667+0000","last_fullsized":"2023-11-24T08:12:59.054667+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:47.415606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.057879+0000","last_change":"2023-11-24T08:12:59.071625+0000","last_active":"2023-11-24T08:13:00.057879+0000","last_peered":"2023-11-24T08:13:00.057879+0000","last_clean":"2023-11-24T08:13:00.057879+0000","last_became_active":"2023-11-24T08:12:59.069926+0000","last_became_peered":"2023-11-24T08:12:59.069926+0000","last_unstale":"2023-11-24T08:13:00.057879+0000","last_undegraded":"2023-11-24T08:13:00.057879+0000","last_fullsized":"2023-11-24T08:13:00.057879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:08.419089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.036723+0000","last_change":"2023-11-24T08:12:58.061502+0000","last_active":"2023-11-24T08:12:59.036723+0000","last_peered":"2023-11-24T08:12:59.036723+0000","last_clean":"2023-11-24T08:12:59.036723+0000","last_became_active":"2023-11-24T08:12:58.061225+0000","last_became_peered":"2023-11-24T08:12:58.061225+0000","last_unstale":"2023-11-24T08:12:59.036723+0000","last_undegraded":"2023-11-24T08:12:59.036723+0000","last_fullsized":"2023-11-24T08:12:59.036723+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:00.672454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-24T08:13:00.054686+0000","last_change":"2023-11-24T08:12:59.061426+0000","last_active":"2023-11-24T08:13:00.054686+0000","last_peered":"2023-11-24T08:13:00.054686+0000","last_clean":"2023-11-24T08:13:00.054686+0000","last_became_active":"2023-11-24T08:12:59.061158+0000","last_became_peered":"2023-11-24T08:12:59.061158+0000","last_unstale":"2023-11-24T08:13:00.054686+0000","last_undegraded":"2023-11-24T08:13:00.054686+0000","last_fullsized":"2023-11-24T08:13:00.054686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:58.011574+0000","last_clean_scrub_stamp":"2023-11-24T08:12:58.011574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:57:58.242377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.081649+0000","last_change":"2023-11-24T08:12:58.059478+0000","last_active":"2023-11-24T08:12:59.081649+0000","last_peered":"2023-11-24T08:12:59.081649+0000","last_clean":"2023-11-24T08:12:59.081649+0000","last_became_active":"2023-11-24T08:12:58.059140+0000","last_became_peered":"2023-11-24T08:12:58.059140+0000","last_unstale":"2023-11-24T08:12:59.081649+0000","last_undegraded":"2023-11-24T08:12:59.081649+0000","last_fullsized":"2023-11-24T08:12:59.081649+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:12:56.994792+0000","last_clean_scrub_stamp":"2023-11-24T08:12:56.994792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:47.415606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":244,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-24T08:12:59.050730+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:12:59.050730+0000","last_peered":"2023-11-24T08:12:59.050730+0000","last_clean":"2023-11-24T08:12:59.050730+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:12:59.050730+0000","last_undegraded":"2023-11-24T08:12:59.050730+0000","last_fullsized":"2023-11-24T08:12:59.050730+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33084,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338756,"statfs":{"total":96636764160,"available":96602886144,"internally_reserved":0,"allocated":1949696,"data_stored":1177670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0880000000000001,"15min":1.0289999999999999},"min":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.516,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.119,"5min":1.1279999999999999,"15min":1.0900000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.6419999999999999,"5min":2.403,"15min":2.7109999999999999},"last":1.02}]},{"osd":1,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.161,"15min":1.145},"min":{"1min":0.77600000000000002,"5min":0.629,"15min":0.46300000000000002},"max":{"1min":1.579,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.323,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.81499999999999995,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.6639999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.151}]},{"osd":2,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1319999999999999,"15min":1.196},"min":{"1min":0.75,"5min":0.67000000000000004,"15min":0.58699999999999997},"max":{"1min":1.4410000000000001,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7669999999999999,"5min":4.0650000000000004,"15min":4.633},"last":0.82999999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.79500000000000004,"5min":0.68700000000000006,"15min":0.504},"max":{"1min":1.6040000000000001,"5min":3.133,"15min":3.133},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.397,"15min":1.3740000000000001},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.706,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.2190000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.331,"15min":1.3340000000000001},"min":{"1min":0.84699999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.635,"5min":2.661,"15min":2.851},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.335},"min":{"1min":0.67700000000000005,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.7929999999999999,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.28}]},{"osd":5,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4910000000000001,"15min":1.4670000000000001},"min":{"1min":0.74099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.841,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.48,"15min":1.48},"min":{"1min":0.89100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9770000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.5469999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.556,"15min":1.546},"min":{"1min":0.77300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0529999999999999,"5min":2.992,"15min":2.992},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4870000000000001,"15min":1.46},"min":{"1min":0.93200000000000005,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.702,"5min":3.798,"15min":3.798},"last":1.405}]}]},{"osd":6,"up_from":13,"seq":55834574951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":587842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.079,"15min":1.0900000000000001},"min":{"1min":0.875,"5min":0.621,"15min":0.621},"max":{"1min":1.6299999999999999,"5min":1.95,"15min":1.95},"last":1.325},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0620000000000001,"15min":1.0940000000000001},"min":{"1min":0.73599999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5960000000000001,"5min":1.986,"15min":2.2069999999999999},"last":1.083}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.149,"15min":1.2390000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.52,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.024},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.141,"15min":1.2490000000000001},"min":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6240000000000001,"5min":1.694,"15min":2.4039999999999999},"last":1.204}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.091,"15min":1.117},"min":{"1min":0.89100000000000001,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.621,"5min":1.6779999999999999,"15min":1.9330000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.2589999999999999,"15min":1.357},"min":{"1min":0.81699999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7749999999999999,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.5940000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3260000000000001,"15min":1.385},"min":{"1min":0.96999999999999997,"5min":0.83399999999999996,"15min":0.71499999999999997},"max":{"1min":1.538,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.286},{"interface":"front","average":{"1min":1.365,"5min":1.339,"15min":1.3460000000000001},"min":{"1min":0.98599999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.9339999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.397}]},{"osd":4,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.3879999999999999,"15min":1.5169999999999999},"min":{"1min":1.0129999999999999,"5min":0.79900000000000004,"15min":0.76500000000000001},"max":{"1min":1.756,"5min":2.895,"15min":2.895},"last":1.756},{"interface":"front","average":{"1min":1.502,"5min":1.452,"15min":1.542},"min":{"1min":1.0940000000000001,"5min":1.056,"15min":0.90800000000000003},"max":{"1min":1.8160000000000001,"5min":2.242,"15min":2.8820000000000001},"last":1.5429999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4199999999999999,"15min":1.5289999999999999},"min":{"1min":0.94599999999999995,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":2.1320000000000001,"5min":2.2250000000000001,"15min":2.629},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.544,"15min":1.663},"min":{"1min":1.3340000000000001,"5min":1.101,"15min":1.022},"max":{"1min":1.8939999999999999,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.8939999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.5569999999999999,"15min":1.6919999999999999},"min":{"1min":1.274,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":2.101,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5600000000000001,"15min":1.6639999999999999},"min":{"1min":1.151,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6890000000000001},"last":1.8080000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1351680,"data_stored":587260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0349999999999999,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.419,"5min":1.694,"15min":3.1179999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.131,"15min":1.181},"min":{"1min":0.85899999999999999,"5min":0.65100000000000002,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":1.859,"15min":2.198},"last":0.97099999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1200000000000001,"15min":1.1639999999999999},"min":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.66700000000000004},"max":{"1min":1.536,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.208,"5min":1.1719999999999999,"15min":1.1519999999999999},"min":{"1min":0.73399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.55,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.218,"15min":1.2170000000000001},"min":{"1min":0.87,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.619,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1379999999999999,"15min":1.169},"min":{"1min":0.84099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5529999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.167}]},{"osd":3,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1339999999999999,"15min":1.1830000000000001},"min":{"1min":0.78300000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.4810000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.129},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.272,"15min":1.337},"min":{"1min":0.79800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.893,"5min":1.9730000000000001,"15min":1.99},"last":1.042}]},{"osd":4,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3100000000000001,"15min":1.3380000000000001},"min":{"1min":0.871,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7310000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.232},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4550000000000001,"15min":1.5449999999999999},"min":{"1min":1.2949999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.0009999999999999,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.3779999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.302,"15min":1.4359999999999999},"min":{"1min":0.91100000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6539999999999999,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":1.29},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.393,"15min":1.4339999999999999},"min":{"1min":1.02,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":1.331}]},{"osd":7,"last update":"Fri Nov 24 08:12:04 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4419999999999999,"15min":1.5780000000000001},"min":{"1min":0.91700000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.129,"5min":2.129,"15min":3.4420000000000002},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5780000000000001,"15min":1.6120000000000001},"min":{"1min":1.2150000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.4530000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1368064,"data_stored":588926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.075,"15min":1.131},"min":{"1min":0.59999999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.0669999999999999,"15min":1.087},"min":{"1min":0.70899999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.516,"5min":1.7150000000000001,"15min":2.222},"last":1.448}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1259999999999999,"15min":1.147},"min":{"1min":0.71499999999999997,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.593,"5min":1.6639999999999999,"15min":1.74},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.194,"5min":1.1619999999999999,"15min":1.1830000000000001},"min":{"1min":0.81699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6080000000000001,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.3169999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.054,"15min":1.1399999999999999},"min":{"1min":0.66600000000000004,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5049999999999999,"5min":1.5780000000000001,"15min":2.1419999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.2210000000000001,"15min":1.3300000000000001},"min":{"1min":0.71199999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.5580000000000001,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":1.5580000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3069999999999999,"15min":1.4019999999999999},"min":{"1min":0.89000000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.756,"5min":2.121,"15min":2.9100000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.3080000000000001},"min":{"1min":0.91200000000000003,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.8280000000000001},"last":1.6599999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.446,"15min":1.498},"min":{"1min":0.996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.9790000000000001,"5min":2.23,"15min":2.23},"last":1.266},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.403,"15min":1.514},"min":{"1min":0.95999999999999996,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.782,"5min":1.8640000000000001,"15min":2.3879999999999999},"last":1.714}]},{"osd":6,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2290000000000001,"15min":1.3999999999999999},"min":{"1min":0.89200000000000002,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.6220000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5,"15min":1.625},"min":{"1min":0.92000000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9590000000000001,"5min":1.9950000000000001,"15min":3.391},"last":1.397}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.452,"15min":1.593},"min":{"1min":1.157,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.825,"5min":2.052,"15min":3.1230000000000002},"last":1.603},{"interface":"front","average":{"1min":1.415,"5min":1.381,"15min":1.5489999999999999},"min":{"1min":1.113,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7929999999999999,"5min":1.857,"15min":3.4849999999999999},"last":1.748}]}]},{"osd":3,"up_from":13,"seq":55834574952,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1363968,"data_stored":588824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.55700000000000005,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.829,"5min":3.181,"15min":10.557},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.093,"5min":0.88800000000000001,"15min":0.91500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":1.0609999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.80200000000000005,"15min":0.83199999999999996},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.496,"5min":2.101,"15min":2.101},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.054,"5min":0.94399999999999995,"15min":1.0009999999999999},"min":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":1.5660000000000001,"5min":3.3260000000000001,"15min":13.090999999999999},"last":1.488}]},{"osd":2,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88700000000000001,"15min":0.95899999999999996},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5780000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.649,"5min":3.2480000000000002,"15min":13.460000000000001},"last":1.377}]},{"osd":4,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.72399999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.79,"5min":1.79,"15min":4.367},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.2110000000000001,"15min":1.3029999999999999},"min":{"1min":0.878,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.5720000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":0.96999999999999997}]},{"osd":5,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.244,"15min":1.3169999999999999},"min":{"1min":0.748,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.0609999999999999,"5min":3.4399999999999999,"15min":13.231},"last":1.631},{"interface":"front","average":{"1min":1.26,"5min":1.1930000000000001,"15min":1.306},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.76,"5min":1.8340000000000001,"15min":5.1180000000000003},"last":1.573}]},{"osd":6,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.331,"15min":1.3979999999999999},"min":{"1min":0.95999999999999996,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":1.9590000000000001,"5min":3.5030000000000001,"15min":13.359999999999999},"last":1.728},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.242,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0150000000000001,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.6970000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.2989999999999999,"15min":1.4159999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":2.101,"15min":4.5060000000000002},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3029999999999999,"15min":1.4339999999999999},"min":{"1min":0.76400000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.887,"5min":3.331,"15min":4.032},"last":1.5980000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1351680,"data_stored":587224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.78300000000000003,"15min":0.88900000000000001},"min":{"1min":0.66700000000000004,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.90300000000000002,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.782,"5min":1.782,"15min":2.5779999999999998},"last":0.56000000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":0.90800000000000003,"15min":1.081},"min":{"1min":0.66100000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5089999999999999,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":0.877},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95799999999999996,"15min":1.091},"min":{"1min":0.69499999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4399999999999999,"5min":1.788,"15min":2.6240000000000001},"last":0.73999999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84099999999999997,"15min":0.92700000000000005},"min":{"1min":0.58699999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5309999999999999,"5min":1.552,"15min":2.2090000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0289999999999999,"15min":1.214},"min":{"1min":0.54400000000000004,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.738,"5min":1.946,"15min":2.9950000000000001},"last":0.83399999999999996}]},{"osd":4,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.298,"15min":1.387},"min":{"1min":1.0029999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.883,"5min":1.893,"15min":2.7160000000000002},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2490000000000001,"15min":1.371},"min":{"1min":0.96699999999999997,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6459999999999999,"5min":1.8779999999999999,"15min":3.3279999999999998},"last":1.075}]},{"osd":5,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.286,"15min":1.4159999999999999},"min":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.73399999999999999},"max":{"1min":1.9319999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2350000000000001,"15min":1.3740000000000001},"min":{"1min":0.79800000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.724,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":0.95899999999999996}]},{"osd":6,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.194,"15min":1.3100000000000001},"min":{"1min":0.90600000000000003,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":1.8640000000000001,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.161},{"interface":"front","average":{"1min":1.532,"5min":1.3520000000000001,"15min":1.4830000000000001},"min":{"1min":1.129,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.069,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":0.92000000000000004}]},{"osd":7,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.329,"15min":1.4730000000000001},"min":{"1min":0.93999999999999995,"5min":0.77700000000000002,"15min":0.71299999999999997},"max":{"1min":1.7669999999999999,"5min":2.1070000000000002,"15min":2.859},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2789999999999999,"15min":1.4379999999999999},"min":{"1min":0.81399999999999995,"5min":0.746,"15min":0.746},"max":{"1min":1.925,"5min":1.9930000000000001,"15min":3.3809999999999998},"last":1.1279999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1347584,"data_stored":587190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.81699999999999995,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3280000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.427,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.135}]},{"osd":2,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81999999999999995,"15min":0.73499999999999999},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":1.3140000000000001,"5min":1.861,"15min":1.861},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.91800000000000004,"15min":0.89000000000000001},"min":{"1min":0.47499999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4039999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0109999999999999,"15min":0.96799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6599999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93200000000000005,"15min":0.89200000000000002},"min":{"1min":0.42899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.629,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.0349999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.27,"15min":1.2470000000000001},"min":{"1min":0.84699999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.7450000000000001,"5min":2.028,"15min":2.7959999999999998},"last":1.421},{"interface":"front","average":{"1min":1.194,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.77200000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.742,"5min":2.1080000000000001,"15min":3.0409999999999999},"last":1.3169999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.268,"15min":1.218},"min":{"1min":0.81999999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6240000000000001,"5min":2.02,"15min":3.7639999999999998},"last":1.294},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.1839999999999999},"min":{"1min":0.75700000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.665,"5min":1.8759999999999999,"15min":3.286},"last":1.2190000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.321,"15min":1.2430000000000001},"min":{"1min":0.91000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6879999999999999,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.258},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3069999999999999,"15min":1.2210000000000001},"min":{"1min":0.80800000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6970000000000001,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.0920000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3380000000000001,"15min":1.3029999999999999},"min":{"1min":0.71899999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7509999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.454},{"interface":"front","average":{"1min":1.367,"5min":1.345,"15min":1.2849999999999999},"min":{"1min":0.83399999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7190000000000001,"5min":3.653,"15min":3.653},"last":1.3759999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574952,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1945600,"data_stored":1178140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.84299999999999997},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.27,"5min":1.425,"15min":1.6020000000000001},"last":0.439},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.97199999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.452,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0229999999999999,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.833,"5min":1.891,"15min":1.891},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.88300000000000001,"15min":0.93899999999999995},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98499999999999999,"15min":1.0920000000000001},"min":{"1min":0.55600000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.5820000000000001,"5min":1.79,"15min":2.0670000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90300000000000002,"15min":1.022},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.641,"5min":1.641,"15min":2.1789999999999998},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2350000000000001,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.619,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0960000000000001,"15min":1.2170000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.569,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":0.83899999999999997}]},{"osd":5,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1439999999999999,"15min":1.282},"min":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.69699999999999995},"max":{"1min":1.7210000000000001,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1879999999999999,"15min":1.377},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5609999999999999,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":1.1080000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.288,"15min":1.405},"min":{"1min":0.84599999999999997,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.792,"5min":1.986,"15min":2.4369999999999998},"last":1.21},{"interface":"front","average":{"1min":1.264,"5min":1.2609999999999999,"15min":1.4079999999999999},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.998,"5min":1.998,"15min":2.738},"last":0.998}]},{"osd":7,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.4910000000000001},"min":{"1min":0.96899999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.1349999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.4750000000000001},"min":{"1min":0.86799999999999999,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":2.0499999999999998,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.2929999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:13:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:13:03.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:13:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.221+0000 7f990ed67640 1 Processor -- start 2023-11-24T08:13:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.221+0000 7f990ed67640 1 -- start start 2023-11-24T08:13:04.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.223+0000 7f990ed67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99080725b0 0x7f9908071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.223+0000 7f990ed67640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f990815de90 0x7f990815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.223+0000 7f990ed67640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990815ebe0 con 0x7f990815d810 2023-11-24T08:13:04.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.223+0000 7f990ed67640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9908070860 con 0x7f99080725b0 2023-11-24T08:13:04.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.223+0000 7f990ed67640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9908070b40 con 0x7f990815de90 2023-11-24T08:13:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99080725b0 0x7f9908071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99080725b0 0x7f9908071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45238/0 (socket says 172.21.15.107:45238) 2023-11-24T08:13:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 -- 172.21.15.107:0/3907427155 learned_addr learned my addr 172.21.15.107:0/3907427155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:04.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990dd65640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f990815d810 0x7f9908071ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58040/0 (socket says 172.21.15.107:58040) 2023-11-24T08:13:04.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990e566640 1 --2- 172.21.15.107:0/3907427155 >> v2:172.21.15.120:3300/0 conn(0x7f990815de90 0x7f990815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 -- 172.21.15.107:0/3907427155 >> v2:172.21.15.120:3300/0 conn(0x7f990815de90 msgr2=0x7f990815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:04.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 --2- 172.21.15.107:0/3907427155 >> v2:172.21.15.120:3300/0 conn(0x7f990815de90 0x7f990815e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 -- 172.21.15.107:0/3907427155 >> v1:172.21.15.120:6789/0 conn(0x7f990815d810 legacy=0x7f9908071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.224+0000 7f990d564640 1 -- 172.21.15.107:0/3907427155 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908070e20 con 0x7f99080725b0 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.225+0000 7f990d564640 1 --2- 172.21.15.107:0/3907427155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f9908071620 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9908070370 tx=0x7f98f802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220e4eb12a67f865 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.225+0000 7f990cd63640 1 -- 172.21.15.107:0/3907427155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8036440 con 0x7f99080725b0 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990cd63640 1 -- 172.21.15.107:0/3907427155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f80365a0 con 0x7f99080725b0 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990cd63640 1 -- 172.21.15.107:0/3907427155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8036930 con 0x7f99080725b0 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 -- 172.21.15.107:0/3907427155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 msgr2=0x7f9908071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 --2- 172.21.15.107:0/3907427155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f9908071620 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9908070370 tx=0x7f98f802e170 comp rx=0 tx=0).stop 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 -- 172.21.15.107:0/3907427155 shutdown_connections 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 --2- 172.21.15.107:0/3907427155 >> v2:172.21.15.120:3300/0 conn(0x7f990815de90 0x7f990815e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 --2- 172.21.15.107:0/3907427155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f9908071620 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 -- 172.21.15.107:0/3907427155 >> 172.21.15.107:0/3907427155 conn(0x7f990806a500 msgr2=0x7f990806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 -- 172.21.15.107:0/3907427155 shutdown_connections 2023-11-24T08:13:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.226+0000 7f990ed67640 1 -- 172.21.15.107:0/3907427155 wait complete. 2023-11-24T08:13:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.228+0000 7f990ed67640 1 Processor -- start 2023-11-24T08:13:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.228+0000 7f990ed67640 1 -- start start 2023-11-24T08:13:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990815d810 0x7f990813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f990815de90 0x7f99081583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9908153920 con 0x7f99080725b0 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9908158f10 con 0x7f990815d810 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990ed67640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99081591f0 con 0x7f990815de90 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990dd65640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990dd65640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45246/0 (socket says 172.21.15.107:45246) 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.229+0000 7f990dd65640 1 -- 172.21.15.107:0/644031675 learned_addr learned my addr 172.21.15.107:0/644031675 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990d564640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990815d810 0x7f990813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 -- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f990815de90 msgr2=0x7f99081583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f990815de90 0x7f99081583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 -- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990815d810 msgr2=0x7f990813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990815d810 0x7f990813f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 -- 172.21.15.107:0/644031675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f800b770 con 0x7f99080725b0 2023-11-24T08:13:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990dd65640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f990400c4d0 tx=0x7f990400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904007480 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99040075e0 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99040049e0 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99081594d0 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.230+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9908159a90 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.231+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990804e840 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.232+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f9904004b40 con 0x7f99080725b0 2023-11-24T08:13:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.233+0000 7f98fe7fc640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 0x7f98ec06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.233+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9904087450 con 0x7f99080725b0 2023-11-24T08:13:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.233+0000 7f990d564640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 0x7f98ec06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.234+0000 7f990d564640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 0x7f98ec06e1a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f98f8038000 tx=0x7f98f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.237+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9904053d20 con 0x7f99080725b0 2023-11-24T08:13:04.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.500+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f99080cce20 con 0x7f99080725b0 2023-11-24T08:13:04.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.506+0000 7f98fe7fc640 1 -- 172.21.15.107:0/644031675 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f99040180e0 con 0x7f99080725b0 2023-11-24T08:13:04.508 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:13:04.508 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:13:04.509 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:13:04.509 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:13:04.509 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:13:04.509 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 msgr2=0x7f98ec06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 0x7f98ec06e1a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f98f8038000 tx=0x7f98f8008040 comp rx=0 tx=0).stop 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 msgr2=0x7f99081400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f990400c4d0 tx=0x7f990400c990 comp rx=0 tx=0).stop 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 shutdown_connections 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98ec06bce0 0x7f98ec06e1a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f990815de90 0x7f99081583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990815d810 0x7f990813f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 --2- 172.21.15.107:0/644031675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99080725b0 0x7f99081400a0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 >> 172.21.15.107:0/644031675 conn(0x7f990806a500 msgr2=0x7f99080cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:04.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 shutdown_connections 2023-11-24T08:13:04.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.509+0000 7f990ed67640 1 -- 172.21.15.107:0/644031675 wait complete. 2023-11-24T08:13:04.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.739+0000 7f95f6a3d640 1 Processor -- start 2023-11-24T08:13:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.739+0000 7f95f6a3d640 1 -- start start 2023-11-24T08:13:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.740+0000 7f95f6a3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.740+0000 7f95f6a3d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 0x7f95f00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.740+0000 7f95f6a3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f00ce0b0 con 0x7f95f00cf450 2023-11-24T08:13:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.740+0000 7f95f6a3d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f00cfe10 con 0x7f95f015e390 2023-11-24T08:13:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.740+0000 7f95f6a3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f0161960 con 0x7f95f0157c70 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.741+0000 7f95effff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.741+0000 7f95f4fda640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 0x7f95f00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.741+0000 7f95e7fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f95f00cf450 0x7f95f015dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58046/0 (socket says 172.21.15.107:58046) 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.741+0000 7f95f4fda640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 0x7f95f00cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:41224/0 (socket says 172.21.15.107:41224) 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.741+0000 7f95e7fff640 1 -- 172.21.15.107:0/579553848 learned_addr learned my addr 172.21.15.107:0/579553848 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.742+0000 7f95effff640 1 -- 172.21.15.107:0/579553848 >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 msgr2=0x7f95f00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.742+0000 7f95effff640 1 --2- 172.21.15.107:0/579553848 >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 0x7f95f00cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.742+0000 7f95effff640 1 -- 172.21.15.107:0/579553848 >> v1:172.21.15.120:6789/0 conn(0x7f95f00cf450 legacy=0x7f95f015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.742+0000 7f95effff640 1 -- 172.21.15.107:0/579553848 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f015e8d0 con 0x7f95f0157c70 2023-11-24T08:13:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.743+0000 7f95effff640 1 --2- 172.21.15.107:0/579553848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f0161df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097a0 tx=0x7f95e002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77fb4c1e46d5616 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.744+0000 7f95ef7fe640 1 -- 172.21.15.107:0/579553848 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0031780 con 0x7f95f0157c70 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.744+0000 7f95ef7fe640 1 -- 172.21.15.107:0/579553848 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e00318e0 con 0x7f95f0157c70 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.744+0000 7f95ef7fe640 1 -- 172.21.15.107:0/579553848 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0031c70 con 0x7f95f0157c70 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 -- 172.21.15.107:0/579553848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 msgr2=0x7f95f0161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/579553848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f0161df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097a0 tx=0x7f95e002b820 comp rx=0 tx=0).stop 2023-11-24T08:13:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 -- 172.21.15.107:0/579553848 shutdown_connections 2023-11-24T08:13:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/579553848 >> v2:172.21.15.120:3300/0 conn(0x7f95f015e390 0x7f95f00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/579553848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f0161df0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 -- 172.21.15.107:0/579553848 >> 172.21.15.107:0/579553848 conn(0x7f95f006a920 msgr2=0x7f95f006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 -- 172.21.15.107:0/579553848 shutdown_connections 2023-11-24T08:13:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.745+0000 7f95f6a3d640 1 -- 172.21.15.107:0/579553848 wait complete. 2023-11-24T08:13:04.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.747+0000 7f95f6a3d640 1 Processor -- start 2023-11-24T08:13:04.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.747+0000 7f95f6a3d640 1 -- start start 2023-11-24T08:13:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.748+0000 7f95f6a3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.748+0000 7f95f6a3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f01f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95f6a3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f015e390 0x7f95f01f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95f6a3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f01f7180 con 0x7f95f00cf450 2023-11-24T08:13:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95f6a3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f01f7460 con 0x7f95f015e390 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95f6a3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f01f7740 con 0x7f95f0157c70 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45258/0 (socket says 172.21.15.107:45258) 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 -- 172.21.15.107:0/2677833549 learned_addr learned my addr 172.21.15.107:0/2677833549 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95e7fff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f01f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95f4fda640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f015e390 0x7f95f01f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 -- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 msgr2=0x7f95f01f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f01f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 -- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f015e390 msgr2=0x7f95f01f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f015e390 0x7f95f01f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.749+0000 7f95effff640 1 -- 172.21.15.107:0/2677833549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0009400 con 0x7f95f00cf450 2023-11-24T08:13:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95effff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f95e0034040 tx=0x7f95e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0032820 con 0x7f95f00cf450 2023-11-24T08:13:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e0032980 con 0x7f95f00cf450 2023-11-24T08:13:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0032d30 con 0x7f95f00cf450 2023-11-24T08:13:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f01f33a0 con 0x7f95f00cf450 2023-11-24T08:13:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.750+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f0155920 con 0x7f95f00cf450 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.751+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f01f38c0 con 0x7f95f00cf450 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.752+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f95e004e440 con 0x7f95f00cf450 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.753+0000 7f95ed7fa640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 0x7f95d006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.753+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95e00aa1a0 con 0x7f95f00cf450 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.754+0000 7f95e7fff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 0x7f95d006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:04.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.754+0000 7f95e7fff640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 0x7f95d006e1d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f95f006d0b0 tx=0x7f95d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:04.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.758+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95f01f38c0 con 0x7f95f00cf450 2023-11-24T08:13:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.986+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95f01f38c0 con 0x7f95f00cf450 2023-11-24T08:13:04.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.987+0000 7f95ed7fa640 1 -- 172.21.15.107:0/2677833549 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f95f01f38c0 con 0x7f95f00cf450 2023-11-24T08:13:04.990 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:04.990 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:04.990 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 59 2023-11-24T08:13:04.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 msgr2=0x7f95d006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 0x7f95d006e1d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f95f006d0b0 tx=0x7f95d8009040 comp rx=0 tx=0).stop 2023-11-24T08:13:04.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 msgr2=0x7f95f01553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f95e0034040 tx=0x7f95e00046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 shutdown_connections 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d006bd10 0x7f95d006e1d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f015e390 0x7f95f01f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f0157c70 0x7f95f01f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 --2- 172.21.15.107:0/2677833549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f00cf450 0x7f95f01553e0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 >> 172.21.15.107:0/2677833549 conn(0x7f95f006a920 msgr2=0x7f95f00b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.990+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 shutdown_connections 2023-11-24T08:13:04.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:04.991+0000 7f95f6a3d640 1 -- 172.21.15.107:0/2677833549 wait complete. 2023-11-24T08:13:05.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:12:58.491334+0000', 'modified': '2023-11-24T08:13:01.555373+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7863}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7863': {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/2728005588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6835', 'nonce': 2728005588}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2728005588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:13:05.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:13:05.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:13:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.637+0000 7f9db7fff640 1 Processor -- start 2023-11-24T08:13:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.637+0000 7f9db7fff640 1 -- start start 2023-11-24T08:13:05.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.639+0000 7f9db7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9db80b1250 0x7f9db8166160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.639+0000 7f9db7fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80ca0e0 con 0x7f9db8070550 2023-11-24T08:13:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.639+0000 7f9db7fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db8072800 con 0x7f9db815b1d0 2023-11-24T08:13:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.639+0000 7f9db7fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db815fde0 con 0x7f9db80b1250 2023-11-24T08:13:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9daffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9db8070550 0x7f9db80b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52764/0 (socket says 172.21.15.107:52764) 2023-11-24T08:13:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9daffff640 1 -- 172.21.15.107:0/953171538 learned_addr learned my addr 172.21.15.107:0/953171538 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981612568 0 0) 0x7f9db80ca0e0 con 0x7f9db8070550 2023-11-24T08:13:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da4003520 con 0x7f9db8070550 2023-11-24T08:13:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db77fe640 1 --2- 172.21.15.107:0/953171538 >> v2:172.21.15.120:3300/0 conn(0x7f9db80b1250 0x7f9db8166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705572778 0 0) 0x7f9da4003520 con 0x7f9db8070550 2023-11-24T08:13:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 >> v2:172.21.15.120:3300/0 conn(0x7f9db80b1250 msgr2=0x7f9db8166160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 --2- 172.21.15.107:0/953171538 >> v2:172.21.15.120:3300/0 conn(0x7f9db80b1250 0x7f9db8166160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 >> v1:172.21.15.120:6789/0 conn(0x7f9db815b1d0 legacy=0x7f9db8071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80c97b0 con 0x7f9db8070550 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9da0001670 con 0x7f9db8070550 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 >> v1:172.21.15.107:6789/0 conn(0x7f9db8070550 legacy=0x7f9db80b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.640+0000 7f9db67fc640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da4003970 0x7f9da4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db67fc640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 0x7f9da4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db67fc640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9da4005180 0x7f9da400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db815fde0 con 0x7f9da40047c0 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da400c0b0 con 0x7f9da4005180 2023-11-24T08:13:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da400c270 con 0x7f9da4003970 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 0x7f9da4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 -- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da4003970 msgr2=0x7f9da4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da4003970 0x7f9da4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 -- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9da4005180 msgr2=0x7f9da400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9da4005180 0x7f9da400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9daffff640 1 -- 172.21.15.107:0/953171538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da4003520 con 0x7f9da40047c0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.641+0000 7f9db77fe640 1 -- 172.21.15.107:0/953171538 reap_dead start 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.642+0000 7f9daffff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 0x7f9da4004c40 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38699b19d8e5e0ae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.642+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0005ad0 con 0x7f9da40047c0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.642+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da0005c50 con 0x7f9da40047c0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.642+0000 7f9db67fc640 1 -- 172.21.15.107:0/953171538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da002c8b0 con 0x7f9da40047c0 2023-11-24T08:13:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 -- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 msgr2=0x7f9da4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 0x7f9da4004c40 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da40057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 -- 172.21.15.107:0/953171538 shutdown_connections 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 --2- 172.21.15.107:0/953171538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9da40047c0 0x7f9da4004c40 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 -- 172.21.15.107:0/953171538 >> 172.21.15.107:0/953171538 conn(0x7f9db80cbb40 msgr2=0x7f9db80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 -- 172.21.15.107:0/953171538 shutdown_connections 2023-11-24T08:13:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.643+0000 7f9db7fff640 1 -- 172.21.15.107:0/953171538 wait complete. 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.645+0000 7f9db7fff640 1 Processor -- start 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.645+0000 7f9db7fff640 1 -- start start 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da40047c0 0x7f9db815a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db809a3e0 0x7f9db809a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db81e6e60 con 0x7f9db80ad3a0 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80add40 con 0x7f9db809a3e0 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9db7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db809ada0 con 0x7f9da40047c0 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45272/0 (socket says 172.21.15.107:45272) 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.646+0000 7f9daffff640 1 -- 172.21.15.107:0/3363849986 learned_addr learned my addr 172.21.15.107:0/3363849986 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9db77fe640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db809a3e0 0x7f9db809a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 -- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da40047c0 msgr2=0x7f9db815a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da40047c0 0x7f9db815a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da40047c0 0x7f9db815a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 -- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db809a3e0 msgr2=0x7f9db809a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db809a3e0 0x7f9db809a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 -- 172.21.15.107:0/3363849986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db809aff0 con 0x7f9db80ad3a0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.647+0000 7f9daffff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f9da002c450 tx=0x7f9da002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.648+0000 7f9daf7fe640 1 -- 172.21.15.107:0/3363849986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da003c040 con 0x7f9db80ad3a0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.648+0000 7f9daf7fe640 1 -- 172.21.15.107:0/3363849986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da002cd80 con 0x7f9db80ad3a0 2023-11-24T08:13:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.648+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db809b2d0 con 0x7f9db80ad3a0 2023-11-24T08:13:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.648+0000 7f9daf7fe640 1 -- 172.21.15.107:0/3363849986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0007290 con 0x7f9db80ad3a0 2023-11-24T08:13:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.648+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db809b890 con 0x7f9db80ad3a0 2023-11-24T08:13:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.650+0000 7f9daf7fe640 1 -- 172.21.15.107:0/3363849986 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f9da0032070 con 0x7f9db80ad3a0 2023-11-24T08:13:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.651+0000 7f9daf7fe640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 0x7f9d9806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.651+0000 7f9daf7fe640 1 -- 172.21.15.107:0/3363849986 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9da00a9890 con 0x7f9db80ad3a0 2023-11-24T08:13:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.651+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 0x7f9d9806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.652+0000 7f9dc01ab740 1 Processor -- start 2023-11-24T08:13:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.652+0000 7f9dc01ab740 1 -- start start 2023-11-24T08:13:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.652+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 0x7f9d9806e1a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9db8064320 tx=0x7f9da8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f4839df450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a22730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48393d880 con 0x55f4839d9d70 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48394cae0 con 0x55f4839de7b0 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f48394a1d0 con 0x55f4839df070 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45278/0 (socket says 172.21.15.107:45278) 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9daffff640 1 -- 172.21.15.107:0/2019463121 learned_addr learned my addr 172.21.15.107:0/2019463121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9daffff640 1 -- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 msgr2=0x55f4839df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.653+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f4839df450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9db77fe640 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a22730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9daffff640 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f4839df450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9daffff640 1 -- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 msgr2=0x55f483a22730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9daffff640 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a22730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9daffff640 1 -- 172.21.15.107:0/2019463121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f48390d860 con 0x55f4839d9d70 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9daffff640 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da0079830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da10ae91922a5aba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9dacff9640 1 -- 172.21.15.107:0/2019463121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0079cf0 con 0x55f4839d9d70 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9dacff9640 1 -- 172.21.15.107:0/2019463121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da007de40 con 0x55f4839d9d70 2023-11-24T08:13:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.654+0000 7f9dacff9640 1 -- 172.21.15.107:0/2019463121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0081ff0 con 0x55f4839d9d70 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 -- 172.21.15.107:0/2019463121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 msgr2=0x55f4839de270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da0079830 comp rx=0 tx=0).stop 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 -- 172.21.15.107:0/2019463121 shutdown_connections 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a22730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839de270 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/2019463121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f4839df450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 -- 172.21.15.107:0/2019463121 >> 172.21.15.107:0/2019463121 conn(0x55f4839a0c70 msgr2=0x55f483a24b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 -- 172.21.15.107:0/2019463121 shutdown_connections 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.655+0000 7f9dc01ab740 1 -- 172.21.15.107:0/2019463121 wait complete. 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.656+0000 7f9dc01ab740 1 -- start start 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.656+0000 7f9dc01ab740 1 Processor -- start 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 0x55f483a2b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f483945ac0 con 0x55f4839df070 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4838e22f0 con 0x55f4839de7b0 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4838ffc70 con 0x55f4839d9d70 2023-11-24T08:13:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9dc01ab740 10 client.0 _ll_register_callbacks cb 0x55f4839043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45292/0 (socket says 172.21.15.107:45292) 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9db77fe640 1 -- 172.21.15.107:0/1440017591 learned_addr learned my addr 172.21.15.107:0/1440017591 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9daffff640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.657+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 0x55f483a2b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 -- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 msgr2=0x55f483a2b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 0x55f483a2b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 -- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 msgr2=0x55f483a261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 -- 172.21.15.107:0/1440017591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db809b890 con 0x55f4839df070 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 0x55f483a2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9daffff640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9db77fe640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9db80717d0 tx=0x7f9db0007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930b95e76f112dcb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0003990 con 0x55f4839df070 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9db0003af0 con 0x55f4839df070 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f48393eaa0 con 0x55f4839df070 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db0003cb0 con 0x55f4839df070 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9dc01ab740 10 client.8001 fetch_fsmap learned FSMap version 59 2023-11-24T08:13:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.658+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x55f483944140 con 0x55f4839df070 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f48393eaa0 con 0x55f4839df070 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 59) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9db00106d0 con 0x55f4839df070 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 10 client.8001 fetch_fsmap finished waiting for FSMap version 59 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 10 client.8001 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 20 client.8001 populate_metadata read hostname 'smithi107' 2023-11-24T08:13:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a3fb10 0x55f483a41f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 4 client.8001 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 --> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f48393e7c0 con 0x55f483a3fb10 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.659+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a3fb10 0x55f483a41f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.660+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a3fb10 0x55f483a41f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40cd9b65f22bc41 server_cookie=e0ff853d4b0e738f in_seq=0 out_seq=0 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.660+0000 7f9d95ffb640 10 client.8001 ms_handle_connect on v2:172.21.15.107:6835/2728005588 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1440017591 <== mds.0 v2:172.21.15.107:6835/2728005588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f48393e7c0 con 0x55f483a3fb10 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9d95ffb640 10 client.8001 handle_command_reply: tid=0 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 client.8001 shutdown 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 msgr2=0x55f483a26720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9db80717d0 tx=0x7f9db0007b10 comp rx=0 tx=0).stop 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 shutdown_connections 2023-11-24T08:13:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a3fb10 0x55f483a41f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839df070 0x55f483a26720 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839de7b0 0x55f483a261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9d95ffb640 0 client.8001 ms_handle_reset on v2:172.21.15.107:6835/2728005588 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1440017591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839d9d70 0x55f483a2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 >> 172.21.15.107:0/1440017591 conn(0x55f4839a0c70 msgr2=0x55f4839df450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 shutdown_connections 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1440017591 wait complete. 2023-11-24T08:13:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.661+0000 7f9dc01ab740 20 client.8001 trim_cache size 0 max 16384 2023-11-24T08:13:05.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.678+0000 7f9dc01ab740 1 Processor -- start 2023-11-24T08:13:05.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.678+0000 7f9dc01ab740 1 -- start start 2023-11-24T08:13:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839de7b0 0x55f4839df450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839df070 0x55f483a293d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4836a8f20 con 0x55f4839d9d70 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f483947a40 con 0x55f4839df070 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f483a37570 con 0x55f4839de7b0 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9db6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9db6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45298/0 (socket says 172.21.15.107:45298) 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.679+0000 7f9db6ffd640 1 -- 172.21.15.107:0/3178255941 learned_addr learned my addr 172.21.15.107:0/3178255941 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 -- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839de7b0 msgr2=0x55f4839df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839de7b0 0x55f4839df450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 -- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839df070 msgr2=0x55f483a293d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839df070 0x55f483a293d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 -- 172.21.15.107:0/3178255941 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4838e1b80 con 0x55f4839d9d70 2023-11-24T08:13:05.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f9da8007c10 tx=0x7f9da8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2646f027dd664c7a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3178255941 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8004ce0 con 0x55f4839d9d70 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3178255941 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da801b070 con 0x55f4839d9d70 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.680+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3178255941 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da802c640 con 0x55f4839d9d70 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 -- 172.21.15.107:0/3178255941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 msgr2=0x55f4839ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f9da8007c10 tx=0x7f9da8013000 comp rx=0 tx=0).stop 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 -- 172.21.15.107:0/3178255941 shutdown_connections 2023-11-24T08:13:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839df070 0x55f483a293d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839de7b0 0x55f4839df450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/3178255941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839d9d70 0x55f4839ff8b0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 -- 172.21.15.107:0/3178255941 >> 172.21.15.107:0/3178255941 conn(0x55f4839a0c70 msgr2=0x55f4839d8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 -- 172.21.15.107:0/3178255941 shutdown_connections 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.681+0000 7f9dc01ab740 1 -- 172.21.15.107:0/3178255941 wait complete. 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.682+0000 7f9dc01ab740 1 -- start start 2023-11-24T08:13:05.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.682+0000 7f9dc01ab740 1 Processor -- start 2023-11-24T08:13:05.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839d9d70 0x55f483a26030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f483a24b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f483a0bca0 con 0x55f4839de7b0 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f483a027e0 con 0x55f4839d9d70 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f483a067f0 con 0x55f4839df070 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9dc01ab740 10 client.0 _ll_register_callbacks cb 0x7f9db0007330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9daffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45308/0 (socket says 172.21.15.107:45308) 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9daffff640 1 -- 172.21.15.107:0/1587224506 learned_addr learned my addr 172.21.15.107:0/1587224506 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839d9d70 0x55f483a26030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9db77fe640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f483a24b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9daffff640 1 -- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 msgr2=0x55f483a24b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.683+0000 7f9daffff640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f483a24b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9daffff640 1 -- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839d9d70 msgr2=0x55f483a26030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9daffff640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839d9d70 0x55f483a26030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9daffff640 1 -- 172.21.15.107:0/1587224506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f483907170 con 0x55f4839de7b0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9daffff640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001930 tx=0x7f9da0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efae246364f8b40 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0079cf0 con 0x55f4839de7b0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9da0081ff0 con 0x55f4839de7b0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9da007de40 con 0x55f4839de7b0 2023-11-24T08:13:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9da00a9260 con 0x55f4839de7b0 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 10 client.8007 fetch_fsmap learned FSMap version 59 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x55f4838edf40 con 0x55f4839de7b0 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da009c070 con 0x55f4839de7b0 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 59) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9da00a9400 con 0x55f4839de7b0 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 10 client.8007 fetch_fsmap finished waiting for FSMap version 59 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 10 client.8007 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 20 client.8007 populate_metadata read hostname 'smithi107' 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.684+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a25190 0x55f483a25570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.685+0000 7f9dc01ab740 4 client.8007 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.685+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 --> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4836466c0 con 0x55f483a25190 2023-11-24T08:13:05.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.685+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a25190 0x55f483a25570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.685+0000 7f9db6ffd640 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a25190 0x55f483a25570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55c6987f6ba84a8 server_cookie=82aca4729e10722e in_seq=0 out_seq=0 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.685+0000 7f9d95ffb640 10 client.8007 ms_handle_connect on v2:172.21.15.107:6835/2728005588 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9d95ffb640 1 -- 172.21.15.107:0/1587224506 <== mds.0 v2:172.21.15.107:6835/2728005588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55f4836466c0 con 0x55f483a25190 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9d95ffb640 10 client.8007 handle_command_reply: tid=0 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 client.8007 shutdown 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 msgr2=0x55f483a26570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001930 tx=0x7f9da0001a00 comp rx=0 tx=0).stop 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 shutdown_connections 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:6835/2728005588,v1:172.21.15.107:6837/2728005588] conn(0x55f483a25190 0x55f483a25570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f4839df070 0x55f483a24b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f4839de7b0 0x55f483a26570 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 --2- 172.21.15.107:0/1587224506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4839d9d70 0x55f483a26030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 >> 172.21.15.107:0/1587224506 conn(0x55f4839a0c70 msgr2=0x55f4839ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9d95ffb640 0 client.8007 ms_handle_reset on v2:172.21.15.107:6835/2728005588 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 shutdown_connections 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 1 -- 172.21.15.107:0/1587224506 wait complete. 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.686+0000 7f9dc01ab740 20 client.8007 trim_cache size 0 max 16384 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:13:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 7863, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6359740629999999, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 59, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 109, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 109, 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.080883997000001 2023-11-24T08:13:05.694 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:13:05.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 msgr2=0x7f9d9806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 0x7f9d9806e1a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9db8064320 tx=0x7f9da8009040 comp rx=0 tx=0).stop 2023-11-24T08:13:05.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 msgr2=0x7f9db80ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f9da002c450 tx=0x7f9da002e070 comp rx=0 tx=0).stop 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 shutdown_connections 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d9806bce0 0x7f9d9806e1a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db809a3e0 0x7f9db809a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db80ad3a0 0x7f9db80ad800 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 --2- 172.21.15.107:0/3363849986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9da40047c0 0x7f9db815a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 >> 172.21.15.107:0/3363849986 conn(0x7f9db80cbb40 msgr2=0x7f9db806c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 shutdown_connections 2023-11-24T08:13:05.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.687+0000 7f9db7fff640 1 -- 172.21.15.107:0/3363849986 wait complete. 2023-11-24T08:13:05.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:13:05.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.910+0000 7f5efb61c640 1 Processor -- start 2023-11-24T08:13:05.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.911+0000 7f5efb61c640 1 -- start start 2023-11-24T08:13:05.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ef415c7f0 0x7f5ef406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ef40cf450 0x7f5ef40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cdfb0 0x7f5ef40ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef40cfe40 con 0x7f5ef415c7f0 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef4156010 con 0x7f5ef40cf450 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5efb61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef406ce30 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5ef93b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ef415c7f0 0x7f5ef406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5ef93b8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ef415c7f0 0x7f5ef406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45314/0 (socket says 172.21.15.107:45314) 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.912+0000 7f5ef93b8640 1 -- 172.21.15.107:0/2758132474 learned_addr learned my addr 172.21.15.107:0/2758132474 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.913+0000 7f5ef93b8640 1 -- 172.21.15.107:0/2758132474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cdfb0 msgr2=0x7f5ef40ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.913+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/2758132474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cdfb0 0x7f5ef40ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.913+0000 7f5ef93b8640 1 -- 172.21.15.107:0/2758132474 >> v2:172.21.15.120:3300/0 conn(0x7f5ef40cf450 msgr2=0x7f5ef40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:05.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.913+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/2758132474 >> v2:172.21.15.120:3300/0 conn(0x7f5ef40cf450 0x7f5ef40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.913+0000 7f5ef93b8640 1 -- 172.21.15.107:0/2758132474 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef40cece0 con 0x7f5ef415c7f0 2023-11-24T08:13:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.914+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/2758132474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef415c7f0 0x7f5ef406db20 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4070410 tx=0x7f5ef002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c41528cb82072ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.914+0000 7f5eebfff640 1 -- 172.21.15.107:0/2758132474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0031880 con 0x7f5ef415c7f0 2023-11-24T08:13:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.914+0000 7f5eebfff640 1 -- 172.21.15.107:0/2758132474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef00319e0 con 0x7f5ef415c7f0 2023-11-24T08:13:05.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.914+0000 7f5eebfff640 1 -- 172.21.15.107:0/2758132474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0031d70 con 0x7f5ef415c7f0 2023-11-24T08:13:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 -- 172.21.15.107:0/2758132474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef415c7f0 msgr2=0x7f5ef406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 --2- 172.21.15.107:0/2758132474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef415c7f0 0x7f5ef406db20 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4070410 tx=0x7f5ef002b920 comp rx=0 tx=0).stop 2023-11-24T08:13:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 -- 172.21.15.107:0/2758132474 shutdown_connections 2023-11-24T08:13:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 --2- 172.21.15.107:0/2758132474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cdfb0 0x7f5ef40ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 --2- 172.21.15.107:0/2758132474 >> v2:172.21.15.120:3300/0 conn(0x7f5ef40cf450 0x7f5ef40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 --2- 172.21.15.107:0/2758132474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef415c7f0 0x7f5ef406db20 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 -- 172.21.15.107:0/2758132474 >> 172.21.15.107:0/2758132474 conn(0x7f5ef406a920 msgr2=0x7f5ef406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:05.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 -- 172.21.15.107:0/2758132474 shutdown_connections 2023-11-24T08:13:05.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.915+0000 7f5efb61c640 1 -- 172.21.15.107:0/2758132474 wait complete. 2023-11-24T08:13:05.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.917+0000 7f5efb61c640 1 Processor -- start 2023-11-24T08:13:05.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.917+0000 7f5efb61c640 1 -- start start 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 0x7f5ef4148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef415c7f0 0x7f5ef4145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef415eb40 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef41581b0 con 0x7f5ef415c7f0 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5efb61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef41561f0 con 0x7f5ef40cf450 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.918+0000 7f5ef93b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45324/0 (socket says 172.21.15.107:45324) 2023-11-24T08:13:05.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 -- 172.21.15.107:0/203752392 learned_addr learned my addr 172.21.15.107:0/203752392 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef8bb7640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 0x7f5ef4148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef9bb9640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef415c7f0 0x7f5ef4145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 -- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 msgr2=0x7f5ef4148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 0x7f5ef4148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 -- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef415c7f0 msgr2=0x7f5ef4145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef415c7f0 0x7f5ef4145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 -- 172.21.15.107:0/203752392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00094d0 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef8bb7640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 0x7f5ef4148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.919+0000 7f5ef93b8640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5ef002b960 tx=0x7f5ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.920+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0031850 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.920+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef00443f0 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.920+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00447a0 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.920+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef4156460 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.920+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef4148780 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.921+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef4067470 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.922+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f5ef0044900 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.923+0000 7f5ee9ffb640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 0x7f5ed806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.923+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ef00aa9b0 con 0x7f5ef40cdfb0 2023-11-24T08:13:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.923+0000 7f5ef8bb7640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 0x7f5ed806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.924+0000 7f5ef8bb7640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 0x7f5ed806e1a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4006fd0 tx=0x7f5ee4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:05.928+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ef0077540 con 0x7f5ef40cdfb0 2023-11-24T08:13:06.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.163+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5ef415fc20 con 0x7f5ef40cdfb0 2023-11-24T08:13:06.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.164+0000 7f5ee9ffb640 1 -- 172.21.15.107:0/203752392 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5ef007ab60 con 0x7f5ef40cdfb0 2023-11-24T08:13:06.166 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:13:06.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 msgr2=0x7f5ed806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:06.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 0x7f5ed806e1a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4006fd0 tx=0x7f5ee4008040 comp rx=0 tx=0).stop 2023-11-24T08:13:06.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 msgr2=0x7f5ef4145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5ef002b960 tx=0x7f5ef00046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 shutdown_connections 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ed806bce0 0x7f5ed806e1a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef415c7f0 0x7f5ef4145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef40cf450 0x7f5ef4148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 --2- 172.21.15.107:0/203752392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef40cdfb0 0x7f5ef4145100 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 >> 172.21.15.107:0/203752392 conn(0x7f5ef406a920 msgr2=0x7f5ef40af940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 shutdown_connections 2023-11-24T08:13:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:06.167+0000 7f5efb61c640 1 -- 172.21.15.107:0/203752392 wait complete. 2023-11-24T08:13:06.188 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:13:06.189 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:06.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.190 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:13:06.244 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:13:06.244 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:13:06.244 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:06.245 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:13:06.246 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:06.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:06.246 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:13:06.246 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:06.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:06.247 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:13:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:13:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41884sec preferred_lft 41884sec 2023-11-24T08:13:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:13:06.248 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.248 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:13:06.248 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:06.248 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:13:06.248 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.249 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:13:06.249 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.249 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:06.250 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:13:06.250 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:13:06.250 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.250 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:06.250 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:13:06.251 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:13:06.251 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:06.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.251 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:13:06.337 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:13:06.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.341 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:13:06.393 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:13:06.393 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:13:06.393 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:13:06.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:13:06.393 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:13:06.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.394 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:13:06.394 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:13:06.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:13:06.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:13:06.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:13:06.395 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:13:06.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.576 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:13:06.627 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:13:06.628 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:13:06.628 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:06.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:13:06.722 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:06.723 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:06.778 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:06.778 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:06.778 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:06.830 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:13:06.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:06.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:06.969 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:06.970 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:07.005 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:07.005 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:07.006 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:13:07.006 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:13:07.006 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:13:07.007 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:13:07.007 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:13:07.007 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:13:07.008 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:13:07.008 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:13:07.014 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:13:07.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:07.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.102 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:13:07.165 INFO:teuthology.orchestra.run.smithi107.stdout:client8016 2023-11-24T08:13:07.165 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:07.165 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:07.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:13:07.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.474+0000 7f318f599640 1 Processor -- start 2023-11-24T08:13:07.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.475+0000 7f318f599640 1 -- start start 2023-11-24T08:13:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.476+0000 7f318f599640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f31880ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.476+0000 7f318f599640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31880d0060 0x7f3188161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.476+0000 7f318f599640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880d1d20 con 0x7f31880cfbe0 2023-11-24T08:13:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.476+0000 7f318f599640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880b90c0 con 0x7f31880d0060 2023-11-24T08:13:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.476+0000 7f318f599640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31880d04f0 con 0x7f31880b8ce0 2023-11-24T08:13:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318dd96640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f31880cfbe0 0x7f3188161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52768/0 (socket says 172.21.15.107:52768) 2023-11-24T08:13:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318dd96640 1 -- 172.21.15.107:0/433570076 learned_addr learned my addr 172.21.15.107:0/433570076 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318ed98640 1 --2- 172.21.15.107:0/433570076 >> v2:172.21.15.120:3300/0 conn(0x7f31880d0060 0x7f3188161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318e597640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f31880ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946735097 0 0) 0x7f31880d1d20 con 0x7f31880cfbe0 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3174003520 con 0x7f31880cfbe0 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417964164 0 0) 0x7f3174003520 con 0x7f31880cfbe0 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 msgr2=0x7f31880ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f31880ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 >> v2:172.21.15.120:3300/0 conn(0x7f31880d0060 msgr2=0x7f3188161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 --2- 172.21.15.107:0/433570076 >> v2:172.21.15.120:3300/0 conn(0x7f31880d0060 0x7f3188161770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880cf2d0 con 0x7f31880cfbe0 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.477+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3178001670 con 0x7f31880cfbe0 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 >> v1:172.21.15.107:6789/0 conn(0x7f31880cfbe0 legacy=0x7f3188161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f3174003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174003ef0 0x7f31740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174004d00 0x7f3174005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31880b90c0 con 0x7f3174003ef0 2023-11-24T08:13:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31880d04f0 con 0x7f3174004d00 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317400b980 con 0x7f31880b8ce0 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318dd96640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174003ef0 0x7f31740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318e597640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f3174003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.478+0000 7f318ed98640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174004d00 0x7f3174005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318dd96640 1 -- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 msgr2=0x7f3174003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318dd96640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f3174003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318dd96640 1 -- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174004d00 msgr2=0x7f3174005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318dd96640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174004d00 0x7f3174005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318dd96640 1 -- 172.21.15.107:0/433570076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3174003520 con 0x7f3174003ef0 2023-11-24T08:13:07.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318e597640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880b8ce0 0x7f3174003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:13:07.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.479+0000 7f318ed98640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174004d00 0x7f3174005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:13:07.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.480+0000 7f318dd96640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174003ef0 0x7f31740047c0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f31740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d5fe093adab4bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:07.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.480+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317802cae0 con 0x7f3174003ef0 2023-11-24T08:13:07.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.480+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317802cc40 con 0x7f3174003ef0 2023-11-24T08:13:07.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.480+0000 7f318d595640 1 -- 172.21.15.107:0/433570076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178005af0 con 0x7f3174003ef0 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 -- 172.21.15.107:0/433570076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174003ef0 msgr2=0x7f31740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 --2- 172.21.15.107:0/433570076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174003ef0 0x7f31740047c0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f31740057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318ed98640 1 -- 172.21.15.107:0/433570076 reap_dead start 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 -- 172.21.15.107:0/433570076 shutdown_connections 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 -- 172.21.15.107:0/433570076 >> 172.21.15.107:0/433570076 conn(0x7f318809c530 msgr2=0x7f31880d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 -- 172.21.15.107:0/433570076 shutdown_connections 2023-11-24T08:13:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.481+0000 7f318f599640 1 -- 172.21.15.107:0/433570076 wait complete. 2023-11-24T08:13:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.483+0000 7f318f599640 1 Processor -- start 2023-11-24T08:13:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.483+0000 7f318f599640 1 -- start start 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 0x7f31880a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880cce80 0x7f318814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318809d8c0 con 0x7f31880cca00 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318809f760 con 0x7f3174003ef0 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318f599640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318809fa40 con 0x7f31880cce80 2023-11-24T08:13:07.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318dd96640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318dd96640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45342/0 (socket says 172.21.15.107:45342) 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.484+0000 7f318dd96640 1 -- 172.21.15.107:0/1676027175 learned_addr learned my addr 172.21.15.107:0/1676027175 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318e597640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 0x7f31880a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 -- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880cce80 msgr2=0x7f318814f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318ed98640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880cce80 0x7f318814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880cce80 0x7f318814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 -- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 msgr2=0x7f31880a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 0x7f31880a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 -- 172.21.15.107:0/1676027175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318809fd20 con 0x7f31880cca00 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318e597640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 0x7f31880a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f318dd96640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f3178001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:07.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178039070 con 0x7f31880cca00 2023-11-24T08:13:07.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178005c80 con 0x7f31880cca00 2023-11-24T08:13:07.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.485+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317803c040 con 0x7f31880cca00 2023-11-24T08:13:07.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.486+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31880a0000 con 0x7f31880cca00 2023-11-24T08:13:07.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.486+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318809d160 con 0x7f31880cca00 2023-11-24T08:13:07.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.487+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318809d320 con 0x7f31880cca00 2023-11-24T08:13:07.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.488+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f3178032070 con 0x7f31880cca00 2023-11-24T08:13:07.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.489+0000 7f317effd640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 0x7f316c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:07.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.489+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31780a9390 con 0x7f31880cca00 2023-11-24T08:13:07.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.489+0000 7f318e597640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 0x7f316c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:07.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.490+0000 7f318e597640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 0x7f316c06e1a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3184006d40 tx=0x7f3184008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.493+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3178079920 con 0x7f31880cca00 2023-11-24T08:13:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.719+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f31880727b0 con 0x7f31880cca00 2023-11-24T08:13:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.720+0000 7f317effd640 1 -- 172.21.15.107:0/1676027175 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3178046020 con 0x7f31880cca00 2023-11-24T08:13:07.722 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:13:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 msgr2=0x7f316c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 0x7f316c06e1a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3184006d40 tx=0x7f3184008040 comp rx=0 tx=0).stop 2023-11-24T08:13:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 msgr2=0x7f318809ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f3178001a00 comp rx=0 tx=0).stop 2023-11-24T08:13:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 shutdown_connections 2023-11-24T08:13:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316c06bce0 0x7f316c06e1a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31880cce80 0x7f318814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31880cca00 0x7f318809ec80 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 --2- 172.21.15.107:0/1676027175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174003ef0 0x7f31880a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 >> 172.21.15.107:0/1676027175 conn(0x7f318809c530 msgr2=0x7f318806c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 shutdown_connections 2023-11-24T08:13:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:07.723+0000 7f318f599640 1 -- 172.21.15.107:0/1676027175 wait complete. 2023-11-24T08:13:07.744 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:13:07.744 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:13:07.745 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:07.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.746 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:13:07.810 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:13:07.811 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41881sec preferred_lft 41881sec 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.812 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:07.813 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:13:07.814 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:13:07.814 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:07.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.815 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:13:07.903 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:13:07.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.907 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:13:07.962 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:13:07.962 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:13:07.962 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:13:07.963 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:13:07.963 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:13:07.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:07.963 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:13:07.963 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:13:07.963 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:13:07.964 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:13:07.964 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:13:07.964 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:13:08.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:08.153 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:13:08.200 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:13:08.200 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:13:08.201 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:08.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:08.202 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:13:08.344 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:08.344 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:08.400 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:08.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:08.400 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:08.451 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:13:08.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:08.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:08.589 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:08.589 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:08.625 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:08.626 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:08.626 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:13:08.626 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:13:08.626 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:13:08.629 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:13:08.629 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:13:08.629 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:13:08.629 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:13:08.629 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:13:08.634 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:13:08.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:08.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:08.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:08.720 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:13:08.778 INFO:teuthology.orchestra.run.smithi120.stdout:client8025 2023-11-24T08:13:08.779 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:08.779 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:08.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:13:09.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.059+0000 7f1673661640 1 Processor -- start 2023-11-24T08:13:09.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.059+0000 7f1673661640 1 -- start start 2023-11-24T08:13:09.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.060+0000 7f1673661640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f166c15c7f0 0x7f166c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.060+0000 7f1673661640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.060+0000 7f1673661640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c0cece0 con 0x7f166c0cf450 2023-11-24T08:13:09.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.060+0000 7f1673661640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0cfe40 con 0x7f166c15c7f0 2023-11-24T08:13:09.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.060+0000 7f1673661640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c156010 con 0x7f166c0ce6c0 2023-11-24T08:13:09.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f166c15c7f0 0x7f166c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f166c15c7f0 0x7f166c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41272/0 (socket says 172.21.15.107:41272) 2023-11-24T08:13:09.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 -- 172.21.15.107:0/1662494017 learned_addr learned my addr 172.21.15.107:0/1662494017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:09.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f1671bfe640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 -- 172.21.15.107:0/1662494017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 msgr2=0x7f166c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:09.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 -- 172.21.15.107:0/1662494017 >> v1:172.21.15.120:6789/0 conn(0x7f166c0cf450 legacy=0x7f166c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:09.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.061+0000 7f16713fd640 1 -- 172.21.15.107:0/1662494017 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c06d040 con 0x7f166c15c7f0 2023-11-24T08:13:09.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.062+0000 7f16713fd640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c15c7f0 0x7f166c06db20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f1668009870 tx=0x7f166802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e56b08ba18f0e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1663fff640 1 -- 172.21.15.107:0/1662494017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668031640 con 0x7f166c15c7f0 2023-11-24T08:13:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1663fff640 1 -- 172.21.15.107:0/1662494017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16680317a0 con 0x7f166c15c7f0 2023-11-24T08:13:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1663fff640 1 -- 172.21.15.107:0/1662494017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668031b30 con 0x7f166c15c7f0 2023-11-24T08:13:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 -- 172.21.15.107:0/1662494017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c15c7f0 msgr2=0x7f166c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c15c7f0 0x7f166c06db20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f1668009870 tx=0x7f166802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 -- 172.21.15.107:0/1662494017 shutdown_connections 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 --2- 172.21.15.107:0/1662494017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c15c7f0 0x7f166c06db20 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 -- 172.21.15.107:0/1662494017 >> 172.21.15.107:0/1662494017 conn(0x7f166c06a920 msgr2=0x7f166c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 -- 172.21.15.107:0/1662494017 shutdown_connections 2023-11-24T08:13:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.063+0000 7f1673661640 1 -- 172.21.15.107:0/1662494017 wait complete. 2023-11-24T08:13:09.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.065+0000 7f1673661640 1 Processor -- start 2023-11-24T08:13:09.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.065+0000 7f1673661640 1 -- start start 2023-11-24T08:13:09.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f166c15c7f0 0x7f166c156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0ad460 con 0x7f166c0cf450 2023-11-24T08:13:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0ac5b0 con 0x7f166c15c7f0 2023-11-24T08:13:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1673661640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0ac850 con 0x7f166c0ce6c0 2023-11-24T08:13:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.066+0000 7f1670bfc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41288/0 (socket says 172.21.15.107:41288) 2023-11-24T08:13:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 -- 172.21.15.107:0/1732046128 learned_addr learned my addr 172.21.15.107:0/1732046128 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 -- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 msgr2=0x7f166c0cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1671bfe640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f166c15c7f0 0x7f166c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 -- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f166c15c7f0 msgr2=0x7f166c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f166c15c7f0 0x7f166c156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 -- 172.21.15.107:0/1732046128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680094d0 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1670bfc640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c430 tx=0x7f165c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.067+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c011790 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.068+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c0118f0 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.068+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c0acb30 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.068+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c011ce0 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.068+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c0ad0f0 con 0x7f166c0cf450 2023-11-24T08:13:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.069+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c067470 con 0x7f166c0cf450 2023-11-24T08:13:09.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.070+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f165c01c460 con 0x7f166c0cf450 2023-11-24T08:13:09.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.071+0000 7f1661ffb640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 0x7f165006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.072+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f165c088390 con 0x7f166c0cf450 2023-11-24T08:13:09.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.072+0000 7f16713fd640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 0x7f165006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.072+0000 7f16713fd640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 0x7f165006e1a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1668036000 tx=0x7f1668030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.076+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f165c054b30 con 0x7f166c0cf450 2023-11-24T08:13:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.304+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f166c0cd730 con 0x7f166c0cf450 2023-11-24T08:13:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.358+0000 7f1661ffb640 1 -- 172.21.15.107:0/1732046128 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 136+0+0 (secure 0 0 0) 0x7f165c058150 con 0x7f166c0cf450 2023-11-24T08:13:09.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 msgr2=0x7f165006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 0x7f165006e1a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1668036000 tx=0x7f1668030000 comp rx=0 tx=0).stop 2023-11-24T08:13:09.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 msgr2=0x7f166c1f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c430 tx=0x7f165c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 shutdown_connections 2023-11-24T08:13:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f165006bce0 0x7f165006e1a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f166c15c7f0 0x7f166c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f166c0cf450 0x7f166c1f2b30 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 --2- 172.21.15.107:0/1732046128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f166c0ce6c0 0x7f166c0cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 >> 172.21.15.107:0/1732046128 conn(0x7f166c06a920 msgr2=0x7f166c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 shutdown_connections 2023-11-24T08:13:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.361+0000 7f1673661640 1 -- 172.21.15.107:0/1732046128 wait complete. 2023-11-24T08:13:09.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:13:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.593+0000 7f033f59e640 1 Processor -- start 2023-11-24T08:13:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.594+0000 7f033f59e640 1 -- start start 2023-11-24T08:13:09.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.595+0000 7f033f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f0340071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.595+0000 7f033f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400ca0e0 con 0x7f03400b1250 2023-11-24T08:13:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.595+0000 7f033f59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0340072800 con 0x7f0340070550 2023-11-24T08:13:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.595+0000 7f033f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03400c97b0 con 0x7f034015b1d0 2023-11-24T08:13:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03400b1250 0x7f034015edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46176/0 (socket says 172.21.15.107:46176) 2023-11-24T08:13:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033ed9d640 1 -- 172.21.15.107:0/780065513 learned_addr learned my addr 172.21.15.107:0/780065513 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033e59c640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f0340071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804268197 0 0) 0x7f03400ca0e0 con 0x7f03400b1250 2023-11-24T08:13:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c003520 con 0x7f03400b1250 2023-11-24T08:13:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041869816 0 0) 0x7f032c003520 con 0x7f03400b1250 2023-11-24T08:13:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 msgr2=0x7f0340071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f0340071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 >> v1:172.21.15.120:6789/0 conn(0x7f0340070550 legacy=0x7f03400b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400c9a90 con 0x7f03400b1250 2023-11-24T08:13:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f03380019f0 con 0x7f03400b1250 2023-11-24T08:13:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.596+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 >> v1:172.21.15.107:6789/0 conn(0x7f03400b1250 legacy=0x7f034015edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f032c0039a0 0x7f032c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0047c0 0x7f032c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f032c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03400c97b0 con 0x7f032c0039a0 2023-11-24T08:13:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c00ba30 con 0x7f032c0047c0 2023-11-24T08:13:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c00bbf0 con 0x7f034015b1d0 2023-11-24T08:13:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f032c0039a0 0x7f032c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033ed9d640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f032c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f0337fff640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0047c0 0x7f032c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 -- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 msgr2=0x7f032c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015b1d0 0x7f032c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 -- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0047c0 msgr2=0x7f032c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0047c0 0x7f032c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f033e59c640 1 -- 172.21.15.107:0/780065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c003520 con 0x7f032c0039a0 2023-11-24T08:13:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.597+0000 7f0337fff640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0047c0 0x7f032c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.598+0000 7f033e59c640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f032c0039a0 0x7f032c003dc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0330009a90 tx=0x7f032c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c96d3fb32524ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033002c660 con 0x7f032c0039a0 2023-11-24T08:13:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033002c7c0 con 0x7f032c0039a0 2023-11-24T08:13:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033dd9b640 1 -- 172.21.15.107:0/780065513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033002cb50 con 0x7f032c0039a0 2023-11-24T08:13:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033f59e640 1 -- 172.21.15.107:0/780065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f032c0039a0 msgr2=0x7f032c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033f59e640 1 --2- 172.21.15.107:0/780065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f032c0039a0 0x7f032c003dc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0330009a90 tx=0x7f032c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033ed9d640 1 -- 172.21.15.107:0/780065513 reap_dead start 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033f59e640 1 -- 172.21.15.107:0/780065513 shutdown_connections 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.599+0000 7f033f59e640 1 -- 172.21.15.107:0/780065513 >> 172.21.15.107:0/780065513 conn(0x7f03400cbb40 msgr2=0x7f03400cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.600+0000 7f033f59e640 1 -- 172.21.15.107:0/780065513 shutdown_connections 2023-11-24T08:13:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.600+0000 7f033f59e640 1 -- 172.21.15.107:0/780065513 wait complete. 2023-11-24T08:13:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.601+0000 7f033f59e640 1 Processor -- start 2023-11-24T08:13:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.601+0000 7f033f59e640 1 -- start start 2023-11-24T08:13:09.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 0x7f0340153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015d900 0x7f034015dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034015e2c0 con 0x7f03400c8450 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340153fc0 con 0x7f032c0039a0 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.602+0000 7f033f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03400c8ed0 con 0x7f034015d900 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41298/0 (socket says 172.21.15.107:41298) 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 -- 172.21.15.107:0/36041036 learned_addr learned my addr 172.21.15.107:0/36041036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f033e59c640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 0x7f0340153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 -- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015d900 msgr2=0x7f034015dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f033ed9d640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015d900 0x7f034015dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015d900 0x7f034015dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 -- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 msgr2=0x7f0340153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 0x7f0340153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 -- 172.21.15.107:0/36041036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400b1ce0 con 0x7f03400c8450 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f033e59c640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 0x7f0340153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.603+0000 7f0337fff640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f032800d7a0 tx=0x7f032800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.604+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328017070 con 0x7f03400c8450 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.604+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0328005070 con 0x7f03400c8450 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.604+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328005420 con 0x7f03400c8450 2023-11-24T08:13:09.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.604+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03400b1fc0 con 0x7f03400c8450 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.604+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03400b2580 con 0x7f03400c8450 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.605+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034004e840 con 0x7f03400c8450 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.606+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f0328007030 con 0x7f03400c8450 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.607+0000 7f0336ffd640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 0x7f032006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.607+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0328087d30 con 0x7f03400c8450 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.607+0000 7f033e59c640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 0x7f032006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.608+0000 7f033e59c640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 0x7f032006e1d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0330036000 tx=0x7f0330030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.611+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0328054550 con 0x7f03400c8450 2023-11-24T08:13:09.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.820+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f03400ca570 con 0x7f032006bd10 2023-11-24T08:13:09.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.822+0000 7f0336ffd640 1 -- 172.21.15.107:0/36041036 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f03300041e0 con 0x7f032006bd10 2023-11-24T08:13:09.824 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:13:09.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:09.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:13:09.824 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:09.825 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:13:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 msgr2=0x7f032006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 0x7f032006e1d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0330036000 tx=0x7f0330030000 comp rx=0 tx=0).stop 2023-11-24T08:13:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 msgr2=0x7f03400c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f032800d7a0 tx=0x7f032800dc60 comp rx=0 tx=0).stop 2023-11-24T08:13:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 shutdown_connections 2023-11-24T08:13:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f032006bd10 0x7f032006e1d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f034015d900 0x7f034015dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03400c8450 0x7f03400c88d0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 --2- 172.21.15.107:0/36041036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f032c0039a0 0x7f0340153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 >> 172.21.15.107:0/36041036 conn(0x7f03400cbb40 msgr2=0x7f03400af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 shutdown_connections 2023-11-24T08:13:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:09.825+0000 7f033f59e640 1 -- 172.21.15.107:0/36041036 wait complete. 2023-11-24T08:13:09.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:13:10.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.054+0000 7f221f2cf640 1 Processor -- start 2023-11-24T08:13:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.054+0000 7f221f2cf640 1 -- start start 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221f2cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22181684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221f2cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218161a10 con 0x7f221806e160 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221f2cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218161cf0 con 0x7f2218146eb0 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221f2cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2218161fd0 con 0x7f22180cd2e0 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221c86a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f221806e160 0x7f22180ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46190/0 (socket says 172.21.15.107:46190) 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.056+0000 7f221c86a640 1 -- 172.21.15.107:0/1984888653 learned_addr learned my addr 172.21.15.107:0/1984888653 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f221d86c640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22181684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356559262 0 0) 0x7f2218161a10 con 0x7f221806e160 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204003520 con 0x7f221806e160 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196292341 0 0) 0x7f2204003520 con 0x7f221806e160 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 msgr2=0x7f22181684e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22181684e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 >> v1:172.21.15.120:6789/0 conn(0x7f2218146eb0 legacy=0x7f2218161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218070660 con 0x7f221806e160 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2208001690 con 0x7f221806e160 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 >> v1:172.21.15.107:6789/0 conn(0x7f221806e160 legacy=0x7f22180ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f22040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2204004d00 0x7f2204005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218161fd0 con 0x7f2204003f20 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220400ba60 con 0x7f2204004d00 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.057+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220400bc20 con 0x7f22180cd2e0 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f22040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221d86c640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2204004d00 0x7f2204005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221d06b640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 -- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 msgr2=0x7f22040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 -- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2204004d00 msgr2=0x7f2204005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2204004d00 0x7f2204005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221c86a640 1 -- 172.21.15.107:0/1984888653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204003520 con 0x7f2204003f20 2023-11-24T08:13:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.058+0000 7f221d06b640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180cd2e0 0x7f22040039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.059+0000 7f221c86a640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f22040047c0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f220800c210 tx=0x7f22040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eb6f480e025acd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.059+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208005ad0 con 0x7f2204003f20 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.059+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208005c30 con 0x7f2204003f20 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.059+0000 7f220ffff640 1 -- 172.21.15.107:0/1984888653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220802c870 con 0x7f2204003f20 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 -- 172.21.15.107:0/1984888653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 msgr2=0x7f22040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1984888653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f22040047c0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f220800c210 tx=0x7f22040057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221d86c640 1 -- 172.21.15.107:0/1984888653 reap_dead start 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 -- 172.21.15.107:0/1984888653 shutdown_connections 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 -- 172.21.15.107:0/1984888653 >> 172.21.15.107:0/1984888653 conn(0x7f22180b0c50 msgr2=0x7f22180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 -- 172.21.15.107:0/1984888653 shutdown_connections 2023-11-24T08:13:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.060+0000 7f221f2cf640 1 -- 172.21.15.107:0/1984888653 wait complete. 2023-11-24T08:13:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.062+0000 7f221f2cf640 1 Processor -- start 2023-11-24T08:13:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.062+0000 7f221f2cf640 1 -- start start 2023-11-24T08:13:10.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180b3a50 0x7f22180c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f221815cd70 0x7f221815d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221815d730 con 0x7f2204003f20 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22181f7620 con 0x7f221815cd70 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221f2cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22181f6860 con 0x7f22180b3a50 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221d06b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221d06b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41314/0 (socket says 172.21.15.107:41314) 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.063+0000 7f221d06b640 1 -- 172.21.15.107:0/1797982138 learned_addr learned my addr 172.21.15.107:0/1797982138 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221c86a640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180b3a50 0x7f22180c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d86c640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f221815cd70 0x7f221815d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 -- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180b3a50 msgr2=0x7f22180c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180b3a50 0x7f22180c5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 -- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f221815cd70 msgr2=0x7f221815d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f221815cd70 0x7f221815d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 -- 172.21.15.107:0/1797982138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22181f6b40 con 0x7f2204003f20 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221d06b640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f221400d7a0 tx=0x7f221400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214017070 con 0x7f2204003f20 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2214005070 con 0x7f2204003f20 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214005420 con 0x7f2204003f20 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22181f6e20 con 0x7f2204003f20 2023-11-24T08:13:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.064+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22180b4ad0 con 0x7f2204003f20 2023-11-24T08:13:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.065+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22180cb560 con 0x7f2204003f20 2023-11-24T08:13:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.066+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7f2214007030 con 0x7f2204003f20 2023-11-24T08:13:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.067+0000 7f220dffb640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 0x7f21fc06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.067+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2214087cd0 con 0x7f2204003f20 2023-11-24T08:13:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.067+0000 7f221c86a640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 0x7f21fc06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.068+0000 7f221c86a640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 0x7f21fc06e1a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2208034000 tx=0x7f2208037040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.072+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2214054550 con 0x7f2204003f20 2023-11-24T08:13:10.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.262+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f221804e8b0 con 0x7f2204003f20 2023-11-24T08:13:10.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.263+0000 7f220dffb640 1 -- 172.21.15.107:0/1797982138 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2214057b70 con 0x7f2204003f20 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 msgr2=0x7f21fc06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 0x7f21fc06e1a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2208034000 tx=0x7f2208037040 comp rx=0 tx=0).stop 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 msgr2=0x7f2218143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f221400d7a0 tx=0x7f221400dc60 comp rx=0 tx=0).stop 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 shutdown_connections 2023-11-24T08:13:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21fc06bce0 0x7f21fc06e1a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f221815cd70 0x7f221815d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22180b3a50 0x7f22180c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 --2- 172.21.15.107:0/1797982138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2204003f20 0x7f2218143900 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 >> 172.21.15.107:0/1797982138 conn(0x7f22180b0c50 msgr2=0x7f22180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 shutdown_connections 2023-11-24T08:13:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.266+0000 7f221f2cf640 1 -- 172.21.15.107:0/1797982138 wait complete. 2023-11-24T08:13:10.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol6314 2023-11-24T08:13:10.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.502+0000 7fa5d429b640 1 Processor -- start 2023-11-24T08:13:10.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.502+0000 7fa5d429b640 1 -- start start 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa5cc0cbf20 0x7fa5cc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc15b820 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cc3a0 con 0x7fa5cc0cbf20 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d429b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc15eea0 con 0x7fa5cc15b440 2023-11-24T08:13:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d2838640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d2838640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41330/0 (socket says 172.21.15.107:41330) 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.504+0000 7fa5d2838640 1 -- 172.21.15.107:0/1164174452 learned_addr learned my addr 172.21.15.107:0/1164174452 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d1836640 1 --2- 172.21.15.107:0/1164174452 >> v2:172.21.15.120:3300/0 conn(0x7fa5cc0cbf20 0x7fa5cc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d2838640 1 -- 172.21.15.107:0/1164174452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 msgr2=0x7fa5cc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d2838640 1 --2- 172.21.15.107:0/1164174452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d2838640 1 -- 172.21.15.107:0/1164174452 >> v2:172.21.15.120:3300/0 conn(0x7fa5cc0cbf20 msgr2=0x7fa5cc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d2838640 1 --2- 172.21.15.107:0/1164174452 >> v2:172.21.15.120:3300/0 conn(0x7fa5cc0cbf20 0x7fa5cc0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.505+0000 7fa5d2838640 1 -- 172.21.15.107:0/1164174452 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0abf30 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.506+0000 7fa5d2838640 1 --2- 172.21.15.107:0/1164174452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0cee60 tx=0x7fa5c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec23f4fea8c6d16c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.506+0000 7fa5d1035640 1 -- 172.21.15.107:0/1164174452 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c402c9f0 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.506+0000 7fa5d1035640 1 -- 172.21.15.107:0/1164174452 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c402cb50 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.506+0000 7fa5d1035640 1 -- 172.21.15.107:0/1164174452 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c40343f0 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 -- 172.21.15.107:0/1164174452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 msgr2=0x7fa5cc0c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 --2- 172.21.15.107:0/1164174452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0cee60 tx=0x7fa5c402be50 comp rx=0 tx=0).stop 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 -- 172.21.15.107:0/1164174452 shutdown_connections 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 --2- 172.21.15.107:0/1164174452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc0c3ec0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 --2- 172.21.15.107:0/1164174452 >> v2:172.21.15.120:3300/0 conn(0x7fa5cc0cbf20 0x7fa5cc0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 --2- 172.21.15.107:0/1164174452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 -- 172.21.15.107:0/1164174452 >> 172.21.15.107:0/1164174452 conn(0x7fa5cc0b0c50 msgr2=0x7fa5cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 -- 172.21.15.107:0/1164174452 shutdown_connections 2023-11-24T08:13:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.507+0000 7fa5d429b640 1 -- 172.21.15.107:0/1164174452 wait complete. 2023-11-24T08:13:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.509+0000 7fa5d429b640 1 Processor -- start 2023-11-24T08:13:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.509+0000 7fa5d429b640 1 -- start start 2023-11-24T08:13:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 0x7fa5cc155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0afc90 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0af330 con 0x7fa5cc0cbf20 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d429b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc152fa0 con 0x7fa5cc15b440 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d2037640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d2037640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41340/0 (socket says 172.21.15.107:41340) 2023-11-24T08:13:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.510+0000 7fa5d2037640 1 -- 172.21.15.107:0/2874692535 learned_addr learned my addr 172.21.15.107:0/2874692535 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d1836640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 0x7fa5cc155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2838640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 -- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 msgr2=0x7fa5cc156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 -- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 msgr2=0x7fa5cc155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 0x7fa5cc155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 -- 172.21.15.107:0/2874692535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4009b30 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d1836640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 0x7fa5cc155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d2037640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b410 tx=0x7fa5c800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011830 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c80119b0 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011d60 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.511+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc0af5d0 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.512+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc148050 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.513+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc04e8b0 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.514+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fa5c801a400 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.515+0000 7fa5c2ffd640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 0x7fa5b006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.515+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5c8088500 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.515+0000 7fa5d1836640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 0x7fa5b006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.516+0000 7fa5d1836640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 0x7fa5b006e1a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc151c20 tx=0x7fa5bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:10.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.519+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5c8054d80 con 0x7fa5cc0c57e0 2023-11-24T08:13:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.736+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol6314", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5cc15f6b0 con 0x7fa5b006bce0 2023-11-24T08:13:10.753 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:10.751+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:10.753 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:10.751+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:10.753 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:10.751+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:10.753 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:10.751+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:10.753 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:10.751+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:10.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.817+0000 7fa5c2ffd640 1 -- 172.21.15.107:0/2874692535 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa5bc00c040 con 0x7fa5b006bce0 2023-11-24T08:13:10.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.820+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 msgr2=0x7fa5b006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.820+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 0x7fa5b006e1a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc151c20 tx=0x7fa5bc008040 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.820+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 msgr2=0x7fa5cc147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.820+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b410 tx=0x7fa5c800b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 shutdown_connections 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5b006bce0 0x7fa5b006e1a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5cc15b440 0x7fa5cc156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5cc0cbf20 0x7fa5cc155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 --2- 172.21.15.107:0/2874692535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc0c57e0 0x7fa5cc147b10 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 >> 172.21.15.107:0/2874692535 conn(0x7fa5cc0b0c50 msgr2=0x7fa5cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 shutdown_connections 2023-11-24T08:13:10.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:10.821+0000 7fa5d429b640 1 -- 172.21.15.107:0/2874692535 wait complete. 2023-11-24T08:13:10.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol6314 2023-11-24T08:13:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.045+0000 7fa0bcf97640 1 Processor -- start 2023-11-24T08:13:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.045+0000 7fa0bcf97640 1 -- start start 2023-11-24T08:13:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.046+0000 7fa0bcf97640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.046+0000 7fa0bcf97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80c57e0 0x7fa0b806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.046+0000 7fa0bcf97640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b815eea0 con 0x7fa0b815b440 2023-11-24T08:13:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.046+0000 7fa0bcf97640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b815b820 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.046+0000 7fa0bcf97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80abf30 con 0x7fa0b80c57e0 2023-11-24T08:13:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41384/0 (socket says 172.21.15.107:41384) 2023-11-24T08:13:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 -- 172.21.15.107:0/1915194224 learned_addr learned my addr 172.21.15.107:0/1915194224 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b6d9d640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80c57e0 0x7fa0b806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 -- 172.21.15.107:0/1915194224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80c57e0 msgr2=0x7fa0b806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80c57e0 0x7fa0b806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 -- 172.21.15.107:0/1915194224 >> v1:172.21.15.120:6789/0 conn(0x7fa0b815b440 legacy=0x7fa0b80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.047+0000 7fa0b5d9b640 1 -- 172.21.15.107:0/1915194224 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b815f0f0 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.048+0000 7fa0b5d9b640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009870 tx=0x7fa0ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89430b22fc53c37 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0b559a640 1 -- 172.21.15.107:0/1915194224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0317c0 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0b559a640 1 -- 172.21.15.107:0/1915194224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac031920 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0b559a640 1 -- 172.21.15.107:0/1915194224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac031cb0 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 -- 172.21.15.107:0/1915194224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80cc0f0 msgr2=0x7fa0b80c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009870 tx=0x7fa0ac02b860 comp rx=0 tx=0).stop 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 -- 172.21.15.107:0/1915194224 shutdown_connections 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80c57e0 0x7fa0b806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 --2- 172.21.15.107:0/1915194224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80cc0f0 0x7fa0b80c52a0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 -- 172.21.15.107:0/1915194224 >> 172.21.15.107:0/1915194224 conn(0x7fa0b80b0c50 msgr2=0x7fa0b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.049+0000 7fa0bcf97640 1 -- 172.21.15.107:0/1915194224 shutdown_connections 2023-11-24T08:13:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.050+0000 7fa0bcf97640 1 -- 172.21.15.107:0/1915194224 wait complete. 2023-11-24T08:13:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.051+0000 7fa0bcf97640 1 Processor -- start 2023-11-24T08:13:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.051+0000 7fa0bcf97640 1 -- start start 2023-11-24T08:13:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80c57e0 0x7fa0b80b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80cc0f0 0x7fa0b8148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b815b440 0x7fa0b80ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80af9d0 con 0x7fa0b80c57e0 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8149060 con 0x7fa0b815b440 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0bcf97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8158c60 con 0x7fa0b80cc0f0 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.052+0000 7fa0b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80c57e0 0x7fa0b80b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80c57e0 0x7fa0b80b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41396/0 (socket says 172.21.15.107:41396) 2023-11-24T08:13:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 -- 172.21.15.107:0/793456295 learned_addr learned my addr 172.21.15.107:0/793456295 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 -- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80cc0f0 msgr2=0x7fa0b8148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b5d9b640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80cc0f0 0x7fa0b8148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b80cc0f0 0x7fa0b8148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 -- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b815b440 msgr2=0x7fa0b80ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b6d9d640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b815b440 0x7fa0b80ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b815b440 0x7fa0b80ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 -- 172.21.15.107:0/793456295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac0094d0 con 0x7fa0b80c57e0 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b6d9d640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b815b440 0x7fa0b80ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0b659c640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b80c57e0 0x7fa0b80b06d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800b3f0 tx=0x7fa0a800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.053+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8011830 con 0x7fa0b80c57e0 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.054+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a8011990 con 0x7fa0b80c57e0 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.054+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8011d40 con 0x7fa0b80c57e0 2023-11-24T08:13:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.054+0000 7fa0bcf97640 1 -- 172.21.15.107:0/793456295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b80aeeb0 con 0x7fa0b80c57e0 2023-11-24T08:13:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.054+0000 7fa0bcf97640 1 -- 172.21.15.107:0/793456295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b80af410 con 0x7fa0b80c57e0 2023-11-24T08:13:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.055+0000 7fa0bcf97640 1 -- 172.21.15.107:0/793456295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b8158a10 con 0x7fa0b80c57e0 2023-11-24T08:13:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.056+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fa0a801a400 con 0x7fa0b80c57e0 2023-11-24T08:13:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.057+0000 7fa0a6ffd640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa09806bce0 0x7fa09806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.057+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0a8087f70 con 0x7fa0b80c57e0 2023-11-24T08:13:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.057+0000 7fa0b5d9b640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa09806bce0 0x7fa09806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.058+0000 7fa0b5d9b640 1 --2- 172.21.15.107:0/793456295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa09806bce0 0x7fa09806e1a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac036000 tx=0x7fa0ac03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.061+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0a8054a70 con 0x7fa0b80c57e0 2023-11-24T08:13:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.288+0000 7fa0bcf97640 1 -- 172.21.15.107:0/793456295 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol6314", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b815f990 con 0x7fa09806bce0 2023-11-24T08:13:11.298 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.296+0000 7f376379e640 -1 mgr.server reply reply (95) Operation not supported operation 'clone-status' is not allowed on subvolume 'subvol6314' of type subvolume 2023-11-24T08:13:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.296+0000 7fa0a6ffd640 1 -- 172.21.15.107:0/793456295 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: -95 operation 'clone-status' is not allowed on subvolume 'subvol6314' of type subvolume) v1 ==== 91+0+0 (secure 0 0 0) 0x7fa0ac031b90 con 0x7fa09806bce0 2023-11-24T08:13:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENOTSUP: operation 'clone-status' is not allowed on subvolume 'subvol6314' of type subvolume 2023-11-24T08:13:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 95 2023-11-24T08:13:11.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol6314 2023-11-24T08:13:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.531+0000 7fe764f90640 1 Processor -- start 2023-11-24T08:13:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.531+0000 7fe764f90640 1 -- start start 2023-11-24T08:13:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe764f90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe76015b2f0 0x7fe7600aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe764f90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe76006b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe764f90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600b4eb0 con 0x7fe7600b58a0 2023-11-24T08:13:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe764f90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7600b5620 con 0x7fe76015b2f0 2023-11-24T08:13:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe764f90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600cbf10 con 0x7fe76015b6d0 2023-11-24T08:13:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe75ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe7600b58a0 0x7fe7600ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46194/0 (socket says 172.21.15.107:46194) 2023-11-24T08:13:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe75ed9d640 1 -- 172.21.15.107:0/273343221 learned_addr learned my addr 172.21.15.107:0/273343221 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe757fff640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe76006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe75e59c640 1 --2- 172.21.15.107:0/273343221 >> v2:172.21.15.120:3300/0 conn(0x7fe76015b2f0 0x7fe7600aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.533+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561398390 0 0) 0x7fe7600b4eb0 con 0x7fe7600b58a0 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c003520 con 0x7fe7600b58a0 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1112747004 0 0) 0x7fe74c003520 con 0x7fe7600b58a0 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 msgr2=0x7fe76006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe76006b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 >> v2:172.21.15.120:3300/0 conn(0x7fe76015b2f0 msgr2=0x7fe7600aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 --2- 172.21.15.107:0/273343221 >> v2:172.21.15.120:3300/0 conn(0x7fe76015b2f0 0x7fe7600aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600cc1f0 con 0x7fe7600b58a0 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe7580019f0 con 0x7fe7600b58a0 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 >> v1:172.21.15.107:6789/0 conn(0x7fe7600b58a0 legacy=0x7fe7600ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe74c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe74c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600b5620 con 0x7fe74c003970 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600cbf10 con 0x7fe74c0047c0 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c00ba00 con 0x7fe76015b6d0 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.534+0000 7fe75e59c640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe74c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75ed9d640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe74c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75e59c640 1 -- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 msgr2=0x7fe74c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe757fff640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75e59c640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe76015b6d0 0x7fe74c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75e59c640 1 -- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c0047c0 msgr2=0x7fe74c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75e59c640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.535+0000 7fe75e59c640 1 -- 172.21.15.107:0/273343221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c003520 con 0x7fe74c003970 2023-11-24T08:13:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.536+0000 7fe75e59c640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe74c003d90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe750009ac0 tx=0x7fe74c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a682d0409263ec22 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.536+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75002c710 con 0x7fe74c003970 2023-11-24T08:13:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.536+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75002c870 con 0x7fe74c003970 2023-11-24T08:13:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.536+0000 7fe75dd9b640 1 -- 172.21.15.107:0/273343221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75002cc00 con 0x7fe74c003970 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 -- 172.21.15.107:0/273343221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 msgr2=0x7fe74c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 --2- 172.21.15.107:0/273343221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe74c003d90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe750009ac0 tx=0x7fe74c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe75ed9d640 1 -- 172.21.15.107:0/273343221 reap_dead start 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 -- 172.21.15.107:0/273343221 shutdown_connections 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 -- 172.21.15.107:0/273343221 >> 172.21.15.107:0/273343221 conn(0x7fe760070fa0 msgr2=0x7fe7600713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 -- 172.21.15.107:0/273343221 shutdown_connections 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.537+0000 7fe764f90640 1 -- 172.21.15.107:0/273343221 wait complete. 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.539+0000 7fe764f90640 1 Processor -- start 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.539+0000 7fe764f90640 1 -- start start 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7600b4020 0x7fe7600ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe76015dcf0 0x7fe76015e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76015e6c0 con 0x7fe74c003970 2023-11-24T08:13:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600ae590 con 0x7fe76015dcf0 2023-11-24T08:13:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe764f90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600b2ca0 con 0x7fe7600b4020 2023-11-24T08:13:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe75e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe75e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41410/0 (socket says 172.21.15.107:41410) 2023-11-24T08:13:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.540+0000 7fe75e59c640 1 -- 172.21.15.107:0/85687264 learned_addr learned my addr 172.21.15.107:0/85687264 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 -- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7600b4020 msgr2=0x7fe7600ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7600b4020 0x7fe7600ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 -- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe76015dcf0 msgr2=0x7fe76015e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe76015dcf0 0x7fe76015e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 -- 172.21.15.107:0/85687264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600b2f80 con 0x7fe74c003970 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe75e59c640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe750037040 tx=0x7fe750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750003810 con 0x7fe74c003970 2023-11-24T08:13:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750003970 con 0x7fe74c003970 2023-11-24T08:13:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7600b3260 con 0x7fe74c003970 2023-11-24T08:13:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750003d00 con 0x7fe74c003970 2023-11-24T08:13:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.541+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7600b3820 con 0x7fe74c003970 2023-11-24T08:13:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.542+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76004e8b0 con 0x7fe74c003970 2023-11-24T08:13:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.544+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) v1 ==== 90557+0+0 (secure 0 0 0) 0x7fe750040450 con 0x7fe74c003970 2023-11-24T08:13:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.544+0000 7fe756ffd640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 0x7fe74006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.545+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7500aa270 con 0x7fe74c003970 2023-11-24T08:13:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.545+0000 7fe757fff640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 0x7fe74006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.545+0000 7fe757fff640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 0x7fe74006e1d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe7600b46a0 tx=0x7fe748008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.549+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe750076bb0 con 0x7fe74c003970 2023-11-24T08:13:11.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:13:11.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:13:11.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.767+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol6314", "target": ["mon-mgr", ""]}) v1 -- 0x7fe760155c00 con 0x7fe74006bd10 2023-11-24T08:13:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.789+0000 7fe756ffd640 1 -- 172.21.15.107:0/85687264 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe760155c00 con 0x7fe74006bd10 2023-11-24T08:13:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 msgr2=0x7fe74006e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 0x7fe74006e1d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe7600b46a0 tx=0x7fe748008040 comp rx=0 tx=0).stop 2023-11-24T08:13:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 msgr2=0x7fe760151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe750037040 tx=0x7fe750004730 comp rx=0 tx=0).stop 2023-11-24T08:13:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 shutdown_connections 2023-11-24T08:13:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe74006bd10 0x7fe74006e1d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe76015dcf0 0x7fe76015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7600b4020 0x7fe7600ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 --2- 172.21.15.107:0/85687264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe74c003970 0x7fe760151dc0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:11.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 >> 172.21.15.107:0/85687264 conn(0x7fe760070fa0 msgr2=0x7fe7600cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:11.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.793+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 shutdown_connections 2023-11-24T08:13:11.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:11.794+0000 7fe764f90640 1 -- 172.21.15.107:0/85687264 wait complete. 2023-11-24T08:13:11.799 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.798+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.800 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.798+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.800 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.798+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.800 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.798+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.800 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.798+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:11.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:13:11.818 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.815+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.818 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.815+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.818 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.815+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.818 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.815+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.818 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:11.815+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:11.908 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:13:11.910 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:13:11.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:13:12.009 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:13:12.133 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:13:12.134 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:12.134 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:12.194 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:12.194 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:12.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.469+0000 7fec698cf640 1 Processor -- start 2023-11-24T08:13:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.469+0000 7fec698cf640 1 -- start start 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec698cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec6406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec698cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec640ce6c0 0x7fec640cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec698cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640cece0 con 0x7fec640cf450 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec698cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec64156010 con 0x7fec640ce6c0 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec698cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6406ce30 con 0x7fec6415c7f0 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec5a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fec640cf450 0x7fec640cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46210/0 (socket says 172.21.15.107:46210) 2023-11-24T08:13:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec5a7fc640 1 -- 172.21.15.107:0/2207394126 learned_addr learned my addr 172.21.15.107:0/2207394126 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec62ffd640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec6406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.471+0000 7fec637fe640 1 --2- 172.21.15.107:0/2207394126 >> v2:172.21.15.120:3300/0 conn(0x7fec640ce6c0 0x7fec640cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029376305 0 0) 0x7fec640cece0 con 0x7fec640cf450 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50003520 con 0x7fec640cf450 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55472255 0 0) 0x7fec50003520 con 0x7fec640cf450 2023-11-24T08:13:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 msgr2=0x7fec6406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec6406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 >> v2:172.21.15.120:3300/0 conn(0x7fec640ce6c0 msgr2=0x7fec640cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 --2- 172.21.15.107:0/2207394126 >> v2:172.21.15.120:3300/0 conn(0x7fec640ce6c0 0x7fec640cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6406d040 con 0x7fec640cf450 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fec4c001670 con 0x7fec640cf450 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 >> v1:172.21.15.107:6789/0 conn(0x7fec640cf450 legacy=0x7fec640cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec50003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003ef0 0x7fec500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 0x7fec50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec64156010 con 0x7fec50003ef0 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6406ce30 con 0x7fec50004d00 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5000b9d0 con 0x7fec6415c7f0 2023-11-24T08:13:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec637fe640 1 --2- 172.21.15.107:0/2207394126 >> v2:172.21.15.120:3300/0 conn(0x7fec640ce6c0 0x7fec640cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.472+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003ef0 0x7fec500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec62ffd640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec50003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec5a7fc640 1 -- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 msgr2=0x7fec50003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec637fe640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 0x7fec50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec50003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec5a7fc640 1 -- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 msgr2=0x7fec50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 0x7fec50005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec5a7fc640 1 -- 172.21.15.107:0/2207394126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50003520 con 0x7fec50003ef0 2023-11-24T08:13:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.473+0000 7fec62ffd640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415c7f0 0x7fec50003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.474+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003ef0 0x7fec500047c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c210 tx=0x7fec500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b9c55d99b499a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.474+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c02cb00 con 0x7fec50003ef0 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.474+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c02cc80 con 0x7fec50003ef0 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.474+0000 7fec627fc640 1 -- 172.21.15.107:0/2207394126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0058d0 con 0x7fec50003ef0 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 -- 172.21.15.107:0/2207394126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003ef0 msgr2=0x7fec500047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 --2- 172.21.15.107:0/2207394126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003ef0 0x7fec500047c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c210 tx=0x7fec500057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec637fe640 1 -- 172.21.15.107:0/2207394126 reap_dead start 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 -- 172.21.15.107:0/2207394126 shutdown_connections 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 -- 172.21.15.107:0/2207394126 >> 172.21.15.107:0/2207394126 conn(0x7fec6406a920 msgr2=0x7fec6406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 -- 172.21.15.107:0/2207394126 shutdown_connections 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.475+0000 7fec698cf640 1 -- 172.21.15.107:0/2207394126 wait complete. 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.477+0000 7fec698cf640 1 Processor -- start 2023-11-24T08:13:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.477+0000 7fec698cf640 1 -- start start 2023-11-24T08:13:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 0x7fec640ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 0x7fec640af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec64157c40 con 0x7fec641577a0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec64157f20 con 0x7fec640ada30 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec698cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec64158200 con 0x7fec50003ef0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec5a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec5a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41478/0 (socket says 172.21.15.107:41478) 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.478+0000 7fec5a7fc640 1 -- 172.21.15.107:0/1070341333 learned_addr learned my addr 172.21.15.107:0/1070341333 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec62ffd640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 0x7fec640ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec637fe640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 0x7fec640af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 -- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 msgr2=0x7fec640ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 0x7fec640ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 -- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 msgr2=0x7fec640af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 0x7fec640af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 -- 172.21.15.107:0/1070341333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec641584e0 con 0x7fec641577a0 2023-11-24T08:13:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec637fe640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 0x7fec640af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec62ffd640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 0x7fec640ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec5a7fc640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fec4c02c4a0 tx=0x7fec4c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.479+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c03c040 con 0x7fec641577a0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.480+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c007270 con 0x7fec641577a0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.480+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c007620 con 0x7fec641577a0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.480+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec641587c0 con 0x7fec641577a0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.480+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec64158ec0 con 0x7fec641577a0 2023-11-24T08:13:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.481+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec64067470 con 0x7fec641577a0 2023-11-24T08:13:12.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.482+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) v1 ==== 90608+0+0 (secure 0 0 0) 0x7fec4c032070 con 0x7fec641577a0 2023-11-24T08:13:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.483+0000 7fec60ff9640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 0x7fec4406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.483+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec4c0a9cb0 con 0x7fec641577a0 2023-11-24T08:13:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.483+0000 7fec62ffd640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 0x7fec4406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.484+0000 7fec62ffd640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 0x7fec4406e230 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fec540099c0 tx=0x7fec54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.487+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec4c076520 con 0x7fec641577a0 2023-11-24T08:13:12.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.682+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec640cd730 con 0x7fec641577a0 2023-11-24T08:13:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.685+0000 7fec60ff9640 1 -- 172.21.15.107:0/1070341333 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7fec4c079b40 con 0x7fec641577a0 2023-11-24T08:13:12.722 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:12.723 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 msgr2=0x7fec4406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 0x7fec4406e230 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fec540099c0 tx=0x7fec54008040 comp rx=0 tx=0).stop 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 msgr2=0x7fec640af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fec4c02c4a0 tx=0x7fec4c000fe0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 shutdown_connections 2023-11-24T08:13:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4406bd70 0x7fec4406e230 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec640ada30 0x7fec640af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641577a0 0x7fec640af230 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 --2- 172.21.15.107:0/1070341333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec50003ef0 0x7fec640ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 >> 172.21.15.107:0/1070341333 conn(0x7fec6406a920 msgr2=0x7fec640cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 shutdown_connections 2023-11-24T08:13:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.688+0000 7fec698cf640 1 -- 172.21.15.107:0/1070341333 wait complete. 2023-11-24T08:13:12.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:56.294075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:57.505929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:13:12.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:13:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.941+0000 7f6a74990640 1 Processor -- start 2023-11-24T08:13:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.942+0000 7f6a74990640 1 -- start start 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.943+0000 7f6a74990640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a70140520 0x7f6a700c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.943+0000 7f6a74990640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70161b10 0x7f6a7006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.943+0000 7f6a74990640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a70161fa0 con 0x7f6a700c65f0 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.943+0000 7f6a74990640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700c6a70 con 0x7f6a70140520 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.943+0000 7f6a74990640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7015de00 con 0x7f6a70161b10 2023-11-24T08:13:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a70140520 0x7f6a700c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a70140520 0x7f6a700c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41486/0 (socket says 172.21.15.107:41486) 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 -- 172.21.15.107:0/2357599390 learned_addr learned my addr 172.21.15.107:0/2357599390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 -- 172.21.15.107:0/2357599390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70161b10 msgr2=0x7f6a7006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6ed9d640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70161b10 0x7f6a7006ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70161b10 0x7f6a7006ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 -- 172.21.15.107:0/2357599390 >> v1:172.21.15.120:6789/0 conn(0x7f6a700c65f0 legacy=0x7f6a70161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.944+0000 7f6a6e59c640 1 -- 172.21.15.107:0/2357599390 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7015e0e0 con 0x7f6a70140520 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.945+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70140520 0x7f6a700c60b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009870 tx=0x7f6a5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5b2aa36f2f5ed8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.946+0000 7f6a6d59a640 1 -- 172.21.15.107:0/2357599390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58031640 con 0x7f6a70140520 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.946+0000 7f6a6d59a640 1 -- 172.21.15.107:0/2357599390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580317a0 con 0x7f6a70140520 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.946+0000 7f6a6d59a640 1 -- 172.21.15.107:0/2357599390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58031b30 con 0x7f6a70140520 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.946+0000 7f6a74990640 1 -- 172.21.15.107:0/2357599390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70140520 msgr2=0x7f6a700c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.946+0000 7f6a74990640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70140520 0x7f6a700c60b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009870 tx=0x7f6a5802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 -- 172.21.15.107:0/2357599390 shutdown_connections 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70161b10 0x7f6a7006ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 --2- 172.21.15.107:0/2357599390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70140520 0x7f6a700c60b0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 -- 172.21.15.107:0/2357599390 >> 172.21.15.107:0/2357599390 conn(0x7f6a700b2e40 msgr2=0x7f6a700b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 -- 172.21.15.107:0/2357599390 shutdown_connections 2023-11-24T08:13:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.947+0000 7f6a74990640 1 -- 172.21.15.107:0/2357599390 wait complete. 2023-11-24T08:13:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.948+0000 7f6a74990640 1 Processor -- start 2023-11-24T08:13:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.949+0000 7f6a74990640 1 -- start start 2023-11-24T08:13:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 0x7f6a7015d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70140520 0x7f6a70153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700cef40 con 0x7f6a70161b10 2023-11-24T08:13:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7015bc80 con 0x7f6a700c65f0 2023-11-24T08:13:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a74990640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7015bf60 con 0x7f6a70140520 2023-11-24T08:13:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41488/0 (socket says 172.21.15.107:41488) 2023-11-24T08:13:12.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 -- 172.21.15.107:0/903978593 learned_addr learned my addr 172.21.15.107:0/903978593 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 0x7f6a7015d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6dd9b640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70140520 0x7f6a70153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 -- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70140520 msgr2=0x7f6a70153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70140520 0x7f6a70153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 -- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 msgr2=0x7f6a7015d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 0x7f6a7015d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6ed9d640 1 -- 172.21.15.107:0/903978593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580094d0 con 0x7f6a70161b10 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.950+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 0x7f6a7015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a6ed9d640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800cd70 tx=0x7f6a6800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68003b90 con 0x7f6a70161b10 2023-11-24T08:13:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68003cf0 con 0x7f6a70161b10 2023-11-24T08:13:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68004240 con 0x7f6a70161b10 2023-11-24T08:13:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7015c240 con 0x7f6a70161b10 2023-11-24T08:13:12.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.951+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7015d680 con 0x7f6a70161b10 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.952+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a700cf0f0 con 0x7f6a70161b10 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.953+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) v1 ==== 90608+0+0 (secure 0 0 0) 0x7f6a680043a0 con 0x7f6a70161b10 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.954+0000 7f6a66ffd640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 0x7f6a5006e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.954+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a68012070 con 0x7f6a70161b10 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.954+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 0x7f6a5006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:12.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.955+0000 7f6a6e59c640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 0x7f6a5006e260 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6a58036000 tx=0x7f6a5803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:12.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:12.958+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a68055560 con 0x7f6a70161b10 2023-11-24T08:13:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.148+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a7004e8b0 con 0x7f6a70161b10 2023-11-24T08:13:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.149+0000 7f6a66ffd640 1 -- 172.21.15.107:0/903978593 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a68058b80 con 0x7f6a70161b10 2023-11-24T08:13:13.151 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:13:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.151+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 msgr2=0x7f6a5006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.151+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 0x7f6a5006e260 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6a58036000 tx=0x7f6a5803c040 comp rx=0 tx=0).stop 2023-11-24T08:13:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.151+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 msgr2=0x7f6a70157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.151+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800cd70 tx=0x7f6a6800a660 comp rx=0 tx=0).stop 2023-11-24T08:13:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 shutdown_connections 2023-11-24T08:13:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6a5006bda0 0x7f6a5006e260 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a70161b10 0x7f6a70157580 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a70140520 0x7f6a70153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 --2- 172.21.15.107:0/903978593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a700c65f0 0x7f6a7015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 >> 172.21.15.107:0/903978593 conn(0x7f6a700b2e40 msgr2=0x7f6a70070290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 shutdown_connections 2023-11-24T08:13:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.152+0000 7f6a74990640 1 -- 172.21.15.107:0/903978593 wait complete. 2023-11-24T08:13:13.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:13:13.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.390+0000 7fd6c40c9640 1 Processor -- start 2023-11-24T08:13:13.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.390+0000 7fd6c40c9640 1 -- start start 2023-11-24T08:13:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c40c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6bc072160 0x7fd6bc15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c40c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc06de10 0x7fd6bc0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c40c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc15ec90 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c40c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc0cd530 con 0x7fd6bc072160 2023-11-24T08:13:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c40c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc0c4ed0 con 0x7fd6bc06de10 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c1e65640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6bc072160 0x7fd6bc15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.392+0000 7fd6c2666640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc06de10 0x7fd6bc0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6bc072160 0x7fd6bc15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:50302/0 (socket says 172.21.15.107:50302) 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 -- 172.21.15.107:0/3964770085 learned_addr learned my addr 172.21.15.107:0/3964770085 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 -- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc06de10 msgr2=0x7fd6bc0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 --2- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc06de10 0x7fd6bc0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 -- 172.21.15.107:0/3964770085 >> v1:172.21.15.120:6789/0 conn(0x7fd6bc0c49f0 legacy=0x7fd6bc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.393+0000 7fd6c1e65640 1 -- 172.21.15.107:0/3964770085 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0c51b0 con 0x7fd6bc072160 2023-11-24T08:13:13.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.395+0000 7fd6c1e65640 1 --2- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc072160 0x7fd6bc15e750 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bb10 tx=0x7fd6b802e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbc774af58605709 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.400+0000 7fd6c0e63640 1 -- 172.21.15.107:0/3964770085 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8036440 con 0x7fd6bc072160 2023-11-24T08:13:13.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.400+0000 7fd6c0e63640 1 -- 172.21.15.107:0/3964770085 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b80365a0 con 0x7fd6bc072160 2023-11-24T08:13:13.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.400+0000 7fd6c0e63640 1 -- 172.21.15.107:0/3964770085 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8036930 con 0x7fd6bc072160 2023-11-24T08:13:13.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 -- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc072160 msgr2=0x7fd6bc15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc072160 0x7fd6bc15e750 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bb10 tx=0x7fd6b802e150 comp rx=0 tx=0).stop 2023-11-24T08:13:13.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 -- 172.21.15.107:0/3964770085 shutdown_connections 2023-11-24T08:13:13.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc06de10 0x7fd6bc0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/3964770085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc072160 0x7fd6bc15e750 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 -- 172.21.15.107:0/3964770085 >> 172.21.15.107:0/3964770085 conn(0x7fd6bc0b0c50 msgr2=0x7fd6bc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:13.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 -- 172.21.15.107:0/3964770085 shutdown_connections 2023-11-24T08:13:13.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.401+0000 7fd6c40c9640 1 -- 172.21.15.107:0/3964770085 wait complete. 2023-11-24T08:13:13.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.403+0000 7fd6c40c9640 1 Processor -- start 2023-11-24T08:13:13.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.403+0000 7fd6c40c9640 1 -- start start 2023-11-24T08:13:13.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc06de10 0x7fd6bc15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc072160 0x7fd6bc14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1595d0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc15b5f0 con 0x7fd6bc06de10 2023-11-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c40c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc06da70 con 0x7fd6bc072160 2023-11-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.404+0000 7fd6c2666640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41504/0 (socket says 172.21.15.107:41504) 2023-11-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 -- 172.21.15.107:0/989585594 learned_addr learned my addr 172.21.15.107:0/989585594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c1664640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc072160 0x7fd6bc14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c1e65640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc06de10 0x7fd6bc15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 -- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc072160 msgr2=0x7fd6bc14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc072160 0x7fd6bc14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 -- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc06de10 msgr2=0x7fd6bc15c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc06de10 0x7fd6bc15c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 -- 172.21.15.107:0/989585594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b800b770 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.405+0000 7fd6c2666640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400efc0 tx=0x7fd6b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.406+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4009ca0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.406+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b400f040 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.406+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc15b840 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.406+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40146c0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.406+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc15be00 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.407+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc04e8b0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.408+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) v1 ==== 90608+0+0 (secure 0 0 0) 0x7fd6b4014820 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.409+0000 7fd6b27fc640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 0x7fd6a006e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.409+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6b4088f20 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.409+0000 7fd6c1e65640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 0x7fd6a006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.410+0000 7fd6c1e65640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 0x7fd6a006e260 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8038000 tx=0x7fd6b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.414+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6b4055a80 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.604+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6bc15dab0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.604+0000 7fd6b27fc640 1 -- 172.21.15.107:0/989585594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6b40180a0 con 0x7fd6bc0c49f0 2023-11-24T08:13:13.607 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 msgr2=0x7fd6a006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 0x7fd6a006e260 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8038000 tx=0x7fd6b8008040 comp rx=0 tx=0).stop 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 msgr2=0x7fd6bc0abfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400efc0 tx=0x7fd6b400c490 comp rx=0 tx=0).stop 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 shutdown_connections 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd6a006bda0 0x7fd6a006e260 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6bc0c49f0 0x7fd6bc0abfa0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6bc072160 0x7fd6bc14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 --2- 172.21.15.107:0/989585594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6bc06de10 0x7fd6bc15c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.607+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 >> 172.21.15.107:0/989585594 conn(0x7fd6bc0b0c50 msgr2=0x7fd6bc0ad0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.608+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 shutdown_connections 2023-11-24T08:13:13.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.608+0000 7fd6c40c9640 1 -- 172.21.15.107:0/989585594 wait complete. 2023-11-24T08:13:13.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:13:13.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.828+0000 7f737e50f640 1 Processor -- start 2023-11-24T08:13:13.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.828+0000 7f737e50f640 1 -- start start 2023-11-24T08:13:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737e50f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f73780cc0f0 0x7f73780c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737e50f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 0x7f737806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737e50f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737815eea0 con 0x7f737815b440 2023-11-24T08:13:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737e50f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737815b820 con 0x7f73780cc0f0 2023-11-24T08:13:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737e50f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73780abf30 con 0x7f73780c57e0 2023-11-24T08:13:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f7377fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f737815b440 0x7f73780cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46224/0 (socket says 172.21.15.107:46224) 2023-11-24T08:13:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f7377fff640 1 -- 172.21.15.107:0/392373158 learned_addr learned my addr 172.21.15.107:0/392373158 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f737caac640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 0x7f737806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f736ffff640 1 --2- 172.21.15.107:0/392373158 >> v2:172.21.15.120:3300/0 conn(0x7f73780cc0f0 0x7f73780c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.830+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190611777 0 0) 0x7f737815eea0 con 0x7f737815b440 2023-11-24T08:13:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003520 con 0x7f737815b440 2023-11-24T08:13:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447356177 0 0) 0x7f7364003520 con 0x7f737815b440 2023-11-24T08:13:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 msgr2=0x7f737806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 0x7f737806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 >> v2:172.21.15.120:3300/0 conn(0x7f73780cc0f0 msgr2=0x7f73780c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 --2- 172.21.15.107:0/392373158 >> v2:172.21.15.120:3300/0 conn(0x7f73780cc0f0 0x7f73780c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737815f0f0 con 0x7f737815b440 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7368001670 con 0x7f737815b440 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 >> v1:172.21.15.107:6789/0 conn(0x7f737815b440 legacy=0x7f73780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 0x7f7364003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003ef0 0x7f73640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737815b820 con 0x7f7364004d00 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780abf30 con 0x7f7364003ef0 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736400b9d0 con 0x7f73780c57e0 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.831+0000 7f737caac640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f736ffff640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003ef0 0x7f73640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f737caac640 1 -- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 msgr2=0x7f7364003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f737caac640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73780c57e0 0x7f7364003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f737caac640 1 -- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003ef0 msgr2=0x7f73640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f737caac640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003ef0 0x7f73640047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.832+0000 7f737caac640 1 -- 172.21.15.107:0/392373158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364003520 con 0x7f7364004d00 2023-11-24T08:13:13.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f737caac640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7364004d00 0x7f7364005180 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f737000e170 tx=0x7f73640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f919b94e2b6032b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737002cc20 con 0x7f7364004d00 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737000f040 con 0x7f7364004d00 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f73777fe640 1 -- 172.21.15.107:0/392373158 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370034440 con 0x7f7364004d00 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f737e50f640 1 -- 172.21.15.107:0/392373158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7364004d00 msgr2=0x7f7364005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.833+0000 7f737e50f640 1 --2- 172.21.15.107:0/392373158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7364004d00 0x7f7364005180 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f737000e170 tx=0x7f73640057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.834+0000 7f737caac640 1 -- 172.21.15.107:0/392373158 reap_dead start 2023-11-24T08:13:13.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.834+0000 7f737e50f640 1 -- 172.21.15.107:0/392373158 shutdown_connections 2023-11-24T08:13:13.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.834+0000 7f737e50f640 1 -- 172.21.15.107:0/392373158 >> 172.21.15.107:0/392373158 conn(0x7f73780b0c50 msgr2=0x7f73780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:13.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.834+0000 7f737e50f640 1 -- 172.21.15.107:0/392373158 shutdown_connections 2023-11-24T08:13:13.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.834+0000 7f737e50f640 1 -- 172.21.15.107:0/392373158 wait complete. 2023-11-24T08:13:13.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.835+0000 7f737e50f640 1 Processor -- start 2023-11-24T08:13:13.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.836+0000 7f737e50f640 1 -- start start 2023-11-24T08:13:13.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364004d00 0x7f73780c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7378154250 0x7f73780c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780c8a20 con 0x7f737814ec40 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378154700 con 0x7f7364004d00 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f737e50f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73781f4dd0 con 0x7f7378154250 2023-11-24T08:13:13.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41516/0 (socket says 172.21.15.107:41516) 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 -- 172.21.15.107:0/2061717534 learned_addr learned my addr 172.21.15.107:0/2061717534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 -- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7378154250 msgr2=0x7f73780c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7378154250 0x7f73780c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 -- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364004d00 msgr2=0x7f73780c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:13.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364004d00 0x7f73780c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 -- 172.21.15.107:0/2061717534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73781f50b0 con 0x7f737814ec40 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.837+0000 7f736ffff640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7360010910 tx=0x7f7360010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.838+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360009690 con 0x7f737814ec40 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.838+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736000eba0 con 0x7f737814ec40 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.838+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736001c400 con 0x7f737814ec40 2023-11-24T08:13:13.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.838+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73781f5390 con 0x7f737814ec40 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.838+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737814f0c0 con 0x7f737814ec40 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.839+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73781f58b0 con 0x7f737814ec40 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.840+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) v1 ==== 90608+0+0 (secure 0 0 0) 0x7f736001c560 con 0x7f737814ec40 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.841+0000 7f73757fa640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 0x7f735806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.841+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7360088920 con 0x7f737814ec40 2023-11-24T08:13:13.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.841+0000 7f7377fff640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 0x7f735806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:13.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.842+0000 7f7377fff640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 0x7f735806e260 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7368000f60 tx=0x7f73680015c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:13.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:13.845+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73600551e0 con 0x7f737814ec40 2023-11-24T08:13:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.033+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73780b1560 con 0x7f737814ec40 2023-11-24T08:13:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.034+0000 7f73757fa640 1 -- 172.21.15.107:0/2061717534 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7360058800 con 0x7f737814ec40 2023-11-24T08:13:14.037 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:13:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 msgr2=0x7f735806e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 0x7f735806e260 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7368000f60 tx=0x7f73680015c0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 msgr2=0x7f7378153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7360010910 tx=0x7f7360010dd0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 shutdown_connections 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f735806bda0 0x7f735806e260 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7378154250 0x7f73780c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f737814ec40 0x7f7378153d10 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 --2- 172.21.15.107:0/2061717534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364004d00 0x7f73780c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 >> 172.21.15.107:0/2061717534 conn(0x7f73780b0c50 msgr2=0x7f737806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 shutdown_connections 2023-11-24T08:13:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.037+0000 7f737e50f640 1 -- 172.21.15.107:0/2061717534 wait complete. 2023-11-24T08:13:14.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.255+0000 7f435ec39640 1 Processor -- start 2023-11-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.255+0000 7f435ec39640 1 -- start start 2023-11-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.257+0000 7f435ec39640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f435815b1d0 0x7f4358071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.257+0000 7f435ec39640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.257+0000 7f435ec39640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43580ca0e0 con 0x7f43580b1080 2023-11-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.257+0000 7f435ec39640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435815fde0 con 0x7f435815b1d0 2023-11-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.257+0000 7f435ec39640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43580b1510 con 0x7f4358070550 2023-11-24T08:13:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435dc37640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f435815b1d0 0x7f4358071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:48302/0 (socket says 172.21.15.107:48302) 2023-11-24T08:13:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 -- 172.21.15.107:0/2759941061 learned_addr learned my addr 172.21.15.107:0/2759941061 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 -- 172.21.15.107:0/2759941061 >> v2:172.21.15.120:3300/0 conn(0x7f435815b1d0 msgr2=0x7f4358071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 --2- 172.21.15.107:0/2759941061 >> v2:172.21.15.120:3300/0 conn(0x7f435815b1d0 0x7f4358071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 -- 172.21.15.107:0/2759941061 >> v1:172.21.15.120:6789/0 conn(0x7f43580b1080 legacy=0x7f435806c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.258+0000 7f435d436640 1 -- 172.21.15.107:0/2759941061 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358072800 con 0x7f4358070550 2023-11-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.260+0000 7f435d436640 1 --2- 172.21.15.107:0/2759941061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4348009870 tx=0x7f434802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348e2f127330cc0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.260+0000 7f435cc35640 1 -- 172.21.15.107:0/2759941061 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348030840 con 0x7f4358070550 2023-11-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435cc35640 1 -- 172.21.15.107:0/2759941061 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43480309a0 con 0x7f4358070550 2023-11-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435cc35640 1 -- 172.21.15.107:0/2759941061 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348030d30 con 0x7f4358070550 2023-11-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 -- 172.21.15.107:0/2759941061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 msgr2=0x7f43580b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 --2- 172.21.15.107:0/2759941061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4348009870 tx=0x7f434802b980 comp rx=0 tx=0).stop 2023-11-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 -- 172.21.15.107:0/2759941061 shutdown_connections 2023-11-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 --2- 172.21.15.107:0/2759941061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4358070550 0x7f43580b0b40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 --2- 172.21.15.107:0/2759941061 >> v2:172.21.15.120:3300/0 conn(0x7f435815b1d0 0x7f4358071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 -- 172.21.15.107:0/2759941061 >> 172.21.15.107:0/2759941061 conn(0x7f43580cbb40 msgr2=0x7f43580cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 -- 172.21.15.107:0/2759941061 shutdown_connections 2023-11-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.261+0000 7f435ec39640 1 -- 172.21.15.107:0/2759941061 wait complete. 2023-11-24T08:13:14.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.263+0000 7f435ec39640 1 Processor -- start 2023-11-24T08:13:14.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.263+0000 7f435ec39640 1 -- start start 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43580b1080 0x7f435809c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f435815b1d0 0x7f43580999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43580c3b10 con 0x7f4358070550 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358099f10 con 0x7f43580b1080 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435ec39640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43580c3d80 con 0x7f435815b1d0 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435dc37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435dc37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41522/0 (socket says 172.21.15.107:41522) 2023-11-24T08:13:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.264+0000 7f435dc37640 1 -- 172.21.15.107:0/2796296495 learned_addr learned my addr 172.21.15.107:0/2796296495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435e438640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f435815b1d0 0x7f43580999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435d436640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43580b1080 0x7f435809c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 -- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f435815b1d0 msgr2=0x7f43580999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f435815b1d0 0x7f43580999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 -- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43580b1080 msgr2=0x7f435809c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43580b1080 0x7f435809c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 -- 172.21.15.107:0/2796296495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480094d0 con 0x7f4358070550 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435dc37640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f435400e870 tx=0x7f435400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540094f0 con 0x7f4358070550 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400c940 con 0x7f4358070550 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435400ccf0 con 0x7f4358070550 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.265+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43580c4060 con 0x7f4358070550 2023-11-24T08:13:14.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.266+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43580cebf0 con 0x7f4358070550 2023-11-24T08:13:14.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.267+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435804e840 con 0x7f4358070550 2023-11-24T08:13:14.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.268+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) v1 ==== 90608+0+0 (secure 0 0 0) 0x7f4354013470 con 0x7f4358070550 2023-11-24T08:13:14.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.268+0000 7f434e7fc640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 0x7f433c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.268+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4354088210 con 0x7f4358070550 2023-11-24T08:13:14.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.269+0000 7f435d436640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 0x7f433c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.269+0000 7f435d436640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 0x7f433c06e230 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4348036000 tx=0x7f4348008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:14.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.273+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4354054cf0 con 0x7f4358070550 2023-11-24T08:13:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.420+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f43540550c0 con 0x7f4358070550 2023-11-24T08:13:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.481+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f43580c2800 con 0x7f4358070550 2023-11-24T08:13:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.482+0000 7f434e7fc640 1 -- 172.21.15.107:0/2796296495 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4354074220 con 0x7f4358070550 2023-11-24T08:13:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:13:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 msgr2=0x7f433c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 0x7f433c06e230 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4348036000 tx=0x7f4348008030 comp rx=0 tx=0).stop 2023-11-24T08:13:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 msgr2=0x7f435809b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f435400e870 tx=0x7f435400ed30 comp rx=0 tx=0).stop 2023-11-24T08:13:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 shutdown_connections 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f433c06bd70 0x7f433c06e230 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f435815b1d0 0x7f43580999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43580b1080 0x7f435809c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 --2- 172.21.15.107:0/2796296495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4358070550 0x7f435809b9b0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 >> 172.21.15.107:0/2796296495 conn(0x7f43580cbb40 msgr2=0x7f43580acad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 shutdown_connections 2023-11-24T08:13:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.485+0000 7f435ec39640 1 -- 172.21.15.107:0/2796296495 wait complete. 2023-11-24T08:13:14.506 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:13:14.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:14.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:14.608 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:13:14.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:14.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:13:14.658 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:14.658 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:14.719 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:14.719 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:14.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:14.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.995+0000 7fe5bcc75640 1 Processor -- start 2023-11-24T08:13:14.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.995+0000 7fe5bcc75640 1 -- start start 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b8157c70 0x7fe5b8161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5b80cf450 0x7fe5b80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815dc80 0x7fe5b815e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b80cfe40 con 0x7fe5b8157c70 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b80ce0b0 con 0x7fe5b80cf450 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5bcc75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b8161960 con 0x7fe5b815dc80 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b8157c70 0x7fe5b8161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b8157c70 0x7fe5b8161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41530/0 (socket says 172.21.15.107:41530) 2023-11-24T08:13:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.997+0000 7fe5b659c640 1 -- 172.21.15.107:0/973247937 learned_addr learned my addr 172.21.15.107:0/973247937 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b6d9d640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815dc80 0x7fe5b815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5affff640 1 --2- 172.21.15.107:0/973247937 >> v2:172.21.15.120:3300/0 conn(0x7fe5b80cf450 0x7fe5b80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b659c640 1 -- 172.21.15.107:0/973247937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815dc80 msgr2=0x7fe5b815e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b659c640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815dc80 0x7fe5b815e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b659c640 1 -- 172.21.15.107:0/973247937 >> v2:172.21.15.120:3300/0 conn(0x7fe5b80cf450 msgr2=0x7fe5b80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b659c640 1 --2- 172.21.15.107:0/973247937 >> v2:172.21.15.120:3300/0 conn(0x7fe5b80cf450 0x7fe5b80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.998+0000 7fe5b659c640 1 -- 172.21.15.107:0/973247937 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b815e650 con 0x7fe5b8157c70 2023-11-24T08:13:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:14.999+0000 7fe5b659c640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b8157c70 0x7fe5b8161df0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80b44b0 tx=0x7fe5a802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f4f378ff691546 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5b5d9b640 1 -- 172.21.15.107:0/973247937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80317c0 con 0x7fe5b8157c70 2023-11-24T08:13:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5b5d9b640 1 -- 172.21.15.107:0/973247937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a8031920 con 0x7fe5b8157c70 2023-11-24T08:13:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5b5d9b640 1 -- 172.21.15.107:0/973247937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8031cb0 con 0x7fe5b8157c70 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 -- 172.21.15.107:0/973247937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b8157c70 msgr2=0x7fe5b8161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b8157c70 0x7fe5b8161df0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80b44b0 tx=0x7fe5a802b860 comp rx=0 tx=0).stop 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 -- 172.21.15.107:0/973247937 shutdown_connections 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815dc80 0x7fe5b815e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/973247937 >> v2:172.21.15.120:3300/0 conn(0x7fe5b80cf450 0x7fe5b80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/973247937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b8157c70 0x7fe5b8161df0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.000+0000 7fe5bcc75640 1 -- 172.21.15.107:0/973247937 >> 172.21.15.107:0/973247937 conn(0x7fe5b806a920 msgr2=0x7fe5b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.001+0000 7fe5bcc75640 1 -- 172.21.15.107:0/973247937 shutdown_connections 2023-11-24T08:13:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.001+0000 7fe5bcc75640 1 -- 172.21.15.107:0/973247937 wait complete. 2023-11-24T08:13:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.002+0000 7fe5bcc75640 1 Processor -- start 2023-11-24T08:13:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.002+0000 7fe5bcc75640 1 -- start start 2023-11-24T08:13:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b80cf450 0x7fe5b8155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b8157c70 0x7fe5b81f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81f47b0 con 0x7fe5b815dc80 2023-11-24T08:13:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81f4180 con 0x7fe5b8157c70 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.003+0000 7fe5bcc75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81f3ab0 con 0x7fe5b80cf450 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41542/0 (socket says 172.21.15.107:41542) 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 -- 172.21.15.107:0/3079624471 learned_addr learned my addr 172.21.15.107:0/3079624471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b659c640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b80cf450 0x7fe5b8155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5affff640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b8157c70 0x7fe5b81f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 -- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b80cf450 msgr2=0x7fe5b8155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b80cf450 0x7fe5b8155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 -- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b8157c70 msgr2=0x7fe5b81f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b8157c70 0x7fe5b81f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.004+0000 7fe5b6d9d640 1 -- 172.21.15.107:0/3079624471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80094d0 con 0x7fe5b815dc80 2023-11-24T08:13:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5b6d9d640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0007d90 tx=0x7fe5b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0014420 con 0x7fe5b815dc80 2023-11-24T08:13:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b00145a0 con 0x7fe5b815dc80 2023-11-24T08:13:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b81f3d90 con 0x7fe5b815dc80 2023-11-24T08:13:15.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0014950 con 0x7fe5b815dc80 2023-11-24T08:13:15.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.005+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b809cf50 con 0x7fe5b815dc80 2023-11-24T08:13:15.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.006+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b81f3fd0 con 0x7fe5b815dc80 2023-11-24T08:13:15.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.007+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fe5b0014ab0 con 0x7fe5b815dc80 2023-11-24T08:13:15.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.008+0000 7fe5aeffd640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 0x7fe59806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.008+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5b0088d70 con 0x7fe5b815dc80 2023-11-24T08:13:15.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.008+0000 7fe5b659c640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 0x7fe59806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.009+0000 7fe5b659c640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 0x7fe59806e350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8036000 tx=0x7fe5a803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.013+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5b0055550 con 0x7fe5b815dc80 2023-11-24T08:13:15.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.210+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b81f48f0 con 0x7fe5b815dc80 2023-11-24T08:13:15.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.212+0000 7fe5aeffd640 1 -- 172.21.15.107:0/3079624471 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7fe5b81f48f0 con 0x7fe5b815dc80 2023-11-24T08:13:15.215 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:15.215 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.215+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 msgr2=0x7fe59806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.215+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 0x7fe59806e350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8036000 tx=0x7fe5a803d040 comp rx=0 tx=0).stop 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.215+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 msgr2=0x7fe5b81f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.215+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0007d90 tx=0x7fe5b000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 shutdown_connections 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59806be90 0x7fe59806e350 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b815dc80 0x7fe5b81f34b0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b8157c70 0x7fe5b81f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 --2- 172.21.15.107:0/3079624471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b80cf450 0x7fe5b8155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 >> 172.21.15.107:0/3079624471 conn(0x7fe5b806a920 msgr2=0x7fe5b815efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 shutdown_connections 2023-11-24T08:13:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.216+0000 7fe5bcc75640 1 -- 172.21.15.107:0/3079624471 wait complete. 2023-11-24T08:13:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:56.294075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:57.505929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:13:15.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:13:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.451+0000 7facddb1d640 1 Processor -- start 2023-11-24T08:13:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.451+0000 7facddb1d640 1 -- start start 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facddb1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd815b440 0x7facd80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facddb1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7facd80c59b0 0x7facd806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facddb1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd80cc3a0 con 0x7facd80cbf20 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facddb1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd815b820 con 0x7facd80c59b0 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facddb1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd815eea0 con 0x7facd815b440 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facd7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7facd80c59b0 0x7facd806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facceffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7facd80cbf20 0x7facd80c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46228/0 (socket says 172.21.15.107:46228) 2023-11-24T08:13:15.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facceffd640 1 -- 172.21.15.107:0/2826043695 learned_addr learned my addr 172.21.15.107:0/2826043695 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:15.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.453+0000 7facd77fe640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd815b440 0x7facd80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd6ffd640 1 -- 172.21.15.107:0/2826043695 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982381166 0 0) 0x7facc0001490 con 0x7facd80cbf20 2023-11-24T08:13:15.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd6ffd640 1 -- 172.21.15.107:0/2826043695 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4003520 con 0x7facd80cbf20 2023-11-24T08:13:15.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd7fff640 1 -- 172.21.15.107:0/2826043695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd815b440 msgr2=0x7facd80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd7fff640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd815b440 0x7facd80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd7fff640 1 -- 172.21.15.107:0/2826043695 >> v1:172.21.15.107:6789/0 conn(0x7facd80cbf20 legacy=0x7facd80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.454+0000 7facd7fff640 1 -- 172.21.15.107:0/2826043695 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80abf30 con 0x7facd80c59b0 2023-11-24T08:13:15.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.455+0000 7facd7fff640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd80c59b0 0x7facd806c5f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7facd00097a0 tx=0x7facd002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b903c0ca66d1538 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facd6ffd640 1 -- 172.21.15.107:0/2826043695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd002e070 con 0x7facd80c59b0 2023-11-24T08:13:15.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facd6ffd640 1 -- 172.21.15.107:0/2826043695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd00315a0 con 0x7facd80c59b0 2023-11-24T08:13:15.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facd6ffd640 1 -- 172.21.15.107:0/2826043695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0031910 con 0x7facd80c59b0 2023-11-24T08:13:15.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 -- 172.21.15.107:0/2826043695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd80c59b0 msgr2=0x7facd806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd80c59b0 0x7facd806c5f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7facd00097a0 tx=0x7facd002b840 comp rx=0 tx=0).stop 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 -- 172.21.15.107:0/2826043695 shutdown_connections 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd80c59b0 0x7facd806c5f0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 --2- 172.21.15.107:0/2826043695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd815b440 0x7facd80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 -- 172.21.15.107:0/2826043695 >> 172.21.15.107:0/2826043695 conn(0x7facd80b0c50 msgr2=0x7facd80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 -- 172.21.15.107:0/2826043695 shutdown_connections 2023-11-24T08:13:15.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.456+0000 7facddb1d640 1 -- 172.21.15.107:0/2826043695 wait complete. 2023-11-24T08:13:15.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.458+0000 7facddb1d640 1 Processor -- start 2023-11-24T08:13:15.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.458+0000 7facddb1d640 1 -- start start 2023-11-24T08:13:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd80c59b0 0x7facd81542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facd80cbf20 0x7facd80c4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd8156550 con 0x7facd815b440 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80ac1f0 con 0x7facd80cbf20 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.459+0000 7facddb1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c7b60 con 0x7facd80c59b0 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41556/0 (socket says 172.21.15.107:41556) 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 -- 172.21.15.107:0/256071764 learned_addr learned my addr 172.21.15.107:0/256071764 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd77fe640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd80c59b0 0x7facd81542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facceffd640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facd80cbf20 0x7facd80c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 -- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd80c59b0 msgr2=0x7facd81542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd80c59b0 0x7facd81542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 -- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facd80cbf20 msgr2=0x7facd80c4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facd80cbf20 0x7facd80c4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.460+0000 7facd7fff640 1 -- 172.21.15.107:0/256071764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0009400 con 0x7facd815b440 2023-11-24T08:13:15.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facd7fff640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7facd0034040 tx=0x7facd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0032630 con 0x7facd815b440 2023-11-24T08:13:15.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0032790 con 0x7facd815b440 2023-11-24T08:13:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd80c7e40 con 0x7facd815b440 2023-11-24T08:13:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0032b20 con 0x7facd815b440 2023-11-24T08:13:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.461+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd80c8400 con 0x7facd815b440 2023-11-24T08:13:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.462+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd80ac3b0 con 0x7facd815b440 2023-11-24T08:13:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.463+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7facd0032c80 con 0x7facd815b440 2023-11-24T08:13:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.464+0000 7facd4ff9640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 0x7facb806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.464+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facd00a9be0 con 0x7facd815b440 2023-11-24T08:13:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.465+0000 7facd77fe640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 0x7facb806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.465+0000 7facd77fe640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 0x7facb806e350 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7facc8006fd0 tx=0x7facc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.469+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facd0076790 con 0x7facd815b440 2023-11-24T08:13:15.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.656+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7facd804e8b0 con 0x7facd815b440 2023-11-24T08:13:15.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.657+0000 7facd4ff9640 1 -- 172.21.15.107:0/256071764 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7facd0079db0 con 0x7facd815b440 2023-11-24T08:13:15.659 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:13:15.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.659+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 msgr2=0x7facb806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.659+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 0x7facb806e350 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7facc8006fd0 tx=0x7facc8008040 comp rx=0 tx=0).stop 2023-11-24T08:13:15.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.659+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 msgr2=0x7facd80c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.659+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7facd0034040 tx=0x7facd00046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 shutdown_connections 2023-11-24T08:13:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7facb806be90 0x7facb806e350 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facd815b440 0x7facd80c4bc0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facd80cbf20 0x7facd80c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 --2- 172.21.15.107:0/256071764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7facd80c59b0 0x7facd81542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 >> 172.21.15.107:0/256071764 conn(0x7facd80b0c50 msgr2=0x7facd8154a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 shutdown_connections 2023-11-24T08:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.660+0000 7facddb1d640 1 -- 172.21.15.107:0/256071764 wait complete. 2023-11-24T08:13:15.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:13:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.888+0000 7f1e99341640 1 Processor -- start 2023-11-24T08:13:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.888+0000 7f1e99341640 1 -- start start 2023-11-24T08:13:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e99341640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1e940cc0f0 0x7f1e940c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e99341640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9415eea0 con 0x7f1e9415b440 2023-11-24T08:13:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e99341640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940abf30 con 0x7f1e940c57e0 2023-11-24T08:13:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e99341640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9415b820 con 0x7f1e940cc0f0 2023-11-24T08:13:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e92ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e9415b440 0x7f1e940cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46230/0 (socket says 172.21.15.107:46230) 2023-11-24T08:13:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e92ffd640 1 -- 172.21.15.107:0/2338717684 learned_addr learned my addr 172.21.15.107:0/2338717684 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e927fc640 1 --2- 172.21.15.107:0/2338717684 >> v2:172.21.15.120:3300/0 conn(0x7f1e940cc0f0 0x7f1e940c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558219486 0 0) 0x7f1e9415eea0 con 0x7f1e9415b440 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.890+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c003520 con 0x7f1e9415b440 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019992260 0 0) 0x7f1e7c003520 con 0x7f1e9415b440 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 >> v2:172.21.15.120:3300/0 conn(0x7f1e940cc0f0 msgr2=0x7f1e940c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 --2- 172.21.15.107:0/2338717684 >> v2:172.21.15.120:3300/0 conn(0x7f1e940cc0f0 0x7f1e940c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 >> v1:172.21.15.120:6789/0 conn(0x7f1e940c57e0 legacy=0x7f1e9406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9415f0f0 con 0x7f1e9415b440 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1e84001670 con 0x7f1e9415b440 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 >> v1:172.21.15.107:6789/0 conn(0x7f1e9415b440 legacy=0x7f1e940cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e7c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e7c0047c0 0x7f1e7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e7c005180 0x7f1e7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9415b820 con 0x7f1e7c003970 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c00c0b0 con 0x7f1e7c005180 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c00c270 con 0x7f1e7c0047c0 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e7c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.891+0000 7f1e937fe640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e7c005180 0x7f1e7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e927fc640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e7c0047c0 0x7f1e7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e92ffd640 1 -- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e7c0047c0 msgr2=0x7f1e7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e7c0047c0 0x7f1e7c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e92ffd640 1 -- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e7c005180 msgr2=0x7f1e7c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e7c005180 0x7f1e7c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e92ffd640 1 -- 172.21.15.107:0/2338717684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c003520 con 0x7f1e7c003970 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.892+0000 7f1e937fe640 1 -- 172.21.15.107:0/2338717684 reap_dead start 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e7c003d70 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400c210 tx=0x7f1e7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c5a2d8bb789e98 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8402cae0 con 0x7f1e7c003970 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8402cc60 con 0x7f1e7c003970 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e91ffb640 1 -- 172.21.15.107:0/2338717684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84005b10 con 0x7f1e7c003970 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e99341640 1 -- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 msgr2=0x7f1e7c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.893+0000 7f1e99341640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e7c003d70 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400c210 tx=0x7f1e7c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.894+0000 7f1e99341640 1 -- 172.21.15.107:0/2338717684 shutdown_connections 2023-11-24T08:13:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.894+0000 7f1e99341640 1 --2- 172.21.15.107:0/2338717684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e7c003d70 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.894+0000 7f1e99341640 1 -- 172.21.15.107:0/2338717684 >> 172.21.15.107:0/2338717684 conn(0x7f1e940b0c50 msgr2=0x7f1e940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.894+0000 7f1e99341640 1 -- 172.21.15.107:0/2338717684 shutdown_connections 2023-11-24T08:13:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.894+0000 7f1e99341640 1 -- 172.21.15.107:0/2338717684 wait complete. 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.895+0000 7f1e99341640 1 Processor -- start 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.896+0000 7f1e99341640 1 -- start start 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.896+0000 7f1e99341640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.896+0000 7f1e99341640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e940999d0 0x7f1e94099e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.896+0000 7f1e99341640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e941564e0 0x7f1e94156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e99341640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e94156ea0 con 0x7f1e7c003970 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e99341640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9409b4c0 con 0x7f1e940999d0 2023-11-24T08:13:15.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e99341640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9409a420 con 0x7f1e941564e0 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41572/0 (socket says 172.21.15.107:41572) 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 -- 172.21.15.107:0/285454769 learned_addr learned my addr 172.21.15.107:0/285454769 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e937fe640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e941564e0 0x7f1e94156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 -- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e941564e0 msgr2=0x7f1e94156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e941564e0 0x7f1e94156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e927fc640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e940999d0 0x7f1e94099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 -- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e940999d0 msgr2=0x7f1e94099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e940999d0 0x7f1e94099e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.897+0000 7f1e92ffd640 1 -- 172.21.15.107:0/285454769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9409a700 con 0x7f1e7c003970 2023-11-24T08:13:15.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e92ffd640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1e8402c480 tx=0x7f1e8402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8403c040 con 0x7f1e7c003970 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e84001630 con 0x7f1e7c003970 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84007270 con 0x7f1e7c003970 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9409a9e0 con 0x7f1e7c003970 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.898+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9409afa0 con 0x7f1e7c003970 2023-11-24T08:13:15.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.899+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9409b130 con 0x7f1e7c003970 2023-11-24T08:13:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.900+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f1e84032070 con 0x7f1e7c003970 2023-11-24T08:13:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.902+0000 7f1e7b7fe640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 0x7f1e8006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.902+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e840a9390 con 0x7f1e7c003970 2023-11-24T08:13:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.902+0000 7f1e927fc640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 0x7f1e8006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.903+0000 7f1e927fc640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 0x7f1e8006e380 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c4ef0 tx=0x7f1e88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:15.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:15.906+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e84075e00 con 0x7f1e7c003970 2023-11-24T08:13:16.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.097+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e9404e8b0 con 0x7f1e7c003970 2023-11-24T08:13:16.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.098+0000 7f1e7b7fe640 1 -- 172.21.15.107:0/285454769 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e84079420 con 0x7f1e7c003970 2023-11-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:13:16.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 msgr2=0x7f1e8006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 0x7f1e8006e380 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c4ef0 tx=0x7f1e88008040 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 msgr2=0x7f1e9415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1e8402c480 tx=0x7f1e8402e070 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 shutdown_connections 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e8006bec0 0x7f1e8006e380 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e941564e0 0x7f1e94156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e940999d0 0x7f1e94099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 --2- 172.21.15.107:0/285454769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e7c003970 0x7f1e9415d9e0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 >> 172.21.15.107:0/285454769 conn(0x7f1e940b0c50 msgr2=0x7f1e940ad6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 shutdown_connections 2023-11-24T08:13:16.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.100+0000 7f1e99341640 1 -- 172.21.15.107:0/285454769 wait complete. 2023-11-24T08:13:16.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:13:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.340+0000 7fad23ee0640 1 Processor -- start 2023-11-24T08:13:16.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.341+0000 7fad23ee0640 1 -- start start 2023-11-24T08:13:16.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.342+0000 7fad23ee0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad1c072160 0x7fad1c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.342+0000 7fad23ee0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c15ec90 con 0x7fad1c06de10 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.342+0000 7fad23ee0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0cd530 con 0x7fad1c0c49f0 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.342+0000 7fad23ee0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c0c4ed0 con 0x7fad1c072160 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad21c7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad1c072160 0x7fad1c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad2247d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fad1c06de10 0x7fad1c0705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46232/0 (socket says 172.21.15.107:46232) 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad2247d640 1 -- 172.21.15.107:0/866118052 learned_addr learned my addr 172.21.15.107:0/866118052 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad21c7c640 1 -- 172.21.15.107:0/866118052 >> v1:172.21.15.120:6789/0 conn(0x7fad1c0c49f0 legacy=0x7fad1c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:16.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad21c7c640 1 -- 172.21.15.107:0/866118052 >> v1:172.21.15.107:6789/0 conn(0x7fad1c06de10 legacy=0x7fad1c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad20c7a640 1 -- 172.21.15.107:0/866118052 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545767536 0 0) 0x7fad14001490 con 0x7fad1c06de10 2023-11-24T08:13:16.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.343+0000 7fad21c7c640 1 -- 172.21.15.107:0/866118052 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0c51b0 con 0x7fad1c072160 2023-11-24T08:13:16.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad21c7c640 1 --2- 172.21.15.107:0/866118052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c15e750 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e50fd375538c9d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad20c7a640 1 -- 172.21.15.107:0/866118052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18030610 con 0x7fad1c072160 2023-11-24T08:13:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad20c7a640 1 -- 172.21.15.107:0/866118052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad18030770 con 0x7fad1c072160 2023-11-24T08:13:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad20c7a640 1 -- 172.21.15.107:0/866118052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18030b00 con 0x7fad1c072160 2023-11-24T08:13:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad23ee0640 1 -- 172.21.15.107:0/866118052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 msgr2=0x7fad1c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad23ee0640 1 --2- 172.21.15.107:0/866118052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c15e750 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b960 comp rx=0 tx=0).stop 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad23ee0640 1 -- 172.21.15.107:0/866118052 shutdown_connections 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad23ee0640 1 --2- 172.21.15.107:0/866118052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c15e750 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.345+0000 7fad23ee0640 1 -- 172.21.15.107:0/866118052 >> 172.21.15.107:0/866118052 conn(0x7fad1c0b0c50 msgr2=0x7fad1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.346+0000 7fad23ee0640 1 -- 172.21.15.107:0/866118052 shutdown_connections 2023-11-24T08:13:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.346+0000 7fad23ee0640 1 -- 172.21.15.107:0/866118052 wait complete. 2023-11-24T08:13:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.347+0000 7fad23ee0640 1 Processor -- start 2023-11-24T08:13:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.347+0000 7fad23ee0640 1 -- start start 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad1c06de10 0x7fad1c148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c0c49f0 0x7fad1c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c09c810 con 0x7fad1c072160 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c14c3a0 con 0x7fad1c0c49f0 2023-11-24T08:13:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.348+0000 7fad23ee0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c09c530 con 0x7fad1c06de10 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41600/0 (socket says 172.21.15.107:41600) 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 -- 172.21.15.107:0/4214153059 learned_addr learned my addr 172.21.15.107:0/4214153059 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad21c7c640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad1c06de10 0x7fad1c148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2247d640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c0c49f0 0x7fad1c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 -- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad1c06de10 msgr2=0x7fad1c148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad1c06de10 0x7fad1c148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 -- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c0c49f0 msgr2=0x7fad1c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c0c49f0 0x7fad1c158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 -- 172.21.15.107:0/4214153059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18009400 con 0x7fad1c072160 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.349+0000 7fad2147b640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c400 tx=0x7fad0c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.350+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0052d0 con 0x7fad1c072160 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.350+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c005430 con 0x7fad1c072160 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.350+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c003830 con 0x7fad1c072160 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.350+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c14c640 con 0x7fad1c072160 2023-11-24T08:13:16.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.350+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c14d360 con 0x7fad1c072160 2023-11-24T08:13:16.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.351+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c04e8b0 con 0x7fad1c072160 2023-11-24T08:13:16.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.352+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fad0c003990 con 0x7fad1c072160 2023-11-24T08:13:16.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.353+0000 7fad127fc640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 0x7fad0006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.353+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad0c087a10 con 0x7fad1c072160 2023-11-24T08:13:16.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.353+0000 7fad21c7c640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 0x7fad0006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.354+0000 7fad21c7c640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 0x7fad0006e380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fad18036000 tx=0x7fad18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.357+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad0c0541f0 con 0x7fad1c072160 2023-11-24T08:13:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.547+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad1c0b4fb0 con 0x7fad1c072160 2023-11-24T08:13:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.547+0000 7fad127fc640 1 -- 172.21.15.107:0/4214153059 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad0c017020 con 0x7fad1c072160 2023-11-24T08:13:16.549 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 msgr2=0x7fad0006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 0x7fad0006e380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fad18036000 tx=0x7fad18008030 comp rx=0 tx=0).stop 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 msgr2=0x7fad1c14ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c400 tx=0x7fad0c00c8c0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 shutdown_connections 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad0006bec0 0x7fad0006e380 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c0c49f0 0x7fad1c158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad1c072160 0x7fad1c14ce20 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 --2- 172.21.15.107:0/4214153059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad1c06de10 0x7fad1c148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 >> 172.21.15.107:0/4214153059 conn(0x7fad1c0b0c50 msgr2=0x7fad1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 shutdown_connections 2023-11-24T08:13:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.550+0000 7fad23ee0640 1 -- 172.21.15.107:0/4214153059 wait complete. 2023-11-24T08:13:16.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:13:16.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.775+0000 7fd300883640 1 Processor -- start 2023-11-24T08:13:16.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.776+0000 7fd300883640 1 -- start start 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.777+0000 7fd300883640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 0x7fd2fc0b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.777+0000 7fd300883640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0b5430 0x7fd2fc167280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.777+0000 7fd300883640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0b37b0 con 0x7fd2fc0999d0 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.777+0000 7fd300883640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc06f9b0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.777+0000 7fd300883640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc06d900 con 0x7fd2fc15d620 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0b5430 0x7fd2fc167280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0b5430 0x7fd2fc167280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41604/0 (socket says 172.21.15.107:41604) 2023-11-24T08:13:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 -- 172.21.15.107:0/2457501624 learned_addr learned my addr 172.21.15.107:0/2457501624 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/2457501624 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 0x7fd2fc0b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 -- 172.21.15.107:0/2457501624 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 msgr2=0x7fd2fc0b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 --2- 172.21.15.107:0/2457501624 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 0x7fd2fc0b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 -- 172.21.15.107:0/2457501624 >> v1:172.21.15.120:6789/0 conn(0x7fd2fc0999d0 legacy=0x7fd2fc0b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fad9d640 1 -- 172.21.15.107:0/2457501624 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc06dbe0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.778+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/2457501624 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 0x7fd2fc0b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.779+0000 7fd2fad9d640 1 --2- 172.21.15.107:0/2457501624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc167280 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc06d0e0 tx=0x7fd2f402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4961025bda78789 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.780+0000 7fd2f959a640 1 -- 172.21.15.107:0/2457501624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40317e0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.780+0000 7fd2f959a640 1 -- 172.21.15.107:0/2457501624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f4031940 con 0x7fd2fc0b5430 2023-11-24T08:13:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.780+0000 7fd2f959a640 1 -- 172.21.15.107:0/2457501624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4031cd0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.780+0000 7fd300883640 1 -- 172.21.15.107:0/2457501624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 msgr2=0x7fd2fc167280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.780+0000 7fd300883640 1 --2- 172.21.15.107:0/2457501624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc167280 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc06d0e0 tx=0x7fd2f402b880 comp rx=0 tx=0).stop 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 -- 172.21.15.107:0/2457501624 shutdown_connections 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 --2- 172.21.15.107:0/2457501624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc167280 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 --2- 172.21.15.107:0/2457501624 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc15d620 0x7fd2fc0b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 -- 172.21.15.107:0/2457501624 >> 172.21.15.107:0/2457501624 conn(0x7fd2fc06c090 msgr2=0x7fd2fc0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 -- 172.21.15.107:0/2457501624 shutdown_connections 2023-11-24T08:13:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.781+0000 7fd300883640 1 -- 172.21.15.107:0/2457501624 wait complete. 2023-11-24T08:13:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.782+0000 7fd300883640 1 Processor -- start 2023-11-24T08:13:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.782+0000 7fd300883640 1 -- start start 2023-11-24T08:13:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 0x7fd2fc0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 0x7fd2fc155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0cc770 con 0x7fd2fc0b5430 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0ca670 con 0x7fd2fc15d620 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.783+0000 7fd300883640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc15daa0 con 0x7fd2fc0999d0 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41618/0 (socket says 172.21.15.107:41618) 2023-11-24T08:13:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 -- 172.21.15.107:0/3905978323 learned_addr learned my addr 172.21.15.107:0/3905978323 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 0x7fd2fc0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2fad9d640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 0x7fd2fc155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 -- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 msgr2=0x7fd2fc0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 0x7fd2fc0ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 -- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 msgr2=0x7fd2fc155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 0x7fd2fc155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 -- 172.21.15.107:0/3905978323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f40094d0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 0x7fd2fc0ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2fad9d640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 0x7fd2fc155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.784+0000 7fd2f9d9b640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c410 tx=0x7fd2e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.785+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40035f0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.785+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4003750 con 0x7fd2fc0b5430 2023-11-24T08:13:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.785+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4003b40 con 0x7fd2fc0b5430 2023-11-24T08:13:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.785+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc15dd80 con 0x7fd2fc0b5430 2023-11-24T08:13:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.785+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc15e340 con 0x7fd2fc0b5430 2023-11-24T08:13:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.786+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc04e8b0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.787+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fd2e4003ca0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.788+0000 7fd2f2ffd640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 0x7fd2dc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.788+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2e4087660 con 0x7fd2fc0b5430 2023-11-24T08:13:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.788+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 0x7fd2dc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.789+0000 7fd2fa59c640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 0x7fd2dc06e350 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cb1e0 tx=0x7fd2ec0074b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.793+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2e408f070 con 0x7fd2fc0b5430 2023-11-24T08:13:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.996+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd2fc0cc8b0 con 0x7fd2fc0b5430 2023-11-24T08:13:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.996+0000 7fd2f2ffd640 1 -- 172.21.15.107:0/3905978323 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd2e4017080 con 0x7fd2fc0b5430 2023-11-24T08:13:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:13:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 msgr2=0x7fd2dc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 0x7fd2dc06e350 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cb1e0 tx=0x7fd2ec0074b0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 msgr2=0x7fd2fc0cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c410 tx=0x7fd2e400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 shutdown_connections 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06be90 0x7fd2dc06e350 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d620 0x7fd2fc155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0b5430 0x7fd2fc0cc230 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 --2- 172.21.15.107:0/3905978323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc0999d0 0x7fd2fc0ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 >> 172.21.15.107:0/3905978323 conn(0x7fd2fc06c090 msgr2=0x7fd2fc156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 shutdown_connections 2023-11-24T08:13:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:16.999+0000 7fd300883640 1 -- 172.21.15.107:0/3905978323 wait complete. 2023-11-24T08:13:17.020 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:13:17.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:17.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:17.136 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:13:17.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:17.137 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:13:17.191 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:17.191 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:17.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:13:17.254 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:17.254 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:17.313 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:13:17.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.535+0000 7f592ed83640 1 Processor -- start 2023-11-24T08:13:17.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.535+0000 7f592ed83640 1 -- start start 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.536+0000 7f592ed83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f592815c7f0 0x7f592806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.536+0000 7f592ed83640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59280cf450 0x7f59280cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.536+0000 7f592ed83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cdfb0 0x7f59280ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.536+0000 7f592ed83640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59280cfe40 con 0x7f592815c7f0 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592ed83640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5928156010 con 0x7f59280cf450 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592ed83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592806ce30 con 0x7f59280cdfb0 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f592815c7f0 0x7f592806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f592815c7f0 0x7f592806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41634/0 (socket says 172.21.15.107:41634) 2023-11-24T08:13:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 -- 172.21.15.107:0/2829388918 learned_addr learned my addr 172.21.15.107:0/2829388918 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 -- 172.21.15.107:0/2829388918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cdfb0 msgr2=0x7f59280ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 --2- 172.21.15.107:0/2829388918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cdfb0 0x7f59280ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 -- 172.21.15.107:0/2829388918 >> v2:172.21.15.120:3300/0 conn(0x7f59280cf450 msgr2=0x7f59280cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 --2- 172.21.15.107:0/2829388918 >> v2:172.21.15.120:3300/0 conn(0x7f59280cf450 0x7f59280cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.537+0000 7f592cb1f640 1 -- 172.21.15.107:0/2829388918 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59280cece0 con 0x7f592815c7f0 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.538+0000 7f592cb1f640 1 --2- 172.21.15.107:0/2829388918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f592806db20 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f5928070410 tx=0x7f591802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46458528e33d85e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.538+0000 7f59277fe640 1 -- 172.21.15.107:0/2829388918 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918031870 con 0x7f592815c7f0 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f59277fe640 1 -- 172.21.15.107:0/2829388918 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59180319d0 con 0x7f592815c7f0 2023-11-24T08:13:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f59277fe640 1 -- 172.21.15.107:0/2829388918 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918031d60 con 0x7f592815c7f0 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 -- 172.21.15.107:0/2829388918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 msgr2=0x7f592806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 --2- 172.21.15.107:0/2829388918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f592806db20 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f5928070410 tx=0x7f591802b910 comp rx=0 tx=0).stop 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 -- 172.21.15.107:0/2829388918 shutdown_connections 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 --2- 172.21.15.107:0/2829388918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cdfb0 0x7f59280ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 --2- 172.21.15.107:0/2829388918 >> v2:172.21.15.120:3300/0 conn(0x7f59280cf450 0x7f59280cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 --2- 172.21.15.107:0/2829388918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f592806db20 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 -- 172.21.15.107:0/2829388918 >> 172.21.15.107:0/2829388918 conn(0x7f592806a920 msgr2=0x7f592806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 -- 172.21.15.107:0/2829388918 shutdown_connections 2023-11-24T08:13:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.539+0000 7f592ed83640 1 -- 172.21.15.107:0/2829388918 wait complete. 2023-11-24T08:13:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.541+0000 7f592ed83640 1 Processor -- start 2023-11-24T08:13:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.541+0000 7f592ed83640 1 -- start start 2023-11-24T08:13:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59280cdfb0 0x7f59280ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cf450 0x7f5928149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5928149a40 con 0x7f592815c7f0 2023-11-24T08:13:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59280acfb0 con 0x7f59280cdfb0 2023-11-24T08:13:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.542+0000 7f592ed83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59280ad290 con 0x7f59280cf450 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41636/0 (socket says 172.21.15.107:41636) 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 -- 172.21.15.107:0/4151477984 learned_addr learned my addr 172.21.15.107:0/4151477984 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f5927fff640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cf450 0x7f5928149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592cb1f640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59280cdfb0 0x7f59280ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 -- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cf450 msgr2=0x7f5928149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cf450 0x7f5928149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 -- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59280cdfb0 msgr2=0x7f59280ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59280cdfb0 0x7f59280ac9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.543+0000 7f592d320640 1 -- 172.21.15.107:0/4151477984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180094d0 con 0x7f592815c7f0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f592d320640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f592000b3e0 tx=0x7f5920009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920011830 con 0x7f592815c7f0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5920011990 con 0x7f592815c7f0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920011d20 con 0x7f592815c7f0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59280ad570 con 0x7f592815c7f0 2023-11-24T08:13:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.544+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59281579d0 con 0x7f592815c7f0 2023-11-24T08:13:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.545+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928067470 con 0x7f592815c7f0 2023-11-24T08:13:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.546+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f592001a400 con 0x7f592815c7f0 2023-11-24T08:13:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.547+0000 7f59257fa640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 0x7f591406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.548+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59200885b0 con 0x7f592815c7f0 2023-11-24T08:13:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.548+0000 7f592cb1f640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 0x7f591406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.548+0000 7f592cb1f640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 0x7f591406e350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5918036000 tx=0x7f591803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:17.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.552+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5920054d90 con 0x7f592815c7f0 2023-11-24T08:13:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.784+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f592809d080 con 0x7f592815c7f0 2023-11-24T08:13:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.784+0000 7f59257fa640 1 -- 172.21.15.107:0/4151477984 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f5920015070 con 0x7f592815c7f0 2023-11-24T08:13:17.786 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:17.787 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-11-24T08:13:17.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 msgr2=0x7f591406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 0x7f591406e350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5918036000 tx=0x7f591803c040 comp rx=0 tx=0).stop 2023-11-24T08:13:17.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 msgr2=0x7f5928157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f592000b3e0 tx=0x7f5920009f90 comp rx=0 tx=0).stop 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 shutdown_connections 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f591406be90 0x7f591406e350 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f592815c7f0 0x7f5928157490 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59280cf450 0x7f5928149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 --2- 172.21.15.107:0/4151477984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59280cdfb0 0x7f59280ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 >> 172.21.15.107:0/4151477984 conn(0x7f592806a920 msgr2=0x7f59280ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 shutdown_connections 2023-11-24T08:13:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:17.787+0000 7f592ed83640 1 -- 172.21.15.107:0/4151477984 wait complete. 2023-11-24T08:13:17.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.019+0000 7ff27efa9640 1 Processor -- start 2023-11-24T08:13:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.019+0000 7ff27efa9640 1 -- start start 2023-11-24T08:13:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.020+0000 7ff27efa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff2780abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.020+0000 7ff27efa9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff2780ce640 0x7ff2780b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.020+0000 7ff27efa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2780ce270 con 0x7ff2780cc470 2023-11-24T08:13:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.020+0000 7ff27efa9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27806c560 con 0x7ff2780ce640 2023-11-24T08:13:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff27efa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806d0a0 con 0x7ff2780cfb60 2023-11-24T08:13:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff27cd45640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff2780cc470 0x7ff2780cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55622/0 (socket says 172.21.15.107:55622) 2023-11-24T08:13:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff27cd45640 1 -- 172.21.15.107:0/3940783309 learned_addr learned my addr 172.21.15.107:0/3940783309 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff277fff640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff2780abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff27d546640 1 --2- 172.21.15.107:0/3940783309 >> v2:172.21.15.120:3300/0 conn(0x7ff2780ce640 0x7ff2780b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258757808 0 0) 0x7ff2780ce270 con 0x7ff2780cc470 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.021+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268003520 con 0x7ff2780cc470 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3731823021 0 0) 0x7ff268003520 con 0x7ff2780cc470 2023-11-24T08:13:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 msgr2=0x7ff2780abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff2780abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 >> v2:172.21.15.120:3300/0 conn(0x7ff2780ce640 msgr2=0x7ff2780b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 --2- 172.21.15.107:0/3940783309 >> v2:172.21.15.120:3300/0 conn(0x7ff2780ce640 0x7ff2780b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780b4200 con 0x7ff2780cc470 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff26c001670 con 0x7ff2780cc470 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 >> v1:172.21.15.107:6789/0 conn(0x7ff2780cc470 legacy=0x7ff2780cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff268003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff268003f40 0x7ff2680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff268004d00 0x7ff268005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806c560 con 0x7ff268004d00 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806d0a0 con 0x7ff268003f40 2023-11-24T08:13:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800b980 con 0x7ff2780cfb60 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff27d546640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff268004d00 0x7ff268005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.022+0000 7ff27cd45640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff268003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff277fff640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff268003f40 0x7ff2680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff27d546640 1 -- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 msgr2=0x7ff268003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff27d546640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2780cfb60 0x7ff268003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff27d546640 1 -- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff268003f40 msgr2=0x7ff2680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff27d546640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff268003f40 0x7ff2680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.023+0000 7ff27d546640 1 -- 172.21.15.107:0/3940783309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268003520 con 0x7ff268004d00 2023-11-24T08:13:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.024+0000 7ff27d546640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff268004d00 0x7ff268005180 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7ff27000e170 tx=0x7ff2680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b8be16b8b089f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.024+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27002cc20 con 0x7ff268004d00 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.024+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000f040 con 0x7ff268004d00 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.024+0000 7ff2777fe640 1 -- 172.21.15.107:0/3940783309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270034440 con 0x7ff268004d00 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 -- 172.21.15.107:0/3940783309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff268004d00 msgr2=0x7ff268005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3940783309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff268004d00 0x7ff268005180 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7ff27000e170 tx=0x7ff2680057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27d546640 1 -- 172.21.15.107:0/3940783309 reap_dead start 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 -- 172.21.15.107:0/3940783309 shutdown_connections 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 -- 172.21.15.107:0/3940783309 >> 172.21.15.107:0/3940783309 conn(0x7ff27806a920 msgr2=0x7ff27806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 -- 172.21.15.107:0/3940783309 shutdown_connections 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.025+0000 7ff27efa9640 1 -- 172.21.15.107:0/3940783309 wait complete. 2023-11-24T08:13:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.027+0000 7ff27efa9640 1 Processor -- start 2023-11-24T08:13:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.027+0000 7ff27efa9640 1 -- start start 2023-11-24T08:13:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff268004d00 0x7ff27815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27815d750 0x7ff27815dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27814cb10 con 0x7ff27814be20 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278165050 con 0x7ff27815d750 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff27efa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff278164a90 con 0x7ff268004d00 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff277fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff277fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36250/0 (socket says 172.21.15.107:36250) 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.028+0000 7ff277fff640 1 -- 172.21.15.107:0/3979097386 learned_addr learned my addr 172.21.15.107:0/3979097386 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff27cd45640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff268004d00 0x7ff27815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff27d546640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27815d750 0x7ff27815dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 -- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff268004d00 msgr2=0x7ff27815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff268004d00 0x7ff27815cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 -- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27815d750 msgr2=0x7ff27815dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27815d750 0x7ff27815dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 -- 172.21.15.107:0/3979097386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278164d70 con 0x7ff27814be20 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff277fff640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7ff264011540 tx=0x7ff264011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.029+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264015930 con 0x7ff27814be20 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.030+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff264015a90 con 0x7ff27814be20 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.030+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27814c7a0 con 0x7ff27814be20 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.030+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264015e60 con 0x7ff27814be20 2023-11-24T08:13:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.030+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27814e700 con 0x7ff27814be20 2023-11-24T08:13:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.031+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278067470 con 0x7ff27814be20 2023-11-24T08:13:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.032+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7ff26400f460 con 0x7ff27814be20 2023-11-24T08:13:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.033+0000 7ff2757fa640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 0x7ff25c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.033+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff264088c60 con 0x7ff27814be20 2023-11-24T08:13:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.033+0000 7ff27cd45640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 0x7ff25c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.034+0000 7ff27cd45640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 0x7ff25c06e350 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff26c001050 tx=0x7ff26c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.037+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2640bee90 con 0x7ff27814be20 2023-11-24T08:13:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.228+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2780cd910 con 0x7ff27814be20 2023-11-24T08:13:18.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.229+0000 7ff2757fa640 1 -- 172.21.15.107:0/3979097386 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7ff264022090 con 0x7ff27814be20 2023-11-24T08:13:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:18.232 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:18.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 msgr2=0x7ff25c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 0x7ff25c06e350 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff26c001050 tx=0x7ff26c009040 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 msgr2=0x7ff27815d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7ff264011540 tx=0x7ff264011a00 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 shutdown_connections 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff25c06be90 0x7ff25c06e350 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27815d750 0x7ff27815dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27814be20 0x7ff27815d210 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 --2- 172.21.15.107:0/3979097386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff268004d00 0x7ff27815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.232+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 >> 172.21.15.107:0/3979097386 conn(0x7ff27806a920 msgr2=0x7ff2780abfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.233+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 shutdown_connections 2023-11-24T08:13:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.233+0000 7ff27efa9640 1 -- 172.21.15.107:0/3979097386 wait complete. 2023-11-24T08:13:18.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:12:56.294075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:12:57.505929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:13:18.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:13:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.471+0000 7fb266c90640 1 Processor -- start 2023-11-24T08:13:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.471+0000 7fb266c90640 1 -- start start 2023-11-24T08:13:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb266c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb26015f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb266c90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26013eb00 con 0x7fb26006ff70 2023-11-24T08:13:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb266c90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26013ede0 con 0x7fb2600cf4f0 2023-11-24T08:13:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb266c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26015fe50 con 0x7fb26015f530 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb257fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb26006ff70 0x7fb2600cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55624/0 (socket says 172.21.15.107:55624) 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.473+0000 7fb257fff640 1 -- 172.21.15.107:0/3572832624 learned_addr learned my addr 172.21.15.107:0/3572832624 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821255889 0 0) 0x7fb26013eb00 con 0x7fb26006ff70 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb264a2c640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb26015f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb248003520 con 0x7fb26006ff70 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959797774 0 0) 0x7fb248003520 con 0x7fb26006ff70 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 msgr2=0x7fb26015f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb26015f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 >> v1:172.21.15.120:6789/0 conn(0x7fb2600cf4f0 legacy=0x7fb26013f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb260070750 con 0x7fb26006ff70 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb250001670 con 0x7fb26006ff70 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 >> v1:172.21.15.107:6789/0 conn(0x7fb26006ff70 legacy=0x7fb2600cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2480039a0 0x7fb248003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb2480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb248004d00 0x7fb248005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26015fe50 con 0x7fb2480039a0 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800bab0 con 0x7fb248004d00 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.474+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800bc70 con 0x7fb26015f530 2023-11-24T08:13:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2480039a0 0x7fb248003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 -- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 msgr2=0x7fb2480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb257fff640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb2480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb26015f530 0x7fb2480047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 -- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb248004d00 msgr2=0x7fb248005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb248004d00 0x7fb248005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.475+0000 7fb264a2c640 1 -- 172.21.15.107:0/3572832624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248003520 con 0x7fb2480039a0 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.476+0000 7fb264a2c640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2480039a0 0x7fb248003dc0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009fc0 tx=0x7fb2480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c27ccc3015c51f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.476+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c02cb40 con 0x7fb2480039a0 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.476+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c00f040 con 0x7fb2480039a0 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.476+0000 7fb2577fe640 1 -- 172.21.15.107:0/3572832624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c034440 con 0x7fb2480039a0 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 -- 172.21.15.107:0/3572832624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2480039a0 msgr2=0x7fb248003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 --2- 172.21.15.107:0/3572832624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2480039a0 0x7fb248003dc0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009fc0 tx=0x7fb2480057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb26522d640 1 -- 172.21.15.107:0/3572832624 reap_dead start 2023-11-24T08:13:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 -- 172.21.15.107:0/3572832624 shutdown_connections 2023-11-24T08:13:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 -- 172.21.15.107:0/3572832624 >> 172.21.15.107:0/3572832624 conn(0x7fb2600b2e40 msgr2=0x7fb2600b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 -- 172.21.15.107:0/3572832624 shutdown_connections 2023-11-24T08:13:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.477+0000 7fb266c90640 1 -- 172.21.15.107:0/3572832624 wait complete. 2023-11-24T08:13:18.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.479+0000 7fb266c90640 1 Processor -- start 2023-11-24T08:13:18.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.479+0000 7fb266c90640 1 -- start start 2023-11-24T08:13:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2480039a0 0x7fb2600cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb26013e500 0x7fb2600b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600ccc30 con 0x7fb2600b44e0 2023-11-24T08:13:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600b5ba0 con 0x7fb26013e500 2023-11-24T08:13:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb266c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2601f6b60 con 0x7fb2480039a0 2023-11-24T08:13:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb26522d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb26522d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36276/0 (socket says 172.21.15.107:36276) 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb26522d640 1 -- 172.21.15.107:0/1443895365 learned_addr learned my addr 172.21.15.107:0/1443895365 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.480+0000 7fb264a2c640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2480039a0 0x7fb2600cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb257fff640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb26013e500 0x7fb2600b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 -- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2480039a0 msgr2=0x7fb2600cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2480039a0 0x7fb2600cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 -- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb26013e500 msgr2=0x7fb2600b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb26013e500 0x7fb2600b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 -- 172.21.15.107:0/1443895365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2601f6e40 con 0x7fb2600b44e0 2023-11-24T08:13:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb26522d640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b0d50 tx=0x7fb25800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2580072e0 con 0x7fb2600b44e0 2023-11-24T08:13:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb258007440 con 0x7fb2600b44e0 2023-11-24T08:13:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2601f7120 con 0x7fb2600b44e0 2023-11-24T08:13:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258004020 con 0x7fb2600b44e0 2023-11-24T08:13:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.481+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26013e960 con 0x7fb2600b44e0 2023-11-24T08:13:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.483+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2600b5df0 con 0x7fb2600b44e0 2023-11-24T08:13:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.484+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb2580041a0 con 0x7fb2600b44e0 2023-11-24T08:13:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.484+0000 7fb2557fa640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 0x7fb24c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.485+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb258087540 con 0x7fb2600b44e0 2023-11-24T08:13:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.485+0000 7fb264a2c640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 0x7fb24c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.485+0000 7fb264a2c640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 0x7fb24c06e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb25c036000 tx=0x7fb25c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.489+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2600b5df0 con 0x7fb2600b44e0 2023-11-24T08:13:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.678+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2600b5df0 con 0x7fb2600b44e0 2023-11-24T08:13:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.679+0000 7fb2557fa640 1 -- 172.21.15.107:0/1443895365 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2600b5df0 con 0x7fb2600b44e0 2023-11-24T08:13:18.681 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.681+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 msgr2=0x7fb24c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.681+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 0x7fb24c06e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb25c036000 tx=0x7fb25c02e000 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 msgr2=0x7fb2600cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b0d50 tx=0x7fb25800c820 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 shutdown_connections 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24c06be90 0x7fb24c06e350 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2600b44e0 0x7fb2600cdb40 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb26013e500 0x7fb2600b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 --2- 172.21.15.107:0/1443895365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2480039a0 0x7fb2600cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 >> 172.21.15.107:0/1443895365 conn(0x7fb2600b2e40 msgr2=0x7fb2600c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 shutdown_connections 2023-11-24T08:13:18.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.682+0000 7fb266c90640 1 -- 172.21.15.107:0/1443895365 wait complete. 2023-11-24T08:13:18.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:13:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.919+0000 7f00b7061640 1 Processor -- start 2023-11-24T08:13:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.919+0000 7f00b7061640 1 -- start start 2023-11-24T08:13:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.920+0000 7f00b7061640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00b00d4010 0x7f00b00cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.920+0000 7f00b7061640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00cf300 con 0x7f00b015d580 2023-11-24T08:13:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.920+0000 7f00b7061640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00d1090 con 0x7f00b00cfc80 2023-11-24T08:13:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.920+0000 7f00b7061640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b00d1370 con 0x7f00b00d4010 2023-11-24T08:13:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.920+0000 7f00b4dfd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f00b015d580 0x7f00b00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55634/0 (socket says 172.21.15.107:55634) 2023-11-24T08:13:18.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00b4dfd640 1 -- 172.21.15.107:0/2643216005 learned_addr learned my addr 172.21.15.107:0/2643216005 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00affff640 1 --2- 172.21.15.107:0/2643216005 >> v2:172.21.15.120:3300/0 conn(0x7f00b00d4010 0x7f00b00cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039919146 0 0) 0x7f00b00cf300 con 0x7f00b015d580 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c003520 con 0x7f00b015d580 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431534625 0 0) 0x7f009c003520 con 0x7f00b015d580 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 >> v2:172.21.15.120:3300/0 conn(0x7f00b00d4010 msgr2=0x7f00b00cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 --2- 172.21.15.107:0/2643216005 >> v2:172.21.15.120:3300/0 conn(0x7f00b00d4010 0x7f00b00cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 >> v1:172.21.15.120:6789/0 conn(0x7f00b00cfc80 legacy=0x7f00b00d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00d1650 con 0x7f00b015d580 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.921+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f00a00019f0 con 0x7f00b015d580 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 >> v1:172.21.15.107:6789/0 conn(0x7f00b015d580 legacy=0x7f00b00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 0x7f009c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0047c0 0x7f009c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c005180 0x7f009c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00d1370 con 0x7f009c0039c0 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00c080 con 0x7f009c005180 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00c240 con 0x7f009c0047c0 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 0x7f009c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 -- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0047c0 msgr2=0x7f009c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0047c0 0x7f009c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 -- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c005180 msgr2=0x7f009c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:18.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c005180 0x7f009c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b4dfd640 1 -- 172.21.15.107:0/2643216005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c003520 con 0x7f009c0039c0 2023-11-24T08:13:18.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.922+0000 7f00b55fe640 1 -- 172.21.15.107:0/2643216005 reap_dead start 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 0x7f009c003de0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f00a000a760 tx=0x7f009c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46efbc30ae9a86e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0034630 con 0x7f009c0039c0 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a002f070 con 0x7f009c0039c0 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00af7fe640 1 -- 172.21.15.107:0/2643216005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a00349f0 con 0x7f009c0039c0 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b7061640 1 -- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 msgr2=0x7f009c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b7061640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 0x7f009c003de0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f00a000a760 tx=0x7f009c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b7061640 1 -- 172.21.15.107:0/2643216005 shutdown_connections 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b7061640 1 --2- 172.21.15.107:0/2643216005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009c0039c0 0x7f009c003de0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.924+0000 7f00b7061640 1 -- 172.21.15.107:0/2643216005 >> 172.21.15.107:0/2643216005 conn(0x7f00b009c770 msgr2=0x7f00b00d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.925+0000 7f00b7061640 1 -- 172.21.15.107:0/2643216005 shutdown_connections 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.925+0000 7f00b7061640 1 -- 172.21.15.107:0/2643216005 wait complete. 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.926+0000 7f00b7061640 1 Processor -- start 2023-11-24T08:13:18.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.927+0000 7f00b7061640 1 -- start start 2023-11-24T08:13:18.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.927+0000 7f00b7061640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0039c0 0x7f00b00ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.927+0000 7f00b7061640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 0x7f00b0158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b7061640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b7061640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00ca4f0 con 0x7f00b01595b0 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b7061640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0158ef0 con 0x7f00b0159150 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b7061640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0159b80 con 0x7f009c0039c0 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36298/0 (socket says 172.21.15.107:36298) 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 -- 172.21.15.107:0/3244029708 learned_addr learned my addr 172.21.15.107:0/3244029708 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0039c0 0x7f00b00ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00affff640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 0x7f00b0158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 -- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0039c0 msgr2=0x7f00b00ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0039c0 0x7f00b00ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 -- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 msgr2=0x7f00b0158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 0x7f00b0158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00b55fe640 1 -- 172.21.15.107:0/3244029708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b0159e60 con 0x7f00b01595b0 2023-11-24T08:13:18.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.928+0000 7f00affff640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 0x7f00b0158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:18.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00b55fe640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f00b00d25d0 tx=0x7f00a800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8007540 con 0x7f00b01595b0 2023-11-24T08:13:18.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a80057c0 con 0x7f00b01595b0 2023-11-24T08:13:18.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8005b70 con 0x7f00b01595b0 2023-11-24T08:13:18.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b015a140 con 0x7f00b01595b0 2023-11-24T08:13:18.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.929+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b00cc880 con 0x7f00b01595b0 2023-11-24T08:13:18.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.930+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b00ca980 con 0x7f00b01595b0 2023-11-24T08:13:18.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.931+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f00a8005cd0 con 0x7f00b01595b0 2023-11-24T08:13:18.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.932+0000 7f00ad7fa640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 0x7f009006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:18.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.932+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00a80876e0 con 0x7f00b01595b0 2023-11-24T08:13:18.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.932+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 0x7f009006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:18.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.933+0000 7f00b4dfd640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 0x7f009006e350 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f00a0034000 tx=0x7f00a002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:18.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:18.937+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00b00ca980 con 0x7f00b01595b0 2023-11-24T08:13:19.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.125+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00b00ca980 con 0x7f00b01595b0 2023-11-24T08:13:19.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.126+0000 7f00ad7fa640 1 -- 172.21.15.107:0/3244029708 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00b00ca980 con 0x7f00b01595b0 2023-11-24T08:13:19.128 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 msgr2=0x7f009006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 0x7f009006e350 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f00a0034000 tx=0x7f00a002f000 comp rx=0 tx=0).stop 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 msgr2=0x7f00b01589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f00b00d25d0 tx=0x7f00a800c820 comp rx=0 tx=0).stop 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 shutdown_connections 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f009006be90 0x7f009006e350 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00b01595b0 0x7f00b01589b0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00b0159150 0x7f00b0158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 --2- 172.21.15.107:0/3244029708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009c0039c0 0x7f00b00ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 >> 172.21.15.107:0/3244029708 conn(0x7f00b009c770 msgr2=0x7f00b00b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 shutdown_connections 2023-11-24T08:13:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.129+0000 7f00b7061640 1 -- 172.21.15.107:0/3244029708 wait complete. 2023-11-24T08:13:19.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:13:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.365+0000 7fb2675e5640 1 Processor -- start 2023-11-24T08:13:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.365+0000 7fb2675e5640 1 -- start start 2023-11-24T08:13:19.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb2675e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb26015b440 0x7fb2600cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb2675e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb2600c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb2675e5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26015eea0 con 0x7fb2600c57e0 2023-11-24T08:13:19.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb2675e5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2600cc3d0 con 0x7fb26015b440 2023-11-24T08:13:19.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb2675e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26015b820 con 0x7fb2600cbf20 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb265381640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb26015b440 0x7fb2600cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb265381640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb26015b440 0x7fb2600cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36308/0 (socket says 172.21.15.107:36308) 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb265381640 1 -- 172.21.15.107:0/518960612 learned_addr learned my addr 172.21.15.107:0/518960612 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.367+0000 7fb265b82640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb2600c57e0 0x7fb2600c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55636/0 (socket says 172.21.15.107:55636) 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb264b80640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb2600c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb265381640 1 -- 172.21.15.107:0/518960612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 msgr2=0x7fb2600c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:19.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb265381640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb2600c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb265381640 1 -- 172.21.15.107:0/518960612 >> v1:172.21.15.107:6789/0 conn(0x7fb2600c57e0 legacy=0x7fb2600c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb265381640 1 -- 172.21.15.107:0/518960612 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26006d600 con 0x7fb26015b440 2023-11-24T08:13:19.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.368+0000 7fb25ffff640 1 -- 172.21.15.107:0/518960612 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403525052 0 0) 0x7fb26015eea0 con 0x7fb2600c57e0 2023-11-24T08:13:19.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.369+0000 7fb265381640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb2600cb9e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb254009e50 tx=0x7fb25402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4f3209a982b75b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.369+0000 7fb25ffff640 1 -- 172.21.15.107:0/518960612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25402ca70 con 0x7fb26015b440 2023-11-24T08:13:19.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.369+0000 7fb25ffff640 1 -- 172.21.15.107:0/518960612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25402cbd0 con 0x7fb26015b440 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.369+0000 7fb25ffff640 1 -- 172.21.15.107:0/518960612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2540343f0 con 0x7fb26015b440 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 -- 172.21.15.107:0/518960612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 msgr2=0x7fb2600cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb2600cb9e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb254009e50 tx=0x7fb25402bed0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 -- 172.21.15.107:0/518960612 shutdown_connections 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb2600c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 --2- 172.21.15.107:0/518960612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb2600cb9e0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 -- 172.21.15.107:0/518960612 >> 172.21.15.107:0/518960612 conn(0x7fb2600b0c50 msgr2=0x7fb2600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 -- 172.21.15.107:0/518960612 shutdown_connections 2023-11-24T08:13:19.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.370+0000 7fb2675e5640 1 -- 172.21.15.107:0/518960612 wait complete. 2023-11-24T08:13:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.372+0000 7fb2675e5640 1 Processor -- start 2023-11-24T08:13:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.372+0000 7fb2675e5640 1 -- start start 2023-11-24T08:13:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2600c57e0 0x7fb260099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb26009a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600b3bd0 con 0x7fb26015b440 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600b3e20 con 0x7fb2600c57e0 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb2675e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600b4100 con 0x7fb2600cbf20 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb265b82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb265b82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36322/0 (socket says 172.21.15.107:36322) 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.373+0000 7fb265b82640 1 -- 172.21.15.107:0/2801358211 learned_addr learned my addr 172.21.15.107:0/2801358211 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265381640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2600c57e0 0x7fb260099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb264b80640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb26009a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 -- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 msgr2=0x7fb26009a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb26009a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 -- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2600c57e0 msgr2=0x7fb260099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2600c57e0 0x7fb260099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 -- 172.21.15.107:0/2801358211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb254009b30 con 0x7fb26015b440 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb265b82640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb25800c450 tx=0x7fb25800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2580074e0 con 0x7fb26015b440 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb258007660 con 0x7fb26015b440 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258004a20 con 0x7fb26015b440 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.374+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2600b43e0 con 0x7fb26015b440 2023-11-24T08:13:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.375+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb260157280 con 0x7fb26015b440 2023-11-24T08:13:19.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.376+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2600c7190 con 0x7fb26015b440 2023-11-24T08:13:19.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.377+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb258004b80 con 0x7fb26015b440 2023-11-24T08:13:19.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.378+0000 7fb25dffb640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 0x7fb24406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.378+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2580875f0 con 0x7fb26015b440 2023-11-24T08:13:19.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.378+0000 7fb265381640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 0x7fb24406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.379+0000 7fb265381640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 0x7fb24406e350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb254035000 tx=0x7fb254030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.382+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb258053dd0 con 0x7fb26015b440 2023-11-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.575+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb26009d070 con 0x7fb26015b440 2023-11-24T08:13:19.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.576+0000 7fb25dffb640 1 -- 172.21.15.107:0/2801358211 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb258018070 con 0x7fb26015b440 2023-11-24T08:13:19.578 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.578+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 msgr2=0x7fb24406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.578+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 0x7fb24406e350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb254035000 tx=0x7fb254030000 comp rx=0 tx=0).stop 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.578+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 msgr2=0x7fb26009a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.578+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb25800c450 tx=0x7fb25800c910 comp rx=0 tx=0).stop 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 shutdown_connections 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb24406be90 0x7fb24406e350 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb26015b440 0x7fb26009a920 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2600cbf20 0x7fb26009a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 --2- 172.21.15.107:0/2801358211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2600c57e0 0x7fb260099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 >> 172.21.15.107:0/2801358211 conn(0x7fb2600b0c50 msgr2=0x7fb2600c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 shutdown_connections 2023-11-24T08:13:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.579+0000 7fb2675e5640 1 -- 172.21.15.107:0/2801358211 wait complete. 2023-11-24T08:13:19.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:19.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.811+0000 7f994d2c9640 1 Processor -- start 2023-11-24T08:13:19.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.811+0000 7f994d2c9640 1 -- start start 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.812+0000 7f994d2c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f994806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.812+0000 7f994d2c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99480cf450 0x7f99480cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f994d2c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480cece0 con 0x7f99480cdfb0 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f994d2c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9948156010 con 0x7f99480cf450 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f994d2c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994806ce30 con 0x7f994815c7f0 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99480cf450 0x7f99480cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99480cf450 0x7f99480cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36336/0 (socket says 172.21.15.107:36336) 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 -- 172.21.15.107:0/3647558092 learned_addr learned my addr 172.21.15.107:0/3647558092 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f9946ffd640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f994806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 -- 172.21.15.107:0/3647558092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 msgr2=0x7f994806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f994806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 -- 172.21.15.107:0/3647558092 >> v1:172.21.15.120:6789/0 conn(0x7f99480cdfb0 legacy=0x7f99480ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.813+0000 7f99467fc640 1 -- 172.21.15.107:0/3647558092 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994806d040 con 0x7f99480cf450 2023-11-24T08:13:19.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.815+0000 7f99467fc640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480cf8d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f993c009870 tx=0x7f993c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8a43d1caad7740 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.815+0000 7f9945ffb640 1 -- 172.21.15.107:0/3647558092 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c031640 con 0x7f99480cf450 2023-11-24T08:13:19.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.815+0000 7f9945ffb640 1 -- 172.21.15.107:0/3647558092 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c0317a0 con 0x7f99480cf450 2023-11-24T08:13:19.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.815+0000 7f9945ffb640 1 -- 172.21.15.107:0/3647558092 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c031b30 con 0x7f99480cf450 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 -- 172.21.15.107:0/3647558092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 msgr2=0x7f99480cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480cf8d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f993c009870 tx=0x7f993c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 -- 172.21.15.107:0/3647558092 shutdown_connections 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480cf8d0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3647558092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f994806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 -- 172.21.15.107:0/3647558092 >> 172.21.15.107:0/3647558092 conn(0x7f994806a920 msgr2=0x7f994806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 -- 172.21.15.107:0/3647558092 shutdown_connections 2023-11-24T08:13:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.816+0000 7f994d2c9640 1 -- 172.21.15.107:0/3647558092 wait complete. 2023-11-24T08:13:19.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.818+0000 7f994d2c9640 1 Processor -- start 2023-11-24T08:13:19.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.818+0000 7f994d2c9640 1 -- start start 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99480cdfb0 0x7f9948158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f9948071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99481f3fc0 con 0x7f99480cf450 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99481f3620 con 0x7f99480cdfb0 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f994d2c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99481f3900 con 0x7f994815c7f0 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f99467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f99467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36350/0 (socket says 172.21.15.107:36350) 2023-11-24T08:13:19.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f99467fc640 1 -- 172.21.15.107:0/3907453898 learned_addr learned my addr 172.21.15.107:0/3907453898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f9946ffd640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99480cdfb0 0x7f9948158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.819+0000 7f99477fe640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f9948071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 -- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 msgr2=0x7f9948071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f9948071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 -- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99480cdfb0 msgr2=0x7f9948158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99480cdfb0 0x7f9948158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 -- 172.21.15.107:0/3907453898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c0094d0 con 0x7f99480cf450 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99477fe640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f9948071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f99467fc640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c03ac60 con 0x7f99480cf450 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c03adc0 con 0x7f99480cf450 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c044630 con 0x7f99480cf450 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99481f3be0 con 0x7f99480cf450 2023-11-24T08:13:19.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.820+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99481f3270 con 0x7f99480cf450 2023-11-24T08:13:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.822+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99481f3d70 con 0x7f99480cf450 2023-11-24T08:13:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.823+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f993c02f050 con 0x7f99480cf450 2023-11-24T08:13:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.823+0000 7f992f7fe640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 0x7f993406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.824+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f993c0aa2f0 con 0x7f99480cf450 2023-11-24T08:13:19.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.824+0000 7f9946ffd640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 0x7f993406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:19.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.824+0000 7f9946ffd640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 0x7f993406e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9938006fd0 tx=0x7f9938008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:19.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:19.828+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f993c076b30 con 0x7f99480cf450 2023-11-24T08:13:20.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.064+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99481f4100 con 0x7f99480cf450 2023-11-24T08:13:20.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.065+0000 7f992f7fe640 1 -- 172.21.15.107:0/3907453898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f99481f4100 con 0x7f99480cf450 2023-11-24T08:13:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:20.067 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.067+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 msgr2=0x7f993406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.067+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 0x7f993406e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9938006fd0 tx=0x7f9938008040 comp rx=0 tx=0).stop 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 msgr2=0x7f99480718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 shutdown_connections 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f993406be90 0x7f993406e350 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994815c7f0 0x7f9948071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99480cf450 0x7f99480718b0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 --2- 172.21.15.107:0/3907453898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99480cdfb0 0x7f9948158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 >> 172.21.15.107:0/3907453898 conn(0x7f994806a920 msgr2=0x7f9948073530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 shutdown_connections 2023-11-24T08:13:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.068+0000 7f994d2c9640 1 -- 172.21.15.107:0/3907453898 wait complete. 2023-11-24T08:13:20.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:20.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.296+0000 7f1b5ad61640 1 Processor -- start 2023-11-24T08:13:20.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.297+0000 7f1b5ad61640 1 -- start start 2023-11-24T08:13:20.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.298+0000 7f1b5ad61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b5415eaf0 0x7f1b540b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.298+0000 7f1b5ad61640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b540b3890 0x7f1b54167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.298+0000 7f1b5ad61640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5406d180 con 0x7f1b541461d0 2023-11-24T08:13:20.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.298+0000 7f1b5ad61640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b540b3ce0 con 0x7f1b540b3890 2023-11-24T08:13:20.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.298+0000 7f1b5ad61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406ba70 con 0x7f1b5415eaf0 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.299+0000 7f1b592fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b540b3890 0x7f1b54167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.299+0000 7f1b592fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b540b3890 0x7f1b54167110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:37580/0 (socket says 172.21.15.107:37580) 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.299+0000 7f1b53fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b5415eaf0 0x7f1b540b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.299+0000 7f1b592fe640 1 -- 172.21.15.107:0/3131503870 learned_addr learned my addr 172.21.15.107:0/3131503870 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.300+0000 7f1b592fe640 1 -- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b5415eaf0 msgr2=0x7f1b540b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.300+0000 7f1b592fe640 1 --2- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b5415eaf0 0x7f1b540b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.300+0000 7f1b592fe640 1 -- 172.21.15.107:0/3131503870 >> v1:172.21.15.120:6789/0 conn(0x7f1b541461d0 legacy=0x7f1b540b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.300+0000 7f1b592fe640 1 -- 172.21.15.107:0/3131503870 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5406bd50 con 0x7f1b540b3890 2023-11-24T08:13:20.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.301+0000 7f1b592fe640 1 --2- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b540b3890 0x7f1b54167110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009870 tx=0x7f1b4c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=240ce79f86423ae2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.301+0000 7f1b537fe640 1 -- 172.21.15.107:0/3131503870 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c030840 con 0x7f1b540b3890 2023-11-24T08:13:20.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.301+0000 7f1b537fe640 1 -- 172.21.15.107:0/3131503870 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c0309a0 con 0x7f1b540b3890 2023-11-24T08:13:20.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.301+0000 7f1b537fe640 1 -- 172.21.15.107:0/3131503870 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c030d30 con 0x7f1b540b3890 2023-11-24T08:13:20.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 -- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b540b3890 msgr2=0x7f1b54167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b540b3890 0x7f1b54167110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009870 tx=0x7f1b4c02b980 comp rx=0 tx=0).stop 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 -- 172.21.15.107:0/3131503870 shutdown_connections 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b540b3890 0x7f1b54167110 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/3131503870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b5415eaf0 0x7f1b540b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 -- 172.21.15.107:0/3131503870 >> 172.21.15.107:0/3131503870 conn(0x7f1b54061340 msgr2=0x7f1b54061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 -- 172.21.15.107:0/3131503870 shutdown_connections 2023-11-24T08:13:20.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.302+0000 7f1b5ad61640 1 -- 172.21.15.107:0/3131503870 wait complete. 2023-11-24T08:13:20.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.304+0000 7f1b5ad61640 1 Processor -- start 2023-11-24T08:13:20.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.304+0000 7f1b5ad61640 1 -- start start 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b540b3890 0x7f1b540d0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b541461d0 0x7f1b5415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540cf4e0 con 0x7f1b5415eaf0 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5415c7f0 con 0x7f1b541461d0 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.305+0000 7f1b5ad61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540ce9a0 con 0x7f1b540b3890 2023-11-24T08:13:20.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36362/0 (socket says 172.21.15.107:36362) 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 -- 172.21.15.107:0/815029062 learned_addr learned my addr 172.21.15.107:0/815029062 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b53fff640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b541461d0 0x7f1b5415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b58afd640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b540b3890 0x7f1b540d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 -- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b540b3890 msgr2=0x7f1b540d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b540b3890 0x7f1b540d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 -- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b541461d0 msgr2=0x7f1b5415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b541461d0 0x7f1b5415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 -- 172.21.15.107:0/815029062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0094d0 con 0x7f1b5415eaf0 2023-11-24T08:13:20.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.306+0000 7f1b592fe640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009870 tx=0x7f1b4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.307+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c039c90 con 0x7f1b5415eaf0 2023-11-24T08:13:20.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.307+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c039e10 con 0x7f1b5415eaf0 2023-11-24T08:13:20.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.307+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c043640 con 0x7f1b5415eaf0 2023-11-24T08:13:20.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.307+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b540cec40 con 0x7f1b5415eaf0 2023-11-24T08:13:20.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.307+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b540cf200 con 0x7f1b5415eaf0 2023-11-24T08:13:20.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.308+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b541605b0 con 0x7f1b5415eaf0 2023-11-24T08:13:20.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.309+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f1b4c0437a0 con 0x7f1b5415eaf0 2023-11-24T08:13:20.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.310+0000 7f1b517fa640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 0x7f1b3406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.310+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b4c03e070 con 0x7f1b5415eaf0 2023-11-24T08:13:20.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.310+0000 7f1b58afd640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 0x7f1b3406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.311+0000 7f1b58afd640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 0x7f1b3406e350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1b440012e0 tx=0x7f1b44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.315+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b4c077c80 con 0x7f1b5415eaf0 2023-11-24T08:13:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.510+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b54067470 con 0x7f1b5415eaf0 2023-11-24T08:13:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.513+0000 7f1b517fa640 1 -- 172.21.15.107:0/815029062 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f1b4c034020 con 0x7f1b5415eaf0 2023-11-24T08:13:20.515 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:20.515 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.515+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 msgr2=0x7f1b3406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.515+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 0x7f1b3406e350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1b440012e0 tx=0x7f1b44009040 comp rx=0 tx=0).stop 2023-11-24T08:13:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.515+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 msgr2=0x7f1b540ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.515+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009870 tx=0x7f1b4c004730 comp rx=0 tx=0).stop 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 shutdown_connections 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b3406be90 0x7f1b3406e350 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5415eaf0 0x7f1b540ce460 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b541461d0 0x7f1b5415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 --2- 172.21.15.107:0/815029062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b540b3890 0x7f1b540d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 >> 172.21.15.107:0/815029062 conn(0x7f1b54061340 msgr2=0x7f1b541607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:20.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 shutdown_connections 2023-11-24T08:13:20.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.516+0000 7f1b5ad61640 1 -- 172.21.15.107:0/815029062 wait complete. 2023-11-24T08:13:20.537 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:13:20.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.744+0000 7f83d8007640 1 Processor -- start 2023-11-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.744+0000 7f83d8007640 1 -- start start 2023-11-24T08:13:20.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.745+0000 7f83d8007640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83d015df80 0x7f83d00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.745+0000 7f83d8007640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 0x7f83d015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.745+0000 7f83d8007640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d015e3f0 con 0x7f83d00b2250 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.745+0000 7f83d8007640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d015a990 con 0x7f83d015df80 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.745+0000 7f83d8007640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d015ab80 con 0x7f83d00cd470 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d55a2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83d015df80 0x7f83d00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d5da3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f83d00b2250 0x7f83d00b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55642/0 (socket says 172.21.15.107:55642) 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d5da3640 1 -- 172.21.15.107:0/2768880829 learned_addr learned my addr 172.21.15.107:0/2768880829 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d65a4640 1 --2- 172.21.15.107:0/2768880829 >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 0x7f83d015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d55a2640 1 -- 172.21.15.107:0/2768880829 >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 msgr2=0x7f83d015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d55a2640 1 --2- 172.21.15.107:0/2768880829 >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 0x7f83d015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d55a2640 1 -- 172.21.15.107:0/2768880829 >> v1:172.21.15.107:6789/0 conn(0x7f83d00b2250 legacy=0x7f83d00b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d4da1640 1 -- 172.21.15.107:0/2768880829 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273326690 0 0) 0x7f83d015e3f0 con 0x7f83d00b2250 2023-11-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d55a2640 1 -- 172.21.15.107:0/2768880829 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00b2d40 con 0x7f83d015df80 2023-11-24T08:13:20.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.746+0000 7f83d65a4640 1 --2- 172.21.15.107:0/2768880829 >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 0x7f83d015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.747+0000 7f83d55a2640 1 --2- 172.21.15.107:0/2768880829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d00ccf30 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f83d00b2fa0 tx=0x7f83c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8ebe5fe2b28fb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.747+0000 7f83d4da1640 1 -- 172.21.15.107:0/2768880829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031830 con 0x7f83d015df80 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d4da1640 1 -- 172.21.15.107:0/2768880829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0031990 con 0x7f83d015df80 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d4da1640 1 -- 172.21.15.107:0/2768880829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031d20 con 0x7f83d015df80 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 -- 172.21.15.107:0/2768880829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 msgr2=0x7f83d00ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 --2- 172.21.15.107:0/2768880829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d00ccf30 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f83d00b2fa0 tx=0x7f83c002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 -- 172.21.15.107:0/2768880829 shutdown_connections 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 --2- 172.21.15.107:0/2768880829 >> v2:172.21.15.120:3300/0 conn(0x7f83d00cd470 0x7f83d015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 --2- 172.21.15.107:0/2768880829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d00ccf30 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 -- 172.21.15.107:0/2768880829 >> 172.21.15.107:0/2768880829 conn(0x7f83d006a920 msgr2=0x7f83d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 -- 172.21.15.107:0/2768880829 shutdown_connections 2023-11-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.748+0000 7f83d8007640 1 -- 172.21.15.107:0/2768880829 wait complete. 2023-11-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.750+0000 7f83d8007640 1 Processor -- start 2023-11-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.750+0000 7f83d8007640 1 -- start start 2023-11-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 0x7f83d014b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d00cd470 0x7f83d014c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d014cd20 con 0x7f83d015df80 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d0151590 con 0x7f83d00b2250 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d8007640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d0151870 con 0x7f83d00cd470 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d65a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d65a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36384/0 (socket says 172.21.15.107:36384) 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.751+0000 7f83d65a4640 1 -- 172.21.15.107:0/725250749 learned_addr learned my addr 172.21.15.107:0/725250749 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d5da3640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 0x7f83d014b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 -- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d00cd470 msgr2=0x7f83d014c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d55a2640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d00cd470 0x7f83d014c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d00cd470 0x7f83d014c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 -- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 msgr2=0x7f83d014b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 0x7f83d014b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 -- 172.21.15.107:0/725250749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c0009400 con 0x7f83d015df80 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d5da3640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 0x7f83d014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d65a4640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f83c800ce40 tx=0x7f83c800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c800d840 con 0x7f83d015df80 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c800d9a0 con 0x7f83d015df80 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d014e3c0 con 0x7f83d015df80 2023-11-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c800dd50 con 0x7f83d015df80 2023-11-24T08:13:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.752+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d015b950 con 0x7f83d015df80 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.754+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d014e220 con 0x7f83d015df80 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.754+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f83c801e3b0 con 0x7f83d015df80 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.755+0000 7f83c67fc640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 0x7f83b406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.755+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83c8088ed0 con 0x7f83d015df80 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.756+0000 7f83d5da3640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 0x7f83b406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.756+0000 7f83d5da3640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 0x7f83b406e350 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000f60 tx=0x7f83cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:20.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.759+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83d014e220 con 0x7f83d015df80 2023-11-24T08:13:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.983+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83d014e8e0 con 0x7f83d015df80 2023-11-24T08:13:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.985+0000 7f83c67fc640 1 -- 172.21.15.107:0/725250749 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f83d014e8e0 con 0x7f83d015df80 2023-11-24T08:13:20.987 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:20.987 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:20.987 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-11-24T08:13:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 msgr2=0x7f83b406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 0x7f83b406e350 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000f60 tx=0x7f83cc009040 comp rx=0 tx=0).stop 2023-11-24T08:13:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 msgr2=0x7f83d0144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f83c800ce40 tx=0x7f83c800a710 comp rx=0 tx=0).stop 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 shutdown_connections 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83b406be90 0x7f83b406e350 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83d015df80 0x7f83d0144d00 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d00cd470 0x7f83d014c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 --2- 172.21.15.107:0/725250749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d00b2250 0x7f83d014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 >> 172.21.15.107:0/725250749 conn(0x7f83d006a920 msgr2=0x7f83d00734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.987+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 shutdown_connections 2023-11-24T08:13:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:20.988+0000 7f83d8007640 1 -- 172.21.15.107:0/725250749 wait complete. 2023-11-24T08:13:21.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:21.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.212+0000 7fd5fc839640 1 Processor -- start 2023-11-24T08:13:21.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.213+0000 7fd5fc839640 1 -- start start 2023-11-24T08:13:21.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.214+0000 7fd5fc839640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f8071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.214+0000 7fd5fc839640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5f815a970 0x7fd5f8163430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.214+0000 7fd5fc839640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f80ce870 con 0x7fd5f8159e80 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.214+0000 7fd5fc839640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f80cdc60 con 0x7fd5f815a970 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.214+0000 7fd5fc839640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80cdf40 con 0x7fd5f806d5c0 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5f815a970 0x7fd5f8163430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5f815a970 0x7fd5f8163430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36394/0 (socket says 172.21.15.107:36394) 2023-11-24T08:13:21.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 -- 172.21.15.107:0/2711717157 learned_addr learned my addr 172.21.15.107:0/2711717157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:21.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 -- 172.21.15.107:0/2711717157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 msgr2=0x7fd5f8071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:21.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 --2- 172.21.15.107:0/2711717157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f8071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 -- 172.21.15.107:0/2711717157 >> v1:172.21.15.120:6789/0 conn(0x7fd5f8159e80 legacy=0x7fd5f815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:21.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.215+0000 7fd5f7fff640 1 -- 172.21.15.107:0/2711717157 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f80ce220 con 0x7fd5f815a970 2023-11-24T08:13:21.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.216+0000 7fd5f7fff640 1 --2- 172.21.15.107:0/2711717157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f815a970 0x7fd5f8163430 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000bb10 tx=0x7fd5f002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c166a0b56bf4ac2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.216+0000 7fd5f67fc640 1 -- 172.21.15.107:0/2711717157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f002ede0 con 0x7fd5f815a970 2023-11-24T08:13:21.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.216+0000 7fd5f67fc640 1 -- 172.21.15.107:0/2711717157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f0036440 con 0x7fd5f815a970 2023-11-24T08:13:21.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.216+0000 7fd5f67fc640 1 -- 172.21.15.107:0/2711717157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00367d0 con 0x7fd5f815a970 2023-11-24T08:13:21.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 -- 172.21.15.107:0/2711717157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f815a970 msgr2=0x7fd5f8163430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 --2- 172.21.15.107:0/2711717157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f815a970 0x7fd5f8163430 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000bb10 tx=0x7fd5f002e030 comp rx=0 tx=0).stop 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 -- 172.21.15.107:0/2711717157 shutdown_connections 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 --2- 172.21.15.107:0/2711717157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f815a970 0x7fd5f8163430 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 --2- 172.21.15.107:0/2711717157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f8071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 -- 172.21.15.107:0/2711717157 >> 172.21.15.107:0/2711717157 conn(0x7fd5f80cb830 msgr2=0x7fd5f80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 -- 172.21.15.107:0/2711717157 shutdown_connections 2023-11-24T08:13:21.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.217+0000 7fd5fc839640 1 -- 172.21.15.107:0/2711717157 wait complete. 2023-11-24T08:13:21.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.219+0000 7fd5fc839640 1 Processor -- start 2023-11-24T08:13:21.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.219+0000 7fd5fc839640 1 -- start start 2023-11-24T08:13:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 0x7fd5f8151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f814ce60 con 0x7fd5f8159e80 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f813a210 con 0x7fd5f815a970 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5fc839640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f813a4f0 con 0x7fd5f806d5c0 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36408/0 (socket says 172.21.15.107:36408) 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 -- 172.21.15.107:0/1156011906 learned_addr learned my addr 172.21.15.107:0/1156011906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f7fff640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 0x7fd5f8151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 -- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 msgr2=0x7fd5f80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 -- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 msgr2=0x7fd5f8151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 0x7fd5f8151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f6ffd640 1 -- 172.21.15.107:0/1156011906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f000b770 con 0x7fd5f8159e80 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.221+0000 7fd5f7fff640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 0x7fd5f8151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5f6ffd640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000b9c0 tx=0x7fd5e000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000cba0 con 0x7fd5f8159e80 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e000cd20 con 0x7fd5f8159e80 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0014680 con 0x7fd5f8159e80 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f813a7d0 con 0x7fd5f8159e80 2023-11-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.222+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f814a4b0 con 0x7fd5f8159e80 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.223+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f813a9c0 con 0x7fd5f8159e80 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.224+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fd5e001d3b0 con 0x7fd5f8159e80 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.225+0000 7fd5effff640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 0x7fd5d806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.225+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5e0088440 con 0x7fd5f8159e80 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.226+0000 7fd5f77fe640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 0x7fd5d806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.226+0000 7fd5f77fe640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 0x7fd5d806e350 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009650 tx=0x7fd5e8006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.229+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5e0054c20 con 0x7fd5f8159e80 2023-11-24T08:13:21.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.438+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f804e840 con 0x7fd5f8159e80 2023-11-24T08:13:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.439+0000 7fd5effff640 1 -- 172.21.15.107:0/1156011906 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd5e0019070 con 0x7fd5f8159e80 2023-11-24T08:13:21.441 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:21.442 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:12:58.491334+0000","modified":"2023-11-24T08:13:01.555373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2728005588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2728005588},{"type":"v1","addr":"172.21.15.107:6837","nonce":2728005588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-24T08:13:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 msgr2=0x7fd5d806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 0x7fd5d806e350 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009650 tx=0x7fd5e8006ba0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 msgr2=0x7fd5f80b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000b9c0 tx=0x7fd5e000be80 comp rx=0 tx=0).stop 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 shutdown_connections 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5d806be90 0x7fd5d806e350 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5f815a970 0x7fd5f8151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5f8159e80 0x7fd5f80b38f0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 --2- 172.21.15.107:0/1156011906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5f806d5c0 0x7fd5f80b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 >> 172.21.15.107:0/1156011906 conn(0x7fd5f80cb830 msgr2=0x7fd5f8153060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 shutdown_connections 2023-11-24T08:13:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.442+0000 7fd5fc839640 1 -- 172.21.15.107:0/1156011906 wait complete. 2023-11-24T08:13:21.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:21.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.658+0000 7ff5bb464640 1 Processor -- start 2023-11-24T08:13:21.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.658+0000 7ff5bb464640 1 -- start start 2023-11-24T08:13:21.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.659+0000 7ff5bb464640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b415af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.659+0000 7ff5bb464640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b406db20 0x7ff5b40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.659+0000 7ff5bb464640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b40cec00 con 0x7ff5b406d6c0 2023-11-24T08:13:21.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.659+0000 7ff5bb464640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b413f9e0 con 0x7ff5b406db20 2023-11-24T08:13:21.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.659+0000 7ff5bb464640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b413fcc0 con 0x7ff5b413d120 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b406db20 0x7ff5b40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b406db20 0x7ff5b40b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36420/0 (socket says 172.21.15.107:36420) 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 -- 172.21.15.107:0/1278015593 learned_addr learned my addr 172.21.15.107:0/1278015593 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5ba462640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 -- 172.21.15.107:0/1278015593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 msgr2=0x7ff5b415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 -- 172.21.15.107:0/1278015593 >> v1:172.21.15.120:6789/0 conn(0x7ff5b406d6c0 legacy=0x7ff5b40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5bac63640 1 -- 172.21.15.107:0/1278015593 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b413ffa0 con 0x7ff5b406db20 2023-11-24T08:13:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.660+0000 7ff5ba462640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.661+0000 7ff5bac63640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b40b1180 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00d150 tx=0x7ff5ac02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d41c74f7e0d669 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.661+0000 7ff5b9460640 1 -- 172.21.15.107:0/1278015593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac02e9e0 con 0x7ff5b406db20 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5b9460640 1 -- 172.21.15.107:0/1278015593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ac02eb40 con 0x7ff5b406db20 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5b9460640 1 -- 172.21.15.107:0/1278015593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac009600 con 0x7ff5b406db20 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 -- 172.21.15.107:0/1278015593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 msgr2=0x7ff5b40b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b40b1180 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00d150 tx=0x7ff5ac02e040 comp rx=0 tx=0).stop 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 -- 172.21.15.107:0/1278015593 shutdown_connections 2023-11-24T08:13:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b40b1180 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 --2- 172.21.15.107:0/1278015593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 -- 172.21.15.107:0/1278015593 >> 172.21.15.107:0/1278015593 conn(0x7ff5b40cb830 msgr2=0x7ff5b40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 -- 172.21.15.107:0/1278015593 shutdown_connections 2023-11-24T08:13:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.662+0000 7ff5bb464640 1 -- 172.21.15.107:0/1278015593 wait complete. 2023-11-24T08:13:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.664+0000 7ff5bb464640 1 Processor -- start 2023-11-24T08:13:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.664+0000 7ff5bb464640 1 -- start start 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 0x7ff5b40b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b413e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b413aba0 con 0x7ff5b406db20 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40b4860 con 0x7ff5b406d6c0 2023-11-24T08:13:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5bb464640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b41f7010 con 0x7ff5b413d120 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.665+0000 7ff5b9c61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36426/0 (socket says 172.21.15.107:36426) 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 -- 172.21.15.107:0/350976424 learned_addr learned my addr 172.21.15.107:0/350976424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5bac63640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5ba462640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 0x7ff5b40b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 -- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 msgr2=0x7ff5b413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 -- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 msgr2=0x7ff5b40b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 0x7ff5b40b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 -- 172.21.15.107:0/350976424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac00bd40 con 0x7ff5b406db20 2023-11-24T08:13:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5ba462640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 0x7ff5b40b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.666+0000 7ff5b9c61640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c460 tx=0x7ff5a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.667+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4012830 con 0x7ff5b406db20 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.667+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a40129d0 con 0x7ff5b406db20 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.667+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b413a550 con 0x7ff5b406db20 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.667+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4012d80 con 0x7ff5b406db20 2023-11-24T08:13:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.667+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b413a1d0 con 0x7ff5b406db20 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.668+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b413ad50 con 0x7ff5b406db20 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.669+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7ff5a401a3b0 con 0x7ff5b406db20 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.670+0000 7ff5aaffd640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 0x7ff59806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.670+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5a40883c0 con 0x7ff5b406db20 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.670+0000 7ff5ba462640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 0x7ff59806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:21.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.670+0000 7ff5ba462640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 0x7ff59806e350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0006fd0 tx=0x7ff5b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:21.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.674+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5b413ad50 con 0x7ff5b406db20 2023-11-24T08:13:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.850+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b413ad50 con 0x7ff5b406db20 2023-11-24T08:13:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.852+0000 7ff5aaffd640 1 -- 172.21.15.107:0/350976424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16519 (secure 0 0 0) 0x7ff5b413ad50 con 0x7ff5b406db20 2023-11-24T08:13:21.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:21.854 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":109,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:00.046363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:12:56.294075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-11-24T08:12:57.505929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:21.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 msgr2=0x7ff59806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 0x7ff59806e350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0006fd0 tx=0x7ff5b0008040 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 msgr2=0x7ff5b413e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c460 tx=0x7ff5a400c920 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 shutdown_connections 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff59806be90 0x7ff59806e350 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b413d120 0x7ff5b413e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406db20 0x7ff5b413e010 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 --2- 172.21.15.107:0/350976424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b406d6c0 0x7ff5b40b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 >> 172.21.15.107:0/350976424 conn(0x7ff5b40cb830 msgr2=0x7ff5b40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:21.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 shutdown_connections 2023-11-24T08:13:21.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:21.855+0000 7ff5bb464640 1 -- 172.21.15.107:0/350976424 wait complete. 2023-11-24T08:13:21.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:13:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.071+0000 7efc29f63640 1 Processor -- start 2023-11-24T08:13:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.071+0000 7efc29f63640 1 -- start start 2023-11-24T08:13:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efc2415b1d0 0x7efc24071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc24070550 0x7efc240b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2406c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc240b14f0 con 0x7efc24070550 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc240ca0e0 con 0x7efc2415b1d0 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc29f63640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc24072800 con 0x7efc240b1080 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc23fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc24070550 0x7efc240b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc23fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc24070550 0x7efc240b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36442/0 (socket says 172.21.15.107:36442) 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.072+0000 7efc23fff640 1 -- 172.21.15.107:0/1993836225 learned_addr learned my addr 172.21.15.107:0/1993836225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.073+0000 7efc23fff640 1 -- 172.21.15.107:0/1993836225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 msgr2=0x7efc2406c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.073+0000 7efc23fff640 1 --2- 172.21.15.107:0/1993836225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2406c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.073+0000 7efc23fff640 1 -- 172.21.15.107:0/1993836225 >> v2:172.21.15.120:3300/0 conn(0x7efc2415b1d0 msgr2=0x7efc24071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.073+0000 7efc23fff640 1 --2- 172.21.15.107:0/1993836225 >> v2:172.21.15.120:3300/0 conn(0x7efc2415b1d0 0x7efc24071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.073+0000 7efc23fff640 1 -- 172.21.15.107:0/1993836225 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc240c97b0 con 0x7efc24070550 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.074+0000 7efc23fff640 1 --2- 172.21.15.107:0/1993836225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc24070550 0x7efc240b0b40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7efc2406e7a0 tx=0x7efc1002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a6bbe0fc583711 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.074+0000 7efc237fe640 1 -- 172.21.15.107:0/1993836225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10031860 con 0x7efc24070550 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.074+0000 7efc237fe640 1 -- 172.21.15.107:0/1993836225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc100319c0 con 0x7efc24070550 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.074+0000 7efc237fe640 1 -- 172.21.15.107:0/1993836225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10031d50 con 0x7efc24070550 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 -- 172.21.15.107:0/1993836225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc24070550 msgr2=0x7efc240b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 --2- 172.21.15.107:0/1993836225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc24070550 0x7efc240b0b40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7efc2406e7a0 tx=0x7efc1002b900 comp rx=0 tx=0).stop 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 -- 172.21.15.107:0/1993836225 shutdown_connections 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 --2- 172.21.15.107:0/1993836225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2406c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 --2- 172.21.15.107:0/1993836225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc24070550 0x7efc240b0b40 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 --2- 172.21.15.107:0/1993836225 >> v2:172.21.15.120:3300/0 conn(0x7efc2415b1d0 0x7efc24071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 -- 172.21.15.107:0/1993836225 >> 172.21.15.107:0/1993836225 conn(0x7efc240cbb40 msgr2=0x7efc240cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 -- 172.21.15.107:0/1993836225 shutdown_connections 2023-11-24T08:13:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.075+0000 7efc29f63640 1 -- 172.21.15.107:0/1993836225 wait complete. 2023-11-24T08:13:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.076+0000 7efc29f63640 1 Processor -- start 2023-11-24T08:13:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.076+0000 7efc29f63640 1 -- start start 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc24070550 0x7efc240cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc240cd3f0 con 0x7efc2415b1d0 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc240cdd50 con 0x7efc24070550 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.077+0000 7efc29f63640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc240c4da0 con 0x7efc240b1080 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36454/0 (socket says 172.21.15.107:36454) 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 -- 172.21.15.107:0/4044989947 learned_addr learned my addr 172.21.15.107:0/4044989947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 -- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 msgr2=0x7efc2415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 -- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc24070550 msgr2=0x7efc240cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc24070550 0x7efc240cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 -- 172.21.15.107:0/4044989947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100094d0 con 0x7efc2415b1d0 2023-11-24T08:13:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc29762640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00b970 tx=0x7efc1c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.078+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c00cb40 con 0x7efc2415b1d0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.079+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c00ccc0 con 0x7efc2415b1d0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.079+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c014680 con 0x7efc2415b1d0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.079+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc240ccd30 con 0x7efc2415b1d0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.079+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2415aa20 con 0x7efc2415b1d0 2023-11-24T08:13:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.080+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc240cd530 con 0x7efc2415b1d0 2023-11-24T08:13:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.081+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7efc1c01d3b0 con 0x7efc2415b1d0 2023-11-24T08:13:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.081+0000 7efc217fa640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 0x7efc0806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.082+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc1c088400 con 0x7efc2415b1d0 2023-11-24T08:13:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.082+0000 7efc28f61640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 0x7efc0806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.082+0000 7efc28f61640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 0x7efc0806e350 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7efc18009710 tx=0x7efc18006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.086+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc240cd530 con 0x7efc2415b1d0 2023-11-24T08:13:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.308+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efc240cd530 con 0x7efc2415b1d0 2023-11-24T08:13:22.310 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:13:22.309+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:13:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.312+0000 7efc217fa640 1 -- 172.21.15.107:0/4044989947 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) v1 ==== 161+0+0 (secure 0 0 0) 0x7efc240cd530 con 0x7efc2415b1d0 2023-11-24T08:13:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 msgr2=0x7efc0806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 0x7efc0806e350 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7efc18009710 tx=0x7efc18006c90 comp rx=0 tx=0).stop 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 msgr2=0x7efc240c4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00b970 tx=0x7efc1c00be30 comp rx=0 tx=0).stop 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 shutdown_connections 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc0806be90 0x7efc0806e350 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc2415b1d0 0x7efc240c4340 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc240b1080 0x7efc2415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 --2- 172.21.15.107:0/4044989947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc24070550 0x7efc240cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 >> 172.21.15.107:0/4044989947 conn(0x7efc240cbb40 msgr2=0x7efc240abec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 shutdown_connections 2023-11-24T08:13:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.316+0000 7efc29f63640 1 -- 172.21.15.107:0/4044989947 wait complete. 2023-11-24T08:13:22.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:13:22.432 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:13:22.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.535+0000 7fab35a7c640 1 Processor -- start 2023-11-24T08:13:22.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.535+0000 7fab35a7c640 1 -- start start 2023-11-24T08:13:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.536+0000 7fab35a7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab300cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.536+0000 7fab35a7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab300c59b0 0x7fab3006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.536+0000 7fab35a7c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab300cc3a0 con 0x7fab300cbf20 2023-11-24T08:13:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.536+0000 7fab35a7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3015b820 con 0x7fab300c59b0 2023-11-24T08:13:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.536+0000 7fab35a7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3015eea0 con 0x7fab3015b440 2023-11-24T08:13:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.537+0000 7fab2ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab300c59b0 0x7fab3006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.537+0000 7fab2f7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab300cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.537+0000 7fab2ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab300c59b0 0x7fab3006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:37694/0 (socket says 172.21.15.107:37694) 2023-11-24T08:13:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.537+0000 7fab2ffff640 1 -- 172.21.15.107:0/2509465535 learned_addr learned my addr 172.21.15.107:0/2509465535 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.538+0000 7fab2ffff640 1 -- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 msgr2=0x7fab300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.538+0000 7fab2ffff640 1 --2- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab300cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.538+0000 7fab2ffff640 1 -- 172.21.15.107:0/2509465535 >> v1:172.21.15.120:6789/0 conn(0x7fab300cbf20 legacy=0x7fab300c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.538+0000 7fab2ffff640 1 -- 172.21.15.107:0/2509465535 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300abf30 con 0x7fab300c59b0 2023-11-24T08:13:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.539+0000 7fab2ffff640 1 --2- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300c59b0 0x7fab3006c5f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fab28009e50 tx=0x7fab2802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573760420c44f923 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.539+0000 7fab2effd640 1 -- 172.21.15.107:0/2509465535 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2802ca70 con 0x7fab300c59b0 2023-11-24T08:13:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.539+0000 7fab2effd640 1 -- 172.21.15.107:0/2509465535 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2802cbd0 con 0x7fab300c59b0 2023-11-24T08:13:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.539+0000 7fab2effd640 1 -- 172.21.15.107:0/2509465535 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280343f0 con 0x7fab300c59b0 2023-11-24T08:13:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 -- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300c59b0 msgr2=0x7fab3006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 --2- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300c59b0 0x7fab3006c5f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fab28009e50 tx=0x7fab2802bed0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 -- 172.21.15.107:0/2509465535 shutdown_connections 2023-11-24T08:13:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 --2- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300c59b0 0x7fab3006c5f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 --2- 172.21.15.107:0/2509465535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab300cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 -- 172.21.15.107:0/2509465535 >> 172.21.15.107:0/2509465535 conn(0x7fab300b0c50 msgr2=0x7fab300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 -- 172.21.15.107:0/2509465535 shutdown_connections 2023-11-24T08:13:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.540+0000 7fab35a7c640 1 -- 172.21.15.107:0/2509465535 wait complete. 2023-11-24T08:13:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.542+0000 7fab35a7c640 1 Processor -- start 2023-11-24T08:13:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.542+0000 7fab35a7c640 1 -- start start 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 0x7fab30070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab30153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab30159f30 con 0x7fab300c59b0 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300cd6e0 con 0x7fab300cbf20 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab35a7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab300cd960 con 0x7fab3015b440 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab2f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab2f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36474/0 (socket says 172.21.15.107:36474) 2023-11-24T08:13:22.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.543+0000 7fab2f7fe640 1 -- 172.21.15.107:0/3435535303 learned_addr learned my addr 172.21.15.107:0/3435535303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab27fff640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 0x7fab30070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2ffff640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab30153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 -- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 msgr2=0x7fab30153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab30153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 -- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 msgr2=0x7fab30070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 0x7fab30070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 -- 172.21.15.107:0/3435535303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab28009b30 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab27fff640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 0x7fab30070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2f7fe640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fab2000e870 tx=0x7fab2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200094f0 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2000c940 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab30159590 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2000ccf0 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.544+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab30159b50 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.545+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3015a250 con 0x7fab300c59b0 2023-11-24T08:13:22.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.546+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fab20013470 con 0x7fab300c59b0 2023-11-24T08:13:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.547+0000 7fab2cff9640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 0x7fab1406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.548+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fab200882d0 con 0x7fab300c59b0 2023-11-24T08:13:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.548+0000 7fab27fff640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 0x7fab1406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.548+0000 7fab27fff640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 0x7fab1406e350 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fab300b15a0 tx=0x7fab18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:22.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.552+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab20054d50 con 0x7fab300c59b0 2023-11-24T08:13:22.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:22.778+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fab3004e8b0 con 0x7fab300c59b0 2023-11-24T08:13:23.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.316+0000 7fab2cff9640 1 -- 172.21.15.107:0/3435535303 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab20058370 con 0x7fab300c59b0 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 msgr2=0x7fab1406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 0x7fab1406e350 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fab300b15a0 tx=0x7fab18009040 comp rx=0 tx=0).stop 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 msgr2=0x7fab300cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fab2000e870 tx=0x7fab2000ed30 comp rx=0 tx=0).stop 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 shutdown_connections 2023-11-24T08:13:23.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab1406be90 0x7fab1406e350 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3015b440 0x7fab30153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab300cbf20 0x7fab30070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 --2- 172.21.15.107:0/3435535303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab300c59b0 0x7fab300cd1a0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 >> 172.21.15.107:0/3435535303 conn(0x7fab300b0c50 msgr2=0x7fab300cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 shutdown_connections 2023-11-24T08:13:23.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.320+0000 7fab35a7c640 1 -- 172.21.15.107:0/3435535303 wait complete. 2023-11-24T08:13:23.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:13:23.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.554+0000 7f0a80cda640 1 Processor -- start 2023-11-24T08:13:23.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.554+0000 7f0a80cda640 1 -- start start 2023-11-24T08:13:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a80cda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a7c15df80 0x7f0a7c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a80cda640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c15e3f0 con 0x7f0a7c0cd470 2023-11-24T08:13:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a80cda640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c15ab80 con 0x7f0a7c0b2250 2023-11-24T08:13:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a80cda640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c15a990 con 0x7f0a7c15df80 2023-11-24T08:13:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a79d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a7c15df80 0x7f0a7c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a7ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0a7c0cd470 0x7f0a7c15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55654/0 (socket says 172.21.15.107:55654) 2023-11-24T08:13:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a7ad9d640 1 -- 172.21.15.107:0/410208543 learned_addr learned my addr 172.21.15.107:0/410208543 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.556+0000 7f0a79d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a7c15df80 0x7f0a7c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36480/0 (socket says 172.21.15.107:36480) 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.557+0000 7f0a7959a640 1 -- 172.21.15.107:0/410208543 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837420127 0 0) 0x7f0a7c15e3f0 con 0x7f0a7c0cd470 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.557+0000 7f0a7959a640 1 -- 172.21.15.107:0/410208543 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a64005730 con 0x7f0a7c0cd470 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.557+0000 7f0a79d9b640 1 -- 172.21.15.107:0/410208543 >> v1:172.21.15.120:6789/0 conn(0x7f0a7c0b2250 legacy=0x7f0a7c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.557+0000 7f0a79d9b640 1 -- 172.21.15.107:0/410208543 >> v1:172.21.15.107:6789/0 conn(0x7f0a7c0cd470 legacy=0x7f0a7c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.557+0000 7f0a79d9b640 1 -- 172.21.15.107:0/410208543 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0b2d40 con 0x7f0a7c15df80 2023-11-24T08:13:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.558+0000 7f0a79d9b640 1 --2- 172.21.15.107:0/410208543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c15df80 0x7f0a7c0ccf30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f0a700097a0 tx=0x7f0a7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f166c98fe0debd61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.558+0000 7f0a7959a640 1 -- 172.21.15.107:0/410208543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a70031830 con 0x7f0a7c15df80 2023-11-24T08:13:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.558+0000 7f0a7959a640 1 -- 172.21.15.107:0/410208543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a70031990 con 0x7f0a7c15df80 2023-11-24T08:13:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.558+0000 7f0a7959a640 1 -- 172.21.15.107:0/410208543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a70031d20 con 0x7f0a7c15df80 2023-11-24T08:13:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 -- 172.21.15.107:0/410208543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c15df80 msgr2=0x7f0a7c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 --2- 172.21.15.107:0/410208543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c15df80 0x7f0a7c0ccf30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f0a700097a0 tx=0x7f0a7002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 -- 172.21.15.107:0/410208543 shutdown_connections 2023-11-24T08:13:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 --2- 172.21.15.107:0/410208543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c15df80 0x7f0a7c0ccf30 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 -- 172.21.15.107:0/410208543 >> 172.21.15.107:0/410208543 conn(0x7f0a7c06a920 msgr2=0x7f0a7c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 -- 172.21.15.107:0/410208543 shutdown_connections 2023-11-24T08:13:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.559+0000 7f0a80cda640 1 -- 172.21.15.107:0/410208543 wait complete. 2023-11-24T08:13:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.561+0000 7f0a80cda640 1 Processor -- start 2023-11-24T08:13:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.561+0000 7f0a80cda640 1 -- start start 2023-11-24T08:13:23.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:23.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a7c0cd470 0x7f0a7c147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:23.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 0x7f0a7c148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:23.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c072540 con 0x7f0a7c0b2250 2023-11-24T08:13:23.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c0718c0 con 0x7f0a7c0cd470 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a80cda640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c071ba0 con 0x7f0a7c15df80 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a7a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a7a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36488/0 (socket says 172.21.15.107:36488) 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a7a59c640 1 -- 172.21.15.107:0/1026069735 learned_addr learned my addr 172.21.15.107:0/1026069735 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.562+0000 7f0a79d9b640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a7c0cd470 0x7f0a7c147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7ad9d640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 0x7f0a7c148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 -- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 msgr2=0x7f0a7c148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 0x7f0a7c148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 -- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a7c0cd470 msgr2=0x7f0a7c147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a7c0cd470 0x7f0a7c147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 -- 172.21.15.107:0/1026069735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a70009400 con 0x7f0a7c0b2250 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7ad9d640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 0x7f0a7c148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a7a59c640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c430 tx=0x7f0a6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c011790 con 0x7f0a7c0b2250 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c0118f0 con 0x7f0a7c0b2250 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c071e80 con 0x7f0a7c0b2250 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c011ce0 con 0x7f0a7c0b2250 2023-11-24T08:13:23.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.563+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c14c250 con 0x7f0a7c0b2250 2023-11-24T08:13:23.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.565+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c072680 con 0x7f0a7c0b2250 2023-11-24T08:13:23.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.566+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f0a6c01c460 con 0x7f0a7c0b2250 2023-11-24T08:13:23.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.567+0000 7f0a6affd640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 0x7f0a5c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:23.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.567+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0a6c088410 con 0x7f0a7c0b2250 2023-11-24T08:13:23.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.567+0000 7f0a79d9b640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 0x7f0a5c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:23.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.568+0000 7f0a79d9b640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 0x7f0a5c06e350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0a70036000 tx=0x7f0a7003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:23.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.572+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a6c054c50 con 0x7f0a7c0b2250 2023-11-24T08:13:23.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:23.765+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a7c14c3e0 con 0x7f0a7c0b2250 2023-11-24T08:13:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.468+0000 7f0a6affd640 1 -- 172.21.15.107:0/1026069735 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0a6c058270 con 0x7f0a7c0b2250 2023-11-24T08:13:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:13:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.471+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 msgr2=0x7f0a5c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.471+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 0x7f0a5c06e350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0a70036000 tx=0x7f0a7003c040 comp rx=0 tx=0).stop 2023-11-24T08:13:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.471+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 msgr2=0x7f0a7c14bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.471+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c430 tx=0x7f0a6c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 shutdown_connections 2023-11-24T08:13:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0a5c06be90 0x7f0a5c06e350 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a7c15df80 0x7f0a7c148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a7c0cd470 0x7f0a7c147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 --2- 172.21.15.107:0/1026069735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c0b2250 0x7f0a7c14bd10 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 >> 172.21.15.107:0/1026069735 conn(0x7f0a7c06a920 msgr2=0x7f0a7c15c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.472+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 shutdown_connections 2023-11-24T08:13:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.474+0000 7f0a80cda640 1 -- 172.21.15.107:0/1026069735 wait complete. 2023-11-24T08:13:24.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:13:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.708+0000 7fb348ffc640 1 Processor -- start 2023-11-24T08:13:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.708+0000 7fb348ffc640 1 -- start start 2023-11-24T08:13:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb348ffc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb34415df80 0x7fb3440ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb348ffc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb34415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb348ffc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34415e3f0 con 0x7fb3440b2250 2023-11-24T08:13:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb348ffc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34415a990 con 0x7fb34415df80 2023-11-24T08:13:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb348ffc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415ab80 con 0x7fb3440cd470 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb342d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb34415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb341d9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb34415df80 0x7fb3440ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb342d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb34415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:35574/0 (socket says 172.21.15.107:35574) 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.710+0000 7fb342d9d640 1 -- 172.21.15.107:0/379622258 learned_addr learned my addr 172.21.15.107:0/379622258 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.711+0000 7fb341d9b640 1 -- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 msgr2=0x7fb34415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.711+0000 7fb341d9b640 1 --2- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb34415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.711+0000 7fb341d9b640 1 -- 172.21.15.107:0/379622258 >> v1:172.21.15.120:6789/0 conn(0x7fb3440b2250 legacy=0x7fb3440b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.711+0000 7fb341d9b640 1 -- 172.21.15.107:0/379622258 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440b2d40 con 0x7fb34415df80 2023-11-24T08:13:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.712+0000 7fb341d9b640 1 --2- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34415df80 0x7fb3440ccf30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097a0 tx=0x7fb33802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26812e2d3bf90acd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.713+0000 7fb34159a640 1 -- 172.21.15.107:0/379622258 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33802d040 con 0x7fb34415df80 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.713+0000 7fb34159a640 1 -- 172.21.15.107:0/379622258 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338031810 con 0x7fb34415df80 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.713+0000 7fb34159a640 1 -- 172.21.15.107:0/379622258 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338031ba0 con 0x7fb34415df80 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.713+0000 7fb348ffc640 1 -- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34415df80 msgr2=0x7fb3440ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.713+0000 7fb348ffc640 1 --2- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34415df80 0x7fb3440ccf30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097a0 tx=0x7fb33802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 -- 172.21.15.107:0/379622258 shutdown_connections 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 --2- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb34415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 --2- 172.21.15.107:0/379622258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34415df80 0x7fb3440ccf30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 -- 172.21.15.107:0/379622258 >> 172.21.15.107:0/379622258 conn(0x7fb34406a920 msgr2=0x7fb34406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 -- 172.21.15.107:0/379622258 shutdown_connections 2023-11-24T08:13:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.714+0000 7fb348ffc640 1 -- 172.21.15.107:0/379622258 wait complete. 2023-11-24T08:13:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.715+0000 7fb348ffc640 1 Processor -- start 2023-11-24T08:13:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 -- start start 2023-11-24T08:13:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 0x7fb344146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb344153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb344098ed0 con 0x7fb34415df80 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415cee0 con 0x7fb3440b2250 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.716+0000 7fb348ffc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb344143f80 con 0x7fb3440cd470 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36502/0 (socket says 172.21.15.107:36502) 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 -- 172.21.15.107:0/1745669516 learned_addr learned my addr 172.21.15.107:0/1745669516 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb341d9b640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb344153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 -- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 msgr2=0x7fb344153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb344153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 -- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 msgr2=0x7fb344146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb34259c640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 0x7fb344146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 0x7fb344146b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb342d9d640 1 -- 172.21.15.107:0/1745669516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338009400 con 0x7fb34415df80 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.717+0000 7fb34259c640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 0x7fb344146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb342d9d640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00efc0 tx=0x7fb33c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c00cd20 con 0x7fb34415df80 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c00f040 con 0x7fb34415df80 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c003bb0 con 0x7fb34415df80 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb344154570 con 0x7fb34415df80 2023-11-24T08:13:24.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.718+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34415cb70 con 0x7fb34415df80 2023-11-24T08:13:24.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.719+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb344154a90 con 0x7fb34415df80 2023-11-24T08:13:24.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.720+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb33c003d10 con 0x7fb34415df80 2023-11-24T08:13:24.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.721+0000 7fb332ffd640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 0x7fb32406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:24.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.721+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fb33c0889c0 con 0x7fb34415df80 2023-11-24T08:13:24.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.721+0000 7fb34259c640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 0x7fb32406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:24.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.722+0000 7fb34259c640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 0x7fb32406e380 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb334000f60 tx=0x7fb334009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:24.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.725+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb344154a90 con 0x7fb34415df80 2023-11-24T08:13:24.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:24.921+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb344154a90 con 0x7fb34415df80 2023-11-24T08:13:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.909+0000 7fb332ffd640 1 -- 172.21.15.107:0/1745669516 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb344154a90 con 0x7fb34415df80 2023-11-24T08:13:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.913+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 msgr2=0x7fb32406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.914+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 0x7fb32406e380 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb334000f60 tx=0x7fb334009040 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.914+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 msgr2=0x7fb344098990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.914+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00efc0 tx=0x7fb33c00c490 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 shutdown_connections 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb32406bec0 0x7fb32406e380 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb34415df80 0x7fb344098990 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3440cd470 0x7fb344153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 --2- 172.21.15.107:0/1745669516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3440b2250 0x7fb344146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 >> 172.21.15.107:0/1745669516 conn(0x7fb34406a920 msgr2=0x7fb3440cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 shutdown_connections 2023-11-24T08:13:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:25.916+0000 7fb348ffc640 1 -- 172.21.15.107:0/1745669516 wait complete. 2023-11-24T08:13:25.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:13:26.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.173+0000 7fbd99f7d640 1 Processor -- start 2023-11-24T08:13:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.173+0000 7fbd99f7d640 1 -- start start 2023-11-24T08:13:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.174+0000 7fbd99f7d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd940c9650 0x7fbd940c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.174+0000 7fbd99f7d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd94166510 con 0x7fbd941660b0 2023-11-24T08:13:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.174+0000 7fbd99f7d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd941655f0 con 0x7fbd940ccbd0 2023-11-24T08:13:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.174+0000 7fbd99f7d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940cc380 con 0x7fbd940c9650 2023-11-24T08:13:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd93fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd940c9650 0x7fbd940c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd92ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd941660b0 0x7fbd94162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55662/0 (socket says 172.21.15.107:55662) 2023-11-24T08:13:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd92ffd640 1 -- 172.21.15.107:0/2442509688 learned_addr learned my addr 172.21.15.107:0/2442509688 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd93fff640 1 -- 172.21.15.107:0/2442509688 >> v1:172.21.15.120:6789/0 conn(0x7fbd940ccbd0 legacy=0x7fbd940ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd93fff640 1 -- 172.21.15.107:0/2442509688 >> v1:172.21.15.107:6789/0 conn(0x7fbd941660b0 legacy=0x7fbd94162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd93fff640 1 -- 172.21.15.107:0/2442509688 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94162b30 con 0x7fbd940c9650 2023-11-24T08:13:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.175+0000 7fbd927fc640 1 -- 172.21.15.107:0/2442509688 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725815910 0 0) 0x7fbd88001110 con 0x7fbd941660b0 2023-11-24T08:13:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.176+0000 7fbd93fff640 1 --2- 172.21.15.107:0/2442509688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd940c9ae0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d90 tx=0x7fbd8c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b297087eebb816 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd927fc640 1 -- 172.21.15.107:0/2442509688 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02cc60 con 0x7fbd940c9650 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd927fc640 1 -- 172.21.15.107:0/2442509688 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c02cdc0 con 0x7fbd940c9650 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd927fc640 1 -- 172.21.15.107:0/2442509688 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c034660 con 0x7fbd940c9650 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd99f7d640 1 -- 172.21.15.107:0/2442509688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 msgr2=0x7fbd940c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/2442509688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd940c9ae0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d90 tx=0x7fbd8c02beb0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd99f7d640 1 -- 172.21.15.107:0/2442509688 shutdown_connections 2023-11-24T08:13:26.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/2442509688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd940c9ae0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.177+0000 7fbd99f7d640 1 -- 172.21.15.107:0/2442509688 >> 172.21.15.107:0/2442509688 conn(0x7fbd940b0c50 msgr2=0x7fbd940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.178+0000 7fbd99f7d640 1 -- 172.21.15.107:0/2442509688 shutdown_connections 2023-11-24T08:13:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.178+0000 7fbd99f7d640 1 -- 172.21.15.107:0/2442509688 wait complete. 2023-11-24T08:13:26.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.179+0000 7fbd99f7d640 1 Processor -- start 2023-11-24T08:13:26.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.179+0000 7fbd99f7d640 1 -- start start 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 0x7fbd941546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 0x7fbd941566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941fb9e0 con 0x7fbd940c9650 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94070760 con 0x7fbd940ccbd0 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.180+0000 7fbd99f7d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941fad60 con 0x7fbd941660b0 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36524/0 (socket says 172.21.15.107:36524) 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 -- 172.21.15.107:0/3647428429 learned_addr learned my addr 172.21.15.107:0/3647428429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd92ffd640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 0x7fbd941546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd93fff640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 0x7fbd941566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 -- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 msgr2=0x7fbd941566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 0x7fbd941566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 -- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 msgr2=0x7fbd941546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 0x7fbd941546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 -- 172.21.15.107:0/3647428429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c009a70 con 0x7fbd940c9650 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd92ffd640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 0x7fbd941546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd93fff640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 0x7fbd941566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.181+0000 7fbd937fe640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400c460 tx=0x7fbd8400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.182+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84012830 con 0x7fbd940c9650 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.182+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840129d0 con 0x7fbd940c9650 2023-11-24T08:13:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.182+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84012d80 con 0x7fbd940c9650 2023-11-24T08:13:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.182+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd941fb040 con 0x7fbd940c9650 2023-11-24T08:13:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.182+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd941fb600 con 0x7fbd940c9650 2023-11-24T08:13:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.183+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9404e8b0 con 0x7fbd940c9650 2023-11-24T08:13:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.184+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fbd8401a3b0 con 0x7fbd940c9650 2023-11-24T08:13:26.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.185+0000 7fbd7bfff640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 0x7fbd8006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.185+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbd84088110 con 0x7fbd940c9650 2023-11-24T08:13:26.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.185+0000 7fbd92ffd640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 0x7fbd8006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.186+0000 7fbd92ffd640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 0x7fbd8006e350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbd94067970 tx=0x7fbd88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.189+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd840548f0 con 0x7fbd940c9650 2023-11-24T08:13:26.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.379+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbd9412c1b0 con 0x7fbd940c9650 2023-11-24T08:13:26.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.386+0000 7fbd7bfff640 1 -- 172.21.15.107:0/3647428429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbd84057f10 con 0x7fbd940c9650 2023-11-24T08:13:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 msgr2=0x7fbd8006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 0x7fbd8006e350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbd94067970 tx=0x7fbd88009040 comp rx=0 tx=0).stop 2023-11-24T08:13:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 msgr2=0x7fbd941f8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400c460 tx=0x7fbd8400c920 comp rx=0 tx=0).stop 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 shutdown_connections 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbd8006be90 0x7fbd8006e350 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd941660b0 0x7fbd941566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd940ccbd0 0x7fbd941546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 --2- 172.21.15.107:0/3647428429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd940c9650 0x7fbd941f8150 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 >> 172.21.15.107:0/3647428429 conn(0x7fbd940b0c50 msgr2=0x7fbd940af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:26.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 shutdown_connections 2023-11-24T08:13:26.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.389+0000 7fbd99f7d640 1 -- 172.21.15.107:0/3647428429 wait complete. 2023-11-24T08:13:26.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status' 2023-11-24T08:13:26.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.620+0000 7f7c949c9640 1 Processor -- start 2023-11-24T08:13:26.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.621+0000 7f7c949c9640 1 -- start start 2023-11-24T08:13:26.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.622+0000 7f7c949c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c900ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.622+0000 7f7c949c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9015ab80 con 0x7f7c900cd470 2023-11-24T08:13:26.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.622+0000 7f7c949c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9015e3c0 con 0x7f7c900b2250 2023-11-24T08:13:26.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.622+0000 7f7c949c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9015a990 con 0x7f7c9015df80 2023-11-24T08:13:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7c900cd470 0x7f7c9015b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55666/0 (socket says 172.21.15.107:55666) 2023-11-24T08:13:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8ed9d640 1 -- 172.21.15.107:0/4179726593 learned_addr learned my addr 172.21.15.107:0/4179726593 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c85d9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c900ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317492249 0 0) 0x7f7c9015ab80 con 0x7f7c900cd470 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c005730 con 0x7f7c900cd470 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715734627 0 0) 0x7f7c7c005730 con 0x7f7c900cd470 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 msgr2=0x7f7c900ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c900ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 >> v1:172.21.15.120:6789/0 conn(0x7f7c900b2250 legacy=0x7f7c900b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900b2d40 con 0x7f7c900cd470 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7c880019f0 con 0x7f7c900cd470 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.623+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 >> v1:172.21.15.107:6789/0 conn(0x7f7c900cd470 legacy=0x7f7c9015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7c005bb0 0x7f7c7c005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c7c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c7c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c00dc70 con 0x7f7c7c006f10 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c00dde0 con 0x7f7c7c005bb0 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c00dfa0 con 0x7f7c9015df80 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c7c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8e59c640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7c005bb0 0x7f7c7c005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c85d9b640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c7c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 -- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 msgr2=0x7f7c7c0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015df80 0x7f7c7c0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 -- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7c005bb0 msgr2=0x7f7c7c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7c005bb0 0x7f7c7c005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.624+0000 7f7c8ed9d640 1 -- 172.21.15.107:0/4179726593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c005730 con 0x7f7c7c006f10 2023-11-24T08:13:26.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.625+0000 7f7c8ed9d640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c7c007390 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f7c88002230 tx=0x7f7c7c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c9525ac5576de4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88034630 con 0x7f7c7c006f10 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8802f070 con 0x7f7c7c006f10 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c8dd9b640 1 -- 172.21.15.107:0/4179726593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88034a10 con 0x7f7c7c006f10 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c949c9640 1 -- 172.21.15.107:0/4179726593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 msgr2=0x7f7c7c007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c949c9640 1 --2- 172.21.15.107:0/4179726593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c7c007390 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f7c88002230 tx=0x7f7c7c007a00 comp rx=0 tx=0).stop 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c8ed9d640 1 -- 172.21.15.107:0/4179726593 reap_dead start 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c949c9640 1 -- 172.21.15.107:0/4179726593 shutdown_connections 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.626+0000 7f7c949c9640 1 -- 172.21.15.107:0/4179726593 >> 172.21.15.107:0/4179726593 conn(0x7f7c9006a920 msgr2=0x7f7c9006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.627+0000 7f7c949c9640 1 -- 172.21.15.107:0/4179726593 shutdown_connections 2023-11-24T08:13:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.627+0000 7f7c949c9640 1 -- 172.21.15.107:0/4179726593 wait complete. 2023-11-24T08:13:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.628+0000 7f7c949c9640 1 Processor -- start 2023-11-24T08:13:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.628+0000 7f7c949c9640 1 -- start start 2023-11-24T08:13:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.629+0000 7f7c949c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.629+0000 7f7c949c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c900b1220 0x7f7c90151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.629+0000 7f7c949c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015caa0 0x7f7c9015cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.629+0000 7f7c949c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9015d460 con 0x7f7c7c006f10 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c949c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c901f6890 con 0x7f7c900b1220 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c949c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900adcf0 con 0x7f7c9015caa0 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36546/0 (socket says 172.21.15.107:36546) 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 -- 172.21.15.107:0/1458688946 learned_addr learned my addr 172.21.15.107:0/1458688946 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 -- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015caa0 msgr2=0x7f7c9015cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015caa0 0x7f7c9015cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 -- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c900b1220 msgr2=0x7f7c90151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c900b1220 0x7f7c90151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 -- 172.21.15.107:0/1458688946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c901f6ae0 con 0x7f7c7c006f10 2023-11-24T08:13:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.630+0000 7f7c8e59c640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000e450 tx=0x7f7c8000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.631+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c800053a0 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.631+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c800035f0 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.631+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80003980 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.631+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c901f6dc0 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.631+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c900ada00 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.632+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c90067470 con 0x7f7c7c006f10 2023-11-24T08:13:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.633+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f7c80003ae0 con 0x7f7c7c006f10 2023-11-24T08:13:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.634+0000 7f7c877fe640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 0x7f7c7006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.634+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c80088180 con 0x7f7c7c006f10 2023-11-24T08:13:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.635+0000 7f7c85d9b640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 0x7f7c7006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.635+0000 7f7c85d9b640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 0x7f7c7006e350 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7c90152010 tx=0x7f7c78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:26.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.639+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c800549c0 con 0x7f7c7c006f10 2023-11-24T08:13:26.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:26.899+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status"]} v 0) v1 -- 0x7f7c900cb7a0 con 0x7f7c7c006f10 2023-11-24T08:13:27.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.493+0000 7f7c877fe640 1 -- 172.21.15.107:0/1458688946 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_non_clone_status"]}]=0 v447) v1 ==== 154+0+0 (secure 0 0 0) 0x7f7c80057fe0 con 0x7f7c7c006f10 2023-11-24T08:13:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 msgr2=0x7f7c7006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 0x7f7c7006e350 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7c90152010 tx=0x7f7c78008040 comp rx=0 tx=0).stop 2023-11-24T08:13:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 msgr2=0x7f7c900ad4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000e450 tx=0x7f7c8000e910 comp rx=0 tx=0).stop 2023-11-24T08:13:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 shutdown_connections 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c7006be90 0x7f7c7006e350 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c9015caa0 0x7f7c9015cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c900b1220 0x7f7c90151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 --2- 172.21.15.107:0/1458688946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7c006f10 0x7f7c900ad4c0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 >> 172.21.15.107:0/1458688946 conn(0x7f7c9006a920 msgr2=0x7f7c90158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 shutdown_connections 2023-11-24T08:13:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.497+0000 7f7c949c9640 1 -- 172.21.15.107:0/1458688946 wait complete. 2023-11-24T08:13:27.520 INFO:tasks.cephfs_test_runner:test_non_clone_status (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:13:27.521 INFO:tasks.cephfs_test_runner:Starting test: test_periodic_async_work (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) to validate that the async thread (purge thread in this case) will 2023-11-24T08:13:27.521 INFO:tasks.cephfs_test_runner:test_periodic_async_work (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:13:27.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:27.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.732+0000 7f82d7f33640 1 Processor -- start 2023-11-24T08:13:27.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.733+0000 7f82d7f33640 1 -- start start 2023-11-24T08:13:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d7f33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82d006aa30 0x7f82d00cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d7f33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b2690 con 0x7f82d00cf420 2023-11-24T08:13:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d7f33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d006ed00 con 0x7f82d00b28d0 2023-11-24T08:13:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d7f33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d00cc7f0 con 0x7f82d006aa30 2023-11-24T08:13:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d5ccf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82d006aa30 0x7f82d00cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d64d0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82d00cf420 0x7f82d01616d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55676/0 (socket says 172.21.15.107:55676) 2023-11-24T08:13:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.734+0000 7f82d64d0640 1 -- 172.21.15.107:0/3639810138 learned_addr learned my addr 172.21.15.107:0/3639810138 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.735+0000 7f82d5ccf640 1 -- 172.21.15.107:0/3639810138 >> v1:172.21.15.120:6789/0 conn(0x7f82d00b28d0 legacy=0x7f82d00cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.735+0000 7f82d5ccf640 1 -- 172.21.15.107:0/3639810138 >> v1:172.21.15.107:6789/0 conn(0x7f82d00cf420 legacy=0x7f82d01616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.735+0000 7f82d5ccf640 1 -- 172.21.15.107:0/3639810138 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d006efe0 con 0x7f82d006aa30 2023-11-24T08:13:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.736+0000 7f82d5ccf640 1 --2- 172.21.15.107:0/3639810138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d006aa30 0x7f82d00cc2b0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009d90 tx=0x7f82c002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08b48a4ee72830a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.736+0000 7f82d4ccd640 1 -- 172.21.15.107:0/3639810138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c002cd00 con 0x7f82d006aa30 2023-11-24T08:13:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d4ccd640 1 -- 172.21.15.107:0/3639810138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c002ce60 con 0x7f82d006aa30 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d4ccd640 1 -- 172.21.15.107:0/3639810138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c0034660 con 0x7f82d006aa30 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 -- 172.21.15.107:0/3639810138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d006aa30 msgr2=0x7f82d00cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 --2- 172.21.15.107:0/3639810138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d006aa30 0x7f82d00cc2b0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009d90 tx=0x7f82c002bf50 comp rx=0 tx=0).stop 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 -- 172.21.15.107:0/3639810138 shutdown_connections 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 --2- 172.21.15.107:0/3639810138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d006aa30 0x7f82d00cc2b0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 -- 172.21.15.107:0/3639810138 >> 172.21.15.107:0/3639810138 conn(0x7f82d00999d0 msgr2=0x7f82d00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 -- 172.21.15.107:0/3639810138 shutdown_connections 2023-11-24T08:13:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.737+0000 7f82d7f33640 1 -- 172.21.15.107:0/3639810138 wait complete. 2023-11-24T08:13:27.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.740+0000 7f82d7f33640 1 Processor -- start 2023-11-24T08:13:27.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.740+0000 7f82d7f33640 1 -- start start 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 0x7f82d00c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 0x7f82d00ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00c88a0 con 0x7f82d00cf420 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00c9140 con 0x7f82d00b28d0 2023-11-24T08:13:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d7f33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00c9b10 con 0x7f82d006aa30 2023-11-24T08:13:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d64d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d64d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36574/0 (socket says 172.21.15.107:36574) 2023-11-24T08:13:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.741+0000 7f82d64d0640 1 -- 172.21.15.107:0/4156578839 learned_addr learned my addr 172.21.15.107:0/4156578839 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d5ccf640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 0x7f82d00c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d54ce640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 0x7f82d00ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 -- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 msgr2=0x7f82d00c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 0x7f82d00c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 -- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 msgr2=0x7f82d00ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 0x7f82d00ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 -- 172.21.15.107:0/4156578839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0009a70 con 0x7f82d00cf420 2023-11-24T08:13:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d54ce640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 0x7f82d00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d5ccf640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 0x7f82d00c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d64d0640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c3c0 tx=0x7f82c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8005ce0 con 0x7f82d00cf420 2023-11-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8005e40 con 0x7f82d00cf420 2023-11-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d00ae900 con 0x7f82d00cf420 2023-11-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.742+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8007310 con 0x7f82d00cf420 2023-11-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.743+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d00c8dd0 con 0x7f82d00cf420 2023-11-24T08:13:27.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.744+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0067470 con 0x7f82d00cf420 2023-11-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.745+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f82c8007470 con 0x7f82d00cf420 2023-11-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.746+0000 7f82ce7fc640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 0x7f82bc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.746+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82c8087900 con 0x7f82d00cf420 2023-11-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.746+0000 7f82d5ccf640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 0x7f82bc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.747+0000 7f82d5ccf640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 0x7f82bc06e350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f82c0036000 tx=0x7f82c0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:27.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.750+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82c8054140 con 0x7f82d00cf420 2023-11-24T08:13:27.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.943+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d00c9280 con 0x7f82d00cf420 2023-11-24T08:13:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.944+0000 7f82ce7fc640 1 -- 172.21.15.107:0/4156578839 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+12998 (secure 0 0 0) 0x7f82c8017090 con 0x7f82d00cf420 2023-11-24T08:13:27.946 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:27.946 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":112,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:25.907296+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/2728005588":"2023-11-25T08:13:22.310235+0000","172.21.15.107:6835/2728005588":"2023-11-25T08:13:22.310235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:27.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 msgr2=0x7f82bc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 0x7f82bc06e350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f82c0036000 tx=0x7f82c0030000 comp rx=0 tx=0).stop 2023-11-24T08:13:27.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 msgr2=0x7f82d01566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:27.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c3c0 tx=0x7f82c800c880 comp rx=0 tx=0).stop 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 shutdown_connections 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82bc06be90 0x7f82bc06e350 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d00cf420 0x7f82d01566b0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d00b28d0 0x7f82d00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 --2- 172.21.15.107:0/4156578839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d006aa30 0x7f82d00c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 >> 172.21.15.107:0/4156578839 conn(0x7f82d00999d0 msgr2=0x7f82d00afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:27.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 shutdown_connections 2023-11-24T08:13:27.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:27.947+0000 7f82d7f33640 1 -- 172.21.15.107:0/4156578839 wait complete. 2023-11-24T08:13:27.967 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:13:27.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:13:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.182+0000 7fe6fa236640 1 Processor -- start 2023-11-24T08:13:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.182+0000 7fe6fa236640 1 -- start start 2023-11-24T08:13:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6fa236640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6f40cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6fa236640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6f406d600 0x7fe6f40cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6fa236640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f415e400 con 0x7fe6f415d7f0 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6fa236640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f40ce880 con 0x7fe6f406d600 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6fa236640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40c46e0 con 0x7fe6f40c4220 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe6f415d7f0 0x7fe6f40c5990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43502/0 (socket says 172.21.15.107:43502) 2023-11-24T08:13:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f37fe640 1 -- 172.21.15.107:0/1626551229 learned_addr learned my addr 172.21.15.107:0/1626551229 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231780075 0 0) 0x7fe6f415e400 con 0x7fe6f415d7f0 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6dc003440 con 0x7fe6f415d7f0 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f3fff640 1 --2- 172.21.15.107:0/1626551229 >> v2:172.21.15.120:3300/0 conn(0x7fe6f406d600 0x7fe6f40cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.184+0000 7fe6f2ffd640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6f40cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657095864 0 0) 0x7fe6dc003440 con 0x7fe6f415d7f0 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 msgr2=0x7fe6f40cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6f40cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 >> v2:172.21.15.120:3300/0 conn(0x7fe6f406d600 msgr2=0x7fe6f40cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/1626551229 >> v2:172.21.15.120:3300/0 conn(0x7fe6f406d600 0x7fe6f40cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40c49c0 con 0x7fe6f415d7f0 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe6e4001670 con 0x7fe6f415d7f0 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f2ffd640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6f40cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 >> v1:172.21.15.107:6789/0 conn(0x7fe6f415d7f0 legacy=0x7fe6f40c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6dc003a60 0x7fe6dc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc008850 0x7fe6dc00ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6dc00d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40c46e0 con 0x7fe6dc003a60 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc00db10 con 0x7fe6dc008850 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc00dcd0 con 0x7fe6f40c4220 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6dc003a60 0x7fe6dc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f2ffd640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc008850 0x7fe6dc00ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 -- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 msgr2=0x7fe6dc00d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f40c4220 0x7fe6dc00d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 -- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc008850 msgr2=0x7fe6dc00ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc008850 0x7fe6dc00ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.185+0000 7fe6f37fe640 1 -- 172.21.15.107:0/1626551229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc003440 con 0x7fe6dc003a60 2023-11-24T08:13:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6dc003a60 0x7fe6dc003e80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400c110 tx=0x7fe6dc0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e040c6bd786fe1cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4005ad0 con 0x7fe6dc003a60 2023-11-24T08:13:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4005c50 con 0x7fe6dc003a60 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6f27fc640 1 -- 172.21.15.107:0/1626551229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e402c830 con 0x7fe6dc003a60 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6fa236640 1 -- 172.21.15.107:0/1626551229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6dc003a60 msgr2=0x7fe6dc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6fa236640 1 --2- 172.21.15.107:0/1626551229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6dc003a60 0x7fe6dc003e80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400c110 tx=0x7fe6dc0036f0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6f3fff640 1 -- 172.21.15.107:0/1626551229 reap_dead start 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6fa236640 1 -- 172.21.15.107:0/1626551229 shutdown_connections 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.187+0000 7fe6fa236640 1 -- 172.21.15.107:0/1626551229 >> 172.21.15.107:0/1626551229 conn(0x7fe6f40b0c50 msgr2=0x7fe6f40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.188+0000 7fe6fa236640 1 -- 172.21.15.107:0/1626551229 shutdown_connections 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.188+0000 7fe6fa236640 1 -- 172.21.15.107:0/1626551229 wait complete. 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.189+0000 7fe6fa236640 1 Processor -- start 2023-11-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 -- start start 2023-11-24T08:13:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 0x7fe6f41f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 0x7fe6f40cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41f76f0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41f6760 con 0x7fe6dc003a60 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.190+0000 7fe6fa236640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40cc170 con 0x7fe6f41f2a30 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47384/0 (socket says 172.21.15.107:47384) 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 -- 172.21.15.107:0/4174840572 learned_addr learned my addr 172.21.15.107:0/4174840572 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 0x7fe6f41f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f2ffd640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 0x7fe6f40cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 -- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 msgr2=0x7fe6f40cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 0x7fe6f40cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 -- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 msgr2=0x7fe6f41f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 0x7fe6f41f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f3fff640 1 -- 172.21.15.107:0/4174840572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40cc450 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 0x7fe6f41f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.191+0000 7fe6f2ffd640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 0x7fe6f40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6f3fff640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fe6f415b350 tx=0x7fe6ec00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec010040 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec00bce0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec014640 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f41f69b0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.192+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f41f2eb0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.193+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f41f6ed0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.194+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fe6ec0147a0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.195+0000 7fe6dbfff640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 0x7fe6e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.196+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6ec0889b0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.196+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 0x7fe6e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.196+0000 7fe6f37fe640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 0x7fe6e006e350 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4034000 tx=0x7fe6e4007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.200+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6f41f6ed0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.388+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6f41f6ed0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.389+0000 7fe6dbfff640 1 -- 172.21.15.107:0/4174840572 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6f41f6ed0 con 0x7fe6f40cbcd0 2023-11-24T08:13:28.391 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 msgr2=0x7fe6e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 0x7fe6e006e350 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4034000 tx=0x7fe6e4007000 comp rx=0 tx=0).stop 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 msgr2=0x7fe6f415c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fe6f415b350 tx=0x7fe6ec00b480 comp rx=0 tx=0).stop 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 shutdown_connections 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe6e006be90 0x7fe6e006e350 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6f40cbcd0 0x7fe6f415c000 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f41f2a30 0x7fe6f40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 --2- 172.21.15.107:0/4174840572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc003a60 0x7fe6f41f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 >> 172.21.15.107:0/4174840572 conn(0x7fe6f40b0c50 msgr2=0x7fe6f415b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 shutdown_connections 2023-11-24T08:13:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.392+0000 7fe6fa236640 1 -- 172.21.15.107:0/4174840572 wait complete. 2023-11-24T08:13:28.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work' 2023-11-24T08:13:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.627+0000 7f7c7c561640 1 Processor -- start 2023-11-24T08:13:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.627+0000 7f7c7c561640 1 -- start start 2023-11-24T08:13:28.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7c561640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c7415c7f0 0x7f7c7406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7c561640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c740cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7c561640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740cece0 con 0x7f7c740cf450 2023-11-24T08:13:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7c561640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c740cfe40 con 0x7f7c7415c7f0 2023-11-24T08:13:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7c561640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74156010 con 0x7f7c740ce6c0 2023-11-24T08:13:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7a2fd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c7415c7f0 0x7f7c7406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7aafe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c740cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7a2fd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c7415c7f0 0x7f7c7406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:36320/0 (socket says 172.21.15.107:36320) 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.629+0000 7f7c7a2fd640 1 -- 172.21.15.107:0/4120254294 learned_addr learned my addr 172.21.15.107:0/4120254294 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.630+0000 7f7c7a2fd640 1 -- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 msgr2=0x7f7c740cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.630+0000 7f7c7a2fd640 1 --2- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c740cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.630+0000 7f7c7a2fd640 1 -- 172.21.15.107:0/4120254294 >> v1:172.21.15.120:6789/0 conn(0x7f7c740cf450 legacy=0x7f7c740cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.630+0000 7f7c7a2fd640 1 -- 172.21.15.107:0/4120254294 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7406d040 con 0x7f7c7415c7f0 2023-11-24T08:13:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.632+0000 7f7c7a2fd640 1 --2- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7415c7f0 0x7f7c7406db20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009870 tx=0x7f7c7002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24a947ed38bec11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.632+0000 7f7c792fb640 1 -- 172.21.15.107:0/4120254294 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70030840 con 0x7f7c7415c7f0 2023-11-24T08:13:28.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.632+0000 7f7c792fb640 1 -- 172.21.15.107:0/4120254294 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c700309a0 con 0x7f7c7415c7f0 2023-11-24T08:13:28.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.632+0000 7f7c792fb640 1 -- 172.21.15.107:0/4120254294 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70030d30 con 0x7f7c7415c7f0 2023-11-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 -- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7415c7f0 msgr2=0x7f7c7406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 --2- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7415c7f0 0x7f7c7406db20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009870 tx=0x7f7c7002b980 comp rx=0 tx=0).stop 2023-11-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 -- 172.21.15.107:0/4120254294 shutdown_connections 2023-11-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 --2- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c740cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 --2- 172.21.15.107:0/4120254294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c7415c7f0 0x7f7c7406db20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 -- 172.21.15.107:0/4120254294 >> 172.21.15.107:0/4120254294 conn(0x7f7c7406a920 msgr2=0x7f7c7406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 -- 172.21.15.107:0/4120254294 shutdown_connections 2023-11-24T08:13:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.633+0000 7f7c7c561640 1 -- 172.21.15.107:0/4120254294 wait complete. 2023-11-24T08:13:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.635+0000 7f7c7c561640 1 Processor -- start 2023-11-24T08:13:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.635+0000 7f7c7c561640 1 -- start start 2023-11-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c7414da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c740cf450 0x7f7c74098010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7414b330 con 0x7f7c7415c7f0 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7414b610 con 0x7f7c740cf450 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.636+0000 7f7c7c561640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7414b8f0 con 0x7f7c740ce6c0 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47394/0 (socket says 172.21.15.107:47394) 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 -- 172.21.15.107:0/2311815960 learned_addr learned my addr 172.21.15.107:0/2311815960 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7a2fd640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c7414da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 -- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 msgr2=0x7f7c7414da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c79afc640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c740cf450 0x7f7c74098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c7414da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 -- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c740cf450 msgr2=0x7f7c74098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c740cf450 0x7f7c74098010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 -- 172.21.15.107:0/2311815960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700094d0 con 0x7f7c7415c7f0 2023-11-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.637+0000 7f7c7aafe640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00efc0 tx=0x7f7c6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.638+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c009b10 con 0x7f7c7415c7f0 2023-11-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.638+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c00f040 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.638+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c74097810 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.638+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0146c0 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.638+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7414df90 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.639+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74067470 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.640+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f7c6c014820 con 0x7f7c7415c7f0 2023-11-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.641+0000 7f7c6affd640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 0x7f7c5806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:28.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.641+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c6c088f90 con 0x7f7c7415c7f0 2023-11-24T08:13:28.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.642+0000 7f7c7a2fd640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 0x7f7c5806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:28.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.642+0000 7f7c7a2fd640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 0x7f7c5806e350 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7c70036000 tx=0x7f7c7003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.646+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c6c055770 con 0x7f7c7415c7f0 2023-11-24T08:13:28.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:28.908+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work"]} v 0) v1 -- 0x7f7c740cd730 con 0x7f7c7415c7f0 2023-11-24T08:13:29.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.508+0000 7f7c6affd640 1 -- 172.21.15.107:0/2311815960 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work"]}]=0 v449) v1 ==== 160+0+0 (secure 0 0 0) 0x7f7c6c018020 con 0x7f7c7415c7f0 2023-11-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.512+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 msgr2=0x7f7c5806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.512+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 0x7f7c5806e350 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7c70036000 tx=0x7f7c7003b040 comp rx=0 tx=0).stop 2023-11-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.512+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 msgr2=0x7f7c740b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.512+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00efc0 tx=0x7f7c6c00c490 comp rx=0 tx=0).stop 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 shutdown_connections 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c5806be90 0x7f7c5806e350 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c7415c7f0 0x7f7c740b2f30 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c740cf450 0x7f7c74098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 --2- 172.21.15.107:0/2311815960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c740ce6c0 0x7f7c7414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 >> 172.21.15.107:0/2311815960 conn(0x7f7c7406a920 msgr2=0x7f7c740afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 shutdown_connections 2023-11-24T08:13:29.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.513+0000 7f7c7c561640 1 -- 172.21.15.107:0/2311815960 wait complete. 2023-11-24T08:13:29.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:13:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.753+0000 7f536e072640 1 Processor -- start 2023-11-24T08:13:29.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.754+0000 7f536e072640 1 -- start start 2023-11-24T08:13:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.755+0000 7f536e072640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53680c59b0 0x7f5368166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.755+0000 7f536e072640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680c5dc0 con 0x7f53680cc0f0 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.755+0000 7f536e072640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536815eea0 con 0x7f536815b440 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.755+0000 7f536e072640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536815b820 con 0x7f53680c59b0 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53680c59b0 0x7f5368166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53680c59b0 0x7f5368166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47400/0 (socket says 172.21.15.107:47400) 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 -- 172.21.15.107:0/4098537244 learned_addr learned my addr 172.21.15.107:0/4098537244 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 -- 172.21.15.107:0/4098537244 >> v1:172.21.15.120:6789/0 conn(0x7f536815b440 legacy=0x7f53680cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 -- 172.21.15.107:0/4098537244 >> v1:172.21.15.107:6789/0 conn(0x7f53680cc0f0 legacy=0x7f53680c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5367fff640 1 -- 172.21.15.107:0/4098537244 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536806d600 con 0x7f53680c59b0 2023-11-24T08:13:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.756+0000 7f5366ffd640 1 -- 172.21.15.107:0/4098537244 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399967546 0 0) 0x7f53680c5dc0 con 0x7f53680cc0f0 2023-11-24T08:13:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.757+0000 7f5367fff640 1 --2- 172.21.15.107:0/4098537244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368166230 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f5360009e50 tx=0x7f536002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccb618a65b67de9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f5366ffd640 1 -- 172.21.15.107:0/4098537244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536002cb20 con 0x7f53680c59b0 2023-11-24T08:13:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f5366ffd640 1 -- 172.21.15.107:0/4098537244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536002cc80 con 0x7f53680c59b0 2023-11-24T08:13:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f5366ffd640 1 -- 172.21.15.107:0/4098537244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53600343f0 con 0x7f53680c59b0 2023-11-24T08:13:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f536e072640 1 -- 172.21.15.107:0/4098537244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 msgr2=0x7f5368166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f536e072640 1 --2- 172.21.15.107:0/4098537244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368166230 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f5360009e50 tx=0x7f536002bf80 comp rx=0 tx=0).stop 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f536e072640 1 -- 172.21.15.107:0/4098537244 shutdown_connections 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f536e072640 1 --2- 172.21.15.107:0/4098537244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368166230 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.758+0000 7f536e072640 1 -- 172.21.15.107:0/4098537244 >> 172.21.15.107:0/4098537244 conn(0x7f53680b0c50 msgr2=0x7f53680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.759+0000 7f536e072640 1 -- 172.21.15.107:0/4098537244 shutdown_connections 2023-11-24T08:13:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.759+0000 7f536e072640 1 -- 172.21.15.107:0/4098537244 wait complete. 2023-11-24T08:13:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.760+0000 7f536e072640 1 Processor -- start 2023-11-24T08:13:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.761+0000 7f536e072640 1 -- start start 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53680cc0f0 0x7f53680acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536815b440 0x7f53680b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680ad4b0 con 0x7f53680c59b0 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b32e0 con 0x7f53680cc0f0 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f536e072640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b35c0 con 0x7f536815b440 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47408/0 (socket says 172.21.15.107:47408) 2023-11-24T08:13:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 -- 172.21.15.107:0/3155308404 learned_addr learned my addr 172.21.15.107:0/3155308404 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 -- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536815b440 msgr2=0x7f53680b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f5367fff640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536815b440 0x7f53680b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f535ffff640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53680cc0f0 0x7f53680acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536815b440 0x7f53680b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 -- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53680cc0f0 msgr2=0x7f53680acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53680cc0f0 0x7f53680acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.762+0000 7f53677fe640 1 -- 172.21.15.107:0/3155308404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5360009b30 con 0x7f53680c59b0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f53677fe640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f535800c400 tx=0x7f535800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53580052d0 con 0x7f53680c59b0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5358005430 con 0x7f53680c59b0 2023-11-24T08:13:29.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53680b38a0 con 0x7f53680c59b0 2023-11-24T08:13:29.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358003830 con 0x7f53680c59b0 2023-11-24T08:13:29.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.763+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53680ad070 con 0x7f53680c59b0 2023-11-24T08:13:29.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.764+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53680b30e0 con 0x7f53680c59b0 2023-11-24T08:13:29.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.765+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f5358003990 con 0x7f53680c59b0 2023-11-24T08:13:29.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.766+0000 7f5364ff9640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 0x7f534806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:29.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.766+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5358087a30 con 0x7f53680c59b0 2023-11-24T08:13:29.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.766+0000 7f535ffff640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 0x7f534806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:29.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.767+0000 7f535ffff640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 0x7f534806e350 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f53680b06f0 tx=0x7f5350009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:29.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.770+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5358054210 con 0x7f53680c59b0 2023-11-24T08:13:29.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.948+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f536804e8b0 con 0x7f53680c59b0 2023-11-24T08:13:29.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.954+0000 7f5364ff9640 1 -- 172.21.15.107:0/3155308404 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f53580170c0 con 0x7f53680c59b0 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.956+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 msgr2=0x7f534806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.956+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 0x7f534806e350 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f53680b06f0 tx=0x7f5350009040 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.956+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 msgr2=0x7f5368153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.956+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f535800c400 tx=0x7f535800c8c0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 shutdown_connections 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f534806be90 0x7f534806e350 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536815b440 0x7f53680b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53680cc0f0 0x7f53680acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 --2- 172.21.15.107:0/3155308404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53680c59b0 0x7f5368153ec0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:29.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 >> 172.21.15.107:0/3155308404 conn(0x7f53680b0c50 msgr2=0x7f53680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:29.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 shutdown_connections 2023-11-24T08:13:29.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:29.957+0000 7f536e072640 1 -- 172.21.15.107:0/3155308404 wait complete. 2023-11-24T08:13:29.977 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:13:29.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:13:30.086 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:13:30.212 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:13:30.212 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:30.213 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:30.273 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:13:30.273 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:30.273 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:30.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:13:30.332 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:30.332 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:30.392 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:13:30.392 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:30.393 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:30.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:13:30.453 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:13:30.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.681+0000 7f2713997640 1 Processor -- start 2023-11-24T08:13:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.681+0000 7f2713997640 1 -- start start 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2713997640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f270c070b80 0x7f270c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2713997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2713997640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0b4230 con 0x7f270c0707a0 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2713997640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270c0b3570 con 0x7f270c070b80 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2713997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c0b3f50 con 0x7f270c0b30c0 2023-11-24T08:13:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.683+0000 7f2711f34640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2710f32640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f270c070b80 0x7f270c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2711f34640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:41590/0 (socket says 172.21.15.107:41590) 2023-11-24T08:13:30.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2711f34640 1 -- 172.21.15.107:0/1950604692 learned_addr learned my addr 172.21.15.107:0/1950604692 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:30.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2710f32640 1 -- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 msgr2=0x7f270c06d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:30.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2710f32640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2710f32640 1 -- 172.21.15.107:0/1950604692 >> v1:172.21.15.120:6789/0 conn(0x7f270c0707a0 legacy=0x7f270c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2710f32640 1 -- 172.21.15.107:0/1950604692 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c0b25e0 con 0x7f270c070b80 2023-11-24T08:13:30.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.684+0000 7f2711f34640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:13:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.686+0000 7f2710f32640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c070b80 0x7f270c0b3930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15eddcad0eaea927 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2703fff640 1 -- 172.21.15.107:0/1950604692 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc031780 con 0x7f270c070b80 2023-11-24T08:13:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2703fff640 1 -- 172.21.15.107:0/1950604692 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc0318e0 con 0x7f270c070b80 2023-11-24T08:13:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2703fff640 1 -- 172.21.15.107:0/1950604692 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc031c70 con 0x7f270c070b80 2023-11-24T08:13:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 -- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c070b80 msgr2=0x7f270c0b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c070b80 0x7f270c0b3930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b820 comp rx=0 tx=0).stop 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 -- 172.21.15.107:0/1950604692 shutdown_connections 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c06d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 --2- 172.21.15.107:0/1950604692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c070b80 0x7f270c0b3930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.687+0000 7f2713997640 1 -- 172.21.15.107:0/1950604692 >> 172.21.15.107:0/1950604692 conn(0x7f270c0b0be0 msgr2=0x7f270c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.688+0000 7f2713997640 1 -- 172.21.15.107:0/1950604692 shutdown_connections 2023-11-24T08:13:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.688+0000 7f2713997640 1 -- 172.21.15.107:0/1950604692 wait complete. 2023-11-24T08:13:30.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.689+0000 7f2713997640 1 Processor -- start 2023-11-24T08:13:30.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.689+0000 7f2713997640 1 -- start start 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 0x7f270c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c14ea40 con 0x7f270c070b80 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c14d580 con 0x7f270c0707a0 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.690+0000 7f2713997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c09aff0 con 0x7f270c0b30c0 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47418/0 (socket says 172.21.15.107:47418) 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 -- 172.21.15.107:0/992448862 learned_addr learned my addr 172.21.15.107:0/992448862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2711733640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 0x7f270c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 -- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 msgr2=0x7f270c0ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2711f34640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c0ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 -- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 msgr2=0x7f270c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 0x7f270c0999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 -- 172.21.15.107:0/992448862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2704009b30 con 0x7f270c070b80 2023-11-24T08:13:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2711733640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 0x7f270c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.691+0000 7f2710f32640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f26fc034040 tx=0x7f26fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.692+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc032820 con 0x7f270c070b80 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.692+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc032980 con 0x7f270c070b80 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.692+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc009400 con 0x7f270c070b80 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.692+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc032d30 con 0x7f270c070b80 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.692+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c09b5b0 con 0x7f270c070b80 2023-11-24T08:13:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.693+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c14eb80 con 0x7f270c070b80 2023-11-24T08:13:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.694+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f26fc04e440 con 0x7f270c070b80 2023-11-24T08:13:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.695+0000 7f2701ffb640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 0x7f26f006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.695+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26fc0aa590 con 0x7f270c070b80 2023-11-24T08:13:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.695+0000 7f2711733640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 0x7f26f006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.696+0000 7f2711733640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 0x7f26f006e350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2708000f60 tx=0x7f2708009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.699+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f270c14eb80 con 0x7f270c070b80 2023-11-24T08:13:30.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.926+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f270c14eb80 con 0x7f270c070b80 2023-11-24T08:13:30.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.927+0000 7f2701ffb640 1 -- 172.21.15.107:0/992448862 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f270c14eb80 con 0x7f270c070b80 2023-11-24T08:13:30.929 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:30.929 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":62,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/115164203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":115164203},{"type":"v1","addr":"172.21.15.120:6834","nonce":115164203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3402497925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3402497925},{"type":"v1","addr":"172.21.15.120:6835","nonce":3402497925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7860,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/840247672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":840247672},{"type":"v1","addr":"172.21.15.107:6836","nonce":840247672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":8214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/483009595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":483009595},{"type":"v1","addr":"172.21.15.107:6837","nonce":483009595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[]} 2023-11-24T08:13:30.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 62 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 msgr2=0x7f26f006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 0x7f26f006e350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2708000f60 tx=0x7f2708009040 comp rx=0 tx=0).stop 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 msgr2=0x7f270c09a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f26fc034040 tx=0x7f26fc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 shutdown_connections 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26f006be90 0x7f26f006e350 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f270c0b30c0 0x7f270c0ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f270c070b80 0x7f270c09a660 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 --2- 172.21.15.107:0/992448862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f270c0707a0 0x7f270c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 >> 172.21.15.107:0/992448862 conn(0x7f270c0b0be0 msgr2=0x7f270c0ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 shutdown_connections 2023-11-24T08:13:30.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:30.930+0000 7f2713997640 1 -- 172.21.15.107:0/992448862 wait complete. 2023-11-24T08:13:30.952 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:13:30.952 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:13:30.952 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:13:30.952 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:13:30.952 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:13:30.952 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:13:30.953 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:13:30.953 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:13:30.953 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:13:30.953 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:13:30.953 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:13:30.953 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:13:30.953 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:13:30.954 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:13:30.954 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:13:30.954 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:13:30.968 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:13:30.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:13:30.971 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:13:30.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:13:30.975 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:13:30.975 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:13:30.977 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:13:30.977 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:13:31.020 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:13:31.020 DEBUG:teuthology.parallel:result is None 2023-11-24T08:13:31.025 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:13:31.025 DEBUG:teuthology.parallel:result is None 2023-11-24T08:13:31.027 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:13:31.028 DEBUG:teuthology.parallel:result is None 2023-11-24T08:13:31.032 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:13:31.032 DEBUG:teuthology.parallel:result is None 2023-11-24T08:13:31.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:13:31.179 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:13:31.182 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:13:31.187 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:13:31.187 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:13:31.189 INFO:tasks.ceph.mds.d.smithi120.stderr:2023-11-24T08:13:31.188+0000 7f448467f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-11-24T08:13:31.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.220+0000 7f361791d640 1 Processor -- start 2023-11-24T08:13:31.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.221+0000 7f361791d640 1 -- start start 2023-11-24T08:13:31.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f361791d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3610146eb0 0x7f3610161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f361791d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3610161a10 con 0x7f36100cd2e0 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f361791d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3610161cf0 con 0x7f361006e160 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f361791d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3610161fd0 con 0x7f3610146eb0 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f36156b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3610146eb0 0x7f3610161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f3615eba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f36100cd2e0 0x7f3610070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43528/0 (socket says 172.21.15.107:43528) 2023-11-24T08:13:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.222+0000 7f3615eba640 1 -- 172.21.15.107:0/1669201617 learned_addr learned my addr 172.21.15.107:0/1669201617 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.223+0000 7f3607fff640 1 -- 172.21.15.107:0/1669201617 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695267423 0 0) 0x7f3610161cf0 con 0x7f361006e160 2023-11-24T08:13:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.223+0000 7f36156b9640 1 -- 172.21.15.107:0/1669201617 >> v1:172.21.15.120:6789/0 conn(0x7f361006e160 legacy=0x7f36100ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.223+0000 7f36156b9640 1 -- 172.21.15.107:0/1669201617 >> v1:172.21.15.107:6789/0 conn(0x7f36100cd2e0 legacy=0x7f3610070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.223+0000 7f36156b9640 1 -- 172.21.15.107:0/1669201617 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3610070660 con 0x7f3610146eb0 2023-11-24T08:13:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.224+0000 7f36156b9640 1 --2- 172.21.15.107:0/1669201617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f3610161150 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f360c0097a0 tx=0x7f360c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bef1d7f6f3723a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.224+0000 7f3607fff640 1 -- 172.21.15.107:0/1669201617 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c030710 con 0x7f3610146eb0 2023-11-24T08:13:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.224+0000 7f3607fff640 1 -- 172.21.15.107:0/1669201617 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c02e050 con 0x7f3610146eb0 2023-11-24T08:13:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.224+0000 7f3607fff640 1 -- 172.21.15.107:0/1669201617 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c030a90 con 0x7f3610146eb0 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 -- 172.21.15.107:0/1669201617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 msgr2=0x7f3610161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 --2- 172.21.15.107:0/1669201617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f3610161150 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f360c0097a0 tx=0x7f360c02b790 comp rx=0 tx=0).stop 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 -- 172.21.15.107:0/1669201617 shutdown_connections 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 --2- 172.21.15.107:0/1669201617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f3610161150 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 -- 172.21.15.107:0/1669201617 >> 172.21.15.107:0/1669201617 conn(0x7f36100b0c50 msgr2=0x7f36100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 -- 172.21.15.107:0/1669201617 shutdown_connections 2023-11-24T08:13:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.225+0000 7f361791d640 1 -- 172.21.15.107:0/1669201617 wait complete. 2023-11-24T08:13:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.227+0000 7f361791d640 1 Processor -- start 2023-11-24T08:13:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.227+0000 7f361791d640 1 -- start start 2023-11-24T08:13:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f361006e160 0x7f36101f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36100cd2e0 0x7f3610143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361009a370 con 0x7f3610146eb0 2023-11-24T08:13:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36101f3f80 con 0x7f36100cd2e0 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f361791d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36100999d0 con 0x7f361006e160 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47456/0 (socket says 172.21.15.107:47456) 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 -- 172.21.15.107:0/2861561043 learned_addr learned my addr 172.21.15.107:0/2861561043 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f36156b9640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f361006e160 0x7f36101f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3614eb8640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36100cd2e0 0x7f3610143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 -- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f361006e160 msgr2=0x7f36101f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f361006e160 0x7f36101f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 -- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36100cd2e0 msgr2=0x7f3610143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36100cd2e0 0x7f3610143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.228+0000 7f3615eba640 1 -- 172.21.15.107:0/2861561043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c009400 con 0x7f3610146eb0 2023-11-24T08:13:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f3615eba640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f360800caf0 tx=0x7f3608002070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800a700 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360800a880 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800acb0 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3610099cb0 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.229+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361009c120 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.230+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361009a4b0 con 0x7f3610146eb0 2023-11-24T08:13:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.231+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f36080070d0 con 0x7f3610146eb0 2023-11-24T08:13:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.232+0000 7f3605ffb640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 0x7f35f406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.233+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3608087bd0 con 0x7f3610146eb0 2023-11-24T08:13:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.233+0000 7f36156b9640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 0x7f35f406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.233+0000 7f36156b9640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 0x7f35f406e350 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f360c034000 tx=0x7f360c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.237+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f361009a4b0 con 0x7f3610146eb0 2023-11-24T08:13:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.436+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f361009a4b0 con 0x7f3610146eb0 2023-11-24T08:13:31.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.548+0000 7f3605ffb640 1 -- 172.21.15.107:0/2861561043 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v114) v1 ==== 97+0+0 (secure 0 0 0) 0x7f361009a4b0 con 0x7f3610146eb0 2023-11-24T08:13:31.550 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 msgr2=0x7f35f406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 0x7f35f406e350 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f360c034000 tx=0x7f360c02f000 comp rx=0 tx=0).stop 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 msgr2=0x7f361009bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f360800caf0 tx=0x7f3608002070 comp rx=0 tx=0).stop 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 shutdown_connections 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35f406be90 0x7f35f406e350 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3610146eb0 0x7f361009bbe0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36100cd2e0 0x7f3610143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 --2- 172.21.15.107:0/2861561043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f361006e160 0x7f36101f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 >> 172.21.15.107:0/2861561043 conn(0x7f36100b0c50 msgr2=0x7f3610071950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 shutdown_connections 2023-11-24T08:13:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.551+0000 7f361791d640 1 -- 172.21.15.107:0/2861561043 wait complete. 2023-11-24T08:13:31.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:13:31.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.766+0000 7f28509ef640 1 Processor -- start 2023-11-24T08:13:31.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.766+0000 7f28509ef640 1 -- start start 2023-11-24T08:13:31.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.767+0000 7f28509ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f284c15e6b0 0x7f284c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.767+0000 7f28509ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f284c0b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f28509ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284c15dc10 con 0x7f284c157a90 2023-11-24T08:13:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f28509ef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284c15eb60 con 0x7f284c15e6b0 2023-11-24T08:13:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f28509ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c06d350 con 0x7f284c06cee0 2023-11-24T08:13:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f284b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f284c157a90 0x7f284c15a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43532/0 (socket says 172.21.15.107:43532) 2023-11-24T08:13:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f284b7fe640 1 -- 172.21.15.107:0/2652813170 learned_addr learned my addr 172.21.15.107:0/2652813170 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635976765 0 0) 0x7f284c15dc10 con 0x7f284c157a90 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2838003520 con 0x7f284c157a90 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f284bfff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f284c0b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.768+0000 7f2843fff640 1 --2- 172.21.15.107:0/2652813170 >> v2:172.21.15.120:3300/0 conn(0x7f284c15e6b0 0x7f284c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705013431 0 0) 0x7f2838003520 con 0x7f284c157a90 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 msgr2=0x7f284c0b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f284c0b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 >> v2:172.21.15.120:3300/0 conn(0x7f284c15e6b0 msgr2=0x7f284c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 --2- 172.21.15.107:0/2652813170 >> v2:172.21.15.120:3300/0 conn(0x7f284c15e6b0 0x7f284c06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c15b130 con 0x7f284c157a90 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f283c001a10 con 0x7f284c157a90 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 >> v1:172.21.15.107:6789/0 conn(0x7f284c157a90 legacy=0x7f284c15a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f2838003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2838003fa0 0x7f28380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2838004d00 0x7f2838005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c15eb60 con 0x7f2838004d00 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c06d350 con 0x7f2838003fa0 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283800ba80 con 0x7f284c06cee0 2023-11-24T08:13:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284bfff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2838004d00 0x7f2838005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284b7fe640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f2838003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284bfff640 1 -- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 msgr2=0x7f2838003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f2843fff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2838003fa0 0x7f28380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284bfff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f284c06cee0 0x7f2838003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.769+0000 7f284bfff640 1 -- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2838003fa0 msgr2=0x7f28380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.770+0000 7f284bfff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2838003fa0 0x7f28380047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.770+0000 7f284bfff640 1 -- 172.21.15.107:0/2652813170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838003520 con 0x7f2838004d00 2023-11-24T08:13:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f284bfff640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2838004d00 0x7f2838005180 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f284400ba30 tx=0x7f28380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173ed5865f6de5e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844032640 con 0x7f2838004d00 2023-11-24T08:13:31.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28440327a0 con 0x7f2838004d00 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f284affd640 1 -- 172.21.15.107:0/2652813170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844032b30 con 0x7f2838004d00 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f28509ef640 1 -- 172.21.15.107:0/2652813170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2838004d00 msgr2=0x7f2838005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f28509ef640 1 --2- 172.21.15.107:0/2652813170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2838004d00 0x7f2838005180 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f284400ba30 tx=0x7f28380057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f284bfff640 1 -- 172.21.15.107:0/2652813170 reap_dead start 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f28509ef640 1 -- 172.21.15.107:0/2652813170 shutdown_connections 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f28509ef640 1 -- 172.21.15.107:0/2652813170 >> 172.21.15.107:0/2652813170 conn(0x7f284c0b4630 msgr2=0x7f284c0b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.771+0000 7f28509ef640 1 -- 172.21.15.107:0/2652813170 shutdown_connections 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.772+0000 7f28509ef640 1 -- 172.21.15.107:0/2652813170 wait complete. 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.773+0000 7f28509ef640 1 Processor -- start 2023-11-24T08:13:31.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.774+0000 7f28509ef640 1 -- start start 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.774+0000 7f28509ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2838004d00 0x7f284c0c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f28509ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f28509ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c140230 0x7f284c1406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f28509ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c0c89b0 con 0x7f284c1522b0 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f28509ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c13f1b0 con 0x7f284c140230 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f28509ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c13f470 con 0x7f2838004d00 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47460/0 (socket says 172.21.15.107:47460) 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 -- 172.21.15.107:0/3713898838 learned_addr learned my addr 172.21.15.107:0/3713898838 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 -- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2838004d00 msgr2=0x7f284c0c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2838004d00 0x7f284c0c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 -- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c140230 msgr2=0x7f284c1406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c140230 0x7f284c1406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 -- 172.21.15.107:0/3713898838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c13f750 con 0x7f284c1522b0 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.775+0000 7f2843fff640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f283400e7e0 tx=0x7f283400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.776+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283400c650 con 0x7f284c1522b0 2023-11-24T08:13:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.776+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283400c7b0 con 0x7f284c1522b0 2023-11-24T08:13:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.776+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283400cb20 con 0x7f284c1522b0 2023-11-24T08:13:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.776+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c13fa30 con 0x7f284c1522b0 2023-11-24T08:13:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.776+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c1529b0 con 0x7f284c1522b0 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.777+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c04e840 con 0x7f284c1522b0 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.778+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f283400cc80 con 0x7f284c1522b0 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.779+0000 7f2848ff9640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 0x7f282c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.779+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2834087f80 con 0x7f284c1522b0 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.779+0000 7f284b7fe640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 0x7f282c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.780+0000 7f284b7fe640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 0x7f282c06e340 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f283c007c40 tx=0x7f283c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:31.783+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28340549f0 con 0x7f284c1522b0 2023-11-24T08:13:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.024+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f284c072640 con 0x7f284c1522b0 2023-11-24T08:13:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.026+0000 7f2848ff9640 1 -- 172.21.15.107:0/3713898838 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2834021070 con 0x7f284c1522b0 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:13:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:13:32.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:13:32.031 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.033 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:13:32.034 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:13:32.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.034 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.036 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:13:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.038 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.041 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:13:32.042 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:13:32.043 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:13:32.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:13:32.046 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.047 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:13:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:13:32.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.051 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.052 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:13:32.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:13:32.054 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:13:32.055 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.055 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:32.055 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:13:32.055 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 msgr2=0x7f282c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 0x7f282c06e340 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f283c007c40 tx=0x7f283c009040 comp rx=0 tx=0).stop 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 msgr2=0x7f284c13e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f283400e7e0 tx=0x7f283400eca0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 shutdown_connections 2023-11-24T08:13:32.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282c06be80 0x7f282c06e340 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c140230 0x7f284c1406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f284c1522b0 0x7f284c13e860 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 --2- 172.21.15.107:0/3713898838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2838004d00 0x7f284c0c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 >> 172.21.15.107:0/3713898838 conn(0x7f284c0b4630 msgr2=0x7f284c0cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 shutdown_connections 2023-11-24T08:13:32.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.028+0000 7f28509ef640 1 -- 172.21.15.107:0/3713898838 wait complete. 2023-11-24T08:13:32.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.270+0000 7f052061c640 1 Processor -- start 2023-11-24T08:13:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.271+0000 7f052061c640 1 -- start start 2023-11-24T08:13:32.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f052061c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05180b2250 0x7f051815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f052061c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f05180b2aa0 0x7f051815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f052061c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051815ab80 con 0x7f05180b2630 2023-11-24T08:13:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f052061c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051815a990 con 0x7f05180b2aa0 2023-11-24T08:13:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f052061c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051815e680 con 0x7f05180b2250 2023-11-24T08:13:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f051ebb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f05180b2aa0 0x7f051815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.272+0000 7f051dbb7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f05180b2630 0x7f05180ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43548/0 (socket says 172.21.15.107:43548) 2023-11-24T08:13:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051dbb7640 1 -- 172.21.15.107:0/4129381312 learned_addr learned my addr 172.21.15.107:0/4129381312 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051e3b8640 1 --2- 172.21.15.107:0/4129381312 >> v2:172.21.15.120:3300/0 conn(0x7f05180b2250 0x7f051815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051ebb9640 1 -- 172.21.15.107:0/4129381312 >> v2:172.21.15.120:3300/0 conn(0x7f05180b2250 msgr2=0x7f051815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051ebb9640 1 --2- 172.21.15.107:0/4129381312 >> v2:172.21.15.120:3300/0 conn(0x7f05180b2250 0x7f051815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051ebb9640 1 -- 172.21.15.107:0/4129381312 >> v1:172.21.15.107:6789/0 conn(0x7f05180b2630 legacy=0x7f05180ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051ebb9640 1 -- 172.21.15.107:0/4129381312 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051815e960 con 0x7f05180b2aa0 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.273+0000 7f051d3b6640 1 -- 172.21.15.107:0/4129381312 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790878278 0 0) 0x7f051815ab80 con 0x7f05180b2630 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.274+0000 7f051ebb9640 1 --2- 172.21.15.107:0/4129381312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2aa0 0x7f051815b080 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0518072a70 tx=0x7f051002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30eb1c26735e8c1b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.275+0000 7f051d3b6640 1 -- 172.21.15.107:0/4129381312 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510031830 con 0x7f05180b2aa0 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.275+0000 7f051d3b6640 1 -- 172.21.15.107:0/4129381312 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510031990 con 0x7f05180b2aa0 2023-11-24T08:13:32.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.275+0000 7f051d3b6640 1 -- 172.21.15.107:0/4129381312 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510031d20 con 0x7f05180b2aa0 2023-11-24T08:13:32.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 -- 172.21.15.107:0/4129381312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2aa0 msgr2=0x7f051815b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 --2- 172.21.15.107:0/4129381312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2aa0 0x7f051815b080 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0518072a70 tx=0x7f051002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 -- 172.21.15.107:0/4129381312 shutdown_connections 2023-11-24T08:13:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 --2- 172.21.15.107:0/4129381312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2aa0 0x7f051815b080 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 --2- 172.21.15.107:0/4129381312 >> v2:172.21.15.120:3300/0 conn(0x7f05180b2250 0x7f051815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 -- 172.21.15.107:0/4129381312 >> 172.21.15.107:0/4129381312 conn(0x7f051806a920 msgr2=0x7f051806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 -- 172.21.15.107:0/4129381312 shutdown_connections 2023-11-24T08:13:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.276+0000 7f052061c640 1 -- 172.21.15.107:0/4129381312 wait complete. 2023-11-24T08:13:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.278+0000 7f052061c640 1 Processor -- start 2023-11-24T08:13:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.278+0000 7f052061c640 1 -- start start 2023-11-24T08:13:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05180b2250 0x7f05180ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05180b2aa0 0x7f051814d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05180c92e0 con 0x7f05180b2630 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05180c86d0 con 0x7f05180b2250 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f052061c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05180c89b0 con 0x7f05180b2aa0 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f051dbb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f051dbb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47484/0 (socket says 172.21.15.107:47484) 2023-11-24T08:13:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.279+0000 7f051dbb7640 1 -- 172.21.15.107:0/924860603 learned_addr learned my addr 172.21.15.107:0/924860603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051e3b8640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05180b2250 0x7f05180ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 -- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05180b2aa0 msgr2=0x7f051814d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05180b2aa0 0x7f051814d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 -- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05180b2250 msgr2=0x7f05180ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05180b2250 0x7f05180ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 -- 172.21.15.107:0/924860603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510009400 con 0x7f05180b2630 2023-11-24T08:13:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f051dbb7640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f050400c4d0 tx=0x7f050400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040074e0 con 0x7f05180b2630 2023-11-24T08:13:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504007660 con 0x7f05180b2630 2023-11-24T08:13:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504004a20 con 0x7f05180b2630 2023-11-24T08:13:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.280+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05180cd7c0 con 0x7f05180b2630 2023-11-24T08:13:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.281+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05180c8f70 con 0x7f05180b2630 2023-11-24T08:13:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.282+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05180c9100 con 0x7f05180b2630 2023-11-24T08:13:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.283+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f0504004b80 con 0x7f05180b2630 2023-11-24T08:13:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.284+0000 7f0516ffd640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 0x7f04fc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.284+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f05040876a0 con 0x7f05180b2630 2023-11-24T08:13:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.284+0000 7f051e3b8640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 0x7f04fc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.285+0000 7f051e3b8640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 0x7f04fc06e350 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f050c009f30 tx=0x7f050c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.288+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05180c9100 con 0x7f05180b2630 2023-11-24T08:13:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.481+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05180c9100 con 0x7f05180b2630 2023-11-24T08:13:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.483+0000 7f0516ffd640 1 -- 172.21.15.107:0/924860603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f05180c9100 con 0x7f05180b2630 2023-11-24T08:13:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:32.487 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":116,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:31.564866+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 msgr2=0x7f04fc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 0x7f04fc06e350 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f050c009f30 tx=0x7f050c00b040 comp rx=0 tx=0).stop 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 msgr2=0x7f0518071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f050400c4d0 tx=0x7f050400c990 comp rx=0 tx=0).stop 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 shutdown_connections 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04fc06be90 0x7f04fc06e350 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05180b2aa0 0x7f051814d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05180b2630 0x7f0518071880 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 --2- 172.21.15.107:0/924860603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05180b2250 0x7f05180ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.486+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 >> 172.21.15.107:0/924860603 conn(0x7f051806a920 msgr2=0x7f05180ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.487+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 shutdown_connections 2023-11-24T08:13:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.487+0000 7f052061c640 1 -- 172.21.15.107:0/924860603 wait complete. 2023-11-24T08:13:32.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:13:32.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:13:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.723+0000 7fa1cb200640 1 Processor -- start 2023-11-24T08:13:32.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.724+0000 7fa1cb200640 1 -- start start 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c415d270 0x7fa1c415d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1cb200640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa1c40cf220 0x7fa1c41616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1cb200640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c415db90 con 0x7fa1c4161260 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1cb200640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c4161ce0 con 0x7fa1c40cf220 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1cb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c4161fc0 con 0x7fa1c415d270 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1c979d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa1c40cf220 0x7fa1c41616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1c3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa1c4161260 0x7fa1c40ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43554/0 (socket says 172.21.15.107:43554) 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.725+0000 7fa1c3fff640 1 -- 172.21.15.107:0/809603844 learned_addr learned my addr 172.21.15.107:0/809603844 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c8f9c640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c415d270 0x7fa1c415d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c979d640 1 -- 172.21.15.107:0/809603844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c415d270 msgr2=0x7fa1c415d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c979d640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c415d270 0x7fa1c415d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c979d640 1 -- 172.21.15.107:0/809603844 >> v1:172.21.15.107:6789/0 conn(0x7fa1c4161260 legacy=0x7fa1c40ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c979d640 1 -- 172.21.15.107:0/809603844 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40ce310 con 0x7fa1c40cf220 2023-11-24T08:13:32.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.726+0000 7fa1c37fe640 1 -- 172.21.15.107:0/809603844 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062868855 0 0) 0x7fa1c415db90 con 0x7fa1c4161260 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.727+0000 7fa1c979d640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41616c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0097a0 tx=0x7fa1bc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd3d266f9e74f0f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.727+0000 7fa1c37fe640 1 -- 172.21.15.107:0/809603844 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc02d040 con 0x7fa1c40cf220 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.727+0000 7fa1c37fe640 1 -- 172.21.15.107:0/809603844 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc031810 con 0x7fa1c40cf220 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.727+0000 7fa1c37fe640 1 -- 172.21.15.107:0/809603844 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc031ba0 con 0x7fa1c40cf220 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 -- 172.21.15.107:0/809603844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 msgr2=0x7fa1c41616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41616c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0097a0 tx=0x7fa1bc02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 -- 172.21.15.107:0/809603844 shutdown_connections 2023-11-24T08:13:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41616c0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 --2- 172.21.15.107:0/809603844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c415d270 0x7fa1c415d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 -- 172.21.15.107:0/809603844 >> 172.21.15.107:0/809603844 conn(0x7fa1c408af60 msgr2=0x7fa1c409c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 -- 172.21.15.107:0/809603844 shutdown_connections 2023-11-24T08:13:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.728+0000 7fa1cb200640 1 -- 172.21.15.107:0/809603844 wait complete. 2023-11-24T08:13:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.730+0000 7fa1cb200640 1 Processor -- start 2023-11-24T08:13:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.730+0000 7fa1cb200640 1 -- start start 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1c415d270 0x7fa1c41f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c4161260 0x7fa1c40c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40b08d0 con 0x7fa1c40cf220 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41f0d80 con 0x7fa1c415d270 2023-11-24T08:13:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1cb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41f42c0 con 0x7fa1c4161260 2023-11-24T08:13:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1c8f9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1c8f9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47502/0 (socket says 172.21.15.107:47502) 2023-11-24T08:13:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.731+0000 7fa1c8f9c640 1 -- 172.21.15.107:0/3999313540 learned_addr learned my addr 172.21.15.107:0/3999313540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:32.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c979d640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c4161260 0x7fa1c40c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 -- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c4161260 msgr2=0x7fa1c40c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c4161260 0x7fa1c40c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c3fff640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1c415d270 0x7fa1c41f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 -- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1c415d270 msgr2=0x7fa1c41f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1c415d270 0x7fa1c41f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 -- 172.21.15.107:0/3999313540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc009400 con 0x7fa1c40cf220 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c8f9c640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4007ef0 tx=0x7fa1b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4014620 con 0x7fa1c40cf220 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b4014780 con 0x7fa1c40cf220 2023-11-24T08:13:32.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4014b30 con 0x7fa1c40cf220 2023-11-24T08:13:32.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.732+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c40c73b0 con 0x7fa1c40cf220 2023-11-24T08:13:32.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.733+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c40c7970 con 0x7fa1c40cf220 2023-11-24T08:13:32.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.734+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c404e8b0 con 0x7fa1c40cf220 2023-11-24T08:13:32.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.734+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fa1b4014c90 con 0x7fa1c40cf220 2023-11-24T08:13:32.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.735+0000 7fa1c17fa640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 0x7fa1a406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:32.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.735+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1b4088eb0 con 0x7fa1c40cf220 2023-11-24T08:13:32.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.736+0000 7fa1c3fff640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 0x7fa1a406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:32.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.736+0000 7fa1c3fff640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 0x7fa1a406e350 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:32.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.740+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1b4055690 con 0x7fa1c40cf220 2023-11-24T08:13:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.928+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1c40ca350 con 0x7fa1c40cf220 2023-11-24T08:13:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.929+0000 7fa1c17fa640 1 -- 172.21.15.107:0/3999313540 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1b4058cb0 con 0x7fa1c40cf220 2023-11-24T08:13:32.931 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 msgr2=0x7fa1a406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 0x7fa1a406e350 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac009040 comp rx=0 tx=0).stop 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 msgr2=0x7fa1c41f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4007ef0 tx=0x7fa1b400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 shutdown_connections 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1a406be90 0x7fa1a406e350 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1c4161260 0x7fa1c40c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1c415d270 0x7fa1c41f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 --2- 172.21.15.107:0/3999313540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c40cf220 0x7fa1c41f0840 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 >> 172.21.15.107:0/3999313540 conn(0x7fa1c408af60 msgr2=0x7fa1c40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 shutdown_connections 2023-11-24T08:13:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:32.932+0000 7fa1cb200640 1 -- 172.21.15.107:0/3999313540 wait complete. 2023-11-24T08:13:32.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:13:33.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.155+0000 7f0e064ad640 1 Processor -- start 2023-11-24T08:13:33.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.155+0000 7f0e064ad640 1 -- start start 2023-11-24T08:13:33.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.156+0000 7f0e064ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e0013d120 0x7f0e0015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.156+0000 7f0e064ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000cec00 con 0x7f0e0006d6c0 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.156+0000 7f0e064ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000b1890 con 0x7f0e0006db20 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.156+0000 7f0e064ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0013f9e0 con 0x7f0e0013d120 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e054ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e0013d120 0x7f0e0015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e04caa640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e0006d6c0 0x7f0e000b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43566/0 (socket says 172.21.15.107:43566) 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e054ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e0013d120 0x7f0e0015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47510/0 (socket says 172.21.15.107:47510) 2023-11-24T08:13:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e04caa640 1 -- 172.21.15.107:0/1692440783 learned_addr learned my addr 172.21.15.107:0/1692440783 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e054ab640 1 -- 172.21.15.107:0/1692440783 >> v1:172.21.15.120:6789/0 conn(0x7f0e0006db20 legacy=0x7f0e000b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e054ab640 1 -- 172.21.15.107:0/1692440783 >> v1:172.21.15.107:6789/0 conn(0x7f0e0006d6c0 legacy=0x7f0e000b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.157+0000 7f0e054ab640 1 -- 172.21.15.107:0/1692440783 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0013fcc0 con 0x7f0e0013d120 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.158+0000 7f0e054ab640 1 --2- 172.21.15.107:0/1692440783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e0015af40 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0097a0 tx=0x7f0dfc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b32dfa0254ed5f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.158+0000 7f0df7fff640 1 -- 172.21.15.107:0/1692440783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc02e070 con 0x7f0e0013d120 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0df7fff640 1 -- 172.21.15.107:0/1692440783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0315a0 con 0x7f0e0013d120 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0df7fff640 1 -- 172.21.15.107:0/1692440783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc031930 con 0x7f0e0013d120 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 -- 172.21.15.107:0/1692440783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 msgr2=0x7f0e0015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1692440783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e0015af40 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0097a0 tx=0x7f0dfc02b840 comp rx=0 tx=0).stop 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 -- 172.21.15.107:0/1692440783 shutdown_connections 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1692440783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e0015af40 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 -- 172.21.15.107:0/1692440783 >> 172.21.15.107:0/1692440783 conn(0x7f0e000cb830 msgr2=0x7f0e000cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 -- 172.21.15.107:0/1692440783 shutdown_connections 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.159+0000 7f0e064ad640 1 -- 172.21.15.107:0/1692440783 wait complete. 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.161+0000 7f0e064ad640 1 Processor -- start 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.161+0000 7f0e064ad640 1 -- start start 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 0x7f0e0013c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e0006db20 0x7f0e000b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015af20 con 0x7f0e0013d120 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015b200 con 0x7f0e0006d6c0 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e064ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015b4e0 con 0x7f0e0006db20 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e05cac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e05cac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47518/0 (socket says 172.21.15.107:47518) 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.162+0000 7f0e05cac640 1 -- 172.21.15.107:0/1338996152 learned_addr learned my addr 172.21.15.107:0/1338996152 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 -- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e0006db20 msgr2=0x7f0e000b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e0006db20 0x7f0e000b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 -- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 msgr2=0x7f0e0013c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e054ab640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 0x7f0e0013c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 0x7f0e0013c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 -- 172.21.15.107:0/1338996152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc009400 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e054ab640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 0x7f0e0013c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e05cac640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c780 tx=0x7f0df800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8004b60 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df800ce90 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0015b7c0 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df800a850 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.163+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0015bd80 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.164+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0015bfa0 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.165+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f0df800a9b0 con 0x7f0e0013d120 2023-11-24T08:13:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.166+0000 7f0df5ffb640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 0x7f0de406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.166+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0df8088650 con 0x7f0e0013d120 2023-11-24T08:13:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.167+0000 7f0e054ab640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 0x7f0de406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.167+0000 7f0e054ab640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 0x7f0de406e350 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc036000 tx=0x7f0dfc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.171+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0df8054e30 con 0x7f0e0013d120 2023-11-24T08:13:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.361+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0e0004e840 con 0x7f0e0013d120 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.362+0000 7f0df5ffb640 1 -- 172.21.15.107:0/1338996152 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v116) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0df8058450 con 0x7f0e0013d120 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:13:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:13:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:33.365 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:13:33.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 msgr2=0x7f0de406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 0x7f0de406e350 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc036000 tx=0x7f0dfc03d040 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 msgr2=0x7f0e00071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c780 tx=0x7f0df800cc40 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 shutdown_connections 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0de406be90 0x7f0de406e350 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e0013d120 0x7f0e00071260 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e0006db20 0x7f0e000b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 --2- 172.21.15.107:0/1338996152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e0006d6c0 0x7f0e0013c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.364+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 >> 172.21.15.107:0/1338996152 conn(0x7f0e000cb830 msgr2=0x7f0e0013df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.365+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 shutdown_connections 2023-11-24T08:13:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.365+0000 7f0e064ad640 1 -- 172.21.15.107:0/1338996152 wait complete. 2023-11-24T08:13:33.386 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:13:33.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:13:33.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.605+0000 7f8b5f59e640 1 Processor -- start 2023-11-24T08:13:33.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.605+0000 7f8b5f59e640 1 -- start start 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8b600b2ad0 0x7f8b6006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b60143c10 con 0x7f8b600b2f60 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5f59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b600705b0 con 0x7f8b601431f0 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5f59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b600b4fd0 con 0x7f8b600b2ad0 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5dd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8b600b2ad0 0x7f8b6006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8b600b2f60 0x7f8b60071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43574/0 (socket says 172.21.15.107:43574) 2023-11-24T08:13:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.607+0000 7f8b5ed9d640 1 -- 172.21.15.107:0/1098069236 learned_addr learned my addr 172.21.15.107:0/1098069236 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.608+0000 7f8b5dd9b640 1 -- 172.21.15.107:0/1098069236 >> v1:172.21.15.120:6789/0 conn(0x7f8b601431f0 legacy=0x7f8b6015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.608+0000 7f8b5dd9b640 1 -- 172.21.15.107:0/1098069236 >> v1:172.21.15.107:6789/0 conn(0x7f8b600b2f60 legacy=0x7f8b60071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.608+0000 7f8b5dd9b640 1 -- 172.21.15.107:0/1098069236 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60070890 con 0x7f8b600b2ad0 2023-11-24T08:13:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.608+0000 7f8b5d59a640 1 -- 172.21.15.107:0/1098069236 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875459127 0 0) 0x7f8b60143c10 con 0x7f8b600b2f60 2023-11-24T08:13:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.609+0000 7f8b5dd9b640 1 --2- 172.21.15.107:0/1098069236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b600b2ad0 0x7f8b6006d6a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009790 tx=0x7f8b4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c103011055ac5c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.609+0000 7f8b5d59a640 1 -- 172.21.15.107:0/1098069236 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4802e070 con 0x7f8b600b2ad0 2023-11-24T08:13:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.609+0000 7f8b5d59a640 1 -- 172.21.15.107:0/1098069236 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480315a0 con 0x7f8b600b2ad0 2023-11-24T08:13:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.609+0000 7f8b5d59a640 1 -- 172.21.15.107:0/1098069236 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48031930 con 0x7f8b600b2ad0 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1098069236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b600b2ad0 msgr2=0x7f8b6006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1098069236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b600b2ad0 0x7f8b6006d6a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009790 tx=0x7f8b4802b840 comp rx=0 tx=0).stop 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1098069236 shutdown_connections 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1098069236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b600b2ad0 0x7f8b6006d6a0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1098069236 >> 172.21.15.107:0/1098069236 conn(0x7f8b600cb8a0 msgr2=0x7f8b600cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1098069236 shutdown_connections 2023-11-24T08:13:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.610+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1098069236 wait complete. 2023-11-24T08:13:33.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.612+0000 7f8b5f59e640 1 Processor -- start 2023-11-24T08:13:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.612+0000 7f8b5f59e640 1 -- start start 2023-11-24T08:13:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b600b2ad0 0x7f8b60159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 0x7f8b600cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600ca970 con 0x7f8b601431f0 2023-11-24T08:13:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600c9fd0 con 0x7f8b600b2f60 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600ca2b0 con 0x7f8b600b2ad0 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47534/0 (socket says 172.21.15.107:47534) 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5ed9d640 1 -- 172.21.15.107:0/1274355487 learned_addr learned my addr 172.21.15.107:0/1274355487 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5dd9b640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 0x7f8b600cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5ed9d640 1 -- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b600b2ad0 msgr2=0x7f8b60159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.613+0000 7f8b5ed9d640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b600b2ad0 0x7f8b60159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5ed9d640 1 -- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 msgr2=0x7f8b600cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5ed9d640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 0x7f8b600cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5ed9d640 1 -- 172.21.15.107:0/1274355487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480093a0 con 0x7f8b601431f0 2023-11-24T08:13:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5dd9b640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 0x7f8b600cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5ed9d640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c7e0 tx=0x7f8b5800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800a640 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5800a7c0 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b600ca590 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800ab70 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.614+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b600c9c80 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.615+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b600caab0 con 0x7f8b601431f0 2023-11-24T08:13:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.617+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f8b5800acd0 con 0x7f8b601431f0 2023-11-24T08:13:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.617+0000 7f8b56ffd640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 0x7f8b4006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.618+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b58087a70 con 0x7f8b601431f0 2023-11-24T08:13:33.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.618+0000 7f8b5e59c640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 0x7f8b4006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.618+0000 7f8b5e59c640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 0x7f8b4006e350 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b50006fd0 tx=0x7f8b50006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.623+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b600caab0 con 0x7f8b601431f0 2023-11-24T08:13:33.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:33.817+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b600caab0 con 0x7f8b601431f0 2023-11-24T08:13:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.566+0000 7f8b56ffd640 1 -- 172.21.15.107:0/1274355487 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v117) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b600caab0 con 0x7f8b601431f0 2023-11-24T08:13:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:13:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 msgr2=0x7f8b4006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 0x7f8b4006e350 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b50006fd0 tx=0x7f8b50006bc0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 msgr2=0x7f8b601e5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c7e0 tx=0x7f8b5800cca0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 shutdown_connections 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8b4006be90 0x7f8b4006e350 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b601431f0 0x7f8b601e5c70 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b600b2f60 0x7f8b600cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 --2- 172.21.15.107:0/1274355487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b600b2ad0 0x7f8b60159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 >> 172.21.15.107:0/1274355487 conn(0x7f8b600cb8a0 msgr2=0x7f8b6006ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 shutdown_connections 2023-11-24T08:13:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.570+0000 7f8b5f59e640 1 -- 172.21.15.107:0/1274355487 wait complete. 2023-11-24T08:13:34.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:13:34.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.795+0000 7f842e5a9640 1 Processor -- start 2023-11-24T08:13:34.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.796+0000 7f842e5a9640 1 -- start start 2023-11-24T08:13:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f842e5a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8428072160 0x7f842815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f842e5a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84280c49f0 0x7f842806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f842e5a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842815ec90 con 0x7f842806dc40 2023-11-24T08:13:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f842e5a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842806e090 con 0x7f84280c49f0 2023-11-24T08:13:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f842e5a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84280cd530 con 0x7f8428072160 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f841ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84280c49f0 0x7f842806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f841ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84280c49f0 0x7f842806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47536/0 (socket says 172.21.15.107:47536) 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.797+0000 7f841ffff640 1 -- 172.21.15.107:0/1633567577 learned_addr learned my addr 172.21.15.107:0/1633567577 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.798+0000 7f8427fff640 1 --2- 172.21.15.107:0/1633567577 >> v2:172.21.15.120:3300/0 conn(0x7f8428072160 0x7f842815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.798+0000 7f841ffff640 1 -- 172.21.15.107:0/1633567577 >> v2:172.21.15.120:3300/0 conn(0x7f8428072160 msgr2=0x7f842815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.798+0000 7f841ffff640 1 --2- 172.21.15.107:0/1633567577 >> v2:172.21.15.120:3300/0 conn(0x7f8428072160 0x7f842815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.798+0000 7f841ffff640 1 -- 172.21.15.107:0/1633567577 >> v1:172.21.15.120:6789/0 conn(0x7f842806dc40 legacy=0x7f84280cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.798+0000 7f841ffff640 1 -- 172.21.15.107:0/1633567577 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280c4ed0 con 0x7f84280c49f0 2023-11-24T08:13:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.799+0000 7f841ffff640 1 --2- 172.21.15.107:0/1633567577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f842806d700 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f84280ccae0 tx=0x7f841002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665a95e2f0e42dda server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.799+0000 7f84277fe640 1 -- 172.21.15.107:0/1633567577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410031870 con 0x7f84280c49f0 2023-11-24T08:13:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.799+0000 7f84277fe640 1 -- 172.21.15.107:0/1633567577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84100319d0 con 0x7f84280c49f0 2023-11-24T08:13:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.799+0000 7f84277fe640 1 -- 172.21.15.107:0/1633567577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410031d60 con 0x7f84280c49f0 2023-11-24T08:13:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 -- 172.21.15.107:0/1633567577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 msgr2=0x7f842806d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 --2- 172.21.15.107:0/1633567577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f842806d700 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f84280ccae0 tx=0x7f841002b910 comp rx=0 tx=0).stop 2023-11-24T08:13:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 -- 172.21.15.107:0/1633567577 shutdown_connections 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 --2- 172.21.15.107:0/1633567577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f842806d700 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 --2- 172.21.15.107:0/1633567577 >> v2:172.21.15.120:3300/0 conn(0x7f8428072160 0x7f842815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 -- 172.21.15.107:0/1633567577 >> 172.21.15.107:0/1633567577 conn(0x7f84280b0c50 msgr2=0x7f84280b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 -- 172.21.15.107:0/1633567577 shutdown_connections 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.800+0000 7f842e5a9640 1 -- 172.21.15.107:0/1633567577 wait complete. 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.802+0000 7f842e5a9640 1 Processor -- start 2023-11-24T08:13:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.802+0000 7f842e5a9640 1 -- start start 2023-11-24T08:13:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f842806dc40 0x7f842815cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8428072160 0x7f8428149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84280c6d00 con 0x7f84280c49f0 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842815a470 con 0x7f8428072160 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842e5a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842814a080 con 0x7f842806dc40 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47546/0 (socket says 172.21.15.107:47546) 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 -- 172.21.15.107:0/2174037108 learned_addr learned my addr 172.21.15.107:0/2174037108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f841ffff640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8428072160 0x7f8428149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 -- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f842806dc40 msgr2=0x7f842815cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f842806dc40 0x7f842815cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 -- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8428072160 msgr2=0x7f8428149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8428072160 0x7f8428149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.803+0000 7f842cb46640 1 -- 172.21.15.107:0/2174037108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84100094d0 con 0x7f84280c49f0 2023-11-24T08:13:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f842cb46640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f842000b9a0 tx=0x7f842000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000cb40 con 0x7f84280c49f0 2023-11-24T08:13:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842000cca0 con 0x7f84280c49f0 2023-11-24T08:13:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8420014680 con 0x7f84280c49f0 2023-11-24T08:13:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842809a930 con 0x7f84280c49f0 2023-11-24T08:13:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.804+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842809aef0 con 0x7f84280c49f0 2023-11-24T08:13:34.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.805+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842804e8b0 con 0x7f84280c49f0 2023-11-24T08:13:34.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.807+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f842001d3b0 con 0x7f84280c49f0 2023-11-24T08:13:34.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.808+0000 7f84257fa640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 0x7f840806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:34.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.808+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f84200881e0 con 0x7f84280c49f0 2023-11-24T08:13:34.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.808+0000 7f8427fff640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 0x7f840806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:34.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.809+0000 7f8427fff640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 0x7f840806e350 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84180098f0 tx=0x7f8418008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:34.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:34.813+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8420054a20 con 0x7f84280c49f0 2023-11-24T08:13:35.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.005+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f842815dab0 con 0x7f84280c49f0 2023-11-24T08:13:35.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.573+0000 7f84257fa640 1 -- 172.21.15.107:0/2174037108 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v118) v1 ==== 170+0+0 (secure 0 0 0) 0x7f84200190d0 con 0x7f84280c49f0 2023-11-24T08:13:35.575 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:13:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 msgr2=0x7f840806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 0x7f840806e350 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84180098f0 tx=0x7f8418008040 comp rx=0 tx=0).stop 2023-11-24T08:13:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 msgr2=0x7f8428149b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f842000b9a0 tx=0x7f842000be60 comp rx=0 tx=0).stop 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 shutdown_connections 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f840806be90 0x7f840806e350 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84280c49f0 0x7f8428149b40 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8428072160 0x7f8428149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 --2- 172.21.15.107:0/2174037108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f842806dc40 0x7f842815cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.577+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 >> 172.21.15.107:0/2174037108 conn(0x7f84280b0c50 msgr2=0x7f842815e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.578+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 shutdown_connections 2023-11-24T08:13:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.578+0000 7f842e5a9640 1 -- 172.21.15.107:0/2174037108 wait complete. 2023-11-24T08:13:35.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:13:35.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.800+0000 7fe00b59e640 1 Processor -- start 2023-11-24T08:13:35.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.801+0000 7fe00b59e640 1 -- start start 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0040ccbd0 0x7fe0041660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccfb0 0x7fe004162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe004162540 0x7fe0040c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0041655f0 con 0x7fe0040ccbd0 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0040cc380 con 0x7fe004162540 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe004162b30 con 0x7fe0040ccfb0 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.802+0000 7fe00933a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0040ccbd0 0x7fe0041660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0040ccbd0 0x7fe0041660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47550/0 (socket says 172.21.15.107:47550) 2023-11-24T08:13:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 -- 172.21.15.107:0/4116110485 learned_addr learned my addr 172.21.15.107:0/4116110485 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:35.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 -- 172.21.15.107:0/4116110485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccfb0 msgr2=0x7fe004162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:35.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 --2- 172.21.15.107:0/4116110485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccfb0 0x7fe004162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 -- 172.21.15.107:0/4116110485 >> v2:172.21.15.120:3300/0 conn(0x7fe004162540 msgr2=0x7fe0040c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:35.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 --2- 172.21.15.107:0/4116110485 >> v2:172.21.15.120:3300/0 conn(0x7fe004162540 0x7fe0040c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.803+0000 7fe00933a640 1 -- 172.21.15.107:0/4116110485 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe004162e10 con 0x7fe0040ccbd0 2023-11-24T08:13:35.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.804+0000 7fe00933a640 1 --2- 172.21.15.107:0/4116110485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccbd0 0x7fe0041660b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe000009870 tx=0x7fe00002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad933f8286502c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:35.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.804+0000 7fdffbfff640 1 -- 172.21.15.107:0/4116110485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000031640 con 0x7fe0040ccbd0 2023-11-24T08:13:35.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.804+0000 7fdffbfff640 1 -- 172.21.15.107:0/4116110485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0000317a0 con 0x7fe0040ccbd0 2023-11-24T08:13:35.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fdffbfff640 1 -- 172.21.15.107:0/4116110485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000031b30 con 0x7fe0040ccbd0 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 -- 172.21.15.107:0/4116110485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccbd0 msgr2=0x7fe0041660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 --2- 172.21.15.107:0/4116110485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccbd0 0x7fe0041660b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe000009870 tx=0x7fe00002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 -- 172.21.15.107:0/4116110485 shutdown_connections 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 --2- 172.21.15.107:0/4116110485 >> v2:172.21.15.120:3300/0 conn(0x7fe004162540 0x7fe0040c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 --2- 172.21.15.107:0/4116110485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccfb0 0x7fe004162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 --2- 172.21.15.107:0/4116110485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccbd0 0x7fe0041660b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 -- 172.21.15.107:0/4116110485 >> 172.21.15.107:0/4116110485 conn(0x7fe0040b0c50 msgr2=0x7fe0040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 -- 172.21.15.107:0/4116110485 shutdown_connections 2023-11-24T08:13:35.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.805+0000 7fe00b59e640 1 -- 172.21.15.107:0/4116110485 wait complete. 2023-11-24T08:13:35.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.807+0000 7fe00b59e640 1 Processor -- start 2023-11-24T08:13:35.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.807+0000 7fe00b59e640 1 -- start start 2023-11-24T08:13:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccbd0 0x7fe0041eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe004162540 0x7fe0040b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040704c0 con 0x7fe0040ccfb0 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040707a0 con 0x7fe004162540 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe00b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040ad2b0 con 0x7fe0040ccbd0 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.808+0000 7fe008b39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47564/0 (socket says 172.21.15.107:47564) 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 -- 172.21.15.107:0/1534540495 learned_addr learned my addr 172.21.15.107:0/1534540495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 -- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccbd0 msgr2=0x7fe0041eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccbd0 0x7fe0041eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 -- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe004162540 msgr2=0x7fe0040b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe009b3b640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe004162540 0x7fe0040b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe004162540 0x7fe0040b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 -- 172.21.15.107:0/1534540495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0000094d0 con 0x7fe0040ccfb0 2023-11-24T08:13:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.809+0000 7fe008b39640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fdff400b4a0 tx=0x7fdff400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.810+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4010610 con 0x7fe0040ccfb0 2023-11-24T08:13:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.810+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff4010770 con 0x7fe0040ccfb0 2023-11-24T08:13:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.810+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4010b20 con 0x7fe0040ccfb0 2023-11-24T08:13:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.810+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0040ad590 con 0x7fe0040ccfb0 2023-11-24T08:13:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.810+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00414bb20 con 0x7fe0040ccfb0 2023-11-24T08:13:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.811+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00404e8b0 con 0x7fe0040ccfb0 2023-11-24T08:13:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.812+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fdff4010c80 con 0x7fe0040ccfb0 2023-11-24T08:13:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.813+0000 7fdff9ffb640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 0x7fdfe806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.813+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fdff4088110 con 0x7fe0040ccfb0 2023-11-24T08:13:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.813+0000 7fe00933a640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 0x7fdfe806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:35.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.814+0000 7fe00933a640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 0x7fdfe806e350 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe000036000 tx=0x7fe000030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:35.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:35.818+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdff40548f0 con 0x7fe0040ccfb0 2023-11-24T08:13:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.042+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe0040b4b20 con 0x7fe0040ccfb0 2023-11-24T08:13:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.599+0000 7fdff9ffb640 1 -- 172.21.15.107:0/1534540495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v65) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdff401e020 con 0x7fe0040ccfb0 2023-11-24T08:13:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 16 and data pool 17 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.603+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 msgr2=0x7fdfe806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.603+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 0x7fdfe806e350 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe000036000 tx=0x7fe000030000 comp rx=0 tx=0).stop 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.603+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 msgr2=0x7fe0041eb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.603+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fdff400b4a0 tx=0x7fdff400b960 comp rx=0 tx=0).stop 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 shutdown_connections 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfe806be90 0x7fdfe806e350 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe004162540 0x7fe0040b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0040ccfb0 0x7fe0041eb990 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 --2- 172.21.15.107:0/1534540495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0040ccbd0 0x7fe0041eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 >> 172.21.15.107:0/1534540495 conn(0x7fe0040b0c50 msgr2=0x7fe00406c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 shutdown_connections 2023-11-24T08:13:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.604+0000 7fe00b59e640 1 -- 172.21.15.107:0/1534540495 wait complete. 2023-11-24T08:13:36.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.826+0000 7f1c4fbc5640 1 Processor -- start 2023-11-24T08:13:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.826+0000 7f1c4fbc5640 1 -- start start 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4fbc5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c4815c7f0 0x7f1c4806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4fbc5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c480ce6c0 0x7f1c480cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4fbc5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480cece0 con 0x7f1c480cf450 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4fbc5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480cfe40 con 0x7f1c480ce6c0 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4fbc5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480ceaf0 con 0x7f1c4815c7f0 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4d160640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c480cf450 0x7f1c480cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43582/0 (socket says 172.21.15.107:43582) 2023-11-24T08:13:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4d160640 1 -- 172.21.15.107:0/4121438895 learned_addr learned my addr 172.21.15.107:0/4121438895 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.827+0000 7f1c4e162640 1 --2- 172.21.15.107:0/4121438895 >> v2:172.21.15.107:3300/0 conn(0x7f1c480ce6c0 0x7f1c480cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.828+0000 7f1c4d961640 1 --2- 172.21.15.107:0/4121438895 >> v2:172.21.15.120:3300/0 conn(0x7f1c4815c7f0 0x7f1c4806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.828+0000 7f1c4e162640 1 -- 172.21.15.107:0/4121438895 >> v2:172.21.15.120:3300/0 conn(0x7f1c4815c7f0 msgr2=0x7f1c4806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.828+0000 7f1c4e162640 1 --2- 172.21.15.107:0/4121438895 >> v2:172.21.15.120:3300/0 conn(0x7f1c4815c7f0 0x7f1c4806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.828+0000 7f1c4e162640 1 -- 172.21.15.107:0/4121438895 >> v1:172.21.15.107:6789/0 conn(0x7f1c480cf450 legacy=0x7f1c480cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.828+0000 7f1c4e162640 1 -- 172.21.15.107:0/4121438895 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4806d040 con 0x7f1c480ce6c0 2023-11-24T08:13:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.829+0000 7f1c4e162640 1 --2- 172.21.15.107:0/4121438895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c480cf900 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f1c480d0130 tx=0x7f1c4002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41437096e01cd88b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.829+0000 7f1c4c95f640 1 -- 172.21.15.107:0/4121438895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002cc70 con 0x7f1c480ce6c0 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.829+0000 7f1c4c95f640 1 -- 172.21.15.107:0/4121438895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4002cdd0 con 0x7f1c480ce6c0 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.829+0000 7f1c4c95f640 1 -- 172.21.15.107:0/4121438895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40034660 con 0x7f1c480ce6c0 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/4121438895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 msgr2=0x7f1c480cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/4121438895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c480cf900 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f1c480d0130 tx=0x7f1c4002bec0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/4121438895 shutdown_connections 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/4121438895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c480cf900 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/4121438895 >> v2:172.21.15.120:3300/0 conn(0x7f1c4815c7f0 0x7f1c4806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/4121438895 >> 172.21.15.107:0/4121438895 conn(0x7f1c4806a920 msgr2=0x7f1c4806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/4121438895 shutdown_connections 2023-11-24T08:13:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.830+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/4121438895 wait complete. 2023-11-24T08:13:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.832+0000 7f1c4fbc5640 1 Processor -- start 2023-11-24T08:13:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.832+0000 7f1c4fbc5640 1 -- start start 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c480cf450 0x7f1c480ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c4815c7f0 0x7f1c4814cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48155de0 con 0x7f1c480ce6c0 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48156680 con 0x7f1c4815c7f0 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4fbc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480ae360 con 0x7f1c480cf450 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47578/0 (socket says 172.21.15.107:47578) 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 -- 172.21.15.107:0/2354399827 learned_addr learned my addr 172.21.15.107:0/2354399827 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4e162640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c4815c7f0 0x7f1c4814cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 -- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c480cf450 msgr2=0x7f1c480ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c480cf450 0x7f1c480ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 -- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c4815c7f0 msgr2=0x7f1c4814cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c4815c7f0 0x7f1c4814cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.833+0000 7f1c4d961640 1 -- 172.21.15.107:0/2354399827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40009a70 con 0x7f1c480ce6c0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c4d961640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c7f0 tx=0x7f1c4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44004220 con 0x7f1c480ce6c0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44004380 con 0x7f1c480ce6c0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44007840 con 0x7f1c480ce6c0 2023-11-24T08:13:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c48156030 con 0x7f1c480ce6c0 2023-11-24T08:13:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.834+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48155510 con 0x7f1c480ce6c0 2023-11-24T08:13:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.835+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48067470 con 0x7f1c480ce6c0 2023-11-24T08:13:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.836+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f1c44007bb0 con 0x7f1c480ce6c0 2023-11-24T08:13:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.837+0000 7f1c3e7fc640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 0x7f1c2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.837+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c44088360 con 0x7f1c480ce6c0 2023-11-24T08:13:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.837+0000 7f1c4d160640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 0x7f1c2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.838+0000 7f1c4d160640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 0x7f1c2c06e350 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1c4806e290 tx=0x7f1c38007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:36.841+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c44054ba0 con 0x7f1c480ce6c0 2023-11-24T08:13:37.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.027+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c480cd730 con 0x7f1c480ce6c0 2023-11-24T08:13:37.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.030+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/2354399827 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16540 (secure 0 0 0) 0x7f1c4401e070 con 0x7f1c480ce6c0 2023-11-24T08:13:37.033 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:37.033 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:36.043848+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:37.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 msgr2=0x7f1c2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 0x7f1c2c06e350 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1c4806e290 tx=0x7f1c38007a40 comp rx=0 tx=0).stop 2023-11-24T08:13:37.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 msgr2=0x7f1c4815af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c7f0 tx=0x7f1c4400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 shutdown_connections 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c2c06be90 0x7f1c2c06e350 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c4815c7f0 0x7f1c4814cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c480cf450 0x7f1c480ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 --2- 172.21.15.107:0/2354399827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c480ce6c0 0x7f1c4815af30 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 >> 172.21.15.107:0/2354399827 conn(0x7f1c4806a920 msgr2=0x7f1c480afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 shutdown_connections 2023-11-24T08:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.033+0000 7f1c4fbc5640 1 -- 172.21.15.107:0/2354399827 wait complete. 2023-11-24T08:13:37.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:37.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.258+0000 7fd53d597640 1 Processor -- start 2023-11-24T08:13:37.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.258+0000 7fd53d597640 1 -- start start 2023-11-24T08:13:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.259+0000 7fd53d597640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd5380cf5d0 0x7fd53806d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.259+0000 7fd53d597640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd53815b320 0x7fd53806b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.259+0000 7fd53d597640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53815e8b0 con 0x7fd53815af40 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.259+0000 7fd53d597640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53815eb90 con 0x7fd53815b320 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.259+0000 7fd53d597640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53815e6a0 con 0x7fd5380cf5d0 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd53815b320 0x7fd53806b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd53815b320 0x7fd53806b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47582/0 (socket says 172.21.15.107:47582) 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 -- 172.21.15.107:0/3623007305 learned_addr learned my addr 172.21.15.107:0/3623007305 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 -- 172.21.15.107:0/3623007305 >> v2:172.21.15.120:3300/0 conn(0x7fd5380cf5d0 msgr2=0x7fd53806d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 --2- 172.21.15.107:0/3623007305 >> v2:172.21.15.120:3300/0 conn(0x7fd5380cf5d0 0x7fd53806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 -- 172.21.15.107:0/3623007305 >> v1:172.21.15.107:6789/0 conn(0x7fd53815af40 legacy=0x7fd5380ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.260+0000 7fd5377fe640 1 -- 172.21.15.107:0/3623007305 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd538070180 con 0x7fd53815b320 2023-11-24T08:13:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.261+0000 7fd5377fe640 1 --2- 172.21.15.107:0/3623007305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd53806b9c0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd53806b3b0 tx=0x7fd53002bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d92da95324f580 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.261+0000 7fd5367fc640 1 -- 172.21.15.107:0/3623007305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53002cb30 con 0x7fd53815b320 2023-11-24T08:13:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.261+0000 7fd5367fc640 1 -- 172.21.15.107:0/3623007305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53002cc90 con 0x7fd53815b320 2023-11-24T08:13:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.261+0000 7fd5367fc640 1 -- 172.21.15.107:0/3623007305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5300343f0 con 0x7fd53815b320 2023-11-24T08:13:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 -- 172.21.15.107:0/3623007305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 msgr2=0x7fd53806b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 --2- 172.21.15.107:0/3623007305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd53806b9c0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd53806b3b0 tx=0x7fd53002bf90 comp rx=0 tx=0).stop 2023-11-24T08:13:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 -- 172.21.15.107:0/3623007305 shutdown_connections 2023-11-24T08:13:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 --2- 172.21.15.107:0/3623007305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd53806b9c0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 --2- 172.21.15.107:0/3623007305 >> v2:172.21.15.120:3300/0 conn(0x7fd5380cf5d0 0x7fd53806d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 -- 172.21.15.107:0/3623007305 >> 172.21.15.107:0/3623007305 conn(0x7fd53806e7f0 msgr2=0x7fd53806ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 -- 172.21.15.107:0/3623007305 shutdown_connections 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.262+0000 7fd53d597640 1 -- 172.21.15.107:0/3623007305 wait complete. 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.264+0000 7fd53d597640 1 Processor -- start 2023-11-24T08:13:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.264+0000 7fd53d597640 1 -- start start 2023-11-24T08:13:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5380cf5d0 0x7fd5380ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd53815af40 0x7fd5380cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380cd360 con 0x7fd53815b320 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380ad7d0 con 0x7fd53815af40 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd53d597640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380b5fd0 con 0x7fd5380cf5d0 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47584/0 (socket says 172.21.15.107:47584) 2023-11-24T08:13:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 -- 172.21.15.107:0/318941645 learned_addr learned my addr 172.21.15.107:0/318941645 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 -- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5380cf5d0 msgr2=0x7fd5380ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5380cf5d0 0x7fd5380ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 -- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd53815af40 msgr2=0x7fd5380cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd53815af40 0x7fd5380cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.265+0000 7fd5377fe640 1 -- 172.21.15.107:0/318941645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530009b30 con 0x7fd53815b320 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd5377fe640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd530036040 tx=0x7fd530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53003e450 con 0x7fd53815b320 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53003e5d0 con 0x7fd53815b320 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53003e780 con 0x7fd53815b320 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5380b62a0 con 0x7fd53815b320 2023-11-24T08:13:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.266+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5380b6860 con 0x7fd53815b320 2023-11-24T08:13:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.267+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd538067470 con 0x7fd53815b320 2023-11-24T08:13:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.268+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fd53003e8e0 con 0x7fd53815b320 2023-11-24T08:13:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.269+0000 7fd52ffff640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 0x7fd51806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.269+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5300aae50 con 0x7fd53815b320 2023-11-24T08:13:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.269+0000 7fd536ffd640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 0x7fd51806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.270+0000 7fd536ffd640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 0x7fd51806e380 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd528000f60 tx=0x7fd5280079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.273+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd530077690 con 0x7fd53815b320 2023-11-24T08:13:37.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.458+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5380b7270 con 0x7fd53815b320 2023-11-24T08:13:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.461+0000 7fd52ffff640 1 -- 172.21.15.107:0/318941645 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16540 (secure 0 0 0) 0x7fd53007acb0 con 0x7fd53815b320 2023-11-24T08:13:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:37.464 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:36.043848+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 msgr2=0x7fd51806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 0x7fd51806e380 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd528000f60 tx=0x7fd5280079b0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 msgr2=0x7fd5380b5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd530036040 tx=0x7fd530004a40 comp rx=0 tx=0).stop 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 shutdown_connections 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd51806bec0 0x7fd51806e380 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd53815b320 0x7fd5380b5a90 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd53815af40 0x7fd5380cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 --2- 172.21.15.107:0/318941645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5380cf5d0 0x7fd5380ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 >> 172.21.15.107:0/318941645 conn(0x7fd53806e7f0 msgr2=0x7fd53806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 shutdown_connections 2023-11-24T08:13:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.464+0000 7fd53d597640 1 -- 172.21.15.107:0/318941645 wait complete. 2023-11-24T08:13:37.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:13:37.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.697+0000 7fb873456640 1 Processor -- start 2023-11-24T08:13:37.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.697+0000 7fb873456640 1 -- start start 2023-11-24T08:13:37.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb873456640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb86c15b440 0x7fb86c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb873456640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb873456640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c15eea0 con 0x7fb86c0cbf20 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb873456640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c0c5e00 con 0x7fb86c15b440 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb873456640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0cc3d0 con 0x7fb86c0c59b0 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb8711f2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb86c15b440 0x7fb86c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.699+0000 7fb8719f3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:41768/0 (socket says 172.21.15.107:41768) 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 -- 172.21.15.107:0/123318709 learned_addr learned my addr 172.21.15.107:0/123318709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 -- 172.21.15.107:0/123318709 >> v2:172.21.15.120:3300/0 conn(0x7fb86c15b440 msgr2=0x7fb86c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 --2- 172.21.15.107:0/123318709 >> v2:172.21.15.120:3300/0 conn(0x7fb86c15b440 0x7fb86c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 -- 172.21.15.107:0/123318709 >> v1:172.21.15.120:6789/0 conn(0x7fb86c0cbf20 legacy=0x7fb86c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.700+0000 7fb8719f3640 1 -- 172.21.15.107:0/123318709 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0abf30 con 0x7fb86c0c59b0 2023-11-24T08:13:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.702+0000 7fb8719f3640 1 --2- 172.21.15.107:0/123318709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb86402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eee514e04802494 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.702+0000 7fb863fff640 1 -- 172.21.15.107:0/123318709 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86402ca70 con 0x7fb86c0c59b0 2023-11-24T08:13:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.702+0000 7fb863fff640 1 -- 172.21.15.107:0/123318709 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86402cbd0 con 0x7fb86c0c59b0 2023-11-24T08:13:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.702+0000 7fb863fff640 1 -- 172.21.15.107:0/123318709 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8640343f0 con 0x7fb86c0c59b0 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 -- 172.21.15.107:0/123318709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 msgr2=0x7fb86c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 --2- 172.21.15.107:0/123318709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb86402bed0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 -- 172.21.15.107:0/123318709 shutdown_connections 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 --2- 172.21.15.107:0/123318709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0c59b0 0x7fb86c06c5f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 --2- 172.21.15.107:0/123318709 >> v2:172.21.15.120:3300/0 conn(0x7fb86c15b440 0x7fb86c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 -- 172.21.15.107:0/123318709 >> 172.21.15.107:0/123318709 conn(0x7fb86c0b0c50 msgr2=0x7fb86c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 -- 172.21.15.107:0/123318709 shutdown_connections 2023-11-24T08:13:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.703+0000 7fb873456640 1 -- 172.21.15.107:0/123318709 wait complete. 2023-11-24T08:13:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.705+0000 7fb873456640 1 Processor -- start 2023-11-24T08:13:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.705+0000 7fb873456640 1 -- start start 2023-11-24T08:13:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0c59b0 0x7fb86c158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 0x7fb86c154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c15a440 con 0x7fb86c15b440 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c15c470 con 0x7fb86c0c59b0 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb873456640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c159010 con 0x7fb86c0cbf20 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb8719f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb8719f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47594/0 (socket says 172.21.15.107:47594) 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb8719f3640 1 -- 172.21.15.107:0/623988578 learned_addr learned my addr 172.21.15.107:0/623988578 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.706+0000 7fb8711f2640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0c59b0 0x7fb86c158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8709f1640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 0x7fb86c154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 -- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 msgr2=0x7fb86c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 0x7fb86c154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 -- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0c59b0 msgr2=0x7fb86c158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0c59b0 0x7fb86c158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 -- 172.21.15.107:0/623988578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864009b30 con 0x7fb86c15b440 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8709f1640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 0x7fb86c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:37.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb8719f3640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb864004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864009510 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86403e450 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86403e800 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.707+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c1592f0 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.708+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c1598b0 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.709+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c15c240 con 0x7fb86c15b440 2023-11-24T08:13:37.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.710+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb86403e960 con 0x7fb86c15b440 2023-11-24T08:13:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.710+0000 7fb861ffb640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 0x7fb85006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:37.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.711+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8640ab140 con 0x7fb86c15b440 2023-11-24T08:13:37.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.711+0000 7fb8711f2640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 0x7fb85006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:37.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.711+0000 7fb8711f2640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 0x7fb85006e350 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb868009a30 tx=0x7fb868008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.715+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb86c15c240 con 0x7fb86c15b440 2023-11-24T08:13:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:37.925+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb86c15c5e0 con 0x7fb86c15b440 2023-11-24T08:13:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.641+0000 7fb861ffb640 1 -- 172.21.15.107:0/623988578 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v68) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb864077980 con 0x7fb86c15b440 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 msgr2=0x7fb85006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 0x7fb85006e350 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb868009a30 tx=0x7fb868008040 comp rx=0 tx=0).stop 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 msgr2=0x7fb86c159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb864004a40 comp rx=0 tx=0).stop 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 shutdown_connections 2023-11-24T08:13:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb85006be90 0x7fb85006e350 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb86c15b440 0x7fb86c159f00 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86c0cbf20 0x7fb86c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 --2- 172.21.15.107:0/623988578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb86c0c59b0 0x7fb86c158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 >> 172.21.15.107:0/623988578 conn(0x7fb86c0b0c50 msgr2=0x7fb86c0c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 shutdown_connections 2023-11-24T08:13:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.645+0000 7fb873456640 1 -- 172.21.15.107:0/623988578 wait complete. 2023-11-24T08:13:38.668 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:13:38.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:13:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.883+0000 7f027561c640 1 Processor -- start 2023-11-24T08:13:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.883+0000 7f027561c640 1 -- start start 2023-11-24T08:13:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f027561c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f02700c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f027561c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02700c65f0 0x7f0270161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f027561c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027015de00 con 0x7f0270161940 2023-11-24T08:13:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f027561c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02700c6a70 con 0x7f02700c65f0 2023-11-24T08:13:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f027561c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027015e0e0 con 0x7f0270140520 2023-11-24T08:13:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f026f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0270161940 0x7f0270142d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41852/0 (socket says 172.21.15.107:41852) 2023-11-24T08:13:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.885+0000 7f026f7fe640 1 -- 172.21.15.107:0/141590516 learned_addr learned my addr 172.21.15.107:0/141590516 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026e7fc640 1 --2- 172.21.15.107:0/141590516 >> v2:172.21.15.107:3300/0 conn(0x7f02700c65f0 0x7f0270161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026effd640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f02700c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026e7fc640 1 -- 172.21.15.107:0/141590516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 msgr2=0x7f02700c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026e7fc640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f02700c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026e7fc640 1 -- 172.21.15.107:0/141590516 >> v1:172.21.15.107:6789/0 conn(0x7f0270161940 legacy=0x7f0270142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026e7fc640 1 -- 172.21.15.107:0/141590516 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027015e3c0 con 0x7f02700c65f0 2023-11-24T08:13:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.886+0000 7f026effd640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f02700c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:13:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.887+0000 7f026e7fc640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02700c65f0 0x7f0270161400 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f02580097a0 tx=0x7f025802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6c0ab6d4b887d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f026dffb640 1 -- 172.21.15.107:0/141590516 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025802d040 con 0x7f02700c65f0 2023-11-24T08:13:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f026dffb640 1 -- 172.21.15.107:0/141590516 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0258031810 con 0x7f02700c65f0 2023-11-24T08:13:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f026dffb640 1 -- 172.21.15.107:0/141590516 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258031ba0 con 0x7f02700c65f0 2023-11-24T08:13:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 -- 172.21.15.107:0/141590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02700c65f0 msgr2=0x7f0270161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02700c65f0 0x7f0270161400 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f02580097a0 tx=0x7f025802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 -- 172.21.15.107:0/141590516 shutdown_connections 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02700c65f0 0x7f0270161400 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 --2- 172.21.15.107:0/141590516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f02700c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 -- 172.21.15.107:0/141590516 >> 172.21.15.107:0/141590516 conn(0x7f02700b2e40 msgr2=0x7f02700b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 -- 172.21.15.107:0/141590516 shutdown_connections 2023-11-24T08:13:38.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.888+0000 7f027561c640 1 -- 172.21.15.107:0/141590516 wait complete. 2023-11-24T08:13:38.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.890+0000 7f027561c640 1 Processor -- start 2023-11-24T08:13:38.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.890+0000 7f027561c640 1 -- start start 2023-11-24T08:13:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 0x7f02700b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f0270157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02701498c0 con 0x7f0270161940 2023-11-24T08:13:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027013de60 con 0x7f02700c65f0 2023-11-24T08:13:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f027561c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027013e140 con 0x7f0270140520 2023-11-24T08:13:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f026f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f026f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33084/0 (socket says 172.21.15.107:33084) 2023-11-24T08:13:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.891+0000 7f026f7fe640 1 -- 172.21.15.107:0/1993528558 learned_addr learned my addr 172.21.15.107:0/1993528558 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026e7fc640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f0270157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 -- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 msgr2=0x7f0270157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026effd640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 0x7f02700b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f0270157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 -- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 msgr2=0x7f02700b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 0x7f02700b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 -- 172.21.15.107:0/1993528558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009400 con 0x7f0270161940 2023-11-24T08:13:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026e7fc640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f0270157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026effd640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 0x7f02700b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f026f7fe640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f026800c420 tx=0x7f026800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.892+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268005ce0 con 0x7f0270161940 2023-11-24T08:13:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.893+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0268005e40 con 0x7f0270161940 2023-11-24T08:13:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.893+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02680072d0 con 0x7f0270161940 2023-11-24T08:13:38.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.893+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027013e420 con 0x7f0270161940 2023-11-24T08:13:38.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.893+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027013e9e0 con 0x7f0270161940 2023-11-24T08:13:38.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.894+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0270161dd0 con 0x7f0270161940 2023-11-24T08:13:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.895+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f02680074f0 con 0x7f0270161940 2023-11-24T08:13:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.896+0000 7f02677fe640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 0x7f025006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.896+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02680878e0 con 0x7f0270161940 2023-11-24T08:13:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.896+0000 7f026effd640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 0x7f025006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.897+0000 7f026effd640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 0x7f025006e350 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f02600094c0 tx=0x7f026000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:38.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:38.901+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02680541d0 con 0x7f0270161940 2023-11-24T08:13:39.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.128+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f027013ebd0 con 0x7f0270161940 2023-11-24T08:13:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.653+0000 7f02677fe640 1 -- 172.21.15.107:0/1993528558 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v69) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0268017020 con 0x7f0270161940 2023-11-24T08:13:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 msgr2=0x7f025006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 0x7f025006e350 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f02600094c0 tx=0x7f026000b040 comp rx=0 tx=0).stop 2023-11-24T08:13:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 msgr2=0x7f0270149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f026800c420 tx=0x7f026800c8e0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 shutdown_connections 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f025006be90 0x7f025006e350 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0270161940 0x7f0270149380 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0270140520 0x7f0270157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 --2- 172.21.15.107:0/1993528558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02700c65f0 0x7f02700b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 >> 172.21.15.107:0/1993528558 conn(0x7f02700b2e40 msgr2=0x7f027015af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 shutdown_connections 2023-11-24T08:13:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.657+0000 7f027561c640 1 -- 172.21.15.107:0/1993528558 wait complete. 2023-11-24T08:13:39.680 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:13:39.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.895+0000 7f133a459640 1 Processor -- start 2023-11-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.895+0000 7f133a459640 1 -- start start 2023-11-24T08:13:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f133a459640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f1334162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f133a459640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1334162eb0 con 0x7f13340703f0 2023-11-24T08:13:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f133a459640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13341654a0 con 0x7f13340cd160 2023-11-24T08:13:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f133a459640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13340ccf00 con 0x7f133414b100 2023-11-24T08:13:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f1338c56640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f13340703f0 0x7f1334165c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41862/0 (socket says 172.21.15.107:41862) 2023-11-24T08:13:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f1338c56640 1 -- 172.21.15.107:0/1618998246 learned_addr learned my addr 172.21.15.107:0/1618998246 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f1339457640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f1334162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.897+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704460421 0 0) 0x7f1324001110 con 0x7f13340703f0 2023-11-24T08:13:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320003520 con 0x7f13340703f0 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527272844 0 0) 0x7f1320003520 con 0x7f13340703f0 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 msgr2=0x7f1334162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f1334162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 >> v1:172.21.15.120:6789/0 conn(0x7f13340cd160 legacy=0x7f13340b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334166320 con 0x7f13340703f0 2023-11-24T08:13:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1324001800 con 0x7f13340703f0 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 >> v1:172.21.15.107:6789/0 conn(0x7f13340703f0 legacy=0x7f1334165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f13200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320003fd0 0x7f13200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1320004d00 0x7f1320005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13340ccf00 con 0x7f1320004d00 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132000bbc0 con 0x7f1320003fd0 2023-11-24T08:13:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132000bd80 con 0x7f133414b100 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.898+0000 7f1339c58640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1320004d00 0x7f1320005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339457640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f13200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339c58640 1 -- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 msgr2=0x7f13200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339c58640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f133414b100 0x7f13200039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339c58640 1 -- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320003fd0 msgr2=0x7f13200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:39.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339c58640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320003fd0 0x7f13200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.899+0000 7f1339c58640 1 -- 172.21.15.107:0/1618998246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1320003520 con 0x7f1320004d00 2023-11-24T08:13:39.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.900+0000 7f1339c58640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1320004d00 0x7f1320005180 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f132c001e50 tx=0x7f13200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e659fdc622e36ba7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:39.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.900+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c003b70 con 0x7f1320004d00 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.900+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c003cd0 con 0x7f1320004d00 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.900+0000 7f132bfff640 1 -- 172.21.15.107:0/1618998246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c005310 con 0x7f1320004d00 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 -- 172.21.15.107:0/1618998246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1320004d00 msgr2=0x7f1320005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 --2- 172.21.15.107:0/1618998246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1320004d00 0x7f1320005180 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f132c001e50 tx=0x7f13200057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f1339c58640 1 -- 172.21.15.107:0/1618998246 reap_dead start 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 -- 172.21.15.107:0/1618998246 shutdown_connections 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 -- 172.21.15.107:0/1618998246 >> 172.21.15.107:0/1618998246 conn(0x7f13340cb830 msgr2=0x7f13340cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 -- 172.21.15.107:0/1618998246 shutdown_connections 2023-11-24T08:13:39.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.901+0000 7f133a459640 1 -- 172.21.15.107:0/1618998246 wait complete. 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.903+0000 7f133a459640 1 Processor -- start 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.903+0000 7f133a459640 1 -- start start 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320004d00 0x7f13340b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1334138d60 0x7f133413b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133413c530 con 0x7f13340aaa10 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334139fd0 con 0x7f1320004d00 2023-11-24T08:13:39.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f133a459640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13341391f0 con 0x7f1334138d60 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33102/0 (socket says 172.21.15.107:33102) 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 -- 172.21.15.107:0/2078643557 learned_addr learned my addr 172.21.15.107:0/2078643557 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1339457640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320004d00 0x7f13340b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 -- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1334138d60 msgr2=0x7f133413b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1334138d60 0x7f133413b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 -- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320004d00 msgr2=0x7f13340b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320004d00 0x7f13340b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.904+0000 7f1338c56640 1 -- 172.21.15.107:0/2078643557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13341396a0 con 0x7f13340aaa10 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f1338c56640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f132400c730 tx=0x7f132400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13240074e0 con 0x7f13340aaa10 2023-11-24T08:13:39.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1324007640 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1334139980 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324004a00 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.905+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13340b4080 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.906+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133413a180 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.907+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f1324004b60 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.908+0000 7f1329ffb640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 0x7f131806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.908+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13240876c0 con 0x7f13340aaa10 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.908+0000 7f1339457640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 0x7f131806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:39.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.909+0000 7f1339457640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 0x7f131806e380 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:39.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:39.912+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f133413a180 con 0x7f13340aaa10 2023-11-24T08:13:40.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.140+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1334162eb0 con 0x7f13340aaa10 2023-11-24T08:13:40.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.141+0000 7f1329ffb640 1 -- 172.21.15.107:0/2078643557 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f1334162eb0 con 0x7f13340aaa10 2023-11-24T08:13:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:40.143 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:13:40.144 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 69 2023-11-24T08:13:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 msgr2=0x7f131806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 0x7f131806e380 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330008040 comp rx=0 tx=0).stop 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 msgr2=0x7f1334138820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f132400c730 tx=0x7f132400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 shutdown_connections 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f131806bec0 0x7f131806e380 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1334138d60 0x7f133413b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13340aaa10 0x7f1334138820 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 --2- 172.21.15.107:0/2078643557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1320004d00 0x7f13340b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 >> 172.21.15.107:0/2078643557 conn(0x7f13340cb830 msgr2=0x7f13340aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.143+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 shutdown_connections 2023-11-24T08:13:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.144+0000 7f133a459640 1 -- 172.21.15.107:0/2078643557 wait complete. 2023-11-24T08:13:40.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:13:40.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.373+0000 7f8d66101640 1 Processor -- start 2023-11-24T08:13:40.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.373+0000 7f8d66101640 1 -- start start 2023-11-24T08:13:40.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d66101640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d600b2250 0x7f8d6015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d66101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600b2630 0x7f8d600ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d66101640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6015ab80 con 0x7f8d600cd470 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d66101640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6015a990 con 0x7f8d600b2250 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d66101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6015e680 con 0x7f8d600b2630 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d5f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d600b2250 0x7f8d6015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d5f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d600b2250 0x7f8d6015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33104/0 (socket says 172.21.15.107:33104) 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d5f7fe640 1 -- 172.21.15.107:0/1601329572 learned_addr learned my addr 172.21.15.107:0/1601329572 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.375+0000 7f8d56ffd640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600b2630 0x7f8d600ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.376+0000 7f8d5f7fe640 1 -- 172.21.15.107:0/1601329572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600b2630 msgr2=0x7f8d600ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.376+0000 7f8d5f7fe640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600b2630 0x7f8d600ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.376+0000 7f8d5f7fe640 1 -- 172.21.15.107:0/1601329572 >> v1:172.21.15.120:6789/0 conn(0x7f8d600cd470 legacy=0x7f8d6015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.376+0000 7f8d5f7fe640 1 -- 172.21.15.107:0/1601329572 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6015e960 con 0x7f8d600b2250 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d5f7fe640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2250 0x7f8d6015df80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009870 tx=0x7f8d5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b339109004576b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d5effd640 1 -- 172.21.15.107:0/1601329572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50031640 con 0x7f8d600b2250 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d5effd640 1 -- 172.21.15.107:0/1601329572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d500317a0 con 0x7f8d600b2250 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d5effd640 1 -- 172.21.15.107:0/1601329572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50031b30 con 0x7f8d600b2250 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d66101640 1 -- 172.21.15.107:0/1601329572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2250 msgr2=0x7f8d6015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.377+0000 7f8d66101640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2250 0x7f8d6015df80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009870 tx=0x7f8d5002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 -- 172.21.15.107:0/1601329572 shutdown_connections 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600b2630 0x7f8d600ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 --2- 172.21.15.107:0/1601329572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2250 0x7f8d6015df80 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 -- 172.21.15.107:0/1601329572 >> 172.21.15.107:0/1601329572 conn(0x7f8d6006a920 msgr2=0x7f8d6006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 -- 172.21.15.107:0/1601329572 shutdown_connections 2023-11-24T08:13:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.378+0000 7f8d66101640 1 -- 172.21.15.107:0/1601329572 wait complete. 2023-11-24T08:13:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.379+0000 7f8d66101640 1 Processor -- start 2023-11-24T08:13:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.380+0000 7f8d66101640 1 -- start start 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.380+0000 7f8d66101640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d600b2250 0x7f8d60157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.380+0000 7f8d66101640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d66101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600cd470 0x7f8d600aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d66101640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600af100 con 0x7f8d600b2630 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d66101640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60156a30 con 0x7f8d600b2250 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d66101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60156d10 con 0x7f8d600cd470 2023-11-24T08:13:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33110/0 (socket says 172.21.15.107:33110) 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 -- 172.21.15.107:0/3007361266 learned_addr learned my addr 172.21.15.107:0/3007361266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 -- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600cd470 msgr2=0x7f8d600aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d5f7fe640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d600b2250 0x7f8d60157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600cd470 0x7f8d600aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 -- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d600b2250 msgr2=0x7f8d60157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d600b2250 0x7f8d60157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.381+0000 7f8d56ffd640 1 -- 172.21.15.107:0/3007361266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d500094d0 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d56ffd640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800d800 tx=0x7f8d4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4800b6c0 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4800b820 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4800bbb0 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d60156ff0 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.382+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601575b0 con 0x7f8d600b2630 2023-11-24T08:13:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.383+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60157770 con 0x7f8d600b2630 2023-11-24T08:13:40.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.384+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f8d4800bd10 con 0x7f8d600b2630 2023-11-24T08:13:40.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.385+0000 7f8d5cff9640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 0x7f8d4006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.385+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d480886d0 con 0x7f8d600b2630 2023-11-24T08:13:40.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.385+0000 7f8d5f7fe640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 0x7f8d4006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.386+0000 7f8d5f7fe640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 0x7f8d4006e350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8d50036000 tx=0x7f8d5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.390+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d48055230 con 0x7f8d600b2630 2023-11-24T08:13:40.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.580+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d600b3bc0 con 0x7f8d600b2630 2023-11-24T08:13:40.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.583+0000 7f8d5cff9640 1 -- 172.21.15.107:0/3007361266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7f8d48058850 con 0x7f8d600b2630 2023-11-24T08:13:40.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:40.586 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 msgr2=0x7f8d4006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 0x7f8d4006e350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8d50036000 tx=0x7f8d5003c040 comp rx=0 tx=0).stop 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 msgr2=0x7f8d601563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800d800 tx=0x7f8d4800dcc0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 shutdown_connections 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d4006be90 0x7f8d4006e350 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d600cd470 0x7f8d600aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600b2630 0x7f8d601563a0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 --2- 172.21.15.107:0/3007361266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d600b2250 0x7f8d60157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 >> 172.21.15.107:0/3007361266 conn(0x7f8d6006a920 msgr2=0x7f8d600cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 shutdown_connections 2023-11-24T08:13:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.586+0000 7f8d66101640 1 -- 172.21.15.107:0/3007361266 wait complete. 2023-11-24T08:13:40.606 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:13:40.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:13:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.817+0000 7f5f86b4c640 1 Processor -- start 2023-11-24T08:13:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.817+0000 7f5f86b4c640 1 -- start start 2023-11-24T08:13:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.818+0000 7f5f86b4c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f8015c7f0 0x7f5f8006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.818+0000 7f5f86b4c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5f800cf450 0x7f5f800cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.818+0000 7f5f86b4c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f800cece0 con 0x7f5f800cdfb0 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.818+0000 7f5f86b4c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800cfe40 con 0x7f5f8015c7f0 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.818+0000 7f5f86b4c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800ceaf0 con 0x7f5f800cf450 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f8015c7f0 0x7f5f8006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f8015c7f0 0x7f5f8006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33118/0 (socket says 172.21.15.107:33118) 2023-11-24T08:13:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 -- 172.21.15.107:0/485210728 learned_addr learned my addr 172.21.15.107:0/485210728 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f7ffff640 1 --2- 172.21.15.107:0/485210728 >> v2:172.21.15.120:3300/0 conn(0x7f5f800cf450 0x7f5f800cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 -- 172.21.15.107:0/485210728 >> v2:172.21.15.120:3300/0 conn(0x7f5f800cf450 msgr2=0x7f5f800cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 --2- 172.21.15.107:0/485210728 >> v2:172.21.15.120:3300/0 conn(0x7f5f800cf450 0x7f5f800cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 -- 172.21.15.107:0/485210728 >> v1:172.21.15.107:6789/0 conn(0x7f5f800cdfb0 legacy=0x7f5f800ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.819+0000 7f5f848e8640 1 -- 172.21.15.107:0/485210728 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8006d040 con 0x7f5f8015c7f0 2023-11-24T08:13:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.820+0000 7f5f848e8640 1 --2- 172.21.15.107:0/485210728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f8006db20 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5f80070410 tx=0x7f5f7402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee758af914276b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f7f7fe640 1 -- 172.21.15.107:0/485210728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74030860 con 0x7f5f8015c7f0 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f7f7fe640 1 -- 172.21.15.107:0/485210728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f740309c0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f7f7fe640 1 -- 172.21.15.107:0/485210728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74030d50 con 0x7f5f8015c7f0 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 -- 172.21.15.107:0/485210728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 msgr2=0x7f5f8006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/485210728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f8006db20 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5f80070410 tx=0x7f5f7402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 -- 172.21.15.107:0/485210728 shutdown_connections 2023-11-24T08:13:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/485210728 >> v2:172.21.15.120:3300/0 conn(0x7f5f800cf450 0x7f5f800cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/485210728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f8006db20 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 -- 172.21.15.107:0/485210728 >> 172.21.15.107:0/485210728 conn(0x7f5f8006a920 msgr2=0x7f5f8006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 -- 172.21.15.107:0/485210728 shutdown_connections 2023-11-24T08:13:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.821+0000 7f5f86b4c640 1 -- 172.21.15.107:0/485210728 wait complete. 2023-11-24T08:13:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.823+0000 7f5f86b4c640 1 Processor -- start 2023-11-24T08:13:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.823+0000 7f5f86b4c640 1 -- start start 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f800cdfb0 0x7f5f801f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 0x7f5f800b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f801f4240 con 0x7f5f8015c7f0 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f801f3c80 con 0x7f5f800cf450 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.824+0000 7f5f86b4c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f801f3f60 con 0x7f5f800cdfb0 2023-11-24T08:13:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33132/0 (socket says 172.21.15.107:33132) 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 -- 172.21.15.107:0/1899346472 learned_addr learned my addr 172.21.15.107:0/1899346472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f848e8640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f800cdfb0 0x7f5f801f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f7ffff640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 0x7f5f800b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 -- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f800cdfb0 msgr2=0x7f5f801f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f800cdfb0 0x7f5f801f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 -- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 msgr2=0x7f5f800b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 0x7f5f800b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 -- 172.21.15.107:0/1899346472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740094d0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f7ffff640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 0x7f5f800b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.825+0000 7f5f850e9640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c730 tx=0x7f5f7800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.826+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78003b70 con 0x7f5f8015c7f0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.826+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f78003cf0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.826+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78003ea0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.826+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f800ac4d0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.826+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f801f37b0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.827+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f800ac9f0 con 0x7f5f8015c7f0 2023-11-24T08:13:40.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.828+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f5f78005520 con 0x7f5f8015c7f0 2023-11-24T08:13:40.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.829+0000 7f5f7d7fa640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 0x7f5f6406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:40.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.829+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f78088460 con 0x7f5f8015c7f0 2023-11-24T08:13:40.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.829+0000 7f5f848e8640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 0x7f5f6406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.830+0000 7f5f848e8640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 0x7f5f6406e3a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5f74036000 tx=0x7f5f7403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:40.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:40.834+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f800ac9f0 con 0x7f5f8015c7f0 2023-11-24T08:13:41.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.023+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f801f40d0 con 0x7f5f6406bee0 2023-11-24T08:13:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.045+0000 7f5f7d7fa640 1 -- 172.21.15.107:0/1899346472 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287602 (secure 0 0 0) 0x7f5f801f40d0 con 0x7f5f6406bee0 2023-11-24T08:13:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:13:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 msgr2=0x7f5f6406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 0x7f5f6406e3a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5f74036000 tx=0x7f5f7403b040 comp rx=0 tx=0).stop 2023-11-24T08:13:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 msgr2=0x7f5f800b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c730 tx=0x7f5f7800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 shutdown_connections 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5f6406bee0 0x7f5f6406e3a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f8015c7f0 0x7f5f800b37a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f800cf450 0x7f5f800b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 --2- 172.21.15.107:0/1899346472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f800cdfb0 0x7f5f801f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.052+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 >> 172.21.15.107:0/1899346472 conn(0x7f5f8006a920 msgr2=0x7f5f800cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.053+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 shutdown_connections 2023-11-24T08:13:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.053+0000 7f5f86b4c640 1 -- 172.21.15.107:0/1899346472 wait complete. 2023-11-24T08:13:41.075 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2023-11-24T08:13:39.597978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283076,"kb_used_data":12824,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754691644,"statfs":{"total":773094113280,"available":772804243456,"internally_reserved":0,"allocated":13131776,"data_stored":6363476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":276614718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.029132"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646601+0000","last_change":"2023-11-24T08:13:36.632799+0000","last_active":"2023-11-24T08:13:36.646601+0000","last_peered":"2023-11-24T08:13:36.646601+0000","last_clean":"2023-11-24T08:13:36.646601+0000","last_became_active":"2023-11-24T08:13:36.629432+0000","last_became_peered":"2023-11-24T08:13:36.629432+0000","last_unstale":"2023-11-24T08:13:36.646601+0000","last_undegraded":"2023-11-24T08:13:36.646601+0000","last_fullsized":"2023-11-24T08:13:36.646601+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:39.870154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636297+0000","last_change":"2023-11-24T08:13:36.640528+0000","last_active":"2023-11-24T08:13:37.636297+0000","last_peered":"2023-11-24T08:13:37.636297+0000","last_clean":"2023-11-24T08:13:37.636297+0000","last_became_active":"2023-11-24T08:13:36.639049+0000","last_became_peered":"2023-11-24T08:13:36.639049+0000","last_unstale":"2023-11-24T08:13:37.636297+0000","last_undegraded":"2023-11-24T08:13:37.636297+0000","last_fullsized":"2023-11-24T08:13:37.636297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175223+0000","last_change":"2023-11-24T08:13:36.633569+0000","last_active":"2023-11-24T08:13:38.175223+0000","last_peered":"2023-11-24T08:13:38.175223+0000","last_clean":"2023-11-24T08:13:38.175223+0000","last_became_active":"2023-11-24T08:13:36.633245+0000","last_became_peered":"2023-11-24T08:13:36.633245+0000","last_unstale":"2023-11-24T08:13:38.175223+0000","last_undegraded":"2023-11-24T08:13:38.175223+0000","last_fullsized":"2023-11-24T08:13:38.175223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:02.590196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646950+0000","last_change":"2023-11-24T08:13:36.637419+0000","last_active":"2023-11-24T08:13:36.646950+0000","last_peered":"2023-11-24T08:13:36.646950+0000","last_clean":"2023-11-24T08:13:36.646950+0000","last_became_active":"2023-11-24T08:13:36.636461+0000","last_became_peered":"2023-11-24T08:13:36.636461+0000","last_unstale":"2023-11-24T08:13:36.646950+0000","last_undegraded":"2023-11-24T08:13:36.646950+0000","last_fullsized":"2023-11-24T08:13:36.646950+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633553+0000","last_change":"2023-11-24T08:13:36.633865+0000","last_active":"2023-11-24T08:13:37.633553+0000","last_peered":"2023-11-24T08:13:37.633553+0000","last_clean":"2023-11-24T08:13:37.633553+0000","last_became_active":"2023-11-24T08:13:36.633073+0000","last_became_peered":"2023-11-24T08:13:36.633073+0000","last_unstale":"2023-11-24T08:13:37.633553+0000","last_undegraded":"2023-11-24T08:13:37.633553+0000","last_fullsized":"2023-11-24T08:13:37.633553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:37.962745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641013+0000","last_change":"2023-11-24T08:13:36.631124+0000","last_active":"2023-11-24T08:13:36.641013+0000","last_peered":"2023-11-24T08:13:36.641013+0000","last_clean":"2023-11-24T08:13:36.641013+0000","last_became_active":"2023-11-24T08:13:36.630792+0000","last_became_peered":"2023-11-24T08:13:36.630792+0000","last_unstale":"2023-11-24T08:13:36.641013+0000","last_undegraded":"2023-11-24T08:13:36.641013+0000","last_fullsized":"2023-11-24T08:13:36.641013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:47:37.309260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633131+0000","last_change":"2023-11-24T08:13:36.619711+0000","last_active":"2023-11-24T08:13:37.633131+0000","last_peered":"2023-11-24T08:13:37.633131+0000","last_clean":"2023-11-24T08:13:37.633131+0000","last_became_active":"2023-11-24T08:13:36.619089+0000","last_became_peered":"2023-11-24T08:13:36.619089+0000","last_unstale":"2023-11-24T08:13:37.633131+0000","last_undegraded":"2023-11-24T08:13:37.633131+0000","last_fullsized":"2023-11-24T08:13:37.633131+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:10:10.166547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174606+0000","last_change":"2023-11-24T08:13:36.628482+0000","last_active":"2023-11-24T08:13:38.174606+0000","last_peered":"2023-11-24T08:13:38.174606+0000","last_clean":"2023-11-24T08:13:38.174606+0000","last_became_active":"2023-11-24T08:13:36.627494+0000","last_became_peered":"2023-11-24T08:13:36.627494+0000","last_unstale":"2023-11-24T08:13:38.174606+0000","last_undegraded":"2023-11-24T08:13:38.174606+0000","last_fullsized":"2023-11-24T08:13:38.174606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643908+0000","last_change":"2023-11-24T08:13:36.629705+0000","last_active":"2023-11-24T08:13:36.643908+0000","last_peered":"2023-11-24T08:13:36.643908+0000","last_clean":"2023-11-24T08:13:36.643908+0000","last_became_active":"2023-11-24T08:13:36.629095+0000","last_became_peered":"2023-11-24T08:13:36.629095+0000","last_unstale":"2023-11-24T08:13:36.643908+0000","last_undegraded":"2023-11-24T08:13:36.643908+0000","last_fullsized":"2023-11-24T08:13:36.643908+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:23.314730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648045+0000","last_change":"2023-11-24T08:13:36.641428+0000","last_active":"2023-11-24T08:13:36.648045+0000","last_peered":"2023-11-24T08:13:36.648045+0000","last_clean":"2023-11-24T08:13:36.648045+0000","last_became_active":"2023-11-24T08:13:36.641123+0000","last_became_peered":"2023-11-24T08:13:36.641123+0000","last_unstale":"2023-11-24T08:13:36.648045+0000","last_undegraded":"2023-11-24T08:13:36.648045+0000","last_fullsized":"2023-11-24T08:13:36.648045+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:02.946543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636056+0000","last_change":"2023-11-24T08:13:36.632254+0000","last_active":"2023-11-24T08:13:37.636056+0000","last_peered":"2023-11-24T08:13:37.636056+0000","last_clean":"2023-11-24T08:13:37.636056+0000","last_became_active":"2023-11-24T08:13:36.631973+0000","last_became_peered":"2023-11-24T08:13:36.631973+0000","last_unstale":"2023-11-24T08:13:37.636056+0000","last_undegraded":"2023-11-24T08:13:37.636056+0000","last_fullsized":"2023-11-24T08:13:37.636056+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:33:16.942552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647787+0000","last_change":"2023-11-24T08:13:36.640033+0000","last_active":"2023-11-24T08:13:36.647787+0000","last_peered":"2023-11-24T08:13:36.647787+0000","last_clean":"2023-11-24T08:13:36.647787+0000","last_became_active":"2023-11-24T08:13:36.639786+0000","last_became_peered":"2023-11-24T08:13:36.639786+0000","last_unstale":"2023-11-24T08:13:36.647787+0000","last_undegraded":"2023-11-24T08:13:36.647787+0000","last_fullsized":"2023-11-24T08:13:36.647787+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175291+0000","last_change":"2023-11-24T08:13:36.634533+0000","last_active":"2023-11-24T08:13:38.175291+0000","last_peered":"2023-11-24T08:13:38.175291+0000","last_clean":"2023-11-24T08:13:38.175291+0000","last_became_active":"2023-11-24T08:13:36.634206+0000","last_became_peered":"2023-11-24T08:13:36.634206+0000","last_unstale":"2023-11-24T08:13:38.175291+0000","last_undegraded":"2023-11-24T08:13:38.175291+0000","last_fullsized":"2023-11-24T08:13:38.175291+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:26.054148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174485+0000","last_change":"2023-11-24T08:13:36.619182+0000","last_active":"2023-11-24T08:13:38.174485+0000","last_peered":"2023-11-24T08:13:38.174485+0000","last_clean":"2023-11-24T08:13:38.174485+0000","last_became_active":"2023-11-24T08:13:36.618900+0000","last_became_peered":"2023-11-24T08:13:36.618900+0000","last_unstale":"2023-11-24T08:13:38.174485+0000","last_undegraded":"2023-11-24T08:13:38.174485+0000","last_fullsized":"2023-11-24T08:13:38.174485+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641612+0000","last_change":"2023-11-24T08:13:36.631407+0000","last_active":"2023-11-24T08:13:36.641612+0000","last_peered":"2023-11-24T08:13:36.641612+0000","last_clean":"2023-11-24T08:13:36.641612+0000","last_became_active":"2023-11-24T08:13:36.631120+0000","last_became_peered":"2023-11-24T08:13:36.631120+0000","last_unstale":"2023-11-24T08:13:36.641612+0000","last_undegraded":"2023-11-24T08:13:36.641612+0000","last_fullsized":"2023-11-24T08:13:36.641612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:19:40.170869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644077+0000","last_change":"2023-11-24T08:13:36.631511+0000","last_active":"2023-11-24T08:13:36.644077+0000","last_peered":"2023-11-24T08:13:36.644077+0000","last_clean":"2023-11-24T08:13:36.644077+0000","last_became_active":"2023-11-24T08:13:36.631001+0000","last_became_peered":"2023-11-24T08:13:36.631001+0000","last_unstale":"2023-11-24T08:13:36.644077+0000","last_undegraded":"2023-11-24T08:13:36.644077+0000","last_fullsized":"2023-11-24T08:13:36.644077+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636545+0000","last_change":"2023-11-24T08:13:36.635293+0000","last_active":"2023-11-24T08:13:37.636545+0000","last_peered":"2023-11-24T08:13:37.636545+0000","last_clean":"2023-11-24T08:13:37.636545+0000","last_became_active":"2023-11-24T08:13:36.634297+0000","last_became_peered":"2023-11-24T08:13:36.634297+0000","last_unstale":"2023-11-24T08:13:37.636545+0000","last_undegraded":"2023-11-24T08:13:37.636545+0000","last_fullsized":"2023-11-24T08:13:37.636545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:53.189958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636762+0000","last_change":"2023-11-24T08:13:36.633291+0000","last_active":"2023-11-24T08:13:36.636762+0000","last_peered":"2023-11-24T08:13:36.636762+0000","last_clean":"2023-11-24T08:13:36.636762+0000","last_became_active":"2023-11-24T08:13:36.632857+0000","last_became_peered":"2023-11-24T08:13:36.632857+0000","last_unstale":"2023-11-24T08:13:36.636762+0000","last_undegraded":"2023-11-24T08:13:36.636762+0000","last_fullsized":"2023-11-24T08:13:36.636762+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:05.002635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175314+0000","last_change":"2023-11-24T08:13:36.640480+0000","last_active":"2023-11-24T08:13:38.175314+0000","last_peered":"2023-11-24T08:13:38.175314+0000","last_clean":"2023-11-24T08:13:38.175314+0000","last_became_active":"2023-11-24T08:13:36.639810+0000","last_became_peered":"2023-11-24T08:13:36.639810+0000","last_unstale":"2023-11-24T08:13:38.175314+0000","last_undegraded":"2023-11-24T08:13:38.175314+0000","last_fullsized":"2023-11-24T08:13:38.175314+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:56.997882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636011+0000","last_change":"2023-11-24T08:13:36.638500+0000","last_active":"2023-11-24T08:13:37.636011+0000","last_peered":"2023-11-24T08:13:37.636011+0000","last_clean":"2023-11-24T08:13:37.636011+0000","last_became_active":"2023-11-24T08:13:36.637218+0000","last_became_peered":"2023-11-24T08:13:36.637218+0000","last_unstale":"2023-11-24T08:13:37.636011+0000","last_undegraded":"2023-11-24T08:13:37.636011+0000","last_fullsized":"2023-11-24T08:13:37.636011+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:03:06.812527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635846+0000","last_change":"2023-11-24T08:13:36.630605+0000","last_active":"2023-11-24T08:13:37.635846+0000","last_peered":"2023-11-24T08:13:37.635846+0000","last_clean":"2023-11-24T08:13:37.635846+0000","last_became_active":"2023-11-24T08:13:36.630060+0000","last_became_peered":"2023-11-24T08:13:36.630060+0000","last_unstale":"2023-11-24T08:13:37.635846+0000","last_undegraded":"2023-11-24T08:13:37.635846+0000","last_fullsized":"2023-11-24T08:13:37.635846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:23.032120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646383+0000","last_change":"2023-11-24T08:13:36.634040+0000","last_active":"2023-11-24T08:13:36.646383+0000","last_peered":"2023-11-24T08:13:36.646383+0000","last_clean":"2023-11-24T08:13:36.646383+0000","last_became_active":"2023-11-24T08:13:36.630133+0000","last_became_peered":"2023-11-24T08:13:36.630133+0000","last_unstale":"2023-11-24T08:13:36.646383+0000","last_undegraded":"2023-11-24T08:13:36.646383+0000","last_fullsized":"2023-11-24T08:13:36.646383+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634603+0000","last_change":"2023-11-24T08:13:36.631917+0000","last_active":"2023-11-24T08:13:37.634603+0000","last_peered":"2023-11-24T08:13:37.634603+0000","last_clean":"2023-11-24T08:13:37.634603+0000","last_became_active":"2023-11-24T08:13:36.630938+0000","last_became_peered":"2023-11-24T08:13:36.630938+0000","last_unstale":"2023-11-24T08:13:37.634603+0000","last_undegraded":"2023-11-24T08:13:37.634603+0000","last_fullsized":"2023-11-24T08:13:37.634603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:45:04.376236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646214+0000","last_change":"2023-11-24T08:13:36.632648+0000","last_active":"2023-11-24T08:13:36.646214+0000","last_peered":"2023-11-24T08:13:36.646214+0000","last_clean":"2023-11-24T08:13:36.646214+0000","last_became_active":"2023-11-24T08:13:36.632360+0000","last_became_peered":"2023-11-24T08:13:36.632360+0000","last_unstale":"2023-11-24T08:13:36.646214+0000","last_undegraded":"2023-11-24T08:13:36.646214+0000","last_fullsized":"2023-11-24T08:13:36.646214+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:53.367338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636421+0000","last_change":"2023-11-24T08:13:36.636739+0000","last_active":"2023-11-24T08:13:37.636421+0000","last_peered":"2023-11-24T08:13:37.636421+0000","last_clean":"2023-11-24T08:13:37.636421+0000","last_became_active":"2023-11-24T08:13:36.636430+0000","last_became_peered":"2023-11-24T08:13:36.636430+0000","last_unstale":"2023-11-24T08:13:37.636421+0000","last_undegraded":"2023-11-24T08:13:37.636421+0000","last_fullsized":"2023-11-24T08:13:37.636421+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:12.778241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636731+0000","last_change":"2023-11-24T08:13:36.628886+0000","last_active":"2023-11-24T08:13:36.636731+0000","last_peered":"2023-11-24T08:13:36.636731+0000","last_clean":"2023-11-24T08:13:36.636731+0000","last_became_active":"2023-11-24T08:13:36.628628+0000","last_became_peered":"2023-11-24T08:13:36.628628+0000","last_unstale":"2023-11-24T08:13:36.636731+0000","last_undegraded":"2023-11-24T08:13:36.636731+0000","last_fullsized":"2023-11-24T08:13:36.636731+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:32.362821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648737+0000","last_change":"2023-11-24T08:13:36.644383+0000","last_active":"2023-11-24T08:13:36.648737+0000","last_peered":"2023-11-24T08:13:36.648737+0000","last_clean":"2023-11-24T08:13:36.648737+0000","last_became_active":"2023-11-24T08:13:36.643474+0000","last_became_peered":"2023-11-24T08:13:36.643474+0000","last_unstale":"2023-11-24T08:13:36.648737+0000","last_undegraded":"2023-11-24T08:13:36.648737+0000","last_fullsized":"2023-11-24T08:13:36.648737+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:30.041085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646772+0000","last_change":"2023-11-24T08:13:36.639626+0000","last_active":"2023-11-24T08:13:36.646772+0000","last_peered":"2023-11-24T08:13:36.646772+0000","last_clean":"2023-11-24T08:13:36.646772+0000","last_became_active":"2023-11-24T08:13:36.638866+0000","last_became_peered":"2023-11-24T08:13:36.638866+0000","last_unstale":"2023-11-24T08:13:36.646772+0000","last_undegraded":"2023-11-24T08:13:36.646772+0000","last_fullsized":"2023-11-24T08:13:36.646772+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:23.142549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635192+0000","last_change":"2023-11-24T08:13:36.638103+0000","last_active":"2023-11-24T08:13:37.635192+0000","last_peered":"2023-11-24T08:13:37.635192+0000","last_clean":"2023-11-24T08:13:37.635192+0000","last_became_active":"2023-11-24T08:13:36.637795+0000","last_became_peered":"2023-11-24T08:13:36.637795+0000","last_unstale":"2023-11-24T08:13:37.635192+0000","last_undegraded":"2023-11-24T08:13:37.635192+0000","last_fullsized":"2023-11-24T08:13:37.635192+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:01.128359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633299+0000","last_change":"2023-11-24T08:13:36.625982+0000","last_active":"2023-11-24T08:13:37.633299+0000","last_peered":"2023-11-24T08:13:37.633299+0000","last_clean":"2023-11-24T08:13:37.633299+0000","last_became_active":"2023-11-24T08:13:36.625730+0000","last_became_peered":"2023-11-24T08:13:36.625730+0000","last_unstale":"2023-11-24T08:13:37.633299+0000","last_undegraded":"2023-11-24T08:13:37.633299+0000","last_fullsized":"2023-11-24T08:13:37.633299+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:11.651604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644001+0000","last_change":"2023-11-24T08:13:36.629834+0000","last_active":"2023-11-24T08:13:36.644001+0000","last_peered":"2023-11-24T08:13:36.644001+0000","last_clean":"2023-11-24T08:13:36.644001+0000","last_became_active":"2023-11-24T08:13:36.628483+0000","last_became_peered":"2023-11-24T08:13:36.628483+0000","last_unstale":"2023-11-24T08:13:36.644001+0000","last_undegraded":"2023-11-24T08:13:36.644001+0000","last_fullsized":"2023-11-24T08:13:36.644001+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:47.568775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641359+0000","last_change":"2023-11-24T08:13:36.632468+0000","last_active":"2023-11-24T08:13:36.641359+0000","last_peered":"2023-11-24T08:13:36.641359+0000","last_clean":"2023-11-24T08:13:36.641359+0000","last_became_active":"2023-11-24T08:13:36.632155+0000","last_became_peered":"2023-11-24T08:13:36.632155+0000","last_unstale":"2023-11-24T08:13:36.641359+0000","last_undegraded":"2023-11-24T08:13:36.641359+0000","last_fullsized":"2023-11-24T08:13:36.641359+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:28.735860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643027+0000","last_change":"2023-11-24T08:13:36.640155+0000","last_active":"2023-11-24T08:13:36.643027+0000","last_peered":"2023-11-24T08:13:36.643027+0000","last_clean":"2023-11-24T08:13:36.643027+0000","last_became_active":"2023-11-24T08:13:36.639762+0000","last_became_peered":"2023-11-24T08:13:36.639762+0000","last_unstale":"2023-11-24T08:13:36.643027+0000","last_undegraded":"2023-11-24T08:13:36.643027+0000","last_fullsized":"2023-11-24T08:13:36.643027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:08.754462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"119'4","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634308+0000","last_change":"2023-11-24T08:13:35.609359+0000","last_active":"2023-11-24T08:13:37.634308+0000","last_peered":"2023-11-24T08:13:37.634308+0000","last_clean":"2023-11-24T08:13:37.634308+0000","last_became_active":"2023-11-24T08:13:35.609064+0000","last_became_peered":"2023-11-24T08:13:35.609064+0000","last_unstale":"2023-11-24T08:13:37.634308+0000","last_undegraded":"2023-11-24T08:13:37.634308+0000","last_fullsized":"2023-11-24T08:13:37.634308+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:01.055173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.625261+0000","last_change":"2023-11-24T08:13:36.623428+0000","last_active":"2023-11-24T08:13:36.625261+0000","last_peered":"2023-11-24T08:13:36.625261+0000","last_clean":"2023-11-24T08:13:36.625261+0000","last_became_active":"2023-11-24T08:13:36.618036+0000","last_became_peered":"2023-11-24T08:13:36.618036+0000","last_unstale":"2023-11-24T08:13:36.625261+0000","last_undegraded":"2023-11-24T08:13:36.625261+0000","last_fullsized":"2023-11-24T08:13:36.625261+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:04.557057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"119'1","reported_seq":13,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175052+0000","last_change":"2023-11-24T08:13:35.608591+0000","last_active":"2023-11-24T08:13:38.175052+0000","last_peered":"2023-11-24T08:13:38.175052+0000","last_clean":"2023-11-24T08:13:38.175052+0000","last_became_active":"2023-11-24T08:13:35.608178+0000","last_became_peered":"2023-11-24T08:13:35.608178+0000","last_unstale":"2023-11-24T08:13:38.175052+0000","last_undegraded":"2023-11-24T08:13:38.175052+0000","last_fullsized":"2023-11-24T08:13:38.175052+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:52.353539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.639427+0000","last_change":"2023-11-24T08:13:36.631000+0000","last_active":"2023-11-24T08:13:36.639427+0000","last_peered":"2023-11-24T08:13:36.639427+0000","last_clean":"2023-11-24T08:13:36.639427+0000","last_became_active":"2023-11-24T08:13:36.630659+0000","last_became_peered":"2023-11-24T08:13:36.630659+0000","last_unstale":"2023-11-24T08:13:36.639427+0000","last_undegraded":"2023-11-24T08:13:36.639427+0000","last_fullsized":"2023-11-24T08:13:36.639427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:27.101355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633785+0000","last_change":"2023-11-24T08:13:35.611575+0000","last_active":"2023-11-24T08:13:37.633785+0000","last_peered":"2023-11-24T08:13:37.633785+0000","last_clean":"2023-11-24T08:13:37.633785+0000","last_became_active":"2023-11-24T08:13:35.611306+0000","last_became_peered":"2023-11-24T08:13:35.611306+0000","last_unstale":"2023-11-24T08:13:37.633785+0000","last_undegraded":"2023-11-24T08:13:37.633785+0000","last_fullsized":"2023-11-24T08:13:37.633785+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:46.151145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175637+0000","last_change":"2023-11-24T08:13:36.640645+0000","last_active":"2023-11-24T08:13:38.175637+0000","last_peered":"2023-11-24T08:13:38.175637+0000","last_clean":"2023-11-24T08:13:38.175637+0000","last_became_active":"2023-11-24T08:13:36.640215+0000","last_became_peered":"2023-11-24T08:13:36.640215+0000","last_unstale":"2023-11-24T08:13:38.175637+0000","last_undegraded":"2023-11-24T08:13:38.175637+0000","last_fullsized":"2023-11-24T08:13:38.175637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:55.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633349+0000","last_change":"2023-11-24T08:13:35.612082+0000","last_active":"2023-11-24T08:13:37.633349+0000","last_peered":"2023-11-24T08:13:37.633349+0000","last_clean":"2023-11-24T08:13:37.633349+0000","last_became_active":"2023-11-24T08:13:35.611841+0000","last_became_peered":"2023-11-24T08:13:35.611841+0000","last_unstale":"2023-11-24T08:13:37.633349+0000","last_undegraded":"2023-11-24T08:13:37.633349+0000","last_fullsized":"2023-11-24T08:13:37.633349+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:04:29.687925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646772+0000","last_change":"2023-11-24T08:13:36.635274+0000","last_active":"2023-11-24T08:13:36.646772+0000","last_peered":"2023-11-24T08:13:36.646772+0000","last_clean":"2023-11-24T08:13:36.646772+0000","last_became_active":"2023-11-24T08:13:36.634918+0000","last_became_peered":"2023-11-24T08:13:36.634918+0000","last_unstale":"2023-11-24T08:13:36.646772+0000","last_undegraded":"2023-11-24T08:13:36.646772+0000","last_fullsized":"2023-11-24T08:13:36.646772+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:53.367338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"119'1","reported_seq":13,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633460+0000","last_change":"2023-11-24T08:13:35.611604+0000","last_active":"2023-11-24T08:13:37.633460+0000","last_peered":"2023-11-24T08:13:37.633460+0000","last_clean":"2023-11-24T08:13:37.633460+0000","last_became_active":"2023-11-24T08:13:35.611344+0000","last_became_peered":"2023-11-24T08:13:35.611344+0000","last_unstale":"2023-11-24T08:13:37.633460+0000","last_undegraded":"2023-11-24T08:13:37.633460+0000","last_fullsized":"2023-11-24T08:13:37.633460+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:47:42.061209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635399+0000","last_change":"2023-11-24T08:13:36.635465+0000","last_active":"2023-11-24T08:13:37.635399+0000","last_peered":"2023-11-24T08:13:37.635399+0000","last_clean":"2023-11-24T08:13:37.635399+0000","last_became_active":"2023-11-24T08:13:36.635145+0000","last_became_peered":"2023-11-24T08:13:36.635145+0000","last_unstale":"2023-11-24T08:13:37.635399+0000","last_undegraded":"2023-11-24T08:13:37.635399+0000","last_fullsized":"2023-11-24T08:13:37.635399+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:46:14.783732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640268+0000","last_change":"2023-11-24T08:13:35.607784+0000","last_active":"2023-11-24T08:13:36.640268+0000","last_peered":"2023-11-24T08:13:36.640268+0000","last_clean":"2023-11-24T08:13:36.640268+0000","last_became_active":"2023-11-24T08:13:35.606631+0000","last_became_peered":"2023-11-24T08:13:35.606631+0000","last_unstale":"2023-11-24T08:13:36.640268+0000","last_undegraded":"2023-11-24T08:13:36.640268+0000","last_fullsized":"2023-11-24T08:13:36.640268+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:01.576398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644095+0000","last_change":"2023-11-24T08:13:36.631702+0000","last_active":"2023-11-24T08:13:36.644095+0000","last_peered":"2023-11-24T08:13:36.644095+0000","last_clean":"2023-11-24T08:13:36.644095+0000","last_became_active":"2023-11-24T08:13:36.631279+0000","last_became_peered":"2023-11-24T08:13:36.631279+0000","last_unstale":"2023-11-24T08:13:36.644095+0000","last_undegraded":"2023-11-24T08:13:36.644095+0000","last_fullsized":"2023-11-24T08:13:36.644095+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:12.778241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634638+0000","last_change":"2023-11-24T08:13:35.603666+0000","last_active":"2023-11-24T08:13:37.634638+0000","last_peered":"2023-11-24T08:13:37.634638+0000","last_clean":"2023-11-24T08:13:37.634638+0000","last_became_active":"2023-11-24T08:13:35.603331+0000","last_became_peered":"2023-11-24T08:13:35.603331+0000","last_unstale":"2023-11-24T08:13:37.634638+0000","last_undegraded":"2023-11-24T08:13:37.634638+0000","last_fullsized":"2023-11-24T08:13:37.634638+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:03.543258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633191+0000","last_change":"2023-11-24T08:13:36.621729+0000","last_active":"2023-11-24T08:13:37.633191+0000","last_peered":"2023-11-24T08:13:37.633191+0000","last_clean":"2023-11-24T08:13:37.633191+0000","last_became_active":"2023-11-24T08:13:36.621335+0000","last_became_peered":"2023-11-24T08:13:36.621335+0000","last_unstale":"2023-11-24T08:13:37.633191+0000","last_undegraded":"2023-11-24T08:13:37.633191+0000","last_fullsized":"2023-11-24T08:13:37.633191+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:09:49.865067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.615951+0000","last_change":"2023-11-24T08:13:35.617740+0000","last_active":"2023-11-24T08:13:36.615951+0000","last_peered":"2023-11-24T08:13:36.615951+0000","last_clean":"2023-11-24T08:13:36.615951+0000","last_became_active":"2023-11-24T08:13:35.617472+0000","last_became_peered":"2023-11-24T08:13:35.617472+0000","last_unstale":"2023-11-24T08:13:36.615951+0000","last_undegraded":"2023-11-24T08:13:36.615951+0000","last_fullsized":"2023-11-24T08:13:36.615951+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:01.576398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634097+0000","last_change":"2023-11-24T08:13:36.639946+0000","last_active":"2023-11-24T08:13:37.634097+0000","last_peered":"2023-11-24T08:13:37.634097+0000","last_clean":"2023-11-24T08:13:37.634097+0000","last_became_active":"2023-11-24T08:13:36.639666+0000","last_became_peered":"2023-11-24T08:13:36.639666+0000","last_unstale":"2023-11-24T08:13:37.634097+0000","last_undegraded":"2023-11-24T08:13:37.634097+0000","last_fullsized":"2023-11-24T08:13:37.634097+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:18.995500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633594+0000","last_change":"2023-11-24T08:13:35.612067+0000","last_active":"2023-11-24T08:13:37.633594+0000","last_peered":"2023-11-24T08:13:37.633594+0000","last_clean":"2023-11-24T08:13:37.633594+0000","last_became_active":"2023-11-24T08:13:35.611792+0000","last_became_peered":"2023-11-24T08:13:35.611792+0000","last_unstale":"2023-11-24T08:13:37.633594+0000","last_undegraded":"2023-11-24T08:13:37.633594+0000","last_fullsized":"2023-11-24T08:13:37.633594+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:29.918709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640802+0000","last_change":"2023-11-24T08:13:36.639152+0000","last_active":"2023-11-24T08:13:36.640802+0000","last_peered":"2023-11-24T08:13:36.640802+0000","last_clean":"2023-11-24T08:13:36.640802+0000","last_became_active":"2023-11-24T08:13:36.638881+0000","last_became_peered":"2023-11-24T08:13:36.638881+0000","last_unstale":"2023-11-24T08:13:36.640802+0000","last_undegraded":"2023-11-24T08:13:36.640802+0000","last_fullsized":"2023-11-24T08:13:36.640802+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:47.135957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.619968+0000","last_change":"2023-11-24T08:13:35.607253+0000","last_active":"2023-11-24T08:13:36.619968+0000","last_peered":"2023-11-24T08:13:36.619968+0000","last_clean":"2023-11-24T08:13:36.619968+0000","last_became_active":"2023-11-24T08:13:35.606727+0000","last_became_peered":"2023-11-24T08:13:35.606727+0000","last_unstale":"2023-11-24T08:13:36.619968+0000","last_undegraded":"2023-11-24T08:13:36.619968+0000","last_fullsized":"2023-11-24T08:13:36.619968+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:55.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646359+0000","last_change":"2023-11-24T08:13:36.634674+0000","last_active":"2023-11-24T08:13:36.646359+0000","last_peered":"2023-11-24T08:13:36.646359+0000","last_clean":"2023-11-24T08:13:36.646359+0000","last_became_active":"2023-11-24T08:13:36.634415+0000","last_became_peered":"2023-11-24T08:13:36.634415+0000","last_unstale":"2023-11-24T08:13:36.646359+0000","last_undegraded":"2023-11-24T08:13:36.646359+0000","last_fullsized":"2023-11-24T08:13:36.646359+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"119'3","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.649297+0000","last_change":"2023-11-24T08:13:35.612998+0000","last_active":"2023-11-24T08:13:36.649297+0000","last_peered":"2023-11-24T08:13:36.649297+0000","last_clean":"2023-11-24T08:13:36.649297+0000","last_became_active":"2023-11-24T08:13:35.612719+0000","last_became_peered":"2023-11-24T08:13:35.612719+0000","last_unstale":"2023-11-24T08:13:36.649297+0000","last_undegraded":"2023-11-24T08:13:36.649297+0000","last_fullsized":"2023-11-24T08:13:36.649297+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:13.144701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633797+0000","last_change":"2023-11-24T08:13:36.629070+0000","last_active":"2023-11-24T08:13:37.633797+0000","last_peered":"2023-11-24T08:13:37.633797+0000","last_clean":"2023-11-24T08:13:37.633797+0000","last_became_active":"2023-11-24T08:13:36.628739+0000","last_became_peered":"2023-11-24T08:13:36.628739+0000","last_unstale":"2023-11-24T08:13:37.633797+0000","last_undegraded":"2023-11-24T08:13:37.633797+0000","last_fullsized":"2023-11-24T08:13:37.633797+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:12.207560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"119'6","reported_seq":17,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.650864+0000","last_change":"2023-11-24T08:13:35.617015+0000","last_active":"2023-11-24T08:13:36.650864+0000","last_peered":"2023-11-24T08:13:36.650864+0000","last_clean":"2023-11-24T08:13:36.650864+0000","last_became_active":"2023-11-24T08:13:35.616760+0000","last_became_peered":"2023-11-24T08:13:35.616760+0000","last_unstale":"2023-11-24T08:13:36.650864+0000","last_undegraded":"2023-11-24T08:13:36.650864+0000","last_fullsized":"2023-11-24T08:13:36.650864+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:22.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.626707+0000","last_change":"2023-11-24T08:13:36.624254+0000","last_active":"2023-11-24T08:13:36.626707+0000","last_peered":"2023-11-24T08:13:36.626707+0000","last_clean":"2023-11-24T08:13:36.626707+0000","last_became_active":"2023-11-24T08:13:36.623919+0000","last_became_peered":"2023-11-24T08:13:36.623919+0000","last_unstale":"2023-11-24T08:13:36.626707+0000","last_undegraded":"2023-11-24T08:13:36.626707+0000","last_fullsized":"2023-11-24T08:13:36.626707+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:06.371472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.637596+0000","last_change":"2023-11-24T08:13:35.608881+0000","last_active":"2023-11-24T08:13:36.637596+0000","last_peered":"2023-11-24T08:13:36.637596+0000","last_clean":"2023-11-24T08:13:36.637596+0000","last_became_active":"2023-11-24T08:13:35.608619+0000","last_became_peered":"2023-11-24T08:13:35.608619+0000","last_unstale":"2023-11-24T08:13:36.637596+0000","last_undegraded":"2023-11-24T08:13:36.637596+0000","last_fullsized":"2023-11-24T08:13:36.637596+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:49:33.927456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646421+0000","last_change":"2023-11-24T08:13:36.636030+0000","last_active":"2023-11-24T08:13:36.646421+0000","last_peered":"2023-11-24T08:13:36.646421+0000","last_clean":"2023-11-24T08:13:36.646421+0000","last_became_active":"2023-11-24T08:13:36.635633+0000","last_became_peered":"2023-11-24T08:13:36.635633+0000","last_unstale":"2023-11-24T08:13:36.646421+0000","last_undegraded":"2023-11-24T08:13:36.646421+0000","last_fullsized":"2023-11-24T08:13:36.646421+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:47.568775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.608067+0000","last_change":"2023-11-24T08:13:35.618133+0000","last_active":"2023-11-24T08:13:36.608067+0000","last_peered":"2023-11-24T08:13:36.608067+0000","last_clean":"2023-11-24T08:13:36.608067+0000","last_became_active":"2023-11-24T08:13:35.617840+0000","last_became_peered":"2023-11-24T08:13:35.617840+0000","last_unstale":"2023-11-24T08:13:36.608067+0000","last_undegraded":"2023-11-24T08:13:36.608067+0000","last_fullsized":"2023-11-24T08:13:36.608067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:24:08.409208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646716+0000","last_change":"2023-11-24T08:13:36.637680+0000","last_active":"2023-11-24T08:13:36.646716+0000","last_peered":"2023-11-24T08:13:36.646716+0000","last_clean":"2023-11-24T08:13:36.646716+0000","last_became_active":"2023-11-24T08:13:36.636949+0000","last_became_peered":"2023-11-24T08:13:36.636949+0000","last_unstale":"2023-11-24T08:13:36.646716+0000","last_undegraded":"2023-11-24T08:13:36.646716+0000","last_fullsized":"2023-11-24T08:13:36.646716+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"119'8","reported_seq":20,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.632780+0000","last_change":"2023-11-24T08:13:35.618903+0000","last_active":"2023-11-24T08:13:37.632780+0000","last_peered":"2023-11-24T08:13:37.632780+0000","last_clean":"2023-11-24T08:13:37.632780+0000","last_became_active":"2023-11-24T08:13:35.618617+0000","last_became_peered":"2023-11-24T08:13:35.618617+0000","last_unstale":"2023-11-24T08:13:37.632780+0000","last_undegraded":"2023-11-24T08:13:37.632780+0000","last_fullsized":"2023-11-24T08:13:37.632780+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:27.722061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634227+0000","last_change":"2023-11-24T08:13:36.645821+0000","last_active":"2023-11-24T08:13:37.634227+0000","last_peered":"2023-11-24T08:13:37.634227+0000","last_clean":"2023-11-24T08:13:37.634227+0000","last_became_active":"2023-11-24T08:13:36.645110+0000","last_became_peered":"2023-11-24T08:13:36.645110+0000","last_unstale":"2023-11-24T08:13:37.634227+0000","last_undegraded":"2023-11-24T08:13:37.634227+0000","last_fullsized":"2023-11-24T08:13:37.634227+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:45.975169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647704+0000","last_change":"2023-11-24T08:13:35.616059+0000","last_active":"2023-11-24T08:13:36.647704+0000","last_peered":"2023-11-24T08:13:36.647704+0000","last_clean":"2023-11-24T08:13:36.647704+0000","last_became_active":"2023-11-24T08:13:35.615801+0000","last_became_peered":"2023-11-24T08:13:35.615801+0000","last_unstale":"2023-11-24T08:13:36.647704+0000","last_undegraded":"2023-11-24T08:13:36.647704+0000","last_fullsized":"2023-11-24T08:13:36.647704+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:05.357674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648706+0000","last_change":"2023-11-24T08:13:36.644154+0000","last_active":"2023-11-24T08:13:36.648706+0000","last_peered":"2023-11-24T08:13:36.648706+0000","last_clean":"2023-11-24T08:13:36.648706+0000","last_became_active":"2023-11-24T08:13:36.643882+0000","last_became_peered":"2023-11-24T08:13:36.643882+0000","last_unstale":"2023-11-24T08:13:36.648706+0000","last_undegraded":"2023-11-24T08:13:36.648706+0000","last_fullsized":"2023-11-24T08:13:36.648706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:04:30.701723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634319+0000","last_change":"2023-11-24T08:13:35.605118+0000","last_active":"2023-11-24T08:13:37.634319+0000","last_peered":"2023-11-24T08:13:37.634319+0000","last_clean":"2023-11-24T08:13:37.634319+0000","last_became_active":"2023-11-24T08:13:35.604833+0000","last_became_peered":"2023-11-24T08:13:35.604833+0000","last_unstale":"2023-11-24T08:13:37.634319+0000","last_undegraded":"2023-11-24T08:13:37.634319+0000","last_fullsized":"2023-11-24T08:13:37.634319+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:49:33.927456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175075+0000","last_change":"2023-11-24T08:13:36.643849+0000","last_active":"2023-11-24T08:13:38.175075+0000","last_peered":"2023-11-24T08:13:38.175075+0000","last_clean":"2023-11-24T08:13:38.175075+0000","last_became_active":"2023-11-24T08:13:36.643556+0000","last_became_peered":"2023-11-24T08:13:36.643556+0000","last_unstale":"2023-11-24T08:13:38.175075+0000","last_undegraded":"2023-11-24T08:13:38.175075+0000","last_fullsized":"2023-11-24T08:13:38.175075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:23.142549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.642967+0000","last_change":"2023-11-24T08:13:35.612757+0000","last_active":"2023-11-24T08:13:36.642967+0000","last_peered":"2023-11-24T08:13:36.642967+0000","last_clean":"2023-11-24T08:13:36.642967+0000","last_became_active":"2023-11-24T08:13:35.612488+0000","last_became_peered":"2023-11-24T08:13:35.612488+0000","last_unstale":"2023-11-24T08:13:36.642967+0000","last_undegraded":"2023-11-24T08:13:36.642967+0000","last_fullsized":"2023-11-24T08:13:36.642967+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:50:58.908133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175656+0000","last_change":"2023-11-24T08:13:36.630930+0000","last_active":"2023-11-24T08:13:38.175656+0000","last_peered":"2023-11-24T08:13:38.175656+0000","last_clean":"2023-11-24T08:13:38.175656+0000","last_became_active":"2023-11-24T08:13:36.630665+0000","last_became_peered":"2023-11-24T08:13:36.630665+0000","last_unstale":"2023-11-24T08:13:38.175656+0000","last_undegraded":"2023-11-24T08:13:38.175656+0000","last_fullsized":"2023-11-24T08:13:38.175656+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:16:25.987306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633713+0000","last_change":"2023-11-24T08:13:35.610547+0000","last_active":"2023-11-24T08:13:37.633713+0000","last_peered":"2023-11-24T08:13:37.633713+0000","last_clean":"2023-11-24T08:13:37.633713+0000","last_became_active":"2023-11-24T08:13:35.610005+0000","last_became_peered":"2023-11-24T08:13:35.610005+0000","last_unstale":"2023-11-24T08:13:37.633713+0000","last_undegraded":"2023-11-24T08:13:37.633713+0000","last_fullsized":"2023-11-24T08:13:37.633713+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:24.209025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174591+0000","last_change":"2023-11-24T08:13:36.633628+0000","last_active":"2023-11-24T08:13:38.174591+0000","last_peered":"2023-11-24T08:13:38.174591+0000","last_clean":"2023-11-24T08:13:38.174591+0000","last_became_active":"2023-11-24T08:13:36.633293+0000","last_became_peered":"2023-11-24T08:13:36.633293+0000","last_unstale":"2023-11-24T08:13:38.174591+0000","last_undegraded":"2023-11-24T08:13:38.174591+0000","last_fullsized":"2023-11-24T08:13:38.174591+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.642448+0000","last_change":"2023-11-24T08:13:35.601288+0000","last_active":"2023-11-24T08:13:36.642448+0000","last_peered":"2023-11-24T08:13:36.642448+0000","last_clean":"2023-11-24T08:13:36.642448+0000","last_became_active":"2023-11-24T08:13:35.600945+0000","last_became_peered":"2023-11-24T08:13:35.600945+0000","last_unstale":"2023-11-24T08:13:36.642448+0000","last_undegraded":"2023-11-24T08:13:36.642448+0000","last_fullsized":"2023-11-24T08:13:36.642448+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:29.918709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644469+0000","last_change":"2023-11-24T08:13:36.631988+0000","last_active":"2023-11-24T08:13:36.644469+0000","last_peered":"2023-11-24T08:13:36.644469+0000","last_clean":"2023-11-24T08:13:36.644469+0000","last_became_active":"2023-11-24T08:13:36.631724+0000","last_became_peered":"2023-11-24T08:13:36.631724+0000","last_unstale":"2023-11-24T08:13:36.644469+0000","last_undegraded":"2023-11-24T08:13:36.644469+0000","last_fullsized":"2023-11-24T08:13:36.644469+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633054+0000","last_change":"2023-11-24T08:13:35.608321+0000","last_active":"2023-11-24T08:13:37.633054+0000","last_peered":"2023-11-24T08:13:37.633054+0000","last_clean":"2023-11-24T08:13:37.633054+0000","last_became_active":"2023-11-24T08:13:35.608009+0000","last_became_peered":"2023-11-24T08:13:35.608009+0000","last_unstale":"2023-11-24T08:13:37.633054+0000","last_undegraded":"2023-11-24T08:13:37.633054+0000","last_fullsized":"2023-11-24T08:13:37.633054+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:07.740664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174586+0000","last_change":"2023-11-24T08:13:36.641630+0000","last_active":"2023-11-24T08:13:38.174586+0000","last_peered":"2023-11-24T08:13:38.174586+0000","last_clean":"2023-11-24T08:13:38.174586+0000","last_became_active":"2023-11-24T08:13:36.641223+0000","last_became_peered":"2023-11-24T08:13:36.641223+0000","last_unstale":"2023-11-24T08:13:38.174586+0000","last_undegraded":"2023-11-24T08:13:38.174586+0000","last_fullsized":"2023-11-24T08:13:38.174586+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.622918+0000","last_change":"2023-11-24T08:13:35.602215+0000","last_active":"2023-11-24T08:13:36.622918+0000","last_peered":"2023-11-24T08:13:36.622918+0000","last_clean":"2023-11-24T08:13:36.622918+0000","last_became_active":"2023-11-24T08:13:35.601939+0000","last_became_peered":"2023-11-24T08:13:35.601939+0000","last_unstale":"2023-11-24T08:13:36.622918+0000","last_undegraded":"2023-11-24T08:13:36.622918+0000","last_fullsized":"2023-11-24T08:13:36.622918+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:46.151145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646738+0000","last_change":"2023-11-24T08:13:36.637896+0000","last_active":"2023-11-24T08:13:36.646738+0000","last_peered":"2023-11-24T08:13:36.646738+0000","last_clean":"2023-11-24T08:13:36.646738+0000","last_became_active":"2023-11-24T08:13:36.637382+0000","last_became_peered":"2023-11-24T08:13:36.637382+0000","last_unstale":"2023-11-24T08:13:36.646738+0000","last_undegraded":"2023-11-24T08:13:36.646738+0000","last_fullsized":"2023-11-24T08:13:36.646738+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.616311+0000","last_change":"2023-11-24T08:13:35.614771+0000","last_active":"2023-11-24T08:13:36.616311+0000","last_peered":"2023-11-24T08:13:36.616311+0000","last_clean":"2023-11-24T08:13:36.616311+0000","last_became_active":"2023-11-24T08:13:35.614499+0000","last_became_peered":"2023-11-24T08:13:35.614499+0000","last_unstale":"2023-11-24T08:13:36.616311+0000","last_undegraded":"2023-11-24T08:13:36.616311+0000","last_fullsized":"2023-11-24T08:13:36.616311+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:25.040350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640775+0000","last_change":"2023-11-24T08:13:36.638956+0000","last_active":"2023-11-24T08:13:36.640775+0000","last_peered":"2023-11-24T08:13:36.640775+0000","last_clean":"2023-11-24T08:13:36.640775+0000","last_became_active":"2023-11-24T08:13:36.638675+0000","last_became_peered":"2023-11-24T08:13:36.638675+0000","last_unstale":"2023-11-24T08:13:36.640775+0000","last_undegraded":"2023-11-24T08:13:36.640775+0000","last_fullsized":"2023-11-24T08:13:36.640775+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:58.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647925+0000","last_change":"2023-11-24T08:13:35.615266+0000","last_active":"2023-11-24T08:13:36.647925+0000","last_peered":"2023-11-24T08:13:36.647925+0000","last_clean":"2023-11-24T08:13:36.647925+0000","last_became_active":"2023-11-24T08:13:35.614996+0000","last_became_peered":"2023-11-24T08:13:35.614996+0000","last_unstale":"2023-11-24T08:13:36.647925+0000","last_undegraded":"2023-11-24T08:13:36.647925+0000","last_fullsized":"2023-11-24T08:13:36.647925+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:25.040350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646627+0000","last_change":"2023-11-24T08:13:36.633380+0000","last_active":"2023-11-24T08:13:36.646627+0000","last_peered":"2023-11-24T08:13:36.646627+0000","last_clean":"2023-11-24T08:13:36.646627+0000","last_became_active":"2023-11-24T08:13:36.633085+0000","last_became_peered":"2023-11-24T08:13:36.633085+0000","last_unstale":"2023-11-24T08:13:36.646627+0000","last_undegraded":"2023-11-24T08:13:36.646627+0000","last_fullsized":"2023-11-24T08:13:36.646627+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:17:12.969900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.609277+0000","last_change":"2023-11-24T08:13:35.609014+0000","last_active":"2023-11-24T08:13:36.609277+0000","last_peered":"2023-11-24T08:13:36.609277+0000","last_clean":"2023-11-24T08:13:36.609277+0000","last_became_active":"2023-11-24T08:13:35.607514+0000","last_became_peered":"2023-11-24T08:13:35.607514+0000","last_unstale":"2023-11-24T08:13:36.609277+0000","last_undegraded":"2023-11-24T08:13:36.609277+0000","last_fullsized":"2023-11-24T08:13:36.609277+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:00.114560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646930+0000","last_change":"2023-11-24T08:13:36.636653+0000","last_active":"2023-11-24T08:13:36.646930+0000","last_peered":"2023-11-24T08:13:36.646930+0000","last_clean":"2023-11-24T08:13:36.646930+0000","last_became_active":"2023-11-24T08:13:36.636148+0000","last_became_peered":"2023-11-24T08:13:36.636148+0000","last_unstale":"2023-11-24T08:13:36.646930+0000","last_undegraded":"2023-11-24T08:13:36.646930+0000","last_fullsized":"2023-11-24T08:13:36.646930+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.620266+0000","last_change":"2023-11-24T08:13:35.617006+0000","last_active":"2023-11-24T08:13:36.620266+0000","last_peered":"2023-11-24T08:13:36.620266+0000","last_clean":"2023-11-24T08:13:36.620266+0000","last_became_active":"2023-11-24T08:13:35.616691+0000","last_became_peered":"2023-11-24T08:13:35.616691+0000","last_unstale":"2023-11-24T08:13:36.620266+0000","last_undegraded":"2023-11-24T08:13:36.620266+0000","last_fullsized":"2023-11-24T08:13:36.620266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:54.475312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636943+0000","last_change":"2023-11-24T08:13:36.634182+0000","last_active":"2023-11-24T08:13:36.636943+0000","last_peered":"2023-11-24T08:13:36.636943+0000","last_clean":"2023-11-24T08:13:36.636943+0000","last_became_active":"2023-11-24T08:13:36.633862+0000","last_became_peered":"2023-11-24T08:13:36.633862+0000","last_unstale":"2023-11-24T08:13:36.636943+0000","last_undegraded":"2023-11-24T08:13:36.636943+0000","last_fullsized":"2023-11-24T08:13:36.636943+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:45:04.376236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.621632+0000","last_change":"2023-11-24T08:13:35.607458+0000","last_active":"2023-11-24T08:13:36.621632+0000","last_peered":"2023-11-24T08:13:36.621632+0000","last_clean":"2023-11-24T08:13:36.621632+0000","last_became_active":"2023-11-24T08:13:35.607216+0000","last_became_peered":"2023-11-24T08:13:35.607216+0000","last_unstale":"2023-11-24T08:13:36.621632+0000","last_undegraded":"2023-11-24T08:13:36.621632+0000","last_fullsized":"2023-11-24T08:13:36.621632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:13.144701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635112+0000","last_change":"2023-11-24T08:13:36.634807+0000","last_active":"2023-11-24T08:13:37.635112+0000","last_peered":"2023-11-24T08:13:37.635112+0000","last_clean":"2023-11-24T08:13:37.635112+0000","last_became_active":"2023-11-24T08:13:36.634614+0000","last_became_peered":"2023-11-24T08:13:36.634614+0000","last_unstale":"2023-11-24T08:13:37.635112+0000","last_undegraded":"2023-11-24T08:13:37.635112+0000","last_fullsized":"2023-11-24T08:13:37.635112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:50:59.921931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633550+0000","last_change":"2023-11-24T08:13:35.606491+0000","last_active":"2023-11-24T08:13:37.633550+0000","last_peered":"2023-11-24T08:13:37.633550+0000","last_clean":"2023-11-24T08:13:37.633550+0000","last_became_active":"2023-11-24T08:13:35.605798+0000","last_became_peered":"2023-11-24T08:13:35.605798+0000","last_unstale":"2023-11-24T08:13:37.633550+0000","last_undegraded":"2023-11-24T08:13:37.633550+0000","last_fullsized":"2023-11-24T08:13:37.633550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:24.974669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635573+0000","last_change":"2023-11-24T08:13:36.641154+0000","last_active":"2023-11-24T08:13:37.635573+0000","last_peered":"2023-11-24T08:13:37.635573+0000","last_clean":"2023-11-24T08:13:37.635573+0000","last_became_active":"2023-11-24T08:13:36.640855+0000","last_became_peered":"2023-11-24T08:13:36.640855+0000","last_unstale":"2023-11-24T08:13:37.635573+0000","last_undegraded":"2023-11-24T08:13:37.635573+0000","last_fullsized":"2023-11-24T08:13:37.635573+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:18.111589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634958+0000","last_change":"2023-11-24T08:13:35.593966+0000","last_active":"2023-11-24T08:13:37.634958+0000","last_peered":"2023-11-24T08:13:37.634958+0000","last_clean":"2023-11-24T08:13:37.634958+0000","last_became_active":"2023-11-24T08:13:35.593014+0000","last_became_peered":"2023-11-24T08:13:35.593014+0000","last_unstale":"2023-11-24T08:13:37.634958+0000","last_undegraded":"2023-11-24T08:13:37.634958+0000","last_fullsized":"2023-11-24T08:13:37.634958+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:57.719449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175348+0000","last_change":"2023-11-24T08:13:36.622931+0000","last_active":"2023-11-24T08:13:38.175348+0000","last_peered":"2023-11-24T08:13:38.175348+0000","last_clean":"2023-11-24T08:13:38.175348+0000","last_became_active":"2023-11-24T08:13:36.621653+0000","last_became_peered":"2023-11-24T08:13:36.621653+0000","last_unstale":"2023-11-24T08:13:38.175348+0000","last_undegraded":"2023-11-24T08:13:38.175348+0000","last_fullsized":"2023-11-24T08:13:38.175348+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:14.158500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.632862+0000","last_change":"2023-11-24T08:13:35.609006+0000","last_active":"2023-11-24T08:13:37.632862+0000","last_peered":"2023-11-24T08:13:37.632862+0000","last_clean":"2023-11-24T08:13:37.632862+0000","last_became_active":"2023-11-24T08:13:35.608698+0000","last_became_peered":"2023-11-24T08:13:35.608698+0000","last_unstale":"2023-11-24T08:13:37.632862+0000","last_undegraded":"2023-11-24T08:13:37.632862+0000","last_fullsized":"2023-11-24T08:13:37.632862+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:29.027287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646644+0000","last_change":"2023-11-24T08:13:36.634584+0000","last_active":"2023-11-24T08:13:36.646644+0000","last_peered":"2023-11-24T08:13:36.646644+0000","last_clean":"2023-11-24T08:13:36.646644+0000","last_became_active":"2023-11-24T08:13:36.634322+0000","last_became_peered":"2023-11-24T08:13:36.634322+0000","last_unstale":"2023-11-24T08:13:36.646644+0000","last_undegraded":"2023-11-24T08:13:36.646644+0000","last_fullsized":"2023-11-24T08:13:36.646644+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:53.189958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.616559+0000","last_change":"2023-11-24T08:13:35.614084+0000","last_active":"2023-11-24T08:13:36.616559+0000","last_peered":"2023-11-24T08:13:36.616559+0000","last_clean":"2023-11-24T08:13:36.616559+0000","last_became_active":"2023-11-24T08:13:35.613174+0000","last_became_peered":"2023-11-24T08:13:35.613174+0000","last_unstale":"2023-11-24T08:13:36.616559+0000","last_undegraded":"2023-11-24T08:13:36.616559+0000","last_fullsized":"2023-11-24T08:13:36.616559+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:55.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643674+0000","last_change":"2023-11-24T08:13:36.632488+0000","last_active":"2023-11-24T08:13:36.643674+0000","last_peered":"2023-11-24T08:13:36.643674+0000","last_clean":"2023-11-24T08:13:36.643674+0000","last_became_active":"2023-11-24T08:13:36.632121+0000","last_became_peered":"2023-11-24T08:13:36.632121+0000","last_unstale":"2023-11-24T08:13:36.643674+0000","last_undegraded":"2023-11-24T08:13:36.643674+0000","last_fullsized":"2023-11-24T08:13:36.643674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:17:12.969900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633262+0000","last_change":"2023-11-24T08:13:35.615588+0000","last_active":"2023-11-24T08:13:37.633262+0000","last_peered":"2023-11-24T08:13:37.633262+0000","last_clean":"2023-11-24T08:13:37.633262+0000","last_became_active":"2023-11-24T08:13:35.615319+0000","last_became_peered":"2023-11-24T08:13:35.615319+0000","last_unstale":"2023-11-24T08:13:37.633262+0000","last_undegraded":"2023-11-24T08:13:37.633262+0000","last_fullsized":"2023-11-24T08:13:37.633262+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:10.801280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.608467+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:13:36.608467+0000","last_peered":"2023-11-24T08:13:36.608467+0000","last_clean":"2023-11-24T08:13:36.608467+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:13:36.608467+0000","last_undegraded":"2023-11-24T08:13:36.608467+0000","last_fullsized":"2023-11-24T08:13:36.608467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33228,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338612,"statfs":{"total":96636764160,"available":96602738688,"internally_reserved":0,"allocated":2093056,"data_stored":1241374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0880000000000001,"15min":1.0289999999999999},"min":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.516,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.119,"5min":1.1279999999999999,"15min":1.0900000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.6419999999999999,"5min":2.403,"15min":2.7109999999999999},"last":0.93300000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.161,"15min":1.145},"min":{"1min":0.77600000000000002,"5min":0.629,"15min":0.46300000000000002},"max":{"1min":1.579,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.873},{"interface":"front","average":{"1min":1.323,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.81499999999999995,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.6639999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.26}]},{"osd":2,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1319999999999999,"15min":1.196},"min":{"1min":0.75,"5min":0.67000000000000004,"15min":0.58699999999999997},"max":{"1min":1.4410000000000001,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":1.774},{"interface":"front","average":{"1min":1.232,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7669999999999999,"5min":4.0650000000000004,"15min":4.633},"last":1.171}]},{"osd":3,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.79500000000000004,"5min":0.68700000000000006,"15min":0.504},"max":{"1min":1.6040000000000001,"5min":3.133,"15min":3.133},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.397,"15min":1.3740000000000001},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.706,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.0860000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.331,"15min":1.3340000000000001},"min":{"1min":0.84699999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.635,"5min":2.661,"15min":2.851},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.335},"min":{"1min":0.67700000000000005,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.7929999999999999,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.6240000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4910000000000001,"15min":1.4670000000000001},"min":{"1min":0.74099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.841,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.48,"15min":1.48},"min":{"1min":0.89100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9770000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.8029999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.556,"15min":1.546},"min":{"1min":0.77300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0529999999999999,"5min":2.992,"15min":2.992},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4870000000000001,"15min":1.46},"min":{"1min":0.93200000000000005,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.702,"5min":3.798,"15min":3.798},"last":1.8859999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1482752,"data_stored":644160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.079,"15min":1.0900000000000001},"min":{"1min":0.875,"5min":0.621,"15min":0.621},"max":{"1min":1.6299999999999999,"5min":1.95,"15min":1.95},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0620000000000001,"15min":1.0940000000000001},"min":{"1min":0.73599999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5960000000000001,"5min":1.986,"15min":2.2069999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.149,"15min":1.2390000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.52,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.018},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.141,"15min":1.2490000000000001},"min":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6240000000000001,"5min":1.694,"15min":2.4039999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.091,"15min":1.117},"min":{"1min":0.89100000000000001,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.621,"5min":1.6779999999999999,"15min":1.9330000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.379,"5min":1.2589999999999999,"15min":1.357},"min":{"1min":0.81699999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7749999999999999,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.55}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3260000000000001,"15min":1.385},"min":{"1min":0.96999999999999997,"5min":0.83399999999999996,"15min":0.71499999999999997},"max":{"1min":1.538,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.339,"15min":1.3460000000000001},"min":{"1min":0.98599999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.9339999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.0760000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.3879999999999999,"15min":1.5169999999999999},"min":{"1min":1.0129999999999999,"5min":0.79900000000000004,"15min":0.76500000000000001},"max":{"1min":1.756,"5min":2.895,"15min":2.895},"last":1.655},{"interface":"front","average":{"1min":1.502,"5min":1.452,"15min":1.542},"min":{"1min":1.0940000000000001,"5min":1.056,"15min":0.90800000000000003},"max":{"1min":1.8160000000000001,"5min":2.242,"15min":2.8820000000000001},"last":1.454}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4199999999999999,"15min":1.5289999999999999},"min":{"1min":0.94599999999999995,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":2.1320000000000001,"5min":2.2250000000000001,"15min":2.629},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.544,"15min":1.663},"min":{"1min":1.3340000000000001,"5min":1.101,"15min":1.022},"max":{"1min":1.8939999999999999,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.7629999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.5569999999999999,"15min":1.6919999999999999},"min":{"1min":1.274,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":2.101,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.379},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5600000000000001,"15min":1.6639999999999999},"min":{"1min":1.151,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6890000000000001},"last":1.593}]}]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":650994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.02,"15min":1.101},"min":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5,"5min":1.694,"15min":3.1179999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.123,"15min":1.1719999999999999},"min":{"1min":0.67700000000000005,"5min":0.65100000000000002,"15min":0.60999999999999999},"max":{"1min":1.627,"5min":1.859,"15min":2.198},"last":1.5189999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1299999999999999,"15min":1.159},"min":{"1min":0.73099999999999998,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.3799999999999999,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1759999999999999,"15min":1.1539999999999999},"min":{"1min":0.78100000000000003,"5min":0.72599999999999998,"15min":0.48699999999999999},"max":{"1min":1.5289999999999999,"5min":1.659,"15min":1.6910000000000001},"last":1.115}]},{"osd":2,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1990000000000001,"15min":1.2110000000000001},"min":{"1min":0.86899999999999999,"5min":0.65100000000000002,"15min":0.51600000000000001},"max":{"1min":1.5669999999999999,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.87},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1679999999999999,"15min":1.169},"min":{"1min":0.84999999999999998,"5min":0.69999999999999996,"15min":0.64200000000000002},"max":{"1min":1.639,"5min":1.7370000000000001,"15min":1.8129999999999999},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1459999999999999,"15min":1.1759999999999999},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.432},"max":{"1min":1.6080000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.29,"15min":1.3280000000000001},"min":{"1min":0.93799999999999994,"5min":0.79800000000000004,"15min":0.55200000000000005},"max":{"1min":1.5549999999999999,"5min":1.9730000000000001,"15min":1.99},"last":1.214}]},{"osd":4,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3200000000000001,"15min":1.333},"min":{"1min":0.79400000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.732,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.27},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":1.169,"5min":0.75600000000000001,"15min":0.46700000000000003},"max":{"1min":1.694,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.389}]},{"osd":6,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.337,"15min":1.427},"min":{"1min":0.88200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.766,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":1.0049999999999999,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":1.8580000000000001,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.4690000000000001,"15min":1.5700000000000001},"min":{"1min":1.105,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.869,"5min":2.129,"15min":3.4420000000000002},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.601,"15min":1.6120000000000001},"min":{"1min":1.3320000000000001,"5min":0.78900000000000003,"15min":0.71199999999999997},"max":{"1min":1.9319999999999999,"5min":2.181,"15min":2.181},"last":1.4810000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36800,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335040,"statfs":{"total":96636764160,"available":96599080960,"internally_reserved":0,"allocated":1490944,"data_stored":650390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.075,"15min":1.131},"min":{"1min":0.59999999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.0669999999999999,"15min":1.087},"min":{"1min":0.70899999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.516,"5min":1.7150000000000001,"15min":2.222},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1259999999999999,"15min":1.147},"min":{"1min":0.71499999999999997,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.593,"5min":1.6639999999999999,"15min":1.74},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.1619999999999999,"15min":1.1830000000000001},"min":{"1min":0.81699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6080000000000001,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.4339999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.054,"15min":1.1399999999999999},"min":{"1min":0.66600000000000004,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5049999999999999,"5min":1.5780000000000001,"15min":2.1419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.181,"5min":1.2210000000000001,"15min":1.3300000000000001},"min":{"1min":0.71199999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.5580000000000001,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":1.2330000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3069999999999999,"15min":1.4019999999999999},"min":{"1min":0.89000000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.756,"5min":2.121,"15min":2.9100000000000001},"last":1.528},{"interface":"front","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.3080000000000001},"min":{"1min":0.91200000000000003,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.8280000000000001},"last":1.109}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.446,"15min":1.498},"min":{"1min":0.996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.9790000000000001,"5min":2.23,"15min":2.23},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.403,"15min":1.514},"min":{"1min":0.95999999999999996,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.782,"5min":1.8640000000000001,"15min":2.3879999999999999},"last":1.571}]},{"osd":6,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2290000000000001,"15min":1.3999999999999999},"min":{"1min":0.89200000000000002,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.6220000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5,"15min":1.625},"min":{"1min":0.92000000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9590000000000001,"5min":1.9950000000000001,"15min":3.391},"last":1.603}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.452,"15min":1.593},"min":{"1min":1.157,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.825,"5min":2.052,"15min":3.1230000000000002},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.381,"15min":1.5489999999999999},"min":{"1min":1.113,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7929999999999999,"5min":1.857,"15min":3.4849999999999999},"last":1.484}]}]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36800,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335040,"statfs":{"total":96636764160,"available":96599080960,"internally_reserved":0,"allocated":1495040,"data_stored":645772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.55700000000000005,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.829,"5min":3.181,"15min":10.557},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.093,"5min":0.88800000000000001,"15min":0.91500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":1.3540000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.80200000000000005,"15min":0.83199999999999996},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.496,"5min":2.101,"15min":2.101},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.054,"5min":0.94399999999999995,"15min":1.0009999999999999},"min":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":1.5660000000000001,"5min":3.3260000000000001,"15min":13.090999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88700000000000001,"15min":0.95899999999999996},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5780000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.649,"5min":3.2480000000000002,"15min":13.460000000000001},"last":1.298}]},{"osd":4,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.72399999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.79,"5min":1.79,"15min":4.367},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.2110000000000001,"15min":1.3029999999999999},"min":{"1min":0.878,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.5720000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.262}]},{"osd":5,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.244,"15min":1.3169999999999999},"min":{"1min":0.748,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.0609999999999999,"5min":3.4399999999999999,"15min":13.231},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.1930000000000001,"15min":1.306},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.76,"5min":1.8340000000000001,"15min":5.1180000000000003},"last":1.4279999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.331,"15min":1.3979999999999999},"min":{"1min":0.95999999999999996,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":1.9590000000000001,"5min":3.5030000000000001,"15min":13.359999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.242,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0150000000000001,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.0820000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.2989999999999999,"15min":1.4159999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":2.101,"15min":4.5060000000000002},"last":1.476},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3029999999999999,"15min":1.4339999999999999},"min":{"1min":0.76400000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.887,"5min":3.331,"15min":4.032},"last":0.997}]}]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1490944,"data_stored":650390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.78300000000000003,"15min":0.88900000000000001},"min":{"1min":0.66700000000000004,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.746},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.90300000000000002,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.782,"5min":1.782,"15min":2.5779999999999998},"last":1.2230000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":0.90800000000000003,"15min":1.081},"min":{"1min":0.66100000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5089999999999999,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95799999999999996,"15min":1.091},"min":{"1min":0.69499999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4399999999999999,"5min":1.788,"15min":2.6240000000000001},"last":1.0189999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84099999999999997,"15min":0.92700000000000005},"min":{"1min":0.58699999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5309999999999999,"5min":1.552,"15min":2.2090000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0289999999999999,"15min":1.214},"min":{"1min":0.54400000000000004,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.738,"5min":1.946,"15min":2.9950000000000001},"last":1.1950000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.298,"15min":1.387},"min":{"1min":1.0029999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.883,"5min":1.893,"15min":2.7160000000000002},"last":1.405},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2490000000000001,"15min":1.371},"min":{"1min":0.96699999999999997,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6459999999999999,"5min":1.8779999999999999,"15min":3.3279999999999998},"last":1.258}]},{"osd":5,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.286,"15min":1.4159999999999999},"min":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.73399999999999999},"max":{"1min":1.9319999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2350000000000001,"15min":1.3740000000000001},"min":{"1min":0.79800000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.724,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":1.3360000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.194,"15min":1.3100000000000001},"min":{"1min":0.90600000000000003,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":1.8640000000000001,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.3520000000000001,"15min":1.4830000000000001},"min":{"1min":1.129,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.069,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.3660000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.329,"15min":1.4730000000000001},"min":{"1min":0.93999999999999995,"5min":0.77700000000000002,"15min":0.71299999999999997},"max":{"1min":1.7669999999999999,"5min":2.1070000000000002,"15min":2.859},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2789999999999999,"15min":1.4379999999999999},"min":{"1min":0.81399999999999995,"5min":0.746,"15min":0.746},"max":{"1min":1.925,"5min":1.9930000000000001,"15min":3.3809999999999998},"last":1.1559999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1486848,"data_stored":644214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.81699999999999995,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3280000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.427,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.141}]},{"osd":2,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81999999999999995,"15min":0.73499999999999999},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":1.3140000000000001,"5min":1.861,"15min":1.861},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.91800000000000004,"15min":0.89000000000000001},"min":{"1min":0.47499999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4039999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.1930000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0109999999999999,"15min":0.96799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6599999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.046},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93200000000000005,"15min":0.89200000000000002},"min":{"1min":0.42899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.629,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.27,"15min":1.2470000000000001},"min":{"1min":0.84699999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.7450000000000001,"5min":2.028,"15min":2.7959999999999998},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.77200000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.742,"5min":2.1080000000000001,"15min":3.0409999999999999},"last":1.2430000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.268,"15min":1.218},"min":{"1min":0.81999999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6240000000000001,"5min":2.02,"15min":3.7639999999999998},"last":1.405},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.1839999999999999},"min":{"1min":0.75700000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.665,"5min":1.8759999999999999,"15min":3.286},"last":0.98199999999999998}]},{"osd":6,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.321,"15min":1.2430000000000001},"min":{"1min":0.91000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6879999999999999,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.379},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3069999999999999,"15min":1.2210000000000001},"min":{"1min":0.80800000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6970000000000001,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.3380000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3380000000000001,"15min":1.3029999999999999},"min":{"1min":0.71899999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7509999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.319},{"interface":"front","average":{"1min":1.367,"5min":1.345,"15min":1.2849999999999999},"min":{"1min":0.83399999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7190000000000001,"5min":3.653,"15min":3.653},"last":1.117}]}]},{"osd":0,"up_from":13,"seq":55834574959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37384,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334456,"statfs":{"total":96636764160,"available":96598482944,"internally_reserved":0,"allocated":2093056,"data_stored":1236182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.84299999999999997},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.27,"5min":1.425,"15min":1.6020000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.97199999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.452,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.363}]},{"osd":2,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0229999999999999,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.833,"5min":1.891,"15min":1.891},"last":1.141},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.88300000000000001,"15min":0.93899999999999995},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3120000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98499999999999999,"15min":1.0920000000000001},"min":{"1min":0.55600000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.5820000000000001,"5min":1.79,"15min":2.0670000000000002},"last":1.456},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90300000000000002,"15min":1.022},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.641,"5min":1.641,"15min":2.1789999999999998},"last":1.4079999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2350000000000001,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.619,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":1.546},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0960000000000001,"15min":1.2170000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.569,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1439999999999999,"15min":1.282},"min":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.69699999999999995},"max":{"1min":1.7210000000000001,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":1.504},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1879999999999999,"15min":1.377},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5609999999999999,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":1.1870000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.288,"15min":1.405},"min":{"1min":0.84599999999999997,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.792,"5min":1.986,"15min":2.4369999999999998},"last":1.583},{"interface":"front","average":{"1min":1.264,"5min":1.2609999999999999,"15min":1.4079999999999999},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.998,"5min":1.998,"15min":2.738},"last":1.6180000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.4910000000000001},"min":{"1min":0.96899999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.1349999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.4750000000000001},"min":{"1min":0.86799999999999999,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":2.0499999999999998,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.6499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:13:41.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:13:41.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.331+0000 7f1e953a9640 1 Processor -- start 2023-11-24T08:13:41.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.331+0000 7f1e953a9640 1 -- start start 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e9015c7f0 0x7f1e9006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1e900cf450 0x7f1e900cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e900cdfb0 0x7f1e900ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e90156010 con 0x7f1e900cdfb0 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9006ce30 con 0x7f1e900cf450 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.333+0000 7f1e953a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e900cfe10 con 0x7f1e9015c7f0 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e900cdfb0 0x7f1e900ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e900cdfb0 0x7f1e900ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33138/0 (socket says 172.21.15.107:33138) 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4097876865 learned_addr learned my addr 172.21.15.107:0/4097876865 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e867fc640 1 --2- 172.21.15.107:0/4097876865 >> v2:172.21.15.120:3300/0 conn(0x7f1e900cf450 0x7f1e900cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8effd640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e9015c7f0 0x7f1e9006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4097876865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e9015c7f0 msgr2=0x7f1e9006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e9015c7f0 0x7f1e9006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4097876865 >> v2:172.21.15.120:3300/0 conn(0x7f1e900cf450 msgr2=0x7f1e900cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4097876865 >> v2:172.21.15.120:3300/0 conn(0x7f1e900cf450 0x7f1e900cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.334+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4097876865 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e900cece0 con 0x7f1e900cdfb0 2023-11-24T08:13:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.335+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e900cdfb0 0x7f1e900ce440 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1e9006b550 tx=0x7f1e8802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0899d4d7349e9c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e8e7fc640 1 -- 172.21.15.107:0/4097876865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8802ca70 con 0x7f1e900cdfb0 2023-11-24T08:13:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e8e7fc640 1 -- 172.21.15.107:0/4097876865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8802cbd0 con 0x7f1e900cdfb0 2023-11-24T08:13:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e8e7fc640 1 -- 172.21.15.107:0/4097876865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880343f0 con 0x7f1e900cdfb0 2023-11-24T08:13:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 -- 172.21.15.107:0/4097876865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e900cdfb0 msgr2=0x7f1e900ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e900cdfb0 0x7f1e900ce440 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1e9006b550 tx=0x7f1e8802bed0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 -- 172.21.15.107:0/4097876865 shutdown_connections 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e900cdfb0 0x7f1e900ce440 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4097876865 >> v2:172.21.15.120:3300/0 conn(0x7f1e900cf450 0x7f1e900cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4097876865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e9015c7f0 0x7f1e9006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.336+0000 7f1e953a9640 1 -- 172.21.15.107:0/4097876865 >> 172.21.15.107:0/4097876865 conn(0x7f1e9006a920 msgr2=0x7f1e9006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.337+0000 7f1e953a9640 1 -- 172.21.15.107:0/4097876865 shutdown_connections 2023-11-24T08:13:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.337+0000 7f1e953a9640 1 -- 172.21.15.107:0/4097876865 wait complete. 2023-11-24T08:13:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.338+0000 7f1e953a9640 1 Processor -- start 2023-11-24T08:13:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.338+0000 7f1e953a9640 1 -- start start 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e900cdfb0 0x7f1e90157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e900cf450 0x7f1e90149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90156c30 con 0x7f1e9015c7f0 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90156f10 con 0x7f1e900cdfb0 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.339+0000 7f1e953a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e901571f0 con 0x7f1e900cf450 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33142/0 (socket says 172.21.15.107:33142) 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4018964961 learned_addr learned my addr 172.21.15.107:0/4018964961 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8effd640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e900cdfb0 0x7f1e90157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e900cf450 msgr2=0x7f1e90149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e900cf450 0x7f1e90149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e900cdfb0 msgr2=0x7f1e90157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e900cdfb0 0x7f1e90157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 -- 172.21.15.107:0/4018964961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88009b30 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.340+0000 7f1e8f7fe640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1e88036040 tx=0x7f1e88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.341+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88009510 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.341+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8803e450 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.341+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e901574d0 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.341+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8803e7e0 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.341+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e90154a10 con 0x7f1e9015c7f0 2023-11-24T08:13:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.342+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e901579f0 con 0x7f1e9015c7f0 2023-11-24T08:13:41.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.343+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f1e8803e940 con 0x7f1e9015c7f0 2023-11-24T08:13:41.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.344+0000 7f1e87fff640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 0x7f1e7006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.345+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e880aaeb0 con 0x7f1e9015c7f0 2023-11-24T08:13:41.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.345+0000 7f1e8effd640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 0x7f1e7006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.345+0000 7f1e8effd640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 0x7f1e7006e380 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1e80006fd0 tx=0x7f1e80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.349+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e88077690 con 0x7f1e9015c7f0 2023-11-24T08:13:41.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.535+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e9006d600 con 0x7f1e7006bec0 2023-11-24T08:13:41.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.558+0000 7f1e87fff640 1 -- 172.21.15.107:0/4018964961 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287602 (secure 0 0 0) 0x7f1e8000b200 con 0x7f1e7006bec0 2023-11-24T08:13:41.564 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:13:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.564+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 msgr2=0x7f1e7006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.564+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 0x7f1e7006e380 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1e80006fd0 tx=0x7f1e80008040 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.564+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 msgr2=0x7f1e90156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.564+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1e88036040 tx=0x7f1e88004a40 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 shutdown_connections 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e7006bec0 0x7f1e7006e380 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e9015c7f0 0x7f1e90156170 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e900cf450 0x7f1e90149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 --2- 172.21.15.107:0/4018964961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e900cdfb0 0x7f1e90157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 >> 172.21.15.107:0/4018964961 conn(0x7f1e9006a920 msgr2=0x7f1e9006db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 shutdown_connections 2023-11-24T08:13:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.565+0000 7f1e953a9640 1 -- 172.21.15.107:0/4018964961 wait complete. 2023-11-24T08:13:41.587 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2023-11-24T08:13:39.597978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283076,"kb_used_data":12824,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754691644,"statfs":{"total":773094113280,"available":772804243456,"internally_reserved":0,"allocated":13131776,"data_stored":6363476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":276614718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.029132"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646601+0000","last_change":"2023-11-24T08:13:36.632799+0000","last_active":"2023-11-24T08:13:36.646601+0000","last_peered":"2023-11-24T08:13:36.646601+0000","last_clean":"2023-11-24T08:13:36.646601+0000","last_became_active":"2023-11-24T08:13:36.629432+0000","last_became_peered":"2023-11-24T08:13:36.629432+0000","last_unstale":"2023-11-24T08:13:36.646601+0000","last_undegraded":"2023-11-24T08:13:36.646601+0000","last_fullsized":"2023-11-24T08:13:36.646601+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:39.870154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636297+0000","last_change":"2023-11-24T08:13:36.640528+0000","last_active":"2023-11-24T08:13:37.636297+0000","last_peered":"2023-11-24T08:13:37.636297+0000","last_clean":"2023-11-24T08:13:37.636297+0000","last_became_active":"2023-11-24T08:13:36.639049+0000","last_became_peered":"2023-11-24T08:13:36.639049+0000","last_unstale":"2023-11-24T08:13:37.636297+0000","last_undegraded":"2023-11-24T08:13:37.636297+0000","last_fullsized":"2023-11-24T08:13:37.636297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175223+0000","last_change":"2023-11-24T08:13:36.633569+0000","last_active":"2023-11-24T08:13:38.175223+0000","last_peered":"2023-11-24T08:13:38.175223+0000","last_clean":"2023-11-24T08:13:38.175223+0000","last_became_active":"2023-11-24T08:13:36.633245+0000","last_became_peered":"2023-11-24T08:13:36.633245+0000","last_unstale":"2023-11-24T08:13:38.175223+0000","last_undegraded":"2023-11-24T08:13:38.175223+0000","last_fullsized":"2023-11-24T08:13:38.175223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:02.590196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646950+0000","last_change":"2023-11-24T08:13:36.637419+0000","last_active":"2023-11-24T08:13:36.646950+0000","last_peered":"2023-11-24T08:13:36.646950+0000","last_clean":"2023-11-24T08:13:36.646950+0000","last_became_active":"2023-11-24T08:13:36.636461+0000","last_became_peered":"2023-11-24T08:13:36.636461+0000","last_unstale":"2023-11-24T08:13:36.646950+0000","last_undegraded":"2023-11-24T08:13:36.646950+0000","last_fullsized":"2023-11-24T08:13:36.646950+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633553+0000","last_change":"2023-11-24T08:13:36.633865+0000","last_active":"2023-11-24T08:13:37.633553+0000","last_peered":"2023-11-24T08:13:37.633553+0000","last_clean":"2023-11-24T08:13:37.633553+0000","last_became_active":"2023-11-24T08:13:36.633073+0000","last_became_peered":"2023-11-24T08:13:36.633073+0000","last_unstale":"2023-11-24T08:13:37.633553+0000","last_undegraded":"2023-11-24T08:13:37.633553+0000","last_fullsized":"2023-11-24T08:13:37.633553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:37.962745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641013+0000","last_change":"2023-11-24T08:13:36.631124+0000","last_active":"2023-11-24T08:13:36.641013+0000","last_peered":"2023-11-24T08:13:36.641013+0000","last_clean":"2023-11-24T08:13:36.641013+0000","last_became_active":"2023-11-24T08:13:36.630792+0000","last_became_peered":"2023-11-24T08:13:36.630792+0000","last_unstale":"2023-11-24T08:13:36.641013+0000","last_undegraded":"2023-11-24T08:13:36.641013+0000","last_fullsized":"2023-11-24T08:13:36.641013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:47:37.309260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633131+0000","last_change":"2023-11-24T08:13:36.619711+0000","last_active":"2023-11-24T08:13:37.633131+0000","last_peered":"2023-11-24T08:13:37.633131+0000","last_clean":"2023-11-24T08:13:37.633131+0000","last_became_active":"2023-11-24T08:13:36.619089+0000","last_became_peered":"2023-11-24T08:13:36.619089+0000","last_unstale":"2023-11-24T08:13:37.633131+0000","last_undegraded":"2023-11-24T08:13:37.633131+0000","last_fullsized":"2023-11-24T08:13:37.633131+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:10:10.166547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174606+0000","last_change":"2023-11-24T08:13:36.628482+0000","last_active":"2023-11-24T08:13:38.174606+0000","last_peered":"2023-11-24T08:13:38.174606+0000","last_clean":"2023-11-24T08:13:38.174606+0000","last_became_active":"2023-11-24T08:13:36.627494+0000","last_became_peered":"2023-11-24T08:13:36.627494+0000","last_unstale":"2023-11-24T08:13:38.174606+0000","last_undegraded":"2023-11-24T08:13:38.174606+0000","last_fullsized":"2023-11-24T08:13:38.174606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643908+0000","last_change":"2023-11-24T08:13:36.629705+0000","last_active":"2023-11-24T08:13:36.643908+0000","last_peered":"2023-11-24T08:13:36.643908+0000","last_clean":"2023-11-24T08:13:36.643908+0000","last_became_active":"2023-11-24T08:13:36.629095+0000","last_became_peered":"2023-11-24T08:13:36.629095+0000","last_unstale":"2023-11-24T08:13:36.643908+0000","last_undegraded":"2023-11-24T08:13:36.643908+0000","last_fullsized":"2023-11-24T08:13:36.643908+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:23.314730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648045+0000","last_change":"2023-11-24T08:13:36.641428+0000","last_active":"2023-11-24T08:13:36.648045+0000","last_peered":"2023-11-24T08:13:36.648045+0000","last_clean":"2023-11-24T08:13:36.648045+0000","last_became_active":"2023-11-24T08:13:36.641123+0000","last_became_peered":"2023-11-24T08:13:36.641123+0000","last_unstale":"2023-11-24T08:13:36.648045+0000","last_undegraded":"2023-11-24T08:13:36.648045+0000","last_fullsized":"2023-11-24T08:13:36.648045+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:04:02.946543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636056+0000","last_change":"2023-11-24T08:13:36.632254+0000","last_active":"2023-11-24T08:13:37.636056+0000","last_peered":"2023-11-24T08:13:37.636056+0000","last_clean":"2023-11-24T08:13:37.636056+0000","last_became_active":"2023-11-24T08:13:36.631973+0000","last_became_peered":"2023-11-24T08:13:36.631973+0000","last_unstale":"2023-11-24T08:13:37.636056+0000","last_undegraded":"2023-11-24T08:13:37.636056+0000","last_fullsized":"2023-11-24T08:13:37.636056+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:33:16.942552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647787+0000","last_change":"2023-11-24T08:13:36.640033+0000","last_active":"2023-11-24T08:13:36.647787+0000","last_peered":"2023-11-24T08:13:36.647787+0000","last_clean":"2023-11-24T08:13:36.647787+0000","last_became_active":"2023-11-24T08:13:36.639786+0000","last_became_peered":"2023-11-24T08:13:36.639786+0000","last_unstale":"2023-11-24T08:13:36.647787+0000","last_undegraded":"2023-11-24T08:13:36.647787+0000","last_fullsized":"2023-11-24T08:13:36.647787+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175291+0000","last_change":"2023-11-24T08:13:36.634533+0000","last_active":"2023-11-24T08:13:38.175291+0000","last_peered":"2023-11-24T08:13:38.175291+0000","last_clean":"2023-11-24T08:13:38.175291+0000","last_became_active":"2023-11-24T08:13:36.634206+0000","last_became_peered":"2023-11-24T08:13:36.634206+0000","last_unstale":"2023-11-24T08:13:38.175291+0000","last_undegraded":"2023-11-24T08:13:38.175291+0000","last_fullsized":"2023-11-24T08:13:38.175291+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:26.054148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174485+0000","last_change":"2023-11-24T08:13:36.619182+0000","last_active":"2023-11-24T08:13:38.174485+0000","last_peered":"2023-11-24T08:13:38.174485+0000","last_clean":"2023-11-24T08:13:38.174485+0000","last_became_active":"2023-11-24T08:13:36.618900+0000","last_became_peered":"2023-11-24T08:13:36.618900+0000","last_unstale":"2023-11-24T08:13:38.174485+0000","last_undegraded":"2023-11-24T08:13:38.174485+0000","last_fullsized":"2023-11-24T08:13:38.174485+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641612+0000","last_change":"2023-11-24T08:13:36.631407+0000","last_active":"2023-11-24T08:13:36.641612+0000","last_peered":"2023-11-24T08:13:36.641612+0000","last_clean":"2023-11-24T08:13:36.641612+0000","last_became_active":"2023-11-24T08:13:36.631120+0000","last_became_peered":"2023-11-24T08:13:36.631120+0000","last_unstale":"2023-11-24T08:13:36.641612+0000","last_undegraded":"2023-11-24T08:13:36.641612+0000","last_fullsized":"2023-11-24T08:13:36.641612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:19:40.170869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644077+0000","last_change":"2023-11-24T08:13:36.631511+0000","last_active":"2023-11-24T08:13:36.644077+0000","last_peered":"2023-11-24T08:13:36.644077+0000","last_clean":"2023-11-24T08:13:36.644077+0000","last_became_active":"2023-11-24T08:13:36.631001+0000","last_became_peered":"2023-11-24T08:13:36.631001+0000","last_unstale":"2023-11-24T08:13:36.644077+0000","last_undegraded":"2023-11-24T08:13:36.644077+0000","last_fullsized":"2023-11-24T08:13:36.644077+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636545+0000","last_change":"2023-11-24T08:13:36.635293+0000","last_active":"2023-11-24T08:13:37.636545+0000","last_peered":"2023-11-24T08:13:37.636545+0000","last_clean":"2023-11-24T08:13:37.636545+0000","last_became_active":"2023-11-24T08:13:36.634297+0000","last_became_peered":"2023-11-24T08:13:36.634297+0000","last_unstale":"2023-11-24T08:13:37.636545+0000","last_undegraded":"2023-11-24T08:13:37.636545+0000","last_fullsized":"2023-11-24T08:13:37.636545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:53.189958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636762+0000","last_change":"2023-11-24T08:13:36.633291+0000","last_active":"2023-11-24T08:13:36.636762+0000","last_peered":"2023-11-24T08:13:36.636762+0000","last_clean":"2023-11-24T08:13:36.636762+0000","last_became_active":"2023-11-24T08:13:36.632857+0000","last_became_peered":"2023-11-24T08:13:36.632857+0000","last_unstale":"2023-11-24T08:13:36.636762+0000","last_undegraded":"2023-11-24T08:13:36.636762+0000","last_fullsized":"2023-11-24T08:13:36.636762+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:05.002635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175314+0000","last_change":"2023-11-24T08:13:36.640480+0000","last_active":"2023-11-24T08:13:38.175314+0000","last_peered":"2023-11-24T08:13:38.175314+0000","last_clean":"2023-11-24T08:13:38.175314+0000","last_became_active":"2023-11-24T08:13:36.639810+0000","last_became_peered":"2023-11-24T08:13:36.639810+0000","last_unstale":"2023-11-24T08:13:38.175314+0000","last_undegraded":"2023-11-24T08:13:38.175314+0000","last_fullsized":"2023-11-24T08:13:38.175314+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:56.997882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636011+0000","last_change":"2023-11-24T08:13:36.638500+0000","last_active":"2023-11-24T08:13:37.636011+0000","last_peered":"2023-11-24T08:13:37.636011+0000","last_clean":"2023-11-24T08:13:37.636011+0000","last_became_active":"2023-11-24T08:13:36.637218+0000","last_became_peered":"2023-11-24T08:13:36.637218+0000","last_unstale":"2023-11-24T08:13:37.636011+0000","last_undegraded":"2023-11-24T08:13:37.636011+0000","last_fullsized":"2023-11-24T08:13:37.636011+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:03:06.812527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635846+0000","last_change":"2023-11-24T08:13:36.630605+0000","last_active":"2023-11-24T08:13:37.635846+0000","last_peered":"2023-11-24T08:13:37.635846+0000","last_clean":"2023-11-24T08:13:37.635846+0000","last_became_active":"2023-11-24T08:13:36.630060+0000","last_became_peered":"2023-11-24T08:13:36.630060+0000","last_unstale":"2023-11-24T08:13:37.635846+0000","last_undegraded":"2023-11-24T08:13:37.635846+0000","last_fullsized":"2023-11-24T08:13:37.635846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:23.032120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646383+0000","last_change":"2023-11-24T08:13:36.634040+0000","last_active":"2023-11-24T08:13:36.646383+0000","last_peered":"2023-11-24T08:13:36.646383+0000","last_clean":"2023-11-24T08:13:36.646383+0000","last_became_active":"2023-11-24T08:13:36.630133+0000","last_became_peered":"2023-11-24T08:13:36.630133+0000","last_unstale":"2023-11-24T08:13:36.646383+0000","last_undegraded":"2023-11-24T08:13:36.646383+0000","last_fullsized":"2023-11-24T08:13:36.646383+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634603+0000","last_change":"2023-11-24T08:13:36.631917+0000","last_active":"2023-11-24T08:13:37.634603+0000","last_peered":"2023-11-24T08:13:37.634603+0000","last_clean":"2023-11-24T08:13:37.634603+0000","last_became_active":"2023-11-24T08:13:36.630938+0000","last_became_peered":"2023-11-24T08:13:36.630938+0000","last_unstale":"2023-11-24T08:13:37.634603+0000","last_undegraded":"2023-11-24T08:13:37.634603+0000","last_fullsized":"2023-11-24T08:13:37.634603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:45:04.376236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646214+0000","last_change":"2023-11-24T08:13:36.632648+0000","last_active":"2023-11-24T08:13:36.646214+0000","last_peered":"2023-11-24T08:13:36.646214+0000","last_clean":"2023-11-24T08:13:36.646214+0000","last_became_active":"2023-11-24T08:13:36.632360+0000","last_became_peered":"2023-11-24T08:13:36.632360+0000","last_unstale":"2023-11-24T08:13:36.646214+0000","last_undegraded":"2023-11-24T08:13:36.646214+0000","last_fullsized":"2023-11-24T08:13:36.646214+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:53.367338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.636421+0000","last_change":"2023-11-24T08:13:36.636739+0000","last_active":"2023-11-24T08:13:37.636421+0000","last_peered":"2023-11-24T08:13:37.636421+0000","last_clean":"2023-11-24T08:13:37.636421+0000","last_became_active":"2023-11-24T08:13:36.636430+0000","last_became_peered":"2023-11-24T08:13:36.636430+0000","last_unstale":"2023-11-24T08:13:37.636421+0000","last_undegraded":"2023-11-24T08:13:37.636421+0000","last_fullsized":"2023-11-24T08:13:37.636421+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:12.778241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636731+0000","last_change":"2023-11-24T08:13:36.628886+0000","last_active":"2023-11-24T08:13:36.636731+0000","last_peered":"2023-11-24T08:13:36.636731+0000","last_clean":"2023-11-24T08:13:36.636731+0000","last_became_active":"2023-11-24T08:13:36.628628+0000","last_became_peered":"2023-11-24T08:13:36.628628+0000","last_unstale":"2023-11-24T08:13:36.636731+0000","last_undegraded":"2023-11-24T08:13:36.636731+0000","last_fullsized":"2023-11-24T08:13:36.636731+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:32.362821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648737+0000","last_change":"2023-11-24T08:13:36.644383+0000","last_active":"2023-11-24T08:13:36.648737+0000","last_peered":"2023-11-24T08:13:36.648737+0000","last_clean":"2023-11-24T08:13:36.648737+0000","last_became_active":"2023-11-24T08:13:36.643474+0000","last_became_peered":"2023-11-24T08:13:36.643474+0000","last_unstale":"2023-11-24T08:13:36.648737+0000","last_undegraded":"2023-11-24T08:13:36.648737+0000","last_fullsized":"2023-11-24T08:13:36.648737+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:30.041085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646772+0000","last_change":"2023-11-24T08:13:36.639626+0000","last_active":"2023-11-24T08:13:36.646772+0000","last_peered":"2023-11-24T08:13:36.646772+0000","last_clean":"2023-11-24T08:13:36.646772+0000","last_became_active":"2023-11-24T08:13:36.638866+0000","last_became_peered":"2023-11-24T08:13:36.638866+0000","last_unstale":"2023-11-24T08:13:36.646772+0000","last_undegraded":"2023-11-24T08:13:36.646772+0000","last_fullsized":"2023-11-24T08:13:36.646772+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:23.142549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635192+0000","last_change":"2023-11-24T08:13:36.638103+0000","last_active":"2023-11-24T08:13:37.635192+0000","last_peered":"2023-11-24T08:13:37.635192+0000","last_clean":"2023-11-24T08:13:37.635192+0000","last_became_active":"2023-11-24T08:13:36.637795+0000","last_became_peered":"2023-11-24T08:13:36.637795+0000","last_unstale":"2023-11-24T08:13:37.635192+0000","last_undegraded":"2023-11-24T08:13:37.635192+0000","last_fullsized":"2023-11-24T08:13:37.635192+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:01.128359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633299+0000","last_change":"2023-11-24T08:13:36.625982+0000","last_active":"2023-11-24T08:13:37.633299+0000","last_peered":"2023-11-24T08:13:37.633299+0000","last_clean":"2023-11-24T08:13:37.633299+0000","last_became_active":"2023-11-24T08:13:36.625730+0000","last_became_peered":"2023-11-24T08:13:36.625730+0000","last_unstale":"2023-11-24T08:13:37.633299+0000","last_undegraded":"2023-11-24T08:13:37.633299+0000","last_fullsized":"2023-11-24T08:13:37.633299+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:11.651604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644001+0000","last_change":"2023-11-24T08:13:36.629834+0000","last_active":"2023-11-24T08:13:36.644001+0000","last_peered":"2023-11-24T08:13:36.644001+0000","last_clean":"2023-11-24T08:13:36.644001+0000","last_became_active":"2023-11-24T08:13:36.628483+0000","last_became_peered":"2023-11-24T08:13:36.628483+0000","last_unstale":"2023-11-24T08:13:36.644001+0000","last_undegraded":"2023-11-24T08:13:36.644001+0000","last_fullsized":"2023-11-24T08:13:36.644001+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:47.568775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.641359+0000","last_change":"2023-11-24T08:13:36.632468+0000","last_active":"2023-11-24T08:13:36.641359+0000","last_peered":"2023-11-24T08:13:36.641359+0000","last_clean":"2023-11-24T08:13:36.641359+0000","last_became_active":"2023-11-24T08:13:36.632155+0000","last_became_peered":"2023-11-24T08:13:36.632155+0000","last_unstale":"2023-11-24T08:13:36.641359+0000","last_undegraded":"2023-11-24T08:13:36.641359+0000","last_fullsized":"2023-11-24T08:13:36.641359+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:28.735860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643027+0000","last_change":"2023-11-24T08:13:36.640155+0000","last_active":"2023-11-24T08:13:36.643027+0000","last_peered":"2023-11-24T08:13:36.643027+0000","last_clean":"2023-11-24T08:13:36.643027+0000","last_became_active":"2023-11-24T08:13:36.639762+0000","last_became_peered":"2023-11-24T08:13:36.639762+0000","last_unstale":"2023-11-24T08:13:36.643027+0000","last_undegraded":"2023-11-24T08:13:36.643027+0000","last_fullsized":"2023-11-24T08:13:36.643027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:08.754462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"119'4","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634308+0000","last_change":"2023-11-24T08:13:35.609359+0000","last_active":"2023-11-24T08:13:37.634308+0000","last_peered":"2023-11-24T08:13:37.634308+0000","last_clean":"2023-11-24T08:13:37.634308+0000","last_became_active":"2023-11-24T08:13:35.609064+0000","last_became_peered":"2023-11-24T08:13:35.609064+0000","last_unstale":"2023-11-24T08:13:37.634308+0000","last_undegraded":"2023-11-24T08:13:37.634308+0000","last_fullsized":"2023-11-24T08:13:37.634308+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:01.055173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.625261+0000","last_change":"2023-11-24T08:13:36.623428+0000","last_active":"2023-11-24T08:13:36.625261+0000","last_peered":"2023-11-24T08:13:36.625261+0000","last_clean":"2023-11-24T08:13:36.625261+0000","last_became_active":"2023-11-24T08:13:36.618036+0000","last_became_peered":"2023-11-24T08:13:36.618036+0000","last_unstale":"2023-11-24T08:13:36.625261+0000","last_undegraded":"2023-11-24T08:13:36.625261+0000","last_fullsized":"2023-11-24T08:13:36.625261+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:04.557057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"119'1","reported_seq":13,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175052+0000","last_change":"2023-11-24T08:13:35.608591+0000","last_active":"2023-11-24T08:13:38.175052+0000","last_peered":"2023-11-24T08:13:38.175052+0000","last_clean":"2023-11-24T08:13:38.175052+0000","last_became_active":"2023-11-24T08:13:35.608178+0000","last_became_peered":"2023-11-24T08:13:35.608178+0000","last_unstale":"2023-11-24T08:13:38.175052+0000","last_undegraded":"2023-11-24T08:13:38.175052+0000","last_fullsized":"2023-11-24T08:13:38.175052+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:52.353539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.639427+0000","last_change":"2023-11-24T08:13:36.631000+0000","last_active":"2023-11-24T08:13:36.639427+0000","last_peered":"2023-11-24T08:13:36.639427+0000","last_clean":"2023-11-24T08:13:36.639427+0000","last_became_active":"2023-11-24T08:13:36.630659+0000","last_became_peered":"2023-11-24T08:13:36.630659+0000","last_unstale":"2023-11-24T08:13:36.639427+0000","last_undegraded":"2023-11-24T08:13:36.639427+0000","last_fullsized":"2023-11-24T08:13:36.639427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:27.101355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633785+0000","last_change":"2023-11-24T08:13:35.611575+0000","last_active":"2023-11-24T08:13:37.633785+0000","last_peered":"2023-11-24T08:13:37.633785+0000","last_clean":"2023-11-24T08:13:37.633785+0000","last_became_active":"2023-11-24T08:13:35.611306+0000","last_became_peered":"2023-11-24T08:13:35.611306+0000","last_unstale":"2023-11-24T08:13:37.633785+0000","last_undegraded":"2023-11-24T08:13:37.633785+0000","last_fullsized":"2023-11-24T08:13:37.633785+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:46.151145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175637+0000","last_change":"2023-11-24T08:13:36.640645+0000","last_active":"2023-11-24T08:13:38.175637+0000","last_peered":"2023-11-24T08:13:38.175637+0000","last_clean":"2023-11-24T08:13:38.175637+0000","last_became_active":"2023-11-24T08:13:36.640215+0000","last_became_peered":"2023-11-24T08:13:36.640215+0000","last_unstale":"2023-11-24T08:13:38.175637+0000","last_undegraded":"2023-11-24T08:13:38.175637+0000","last_fullsized":"2023-11-24T08:13:38.175637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:55.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633349+0000","last_change":"2023-11-24T08:13:35.612082+0000","last_active":"2023-11-24T08:13:37.633349+0000","last_peered":"2023-11-24T08:13:37.633349+0000","last_clean":"2023-11-24T08:13:37.633349+0000","last_became_active":"2023-11-24T08:13:35.611841+0000","last_became_peered":"2023-11-24T08:13:35.611841+0000","last_unstale":"2023-11-24T08:13:37.633349+0000","last_undegraded":"2023-11-24T08:13:37.633349+0000","last_fullsized":"2023-11-24T08:13:37.633349+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:04:29.687925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646772+0000","last_change":"2023-11-24T08:13:36.635274+0000","last_active":"2023-11-24T08:13:36.646772+0000","last_peered":"2023-11-24T08:13:36.646772+0000","last_clean":"2023-11-24T08:13:36.646772+0000","last_became_active":"2023-11-24T08:13:36.634918+0000","last_became_peered":"2023-11-24T08:13:36.634918+0000","last_unstale":"2023-11-24T08:13:36.646772+0000","last_undegraded":"2023-11-24T08:13:36.646772+0000","last_fullsized":"2023-11-24T08:13:36.646772+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:52:53.367338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"119'1","reported_seq":13,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633460+0000","last_change":"2023-11-24T08:13:35.611604+0000","last_active":"2023-11-24T08:13:37.633460+0000","last_peered":"2023-11-24T08:13:37.633460+0000","last_clean":"2023-11-24T08:13:37.633460+0000","last_became_active":"2023-11-24T08:13:35.611344+0000","last_became_peered":"2023-11-24T08:13:35.611344+0000","last_unstale":"2023-11-24T08:13:37.633460+0000","last_undegraded":"2023-11-24T08:13:37.633460+0000","last_fullsized":"2023-11-24T08:13:37.633460+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:47:42.061209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635399+0000","last_change":"2023-11-24T08:13:36.635465+0000","last_active":"2023-11-24T08:13:37.635399+0000","last_peered":"2023-11-24T08:13:37.635399+0000","last_clean":"2023-11-24T08:13:37.635399+0000","last_became_active":"2023-11-24T08:13:36.635145+0000","last_became_peered":"2023-11-24T08:13:36.635145+0000","last_unstale":"2023-11-24T08:13:37.635399+0000","last_undegraded":"2023-11-24T08:13:37.635399+0000","last_fullsized":"2023-11-24T08:13:37.635399+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:46:14.783732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640268+0000","last_change":"2023-11-24T08:13:35.607784+0000","last_active":"2023-11-24T08:13:36.640268+0000","last_peered":"2023-11-24T08:13:36.640268+0000","last_clean":"2023-11-24T08:13:36.640268+0000","last_became_active":"2023-11-24T08:13:35.606631+0000","last_became_peered":"2023-11-24T08:13:35.606631+0000","last_unstale":"2023-11-24T08:13:36.640268+0000","last_undegraded":"2023-11-24T08:13:36.640268+0000","last_fullsized":"2023-11-24T08:13:36.640268+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:01.576398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644095+0000","last_change":"2023-11-24T08:13:36.631702+0000","last_active":"2023-11-24T08:13:36.644095+0000","last_peered":"2023-11-24T08:13:36.644095+0000","last_clean":"2023-11-24T08:13:36.644095+0000","last_became_active":"2023-11-24T08:13:36.631279+0000","last_became_peered":"2023-11-24T08:13:36.631279+0000","last_unstale":"2023-11-24T08:13:36.644095+0000","last_undegraded":"2023-11-24T08:13:36.644095+0000","last_fullsized":"2023-11-24T08:13:36.644095+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:12.778241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634638+0000","last_change":"2023-11-24T08:13:35.603666+0000","last_active":"2023-11-24T08:13:37.634638+0000","last_peered":"2023-11-24T08:13:37.634638+0000","last_clean":"2023-11-24T08:13:37.634638+0000","last_became_active":"2023-11-24T08:13:35.603331+0000","last_became_peered":"2023-11-24T08:13:35.603331+0000","last_unstale":"2023-11-24T08:13:37.634638+0000","last_undegraded":"2023-11-24T08:13:37.634638+0000","last_fullsized":"2023-11-24T08:13:37.634638+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:03.543258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633191+0000","last_change":"2023-11-24T08:13:36.621729+0000","last_active":"2023-11-24T08:13:37.633191+0000","last_peered":"2023-11-24T08:13:37.633191+0000","last_clean":"2023-11-24T08:13:37.633191+0000","last_became_active":"2023-11-24T08:13:36.621335+0000","last_became_peered":"2023-11-24T08:13:36.621335+0000","last_unstale":"2023-11-24T08:13:37.633191+0000","last_undegraded":"2023-11-24T08:13:37.633191+0000","last_fullsized":"2023-11-24T08:13:37.633191+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:09:49.865067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.615951+0000","last_change":"2023-11-24T08:13:35.617740+0000","last_active":"2023-11-24T08:13:36.615951+0000","last_peered":"2023-11-24T08:13:36.615951+0000","last_clean":"2023-11-24T08:13:36.615951+0000","last_became_active":"2023-11-24T08:13:35.617472+0000","last_became_peered":"2023-11-24T08:13:35.617472+0000","last_unstale":"2023-11-24T08:13:36.615951+0000","last_undegraded":"2023-11-24T08:13:36.615951+0000","last_fullsized":"2023-11-24T08:13:36.615951+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:01.576398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634097+0000","last_change":"2023-11-24T08:13:36.639946+0000","last_active":"2023-11-24T08:13:37.634097+0000","last_peered":"2023-11-24T08:13:37.634097+0000","last_clean":"2023-11-24T08:13:37.634097+0000","last_became_active":"2023-11-24T08:13:36.639666+0000","last_became_peered":"2023-11-24T08:13:36.639666+0000","last_unstale":"2023-11-24T08:13:37.634097+0000","last_undegraded":"2023-11-24T08:13:37.634097+0000","last_fullsized":"2023-11-24T08:13:37.634097+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:18.995500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633594+0000","last_change":"2023-11-24T08:13:35.612067+0000","last_active":"2023-11-24T08:13:37.633594+0000","last_peered":"2023-11-24T08:13:37.633594+0000","last_clean":"2023-11-24T08:13:37.633594+0000","last_became_active":"2023-11-24T08:13:35.611792+0000","last_became_peered":"2023-11-24T08:13:35.611792+0000","last_unstale":"2023-11-24T08:13:37.633594+0000","last_undegraded":"2023-11-24T08:13:37.633594+0000","last_fullsized":"2023-11-24T08:13:37.633594+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:29.918709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640802+0000","last_change":"2023-11-24T08:13:36.639152+0000","last_active":"2023-11-24T08:13:36.640802+0000","last_peered":"2023-11-24T08:13:36.640802+0000","last_clean":"2023-11-24T08:13:36.640802+0000","last_became_active":"2023-11-24T08:13:36.638881+0000","last_became_peered":"2023-11-24T08:13:36.638881+0000","last_unstale":"2023-11-24T08:13:36.640802+0000","last_undegraded":"2023-11-24T08:13:36.640802+0000","last_fullsized":"2023-11-24T08:13:36.640802+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:47.135957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.619968+0000","last_change":"2023-11-24T08:13:35.607253+0000","last_active":"2023-11-24T08:13:36.619968+0000","last_peered":"2023-11-24T08:13:36.619968+0000","last_clean":"2023-11-24T08:13:36.619968+0000","last_became_active":"2023-11-24T08:13:35.606727+0000","last_became_peered":"2023-11-24T08:13:35.606727+0000","last_unstale":"2023-11-24T08:13:36.619968+0000","last_undegraded":"2023-11-24T08:13:36.619968+0000","last_fullsized":"2023-11-24T08:13:36.619968+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:55.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646359+0000","last_change":"2023-11-24T08:13:36.634674+0000","last_active":"2023-11-24T08:13:36.646359+0000","last_peered":"2023-11-24T08:13:36.646359+0000","last_clean":"2023-11-24T08:13:36.646359+0000","last_became_active":"2023-11-24T08:13:36.634415+0000","last_became_peered":"2023-11-24T08:13:36.634415+0000","last_unstale":"2023-11-24T08:13:36.646359+0000","last_undegraded":"2023-11-24T08:13:36.646359+0000","last_fullsized":"2023-11-24T08:13:36.646359+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"119'3","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.649297+0000","last_change":"2023-11-24T08:13:35.612998+0000","last_active":"2023-11-24T08:13:36.649297+0000","last_peered":"2023-11-24T08:13:36.649297+0000","last_clean":"2023-11-24T08:13:36.649297+0000","last_became_active":"2023-11-24T08:13:35.612719+0000","last_became_peered":"2023-11-24T08:13:35.612719+0000","last_unstale":"2023-11-24T08:13:36.649297+0000","last_undegraded":"2023-11-24T08:13:36.649297+0000","last_fullsized":"2023-11-24T08:13:36.649297+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:13.144701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633797+0000","last_change":"2023-11-24T08:13:36.629070+0000","last_active":"2023-11-24T08:13:37.633797+0000","last_peered":"2023-11-24T08:13:37.633797+0000","last_clean":"2023-11-24T08:13:37.633797+0000","last_became_active":"2023-11-24T08:13:36.628739+0000","last_became_peered":"2023-11-24T08:13:36.628739+0000","last_unstale":"2023-11-24T08:13:37.633797+0000","last_undegraded":"2023-11-24T08:13:37.633797+0000","last_fullsized":"2023-11-24T08:13:37.633797+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:12.207560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"119'6","reported_seq":17,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.650864+0000","last_change":"2023-11-24T08:13:35.617015+0000","last_active":"2023-11-24T08:13:36.650864+0000","last_peered":"2023-11-24T08:13:36.650864+0000","last_clean":"2023-11-24T08:13:36.650864+0000","last_became_active":"2023-11-24T08:13:35.616760+0000","last_became_peered":"2023-11-24T08:13:35.616760+0000","last_unstale":"2023-11-24T08:13:36.650864+0000","last_undegraded":"2023-11-24T08:13:36.650864+0000","last_fullsized":"2023-11-24T08:13:36.650864+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:22.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.626707+0000","last_change":"2023-11-24T08:13:36.624254+0000","last_active":"2023-11-24T08:13:36.626707+0000","last_peered":"2023-11-24T08:13:36.626707+0000","last_clean":"2023-11-24T08:13:36.626707+0000","last_became_active":"2023-11-24T08:13:36.623919+0000","last_became_peered":"2023-11-24T08:13:36.623919+0000","last_unstale":"2023-11-24T08:13:36.626707+0000","last_undegraded":"2023-11-24T08:13:36.626707+0000","last_fullsized":"2023-11-24T08:13:36.626707+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:06.371472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.637596+0000","last_change":"2023-11-24T08:13:35.608881+0000","last_active":"2023-11-24T08:13:36.637596+0000","last_peered":"2023-11-24T08:13:36.637596+0000","last_clean":"2023-11-24T08:13:36.637596+0000","last_became_active":"2023-11-24T08:13:35.608619+0000","last_became_peered":"2023-11-24T08:13:35.608619+0000","last_unstale":"2023-11-24T08:13:36.637596+0000","last_undegraded":"2023-11-24T08:13:36.637596+0000","last_fullsized":"2023-11-24T08:13:36.637596+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:49:33.927456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646421+0000","last_change":"2023-11-24T08:13:36.636030+0000","last_active":"2023-11-24T08:13:36.646421+0000","last_peered":"2023-11-24T08:13:36.646421+0000","last_clean":"2023-11-24T08:13:36.646421+0000","last_became_active":"2023-11-24T08:13:36.635633+0000","last_became_peered":"2023-11-24T08:13:36.635633+0000","last_unstale":"2023-11-24T08:13:36.646421+0000","last_undegraded":"2023-11-24T08:13:36.646421+0000","last_fullsized":"2023-11-24T08:13:36.646421+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:13:47.568775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.608067+0000","last_change":"2023-11-24T08:13:35.618133+0000","last_active":"2023-11-24T08:13:36.608067+0000","last_peered":"2023-11-24T08:13:36.608067+0000","last_clean":"2023-11-24T08:13:36.608067+0000","last_became_active":"2023-11-24T08:13:35.617840+0000","last_became_peered":"2023-11-24T08:13:35.617840+0000","last_unstale":"2023-11-24T08:13:36.608067+0000","last_undegraded":"2023-11-24T08:13:36.608067+0000","last_fullsized":"2023-11-24T08:13:36.608067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:24:08.409208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646716+0000","last_change":"2023-11-24T08:13:36.637680+0000","last_active":"2023-11-24T08:13:36.646716+0000","last_peered":"2023-11-24T08:13:36.646716+0000","last_clean":"2023-11-24T08:13:36.646716+0000","last_became_active":"2023-11-24T08:13:36.636949+0000","last_became_peered":"2023-11-24T08:13:36.636949+0000","last_unstale":"2023-11-24T08:13:36.646716+0000","last_undegraded":"2023-11-24T08:13:36.646716+0000","last_fullsized":"2023-11-24T08:13:36.646716+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"119'8","reported_seq":20,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.632780+0000","last_change":"2023-11-24T08:13:35.618903+0000","last_active":"2023-11-24T08:13:37.632780+0000","last_peered":"2023-11-24T08:13:37.632780+0000","last_clean":"2023-11-24T08:13:37.632780+0000","last_became_active":"2023-11-24T08:13:35.618617+0000","last_became_peered":"2023-11-24T08:13:35.618617+0000","last_unstale":"2023-11-24T08:13:37.632780+0000","last_undegraded":"2023-11-24T08:13:37.632780+0000","last_fullsized":"2023-11-24T08:13:37.632780+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:27.722061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634227+0000","last_change":"2023-11-24T08:13:36.645821+0000","last_active":"2023-11-24T08:13:37.634227+0000","last_peered":"2023-11-24T08:13:37.634227+0000","last_clean":"2023-11-24T08:13:37.634227+0000","last_became_active":"2023-11-24T08:13:36.645110+0000","last_became_peered":"2023-11-24T08:13:36.645110+0000","last_unstale":"2023-11-24T08:13:37.634227+0000","last_undegraded":"2023-11-24T08:13:37.634227+0000","last_fullsized":"2023-11-24T08:13:37.634227+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:45.975169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647704+0000","last_change":"2023-11-24T08:13:35.616059+0000","last_active":"2023-11-24T08:13:36.647704+0000","last_peered":"2023-11-24T08:13:36.647704+0000","last_clean":"2023-11-24T08:13:36.647704+0000","last_became_active":"2023-11-24T08:13:35.615801+0000","last_became_peered":"2023-11-24T08:13:35.615801+0000","last_unstale":"2023-11-24T08:13:36.647704+0000","last_undegraded":"2023-11-24T08:13:36.647704+0000","last_fullsized":"2023-11-24T08:13:36.647704+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:05.357674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.648706+0000","last_change":"2023-11-24T08:13:36.644154+0000","last_active":"2023-11-24T08:13:36.648706+0000","last_peered":"2023-11-24T08:13:36.648706+0000","last_clean":"2023-11-24T08:13:36.648706+0000","last_became_active":"2023-11-24T08:13:36.643882+0000","last_became_peered":"2023-11-24T08:13:36.643882+0000","last_unstale":"2023-11-24T08:13:36.648706+0000","last_undegraded":"2023-11-24T08:13:36.648706+0000","last_fullsized":"2023-11-24T08:13:36.648706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:04:30.701723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"119'2","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634319+0000","last_change":"2023-11-24T08:13:35.605118+0000","last_active":"2023-11-24T08:13:37.634319+0000","last_peered":"2023-11-24T08:13:37.634319+0000","last_clean":"2023-11-24T08:13:37.634319+0000","last_became_active":"2023-11-24T08:13:35.604833+0000","last_became_peered":"2023-11-24T08:13:35.604833+0000","last_unstale":"2023-11-24T08:13:37.634319+0000","last_undegraded":"2023-11-24T08:13:37.634319+0000","last_fullsized":"2023-11-24T08:13:37.634319+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:49:33.927456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175075+0000","last_change":"2023-11-24T08:13:36.643849+0000","last_active":"2023-11-24T08:13:38.175075+0000","last_peered":"2023-11-24T08:13:38.175075+0000","last_clean":"2023-11-24T08:13:38.175075+0000","last_became_active":"2023-11-24T08:13:36.643556+0000","last_became_peered":"2023-11-24T08:13:36.643556+0000","last_unstale":"2023-11-24T08:13:38.175075+0000","last_undegraded":"2023-11-24T08:13:38.175075+0000","last_fullsized":"2023-11-24T08:13:38.175075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:23.142549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.642967+0000","last_change":"2023-11-24T08:13:35.612757+0000","last_active":"2023-11-24T08:13:36.642967+0000","last_peered":"2023-11-24T08:13:36.642967+0000","last_clean":"2023-11-24T08:13:36.642967+0000","last_became_active":"2023-11-24T08:13:35.612488+0000","last_became_peered":"2023-11-24T08:13:35.612488+0000","last_unstale":"2023-11-24T08:13:36.642967+0000","last_undegraded":"2023-11-24T08:13:36.642967+0000","last_fullsized":"2023-11-24T08:13:36.642967+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:50:58.908133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175656+0000","last_change":"2023-11-24T08:13:36.630930+0000","last_active":"2023-11-24T08:13:38.175656+0000","last_peered":"2023-11-24T08:13:38.175656+0000","last_clean":"2023-11-24T08:13:38.175656+0000","last_became_active":"2023-11-24T08:13:36.630665+0000","last_became_peered":"2023-11-24T08:13:36.630665+0000","last_unstale":"2023-11-24T08:13:38.175656+0000","last_undegraded":"2023-11-24T08:13:38.175656+0000","last_fullsized":"2023-11-24T08:13:38.175656+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:16:25.987306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633713+0000","last_change":"2023-11-24T08:13:35.610547+0000","last_active":"2023-11-24T08:13:37.633713+0000","last_peered":"2023-11-24T08:13:37.633713+0000","last_clean":"2023-11-24T08:13:37.633713+0000","last_became_active":"2023-11-24T08:13:35.610005+0000","last_became_peered":"2023-11-24T08:13:35.610005+0000","last_unstale":"2023-11-24T08:13:37.633713+0000","last_undegraded":"2023-11-24T08:13:37.633713+0000","last_fullsized":"2023-11-24T08:13:37.633713+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:24.209025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174591+0000","last_change":"2023-11-24T08:13:36.633628+0000","last_active":"2023-11-24T08:13:38.174591+0000","last_peered":"2023-11-24T08:13:38.174591+0000","last_clean":"2023-11-24T08:13:38.174591+0000","last_became_active":"2023-11-24T08:13:36.633293+0000","last_became_peered":"2023-11-24T08:13:36.633293+0000","last_unstale":"2023-11-24T08:13:38.174591+0000","last_undegraded":"2023-11-24T08:13:38.174591+0000","last_fullsized":"2023-11-24T08:13:38.174591+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:42.455403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.642448+0000","last_change":"2023-11-24T08:13:35.601288+0000","last_active":"2023-11-24T08:13:36.642448+0000","last_peered":"2023-11-24T08:13:36.642448+0000","last_clean":"2023-11-24T08:13:36.642448+0000","last_became_active":"2023-11-24T08:13:35.600945+0000","last_became_peered":"2023-11-24T08:13:35.600945+0000","last_unstale":"2023-11-24T08:13:36.642448+0000","last_undegraded":"2023-11-24T08:13:36.642448+0000","last_fullsized":"2023-11-24T08:13:36.642448+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:29.918709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.644469+0000","last_change":"2023-11-24T08:13:36.631988+0000","last_active":"2023-11-24T08:13:36.644469+0000","last_peered":"2023-11-24T08:13:36.644469+0000","last_clean":"2023-11-24T08:13:36.644469+0000","last_became_active":"2023-11-24T08:13:36.631724+0000","last_became_peered":"2023-11-24T08:13:36.631724+0000","last_unstale":"2023-11-24T08:13:36.644469+0000","last_undegraded":"2023-11-24T08:13:36.644469+0000","last_fullsized":"2023-11-24T08:13:36.644469+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:51.092619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633054+0000","last_change":"2023-11-24T08:13:35.608321+0000","last_active":"2023-11-24T08:13:37.633054+0000","last_peered":"2023-11-24T08:13:37.633054+0000","last_clean":"2023-11-24T08:13:37.633054+0000","last_became_active":"2023-11-24T08:13:35.608009+0000","last_became_peered":"2023-11-24T08:13:35.608009+0000","last_unstale":"2023-11-24T08:13:37.633054+0000","last_undegraded":"2023-11-24T08:13:37.633054+0000","last_fullsized":"2023-11-24T08:13:37.633054+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:07.740664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.174586+0000","last_change":"2023-11-24T08:13:36.641630+0000","last_active":"2023-11-24T08:13:38.174586+0000","last_peered":"2023-11-24T08:13:38.174586+0000","last_clean":"2023-11-24T08:13:38.174586+0000","last_became_active":"2023-11-24T08:13:36.641223+0000","last_became_peered":"2023-11-24T08:13:36.641223+0000","last_unstale":"2023-11-24T08:13:38.174586+0000","last_undegraded":"2023-11-24T08:13:38.174586+0000","last_fullsized":"2023-11-24T08:13:38.174586+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.622918+0000","last_change":"2023-11-24T08:13:35.602215+0000","last_active":"2023-11-24T08:13:36.622918+0000","last_peered":"2023-11-24T08:13:36.622918+0000","last_clean":"2023-11-24T08:13:36.622918+0000","last_became_active":"2023-11-24T08:13:35.601939+0000","last_became_peered":"2023-11-24T08:13:35.601939+0000","last_unstale":"2023-11-24T08:13:36.622918+0000","last_undegraded":"2023-11-24T08:13:36.622918+0000","last_fullsized":"2023-11-24T08:13:36.622918+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:46.151145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646738+0000","last_change":"2023-11-24T08:13:36.637896+0000","last_active":"2023-11-24T08:13:36.646738+0000","last_peered":"2023-11-24T08:13:36.646738+0000","last_clean":"2023-11-24T08:13:36.646738+0000","last_became_active":"2023-11-24T08:13:36.637382+0000","last_became_peered":"2023-11-24T08:13:36.637382+0000","last_unstale":"2023-11-24T08:13:36.646738+0000","last_undegraded":"2023-11-24T08:13:36.646738+0000","last_fullsized":"2023-11-24T08:13:36.646738+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:24.208682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.616311+0000","last_change":"2023-11-24T08:13:35.614771+0000","last_active":"2023-11-24T08:13:36.616311+0000","last_peered":"2023-11-24T08:13:36.616311+0000","last_clean":"2023-11-24T08:13:36.616311+0000","last_became_active":"2023-11-24T08:13:35.614499+0000","last_became_peered":"2023-11-24T08:13:35.614499+0000","last_unstale":"2023-11-24T08:13:36.616311+0000","last_undegraded":"2023-11-24T08:13:36.616311+0000","last_fullsized":"2023-11-24T08:13:36.616311+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:25.040350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.640775+0000","last_change":"2023-11-24T08:13:36.638956+0000","last_active":"2023-11-24T08:13:36.640775+0000","last_peered":"2023-11-24T08:13:36.640775+0000","last_clean":"2023-11-24T08:13:36.640775+0000","last_became_active":"2023-11-24T08:13:36.638675+0000","last_became_peered":"2023-11-24T08:13:36.638675+0000","last_unstale":"2023-11-24T08:13:36.640775+0000","last_undegraded":"2023-11-24T08:13:36.640775+0000","last_fullsized":"2023-11-24T08:13:36.640775+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:58.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"119'2","reported_seq":13,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.647925+0000","last_change":"2023-11-24T08:13:35.615266+0000","last_active":"2023-11-24T08:13:36.647925+0000","last_peered":"2023-11-24T08:13:36.647925+0000","last_clean":"2023-11-24T08:13:36.647925+0000","last_became_active":"2023-11-24T08:13:35.614996+0000","last_became_peered":"2023-11-24T08:13:35.614996+0000","last_unstale":"2023-11-24T08:13:36.647925+0000","last_undegraded":"2023-11-24T08:13:36.647925+0000","last_fullsized":"2023-11-24T08:13:36.647925+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:44:25.040350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646627+0000","last_change":"2023-11-24T08:13:36.633380+0000","last_active":"2023-11-24T08:13:36.646627+0000","last_peered":"2023-11-24T08:13:36.646627+0000","last_clean":"2023-11-24T08:13:36.646627+0000","last_became_active":"2023-11-24T08:13:36.633085+0000","last_became_peered":"2023-11-24T08:13:36.633085+0000","last_unstale":"2023-11-24T08:13:36.646627+0000","last_undegraded":"2023-11-24T08:13:36.646627+0000","last_fullsized":"2023-11-24T08:13:36.646627+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:17:12.969900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.609277+0000","last_change":"2023-11-24T08:13:35.609014+0000","last_active":"2023-11-24T08:13:36.609277+0000","last_peered":"2023-11-24T08:13:36.609277+0000","last_clean":"2023-11-24T08:13:36.609277+0000","last_became_active":"2023-11-24T08:13:35.607514+0000","last_became_peered":"2023-11-24T08:13:35.607514+0000","last_unstale":"2023-11-24T08:13:36.609277+0000","last_undegraded":"2023-11-24T08:13:36.609277+0000","last_fullsized":"2023-11-24T08:13:36.609277+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:00.114560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646930+0000","last_change":"2023-11-24T08:13:36.636653+0000","last_active":"2023-11-24T08:13:36.646930+0000","last_peered":"2023-11-24T08:13:36.646930+0000","last_clean":"2023-11-24T08:13:36.646930+0000","last_became_active":"2023-11-24T08:13:36.636148+0000","last_became_peered":"2023-11-24T08:13:36.636148+0000","last_unstale":"2023-11-24T08:13:36.646930+0000","last_undegraded":"2023-11-24T08:13:36.646930+0000","last_fullsized":"2023-11-24T08:13:36.646930+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:07.664771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.620266+0000","last_change":"2023-11-24T08:13:35.617006+0000","last_active":"2023-11-24T08:13:36.620266+0000","last_peered":"2023-11-24T08:13:36.620266+0000","last_clean":"2023-11-24T08:13:36.620266+0000","last_became_active":"2023-11-24T08:13:35.616691+0000","last_became_peered":"2023-11-24T08:13:35.616691+0000","last_unstale":"2023-11-24T08:13:36.620266+0000","last_undegraded":"2023-11-24T08:13:36.620266+0000","last_fullsized":"2023-11-24T08:13:36.620266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:54.475312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.636943+0000","last_change":"2023-11-24T08:13:36.634182+0000","last_active":"2023-11-24T08:13:36.636943+0000","last_peered":"2023-11-24T08:13:36.636943+0000","last_clean":"2023-11-24T08:13:36.636943+0000","last_became_active":"2023-11-24T08:13:36.633862+0000","last_became_peered":"2023-11-24T08:13:36.633862+0000","last_unstale":"2023-11-24T08:13:36.636943+0000","last_undegraded":"2023-11-24T08:13:36.636943+0000","last_fullsized":"2023-11-24T08:13:36.636943+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:45:04.376236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.621632+0000","last_change":"2023-11-24T08:13:35.607458+0000","last_active":"2023-11-24T08:13:36.621632+0000","last_peered":"2023-11-24T08:13:36.621632+0000","last_clean":"2023-11-24T08:13:36.621632+0000","last_became_active":"2023-11-24T08:13:35.607216+0000","last_became_peered":"2023-11-24T08:13:35.607216+0000","last_unstale":"2023-11-24T08:13:36.621632+0000","last_undegraded":"2023-11-24T08:13:36.621632+0000","last_fullsized":"2023-11-24T08:13:36.621632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:13.144701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635112+0000","last_change":"2023-11-24T08:13:36.634807+0000","last_active":"2023-11-24T08:13:37.635112+0000","last_peered":"2023-11-24T08:13:37.635112+0000","last_clean":"2023-11-24T08:13:37.635112+0000","last_became_active":"2023-11-24T08:13:36.634614+0000","last_became_peered":"2023-11-24T08:13:36.634614+0000","last_unstale":"2023-11-24T08:13:37.635112+0000","last_undegraded":"2023-11-24T08:13:37.635112+0000","last_fullsized":"2023-11-24T08:13:37.635112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:50:59.921931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633550+0000","last_change":"2023-11-24T08:13:35.606491+0000","last_active":"2023-11-24T08:13:37.633550+0000","last_peered":"2023-11-24T08:13:37.633550+0000","last_clean":"2023-11-24T08:13:37.633550+0000","last_became_active":"2023-11-24T08:13:35.605798+0000","last_became_peered":"2023-11-24T08:13:35.605798+0000","last_unstale":"2023-11-24T08:13:37.633550+0000","last_undegraded":"2023-11-24T08:13:37.633550+0000","last_fullsized":"2023-11-24T08:13:37.633550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:24.974669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.635573+0000","last_change":"2023-11-24T08:13:36.641154+0000","last_active":"2023-11-24T08:13:37.635573+0000","last_peered":"2023-11-24T08:13:37.635573+0000","last_clean":"2023-11-24T08:13:37.635573+0000","last_became_active":"2023-11-24T08:13:36.640855+0000","last_became_peered":"2023-11-24T08:13:36.640855+0000","last_unstale":"2023-11-24T08:13:37.635573+0000","last_undegraded":"2023-11-24T08:13:37.635573+0000","last_fullsized":"2023-11-24T08:13:37.635573+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:18.111589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.634958+0000","last_change":"2023-11-24T08:13:35.593966+0000","last_active":"2023-11-24T08:13:37.634958+0000","last_peered":"2023-11-24T08:13:37.634958+0000","last_clean":"2023-11-24T08:13:37.634958+0000","last_became_active":"2023-11-24T08:13:35.593014+0000","last_became_peered":"2023-11-24T08:13:35.593014+0000","last_unstale":"2023-11-24T08:13:37.634958+0000","last_undegraded":"2023-11-24T08:13:37.634958+0000","last_fullsized":"2023-11-24T08:13:37.634958+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:57.719449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:38.175348+0000","last_change":"2023-11-24T08:13:36.622931+0000","last_active":"2023-11-24T08:13:38.175348+0000","last_peered":"2023-11-24T08:13:38.175348+0000","last_clean":"2023-11-24T08:13:38.175348+0000","last_became_active":"2023-11-24T08:13:36.621653+0000","last_became_peered":"2023-11-24T08:13:36.621653+0000","last_unstale":"2023-11-24T08:13:38.175348+0000","last_undegraded":"2023-11-24T08:13:38.175348+0000","last_fullsized":"2023-11-24T08:13:38.175348+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:59:14.158500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.632862+0000","last_change":"2023-11-24T08:13:35.609006+0000","last_active":"2023-11-24T08:13:37.632862+0000","last_peered":"2023-11-24T08:13:37.632862+0000","last_clean":"2023-11-24T08:13:37.632862+0000","last_became_active":"2023-11-24T08:13:35.608698+0000","last_became_peered":"2023-11-24T08:13:35.608698+0000","last_unstale":"2023-11-24T08:13:37.632862+0000","last_undegraded":"2023-11-24T08:13:37.632862+0000","last_fullsized":"2023-11-24T08:13:37.632862+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:29.027287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.646644+0000","last_change":"2023-11-24T08:13:36.634584+0000","last_active":"2023-11-24T08:13:36.646644+0000","last_peered":"2023-11-24T08:13:36.646644+0000","last_clean":"2023-11-24T08:13:36.646644+0000","last_became_active":"2023-11-24T08:13:36.634322+0000","last_became_peered":"2023-11-24T08:13:36.634322+0000","last_unstale":"2023-11-24T08:13:36.646644+0000","last_undegraded":"2023-11-24T08:13:36.646644+0000","last_fullsized":"2023-11-24T08:13:36.646644+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:53.189958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.616559+0000","last_change":"2023-11-24T08:13:35.614084+0000","last_active":"2023-11-24T08:13:36.616559+0000","last_peered":"2023-11-24T08:13:36.616559+0000","last_clean":"2023-11-24T08:13:36.616559+0000","last_became_active":"2023-11-24T08:13:35.613174+0000","last_became_peered":"2023-11-24T08:13:35.613174+0000","last_unstale":"2023-11-24T08:13:36.616559+0000","last_undegraded":"2023-11-24T08:13:36.616559+0000","last_fullsized":"2023-11-24T08:13:36.616559+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:55.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.643674+0000","last_change":"2023-11-24T08:13:36.632488+0000","last_active":"2023-11-24T08:13:36.643674+0000","last_peered":"2023-11-24T08:13:36.643674+0000","last_clean":"2023-11-24T08:13:36.643674+0000","last_became_active":"2023-11-24T08:13:36.632121+0000","last_became_peered":"2023-11-24T08:13:36.632121+0000","last_unstale":"2023-11-24T08:13:36.643674+0000","last_undegraded":"2023-11-24T08:13:36.643674+0000","last_fullsized":"2023-11-24T08:13:36.643674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:35.567653+0000","last_clean_scrub_stamp":"2023-11-24T08:13:35.567653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:17:12.969900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-11-24T08:13:37.633262+0000","last_change":"2023-11-24T08:13:35.615588+0000","last_active":"2023-11-24T08:13:37.633262+0000","last_peered":"2023-11-24T08:13:37.633262+0000","last_clean":"2023-11-24T08:13:37.633262+0000","last_became_active":"2023-11-24T08:13:35.615319+0000","last_became_peered":"2023-11-24T08:13:35.615319+0000","last_unstale":"2023-11-24T08:13:37.633262+0000","last_undegraded":"2023-11-24T08:13:37.633262+0000","last_fullsized":"2023-11-24T08:13:37.633262+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:13:34.553855+0000","last_clean_scrub_stamp":"2023-11-24T08:13:34.553855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:10.801280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-11-24T08:13:36.608467+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:13:36.608467+0000","last_peered":"2023-11-24T08:13:36.608467+0000","last_clean":"2023-11-24T08:13:36.608467+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:13:36.608467+0000","last_undegraded":"2023-11-24T08:13:36.608467+0000","last_fullsized":"2023-11-24T08:13:36.608467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33228,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338612,"statfs":{"total":96636764160,"available":96602738688,"internally_reserved":0,"allocated":2093056,"data_stored":1241374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0880000000000001,"15min":1.0289999999999999},"min":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.516,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.119,"5min":1.1279999999999999,"15min":1.0900000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.6419999999999999,"5min":2.403,"15min":2.7109999999999999},"last":0.93300000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.161,"15min":1.145},"min":{"1min":0.77600000000000002,"5min":0.629,"15min":0.46300000000000002},"max":{"1min":1.579,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.873},{"interface":"front","average":{"1min":1.323,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.81499999999999995,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.6639999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.26}]},{"osd":2,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1319999999999999,"15min":1.196},"min":{"1min":0.75,"5min":0.67000000000000004,"15min":0.58699999999999997},"max":{"1min":1.4410000000000001,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":1.774},{"interface":"front","average":{"1min":1.232,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7669999999999999,"5min":4.0650000000000004,"15min":4.633},"last":1.171}]},{"osd":3,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.79500000000000004,"5min":0.68700000000000006,"15min":0.504},"max":{"1min":1.6040000000000001,"5min":3.133,"15min":3.133},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.397,"15min":1.3740000000000001},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.706,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.0860000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.331,"15min":1.3340000000000001},"min":{"1min":0.84699999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.635,"5min":2.661,"15min":2.851},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.335},"min":{"1min":0.67700000000000005,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.7929999999999999,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.6240000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4910000000000001,"15min":1.4670000000000001},"min":{"1min":0.74099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.841,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.48,"15min":1.48},"min":{"1min":0.89100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9770000000000001,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.8029999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.556,"15min":1.546},"min":{"1min":0.77300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.0529999999999999,"5min":2.992,"15min":2.992},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4870000000000001,"15min":1.46},"min":{"1min":0.93200000000000005,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.702,"5min":3.798,"15min":3.798},"last":1.8859999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1482752,"data_stored":644160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.079,"15min":1.0900000000000001},"min":{"1min":0.875,"5min":0.621,"15min":0.621},"max":{"1min":1.6299999999999999,"5min":1.95,"15min":1.95},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0620000000000001,"15min":1.0940000000000001},"min":{"1min":0.73599999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5960000000000001,"5min":1.986,"15min":2.2069999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.149,"15min":1.2390000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.52,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.018},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.141,"15min":1.2490000000000001},"min":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6240000000000001,"5min":1.694,"15min":2.4039999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.091,"15min":1.117},"min":{"1min":0.89100000000000001,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.621,"5min":1.6779999999999999,"15min":1.9330000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.379,"5min":1.2589999999999999,"15min":1.357},"min":{"1min":0.81699999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.7749999999999999,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.55}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3260000000000001,"15min":1.385},"min":{"1min":0.96999999999999997,"5min":0.83399999999999996,"15min":0.71499999999999997},"max":{"1min":1.538,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.339,"15min":1.3460000000000001},"min":{"1min":0.98599999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.9339999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.0760000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.3879999999999999,"15min":1.5169999999999999},"min":{"1min":1.0129999999999999,"5min":0.79900000000000004,"15min":0.76500000000000001},"max":{"1min":1.756,"5min":2.895,"15min":2.895},"last":1.655},{"interface":"front","average":{"1min":1.502,"5min":1.452,"15min":1.542},"min":{"1min":1.0940000000000001,"5min":1.056,"15min":0.90800000000000003},"max":{"1min":1.8160000000000001,"5min":2.242,"15min":2.8820000000000001},"last":1.454}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4199999999999999,"15min":1.5289999999999999},"min":{"1min":0.94599999999999995,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":2.1320000000000001,"5min":2.2250000000000001,"15min":2.629},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.544,"15min":1.663},"min":{"1min":1.3340000000000001,"5min":1.101,"15min":1.022},"max":{"1min":1.8939999999999999,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.7629999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.5569999999999999,"15min":1.6919999999999999},"min":{"1min":1.274,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":2.101,"5min":2.1850000000000001,"15min":3.0329999999999999},"last":1.379},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5600000000000001,"15min":1.6639999999999999},"min":{"1min":1.151,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6890000000000001},"last":1.593}]}]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1499136,"data_stored":650994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.02,"15min":1.101},"min":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5,"5min":1.694,"15min":3.1179999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.123,"15min":1.1719999999999999},"min":{"1min":0.67700000000000005,"5min":0.65100000000000002,"15min":0.60999999999999999},"max":{"1min":1.627,"5min":1.859,"15min":2.198},"last":1.5189999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1299999999999999,"15min":1.159},"min":{"1min":0.73099999999999998,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.3799999999999999,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1759999999999999,"15min":1.1539999999999999},"min":{"1min":0.78100000000000003,"5min":0.72599999999999998,"15min":0.48699999999999999},"max":{"1min":1.5289999999999999,"5min":1.659,"15min":1.6910000000000001},"last":1.115}]},{"osd":2,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1990000000000001,"15min":1.2110000000000001},"min":{"1min":0.86899999999999999,"5min":0.65100000000000002,"15min":0.51600000000000001},"max":{"1min":1.5669999999999999,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.87},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1679999999999999,"15min":1.169},"min":{"1min":0.84999999999999998,"5min":0.69999999999999996,"15min":0.64200000000000002},"max":{"1min":1.639,"5min":1.7370000000000001,"15min":1.8129999999999999},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1459999999999999,"15min":1.1759999999999999},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.432},"max":{"1min":1.6080000000000001,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.29,"15min":1.3280000000000001},"min":{"1min":0.93799999999999994,"5min":0.79800000000000004,"15min":0.55200000000000005},"max":{"1min":1.5549999999999999,"5min":1.9730000000000001,"15min":1.99},"last":1.214}]},{"osd":4,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3200000000000001,"15min":1.333},"min":{"1min":0.79400000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.732,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.27},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":1.169,"5min":0.75600000000000001,"15min":0.46700000000000003},"max":{"1min":1.694,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.389}]},{"osd":6,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.337,"15min":1.427},"min":{"1min":0.88200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.766,"5min":2.1240000000000001,"15min":3.2949999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":1.0049999999999999,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":1.8580000000000001,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Fri Nov 24 08:13:09 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.4690000000000001,"15min":1.5700000000000001},"min":{"1min":1.105,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.869,"5min":2.129,"15min":3.4420000000000002},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.601,"15min":1.6120000000000001},"min":{"1min":1.3320000000000001,"5min":0.78900000000000003,"15min":0.71199999999999997},"max":{"1min":1.9319999999999999,"5min":2.181,"15min":2.181},"last":1.4810000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36800,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335040,"statfs":{"total":96636764160,"available":96599080960,"internally_reserved":0,"allocated":1490944,"data_stored":650390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.075,"15min":1.131},"min":{"1min":0.59999999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.0669999999999999,"15min":1.087},"min":{"1min":0.70899999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.516,"5min":1.7150000000000001,"15min":2.222},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1259999999999999,"15min":1.147},"min":{"1min":0.71499999999999997,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.593,"5min":1.6639999999999999,"15min":1.74},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.1619999999999999,"15min":1.1830000000000001},"min":{"1min":0.81699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6080000000000001,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.4339999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.054,"15min":1.1399999999999999},"min":{"1min":0.66600000000000004,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.5049999999999999,"5min":1.5780000000000001,"15min":2.1419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.181,"5min":1.2210000000000001,"15min":1.3300000000000001},"min":{"1min":0.71199999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.5580000000000001,"5min":1.8100000000000001,"15min":2.7229999999999999},"last":1.2330000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3069999999999999,"15min":1.4019999999999999},"min":{"1min":0.89000000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.756,"5min":2.121,"15min":2.9100000000000001},"last":1.528},{"interface":"front","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.3080000000000001},"min":{"1min":0.91200000000000003,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.8280000000000001},"last":1.109}]},{"osd":5,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.446,"15min":1.498},"min":{"1min":0.996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.9790000000000001,"5min":2.23,"15min":2.23},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.403,"15min":1.514},"min":{"1min":0.95999999999999996,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.782,"5min":1.8640000000000001,"15min":2.3879999999999999},"last":1.571}]},{"osd":6,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2290000000000001,"15min":1.3999999999999999},"min":{"1min":0.89200000000000002,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.6220000000000001,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5,"15min":1.625},"min":{"1min":0.92000000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9590000000000001,"5min":1.9950000000000001,"15min":3.391},"last":1.603}]},{"osd":7,"last update":"Fri Nov 24 08:13:02 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.452,"15min":1.593},"min":{"1min":1.157,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.825,"5min":2.052,"15min":3.1230000000000002},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.381,"15min":1.5489999999999999},"min":{"1min":1.113,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7929999999999999,"5min":1.857,"15min":3.4849999999999999},"last":1.484}]}]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36800,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335040,"statfs":{"total":96636764160,"available":96599080960,"internally_reserved":0,"allocated":1495040,"data_stored":645772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.55700000000000005,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.829,"5min":3.181,"15min":10.557},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.093,"5min":0.88800000000000001,"15min":0.91500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":1.3540000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.80200000000000005,"15min":0.83199999999999996},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.496,"5min":2.101,"15min":2.101},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.054,"5min":0.94399999999999995,"15min":1.0009999999999999},"min":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45100000000000001},"max":{"1min":1.5660000000000001,"5min":3.3260000000000001,"15min":13.090999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88700000000000001,"15min":0.95899999999999996},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5780000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.649,"5min":3.2480000000000002,"15min":13.460000000000001},"last":1.298}]},{"osd":4,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.72399999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.79,"5min":1.79,"15min":4.367},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.2110000000000001,"15min":1.3029999999999999},"min":{"1min":0.878,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.5720000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.262}]},{"osd":5,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.244,"15min":1.3169999999999999},"min":{"1min":0.748,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.0609999999999999,"5min":3.4399999999999999,"15min":13.231},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.1930000000000001,"15min":1.306},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.76,"5min":1.8340000000000001,"15min":5.1180000000000003},"last":1.4279999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.331,"15min":1.3979999999999999},"min":{"1min":0.95999999999999996,"5min":0.751,"15min":0.72899999999999998},"max":{"1min":1.9590000000000001,"5min":3.5030000000000001,"15min":13.359999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.242,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0150000000000001,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.0820000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.2989999999999999,"15min":1.4159999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":2.101,"15min":4.5060000000000002},"last":1.476},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3029999999999999,"15min":1.4339999999999999},"min":{"1min":0.76400000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.887,"5min":3.331,"15min":4.032},"last":0.997}]}]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1490944,"data_stored":650390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.78300000000000003,"15min":0.88900000000000001},"min":{"1min":0.66700000000000004,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.746},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.90300000000000002,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.782,"5min":1.782,"15min":2.5779999999999998},"last":1.2230000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":0.90800000000000003,"15min":1.081},"min":{"1min":0.66100000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5089999999999999,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95799999999999996,"15min":1.091},"min":{"1min":0.69499999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4399999999999999,"5min":1.788,"15min":2.6240000000000001},"last":1.0189999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84099999999999997,"15min":0.92700000000000005},"min":{"1min":0.58699999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.5309999999999999,"5min":1.552,"15min":2.2090000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0289999999999999,"15min":1.214},"min":{"1min":0.54400000000000004,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.738,"5min":1.946,"15min":2.9950000000000001},"last":1.1950000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.298,"15min":1.387},"min":{"1min":1.0029999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.883,"5min":1.893,"15min":2.7160000000000002},"last":1.405},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2490000000000001,"15min":1.371},"min":{"1min":0.96699999999999997,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6459999999999999,"5min":1.8779999999999999,"15min":3.3279999999999998},"last":1.258}]},{"osd":5,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.286,"15min":1.4159999999999999},"min":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.73399999999999999},"max":{"1min":1.9319999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2350000000000001,"15min":1.3740000000000001},"min":{"1min":0.79800000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.724,"5min":1.8759999999999999,"15min":2.7839999999999998},"last":1.3360000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.194,"15min":1.3100000000000001},"min":{"1min":0.90600000000000003,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":1.8640000000000001,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.3520000000000001,"15min":1.4830000000000001},"min":{"1min":1.129,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":2.069,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.3660000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.329,"15min":1.4730000000000001},"min":{"1min":0.93999999999999995,"5min":0.77700000000000002,"15min":0.71299999999999997},"max":{"1min":1.7669999999999999,"5min":2.1070000000000002,"15min":2.859},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2789999999999999,"15min":1.4379999999999999},"min":{"1min":0.81399999999999995,"5min":0.746,"15min":0.746},"max":{"1min":1.925,"5min":1.9930000000000001,"15min":3.3809999999999998},"last":1.1559999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1486848,"data_stored":644214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.81699999999999995,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3280000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.427,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.141}]},{"osd":2,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81999999999999995,"15min":0.73499999999999999},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":1.3140000000000001,"5min":1.861,"15min":1.861},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.91800000000000004,"15min":0.89000000000000001},"min":{"1min":0.47499999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4039999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.1930000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0109999999999999,"15min":0.96799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6599999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.046},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93200000000000005,"15min":0.89200000000000002},"min":{"1min":0.42899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.629,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.27,"15min":1.2470000000000001},"min":{"1min":0.84699999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.7450000000000001,"5min":2.028,"15min":2.7959999999999998},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.77200000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.742,"5min":2.1080000000000001,"15min":3.0409999999999999},"last":1.2430000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.268,"15min":1.218},"min":{"1min":0.81999999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6240000000000001,"5min":2.02,"15min":3.7639999999999998},"last":1.405},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.1839999999999999},"min":{"1min":0.75700000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.665,"5min":1.8759999999999999,"15min":3.286},"last":0.98199999999999998}]},{"osd":6,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.321,"15min":1.2430000000000001},"min":{"1min":0.91000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6879999999999999,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.379},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3069999999999999,"15min":1.2210000000000001},"min":{"1min":0.80800000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6970000000000001,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.3380000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:13:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3380000000000001,"15min":1.3029999999999999},"min":{"1min":0.71899999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7509999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.319},{"interface":"front","average":{"1min":1.367,"5min":1.345,"15min":1.2849999999999999},"min":{"1min":0.83399999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7190000000000001,"5min":3.653,"15min":3.653},"last":1.117}]}]},{"osd":0,"up_from":13,"seq":55834574959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37384,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334456,"statfs":{"total":96636764160,"available":96598482944,"internally_reserved":0,"allocated":2093056,"data_stored":1236182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.84299999999999997},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.27,"5min":1.425,"15min":1.6020000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.97199999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.452,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.363}]},{"osd":2,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0229999999999999,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.833,"5min":1.891,"15min":1.891},"last":1.141},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.88300000000000001,"15min":0.93899999999999995},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3120000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98499999999999999,"15min":1.0920000000000001},"min":{"1min":0.55600000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.5820000000000001,"5min":1.79,"15min":2.0670000000000002},"last":1.456},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90300000000000002,"15min":1.022},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.641,"5min":1.641,"15min":2.1789999999999998},"last":1.4079999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2350000000000001,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.619,"5min":3.7610000000000001,"15min":3.7610000000000001},"last":1.546},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0960000000000001,"15min":1.2170000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.569,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1439999999999999,"15min":1.282},"min":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.69699999999999995},"max":{"1min":1.7210000000000001,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":1.504},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1879999999999999,"15min":1.377},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5609999999999999,"5min":1.7689999999999999,"15min":2.7240000000000002},"last":1.1870000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.288,"15min":1.405},"min":{"1min":0.84599999999999997,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.792,"5min":1.986,"15min":2.4369999999999998},"last":1.583},{"interface":"front","average":{"1min":1.264,"5min":1.2609999999999999,"15min":1.4079999999999999},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.998,"5min":1.998,"15min":2.738},"last":1.6180000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:12:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.4910000000000001},"min":{"1min":0.96899999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.1349999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.4750000000000001},"min":{"1min":0.86799999999999999,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":2.0499999999999998,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.6499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:13:41.592 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:13:41.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:13:41.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:13:41.838 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:13:41.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.857+0000 7fa78a7a9640 1 Processor -- start 2023-11-24T08:13:41.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.857+0000 7fa78a7a9640 1 -- start start 2023-11-24T08:13:41.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.858+0000 7fa78a7a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 0x7fa7841681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.858+0000 7fa78a7a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78415db00 con 0x7fa78415d720 2023-11-24T08:13:41.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.858+0000 7fa78a7a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78406d1a0 con 0x7fa7840d3a40 2023-11-24T08:13:41.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.858+0000 7fa78a7a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840d0ce0 con 0x7fa7840b1ed0 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa783fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa78415d720 0x7fa7840d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41886/0 (socket says 172.21.15.107:41886) 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa783fff640 1 -- 172.21.15.107:0/1721447851 learned_addr learned my addr 172.21.15.107:0/1721447851 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa788d46640 1 --2- 172.21.15.107:0/1721447851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 0x7fa7841681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015127273 0 0) 0x7fa78415db00 con 0x7fa78415d720 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa770003520 con 0x7fa78415d720 2023-11-24T08:13:41.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa788d46640 1 -- 172.21.15.107:0/1721447851 >> v1:172.21.15.120:6789/0 conn(0x7fa7840d3a40 legacy=0x7fa78406ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448712266 0 0) 0x7fa78406d1a0 con 0x7fa7840d3a40 2023-11-24T08:13:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa788d46640 1 -- 172.21.15.107:0/1721447851 >> v1:172.21.15.107:6789/0 conn(0x7fa78415d720 legacy=0x7fa7840d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.859+0000 7fa788d46640 1 -- 172.21.15.107:0/1721447851 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78406d400 con 0x7fa7840b1ed0 2023-11-24T08:13:41.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.861+0000 7fa788d46640 1 --2- 172.21.15.107:0/1721447851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 0x7fa7841681d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009dc0 tx=0x7fa77c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e9cb8b30bd19f71 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.861+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c02cc40 con 0x7fa7840b1ed0 2023-11-24T08:13:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.861+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c02cda0 con 0x7fa7840b1ed0 2023-11-24T08:13:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.861+0000 7fa7837fe640 1 -- 172.21.15.107:0/1721447851 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c034640 con 0x7fa7840b1ed0 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1721447851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 msgr2=0x7fa7841681d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1721447851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 0x7fa7841681d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009dc0 tx=0x7fa77c02be90 comp rx=0 tx=0).stop 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1721447851 shutdown_connections 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1721447851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7840b1ed0 0x7fa7841681d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1721447851 >> 172.21.15.107:0/1721447851 conn(0x7fa78408af60 msgr2=0x7fa78409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1721447851 shutdown_connections 2023-11-24T08:13:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.862+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1721447851 wait complete. 2023-11-24T08:13:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.864+0000 7fa78a7a9640 1 Processor -- start 2023-11-24T08:13:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.864+0000 7fa78a7a9640 1 -- start start 2023-11-24T08:13:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7840b1ed0 0x7fa78415bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa78415d720 0x7fa78415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78415ef60 con 0x7fa7840d3a40 2023-11-24T08:13:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840ca4f0 con 0x7fa7840b1ed0 2023-11-24T08:13:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa78a7a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840ca7d0 con 0x7fa78415d720 2023-11-24T08:13:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33156/0 (socket says 172.21.15.107:33156) 2023-11-24T08:13:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa77bfff640 1 -- 172.21.15.107:0/1637934577 learned_addr learned my addr 172.21.15.107:0/1637934577 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa783fff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7840b1ed0 0x7fa78415bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.865+0000 7fa788d46640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa78415d720 0x7fa78415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 -- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa78415d720 msgr2=0x7fa78415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa78415d720 0x7fa78415e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 -- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7840b1ed0 msgr2=0x7fa78415bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:41.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7840b1ed0 0x7fa78415bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 -- 172.21.15.107:0/1637934577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c009a70 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.866+0000 7fa77bfff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c450 tx=0x7fa76c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.867+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0052d0 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.867+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c005430 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.867+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c012870 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.867+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7840caab0 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.867+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7840b6570 con 0x7fa7840d3a40 2023-11-24T08:13:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.868+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78404e8b0 con 0x7fa7840d3a40 2023-11-24T08:13:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.869+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fa76c0129d0 con 0x7fa7840d3a40 2023-11-24T08:13:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.870+0000 7fa7817fa640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 0x7fa76406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.870+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa76c087e00 con 0x7fa7840d3a40 2023-11-24T08:13:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.870+0000 7fa783fff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 0x7fa76406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.871+0000 7fa783fff640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 0x7fa76406e380 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa774005880 tx=0x7fa774009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:41.875+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa76c0547b0 con 0x7fa7840d3a40 2023-11-24T08:13:42.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.132+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa78415f0a0 con 0x7fa7840d3a40 2023-11-24T08:13:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.137+0000 7fa7817fa640 1 -- 172.21.15.107:0/1637934577 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa76c057dd0 con 0x7fa7840d3a40 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:13:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:13:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 msgr2=0x7fa76406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 0x7fa76406e380 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa774005880 tx=0x7fa774009040 comp rx=0 tx=0).stop 2023-11-24T08:13:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 msgr2=0x7fa7840b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c450 tx=0x7fa76c00c910 comp rx=0 tx=0).stop 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 shutdown_connections 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa76406bec0 0x7fa76406e380 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa78415d720 0x7fa78415e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7840d3a40 0x7fa7840b6030 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 --2- 172.21.15.107:0/1637934577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7840b1ed0 0x7fa78415bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 >> 172.21.15.107:0/1637934577 conn(0x7fa78408af60 msgr2=0x7fa7840734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 shutdown_connections 2023-11-24T08:13:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.141+0000 7fa78a7a9640 1 -- 172.21.15.107:0/1637934577 wait complete. 2023-11-24T08:13:42.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:13:42.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.372+0000 7f649e7c9640 1 Processor -- start 2023-11-24T08:13:42.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.372+0000 7f649e7c9640 1 -- start start 2023-11-24T08:13:42.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f649e7c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64980cc470 0x7f64980abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f649e7c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64980ce640 0x7f64980b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f649e7c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64980ce270 con 0x7f64980cf450 2023-11-24T08:13:42.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f649e7c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649806c560 con 0x7f64980cc470 2023-11-24T08:13:42.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f649e7c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649806d0a0 con 0x7f64980ce640 2023-11-24T08:13:42.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f6497fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64980cc470 0x7f64980abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f6497fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64980cc470 0x7f64980abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33170/0 (socket says 172.21.15.107:33170) 2023-11-24T08:13:42.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.374+0000 7f6497fff640 1 -- 172.21.15.107:0/902566021 learned_addr learned my addr 172.21.15.107:0/902566021 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:42.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.375+0000 7f64977fe640 1 --2- 172.21.15.107:0/902566021 >> v2:172.21.15.120:3300/0 conn(0x7f64980ce640 0x7f64980b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.375+0000 7f6497fff640 1 -- 172.21.15.107:0/902566021 >> v2:172.21.15.120:3300/0 conn(0x7f64980ce640 msgr2=0x7f64980b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.375+0000 7f6497fff640 1 --2- 172.21.15.107:0/902566021 >> v2:172.21.15.120:3300/0 conn(0x7f64980ce640 0x7f64980b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.375+0000 7f6497fff640 1 -- 172.21.15.107:0/902566021 >> v1:172.21.15.120:6789/0 conn(0x7f64980cf450 legacy=0x7f649806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.375+0000 7f6497fff640 1 -- 172.21.15.107:0/902566021 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980b4200 con 0x7f64980cc470 2023-11-24T08:13:42.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.376+0000 7f6497fff640 1 --2- 172.21.15.107:0/902566021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cc470 0x7f64980abfc0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f64980ae200 tx=0x7f648802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c663be80e3542ea1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:42.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.377+0000 7f6496ffd640 1 -- 172.21.15.107:0/902566021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488031870 con 0x7f64980cc470 2023-11-24T08:13:42.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.377+0000 7f6496ffd640 1 -- 172.21.15.107:0/902566021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64880319d0 con 0x7f64980cc470 2023-11-24T08:13:42.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.377+0000 7f6496ffd640 1 -- 172.21.15.107:0/902566021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488031d60 con 0x7f64980cc470 2023-11-24T08:13:42.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 -- 172.21.15.107:0/902566021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cc470 msgr2=0x7f64980abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 --2- 172.21.15.107:0/902566021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cc470 0x7f64980abfc0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f64980ae200 tx=0x7f648802b910 comp rx=0 tx=0).stop 2023-11-24T08:13:42.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 -- 172.21.15.107:0/902566021 shutdown_connections 2023-11-24T08:13:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 --2- 172.21.15.107:0/902566021 >> v2:172.21.15.120:3300/0 conn(0x7f64980ce640 0x7f64980b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 --2- 172.21.15.107:0/902566021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cc470 0x7f64980abfc0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 -- 172.21.15.107:0/902566021 >> 172.21.15.107:0/902566021 conn(0x7f649806a920 msgr2=0x7f649806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 -- 172.21.15.107:0/902566021 shutdown_connections 2023-11-24T08:13:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.378+0000 7f649e7c9640 1 -- 172.21.15.107:0/902566021 wait complete. 2023-11-24T08:13:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.380+0000 7f649e7c9640 1 Processor -- start 2023-11-24T08:13:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.380+0000 7f649e7c9640 1 -- start start 2023-11-24T08:13:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.381+0000 7f649e7c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64980cc470 0x7f64980714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649e7c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 0x7f64981eb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649e7c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649e7c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981f7870 con 0x7f64980cf450 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649e7c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498157500 con 0x7f64980ce640 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649e7c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64981f7ac0 con 0x7f64980cc470 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33174/0 (socket says 172.21.15.107:33174) 2023-11-24T08:13:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 -- 172.21.15.107:0/1119954855 learned_addr learned my addr 172.21.15.107:0/1119954855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:42.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f6497fff640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64980cc470 0x7f64980714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 -- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64980cc470 msgr2=0x7f64980714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f64977fe640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 0x7f64981eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64980cc470 0x7f64980714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 -- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 msgr2=0x7f64981eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 0x7f64981eb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f649cd66640 1 -- 172.21.15.107:0/1119954855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880094d0 con 0x7f64980cf450 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.382+0000 7f64977fe640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 0x7f64981eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:42.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f649cd66640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f649000c780 tx=0x7f649000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:42.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490003d80 con 0x7f64980cf450 2023-11-24T08:13:42.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649000ce30 con 0x7f64980cf450 2023-11-24T08:13:42.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490005290 con 0x7f64980cf450 2023-11-24T08:13:42.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64981f7da0 con 0x7f64980cf450 2023-11-24T08:13:42.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.383+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6498157190 con 0x7f64980cf450 2023-11-24T08:13:42.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.384+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64981f82c0 con 0x7f64980cf450 2023-11-24T08:13:42.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.385+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f64900053f0 con 0x7f64980cf450 2023-11-24T08:13:42.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.386+0000 7f6494ff9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 0x7f648406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:42.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.386+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64900878f0 con 0x7f64980cf450 2023-11-24T08:13:42.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.386+0000 7f6497fff640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 0x7f648406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:42.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.387+0000 7f6497fff640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 0x7f648406e350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6488036000 tx=0x7f648803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:42.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.391+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64981f82c0 con 0x7f64980cf450 2023-11-24T08:13:42.563 INFO:tasks.ceph.mds.d.smithi120.stderr:2023-11-24T08:13:42.562+0000 7f447e673640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:13:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.622+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64981f82c0 con 0x7f64980cf450 2023-11-24T08:13:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.623+0000 7f6494ff9640 1 -- 172.21.15.107:0/1119954855 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f64981f82c0 con 0x7f64980cf450 2023-11-24T08:13:42.625 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:13:42.626 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:13:42.626 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 69 2023-11-24T08:13:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.625+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 msgr2=0x7f648406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 0x7f648406e350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6488036000 tx=0x7f648803c040 comp rx=0 tx=0).stop 2023-11-24T08:13:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 msgr2=0x7f64981eba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f649000c780 tx=0x7f649000cc40 comp rx=0 tx=0).stop 2023-11-24T08:13:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 shutdown_connections 2023-11-24T08:13:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f648406be90 0x7f648406e350 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64980cf450 0x7f64981eba70 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64980ce640 0x7f64981eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 --2- 172.21.15.107:0/1119954855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64980cc470 0x7f64980714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 >> 172.21.15.107:0/1119954855 conn(0x7f649806a920 msgr2=0x7f649806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 shutdown_connections 2023-11-24T08:13:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:42.626+0000 7f649e7c9640 1 -- 172.21.15.107:0/1119954855 wait complete. 2023-11-24T08:13:43.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:13:36.044229+0000', 'modified': '2023-11-24T08:13:39.130128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8304}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8304': {'gid': 8304, 'name': 'd', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6835/74799278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6833', 'nonce': 74799278}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 74799278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:13:43.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:13:43.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-24T08:13:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.274+0000 7fd591890640 1 Processor -- start 2023-11-24T08:13:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.275+0000 7fd591890640 1 -- start start 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.276+0000 7fd591890640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd58c0cc0f0 0x7fd58c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.276+0000 7fd591890640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd58c0c57e0 0x7fd58c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.276+0000 7fd591890640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58c15eea0 con 0x7fd58c15b440 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.276+0000 7fd591890640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c15b820 con 0x7fd58c0c57e0 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.276+0000 7fd591890640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c0abf30 con 0x7fd58c0cc0f0 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd58c15b440 0x7fd58c0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41898/0 (socket says 172.21.15.107:41898) 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58affd640 1 -- 172.21.15.107:0/258440157 learned_addr learned my addr 172.21.15.107:0/258440157 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/258440157 >> v2:172.21.15.107:3300/0 conn(0x7fd58c0c57e0 0x7fd58c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/258440157 >> v2:172.21.15.120:3300/0 conn(0x7fd58c0cc0f0 0x7fd58c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58b7fe640 1 -- 172.21.15.107:0/258440157 >> v2:172.21.15.120:3300/0 conn(0x7fd58c0cc0f0 msgr2=0x7fd58c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/258440157 >> v2:172.21.15.120:3300/0 conn(0x7fd58c0cc0f0 0x7fd58c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58b7fe640 1 -- 172.21.15.107:0/258440157 >> v1:172.21.15.107:6789/0 conn(0x7fd58c15b440 legacy=0x7fd58c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.277+0000 7fd58b7fe640 1 -- 172.21.15.107:0/258440157 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c15f0f0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.279+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/258440157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c06c5f0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06e720 tx=0x7fd58402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416439a6b07a274d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.279+0000 7fd589ffb640 1 -- 172.21.15.107:0/258440157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58402e070 con 0x7fd58c0c57e0 2023-11-24T08:13:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.279+0000 7fd589ffb640 1 -- 172.21.15.107:0/258440157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5840315a0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.279+0000 7fd589ffb640 1 -- 172.21.15.107:0/258440157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584031930 con 0x7fd58c0c57e0 2023-11-24T08:13:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 -- 172.21.15.107:0/258440157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 msgr2=0x7fd58c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 --2- 172.21.15.107:0/258440157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c06c5f0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06e720 tx=0x7fd58402b840 comp rx=0 tx=0).stop 2023-11-24T08:13:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 -- 172.21.15.107:0/258440157 shutdown_connections 2023-11-24T08:13:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 --2- 172.21.15.107:0/258440157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c06c5f0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 --2- 172.21.15.107:0/258440157 >> v2:172.21.15.120:3300/0 conn(0x7fd58c0cc0f0 0x7fd58c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 -- 172.21.15.107:0/258440157 >> 172.21.15.107:0/258440157 conn(0x7fd58c0b0c50 msgr2=0x7fd58c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 -- 172.21.15.107:0/258440157 shutdown_connections 2023-11-24T08:13:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.280+0000 7fd591890640 1 -- 172.21.15.107:0/258440157 wait complete. 2023-11-24T08:13:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.282+0000 7fd591890640 1 Processor -- start 2023-11-24T08:13:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.282+0000 7fd591890640 1 -- start start 2023-11-24T08:13:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.283+0000 7fd591890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.283+0000 7fd591890640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 0x7fd58c0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd591890640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58c15b440 0x7fd58c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd591890640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c155aa0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd591890640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c1554b0 con 0x7fd58c0cc0f0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd591890640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c15a7f0 con 0x7fd58c15b440 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33202/0 (socket says 172.21.15.107:33202) 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 -- 172.21.15.107:0/3407861833 learned_addr learned my addr 172.21.15.107:0/3407861833 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 0x7fd58c0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58c15b440 0x7fd58c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 -- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58c15b440 msgr2=0x7fd58c159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58c15b440 0x7fd58c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 -- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 msgr2=0x7fd58c0afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 0x7fd58c0afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58affd640 1 -- 172.21.15.107:0/3407861833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584009400 con 0x7fd58c0c57e0 2023-11-24T08:13:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.284+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 0x7fd58c0afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd58affd640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c480 tx=0x7fd57c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd583fff640 1 -- 172.21.15.107:0/3407861833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c005ce0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd583fff640 1 -- 172.21.15.107:0/3407861833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c005e40 con 0x7fd58c0c57e0 2023-11-24T08:13:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c15aad0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd583fff640 1 -- 172.21.15.107:0/3407861833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0072f0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.285+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c156400 con 0x7fd58c0c57e0 2023-11-24T08:13:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.287+0000 7fd583fff640 1 -- 172.21.15.107:0/3407861833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fd57c007510 con 0x7fd58c0c57e0 2023-11-24T08:13:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.289+0000 7fd583fff640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 0x7fd57006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.289+0000 7fd583fff640 1 -- 172.21.15.107:0/3407861833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd57c0878d0 con 0x7fd58c0c57e0 2023-11-24T08:13:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.289+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 0x7fd57006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.290+0000 7fd594427740 1 Processor -- start 2023-11-24T08:13:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.290+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 0x7fd57006e350 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0b4d10 tx=0x7fd57400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.290+0000 7fd594427740 1 -- start start 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e6ea0 0x5649719e7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x5649719e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x564971945880 con 0x5649719e65e0 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564971954ae0 con 0x5649719e1ba0 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649719521d0 con 0x5649719e6ea0 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33204/0 (socket says 172.21.15.107:33204) 2023-11-24T08:13:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.291+0000 7fd58b7fe640 1 -- 172.21.15.107:0/1350233675 learned_addr learned my addr 172.21.15.107:0/1350233675 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 -- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e6ea0 msgr2=0x5649719e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e6ea0 0x5649719e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 -- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 msgr2=0x5649719e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x5649719e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 -- 172.21.15.107:0/1350233675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564971915860 con 0x5649719e65e0 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd584009860 tx=0x7fd584008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aaf5b81f2876a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd5817fa640 1 -- 172.21.15.107:0/1350233675 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5840318d0 con 0x5649719e65e0 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd5817fa640 1 -- 172.21.15.107:0/1350233675 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584031a30 con 0x5649719e65e0 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.292+0000 7fd5817fa640 1 -- 172.21.15.107:0/1350233675 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584031da0 con 0x5649719e65e0 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 -- 172.21.15.107:0/1350233675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 msgr2=0x564971a2a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd584009860 tx=0x7fd584008030 comp rx=0 tx=0).stop 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 -- 172.21.15.107:0/1350233675 shutdown_connections 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e65e0 0x564971a2a560 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x5649719e60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 --2- 172.21.15.107:0/1350233675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e6ea0 0x5649719e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 -- 172.21.15.107:0/1350233675 >> 172.21.15.107:0/1350233675 conn(0x5649719a8c70 msgr2=0x564971a2c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 -- 172.21.15.107:0/1350233675 shutdown_connections 2023-11-24T08:13:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.293+0000 7fd594427740 1 -- 172.21.15.107:0/1350233675 wait complete. 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.294+0000 7fd594427740 1 -- start start 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.294+0000 7fd594427740 1 Processor -- start 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a11310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a12f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497194dac0 con 0x5649719e1ba0 2023-11-24T08:13:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649718ea2f0 con 0x5649719e6ea0 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564971907c70 con 0x5649719e65e0 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd594427740 10 client.0 _ll_register_callbacks cb 0x56497190c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33212/0 (socket says 172.21.15.107:33212) 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 -- 172.21.15.107:0/3919364387 learned_addr learned my addr 172.21.15.107:0/3919364387 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 -- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 msgr2=0x564971a11310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a11310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 -- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 msgr2=0x564971a12f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a12f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.295+0000 7fd58affd640 1 -- 172.21.15.107:0/3919364387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c15f0f0 con 0x5649719e1ba0 2023-11-24T08:13:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd58affd640 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c450 tx=0x7fd57c057f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3ccfce1dee3efa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd57a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0600d0 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd57c05fc40 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564971946aa0 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd57c05fe20 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd594427740 10 client.8448 fetch_fsmap learned FSMap version 69 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x56497194c140 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.296+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564971946aa0 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 69) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd57c053c70 con 0x5649719e1ba0 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 10 client.8448 fetch_fsmap finished waiting for FSMap version 69 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 10 client.8448 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T08:13:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 20 client.8448 populate_metadata read hostname 'smithi107' 2023-11-24T08:13:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a13550 0x564971a0f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 4 client.8448 mds_command: new command op to 8304 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:13:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 --> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649719467c0 con 0x564971a13550 2023-11-24T08:13:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.297+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a13550 0x564971a0f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.298+0000 7fd57a7fc640 10 client.8448 ms_handle_connect on v2:172.21.15.120:6833/74799278 2023-11-24T08:13:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.298+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a13550 0x564971a0f2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8d61ad1dd9ff114 server_cookie=86a38eb99d5283c5 in_seq=0 out_seq=0 2023-11-24T08:13:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd57a7fc640 1 -- 172.21.15.107:0/3919364387 <== mds.0 v2:172.21.15.120:6833/74799278 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5649719467c0 con 0x564971a13550 2023-11-24T08:13:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd57a7fc640 10 client.8448 handle_command_reply: tid=0 2023-11-24T08:13:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 client.8448 shutdown 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 msgr2=0x564971a10dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c450 tx=0x7fd57c057f40 comp rx=0 tx=0).stop 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 shutdown_connections 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a13550 0x564971a0f2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a12f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a11310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 --2- 172.21.15.107:0/3919364387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a10dd0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd57a7fc640 0 client.8448 ms_handle_reset on v2:172.21.15.120:6833/74799278 2023-11-24T08:13:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 >> 172.21.15.107:0/3919364387 conn(0x5649719a8c70 msgr2=0x5649719e0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.301+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 shutdown_connections 2023-11-24T08:13:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.302+0000 7fd594427740 1 -- 172.21.15.107:0/3919364387 wait complete. 2023-11-24T08:13:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.302+0000 7fd594427740 20 client.8448 trim_cache size 0 max 16384 2023-11-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.318+0000 7fd594427740 1 Processor -- start 2023-11-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 -- start start 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x564971a12f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x5649719e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649716b0f20 con 0x5649719e6ea0 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649719e1870 con 0x5649719e1ba0 2023-11-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.319+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564971a11900 con 0x5649719e65e0 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33228/0 (socket says 172.21.15.107:33228) 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 -- 172.21.15.107:0/2420676555 learned_addr learned my addr 172.21.15.107:0/2420676555 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x5649719e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58affd640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x564971a12f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 -- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 msgr2=0x5649719e0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x5649719e0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 -- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 msgr2=0x564971a12f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x564971a12f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58b7fe640 1 -- 172.21.15.107:0/2420676555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649718e9b80 con 0x5649719e6ea0 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x5649719e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.320+0000 7fd58affd640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x564971a12f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd58b7fe640 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fd584000c00 tx=0x7fd58402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45fc892a755b80e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd581ffb640 1 -- 172.21.15.107:0/2420676555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584031da0 con 0x5649719e6ea0 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd581ffb640 1 -- 172.21.15.107:0/2420676555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584031f00 con 0x5649719e6ea0 2023-11-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd581ffb640 1 -- 172.21.15.107:0/2420676555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584004240 con 0x5649719e6ea0 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 -- 172.21.15.107:0/2420676555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 msgr2=0x564971a3e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fd584000c00 tx=0x7fd58402e000 comp rx=0 tx=0).stop 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 -- 172.21.15.107:0/2420676555 shutdown_connections 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e6ea0 0x564971a3e580 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x5649719e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 --2- 172.21.15.107:0/2420676555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e1ba0 0x564971a12f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.321+0000 7fd594427740 1 -- 172.21.15.107:0/2420676555 >> 172.21.15.107:0/2420676555 conn(0x5649719a8c70 msgr2=0x564971a30e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.322+0000 7fd594427740 1 -- 172.21.15.107:0/2420676555 shutdown_connections 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.322+0000 7fd594427740 1 -- 172.21.15.107:0/2420676555 wait complete. 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.322+0000 7fd594427740 1 -- start start 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.322+0000 7fd594427740 1 Processor -- start 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a3ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a45ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649719e7f50 con 0x5649719e1ba0 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564971a3e7b0 con 0x5649719e6ea0 2023-11-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564971a0ed50 con 0x5649719e65e0 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.323+0000 7fd594427740 10 client.0 _ll_register_callbacks cb 0x7fd57c0541c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33230/0 (socket says 172.21.15.107:33230) 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 -- 172.21.15.107:0/1486624685 learned_addr learned my addr 172.21.15.107:0/1486624685 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a3ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 -- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 msgr2=0x564971a3ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a3ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 -- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 msgr2=0x564971a45ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a45ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 -- 172.21.15.107:0/1486624685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564971915860 con 0x5649719e1ba0 2023-11-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd58affd640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fd57c054330 tx=0x7fd57c0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f35e9af7c4fdb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd57a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.324+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c001160 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd57c0600d0 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd584031a30 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd57c06bb20 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 10 client.8454 fetch_fsmap learned FSMap version 69 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x56497190f170 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c07a070 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 69) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd57c06bc80 con 0x5649719e1ba0 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 10 client.8454 fetch_fsmap finished waiting for FSMap version 69 2023-11-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 10 client.8454 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T08:13:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 20 client.8454 populate_metadata read hostname 'smithi107' 2023-11-24T08:13:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a0c580 0x564971a461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 4 client.8454 mds_command: new command op to 8304 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:13:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.325+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 --> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56497164e6c0 con 0x564971a0c580 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.326+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a0c580 0x564971a461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.326+0000 7fd57a7fc640 10 client.8454 ms_handle_connect on v2:172.21.15.120:6833/74799278 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.326+0000 7fd58a7fc640 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a0c580 0x564971a461e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6611619ce5fafdad server_cookie=1361d8e92c4843e2 in_seq=0 out_seq=0 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd57a7fc640 1 -- 172.21.15.107:0/1486624685 <== mds.0 v2:172.21.15.120:6833/74799278 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56497164e6c0 con 0x564971a0c580 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd57a7fc640 10 client.8454 handle_command_reply: tid=0 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 client.8454 shutdown 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 msgr2=0x564971a3e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fd57c054330 tx=0x7fd57c0077c0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 shutdown_connections 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:6833/74799278,v1:172.21.15.120:6835/74799278] conn(0x564971a0c580 0x564971a461e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649719e6ea0 0x564971a45ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649719e65e0 0x564971a3ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd57a7fc640 0 client.8454 ms_handle_reset on v2:172.21.15.120:6833/74799278 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 --2- 172.21.15.107:0/1486624685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649719e1ba0 0x564971a3e920 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.327+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 >> 172.21.15.107:0/1486624685 conn(0x5649719a8c70 msgr2=0x564971a2c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 shutdown_connections 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd594427740 1 -- 172.21.15.107:0/1486624685 wait complete. 2023-11-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd594427740 20 client.8454 trim_cache size 0 max 16384 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 8304, 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.7253437219999999, 2023-11-24T08:13:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 69, 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 120, 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 120, 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.136042719000001 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 msgr2=0x7fd57006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 0x7fd57006e350 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0b4d10 tx=0x7fd57400b040 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 msgr2=0x7fd58c155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c480 tx=0x7fd57c00c940 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 shutdown_connections 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd57006be90 0x7fd57006e350 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58c15b440 0x7fd58c159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd58c0cc0f0 0x7fd58c0afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 --2- 172.21.15.107:0/3407861833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd58c0c57e0 0x7fd58c155ec0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.328+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 >> 172.21.15.107:0/3407861833 conn(0x7fd58c0b0c50 msgr2=0x7fd58c0c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.329+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 shutdown_connections 2023-11-24T08:13:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.329+0000 7fd591890640 1 -- 172.21.15.107:0/3407861833 wait complete. 2023-11-24T08:13:43.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:13:43.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.556+0000 7fb8fd890640 1 Processor -- start 2023-11-24T08:13:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.556+0000 7fb8fd890640 1 -- start start 2023-11-24T08:13:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8fd890640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f815df80 0x7fb8f80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8fd890640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8f80cd470 0x7fb8f815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8fd890640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f815e3f0 con 0x7fb8f80b2250 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8fd890640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f815a990 con 0x7fb8f815df80 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8fd890640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f815ab80 con 0x7fb8f80cd470 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8ee7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f815df80 0x7fb8f80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8ee7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f815df80 0x7fb8f80ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33238/0 (socket says 172.21.15.107:33238) 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.558+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/1283266676 learned_addr learned my addr 172.21.15.107:0/1283266676 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.559+0000 7fb8f77fe640 1 --2- 172.21.15.107:0/1283266676 >> v2:172.21.15.120:3300/0 conn(0x7fb8f80cd470 0x7fb8f815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.559+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/1283266676 >> v2:172.21.15.120:3300/0 conn(0x7fb8f80cd470 msgr2=0x7fb8f815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.559+0000 7fb8ee7fc640 1 --2- 172.21.15.107:0/1283266676 >> v2:172.21.15.120:3300/0 conn(0x7fb8f80cd470 0x7fb8f815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.559+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/1283266676 >> v1:172.21.15.120:6789/0 conn(0x7fb8f80b2250 legacy=0x7fb8f80b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.559+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/1283266676 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f80b2d40 con 0x7fb8f815df80 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.560+0000 7fb8ee7fc640 1 --2- 172.21.15.107:0/1283266676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f80ccf30 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80b2fa0 tx=0x7fb8e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901417da6e0613fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.560+0000 7fb8f67fc640 1 -- 172.21.15.107:0/1283266676 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0031870 con 0x7fb8f815df80 2023-11-24T08:13:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.560+0000 7fb8f67fc640 1 -- 172.21.15.107:0/1283266676 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e00319d0 con 0x7fb8f815df80 2023-11-24T08:13:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.560+0000 7fb8f67fc640 1 -- 172.21.15.107:0/1283266676 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0031d60 con 0x7fb8f815df80 2023-11-24T08:13:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 -- 172.21.15.107:0/1283266676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 msgr2=0x7fb8f80ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1283266676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f80ccf30 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80b2fa0 tx=0x7fb8e002b910 comp rx=0 tx=0).stop 2023-11-24T08:13:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 -- 172.21.15.107:0/1283266676 shutdown_connections 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1283266676 >> v2:172.21.15.120:3300/0 conn(0x7fb8f80cd470 0x7fb8f815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1283266676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f80ccf30 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 -- 172.21.15.107:0/1283266676 >> 172.21.15.107:0/1283266676 conn(0x7fb8f806a920 msgr2=0x7fb8f806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 -- 172.21.15.107:0/1283266676 shutdown_connections 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.561+0000 7fb8fd890640 1 -- 172.21.15.107:0/1283266676 wait complete. 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.563+0000 7fb8fd890640 1 Processor -- start 2023-11-24T08:13:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 -- start start 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 0x7fb8f8153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f80cd470 0x7fb8f8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f814e6e0 con 0x7fb8f815df80 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.564+0000 7fb8fd890640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80afd40 con 0x7fb8f80b2250 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8fd890640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80b0020 con 0x7fb8f80cd470 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33254/0 (socket says 172.21.15.107:33254) 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 -- 172.21.15.107:0/1342340695 learned_addr learned my addr 172.21.15.107:0/1342340695 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f6ffd640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 0x7fb8f8153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 -- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f80cd470 msgr2=0x7fb8f8155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f80cd470 0x7fb8f8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 -- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 msgr2=0x7fb8f8153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 0x7fb8f8153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f77fe640 1 -- 172.21.15.107:0/1342340695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e00094d0 con 0x7fb8f815df80 2023-11-24T08:13:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.565+0000 7fb8f6ffd640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 0x7fb8f8153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8f77fe640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007d90 tx=0x7fb8f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0014420 con 0x7fb8f815df80 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f00145a0 con 0x7fb8f815df80 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0014950 con 0x7fb8f815df80 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f8153730 con 0x7fb8f815df80 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.566+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f814db80 con 0x7fb8f815df80 2023-11-24T08:13:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.567+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f8153590 con 0x7fb8f815df80 2023-11-24T08:13:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.568+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb8f0014ab0 con 0x7fb8f815df80 2023-11-24T08:13:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.569+0000 7fb8f4ff9640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 0x7fb8d806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.569+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8f0088da0 con 0x7fb8f815df80 2023-11-24T08:13:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.569+0000 7fb8f6ffd640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 0x7fb8d806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.570+0000 7fb8f6ffd640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 0x7fb8d806e350 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8003eb0 tx=0x7fb8e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:43.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.574+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8f8153590 con 0x7fb8f815df80 2023-11-24T08:13:43.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.807+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb8f8153c50 con 0x7fb8f815df80 2023-11-24T08:13:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.808+0000 7fb8f4ff9640 1 -- 172.21.15.107:0/1342340695 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v69) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8f8153c50 con 0x7fb8f815df80 2023-11-24T08:13:43.810 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:13:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 msgr2=0x7fb8d806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 0x7fb8d806e350 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8003eb0 tx=0x7fb8e8009040 comp rx=0 tx=0).stop 2023-11-24T08:13:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 msgr2=0x7fb8f814d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007d90 tx=0x7fb8f000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 shutdown_connections 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d806be90 0x7fb8d806e350 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f815df80 0x7fb8f814d640 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f80cd470 0x7fb8f8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 --2- 172.21.15.107:0/1342340695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f80b2250 0x7fb8f8153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 >> 172.21.15.107:0/1342340695 conn(0x7fb8f806a920 msgr2=0x7fb8f81585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 shutdown_connections 2023-11-24T08:13:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:43.811+0000 7fb8fd890640 1 -- 172.21.15.107:0/1342340695 wait complete. 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:13:43.833 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:43.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:43.834 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:13:43.886 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:13:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:13:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:13:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41846sec preferred_lft 41846sec 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.888 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:13:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:13:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:43.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:43.889 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:13:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:13:43.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:43.986 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:13:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:13:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:13:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:13:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:13:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:13:44.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:13:44.039 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:13:44.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.224 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:13:44.274 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:13:44.275 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:13:44.276 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:44.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:13:44.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:44.378 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:44.431 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:13:44.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:44.431 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:44.483 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:13:44.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:44.626 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:44.626 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:13:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:13:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:13:44.663 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:13:44.663 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:13:44.663 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:13:44.663 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:13:44.663 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:13:44.667 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:13:44.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:13:44.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:44.744 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:13:44.813 INFO:teuthology.orchestra.run.smithi107.stdout:client6184 2023-11-24T08:13:44.813 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:13:44.813 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:44.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:13:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.118+0000 7fb576f90640 1 Processor -- start 2023-11-24T08:13:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.118+0000 7fb576f90640 1 -- start start 2023-11-24T08:13:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.119+0000 7fb576f90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5700cf450 0x7fb5700cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb576f90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb5700cdfb0 0x7fb5700ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb576f90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5700cece0 con 0x7fb57015c7f0 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb576f90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700cfe40 con 0x7fb5700cf450 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb576f90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb570156010 con 0x7fb5700cdfb0 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5700cf450 0x7fb5700cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5700cf450 0x7fb5700cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33270/0 (socket says 172.21.15.107:33270) 2023-11-24T08:13:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 -- 172.21.15.107:0/1613220277 learned_addr learned my addr 172.21.15.107:0/1613220277 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 -- 172.21.15.107:0/1613220277 >> v2:172.21.15.120:3300/0 conn(0x7fb5700cdfb0 msgr2=0x7fb5700ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 --2- 172.21.15.107:0/1613220277 >> v2:172.21.15.120:3300/0 conn(0x7fb5700cdfb0 0x7fb5700ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 -- 172.21.15.107:0/1613220277 >> v1:172.21.15.120:6789/0 conn(0x7fb57015c7f0 legacy=0x7fb57006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.120+0000 7fb56ffff640 1 -- 172.21.15.107:0/1613220277 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57006d040 con 0x7fb5700cf450 2023-11-24T08:13:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.121+0000 7fb56ffff640 1 --2- 172.21.15.107:0/1613220277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700cf900 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fb5700d00e0 tx=0x7fb55c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2143d27fd252bcc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1613220277 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c031870 con 0x7fb5700cf450 2023-11-24T08:13:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1613220277 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0319d0 con 0x7fb5700cf450 2023-11-24T08:13:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1613220277 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c031d60 con 0x7fb5700cf450 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 -- 172.21.15.107:0/1613220277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 msgr2=0x7fb5700cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 --2- 172.21.15.107:0/1613220277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700cf900 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fb5700d00e0 tx=0x7fb55c02b910 comp rx=0 tx=0).stop 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 -- 172.21.15.107:0/1613220277 shutdown_connections 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 --2- 172.21.15.107:0/1613220277 >> v2:172.21.15.120:3300/0 conn(0x7fb5700cdfb0 0x7fb5700ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 --2- 172.21.15.107:0/1613220277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700cf900 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 -- 172.21.15.107:0/1613220277 >> 172.21.15.107:0/1613220277 conn(0x7fb57006a920 msgr2=0x7fb57006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.122+0000 7fb576f90640 1 -- 172.21.15.107:0/1613220277 shutdown_connections 2023-11-24T08:13:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.123+0000 7fb576f90640 1 -- 172.21.15.107:0/1613220277 wait complete. 2023-11-24T08:13:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.124+0000 7fb576f90640 1 Processor -- start 2023-11-24T08:13:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.124+0000 7fb576f90640 1 -- start start 2023-11-24T08:13:45.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 0x7fb570153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb57015c7f0 0x7fb5700c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700b2c90 con 0x7fb5700cf450 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700b20e0 con 0x7fb5700cdfb0 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.125+0000 7fb576f90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb570153bc0 con 0x7fb57015c7f0 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33276/0 (socket says 172.21.15.107:33276) 2023-11-24T08:13:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 -- 172.21.15.107:0/534830640 learned_addr learned my addr 172.21.15.107:0/534830640 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 -- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb57015c7f0 msgr2=0x7fb5700c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb57015c7f0 0x7fb5700c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 -- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 msgr2=0x7fb570153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb574d2c640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 0x7fb570153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 0x7fb570153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 -- 172.21.15.107:0/534830640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0094d0 con 0x7fb5700cf450 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb574d2c640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 0x7fb570153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.126+0000 7fb56ffff640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.127+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c031840 con 0x7fb5700cf450 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.127+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0443f0 con 0x7fb5700cf450 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.127+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5700b22f0 con 0x7fb5700cf450 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.127+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0447a0 con 0x7fb5700cf450 2023-11-24T08:13:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.127+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5700b28b0 con 0x7fb5700cf450 2023-11-24T08:13:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.128+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5700b2dd0 con 0x7fb5700cf450 2023-11-24T08:13:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.129+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fb55c044900 con 0x7fb5700cf450 2023-11-24T08:13:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.130+0000 7fb56d7fa640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 0x7fb55406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.130+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb55c02d040 con 0x7fb5700cf450 2023-11-24T08:13:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.130+0000 7fb574d2c640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 0x7fb55406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.131+0000 7fb574d2c640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 0x7fb55406e350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb57015b300 tx=0x7fb564008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.134+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5700b2dd0 con 0x7fb5700cf450 2023-11-24T08:13:45.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.365+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb5700b2dd0 con 0x7fb5700cf450 2023-11-24T08:13:45.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.366+0000 7fb56d7fa640 1 -- 172.21.15.107:0/534830640 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v69) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb5700b2dd0 con 0x7fb5700cf450 2023-11-24T08:13:45.368 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 msgr2=0x7fb55406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 0x7fb55406e350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb57015b300 tx=0x7fb564008040 comp rx=0 tx=0).stop 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 msgr2=0x7fb5700aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 shutdown_connections 2023-11-24T08:13:45.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb55406be90 0x7fb55406e350 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb57015c7f0 0x7fb5700c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5700cf450 0x7fb5700aceb0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 --2- 172.21.15.107:0/534830640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5700cdfb0 0x7fb570153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 >> 172.21.15.107:0/534830640 conn(0x7fb57006a920 msgr2=0x7fb5700af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 shutdown_connections 2023-11-24T08:13:45.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:45.369+0000 7fb576f90640 1 -- 172.21.15.107:0/534830640 wait complete. 2023-11-24T08:13:45.390 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:13:45.390 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:13:45.390 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:13:45.391 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:45.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.391 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:13:45.452 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41843sec preferred_lft 41843sec 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:13:45.453 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.454 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:13:45.454 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:13:45.454 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:13:45.454 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:13:45.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.454 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:13:45.545 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:13:45.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.550 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:13:45.602 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:13:45.603 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:13:45.603 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:13:45.603 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:13:45.603 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:13:45.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.603 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:13:45.604 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:13:45.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:13:45.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:13:45.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:13:45.604 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:13:45.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.777 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:13:45.829 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:13:45.830 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:13:45.831 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:45.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:45.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:13:45.928 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:45.928 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:45.983 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:13:45.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:13:45.984 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:46.037 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:13:46.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:46.037 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:46.176 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:46.176 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:13:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:13:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:13:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:13:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:13:46.209 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:13:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:13:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:13:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:13:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:13:46.215 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:13:46.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:46.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:13:46.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:46.302 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:13:46.356 INFO:teuthology.orchestra.run.smithi120.stdout:client6187 2023-11-24T08:13:46.356 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:13:46.356 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:13:46.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:13:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.630+0000 7fc6e9656640 1 Processor -- start 2023-11-24T08:13:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.631+0000 7fc6e9656640 1 -- start start 2023-11-24T08:13:46.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.632+0000 7fc6e9656640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.632+0000 7fc6e9656640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc6e40ce640 0x7fc6e40b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.632+0000 7fc6e9656640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40ce270 con 0x7fc6e40cc470 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e9656640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e406c560 con 0x7fc6e40ce640 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e9656640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e406d0a0 con 0x7fc6e40cfb60 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e27fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc6e40ce640 0x7fc6e40b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e27fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:60222/0 (socket says 172.21.15.107:60222) 2023-11-24T08:13:46.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.633+0000 7fc6e27fc640 1 -- 172.21.15.107:0/2170304987 learned_addr learned my addr 172.21.15.107:0/2170304987 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.634+0000 7fc6e27fc640 1 -- 172.21.15.107:0/2170304987 >> v2:172.21.15.120:3300/0 conn(0x7fc6e40ce640 msgr2=0x7fc6e40b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.634+0000 7fc6e27fc640 1 --2- 172.21.15.107:0/2170304987 >> v2:172.21.15.120:3300/0 conn(0x7fc6e40ce640 0x7fc6e40b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.634+0000 7fc6e27fc640 1 -- 172.21.15.107:0/2170304987 >> v1:172.21.15.120:6789/0 conn(0x7fc6e40cc470 legacy=0x7fc6e40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.634+0000 7fc6e27fc640 1 -- 172.21.15.107:0/2170304987 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e40b4200 con 0x7fc6e40cfb60 2023-11-24T08:13:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.636+0000 7fc6e27fc640 1 --2- 172.21.15.107:0/2170304987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=661dcc93444e0c7b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.636+0000 7fc6e1ffb640 1 -- 172.21.15.107:0/2170304987 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc030840 con 0x7fc6e40cfb60 2023-11-24T08:13:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.636+0000 7fc6e1ffb640 1 -- 172.21.15.107:0/2170304987 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc0309a0 con 0x7fc6e40cfb60 2023-11-24T08:13:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.636+0000 7fc6e1ffb640 1 -- 172.21.15.107:0/2170304987 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc030d30 con 0x7fc6e40cfb60 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 -- 172.21.15.107:0/2170304987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 msgr2=0x7fc6e40abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 --2- 172.21.15.107:0/2170304987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b980 comp rx=0 tx=0).stop 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 -- 172.21.15.107:0/2170304987 shutdown_connections 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 --2- 172.21.15.107:0/2170304987 >> v2:172.21.15.120:3300/0 conn(0x7fc6e40ce640 0x7fc6e40b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 --2- 172.21.15.107:0/2170304987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40cfb60 0x7fc6e40abfc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 -- 172.21.15.107:0/2170304987 >> 172.21.15.107:0/2170304987 conn(0x7fc6e406a920 msgr2=0x7fc6e406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 -- 172.21.15.107:0/2170304987 shutdown_connections 2023-11-24T08:13:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.637+0000 7fc6e9656640 1 -- 172.21.15.107:0/2170304987 wait complete. 2023-11-24T08:13:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.639+0000 7fc6e9656640 1 Processor -- start 2023-11-24T08:13:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.639+0000 7fc6e9656640 1 -- start start 2023-11-24T08:13:46.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6e40cc470 0x7fc6e41573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40ce640 0x7fc6e4149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4163390 con 0x7fc6e40cfb60 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4163670 con 0x7fc6e40cc470 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e9656640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4163950 con 0x7fc6e40ce640 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33286/0 (socket says 172.21.15.107:33286) 2023-11-24T08:13:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e37fe640 1 -- 172.21.15.107:0/3547565376 learned_addr learned my addr 172.21.15.107:0/3547565376 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e2ffd640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6e40cc470 0x7fc6e41573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.640+0000 7fc6e27fc640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40ce640 0x7fc6e4149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 -- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40ce640 msgr2=0x7fc6e4149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40ce640 0x7fc6e4149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 -- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6e40cc470 msgr2=0x7fc6e41573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6e40cc470 0x7fc6e41573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 -- 172.21.15.107:0/3547565376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0094d0 con 0x7fc6e40cfb60 2023-11-24T08:13:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e37fe640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00e860 tx=0x7fc6dc00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00c6d0 con 0x7fc6e40cfb60 2023-11-24T08:13:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00c830 con 0x7fc6e40cfb60 2023-11-24T08:13:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00cbe0 con 0x7fc6e40cfb60 2023-11-24T08:13:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e4163c30 con 0x7fc6e40cfb60 2023-11-24T08:13:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.641+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e4162b40 con 0x7fc6e40cfb60 2023-11-24T08:13:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.642+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4164150 con 0x7fc6e40cfb60 2023-11-24T08:13:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.644+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fc6dc013470 con 0x7fc6e40cfb60 2023-11-24T08:13:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.644+0000 7fc6db7fe640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 0x7fc6c406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.645+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6dc088600 con 0x7fc6e40cfb60 2023-11-24T08:13:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.645+0000 7fc6e2ffd640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 0x7fc6c406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.645+0000 7fc6e2ffd640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 0x7fc6c406e350 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4001d40 tx=0x7fc6d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.649+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6e4164150 con 0x7fc6e40cfb60 2023-11-24T08:13:46.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.872+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc6e4164150 con 0x7fc6e40cfb60 2023-11-24T08:13:46.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.945+0000 7fc6db7fe640 1 -- 172.21.15.107:0/3547565376 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v70) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc6e4164150 con 0x7fc6e40cfb60 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 msgr2=0x7fc6c406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 0x7fc6c406e350 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4001d40 tx=0x7fc6d4009040 comp rx=0 tx=0).stop 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 msgr2=0x7fc6e414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00e860 tx=0x7fc6dc00ed20 comp rx=0 tx=0).stop 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 shutdown_connections 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6c406be90 0x7fc6c406e350 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6e40cfb60 0x7fc6e414cbe0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6e40ce640 0x7fc6e4149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 --2- 172.21.15.107:0/3547565376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6e40cc470 0x7fc6e41573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 >> 172.21.15.107:0/3547565376 conn(0x7fc6e406a920 msgr2=0x7fc6e40b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 shutdown_connections 2023-11-24T08:13:46.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:46.948+0000 7fc6e9656640 1 -- 172.21.15.107:0/3547565376 wait complete. 2023-11-24T08:13:46.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:13:47.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.176+0000 7f4a2d459640 1 Processor -- start 2023-11-24T08:13:47.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.176+0000 7f4a2d459640 1 -- start start 2023-11-24T08:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2d459640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4a2815b1d0 0x7f4a28071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2d459640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a28070550 0x7f4a280b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2d459640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a280ca0e0 con 0x7f4a280b1080 2023-11-24T08:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2d459640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2815fde0 con 0x7f4a2815b1d0 2023-11-24T08:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2d459640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a280b14a0 con 0x7f4a28070550 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a27fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4a2815b1d0 0x7f4a28071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2cc58640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4a280b1080 0x7f4a2806c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41906/0 (socket says 172.21.15.107:41906) 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.178+0000 7f4a2cc58640 1 -- 172.21.15.107:0/288645255 learned_addr learned my addr 172.21.15.107:0/288645255 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.179+0000 7f4a1ffff640 1 --2- 172.21.15.107:0/288645255 >> v2:172.21.15.120:3300/0 conn(0x7f4a28070550 0x7f4a280b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.179+0000 7f4a27fff640 1 -- 172.21.15.107:0/288645255 >> v2:172.21.15.120:3300/0 conn(0x7f4a28070550 msgr2=0x7f4a280b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.179+0000 7f4a27fff640 1 --2- 172.21.15.107:0/288645255 >> v2:172.21.15.120:3300/0 conn(0x7f4a28070550 0x7f4a280b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.179+0000 7f4a27fff640 1 -- 172.21.15.107:0/288645255 >> v1:172.21.15.107:6789/0 conn(0x7f4a280b1080 legacy=0x7f4a2806c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.179+0000 7f4a27fff640 1 -- 172.21.15.107:0/288645255 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a28072800 con 0x7f4a2815b1d0 2023-11-24T08:13:47.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.180+0000 7f4a27fff640 1 --2- 172.21.15.107:0/288645255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a2815b1d0 0x7f4a28071f40 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4a280cef60 tx=0x7f4a1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2897ece67a251b11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a277fe640 1 -- 172.21.15.107:0/288645255 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18031830 con 0x7f4a2815b1d0 2023-11-24T08:13:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a277fe640 1 -- 172.21.15.107:0/288645255 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a18031990 con 0x7f4a2815b1d0 2023-11-24T08:13:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a277fe640 1 -- 172.21.15.107:0/288645255 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18031d20 con 0x7f4a2815b1d0 2023-11-24T08:13:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 -- 172.21.15.107:0/288645255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a2815b1d0 msgr2=0x7f4a28071f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 --2- 172.21.15.107:0/288645255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a2815b1d0 0x7f4a28071f40 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4a280cef60 tx=0x7f4a1802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 -- 172.21.15.107:0/288645255 shutdown_connections 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 --2- 172.21.15.107:0/288645255 >> v2:172.21.15.120:3300/0 conn(0x7f4a28070550 0x7f4a280b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 --2- 172.21.15.107:0/288645255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a2815b1d0 0x7f4a28071f40 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 -- 172.21.15.107:0/288645255 >> 172.21.15.107:0/288645255 conn(0x7f4a280cbb40 msgr2=0x7f4a280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.181+0000 7f4a2d459640 1 -- 172.21.15.107:0/288645255 shutdown_connections 2023-11-24T08:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.182+0000 7f4a2d459640 1 -- 172.21.15.107:0/288645255 wait complete. 2023-11-24T08:13:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.183+0000 7f4a2d459640 1 Processor -- start 2023-11-24T08:13:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.183+0000 7f4a2d459640 1 -- start start 2023-11-24T08:13:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a28070550 0x7f4a2814c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a2815b1d0 0x7f4a280c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281f4e10 con 0x7f4a280b1080 2023-11-24T08:13:47.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281f4260 con 0x7f4a2815b1d0 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.184+0000 7f4a2d459640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281495f0 con 0x7f4a28070550 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33302/0 (socket says 172.21.15.107:33302) 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 -- 172.21.15.107:0/174962661 learned_addr learned my addr 172.21.15.107:0/174962661 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 -- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a28070550 msgr2=0x7f4a2814c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a28070550 0x7f4a2814c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a2cc58640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a2815b1d0 0x7f4a280c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 -- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a2815b1d0 msgr2=0x7f4a280c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a2815b1d0 0x7f4a280c2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 -- 172.21.15.107:0/174962661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18009400 con 0x7f4a280b1080 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.185+0000 7f4a1ffff640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000d800 tx=0x7f4a1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.186+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1000b6c0 con 0x7f4a280b1080 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.186+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1000b820 con 0x7f4a280b1080 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.186+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a280b4b10 con 0x7f4a280b1080 2023-11-24T08:13:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.186+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1000bbb0 con 0x7f4a280b1080 2023-11-24T08:13:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.186+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a281f4470 con 0x7f4a280b1080 2023-11-24T08:13:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.187+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a281f5080 con 0x7f4a280b1080 2023-11-24T08:13:47.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.188+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f4a1000bd10 con 0x7f4a280b1080 2023-11-24T08:13:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.189+0000 7f4a257fa640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 0x7f4a0806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.190+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a10088730 con 0x7f4a280b1080 2023-11-24T08:13:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.190+0000 7f4a27fff640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 0x7f4a0806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.190+0000 7f4a27fff640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 0x7f4a0806e350 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a18030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.194+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a10055230 con 0x7f4a280b1080 2023-11-24T08:13:47.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.402+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a280c9d70 con 0x7f4a0806be90 2023-11-24T08:13:47.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.406+0000 7f4a257fa640 1 -- 172.21.15.107:0/174962661 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4a18031b00 con 0x7f4a0806be90 2023-11-24T08:13:47.408 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:13:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:13:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:13:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:13:47.408 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:13:47.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 msgr2=0x7f4a0806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 0x7f4a0806e350 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a18030000 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 msgr2=0x7f4a281490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000d800 tx=0x7f4a1000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 shutdown_connections 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0806be90 0x7f4a0806e350 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a2815b1d0 0x7f4a280c2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a280b1080 0x7f4a281490b0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 --2- 172.21.15.107:0/174962661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a28070550 0x7f4a2814c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 >> 172.21.15.107:0/174962661 conn(0x7f4a280cbb40 msgr2=0x7f4a280ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.408+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 shutdown_connections 2023-11-24T08:13:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.409+0000 7f4a2d459640 1 -- 172.21.15.107:0/174962661 wait complete. 2023-11-24T08:13:47.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:13:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.634+0000 7fceaa25c640 1 Processor -- start 2023-11-24T08:13:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.634+0000 7fceaa25c640 1 -- start start 2023-11-24T08:13:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fceaa25c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcea406de10 0x7fcea415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fceaa25c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea415ec90 con 0x7fcea4072160 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fceaa25c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40cd530 con 0x7fcea40c49f0 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fceaa25c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea40c4ed0 con 0x7fcea406de10 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fcea37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcea4072160 0x7fcea406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41920/0 (socket says 172.21.15.107:41920) 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fcea37fe640 1 -- 172.21.15.107:0/3211179286 learned_addr learned my addr 172.21.15.107:0/3211179286 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.636+0000 7fce9affd640 1 --2- 172.21.15.107:0/3211179286 >> v2:172.21.15.120:3300/0 conn(0x7fcea406de10 0x7fcea415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901244104 0 0) 0x7fce94001490 con 0x7fcea4072160 2023-11-24T08:13:47.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce90003520 con 0x7fcea4072160 2023-11-24T08:13:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2156061537 0 0) 0x7fce90003520 con 0x7fcea4072160 2023-11-24T08:13:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 >> v2:172.21.15.120:3300/0 conn(0x7fcea406de10 msgr2=0x7fcea415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 --2- 172.21.15.107:0/3211179286 >> v2:172.21.15.120:3300/0 conn(0x7fcea406de10 0x7fcea415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 >> v1:172.21.15.120:6789/0 conn(0x7fcea40c49f0 legacy=0x7fcea40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40c51b0 con 0x7fcea4072160 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fce94001b80 con 0x7fcea4072160 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 >> v1:172.21.15.107:6789/0 conn(0x7fcea4072160 legacy=0x7fcea406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 0x7fce90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900047c0 0x7fce90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90005180 0x7fce9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c4ed0 con 0x7fce90003970 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9000c020 con 0x7fce90005180 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.637+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9000e3f0 con 0x7fce900047c0 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 0x7fce90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 -- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900047c0 msgr2=0x7fce90004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900047c0 0x7fce90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 -- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90005180 msgr2=0x7fce9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90005180 0x7fce9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea37fe640 1 -- 172.21.15.107:0/3211179286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce90003520 con 0x7fce90003970 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.638+0000 7fcea3fff640 1 -- 172.21.15.107:0/3211179286 reap_dead start 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.639+0000 7fcea37fe640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 0x7fce90003d90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fce9400a830 tx=0x7fce900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526e9f90efa2ae11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.639+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94034630 con 0x7fce90003970 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.639+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9402f070 con 0x7fce90003970 2023-11-24T08:13:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.639+0000 7fcea2ffd640 1 -- 172.21.15.107:0/3211179286 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94034a30 con 0x7fce90003970 2023-11-24T08:13:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 -- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 msgr2=0x7fce90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 0x7fce90003d90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fce9400a830 tx=0x7fce900057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 -- 172.21.15.107:0/3211179286 shutdown_connections 2023-11-24T08:13:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 --2- 172.21.15.107:0/3211179286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce90003970 0x7fce90003d90 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 -- 172.21.15.107:0/3211179286 >> 172.21.15.107:0/3211179286 conn(0x7fcea40b0c50 msgr2=0x7fcea40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 -- 172.21.15.107:0/3211179286 shutdown_connections 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.640+0000 7fceaa25c640 1 -- 172.21.15.107:0/3211179286 wait complete. 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.642+0000 7fceaa25c640 1 Processor -- start 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.642+0000 7fceaa25c640 1 -- start start 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90003970 0x7fcea4146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcea415d040 0x7fcea40cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea41469b0 con 0x7fcea415a470 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415ae30 con 0x7fce90003970 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fceaa25c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40cbdb0 con 0x7fcea415d040 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33316/0 (socket says 172.21.15.107:33316) 2023-11-24T08:13:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fcea3fff640 1 -- 172.21.15.107:0/942960795 learned_addr learned my addr 172.21.15.107:0/942960795 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fcea37fe640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90003970 0x7fcea4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.643+0000 7fce9affd640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcea415d040 0x7fcea40cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 -- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcea415d040 msgr2=0x7fcea40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcea415d040 0x7fcea40cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 -- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90003970 msgr2=0x7fcea4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90003970 0x7fcea4146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 -- 172.21.15.107:0/942960795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40cc030 con 0x7fcea415a470 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea3fff640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c4e00 tx=0x7fce9c00c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c007270 con 0x7fcea415a470 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c0073d0 con 0x7fcea415a470 2023-11-24T08:13:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea40cc310 con 0x7fcea415a470 2023-11-24T08:13:47.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0057c0 con 0x7fcea415a470 2023-11-24T08:13:47.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.644+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4148760 con 0x7fcea415a470 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.645+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40cc830 con 0x7fcea415a470 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.646+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7fce9c005920 con 0x7fcea415a470 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.647+0000 7fcea0ff9640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 0x7fce8406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.647+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce9c087210 con 0x7fcea415a470 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.648+0000 7fcea37fe640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 0x7fce8406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.648+0000 7fcea37fe640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 0x7fce8406e350 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fce94034000 tx=0x7fce9402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:47.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.652+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcea40cc830 con 0x7fcea415a470 2023-11-24T08:13:47.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.841+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcea40cc830 con 0x7fcea415a470 2023-11-24T08:13:47.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.843+0000 7fcea0ff9640 1 -- 172.21.15.107:0/942960795 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcea40cc830 con 0x7fcea415a470 2023-11-24T08:13:47.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 msgr2=0x7fce8406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 0x7fce8406e350 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fce94034000 tx=0x7fce9402f000 comp rx=0 tx=0).stop 2023-11-24T08:13:47.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 msgr2=0x7fcea415a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:47.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c4e00 tx=0x7fce9c00c570 comp rx=0 tx=0).stop 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 shutdown_connections 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce8406be90 0x7fce8406e350 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea415a470 0x7fcea415a8f0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcea415d040 0x7fcea40cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 --2- 172.21.15.107:0/942960795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce90003970 0x7fcea4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 >> 172.21.15.107:0/942960795 conn(0x7fcea40b0c50 msgr2=0x7fcea40ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 shutdown_connections 2023-11-24T08:13:47.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:47.845+0000 7fceaa25c640 1 -- 172.21.15.107:0/942960795 wait complete. 2023-11-24T08:13:47.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/periodic_async_work True 2023-11-24T08:13:48.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.071+0000 7f638b267640 1 Processor -- start 2023-11-24T08:13:48.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.071+0000 7f638b267640 1 -- start start 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f638b267640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 0x7f638406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f638b267640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638415ed20 con 0x7f63840b0b40 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f638b267640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638415f8a0 con 0x7f638415b260 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f638b267640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638415b640 con 0x7f63840b0fb0 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f6389a64640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f63840b0b40 0x7f638406c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36250/0 (socket says 172.21.15.107:36250) 2023-11-24T08:13:48.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.072+0000 7f6389a64640 1 -- 172.21.15.107:0/2012289719 learned_addr learned my addr 172.21.15.107:0/2012289719 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:13:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048253722 0 0) 0x7f638415ed20 con 0x7f63840b0b40 2023-11-24T08:13:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f638aa66640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 0x7f638406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6370003520 con 0x7f63840b0b40 2023-11-24T08:13:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080883953 0 0) 0x7f6370003520 con 0x7f63840b0b40 2023-11-24T08:13:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 msgr2=0x7f638406ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 0x7f638406ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 >> v1:172.21.15.120:6789/0 conn(0x7f638415b260 legacy=0x7f638406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638406e750 con 0x7f63840b0b40 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6374001670 con 0x7f63840b0b40 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 >> v1:172.21.15.107:6789/0 conn(0x7f63840b0b40 legacy=0x7f638406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63700039a0 0x7f6370003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700047c0 0x7f6370004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 0x7f637000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638415b640 con 0x7f63700039a0 2023-11-24T08:13:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370005500 con 0x7f63700047c0 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637000bec0 con 0x7f63840b0fb0 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.073+0000 7f638a265640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63700039a0 0x7f6370003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.074+0000 7f638a265640 1 -- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 msgr2=0x7f637000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.074+0000 7f638a265640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63840b0fb0 0x7f637000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.074+0000 7f638a265640 1 -- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700047c0 msgr2=0x7f6370004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.074+0000 7f638a265640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700047c0 0x7f6370004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.074+0000 7f638a265640 1 -- 172.21.15.107:0/2012289719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6370003520 con 0x7f63700039a0 2023-11-24T08:13:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.075+0000 7f638a265640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63700039a0 0x7f6370003dc0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6380001750 tx=0x7f63700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d936acebc4695e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.075+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63800052d0 con 0x7f63700039a0 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.075+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638002e050 con 0x7f63700039a0 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.075+0000 7f6389263640 1 -- 172.21.15.107:0/2012289719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63800035f0 con 0x7f63700039a0 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 -- 172.21.15.107:0/2012289719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63700039a0 msgr2=0x7f6370003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 --2- 172.21.15.107:0/2012289719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63700039a0 0x7f6370003dc0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6380001750 tx=0x7f63700057f0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638aa66640 1 -- 172.21.15.107:0/2012289719 reap_dead start 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 -- 172.21.15.107:0/2012289719 shutdown_connections 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 -- 172.21.15.107:0/2012289719 >> 172.21.15.107:0/2012289719 conn(0x7f63840cbb40 msgr2=0x7f63840cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 -- 172.21.15.107:0/2012289719 shutdown_connections 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.076+0000 7f638b267640 1 -- 172.21.15.107:0/2012289719 wait complete. 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.078+0000 7f638b267640 1 Processor -- start 2023-11-24T08:13:48.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.078+0000 7f638b267640 1 -- start start 2023-11-24T08:13:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700039a0 0x7f638409cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 0x7f638415dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638414c740 con 0x7f638406cb50 2023-11-24T08:13:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638406d510 con 0x7f63700039a0 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638b267640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6384099e70 con 0x7f638414c2c0 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58968/0 (socket says 172.21.15.107:58968) 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 -- 172.21.15.107:0/2545864677 learned_addr learned my addr 172.21.15.107:0/2545864677 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638a265640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700039a0 0x7f638409cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f6389a64640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 0x7f638415dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 -- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 msgr2=0x7f638415dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 0x7f638415dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 -- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700039a0 msgr2=0x7f638409cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700039a0 0x7f638409cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f638aa66640 1 -- 172.21.15.107:0/2545864677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638415ecb0 con 0x7f638406cb50 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.079+0000 7f6389a64640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 0x7f638415dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:13:48.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f638aa66640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f63840cb9b0 tx=0x7f637c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c018070 con 0x7f638406cb50 2023-11-24T08:13:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00eb90 con 0x7f638406cb50 2023-11-24T08:13:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c003990 con 0x7f638406cb50 2023-11-24T08:13:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638415e6f0 con 0x7f638406cb50 2023-11-24T08:13:48.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.080+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6384099b00 con 0x7f638406cb50 2023-11-24T08:13:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.081+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638404e840 con 0x7f638406cb50 2023-11-24T08:13:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.082+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) v1 ==== 90710+0+0 (secure 0 0 0) 0x7f637c014420 con 0x7f638406cb50 2023-11-24T08:13:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.083+0000 7f637affd640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 0x7f636806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.083+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f637c088240 con 0x7f638406cb50 2023-11-24T08:13:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.083+0000 7f638a265640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 0x7f636806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:48.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.084+0000 7f638a265640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 0x7f636806e380 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6380033000 tx=0x7f6380009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.087+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f637c054a80 con 0x7f638406cb50 2023-11-24T08:13:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.276+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/periodic_async_work}] v 0) v1 -- 0x7f63840ae710 con 0x7f638406cb50 2023-11-24T08:13:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.282+0000 7f637affd640 1 -- 172.21.15.107:0/2545864677 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/periodic_async_work}]=0 v31)=0 v31) v1 ==== 132+0+0 (secure 0 0 0) 0x7f637c0580a0 con 0x7f638406cb50 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.285+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 msgr2=0x7f636806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.285+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 0x7f636806e380 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6380033000 tx=0x7f6380009030 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.285+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 msgr2=0x7f638406cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.285+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f63840cb9b0 tx=0x7f637c00e4b0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 shutdown_connections 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f636806bec0 0x7f636806e380 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f638406cb50 0x7f638406cfd0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f638414c2c0 0x7f638415dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 --2- 172.21.15.107:0/2545864677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63700039a0 0x7f638409cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 >> 172.21.15.107:0/2545864677 conn(0x7f63840cbb40 msgr2=0x7f63840b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 shutdown_connections 2023-11-24T08:13:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:48.286+0000 7f638b267640 1 -- 172.21.15.107:0/2545864677 wait complete. 2023-11-24T08:13:48.305 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.304+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.305 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.304+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.306 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.304+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.306 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.304+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.306 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.304+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:48.308 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_deleting/subvol) 2023-11-24T08:13:48.309 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.306+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.309 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.306+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.309 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.306+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.309 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.306+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.309 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:48.306+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:48.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:48.395 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:13:48.442 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:13:53.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:53.344+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:53.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:53.344+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:53.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:53.344+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:53.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:53.344+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:53.345 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:13:53.344+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:13:53.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:13:53.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:13:53.506 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:13:53.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/periodic_async_work False 2023-11-24T08:13:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.724+0000 7f68c03b0640 1 Processor -- start 2023-11-24T08:13:53.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.724+0000 7f68c03b0640 1 -- start start 2023-11-24T08:13:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.726+0000 7f68c03b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.726+0000 7f68c03b0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68b80c49f0 0x7f68b80705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.726+0000 7f68c03b0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b815ec90 con 0x7f68b8072160 2023-11-24T08:13:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.726+0000 7f68c03b0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80cd530 con 0x7f68b80c49f0 2023-11-24T08:13:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.726+0000 7f68c03b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80c4ed0 con 0x7f68b806de10 2023-11-24T08:13:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68be94d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68b80c49f0 0x7f68b80705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:46346/0 (socket says 172.21.15.107:46346) 2023-11-24T08:13:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 -- 172.21.15.107:0/2113417382 learned_addr learned my addr 172.21.15.107:0/2113417382 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:53.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 -- 172.21.15.107:0/2113417382 >> v2:172.21.15.120:3300/0 conn(0x7f68b80c49f0 msgr2=0x7f68b80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:13:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 --2- 172.21.15.107:0/2113417382 >> v2:172.21.15.120:3300/0 conn(0x7f68b80c49f0 0x7f68b80705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 -- 172.21.15.107:0/2113417382 >> v1:172.21.15.120:6789/0 conn(0x7f68b8072160 legacy=0x7f68b806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.727+0000 7f68bd94b640 1 -- 172.21.15.107:0/2113417382 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80c51b0 con 0x7f68b806de10 2023-11-24T08:13:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.729+0000 7f68bd94b640 1 --2- 172.21.15.107:0/2113417382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009870 tx=0x7f68ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6759b8bd4979cfe3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:53.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.729+0000 7f68bd14a640 1 -- 172.21.15.107:0/2113417382 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac030840 con 0x7f68b806de10 2023-11-24T08:13:53.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.729+0000 7f68bd14a640 1 -- 172.21.15.107:0/2113417382 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac0309a0 con 0x7f68b806de10 2023-11-24T08:13:53.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.729+0000 7f68bd14a640 1 -- 172.21.15.107:0/2113417382 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac030d30 con 0x7f68b806de10 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 -- 172.21.15.107:0/2113417382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 msgr2=0x7f68b815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 --2- 172.21.15.107:0/2113417382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009870 tx=0x7f68ac02b980 comp rx=0 tx=0).stop 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 -- 172.21.15.107:0/2113417382 shutdown_connections 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 --2- 172.21.15.107:0/2113417382 >> v2:172.21.15.120:3300/0 conn(0x7f68b80c49f0 0x7f68b80705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 --2- 172.21.15.107:0/2113417382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b806de10 0x7f68b815e750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 -- 172.21.15.107:0/2113417382 >> 172.21.15.107:0/2113417382 conn(0x7f68b80b0c50 msgr2=0x7f68b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:53.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 -- 172.21.15.107:0/2113417382 shutdown_connections 2023-11-24T08:13:53.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.730+0000 7f68c03b0640 1 -- 172.21.15.107:0/2113417382 wait complete. 2023-11-24T08:13:53.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.732+0000 7f68c03b0640 1 Processor -- start 2023-11-24T08:13:53.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.732+0000 7f68c03b0640 1 -- start start 2023-11-24T08:13:53.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 0x7f68b8150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 0x7f68b815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b80c49f0 0x7f68b815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b815cea0 con 0x7f68b806de10 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81514d0 con 0x7f68b80c49f0 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68c03b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b815d2c0 con 0x7f68b8072160 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68bd94b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 0x7f68b815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.733+0000 7f68be14c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 0x7f68b8150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68bd94b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 0x7f68b815b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:46360/0 (socket says 172.21.15.107:46360) 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68bd94b640 1 -- 172.21.15.107:0/3370381885 learned_addr learned my addr 172.21.15.107:0/3370381885 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:13:53.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be94d640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b80c49f0 0x7f68b815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 -- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 msgr2=0x7f68b815b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 0x7f68b815b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 -- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b80c49f0 msgr2=0x7f68b815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b80c49f0 0x7f68b815bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 -- 172.21.15.107:0/3370381885 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0094d0 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.734+0000 7f68be14c640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 0x7f68b8150f90 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c3c0 tx=0x7f68b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.735+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b40052d0 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.735+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4005430 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.735+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b809c9e0 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.735+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4003830 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.735+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b815c500 con 0x7f68b806de10 2023-11-24T08:13:53.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.736+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b815ea50 con 0x7f68b806de10 2023-11-24T08:13:53.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.737+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) v1 ==== 90812+0+0 (secure 0 0 0) 0x7f68b4003990 con 0x7f68b806de10 2023-11-24T08:13:53.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.739+0000 7f68aaffd640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 0x7f689c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:13:53.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.739+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68b40877a0 con 0x7f68b806de10 2023-11-24T08:13:53.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.739+0000 7f68bd94b640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 0x7f689c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:13:53.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.740+0000 7f68bd94b640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 0x7f689c06e470 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009870 tx=0x7f68ac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:13:53.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.746+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68b40bde90 con 0x7f68b806de10 2023-11-24T08:13:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.937+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/periodic_async_work}] v 0) v1 -- 0x7f68b815ec90 con 0x7f68b806de10 2023-11-24T08:13:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.945+0000 7f68aaffd640 1 -- 172.21.15.107:0/3370381885 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/periodic_async_work}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7f68b815ec90 con 0x7f68b806de10 2023-11-24T08:13:53.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 msgr2=0x7f689c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 0x7f689c06e470 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009870 tx=0x7f68ac008030 comp rx=0 tx=0).stop 2023-11-24T08:13:53.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 msgr2=0x7f68b8150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:13:53.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 0x7f68b8150f90 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c3c0 tx=0x7f68b400c880 comp rx=0 tx=0).stop 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 shutdown_connections 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f689c06bfb0 0x7f689c06e470 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b80c49f0 0x7f68b815bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b8072160 0x7f68b815b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 --2- 172.21.15.107:0/3370381885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b806de10 0x7f68b8150f90 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:13:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 >> 172.21.15.107:0/3370381885 conn(0x7f68b80b0c50 msgr2=0x7f68b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:13:53.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 shutdown_connections 2023-11-24T08:13:53.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:13:53.948+0000 7f68c03b0640 1 -- 172.21.15.107:0/3370381885 wait complete. 2023-11-24T08:14:03.969 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:14:03.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:14:04.093 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:14:04.222 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:14:04.222 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:04.223 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:04.284 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:04.284 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:04.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.553+0000 7f96fb59e640 1 Processor -- start 2023-11-24T08:14:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.553+0000 7f96fb59e640 1 -- start start 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fb59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96fc144b30 0x7f96fc15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fb59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f96fc0b1de0 0x7f96fc0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96fc15bf80 con 0x7f96fc15c410 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fb59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0b27b0 con 0x7f96fc144b30 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fb59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0b1560 con 0x7f96fc0b1de0 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fa59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96fc144b30 0x7f96fc15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96f9d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f96fc15c410 0x7f96fc0b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38196/0 (socket says 172.21.15.107:38196) 2023-11-24T08:14:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96f9d9b640 1 -- 172.21.15.107:0/354505298 learned_addr learned my addr 172.21.15.107:0/354505298 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96fad9d640 1 --2- 172.21.15.107:0/354505298 >> v2:172.21.15.120:3300/0 conn(0x7f96fc0b1de0 0x7f96fc0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.556+0000 7f96fa59c640 1 -- 172.21.15.107:0/354505298 >> v2:172.21.15.120:3300/0 conn(0x7f96fc0b1de0 msgr2=0x7f96fc0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.556+0000 7f96fa59c640 1 --2- 172.21.15.107:0/354505298 >> v2:172.21.15.120:3300/0 conn(0x7f96fc0b1de0 0x7f96fc0b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.555+0000 7f96f959a640 1 -- 172.21.15.107:0/354505298 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738574898 0 0) 0x7f96fc15bf80 con 0x7f96fc15c410 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.556+0000 7f96fa59c640 1 -- 172.21.15.107:0/354505298 >> v1:172.21.15.107:6789/0 conn(0x7f96fc15c410 legacy=0x7f96fc0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.556+0000 7f96fa59c640 1 -- 172.21.15.107:0/354505298 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc0b1280 con 0x7f96fc144b30 2023-11-24T08:14:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.557+0000 7f96fa59c640 1 --2- 172.21.15.107:0/354505298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc144b30 0x7f96fc15f920 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f96fc072a50 tx=0x7f96ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f47ad82b8b61e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.557+0000 7f96f959a640 1 -- 172.21.15.107:0/354505298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec031830 con 0x7f96fc144b30 2023-11-24T08:14:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.557+0000 7f96f959a640 1 -- 172.21.15.107:0/354505298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec031990 con 0x7f96fc144b30 2023-11-24T08:14:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.557+0000 7f96f959a640 1 -- 172.21.15.107:0/354505298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec031d20 con 0x7f96fc144b30 2023-11-24T08:14:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 -- 172.21.15.107:0/354505298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc144b30 msgr2=0x7f96fc15f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 --2- 172.21.15.107:0/354505298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc144b30 0x7f96fc15f920 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f96fc072a50 tx=0x7f96ec02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 -- 172.21.15.107:0/354505298 shutdown_connections 2023-11-24T08:14:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 --2- 172.21.15.107:0/354505298 >> v2:172.21.15.120:3300/0 conn(0x7f96fc0b1de0 0x7f96fc0b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 --2- 172.21.15.107:0/354505298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc144b30 0x7f96fc15f920 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 -- 172.21.15.107:0/354505298 >> 172.21.15.107:0/354505298 conn(0x7f96fc0cb830 msgr2=0x7f96fc0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 -- 172.21.15.107:0/354505298 shutdown_connections 2023-11-24T08:14:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.558+0000 7f96fb59e640 1 -- 172.21.15.107:0/354505298 wait complete. 2023-11-24T08:14:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.560+0000 7f96fb59e640 1 Processor -- start 2023-11-24T08:14:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.560+0000 7f96fb59e640 1 -- start start 2023-11-24T08:14:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96fc144b30 0x7f96fc0a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96fc15c410 0x7f96fc0a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc142540 con 0x7f96fc0b1de0 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc142720 con 0x7f96fc144b30 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fb59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc142a00 con 0x7f96fc15c410 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fa59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fa59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44688/0 (socket says 172.21.15.107:44688) 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.561+0000 7f96fa59c640 1 -- 172.21.15.107:0/1381621023 learned_addr learned my addr 172.21.15.107:0/1381621023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96f9d9b640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96fc144b30 0x7f96fc0a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 -- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96fc15c410 msgr2=0x7f96fc0a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96fc15c410 0x7f96fc0a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 -- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96fc144b30 msgr2=0x7f96fc0a8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96fc144b30 0x7f96fc0a8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 -- 172.21.15.107:0/1381621023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec009400 con 0x7f96fc0b1de0 2023-11-24T08:14:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fa59c640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009890 tx=0x7f96ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec03ba70 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec03bbf0 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec0453f0 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.562+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc142ce0 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.563+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc1432a0 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.564+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc143490 con 0x7f96fc0b1de0 2023-11-24T08:14:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.565+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f96ec045550 con 0x7f96fc0b1de0 2023-11-24T08:14:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.566+0000 7f96eaffd640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 0x7f96dc06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.566+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96ec0aa740 con 0x7f96fc0b1de0 2023-11-24T08:14:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.566+0000 7f96f9d9b640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 0x7f96dc06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.566+0000 7f96f9d9b640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 0x7f96dc06e530 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f96fc141ac0 tx=0x7f96f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:04.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.571+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96ec077050 con 0x7f96fc0b1de0 2023-11-24T08:14:04.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.761+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96fc04e840 con 0x7f96fc0b1de0 2023-11-24T08:14:04.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.763+0000 7f96eaffd640 1 -- 172.21.15.107:0/1381621023 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7f96ec07a670 con 0x7f96fc0b1de0 2023-11-24T08:14:04.765 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:04.766 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:04.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 msgr2=0x7f96dc06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 0x7f96dc06e530 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f96fc141ac0 tx=0x7f96f0009040 comp rx=0 tx=0).stop 2023-11-24T08:14:04.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 msgr2=0x7f96fc137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:04.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009890 tx=0x7f96ec0046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 shutdown_connections 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96dc06c070 0x7f96dc06e530 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96fc15c410 0x7f96fc0a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96fc144b30 0x7f96fc0a8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 --2- 172.21.15.107:0/1381621023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96fc0b1de0 0x7f96fc137950 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 >> 172.21.15.107:0/1381621023 conn(0x7f96fc0cb830 msgr2=0x7f96fc0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 shutdown_connections 2023-11-24T08:14:04.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.766+0000 7f96fb59e640 1 -- 172.21.15.107:0/1381621023 wait complete. 2023-11-24T08:14:04.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:13:33.870908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:13:35.060534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:14:04.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:14:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.997+0000 7f837c11d640 1 Processor -- start 2023-11-24T08:14:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.997+0000 7f837c11d640 1 -- start start 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f837415b440 0x7f83740cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83740cbf20 0x7f83740c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740c57e0 0x7f83740c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83740cc3d0 con 0x7f837415b440 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837415b820 con 0x7f83740cbf20 2023-11-24T08:14:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837c11d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837415eea0 con 0x7f83740c57e0 2023-11-24T08:14:05.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f8379eb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f837415b440 0x7f83740cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f8379eb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f837415b440 0x7f83740cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44698/0 (socket says 172.21.15.107:44698) 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f8379eb9640 1 -- 172.21.15.107:0/4016238632 learned_addr learned my addr 172.21.15.107:0/4016238632 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f83796b8640 1 --2- 172.21.15.107:0/4016238632 >> v2:172.21.15.120:3300/0 conn(0x7f83740cbf20 0x7f83740c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:04.999+0000 7f837a6ba640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740c57e0 0x7f83740c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.000+0000 7f8379eb9640 1 -- 172.21.15.107:0/4016238632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740c57e0 msgr2=0x7f83740c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.000+0000 7f8379eb9640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740c57e0 0x7f83740c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.000+0000 7f8379eb9640 1 -- 172.21.15.107:0/4016238632 >> v2:172.21.15.120:3300/0 conn(0x7f83740cbf20 msgr2=0x7f83740c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.000+0000 7f8379eb9640 1 --2- 172.21.15.107:0/4016238632 >> v2:172.21.15.120:3300/0 conn(0x7f83740cbf20 0x7f83740c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.000+0000 7f8379eb9640 1 -- 172.21.15.107:0/4016238632 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83740abf30 con 0x7f837415b440 2023-11-24T08:14:05.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.001+0000 7f8379eb9640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f83740cb9e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f837406e890 tx=0x7f837002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137c83afd9617684 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.001+0000 7f8378eb7640 1 -- 172.21.15.107:0/4016238632 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83700317e0 con 0x7f837415b440 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.001+0000 7f8378eb7640 1 -- 172.21.15.107:0/4016238632 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370031940 con 0x7f837415b440 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.001+0000 7f8378eb7640 1 -- 172.21.15.107:0/4016238632 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370031cd0 con 0x7f837415b440 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 -- 172.21.15.107:0/4016238632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 msgr2=0x7f83740cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f83740cb9e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f837406e890 tx=0x7f837002b880 comp rx=0 tx=0).stop 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 -- 172.21.15.107:0/4016238632 shutdown_connections 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740c57e0 0x7f83740c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 --2- 172.21.15.107:0/4016238632 >> v2:172.21.15.120:3300/0 conn(0x7f83740cbf20 0x7f83740c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 --2- 172.21.15.107:0/4016238632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f83740cb9e0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 -- 172.21.15.107:0/4016238632 >> 172.21.15.107:0/4016238632 conn(0x7f83740b0c50 msgr2=0x7f83740b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 -- 172.21.15.107:0/4016238632 shutdown_connections 2023-11-24T08:14:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.002+0000 7f837c11d640 1 -- 172.21.15.107:0/4016238632 wait complete. 2023-11-24T08:14:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.004+0000 7f837c11d640 1 Processor -- start 2023-11-24T08:14:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.004+0000 7f837c11d640 1 -- start start 2023-11-24T08:14:05.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83740c57e0 0x7f8374153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740cbf20 0x7f83741521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837415df30 con 0x7f837415b440 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837415d2b0 con 0x7f83740c57e0 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837c11d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837415d590 con 0x7f83740cbf20 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837a6ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837a6ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44714/0 (socket says 172.21.15.107:44714) 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f837a6ba640 1 -- 172.21.15.107:0/548171470 learned_addr learned my addr 172.21.15.107:0/548171470 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.005+0000 7f8379eb9640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83740c57e0 0x7f8374153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f83796b8640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740cbf20 0x7f83741521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 -- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740cbf20 msgr2=0x7f83741521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740cbf20 0x7f83741521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 -- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83740c57e0 msgr2=0x7f8374153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83740c57e0 0x7f8374153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 -- 172.21.15.107:0/548171470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700094d0 con 0x7f837415b440 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f837a6ba640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f836c007e60 tx=0x7f836c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.006+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c014620 con 0x7f837415b440 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.007+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c00ee90 con 0x7f837415b440 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.007+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0149e0 con 0x7f837415b440 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.007+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837415d870 con 0x7f837415b440 2023-11-24T08:14:05.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.007+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8374154130 con 0x7f837415b440 2023-11-24T08:14:05.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.008+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83740c54b0 con 0x7f837415b440 2023-11-24T08:14:05.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.009+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f836c014b40 con 0x7f837415b440 2023-11-24T08:14:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.010+0000 7f836a7fc640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 0x7f835806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.010+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f836c088e90 con 0x7f837415b440 2023-11-24T08:14:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.010+0000 7f8379eb9640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 0x7f835806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.011+0000 7f8379eb9640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 0x7f835806e500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8370036000 tx=0x7f837003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.015+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f836c055680 con 0x7f837415b440 2023-11-24T08:14:05.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.204+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83740c5bf0 con 0x7f837415b440 2023-11-24T08:14:05.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.205+0000 7f836a7fc640 1 -- 172.21.15.107:0/548171470 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f836c058ca0 con 0x7f837415b440 2023-11-24T08:14:05.207 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 msgr2=0x7f835806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 0x7f835806e500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8370036000 tx=0x7f837003d040 comp rx=0 tx=0).stop 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 msgr2=0x7f837415b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f836c007e60 tx=0x7f836c00e4b0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 shutdown_connections 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f835806c040 0x7f835806e500 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f837415b440 0x7f837415b960 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83740cbf20 0x7f83741521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 --2- 172.21.15.107:0/548171470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83740c57e0 0x7f8374153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 >> 172.21.15.107:0/548171470 conn(0x7f83740b0c50 msgr2=0x7f8374071810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 shutdown_connections 2023-11-24T08:14:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.208+0000 7f837c11d640 1 -- 172.21.15.107:0/548171470 wait complete. 2023-11-24T08:14:05.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:14:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.447+0000 7fce8977c640 1 Processor -- start 2023-11-24T08:14:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.448+0000 7fce8977c640 1 -- start start 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce8977c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce840ce260 0x7fce840d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce8977c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840d21d0 con 0x7fce841608f0 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce8977c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce84160cd0 con 0x7fce840ce710 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce8977c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840b8f60 con 0x7fce840ce260 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce83fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce841608f0 0x7fce840d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38206/0 (socket says 172.21.15.107:38206) 2023-11-24T08:14:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce83fff640 1 -- 172.21.15.107:0/518743025 learned_addr learned my addr 172.21.15.107:0/518743025 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.449+0000 7fce837fe640 1 --2- 172.21.15.107:0/518743025 >> v2:172.21.15.107:3300/0 conn(0x7fce840ce260 0x7fce840d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.450+0000 7fce82ffd640 1 -- 172.21.15.107:0/518743025 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071382872 0 0) 0x7fce840d21d0 con 0x7fce841608f0 2023-11-24T08:14:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.450+0000 7fce82ffd640 1 -- 172.21.15.107:0/518743025 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce70003520 con 0x7fce841608f0 2023-11-24T08:14:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.450+0000 7fce837fe640 1 -- 172.21.15.107:0/518743025 >> v1:172.21.15.120:6789/0 conn(0x7fce840ce710 legacy=0x7fce840b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.450+0000 7fce837fe640 1 -- 172.21.15.107:0/518743025 >> v1:172.21.15.107:6789/0 conn(0x7fce841608f0 legacy=0x7fce840d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.450+0000 7fce837fe640 1 -- 172.21.15.107:0/518743025 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840cdcb0 con 0x7fce840ce260 2023-11-24T08:14:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.451+0000 7fce837fe640 1 --2- 172.21.15.107:0/518743025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce840d3180 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0097a0 tx=0x7fce6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c450d05099f8c2a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce82ffd640 1 -- 172.21.15.107:0/518743025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c031830 con 0x7fce840ce260 2023-11-24T08:14:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce82ffd640 1 -- 172.21.15.107:0/518743025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c031990 con 0x7fce840ce260 2023-11-24T08:14:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce82ffd640 1 -- 172.21.15.107:0/518743025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c031d00 con 0x7fce840ce260 2023-11-24T08:14:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 -- 172.21.15.107:0/518743025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 msgr2=0x7fce840d3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 --2- 172.21.15.107:0/518743025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce840d3180 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0097a0 tx=0x7fce6c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 -- 172.21.15.107:0/518743025 shutdown_connections 2023-11-24T08:14:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 --2- 172.21.15.107:0/518743025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce840d3180 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 -- 172.21.15.107:0/518743025 >> 172.21.15.107:0/518743025 conn(0x7fce84061340 msgr2=0x7fce84061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 -- 172.21.15.107:0/518743025 shutdown_connections 2023-11-24T08:14:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.452+0000 7fce8977c640 1 -- 172.21.15.107:0/518743025 wait complete. 2023-11-24T08:14:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.454+0000 7fce8977c640 1 Processor -- start 2023-11-24T08:14:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.454+0000 7fce8977c640 1 -- start start 2023-11-24T08:14:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce840ce710 0x7fce840c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce841608f0 0x7fce840c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce840c7c70 con 0x7fce840ce260 2023-11-24T08:14:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce840ca370 con 0x7fce840ce710 2023-11-24T08:14:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce8977c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840c9ac0 con 0x7fce841608f0 2023-11-24T08:14:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.455+0000 7fce83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44722/0 (socket says 172.21.15.107:44722) 2023-11-24T08:14:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 -- 172.21.15.107:0/2737473023 learned_addr learned my addr 172.21.15.107:0/2737473023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 -- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce841608f0 msgr2=0x7fce840c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce841608f0 0x7fce840c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 -- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce840ce710 msgr2=0x7fce840c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce840ce710 0x7fce840c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 -- 172.21.15.107:0/2737473023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c009400 con 0x7fce840ce260 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce83fff640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fce7400caa0 tx=0x7fce7400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7400ac00 con 0x7fce840ce260 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7400ad60 con 0x7fce840ce260 2023-11-24T08:14:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74012620 con 0x7fce840ce260 2023-11-24T08:14:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.456+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce840c7990 con 0x7fce840ce260 2023-11-24T08:14:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.457+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce840ca860 con 0x7fce840ce260 2023-11-24T08:14:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.458+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8404e840 con 0x7fce840ce260 2023-11-24T08:14:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.459+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fce74007070 con 0x7fce840ce260 2023-11-24T08:14:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.459+0000 7fce80ff9640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 0x7fce6406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.460+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce740884d0 con 0x7fce840ce260 2023-11-24T08:14:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.460+0000 7fce837fe640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 0x7fce6406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.460+0000 7fce837fe640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 0x7fce6406e500 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fce6c036000 tx=0x7fce6c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.464+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce74054cc0 con 0x7fce840ce260 2023-11-24T08:14:05.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.653+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce840d20b0 con 0x7fce840ce260 2023-11-24T08:14:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.654+0000 7fce80ff9640 1 -- 172.21.15.107:0/2737473023 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce74017070 con 0x7fce840ce260 2023-11-24T08:14:05.656 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:14:05.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 msgr2=0x7fce6406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 0x7fce6406e500 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fce6c036000 tx=0x7fce6c030000 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 msgr2=0x7fce841559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fce7400caa0 tx=0x7fce7400cf60 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 shutdown_connections 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce6406c040 0x7fce6406e500 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce841608f0 0x7fce840c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce840ce710 0x7fce840c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 --2- 172.21.15.107:0/2737473023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce840ce260 0x7fce841559f0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.656+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 >> 172.21.15.107:0/2737473023 conn(0x7fce84061340 msgr2=0x7fce840b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:05.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.657+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 shutdown_connections 2023-11-24T08:14:05.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.657+0000 7fce8977c640 1 -- 172.21.15.107:0/2737473023 wait complete. 2023-11-24T08:14:05.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:14:05.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.895+0000 7fdf0b08c640 1 Processor -- start 2023-11-24T08:14:05.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.896+0000 7fdf0b08c640 1 -- start start 2023-11-24T08:14:05.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0b08c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf0415d7d0 0x7fdf040cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0b08c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040d15e0 con 0x7fdf0415d3f0 2023-11-24T08:14:05.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0b08c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040d0280 con 0x7fdf040d1f40 2023-11-24T08:14:05.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0b08c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040b0eb0 con 0x7fdf0415d7d0 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf09889640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf0415d7d0 0x7fdf040cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0a08a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdf0415d3f0 0x7fdf040d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38216/0 (socket says 172.21.15.107:38216) 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.897+0000 7fdf0a08a640 1 -- 172.21.15.107:0/2682524598 learned_addr learned my addr 172.21.15.107:0/2682524598 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.898+0000 7fdf09889640 1 -- 172.21.15.107:0/2682524598 >> v1:172.21.15.120:6789/0 conn(0x7fdf040d1f40 legacy=0x7fdf040ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.898+0000 7fdf09889640 1 -- 172.21.15.107:0/2682524598 >> v1:172.21.15.107:6789/0 conn(0x7fdf0415d3f0 legacy=0x7fdf040d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.898+0000 7fdf09889640 1 -- 172.21.15.107:0/2682524598 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040d1300 con 0x7fdf0415d7d0 2023-11-24T08:14:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.898+0000 7fdf09088640 1 -- 172.21.15.107:0/2682524598 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391815156 0 0) 0x7fdf040d15e0 con 0x7fdf0415d3f0 2023-11-24T08:14:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.899+0000 7fdf09889640 1 --2- 172.21.15.107:0/2682524598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf0415d7d0 0x7fdf040cdd40 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdef00097a0 tx=0x7fdef002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1b674c63c9595b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.899+0000 7fdf09088640 1 -- 172.21.15.107:0/2682524598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0031830 con 0x7fdf0415d7d0 2023-11-24T08:14:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.899+0000 7fdf09088640 1 -- 172.21.15.107:0/2682524598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef0031990 con 0x7fdf0415d7d0 2023-11-24T08:14:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.899+0000 7fdf09088640 1 -- 172.21.15.107:0/2682524598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0031d20 con 0x7fdf0415d7d0 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 -- 172.21.15.107:0/2682524598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf0415d7d0 msgr2=0x7fdf040cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/2682524598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf0415d7d0 0x7fdf040cdd40 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdef00097a0 tx=0x7fdef002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 -- 172.21.15.107:0/2682524598 shutdown_connections 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/2682524598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf0415d7d0 0x7fdf040cdd40 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 -- 172.21.15.107:0/2682524598 >> 172.21.15.107:0/2682524598 conn(0x7fdf0409c5b0 msgr2=0x7fdf040d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 -- 172.21.15.107:0/2682524598 shutdown_connections 2023-11-24T08:14:05.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.900+0000 7fdf0b08c640 1 -- 172.21.15.107:0/2682524598 wait complete. 2023-11-24T08:14:05.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.902+0000 7fdf0b08c640 1 Processor -- start 2023-11-24T08:14:05.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.902+0000 7fdf0b08c640 1 -- start start 2023-11-24T08:14:05.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf0415d3f0 0x7fdf0415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf0415d7d0 0x7fdf0415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b75c0 con 0x7fdf040d1f40 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b78a0 con 0x7fdf0415d7d0 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0b08c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b7b80 con 0x7fdf0415d3f0 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44738/0 (socket says 172.21.15.107:44738) 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 -- 172.21.15.107:0/1867272174 learned_addr learned my addr 172.21.15.107:0/1867272174 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 -- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf0415d3f0 msgr2=0x7fdf0415b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf0415d3f0 0x7fdf0415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a88b640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf0415d7d0 0x7fdf0415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 -- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf0415d7d0 msgr2=0x7fdf0415bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf0415d7d0 0x7fdf0415bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.903+0000 7fdf0a08a640 1 -- 172.21.15.107:0/1867272174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef0009400 con 0x7fdf040d1f40 2023-11-24T08:14:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf0a08a640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c890 tx=0x7fdef800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef800a640 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef800a7c0 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef800ab70 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040b7e60 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.904+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040b8ba0 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.905+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf04067990 con 0x7fdf040d1f40 2023-11-24T08:14:05.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.906+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fdef800acd0 con 0x7fdf040d1f40 2023-11-24T08:14:05.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.908+0000 7fdf02ffd640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 0x7fdee806e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:05.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.908+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdef8087fb0 con 0x7fdf040d1f40 2023-11-24T08:14:05.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.908+0000 7fdf09889640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 0x7fdee806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:05.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.909+0000 7fdf09889640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 0x7fdee806e550 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdef0036000 tx=0x7fdef0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:05.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:05.912+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdef8054740 con 0x7fdf040d1f40 2023-11-24T08:14:06.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.099+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf040d06f0 con 0x7fdf040d1f40 2023-11-24T08:14:06.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.100+0000 7fdf02ffd640 1 -- 172.21.15.107:0/1867272174 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf040d06f0 con 0x7fdf040d1f40 2023-11-24T08:14:06.103 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:14:06.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 msgr2=0x7fdee806e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 0x7fdee806e550 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdef0036000 tx=0x7fdef0030000 comp rx=0 tx=0).stop 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 msgr2=0x7fdf040b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c890 tx=0x7fdef800cd50 comp rx=0 tx=0).stop 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 shutdown_connections 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdee806c090 0x7fdee806e550 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf0415d7d0 0x7fdf0415bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf0415d3f0 0x7fdf0415b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 --2- 172.21.15.107:0/1867272174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf040d1f40 0x7fdf040b8660 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 >> 172.21.15.107:0/1867272174 conn(0x7fdf0409c5b0 msgr2=0x7fdf040d1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:06.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 shutdown_connections 2023-11-24T08:14:06.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.103+0000 7fdf0b08c640 1 -- 172.21.15.107:0/1867272174 wait complete. 2023-11-24T08:14:06.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:14:06.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.332+0000 7fee1359e640 1 Processor -- start 2023-11-24T08:14:06.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.332+0000 7fee1359e640 1 -- start start 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee14070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee140b15e0 0x7fee1415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140b1a20 con 0x7fee1406d640 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140cd920 con 0x7fee140b15e0 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415e2e0 con 0x7fee1415a1c0 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee12d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fee1406d640 0x7fee140ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38232/0 (socket says 172.21.15.107:38232) 2023-11-24T08:14:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee12d9d640 1 -- 172.21.15.107:0/1615846032 learned_addr learned my addr 172.21.15.107:0/1615846032 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee0bfff640 1 --2- 172.21.15.107:0/1615846032 >> v2:172.21.15.120:3300/0 conn(0x7fee140b15e0 0x7fee1415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723200416 0 0) 0x7fee0c001490 con 0x7fee1406d640 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.334+0000 7fee1259c640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee14070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee00005730 con 0x7fee1406d640 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124397338 0 0) 0x7fee00005730 con 0x7fee1406d640 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 msgr2=0x7fee14070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:06.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee14070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 >> v2:172.21.15.120:3300/0 conn(0x7fee140b15e0 msgr2=0x7fee1415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 --2- 172.21.15.107:0/1615846032 >> v2:172.21.15.120:3300/0 conn(0x7fee140b15e0 0x7fee1415dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1415e580 con 0x7fee1406d640 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fee0c001b80 con 0x7fee1406d640 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee0bfff640 1 --2- 172.21.15.107:0/1615846032 >> v2:172.21.15.120:3300/0 conn(0x7fee140b15e0 0x7fee1415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 >> v1:172.21.15.107:6789/0 conn(0x7fee1406d640 legacy=0x7fee140ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee00005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee00006100 0x7fee000069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee00007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140cd920 con 0x7fee00006f10 2023-11-24T08:14:06.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415e2e0 con 0x7fee00006100 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0000dc70 con 0x7fee1415a1c0 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.335+0000 7fee12d9d640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee00007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee1259c640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee00005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee0bfff640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee00006100 0x7fee000069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee12d9d640 1 -- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 msgr2=0x7fee00005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee12d9d640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee00005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee12d9d640 1 -- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee00006100 msgr2=0x7fee000069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee12d9d640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee00006100 0x7fee000069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee12d9d640 1 -- 172.21.15.107:0/1615846032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00005730 con 0x7fee00006f10 2023-11-24T08:14:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.336+0000 7fee1259c640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415a1c0 0x7fee00005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:06.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.337+0000 7fee12d9d640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee00007390 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00a830 tx=0x7fee00007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c621823deb58bba5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.337+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c034630 con 0x7fee00006f10 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.337+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c02f070 con 0x7fee00006f10 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.337+0000 7fee11d9b640 1 -- 172.21.15.107:0/1615846032 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c034a30 con 0x7fee00006f10 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 -- 172.21.15.107:0/1615846032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 msgr2=0x7fee00007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 --2- 172.21.15.107:0/1615846032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee00007390 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00a830 tx=0x7fee00007a00 comp rx=0 tx=0).stop 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee12d9d640 1 -- 172.21.15.107:0/1615846032 reap_dead start 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 -- 172.21.15.107:0/1615846032 shutdown_connections 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 -- 172.21.15.107:0/1615846032 >> 172.21.15.107:0/1615846032 conn(0x7fee140cbb40 msgr2=0x7fee140cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 -- 172.21.15.107:0/1615846032 shutdown_connections 2023-11-24T08:14:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.338+0000 7fee1359e640 1 -- 172.21.15.107:0/1615846032 wait complete. 2023-11-24T08:14:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.340+0000 7fee1359e640 1 Processor -- start 2023-11-24T08:14:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.340+0000 7fee1359e640 1 -- start start 2023-11-24T08:14:06.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee14154380 0x7fee1415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415bb40 0x7fee14153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415c050 con 0x7fee00006f10 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415c330 con 0x7fee14154380 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415c610 con 0x7fee1415bb40 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44762/0 (socket says 172.21.15.107:44762) 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 -- 172.21.15.107:0/4018630505 learned_addr learned my addr 172.21.15.107:0/4018630505 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 -- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415bb40 msgr2=0x7fee14153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415bb40 0x7fee14153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 -- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee14154380 msgr2=0x7fee1415b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee14154380 0x7fee1415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.341+0000 7fee1259c640 1 -- 172.21.15.107:0/4018630505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1415c8f0 con 0x7fee00006f10 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee1259c640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fee04010830 tx=0x7fee04010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0400eb60 con 0x7fee00006f10 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0400ecc0 con 0x7fee00006f10 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0401c400 con 0x7fee00006f10 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140b1bc0 con 0x7fee00006f10 2023-11-24T08:14:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.342+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee14156740 con 0x7fee00006f10 2023-11-24T08:14:06.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.343+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140b1d50 con 0x7fee00006f10 2023-11-24T08:14:06.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.344+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fee0401c560 con 0x7fee00006f10 2023-11-24T08:14:06.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.345+0000 7fee0affd640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 0x7fedf406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:06.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.345+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee04088a70 con 0x7fee00006f10 2023-11-24T08:14:06.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.345+0000 7fee0bfff640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 0x7fedf406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:06.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.346+0000 7fee0bfff640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 0x7fedf406e500 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fedfc006f00 tx=0x7fedfc006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.349+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee04055170 con 0x7fee00006f10 2023-11-24T08:14:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.554+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee14156740 con 0x7fee00006f10 2023-11-24T08:14:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.554+0000 7fee0affd640 1 -- 172.21.15.107:0/4018630505 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee04021020 con 0x7fee00006f10 2023-11-24T08:14:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:14:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 msgr2=0x7fedf406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 0x7fedf406e500 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fedfc006f00 tx=0x7fedfc006df0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 msgr2=0x7fee14156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fee04010830 tx=0x7fee04010cf0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 shutdown_connections 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fedf406c040 0x7fedf406e500 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee1415bb40 0x7fee14153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee14154380 0x7fee1415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 --2- 172.21.15.107:0/4018630505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee00006f10 0x7fee14156020 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 >> 172.21.15.107:0/4018630505 conn(0x7fee140cbb40 msgr2=0x7fee14159840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 shutdown_connections 2023-11-24T08:14:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:06.557+0000 7fee1359e640 1 -- 172.21.15.107:0/4018630505 wait complete. 2023-11-24T08:14:06.577 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:14:06.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:06.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:14:06.679 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:14:06.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:06.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:14:06.730 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:06.730 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:06.789 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:06.789 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:06.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:07.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.064+0000 7feb0e935640 1 Processor -- start 2023-11-24T08:14:07.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.064+0000 7feb0e935640 1 -- start start 2023-11-24T08:14:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.065+0000 7feb0e935640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb08144b30 0x7feb0815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.065+0000 7feb0e935640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0815bf80 con 0x7feb080b1de0 2023-11-24T08:14:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.065+0000 7feb0e935640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080b1560 con 0x7feb0815c410 2023-11-24T08:14:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.065+0000 7feb0e935640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb080b1280 con 0x7feb08144b30 2023-11-24T08:14:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.066+0000 7feb0e134640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feb080b1de0 0x7feb080b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38236/0 (socket says 172.21.15.107:38236) 2023-11-24T08:14:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.066+0000 7feb0e134640 1 -- 172.21.15.107:0/3425880512 learned_addr learned my addr 172.21.15.107:0/3425880512 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.066+0000 7feb0d933640 1 --2- 172.21.15.107:0/3425880512 >> v2:172.21.15.120:3300/0 conn(0x7feb08144b30 0x7feb0815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.066+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088480679 0 0) 0x7feb0815bf80 con 0x7feb080b1de0 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.066+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf4003520 con 0x7feb080b1de0 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584981629 0 0) 0x7feaf4003520 con 0x7feb080b1de0 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 >> v2:172.21.15.120:3300/0 conn(0x7feb08144b30 msgr2=0x7feb0815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 --2- 172.21.15.107:0/3425880512 >> v2:172.21.15.120:3300/0 conn(0x7feb08144b30 0x7feb0815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 >> v1:172.21.15.120:6789/0 conn(0x7feb0815c410 legacy=0x7feb080b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0815b930 con 0x7feb080b1de0 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7feb000019f0 con 0x7feb080b1de0 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 >> v1:172.21.15.107:6789/0 conn(0x7feb080b1de0 legacy=0x7feb080b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40039c0 0x7feaf4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 0x7feaf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf4005180 0x7feaf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080b1280 con 0x7feaf40047c0 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf400c080 con 0x7feaf40039c0 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf400c240 con 0x7feaf4005180 2023-11-24T08:14:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d132640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 0x7feaf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d933640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40039c0 0x7feaf4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0e134640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf4005180 0x7feaf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d132640 1 -- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf4005180 msgr2=0x7feaf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d132640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf4005180 0x7feaf400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d132640 1 -- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40039c0 msgr2=0x7feaf4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.067+0000 7feb0d132640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40039c0 0x7feaf4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.068+0000 7feb0d132640 1 -- 172.21.15.107:0/3425880512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4003520 con 0x7feaf40047c0 2023-11-24T08:14:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.068+0000 7feb0e134640 1 -- 172.21.15.107:0/3425880512 reap_dead start 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0d132640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 0x7feaf4004c40 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7feaf8001750 tx=0x7feaf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124cf53e2bbdf258 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf80052d0 con 0x7feaf40047c0 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf802e050 con 0x7feaf40047c0 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0c931640 1 -- 172.21.15.107:0/3425880512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8003830 con 0x7feaf40047c0 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0e935640 1 -- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 msgr2=0x7feaf4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0e935640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 0x7feaf4004c40 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7feaf8001750 tx=0x7feaf40057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0e935640 1 -- 172.21.15.107:0/3425880512 shutdown_connections 2023-11-24T08:14:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0e935640 1 --2- 172.21.15.107:0/3425880512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feaf40047c0 0x7feaf4004c40 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.069+0000 7feb0e935640 1 -- 172.21.15.107:0/3425880512 >> 172.21.15.107:0/3425880512 conn(0x7feb080cb830 msgr2=0x7feb080cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.070+0000 7feb0e935640 1 -- 172.21.15.107:0/3425880512 shutdown_connections 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.070+0000 7feb0e935640 1 -- 172.21.15.107:0/3425880512 wait complete. 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.071+0000 7feb0e935640 1 Processor -- start 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.071+0000 7feb0e935640 1 -- start start 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40047c0 0x7feb080c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb0815a1d0 0x7feb080c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb081f4d10 con 0x7feb081f4350 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080c56a0 con 0x7feaf40047c0 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e935640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080c6500 con 0x7feb0815a1d0 2023-11-24T08:14:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.072+0000 7feb0e134640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44774/0 (socket says 172.21.15.107:44774) 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 -- 172.21.15.107:0/2156463420 learned_addr learned my addr 172.21.15.107:0/2156463420 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 -- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb0815a1d0 msgr2=0x7feb080c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb0815a1d0 0x7feb080c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 -- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40047c0 msgr2=0x7feb080c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40047c0 0x7feb080c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 -- 172.21.15.107:0/2156463420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080c58f0 con 0x7feb081f4350 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e134640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7feb0815c7f0 tx=0x7feb0000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00004d60 con 0x7feb081f4350 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0000bb80 con 0x7feb081f4350 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00007070 con 0x7feb081f4350 2023-11-24T08:14:07.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.073+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb080c5bd0 con 0x7feb081f4350 2023-11-24T08:14:07.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.074+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb080c6190 con 0x7feb081f4350 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.075+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb08067990 con 0x7feb081f4350 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.075+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7feb000071d0 con 0x7feb081f4350 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.076+0000 7feafe7fc640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 0x7feaec06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.076+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb00088860 con 0x7feb081f4350 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.077+0000 7feb0d933640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 0x7feaec06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.077+0000 7feb0d933640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 0x7feaec06e500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feb04009990 tx=0x7feb04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.081+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb00055110 con 0x7feb081f4350 2023-11-24T08:14:07.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.271+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb0804e840 con 0x7feb081f4350 2023-11-24T08:14:07.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.272+0000 7feafe7fc640 1 -- 172.21.15.107:0/2156463420 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7feb00058730 con 0x7feb081f4350 2023-11-24T08:14:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:07.275 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.275+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 msgr2=0x7feaec06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.275+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 0x7feaec06e500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feb04009990 tx=0x7feb04008040 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.275+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 msgr2=0x7feb081f47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.275+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7feb0815c7f0 tx=0x7feb0000b4a0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.275+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 shutdown_connections 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feaec06c040 0x7feaec06e500 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb081f4350 0x7feb081f47d0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb0815a1d0 0x7feb080c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 --2- 172.21.15.107:0/2156463420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf40047c0 0x7feb080c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 >> 172.21.15.107:0/2156463420 conn(0x7feb080cb830 msgr2=0x7feb080cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 shutdown_connections 2023-11-24T08:14:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.276+0000 7feb0e935640 1 -- 172.21.15.107:0/2156463420 wait complete. 2023-11-24T08:14:07.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:13:33.870908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:13:35.060534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:14:07.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:14:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.505+0000 7fc354fef640 1 Processor -- start 2023-11-24T08:14:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.505+0000 7fc354fef640 1 -- start start 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.506+0000 7fc354fef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc3500b0b40 0x7fc3500b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc354fef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc35015ded0 0x7fc35015e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc354fef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35006d430 con 0x7fc35015a1c0 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc354fef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35015ec50 con 0x7fc35015ded0 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc354fef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35006cb00 con 0x7fc3500b0b40 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc35015a1c0 0x7fc35006d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38244/0 (socket says 172.21.15.107:38244) 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34f7fe640 1 -- 172.21.15.107:0/2905780700 learned_addr learned my addr 172.21.15.107:0/2905780700 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34ffff640 1 --2- 172.21.15.107:0/2905780700 >> v2:172.21.15.107:3300/0 conn(0x7fc35015ded0 0x7fc35015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34effd640 1 --2- 172.21.15.107:0/2905780700 >> v2:172.21.15.120:3300/0 conn(0x7fc3500b0b40 0x7fc3500b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34e7fc640 1 -- 172.21.15.107:0/2905780700 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896787355 0 0) 0x7fc35006d430 con 0x7fc35015a1c0 2023-11-24T08:14:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.507+0000 7fc34e7fc640 1 -- 172.21.15.107:0/2905780700 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc338003520 con 0x7fc35015a1c0 2023-11-24T08:14:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.508+0000 7fc34ffff640 1 -- 172.21.15.107:0/2905780700 >> v2:172.21.15.120:3300/0 conn(0x7fc3500b0b40 msgr2=0x7fc3500b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.508+0000 7fc34ffff640 1 --2- 172.21.15.107:0/2905780700 >> v2:172.21.15.120:3300/0 conn(0x7fc3500b0b40 0x7fc3500b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.508+0000 7fc34ffff640 1 -- 172.21.15.107:0/2905780700 >> v1:172.21.15.107:6789/0 conn(0x7fc35015a1c0 legacy=0x7fc35006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.508+0000 7fc34ffff640 1 -- 172.21.15.107:0/2905780700 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500b14e0 con 0x7fc35015ded0 2023-11-24T08:14:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.509+0000 7fc34ffff640 1 --2- 172.21.15.107:0/2905780700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc35015e360 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc3500cee20 tx=0x7fc34802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ee30b397c95d64 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.509+0000 7fc34e7fc640 1 -- 172.21.15.107:0/2905780700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34802cc80 con 0x7fc35015ded0 2023-11-24T08:14:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.509+0000 7fc34e7fc640 1 -- 172.21.15.107:0/2905780700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34802cde0 con 0x7fc35015ded0 2023-11-24T08:14:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.509+0000 7fc34e7fc640 1 -- 172.21.15.107:0/2905780700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348034660 con 0x7fc35015ded0 2023-11-24T08:14:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 -- 172.21.15.107:0/2905780700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 msgr2=0x7fc35015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 --2- 172.21.15.107:0/2905780700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc35015e360 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc3500cee20 tx=0x7fc34802bed0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 -- 172.21.15.107:0/2905780700 shutdown_connections 2023-11-24T08:14:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 --2- 172.21.15.107:0/2905780700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc35015e360 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 --2- 172.21.15.107:0/2905780700 >> v2:172.21.15.120:3300/0 conn(0x7fc3500b0b40 0x7fc3500b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 -- 172.21.15.107:0/2905780700 >> 172.21.15.107:0/2905780700 conn(0x7fc3500cbb40 msgr2=0x7fc3500cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 -- 172.21.15.107:0/2905780700 shutdown_connections 2023-11-24T08:14:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.510+0000 7fc354fef640 1 -- 172.21.15.107:0/2905780700 wait complete. 2023-11-24T08:14:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.512+0000 7fc354fef640 1 Processor -- start 2023-11-24T08:14:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.512+0000 7fc354fef640 1 -- start start 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3500b0b40 0x7fc3501f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc35015a1c0 0x7fc35009af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35015d0f0 con 0x7fc35015ded0 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35015d390 con 0x7fc35015a1c0 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc354fef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35015d670 con 0x7fc3500b0b40 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44784/0 (socket says 172.21.15.107:44784) 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 -- 172.21.15.107:0/3832814262 learned_addr learned my addr 172.21.15.107:0/3832814262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34effd640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc35015a1c0 0x7fc35009af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 -- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3500b0b40 msgr2=0x7fc3501f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3500b0b40 0x7fc3501f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 -- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc35015a1c0 msgr2=0x7fc35009af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.513+0000 7fc34ffff640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc35015a1c0 0x7fc35009af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc34ffff640 1 -- 172.21.15.107:0/3832814262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348009a70 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc34ffff640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc348009e80 tx=0x7fc348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348040650 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3480407b0 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348040b60 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35015ea20 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.514+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3501f2dc0 con 0x7fc35015ded0 2023-11-24T08:14:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.515+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3501f3380 con 0x7fc35015ded0 2023-11-24T08:14:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.516+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc348040cc0 con 0x7fc35015ded0 2023-11-24T08:14:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.518+0000 7fc337fff640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 0x7fc33c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.518+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3480aaea0 con 0x7fc35015ded0 2023-11-24T08:14:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.518+0000 7fc34f7fe640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 0x7fc33c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.519+0000 7fc34f7fe640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 0x7fc33c06e500 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc340001610 tx=0x7fc340007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.522+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3501f3380 con 0x7fc35015ded0 2023-11-24T08:14:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.718+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3501f3380 con 0x7fc35015ded0 2023-11-24T08:14:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.718+0000 7fc337fff640 1 -- 172.21.15.107:0/3832814262 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3501f3380 con 0x7fc35015ded0 2023-11-24T08:14:07.720 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:14:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 msgr2=0x7fc33c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 0x7fc33c06e500 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc340001610 tx=0x7fc340007930 comp rx=0 tx=0).stop 2023-11-24T08:14:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 msgr2=0x7fc3501f3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc348009e80 tx=0x7fc348004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 shutdown_connections 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc33c06c040 0x7fc33c06e500 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc35015ded0 0x7fc3501f3660 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc35015a1c0 0x7fc35009af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 --2- 172.21.15.107:0/3832814262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3500b0b40 0x7fc3501f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 >> 172.21.15.107:0/3832814262 conn(0x7fc3500cbb40 msgr2=0x7fc3500b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 shutdown_connections 2023-11-24T08:14:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.721+0000 7fc354fef640 1 -- 172.21.15.107:0/3832814262 wait complete. 2023-11-24T08:14:07.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:14:07.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.947+0000 7f8fe2d16640 1 Processor -- start 2023-11-24T08:14:07.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.948+0000 7f8fe2d16640 1 -- start start 2023-11-24T08:14:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.949+0000 7f8fe2d16640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fdc06f360 0x7f8fdc0cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.949+0000 7f8fe2d16640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc06bb90 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.949+0000 7f8fe2d16640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc06be70 con 0x7f8fdc15d380 2023-11-24T08:14:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.949+0000 7f8fe2d16640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fdc06b980 con 0x7f8fdc06f360 2023-11-24T08:14:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.949+0000 7f8fe12b3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8fdc0ce2c0 0x7f8fdc070100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34650/0 (socket says 172.21.15.107:34650) 2023-11-24T08:14:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fe12b3640 1 -- 172.21.15.107:0/3191525540 learned_addr learned my addr 172.21.15.107:0/3191525540 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3191525540 >> v2:172.21.15.120:3300/0 conn(0x7f8fdc06f360 0x7f8fdc0cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957361487 0 0) 0x7f8fdc06bb90 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc8003520 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832107619 0 0) 0x7f8fc8003520 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 >> v2:172.21.15.120:3300/0 conn(0x7f8fdc06f360 msgr2=0x7f8fdc0cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3191525540 >> v2:172.21.15.120:3300/0 conn(0x7f8fdc06f360 0x7f8fdc0cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 >> v1:172.21.15.120:6789/0 conn(0x7f8fdc15d380 legacy=0x7f8fdc0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc06c150 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8fd4001670 con 0x7f8fdc0ce2c0 2023-11-24T08:14:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.950+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 >> v1:172.21.15.107:6789/0 conn(0x7f8fdc0ce2c0 legacy=0x7f8fdc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80039c0 0x7f8fc8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 0x7f8fc8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc8005180 0x7f8fc800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc06b980 con 0x7f8fc80047c0 2023-11-24T08:14:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc800c080 con 0x7f8fc8005180 2023-11-24T08:14:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc800c240 con 0x7f8fc80039c0 2023-11-24T08:14:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 0x7f8fc8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80039c0 msgr2=0x7f8fc8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fe12b3640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc8005180 0x7f8fc800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80039c0 0x7f8fc8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc8005180 msgr2=0x7f8fc800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc8005180 0x7f8fc800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fe12b3640 1 -- 172.21.15.107:0/3191525540 reap_dead start 2023-11-24T08:14:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.951+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3191525540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8003520 con 0x7f8fc80047c0 2023-11-24T08:14:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.952+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 0x7f8fc8004c40 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009ac0 tx=0x7f8fc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e7a7bbb3cd7395 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc402c6b0 con 0x7f8fc80047c0 2023-11-24T08:14:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc402c810 con 0x7f8fc80047c0 2023-11-24T08:14:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3191525540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc402cba0 con 0x7f8fc80047c0 2023-11-24T08:14:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 msgr2=0x7f8fc8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 0x7f8fc8004c40 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009ac0 tx=0x7f8fc80057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3191525540 shutdown_connections 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3191525540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fc80047c0 0x7f8fc8004c40 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3191525540 >> 172.21.15.107:0/3191525540 conn(0x7f8fdc06e7d0 msgr2=0x7f8fdc06ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3191525540 shutdown_connections 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.953+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3191525540 wait complete. 2023-11-24T08:14:07.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.955+0000 7f8fe2d16640 1 Processor -- start 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.955+0000 7f8fe2d16640 1 -- start start 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80047c0 0x7f8fdc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fdc1f4210 0x7f8fdc1f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0c5ee0 con 0x7f8fdc0c9010 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0c5640 con 0x7f8fdc1f4210 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.956+0000 7f8fe2d16640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0c5920 con 0x7f8fc80047c0 2023-11-24T08:14:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41448/0 (socket says 172.21.15.107:41448) 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3556533898 learned_addr learned my addr 172.21.15.107:0/3556533898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80047c0 msgr2=0x7f8fdc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80047c0 0x7f8fdc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fdc1f4210 msgr2=0x7f8fdc1f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fdc1f4210 0x7f8fdc1f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 -- 172.21.15.107:0/3556533898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0c5c00 con 0x7f8fdc0c9010 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.957+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c250 tx=0x7f8fc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.958+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003810 con 0x7f8fdc0c9010 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.958+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4003970 con 0x7f8fdc0c9010 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.958+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc1f3cf0 con 0x7f8fdc0c9010 2023-11-24T08:14:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.958+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003d00 con 0x7f8fdc0c9010 2023-11-24T08:14:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.958+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc1f39c0 con 0x7f8fdc0c9010 2023-11-24T08:14:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.959+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc04e8b0 con 0x7f8fdc0c9010 2023-11-24T08:14:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.960+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f8fc4047450 con 0x7f8fdc0c9010 2023-11-24T08:14:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.961+0000 7f8fd97fa640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 0x7f8fbc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.961+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fc40aa540 con 0x7f8fdc0c9010 2023-11-24T08:14:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.961+0000 7f8fe0ab2640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 0x7f8fbc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.962+0000 7f8fe0ab2640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 0x7f8fbc06e500 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc001c30 tx=0x7f8fcc007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:07.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:07.965+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8fc40770f0 con 0x7f8fdc0c9010 2023-11-24T08:14:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.155+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fdc0c9780 con 0x7f8fdc0c9010 2023-11-24T08:14:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.156+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3556533898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8fc4003e60 con 0x7f8fdc0c9010 2023-11-24T08:14:08.158 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 msgr2=0x7f8fbc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 0x7f8fbc06e500 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc001c30 tx=0x7f8fcc007d50 comp rx=0 tx=0).stop 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 msgr2=0x7f8fdc1f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c250 tx=0x7f8fc4004730 comp rx=0 tx=0).stop 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 shutdown_connections 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fbc06c040 0x7f8fbc06e500 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fdc1f4210 0x7f8fdc1f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc0c9010 0x7f8fdc1f3480 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 --2- 172.21.15.107:0/3556533898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc80047c0 0x7f8fdc0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 >> 172.21.15.107:0/3556533898 conn(0x7f8fdc06e7d0 msgr2=0x7f8fdc0b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:08.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 shutdown_connections 2023-11-24T08:14:08.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.159+0000 7f8fe2d16640 1 -- 172.21.15.107:0/3556533898 wait complete. 2023-11-24T08:14:08.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:14:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.387+0000 7f558bfff640 1 Processor -- start 2023-11-24T08:14:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.387+0000 7f558bfff640 1 -- start start 2023-11-24T08:14:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0b0b40 0x7f558c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f558c15ded0 0x7f558c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558bfff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c06d430 con 0x7f558c15a1c0 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558bfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c15ec50 con 0x7f558c15ded0 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c06cb00 con 0x7f558c0b0b40 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558b7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f558c15ded0 0x7f558c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558a7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0b0b40 0x7f558c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558b7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f558c15ded0 0x7f558c15e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:44404/0 (socket says 172.21.15.107:44404) 2023-11-24T08:14:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.389+0000 7f558b7fe640 1 -- 172.21.15.107:0/4294789523 learned_addr learned my addr 172.21.15.107:0/4294789523 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.390+0000 7f558b7fe640 1 -- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0b0b40 msgr2=0x7f558c0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.390+0000 7f558b7fe640 1 --2- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0b0b40 0x7f558c0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.390+0000 7f558b7fe640 1 -- 172.21.15.107:0/4294789523 >> v1:172.21.15.120:6789/0 conn(0x7f558c15a1c0 legacy=0x7f558c06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.390+0000 7f558b7fe640 1 -- 172.21.15.107:0/4294789523 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0b14e0 con 0x7f558c15ded0 2023-11-24T08:14:08.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.391+0000 7f558b7fe640 1 --2- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15ded0 0x7f558c15e360 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5584009e50 tx=0x7f558402e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de90d3c87240463e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.392+0000 7f5589ffb640 1 -- 172.21.15.107:0/4294789523 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558402ed00 con 0x7f558c15ded0 2023-11-24T08:14:08.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.392+0000 7f5589ffb640 1 -- 172.21.15.107:0/4294789523 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558402ee60 con 0x7f558c15ded0 2023-11-24T08:14:08.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.392+0000 7f5589ffb640 1 -- 172.21.15.107:0/4294789523 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584036610 con 0x7f558c15ded0 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.392+0000 7f558bfff640 1 -- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15ded0 msgr2=0x7f558c15e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.392+0000 7f558bfff640 1 --2- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15ded0 0x7f558c15e360 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5584009e50 tx=0x7f558402e160 comp rx=0 tx=0).stop 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 -- 172.21.15.107:0/4294789523 shutdown_connections 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 --2- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15ded0 0x7f558c15e360 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 --2- 172.21.15.107:0/4294789523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0b0b40 0x7f558c0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 -- 172.21.15.107:0/4294789523 >> 172.21.15.107:0/4294789523 conn(0x7f558c0cbb40 msgr2=0x7f558c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 -- 172.21.15.107:0/4294789523 shutdown_connections 2023-11-24T08:14:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.393+0000 7f558bfff640 1 -- 172.21.15.107:0/4294789523 wait complete. 2023-11-24T08:14:08.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.394+0000 7f558bfff640 1 Processor -- start 2023-11-24T08:14:08.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 -- start start 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15a1c0 0x7f558c159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c15ded0 0x7f558c159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0c7010 con 0x7f558c0b0b40 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0c72f0 con 0x7f558c15a1c0 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.395+0000 7f558bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0c75d0 con 0x7f558c15ded0 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41456/0 (socket says 172.21.15.107:41456) 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 -- 172.21.15.107:0/41773073 learned_addr learned my addr 172.21.15.107:0/41773073 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558a7fc640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15a1c0 0x7f558c159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558b7fe640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c15ded0 0x7f558c159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 -- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c15ded0 msgr2=0x7f558c159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c15ded0 0x7f558c159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 -- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15a1c0 msgr2=0x7f558c159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15a1c0 0x7f558c159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.396+0000 7f558affd640 1 -- 172.21.15.107:0/41773073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584009b30 con 0x7f558c0b0b40 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f558affd640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c450 tx=0x7f557c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c0052d0 con 0x7f558c0b0b40 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c005430 con 0x7f558c0b0b40 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c15ea20 con 0x7f558c0b0b40 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c012870 con 0x7f558c0b0b40 2023-11-24T08:14:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.397+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c0c8270 con 0x7f558c0b0b40 2023-11-24T08:14:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.398+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c15a5a0 con 0x7f558c0b0b40 2023-11-24T08:14:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.399+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f557c0129d0 con 0x7f558c0b0b40 2023-11-24T08:14:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.400+0000 7f5573fff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 0x7f557806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.400+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f557c087fd0 con 0x7f558c0b0b40 2023-11-24T08:14:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.401+0000 7f558a7fc640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 0x7f557806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.401+0000 7f558a7fc640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 0x7f557806e500 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f558c15c490 tx=0x7f5580009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.405+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f558c15a5a0 con 0x7f558c0b0b40 2023-11-24T08:14:08.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.592+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f558c15a980 con 0x7f558c0b0b40 2023-11-24T08:14:08.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.593+0000 7f5573fff640 1 -- 172.21.15.107:0/41773073 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7f558c15a980 con 0x7f558c0b0b40 2023-11-24T08:14:08.595 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:14:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 msgr2=0x7f557806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 0x7f557806e500 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f558c15c490 tx=0x7f5580009190 comp rx=0 tx=0).stop 2023-11-24T08:14:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 msgr2=0x7f558c0c7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c450 tx=0x7f557c00c910 comp rx=0 tx=0).stop 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 shutdown_connections 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f557806c040 0x7f557806e500 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c15ded0 0x7f558c159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15a1c0 0x7f558c159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 --2- 172.21.15.107:0/41773073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0b0b40 0x7f558c0c7d30 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 >> 172.21.15.107:0/41773073 conn(0x7f558c0cbb40 msgr2=0x7f558c0cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.595+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 shutdown_connections 2023-11-24T08:14:08.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.596+0000 7f558bfff640 1 -- 172.21.15.107:0/41773073 wait complete. 2023-11-24T08:14:08.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:14:08.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.818+0000 7fd61d0ff640 1 Processor -- start 2023-11-24T08:14:08.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.818+0000 7fd61d0ff640 1 -- start start 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd61d0ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6180b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd61d0ff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6180ca0e0 con 0x7fd61815b1d0 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd61d0ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd618072800 con 0x7fd6180b1080 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd61d0ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180b1510 con 0x7fd618070550 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd617fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd61815b1d0 0x7fd618071f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34666/0 (socket says 172.21.15.107:34666) 2023-11-24T08:14:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd617fff640 1 -- 172.21.15.107:0/2852185886 learned_addr learned my addr 172.21.15.107:0/2852185886 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742441130 0 0) 0x7fd6180ca0e0 con 0x7fd61815b1d0 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6180b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.820+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd604003520 con 0x7fd61815b1d0 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605284006 0 0) 0x7fd604003520 con 0x7fd61815b1d0 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 msgr2=0x7fd6180b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6180b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 >> v1:172.21.15.120:6789/0 conn(0x7fd6180b1080 legacy=0x7fd61815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180c97b0 con 0x7fd61815b1d0 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd6080019f0 con 0x7fd61815b1d0 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 >> v1:172.21.15.107:6789/0 conn(0x7fd61815b1d0 legacy=0x7fd618071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd604004000 0x7fd6040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd604005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180b1510 con 0x7fd604004d00 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60400bb70 con 0x7fd604004000 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60400bd30 con 0x7fd618070550 2023-11-24T08:14:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd61c8fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd604005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.821+0000 7fd617fff640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd604004000 0x7fd6040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd61c8fe640 1 -- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 msgr2=0x7fd6040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd61c8fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618070550 0x7fd6040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd61c8fe640 1 -- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd604004000 msgr2=0x7fd6040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd61c8fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd604004000 0x7fd6040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd61c8fe640 1 -- 172.21.15.107:0/2852185886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604003520 con 0x7fd604004d00 2023-11-24T08:14:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.822+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd604004000 0x7fd6040047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.823+0000 7fd61c8fe640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd604005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd610003db0 tx=0x7fd6040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c86aca8294987b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.823+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610032600 con 0x7fd604004d00 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.823+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61002e050 con 0x7fd604004d00 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.823+0000 7fd6177fe640 1 -- 172.21.15.107:0/2852185886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6100329e0 con 0x7fd604004d00 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 -- 172.21.15.107:0/2852185886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 msgr2=0x7fd604005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/2852185886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd604005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd610003db0 tx=0x7fd6040057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61c8fe640 1 -- 172.21.15.107:0/2852185886 reap_dead start 2023-11-24T08:14:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 -- 172.21.15.107:0/2852185886 shutdown_connections 2023-11-24T08:14:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 -- 172.21.15.107:0/2852185886 >> 172.21.15.107:0/2852185886 conn(0x7fd6180cbb40 msgr2=0x7fd6180cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 -- 172.21.15.107:0/2852185886 shutdown_connections 2023-11-24T08:14:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.824+0000 7fd61d0ff640 1 -- 172.21.15.107:0/2852185886 wait complete. 2023-11-24T08:14:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.826+0000 7fd61d0ff640 1 Processor -- start 2023-11-24T08:14:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.826+0000 7fd61d0ff640 1 -- start start 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618099e10 0x7fd6181f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd61809b7d0 0x7fd61809bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180ce620 con 0x7fd604004d00 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180cdd80 con 0x7fd61809b7d0 2023-11-24T08:14:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd61d0ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180ce060 con 0x7fd618099e10 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd617fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd617fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41460/0 (socket says 172.21.15.107:41460) 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.827+0000 7fd617fff640 1 -- 172.21.15.107:0/1659321535 learned_addr learned my addr 172.21.15.107:0/1659321535 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 -- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618099e10 msgr2=0x7fd6181f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618099e10 0x7fd6181f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd61c8fe640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd61809b7d0 0x7fd61809bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618099e10 0x7fd6181f2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 -- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd61809b7d0 msgr2=0x7fd61809bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd61809b7d0 0x7fd61809bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 -- 172.21.15.107:0/1659321535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180ce340 con 0x7fd604004d00 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd617fff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd608007ae0 tx=0x7fd60800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608013420 con 0x7fd604004d00 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6080135a0 con 0x7fd604004d00 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608013950 con 0x7fd604004d00 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6180c62c0 con 0x7fd604004d00 2023-11-24T08:14:08.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.828+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61809a290 con 0x7fd604004d00 2023-11-24T08:14:08.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.829+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6180c64b0 con 0x7fd604004d00 2023-11-24T08:14:08.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.830+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fd608013ab0 con 0x7fd604004d00 2023-11-24T08:14:08.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.831+0000 7fd6157fa640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 0x7fd5f806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:08.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.831+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd608088e20 con 0x7fd604004d00 2023-11-24T08:14:08.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.832+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 0x7fd5f806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:08.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.832+0000 7fd60f7fe640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 0x7fd5f806e500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd618144e60 tx=0x7fd600008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:08.836+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd608055610 con 0x7fd604004d00 2023-11-24T08:14:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.040+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd61804e840 con 0x7fd604004d00 2023-11-24T08:14:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.040+0000 7fd6157fa640 1 -- 172.21.15.107:0/1659321535 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd608058c30 con 0x7fd604004d00 2023-11-24T08:14:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 msgr2=0x7fd5f806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 0x7fd5f806e500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd618144e60 tx=0x7fd600008040 comp rx=0 tx=0).stop 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 msgr2=0x7fd6180cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd608007ae0 tx=0x7fd60800e530 comp rx=0 tx=0).stop 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 shutdown_connections 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5f806c040 0x7fd5f806e500 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd61809b7d0 0x7fd61809bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd618099e10 0x7fd6181f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 --2- 172.21.15.107:0/1659321535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd604004d00 0x7fd6180cd6a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 >> 172.21.15.107:0/1659321535 conn(0x7fd6180cbb40 msgr2=0x7fd6180ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 shutdown_connections 2023-11-24T08:14:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.043+0000 7fd61d0ff640 1 -- 172.21.15.107:0/1659321535 wait complete. 2023-11-24T08:14:09.063 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:14:09.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:09.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:14:09.179 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:14:09.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:09.179 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:14:09.230 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:09.230 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:09.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:14:09.290 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:09.290 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:09.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:14:09.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:09.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.567+0000 7fb8f6248640 1 Processor -- start 2023-11-24T08:14:09.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.567+0000 7fb8f6248640 1 -- start start 2023-11-24T08:14:09.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.568+0000 7fb8f6248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 0x7fb8f00cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.568+0000 7fb8f6248640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f00cdfb0 0x7fb8f00ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.568+0000 7fb8f6248640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f00cece0 con 0x7fb8f015c7f0 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.568+0000 7fb8f6248640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f0156010 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.568+0000 7fb8f6248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f006ce30 con 0x7fb8f00cf450 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8ef7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8f015c7f0 0x7fb8f006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34670/0 (socket says 172.21.15.107:34670) 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/2806402836 learned_addr learned my addr 172.21.15.107:0/2806402836 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8effff640 1 --2- 172.21.15.107:0/2806402836 >> v2:172.21.15.107:3300/0 conn(0x7fb8f00cdfb0 0x7fb8f00ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:09.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/2806402836 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309498000 0 0) 0x7fb8f00cece0 con 0x7fb8f015c7f0 2023-11-24T08:14:09.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8eeffd640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 0x7fb8f00cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/2806402836 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d8003520 con 0x7fb8f015c7f0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8effff640 1 -- 172.21.15.107:0/2806402836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 msgr2=0x7fb8f00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8effff640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 0x7fb8f00cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8effff640 1 -- 172.21.15.107:0/2806402836 >> v1:172.21.15.107:6789/0 conn(0x7fb8f015c7f0 legacy=0x7fb8f006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8effff640 1 -- 172.21.15.107:0/2806402836 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f006d040 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.569+0000 7fb8eeffd640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 0x7fb8f00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8effff640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f00ce440 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009d90 tx=0x7fb8e802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b38623cd8b4679 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/2806402836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e802cc60 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/2806402836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e802cdc0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8ee7fc640 1 -- 172.21.15.107:0/2806402836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8034660 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 -- 172.21.15.107:0/2806402836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 msgr2=0x7fb8f00ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f00ce440 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009d90 tx=0x7fb8e802beb0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 -- 172.21.15.107:0/2806402836 shutdown_connections 2023-11-24T08:14:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f00ce440 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 --2- 172.21.15.107:0/2806402836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00cf450 0x7fb8f00cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.571+0000 7fb8f6248640 1 -- 172.21.15.107:0/2806402836 >> 172.21.15.107:0/2806402836 conn(0x7fb8f006a920 msgr2=0x7fb8f006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.572+0000 7fb8f6248640 1 -- 172.21.15.107:0/2806402836 shutdown_connections 2023-11-24T08:14:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.572+0000 7fb8f6248640 1 -- 172.21.15.107:0/2806402836 wait complete. 2023-11-24T08:14:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.573+0000 7fb8f6248640 1 Processor -- start 2023-11-24T08:14:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.573+0000 7fb8f6248640 1 -- start start 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00cf450 0x7fb8f00b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f015c7f0 0x7fb8f0158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01f6670 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01f3510 con 0x7fb8f00cf450 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.574+0000 7fb8f6248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01f37f0 con 0x7fb8f015c7f0 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41482/0 (socket says 172.21.15.107:41482) 2023-11-24T08:14:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/3422282246 learned_addr learned my addr 172.21.15.107:0/3422282246 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f015c7f0 msgr2=0x7fb8f0158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f015c7f0 0x7fb8f0158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00cf450 msgr2=0x7fb8f00b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00cf450 0x7fb8f00b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/3422282246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8009a70 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.575+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000c410 tx=0x7fb8e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.576+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0005ce0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.576+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e0005e40 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.576+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f01f3ad0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.576+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e00072f0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.576+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f0148930 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.577+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0067470 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.578+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fb8e0007510 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.579+0000 7fb8d7fff640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 0x7fb8dc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.579+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8e0087be0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.579+0000 7fb8eeffd640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 0x7fb8dc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:09.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.580+0000 7fb8eeffd640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 0x7fb8dc06e500 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4006fd0 tx=0x7fb8e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:09.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.583+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8e0054490 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.814+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f00cf9b0 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.815+0000 7fb8d7fff640 1 -- 172.21.15.107:0/3422282246 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fb8e0017090 con 0x7fb8f00cdfb0 2023-11-24T08:14:09.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:09.817 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:14:09.818 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 msgr2=0x7fb8dc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 0x7fb8dc06e500 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4006fd0 tx=0x7fb8e4008040 comp rx=0 tx=0).stop 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 msgr2=0x7fb8f01483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000c410 tx=0x7fb8e000c8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 shutdown_connections 2023-11-24T08:14:09.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8dc06c040 0x7fb8dc06e500 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f015c7f0 0x7fb8f0158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00cf450 0x7fb8f00b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 --2- 172.21.15.107:0/3422282246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00cdfb0 0x7fb8f01483f0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.818+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 >> 172.21.15.107:0/3422282246 conn(0x7fb8f006a920 msgr2=0x7fb8f01554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.819+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 shutdown_connections 2023-11-24T08:14:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:09.819+0000 7fb8f6248640 1 -- 172.21.15.107:0/3422282246 wait complete. 2023-11-24T08:14:09.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:10.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.050+0000 7f762f59e640 1 Processor -- start 2023-11-24T08:14:10.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.050+0000 7f762f59e640 1 -- start start 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f763013d120 0x7f763015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762f59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f763006d6c0 0x7f763006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763006e060 con 0x7f76300b0a70 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762f59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763013f9e0 con 0x7f763013d120 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762f59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763013fcc0 con 0x7f763006d6c0 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f763013d120 0x7f763015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f763013d120 0x7f763015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41494/0 (socket says 172.21.15.107:41494) 2023-11-24T08:14:10.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.052+0000 7f762e59c640 1 -- 172.21.15.107:0/2247188769 learned_addr learned my addr 172.21.15.107:0/2247188769 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:10.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.053+0000 7f762e59c640 1 -- 172.21.15.107:0/2247188769 >> v2:172.21.15.120:3300/0 conn(0x7f763006d6c0 msgr2=0x7f763006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.053+0000 7f762e59c640 1 --2- 172.21.15.107:0/2247188769 >> v2:172.21.15.120:3300/0 conn(0x7f763006d6c0 0x7f763006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.053+0000 7f762e59c640 1 -- 172.21.15.107:0/2247188769 >> v1:172.21.15.107:6789/0 conn(0x7f76300b0a70 legacy=0x7f76300b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.053+0000 7f762e59c640 1 -- 172.21.15.107:0/2247188769 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76300cec00 con 0x7f763013d120 2023-11-24T08:14:10.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.054+0000 7f762e59c640 1 --2- 172.21.15.107:0/2247188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763013d120 0x7f763015af40 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f7630072a70 tx=0x7f762002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec2f7cd692b942c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.054+0000 7f762dd9b640 1 -- 172.21.15.107:0/2247188769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762002f8e0 con 0x7f763013d120 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.054+0000 7f762dd9b640 1 -- 172.21.15.107:0/2247188769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762002fa40 con 0x7f763013d120 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.054+0000 7f762dd9b640 1 -- 172.21.15.107:0/2247188769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762002fdd0 con 0x7f763013d120 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 -- 172.21.15.107:0/2247188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763013d120 msgr2=0x7f763015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 --2- 172.21.15.107:0/2247188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763013d120 0x7f763015af40 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f7630072a70 tx=0x7f762002db20 comp rx=0 tx=0).stop 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 -- 172.21.15.107:0/2247188769 shutdown_connections 2023-11-24T08:14:10.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 --2- 172.21.15.107:0/2247188769 >> v2:172.21.15.120:3300/0 conn(0x7f763006d6c0 0x7f763006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 --2- 172.21.15.107:0/2247188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763013d120 0x7f763015af40 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 -- 172.21.15.107:0/2247188769 >> 172.21.15.107:0/2247188769 conn(0x7f76300cb830 msgr2=0x7f76300cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:10.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 -- 172.21.15.107:0/2247188769 shutdown_connections 2023-11-24T08:14:10.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.055+0000 7f762f59e640 1 -- 172.21.15.107:0/2247188769 wait complete. 2023-11-24T08:14:10.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.056+0000 7f762f59e640 1 Processor -- start 2023-11-24T08:14:10.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.057+0000 7f762f59e640 1 -- start start 2023-11-24T08:14:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300b0a70 0x7f76300ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 0x7f763013bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7630158250 con 0x7f763006d6c0 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76300ad290 con 0x7f76300b0a70 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300ad570 con 0x7f763013d120 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41496/0 (socket says 172.21.15.107:41496) 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 -- 172.21.15.107:0/2694728177 learned_addr learned my addr 172.21.15.107:0/2694728177 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f7627fff640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300b0a70 0x7f76300ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762ed9d640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 0x7f763013bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 -- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 msgr2=0x7f763013bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 0x7f763013bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 -- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300b0a70 msgr2=0x7f76300ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300b0a70 0x7f76300ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762e59c640 1 -- 172.21.15.107:0/2694728177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200094d0 con 0x7f763006d6c0 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.058+0000 7f762ed9d640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 0x7f763013bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:10.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f762e59c640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f762002db60 tx=0x7f76200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762002f880 con 0x7f763006d6c0 2023-11-24T08:14:10.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620042470 con 0x7f763006d6c0 2023-11-24T08:14:10.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76300ad850 con 0x7f763006d6c0 2023-11-24T08:14:10.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620042820 con 0x7f763006d6c0 2023-11-24T08:14:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.059+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76301f3200 con 0x7f763006d6c0 2023-11-24T08:14:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.060+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763004e840 con 0x7f763006d6c0 2023-11-24T08:14:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.061+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f7620042980 con 0x7f763006d6c0 2023-11-24T08:14:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.062+0000 7f7626ffd640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 0x7f761006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.062+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7620033070 con 0x7f763006d6c0 2023-11-24T08:14:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.062+0000 7f7627fff640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 0x7f761006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.063+0000 7f7627fff640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 0x7f761006e500 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f763006de90 tx=0x7f7618008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.067+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7620077ff0 con 0x7f763006d6c0 2023-11-24T08:14:10.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.262+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76300ccf10 con 0x7f763006d6c0 2023-11-24T08:14:10.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.264+0000 7f7626ffd640 1 -- 172.21.15.107:0/2694728177 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7f7620046090 con 0x7f763006d6c0 2023-11-24T08:14:10.266 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:10.266 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 msgr2=0x7f761006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 0x7f761006e500 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f763006de90 tx=0x7f7618008040 comp rx=0 tx=0).stop 2023-11-24T08:14:10.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 msgr2=0x7f7630138d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f762002db60 tx=0x7f76200046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 shutdown_connections 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761006c040 0x7f761006e500 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763013d120 0x7f763013bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300b0a70 0x7f76300ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 --2- 172.21.15.107:0/2694728177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f763006d6c0 0x7f7630138d10 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 >> 172.21.15.107:0/2694728177 conn(0x7f76300cb830 msgr2=0x7f76300cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 shutdown_connections 2023-11-24T08:14:10.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.266+0000 7f762f59e640 1 -- 172.21.15.107:0/2694728177 wait complete. 2023-11-24T08:14:10.289 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:13:33.870908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:13:35.060534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:14:10.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:14:10.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.504+0000 7fc582a7c640 1 Processor -- start 2023-11-24T08:14:10.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.504+0000 7fc582a7c640 1 -- start start 2023-11-24T08:14:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc582a7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc57c0cc0f0 0x7fc57c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc582a7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c15eea0 con 0x7fc57c15b440 2023-11-24T08:14:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc582a7c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c0abf30 con 0x7fc57c0c57e0 2023-11-24T08:14:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc582a7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c15b820 con 0x7fc57c0cc0f0 2023-11-24T08:14:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc580818640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc57c15b440 0x7fc57c0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34686/0 (socket says 172.21.15.107:34686) 2023-11-24T08:14:10.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.506+0000 7fc580818640 1 -- 172.21.15.107:0/372238416 learned_addr learned my addr 172.21.15.107:0/372238416 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:10.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc573fff640 1 --2- 172.21.15.107:0/372238416 >> v2:172.21.15.120:3300/0 conn(0x7fc57c0cc0f0 0x7fc57c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006014880 0 0) 0x7fc57c15eea0 con 0x7fc57c15b440 2023-11-24T08:14:10.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc564003520 con 0x7fc57c15b440 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2032732072 0 0) 0x7fc564003520 con 0x7fc57c15b440 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 >> v2:172.21.15.120:3300/0 conn(0x7fc57c0cc0f0 msgr2=0x7fc57c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 --2- 172.21.15.107:0/372238416 >> v2:172.21.15.120:3300/0 conn(0x7fc57c0cc0f0 0x7fc57c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 >> v1:172.21.15.120:6789/0 conn(0x7fc57c0c57e0 legacy=0x7fc57c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c15f0f0 con 0x7fc57c15b440 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc578001670 con 0x7fc57c15b440 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 >> v1:172.21.15.107:6789/0 conn(0x7fc57c15b440 legacy=0x7fc57c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 0x7fc564003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.507+0000 7fc5737fe640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc564005180 0x7fc56400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c15b820 con 0x7fc564003970 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c120 con 0x7fc564005180 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c2e0 con 0x7fc5640047c0 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 0x7fc564003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 -- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5640047c0 msgr2=0x7fc564004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc581019640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc564005180 0x7fc56400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc573fff640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 -- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc564005180 msgr2=0x7fc56400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc564005180 0x7fc56400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc580818640 1 -- 172.21.15.107:0/372238416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc564003520 con 0x7fc564003970 2023-11-24T08:14:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.508+0000 7fc581019640 1 -- 172.21.15.107:0/372238416 reap_dead start 2023-11-24T08:14:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.509+0000 7fc580818640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 0x7fc564003d90 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c240 tx=0x7fc5640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b335bc464ced453 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578005ad0 con 0x7fc564003970 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578005c50 con 0x7fc564003970 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc5737fe640 1 -- 172.21.15.107:0/372238416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57802c8b0 con 0x7fc564003970 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 -- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 msgr2=0x7fc564003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 0x7fc564003d90 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c240 tx=0x7fc5640057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 -- 172.21.15.107:0/372238416 shutdown_connections 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 --2- 172.21.15.107:0/372238416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc564003970 0x7fc564003d90 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 -- 172.21.15.107:0/372238416 >> 172.21.15.107:0/372238416 conn(0x7fc57c0b0c50 msgr2=0x7fc57c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 -- 172.21.15.107:0/372238416 shutdown_connections 2023-11-24T08:14:10.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.510+0000 7fc582a7c640 1 -- 172.21.15.107:0/372238416 wait complete. 2023-11-24T08:14:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.512+0000 7fc582a7c640 1 Processor -- start 2023-11-24T08:14:10.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.512+0000 7fc582a7c640 1 -- start start 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.513+0000 7fc582a7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc564003970 0x7fc57c155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.513+0000 7fc582a7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 0x7fc57c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.513+0000 7fc582a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.513+0000 7fc582a7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c155860 con 0x7fc57c14f610 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc582a7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c14f450 con 0x7fc57c14fb60 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc582a7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c15a5b0 con 0x7fc564003970 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41518/0 (socket says 172.21.15.107:41518) 2023-11-24T08:14:10.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 -- 172.21.15.107:0/3600415909 learned_addr learned my addr 172.21.15.107:0/3600415909 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc580818640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc564003970 0x7fc57c155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc573fff640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 0x7fc57c159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 -- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc564003970 msgr2=0x7fc57c155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc564003970 0x7fc57c155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 -- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 msgr2=0x7fc57c159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 0x7fc57c159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc581019640 1 -- 172.21.15.107:0/3600415909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c099a70 con 0x7fc57c14f610 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.514+0000 7fc573fff640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 0x7fc57c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc581019640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fc57c06dc00 tx=0x7fc57400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574003d80 con 0x7fc57c14f610 2023-11-24T08:14:10.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574005070 con 0x7fc57c14f610 2023-11-24T08:14:10.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574005400 con 0x7fc57c14f610 2023-11-24T08:14:10.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c099d50 con 0x7fc57c14f610 2023-11-24T08:14:10.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.515+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c09a310 con 0x7fc57c14f610 2023-11-24T08:14:10.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.516+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c04e8b0 con 0x7fc57c14f610 2023-11-24T08:14:10.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.517+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc574005560 con 0x7fc57c14f610 2023-11-24T08:14:10.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.518+0000 7fc5717fa640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 0x7fc56806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.518+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc574087e40 con 0x7fc57c14f610 2023-11-24T08:14:10.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.518+0000 7fc580818640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 0x7fc56806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.519+0000 7fc580818640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 0x7fc56806e500 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc578034000 tx=0x7fc578009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.523+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc574054540 con 0x7fc57c14f610 2023-11-24T08:14:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.703+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc57c06dc80 con 0x7fc57c14f610 2023-11-24T08:14:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.704+0000 7fc5717fa640 1 -- 172.21.15.107:0/3600415909 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc574018020 con 0x7fc57c14f610 2023-11-24T08:14:10.705 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:14:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 msgr2=0x7fc56806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 0x7fc56806e500 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc578034000 tx=0x7fc578009030 comp rx=0 tx=0).stop 2023-11-24T08:14:10.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 msgr2=0x7fc57c15a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fc57c06dc00 tx=0x7fc57400d9d0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 shutdown_connections 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc56806c040 0x7fc56806e500 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc57c14f610 0x7fc57c15a070 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc57c14fb60 0x7fc57c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 --2- 172.21.15.107:0/3600415909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc564003970 0x7fc57c155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 >> 172.21.15.107:0/3600415909 conn(0x7fc57c0b0c50 msgr2=0x7fc57c06c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:10.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 shutdown_connections 2023-11-24T08:14:10.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.706+0000 7fc582a7c640 1 -- 172.21.15.107:0/3600415909 wait complete. 2023-11-24T08:14:10.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:14:10.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.926+0000 7f007e28c640 1 Processor -- start 2023-11-24T08:14:10.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.927+0000 7f007e28c640 1 -- start start 2023-11-24T08:14:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007e28c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 0x7f0078161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007e28c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00780cc3e0 0x7f007806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007e28c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780ced70 con 0x7f00780cceb0 2023-11-24T08:14:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007e28c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0078161f70 con 0x7f00780cc3e0 2023-11-24T08:14:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007e28c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007806e870 con 0x7f00780cb3c0 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007da8b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00780cc3e0 0x7f007806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007da8b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00780cc3e0 0x7f007806c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41528/0 (socket says 172.21.15.107:41528) 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.928+0000 7f007da8b640 1 -- 172.21.15.107:0/3257019755 learned_addr learned my addr 172.21.15.107:0/3257019755 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007d28a640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 0x7f0078161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007da8b640 1 -- 172.21.15.107:0/3257019755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 msgr2=0x7f0078161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007da8b640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 0x7f0078161a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007da8b640 1 -- 172.21.15.107:0/3257019755 >> v1:172.21.15.120:6789/0 conn(0x7f00780cceb0 legacy=0x7f00780cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007da8b640 1 -- 172.21.15.107:0/3257019755 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007806e2b0 con 0x7f00780cc3e0 2023-11-24T08:14:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.929+0000 7f007d28a640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 0x7f0078161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.930+0000 7f007da8b640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cc3e0 0x7f007806c7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0070009e50 tx=0x7f007002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964b76670f69547b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.930+0000 7f006ffff640 1 -- 172.21.15.107:0/3257019755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007002c9f0 con 0x7f00780cc3e0 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.930+0000 7f006ffff640 1 -- 172.21.15.107:0/3257019755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007002cb50 con 0x7f00780cc3e0 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.930+0000 7f006ffff640 1 -- 172.21.15.107:0/3257019755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00700343f0 con 0x7f00780cc3e0 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 -- 172.21.15.107:0/3257019755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cc3e0 msgr2=0x7f007806c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cc3e0 0x7f007806c7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0070009e50 tx=0x7f007002be50 comp rx=0 tx=0).stop 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 -- 172.21.15.107:0/3257019755 shutdown_connections 2023-11-24T08:14:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cc3e0 0x7f007806c7d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 --2- 172.21.15.107:0/3257019755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cb3c0 0x7f0078161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 -- 172.21.15.107:0/3257019755 >> 172.21.15.107:0/3257019755 conn(0x7f0078071b20 msgr2=0x7f0078071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 -- 172.21.15.107:0/3257019755 shutdown_connections 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.931+0000 7f007e28c640 1 -- 172.21.15.107:0/3257019755 wait complete. 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.933+0000 7f007e28c640 1 Processor -- start 2023-11-24T08:14:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.933+0000 7f007e28c640 1 -- start start 2023-11-24T08:14:10.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cc3e0 0x7f007809c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00780cceb0 0x7f0078097b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007815c1d0 con 0x7f00780cb3c0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007815c4b0 con 0x7f00780cceb0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007e28c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007815c790 con 0x7f00780cc3e0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41532/0 (socket says 172.21.15.107:41532) 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 -- 172.21.15.107:0/164715591 learned_addr learned my addr 172.21.15.107:0/164715591 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007da8b640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00780cceb0 0x7f0078097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007ca89640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cc3e0 0x7f007809c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 -- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cc3e0 msgr2=0x7f007809c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cc3e0 0x7f007809c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 -- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00780cceb0 msgr2=0x7f0078097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00780cceb0 0x7f0078097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.934+0000 7f007d28a640 1 -- 172.21.15.107:0/164715591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0070009b30 con 0x7f00780cb3c0 2023-11-24T08:14:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f007d28a640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f006800c7f0 tx=0x7f006800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068007a30 con 0x7f00780cb3c0 2023-11-24T08:14:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068007b90 con 0x7f00780cb3c0 2023-11-24T08:14:10.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006800ce40 con 0x7f00780cb3c0 2023-11-24T08:14:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007815ca70 con 0x7f00780cb3c0 2023-11-24T08:14:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.935+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007815bee0 con 0x7f00780cb3c0 2023-11-24T08:14:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.936+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00780ccd00 con 0x7f00780cb3c0 2023-11-24T08:14:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.937+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f00680113f0 con 0x7f00780cb3c0 2023-11-24T08:14:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.938+0000 7f006dffb640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 0x7f006406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.938+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0068088780 con 0x7f00780cb3c0 2023-11-24T08:14:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.938+0000 7f007ca89640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 0x7f006406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.939+0000 7f007ca89640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 0x7f006406e500 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f007815b930 tx=0x7f0074008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:10.942+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0068054f10 con 0x7f00780cb3c0 2023-11-24T08:14:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.117+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00780cd2f0 con 0x7f00780cb3c0 2023-11-24T08:14:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.118+0000 7f006dffb640 1 -- 172.21.15.107:0/164715591 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00780cd2f0 con 0x7f00780cb3c0 2023-11-24T08:14:11.120 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 msgr2=0x7f006406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 0x7f006406e500 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f007815b930 tx=0x7f0074008040 comp rx=0 tx=0).stop 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 msgr2=0x7f007815b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f006800c7f0 tx=0x7f006800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 shutdown_connections 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f006406c040 0x7f006406e500 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00780cceb0 0x7f0078097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00780cc3e0 0x7f007809c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 --2- 172.21.15.107:0/164715591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00780cb3c0 0x7f007815b9a0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 >> 172.21.15.107:0/164715591 conn(0x7f0078071b20 msgr2=0x7f0078160a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 shutdown_connections 2023-11-24T08:14:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.121+0000 7f007e28c640 1 -- 172.21.15.107:0/164715591 wait complete. 2023-11-24T08:14:11.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:14:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.351+0000 7fe86d705640 1 Processor -- start 2023-11-24T08:14:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.351+0000 7fe86d705640 1 -- start start 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe86d705640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe86d705640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe8680cbf20 0x7fe8680c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe86d705640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8680cc3a0 con 0x7fe8680c57e0 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe86d705640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86815b820 con 0x7fe8680cbf20 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe86d705640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86815eea0 con 0x7fe86815b440 2023-11-24T08:14:11.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.353+0000 7fe866ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:58230/0 (socket says 172.21.15.107:58230) 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 -- 172.21.15.107:0/3939937140 learned_addr learned my addr 172.21.15.107:0/3939937140 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe85e7fc640 1 --2- 172.21.15.107:0/3939937140 >> v2:172.21.15.120:3300/0 conn(0x7fe8680cbf20 0x7fe8680c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 -- 172.21.15.107:0/3939937140 >> v2:172.21.15.120:3300/0 conn(0x7fe8680cbf20 msgr2=0x7fe8680c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 --2- 172.21.15.107:0/3939937140 >> v2:172.21.15.120:3300/0 conn(0x7fe8680cbf20 0x7fe8680c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 -- 172.21.15.107:0/3939937140 >> v1:172.21.15.120:6789/0 conn(0x7fe8680c57e0 legacy=0x7fe8680c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.354+0000 7fe866ffd640 1 -- 172.21.15.107:0/3939937140 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86806d600 con 0x7fe86815b440 2023-11-24T08:14:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.356+0000 7fe866ffd640 1 --2- 172.21.15.107:0/3939937140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe858009870 tx=0x7fe85802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1435225f87f3247c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.356+0000 7fe8667fc640 1 -- 172.21.15.107:0/3939937140 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858031640 con 0x7fe86815b440 2023-11-24T08:14:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.356+0000 7fe8667fc640 1 -- 172.21.15.107:0/3939937140 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8580317a0 con 0x7fe86815b440 2023-11-24T08:14:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.356+0000 7fe8667fc640 1 -- 172.21.15.107:0/3939937140 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858031b30 con 0x7fe86815b440 2023-11-24T08:14:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 -- 172.21.15.107:0/3939937140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 msgr2=0x7fe8680cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 --2- 172.21.15.107:0/3939937140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe858009870 tx=0x7fe85802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 -- 172.21.15.107:0/3939937140 shutdown_connections 2023-11-24T08:14:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 --2- 172.21.15.107:0/3939937140 >> v2:172.21.15.120:3300/0 conn(0x7fe8680cbf20 0x7fe8680c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 --2- 172.21.15.107:0/3939937140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe86815b440 0x7fe8680cb9e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 -- 172.21.15.107:0/3939937140 >> 172.21.15.107:0/3939937140 conn(0x7fe8680b0c50 msgr2=0x7fe8680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:11.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 -- 172.21.15.107:0/3939937140 shutdown_connections 2023-11-24T08:14:11.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.357+0000 7fe86d705640 1 -- 172.21.15.107:0/3939937140 wait complete. 2023-11-24T08:14:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.359+0000 7fe86d705640 1 Processor -- start 2023-11-24T08:14:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.359+0000 7fe86d705640 1 -- start start 2023-11-24T08:14:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8680c57e0 0x7fe86815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8680cbf20 0x7fe8680b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe868156cc0 con 0x7fe86815b440 2023-11-24T08:14:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680b4aa0 con 0x7fe8680c57e0 2023-11-24T08:14:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.360+0000 7fe86d705640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8681e4c70 con 0x7fe8680cbf20 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41548/0 (socket says 172.21.15.107:41548) 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 -- 172.21.15.107:0/162041494 learned_addr learned my addr 172.21.15.107:0/162041494 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe866ffd640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8680c57e0 0x7fe86815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe85e7fc640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8680cbf20 0x7fe8680b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 -- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8680cbf20 msgr2=0x7fe8680b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8680cbf20 0x7fe8680b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 -- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8680c57e0 msgr2=0x7fe86815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8680c57e0 0x7fe86815dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.361+0000 7fe8677fe640 1 -- 172.21.15.107:0/162041494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8580094d0 con 0x7fe86815b440 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe8677fe640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c710 tx=0x7fe86000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860006d50 con 0x7fe86815b440 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86000ce20 con 0x7fe86815b440 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860003bb0 con 0x7fe86815b440 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8681e4f50 con 0x7fe86815b440 2023-11-24T08:14:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.362+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8681e5510 con 0x7fe86815b440 2023-11-24T08:14:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.363+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8681e5730 con 0x7fe86815b440 2023-11-24T08:14:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.365+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fe860003d10 con 0x7fe86815b440 2023-11-24T08:14:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.366+0000 7fe85ffff640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 0x7fe848072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.366+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe860087db0 con 0x7fe86815b440 2023-11-24T08:14:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.366+0000 7fe866ffd640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 0x7fe848072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.367+0000 7fe866ffd640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 0x7fe848072940 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe858036000 tx=0x7fe85803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.370+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8600580e0 con 0x7fe86815b440 2023-11-24T08:14:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.561+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8680cc3a0 con 0x7fe86815b440 2023-11-24T08:14:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.562+0000 7fe85ffff640 1 -- 172.21.15.107:0/162041494 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8680cc3a0 con 0x7fe86815b440 2023-11-24T08:14:11.564 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:14:11.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 msgr2=0x7fe848072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 0x7fe848072940 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe858036000 tx=0x7fe85803c040 comp rx=0 tx=0).stop 2023-11-24T08:14:11.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 msgr2=0x7fe8680b43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c710 tx=0x7fe86000cbd0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 shutdown_connections 2023-11-24T08:14:11.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe848070480 0x7fe848072940 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b440 0x7fe8680b43d0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8680cbf20 0x7fe8680b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 --2- 172.21.15.107:0/162041494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8680c57e0 0x7fe86815dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 >> 172.21.15.107:0/162041494 conn(0x7fe8680b0c50 msgr2=0x7fe86806cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:11.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 shutdown_connections 2023-11-24T08:14:11.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.565+0000 7fe86d705640 1 -- 172.21.15.107:0/162041494 wait complete. 2023-11-24T08:14:11.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.789+0000 7fe1bf2ff640 1 Processor -- start 2023-11-24T08:14:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.789+0000 7fe1bf2ff640 1 -- start start 2023-11-24T08:14:11.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bf2ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b815c410 0x7fe1b815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bf2ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1b80b1de0 0x7fe1b80b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bf2ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b815bf80 con 0x7fe1b8144b30 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bf2ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80b27b0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bf2ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80b1560 con 0x7fe1b815c410 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1b80b1de0 0x7fe1b80b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1b80b1de0 0x7fe1b80b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41554/0 (socket says 172.21.15.107:41554) 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 -- 172.21.15.107:0/927005970 learned_addr learned my addr 172.21.15.107:0/927005970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1bdafc640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b815c410 0x7fe1b815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 -- 172.21.15.107:0/927005970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b815c410 msgr2=0x7fe1b815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b815c410 0x7fe1b815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 -- 172.21.15.107:0/927005970 >> v1:172.21.15.120:6789/0 conn(0x7fe1b8144b30 legacy=0x7fe1b80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.791+0000 7fe1beafe640 1 -- 172.21.15.107:0/927005970 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80b1280 con 0x7fe1b80b1de0 2023-11-24T08:14:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1beafe640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b80b2270 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009e50 tx=0x7fe1b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5769e8c8f6bd855d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1bd2fb640 1 -- 172.21.15.107:0/927005970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b002ca70 con 0x7fe1b80b1de0 2023-11-24T08:14:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1bd2fb640 1 -- 172.21.15.107:0/927005970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b002cbd0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1bd2fb640 1 -- 172.21.15.107:0/927005970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00343f0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/927005970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 msgr2=0x7fe1b80b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.793+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b80b2270 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009e50 tx=0x7fe1b002bed0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/927005970 shutdown_connections 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b80b2270 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/927005970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b815c410 0x7fe1b815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/927005970 >> 172.21.15.107:0/927005970 conn(0x7fe1b80cb830 msgr2=0x7fe1b80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/927005970 shutdown_connections 2023-11-24T08:14:11.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.794+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/927005970 wait complete. 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.795+0000 7fe1bf2ff640 1 Processor -- start 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 -- start start 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b8144b30 0x7fe1b81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 0x7fe1b813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81419f0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b813e230 con 0x7fe1b815c410 2023-11-24T08:14:11.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.796+0000 7fe1bf2ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b8141c70 con 0x7fe1b8144b30 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41564/0 (socket says 172.21.15.107:41564) 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 -- 172.21.15.107:0/1728551993 learned_addr learned my addr 172.21.15.107:0/1728551993 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1beafe640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 0x7fe1b813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 -- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b8144b30 msgr2=0x7fe1b81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b8144b30 0x7fe1b81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 -- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 msgr2=0x7fe1b813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 0x7fe1b813dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 -- 172.21.15.107:0/1728551993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0009b30 con 0x7fe1b80b1de0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1beafe640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 0x7fe1b813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.797+0000 7fe1be2fd640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c430 tx=0x7fe1b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.798+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011790 con 0x7fe1b80b1de0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.798+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b40118f0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.798+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011ce0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.798+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b8141f50 con 0x7fe1b80b1de0 2023-11-24T08:14:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.798+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b8142510 con 0x7fe1b80b1de0 2023-11-24T08:14:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.799+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b804e840 con 0x7fe1b80b1de0 2023-11-24T08:14:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.800+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fe1b401c460 con 0x7fe1b80b1de0 2023-11-24T08:14:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.801+0000 7fe1aaffd640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 0x7fe19c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.801+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1b40884c0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.801+0000 7fe1bdafc640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 0x7fe19c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.802+0000 7fe1bdafc640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 0x7fe19c06e500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe1b815a8f0 tx=0x7fe1ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:11.806+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1b4054cb0 con 0x7fe1b80b1de0 2023-11-24T08:14:11.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:14:11.983 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:14:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.034+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b806d860 con 0x7fe1b80b1de0 2023-11-24T08:14:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.035+0000 7fe1aaffd640 1 -- 172.21.15.107:0/1728551993 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe1b40582d0 con 0x7fe1b80b1de0 2023-11-24T08:14:12.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:12.037 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:14:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-11-24T08:14:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 msgr2=0x7fe19c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 0x7fe19c06e500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe1b815a8f0 tx=0x7fe1ac008040 comp rx=0 tx=0).stop 2023-11-24T08:14:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 msgr2=0x7fe1b81f30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c430 tx=0x7fe1b400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 shutdown_connections 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe19c06c040 0x7fe19c06e500 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b815c410 0x7fe1b813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1b8144b30 0x7fe1b81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 --2- 172.21.15.107:0/1728551993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b80b1de0 0x7fe1b81f30e0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 >> 172.21.15.107:0/1728551993 conn(0x7fe1b80cb830 msgr2=0x7fe1b80722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 shutdown_connections 2023-11-24T08:14:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.037+0000 7fe1bf2ff640 1 -- 172.21.15.107:0/1728551993 wait complete. 2023-11-24T08:14:12.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.245+0000 7fed2df41640 1 Processor -- start 2023-11-24T08:14:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.245+0000 7fed2df41640 1 -- start start 2023-11-24T08:14:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.246+0000 7fed2df41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed280cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.246+0000 7fed2df41640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280cece0 con 0x7fed280cf450 2023-11-24T08:14:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.246+0000 7fed2df41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2806d040 con 0x7fed2815c7f0 2023-11-24T08:14:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.246+0000 7fed2df41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280cfe40 con 0x7fed280ce6c0 2023-11-24T08:14:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.247+0000 7fed26ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fed280cf450 0x7fed280cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34688/0 (socket says 172.21.15.107:34688) 2023-11-24T08:14:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.247+0000 7fed26ffd640 1 -- 172.21.15.107:0/3727734567 learned_addr learned my addr 172.21.15.107:0/3727734567 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.247+0000 7fed27fff640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed280cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.247+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322586155 0 0) 0x7fed280cece0 con 0x7fed280cf450 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.247+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed280cf450 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423446420 0 0) 0x7fed10003520 con 0x7fed280cf450 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 msgr2=0x7fed280cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed280cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 >> v1:172.21.15.120:6789/0 conn(0x7fed2815c7f0 legacy=0x7fed2806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2806d320 con 0x7fed280cf450 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fed1c001670 con 0x7fed280cf450 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 >> v1:172.21.15.107:6789/0 conn(0x7fed280cf450 legacy=0x7fed280cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed100039a0 0x7fed10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280cfe40 con 0x7fed100047c0 2023-11-24T08:14:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000ba60 con 0x7fed100039a0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000bc20 con 0x7fed280ce6c0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed26ffd640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed27fff640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed26ffd640 1 -- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 msgr2=0x7fed10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed277fe640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed100039a0 0x7fed10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed26ffd640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed280ce6c0 0x7fed10005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.248+0000 7fed26ffd640 1 -- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed100039a0 msgr2=0x7fed10003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.249+0000 7fed26ffd640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed100039a0 0x7fed10003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.249+0000 7fed26ffd640 1 -- 172.21.15.107:0/3727734567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10003520 con 0x7fed100047c0 2023-11-24T08:14:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.249+0000 7fed277fe640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed100039a0 0x7fed10003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed26ffd640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed10004c40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fed1c001eb0 tx=0x7fed100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1160020526c977eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c005ad0 con 0x7fed100047c0 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c005c30 con 0x7fed100047c0 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed267fc640 1 -- 172.21.15.107:0/3727734567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c02c7c0 con 0x7fed100047c0 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed2df41640 1 -- 172.21.15.107:0/3727734567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 msgr2=0x7fed10004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed2df41640 1 --2- 172.21.15.107:0/3727734567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed10004c40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fed1c001eb0 tx=0x7fed100057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.250+0000 7fed27fff640 1 -- 172.21.15.107:0/3727734567 reap_dead start 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.251+0000 7fed2df41640 1 -- 172.21.15.107:0/3727734567 shutdown_connections 2023-11-24T08:14:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.251+0000 7fed2df41640 1 -- 172.21.15.107:0/3727734567 >> 172.21.15.107:0/3727734567 conn(0x7fed2806a920 msgr2=0x7fed2806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.251+0000 7fed2df41640 1 -- 172.21.15.107:0/3727734567 shutdown_connections 2023-11-24T08:14:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.251+0000 7fed2df41640 1 -- 172.21.15.107:0/3727734567 wait complete. 2023-11-24T08:14:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.252+0000 7fed2df41640 1 Processor -- start 2023-11-24T08:14:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.253+0000 7fed2df41640 1 -- start start 2023-11-24T08:14:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.253+0000 7fed2df41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.253+0000 7fed2df41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed2814b180 0x7fed28151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed2df41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed28152480 0x7fed28156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed2df41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed28152900 con 0x7fed100047c0 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed2df41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2814afa0 con 0x7fed28152480 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed2df41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed281577c0 con 0x7fed2814b180 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41592/0 (socket says 172.21.15.107:41592) 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 -- 172.21.15.107:0/4156175185 learned_addr learned my addr 172.21.15.107:0/4156175185 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 -- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed2814b180 msgr2=0x7fed28151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed2814b180 0x7fed28151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 -- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed28152480 msgr2=0x7fed28156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed28152480 0x7fed28156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 -- 172.21.15.107:0/4156175185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28157aa0 con 0x7fed100047c0 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.254+0000 7fed277fe640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fed1800d770 tx=0x7fed1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.255+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18005270 con 0x7fed100047c0 2023-11-24T08:14:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.255+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed180053d0 con 0x7fed100047c0 2023-11-24T08:14:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.255+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1800b850 con 0x7fed100047c0 2023-11-24T08:14:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.255+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28155f80 con 0x7fed100047c0 2023-11-24T08:14:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.255+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed28156540 con 0x7fed100047c0 2023-11-24T08:14:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.256+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed28155de0 con 0x7fed100047c0 2023-11-24T08:14:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.257+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fed1800b9b0 con 0x7fed100047c0 2023-11-24T08:14:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.258+0000 7fed0ffff640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 0x7fed1406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.258+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed18088510 con 0x7fed100047c0 2023-11-24T08:14:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.258+0000 7fed26ffd640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 0x7fed1406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.259+0000 7fed26ffd640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 0x7fed1406e530 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fed1c034000 tx=0x7fed1c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.262+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed28155de0 con 0x7fed100047c0 2023-11-24T08:14:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.455+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed28155de0 con 0x7fed100047c0 2023-11-24T08:14:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.458+0000 7fed0ffff640 1 -- 172.21.15.107:0/4156175185 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7fed28155de0 con 0x7fed100047c0 2023-11-24T08:14:12.460 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:12.460 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.460+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 msgr2=0x7fed1406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.460+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 0x7fed1406e530 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fed1c034000 tx=0x7fed1c007000 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.460+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 msgr2=0x7fed2814aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.460+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fed1800d770 tx=0x7fed1800dc30 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 shutdown_connections 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed1406c070 0x7fed1406e530 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed28152480 0x7fed28156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed2814b180 0x7fed28151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 --2- 172.21.15.107:0/4156175185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed100047c0 0x7fed2814aa60 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 >> 172.21.15.107:0/4156175185 conn(0x7fed2806a920 msgr2=0x7fed281597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 shutdown_connections 2023-11-24T08:14:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.461+0000 7fed2df41640 1 -- 172.21.15.107:0/4156175185 wait complete. 2023-11-24T08:14:12.481 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:14:12.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.689+0000 7f66d9836640 1 Processor -- start 2023-11-24T08:14:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.689+0000 7f66d9836640 1 -- start start 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66d415c7f0 0x7f66d406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 0x7f66d40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 0x7f66d40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d4156010 con 0x7f66d415c7f0 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d406ce30 con 0x7f66d40cdfb0 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d9836640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40cfe10 con 0x7f66d40cf450 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66d415c7f0 0x7f66d406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66d415c7f0 0x7f66d406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41602/0 (socket says 172.21.15.107:41602) 2023-11-24T08:14:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 -- 172.21.15.107:0/3812373856 learned_addr learned my addr 172.21.15.107:0/3812373856 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d37fe640 1 --2- 172.21.15.107:0/3812373856 >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 0x7f66d40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d27fc640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 0x7f66d40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 -- 172.21.15.107:0/3812373856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 msgr2=0x7f66d40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 0x7f66d40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 -- 172.21.15.107:0/3812373856 >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 msgr2=0x7f66d40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/3812373856 >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 0x7f66d40ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d2ffd640 1 -- 172.21.15.107:0/3812373856 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40cece0 con 0x7f66d415c7f0 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d37fe640 1 --2- 172.21.15.107:0/3812373856 >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 0x7f66d40ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.691+0000 7f66d27fc640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 0x7f66d40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.692+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d415c7f0 0x7f66d406db20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f66d40aebc0 tx=0x7f66c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a6c794d05a5ab0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.693+0000 7f66d1ffb640 1 -- 172.21.15.107:0/3812373856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031870 con 0x7f66d415c7f0 2023-11-24T08:14:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.693+0000 7f66d1ffb640 1 -- 172.21.15.107:0/3812373856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c40319d0 con 0x7f66d415c7f0 2023-11-24T08:14:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.693+0000 7f66d1ffb640 1 -- 172.21.15.107:0/3812373856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031d60 con 0x7f66d415c7f0 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.693+0000 7f66d9836640 1 -- 172.21.15.107:0/3812373856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d415c7f0 msgr2=0x7f66d406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.693+0000 7f66d9836640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d415c7f0 0x7f66d406db20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f66d40aebc0 tx=0x7f66c402b910 comp rx=0 tx=0).stop 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 -- 172.21.15.107:0/3812373856 shutdown_connections 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 --2- 172.21.15.107:0/3812373856 >> v2:172.21.15.120:3300/0 conn(0x7f66d40cdfb0 0x7f66d40ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cf450 0x7f66d40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 --2- 172.21.15.107:0/3812373856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d415c7f0 0x7f66d406db20 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 -- 172.21.15.107:0/3812373856 >> 172.21.15.107:0/3812373856 conn(0x7f66d406a920 msgr2=0x7f66d406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 -- 172.21.15.107:0/3812373856 shutdown_connections 2023-11-24T08:14:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.694+0000 7f66d9836640 1 -- 172.21.15.107:0/3812373856 wait complete. 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.695+0000 7f66d9836640 1 Processor -- start 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.695+0000 7f66d9836640 1 -- start start 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cdfb0 0x7f66d41f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66d415c7f0 0x7f66d4154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d415a780 con 0x7f66d40cf450 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d415a330 con 0x7f66d415c7f0 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.696+0000 7f66d9836640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d41f3890 con 0x7f66d40cdfb0 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41608/0 (socket says 172.21.15.107:41608) 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 -- 172.21.15.107:0/1601933971 learned_addr learned my addr 172.21.15.107:0/1601933971 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cdfb0 0x7f66d41f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 -- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cdfb0 msgr2=0x7f66d41f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d37fe640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66d415c7f0 0x7f66d4154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cdfb0 0x7f66d41f4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 -- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66d415c7f0 msgr2=0x7f66d4154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66d415c7f0 0x7f66d4154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 -- 172.21.15.107:0/1601933971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40094d0 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.697+0000 7f66d27fc640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f66c800c7f0 tx=0x7f66c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.698+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8007a30 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.698+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c8007b90 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.698+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c800ce40 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.698+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d41f3b70 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.698+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d41f4130 con 0x7f66d40cf450 2023-11-24T08:14:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.699+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d4067470 con 0x7f66d40cf450 2023-11-24T08:14:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.700+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f66c80113f0 con 0x7f66d40cf450 2023-11-24T08:14:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.701+0000 7f66bbfff640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 0x7f66c006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.701+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66c8088730 con 0x7f66d40cf450 2023-11-24T08:14:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.701+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 0x7f66c006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.702+0000 7f66d2ffd640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 0x7f66c006e500 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:12.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.705+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66c8054ec0 con 0x7f66d40cf450 2023-11-24T08:14:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.935+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66d415a8c0 con 0x7f66d40cf450 2023-11-24T08:14:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.936+0000 7f66bbfff640 1 -- 172.21.15.107:0/1601933971 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f66c801f020 con 0x7f66d40cf450 2023-11-24T08:14:12.938 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:12.938 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:14:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 msgr2=0x7f66c006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 0x7f66c006e500 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c403c040 comp rx=0 tx=0).stop 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 msgr2=0x7f66d4157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f66c800c7f0 tx=0x7f66c800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 shutdown_connections 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c006c040 0x7f66c006e500 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66d415c7f0 0x7f66d4154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66d40cf450 0x7f66d4157470 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 --2- 172.21.15.107:0/1601933971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66d40cdfb0 0x7f66d41f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 >> 172.21.15.107:0/1601933971 conn(0x7f66d406a920 msgr2=0x7f66d415cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 shutdown_connections 2023-11-24T08:14:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:12.939+0000 7f66d9836640 1 -- 172.21.15.107:0/1601933971 wait complete. 2023-11-24T08:14:12.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.168+0000 7fe256f47640 1 Processor -- start 2023-11-24T08:14:13.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.169+0000 7fe256f47640 1 -- start start 2023-11-24T08:14:13.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 0x7fe250071460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 0x7fe25006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe25006aa40 0x7fe250160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25006aed0 con 0x7fe25006aa40 2023-11-24T08:14:13.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe250071d60 con 0x7fe250146d60 2023-11-24T08:14:13.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.170+0000 7fe256f47640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe250072040 con 0x7fe2500d0a30 2023-11-24T08:14:13.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe25006aa40 0x7fe250160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe25006aa40 0x7fe250160ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41624/0 (socket says 172.21.15.107:41624) 2023-11-24T08:14:13.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 -- 172.21.15.107:0/430754534 learned_addr learned my addr 172.21.15.107:0/430754534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe255f45640 1 --2- 172.21.15.107:0/430754534 >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 0x7fe250071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 -- 172.21.15.107:0/430754534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 msgr2=0x7fe25006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe255744640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 0x7fe25006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 0x7fe25006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 -- 172.21.15.107:0/430754534 >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 msgr2=0x7fe250071460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 --2- 172.21.15.107:0/430754534 >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 0x7fe250071460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe256746640 1 -- 172.21.15.107:0/430754534 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250072320 con 0x7fe25006aa40 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe255f45640 1 --2- 172.21.15.107:0/430754534 >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 0x7fe250071460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:13.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.171+0000 7fe255744640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 0x7fe25006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.172+0000 7fe256746640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe25006aa40 0x7fe250160ea0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe2500b6160 tx=0x7fe24802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f7d5c6c4095714 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe254f43640 1 -- 172.21.15.107:0/430754534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24802ca80 con 0x7fe25006aa40 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe254f43640 1 -- 172.21.15.107:0/430754534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24802cbe0 con 0x7fe25006aa40 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe254f43640 1 -- 172.21.15.107:0/430754534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2480343f0 con 0x7fe25006aa40 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 -- 172.21.15.107:0/430754534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe25006aa40 msgr2=0x7fe250160ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe25006aa40 0x7fe250160ea0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe2500b6160 tx=0x7fe24802bee0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 -- 172.21.15.107:0/430754534 shutdown_connections 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe25006aa40 0x7fe250160ea0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 --2- 172.21.15.107:0/430754534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2500d0a30 0x7fe25006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 --2- 172.21.15.107:0/430754534 >> v2:172.21.15.120:3300/0 conn(0x7fe250146d60 0x7fe250071460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 -- 172.21.15.107:0/430754534 >> 172.21.15.107:0/430754534 conn(0x7fe2500cfed0 msgr2=0x7fe2500d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.173+0000 7fe256f47640 1 -- 172.21.15.107:0/430754534 shutdown_connections 2023-11-24T08:14:13.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.174+0000 7fe256f47640 1 -- 172.21.15.107:0/430754534 wait complete. 2023-11-24T08:14:13.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.175+0000 7fe256f47640 1 Processor -- start 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.175+0000 7fe256f47640 1 -- start start 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe25006aa40 0x7fe2501f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 0x7fe2501605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501f2d30 con 0x7fe250146d60 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501f67f0 con 0x7fe2500d0a30 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.176+0000 7fe256f47640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501f6ad0 con 0x7fe25006aa40 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41630/0 (socket says 172.21.15.107:41630) 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 -- 172.21.15.107:0/3326836759 learned_addr learned my addr 172.21.15.107:0/3326836759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe255744640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 0x7fe2501605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 -- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe25006aa40 msgr2=0x7fe2501f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe255f45640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe25006aa40 0x7fe2501f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe25006aa40 0x7fe2501f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 -- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 msgr2=0x7fe2501605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 0x7fe2501605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 -- 172.21.15.107:0/3326836759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248009b30 con 0x7fe250146d60 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe255744640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 0x7fe2501605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.177+0000 7fe256746640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fe248036040 tx=0x7fe248004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.178+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24803e450 con 0x7fe250146d60 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.178+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24803e5d0 con 0x7fe250146d60 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.178+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24803e980 con 0x7fe250146d60 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.178+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2501f6db0 con 0x7fe250146d60 2023-11-24T08:14:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.178+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2501f7090 con 0x7fe250146d60 2023-11-24T08:14:13.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.179+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2501f2f00 con 0x7fe250146d60 2023-11-24T08:14:13.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.180+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fe24802cd50 con 0x7fe250146d60 2023-11-24T08:14:13.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.181+0000 7fe2467fc640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 0x7fe23406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.181+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2480ab160 con 0x7fe250146d60 2023-11-24T08:14:13.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.181+0000 7fe255f45640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 0x7fe23406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.182+0000 7fe255f45640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 0x7fe23406e500 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b840 tx=0x7fe24c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.185+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe248077b20 con 0x7fe250146d60 2023-11-24T08:14:13.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.414+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe25004e840 con 0x7fe250146d60 2023-11-24T08:14:13.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.415+0000 7fe2467fc640 1 -- 172.21.15.107:0/3326836759 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe248043020 con 0x7fe250146d60 2023-11-24T08:14:13.417 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:13.418 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:13:36.044229+0000","modified":"2023-11-24T08:13:39.130128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8304":{"gid":8304,"name":"d","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/74799278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":74799278},{"type":"v1","addr":"172.21.15.120:6835","nonce":74799278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-24T08:14:13.418 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 msgr2=0x7fe23406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 0x7fe23406e500 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b840 tx=0x7fe24c008040 comp rx=0 tx=0).stop 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 msgr2=0x7fe250144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fe248036040 tx=0x7fe248004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 shutdown_connections 2023-11-24T08:14:13.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe23406c040 0x7fe23406e500 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe250146d60 0x7fe250144df0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2500d0a30 0x7fe2501605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 --2- 172.21.15.107:0/3326836759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe25006aa40 0x7fe2501f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 >> 172.21.15.107:0/3326836759 conn(0x7fe2500cfed0 msgr2=0x7fe2500cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 shutdown_connections 2023-11-24T08:14:13.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.418+0000 7fe256f47640 1 -- 172.21.15.107:0/3326836759 wait complete. 2023-11-24T08:14:13.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.652+0000 7fc3f8a9f640 1 Processor -- start 2023-11-24T08:14:13.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.652+0000 7fc3f8a9f640 1 -- start start 2023-11-24T08:14:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.653+0000 7fc3f8a9f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f415c410 0x7fc3f415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.653+0000 7fc3f8a9f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc3f40b1de0 0x7fc3f40b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.653+0000 7fc3f8a9f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f415bf80 con 0x7fc3f4144b30 2023-11-24T08:14:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.653+0000 7fc3f8a9f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f40b27b0 con 0x7fc3f40b1de0 2023-11-24T08:14:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.653+0000 7fc3f8a9f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40b1560 con 0x7fc3f415c410 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc3f4144b30 0x7fc3f40b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34698/0 (socket says 172.21.15.107:34698) 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f37fe640 1 -- 172.21.15.107:0/972914268 learned_addr learned my addr 172.21.15.107:0/972914268 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/972914268 >> v2:172.21.15.107:3300/0 conn(0x7fc3f40b1de0 0x7fc3f40b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3ebfff640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f415c410 0x7fc3f415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f2ffd640 1 -- 172.21.15.107:0/972914268 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140287907 0 0) 0x7fc3f415bf80 con 0x7fc3f4144b30 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f2ffd640 1 -- 172.21.15.107:0/972914268 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e0003520 con 0x7fc3f4144b30 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f3fff640 1 -- 172.21.15.107:0/972914268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f415c410 msgr2=0x7fc3f415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f415c410 0x7fc3f415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f3fff640 1 -- 172.21.15.107:0/972914268 >> v1:172.21.15.107:6789/0 conn(0x7fc3f4144b30 legacy=0x7fc3f40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.654+0000 7fc3f3fff640 1 -- 172.21.15.107:0/972914268 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40b1280 con 0x7fc3f40b1de0 2023-11-24T08:14:13.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f40b1de0 0x7fc3f40b2270 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009d90 tx=0x7fc3ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31318d5093cf172 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f2ffd640 1 -- 172.21.15.107:0/972914268 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec02cc50 con 0x7fc3f40b1de0 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f2ffd640 1 -- 172.21.15.107:0/972914268 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec02cdb0 con 0x7fc3f40b1de0 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f2ffd640 1 -- 172.21.15.107:0/972914268 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec034660 con 0x7fc3f40b1de0 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/972914268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f40b1de0 msgr2=0x7fc3f40b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.656+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f40b1de0 0x7fc3f40b2270 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009d90 tx=0x7fc3ec02bea0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/972914268 shutdown_connections 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f40b1de0 0x7fc3f40b2270 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/972914268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f415c410 0x7fc3f415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/972914268 >> 172.21.15.107:0/972914268 conn(0x7fc3f40cb830 msgr2=0x7fc3f40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/972914268 shutdown_connections 2023-11-24T08:14:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.657+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/972914268 wait complete. 2023-11-24T08:14:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.658+0000 7fc3f8a9f640 1 Processor -- start 2023-11-24T08:14:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.659+0000 7fc3f8a9f640 1 -- start start 2023-11-24T08:14:13.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f40b1de0 0x7fc3f413e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 0x7fc3f415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4136200 con 0x7fc3f415c410 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41364e0 con 0x7fc3f4144b30 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f8a9f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41367c0 con 0x7fc3f40b1de0 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41644/0 (socket says 172.21.15.107:41644) 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 -- 172.21.15.107:0/3449299142 learned_addr learned my addr 172.21.15.107:0/3449299142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f37fe640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f40b1de0 0x7fc3f413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3ebfff640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 0x7fc3f415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 -- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f40b1de0 msgr2=0x7fc3f413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f40b1de0 0x7fc3f413e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 -- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 msgr2=0x7fc3f415a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 0x7fc3f415a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3f3fff640 1 -- 172.21.15.107:0/3449299142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec009a70 con 0x7fc3f415c410 2023-11-24T08:14:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.660+0000 7fc3ebfff640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 0x7fc3f415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:13.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f3fff640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009d90 tx=0x7fc3ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec03e610 con 0x7fc3f415c410 2023-11-24T08:14:13.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec03e770 con 0x7fc3f415c410 2023-11-24T08:14:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f4136aa0 con 0x7fc3f415c410 2023-11-24T08:14:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec03eb20 con 0x7fc3f415c410 2023-11-24T08:14:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.661+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f413e650 con 0x7fc3f415c410 2023-11-24T08:14:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.662+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f404e840 con 0x7fc3f415c410 2023-11-24T08:14:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.663+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc3ec03b050 con 0x7fc3f415c410 2023-11-24T08:14:13.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.664+0000 7fc3f0ff9640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 0x7fc3d406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:13.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.664+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3ec0aaa20 con 0x7fc3f415c410 2023-11-24T08:14:13.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.665+0000 7fc3f37fe640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 0x7fc3d406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:13.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.665+0000 7fc3f37fe640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 0x7fc3d406e500 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40011e0 tx=0x7fc3e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:13.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.669+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3ec077640 con 0x7fc3f415c410 2023-11-24T08:14:13.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.864+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3f406d860 con 0x7fc3f415c410 2023-11-24T08:14:13.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.866+0000 7fc3f0ff9640 1 -- 172.21.15.107:0/3449299142 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16527 (secure 0 0 0) 0x7fc3ec04e020 con 0x7fc3f415c410 2023-11-24T08:14:13.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:13.868 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:13:37.619539+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:13:33.870908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-11-24T08:13:35.060534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:13.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.868+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 msgr2=0x7fc3d406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.868+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 0x7fc3d406e500 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40011e0 tx=0x7fc3e4009040 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.868+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 msgr2=0x7fc3f415a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.868+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009d90 tx=0x7fc3ec004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.868+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 shutdown_connections 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3d406c040 0x7fc3d406e500 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f415c410 0x7fc3f415a710 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3f4144b30 0x7fc3f415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 --2- 172.21.15.107:0/3449299142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3f40b1de0 0x7fc3f413e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 >> 172.21.15.107:0/3449299142 conn(0x7fc3f40cb830 msgr2=0x7fc3f415c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 shutdown_connections 2023-11-24T08:14:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:13.869+0000 7fc3f8a9f640 1 -- 172.21.15.107:0/3449299142 wait complete. 2023-11-24T08:14:13.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:14:14.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.090+0000 7f6e7a5a7640 1 Processor -- start 2023-11-24T08:14:14.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.090+0000 7f6e7a5a7640 1 -- start start 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e7a5a7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e7415b1d0 0x7f6e74071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e7a5a7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e740b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e7a5a7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e740ca0e0 con 0x7f6e740b1080 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e7a5a7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7415fde0 con 0x7f6e7415b1d0 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e7a5a7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740b1510 con 0x7f6e74070550 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e79da6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e740b1080 0x7f6e7406c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34702/0 (socket says 172.21.15.107:34702) 2023-11-24T08:14:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e79da6640 1 -- 172.21.15.107:0/2262300155 learned_addr learned my addr 172.21.15.107:0/2262300155 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.092+0000 7f6e78da4640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e740b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e795a5640 1 --2- 172.21.15.107:0/2262300155 >> v2:172.21.15.120:3300/0 conn(0x7f6e7415b1d0 0x7f6e74071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595856646 0 0) 0x7f6e740ca0e0 con 0x7f6e740b1080 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e60003520 con 0x7f6e740b1080 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478901529 0 0) 0x7f6e60003520 con 0x7f6e740b1080 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 msgr2=0x7f6e740b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e740b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 >> v2:172.21.15.120:3300/0 conn(0x7f6e7415b1d0 msgr2=0x7f6e74071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/2262300155 >> v2:172.21.15.120:3300/0 conn(0x7f6e7415b1d0 0x7f6e74071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74072800 con 0x7f6e740b1080 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6e6c0019f0 con 0x7f6e740b1080 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 >> v1:172.21.15.107:6789/0 conn(0x7f6e740b1080 legacy=0x7f6e7406c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e60003970 0x7f6e60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e60004d00 0x7f6e60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7415fde0 con 0x7f6e60004d00 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740b1510 con 0x7f6e60003970 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.093+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6000b980 con 0x7f6e74070550 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e60004d00 0x7f6e60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e78da4640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e795a5640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e60003970 0x7f6e60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 -- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 msgr2=0x7f6e600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e74070550 0x7f6e600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 -- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e60003970 msgr2=0x7f6e60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e60003970 0x7f6e60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e79da6640 1 -- 172.21.15.107:0/2262300155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60003520 con 0x7f6e60004d00 2023-11-24T08:14:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.094+0000 7f6e795a5640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e60003970 0x7f6e60003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.095+0000 7f6e79da6640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e60004d00 0x7f6e60005180 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00a860 tx=0x7f6e600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289cc93ff84e9975 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.095+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c034630 con 0x7f6e60004d00 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.095+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c02f070 con 0x7f6e60004d00 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.095+0000 7f6e6bfff640 1 -- 172.21.15.107:0/2262300155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0349f0 con 0x7f6e60004d00 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/2262300155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e60004d00 msgr2=0x7f6e60005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/2262300155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e60004d00 0x7f6e60005180 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00a860 tx=0x7f6e600057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e79da6640 1 -- 172.21.15.107:0/2262300155 reap_dead start 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/2262300155 shutdown_connections 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/2262300155 >> 172.21.15.107:0/2262300155 conn(0x7f6e740cbb40 msgr2=0x7f6e740cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/2262300155 shutdown_connections 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.096+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/2262300155 wait complete. 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.098+0000 7f6e7a5a7640 1 Processor -- start 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.098+0000 7f6e7a5a7640 1 -- start start 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e60004d00 0x7f6e7409c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740c4010 0x7f6e740c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740c23c0 con 0x7f6e740c1a00 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740b3470 con 0x7f6e740c4010 2023-11-24T08:14:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e7a5a7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740b2860 con 0x7f6e60004d00 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e78da4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e78da4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41668/0 (socket says 172.21.15.107:41668) 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.099+0000 7f6e78da4640 1 -- 172.21.15.107:0/382618280 learned_addr learned my addr 172.21.15.107:0/382618280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e795a5640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e60004d00 0x7f6e7409c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e79da6640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740c4010 0x7f6e740c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 -- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e60004d00 msgr2=0x7f6e7409c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e60004d00 0x7f6e7409c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 -- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740c4010 msgr2=0x7f6e740c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740c4010 0x7f6e740c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 -- 172.21.15.107:0/382618280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e740b2b40 con 0x7f6e740c1a00 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e78da4640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f6e64011600 tx=0x7f6e64011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6400f920 con 0x7f6e740c1a00 2023-11-24T08:14:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6400fa80 con 0x7f6e740c1a00 2023-11-24T08:14:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6400fdf0 con 0x7f6e740c1a00 2023-11-24T08:14:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.100+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e740b2e20 con 0x7f6e740c1a00 2023-11-24T08:14:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.101+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e740c49d0 con 0x7f6e740c1a00 2023-11-24T08:14:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.102+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7404e840 con 0x7f6e740c1a00 2023-11-24T08:14:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.103+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f6e64015460 con 0x7f6e740c1a00 2023-11-24T08:14:14.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.104+0000 7f6e69ffb640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 0x7f6e5806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.104+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e64088670 con 0x7f6e740c1a00 2023-11-24T08:14:14.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.104+0000 7f6e795a5640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 0x7f6e5806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.105+0000 7f6e795a5640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 0x7f6e5806e530 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6e740cee70 tx=0x7f6e70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.108+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e640551d0 con 0x7f6e740c1a00 2023-11-24T08:14:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.337+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6e7409ce40 con 0x7f6e740c1a00 2023-11-24T08:14:14.340 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:14:14.338+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:14:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.341+0000 7f6e69ffb640 1 -- 172.21.15.107:0/382618280 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6e64022090 con 0x7f6e740c1a00 2023-11-24T08:14:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:14:14.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 msgr2=0x7f6e5806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 0x7f6e5806e530 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6e740cee70 tx=0x7f6e70008040 comp rx=0 tx=0).stop 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 msgr2=0x7f6e740c1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f6e64011600 tx=0x7f6e64011ac0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 shutdown_connections 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e5806c070 0x7f6e5806e530 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740c4010 0x7f6e740c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e740c1a00 0x7f6e740c1e80 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 --2- 172.21.15.107:0/382618280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e60004d00 0x7f6e7409c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 >> 172.21.15.107:0/382618280 conn(0x7f6e740cbb40 msgr2=0x7f6e740ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:14.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 shutdown_connections 2023-11-24T08:14:14.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.344+0000 7f6e7a5a7640 1 -- 172.21.15.107:0/382618280 wait complete. 2023-11-24T08:14:14.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:14:14.378 INFO:tasks.ceph.mds.d.smithi120.stderr: -9336> 2023-11-24T08:13:31.188+0000 7f448467f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-11-24T08:14:14.383 INFO:tasks.ceph.mds.d.smithi120.stderr: -8085> 2023-11-24T08:13:42.562+0000 7f447e673640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:14:14.459 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:14:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.566+0000 7fc704ada640 1 Processor -- start 2023-11-24T08:14:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.566+0000 7fc704ada640 1 -- start start 2023-11-24T08:14:14.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.567+0000 7fc704ada640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 0x7fc7000cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc704ada640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7000c59b0 0x7fc70006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc704ada640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7000cc3a0 con 0x7fc7000cbf20 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc704ada640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70015b820 con 0x7fc7000c59b0 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc704ada640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70015eea0 con 0x7fc70015b440 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6f7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc7000cbf20 0x7fc7000c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34704/0 (socket says 172.21.15.107:34704) 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4043762170 learned_addr learned my addr 172.21.15.107:0/4043762170 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/4043762170 >> v2:172.21.15.107:3300/0 conn(0x7fc7000c59b0 0x7fc70006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fe59c640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 0x7fc7000cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fdd9b640 1 -- 172.21.15.107:0/4043762170 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575152877 0 0) 0x7fc6e8001110 con 0x7fc7000cbf20 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fdd9b640 1 -- 172.21.15.107:0/4043762170 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003520 con 0x7fc7000cbf20 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fed9d640 1 -- 172.21.15.107:0/4043762170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 msgr2=0x7fc7000cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 0x7fc7000cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fed9d640 1 -- 172.21.15.107:0/4043762170 >> v1:172.21.15.107:6789/0 conn(0x7fc7000cbf20 legacy=0x7fc7000c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.568+0000 7fc6fed9d640 1 -- 172.21.15.107:0/4043762170 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000abf30 con 0x7fc7000c59b0 2023-11-24T08:14:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.569+0000 7fc6fe59c640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 0x7fc7000cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:14:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7000c59b0 0x7fc70006c5f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009d90 tx=0x7fc6f802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d1f0d7c8ef936d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc6fdd9b640 1 -- 172.21.15.107:0/4043762170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f802cbc0 con 0x7fc7000c59b0 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc6fdd9b640 1 -- 172.21.15.107:0/4043762170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f802cd20 con 0x7fc7000c59b0 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc6fdd9b640 1 -- 172.21.15.107:0/4043762170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8034660 con 0x7fc7000c59b0 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 -- 172.21.15.107:0/4043762170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7000c59b0 msgr2=0x7fc70006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7000c59b0 0x7fc70006c5f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009d90 tx=0x7fc6f802be10 comp rx=0 tx=0).stop 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 -- 172.21.15.107:0/4043762170 shutdown_connections 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7000c59b0 0x7fc70006c5f0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 --2- 172.21.15.107:0/4043762170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc70015b440 0x7fc7000cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.570+0000 7fc704ada640 1 -- 172.21.15.107:0/4043762170 >> 172.21.15.107:0/4043762170 conn(0x7fc7000b0c50 msgr2=0x7fc7000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.571+0000 7fc704ada640 1 -- 172.21.15.107:0/4043762170 shutdown_connections 2023-11-24T08:14:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.571+0000 7fc704ada640 1 -- 172.21.15.107:0/4043762170 wait complete. 2023-11-24T08:14:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.572+0000 7fc704ada640 1 Processor -- start 2023-11-24T08:14:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.572+0000 7fc704ada640 1 -- start start 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7000c59b0 0x7fc700156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7000cbf20 0x7fc7000afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700159a00 con 0x7fc70015b440 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700159e20 con 0x7fc7000cbf20 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.573+0000 7fc704ada640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc700158d80 con 0x7fc7000c59b0 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41686/0 (socket says 172.21.15.107:41686) 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 -- 172.21.15.107:0/640152314 learned_addr learned my addr 172.21.15.107:0/640152314 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 -- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7000c59b0 msgr2=0x7fc700156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7000c59b0 0x7fc700156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 -- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7000cbf20 msgr2=0x7fc7000afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7000cbf20 0x7fc7000afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 -- 172.21.15.107:0/640152314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8009a70 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6fed9d640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009d90 tx=0x7fc6f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.574+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8040650 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.575+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f802ce90 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.575+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80409f0 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.575+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700159060 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.575+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc700159620 con 0x7fc70015b440 2023-11-24T08:14:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.576+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70004e8b0 con 0x7fc70015b440 2023-11-24T08:14:14.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.577+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc6f8040b50 con 0x7fc70015b440 2023-11-24T08:14:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.578+0000 7fc6f6ffd640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 0x7fc6e006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.578+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc6f80aae90 con 0x7fc70015b440 2023-11-24T08:14:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.578+0000 7fc6fe59c640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 0x7fc6e006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.579+0000 7fc6fe59c640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 0x7fc6e006e500 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0006fd0 tx=0x7fc6f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:14.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.583+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6f8077990 con 0x7fc70015b440 2023-11-24T08:14:14.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:14.795+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc70006dc80 con 0x7fc70015b440 2023-11-24T08:14:15.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.344+0000 7fc6f6ffd640 1 -- 172.21.15.107:0/640152314 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc6f8009450 con 0x7fc70015b440 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.348+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 msgr2=0x7fc6e006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.348+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 0x7fc6e006e500 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0006fd0 tx=0x7fc6f0008040 comp rx=0 tx=0).stop 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.348+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 msgr2=0x7fc7001554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.348+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009d90 tx=0x7fc6f8004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.348+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 shutdown_connections 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc6e006c040 0x7fc6e006e500 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70015b440 0x7fc7001554b0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7000cbf20 0x7fc7000afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 --2- 172.21.15.107:0/640152314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7000c59b0 0x7fc700156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 >> 172.21.15.107:0/640152314 conn(0x7fc7000b0c50 msgr2=0x7fc700070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:15.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 shutdown_connections 2023-11-24T08:14:15.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.349+0000 7fc704ada640 1 -- 172.21.15.107:0/640152314 wait complete. 2023-11-24T08:14:15.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:14:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.586+0000 7f6fc12e0640 1 Processor -- start 2023-11-24T08:14:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.586+0000 7f6fc12e0640 1 -- start start 2023-11-24T08:14:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fbc157c70 0x7f6fbc161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fbc15dc80 0x7f6fbc15e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc0cfe10 con 0x7f6fbc157c70 2023-11-24T08:14:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc0ce0b0 con 0x7f6fbc15dc80 2023-11-24T08:14:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.587+0000 7f6fc12e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc161960 con 0x7f6fbc0cf450 2023-11-24T08:14:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fbc157c70 0x7f6fbc161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fbc157c70 0x7f6fbc161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41698/0 (socket says 172.21.15.107:41698) 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 -- 172.21.15.107:0/2928823969 learned_addr learned my addr 172.21.15.107:0/2928823969 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fba7fc640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbb7fe640 1 --2- 172.21.15.107:0/2928823969 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc15dc80 0x7f6fbc15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 -- 172.21.15.107:0/2928823969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 msgr2=0x7f6fbc0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 -- 172.21.15.107:0/2928823969 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc15dc80 msgr2=0x7f6fbc15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/2928823969 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc15dc80 0x7f6fbc15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.588+0000 7f6fbaffd640 1 -- 172.21.15.107:0/2928823969 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc15e650 con 0x7f6fbc157c70 2023-11-24T08:14:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc161df0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0cefc0 tx=0x7f6fac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e270b8ada319e50f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fb9ffb640 1 -- 172.21.15.107:0/2928823969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0317e0 con 0x7f6fbc157c70 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fb9ffb640 1 -- 172.21.15.107:0/2928823969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac031940 con 0x7f6fbc157c70 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fb9ffb640 1 -- 172.21.15.107:0/2928823969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac031cd0 con 0x7f6fbc157c70 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fc12e0640 1 -- 172.21.15.107:0/2928823969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 msgr2=0x7f6fbc161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.590+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc161df0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0cefc0 tx=0x7f6fac02b880 comp rx=0 tx=0).stop 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 -- 172.21.15.107:0/2928823969 shutdown_connections 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/2928823969 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc15dc80 0x7f6fbc15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/2928823969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc161df0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 -- 172.21.15.107:0/2928823969 >> 172.21.15.107:0/2928823969 conn(0x7f6fbc06a920 msgr2=0x7f6fbc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 -- 172.21.15.107:0/2928823969 shutdown_connections 2023-11-24T08:14:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.591+0000 7f6fc12e0640 1 -- 172.21.15.107:0/2928823969 wait complete. 2023-11-24T08:14:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.592+0000 7f6fc12e0640 1 Processor -- start 2023-11-24T08:14:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.593+0000 7f6fc12e0640 1 -- start start 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.593+0000 7f6fc12e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.593+0000 7f6fc12e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.593+0000 7f6fc12e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 0x7f6fbc148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fc12e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc1463d0 con 0x7f6fbc157c70 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fc12e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0b0060 con 0x7f6fbc15dc80 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fc12e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc1457c0 con 0x7f6fbc0cf450 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41700/0 (socket says 172.21.15.107:41700) 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 -- 172.21.15.107:0/499017304 learned_addr learned my addr 172.21.15.107:0/499017304 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fbb7fe640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 0x7f6fbc148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 -- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 msgr2=0x7f6fbc0afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 -- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 msgr2=0x7f6fbc148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 0x7f6fbc148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fba7fc640 1 -- 172.21.15.107:0/499017304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0094d0 con 0x7f6fbc157c70 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.594+0000 7f6fbb7fe640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 0x7f6fbc148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fba7fc640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000d5f0 tx=0x7f6fb000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00125f0 con 0x7f6fbc157c70 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0012750 con 0x7f6fbc157c70 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0012b00 con 0x7f6fbc157c70 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc145aa0 con 0x7f6fbc157c70 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.595+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc146060 con 0x7f6fbc157c70 2023-11-24T08:14:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.596+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc067470 con 0x7f6fbc157c70 2023-11-24T08:14:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.597+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f6fb0012c60 con 0x7f6fbc157c70 2023-11-24T08:14:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.598+0000 7f6fa37fe640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 0x7f6fa806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.598+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6fb00885f0 con 0x7f6fbc157c70 2023-11-24T08:14:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.598+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 0x7f6fa806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.599+0000 7f6fbaffd640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 0x7f6fa806e530 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6fac036000 tx=0x7f6fac03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.603+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fb00bee90 con 0x7f6fbc157c70 2023-11-24T08:14:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:15.801+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fbc160f50 con 0x7f6fbc157c70 2023-11-24T08:14:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.767+0000 7f6fa37fe640 1 -- 172.21.15.107:0/499017304 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6fb0055150 con 0x7f6fbc157c70 2023-11-24T08:14:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.771+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 msgr2=0x7f6fa806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.771+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 0x7f6fa806e530 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6fac036000 tx=0x7f6fac03d040 comp rx=0 tx=0).stop 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.771+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 msgr2=0x7f6fbc0cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.771+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000d5f0 tx=0x7f6fb000dab0 comp rx=0 tx=0).stop 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 shutdown_connections 2023-11-24T08:14:16.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fa806c070 0x7f6fa806e530 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc15dc80 0x7f6fbc148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fbc157c70 0x7f6fbc0cad80 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 --2- 172.21.15.107:0/499017304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0cf450 0x7f6fbc0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 >> 172.21.15.107:0/499017304 conn(0x7f6fbc06a920 msgr2=0x7f6fbc160540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 shutdown_connections 2023-11-24T08:14:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:16.772+0000 7f6fc12e0640 1 -- 172.21.15.107:0/499017304 wait complete. 2023-11-24T08:14:16.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:14:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.010+0000 7fd9d254c640 1 Processor -- start 2023-11-24T08:14:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.010+0000 7fd9d254c640 1 -- start start 2023-11-24T08:14:17.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.011+0000 7fd9d254c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9cc0cf450 0x7fd9cc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9d254c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc0cece0 con 0x7fd9cc0cdfb0 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9d254c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc06d040 con 0x7fd9cc15c7f0 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9d254c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9cc0cfe40 con 0x7fd9cc0cf450 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9c3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9cc0cf450 0x7fd9cc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9d0ae9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9cc0cdfb0 0x7fd9cc0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34708/0 (socket says 172.21.15.107:34708) 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9d0ae9640 1 -- 172.21.15.107:0/1913331180 learned_addr learned my addr 172.21.15.107:0/1913331180 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9c3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9cc0cf450 0x7fd9cc0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41714/0 (socket says 172.21.15.107:41714) 2023-11-24T08:14:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9c3fff640 1 -- 172.21.15.107:0/1913331180 >> v1:172.21.15.120:6789/0 conn(0x7fd9cc15c7f0 legacy=0x7fd9cc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9c3fff640 1 -- 172.21.15.107:0/1913331180 >> v1:172.21.15.107:6789/0 conn(0x7fd9cc0cdfb0 legacy=0x7fd9cc0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9c3fff640 1 -- 172.21.15.107:0/1913331180 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc06d320 con 0x7fd9cc0cf450 2023-11-24T08:14:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.012+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1913331180 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79832774 0 0) 0x7fd9cc0cece0 con 0x7fd9cc0cdfb0 2023-11-24T08:14:17.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9c3fff640 1 --2- 172.21.15.107:0/1913331180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc0cf900 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40097a0 tx=0x7fd9b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5903f4d84170f1fd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:17.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1913331180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4031830 con 0x7fd9cc0cf450 2023-11-24T08:14:17.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1913331180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4031990 con 0x7fd9cc0cf450 2023-11-24T08:14:17.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9cb7fe640 1 -- 172.21.15.107:0/1913331180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4031d20 con 0x7fd9cc0cf450 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9d254c640 1 -- 172.21.15.107:0/1913331180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 msgr2=0x7fd9cc0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.014+0000 7fd9d254c640 1 --2- 172.21.15.107:0/1913331180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc0cf900 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40097a0 tx=0x7fd9b402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.015+0000 7fd9d254c640 1 -- 172.21.15.107:0/1913331180 shutdown_connections 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.015+0000 7fd9d254c640 1 --2- 172.21.15.107:0/1913331180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc0cf900 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.015+0000 7fd9d254c640 1 -- 172.21.15.107:0/1913331180 >> 172.21.15.107:0/1913331180 conn(0x7fd9cc06a920 msgr2=0x7fd9cc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.015+0000 7fd9d254c640 1 -- 172.21.15.107:0/1913331180 shutdown_connections 2023-11-24T08:14:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.015+0000 7fd9d254c640 1 -- 172.21.15.107:0/1913331180 wait complete. 2023-11-24T08:14:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.016+0000 7fd9d254c640 1 Processor -- start 2023-11-24T08:14:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 -- start start 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 0x7fd9cc1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9cc15c7f0 0x7fd9cc098510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1596d0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1f1570 con 0x7fd9cc15c7f0 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.017+0000 7fd9d254c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc158990 con 0x7fd9cc0cdfb0 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41730/0 (socket says 172.21.15.107:41730) 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 -- 172.21.15.107:0/3401321428 learned_addr learned my addr 172.21.15.107:0/3401321428 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9cbfff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 0x7fd9cc1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 -- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 msgr2=0x7fd9cc1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 0x7fd9cc1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9d0ae9640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9cc15c7f0 0x7fd9cc098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 -- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9cc15c7f0 msgr2=0x7fd9cc098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9cc15c7f0 0x7fd9cc098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9c3fff640 1 -- 172.21.15.107:0/3401321428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b4009400 con 0x7fd9cc0cf450 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.018+0000 7fd9cbfff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 0x7fd9cc1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9c3fff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40098c0 tx=0x7fd9b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b403ba70 con 0x7fd9cc0cf450 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b403bbd0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40453f0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc158c70 con 0x7fd9cc0cf450 2023-11-24T08:14:17.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.019+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc159c00 con 0x7fd9cc0cf450 2023-11-24T08:14:17.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.020+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc067470 con 0x7fd9cc0cf450 2023-11-24T08:14:17.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.021+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fd9b404e450 con 0x7fd9cc0cf450 2023-11-24T08:14:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.022+0000 7fd9c97fa640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 0x7fd9ac06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.022+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fd9b40aa7d0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.022+0000 7fd9cbfff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 0x7fd9ac06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.023+0000 7fd9cbfff640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 0x7fd9ac06e500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00a830 tx=0x7fd9bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:17.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.026+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9b4076fc0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.218+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd9cc0cd730 con 0x7fd9cc0cf450 2023-11-24T08:14:17.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.777+0000 7fd9c97fa640 1 -- 172.21.15.107:0/3401321428 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd9b407a5e0 con 0x7fd9cc0cf450 2023-11-24T08:14:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 msgr2=0x7fd9ac06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 0x7fd9ac06e500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00a830 tx=0x7fd9bc008040 comp rx=0 tx=0).stop 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 msgr2=0x7fd9cc155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40098c0 tx=0x7fd9b40046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 shutdown_connections 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd9ac06c040 0x7fd9ac06e500 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9cc15c7f0 0x7fd9cc098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9cc0cf450 0x7fd9cc155e50 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 --2- 172.21.15.107:0/3401321428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9cc0cdfb0 0x7fd9cc1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 >> 172.21.15.107:0/3401321428 conn(0x7fd9cc06a920 msgr2=0x7fd9cc0af110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 shutdown_connections 2023-11-24T08:14:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:17.781+0000 7fd9d254c640 1 -- 172.21.15.107:0/3401321428 wait complete. 2023-11-24T08:14:17.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:14:18.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.006+0000 7ff30c71d640 1 Processor -- start 2023-11-24T08:14:18.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.006+0000 7ff30c71d640 1 -- start start 2023-11-24T08:14:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.007+0000 7ff30c71d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 0x7ff30415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30c71d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3040c49f0 0x7ff30406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30c71d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30415ec90 con 0x7ff30406dc40 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30c71d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30406e090 con 0x7ff3040c49f0 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30c71d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3040cd530 con 0x7ff304072160 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3040c49f0 0x7ff30406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3040c49f0 0x7ff30406d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55400/0 (socket says 172.21.15.107:55400) 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 -- 172.21.15.107:0/1577558167 learned_addr learned my addr 172.21.15.107:0/1577558167 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30acba640 1 --1- 172.21.15.107:0/1577558167 >> v1:172.21.15.107:6789/0 conn(0x7ff30406dc40 0x7ff3040cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45718/0 (socket says 172.21.15.107:45718) 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff30a4b9640 1 --2- 172.21.15.107:0/1577558167 >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 0x7ff30415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 -- 172.21.15.107:0/1577558167 >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 msgr2=0x7ff30415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 --2- 172.21.15.107:0/1577558167 >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 0x7ff30415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.008+0000 7ff309cb8640 1 -- 172.21.15.107:0/1577558167 >> v1:172.21.15.107:6789/0 conn(0x7ff30406dc40 legacy=0x7ff3040cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.009+0000 7ff309cb8640 1 -- 172.21.15.107:0/1577558167 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040c4ed0 con 0x7ff3040c49f0 2023-11-24T08:14:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.009+0000 7ff30a4b9640 1 --2- 172.21.15.107:0/1577558167 >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 0x7ff30415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff309cb8640 1 --2- 172.21.15.107:0/1577558167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3040c49f0 0x7ff30406d700 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ff3040ccae0 tx=0x7ff2f402e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3efa0360242a6f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff3094b7640 1 -- 172.21.15.107:0/1577558167 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4036440 con 0x7ff3040c49f0 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff3094b7640 1 -- 172.21.15.107:0/1577558167 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f40365a0 con 0x7ff3040c49f0 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff3094b7640 1 -- 172.21.15.107:0/1577558167 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4036930 con 0x7ff3040c49f0 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff30c71d640 1 -- 172.21.15.107:0/1577558167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3040c49f0 msgr2=0x7ff30406d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.010+0000 7ff30c71d640 1 --2- 172.21.15.107:0/1577558167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3040c49f0 0x7ff30406d700 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ff3040ccae0 tx=0x7ff2f402e180 comp rx=0 tx=0).stop 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 -- 172.21.15.107:0/1577558167 shutdown_connections 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 --2- 172.21.15.107:0/1577558167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3040c49f0 0x7ff30406d700 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 --2- 172.21.15.107:0/1577558167 >> v2:172.21.15.120:3300/0 conn(0x7ff304072160 0x7ff30415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 -- 172.21.15.107:0/1577558167 >> 172.21.15.107:0/1577558167 conn(0x7ff3040b0c50 msgr2=0x7ff3040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 -- 172.21.15.107:0/1577558167 shutdown_connections 2023-11-24T08:14:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.011+0000 7ff30c71d640 1 -- 172.21.15.107:0/1577558167 wait complete. 2023-11-24T08:14:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.012+0000 7ff30c71d640 1 Processor -- start 2023-11-24T08:14:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.013+0000 7ff30c71d640 1 -- start start 2023-11-24T08:14:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.013+0000 7ff30c71d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.013+0000 7ff30c71d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 0x7ff3041522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.013+0000 7ff30c71d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3040c49f0 0x7ff3041585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.013+0000 7ff30c71d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3041527e0 con 0x7ff30406dc40 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30c71d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b2de0 con 0x7ff304072160 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30c71d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3041579b0 con 0x7ff3040c49f0 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55408/0 (socket says 172.21.15.107:55408) 2023-11-24T08:14:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 -- 172.21.15.107:0/3320504484 learned_addr learned my addr 172.21.15.107:0/3320504484 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff309cb8640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 0x7ff3041522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 -- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3040c49f0 msgr2=0x7ff3041585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3040c49f0 0x7ff3041585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 -- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 msgr2=0x7ff3041522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 0x7ff3041522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff30a4b9640 1 -- 172.21.15.107:0/3320504484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f400b770 con 0x7ff30406dc40 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.014+0000 7ff309cb8640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 0x7ff3041522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff30a4b9640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c7f0 tx=0x7ff30000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300007a30 con 0x7ff30406dc40 2023-11-24T08:14:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300007b90 con 0x7ff30406dc40 2023-11-24T08:14:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30000ce40 con 0x7ff30406dc40 2023-11-24T08:14:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3040b3940 con 0x7ff30406dc40 2023-11-24T08:14:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.015+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff304158b10 con 0x7ff30406dc40 2023-11-24T08:14:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.016+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3040cfdd0 con 0x7ff30406dc40 2023-11-24T08:14:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.017+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7ff3000113f0 con 0x7ff30406dc40 2023-11-24T08:14:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.018+0000 7ff2faffd640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 0x7ff2e806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.018+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff3000886e0 con 0x7ff30406dc40 2023-11-24T08:14:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.018+0000 7ff309cb8640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 0x7ff2e806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.019+0000 7ff309cb8640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 0x7ff2e806e530 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4038000 tx=0x7ff2f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.022+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff300054e70 con 0x7ff30406dc40 2023-11-24T08:14:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.217+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff30404e8b0 con 0x7ff30406dc40 2023-11-24T08:14:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.222+0000 7ff2faffd640 1 -- 172.21.15.107:0/3320504484 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff30001f0a0 con 0x7ff30406dc40 2023-11-24T08:14:18.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 msgr2=0x7ff2e806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 0x7ff2e806e530 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4038000 tx=0x7ff2f4008040 comp rx=0 tx=0).stop 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 msgr2=0x7ff3041572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c7f0 tx=0x7ff30000cbf0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 shutdown_connections 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e806c070 0x7ff2e806e530 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3040c49f0 0x7ff3041585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff304072160 0x7ff3041522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 --2- 172.21.15.107:0/3320504484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30406dc40 0x7ff3041572d0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.225+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 >> 172.21.15.107:0/3320504484 conn(0x7ff3040b0c50 msgr2=0x7ff3040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.226+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 shutdown_connections 2023-11-24T08:14:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.226+0000 7ff30c71d640 1 -- 172.21.15.107:0/3320504484 wait complete. 2023-11-24T08:14:18.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/periodic_async_work 2023-11-24T08:14:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.450+0000 7f25ca9a5640 1 Processor -- start 2023-11-24T08:14:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.450+0000 7f25ca9a5640 1 -- start start 2023-11-24T08:14:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.451+0000 7f25ca9a5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25c40ccbd0 0x7f25c41660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.451+0000 7f25ca9a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c40c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.451+0000 7f25ca9a5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c41655f0 con 0x7f25c40ccfb0 2023-11-24T08:14:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.451+0000 7f25ca9a5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40cc380 con 0x7f25c40ccbd0 2023-11-24T08:14:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.451+0000 7f25ca9a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4162b30 con 0x7f25c40c9650 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.452+0000 7f25c3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25c40ccbd0 0x7f25c41660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.452+0000 7f25c8f42640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c40c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.452+0000 7f25c3fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25c40ccbd0 0x7f25c41660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:59442/0 (socket says 172.21.15.107:59442) 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.452+0000 7f25c3fff640 1 -- 172.21.15.107:0/2241860003 learned_addr learned my addr 172.21.15.107:0/2241860003 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.453+0000 7f25c3fff640 1 -- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 msgr2=0x7f25c40c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.453+0000 7f25c3fff640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c40c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.453+0000 7f25c3fff640 1 -- 172.21.15.107:0/2241860003 >> v1:172.21.15.120:6789/0 conn(0x7f25c40ccfb0 legacy=0x7f25c4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.453+0000 7f25c3fff640 1 -- 172.21.15.107:0/2241860003 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4162e10 con 0x7f25c40ccbd0 2023-11-24T08:14:18.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.453+0000 7f25c8f42640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c40c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:14:18.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.454+0000 7f25c3fff640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c41660b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009870 tx=0x7f25b402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d583bff8a35512d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.455+0000 7f25c37fe640 1 -- 172.21.15.107:0/2241860003 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4030840 con 0x7f25c40ccbd0 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.455+0000 7f25c37fe640 1 -- 172.21.15.107:0/2241860003 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b40309a0 con 0x7f25c40ccbd0 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.455+0000 7f25c37fe640 1 -- 172.21.15.107:0/2241860003 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4030d30 con 0x7f25c40ccbd0 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 -- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 msgr2=0x7f25c41660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c41660b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009870 tx=0x7f25b402b980 comp rx=0 tx=0).stop 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 -- 172.21.15.107:0/2241860003 shutdown_connections 2023-11-24T08:14:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c40c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/2241860003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c41660b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 -- 172.21.15.107:0/2241860003 >> 172.21.15.107:0/2241860003 conn(0x7f25c40b0c50 msgr2=0x7f25c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 -- 172.21.15.107:0/2241860003 shutdown_connections 2023-11-24T08:14:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.456+0000 7f25ca9a5640 1 -- 172.21.15.107:0/2241860003 wait complete. 2023-11-24T08:14:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.458+0000 7f25ca9a5640 1 Processor -- start 2023-11-24T08:14:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.458+0000 7f25ca9a5640 1 -- start start 2023-11-24T08:14:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c4158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4164330 con 0x7f25c40ccfb0 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4163a20 con 0x7f25c40ccbd0 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25ca9a5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4163550 con 0x7f25c40c9650 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55414/0 (socket says 172.21.15.107:55414) 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 -- 172.21.15.107:0/1178739444 learned_addr learned my addr 172.21.15.107:0/1178739444 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25bbfff640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c3fff640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c4158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 -- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 msgr2=0x7f25c4158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c4158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.459+0000 7f25c8f42640 1 -- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 msgr2=0x7f25c415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c8f42640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c415d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c8f42640 1 -- 172.21.15.107:0/1178739444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc009a70 con 0x7f25c40ccfb0 2023-11-24T08:14:18.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c8f42640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00efc0 tx=0x7f25bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc009cd0 con 0x7f25c40ccfb0 2023-11-24T08:14:18.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00f040 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc003990 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b40094d0 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.460+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c4163f50 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.461+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c4164470 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.462+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f25bc014480 con 0x7f25c40ccfb0 2023-11-24T08:14:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.463+0000 7f25c17fa640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 0x7f25a406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.463+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f25bc088e50 con 0x7f25c40ccfb0 2023-11-24T08:14:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.464+0000 7f25c3fff640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 0x7f25a406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.464+0000 7f25c3fff640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 0x7f25a406e500 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036000 tx=0x7f25b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.468+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25bc055950 con 0x7f25c40ccfb0 2023-11-24T08:14:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.659+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/periodic_async_work"} v 0) v1 -- 0x7f25c4163830 con 0x7f25c40ccfb0 2023-11-24T08:14:18.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.666+0000 7f25c17fa640 1 -- 172.21.15.107:0/1178739444 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/periodic_async_work"}]=0 v34) v1 ==== 114+0+0 (secure 0 0 0) 0x7f25bc018020 con 0x7f25c40ccfb0 2023-11-24T08:14:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 msgr2=0x7f25a406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 0x7f25a406e500 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036000 tx=0x7f25b4008030 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 msgr2=0x7f25c4164750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00efc0 tx=0x7f25bc00c490 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 shutdown_connections 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25a406c040 0x7f25a406e500 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c40ccfb0 0x7f25c4164750 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25c40ccbd0 0x7f25c415d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 --2- 172.21.15.107:0/1178739444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25c40c9650 0x7f25c4158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.669+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 >> 172.21.15.107:0/1178739444 conn(0x7f25c40b0c50 msgr2=0x7f25c40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.670+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 shutdown_connections 2023-11-24T08:14:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.670+0000 7f25ca9a5640 1 -- 172.21.15.107:0/1178739444 wait complete. 2023-11-24T08:14:18.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work' 2023-11-24T08:14:18.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.895+0000 7fb54a26b640 1 Processor -- start 2023-11-24T08:14:18.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.895+0000 7fb54a26b640 1 -- start start 2023-11-24T08:14:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.896+0000 7fb54a26b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb54406de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb54a26b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440b3a40 con 0x7fb5440cba40 2023-11-24T08:14:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb54a26b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440cdfa0 con 0x7fb5440b21d0 2023-11-24T08:14:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb54a26b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440cfb60 con 0x7fb544159ce0 2023-11-24T08:14:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb548808640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb5440cba40 0x7fb544164af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45732/0 (socket says 172.21.15.107:45732) 2023-11-24T08:14:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb548808640 1 -- 172.21.15.107:0/3422021501 learned_addr learned my addr 172.21.15.107:0/3422021501 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb543fff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb54406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288175552 0 0) 0x7fb5440b3a40 con 0x7fb5440cba40 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.897+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb530003520 con 0x7fb5440cba40 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 992298554 0 0) 0x7fb530003520 con 0x7fb5440cba40 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 msgr2=0x7fb54406de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb54406de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 >> v1:172.21.15.120:6789/0 conn(0x7fb5440b21d0 legacy=0x7fb5440cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440cd9e0 con 0x7fb5440cba40 2023-11-24T08:14:18.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb53c0019f0 con 0x7fb5440cba40 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 >> v1:172.21.15.107:6789/0 conn(0x7fb5440cba40 legacy=0x7fb544164af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb5300039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb530003f70 0x7fb5300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530004d00 0x7fb530005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440cfb60 con 0x7fb530003f70 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53000dcd0 con 0x7fb530004d00 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53000de90 con 0x7fb544159ce0 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb53bfff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb530003f70 0x7fb5300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb543fff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb5300039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.898+0000 7fb548808640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530004d00 0x7fb530005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb53bfff640 1 -- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 msgr2=0x7fb5300039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:18.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb53bfff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb5300039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb53bfff640 1 -- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530004d00 msgr2=0x7fb530005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb53bfff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530004d00 0x7fb530005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb53bfff640 1 -- 172.21.15.107:0/3422021501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb530003520 con 0x7fb530003f70 2023-11-24T08:14:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb543fff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb544159ce0 0x7fb5300039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:18.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.899+0000 7fb548808640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530004d00 0x7fb530005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.900+0000 7fb53bfff640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb530003f70 0x7fb5300047c0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb52c001750 tx=0x7fb5300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f523cd8eae8d65ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.900+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0052d0 con 0x7fb530003f70 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.900+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c02e050 con 0x7fb530003f70 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.900+0000 7fb5437fe640 1 -- 172.21.15.107:0/3422021501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c003830 con 0x7fb530003f70 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 -- 172.21.15.107:0/3422021501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb530003f70 msgr2=0x7fb5300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 --2- 172.21.15.107:0/3422021501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb530003f70 0x7fb5300047c0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb52c001750 tx=0x7fb5300057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb548808640 1 -- 172.21.15.107:0/3422021501 reap_dead start 2023-11-24T08:14:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 -- 172.21.15.107:0/3422021501 shutdown_connections 2023-11-24T08:14:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 -- 172.21.15.107:0/3422021501 >> 172.21.15.107:0/3422021501 conn(0x7fb54406a920 msgr2=0x7fb54406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 -- 172.21.15.107:0/3422021501 shutdown_connections 2023-11-24T08:14:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.901+0000 7fb54a26b640 1 -- 172.21.15.107:0/3422021501 wait complete. 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.903+0000 7fb54a26b640 1 Processor -- start 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.903+0000 7fb54a26b640 1 -- start start 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530003f70 0x7fb544150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 0x7fb54413b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544072020 con 0x7fb544071660 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544151780 con 0x7fb530003f70 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb54a26b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441506e0 con 0x7fb5441f2ed0 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55422/0 (socket says 172.21.15.107:55422) 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 -- 172.21.15.107:0/666204357 learned_addr learned my addr 172.21.15.107:0/666204357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb543fff640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530003f70 0x7fb544150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb53bfff640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 0x7fb54413b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 -- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 msgr2=0x7fb54413b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 0x7fb54413b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 -- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530003f70 msgr2=0x7fb544150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530003f70 0x7fb544150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb548808640 1 -- 172.21.15.107:0/666204357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5441509c0 con 0x7fb544071660 2023-11-24T08:14:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.904+0000 7fb53bfff640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 0x7fb54413b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb548808640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb54406e9d0 tx=0x7fb53c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c004d60 con 0x7fb544071660 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c00bb80 con 0x7fb544071660 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c007110 con 0x7fb544071660 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb544150ca0 con 0x7fb544071660 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.905+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb544151260 con 0x7fb544071660 2023-11-24T08:14:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.906+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb544067470 con 0x7fb544071660 2023-11-24T08:14:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.907+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fb53c007270 con 0x7fb544071660 2023-11-24T08:14:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.908+0000 7fb5417fa640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 0x7fb52406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.908+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb53c0889a0 con 0x7fb544071660 2023-11-24T08:14:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.908+0000 7fb543fff640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 0x7fb52406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:18.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.909+0000 7fb543fff640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 0x7fb52406e500 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb534009aa0 tx=0x7fb534008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:18.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:18.913+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb53c055130 con 0x7fb544071660 2023-11-24T08:14:19.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.174+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work"]} v 0) v1 -- 0x7fb544072830 con 0x7fb544071660 2023-11-24T08:14:19.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.785+0000 7fb5417fa640 1 -- 172.21.15.107:0/666204357 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_periodic_async_work"]}]=0 v491) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb53c058750 con 0x7fb544071660 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.790+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 msgr2=0x7fb52406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.790+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 0x7fb52406e500 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb534009aa0 tx=0x7fb534008040 comp rx=0 tx=0).stop 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.790+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 msgr2=0x7fb544071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.790+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb54406e9d0 tx=0x7fb53c00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 shutdown_connections 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb52406c040 0x7fb52406e500 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb544071660 0x7fb544071ae0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5441f2ed0 0x7fb54413b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 --2- 172.21.15.107:0/666204357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530003f70 0x7fb544150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 >> 172.21.15.107:0/666204357 conn(0x7fb54406a920 msgr2=0x7fb5440ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 shutdown_connections 2023-11-24T08:14:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:19.791+0000 7fb54a26b640 1 -- 172.21.15.107:0/666204357 wait complete. 2023-11-24T08:14:19.815 INFO:tasks.cephfs_test_runner:to validate that the async thread (purge thread in this case) will ... ok 2023-11-24T08:14:19.815 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:14:19.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.021+0000 7f758a2cf640 1 Processor -- start 2023-11-24T08:14:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.021+0000 7f758a2cf640 1 -- start start 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.022+0000 7f758a2cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f758415c7f0 0x7f758406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.022+0000 7f758a2cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75840cf450 0x7f75840cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f758a2cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840cece0 con 0x7f75840cdfb0 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f758a2cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75840cfe40 con 0x7f75840cf450 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f758a2cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75840ceaf0 con 0x7f758415c7f0 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75840cf450 0x7f75840cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75840cf450 0x7f75840cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55434/0 (socket says 172.21.15.107:55434) 2023-11-24T08:14:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 -- 172.21.15.107:0/1861875283 learned_addr learned my addr 172.21.15.107:0/1861875283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f7583fff640 1 --2- 172.21.15.107:0/1861875283 >> v2:172.21.15.120:3300/0 conn(0x7f758415c7f0 0x7f758406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 -- 172.21.15.107:0/1861875283 >> v2:172.21.15.120:3300/0 conn(0x7f758415c7f0 msgr2=0x7f758406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 --2- 172.21.15.107:0/1861875283 >> v2:172.21.15.120:3300/0 conn(0x7f758415c7f0 0x7f758406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 -- 172.21.15.107:0/1861875283 >> v1:172.21.15.120:6789/0 conn(0x7f75840cdfb0 legacy=0x7f75840ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.023+0000 7f75837fe640 1 -- 172.21.15.107:0/1861875283 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758406d040 con 0x7f75840cf450 2023-11-24T08:14:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.024+0000 7f75837fe640 1 --2- 172.21.15.107:0/1861875283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75840cf450 0x7f75840cf900 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f75840d0130 tx=0x7f757802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2e8c47ad3dc259 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f7582ffd640 1 -- 172.21.15.107:0/1861875283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578031890 con 0x7f75840cf450 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f7582ffd640 1 -- 172.21.15.107:0/1861875283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75780319f0 con 0x7f75840cf450 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f7582ffd640 1 -- 172.21.15.107:0/1861875283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578031d80 con 0x7f75840cf450 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 -- 172.21.15.107:0/1861875283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75840cf450 msgr2=0x7f75840cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 --2- 172.21.15.107:0/1861875283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75840cf450 0x7f75840cf900 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f75840d0130 tx=0x7f757802b930 comp rx=0 tx=0).stop 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 -- 172.21.15.107:0/1861875283 shutdown_connections 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 --2- 172.21.15.107:0/1861875283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75840cf450 0x7f75840cf900 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 --2- 172.21.15.107:0/1861875283 >> v2:172.21.15.120:3300/0 conn(0x7f758415c7f0 0x7f758406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.025+0000 7f758a2cf640 1 -- 172.21.15.107:0/1861875283 >> 172.21.15.107:0/1861875283 conn(0x7f758406a920 msgr2=0x7f758406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.026+0000 7f758a2cf640 1 -- 172.21.15.107:0/1861875283 shutdown_connections 2023-11-24T08:14:20.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.026+0000 7f758a2cf640 1 -- 172.21.15.107:0/1861875283 wait complete. 2023-11-24T08:14:20.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.027+0000 7f758a2cf640 1 Processor -- start 2023-11-24T08:14:20.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.027+0000 7f758a2cf640 1 -- start start 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75840cdfb0 0x7f7584150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75840cf450 0x7f75841f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75840b2e00 con 0x7f758415c7f0 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75840b27b0 con 0x7f75840cdfb0 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.028+0000 7f758a2cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7584151420 con 0x7f75840cf450 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55450/0 (socket says 172.21.15.107:55450) 2023-11-24T08:14:20.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 -- 172.21.15.107:0/3263550572 learned_addr learned my addr 172.21.15.107:0/3263550572 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 -- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75840cf450 msgr2=0x7f75841f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f75837fe640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75840cf450 0x7f75841f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f7583fff640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75840cdfb0 0x7f7584150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75840cf450 0x7f75841f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 -- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75840cdfb0 msgr2=0x7f7584150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75840cdfb0 0x7f7584150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 -- 172.21.15.107:0/3263550572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780094d0 con 0x7f758415c7f0 2023-11-24T08:14:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.029+0000 7f758886c640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c7b0 tx=0x7f757c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.030+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c003d80 con 0x7f758415c7f0 2023-11-24T08:14:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.030+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c00ce60 con 0x7f758415c7f0 2023-11-24T08:14:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.030+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0052b0 con 0x7f758415c7f0 2023-11-24T08:14:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.030+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75840b2a20 con 0x7f758415c7f0 2023-11-24T08:14:20.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.030+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75841f3750 con 0x7f758415c7f0 2023-11-24T08:14:20.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.031+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75840b2f40 con 0x7f758415c7f0 2023-11-24T08:14:20.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.032+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f757c006b10 con 0x7f758415c7f0 2023-11-24T08:14:20.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.033+0000 7f7580ff9640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 0x7f757006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.033+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f757c087840 con 0x7f758415c7f0 2023-11-24T08:14:20.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.033+0000 7f7583fff640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 0x7f757006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.034+0000 7f7583fff640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 0x7f757006e500 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f757400b7c0 tx=0x7f7574008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.038+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75840b2f40 con 0x7f758415c7f0 2023-11-24T08:14:20.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.229+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75840b2f40 con 0x7f758415c7f0 2023-11-24T08:14:20.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.231+0000 7f7580ff9640 1 -- 172.21.15.107:0/3263550572 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f75840b2f40 con 0x7f758415c7f0 2023-11-24T08:14:20.233 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:20.233 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":123,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:14:17.769096+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6835/74799278":"2023-11-25T08:14:14.339708+0000","172.21.15.120:6833/74799278":"2023-11-25T08:14:14.339708+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.233+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 msgr2=0x7f757006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.233+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 0x7f757006e500 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f757400b7c0 tx=0x7f7574008040 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.233+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 msgr2=0x7f7584154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.233+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c7b0 tx=0x7f757c00cc70 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 shutdown_connections 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f757006c040 0x7f757006e500 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f758415c7f0 0x7f7584154550 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75840cf450 0x7f75841f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 --2- 172.21.15.107:0/3263550572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75840cdfb0 0x7f7584150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 >> 172.21.15.107:0/3263550572 conn(0x7f758406a920 msgr2=0x7f7584153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 shutdown_connections 2023-11-24T08:14:20.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.234+0000 7f758a2cf640 1 -- 172.21.15.107:0/3263550572 wait complete. 2023-11-24T08:14:20.254 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:14:20.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:14:20.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.469+0000 7f0afc8e6640 1 Processor -- start 2023-11-24T08:14:20.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.469+0000 7f0afc8e6640 1 -- start start 2023-11-24T08:14:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.470+0000 7f0afc8e6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0af806c7c0 0x7f0af8071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.470+0000 7f0afc8e6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 0x7f0af80ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.470+0000 7f0afc8e6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af8070860 con 0x7f0af815aa80 2023-11-24T08:14:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.470+0000 7f0afc8e6640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0af8159f40 con 0x7f0af806c7c0 2023-11-24T08:14:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.470+0000 7f0afc8e6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af815b020 con 0x7f0af8071620 2023-11-24T08:14:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.471+0000 7f0af6d9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0af806c7c0 0x7f0af8071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.471+0000 7f0af6d9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0af806c7c0 0x7f0af8071eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:59500/0 (socket says 172.21.15.107:59500) 2023-11-24T08:14:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.471+0000 7f0af6d9d640 1 -- 172.21.15.107:0/2030059476 learned_addr learned my addr 172.21.15.107:0/2030059476 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.471+0000 7f0af759e640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 0x7f0af80ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.472+0000 7f0af6d9d640 1 -- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 msgr2=0x7f0af80ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.472+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 0x7f0af80ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.472+0000 7f0af6d9d640 1 -- 172.21.15.107:0/2030059476 >> v1:172.21.15.120:6789/0 conn(0x7f0af815aa80 legacy=0x7f0af806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.472+0000 7f0af6d9d640 1 -- 172.21.15.107:0/2030059476 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af8070b40 con 0x7f0af806c7c0 2023-11-24T08:14:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.472+0000 7f0af759e640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 0x7f0af80ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:14:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.474+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af806c7c0 0x7f0af8071eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800bb10 tx=0x7f0ae802e1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd0804efbee447d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.474+0000 7f0af5d9b640 1 -- 172.21.15.107:0/2030059476 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8036440 con 0x7f0af806c7c0 2023-11-24T08:14:20.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.474+0000 7f0af5d9b640 1 -- 172.21.15.107:0/2030059476 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae80365a0 con 0x7f0af806c7c0 2023-11-24T08:14:20.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.474+0000 7f0af5d9b640 1 -- 172.21.15.107:0/2030059476 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8036930 con 0x7f0af806c7c0 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 -- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af806c7c0 msgr2=0x7f0af8071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af806c7c0 0x7f0af8071eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800bb10 tx=0x7f0ae802e1f0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 -- 172.21.15.107:0/2030059476 shutdown_connections 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af8071620 0x7f0af80ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/2030059476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af806c7c0 0x7f0af8071eb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 -- 172.21.15.107:0/2030059476 >> 172.21.15.107:0/2030059476 conn(0x7f0af80cbb40 msgr2=0x7f0af80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:20.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 -- 172.21.15.107:0/2030059476 shutdown_connections 2023-11-24T08:14:20.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.475+0000 7f0afc8e6640 1 -- 172.21.15.107:0/2030059476 wait complete. 2023-11-24T08:14:20.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.477+0000 7f0afc8e6640 1 Processor -- start 2023-11-24T08:14:20.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.477+0000 7f0afc8e6640 1 -- start start 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 0x7f0af8071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af8071620 0x7f0af80b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af80c71d0 con 0x7f0af815aa80 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af80c6b50 con 0x7f0af8071620 2023-11-24T08:14:20.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0afc8e6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af80b2750 con 0x7f0af806c7c0 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55454/0 (socket says 172.21.15.107:55454) 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 -- 172.21.15.107:0/4089728831 learned_addr learned my addr 172.21.15.107:0/4089728831 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af659c640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af8071620 0x7f0af80b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 0x7f0af8071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 -- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 msgr2=0x7f0af8071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 0x7f0af8071d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 -- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af8071620 msgr2=0x7f0af80b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af8071620 0x7f0af80b1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af759e640 1 -- 172.21.15.107:0/4089728831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0009a70 con 0x7f0af815aa80 2023-11-24T08:14:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.478+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 0x7f0af8071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0af759e640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f0af000efc0 tx=0x7f0af0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0009d70 con 0x7f0af815aa80 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af000f040 con 0x7f0af815aa80 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae800b770 con 0x7f0af815aa80 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0003990 con 0x7f0af815aa80 2023-11-24T08:14:20.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.479+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af81555d0 con 0x7f0af815aa80 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.480+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af80c7310 con 0x7f0af815aa80 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.481+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f0af00144a0 con 0x7f0af815aa80 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.482+0000 7f0adf7fe640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 0x7f0ae406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.482+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0af0088df0 con 0x7f0af815aa80 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.482+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 0x7f0ae406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.483+0000 7f0af6d9d640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 0x7f0ae406e530 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8038000 tx=0x7f0ae8032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.486+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0af80c7310 con 0x7f0af815aa80 2023-11-24T08:14:20.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.679+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0af80c7310 con 0x7f0af815aa80 2023-11-24T08:14:20.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.680+0000 7f0adf7fe640 1 -- 172.21.15.107:0/4089728831 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0af80c7310 con 0x7f0af815aa80 2023-11-24T08:14:20.682 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 msgr2=0x7f0ae406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 0x7f0ae406e530 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8038000 tx=0x7f0ae8032000 comp rx=0 tx=0).stop 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 msgr2=0x7f0af814dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f0af000efc0 tx=0x7f0af0009f90 comp rx=0 tx=0).stop 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 shutdown_connections 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ae406c070 0x7f0ae406e530 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0af815aa80 0x7f0af814dfb0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0af8071620 0x7f0af80b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 --2- 172.21.15.107:0/4089728831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0af806c7c0 0x7f0af8071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 >> 172.21.15.107:0/4089728831 conn(0x7f0af80cbb40 msgr2=0x7f0af815a230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 shutdown_connections 2023-11-24T08:14:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.683+0000 7f0afc8e6640 1 -- 172.21.15.107:0/4089728831 wait complete. 2023-11-24T08:14:20.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath' 2023-11-24T08:14:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.911+0000 7f8308b7c640 1 Processor -- start 2023-11-24T08:14:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.911+0000 7f8308b7c640 1 -- start start 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f8308b7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f830415a4f0 0x7f83040b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f8308b7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83040b27d0 0x7f83040ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f8308b7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040b2c00 con 0x7f83040cdfb0 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f8308b7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040ce980 con 0x7f830415a4f0 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f8308b7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040b3a40 con 0x7f83040b27d0 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.912+0000 7f830259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f830415a4f0 0x7f83040b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f830415a4f0 0x7f83040b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55456/0 (socket says 172.21.15.107:55456) 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 -- 172.21.15.107:0/828334348 learned_addr learned my addr 172.21.15.107:0/828334348 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f82f9d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f83040cdfb0 0x7f83040b20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45746/0 (socket says 172.21.15.107:45746) 2023-11-24T08:14:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 -- 172.21.15.107:0/828334348 >> v2:172.21.15.120:3300/0 conn(0x7f83040b27d0 msgr2=0x7f83040ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 --2- 172.21.15.107:0/828334348 >> v2:172.21.15.120:3300/0 conn(0x7f83040b27d0 0x7f83040ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 -- 172.21.15.107:0/828334348 >> v1:172.21.15.107:6789/0 conn(0x7f83040cdfb0 legacy=0x7f83040b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f830259c640 1 -- 172.21.15.107:0/828334348 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040b2e30 con 0x7f830415a4f0 2023-11-24T08:14:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.913+0000 7f8301d9b640 1 -- 172.21.15.107:0/828334348 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593548803 0 0) 0x7f83040b2c00 con 0x7f83040cdfb0 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.914+0000 7f830259c640 1 --2- 172.21.15.107:0/828334348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830415a4f0 0x7f83040b3ed0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f83040d0080 tx=0x7f82f402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301be3a9ad904024 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.914+0000 7f8301d9b640 1 -- 172.21.15.107:0/828334348 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4036440 con 0x7f830415a4f0 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.914+0000 7f8301d9b640 1 -- 172.21.15.107:0/828334348 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f40365a0 con 0x7f830415a4f0 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.914+0000 7f8301d9b640 1 -- 172.21.15.107:0/828334348 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4036930 con 0x7f830415a4f0 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 -- 172.21.15.107:0/828334348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830415a4f0 msgr2=0x7f83040b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 --2- 172.21.15.107:0/828334348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830415a4f0 0x7f83040b3ed0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f83040d0080 tx=0x7f82f402e170 comp rx=0 tx=0).stop 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 -- 172.21.15.107:0/828334348 shutdown_connections 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 --2- 172.21.15.107:0/828334348 >> v2:172.21.15.120:3300/0 conn(0x7f83040b27d0 0x7f83040ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 --2- 172.21.15.107:0/828334348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830415a4f0 0x7f83040b3ed0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 -- 172.21.15.107:0/828334348 >> 172.21.15.107:0/828334348 conn(0x7f83040b0c70 msgr2=0x7f83040b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 -- 172.21.15.107:0/828334348 shutdown_connections 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.915+0000 7f8308b7c640 1 -- 172.21.15.107:0/828334348 wait complete. 2023-11-24T08:14:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.917+0000 7f8308b7c640 1 Processor -- start 2023-11-24T08:14:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.917+0000 7f8308b7c640 1 -- start start 2023-11-24T08:14:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 0x7f830415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 0x7f83041509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041f2640 con 0x7f83040cdfb0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041512b0 con 0x7f830415a4f0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f8308b7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8304151590 con 0x7f83040b27d0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f82f9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f82f9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55458/0 (socket says 172.21.15.107:55458) 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f82f9d9b640 1 -- 172.21.15.107:0/2373624155 learned_addr learned my addr 172.21.15.107:0/2373624155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.918+0000 7f830259c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 0x7f830415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f8302d9d640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 0x7f83041509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 -- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 msgr2=0x7f830415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 0x7f830415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 -- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 msgr2=0x7f83041509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 0x7f83041509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 -- 172.21.15.107:0/2373624155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f400b770 con 0x7f83040cdfb0 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f830259c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 0x7f830415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f8302d9d640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 0x7f83041509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82f9d9b640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00c4d0 tx=0x7f82ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0074e0 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec007660 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec004a20 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8304151870 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.919+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304150ef0 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.920+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830404e8b0 con 0x7f83040cdfb0 2023-11-24T08:14:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.921+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f82ec004b80 con 0x7f83040cdfb0 2023-11-24T08:14:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.923+0000 7f82fb7fe640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 0x7f82e406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.923+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82ec087720 con 0x7f83040cdfb0 2023-11-24T08:14:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.923+0000 7f830259c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 0x7f82e406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.924+0000 7f830259c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 0x7f82e406e500 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f82f4038000 tx=0x7f82f4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:20.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:20.927+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82ec08f070 con 0x7f83040cdfb0 2023-11-24T08:14:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.166+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7f83040ada50 con 0x7f83040cdfb0 2023-11-24T08:14:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.798+0000 7f82fb7fe640 1 -- 172.21.15.107:0/2373624155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath"]}]=0 v493) v1 ==== 176+0+0 (secure 0 0 0) 0x7f82ec018070 con 0x7f83040cdfb0 2023-11-24T08:14:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 msgr2=0x7f82e406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 0x7f82e406e500 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f82f4038000 tx=0x7f82f4032000 comp rx=0 tx=0).stop 2023-11-24T08:14:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 msgr2=0x7f83041f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00c4d0 tx=0x7f82ec00c990 comp rx=0 tx=0).stop 2023-11-24T08:14:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 shutdown_connections 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82e406c040 0x7f82e406e500 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830415a4f0 0x7f83041509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83040cdfb0 0x7f83041f2100 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 --2- 172.21.15.107:0/2373624155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b27d0 0x7f830415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 >> 172.21.15.107:0/2373624155 conn(0x7f83040b0c70 msgr2=0x7f83040ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 shutdown_connections 2023-11-24T08:14:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:21.802+0000 7f8308b7c640 1 -- 172.21.15.107:0/2373624155 wait complete. 2023-11-24T08:14:21.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:14:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.025+0000 7f1a5e89b640 1 Processor -- start 2023-11-24T08:14:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.025+0000 7f1a5e89b640 1 -- start start 2023-11-24T08:14:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a580ccbd0 0x7f1a581660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a580ccfb0 0x7f1a58162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a58162540 0x7f1a580c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a581655f0 con 0x7f1a58162540 2023-11-24T08:14:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a580cc380 con 0x7f1a580ccfb0 2023-11-24T08:14:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5e89b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a58162b30 con 0x7f1a580ccbd0 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5ce38640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a58162540 0x7f1a580c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5ce38640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a58162540 0x7f1a580c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55462/0 (socket says 172.21.15.107:55462) 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a5ce38640 1 -- 172.21.15.107:0/3100347052 learned_addr learned my addr 172.21.15.107:0/3100347052 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a57fff640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a580ccbd0 0x7f1a581660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.027+0000 7f1a4ffff640 1 --2- 172.21.15.107:0/3100347052 >> v2:172.21.15.120:3300/0 conn(0x7f1a580ccfb0 0x7f1a58162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.028+0000 7f1a5ce38640 1 -- 172.21.15.107:0/3100347052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a580ccbd0 msgr2=0x7f1a581660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.028+0000 7f1a5ce38640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a580ccbd0 0x7f1a581660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.028+0000 7f1a5ce38640 1 -- 172.21.15.107:0/3100347052 >> v2:172.21.15.120:3300/0 conn(0x7f1a580ccfb0 msgr2=0x7f1a58162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.028+0000 7f1a5ce38640 1 --2- 172.21.15.107:0/3100347052 >> v2:172.21.15.120:3300/0 conn(0x7f1a580ccfb0 0x7f1a58162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.028+0000 7f1a5ce38640 1 -- 172.21.15.107:0/3100347052 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58162e10 con 0x7f1a58162540 2023-11-24T08:14:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.029+0000 7f1a5ce38640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a58162540 0x7f1a580c9650 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009e50 tx=0x7f1a5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ee0a0dadf522d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.029+0000 7f1a577fe640 1 -- 172.21.15.107:0/3100347052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5002ca70 con 0x7f1a58162540 2023-11-24T08:14:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.029+0000 7f1a577fe640 1 -- 172.21.15.107:0/3100347052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5002cbd0 con 0x7f1a58162540 2023-11-24T08:14:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.029+0000 7f1a577fe640 1 -- 172.21.15.107:0/3100347052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500343f0 con 0x7f1a58162540 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 -- 172.21.15.107:0/3100347052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a58162540 msgr2=0x7f1a580c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a58162540 0x7f1a580c9650 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009e50 tx=0x7f1a5002bed0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 -- 172.21.15.107:0/3100347052 shutdown_connections 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a58162540 0x7f1a580c9650 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/3100347052 >> v2:172.21.15.120:3300/0 conn(0x7f1a580ccfb0 0x7f1a58162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/3100347052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a580ccbd0 0x7f1a581660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 -- 172.21.15.107:0/3100347052 >> 172.21.15.107:0/3100347052 conn(0x7f1a580b0c50 msgr2=0x7f1a580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 -- 172.21.15.107:0/3100347052 shutdown_connections 2023-11-24T08:14:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.030+0000 7f1a5e89b640 1 -- 172.21.15.107:0/3100347052 wait complete. 2023-11-24T08:14:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.032+0000 7f1a5e89b640 1 Processor -- start 2023-11-24T08:14:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.032+0000 7f1a5e89b640 1 -- start start 2023-11-24T08:14:22.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 0x7f1a580c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a58162540 0x7f1a581645e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580704c0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580c8b60 con 0x7f1a580ccbd0 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a5e89b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a58070710 con 0x7f1a58162540 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55474/0 (socket says 172.21.15.107:55474) 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.033+0000 7f1a4ffff640 1 -- 172.21.15.107:0/1147677194 learned_addr learned my addr 172.21.15.107:0/1147677194 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a57fff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 0x7f1a580c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 -- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a58162540 msgr2=0x7f1a581645e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a58162540 0x7f1a581645e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 -- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 msgr2=0x7f1a580c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 0x7f1a580c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 -- 172.21.15.107:0/1147677194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009b30 con 0x7f1a580ccfb0 2023-11-24T08:14:22.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a57fff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 0x7f1a580c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a4ffff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000d800 tx=0x7f1a4000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000b6c0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4000b820 con 0x7f1a580ccfb0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.034+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580709f0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.035+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000bbb0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.035+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a580c8720 con 0x7f1a580ccfb0 2023-11-24T08:14:22.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.036+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a580c88b0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.037+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f1a4000bd10 con 0x7f1a580ccfb0 2023-11-24T08:14:22.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.037+0000 7f1a557fa640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 0x7f1a3806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.038+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a40088760 con 0x7f1a580ccfb0 2023-11-24T08:14:22.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.038+0000 7f1a57fff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 0x7f1a3806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.039+0000 7f1a57fff640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 0x7f1a3806e500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1a48006fd0 tx=0x7f1a48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.042+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a400552c0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.231+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1a5804e8b0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.237+0000 7f1a557fa640 1 -- 172.21.15.107:0/1147677194 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1a400588e0 con 0x7f1a580ccfb0 2023-11-24T08:14:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.239+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 msgr2=0x7f1a3806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 0x7f1a3806e500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1a48006fd0 tx=0x7f1a48008040 comp rx=0 tx=0).stop 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 msgr2=0x7f1a581640a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000d800 tx=0x7f1a4000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 shutdown_connections 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a3806c040 0x7f1a3806e500 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a58162540 0x7f1a581645e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a580ccfb0 0x7f1a581640a0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 --2- 172.21.15.107:0/1147677194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a580ccbd0 0x7f1a580c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 >> 172.21.15.107:0/1147677194 conn(0x7f1a580b0c50 msgr2=0x7f1a5806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 shutdown_connections 2023-11-24T08:14:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.240+0000 7f1a5e89b640 1 -- 172.21.15.107:0/1147677194 wait complete. 2023-11-24T08:14:22.258 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:14:22.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:14:22.368 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:14:22.489 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:14:22.490 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:22.490 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:22.546 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:14:22.546 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:22.546 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:22.601 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:14:22.602 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:22.602 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:22.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:14:22.662 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:22.662 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:22.722 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:14:22.722 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:14:22.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:22.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.938+0000 7f26d2333640 1 Processor -- start 2023-11-24T08:14:22.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.938+0000 7f26d2333640 1 -- start start 2023-11-24T08:14:22.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26cc159ce0 0x7f26cc06de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0b21d0 0x7f26cc0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 0x7f26cc15d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26cc0cc410 con 0x7f26cc159ce0 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26cc0b3a40 con 0x7f26cc0cbf80 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.939+0000 7f26d2333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0cdfa0 con 0x7f26cc0b21d0 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26cc159ce0 0x7f26cc06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26cc159ce0 0x7f26cc06de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55482/0 (socket says 172.21.15.107:55482) 2023-11-24T08:14:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 -- 172.21.15.107:0/3620452489 learned_addr learned my addr 172.21.15.107:0/3620452489 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26d08d0640 1 --2- 172.21.15.107:0/3620452489 >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 0x7f26cc15d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cb7fe640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0b21d0 0x7f26cc0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 -- 172.21.15.107:0/3620452489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0b21d0 msgr2=0x7f26cc0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0b21d0 0x7f26cc0cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 -- 172.21.15.107:0/3620452489 >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 msgr2=0x7f26cc15d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 --2- 172.21.15.107:0/3620452489 >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 0x7f26cc15d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26cbfff640 1 -- 172.21.15.107:0/3620452489 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc0cd9e0 con 0x7f26cc159ce0 2023-11-24T08:14:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.940+0000 7f26d08d0640 1 --2- 172.21.15.107:0/3620452489 >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 0x7f26cc15d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.941+0000 7f26cbfff640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc159ce0 0x7f26cc06de80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0b5270 tx=0x7f26bc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85ca1f491aaaea8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26caffd640 1 -- 172.21.15.107:0/3620452489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc031860 con 0x7f26cc159ce0 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26caffd640 1 -- 172.21.15.107:0/3620452489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc0319c0 con 0x7f26cc159ce0 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26caffd640 1 -- 172.21.15.107:0/3620452489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc031d50 con 0x7f26cc159ce0 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 -- 172.21.15.107:0/3620452489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc159ce0 msgr2=0x7f26cc06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc159ce0 0x7f26cc06de80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0b5270 tx=0x7f26bc02b900 comp rx=0 tx=0).stop 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 -- 172.21.15.107:0/3620452489 shutdown_connections 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 --2- 172.21.15.107:0/3620452489 >> v2:172.21.15.120:3300/0 conn(0x7f26cc0cbf80 0x7f26cc15d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0b21d0 0x7f26cc0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 --2- 172.21.15.107:0/3620452489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc159ce0 0x7f26cc06de80 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.942+0000 7f26d2333640 1 -- 172.21.15.107:0/3620452489 >> 172.21.15.107:0/3620452489 conn(0x7f26cc06a920 msgr2=0x7f26cc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.943+0000 7f26d2333640 1 -- 172.21.15.107:0/3620452489 shutdown_connections 2023-11-24T08:14:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.943+0000 7f26d2333640 1 -- 172.21.15.107:0/3620452489 wait complete. 2023-11-24T08:14:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.944+0000 7f26d2333640 1 Processor -- start 2023-11-24T08:14:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.944+0000 7f26d2333640 1 -- start start 2023-11-24T08:14:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0cbf80 0x7f26cc14a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26cc159ce0 0x7f26cc0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc150300 con 0x7f26cc0b21d0 2023-11-24T08:14:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc14f2a0 con 0x7f26cc159ce0 2023-11-24T08:14:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26d2333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc14f540 con 0x7f26cc0cbf80 2023-11-24T08:14:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.945+0000 7f26cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55498/0 (socket says 172.21.15.107:55498) 2023-11-24T08:14:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 -- 172.21.15.107:0/899515629 learned_addr learned my addr 172.21.15.107:0/899515629 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 -- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0cbf80 msgr2=0x7f26cc14a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cb7fe640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0cbf80 0x7f26cc14a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26d08d0640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26cc159ce0 0x7f26cc0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0cbf80 0x7f26cc14a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 -- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26cc159ce0 msgr2=0x7f26cc0c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26cc159ce0 0x7f26cc0c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:22.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 -- 172.21.15.107:0/899515629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc0094d0 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.946+0000 7f26cbfff640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f26bc000c00 tx=0x7f26bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.947+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc031830 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.947+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc0443f0 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.947+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc0447a0 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.947+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26cc14f820 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.947+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc14fde0 con 0x7f26cc0b21d0 2023-11-24T08:14:22.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.948+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc067470 con 0x7f26cc0b21d0 2023-11-24T08:14:22.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.949+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f26bc04d450 con 0x7f26cc0b21d0 2023-11-24T08:14:22.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.950+0000 7f26c8ff9640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 0x7f26b806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:22.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.950+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26bc0aac90 con 0x7f26cc0b21d0 2023-11-24T08:14:22.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.950+0000 7f26cb7fe640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 0x7f26b806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:22.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.951+0000 7f26cb7fe640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 0x7f26b806e530 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006fd0 tx=0x7f26c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:22.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:22.954+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26bc0773f0 con 0x7f26cc0b21d0 2023-11-24T08:14:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.181+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26cc0ce180 con 0x7f26cc0b21d0 2023-11-24T08:14:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.182+0000 7f26c8ff9640 1 -- 172.21.15.107:0/899515629 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f26bc035030 con 0x7f26cc0b21d0 2023-11-24T08:14:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:23.184 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5669,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3208643552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3208643552},{"type":"v1","addr":"172.21.15.120:6835","nonce":3208643552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6127,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2869970138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2869970138},{"type":"v1","addr":"172.21.15.107:6837","nonce":2869970138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2117970131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2117970131},{"type":"v1","addr":"172.21.15.107:6835","nonce":2117970131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8301,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1594235330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1594235330},{"type":"v1","addr":"172.21.15.120:6834","nonce":1594235330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[]} 2023-11-24T08:14:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 72 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.184+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 msgr2=0x7f26b806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.184+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 0x7f26b806e530 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006fd0 tx=0x7f26c0008040 comp rx=0 tx=0).stop 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.184+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 msgr2=0x7f26cc1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.184+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f26bc000c00 tx=0x7f26bc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 shutdown_connections 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f26b806c070 0x7f26b806e530 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26cc159ce0 0x7f26cc0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26cc0cbf80 0x7f26cc14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 --2- 172.21.15.107:0/899515629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26cc0b21d0 0x7f26cc1511a0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 >> 172.21.15.107:0/899515629 conn(0x7f26cc06a920 msgr2=0x7f26cc06c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:23.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 shutdown_connections 2023-11-24T08:14:23.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.185+0000 7f26d2333640 1 -- 172.21.15.107:0/899515629 wait complete. 2023-11-24T08:14:23.206 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:14:23.206 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:14:23.206 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:14:23.206 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:14:23.207 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:14:23.207 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:14:23.207 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:14:23.207 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:14:23.207 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:14:23.207 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:14:23.207 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:14:23.207 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:14:23.208 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:14:23.208 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:14:23.208 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:14:23.208 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:14:23.227 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:14:23.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:14:23.232 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:14:23.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:14:23.233 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:14:23.233 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:14:23.234 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:14:23.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:14:23.280 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:14:23.280 DEBUG:teuthology.parallel:result is None 2023-11-24T08:14:23.283 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:14:23.283 DEBUG:teuthology.parallel:result is None 2023-11-24T08:14:23.288 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:14:23.288 DEBUG:teuthology.parallel:result is None 2023-11-24T08:14:23.290 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:14:23.291 DEBUG:teuthology.parallel:result is None 2023-11-24T08:14:23.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:14:23.436 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:14:23.440 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:14:23.450 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:14:23.452 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:14:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.474+0000 7f0cac49b640 1 Processor -- start 2023-11-24T08:14:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.474+0000 7f0cac49b640 1 -- start start 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0cac49b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ca4072160 0x7f0ca415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0cac49b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0cac49b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca415ec90 con 0x7f0ca406dc40 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0cac49b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca406e0f0 con 0x7f0ca4072160 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0cac49b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40cd530 con 0x7f0ca40c49f0 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caa237640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ca4072160 0x7f0ca415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0ca9a36640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caaa38640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0ca406dc40 0x7f0ca40cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45756/0 (socket says 172.21.15.107:45756) 2023-11-24T08:14:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caaa38640 1 -- 172.21.15.107:0/2457288522 learned_addr learned my addr 172.21.15.107:0/2457288522 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caa237640 1 -- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 msgr2=0x7f0ca406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caa237640 1 --2- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caa237640 1 -- 172.21.15.107:0/2457288522 >> v1:172.21.15.107:6789/0 conn(0x7f0ca406dc40 legacy=0x7f0ca40cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.476+0000 7f0caa237640 1 -- 172.21.15.107:0/2457288522 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40c4ed0 con 0x7f0ca4072160 2023-11-24T08:14:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.478+0000 7f0caa237640 1 --2- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca415e750 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097a0 tx=0x7f0ca002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dc562519c60cf15 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.478+0000 7f0ca9235640 1 -- 172.21.15.107:0/2457288522 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca002d040 con 0x7f0ca4072160 2023-11-24T08:14:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.478+0000 7f0ca9235640 1 -- 172.21.15.107:0/2457288522 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0031810 con 0x7f0ca4072160 2023-11-24T08:14:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.478+0000 7f0ca9235640 1 -- 172.21.15.107:0/2457288522 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0031ba0 con 0x7f0ca4072160 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 -- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 msgr2=0x7f0ca415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca415e750 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097a0 tx=0x7f0ca002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 -- 172.21.15.107:0/2457288522 shutdown_connections 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2457288522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca415e750 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 -- 172.21.15.107:0/2457288522 >> 172.21.15.107:0/2457288522 conn(0x7f0ca40b0c50 msgr2=0x7f0ca40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 -- 172.21.15.107:0/2457288522 shutdown_connections 2023-11-24T08:14:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.479+0000 7f0cac49b640 1 -- 172.21.15.107:0/2457288522 wait complete. 2023-11-24T08:14:23.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.481+0000 7f0cac49b640 1 Processor -- start 2023-11-24T08:14:23.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.481+0000 7f0cac49b640 1 -- start start 2023-11-24T08:14:23.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca40cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca409ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40cb380 con 0x7f0ca406dc40 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4099db0 con 0x7f0ca4072160 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0cac49b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca409a090 con 0x7f0ca40c49f0 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55538/0 (socket says 172.21.15.107:55538) 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 -- 172.21.15.107:0/2416243439 learned_addr learned my addr 172.21.15.107:0/2416243439 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0ca9a36640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca40cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caaa38640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca409ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 -- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 msgr2=0x7f0ca409ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca409ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 -- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 msgr2=0x7f0ca40cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca40cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.482+0000 7f0caa237640 1 -- 172.21.15.107:0/2416243439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009400 con 0x7f0ca406dc40 2023-11-24T08:14:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0caa237640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00098c0 tx=0x7f0ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0006e10 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca003ad60 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0044630 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca409a370 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.483+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca409a930 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.484+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca404e8b0 con 0x7f0ca406dc40 2023-11-24T08:14:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.485+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f0ca004d450 con 0x7f0ca406dc40 2023-11-24T08:14:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.486+0000 7f0c9affd640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 0x7f0c8806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.486+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0ca00aa6f0 con 0x7f0ca406dc40 2023-11-24T08:14:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.486+0000 7f0ca9a36640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 0x7f0c8806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.487+0000 7f0ca9a36640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 0x7f0c8806e500 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0ca415cc50 tx=0x7f0c94009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.491+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca0076f30 con 0x7f0ca406dc40 2023-11-24T08:14:23.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.694+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0ca409ab20 con 0x7f0ca406dc40 2023-11-24T08:14:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.824+0000 7f0c9affd640 1 -- 172.21.15.107:0/2416243439 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v125) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0ca007a550 con 0x7f0ca406dc40 2023-11-24T08:14:23.827 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.828+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 msgr2=0x7f0c8806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.828+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 0x7f0c8806e500 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0ca415cc50 tx=0x7f0c94009190 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.828+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 msgr2=0x7f0ca414dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.828+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00098c0 tx=0x7f0ca00046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 shutdown_connections 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8806c040 0x7f0c8806e500 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca40c49f0 0x7f0ca409ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca4072160 0x7f0ca40cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 --2- 172.21.15.107:0/2416243439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca406dc40 0x7f0ca414dd20 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 >> 172.21.15.107:0/2416243439 conn(0x7f0ca40b0c50 msgr2=0x7f0ca40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 shutdown_connections 2023-11-24T08:14:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:23.829+0000 7f0cac49b640 1 -- 172.21.15.107:0/2416243439 wait complete. 2023-11-24T08:14:23.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:14:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.067+0000 7feb8acbb640 1 Processor -- start 2023-11-24T08:14:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.067+0000 7feb8acbb640 1 -- start start 2023-11-24T08:14:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.069+0000 7feb8acbb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb8413e0e0 0x7feb8406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.069+0000 7feb8acbb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb84070a80 0x7feb8413e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.069+0000 7feb8acbb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8415b7a0 con 0x7feb8406cee0 2023-11-24T08:14:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.069+0000 7feb8acbb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8406d390 con 0x7feb84070a80 2023-11-24T08:14:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.069+0000 7feb8acbb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8415ba60 con 0x7feb8413e0e0 2023-11-24T08:14:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb894b8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feb8406cee0 0x7feb84070370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45758/0 (socket says 172.21.15.107:45758) 2023-11-24T08:14:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb894b8640 1 -- 172.21.15.107:0/2889489712 learned_addr learned my addr 172.21.15.107:0/2889489712 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/2889489712 >> v2:172.21.15.107:3300/0 conn(0x7feb84070a80 0x7feb8413e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb89cb9640 1 --2- 172.21.15.107:0/2889489712 >> v2:172.21.15.120:3300/0 conn(0x7feb8413e0e0 0x7feb8406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb88cb7640 1 -- 172.21.15.107:0/2889489712 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410894887 0 0) 0x7feb8415b7a0 con 0x7feb8406cee0 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb88cb7640 1 -- 172.21.15.107:0/2889489712 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003520 con 0x7feb8406cee0 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb8a4ba640 1 -- 172.21.15.107:0/2889489712 >> v2:172.21.15.120:3300/0 conn(0x7feb8413e0e0 msgr2=0x7feb8406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/2889489712 >> v2:172.21.15.120:3300/0 conn(0x7feb8413e0e0 0x7feb8406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb8a4ba640 1 -- 172.21.15.107:0/2889489712 >> v1:172.21.15.107:6789/0 conn(0x7feb8406cee0 legacy=0x7feb84070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.070+0000 7feb8a4ba640 1 -- 172.21.15.107:0/2889489712 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8415bd40 con 0x7feb84070a80 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.071+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/2889489712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb84070a80 0x7feb8413e810 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7feb8406e520 tx=0x7feb7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5678ffa5e9cd3748 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb88cb7640 1 -- 172.21.15.107:0/2889489712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c02cc80 con 0x7feb84070a80 2023-11-24T08:14:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb88cb7640 1 -- 172.21.15.107:0/2889489712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c02cde0 con 0x7feb84070a80 2023-11-24T08:14:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb88cb7640 1 -- 172.21.15.107:0/2889489712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c034660 con 0x7feb84070a80 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 -- 172.21.15.107:0/2889489712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb84070a80 msgr2=0x7feb8413e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 --2- 172.21.15.107:0/2889489712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb84070a80 0x7feb8413e810 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7feb8406e520 tx=0x7feb7c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 -- 172.21.15.107:0/2889489712 shutdown_connections 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 --2- 172.21.15.107:0/2889489712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb84070a80 0x7feb8413e810 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 --2- 172.21.15.107:0/2889489712 >> v2:172.21.15.120:3300/0 conn(0x7feb8413e0e0 0x7feb8406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 -- 172.21.15.107:0/2889489712 >> 172.21.15.107:0/2889489712 conn(0x7feb840b4630 msgr2=0x7feb840b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 -- 172.21.15.107:0/2889489712 shutdown_connections 2023-11-24T08:14:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.072+0000 7feb8acbb640 1 -- 172.21.15.107:0/2889489712 wait complete. 2023-11-24T08:14:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.074+0000 7feb8acbb640 1 Processor -- start 2023-11-24T08:14:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.074+0000 7feb8acbb640 1 -- start start 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb8406cee0 0x7feb84154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb84070a80 0x7feb8413a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb841f3ba0 con 0x7feb8413e0e0 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840b23e0 con 0x7feb84070a80 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.075+0000 7feb8acbb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb840b26c0 con 0x7feb8406cee0 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55550/0 (socket says 172.21.15.107:55550) 2023-11-24T08:14:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 -- 172.21.15.107:0/3324686525 learned_addr learned my addr 172.21.15.107:0/3324686525 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 -- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb8406cee0 msgr2=0x7feb84154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb894b8640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb84070a80 0x7feb8413a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb8406cee0 0x7feb84154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 -- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb84070a80 msgr2=0x7feb8413a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb84070a80 0x7feb8413a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 -- 172.21.15.107:0/3324686525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c009a70 con 0x7feb8413e0e0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.076+0000 7feb8a4ba640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7feb7c02c4e0 tx=0x7feb7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.077+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c03e610 con 0x7feb8413e0e0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.077+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c03e770 con 0x7feb8413e0e0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.077+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c03eb60 con 0x7feb8413e0e0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.077+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb840b29a0 con 0x7feb8413e0e0 2023-11-24T08:14:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.077+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb840b2f60 con 0x7feb8413e0e0 2023-11-24T08:14:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.078+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb840b3180 con 0x7feb8413e0e0 2023-11-24T08:14:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.079+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7feb7c03b050 con 0x7feb8413e0e0 2023-11-24T08:14:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.080+0000 7feb7a7fc640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 0x7feb6806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.080+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb7c0aa3b0 con 0x7feb8413e0e0 2023-11-24T08:14:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.080+0000 7feb89cb9640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 0x7feb6806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.081+0000 7feb89cb9640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 0x7feb6806e500 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7feb80006fd0 tx=0x7feb80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.084+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb7c076eb0 con 0x7feb8413e0e0 2023-11-24T08:14:24.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.338+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7feb8404e840 con 0x7feb8413e0e0 2023-11-24T08:14:24.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.339+0000 7feb7a7fc640 1 -- 172.21.15.107:0/3324686525 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7feb7c04d090 con 0x7feb8413e0e0 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.343 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:14:24.344 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:14:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:14:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.354 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:14:24.355 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:14:24.356 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:24.357 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:14:24.358 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:14:24.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.341+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 msgr2=0x7feb6806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.341+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 0x7feb6806e500 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7feb80006fd0 tx=0x7feb80008040 comp rx=0 tx=0).stop 2023-11-24T08:14:24.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.341+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 msgr2=0x7feb8415a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.341+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7feb7c02c4e0 tx=0x7feb7c004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 shutdown_connections 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb6806c040 0x7feb6806e500 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb8413e0e0 0x7feb8415a270 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb84070a80 0x7feb8413a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 --2- 172.21.15.107:0/3324686525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb8406cee0 0x7feb84154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 >> 172.21.15.107:0/3324686525 conn(0x7feb840b4630 msgr2=0x7feb8413f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 shutdown_connections 2023-11-24T08:14:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.342+0000 7feb8acbb640 1 -- 172.21.15.107:0/3324686525 wait complete. 2023-11-24T08:14:24.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.569+0000 7fec6b31d640 1 Processor -- start 2023-11-24T08:14:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.570+0000 7fec6b31d640 1 -- start start 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec6b31d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 0x7fec640ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec6b31d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec640cd470 0x7fec6415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec6b31d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6415ab80 con 0x7fec640b2250 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec6b31d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6415a990 con 0x7fec640cd470 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec6b31d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6415e3c0 con 0x7fec6415df80 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec690b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fec640b2250 0x7fec640b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45766/0 (socket says 172.21.15.107:45766) 2023-11-24T08:14:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.571+0000 7fec690b9640 1 -- 172.21.15.107:0/1059882366 learned_addr learned my addr 172.21.15.107:0/1059882366 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec698ba640 1 --2- 172.21.15.107:0/1059882366 >> v2:172.21.15.120:3300/0 conn(0x7fec640cd470 0x7fec6415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec688b8640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 0x7fec640ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34062760 0 0) 0x7fec6415ab80 con 0x7fec640b2250 2023-11-24T08:14:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50003520 con 0x7fec640b2250 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 697916770 0 0) 0x7fec50003520 con 0x7fec640b2250 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 msgr2=0x7fec640ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 0x7fec640ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 >> v2:172.21.15.120:3300/0 conn(0x7fec640cd470 msgr2=0x7fec6415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 --2- 172.21.15.107:0/1059882366 >> v2:172.21.15.120:3300/0 conn(0x7fec640cd470 0x7fec6415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640b2d40 con 0x7fec640b2250 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fec60001670 con 0x7fec640b2250 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 >> v1:172.21.15.107:6789/0 conn(0x7fec640b2250 legacy=0x7fec640b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003970 0x7fec50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 0x7fec500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 0x7fec50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6415e3c0 con 0x7fec50003970 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5000db90 con 0x7fec50004d00 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.572+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5000dcd0 con 0x7fec6415df80 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003970 0x7fec50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 -- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 msgr2=0x7fec500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec6415df80 0x7fec500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 -- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 msgr2=0x7fec50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50004d00 0x7fec50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.573+0000 7fec690b9640 1 -- 172.21.15.107:0/1059882366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50003520 con 0x7fec50003970 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.574+0000 7fec690b9640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003970 0x7fec50003d90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c110 tx=0x7fec500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7031acfd59e3751b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.574+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60005ad0 con 0x7fec50003970 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.574+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec60005c50 con 0x7fec50003970 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.574+0000 7fec5bfff640 1 -- 172.21.15.107:0/1059882366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6002ca70 con 0x7fec50003970 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 -- 172.21.15.107:0/1059882366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003970 msgr2=0x7fec50003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 --2- 172.21.15.107:0/1059882366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50003970 0x7fec50003d90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c110 tx=0x7fec500057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec698ba640 1 -- 172.21.15.107:0/1059882366 reap_dead start 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 -- 172.21.15.107:0/1059882366 shutdown_connections 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 -- 172.21.15.107:0/1059882366 >> 172.21.15.107:0/1059882366 conn(0x7fec6406a920 msgr2=0x7fec6406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 -- 172.21.15.107:0/1059882366 shutdown_connections 2023-11-24T08:14:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.575+0000 7fec6b31d640 1 -- 172.21.15.107:0/1059882366 wait complete. 2023-11-24T08:14:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.577+0000 7fec6b31d640 1 Processor -- start 2023-11-24T08:14:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.577+0000 7fec6b31d640 1 -- start start 2023-11-24T08:14:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50003970 0x7fec64152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec64147220 0x7fec64154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640aed70 con 0x7fec641476a0 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec641532f0 con 0x7fec50003970 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec6b31d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec64155790 con 0x7fec64147220 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec698ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec698ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55574/0 (socket says 172.21.15.107:55574) 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.578+0000 7fec698ba640 1 -- 172.21.15.107:0/3924300757 learned_addr learned my addr 172.21.15.107:0/3924300757 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 -- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec64147220 msgr2=0x7fec64154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec64147220 0x7fec64154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 -- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50003970 msgr2=0x7fec64152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec690b9640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50003970 0x7fec64152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50003970 0x7fec64152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 -- 172.21.15.107:0/3924300757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec64155a70 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec698ba640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fec640b2fa0 tx=0x7fec5c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c00cb80 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c00f040 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec64155d50 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c014480 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.579+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec64152a40 con 0x7fec641476a0 2023-11-24T08:14:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.581+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec640aeeb0 con 0x7fec641476a0 2023-11-24T08:14:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.582+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fec5c0145e0 con 0x7fec641476a0 2023-11-24T08:14:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.583+0000 7fec59ffb640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 0x7fec4806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.583+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec5c088b20 con 0x7fec641476a0 2023-11-24T08:14:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.583+0000 7fec690b9640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 0x7fec4806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.584+0000 7fec690b9640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 0x7fec4806e500 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec60034000 tx=0x7fec60007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.587+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec640aeeb0 con 0x7fec641476a0 2023-11-24T08:14:24.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.781+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec640aeeb0 con 0x7fec641476a0 2023-11-24T08:14:24.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.783+0000 7fec59ffb640 1 -- 172.21.15.107:0/3924300757 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fec640aeeb0 con 0x7fec641476a0 2023-11-24T08:14:24.785 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:24.785 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":127,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:14:23.840367+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 msgr2=0x7fec4806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 0x7fec4806e500 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec60034000 tx=0x7fec60007000 comp rx=0 tx=0).stop 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 msgr2=0x7fec64154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fec640b2fa0 tx=0x7fec5c00ef20 comp rx=0 tx=0).stop 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 shutdown_connections 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec4806c040 0x7fec4806e500 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec641476a0 0x7fec64154cc0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec64147220 0x7fec64154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 --2- 172.21.15.107:0/3924300757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec50003970 0x7fec64152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 >> 172.21.15.107:0/3924300757 conn(0x7fec6406a920 msgr2=0x7fec640705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 shutdown_connections 2023-11-24T08:14:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:24.786+0000 7fec6b31d640 1 -- 172.21.15.107:0/3924300757 wait complete. 2023-11-24T08:14:24.808 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:14:24.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:14:25.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.020+0000 7fd24b983640 1 Processor -- start 2023-11-24T08:14:25.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.020+0000 7fd24b983640 1 -- start start 2023-11-24T08:14:25.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24b983640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2440c67c0 0x7fd244161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24b983640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24415de00 con 0x7fd244140520 2023-11-24T08:14:25.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24b983640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24415e0e0 con 0x7fd244161940 2023-11-24T08:14:25.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24b983640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24415e3c0 con 0x7fd2440c67c0 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24971f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd244140520 0x7fd2440c60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45782/0 (socket says 172.21.15.107:45782) 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd24971f640 1 -- 172.21.15.107:0/1131887471 learned_addr learned my addr 172.21.15.107:0/1131887471 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.022+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1131887471 >> v2:172.21.15.120:3300/0 conn(0x7fd2440c67c0 0x7fd244161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410394109 0 0) 0x7fd24415de00 con 0x7fd244140520 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230003520 con 0x7fd244140520 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735594197 0 0) 0x7fd230003520 con 0x7fd244140520 2023-11-24T08:14:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 >> v2:172.21.15.120:3300/0 conn(0x7fd2440c67c0 msgr2=0x7fd244161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 --2- 172.21.15.107:0/1131887471 >> v2:172.21.15.120:3300/0 conn(0x7fd2440c67c0 0x7fd244161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 >> v1:172.21.15.120:6789/0 conn(0x7fd244161940 legacy=0x7fd24406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24415e6a0 con 0x7fd244140520 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd2400019f0 con 0x7fd244140520 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 >> v1:172.21.15.107:6789/0 conn(0x7fd244140520 legacy=0x7fd2440c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd230003970 0x7fd230003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 0x7fd230004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd230005180 0x7fd23000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415e3c0 con 0x7fd2300047c0 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000c020 con 0x7fd230005180 2023-11-24T08:14:25.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000c1e0 con 0x7fd230003970 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.023+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 0x7fd230004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd248f1e640 1 -- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd230003970 msgr2=0x7fd230003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd24971f640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd230003970 0x7fd230003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd230003970 0x7fd230003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd249f20640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd230005180 0x7fd23000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd248f1e640 1 -- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd230005180 msgr2=0x7fd23000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd230005180 0x7fd23000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd248f1e640 1 -- 172.21.15.107:0/1131887471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230003520 con 0x7fd2300047c0 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd249f20640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd230005180 0x7fd23000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.024+0000 7fd249f20640 1 -- 172.21.15.107:0/1131887471 reap_dead start 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.025+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 0x7fd230004c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fd234009ac0 tx=0x7fd2300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b532431418dd66d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.025+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23402c6b0 con 0x7fd2300047c0 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.025+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23402c810 con 0x7fd2300047c0 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.025+0000 7fd23bfff640 1 -- 172.21.15.107:0/1131887471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23402cba0 con 0x7fd2300047c0 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 -- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 msgr2=0x7fd230004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 0x7fd230004c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fd234009ac0 tx=0x7fd2300057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 -- 172.21.15.107:0/1131887471 shutdown_connections 2023-11-24T08:14:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 --2- 172.21.15.107:0/1131887471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2300047c0 0x7fd230004c40 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 -- 172.21.15.107:0/1131887471 >> 172.21.15.107:0/1131887471 conn(0x7fd2440b2e40 msgr2=0x7fd2440b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 -- 172.21.15.107:0/1131887471 shutdown_connections 2023-11-24T08:14:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.026+0000 7fd24b983640 1 -- 172.21.15.107:0/1131887471 wait complete. 2023-11-24T08:14:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.028+0000 7fd24b983640 1 Processor -- start 2023-11-24T08:14:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.028+0000 7fd24b983640 1 -- start start 2023-11-24T08:14:25.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2300047c0 0x7fd24413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 0x7fd244159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24413c700 con 0x7fd24406c490 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24406c9f0 con 0x7fd24406c010 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24b983640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24406ccd0 con 0x7fd2300047c0 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd249f20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd249f20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55586/0 (socket says 172.21.15.107:55586) 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd249f20640 1 -- 172.21.15.107:0/1695467082 learned_addr learned my addr 172.21.15.107:0/1695467082 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 0x7fd244159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.029+0000 7fd24971f640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2300047c0 0x7fd24413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 -- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2300047c0 msgr2=0x7fd24413c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2300047c0 0x7fd24413c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 -- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 msgr2=0x7fd244159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 0x7fd244159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 -- 172.21.15.107:0/1695467082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24406cfb0 con 0x7fd24406c490 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd248f1e640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 0x7fd244159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd249f20640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fd2440b7380 tx=0x7fd23c00d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00b070 con 0x7fd24406c490 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c0052f0 con 0x7fd24406c490 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c003830 con 0x7fd24406c490 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24406d290 con 0x7fd24406c490 2023-11-24T08:14:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.030+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2440cfc00 con 0x7fd24406c490 2023-11-24T08:14:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.031+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24404e8b0 con 0x7fd24406c490 2023-11-24T08:14:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.032+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fd23c003990 con 0x7fd24406c490 2023-11-24T08:14:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.034+0000 7fd239ffb640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 0x7fd22806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.034+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd23c087a40 con 0x7fd24406c490 2023-11-24T08:14:25.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.034+0000 7fd24971f640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 0x7fd22806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.034+0000 7fd24971f640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 0x7fd22806e530 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd240007c40 tx=0x7fd240009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.038+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd23c0541d0 con 0x7fd24406c490 2023-11-24T08:14:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.229+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd24413cd60 con 0x7fd24406c490 2023-11-24T08:14:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.229+0000 7fd239ffb640 1 -- 172.21.15.107:0/1695467082 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd23c0180a0 con 0x7fd24406c490 2023-11-24T08:14:25.231 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:14:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 msgr2=0x7fd22806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 0x7fd22806e530 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd240007c40 tx=0x7fd240009040 comp rx=0 tx=0).stop 2023-11-24T08:14:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 msgr2=0x7fd24415a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fd2440b7380 tx=0x7fd23c00d5e0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 shutdown_connections 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd22806c070 0x7fd22806e530 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd24406c490 0x7fd24415a000 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd24406c010 0x7fd244159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 --2- 172.21.15.107:0/1695467082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2300047c0 0x7fd24413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.232+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 >> 172.21.15.107:0/1695467082 conn(0x7fd2440b2e40 msgr2=0x7fd2440cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.233+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 shutdown_connections 2023-11-24T08:14:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.233+0000 7fd24b983640 1 -- 172.21.15.107:0/1695467082 wait complete. 2023-11-24T08:14:25.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:14:25.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.465+0000 7f51a0d33640 1 Processor -- start 2023-11-24T08:14:25.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.466+0000 7f51a0d33640 1 -- start start 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f51a0d33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f519c15c7f0 0x7f519c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f51a0d33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c0cece0 con 0x7f519c0ce6c0 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f51a0d33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c06d040 con 0x7f519c0cf450 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f51a0d33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519c0ceaf0 con 0x7f519c15c7f0 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f519a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f519c15c7f0 0x7f519c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f519ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f519c0ce6c0 0x7f519c0714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45796/0 (socket says 172.21.15.107:45796) 2023-11-24T08:14:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.467+0000 7f519ad9d640 1 -- 172.21.15.107:0/1409145406 learned_addr learned my addr 172.21.15.107:0/1409145406 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:25.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.468+0000 7f519a59c640 1 -- 172.21.15.107:0/1409145406 >> v1:172.21.15.120:6789/0 conn(0x7f519c0cf450 legacy=0x7f519c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.468+0000 7f519a59c640 1 -- 172.21.15.107:0/1409145406 >> v1:172.21.15.107:6789/0 conn(0x7f519c0ce6c0 legacy=0x7f519c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.468+0000 7f5199d9b640 1 -- 172.21.15.107:0/1409145406 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546453160 0 0) 0x7f519c0cece0 con 0x7f519c0ce6c0 2023-11-24T08:14:25.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.468+0000 7f519a59c640 1 -- 172.21.15.107:0/1409145406 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c06d320 con 0x7f519c15c7f0 2023-11-24T08:14:25.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f519a59c640 1 --2- 172.21.15.107:0/1409145406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c06db20 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f518c0097a0 tx=0x7f518c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0350c6514b00621 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f5199d9b640 1 -- 172.21.15.107:0/1409145406 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c031830 con 0x7f519c15c7f0 2023-11-24T08:14:25.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f5199d9b640 1 -- 172.21.15.107:0/1409145406 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c031990 con 0x7f519c15c7f0 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f5199d9b640 1 -- 172.21.15.107:0/1409145406 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c031d20 con 0x7f519c15c7f0 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f51a0d33640 1 -- 172.21.15.107:0/1409145406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 msgr2=0x7f519c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.469+0000 7f51a0d33640 1 --2- 172.21.15.107:0/1409145406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c06db20 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f518c0097a0 tx=0x7f518c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.470+0000 7f51a0d33640 1 -- 172.21.15.107:0/1409145406 shutdown_connections 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.470+0000 7f51a0d33640 1 --2- 172.21.15.107:0/1409145406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c06db20 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.470+0000 7f51a0d33640 1 -- 172.21.15.107:0/1409145406 >> 172.21.15.107:0/1409145406 conn(0x7f519c06a920 msgr2=0x7f519c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.470+0000 7f51a0d33640 1 -- 172.21.15.107:0/1409145406 shutdown_connections 2023-11-24T08:14:25.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.470+0000 7f51a0d33640 1 -- 172.21.15.107:0/1409145406 wait complete. 2023-11-24T08:14:25.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.471+0000 7f51a0d33640 1 Processor -- start 2023-11-24T08:14:25.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 -- start start 2023-11-24T08:14:25.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce6c0 0x7f519c157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 0x7f519c14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c156f00 con 0x7f519c15c7f0 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c157860 con 0x7f519c0cf450 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.472+0000 7f51a0d33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c156280 con 0x7f519c0ce6c0 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55602/0 (socket says 172.21.15.107:55602) 2023-11-24T08:14:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 -- 172.21.15.107:0/2070663202 learned_addr learned my addr 172.21.15.107:0/2070663202 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 -- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce6c0 msgr2=0x7f519c157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce6c0 0x7f519c157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 -- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 msgr2=0x7f519c14d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f5191d9b640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 0x7f519c14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 0x7f519c14d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 -- 172.21.15.107:0/2070663202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c009400 con 0x7f519c15c7f0 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f5191d9b640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 0x7f519c14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.473+0000 7f519ad9d640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5194002230 tx=0x7f519400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.474+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400ce30 con 0x7f519c15c7f0 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.474+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519400a640 con 0x7f519c15c7f0 2023-11-24T08:14:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.474+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400a9f0 con 0x7f519c15c7f0 2023-11-24T08:14:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.474+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c156560 con 0x7f519c15c7f0 2023-11-24T08:14:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.474+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c156b20 con 0x7f519c15c7f0 2023-11-24T08:14:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.475+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c0cdf30 con 0x7f519c15c7f0 2023-11-24T08:14:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.476+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f519400ab50 con 0x7f519c15c7f0 2023-11-24T08:14:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.477+0000 7f51937fe640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 0x7f518006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.477+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5194087ae0 con 0x7f519c15c7f0 2023-11-24T08:14:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.477+0000 7f519a59c640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 0x7f518006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.478+0000 7f519a59c640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 0x7f518006e500 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f518c036000 tx=0x7f518c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.482+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5194054240 con 0x7f519c15c7f0 2023-11-24T08:14:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.678+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f519c157040 con 0x7f519c15c7f0 2023-11-24T08:14:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.679+0000 7f51937fe640 1 -- 172.21.15.107:0/2070663202 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v127) v1 ==== 84+0+65 (secure 0 0 0) 0x7f519c157040 con 0x7f519c15c7f0 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:25.681 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.681+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 msgr2=0x7f518006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.681+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 0x7f518006e500 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f518c036000 tx=0x7f518c030000 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.681+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 msgr2=0x7f519c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.681+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5194002230 tx=0x7f519400cbe0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 shutdown_connections 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518006c040 0x7f518006e500 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f519c15c7f0 0x7f519c0b3420 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0cf450 0x7f519c14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 --2- 172.21.15.107:0/2070663202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce6c0 0x7f519c157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 >> 172.21.15.107:0/2070663202 conn(0x7f519c06a920 msgr2=0x7f519c15d450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 shutdown_connections 2023-11-24T08:14:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.682+0000 7f51a0d33640 1 -- 172.21.15.107:0/2070663202 wait complete. 2023-11-24T08:14:25.703 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:14:25.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:14:25.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.907+0000 7fb099e32640 1 Processor -- start 2023-11-24T08:14:25.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.908+0000 7fb099e32640 1 -- start start 2023-11-24T08:14:25.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.909+0000 7fb099e32640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 0x7fb0940d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.909+0000 7fb099e32640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0940d12e0 0x7fb0940cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.909+0000 7fb099e32640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09415ea30 con 0x7fb0940d0120 2023-11-24T08:14:25.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.909+0000 7fb099e32640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940d05b0 con 0x7fb0940d12e0 2023-11-24T08:14:25.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.909+0000 7fb099e32640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415de20 con 0x7fb094160b00 2023-11-24T08:14:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.910+0000 7fb098e30640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 0x7fb0940d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.910+0000 7fb099631640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb0940d0120 0x7fb09415ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:47486/0 (socket says 172.21.15.107:47486) 2023-11-24T08:14:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.910+0000 7fb099631640 1 -- 172.21.15.107:0/1606584251 learned_addr learned my addr 172.21.15.107:0/1606584251 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.910+0000 7fb093fff640 1 --2- 172.21.15.107:0/1606584251 >> v2:172.21.15.120:3300/0 conn(0x7fb0940d12e0 0x7fb0940cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.911+0000 7fb098e30640 1 -- 172.21.15.107:0/1606584251 >> v2:172.21.15.120:3300/0 conn(0x7fb0940d12e0 msgr2=0x7fb0940cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.911+0000 7fb098e30640 1 --2- 172.21.15.107:0/1606584251 >> v2:172.21.15.120:3300/0 conn(0x7fb0940d12e0 0x7fb0940cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.911+0000 7fb098e30640 1 -- 172.21.15.107:0/1606584251 >> v1:172.21.15.120:6789/0 conn(0x7fb0940d0120 legacy=0x7fb09415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.911+0000 7fb098e30640 1 -- 172.21.15.107:0/1606584251 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09415e100 con 0x7fb094160b00 2023-11-24T08:14:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.912+0000 7fb098e30640 1 --2- 172.21.15.107:0/1606584251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 0x7fb0940d0da0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb084009d90 tx=0x7fb08402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9feddaa9f1a8a2bb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.912+0000 7fb0937fe640 1 -- 172.21.15.107:0/1606584251 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08402cc50 con 0x7fb094160b00 2023-11-24T08:14:25.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb0937fe640 1 -- 172.21.15.107:0/1606584251 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08402cdb0 con 0x7fb094160b00 2023-11-24T08:14:25.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb0937fe640 1 -- 172.21.15.107:0/1606584251 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084034660 con 0x7fb094160b00 2023-11-24T08:14:25.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 -- 172.21.15.107:0/1606584251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 msgr2=0x7fb0940d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 --2- 172.21.15.107:0/1606584251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 0x7fb0940d0da0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb084009d90 tx=0x7fb08402bea0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 -- 172.21.15.107:0/1606584251 shutdown_connections 2023-11-24T08:14:25.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 --2- 172.21.15.107:0/1606584251 >> v2:172.21.15.120:3300/0 conn(0x7fb0940d12e0 0x7fb0940cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 --2- 172.21.15.107:0/1606584251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb094160b00 0x7fb0940d0da0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 -- 172.21.15.107:0/1606584251 >> 172.21.15.107:0/1606584251 conn(0x7fb09409c510 msgr2=0x7fb0940d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:25.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 -- 172.21.15.107:0/1606584251 shutdown_connections 2023-11-24T08:14:25.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.913+0000 7fb099e32640 1 -- 172.21.15.107:0/1606584251 wait complete. 2023-11-24T08:14:25.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.915+0000 7fb099e32640 1 Processor -- start 2023-11-24T08:14:25.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.915+0000 7fb099e32640 1 -- start start 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0940d0120 0x7fb0941521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940d12e0 0x7fb0940b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940cade0 con 0x7fb094160b00 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940b49e0 con 0x7fb0940d12e0 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.916+0000 7fb099e32640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940ca440 con 0x7fb0940d0120 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55612/0 (socket says 172.21.15.107:55612) 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 -- 172.21.15.107:0/3934822534 learned_addr learned my addr 172.21.15.107:0/3934822534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb093fff640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940d12e0 0x7fb0940b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb098e30640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0940d0120 0x7fb0941521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 -- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0940d0120 msgr2=0x7fb0941521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0940d0120 0x7fb0941521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 -- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940d12e0 msgr2=0x7fb0940b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940d12e0 0x7fb0940b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 -- 172.21.15.107:0/3934822534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084009a70 con 0x7fb094160b00 2023-11-24T08:14:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb099631640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c420 tx=0x7fb08c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.917+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c005ce0 con 0x7fb094160b00 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.918+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c005e40 con 0x7fb094160b00 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.918+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0940ca720 con 0x7fb094160b00 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.918+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c0072d0 con 0x7fb094160b00 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.918+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094152730 con 0x7fb094160b00 2023-11-24T08:14:25.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.919+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09404e840 con 0x7fb094160b00 2023-11-24T08:14:25.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.920+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fb08c007430 con 0x7fb094160b00 2023-11-24T08:14:25.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.921+0000 7fb0917fa640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 0x7fb07406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:25.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.921+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb08c087940 con 0x7fb094160b00 2023-11-24T08:14:25.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.921+0000 7fb098e30640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 0x7fb07406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:25.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.922+0000 7fb098e30640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 0x7fb07406e530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb084036000 tx=0x7fb084030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:25.925+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb08c054180 con 0x7fb094160b00 2023-11-24T08:14:26.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.115+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb09415ee10 con 0x7fb094160b00 2023-11-24T08:14:26.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.953+0000 7fb0917fa640 1 -- 172.21.15.107:0/3934822534 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v128) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb08c017020 con 0x7fb094160b00 2023-11-24T08:14:26.955 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.957+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 msgr2=0x7fb07406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.957+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 0x7fb07406e530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb084036000 tx=0x7fb084030000 comp rx=0 tx=0).stop 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.957+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 msgr2=0x7fb09406beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.957+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c420 tx=0x7fb08c00c8e0 comp rx=0 tx=0).stop 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 shutdown_connections 2023-11-24T08:14:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb07406c070 0x7fb07406e530 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb094160b00 0x7fb09406beb0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940d12e0 0x7fb0940b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 --2- 172.21.15.107:0/3934822534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0940d0120 0x7fb0941521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 >> 172.21.15.107:0/3934822534 conn(0x7fb09409c510 msgr2=0x7fb0940cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 shutdown_connections 2023-11-24T08:14:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:26.958+0000 7fb099e32640 1 -- 172.21.15.107:0/3934822534 wait complete. 2023-11-24T08:14:26.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:14:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.191+0000 7f53e0d2c640 1 Processor -- start 2023-11-24T08:14:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.191+0000 7f53e0d2c640 1 -- start start 2023-11-24T08:14:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.192+0000 7f53e0d2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53dc0c9650 0x7f53dc0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.192+0000 7f53e0d2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53dc166510 con 0x7f53dc0ccbd0 2023-11-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.192+0000 7f53e0d2c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53dc1655f0 con 0x7f53dc1660b0 2023-11-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.192+0000 7f53e0d2c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc0cc380 con 0x7f53dc0c9650 2023-11-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53dc0c9650 0x7f53dc0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53dc0c9650 0x7f53dc0c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55626/0 (socket says 172.21.15.107:55626) 2023-11-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 -- 172.21.15.107:0/2905919566 learned_addr learned my addr 172.21.15.107:0/2905919566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 -- 172.21.15.107:0/2905919566 >> v1:172.21.15.120:6789/0 conn(0x7f53dc1660b0 legacy=0x7f53dc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 -- 172.21.15.107:0/2905919566 >> v1:172.21.15.107:6789/0 conn(0x7f53dc0ccbd0 legacy=0x7f53dc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53d959a640 1 -- 172.21.15.107:0/2905919566 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626903489 0 0) 0x7f53cc001490 con 0x7f53dc0ccbd0 2023-11-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.193+0000 7f53dad9d640 1 -- 172.21.15.107:0/2905919566 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc162b30 con 0x7f53dc0c9650 2023-11-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53dad9d640 1 --2- 172.21.15.107:0/2905919566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc0c9650 0x7f53dc0c9ae0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009870 tx=0x7f53d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8142f1c29f12b9db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53d959a640 1 -- 172.21.15.107:0/2905919566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4030860 con 0x7f53dc0c9650 2023-11-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53d959a640 1 -- 172.21.15.107:0/2905919566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d40309c0 con 0x7f53dc0c9650 2023-11-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53d959a640 1 -- 172.21.15.107:0/2905919566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4030d50 con 0x7f53dc0c9650 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53e0d2c640 1 -- 172.21.15.107:0/2905919566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc0c9650 msgr2=0x7f53dc0c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.195+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/2905919566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc0c9650 0x7f53dc0c9ae0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009870 tx=0x7f53d402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.196+0000 7f53e0d2c640 1 -- 172.21.15.107:0/2905919566 shutdown_connections 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.196+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/2905919566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc0c9650 0x7f53dc0c9ae0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.196+0000 7f53e0d2c640 1 -- 172.21.15.107:0/2905919566 >> 172.21.15.107:0/2905919566 conn(0x7f53dc0b0c50 msgr2=0x7f53dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.196+0000 7f53e0d2c640 1 -- 172.21.15.107:0/2905919566 shutdown_connections 2023-11-24T08:14:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.196+0000 7f53e0d2c640 1 -- 172.21.15.107:0/2905919566 wait complete. 2023-11-24T08:14:27.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.197+0000 7f53e0d2c640 1 Processor -- start 2023-11-24T08:14:27.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.198+0000 7f53e0d2c640 1 -- start start 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.198+0000 7f53e0d2c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53dc0c9650 0x7f53dc154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53e0d2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53dc0ccbd0 0x7f53dc163130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53e0d2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53e0d2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc155000 con 0x7f53dc1660b0 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53e0d2c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc164720 con 0x7f53dc0c9650 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53e0d2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc163b10 con 0x7f53dc0ccbd0 2023-11-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55630/0 (socket says 172.21.15.107:55630) 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 -- 172.21.15.107:0/1349956721 learned_addr learned my addr 172.21.15.107:0/1349956721 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53da59c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53dc0c9650 0x7f53dc154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53d9d9b640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53dc0ccbd0 0x7f53dc163130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 -- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53dc0ccbd0 msgr2=0x7f53dc163130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53dc0ccbd0 0x7f53dc163130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 -- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53dc0c9650 msgr2=0x7f53dc154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53dc0c9650 0x7f53dc154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.199+0000 7f53dad9d640 1 -- 172.21.15.107:0/1349956721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40094d0 con 0x7f53dc1660b0 2023-11-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53dad9d640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f53d402b9e0 tx=0x7f53d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4039d90 con 0x7f53dc1660b0 2023-11-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d40433f0 con 0x7f53dc1660b0 2023-11-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53dc163df0 con 0x7f53dc1660b0 2023-11-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d40437a0 con 0x7f53dc1660b0 2023-11-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.200+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc1643b0 con 0x7f53dc1660b0 2023-11-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.201+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc04e8b0 con 0x7f53dc1660b0 2023-11-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.202+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f53d404c3b0 con 0x7f53dc1660b0 2023-11-24T08:14:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.203+0000 7f53d2ffd640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 0x7f53bc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.203+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f53d40ab2b0 con 0x7f53dc1660b0 2023-11-24T08:14:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.204+0000 7f53da59c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 0x7f53bc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.204+0000 7f53da59c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 0x7f53bc06e500 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0012e0 tx=0x7f53cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:27.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.207+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53d4077ba0 con 0x7f53dc1660b0 2023-11-24T08:14:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.400+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f53dc155880 con 0x7f53dc1660b0 2023-11-24T08:14:27.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.969+0000 7f53d2ffd640 1 -- 172.21.15.107:0/1349956721 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v129) v1 ==== 170+0+0 (secure 0 0 0) 0x7f53d40480d0 con 0x7f53dc1660b0 2023-11-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 msgr2=0x7f53bc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 0x7f53bc06e500 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0012e0 tx=0x7f53cc009040 comp rx=0 tx=0).stop 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 msgr2=0x7f53dc161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f53d402b9e0 tx=0x7f53d4004730 comp rx=0 tx=0).stop 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 shutdown_connections 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53bc06c040 0x7f53bc06e500 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53dc1660b0 0x7f53dc161010 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53dc0ccbd0 0x7f53dc163130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 --2- 172.21.15.107:0/1349956721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53dc0c9650 0x7f53dc154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 >> 172.21.15.107:0/1349956721 conn(0x7f53dc0b0c50 msgr2=0x7f53dc0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 shutdown_connections 2023-11-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:27.973+0000 7f53e0d2c640 1 -- 172.21.15.107:0/1349956721 wait complete. 2023-11-24T08:14:28.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:14:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.189+0000 7f180b61c640 1 Processor -- start 2023-11-24T08:14:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.189+0000 7f180b61c640 1 -- start start 2023-11-24T08:14:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f180b61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f180b61c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 0x7f180406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f180b61c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18040cc3a0 con 0x7f18040cbf20 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f180b61c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180415b820 con 0x7f18040c59b0 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f180b61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180415eea0 con 0x7f180415b440 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f18093b8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f18093b8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:58384/0 (socket says 172.21.15.107:58384) 2023-11-24T08:14:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f1809bb9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 0x7f180406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.191+0000 7f18093b8640 1 -- 172.21.15.107:0/2384682655 learned_addr learned my addr 172.21.15.107:0/2384682655 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.192+0000 7f18093b8640 1 -- 172.21.15.107:0/2384682655 >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 msgr2=0x7f180406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.192+0000 7f18093b8640 1 --2- 172.21.15.107:0/2384682655 >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 0x7f180406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.192+0000 7f18093b8640 1 -- 172.21.15.107:0/2384682655 >> v1:172.21.15.120:6789/0 conn(0x7f18040cbf20 legacy=0x7f18040c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.192+0000 7f18093b8640 1 -- 172.21.15.107:0/2384682655 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18040abf30 con 0x7f180415b440 2023-11-24T08:14:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.192+0000 7f1809bb9640 1 --2- 172.21.15.107:0/2384682655 >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 0x7f180406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:14:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.194+0000 7f18093b8640 1 --2- 172.21.15.107:0/2384682655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9799936768d4f977 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.194+0000 7f17fbfff640 1 -- 172.21.15.107:0/2384682655 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800031640 con 0x7f180415b440 2023-11-24T08:14:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.194+0000 7f17fbfff640 1 -- 172.21.15.107:0/2384682655 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18000317a0 con 0x7f180415b440 2023-11-24T08:14:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.194+0000 7f17fbfff640 1 -- 172.21.15.107:0/2384682655 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800031b30 con 0x7f180415b440 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 -- 172.21.15.107:0/2384682655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 msgr2=0x7f18040cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 --2- 172.21.15.107:0/2384682655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 -- 172.21.15.107:0/2384682655 shutdown_connections 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 --2- 172.21.15.107:0/2384682655 >> v2:172.21.15.120:3300/0 conn(0x7f18040c59b0 0x7f180406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 --2- 172.21.15.107:0/2384682655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f180415b440 0x7f18040cb9e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 -- 172.21.15.107:0/2384682655 >> 172.21.15.107:0/2384682655 conn(0x7f18040b0c50 msgr2=0x7f18040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 -- 172.21.15.107:0/2384682655 shutdown_connections 2023-11-24T08:14:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.195+0000 7f180b61c640 1 -- 172.21.15.107:0/2384682655 wait complete. 2023-11-24T08:14:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.197+0000 7f180b61c640 1 Processor -- start 2023-11-24T08:14:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.197+0000 7f180b61c640 1 -- start start 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 0x7f1804156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f180415b440 0x7f18040c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18041e7d40 con 0x7f18040cbf20 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18041560c0 con 0x7f180415b440 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f180b61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18040c6c10 con 0x7f18040c59b0 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f1808bb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f1808bb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36432/0 (socket says 172.21.15.107:36432) 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.198+0000 7f1808bb7640 1 -- 172.21.15.107:0/2809185732 learned_addr learned my addr 172.21.15.107:0/2809185732 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f18093b8640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 0x7f1804156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1809bb9640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f180415b440 0x7f18040c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 -- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 msgr2=0x7f1804156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 0x7f1804156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 -- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f180415b440 msgr2=0x7f18040c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f180415b440 0x7f18040c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 -- 172.21.15.107:0/2809185732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc009a70 con 0x7f18040cbf20 2023-11-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f18093b8640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 0x7f1804156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f1808bb7640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c400 tx=0x7f17f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f40052d0 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f4005450 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4007260 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18000094d0 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.199+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180415c820 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.201+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180415ca40 con 0x7f18040cbf20 2023-11-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.202+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f17f40073c0 con 0x7f18040cbf20 2023-11-24T08:14:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.203+0000 7f17f9ffb640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 0x7f17e806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.203+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f17f4087830 con 0x7f18040cbf20 2023-11-24T08:14:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.203+0000 7f18093b8640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 0x7f17e806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.204+0000 7f18093b8640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 0x7f17e806e500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1800036000 tx=0x7f180003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:28.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.208+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17f4053fc0 con 0x7f18040cbf20 2023-11-24T08:14:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.422+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f18040cc3a0 con 0x7f18040cbf20 2023-11-24T08:14:28.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.973+0000 7f17f9ffb640 1 -- 172.21.15.107:0/2809185732 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f18040cc3a0 con 0x7f18040cbf20 2023-11-24T08:14:28.976 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 18 and data pool 19 2023-11-24T08:14:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 msgr2=0x7f17e806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 0x7f17e806e500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1800036000 tx=0x7f180003c040 comp rx=0 tx=0).stop 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 msgr2=0x7f1804155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c400 tx=0x7f17f400c8c0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 shutdown_connections 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17e806c040 0x7f17e806e500 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f180415b440 0x7f18040c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18040cbf20 0x7f1804155b80 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 --2- 172.21.15.107:0/2809185732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18040c59b0 0x7f1804156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.977+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 >> 172.21.15.107:0/2809185732 conn(0x7f18040b0c50 msgr2=0x7f18040c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.978+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 shutdown_connections 2023-11-24T08:14:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:28.978+0000 7f180b61c640 1 -- 172.21.15.107:0/2809185732 wait complete. 2023-11-24T08:14:29.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:29.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.189+0000 7f31eecc9640 1 Processor -- start 2023-11-24T08:14:29.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.190+0000 7f31eecc9640 1 -- start start 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eecc9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31e815e390 0x7f31e8161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eecc9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e80ce0b0 con 0x7f31e8157c70 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eecc9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e8161960 con 0x7f31e80cfb60 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eecc9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e815e8d0 con 0x7f31e815e390 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eca65640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f31e8157c70 0x7f31e80cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36948/0 (socket says 172.21.15.107:36948) 2023-11-24T08:14:29.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.191+0000 7f31eca65640 1 -- 172.21.15.107:0/3393799587 learned_addr learned my addr 172.21.15.107:0/3393799587 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:29.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057917145 0 0) 0x7f31e80ce0b0 con 0x7f31e8157c70 2023-11-24T08:14:29.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d4003520 con 0x7f31e8157c70 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31ed266640 1 --2- 172.21.15.107:0/3393799587 >> v2:172.21.15.120:3300/0 conn(0x7f31e815e390 0x7f31e8161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2296578271 0 0) 0x7f31d4003520 con 0x7f31e8157c70 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 >> v2:172.21.15.120:3300/0 conn(0x7f31e815e390 msgr2=0x7f31e8161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 --2- 172.21.15.107:0/3393799587 >> v2:172.21.15.120:3300/0 conn(0x7f31e815e390 0x7f31e8161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 >> v1:172.21.15.120:6789/0 conn(0x7f31e80cfb60 legacy=0x7f31e815dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e815ebb0 con 0x7f31e8157c70 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f31d80019f0 con 0x7f31e8157c70 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 >> v1:172.21.15.107:6789/0 conn(0x7f31e8157c70 legacy=0x7f31e80cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40039c0 0x7f31d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 0x7f31d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31d4005250 0x7f31d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e815e8d0 con 0x7f31d40047c0 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d400c1a0 con 0x7f31d40039c0 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d400c360 con 0x7f31d4005250 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.192+0000 7f31e7fff640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 0x7f31d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31eca65640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40039c0 0x7f31d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31e7fff640 1 -- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31d4005250 msgr2=0x7f31d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31e7fff640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31d4005250 0x7f31d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31e7fff640 1 -- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40039c0 msgr2=0x7f31d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31e7fff640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40039c0 0x7f31d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31e7fff640 1 -- 172.21.15.107:0/3393799587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4003520 con 0x7f31d40047c0 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.193+0000 7f31ed266640 1 -- 172.21.15.107:0/3393799587 reap_dead start 2023-11-24T08:14:29.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.194+0000 7f31e7fff640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 0x7f31d4004c40 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f31d0003db0 tx=0x7f31d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6daecbf626e78fdc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.194+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0032600 con 0x7f31d40047c0 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.194+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d002e050 con 0x7f31d40047c0 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.194+0000 7f31e77fe640 1 -- 172.21.15.107:0/3393799587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00329e0 con 0x7f31d40047c0 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 -- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 msgr2=0x7f31d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 0x7f31d4004c40 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f31d0003db0 tx=0x7f31d40057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 -- 172.21.15.107:0/3393799587 shutdown_connections 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 --2- 172.21.15.107:0/3393799587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31d40047c0 0x7f31d4004c40 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 -- 172.21.15.107:0/3393799587 >> 172.21.15.107:0/3393799587 conn(0x7f31e806a920 msgr2=0x7f31e806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 -- 172.21.15.107:0/3393799587 shutdown_connections 2023-11-24T08:14:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.195+0000 7f31eecc9640 1 -- 172.21.15.107:0/3393799587 wait complete. 2023-11-24T08:14:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.197+0000 7f31eecc9640 1 Processor -- start 2023-11-24T08:14:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.197+0000 7f31eecc9640 1 -- start start 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40047c0 0x7f31e8147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31e80ca600 0x7f31e81560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e81f3f30 con 0x7f31e80c8360 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e81f33b0 con 0x7f31d40047c0 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eecc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e81f3690 con 0x7f31e80ca600 2023-11-24T08:14:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36448/0 (socket says 172.21.15.107:36448) 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 -- 172.21.15.107:0/238947174 learned_addr learned my addr 172.21.15.107:0/238947174 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31eca65640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40047c0 0x7f31e8147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 -- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31e80ca600 msgr2=0x7f31e81560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31ed266640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31e80ca600 0x7f31e81560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31e80ca600 0x7f31e81560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 -- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40047c0 msgr2=0x7f31e8147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40047c0 0x7f31e8147700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.198+0000 7f31e7fff640 1 -- 172.21.15.107:0/238947174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e81f3970 con 0x7f31e80c8360 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31e7fff640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f31d0035040 tx=0x7f31d00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d003c410 con 0x7f31e80c8360 2023-11-24T08:14:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d003c570 con 0x7f31e80c8360 2023-11-24T08:14:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e81f3c50 con 0x7f31e80c8360 2023-11-24T08:14:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d003c900 con 0x7f31e80c8360 2023-11-24T08:14:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.199+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e81f2fc0 con 0x7f31e80c8360 2023-11-24T08:14:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.200+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e80caa10 con 0x7f31e80c8360 2023-11-24T08:14:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.201+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f31d0039050 con 0x7f31e80c8360 2023-11-24T08:14:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.202+0000 7f31e57fa640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 0x7f31c806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.203+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31d00a9970 con 0x7f31e80c8360 2023-11-24T08:14:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.203+0000 7f31eca65640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 0x7f31c806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.203+0000 7f31eca65640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 0x7f31c806e500 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f31d8001f50 tx=0x7f31d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.207+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31d00762d0 con 0x7f31e80c8360 2023-11-24T08:14:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.383+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31e8067470 con 0x7f31e80c8360 2023-11-24T08:14:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.386+0000 7f31e57fa640 1 -- 172.21.15.107:0/238947174 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f31d004b090 con 0x7f31e80c8360 2023-11-24T08:14:29.388 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:29.389 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:14:28.424363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 msgr2=0x7f31c806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 0x7f31c806e500 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f31d8001f50 tx=0x7f31d8009040 comp rx=0 tx=0).stop 2023-11-24T08:14:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 msgr2=0x7f31e80ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f31d0035040 tx=0x7f31d00036f0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 shutdown_connections 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31c806c040 0x7f31c806e500 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31e80ca600 0x7f31e81560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31e80c8360 0x7f31e80ca0c0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 --2- 172.21.15.107:0/238947174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31d40047c0 0x7f31e8147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 >> 172.21.15.107:0/238947174 conn(0x7f31e806a920 msgr2=0x7f31e806c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 shutdown_connections 2023-11-24T08:14:29.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.389+0000 7f31eecc9640 1 -- 172.21.15.107:0/238947174 wait complete. 2023-11-24T08:14:29.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.613+0000 7f4a223ff640 1 Processor -- start 2023-11-24T08:14:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.614+0000 7f4a223ff640 1 -- start start 2023-11-24T08:14:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.615+0000 7f4a223ff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a1c15c410 0x7f4a1c15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.615+0000 7f4a223ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.615+0000 7f4a223ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c15bf80 con 0x7f4a1c144b30 2023-11-24T08:14:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.615+0000 7f4a223ff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c0b27b0 con 0x7f4a1c15c410 2023-11-24T08:14:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.615+0000 7f4a223ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0b1560 con 0x7f4a1c0b1de0 2023-11-24T08:14:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.616+0000 7f4a20bfc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a1c15c410 0x7f4a1c15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.616+0000 7f4a20bfc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a1c15c410 0x7f4a1c15f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:57492/0 (socket says 172.21.15.107:57492) 2023-11-24T08:14:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.616+0000 7f4a20bfc640 1 -- 172.21.15.107:0/3678951452 learned_addr learned my addr 172.21.15.107:0/3678951452 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.616+0000 7f4a21bfe640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.617+0000 7f4a20bfc640 1 -- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 msgr2=0x7f4a1c0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.617+0000 7f4a20bfc640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c0b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.617+0000 7f4a20bfc640 1 -- 172.21.15.107:0/3678951452 >> v1:172.21.15.120:6789/0 conn(0x7f4a1c144b30 legacy=0x7f4a1c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.617+0000 7f4a20bfc640 1 -- 172.21.15.107:0/3678951452 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0b1280 con 0x7f4a1c15c410 2023-11-24T08:14:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.618+0000 7f4a20bfc640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c15f920 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009870 tx=0x7f4a0c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9adf533a8c2768d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.619+0000 7f4a13fff640 1 -- 172.21.15.107:0/3678951452 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c030840 con 0x7f4a1c15c410 2023-11-24T08:14:29.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.619+0000 7f4a13fff640 1 -- 172.21.15.107:0/3678951452 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c0309a0 con 0x7f4a1c15c410 2023-11-24T08:14:29.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.619+0000 7f4a13fff640 1 -- 172.21.15.107:0/3678951452 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c030d30 con 0x7f4a1c15c410 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.619+0000 7f4a223ff640 1 -- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 msgr2=0x7f4a1c15f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.619+0000 7f4a223ff640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c15f920 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009870 tx=0x7f4a0c02b980 comp rx=0 tx=0).stop 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 -- 172.21.15.107:0/3678951452 shutdown_connections 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c0b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 --2- 172.21.15.107:0/3678951452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c15f920 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 -- 172.21.15.107:0/3678951452 >> 172.21.15.107:0/3678951452 conn(0x7f4a1c0cb830 msgr2=0x7f4a1c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 -- 172.21.15.107:0/3678951452 shutdown_connections 2023-11-24T08:14:29.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.620+0000 7f4a223ff640 1 -- 172.21.15.107:0/3678951452 wait complete. 2023-11-24T08:14:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.621+0000 7f4a223ff640 1 Processor -- start 2023-11-24T08:14:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.622+0000 7f4a223ff640 1 -- start start 2023-11-24T08:14:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c15e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c0caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1f6540 con 0x7f4a1c144b30 2023-11-24T08:14:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1f6790 con 0x7f4a1c15c410 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a223ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1f6a70 con 0x7f4a1c0b1de0 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36460/0 (socket says 172.21.15.107:36460) 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 -- 172.21.15.107:0/1920420512 learned_addr learned my addr 172.21.15.107:0/1920420512 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a21bfe640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c0caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 -- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 msgr2=0x7f4a1c15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 -- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 msgr2=0x7f4a1c0caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c0caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.623+0000 7f4a20bfc640 1 -- 172.21.15.107:0/1920420512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c0094d0 con 0x7f4a1c144b30 2023-11-24T08:14:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a20bfc640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c02b9c0 tx=0x7f4a0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c03e070 con 0x7f4a1c144b30 2023-11-24T08:14:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c039a10 con 0x7f4a1c144b30 2023-11-24T08:14:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c039de0 con 0x7f4a1c144b30 2023-11-24T08:14:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c1f6d50 con 0x7f4a1c144b30 2023-11-24T08:14:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.624+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c0c91b0 con 0x7f4a1c144b30 2023-11-24T08:14:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.625+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c1f7270 con 0x7f4a1c144b30 2023-11-24T08:14:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.626+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f4a0c04b400 con 0x7f4a1c144b30 2023-11-24T08:14:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.627+0000 7f4a11ffb640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 0x7f4a0006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.627+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a0c0aab90 con 0x7f4a1c144b30 2023-11-24T08:14:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.627+0000 7f4a213fd640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 0x7f4a0006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.628+0000 7f4a213fd640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 0x7f4a0006e500 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000f60 tx=0x7f4a180079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:29.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.631+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a1c1f7270 con 0x7f4a1c144b30 2023-11-24T08:14:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.809+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a1c1f7270 con 0x7f4a1c144b30 2023-11-24T08:14:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.812+0000 7f4a11ffb640 1 -- 172.21.15.107:0/1920420512 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f4a1c1f7270 con 0x7f4a1c144b30 2023-11-24T08:14:29.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:29.815 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:14:28.424363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 msgr2=0x7f4a0006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 0x7f4a0006e500 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000f60 tx=0x7f4a180079b0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 msgr2=0x7f4a1c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c02b9c0 tx=0x7f4a0c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 shutdown_connections 2023-11-24T08:14:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a0006c040 0x7f4a0006e500 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a1c15c410 0x7f4a1c0caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a1c144b30 0x7f4a1c0caa20 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 --2- 172.21.15.107:0/1920420512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a1c0b1de0 0x7f4a1c15e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 >> 172.21.15.107:0/1920420512 conn(0x7f4a1c0cb830 msgr2=0x7f4a1c15c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 shutdown_connections 2023-11-24T08:14:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:29.815+0000 7f4a223ff640 1 -- 172.21.15.107:0/1920420512 wait complete. 2023-11-24T08:14:29.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:14:30.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.042+0000 7f351fcbb640 1 Processor -- start 2023-11-24T08:14:30.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.043+0000 7f351fcbb640 1 -- start start 2023-11-24T08:14:30.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351fcbb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f351806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351fcbb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35180d2000 0x7f35180cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351fcbb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180ce410 con 0x7f35180d0200 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351fcbb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180d06f0 con 0x7f35180d2000 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351fcbb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180d09d0 con 0x7f35180d1c20 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351d256640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35180d2000 0x7f35180cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351e258640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f35180d0200 0x7f35181605f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36954/0 (socket says 172.21.15.107:36954) 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.044+0000 7f351e258640 1 -- 172.21.15.107:0/3724753902 learned_addr learned my addr 172.21.15.107:0/3724753902 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.045+0000 7f351da57640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f351806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.045+0000 7f351d256640 1 -- 172.21.15.107:0/3724753902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 msgr2=0x7f351806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.045+0000 7f351d256640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f351806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.045+0000 7f351d256640 1 -- 172.21.15.107:0/3724753902 >> v1:172.21.15.107:6789/0 conn(0x7f35180d0200 legacy=0x7f35181605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:30.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.045+0000 7f351d256640 1 -- 172.21.15.107:0/3724753902 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180ce0a0 con 0x7f35180d2000 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.046+0000 7f351d256640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d2000 0x7f35180cfcc0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f35080097a0 tx=0x7f350802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a147e1f167685549 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.046+0000 7f351ca55640 1 -- 172.21.15.107:0/3724753902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350802d040 con 0x7f35180d2000 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.046+0000 7f351ca55640 1 -- 172.21.15.107:0/3724753902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508031810 con 0x7f35180d2000 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.046+0000 7f351ca55640 1 -- 172.21.15.107:0/3724753902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508031ba0 con 0x7f35180d2000 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 -- 172.21.15.107:0/3724753902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d2000 msgr2=0x7f35180cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d2000 0x7f35180cfcc0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f35080097a0 tx=0x7f350802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 -- 172.21.15.107:0/3724753902 shutdown_connections 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d2000 0x7f35180cfcc0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 --2- 172.21.15.107:0/3724753902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f351806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 -- 172.21.15.107:0/3724753902 >> 172.21.15.107:0/3724753902 conn(0x7f351808cea0 msgr2=0x7f351809c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 -- 172.21.15.107:0/3724753902 shutdown_connections 2023-11-24T08:14:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.047+0000 7f351fcbb640 1 -- 172.21.15.107:0/3724753902 wait complete. 2023-11-24T08:14:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.048+0000 7f351fcbb640 1 Processor -- start 2023-11-24T08:14:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 -- start start 2023-11-24T08:14:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f3518153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180d2000 0x7f35180a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180b0ef0 con 0x7f35180d0200 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180b0020 con 0x7f35180d2000 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.049+0000 7f351fcbb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351815e240 con 0x7f35180d1c20 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36486/0 (socket says 172.21.15.107:36486) 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 -- 172.21.15.107:0/1469491460 learned_addr learned my addr 172.21.15.107:0/1469491460 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 -- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 msgr2=0x7f3518153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f3518153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 -- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180d2000 msgr2=0x7f35180a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180d2000 0x7f35180a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 -- 172.21.15.107:0/1469491460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508009400 con 0x7f35180d0200 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f351da57640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f351400ce40 tx=0x7f351400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514003b90 con 0x7f35180d0200 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514003cf0 con 0x7f35180d0200 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.050+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514004240 con 0x7f35180d0200 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.051+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35180b0270 con 0x7f35180d0200 2023-11-24T08:14:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.051+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35180b0830 con 0x7f35180d0200 2023-11-24T08:14:30.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.052+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180b1030 con 0x7f35180d0200 2023-11-24T08:14:30.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.053+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f35140043a0 con 0x7f35180d0200 2023-11-24T08:14:30.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.054+0000 7f350e7fc640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 0x7f34fc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:30.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.054+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3514012070 con 0x7f35180d0200 2023-11-24T08:14:30.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.054+0000 7f351d256640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 0x7f34fc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:30.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.054+0000 7f351d256640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 0x7f34fc06e500 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3508036000 tx=0x7f3508030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:30.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.058+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35180b1030 con 0x7f35180d0200 2023-11-24T08:14:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:30.268+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f35180b1030 con 0x7f35180d0200 2023-11-24T08:14:31.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.005+0000 7f350e7fc640 1 -- 172.21.15.107:0/1469491460 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35180b1030 con 0x7f35180d0200 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 msgr2=0x7f34fc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 0x7f34fc06e500 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3508036000 tx=0x7f3508030000 comp rx=0 tx=0).stop 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 msgr2=0x7f35180b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f351400ce40 tx=0x7f351400a660 comp rx=0 tx=0).stop 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 shutdown_connections 2023-11-24T08:14:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34fc06c040 0x7f34fc06e500 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35180d2000 0x7f35180a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35180d1c20 0x7f3518153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 --2- 172.21.15.107:0/1469491460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35180d0200 0x7f35180b22e0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 >> 172.21.15.107:0/1469491460 conn(0x7f351808cea0 msgr2=0x7f35180d3140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.008+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 shutdown_connections 2023-11-24T08:14:31.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.009+0000 7f351fcbb640 1 -- 172.21.15.107:0/1469491460 wait complete. 2023-11-24T08:14:31.029 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:14:31.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:14:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.222+0000 7f46a3733640 1 Processor -- start 2023-11-24T08:14:31.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.222+0000 7f46a3733640 1 -- start start 2023-11-24T08:14:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.224+0000 7f46a3733640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 0x7f469c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.224+0000 7f46a3733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.224+0000 7f46a3733640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c15ec90 con 0x7f469c06dc40 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.224+0000 7f46a3733640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469c06e0f0 con 0x7f469c072160 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.224+0000 7f46a3733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0cd530 con 0x7f469c0c49f0 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.225+0000 7f46a14cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 0x7f469c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.225+0000 7f46a0cce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.225+0000 7f46a14cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 0x7f469c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:57532/0 (socket says 172.21.15.107:57532) 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.225+0000 7f46a1cd0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f469c06dc40 0x7f469c0cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:43120/0 (socket says 172.21.15.107:43120) 2023-11-24T08:14:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.225+0000 7f46a14cf640 1 -- 172.21.15.107:0/2271506928 learned_addr learned my addr 172.21.15.107:0/2271506928 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.226+0000 7f46a0cce640 1 -- 172.21.15.107:0/2271506928 >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 msgr2=0x7f469c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.226+0000 7f46a0cce640 1 --2- 172.21.15.107:0/2271506928 >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 0x7f469c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.226+0000 7f46a0cce640 1 -- 172.21.15.107:0/2271506928 >> v1:172.21.15.120:6789/0 conn(0x7f469c06dc40 legacy=0x7f469c0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.226+0000 7f46a0cce640 1 -- 172.21.15.107:0/2271506928 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0c4ed0 con 0x7f469c0c49f0 2023-11-24T08:14:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.227+0000 7f46a0cce640 1 --2- 172.21.15.107:0/2271506928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c06d700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f46900097a0 tx=0x7f469002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47ce04432def9b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f468ffff640 1 -- 172.21.15.107:0/2271506928 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469002d040 con 0x7f469c0c49f0 2023-11-24T08:14:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f468ffff640 1 -- 172.21.15.107:0/2271506928 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4690031810 con 0x7f469c0c49f0 2023-11-24T08:14:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f468ffff640 1 -- 172.21.15.107:0/2271506928 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690031ba0 con 0x7f469c0c49f0 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 -- 172.21.15.107:0/2271506928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 msgr2=0x7f469c06d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 --2- 172.21.15.107:0/2271506928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c06d700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f46900097a0 tx=0x7f469002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 -- 172.21.15.107:0/2271506928 shutdown_connections 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 --2- 172.21.15.107:0/2271506928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c06d700 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 --2- 172.21.15.107:0/2271506928 >> v2:172.21.15.120:3300/0 conn(0x7f469c072160 0x7f469c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.228+0000 7f46a3733640 1 -- 172.21.15.107:0/2271506928 >> 172.21.15.107:0/2271506928 conn(0x7f469c0b0c50 msgr2=0x7f469c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.229+0000 7f46a3733640 1 -- 172.21.15.107:0/2271506928 shutdown_connections 2023-11-24T08:14:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.229+0000 7f46a3733640 1 -- 172.21.15.107:0/2271506928 wait complete. 2023-11-24T08:14:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.230+0000 7f46a3733640 1 Processor -- start 2023-11-24T08:14:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.230+0000 7f46a3733640 1 -- start start 2023-11-24T08:14:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f469c072160 0x7f469c156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15c8e0 con 0x7f469c06dc40 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15abf0 con 0x7f469c072160 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.231+0000 7f46a3733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15aed0 con 0x7f469c0c49f0 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36490/0 (socket says 172.21.15.107:36490) 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 -- 172.21.15.107:0/399131764 learned_addr learned my addr 172.21.15.107:0/399131764 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a1cd0640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a0cce640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f469c072160 0x7f469c156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 -- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 msgr2=0x7f469c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c15d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 -- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f469c072160 msgr2=0x7f469c156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f469c072160 0x7f469c156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 -- 172.21.15.107:0/399131764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4690009400 con 0x7f469c06dc40 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a1cd0640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.232+0000 7f46a14cf640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4698009fd0 tx=0x7f469800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.233+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469800c940 con 0x7f469c06dc40 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.233+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469800caa0 con 0x7f469c06dc40 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.233+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c15b1b0 con 0x7f469c06dc40 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.233+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469800ce30 con 0x7f469c06dc40 2023-11-24T08:14:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.233+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c157340 con 0x7f469c06dc40 2023-11-24T08:14:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.234+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c15ced0 con 0x7f469c06dc40 2023-11-24T08:14:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.235+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f4698013470 con 0x7f469c06dc40 2023-11-24T08:14:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.236+0000 7f468dffb640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 0x7f468006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.236+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4698088790 con 0x7f469c06dc40 2023-11-24T08:14:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.236+0000 7f46a0cce640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 0x7f468006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.237+0000 7f46a0cce640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 0x7f468006e530 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f469003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.240+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4698054ef0 con 0x7f469c06dc40 2023-11-24T08:14:31.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:31.462+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f469c15ca20 con 0x7f469c06dc40 2023-11-24T08:14:32.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.023+0000 7f468dffb640 1 -- 172.21.15.107:0/399131764 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4698021020 con 0x7f469c06dc40 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 msgr2=0x7f468006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 0x7f468006e530 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f469003c040 comp rx=0 tx=0).stop 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 msgr2=0x7f469c0c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4698009fd0 tx=0x7f469800ed40 comp rx=0 tx=0).stop 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 shutdown_connections 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f468006c070 0x7f468006e530 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f469c0c49f0 0x7f469c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f469c072160 0x7f469c156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 --2- 172.21.15.107:0/399131764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f469c06dc40 0x7f469c0c8db0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 >> 172.21.15.107:0/399131764 conn(0x7f469c0b0c50 msgr2=0x7f469c06cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:32.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 shutdown_connections 2023-11-24T08:14:32.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.027+0000 7f46a3733640 1 -- 172.21.15.107:0/399131764 wait complete. 2023-11-24T08:14:32.048 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:14:32.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:32.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.250+0000 7fce27227640 1 Processor -- start 2023-11-24T08:14:32.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.250+0000 7fce27227640 1 -- start start 2023-11-24T08:14:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce27227640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce201684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce27227640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200cef60 con 0x7fce200d1740 2023-11-24T08:14:32.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce27227640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b2ac0 con 0x7fce200d0f40 2023-11-24T08:14:32.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce27227640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200b2da0 con 0x7fce201660e0 2023-11-24T08:14:32.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce26225640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce200d1740 0x7fce200ce780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36958/0 (socket says 172.21.15.107:36958) 2023-11-24T08:14:32.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce26225640 1 -- 172.21.15.107:0/1445389621 learned_addr learned my addr 172.21.15.107:0/1445389621 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775312418 0 0) 0x7fce200cef60 con 0x7fce200d1740 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce26a26640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce201684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.252+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c003520 con 0x7fce200d1740 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345791212 0 0) 0x7fce0c003520 con 0x7fce200d1740 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 msgr2=0x7fce201684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce201684d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 >> v1:172.21.15.120:6789/0 conn(0x7fce200d0f40 legacy=0x7fce200d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200b3080 con 0x7fce200d1740 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fce1c001670 con 0x7fce200d1740 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 >> v1:172.21.15.107:6789/0 conn(0x7fce200d1740 legacy=0x7fce200ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce0c004d00 0x7fce0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200b2da0 con 0x7fce0c0039a0 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c00bba0 con 0x7fce0c004d00 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c00bd60 con 0x7fce201660e0 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.253+0000 7fce26225640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26a26640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce0c004d00 0x7fce0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce25a24640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26225640 1 -- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 msgr2=0x7fce0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26225640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26225640 1 -- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce0c004d00 msgr2=0x7fce0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26225640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce0c004d00 0x7fce0c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26225640 1 -- 172.21.15.107:0/1445389621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c003520 con 0x7fce0c0039a0 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce26a26640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce0c004d00 0x7fce0c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.254+0000 7fce25a24640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce201660e0 0x7fce0c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.255+0000 7fce26225640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce0c003dc0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c240 tx=0x7fce0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1942f3729018e3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.255+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c005ad0 con 0x7fce0c0039a0 2023-11-24T08:14:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.255+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c005c30 con 0x7fce0c0039a0 2023-11-24T08:14:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.255+0000 7fce25223640 1 -- 172.21.15.107:0/1445389621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c02c8b0 con 0x7fce0c0039a0 2023-11-24T08:14:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 -- 172.21.15.107:0/1445389621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 msgr2=0x7fce0c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 --2- 172.21.15.107:0/1445389621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce0c003dc0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c240 tx=0x7fce0c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce26a26640 1 -- 172.21.15.107:0/1445389621 reap_dead start 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 -- 172.21.15.107:0/1445389621 shutdown_connections 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 -- 172.21.15.107:0/1445389621 >> 172.21.15.107:0/1445389621 conn(0x7fce2009c500 msgr2=0x7fce200d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 -- 172.21.15.107:0/1445389621 shutdown_connections 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.256+0000 7fce27227640 1 -- 172.21.15.107:0/1445389621 wait complete. 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.257+0000 7fce27227640 1 Processor -- start 2023-11-24T08:14:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.258+0000 7fce27227640 1 -- start start 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.258+0000 7fce27227640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.258+0000 7fce27227640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20155ed0 0x7fce20155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.258+0000 7fce27227640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce2015fa40 0x7fce200cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce27227640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015eed0 con 0x7fce0c0039a0 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce27227640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015e500 con 0x7fce20155ed0 2023-11-24T08:14:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce27227640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce20159ec0 con 0x7fce2015fa40 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36514/0 (socket says 172.21.15.107:36514) 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 -- 172.21.15.107:0/4260231128 learned_addr learned my addr 172.21.15.107:0/4260231128 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 -- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce2015fa40 msgr2=0x7fce200cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce2015fa40 0x7fce200cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce25a24640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20155ed0 0x7fce20155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 -- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20155ed0 msgr2=0x7fce20155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20155ed0 0x7fce20155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.259+0000 7fce26225640 1 -- 172.21.15.107:0/4260231128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2015dbd0 con 0x7fce0c0039a0 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce26225640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fce1c02c450 tx=0x7fce1c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c039070 con 0x7fce0c0039a0 2023-11-24T08:14:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c02cce0 con 0x7fce0c0039a0 2023-11-24T08:14:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c03c040 con 0x7fce0c0039a0 2023-11-24T08:14:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2015deb0 con 0x7fce0c0039a0 2023-11-24T08:14:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.260+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce20155c40 con 0x7fce0c0039a0 2023-11-24T08:14:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.261+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2004e840 con 0x7fce0c0039a0 2023-11-24T08:14:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.262+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fce1c032070 con 0x7fce0c0039a0 2023-11-24T08:14:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.263+0000 7fce16ffd640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 0x7fce0406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.263+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce1c0a9630 con 0x7fce0c0039a0 2023-11-24T08:14:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.263+0000 7fce25a24640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 0x7fce0406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.264+0000 7fce25a24640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 0x7fce0406e500 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fce10007fd0 tx=0x7fce10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.267+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce1c076190 con 0x7fce0c0039a0 2023-11-24T08:14:32.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.499+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce200b5760 con 0x7fce0c0039a0 2023-11-24T08:14:32.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.500+0000 7fce16ffd640 1 -- 172.21.15.107:0/4260231128 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fce1c0797b0 con 0x7fce0c0039a0 2023-11-24T08:14:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:32.502 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:14:32.502 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 78 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.502+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 msgr2=0x7fce0406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.502+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 0x7fce0406e500 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fce10007fd0 tx=0x7fce10009040 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.502+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 msgr2=0x7fce2015e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.502+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fce1c02c450 tx=0x7fce1c001a00 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 shutdown_connections 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce0406c040 0x7fce0406e500 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce2015fa40 0x7fce200cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20155ed0 0x7fce20155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 --2- 172.21.15.107:0/4260231128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c0039a0 0x7fce2015e990 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 >> 172.21.15.107:0/4260231128 conn(0x7fce2009c500 msgr2=0x7fce200d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:32.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 shutdown_connections 2023-11-24T08:14:32.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.503+0000 7fce27227640 1 -- 172.21.15.107:0/4260231128 wait complete. 2023-11-24T08:14:32.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:14:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.720+0000 7f6bee34e640 1 Processor -- start 2023-11-24T08:14:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.720+0000 7f6bee34e640 1 -- start start 2023-11-24T08:14:32.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.721+0000 7f6bee34e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6be815c410 0x7f6be815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6bee34e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be815bf80 con 0x7f6be8144b30 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6bee34e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80b1560 con 0x7f6be80b1de0 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6bee34e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80b1280 con 0x7f6be815c410 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6bed34c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6be8144b30 0x7f6be80b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36966/0 (socket says 172.21.15.107:36966) 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6bed34c640 1 -- 172.21.15.107:0/1637202965 learned_addr learned my addr 172.21.15.107:0/1637202965 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:32.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6becb4b640 1 --2- 172.21.15.107:0/1637202965 >> v2:172.21.15.107:3300/0 conn(0x7f6be815c410 0x7f6be815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6becb4b640 1 -- 172.21.15.107:0/1637202965 >> v1:172.21.15.120:6789/0 conn(0x7f6be80b1de0 legacy=0x7f6be80b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6becb4b640 1 -- 172.21.15.107:0/1637202965 >> v1:172.21.15.107:6789/0 conn(0x7f6be8144b30 legacy=0x7f6be80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.722+0000 7f6becb4b640 1 -- 172.21.15.107:0/1637202965 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be815b930 con 0x7f6be815c410 2023-11-24T08:14:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.723+0000 7f6becb4b640 1 --2- 172.21.15.107:0/1637202965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be815f920 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80097a0 tx=0x7f6bd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c9e78cbbdad34f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.724+0000 7f6bdffff640 1 -- 172.21.15.107:0/1637202965 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8031830 con 0x7f6be815c410 2023-11-24T08:14:32.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.724+0000 7f6bdffff640 1 -- 172.21.15.107:0/1637202965 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8031990 con 0x7f6be815c410 2023-11-24T08:14:32.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.724+0000 7f6bdffff640 1 -- 172.21.15.107:0/1637202965 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8031d20 con 0x7f6be815c410 2023-11-24T08:14:32.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.724+0000 7f6bee34e640 1 -- 172.21.15.107:0/1637202965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 msgr2=0x7f6be815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.724+0000 7f6bee34e640 1 --2- 172.21.15.107:0/1637202965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be815f920 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80097a0 tx=0x7f6bd802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.725+0000 7f6bee34e640 1 -- 172.21.15.107:0/1637202965 shutdown_connections 2023-11-24T08:14:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.725+0000 7f6bee34e640 1 --2- 172.21.15.107:0/1637202965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be815f920 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.725+0000 7f6bee34e640 1 -- 172.21.15.107:0/1637202965 >> 172.21.15.107:0/1637202965 conn(0x7f6be80cb830 msgr2=0x7f6be80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.725+0000 7f6bee34e640 1 -- 172.21.15.107:0/1637202965 shutdown_connections 2023-11-24T08:14:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.725+0000 7f6bee34e640 1 -- 172.21.15.107:0/1637202965 wait complete. 2023-11-24T08:14:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.726+0000 7f6bee34e640 1 Processor -- start 2023-11-24T08:14:32.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.726+0000 7f6bee34e640 1 -- start start 2023-11-24T08:14:32.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 0x7f6be814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6be8144b30 0x7f6be806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be814dc00 con 0x7f6be815c410 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8141d50 con 0x7f6be8144b30 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.727+0000 7f6bee34e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8142030 con 0x7f6be80b1de0 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36534/0 (socket says 172.21.15.107:36534) 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 -- 172.21.15.107:0/2291452689 learned_addr learned my addr 172.21.15.107:0/2291452689 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bed34c640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 0x7f6be814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6becb4b640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6be8144b30 0x7f6be806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 -- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 msgr2=0x7f6be814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 0x7f6be814d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 -- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6be8144b30 msgr2=0x7f6be806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6be8144b30 0x7f6be806cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 -- 172.21.15.107:0/2291452689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8009400 con 0x7f6be815c410 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bed34c640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 0x7f6be814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.728+0000 7f6bedb4d640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f6be000b970 tx=0x7f6be000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.729+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be000cb40 con 0x7f6be815c410 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.729+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be000cca0 con 0x7f6be815c410 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.729+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be0014680 con 0x7f6be815c410 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.729+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8142310 con 0x7f6be815c410 2023-11-24T08:14:32.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.729+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be81428d0 con 0x7f6be815c410 2023-11-24T08:14:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.730+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be804e840 con 0x7f6be815c410 2023-11-24T08:14:32.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.731+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f6be001d3b0 con 0x7f6be815c410 2023-11-24T08:14:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.732+0000 7f6bddffb640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 0x7f6bcc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.732+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6be0087ff0 con 0x7f6be815c410 2023-11-24T08:14:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.732+0000 7f6bed34c640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 0x7f6bcc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:32.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.733+0000 7f6bed34c640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 0x7f6bcc06e500 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6be4005880 tx=0x7f6be4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:32.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.736+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6be0054b50 con 0x7f6be815c410 2023-11-24T08:14:32.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.925+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6be815db40 con 0x7f6be815c410 2023-11-24T08:14:32.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.928+0000 7f6bddffb640 1 -- 172.21.15.107:0/2291452689 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f6be0019090 con 0x7f6be815c410 2023-11-24T08:14:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:32.930 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":131,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:14:29.993459+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.930+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 msgr2=0x7f6bcc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.930+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 0x7f6bcc06e500 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6be4005880 tx=0x7f6be4009040 comp rx=0 tx=0).stop 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.930+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 msgr2=0x7f6be81e6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.930+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f6be000b970 tx=0x7f6be000be30 comp rx=0 tx=0).stop 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 shutdown_connections 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6bcc06c040 0x7f6bcc06e500 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6be815c410 0x7f6be81e6d90 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6be8144b30 0x7f6be806cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 --2- 172.21.15.107:0/2291452689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6be80b1de0 0x7f6be814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 >> 172.21.15.107:0/2291452689 conn(0x7f6be80cb830 msgr2=0x7f6be80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 shutdown_connections 2023-11-24T08:14:32.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:32.931+0000 7f6bee34e640 1 -- 172.21.15.107:0/2291452689 wait complete. 2023-11-24T08:14:32.952 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:14:32.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:14:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.163+0000 7ff4e0c56640 1 Processor -- start 2023-11-24T08:14:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.163+0000 7ff4e0c56640 1 -- start start 2023-11-24T08:14:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.164+0000 7ff4e0c56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4dc15c7f0 0x7ff4dc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.164+0000 7ff4e0c56640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff4dc0ce6c0 0x7ff4dc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.164+0000 7ff4e0c56640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4dc0cece0 con 0x7ff4dc0cf450 2023-11-24T08:14:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.164+0000 7ff4e0c56640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0cfe40 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.164+0000 7ff4e0c56640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0ceaf0 con 0x7ff4dc0ce6c0 2023-11-24T08:14:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4dc15c7f0 0x7ff4dc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4dc15c7f0 0x7ff4dc06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36550/0 (socket says 172.21.15.107:36550) 2023-11-24T08:14:33.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 -- 172.21.15.107:0/4245168172 learned_addr learned my addr 172.21.15.107:0/4245168172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4dad9d640 1 --2- 172.21.15.107:0/4245168172 >> v2:172.21.15.120:3300/0 conn(0x7ff4dc0ce6c0 0x7ff4dc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 -- 172.21.15.107:0/4245168172 >> v2:172.21.15.120:3300/0 conn(0x7ff4dc0ce6c0 msgr2=0x7ff4dc0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 --2- 172.21.15.107:0/4245168172 >> v2:172.21.15.120:3300/0 conn(0x7ff4dc0ce6c0 0x7ff4dc0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 -- 172.21.15.107:0/4245168172 >> v1:172.21.15.120:6789/0 conn(0x7ff4dc0cf450 legacy=0x7ff4dc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.165+0000 7ff4da59c640 1 -- 172.21.15.107:0/4245168172 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc06d040 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.166+0000 7ff4da59c640 1 --2- 172.21.15.107:0/4245168172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc06db20 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0d00e0 tx=0x7ff4cc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f0ea3cd58143c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4d959a640 1 -- 172.21.15.107:0/4245168172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc02cb20 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4d959a640 1 -- 172.21.15.107:0/4245168172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc02cc80 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4d959a640 1 -- 172.21.15.107:0/4245168172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc0343f0 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 -- 172.21.15.107:0/4245168172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 msgr2=0x7ff4dc06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/4245168172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc06db20 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0d00e0 tx=0x7ff4cc02bf80 comp rx=0 tx=0).stop 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 -- 172.21.15.107:0/4245168172 shutdown_connections 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/4245168172 >> v2:172.21.15.120:3300/0 conn(0x7ff4dc0ce6c0 0x7ff4dc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/4245168172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc06db20 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 -- 172.21.15.107:0/4245168172 >> 172.21.15.107:0/4245168172 conn(0x7ff4dc06a920 msgr2=0x7ff4dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 -- 172.21.15.107:0/4245168172 shutdown_connections 2023-11-24T08:14:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.167+0000 7ff4e0c56640 1 -- 172.21.15.107:0/4245168172 wait complete. 2023-11-24T08:14:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.169+0000 7ff4e0c56640 1 Processor -- start 2023-11-24T08:14:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.169+0000 7ff4e0c56640 1 -- start start 2023-11-24T08:14:33.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4dc0ce6c0 0x7ff4dc155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4dc0cf450 0x7ff4dc0c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0c9670 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0c9950 con 0x7ff4dc0cf450 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4e0c56640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0c9c30 con 0x7ff4dc0ce6c0 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.170+0000 7ff4dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36566/0 (socket says 172.21.15.107:36566) 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 -- 172.21.15.107:0/138516322 learned_addr learned my addr 172.21.15.107:0/138516322 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 -- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4dc0ce6c0 msgr2=0x7ff4dc155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4dc0ce6c0 0x7ff4dc155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 -- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4dc0cf450 msgr2=0x7ff4dc0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4dc0cf450 0x7ff4dc0c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 -- 172.21.15.107:0/138516322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc009b30 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4dad9d640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400d580 tx=0x7ff4d400da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40125f0 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4012770 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4012b20 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.171+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc0c9f10 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.172+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc0c92b0 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.173+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc0c9440 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.174+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7ff4d4012c80 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.175+0000 7ff4caffd640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 0x7ff4bc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.175+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4d40886f0 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.175+0000 7ff4da59c640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 0x7ff4bc06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.176+0000 7ff4da59c640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 0x7ff4bc06e500 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc035000 tx=0x7ff4cc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.179+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4d40bee90 con 0x7ff4dc15c7f0 2023-11-24T08:14:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.365+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4dc073bc0 con 0x7ff4bc06c040 2023-11-24T08:14:33.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.387+0000 7ff4caffd640 1 -- 172.21.15.107:0/138516322 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282233 (secure 0 0 0) 0x7ff4dc073bc0 con 0x7ff4bc06c040 2023-11-24T08:14:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:33.395 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:14:33.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 msgr2=0x7ff4bc06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 0x7ff4bc06e500 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc035000 tx=0x7ff4cc02e000 comp rx=0 tx=0).stop 2023-11-24T08:14:33.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 msgr2=0x7ff4dc0ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400d580 tx=0x7ff4d400da40 comp rx=0 tx=0).stop 2023-11-24T08:14:33.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 shutdown_connections 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4bc06c040 0x7ff4bc06e500 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4dc15c7f0 0x7ff4dc0ad010 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4dc0cf450 0x7ff4dc0c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 --2- 172.21.15.107:0/138516322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4dc0ce6c0 0x7ff4dc155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 >> 172.21.15.107:0/138516322 conn(0x7ff4dc06a920 msgr2=0x7ff4dc0720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.394+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 shutdown_connections 2023-11-24T08:14:33.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.395+0000 7ff4e0c56640 1 -- 172.21.15.107:0/138516322 wait complete. 2023-11-24T08:14:33.415 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2023-11-24T08:14:31.615950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283960,"kb_used_data":13700,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754690760,"statfs":{"total":773094113280,"available":772803338240,"internally_reserved":0,"allocated":14028800,"data_stored":6766031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":276614718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.660341"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.985011+0000","last_change":"2023-11-24T08:14:27.985011+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.985011+0000","last_undegraded":"2023-11-24T08:14:27.985011+0000","last_fullsized":"2023-11-24T08:14:27.985011+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983631+0000","last_change":"2023-11-24T08:14:27.983631+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983631+0000","last_undegraded":"2023-11-24T08:14:27.983631+0000","last_fullsized":"2023-11-24T08:14:27.983631+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.988864+0000","last_change":"2023-11-24T08:14:27.988864+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.988864+0000","last_undegraded":"2023-11-24T08:14:27.988864+0000","last_fullsized":"2023-11-24T08:14:27.988864+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984361+0000","last_change":"2023-11-24T08:14:27.984361+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984361+0000","last_undegraded":"2023-11-24T08:14:27.984361+0000","last_fullsized":"2023-11-24T08:14:27.984361+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.992919+0000","last_change":"2023-11-24T08:14:27.992919+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.992919+0000","last_undegraded":"2023-11-24T08:14:27.992919+0000","last_fullsized":"2023-11-24T08:14:27.992919+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:28.002587+0000","last_change":"2023-11-24T08:14:28.002587+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:28.002587+0000","last_undegraded":"2023-11-24T08:14:28.002587+0000","last_fullsized":"2023-11-24T08:14:28.002587+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.997279+0000","last_change":"2023-11-24T08:14:27.997279+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.997279+0000","last_undegraded":"2023-11-24T08:14:27.997279+0000","last_fullsized":"2023-11-24T08:14:27.997279+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.986278+0000","last_change":"2023-11-24T08:14:27.986278+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.986278+0000","last_undegraded":"2023-11-24T08:14:27.986278+0000","last_fullsized":"2023-11-24T08:14:27.986278+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.988140+0000","last_change":"2023-11-24T08:14:27.988140+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.988140+0000","last_undegraded":"2023-11-24T08:14:27.988140+0000","last_fullsized":"2023-11-24T08:14:27.988140+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.980966+0000","last_change":"2023-11-24T08:14:27.980966+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.980966+0000","last_undegraded":"2023-11-24T08:14:27.980966+0000","last_fullsized":"2023-11-24T08:14:27.980966+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.987821+0000","last_change":"2023-11-24T08:14:27.987821+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.987821+0000","last_undegraded":"2023-11-24T08:14:27.987821+0000","last_fullsized":"2023-11-24T08:14:27.987821+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983978+0000","last_change":"2023-11-24T08:14:27.983978+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983978+0000","last_undegraded":"2023-11-24T08:14:27.983978+0000","last_fullsized":"2023-11-24T08:14:27.983978+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983868+0000","last_change":"2023-11-24T08:14:27.983868+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983868+0000","last_undegraded":"2023-11-24T08:14:27.983868+0000","last_fullsized":"2023-11-24T08:14:27.983868+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.980957+0000","last_change":"2023-11-24T08:14:27.980957+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.980957+0000","last_undegraded":"2023-11-24T08:14:27.980957+0000","last_fullsized":"2023-11-24T08:14:27.980957+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.985264+0000","last_change":"2023-11-24T08:14:27.985264+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.985264+0000","last_undegraded":"2023-11-24T08:14:27.985264+0000","last_fullsized":"2023-11-24T08:14:27.985264+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983458+0000","last_change":"2023-11-24T08:14:27.983458+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983458+0000","last_undegraded":"2023-11-24T08:14:27.983458+0000","last_fullsized":"2023-11-24T08:14:27.983458+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:28.000009+0000","last_change":"2023-11-24T08:14:28.000009+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:28.000009+0000","last_undegraded":"2023-11-24T08:14:28.000009+0000","last_fullsized":"2023-11-24T08:14:28.000009+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.989978+0000","last_change":"2023-11-24T08:14:27.989978+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.989978+0000","last_undegraded":"2023-11-24T08:14:27.989978+0000","last_fullsized":"2023-11-24T08:14:27.989978+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.987342+0000","last_change":"2023-11-24T08:14:27.987342+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.987342+0000","last_undegraded":"2023-11-24T08:14:27.987342+0000","last_fullsized":"2023-11-24T08:14:27.987342+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984705+0000","last_change":"2023-11-24T08:14:27.984705+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984705+0000","last_undegraded":"2023-11-24T08:14:27.984705+0000","last_fullsized":"2023-11-24T08:14:27.984705+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.997967+0000","last_change":"2023-11-24T08:14:27.997967+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.997967+0000","last_undegraded":"2023-11-24T08:14:27.997967+0000","last_fullsized":"2023-11-24T08:14:27.997967+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.982284+0000","last_change":"2023-11-24T08:14:27.982284+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.982284+0000","last_undegraded":"2023-11-24T08:14:27.982284+0000","last_fullsized":"2023-11-24T08:14:27.982284+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983619+0000","last_change":"2023-11-24T08:14:27.983619+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983619+0000","last_undegraded":"2023-11-24T08:14:27.983619+0000","last_fullsized":"2023-11-24T08:14:27.983619+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.982655+0000","last_change":"2023-11-24T08:14:27.982655+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.982655+0000","last_undegraded":"2023-11-24T08:14:27.982655+0000","last_fullsized":"2023-11-24T08:14:27.982655+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.981226+0000","last_change":"2023-11-24T08:14:27.981226+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.981226+0000","last_undegraded":"2023-11-24T08:14:27.981226+0000","last_fullsized":"2023-11-24T08:14:27.981226+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.989104+0000","last_change":"2023-11-24T08:14:27.989104+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.989104+0000","last_undegraded":"2023-11-24T08:14:27.989104+0000","last_fullsized":"2023-11-24T08:14:27.989104+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984473+0000","last_change":"2023-11-24T08:14:27.984473+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984473+0000","last_undegraded":"2023-11-24T08:14:27.984473+0000","last_fullsized":"2023-11-24T08:14:27.984473+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.992630+0000","last_change":"2023-11-24T08:14:27.992630+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.992630+0000","last_undegraded":"2023-11-24T08:14:27.992630+0000","last_fullsized":"2023-11-24T08:14:27.992630+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983444+0000","last_change":"2023-11-24T08:14:27.983444+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983444+0000","last_undegraded":"2023-11-24T08:14:27.983444+0000","last_fullsized":"2023-11-24T08:14:27.983444+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983312+0000","last_change":"2023-11-24T08:14:27.983312+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983312+0000","last_undegraded":"2023-11-24T08:14:27.983312+0000","last_fullsized":"2023-11-24T08:14:27.983312+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.988754+0000","last_change":"2023-11-24T08:14:27.988754+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.988754+0000","last_undegraded":"2023-11-24T08:14:27.988754+0000","last_fullsized":"2023-11-24T08:14:27.988754+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:28.003658+0000","last_change":"2023-11-24T08:14:28.003658+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:28.003658+0000","last_undegraded":"2023-11-24T08:14:28.003658+0000","last_fullsized":"2023-11-24T08:14:28.003658+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.985245+0000","last_change":"2023-11-24T08:14:27.985245+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.985245+0000","last_undegraded":"2023-11-24T08:14:27.985245+0000","last_fullsized":"2023-11-24T08:14:27.985245+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.011024+0000","last_change":"2023-11-24T08:14:28.007836+0000","last_active":"2023-11-24T08:14:28.011024+0000","last_peered":"2023-11-24T08:14:28.011024+0000","last_clean":"2023-11-24T08:14:28.011024+0000","last_became_active":"2023-11-24T08:14:28.007500+0000","last_became_peered":"2023-11-24T08:14:28.007500+0000","last_unstale":"2023-11-24T08:14:28.011024+0000","last_undegraded":"2023-11-24T08:14:28.011024+0000","last_fullsized":"2023-11-24T08:14:28.011024+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:29:30.271202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984716+0000","last_change":"2023-11-24T08:14:27.984716+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984716+0000","last_undegraded":"2023-11-24T08:14:27.984716+0000","last_fullsized":"2023-11-24T08:14:27.984716+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.010542+0000","last_change":"2023-11-24T08:14:28.003938+0000","last_active":"2023-11-24T08:14:28.010542+0000","last_peered":"2023-11-24T08:14:28.010542+0000","last_clean":"2023-11-24T08:14:28.010542+0000","last_became_active":"2023-11-24T08:14:28.001142+0000","last_became_peered":"2023-11-24T08:14:28.001142+0000","last_unstale":"2023-11-24T08:14:28.010542+0000","last_undegraded":"2023-11-24T08:14:28.010542+0000","last_fullsized":"2023-11-24T08:14:28.010542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:43.588181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.990989+0000","last_change":"2023-11-24T08:14:27.990989+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.990989+0000","last_undegraded":"2023-11-24T08:14:27.990989+0000","last_fullsized":"2023-11-24T08:14:27.990989+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:27.995471+0000","last_change":"2023-11-24T08:14:27.993105+0000","last_active":"2023-11-24T08:14:27.995471+0000","last_peered":"2023-11-24T08:14:27.995471+0000","last_clean":"2023-11-24T08:14:27.995471+0000","last_became_active":"2023-11-24T08:14:27.992854+0000","last_became_peered":"2023-11-24T08:14:27.992854+0000","last_unstale":"2023-11-24T08:14:27.995471+0000","last_undegraded":"2023-11-24T08:14:27.995471+0000","last_fullsized":"2023-11-24T08:14:27.995471+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:53:36.500941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.993124+0000","last_change":"2023-11-24T08:14:27.993124+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.993124+0000","last_undegraded":"2023-11-24T08:14:27.993124+0000","last_fullsized":"2023-11-24T08:14:27.993124+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.003382+0000","last_change":"2023-11-24T08:14:27.996245+0000","last_active":"2023-11-24T08:14:28.003382+0000","last_peered":"2023-11-24T08:14:28.003382+0000","last_clean":"2023-11-24T08:14:28.003382+0000","last_became_active":"2023-11-24T08:14:27.995915+0000","last_became_peered":"2023-11-24T08:14:27.995915+0000","last_unstale":"2023-11-24T08:14:28.003382+0000","last_undegraded":"2023-11-24T08:14:28.003382+0000","last_fullsized":"2023-11-24T08:14:28.003382+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:38.506418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.988324+0000","last_change":"2023-11-24T08:14:27.988324+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.988324+0000","last_undegraded":"2023-11-24T08:14:27.988324+0000","last_fullsized":"2023-11-24T08:14:27.988324+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:27.997384+0000","last_change":"2023-11-24T08:14:27.993128+0000","last_active":"2023-11-24T08:14:27.997384+0000","last_peered":"2023-11-24T08:14:27.997384+0000","last_clean":"2023-11-24T08:14:27.997384+0000","last_became_active":"2023-11-24T08:14:27.992403+0000","last_became_peered":"2023-11-24T08:14:27.992403+0000","last_unstale":"2023-11-24T08:14:27.997384+0000","last_undegraded":"2023-11-24T08:14:27.997384+0000","last_fullsized":"2023-11-24T08:14:27.997384+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:56.373096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:28.001564+0000","last_change":"2023-11-24T08:14:28.001564+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:28.001564+0000","last_undegraded":"2023-11-24T08:14:28.001564+0000","last_fullsized":"2023-11-24T08:14:28.001564+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:27.995162+0000","last_change":"2023-11-24T08:14:27.992627+0000","last_active":"2023-11-24T08:14:27.995162+0000","last_peered":"2023-11-24T08:14:27.995162+0000","last_clean":"2023-11-24T08:14:27.995162+0000","last_became_active":"2023-11-24T08:14:27.992364+0000","last_became_peered":"2023-11-24T08:14:27.992364+0000","last_unstale":"2023-11-24T08:14:27.995162+0000","last_undegraded":"2023-11-24T08:14:27.995162+0000","last_fullsized":"2023-11-24T08:14:27.995162+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:22:46.164058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.985258+0000","last_change":"2023-11-24T08:14:27.985258+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.985258+0000","last_undegraded":"2023-11-24T08:14:27.985258+0000","last_fullsized":"2023-11-24T08:14:27.985258+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.011469+0000","last_change":"2023-11-24T08:14:28.008075+0000","last_active":"2023-11-24T08:14:28.011469+0000","last_peered":"2023-11-24T08:14:28.011469+0000","last_clean":"2023-11-24T08:14:28.011469+0000","last_became_active":"2023-11-24T08:14:28.007469+0000","last_became_peered":"2023-11-24T08:14:28.007469+0000","last_unstale":"2023-11-24T08:14:28.011469+0000","last_undegraded":"2023-11-24T08:14:28.011469+0000","last_fullsized":"2023-11-24T08:14:28.011469+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:53.960657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983769+0000","last_change":"2023-11-24T08:14:27.983769+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983769+0000","last_undegraded":"2023-11-24T08:14:27.983769+0000","last_fullsized":"2023-11-24T08:14:27.983769+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.003498+0000","last_change":"2023-11-24T08:14:27.996516+0000","last_active":"2023-11-24T08:14:28.003498+0000","last_peered":"2023-11-24T08:14:28.003498+0000","last_clean":"2023-11-24T08:14:28.003498+0000","last_became_active":"2023-11-24T08:14:27.996247+0000","last_became_peered":"2023-11-24T08:14:27.996247+0000","last_unstale":"2023-11-24T08:14:28.003498+0000","last_undegraded":"2023-11-24T08:14:28.003498+0000","last_fullsized":"2023-11-24T08:14:28.003498+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:25:00.793468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984819+0000","last_change":"2023-11-24T08:14:27.984819+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984819+0000","last_undegraded":"2023-11-24T08:14:27.984819+0000","last_fullsized":"2023-11-24T08:14:27.984819+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.010582+0000","last_change":"2023-11-24T08:14:28.004957+0000","last_active":"2023-11-24T08:14:28.010582+0000","last_peered":"2023-11-24T08:14:28.010582+0000","last_clean":"2023-11-24T08:14:28.010582+0000","last_became_active":"2023-11-24T08:14:28.004639+0000","last_became_peered":"2023-11-24T08:14:28.004639+0000","last_unstale":"2023-11-24T08:14:28.010582+0000","last_undegraded":"2023-11-24T08:14:28.010582+0000","last_fullsized":"2023-11-24T08:14:28.010582+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:34:08.313014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.995281+0000","last_change":"2023-11-24T08:14:27.995281+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.995281+0000","last_undegraded":"2023-11-24T08:14:27.995281+0000","last_fullsized":"2023-11-24T08:14:27.995281+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.009663+0000","last_change":"2023-11-24T08:14:28.006763+0000","last_active":"2023-11-24T08:14:28.009663+0000","last_peered":"2023-11-24T08:14:28.009663+0000","last_clean":"2023-11-24T08:14:28.009663+0000","last_became_active":"2023-11-24T08:14:28.006477+0000","last_became_peered":"2023-11-24T08:14:28.006477+0000","last_unstale":"2023-11-24T08:14:28.009663+0000","last_undegraded":"2023-11-24T08:14:28.009663+0000","last_fullsized":"2023-11-24T08:14:28.009663+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:12:46.540300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.989967+0000","last_change":"2023-11-24T08:14:27.989967+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.989967+0000","last_undegraded":"2023-11-24T08:14:27.989967+0000","last_fullsized":"2023-11-24T08:14:27.989967+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.006300+0000","last_change":"2023-11-24T08:14:28.000140+0000","last_active":"2023-11-24T08:14:28.006300+0000","last_peered":"2023-11-24T08:14:28.006300+0000","last_clean":"2023-11-24T08:14:28.006300+0000","last_became_active":"2023-11-24T08:14:27.999821+0000","last_became_peered":"2023-11-24T08:14:27.999821+0000","last_unstale":"2023-11-24T08:14:28.006300+0000","last_undegraded":"2023-11-24T08:14:28.006300+0000","last_fullsized":"2023-11-24T08:14:28.006300+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:22:23.733282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.992150+0000","last_change":"2023-11-24T08:14:27.992150+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.992150+0000","last_undegraded":"2023-11-24T08:14:27.992150+0000","last_fullsized":"2023-11-24T08:14:27.992150+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.013960+0000","last_change":"2023-11-24T08:14:28.009062+0000","last_active":"2023-11-24T08:14:28.013960+0000","last_peered":"2023-11-24T08:14:28.013960+0000","last_clean":"2023-11-24T08:14:28.013960+0000","last_became_active":"2023-11-24T08:14:28.008166+0000","last_became_peered":"2023-11-24T08:14:28.008166+0000","last_unstale":"2023-11-24T08:14:28.013960+0000","last_undegraded":"2023-11-24T08:14:28.013960+0000","last_fullsized":"2023-11-24T08:14:28.013960+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:15.579143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983005+0000","last_change":"2023-11-24T08:14:27.983005+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983005+0000","last_undegraded":"2023-11-24T08:14:27.983005+0000","last_fullsized":"2023-11-24T08:14:27.983005+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.012246+0000","last_change":"2023-11-24T08:14:28.008303+0000","last_active":"2023-11-24T08:14:28.012246+0000","last_peered":"2023-11-24T08:14:28.012246+0000","last_clean":"2023-11-24T08:14:28.012246+0000","last_became_active":"2023-11-24T08:14:28.008028+0000","last_became_peered":"2023-11-24T08:14:28.008028+0000","last_unstale":"2023-11-24T08:14:28.012246+0000","last_undegraded":"2023-11-24T08:14:28.012246+0000","last_fullsized":"2023-11-24T08:14:28.012246+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:53.439432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.981757+0000","last_change":"2023-11-24T08:14:27.981757+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.981757+0000","last_undegraded":"2023-11-24T08:14:27.981757+0000","last_fullsized":"2023-11-24T08:14:27.981757+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.003538+0000","last_change":"2023-11-24T08:14:27.999783+0000","last_active":"2023-11-24T08:14:28.003538+0000","last_peered":"2023-11-24T08:14:28.003538+0000","last_clean":"2023-11-24T08:14:28.003538+0000","last_became_active":"2023-11-24T08:14:27.999360+0000","last_became_peered":"2023-11-24T08:14:27.999360+0000","last_unstale":"2023-11-24T08:14:28.003538+0000","last_undegraded":"2023-11-24T08:14:28.003538+0000","last_fullsized":"2023-11-24T08:14:28.003538+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:09:01.899664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984839+0000","last_change":"2023-11-24T08:14:27.984839+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984839+0000","last_undegraded":"2023-11-24T08:14:27.984839+0000","last_fullsized":"2023-11-24T08:14:27.984839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.005847+0000","last_change":"2023-11-24T08:14:28.001801+0000","last_active":"2023-11-24T08:14:28.005847+0000","last_peered":"2023-11-24T08:14:28.005847+0000","last_clean":"2023-11-24T08:14:28.005847+0000","last_became_active":"2023-11-24T08:14:28.001356+0000","last_became_peered":"2023-11-24T08:14:28.001356+0000","last_unstale":"2023-11-24T08:14:28.005847+0000","last_undegraded":"2023-11-24T08:14:28.005847+0000","last_fullsized":"2023-11-24T08:14:28.005847+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:15:33.825864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.986729+0000","last_change":"2023-11-24T08:14:27.986729+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.986729+0000","last_undegraded":"2023-11-24T08:14:27.986729+0000","last_fullsized":"2023-11-24T08:14:27.986729+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.010552+0000","last_change":"2023-11-24T08:14:28.004863+0000","last_active":"2023-11-24T08:14:28.010552+0000","last_peered":"2023-11-24T08:14:28.010552+0000","last_clean":"2023-11-24T08:14:28.010552+0000","last_became_active":"2023-11-24T08:14:28.004582+0000","last_became_peered":"2023-11-24T08:14:28.004582+0000","last_unstale":"2023-11-24T08:14:28.010552+0000","last_undegraded":"2023-11-24T08:14:28.010552+0000","last_fullsized":"2023-11-24T08:14:28.010552+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:03:58.182988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.994612+0000","last_change":"2023-11-24T08:14:27.994612+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.994612+0000","last_undegraded":"2023-11-24T08:14:27.994612+0000","last_fullsized":"2023-11-24T08:14:27.994612+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.007244+0000","last_change":"2023-11-24T08:14:28.005860+0000","last_active":"2023-11-24T08:14:28.007244+0000","last_peered":"2023-11-24T08:14:28.007244+0000","last_clean":"2023-11-24T08:14:28.007244+0000","last_became_active":"2023-11-24T08:14:28.004165+0000","last_became_peered":"2023-11-24T08:14:28.004165+0000","last_unstale":"2023-11-24T08:14:28.007244+0000","last_undegraded":"2023-11-24T08:14:28.007244+0000","last_fullsized":"2023-11-24T08:14:28.007244+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:05:51.357623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.984219+0000","last_change":"2023-11-24T08:14:27.984219+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.984219+0000","last_undegraded":"2023-11-24T08:14:27.984219+0000","last_fullsized":"2023-11-24T08:14:27.984219+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.009234+0000","last_change":"2023-11-24T08:14:28.003777+0000","last_active":"2023-11-24T08:14:28.009234+0000","last_peered":"2023-11-24T08:14:28.009234+0000","last_clean":"2023-11-24T08:14:28.009234+0000","last_became_active":"2023-11-24T08:14:28.003459+0000","last_became_peered":"2023-11-24T08:14:28.003459+0000","last_unstale":"2023-11-24T08:14:28.009234+0000","last_undegraded":"2023-11-24T08:14:28.009234+0000","last_fullsized":"2023-11-24T08:14:28.009234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:17.424610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.981398+0000","last_change":"2023-11-24T08:14:27.981398+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.981398+0000","last_undegraded":"2023-11-24T08:14:27.981398+0000","last_fullsized":"2023-11-24T08:14:27.981398+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.006775+0000","last_change":"2023-11-24T08:14:28.001321+0000","last_active":"2023-11-24T08:14:28.006775+0000","last_peered":"2023-11-24T08:14:28.006775+0000","last_clean":"2023-11-24T08:14:28.006775+0000","last_became_active":"2023-11-24T08:14:28.001049+0000","last_became_peered":"2023-11-24T08:14:28.001049+0000","last_unstale":"2023-11-24T08:14:28.006775+0000","last_undegraded":"2023-11-24T08:14:28.006775+0000","last_fullsized":"2023-11-24T08:14:28.006775+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:04.148702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.982844+0000","last_change":"2023-11-24T08:14:27.982844+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.982844+0000","last_undegraded":"2023-11-24T08:14:27.982844+0000","last_fullsized":"2023-11-24T08:14:27.982844+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.010579+0000","last_change":"2023-11-24T08:14:28.004913+0000","last_active":"2023-11-24T08:14:28.010579+0000","last_peered":"2023-11-24T08:14:28.010579+0000","last_clean":"2023-11-24T08:14:28.010579+0000","last_became_active":"2023-11-24T08:14:28.004565+0000","last_became_peered":"2023-11-24T08:14:28.004565+0000","last_unstale":"2023-11-24T08:14:28.010579+0000","last_undegraded":"2023-11-24T08:14:28.010579+0000","last_fullsized":"2023-11-24T08:14:28.010579+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:44.560419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.997154+0000","last_change":"2023-11-24T08:14:27.997154+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.997154+0000","last_undegraded":"2023-11-24T08:14:27.997154+0000","last_fullsized":"2023-11-24T08:14:27.997154+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.012322+0000","last_change":"2023-11-24T08:14:28.007712+0000","last_active":"2023-11-24T08:14:28.012322+0000","last_peered":"2023-11-24T08:14:28.012322+0000","last_clean":"2023-11-24T08:14:28.012322+0000","last_became_active":"2023-11-24T08:14:28.007446+0000","last_became_peered":"2023-11-24T08:14:28.007446+0000","last_unstale":"2023-11-24T08:14:28.012322+0000","last_undegraded":"2023-11-24T08:14:28.012322+0000","last_fullsized":"2023-11-24T08:14:28.012322+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:59:34.744501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.981112+0000","last_change":"2023-11-24T08:14:27.981112+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.981112+0000","last_undegraded":"2023-11-24T08:14:27.981112+0000","last_fullsized":"2023-11-24T08:14:27.981112+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.004035+0000","last_change":"2023-11-24T08:14:28.001680+0000","last_active":"2023-11-24T08:14:28.004035+0000","last_peered":"2023-11-24T08:14:28.004035+0000","last_clean":"2023-11-24T08:14:28.004035+0000","last_became_active":"2023-11-24T08:14:28.001431+0000","last_became_peered":"2023-11-24T08:14:28.001431+0000","last_unstale":"2023-11-24T08:14:28.004035+0000","last_undegraded":"2023-11-24T08:14:28.004035+0000","last_fullsized":"2023-11-24T08:14:28.004035+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:56.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.989910+0000","last_change":"2023-11-24T08:14:27.989910+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.989910+0000","last_undegraded":"2023-11-24T08:14:27.989910+0000","last_fullsized":"2023-11-24T08:14:27.989910+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.007216+0000","last_change":"2023-11-24T08:14:28.005732+0000","last_active":"2023-11-24T08:14:28.007216+0000","last_peered":"2023-11-24T08:14:28.007216+0000","last_clean":"2023-11-24T08:14:28.007216+0000","last_became_active":"2023-11-24T08:14:28.004637+0000","last_became_peered":"2023-11-24T08:14:28.004637+0000","last_unstale":"2023-11-24T08:14:28.007216+0000","last_undegraded":"2023-11-24T08:14:28.007216+0000","last_fullsized":"2023-11-24T08:14:28.007216+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:39:06.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:28.000915+0000","last_change":"2023-11-24T08:14:28.000915+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:28.000915+0000","last_undegraded":"2023-11-24T08:14:28.000915+0000","last_fullsized":"2023-11-24T08:14:28.000915+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.005978+0000","last_change":"2023-11-24T08:14:28.001910+0000","last_active":"2023-11-24T08:14:28.005978+0000","last_peered":"2023-11-24T08:14:28.005978+0000","last_clean":"2023-11-24T08:14:28.005978+0000","last_became_active":"2023-11-24T08:14:28.001250+0000","last_became_peered":"2023-11-24T08:14:28.001250+0000","last_unstale":"2023-11-24T08:14:28.005978+0000","last_undegraded":"2023-11-24T08:14:28.005978+0000","last_fullsized":"2023-11-24T08:14:28.005978+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:11:01.537009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.985782+0000","last_change":"2023-11-24T08:14:27.985782+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.985782+0000","last_undegraded":"2023-11-24T08:14:27.985782+0000","last_fullsized":"2023-11-24T08:14:27.985782+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.013872+0000","last_change":"2023-11-24T08:14:28.010054+0000","last_active":"2023-11-24T08:14:28.013872+0000","last_peered":"2023-11-24T08:14:28.013872+0000","last_clean":"2023-11-24T08:14:28.013872+0000","last_became_active":"2023-11-24T08:14:28.009808+0000","last_became_peered":"2023-11-24T08:14:28.009808+0000","last_unstale":"2023-11-24T08:14:28.013872+0000","last_undegraded":"2023-11-24T08:14:28.013872+0000","last_fullsized":"2023-11-24T08:14:28.013872+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:30:08.431761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.980740+0000","last_change":"2023-11-24T08:14:27.980740+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.980740+0000","last_undegraded":"2023-11-24T08:14:27.980740+0000","last_fullsized":"2023-11-24T08:14:27.980740+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.002718+0000","last_change":"2023-11-24T08:14:27.994656+0000","last_active":"2023-11-24T08:14:28.002718+0000","last_peered":"2023-11-24T08:14:28.002718+0000","last_clean":"2023-11-24T08:14:28.002718+0000","last_became_active":"2023-11-24T08:14:27.994381+0000","last_became_peered":"2023-11-24T08:14:27.994381+0000","last_unstale":"2023-11-24T08:14:28.002718+0000","last_undegraded":"2023-11-24T08:14:28.002718+0000","last_fullsized":"2023-11-24T08:14:28.002718+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:22:50.103708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.997611+0000","last_change":"2023-11-24T08:14:27.997611+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.997611+0000","last_undegraded":"2023-11-24T08:14:27.997611+0000","last_fullsized":"2023-11-24T08:14:27.997611+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:27.999902+0000","last_change":"2023-11-24T08:14:27.994554+0000","last_active":"2023-11-24T08:14:27.999902+0000","last_peered":"2023-11-24T08:14:27.999902+0000","last_clean":"2023-11-24T08:14:27.999902+0000","last_became_active":"2023-11-24T08:14:27.994294+0000","last_became_peered":"2023-11-24T08:14:27.994294+0000","last_unstale":"2023-11-24T08:14:27.999902+0000","last_undegraded":"2023-11-24T08:14:27.999902+0000","last_fullsized":"2023-11-24T08:14:27.999902+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:36:55.191757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.987395+0000","last_change":"2023-11-24T08:14:27.987395+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.987395+0000","last_undegraded":"2023-11-24T08:14:27.987395+0000","last_fullsized":"2023-11-24T08:14:27.987395+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.005890+0000","last_change":"2023-11-24T08:14:28.004634+0000","last_active":"2023-11-24T08:14:28.005890+0000","last_peered":"2023-11-24T08:14:28.005890+0000","last_clean":"2023-11-24T08:14:28.005890+0000","last_became_active":"2023-11-24T08:14:28.004254+0000","last_became_peered":"2023-11-24T08:14:28.004254+0000","last_unstale":"2023-11-24T08:14:28.005890+0000","last_undegraded":"2023-11-24T08:14:28.005890+0000","last_fullsized":"2023-11-24T08:14:28.005890+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:55.927518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.990835+0000","last_change":"2023-11-24T08:14:27.990835+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.990835+0000","last_undegraded":"2023-11-24T08:14:27.990835+0000","last_fullsized":"2023-11-24T08:14:27.990835+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.013810+0000","last_change":"2023-11-24T08:14:28.009103+0000","last_active":"2023-11-24T08:14:28.013810+0000","last_peered":"2023-11-24T08:14:28.013810+0000","last_clean":"2023-11-24T08:14:28.013810+0000","last_became_active":"2023-11-24T08:14:28.008680+0000","last_became_peered":"2023-11-24T08:14:28.008680+0000","last_unstale":"2023-11-24T08:14:28.013810+0000","last_undegraded":"2023-11-24T08:14:28.013810+0000","last_fullsized":"2023-11-24T08:14:28.013810+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:40.490082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.982839+0000","last_change":"2023-11-24T08:14:27.982839+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.982839+0000","last_undegraded":"2023-11-24T08:14:27.982839+0000","last_fullsized":"2023-11-24T08:14:27.982839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.007327+0000","last_change":"2023-11-24T08:14:28.005030+0000","last_active":"2023-11-24T08:14:28.007327+0000","last_peered":"2023-11-24T08:14:28.007327+0000","last_clean":"2023-11-24T08:14:28.007327+0000","last_became_active":"2023-11-24T08:14:28.004745+0000","last_became_peered":"2023-11-24T08:14:28.004745+0000","last_unstale":"2023-11-24T08:14:28.007327+0000","last_undegraded":"2023-11-24T08:14:28.007327+0000","last_fullsized":"2023-11-24T08:14:28.007327+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:24.733828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.983356+0000","last_change":"2023-11-24T08:14:27.983356+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.983356+0000","last_undegraded":"2023-11-24T08:14:27.983356+0000","last_fullsized":"2023-11-24T08:14:27.983356+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.009782+0000","last_change":"2023-11-24T08:14:28.006409+0000","last_active":"2023-11-24T08:14:28.009782+0000","last_peered":"2023-11-24T08:14:28.009782+0000","last_clean":"2023-11-24T08:14:28.009782+0000","last_became_active":"2023-11-24T08:14:28.006014+0000","last_became_peered":"2023-11-24T08:14:28.006014+0000","last_unstale":"2023-11-24T08:14:28.009782+0000","last_undegraded":"2023-11-24T08:14:28.009782+0000","last_fullsized":"2023-11-24T08:14:28.009782+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:25:36.583711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":129,"state":"creating+peering","last_fresh":"2023-11-24T08:14:27.990796+0000","last_change":"2023-11-24T08:14:27.990796+0000","last_active":"2023-11-24T08:14:27.955401+0000","last_peered":"2023-11-24T08:14:27.955401+0000","last_clean":"2023-11-24T08:14:27.955401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T08:14:27.990796+0000","last_undegraded":"2023-11-24T08:14:27.990796+0000","last_fullsized":"2023-11-24T08:14:27.990796+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.013817+0000","last_change":"2023-11-24T08:14:28.009227+0000","last_active":"2023-11-24T08:14:28.013817+0000","last_peered":"2023-11-24T08:14:28.013817+0000","last_clean":"2023-11-24T08:14:28.013817+0000","last_became_active":"2023-11-24T08:14:28.008642+0000","last_became_peered":"2023-11-24T08:14:28.008642+0000","last_unstale":"2023-11-24T08:14:28.013817+0000","last_undegraded":"2023-11-24T08:14:28.013817+0000","last_fullsized":"2023-11-24T08:14:28.013817+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:28.488940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":259,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-11-24T08:14:26.973487+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:14:26.973487+0000","last_peered":"2023-11-24T08:14:26.973487+0000","last_clean":"2023-11-24T08:14:26.973487+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:14:26.973487+0000","last_undegraded":"2023-11-24T08:14:26.973487+0000","last_fullsized":"2023-11-24T08:14:26.973487+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574969,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1290883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31914444},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0780000000000001,"15min":1.03},"min":{"1min":0.65600000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.3859999999999999,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0820000000000001,"15min":1.0860000000000001},"min":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.3120000000000001,"5min":2.3340000000000001,"15min":2.7109999999999999},"last":1.0660000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.157,"15min":1.1419999999999999},"min":{"1min":0.78400000000000003,"5min":0.64700000000000002,"15min":0.46300000000000002},"max":{"1min":1.5029999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.002},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2290000000000001,"15min":1.202},"min":{"1min":0.84699999999999998,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.601,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.177}]},{"osd":2,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1850000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.58699999999999997},"max":{"1min":1.774,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":1.371},{"interface":"front","average":{"1min":1.246,"5min":1.3220000000000001,"15min":1.343},"min":{"1min":0.82999999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.667,"5min":4.0650000000000004,"15min":4.633},"last":1.3240000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2190000000000001,"15min":1.169},"min":{"1min":0.98499999999999999,"5min":0.79500000000000004,"15min":0.504},"max":{"1min":1.4330000000000001,"5min":3.133,"15min":3.133},"last":1.466},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.401,"15min":1.3720000000000001},"min":{"1min":0.98199999999999998,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.8009999999999999,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.1459999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3109999999999999,"15min":1.3340000000000001},"min":{"1min":0.79700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.871,"5min":2.661,"15min":2.851},"last":1.294},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":1.0669999999999999,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.6240000000000001,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.583}]},{"osd":5,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.472,"15min":1.464},"min":{"1min":0.99399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.016,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.53,"5min":1.4630000000000001,"15min":1.484},"min":{"1min":1.0289999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.1230000000000002,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.431}]},{"osd":6,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.546,"15min":1.5509999999999999},"min":{"1min":1.1950000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.3039999999999998,"5min":2.992,"15min":2.992},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4830000000000001,"15min":1.464},"min":{"1min":1.2589999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.929,"5min":3.798,"15min":3.798},"last":1.52}]}]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1597440,"data_stored":694240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0760000000000001,"15min":1.0880000000000001},"min":{"1min":0.63700000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.377,"5min":1.95,"15min":1.95},"last":1.081},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0800000000000001,"15min":1.095},"min":{"1min":0.753,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.804,"5min":1.986,"15min":2.2069999999999999},"last":0.89000000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1719999999999999,"15min":1.2290000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5840000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1399999999999999,"15min":1.2390000000000001},"min":{"1min":0.72999999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.4039999999999999},"last":1.1970000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0920000000000001,"15min":1.117},"min":{"1min":0.76100000000000001,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.544,"5min":1.6779999999999999,"15min":1.9330000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.27,"15min":1.345},"min":{"1min":0.93600000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.615,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.6100000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.323,"15min":1.3740000000000001},"min":{"1min":0.86399999999999999,"5min":0.83399999999999996,"15min":0.71499999999999997},"max":{"1min":1.7410000000000001,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.343,"15min":1.343},"min":{"1min":0.90000000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.7949999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.3480000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4199999999999999,"15min":1.506},"min":{"1min":0.95799999999999996,"5min":0.79900000000000004,"15min":0.76500000000000001},"max":{"1min":2.0790000000000002,"5min":2.895,"15min":2.895},"last":1.708},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4690000000000001,"15min":1.538},"min":{"1min":1.165,"5min":1.056,"15min":0.90800000000000003},"max":{"1min":2.1909999999999998,"5min":2.242,"15min":2.8820000000000001},"last":1.2849999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.429,"15min":1.5189999999999999},"min":{"1min":0.91000000000000003,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":1.9730000000000001,"5min":2.1709999999999998,"15min":2.629},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.5569999999999999,"15min":1.6519999999999999},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.95,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.921}]},{"osd":7,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.5649999999999999,"15min":1.6779999999999999},"min":{"1min":1.135,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":2.125,"5min":2.125,"15min":3.0329999999999999},"last":1.651},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5820000000000001,"15min":1.6579999999999999},"min":{"1min":1.1910000000000001,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.254,"5min":2.2909999999999999,"15min":2.6890000000000001},"last":1.5589999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574968,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1609728,"data_stored":700515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.026,"15min":1.0900000000000001},"min":{"1min":0.65300000000000002,"5min":0.58599999999999997,"15min":0.58299999999999996},"max":{"1min":1.4690000000000001,"5min":1.694,"15min":3.1179999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.054,"5min":1.119,"15min":1.1619999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5960000000000001,"5min":1.859,"15min":2.198},"last":1.002}]},{"osd":1,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1399999999999999,"15min":1.1559999999999999},"min":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.5620000000000001,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1970000000000001,"15min":1.1579999999999999},"min":{"1min":0.78500000000000003,"5min":0.73399999999999999,"15min":0.48699999999999999},"max":{"1min":1.611,"5min":1.659,"15min":1.6910000000000001},"last":1.3}]},{"osd":2,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1779999999999999,"15min":1.202},"min":{"1min":0.73299999999999998,"5min":0.65100000000000002,"15min":0.51600000000000001},"max":{"1min":1.671,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.359},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.143,"15min":1.1659999999999999},"min":{"1min":0.83799999999999997,"5min":0.72499999999999998,"15min":0.64200000000000002},"max":{"1min":1.552,"5min":1.6839999999999999,"15min":1.8129999999999999},"last":1.054}]},{"osd":3,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1240000000000001,"15min":1.1699999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.28,"15min":1.3149999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.55200000000000005},"max":{"1min":1.522,"5min":1.9730000000000001,"15min":1.99},"last":1.1299999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3080000000000001,"15min":1.3280000000000001},"min":{"1min":0.66400000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.518,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.405},{"interface":"front","average":{"1min":1.353,"5min":1.452,"15min":1.5189999999999999},"min":{"1min":0.86299999999999999,"5min":0.75600000000000001,"15min":0.46700000000000003},"max":{"1min":1.7709999999999999,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.53}]},{"osd":6,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2969999999999999,"15min":1.407},"min":{"1min":0.63600000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":3.2949999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.403,"15min":1.4259999999999999},"min":{"1min":0.71299999999999997,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":1.6739999999999999,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":1.4810000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.554},"min":{"1min":0.69899999999999995,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":3.4420000000000002},"last":1.841},{"interface":"front","average":{"1min":1.52,"5min":1.587,"15min":1.6040000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":1.869,"5min":2.181,"15min":2.181},"last":1.7410000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1597440,"data_stored":694240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.0760000000000001,"15min":1.1279999999999999},"min":{"1min":0.70399999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0620000000000001,"15min":1.083},"min":{"1min":0.68400000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.7150000000000001,"15min":2.222},"last":0.91000000000000003}]},{"osd":1,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.111,"15min":1.1499999999999999},"min":{"1min":0.73699999999999999,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.202},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1639999999999999,"15min":1.1819999999999999},"min":{"1min":0.73199999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6279999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.0229999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0740000000000001,"15min":1.135},"min":{"1min":0.80700000000000005,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.552,"5min":1.5780000000000001,"15min":2.1419999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.212,"15min":1.321},"min":{"1min":0.88300000000000001,"5min":0.505,"15min":0.505},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.0740000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.306,"15min":1.3959999999999999},"min":{"1min":0.93100000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.26,"15min":1.3049999999999999},"min":{"1min":0.92700000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6499999999999999,"5min":1.6599999999999999,"15min":1.8280000000000001},"last":1.262}]},{"osd":5,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4419999999999999,"15min":1.4970000000000001},"min":{"1min":0.71999999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.417,"15min":1.504},"min":{"1min":0.78200000000000003,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.3879999999999999},"last":1.3720000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.234,"15min":1.381},"min":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.651,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.155},{"interface":"front","average":{"1min":1.522,"5min":1.494,"15min":1.6100000000000001},"min":{"1min":0.79000000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":3.391},"last":1.5620000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4650000000000001,"15min":1.579},"min":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.9370000000000001,"5min":2.052,"15min":3.1230000000000002},"last":1.514},{"interface":"front","average":{"1min":1.397,"5min":1.387,"15min":1.528},"min":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6879999999999999,"5min":1.857,"15min":3.4849999999999999},"last":1.478}]}]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36912,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334928,"statfs":{"total":96636764160,"available":96598966272,"internally_reserved":0,"allocated":1609728,"data_stored":700515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89100000000000001,"15min":0.86099999999999999},"min":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.643,"5min":3.181,"15min":10.557},"last":1.179},{"interface":"front","average":{"1min":0.999,"5min":0.93700000000000006,"15min":0.92100000000000004},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6679999999999999,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":1.1240000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.85199999999999998,"15min":0.83699999999999997},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.538,"5min":2.101,"15min":2.101},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.99199999999999999,"15min":1.008},"min":{"1min":0.51700000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"max":{"1min":1.6859999999999999,"5min":3.3260000000000001,"15min":13.090999999999999},"last":1.3029999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.93799999999999994,"15min":0.95899999999999996},"min":{"1min":0.48099999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":1.6060000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":0.876},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.087,"15min":1.0860000000000001},"min":{"1min":0.56100000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.708,"5min":3.2480000000000002,"15min":13.460000000000001},"last":1.446}]},{"osd":4,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.282,"15min":1.2849999999999999},"min":{"1min":0.83799999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":4.367},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.22,"15min":1.29},"min":{"1min":0.94599999999999995,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.514,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.274}]},{"osd":5,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.306,"15min":1.323},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.857,"5min":3.4399999999999999,"15min":13.231},"last":1.508},{"interface":"front","average":{"1min":1.379,"5min":1.2490000000000001,"15min":1.3080000000000001},"min":{"1min":0.82199999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.956,"5min":1.956,"15min":5.1180000000000003},"last":1.24}]},{"osd":6,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.379,"15min":1.399},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.72899999999999998},"max":{"1min":1.931,"5min":3.5030000000000001,"15min":13.359999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.268,"15min":1.323},"min":{"1min":0.77800000000000002,"5min":0.73799999999999999,"15min":0.67000000000000004},"max":{"1min":2.024,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":1.3859999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.349,"15min":1.411},"min":{"1min":0.77500000000000002,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0790000000000002,"5min":2.101,"15min":4.5060000000000002},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3420000000000001,"15min":1.4199999999999999},"min":{"1min":0.87,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7969999999999999,"5min":3.331,"15min":4.032},"last":1.476}]}]},{"osd":2,"up_from":13,"seq":55834574967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36916,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334924,"statfs":{"total":96636764160,"available":96598962176,"internally_reserved":0,"allocated":1609728,"data_stored":700515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78500000000000003,"15min":0.879},"min":{"1min":0.46800000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.3500000000000001,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.90900000000000003,"15min":0.97699999999999998},"min":{"1min":0.46400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3919999999999999,"5min":1.782,"15min":2.5779999999999998},"last":0.74099999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.91900000000000004,"15min":1.0600000000000001},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.429,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.95399999999999996,"15min":1.075},"min":{"1min":0.52400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.6080000000000001,"5min":1.637,"15min":2.6240000000000001},"last":1.391}]},{"osd":3,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.84599999999999997,"15min":0.92100000000000004},"min":{"1min":0.48299999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.4410000000000001,"5min":1.552,"15min":2.2090000000000001},"last":1},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0309999999999999,"15min":1.194},"min":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.5249999999999999,"5min":1.8480000000000001,"15min":2.9950000000000001},"last":1.345}]},{"osd":4,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.375},"min":{"1min":0.72399999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":1.53},{"interface":"front","average":{"1min":1.278,"5min":1.26,"15min":1.359},"min":{"1min":0.90200000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.921,"5min":1.921,"15min":3.3279999999999998},"last":1.484}]},{"osd":5,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2749999999999999,"15min":1.3979999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.73399999999999999},"max":{"1min":1.8959999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.149},{"interface":"front","average":{"1min":1.256,"5min":1.2290000000000001,"15min":1.359},"min":{"1min":0.85199999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.7839999999999998},"last":1.704}]},{"osd":6,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.194,"15min":1.2989999999999999},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":2.1709999999999998,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.107},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.337,"15min":1.4630000000000001},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":1.944,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.4450000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.325,"15min":1.4550000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.71299999999999997},"max":{"1min":2.0760000000000001,"5min":2.1070000000000002,"15min":2.859},"last":1.599},{"interface":"front","average":{"1min":1.266,"5min":1.258,"15min":1.4179999999999999},"min":{"1min":0.86799999999999999,"5min":0.746,"15min":0.746},"max":{"1min":1.97,"5min":1.9930000000000001,"15min":3.3809999999999998},"last":1.2929999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36916,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334924,"statfs":{"total":96636764160,"available":96598962176,"internally_reserved":0,"allocated":1609728,"data_stored":700515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84299999999999997,"15min":0.78500000000000003},"min":{"1min":0.36699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.5900000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.91400000000000003,"15min":0.85899999999999999},"min":{"1min":0.44,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.5189999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.746},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.679,"5min":1.861,"15min":1.861},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.94999999999999996,"15min":0.89800000000000002},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5609999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.1100000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0329999999999999,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.92300000000000004,"15min":0.89200000000000002},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.45,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.179}]},{"osd":4,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2849999999999999,"15min":1.2529999999999999},"min":{"1min":0.88700000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.827,"5min":2.028,"15min":2.7959999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.90000000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.649,"5min":2.1080000000000001,"15min":3.0409999999999999},"last":1.0600000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.274,"15min":1.224},"min":{"1min":0.80000000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8520000000000001,"5min":2.02,"15min":3.7639999999999998},"last":1.276},{"interface":"front","average":{"1min":1.206,"5min":1.1970000000000001,"15min":1.1879999999999999},"min":{"1min":0.78100000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.246,"5min":2.246,"15min":3.286},"last":1.038}]},{"osd":6,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3180000000000001,"15min":1.2470000000000001},"min":{"1min":0.79100000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8759999999999999,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.2909999999999999,"15min":1.2270000000000001},"min":{"1min":0.72299999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.788,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.208}]},{"osd":7,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3300000000000001,"15min":1.302},"min":{"1min":0.81200000000000006,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7709999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.343},{"interface":"front","average":{"1min":1.377,"5min":1.355,"15min":1.2949999999999999},"min":{"1min":0.96199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3620000000000001,"5min":3.653,"15min":3.653},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574969,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37484,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334356,"statfs":{"total":96636764160,"available":96598380544,"internally_reserved":0,"allocated":2191360,"data_stored":1284608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71399999999999997,"15min":0.82399999999999995},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":1.425,"15min":1.6020000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.84499999999999997,"15min":0.95799999999999996},"min":{"1min":0.47499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.4610000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.97199999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.016,"15min":1.05},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.5209999999999999,"5min":1.891,"15min":1.891},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.92900000000000005},"min":{"1min":0.47499999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3799999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.044}]},{"osd":3,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.97099999999999997,"15min":1.0740000000000001},"min":{"1min":0.54800000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.456,"5min":1.79,"15min":2.0670000000000002},"last":1.3},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":1.0049999999999999},"min":{"1min":0.44900000000000001,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.4079999999999999,"5min":1.641,"15min":2.1789999999999998},"last":1.103}]},{"osd":4,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1859999999999999,"15min":1.3480000000000001},"min":{"1min":0.76400000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.724,"5min":2.3010000000000002,"15min":3.7610000000000001},"last":1.548},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.109,"15min":1.2010000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.54,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":1.349}]},{"osd":5,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.266},"min":{"1min":0.79700000000000004,"5min":0.72799999999999998,"15min":0.69699999999999995},"max":{"1min":1.5649999999999999,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.173,"15min":1.349},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.514,"5min":1.75,"15min":2.7240000000000002},"last":1.196}]},{"osd":6,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.28,"15min":1.3839999999999999},"min":{"1min":0.78000000000000003,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.714,"5min":1.986,"15min":2.4369999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.256,"15min":1.3859999999999999},"min":{"1min":0.81399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7030000000000001,"5min":1.998,"15min":2.738},"last":1.4379999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3560000000000001,"15min":1.4710000000000001},"min":{"1min":0.88,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.7410000000000001,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.3320000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.776,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.4970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:14:33.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:14:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.675+0000 7fc9fce1c640 1 Processor -- start 2023-11-24T08:14:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.675+0000 7fc9fce1c640 1 -- start start 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9f815c7f0 0x7fc9f806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cf450 0x7fc9f80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9f80cdfb0 0x7fc9f80ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f8156010 con 0x7fc9f815c7f0 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f806ce30 con 0x7fc9f80cdfb0 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9fce1c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80cfe10 con 0x7fc9f80cf450 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9f815c7f0 0x7fc9f806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9f815c7f0 0x7fc9f806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36578/0 (socket says 172.21.15.107:36578) 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 -- 172.21.15.107:0/3023321273 learned_addr learned my addr 172.21.15.107:0/3023321273 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f5d9b640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cf450 0x7fc9f80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f6d9d640 1 --2- 172.21.15.107:0/3023321273 >> v2:172.21.15.120:3300/0 conn(0x7fc9f80cdfb0 0x7fc9f80ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 -- 172.21.15.107:0/3023321273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cf450 msgr2=0x7fc9f80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cf450 0x7fc9f80cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 -- 172.21.15.107:0/3023321273 >> v2:172.21.15.120:3300/0 conn(0x7fc9f80cdfb0 msgr2=0x7fc9f80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 --2- 172.21.15.107:0/3023321273 >> v2:172.21.15.120:3300/0 conn(0x7fc9f80cdfb0 0x7fc9f80ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.677+0000 7fc9f659c640 1 -- 172.21.15.107:0/3023321273 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80cece0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.679+0000 7fc9f659c640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f806db20 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80aebc0 tx=0x7fc9e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df200493619c427f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.679+0000 7fc9f559a640 1 -- 172.21.15.107:0/3023321273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e802ca70 con 0x7fc9f815c7f0 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.679+0000 7fc9f559a640 1 -- 172.21.15.107:0/3023321273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e802cbd0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.679+0000 7fc9f559a640 1 -- 172.21.15.107:0/3023321273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80343f0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 -- 172.21.15.107:0/3023321273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 msgr2=0x7fc9f806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f806db20 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80aebc0 tx=0x7fc9e802bed0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 -- 172.21.15.107:0/3023321273 shutdown_connections 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/3023321273 >> v2:172.21.15.120:3300/0 conn(0x7fc9f80cdfb0 0x7fc9f80ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cf450 0x7fc9f80cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/3023321273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f806db20 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 -- 172.21.15.107:0/3023321273 >> 172.21.15.107:0/3023321273 conn(0x7fc9f806a920 msgr2=0x7fc9f806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 -- 172.21.15.107:0/3023321273 shutdown_connections 2023-11-24T08:14:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.680+0000 7fc9fce1c640 1 -- 172.21.15.107:0/3023321273 wait complete. 2023-11-24T08:14:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.681+0000 7fc9fce1c640 1 Processor -- start 2023-11-24T08:14:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.682+0000 7fc9fce1c640 1 -- start start 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.682+0000 7fc9fce1c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cdfb0 0x7fc9f815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9fce1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 0x7fc9f8151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9fce1c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9fce1c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b2d20 con 0x7fc9f815c7f0 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9fce1c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8072770 con 0x7fc9f80cf450 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9fce1c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8159e10 con 0x7fc9f80cdfb0 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36592/0 (socket says 172.21.15.107:36592) 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 -- 172.21.15.107:0/2829195997 learned_addr learned my addr 172.21.15.107:0/2829195997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f5d9b640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 0x7fc9f8151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f659c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cdfb0 0x7fc9f815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 -- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cdfb0 msgr2=0x7fc9f815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cdfb0 0x7fc9f815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 -- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 msgr2=0x7fc9f8151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 0x7fc9f8151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f6d9d640 1 -- 172.21.15.107:0/2829195997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8009b30 con 0x7fc9f815c7f0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.683+0000 7fc9f5d9b640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 0x7fc9f8151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9f6d9d640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000d500 tx=0x7fc9f000d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0014620 con 0x7fc9f815c7f0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f00147a0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0014b50 con 0x7fc9f815c7f0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f8159850 con 0x7fc9f815c7f0 2023-11-24T08:14:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.684+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f814a7a0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.685+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f80b2e90 con 0x7fc9f815c7f0 2023-11-24T08:14:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.686+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc9f0014cb0 con 0x7fc9f815c7f0 2023-11-24T08:14:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.688+0000 7fc9eeffd640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 0x7fc9d806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.688+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9f0088560 con 0x7fc9f815c7f0 2023-11-24T08:14:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.688+0000 7fc9f659c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 0x7fc9d806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.688+0000 7fc9f659c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 0x7fc9d806e500 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8035000 tx=0x7fc9e8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:33.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.692+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9f80b2e90 con 0x7fc9f815c7f0 2023-11-24T08:14:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.880+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9f806e3b0 con 0x7fc9d806c040 2023-11-24T08:14:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.902+0000 7fc9eeffd640 1 -- 172.21.15.107:0/2829195997 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287757 (secure 0 0 0) 0x7fc9f806e3b0 con 0x7fc9d806c040 2023-11-24T08:14:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.908+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 msgr2=0x7fc9d806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.908+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 0x7fc9d806e500 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8035000 tx=0x7fc9e8030000 comp rx=0 tx=0).stop 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.908+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 msgr2=0x7fc9f8072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.908+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000d500 tx=0x7fc9f000d9c0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 shutdown_connections 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9d806c040 0x7fc9d806e500 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f815c7f0 0x7fc9f8072230 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9f80cf450 0x7fc9f8151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 --2- 172.21.15.107:0/2829195997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9f80cdfb0 0x7fc9f815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 >> 172.21.15.107:0/2829195997 conn(0x7fc9f806a920 msgr2=0x7fc9f80aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 shutdown_connections 2023-11-24T08:14:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:33.909+0000 7fc9fce1c640 1 -- 172.21.15.107:0/2829195997 wait complete. 2023-11-24T08:14:33.930 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2023-11-24T08:14:33.616535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":288412,"kb_used_data":13992,"kb_used_omap":12,"kb_used_meta":274291,"kb_avail":754686308,"statfs":{"total":773094113280,"available":772798779392,"internally_reserved":0,"allocated":14327808,"data_stored":6894276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":280874558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001675"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570624+0000","last_change":"2023-11-24T08:14:29.006146+0000","last_active":"2023-11-24T08:14:31.570624+0000","last_peered":"2023-11-24T08:14:31.570624+0000","last_clean":"2023-11-24T08:14:31.570624+0000","last_became_active":"2023-11-24T08:14:29.005760+0000","last_became_peered":"2023-11-24T08:14:29.005760+0000","last_unstale":"2023-11-24T08:14:31.570624+0000","last_undegraded":"2023-11-24T08:14:31.570624+0000","last_fullsized":"2023-11-24T08:14:31.570624+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:03:59.200275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572457+0000","last_change":"2023-11-24T08:14:29.005768+0000","last_active":"2023-11-24T08:14:31.572457+0000","last_peered":"2023-11-24T08:14:31.572457+0000","last_clean":"2023-11-24T08:14:31.572457+0000","last_became_active":"2023-11-24T08:14:29.005511+0000","last_became_peered":"2023-11-24T08:14:29.005511+0000","last_unstale":"2023-11-24T08:14:31.572457+0000","last_undegraded":"2023-11-24T08:14:31.572457+0000","last_fullsized":"2023-11-24T08:14:31.572457+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:18.441897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571401+0000","last_change":"2023-11-24T08:14:28.999654+0000","last_active":"2023-11-24T08:14:31.571401+0000","last_peered":"2023-11-24T08:14:31.571401+0000","last_clean":"2023-11-24T08:14:31.571401+0000","last_became_active":"2023-11-24T08:14:28.999349+0000","last_became_peered":"2023-11-24T08:14:28.999349+0000","last_unstale":"2023-11-24T08:14:31.571401+0000","last_undegraded":"2023-11-24T08:14:31.571401+0000","last_fullsized":"2023-11-24T08:14:31.571401+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:34:27.146354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571919+0000","last_change":"2023-11-24T08:14:29.018095+0000","last_active":"2023-11-24T08:14:31.571919+0000","last_peered":"2023-11-24T08:14:31.571919+0000","last_clean":"2023-11-24T08:14:31.571919+0000","last_became_active":"2023-11-24T08:14:29.017829+0000","last_became_peered":"2023-11-24T08:14:29.017829+0000","last_unstale":"2023-11-24T08:14:31.571919+0000","last_undegraded":"2023-11-24T08:14:31.571919+0000","last_fullsized":"2023-11-24T08:14:31.571919+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:14:15.702479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570670+0000","last_change":"2023-11-24T08:14:28.997275+0000","last_active":"2023-11-24T08:14:31.570670+0000","last_peered":"2023-11-24T08:14:31.570670+0000","last_clean":"2023-11-24T08:14:31.570670+0000","last_became_active":"2023-11-24T08:14:28.996913+0000","last_became_peered":"2023-11-24T08:14:28.996913+0000","last_unstale":"2023-11-24T08:14:31.570670+0000","last_undegraded":"2023-11-24T08:14:31.570670+0000","last_fullsized":"2023-11-24T08:14:31.570670+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:24.700535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008943+0000","last_change":"2023-11-24T08:14:29.015122+0000","last_active":"2023-11-24T08:14:30.008943+0000","last_peered":"2023-11-24T08:14:30.008943+0000","last_clean":"2023-11-24T08:14:30.008943+0000","last_became_active":"2023-11-24T08:14:29.014821+0000","last_became_peered":"2023-11-24T08:14:29.014821+0000","last_unstale":"2023-11-24T08:14:30.008943+0000","last_undegraded":"2023-11-24T08:14:30.008943+0000","last_fullsized":"2023-11-24T08:14:30.008943+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:21:59.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009130+0000","last_change":"2023-11-24T08:14:29.007041+0000","last_active":"2023-11-24T08:14:30.009130+0000","last_peered":"2023-11-24T08:14:30.009130+0000","last_clean":"2023-11-24T08:14:30.009130+0000","last_became_active":"2023-11-24T08:14:29.006787+0000","last_became_peered":"2023-11-24T08:14:29.006787+0000","last_unstale":"2023-11-24T08:14:30.009130+0000","last_undegraded":"2023-11-24T08:14:30.009130+0000","last_fullsized":"2023-11-24T08:14:30.009130+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:00:24.270473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570346+0000","last_change":"2023-11-24T08:14:29.009238+0000","last_active":"2023-11-24T08:14:31.570346+0000","last_peered":"2023-11-24T08:14:31.570346+0000","last_clean":"2023-11-24T08:14:31.570346+0000","last_became_active":"2023-11-24T08:14:29.008929+0000","last_became_peered":"2023-11-24T08:14:29.008929+0000","last_unstale":"2023-11-24T08:14:31.570346+0000","last_undegraded":"2023-11-24T08:14:31.570346+0000","last_fullsized":"2023-11-24T08:14:31.570346+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:55.272528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571951+0000","last_change":"2023-11-24T08:14:28.995300+0000","last_active":"2023-11-24T08:14:31.571951+0000","last_peered":"2023-11-24T08:14:31.571951+0000","last_clean":"2023-11-24T08:14:31.571951+0000","last_became_active":"2023-11-24T08:14:28.995024+0000","last_became_peered":"2023-11-24T08:14:28.995024+0000","last_unstale":"2023-11-24T08:14:31.571951+0000","last_undegraded":"2023-11-24T08:14:31.571951+0000","last_fullsized":"2023-11-24T08:14:31.571951+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:04.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572673+0000","last_change":"2023-11-24T08:14:29.003601+0000","last_active":"2023-11-24T08:14:31.572673+0000","last_peered":"2023-11-24T08:14:31.572673+0000","last_clean":"2023-11-24T08:14:31.572673+0000","last_became_active":"2023-11-24T08:14:29.003065+0000","last_became_peered":"2023-11-24T08:14:29.003065+0000","last_unstale":"2023-11-24T08:14:31.572673+0000","last_undegraded":"2023-11-24T08:14:31.572673+0000","last_fullsized":"2023-11-24T08:14:31.572673+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:49.385631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568841+0000","last_change":"2023-11-24T08:14:29.004595+0000","last_active":"2023-11-24T08:14:31.568841+0000","last_peered":"2023-11-24T08:14:31.568841+0000","last_clean":"2023-11-24T08:14:31.568841+0000","last_became_active":"2023-11-24T08:14:29.004304+0000","last_became_peered":"2023-11-24T08:14:29.004304+0000","last_unstale":"2023-11-24T08:14:31.568841+0000","last_undegraded":"2023-11-24T08:14:31.568841+0000","last_fullsized":"2023-11-24T08:14:31.568841+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:11:48.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571173+0000","last_change":"2023-11-24T08:14:28.996654+0000","last_active":"2023-11-24T08:14:31.571173+0000","last_peered":"2023-11-24T08:14:31.571173+0000","last_clean":"2023-11-24T08:14:31.571173+0000","last_became_active":"2023-11-24T08:14:28.996373+0000","last_became_peered":"2023-11-24T08:14:28.996373+0000","last_unstale":"2023-11-24T08:14:31.571173+0000","last_undegraded":"2023-11-24T08:14:31.571173+0000","last_fullsized":"2023-11-24T08:14:31.571173+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:42.736994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571285+0000","last_change":"2023-11-24T08:14:29.013609+0000","last_active":"2023-11-24T08:14:31.571285+0000","last_peered":"2023-11-24T08:14:31.571285+0000","last_clean":"2023-11-24T08:14:31.571285+0000","last_became_active":"2023-11-24T08:14:29.011814+0000","last_became_peered":"2023-11-24T08:14:29.011814+0000","last_unstale":"2023-11-24T08:14:31.571285+0000","last_undegraded":"2023-11-24T08:14:31.571285+0000","last_fullsized":"2023-11-24T08:14:31.571285+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:05.165989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570847+0000","last_change":"2023-11-24T08:14:29.011203+0000","last_active":"2023-11-24T08:14:31.570847+0000","last_peered":"2023-11-24T08:14:31.570847+0000","last_clean":"2023-11-24T08:14:31.570847+0000","last_became_active":"2023-11-24T08:14:29.010779+0000","last_became_peered":"2023-11-24T08:14:29.010779+0000","last_unstale":"2023-11-24T08:14:31.570847+0000","last_undegraded":"2023-11-24T08:14:31.570847+0000","last_fullsized":"2023-11-24T08:14:31.570847+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:43.480367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570095+0000","last_change":"2023-11-24T08:14:29.016822+0000","last_active":"2023-11-24T08:14:31.570095+0000","last_peered":"2023-11-24T08:14:31.570095+0000","last_clean":"2023-11-24T08:14:31.570095+0000","last_became_active":"2023-11-24T08:14:29.016560+0000","last_became_peered":"2023-11-24T08:14:29.016560+0000","last_unstale":"2023-11-24T08:14:31.570095+0000","last_undegraded":"2023-11-24T08:14:31.570095+0000","last_fullsized":"2023-11-24T08:14:31.570095+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:46:10.499337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008391+0000","last_change":"2023-11-24T08:14:29.006282+0000","last_active":"2023-11-24T08:14:30.008391+0000","last_peered":"2023-11-24T08:14:30.008391+0000","last_clean":"2023-11-24T08:14:30.008391+0000","last_became_active":"2023-11-24T08:14:29.005939+0000","last_became_peered":"2023-11-24T08:14:29.005939+0000","last_unstale":"2023-11-24T08:14:30.008391+0000","last_undegraded":"2023-11-24T08:14:30.008391+0000","last_fullsized":"2023-11-24T08:14:30.008391+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:11:48.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009571+0000","last_change":"2023-11-24T08:14:28.998796+0000","last_active":"2023-11-24T08:14:30.009571+0000","last_peered":"2023-11-24T08:14:30.009571+0000","last_clean":"2023-11-24T08:14:30.009571+0000","last_became_active":"2023-11-24T08:14:28.998497+0000","last_became_peered":"2023-11-24T08:14:28.998497+0000","last_unstale":"2023-11-24T08:14:30.009571+0000","last_undegraded":"2023-11-24T08:14:30.009571+0000","last_fullsized":"2023-11-24T08:14:30.009571+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:15:52.104000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009258+0000","last_change":"2023-11-24T08:14:28.999164+0000","last_active":"2023-11-24T08:14:30.009258+0000","last_peered":"2023-11-24T08:14:30.009258+0000","last_clean":"2023-11-24T08:14:30.009258+0000","last_became_active":"2023-11-24T08:14:28.998883+0000","last_became_peered":"2023-11-24T08:14:28.998883+0000","last_unstale":"2023-11-24T08:14:30.009258+0000","last_undegraded":"2023-11-24T08:14:30.009258+0000","last_fullsized":"2023-11-24T08:14:30.009258+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:18:43.478888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569153+0000","last_change":"2023-11-24T08:14:28.999289+0000","last_active":"2023-11-24T08:14:31.569153+0000","last_peered":"2023-11-24T08:14:31.569153+0000","last_clean":"2023-11-24T08:14:31.569153+0000","last_became_active":"2023-11-24T08:14:28.996589+0000","last_became_peered":"2023-11-24T08:14:28.996589+0000","last_unstale":"2023-11-24T08:14:31.569153+0000","last_undegraded":"2023-11-24T08:14:31.569153+0000","last_fullsized":"2023-11-24T08:14:31.569153+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:51.804622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008835+0000","last_change":"2023-11-24T08:14:29.001495+0000","last_active":"2023-11-24T08:14:30.008835+0000","last_peered":"2023-11-24T08:14:30.008835+0000","last_clean":"2023-11-24T08:14:30.008835+0000","last_became_active":"2023-11-24T08:14:29.001213+0000","last_became_peered":"2023-11-24T08:14:29.001213+0000","last_unstale":"2023-11-24T08:14:30.008835+0000","last_undegraded":"2023-11-24T08:14:30.008835+0000","last_fullsized":"2023-11-24T08:14:30.008835+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:25.447000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569406+0000","last_change":"2023-11-24T08:14:28.997204+0000","last_active":"2023-11-24T08:14:31.569406+0000","last_peered":"2023-11-24T08:14:31.569406+0000","last_clean":"2023-11-24T08:14:31.569406+0000","last_became_active":"2023-11-24T08:14:28.996504+0000","last_became_peered":"2023-11-24T08:14:28.996504+0000","last_unstale":"2023-11-24T08:14:31.569406+0000","last_undegraded":"2023-11-24T08:14:31.569406+0000","last_fullsized":"2023-11-24T08:14:31.569406+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:49:28.385483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568289+0000","last_change":"2023-11-24T08:14:29.002687+0000","last_active":"2023-11-24T08:14:31.568289+0000","last_peered":"2023-11-24T08:14:31.568289+0000","last_clean":"2023-11-24T08:14:31.568289+0000","last_became_active":"2023-11-24T08:14:29.002398+0000","last_became_peered":"2023-11-24T08:14:29.002398+0000","last_unstale":"2023-11-24T08:14:31.568289+0000","last_undegraded":"2023-11-24T08:14:31.568289+0000","last_fullsized":"2023-11-24T08:14:31.568289+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:23.896195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570864+0000","last_change":"2023-11-24T08:14:28.996655+0000","last_active":"2023-11-24T08:14:31.570864+0000","last_peered":"2023-11-24T08:14:31.570864+0000","last_clean":"2023-11-24T08:14:31.570864+0000","last_became_active":"2023-11-24T08:14:28.991990+0000","last_became_peered":"2023-11-24T08:14:28.991990+0000","last_unstale":"2023-11-24T08:14:31.570864+0000","last_undegraded":"2023-11-24T08:14:31.570864+0000","last_fullsized":"2023-11-24T08:14:31.570864+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:25.273269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.567997+0000","last_change":"2023-11-24T08:14:28.993299+0000","last_active":"2023-11-24T08:14:31.567997+0000","last_peered":"2023-11-24T08:14:31.567997+0000","last_clean":"2023-11-24T08:14:31.567997+0000","last_became_active":"2023-11-24T08:14:28.992998+0000","last_became_peered":"2023-11-24T08:14:28.992998+0000","last_unstale":"2023-11-24T08:14:31.567997+0000","last_undegraded":"2023-11-24T08:14:31.567997+0000","last_fullsized":"2023-11-24T08:14:31.567997+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:48:19.275354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572432+0000","last_change":"2023-11-24T08:14:29.014702+0000","last_active":"2023-11-24T08:14:31.572432+0000","last_peered":"2023-11-24T08:14:31.572432+0000","last_clean":"2023-11-24T08:14:31.572432+0000","last_became_active":"2023-11-24T08:14:29.014403+0000","last_became_peered":"2023-11-24T08:14:29.014403+0000","last_unstale":"2023-11-24T08:14:31.572432+0000","last_undegraded":"2023-11-24T08:14:31.572432+0000","last_fullsized":"2023-11-24T08:14:31.572432+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:54.977945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008888+0000","last_change":"2023-11-24T08:14:29.009118+0000","last_active":"2023-11-24T08:14:30.008888+0000","last_peered":"2023-11-24T08:14:30.008888+0000","last_clean":"2023-11-24T08:14:30.008888+0000","last_became_active":"2023-11-24T08:14:29.008838+0000","last_became_peered":"2023-11-24T08:14:29.008838+0000","last_unstale":"2023-11-24T08:14:30.008888+0000","last_undegraded":"2023-11-24T08:14:30.008888+0000","last_fullsized":"2023-11-24T08:14:30.008888+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:27:29.888143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571108+0000","last_change":"2023-11-24T08:14:29.009677+0000","last_active":"2023-11-24T08:14:31.571108+0000","last_peered":"2023-11-24T08:14:31.571108+0000","last_clean":"2023-11-24T08:14:31.571108+0000","last_became_active":"2023-11-24T08:14:29.007621+0000","last_became_peered":"2023-11-24T08:14:29.007621+0000","last_unstale":"2023-11-24T08:14:31.571108+0000","last_undegraded":"2023-11-24T08:14:31.571108+0000","last_fullsized":"2023-11-24T08:14:31.571108+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:44.605468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.573018+0000","last_change":"2023-11-24T08:14:28.993292+0000","last_active":"2023-11-24T08:14:31.573018+0000","last_peered":"2023-11-24T08:14:31.573018+0000","last_clean":"2023-11-24T08:14:31.573018+0000","last_became_active":"2023-11-24T08:14:28.993012+0000","last_became_peered":"2023-11-24T08:14:28.993012+0000","last_unstale":"2023-11-24T08:14:31.573018+0000","last_undegraded":"2023-11-24T08:14:31.573018+0000","last_fullsized":"2023-11-24T08:14:31.573018+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:54.977945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571368+0000","last_change":"2023-11-24T08:14:29.017375+0000","last_active":"2023-11-24T08:14:31.571368+0000","last_peered":"2023-11-24T08:14:31.571368+0000","last_clean":"2023-11-24T08:14:31.571368+0000","last_became_active":"2023-11-24T08:14:29.016409+0000","last_became_peered":"2023-11-24T08:14:29.016409+0000","last_unstale":"2023-11-24T08:14:31.571368+0000","last_undegraded":"2023-11-24T08:14:31.571368+0000","last_fullsized":"2023-11-24T08:14:31.571368+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:45.577706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569907+0000","last_change":"2023-11-24T08:14:29.012816+0000","last_active":"2023-11-24T08:14:31.569907+0000","last_peered":"2023-11-24T08:14:31.569907+0000","last_clean":"2023-11-24T08:14:31.569907+0000","last_became_active":"2023-11-24T08:14:29.011147+0000","last_became_peered":"2023-11-24T08:14:29.011147+0000","last_unstale":"2023-11-24T08:14:31.569907+0000","last_undegraded":"2023-11-24T08:14:31.569907+0000","last_fullsized":"2023-11-24T08:14:31.569907+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:53.516107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569096+0000","last_change":"2023-11-24T08:14:29.015126+0000","last_active":"2023-11-24T08:14:31.569096+0000","last_peered":"2023-11-24T08:14:31.569096+0000","last_clean":"2023-11-24T08:14:31.569096+0000","last_became_active":"2023-11-24T08:14:29.014642+0000","last_became_peered":"2023-11-24T08:14:29.014642+0000","last_unstale":"2023-11-24T08:14:31.569096+0000","last_undegraded":"2023-11-24T08:14:31.569096+0000","last_fullsized":"2023-11-24T08:14:31.569096+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:23.346076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008562+0000","last_change":"2023-11-24T08:14:29.008608+0000","last_active":"2023-11-24T08:14:30.008562+0000","last_peered":"2023-11-24T08:14:30.008562+0000","last_clean":"2023-11-24T08:14:30.008562+0000","last_became_active":"2023-11-24T08:14:29.008287+0000","last_became_peered":"2023-11-24T08:14:29.008287+0000","last_unstale":"2023-11-24T08:14:30.008562+0000","last_undegraded":"2023-11-24T08:14:30.008562+0000","last_fullsized":"2023-11-24T08:14:30.008562+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:06.648148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009013+0000","last_change":"2023-11-24T08:14:28.999548+0000","last_active":"2023-11-24T08:14:30.009013+0000","last_peered":"2023-11-24T08:14:30.009013+0000","last_clean":"2023-11-24T08:14:30.009013+0000","last_became_active":"2023-11-24T08:14:28.995469+0000","last_became_peered":"2023-11-24T08:14:28.995469+0000","last_unstale":"2023-11-24T08:14:30.009013+0000","last_undegraded":"2023-11-24T08:14:30.009013+0000","last_fullsized":"2023-11-24T08:14:30.009013+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:00:55.137710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"130'4","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.016874+0000","last_change":"2023-11-24T08:14:28.007836+0000","last_active":"2023-11-24T08:14:29.016874+0000","last_peered":"2023-11-24T08:14:29.016874+0000","last_clean":"2023-11-24T08:14:29.016874+0000","last_became_active":"2023-11-24T08:14:28.007500+0000","last_became_peered":"2023-11-24T08:14:28.007500+0000","last_unstale":"2023-11-24T08:14:29.016874+0000","last_undegraded":"2023-11-24T08:14:29.016874+0000","last_fullsized":"2023-11-24T08:14:29.016874+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:25:36.583711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572166+0000","last_change":"2023-11-24T08:14:29.009685+0000","last_active":"2023-11-24T08:14:31.572166+0000","last_peered":"2023-11-24T08:14:31.572166+0000","last_clean":"2023-11-24T08:14:31.572166+0000","last_became_active":"2023-11-24T08:14:29.009287+0000","last_became_peered":"2023-11-24T08:14:29.009287+0000","last_unstale":"2023-11-24T08:14:31.572166+0000","last_undegraded":"2023-11-24T08:14:31.572166+0000","last_fullsized":"2023-11-24T08:14:31.572166+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:00:24.270473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"130'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.010570+0000","last_change":"2023-11-24T08:14:28.003938+0000","last_active":"2023-11-24T08:14:29.010570+0000","last_peered":"2023-11-24T08:14:29.010570+0000","last_clean":"2023-11-24T08:14:29.010570+0000","last_became_active":"2023-11-24T08:14:28.001142+0000","last_became_peered":"2023-11-24T08:14:28.001142+0000","last_unstale":"2023-11-24T08:14:29.010570+0000","last_undegraded":"2023-11-24T08:14:29.010570+0000","last_fullsized":"2023-11-24T08:14:29.010570+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:57.741934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570226+0000","last_change":"2023-11-24T08:14:29.003112+0000","last_active":"2023-11-24T08:14:31.570226+0000","last_peered":"2023-11-24T08:14:31.570226+0000","last_clean":"2023-11-24T08:14:31.570226+0000","last_became_active":"2023-11-24T08:14:29.002773+0000","last_became_peered":"2023-11-24T08:14:29.002773+0000","last_unstale":"2023-11-24T08:14:31.570226+0000","last_undegraded":"2023-11-24T08:14:31.570226+0000","last_fullsized":"2023-11-24T08:14:31.570226+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:51.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.016744+0000","last_change":"2023-11-24T08:14:27.993105+0000","last_active":"2023-11-24T08:14:29.016744+0000","last_peered":"2023-11-24T08:14:29.016744+0000","last_clean":"2023-11-24T08:14:29.016744+0000","last_became_active":"2023-11-24T08:14:27.992854+0000","last_became_peered":"2023-11-24T08:14:27.992854+0000","last_unstale":"2023-11-24T08:14:29.016744+0000","last_undegraded":"2023-11-24T08:14:29.016744+0000","last_fullsized":"2023-11-24T08:14:29.016744+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:23.683248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571039+0000","last_change":"2023-11-24T08:14:29.003659+0000","last_active":"2023-11-24T08:14:31.571039+0000","last_peered":"2023-11-24T08:14:31.571039+0000","last_clean":"2023-11-24T08:14:31.571039+0000","last_became_active":"2023-11-24T08:14:29.003397+0000","last_became_peered":"2023-11-24T08:14:29.003397+0000","last_unstale":"2023-11-24T08:14:31.571039+0000","last_undegraded":"2023-11-24T08:14:31.571039+0000","last_fullsized":"2023-11-24T08:14:31.571039+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:48:19.275354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.011954+0000","last_change":"2023-11-24T08:14:27.996245+0000","last_active":"2023-11-24T08:14:29.011954+0000","last_peered":"2023-11-24T08:14:29.011954+0000","last_clean":"2023-11-24T08:14:29.011954+0000","last_became_active":"2023-11-24T08:14:27.995915+0000","last_became_peered":"2023-11-24T08:14:27.995915+0000","last_unstale":"2023-11-24T08:14:29.011954+0000","last_undegraded":"2023-11-24T08:14:29.011954+0000","last_fullsized":"2023-11-24T08:14:29.011954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:53:44.737799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008531+0000","last_change":"2023-11-24T08:14:28.997349+0000","last_active":"2023-11-24T08:14:30.008531+0000","last_peered":"2023-11-24T08:14:30.008531+0000","last_clean":"2023-11-24T08:14:30.008531+0000","last_became_active":"2023-11-24T08:14:28.997101+0000","last_became_peered":"2023-11-24T08:14:28.997101+0000","last_unstale":"2023-11-24T08:14:30.008531+0000","last_undegraded":"2023-11-24T08:14:30.008531+0000","last_fullsized":"2023-11-24T08:14:30.008531+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:45:51.804622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"130'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.013907+0000","last_change":"2023-11-24T08:14:27.993128+0000","last_active":"2023-11-24T08:14:29.013907+0000","last_peered":"2023-11-24T08:14:29.013907+0000","last_clean":"2023-11-24T08:14:29.013907+0000","last_became_active":"2023-11-24T08:14:27.992403+0000","last_became_peered":"2023-11-24T08:14:27.992403+0000","last_unstale":"2023-11-24T08:14:29.013907+0000","last_undegraded":"2023-11-24T08:14:29.013907+0000","last_fullsized":"2023-11-24T08:14:29.013907+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:51:51.292392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009121+0000","last_change":"2023-11-24T08:14:29.005530+0000","last_active":"2023-11-24T08:14:30.009121+0000","last_peered":"2023-11-24T08:14:30.009121+0000","last_clean":"2023-11-24T08:14:30.009121+0000","last_became_active":"2023-11-24T08:14:29.003556+0000","last_became_peered":"2023-11-24T08:14:29.003556+0000","last_unstale":"2023-11-24T08:14:30.009121+0000","last_undegraded":"2023-11-24T08:14:30.009121+0000","last_fullsized":"2023-11-24T08:14:30.009121+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:04.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"130'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.011432+0000","last_change":"2023-11-24T08:14:27.992627+0000","last_active":"2023-11-24T08:14:29.011432+0000","last_peered":"2023-11-24T08:14:29.011432+0000","last_clean":"2023-11-24T08:14:29.011432+0000","last_became_active":"2023-11-24T08:14:27.992364+0000","last_became_peered":"2023-11-24T08:14:27.992364+0000","last_unstale":"2023-11-24T08:14:29.011432+0000","last_undegraded":"2023-11-24T08:14:29.011432+0000","last_fullsized":"2023-11-24T08:14:29.011432+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:25:18.599641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570724+0000","last_change":"2023-11-24T08:14:29.007254+0000","last_active":"2023-11-24T08:14:31.570724+0000","last_peered":"2023-11-24T08:14:31.570724+0000","last_clean":"2023-11-24T08:14:31.570724+0000","last_became_active":"2023-11-24T08:14:29.006997+0000","last_became_peered":"2023-11-24T08:14:29.006997+0000","last_unstale":"2023-11-24T08:14:31.570724+0000","last_undegraded":"2023-11-24T08:14:31.570724+0000","last_fullsized":"2023-11-24T08:14:31.570724+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:44:55.901480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.990402+0000","last_change":"2023-11-24T08:14:28.008075+0000","last_active":"2023-11-24T08:14:28.990402+0000","last_peered":"2023-11-24T08:14:28.990402+0000","last_clean":"2023-11-24T08:14:28.990402+0000","last_became_active":"2023-11-24T08:14:28.007469+0000","last_became_peered":"2023-11-24T08:14:28.007469+0000","last_unstale":"2023-11-24T08:14:28.990402+0000","last_undegraded":"2023-11-24T08:14:28.990402+0000","last_fullsized":"2023-11-24T08:14:28.990402+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:28.694555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570167+0000","last_change":"2023-11-24T08:14:29.008624+0000","last_active":"2023-11-24T08:14:31.570167+0000","last_peered":"2023-11-24T08:14:31.570167+0000","last_clean":"2023-11-24T08:14:31.570167+0000","last_became_active":"2023-11-24T08:14:29.008344+0000","last_became_peered":"2023-11-24T08:14:29.008344+0000","last_unstale":"2023-11-24T08:14:31.570167+0000","last_undegraded":"2023-11-24T08:14:31.570167+0000","last_fullsized":"2023-11-24T08:14:31.570167+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:00.052520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.988548+0000","last_change":"2023-11-24T08:14:27.996516+0000","last_active":"2023-11-24T08:14:28.988548+0000","last_peered":"2023-11-24T08:14:28.988548+0000","last_clean":"2023-11-24T08:14:28.988548+0000","last_became_active":"2023-11-24T08:14:27.996247+0000","last_became_peered":"2023-11-24T08:14:27.996247+0000","last_unstale":"2023-11-24T08:14:28.988548+0000","last_undegraded":"2023-11-24T08:14:28.988548+0000","last_fullsized":"2023-11-24T08:14:28.988548+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:14.402581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570625+0000","last_change":"2023-11-24T08:14:29.006614+0000","last_active":"2023-11-24T08:14:31.570625+0000","last_peered":"2023-11-24T08:14:31.570625+0000","last_clean":"2023-11-24T08:14:31.570625+0000","last_became_active":"2023-11-24T08:14:29.006032+0000","last_became_peered":"2023-11-24T08:14:29.006032+0000","last_unstale":"2023-11-24T08:14:31.570625+0000","last_undegraded":"2023-11-24T08:14:31.570625+0000","last_fullsized":"2023-11-24T08:14:31.570625+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:34:09.330301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.986482+0000","last_change":"2023-11-24T08:14:28.004957+0000","last_active":"2023-11-24T08:14:28.986482+0000","last_peered":"2023-11-24T08:14:28.986482+0000","last_clean":"2023-11-24T08:14:28.986482+0000","last_became_active":"2023-11-24T08:14:28.004639+0000","last_became_peered":"2023-11-24T08:14:28.004639+0000","last_unstale":"2023-11-24T08:14:28.986482+0000","last_undegraded":"2023-11-24T08:14:28.986482+0000","last_fullsized":"2023-11-24T08:14:28.986482+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:15.579143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.573764+0000","last_change":"2023-11-24T08:14:28.991483+0000","last_active":"2023-11-24T08:14:31.573764+0000","last_peered":"2023-11-24T08:14:31.573764+0000","last_clean":"2023-11-24T08:14:31.573764+0000","last_became_active":"2023-11-24T08:14:28.991180+0000","last_became_peered":"2023-11-24T08:14:28.991180+0000","last_unstale":"2023-11-24T08:14:31.573764+0000","last_undegraded":"2023-11-24T08:14:31.573764+0000","last_fullsized":"2023-11-24T08:14:31.573764+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:18.441897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.986786+0000","last_change":"2023-11-24T08:14:28.006763+0000","last_active":"2023-11-24T08:14:28.986786+0000","last_peered":"2023-11-24T08:14:28.986786+0000","last_clean":"2023-11-24T08:14:28.986786+0000","last_became_active":"2023-11-24T08:14:28.006477+0000","last_became_peered":"2023-11-24T08:14:28.006477+0000","last_unstale":"2023-11-24T08:14:28.986786+0000","last_undegraded":"2023-11-24T08:14:28.986786+0000","last_fullsized":"2023-11-24T08:14:28.986786+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:22.328789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570272+0000","last_change":"2023-11-24T08:14:29.013356+0000","last_active":"2023-11-24T08:14:31.570272+0000","last_peered":"2023-11-24T08:14:31.570272+0000","last_clean":"2023-11-24T08:14:31.570272+0000","last_became_active":"2023-11-24T08:14:29.012684+0000","last_became_peered":"2023-11-24T08:14:29.012684+0000","last_unstale":"2023-11-24T08:14:31.570272+0000","last_undegraded":"2023-11-24T08:14:31.570272+0000","last_fullsized":"2023-11-24T08:14:31.570272+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:59.747597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"130'3","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.010639+0000","last_change":"2023-11-24T08:14:28.000140+0000","last_active":"2023-11-24T08:14:29.010639+0000","last_peered":"2023-11-24T08:14:29.010639+0000","last_clean":"2023-11-24T08:14:29.010639+0000","last_became_active":"2023-11-24T08:14:27.999821+0000","last_became_peered":"2023-11-24T08:14:27.999821+0000","last_unstale":"2023-11-24T08:14:29.010639+0000","last_undegraded":"2023-11-24T08:14:29.010639+0000","last_fullsized":"2023-11-24T08:14:29.010639+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:31.240616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571351+0000","last_change":"2023-11-24T08:14:29.007203+0000","last_active":"2023-11-24T08:14:31.571351+0000","last_peered":"2023-11-24T08:14:31.571351+0000","last_clean":"2023-11-24T08:14:31.571351+0000","last_became_active":"2023-11-24T08:14:29.006789+0000","last_became_peered":"2023-11-24T08:14:29.006789+0000","last_unstale":"2023-11-24T08:14:31.571351+0000","last_undegraded":"2023-11-24T08:14:31.571351+0000","last_fullsized":"2023-11-24T08:14:31.571351+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:23.896195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"130'6","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.019938+0000","last_change":"2023-11-24T08:14:28.009062+0000","last_active":"2023-11-24T08:14:29.019938+0000","last_peered":"2023-11-24T08:14:29.019938+0000","last_clean":"2023-11-24T08:14:29.019938+0000","last_became_active":"2023-11-24T08:14:28.008166+0000","last_became_peered":"2023-11-24T08:14:28.008166+0000","last_unstale":"2023-11-24T08:14:29.019938+0000","last_undegraded":"2023-11-24T08:14:29.019938+0000","last_fullsized":"2023-11-24T08:14:29.019938+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:28.694555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571622+0000","last_change":"2023-11-24T08:14:29.009434+0000","last_active":"2023-11-24T08:14:31.571622+0000","last_peered":"2023-11-24T08:14:31.571622+0000","last_clean":"2023-11-24T08:14:31.571622+0000","last_became_active":"2023-11-24T08:14:29.009015+0000","last_became_peered":"2023-11-24T08:14:29.009015+0000","last_unstale":"2023-11-24T08:14:31.571622+0000","last_undegraded":"2023-11-24T08:14:31.571622+0000","last_fullsized":"2023-11-24T08:14:31.571622+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:22:24.750569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.990807+0000","last_change":"2023-11-24T08:14:28.008303+0000","last_active":"2023-11-24T08:14:28.990807+0000","last_peered":"2023-11-24T08:14:28.990807+0000","last_clean":"2023-11-24T08:14:28.990807+0000","last_became_active":"2023-11-24T08:14:28.008028+0000","last_became_peered":"2023-11-24T08:14:28.008028+0000","last_unstale":"2023-11-24T08:14:28.990807+0000","last_undegraded":"2023-11-24T08:14:28.990807+0000","last_fullsized":"2023-11-24T08:14:28.990807+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:07.625261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568762+0000","last_change":"2023-11-24T08:14:29.012009+0000","last_active":"2023-11-24T08:14:31.568762+0000","last_peered":"2023-11-24T08:14:31.568762+0000","last_clean":"2023-11-24T08:14:31.568762+0000","last_became_active":"2023-11-24T08:14:29.011746+0000","last_became_peered":"2023-11-24T08:14:29.011746+0000","last_unstale":"2023-11-24T08:14:31.568762+0000","last_undegraded":"2023-11-24T08:14:31.568762+0000","last_fullsized":"2023-11-24T08:14:31.568762+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:31:58.759221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.990627+0000","last_change":"2023-11-24T08:14:27.999783+0000","last_active":"2023-11-24T08:14:28.990627+0000","last_peered":"2023-11-24T08:14:28.990627+0000","last_clean":"2023-11-24T08:14:28.990627+0000","last_became_active":"2023-11-24T08:14:27.999360+0000","last_became_peered":"2023-11-24T08:14:27.999360+0000","last_unstale":"2023-11-24T08:14:28.990627+0000","last_undegraded":"2023-11-24T08:14:28.990627+0000","last_fullsized":"2023-11-24T08:14:28.990627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:44.560419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571718+0000","last_change":"2023-11-24T08:14:28.992671+0000","last_active":"2023-11-24T08:14:31.571718+0000","last_peered":"2023-11-24T08:14:31.571718+0000","last_clean":"2023-11-24T08:14:31.571718+0000","last_became_active":"2023-11-24T08:14:28.991607+0000","last_became_peered":"2023-11-24T08:14:28.991607+0000","last_unstale":"2023-11-24T08:14:31.571718+0000","last_undegraded":"2023-11-24T08:14:31.571718+0000","last_fullsized":"2023-11-24T08:14:31.571718+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:06.648148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"130'8","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.019033+0000","last_change":"2023-11-24T08:14:28.001801+0000","last_active":"2023-11-24T08:14:29.019033+0000","last_peered":"2023-11-24T08:14:29.019033+0000","last_clean":"2023-11-24T08:14:29.019033+0000","last_became_active":"2023-11-24T08:14:28.001356+0000","last_became_peered":"2023-11-24T08:14:28.001356+0000","last_unstale":"2023-11-24T08:14:29.019033+0000","last_undegraded":"2023-11-24T08:14:29.019033+0000","last_fullsized":"2023-11-24T08:14:29.019033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:17:17.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.571785+0000","last_change":"2023-11-24T08:14:29.017582+0000","last_active":"2023-11-24T08:14:31.571785+0000","last_peered":"2023-11-24T08:14:31.571785+0000","last_clean":"2023-11-24T08:14:31.571785+0000","last_became_active":"2023-11-24T08:14:29.017282+0000","last_became_peered":"2023-11-24T08:14:29.017282+0000","last_unstale":"2023-11-24T08:14:31.571785+0000","last_undegraded":"2023-11-24T08:14:31.571785+0000","last_fullsized":"2023-11-24T08:14:31.571785+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:21:59.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"130'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.013270+0000","last_change":"2023-11-24T08:14:28.004863+0000","last_active":"2023-11-24T08:14:29.013270+0000","last_peered":"2023-11-24T08:14:29.013270+0000","last_clean":"2023-11-24T08:14:29.013270+0000","last_became_active":"2023-11-24T08:14:28.004582+0000","last_became_peered":"2023-11-24T08:14:28.004582+0000","last_unstale":"2023-11-24T08:14:29.013270+0000","last_undegraded":"2023-11-24T08:14:29.013270+0000","last_fullsized":"2023-11-24T08:14:29.013270+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:42.463080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.573857+0000","last_change":"2023-11-24T08:14:28.994846+0000","last_active":"2023-11-24T08:14:31.573857+0000","last_peered":"2023-11-24T08:14:31.573857+0000","last_clean":"2023-11-24T08:14:31.573857+0000","last_became_active":"2023-11-24T08:14:28.994464+0000","last_became_peered":"2023-11-24T08:14:28.994464+0000","last_unstale":"2023-11-24T08:14:31.573857+0000","last_undegraded":"2023-11-24T08:14:31.573857+0000","last_fullsized":"2023-11-24T08:14:31.573857+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:49.385631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.019729+0000","last_change":"2023-11-24T08:14:28.005860+0000","last_active":"2023-11-24T08:14:29.019729+0000","last_peered":"2023-11-24T08:14:29.019729+0000","last_clean":"2023-11-24T08:14:29.019729+0000","last_became_active":"2023-11-24T08:14:28.004165+0000","last_became_peered":"2023-11-24T08:14:28.004165+0000","last_unstale":"2023-11-24T08:14:29.019729+0000","last_undegraded":"2023-11-24T08:14:29.019729+0000","last_fullsized":"2023-11-24T08:14:29.019729+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:58.730310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008982+0000","last_change":"2023-11-24T08:14:29.014005+0000","last_active":"2023-11-24T08:14:30.008982+0000","last_peered":"2023-11-24T08:14:30.008982+0000","last_clean":"2023-11-24T08:14:30.008982+0000","last_became_active":"2023-11-24T08:14:29.013701+0000","last_became_peered":"2023-11-24T08:14:29.013701+0000","last_unstale":"2023-11-24T08:14:30.008982+0000","last_undegraded":"2023-11-24T08:14:30.008982+0000","last_fullsized":"2023-11-24T08:14:30.008982+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:38.998968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"130'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.009899+0000","last_change":"2023-11-24T08:14:28.003777+0000","last_active":"2023-11-24T08:14:29.009899+0000","last_peered":"2023-11-24T08:14:29.009899+0000","last_clean":"2023-11-24T08:14:29.009899+0000","last_became_active":"2023-11-24T08:14:28.003459+0000","last_became_peered":"2023-11-24T08:14:28.003459+0000","last_unstale":"2023-11-24T08:14:29.009899+0000","last_undegraded":"2023-11-24T08:14:29.009899+0000","last_fullsized":"2023-11-24T08:14:29.009899+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:41.719707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572330+0000","last_change":"2023-11-24T08:14:29.008786+0000","last_active":"2023-11-24T08:14:31.572330+0000","last_peered":"2023-11-24T08:14:31.572330+0000","last_clean":"2023-11-24T08:14:31.572330+0000","last_became_active":"2023-11-24T08:14:29.008488+0000","last_became_peered":"2023-11-24T08:14:29.008488+0000","last_unstale":"2023-11-24T08:14:31.572330+0000","last_undegraded":"2023-11-24T08:14:31.572330+0000","last_fullsized":"2023-11-24T08:14:31.572330+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:15.530298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.982458+0000","last_change":"2023-11-24T08:14:28.001321+0000","last_active":"2023-11-24T08:14:28.982458+0000","last_peered":"2023-11-24T08:14:28.982458+0000","last_clean":"2023-11-24T08:14:28.982458+0000","last_became_active":"2023-11-24T08:14:28.001049+0000","last_became_peered":"2023-11-24T08:14:28.001049+0000","last_unstale":"2023-11-24T08:14:28.982458+0000","last_undegraded":"2023-11-24T08:14:28.982458+0000","last_fullsized":"2023-11-24T08:14:28.982458+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:38.939236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570029+0000","last_change":"2023-11-24T08:14:29.005510+0000","last_active":"2023-11-24T08:14:31.570029+0000","last_peered":"2023-11-24T08:14:31.570029+0000","last_clean":"2023-11-24T08:14:31.570029+0000","last_became_active":"2023-11-24T08:14:29.005232+0000","last_became_peered":"2023-11-24T08:14:29.005232+0000","last_unstale":"2023-11-24T08:14:31.570029+0000","last_undegraded":"2023-11-24T08:14:31.570029+0000","last_fullsized":"2023-11-24T08:14:31.570029+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:07.171480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.011107+0000","last_change":"2023-11-24T08:14:28.004913+0000","last_active":"2023-11-24T08:14:29.011107+0000","last_peered":"2023-11-24T08:14:29.011107+0000","last_clean":"2023-11-24T08:14:29.011107+0000","last_became_active":"2023-11-24T08:14:28.004565+0000","last_became_peered":"2023-11-24T08:14:28.004565+0000","last_unstale":"2023-11-24T08:14:29.011107+0000","last_undegraded":"2023-11-24T08:14:29.011107+0000","last_fullsized":"2023-11-24T08:14:29.011107+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:53:44.737799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009538+0000","last_change":"2023-11-24T08:14:29.005775+0000","last_active":"2023-11-24T08:14:30.009538+0000","last_peered":"2023-11-24T08:14:30.009538+0000","last_clean":"2023-11-24T08:14:30.009538+0000","last_became_active":"2023-11-24T08:14:29.005382+0000","last_became_peered":"2023-11-24T08:14:29.005382+0000","last_unstale":"2023-11-24T08:14:30.009538+0000","last_undegraded":"2023-11-24T08:14:30.009538+0000","last_fullsized":"2023-11-24T08:14:30.009538+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:29:56.182188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.986827+0000","last_change":"2023-11-24T08:14:28.007712+0000","last_active":"2023-11-24T08:14:28.986827+0000","last_peered":"2023-11-24T08:14:28.986827+0000","last_clean":"2023-11-24T08:14:28.986827+0000","last_became_active":"2023-11-24T08:14:28.007446+0000","last_became_peered":"2023-11-24T08:14:28.007446+0000","last_unstale":"2023-11-24T08:14:28.986827+0000","last_undegraded":"2023-11-24T08:14:28.986827+0000","last_fullsized":"2023-11-24T08:14:28.986827+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:49:27.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569782+0000","last_change":"2023-11-24T08:14:29.007598+0000","last_active":"2023-11-24T08:14:31.569782+0000","last_peered":"2023-11-24T08:14:31.569782+0000","last_clean":"2023-11-24T08:14:31.569782+0000","last_became_active":"2023-11-24T08:14:29.006781+0000","last_became_peered":"2023-11-24T08:14:29.006781+0000","last_unstale":"2023-11-24T08:14:31.569782+0000","last_undegraded":"2023-11-24T08:14:31.569782+0000","last_fullsized":"2023-11-24T08:14:31.569782+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:14:39.956523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.987369+0000","last_change":"2023-11-24T08:14:28.001680+0000","last_active":"2023-11-24T08:14:28.987369+0000","last_peered":"2023-11-24T08:14:28.987369+0000","last_clean":"2023-11-24T08:14:28.987369+0000","last_became_active":"2023-11-24T08:14:28.001431+0000","last_became_peered":"2023-11-24T08:14:28.001431+0000","last_unstale":"2023-11-24T08:14:28.987369+0000","last_undegraded":"2023-11-24T08:14:28.987369+0000","last_fullsized":"2023-11-24T08:14:28.987369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:50.031260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568676+0000","last_change":"2023-11-24T08:14:29.014637+0000","last_active":"2023-11-24T08:14:31.568676+0000","last_peered":"2023-11-24T08:14:31.568676+0000","last_clean":"2023-11-24T08:14:31.568676+0000","last_became_active":"2023-11-24T08:14:29.014347+0000","last_became_peered":"2023-11-24T08:14:29.014347+0000","last_unstale":"2023-11-24T08:14:31.568676+0000","last_undegraded":"2023-11-24T08:14:31.568676+0000","last_fullsized":"2023-11-24T08:14:31.568676+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:07:24.155123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.986630+0000","last_change":"2023-11-24T08:14:28.005732+0000","last_active":"2023-11-24T08:14:28.986630+0000","last_peered":"2023-11-24T08:14:28.986630+0000","last_clean":"2023-11-24T08:14:28.986630+0000","last_became_active":"2023-11-24T08:14:28.004637+0000","last_became_peered":"2023-11-24T08:14:28.004637+0000","last_unstale":"2023-11-24T08:14:28.986630+0000","last_undegraded":"2023-11-24T08:14:28.986630+0000","last_fullsized":"2023-11-24T08:14:28.986630+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:22.328789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.010041+0000","last_change":"2023-11-24T08:14:29.004125+0000","last_active":"2023-11-24T08:14:30.010041+0000","last_peered":"2023-11-24T08:14:30.010041+0000","last_clean":"2023-11-24T08:14:30.010041+0000","last_became_active":"2023-11-24T08:14:29.003782+0000","last_became_peered":"2023-11-24T08:14:29.003782+0000","last_unstale":"2023-11-24T08:14:30.010041+0000","last_undegraded":"2023-11-24T08:14:30.010041+0000","last_fullsized":"2023-11-24T08:14:30.010041+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:12:00.484690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.015364+0000","last_change":"2023-11-24T08:14:28.001910+0000","last_active":"2023-11-24T08:14:29.015364+0000","last_peered":"2023-11-24T08:14:29.015364+0000","last_clean":"2023-11-24T08:14:29.015364+0000","last_became_active":"2023-11-24T08:14:28.001250+0000","last_became_peered":"2023-11-24T08:14:28.001250+0000","last_unstale":"2023-11-24T08:14:29.015364+0000","last_undegraded":"2023-11-24T08:14:29.015364+0000","last_fullsized":"2023-11-24T08:14:29.015364+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:15:05.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570156+0000","last_change":"2023-11-24T08:14:29.015898+0000","last_active":"2023-11-24T08:14:31.570156+0000","last_peered":"2023-11-24T08:14:31.570156+0000","last_clean":"2023-11-24T08:14:31.570156+0000","last_became_active":"2023-11-24T08:14:29.015633+0000","last_became_peered":"2023-11-24T08:14:29.015633+0000","last_unstale":"2023-11-24T08:14:31.570156+0000","last_undegraded":"2023-11-24T08:14:31.570156+0000","last_fullsized":"2023-11-24T08:14:31.570156+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:15.419868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"131'1","reported_seq":13,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.569541+0000","last_change":"2023-11-24T08:14:28.010054+0000","last_active":"2023-11-24T08:14:31.569541+0000","last_peered":"2023-11-24T08:14:31.569541+0000","last_clean":"2023-11-24T08:14:31.569541+0000","last_became_active":"2023-11-24T08:14:28.009808+0000","last_became_peered":"2023-11-24T08:14:28.009808+0000","last_unstale":"2023-11-24T08:14:31.569541+0000","last_undegraded":"2023-11-24T08:14:31.569541+0000","last_fullsized":"2023-11-24T08:14:31.569541+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:53:36.500941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570536+0000","last_change":"2023-11-24T08:14:29.015567+0000","last_active":"2023-11-24T08:14:31.570536+0000","last_peered":"2023-11-24T08:14:31.570536+0000","last_clean":"2023-11-24T08:14:31.570536+0000","last_became_active":"2023-11-24T08:14:29.015233+0000","last_became_peered":"2023-11-24T08:14:29.015233+0000","last_unstale":"2023-11-24T08:14:31.570536+0000","last_undegraded":"2023-11-24T08:14:31.570536+0000","last_fullsized":"2023-11-24T08:14:31.570536+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:15:34.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.990627+0000","last_change":"2023-11-24T08:14:27.994656+0000","last_active":"2023-11-24T08:14:28.990627+0000","last_peered":"2023-11-24T08:14:28.990627+0000","last_clean":"2023-11-24T08:14:28.990627+0000","last_became_active":"2023-11-24T08:14:27.994381+0000","last_became_peered":"2023-11-24T08:14:27.994381+0000","last_unstale":"2023-11-24T08:14:28.990627+0000","last_undegraded":"2023-11-24T08:14:28.990627+0000","last_fullsized":"2023-11-24T08:14:28.990627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:04.148702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.009772+0000","last_change":"2023-11-24T08:14:29.013166+0000","last_active":"2023-11-24T08:14:30.009772+0000","last_peered":"2023-11-24T08:14:30.009772+0000","last_clean":"2023-11-24T08:14:30.009772+0000","last_became_active":"2023-11-24T08:14:29.012410+0000","last_became_peered":"2023-11-24T08:14:29.012410+0000","last_unstale":"2023-11-24T08:14:30.009772+0000","last_undegraded":"2023-11-24T08:14:30.009772+0000","last_fullsized":"2023-11-24T08:14:30.009772+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:51.649363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.987964+0000","last_change":"2023-11-24T08:14:27.994554+0000","last_active":"2023-11-24T08:14:28.987964+0000","last_peered":"2023-11-24T08:14:28.987964+0000","last_clean":"2023-11-24T08:14:28.987964+0000","last_became_active":"2023-11-24T08:14:27.994294+0000","last_became_peered":"2023-11-24T08:14:27.994294+0000","last_unstale":"2023-11-24T08:14:28.987964+0000","last_undegraded":"2023-11-24T08:14:28.987964+0000","last_fullsized":"2023-11-24T08:14:28.987964+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:50.031260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568794+0000","last_change":"2023-11-24T08:14:29.013561+0000","last_active":"2023-11-24T08:14:31.568794+0000","last_peered":"2023-11-24T08:14:31.568794+0000","last_clean":"2023-11-24T08:14:31.568794+0000","last_became_active":"2023-11-24T08:14:29.013254+0000","last_became_peered":"2023-11-24T08:14:29.013254+0000","last_unstale":"2023-11-24T08:14:31.568794+0000","last_undegraded":"2023-11-24T08:14:31.568794+0000","last_fullsized":"2023-11-24T08:14:31.568794+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:17:49.692757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.987164+0000","last_change":"2023-11-24T08:14:28.004634+0000","last_active":"2023-11-24T08:14:28.987164+0000","last_peered":"2023-11-24T08:14:28.987164+0000","last_clean":"2023-11-24T08:14:28.987164+0000","last_became_active":"2023-11-24T08:14:28.004254+0000","last_became_peered":"2023-11-24T08:14:28.004254+0000","last_unstale":"2023-11-24T08:14:28.987164+0000","last_undegraded":"2023-11-24T08:14:28.987164+0000","last_fullsized":"2023-11-24T08:14:28.987164+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:52.498820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.572812+0000","last_change":"2023-11-24T08:14:29.009653+0000","last_active":"2023-11-24T08:14:31.572812+0000","last_peered":"2023-11-24T08:14:31.572812+0000","last_clean":"2023-11-24T08:14:31.572812+0000","last_became_active":"2023-11-24T08:14:29.009343+0000","last_became_peered":"2023-11-24T08:14:29.009343+0000","last_unstale":"2023-11-24T08:14:31.572812+0000","last_undegraded":"2023-11-24T08:14:31.572812+0000","last_fullsized":"2023-11-24T08:14:31.572812+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:15.530298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.990512+0000","last_change":"2023-11-24T08:14:28.009103+0000","last_active":"2023-11-24T08:14:28.990512+0000","last_peered":"2023-11-24T08:14:28.990512+0000","last_clean":"2023-11-24T08:14:28.990512+0000","last_became_active":"2023-11-24T08:14:28.008680+0000","last_became_peered":"2023-11-24T08:14:28.008680+0000","last_unstale":"2023-11-24T08:14:28.990512+0000","last_undegraded":"2023-11-24T08:14:28.990512+0000","last_fullsized":"2023-11-24T08:14:28.990512+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:39:06.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:30.008372+0000","last_change":"2023-11-24T08:14:29.007901+0000","last_active":"2023-11-24T08:14:30.008372+0000","last_peered":"2023-11-24T08:14:30.008372+0000","last_clean":"2023-11-24T08:14:30.008372+0000","last_became_active":"2023-11-24T08:14:29.007269+0000","last_became_peered":"2023-11-24T08:14:29.007269+0000","last_unstale":"2023-11-24T08:14:30.008372+0000","last_undegraded":"2023-11-24T08:14:30.008372+0000","last_fullsized":"2023-11-24T08:14:30.008372+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:23.896195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.987594+0000","last_change":"2023-11-24T08:14:28.005030+0000","last_active":"2023-11-24T08:14:28.987594+0000","last_peered":"2023-11-24T08:14:28.987594+0000","last_clean":"2023-11-24T08:14:28.987594+0000","last_became_active":"2023-11-24T08:14:28.004745+0000","last_became_peered":"2023-11-24T08:14:28.004745+0000","last_unstale":"2023-11-24T08:14:28.987594+0000","last_undegraded":"2023-11-24T08:14:28.987594+0000","last_fullsized":"2023-11-24T08:14:28.987594+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:58.730310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.570574+0000","last_change":"2023-11-24T08:14:29.016776+0000","last_active":"2023-11-24T08:14:31.570574+0000","last_peered":"2023-11-24T08:14:31.570574+0000","last_clean":"2023-11-24T08:14:31.570574+0000","last_became_active":"2023-11-24T08:14:29.016361+0000","last_became_peered":"2023-11-24T08:14:29.016361+0000","last_unstale":"2023-11-24T08:14:31.570574+0000","last_undegraded":"2023-11-24T08:14:31.570574+0000","last_fullsized":"2023-11-24T08:14:31.570574+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:16.596430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.986756+0000","last_change":"2023-11-24T08:14:28.006409+0000","last_active":"2023-11-24T08:14:28.986756+0000","last_peered":"2023-11-24T08:14:28.986756+0000","last_clean":"2023-11-24T08:14:28.986756+0000","last_became_active":"2023-11-24T08:14:28.006014+0000","last_became_peered":"2023-11-24T08:14:28.006014+0000","last_unstale":"2023-11-24T08:14:28.986756+0000","last_undegraded":"2023-11-24T08:14:28.986756+0000","last_fullsized":"2023-11-24T08:14:28.986756+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:49:27.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-24T08:14:31.568835+0000","last_change":"2023-11-24T08:14:29.016608+0000","last_active":"2023-11-24T08:14:31.568835+0000","last_peered":"2023-11-24T08:14:31.568835+0000","last_clean":"2023-11-24T08:14:31.568835+0000","last_became_active":"2023-11-24T08:14:29.016311+0000","last_became_peered":"2023-11-24T08:14:29.016311+0000","last_unstale":"2023-11-24T08:14:31.568835+0000","last_undegraded":"2023-11-24T08:14:31.568835+0000","last_fullsized":"2023-11-24T08:14:31.568835+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:27.955401+0000","last_clean_scrub_stamp":"2023-11-24T08:14:27.955401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:25:19.616928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"130'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:29.014309+0000","last_change":"2023-11-24T08:14:28.009227+0000","last_active":"2023-11-24T08:14:29.014309+0000","last_peered":"2023-11-24T08:14:29.014309+0000","last_clean":"2023-11-24T08:14:29.014309+0000","last_became_active":"2023-11-24T08:14:28.008642+0000","last_became_peered":"2023-11-24T08:14:28.008642+0000","last_unstale":"2023-11-24T08:14:29.014309+0000","last_undegraded":"2023-11-24T08:14:29.014309+0000","last_fullsized":"2023-11-24T08:14:29.014309+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:14:26.938114+0000","last_clean_scrub_stamp":"2023-11-24T08:14:26.938114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:41:56.053825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":261,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-11-24T08:14:28.984775+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:14:28.984775+0000","last_peered":"2023-11-24T08:14:28.984775+0000","last_clean":"2023-11-24T08:14:28.984775+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:14:28.984775+0000","last_undegraded":"2023-11-24T08:14:28.984775+0000","last_fullsized":"2023-11-24T08:14:28.984775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37540,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334300,"statfs":{"total":96636764160,"available":96598323200,"internally_reserved":0,"allocated":2252800,"data_stored":1306208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0780000000000001,"15min":1.03},"min":{"1min":0.65600000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.3859999999999999,"5min":2.7240000000000002,"15min":2.7240000000000002},"last":1.018},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0820000000000001,"15min":1.0860000000000001},"min":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.52900000000000003},"max":{"1min":1.3120000000000001,"5min":2.3340000000000001,"15min":2.7109999999999999},"last":1.581}]},{"osd":1,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.157,"15min":1.1419999999999999},"min":{"1min":0.78400000000000003,"5min":0.64700000000000002,"15min":0.46300000000000002},"max":{"1min":1.5029999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.403},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2290000000000001,"15min":1.202},"min":{"1min":0.84699999999999998,"5min":0.70299999999999996,"15min":0.628},"max":{"1min":1.601,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.085}]},{"osd":2,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1850000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.58699999999999997},"max":{"1min":1.774,"5min":2.4430000000000001,"15min":4.7000000000000002},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.246,"5min":1.3220000000000001,"15min":1.343},"min":{"1min":0.82999999999999996,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.667,"5min":4.0650000000000004,"15min":4.633},"last":1.5329999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2190000000000001,"15min":1.169},"min":{"1min":0.98499999999999999,"5min":0.79500000000000004,"15min":0.504},"max":{"1min":1.4330000000000001,"5min":3.133,"15min":3.133},"last":1.456},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.401,"15min":1.3720000000000001},"min":{"1min":0.98199999999999998,"5min":0.82399999999999995,"15min":0.65700000000000003},"max":{"1min":1.8009999999999999,"5min":3.9209999999999998,"15min":3.9209999999999998},"last":1.2609999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3109999999999999,"15min":1.3340000000000001},"min":{"1min":0.79700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.871,"5min":2.661,"15min":2.851},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":1.0669999999999999,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.6240000000000001,"5min":3.4870000000000001,"15min":3.4870000000000001},"last":1.1439999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.472,"15min":1.464},"min":{"1min":0.99399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.016,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.645},{"interface":"front","average":{"1min":1.53,"5min":1.4630000000000001,"15min":1.484},"min":{"1min":1.0289999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.1230000000000002,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.4930000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.546,"15min":1.5509999999999999},"min":{"1min":1.1950000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.3039999999999998,"5min":2.992,"15min":2.992},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4830000000000001,"15min":1.464},"min":{"1min":1.2589999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"max":{"1min":1.929,"5min":3.798,"15min":3.798},"last":1.3100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574969,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1646592,"data_stored":715202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0760000000000001,"15min":1.0880000000000001},"min":{"1min":0.63700000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.377,"5min":1.95,"15min":1.95},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0800000000000001,"15min":1.095},"min":{"1min":0.753,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.804,"5min":1.986,"15min":2.2069999999999999},"last":0.80600000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1719999999999999,"15min":1.2290000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5840000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.101},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1399999999999999,"15min":1.2390000000000001},"min":{"1min":0.72999999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.4039999999999999},"last":1.2889999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0920000000000001,"15min":1.117},"min":{"1min":0.76100000000000001,"5min":0.68700000000000006,"15min":0.68400000000000005},"max":{"1min":1.544,"5min":1.6779999999999999,"15min":1.9330000000000001},"last":1.573},{"interface":"front","average":{"1min":1.262,"5min":1.27,"15min":1.345},"min":{"1min":0.93600000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.615,"5min":2.0369999999999999,"15min":2.3239999999999998},"last":1.2170000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.323,"15min":1.3740000000000001},"min":{"1min":0.86399999999999999,"5min":0.83399999999999996,"15min":0.71499999999999997},"max":{"1min":1.7410000000000001,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.343,"15min":1.343},"min":{"1min":0.90000000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.7949999999999999,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.677}]},{"osd":4,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4199999999999999,"15min":1.506},"min":{"1min":0.95799999999999996,"5min":0.79900000000000004,"15min":0.76500000000000001},"max":{"1min":2.0790000000000002,"5min":2.895,"15min":2.895},"last":1.345},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4690000000000001,"15min":1.538},"min":{"1min":1.165,"5min":1.056,"15min":0.90800000000000003},"max":{"1min":2.1909999999999998,"5min":2.242,"15min":2.8820000000000001},"last":1.506}]},{"osd":5,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.429,"15min":1.5189999999999999},"min":{"1min":0.91000000000000003,"5min":0.88600000000000001,"15min":0.79300000000000004},"max":{"1min":1.9730000000000001,"5min":2.1709999999999998,"15min":2.629},"last":1.857},{"interface":"front","average":{"1min":1.583,"5min":1.5569999999999999,"15min":1.6519999999999999},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.95,"5min":2.4249999999999998,"15min":2.7759999999999998},"last":1.4410000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.5649999999999999,"15min":1.6779999999999999},"min":{"1min":1.135,"5min":1.04,"15min":0.95399999999999996},"max":{"1min":2.125,"5min":2.125,"15min":3.0329999999999999},"last":1.8859999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5820000000000001,"15min":1.6579999999999999},"min":{"1min":1.1910000000000001,"5min":0.95999999999999996,"15min":0.88200000000000001},"max":{"1min":2.254,"5min":2.2909999999999999,"15min":2.6890000000000001},"last":1.766}]}]},{"osd":5,"up_from":13,"seq":55834574969,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32784,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339056,"statfs":{"total":96636764160,"available":96603193344,"internally_reserved":0,"allocated":1642496,"data_stored":713702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.026,"15min":1.0900000000000001},"min":{"1min":0.65300000000000002,"5min":0.58599999999999997,"15min":0.58299999999999996},"max":{"1min":1.4690000000000001,"5min":1.694,"15min":3.1179999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.054,"5min":1.119,"15min":1.1619999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5960000000000001,"5min":1.859,"15min":2.198},"last":0.94099999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1399999999999999,"15min":1.1559999999999999},"min":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.5620000000000001,"5min":1.7070000000000001,"15min":2.0529999999999999},"last":1.01},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1970000000000001,"15min":1.1579999999999999},"min":{"1min":0.78500000000000003,"5min":0.73399999999999999,"15min":0.48699999999999999},"max":{"1min":1.611,"5min":1.659,"15min":1.6910000000000001},"last":0.877}]},{"osd":2,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1779999999999999,"15min":1.202},"min":{"1min":0.73299999999999998,"5min":0.65100000000000002,"15min":0.51600000000000001},"max":{"1min":1.671,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.143,"15min":1.1659999999999999},"min":{"1min":0.83799999999999997,"5min":0.72499999999999998,"15min":0.64200000000000002},"max":{"1min":1.552,"5min":1.6839999999999999,"15min":1.8129999999999999},"last":1.3069999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1240000000000001,"15min":1.1699999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.28,"15min":1.3149999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.55200000000000005},"max":{"1min":1.522,"5min":1.9730000000000001,"15min":1.99},"last":1.1970000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3080000000000001,"15min":1.3280000000000001},"min":{"1min":0.66400000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.518,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.452,"15min":1.5189999999999999},"min":{"1min":0.86299999999999999,"5min":0.75600000000000001,"15min":0.46700000000000003},"max":{"1min":1.7709999999999999,"5min":2.1019999999999999,"15min":3.5030000000000001},"last":1.1040000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2969999999999999,"15min":1.407},"min":{"1min":0.63600000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":3.2949999999999999},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.403,"15min":1.4259999999999999},"min":{"1min":0.71299999999999997,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":1.6739999999999999,"5min":1.9490000000000001,"15min":2.1099999999999999},"last":1.76}]},{"osd":7,"last update":"Fri Nov 24 08:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.554},"min":{"1min":0.69899999999999995,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":3.4420000000000002},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.587,"15min":1.6040000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":1.869,"5min":2.181,"15min":2.181},"last":1.911}]}]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1638400,"data_stored":713660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.0760000000000001,"15min":1.1279999999999999},"min":{"1min":0.70399999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0620000000000001,"15min":1.083},"min":{"1min":0.68400000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.7150000000000001,"15min":2.222},"last":0.85099999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.111,"15min":1.1499999999999999},"min":{"1min":0.73699999999999999,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1639999999999999,"15min":1.1819999999999999},"min":{"1min":0.73199999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6279999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":1.272}]},{"osd":2,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0740000000000001,"15min":1.135},"min":{"1min":0.80700000000000005,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.552,"5min":1.5780000000000001,"15min":2.1419999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.212,"15min":1.321},"min":{"1min":0.88300000000000001,"5min":0.505,"15min":0.505},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.347}]},{"osd":3,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.306,"15min":1.3959999999999999},"min":{"1min":0.93100000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.26,"15min":1.3049999999999999},"min":{"1min":0.92700000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6499999999999999,"5min":1.6599999999999999,"15min":1.8280000000000001},"last":1.492}]},{"osd":5,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4419999999999999,"15min":1.4970000000000001},"min":{"1min":0.71999999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.417,"15min":1.504},"min":{"1min":0.78200000000000003,"5min":0.67000000000000004,"15min":0.60199999999999998},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.3879999999999999},"last":1.76}]},{"osd":6,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.234,"15min":1.381},"min":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.50900000000000001},"max":{"1min":1.651,"5min":1.7110000000000001,"15min":3.5899999999999999},"last":1.425},{"interface":"front","average":{"1min":1.522,"5min":1.494,"15min":1.6100000000000001},"min":{"1min":0.79000000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":3.391},"last":1.9399999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4650000000000001,"15min":1.579},"min":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.9370000000000001,"5min":2.052,"15min":3.1230000000000002},"last":1.679},{"interface":"front","average":{"1min":1.397,"5min":1.387,"15min":1.528},"min":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6879999999999999,"5min":1.857,"15min":3.4849999999999999},"last":1.528}]}]},{"osd":3,"up_from":13,"seq":55834574970,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1642496,"data_stored":713750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89100000000000001,"15min":0.86099999999999999},"min":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.643,"5min":3.181,"15min":10.557},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.999,"5min":0.93700000000000006,"15min":0.92100000000000004},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6679999999999999,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.85199999999999998,"15min":0.83699999999999997},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.538,"5min":2.101,"15min":2.101},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.99199999999999999,"15min":1.008},"min":{"1min":0.51700000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"max":{"1min":1.6859999999999999,"5min":3.3260000000000001,"15min":13.090999999999999},"last":0.379}]},{"osd":2,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.93799999999999994,"15min":0.95899999999999996},"min":{"1min":0.48099999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":1.6060000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.087,"15min":1.0860000000000001},"min":{"1min":0.56100000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.708,"5min":3.2480000000000002,"15min":13.460000000000001},"last":0.45200000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.282,"15min":1.2849999999999999},"min":{"1min":0.83799999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":4.367},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.22,"15min":1.29},"min":{"1min":0.94599999999999995,"5min":0.68899999999999995,"15min":0.64200000000000002},"max":{"1min":1.514,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.306,"15min":1.323},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.857,"5min":3.4399999999999999,"15min":13.231},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.379,"5min":1.2490000000000001,"15min":1.3080000000000001},"min":{"1min":0.82199999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.956,"5min":1.956,"15min":5.1180000000000003},"last":0.81100000000000005}]},{"osd":6,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.379,"15min":1.399},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.72899999999999998},"max":{"1min":1.931,"5min":3.5030000000000001,"15min":13.359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.28,"5min":1.268,"15min":1.323},"min":{"1min":0.77800000000000002,"5min":0.73799999999999999,"15min":0.67000000000000004},"max":{"1min":2.024,"5min":2.0870000000000002,"15min":3.4409999999999998},"last":0.91800000000000004}]},{"osd":7,"last update":"Fri Nov 24 08:13:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.349,"15min":1.411},"min":{"1min":0.77500000000000002,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0790000000000002,"5min":2.101,"15min":4.5060000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3420000000000001,"15min":1.4199999999999999},"min":{"1min":0.87,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7969999999999999,"5min":3.331,"15min":4.032},"last":0.78400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574968,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1642496,"data_stored":714206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78500000000000003,"15min":0.879},"min":{"1min":0.46800000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.3500000000000001,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.90900000000000003,"15min":0.97699999999999998},"min":{"1min":0.46400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3919999999999999,"5min":1.782,"15min":2.5779999999999998},"last":1.266}]},{"osd":1,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.91900000000000004,"15min":1.0600000000000001},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.429,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.95399999999999996,"15min":1.075},"min":{"1min":0.52400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.6080000000000001,"5min":1.637,"15min":2.6240000000000001},"last":1.413}]},{"osd":3,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.84599999999999997,"15min":0.92100000000000004},"min":{"1min":0.48299999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.4410000000000001,"5min":1.552,"15min":2.2090000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0309999999999999,"15min":1.194},"min":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.50900000000000001},"max":{"1min":1.5249999999999999,"5min":1.8480000000000001,"15min":2.9950000000000001},"last":1.004}]},{"osd":4,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.375},"min":{"1min":0.72399999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":1.51},{"interface":"front","average":{"1min":1.278,"5min":1.26,"15min":1.359},"min":{"1min":0.90200000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.921,"5min":1.921,"15min":3.3279999999999998},"last":1.335}]},{"osd":5,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2749999999999999,"15min":1.3979999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.73399999999999999},"max":{"1min":1.8959999999999999,"5min":2.3149999999999999,"15min":2.508},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.2290000000000001,"15min":1.359},"min":{"1min":0.85199999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.7839999999999998},"last":1.601}]},{"osd":6,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.194,"15min":1.2989999999999999},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":2.1709999999999998,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.337,"15min":1.4630000000000001},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":1.944,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.1160000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:14:06 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.325,"15min":1.4550000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.71299999999999997},"max":{"1min":2.0760000000000001,"5min":2.1070000000000002,"15min":2.859},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.258,"15min":1.4179999999999999},"min":{"1min":0.86799999999999999,"5min":0.746,"15min":0.746},"max":{"1min":1.97,"5min":1.9930000000000001,"15min":3.3809999999999998},"last":1.1579999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36936,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334904,"statfs":{"total":96636764160,"available":96598941696,"internally_reserved":0,"allocated":1634304,"data_stored":713590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84299999999999997,"15min":0.78500000000000003},"min":{"1min":0.36699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.5900000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.91400000000000003,"15min":0.85899999999999999},"min":{"1min":0.44,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.5189999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.746},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.679,"5min":1.861,"15min":1.861},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.94999999999999996,"15min":0.89800000000000002},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5609999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.1100000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0329999999999999,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.92300000000000004,"15min":0.89200000000000002},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.45,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.179}]},{"osd":4,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2849999999999999,"15min":1.2529999999999999},"min":{"1min":0.88700000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.827,"5min":2.028,"15min":2.7959999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.90000000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.649,"5min":2.1080000000000001,"15min":3.0409999999999999},"last":1.0600000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.274,"15min":1.224},"min":{"1min":0.80000000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8520000000000001,"5min":2.02,"15min":3.7639999999999998},"last":1.276},{"interface":"front","average":{"1min":1.206,"5min":1.1970000000000001,"15min":1.1879999999999999},"min":{"1min":0.78100000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.246,"5min":2.246,"15min":3.286},"last":1.038}]},{"osd":6,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3180000000000001,"15min":1.2470000000000001},"min":{"1min":0.79100000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8759999999999999,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.2909999999999999,"15min":1.2270000000000001},"min":{"1min":0.72299999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.788,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":1.208}]},{"osd":7,"last update":"Fri Nov 24 08:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3300000000000001,"15min":1.302},"min":{"1min":0.81200000000000006,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7709999999999999,"5min":2.7280000000000002,"15min":3.1760000000000002},"last":1.343},{"interface":"front","average":{"1min":1.377,"5min":1.355,"15min":1.2949999999999999},"min":{"1min":0.96199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3620000000000001,"5min":3.653,"15min":3.653},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574970,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37516,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334324,"statfs":{"total":96636764160,"available":96598347776,"internally_reserved":0,"allocated":2228224,"data_stored":1303958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71399999999999997,"15min":0.82399999999999995},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":1.425,"15min":1.6020000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.84499999999999997,"15min":0.95799999999999996},"min":{"1min":0.47499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.4610000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.97199999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.016,"15min":1.05},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.47699999999999998},"max":{"1min":1.5209999999999999,"5min":1.891,"15min":1.891},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.92900000000000005},"min":{"1min":0.47499999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3799999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.044}]},{"osd":3,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.97099999999999997,"15min":1.0740000000000001},"min":{"1min":0.54800000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.456,"5min":1.79,"15min":2.0670000000000002},"last":1.3},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":1.0049999999999999},"min":{"1min":0.44900000000000001,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.4079999999999999,"5min":1.641,"15min":2.1789999999999998},"last":1.103}]},{"osd":4,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1859999999999999,"15min":1.3480000000000001},"min":{"1min":0.76400000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.724,"5min":2.3010000000000002,"15min":3.7610000000000001},"last":1.548},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.109,"15min":1.2010000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.54,"5min":1.6240000000000001,"15min":2.2839999999999998},"last":1.349}]},{"osd":5,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.266},"min":{"1min":0.79700000000000004,"5min":0.72799999999999998,"15min":0.69699999999999995},"max":{"1min":1.5649999999999999,"5min":1.9730000000000001,"15min":2.5390000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.173,"15min":1.349},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.514,"5min":1.75,"15min":2.7240000000000002},"last":1.196}]},{"osd":6,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.28,"15min":1.3839999999999999},"min":{"1min":0.78000000000000003,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":1.714,"5min":1.986,"15min":2.4369999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.256,"15min":1.3859999999999999},"min":{"1min":0.81399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7030000000000001,"5min":1.998,"15min":2.738},"last":1.4379999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:13:58 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3560000000000001,"15min":1.4710000000000001},"min":{"1min":0.88,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.7410000000000001,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.3320000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.776,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.4970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:14:33.935 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:14:33.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:14:34.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.197+0000 7fde4d8c2640 1 Processor -- start 2023-11-24T08:14:34.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.198+0000 7fde4d8c2640 1 -- start start 2023-11-24T08:14:34.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.199+0000 7fde4d8c2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fde480c67c0 0x7fde48161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.199+0000 7fde4d8c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde48161940 0x7fde4806ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.199+0000 7fde4d8c2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4815de00 con 0x7fde48140520 2023-11-24T08:14:34.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.199+0000 7fde4d8c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4815e0e0 con 0x7fde48161940 2023-11-24T08:14:34.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.199+0000 7fde4d8c2640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4815e3c0 con 0x7fde480c67c0 2023-11-24T08:14:34.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde48161940 0x7fde4806ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde48161940 0x7fde4806ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36608/0 (socket says 172.21.15.107:36608) 2023-11-24T08:14:34.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 -- 172.21.15.107:0/3552123797 learned_addr learned my addr 172.21.15.107:0/3552123797 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:34.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde3e7fc640 1 --2- 172.21.15.107:0/3552123797 >> v2:172.21.15.120:3300/0 conn(0x7fde480c67c0 0x7fde48161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 -- 172.21.15.107:0/3552123797 >> v2:172.21.15.120:3300/0 conn(0x7fde480c67c0 msgr2=0x7fde48161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 --2- 172.21.15.107:0/3552123797 >> v2:172.21.15.120:3300/0 conn(0x7fde480c67c0 0x7fde48161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 -- 172.21.15.107:0/3552123797 >> v1:172.21.15.120:6789/0 conn(0x7fde48140520 legacy=0x7fde480c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.200+0000 7fde477fe640 1 -- 172.21.15.107:0/3552123797 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4815e6a0 con 0x7fde48161940 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.201+0000 7fde477fe640 1 --2- 172.21.15.107:0/3552123797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4806ff70 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fde480b0c30 tx=0x7fde4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c8c84638896ef1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.201+0000 7fde467fc640 1 -- 172.21.15.107:0/3552123797 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4002ca80 con 0x7fde48161940 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.201+0000 7fde467fc640 1 -- 172.21.15.107:0/3552123797 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4002cbe0 con 0x7fde48161940 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.201+0000 7fde467fc640 1 -- 172.21.15.107:0/3552123797 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400343f0 con 0x7fde48161940 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 -- 172.21.15.107:0/3552123797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 msgr2=0x7fde4806ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/3552123797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4806ff70 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fde480b0c30 tx=0x7fde4002bee0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 -- 172.21.15.107:0/3552123797 shutdown_connections 2023-11-24T08:14:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/3552123797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4806ff70 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/3552123797 >> v2:172.21.15.120:3300/0 conn(0x7fde480c67c0 0x7fde48161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 -- 172.21.15.107:0/3552123797 >> 172.21.15.107:0/3552123797 conn(0x7fde480b2e40 msgr2=0x7fde480b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 -- 172.21.15.107:0/3552123797 shutdown_connections 2023-11-24T08:14:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.202+0000 7fde4d8c2640 1 -- 172.21.15.107:0/3552123797 wait complete. 2023-11-24T08:14:34.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.204+0000 7fde4d8c2640 1 Processor -- start 2023-11-24T08:14:34.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.204+0000 7fde4d8c2640 1 -- start start 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde480c67c0 0x7fde48139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde48140520 0x7fde4806cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cf660 con 0x7fde48161940 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480ced30 con 0x7fde48140520 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde4d8c2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cf010 con 0x7fde480c67c0 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36622/0 (socket says 172.21.15.107:36622) 2023-11-24T08:14:34.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.205+0000 7fde477fe640 1 -- 172.21.15.107:0/2313079467 learned_addr learned my addr 172.21.15.107:0/2313079467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde46ffd640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde480c67c0 0x7fde48139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde3e7fc640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde48140520 0x7fde4806cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 -- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde480c67c0 msgr2=0x7fde48139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde480c67c0 0x7fde48139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 -- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde48140520 msgr2=0x7fde4806cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde48140520 0x7fde4806cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 -- 172.21.15.107:0/2313079467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009b30 con 0x7fde48161940 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde477fe640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fde40036040 tx=0x7fde40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4003e450 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4003e5d0 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4003e980 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde480cf2f0 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.206+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde481572a0 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.208+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4804e8b0 con 0x7fde48161940 2023-11-24T08:14:34.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.209+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fde4002cd50 con 0x7fde48161940 2023-11-24T08:14:34.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.209+0000 7fde44ff9640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 0x7fde2806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.210+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde400ab1c0 con 0x7fde48161940 2023-11-24T08:14:34.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.210+0000 7fde46ffd640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 0x7fde2806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.210+0000 7fde46ffd640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 0x7fde2806e500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde38007fd0 tx=0x7fde38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.214+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde40077b20 con 0x7fde48161940 2023-11-24T08:14:34.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.473+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fde480cf7a0 con 0x7fde48161940 2023-11-24T08:14:34.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.478+0000 7fde44ff9640 1 -- 172.21.15.107:0/2313079467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fde40043020 con 0x7fde48161940 2023-11-24T08:14:34.480 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:14:34.480 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:14:34.480 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:14:34.480 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:14:34.481 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:14:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 msgr2=0x7fde2806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 0x7fde2806e500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde38007fd0 tx=0x7fde38009040 comp rx=0 tx=0).stop 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 msgr2=0x7fde4813c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fde40036040 tx=0x7fde40004a40 comp rx=0 tx=0).stop 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 shutdown_connections 2023-11-24T08:14:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde2806c040 0x7fde2806e500 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde48161940 0x7fde4813c970 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde48140520 0x7fde4806cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 --2- 172.21.15.107:0/2313079467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde480c67c0 0x7fde48139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 >> 172.21.15.107:0/2313079467 conn(0x7fde480b2e40 msgr2=0x7fde480cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 shutdown_connections 2023-11-24T08:14:34.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.481+0000 7fde4d8c2640 1 -- 172.21.15.107:0/2313079467 wait complete. 2023-11-24T08:14:34.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:14:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.704+0000 7f808eeb4640 1 Processor -- start 2023-11-24T08:14:34.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.705+0000 7f808eeb4640 1 -- start start 2023-11-24T08:14:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.706+0000 7f808eeb4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f808815e390 0x7f8088161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.706+0000 7f808eeb4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80880ce0b0 con 0x7f80880cfb60 2023-11-24T08:14:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.706+0000 7f808eeb4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8088161960 con 0x7f8088157c70 2023-11-24T08:14:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.706+0000 7f808eeb4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808815e8d0 con 0x7f808815e390 2023-11-24T08:14:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f8087fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f80880cfb60 0x7f808815dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36970/0 (socket says 172.21.15.107:36970) 2023-11-24T08:14:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f8087fff640 1 -- 172.21.15.107:0/1018367709 learned_addr learned my addr 172.21.15.107:0/1018367709 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f808d451640 1 --2- 172.21.15.107:0/1018367709 >> v2:172.21.15.120:3300/0 conn(0x7f808815e390 0x7f8088161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008865347 0 0) 0x7f80880ce0b0 con 0x7f80880cfb60 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8070003520 con 0x7f80880cfb60 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2759456987 0 0) 0x7f8070003520 con 0x7f80880cfb60 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 >> v2:172.21.15.120:3300/0 conn(0x7f808815e390 msgr2=0x7f8088161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 --2- 172.21.15.107:0/1018367709 >> v2:172.21.15.120:3300/0 conn(0x7f808815e390 0x7f8088161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 >> v1:172.21.15.120:6789/0 conn(0x7f8088157c70 legacy=0x7f80880cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808815ebb0 con 0x7f80880cfb60 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.707+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f807c001670 con 0x7f80880cfb60 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 >> v1:172.21.15.107:6789/0 conn(0x7f80880cfb60 legacy=0x7f808815dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80700039c0 0x7f8070003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 0x7f8070004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700051d0 0x7f807000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808815e8d0 con 0x7f80700047c0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807000c120 con 0x7f80700039c0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807000c2e0 con 0x7f80700051d0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 0x7f8070004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f808cc50640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80700039c0 0x7f8070003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 -- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700051d0 msgr2=0x7f807000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700051d0 0x7f807000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 -- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80700039c0 msgr2=0x7f8070003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80700039c0 0x7f8070003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f8087fff640 1 -- 172.21.15.107:0/1018367709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8070003520 con 0x7f80700047c0 2023-11-24T08:14:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.708+0000 7f808d451640 1 -- 172.21.15.107:0/1018367709 reap_dead start 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.709+0000 7f8087fff640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 0x7f8070004c40 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f807c007fb0 tx=0x7f80700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f3eb3c61e4042f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c005ad0 con 0x7f80700047c0 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c005c30 con 0x7f80700047c0 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f80877fe640 1 -- 172.21.15.107:0/1018367709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c007070 con 0x7f80700047c0 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 -- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 msgr2=0x7f8070004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 0x7f8070004c40 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f807c007fb0 tx=0x7f80700057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 -- 172.21.15.107:0/1018367709 shutdown_connections 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 --2- 172.21.15.107:0/1018367709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80700047c0 0x7f8070004c40 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 -- 172.21.15.107:0/1018367709 >> 172.21.15.107:0/1018367709 conn(0x7f808806a920 msgr2=0x7f808806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 -- 172.21.15.107:0/1018367709 shutdown_connections 2023-11-24T08:14:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.710+0000 7f808eeb4640 1 -- 172.21.15.107:0/1018367709 wait complete. 2023-11-24T08:14:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.712+0000 7f808eeb4640 1 Processor -- start 2023-11-24T08:14:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.712+0000 7f808eeb4640 1 -- start start 2023-11-24T08:14:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700047c0 0x7f80881f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80880cbf00 0x7f80881560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088154a10 con 0x7f808814a610 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c8360 con 0x7f80880cbf00 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f808eeb4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c8640 con 0x7f80700047c0 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f8087fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f8087fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36648/0 (socket says 172.21.15.107:36648) 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.713+0000 7f8087fff640 1 -- 172.21.15.107:0/808540194 learned_addr learned my addr 172.21.15.107:0/808540194 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f808cc50640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700047c0 0x7f80881f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 -- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700047c0 msgr2=0x7f80881f3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f808d451640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80880cbf00 0x7f80881560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700047c0 0x7f80881f3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 -- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80880cbf00 msgr2=0x7f80881560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80880cbf00 0x7f80881560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 -- 172.21.15.107:0/808540194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80880c8920 con 0x7f808814a610 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f8087fff640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f807c02c310 tx=0x7f80881618d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c03c070 con 0x7f808814a610 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c007280 con 0x7f808814a610 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0075f0 con 0x7f808814a610 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.714+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80880c8c00 con 0x7f808814a610 2023-11-24T08:14:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.715+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80880c91c0 con 0x7f808814a610 2023-11-24T08:14:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.716+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8088067470 con 0x7f808814a610 2023-11-24T08:14:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.717+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f807c032070 con 0x7f808814a610 2023-11-24T08:14:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.718+0000 7f80857fa640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 0x7f807406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.718+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f807c0a9180 con 0x7f808814a610 2023-11-24T08:14:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.718+0000 7f808cc50640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 0x7f807406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.719+0000 7f808cc50640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 0x7f807406e530 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f80880681f0 tx=0x7f8078009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:34.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.722+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f807c0759a0 con 0x7f808814a610 2023-11-24T08:14:34.840 INFO:tasks.ceph.mds.d.smithi120.stderr:2023-11-24T08:14:34.838+0000 7f28818dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:14:34.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.949+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f808815d5e0 con 0x7f808814a610 2023-11-24T08:14:34.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.950+0000 7f80857fa640 1 -- 172.21.15.107:0/808540194 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f807c045020 con 0x7f808814a610 2023-11-24T08:14:34.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:14:34.952 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:14:34.953 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 78 2023-11-24T08:14:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 msgr2=0x7f807406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 0x7f807406e530 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f80880681f0 tx=0x7f8078009040 comp rx=0 tx=0).stop 2023-11-24T08:14:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 msgr2=0x7f80880cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f807c02c310 tx=0x7f80881618d0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 shutdown_connections 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f807406c070 0x7f807406e530 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80880cbf00 0x7f80881560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f808814a610 0x7f80880cb9c0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 --2- 172.21.15.107:0/808540194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80700047c0 0x7f80881f3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 >> 172.21.15.107:0/808540194 conn(0x7f808806a920 msgr2=0x7f80880afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 shutdown_connections 2023-11-24T08:14:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:34.953+0000 7f808eeb4640 1 -- 172.21.15.107:0/808540194 wait complete. 2023-11-24T08:14:35.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:14:28.424695+0000', 'modified': '2023-11-24T08:14:31.463839+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8754}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8754': {'gid': 8754, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6835/2095975197', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 2095975197}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 2095975197}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:14:35.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:14:35.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-24T08:14:35.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.601+0000 7f5c82246640 1 Processor -- start 2023-11-24T08:14:35.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.601+0000 7f5c82246640 1 -- start start 2023-11-24T08:14:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.602+0000 7f5c82246640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.602+0000 7f5c82246640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.602+0000 7f5c82246640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c0cc3a0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.602+0000 7f5c82246640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c15b820 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.602+0000 7f5c82246640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c15eea0 con 0x7f5c7c15b440 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36650/0 (socket says 172.21.15.107:36650) 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 -- 172.21.15.107:0/4041550179 learned_addr learned my addr 172.21.15.107:0/4041550179 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 -- 172.21.15.107:0/4041550179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 msgr2=0x7f5c7c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 -- 172.21.15.107:0/4041550179 >> v1:172.21.15.120:6789/0 conn(0x7f5c7c0c57e0 legacy=0x7f5c7c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c73fff640 1 -- 172.21.15.107:0/4041550179 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c06d600 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.603+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:35.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.604+0000 7f5c73fff640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009870 tx=0x7f5c6402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869a4303d8e47f1f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.604+0000 7f5c7affd640 1 -- 172.21.15.107:0/4041550179 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640317c0 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c7affd640 1 -- 172.21.15.107:0/4041550179 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64031920 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c7affd640 1 -- 172.21.15.107:0/4041550179 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64031cb0 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 -- 172.21.15.107:0/4041550179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0cbf20 msgr2=0x7f5c7c0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009870 tx=0x7f5c6402b860 comp rx=0 tx=0).stop 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 -- 172.21.15.107:0/4041550179 shutdown_connections 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c0c52a0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 --2- 172.21.15.107:0/4041550179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 -- 172.21.15.107:0/4041550179 >> 172.21.15.107:0/4041550179 conn(0x7f5c7c0b0c50 msgr2=0x7f5c7c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 -- 172.21.15.107:0/4041550179 shutdown_connections 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.605+0000 7f5c82246640 1 -- 172.21.15.107:0/4041550179 wait complete. 2023-11-24T08:14:35.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.607+0000 7f5c82246640 1 Processor -- start 2023-11-24T08:14:35.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.607+0000 7f5c82246640 1 -- start start 2023-11-24T08:14:35.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c1554b0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c155700 con 0x7f5c7c0cbf20 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c82246640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c1559e0 con 0x7f5c7c15b440 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36652/0 (socket says 172.21.15.107:36652) 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.608+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/1582651321 learned_addr learned my addr 172.21.15.107:0/1582651321 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 msgr2=0x7f5c7c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c73fff640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c7c0cbf20 msgr2=0x7f5c7c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/1582651321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640094d0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c7f0 tx=0x7f5c6c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c78ff9640 1 -- 172.21.15.107:0/1582651321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c007820 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c78ff9640 1 -- 172.21.15.107:0/1582651321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c007980 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c78ff9640 1 -- 172.21.15.107:0/1582651321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c007cf0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.609+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7c155cc0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.610+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c156280 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.612+0000 7f5c78ff9640 1 -- 172.21.15.107:0/1582651321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f5c6c0113f0 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.613+0000 7f5c78ff9640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 0x7f5c5c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.613+0000 7f5c78ff9640 1 -- 172.21.15.107:0/1582651321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c6c088870 con 0x7f5c7c0c57e0 2023-11-24T08:14:35.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.613+0000 7f5c73fff640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 0x7f5c5c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.614+0000 7f5c73fff640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 0x7f5c5c06e530 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5c64036000 tx=0x7f5c6403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.614+0000 7f5c84ddd740 1 Processor -- start 2023-11-24T08:14:35.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.614+0000 7f5c84ddd740 1 -- start start 2023-11-24T08:14:35.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556269585ba0 0x55626958a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958a5e0 0x5562695ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562694e9880 con 0x55626958aea0 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562694f8ae0 con 0x556269585ba0 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562694f61d0 con 0x55626958a5e0 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36662/0 (socket says 172.21.15.107:36662) 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/3130978842 learned_addr learned my addr 172.21.15.107:0/3130978842 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958a5e0 msgr2=0x5562695ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958a5e0 0x5562695ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556269585ba0 msgr2=0x55626958a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556269585ba0 0x55626958a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.615+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/3130978842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562694b9860 con 0x55626958aea0 2023-11-24T08:14:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.616+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c058c50 tx=0x7f5c6c058730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc79f206e21d6023 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.616+0000 7f5c717fa640 1 -- 172.21.15.107:0/3130978842 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c088b00 con 0x55626958aea0 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.616+0000 7f5c717fa640 1 -- 172.21.15.107:0/3130978842 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c0553d0 con 0x55626958aea0 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.616+0000 7f5c717fa640 1 -- 172.21.15.107:0/3130978842 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c060d30 con 0x55626958aea0 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- 172.21.15.107:0/3130978842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 msgr2=0x55626958b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c058c50 tx=0x7f5c6c058730 comp rx=0 tx=0).stop 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- 172.21.15.107:0/3130978842 shutdown_connections 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958a5e0 0x5562695ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556269585ba0 0x55626958a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/3130978842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x55626958b280 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- 172.21.15.107:0/3130978842 >> 172.21.15.107:0/3130978842 conn(0x55626954cc70 msgr2=0x5562695d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- 172.21.15.107:0/3130978842 shutdown_connections 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- 172.21.15.107:0/3130978842 wait complete. 2023-11-24T08:14:35.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.617+0000 7f5c84ddd740 1 -- start start 2023-11-24T08:14:35.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.618+0000 7f5c84ddd740 1 Processor -- start 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556269585ba0 0x5562695b1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562694f1ac0 con 0x55626958aea0 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626948e2f0 con 0x55626958a5e0 2023-11-24T08:14:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562694abc70 con 0x556269585ba0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c84ddd740 10 client.0 _ll_register_callbacks cb 0x5562694b03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36664/0 (socket says 172.21.15.107:36664) 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 -- 172.21.15.107:0/347055049 learned_addr learned my addr 172.21.15.107:0/347055049 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556269585ba0 0x5562695b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c73fff640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 -- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556269585ba0 msgr2=0x5562695b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556269585ba0 0x5562695b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.619+0000 7f5c7bfff640 1 -- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 msgr2=0x5562695c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c7bfff640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c7bfff640 1 -- 172.21.15.107:0/347055049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c156280 con 0x55626958aea0 2023-11-24T08:14:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c7bfff640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400c1e0 tx=0x7f5c7400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82bd42a0a3f8bd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74011790 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c740118f0 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562694eaaa0 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c74003d90 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c84ddd740 10 client.8877 fetch_fsmap learned FSMap version 78 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.620+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5562694f0140 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5562694eaaa0 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5c74006b50 con 0x55626958aea0 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 10 client.8877 fetch_fsmap finished waiting for FSMap version 78 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 10 client.8877 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T08:14:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 20 client.8877 populate_metadata read hostname 'smithi107' 2023-11-24T08:14:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695cb290 0x5562695d3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 4 client.8877 mds_command: new command op to 8754 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:14:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 --> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562694ea7c0 con 0x5562695cb290 2023-11-24T08:14:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.621+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695cb290 0x5562695d3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.622+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695cb290 0x5562695d3900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67b3d367ff2b06ae server_cookie=48544a7b0a7bc958 in_seq=0 out_seq=0 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.622+0000 7f5c5a7fc640 10 client.8877 ms_handle_connect on v2:172.21.15.120:6834/2095975197 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.624+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/347055049 <== mds.0 v2:172.21.15.120:6834/2095975197 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5562694ea7c0 con 0x5562695cb290 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.624+0000 7f5c5a7fc640 10 client.8877 handle_command_reply: tid=0 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 client.8877 shutdown 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 msgr2=0x5562695d8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400c1e0 tx=0x7f5c7400c6a0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 shutdown_connections 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695cb290 0x5562695d3900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55626958aea0 0x5562695d8820 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/347055049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556269585ba0 0x5562695b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c5a7fc640 0 client.8877 ms_handle_reset on v2:172.21.15.120:6834/2095975197 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 >> 172.21.15.107:0/347055049 conn(0x55626954cc70 msgr2=0x556269584b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 shutdown_connections 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 1 -- 172.21.15.107:0/347055049 wait complete. 2023-11-24T08:14:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.625+0000 7f5c84ddd740 20 client.8877 trim_cache size 0 max 16384 2023-11-24T08:14:35.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.642+0000 7f5c84ddd740 1 Processor -- start 2023-11-24T08:14:35.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.642+0000 7f5c84ddd740 1 -- start start 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695b94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695d7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x556269254f20 con 0x556269585ba0 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562695ca490 con 0x55626958a5e0 2023-11-24T08:14:35.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562695c7540 con 0x55626958aea0 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.643+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36668/0 (socket says 172.21.15.107:36668) 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/2879843087 learned_addr learned my addr 172.21.15.107:0/2879843087 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c73fff640 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695b94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7bfff640 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695d7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 msgr2=0x5562695d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695d7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 msgr2=0x5562695b94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695b94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/2879843087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55626948db80 con 0x556269585ba0 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c057010 tx=0x7f5c6c06d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b9c01ae902c693 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.644+0000 7f5c71ffb640 1 -- 172.21.15.107:0/2879843087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c088ab0 con 0x556269585ba0 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c71ffb640 1 -- 172.21.15.107:0/2879843087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c088c10 con 0x556269585ba0 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c71ffb640 1 -- 172.21.15.107:0/2879843087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c060d50 con 0x556269585ba0 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 -- 172.21.15.107:0/2879843087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 msgr2=0x5562695c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c057010 tx=0x7f5c6c06d000 comp rx=0 tx=0).stop 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 -- 172.21.15.107:0/2879843087 shutdown_connections 2023-11-24T08:14:35.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695d7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x5562695b94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/2879843087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695c8f20 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 -- 172.21.15.107:0/2879843087 >> 172.21.15.107:0/2879843087 conn(0x55626954cc70 msgr2=0x5562695d38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 -- 172.21.15.107:0/2879843087 shutdown_connections 2023-11-24T08:14:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.645+0000 7f5c84ddd740 1 -- 172.21.15.107:0/2879843087 wait complete. 2023-11-24T08:14:35.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.646+0000 7f5c84ddd740 1 -- start start 2023-11-24T08:14:35.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.646+0000 7f5c84ddd740 1 Processor -- start 2023-11-24T08:14:35.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x55626963f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562695df240 con 0x556269585ba0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562695b2050 con 0x55626958a5e0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562695df450 con 0x55626958aea0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c84ddd740 10 client.0 _ll_register_callbacks cb 0x7f5c74003c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36678/0 (socket says 172.21.15.107:36678) 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.647+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/97280139 learned_addr learned my addr 172.21.15.107:0/97280139 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 msgr2=0x5562695b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7bfff640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c73fff640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x55626963f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 msgr2=0x55626963f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x55626963f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 -- 172.21.15.107:0/97280139 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562694b9860 con 0x556269585ba0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c7b7fe640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c7c0 tx=0x7f5c6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b58872954dba32 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c088ab0 con 0x556269585ba0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c6c060d30 con 0x556269585ba0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c6c0553d0 con 0x556269585ba0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c6c05db50 con 0x556269585ba0 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c84ddd740 10 client.8883 fetch_fsmap learned FSMap version 78 2023-11-24T08:14:35.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.648+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5562694b3170 con 0x556269585ba0 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c0553d0 con 0x556269585ba0 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5c6c077050 con 0x556269585ba0 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 10 client.8883 fetch_fsmap finished waiting for FSMap version 78 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 10 client.8883 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 20 client.8883 populate_metadata read hostname 'smithi107' 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695d7990 0x5562695d7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 4 client.8883 mds_command: new command op to 8754 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 --> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5562691f26c0 con 0x5562695d7990 2023-11-24T08:14:35.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.649+0000 7f5c73fff640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695d7990 0x5562695d7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.650+0000 7f5c5a7fc640 10 client.8883 ms_handle_connect on v2:172.21.15.120:6834/2095975197 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.650+0000 7f5c73fff640 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695d7990 0x5562695d7d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=184d463091417619 server_cookie=478121ff3fbaa0a7 in_seq=0 out_seq=0 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c5a7fc640 1 -- 172.21.15.107:0/97280139 <== mds.0 v2:172.21.15.120:6834/2095975197 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5562691f26c0 con 0x5562695d7990 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c5a7fc640 10 client.8883 handle_command_reply: tid=0 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 client.8883 shutdown 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 msgr2=0x5562695b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c7c0 tx=0x7f5c6c004730 comp rx=0 tx=0).stop 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 shutdown_connections 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:6834/2095975197,v1:172.21.15.120:6835/2095975197] conn(0x5562695d7990 0x5562695d7d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55626958aea0 0x5562695b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55626958a5e0 0x55626963f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 --2- 172.21.15.107:0/97280139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556269585ba0 0x5562695b0000 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c5a7fc640 0 client.8883 ms_handle_reset on v2:172.21.15.120:6834/2095975197 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.651+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 >> 172.21.15.107:0/97280139 conn(0x55626954cc70 msgr2=0x5562695b94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 shutdown_connections 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c84ddd740 1 -- 172.21.15.107:0/97280139 wait complete. 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c84ddd740 20 client.8883 trim_cache size 0 max 16384 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:14:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 8754, 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6748780840000004, 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 78, 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 131, 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 131, 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.196025228 2023-11-24T08:14:35.658 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 msgr2=0x7f5c5c06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 0x7f5c5c06e530 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5c64036000 tx=0x7f5c6403d040 comp rx=0 tx=0).stop 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 msgr2=0x7f5c7c154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c7f0 tx=0x7f5c6c00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 shutdown_connections 2023-11-24T08:14:35.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c5c06c070 0x7f5c5c06e530 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c7c15b440 0x7f5c7c159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c7c0cbf20 0x7f5c7c157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 --2- 172.21.15.107:0/1582651321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c7c0c57e0 0x7f5c7c154ac0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 >> 172.21.15.107:0/1582651321 conn(0x7f5c7c0b0c50 msgr2=0x7f5c7c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 shutdown_connections 2023-11-24T08:14:35.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.652+0000 7f5c82246640 1 -- 172.21.15.107:0/1582651321 wait complete. 2023-11-24T08:14:35.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:14:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.867+0000 7f5e85883640 1 Processor -- start 2023-11-24T08:14:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.868+0000 7f5e85883640 1 -- start start 2023-11-24T08:14:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.869+0000 7f5e85883640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e800cf450 0x7f5e800cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.869+0000 7f5e85883640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800cece0 con 0x7f5e800cdfb0 2023-11-24T08:14:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.869+0000 7f5e85883640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8006d040 con 0x7f5e8015c7f0 2023-11-24T08:14:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.869+0000 7f5e85883640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800cfe40 con 0x7f5e800cf450 2023-11-24T08:14:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5e800cdfb0 0x7f5e800ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36980/0 (socket says 172.21.15.107:36980) 2023-11-24T08:14:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7f7fe640 1 -- 172.21.15.107:0/1905782917 learned_addr learned my addr 172.21.15.107:0/1905782917 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7e7fc640 1 --2- 172.21.15.107:0/1905782917 >> v2:172.21.15.120:3300/0 conn(0x7f5e800cf450 0x7f5e800cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552168004 0 0) 0x7f5e800cece0 con 0x7f5e800cdfb0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e68003520 con 0x7f5e800cdfb0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.870+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186320591 0 0) 0x7f5e68003520 con 0x7f5e800cdfb0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 >> v2:172.21.15.120:3300/0 conn(0x7f5e800cf450 msgr2=0x7f5e800cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 --2- 172.21.15.107:0/1905782917 >> v2:172.21.15.120:3300/0 conn(0x7f5e800cf450 0x7f5e800cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 >> v1:172.21.15.120:6789/0 conn(0x7f5e8015c7f0 legacy=0x7f5e8006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8006d320 con 0x7f5e800cdfb0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5e78001670 con 0x7f5e800cdfb0 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 >> v1:172.21.15.107:6789/0 conn(0x7f5e800cdfb0 legacy=0x7f5e800ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e680047c0 0x7f5e68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e68005180 0x7f5e6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800cfe40 con 0x7f5e680039c0 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6800c080 con 0x7f5e680047c0 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6800c240 con 0x7f5e68005180 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 -- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e68005180 msgr2=0x7f5e6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e68005180 0x7f5e6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 -- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e680047c0 msgr2=0x7f5e68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e680047c0 0x7f5e68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7effd640 1 -- 172.21.15.107:0/1905782917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68003520 con 0x7f5e680039c0 2023-11-24T08:14:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.871+0000 7f5e7f7fe640 1 -- 172.21.15.107:0/1905782917 reap_dead start 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.872+0000 7f5e7effd640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e68003de0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f5e70001750 tx=0x7f5e680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd687c73b80218b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e700052d0 con 0x7f5e680039c0 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7002e050 con 0x7f5e680039c0 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e7dffb640 1 -- 172.21.15.107:0/1905782917 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70003830 con 0x7f5e680039c0 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 -- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 msgr2=0x7f5e68003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e68003de0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f5e70001750 tx=0x7f5e680057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 -- 172.21.15.107:0/1905782917 shutdown_connections 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 --2- 172.21.15.107:0/1905782917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e68003de0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 -- 172.21.15.107:0/1905782917 >> 172.21.15.107:0/1905782917 conn(0x7f5e8006a920 msgr2=0x7f5e8006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 -- 172.21.15.107:0/1905782917 shutdown_connections 2023-11-24T08:14:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.873+0000 7f5e85883640 1 -- 172.21.15.107:0/1905782917 wait complete. 2023-11-24T08:14:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.875+0000 7f5e85883640 1 Processor -- start 2023-11-24T08:14:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.875+0000 7f5e85883640 1 -- start start 2023-11-24T08:14:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e801555c0 0x7f5e80155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e80151730 0x7f5e80154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801512e0 con 0x7f5e680039c0 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801535a0 con 0x7f5e80151730 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e85883640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80156320 con 0x7f5e801555c0 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36698/0 (socket says 172.21.15.107:36698) 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.876+0000 7f5e7effd640 1 -- 172.21.15.107:0/3697788011 learned_addr learned my addr 172.21.15.107:0/3697788011 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 -- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e801555c0 msgr2=0x7f5e80155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e801555c0 0x7f5e80155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 -- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e80151730 msgr2=0x7f5e80154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e80151730 0x7f5e80154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 -- 172.21.15.107:0/3697788011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e801532c0 con 0x7f5e680039c0 2023-11-24T08:14:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e7effd640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f5e70005040 tx=0x7f5e70001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e700076c0 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e70007a30 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70006c00 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.877+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e800acb70 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.878+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e80152ee0 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.879+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e800ac9d0 con 0x7f5e680039c0 2023-11-24T08:14:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.880+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f5e700035f0 con 0x7f5e680039c0 2023-11-24T08:14:35.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.880+0000 7f5e67fff640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 0x7f5e6c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:35.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.881+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e700aa1c0 con 0x7f5e680039c0 2023-11-24T08:14:35.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.881+0000 7f5e7e7fc640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 0x7f5e6c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:35.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.881+0000 7f5e7e7fc640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 0x7f5e6c06e500 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5e80155400 tx=0x7f5e7400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:35.885+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e800ac9d0 con 0x7f5e680039c0 2023-11-24T08:14:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.117+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5e800ac9d0 con 0x7f5e680039c0 2023-11-24T08:14:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.118+0000 7f5e67fff640 1 -- 172.21.15.107:0/3697788011 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5e800ac9d0 con 0x7f5e680039c0 2023-11-24T08:14:36.119 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:14:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 msgr2=0x7f5e6c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 0x7f5e6c06e500 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5e80155400 tx=0x7f5e7400b040 comp rx=0 tx=0).stop 2023-11-24T08:14:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 msgr2=0x7f5e800adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f5e70005040 tx=0x7f5e70001120 comp rx=0 tx=0).stop 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 shutdown_connections 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5e6c06c040 0x7f5e6c06e500 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e80151730 0x7f5e80154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e801555c0 0x7f5e80155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 --2- 172.21.15.107:0/3697788011 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e680039c0 0x7f5e800adbf0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 >> 172.21.15.107:0/3697788011 conn(0x7f5e8006a920 msgr2=0x7f5e8006c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 shutdown_connections 2023-11-24T08:14:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:36.120+0000 7f5e85883640 1 -- 172.21.15.107:0/3697788011 wait complete. 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:14:36.141 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:14:36.142 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:14:36.142 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:14:36.142 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:14:36.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.142 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:36.193 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41794sec preferred_lft 41794sec 2023-11-24T08:14:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:14:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:14:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:14:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:36.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.196 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:14:36.287 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:14:36.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.291 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:14:36.344 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:14:36.345 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:14:36.345 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:14:36.345 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:14:36.345 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:14:36.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.346 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:14:36.346 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:14:36.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:14:36.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:14:36.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:14:36.347 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:14:36.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.534 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:14:36.589 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:14:36.590 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:14:36.590 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:14:36.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:14:36.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:14:36.683 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:14:36.735 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:14:36.735 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:14:36.735 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:14:36.786 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:14:36.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:36.924 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:36.925 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:14:36.960 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:36.961 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:14:36.961 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:14:36.961 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:14:36.961 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:14:36.962 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:14:36.962 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:14:36.963 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:14:36.963 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:14:36.963 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:14:36.966 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:14:36.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:36.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:14:37.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:37.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:14:37.122 INFO:teuthology.orchestra.run.smithi107.stdout:client6421 2023-11-24T08:14:37.123 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:37.123 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:37.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:14:37.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.387+0000 7f651677a640 1 Processor -- start 2023-11-24T08:14:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.388+0000 7f651677a640 1 -- start start 2023-11-24T08:14:37.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.389+0000 7f651677a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f651015c410 0x7f651015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.389+0000 7f651677a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651015bf80 con 0x7f6510144b30 2023-11-24T08:14:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.389+0000 7f651677a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65100b1560 con 0x7f65100b1de0 2023-11-24T08:14:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.389+0000 7f651677a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65100b1280 con 0x7f651015c410 2023-11-24T08:14:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6515778640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6510144b30 0x7f65100b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36994/0 (socket says 172.21.15.107:36994) 2023-11-24T08:14:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6515778640 1 -- 172.21.15.107:0/113761294 learned_addr learned my addr 172.21.15.107:0/113761294 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499092639 0 0) 0x7f651015bf80 con 0x7f6510144b30 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc003520 con 0x7f6510144b30 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6514f77640 1 --2- 172.21.15.107:0/113761294 >> v2:172.21.15.120:3300/0 conn(0x7f651015c410 0x7f651015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255969905 0 0) 0x7f64fc003520 con 0x7f6510144b30 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 >> v2:172.21.15.120:3300/0 conn(0x7f651015c410 msgr2=0x7f651015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:37.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 --2- 172.21.15.107:0/113761294 >> v2:172.21.15.120:3300/0 conn(0x7f651015c410 0x7f651015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 >> v1:172.21.15.120:6789/0 conn(0x7f65100b1de0 legacy=0x7f65100b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651015b930 con 0x7f6510144b30 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.390+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f650c001670 con 0x7f6510144b30 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 >> v1:172.21.15.107:6789/0 conn(0x7f6510144b30 legacy=0x7f65100b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f64fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc0047c0 0x7f64fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc0051d0 0x7f64fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100b1280 con 0x7f64fc0039c0 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00c120 con 0x7f64fc0051d0 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00c2e0 con 0x7f64fc0047c0 2023-11-24T08:14:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f64fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515f79640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc0051d0 0x7f64fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 -- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc0047c0 msgr2=0x7f64fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc0047c0 0x7f64fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 -- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc0051d0 msgr2=0x7f64fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc0051d0 0x7f64fc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515778640 1 -- 172.21.15.107:0/113761294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc003520 con 0x7f64fc0039c0 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515f79640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc0051d0 0x7f64fc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.391+0000 7f6515f79640 1 -- 172.21.15.107:0/113761294 reap_dead start 2023-11-24T08:14:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.392+0000 7f6515778640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f64fc003de0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c0e0 tx=0x7f64fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0444e56681328f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:37.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c005ad0 con 0x7f64fc0039c0 2023-11-24T08:14:37.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c005c30 con 0x7f64fc0039c0 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f6507fff640 1 -- 172.21.15.107:0/113761294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c02ca50 con 0x7f64fc0039c0 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 -- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 msgr2=0x7f64fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f64fc003de0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c0e0 tx=0x7f64fc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 -- 172.21.15.107:0/113761294 shutdown_connections 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 --2- 172.21.15.107:0/113761294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f64fc003de0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 -- 172.21.15.107:0/113761294 >> 172.21.15.107:0/113761294 conn(0x7f65100cb830 msgr2=0x7f65100cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 -- 172.21.15.107:0/113761294 shutdown_connections 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.393+0000 7f651677a640 1 -- 172.21.15.107:0/113761294 wait complete. 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.395+0000 7f651677a640 1 Processor -- start 2023-11-24T08:14:37.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.395+0000 7f651677a640 1 -- start start 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 0x7f65100c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100cd880 0x7f651015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c97b0 con 0x7f64fc0039c0 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c9a90 con 0x7f65100ca850 2023-11-24T08:14:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f651677a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c9d70 con 0x7f65100cd880 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f6515778640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f6515778640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36706/0 (socket says 172.21.15.107:36706) 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.396+0000 7f6515778640 1 -- 172.21.15.107:0/865091234 learned_addr learned my addr 172.21.15.107:0/865091234 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6514f77640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 0x7f65100c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515f79640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100cd880 0x7f651015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 -- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100cd880 msgr2=0x7f651015a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100cd880 0x7f651015a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 -- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 msgr2=0x7f65100c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 0x7f65100c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 -- 172.21.15.107:0/865091234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100ca050 con 0x7f64fc0039c0 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6514f77640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 0x7f65100c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6515778640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c0e0 tx=0x7f650c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.397+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c03c040 con 0x7f64fc0039c0 2023-11-24T08:14:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.398+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c02cd60 con 0x7f64fc0039c0 2023-11-24T08:14:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.398+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65100ca330 con 0x7f64fc0039c0 2023-11-24T08:14:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.398+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0072b0 con 0x7f64fc0039c0 2023-11-24T08:14:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.398+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65100cdd00 con 0x7f64fc0039c0 2023-11-24T08:14:37.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.399+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65100cac30 con 0x7f64fc0039c0 2023-11-24T08:14:37.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.400+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f650c032070 con 0x7f64fc0039c0 2023-11-24T08:14:37.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.401+0000 7f6505ffb640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 0x7f64f406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:37.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.401+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f650c0a96c0 con 0x7f64fc0039c0 2023-11-24T08:14:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.401+0000 7f6514f77640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 0x7f64f406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.401+0000 7f6514f77640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 0x7f64f406e500 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6500006a90 tx=0x7f6500008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:37.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.405+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f650c076220 con 0x7f64fc0039c0 2023-11-24T08:14:37.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.632+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65100cdfb0 con 0x7f64fc0039c0 2023-11-24T08:14:37.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.633+0000 7f6505ffb640 1 -- 172.21.15.107:0/865091234 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f650c079840 con 0x7f64fc0039c0 2023-11-24T08:14:37.635 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:14:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 msgr2=0x7f64f406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 0x7f64f406e500 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6500006a90 tx=0x7f6500008040 comp rx=0 tx=0).stop 2023-11-24T08:14:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 msgr2=0x7f65100c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c0e0 tx=0x7f650c02e070 comp rx=0 tx=0).stop 2023-11-24T08:14:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 shutdown_connections 2023-11-24T08:14:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64f406c040 0x7f64f406e500 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65100cd880 0x7f651015a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65100ca850 0x7f65100c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 --2- 172.21.15.107:0/865091234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64fc0039c0 0x7f65100c91d0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 >> 172.21.15.107:0/865091234 conn(0x7f65100cb830 msgr2=0x7f65100cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 shutdown_connections 2023-11-24T08:14:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:37.636+0000 7f651677a640 1 -- 172.21.15.107:0/865091234 wait complete. 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:14:37.658 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:14:37.659 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:14:37.659 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:14:37.659 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:14:37.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:37.661 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:14:37.722 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:14:37.723 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:14:37.723 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:14:37.723 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.723 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:14:37.723 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.724 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:37.724 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:37.724 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:14:37.724 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:37.724 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:14:37.725 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41791sec preferred_lft 41791sec 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:14:37.726 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:14:37.727 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.728 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:14:37.728 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:14:37.728 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:14:37.728 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:14:37.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:37.729 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:14:37.817 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:14:37.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:37.822 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:14:37.879 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:14:37.879 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:14:37.879 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:14:37.880 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:14:37.880 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:14:37.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:14:37.881 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:14:38.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:38.065 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:14:38.114 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:14:38.114 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:14:38.114 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:14:38.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:38.115 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:14:38.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:14:38.258 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:14:38.311 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:14:38.311 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:14:38.312 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:14:38.361 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:14:38.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:38.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:38.502 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:38.502 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:14:38.534 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:14:38.534 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:14:38.535 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:14:38.535 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:14:38.536 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:14:38.538 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:14:38.538 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:14:38.538 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:14:38.538 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:14:38.538 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:14:38.542 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:14:38.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:38.542 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:14:38.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:38.622 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:14:38.675 INFO:teuthology.orchestra.run.smithi120.stdout:client8898 2023-11-24T08:14:38.675 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:14:38.675 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:38.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:14:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.954+0000 7f25b1683640 1 Processor -- start 2023-11-24T08:14:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.955+0000 7f25b1683640 1 -- start start 2023-11-24T08:14:38.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25b1683640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25ac159ce0 0x7f25ac06de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25b1683640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 0x7f25ac0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25b1683640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0b3a40 con 0x7f25ac0cbf80 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25b1683640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac0cc410 con 0x7f25ac159ce0 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25b1683640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0cdfa0 con 0x7f25ac0b21d0 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25ab7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25ac0cbf80 0x7f25ac15d760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51222/0 (socket says 172.21.15.107:51222) 2023-11-24T08:14:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.956+0000 7f25ab7fe640 1 -- 172.21.15.107:0/3677211639 learned_addr learned my addr 172.21.15.107:0/3677211639 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aaffd640 1 --2- 172.21.15.107:0/3677211639 >> v2:172.21.15.120:3300/0 conn(0x7f25ac159ce0 0x7f25ac06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89295925 0 0) 0x7f25ac0b3a40 con 0x7f25ac0cbf80 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003440 con 0x7f25ac0cbf80 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25a3fff640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 0x7f25ac0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1800302196 0 0) 0x7f2598003440 con 0x7f25ac0cbf80 2023-11-24T08:14:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 msgr2=0x7f25ac0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 0x7f25ac0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 >> v2:172.21.15.120:3300/0 conn(0x7f25ac159ce0 msgr2=0x7f25ac06de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 --2- 172.21.15.107:0/3677211639 >> v2:172.21.15.120:3300/0 conn(0x7f25ac159ce0 0x7f25ac06de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0cd9e0 con 0x7f25ac0cbf80 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f25a40019f0 con 0x7f25ac0cbf80 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 >> v1:172.21.15.107:6789/0 conn(0x7f25ac0cbf80 legacy=0x7f25ac15d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.957+0000 7f25aa7fc640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f2598003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aa7fc640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2598008850 0x7f259800ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aa7fc640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 0x7f259800d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0cc410 con 0x7f2598003a60 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0cdfa0 con 0x7f2598008850 2023-11-24T08:14:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259800db90 con 0x7f25ac0b21d0 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f2598003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 -- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 msgr2=0x7f259800d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b21d0 0x7f259800d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 -- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2598008850 msgr2=0x7f259800ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25a3fff640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2598008850 0x7f259800ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2598008850 0x7f259800ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.958+0000 7f25aaffd640 1 -- 172.21.15.107:0/3677211639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598003440 con 0x7f2598003a60 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.959+0000 7f25aaffd640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f2598003e80 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ac0 tx=0x7f25980036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0542b8ceaa02dc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.959+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02c710 con 0x7f2598003a60 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c02c870 con 0x7f2598003a60 2023-11-24T08:14:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25aa7fc640 1 -- 172.21.15.107:0/3677211639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02cc00 con 0x7f2598003a60 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 -- 172.21.15.107:0/3677211639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 msgr2=0x7f2598003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 --2- 172.21.15.107:0/3677211639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f2598003e80 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ac0 tx=0x7f25980036f0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25ab7fe640 1 -- 172.21.15.107:0/3677211639 reap_dead start 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 -- 172.21.15.107:0/3677211639 shutdown_connections 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 -- 172.21.15.107:0/3677211639 >> 172.21.15.107:0/3677211639 conn(0x7f25ac06a920 msgr2=0x7f25ac06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 -- 172.21.15.107:0/3677211639 shutdown_connections 2023-11-24T08:14:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.960+0000 7f25b1683640 1 -- 172.21.15.107:0/3677211639 wait complete. 2023-11-24T08:14:38.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.962+0000 7f25b1683640 1 Processor -- start 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.962+0000 7f25b1683640 1 -- start start 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b0e70 0x7f25ac1efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac1eed30 0x7f25ac1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac14c100 con 0x7f2598003a60 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1f0b50 con 0x7f25ac1eed30 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25b1683640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1f0430 con 0x7f25ac0b0e70 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48032/0 (socket says 172.21.15.107:48032) 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.963+0000 7f25aaffd640 1 -- 172.21.15.107:0/2574097060 learned_addr learned my addr 172.21.15.107:0/2574097060 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 -- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b0e70 msgr2=0x7f25ac1efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b0e70 0x7f25ac1efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 -- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac1eed30 msgr2=0x7f25ac1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac1eed30 0x7f25ac1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 -- 172.21.15.107:0/2574097060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac14c2e0 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25aaffd640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f259c02c2b0 tx=0x7f259c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c003810 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c02de90 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac14c5c0 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c003b70 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.964+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac14cb80 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.965+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac067470 con 0x7f2598003a60 2023-11-24T08:14:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.966+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f259c003cd0 con 0x7f2598003a60 2023-11-24T08:14:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.967+0000 7f25a37fe640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 0x7f258c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.967+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f259c0aa080 con 0x7f2598003a60 2023-11-24T08:14:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.968+0000 7f25a3fff640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 0x7f258c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.968+0000 7f25a3fff640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 0x7f258c06e500 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2594006a90 tx=0x7f2594006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:38.972+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f259c076b80 con 0x7f2598003a60 2023-11-24T08:14:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:39.208+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25ac0ce180 con 0x7f2598003a60 2023-11-24T08:14:40.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.078+0000 7f25a37fe640 1 -- 172.21.15.107:0/2574097060 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 136+0+0 (secure 0 0 0) 0x7f259c07a1a0 con 0x7f2598003a60 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 msgr2=0x7f258c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 0x7f258c06e500 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2594006a90 tx=0x7f2594006a20 comp rx=0 tx=0).stop 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 msgr2=0x7f25ac1f0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f259c02c2b0 tx=0x7f259c004730 comp rx=0 tx=0).stop 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 shutdown_connections 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f258c06c040 0x7f258c06e500 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac1eed30 0x7f25ac1506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b0e70 0x7f25ac1efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 --2- 172.21.15.107:0/2574097060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2598003a60 0x7f25ac1f0610 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 >> 172.21.15.107:0/2574097060 conn(0x7f25ac06a920 msgr2=0x7f25ac0abfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 shutdown_connections 2023-11-24T08:14:40.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.082+0000 7f25b1683640 1 -- 172.21.15.107:0/2574097060 wait complete. 2023-11-24T08:14:40.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:14:40.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.313+0000 7f646cb2c640 1 Processor -- start 2023-11-24T08:14:40.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.314+0000 7f646cb2c640 1 -- start start 2023-11-24T08:14:40.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.315+0000 7f646cb2c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64680c59b0 0x7f6468166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.315+0000 7f646cb2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64680c5dc0 con 0x7f64680cc0f0 2023-11-24T08:14:40.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.315+0000 7f646cb2c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646815eea0 con 0x7f646815b440 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.315+0000 7f646cb2c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646815b820 con 0x7f64680c59b0 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.316+0000 7f6465d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f64680cc0f0 0x7f64680c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51224/0 (socket says 172.21.15.107:51224) 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.316+0000 7f6465d9b640 1 -- 172.21.15.107:0/2684015638 learned_addr learned my addr 172.21.15.107:0/2684015638 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.316+0000 7f6466d9d640 1 --2- 172.21.15.107:0/2684015638 >> v2:172.21.15.120:3300/0 conn(0x7f64680c59b0 0x7f6468166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.316+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959607658 0 0) 0x7f64680c5dc0 con 0x7f64680cc0f0 2023-11-24T08:14:40.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.316+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454003520 con 0x7f64680cc0f0 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407180473 0 0) 0x7f6454003520 con 0x7f64680cc0f0 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 >> v2:172.21.15.120:3300/0 conn(0x7f64680c59b0 msgr2=0x7f6468166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 --2- 172.21.15.107:0/2684015638 >> v2:172.21.15.120:3300/0 conn(0x7f64680c59b0 0x7f6468166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 >> v1:172.21.15.120:6789/0 conn(0x7f646815b440 legacy=0x7f64680cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646806d600 con 0x7f64680cc0f0 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f64500019f0 con 0x7f64680cc0f0 2023-11-24T08:14:40.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 >> v1:172.21.15.107:6789/0 conn(0x7f64680cc0f0 legacy=0x7f64680c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 0x7f6454003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64540047c0 0x7f6454004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454005180 0x7f645400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646815b820 con 0x7f6454003970 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c120 con 0x7f64540047c0 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c2e0 con 0x7f6454005180 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.317+0000 7f646659c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 0x7f6454003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f6465d9b640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64540047c0 0x7f6454004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f6466d9d640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454005180 0x7f645400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f646659c640 1 -- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454005180 msgr2=0x7f645400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f646659c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454005180 0x7f645400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f646659c640 1 -- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64540047c0 msgr2=0x7f6454004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f646659c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64540047c0 0x7f6454004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f646659c640 1 -- 172.21.15.107:0/2684015638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454003520 con 0x7f6454003970 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f6466d9d640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454005180 0x7f645400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.318+0000 7f6466d9d640 1 -- 172.21.15.107:0/2684015638 reap_dead start 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.319+0000 7f646659c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 0x7f6454003d90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6458001750 tx=0x7f64540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930e7b2f6b3b4c9e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.319+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64580052d0 con 0x7f6454003970 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.319+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645802e050 con 0x7f6454003970 2023-11-24T08:14:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.319+0000 7f646559a640 1 -- 172.21.15.107:0/2684015638 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458003830 con 0x7f6454003970 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 -- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 msgr2=0x7f6454003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 0x7f6454003d90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6458001750 tx=0x7f64540057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 -- 172.21.15.107:0/2684015638 shutdown_connections 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 --2- 172.21.15.107:0/2684015638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6454003970 0x7f6454003d90 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 -- 172.21.15.107:0/2684015638 >> 172.21.15.107:0/2684015638 conn(0x7f64680b0c50 msgr2=0x7f64680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 -- 172.21.15.107:0/2684015638 shutdown_connections 2023-11-24T08:14:40.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.320+0000 7f646cb2c640 1 -- 172.21.15.107:0/2684015638 wait complete. 2023-11-24T08:14:40.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.322+0000 7f646cb2c640 1 Processor -- start 2023-11-24T08:14:40.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.322+0000 7f646cb2c640 1 -- start start 2023-11-24T08:14:40.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454003970 0x7f646814c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 0x7f64680b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646814cf00 con 0x7f64680b34f0 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680b3060 con 0x7f6468156900 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646cb2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64680b21f0 con 0x7f6454003970 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48040/0 (socket says 172.21.15.107:48040) 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 -- 172.21.15.107:0/1943636729 learned_addr learned my addr 172.21.15.107:0/1943636729 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f646659c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454003970 0x7f646814c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6465d9b640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 0x7f64680b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 -- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454003970 msgr2=0x7f646814c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454003970 0x7f646814c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 -- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 msgr2=0x7f64680b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 0x7f64680b1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6466d9d640 1 -- 172.21.15.107:0/1943636729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680b2450 con 0x7f64680b34f0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.323+0000 7f6465d9b640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 0x7f64680b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f6466d9d640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f646806e890 tx=0x7f6460010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646000e840 con 0x7f64680b34f0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646000e9a0 con 0x7f64680b34f0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64680b2730 con 0x7f64680b34f0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646000ed30 con 0x7f64680b34f0 2023-11-24T08:14:40.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.324+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64680b2cf0 con 0x7f64680b34f0 2023-11-24T08:14:40.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.325+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64680b2e80 con 0x7f64680b34f0 2023-11-24T08:14:40.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.326+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f6460021460 con 0x7f64680b34f0 2023-11-24T08:14:40.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.328+0000 7f645effd640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 0x7f644806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.328+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f64600883c0 con 0x7f64680b34f0 2023-11-24T08:14:40.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.328+0000 7f646659c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 0x7f644806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.329+0000 7f646659c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 0x7f644806e530 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6458033000 tx=0x7f6458009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.333+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6460054c60 con 0x7f64680b34f0 2023-11-24T08:14:40.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.542+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f64680c5330 con 0x7f644806c070 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.545+0000 7f645effd640 1 -- 172.21.15.107:0/1943636729 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f64680c5330 con 0x7f644806c070 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:40.547 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:14:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 msgr2=0x7f644806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 0x7f644806e530 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6458033000 tx=0x7f6458009030 comp rx=0 tx=0).stop 2023-11-24T08:14:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 msgr2=0x7f64680afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f646806e890 tx=0x7f6460010c20 comp rx=0 tx=0).stop 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 shutdown_connections 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f644806c070 0x7f644806e530 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64680b34f0 0x7f64680afd50 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6468156900 0x7f64680b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 --2- 172.21.15.107:0/1943636729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6454003970 0x7f646814c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 >> 172.21.15.107:0/1943636729 conn(0x7f64680b0c50 msgr2=0x7f6468071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.547+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 shutdown_connections 2023-11-24T08:14:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.548+0000 7f646cb2c640 1 -- 172.21.15.107:0/1943636729 wait complete. 2023-11-24T08:14:40.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:14:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.779+0000 7fd986c13640 1 Processor -- start 2023-11-24T08:14:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.779+0000 7fd986c13640 1 -- start start 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986c13640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd98006d080 0x7fd9800ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986c13640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd9800ac840 0x7fd9800ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986c13640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800cdf70 con 0x7fd98015d5f0 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986c13640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9800cb3d0 con 0x7fd98006d080 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986c13640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9800cbfa0 con 0x7fd9800ac840 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd985c11640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd98015d5f0 0x7fd98006c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51232/0 (socket says 172.21.15.107:51232) 2023-11-24T08:14:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd985c11640 1 -- 172.21.15.107:0/1616425044 learned_addr learned my addr 172.21.15.107:0/1616425044 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd985410640 1 --2- 172.21.15.107:0/1616425044 >> v2:172.21.15.107:3300/0 conn(0x7fd98006d080 0x7fd9800ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.781+0000 7fd986412640 1 --2- 172.21.15.107:0/1616425044 >> v2:172.21.15.120:3300/0 conn(0x7fd9800ac840 0x7fd9800ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd984c0f640 1 -- 172.21.15.107:0/1616425044 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395674489 0 0) 0x7fd9800cdf70 con 0x7fd98015d5f0 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd984c0f640 1 -- 172.21.15.107:0/1616425044 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c003520 con 0x7fd98015d5f0 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd985410640 1 -- 172.21.15.107:0/1616425044 >> v2:172.21.15.120:3300/0 conn(0x7fd9800ac840 msgr2=0x7fd9800ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd985410640 1 --2- 172.21.15.107:0/1616425044 >> v2:172.21.15.120:3300/0 conn(0x7fd9800ac840 0x7fd9800ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd985410640 1 -- 172.21.15.107:0/1616425044 >> v1:172.21.15.107:6789/0 conn(0x7fd98015d5f0 legacy=0x7fd98006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.782+0000 7fd985410640 1 -- 172.21.15.107:0/1616425044 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800cc280 con 0x7fd98006d080 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.783+0000 7fd985410640 1 --2- 172.21.15.107:0/1616425044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9800ae020 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd9800b4550 tx=0x7fd97002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb8e83c224a7514 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.783+0000 7fd984c0f640 1 -- 172.21.15.107:0/1616425044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97002e070 con 0x7fd98006d080 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.783+0000 7fd984c0f640 1 -- 172.21.15.107:0/1616425044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700315a0 con 0x7fd98006d080 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.783+0000 7fd984c0f640 1 -- 172.21.15.107:0/1616425044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031930 con 0x7fd98006d080 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 -- 172.21.15.107:0/1616425044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 msgr2=0x7fd9800ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 --2- 172.21.15.107:0/1616425044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9800ae020 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd9800b4550 tx=0x7fd97002b840 comp rx=0 tx=0).stop 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 -- 172.21.15.107:0/1616425044 shutdown_connections 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 --2- 172.21.15.107:0/1616425044 >> v2:172.21.15.120:3300/0 conn(0x7fd9800ac840 0x7fd9800ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 --2- 172.21.15.107:0/1616425044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9800ae020 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 -- 172.21.15.107:0/1616425044 >> 172.21.15.107:0/1616425044 conn(0x7fd98006a500 msgr2=0x7fd98006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 -- 172.21.15.107:0/1616425044 shutdown_connections 2023-11-24T08:14:40.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.784+0000 7fd986c13640 1 -- 172.21.15.107:0/1616425044 wait complete. 2023-11-24T08:14:40.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.786+0000 7fd986c13640 1 Processor -- start 2023-11-24T08:14:40.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.786+0000 7fd986c13640 1 -- start start 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9800ac840 0x7fd98015e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98015d5f0 0x7fd9800aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801f6ee0 con 0x7fd98006d080 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801f6260 con 0x7fd98015d5f0 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd986c13640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801f6540 con 0x7fd9800ac840 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48062/0 (socket says 172.21.15.107:48062) 2023-11-24T08:14:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 -- 172.21.15.107:0/1282619519 learned_addr learned my addr 172.21.15.107:0/1282619519 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 -- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9800ac840 msgr2=0x7fd98015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9800ac840 0x7fd98015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 -- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98015d5f0 msgr2=0x7fd9800aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98015d5f0 0x7fd9800aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.787+0000 7fd985c11640 1 -- 172.21.15.107:0/1282619519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970009400 con 0x7fd98006d080 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd985c11640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00c500 tx=0x7fd97c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c005ce0 con 0x7fd98006d080 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c005e40 con 0x7fd98006d080 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9801f6820 con 0x7fd98006d080 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c007330 con 0x7fd98006d080 2023-11-24T08:14:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.788+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd980144ce0 con 0x7fd98006d080 2023-11-24T08:14:40.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.789+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9801f7020 con 0x7fd98006d080 2023-11-24T08:14:40.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.790+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fd97c007550 con 0x7fd98006d080 2023-11-24T08:14:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.791+0000 7fd9767fc640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 0x7fd96406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.791+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd97c087c30 con 0x7fd98006d080 2023-11-24T08:14:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.791+0000 7fd985410640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 0x7fd96406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.792+0000 7fd985410640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 0x7fd96406e500 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd970036000 tx=0x7fd970030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.795+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9801f7020 con 0x7fd98006d080 2023-11-24T08:14:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.974+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd9801f7020 con 0x7fd98006d080 2023-11-24T08:14:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.975+0000 7fd9767fc640 1 -- 172.21.15.107:0/1282619519 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd9801f7020 con 0x7fd98006d080 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.977+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 msgr2=0x7fd96406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.977+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 0x7fd96406e500 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd970036000 tx=0x7fd970030000 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 msgr2=0x7fd9801447a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00c500 tx=0x7fd97c00c9c0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 shutdown_connections 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96406c040 0x7fd96406e500 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98015d5f0 0x7fd9800aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9800ac840 0x7fd98015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 --2- 172.21.15.107:0/1282619519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98006d080 0x7fd9801447a0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 >> 172.21.15.107:0/1282619519 conn(0x7fd98006a500 msgr2=0x7fd9800b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 shutdown_connections 2023-11-24T08:14:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:40.978+0000 7fd986c13640 1 -- 172.21.15.107:0/1282619519 wait complete. 2023-11-24T08:14:40.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol5827 --mode=777 2023-11-24T08:14:41.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.194+0000 7f3dcc8e7640 1 Processor -- start 2023-11-24T08:14:41.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.195+0000 7f3dcc8e7640 1 -- start start 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dcc8e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dc815c7f0 0x7f3dc806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dcc8e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dcc8e7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80cece0 con 0x7f3dc80cf450 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dcc8e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc80cfe40 con 0x7f3dc815c7f0 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dcc8e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8156010 con 0x7f3dc80ce6c0 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dc659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dc815c7f0 0x7f3dc806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.196+0000 7f3dc659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dc815c7f0 0x7f3dc806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48064/0 (socket says 172.21.15.107:48064) 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc659c640 1 -- 172.21.15.107:0/730638984 learned_addr learned my addr 172.21.15.107:0/730638984 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc6d9d640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc659c640 1 -- 172.21.15.107:0/730638984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 msgr2=0x7f3dc80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc659c640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc659c640 1 -- 172.21.15.107:0/730638984 >> v1:172.21.15.120:6789/0 conn(0x7f3dc80cf450 legacy=0x7f3dc80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.197+0000 7f3dc659c640 1 -- 172.21.15.107:0/730638984 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc806d040 con 0x7f3dc815c7f0 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.198+0000 7f3dc659c640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc806db20 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b2084ff7b913fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.198+0000 7f3dc559a640 1 -- 172.21.15.107:0/730638984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8030840 con 0x7f3dc815c7f0 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.198+0000 7f3dc559a640 1 -- 172.21.15.107:0/730638984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db80309a0 con 0x7f3dc815c7f0 2023-11-24T08:14:41.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.198+0000 7f3dc559a640 1 -- 172.21.15.107:0/730638984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8030d30 con 0x7f3dc815c7f0 2023-11-24T08:14:41.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/730638984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 msgr2=0x7f3dc806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc806db20 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802b980 comp rx=0 tx=0).stop 2023-11-24T08:14:41.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/730638984 shutdown_connections 2023-11-24T08:14:41.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/730638984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc806db20 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/730638984 >> 172.21.15.107:0/730638984 conn(0x7f3dc806a920 msgr2=0x7f3dc806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/730638984 shutdown_connections 2023-11-24T08:14:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.199+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/730638984 wait complete. 2023-11-24T08:14:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.201+0000 7f3dcc8e7640 1 Processor -- start 2023-11-24T08:14:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.201+0000 7f3dcc8e7640 1 -- start start 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc8156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 0x7f3dc8156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80b2d50 con 0x7f3dc815c7f0 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80b2180 con 0x7f3dc80cf450 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dcc8e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80b2420 con 0x7f3dc80ce6c0 2023-11-24T08:14:41.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48070/0 (socket says 172.21.15.107:48070) 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc6d9d640 1 -- 172.21.15.107:0/767765145 learned_addr learned my addr 172.21.15.107:0/767765145 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc659c640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc8156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc5d9b640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 0x7f3dc8156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.202+0000 7f3dc6d9d640 1 -- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 msgr2=0x7f3dc8156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc6d9d640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc8156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc6d9d640 1 -- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 msgr2=0x7f3dc8156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc6d9d640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 0x7f3dc8156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc6d9d640 1 -- 172.21.15.107:0/767765145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80094d0 con 0x7f3dc815c7f0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc5d9b640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 0x7f3dc8156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dc6d9d640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000cd20 tx=0x7f3dc000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0003b90 con 0x7f3dc815c7f0 2023-11-24T08:14:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc0003cf0 con 0x7f3dc815c7f0 2023-11-24T08:14:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc80b2700 con 0x7f3dc815c7f0 2023-11-24T08:14:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0004240 con 0x7f3dc815c7f0 2023-11-24T08:14:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.203+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc81570a0 con 0x7f3dc815c7f0 2023-11-24T08:14:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.204+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc80b2f00 con 0x7f3dc815c7f0 2023-11-24T08:14:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.205+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7f3dc00043a0 con 0x7f3dc815c7f0 2023-11-24T08:14:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.206+0000 7f3dbeffd640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 0x7f3da806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.206+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3dc0012070 con 0x7f3dc815c7f0 2023-11-24T08:14:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.206+0000 7f3dc659c640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 0x7f3da806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.207+0000 7f3dc659c640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 0x7f3da806e500 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3db8036000 tx=0x7f3db803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.210+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3dc80b2f00 con 0x7f3dc815c7f0 2023-11-24T08:14:41.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.409+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol5827", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc806e060 con 0x7f3da806c040 2023-11-24T08:14:41.417 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:41.415+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:41.417 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:41.415+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:41.418 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:41.415+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:41.418 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:41.415+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:41.418 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:41.415+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:41.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.496+0000 7f3dbeffd640 1 -- 172.21.15.107:0/767765145 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3db8030ce0 con 0x7f3da806c040 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.498+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 msgr2=0x7f3da806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.498+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 0x7f3da806e500 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3db8036000 tx=0x7f3db803b040 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.498+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 msgr2=0x7f3dc815f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.498+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000cd20 tx=0x7f3dc000a660 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 shutdown_connections 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3da806c040 0x7f3da806e500 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dc815c7f0 0x7f3dc815f130 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80cf450 0x7f3dc8156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 --2- 172.21.15.107:0/767765145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc80ce6c0 0x7f3dc8156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 >> 172.21.15.107:0/767765145 conn(0x7f3dc806a920 msgr2=0x7f3dc806ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 shutdown_connections 2023-11-24T08:14:41.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.499+0000 7f3dcc8e7640 1 -- 172.21.15.107:0/767765145 wait complete. 2023-11-24T08:14:41.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol5827 2023-11-24T08:14:41.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.727+0000 7fc23b55a640 1 Processor -- start 2023-11-24T08:14:41.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.727+0000 7fc23b55a640 1 -- start start 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc23415b440 0x7fc2340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2340cbf20 0x7fc2340c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2340c57e0 0x7fc2340c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2340cc3d0 con 0x7fc2340c57e0 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23415b820 con 0x7fc23415b440 2023-11-24T08:14:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc23b55a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23415eea0 con 0x7fc2340cbf20 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.729+0000 7fc239af7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2340c57e0 0x7fc2340c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc238af5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2340cbf20 0x7fc2340c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc2392f6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc23415b440 0x7fc2340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2340c57e0 0x7fc2340c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48096/0 (socket says 172.21.15.107:48096) 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 -- 172.21.15.107:0/1401066742 learned_addr learned my addr 172.21.15.107:0/1401066742 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 -- 172.21.15.107:0/1401066742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2340cbf20 msgr2=0x7fc2340c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:41.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 --2- 172.21.15.107:0/1401066742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2340cbf20 0x7fc2340c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 -- 172.21.15.107:0/1401066742 >> v2:172.21.15.120:3300/0 conn(0x7fc23415b440 msgr2=0x7fc2340cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 --2- 172.21.15.107:0/1401066742 >> v2:172.21.15.120:3300/0 conn(0x7fc23415b440 0x7fc2340cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.730+0000 7fc239af7640 1 -- 172.21.15.107:0/1401066742 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340abf30 con 0x7fc2340c57e0 2023-11-24T08:14:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.732+0000 7fc239af7640 1 --2- 172.21.15.107:0/1401066742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340c57e0 0x7fc2340c3ec0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc23406e770 tx=0x7fc22c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9761ea3adfb974 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.732+0000 7fc22bfff640 1 -- 172.21.15.107:0/1401066742 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c02c9e0 con 0x7fc2340c57e0 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.732+0000 7fc22bfff640 1 -- 172.21.15.107:0/1401066742 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c02cb40 con 0x7fc2340c57e0 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.732+0000 7fc22bfff640 1 -- 172.21.15.107:0/1401066742 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0343f0 con 0x7fc2340c57e0 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 -- 172.21.15.107:0/1401066742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340c57e0 msgr2=0x7fc2340c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 --2- 172.21.15.107:0/1401066742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340c57e0 0x7fc2340c3ec0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc23406e770 tx=0x7fc22c02be40 comp rx=0 tx=0).stop 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 -- 172.21.15.107:0/1401066742 shutdown_connections 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 --2- 172.21.15.107:0/1401066742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340c57e0 0x7fc2340c3ec0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 --2- 172.21.15.107:0/1401066742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2340cbf20 0x7fc2340c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 --2- 172.21.15.107:0/1401066742 >> v2:172.21.15.120:3300/0 conn(0x7fc23415b440 0x7fc2340cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 -- 172.21.15.107:0/1401066742 >> 172.21.15.107:0/1401066742 conn(0x7fc2340b0c50 msgr2=0x7fc2340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 -- 172.21.15.107:0/1401066742 shutdown_connections 2023-11-24T08:14:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.733+0000 7fc23b55a640 1 -- 172.21.15.107:0/1401066742 wait complete. 2023-11-24T08:14:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.735+0000 7fc23b55a640 1 Processor -- start 2023-11-24T08:14:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.735+0000 7fc23b55a640 1 -- start start 2023-11-24T08:14:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2340c57e0 0x7fc2340ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc23415b440 0x7fc23415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340b35e0 con 0x7fc2340cbf20 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340b38c0 con 0x7fc2340c57e0 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc23b55a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340b3ba0 con 0x7fc23415b440 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc238af5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc238af5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48104/0 (socket says 172.21.15.107:48104) 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc238af5640 1 -- 172.21.15.107:0/3589681409 learned_addr learned my addr 172.21.15.107:0/3589681409 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.736+0000 7fc2392f6640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2340c57e0 0x7fc2340ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc239af7640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc23415b440 0x7fc23415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 -- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc23415b440 msgr2=0x7fc23415b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc23415b440 0x7fc23415b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 -- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2340c57e0 msgr2=0x7fc2340ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2340c57e0 0x7fc2340ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 -- 172.21.15.107:0/3589681409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c009b30 con 0x7fc2340cbf20 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc238af5640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e890 tx=0x7fc22400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400c940 con 0x7fc2340cbf20 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22400caa0 con 0x7fc2340cbf20 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2341f4270 con 0x7fc2340cbf20 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400ce30 con 0x7fc2340cbf20 2023-11-24T08:14:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.737+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23415bea0 con 0x7fc2340cbf20 2023-11-24T08:14:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.738+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23404e8b0 con 0x7fc2340cbf20 2023-11-24T08:14:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.740+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) v1 ==== 90863+0+0 (secure 0 0 0) 0x7fc224013470 con 0x7fc2340cbf20 2023-11-24T08:14:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.741+0000 7fc229ffb640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 0x7fc21806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.741+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc224088760 con 0x7fc2340cbf20 2023-11-24T08:14:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.741+0000 7fc2392f6640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 0x7fc21806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.742+0000 7fc2392f6640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 0x7fc21806e500 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc230009a30 tx=0x7fc230008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.745+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc224054f50 con 0x7fc2340cbf20 2023-11-24T08:14:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.968+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol5827", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2340b3a30 con 0x7fc21806c040 2023-11-24T08:14:41.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.974+0000 7fc229ffb640 1 -- 172.21.15.107:0/3589681409 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fc2340b3a30 con 0x7fc21806c040 2023-11-24T08:14:41.976 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 msgr2=0x7fc21806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 0x7fc21806e500 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc230009a30 tx=0x7fc230008040 comp rx=0 tx=0).stop 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 msgr2=0x7fc2340ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e890 tx=0x7fc22400ed50 comp rx=0 tx=0).stop 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 shutdown_connections 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc21806c040 0x7fc21806e500 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc23415b440 0x7fc23415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2340cbf20 0x7fc2340ce5a0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 --2- 172.21.15.107:0/3589681409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2340c57e0 0x7fc2340ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 >> 172.21.15.107:0/3589681409 conn(0x7fc2340b0c50 msgr2=0x7fc234071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 shutdown_connections 2023-11-24T08:14:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:41.977+0000 7fc23b55a640 1 -- 172.21.15.107:0/3589681409 wait complete. 2023-11-24T08:14:41.998 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol5827 with 64 files each 1MB size under directory volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459 2023-11-24T08:14:41.998 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:41.998 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.051 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:14:42.147 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:14:42.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.193 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244162 s, 42.9 MB/s 2023-11-24T08:14:42.194 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.195 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.386 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.387 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.387 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262793 s, 39.9 MB/s 2023-11-24T08:14:42.388 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.389 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.445 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.524 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.525 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.525 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236453 s, 44.3 MB/s 2023-11-24T08:14:42.526 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.526 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.584 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258667 s, 40.5 MB/s 2023-11-24T08:14:42.667 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.667 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287169 s, 36.5 MB/s 2023-11-24T08:14:42.809 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.809 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:42.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:42.864 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253748 s, 41.3 MB/s 2023-11-24T08:14:42.947 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:42.947 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257481 s, 40.7 MB/s 2023-11-24T08:14:43.086 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.086 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.144 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234168 s, 44.8 MB/s 2023-11-24T08:14:43.223 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.223 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.282 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204603 s, 51.2 MB/s 2023-11-24T08:14:43.358 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.358 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.495 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.495 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.495 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288797 s, 36.3 MB/s 2023-11-24T08:14:43.496 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.496 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.554 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165395 s, 63.4 MB/s 2023-11-24T08:14:43.626 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.626 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.676 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.801 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.801 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.801 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232737 s, 45.1 MB/s 2023-11-24T08:14:43.803 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.803 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:43.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:43.860 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237597 s, 44.1 MB/s 2023-11-24T08:14:43.942 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:43.942 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.000 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.085 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.085 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.085 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311029 s, 33.7 MB/s 2023-11-24T08:14:44.087 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.087 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.144 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.221 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208596 s, 50.3 MB/s 2023-11-24T08:14:44.221 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.221 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.278 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206899 s, 50.7 MB/s 2023-11-24T08:14:44.351 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.352 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.409 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.491 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.491 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.491 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266702 s, 39.3 MB/s 2023-11-24T08:14:44.492 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.493 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.627 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.627 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.627 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214281 s, 48.9 MB/s 2023-11-24T08:14:44.629 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.629 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.766 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228506 s, 45.9 MB/s 2023-11-24T08:14:44.768 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.769 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.827 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:44.911 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03 s, 35.0 MB/s 2023-11-24T08:14:44.913 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:44.914 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:44.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:44.971 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223593 s, 46.9 MB/s 2023-11-24T08:14:45.051 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.052 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.105 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.224 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.225 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.225 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202923 s, 51.7 MB/s 2023-11-24T08:14:45.226 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.226 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.284 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221767 s, 47.3 MB/s 2023-11-24T08:14:45.363 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.363 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.421 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.504 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.504 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.504 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270227 s, 38.8 MB/s 2023-11-24T08:14:45.506 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.506 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.564 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213303 s, 49.2 MB/s 2023-11-24T08:14:45.642 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.642 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.696 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248511 s, 42.2 MB/s 2023-11-24T08:14:45.778 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.778 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:45.896 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:45.896 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018709 s, 56.0 MB/s 2023-11-24T08:14:45.898 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:45.898 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:45.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:45.954 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243895 s, 43.0 MB/s 2023-11-24T08:14:46.032 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.032 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.090 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.171 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.172 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225948 s, 46.4 MB/s 2023-11-24T08:14:46.173 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.173 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.306 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.307 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.307 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224694 s, 46.7 MB/s 2023-11-24T08:14:46.308 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.308 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.366 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.442 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.442 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.442 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213504 s, 49.1 MB/s 2023-11-24T08:14:46.444 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.444 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.500 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.580 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.580 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237087 s, 44.2 MB/s 2023-11-24T08:14:46.581 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.581 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.640 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.719 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.719 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.719 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238488 s, 44.0 MB/s 2023-11-24T08:14:46.720 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.721 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.777 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.854 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.854 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.854 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192001 s, 54.6 MB/s 2023-11-24T08:14:46.855 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.855 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:46.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:46.911 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:46.985 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:46.986 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:46.986 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217403 s, 48.2 MB/s 2023-11-24T08:14:46.987 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:46.987 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.121 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.122 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238629 s, 43.9 MB/s 2023-11-24T08:14:47.123 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.123 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.255 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.255 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.255 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195325 s, 53.7 MB/s 2023-11-24T08:14:47.257 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.257 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.316 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235052 s, 44.6 MB/s 2023-11-24T08:14:47.395 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.395 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.536 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.536 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.536 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234316 s, 44.8 MB/s 2023-11-24T08:14:47.538 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.538 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.669 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.669 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021954 s, 47.8 MB/s 2023-11-24T08:14:47.671 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.808 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.808 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.808 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023676 s, 44.3 MB/s 2023-11-24T08:14:47.809 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.809 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.862 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:47.935 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:47.935 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:47.935 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023247 s, 45.1 MB/s 2023-11-24T08:14:47.936 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:47.936 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:47.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:47.992 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.064 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.065 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.065 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209635 s, 50.0 MB/s 2023-11-24T08:14:48.066 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.066 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.124 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.197 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.197 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.197 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178605 s, 58.7 MB/s 2023-11-24T08:14:48.198 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.198 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.258 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218971 s, 47.9 MB/s 2023-11-24T08:14:48.337 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.337 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.397 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.476 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.477 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.477 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217214 s, 48.3 MB/s 2023-11-24T08:14:48.478 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.478 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232001 s, 45.2 MB/s 2023-11-24T08:14:48.617 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.617 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.674 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228442 s, 45.9 MB/s 2023-11-24T08:14:48.754 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.754 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.811 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222269 s, 47.2 MB/s 2023-11-24T08:14:48.883 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:48.884 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:48.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:48.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237372 s, 44.2 MB/s 2023-11-24T08:14:49.019 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.019 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.074 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.151 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.151 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.151 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246924 s, 42.5 MB/s 2023-11-24T08:14:49.153 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.153 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.286 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.286 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.286 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230825 s, 45.4 MB/s 2023-11-24T08:14:49.288 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.288 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210446 s, 49.8 MB/s 2023-11-24T08:14:49.423 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.424 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.480 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.552 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.552 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.553 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176488 s, 59.4 MB/s 2023-11-24T08:14:49.554 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.554 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.683 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.683 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.684 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217011 s, 48.3 MB/s 2023-11-24T08:14:49.685 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.685 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.742 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.819 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.819 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.819 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022193 s, 47.2 MB/s 2023-11-24T08:14:49.820 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.820 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:49.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:49.878 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:49.961 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272286 s, 38.5 MB/s 2023-11-24T08:14:49.962 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:49.962 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.021 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.099 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.100 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.100 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243246 s, 43.1 MB/s 2023-11-24T08:14:50.101 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.101 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.155 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.225 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.226 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.226 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186339 s, 56.3 MB/s 2023-11-24T08:14:50.227 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.227 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.281 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.350 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.351 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.351 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183659 s, 57.1 MB/s 2023-11-24T08:14:50.352 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.352 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.410 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198474 s, 52.8 MB/s 2023-11-24T08:14:50.487 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.488 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.544 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231956 s, 45.2 MB/s 2023-11-24T08:14:50.625 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.625 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.753 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.753 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196589 s, 53.3 MB/s 2023-11-24T08:14:50.754 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:14:50.754 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:14:50.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:14:50.812 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:14:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:14:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:14:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020732 s, 50.6 MB/s 2023-11-24T08:14:50.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol5827 subvol_snap7985 2023-11-24T08:14:51.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.111+0000 7f4db870c640 1 Processor -- start 2023-11-24T08:14:51.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.112+0000 7f4db870c640 1 -- start start 2023-11-24T08:14:51.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db870c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4db00b2400 0x7f4db006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db870c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4db00af400 0x7f4db006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db870c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db00cfed0 con 0x7f4db015d920 2023-11-24T08:14:51.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db870c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db00caae0 con 0x7f4db00af400 2023-11-24T08:14:51.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db870c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db00cadc0 con 0x7f4db00b2400 2023-11-24T08:14:51.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db6ca9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4db00af400 0x7f4db006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.113+0000 7f4db6ca9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4db00af400 0x7f4db006aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55262/0 (socket says 172.21.15.107:55262) 2023-11-24T08:14:51.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.114+0000 7f4db6ca9640 1 -- 172.21.15.107:0/501401396 learned_addr learned my addr 172.21.15.107:0/501401396 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:51.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.114+0000 7f4db6ca9640 1 -- 172.21.15.107:0/501401396 >> v2:172.21.15.120:3300/0 conn(0x7f4db00b2400 msgr2=0x7f4db006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:51.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.114+0000 7f4db6ca9640 1 --2- 172.21.15.107:0/501401396 >> v2:172.21.15.120:3300/0 conn(0x7f4db00b2400 0x7f4db006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.114+0000 7f4db6ca9640 1 -- 172.21.15.107:0/501401396 >> v1:172.21.15.120:6789/0 conn(0x7f4db015d920 legacy=0x7f4db00acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:51.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.114+0000 7f4db6ca9640 1 -- 172.21.15.107:0/501401396 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db00cb0a0 con 0x7f4db00af400 2023-11-24T08:14:51.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.115+0000 7f4db6ca9640 1 --2- 172.21.15.107:0/501401396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db006aa40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f4db00ce420 tx=0x7f4da802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e61409e0c303752 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.115+0000 7f4db54a6640 1 -- 172.21.15.107:0/501401396 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da802cb20 con 0x7f4db00af400 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.115+0000 7f4db54a6640 1 -- 172.21.15.107:0/501401396 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da802cc80 con 0x7f4db00af400 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.115+0000 7f4db54a6640 1 -- 172.21.15.107:0/501401396 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da80343f0 con 0x7f4db00af400 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 -- 172.21.15.107:0/501401396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 msgr2=0x7f4db006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 --2- 172.21.15.107:0/501401396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db006aa40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f4db00ce420 tx=0x7f4da802bf80 comp rx=0 tx=0).stop 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 -- 172.21.15.107:0/501401396 shutdown_connections 2023-11-24T08:14:51.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 --2- 172.21.15.107:0/501401396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db006aa40 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 --2- 172.21.15.107:0/501401396 >> v2:172.21.15.120:3300/0 conn(0x7f4db00b2400 0x7f4db006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 -- 172.21.15.107:0/501401396 >> 172.21.15.107:0/501401396 conn(0x7f4db00999d0 msgr2=0x7f4db00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:51.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 -- 172.21.15.107:0/501401396 shutdown_connections 2023-11-24T08:14:51.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.116+0000 7f4db870c640 1 -- 172.21.15.107:0/501401396 wait complete. 2023-11-24T08:14:51.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.118+0000 7f4db870c640 1 Processor -- start 2023-11-24T08:14:51.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.118+0000 7f4db870c640 1 -- start start 2023-11-24T08:14:51.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db00b2400 0x7f4db00c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db015d920 0x7f4db00d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00c7760 con 0x7f4db00af400 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00c7a40 con 0x7f4db00b2400 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db870c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00c7d20 con 0x7f4db015d920 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db64a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db64a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55268/0 (socket says 172.21.15.107:55268) 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.119+0000 7f4db64a8640 1 -- 172.21.15.107:0/3951343103 learned_addr learned my addr 172.21.15.107:0/3951343103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 -- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db015d920 msgr2=0x7f4db00d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db015d920 0x7f4db00d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 -- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db00b2400 msgr2=0x7f4db00c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db00b2400 0x7f4db00c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 -- 172.21.15.107:0/3951343103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da8009b30 con 0x7f4db00af400 2023-11-24T08:14:51.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db64a8640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b3f0 tx=0x7f4dac00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac011830 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac011990 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db00c8000 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac011d40 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.120+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db00c8800 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.122+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db00c8520 con 0x7f4db00af400 2023-11-24T08:14:51.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.123+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 27) v1 ==== 90914+0+0 (secure 0 0 0) 0x7f4dac01a400 con 0x7f4db00af400 2023-11-24T08:14:51.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.123+0000 7f4da6ffd640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 0x7f4d9406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.124+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4dac088410 con 0x7f4db00af400 2023-11-24T08:14:51.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.124+0000 7f4db5ca7640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 0x7f4d9406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.124+0000 7f4db5ca7640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 0x7f4d9406e590 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4db015b2f0 tx=0x7f4da0006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.128+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4dac054bd0 con 0x7f4db00af400 2023-11-24T08:14:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.347+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol5827", "snap_name": "subvol_snap7985", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db00ca770 con 0x7f4d9406c0d0 2023-11-24T08:14:51.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.396+0000 7f4da6ffd640 1 -- 172.21.15.107:0/3951343103 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4da0013400 con 0x7f4d9406c0d0 2023-11-24T08:14:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.398+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 msgr2=0x7f4d9406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.398+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 0x7f4d9406e590 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4db015b2f0 tx=0x7f4da0006c90 comp rx=0 tx=0).stop 2023-11-24T08:14:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.398+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 msgr2=0x7f4db014c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.398+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b3f0 tx=0x7f4dac00b8b0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 shutdown_connections 2023-11-24T08:14:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d9406c0d0 0x7f4d9406e590 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db015d920 0x7f4db00d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db00b2400 0x7f4db00c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 --2- 172.21.15.107:0/3951343103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4db00af400 0x7f4db014c810 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 >> 172.21.15.107:0/3951343103 conn(0x7f4db00999d0 msgr2=0x7f4db00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:51.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 shutdown_connections 2023-11-24T08:14:51.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.399+0000 7f4db870c640 1 -- 172.21.15.107:0/3951343103 wait complete. 2023-11-24T08:14:51.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 2 2023-11-24T08:14:51.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.636+0000 7f436ea23640 1 Processor -- start 2023-11-24T08:14:51.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.637+0000 7f436ea23640 1 -- start start 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f436ea23640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43680c49f0 0x7f436815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f436ea23640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436815ec90 con 0x7f436806de10 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f436ea23640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43680cd530 con 0x7f4368072160 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f436ea23640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43680c4ed0 con 0x7f43680c49f0 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f43677fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f436806de10 0x7f43680705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48552/0 (socket says 172.21.15.107:48552) 2023-11-24T08:14:51.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.638+0000 7f43677fe640 1 -- 172.21.15.107:0/850561299 learned_addr learned my addr 172.21.15.107:0/850561299 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201480865 0 0) 0x7f4358001490 con 0x7f436806de10 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4354003520 con 0x7f436806de10 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f436cfc0640 1 --2- 172.21.15.107:0/850561299 >> v2:172.21.15.120:3300/0 conn(0x7f43680c49f0 0x7f436815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585825830 0 0) 0x7f4354003520 con 0x7f436806de10 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 >> v2:172.21.15.120:3300/0 conn(0x7f43680c49f0 msgr2=0x7f436815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:51.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 --2- 172.21.15.107:0/850561299 >> v2:172.21.15.120:3300/0 conn(0x7f43680c49f0 0x7f436815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 >> v1:172.21.15.120:6789/0 conn(0x7f4368072160 legacy=0x7f436806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43680c51b0 con 0x7f436806de10 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4358001b80 con 0x7f436806de10 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 >> v1:172.21.15.107:6789/0 conn(0x7f436806de10 legacy=0x7f43680705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f4354003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43540047c0 0x7f4354004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43540051d0 0x7f435400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680c4ed0 con 0x7f43540039c0 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435400e330 con 0x7f43540047c0 2023-11-24T08:14:51.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.639+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435400e4f0 con 0x7f43540051d0 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f4354003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 -- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43540051d0 msgr2=0x7f435400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43540051d0 0x7f435400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 -- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43540047c0 msgr2=0x7f4354004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43540047c0 0x7f4354004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f4367fff640 1 -- 172.21.15.107:0/850561299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354003520 con 0x7f43540039c0 2023-11-24T08:14:51.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.640+0000 7f436cfc0640 1 -- 172.21.15.107:0/850561299 reap_dead start 2023-11-24T08:14:51.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.641+0000 7f4367fff640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f4354003de0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4350001750 tx=0x7f43540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50bc81b1cd113d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.641+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43500052d0 con 0x7f43540039c0 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.641+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435002e050 con 0x7f43540039c0 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.641+0000 7f4366ffd640 1 -- 172.21.15.107:0/850561299 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350003830 con 0x7f43540039c0 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 -- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 msgr2=0x7f4354003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f4354003de0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4350001750 tx=0x7f43540057f0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 -- 172.21.15.107:0/850561299 shutdown_connections 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 --2- 172.21.15.107:0/850561299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f4354003de0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 -- 172.21.15.107:0/850561299 >> 172.21.15.107:0/850561299 conn(0x7f43680b0c50 msgr2=0x7f43680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 -- 172.21.15.107:0/850561299 shutdown_connections 2023-11-24T08:14:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.642+0000 7f436ea23640 1 -- 172.21.15.107:0/850561299 wait complete. 2023-11-24T08:14:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.644+0000 7f436ea23640 1 Processor -- start 2023-11-24T08:14:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.644+0000 7f436ea23640 1 -- start start 2023-11-24T08:14:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f436814c500 0x7f436814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43680cd750 0x7f43680cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680ce110 con 0x7f43540039c0 2023-11-24T08:14:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680b3e40 con 0x7f436814c500 2023-11-24T08:14:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f436ea23640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43680b3550 con 0x7f43680cd750 2023-11-24T08:14:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f4367fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f4367fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55282/0 (socket says 172.21.15.107:55282) 2023-11-24T08:14:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.645+0000 7f4367fff640 1 -- 172.21.15.107:0/4122383354 learned_addr learned my addr 172.21.15.107:0/4122383354 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 -- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43680cd750 msgr2=0x7f43680cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43680cd750 0x7f43680cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 -- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f436814c500 msgr2=0x7f436814f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f436814c500 0x7f436814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 -- 172.21.15.107:0/4122383354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436814fcb0 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4367fff640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f4350000c00 tx=0x7f4350001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43500076c0 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4350007a30 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.646+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350006be0 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.647+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436815d1f0 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.647+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43680b3ad0 con 0x7f43540039c0 2023-11-24T08:14:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.648+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436806eed0 con 0x7f43540039c0 2023-11-24T08:14:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.649+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 27) v1 ==== 90914+0+0 (secure 0 0 0) 0x7f4350006d40 con 0x7f43540039c0 2023-11-24T08:14:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.650+0000 7f4364ff9640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 0x7f434806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.650+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f43500aa190 con 0x7f43540039c0 2023-11-24T08:14:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.650+0000 7f43677fe640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 0x7f434806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.651+0000 7f43677fe640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 0x7f434806e5c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f43580015f0 tx=0x7f4358001560 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.654+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4350076970 con 0x7f43540039c0 2023-11-24T08:14:51.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.843+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f436815e6f0 con 0x7f43540039c0 2023-11-24T08:14:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.850+0000 7f4364ff9640 1 -- 172.21.15.107:0/4122383354 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v36)=0 v36) v1 ==== 130+0+0 (secure 0 0 0) 0x7f4350079f90 con 0x7f43540039c0 2023-11-24T08:14:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.853+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 msgr2=0x7f434806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.853+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 0x7f434806e5c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f43580015f0 tx=0x7f4358001560 comp rx=0 tx=0).stop 2023-11-24T08:14:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 msgr2=0x7f436814cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f4350000c00 tx=0x7f4350001120 comp rx=0 tx=0).stop 2023-11-24T08:14:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 shutdown_connections 2023-11-24T08:14:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f434806c100 0x7f434806e5c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43680cd750 0x7f43680cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f436814c500 0x7f436814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 --2- 172.21.15.107:0/4122383354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540039c0 0x7f436814cd00 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 >> 172.21.15.107:0/4122383354 conn(0x7f43680b0c50 msgr2=0x7f43680cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 shutdown_connections 2023-11-24T08:14:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:51.854+0000 7f436ea23640 1 -- 172.21.15.107:0/4122383354 wait complete. 2023-11-24T08:14:51.874 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:51.872+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:51.874 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:51.872+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:51.875 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:51.872+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:51.875 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:51.872+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:51.875 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:51.872+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:51.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol5827 subvol_snap7985 subvol_clone7555 2023-11-24T08:14:52.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.075+0000 7fd651aaa640 1 Processor -- start 2023-11-24T08:14:52.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.075+0000 7fd651aaa640 1 -- start start 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.076+0000 7fd651aaa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd64c06dd50 0x7fd64c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.076+0000 7fd651aaa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c15e580 con 0x7fd64c070ea0 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.076+0000 7fd651aaa640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c15e860 con 0x7fd64c15a1c0 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd651aaa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c0cd920 con 0x7fd64c06dd50 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd6512a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd64c06dd50 0x7fd64c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd64bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd64c070ea0 0x7fd64c06d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48558/0 (socket says 172.21.15.107:48558) 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd64bfff640 1 -- 172.21.15.107:0/3638729946 learned_addr learned my addr 172.21.15.107:0/3638729946 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd6512a9640 1 -- 172.21.15.107:0/3638729946 >> v1:172.21.15.120:6789/0 conn(0x7fd64c15a1c0 legacy=0x7fd64c15dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd6512a9640 1 -- 172.21.15.107:0/3638729946 >> v1:172.21.15.107:6789/0 conn(0x7fd64c070ea0 legacy=0x7fd64c06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.077+0000 7fd6512a9640 1 -- 172.21.15.107:0/3638729946 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c15eb40 con 0x7fd64c06dd50 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.078+0000 7fd6512a9640 1 --2- 172.21.15.107:0/3638729946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c06dd50 0x7fd64c0b15e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd644009d90 tx=0x7fd64402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618b5f1883f4b2e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd64b7fe640 1 -- 172.21.15.107:0/3638729946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64402ccf0 con 0x7fd64c06dd50 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd64b7fe640 1 -- 172.21.15.107:0/3638729946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64402ce50 con 0x7fd64c06dd50 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd64b7fe640 1 -- 172.21.15.107:0/3638729946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644034660 con 0x7fd64c06dd50 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 -- 172.21.15.107:0/3638729946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c06dd50 msgr2=0x7fd64c0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 --2- 172.21.15.107:0/3638729946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c06dd50 0x7fd64c0b15e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd644009d90 tx=0x7fd64402bf40 comp rx=0 tx=0).stop 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 -- 172.21.15.107:0/3638729946 shutdown_connections 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 --2- 172.21.15.107:0/3638729946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c06dd50 0x7fd64c0b15e0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 -- 172.21.15.107:0/3638729946 >> 172.21.15.107:0/3638729946 conn(0x7fd64c0cbb40 msgr2=0x7fd64c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:52.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 -- 172.21.15.107:0/3638729946 shutdown_connections 2023-11-24T08:14:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.079+0000 7fd651aaa640 1 -- 172.21.15.107:0/3638729946 wait complete. 2023-11-24T08:14:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.081+0000 7fd651aaa640 1 Processor -- start 2023-11-24T08:14:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.081+0000 7fd651aaa640 1 -- start start 2023-11-24T08:14:52.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd64c06dd50 0x7fd64c144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd64c15a1c0 0x7fd64c0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c1f7580 con 0x7fd64c070ea0 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c1f4450 con 0x7fd64c15a1c0 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd651aaa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c1f79a0 con 0x7fd64c06dd50 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.082+0000 7fd64bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55322/0 (socket says 172.21.15.107:55322) 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 -- 172.21.15.107:0/2064225472 learned_addr learned my addr 172.21.15.107:0/2064225472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 -- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd64c06dd50 msgr2=0x7fd64c144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd64c06dd50 0x7fd64c144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 -- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd64c15a1c0 msgr2=0x7fd64c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd64c15a1c0 0x7fd64c0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 -- 172.21.15.107:0/2064225472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644009a70 con 0x7fd64c070ea0 2023-11-24T08:14:52.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd64bfff640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c4d0 tx=0x7fd63800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638005ce0 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638005e40 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638007270 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.083+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c14e130 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.084+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c1f6ec0 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.085+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c04e840 con 0x7fd64c070ea0 2023-11-24T08:14:52.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.086+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 27) v1 ==== 90914+0+0 (secure 0 0 0) 0x7fd6380073d0 con 0x7fd64c070ea0 2023-11-24T08:14:52.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.086+0000 7fd6497fa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 0x7fd63006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.087+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6380878a0 con 0x7fd64c070ea0 2023-11-24T08:14:52.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.087+0000 7fd650aa8640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 0x7fd63006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.088+0000 7fd650aa8640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 0x7fd63006e590 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd640006fd0 tx=0x7fd640006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.091+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6380540d0 con 0x7fd64c070ea0 2023-11-24T08:14:52.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.318+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol5827", "snap_name": "subvol_snap7985", "target_sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7fd64c0ceaf0 con 0x7fd63006c0d0 2023-11-24T08:14:52.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.463+0000 7fd6497fa640 1 -- 172.21.15.107:0/2064225472 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd64c0ceaf0 con 0x7fd63006c0d0 2023-11-24T08:14:52.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 msgr2=0x7fd63006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 0x7fd63006e590 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd640006fd0 tx=0x7fd640006bc0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 msgr2=0x7fd64c0afa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c4d0 tx=0x7fd63800c990 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 shutdown_connections 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd63006c0d0 0x7fd63006e590 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd64c15a1c0 0x7fd64c0affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd64c070ea0 0x7fd64c0afa80 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 --2- 172.21.15.107:0/2064225472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd64c06dd50 0x7fd64c144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 >> 172.21.15.107:0/2064225472 conn(0x7fd64c0cbb40 msgr2=0x7fd64c0ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.467+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 shutdown_connections 2023-11-24T08:14:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.468+0000 7fd651aaa640 1 -- 172.21.15.107:0/2064225472 wait complete. 2023-11-24T08:14:52.478 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.476+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.478 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.476+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.478 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.476+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.478 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.476+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.478 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.476+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone7555 2023-11-24T08:14:52.508 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.506+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.508 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.506+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.508 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.506+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.508 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.506+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.508 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.506+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:14:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.682+0000 7fd98d007640 1 Processor -- start 2023-11-24T08:14:52.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.682+0000 7fd98d007640 1 -- start start 2023-11-24T08:14:52.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.683+0000 7fd98d007640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd988072160 0x7fd98815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.683+0000 7fd98d007640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd98806de10 0x7fd9880705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.683+0000 7fd98d007640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98815ec90 con 0x7fd9880c49f0 2023-11-24T08:14:52.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.683+0000 7fd98d007640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9880cd530 con 0x7fd98806de10 2023-11-24T08:14:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.683+0000 7fd98d007640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880c4ed0 con 0x7fd988072160 2023-11-24T08:14:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.684+0000 7fd98659c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd9880c49f0 0x7fd98806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:55168/0 (socket says 172.21.15.107:55168) 2023-11-24T08:14:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.684+0000 7fd98759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd98806de10 0x7fd9880705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.684+0000 7fd986d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd988072160 0x7fd98815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.684+0000 7fd98659c640 1 -- 172.21.15.107:0/1974929661 learned_addr learned my addr 172.21.15.107:0/1974929661 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:52.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.685+0000 7fd98759e640 1 -- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd988072160 msgr2=0x7fd98815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:52.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.685+0000 7fd98759e640 1 --2- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd988072160 0x7fd98815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.685+0000 7fd98759e640 1 -- 172.21.15.107:0/1974929661 >> v1:172.21.15.120:6789/0 conn(0x7fd9880c49f0 legacy=0x7fd98806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.685+0000 7fd98759e640 1 -- 172.21.15.107:0/1974929661 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9880c51b0 con 0x7fd98806de10 2023-11-24T08:14:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.687+0000 7fd98759e640 1 --2- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98806de10 0x7fd9880705c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd980009d90 tx=0x7fd98002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fad8dc0dfb4e997d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd985d9b640 1 -- 172.21.15.107:0/1974929661 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98002cc60 con 0x7fd98806de10 2023-11-24T08:14:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd985d9b640 1 -- 172.21.15.107:0/1974929661 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98002cdc0 con 0x7fd98806de10 2023-11-24T08:14:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd985d9b640 1 -- 172.21.15.107:0/1974929661 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980034660 con 0x7fd98806de10 2023-11-24T08:14:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 -- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98806de10 msgr2=0x7fd9880705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 --2- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98806de10 0x7fd9880705c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd980009d90 tx=0x7fd98002beb0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 -- 172.21.15.107:0/1974929661 shutdown_connections 2023-11-24T08:14:52.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 --2- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd98806de10 0x7fd9880705c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 --2- 172.21.15.107:0/1974929661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd988072160 0x7fd98815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 -- 172.21.15.107:0/1974929661 >> 172.21.15.107:0/1974929661 conn(0x7fd9880b0c50 msgr2=0x7fd9880b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:52.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 -- 172.21.15.107:0/1974929661 shutdown_connections 2023-11-24T08:14:52.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.693+0000 7fd98d007640 1 -- 172.21.15.107:0/1974929661 wait complete. 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.695+0000 7fd98d007640 1 Processor -- start 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.695+0000 7fd98d007640 1 -- start start 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd98806de10 0x7fd9880c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd988072160 0x7fd9880c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9880c49f0 0x7fd9881e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880c6ff0 con 0x7fd9880c49f0 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880c8fc0 con 0x7fd988072160 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.696+0000 7fd98d007640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880c71e0 con 0x7fd98806de10 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9880c49f0 0x7fd9881e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9880c49f0 0x7fd9881e5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55378/0 (socket says 172.21.15.107:55378) 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 -- 172.21.15.107:0/3222888074 learned_addr learned my addr 172.21.15.107:0/3222888074 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98659c640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd988072160 0x7fd9880c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 -- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd98806de10 msgr2=0x7fd9880c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd98806de10 0x7fd9880c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 -- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd988072160 msgr2=0x7fd9880c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd988072160 0x7fd9880c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 -- 172.21.15.107:0/3222888074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd980009a70 con 0x7fd9880c49f0 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98659c640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd988072160 0x7fd9880c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:14:52.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.697+0000 7fd98759e640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9880c49f0 0x7fd9881e5c20 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fd98002c4c0 tx=0x7fd980004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.698+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98003e610 con 0x7fd9880c49f0 2023-11-24T08:14:52.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.698+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98003e770 con 0x7fd9880c49f0 2023-11-24T08:14:52.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.698+0000 7fd98d007640 1 -- 172.21.15.107:0/3222888074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9880c74c0 con 0x7fd9880c49f0 2023-11-24T08:14:52.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.698+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98003eb20 con 0x7fd9880c49f0 2023-11-24T08:14:52.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.698+0000 7fd98d007640 1 -- 172.21.15.107:0/3222888074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9880c7a80 con 0x7fd9880c49f0 2023-11-24T08:14:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.699+0000 7fd98d007640 1 -- 172.21.15.107:0/3222888074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98804e8b0 con 0x7fd9880c49f0 2023-11-24T08:14:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.700+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 27) v1 ==== 90914+0+0 (secure 0 0 0) 0x7fd98003b050 con 0x7fd9880c49f0 2023-11-24T08:14:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.701+0000 7fd96f7fe640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97406c0d0 0x7fd97406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.701+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd9800aaab0 con 0x7fd9880c49f0 2023-11-24T08:14:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.701+0000 7fd986d9d640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97406c0d0 0x7fd97406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:52.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.702+0000 7fd986d9d640 1 --2- 172.21.15.107:0/3222888074 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd97406c0d0 0x7fd97406e590 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd978006a90 tx=0x7fd978006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:52.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.705+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9800776d0 con 0x7fd9880c49f0 2023-11-24T08:14:52.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.926+0000 7fd98d007640 1 -- 172.21.15.107:0/3222888074 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9880c5a50 con 0x7fd97406c0d0 2023-11-24T08:14:52.941 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:14:52.939+0000 7f376379e640 -1 mgr.server reply reply (11) Resource temporarily unavailable subvolume "subvol_clone7555" is not ready for operation "getpath" 2023-11-24T08:14:52.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:52.939+0000 7fd96f7fe640 1 -- 172.21.15.107:0/3222888074 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: -11 subvolume "subvol_clone7555" is not ready for operation "getpath") v1 ==== 73+0+0 (secure 0 0 0) 0x7fd9780096c0 con 0x7fd97406c0d0 2023-11-24T08:14:52.941 INFO:teuthology.orchestra.run.smithi107.stderr:Error EAGAIN: subvolume "subvol_clone7555" is not ready for operation "getpath" 2023-11-24T08:14:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 11 2023-11-24T08:14:52.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7555 2023-11-24T08:14:53.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.212+0000 7fd98ab33640 1 Processor -- start 2023-11-24T08:14:53.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.212+0000 7fd98ab33640 1 -- start start 2023-11-24T08:14:53.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd98ab33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9840cc0f0 0x7fd9840c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd98ab33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd9840c57e0 0x7fd98406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd98ab33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98415eea0 con 0x7fd98415b440 2023-11-24T08:14:53.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd98ab33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98415b820 con 0x7fd9840cc0f0 2023-11-24T08:14:53.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd98ab33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9840abf30 con 0x7fd9840c57e0 2023-11-24T08:14:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd983fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9840cc0f0 0x7fd9840c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd9888cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd98415b440 0x7fd9840cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48574/0 (socket says 172.21.15.107:48574) 2023-11-24T08:14:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.214+0000 7fd9888cf640 1 -- 172.21.15.107:0/4245393548 learned_addr learned my addr 172.21.15.107:0/4245393548 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.215+0000 7fd9890d0640 1 --2- 172.21.15.107:0/4245393548 >> v2:172.21.15.120:3300/0 conn(0x7fd9840c57e0 0x7fd98406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:53.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.215+0000 7fd983fff640 1 -- 172.21.15.107:0/4245393548 >> v2:172.21.15.120:3300/0 conn(0x7fd9840c57e0 msgr2=0x7fd98406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:53.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.215+0000 7fd983fff640 1 --2- 172.21.15.107:0/4245393548 >> v2:172.21.15.120:3300/0 conn(0x7fd9840c57e0 0x7fd98406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.215+0000 7fd983fff640 1 -- 172.21.15.107:0/4245393548 >> v1:172.21.15.107:6789/0 conn(0x7fd98415b440 legacy=0x7fd9840cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.215+0000 7fd983fff640 1 -- 172.21.15.107:0/4245393548 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98415f0f0 con 0x7fd9840cc0f0 2023-11-24T08:14:53.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.216+0000 7fd983fff640 1 --2- 172.21.15.107:0/4245393548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9840cc0f0 0x7fd9840c52a0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fd9840c95e0 tx=0x7fd97002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48710b1ac99645b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.216+0000 7fd9837fe640 1 -- 172.21.15.107:0/4245393548 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97002e070 con 0x7fd9840cc0f0 2023-11-24T08:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd9837fe640 1 -- 172.21.15.107:0/4245393548 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700315a0 con 0x7fd9840cc0f0 2023-11-24T08:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd9837fe640 1 -- 172.21.15.107:0/4245393548 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031930 con 0x7fd9840cc0f0 2023-11-24T08:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 -- 172.21.15.107:0/4245393548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9840cc0f0 msgr2=0x7fd9840c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 --2- 172.21.15.107:0/4245393548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9840cc0f0 0x7fd9840c52a0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fd9840c95e0 tx=0x7fd97002b840 comp rx=0 tx=0).stop 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 -- 172.21.15.107:0/4245393548 shutdown_connections 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 --2- 172.21.15.107:0/4245393548 >> v2:172.21.15.120:3300/0 conn(0x7fd9840c57e0 0x7fd98406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 --2- 172.21.15.107:0/4245393548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9840cc0f0 0x7fd9840c52a0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 -- 172.21.15.107:0/4245393548 >> 172.21.15.107:0/4245393548 conn(0x7fd9840b0c50 msgr2=0x7fd9840b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 -- 172.21.15.107:0/4245393548 shutdown_connections 2023-11-24T08:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.217+0000 7fd98ab33640 1 -- 172.21.15.107:0/4245393548 wait complete. 2023-11-24T08:14:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.219+0000 7fd98ab33640 1 Processor -- start 2023-11-24T08:14:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.219+0000 7fd98ab33640 1 -- start start 2023-11-24T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9840c57e0 0x7fd9841f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9840cc0f0 0x7fd9840ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984153180 con 0x7fd98415b440 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984158d60 con 0x7fd9840c57e0 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd98ab33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9840cd190 con 0x7fd9840cc0f0 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd9890d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd9890d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55394/0 (socket says 172.21.15.107:55394) 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.220+0000 7fd9890d0640 1 -- 172.21.15.107:0/352914191 learned_addr learned my addr 172.21.15.107:0/352914191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 -- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9840cc0f0 msgr2=0x7fd9840ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9840cc0f0 0x7fd9840ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 -- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9840c57e0 msgr2=0x7fd9841f75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9840c57e0 0x7fd9841f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 -- 172.21.15.107:0/352914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970009400 con 0x7fd98415b440 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9890d0640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007e70 tx=0x7fd97c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c014620 con 0x7fd98415b440 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c0147a0 con 0x7fd98415b440 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9840cd470 con 0x7fd98415b440 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.221+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c014b50 con 0x7fd98415b440 2023-11-24T08:14:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.222+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9840cda30 con 0x7fd98415b440 2023-11-24T08:14:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.223+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98406ef60 con 0x7fd98415b440 2023-11-24T08:14:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.223+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 27) v1 ==== 90914+0+0 (secure 0 0 0) 0x7fd97c014cb0 con 0x7fd98415b440 2023-11-24T08:14:53.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.225+0000 7fd9817fa640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 0x7fd96806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:53.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.225+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd97c088f50 con 0x7fd98415b440 2023-11-24T08:14:53.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.225+0000 7fd9888cf640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 0x7fd96806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:53.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.226+0000 7fd9888cf640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 0x7fd96806e5c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ce6e0 tx=0x7fd978007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:53.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.231+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd97c055730 con 0x7fd98415b440 2023-11-24T08:14:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.459+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7fd98415f990 con 0x7fd96806c100 2023-11-24T08:14:53.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.468+0000 7fd9817fa640 1 -- 172.21.15.107:0/352914191 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7fd978001210 con 0x7fd96806c100 2023-11-24T08:14:53.470 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:14:53.470 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:14:53.470 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:14:53.470 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol5827", 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap7985" 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:53.471 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:14:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 msgr2=0x7fd96806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 0x7fd96806e5c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ce6e0 tx=0x7fd978007a40 comp rx=0 tx=0).stop 2023-11-24T08:14:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 msgr2=0x7fd9840ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007e70 tx=0x7fd97c00e4b0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 shutdown_connections 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd96806c100 0x7fd96806e5c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd98415b440 0x7fd9840ad8a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9840cc0f0 0x7fd9840ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 --2- 172.21.15.107:0/352914191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9840c57e0 0x7fd9841f75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 >> 172.21.15.107:0/352914191 conn(0x7fd9840b0c50 msgr2=0x7fd9840713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 shutdown_connections 2023-11-24T08:14:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:53.471+0000 7fd98ab33640 1 -- 172.21.15.107:0/352914191 wait complete. 2023-11-24T08:14:54.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7555 2023-11-24T08:14:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.714+0000 7f25b3a07640 1 Processor -- start 2023-11-24T08:14:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.714+0000 7f25b3a07640 1 -- start start 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b3a07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25ac0d3a40 0x7f25ac06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b3a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac06bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b3a07640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac15db00 con 0x7f25ac15d720 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b3a07640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac0d0ce0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b3a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac06d400 con 0x7f25ac0b1ed0 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b17a3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25ac15d720 0x7f25ac0d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48584/0 (socket says 172.21.15.107:48584) 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b17a3640 1 -- 172.21.15.107:0/2628597490 learned_addr learned my addr 172.21.15.107:0/2628597490 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/2628597490 >> v2:172.21.15.107:3300/0 conn(0x7f25ac0d3a40 0x7f25ac06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b0fa2640 1 -- 172.21.15.107:0/2628597490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 msgr2=0x7f25ac06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/2628597490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac06bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.716+0000 7f25b0fa2640 1 -- 172.21.15.107:0/2628597490 >> v1:172.21.15.107:6789/0 conn(0x7f25ac15d720 legacy=0x7f25ac0d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.717+0000 7f25b0fa2640 1 -- 172.21.15.107:0/2628597490 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0b0100 con 0x7f25ac0d3a40 2023-11-24T08:14:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/2628597490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac06ba80 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0d0e80 tx=0x7f259c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706d6c377e036a1c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25a3fff640 1 -- 172.21.15.107:0/2628597490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c031770 con 0x7f25ac0d3a40 2023-11-24T08:14:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25a3fff640 1 -- 172.21.15.107:0/2628597490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c0318d0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25a3fff640 1 -- 172.21.15.107:0/2628597490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c031c60 con 0x7f25ac0d3a40 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25b3a07640 1 -- 172.21.15.107:0/2628597490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 msgr2=0x7f25ac06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.718+0000 7f25b3a07640 1 --2- 172.21.15.107:0/2628597490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac06ba80 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0d0e80 tx=0x7f259c02b810 comp rx=0 tx=0).stop 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 -- 172.21.15.107:0/2628597490 shutdown_connections 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 --2- 172.21.15.107:0/2628597490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac06bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 --2- 172.21.15.107:0/2628597490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac06ba80 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 -- 172.21.15.107:0/2628597490 >> 172.21.15.107:0/2628597490 conn(0x7f25ac08af60 msgr2=0x7f25ac09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 -- 172.21.15.107:0/2628597490 shutdown_connections 2023-11-24T08:14:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.719+0000 7f25b3a07640 1 -- 172.21.15.107:0/2628597490 wait complete. 2023-11-24T08:14:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.720+0000 7f25b3a07640 1 Processor -- start 2023-11-24T08:14:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.720+0000 7f25b3a07640 1 -- start start 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac15d720 0x7f25ac1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0ccfc0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0c8be0 con 0x7f25ac15d720 2023-11-24T08:14:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.721+0000 7f25b3a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0c7fd0 con 0x7f25ac0b1ed0 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55420/0 (socket says 172.21.15.107:55420) 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 -- 172.21.15.107:0/1806667233 learned_addr learned my addr 172.21.15.107:0/1806667233 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b17a3640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 -- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 msgr2=0x7f25ac0c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac0c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 -- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac15d720 msgr2=0x7f25ac1585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b1fa4640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac15d720 0x7f25ac1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac15d720 0x7f25ac1585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 -- 172.21.15.107:0/1806667233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c009400 con 0x7f25ac0d3a40 2023-11-24T08:14:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.722+0000 7f25b0fa2640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f259c009890 tx=0x7f259c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.723+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c0328d0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.723+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c032a30 con 0x7f25ac0d3a40 2023-11-24T08:14:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.723+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac0c82b0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.723+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c032de0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.723+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac0c8870 con 0x7f25ac0d3a40 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.724+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac0c8d90 con 0x7f25ac0d3a40 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.725+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f259c04e450 con 0x7f25ac0d3a40 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.726+0000 7f25a1ffb640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 0x7f259006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.726+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f259c0aa2b0 con 0x7f25ac0d3a40 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.726+0000 7f25b17a3640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 0x7f259006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.727+0000 7f25b17a3640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 0x7f259006e740 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f25a8005880 tx=0x7f25a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.730+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25ac0c8d90 con 0x7f25ac0d3a40 2023-11-24T08:14:54.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:54.956+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f25ac0cfb80 con 0x7f259006c280 2023-11-24T08:14:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.494+0000 7f25a1ffb640 1 -- 172.21.15.107:0/1806667233 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f25ac0cfb80 con 0x7f259006c280 2023-11-24T08:14:58.496 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:14:58.496 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:14:58.496 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:14:58.496 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:14:58.496 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:14:58.497 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol5827", 2023-11-24T08:14:58.497 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap7985" 2023-11-24T08:14:58.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:58.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:14:58.497 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:14:58.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 msgr2=0x7f259006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:58.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 0x7f259006e740 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f25a8005880 tx=0x7f25a8009040 comp rx=0 tx=0).stop 2023-11-24T08:14:58.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 msgr2=0x7f25ac1578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:58.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f259c009890 tx=0x7f259c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 shutdown_connections 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f259006c280 0x7f259006e740 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ac15d720 0x7f25ac1585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ac0d3a40 0x7f25ac1578e0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 --2- 172.21.15.107:0/1806667233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ac0b1ed0 0x7f25ac0c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.498+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 >> 172.21.15.107:0/1806667233 conn(0x7f25ac08af60 msgr2=0x7f25ac15fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.499+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 shutdown_connections 2023-11-24T08:14:58.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:58.499+0000 7f25b3a07640 1 -- 172.21.15.107:0/1806667233 wait complete. 2023-11-24T08:14:59.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7555 2023-11-24T08:14:59.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.747+0000 7f01a3fff640 1 Processor -- start 2023-11-24T08:14:59.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.747+0000 7f01a3fff640 1 -- start start 2023-11-24T08:14:59.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.749+0000 7f01a3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a40b0b40 0x7f01a40b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.749+0000 7f01a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01a406c5b0 0x7f01a406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.749+0000 7f01a3fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a415ed20 con 0x7f01a415b260 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.749+0000 7f01a3fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a415b640 con 0x7f01a406c5b0 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.749+0000 7f01a3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a415f8a0 con 0x7f01a40b0b40 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01a406c5b0 0x7f01a406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f01a415b260 0x7f01a406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44116/0 (socket says 172.21.15.107:44116) 2023-11-24T08:14:59.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a2ffd640 1 -- 172.21.15.107:0/1960306534 learned_addr learned my addr 172.21.15.107:0/1960306534 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a27fc640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a40b0b40 0x7f01a40b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a37fe640 1 -- 172.21.15.107:0/1960306534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a40b0b40 msgr2=0x7f01a40b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a37fe640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a40b0b40 0x7f01a40b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a37fe640 1 -- 172.21.15.107:0/1960306534 >> v1:172.21.15.107:6789/0 conn(0x7f01a415b260 legacy=0x7f01a406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a37fe640 1 -- 172.21.15.107:0/1960306534 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a40b14c0 con 0x7f01a406c5b0 2023-11-24T08:14:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.750+0000 7f01a1ffb640 1 -- 172.21.15.107:0/1960306534 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679997125 0 0) 0x7f01a415ed20 con 0x7f01a415b260 2023-11-24T08:14:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.751+0000 7f01a37fe640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a406c5b0 0x7f01a406ca40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097a0 tx=0x7f019c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ac13a11fcacb5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.752+0000 7f01a1ffb640 1 -- 172.21.15.107:0/1960306534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c02f050 con 0x7f01a406c5b0 2023-11-24T08:14:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.752+0000 7f01a1ffb640 1 -- 172.21.15.107:0/1960306534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c031700 con 0x7f01a406c5b0 2023-11-24T08:14:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.752+0000 7f01a1ffb640 1 -- 172.21.15.107:0/1960306534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c031a90 con 0x7f01a406c5b0 2023-11-24T08:14:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.752+0000 7f01a3fff640 1 -- 172.21.15.107:0/1960306534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a406c5b0 msgr2=0x7f01a406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.752+0000 7f01a3fff640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a406c5b0 0x7f01a406ca40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097a0 tx=0x7f019c02b7a0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 -- 172.21.15.107:0/1960306534 shutdown_connections 2023-11-24T08:14:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a406c5b0 0x7f01a406ca40 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 --2- 172.21.15.107:0/1960306534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a40b0b40 0x7f01a40b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 -- 172.21.15.107:0/1960306534 >> 172.21.15.107:0/1960306534 conn(0x7f01a40cbb40 msgr2=0x7f01a40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:14:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 -- 172.21.15.107:0/1960306534 shutdown_connections 2023-11-24T08:14:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.753+0000 7f01a3fff640 1 -- 172.21.15.107:0/1960306534 wait complete. 2023-11-24T08:14:59.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.755+0000 7f01a3fff640 1 Processor -- start 2023-11-24T08:14:59.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.755+0000 7f01a3fff640 1 -- start start 2023-11-24T08:14:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a406c5b0 0x7f01a409a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a40b0b40 0x7f01a4151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a409c3b0 con 0x7f01a415b260 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40999d0 con 0x7f01a40b0b40 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.756+0000 7f01a3fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a409a510 con 0x7f01a406c5b0 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49262/0 (socket says 172.21.15.107:49262) 2023-11-24T08:14:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 -- 172.21.15.107:0/4058784952 learned_addr learned my addr 172.21.15.107:0/4058784952 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:14:59.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a2ffd640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a406c5b0 0x7f01a409a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a27fc640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a40b0b40 0x7f01a4151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 -- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a406c5b0 msgr2=0x7f01a409a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a406c5b0 0x7f01a409a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 -- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a40b0b40 msgr2=0x7f01a4151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a40b0b40 0x7f01a4151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.757+0000 7f01a37fe640 1 -- 172.21.15.107:0/4058784952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c009400 con 0x7f01a415b260 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f01a37fe640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097a0 tx=0x7f019c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c033a30 con 0x7f01a415b260 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c033b90 con 0x7f01a415b260 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a4151990 con 0x7f01a415b260 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c004030 con 0x7f01a415b260 2023-11-24T08:14:59.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.758+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a4099ec0 con 0x7f01a415b260 2023-11-24T08:14:59.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.760+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a404e840 con 0x7f01a415b260 2023-11-24T08:14:59.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.761+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f019c03a070 con 0x7f01a415b260 2023-11-24T08:14:59.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.762+0000 7f019bfff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 0x7f018806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:14:59.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.762+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f019c0aa490 con 0x7f01a415b260 2023-11-24T08:14:59.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.762+0000 7f01a2ffd640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 0x7f018806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:14:59.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.763+0000 7f01a2ffd640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 0x7f018806e740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0194000f60 tx=0x7f0194007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:14:59.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:14:59.767+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f019c076b30 con 0x7f01a415b260 2023-11-24T08:15:00.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.016+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f01a415ef70 con 0x7f018806c280 2023-11-24T08:15:00.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.029+0000 7f019bfff640 1 -- 172.21.15.107:0/4058784952 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f0194007270 con 0x7f018806c280 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol5827", 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap7985" 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:15:00.030 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:15:00.031 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 msgr2=0x7f018806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 0x7f018806e740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0194000f60 tx=0x7f0194007c00 comp rx=0 tx=0).stop 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 msgr2=0x7f01a415d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097a0 tx=0x7f019c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 shutdown_connections 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f018806c280 0x7f018806e740 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a415b260 0x7f01a415d790 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a40b0b40 0x7f01a4151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:00.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 --2- 172.21.15.107:0/4058784952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a406c5b0 0x7f01a409a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:00.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 >> 172.21.15.107:0/4058784952 conn(0x7f01a40cbb40 msgr2=0x7f01a40acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:00.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 shutdown_connections 2023-11-24T08:15:00.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:00.032+0000 7f01a3fff640 1 -- 172.21.15.107:0/4058784952 wait complete. 2023-11-24T08:15:01.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7555 2023-11-24T08:15:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.266+0000 7ff56ea21640 1 Processor -- start 2023-11-24T08:15:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.266+0000 7ff56ea21640 1 -- start start 2023-11-24T08:15:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56ea21640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5680b3740 0x7ff56815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56ea21640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff56815f380 0x7ff5680cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56ea21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56815fbb0 con 0x7ff56813df60 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56ea21640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56815bac0 con 0x7ff56815f380 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56ea21640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56815bda0 con 0x7ff5680b3740 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56e220640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff56815f380 0x7ff5680cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.268+0000 7ff56da1f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff56813df60 0x7ff5680b3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44130/0 (socket says 172.21.15.107:44130) 2023-11-24T08:15:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56da1f640 1 -- 172.21.15.107:0/1915792871 learned_addr learned my addr 172.21.15.107:0/1915792871 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56e220640 1 -- 172.21.15.107:0/1915792871 >> v2:172.21.15.120:3300/0 conn(0x7ff5680b3740 msgr2=0x7ff56815ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56e220640 1 --2- 172.21.15.107:0/1915792871 >> v2:172.21.15.120:3300/0 conn(0x7ff5680b3740 0x7ff56815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56e220640 1 -- 172.21.15.107:0/1915792871 >> v1:172.21.15.107:6789/0 conn(0x7ff56813df60 legacy=0x7ff5680b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56e220640 1 -- 172.21.15.107:0/1915792871 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56815c080 con 0x7ff56815f380 2023-11-24T08:15:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.269+0000 7ff56ca1d640 1 -- 172.21.15.107:0/1915792871 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863133490 0 0) 0x7ff56815fbb0 con 0x7ff56813df60 2023-11-24T08:15:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.270+0000 7ff56e220640 1 --2- 172.21.15.107:0/1915792871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56815f380 0x7ff5680cfa90 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff5680d2090 tx=0x7ff56002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d812063fb0f1b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.270+0000 7ff56ca1d640 1 -- 172.21.15.107:0/1915792871 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56002cc80 con 0x7ff56815f380 2023-11-24T08:15:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.271+0000 7ff56ca1d640 1 -- 172.21.15.107:0/1915792871 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56002cde0 con 0x7ff56815f380 2023-11-24T08:15:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.271+0000 7ff56ca1d640 1 -- 172.21.15.107:0/1915792871 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560034660 con 0x7ff56815f380 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.271+0000 7ff56ea21640 1 -- 172.21.15.107:0/1915792871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56815f380 msgr2=0x7ff5680cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.271+0000 7ff56ea21640 1 --2- 172.21.15.107:0/1915792871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56815f380 0x7ff5680cfa90 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff5680d2090 tx=0x7ff56002bed0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 -- 172.21.15.107:0/1915792871 shutdown_connections 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 --2- 172.21.15.107:0/1915792871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56815f380 0x7ff5680cfa90 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 --2- 172.21.15.107:0/1915792871 >> v2:172.21.15.120:3300/0 conn(0x7ff5680b3740 0x7ff56815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 -- 172.21.15.107:0/1915792871 >> 172.21.15.107:0/1915792871 conn(0x7ff568061100 msgr2=0x7ff568061510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 -- 172.21.15.107:0/1915792871 shutdown_connections 2023-11-24T08:15:01.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.272+0000 7ff56ea21640 1 -- 172.21.15.107:0/1915792871 wait complete. 2023-11-24T08:15:01.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.274+0000 7ff56ea21640 1 Processor -- start 2023-11-24T08:15:01.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.274+0000 7ff56ea21640 1 -- start start 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5680b3740 0x7ff56815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff56815f380 0x7ff568153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56806b7b0 con 0x7ff56813df60 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56806ba90 con 0x7ff5680b3740 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56ea21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56806bd70 con 0x7ff56815f380 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49286/0 (socket says 172.21.15.107:49286) 2023-11-24T08:15:01.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 -- 172.21.15.107:0/2661256623 learned_addr learned my addr 172.21.15.107:0/2661256623 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:01.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56da1f640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5680b3740 0x7ff56815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 -- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff56815f380 msgr2=0x7ff568153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff56815f380 0x7ff568153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 -- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5680b3740 msgr2=0x7ff56815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5680b3740 0x7ff56815a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.276+0000 7ff56d21e640 1 -- 172.21.15.107:0/2661256623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560009a70 con 0x7ff56813df60 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff56d21e640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558010610 con 0x7ff56813df60 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff558010770 con 0x7ff56813df60 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558010b20 con 0x7ff56813df60 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56815c720 con 0x7ff56813df60 2023-11-24T08:15:01.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.277+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56806c330 con 0x7ff56813df60 2023-11-24T08:15:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.278+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56806c550 con 0x7ff56813df60 2023-11-24T08:15:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.279+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7ff558010c80 con 0x7ff56813df60 2023-11-24T08:15:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.280+0000 7ff55e7fc640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 0x7ff54c06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.280+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5580882e0 con 0x7ff56813df60 2023-11-24T08:15:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.281+0000 7ff56da1f640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 0x7ff54c06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.281+0000 7ff56da1f640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 0x7ff54c06e740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff564000f60 tx=0x7ff564007b70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:01.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.286+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff558054a10 con 0x7ff56813df60 2023-11-24T08:15:01.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:01.542+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7ff56813d680 con 0x7ff54c06c280 2023-11-24T08:15:02.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.590+0000 7ff55e7fc640 1 -- 172.21.15.107:0/2661256623 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7ff56813d680 con 0x7ff54c06c280 2023-11-24T08:15:02.591 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:15:02.591 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:15:02.591 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:15:02.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:15:02.592 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:15:02.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 msgr2=0x7ff54c06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 0x7ff54c06e740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff564000f60 tx=0x7ff564007b70 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 msgr2=0x7ff56806c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 shutdown_connections 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff54c06c280 0x7ff54c06e740 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff56815f380 0x7ff568153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff56813df60 0x7ff56806c850 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 --2- 172.21.15.107:0/2661256623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5680b3740 0x7ff56815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.593+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 >> 172.21.15.107:0/2661256623 conn(0x7ff568061100 msgr2=0x7ff5681504c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.594+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 shutdown_connections 2023-11-24T08:15:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.594+0000 7ff56ea21640 1 -- 172.21.15.107:0/2661256623 wait complete. 2023-11-24T08:15:02.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone7555 2023-11-24T08:15:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.834+0000 7f75e731d640 1 Processor -- start 2023-11-24T08:15:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.834+0000 7f75e731d640 1 -- start start 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.835+0000 7f75e731d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75e0072160 0x7f75e015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e731d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75e00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e731d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e015ec90 con 0x7f75e00c49f0 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e731d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e00cd530 con 0x7f75e0072160 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e731d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00c4ed0 con 0x7f75e006de10 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e48b8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75e00c49f0 0x7f75e006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44144/0 (socket says 172.21.15.107:44144) 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e48b8640 1 -- 172.21.15.107:0/3145019469 learned_addr learned my addr 172.21.15.107:0/3145019469 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:02.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e50b9640 1 --2- 172.21.15.107:0/3145019469 >> v2:172.21.15.120:3300/0 conn(0x7f75e0072160 0x7f75e015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75e58ba640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75e00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.836+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918644075 0 0) 0x7f75d0001110 con 0x7f75e00c49f0 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc003520 con 0x7f75e00c49f0 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165434547 0 0) 0x7f75cc003520 con 0x7f75e00c49f0 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 msgr2=0x7f75e00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75e00705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 >> v2:172.21.15.120:3300/0 conn(0x7f75e0072160 msgr2=0x7f75e015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 --2- 172.21.15.107:0/3145019469 >> v2:172.21.15.120:3300/0 conn(0x7f75e0072160 0x7f75e015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e00c51b0 con 0x7f75e00c49f0 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f75d0001800 con 0x7f75e00c49f0 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 >> v1:172.21.15.107:6789/0 conn(0x7f75e00c49f0 legacy=0x7f75e006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75cc003970 0x7f75cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00cd530 con 0x7f75cc0047c0 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00c4ed0 con 0x7f75cc003970 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc00b980 con 0x7f75e006de10 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.837+0000 7f75e48b8640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e58ba640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e48b8640 1 -- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 msgr2=0x7f75cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:02.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e48b8640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e48b8640 1 -- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75cc003970 msgr2=0x7f75cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e50b9640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75cc003970 0x7f75cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e48b8640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75cc003970 0x7f75cc003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e48b8640 1 -- 172.21.15.107:0/3145019469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc003520 con 0x7f75cc0047c0 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.838+0000 7f75e58ba640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e006de10 0x7f75cc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.839+0000 7f75e48b8640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75cc004c40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f75d000c210 tx=0x7f75cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b1c3b27e899d2c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.839+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d002cae0 con 0x7f75cc0047c0 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.839+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d002cc60 con 0x7f75cc0047c0 2023-11-24T08:15:02.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.839+0000 7f75d7fff640 1 -- 172.21.15.107:0/3145019469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00058d0 con 0x7f75cc0047c0 2023-11-24T08:15:02.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 -- 172.21.15.107:0/3145019469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 msgr2=0x7f75cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 --2- 172.21.15.107:0/3145019469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75cc004c40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f75d000c210 tx=0x7f75cc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e58ba640 1 -- 172.21.15.107:0/3145019469 reap_dead start 2023-11-24T08:15:02.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 -- 172.21.15.107:0/3145019469 shutdown_connections 2023-11-24T08:15:02.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 -- 172.21.15.107:0/3145019469 >> 172.21.15.107:0/3145019469 conn(0x7f75e00b0c50 msgr2=0x7f75e00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:02.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 -- 172.21.15.107:0/3145019469 shutdown_connections 2023-11-24T08:15:02.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.840+0000 7f75e731d640 1 -- 172.21.15.107:0/3145019469 wait complete. 2023-11-24T08:15:02.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.842+0000 7f75e731d640 1 Processor -- start 2023-11-24T08:15:02.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.842+0000 7f75e731d640 1 -- start start 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e00c8740 0x7f75e015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e00c8bc0 0x7f75e015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e015c5f0 con 0x7f75cc0047c0 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0159530 con 0x7f75e00c8bc0 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e731d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00c7f80 con 0x7f75e00c8740 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e50b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e50b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49304/0 (socket says 172.21.15.107:49304) 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e50b9640 1 -- 172.21.15.107:0/2396878343 learned_addr learned my addr 172.21.15.107:0/2396878343 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.843+0000 7f75e48b8640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e00c8740 0x7f75e015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 -- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e00c8740 msgr2=0x7f75e015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e00c8740 0x7f75e015b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e58ba640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e00c8bc0 0x7f75e015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 -- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e00c8bc0 msgr2=0x7f75e015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e00c8bc0 0x7f75e015ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 -- 172.21.15.107:0/2396878343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e015d150 con 0x7f75cc0047c0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e50b9640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0108e0 tx=0x7f75dc010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00eb60 con 0x7f75cc0047c0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc00ecc0 con 0x7f75cc0047c0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e015c030 con 0x7f75cc0047c0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc01c400 con 0x7f75cc0047c0 2023-11-24T08:15:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.844+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e00c8220 con 0x7f75cc0047c0 2023-11-24T08:15:02.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.846+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e00c7de0 con 0x7f75cc0047c0 2023-11-24T08:15:02.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.846+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f75dc01c560 con 0x7f75cc0047c0 2023-11-24T08:15:02.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.847+0000 7f75d5ffb640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 0x7f75c406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:02.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.848+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f75dc088990 con 0x7f75cc0047c0 2023-11-24T08:15:02.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.848+0000 7f75e48b8640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 0x7f75c406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:02.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.848+0000 7f75e48b8640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 0x7f75c406e6d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f75d0034000 tx=0x7f75d0007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:02.852+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75e00c7de0 con 0x7f75cc0047c0 2023-11-24T08:15:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.070+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f75e015dff0 con 0x7f75c406c210 2023-11-24T08:15:03.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.083+0000 7f75d5ffb640 1 -- 172.21.15.107:0/2396878343 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f75e015dff0 con 0x7f75c406c210 2023-11-24T08:15:03.084 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.085+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 msgr2=0x7f75c406e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 0x7f75c406e6d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f75d0034000 tx=0x7f75d0007000 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 msgr2=0x7f75e0158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0108e0 tx=0x7f75dc010da0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 shutdown_connections 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f75c406c210 0x7f75c406e6d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e00c8bc0 0x7f75e015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75e00c8740 0x7f75e015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 --2- 172.21.15.107:0/2396878343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75cc0047c0 0x7f75e0158ff0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 >> 172.21.15.107:0/2396878343 conn(0x7f75e00b0c50 msgr2=0x7f75e00cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 shutdown_connections 2023-11-24T08:15:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.086+0000 7f75e731d640 1 -- 172.21.15.107:0/2396878343 wait complete. 2023-11-24T08:15:03.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol5827 2023-11-24T08:15:03.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.313+0000 7f0d5590f640 1 Processor -- start 2023-11-24T08:15:03.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.313+0000 7f0d5590f640 1 -- start start 2023-11-24T08:15:03.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.315+0000 7f0d5590f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.315+0000 7f0d5590f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d500cf4f0 0x7f0d500700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.315+0000 7f0d5590f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5015c630 con 0x7f0d5013ea00 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.315+0000 7f0d5590f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5013e7c0 con 0x7f0d500cf4f0 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.315+0000 7f0d5590f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5015b810 con 0x7f0d5013e3e0 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40620/0 (socket says 172.21.15.107:40620) 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 -- 172.21.15.107:0/734727916 learned_addr learned my addr 172.21.15.107:0/734727916 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4f7fe640 1 --2- 172.21.15.107:0/734727916 >> v2:172.21.15.120:3300/0 conn(0x7f0d500cf4f0 0x7f0d500700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 -- 172.21.15.107:0/734727916 >> v2:172.21.15.120:3300/0 conn(0x7f0d500cf4f0 msgr2=0x7f0d500700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 --2- 172.21.15.107:0/734727916 >> v2:172.21.15.120:3300/0 conn(0x7f0d500cf4f0 0x7f0d500700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 -- 172.21.15.107:0/734727916 >> v1:172.21.15.120:6789/0 conn(0x7f0d5013ea00 legacy=0x7f0d500cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.316+0000 7f0d4effd640 1 -- 172.21.15.107:0/734727916 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5015ba20 con 0x7f0d5013e3e0 2023-11-24T08:15:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.318+0000 7f0d4effd640 1 --2- 172.21.15.107:0/734727916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009870 tx=0x7f0d4002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0bd02efcf3f0a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.318+0000 7f0d4dffb640 1 -- 172.21.15.107:0/734727916 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40031640 con 0x7f0d5013e3e0 2023-11-24T08:15:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.318+0000 7f0d4dffb640 1 -- 172.21.15.107:0/734727916 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d400317a0 con 0x7f0d5013e3e0 2023-11-24T08:15:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.318+0000 7f0d4dffb640 1 -- 172.21.15.107:0/734727916 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40031b30 con 0x7f0d5013e3e0 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 -- 172.21.15.107:0/734727916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 msgr2=0x7f0d500ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 --2- 172.21.15.107:0/734727916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009870 tx=0x7f0d4002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 -- 172.21.15.107:0/734727916 shutdown_connections 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 --2- 172.21.15.107:0/734727916 >> v2:172.21.15.120:3300/0 conn(0x7f0d500cf4f0 0x7f0d500700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 --2- 172.21.15.107:0/734727916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013e3e0 0x7f0d500ce3e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 -- 172.21.15.107:0/734727916 >> 172.21.15.107:0/734727916 conn(0x7f0d500b2e40 msgr2=0x7f0d500b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:03.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 -- 172.21.15.107:0/734727916 shutdown_connections 2023-11-24T08:15:03.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.319+0000 7f0d5590f640 1 -- 172.21.15.107:0/734727916 wait complete. 2023-11-24T08:15:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.321+0000 7f0d5590f640 1 Processor -- start 2023-11-24T08:15:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.321+0000 7f0d5590f640 1 -- start start 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d5013e3e0 0x7f0d5013c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013ea00 0x7f0d500c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5013a290 con 0x7f0d500cf4f0 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5013d200 con 0x7f0d5013e3e0 2023-11-24T08:15:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.322+0000 7f0d5590f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500b46b0 con 0x7f0d5013ea00 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49314/0 (socket says 172.21.15.107:49314) 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 -- 172.21.15.107:0/871079622 learned_addr learned my addr 172.21.15.107:0/871079622 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4e7fc640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d5013e3e0 0x7f0d5013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4f7fe640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013ea00 0x7f0d500c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 -- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013ea00 msgr2=0x7f0d500c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013ea00 0x7f0d500c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 -- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d5013e3e0 msgr2=0x7f0d5013c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d5013e3e0 0x7f0d5013c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 -- 172.21.15.107:0/871079622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400094d0 con 0x7f0d500cf4f0 2023-11-24T08:15:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.323+0000 7f0d4effd640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0d4002b930 tx=0x7f0d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.324+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4003a930 con 0x7f0d500cf4f0 2023-11-24T08:15:03.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.324+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4003aa90 con 0x7f0d500cf4f0 2023-11-24T08:15:03.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.324+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5013cf20 con 0x7f0d500cf4f0 2023-11-24T08:15:03.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.324+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4003ae40 con 0x7f0d500cf4f0 2023-11-24T08:15:03.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.324+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d500b4340 con 0x7f0d500cf4f0 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.325+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d501578e0 con 0x7f0d500cf4f0 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.326+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f0d4002f050 con 0x7f0d500cf4f0 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.327+0000 7f0d37fff640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 0x7f0d3c06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.327+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0d400aa0c0 con 0x7f0d500cf4f0 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.327+0000 7f0d4e7fc640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 0x7f0d3c06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.328+0000 7f0d4e7fc640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 0x7f0d3c06e740 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0d501f4710 tx=0x7f0d44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.332+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d40076c20 con 0x7f0d500cf4f0 2023-11-24T08:15:03.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.551+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol5827", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d5015c2c0 con 0x7f0d3c06c280 2023-11-24T08:15:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.572+0000 7f0d37fff640 1 -- 172.21.15.107:0/871079622 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f0d44001360 con 0x7f0d3c06c280 2023-11-24T08:15:03.573 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol5827/4b8b606b-b8db-4e06-97ea-277a5f63a459 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 msgr2=0x7f0d3c06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 0x7f0d3c06e740 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0d501f4710 tx=0x7f0d44009040 comp rx=0 tx=0).stop 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 msgr2=0x7f0d50139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0d4002b930 tx=0x7f0d400046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 shutdown_connections 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d3c06c280 0x7f0d3c06e740 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d5013ea00 0x7f0d500c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d5013e3e0 0x7f0d5013c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 --2- 172.21.15.107:0/871079622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d500cf4f0 0x7f0d50139d50 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 >> 172.21.15.107:0/871079622 conn(0x7f0d500b2e40 msgr2=0x7f0d500700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 shutdown_connections 2023-11-24T08:15:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.575+0000 7f0d5590f640 1 -- 172.21.15.107:0/871079622 wait complete. 2023-11-24T08:15:03.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone7555 2023-11-24T08:15:03.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.813+0000 7f36022ad640 1 Processor -- start 2023-11-24T08:15:03.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.814+0000 7f36022ad640 1 -- start start 2023-11-24T08:15:03.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.815+0000 7f36022ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35fc06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.815+0000 7f36022ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc06dcf0 con 0x7f35fc161060 2023-11-24T08:15:03.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.815+0000 7f36022ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc161ec0 con 0x7f35fc0b3330 2023-11-24T08:15:03.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.815+0000 7f36022ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc06ec10 con 0x7f35fc0b3790 2023-11-24T08:15:03.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.816+0000 7f35fbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f35fc161060 0x7f35fc161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44160/0 (socket says 172.21.15.107:44160) 2023-11-24T08:15:03.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.816+0000 7f35fbfff640 1 -- 172.21.15.107:0/923444180 learned_addr learned my addr 172.21.15.107:0/923444180 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.816+0000 7f360084a640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35fc06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.816+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265155081 0 0) 0x7f35fc06dcf0 con 0x7f35fc161060 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.816+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e8003520 con 0x7f35fc161060 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2470486422 0 0) 0x7f35e8003520 con 0x7f35fc161060 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 msgr2=0x7f35fc06adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35fc06adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 >> v1:172.21.15.120:6789/0 conn(0x7f35fc0b3330 legacy=0x7f35fc06a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc06df90 con 0x7f35fc161060 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f35ec0019f0 con 0x7f35fc161060 2023-11-24T08:15:03.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 >> v1:172.21.15.107:6789/0 conn(0x7f35fc161060 legacy=0x7f35fc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e80039a0 0x7f35e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35e8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc06ec10 con 0x7f35e80047c0 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e800bb10 con 0x7f35e80039a0 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e800bcd0 con 0x7f35fc0b3790 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f360084a640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35e8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35fbfff640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e80039a0 0x7f35e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.817+0000 7f35fb7fe640 1 -- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 msgr2=0x7f35e8005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.818+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3790 0x7f35e8005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.818+0000 7f35fb7fe640 1 -- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e80039a0 msgr2=0x7f35e8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.818+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e80039a0 0x7f35e8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.818+0000 7f35fb7fe640 1 -- 172.21.15.107:0/923444180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8003520 con 0x7f35e80047c0 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35e8004c40 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001770 tx=0x7f35e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180f9b1770e035a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e40052d0 con 0x7f35e80047c0 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e402e050 con 0x7f35e80047c0 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f35faffd640 1 -- 172.21.15.107:0/923444180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4003640 con 0x7f35e80047c0 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f36022ad640 1 -- 172.21.15.107:0/923444180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 msgr2=0x7f35e8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f36022ad640 1 --2- 172.21.15.107:0/923444180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35e8004c40 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001770 tx=0x7f35e80057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f360084a640 1 -- 172.21.15.107:0/923444180 reap_dead start 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f36022ad640 1 -- 172.21.15.107:0/923444180 shutdown_connections 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.819+0000 7f36022ad640 1 -- 172.21.15.107:0/923444180 >> 172.21.15.107:0/923444180 conn(0x7f35fc0cf900 msgr2=0x7f35fc0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.820+0000 7f36022ad640 1 -- 172.21.15.107:0/923444180 shutdown_connections 2023-11-24T08:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.820+0000 7f36022ad640 1 -- 172.21.15.107:0/923444180 wait complete. 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.821+0000 7f36022ad640 1 Processor -- start 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.821+0000 7f36022ad640 1 -- start start 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0cc510 0x7f35fc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1f3540 0x7f35fc1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0cd750 con 0x7f35e80047c0 2023-11-24T08:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0ccad0 con 0x7f35fc1f3540 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.822+0000 7f36022ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0ccdb0 con 0x7f35fc0cc510 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49340/0 (socket says 172.21.15.107:49340) 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 -- 172.21.15.107:0/1340880900 learned_addr learned my addr 172.21.15.107:0/1340880900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 -- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0cc510 msgr2=0x7f35fc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0cc510 0x7f35fc0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0cc510 0x7f35fc0ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 -- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1f3540 msgr2=0x7f35fc1f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1f3540 0x7f35fc1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 -- 172.21.15.107:0/1340880900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc0cd090 con 0x7f35e80047c0 2023-11-24T08:15:03.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.823+0000 7f35fbfff640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f35ec002070 tx=0x7f35ec00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.824+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec004b60 con 0x7f35e80047c0 2023-11-24T08:15:03.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.824+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec004cc0 con 0x7f35e80047c0 2023-11-24T08:15:03.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.824+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0072d0 con 0x7f35e80047c0 2023-11-24T08:15:03.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.824+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc0cd370 con 0x7f35e80047c0 2023-11-24T08:15:03.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.824+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc0cec80 con 0x7f35e80047c0 2023-11-24T08:15:03.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.825+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc0cd890 con 0x7f35e80047c0 2023-11-24T08:15:03.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.826+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f35ec007430 con 0x7f35e80047c0 2023-11-24T08:15:03.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.827+0000 7f35f8ff9640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 0x7f35dc06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:03.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.827+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f35ec088650 con 0x7f35e80047c0 2023-11-24T08:15:03.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.827+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 0x7f35dc06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:03.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.828+0000 7f35fb7fe640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 0x7f35dc06e740 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001770 tx=0x7f35e4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:03.831+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35ec054de0 con 0x7f35e80047c0 2023-11-24T08:15:04.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.049+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f35fc06e830 con 0x7f35dc06c280 2023-11-24T08:15:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.061+0000 7f35f8ff9640 1 -- 172.21.15.107:0/1340880900 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f35e40054b0 con 0x7f35dc06c280 2023-11-24T08:15:04.062 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3 2023-11-24T08:15:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 msgr2=0x7f35dc06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 0x7f35dc06e740 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001770 tx=0x7f35e4009030 comp rx=0 tx=0).stop 2023-11-24T08:15:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 msgr2=0x7f35fc153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f35ec002070 tx=0x7f35ec00b4a0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 shutdown_connections 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35dc06c280 0x7f35dc06e740 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1f3540 0x7f35fc1f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0cc510 0x7f35fc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 --2- 172.21.15.107:0/1340880900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e80047c0 0x7f35fc153010 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 >> 172.21.15.107:0/1340880900 conn(0x7f35fc0cf900 msgr2=0x7f35fc15cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:04.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 shutdown_connections 2023-11-24T08:15:04.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.064+0000 7f36022ad640 1 -- 172.21.15.107:0/1340880900 wait complete. 2023-11-24T08:15:04.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.083 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:15:04.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.132 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3) 2023-11-24T08:15:04.183 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:15:04.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone7555 2023-11-24T08:15:04.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.397+0000 7f14a5fe6640 1 Processor -- start 2023-11-24T08:15:04.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.397+0000 7f14a5fe6640 1 -- start start 2023-11-24T08:15:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f14a5fe6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a00d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f14a5fe6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14a015d7d0 0x7f14a00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f14a5fe6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a00d15e0 con 0x7f14a00d0e40 2023-11-24T08:15:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f14a5fe6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a00d0280 con 0x7f14a015d7d0 2023-11-24T08:15:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f14a5fe6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00d1300 con 0x7f14a015d3f0 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f149ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14a015d7d0 0x7f14a00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f149ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14a015d7d0 0x7f14a00d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49342/0 (socket says 172.21.15.107:49342) 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.399+0000 7f149ffff640 1 -- 172.21.15.107:0/634292766 learned_addr learned my addr 172.21.15.107:0/634292766 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.400+0000 7f14a4fe4640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a00d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.400+0000 7f149ffff640 1 -- 172.21.15.107:0/634292766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 msgr2=0x7f14a00d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:04.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.400+0000 7f149ffff640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a00d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.400+0000 7f149ffff640 1 -- 172.21.15.107:0/634292766 >> v1:172.21.15.120:6789/0 conn(0x7f14a00d0e40 legacy=0x7f14a00afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:04.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.400+0000 7f149ffff640 1 -- 172.21.15.107:0/634292766 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a00cfc30 con 0x7f14a015d7d0 2023-11-24T08:15:04.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.401+0000 7f149ffff640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00d1d70 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f148c009870 tx=0x7f148c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e666252ede9a84c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:04.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.401+0000 7f149f7fe640 1 -- 172.21.15.107:0/634292766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c031640 con 0x7f14a015d7d0 2023-11-24T08:15:04.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.401+0000 7f149f7fe640 1 -- 172.21.15.107:0/634292766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c0317a0 con 0x7f14a015d7d0 2023-11-24T08:15:04.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.401+0000 7f149f7fe640 1 -- 172.21.15.107:0/634292766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c031b30 con 0x7f14a015d7d0 2023-11-24T08:15:04.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 -- 172.21.15.107:0/634292766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 msgr2=0x7f14a00d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00d1d70 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f148c009870 tx=0x7f148c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 -- 172.21.15.107:0/634292766 shutdown_connections 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00d1d70 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/634292766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a00d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 -- 172.21.15.107:0/634292766 >> 172.21.15.107:0/634292766 conn(0x7f14a009c5b0 msgr2=0x7f14a00d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 -- 172.21.15.107:0/634292766 shutdown_connections 2023-11-24T08:15:04.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.402+0000 7f14a5fe6640 1 -- 172.21.15.107:0/634292766 wait complete. 2023-11-24T08:15:04.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.404+0000 7f14a5fe6640 1 Processor -- start 2023-11-24T08:15:04.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.404+0000 7f14a5fe6640 1 -- start start 2023-11-24T08:15:04.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14a00d0e40 0x7f14a01f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a0160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00b6d80 con 0x7f14a015d7d0 2023-11-24T08:15:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01f4400 con 0x7f14a00d0e40 2023-11-24T08:15:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a5fe6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00b6200 con 0x7f14a015d3f0 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a57e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a57e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49346/0 (socket says 172.21.15.107:49346) 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.405+0000 7f14a57e5640 1 -- 172.21.15.107:0/4287536210 learned_addr learned my addr 172.21.15.107:0/4287536210 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f149ffff640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a0160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 -- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 msgr2=0x7f14a0160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a0160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 -- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14a00d0e40 msgr2=0x7f14a01f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14a00d0e40 0x7f14a01f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 -- 172.21.15.107:0/4287536210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0094d0 con 0x7f14a015d7d0 2023-11-24T08:15:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a57e5640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f149800c730 tx=0x7f149800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:04.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800aa50 con 0x7f14a015d7d0 2023-11-24T08:15:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149800abb0 con 0x7f14a015d7d0 2023-11-24T08:15:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800ce40 con 0x7f14a015d7d0 2023-11-24T08:15:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.406+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a00b64e0 con 0x7f14a015d7d0 2023-11-24T08:15:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.407+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a00b6aa0 con 0x7f14a015d7d0 2023-11-24T08:15:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.408+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a004e840 con 0x7f14a015d7d0 2023-11-24T08:15:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.409+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f1498012420 con 0x7f14a015d7d0 2023-11-24T08:15:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.409+0000 7f149d7fa640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 0x7f148406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.410+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1498088b70 con 0x7f14a015d7d0 2023-11-24T08:15:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.410+0000 7f14a4fe4640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 0x7f148406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.410+0000 7f14a4fe4640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 0x7f148406e740 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f14a00b4a20 tx=0x7f1494008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.414+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14980552a0 con 0x7f14a015d7d0 2023-11-24T08:15:04.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.634+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f14a00d0710 con 0x7f148406c280 2023-11-24T08:15:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.653+0000 7f149d7fa640 1 -- 172.21.15.107:0/4287536210 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7f1494009e40 con 0x7f148406c280 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:14:41", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 67108864, 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:14:52", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:15:01", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:15:04.656 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:15:04.657 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:14:50", 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3", 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:15:04.658 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:15:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 msgr2=0x7f148406e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 0x7f148406e740 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f14a00b4a20 tx=0x7f1494008040 comp rx=0 tx=0).stop 2023-11-24T08:15:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 msgr2=0x7f14a00b7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f149800c730 tx=0x7f149800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 shutdown_connections 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f148406c280 0x7f148406e740 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14a015d7d0 0x7f14a00b7a20 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14a015d3f0 0x7f14a0160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 --2- 172.21.15.107:0/4287536210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14a00d0e40 0x7f14a01f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 >> 172.21.15.107:0/4287536210 conn(0x7f14a009c5b0 msgr2=0x7f14a00c9650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 shutdown_connections 2023-11-24T08:15:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:04.656+0000 7f14a5fe6640 1 -- 172.21.15.107:0/4287536210 wait complete. 2023-11-24T08:15:04.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.677 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.725 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459: ceph.quota.max_bytes: No such attribute 2023-11-24T08:15:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:15:04.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.777 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:15:04.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34 2023-11-24T08:15:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13 2023-11-24T08:15:04.886 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29 2023-11-24T08:15:04.887 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38 2023-11-24T08:15:04.888 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19 2023-11-24T08:15:04.888 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7 2023-11-24T08:15:04.888 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48 2023-11-24T08:15:04.888 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57 2023-11-24T08:15:04.888 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23 2023-11-24T08:15:04.889 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52 2023-11-24T08:15:04.890 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12 2023-11-24T08:15:04.891 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43 2023-11-24T08:15:04.892 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16 2023-11-24T08:15:04.892 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31 2023-11-24T08:15:04.892 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1 2023-11-24T08:15:04.892 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60 2023-11-24T08:15:04.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.893 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:04.943 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:15:04.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.945 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/) 2023-11-24T08:15:04.994 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:15:04.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:04.996 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:05.044 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:15:05.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.045 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/) 2023-11-24T08:15:05.098 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:15:05.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:05.150 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:15:05.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.152 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/) 2023-11-24T08:15:05.208 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:15:05.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.209 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459) 2023-11-24T08:15:05.260 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:05.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/) 2023-11-24T08:15:05.310 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:05.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.311 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49) 2023-11-24T08:15:05.362 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:05.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.363 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.49) 2023-11-24T08:15:05.426 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:05.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.427 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49) 2023-11-24T08:15:05.480 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:05.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.482 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.49) 2023-11-24T08:15:05.535 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:05.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.536 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49) 2023-11-24T08:15:05.588 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:05.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.590 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.49) 2023-11-24T08:15:05.641 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:05.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.642 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.49) 2023-11-24T08:15:05.694 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:05.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.696 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.49) 2023-11-24T08:15:05.751 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:05.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.753 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22) 2023-11-24T08:15:05.805 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:05.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.806 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.22) 2023-11-24T08:15:05.858 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:05.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.859 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22) 2023-11-24T08:15:05.908 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:05.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.909 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.22) 2023-11-24T08:15:05.961 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:05.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:05.963 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22) 2023-11-24T08:15:06.017 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.019 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.22) 2023-11-24T08:15:06.070 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.071 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.22) 2023-11-24T08:15:06.125 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:06.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.126 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.22) 2023-11-24T08:15:06.178 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:06.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.180 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20) 2023-11-24T08:15:06.230 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:06.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.232 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.20) 2023-11-24T08:15:06.283 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:06.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.285 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20) 2023-11-24T08:15:06.336 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:06.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.20) 2023-11-24T08:15:06.391 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:06.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20) 2023-11-24T08:15:06.445 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.446 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.20) 2023-11-24T08:15:06.499 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.501 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.20) 2023-11-24T08:15:06.553 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:06.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.555 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.20) 2023-11-24T08:15:06.606 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:06.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34) 2023-11-24T08:15:06.659 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:06.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.660 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.34) 2023-11-24T08:15:06.713 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:06.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.714 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34) 2023-11-24T08:15:06.765 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.767 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.34) 2023-11-24T08:15:06.819 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:06.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.820 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34) 2023-11-24T08:15:06.873 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.875 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.34) 2023-11-24T08:15:06.927 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:06.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.928 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.34) 2023-11-24T08:15:06.981 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:06.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:06.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.34) 2023-11-24T08:15:07.033 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:07.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.035 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55) 2023-11-24T08:15:07.085 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:07.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.086 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.55) 2023-11-24T08:15:07.139 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:07.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55) 2023-11-24T08:15:07.193 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:07.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.55) 2023-11-24T08:15:07.250 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:07.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55) 2023-11-24T08:15:07.305 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:07.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.55) 2023-11-24T08:15:07.360 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:07.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.361 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.55) 2023-11-24T08:15:07.411 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:07.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.412 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.55) 2023-11-24T08:15:07.512 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:07.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.513 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2) 2023-11-24T08:15:07.568 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:07.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.569 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.2) 2023-11-24T08:15:07.623 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:07.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2) 2023-11-24T08:15:07.678 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:07.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.679 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.2) 2023-11-24T08:15:07.732 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:07.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.733 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2) 2023-11-24T08:15:07.785 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:07.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.787 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.2) 2023-11-24T08:15:07.840 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:07.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.842 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.2) 2023-11-24T08:15:07.894 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:07.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.896 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.2) 2023-11-24T08:15:07.949 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:07.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:07.950 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35) 2023-11-24T08:15:08.001 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.003 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.35) 2023-11-24T08:15:08.054 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35) 2023-11-24T08:15:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:08.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.107 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.35) 2023-11-24T08:15:08.158 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:08.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.159 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35) 2023-11-24T08:15:08.209 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:08.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.211 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.35) 2023-11-24T08:15:08.263 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:08.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.264 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.35) 2023-11-24T08:15:08.315 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:08.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.316 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.35) 2023-11-24T08:15:08.365 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:08.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.366 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30) 2023-11-24T08:15:08.415 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.416 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.30) 2023-11-24T08:15:08.470 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30) 2023-11-24T08:15:08.522 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:08.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.524 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.30) 2023-11-24T08:15:08.576 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:08.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.576 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30) 2023-11-24T08:15:08.629 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:08.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.630 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.30) 2023-11-24T08:15:08.686 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:08.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.30) 2023-11-24T08:15:08.738 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:08.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.739 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.30) 2023-11-24T08:15:08.787 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:08.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.788 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14) 2023-11-24T08:15:08.888 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.889 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.14) 2023-11-24T08:15:08.944 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:08.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:08.945 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14) 2023-11-24T08:15:08.998 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:08.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.000 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.14) 2023-11-24T08:15:09.049 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:09.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.051 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14) 2023-11-24T08:15:09.098 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.100 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.14) 2023-11-24T08:15:09.151 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.152 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.14) 2023-11-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:09.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.14) 2023-11-24T08:15:09.257 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:09.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.259 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26) 2023-11-24T08:15:09.312 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:09.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.313 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.26) 2023-11-24T08:15:09.368 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:09.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.369 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26) 2023-11-24T08:15:09.421 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:09.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.422 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.26) 2023-11-24T08:15:09.474 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:09.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.475 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26) 2023-11-24T08:15:09.524 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.26) 2023-11-24T08:15:09.576 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.577 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.26) 2023-11-24T08:15:09.629 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:09.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.631 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.26) 2023-11-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:09.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51) 2023-11-24T08:15:09.734 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:09.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.736 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.51) 2023-11-24T08:15:09.786 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:09.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.787 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51) 2023-11-24T08:15:09.839 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:09.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.840 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.51) 2023-11-24T08:15:09.893 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:09.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.894 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51) 2023-11-24T08:15:09.945 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.947 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.51) 2023-11-24T08:15:09.998 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:09.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:09.999 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.51) 2023-11-24T08:15:10.048 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:10.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.51) 2023-11-24T08:15:10.101 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:10.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50) 2023-11-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:10.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.156 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.50) 2023-11-24T08:15:10.212 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:10.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.214 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50) 2023-11-24T08:15:10.266 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:10.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.268 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.50) 2023-11-24T08:15:10.325 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:10.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.327 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50) 2023-11-24T08:15:10.379 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:10.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.381 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.50) 2023-11-24T08:15:10.429 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:10.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.430 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.50) 2023-11-24T08:15:10.481 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:10.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.482 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.50) 2023-11-24T08:15:10.534 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:10.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.535 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13) 2023-11-24T08:15:10.587 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:10.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.589 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.13) 2023-11-24T08:15:10.643 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:10.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.644 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13) 2023-11-24T08:15:10.696 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:10.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.698 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.13) 2023-11-24T08:15:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:10.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13) 2023-11-24T08:15:10.803 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:10.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.13) 2023-11-24T08:15:10.860 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:10.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.861 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.13) 2023-11-24T08:15:10.912 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:10.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.913 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.13) 2023-11-24T08:15:10.965 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:10.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:10.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10) 2023-11-24T08:15:11.017 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.018 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.10) 2023-11-24T08:15:11.072 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.073 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10) 2023-11-24T08:15:11.125 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:11.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.126 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.10) 2023-11-24T08:15:11.177 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:11.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.178 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10) 2023-11-24T08:15:11.232 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:11.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.232 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.10) 2023-11-24T08:15:11.288 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:11.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.10) 2023-11-24T08:15:11.343 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:11.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.10) 2023-11-24T08:15:11.397 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:11.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.398 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47) 2023-11-24T08:15:11.450 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.452 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.47) 2023-11-24T08:15:11.503 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47) 2023-11-24T08:15:11.553 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:11.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.554 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.47) 2023-11-24T08:15:11.603 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:11.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.605 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47) 2023-11-24T08:15:11.662 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:11.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.663 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.47) 2023-11-24T08:15:11.715 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:11.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.716 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.47) 2023-11-24T08:15:11.769 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:11.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.770 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.47) 2023-11-24T08:15:11.824 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:11.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5) 2023-11-24T08:15:11.880 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.881 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.5) 2023-11-24T08:15:11.936 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:11.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.937 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5) 2023-11-24T08:15:11.987 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:11.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:11.988 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.5) 2023-11-24T08:15:12.042 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:12.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5) 2023-11-24T08:15:12.095 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:12.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.097 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.5) 2023-11-24T08:15:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:12.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.5) 2023-11-24T08:15:12.203 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:12.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.204 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.5) 2023-11-24T08:15:12.257 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:12.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.258 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37) 2023-11-24T08:15:12.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:15:12.314 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:15:12.316 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:12.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.316 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.37) 2023-11-24T08:15:12.402 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:12.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37) 2023-11-24T08:15:12.523 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:12.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.37) 2023-11-24T08:15:12.574 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:12.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.576 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37) 2023-11-24T08:15:12.626 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:12.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.37) 2023-11-24T08:15:12.680 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:12.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.682 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.37) 2023-11-24T08:15:12.738 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:12.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.741 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.37) 2023-11-24T08:15:12.796 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:12.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.798 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39) 2023-11-24T08:15:12.850 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:12.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.852 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.39) 2023-11-24T08:15:12.907 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:12.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.909 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39) 2023-11-24T08:15:12.964 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:12.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:12.965 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.39) 2023-11-24T08:15:13.018 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:13.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.020 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39) 2023-11-24T08:15:13.073 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.39) 2023-11-24T08:15:13.131 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.132 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.39) 2023-11-24T08:15:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:13.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.186 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.39) 2023-11-24T08:15:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:13.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54) 2023-11-24T08:15:13.290 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:13.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.54) 2023-11-24T08:15:13.344 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:13.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54) 2023-11-24T08:15:13.397 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:13.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.398 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.54) 2023-11-24T08:15:13.447 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:13.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.449 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54) 2023-11-24T08:15:13.501 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.503 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.54) 2023-11-24T08:15:13.556 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.558 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.54) 2023-11-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:13.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.54) 2023-11-24T08:15:13.658 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:13.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.659 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36) 2023-11-24T08:15:13.714 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:13.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.36) 2023-11-24T08:15:13.770 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:13.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.772 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36) 2023-11-24T08:15:13.824 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:13.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.825 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.36) 2023-11-24T08:15:13.875 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:13.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.876 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36) 2023-11-24T08:15:13.926 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.927 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.36) 2023-11-24T08:15:13.978 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:13.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:13.979 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.36) 2023-11-24T08:15:14.029 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:14.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.030 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.36) 2023-11-24T08:15:14.079 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:14.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.081 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18) 2023-11-24T08:15:14.129 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:14.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.130 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.18) 2023-11-24T08:15:14.178 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:14.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.180 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18) 2023-11-24T08:15:14.228 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:14.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.18) 2023-11-24T08:15:14.329 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:14.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.330 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18) 2023-11-24T08:15:14.382 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:14.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.383 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.18) 2023-11-24T08:15:14.435 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:14.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.436 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.18) 2023-11-24T08:15:14.489 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:14.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.18) 2023-11-24T08:15:14.542 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:14.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17) 2023-11-24T08:15:14.593 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:14.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.594 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.17) 2023-11-24T08:15:14.645 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:14.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.647 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17) 2023-11-24T08:15:14.694 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:14.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.695 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.17) 2023-11-24T08:15:14.743 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:14.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.745 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17) 2023-11-24T08:15:14.794 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:14.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.796 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.17) 2023-11-24T08:15:14.845 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:14.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.847 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.17) 2023-11-24T08:15:14.900 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:14.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.901 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.17) 2023-11-24T08:15:14.952 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:14.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:14.954 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4) 2023-11-24T08:15:15.007 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.008 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.4) 2023-11-24T08:15:15.062 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.064 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4) 2023-11-24T08:15:15.119 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:15.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.120 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.4) 2023-11-24T08:15:15.174 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:15.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.176 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4) 2023-11-24T08:15:15.226 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:15.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.4) 2023-11-24T08:15:15.280 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:15.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.282 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.4) 2023-11-24T08:15:15.332 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:15.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.333 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.4) 2023-11-24T08:15:15.385 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:15.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58) 2023-11-24T08:15:15.436 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.437 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.58) 2023-11-24T08:15:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.487 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58) 2023-11-24T08:15:15.534 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:15.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.535 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.58) 2023-11-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:15.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.584 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58) 2023-11-24T08:15:15.631 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:15.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.58) 2023-11-24T08:15:15.733 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:15.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.734 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.58) 2023-11-24T08:15:15.785 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:15.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.786 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.58) 2023-11-24T08:15:15.838 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:15.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.839 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29) 2023-11-24T08:15:15.891 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.893 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.29) 2023-11-24T08:15:15.948 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:15.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:15.949 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29) 2023-11-24T08:15:16.000 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.002 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.29) 2023-11-24T08:15:16.049 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29) 2023-11-24T08:15:16.149 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:16.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.151 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.29) 2023-11-24T08:15:16.207 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:16.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.209 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.29) 2023-11-24T08:15:16.263 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:16.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.264 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.29) 2023-11-24T08:15:16.316 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:16.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38) 2023-11-24T08:15:16.369 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:16.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.370 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.38) 2023-11-24T08:15:16.424 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:16.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.425 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38) 2023-11-24T08:15:16.479 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.481 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.38) 2023-11-24T08:15:16.531 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.532 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38) 2023-11-24T08:15:16.579 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:16.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.38) 2023-11-24T08:15:16.631 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:16.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.632 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.38) 2023-11-24T08:15:16.683 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:16.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.38) 2023-11-24T08:15:16.736 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:16.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.738 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19) 2023-11-24T08:15:16.790 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:16.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.791 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.19) 2023-11-24T08:15:16.849 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:16.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.850 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19) 2023-11-24T08:15:16.903 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.904 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.19) 2023-11-24T08:15:16.956 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:16.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:16.957 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19) 2023-11-24T08:15:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.017 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.19) 2023-11-24T08:15:17.067 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.19) 2023-11-24T08:15:17.119 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:17.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.121 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.19) 2023-11-24T08:15:17.172 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:17.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.174 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7) 2023-11-24T08:15:17.226 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:17.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.7) 2023-11-24T08:15:17.281 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:17.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.282 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7) 2023-11-24T08:15:17.335 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:17.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.336 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.7) 2023-11-24T08:15:17.389 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:17.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7) 2023-11-24T08:15:17.443 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.7) 2023-11-24T08:15:17.498 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.7) 2023-11-24T08:15:17.551 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:17.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.7) 2023-11-24T08:15:17.605 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:17.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48) 2023-11-24T08:15:17.662 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:17.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.663 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.48) 2023-11-24T08:15:17.717 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:17.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.718 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48) 2023-11-24T08:15:17.769 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:17.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.771 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.48) 2023-11-24T08:15:17.823 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:17.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.825 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48) 2023-11-24T08:15:17.882 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.882 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.48) 2023-11-24T08:15:17.936 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:17.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.937 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.48) 2023-11-24T08:15:17.989 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:17.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:17.990 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.48) 2023-11-24T08:15:18.043 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:18.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57) 2023-11-24T08:15:18.097 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:18.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.098 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.57) 2023-11-24T08:15:18.152 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:18.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57) 2023-11-24T08:15:18.210 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.214 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.57) 2023-11-24T08:15:18.265 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:18.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57) 2023-11-24T08:15:18.323 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:18.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.324 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.57) 2023-11-24T08:15:18.377 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:18.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.378 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.57) 2023-11-24T08:15:18.432 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:18.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.433 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.57) 2023-11-24T08:15:18.486 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:18.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.487 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6) 2023-11-24T08:15:18.539 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:18.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.540 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.6) 2023-11-24T08:15:18.593 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:18.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6) 2023-11-24T08:15:18.644 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:18.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.645 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.6) 2023-11-24T08:15:18.694 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:18.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.695 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6) 2023-11-24T08:15:18.749 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:18.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.6) 2023-11-24T08:15:18.800 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:18.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.801 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.6) 2023-11-24T08:15:18.851 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:18.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.852 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.6) 2023-11-24T08:15:18.909 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:18.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.914 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53) 2023-11-24T08:15:18.968 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:18.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:18.970 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.53) 2023-11-24T08:15:19.026 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:19.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53) 2023-11-24T08:15:19.080 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.081 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.53) 2023-11-24T08:15:19.133 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.135 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53) 2023-11-24T08:15:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:19.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.185 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.53) 2023-11-24T08:15:19.236 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:19.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.53) 2023-11-24T08:15:19.290 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:19.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.53) 2023-11-24T08:15:19.345 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:19.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.348 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40) 2023-11-24T08:15:19.402 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:19.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.404 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.40) 2023-11-24T08:15:19.458 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:19.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.460 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40) 2023-11-24T08:15:19.514 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.516 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.40) 2023-11-24T08:15:19.569 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.571 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40) 2023-11-24T08:15:19.625 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:19.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.40) 2023-11-24T08:15:19.678 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:19.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.40) 2023-11-24T08:15:19.730 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:19.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.734 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.40) 2023-11-24T08:15:19.781 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:19.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.783 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42) 2023-11-24T08:15:19.834 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:19.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.835 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.42) 2023-11-24T08:15:19.890 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:19.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.891 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42) 2023-11-24T08:15:19.943 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.42) 2023-11-24T08:15:19.995 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:19.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:19.997 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42) 2023-11-24T08:15:20.048 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.42) 2023-11-24T08:15:20.099 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.42) 2023-11-24T08:15:20.153 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:20.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.42) 2023-11-24T08:15:20.205 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:20.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0) 2023-11-24T08:15:20.259 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:20.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.260 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.0) 2023-11-24T08:15:20.313 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:20.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.315 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0) 2023-11-24T08:15:20.366 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:20.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.0) 2023-11-24T08:15:20.419 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:20.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.421 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0) 2023-11-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.0) 2023-11-24T08:15:20.525 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.0) 2023-11-24T08:15:20.579 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:20.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.0) 2023-11-24T08:15:20.632 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:20.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27) 2023-11-24T08:15:20.688 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:20.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.689 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.27) 2023-11-24T08:15:20.748 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:20.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.750 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27) 2023-11-24T08:15:20.802 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:20.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.27) 2023-11-24T08:15:20.856 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:20.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27) 2023-11-24T08:15:20.912 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.914 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.27) 2023-11-24T08:15:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:20.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:20.967 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.27) 2023-11-24T08:15:21.018 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:21.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.019 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.27) 2023-11-24T08:15:21.071 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:21.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.072 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25) 2023-11-24T08:15:21.124 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:21.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.125 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.25) 2023-11-24T08:15:21.178 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:21.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.180 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25) 2023-11-24T08:15:21.232 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:21.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.233 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.25) 2023-11-24T08:15:21.286 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:21.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25) 2023-11-24T08:15:21.340 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:21.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.25) 2023-11-24T08:15:21.398 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:21.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.25) 2023-11-24T08:15:21.457 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:21.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.458 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.25) 2023-11-24T08:15:21.511 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:21.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.512 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32) 2023-11-24T08:15:21.563 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:21.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.565 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.32) 2023-11-24T08:15:21.618 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:21.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.619 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32) 2023-11-24T08:15:21.672 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:21.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.673 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.32) 2023-11-24T08:15:21.725 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:21.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32) 2023-11-24T08:15:21.781 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:21.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.32) 2023-11-24T08:15:21.838 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:21.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.840 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.32) 2023-11-24T08:15:21.893 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:21.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.894 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.32) 2023-11-24T08:15:21.948 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:21.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:21.949 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28) 2023-11-24T08:15:22.002 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.28) 2023-11-24T08:15:22.060 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.062 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28) 2023-11-24T08:15:22.114 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:22.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.115 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.28) 2023-11-24T08:15:22.166 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:22.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28) 2023-11-24T08:15:22.218 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:22.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.220 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.28) 2023-11-24T08:15:22.273 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:22.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.28) 2023-11-24T08:15:22.327 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:22.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.328 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.28) 2023-11-24T08:15:22.385 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:22.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21) 2023-11-24T08:15:22.441 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.21) 2023-11-24T08:15:22.491 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.492 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21) 2023-11-24T08:15:22.543 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:22.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.544 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.21) 2023-11-24T08:15:22.596 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:22.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21) 2023-11-24T08:15:22.649 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:22.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.651 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.21) 2023-11-24T08:15:22.702 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:22.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.704 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.21) 2023-11-24T08:15:22.754 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:22.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.756 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.21) 2023-11-24T08:15:22.807 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:22.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.808 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9) 2023-11-24T08:15:22.862 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.9) 2023-11-24T08:15:22.919 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:22.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.920 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9) 2023-11-24T08:15:22.973 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:22.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:22.974 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.9) 2023-11-24T08:15:23.027 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:23.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.029 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9) 2023-11-24T08:15:23.079 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.080 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.9) 2023-11-24T08:15:23.131 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.133 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.9) 2023-11-24T08:15:23.185 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:23.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.186 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.9) 2023-11-24T08:15:23.236 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:23.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.238 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23) 2023-11-24T08:15:23.290 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:23.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.23) 2023-11-24T08:15:23.342 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:23.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.344 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23) 2023-11-24T08:15:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:23.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.402 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.23) 2023-11-24T08:15:23.455 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:23.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.456 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23) 2023-11-24T08:15:23.510 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.512 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.23) 2023-11-24T08:15:23.564 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.565 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.23) 2023-11-24T08:15:23.615 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:23.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.616 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.23) 2023-11-24T08:15:23.666 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:23.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.668 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15) 2023-11-24T08:15:23.721 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:23.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.722 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.15) 2023-11-24T08:15:23.776 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:23.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15) 2023-11-24T08:15:23.829 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:23.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.831 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.15) 2023-11-24T08:15:23.882 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:23.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.883 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15) 2023-11-24T08:15:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.936 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.15) 2023-11-24T08:15:23.987 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:23.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:23.989 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.15) 2023-11-24T08:15:24.039 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:24.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.041 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.15) 2023-11-24T08:15:24.086 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:24.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.087 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8) 2023-11-24T08:15:24.188 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:24.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.8) 2023-11-24T08:15:24.244 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:24.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.246 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8) 2023-11-24T08:15:24.303 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:24.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.304 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.8) 2023-11-24T08:15:24.358 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:24.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8) 2023-11-24T08:15:24.412 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:24.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.8) 2023-11-24T08:15:24.466 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:24.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.467 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.8) 2023-11-24T08:15:24.519 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:24.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.521 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.8) 2023-11-24T08:15:24.578 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:24.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46) 2023-11-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:24.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.46) 2023-11-24T08:15:24.691 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:24.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.693 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46) 2023-11-24T08:15:24.748 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:24.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.749 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.46) 2023-11-24T08:15:24.803 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:24.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46) 2023-11-24T08:15:24.859 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:24.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.861 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.46) 2023-11-24T08:15:24.909 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:24.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.46) 2023-11-24T08:15:24.959 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:24.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:24.961 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.46) 2023-11-24T08:15:25.013 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:25.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.015 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33) 2023-11-24T08:15:25.067 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:25.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.33) 2023-11-24T08:15:25.114 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:25.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33) 2023-11-24T08:15:25.214 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:25.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.215 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.33) 2023-11-24T08:15:25.269 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:25.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.270 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33) 2023-11-24T08:15:25.322 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:25.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.33) 2023-11-24T08:15:25.376 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:25.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.377 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.33) 2023-11-24T08:15:25.430 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:25.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.431 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.33) 2023-11-24T08:15:25.485 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:25.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.486 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45) 2023-11-24T08:15:25.540 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:25.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.45) 2023-11-24T08:15:25.597 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:25.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.598 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45) 2023-11-24T08:15:25.651 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:25.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.45) 2023-11-24T08:15:25.708 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:25.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.709 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45) 2023-11-24T08:15:25.762 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:25.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:25.763 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.45) 2023-11-24T08:15:25.819 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:26.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.173 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.45) 2023-11-24T08:15:26.234 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:26.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.45) 2023-11-24T08:15:26.294 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:26.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61) 2023-11-24T08:15:26.347 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:26.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.348 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.61) 2023-11-24T08:15:26.397 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:26.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61) 2023-11-24T08:15:26.448 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:26.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.449 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.61) 2023-11-24T08:15:26.498 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:26.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61) 2023-11-24T08:15:26.550 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:26.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.61) 2023-11-24T08:15:26.607 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:26.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.607 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.61) 2023-11-24T08:15:26.659 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:26.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.660 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.61) 2023-11-24T08:15:26.711 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:26.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.713 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62) 2023-11-24T08:15:26.767 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:26.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.772 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.62) 2023-11-24T08:15:26.828 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:26.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.829 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62) 2023-11-24T08:15:26.886 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:26.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.887 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.62) 2023-11-24T08:15:26.937 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:26.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.939 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62) 2023-11-24T08:15:26.988 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:26.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:26.990 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.62) 2023-11-24T08:15:27.041 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:27.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.042 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.62) 2023-11-24T08:15:27.095 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:27.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.096 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.62) 2023-11-24T08:15:27.152 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:27.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52) 2023-11-24T08:15:27.205 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:27.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.52) 2023-11-24T08:15:27.260 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:27.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.262 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52) 2023-11-24T08:15:27.309 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:27.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.311 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.52) 2023-11-24T08:15:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:27.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.362 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52) 2023-11-24T08:15:27.412 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:27.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.52) 2023-11-24T08:15:27.463 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:27.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.464 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.52) 2023-11-24T08:15:27.512 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:27.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.513 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.52) 2023-11-24T08:15:27.559 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:27.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.561 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11) 2023-11-24T08:15:27.610 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:27.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.611 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.11) 2023-11-24T08:15:27.665 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:27.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.666 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11) 2023-11-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:27.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.720 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.11) 2023-11-24T08:15:27.770 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:27.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.771 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11) 2023-11-24T08:15:27.822 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:27.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.823 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.11) 2023-11-24T08:15:27.877 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:27.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.878 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.11) 2023-11-24T08:15:27.930 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:27.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.931 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.11) 2023-11-24T08:15:27.981 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:27.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:27.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44) 2023-11-24T08:15:28.036 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:28.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.038 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.44) 2023-11-24T08:15:28.089 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:28.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.091 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44) 2023-11-24T08:15:28.138 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:28.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.139 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.44) 2023-11-24T08:15:28.189 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:28.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44) 2023-11-24T08:15:28.239 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:28.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.241 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.44) 2023-11-24T08:15:28.296 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:28.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.297 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.44) 2023-11-24T08:15:28.352 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:28.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.353 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.44) 2023-11-24T08:15:28.405 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:28.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.406 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3) 2023-11-24T08:15:28.458 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:28.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.459 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.3) 2023-11-24T08:15:28.514 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:28.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.515 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3) 2023-11-24T08:15:28.563 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:28.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.564 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.3) 2023-11-24T08:15:28.608 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:28.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.610 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3) 2023-11-24T08:15:28.709 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:28.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.3) 2023-11-24T08:15:28.758 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:28.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.759 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.3) 2023-11-24T08:15:28.808 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:28.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.810 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.3) 2023-11-24T08:15:28.862 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:28.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24) 2023-11-24T08:15:28.908 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:28.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:28.909 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.24) 2023-11-24T08:15:29.005 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:29.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.007 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24) 2023-11-24T08:15:29.054 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:29.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.24) 2023-11-24T08:15:29.156 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:29.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.158 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24) 2023-11-24T08:15:29.210 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:29.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.211 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.24) 2023-11-24T08:15:29.268 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:29.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.270 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.24) 2023-11-24T08:15:29.321 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:29.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.24) 2023-11-24T08:15:29.374 INFO:teuthology.orchestra.run.smithi107.stdout:1700813685 2023-11-24T08:15:29.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.375 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56) 2023-11-24T08:15:29.425 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:29.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.427 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.56) 2023-11-24T08:15:29.480 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:29.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.482 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56) 2023-11-24T08:15:29.534 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:29.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.535 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.56) 2023-11-24T08:15:29.590 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:29.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.590 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56) 2023-11-24T08:15:29.643 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:29.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.645 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.56) 2023-11-24T08:15:29.697 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:29.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.699 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.56) 2023-11-24T08:15:29.751 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:29.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.752 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.56) 2023-11-24T08:15:29.804 INFO:teuthology.orchestra.run.smithi107.stdout:1700813689 2023-11-24T08:15:29.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.806 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63) 2023-11-24T08:15:29.863 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:29.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.864 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.63) 2023-11-24T08:15:29.919 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:29.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.920 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63) 2023-11-24T08:15:29.975 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:29.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:29.976 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.63) 2023-11-24T08:15:30.028 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:30.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.029 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63) 2023-11-24T08:15:30.084 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.085 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.63) 2023-11-24T08:15:30.135 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.136 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.63) 2023-11-24T08:15:30.185 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:30.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.187 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.63) 2023-11-24T08:15:30.239 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:30.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59) 2023-11-24T08:15:30.289 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:30.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.59) 2023-11-24T08:15:30.341 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:30.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59) 2023-11-24T08:15:30.394 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:30.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.395 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.59) 2023-11-24T08:15:30.447 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:30.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.448 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59) 2023-11-24T08:15:30.501 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.502 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.59) 2023-11-24T08:15:30.555 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.557 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.59) 2023-11-24T08:15:30.607 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:30.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.59) 2023-11-24T08:15:30.656 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:30.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41) 2023-11-24T08:15:30.710 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:30.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.41) 2023-11-24T08:15:30.765 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:30.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.765 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41) 2023-11-24T08:15:30.816 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:30.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.41) 2023-11-24T08:15:30.867 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:30.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41) 2023-11-24T08:15:30.918 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.919 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.41) 2023-11-24T08:15:30.970 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:30.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:30.971 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.41) 2023-11-24T08:15:31.022 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:31.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.023 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.41) 2023-11-24T08:15:31.075 INFO:teuthology.orchestra.run.smithi107.stdout:1700813687 2023-11-24T08:15:31.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.078 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12) 2023-11-24T08:15:31.133 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:31.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.135 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.12) 2023-11-24T08:15:31.194 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:31.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12) 2023-11-24T08:15:31.251 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:31.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.253 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.12) 2023-11-24T08:15:31.305 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:31.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.306 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12) 2023-11-24T08:15:31.364 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:31.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.365 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.12) 2023-11-24T08:15:31.417 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:31.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.12) 2023-11-24T08:15:31.471 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:31.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.12) 2023-11-24T08:15:31.524 INFO:teuthology.orchestra.run.smithi107.stdout:1700813683 2023-11-24T08:15:31.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43) 2023-11-24T08:15:31.577 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:31.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.578 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.43) 2023-11-24T08:15:31.631 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:31.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43) 2023-11-24T08:15:31.684 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:31.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.43) 2023-11-24T08:15:31.735 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:31.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43) 2023-11-24T08:15:31.785 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:31.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.788 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.43) 2023-11-24T08:15:31.837 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:31.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.838 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.43) 2023-11-24T08:15:31.888 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:31.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.889 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.43) 2023-11-24T08:15:31.939 INFO:teuthology.orchestra.run.smithi107.stdout:1700813688 2023-11-24T08:15:31.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16) 2023-11-24T08:15:31.993 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:31.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:31.995 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.16) 2023-11-24T08:15:32.046 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:32.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.048 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16) 2023-11-24T08:15:32.094 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:32.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.095 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.16) 2023-11-24T08:15:32.192 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:32.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.193 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16) 2023-11-24T08:15:32.246 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:32.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.247 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.16) 2023-11-24T08:15:32.301 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:32.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.302 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.16) 2023-11-24T08:15:32.354 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:32.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.355 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.16) 2023-11-24T08:15:32.407 INFO:teuthology.orchestra.run.smithi107.stdout:1700813684 2023-11-24T08:15:32.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.408 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31) 2023-11-24T08:15:32.462 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:32.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.463 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.31) 2023-11-24T08:15:32.515 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:32.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.517 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31) 2023-11-24T08:15:32.568 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:32.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.570 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.31) 2023-11-24T08:15:32.619 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:32.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.621 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31) 2023-11-24T08:15:32.672 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:32.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.674 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.31) 2023-11-24T08:15:32.726 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:32.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.31) 2023-11-24T08:15:32.779 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:32.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.31) 2023-11-24T08:15:32.832 INFO:teuthology.orchestra.run.smithi107.stdout:1700813686 2023-11-24T08:15:32.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.833 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1) 2023-11-24T08:15:32.887 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:32.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.889 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.1) 2023-11-24T08:15:32.945 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:32.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.947 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1) 2023-11-24T08:15:32.997 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:32.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:32.999 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.1) 2023-11-24T08:15:33.050 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:33.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.051 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1) 2023-11-24T08:15:33.102 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:33.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.1) 2023-11-24T08:15:33.152 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:33.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.1) 2023-11-24T08:15:33.202 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:33.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.203 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.1) 2023-11-24T08:15:33.252 INFO:teuthology.orchestra.run.smithi107.stdout:1700813682 2023-11-24T08:15:33.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.254 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60) 2023-11-24T08:15:33.302 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:33.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.303 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.60) 2023-11-24T08:15:33.352 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:15:33.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.353 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60) 2023-11-24T08:15:33.404 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:33.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.406 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.60) 2023-11-24T08:15:33.454 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:15:33.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60) 2023-11-24T08:15:33.507 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:33.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.509 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.60) 2023-11-24T08:15:33.559 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:15:33.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5827/.snap/subvol_snap7985/4b8b606b-b8db-4e06-97ea-277a5f63a459/subvolume_file.60) 2023-11-24T08:15:33.612 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:33.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:33.613 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7555/063c406a-e9dd-4793-bce0-f0f732152cd3/subvolume_file.60) 2023-11-24T08:15:33.667 INFO:teuthology.orchestra.run.smithi107.stdout:1700813690 2023-11-24T08:15:33.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol5827 subvol_snap7985 2023-11-24T08:15:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.879+0000 7f5e12e32640 1 Processor -- start 2023-11-24T08:15:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.879+0000 7f5e12e32640 1 -- start start 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12e32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5e0c0610b0 0x7f5e0c06aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12e32640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 0x7f5e0c0ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12e32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c06b810 con 0x7f5e0c0b2880 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12e32640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c06baf0 con 0x7f5e0c0610b0 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12e32640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c06bdd0 con 0x7f5e0c0ad040 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e1162f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5e0c0610b0 0x7f5e0c06aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e11e30640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5e0c0b2880 0x7f5e0c0b2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57728/0 (socket says 172.21.15.107:57728) 2023-11-24T08:15:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e11e30640 1 -- 172.21.15.107:0/755343047 learned_addr learned my addr 172.21.15.107:0/755343047 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e12631640 1 --2- 172.21.15.107:0/755343047 >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 0x7f5e0c0ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e1162f640 1 -- 172.21.15.107:0/755343047 >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 msgr2=0x7f5e0c0ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e1162f640 1 --2- 172.21.15.107:0/755343047 >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 0x7f5e0c0ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e1162f640 1 -- 172.21.15.107:0/755343047 >> v1:172.21.15.107:6789/0 conn(0x7f5e0c0b2880 legacy=0x7f5e0c0b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.881+0000 7f5e1162f640 1 -- 172.21.15.107:0/755343047 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0ac840 con 0x7f5e0c0610b0 2023-11-24T08:15:33.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.882+0000 7f5e12631640 1 --2- 172.21.15.107:0/755343047 >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 0x7f5e0c0ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:33.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e1162f640 1 --2- 172.21.15.107:0/755343047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0610b0 0x7f5e0c06aee0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0097a0 tx=0x7f5dfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f75559f82e1a42 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:33.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e10e2e640 1 -- 172.21.15.107:0/755343047 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc031830 con 0x7f5e0c0610b0 2023-11-24T08:15:33.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e10e2e640 1 -- 172.21.15.107:0/755343047 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc031990 con 0x7f5e0c0610b0 2023-11-24T08:15:33.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e10e2e640 1 -- 172.21.15.107:0/755343047 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc031d20 con 0x7f5e0c0610b0 2023-11-24T08:15:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e12e32640 1 -- 172.21.15.107:0/755343047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0610b0 msgr2=0x7f5e0c06aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.883+0000 7f5e12e32640 1 --2- 172.21.15.107:0/755343047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0610b0 0x7f5e0c06aee0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0097a0 tx=0x7f5dfc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 -- 172.21.15.107:0/755343047 shutdown_connections 2023-11-24T08:15:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 --2- 172.21.15.107:0/755343047 >> v2:172.21.15.120:3300/0 conn(0x7f5e0c0ad040 0x7f5e0c0ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 --2- 172.21.15.107:0/755343047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0610b0 0x7f5e0c06aee0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 -- 172.21.15.107:0/755343047 >> 172.21.15.107:0/755343047 conn(0x7f5e0c0cf620 msgr2=0x7f5e0c0cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 -- 172.21.15.107:0/755343047 shutdown_connections 2023-11-24T08:15:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.884+0000 7f5e12e32640 1 -- 172.21.15.107:0/755343047 wait complete. 2023-11-24T08:15:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.885+0000 7f5e12e32640 1 Processor -- start 2023-11-24T08:15:33.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.886+0000 7f5e12e32640 1 -- start start 2023-11-24T08:15:33.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e0c0610b0 0x7f5e0c143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 0x7f5e0c0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1565b0 con 0x7f5e0c0ad040 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c156890 con 0x7f5e0c0b2880 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12e32640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c156b70 con 0x7f5e0c0610b0 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53934/0 (socket says 172.21.15.107:53934) 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 -- 172.21.15.107:0/3475029089 learned_addr learned my addr 172.21.15.107:0/3475029089 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e11e30640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e0c0610b0 0x7f5e0c143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 -- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e0c0610b0 msgr2=0x7f5e0c143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e0c0610b0 0x7f5e0c143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 -- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 msgr2=0x7f5e0c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12631640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 0x7f5e0c0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 0x7f5e0c0affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e1162f640 1 -- 172.21.15.107:0/3475029089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc009400 con 0x7f5e0c0ad040 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.887+0000 7f5e12631640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 0x7f5e0c0affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e1162f640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc02bee0 tx=0x7f5dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:33.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc03ba70 con 0x7f5e0c0ad040 2023-11-24T08:15:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc03bbf0 con 0x7f5e0c0ad040 2023-11-24T08:15:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc0453f0 con 0x7f5e0c0ad040 2023-11-24T08:15:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c156e50 con 0x7f5e0c0ad040 2023-11-24T08:15:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.888+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1444a0 con 0x7f5e0c0ad040 2023-11-24T08:15:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.889+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c157370 con 0x7f5e0c0ad040 2023-11-24T08:15:33.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.890+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f5dfc04e450 con 0x7f5e0c0ad040 2023-11-24T08:15:33.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.892+0000 7f5e027fc640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 0x7f5df006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.892+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5dfc0aa560 con 0x7f5e0c0ad040 2023-11-24T08:15:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.892+0000 7f5e11e30640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 0x7f5df006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.893+0000 7f5e11e30640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 0x7f5df006e740 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5e08005880 tx=0x7f5e08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:33.897+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e0c157370 con 0x7f5e0c0ad040 2023-11-24T08:15:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.120+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol5827", "snap_name": "subvol_snap7985", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e0c15d4d0 con 0x7f5df006c280 2023-11-24T08:15:34.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.197+0000 7f5e027fc640 1 -- 172.21.15.107:0/3475029089 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5e080012c0 con 0x7f5df006c280 2023-11-24T08:15:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 msgr2=0x7f5df006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 0x7f5df006e740 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5e08005880 tx=0x7f5e08009040 comp rx=0 tx=0).stop 2023-11-24T08:15:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 msgr2=0x7f5e0c153930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc02bee0 tx=0x7f5dfc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 shutdown_connections 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5df006c280 0x7f5df006e740 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0b2880 0x7f5e0c0affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c0ad040 0x7f5e0c153930 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 --2- 172.21.15.107:0/3475029089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e0c0610b0 0x7f5e0c143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 >> 172.21.15.107:0/3475029089 conn(0x7f5e0c0cf620 msgr2=0x7f5e0c15b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 shutdown_connections 2023-11-24T08:15:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.201+0000 7f5e12e32640 1 -- 172.21.15.107:0/3475029089 wait complete. 2023-11-24T08:15:34.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol5827 2023-11-24T08:15:34.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.431+0000 7f68f2bdc640 1 Processor -- start 2023-11-24T08:15:34.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.431+0000 7f68f2bdc640 1 -- start start 2023-11-24T08:15:34.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f2bdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68ec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f2bdc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0cc3a0 con 0x7f68ec0c59b0 2023-11-24T08:15:34.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f2bdc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec15eea0 con 0x7f68ec0cbf20 2023-11-24T08:15:34.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f2bdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec15b820 con 0x7f68ec15b440 2023-11-24T08:15:34.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f1179640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f68ec0c59b0 0x7f68ec06c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57730/0 (socket says 172.21.15.107:57730) 2023-11-24T08:15:34.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68f1179640 1 -- 172.21.15.107:0/1417267927 learned_addr learned my addr 172.21.15.107:0/1417267927 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:34.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.433+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393423000 0 0) 0x7f68e4001490 con 0x7f68ec0c59b0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68dc003520 con 0x7f68ec0c59b0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68f0978640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68ec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3951099806 0 0) 0x7f68dc003520 con 0x7f68ec0c59b0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 msgr2=0x7f68ec0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68ec0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 >> v1:172.21.15.120:6789/0 conn(0x7f68ec0cbf20 legacy=0x7f68ec0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0abf30 con 0x7f68ec0c59b0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f68e4001b80 con 0x7f68ec0c59b0 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 >> v1:172.21.15.107:6789/0 conn(0x7f68ec0c59b0 legacy=0x7f68ec06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68dc0039a0 0x7f68dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc004d00 0x7f68dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec15b820 con 0x7f68dc0039a0 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc00bc00 con 0x7f68dc004d00 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.434+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc00bdc0 con 0x7f68ec15b440 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68dc0039a0 0x7f68dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68ebfff640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f1179640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc004d00 0x7f68dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 -- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 msgr2=0x7f68dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec15b440 0x7f68dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 -- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc004d00 msgr2=0x7f68dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc004d00 0x7f68dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.435+0000 7f68f0978640 1 -- 172.21.15.107:0/1417267927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc003520 con 0x7f68dc0039a0 2023-11-24T08:15:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.436+0000 7f68f0978640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68dc0039a0 0x7f68dc003dc0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f68e000ba30 tx=0x7f68dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c290bdc24ef47824 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.436+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0032640 con 0x7f68dc0039a0 2023-11-24T08:15:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.436+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e00327a0 con 0x7f68dc0039a0 2023-11-24T08:15:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.436+0000 7f68eb7fe640 1 -- 172.21.15.107:0/1417267927 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0032b30 con 0x7f68dc0039a0 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 -- 172.21.15.107:0/1417267927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68dc0039a0 msgr2=0x7f68dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/1417267927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68dc0039a0 0x7f68dc003dc0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f68e000ba30 tx=0x7f68dc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f1179640 1 -- 172.21.15.107:0/1417267927 reap_dead start 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 -- 172.21.15.107:0/1417267927 shutdown_connections 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 -- 172.21.15.107:0/1417267927 >> 172.21.15.107:0/1417267927 conn(0x7f68ec0b0c50 msgr2=0x7f68ec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 -- 172.21.15.107:0/1417267927 shutdown_connections 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.437+0000 7f68f2bdc640 1 -- 172.21.15.107:0/1417267927 wait complete. 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.439+0000 7f68f2bdc640 1 Processor -- start 2023-11-24T08:15:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.439+0000 7f68f2bdc640 1 -- start start 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc0039a0 0x7f68ec1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec0cda90 0x7f68ec0cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0ce450 con 0x7f68ec156e30 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0c8740 con 0x7f68dc0039a0 2023-11-24T08:15:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68f2bdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec154960 con 0x7f68ec0cda90 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53958/0 (socket says 172.21.15.107:53958) 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.440+0000 7f68ebfff640 1 -- 172.21.15.107:0/704985769 learned_addr learned my addr 172.21.15.107:0/704985769 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68f1179640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec0cda90 0x7f68ec0cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68f0978640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc0039a0 0x7f68ec1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 -- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec0cda90 msgr2=0x7f68ec0cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec0cda90 0x7f68ec0cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 -- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc0039a0 msgr2=0x7f68ec1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc0039a0 0x7f68ec1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 -- 172.21.15.107:0/704985769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec154c40 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68ebfff640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f68d800e4d0 tx=0x7f68d800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80053a0 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d80035f0 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8003980 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.441+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec154f20 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.442+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec0c8310 con 0x7f68ec156e30 2023-11-24T08:15:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.443+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec04e8b0 con 0x7f68ec156e30 2023-11-24T08:15:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.444+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f68d8003ae0 con 0x7f68ec156e30 2023-11-24T08:15:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.445+0000 7f68e97fa640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 0x7f68d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.445+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f68d8088430 con 0x7f68ec156e30 2023-11-24T08:15:34.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.445+0000 7f68f0978640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 0x7f68d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.446+0000 7f68f0978640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 0x7f68d006e770 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f68e000ba00 tx=0x7f68e003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.450+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68d8054bc0 con 0x7f68ec156e30 2023-11-24T08:15:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.666+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol5827", "target": ["mon-mgr", ""]}) v1 -- 0x7f68ec0c8530 con 0x7f68d006c2b0 2023-11-24T08:15:34.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.719+0000 7f68e97fa640 1 -- 172.21.15.107:0/704985769 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68ec0c8530 con 0x7f68d006c2b0 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 msgr2=0x7f68d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 0x7f68d006e770 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f68e000ba00 tx=0x7f68e003d040 comp rx=0 tx=0).stop 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 msgr2=0x7f68ec0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f68d800e4d0 tx=0x7f68d800e990 comp rx=0 tx=0).stop 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 shutdown_connections 2023-11-24T08:15:34.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68d006c2b0 0x7f68d006e770 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68ec0cda90 0x7f68ec0cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68ec156e30 0x7f68ec0c7dd0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 --2- 172.21.15.107:0/704985769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68dc0039a0 0x7f68ec1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 >> 172.21.15.107:0/704985769 conn(0x7f68ec0b0c50 msgr2=0x7f68ec071980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 shutdown_connections 2023-11-24T08:15:34.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.722+0000 7f68f2bdc640 1 -- 172.21.15.107:0/704985769 wait complete. 2023-11-24T08:15:34.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone7555 2023-11-24T08:15:34.751 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:15:34.746+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:15:34.751 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:15:34.746+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:15:34.751 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:15:34.746+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:15:34.751 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:15:34.746+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:15:34.751 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:15:34.746+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:15:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.937+0000 7f2acbf01640 1 Processor -- start 2023-11-24T08:15:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.938+0000 7f2acbf01640 1 -- start start 2023-11-24T08:15:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.939+0000 7f2acbf01640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ac406de10 0x7f2ac415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.939+0000 7f2acbf01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 0x7f2ac40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.939+0000 7f2acbf01640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac415ec90 con 0x7f2ac4072160 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.939+0000 7f2acbf01640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac40cd530 con 0x7f2ac406de10 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.939+0000 7f2acbf01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac40c4ed0 con 0x7f2ac40c49f0 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac949c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ac406de10 0x7f2ac415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac9c9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2ac4072160 0x7f2ac406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57732/0 (socket says 172.21.15.107:57732) 2023-11-24T08:15:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac9c9d640 1 -- 172.21.15.107:0/1186296257 learned_addr learned my addr 172.21.15.107:0/1186296257 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2aca49e640 1 --2- 172.21.15.107:0/1186296257 >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 0x7f2ac40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac949c640 1 -- 172.21.15.107:0/1186296257 >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 msgr2=0x7f2ac40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac949c640 1 --2- 172.21.15.107:0/1186296257 >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 0x7f2ac40705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac949c640 1 -- 172.21.15.107:0/1186296257 >> v1:172.21.15.107:6789/0 conn(0x7f2ac4072160 legacy=0x7f2ac406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac8c9b640 1 -- 172.21.15.107:0/1186296257 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28547429 0 0) 0x7f2ac0001110 con 0x7f2ac4072160 2023-11-24T08:15:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2ac949c640 1 -- 172.21.15.107:0/1186296257 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40c51b0 con 0x7f2ac406de10 2023-11-24T08:15:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.940+0000 7f2aca49e640 1 --2- 172.21.15.107:0/1186296257 >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 0x7f2ac40705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.941+0000 7f2ac949c640 1 --2- 172.21.15.107:0/1186296257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac415e750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4072a30 tx=0x7f2ab402e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8838dd91f9d4197a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2ac8c9b640 1 -- 172.21.15.107:0/1186296257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab402ee60 con 0x7f2ac406de10 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2ac8c9b640 1 -- 172.21.15.107:0/1186296257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab4036440 con 0x7f2ac406de10 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2ac8c9b640 1 -- 172.21.15.107:0/1186296257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40367d0 con 0x7f2ac406de10 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2acbf01640 1 -- 172.21.15.107:0/1186296257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 msgr2=0x7f2ac415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2acbf01640 1 --2- 172.21.15.107:0/1186296257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac415e750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4072a30 tx=0x7f2ab402e0b0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2acbf01640 1 -- 172.21.15.107:0/1186296257 shutdown_connections 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2acbf01640 1 --2- 172.21.15.107:0/1186296257 >> v2:172.21.15.120:3300/0 conn(0x7f2ac40c49f0 0x7f2ac40705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.942+0000 7f2acbf01640 1 --2- 172.21.15.107:0/1186296257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac415e750 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.943+0000 7f2acbf01640 1 -- 172.21.15.107:0/1186296257 >> 172.21.15.107:0/1186296257 conn(0x7f2ac40b0c50 msgr2=0x7f2ac40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.943+0000 7f2acbf01640 1 -- 172.21.15.107:0/1186296257 shutdown_connections 2023-11-24T08:15:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.943+0000 7f2acbf01640 1 -- 172.21.15.107:0/1186296257 wait complete. 2023-11-24T08:15:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.944+0000 7f2acbf01640 1 Processor -- start 2023-11-24T08:15:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.945+0000 7f2acbf01640 1 -- start start 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4072160 0x7f2ac40999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ac40c49f0 0x7f2ac409bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac409a420 con 0x7f2ac406de10 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac409a700 con 0x7f2ac4072160 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2acbf01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac409a9e0 con 0x7f2ac40c49f0 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53988/0 (socket says 172.21.15.107:53988) 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 -- 172.21.15.107:0/381368185 learned_addr learned my addr 172.21.15.107:0/381368185 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 -- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ac40c49f0 msgr2=0x7f2ac409bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ac40c49f0 0x7f2ac409bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 -- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4072160 msgr2=0x7f2ac40999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4072160 0x7f2ac40999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.946+0000 7f2ac9c9d640 1 -- 172.21.15.107:0/381368185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab400b6a0 con 0x7f2ac406de10 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2ac9c9d640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000c780 tx=0x7f2ac000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac00072d0 con 0x7f2ac406de10 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0007450 con 0x7f2ac406de10 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac409acc0 con 0x7f2ac406de10 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac00047e0 con 0x7f2ac406de10 2023-11-24T08:15:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.947+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac4158d20 con 0x7f2ac406de10 2023-11-24T08:15:34.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.948+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac409b1e0 con 0x7f2ac406de10 2023-11-24T08:15:34.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.949+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 28) v1 ==== 91067+0+0 (secure 0 0 0) 0x7f2ac0004960 con 0x7f2ac406de10 2023-11-24T08:15:34.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.950+0000 7f2aba7fc640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 0x7f2aa806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:34.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.950+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ac0087ad0 con 0x7f2ac406de10 2023-11-24T08:15:34.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.950+0000 7f2ac949c640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 0x7f2aa806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:34.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.951+0000 7f2ac949c640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 0x7f2aa806e740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4038000 tx=0x7f2ab4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:34.955+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ac409b1e0 con 0x7f2ac406de10 2023-11-24T08:15:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.168+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone7555", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ac40c5a50 con 0x7f2aa806c280 2023-11-24T08:15:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.208+0000 7f2aba7fc640 1 -- 172.21.15.107:0/381368185 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ab40365b0 con 0x7f2aa806c280 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 msgr2=0x7f2aa806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 0x7f2aa806e740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4038000 tx=0x7f2ab4030000 comp rx=0 tx=0).stop 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 msgr2=0x7f2ac41587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000c780 tx=0x7f2ac000cc40 comp rx=0 tx=0).stop 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 shutdown_connections 2023-11-24T08:15:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa806c280 0x7f2aa806e740 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.210+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ac40c49f0 0x7f2ac409bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.211+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4072160 0x7f2ac40999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.211+0000 7f2acbf01640 1 --2- 172.21.15.107:0/381368185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ac406de10 0x7f2ac41587e0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.211+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 >> 172.21.15.107:0/381368185 conn(0x7f2ac40b0c50 msgr2=0x7f2ac40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.211+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 shutdown_connections 2023-11-24T08:15:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:35.211+0000 7f2acbf01640 1 -- 172.21.15.107:0/381368185 wait complete. 2023-11-24T08:15:35.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:35.230 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:15:35.297 INFO:teuthology.orchestra.run.smithi107.stdout:4 2023-11-24T08:15:40.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:40.300 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:15:40.364 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:15:40.366 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:15:40.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:15:40.482 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:15:40.607 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:15:40.607 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:15:40.608 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:40.670 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:15:40.671 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:40.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:40.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.938+0000 7ff88759e640 1 Processor -- start 2023-11-24T08:15:40.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.938+0000 7ff88759e640 1 -- start start 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.939+0000 7ff88759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 0x7ff88815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.939+0000 7ff88759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8880b1de0 0x7ff8880b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.939+0000 7ff88759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88815bf80 con 0x7ff888144b30 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.939+0000 7ff88759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8880b27b0 con 0x7ff8880b1de0 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff88759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8880b1560 con 0x7ff88815c410 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8880b1de0 0x7ff8880b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8880b1de0 0x7ff8880b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58912/0 (socket says 172.21.15.107:58912) 2023-11-24T08:15:40.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 -- 172.21.15.107:0/2951526285 learned_addr learned my addr 172.21.15.107:0/2951526285 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff87dd9b640 1 --2- 172.21.15.107:0/2951526285 >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 0x7ff88815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 -- 172.21.15.107:0/2951526285 >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 msgr2=0x7ff88815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 --2- 172.21.15.107:0/2951526285 >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 0x7ff88815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 -- 172.21.15.107:0/2951526285 >> v1:172.21.15.120:6789/0 conn(0x7ff888144b30 legacy=0x7ff8880b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.940+0000 7ff886d9d640 1 -- 172.21.15.107:0/2951526285 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8880b1280 con 0x7ff8880b1de0 2023-11-24T08:15:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.941+0000 7ff87dd9b640 1 --2- 172.21.15.107:0/2951526285 >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 0x7ff88815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.942+0000 7ff886d9d640 1 --2- 172.21.15.107:0/2951526285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8880b1de0 0x7ff8880b2270 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff8880cc7a0 tx=0x7ff88002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a40f5d574cb215 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.942+0000 7ff885d9b640 1 -- 172.21.15.107:0/2951526285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88002ca80 con 0x7ff8880b1de0 2023-11-24T08:15:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.942+0000 7ff885d9b640 1 -- 172.21.15.107:0/2951526285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88002cbe0 con 0x7ff8880b1de0 2023-11-24T08:15:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.942+0000 7ff885d9b640 1 -- 172.21.15.107:0/2951526285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800343f0 con 0x7ff8880b1de0 2023-11-24T08:15:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 -- 172.21.15.107:0/2951526285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8880b1de0 msgr2=0x7ff8880b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 --2- 172.21.15.107:0/2951526285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8880b1de0 0x7ff8880b2270 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff8880cc7a0 tx=0x7ff88002bee0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 -- 172.21.15.107:0/2951526285 shutdown_connections 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 --2- 172.21.15.107:0/2951526285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8880b1de0 0x7ff8880b2270 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 --2- 172.21.15.107:0/2951526285 >> v2:172.21.15.120:3300/0 conn(0x7ff88815c410 0x7ff88815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 -- 172.21.15.107:0/2951526285 >> 172.21.15.107:0/2951526285 conn(0x7ff8880cb830 msgr2=0x7ff8880cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 -- 172.21.15.107:0/2951526285 shutdown_connections 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.943+0000 7ff88759e640 1 -- 172.21.15.107:0/2951526285 wait complete. 2023-11-24T08:15:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.945+0000 7ff88759e640 1 Processor -- start 2023-11-24T08:15:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.945+0000 7ff88759e640 1 -- start start 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 0x7ff888071410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff888144b30 0x7ff8880702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8881f40d0 con 0x7ff88815c410 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88813b620 con 0x7ff8880b1de0 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff88759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88813b010 con 0x7ff888144b30 2023-11-24T08:15:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff886d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff886d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58922/0 (socket says 172.21.15.107:58922) 2023-11-24T08:15:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.946+0000 7ff886d9d640 1 -- 172.21.15.107:0/2743879595 learned_addr learned my addr 172.21.15.107:0/2743879595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff88659c640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 0x7ff888071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff87dd9b640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff888144b30 0x7ff8880702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 -- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff888144b30 msgr2=0x7ff8880702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff888144b30 0x7ff8880702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 -- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 msgr2=0x7ff888071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 0x7ff888071410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 -- 172.21.15.107:0/2743879595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009b30 con 0x7ff88815c410 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff88659c640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 0x7ff888071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff886d9d640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff880036040 tx=0x7ff880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88003e450 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88003e5d0 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8881f3b10 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.947+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88003e980 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.948+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88813b2b0 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.949+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88813b7d0 con 0x7ff88815c410 2023-11-24T08:15:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.950+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7ff88002cd50 con 0x7ff88815c410 2023-11-24T08:15:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.951+0000 7ff87f7fe640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 0x7ff86806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.951+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7ff8800aaef0 con 0x7ff88815c410 2023-11-24T08:15:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.951+0000 7ff88659c640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 0x7ff86806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.952+0000 7ff88659c640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 0x7ff86806e7d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff878003eb0 tx=0x7ff878009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:40.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:40.956+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff88813b7d0 con 0x7ff88815c410 2023-11-24T08:15:41.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.170+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff88813b7d0 con 0x7ff88815c410 2023-11-24T08:15:41.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.172+0000 7ff87f7fe640 1 -- 172.21.15.107:0/2743879595 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16731 (secure 0 0 0) 0x7ff88813b7d0 con 0x7ff88815c410 2023-11-24T08:15:41.174 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:41.174 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":134,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:38.744776+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.5","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:41.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 msgr2=0x7ff86806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 0x7ff86806e7d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff878003eb0 tx=0x7ff878009040 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 msgr2=0x7ff8881f3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff880036040 tx=0x7ff880004a40 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 shutdown_connections 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff86806c310 0x7ff86806e7d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff88815c410 0x7ff8881f3070 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff888144b30 0x7ff8880702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 --2- 172.21.15.107:0/2743879595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8880b1de0 0x7ff888071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 >> 172.21.15.107:0/2743879595 conn(0x7ff8880cb830 msgr2=0x7ff8880b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 shutdown_connections 2023-11-24T08:15:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.174+0000 7ff88759e640 1 -- 172.21.15.107:0/2743879595 wait complete. 2023-11-24T08:15:41.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:14:26.165775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:14:27.450883+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:15:41.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:15:41.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.385+0000 7f96e3ae7640 1 Processor -- start 2023-11-24T08:15:41.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.385+0000 7f96e3ae7640 1 -- start start 2023-11-24T08:15:41.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e3ae7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e3ae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc15eea0 con 0x7f96dc0c57e0 2023-11-24T08:15:41.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e3ae7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc0abf30 con 0x7f96dc15b440 2023-11-24T08:15:41.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e3ae7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc15b820 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e1082640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e1082640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58926/0 (socket says 172.21.15.107:58926) 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.388+0000 7f96e1082640 1 -- 172.21.15.107:0/3650732838 learned_addr learned my addr 172.21.15.107:0/3650732838 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.387+0000 7f96e2084640 1 --1- 172.21.15.107:0/3650732838 >> v1:172.21.15.107:6789/0 conn(0x7f96dc0c57e0 0x7f96dc06c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36998/0 (socket says 172.21.15.107:36998) 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.388+0000 7f96e1082640 1 -- 172.21.15.107:0/3650732838 >> v1:172.21.15.120:6789/0 conn(0x7f96dc15b440 legacy=0x7f96dc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.388+0000 7f96e1082640 1 -- 172.21.15.107:0/3650732838 >> v1:172.21.15.107:6789/0 conn(0x7f96dc0c57e0 legacy=0x7f96dc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.388+0000 7f96e1082640 1 -- 172.21.15.107:0/3650732838 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc15f0f0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.389+0000 7f96e1082640 1 --2- 172.21.15.107:0/3650732838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009870 tx=0x7f96d002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925ac53cc12d9c4d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.389+0000 7f96e0881640 1 -- 172.21.15.107:0/3650732838 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0031860 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.389+0000 7f96e0881640 1 -- 172.21.15.107:0/3650732838 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d00319c0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.389+0000 7f96e0881640 1 -- 172.21.15.107:0/3650732838 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0031d50 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 -- 172.21.15.107:0/3650732838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 msgr2=0x7f96dc0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/3650732838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009870 tx=0x7f96d002b900 comp rx=0 tx=0).stop 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 -- 172.21.15.107:0/3650732838 shutdown_connections 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/3650732838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc0c52a0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 -- 172.21.15.107:0/3650732838 >> 172.21.15.107:0/3650732838 conn(0x7f96dc0b0c50 msgr2=0x7f96dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:41.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 -- 172.21.15.107:0/3650732838 shutdown_connections 2023-11-24T08:15:41.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.390+0000 7f96e3ae7640 1 -- 172.21.15.107:0/3650732838 wait complete. 2023-11-24T08:15:41.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.392+0000 7f96e3ae7640 1 Processor -- start 2023-11-24T08:15:41.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.392+0000 7f96e3ae7640 1 -- start start 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96dc0c57e0 0x7f96dc15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96dc15b440 0x7f96dc0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0c74b0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc15a4c0 con 0x7f96dc0c57e0 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e3ae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc154330 con 0x7f96dc15b440 2023-11-24T08:15:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e1082640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e1082640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58942/0 (socket says 172.21.15.107:58942) 2023-11-24T08:15:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.393+0000 7f96e1082640 1 -- 172.21.15.107:0/33803089 learned_addr learned my addr 172.21.15.107:0/33803089 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 -- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96dc15b440 msgr2=0x7f96dc0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96dc15b440 0x7f96dc0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1883640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96dc0c57e0 0x7f96dc15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 -- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96dc0c57e0 msgr2=0x7f96dc15a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96dc0c57e0 0x7f96dc15a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 -- 172.21.15.107:0/33803089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d00094d0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e1082640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000c00 tx=0x7f96d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0031810 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d00443f0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc158bd0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d00447a0 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.394+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc158e70 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.396+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc158a30 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.396+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f96d004d450 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.398+0000 7f96ce7fc640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 0x7f96c006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.398+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f96d00aaa20 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.398+0000 7f96e1883640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 0x7f96c006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.399+0000 7f96e1883640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 0x7f96c006e7d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f96d800a750 tx=0x7f96d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.402+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96dc158a30 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.593+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96dc158a10 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.594+0000 7f96ce7fc640 1 -- 172.21.15.107:0/33803089 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96d0035020 con 0x7f96dc0cc0f0 2023-11-24T08:15:41.596 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 msgr2=0x7f96c006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 0x7f96c006e7d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f96d800a750 tx=0x7f96d8008040 comp rx=0 tx=0).stop 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 msgr2=0x7f96dc153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000c00 tx=0x7f96d00046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 shutdown_connections 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f96c006c310 0x7f96c006e7d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96dc15b440 0x7f96dc0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96dc0cc0f0 0x7f96dc153820 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 --2- 172.21.15.107:0/33803089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96dc0c57e0 0x7f96dc15a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 >> 172.21.15.107:0/33803089 conn(0x7f96dc0b0c50 msgr2=0x7f96dc0711a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:41.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 shutdown_connections 2023-11-24T08:15:41.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.597+0000 7f96e3ae7640 1 -- 172.21.15.107:0/33803089 wait complete. 2023-11-24T08:15:41.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:15:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.821+0000 7f621d215640 1 Processor -- start 2023-11-24T08:15:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.822+0000 7f621d215640 1 -- start start 2023-11-24T08:15:41.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.823+0000 7f621d215640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f621815b440 0x7f62180cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.823+0000 7f621d215640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180c5dc0 con 0x7f62180c59b0 2023-11-24T08:15:41.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.823+0000 7f621d215640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621815eea0 con 0x7f62180cbf20 2023-11-24T08:15:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.823+0000 7f621d215640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62180cc3d0 con 0x7f621815b440 2023-11-24T08:15:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62180c59b0 0x7f621806c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37010/0 (socket says 172.21.15.107:37010) 2023-11-24T08:15:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621759e640 1 -- 172.21.15.107:0/3462855278 learned_addr learned my addr 172.21.15.107:0/3462855278 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115946514 0 0) 0x7f62180c5dc0 con 0x7f62180c59b0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6204003520 con 0x7f62180c59b0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f6216d9d640 1 --2- 172.21.15.107:0/3462855278 >> v2:172.21.15.120:3300/0 conn(0x7f621815b440 0x7f62180cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679138809 0 0) 0x7f6204003520 con 0x7f62180c59b0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 >> v2:172.21.15.120:3300/0 conn(0x7f621815b440 msgr2=0x7f62180cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 --2- 172.21.15.107:0/3462855278 >> v2:172.21.15.120:3300/0 conn(0x7f621815b440 0x7f62180cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 >> v1:172.21.15.120:6789/0 conn(0x7f62180cbf20 legacy=0x7f62180c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180abf30 con 0x7f62180c59b0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f62100019f0 con 0x7f62180c59b0 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.824+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 >> v1:172.21.15.107:6789/0 conn(0x7f62180c59b0 legacy=0x7f621806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62040039c0 0x7f6204003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62040047c0 0x7f6204004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 0x7f620400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180cc3d0 con 0x7f6204005250 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620400c1a0 con 0x7f62040047c0 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620400c360 con 0x7f62040039c0 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 0x7f620400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f6216d9d640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62040039c0 0x7f6204003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f620ffff640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62040047c0 0x7f6204004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 -- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62040039c0 msgr2=0x7f6204003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62040039c0 0x7f6204003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 -- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62040047c0 msgr2=0x7f6204004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62040047c0 0x7f6204004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.825+0000 7f621759e640 1 -- 172.21.15.107:0/3462855278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6204003520 con 0x7f6204005250 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.826+0000 7f621759e640 1 -- 172.21.15.107:0/3462855278 reap_dead start 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621759e640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 0x7f620400b980 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f621000a810 tx=0x7f62040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fd564426616035 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210034630 con 0x7f6204005250 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621002f070 con 0x7f6204005250 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621659c640 1 -- 172.21.15.107:0/3462855278 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210034a10 con 0x7f6204005250 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621d215640 1 -- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 msgr2=0x7f620400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621d215640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 0x7f620400b980 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f621000a810 tx=0x7f62040057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621d215640 1 -- 172.21.15.107:0/3462855278 shutdown_connections 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621d215640 1 --2- 172.21.15.107:0/3462855278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6204005250 0x7f620400b980 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.827+0000 7f621d215640 1 -- 172.21.15.107:0/3462855278 >> 172.21.15.107:0/3462855278 conn(0x7f62180b0c50 msgr2=0x7f62180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.828+0000 7f621d215640 1 -- 172.21.15.107:0/3462855278 shutdown_connections 2023-11-24T08:15:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.828+0000 7f621d215640 1 -- 172.21.15.107:0/3462855278 wait complete. 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.829+0000 7f621d215640 1 Processor -- start 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.830+0000 7f621d215640 1 -- start start 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.830+0000 7f621d215640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6204005250 0x7f6218156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.830+0000 7f621d215640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.830+0000 7f621d215640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 0x7f621815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f621d215640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621815d130 con 0x7f6218070d90 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f621d215640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218071730 con 0x7f6204005250 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f621d215640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6218157180 con 0x7f621815c770 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58964/0 (socket says 172.21.15.107:58964) 2023-11-24T08:15:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 -- 172.21.15.107:0/1008215332 learned_addr learned my addr 172.21.15.107:0/1008215332 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f6216d9d640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6204005250 0x7f6218156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f621759e640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 0x7f621815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 -- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 msgr2=0x7f621815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 0x7f621815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 -- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6204005250 msgr2=0x7f6218156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6204005250 0x7f6218156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f620ffff640 1 -- 172.21.15.107:0/1008215332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180c83d0 con 0x7f6218070d90 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.831+0000 7f621759e640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 0x7f621815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f620ffff640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f620000e490 tx=0x7f620000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62000135f0 con 0x7f6218070d90 2023-11-24T08:15:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6200013750 con 0x7f6218070d90 2023-11-24T08:15:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6200013b00 con 0x7f6218070d90 2023-11-24T08:15:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62180c86b0 con 0x7f6218070d90 2023-11-24T08:15:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.832+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180c8c70 con 0x7f6218070d90 2023-11-24T08:15:41.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.833+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62180c8e90 con 0x7f6218070d90 2023-11-24T08:15:41.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.834+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f6200013c60 con 0x7f6218070d90 2023-11-24T08:15:41.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.835+0000 7f620f7fe640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 0x7f61f806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:41.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.835+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6200088570 con 0x7f6218070d90 2023-11-24T08:15:41.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.836+0000 7f6216d9d640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 0x7f61f806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:41.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.836+0000 7f6216d9d640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 0x7f61f806e800 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6208006fd0 tx=0x7f6208008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:41.840+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6200054cf0 con 0x7f6218070d90 2023-11-24T08:15:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.029+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f621804e8b0 con 0x7f6218070d90 2023-11-24T08:15:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.030+0000 7f620f7fe640 1 -- 172.21.15.107:0/1008215332 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6200058310 con 0x7f6218070d90 2023-11-24T08:15:42.032 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 msgr2=0x7f61f806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 0x7f61f806e800 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6208006fd0 tx=0x7f6208008040 comp rx=0 tx=0).stop 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 msgr2=0x7f62180711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f620000e490 tx=0x7f620000e950 comp rx=0 tx=0).stop 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 shutdown_connections 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61f806c340 0x7f61f806e800 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f621815c770 0x7f621815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6218070d90 0x7f62180711f0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 --2- 172.21.15.107:0/1008215332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6204005250 0x7f6218156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 >> 172.21.15.107:0/1008215332 conn(0x7f62180b0c50 msgr2=0x7f62180c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:42.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 shutdown_connections 2023-11-24T08:15:42.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.033+0000 7f621d215640 1 -- 172.21.15.107:0/1008215332 wait complete. 2023-11-24T08:15:42.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:15:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.266+0000 7efcb5a5c640 1 Processor -- start 2023-11-24T08:15:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.266+0000 7efcb5a5c640 1 -- start start 2023-11-24T08:15:42.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcb5a5c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efcb015b440 0x7efcb00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcb5a5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcb5a5c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb015eea0 con 0x7efcb00c57e0 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcb5a5c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00cc3d0 con 0x7efcb015b440 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcb5a5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb015b820 con 0x7efcb00cbf20 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcaeffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efcb015b440 0x7efcb00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcaeffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efcb015b440 0x7efcb00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58972/0 (socket says 172.21.15.107:58972) 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.268+0000 7efcaeffd640 1 -- 172.21.15.107:0/1260037680 learned_addr learned my addr 172.21.15.107:0/1260037680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.269+0000 7efca7fff640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.269+0000 7efcaeffd640 1 -- 172.21.15.107:0/1260037680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 msgr2=0x7efcb00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.269+0000 7efcaeffd640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.269+0000 7efcaeffd640 1 -- 172.21.15.107:0/1260037680 >> v1:172.21.15.120:6789/0 conn(0x7efcb00c57e0 legacy=0x7efcb00c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.269+0000 7efcaeffd640 1 -- 172.21.15.107:0/1260037680 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb006d600 con 0x7efcb015b440 2023-11-24T08:15:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.270+0000 7efcaeffd640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb015b440 0x7efcb00cb9e0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7efca0009870 tx=0x7efca002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788e380a1fcb5671 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcae7fc640 1 -- 172.21.15.107:0/1260037680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca00317c0 con 0x7efcb015b440 2023-11-24T08:15:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcae7fc640 1 -- 172.21.15.107:0/1260037680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0031920 con 0x7efcb015b440 2023-11-24T08:15:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcae7fc640 1 -- 172.21.15.107:0/1260037680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0031cb0 con 0x7efcb015b440 2023-11-24T08:15:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 -- 172.21.15.107:0/1260037680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb015b440 msgr2=0x7efcb00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb015b440 0x7efcb00cb9e0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7efca0009870 tx=0x7efca002b860 comp rx=0 tx=0).stop 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 -- 172.21.15.107:0/1260037680 shutdown_connections 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/1260037680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb015b440 0x7efcb00cb9e0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.271+0000 7efcb5a5c640 1 -- 172.21.15.107:0/1260037680 >> 172.21.15.107:0/1260037680 conn(0x7efcb00b0c50 msgr2=0x7efcb00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.272+0000 7efcb5a5c640 1 -- 172.21.15.107:0/1260037680 shutdown_connections 2023-11-24T08:15:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.272+0000 7efcb5a5c640 1 -- 172.21.15.107:0/1260037680 wait complete. 2023-11-24T08:15:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.273+0000 7efcb5a5c640 1 Processor -- start 2023-11-24T08:15:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.273+0000 7efcb5a5c640 1 -- start start 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb015b440 0x7efcb0156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00b4a50 con 0x7efcb00c57e0 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb015c390 con 0x7efcb015b440 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.274+0000 7efcb5a5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00b4770 con 0x7efcb00cbf20 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58988/0 (socket says 172.21.15.107:58988) 2023-11-24T08:15:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 -- 172.21.15.107:0/3523269579 learned_addr learned my addr 172.21.15.107:0/3523269579 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 -- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 msgr2=0x7efcb00b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 -- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb015b440 msgr2=0x7efcb0156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb015b440 0x7efcb0156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 -- 172.21.15.107:0/3523269579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00094d0 con 0x7efcb00c57e0 2023-11-24T08:15:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.275+0000 7efcaeffd640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7efca0000c00 tx=0x7efca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.276+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0032860 con 0x7efcb00c57e0 2023-11-24T08:15:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.276+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca00329c0 con 0x7efcb00c57e0 2023-11-24T08:15:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.276+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0032d70 con 0x7efcb00c57e0 2023-11-24T08:15:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.276+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb015bcd0 con 0x7efcb00c57e0 2023-11-24T08:15:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.276+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb00ce830 con 0x7efcb00c57e0 2023-11-24T08:15:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.277+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb00cea20 con 0x7efcb00c57e0 2023-11-24T08:15:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.278+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7efca004d3f0 con 0x7efcb00c57e0 2023-11-24T08:15:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.279+0000 7efcacff9640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 0x7efc9006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.279+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7efca00aa180 con 0x7efcb00c57e0 2023-11-24T08:15:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.279+0000 7efca7fff640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 0x7efc9006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.280+0000 7efca7fff640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 0x7efc9006e7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efcb009d0d0 tx=0x7efc98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.283+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efca0076c80 con 0x7efcb00c57e0 2023-11-24T08:15:42.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:15:42.470 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:15:42.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.477+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efcb015bb30 con 0x7efcb00c57e0 2023-11-24T08:15:42.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.477+0000 7efcacff9640 1 -- 172.21.15.107:0/3523269579 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7efcb015bb30 con 0x7efcb00c57e0 2023-11-24T08:15:42.480 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:15:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 msgr2=0x7efc9006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 0x7efc9006e7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efcb009d0d0 tx=0x7efc98008040 comp rx=0 tx=0).stop 2023-11-24T08:15:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 msgr2=0x7efcb00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7efca0000c00 tx=0x7efca00046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 shutdown_connections 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efc9006c310 0x7efc9006e7d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcb015b440 0x7efcb0156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcb00cbf20 0x7efcb00b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 --2- 172.21.15.107:0/3523269579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcb00c57e0 0x7efcb00ce2f0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 >> 172.21.15.107:0/3523269579 conn(0x7efcb00b0c50 msgr2=0x7efcb00cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.480+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 shutdown_connections 2023-11-24T08:15:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.481+0000 7efcb5a5c640 1 -- 172.21.15.107:0/3523269579 wait complete. 2023-11-24T08:15:42.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:15:42.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.743+0000 7fa2767a9640 1 Processor -- start 2023-11-24T08:15:42.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.743+0000 7fa2767a9640 1 -- start start 2023-11-24T08:15:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa2767a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa270161060 0x7fa2701614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa2767a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27006dca0 con 0x7fa2700af000 2023-11-24T08:15:42.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa2767a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27006df40 con 0x7fa27006a670 2023-11-24T08:15:42.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa2767a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2700af3e0 con 0x7fa270161060 2023-11-24T08:15:42.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa26ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa2700af000 0x7fa2700b3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37012/0 (socket says 172.21.15.107:37012) 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa26ffff640 1 -- 172.21.15.107:0/2570600535 learned_addr learned my addr 172.21.15.107:0/2570600535 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418575003 0 0) 0x7fa27006dca0 con 0x7fa2700af000 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.745+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25c003520 con 0x7fa2700af000 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2570600535 >> v2:172.21.15.120:3300/0 conn(0x7fa270161060 0x7fa2701614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346052935 0 0) 0x7fa25c003520 con 0x7fa2700af000 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 >> v2:172.21.15.120:3300/0 conn(0x7fa270161060 msgr2=0x7fa2701614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 --2- 172.21.15.107:0/2570600535 >> v2:172.21.15.120:3300/0 conn(0x7fa270161060 0x7fa2701614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 >> v1:172.21.15.120:6789/0 conn(0x7fa27006a670 legacy=0x7fa27006ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27006e220 con 0x7fa2700af000 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa2600019f0 con 0x7fa2700af000 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 >> v1:172.21.15.107:6789/0 conn(0x7fa2700af000 legacy=0x7fa2700b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0039f0 0x7fa25c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 0x7fa25c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa25c005380 0x7fa25c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700af3e0 con 0x7fa25c0047c0 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c00c210 con 0x7fa25c005380 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c00c3d0 con 0x7fa25c0039f0 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.746+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 0x7fa25c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa274d46640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa25c005380 0x7fa25c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa26f7fe640 1 -- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0039f0 msgr2=0x7fa25c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0039f0 0x7fa25c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa26f7fe640 1 -- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa25c005380 msgr2=0x7fa25c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa25c005380 0x7fa25c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa26f7fe640 1 -- 172.21.15.107:0/2570600535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c003520 con 0x7fa25c0047c0 2023-11-24T08:15:42.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa274d46640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa25c005380 0x7fa25c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:42.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.747+0000 7fa274d46640 1 -- 172.21.15.107:0/2570600535 reap_dead start 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.748+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 0x7fa25c004c40 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa25800a970 tx=0x7fa25c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47bda356c7cc10 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.748+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258005990 con 0x7fa25c0047c0 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.748+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258005af0 con 0x7fa25c0047c0 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.748+0000 7fa26effd640 1 -- 172.21.15.107:0/2570600535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258006020 con 0x7fa25c0047c0 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 -- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 msgr2=0x7fa25c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 0x7fa25c004c40 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa25800a970 tx=0x7fa25c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 -- 172.21.15.107:0/2570600535 shutdown_connections 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2570600535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa25c0047c0 0x7fa25c004c40 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 -- 172.21.15.107:0/2570600535 >> 172.21.15.107:0/2570600535 conn(0x7fa2700cf900 msgr2=0x7fa2700cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:42.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 -- 172.21.15.107:0/2570600535 shutdown_connections 2023-11-24T08:15:42.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.749+0000 7fa2767a9640 1 -- 172.21.15.107:0/2570600535 wait complete. 2023-11-24T08:15:42.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.751+0000 7fa2767a9640 1 Processor -- start 2023-11-24T08:15:42.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.751+0000 7fa2767a9640 1 -- start start 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0047c0 0x7fa27015f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 0x7fa270156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa270155a90 con 0x7fa2701565a0 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27015f9d0 con 0x7fa27014cb10 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa2767a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa270160280 con 0x7fa25c0047c0 2023-11-24T08:15:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa274d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa274d46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59008/0 (socket says 172.21.15.107:59008) 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa274d46640 1 -- 172.21.15.107:0/2087760295 learned_addr learned my addr 172.21.15.107:0/2087760295 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.752+0000 7fa26ffff640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0047c0 0x7fa27015f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 0x7fa270156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 -- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0047c0 msgr2=0x7fa27015f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0047c0 0x7fa27015f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 -- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 msgr2=0x7fa270156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 0x7fa270156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 -- 172.21.15.107:0/2087760295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa270160560 con 0x7fa2701565a0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa26f7fe640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 0x7fa270156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa274d46640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fa2700d0a00 tx=0x7fa26800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268005230 con 0x7fa2701565a0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2680053b0 con 0x7fa2701565a0 2023-11-24T08:15:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268007260 con 0x7fa2701565a0 2023-11-24T08:15:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa270160840 con 0x7fa2701565a0 2023-11-24T08:15:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.753+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27015ef70 con 0x7fa2701565a0 2023-11-24T08:15:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.755+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27015eaf0 con 0x7fa2701565a0 2023-11-24T08:15:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.755+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fa2680073c0 con 0x7fa2701565a0 2023-11-24T08:15:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.757+0000 7fa26cff9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 0x7fa25006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.757+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fa2680872c0 con 0x7fa2701565a0 2023-11-24T08:15:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.757+0000 7fa26ffff640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 0x7fa25006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.758+0000 7fa26ffff640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 0x7fa25006e800 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa2600020b0 tx=0x7fa260001a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.761+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa27015eaf0 con 0x7fa2701565a0 2023-11-24T08:15:42.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.965+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa27015eaf0 con 0x7fa2701565a0 2023-11-24T08:15:42.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.966+0000 7fa26cff9640 1 -- 172.21.15.107:0/2087760295 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v134) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa27015eaf0 con 0x7fa2701565a0 2023-11-24T08:15:42.968 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:15:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 msgr2=0x7fa25006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 0x7fa25006e800 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa2600020b0 tx=0x7fa260001a40 comp rx=0 tx=0).stop 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 msgr2=0x7fa27015a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fa2700d0a00 tx=0x7fa26800d890 comp rx=0 tx=0).stop 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 shutdown_connections 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa25006c340 0x7fa25006e800 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701565a0 0x7fa27015a630 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa27014cb10 0x7fa270156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 --2- 172.21.15.107:0/2087760295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa25c0047c0 0x7fa27015f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 >> 172.21.15.107:0/2087760295 conn(0x7fa2700cf900 msgr2=0x7fa27015d870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 shutdown_connections 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:42.969+0000 7fa2767a9640 1 -- 172.21.15.107:0/2087760295 wait complete. 2023-11-24T08:15:42.989 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:15:42.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:42.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:15:43.133 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:15:43.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:43.134 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:15:43.184 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:15:43.185 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:43.225 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:15:43.225 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:43.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.524+0000 7f509759e640 1 Processor -- start 2023-11-24T08:15:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.524+0000 7f509759e640 1 -- start start 2023-11-24T08:15:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5098143870 0x7f5098146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50980b1180 0x7f50980cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5098161510 con 0x7f5098161080 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5098070460 con 0x7f5098143870 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50981617f0 con 0x7f50980b1180 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f509659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5098143870 0x7f5098146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f508ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5098161080 0x7f50980b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37024/0 (socket says 172.21.15.107:37024) 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.526+0000 7f508ffff640 1 -- 172.21.15.107:0/1934381170 learned_addr learned my addr 172.21.15.107:0/1934381170 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f5096d9d640 1 --2- 172.21.15.107:0/1934381170 >> v2:172.21.15.120:3300/0 conn(0x7f50980b1180 0x7f50980cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f509659c640 1 -- 172.21.15.107:0/1934381170 >> v2:172.21.15.120:3300/0 conn(0x7f50980b1180 msgr2=0x7f50980cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f509659c640 1 --2- 172.21.15.107:0/1934381170 >> v2:172.21.15.120:3300/0 conn(0x7f50980b1180 0x7f50980cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f5095d9b640 1 -- 172.21.15.107:0/1934381170 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669973014 0 0) 0x7f5098161510 con 0x7f5098161080 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f509659c640 1 -- 172.21.15.107:0/1934381170 >> v1:172.21.15.107:6789/0 conn(0x7f5098161080 legacy=0x7f50980b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.527+0000 7f509659c640 1 -- 172.21.15.107:0/1934381170 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098161ad0 con 0x7f5098143870 2023-11-24T08:15:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.529+0000 7f509659c640 1 --2- 172.21.15.107:0/1934381170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f5098146970 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f50980ae9e0 tx=0x7f508802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ca2bea60ec26e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.529+0000 7f5095d9b640 1 -- 172.21.15.107:0/1934381170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088031830 con 0x7f5098143870 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.529+0000 7f5095d9b640 1 -- 172.21.15.107:0/1934381170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5088031990 con 0x7f5098143870 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.529+0000 7f5095d9b640 1 -- 172.21.15.107:0/1934381170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088031d20 con 0x7f5098143870 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 -- 172.21.15.107:0/1934381170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 msgr2=0x7f5098146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 --2- 172.21.15.107:0/1934381170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f5098146970 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f50980ae9e0 tx=0x7f508802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 -- 172.21.15.107:0/1934381170 shutdown_connections 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 --2- 172.21.15.107:0/1934381170 >> v2:172.21.15.120:3300/0 conn(0x7f50980b1180 0x7f50980cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 --2- 172.21.15.107:0/1934381170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f5098146970 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 -- 172.21.15.107:0/1934381170 >> 172.21.15.107:0/1934381170 conn(0x7f50980cb830 msgr2=0x7f50980cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 -- 172.21.15.107:0/1934381170 shutdown_connections 2023-11-24T08:15:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.530+0000 7f509759e640 1 -- 172.21.15.107:0/1934381170 wait complete. 2023-11-24T08:15:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.532+0000 7f509759e640 1 Processor -- start 2023-11-24T08:15:43.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.532+0000 7f509759e640 1 -- start start 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50980b1180 0x7f5098131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5098133730 0x7f50980b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5098132dd0 con 0x7f5098143870 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5098132150 con 0x7f5098133730 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.533+0000 7f509759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5098132430 con 0x7f50980b1180 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59016/0 (socket says 172.21.15.107:59016) 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 -- 172.21.15.107:0/4019607313 learned_addr learned my addr 172.21.15.107:0/4019607313 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 -- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50980b1180 msgr2=0x7f5098131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50980b1180 0x7f5098131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 -- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5098133730 msgr2=0x7f50980b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5098133730 0x7f50980b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 -- 172.21.15.107:0/4019607313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088009400 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.534+0000 7f508ffff640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f508000c430 tx=0x7f508000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.535+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800074e0 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.535+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5080007660 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.535+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080004a20 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.535+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5098132710 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.535+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50980b2450 con 0x7f5098143870 2023-11-24T08:15:43.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.536+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5098132f10 con 0x7f5098143870 2023-11-24T08:15:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.537+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f5080004b80 con 0x7f5098143870 2023-11-24T08:15:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.538+0000 7f508effd640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 0x7f507806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.538+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f50800884d0 con 0x7f5098143870 2023-11-24T08:15:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.538+0000 7f509659c640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 0x7f507806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.539+0000 7f509659c640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 0x7f507806e7d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5088036000 tx=0x7f5088030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.543+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5098132f10 con 0x7f5098143870 2023-11-24T08:15:43.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.737+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5098132f10 con 0x7f5098143870 2023-11-24T08:15:43.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.740+0000 7f508effd640 1 -- 172.21.15.107:0/4019607313 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16731 (secure 0 0 0) 0x7f5098132f10 con 0x7f5098143870 2023-11-24T08:15:43.743 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":134,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:38.744776+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.5","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 msgr2=0x7f507806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 0x7f507806e7d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5088036000 tx=0x7f5088030000 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 msgr2=0x7f50981331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f508000c430 tx=0x7f508000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 shutdown_connections 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f507806c310 0x7f507806e7d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5098133730 0x7f50980b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5098143870 0x7f50981331f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 --2- 172.21.15.107:0/4019607313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50980b1180 0x7f5098131b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.743+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 >> 172.21.15.107:0/4019607313 conn(0x7f50980cb830 msgr2=0x7f50980721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.744+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 shutdown_connections 2023-11-24T08:15:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.744+0000 7f509759e640 1 -- 172.21.15.107:0/4019607313 wait complete. 2023-11-24T08:15:43.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:14:26.165775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:14:27.450883+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:15:43.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:15:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.980+0000 7fe2bc67c640 1 Processor -- start 2023-11-24T08:15:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.980+0000 7fe2bc67c640 1 -- start start 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bc67c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe2b4146eb0 0x7fe2b4161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bc67c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2b40cd2e0 0x7fe2b4070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bc67c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4161a10 con 0x7fe2b406e160 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bc67c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b4161cf0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bc67c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b4161fd0 con 0x7fe2b4146eb0 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2b40cd2e0 0x7fe2b4070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2b40cd2e0 0x7fe2b4070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59022/0 (socket says 172.21.15.107:59022) 2023-11-24T08:15:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 -- 172.21.15.107:0/3676352418 learned_addr learned my addr 172.21.15.107:0/3676352418 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 -- 172.21.15.107:0/3676352418 >> v2:172.21.15.120:3300/0 conn(0x7fe2b4146eb0 msgr2=0x7fe2b4161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 --2- 172.21.15.107:0/3676352418 >> v2:172.21.15.120:3300/0 conn(0x7fe2b4146eb0 0x7fe2b4161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 -- 172.21.15.107:0/3676352418 >> v1:172.21.15.120:6789/0 conn(0x7fe2b406e160 legacy=0x7fe2b40ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.982+0000 7fe2bac19640 1 -- 172.21.15.107:0/3676352418 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4070660 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2bac19640 1 --2- 172.21.15.107:0/3676352418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b4070e40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2b406ea50 tx=0x7fe2ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc94d37eab41ee57 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2b9416640 1 -- 172.21.15.107:0/3676352418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031870 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2b9416640 1 -- 172.21.15.107:0/3676352418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0319d0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2b9416640 1 -- 172.21.15.107:0/3676352418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031d60 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2bc67c640 1 -- 172.21.15.107:0/3676352418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 msgr2=0x7fe2b4070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.984+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/3676352418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b4070e40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2b406ea50 tx=0x7fe2ac02b910 comp rx=0 tx=0).stop 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 -- 172.21.15.107:0/3676352418 shutdown_connections 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/3676352418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b4070e40 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/3676352418 >> v2:172.21.15.120:3300/0 conn(0x7fe2b4146eb0 0x7fe2b4161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 -- 172.21.15.107:0/3676352418 >> 172.21.15.107:0/3676352418 conn(0x7fe2b40b0c50 msgr2=0x7fe2b40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 -- 172.21.15.107:0/3676352418 shutdown_connections 2023-11-24T08:15:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.985+0000 7fe2bc67c640 1 -- 172.21.15.107:0/3676352418 wait complete. 2023-11-24T08:15:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.986+0000 7fe2bc67c640 1 Processor -- start 2023-11-24T08:15:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.987+0000 7fe2bc67c640 1 -- start start 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 0x7fe2b4143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 0x7fe2b41e8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4143a40 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4143ee0 con 0x7fe2b4146eb0 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bc67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415f400 con 0x7fe2b406e160 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59032/0 (socket says 172.21.15.107:59032) 2023-11-24T08:15:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 -- 172.21.15.107:0/1657315221 learned_addr learned my addr 172.21.15.107:0/1657315221 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bac19640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 0x7fe2b41e8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2ba418640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 0x7fe2b4143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 -- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 msgr2=0x7fe2b4143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 0x7fe2b4143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 -- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 msgr2=0x7fe2b41e8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 0x7fe2b41e8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2b9c17640 1 -- 172.21.15.107:0/1657315221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac0094d0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2bac19640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 0x7fe2b41e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.988+0000 7fe2ba418640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 0x7fe2b4143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2b9c17640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c440 tx=0x7fe2a400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4005480 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a40035f0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b415e7f0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40039c0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.989+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b415edb0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.990+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b415f590 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.991+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fe2a4003b20 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.992+0000 7fe2aaffd640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 0x7fe29806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.992+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe2a40878f0 con 0x7fe2b40cd2e0 2023-11-24T08:15:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.992+0000 7fe2ba418640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 0x7fe29806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.993+0000 7fe2ba418640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 0x7fe29806e7d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0007eb0 tx=0x7fe2b00073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:43.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:43.996+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2a40543c0 con 0x7fe2b40cd2e0 2023-11-24T08:15:44.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.187+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2b404e8b0 con 0x7fe2b40cd2e0 2023-11-24T08:15:44.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.188+0000 7fe2aaffd640 1 -- 172.21.15.107:0/1657315221 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2a4017020 con 0x7fe2b40cd2e0 2023-11-24T08:15:44.190 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:15:44.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 msgr2=0x7fe29806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 0x7fe29806e7d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0007eb0 tx=0x7fe2b00073c0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 msgr2=0x7fe2b41e89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c440 tx=0x7fe2a400c900 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 shutdown_connections 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c310 0x7fe29806e7d0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b4146eb0 0x7fe2b41e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2b40cd2e0 0x7fe2b41e89e0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 --2- 172.21.15.107:0/1657315221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b406e160 0x7fe2b4143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 >> 172.21.15.107:0/1657315221 conn(0x7fe2b40b0c50 msgr2=0x7fe2b40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 shutdown_connections 2023-11-24T08:15:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.190+0000 7fe2bc67c640 1 -- 172.21.15.107:0/1657315221 wait complete. 2023-11-24T08:15:44.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:15:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.427+0000 7f8c2dd1d640 1 Processor -- start 2023-11-24T08:15:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.427+0000 7f8c2dd1d640 1 -- start start 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.428+0000 7f8c2dd1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 0x7f8c280c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c2dd1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2815e400 con 0x7f8c2806d600 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c2dd1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c280c5ed0 con 0x7f8c280c4220 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c2dd1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280ce880 con 0x7f8c2815d7f0 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c27fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8c2806d600 0x7f8c280cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37032/0 (socket says 172.21.15.107:37032) 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c27fff640 1 -- 172.21.15.107:0/555815719 learned_addr learned my addr 172.21.15.107:0/555815719 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c277fe640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 0x7f8c280c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.429+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088898929 0 0) 0x7f8c2815e400 con 0x7f8c2806d600 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c10003440 con 0x7f8c2806d600 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313621250 0 0) 0x7f8c10003440 con 0x7f8c2806d600 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 msgr2=0x7f8c280c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 0x7f8c280c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 >> v1:172.21.15.120:6789/0 conn(0x7f8c280c4220 legacy=0x7f8c280cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280c46e0 con 0x7f8c2806d600 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8c200019f0 con 0x7f8c2806d600 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 >> v1:172.21.15.107:6789/0 conn(0x7f8c2806d600 legacy=0x7f8c280cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c10003a90 0x7f8c10003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c10008850 0x7f8c1000acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 0x7f8c1000d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1000dc40 con 0x7f8c10003a90 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1000ddb0 con 0x7f8c10008850 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1000df70 con 0x7f8c2815d7f0 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.430+0000 7f8c277fe640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c10003a90 0x7f8c10003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.431+0000 7f8c277fe640 1 -- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 msgr2=0x7f8c1000d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.431+0000 7f8c277fe640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2815d7f0 0x7f8c1000d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.431+0000 7f8c277fe640 1 -- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c10008850 msgr2=0x7f8c1000acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.431+0000 7f8c277fe640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c10008850 0x7f8c1000acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.431+0000 7f8c277fe640 1 -- 172.21.15.107:0/555815719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10003440 con 0x7f8c10003a90 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.432+0000 7f8c277fe640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c10003a90 0x7f8c10003eb0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009ac0 tx=0x7f8c100036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67925cb1d5bf524a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.432+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1802c710 con 0x7f8c10003a90 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.432+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1802c870 con 0x7f8c10003a90 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.432+0000 7f8c267fc640 1 -- 172.21.15.107:0/555815719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1802cc00 con 0x7f8c10003a90 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/555815719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c10003a90 msgr2=0x7f8c10003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/555815719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c10003a90 0x7f8c10003eb0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009ac0 tx=0x7f8c100036f0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c27fff640 1 -- 172.21.15.107:0/555815719 reap_dead start 2023-11-24T08:15:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/555815719 shutdown_connections 2023-11-24T08:15:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/555815719 >> 172.21.15.107:0/555815719 conn(0x7f8c280b0c50 msgr2=0x7f8c280b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/555815719 shutdown_connections 2023-11-24T08:15:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.433+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/555815719 wait complete. 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.435+0000 7f8c2dd1d640 1 Processor -- start 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.435+0000 7f8c2dd1d640 1 -- start start 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c10003a90 0x7f8c28157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 0x7f8c280afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281f4450 con 0x7f8c280b0120 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280b05a0 con 0x7f8c281f45b0 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c2dd1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28070fe0 con 0x7f8c10003a90 2023-11-24T08:15:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59064/0 (socket says 172.21.15.107:59064) 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 -- 172.21.15.107:0/4037281787 learned_addr learned my addr 172.21.15.107:0/4037281787 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c26ffd640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 0x7f8c280afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c277fe640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c10003a90 0x7f8c28157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 -- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c10003a90 msgr2=0x7f8c28157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c10003a90 0x7f8c28157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 -- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 msgr2=0x7f8c280afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.436+0000 7f8c27fff640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 0x7f8c280afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c27fff640 1 -- 172.21.15.107:0/4037281787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280712c0 con 0x7f8c280b0120 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c26ffd640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 0x7f8c280afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c27fff640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f8c280aeb50 tx=0x7f8c2000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000a610 con 0x7f8c280b0120 2023-11-24T08:15:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2000a790 con 0x7f8c280b0120 2023-11-24T08:15:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000ab40 con 0x7f8c280b0120 2023-11-24T08:15:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c280715a0 con 0x7f8c280b0120 2023-11-24T08:15:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.437+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c281f4a30 con 0x7f8c280b0120 2023-11-24T08:15:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.439+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2804e8b0 con 0x7f8c280b0120 2023-11-24T08:15:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.439+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f8c2000aca0 con 0x7f8c280b0120 2023-11-24T08:15:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.440+0000 7f8c0ffff640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 0x7f8c1406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.441+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8c20087e70 con 0x7f8c280b0120 2023-11-24T08:15:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.441+0000 7f8c277fe640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 0x7f8c1406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.441+0000 7f8c277fe640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 0x7f8c1406e800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c18030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.445+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c20092080 con 0x7f8c280b0120 2023-11-24T08:15:44.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.635+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c280c4000 con 0x7f8c280b0120 2023-11-24T08:15:44.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.636+0000 7f8c0ffff640 1 -- 172.21.15.107:0/4037281787 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c2001a070 con 0x7f8c280b0120 2023-11-24T08:15:44.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 msgr2=0x7f8c1406e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 0x7f8c1406e800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c18030000 comp rx=0 tx=0).stop 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 msgr2=0x7f8c281f36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f8c280aeb50 tx=0x7f8c2000ccd0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 shutdown_connections 2023-11-24T08:15:44.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8c1406c340 0x7f8c1406e800 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c280b0120 0x7f8c281f36d0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c281f45b0 0x7f8c280afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 --2- 172.21.15.107:0/4037281787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c10003a90 0x7f8c28157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 >> 172.21.15.107:0/4037281787 conn(0x7f8c280b0c50 msgr2=0x7f8c28158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 shutdown_connections 2023-11-24T08:15:44.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.639+0000 7f8c2dd1d640 1 -- 172.21.15.107:0/4037281787 wait complete. 2023-11-24T08:15:44.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:15:44.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.872+0000 7f0324c83640 1 Processor -- start 2023-11-24T08:15:44.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.873+0000 7f0324c83640 1 -- start start 2023-11-24T08:15:44.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.874+0000 7f0324c83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03201605f0 0x7f0320160a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.874+0000 7f0324c83640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200ce410 con 0x7f03200d2000 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.874+0000 7f0324c83640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0320160f80 con 0x7f03200d1c20 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.874+0000 7f0324c83640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032006c270 con 0x7f03201605f0 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03201605f0 0x7f0320160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03201605f0 0x7f0320160a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59074/0 (socket says 172.21.15.107:59074) 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 -- 172.21.15.107:0/579781463 learned_addr learned my addr 172.21.15.107:0/579781463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:44.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 -- 172.21.15.107:0/579781463 >> v1:172.21.15.120:6789/0 conn(0x7f03200d1c20 legacy=0x7f032006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:44.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 -- 172.21.15.107:0/579781463 >> v1:172.21.15.107:6789/0 conn(0x7f03200d2000 legacy=0x7f03200cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.875+0000 7f031ed9d640 1 -- 172.21.15.107:0/579781463 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200d06f0 con 0x7f03201605f0 2023-11-24T08:15:44.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.876+0000 7f031ed9d640 1 --2- 172.21.15.107:0/579781463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03201605f0 0x7f0320160a40 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0318009870 tx=0x7f031802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20cb6f1394ea34 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f031d59a640 1 -- 172.21.15.107:0/579781463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318030860 con 0x7f03201605f0 2023-11-24T08:15:44.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f031d59a640 1 -- 172.21.15.107:0/579781463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03180309c0 con 0x7f03201605f0 2023-11-24T08:15:44.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f031d59a640 1 -- 172.21.15.107:0/579781463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318030d50 con 0x7f03201605f0 2023-11-24T08:15:44.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 -- 172.21.15.107:0/579781463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03201605f0 msgr2=0x7f0320160a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 --2- 172.21.15.107:0/579781463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03201605f0 0x7f0320160a40 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0318009870 tx=0x7f031802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 -- 172.21.15.107:0/579781463 shutdown_connections 2023-11-24T08:15:44.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 --2- 172.21.15.107:0/579781463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03201605f0 0x7f0320160a40 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 -- 172.21.15.107:0/579781463 >> 172.21.15.107:0/579781463 conn(0x7f032008cea0 msgr2=0x7f032009c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:44.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 -- 172.21.15.107:0/579781463 shutdown_connections 2023-11-24T08:15:44.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.877+0000 7f0324c83640 1 -- 172.21.15.107:0/579781463 wait complete. 2023-11-24T08:15:44.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.879+0000 7f0324c83640 1 Processor -- start 2023-11-24T08:15:44.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.879+0000 7f0324c83640 1 -- start start 2023-11-24T08:15:44.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03200d2000 0x7f03200cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 0x7f032014f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03200cc460 con 0x7f03200d1c20 2023-11-24T08:15:44.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03200ccf60 con 0x7f03201605f0 2023-11-24T08:15:44.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f0324c83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03200cc6c0 con 0x7f03200d2000 2023-11-24T08:15:44.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.880+0000 7f031e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59078/0 (socket says 172.21.15.107:59078) 2023-11-24T08:15:44.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 -- 172.21.15.107:0/2952352207 learned_addr learned my addr 172.21.15.107:0/2952352207 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031ed9d640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 0x7f032014f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 -- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03200d2000 msgr2=0x7f03200cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03200d2000 0x7f03200cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 -- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 msgr2=0x7f032014f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 0x7f032014f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 -- 172.21.15.107:0/2952352207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180094d0 con 0x7f03200d1c20 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031ed9d640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 0x7f032014f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:44.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.881+0000 7f031e59c640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f031000ba50 tx=0x7f031000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.882+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000cbe0 con 0x7f03200d1c20 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.882+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031000cd80 con 0x7f03200d1c20 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.882+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03200cc9a0 con 0x7f03200d1c20 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.882+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310014680 con 0x7f03200d1c20 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.882+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0320150340 con 0x7f03200d1c20 2023-11-24T08:15:44.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.883+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0320150560 con 0x7f03200d1c20 2023-11-24T08:15:44.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.884+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f03100147e0 con 0x7f03200d1c20 2023-11-24T08:15:44.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.885+0000 7f0316ffd640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 0x7f030006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:44.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.885+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f03100881e0 con 0x7f03200d1c20 2023-11-24T08:15:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.885+0000 7f031dd9b640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 0x7f030006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.886+0000 7f031dd9b640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 0x7f030006e7d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f60 tx=0x7f03080079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:44.889+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0310054d90 con 0x7f03200d1c20 2023-11-24T08:15:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.086+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f032004e8b0 con 0x7f03200d1c20 2023-11-24T08:15:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.087+0000 7f0316ffd640 1 -- 172.21.15.107:0/2952352207 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v135) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0310019070 con 0x7f03200d1c20 2023-11-24T08:15:45.089 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:15:45.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 msgr2=0x7f030006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 0x7f030006e7d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f60 tx=0x7f03080079b0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 msgr2=0x7f0320152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f031000ba50 tx=0x7f031000bf10 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 shutdown_connections 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f030006c310 0x7f030006e7d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03201605f0 0x7f032014f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03200d2000 0x7f03200cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 --2- 172.21.15.107:0/2952352207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03200d1c20 0x7f0320152ed0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.089+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 >> 172.21.15.107:0/2952352207 conn(0x7f032008cea0 msgr2=0x7f03200c9790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.090+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 shutdown_connections 2023-11-24T08:15:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.090+0000 7f0324c83640 1 -- 172.21.15.107:0/2952352207 wait complete. 2023-11-24T08:15:45.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:15:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.321+0000 7f901643d640 1 Processor -- start 2023-11-24T08:15:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.321+0000 7f901643d640 1 -- start start 2023-11-24T08:15:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.322+0000 7f901643d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.322+0000 7f901643d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f901006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f901643d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901015ec90 con 0x7f901006dc40 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f901643d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901006e0f0 con 0x7f9010072160 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f901643d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90100cd530 con 0x7f90100c49f0 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f900ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f900f7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f901006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f900ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:56866/0 (socket says 172.21.15.107:56866) 2023-11-24T08:15:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.323+0000 7f900ffff640 1 -- 172.21.15.107:0/2830996761 learned_addr learned my addr 172.21.15.107:0/2830996761 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.324+0000 7f900f7fe640 1 -- 172.21.15.107:0/2830996761 >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 msgr2=0x7f901015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.324+0000 7f900f7fe640 1 --2- 172.21.15.107:0/2830996761 >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.324+0000 7f900f7fe640 1 -- 172.21.15.107:0/2830996761 >> v1:172.21.15.120:6789/0 conn(0x7f901006dc40 legacy=0x7f90100cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.324+0000 7f900f7fe640 1 -- 172.21.15.107:0/2830996761 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90100c4ed0 con 0x7f90100c49f0 2023-11-24T08:15:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.324+0000 7f900ffff640 1 --2- 172.21.15.107:0/2830996761 >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:15:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.325+0000 7f900f7fe640 1 --2- 172.21.15.107:0/2830996761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f901006d700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80097a0 tx=0x7f8ff802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c64850fa55b1ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f900effd640 1 -- 172.21.15.107:0/2830996761 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff802d040 con 0x7f90100c49f0 2023-11-24T08:15:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f900effd640 1 -- 172.21.15.107:0/2830996761 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff8031810 con 0x7f90100c49f0 2023-11-24T08:15:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f900effd640 1 -- 172.21.15.107:0/2830996761 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8031ba0 con 0x7f90100c49f0 2023-11-24T08:15:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 -- 172.21.15.107:0/2830996761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 msgr2=0x7f901006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 --2- 172.21.15.107:0/2830996761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f901006d700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80097a0 tx=0x7f8ff802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 -- 172.21.15.107:0/2830996761 shutdown_connections 2023-11-24T08:15:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 --2- 172.21.15.107:0/2830996761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f901006d700 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 --2- 172.21.15.107:0/2830996761 >> v2:172.21.15.120:3300/0 conn(0x7f9010072160 0x7f901015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 -- 172.21.15.107:0/2830996761 >> 172.21.15.107:0/2830996761 conn(0x7f90100b0c50 msgr2=0x7f90100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 -- 172.21.15.107:0/2830996761 shutdown_connections 2023-11-24T08:15:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.326+0000 7f901643d640 1 -- 172.21.15.107:0/2830996761 wait complete. 2023-11-24T08:15:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.328+0000 7f901643d640 1 Processor -- start 2023-11-24T08:15:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.328+0000 7f901643d640 1 -- start start 2023-11-24T08:15:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9010072160 0x7f90101546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f9010158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9010155800 con 0x7f901006dc40 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90100c6610 con 0x7f9010072160 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f901643d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9010154be0 con 0x7f90100c49f0 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f900ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.329+0000 7f900ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59080/0 (socket says 172.21.15.107:59080) 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 -- 172.21.15.107:0/188038473 learned_addr learned my addr 172.21.15.107:0/188038473 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 -- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 msgr2=0x7f9010158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f9010158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 -- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9010072160 msgr2=0x7f90101546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9010072160 0x7f90101546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 -- 172.21.15.107:0/188038473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90000094d0 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900ffff640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f900000e9d0 tx=0x7f900000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900000ca60 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900000cbc0 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff8009400 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.330+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000013470 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.331+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9010154e60 con 0x7f901006dc40 2023-11-24T08:15:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.332+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901004e8b0 con 0x7f901006dc40 2023-11-24T08:15:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.333+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f90000135d0 con 0x7f901006dc40 2023-11-24T08:15:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.333+0000 7f900cff9640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 0x7f8ff006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.334+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9000088580 con 0x7f901006dc40 2023-11-24T08:15:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.334+0000 7f900f7fe640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 0x7f8ff006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.335+0000 7f900f7fe640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 0x7f8ff006e7d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8036000 tx=0x7f8ff8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.338+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90000550e0 con 0x7f901006dc40 2023-11-24T08:15:45.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.544+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f901015dab0 con 0x7f901006dc40 2023-11-24T08:15:45.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.545+0000 7f900cff9640 1 -- 172.21.15.107:0/188038473 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9000017080 con 0x7f901006dc40 2023-11-24T08:15:45.547 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 msgr2=0x7f8ff006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 0x7f8ff006e7d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8036000 tx=0x7f8ff8030000 comp rx=0 tx=0).stop 2023-11-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 msgr2=0x7f90100c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f900000e9d0 tx=0x7f900000ee90 comp rx=0 tx=0).stop 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 shutdown_connections 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8ff006c310 0x7f8ff006e7d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90100c49f0 0x7f9010158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9010072160 0x7f90101546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 --2- 172.21.15.107:0/188038473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006dc40 0x7f90100c60d0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 >> 172.21.15.107:0/188038473 conn(0x7f90100b0c50 msgr2=0x7f90100704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 shutdown_connections 2023-11-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:45.548+0000 7f901643d640 1 -- 172.21.15.107:0/188038473 wait complete. 2023-11-24T08:15:45.569 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:15:45.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:45.569 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:15:45.680 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:15:45.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:15:45.681 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:15:45.730 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:15:45.730 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:45.791 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:15:45.791 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:15:45.791 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:45.852 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:15:45.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:15:46.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.064+0000 7f9f2f3ff640 1 Processor -- start 2023-11-24T08:15:46.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.064+0000 7f9f2f3ff640 1 -- start start 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2f3ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9f2806db20 0x7f9f280ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2f3ff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2813f9e0 con 0x7f9f2806d6c0 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2f3ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2813fcc0 con 0x7f9f2813d120 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2f3ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2813ffa0 con 0x7f9f2806db20 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2ebfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9f2806db20 0x7f9f280ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2dbfc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9f2806d6c0 0x7f9f280cdbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37042/0 (socket says 172.21.15.107:37042) 2023-11-24T08:15:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2dbfc640 1 -- 172.21.15.107:0/2184850223 learned_addr learned my addr 172.21.15.107:0/2184850223 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2d3fb640 1 -- 172.21.15.107:0/2184850223 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028781160 0 0) 0x7f9f2813f9e0 con 0x7f9f2806d6c0 2023-11-24T08:15:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2d3fb640 1 -- 172.21.15.107:0/2184850223 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003520 con 0x7f9f2806d6c0 2023-11-24T08:15:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.066+0000 7f9f2ebfe640 1 -- 172.21.15.107:0/2184850223 >> v1:172.21.15.120:6789/0 conn(0x7f9f2813d120 legacy=0x7f9f2815af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.067+0000 7f9f2ebfe640 1 -- 172.21.15.107:0/2184850223 >> v1:172.21.15.107:6789/0 conn(0x7f9f2806d6c0 legacy=0x7f9f280cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.067+0000 7f9f2ebfe640 1 -- 172.21.15.107:0/2184850223 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f28140280 con 0x7f9f2806db20 2023-11-24T08:15:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2ebfe640 1 --2- 172.21.15.107:0/2184850223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806db20 0x7f9f280ce300 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009d90 tx=0x7f9f2002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d271d208136436aa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2d3fb640 1 -- 172.21.15.107:0/2184850223 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2002cbd0 con 0x7f9f2806db20 2023-11-24T08:15:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2d3fb640 1 -- 172.21.15.107:0/2184850223 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2002cd30 con 0x7f9f2806db20 2023-11-24T08:15:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2d3fb640 1 -- 172.21.15.107:0/2184850223 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2002cea0 con 0x7f9f2806db20 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/2184850223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806db20 msgr2=0x7f9f280ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.068+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/2184850223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806db20 0x7f9f280ce300 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009d90 tx=0x7f9f2002be20 comp rx=0 tx=0).stop 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.069+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/2184850223 shutdown_connections 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.069+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/2184850223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806db20 0x7f9f280ce300 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.069+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/2184850223 >> 172.21.15.107:0/2184850223 conn(0x7f9f280cb830 msgr2=0x7f9f280cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.069+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/2184850223 shutdown_connections 2023-11-24T08:15:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.069+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/2184850223 wait complete. 2023-11-24T08:15:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.070+0000 7f9f2f3ff640 1 Processor -- start 2023-11-24T08:15:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.071+0000 7f9f2f3ff640 1 -- start start 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.071+0000 7f9f2f3ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.071+0000 7f9f2f3ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2806db20 0x7f9f2806c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.071+0000 7f9f2f3ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2813d120 0x7f9f2806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2f3ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f28152a10 con 0x7f9f2806d6c0 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2f3ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f281f40c0 con 0x7f9f2806db20 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2f3ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f281f3540 con 0x7f9f2813d120 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59100/0 (socket says 172.21.15.107:59100) 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 -- 172.21.15.107:0/3024876625 learned_addr learned my addr 172.21.15.107:0/3024876625 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2dbfc640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2806db20 0x7f9f2806c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 -- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2813d120 msgr2=0x7f9f2806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2813d120 0x7f9f2806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 -- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2806db20 msgr2=0x7f9f2806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2806db20 0x7f9f2806c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 -- 172.21.15.107:0/3024876625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20009a70 con 0x7f9f2806d6c0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.072+0000 7f9f2e3fd640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400b3f0 tx=0x7f9f2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.073+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24011830 con 0x7f9f2806d6c0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.073+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24011990 con 0x7f9f2806d6c0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.073+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2815a9d0 con 0x7f9f2806d6c0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.073+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24011d40 con 0x7f9f2806d6c0 2023-11-24T08:15:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.073+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f281f3b00 con 0x7f9f2806d6c0 2023-11-24T08:15:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.074+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2804e840 con 0x7f9f2806d6c0 2023-11-24T08:15:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.075+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f9f2401a400 con 0x7f9f2806d6c0 2023-11-24T08:15:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.076+0000 7f9f1effd640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 0x7f9f0c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.076+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9f240880e0 con 0x7f9f2806d6c0 2023-11-24T08:15:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.077+0000 7f9f2dbfc640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 0x7f9f0c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.077+0000 7f9f2dbfc640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 0x7f9f0c06e7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000f60 tx=0x7f9f18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.081+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f24054c40 con 0x7f9f2806d6c0 2023-11-24T08:15:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.317+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f280cf0e0 con 0x7f9f2806d6c0 2023-11-24T08:15:46.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.318+0000 7f9f1effd640 1 -- 172.21.15.107:0/3024876625 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f9f24015020 con 0x7f9f2806d6c0 2023-11-24T08:15:46.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:46.320 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:15:46.321 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 msgr2=0x7f9f0c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 0x7f9f0c06e7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000f60 tx=0x7f9f18009040 comp rx=0 tx=0).stop 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 msgr2=0x7f9f281542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400b3f0 tx=0x7f9f2400b8b0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 shutdown_connections 2023-11-24T08:15:46.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f0c06c310 0x7f9f0c06e7d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2813d120 0x7f9f2806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2806db20 0x7f9f2806c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 --2- 172.21.15.107:0/3024876625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f2806d6c0 0x7f9f281542d0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 >> 172.21.15.107:0/3024876625 conn(0x7f9f280cb830 msgr2=0x7f9f280b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 shutdown_connections 2023-11-24T08:15:46.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.321+0000 7f9f2f3ff640 1 -- 172.21.15.107:0/3024876625 wait complete. 2023-11-24T08:15:46.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.552+0000 7f293a301640 1 Processor -- start 2023-11-24T08:15:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.552+0000 7f293a301640 1 -- start start 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293a301640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cf450 0x7f29340cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293a301640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29340cdfb0 0x7f29340ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293a301640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340cece0 con 0x7f293415c7f0 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293a301640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2934156010 con 0x7f29340cdfb0 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293a301640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293406ce30 con 0x7f29340cf450 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29340cdfb0 0x7f29340ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29340cdfb0 0x7f29340ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59110/0 (socket says 172.21.15.107:59110) 2023-11-24T08:15:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 -- 172.21.15.107:0/4107244826 learned_addr learned my addr 172.21.15.107:0/4107244826 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f292bfff640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cf450 0x7f29340cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 -- 172.21.15.107:0/4107244826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cf450 msgr2=0x7f29340cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cf450 0x7f29340cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.554+0000 7f293889e640 1 -- 172.21.15.107:0/4107244826 >> v1:172.21.15.120:6789/0 conn(0x7f293415c7f0 legacy=0x7f293406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.555+0000 7f293889e640 1 -- 172.21.15.107:0/4107244826 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293406d040 con 0x7f29340cdfb0 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f293889e640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cdfb0 0x7f29340ce440 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4062486c67797bdb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f29337fe640 1 -- 172.21.15.107:0/4107244826 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c02ca70 con 0x7f29340cdfb0 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f29337fe640 1 -- 172.21.15.107:0/4107244826 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c02cbd0 con 0x7f29340cdfb0 2023-11-24T08:15:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f29337fe640 1 -- 172.21.15.107:0/4107244826 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c0343f0 con 0x7f29340cdfb0 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f293a301640 1 -- 172.21.15.107:0/4107244826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cdfb0 msgr2=0x7f29340ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.556+0000 7f293a301640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cdfb0 0x7f29340ce440 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 -- 172.21.15.107:0/4107244826 shutdown_connections 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cdfb0 0x7f29340ce440 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 --2- 172.21.15.107:0/4107244826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cf450 0x7f29340cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 -- 172.21.15.107:0/4107244826 >> 172.21.15.107:0/4107244826 conn(0x7f293406a920 msgr2=0x7f293406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 -- 172.21.15.107:0/4107244826 shutdown_connections 2023-11-24T08:15:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.557+0000 7f293a301640 1 -- 172.21.15.107:0/4107244826 wait complete. 2023-11-24T08:15:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.558+0000 7f293a301640 1 Processor -- start 2023-11-24T08:15:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.558+0000 7f293a301640 1 -- start start 2023-11-24T08:15:46.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cdfb0 0x7f293414c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 0x7f293415df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340ac180 con 0x7f29340cf450 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293414cbd0 con 0x7f293415c7f0 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.559+0000 7f293a301640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29341507b0 con 0x7f29340cdfb0 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59126/0 (socket says 172.21.15.107:59126) 2023-11-24T08:15:46.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 -- 172.21.15.107:0/35109904 learned_addr learned my addr 172.21.15.107:0/35109904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 -- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cdfb0 msgr2=0x7f293414c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f293889e640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 0x7f293415df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f2933fff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cdfb0 0x7f293414c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cdfb0 0x7f293414c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 -- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 msgr2=0x7f293415df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 0x7f293415df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 -- 172.21.15.107:0/35109904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c009b30 con 0x7f29340cf450 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f293889e640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 0x7f293415df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.560+0000 7f292bfff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f291c00c7f0 tx=0x7f291c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.561+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c004220 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.561+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c0043a0 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.561+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c007860 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.561+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2934150a90 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.561+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2934151050 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.562+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29340ac2f0 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.563+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f291c007bd0 con 0x7f29340cf450 2023-11-24T08:15:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.564+0000 7f29317fa640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 0x7f291406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.564+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f291c088660 con 0x7f29340cf450 2023-11-24T08:15:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.564+0000 7f2933fff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 0x7f291406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.565+0000 7f2933fff640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 0x7f291406e800 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f292400a750 tx=0x7f2924008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.568+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f291c054de0 con 0x7f29340cf450 2023-11-24T08:15:46.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.759+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2934067470 con 0x7f29340cf450 2023-11-24T08:15:46.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.762+0000 7f29317fa640 1 -- 172.21.15.107:0/35109904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16687 (secure 0 0 0) 0x7f291c01e070 con 0x7f29340cf450 2023-11-24T08:15:46.764 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:46.764 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:44.960410+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.5","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:46.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.764+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 msgr2=0x7f291406e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.764+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 0x7f291406e800 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f292400a750 tx=0x7f2924008040 comp rx=0 tx=0).stop 2023-11-24T08:15:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.764+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 msgr2=0x7f293415a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.764+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f291c00c7f0 tx=0x7f291c00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 shutdown_connections 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f291406c340 0x7f291406e800 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293415c7f0 0x7f293415df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29340cf450 0x7f293415a5c0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 --2- 172.21.15.107:0/35109904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29340cdfb0 0x7f293414c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 >> 172.21.15.107:0/35109904 conn(0x7f293406a920 msgr2=0x7f29340b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 shutdown_connections 2023-11-24T08:15:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.765+0000 7f293a301640 1 -- 172.21.15.107:0/35109904 wait complete. 2023-11-24T08:15:46.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:14:26.165775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:14:27.450883+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:15:46.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:15:46.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.993+0000 7f03d82cf640 1 Processor -- start 2023-11-24T08:15:46.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.993+0000 7f03d82cf640 1 -- start start 2023-11-24T08:15:46.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03d015b440 0x7f03d00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f03d00cbf20 0x7f03d00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00c57e0 0x7f03d00c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:46.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00cc3d0 con 0x7f03d015b440 2023-11-24T08:15:46.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d015b820 con 0x7f03d00cbf20 2023-11-24T08:15:46.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d82cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015eea0 con 0x7f03d00c57e0 2023-11-24T08:15:46.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d606b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03d015b440 0x7f03d00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d606b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03d015b440 0x7f03d00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59136/0 (socket says 172.21.15.107:59136) 2023-11-24T08:15:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d606b640 1 -- 172.21.15.107:0/3909360398 learned_addr learned my addr 172.21.15.107:0/3909360398 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d586a640 1 --2- 172.21.15.107:0/3909360398 >> v2:172.21.15.120:3300/0 conn(0x7f03d00cbf20 0x7f03d00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.995+0000 7f03d686c640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00c57e0 0x7f03d00c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.996+0000 7f03d606b640 1 -- 172.21.15.107:0/3909360398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00c57e0 msgr2=0x7f03d00c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.996+0000 7f03d606b640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00c57e0 0x7f03d00c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.996+0000 7f03d606b640 1 -- 172.21.15.107:0/3909360398 >> v2:172.21.15.120:3300/0 conn(0x7f03d00cbf20 msgr2=0x7f03d00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.996+0000 7f03d606b640 1 --2- 172.21.15.107:0/3909360398 >> v2:172.21.15.120:3300/0 conn(0x7f03d00cbf20 0x7f03d00c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.996+0000 7f03d606b640 1 -- 172.21.15.107:0/3909360398 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00abf30 con 0x7f03d015b440 2023-11-24T08:15:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.997+0000 7f03d606b640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d015b440 0x7f03d00cb9e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f03d006e890 tx=0x7f03cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c840ed009c0f218b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.997+0000 7f03d5069640 1 -- 172.21.15.107:0/3909360398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0317e0 con 0x7f03d015b440 2023-11-24T08:15:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.997+0000 7f03d5069640 1 -- 172.21.15.107:0/3909360398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc031940 con 0x7f03d015b440 2023-11-24T08:15:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.997+0000 7f03d5069640 1 -- 172.21.15.107:0/3909360398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc031cd0 con 0x7f03d015b440 2023-11-24T08:15:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 -- 172.21.15.107:0/3909360398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d015b440 msgr2=0x7f03d00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d015b440 0x7f03d00cb9e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f03d006e890 tx=0x7f03cc02b880 comp rx=0 tx=0).stop 2023-11-24T08:15:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 -- 172.21.15.107:0/3909360398 shutdown_connections 2023-11-24T08:15:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00c57e0 0x7f03d00c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 --2- 172.21.15.107:0/3909360398 >> v2:172.21.15.120:3300/0 conn(0x7f03d00cbf20 0x7f03d00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 --2- 172.21.15.107:0/3909360398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d015b440 0x7f03d00cb9e0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 -- 172.21.15.107:0/3909360398 >> 172.21.15.107:0/3909360398 conn(0x7f03d00b0c50 msgr2=0x7f03d00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 -- 172.21.15.107:0/3909360398 shutdown_connections 2023-11-24T08:15:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:46.998+0000 7f03d82cf640 1 -- 172.21.15.107:0/3909360398 wait complete. 2023-11-24T08:15:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.000+0000 7f03d82cf640 1 Processor -- start 2023-11-24T08:15:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.000+0000 7f03d82cf640 1 -- start start 2023-11-24T08:15:47.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00cbf20 0x7f03d009a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 0x7f03d01559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00c65e0 con 0x7f03d00c57e0 2023-11-24T08:15:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d009ae60 con 0x7f03d015b440 2023-11-24T08:15:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d82cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d009b140 con 0x7f03d00cbf20 2023-11-24T08:15:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d606b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d606b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59146/0 (socket says 172.21.15.107:59146) 2023-11-24T08:15:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.001+0000 7f03d606b640 1 -- 172.21.15.107:0/37813279 learned_addr learned my addr 172.21.15.107:0/37813279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d586a640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00cbf20 0x7f03d009a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d686c640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 0x7f03d01559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 -- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00cbf20 msgr2=0x7f03d009a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00cbf20 0x7f03d009a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 -- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 msgr2=0x7f03d01559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 0x7f03d01559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 -- 172.21.15.107:0/37813279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0094d0 con 0x7f03d00c57e0 2023-11-24T08:15:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d686c640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 0x7f03d01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d606b640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f03cc000c00 tx=0x7f03cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc032880 con 0x7f03d00c57e0 2023-11-24T08:15:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc032a20 con 0x7f03d00c57e0 2023-11-24T08:15:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc032dd0 con 0x7f03d00c57e0 2023-11-24T08:15:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d009b420 con 0x7f03d00c57e0 2023-11-24T08:15:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.002+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d00c8f10 con 0x7f03d00c57e0 2023-11-24T08:15:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.004+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d009b280 con 0x7f03d00c57e0 2023-11-24T08:15:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.004+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f03cc04e450 con 0x7f03d00c57e0 2023-11-24T08:15:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.005+0000 7f03c6ffd640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 0x7f03b406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.005+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f03cc0aa400 con 0x7f03d00c57e0 2023-11-24T08:15:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.006+0000 7f03d586a640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 0x7f03b406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.006+0000 7f03d586a640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 0x7f03b406e7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f03d00b1410 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.010+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03d009b280 con 0x7f03d00c57e0 2023-11-24T08:15:47.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.200+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03d009b280 con 0x7f03d00c57e0 2023-11-24T08:15:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.201+0000 7f03c6ffd640 1 -- 172.21.15.107:0/37813279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03d009b280 con 0x7f03d00c57e0 2023-11-24T08:15:47.203 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:15:47.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 msgr2=0x7f03b406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 0x7f03b406e7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f03d00b1410 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 msgr2=0x7f03d0150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f03cc000c00 tx=0x7f03cc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 shutdown_connections 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f03b406c310 0x7f03b406e7d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03d015b440 0x7f03d01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03d00cbf20 0x7f03d009a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 --2- 172.21.15.107:0/37813279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d00c57e0 0x7f03d0150420 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 >> 172.21.15.107:0/37813279 conn(0x7f03d00b0c50 msgr2=0x7f03d006c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 shutdown_connections 2023-11-24T08:15:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.204+0000 7f03d82cf640 1 -- 172.21.15.107:0/37813279 wait complete. 2023-11-24T08:15:47.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:15:47.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.440+0000 7fc16f413640 1 Processor -- start 2023-11-24T08:15:47.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.440+0000 7fc16f413640 1 -- start start 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16f413640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc16806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16f413640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc16806e580 0x7fc1680b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16f413640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16806d5c0 con 0x7fc1680b1080 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16f413640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680b1530 con 0x7fc16806e580 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16f413640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16815b640 con 0x7fc16815b260 2023-11-24T08:15:47.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16dc10640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc16806e580 0x7fc1680b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16dc10640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc16806e580 0x7fc1680b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59148/0 (socket says 172.21.15.107:59148) 2023-11-24T08:15:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16dc10640 1 -- 172.21.15.107:0/393103053 learned_addr learned my addr 172.21.15.107:0/393103053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:47.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.442+0000 7fc16e411640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc16806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.443+0000 7fc16dc10640 1 -- 172.21.15.107:0/393103053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 msgr2=0x7fc16806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.443+0000 7fc16dc10640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc16806e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.443+0000 7fc16dc10640 1 -- 172.21.15.107:0/393103053 >> v1:172.21.15.120:6789/0 conn(0x7fc1680b1080 legacy=0x7fc1680ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.443+0000 7fc16dc10640 1 -- 172.21.15.107:0/393103053 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16815ed20 con 0x7fc16806e580 2023-11-24T08:15:47.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.444+0000 7fc16dc10640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc16806e580 0x7fc1680b0b40 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc158009870 tx=0x7fc15802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce18f419d8234169 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.444+0000 7fc16d40f640 1 -- 172.21.15.107:0/393103053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158031740 con 0x7fc16806e580 2023-11-24T08:15:47.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.444+0000 7fc16d40f640 1 -- 172.21.15.107:0/393103053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1580318a0 con 0x7fc16806e580 2023-11-24T08:15:47.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.444+0000 7fc16d40f640 1 -- 172.21.15.107:0/393103053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158031c30 con 0x7fc16806e580 2023-11-24T08:15:47.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 -- 172.21.15.107:0/393103053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc16806e580 msgr2=0x7fc1680b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc16806e580 0x7fc1680b0b40 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc158009870 tx=0x7fc15802b7e0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 -- 172.21.15.107:0/393103053 shutdown_connections 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc16806e580 0x7fc1680b0b40 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 --2- 172.21.15.107:0/393103053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc16806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 -- 172.21.15.107:0/393103053 >> 172.21.15.107:0/393103053 conn(0x7fc1680cbb40 msgr2=0x7fc1680cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 -- 172.21.15.107:0/393103053 shutdown_connections 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.445+0000 7fc16f413640 1 -- 172.21.15.107:0/393103053 wait complete. 2023-11-24T08:15:47.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.447+0000 7fc16f413640 1 Processor -- start 2023-11-24T08:15:47.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.447+0000 7fc16f413640 1 -- start start 2023-11-24T08:15:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc16806e580 0x7fc1680c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc168154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681f7790 con 0x7fc1680b1080 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1680c51a0 con 0x7fc16806e580 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16f413640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16815b820 con 0x7fc16815b260 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59162/0 (socket says 172.21.15.107:59162) 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 -- 172.21.15.107:0/1394893266 learned_addr learned my addr 172.21.15.107:0/1394893266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 -- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 msgr2=0x7fc168154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc168154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 -- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc16806e580 msgr2=0x7fc1680c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc16806e580 0x7fc1680c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.448+0000 7fc16dc10640 1 -- 172.21.15.107:0/1394893266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1580094d0 con 0x7fc1680b1080 2023-11-24T08:15:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc16dc10640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc158000c00 tx=0x7fc1580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158033c70 con 0x7fc1680b1080 2023-11-24T08:15:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158033dd0 con 0x7fc1680b1080 2023-11-24T08:15:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15803c640 con 0x7fc1680b1080 2023-11-24T08:15:47.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16815bb00 con 0x7fc1680b1080 2023-11-24T08:15:47.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.449+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16815c0c0 con 0x7fc1680b1080 2023-11-24T08:15:47.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.450+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16815c2b0 con 0x7fc1680b1080 2023-11-24T08:15:47.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.451+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fc15803a040 con 0x7fc1680b1080 2023-11-24T08:15:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.452+0000 7fc15effd640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 0x7fc14c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.452+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc1580a9ca0 con 0x7fc1680b1080 2023-11-24T08:15:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.452+0000 7fc16e411640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 0x7fc14c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.453+0000 7fc16e411640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 0x7fc14c06e7d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc1640098f0 tx=0x7fc164008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.457+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc158076800 con 0x7fc1680b1080 2023-11-24T08:15:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.646+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc16804e840 con 0x7fc1680b1080 2023-11-24T08:15:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.646+0000 7fc15effd640 1 -- 172.21.15.107:0/1394893266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v135) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc158038020 con 0x7fc1680b1080 2023-11-24T08:15:47.648 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 msgr2=0x7fc14c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 0x7fc14c06e7d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc1640098f0 tx=0x7fc164008040 comp rx=0 tx=0).stop 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 msgr2=0x7fc1681f7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc158000c00 tx=0x7fc1580046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 shutdown_connections 2023-11-24T08:15:47.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc14c06c310 0x7fc14c06e7d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc16815b260 0x7fc168154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1680b1080 0x7fc1681f7250 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 --2- 172.21.15.107:0/1394893266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc16806e580 0x7fc1680c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.649+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 >> 172.21.15.107:0/1394893266 conn(0x7fc1680cbb40 msgr2=0x7fc1680ac070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.650+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 shutdown_connections 2023-11-24T08:15:47.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.650+0000 7fc16f413640 1 -- 172.21.15.107:0/1394893266 wait complete. 2023-11-24T08:15:47.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:15:47.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.873+0000 7f9169101640 1 Processor -- start 2023-11-24T08:15:47.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.874+0000 7f9169101640 1 -- start start 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.875+0000 7f9169101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f916415b440 0x7f91640cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.875+0000 7f9169101640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91640c59b0 0x7f916406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.875+0000 7f9169101640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640cc3a0 con 0x7f91640cbf20 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.875+0000 7f9169101640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916415b820 con 0x7f91640c59b0 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.875+0000 7f9169101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916415eea0 con 0x7f916415b440 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91640c59b0 0x7f916406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f91640cbf20 0x7f91640c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37058/0 (socket says 172.21.15.107:37058) 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916259c640 1 -- 172.21.15.107:0/2506508947 learned_addr learned my addr 172.21.15.107:0/2506508947 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f9162d9d640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f916415b440 0x7f91640cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916359e640 1 -- 172.21.15.107:0/2506508947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f916415b440 msgr2=0x7f91640cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916359e640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f916415b440 0x7f91640cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916359e640 1 -- 172.21.15.107:0/2506508947 >> v1:172.21.15.107:6789/0 conn(0x7f91640cbf20 legacy=0x7f91640c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.876+0000 7f916359e640 1 -- 172.21.15.107:0/2506508947 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640abf30 con 0x7f91640c59b0 2023-11-24T08:15:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.877+0000 7f916359e640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f916406c5f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097a0 tx=0x7f915c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b3c5cde43313a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9161d9b640 1 -- 172.21.15.107:0/2506508947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c031780 con 0x7f91640c59b0 2023-11-24T08:15:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9161d9b640 1 -- 172.21.15.107:0/2506508947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c0318e0 con 0x7f91640c59b0 2023-11-24T08:15:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9161d9b640 1 -- 172.21.15.107:0/2506508947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c031c70 con 0x7f91640c59b0 2023-11-24T08:15:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 -- 172.21.15.107:0/2506508947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 msgr2=0x7f916406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f916406c5f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097a0 tx=0x7f915c02b820 comp rx=0 tx=0).stop 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 -- 172.21.15.107:0/2506508947 shutdown_connections 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f916406c5f0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 --2- 172.21.15.107:0/2506508947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f916415b440 0x7f91640cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 -- 172.21.15.107:0/2506508947 >> 172.21.15.107:0/2506508947 conn(0x7f91640b0c50 msgr2=0x7f91640b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 -- 172.21.15.107:0/2506508947 shutdown_connections 2023-11-24T08:15:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.878+0000 7f9169101640 1 -- 172.21.15.107:0/2506508947 wait complete. 2023-11-24T08:15:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.880+0000 7f9169101640 1 Processor -- start 2023-11-24T08:15:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.880+0000 7f9169101640 1 -- start start 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91640cbf20 0x7f91641f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916415b440 0x7f9164070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91641f3fd0 con 0x7f91640c59b0 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91641571c0 con 0x7f916415b440 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9169101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164156310 con 0x7f91640cbf20 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9162d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9162d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59184/0 (socket says 172.21.15.107:59184) 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.881+0000 7f9162d9d640 1 -- 172.21.15.107:0/2514724657 learned_addr learned my addr 172.21.15.107:0/2514724657 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 -- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91640cbf20 msgr2=0x7f91641f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91640cbf20 0x7f91641f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 -- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916415b440 msgr2=0x7f9164070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916415b440 0x7f9164070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 -- 172.21.15.107:0/2514724657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c009400 con 0x7f91640c59b0 2023-11-24T08:15:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9162d9d640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f915400d5f0 tx=0x7f915400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91540125f0 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154012750 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91641565b0 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.882+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154012b00 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.883+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9164156b70 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.884+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916404e8b0 con 0x7f91640c59b0 2023-11-24T08:15:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.885+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f9154012c60 con 0x7f91640c59b0 2023-11-24T08:15:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.886+0000 7f915b7fe640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 0x7f914406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.886+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9154088b70 con 0x7f91640c59b0 2023-11-24T08:15:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.886+0000 7f916259c640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 0x7f914406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.887+0000 7f916259c640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 0x7f914406e7d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f91640ce690 tx=0x7f914c001660 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:47.890+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91540552f0 con 0x7f91640c59b0 2023-11-24T08:15:48.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.084+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f916406dc80 con 0x7f91640c59b0 2023-11-24T08:15:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.085+0000 7f915b7fe640 1 -- 172.21.15.107:0/2514724657 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v135) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9154058910 con 0x7f91640c59b0 2023-11-24T08:15:48.087 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 msgr2=0x7f914406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 0x7f914406e7d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f91640ce690 tx=0x7f914c001660 comp rx=0 tx=0).stop 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 msgr2=0x7f91640cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f915400d5f0 tx=0x7f915400dab0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 shutdown_connections 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406c310 0x7f914406e7d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916415b440 0x7f9164070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91640cbf20 0x7f91641f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 --2- 172.21.15.107:0/2514724657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91640c59b0 0x7f91640cd8f0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 >> 172.21.15.107:0/2514724657 conn(0x7f91640b0c50 msgr2=0x7f91640ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 shutdown_connections 2023-11-24T08:15:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.088+0000 7f9169101640 1 -- 172.21.15.107:0/2514724657 wait complete. 2023-11-24T08:15:48.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:15:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.322+0000 7f10992d3640 1 Processor -- start 2023-11-24T08:15:48.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.322+0000 7f10992d3640 1 -- start start 2023-11-24T08:15:48.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.323+0000 7f10992d3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10940b2250 0x7f109415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.323+0000 7f10992d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10940ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.323+0000 7f10992d3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109415ab80 con 0x7f10940cd470 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.323+0000 7f10992d3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f109415a990 con 0x7f10940b2250 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.323+0000 7f10992d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f109415e680 con 0x7f10940b2630 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f10937fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10940cd470 0x7f109415b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39170/0 (socket says 172.21.15.107:39170) 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f10937fe640 1 -- 172.21.15.107:0/2029377241 learned_addr learned my addr 172.21.15.107:0/2029377241 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1092ffd640 1 --2- 172.21.15.107:0/2029377241 >> v2:172.21.15.107:3300/0 conn(0x7f10940b2250 0x7f109415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1091ffb640 1 -- 172.21.15.107:0/2029377241 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765336815 0 0) 0x7f109415ab80 con 0x7f10940cd470 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f10927fc640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10940ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1092ffd640 1 -- 172.21.15.107:0/2029377241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 msgr2=0x7f10940ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1092ffd640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10940ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1092ffd640 1 -- 172.21.15.107:0/2029377241 >> v1:172.21.15.107:6789/0 conn(0x7f10940cd470 legacy=0x7f109415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.324+0000 7f1092ffd640 1 -- 172.21.15.107:0/2029377241 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109415e960 con 0x7f10940b2250 2023-11-24T08:15:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.325+0000 7f1092ffd640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f109415df80 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f10840097a0 tx=0x7f108402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3213822478e0a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f1091ffb640 1 -- 172.21.15.107:0/2029377241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084031830 con 0x7f10940b2250 2023-11-24T08:15:48.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f1091ffb640 1 -- 172.21.15.107:0/2029377241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084031990 con 0x7f10940b2250 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f1091ffb640 1 -- 172.21.15.107:0/2029377241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084031d20 con 0x7f10940b2250 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 -- 172.21.15.107:0/2029377241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 msgr2=0x7f109415df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f109415df80 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f10840097a0 tx=0x7f108402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 -- 172.21.15.107:0/2029377241 shutdown_connections 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10940ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 --2- 172.21.15.107:0/2029377241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f109415df80 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 -- 172.21.15.107:0/2029377241 >> 172.21.15.107:0/2029377241 conn(0x7f109406a920 msgr2=0x7f109406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 -- 172.21.15.107:0/2029377241 shutdown_connections 2023-11-24T08:15:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.326+0000 7f10992d3640 1 -- 172.21.15.107:0/2029377241 wait complete. 2023-11-24T08:15:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.328+0000 7f10992d3640 1 Processor -- start 2023-11-24T08:15:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.328+0000 7f10992d3640 1 -- start start 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10941f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 0x7f1094071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10940b1170 con 0x7f10940b2250 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10940af690 con 0x7f10940cd470 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f10992d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1094154af0 con 0x7f10940b2630 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f1092ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f1092ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51232/0 (socket says 172.21.15.107:51232) 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.329+0000 7f1092ffd640 1 -- 172.21.15.107:0/1924056518 learned_addr learned my addr 172.21.15.107:0/1924056518 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f10937fe640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 0x7f1094071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f10927fc640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10941f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 -- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 msgr2=0x7f10941f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10941f2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 -- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 msgr2=0x7f1094071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 0x7f1094071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 -- 172.21.15.107:0/1924056518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084009400 con 0x7f10940b2250 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f10937fe640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 0x7f1094071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f1092ffd640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1084000c00 tx=0x7f10840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084004230 con 0x7f10940b2250 2023-11-24T08:15:48.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.330+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084004390 con 0x7f10940b2250 2023-11-24T08:15:48.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.331+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108403a920 con 0x7f10940b2250 2023-11-24T08:15:48.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.331+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10940aecf0 con 0x7f10940b2250 2023-11-24T08:15:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.331+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10940af2b0 con 0x7f10940b2250 2023-11-24T08:15:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.332+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1094067470 con 0x7f10940b2250 2023-11-24T08:15:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.333+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f108403aa80 con 0x7f10940b2250 2023-11-24T08:15:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.334+0000 7f107b7fe640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 0x7f108006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.334+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f10840aa9d0 con 0x7f10940b2250 2023-11-24T08:15:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.334+0000 7f10927fc640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 0x7f108006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.334+0000 7f10927fc640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 0x7f108006e7d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1088006fd0 tx=0x7f1088008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.338+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1084077530 con 0x7f10940b2250 2023-11-24T08:15:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.565+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10940cea00 con 0x7f10940b2250 2023-11-24T08:15:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.566+0000 7f107b7fe640 1 -- 172.21.15.107:0/1924056518 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f1084035070 con 0x7f10940b2250 2023-11-24T08:15:48.568 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:48.568 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:15:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-11-24T08:15:48.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 msgr2=0x7f108006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 0x7f108006e7d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1088006fd0 tx=0x7f1088008040 comp rx=0 tx=0).stop 2023-11-24T08:15:48.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 msgr2=0x7f10940afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1084000c00 tx=0x7f10840046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 shutdown_connections 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f108006c310 0x7f108006e7d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10940cd470 0x7f1094071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10940b2630 0x7f10941f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 --2- 172.21.15.107:0/1924056518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940b2250 0x7f10940afab0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 >> 172.21.15.107:0/1924056518 conn(0x7f109406a920 msgr2=0x7f109415ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 shutdown_connections 2023-11-24T08:15:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.569+0000 7f10992d3640 1 -- 172.21.15.107:0/1924056518 wait complete. 2023-11-24T08:15:48.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.786+0000 7f294229b640 1 Processor -- start 2023-11-24T08:15:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.786+0000 7f294229b640 1 -- start start 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f294229b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f293c15c7f0 0x7f293c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f294229b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0ce6c0 0x7f293c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f294229b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c0cece0 con 0x7f293c0cf450 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f294229b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c0cfe40 con 0x7f293c15c7f0 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f294229b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c156010 con 0x7f293c0ce6c0 2023-11-24T08:15:48.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f293bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f293c15c7f0 0x7f293c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f293bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f293c15c7f0 0x7f293c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51242/0 (socket says 172.21.15.107:51242) 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.788+0000 7f293bfff640 1 -- 172.21.15.107:0/1583945843 learned_addr learned my addr 172.21.15.107:0/1583945843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.789+0000 7f293bfff640 1 -- 172.21.15.107:0/1583945843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0ce6c0 msgr2=0x7f293c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.789+0000 7f293bfff640 1 --2- 172.21.15.107:0/1583945843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0ce6c0 0x7f293c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.789+0000 7f293bfff640 1 -- 172.21.15.107:0/1583945843 >> v1:172.21.15.120:6789/0 conn(0x7f293c0cf450 legacy=0x7f293c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.789+0000 7f293bfff640 1 -- 172.21.15.107:0/1583945843 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c06d040 con 0x7f293c15c7f0 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.790+0000 7f293bfff640 1 --2- 172.21.15.107:0/1583945843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c06db20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e240dce117cab641 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.790+0000 7f293affd640 1 -- 172.21.15.107:0/1583945843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c031640 con 0x7f293c15c7f0 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.790+0000 7f293affd640 1 -- 172.21.15.107:0/1583945843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c0317a0 con 0x7f293c15c7f0 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.790+0000 7f293affd640 1 -- 172.21.15.107:0/1583945843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c031b30 con 0x7f293c15c7f0 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 -- 172.21.15.107:0/1583945843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 msgr2=0x7f293c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 --2- 172.21.15.107:0/1583945843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c06db20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 -- 172.21.15.107:0/1583945843 shutdown_connections 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 --2- 172.21.15.107:0/1583945843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0ce6c0 0x7f293c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 --2- 172.21.15.107:0/1583945843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c06db20 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 -- 172.21.15.107:0/1583945843 >> 172.21.15.107:0/1583945843 conn(0x7f293c06a920 msgr2=0x7f293c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 -- 172.21.15.107:0/1583945843 shutdown_connections 2023-11-24T08:15:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.791+0000 7f294229b640 1 -- 172.21.15.107:0/1583945843 wait complete. 2023-11-24T08:15:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.793+0000 7f294229b640 1 Processor -- start 2023-11-24T08:15:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.793+0000 7f294229b640 1 -- start start 2023-11-24T08:15:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c0ce6c0 0x7f293c150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0cf450 0x7f293c156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0ad8c0 con 0x7f293c15c7f0 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0b2850 con 0x7f293c0ce6c0 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.794+0000 7f294229b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0b2b30 con 0x7f293c0cf450 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51254/0 (socket says 172.21.15.107:51254) 2023-11-24T08:15:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 -- 172.21.15.107:0/1563024443 learned_addr learned my addr 172.21.15.107:0/1563024443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f293bfff640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c0ce6c0 0x7f293c150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f293b7fe640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0cf450 0x7f293c156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 -- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0cf450 msgr2=0x7f293c156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0cf450 0x7f293c156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 -- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c0ce6c0 msgr2=0x7f293c150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c0ce6c0 0x7f293c150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.795+0000 7f2940838640 1 -- 172.21.15.107:0/1563024443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0094d0 con 0x7f293c15c7f0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f2940838640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f293400b9e0 tx=0x7f293400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400caa0 con 0x7f293c15c7f0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400cc20 con 0x7f293c15c7f0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934014440 con 0x7f293c15c7f0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c0b2e10 con 0x7f293c15c7f0 2023-11-24T08:15:48.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.796+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c151190 con 0x7f293c15c7f0 2023-11-24T08:15:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.797+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c0b3330 con 0x7f293c15c7f0 2023-11-24T08:15:48.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.798+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f293401d3b0 con 0x7f293c15c7f0 2023-11-24T08:15:48.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.799+0000 7f2938ff9640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 0x7f292806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:48.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.799+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2934088210 con 0x7f293c15c7f0 2023-11-24T08:15:48.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.800+0000 7f293bfff640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 0x7f292806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:48.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.800+0000 7f293bfff640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 0x7f292806e7d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f292c036000 tx=0x7f292c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.803+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f293c0b3330 con 0x7f293c15c7f0 2023-11-24T08:15:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.995+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f293c0b3330 con 0x7f293c15c7f0 2023-11-24T08:15:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:48.999+0000 7f2938ff9640 1 -- 172.21.15.107:0/1563024443 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16687 (secure 0 0 0) 0x7f293c0b3330 con 0x7f293c15c7f0 2023-11-24T08:15:49.001 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:49.001 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:44.960410+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.5","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 msgr2=0x7f292806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 0x7f292806e7d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f292c036000 tx=0x7f292c03c040 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 msgr2=0x7f293c15eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f293400b9e0 tx=0x7f293400bea0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 shutdown_connections 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.001+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f292806c310 0x7f292806e7d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f293c15c7f0 0x7f293c15eec0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0cf450 0x7f293c156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 --2- 172.21.15.107:0/1563024443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c0ce6c0 0x7f293c150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 >> 172.21.15.107:0/1563024443 conn(0x7f293c06a920 msgr2=0x7f293c06ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 shutdown_connections 2023-11-24T08:15:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.002+0000 7f294229b640 1 -- 172.21.15.107:0/1563024443 wait complete. 2023-11-24T08:15:49.024 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:15:49.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:15:49.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.219+0000 7f98dcee7640 1 Processor -- start 2023-11-24T08:15:49.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.220+0000 7f98dcee7640 1 -- start start 2023-11-24T08:15:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.221+0000 7f98dcee7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f98d80ce290 0x7f98d80ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.221+0000 7f98dcee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 0x7f98d80ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.221+0000 7f98dcee7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d815bcb0 con 0x7f98d806c4f0 2023-11-24T08:15:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.221+0000 7f98dcee7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d80ce670 con 0x7f98d80ce290 2023-11-24T08:15:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.221+0000 7f98dcee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d815bf90 con 0x7f98d80ac5d0 2023-11-24T08:15:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98cffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98d806c4f0 0x7f98d80abec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39178/0 (socket says 172.21.15.107:39178) 2023-11-24T08:15:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98cffff640 1 -- 172.21.15.107:0/2541612931 learned_addr learned my addr 172.21.15.107:0/2541612931 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98d659c640 1 --2- 172.21.15.107:0/2541612931 >> v2:172.21.15.120:3300/0 conn(0x7f98d80ce290 0x7f98d80ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98d6d9d640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 0x7f98d80ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562445342 0 0) 0x7f98d815bcb0 con 0x7f98d806c4f0 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98d806c4f0 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.222+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3320126448 0 0) 0x7f98c4003520 con 0x7f98d806c4f0 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 msgr2=0x7f98d80ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 0x7f98d80ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 >> v2:172.21.15.120:3300/0 conn(0x7f98d80ce290 msgr2=0x7f98d80ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 --2- 172.21.15.107:0/2541612931 >> v2:172.21.15.120:3300/0 conn(0x7f98d80ce290 0x7f98d80ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d815c270 con 0x7f98d806c4f0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f98c0001670 con 0x7f98d806c4f0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 >> v1:172.21.15.107:6789/0 conn(0x7f98d806c4f0 legacy=0x7f98d80abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c40047c0 0x7f98c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 0x7f98c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80ce670 con 0x7f98c4003970 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d815bf90 con 0x7f98c40047c0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c400b980 con 0x7f98d80ac5d0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98cffff640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c40047c0 0x7f98c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 -- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 msgr2=0x7f98c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d80ac5d0 0x7f98c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 -- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c40047c0 msgr2=0x7f98c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c40047c0 0x7f98c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98d659c640 1 -- 172.21.15.107:0/2541612931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4003520 con 0x7f98c4003970 2023-11-24T08:15:49.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.223+0000 7f98cffff640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c40047c0 0x7f98c4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:49.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98d659c640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98c4003d90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009af0 tx=0x7f98c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e62cd74b7b3582d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c802c710 con 0x7f98c4003970 2023-11-24T08:15:49.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c802c870 con 0x7f98c4003970 2023-11-24T08:15:49.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98d5d9b640 1 -- 172.21.15.107:0/2541612931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c802cc00 con 0x7f98c4003970 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98dcee7640 1 -- 172.21.15.107:0/2541612931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 msgr2=0x7f98c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98dcee7640 1 --2- 172.21.15.107:0/2541612931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98c4003d90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009af0 tx=0x7f98c40057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98d6d9d640 1 -- 172.21.15.107:0/2541612931 reap_dead start 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98dcee7640 1 -- 172.21.15.107:0/2541612931 shutdown_connections 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.225+0000 7f98dcee7640 1 -- 172.21.15.107:0/2541612931 >> 172.21.15.107:0/2541612931 conn(0x7f98d80b0be0 msgr2=0x7f98d80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.226+0000 7f98dcee7640 1 -- 172.21.15.107:0/2541612931 shutdown_connections 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.226+0000 7f98dcee7640 1 -- 172.21.15.107:0/2541612931 wait complete. 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.227+0000 7f98dcee7640 1 Processor -- start 2023-11-24T08:15:49.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.227+0000 7f98dcee7640 1 -- start start 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8143600 0x7f98d8142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8149150 0x7f98d8071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8145830 con 0x7f98c4003970 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8145b10 con 0x7f98d8143600 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.228+0000 7f98dcee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8145df0 con 0x7f98d8149150 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51278/0 (socket says 172.21.15.107:51278) 2023-11-24T08:15:49.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 -- 172.21.15.107:0/240748191 learned_addr learned my addr 172.21.15.107:0/240748191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 -- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8149150 msgr2=0x7f98d8071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8149150 0x7f98d8071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 -- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8143600 msgr2=0x7f98d8142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98cffff640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8143600 0x7f98d8142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8143600 0x7f98d8142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 -- 172.21.15.107:0/240748191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d81460d0 con 0x7f98c4003970 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.229+0000 7f98d659c640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f98c8037040 tx=0x7f98c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.230+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c8003810 con 0x7f98c4003970 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.230+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c802de90 con 0x7f98c4003970 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.230+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c8003b90 con 0x7f98c4003970 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.230+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d81463b0 con 0x7f98c4003970 2023-11-24T08:15:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.230+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d8071b90 con 0x7f98c4003970 2023-11-24T08:15:49.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.231+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d8146540 con 0x7f98c4003970 2023-11-24T08:15:49.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.232+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f98c8003cf0 con 0x7f98c4003970 2023-11-24T08:15:49.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.234+0000 7f98ceffd640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 0x7f98b806e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.234+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f98c80aa090 con 0x7f98c4003970 2023-11-24T08:15:49.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.234+0000 7f98cffff640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 0x7f98b806e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.234+0000 7f98cffff640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 0x7f98b806e850 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f98c00017a0 tx=0x7f98c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.238+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98c8076b90 con 0x7f98c4003970 2023-11-24T08:15:49.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.465+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d804e840 con 0x7f98c4003970 2023-11-24T08:15:49.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.466+0000 7f98ceffd640 1 -- 172.21.15.107:0/240748191 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f98c8039090 con 0x7f98c4003970 2023-11-24T08:15:49.471 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:49.472 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:15:49.476 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-11-24T08:15:49.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.468+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 msgr2=0x7f98b806e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.468+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 0x7f98b806e850 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f98c00017a0 tx=0x7f98c0009040 comp rx=0 tx=0).stop 2023-11-24T08:15:49.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 msgr2=0x7f98d81496a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f98c8037040 tx=0x7f98c8004730 comp rx=0 tx=0).stop 2023-11-24T08:15:49.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 shutdown_connections 2023-11-24T08:15:49.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98b806c390 0x7f98b806e850 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98d8149150 0x7f98d8071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98d8143600 0x7f98d8142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 --2- 172.21.15.107:0/240748191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98c4003970 0x7f98d81496a0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 >> 172.21.15.107:0/240748191 conn(0x7f98d80b0be0 msgr2=0x7f98d80ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 shutdown_connections 2023-11-24T08:15:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.469+0000 7f98dcee7640 1 -- 172.21.15.107:0/240748191 wait complete. 2023-11-24T08:15:49.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:15:49.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.700+0000 7fcfdf59e640 1 Processor -- start 2023-11-24T08:15:49.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.701+0000 7fcfdf59e640 1 -- start start 2023-11-24T08:15:49.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.702+0000 7fcfdf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 0x7fcfe00cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.702+0000 7fcfdf59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcfe006e6a0 0x7fcfe006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.702+0000 7fcfdf59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe0160e30 con 0x7fcfe00999d0 2023-11-24T08:15:49.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.702+0000 7fcfdf59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00cfc90 con 0x7fcfe006e6a0 2023-11-24T08:15:49.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.702+0000 7fcfdf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00cff70 con 0x7fcfe006b820 2023-11-24T08:15:49.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfde59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcfe00999d0 0x7fcfe0099db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39192/0 (socket says 172.21.15.107:39192) 2023-11-24T08:15:49.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfde59c640 1 -- 172.21.15.107:0/4241778693 learned_addr learned my addr 172.21.15.107:0/4241778693 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfded9d640 1 --2- 172.21.15.107:0/4241778693 >> v2:172.21.15.120:3300/0 conn(0x7fcfe006e6a0 0x7fcfe006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 0x7fcfe00cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216412427 0 0) 0x7fcfd0001110 con 0x7fcfe00999d0 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc003520 con 0x7fcfe00999d0 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.703+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774292237 0 0) 0x7fcfcc003520 con 0x7fcfe00999d0 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 msgr2=0x7fcfe00cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 0x7fcfe00cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 >> v2:172.21.15.120:3300/0 conn(0x7fcfe006e6a0 msgr2=0x7fcfe006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 --2- 172.21.15.107:0/4241778693 >> v2:172.21.15.120:3300/0 conn(0x7fcfe006e6a0 0x7fcfe006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00d0250 con 0x7fcfe00999d0 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcfd0001800 con 0x7fcfe00999d0 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 >> v1:172.21.15.107:6789/0 conn(0x7fcfe00999d0 legacy=0x7fcfe0099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc003970 0x7fcfcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc0047c0 0x7fcfcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 0x7fcfcc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00cfc90 con 0x7fcfcc003970 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00cff70 con 0x7fcfcc0047c0 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc00b980 con 0x7fcfe006b820 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc003970 0x7fcfcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 -- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 msgr2=0x7fcfcc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe006b820 0x7fcfcc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 -- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc0047c0 msgr2=0x7fcfcc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:49.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc0047c0 0x7fcfcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.704+0000 7fcfde59c640 1 -- 172.21.15.107:0/4241778693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc003520 con 0x7fcfcc003970 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.705+0000 7fcfde59c640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc003970 0x7fcfcc003d90 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000c210 tx=0x7fcfcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0993b8ff5c8a9f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd002cae0 con 0x7fcfcc003970 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd002cc60 con 0x7fcfcc003970 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfddd9b640 1 -- 172.21.15.107:0/4241778693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00058d0 con 0x7fcfcc003970 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 -- 172.21.15.107:0/4241778693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc003970 msgr2=0x7fcfcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/4241778693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc003970 0x7fcfcc003d90 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000c210 tx=0x7fcfcc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfded9d640 1 -- 172.21.15.107:0/4241778693 reap_dead start 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 -- 172.21.15.107:0/4241778693 shutdown_connections 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 -- 172.21.15.107:0/4241778693 >> 172.21.15.107:0/4241778693 conn(0x7fcfe00b2ae0 msgr2=0x7fcfe00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 -- 172.21.15.107:0/4241778693 shutdown_connections 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.706+0000 7fcfdf59e640 1 -- 172.21.15.107:0/4241778693 wait complete. 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.708+0000 7fcfdf59e640 1 Processor -- start 2023-11-24T08:15:49.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.708+0000 7fcfdf59e640 1 -- start start 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc003970 0x7fcfe015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe0148590 0x7fcfe0148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c6250 con 0x7fcfe0149390 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe015c7e0 con 0x7fcfcc003970 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfdf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c6480 con 0x7fcfe0148590 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51292/0 (socket says 172.21.15.107:51292) 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.709+0000 7fcfded9d640 1 -- 172.21.15.107:0/3586753189 learned_addr learned my addr 172.21.15.107:0/3586753189 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:49.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfde59c640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc003970 0x7fcfe015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 -- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe0148590 msgr2=0x7fcfe0148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe0148590 0x7fcfe0148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 -- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc003970 msgr2=0x7fcfe015bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc003970 0x7fcfe015bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 -- 172.21.15.107:0/3586753189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00c6760 con 0x7fcfe0149390 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfded9d640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fcfe006e540 tx=0x7fcfd800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8010040 con 0x7fcfe0149390 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800bae0 con 0x7fcfe0149390 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe00c6a40 con 0x7fcfe0149390 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800be70 con 0x7fcfe0149390 2023-11-24T08:15:49.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.710+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe00c7000 con 0x7fcfe0149390 2023-11-24T08:15:49.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.711+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe00c7220 con 0x7fcfe0149390 2023-11-24T08:15:49.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.712+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fcfd801f460 con 0x7fcfe0149390 2023-11-24T08:15:49.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.713+0000 7fcfd6ffd640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 0x7fcfc006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:49.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.713+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcfd80886a0 con 0x7fcfe0149390 2023-11-24T08:15:49.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.714+0000 7fcfde59c640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 0x7fcfc006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:49.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.714+0000 7fcfde59c640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 0x7fcfc006e7d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0034000 tx=0x7fcfd0007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:49.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.718+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfd8055200 con 0x7fcfe0149390 2023-11-24T08:15:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.943+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfe0067470 con 0x7fcfe0149390 2023-11-24T08:15:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.944+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/3586753189 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fcfd8058820 con 0x7fcfe0149390 2023-11-24T08:15:49.947 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:49.947 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:14:28.424695+0000","modified":"2023-11-24T08:14:31.463839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8754":{"gid":8754,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/2095975197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2095975197},{"type":"v1","addr":"172.21.15.120:6835","nonce":2095975197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-24T08:15:49.947 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.947+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 msgr2=0x7fcfc006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.947+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 0x7fcfc006e7d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0034000 tx=0x7fcfd0007000 comp rx=0 tx=0).stop 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.947+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 msgr2=0x7fcfe00c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.947+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fcfe006e540 tx=0x7fcfd800b480 comp rx=0 tx=0).stop 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 shutdown_connections 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc006c310 0x7fcfc006e7d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfe0149390 0x7fcfe00c7520 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe0148590 0x7fcfe0148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 --2- 172.21.15.107:0/3586753189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfcc003970 0x7fcfe015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:49.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 >> 172.21.15.107:0/3586753189 conn(0x7fcfe00b2ae0 msgr2=0x7fcfe00adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:49.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 shutdown_connections 2023-11-24T08:15:49.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:49.948+0000 7fcfdf59e640 1 -- 172.21.15.107:0/3586753189 wait complete. 2023-11-24T08:15:49.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:50.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.173+0000 7f9347fff640 1 Processor -- start 2023-11-24T08:15:50.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.174+0000 7f9347fff640 1 -- start start 2023-11-24T08:15:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.175+0000 7f9347fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.175+0000 7f9347fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f93480cd160 0x7f93480cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.175+0000 7f9347fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9348162eb0 con 0x7f93480703f0 2023-11-24T08:15:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.175+0000 7f9347fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93481654a0 con 0x7f93480cd160 2023-11-24T08:15:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.175+0000 7f9347fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93480ccf00 con 0x7f934814b100 2023-11-24T08:15:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.176+0000 7f93477fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f93480cd160 0x7f93480cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.176+0000 7f9346ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.176+0000 7f93477fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f93480cd160 0x7f93480cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:50994/0 (socket says 172.21.15.107:50994) 2023-11-24T08:15:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.176+0000 7f93477fe640 1 -- 172.21.15.107:0/147973329 learned_addr learned my addr 172.21.15.107:0/147973329 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.177+0000 7f93477fe640 1 -- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 msgr2=0x7f9348162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.177+0000 7f93477fe640 1 --2- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.177+0000 7f93477fe640 1 -- 172.21.15.107:0/147973329 >> v1:172.21.15.120:6789/0 conn(0x7f93480703f0 legacy=0x7f9348165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.177+0000 7f93477fe640 1 -- 172.21.15.107:0/147973329 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348166320 con 0x7f93480cd160 2023-11-24T08:15:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.179+0000 7f93477fe640 1 --2- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480cd160 0x7f93480cd5f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9340009e50 tx=0x7f934002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7e4e1324769c083 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.179+0000 7f93467fc640 1 -- 172.21.15.107:0/147973329 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934002ca70 con 0x7f93480cd160 2023-11-24T08:15:50.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f93467fc640 1 -- 172.21.15.107:0/147973329 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934002cbd0 con 0x7f93480cd160 2023-11-24T08:15:50.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f93467fc640 1 -- 172.21.15.107:0/147973329 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93400343f0 con 0x7f93480cd160 2023-11-24T08:15:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 -- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480cd160 msgr2=0x7f93480cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 --2- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480cd160 0x7f93480cd5f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9340009e50 tx=0x7f934002bed0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 -- 172.21.15.107:0/147973329 shutdown_connections 2023-11-24T08:15:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 --2- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480cd160 0x7f93480cd5f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 --2- 172.21.15.107:0/147973329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 -- 172.21.15.107:0/147973329 >> 172.21.15.107:0/147973329 conn(0x7f93480cb830 msgr2=0x7f93480cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:50.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 -- 172.21.15.107:0/147973329 shutdown_connections 2023-11-24T08:15:50.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.180+0000 7f9347fff640 1 -- 172.21.15.107:0/147973329 wait complete. 2023-11-24T08:15:50.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.182+0000 7f9347fff640 1 Processor -- start 2023-11-24T08:15:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.182+0000 7f9347fff640 1 -- start start 2023-11-24T08:15:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 0x7f93480b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480b48d0 con 0x7f93480cd160 2023-11-24T08:15:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9348154300 con 0x7f93480703f0 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f9347fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9348146230 con 0x7f934814b100 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f933e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f933e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51304/0 (socket says 172.21.15.107:51304) 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.183+0000 7f933e7fc640 1 -- 172.21.15.107:0/2913796320 learned_addr learned my addr 172.21.15.107:0/2913796320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9346ffd640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 0x7f93480b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f93477fe640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 -- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 msgr2=0x7f9348153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 -- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 msgr2=0x7f93480b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 0x7f93480b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 -- 172.21.15.107:0/2913796320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9340009b30 con 0x7f93480cd160 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9346ffd640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 0x7f93480b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f933e7fc640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f933000c3c0 tx=0x7f933000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300052d0 con 0x7f93480cd160 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330005450 con 0x7f93480cd160 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330007260 con 0x7f93480cd160 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.184+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93480b4310 con 0x7f93480cd160 2023-11-24T08:15:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.185+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93481464d0 con 0x7f93480cd160 2023-11-24T08:15:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.186+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934804e840 con 0x7f93480cd160 2023-11-24T08:15:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.187+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f93300073c0 con 0x7f93480cd160 2023-11-24T08:15:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.187+0000 7f9344ff9640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 0x7f932806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.187+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f93300878d0 con 0x7f93480cd160 2023-11-24T08:15:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.188+0000 7f9346ffd640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 0x7f932806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.188+0000 7f9346ffd640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 0x7f932806e7d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9348164e10 tx=0x7f9338008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.192+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9330053ff0 con 0x7f93480cd160 2023-11-24T08:15:50.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.384+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9348162eb0 con 0x7f93480cd160 2023-11-24T08:15:50.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.387+0000 7f9344ff9640 1 -- 172.21.15.107:0/2913796320 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16687 (secure 0 0 0) 0x7f9348162eb0 con 0x7f93480cd160 2023-11-24T08:15:50.390 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:50.390 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:44.960410+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:14:26.165775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-11-24T08:14:27.450883+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.5","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:50.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 msgr2=0x7f932806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 0x7f932806e7d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9348164e10 tx=0x7f9338008040 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 msgr2=0x7f9348144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f933000c3c0 tx=0x7f933000c880 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 shutdown_connections 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f932806c310 0x7f932806e7d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f934814b100 0x7f9348153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93480cd160 0x7f9348144940 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 --2- 172.21.15.107:0/2913796320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93480703f0 0x7f93480b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 >> 172.21.15.107:0/2913796320 conn(0x7f93480cb830 msgr2=0x7f93480ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:50.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 shutdown_connections 2023-11-24T08:15:50.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.390+0000 7f9347fff640 1 -- 172.21.15.107:0/2913796320 wait complete. 2023-11-24T08:15:50.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:15:50.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.603+0000 7fc6b44ee640 1 Processor -- start 2023-11-24T08:15:50.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.603+0000 7fc6b44ee640 1 -- start start 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b44ee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc6ac146eb0 0x7fc6ac161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b44ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac0cd2e0 0x7fc6ac070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b44ee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac161a10 con 0x7fc6ac06e160 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b44ee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac161cf0 con 0x7fc6ac146eb0 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b44ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac161fd0 con 0x7fc6ac0cd2e0 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b1a89640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc6ac06e160 0x7fc6ac0ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39200/0 (socket says 172.21.15.107:39200) 2023-11-24T08:15:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b1a89640 1 -- 172.21.15.107:0/1985043786 learned_addr learned my addr 172.21.15.107:0/1985043786 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b228a640 1 --2- 172.21.15.107:0/1985043786 >> v2:172.21.15.120:3300/0 conn(0x7fc6ac146eb0 0x7fc6ac161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.605+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833068990 0 0) 0x7fc6ac161a10 con 0x7fc6ac06e160 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b2a8b640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac0cd2e0 0x7fc6ac070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc698003520 con 0x7fc6ac06e160 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135146781 0 0) 0x7fc698003520 con 0x7fc6ac06e160 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac0cd2e0 msgr2=0x7fc6ac070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac0cd2e0 0x7fc6ac070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 >> v2:172.21.15.120:3300/0 conn(0x7fc6ac146eb0 msgr2=0x7fc6ac161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 --2- 172.21.15.107:0/1985043786 >> v2:172.21.15.120:3300/0 conn(0x7fc6ac146eb0 0x7fc6ac161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac070660 con 0x7fc6ac06e160 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc69c001690 con 0x7fc6ac06e160 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 >> v1:172.21.15.107:6789/0 conn(0x7fc6ac06e160 legacy=0x7fc6ac0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc698003970 0x7fc698003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc698004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc698005180 0x7fc69800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac161cf0 con 0x7fc6980047c0 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac161fd0 con 0x7fc698005180 2023-11-24T08:15:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69800bf40 con 0x7fc698003970 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b2a8b640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac0cd2e0 0x7fc6ac070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.606+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc698004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b2a8b640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc698005180 0x7fc69800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b1a89640 1 -- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc698003970 msgr2=0x7fc698003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc698003970 0x7fc698003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b1a89640 1 -- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc698005180 msgr2=0x7fc69800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc698005180 0x7fc69800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.607+0000 7fc6b1a89640 1 -- 172.21.15.107:0/1985043786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc698003520 con 0x7fc6980047c0 2023-11-24T08:15:50.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.608+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc698004c40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c160 tx=0x7fc6980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea295356d5ce669 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.608+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c005ad0 con 0x7fc6980047c0 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.608+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c005c30 con 0x7fc6980047c0 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.608+0000 7fc6b1288640 1 -- 172.21.15.107:0/1985043786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c02c810 con 0x7fc6980047c0 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 -- 172.21.15.107:0/1985043786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 msgr2=0x7fc698004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/1985043786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc698004c40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c160 tx=0x7fc6980057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b2a8b640 1 -- 172.21.15.107:0/1985043786 reap_dead start 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 -- 172.21.15.107:0/1985043786 shutdown_connections 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 -- 172.21.15.107:0/1985043786 >> 172.21.15.107:0/1985043786 conn(0x7fc6ac0b0c50 msgr2=0x7fc6ac0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 -- 172.21.15.107:0/1985043786 shutdown_connections 2023-11-24T08:15:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.609+0000 7fc6b44ee640 1 -- 172.21.15.107:0/1985043786 wait complete. 2023-11-24T08:15:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.611+0000 7fc6b44ee640 1 Processor -- start 2023-11-24T08:15:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.611+0000 7fc6b44ee640 1 -- start start 2023-11-24T08:15:50.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.611+0000 7fc6b44ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b44ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6ac0b4570 0x7fc6ac0ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b44ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac15ca20 0x7fc6ac15ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b44ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac142cf0 con 0x7fc6980047c0 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b44ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac15d410 con 0x7fc6ac0b4570 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b44ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac09ac50 con 0x7fc6ac15ca20 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51318/0 (socket says 172.21.15.107:51318) 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 -- 172.21.15.107:0/2009454924 learned_addr learned my addr 172.21.15.107:0/2009454924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6ac0b4570 0x7fc6ac0ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 -- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac15ca20 msgr2=0x7fc6ac15ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac15ca20 0x7fc6ac15ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 -- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6ac0b4570 msgr2=0x7fc6ac0ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6ac0b4570 0x7fc6ac0ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.612+0000 7fc6b228a640 1 -- 172.21.15.107:0/2009454924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac142170 con 0x7fc6980047c0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6b228a640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800e860 tx=0x7fc6a800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8009600 con 0x7fc6980047c0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a800c8c0 con 0x7fc6980047c0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800cc50 con 0x7fc6980047c0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac142450 con 0x7fc6980047c0 2023-11-24T08:15:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.613+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac142a10 con 0x7fc6980047c0 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.614+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac04e8b0 con 0x7fc6980047c0 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.615+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fc6a8013470 con 0x7fc6980047c0 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.616+0000 7fc6a2ffd640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 0x7fc69006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.616+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc6a80884d0 con 0x7fc6980047c0 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.616+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 0x7fc69006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:50.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.617+0000 7fc6b1a89640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 0x7fc69006e7d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc69c034000 tx=0x7fc69c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:50.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.621+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6a8058a60 con 0x7fc6980047c0 2023-11-24T08:15:50.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.847+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc6ac09ad90 con 0x7fc6980047c0 2023-11-24T08:15:50.850 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:15:50.849+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:15:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.852+0000 7fc6a2ffd640 1 -- 172.21.15.107:0/2009454924 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v80) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc6a80585f0 con 0x7fc6980047c0 2023-11-24T08:15:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:15:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 msgr2=0x7fc69006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 0x7fc69006e7d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc69c034000 tx=0x7fc69c007000 comp rx=0 tx=0).stop 2023-11-24T08:15:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 msgr2=0x7fc6ac141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800e860 tx=0x7fc6a800ed20 comp rx=0 tx=0).stop 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 shutdown_connections 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc69006c310 0x7fc69006e7d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6ac15ca20 0x7fc6ac15ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6ac0b4570 0x7fc6ac0ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 --2- 172.21.15.107:0/2009454924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6980047c0 0x7fc6ac141b90 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.855+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 >> 172.21.15.107:0/2009454924 conn(0x7fc6ac0b0c50 msgr2=0x7fc6ac0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.856+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 shutdown_connections 2023-11-24T08:15:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:50.856+0000 7fc6b44ee640 1 -- 172.21.15.107:0/2009454924 wait complete. 2023-11-24T08:15:50.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:15:50.974 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:15:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.087+0000 7f8931832640 1 Processor -- start 2023-11-24T08:15:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.087+0000 7f8931832640 1 -- start start 2023-11-24T08:15:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8931832640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f892c072160 0x7f892c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8931832640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 0x7f892c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8931832640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c15ec90 con 0x7f892c0c49f0 2023-11-24T08:15:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8931832640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c0cd530 con 0x7f892c072160 2023-11-24T08:15:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8931832640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c0c4ed0 con 0x7f892c06de10 2023-11-24T08:15:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f892affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f892c072160 0x7f892c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.088+0000 7f8923fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f892c0c49f0 0x7f892c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39208/0 (socket says 172.21.15.107:39208) 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f8923fff640 1 -- 172.21.15.107:0/107814316 learned_addr learned my addr 172.21.15.107:0/107814316 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892b7fe640 1 --2- 172.21.15.107:0/107814316 >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 0x7f892c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892affd640 1 -- 172.21.15.107:0/107814316 >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 msgr2=0x7f892c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892affd640 1 --2- 172.21.15.107:0/107814316 >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 0x7f892c0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892affd640 1 -- 172.21.15.107:0/107814316 >> v1:172.21.15.107:6789/0 conn(0x7f892c0c49f0 legacy=0x7f892c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892affd640 1 -- 172.21.15.107:0/107814316 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0c51b0 con 0x7f892c072160 2023-11-24T08:15:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892b7fe640 1 --2- 172.21.15.107:0/107814316 >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 0x7f892c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:51.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.089+0000 7f892a7fc640 1 -- 172.21.15.107:0/107814316 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208355290 0 0) 0x7f8914001110 con 0x7f892c0c49f0 2023-11-24T08:15:51.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.090+0000 7f892affd640 1 --2- 172.21.15.107:0/107814316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c072160 0x7f892c15e750 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f892c06eab0 tx=0x7f891c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d339db35a9fd696 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.090+0000 7f892a7fc640 1 -- 172.21.15.107:0/107814316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c031830 con 0x7f892c072160 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.090+0000 7f892a7fc640 1 -- 172.21.15.107:0/107814316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c031990 con 0x7f892c072160 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.090+0000 7f892a7fc640 1 -- 172.21.15.107:0/107814316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c031d20 con 0x7f892c072160 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 -- 172.21.15.107:0/107814316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c072160 msgr2=0x7f892c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 --2- 172.21.15.107:0/107814316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c072160 0x7f892c15e750 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f892c06eab0 tx=0x7f891c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 -- 172.21.15.107:0/107814316 shutdown_connections 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 --2- 172.21.15.107:0/107814316 >> v2:172.21.15.120:3300/0 conn(0x7f892c06de10 0x7f892c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 --2- 172.21.15.107:0/107814316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c072160 0x7f892c15e750 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 -- 172.21.15.107:0/107814316 >> 172.21.15.107:0/107814316 conn(0x7f892c0b0c50 msgr2=0x7f892c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 -- 172.21.15.107:0/107814316 shutdown_connections 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.091+0000 7f8931832640 1 -- 172.21.15.107:0/107814316 wait complete. 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.093+0000 7f8931832640 1 Processor -- start 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.093+0000 7f8931832640 1 -- start start 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f892c06de10 0x7f892c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f892c072160 0x7f892c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c15b270 con 0x7f892c0c49f0 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c15a5f0 con 0x7f892c06de10 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8931832640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c15a8d0 con 0x7f892c072160 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f892b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f892b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51344/0 (socket says 172.21.15.107:51344) 2023-11-24T08:15:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f892b7fe640 1 -- 172.21.15.107:0/1003318294 learned_addr learned my addr 172.21.15.107:0/1003318294 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.094+0000 7f8923fff640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f892c072160 0x7f892c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892affd640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f892c06de10 0x7f892c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 -- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f892c072160 msgr2=0x7f892c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f892c072160 0x7f892c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 -- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f892c06de10 msgr2=0x7f892c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f892c06de10 0x7f892c158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 -- 172.21.15.107:0/1003318294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c009400 con 0x7f892c0c49f0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f892b7fe640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f892400cea0 tx=0x7f892400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924003b90 con 0x7f892c0c49f0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924003d10 con 0x7f892c0c49f0 2023-11-24T08:15:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c15abb0 con 0x7f892c0c49f0 2023-11-24T08:15:51.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924004260 con 0x7f892c0c49f0 2023-11-24T08:15:51.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.095+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c159270 con 0x7f892c0c49f0 2023-11-24T08:15:51.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.096+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c15b3b0 con 0x7f892c0c49f0 2023-11-24T08:15:51.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.097+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f89240043c0 con 0x7f892c0c49f0 2023-11-24T08:15:51.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.098+0000 7f8928ff9640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 0x7f890c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:51.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.098+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8924088fd0 con 0x7f892c0c49f0 2023-11-24T08:15:51.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.099+0000 7f892affd640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 0x7f890c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:51.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.099+0000 7f892affd640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 0x7f890c06e7d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f891c036000 tx=0x7f891c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:51.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.103+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8924091070 con 0x7f892c0c49f0 2023-11-24T08:15:51.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.334+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f892c04e8b0 con 0x7f892c0c49f0 2023-11-24T08:15:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.856+0000 7f8928ff9640 1 -- 172.21.15.107:0/1003318294 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8924055750 con 0x7f892c0c49f0 2023-11-24T08:15:51.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 msgr2=0x7f890c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 0x7f890c06e7d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f891c036000 tx=0x7f891c008030 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 msgr2=0x7f892c0b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f892400cea0 tx=0x7f892400cf90 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 shutdown_connections 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f890c06c310 0x7f890c06e7d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f892c0c49f0 0x7f892c0b2d50 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f892c072160 0x7f892c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 --2- 172.21.15.107:0/1003318294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f892c06de10 0x7f892c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.861+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 >> 172.21.15.107:0/1003318294 conn(0x7f892c0b0c50 msgr2=0x7f892c0cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.862+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 shutdown_connections 2023-11-24T08:15:51.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:51.862+0000 7f8931832640 1 -- 172.21.15.107:0/1003318294 wait complete. 2023-11-24T08:15:51.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:15:52.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.106+0000 7f1ce3bb0640 1 Processor -- start 2023-11-24T08:15:52.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.106+0000 7f1ce3bb0640 1 -- start start 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce3bb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cdc15df80 0x7f1cdc0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce3bb0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc15e3f0 con 0x7f1cdc0b2250 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce3bb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc15ab80 con 0x7f1cdc0cd470 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce3bb0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc15a990 con 0x7f1cdc15df80 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce194c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1cdc0b2250 0x7f1cdc0b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39214/0 (socket says 172.21.15.107:39214) 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce194c640 1 -- 172.21.15.107:0/2025267390 learned_addr learned my addr 172.21.15.107:0/2025267390 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:52.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce114b640 1 --2- 172.21.15.107:0/2025267390 >> v2:172.21.15.107:3300/0 conn(0x7f1cdc15df80 0x7f1cdc0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce094a640 1 -- 172.21.15.107:0/2025267390 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405006276 0 0) 0x7f1cdc15e3f0 con 0x7f1cdc0b2250 2023-11-24T08:15:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce094a640 1 -- 172.21.15.107:0/2025267390 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8003520 con 0x7f1cdc0b2250 2023-11-24T08:15:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce114b640 1 -- 172.21.15.107:0/2025267390 >> v1:172.21.15.120:6789/0 conn(0x7f1cdc0cd470 legacy=0x7f1cdc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce114b640 1 -- 172.21.15.107:0/2025267390 >> v1:172.21.15.107:6789/0 conn(0x7f1cdc0b2250 legacy=0x7f1cdc0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:52.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.108+0000 7f1ce114b640 1 -- 172.21.15.107:0/2025267390 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc0b2d40 con 0x7f1cdc15df80 2023-11-24T08:15:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce114b640 1 --2- 172.21.15.107:0/2025267390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc15df80 0x7f1cdc0ccf30 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0097a0 tx=0x7f1ccc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa0c6677801b3fa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce094a640 1 -- 172.21.15.107:0/2025267390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc031830 con 0x7f1cdc15df80 2023-11-24T08:15:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce094a640 1 -- 172.21.15.107:0/2025267390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc031990 con 0x7f1cdc15df80 2023-11-24T08:15:52.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce094a640 1 -- 172.21.15.107:0/2025267390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc031d20 con 0x7f1cdc15df80 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/2025267390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc15df80 msgr2=0x7f1cdc0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.110+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/2025267390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc15df80 0x7f1cdc0ccf30 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0097a0 tx=0x7f1ccc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.111+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/2025267390 shutdown_connections 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.111+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/2025267390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc15df80 0x7f1cdc0ccf30 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.111+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/2025267390 >> 172.21.15.107:0/2025267390 conn(0x7f1cdc06a920 msgr2=0x7f1cdc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.111+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/2025267390 shutdown_connections 2023-11-24T08:15:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.111+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/2025267390 wait complete. 2023-11-24T08:15:52.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.112+0000 7f1ce3bb0640 1 Processor -- start 2023-11-24T08:15:52.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.113+0000 7f1ce3bb0640 1 -- start start 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.113+0000 7f1ce3bb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 0x7f1cdc14e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce3bb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce3bb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cdc15df80 0x7f1cdc157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce3bb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc14ba40 con 0x7f1cdc0cd470 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce3bb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc0b0d90 con 0x7f1cdc15df80 2023-11-24T08:15:52.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce3bb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc0b0180 con 0x7f1cdc0b2250 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51360/0 (socket says 172.21.15.107:51360) 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 -- 172.21.15.107:0/418009598 learned_addr learned my addr 172.21.15.107:0/418009598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce194c640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 0x7f1cdc14e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 -- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 msgr2=0x7f1cdc14e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce214d640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cdc15df80 0x7f1cdc157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 0x7f1cdc14e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 -- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cdc15df80 msgr2=0x7f1cdc157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cdc15df80 0x7f1cdc157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce114b640 1 -- 172.21.15.107:0/418009598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc009400 con 0x7f1cdc0cd470 2023-11-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.114+0000 7f1ce194c640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 0x7f1cdc14e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1ce114b640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc000c00 tx=0x7f1ccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc03a8f0 con 0x7f1cdc0cd470 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc03aa50 con 0x7f1cdc0cd470 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc0b0460 con 0x7f1cdc0cd470 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc03ae00 con 0x7f1cdc0cd470 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.115+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc0b0a20 con 0x7f1cdc0cd470 2023-11-24T08:15:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.116+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc0ae410 con 0x7f1cdc0cd470 2023-11-24T08:15:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.118+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f1ccc04d450 con 0x7f1cdc0cd470 2023-11-24T08:15:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.119+0000 7f1cd27fc640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 0x7f1cc006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:52.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.119+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1ccc0aa340 con 0x7f1cdc0cd470 2023-11-24T08:15:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.119+0000 7f1ce194c640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 0x7f1cc006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.120+0000 7f1ce194c640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 0x7f1cc006e800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80011e0 tx=0x7f1cd8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:52.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.123+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ccc076e40 con 0x7f1cdc0cd470 2023-11-24T08:15:52.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:52.313+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1cdc067470 con 0x7f1cdc0cd470 2023-11-24T08:15:53.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.320+0000 7f1cd27fc640 1 -- 172.21.15.107:0/418009598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1ccc035070 con 0x7f1cdc0cd470 2023-11-24T08:15:53.322 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:15:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 msgr2=0x7f1cc006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 0x7f1cc006e800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80011e0 tx=0x7f1cd8009040 comp rx=0 tx=0).stop 2023-11-24T08:15:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 msgr2=0x7f1cdc14b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc000c00 tx=0x7f1ccc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 shutdown_connections 2023-11-24T08:15:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.323+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1cc006c340 0x7f1cc006e800 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.324+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cdc15df80 0x7f1cdc157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.324+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0cd470 0x7f1cdc14b500 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.324+0000 7f1ce3bb0640 1 --2- 172.21.15.107:0/418009598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cdc0b2250 0x7f1cdc14e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.324+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 >> 172.21.15.107:0/418009598 conn(0x7f1cdc06a920 msgr2=0x7f1cdc15c640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.324+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 shutdown_connections 2023-11-24T08:15:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.325+0000 7f1ce3bb0640 1 -- 172.21.15.107:0/418009598 wait complete. 2023-11-24T08:15:53.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:15:53.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.551+0000 7f9ac4a07640 1 Processor -- start 2023-11-24T08:15:53.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.551+0000 7f9ac4a07640 1 -- start start 2023-11-24T08:15:53.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.551+0000 7f9ac4a07640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ac00cfb60 0x7f9ac0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.551+0000 7f9ac4a07640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00ce0b0 con 0x7f9ac015dc80 2023-11-24T08:15:53.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9ac4a07640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac0161960 con 0x7f9ac0157c70 2023-11-24T08:15:53.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9ac4a07640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac015e8d0 con 0x7f9ac00cfb60 2023-11-24T08:15:53.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9abed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9ac015dc80 0x7f9ac00b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39228/0 (socket says 172.21.15.107:39228) 2023-11-24T08:15:53.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9abed9d640 1 -- 172.21.15.107:0/3027480144 learned_addr learned my addr 172.21.15.107:0/3027480144 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/3027480144 >> v2:172.21.15.120:3300/0 conn(0x7f9ac00cfb60 0x7f9ac0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594725792 0 0) 0x7f9ac00ce0b0 con 0x7f9ac015dc80 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.552+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8003520 con 0x7f9ac015dc80 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119219235 0 0) 0x7f9aa8003520 con 0x7f9ac015dc80 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 >> v2:172.21.15.120:3300/0 conn(0x7f9ac00cfb60 msgr2=0x7f9ac0161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 --2- 172.21.15.107:0/3027480144 >> v2:172.21.15.120:3300/0 conn(0x7f9ac00cfb60 0x7f9ac0161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 >> v1:172.21.15.120:6789/0 conn(0x7f9ac0157c70 legacy=0x7f9ac00cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac015ebb0 con 0x7f9ac015dc80 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9ab80019f0 con 0x7f9ac015dc80 2023-11-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 >> v1:172.21.15.107:6789/0 conn(0x7f9ac015dc80 legacy=0x7f9ac00b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9aa80039c0 0x7f9aa8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9aa80047c0 0x7f9aa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9aa800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac015e8d0 con 0x7f9aa80051d0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800c120 con 0x7f9aa80047c0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800c2e0 con 0x7f9aa80039c0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.553+0000 7f9abed9d640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9aa800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abe59c640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9aa80039c0 0x7f9aa8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9aa80047c0 0x7f9aa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abed9d640 1 -- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9aa80039c0 msgr2=0x7f9aa8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abed9d640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9aa80039c0 0x7f9aa8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abed9d640 1 -- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9aa80047c0 msgr2=0x7f9aa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abed9d640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9aa80047c0 0x7f9aa8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abed9d640 1 -- 172.21.15.107:0/3027480144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8003520 con 0x7f9aa80051d0 2023-11-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.554+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9aa80047c0 0x7f9aa8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.555+0000 7f9abed9d640 1 -- 172.21.15.107:0/3027480144 reap_dead start 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.555+0000 7f9abed9d640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9aa800b980 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800a720 tx=0x7f9aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5cbf679ec352d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.555+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8034630 con 0x7f9aa80051d0 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.555+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802f070 con 0x7f9aa80051d0 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.555+0000 7f9abd59a640 1 -- 172.21.15.107:0/3027480144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80349f0 con 0x7f9aa80051d0 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 -- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 msgr2=0x7f9aa800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9aa800b980 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800a720 tx=0x7f9aa80057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 -- 172.21.15.107:0/3027480144 shutdown_connections 2023-11-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/3027480144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9aa800b980 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 -- 172.21.15.107:0/3027480144 >> 172.21.15.107:0/3027480144 conn(0x7f9ac006a920 msgr2=0x7f9ac006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 -- 172.21.15.107:0/3027480144 shutdown_connections 2023-11-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.556+0000 7f9ac4a07640 1 -- 172.21.15.107:0/3027480144 wait complete. 2023-11-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.558+0000 7f9ac4a07640 1 Processor -- start 2023-11-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.558+0000 7f9ac4a07640 1 -- start start 2023-11-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ac0152db0 0x7f9ac0148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ac00c94a0 0x7f9ac015efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0148980 con 0x7f9aa80051d0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00abfc0 con 0x7f9ac00c94a0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9ac4a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0153370 con 0x7f9ac0152db0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51370/0 (socket says 172.21.15.107:51370) 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 -- 172.21.15.107:0/193557790 learned_addr learned my addr 172.21.15.107:0/193557790 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 -- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ac0152db0 msgr2=0x7f9ac0148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ac0152db0 0x7f9ac0148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ac0152db0 0x7f9ac0148440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abed9d640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ac00c94a0 0x7f9ac015efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 -- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ac00c94a0 msgr2=0x7f9ac015efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ac00c94a0 0x7f9ac015efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.559+0000 7f9abe59c640 1 -- 172.21.15.107:0/193557790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac0153650 con 0x7f9aa80051d0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9abe59c640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e410 tx=0x7f9ab000e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00052e0 con 0x7f9aa80051d0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab0005440 con 0x7f9aa80051d0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00037f0 con 0x7f9aa80051d0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0153930 con 0x7f9aa80051d0 2023-11-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.560+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0152150 con 0x7f9aa80051d0 2023-11-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.561+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac00ac130 con 0x7f9aa80051d0 2023-11-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.562+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f9ab0003950 con 0x7f9aa80051d0 2023-11-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.563+0000 7f9aaeffd640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 0x7f9aa006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.563+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f9ab0087ae0 con 0x7f9aa80051d0 2023-11-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.563+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 0x7f9aa006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.564+0000 7f9abdd9b640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 0x7f9aa006e800 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0160870 tx=0x7f9ab4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:53.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.569+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ab0054600 con 0x7f9aa80051d0 2023-11-24T08:15:53.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:53.763+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ac0061a50 con 0x7f9aa80051d0 2023-11-24T08:15:54.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.333+0000 7f9aaeffd640 1 -- 172.21.15.107:0/193557790 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9ab0019030 con 0x7f9aa80051d0 2023-11-24T08:15:54.335 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:15:54.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.338+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 msgr2=0x7f9aa006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.338+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 0x7f9aa006e800 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0160870 tx=0x7f9ab4008040 comp rx=0 tx=0).stop 2023-11-24T08:15:54.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.338+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 msgr2=0x7f9ac0154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.338+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e410 tx=0x7f9ab000e8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 shutdown_connections 2023-11-24T08:15:54.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9aa006c340 0x7f9aa006e800 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ac00c94a0 0x7f9ac015efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ac0152db0 0x7f9ac0148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 --2- 172.21.15.107:0/193557790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9aa80051d0 0x7f9ac0154c00 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.339+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 >> 172.21.15.107:0/193557790 conn(0x7f9ac006a920 msgr2=0x7f9ac00af480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:54.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.340+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 shutdown_connections 2023-11-24T08:15:54.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.340+0000 7f9ac4a07640 1 -- 172.21.15.107:0/193557790 wait complete. 2023-11-24T08:15:54.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:15:54.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.567+0000 7f4e0f9be640 1 Processor -- start 2023-11-24T08:15:54.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.567+0000 7f4e0f9be640 1 -- start start 2023-11-24T08:15:54.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.568+0000 7f4e0f9be640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e080c4220 0x7f4e080c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.568+0000 7f4e0f9be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0815e400 con 0x7f4e0815dc50 2023-11-24T08:15:54.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.568+0000 7f4e0f9be640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080c5ed0 con 0x7f4e0815d7f0 2023-11-24T08:15:54.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.568+0000 7f4e0f9be640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e080ce880 con 0x7f4e080c4220 2023-11-24T08:15:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4e0df5b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4e0815dc50 0x7f4e08159690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39242/0 (socket says 172.21.15.107:39242) 2023-11-24T08:15:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4e0df5b640 1 -- 172.21.15.107:0/3080015900 learned_addr learned my addr 172.21.15.107:0/3080015900 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/3080015900 >> v2:172.21.15.120:3300/0 conn(0x7f4e080c4220 0x7f4e080c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758150656 0 0) 0x7f4e0815e400 con 0x7f4e0815dc50 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df4003520 con 0x7f4e0815dc50 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 572635336 0 0) 0x7f4df4003520 con 0x7f4e0815dc50 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 >> v2:172.21.15.120:3300/0 conn(0x7f4e080c4220 msgr2=0x7f4e080c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 --2- 172.21.15.107:0/3080015900 >> v2:172.21.15.120:3300/0 conn(0x7f4e080c4220 0x7f4e080c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 >> v1:172.21.15.120:6789/0 conn(0x7f4e0815d7f0 legacy=0x7f4e080cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e080c46e0 con 0x7f4e0815dc50 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.569+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4e000019f0 con 0x7f4e0815dc50 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 >> v1:172.21.15.107:6789/0 conn(0x7f4e0815dc50 legacy=0x7f4e08159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4df4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4df40047c0 0x7f4df4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df40051d0 0x7f4df400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df400c0d0 con 0x7f4df40039c0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df400c240 con 0x7f4df40051d0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df400c400 con 0x7f4df40047c0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4df4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 -- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4df40047c0 msgr2=0x7f4df4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4df40047c0 0x7f4df4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 -- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df40051d0 msgr2=0x7f4df400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0df5b640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df40051d0 0x7f4df400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df40051d0 0x7f4df400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0d75a640 1 -- 172.21.15.107:0/3080015900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4003520 con 0x7f4df40039c0 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0df5b640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df40051d0 0x7f4df400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.570+0000 7f4e0df5b640 1 -- 172.21.15.107:0/3080015900 reap_dead start 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.571+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4df4003dc0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba00 tx=0x7f4df40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b5cf8d27c4802b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04032620 con 0x7f4df40039c0 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04032780 con 0x7f4df40039c0 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4dfffff640 1 -- 172.21.15.107:0/3080015900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04032b10 con 0x7f4df40039c0 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4e0f9be640 1 -- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 msgr2=0x7f4df4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4df4003dc0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba00 tx=0x7f4df40057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4e0f9be640 1 -- 172.21.15.107:0/3080015900 shutdown_connections 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/3080015900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4df4003dc0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.572+0000 7f4e0f9be640 1 -- 172.21.15.107:0/3080015900 >> 172.21.15.107:0/3080015900 conn(0x7f4e080b0c50 msgr2=0x7f4e080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.573+0000 7f4e0f9be640 1 -- 172.21.15.107:0/3080015900 shutdown_connections 2023-11-24T08:15:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.573+0000 7f4e0f9be640 1 -- 172.21.15.107:0/3080015900 wait complete. 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.574+0000 7f4e0f9be640 1 Processor -- start 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.574+0000 7f4e0f9be640 1 -- start start 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e0809a450 0x7f4e0809a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e08152f50 0x7f4e081533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e081591f0 con 0x7f4df40039c0 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0809ae10 con 0x7f4e0809a450 2023-11-24T08:15:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.575+0000 7f4e0f9be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e08099f10 con 0x7f4e08152f50 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51394/0 (socket says 172.21.15.107:51394) 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 -- 172.21.15.107:0/1370226503 learned_addr learned my addr 172.21.15.107:0/1370226503 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 -- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e08152f50 msgr2=0x7f4e081533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e08152f50 0x7f4e081533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 -- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e0809a450 msgr2=0x7f4e0809a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e0809a450 0x7f4e0809a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 -- 172.21.15.107:0/1370226503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e08153f30 con 0x7f4df40039c0 2023-11-24T08:15:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.576+0000 7f4e0d75a640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f4e0402bea0 tx=0x7f4e04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.577+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0403b620 con 0x7f4df40039c0 2023-11-24T08:15:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.577+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0403b780 con 0x7f4df40039c0 2023-11-24T08:15:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.577+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e08158c30 con 0x7f4df40039c0 2023-11-24T08:15:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.577+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0403baf0 con 0x7f4df40039c0 2023-11-24T08:15:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.577+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e080ca5a0 con 0x7f4df40039c0 2023-11-24T08:15:54.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.578+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0804e8b0 con 0x7f4df40039c0 2023-11-24T08:15:54.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.579+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f4e0403bc50 con 0x7f4df40039c0 2023-11-24T08:15:54.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.580+0000 7f4dfdffb640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 0x7f4dec06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:54.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.581+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4e040aa020 con 0x7f4df40039c0 2023-11-24T08:15:54.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.581+0000 7f4e0cf59640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 0x7f4dec06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:54.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.581+0000 7f4e0cf59640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 0x7f4dec06e800 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4df8003cb0 tx=0x7f4df80079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:54.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.585+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e04076b80 con 0x7f4df40039c0 2023-11-24T08:15:54.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.776+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4e080c4000 con 0x7f4df40039c0 2023-11-24T08:15:54.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.783+0000 7f4dfdffb640 1 -- 172.21.15.107:0/1370226503 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4e0407a1a0 con 0x7f4df40039c0 2023-11-24T08:15:54.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 msgr2=0x7f4dec06e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 0x7f4dec06e800 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4df8003cb0 tx=0x7f4df80079b0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 msgr2=0x7f4e080999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:54.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f4e0402bea0 tx=0x7f4e04004730 comp rx=0 tx=0).stop 2023-11-24T08:15:54.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 shutdown_connections 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dec06c340 0x7f4dec06e800 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e08152f50 0x7f4e081533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e0809a450 0x7f4e0809a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 --2- 172.21.15.107:0/1370226503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40039c0 0x7f4e080999d0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 >> 172.21.15.107:0/1370226503 conn(0x7f4e080b0c50 msgr2=0x7f4e08159460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 shutdown_connections 2023-11-24T08:15:54.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:54.786+0000 7f4e0f9be640 1 -- 172.21.15.107:0/1370226503 wait complete. 2023-11-24T08:15:54.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:15:55.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.020+0000 7f8a97189640 1 Processor -- start 2023-11-24T08:15:55.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.020+0000 7f8a97189640 1 -- start start 2023-11-24T08:15:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.021+0000 7f8a97189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 0x7f8a900c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.021+0000 7f8a97189640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9015e400 con 0x7f8a900c4220 2023-11-24T08:15:55.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a97189640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a900c5ed0 con 0x7f8a9006d600 2023-11-24T08:15:55.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a97189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a900ce880 con 0x7f8a9015d7f0 2023-11-24T08:15:55.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a8ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8a900c4220 0x7f8a900cabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39244/0 (socket says 172.21.15.107:39244) 2023-11-24T08:15:55.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a8ffff640 1 -- 172.21.15.107:0/2571963560 learned_addr learned my addr 172.21.15.107:0/2571963560 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a94f25640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 0x7f8a900c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628761362 0 0) 0x7f8a9015e400 con 0x7f8a900c4220 2023-11-24T08:15:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.022+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a80003520 con 0x7f8a900c4220 2023-11-24T08:15:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3209387906 0 0) 0x7f8a80003520 con 0x7f8a900c4220 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 msgr2=0x7f8a900c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 0x7f8a900c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 >> v1:172.21.15.120:6789/0 conn(0x7f8a9006d600 legacy=0x7f8a900cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900c46e0 con 0x7f8a900c4220 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8a7c001670 con 0x7f8a900c4220 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 >> v1:172.21.15.107:6789/0 conn(0x7f8a900c4220 legacy=0x7f8a900cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800047c0 0x7f8a80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 0x7f8a80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000dc40 con 0x7f8a800039a0 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000dd80 con 0x7f8a800047c0 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000df40 con 0x7f8a9015d7f0 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a94f25640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a8ffff640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800047c0 0x7f8a80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.023+0000 7f8a94f25640 1 -- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 msgr2=0x7f8a80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.024+0000 7f8a94f25640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a9015d7f0 0x7f8a80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.024+0000 7f8a94f25640 1 -- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800047c0 msgr2=0x7f8a80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.024+0000 7f8a94f25640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a800047c0 0x7f8a80004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.024+0000 7f8a94f25640 1 -- 172.21.15.107:0/2571963560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80003520 con 0x7f8a800039a0 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a94f25640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a80003dc0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009a90 tx=0x7f8a800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade708a77e3b456 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8402c6b0 con 0x7f8a800039a0 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8402c810 con 0x7f8a800039a0 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a8f7fe640 1 -- 172.21.15.107:0/2571963560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8402cba0 con 0x7f8a800039a0 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a97189640 1 -- 172.21.15.107:0/2571963560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 msgr2=0x7f8a80003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a97189640 1 --2- 172.21.15.107:0/2571963560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a80003dc0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009a90 tx=0x7f8a800057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a95726640 1 -- 172.21.15.107:0/2571963560 reap_dead start 2023-11-24T08:15:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a97189640 1 -- 172.21.15.107:0/2571963560 shutdown_connections 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.025+0000 7f8a97189640 1 -- 172.21.15.107:0/2571963560 >> 172.21.15.107:0/2571963560 conn(0x7f8a900b0c50 msgr2=0x7f8a900b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.026+0000 7f8a97189640 1 -- 172.21.15.107:0/2571963560 shutdown_connections 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.026+0000 7f8a97189640 1 -- 172.21.15.107:0/2571963560 wait complete. 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.027+0000 7f8a97189640 1 Processor -- start 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.028+0000 7f8a97189640 1 -- start start 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.028+0000 7f8a97189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.028+0000 7f8a97189640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a901f61f0 0x7f8a900cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a97189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a901f41c0 0x7f8a90071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a97189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90157810 con 0x7f8a800039a0 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a97189640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901f31c0 con 0x7f8a901f61f0 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a97189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901f34a0 con 0x7f8a901f41c0 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51420/0 (socket says 172.21.15.107:51420) 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 -- 172.21.15.107:0/1919963653 learned_addr learned my addr 172.21.15.107:0/1919963653 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 -- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a901f41c0 msgr2=0x7f8a90071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a901f41c0 0x7f8a90071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 -- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a901f61f0 msgr2=0x7f8a900cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a901f61f0 0x7f8a900cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 -- 172.21.15.107:0/1919963653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a901f3780 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.029+0000 7f8a94f25640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402c250 tx=0x7f8a84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.030+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84003810 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.030+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84003970 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.030+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84003ce0 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.030+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a901f3a60 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.030+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a90071c80 con 0x7f8a800039a0 2023-11-24T08:15:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.031+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9004e8b0 con 0x7f8a800039a0 2023-11-24T08:15:55.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.032+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f8a84040450 con 0x7f8a800039a0 2023-11-24T08:15:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.033+0000 7f8a8d7fa640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 0x7f8a7406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.033+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8a840aa240 con 0x7f8a800039a0 2023-11-24T08:15:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.033+0000 7f8a8ffff640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 0x7f8a7406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.034+0000 7f8a8ffff640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 0x7f8a7406e7d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c001200 tx=0x7f8a7c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.038+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a840b4080 con 0x7f8a800039a0 2023-11-24T08:15:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.238+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f8a900b4b50 con 0x7f8a800039a0 2023-11-24T08:15:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.247+0000 7f8a8d7fa640 1 -- 172.21.15.107:0/1919963653 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v38) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8a840390b0 con 0x7f8a800039a0 2023-11-24T08:15:55.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 msgr2=0x7f8a7406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 0x7f8a7406e7d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c001200 tx=0x7f8a7c009040 comp rx=0 tx=0).stop 2023-11-24T08:15:55.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 msgr2=0x7f8a900ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402c250 tx=0x7f8a84004730 comp rx=0 tx=0).stop 2023-11-24T08:15:55.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 shutdown_connections 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8a7406c310 0x7f8a7406e7d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a901f41c0 0x7f8a90071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a901f61f0 0x7f8a900cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 --2- 172.21.15.107:0/1919963653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a800039a0 0x7f8a900ccd60 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 >> 172.21.15.107:0/1919963653 conn(0x7f8a900b0c50 msgr2=0x7f8a900b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 shutdown_connections 2023-11-24T08:15:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.250+0000 7f8a97189640 1 -- 172.21.15.107:0/1919963653 wait complete. 2023-11-24T08:15:55.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath' 2023-11-24T08:15:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.466+0000 7fddc61ea640 1 Processor -- start 2023-11-24T08:15:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.466+0000 7fddc61ea640 1 -- start start 2023-11-24T08:15:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.467+0000 7fddc61ea640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fddc0160d40 0x7fddc00b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.467+0000 7fddc61ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc00cdf90 con 0x7fddc00b2740 2023-11-24T08:15:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.467+0000 7fddc61ea640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc00ce230 con 0x7fddc00d04f0 2023-11-24T08:15:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.467+0000 7fddc61ea640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc00d2430 con 0x7fddc0160d40 2023-11-24T08:15:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddb6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fddc00b2740 0x7fddc00cfde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39250/0 (socket says 172.21.15.107:39250) 2023-11-24T08:15:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddb6ffd640 1 -- 172.21.15.107:0/831494733 learned_addr learned my addr 172.21.15.107:0/831494733 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/831494733 >> v2:172.21.15.120:3300/0 conn(0x7fddc0160d40 0x7fddc00b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748756921 0 0) 0x7fddc00cdf90 con 0x7fddc00b2740 2023-11-24T08:15:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac003520 con 0x7fddc00b2740 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189643859 0 0) 0x7fddac003520 con 0x7fddc00b2740 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 >> v2:172.21.15.120:3300/0 conn(0x7fddc0160d40 msgr2=0x7fddc00b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 --2- 172.21.15.107:0/831494733 >> v2:172.21.15.120:3300/0 conn(0x7fddc0160d40 0x7fddc00b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.468+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 >> v1:172.21.15.120:6789/0 conn(0x7fddc00d04f0 legacy=0x7fddc00ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc00d1b90 con 0x7fddc00b2740 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdda8001690 con 0x7fddc00b2740 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 >> v1:172.21.15.107:6789/0 conn(0x7fddc00b2740 legacy=0x7fddc00cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 0x7fddac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddac0047c0 0x7fddac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0051d0 0x7fddac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc00d2430 con 0x7fddac0039c0 2023-11-24T08:15:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac00c100 con 0x7fddac0047c0 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac00c2c0 con 0x7fddac0051d0 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 0x7fddac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddb6ffd640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddac0047c0 0x7fddac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 -- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0051d0 msgr2=0x7fddac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0051d0 0x7fddac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 -- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddac0047c0 msgr2=0x7fddac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddac0047c0 0x7fddac004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbf7fe640 1 -- 172.21.15.107:0/831494733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac003520 con 0x7fddac0039c0 2023-11-24T08:15:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.469+0000 7fddbffff640 1 -- 172.21.15.107:0/831494733 reap_dead start 2023-11-24T08:15:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.470+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 0x7fddac003dc0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fddb0007330 tx=0x7fddac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40fe89ed0a5b2ea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb002ccb0 con 0x7fddac0039c0 2023-11-24T08:15:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb002ce10 con 0x7fddac0039c0 2023-11-24T08:15:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddbeffd640 1 -- 172.21.15.107:0/831494733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0034660 con 0x7fddac0039c0 2023-11-24T08:15:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddc61ea640 1 -- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 msgr2=0x7fddac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddc61ea640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 0x7fddac003dc0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fddb0007330 tx=0x7fddac0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddc61ea640 1 -- 172.21.15.107:0/831494733 shutdown_connections 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddc61ea640 1 --2- 172.21.15.107:0/831494733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac0039c0 0x7fddac003dc0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.471+0000 7fddc61ea640 1 -- 172.21.15.107:0/831494733 >> 172.21.15.107:0/831494733 conn(0x7fddc008af60 msgr2=0x7fddc00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.472+0000 7fddc61ea640 1 -- 172.21.15.107:0/831494733 shutdown_connections 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.472+0000 7fddc61ea640 1 -- 172.21.15.107:0/831494733 wait complete. 2023-11-24T08:15:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.473+0000 7fddc61ea640 1 Processor -- start 2023-11-24T08:15:55.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.474+0000 7fddc61ea640 1 -- start start 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.474+0000 7fddc61ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0039c0 0x7fddc01f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.474+0000 7fddc61ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.474+0000 7fddc61ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 0x7fddc00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.474+0000 7fddc61ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc00b9150 con 0x7fddc014b650 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddc61ea640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc014c010 con 0x7fddc00b7760 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddc61ea640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc00b84d0 con 0x7fddac0039c0 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51426/0 (socket says 172.21.15.107:51426) 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 -- 172.21.15.107:0/162961603 learned_addr learned my addr 172.21.15.107:0/162961603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0039c0 0x7fddc01f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddbffff640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 0x7fddc00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 -- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0039c0 msgr2=0x7fddc01f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0039c0 0x7fddc01f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 -- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 msgr2=0x7fddc00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 0x7fddc00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddb6ffd640 1 -- 172.21.15.107:0/162961603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc00b87b0 con 0x7fddc014b650 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.475+0000 7fddbffff640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 0x7fddc00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddb6ffd640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fdda800d7a0 tx=0x7fdda800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda80074e0 con 0x7fddc014b650 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda8007640 con 0x7fddc014b650 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda80049e0 con 0x7fddc014b650 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc00b8a90 con 0x7fddc014b650 2023-11-24T08:15:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.476+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc01f6430 con 0x7fddc014b650 2023-11-24T08:15:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.477+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc0067470 con 0x7fddc014b650 2023-11-24T08:15:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.478+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fdda8004b40 con 0x7fddc014b650 2023-11-24T08:15:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.479+0000 7fddbcff9640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 0x7fdda006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.479+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdda8087ba0 con 0x7fddc014b650 2023-11-24T08:15:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.479+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 0x7fdda006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.480+0000 7fddbf7fe640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 0x7fdda006e7d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.484+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdda8054700 con 0x7fddc014b650 2023-11-24T08:15:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:55.748+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7fddc008b400 con 0x7fddc014b650 2023-11-24T08:15:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.345+0000 7fddbcff9640 1 -- 172.21.15.107:0/162961603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_getpath"]}]=0 v565) v1 ==== 173+0+0 (secure 0 0 0) 0x7fdda8019070 con 0x7fddc014b650 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 msgr2=0x7fdda006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 0x7fdda006e7d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb0030000 comp rx=0 tx=0).stop 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 msgr2=0x7fddc014bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fdda800d7a0 tx=0x7fdda800dc60 comp rx=0 tx=0).stop 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 shutdown_connections 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdda006c310 0x7fdda006e7d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc00b7760 0x7fddc00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddc014b650 0x7fddc014bad0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 --2- 172.21.15.107:0/162961603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddac0039c0 0x7fddc01f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 >> 172.21.15.107:0/162961603 conn(0x7fddc008af60 msgr2=0x7fddc006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:56.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 shutdown_connections 2023-11-24T08:15:56.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.349+0000 7fddc61ea640 1 -- 172.21.15.107:0/162961603 wait complete. 2023-11-24T08:15:56.370 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:15:56.371 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_snapshot_rm (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:15:56.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:15:56.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.580+0000 7f6071a3d640 1 Processor -- start 2023-11-24T08:15:56.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.580+0000 7f6071a3d640 1 -- start start 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f6071a3d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f606c15e390 0x7f606c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f6071a3d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0ce0b0 con 0x7f606c0cfb60 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f6071a3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c161960 con 0x7f606c157c70 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f6071a3d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c15e8d0 con 0x7f606c15e390 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f606c15e390 0x7f606c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:56.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f606c15e390 0x7f606c161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51428/0 (socket says 172.21.15.107:51428) 2023-11-24T08:15:56.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 -- 172.21.15.107:0/3619566503 learned_addr learned my addr 172.21.15.107:0/3619566503 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:56.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f60627fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f606c0cfb60 0x7f606c15dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39260/0 (socket says 172.21.15.107:39260) 2023-11-24T08:15:56.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 -- 172.21.15.107:0/3619566503 >> v1:172.21.15.120:6789/0 conn(0x7f606c157c70 legacy=0x7f606c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 -- 172.21.15.107:0/3619566503 >> v1:172.21.15.107:6789/0 conn(0x7f606c0cfb60 legacy=0x7f606c15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.582+0000 7f606b7fe640 1 -- 172.21.15.107:0/3619566503 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c15ebb0 con 0x7f606c15e390 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.583+0000 7f606b7fe640 1 --2- 172.21.15.107:0/3619566503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c161df0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f6064009e50 tx=0x7f606402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4aa1a9b44cd108d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f606a7fc640 1 -- 172.21.15.107:0/3619566503 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606402ca80 con 0x7f606c15e390 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f606a7fc640 1 -- 172.21.15.107:0/3619566503 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606402cbe0 con 0x7f606c15e390 2023-11-24T08:15:56.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f606a7fc640 1 -- 172.21.15.107:0/3619566503 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60640343f0 con 0x7f606c15e390 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f6071a3d640 1 -- 172.21.15.107:0/3619566503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 msgr2=0x7f606c161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f6071a3d640 1 --2- 172.21.15.107:0/3619566503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c161df0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f6064009e50 tx=0x7f606402bee0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f6071a3d640 1 -- 172.21.15.107:0/3619566503 shutdown_connections 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f6071a3d640 1 --2- 172.21.15.107:0/3619566503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c161df0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.584+0000 7f6071a3d640 1 -- 172.21.15.107:0/3619566503 >> 172.21.15.107:0/3619566503 conn(0x7f606c06a920 msgr2=0x7f606c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.585+0000 7f6071a3d640 1 -- 172.21.15.107:0/3619566503 shutdown_connections 2023-11-24T08:15:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.585+0000 7f6071a3d640 1 -- 172.21.15.107:0/3619566503 wait complete. 2023-11-24T08:15:56.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.586+0000 7f6071a3d640 1 Processor -- start 2023-11-24T08:15:56.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.586+0000 7f6071a3d640 1 -- start start 2023-11-24T08:15:56.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c0cfb60 0x7f606c071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:56.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c157c70 0x7f606c09c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c09d090 con 0x7f606c15e390 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0cafe0 con 0x7f606c157c70 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.587+0000 7f6071a3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0cb2c0 con 0x7f606c0cfb60 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:56.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51438/0 (socket says 172.21.15.107:51438) 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 -- 172.21.15.107:0/2993605443 learned_addr learned my addr 172.21.15.107:0/2993605443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 -- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c0cfb60 msgr2=0x7f606c071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c0cfb60 0x7f606c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 -- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c157c70 msgr2=0x7f606c09c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c157c70 0x7f606c09c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 -- 172.21.15.107:0/2993605443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064009b30 con 0x7f606c15e390 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.588+0000 7f606b7fe640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f6064036040 tx=0x7f6064004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.589+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064009510 con 0x7f606c15e390 2023-11-24T08:15:56.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.589+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606403e450 con 0x7f606c15e390 2023-11-24T08:15:56.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.589+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606403e800 con 0x7f606c15e390 2023-11-24T08:15:56.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.589+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c0cb5a0 con 0x7f606c15e390 2023-11-24T08:15:56.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.589+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c0cbb60 con 0x7f606c15e390 2023-11-24T08:15:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.590+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c0cc140 con 0x7f606c15e390 2023-11-24T08:15:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.591+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f606402cd50 con 0x7f606c15e390 2023-11-24T08:15:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.592+0000 7f6068ff9640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 0x7f604c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.592+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60640aae40 con 0x7f606c15e390 2023-11-24T08:15:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.592+0000 7f606affd640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 0x7f604c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:56.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.593+0000 7f606affd640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 0x7f604c06e7d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f605c003cc0 tx=0x7f605c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:56.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.596+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60640779a0 con 0x7f606c15e390 2023-11-24T08:15:56.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.791+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f606c0cb730 con 0x7f606c15e390 2023-11-24T08:15:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.793+0000 7f6068ff9640 1 -- 172.21.15.107:0/2993605443 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+12998 (secure 0 0 0) 0x7f6064043020 con 0x7f606c15e390 2023-11-24T08:15:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:56.861 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":138,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:15:54.321537+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6835/2095975197":"2023-11-25T08:15:50.849657+0000","172.21.15.120:6834/2095975197":"2023-11-25T08:15:50.849657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 msgr2=0x7f604c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 0x7f604c06e7d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f605c003cc0 tx=0x7f605c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 msgr2=0x7f606c09cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f6064036040 tx=0x7f6064004a40 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 shutdown_connections 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f604c06c310 0x7f604c06e7d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f606c15e390 0x7f606c09cb50 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c157c70 0x7f606c09c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 --2- 172.21.15.107:0/2993605443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c0cfb60 0x7f606c071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 >> 172.21.15.107:0/2993605443 conn(0x7f606c06a920 msgr2=0x7f606c06c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:56.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 shutdown_connections 2023-11-24T08:15:56.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:56.796+0000 7f6071a3d640 1 -- 172.21.15.107:0/2993605443 wait complete. 2023-11-24T08:15:56.864 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:15:56.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:15:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.087+0000 7f19378ad640 1 Processor -- start 2023-11-24T08:15:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.087+0000 7f19378ad640 1 -- start start 2023-11-24T08:15:57.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.088+0000 7f19378ad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f193006de10 0x7f193015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.088+0000 7f19378ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f19300705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.088+0000 7f19378ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193015ec90 con 0x7f1930072160 2023-11-24T08:15:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.088+0000 7f19378ad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19300cd530 con 0x7f193006de10 2023-11-24T08:15:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.088+0000 7f19378ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300c4ed0 con 0x7f19300c49f0 2023-11-24T08:15:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1935649640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1930072160 0x7f193006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39264/0 (socket says 172.21.15.107:39264) 2023-11-24T08:15:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1935649640 1 -- 172.21.15.107:0/3487811031 learned_addr learned my addr 172.21.15.107:0/3487811031 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:57.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f19300705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1934e48640 1 --2- 172.21.15.107:0/3487811031 >> v2:172.21.15.120:3300/0 conn(0x7f193006de10 0x7f193015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578463411 0 0) 0x7f192c001110 con 0x7f1930072160 2023-11-24T08:15:57.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c003520 con 0x7f1930072160 2023-11-24T08:15:57.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.089+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2256400669 0 0) 0x7f191c003520 con 0x7f1930072160 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 msgr2=0x7f19300705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f19300705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 >> v2:172.21.15.120:3300/0 conn(0x7f193006de10 msgr2=0x7f193015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 --2- 172.21.15.107:0/3487811031 >> v2:172.21.15.120:3300/0 conn(0x7f193006de10 0x7f193015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300c51b0 con 0x7f1930072160 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f192c001800 con 0x7f1930072160 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 >> v1:172.21.15.107:6789/0 conn(0x7f1930072160 legacy=0x7f193006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f191c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c003f40 0x7f191c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f191c004d00 0x7f191c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cd530 con 0x7f191c004d00 2023-11-24T08:15:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300c4ed0 con 0x7f191c003f40 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c00b980 con 0x7f19300c49f0 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f191c004d00 0x7f191c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 -- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 msgr2=0x7f191c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1934e48640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c003f40 0x7f191c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935649640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f191c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f191c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 -- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c003f40 msgr2=0x7f191c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c003f40 0x7f191c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1935e4a640 1 -- 172.21.15.107:0/3487811031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c003520 con 0x7f191c004d00 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.090+0000 7f1934e48640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c003f40 0x7f191c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.091+0000 7f1935649640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19300c49f0 0x7f191c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f191c004d00 0x7f191c005180 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f1928009f90 tx=0x7f191c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44deb69de00c550 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192802cb90 con 0x7f191c004d00 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192800f040 con 0x7f191c004d00 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f1927fff640 1 -- 172.21.15.107:0/3487811031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928034440 con 0x7f191c004d00 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f19378ad640 1 -- 172.21.15.107:0/3487811031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f191c004d00 msgr2=0x7f191c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f19378ad640 1 --2- 172.21.15.107:0/3487811031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f191c004d00 0x7f191c005180 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f1928009f90 tx=0x7f191c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f1935e4a640 1 -- 172.21.15.107:0/3487811031 reap_dead start 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f19378ad640 1 -- 172.21.15.107:0/3487811031 shutdown_connections 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.092+0000 7f19378ad640 1 -- 172.21.15.107:0/3487811031 >> 172.21.15.107:0/3487811031 conn(0x7f19300b0c50 msgr2=0x7f19300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.093+0000 7f19378ad640 1 -- 172.21.15.107:0/3487811031 shutdown_connections 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.093+0000 7f19378ad640 1 -- 172.21.15.107:0/3487811031 wait complete. 2023-11-24T08:15:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.094+0000 7f19378ad640 1 Processor -- start 2023-11-24T08:15:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.094+0000 7f19378ad640 1 -- start start 2023-11-24T08:15:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 0x7f193014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 0x7f193009aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300af8c0 con 0x7f19300ae2b0 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300ae820 con 0x7f191c004d00 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.095+0000 7f19378ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300aeb00 con 0x7f193009a650 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51444/0 (socket says 172.21.15.107:51444) 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 -- 172.21.15.107:0/3365815677 learned_addr learned my addr 172.21.15.107:0/3365815677 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1935649640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 0x7f193014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 0x7f193009aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 -- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 msgr2=0x7f193009aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 0x7f193009aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 -- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 msgr2=0x7f193014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 0x7f193014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 -- 172.21.15.107:0/3365815677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300aede0 con 0x7f19300ae2b0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1935649640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 0x7f193014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1935e4a640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 0x7f193009aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.096+0000 7f1934e48640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f19200109d0 tx=0x7f1920010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.097+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192000ec20 con 0x7f19300ae2b0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.097+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192000ed80 con 0x7f19300ae2b0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.097+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192001c620 con 0x7f19300ae2b0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.097+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19300af0c0 con 0x7f19300ae2b0 2023-11-24T08:15:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.097+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193014e780 con 0x7f19300ae2b0 2023-11-24T08:15:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.098+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193004e8b0 con 0x7f19300ae2b0 2023-11-24T08:15:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.099+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f192001c780 con 0x7f19300ae2b0 2023-11-24T08:15:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.100+0000 7f1925ffb640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 0x7f191406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.100+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1920088da0 con 0x7f19300ae2b0 2023-11-24T08:15:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.100+0000 7f1935649640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 0x7f191406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.101+0000 7f1935649640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 0x7f191406e800 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f192c001930 tx=0x7f192c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.104+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1920055570 con 0x7f19300ae2b0 2023-11-24T08:15:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.294+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f193015dab0 con 0x7f19300ae2b0 2023-11-24T08:15:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.295+0000 7f1925ffb640 1 -- 172.21.15.107:0/3365815677 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1920058b90 con 0x7f19300ae2b0 2023-11-24T08:15:57.297 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:15:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 msgr2=0x7f191406e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 0x7f191406e800 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f192c001930 tx=0x7f192c009040 comp rx=0 tx=0).stop 2023-11-24T08:15:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 msgr2=0x7f193014c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f19200109d0 tx=0x7f1920010e90 comp rx=0 tx=0).stop 2023-11-24T08:15:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 shutdown_connections 2023-11-24T08:15:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f191406c340 0x7f191406e800 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193009a650 0x7f193009aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19300ae2b0 0x7f193014c9c0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 --2- 172.21.15.107:0/3365815677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f191c004d00 0x7f193014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 >> 172.21.15.107:0/3365815677 conn(0x7f19300b0c50 msgr2=0x7f19300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 shutdown_connections 2023-11-24T08:15:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.298+0000 7f19378ad640 1 -- 172.21.15.107:0/3365815677 wait complete. 2023-11-24T08:15:57.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm' 2023-11-24T08:15:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.520+0000 7f4fdc98a640 1 Processor -- start 2023-11-24T08:15:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.520+0000 7f4fdc98a640 1 -- start start 2023-11-24T08:15:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fdc98a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4fd8144b30 0x7f4fd815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fdc98a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd80b1de0 0x7f4fd80b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fdc98a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd815bf80 con 0x7f4fd815c410 2023-11-24T08:15:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fdc98a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd80b27b0 con 0x7f4fd8144b30 2023-11-24T08:15:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fdc98a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd80b1560 con 0x7f4fd80b1de0 2023-11-24T08:15:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fd77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4fd8144b30 0x7f4fd815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fceffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4fd815c410 0x7f4fd80b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39274/0 (socket says 172.21.15.107:39274) 2023-11-24T08:15:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.522+0000 7f4fceffd640 1 -- 172.21.15.107:0/2586240409 learned_addr learned my addr 172.21.15.107:0/2586240409 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.523+0000 7f4fd7fff640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd80b1de0 0x7f4fd80b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.523+0000 7f4fd77fe640 1 -- 172.21.15.107:0/2586240409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd80b1de0 msgr2=0x7f4fd80b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.523+0000 7f4fd77fe640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd80b1de0 0x7f4fd80b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.523+0000 7f4fd77fe640 1 -- 172.21.15.107:0/2586240409 >> v1:172.21.15.107:6789/0 conn(0x7f4fd815c410 legacy=0x7f4fd80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.523+0000 7f4fd77fe640 1 -- 172.21.15.107:0/2586240409 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd80b1280 con 0x7f4fd8144b30 2023-11-24T08:15:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.524+0000 7f4fd77fe640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd8144b30 0x7f4fd815f920 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009d90 tx=0x7f4fc802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c602c411a724fd28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.524+0000 7f4fd6ffd640 1 -- 172.21.15.107:0/2586240409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc802cc50 con 0x7f4fd8144b30 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.524+0000 7f4fd6ffd640 1 -- 172.21.15.107:0/2586240409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc802cdb0 con 0x7f4fd8144b30 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.524+0000 7f4fd6ffd640 1 -- 172.21.15.107:0/2586240409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8034660 con 0x7f4fd8144b30 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2586240409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd8144b30 msgr2=0x7f4fd815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd8144b30 0x7f4fd815f920 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009d90 tx=0x7f4fc802bea0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2586240409 shutdown_connections 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd80b1de0 0x7f4fd80b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2586240409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd8144b30 0x7f4fd815f920 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2586240409 >> 172.21.15.107:0/2586240409 conn(0x7f4fd80cb830 msgr2=0x7f4fd80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2586240409 shutdown_connections 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.525+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2586240409 wait complete. 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.527+0000 7f4fdc98a640 1 Processor -- start 2023-11-24T08:15:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.527+0000 7f4fdc98a640 1 -- start start 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fd80b1de0 0x7f4fd8140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 0x7f4fd80cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd80713b0 con 0x7f4fd815c410 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd8071690 con 0x7f4fd80b1de0 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fdc98a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd8071970 con 0x7f4fd8144b30 2023-11-24T08:15:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51448/0 (socket says 172.21.15.107:51448) 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 -- 172.21.15.107:0/2635010241 learned_addr learned my addr 172.21.15.107:0/2635010241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fceffd640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 0x7f4fd80cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 -- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 msgr2=0x7f4fd80cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 0x7f4fd80cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd77fe640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fd80b1de0 0x7f4fd8140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 -- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fd80b1de0 msgr2=0x7f4fd8140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.528+0000 7f4fd7fff640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fd80b1de0 0x7f4fd8140680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fd7fff640 1 -- 172.21.15.107:0/2635010241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8009a70 con 0x7f4fd815c410 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fceffd640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 0x7f4fd80cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fd7fff640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c7f0 tx=0x7f4fd000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0004220 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0004380 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd8071c50 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0007860 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.529+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd8140bc0 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.530+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd80b3610 con 0x7f4fd815c410 2023-11-24T08:15:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.531+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f4fd0007bd0 con 0x7f4fd815c410 2023-11-24T08:15:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.532+0000 7f4fd4ff9640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 0x7f4fb806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.532+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4fd0088600 con 0x7f4fd815c410 2023-11-24T08:15:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.532+0000 7f4fd77fe640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 0x7f4fb806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.533+0000 7f4fd77fe640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 0x7f4fb806e7d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8036000 tx=0x7f4fc8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.536+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fd80b3610 con 0x7f4fd815c410 2023-11-24T08:15:57.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:57.795+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm"]} v 0) v1 -- 0x7f4fd80b3610 con 0x7f4fd815c410 2023-11-24T08:15:58.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.360+0000 7f4fd4ff9640 1 -- 172.21.15.107:0/2635010241 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm"]}]=0 v567) v1 ==== 180+0+0 (secure 0 0 0) 0x7f4fd80b3610 con 0x7f4fd815c410 2023-11-24T08:15:58.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 msgr2=0x7f4fb806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 0x7f4fb806e7d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8036000 tx=0x7f4fc8030000 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 msgr2=0x7f4fd80cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c7f0 tx=0x7f4fd000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 shutdown_connections 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4fb806c310 0x7f4fb806e7d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fd815c410 0x7f4fd80cdd90 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fd8144b30 0x7f4fd80cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 --2- 172.21.15.107:0/2635010241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fd80b1de0 0x7f4fd8140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.363+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 >> 172.21.15.107:0/2635010241 conn(0x7f4fd80cb830 msgr2=0x7f4fd8142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.364+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 shutdown_connections 2023-11-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.364+0000 7f4fdc98a640 1 -- 172.21.15.107:0/2635010241 wait complete. 2023-11-24T08:15:58.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:15:58.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.599+0000 7fd29629b640 1 Processor -- start 2023-11-24T08:15:58.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.600+0000 7fd29629b640 1 -- start start 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd29629b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd29015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd29629b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29015ec90 con 0x7fd29006de10 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd29629b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900cd530 con 0x7fd2900c49f0 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd29629b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c4ed0 con 0x7fd290072160 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd294838640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd29006de10 0x7fd2900705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53658/0 (socket says 172.21.15.107:53658) 2023-11-24T08:15:58.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.601+0000 7fd294838640 1 -- 172.21.15.107:0/3017221948 learned_addr learned my addr 172.21.15.107:0/3017221948 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28ffff640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd29015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934107185 0 0) 0x7fd288001110 con 0x7fd29006de10 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd27c003520 con 0x7fd29006de10 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2191850623 0 0) 0x7fd27c003520 con 0x7fd29006de10 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 msgr2=0x7fd29015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd29015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 >> v1:172.21.15.120:6789/0 conn(0x7fd2900c49f0 legacy=0x7fd29006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900c51b0 con 0x7fd29006de10 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.602+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd288001800 con 0x7fd29006de10 2023-11-24T08:15:58.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 >> v1:172.21.15.107:6789/0 conn(0x7fd29006de10 legacy=0x7fd2900705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd27c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd27c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd27c004d00 0x7fd27c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c4ed0 con 0x7fd27c003f20 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c00ba60 con 0x7fd27c004d00 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c00bc20 con 0x7fd290072160 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd27c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd28ffff640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd27c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd294838640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd27c004d00 0x7fd27c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 -- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 msgr2=0x7fd27c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd290072160 0x7fd27c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 -- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd27c004d00 msgr2=0x7fd27c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:58.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd27c004d00 0x7fd27c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.603+0000 7fd2877fe640 1 -- 172.21.15.107:0/3017221948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c003520 con 0x7fd27c003f20 2023-11-24T08:15:58.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.604+0000 7fd294838640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd27c004d00 0x7fd27c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd2877fe640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd27c0047c0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fd278001750 tx=0x7fd27c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e115ac1f2b5aca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2780052d0 con 0x7fd27c003f20 2023-11-24T08:15:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27802e050 con 0x7fd27c003f20 2023-11-24T08:15:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd28f7fe640 1 -- 172.21.15.107:0/3017221948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278003830 con 0x7fd27c003f20 2023-11-24T08:15:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd29629b640 1 -- 172.21.15.107:0/3017221948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 msgr2=0x7fd27c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.605+0000 7fd29629b640 1 --2- 172.21.15.107:0/3017221948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd27c0047c0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fd278001750 tx=0x7fd27c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.606+0000 7fd294838640 1 -- 172.21.15.107:0/3017221948 reap_dead start 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.606+0000 7fd29629b640 1 -- 172.21.15.107:0/3017221948 shutdown_connections 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.606+0000 7fd29629b640 1 -- 172.21.15.107:0/3017221948 >> 172.21.15.107:0/3017221948 conn(0x7fd2900b0c50 msgr2=0x7fd2900b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.606+0000 7fd29629b640 1 -- 172.21.15.107:0/3017221948 shutdown_connections 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.606+0000 7fd29629b640 1 -- 172.21.15.107:0/3017221948 wait complete. 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.607+0000 7fd29629b640 1 Processor -- start 2023-11-24T08:15:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.608+0000 7fd29629b640 1 -- start start 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.608+0000 7fd29629b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.608+0000 7fd29629b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd29015a710 0x7fd2900cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd29629b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd290147ae0 0x7fd290147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd29629b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900cb280 con 0x7fd27c003f20 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd29629b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29009a2c0 con 0x7fd290147ae0 2023-11-24T08:15:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd29629b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2901484a0 con 0x7fd29015a710 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39588/0 (socket says 172.21.15.107:39588) 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 -- 172.21.15.107:0/637849864 learned_addr learned my addr 172.21.15.107:0/637849864 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 -- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd29015a710 msgr2=0x7fd2900cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd29015a710 0x7fd2900cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 -- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd290147ae0 msgr2=0x7fd290147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd290147ae0 0x7fd290147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.609+0000 7fd28ffff640 1 -- 172.21.15.107:0/637849864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29015ab90 con 0x7fd27c003f20 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd28ffff640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd280007f30 tx=0x7fd280011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28000e030 con 0x7fd27c003f20 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd280016470 con 0x7fd27c003f20 2023-11-24T08:15:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2900cbbe0 con 0x7fd27c003f20 2023-11-24T08:15:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280016820 con 0x7fd27c003f20 2023-11-24T08:15:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.610+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd290099fe0 con 0x7fd27c003f20 2023-11-24T08:15:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.611+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2900cb4f0 con 0x7fd27c003f20 2023-11-24T08:15:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.612+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fd280016980 con 0x7fd27c003f20 2023-11-24T08:15:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.613+0000 7fd28d7fa640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 0x7fd27006e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.613+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd28000f070 con 0x7fd27c003f20 2023-11-24T08:15:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.613+0000 7fd2877fe640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 0x7fd27006e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.614+0000 7fd2877fe640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 0x7fd27006e760 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd278033000 tx=0x7fd27802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.617+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd280055e20 con 0x7fd27c003f20 2023-11-24T08:15:58.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.809+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd29004e8b0 con 0x7fd27c003f20 2023-11-24T08:15:58.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.815+0000 7fd28d7fa640 1 -- 172.21.15.107:0/637849864 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v39)=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd280059440 con 0x7fd27c003f20 2023-11-24T08:15:58.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 msgr2=0x7fd27006e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 0x7fd27006e760 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd278033000 tx=0x7fd27802f000 comp rx=0 tx=0).stop 2023-11-24T08:15:58.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 msgr2=0x7fd290145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:58.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd280007f30 tx=0x7fd280011660 comp rx=0 tx=0).stop 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 shutdown_connections 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd27006c2a0 0x7fd27006e760 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd290147ae0 0x7fd290147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd29015a710 0x7fd2900cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 --2- 172.21.15.107:0/637849864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27c003f20 0x7fd290145400 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 >> 172.21.15.107:0/637849864 conn(0x7fd2900b0c50 msgr2=0x7fd2900ac550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 shutdown_connections 2023-11-24T08:15:58.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:58.818+0000 7fd29629b640 1 -- 172.21.15.107:0/637849864 wait complete. 2023-11-24T08:15:58.839 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:15:58.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:15:58.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:15:59.075 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:15:59.075 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:15:59.075 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:59.135 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:15:59.135 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:15:59.135 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:59.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:15:59.195 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:15:59.195 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:59.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:15:59.251 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:15:59.251 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:15:59.310 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:15:59.310 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:15:59.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:15:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.532+0000 7f5bbc5a9640 1 Processor -- start 2023-11-24T08:15:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.532+0000 7f5bbc5a9640 1 -- start start 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bbc5a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5bb415df80 0x7f5bb40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bbc5a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 0x7f5bb415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bbc5a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb415e3f0 con 0x7f5bb40b2250 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bbc5a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb415a990 con 0x7f5bb415df80 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bbc5a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb415ab80 con 0x7f5bb40cd470 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bba345640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5bb40b2250 0x7f5bb40b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53668/0 (socket says 172.21.15.107:53668) 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bba345640 1 -- 172.21.15.107:0/3654985201 learned_addr learned my addr 172.21.15.107:0/3654985201 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:15:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.534+0000 7f5bb9b44640 1 --2- 172.21.15.107:0/3654985201 >> v2:172.21.15.107:3300/0 conn(0x7f5bb415df80 0x7f5bb40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bbab46640 1 --2- 172.21.15.107:0/3654985201 >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 0x7f5bb415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bb9b44640 1 -- 172.21.15.107:0/3654985201 >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 msgr2=0x7f5bb415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bb9b44640 1 --2- 172.21.15.107:0/3654985201 >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 0x7f5bb415b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bb9b44640 1 -- 172.21.15.107:0/3654985201 >> v1:172.21.15.107:6789/0 conn(0x7f5bb40b2250 legacy=0x7f5bb40b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bb9b44640 1 -- 172.21.15.107:0/3654985201 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40b2d40 con 0x7f5bb415df80 2023-11-24T08:15:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.535+0000 7f5bbab46640 1 --2- 172.21.15.107:0/3654985201 >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 0x7f5bb415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.536+0000 7f5bb9b44640 1 --2- 172.21.15.107:0/3654985201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb40ccf30 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40b2fa0 tx=0x7f5ba002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792637d893a1b298 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.536+0000 7f5bb9343640 1 -- 172.21.15.107:0/3654985201 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0031830 con 0x7f5bb415df80 2023-11-24T08:15:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.536+0000 7f5bb9343640 1 -- 172.21.15.107:0/3654985201 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba0031990 con 0x7f5bb415df80 2023-11-24T08:15:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.536+0000 7f5bb9343640 1 -- 172.21.15.107:0/3654985201 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0031d20 con 0x7f5bb415df80 2023-11-24T08:15:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/3654985201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 msgr2=0x7f5bb40ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/3654985201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb40ccf30 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40b2fa0 tx=0x7f5ba002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/3654985201 shutdown_connections 2023-11-24T08:15:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/3654985201 >> v2:172.21.15.120:3300/0 conn(0x7f5bb40cd470 0x7f5bb415b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/3654985201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb40ccf30 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/3654985201 >> 172.21.15.107:0/3654985201 conn(0x7f5bb406a920 msgr2=0x7f5bb406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/3654985201 shutdown_connections 2023-11-24T08:15:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.537+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/3654985201 wait complete. 2023-11-24T08:15:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.539+0000 7f5bbc5a9640 1 Processor -- start 2023-11-24T08:15:59.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.539+0000 7f5bbc5a9640 1 -- start start 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 0x7f5bb40ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bb40cd470 0x7f5bb4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40b1a00 con 0x7f5bb415df80 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4156a70 con 0x7f5bb40b2250 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbc5a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4156d50 con 0x7f5bb40cd470 2023-11-24T08:15:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbab46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbab46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39608/0 (socket says 172.21.15.107:39608) 2023-11-24T08:15:59.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.540+0000 7f5bbab46640 1 -- 172.21.15.107:0/183832019 learned_addr learned my addr 172.21.15.107:0/183832019 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:15:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bba345640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 0x7f5bb40ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bb9b44640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bb40cd470 0x7f5bb4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 -- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bb40cd470 msgr2=0x7f5bb4157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bb40cd470 0x7f5bb4157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 -- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 msgr2=0x7f5bb40ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 0x7f5bb40ae060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 -- 172.21.15.107:0/183832019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0009400 con 0x7f5bb415df80 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bba345640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 0x7f5bb40ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.541+0000 7f5bbab46640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c5e0 tx=0x7f5bac00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.542+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac011800 con 0x7f5bb415df80 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.542+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac0119a0 con 0x7f5bb415df80 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.542+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb4157030 con 0x7f5bb415df80 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.542+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac011d50 con 0x7f5bb415df80 2023-11-24T08:15:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.542+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb4157d70 con 0x7f5bb415df80 2023-11-24T08:15:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.543+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb4157550 con 0x7f5bb415df80 2023-11-24T08:15:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.544+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f5bac022350 con 0x7f5bb415df80 2023-11-24T08:15:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.545+0000 7f5bb2ffd640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 0x7f5b9806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:15:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.545+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5bac0884a0 con 0x7f5bb415df80 2023-11-24T08:15:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.545+0000 7f5bba345640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 0x7f5b9806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:15:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.546+0000 7f5bba345640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 0x7f5b9806e7d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8005c00 tx=0x7f5ba8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:15:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.549+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bac0bee90 con 0x7f5bb415df80 2023-11-24T08:15:59.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.775+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb4067470 con 0x7f5bb415df80 2023-11-24T08:15:59.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.777+0000 7f5bb2ffd640 1 -- 172.21.15.107:0/183832019 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f5bac01f070 con 0x7f5bb415df80 2023-11-24T08:15:59.778 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:15:59.778 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3655774441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3655774441},{"type":"v1","addr":"172.21.15.120:6835","nonce":3655774441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":6340,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2534333110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2534333110},{"type":"v1","addr":"172.21.15.120:6833","nonce":2534333110}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4218513610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4218513610},{"type":"v1","addr":"172.21.15.107:6835","nonce":4218513610}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2422039284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2422039284},{"type":"v1","addr":"172.21.15.107:6837","nonce":2422039284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[]} 2023-11-24T08:15:59.778 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 81 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.779+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 msgr2=0x7f5b9806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.779+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 0x7f5b9806e7d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8005c00 tx=0x7f5ba8009040 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.779+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 msgr2=0x7f5bb414de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.779+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c5e0 tx=0x7f5bac00caa0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 shutdown_connections 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5b9806c310 0x7f5b9806e7d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb415df80 0x7f5bb414de20 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bb40cd470 0x7f5bb4157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 --2- 172.21.15.107:0/183832019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bb40b2250 0x7f5bb40ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 >> 172.21.15.107:0/183832019 conn(0x7f5bb406a920 msgr2=0x7f5bb4159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:15:59.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 shutdown_connections 2023-11-24T08:15:59.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:15:59.780+0000 7f5bbc5a9640 1 -- 172.21.15.107:0/183832019 wait complete. 2023-11-24T08:15:59.800 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:15:59.800 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:15:59.800 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:15:59.800 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:15:59.801 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:15:59.801 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:15:59.801 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:15:59.801 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:15:59.801 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:15:59.801 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:15:59.801 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:15:59.802 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:15:59.802 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:15:59.802 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:15:59.802 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:15:59.802 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:15:59.816 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:15:59.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:15:59.823 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:15:59.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:15:59.824 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:15:59.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:15:59.827 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:15:59.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:15:59.871 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:15:59.871 DEBUG:teuthology.parallel:result is None 2023-11-24T08:15:59.875 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:15:59.875 DEBUG:teuthology.parallel:result is None 2023-11-24T08:15:59.878 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:15:59.878 DEBUG:teuthology.parallel:result is None 2023-11-24T08:15:59.882 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:15:59.882 DEBUG:teuthology.parallel:result is None 2023-11-24T08:15:59.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:16:00.024 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:16:00.024 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:16:00.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-11-24T08:16:00.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-11-24T08:16:00.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-24T08:16:00.039 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:16:00.041 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:16:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.061+0000 7f8bdf294640 1 Processor -- start 2023-11-24T08:16:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.061+0000 7f8bdf294640 1 -- start start 2023-11-24T08:16:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8bd815b440 0x7f8bd80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80cbf20 0x7f8bd80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8bd80c57e0 0x7f8bd80c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd80cc3d0 con 0x7f8bd815b440 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd815b820 con 0x7f8bd80c57e0 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdf294640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd815eea0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdd030640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8bd815b440 0x7f8bd80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdd030640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8bd815b440 0x7f8bd80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39650/0 (socket says 172.21.15.107:39650) 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdd030640 1 -- 172.21.15.107:0/3305275604 learned_addr learned my addr 172.21.15.107:0/3305275604 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.063+0000 7f8bdc82f640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80cbf20 0x7f8bd80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd831640 1 --2- 172.21.15.107:0/3305275604 >> v2:172.21.15.120:3300/0 conn(0x7f8bd80c57e0 0x7f8bd80c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd030640 1 -- 172.21.15.107:0/3305275604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80cbf20 msgr2=0x7f8bd80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd030640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80cbf20 0x7f8bd80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd030640 1 -- 172.21.15.107:0/3305275604 >> v2:172.21.15.120:3300/0 conn(0x7f8bd80c57e0 msgr2=0x7f8bd80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd030640 1 --2- 172.21.15.107:0/3305275604 >> v2:172.21.15.120:3300/0 conn(0x7f8bd80c57e0 0x7f8bd80c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.064+0000 7f8bdd030640 1 -- 172.21.15.107:0/3305275604 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd80abf30 con 0x7f8bd815b440 2023-11-24T08:16:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.065+0000 7f8bdd030640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd815b440 0x7f8bd80cb9e0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80cb910 tx=0x7f8bd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7518bd418200f9ca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.065+0000 7f8bcffff640 1 -- 172.21.15.107:0/3305275604 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40317e0 con 0x7f8bd815b440 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.065+0000 7f8bcffff640 1 -- 172.21.15.107:0/3305275604 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd4031940 con 0x7f8bd815b440 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.065+0000 7f8bcffff640 1 -- 172.21.15.107:0/3305275604 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4031cd0 con 0x7f8bd815b440 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 -- 172.21.15.107:0/3305275604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd815b440 msgr2=0x7f8bd80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd815b440 0x7f8bd80cb9e0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80cb910 tx=0x7f8bd402b880 comp rx=0 tx=0).stop 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 -- 172.21.15.107:0/3305275604 shutdown_connections 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 --2- 172.21.15.107:0/3305275604 >> v2:172.21.15.120:3300/0 conn(0x7f8bd80c57e0 0x7f8bd80c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80cbf20 0x7f8bd80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 --2- 172.21.15.107:0/3305275604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd815b440 0x7f8bd80cb9e0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 -- 172.21.15.107:0/3305275604 >> 172.21.15.107:0/3305275604 conn(0x7f8bd80b0c50 msgr2=0x7f8bd80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 -- 172.21.15.107:0/3305275604 shutdown_connections 2023-11-24T08:16:00.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.066+0000 7f8bdf294640 1 -- 172.21.15.107:0/3305275604 wait complete. 2023-11-24T08:16:00.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.067+0000 7f8bdf294640 1 Processor -- start 2023-11-24T08:16:00.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.068+0000 7f8bdf294640 1 -- start start 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.068+0000 7f8bdf294640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80c57e0 0x7f8bd815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.068+0000 7f8bdf294640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.068+0000 7f8bdf294640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd815b440 0x7f8bd80b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdf294640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8156aa0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdf294640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80c6340 con 0x7f8bd815b440 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdf294640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd815cac0 con 0x7f8bd80c57e0 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39666/0 (socket says 172.21.15.107:39666) 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 -- 172.21.15.107:0/793974589 learned_addr learned my addr 172.21.15.107:0/793974589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdd030640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80c57e0 0x7f8bd815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdd831640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd815b440 0x7f8bd80b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 -- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80c57e0 msgr2=0x7f8bd815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80c57e0 0x7f8bd815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 -- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd815b440 msgr2=0x7f8bd80b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd815b440 0x7f8bd80b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 -- 172.21.15.107:0/793974589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40094d0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.069+0000 7f8bdc82f640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e9b0 tx=0x7f8bc800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.070+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8009670 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.070+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc800c9a0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.070+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800cd50 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.070+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd815bf40 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.070+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd815c500 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.071+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd804e8b0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.072+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f8bc8013470 con 0x7f8bd80cbf20 2023-11-24T08:16:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.073+0000 7f8bcdffb640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 0x7f8bbc06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.073+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8bc8088430 con 0x7f8bd80cbf20 2023-11-24T08:16:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.073+0000 7f8bdd030640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 0x7f8bbc06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.074+0000 7f8bdd030640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 0x7f8bbc06e800 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036000 tx=0x7f8bd403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:00.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.078+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bc8054fe0 con 0x7f8bd80cbf20 2023-11-24T08:16:00.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:00.276+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8bd80cb740 con 0x7f8bd80cbf20 2023-11-24T08:16:01.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.404+0000 7f8bcdffb640 1 -- 172.21.15.107:0/793974589 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v143) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8bc8021080 con 0x7f8bd80cbf20 2023-11-24T08:16:01.405 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 msgr2=0x7f8bbc06e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 0x7f8bbc06e800 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036000 tx=0x7f8bd403d040 comp rx=0 tx=0).stop 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 msgr2=0x7f8bd80c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e9b0 tx=0x7f8bc800ee70 comp rx=0 tx=0).stop 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 shutdown_connections 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8bbc06c340 0x7f8bbc06e800 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd815b440 0x7f8bd80b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd80cbf20 0x7f8bd80c8ff0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 --2- 172.21.15.107:0/793974589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd80c57e0 0x7f8bd815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 >> 172.21.15.107:0/793974589 conn(0x7f8bd80b0c50 msgr2=0x7f8bd8071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 shutdown_connections 2023-11-24T08:16:01.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.408+0000 7f8bdf294640 1 -- 172.21.15.107:0/793974589 wait complete. 2023-11-24T08:16:01.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:16:01.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.640+0000 7f4ea6b4e640 1 Processor -- start 2023-11-24T08:16:01.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.641+0000 7f4ea6b4e640 1 -- start start 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea6b4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 0x7f4ea00b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea6b4e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4ea00b2570 0x7f4ea006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea6b4e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea0160fb0 con 0x7f4ea0154500 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea6b4e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea01548e0 con 0x7f4ea00b2570 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea6b4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea015dec0 con 0x7f4ea015d9c0 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea5b4c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4ea0154500 0x7f4ea00b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53672/0 (socket says 172.21.15.107:53672) 2023-11-24T08:16:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.642+0000 7f4ea5b4c640 1 -- 172.21.15.107:0/2384484849 learned_addr learned my addr 172.21.15.107:0/2384484849 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea634d640 1 --2- 172.21.15.107:0/2384484849 >> v2:172.21.15.120:3300/0 conn(0x7f4ea00b2570 0x7f4ea006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055971271 0 0) 0x7f4ea0160fb0 con 0x7f4ea0154500 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea534b640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 0x7f4ea00b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c003520 con 0x7f4ea0154500 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3411803531 0 0) 0x7f4e8c003520 con 0x7f4ea0154500 2023-11-24T08:16:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 msgr2=0x7f4ea00b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 0x7f4ea00b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 >> v2:172.21.15.120:3300/0 conn(0x7f4ea00b2570 msgr2=0x7f4ea006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 --2- 172.21.15.107:0/2384484849 >> v2:172.21.15.120:3300/0 conn(0x7f4ea00b2570 0x7f4ea006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea015e1a0 con 0x7f4ea0154500 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4e9c001670 con 0x7f4ea0154500 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 >> v1:172.21.15.107:6789/0 conn(0x7f4ea0154500 legacy=0x7f4ea00b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 0x7f4e8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea01548e0 con 0x7f4e8c003970 2023-11-24T08:16:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea015dec0 con 0x7f4e8c004d00 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.643+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c00b980 con 0x7f4ea015d9c0 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 -- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 msgr2=0x7f4e8c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea015d9c0 0x7f4e8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 -- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c004d00 msgr2=0x7f4e8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c004d00 0x7f4e8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.644+0000 7f4ea5b4c640 1 -- 172.21.15.107:0/2384484849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c003520 con 0x7f4e8c003970 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.645+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00c240 tx=0x7f4e8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0e72666cbaf54a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.645+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c02cb00 con 0x7f4e8c003970 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.645+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c02cc60 con 0x7f4e8c003970 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.645+0000 7f4ea4b4a640 1 -- 172.21.15.107:0/2384484849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c005af0 con 0x7f4e8c003970 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2384484849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c003970 msgr2=0x7f4e8c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2384484849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e8c003970 0x7f4e8c003d90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00c240 tx=0x7f4e8c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea634d640 1 -- 172.21.15.107:0/2384484849 reap_dead start 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2384484849 shutdown_connections 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2384484849 >> 172.21.15.107:0/2384484849 conn(0x7f4ea00b4630 msgr2=0x7f4ea00b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2384484849 shutdown_connections 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.646+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2384484849 wait complete. 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.648+0000 7f4ea6b4e640 1 Processor -- start 2023-11-24T08:16:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.648+0000 7f4ea6b4e640 1 -- start start 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 0x7f4ea0151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 0x7f4ea01515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea0157e00 con 0x7f4ea0156ae0 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea01f2c40 con 0x7f4ea013e310 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea6b4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea0157840 con 0x7f4e8c003970 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39686/0 (socket says 172.21.15.107:39686) 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 -- 172.21.15.107:0/2762011029 learned_addr learned my addr 172.21.15.107:0/2762011029 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea534b640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 0x7f4ea01515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 0x7f4ea0151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 -- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 msgr2=0x7f4ea0151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 0x7f4ea0151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 -- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 msgr2=0x7f4ea01515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 0x7f4ea01515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea634d640 1 -- 172.21.15.107:0/2762011029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea0157b20 con 0x7f4ea0156ae0 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea534b640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 0x7f4ea01515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.649+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 0x7f4ea0151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4ea634d640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cbd50 tx=0x7f4e9800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98010040 con 0x7f4ea0156ae0 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9800bae0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9800be70 con 0x7f4ea0156ae0 2023-11-24T08:16:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea013e9b0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.650+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea01574a0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.651+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea013eed0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.652+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f4e9801e410 con 0x7f4ea0156ae0 2023-11-24T08:16:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.653+0000 7f4e967fc640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 0x7f4e8406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.653+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e98088520 con 0x7f4ea0156ae0 2023-11-24T08:16:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.654+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 0x7f4e8406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.654+0000 7f4ea5b4c640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 0x7f4e8406e7d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c034000 tx=0x7f4e9c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:01.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.657+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ea013eed0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.916+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4ea013eed0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.917+0000 7f4e967fc640 1 -- 172.21.15.107:0/2762011029 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4ea013eed0 con 0x7f4ea0156ae0 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:16:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:16:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:16:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:16:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:16:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:16:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:16:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:16:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:16:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:16:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:16:01.942 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 msgr2=0x7f4e8406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 0x7f4e8406e7d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c034000 tx=0x7f4e9c007000 comp rx=0 tx=0).stop 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 msgr2=0x7f4ea0156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cbd50 tx=0x7f4e9800b480 comp rx=0 tx=0).stop 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 shutdown_connections 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e8406c310 0x7f4e8406e7d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea0156ae0 0x7f4ea0156f60 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea013e310 0x7f4ea01515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 --2- 172.21.15.107:0/2762011029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c003970 0x7f4ea0151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.919+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 >> 172.21.15.107:0/2762011029 conn(0x7f4ea00b4630 msgr2=0x7f4ea00ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.920+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 shutdown_connections 2023-11-24T08:16:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:01.920+0000 7f4ea6b4e640 1 -- 172.21.15.107:0/2762011029 wait complete. 2023-11-24T08:16:01.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:16:02.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.153+0000 7f5401916640 1 Processor -- start 2023-11-24T08:16:02.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.153+0000 7f5401916640 1 -- start start 2023-11-24T08:16:02.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401916640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53fc15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401916640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53fc15bf80 con 0x7f53fc0b1de0 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401916640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53fc0b1560 con 0x7f53fc15c410 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401916640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc0b1280 con 0x7f53fc144b30 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401115640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53fc0b1de0 0x7f53fc0b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53674/0 (socket says 172.21.15.107:53674) 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5401115640 1 -- 172.21.15.107:0/4005069741 learned_addr learned my addr 172.21.15.107:0/4005069741 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.155+0000 7f5400914640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53fc15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448636083 0 0) 0x7f53fc15bf80 con 0x7f53fc0b1de0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ec003520 con 0x7f53fc0b1de0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183937199 0 0) 0x7f53ec003520 con 0x7f53fc0b1de0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 msgr2=0x7f53fc15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53fc15f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 >> v1:172.21.15.120:6789/0 conn(0x7f53fc15c410 legacy=0x7f53fc0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc15b930 con 0x7f53fc0b1de0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f53f40019f0 con 0x7f53fc0b1de0 2023-11-24T08:16:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 >> v1:172.21.15.107:6789/0 conn(0x7f53fc0b1de0 legacy=0x7f53fc0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53ec004d00 0x7f53ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc0b1280 con 0x7f53ec003f70 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00bac0 con 0x7f53ec004d00 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00bc80 con 0x7f53fc144b30 2023-11-24T08:16:02.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.156+0000 7f53fbfff640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f5400914640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f53fbfff640 1 -- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 msgr2=0x7f53ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f53fbfff640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc144b30 0x7f53ec0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f53fbfff640 1 -- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53ec004d00 msgr2=0x7f53ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f53fbfff640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53ec004d00 0x7f53ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.157+0000 7f53fbfff640 1 -- 172.21.15.107:0/4005069741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec003520 con 0x7f53ec003f70 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.158+0000 7f53fbfff640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53ec0047c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f53e8001750 tx=0x7f53ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab596b9e4a085e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.158+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80052d0 con 0x7f53ec003f70 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.158+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e802e050 con 0x7f53ec003f70 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.158+0000 7f53fb7fe640 1 -- 172.21.15.107:0/4005069741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8003830 con 0x7f53ec003f70 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 -- 172.21.15.107:0/4005069741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 msgr2=0x7f53ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 --2- 172.21.15.107:0/4005069741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53ec0047c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f53e8001750 tx=0x7f53ec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401115640 1 -- 172.21.15.107:0/4005069741 reap_dead start 2023-11-24T08:16:02.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 -- 172.21.15.107:0/4005069741 shutdown_connections 2023-11-24T08:16:02.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 -- 172.21.15.107:0/4005069741 >> 172.21.15.107:0/4005069741 conn(0x7f53fc0cb830 msgr2=0x7f53fc0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:02.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 -- 172.21.15.107:0/4005069741 shutdown_connections 2023-11-24T08:16:02.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.159+0000 7f5401916640 1 -- 172.21.15.107:0/4005069741 wait complete. 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.161+0000 7f5401916640 1 Processor -- start 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.161+0000 7f5401916640 1 -- start start 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc15d540 0x7f53fc15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53fc0ccc90 0x7f53fc0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc1429b0 con 0x7f53ec003f70 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc15df00 con 0x7f53fc0ccc90 2023-11-24T08:16:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401916640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc0cda00 con 0x7f53fc15d540 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5400914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5400914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39698/0 (socket says 172.21.15.107:39698) 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5400914640 1 -- 172.21.15.107:0/2184078288 learned_addr learned my addr 172.21.15.107:0/2184078288 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.162+0000 7f5401115640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53fc0ccc90 0x7f53fc0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 -- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc15d540 msgr2=0x7f53fc15d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc15d540 0x7f53fc15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 -- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53fc0ccc90 msgr2=0x7f53fc0cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53fc0ccc90 0x7f53fc0cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 -- 172.21.15.107:0/2184078288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc1420c0 con 0x7f53ec003f70 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5400914640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f53f0010860 tx=0x7f53f0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0009610 con 0x7f53ec003f70 2023-11-24T08:16:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f000eb60 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc142e40 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f001c400 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.163+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc142640 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.164+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc04e840 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.165+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f53f001c560 con 0x7f53ec003f70 2023-11-24T08:16:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.167+0000 7f53f97fa640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 0x7f53e006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.167+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53f0088c80 con 0x7f53ec003f70 2023-11-24T08:16:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.167+0000 7f53fbfff640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 0x7f53e006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.168+0000 7f53fbfff640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 0x7f53e006e800 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f53e8033000 tx=0x7f53e802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.171+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53f0055400 con 0x7f53ec003f70 2023-11-24T08:16:02.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.366+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53fc06d860 con 0x7f53ec003f70 2023-11-24T08:16:02.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.368+0000 7f53f97fa640 1 -- 172.21.15.107:0/2184078288 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f53f0058a20 con 0x7f53ec003f70 2023-11-24T08:16:02.370 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:02.370 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":143,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:16:01.400735+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:16:02.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 msgr2=0x7f53e006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 0x7f53e006e800 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f53e8033000 tx=0x7f53e802f000 comp rx=0 tx=0).stop 2023-11-24T08:16:02.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 msgr2=0x7f53fc13e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f53f0010860 tx=0x7f53f0010d20 comp rx=0 tx=0).stop 2023-11-24T08:16:02.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 shutdown_connections 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f53e006c340 0x7f53e006e800 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53fc0ccc90 0x7f53fc0cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53fc15d540 0x7f53fc15d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 --2- 172.21.15.107:0/2184078288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec003f70 0x7f53fc13e1d0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 >> 172.21.15.107:0/2184078288 conn(0x7f53fc0cb830 msgr2=0x7f53fc15ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 shutdown_connections 2023-11-24T08:16:02.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.371+0000 7f5401916640 1 -- 172.21.15.107:0/2184078288 wait complete. 2023-11-24T08:16:02.392 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:16:02.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:16:02.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.604+0000 7f51a2189640 1 Processor -- start 2023-11-24T08:16:02.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.604+0000 7f51a2189640 1 -- start start 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f51a2189640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f519c15b440 0x7f519c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f51a2189640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c0c5dc0 con 0x7f519c0cbf20 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f51a2189640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c15eea0 con 0x7f519c0c59b0 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f51a2189640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519c0cc3d0 con 0x7f519c15b440 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f519affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f519c0cbf20 0x7f519c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53684/0 (socket says 172.21.15.107:53684) 2023-11-24T08:16:02.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.606+0000 7f519affd640 1 -- 172.21.15.107:0/3628192341 learned_addr learned my addr 172.21.15.107:0/3628192341 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519b7fe640 1 --2- 172.21.15.107:0/3628192341 >> v2:172.21.15.120:3300/0 conn(0x7f519c15b440 0x7f519c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366028084 0 0) 0x7f519c0c5dc0 con 0x7f519c0cbf20 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5184003520 con 0x7f519c0cbf20 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3026981292 0 0) 0x7f5184003520 con 0x7f519c0cbf20 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 >> v2:172.21.15.120:3300/0 conn(0x7f519c15b440 msgr2=0x7f519c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 --2- 172.21.15.107:0/3628192341 >> v2:172.21.15.120:3300/0 conn(0x7f519c15b440 0x7f519c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 >> v1:172.21.15.120:6789/0 conn(0x7f519c0c59b0 legacy=0x7f519c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c0abf30 con 0x7f519c0cbf20 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5190001670 con 0x7f519c0cbf20 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 >> v1:172.21.15.107:6789/0 conn(0x7f519c0cbf20 legacy=0x7f519c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51840039c0 0x7f5184003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51840047c0 0x7f5184004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f518400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0cc3d0 con 0x7f51840051d0 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518400c120 con 0x7f51840039c0 2023-11-24T08:16:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.607+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518400c2e0 con 0x7f51840047c0 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f518400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519b7fe640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51840039c0 0x7f5184003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519affd640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51840047c0 0x7f5184004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 -- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51840047c0 msgr2=0x7f5184004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51840047c0 0x7f5184004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 -- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51840039c0 msgr2=0x7f5184003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51840039c0 0x7f5184003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.608+0000 7f519bfff640 1 -- 172.21.15.107:0/3628192341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184003520 con 0x7f51840051d0 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.609+0000 7f519bfff640 1 -- 172.21.15.107:0/3628192341 reap_dead start 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.609+0000 7f519bfff640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f518400b980 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f5194001e20 tx=0x7f51840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804b29738bf26507 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.609+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194003b70 con 0x7f51840051d0 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.609+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5194003cd0 con 0x7f51840051d0 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.609+0000 7f519a7fc640 1 -- 172.21.15.107:0/3628192341 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194005310 con 0x7f51840051d0 2023-11-24T08:16:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 -- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 msgr2=0x7f518400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f518400b980 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f5194001e20 tx=0x7f51840057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 -- 172.21.15.107:0/3628192341 shutdown_connections 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 --2- 172.21.15.107:0/3628192341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f518400b980 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 -- 172.21.15.107:0/3628192341 >> 172.21.15.107:0/3628192341 conn(0x7f519c0b0c50 msgr2=0x7f519c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 -- 172.21.15.107:0/3628192341 shutdown_connections 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.610+0000 7f51a2189640 1 -- 172.21.15.107:0/3628192341 wait complete. 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.612+0000 7f51a2189640 1 Processor -- start 2023-11-24T08:16:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.612+0000 7f51a2189640 1 -- start start 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce2d0 0x7f519c156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0717a0 0x7f519c071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c072160 con 0x7f51840051d0 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0b21c0 con 0x7f519c0717a0 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f51a2189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0b2c90 con 0x7f519c0ce2d0 2023-11-24T08:16:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39712/0 (socket says 172.21.15.107:39712) 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.613+0000 7f519b7fe640 1 -- 172.21.15.107:0/1015280951 learned_addr learned my addr 172.21.15.107:0/1015280951 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519bfff640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0717a0 0x7f519c071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519affd640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce2d0 0x7f519c156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 -- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce2d0 msgr2=0x7f519c156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce2d0 0x7f519c156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 -- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0717a0 msgr2=0x7f519c071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0717a0 0x7f519c071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 -- 172.21.15.107:0/1015280951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c0b23f0 con 0x7f51840051d0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f519b7fe640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f518c011570 tx=0x7f518c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c015930 con 0x7f51840051d0 2023-11-24T08:16:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c015a90 con 0x7f51840051d0 2023-11-24T08:16:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.614+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c015e60 con 0x7f51840051d0 2023-11-24T08:16:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.615+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c0b26d0 con 0x7f51840051d0 2023-11-24T08:16:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.615+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c0c6f90 con 0x7f51840051d0 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.616+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c0c6df0 con 0x7f51840051d0 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.617+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f518c00f460 con 0x7f51840051d0 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.618+0000 7f5183fff640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 0x7f518806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.618+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f518c088e70 con 0x7f51840051d0 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.618+0000 7f519affd640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 0x7f518806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.619+0000 7f519affd640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 0x7f518806e800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f51900017a0 tx=0x7f5190009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.622+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f519c0c6df0 con 0x7f51840051d0 2023-11-24T08:16:02.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.804+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f519c0c6df0 con 0x7f51840051d0 2023-11-24T08:16:02.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.805+0000 7f5183fff640 1 -- 172.21.15.107:0/1015280951 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f519c0c6df0 con 0x7f51840051d0 2023-11-24T08:16:02.807 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:16:02.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 msgr2=0x7f518806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 0x7f518806e800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f51900017a0 tx=0x7f5190009040 comp rx=0 tx=0).stop 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 msgr2=0x7f519c0b1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f518c011570 tx=0x7f518c011a30 comp rx=0 tx=0).stop 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 shutdown_connections 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f518806c340 0x7f518806e800 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f519c0717a0 0x7f519c071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f519c0ce2d0 0x7f519c156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 --2- 172.21.15.107:0/1015280951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51840051d0 0x7f519c0b1c80 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 >> 172.21.15.107:0/1015280951 conn(0x7f519c0b0c50 msgr2=0x7f519c0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.808+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 shutdown_connections 2023-11-24T08:16:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:02.809+0000 7f51a2189640 1 -- 172.21.15.107:0/1015280951 wait complete. 2023-11-24T08:16:02.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:16:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.033+0000 7fd7effff640 1 Processor -- start 2023-11-24T08:16:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.033+0000 7fd7effff640 1 -- start start 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7effff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7f006db20 0x7fd7f00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7effff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f013f9e0 con 0x7fd7f013d120 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7effff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f013fcc0 con 0x7fd7f006d6c0 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7effff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f013ffa0 con 0x7fd7f006db20 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7eeffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd7f013d120 0x7fd7f015af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53700/0 (socket says 172.21.15.107:53700) 2023-11-24T08:16:03.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.035+0000 7fd7eeffd640 1 -- 172.21.15.107:0/4125047968 learned_addr learned my addr 172.21.15.107:0/4125047968 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:03.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7ef7fe640 1 --2- 172.21.15.107:0/4125047968 >> v2:172.21.15.120:3300/0 conn(0x7fd7f006db20 0x7fd7f00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090892205 0 0) 0x7fd7f013f9e0 con 0x7fd7f013d120 2023-11-24T08:16:03.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d8003520 con 0x7fd7f013d120 2023-11-24T08:16:03.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 762918091 0 0) 0x7fd7d8003520 con 0x7fd7f013d120 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 >> v2:172.21.15.120:3300/0 conn(0x7fd7f006db20 msgr2=0x7fd7f00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 --2- 172.21.15.107:0/4125047968 >> v2:172.21.15.120:3300/0 conn(0x7fd7f006db20 0x7fd7f00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 >> v1:172.21.15.120:6789/0 conn(0x7fd7f006d6c0 legacy=0x7fd7f00cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f0140280 con 0x7fd7f013d120 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd7e0001670 con 0x7fd7f013d120 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 >> v1:172.21.15.107:6789/0 conn(0x7fd7f013d120 legacy=0x7fd7f015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 0x7fd7d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7d80047c0 0x7fd7d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d80052a0 0x7fd7d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f013ffa0 con 0x7fd7d8003970 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d800c170 con 0x7fd7d80047c0 2023-11-24T08:16:03.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.036+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d800c330 con 0x7fd7d80052a0 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 0x7fd7d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 -- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d80052a0 msgr2=0x7fd7d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d80052a0 0x7fd7d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 -- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7d80047c0 msgr2=0x7fd7d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7d80047c0 0x7fd7d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7eeffd640 1 -- 172.21.15.107:0/4125047968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8003520 con 0x7fd7d8003970 2023-11-24T08:16:03.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.037+0000 7fd7ef7fe640 1 -- 172.21.15.107:0/4125047968 reap_dead start 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.038+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 0x7fd7d8003d90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c210 tx=0x7fd7d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39dc84f091f08db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.038+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0005ad0 con 0x7fd7d8003970 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.038+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e0005c30 con 0x7fd7d8003970 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.038+0000 7fd7edffb640 1 -- 172.21.15.107:0/4125047968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e002cab0 con 0x7fd7d8003970 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 -- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 msgr2=0x7fd7d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 0x7fd7d8003d90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c210 tx=0x7fd7d80057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 -- 172.21.15.107:0/4125047968 shutdown_connections 2023-11-24T08:16:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 --2- 172.21.15.107:0/4125047968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7d8003970 0x7fd7d8003d90 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 -- 172.21.15.107:0/4125047968 >> 172.21.15.107:0/4125047968 conn(0x7fd7f00cb830 msgr2=0x7fd7f00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 -- 172.21.15.107:0/4125047968 shutdown_connections 2023-11-24T08:16:03.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.039+0000 7fd7effff640 1 -- 172.21.15.107:0/4125047968 wait complete. 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.041+0000 7fd7effff640 1 Processor -- start 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.041+0000 7fd7effff640 1 -- start start 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d8003970 0x7fd7f013b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 0x7fd7f013c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f0155150 con 0x7fd7f013c5e0 2023-11-24T08:16:03.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f00718d0 con 0x7fd7f013bca0 2023-11-24T08:16:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7effff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f0071bb0 con 0x7fd7d8003970 2023-11-24T08:16:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39734/0 (socket says 172.21.15.107:39734) 2023-11-24T08:16:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.042+0000 7fd7ef7fe640 1 -- 172.21.15.107:0/411443979 learned_addr learned my addr 172.21.15.107:0/411443979 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d8003970 0x7fd7f013b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 -- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d8003970 msgr2=0x7fd7f013b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d8003970 0x7fd7f013b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ee7fc640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 0x7fd7f013c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 -- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 msgr2=0x7fd7f013c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 0x7fd7f013c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 -- 172.21.15.107:0/411443979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f015a9d0 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ee7fc640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 0x7fd7f013c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7ef7fe640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd7f006e210 tx=0x7fd7e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800ed90 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8014420 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f0154de0 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.043+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80147d0 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.044+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f0143830 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.044+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f01552c0 con 0x7fd7f013c5e0 2023-11-24T08:16:03.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.045+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fd7e8014930 con 0x7fd7f013c5e0 2023-11-24T08:16:03.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.046+0000 7fd7d77fe640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 0x7fd7dc06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.046+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7e8088600 con 0x7fd7f013c5e0 2023-11-24T08:16:03.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.047+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 0x7fd7dc06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.047+0000 7fd7eeffd640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 0x7fd7dc06e800 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0034000 tx=0x7fd7e0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.050+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7e8055070 con 0x7fd7f013c5e0 2023-11-24T08:16:03.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.244+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd7f004e840 con 0x7fd7f013c5e0 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.245+0000 7fd7d77fe640 1 -- 172.21.15.107:0/411443979 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v143) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd7e8058690 con 0x7fd7f013c5e0 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:16:03.246 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:16:03.247 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:03.247 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:16:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 msgr2=0x7fd7dc06e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 0x7fd7dc06e800 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0034000 tx=0x7fd7e0009030 comp rx=0 tx=0).stop 2023-11-24T08:16:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 msgr2=0x7fd7f01555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd7f006e210 tx=0x7fd7e800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 shutdown_connections 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd7dc06c340 0x7fd7dc06e800 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7f013c5e0 0x7fd7f01555e0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7f013bca0 0x7fd7f013c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 --2- 172.21.15.107:0/411443979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7d8003970 0x7fd7f013b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 >> 172.21.15.107:0/411443979 conn(0x7fd7f00cb830 msgr2=0x7fd7f0158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 shutdown_connections 2023-11-24T08:16:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.248+0000 7fd7effff640 1 -- 172.21.15.107:0/411443979 wait complete. 2023-11-24T08:16:03.266 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:16:03.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:16:03.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.470+0000 7f57f853e640 1 Processor -- start 2023-11-24T08:16:03.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.470+0000 7f57f853e640 1 -- start start 2023-11-24T08:16:03.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.471+0000 7f57f853e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57f0157c70 0x7f57f0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.471+0000 7f57f853e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 0x7f57f00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.471+0000 7f57f853e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f00ce0b0 con 0x7f57f00cf450 2023-11-24T08:16:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.471+0000 7f57f853e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00cfe40 con 0x7f57f0157c70 2023-11-24T08:16:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.471+0000 7f57f853e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0161960 con 0x7f57f015e390 2023-11-24T08:16:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f5ad9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f57f00cf450 0x7f57f015dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53712/0 (socket says 172.21.15.107:53712) 2023-11-24T08:16:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f5ad9640 1 -- 172.21.15.107:0/904395682 learned_addr learned my addr 172.21.15.107:0/904395682 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f62da640 1 --2- 172.21.15.107:0/904395682 >> v2:172.21.15.120:3300/0 conn(0x7f57f0157c70 0x7f57f0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f6adb640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 0x7f57f00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004047503 0 0) 0x7f57f00ce0b0 con 0x7f57f00cf450 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e0003520 con 0x7f57f00cf450 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167834303 0 0) 0x7f57e0003520 con 0x7f57f00cf450 2023-11-24T08:16:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 msgr2=0x7f57f00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 0x7f57f00cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 >> v2:172.21.15.120:3300/0 conn(0x7f57f0157c70 msgr2=0x7f57f0161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 --2- 172.21.15.107:0/904395682 >> v2:172.21.15.120:3300/0 conn(0x7f57f0157c70 0x7f57f0161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f015e8d0 con 0x7f57f00cf450 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.472+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f57dc0019f0 con 0x7f57f00cf450 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 >> v1:172.21.15.107:6789/0 conn(0x7f57f00cf450 legacy=0x7f57f015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57e0003970 0x7f57e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 0x7f57e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0004d00 0x7f57e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00cfe40 con 0x7f57e0003970 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0161960 con 0x7f57e0004d00 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e000b9b0 con 0x7f57f015e390 2023-11-24T08:16:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57e0003970 0x7f57e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 -- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 msgr2=0x7f57e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57f015e390 0x7f57e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 -- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0004d00 msgr2=0x7f57e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:03.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0004d00 0x7f57e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.473+0000 7f57f62da640 1 -- 172.21.15.107:0/904395682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e0003520 con 0x7f57e0003970 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.474+0000 7f57f62da640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57e0003970 0x7f57e0003d90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009ac0 tx=0x7f57e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5017fb8f6caa7ebc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e402c710 con 0x7f57e0003970 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e402c870 con 0x7f57e0003970 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f52d8640 1 -- 172.21.15.107:0/904395682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e402cc00 con 0x7f57e0003970 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 -- 172.21.15.107:0/904395682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57e0003970 msgr2=0x7f57e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 --2- 172.21.15.107:0/904395682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57e0003970 0x7f57e0003d90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009ac0 tx=0x7f57e00057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f6adb640 1 -- 172.21.15.107:0/904395682 reap_dead start 2023-11-24T08:16:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 -- 172.21.15.107:0/904395682 shutdown_connections 2023-11-24T08:16:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 -- 172.21.15.107:0/904395682 >> 172.21.15.107:0/904395682 conn(0x7f57f006a920 msgr2=0x7f57f006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 -- 172.21.15.107:0/904395682 shutdown_connections 2023-11-24T08:16:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.475+0000 7f57f853e640 1 -- 172.21.15.107:0/904395682 wait complete. 2023-11-24T08:16:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.477+0000 7f57f853e640 1 Processor -- start 2023-11-24T08:16:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.477+0000 7f57f853e640 1 -- start start 2023-11-24T08:16:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e0003970 0x7f57f0150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57f014f130 0x7f57f014f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f014fb20 con 0x7f57f0072130 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00cacb0 con 0x7f57f014f130 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f853e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00caf30 con 0x7f57e0003970 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f5ad9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f5ad9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39748/0 (socket says 172.21.15.107:39748) 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.478+0000 7f57f5ad9640 1 -- 172.21.15.107:0/238225317 learned_addr learned my addr 172.21.15.107:0/238225317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 -- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e0003970 msgr2=0x7f57f0150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f62da640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e0003970 0x7f57f0150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f6adb640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57f014f130 0x7f57f014f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e0003970 0x7f57f0150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 -- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57f014f130 msgr2=0x7f57f014f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57f014f130 0x7f57f014f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 -- 172.21.15.107:0/238225317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00cb210 con 0x7f57f0072130 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f5ad9640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00cb10 tx=0x7f57dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00a860 con 0x7f57f0072130 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00a9c0 con 0x7f57f0072130 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00ad70 con 0x7f57f0072130 2023-11-24T08:16:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f00cb4f0 con 0x7f57f0072130 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.479+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f00cbab0 con 0x7f57f0072130 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.480+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f00cbcd0 con 0x7f57f0072130 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.481+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f57dc0070d0 con 0x7f57f0072130 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.482+0000 7f57eeffd640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 0x7f57d406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.483+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f57dc087eb0 con 0x7f57f0072130 2023-11-24T08:16:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.483+0000 7f57f62da640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 0x7f57d406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.483+0000 7f57f62da640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 0x7f57d406e7d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f57e4036000 tx=0x7f57e4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.487+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57dc054630 con 0x7f57f0072130 2023-11-24T08:16:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:03.674+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f57f0067470 con 0x7f57f0072130 2023-11-24T08:16:04.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.430+0000 7f57eeffd640 1 -- 172.21.15.107:0/238225317 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v144) v1 ==== 138+0+0 (secure 0 0 0) 0x7f57dc018070 con 0x7f57f0072130 2023-11-24T08:16:04.432 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:16:04.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 msgr2=0x7f57d406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:04.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 0x7f57d406e7d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f57e4036000 tx=0x7f57e4030000 comp rx=0 tx=0).stop 2023-11-24T08:16:04.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 msgr2=0x7f57f00c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:04.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00cb10 tx=0x7f57dc00a060 comp rx=0 tx=0).stop 2023-11-24T08:16:04.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 shutdown_connections 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57d406c310 0x7f57d406e7d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57f014f130 0x7f57f014f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57f0072130 0x7f57f00c8dc0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 --2- 172.21.15.107:0/238225317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e0003970 0x7f57f0150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 >> 172.21.15.107:0/238225317 conn(0x7f57f006a920 msgr2=0x7f57f006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 shutdown_connections 2023-11-24T08:16:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.434+0000 7f57f853e640 1 -- 172.21.15.107:0/238225317 wait complete. 2023-11-24T08:16:04.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:16:04.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.652+0000 7f3bbf59e640 1 Processor -- start 2023-11-24T08:16:04.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.652+0000 7f3bbf59e640 1 -- start start 2023-11-24T08:16:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3bc0160890 0x7f3bc00c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bc00cb690 0x7f3bc00b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc0160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc006d170 con 0x7f3bc00cb690 2023-11-24T08:16:04.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc00b1510 con 0x7f3bc0160890 2023-11-24T08:16:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc006c530 con 0x7f3bc00b1080 2023-11-24T08:16:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbdd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bc00cb690 0x7f3bc00b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbdd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bc00cb690 0x7f3bc00b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39762/0 (socket says 172.21.15.107:39762) 2023-11-24T08:16:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/1462733429 learned_addr learned my addr 172.21.15.107:0/1462733429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbe59c640 1 --2- 172.21.15.107:0/1462733429 >> v2:172.21.15.120:3300/0 conn(0x7f3bc0160890 0x7f3bc00c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.654+0000 7f3bbed9d640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc0160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.655+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/1462733429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 msgr2=0x7f3bc0160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.655+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc0160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.655+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/1462733429 >> v2:172.21.15.120:3300/0 conn(0x7f3bc0160890 msgr2=0x7f3bc00c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.655+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/1462733429 >> v2:172.21.15.120:3300/0 conn(0x7f3bc0160890 0x7f3bc00c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.655+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/1462733429 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc006c7d0 con 0x7f3bc00cb690 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.656+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc00b0b40 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00ccc10 tx=0x7f3bb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8c8a3f6b603cdb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.656+0000 7f3bbd59a640 1 -- 172.21.15.107:0/1462733429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4031870 con 0x7f3bc00cb690 2023-11-24T08:16:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.656+0000 7f3bbd59a640 1 -- 172.21.15.107:0/1462733429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb40319d0 con 0x7f3bc00cb690 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.656+0000 7f3bbd59a640 1 -- 172.21.15.107:0/1462733429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4031d60 con 0x7f3bc00cb690 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 -- 172.21.15.107:0/1462733429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 msgr2=0x7f3bc00b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc00b0b40 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00ccc10 tx=0x7f3bb402b910 comp rx=0 tx=0).stop 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 -- 172.21.15.107:0/1462733429 shutdown_connections 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc0160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/1462733429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc00b0b40 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/1462733429 >> v2:172.21.15.120:3300/0 conn(0x7f3bc0160890 0x7f3bc00c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 -- 172.21.15.107:0/1462733429 >> 172.21.15.107:0/1462733429 conn(0x7f3bc00cbb40 msgr2=0x7f3bc00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 -- 172.21.15.107:0/1462733429 shutdown_connections 2023-11-24T08:16:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.657+0000 7f3bbf59e640 1 -- 172.21.15.107:0/1462733429 wait complete. 2023-11-24T08:16:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.659+0000 7f3bbf59e640 1 Processor -- start 2023-11-24T08:16:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.659+0000 7f3bbf59e640 1 -- start start 2023-11-24T08:16:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc00afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc0160890 0x7f3bc015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc00c32b0 con 0x7f3bc00cb690 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc015f230 con 0x7f3bc0160890 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbf59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc009aa50 con 0x7f3bc00b1080 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39766/0 (socket says 172.21.15.107:39766) 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/3517953594 learned_addr learned my addr 172.21.15.107:0/3517953594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbe59c640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc00afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.660+0000 7f3bbed9d640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc0160890 0x7f3bc015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 msgr2=0x7f3bc00afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc00afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc0160890 msgr2=0x7f3bc015e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc0160890 0x7f3bc015e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 -- 172.21.15.107:0/3517953594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40094d0 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbdd9b640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f3bb402b950 tx=0x7f3bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb403ad30 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb40443f0 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40447a0 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc009ad30 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.661+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc00c2ed0 con 0x7f3bc00cb690 2023-11-24T08:16:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.662+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc00c33f0 con 0x7f3bc00cb690 2023-11-24T08:16:04.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.664+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f3bb404d450 con 0x7f3bc00cb690 2023-11-24T08:16:04.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.665+0000 7f3baeffd640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 0x7f3ba006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:04.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.665+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3bb402d040 con 0x7f3bc00cb690 2023-11-24T08:16:04.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.666+0000 7f3bbe59c640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 0x7f3ba006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:04.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.666+0000 7f3bbe59c640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 0x7f3ba006e800 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0006fd0 tx=0x7f3bb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:04.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.669+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bb4077880 con 0x7f3bc00cb690 2023-11-24T08:16:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:04.858+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3bc00c33f0 con 0x7f3bc00cb690 2023-11-24T08:16:05.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.435+0000 7f3baeffd640 1 -- 172.21.15.107:0/3517953594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v145) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3bc00c33f0 con 0x7f3bc00cb690 2023-11-24T08:16:05.437 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.438+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 msgr2=0x7f3ba006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.438+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 0x7f3ba006e800 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0006fd0 tx=0x7f3bb0008040 comp rx=0 tx=0).stop 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 msgr2=0x7f3bc009a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f3bb402b950 tx=0x7f3bb40046c0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 shutdown_connections 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ba006c340 0x7f3ba006e800 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc0160890 0x7f3bc015e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc00cb690 0x7f3bc009a390 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 --2- 172.21.15.107:0/3517953594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bc00b1080 0x7f3bc00afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 >> 172.21.15.107:0/3517953594 conn(0x7f3bc00cbb40 msgr2=0x7f3bc0071750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:05.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 shutdown_connections 2023-11-24T08:16:05.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.439+0000 7f3bbf59e640 1 -- 172.21.15.107:0/3517953594 wait complete. 2023-11-24T08:16:05.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:16:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.661+0000 7f882605c640 1 Processor -- start 2023-11-24T08:16:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.661+0000 7f882605c640 1 -- start start 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882015b440 0x7f88200cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 0x7f88200c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88200c57e0 0x7f88200c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882015b820 con 0x7f88200c57e0 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88200cc3a0 con 0x7f88200cbf20 2023-11-24T08:16:05.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f882605c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882015eea0 con 0x7f882015b440 2023-11-24T08:16:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f881ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88200c57e0 0x7f88200c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f881ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88200c57e0 0x7f88200c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39778/0 (socket says 172.21.15.107:39778) 2023-11-24T08:16:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.663+0000 7f881ffff640 1 -- 172.21.15.107:0/3256038910 learned_addr learned my addr 172.21.15.107:0/3256038910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881f7fe640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882015b440 0x7f88200cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f8817fff640 1 --2- 172.21.15.107:0/3256038910 >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 0x7f88200c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881ffff640 1 -- 172.21.15.107:0/3256038910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882015b440 msgr2=0x7f88200cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881ffff640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882015b440 0x7f88200cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881ffff640 1 -- 172.21.15.107:0/3256038910 >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 msgr2=0x7f88200c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881ffff640 1 --2- 172.21.15.107:0/3256038910 >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 0x7f88200c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f881ffff640 1 -- 172.21.15.107:0/3256038910 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200abf30 con 0x7f88200c57e0 2023-11-24T08:16:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.664+0000 7f8817fff640 1 --2- 172.21.15.107:0/3256038910 >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 0x7f88200c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.665+0000 7f881ffff640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200c57e0 0x7f88200c3ec0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f88200cee60 tx=0x7f881802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc1327a76b037a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f881effd640 1 -- 172.21.15.107:0/3256038910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881802c9e0 con 0x7f88200c57e0 2023-11-24T08:16:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f881effd640 1 -- 172.21.15.107:0/3256038910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881802cb40 con 0x7f88200c57e0 2023-11-24T08:16:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f881effd640 1 -- 172.21.15.107:0/3256038910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88180343f0 con 0x7f88200c57e0 2023-11-24T08:16:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 -- 172.21.15.107:0/3256038910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200c57e0 msgr2=0x7f88200c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200c57e0 0x7f88200c3ec0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f88200cee60 tx=0x7f881802be40 comp rx=0 tx=0).stop 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 -- 172.21.15.107:0/3256038910 shutdown_connections 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200c57e0 0x7f88200c3ec0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 --2- 172.21.15.107:0/3256038910 >> v2:172.21.15.120:3300/0 conn(0x7f88200cbf20 0x7f88200c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 --2- 172.21.15.107:0/3256038910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f882015b440 0x7f88200cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.666+0000 7f882605c640 1 -- 172.21.15.107:0/3256038910 >> 172.21.15.107:0/3256038910 conn(0x7f88200b0c50 msgr2=0x7f88200b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.667+0000 7f882605c640 1 -- 172.21.15.107:0/3256038910 shutdown_connections 2023-11-24T08:16:05.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.667+0000 7f882605c640 1 -- 172.21.15.107:0/3256038910 wait complete. 2023-11-24T08:16:05.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.668+0000 7f882605c640 1 Processor -- start 2023-11-24T08:16:05.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.668+0000 7f882605c640 1 -- start start 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88200c57e0 0x7f88201556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882015b440 0x7f882015e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200c82d0 con 0x7f88200cbf20 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200c85b0 con 0x7f882015b440 2023-11-24T08:16:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.669+0000 7f882605c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882015ba00 con 0x7f88200c57e0 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39780/0 (socket says 172.21.15.107:39780) 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 -- 172.21.15.107:0/1838593339 learned_addr learned my addr 172.21.15.107:0/1838593339 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f881ffff640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882015b440 0x7f882015e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f881f7fe640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88200c57e0 0x7f88201556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 -- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88200c57e0 msgr2=0x7f88201556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88200c57e0 0x7f88201556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 -- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882015b440 msgr2=0x7f882015e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882015b440 0x7f882015e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:05.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 -- 172.21.15.107:0/1838593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8818009b30 con 0x7f88200cbf20 2023-11-24T08:16:05.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.670+0000 7f8817fff640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f880800c7c0 tx=0x7f880800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:05.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.671+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808007a30 con 0x7f88200cbf20 2023-11-24T08:16:05.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.671+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808007b90 con 0x7f88200cbf20 2023-11-24T08:16:05.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.671+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800ce70 con 0x7f88200cbf20 2023-11-24T08:16:05.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.671+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882015bce0 con 0x7f88200cbf20 2023-11-24T08:16:05.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.671+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88200c91f0 con 0x7f88200cbf20 2023-11-24T08:16:05.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.672+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882004e8b0 con 0x7f88200cbf20 2023-11-24T08:16:05.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.674+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f8808024480 con 0x7f88200cbf20 2023-11-24T08:16:05.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.674+0000 7f881cff9640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 0x7f880006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:05.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.674+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f88080888e0 con 0x7f88200cbf20 2023-11-24T08:16:05.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.675+0000 7f881f7fe640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 0x7f880006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:05.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.675+0000 7f881f7fe640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 0x7f880006e7d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8810006fd0 tx=0x7f8810008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:05.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.678+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8808055420 con 0x7f88200cbf20 2023-11-24T08:16:05.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:05.902+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f882006dc80 con 0x7f88200cbf20 2023-11-24T08:16:06.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.483+0000 7f881cff9640 1 -- 172.21.15.107:0/1838593339 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v83) v1 ==== 174+0+0 (secure 0 0 0) 0x7f880801f020 con 0x7f88200cbf20 2023-11-24T08:16:06.490 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 20 and data pool 21 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.492+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 msgr2=0x7f880006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.492+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 0x7f880006e7d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8810006fd0 tx=0x7f8810008040 comp rx=0 tx=0).stop 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.492+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 msgr2=0x7f882015de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.492+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f880800c7c0 tx=0x7f880800cc80 comp rx=0 tx=0).stop 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 shutdown_connections 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880006c310 0x7f880006e7d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f882015b440 0x7f882015e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88200cbf20 0x7f882015de90 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 --2- 172.21.15.107:0/1838593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88200c57e0 0x7f88201556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.494+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 >> 172.21.15.107:0/1838593339 conn(0x7f88200b0c50 msgr2=0x7f8820071590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.495+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 shutdown_connections 2023-11-24T08:16:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.495+0000 7f882605c640 1 -- 172.21.15.107:0/1838593339 wait complete. 2023-11-24T08:16:06.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:16:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.699+0000 7fae3ed7c640 1 Processor -- start 2023-11-24T08:16:06.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.699+0000 7fae3ed7c640 1 -- start start 2023-11-24T08:16:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3ed7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 0x7fae3815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3ed7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae380b2a70 0x7fae3815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3ed7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3815ab80 con 0x7fae380b2630 2023-11-24T08:16:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3ed7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3815a990 con 0x7fae380b2a70 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3ed7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3815e680 con 0x7fae380b2250 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3d319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae380b2a70 0x7fae3815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3d319640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae380b2a70 0x7fae3815b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39796/0 (socket says 172.21.15.107:39796) 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3d319640 1 -- 172.21.15.107:0/2946962471 learned_addr learned my addr 172.21.15.107:0/2946962471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3cb18640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 0x7fae3815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.701+0000 7fae3d319640 1 -- 172.21.15.107:0/2946962471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 msgr2=0x7fae3815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.702+0000 7fae3d319640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 0x7fae3815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.702+0000 7fae3d319640 1 -- 172.21.15.107:0/2946962471 >> v1:172.21.15.107:6789/0 conn(0x7fae380b2630 legacy=0x7fae380ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.702+0000 7fae3d319640 1 -- 172.21.15.107:0/2946962471 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3815e960 con 0x7fae380b2a70 2023-11-24T08:16:06.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.702+0000 7fae377fe640 1 -- 172.21.15.107:0/2946962471 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503639683 0 0) 0x7fae3815ab80 con 0x7fae380b2630 2023-11-24T08:16:06.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.702+0000 7fae3cb18640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 0x7fae3815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:06.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3d319640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2a70 0x7fae3815b080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fae30009e50 tx=0x7fae3002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed73d458e92e986 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:06.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae377fe640 1 -- 172.21.15.107:0/2946962471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3002cb00 con 0x7fae380b2a70 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae377fe640 1 -- 172.21.15.107:0/2946962471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3002cc60 con 0x7fae380b2a70 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae377fe640 1 -- 172.21.15.107:0/2946962471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae300343f0 con 0x7fae380b2a70 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2946962471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2a70 msgr2=0x7fae3815b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2a70 0x7fae3815b080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fae30009e50 tx=0x7fae3002bf60 comp rx=0 tx=0).stop 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2946962471 shutdown_connections 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2a70 0x7fae3815b080 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2946962471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2250 0x7fae3815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.703+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2946962471 >> 172.21.15.107:0/2946962471 conn(0x7fae3806a920 msgr2=0x7fae3806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.704+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2946962471 shutdown_connections 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.704+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2946962471 wait complete. 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.705+0000 7fae3ed7c640 1 Processor -- start 2023-11-24T08:16:06.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.705+0000 7fae3ed7c640 1 -- start start 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b2250 0x7fae3814c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2a70 0x7fae3815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae380b1510 con 0x7fae380b2630 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae380b0990 con 0x7fae380b2250 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae3ed7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae380b0c70 con 0x7fae380b2a70 2023-11-24T08:16:06.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39804/0 (socket says 172.21.15.107:39804) 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.706+0000 7fae37fff640 1 -- 172.21.15.107:0/2281801093 learned_addr learned my addr 172.21.15.107:0/2281801093 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 -- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2a70 msgr2=0x7fae3815cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2a70 0x7fae3815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 -- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b2250 msgr2=0x7fae3814c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b2250 0x7fae3814c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 -- 172.21.15.107:0/2281801093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30009b30 con 0x7fae380b2630 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae37fff640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fae2400c4b0 tx=0x7fae2400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24005480 con 0x7fae380b2630 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae240035f0 con 0x7fae380b2630 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae240039e0 con 0x7fae380b2630 2023-11-24T08:16:06.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae380cd7c0 con 0x7fae380b2630 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.707+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae380b1230 con 0x7fae380b2630 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.708+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae38067470 con 0x7fae380b2630 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.709+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fae24003b40 con 0x7fae380b2630 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.710+0000 7fae357fa640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 0x7fae1c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.710+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae24087910 con 0x7fae380b2630 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.710+0000 7fae3cb18640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 0x7fae1c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:06.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.711+0000 7fae3cb18640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 0x7fae1c06e7d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fae2c006fd0 tx=0x7fae2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:06.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.714+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae24054410 con 0x7fae380b2630 2023-11-24T08:16:06.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2023-11-24T08:16:06.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-11-24T08:16:06.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-24T08:16:06.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.901+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae380cea00 con 0x7fae380b2630 2023-11-24T08:16:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.903+0000 7fae357fa640 1 -- 172.21.15.107:0/2281801093 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16592 (secure 0 0 0) 0x7fae24017070 con 0x7fae380b2630 2023-11-24T08:16:06.906 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:06.906 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":146,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:16:05.904043+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:16:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 msgr2=0x7fae1c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 0x7fae1c06e7d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fae2c006fd0 tx=0x7fae2c008040 comp rx=0 tx=0).stop 2023-11-24T08:16:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 msgr2=0x7fae3815c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fae2400c4b0 tx=0x7fae2400c970 comp rx=0 tx=0).stop 2023-11-24T08:16:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 shutdown_connections 2023-11-24T08:16:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae1c06c310 0x7fae1c06e7d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b2a70 0x7fae3815cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae380b2630 0x7fae3815c9a0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 --2- 172.21.15.107:0/2281801093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b2250 0x7fae3814c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:06.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 >> 172.21.15.107:0/2281801093 conn(0x7fae3806a920 msgr2=0x7fae3815ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:06.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 shutdown_connections 2023-11-24T08:16:06.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:06.906+0000 7fae3ed7c640 1 -- 172.21.15.107:0/2281801093 wait complete. 2023-11-24T08:16:06.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:16:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.132+0000 7ff0d9e44640 1 Processor -- start 2023-11-24T08:16:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.133+0000 7ff0d9e44640 1 -- start start 2023-11-24T08:16:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.134+0000 7ff0d9e44640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0d4144b30 0x7ff0d415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.134+0000 7ff0d9e44640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 0x7ff0d40b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.134+0000 7ff0d9e44640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d415bf80 con 0x7ff0d40b1de0 2023-11-24T08:16:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.134+0000 7ff0d9e44640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d40b1560 con 0x7ff0d4144b30 2023-11-24T08:16:07.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.134+0000 7ff0d9e44640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b0fb0 con 0x7ff0d415c410 2023-11-24T08:16:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d9643640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff0d40b1de0 0x7ff0d406c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53724/0 (socket says 172.21.15.107:53724) 2023-11-24T08:16:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d9643640 1 -- 172.21.15.107:0/440038464 learned_addr learned my addr 172.21.15.107:0/440038464 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/440038464 >> v2:172.21.15.120:3300/0 conn(0x7ff0d4144b30 0x7ff0d415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d3fff640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 0x7ff0d40b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989295597 0 0) 0x7ff0d415bf80 con 0x7ff0d40b1de0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c4003520 con 0x7ff0d40b1de0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787533619 0 0) 0x7ff0c4003520 con 0x7ff0d40b1de0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 msgr2=0x7ff0d40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 0x7ff0d40b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 >> v2:172.21.15.120:3300/0 conn(0x7ff0d4144b30 msgr2=0x7ff0d415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 --2- 172.21.15.107:0/440038464 >> v2:172.21.15.120:3300/0 conn(0x7ff0d4144b30 0x7ff0d415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40b1280 con 0x7ff0d40b1de0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff0cc0019f0 con 0x7ff0d40b1de0 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 >> v1:172.21.15.107:6789/0 conn(0x7ff0d40b1de0 legacy=0x7ff0d406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.135+0000 7ff0d37fe640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d37fe640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 0x7ff0c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b1560 con 0x7ff0c4003970 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b0fb0 con 0x7ff0c40047c0 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c400b980 con 0x7ff0d415c410 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 -- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 msgr2=0x7ff0c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d415c410 0x7ff0c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 -- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c40047c0 msgr2=0x7ff0c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c40047c0 0x7ff0c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.136+0000 7ff0d8e42640 1 -- 172.21.15.107:0/440038464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4003520 con 0x7ff0c4003970 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.137+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009ac0 tx=0x7ff0c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc372947a2149511 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.137+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c802c710 con 0x7ff0c4003970 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.137+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c802c870 con 0x7ff0c4003970 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.137+0000 7ff0d37fe640 1 -- 172.21.15.107:0/440038464 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c802cc00 con 0x7ff0c4003970 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 -- 172.21.15.107:0/440038464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 msgr2=0x7ff0c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/440038464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0c4003d90 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009ac0 tx=0x7ff0c40057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9643640 1 -- 172.21.15.107:0/440038464 reap_dead start 2023-11-24T08:16:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 -- 172.21.15.107:0/440038464 shutdown_connections 2023-11-24T08:16:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 -- 172.21.15.107:0/440038464 >> 172.21.15.107:0/440038464 conn(0x7ff0d40cb830 msgr2=0x7ff0d40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 -- 172.21.15.107:0/440038464 shutdown_connections 2023-11-24T08:16:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.138+0000 7ff0d9e44640 1 -- 172.21.15.107:0/440038464 wait complete. 2023-11-24T08:16:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.140+0000 7ff0d9e44640 1 Processor -- start 2023-11-24T08:16:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.140+0000 7ff0d9e44640 1 -- start start 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d413d4b0 0x7ff0d41f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41f3440 0x7ff0d41f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f3e30 con 0x7ff0c4003970 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d4144610 con 0x7ff0d41f3440 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d9e44640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d413ed10 con 0x7ff0d413d4b0 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d8e42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d8e42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39824/0 (socket says 172.21.15.107:39824) 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.141+0000 7ff0d8e42640 1 -- 172.21.15.107:0/3844968590 learned_addr learned my addr 172.21.15.107:0/3844968590 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 -- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d413d4b0 msgr2=0x7ff0d41f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d413d4b0 0x7ff0d41f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 -- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41f3440 msgr2=0x7ff0d41f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41f3440 0x7ff0d41f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 -- 172.21.15.107:0/3844968590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d413dc70 con 0x7ff0c4003970 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d8e42640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8037040 tx=0x7ff0c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003810 con 0x7ff0c4003970 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8003970 con 0x7ff0c4003970 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003d20 con 0x7ff0c4003970 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d413df50 con 0x7ff0c4003970 2023-11-24T08:16:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.142+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d413e510 con 0x7ff0c4003970 2023-11-24T08:16:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.143+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d404e840 con 0x7ff0c4003970 2023-11-24T08:16:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.144+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7ff0c8040450 con 0x7ff0c4003970 2023-11-24T08:16:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.145+0000 7ff0d17fa640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 0x7ff0b806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.145+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0c80aa160 con 0x7ff0c4003970 2023-11-24T08:16:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.145+0000 7ff0d3fff640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 0x7ff0b806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.146+0000 7ff0d3fff640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 0x7ff0b806e7d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006f00 tx=0x7ff0c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.150+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0c8076de0 con 0x7ff0c4003970 2023-11-24T08:16:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.343+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0d406d860 con 0x7ff0c4003970 2023-11-24T08:16:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.346+0000 7ff0d17fa640 1 -- 172.21.15.107:0/3844968590 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16592 (secure 0 0 0) 0x7ff0c8044090 con 0x7ff0c4003970 2023-11-24T08:16:07.348 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:07.348 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":146,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:16:05.904043+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.348+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 msgr2=0x7ff0b806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 0x7ff0b806e7d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006f00 tx=0x7ff0c0008040 comp rx=0 tx=0).stop 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 msgr2=0x7ff0d41440d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8037040 tx=0x7ff0c8004730 comp rx=0 tx=0).stop 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 shutdown_connections 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0b806c310 0x7ff0b806e7d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41f3440 0x7ff0d41f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0d413d4b0 0x7ff0d41f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 --2- 172.21.15.107:0/3844968590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0c4003970 0x7ff0d41440d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 >> 172.21.15.107:0/3844968590 conn(0x7ff0d40cb830 msgr2=0x7ff0d406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 shutdown_connections 2023-11-24T08:16:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.349+0000 7ff0d9e44640 1 -- 172.21.15.107:0/3844968590 wait complete. 2023-11-24T08:16:07.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:16:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.588+0000 7fe4ac0cf640 1 Processor -- start 2023-11-24T08:16:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.588+0000 7fe4ac0cf640 1 -- start start 2023-11-24T08:16:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4ac0cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe4a415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4ac0cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a415ec90 con 0x7fe4a4072160 2023-11-24T08:16:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4ac0cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40cd530 con 0x7fe4a406de10 2023-11-24T08:16:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4ac0cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40c4ed0 con 0x7fe4a40c49f0 2023-11-24T08:16:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4a9e6b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe4a4072160 0x7fe4a406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53736/0 (socket says 172.21.15.107:53736) 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.590+0000 7fe4a9e6b640 1 -- 172.21.15.107:0/3566340246 learned_addr learned my addr 172.21.15.107:0/3566340246 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe4a415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946130813 0 0) 0x7fe4a0001110 con 0x7fe4a4072160 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe490005730 con 0x7fe4a4072160 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813886657 0 0) 0x7fe490005730 con 0x7fe4a4072160 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 msgr2=0x7fe4a415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe4a415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 >> v1:172.21.15.120:6789/0 conn(0x7fe4a406de10 legacy=0x7fe4a40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40c51b0 con 0x7fe4a4072160 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe4a0001800 con 0x7fe4a4072160 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 >> v1:172.21.15.107:6789/0 conn(0x7fe4a4072160 legacy=0x7fe4a406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe490005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4900061e0 0x7fe4900069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe490006f10 0x7fe490007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40c4ed0 con 0x7fe490006f10 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49000dd50 con 0x7fe4900061e0 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.591+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49000df10 con 0x7fe4a40c49f0 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe490006f10 0x7fe490007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4a966a640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4900061e0 0x7fe4900069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4a9e6b640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe490005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 -- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 msgr2=0x7fe490005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a40c49f0 0x7fe490005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 -- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4900061e0 msgr2=0x7fe4900069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4900061e0 0x7fe4900069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.592+0000 7fe4aa66c640 1 -- 172.21.15.107:0/3566340246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe490005730 con 0x7fe490006f10 2023-11-24T08:16:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.593+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe490006f10 0x7fe490007390 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe4a406ead0 tx=0x7fe490007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0e5c4889606896 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.593+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c02cc70 con 0x7fe490006f10 2023-11-24T08:16:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.593+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c02cdd0 con 0x7fe490006f10 2023-11-24T08:16:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.593+0000 7fe4a8e69640 1 -- 172.21.15.107:0/3566340246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c034680 con 0x7fe490006f10 2023-11-24T08:16:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/3566340246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe490006f10 msgr2=0x7fe490007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/3566340246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe490006f10 0x7fe490007390 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe4a406ead0 tx=0x7fe490007a00 comp rx=0 tx=0).stop 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4aa66c640 1 -- 172.21.15.107:0/3566340246 reap_dead start 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/3566340246 shutdown_connections 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/3566340246 >> 172.21.15.107:0/3566340246 conn(0x7fe4a40b0c50 msgr2=0x7fe4a40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/3566340246 shutdown_connections 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.594+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/3566340246 wait complete. 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.596+0000 7fe4ac0cf640 1 Processor -- start 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.596+0000 7fe4ac0cf640 1 -- start start 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 0x7fe4a40afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 0x7fe4a415b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a415b1d0 con 0x7fe4a41f6f50 2023-11-24T08:16:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40aecd0 con 0x7fe4a41f32f0 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4ac0cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40aefb0 con 0x7fe4a415b4b0 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4aa66c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4aa66c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39844/0 (socket says 172.21.15.107:39844) 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.597+0000 7fe4aa66c640 1 -- 172.21.15.107:0/2367197276 learned_addr learned my addr 172.21.15.107:0/2367197276 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4a9e6b640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 0x7fe4a40afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4a966a640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 0x7fe4a415b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 -- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 msgr2=0x7fe4a415b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 0x7fe4a415b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 -- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 msgr2=0x7fe4a40afc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 0x7fe4a40afc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 -- 172.21.15.107:0/2367197276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a415ae80 con 0x7fe4a41f6f50 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4a966a640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 0x7fe4a415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:07.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4a9e6b640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 0x7fe4a40afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4aa66c640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe49c02c610 tx=0x7fe49c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c03e600 con 0x7fe4a41f6f50 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c03e760 con 0x7fe4a41f6f50 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a415be50 con 0x7fe4a41f6f50 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.598+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c03eaf0 con 0x7fe4a41f6f50 2023-11-24T08:16:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.599+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a41551d0 con 0x7fe4a41f6f50 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.600+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40af170 con 0x7fe4a41f6f50 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.601+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fe49c03ec50 con 0x7fe4a41f6f50 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.602+0000 7fe49a7fc640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 0x7fe48806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.602+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe49c0aa350 con 0x7fe4a41f6f50 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.602+0000 7fe4a9e6b640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 0x7fe48806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.603+0000 7fe4a9e6b640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 0x7fe48806e7d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0001050 tx=0x7fe4a0001320 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.606+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe49c076f70 con 0x7fe4a41f6f50 2023-11-24T08:16:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:07.836+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe4a40af320 con 0x7fe4a41f6f50 2023-11-24T08:16:08.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.505+0000 7fe49a7fc640 1 -- 172.21.15.107:0/2367197276 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v86) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe49c039030 con 0x7fe4a41f6f50 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 msgr2=0x7fe48806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 0x7fe48806e7d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0001050 tx=0x7fe4a0001320 comp rx=0 tx=0).stop 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 msgr2=0x7fe4a41f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe49c02c610 tx=0x7fe49c004ab0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 shutdown_connections 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48806c310 0x7fe48806e7d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a41f6f50 0x7fe4a41f73d0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a415b4b0 0x7fe4a415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 --2- 172.21.15.107:0/2367197276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a41f32f0 0x7fe4a40afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 >> 172.21.15.107:0/2367197276 conn(0x7fe4a40b0c50 msgr2=0x7fe4a40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:08.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 shutdown_connections 2023-11-24T08:16:08.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.509+0000 7fe4ac0cf640 1 -- 172.21.15.107:0/2367197276 wait complete. 2023-11-24T08:16:08.532 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:16:08.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:16:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.753+0000 7ff304e64640 1 Processor -- start 2023-11-24T08:16:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.753+0000 7ff304e64640 1 -- start start 2023-11-24T08:16:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff304e64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff30015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff304e64640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30015ec90 con 0x7ff3000c49f0 2023-11-24T08:16:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff304e64640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3000cd530 con 0x7ff30006de10 2023-11-24T08:16:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff304e64640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000c4ed0 con 0x7ff300072160 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff2f5d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff3000c49f0 0x7ff30006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52020/0 (socket says 172.21.15.107:52020) 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff2f5d9b640 1 -- 172.21.15.107:0/2259370795 learned_addr learned my addr 172.21.15.107:0/2259370795 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff30015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.755+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193692451 0 0) 0x7ff2e8001110 con 0x7ff3000c49f0 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec003520 con 0x7ff3000c49f0 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 189747593 0 0) 0x7ff2ec003520 con 0x7ff3000c49f0 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 msgr2=0x7ff30015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff30015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 >> v1:172.21.15.120:6789/0 conn(0x7ff30006de10 legacy=0x7ff3000705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3000c51b0 con 0x7ff3000c49f0 2023-11-24T08:16:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff2e8001800 con 0x7ff3000c49f0 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 >> v1:172.21.15.107:6789/0 conn(0x7ff3000c49f0 legacy=0x7ff30006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec0039a0 0x7ff2ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff2ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2ec004d00 0x7ff2ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000c4ed0 con 0x7ff2ec004d00 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec00ba30 con 0x7ff2ec0039a0 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec00bbf0 con 0x7ff300072160 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.756+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2ec004d00 0x7ff2ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2f5d9b640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff2ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec0039a0 0x7ff2ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fed9d640 1 -- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 msgr2=0x7ff2ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff300072160 0x7ff2ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fed9d640 1 -- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec0039a0 msgr2=0x7ff2ec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec0039a0 0x7ff2ec003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.757+0000 7ff2fed9d640 1 -- 172.21.15.107:0/2259370795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec003520 con 0x7ff2ec004d00 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.758+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2ec004d00 0x7ff2ec005180 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ff3000b51a0 tx=0x7ff2ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a57a66257afe18 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.758+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80052d0 con 0x7ff2ec004d00 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.758+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f802e050 con 0x7ff2ec004d00 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.758+0000 7ff2fdd9b640 1 -- 172.21.15.107:0/2259370795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80035f0 con 0x7ff2ec004d00 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 -- 172.21.15.107:0/2259370795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2ec004d00 msgr2=0x7ff2ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 --2- 172.21.15.107:0/2259370795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2ec004d00 0x7ff2ec005180 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ff3000b51a0 tx=0x7ff2ec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff2fed9d640 1 -- 172.21.15.107:0/2259370795 reap_dead start 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 -- 172.21.15.107:0/2259370795 shutdown_connections 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 -- 172.21.15.107:0/2259370795 >> 172.21.15.107:0/2259370795 conn(0x7ff3000b0c50 msgr2=0x7ff3000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 -- 172.21.15.107:0/2259370795 shutdown_connections 2023-11-24T08:16:08.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.759+0000 7ff304e64640 1 -- 172.21.15.107:0/2259370795 wait complete. 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.761+0000 7ff304e64640 1 Processor -- start 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.761+0000 7ff304e64640 1 -- start start 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 0x7ff3000b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 0x7ff3000ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30015a1e0 con 0x7ff300159660 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000ca7c0 con 0x7ff2ec004d00 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff304e64640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30014b060 con 0x7ff3000b1c80 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35396/0 (socket says 172.21.15.107:35396) 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 -- 172.21.15.107:0/2197839191 learned_addr learned my addr 172.21.15.107:0/2197839191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 0x7ff3000b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 0x7ff3000ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 -- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 msgr2=0x7ff3000ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 0x7ff3000ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 -- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 msgr2=0x7ff3000b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 0x7ff3000b3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2f5d9b640 1 -- 172.21.15.107:0/2197839191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff300149cc0 con 0x7ff300159660 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2fed9d640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 0x7ff3000ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.762+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 0x7ff3000b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff2f5d9b640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e840 tx=0x7ff2e800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800c8d0 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e800ca50 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800ce00 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff300159c20 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.763+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff300149950 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.764+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30004e8b0 con 0x7ff300159660 2023-11-24T08:16:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.766+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7ff2e801d460 con 0x7ff300159660 2023-11-24T08:16:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.766+0000 7ff2f77fe640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 0x7ff2e006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.767+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2e80884c0 con 0x7ff300159660 2023-11-24T08:16:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.767+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 0x7ff2e006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.767+0000 7ff2fe59c640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 0x7ff2e006e7d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0007460 tx=0x7ff2f000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.771+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2e8054be0 con 0x7ff300159660 2023-11-24T08:16:08.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:08.997+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff300149b10 con 0x7ff300159660 2023-11-24T08:16:09.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.509+0000 7ff2f77fe640 1 -- 172.21.15.107:0/2197839191 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v87) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff2e8058200 con 0x7ff300159660 2023-11-24T08:16:09.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.513+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 msgr2=0x7ff2e006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.513+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 0x7ff2e006e7d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0007460 tx=0x7ff2f000d040 comp rx=0 tx=0).stop 2023-11-24T08:16:09.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.513+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 msgr2=0x7ff3000ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.513+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e840 tx=0x7ff2e800ed00 comp rx=0 tx=0).stop 2023-11-24T08:16:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 shutdown_connections 2023-11-24T08:16:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2e006c310 0x7ff2e006e7d0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000b1c80 0x7ff3000ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff300159660 0x7ff3000ca280 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 --2- 172.21.15.107:0/2197839191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2ec004d00 0x7ff3000b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 >> 172.21.15.107:0/2197839191 conn(0x7ff3000b0c50 msgr2=0x7ff3000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 shutdown_connections 2023-11-24T08:16:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.514+0000 7ff304e64640 1 -- 172.21.15.107:0/2197839191 wait complete. 2023-11-24T08:16:09.537 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:16:09.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:16:09.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.749+0000 7f985bfff640 1 Processor -- start 2023-11-24T08:16:09.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.750+0000 7f985bfff640 1 -- start start 2023-11-24T08:16:09.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f985c0b0490 0x7f985c0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f985c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985bfff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0cf090 con 0x7f985c15e910 2023-11-24T08:16:09.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985bfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c0ced40 con 0x7f985c0b0490 2023-11-24T08:16:09.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0ccba0 con 0x7f985c0cfa60 2023-11-24T08:16:09.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.751+0000 7f985affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f985c15e910 0x7f985c06e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52024/0 (socket says 172.21.15.107:52024) 2023-11-24T08:16:09.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f985affd640 1 -- 172.21.15.107:0/2634126818 learned_addr learned my addr 172.21.15.107:0/2634126818 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:09.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226267830 0 0) 0x7f985c0cf090 con 0x7f985c15e910 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9848003520 con 0x7f985c15e910 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f985b7fe640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f985c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f985a7fc640 1 --2- 172.21.15.107:0/2634126818 >> v2:172.21.15.120:3300/0 conn(0x7f985c0b0490 0x7f985c0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70985688 0 0) 0x7f9848003520 con 0x7f985c15e910 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 msgr2=0x7f985c15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f985c15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 >> v2:172.21.15.120:3300/0 conn(0x7f985c0b0490 msgr2=0x7f985c0cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 --2- 172.21.15.107:0/2634126818 >> v2:172.21.15.120:3300/0 conn(0x7f985c0b0490 0x7f985c0cf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0cce80 con 0x7f985c15e910 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9844001670 con 0x7f985c15e910 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 >> v1:172.21.15.107:6789/0 conn(0x7f985c15e910 legacy=0x7f985c06e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f9848003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848003fa0 0x7f98480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f9848005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0ced40 con 0x7f9848004d00 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.752+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0ccba0 con 0x7f9848003fa0 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984800ba80 con 0x7f985c0cfa60 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f9848005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985affd640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f9848003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985a7fc640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848003fa0 0x7f98480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 -- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 msgr2=0x7f9848003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0cfa60 0x7f9848003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 -- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848003fa0 msgr2=0x7f98480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848003fa0 0x7f98480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985b7fe640 1 -- 172.21.15.107:0/2634126818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9848003520 con 0x7f9848004d00 2023-11-24T08:16:09.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.753+0000 7f985a7fc640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848003fa0 0x7f98480047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:09.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.754+0000 7f985b7fe640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f9848005180 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba30 tx=0x7f98480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba000f99cc40a82 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:09.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854032640 con 0x7f9848004d00 2023-11-24T08:16:09.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98540327a0 con 0x7f9848004d00 2023-11-24T08:16:09.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f9859ffb640 1 -- 172.21.15.107:0/2634126818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854032b30 con 0x7f9848004d00 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 -- 172.21.15.107:0/2634126818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 msgr2=0x7f9848005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 --2- 172.21.15.107:0/2634126818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f9848005180 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba30 tx=0x7f98480057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985b7fe640 1 -- 172.21.15.107:0/2634126818 reap_dead start 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 -- 172.21.15.107:0/2634126818 shutdown_connections 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 -- 172.21.15.107:0/2634126818 >> 172.21.15.107:0/2634126818 conn(0x7f985c0b2ae0 msgr2=0x7f985c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 -- 172.21.15.107:0/2634126818 shutdown_connections 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.755+0000 7f985bfff640 1 -- 172.21.15.107:0/2634126818 wait complete. 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.757+0000 7f985bfff640 1 Processor -- start 2023-11-24T08:16:09.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.757+0000 7f985bfff640 1 -- start start 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f985c0aefe0 0x7f985c1f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0af460 0x7f985c1f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0b6cd0 con 0x7f9848004d00 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1f78c0 con 0x7f985c0aefe0 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1f75e0 con 0x7f985c0af460 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.758+0000 7f985affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35408/0 (socket says 172.21.15.107:35408) 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 -- 172.21.15.107:0/3625164819 learned_addr learned my addr 172.21.15.107:0/3625164819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 -- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0af460 msgr2=0x7f985c1f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0af460 0x7f985c1f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 -- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f985c0aefe0 msgr2=0x7f985c1f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f985c0aefe0 0x7f985c1f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 -- 172.21.15.107:0/3625164819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0c4ab0 con 0x7f9848004d00 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f985affd640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9844001bb0 tx=0x7f984400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:09.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984400c850 con 0x7f9848004d00 2023-11-24T08:16:09.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.759+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984400c9d0 con 0x7f9848004d00 2023-11-24T08:16:09.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.760+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984400cd80 con 0x7f9848004d00 2023-11-24T08:16:09.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.760+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c0c4d90 con 0x7f9848004d00 2023-11-24T08:16:09.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.760+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c158e50 con 0x7f9848004d00 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.761+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c04e840 con 0x7f9848004d00 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.762+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f984401d460 con 0x7f9848004d00 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.763+0000 7f98537fe640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 0x7f983c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.763+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9844088700 con 0x7f9848004d00 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.763+0000 7f985a7fc640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 0x7f983c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.764+0000 7f985a7fc640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 0x7f983c06e7d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f984c0094c0 tx=0x7f984c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:09.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.767+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9844054e80 con 0x7f9848004d00 2023-11-24T08:16:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.997+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f985c15dbc0 con 0x7f9848004d00 2023-11-24T08:16:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:09.998+0000 7f98537fe640 1 -- 172.21.15.107:0/3625164819 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f98440584a0 con 0x7f9848004d00 2023-11-24T08:16:10.000 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:10.000 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:16:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 87 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.000+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 msgr2=0x7f983c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.000+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 0x7f983c06e7d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f984c0094c0 tx=0x7f984c00b040 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.000+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 msgr2=0x7f985c0c5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.000+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9844001bb0 tx=0x7f984400ecd0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 shutdown_connections 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f983c06c310 0x7f983c06e7d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f985c0af460 0x7f985c1f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f985c0aefe0 0x7f985c1f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 --2- 172.21.15.107:0/3625164819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9848004d00 0x7f985c0c5590 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 >> 172.21.15.107:0/3625164819 conn(0x7f985c0b2ae0 msgr2=0x7f985c0c3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 shutdown_connections 2023-11-24T08:16:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.001+0000 7f985bfff640 1 -- 172.21.15.107:0/3625164819 wait complete. 2023-11-24T08:16:10.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:16:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.232+0000 7f9edd4da640 1 Processor -- start 2023-11-24T08:16:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.232+0000 7f9edd4da640 1 -- start start 2023-11-24T08:16:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.234+0000 7f9edd4da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cf450 0x7f9ed80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.234+0000 7f9edd4da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed80cece0 con 0x7f9ed815c7f0 2023-11-24T08:16:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.234+0000 7f9edd4da640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed806d040 con 0x7f9ed80cdfb0 2023-11-24T08:16:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.234+0000 7f9edd4da640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80cfe40 con 0x7f9ed80cf450 2023-11-24T08:16:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9ed815c7f0 0x7f9ed806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52036/0 (socket says 172.21.15.107:52036) 2023-11-24T08:16:10.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed6ffd640 1 -- 172.21.15.107:0/4121519920 learned_addr learned my addr 172.21.15.107:0/4121519920 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:10.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed67fc640 1 --2- 172.21.15.107:0/4121519920 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cf450 0x7f9ed80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316854536 0 0) 0x7f9ed80cece0 con 0x7f9ed815c7f0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec4003520 con 0x7f9ed815c7f0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728919743 0 0) 0x7f9ec4003520 con 0x7f9ed815c7f0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cf450 msgr2=0x7f9ed80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 --2- 172.21.15.107:0/4121519920 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cf450 0x7f9ed80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.235+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 >> v1:172.21.15.120:6789/0 conn(0x7f9ed80cdfb0 legacy=0x7f9ed80ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed806d320 con 0x7f9ed815c7f0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9ec8001670 con 0x7f9ed815c7f0 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 >> v1:172.21.15.107:6789/0 conn(0x7f9ed815c7f0 legacy=0x7f9ed806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ec4003970 0x7f9ec4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ec40047c0 0x7f9ec4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ec400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80cfe40 con 0x7f9ec4005180 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec400c020 con 0x7f9ec4003970 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec400c1e0 con 0x7f9ec40047c0 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ec400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed6ffd640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ec4003970 0x7f9ec4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 -- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ec40047c0 msgr2=0x7f9ec4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed67fc640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ec40047c0 0x7f9ec4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ec40047c0 0x7f9ec4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 -- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ec4003970 msgr2=0x7f9ec4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ec4003970 0x7f9ec4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.236+0000 7f9ed77fe640 1 -- 172.21.15.107:0/4121519920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4003520 con 0x7f9ec4005180 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.237+0000 7f9ed77fe640 1 -- 172.21.15.107:0/4121519920 reap_dead start 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9ed77fe640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ec400b980 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0001e50 tx=0x7f9ec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4364b59e04feb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0003b70 con 0x7f9ec4005180 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed0003d10 con 0x7f9ec4005180 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9ed5ffb640 1 -- 172.21.15.107:0/4121519920 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0005310 con 0x7f9ec4005180 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9edd4da640 1 -- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 msgr2=0x7f9ec400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.238+0000 7f9edd4da640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ec400b980 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0001e50 tx=0x7f9ec40057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.239+0000 7f9edd4da640 1 -- 172.21.15.107:0/4121519920 shutdown_connections 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.239+0000 7f9edd4da640 1 --2- 172.21.15.107:0/4121519920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ec400b980 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.239+0000 7f9edd4da640 1 -- 172.21.15.107:0/4121519920 >> 172.21.15.107:0/4121519920 conn(0x7f9ed806a920 msgr2=0x7f9ed806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:10.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.239+0000 7f9edd4da640 1 -- 172.21.15.107:0/4121519920 shutdown_connections 2023-11-24T08:16:10.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.239+0000 7f9edd4da640 1 -- 172.21.15.107:0/4121519920 wait complete. 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.240+0000 7f9edd4da640 1 Processor -- start 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 -- start start 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed8154720 0x7f9ed81571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed80afd90 0x7f9ed80b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80b0750 con 0x7f9ec4005180 2023-11-24T08:16:10.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8154ba0 con 0x7f9ed8154720 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.241+0000 7f9edd4da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed81555d0 con 0x7f9ed80afd90 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35428/0 (socket says 172.21.15.107:35428) 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 -- 172.21.15.107:0/1612261454 learned_addr learned my addr 172.21.15.107:0/1612261454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 -- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed80afd90 msgr2=0x7f9ed80b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed80afd90 0x7f9ed80b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 -- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed8154720 msgr2=0x7f9ed81571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed8154720 0x7f9ed81571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 -- 172.21.15.107:0/1612261454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed80acf10 con 0x7f9ec4005180 2023-11-24T08:16:10.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.242+0000 7f9ed6ffd640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800e6c0 tx=0x7f9ec800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.243+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800c6c0 con 0x7f9ec4005180 2023-11-24T08:16:10.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.243+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec800c840 con 0x7f9ec4005180 2023-11-24T08:16:10.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.243+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800cbf0 con 0x7f9ec4005180 2023-11-24T08:16:10.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.243+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed80ad1f0 con 0x7f9ec4005180 2023-11-24T08:16:10.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.243+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed80ad7b0 con 0x7f9ec4005180 2023-11-24T08:16:10.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.244+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed806db20 con 0x7f9ec4005180 2023-11-24T08:16:10.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.245+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f9ec800cd50 con 0x7f9ec4005180 2023-11-24T08:16:10.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.246+0000 7f9ecf7fe640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 0x7f9eb806e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.246+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ec8088620 con 0x7f9ec4005180 2023-11-24T08:16:10.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.246+0000 7f9ed67fc640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 0x7f9eb806e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.247+0000 7f9ed67fc640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 0x7f9eb806e820 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80cbfe0 tx=0x7f9ec0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.251+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ec8054d80 con 0x7f9ec4005180 2023-11-24T08:16:10.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.441+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ed8067470 con 0x7f9ec4005180 2023-11-24T08:16:10.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.444+0000 7f9ecf7fe640 1 -- 172.21.15.107:0/1612261454 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f9ec80583a0 con 0x7f9ec4005180 2023-11-24T08:16:10.447 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:10.447 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":147,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:16:07.484877+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:16:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 msgr2=0x7f9eb806e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 0x7f9eb806e820 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80cbfe0 tx=0x7f9ec0008040 comp rx=0 tx=0).stop 2023-11-24T08:16:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 msgr2=0x7f9ed80adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800e6c0 tx=0x7f9ec800eb80 comp rx=0 tx=0).stop 2023-11-24T08:16:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 shutdown_connections 2023-11-24T08:16:10.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9eb806c360 0x7f9eb806e820 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed80afd90 0x7f9ed80b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed8154720 0x7f9ed81571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 --2- 172.21.15.107:0/1612261454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ec4005180 0x7f9ed80adfb0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 >> 172.21.15.107:0/1612261454 conn(0x7f9ed806a920 msgr2=0x7f9ed80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:10.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.446+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 shutdown_connections 2023-11-24T08:16:10.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.447+0000 7f9edd4da640 1 -- 172.21.15.107:0/1612261454 wait complete. 2023-11-24T08:16:10.467 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:16:10.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:16:10.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.672+0000 7fe3b6fdc640 1 Processor -- start 2023-11-24T08:16:10.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.672+0000 7fe3b6fdc640 1 -- start start 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3b6fdc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3b015c7f0 0x7fe3b006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3b6fdc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b00cece0 con 0x7fe3b00cf450 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3b6fdc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b006d040 con 0x7fe3b00ce6c0 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3b6fdc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b00ceaf0 con 0x7fe3b015c7f0 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3affff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3b00cf450 0x7fe3b00cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52044/0 (socket says 172.21.15.107:52044) 2023-11-24T08:16:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3affff640 1 -- 172.21.15.107:0/2535402961 learned_addr learned my addr 172.21.15.107:0/2535402961 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.674+0000 7fe3b4d78640 1 --2- 172.21.15.107:0/2535402961 >> v2:172.21.15.120:3300/0 conn(0x7fe3b015c7f0 0x7fe3b006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596449601 0 0) 0x7fe3b00cece0 con 0x7fe3b00cf450 2023-11-24T08:16:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a0003520 con 0x7fe3b00cf450 2023-11-24T08:16:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245497801 0 0) 0x7fe3a0003520 con 0x7fe3b00cf450 2023-11-24T08:16:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 >> v2:172.21.15.120:3300/0 conn(0x7fe3b015c7f0 msgr2=0x7fe3b006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 --2- 172.21.15.107:0/2535402961 >> v2:172.21.15.120:3300/0 conn(0x7fe3b015c7f0 0x7fe3b006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 >> v1:172.21.15.120:6789/0 conn(0x7fe3b00ce6c0 legacy=0x7fe3b00714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b006d320 con 0x7fe3b00cf450 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe39c001670 con 0x7fe3b00cf450 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 >> v1:172.21.15.107:6789/0 conn(0x7fe3b00cf450 legacy=0x7fe3b00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a0003970 0x7fe3a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a00047c0 0x7fe3a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 0x7fe3a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a000c0a0 con 0x7fe3a0005180 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a000c210 con 0x7fe3a00047c0 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.675+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a000c3d0 con 0x7fe3a0003970 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 0x7fe3a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3affff640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a00047c0 0x7fe3a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b4d78640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a0003970 0x7fe3a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 -- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a0003970 msgr2=0x7fe3a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a0003970 0x7fe3a0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 -- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a00047c0 msgr2=0x7fe3a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a00047c0 0x7fe3a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3b5579640 1 -- 172.21.15.107:0/2535402961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0003520 con 0x7fe3a0005180 2023-11-24T08:16:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.676+0000 7fe3affff640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a00047c0 0x7fe3a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.677+0000 7fe3b5579640 1 -- 172.21.15.107:0/2535402961 reap_dead start 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.677+0000 7fe3b5579640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 0x7fe3a000b980 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8001e50 tx=0x7fe3a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba8f196b279eeed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.677+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8003b70 con 0x7fe3a0005180 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.677+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8003d10 con 0x7fe3a0005180 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.677+0000 7fe3af7fe640 1 -- 172.21.15.107:0/2535402961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8005310 con 0x7fe3a0005180 2023-11-24T08:16:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 msgr2=0x7fe3a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 0x7fe3a000b980 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8001e50 tx=0x7fe3a00057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/2535402961 shutdown_connections 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/2535402961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3a0005180 0x7fe3a000b980 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/2535402961 >> 172.21.15.107:0/2535402961 conn(0x7fe3b006a920 msgr2=0x7fe3b006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/2535402961 shutdown_connections 2023-11-24T08:16:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.678+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/2535402961 wait complete. 2023-11-24T08:16:10.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.680+0000 7fe3b6fdc640 1 Processor -- start 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.680+0000 7fe3b6fdc640 1 -- start start 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a0005180 0x7fe3b00ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3b00af750 0x7fe3b00afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00b0110 con 0x7fe3b01574c0 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00ae7b0 con 0x7fe3a0005180 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3b6fdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00ad710 con 0x7fe3b00af750 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35442/0 (socket says 172.21.15.107:35442) 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.681+0000 7fe3affff640 1 -- 172.21.15.107:0/1994985938 learned_addr learned my addr 172.21.15.107:0/1994985938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 -- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3b00af750 msgr2=0x7fe3b00afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3b00af750 0x7fe3b00afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 -- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a0005180 msgr2=0x7fe3b00ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a0005180 0x7fe3b00ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 -- 172.21.15.107:0/1994985938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00ad9f0 con 0x7fe3b01574c0 2023-11-24T08:16:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3affff640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00d810 tx=0x7fe39c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0074e0 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c007640 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b00adcd0 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c004a00 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.682+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b00ae290 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.684+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b0067470 con 0x7fe3b01574c0 2023-11-24T08:16:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.684+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fe39c004b60 con 0x7fe3b01574c0 2023-11-24T08:16:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.686+0000 7fe3ad7fa640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 0x7fe39406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.686+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe39c0880a0 con 0x7fe3b01574c0 2023-11-24T08:16:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.686+0000 7fe3b4d78640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 0x7fe39406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.687+0000 7fe3b4d78640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 0x7fe39406e7d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4006a90 tx=0x7fe3a4006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:10.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.690+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe39c054c00 con 0x7fe3b01574c0 2023-11-24T08:16:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.879+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3b0159400 con 0x7fe39406c310 2023-11-24T08:16:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.901+0000 7fe3ad7fa640 1 -- 172.21.15.107:0/1994985938 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287633 (secure 0 0 0) 0x7fe3b0159400 con 0x7fe39406c310 2023-11-24T08:16:10.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:16:10.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.906+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 msgr2=0x7fe39406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.906+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 0x7fe39406e7d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4006a90 tx=0x7fe3a4006a20 comp rx=0 tx=0).stop 2023-11-24T08:16:10.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.906+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 msgr2=0x7fe3b01564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:10.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.906+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00d810 tx=0x7fe39c00dcd0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 shutdown_connections 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe39406c310 0x7fe39406e7d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3b00af750 0x7fe3b00afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3b01574c0 0x7fe3b01564b0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 --2- 172.21.15.107:0/1994985938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a0005180 0x7fe3b00ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 >> 172.21.15.107:0/1994985938 conn(0x7fe3b006a920 msgr2=0x7fe3b015dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 shutdown_connections 2023-11-24T08:16:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:10.907+0000 7fe3b6fdc640 1 -- 172.21.15.107:0/1994985938 wait complete. 2023-11-24T08:16:10.928 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2023-11-24T08:16:09.645042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294148,"kb_used_data":15580,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754680572,"statfs":{"total":773094113280,"available":772792905728,"internally_reserved":0,"allocated":15953920,"data_stored":7633080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":285134398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.213155"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407609+0000","last_change":"2023-11-24T08:16:06.503554+0000","last_active":"2023-11-24T08:16:08.407609+0000","last_peered":"2023-11-24T08:16:08.407609+0000","last_clean":"2023-11-24T08:16:08.407609+0000","last_became_active":"2023-11-24T08:16:06.502985+0000","last_became_peered":"2023-11-24T08:16:06.502985+0000","last_unstale":"2023-11-24T08:16:08.407609+0000","last_undegraded":"2023-11-24T08:16:08.407609+0000","last_fullsized":"2023-11-24T08:16:08.407609+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:42.131310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747505+0000","last_change":"2023-11-24T08:16:06.497798+0000","last_active":"2023-11-24T08:16:07.747505+0000","last_peered":"2023-11-24T08:16:07.747505+0000","last_clean":"2023-11-24T08:16:07.747505+0000","last_became_active":"2023-11-24T08:16:06.497537+0000","last_became_peered":"2023-11-24T08:16:06.497537+0000","last_unstale":"2023-11-24T08:16:07.747505+0000","last_undegraded":"2023-11-24T08:16:07.747505+0000","last_fullsized":"2023-11-24T08:16:07.747505+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:02.014109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405676+0000","last_change":"2023-11-24T08:16:06.492396+0000","last_active":"2023-11-24T08:16:08.405676+0000","last_peered":"2023-11-24T08:16:08.405676+0000","last_clean":"2023-11-24T08:16:08.405676+0000","last_became_active":"2023-11-24T08:16:06.492123+0000","last_became_peered":"2023-11-24T08:16:06.492123+0000","last_unstale":"2023-11-24T08:16:08.405676+0000","last_undegraded":"2023-11-24T08:16:08.405676+0000","last_fullsized":"2023-11-24T08:16:08.405676+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:39:55.657306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503234+0000","last_change":"2023-11-24T08:16:06.495470+0000","last_active":"2023-11-24T08:16:07.503234+0000","last_peered":"2023-11-24T08:16:07.503234+0000","last_clean":"2023-11-24T08:16:07.503234+0000","last_became_active":"2023-11-24T08:16:06.495172+0000","last_became_peered":"2023-11-24T08:16:06.495172+0000","last_unstale":"2023-11-24T08:16:07.503234+0000","last_undegraded":"2023-11-24T08:16:07.503234+0000","last_fullsized":"2023-11-24T08:16:07.503234+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:36.772056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500181+0000","last_change":"2023-11-24T08:16:06.494450+0000","last_active":"2023-11-24T08:16:07.500181+0000","last_peered":"2023-11-24T08:16:07.500181+0000","last_clean":"2023-11-24T08:16:07.500181+0000","last_became_active":"2023-11-24T08:16:06.494150+0000","last_became_peered":"2023-11-24T08:16:06.494150+0000","last_unstale":"2023-11-24T08:16:07.500181+0000","last_undegraded":"2023-11-24T08:16:07.500181+0000","last_fullsized":"2023-11-24T08:16:07.500181+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:29.279322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406781+0000","last_change":"2023-11-24T08:16:06.503101+0000","last_active":"2023-11-24T08:16:08.406781+0000","last_peered":"2023-11-24T08:16:08.406781+0000","last_clean":"2023-11-24T08:16:08.406781+0000","last_became_active":"2023-11-24T08:16:06.502803+0000","last_became_peered":"2023-11-24T08:16:06.502803+0000","last_unstale":"2023-11-24T08:16:08.406781+0000","last_undegraded":"2023-11-24T08:16:08.406781+0000","last_fullsized":"2023-11-24T08:16:08.406781+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:49.195067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343008+0000","last_change":"2023-11-24T08:16:06.500807+0000","last_active":"2023-11-24T08:16:08.343008+0000","last_peered":"2023-11-24T08:16:08.343008+0000","last_clean":"2023-11-24T08:16:08.343008+0000","last_became_active":"2023-11-24T08:16:06.494871+0000","last_became_peered":"2023-11-24T08:16:06.494871+0000","last_unstale":"2023-11-24T08:16:08.343008+0000","last_undegraded":"2023-11-24T08:16:08.343008+0000","last_fullsized":"2023-11-24T08:16:08.343008+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:07.776858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406066+0000","last_change":"2023-11-24T08:16:06.497881+0000","last_active":"2023-11-24T08:16:08.406066+0000","last_peered":"2023-11-24T08:16:08.406066+0000","last_clean":"2023-11-24T08:16:08.406066+0000","last_became_active":"2023-11-24T08:16:06.494609+0000","last_became_peered":"2023-11-24T08:16:06.494609+0000","last_unstale":"2023-11-24T08:16:08.406066+0000","last_undegraded":"2023-11-24T08:16:08.406066+0000","last_fullsized":"2023-11-24T08:16:08.406066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:47.882204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747635+0000","last_change":"2023-11-24T08:16:06.502162+0000","last_active":"2023-11-24T08:16:07.747635+0000","last_peered":"2023-11-24T08:16:07.747635+0000","last_clean":"2023-11-24T08:16:07.747635+0000","last_became_active":"2023-11-24T08:16:06.501616+0000","last_became_peered":"2023-11-24T08:16:06.501616+0000","last_unstale":"2023-11-24T08:16:07.747635+0000","last_undegraded":"2023-11-24T08:16:07.747635+0000","last_fullsized":"2023-11-24T08:16:07.747635+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:35.786642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521768+0000","last_change":"2023-11-24T08:16:06.499678+0000","last_active":"2023-11-24T08:16:07.521768+0000","last_peered":"2023-11-24T08:16:07.521768+0000","last_clean":"2023-11-24T08:16:07.521768+0000","last_became_active":"2023-11-24T08:16:06.499374+0000","last_became_peered":"2023-11-24T08:16:06.499374+0000","last_unstale":"2023-11-24T08:16:07.521768+0000","last_undegraded":"2023-11-24T08:16:07.521768+0000","last_fullsized":"2023-11-24T08:16:07.521768+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:11:39.961690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406747+0000","last_change":"2023-11-24T08:16:06.500807+0000","last_active":"2023-11-24T08:16:08.406747+0000","last_peered":"2023-11-24T08:16:08.406747+0000","last_clean":"2023-11-24T08:16:08.406747+0000","last_became_active":"2023-11-24T08:16:06.500526+0000","last_became_peered":"2023-11-24T08:16:06.500526+0000","last_unstale":"2023-11-24T08:16:08.406747+0000","last_undegraded":"2023-11-24T08:16:08.406747+0000","last_fullsized":"2023-11-24T08:16:08.406747+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:55.688570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521093+0000","last_change":"2023-11-24T08:16:06.498544+0000","last_active":"2023-11-24T08:16:07.521093+0000","last_peered":"2023-11-24T08:16:07.521093+0000","last_clean":"2023-11-24T08:16:07.521093+0000","last_became_active":"2023-11-24T08:16:06.498232+0000","last_became_peered":"2023-11-24T08:16:06.498232+0000","last_unstale":"2023-11-24T08:16:07.521093+0000","last_undegraded":"2023-11-24T08:16:07.521093+0000","last_fullsized":"2023-11-24T08:16:07.521093+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:42.672960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500687+0000","last_change":"2023-11-24T08:16:06.495536+0000","last_active":"2023-11-24T08:16:07.500687+0000","last_peered":"2023-11-24T08:16:07.500687+0000","last_clean":"2023-11-24T08:16:07.500687+0000","last_became_active":"2023-11-24T08:16:06.494260+0000","last_became_peered":"2023-11-24T08:16:06.494260+0000","last_unstale":"2023-11-24T08:16:07.500687+0000","last_undegraded":"2023-11-24T08:16:07.500687+0000","last_fullsized":"2023-11-24T08:16:07.500687+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:32.612410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502642+0000","last_change":"2023-11-24T08:16:06.492403+0000","last_active":"2023-11-24T08:16:07.502642+0000","last_peered":"2023-11-24T08:16:07.502642+0000","last_clean":"2023-11-24T08:16:07.502642+0000","last_became_active":"2023-11-24T08:16:06.492158+0000","last_became_peered":"2023-11-24T08:16:06.492158+0000","last_unstale":"2023-11-24T08:16:07.502642+0000","last_undegraded":"2023-11-24T08:16:07.502642+0000","last_fullsized":"2023-11-24T08:16:07.502642+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.885982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520758+0000","last_change":"2023-11-24T08:16:06.506683+0000","last_active":"2023-11-24T08:16:07.520758+0000","last_peered":"2023-11-24T08:16:07.520758+0000","last_clean":"2023-11-24T08:16:07.520758+0000","last_became_active":"2023-11-24T08:16:06.506236+0000","last_became_peered":"2023-11-24T08:16:06.506236+0000","last_unstale":"2023-11-24T08:16:07.520758+0000","last_undegraded":"2023-11-24T08:16:07.520758+0000","last_fullsized":"2023-11-24T08:16:07.520758+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:50.803641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404973+0000","last_change":"2023-11-24T08:16:06.503847+0000","last_active":"2023-11-24T08:16:08.404973+0000","last_peered":"2023-11-24T08:16:08.404973+0000","last_clean":"2023-11-24T08:16:08.404973+0000","last_became_active":"2023-11-24T08:16:06.503554+0000","last_became_peered":"2023-11-24T08:16:06.503554+0000","last_unstale":"2023-11-24T08:16:08.404973+0000","last_undegraded":"2023-11-24T08:16:08.404973+0000","last_fullsized":"2023-11-24T08:16:08.404973+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:27.167457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522109+0000","last_change":"2023-11-24T08:16:06.499283+0000","last_active":"2023-11-24T08:16:07.522109+0000","last_peered":"2023-11-24T08:16:07.522109+0000","last_clean":"2023-11-24T08:16:07.522109+0000","last_became_active":"2023-11-24T08:16:06.498769+0000","last_became_peered":"2023-11-24T08:16:06.498769+0000","last_unstale":"2023-11-24T08:16:07.522109+0000","last_undegraded":"2023-11-24T08:16:07.522109+0000","last_fullsized":"2023-11-24T08:16:07.522109+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:05:29.257509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405896+0000","last_change":"2023-11-24T08:16:06.497994+0000","last_active":"2023-11-24T08:16:08.405896+0000","last_peered":"2023-11-24T08:16:08.405896+0000","last_clean":"2023-11-24T08:16:08.405896+0000","last_became_active":"2023-11-24T08:16:06.497705+0000","last_became_peered":"2023-11-24T08:16:06.497705+0000","last_unstale":"2023-11-24T08:16:08.405896+0000","last_undegraded":"2023-11-24T08:16:08.405896+0000","last_fullsized":"2023-11-24T08:16:08.405896+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:09.039086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407488+0000","last_change":"2023-11-24T08:16:06.496118+0000","last_active":"2023-11-24T08:16:08.407488+0000","last_peered":"2023-11-24T08:16:08.407488+0000","last_clean":"2023-11-24T08:16:08.407488+0000","last_became_active":"2023-11-24T08:16:06.495803+0000","last_became_peered":"2023-11-24T08:16:06.495803+0000","last_unstale":"2023-11-24T08:16:08.407488+0000","last_undegraded":"2023-11-24T08:16:08.407488+0000","last_fullsized":"2023-11-24T08:16:08.407488+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:33.919970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406128+0000","last_change":"2023-11-24T08:16:06.492617+0000","last_active":"2023-11-24T08:16:08.406128+0000","last_peered":"2023-11-24T08:16:08.406128+0000","last_clean":"2023-11-24T08:16:08.406128+0000","last_became_active":"2023-11-24T08:16:06.492320+0000","last_became_peered":"2023-11-24T08:16:06.492320+0000","last_unstale":"2023-11-24T08:16:08.406128+0000","last_undegraded":"2023-11-24T08:16:08.406128+0000","last_fullsized":"2023-11-24T08:16:08.406128+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:02.175235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521408+0000","last_change":"2023-11-24T08:16:06.501869+0000","last_active":"2023-11-24T08:16:07.521408+0000","last_peered":"2023-11-24T08:16:07.521408+0000","last_clean":"2023-11-24T08:16:07.521408+0000","last_became_active":"2023-11-24T08:16:06.501607+0000","last_became_peered":"2023-11-24T08:16:06.501607+0000","last_unstale":"2023-11-24T08:16:07.521408+0000","last_undegraded":"2023-11-24T08:16:07.521408+0000","last_fullsized":"2023-11-24T08:16:07.521408+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:02.014109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502645+0000","last_change":"2023-11-24T08:16:06.491776+0000","last_active":"2023-11-24T08:16:07.502645+0000","last_peered":"2023-11-24T08:16:07.502645+0000","last_clean":"2023-11-24T08:16:07.502645+0000","last_became_active":"2023-11-24T08:16:06.491502+0000","last_became_peered":"2023-11-24T08:16:06.491502+0000","last_unstale":"2023-11-24T08:16:07.502645+0000","last_undegraded":"2023-11-24T08:16:07.502645+0000","last_fullsized":"2023-11-24T08:16:07.502645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:14:56.087388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502334+0000","last_change":"2023-11-24T08:16:06.493445+0000","last_active":"2023-11-24T08:16:07.502334+0000","last_peered":"2023-11-24T08:16:07.502334+0000","last_clean":"2023-11-24T08:16:07.502334+0000","last_became_active":"2023-11-24T08:16:06.493156+0000","last_became_peered":"2023-11-24T08:16:06.493156+0000","last_unstale":"2023-11-24T08:16:07.502334+0000","last_undegraded":"2023-11-24T08:16:07.502334+0000","last_fullsized":"2023-11-24T08:16:07.502334+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:09.039086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520740+0000","last_change":"2023-11-24T08:16:06.502468+0000","last_active":"2023-11-24T08:16:07.520740+0000","last_peered":"2023-11-24T08:16:07.520740+0000","last_clean":"2023-11-24T08:16:07.520740+0000","last_became_active":"2023-11-24T08:16:06.502201+0000","last_became_peered":"2023-11-24T08:16:06.502201+0000","last_unstale":"2023-11-24T08:16:07.520740+0000","last_undegraded":"2023-11-24T08:16:07.520740+0000","last_fullsized":"2023-11-24T08:16:07.520740+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:04.727045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405838+0000","last_change":"2023-11-24T08:16:06.501813+0000","last_active":"2023-11-24T08:16:08.405838+0000","last_peered":"2023-11-24T08:16:08.405838+0000","last_clean":"2023-11-24T08:16:08.405838+0000","last_became_active":"2023-11-24T08:16:06.501545+0000","last_became_peered":"2023-11-24T08:16:06.501545+0000","last_unstale":"2023-11-24T08:16:08.405838+0000","last_undegraded":"2023-11-24T08:16:08.405838+0000","last_fullsized":"2023-11-24T08:16:08.405838+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:37.222297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407055+0000","last_change":"2023-11-24T08:16:06.494379+0000","last_active":"2023-11-24T08:16:08.407055+0000","last_peered":"2023-11-24T08:16:08.407055+0000","last_clean":"2023-11-24T08:16:08.407055+0000","last_became_active":"2023-11-24T08:16:06.494103+0000","last_became_peered":"2023-11-24T08:16:06.494103+0000","last_unstale":"2023-11-24T08:16:08.407055+0000","last_undegraded":"2023-11-24T08:16:08.407055+0000","last_fullsized":"2023-11-24T08:16:08.407055+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:09.500462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405565+0000","last_change":"2023-11-24T08:16:06.494358+0000","last_active":"2023-11-24T08:16:08.405565+0000","last_peered":"2023-11-24T08:16:08.405565+0000","last_clean":"2023-11-24T08:16:08.405565+0000","last_became_active":"2023-11-24T08:16:06.493935+0000","last_became_peered":"2023-11-24T08:16:06.493935+0000","last_unstale":"2023-11-24T08:16:08.405565+0000","last_undegraded":"2023-11-24T08:16:08.405565+0000","last_fullsized":"2023-11-24T08:16:08.405565+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:51:05.860184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343300+0000","last_change":"2023-11-24T08:16:06.491448+0000","last_active":"2023-11-24T08:16:08.343300+0000","last_peered":"2023-11-24T08:16:08.343300+0000","last_clean":"2023-11-24T08:16:08.343300+0000","last_became_active":"2023-11-24T08:16:06.491158+0000","last_became_peered":"2023-11-24T08:16:06.491158+0000","last_unstale":"2023-11-24T08:16:08.343300+0000","last_undegraded":"2023-11-24T08:16:08.343300+0000","last_fullsized":"2023-11-24T08:16:08.343300+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:36.139572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406121+0000","last_change":"2023-11-24T08:16:06.499555+0000","last_active":"2023-11-24T08:16:08.406121+0000","last_peered":"2023-11-24T08:16:08.406121+0000","last_clean":"2023-11-24T08:16:08.406121+0000","last_became_active":"2023-11-24T08:16:06.499283+0000","last_became_peered":"2023-11-24T08:16:06.499283+0000","last_unstale":"2023-11-24T08:16:08.406121+0000","last_undegraded":"2023-11-24T08:16:08.406121+0000","last_fullsized":"2023-11-24T08:16:08.406121+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:46:09.671464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503008+0000","last_change":"2023-11-24T08:16:06.503014+0000","last_active":"2023-11-24T08:16:07.503008+0000","last_peered":"2023-11-24T08:16:07.503008+0000","last_clean":"2023-11-24T08:16:07.503008+0000","last_became_active":"2023-11-24T08:16:06.502201+0000","last_became_peered":"2023-11-24T08:16:06.502201+0000","last_unstale":"2023-11-24T08:16:07.503008+0000","last_undegraded":"2023-11-24T08:16:07.503008+0000","last_fullsized":"2023-11-24T08:16:07.503008+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:52.832521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747346+0000","last_change":"2023-11-24T08:16:06.494306+0000","last_active":"2023-11-24T08:16:07.747346+0000","last_peered":"2023-11-24T08:16:07.747346+0000","last_clean":"2023-11-24T08:16:07.747346+0000","last_became_active":"2023-11-24T08:16:06.494021+0000","last_became_peered":"2023-11-24T08:16:06.494021+0000","last_unstale":"2023-11-24T08:16:07.747346+0000","last_undegraded":"2023-11-24T08:16:07.747346+0000","last_fullsized":"2023-11-24T08:16:07.747346+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:42.672960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405074+0000","last_change":"2023-11-24T08:16:06.489660+0000","last_active":"2023-11-24T08:16:08.405074+0000","last_peered":"2023-11-24T08:16:08.405074+0000","last_clean":"2023-11-24T08:16:08.405074+0000","last_became_active":"2023-11-24T08:16:06.489257+0000","last_became_peered":"2023-11-24T08:16:06.489257+0000","last_unstale":"2023-11-24T08:16:08.405074+0000","last_undegraded":"2023-11-24T08:16:08.405074+0000","last_fullsized":"2023-11-24T08:16:08.405074+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:26:57.091628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521894+0000","last_change":"2023-11-24T08:16:06.499474+0000","last_active":"2023-11-24T08:16:07.521894+0000","last_peered":"2023-11-24T08:16:07.521894+0000","last_clean":"2023-11-24T08:16:07.521894+0000","last_became_active":"2023-11-24T08:16:06.498962+0000","last_became_peered":"2023-11-24T08:16:06.498962+0000","last_unstale":"2023-11-24T08:16:07.521894+0000","last_undegraded":"2023-11-24T08:16:07.521894+0000","last_fullsized":"2023-11-24T08:16:07.521894+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:09:48.423417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"146'4","reported_seq":16,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521880+0000","last_change":"2023-11-24T08:16:05.473035+0000","last_active":"2023-11-24T08:16:07.521880+0000","last_peered":"2023-11-24T08:16:07.521880+0000","last_clean":"2023-11-24T08:16:07.521880+0000","last_became_active":"2023-11-24T08:16:05.472688+0000","last_became_peered":"2023-11-24T08:16:05.472688+0000","last_unstale":"2023-11-24T08:16:07.521880+0000","last_undegraded":"2023-11-24T08:16:07.521880+0000","last_fullsized":"2023-11-24T08:16:07.521880+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:26.583836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.748006+0000","last_change":"2023-11-24T08:16:06.499172+0000","last_active":"2023-11-24T08:16:07.748006+0000","last_peered":"2023-11-24T08:16:07.748006+0000","last_clean":"2023-11-24T08:16:07.748006+0000","last_became_active":"2023-11-24T08:16:06.498908+0000","last_became_peered":"2023-11-24T08:16:06.498908+0000","last_unstale":"2023-11-24T08:16:07.748006+0000","last_undegraded":"2023-11-24T08:16:07.748006+0000","last_fullsized":"2023-11-24T08:16:07.748006+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:09:48.423417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747602+0000","last_change":"2023-11-24T08:16:05.473712+0000","last_active":"2023-11-24T08:16:07.747602+0000","last_peered":"2023-11-24T08:16:07.747602+0000","last_clean":"2023-11-24T08:16:07.747602+0000","last_became_active":"2023-11-24T08:16:05.473227+0000","last_became_peered":"2023-11-24T08:16:05.473227+0000","last_unstale":"2023-11-24T08:16:07.747602+0000","last_undegraded":"2023-11-24T08:16:07.747602+0000","last_fullsized":"2023-11-24T08:16:07.747602+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:01.784224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406297+0000","last_change":"2023-11-24T08:16:06.508929+0000","last_active":"2023-11-24T08:16:08.406297+0000","last_peered":"2023-11-24T08:16:08.406297+0000","last_clean":"2023-11-24T08:16:08.406297+0000","last_became_active":"2023-11-24T08:16:06.506946+0000","last_became_peered":"2023-11-24T08:16:06.506946+0000","last_unstale":"2023-11-24T08:16:08.406297+0000","last_undegraded":"2023-11-24T08:16:08.406297+0000","last_fullsized":"2023-11-24T08:16:08.406297+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:59.854885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502979+0000","last_change":"2023-11-24T08:16:05.469028+0000","last_active":"2023-11-24T08:16:07.502979+0000","last_peered":"2023-11-24T08:16:07.502979+0000","last_clean":"2023-11-24T08:16:07.502979+0000","last_became_active":"2023-11-24T08:16:05.468740+0000","last_became_peered":"2023-11-24T08:16:05.468740+0000","last_unstale":"2023-11-24T08:16:07.502979+0000","last_undegraded":"2023-11-24T08:16:07.502979+0000","last_fullsized":"2023-11-24T08:16:07.502979+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:08.488846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521596+0000","last_change":"2023-11-24T08:16:06.494087+0000","last_active":"2023-11-24T08:16:07.521596+0000","last_peered":"2023-11-24T08:16:07.521596+0000","last_clean":"2023-11-24T08:16:07.521596+0000","last_became_active":"2023-11-24T08:16:06.493751+0000","last_became_peered":"2023-11-24T08:16:06.493751+0000","last_unstale":"2023-11-24T08:16:07.521596+0000","last_undegraded":"2023-11-24T08:16:07.521596+0000","last_fullsized":"2023-11-24T08:16:07.521596+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:35.786642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503331+0000","last_change":"2023-11-24T08:16:05.464029+0000","last_active":"2023-11-24T08:16:07.503331+0000","last_peered":"2023-11-24T08:16:07.503331+0000","last_clean":"2023-11-24T08:16:07.503331+0000","last_became_active":"2023-11-24T08:16:05.463581+0000","last_became_peered":"2023-11-24T08:16:05.463581+0000","last_unstale":"2023-11-24T08:16:07.503331+0000","last_undegraded":"2023-11-24T08:16:07.503331+0000","last_fullsized":"2023-11-24T08:16:07.503331+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:39:54.645689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747425+0000","last_change":"2023-11-24T08:16:06.493299+0000","last_active":"2023-11-24T08:16:07.747425+0000","last_peered":"2023-11-24T08:16:07.747425+0000","last_clean":"2023-11-24T08:16:07.747425+0000","last_became_active":"2023-11-24T08:16:06.492844+0000","last_became_peered":"2023-11-24T08:16:06.492844+0000","last_unstale":"2023-11-24T08:16:07.747425+0000","last_undegraded":"2023-11-24T08:16:07.747425+0000","last_fullsized":"2023-11-24T08:16:07.747425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:30.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406022+0000","last_change":"2023-11-24T08:16:05.475958+0000","last_active":"2023-11-24T08:16:08.406022+0000","last_peered":"2023-11-24T08:16:08.406022+0000","last_clean":"2023-11-24T08:16:08.406022+0000","last_became_active":"2023-11-24T08:16:05.475670+0000","last_became_peered":"2023-11-24T08:16:05.475670+0000","last_unstale":"2023-11-24T08:16:08.406022+0000","last_undegraded":"2023-11-24T08:16:08.406022+0000","last_fullsized":"2023-11-24T08:16:08.406022+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:15:33.634025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502846+0000","last_change":"2023-11-24T08:16:06.495288+0000","last_active":"2023-11-24T08:16:07.502846+0000","last_peered":"2023-11-24T08:16:07.502846+0000","last_clean":"2023-11-24T08:16:07.502846+0000","last_became_active":"2023-11-24T08:16:06.494965+0000","last_became_peered":"2023-11-24T08:16:06.494965+0000","last_unstale":"2023-11-24T08:16:07.502846+0000","last_undegraded":"2023-11-24T08:16:07.502846+0000","last_fullsized":"2023-11-24T08:16:07.502846+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:40.831282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499382+0000","last_change":"2023-11-24T08:16:05.475703+0000","last_active":"2023-11-24T08:16:07.499382+0000","last_peered":"2023-11-24T08:16:07.499382+0000","last_clean":"2023-11-24T08:16:07.499382+0000","last_became_active":"2023-11-24T08:16:05.475382+0000","last_became_peered":"2023-11-24T08:16:05.475382+0000","last_unstale":"2023-11-24T08:16:07.499382+0000","last_undegraded":"2023-11-24T08:16:07.499382+0000","last_fullsized":"2023-11-24T08:16:07.499382+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:59.809160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747736+0000","last_change":"2023-11-24T08:16:06.493435+0000","last_active":"2023-11-24T08:16:07.747736+0000","last_peered":"2023-11-24T08:16:07.747736+0000","last_clean":"2023-11-24T08:16:07.747736+0000","last_became_active":"2023-11-24T08:16:06.493097+0000","last_became_peered":"2023-11-24T08:16:06.493097+0000","last_unstale":"2023-11-24T08:16:07.747736+0000","last_undegraded":"2023-11-24T08:16:07.747736+0000","last_fullsized":"2023-11-24T08:16:07.747736+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:11:39.961690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405428+0000","last_change":"2023-11-24T08:16:05.470646+0000","last_active":"2023-11-24T08:16:08.405428+0000","last_peered":"2023-11-24T08:16:08.405428+0000","last_clean":"2023-11-24T08:16:08.405428+0000","last_became_active":"2023-11-24T08:16:05.470003+0000","last_became_peered":"2023-11-24T08:16:05.470003+0000","last_unstale":"2023-11-24T08:16:08.405428+0000","last_undegraded":"2023-11-24T08:16:08.405428+0000","last_fullsized":"2023-11-24T08:16:08.405428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:23:02.214199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500030+0000","last_change":"2023-11-24T08:16:06.498757+0000","last_active":"2023-11-24T08:16:07.500030+0000","last_peered":"2023-11-24T08:16:07.500030+0000","last_clean":"2023-11-24T08:16:07.500030+0000","last_became_active":"2023-11-24T08:16:06.498499+0000","last_became_peered":"2023-11-24T08:16:06.498499+0000","last_unstale":"2023-11-24T08:16:07.500030+0000","last_undegraded":"2023-11-24T08:16:07.500030+0000","last_fullsized":"2023-11-24T08:16:07.500030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:25.869197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502172+0000","last_change":"2023-11-24T08:16:05.474788+0000","last_active":"2023-11-24T08:16:07.502172+0000","last_peered":"2023-11-24T08:16:07.502172+0000","last_clean":"2023-11-24T08:16:07.502172+0000","last_became_active":"2023-11-24T08:16:05.474469+0000","last_became_peered":"2023-11-24T08:16:05.474469+0000","last_unstale":"2023-11-24T08:16:07.502172+0000","last_undegraded":"2023-11-24T08:16:07.502172+0000","last_fullsized":"2023-11-24T08:16:07.502172+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:54.676954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500672+0000","last_change":"2023-11-24T08:16:06.505815+0000","last_active":"2023-11-24T08:16:07.500672+0000","last_peered":"2023-11-24T08:16:07.500672+0000","last_clean":"2023-11-24T08:16:07.500672+0000","last_became_active":"2023-11-24T08:16:06.505499+0000","last_became_peered":"2023-11-24T08:16:06.505499+0000","last_unstale":"2023-11-24T08:16:07.500672+0000","last_undegraded":"2023-11-24T08:16:07.500672+0000","last_fullsized":"2023-11-24T08:16:07.500672+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:29:07.362844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747944+0000","last_change":"2023-11-24T08:16:05.463018+0000","last_active":"2023-11-24T08:16:07.747944+0000","last_peered":"2023-11-24T08:16:07.747944+0000","last_clean":"2023-11-24T08:16:07.747944+0000","last_became_active":"2023-11-24T08:16:05.461567+0000","last_became_peered":"2023-11-24T08:16:05.461567+0000","last_unstale":"2023-11-24T08:16:07.747944+0000","last_undegraded":"2023-11-24T08:16:07.747944+0000","last_fullsized":"2023-11-24T08:16:07.747944+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:46.099656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521928+0000","last_change":"2023-11-24T08:16:06.499081+0000","last_active":"2023-11-24T08:16:07.521928+0000","last_peered":"2023-11-24T08:16:07.521928+0000","last_clean":"2023-11-24T08:16:07.521928+0000","last_became_active":"2023-11-24T08:16:06.498303+0000","last_became_peered":"2023-11-24T08:16:06.498303+0000","last_unstale":"2023-11-24T08:16:07.521928+0000","last_undegraded":"2023-11-24T08:16:07.521928+0000","last_fullsized":"2023-11-24T08:16:07.521928+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:11.352447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522103+0000","last_change":"2023-11-24T08:16:05.477279+0000","last_active":"2023-11-24T08:16:07.522103+0000","last_peered":"2023-11-24T08:16:07.522103+0000","last_clean":"2023-11-24T08:16:07.522103+0000","last_became_active":"2023-11-24T08:16:05.477022+0000","last_became_peered":"2023-11-24T08:16:05.477022+0000","last_unstale":"2023-11-24T08:16:07.522103+0000","last_undegraded":"2023-11-24T08:16:07.522103+0000","last_fullsized":"2023-11-24T08:16:07.522103+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:15.984683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405724+0000","last_change":"2023-11-24T08:16:06.494192+0000","last_active":"2023-11-24T08:16:08.405724+0000","last_peered":"2023-11-24T08:16:08.405724+0000","last_clean":"2023-11-24T08:16:08.405724+0000","last_became_active":"2023-11-24T08:16:06.493586+0000","last_became_peered":"2023-11-24T08:16:06.493586+0000","last_unstale":"2023-11-24T08:16:08.405724+0000","last_undegraded":"2023-11-24T08:16:08.405724+0000","last_fullsized":"2023-11-24T08:16:08.405724+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:28.523247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"146'6","reported_seq":18,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343624+0000","last_change":"2023-11-24T08:16:05.465947+0000","last_active":"2023-11-24T08:16:08.343624+0000","last_peered":"2023-11-24T08:16:08.343624+0000","last_clean":"2023-11-24T08:16:08.343624+0000","last_became_active":"2023-11-24T08:16:05.465669+0000","last_became_peered":"2023-11-24T08:16:05.465669+0000","last_unstale":"2023-11-24T08:16:08.343624+0000","last_undegraded":"2023-11-24T08:16:08.343624+0000","last_fullsized":"2023-11-24T08:16:08.343624+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:14.380302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405459+0000","last_change":"2023-11-24T08:16:06.498585+0000","last_active":"2023-11-24T08:16:08.405459+0000","last_peered":"2023-11-24T08:16:08.405459+0000","last_clean":"2023-11-24T08:16:08.405459+0000","last_became_active":"2023-11-24T08:16:06.498289+0000","last_became_peered":"2023-11-24T08:16:06.498289+0000","last_unstale":"2023-11-24T08:16:08.405459+0000","last_undegraded":"2023-11-24T08:16:08.405459+0000","last_fullsized":"2023-11-24T08:16:08.405459+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:26:13.648432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405478+0000","last_change":"2023-11-24T08:16:05.464377+0000","last_active":"2023-11-24T08:16:08.405478+0000","last_peered":"2023-11-24T08:16:08.405478+0000","last_clean":"2023-11-24T08:16:08.405478+0000","last_became_active":"2023-11-24T08:16:05.462468+0000","last_became_peered":"2023-11-24T08:16:05.462468+0000","last_unstale":"2023-11-24T08:16:08.405478+0000","last_undegraded":"2023-11-24T08:16:08.405478+0000","last_fullsized":"2023-11-24T08:16:08.405478+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:48.183451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747245+0000","last_change":"2023-11-24T08:16:06.510216+0000","last_active":"2023-11-24T08:16:07.747245+0000","last_peered":"2023-11-24T08:16:07.747245+0000","last_clean":"2023-11-24T08:16:07.747245+0000","last_became_active":"2023-11-24T08:16:06.509905+0000","last_became_peered":"2023-11-24T08:16:06.509905+0000","last_unstale":"2023-11-24T08:16:07.747245+0000","last_undegraded":"2023-11-24T08:16:07.747245+0000","last_fullsized":"2023-11-24T08:16:07.747245+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:50.803641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405396+0000","last_change":"2023-11-24T08:16:05.472489+0000","last_active":"2023-11-24T08:16:08.405396+0000","last_peered":"2023-11-24T08:16:08.405396+0000","last_clean":"2023-11-24T08:16:08.405396+0000","last_became_active":"2023-11-24T08:16:05.471978+0000","last_became_peered":"2023-11-24T08:16:05.471978+0000","last_unstale":"2023-11-24T08:16:08.405396+0000","last_undegraded":"2023-11-24T08:16:08.405396+0000","last_fullsized":"2023-11-24T08:16:08.405396+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:58.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499397+0000","last_change":"2023-11-24T08:16:06.492482+0000","last_active":"2023-11-24T08:16:07.499397+0000","last_peered":"2023-11-24T08:16:07.499397+0000","last_clean":"2023-11-24T08:16:07.499397+0000","last_became_active":"2023-11-24T08:16:06.492073+0000","last_became_peered":"2023-11-24T08:16:06.492073+0000","last_unstale":"2023-11-24T08:16:07.499397+0000","last_undegraded":"2023-11-24T08:16:07.499397+0000","last_fullsized":"2023-11-24T08:16:07.499397+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.750054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406403+0000","last_change":"2023-11-24T08:16:05.456418+0000","last_active":"2023-11-24T08:16:08.406403+0000","last_peered":"2023-11-24T08:16:08.406403+0000","last_clean":"2023-11-24T08:16:08.406403+0000","last_became_active":"2023-11-24T08:16:05.456060+0000","last_became_peered":"2023-11-24T08:16:05.456060+0000","last_unstale":"2023-11-24T08:16:08.406403+0000","last_undegraded":"2023-11-24T08:16:08.406403+0000","last_fullsized":"2023-11-24T08:16:08.406403+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:41.821611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404899+0000","last_change":"2023-11-24T08:16:06.494565+0000","last_active":"2023-11-24T08:16:08.404899+0000","last_peered":"2023-11-24T08:16:08.404899+0000","last_clean":"2023-11-24T08:16:08.404899+0000","last_became_active":"2023-11-24T08:16:06.494262+0000","last_became_peered":"2023-11-24T08:16:06.494262+0000","last_unstale":"2023-11-24T08:16:08.404899+0000","last_undegraded":"2023-11-24T08:16:08.404899+0000","last_fullsized":"2023-11-24T08:16:08.404899+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:42.131310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"146'8","reported_seq":20,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405287+0000","last_change":"2023-11-24T08:16:05.470464+0000","last_active":"2023-11-24T08:16:08.405287+0000","last_peered":"2023-11-24T08:16:08.405287+0000","last_clean":"2023-11-24T08:16:08.405287+0000","last_became_active":"2023-11-24T08:16:05.469750+0000","last_became_peered":"2023-11-24T08:16:05.469750+0000","last_unstale":"2023-11-24T08:16:08.405287+0000","last_undegraded":"2023-11-24T08:16:08.405287+0000","last_fullsized":"2023-11-24T08:16:08.405287+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:13.981312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521981+0000","last_change":"2023-11-24T08:16:06.504930+0000","last_active":"2023-11-24T08:16:07.521981+0000","last_peered":"2023-11-24T08:16:07.521981+0000","last_clean":"2023-11-24T08:16:07.521981+0000","last_became_active":"2023-11-24T08:16:06.504631+0000","last_became_peered":"2023-11-24T08:16:06.504631+0000","last_unstale":"2023-11-24T08:16:07.521981+0000","last_undegraded":"2023-11-24T08:16:07.521981+0000","last_fullsized":"2023-11-24T08:16:07.521981+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:47:54.139953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500301+0000","last_change":"2023-11-24T08:16:05.471508+0000","last_active":"2023-11-24T08:16:07.500301+0000","last_peered":"2023-11-24T08:16:07.500301+0000","last_clean":"2023-11-24T08:16:07.500301+0000","last_became_active":"2023-11-24T08:16:05.471218+0000","last_became_peered":"2023-11-24T08:16:05.471218+0000","last_unstale":"2023-11-24T08:16:07.500301+0000","last_undegraded":"2023-11-24T08:16:07.500301+0000","last_fullsized":"2023-11-24T08:16:07.500301+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:36.210681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406271+0000","last_change":"2023-11-24T08:16:06.497979+0000","last_active":"2023-11-24T08:16:08.406271+0000","last_peered":"2023-11-24T08:16:08.406271+0000","last_clean":"2023-11-24T08:16:08.406271+0000","last_became_active":"2023-11-24T08:16:06.496582+0000","last_became_peered":"2023-11-24T08:16:06.496582+0000","last_unstale":"2023-11-24T08:16:08.406271+0000","last_undegraded":"2023-11-24T08:16:08.406271+0000","last_fullsized":"2023-11-24T08:16:08.406271+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.750054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747693+0000","last_change":"2023-11-24T08:16:05.469048+0000","last_active":"2023-11-24T08:16:07.747693+0000","last_peered":"2023-11-24T08:16:07.747693+0000","last_clean":"2023-11-24T08:16:07.747693+0000","last_became_active":"2023-11-24T08:16:05.468291+0000","last_became_peered":"2023-11-24T08:16:05.468291+0000","last_unstale":"2023-11-24T08:16:07.747693+0000","last_undegraded":"2023-11-24T08:16:07.747693+0000","last_fullsized":"2023-11-24T08:16:07.747693+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:00:53.941171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405737+0000","last_change":"2023-11-24T08:16:06.499502+0000","last_active":"2023-11-24T08:16:08.405737+0000","last_peered":"2023-11-24T08:16:08.405737+0000","last_clean":"2023-11-24T08:16:08.405737+0000","last_became_active":"2023-11-24T08:16:06.499202+0000","last_became_peered":"2023-11-24T08:16:06.499202+0000","last_unstale":"2023-11-24T08:16:08.405737+0000","last_undegraded":"2023-11-24T08:16:08.405737+0000","last_fullsized":"2023-11-24T08:16:08.405737+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:33.792816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406155+0000","last_change":"2023-11-24T08:16:05.473595+0000","last_active":"2023-11-24T08:16:08.406155+0000","last_peered":"2023-11-24T08:16:08.406155+0000","last_clean":"2023-11-24T08:16:08.406155+0000","last_became_active":"2023-11-24T08:16:05.473320+0000","last_became_peered":"2023-11-24T08:16:05.473320+0000","last_unstale":"2023-11-24T08:16:08.406155+0000","last_undegraded":"2023-11-24T08:16:08.406155+0000","last_fullsized":"2023-11-24T08:16:08.406155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:38:32.672128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407219+0000","last_change":"2023-11-24T08:16:06.503795+0000","last_active":"2023-11-24T08:16:08.407219+0000","last_peered":"2023-11-24T08:16:08.407219+0000","last_clean":"2023-11-24T08:16:08.407219+0000","last_became_active":"2023-11-24T08:16:06.503534+0000","last_became_peered":"2023-11-24T08:16:06.503534+0000","last_unstale":"2023-11-24T08:16:08.407219+0000","last_undegraded":"2023-11-24T08:16:08.407219+0000","last_fullsized":"2023-11-24T08:16:08.407219+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:17.448690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747629+0000","last_change":"2023-11-24T08:16:05.464915+0000","last_active":"2023-11-24T08:16:07.747629+0000","last_peered":"2023-11-24T08:16:07.747629+0000","last_clean":"2023-11-24T08:16:07.747629+0000","last_became_active":"2023-11-24T08:16:05.464638+0000","last_became_peered":"2023-11-24T08:16:05.464638+0000","last_unstale":"2023-11-24T08:16:07.747629+0000","last_undegraded":"2023-11-24T08:16:07.747629+0000","last_fullsized":"2023-11-24T08:16:07.747629+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:32.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503193+0000","last_change":"2023-11-24T08:16:06.503282+0000","last_active":"2023-11-24T08:16:07.503193+0000","last_peered":"2023-11-24T08:16:07.503193+0000","last_clean":"2023-11-24T08:16:07.503193+0000","last_became_active":"2023-11-24T08:16:06.501862+0000","last_became_peered":"2023-11-24T08:16:06.501862+0000","last_unstale":"2023-11-24T08:16:07.503193+0000","last_undegraded":"2023-11-24T08:16:07.503193+0000","last_fullsized":"2023-11-24T08:16:07.503193+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:48.205914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343007+0000","last_change":"2023-11-24T08:16:05.464105+0000","last_active":"2023-11-24T08:16:08.343007+0000","last_peered":"2023-11-24T08:16:08.343007+0000","last_clean":"2023-11-24T08:16:08.343007+0000","last_became_active":"2023-11-24T08:16:05.463226+0000","last_became_peered":"2023-11-24T08:16:05.463226+0000","last_unstale":"2023-11-24T08:16:08.343007+0000","last_undegraded":"2023-11-24T08:16:08.343007+0000","last_fullsized":"2023-11-24T08:16:08.343007+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:49.322938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343163+0000","last_change":"2023-11-24T08:16:06.505815+0000","last_active":"2023-11-24T08:16:08.343163+0000","last_peered":"2023-11-24T08:16:08.343163+0000","last_clean":"2023-11-24T08:16:08.343163+0000","last_became_active":"2023-11-24T08:16:06.505528+0000","last_became_peered":"2023-11-24T08:16:06.505528+0000","last_unstale":"2023-11-24T08:16:08.343163+0000","last_undegraded":"2023-11-24T08:16:08.343163+0000","last_fullsized":"2023-11-24T08:16:08.343163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:25:09.051607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520988+0000","last_change":"2023-11-24T08:16:05.472454+0000","last_active":"2023-11-24T08:16:07.520988+0000","last_peered":"2023-11-24T08:16:07.520988+0000","last_clean":"2023-11-24T08:16:07.520988+0000","last_became_active":"2023-11-24T08:16:05.471551+0000","last_became_peered":"2023-11-24T08:16:05.471551+0000","last_unstale":"2023-11-24T08:16:07.520988+0000","last_undegraded":"2023-11-24T08:16:07.520988+0000","last_fullsized":"2023-11-24T08:16:07.520988+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:06.765242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502106+0000","last_change":"2023-11-24T08:16:06.493710+0000","last_active":"2023-11-24T08:16:07.502106+0000","last_peered":"2023-11-24T08:16:07.502106+0000","last_clean":"2023-11-24T08:16:07.502106+0000","last_became_active":"2023-11-24T08:16:06.493374+0000","last_became_peered":"2023-11-24T08:16:06.493374+0000","last_unstale":"2023-11-24T08:16:07.502106+0000","last_undegraded":"2023-11-24T08:16:07.502106+0000","last_fullsized":"2023-11-24T08:16:07.502106+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:33.792816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520642+0000","last_change":"2023-11-24T08:16:05.466740+0000","last_active":"2023-11-24T08:16:07.520642+0000","last_peered":"2023-11-24T08:16:07.520642+0000","last_clean":"2023-11-24T08:16:07.520642+0000","last_became_active":"2023-11-24T08:16:05.466472+0000","last_became_peered":"2023-11-24T08:16:05.466472+0000","last_unstale":"2023-11-24T08:16:07.520642+0000","last_undegraded":"2023-11-24T08:16:07.520642+0000","last_fullsized":"2023-11-24T08:16:07.520642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:01:17.812786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343002+0000","last_change":"2023-11-24T08:16:06.500970+0000","last_active":"2023-11-24T08:16:08.343002+0000","last_peered":"2023-11-24T08:16:08.343002+0000","last_clean":"2023-11-24T08:16:08.343002+0000","last_became_active":"2023-11-24T08:16:06.500558+0000","last_became_peered":"2023-11-24T08:16:06.500558+0000","last_unstale":"2023-11-24T08:16:08.343002+0000","last_undegraded":"2023-11-24T08:16:08.343002+0000","last_fullsized":"2023-11-24T08:16:08.343002+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:23.610229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404786+0000","last_change":"2023-11-24T08:16:05.467270+0000","last_active":"2023-11-24T08:16:08.404786+0000","last_peered":"2023-11-24T08:16:08.404786+0000","last_clean":"2023-11-24T08:16:08.404786+0000","last_became_active":"2023-11-24T08:16:05.466840+0000","last_became_peered":"2023-11-24T08:16:05.466840+0000","last_unstale":"2023-11-24T08:16:08.404786+0000","last_undegraded":"2023-11-24T08:16:08.404786+0000","last_fullsized":"2023-11-24T08:16:08.404786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:54.676954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500924+0000","last_change":"2023-11-24T08:16:06.492643+0000","last_active":"2023-11-24T08:16:07.500924+0000","last_peered":"2023-11-24T08:16:07.500924+0000","last_clean":"2023-11-24T08:16:07.500924+0000","last_became_active":"2023-11-24T08:16:06.492330+0000","last_became_peered":"2023-11-24T08:16:06.492330+0000","last_unstale":"2023-11-24T08:16:07.500924+0000","last_undegraded":"2023-11-24T08:16:07.500924+0000","last_fullsized":"2023-11-24T08:16:07.500924+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:42:16.473668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522082+0000","last_change":"2023-11-24T08:16:05.474751+0000","last_active":"2023-11-24T08:16:07.522082+0000","last_peered":"2023-11-24T08:16:07.522082+0000","last_clean":"2023-11-24T08:16:07.522082+0000","last_became_active":"2023-11-24T08:16:05.474468+0000","last_became_peered":"2023-11-24T08:16:05.474468+0000","last_unstale":"2023-11-24T08:16:07.522082+0000","last_undegraded":"2023-11-24T08:16:07.522082+0000","last_fullsized":"2023-11-24T08:16:07.522082+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:09.359649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406020+0000","last_change":"2023-11-24T08:16:06.498734+0000","last_active":"2023-11-24T08:16:08.406020+0000","last_peered":"2023-11-24T08:16:08.406020+0000","last_clean":"2023-11-24T08:16:08.406020+0000","last_became_active":"2023-11-24T08:16:06.498309+0000","last_became_peered":"2023-11-24T08:16:06.498309+0000","last_unstale":"2023-11-24T08:16:08.406020+0000","last_undegraded":"2023-11-24T08:16:08.406020+0000","last_fullsized":"2023-11-24T08:16:08.406020+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:14:56.087388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407016+0000","last_change":"2023-11-24T08:16:05.474544+0000","last_active":"2023-11-24T08:16:08.407016+0000","last_peered":"2023-11-24T08:16:08.407016+0000","last_clean":"2023-11-24T08:16:08.407016+0000","last_became_active":"2023-11-24T08:16:05.473862+0000","last_became_peered":"2023-11-24T08:16:05.473862+0000","last_unstale":"2023-11-24T08:16:08.407016+0000","last_undegraded":"2023-11-24T08:16:08.407016+0000","last_fullsized":"2023-11-24T08:16:08.407016+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:01.910084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.748140+0000","last_change":"2023-11-24T08:16:06.503128+0000","last_active":"2023-11-24T08:16:07.748140+0000","last_peered":"2023-11-24T08:16:07.748140+0000","last_clean":"2023-11-24T08:16:07.748140+0000","last_became_active":"2023-11-24T08:16:06.502852+0000","last_became_peered":"2023-11-24T08:16:06.502852+0000","last_unstale":"2023-11-24T08:16:07.748140+0000","last_undegraded":"2023-11-24T08:16:07.748140+0000","last_fullsized":"2023-11-24T08:16:07.748140+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:11.352447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407453+0000","last_change":"2023-11-24T08:16:05.462733+0000","last_active":"2023-11-24T08:16:08.407453+0000","last_peered":"2023-11-24T08:16:08.407453+0000","last_clean":"2023-11-24T08:16:08.407453+0000","last_became_active":"2023-11-24T08:16:05.462292+0000","last_became_peered":"2023-11-24T08:16:05.462292+0000","last_unstale":"2023-11-24T08:16:08.407453+0000","last_undegraded":"2023-11-24T08:16:08.407453+0000","last_fullsized":"2023-11-24T08:16:08.407453+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:39.617325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407122+0000","last_change":"2023-11-24T08:16:06.500802+0000","last_active":"2023-11-24T08:16:08.407122+0000","last_peered":"2023-11-24T08:16:08.407122+0000","last_clean":"2023-11-24T08:16:08.407122+0000","last_became_active":"2023-11-24T08:16:06.500499+0000","last_became_peered":"2023-11-24T08:16:06.500499+0000","last_unstale":"2023-11-24T08:16:08.407122+0000","last_undegraded":"2023-11-24T08:16:08.407122+0000","last_fullsized":"2023-11-24T08:16:08.407122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:05.251377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406542+0000","last_change":"2023-11-24T08:16:05.464913+0000","last_active":"2023-11-24T08:16:08.406542+0000","last_peered":"2023-11-24T08:16:08.406542+0000","last_clean":"2023-11-24T08:16:08.406542+0000","last_became_active":"2023-11-24T08:16:05.464604+0000","last_became_peered":"2023-11-24T08:16:05.464604+0000","last_unstale":"2023-11-24T08:16:08.406542+0000","last_undegraded":"2023-11-24T08:16:08.406542+0000","last_fullsized":"2023-11-24T08:16:08.406542+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:20:19.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405156+0000","last_change":"2023-11-24T08:16:06.498071+0000","last_active":"2023-11-24T08:16:08.405156+0000","last_peered":"2023-11-24T08:16:08.405156+0000","last_clean":"2023-11-24T08:16:08.405156+0000","last_became_active":"2023-11-24T08:16:06.497807+0000","last_became_peered":"2023-11-24T08:16:06.497807+0000","last_unstale":"2023-11-24T08:16:08.405156+0000","last_undegraded":"2023-11-24T08:16:08.405156+0000","last_fullsized":"2023-11-24T08:16:08.405156+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:00.820777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343544+0000","last_change":"2023-11-24T08:16:05.470909+0000","last_active":"2023-11-24T08:16:08.343544+0000","last_peered":"2023-11-24T08:16:08.343544+0000","last_clean":"2023-11-24T08:16:08.343544+0000","last_became_active":"2023-11-24T08:16:05.470576+0000","last_became_peered":"2023-11-24T08:16:05.470576+0000","last_unstale":"2023-11-24T08:16:08.343544+0000","last_undegraded":"2023-11-24T08:16:08.343544+0000","last_fullsized":"2023-11-24T08:16:08.343544+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:57:30.064033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500542+0000","last_change":"2023-11-24T08:16:06.491032+0000","last_active":"2023-11-24T08:16:07.500542+0000","last_peered":"2023-11-24T08:16:07.500542+0000","last_clean":"2023-11-24T08:16:07.500542+0000","last_became_active":"2023-11-24T08:16:06.490600+0000","last_became_peered":"2023-11-24T08:16:06.490600+0000","last_unstale":"2023-11-24T08:16:07.500542+0000","last_undegraded":"2023-11-24T08:16:07.500542+0000","last_fullsized":"2023-11-24T08:16:07.500542+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:02.921701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521039+0000","last_change":"2023-11-24T08:16:05.473795+0000","last_active":"2023-11-24T08:16:07.521039+0000","last_peered":"2023-11-24T08:16:07.521039+0000","last_clean":"2023-11-24T08:16:07.521039+0000","last_became_active":"2023-11-24T08:16:05.473360+0000","last_became_peered":"2023-11-24T08:16:05.473360+0000","last_unstale":"2023-11-24T08:16:07.521039+0000","last_undegraded":"2023-11-24T08:16:07.521039+0000","last_fullsized":"2023-11-24T08:16:07.521039+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:22.598612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521211+0000","last_change":"2023-11-24T08:16:06.512282+0000","last_active":"2023-11-24T08:16:07.521211+0000","last_peered":"2023-11-24T08:16:07.521211+0000","last_clean":"2023-11-24T08:16:07.521211+0000","last_became_active":"2023-11-24T08:16:06.511875+0000","last_became_peered":"2023-11-24T08:16:06.511875+0000","last_unstale":"2023-11-24T08:16:07.521211+0000","last_undegraded":"2023-11-24T08:16:07.521211+0000","last_fullsized":"2023-11-24T08:16:07.521211+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:30.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404986+0000","last_change":"2023-11-24T08:16:05.461710+0000","last_active":"2023-11-24T08:16:08.404986+0000","last_peered":"2023-11-24T08:16:08.404986+0000","last_clean":"2023-11-24T08:16:08.404986+0000","last_became_active":"2023-11-24T08:16:05.461401+0000","last_became_peered":"2023-11-24T08:16:05.461401+0000","last_unstale":"2023-11-24T08:16:08.404986+0000","last_undegraded":"2023-11-24T08:16:08.404986+0000","last_fullsized":"2023-11-24T08:16:08.404986+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:46:08.659848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499510+0000","last_change":"2023-11-24T08:16:06.505809+0000","last_active":"2023-11-24T08:16:07.499510+0000","last_peered":"2023-11-24T08:16:07.499510+0000","last_clean":"2023-11-24T08:16:07.499510+0000","last_became_active":"2023-11-24T08:16:06.505516+0000","last_became_peered":"2023-11-24T08:16:06.505516+0000","last_unstale":"2023-11-24T08:16:07.499510+0000","last_undegraded":"2023-11-24T08:16:07.499510+0000","last_fullsized":"2023-11-24T08:16:07.499510+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:32:01.370895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747813+0000","last_change":"2023-11-24T08:16:05.473443+0000","last_active":"2023-11-24T08:16:07.747813+0000","last_peered":"2023-11-24T08:16:07.747813+0000","last_clean":"2023-11-24T08:16:07.747813+0000","last_became_active":"2023-11-24T08:16:05.473132+0000","last_became_peered":"2023-11-24T08:16:05.473132+0000","last_unstale":"2023-11-24T08:16:07.747813+0000","last_undegraded":"2023-11-24T08:16:07.747813+0000","last_fullsized":"2023-11-24T08:16:07.747813+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:15.984683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343330+0000","last_change":"2023-11-24T08:16:06.504898+0000","last_active":"2023-11-24T08:16:08.343330+0000","last_peered":"2023-11-24T08:16:08.343330+0000","last_clean":"2023-11-24T08:16:08.343330+0000","last_became_active":"2023-11-24T08:16:06.504615+0000","last_became_peered":"2023-11-24T08:16:06.504615+0000","last_unstale":"2023-11-24T08:16:08.343330+0000","last_undegraded":"2023-11-24T08:16:08.343330+0000","last_fullsized":"2023-11-24T08:16:08.343330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:27.219214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"146'3","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405923+0000","last_change":"2023-11-24T08:16:05.464230+0000","last_active":"2023-11-24T08:16:08.405923+0000","last_peered":"2023-11-24T08:16:08.405923+0000","last_clean":"2023-11-24T08:16:08.405923+0000","last_became_active":"2023-11-24T08:16:05.462859+0000","last_became_peered":"2023-11-24T08:16:05.462859+0000","last_unstale":"2023-11-24T08:16:08.405923+0000","last_undegraded":"2023-11-24T08:16:08.405923+0000","last_fullsized":"2023-11-24T08:16:08.405923+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:32.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-11-24T08:16:06.482695+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:16:06.482695+0000","last_peered":"2023-11-24T08:16:06.482695+0000","last_clean":"2023-11-24T08:16:06.482695+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:16:06.482695+0000","last_undegraded":"2023-11-24T08:16:06.482695+0000","last_fullsized":"2023-11-24T08:16:06.482695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2441216,"data_stored":1397748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.036},"min":{"1min":0.75800000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.0529999999999999,"15min":1.089},"min":{"1min":0.78200000000000003,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.774,"5min":1.794,"15min":2.7109999999999999},"last":1.339}]},{"osd":1,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.151,"15min":1.1579999999999999},"min":{"1min":0.79900000000000004,"5min":0.64700000000000002,"15min":0.46300000000000002},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":1.486},{"interface":"front","average":{"1min":1.391,"5min":1.2430000000000001,"15min":1.216},"min":{"1min":0.78500000000000003,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.95699999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.117,"15min":1.169},"min":{"1min":0.70499999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.698,"5min":1.8859999999999999,"15min":4.7000000000000002},"last":1.161},{"interface":"front","average":{"1min":1.407,"5min":1.3029999999999999,"15min":1.3400000000000001},"min":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.8480000000000001,"5min":1.889,"15min":4.633},"last":1.7310000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.2110000000000001,"15min":1.1919999999999999},"min":{"1min":0.93200000000000005,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.99,"5min":1.99,"15min":3.133},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.369,"15min":1.3779999999999999},"min":{"1min":1.119,"5min":0.622,"15min":0.622},"max":{"1min":1.98,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.627}]},{"osd":4,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":1.359,"15min":1.3500000000000001},"min":{"1min":1.095,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":2.665,"5min":2.665,"15min":2.851},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.345,"15min":1.345},"min":{"1min":0.84599999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.298}]},{"osd":5,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.476,"15min":1.4730000000000001},"min":{"1min":1.1479999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.8440000000000001},{"interface":"front","average":{"1min":1.786,"5min":1.5229999999999999,"15min":1.5009999999999999},"min":{"1min":1.2529999999999999,"5min":0.622,"15min":0.622},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.6950000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.5840000000000001,"15min":1.5649999999999999},"min":{"1min":1.0740000000000001,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.992},"last":1.583},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.508,"15min":1.4870000000000001},"min":{"1min":1.1859999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":3.798},"last":1.4550000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1851392,"data_stored":806874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1279999999999999,"15min":1.0920000000000001},"min":{"1min":0.70399999999999996,"5min":0.63700000000000001,"15min":0.621},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.95},"last":1.639},{"interface":"front","average":{"1min":1.089,"5min":1.099,"15min":1.0940000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.58099999999999996},"max":{"1min":1.5349999999999999,"5min":1.804,"15min":2.2069999999999999},"last":0.998}]},{"osd":1,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.196,"15min":1.2130000000000001},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.835,"5min":1.835,"15min":2.0230000000000001},"last":1.474},{"interface":"front","average":{"1min":1.232,"5min":1.2070000000000001,"15min":1.228},"min":{"1min":0.877,"5min":0.72999999999999998,"15min":0.65000000000000002},"max":{"1min":1.7070000000000001,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.113,"15min":1.1120000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.323},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.60399999999999998},"max":{"1min":1.6970000000000001,"5min":1.8220000000000001,"15min":2.3239999999999998},"last":1.6970000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3740000000000001,"15min":1.369},"min":{"1min":0.88800000000000001,"5min":0.86399999999999999,"15min":0.71499999999999997},"max":{"1min":2.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3320000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.6850000000000001,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.512}]},{"osd":4,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.452,"15min":1.4830000000000001},"min":{"1min":1.0880000000000001,"5min":0.85399999999999998,"15min":0.76500000000000001},"max":{"1min":1.9399999999999999,"5min":2.1190000000000002,"15min":2.895},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.498,"15min":1.5229999999999999},"min":{"1min":1.0509999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9510000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4370000000000001,"15min":1.4910000000000001},"min":{"1min":0.92900000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7410000000000001,"5min":2.3300000000000001,"15min":2.629},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.571,"15min":1.6220000000000001},"min":{"1min":0.98499999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0289999999999999,"5min":2.165,"15min":2.7759999999999998},"last":1.5620000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.599,"15min":1.6499999999999999},"min":{"1min":1.266,"5min":0.748,"15min":0.748},"max":{"1min":1.97,"5min":2.4100000000000001,"15min":3.0329999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.6060000000000001,"15min":1.6359999999999999},"min":{"1min":1.143,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.927,"5min":2.512,"15min":2.6890000000000001},"last":1.5920000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574988,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1843200,"data_stored":806750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0229999999999999,"15min":1.081},"min":{"1min":0.69399999999999995,"5min":0.58599999999999997,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.694,"15min":3.1179999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0920000000000001,"15min":1.151},"min":{"1min":0.80300000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5600000000000001,"5min":1.859,"15min":2.198},"last":0.91900000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.117,"15min":1.149},"min":{"1min":0.79500000000000004,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":2.0529999999999999},"last":1.466},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1890000000000001,"15min":1.1619999999999999},"min":{"1min":0.84899999999999998,"5min":0.73399999999999999,"15min":0.48699999999999999},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.421}]},{"osd":2,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.82299999999999995,"5min":0.71099999999999997,"15min":0.51600000000000001},"max":{"1min":1.474,"5min":1.768,"15min":1.8520000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.163,"15min":1.1659999999999999},"min":{"1min":0.82199999999999995,"5min":0.72499999999999998,"15min":0.64200000000000002},"max":{"1min":1.784,"5min":1.784,"15min":1.8129999999999999},"last":1.5680000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1339999999999999,"15min":1.1699999999999999},"min":{"1min":0.89800000000000002,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.254,"15min":1.3049999999999999},"min":{"1min":0.876,"5min":0.78000000000000003,"15min":0.55200000000000005},"max":{"1min":1.5680000000000001,"5min":1.893,"15min":1.99},"last":1.2290000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.3120000000000001,"15min":1.3320000000000001},"min":{"1min":0.98399999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.268},{"interface":"front","average":{"1min":1.446,"5min":1.4370000000000001,"15min":1.508},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.46700000000000003},"max":{"1min":1.994,"5min":2.0009999999999999,"15min":3.5030000000000001},"last":1.514}]},{"osd":6,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.306,"15min":1.397},"min":{"1min":0.81899999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.085,"5min":2.085,"15min":3.2949999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.429},"min":{"1min":1.1319999999999999,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.034}]},{"osd":7,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.48,"15min":1.5469999999999999},"min":{"1min":1.0069999999999999,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":3.4420000000000002},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.591,"15min":1.603},"min":{"1min":1.087,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":2.113,"5min":2.181,"15min":2.181},"last":1.845}]}]},{"osd":4,"up_from":13,"seq":55834574990,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1839104,"data_stored":806728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0780000000000001,"15min":1.117},"min":{"1min":0.626,"5min":0.59999999999999998,"15min":0.42699999999999999},"max":{"1min":1.5780000000000001,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.129,"15min":1.0980000000000001},"min":{"1min":0.69299999999999995,"5min":0.63600000000000001,"15min":0.46100000000000002},"max":{"1min":5.452,"5min":5.452,"15min":5.452},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1359999999999999,"15min":1.149},"min":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.1679999999999999,"15min":1.181},"min":{"1min":0.73799999999999999,"5min":0.72099999999999997,"15min":0.51900000000000002},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.3120000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.0980000000000001,"15min":1.129},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.64600000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2350000000000001,"15min":1.3100000000000001},"min":{"1min":0.877,"5min":0.71199999999999997,"15min":0.505},"max":{"1min":1.6950000000000001,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.014}]},{"osd":3,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3380000000000001,"15min":1.3879999999999999},"min":{"1min":0.74399999999999999,"5min":0.73599999999999999,"15min":0.62},"max":{"1min":1.7250000000000001,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.371},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.96399999999999997,"5min":0.91200000000000003,"15min":0.55800000000000005},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.429}]},{"osd":5,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.4950000000000001,"15min":1.5009999999999999},"min":{"1min":1.258,"5min":0.71999999999999997,"15min":0.69299999999999995},"max":{"1min":1.8799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.46,"15min":1.502},"min":{"1min":1.0900000000000001,"5min":0.78200000000000003,"15min":0.60199999999999998},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6100000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.256,"15min":1.367},"min":{"1min":1.016,"5min":0.57499999999999996,"15min":0.50900000000000001},"max":{"1min":1.907,"5min":1.907,"15min":3.5899999999999999},"last":1.524},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.548,"15min":1.6080000000000001},"min":{"1min":1.397,"5min":0.79000000000000004,"15min":0.58099999999999996},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":3.391},"last":1.6539999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.51,"15min":1.5740000000000001},"min":{"1min":1.1850000000000001,"5min":0.63300000000000001,"15min":0.60899999999999999},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.419,"15min":1.5169999999999999},"min":{"1min":1.1359999999999999,"5min":0.63500000000000001,"15min":0.61799999999999999},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":3.4849999999999999},"last":1.25}]}]},{"osd":3,"up_from":13,"seq":55834574989,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37140,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334700,"statfs":{"total":96636764160,"available":96598732800,"internally_reserved":0,"allocated":1843200,"data_stored":806818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.88500000000000001,"15min":0.86299999999999999},"min":{"1min":0.433,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.4059999999999999,"5min":2.1000000000000001,"15min":10.557},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.95599999999999996,"15min":0.91800000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4490000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":0.749}]},{"osd":1,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":1.02,"5min":2.101,"15min":2.101},"last":1.274},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.996,"15min":1.0009999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.4450000000000001,"5min":2.1909999999999998,"15min":13.090999999999999},"last":1.0549999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.94199999999999995,"15min":0.93999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4750000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.0720000000000001,"15min":1.0820000000000001},"min":{"1min":0.52900000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.615,"5min":1.9690000000000001,"15min":13.460000000000001},"last":1.1739999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2809999999999999,"15min":1.2749999999999999},"min":{"1min":0.83099999999999996,"5min":0.71199999999999997,"15min":0.67800000000000005},"max":{"1min":1.639,"5min":2.1840000000000002,"15min":4.367},"last":1.786},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2210000000000001,"15min":1.2629999999999999},"min":{"1min":0.72399999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6120000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.696}]},{"osd":5,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3260000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.9410000000000001,"5min":2.2869999999999999,"15min":13.231},"last":1.335},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2649999999999999,"15min":1.2869999999999999},"min":{"1min":0.74199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.992,"5min":1.992,"15min":5.1180000000000003},"last":1.454}]},{"osd":6,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3720000000000001,"15min":1.3879999999999999},"min":{"1min":0.69499999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0449999999999999,"5min":3.1120000000000001,"15min":13.359999999999999},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2869999999999999,"15min":1.298},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6140000000000001,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.4119999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.343,"15min":1.375},"min":{"1min":0.80200000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.1030000000000002,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.643},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.351,"15min":1.3819999999999999},"min":{"1min":0.83299999999999996,"5min":0.76400000000000001,"15min":0.68799999999999994},"max":{"1min":1.7050000000000001,"5min":3.331,"15min":4.032},"last":1.855}]}]},{"osd":2,"up_from":13,"seq":55834574987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1855488,"data_stored":808410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.81799999999999995,"15min":0.86499999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.32800000000000001},"max":{"1min":1.4299999999999999,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.95299999999999996,"15min":0.96999999999999997},"min":{"1min":0.49299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.6080000000000001,"5min":1.782,"15min":2.5779999999999998},"last":1.244}]},{"osd":1,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":1.0409999999999999},"min":{"1min":0.56899999999999995,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.518,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":1.4850000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.98599999999999999,"15min":1.0580000000000001},"min":{"1min":0.54900000000000004,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.413,"5min":1.637,"15min":2.6240000000000001},"last":1.117}]},{"osd":3,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86699999999999999,"15min":0.91300000000000003},"min":{"1min":0.55200000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3939999999999999,"5min":1.552,"15min":2.2090000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0720000000000001,"15min":1.1719999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.927,"5min":1.927,"15min":2.9950000000000001},"last":1.3660000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.365},"min":{"1min":0.98799999999999999,"5min":0.72399999999999998,"15min":0.70599999999999996},"max":{"1min":1.968,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.274,"15min":1.345},"min":{"1min":0.85499999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8839999999999999,"5min":1.921,"15min":3.3279999999999998},"last":1.863}]},{"osd":5,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.292,"15min":1.3740000000000001},"min":{"1min":0.79100000000000004,"5min":0.75600000000000001,"15min":0.73399999999999999},"max":{"1min":1.6830000000000001,"5min":2.3149999999999999,"15min":2.508},"last":1.694},{"interface":"front","average":{"1min":1.276,"5min":1.264,"15min":1.345},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.74,"5min":2.0070000000000001,"15min":2.7839999999999998},"last":2.0910000000000002}]},{"osd":6,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2170000000000001,"15min":1.284},"min":{"1min":0.81999999999999995,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":1.7490000000000001,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":2.2879999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.3660000000000001,"15min":1.4450000000000001},"min":{"1min":0.79300000000000004,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":1.8140000000000001,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.75}]},{"osd":7,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.359,"15min":1.4390000000000001},"min":{"1min":0.83899999999999997,"5min":0.77500000000000002,"15min":0.71299999999999997},"max":{"1min":1.8420000000000001,"5min":2.1070000000000002,"15min":2.859},"last":2.3999999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.2989999999999999,"15min":1.4019999999999999},"min":{"1min":0.93000000000000005,"5min":0.81399999999999995,"15min":0.746},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":2.0179999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37128,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334712,"statfs":{"total":96636764160,"available":96598745088,"internally_reserved":0,"allocated":1830912,"data_stored":801010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.79300000000000004,"15min":0.78500000000000003},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.006,"5min":1.5900000000000001,"15min":1.7749999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.877,"15min":0.86199999999999999},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.375,"5min":1.6970000000000001,"15min":2.1619999999999999},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80100000000000005,"15min":0.76000000000000001},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":1.4390000000000001,"5min":1.679,"15min":1.861},"last":0.374},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92900000000000005,"15min":0.90400000000000003},"min":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.403,"5min":1.925,"15min":1.925},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":0.97499999999999998},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.5600000000000001,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.92800000000000005,"15min":0.90400000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5409999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.276,"15min":1.2569999999999999},"min":{"1min":0.85199999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8380000000000001,"5min":2.028,"15min":2.7959999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.252},"min":{"1min":0.90300000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7110000000000001,"5min":2.141,"15min":3.0409999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2310000000000001,"15min":1.23},"min":{"1min":0.77600000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.9610000000000001,"15min":3.7639999999999998},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1830000000000001,"15min":1.1910000000000001},"min":{"1min":0.66900000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.821,"5min":2.246,"15min":3.286},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2470000000000001,"15min":1.256},"min":{"1min":0.69299999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.788,"5min":2.5270000000000001,"15min":4.2750000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.198,"5min":1.2330000000000001,"15min":1.2350000000000001},"min":{"1min":0.72099999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.627,"5min":1.9179999999999999,"15min":3.7040000000000002},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3009999999999999,"15min":1.3049999999999999},"min":{"1min":0.80200000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7150000000000001,"5min":2.3959999999999999,"15min":3.1760000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.3149999999999999,"15min":1.3029999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.7270000000000001,"5min":2.3620000000000001,"15min":3.653},"last":0.624}]}]},{"osd":0,"up_from":13,"seq":55834574989,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2449408,"data_stored":1398742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72599999999999998,"15min":0.80100000000000005},"min":{"1min":0.46300000000000002,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.81100000000000005,"15min":0.91600000000000004},"min":{"1min":0.42099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.425,"5min":1.4610000000000001,"15min":2.3620000000000001},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.008,"15min":1.036},"min":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.47699999999999998},"max":{"1min":1.405,"5min":1.833,"15min":1.891},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":0.91400000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.99099999999999999,"15min":1.0509999999999999},"min":{"1min":0.50900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.679,"5min":1.724,"15min":2.0670000000000002},"last":1.2},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91300000000000003,"15min":0.97599999999999998},"min":{"1min":0.48899999999999999,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":1.651}]},{"osd":4,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1639999999999999,"15min":1.306},"min":{"1min":0.73799999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4950000000000001,"5min":1.7729999999999999,"15min":3.7610000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.157,"15min":1.1870000000000001},"min":{"1min":0.83499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.3420000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.169,"15min":1.238},"min":{"1min":0.752,"5min":0.72799999999999998,"15min":0.69699999999999995},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1699999999999999,"15min":1.302},"min":{"1min":0.83299999999999996,"5min":0.72299999999999998,"15min":0.71099999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":2.7240000000000002},"last":1.268}]},{"osd":6,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2689999999999999,"15min":1.349},"min":{"1min":0.93600000000000005,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":2.052,"5min":2.052,"15min":2.4369999999999998},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.262,"5min":1.2669999999999999,"15min":1.3560000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.6899999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.339,"15min":1.4330000000000001},"min":{"1min":0.997,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":1.6120000000000001,"5min":2.3319999999999999,"15min":2.6989999999999998},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.3140000000000001,"15min":1.411},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.875,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.885}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:16:10.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:16:10.990 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:16:10.988+0000 7f5e43522640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:16:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.186+0000 7fabab363640 1 Processor -- start 2023-11-24T08:16:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.186+0000 7fabab363640 1 -- start start 2023-11-24T08:16:11.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.187+0000 7fabab363640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba413d120 0x7faba415af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.187+0000 7fabab363640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 0x7faba406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.187+0000 7fabab363640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba406e060 con 0x7faba40b0a70 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.187+0000 7fabab363640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba413f9e0 con 0x7faba413d120 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.187+0000 7fabab363640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba413fcc0 con 0x7faba406d6c0 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba413d120 0x7faba415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba413d120 0x7faba415af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35458/0 (socket says 172.21.15.107:35458) 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 -- 172.21.15.107:0/3170684867 learned_addr learned my addr 172.21.15.107:0/3170684867 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 -- 172.21.15.107:0/3170684867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 msgr2=0x7faba406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7faba9b60640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 0x7faba406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 0x7faba406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 -- 172.21.15.107:0/3170684867 >> v1:172.21.15.107:6789/0 conn(0x7faba40b0a70 legacy=0x7faba40b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7fabaa361640 1 -- 172.21.15.107:0/3170684867 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40cec00 con 0x7faba413d120 2023-11-24T08:16:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.188+0000 7faba9b60640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 0x7faba406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.189+0000 7fabaa361640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba415af40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7faba0009870 tx=0x7faba002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7fd475fcae8937 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7faba935f640 1 -- 172.21.15.107:0/3170684867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0031640 con 0x7faba413d120 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7faba935f640 1 -- 172.21.15.107:0/3170684867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00317a0 con 0x7faba413d120 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7faba935f640 1 -- 172.21.15.107:0/3170684867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0031b30 con 0x7faba413d120 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 -- 172.21.15.107:0/3170684867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 msgr2=0x7faba415af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba415af40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7faba0009870 tx=0x7faba002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 -- 172.21.15.107:0/3170684867 shutdown_connections 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406d6c0 0x7faba406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 --2- 172.21.15.107:0/3170684867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba415af40 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 -- 172.21.15.107:0/3170684867 >> 172.21.15.107:0/3170684867 conn(0x7faba40cb830 msgr2=0x7faba40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 -- 172.21.15.107:0/3170684867 shutdown_connections 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.190+0000 7fabab363640 1 -- 172.21.15.107:0/3170684867 wait complete. 2023-11-24T08:16:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.192+0000 7fabab363640 1 Processor -- start 2023-11-24T08:16:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.192+0000 7fabab363640 1 -- start start 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406d6c0 0x7faba40b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 0x7faba413c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40b01a0 con 0x7faba413d120 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4071b80 con 0x7faba406d6c0 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabab363640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba4071000 con 0x7faba40b0a70 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.193+0000 7fabaab62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35470/0 (socket says 172.21.15.107:35470) 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 -- 172.21.15.107:0/1997109472 learned_addr learned my addr 172.21.15.107:0/1997109472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7faba9b60640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 0x7faba413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaa361640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406d6c0 0x7faba40b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 -- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 msgr2=0x7faba413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 0x7faba413c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 -- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406d6c0 msgr2=0x7faba40b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406d6c0 0x7faba40b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 -- 172.21.15.107:0/1997109472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba00094d0 con 0x7faba413d120 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7faba9b60640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 0x7faba413c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fabaab62640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c7d0 tx=0x7fab9c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.194+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c003b70 con 0x7faba413d120 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.195+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c003cf0 con 0x7faba413d120 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.195+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba40712e0 con 0x7faba413d120 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.195+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00aa90 con 0x7faba413d120 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.195+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba40718a0 con 0x7faba413d120 2023-11-24T08:16:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.196+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4140240 con 0x7faba413d120 2023-11-24T08:16:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.197+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fab9c00abf0 con 0x7faba413d120 2023-11-24T08:16:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.198+0000 7fab9affd640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 0x7fab8806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.198+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab9c0884e0 con 0x7faba413d120 2023-11-24T08:16:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.198+0000 7fabaa361640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 0x7fab8806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.199+0000 7fabaa361640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 0x7fab8806e7d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faba0036000 tx=0x7faba003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.202+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab9c054c60 con 0x7faba413d120 2023-11-24T08:16:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.389+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faba4140560 con 0x7fab8806c310 2023-11-24T08:16:11.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.411+0000 7fab9affd640 1 -- 172.21.15.107:0/1997109472 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287633 (secure 0 0 0) 0x7faba003a420 con 0x7fab8806c310 2023-11-24T08:16:11.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:16:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 msgr2=0x7fab8806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 0x7fab8806e7d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faba0036000 tx=0x7faba003c040 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 msgr2=0x7faba413c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c7d0 tx=0x7fab9c00cc90 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 shutdown_connections 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8806c310 0x7fab8806e7d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba413d120 0x7faba413c8c0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40b0a70 0x7faba413c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 --2- 172.21.15.107:0/1997109472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406d6c0 0x7faba40b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 >> 172.21.15.107:0/1997109472 conn(0x7faba40cb830 msgr2=0x7faba40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.418+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 shutdown_connections 2023-11-24T08:16:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.419+0000 7fabab363640 1 -- 172.21.15.107:0/1997109472 wait complete. 2023-11-24T08:16:11.441 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2023-11-24T08:16:09.645042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294148,"kb_used_data":15580,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754680572,"statfs":{"total":773094113280,"available":772792905728,"internally_reserved":0,"allocated":15953920,"data_stored":7633080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":285134398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.213155"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407609+0000","last_change":"2023-11-24T08:16:06.503554+0000","last_active":"2023-11-24T08:16:08.407609+0000","last_peered":"2023-11-24T08:16:08.407609+0000","last_clean":"2023-11-24T08:16:08.407609+0000","last_became_active":"2023-11-24T08:16:06.502985+0000","last_became_peered":"2023-11-24T08:16:06.502985+0000","last_unstale":"2023-11-24T08:16:08.407609+0000","last_undegraded":"2023-11-24T08:16:08.407609+0000","last_fullsized":"2023-11-24T08:16:08.407609+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:42.131310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747505+0000","last_change":"2023-11-24T08:16:06.497798+0000","last_active":"2023-11-24T08:16:07.747505+0000","last_peered":"2023-11-24T08:16:07.747505+0000","last_clean":"2023-11-24T08:16:07.747505+0000","last_became_active":"2023-11-24T08:16:06.497537+0000","last_became_peered":"2023-11-24T08:16:06.497537+0000","last_unstale":"2023-11-24T08:16:07.747505+0000","last_undegraded":"2023-11-24T08:16:07.747505+0000","last_fullsized":"2023-11-24T08:16:07.747505+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:02.014109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405676+0000","last_change":"2023-11-24T08:16:06.492396+0000","last_active":"2023-11-24T08:16:08.405676+0000","last_peered":"2023-11-24T08:16:08.405676+0000","last_clean":"2023-11-24T08:16:08.405676+0000","last_became_active":"2023-11-24T08:16:06.492123+0000","last_became_peered":"2023-11-24T08:16:06.492123+0000","last_unstale":"2023-11-24T08:16:08.405676+0000","last_undegraded":"2023-11-24T08:16:08.405676+0000","last_fullsized":"2023-11-24T08:16:08.405676+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:39:55.657306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503234+0000","last_change":"2023-11-24T08:16:06.495470+0000","last_active":"2023-11-24T08:16:07.503234+0000","last_peered":"2023-11-24T08:16:07.503234+0000","last_clean":"2023-11-24T08:16:07.503234+0000","last_became_active":"2023-11-24T08:16:06.495172+0000","last_became_peered":"2023-11-24T08:16:06.495172+0000","last_unstale":"2023-11-24T08:16:07.503234+0000","last_undegraded":"2023-11-24T08:16:07.503234+0000","last_fullsized":"2023-11-24T08:16:07.503234+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:36.772056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500181+0000","last_change":"2023-11-24T08:16:06.494450+0000","last_active":"2023-11-24T08:16:07.500181+0000","last_peered":"2023-11-24T08:16:07.500181+0000","last_clean":"2023-11-24T08:16:07.500181+0000","last_became_active":"2023-11-24T08:16:06.494150+0000","last_became_peered":"2023-11-24T08:16:06.494150+0000","last_unstale":"2023-11-24T08:16:07.500181+0000","last_undegraded":"2023-11-24T08:16:07.500181+0000","last_fullsized":"2023-11-24T08:16:07.500181+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:29.279322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406781+0000","last_change":"2023-11-24T08:16:06.503101+0000","last_active":"2023-11-24T08:16:08.406781+0000","last_peered":"2023-11-24T08:16:08.406781+0000","last_clean":"2023-11-24T08:16:08.406781+0000","last_became_active":"2023-11-24T08:16:06.502803+0000","last_became_peered":"2023-11-24T08:16:06.502803+0000","last_unstale":"2023-11-24T08:16:08.406781+0000","last_undegraded":"2023-11-24T08:16:08.406781+0000","last_fullsized":"2023-11-24T08:16:08.406781+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:49.195067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343008+0000","last_change":"2023-11-24T08:16:06.500807+0000","last_active":"2023-11-24T08:16:08.343008+0000","last_peered":"2023-11-24T08:16:08.343008+0000","last_clean":"2023-11-24T08:16:08.343008+0000","last_became_active":"2023-11-24T08:16:06.494871+0000","last_became_peered":"2023-11-24T08:16:06.494871+0000","last_unstale":"2023-11-24T08:16:08.343008+0000","last_undegraded":"2023-11-24T08:16:08.343008+0000","last_fullsized":"2023-11-24T08:16:08.343008+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:07.776858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406066+0000","last_change":"2023-11-24T08:16:06.497881+0000","last_active":"2023-11-24T08:16:08.406066+0000","last_peered":"2023-11-24T08:16:08.406066+0000","last_clean":"2023-11-24T08:16:08.406066+0000","last_became_active":"2023-11-24T08:16:06.494609+0000","last_became_peered":"2023-11-24T08:16:06.494609+0000","last_unstale":"2023-11-24T08:16:08.406066+0000","last_undegraded":"2023-11-24T08:16:08.406066+0000","last_fullsized":"2023-11-24T08:16:08.406066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:47.882204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747635+0000","last_change":"2023-11-24T08:16:06.502162+0000","last_active":"2023-11-24T08:16:07.747635+0000","last_peered":"2023-11-24T08:16:07.747635+0000","last_clean":"2023-11-24T08:16:07.747635+0000","last_became_active":"2023-11-24T08:16:06.501616+0000","last_became_peered":"2023-11-24T08:16:06.501616+0000","last_unstale":"2023-11-24T08:16:07.747635+0000","last_undegraded":"2023-11-24T08:16:07.747635+0000","last_fullsized":"2023-11-24T08:16:07.747635+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:35.786642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521768+0000","last_change":"2023-11-24T08:16:06.499678+0000","last_active":"2023-11-24T08:16:07.521768+0000","last_peered":"2023-11-24T08:16:07.521768+0000","last_clean":"2023-11-24T08:16:07.521768+0000","last_became_active":"2023-11-24T08:16:06.499374+0000","last_became_peered":"2023-11-24T08:16:06.499374+0000","last_unstale":"2023-11-24T08:16:07.521768+0000","last_undegraded":"2023-11-24T08:16:07.521768+0000","last_fullsized":"2023-11-24T08:16:07.521768+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:11:39.961690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406747+0000","last_change":"2023-11-24T08:16:06.500807+0000","last_active":"2023-11-24T08:16:08.406747+0000","last_peered":"2023-11-24T08:16:08.406747+0000","last_clean":"2023-11-24T08:16:08.406747+0000","last_became_active":"2023-11-24T08:16:06.500526+0000","last_became_peered":"2023-11-24T08:16:06.500526+0000","last_unstale":"2023-11-24T08:16:08.406747+0000","last_undegraded":"2023-11-24T08:16:08.406747+0000","last_fullsized":"2023-11-24T08:16:08.406747+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:55.688570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521093+0000","last_change":"2023-11-24T08:16:06.498544+0000","last_active":"2023-11-24T08:16:07.521093+0000","last_peered":"2023-11-24T08:16:07.521093+0000","last_clean":"2023-11-24T08:16:07.521093+0000","last_became_active":"2023-11-24T08:16:06.498232+0000","last_became_peered":"2023-11-24T08:16:06.498232+0000","last_unstale":"2023-11-24T08:16:07.521093+0000","last_undegraded":"2023-11-24T08:16:07.521093+0000","last_fullsized":"2023-11-24T08:16:07.521093+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:42.672960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500687+0000","last_change":"2023-11-24T08:16:06.495536+0000","last_active":"2023-11-24T08:16:07.500687+0000","last_peered":"2023-11-24T08:16:07.500687+0000","last_clean":"2023-11-24T08:16:07.500687+0000","last_became_active":"2023-11-24T08:16:06.494260+0000","last_became_peered":"2023-11-24T08:16:06.494260+0000","last_unstale":"2023-11-24T08:16:07.500687+0000","last_undegraded":"2023-11-24T08:16:07.500687+0000","last_fullsized":"2023-11-24T08:16:07.500687+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:32.612410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502642+0000","last_change":"2023-11-24T08:16:06.492403+0000","last_active":"2023-11-24T08:16:07.502642+0000","last_peered":"2023-11-24T08:16:07.502642+0000","last_clean":"2023-11-24T08:16:07.502642+0000","last_became_active":"2023-11-24T08:16:06.492158+0000","last_became_peered":"2023-11-24T08:16:06.492158+0000","last_unstale":"2023-11-24T08:16:07.502642+0000","last_undegraded":"2023-11-24T08:16:07.502642+0000","last_fullsized":"2023-11-24T08:16:07.502642+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.885982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520758+0000","last_change":"2023-11-24T08:16:06.506683+0000","last_active":"2023-11-24T08:16:07.520758+0000","last_peered":"2023-11-24T08:16:07.520758+0000","last_clean":"2023-11-24T08:16:07.520758+0000","last_became_active":"2023-11-24T08:16:06.506236+0000","last_became_peered":"2023-11-24T08:16:06.506236+0000","last_unstale":"2023-11-24T08:16:07.520758+0000","last_undegraded":"2023-11-24T08:16:07.520758+0000","last_fullsized":"2023-11-24T08:16:07.520758+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:50.803641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404973+0000","last_change":"2023-11-24T08:16:06.503847+0000","last_active":"2023-11-24T08:16:08.404973+0000","last_peered":"2023-11-24T08:16:08.404973+0000","last_clean":"2023-11-24T08:16:08.404973+0000","last_became_active":"2023-11-24T08:16:06.503554+0000","last_became_peered":"2023-11-24T08:16:06.503554+0000","last_unstale":"2023-11-24T08:16:08.404973+0000","last_undegraded":"2023-11-24T08:16:08.404973+0000","last_fullsized":"2023-11-24T08:16:08.404973+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:27.167457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522109+0000","last_change":"2023-11-24T08:16:06.499283+0000","last_active":"2023-11-24T08:16:07.522109+0000","last_peered":"2023-11-24T08:16:07.522109+0000","last_clean":"2023-11-24T08:16:07.522109+0000","last_became_active":"2023-11-24T08:16:06.498769+0000","last_became_peered":"2023-11-24T08:16:06.498769+0000","last_unstale":"2023-11-24T08:16:07.522109+0000","last_undegraded":"2023-11-24T08:16:07.522109+0000","last_fullsized":"2023-11-24T08:16:07.522109+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:05:29.257509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405896+0000","last_change":"2023-11-24T08:16:06.497994+0000","last_active":"2023-11-24T08:16:08.405896+0000","last_peered":"2023-11-24T08:16:08.405896+0000","last_clean":"2023-11-24T08:16:08.405896+0000","last_became_active":"2023-11-24T08:16:06.497705+0000","last_became_peered":"2023-11-24T08:16:06.497705+0000","last_unstale":"2023-11-24T08:16:08.405896+0000","last_undegraded":"2023-11-24T08:16:08.405896+0000","last_fullsized":"2023-11-24T08:16:08.405896+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:09.039086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407488+0000","last_change":"2023-11-24T08:16:06.496118+0000","last_active":"2023-11-24T08:16:08.407488+0000","last_peered":"2023-11-24T08:16:08.407488+0000","last_clean":"2023-11-24T08:16:08.407488+0000","last_became_active":"2023-11-24T08:16:06.495803+0000","last_became_peered":"2023-11-24T08:16:06.495803+0000","last_unstale":"2023-11-24T08:16:08.407488+0000","last_undegraded":"2023-11-24T08:16:08.407488+0000","last_fullsized":"2023-11-24T08:16:08.407488+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:33.919970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406128+0000","last_change":"2023-11-24T08:16:06.492617+0000","last_active":"2023-11-24T08:16:08.406128+0000","last_peered":"2023-11-24T08:16:08.406128+0000","last_clean":"2023-11-24T08:16:08.406128+0000","last_became_active":"2023-11-24T08:16:06.492320+0000","last_became_peered":"2023-11-24T08:16:06.492320+0000","last_unstale":"2023-11-24T08:16:08.406128+0000","last_undegraded":"2023-11-24T08:16:08.406128+0000","last_fullsized":"2023-11-24T08:16:08.406128+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:02.175235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521408+0000","last_change":"2023-11-24T08:16:06.501869+0000","last_active":"2023-11-24T08:16:07.521408+0000","last_peered":"2023-11-24T08:16:07.521408+0000","last_clean":"2023-11-24T08:16:07.521408+0000","last_became_active":"2023-11-24T08:16:06.501607+0000","last_became_peered":"2023-11-24T08:16:06.501607+0000","last_unstale":"2023-11-24T08:16:07.521408+0000","last_undegraded":"2023-11-24T08:16:07.521408+0000","last_fullsized":"2023-11-24T08:16:07.521408+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:02.014109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502645+0000","last_change":"2023-11-24T08:16:06.491776+0000","last_active":"2023-11-24T08:16:07.502645+0000","last_peered":"2023-11-24T08:16:07.502645+0000","last_clean":"2023-11-24T08:16:07.502645+0000","last_became_active":"2023-11-24T08:16:06.491502+0000","last_became_peered":"2023-11-24T08:16:06.491502+0000","last_unstale":"2023-11-24T08:16:07.502645+0000","last_undegraded":"2023-11-24T08:16:07.502645+0000","last_fullsized":"2023-11-24T08:16:07.502645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:14:56.087388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502334+0000","last_change":"2023-11-24T08:16:06.493445+0000","last_active":"2023-11-24T08:16:07.502334+0000","last_peered":"2023-11-24T08:16:07.502334+0000","last_clean":"2023-11-24T08:16:07.502334+0000","last_became_active":"2023-11-24T08:16:06.493156+0000","last_became_peered":"2023-11-24T08:16:06.493156+0000","last_unstale":"2023-11-24T08:16:07.502334+0000","last_undegraded":"2023-11-24T08:16:07.502334+0000","last_fullsized":"2023-11-24T08:16:07.502334+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:09.039086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520740+0000","last_change":"2023-11-24T08:16:06.502468+0000","last_active":"2023-11-24T08:16:07.520740+0000","last_peered":"2023-11-24T08:16:07.520740+0000","last_clean":"2023-11-24T08:16:07.520740+0000","last_became_active":"2023-11-24T08:16:06.502201+0000","last_became_peered":"2023-11-24T08:16:06.502201+0000","last_unstale":"2023-11-24T08:16:07.520740+0000","last_undegraded":"2023-11-24T08:16:07.520740+0000","last_fullsized":"2023-11-24T08:16:07.520740+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:04.727045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405838+0000","last_change":"2023-11-24T08:16:06.501813+0000","last_active":"2023-11-24T08:16:08.405838+0000","last_peered":"2023-11-24T08:16:08.405838+0000","last_clean":"2023-11-24T08:16:08.405838+0000","last_became_active":"2023-11-24T08:16:06.501545+0000","last_became_peered":"2023-11-24T08:16:06.501545+0000","last_unstale":"2023-11-24T08:16:08.405838+0000","last_undegraded":"2023-11-24T08:16:08.405838+0000","last_fullsized":"2023-11-24T08:16:08.405838+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:37.222297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407055+0000","last_change":"2023-11-24T08:16:06.494379+0000","last_active":"2023-11-24T08:16:08.407055+0000","last_peered":"2023-11-24T08:16:08.407055+0000","last_clean":"2023-11-24T08:16:08.407055+0000","last_became_active":"2023-11-24T08:16:06.494103+0000","last_became_peered":"2023-11-24T08:16:06.494103+0000","last_unstale":"2023-11-24T08:16:08.407055+0000","last_undegraded":"2023-11-24T08:16:08.407055+0000","last_fullsized":"2023-11-24T08:16:08.407055+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:09.500462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405565+0000","last_change":"2023-11-24T08:16:06.494358+0000","last_active":"2023-11-24T08:16:08.405565+0000","last_peered":"2023-11-24T08:16:08.405565+0000","last_clean":"2023-11-24T08:16:08.405565+0000","last_became_active":"2023-11-24T08:16:06.493935+0000","last_became_peered":"2023-11-24T08:16:06.493935+0000","last_unstale":"2023-11-24T08:16:08.405565+0000","last_undegraded":"2023-11-24T08:16:08.405565+0000","last_fullsized":"2023-11-24T08:16:08.405565+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:51:05.860184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343300+0000","last_change":"2023-11-24T08:16:06.491448+0000","last_active":"2023-11-24T08:16:08.343300+0000","last_peered":"2023-11-24T08:16:08.343300+0000","last_clean":"2023-11-24T08:16:08.343300+0000","last_became_active":"2023-11-24T08:16:06.491158+0000","last_became_peered":"2023-11-24T08:16:06.491158+0000","last_unstale":"2023-11-24T08:16:08.343300+0000","last_undegraded":"2023-11-24T08:16:08.343300+0000","last_fullsized":"2023-11-24T08:16:08.343300+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:36.139572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406121+0000","last_change":"2023-11-24T08:16:06.499555+0000","last_active":"2023-11-24T08:16:08.406121+0000","last_peered":"2023-11-24T08:16:08.406121+0000","last_clean":"2023-11-24T08:16:08.406121+0000","last_became_active":"2023-11-24T08:16:06.499283+0000","last_became_peered":"2023-11-24T08:16:06.499283+0000","last_unstale":"2023-11-24T08:16:08.406121+0000","last_undegraded":"2023-11-24T08:16:08.406121+0000","last_fullsized":"2023-11-24T08:16:08.406121+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:46:09.671464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503008+0000","last_change":"2023-11-24T08:16:06.503014+0000","last_active":"2023-11-24T08:16:07.503008+0000","last_peered":"2023-11-24T08:16:07.503008+0000","last_clean":"2023-11-24T08:16:07.503008+0000","last_became_active":"2023-11-24T08:16:06.502201+0000","last_became_peered":"2023-11-24T08:16:06.502201+0000","last_unstale":"2023-11-24T08:16:07.503008+0000","last_undegraded":"2023-11-24T08:16:07.503008+0000","last_fullsized":"2023-11-24T08:16:07.503008+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:52.832521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747346+0000","last_change":"2023-11-24T08:16:06.494306+0000","last_active":"2023-11-24T08:16:07.747346+0000","last_peered":"2023-11-24T08:16:07.747346+0000","last_clean":"2023-11-24T08:16:07.747346+0000","last_became_active":"2023-11-24T08:16:06.494021+0000","last_became_peered":"2023-11-24T08:16:06.494021+0000","last_unstale":"2023-11-24T08:16:07.747346+0000","last_undegraded":"2023-11-24T08:16:07.747346+0000","last_fullsized":"2023-11-24T08:16:07.747346+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:42.672960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405074+0000","last_change":"2023-11-24T08:16:06.489660+0000","last_active":"2023-11-24T08:16:08.405074+0000","last_peered":"2023-11-24T08:16:08.405074+0000","last_clean":"2023-11-24T08:16:08.405074+0000","last_became_active":"2023-11-24T08:16:06.489257+0000","last_became_peered":"2023-11-24T08:16:06.489257+0000","last_unstale":"2023-11-24T08:16:08.405074+0000","last_undegraded":"2023-11-24T08:16:08.405074+0000","last_fullsized":"2023-11-24T08:16:08.405074+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:26:57.091628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521894+0000","last_change":"2023-11-24T08:16:06.499474+0000","last_active":"2023-11-24T08:16:07.521894+0000","last_peered":"2023-11-24T08:16:07.521894+0000","last_clean":"2023-11-24T08:16:07.521894+0000","last_became_active":"2023-11-24T08:16:06.498962+0000","last_became_peered":"2023-11-24T08:16:06.498962+0000","last_unstale":"2023-11-24T08:16:07.521894+0000","last_undegraded":"2023-11-24T08:16:07.521894+0000","last_fullsized":"2023-11-24T08:16:07.521894+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:09:48.423417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"146'4","reported_seq":16,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521880+0000","last_change":"2023-11-24T08:16:05.473035+0000","last_active":"2023-11-24T08:16:07.521880+0000","last_peered":"2023-11-24T08:16:07.521880+0000","last_clean":"2023-11-24T08:16:07.521880+0000","last_became_active":"2023-11-24T08:16:05.472688+0000","last_became_peered":"2023-11-24T08:16:05.472688+0000","last_unstale":"2023-11-24T08:16:07.521880+0000","last_undegraded":"2023-11-24T08:16:07.521880+0000","last_fullsized":"2023-11-24T08:16:07.521880+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:26.583836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.748006+0000","last_change":"2023-11-24T08:16:06.499172+0000","last_active":"2023-11-24T08:16:07.748006+0000","last_peered":"2023-11-24T08:16:07.748006+0000","last_clean":"2023-11-24T08:16:07.748006+0000","last_became_active":"2023-11-24T08:16:06.498908+0000","last_became_peered":"2023-11-24T08:16:06.498908+0000","last_unstale":"2023-11-24T08:16:07.748006+0000","last_undegraded":"2023-11-24T08:16:07.748006+0000","last_fullsized":"2023-11-24T08:16:07.748006+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:09:48.423417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747602+0000","last_change":"2023-11-24T08:16:05.473712+0000","last_active":"2023-11-24T08:16:07.747602+0000","last_peered":"2023-11-24T08:16:07.747602+0000","last_clean":"2023-11-24T08:16:07.747602+0000","last_became_active":"2023-11-24T08:16:05.473227+0000","last_became_peered":"2023-11-24T08:16:05.473227+0000","last_unstale":"2023-11-24T08:16:07.747602+0000","last_undegraded":"2023-11-24T08:16:07.747602+0000","last_fullsized":"2023-11-24T08:16:07.747602+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:01.784224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406297+0000","last_change":"2023-11-24T08:16:06.508929+0000","last_active":"2023-11-24T08:16:08.406297+0000","last_peered":"2023-11-24T08:16:08.406297+0000","last_clean":"2023-11-24T08:16:08.406297+0000","last_became_active":"2023-11-24T08:16:06.506946+0000","last_became_peered":"2023-11-24T08:16:06.506946+0000","last_unstale":"2023-11-24T08:16:08.406297+0000","last_undegraded":"2023-11-24T08:16:08.406297+0000","last_fullsized":"2023-11-24T08:16:08.406297+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:59.854885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502979+0000","last_change":"2023-11-24T08:16:05.469028+0000","last_active":"2023-11-24T08:16:07.502979+0000","last_peered":"2023-11-24T08:16:07.502979+0000","last_clean":"2023-11-24T08:16:07.502979+0000","last_became_active":"2023-11-24T08:16:05.468740+0000","last_became_peered":"2023-11-24T08:16:05.468740+0000","last_unstale":"2023-11-24T08:16:07.502979+0000","last_undegraded":"2023-11-24T08:16:07.502979+0000","last_fullsized":"2023-11-24T08:16:07.502979+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:08.488846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521596+0000","last_change":"2023-11-24T08:16:06.494087+0000","last_active":"2023-11-24T08:16:07.521596+0000","last_peered":"2023-11-24T08:16:07.521596+0000","last_clean":"2023-11-24T08:16:07.521596+0000","last_became_active":"2023-11-24T08:16:06.493751+0000","last_became_peered":"2023-11-24T08:16:06.493751+0000","last_unstale":"2023-11-24T08:16:07.521596+0000","last_undegraded":"2023-11-24T08:16:07.521596+0000","last_fullsized":"2023-11-24T08:16:07.521596+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:35.786642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503331+0000","last_change":"2023-11-24T08:16:05.464029+0000","last_active":"2023-11-24T08:16:07.503331+0000","last_peered":"2023-11-24T08:16:07.503331+0000","last_clean":"2023-11-24T08:16:07.503331+0000","last_became_active":"2023-11-24T08:16:05.463581+0000","last_became_peered":"2023-11-24T08:16:05.463581+0000","last_unstale":"2023-11-24T08:16:07.503331+0000","last_undegraded":"2023-11-24T08:16:07.503331+0000","last_fullsized":"2023-11-24T08:16:07.503331+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:39:54.645689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747425+0000","last_change":"2023-11-24T08:16:06.493299+0000","last_active":"2023-11-24T08:16:07.747425+0000","last_peered":"2023-11-24T08:16:07.747425+0000","last_clean":"2023-11-24T08:16:07.747425+0000","last_became_active":"2023-11-24T08:16:06.492844+0000","last_became_peered":"2023-11-24T08:16:06.492844+0000","last_unstale":"2023-11-24T08:16:07.747425+0000","last_undegraded":"2023-11-24T08:16:07.747425+0000","last_fullsized":"2023-11-24T08:16:07.747425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:30.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406022+0000","last_change":"2023-11-24T08:16:05.475958+0000","last_active":"2023-11-24T08:16:08.406022+0000","last_peered":"2023-11-24T08:16:08.406022+0000","last_clean":"2023-11-24T08:16:08.406022+0000","last_became_active":"2023-11-24T08:16:05.475670+0000","last_became_peered":"2023-11-24T08:16:05.475670+0000","last_unstale":"2023-11-24T08:16:08.406022+0000","last_undegraded":"2023-11-24T08:16:08.406022+0000","last_fullsized":"2023-11-24T08:16:08.406022+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:15:33.634025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502846+0000","last_change":"2023-11-24T08:16:06.495288+0000","last_active":"2023-11-24T08:16:07.502846+0000","last_peered":"2023-11-24T08:16:07.502846+0000","last_clean":"2023-11-24T08:16:07.502846+0000","last_became_active":"2023-11-24T08:16:06.494965+0000","last_became_peered":"2023-11-24T08:16:06.494965+0000","last_unstale":"2023-11-24T08:16:07.502846+0000","last_undegraded":"2023-11-24T08:16:07.502846+0000","last_fullsized":"2023-11-24T08:16:07.502846+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:40.831282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499382+0000","last_change":"2023-11-24T08:16:05.475703+0000","last_active":"2023-11-24T08:16:07.499382+0000","last_peered":"2023-11-24T08:16:07.499382+0000","last_clean":"2023-11-24T08:16:07.499382+0000","last_became_active":"2023-11-24T08:16:05.475382+0000","last_became_peered":"2023-11-24T08:16:05.475382+0000","last_unstale":"2023-11-24T08:16:07.499382+0000","last_undegraded":"2023-11-24T08:16:07.499382+0000","last_fullsized":"2023-11-24T08:16:07.499382+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:59.809160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747736+0000","last_change":"2023-11-24T08:16:06.493435+0000","last_active":"2023-11-24T08:16:07.747736+0000","last_peered":"2023-11-24T08:16:07.747736+0000","last_clean":"2023-11-24T08:16:07.747736+0000","last_became_active":"2023-11-24T08:16:06.493097+0000","last_became_peered":"2023-11-24T08:16:06.493097+0000","last_unstale":"2023-11-24T08:16:07.747736+0000","last_undegraded":"2023-11-24T08:16:07.747736+0000","last_fullsized":"2023-11-24T08:16:07.747736+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:11:39.961690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405428+0000","last_change":"2023-11-24T08:16:05.470646+0000","last_active":"2023-11-24T08:16:08.405428+0000","last_peered":"2023-11-24T08:16:08.405428+0000","last_clean":"2023-11-24T08:16:08.405428+0000","last_became_active":"2023-11-24T08:16:05.470003+0000","last_became_peered":"2023-11-24T08:16:05.470003+0000","last_unstale":"2023-11-24T08:16:08.405428+0000","last_undegraded":"2023-11-24T08:16:08.405428+0000","last_fullsized":"2023-11-24T08:16:08.405428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:23:02.214199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500030+0000","last_change":"2023-11-24T08:16:06.498757+0000","last_active":"2023-11-24T08:16:07.500030+0000","last_peered":"2023-11-24T08:16:07.500030+0000","last_clean":"2023-11-24T08:16:07.500030+0000","last_became_active":"2023-11-24T08:16:06.498499+0000","last_became_peered":"2023-11-24T08:16:06.498499+0000","last_unstale":"2023-11-24T08:16:07.500030+0000","last_undegraded":"2023-11-24T08:16:07.500030+0000","last_fullsized":"2023-11-24T08:16:07.500030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:25.869197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502172+0000","last_change":"2023-11-24T08:16:05.474788+0000","last_active":"2023-11-24T08:16:07.502172+0000","last_peered":"2023-11-24T08:16:07.502172+0000","last_clean":"2023-11-24T08:16:07.502172+0000","last_became_active":"2023-11-24T08:16:05.474469+0000","last_became_peered":"2023-11-24T08:16:05.474469+0000","last_unstale":"2023-11-24T08:16:07.502172+0000","last_undegraded":"2023-11-24T08:16:07.502172+0000","last_fullsized":"2023-11-24T08:16:07.502172+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:54.676954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500672+0000","last_change":"2023-11-24T08:16:06.505815+0000","last_active":"2023-11-24T08:16:07.500672+0000","last_peered":"2023-11-24T08:16:07.500672+0000","last_clean":"2023-11-24T08:16:07.500672+0000","last_became_active":"2023-11-24T08:16:06.505499+0000","last_became_peered":"2023-11-24T08:16:06.505499+0000","last_unstale":"2023-11-24T08:16:07.500672+0000","last_undegraded":"2023-11-24T08:16:07.500672+0000","last_fullsized":"2023-11-24T08:16:07.500672+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:29:07.362844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747944+0000","last_change":"2023-11-24T08:16:05.463018+0000","last_active":"2023-11-24T08:16:07.747944+0000","last_peered":"2023-11-24T08:16:07.747944+0000","last_clean":"2023-11-24T08:16:07.747944+0000","last_became_active":"2023-11-24T08:16:05.461567+0000","last_became_peered":"2023-11-24T08:16:05.461567+0000","last_unstale":"2023-11-24T08:16:07.747944+0000","last_undegraded":"2023-11-24T08:16:07.747944+0000","last_fullsized":"2023-11-24T08:16:07.747944+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:46.099656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521928+0000","last_change":"2023-11-24T08:16:06.499081+0000","last_active":"2023-11-24T08:16:07.521928+0000","last_peered":"2023-11-24T08:16:07.521928+0000","last_clean":"2023-11-24T08:16:07.521928+0000","last_became_active":"2023-11-24T08:16:06.498303+0000","last_became_peered":"2023-11-24T08:16:06.498303+0000","last_unstale":"2023-11-24T08:16:07.521928+0000","last_undegraded":"2023-11-24T08:16:07.521928+0000","last_fullsized":"2023-11-24T08:16:07.521928+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:11.352447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522103+0000","last_change":"2023-11-24T08:16:05.477279+0000","last_active":"2023-11-24T08:16:07.522103+0000","last_peered":"2023-11-24T08:16:07.522103+0000","last_clean":"2023-11-24T08:16:07.522103+0000","last_became_active":"2023-11-24T08:16:05.477022+0000","last_became_peered":"2023-11-24T08:16:05.477022+0000","last_unstale":"2023-11-24T08:16:07.522103+0000","last_undegraded":"2023-11-24T08:16:07.522103+0000","last_fullsized":"2023-11-24T08:16:07.522103+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:15.984683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405724+0000","last_change":"2023-11-24T08:16:06.494192+0000","last_active":"2023-11-24T08:16:08.405724+0000","last_peered":"2023-11-24T08:16:08.405724+0000","last_clean":"2023-11-24T08:16:08.405724+0000","last_became_active":"2023-11-24T08:16:06.493586+0000","last_became_peered":"2023-11-24T08:16:06.493586+0000","last_unstale":"2023-11-24T08:16:08.405724+0000","last_undegraded":"2023-11-24T08:16:08.405724+0000","last_fullsized":"2023-11-24T08:16:08.405724+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:28.523247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"146'6","reported_seq":18,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343624+0000","last_change":"2023-11-24T08:16:05.465947+0000","last_active":"2023-11-24T08:16:08.343624+0000","last_peered":"2023-11-24T08:16:08.343624+0000","last_clean":"2023-11-24T08:16:08.343624+0000","last_became_active":"2023-11-24T08:16:05.465669+0000","last_became_peered":"2023-11-24T08:16:05.465669+0000","last_unstale":"2023-11-24T08:16:08.343624+0000","last_undegraded":"2023-11-24T08:16:08.343624+0000","last_fullsized":"2023-11-24T08:16:08.343624+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:14.380302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405459+0000","last_change":"2023-11-24T08:16:06.498585+0000","last_active":"2023-11-24T08:16:08.405459+0000","last_peered":"2023-11-24T08:16:08.405459+0000","last_clean":"2023-11-24T08:16:08.405459+0000","last_became_active":"2023-11-24T08:16:06.498289+0000","last_became_peered":"2023-11-24T08:16:06.498289+0000","last_unstale":"2023-11-24T08:16:08.405459+0000","last_undegraded":"2023-11-24T08:16:08.405459+0000","last_fullsized":"2023-11-24T08:16:08.405459+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:26:13.648432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405478+0000","last_change":"2023-11-24T08:16:05.464377+0000","last_active":"2023-11-24T08:16:08.405478+0000","last_peered":"2023-11-24T08:16:08.405478+0000","last_clean":"2023-11-24T08:16:08.405478+0000","last_became_active":"2023-11-24T08:16:05.462468+0000","last_became_peered":"2023-11-24T08:16:05.462468+0000","last_unstale":"2023-11-24T08:16:08.405478+0000","last_undegraded":"2023-11-24T08:16:08.405478+0000","last_fullsized":"2023-11-24T08:16:08.405478+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:48.183451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747245+0000","last_change":"2023-11-24T08:16:06.510216+0000","last_active":"2023-11-24T08:16:07.747245+0000","last_peered":"2023-11-24T08:16:07.747245+0000","last_clean":"2023-11-24T08:16:07.747245+0000","last_became_active":"2023-11-24T08:16:06.509905+0000","last_became_peered":"2023-11-24T08:16:06.509905+0000","last_unstale":"2023-11-24T08:16:07.747245+0000","last_undegraded":"2023-11-24T08:16:07.747245+0000","last_fullsized":"2023-11-24T08:16:07.747245+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:03:50.803641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405396+0000","last_change":"2023-11-24T08:16:05.472489+0000","last_active":"2023-11-24T08:16:08.405396+0000","last_peered":"2023-11-24T08:16:08.405396+0000","last_clean":"2023-11-24T08:16:08.405396+0000","last_became_active":"2023-11-24T08:16:05.471978+0000","last_became_peered":"2023-11-24T08:16:05.471978+0000","last_unstale":"2023-11-24T08:16:08.405396+0000","last_undegraded":"2023-11-24T08:16:08.405396+0000","last_fullsized":"2023-11-24T08:16:08.405396+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:58.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499397+0000","last_change":"2023-11-24T08:16:06.492482+0000","last_active":"2023-11-24T08:16:07.499397+0000","last_peered":"2023-11-24T08:16:07.499397+0000","last_clean":"2023-11-24T08:16:07.499397+0000","last_became_active":"2023-11-24T08:16:06.492073+0000","last_became_peered":"2023-11-24T08:16:06.492073+0000","last_unstale":"2023-11-24T08:16:07.499397+0000","last_undegraded":"2023-11-24T08:16:07.499397+0000","last_fullsized":"2023-11-24T08:16:07.499397+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.750054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406403+0000","last_change":"2023-11-24T08:16:05.456418+0000","last_active":"2023-11-24T08:16:08.406403+0000","last_peered":"2023-11-24T08:16:08.406403+0000","last_clean":"2023-11-24T08:16:08.406403+0000","last_became_active":"2023-11-24T08:16:05.456060+0000","last_became_peered":"2023-11-24T08:16:05.456060+0000","last_unstale":"2023-11-24T08:16:08.406403+0000","last_undegraded":"2023-11-24T08:16:08.406403+0000","last_fullsized":"2023-11-24T08:16:08.406403+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:41.821611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404899+0000","last_change":"2023-11-24T08:16:06.494565+0000","last_active":"2023-11-24T08:16:08.404899+0000","last_peered":"2023-11-24T08:16:08.404899+0000","last_clean":"2023-11-24T08:16:08.404899+0000","last_became_active":"2023-11-24T08:16:06.494262+0000","last_became_peered":"2023-11-24T08:16:06.494262+0000","last_unstale":"2023-11-24T08:16:08.404899+0000","last_undegraded":"2023-11-24T08:16:08.404899+0000","last_fullsized":"2023-11-24T08:16:08.404899+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:42.131310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"146'8","reported_seq":20,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405287+0000","last_change":"2023-11-24T08:16:05.470464+0000","last_active":"2023-11-24T08:16:08.405287+0000","last_peered":"2023-11-24T08:16:08.405287+0000","last_clean":"2023-11-24T08:16:08.405287+0000","last_became_active":"2023-11-24T08:16:05.469750+0000","last_became_peered":"2023-11-24T08:16:05.469750+0000","last_unstale":"2023-11-24T08:16:08.405287+0000","last_undegraded":"2023-11-24T08:16:08.405287+0000","last_fullsized":"2023-11-24T08:16:08.405287+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:13.981312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521981+0000","last_change":"2023-11-24T08:16:06.504930+0000","last_active":"2023-11-24T08:16:07.521981+0000","last_peered":"2023-11-24T08:16:07.521981+0000","last_clean":"2023-11-24T08:16:07.521981+0000","last_became_active":"2023-11-24T08:16:06.504631+0000","last_became_peered":"2023-11-24T08:16:06.504631+0000","last_unstale":"2023-11-24T08:16:07.521981+0000","last_undegraded":"2023-11-24T08:16:07.521981+0000","last_fullsized":"2023-11-24T08:16:07.521981+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:47:54.139953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500301+0000","last_change":"2023-11-24T08:16:05.471508+0000","last_active":"2023-11-24T08:16:07.500301+0000","last_peered":"2023-11-24T08:16:07.500301+0000","last_clean":"2023-11-24T08:16:07.500301+0000","last_became_active":"2023-11-24T08:16:05.471218+0000","last_became_peered":"2023-11-24T08:16:05.471218+0000","last_unstale":"2023-11-24T08:16:07.500301+0000","last_undegraded":"2023-11-24T08:16:07.500301+0000","last_fullsized":"2023-11-24T08:16:07.500301+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:36.210681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406271+0000","last_change":"2023-11-24T08:16:06.497979+0000","last_active":"2023-11-24T08:16:08.406271+0000","last_peered":"2023-11-24T08:16:08.406271+0000","last_clean":"2023-11-24T08:16:08.406271+0000","last_became_active":"2023-11-24T08:16:06.496582+0000","last_became_peered":"2023-11-24T08:16:06.496582+0000","last_unstale":"2023-11-24T08:16:08.406271+0000","last_undegraded":"2023-11-24T08:16:08.406271+0000","last_fullsized":"2023-11-24T08:16:08.406271+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:56.750054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747693+0000","last_change":"2023-11-24T08:16:05.469048+0000","last_active":"2023-11-24T08:16:07.747693+0000","last_peered":"2023-11-24T08:16:07.747693+0000","last_clean":"2023-11-24T08:16:07.747693+0000","last_became_active":"2023-11-24T08:16:05.468291+0000","last_became_peered":"2023-11-24T08:16:05.468291+0000","last_unstale":"2023-11-24T08:16:07.747693+0000","last_undegraded":"2023-11-24T08:16:07.747693+0000","last_fullsized":"2023-11-24T08:16:07.747693+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:00:53.941171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405737+0000","last_change":"2023-11-24T08:16:06.499502+0000","last_active":"2023-11-24T08:16:08.405737+0000","last_peered":"2023-11-24T08:16:08.405737+0000","last_clean":"2023-11-24T08:16:08.405737+0000","last_became_active":"2023-11-24T08:16:06.499202+0000","last_became_peered":"2023-11-24T08:16:06.499202+0000","last_unstale":"2023-11-24T08:16:08.405737+0000","last_undegraded":"2023-11-24T08:16:08.405737+0000","last_fullsized":"2023-11-24T08:16:08.405737+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:33.792816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"146'1","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406155+0000","last_change":"2023-11-24T08:16:05.473595+0000","last_active":"2023-11-24T08:16:08.406155+0000","last_peered":"2023-11-24T08:16:08.406155+0000","last_clean":"2023-11-24T08:16:08.406155+0000","last_became_active":"2023-11-24T08:16:05.473320+0000","last_became_peered":"2023-11-24T08:16:05.473320+0000","last_unstale":"2023-11-24T08:16:08.406155+0000","last_undegraded":"2023-11-24T08:16:08.406155+0000","last_fullsized":"2023-11-24T08:16:08.406155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:38:32.672128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407219+0000","last_change":"2023-11-24T08:16:06.503795+0000","last_active":"2023-11-24T08:16:08.407219+0000","last_peered":"2023-11-24T08:16:08.407219+0000","last_clean":"2023-11-24T08:16:08.407219+0000","last_became_active":"2023-11-24T08:16:06.503534+0000","last_became_peered":"2023-11-24T08:16:06.503534+0000","last_unstale":"2023-11-24T08:16:08.407219+0000","last_undegraded":"2023-11-24T08:16:08.407219+0000","last_fullsized":"2023-11-24T08:16:08.407219+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:33:17.448690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747629+0000","last_change":"2023-11-24T08:16:05.464915+0000","last_active":"2023-11-24T08:16:07.747629+0000","last_peered":"2023-11-24T08:16:07.747629+0000","last_clean":"2023-11-24T08:16:07.747629+0000","last_became_active":"2023-11-24T08:16:05.464638+0000","last_became_peered":"2023-11-24T08:16:05.464638+0000","last_unstale":"2023-11-24T08:16:07.747629+0000","last_undegraded":"2023-11-24T08:16:07.747629+0000","last_fullsized":"2023-11-24T08:16:07.747629+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:32.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.503193+0000","last_change":"2023-11-24T08:16:06.503282+0000","last_active":"2023-11-24T08:16:07.503193+0000","last_peered":"2023-11-24T08:16:07.503193+0000","last_clean":"2023-11-24T08:16:07.503193+0000","last_became_active":"2023-11-24T08:16:06.501862+0000","last_became_peered":"2023-11-24T08:16:06.501862+0000","last_unstale":"2023-11-24T08:16:07.503193+0000","last_undegraded":"2023-11-24T08:16:07.503193+0000","last_fullsized":"2023-11-24T08:16:07.503193+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:48.205914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343007+0000","last_change":"2023-11-24T08:16:05.464105+0000","last_active":"2023-11-24T08:16:08.343007+0000","last_peered":"2023-11-24T08:16:08.343007+0000","last_clean":"2023-11-24T08:16:08.343007+0000","last_became_active":"2023-11-24T08:16:05.463226+0000","last_became_peered":"2023-11-24T08:16:05.463226+0000","last_unstale":"2023-11-24T08:16:08.343007+0000","last_undegraded":"2023-11-24T08:16:08.343007+0000","last_fullsized":"2023-11-24T08:16:08.343007+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:49.322938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343163+0000","last_change":"2023-11-24T08:16:06.505815+0000","last_active":"2023-11-24T08:16:08.343163+0000","last_peered":"2023-11-24T08:16:08.343163+0000","last_clean":"2023-11-24T08:16:08.343163+0000","last_became_active":"2023-11-24T08:16:06.505528+0000","last_became_peered":"2023-11-24T08:16:06.505528+0000","last_unstale":"2023-11-24T08:16:08.343163+0000","last_undegraded":"2023-11-24T08:16:08.343163+0000","last_fullsized":"2023-11-24T08:16:08.343163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:25:09.051607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520988+0000","last_change":"2023-11-24T08:16:05.472454+0000","last_active":"2023-11-24T08:16:07.520988+0000","last_peered":"2023-11-24T08:16:07.520988+0000","last_clean":"2023-11-24T08:16:07.520988+0000","last_became_active":"2023-11-24T08:16:05.471551+0000","last_became_peered":"2023-11-24T08:16:05.471551+0000","last_unstale":"2023-11-24T08:16:07.520988+0000","last_undegraded":"2023-11-24T08:16:07.520988+0000","last_fullsized":"2023-11-24T08:16:07.520988+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:02:06.765242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.502106+0000","last_change":"2023-11-24T08:16:06.493710+0000","last_active":"2023-11-24T08:16:07.502106+0000","last_peered":"2023-11-24T08:16:07.502106+0000","last_clean":"2023-11-24T08:16:07.502106+0000","last_became_active":"2023-11-24T08:16:06.493374+0000","last_became_peered":"2023-11-24T08:16:06.493374+0000","last_unstale":"2023-11-24T08:16:07.502106+0000","last_undegraded":"2023-11-24T08:16:07.502106+0000","last_fullsized":"2023-11-24T08:16:07.502106+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:33.792816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.520642+0000","last_change":"2023-11-24T08:16:05.466740+0000","last_active":"2023-11-24T08:16:07.520642+0000","last_peered":"2023-11-24T08:16:07.520642+0000","last_clean":"2023-11-24T08:16:07.520642+0000","last_became_active":"2023-11-24T08:16:05.466472+0000","last_became_peered":"2023-11-24T08:16:05.466472+0000","last_unstale":"2023-11-24T08:16:07.520642+0000","last_undegraded":"2023-11-24T08:16:07.520642+0000","last_fullsized":"2023-11-24T08:16:07.520642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:01:17.812786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343002+0000","last_change":"2023-11-24T08:16:06.500970+0000","last_active":"2023-11-24T08:16:08.343002+0000","last_peered":"2023-11-24T08:16:08.343002+0000","last_clean":"2023-11-24T08:16:08.343002+0000","last_became_active":"2023-11-24T08:16:06.500558+0000","last_became_peered":"2023-11-24T08:16:06.500558+0000","last_unstale":"2023-11-24T08:16:08.343002+0000","last_undegraded":"2023-11-24T08:16:08.343002+0000","last_fullsized":"2023-11-24T08:16:08.343002+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:23.610229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404786+0000","last_change":"2023-11-24T08:16:05.467270+0000","last_active":"2023-11-24T08:16:08.404786+0000","last_peered":"2023-11-24T08:16:08.404786+0000","last_clean":"2023-11-24T08:16:08.404786+0000","last_became_active":"2023-11-24T08:16:05.466840+0000","last_became_peered":"2023-11-24T08:16:05.466840+0000","last_unstale":"2023-11-24T08:16:08.404786+0000","last_undegraded":"2023-11-24T08:16:08.404786+0000","last_fullsized":"2023-11-24T08:16:08.404786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:54.676954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500924+0000","last_change":"2023-11-24T08:16:06.492643+0000","last_active":"2023-11-24T08:16:07.500924+0000","last_peered":"2023-11-24T08:16:07.500924+0000","last_clean":"2023-11-24T08:16:07.500924+0000","last_became_active":"2023-11-24T08:16:06.492330+0000","last_became_peered":"2023-11-24T08:16:06.492330+0000","last_unstale":"2023-11-24T08:16:07.500924+0000","last_undegraded":"2023-11-24T08:16:07.500924+0000","last_fullsized":"2023-11-24T08:16:07.500924+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:42:16.473668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"146'2","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.522082+0000","last_change":"2023-11-24T08:16:05.474751+0000","last_active":"2023-11-24T08:16:07.522082+0000","last_peered":"2023-11-24T08:16:07.522082+0000","last_clean":"2023-11-24T08:16:07.522082+0000","last_became_active":"2023-11-24T08:16:05.474468+0000","last_became_peered":"2023-11-24T08:16:05.474468+0000","last_unstale":"2023-11-24T08:16:07.522082+0000","last_undegraded":"2023-11-24T08:16:07.522082+0000","last_fullsized":"2023-11-24T08:16:07.522082+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:09.359649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406020+0000","last_change":"2023-11-24T08:16:06.498734+0000","last_active":"2023-11-24T08:16:08.406020+0000","last_peered":"2023-11-24T08:16:08.406020+0000","last_clean":"2023-11-24T08:16:08.406020+0000","last_became_active":"2023-11-24T08:16:06.498309+0000","last_became_peered":"2023-11-24T08:16:06.498309+0000","last_unstale":"2023-11-24T08:16:08.406020+0000","last_undegraded":"2023-11-24T08:16:08.406020+0000","last_fullsized":"2023-11-24T08:16:08.406020+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:14:56.087388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407016+0000","last_change":"2023-11-24T08:16:05.474544+0000","last_active":"2023-11-24T08:16:08.407016+0000","last_peered":"2023-11-24T08:16:08.407016+0000","last_clean":"2023-11-24T08:16:08.407016+0000","last_became_active":"2023-11-24T08:16:05.473862+0000","last_became_peered":"2023-11-24T08:16:05.473862+0000","last_unstale":"2023-11-24T08:16:08.407016+0000","last_undegraded":"2023-11-24T08:16:08.407016+0000","last_fullsized":"2023-11-24T08:16:08.407016+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:01.910084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.748140+0000","last_change":"2023-11-24T08:16:06.503128+0000","last_active":"2023-11-24T08:16:07.748140+0000","last_peered":"2023-11-24T08:16:07.748140+0000","last_clean":"2023-11-24T08:16:07.748140+0000","last_became_active":"2023-11-24T08:16:06.502852+0000","last_became_peered":"2023-11-24T08:16:06.502852+0000","last_unstale":"2023-11-24T08:16:07.748140+0000","last_undegraded":"2023-11-24T08:16:07.748140+0000","last_fullsized":"2023-11-24T08:16:07.748140+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:11.352447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407453+0000","last_change":"2023-11-24T08:16:05.462733+0000","last_active":"2023-11-24T08:16:08.407453+0000","last_peered":"2023-11-24T08:16:08.407453+0000","last_clean":"2023-11-24T08:16:08.407453+0000","last_became_active":"2023-11-24T08:16:05.462292+0000","last_became_peered":"2023-11-24T08:16:05.462292+0000","last_unstale":"2023-11-24T08:16:08.407453+0000","last_undegraded":"2023-11-24T08:16:08.407453+0000","last_fullsized":"2023-11-24T08:16:08.407453+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:39.617325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.407122+0000","last_change":"2023-11-24T08:16:06.500802+0000","last_active":"2023-11-24T08:16:08.407122+0000","last_peered":"2023-11-24T08:16:08.407122+0000","last_clean":"2023-11-24T08:16:08.407122+0000","last_became_active":"2023-11-24T08:16:06.500499+0000","last_became_peered":"2023-11-24T08:16:06.500499+0000","last_unstale":"2023-11-24T08:16:08.407122+0000","last_undegraded":"2023-11-24T08:16:08.407122+0000","last_fullsized":"2023-11-24T08:16:08.407122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:05.251377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.406542+0000","last_change":"2023-11-24T08:16:05.464913+0000","last_active":"2023-11-24T08:16:08.406542+0000","last_peered":"2023-11-24T08:16:08.406542+0000","last_clean":"2023-11-24T08:16:08.406542+0000","last_became_active":"2023-11-24T08:16:05.464604+0000","last_became_peered":"2023-11-24T08:16:05.464604+0000","last_unstale":"2023-11-24T08:16:08.406542+0000","last_undegraded":"2023-11-24T08:16:08.406542+0000","last_fullsized":"2023-11-24T08:16:08.406542+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:20:19.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405156+0000","last_change":"2023-11-24T08:16:06.498071+0000","last_active":"2023-11-24T08:16:08.405156+0000","last_peered":"2023-11-24T08:16:08.405156+0000","last_clean":"2023-11-24T08:16:08.405156+0000","last_became_active":"2023-11-24T08:16:06.497807+0000","last_became_peered":"2023-11-24T08:16:06.497807+0000","last_unstale":"2023-11-24T08:16:08.405156+0000","last_undegraded":"2023-11-24T08:16:08.405156+0000","last_fullsized":"2023-11-24T08:16:08.405156+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:00.820777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343544+0000","last_change":"2023-11-24T08:16:05.470909+0000","last_active":"2023-11-24T08:16:08.343544+0000","last_peered":"2023-11-24T08:16:08.343544+0000","last_clean":"2023-11-24T08:16:08.343544+0000","last_became_active":"2023-11-24T08:16:05.470576+0000","last_became_peered":"2023-11-24T08:16:05.470576+0000","last_unstale":"2023-11-24T08:16:08.343544+0000","last_undegraded":"2023-11-24T08:16:08.343544+0000","last_fullsized":"2023-11-24T08:16:08.343544+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:57:30.064033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.500542+0000","last_change":"2023-11-24T08:16:06.491032+0000","last_active":"2023-11-24T08:16:07.500542+0000","last_peered":"2023-11-24T08:16:07.500542+0000","last_clean":"2023-11-24T08:16:07.500542+0000","last_became_active":"2023-11-24T08:16:06.490600+0000","last_became_peered":"2023-11-24T08:16:06.490600+0000","last_unstale":"2023-11-24T08:16:07.500542+0000","last_undegraded":"2023-11-24T08:16:07.500542+0000","last_fullsized":"2023-11-24T08:16:07.500542+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:02.921701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521039+0000","last_change":"2023-11-24T08:16:05.473795+0000","last_active":"2023-11-24T08:16:07.521039+0000","last_peered":"2023-11-24T08:16:07.521039+0000","last_clean":"2023-11-24T08:16:07.521039+0000","last_became_active":"2023-11-24T08:16:05.473360+0000","last_became_peered":"2023-11-24T08:16:05.473360+0000","last_unstale":"2023-11-24T08:16:07.521039+0000","last_undegraded":"2023-11-24T08:16:07.521039+0000","last_fullsized":"2023-11-24T08:16:07.521039+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:14:22.598612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.521211+0000","last_change":"2023-11-24T08:16:06.512282+0000","last_active":"2023-11-24T08:16:07.521211+0000","last_peered":"2023-11-24T08:16:07.521211+0000","last_clean":"2023-11-24T08:16:07.521211+0000","last_became_active":"2023-11-24T08:16:06.511875+0000","last_became_peered":"2023-11-24T08:16:06.511875+0000","last_unstale":"2023-11-24T08:16:07.521211+0000","last_undegraded":"2023-11-24T08:16:07.521211+0000","last_fullsized":"2023-11-24T08:16:07.521211+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:30.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.404986+0000","last_change":"2023-11-24T08:16:05.461710+0000","last_active":"2023-11-24T08:16:08.404986+0000","last_peered":"2023-11-24T08:16:08.404986+0000","last_clean":"2023-11-24T08:16:08.404986+0000","last_became_active":"2023-11-24T08:16:05.461401+0000","last_became_peered":"2023-11-24T08:16:05.461401+0000","last_unstale":"2023-11-24T08:16:08.404986+0000","last_undegraded":"2023-11-24T08:16:08.404986+0000","last_fullsized":"2023-11-24T08:16:08.404986+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:46:08.659848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.499510+0000","last_change":"2023-11-24T08:16:06.505809+0000","last_active":"2023-11-24T08:16:07.499510+0000","last_peered":"2023-11-24T08:16:07.499510+0000","last_clean":"2023-11-24T08:16:07.499510+0000","last_became_active":"2023-11-24T08:16:06.505516+0000","last_became_peered":"2023-11-24T08:16:06.505516+0000","last_unstale":"2023-11-24T08:16:07.499510+0000","last_undegraded":"2023-11-24T08:16:07.499510+0000","last_fullsized":"2023-11-24T08:16:07.499510+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:32:01.370895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:07.747813+0000","last_change":"2023-11-24T08:16:05.473443+0000","last_active":"2023-11-24T08:16:07.747813+0000","last_peered":"2023-11-24T08:16:07.747813+0000","last_clean":"2023-11-24T08:16:07.747813+0000","last_became_active":"2023-11-24T08:16:05.473132+0000","last_became_peered":"2023-11-24T08:16:05.473132+0000","last_unstale":"2023-11-24T08:16:07.747813+0000","last_undegraded":"2023-11-24T08:16:07.747813+0000","last_fullsized":"2023-11-24T08:16:07.747813+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:15.984683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.343330+0000","last_change":"2023-11-24T08:16:06.504898+0000","last_active":"2023-11-24T08:16:08.343330+0000","last_peered":"2023-11-24T08:16:08.343330+0000","last_clean":"2023-11-24T08:16:08.343330+0000","last_became_active":"2023-11-24T08:16:06.504615+0000","last_became_peered":"2023-11-24T08:16:06.504615+0000","last_unstale":"2023-11-24T08:16:08.343330+0000","last_undegraded":"2023-11-24T08:16:08.343330+0000","last_fullsized":"2023-11-24T08:16:08.343330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:05.430102+0000","last_clean_scrub_stamp":"2023-11-24T08:16:05.430102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:27.219214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"146'3","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-24T08:16:08.405923+0000","last_change":"2023-11-24T08:16:05.464230+0000","last_active":"2023-11-24T08:16:08.405923+0000","last_peered":"2023-11-24T08:16:08.405923+0000","last_clean":"2023-11-24T08:16:08.405923+0000","last_became_active":"2023-11-24T08:16:05.462859+0000","last_became_peered":"2023-11-24T08:16:05.462859+0000","last_unstale":"2023-11-24T08:16:08.405923+0000","last_undegraded":"2023-11-24T08:16:08.405923+0000","last_fullsized":"2023-11-24T08:16:08.405923+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:16:04.418485+0000","last_clean_scrub_stamp":"2023-11-24T08:16:04.418485+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:32.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-11-24T08:16:06.482695+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:16:06.482695+0000","last_peered":"2023-11-24T08:16:06.482695+0000","last_clean":"2023-11-24T08:16:06.482695+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:16:06.482695+0000","last_undegraded":"2023-11-24T08:16:06.482695+0000","last_fullsized":"2023-11-24T08:16:06.482695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2441216,"data_stored":1397748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.036},"min":{"1min":0.75800000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.0529999999999999,"15min":1.089},"min":{"1min":0.78200000000000003,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.774,"5min":1.794,"15min":2.7109999999999999},"last":1.339}]},{"osd":1,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.151,"15min":1.1579999999999999},"min":{"1min":0.79900000000000004,"5min":0.64700000000000002,"15min":0.46300000000000002},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":1.486},{"interface":"front","average":{"1min":1.391,"5min":1.2430000000000001,"15min":1.216},"min":{"1min":0.78500000000000003,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.95699999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.117,"15min":1.169},"min":{"1min":0.70499999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.698,"5min":1.8859999999999999,"15min":4.7000000000000002},"last":1.161},{"interface":"front","average":{"1min":1.407,"5min":1.3029999999999999,"15min":1.3400000000000001},"min":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.8480000000000001,"5min":1.889,"15min":4.633},"last":1.7310000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.2110000000000001,"15min":1.1919999999999999},"min":{"1min":0.93200000000000005,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.99,"5min":1.99,"15min":3.133},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.369,"15min":1.3779999999999999},"min":{"1min":1.119,"5min":0.622,"15min":0.622},"max":{"1min":1.98,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.627}]},{"osd":4,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":1.359,"15min":1.3500000000000001},"min":{"1min":1.095,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":2.665,"5min":2.665,"15min":2.851},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.345,"15min":1.345},"min":{"1min":0.84599999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.298}]},{"osd":5,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.476,"15min":1.4730000000000001},"min":{"1min":1.1479999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.8440000000000001},{"interface":"front","average":{"1min":1.786,"5min":1.5229999999999999,"15min":1.5009999999999999},"min":{"1min":1.2529999999999999,"5min":0.622,"15min":0.622},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.6950000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.5840000000000001,"15min":1.5649999999999999},"min":{"1min":1.0740000000000001,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":2.702,"5min":2.702,"15min":2.992},"last":1.583},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.508,"15min":1.4870000000000001},"min":{"1min":1.1859999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":3.798},"last":1.4550000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1851392,"data_stored":806874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1279999999999999,"15min":1.0920000000000001},"min":{"1min":0.70399999999999996,"5min":0.63700000000000001,"15min":0.621},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.95},"last":1.639},{"interface":"front","average":{"1min":1.089,"5min":1.099,"15min":1.0940000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.58099999999999996},"max":{"1min":1.5349999999999999,"5min":1.804,"15min":2.2069999999999999},"last":0.998}]},{"osd":1,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.196,"15min":1.2130000000000001},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.835,"5min":1.835,"15min":2.0230000000000001},"last":1.474},{"interface":"front","average":{"1min":1.232,"5min":1.2070000000000001,"15min":1.228},"min":{"1min":0.877,"5min":0.72999999999999998,"15min":0.65000000000000002},"max":{"1min":1.7070000000000001,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.113,"15min":1.1120000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.323},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.60399999999999998},"max":{"1min":1.6970000000000001,"5min":1.8220000000000001,"15min":2.3239999999999998},"last":1.6970000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3740000000000001,"15min":1.369},"min":{"1min":0.88800000000000001,"5min":0.86399999999999999,"15min":0.71499999999999997},"max":{"1min":2.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3320000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.6850000000000001,"5min":4.8319999999999999,"15min":4.8319999999999999},"last":1.512}]},{"osd":4,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.452,"15min":1.4830000000000001},"min":{"1min":1.0880000000000001,"5min":0.85399999999999998,"15min":0.76500000000000001},"max":{"1min":1.9399999999999999,"5min":2.1190000000000002,"15min":2.895},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.498,"15min":1.5229999999999999},"min":{"1min":1.0509999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9510000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4370000000000001,"15min":1.4910000000000001},"min":{"1min":0.92900000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7410000000000001,"5min":2.3300000000000001,"15min":2.629},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.571,"15min":1.6220000000000001},"min":{"1min":0.98499999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0289999999999999,"5min":2.165,"15min":2.7759999999999998},"last":1.5620000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.599,"15min":1.6499999999999999},"min":{"1min":1.266,"5min":0.748,"15min":0.748},"max":{"1min":1.97,"5min":2.4100000000000001,"15min":3.0329999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.6060000000000001,"15min":1.6359999999999999},"min":{"1min":1.143,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.927,"5min":2.512,"15min":2.6890000000000001},"last":1.5920000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574988,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1843200,"data_stored":806750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0229999999999999,"15min":1.081},"min":{"1min":0.69399999999999995,"5min":0.58599999999999997,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.694,"15min":3.1179999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0920000000000001,"15min":1.151},"min":{"1min":0.80300000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5600000000000001,"5min":1.859,"15min":2.198},"last":0.91900000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.117,"15min":1.149},"min":{"1min":0.79500000000000004,"5min":0.69699999999999995,"15min":0.66700000000000004},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":2.0529999999999999},"last":1.466},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1890000000000001,"15min":1.1619999999999999},"min":{"1min":0.84899999999999998,"5min":0.73399999999999999,"15min":0.48699999999999999},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.421}]},{"osd":2,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.82299999999999995,"5min":0.71099999999999997,"15min":0.51600000000000001},"max":{"1min":1.474,"5min":1.768,"15min":1.8520000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.163,"15min":1.1659999999999999},"min":{"1min":0.82199999999999995,"5min":0.72499999999999998,"15min":0.64200000000000002},"max":{"1min":1.784,"5min":1.784,"15min":1.8129999999999999},"last":1.5680000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1339999999999999,"15min":1.1699999999999999},"min":{"1min":0.89800000000000002,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.254,"15min":1.3049999999999999},"min":{"1min":0.876,"5min":0.78000000000000003,"15min":0.55200000000000005},"max":{"1min":1.5680000000000001,"5min":1.893,"15min":1.99},"last":1.2290000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.3120000000000001,"15min":1.3320000000000001},"min":{"1min":0.98399999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.268},{"interface":"front","average":{"1min":1.446,"5min":1.4370000000000001,"15min":1.508},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.46700000000000003},"max":{"1min":1.994,"5min":2.0009999999999999,"15min":3.5030000000000001},"last":1.514}]},{"osd":6,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.306,"15min":1.397},"min":{"1min":0.81899999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.085,"5min":2.085,"15min":3.2949999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.429},"min":{"1min":1.1319999999999999,"5min":0.69299999999999995,"15min":0.60999999999999999},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.034}]},{"osd":7,"last update":"Fri Nov 24 08:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.48,"15min":1.5469999999999999},"min":{"1min":1.0069999999999999,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":3.4420000000000002},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.591,"15min":1.603},"min":{"1min":1.087,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":2.113,"5min":2.181,"15min":2.181},"last":1.845}]}]},{"osd":4,"up_from":13,"seq":55834574990,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1839104,"data_stored":806728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0780000000000001,"15min":1.117},"min":{"1min":0.626,"5min":0.59999999999999998,"15min":0.42699999999999999},"max":{"1min":1.5780000000000001,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.129,"15min":1.0980000000000001},"min":{"1min":0.69299999999999995,"5min":0.63600000000000001,"15min":0.46100000000000002},"max":{"1min":5.452,"5min":5.452,"15min":5.452},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1359999999999999,"15min":1.149},"min":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.64200000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.1679999999999999,"15min":1.181},"min":{"1min":0.73799999999999999,"5min":0.72099999999999997,"15min":0.51900000000000002},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.3120000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.0980000000000001,"15min":1.129},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.64600000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2350000000000001,"15min":1.3100000000000001},"min":{"1min":0.877,"5min":0.71199999999999997,"15min":0.505},"max":{"1min":1.6950000000000001,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.014}]},{"osd":3,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3380000000000001,"15min":1.3879999999999999},"min":{"1min":0.74399999999999999,"5min":0.73599999999999999,"15min":0.62},"max":{"1min":1.7250000000000001,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.371},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.96399999999999997,"5min":0.91200000000000003,"15min":0.55800000000000005},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.429}]},{"osd":5,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.4950000000000001,"15min":1.5009999999999999},"min":{"1min":1.258,"5min":0.71999999999999997,"15min":0.69299999999999995},"max":{"1min":1.8799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.46,"15min":1.502},"min":{"1min":1.0900000000000001,"5min":0.78200000000000003,"15min":0.60199999999999998},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6100000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.256,"15min":1.367},"min":{"1min":1.016,"5min":0.57499999999999996,"15min":0.50900000000000001},"max":{"1min":1.907,"5min":1.907,"15min":3.5899999999999999},"last":1.524},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.548,"15min":1.6080000000000001},"min":{"1min":1.397,"5min":0.79000000000000004,"15min":0.58099999999999996},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":3.391},"last":1.6539999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.51,"15min":1.5740000000000001},"min":{"1min":1.1850000000000001,"5min":0.63300000000000001,"15min":0.60899999999999999},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.419,"15min":1.5169999999999999},"min":{"1min":1.1359999999999999,"5min":0.63500000000000001,"15min":0.61799999999999999},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":3.4849999999999999},"last":1.25}]}]},{"osd":3,"up_from":13,"seq":55834574989,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37140,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334700,"statfs":{"total":96636764160,"available":96598732800,"internally_reserved":0,"allocated":1843200,"data_stored":806818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.88500000000000001,"15min":0.86299999999999999},"min":{"1min":0.433,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.4059999999999999,"5min":2.1000000000000001,"15min":10.557},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.95599999999999996,"15min":0.91800000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4490000000000001,"5min":1.6890000000000001,"15min":3.8260000000000001},"last":0.749}]},{"osd":1,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":1.02,"5min":2.101,"15min":2.101},"last":1.274},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.996,"15min":1.0009999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.4450000000000001,"5min":2.1909999999999998,"15min":13.090999999999999},"last":1.0549999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.94199999999999995,"15min":0.93999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4750000000000001,"5min":2.3700000000000001,"15min":3.9489999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.0720000000000001,"15min":1.0820000000000001},"min":{"1min":0.52900000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.615,"5min":1.9690000000000001,"15min":13.460000000000001},"last":1.1739999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2809999999999999,"15min":1.2749999999999999},"min":{"1min":0.83099999999999996,"5min":0.71199999999999997,"15min":0.67800000000000005},"max":{"1min":1.639,"5min":2.1840000000000002,"15min":4.367},"last":1.786},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2210000000000001,"15min":1.2629999999999999},"min":{"1min":0.72399999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6120000000000001,"5min":3.0459999999999998,"15min":3.3380000000000001},"last":1.696}]},{"osd":5,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3260000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.9410000000000001,"5min":2.2869999999999999,"15min":13.231},"last":1.335},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2649999999999999,"15min":1.2869999999999999},"min":{"1min":0.74199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.992,"5min":1.992,"15min":5.1180000000000003},"last":1.454}]},{"osd":6,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3720000000000001,"15min":1.3879999999999999},"min":{"1min":0.69499999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0449999999999999,"5min":3.1120000000000001,"15min":13.359999999999999},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2869999999999999,"15min":1.298},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6140000000000001,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.4119999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:16:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.343,"15min":1.375},"min":{"1min":0.80200000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.1030000000000002,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.643},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.351,"15min":1.3819999999999999},"min":{"1min":0.83299999999999996,"5min":0.76400000000000001,"15min":0.68799999999999994},"max":{"1min":1.7050000000000001,"5min":3.331,"15min":4.032},"last":1.855}]}]},{"osd":2,"up_from":13,"seq":55834574987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1855488,"data_stored":808410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.81799999999999995,"15min":0.86499999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.32800000000000001},"max":{"1min":1.4299999999999999,"5min":1.5760000000000001,"15min":1.9610000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.95299999999999996,"15min":0.96999999999999997},"min":{"1min":0.49299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.6080000000000001,"5min":1.782,"15min":2.5779999999999998},"last":1.244}]},{"osd":1,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":1.0409999999999999},"min":{"1min":0.56899999999999995,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.518,"5min":1.7809999999999999,"15min":1.9890000000000001},"last":1.4850000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.98599999999999999,"15min":1.0580000000000001},"min":{"1min":0.54900000000000004,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.413,"5min":1.637,"15min":2.6240000000000001},"last":1.117}]},{"osd":3,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86699999999999999,"15min":0.91300000000000003},"min":{"1min":0.55200000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.3939999999999999,"5min":1.552,"15min":2.2090000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0720000000000001,"15min":1.1719999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.927,"5min":1.927,"15min":2.9950000000000001},"last":1.3660000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.365},"min":{"1min":0.98799999999999999,"5min":0.72399999999999998,"15min":0.70599999999999996},"max":{"1min":1.968,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.274,"15min":1.345},"min":{"1min":0.85499999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8839999999999999,"5min":1.921,"15min":3.3279999999999998},"last":1.863}]},{"osd":5,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.292,"15min":1.3740000000000001},"min":{"1min":0.79100000000000004,"5min":0.75600000000000001,"15min":0.73399999999999999},"max":{"1min":1.6830000000000001,"5min":2.3149999999999999,"15min":2.508},"last":1.694},{"interface":"front","average":{"1min":1.276,"5min":1.264,"15min":1.345},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.74,"5min":2.0070000000000001,"15min":2.7839999999999998},"last":2.0910000000000002}]},{"osd":6,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2170000000000001,"15min":1.284},"min":{"1min":0.81999999999999995,"5min":0.72099999999999997,"15min":0.71199999999999997},"max":{"1min":1.7490000000000001,"5min":2.1989999999999998,"15min":3.2210000000000001},"last":2.2879999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.3660000000000001,"15min":1.4450000000000001},"min":{"1min":0.79300000000000004,"5min":0.78300000000000003,"15min":0.73499999999999999},"max":{"1min":1.8140000000000001,"5min":2.4060000000000001,"15min":2.6629999999999998},"last":1.75}]},{"osd":7,"last update":"Fri Nov 24 08:15:07 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.359,"15min":1.4390000000000001},"min":{"1min":0.83899999999999997,"5min":0.77500000000000002,"15min":0.71299999999999997},"max":{"1min":1.8420000000000001,"5min":2.1070000000000002,"15min":2.859},"last":2.3999999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.2989999999999999,"15min":1.4019999999999999},"min":{"1min":0.93000000000000005,"5min":0.81399999999999995,"15min":0.746},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":2.0179999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37128,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334712,"statfs":{"total":96636764160,"available":96598745088,"internally_reserved":0,"allocated":1830912,"data_stored":801010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.79300000000000004,"15min":0.78500000000000003},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.006,"5min":1.5900000000000001,"15min":1.7749999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.877,"15min":0.86199999999999999},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.375,"5min":1.6970000000000001,"15min":2.1619999999999999},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80100000000000005,"15min":0.76000000000000001},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":1.4390000000000001,"5min":1.679,"15min":1.861},"last":0.374},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92900000000000005,"15min":0.90400000000000003},"min":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.403,"5min":1.925,"15min":1.925},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":0.97499999999999998},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.5600000000000001,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.92800000000000005,"15min":0.90400000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5409999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.276,"15min":1.2569999999999999},"min":{"1min":0.85199999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8380000000000001,"5min":2.028,"15min":2.7959999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.252},"min":{"1min":0.90300000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7110000000000001,"5min":2.141,"15min":3.0409999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2310000000000001,"15min":1.23},"min":{"1min":0.77600000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.9610000000000001,"15min":3.7639999999999998},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1830000000000001,"15min":1.1910000000000001},"min":{"1min":0.66900000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.821,"5min":2.246,"15min":3.286},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2470000000000001,"15min":1.256},"min":{"1min":0.69299999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.788,"5min":2.5270000000000001,"15min":4.2750000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.198,"5min":1.2330000000000001,"15min":1.2350000000000001},"min":{"1min":0.72099999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.627,"5min":1.9179999999999999,"15min":3.7040000000000002},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:16:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3009999999999999,"15min":1.3049999999999999},"min":{"1min":0.80200000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7150000000000001,"5min":2.3959999999999999,"15min":3.1760000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.3149999999999999,"15min":1.3029999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.7270000000000001,"5min":2.3620000000000001,"15min":3.653},"last":0.624}]}]},{"osd":0,"up_from":13,"seq":55834574989,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2449408,"data_stored":1398742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72599999999999998,"15min":0.80100000000000005},"min":{"1min":0.46300000000000002,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.81100000000000005,"15min":0.91600000000000004},"min":{"1min":0.42099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.425,"5min":1.4610000000000001,"15min":2.3620000000000001},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.008,"15min":1.036},"min":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.47699999999999998},"max":{"1min":1.405,"5min":1.833,"15min":1.891},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":0.91400000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.99099999999999999,"15min":1.0509999999999999},"min":{"1min":0.50900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.679,"5min":1.724,"15min":2.0670000000000002},"last":1.2},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91300000000000003,"15min":0.97599999999999998},"min":{"1min":0.48899999999999999,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":1.651}]},{"osd":4,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1639999999999999,"15min":1.306},"min":{"1min":0.73799999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4950000000000001,"5min":1.7729999999999999,"15min":3.7610000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.157,"15min":1.1870000000000001},"min":{"1min":0.83499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.3420000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.169,"15min":1.238},"min":{"1min":0.752,"5min":0.72799999999999998,"15min":0.69699999999999995},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1699999999999999,"15min":1.302},"min":{"1min":0.83299999999999996,"5min":0.72299999999999998,"15min":0.71099999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":2.7240000000000002},"last":1.268}]},{"osd":6,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2689999999999999,"15min":1.349},"min":{"1min":0.93600000000000005,"5min":0.71099999999999997,"15min":0.66800000000000004},"max":{"1min":2.052,"5min":2.052,"15min":2.4369999999999998},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.262,"5min":1.2669999999999999,"15min":1.3560000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.6899999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:15:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.339,"15min":1.4330000000000001},"min":{"1min":0.997,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":1.6120000000000001,"5min":2.3319999999999999,"15min":2.6989999999999998},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.3140000000000001,"15min":1.411},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.875,"5min":2.0710000000000002,"15min":2.3929999999999998},"last":1.885}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:16:11.446 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:16:11.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:16:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.708+0000 7f3309301640 1 Processor -- start 2023-11-24T08:16:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.708+0000 7f3309301640 1 -- start start 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3309301640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33040c9650 0x7f33040c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3309301640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3304166510 con 0x7f33040ccbd0 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3309301640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33041655f0 con 0x7f33041660b0 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3309301640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33040cc380 con 0x7f33040c9650 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33040c9650 0x7f33040c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33040c9650 0x7f33040c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35480/0 (socket says 172.21.15.107:35480) 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 -- 172.21.15.107:0/147985069 learned_addr learned my addr 172.21.15.107:0/147985069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:11.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3302ffd640 1 --1- 172.21.15.107:0/147985069 >> v1:172.21.15.107:6789/0 conn(0x7f33040ccbd0 0x7f33040ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52064/0 (socket says 172.21.15.107:52064) 2023-11-24T08:16:11.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 -- 172.21.15.107:0/147985069 >> v1:172.21.15.120:6789/0 conn(0x7f33041660b0 legacy=0x7f3304162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:11.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 -- 172.21.15.107:0/147985069 >> v1:172.21.15.107:6789/0 conn(0x7f33040ccbd0 legacy=0x7f33040ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f3301ffb640 1 -- 172.21.15.107:0/147985069 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772627736 0 0) 0x7f32f4001110 con 0x7f33040ccbd0 2023-11-24T08:16:11.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.711+0000 7f33037fe640 1 -- 172.21.15.107:0/147985069 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3304162b30 con 0x7f33040c9650 2023-11-24T08:16:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f33037fe640 1 --2- 172.21.15.107:0/147985069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33040c9650 0x7f33040c9ae0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009e50 tx=0x7f32fc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3983214cbb705b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f3301ffb640 1 -- 172.21.15.107:0/147985069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc02cb20 con 0x7f33040c9650 2023-11-24T08:16:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f3301ffb640 1 -- 172.21.15.107:0/147985069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc02cc80 con 0x7f33040c9650 2023-11-24T08:16:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f3301ffb640 1 -- 172.21.15.107:0/147985069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0343f0 con 0x7f33040c9650 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f3309301640 1 -- 172.21.15.107:0/147985069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33040c9650 msgr2=0x7f33040c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.713+0000 7f3309301640 1 --2- 172.21.15.107:0/147985069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33040c9650 0x7f33040c9ae0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009e50 tx=0x7f32fc02bf80 comp rx=0 tx=0).stop 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.714+0000 7f3309301640 1 -- 172.21.15.107:0/147985069 shutdown_connections 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.714+0000 7f3309301640 1 --2- 172.21.15.107:0/147985069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33040c9650 0x7f33040c9ae0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.714+0000 7f3309301640 1 -- 172.21.15.107:0/147985069 >> 172.21.15.107:0/147985069 conn(0x7f33040b0c50 msgr2=0x7f33040b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.714+0000 7f3309301640 1 -- 172.21.15.107:0/147985069 shutdown_connections 2023-11-24T08:16:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.714+0000 7f3309301640 1 -- 172.21.15.107:0/147985069 wait complete. 2023-11-24T08:16:11.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.715+0000 7f3309301640 1 Processor -- start 2023-11-24T08:16:11.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.715+0000 7f3309301640 1 -- start start 2023-11-24T08:16:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33040c9650 0x7f33040c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33040ccbd0 0x7f3304163130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304161980 con 0x7f33041660b0 2023-11-24T08:16:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041613c0 con 0x7f33040ccbd0 2023-11-24T08:16:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.716+0000 7f3309301640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33041616a0 con 0x7f33040c9650 2023-11-24T08:16:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35490/0 (socket says 172.21.15.107:35490) 2023-11-24T08:16:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 -- 172.21.15.107:0/2021536584 learned_addr learned my addr 172.21.15.107:0/2021536584 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f3302ffd640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33040c9650 0x7f33040c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33027fc640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33040ccbd0 0x7f3304163130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 -- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33040c9650 msgr2=0x7f33040c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33040c9650 0x7f33040c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 -- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33040ccbd0 msgr2=0x7f3304163130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33040ccbd0 0x7f3304163130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 -- 172.21.15.107:0/2021536584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc009b30 con 0x7f33041660b0 2023-11-24T08:16:11.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.717+0000 7f33037fe640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f32fc036040 tx=0x7f32fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.718+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc009510 con 0x7f33041660b0 2023-11-24T08:16:11.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.718+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc007e50 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.718+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc03e690 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.718+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3304070820 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.718+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3304157090 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.719+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330404e8b0 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.720+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f32fc03e7f0 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.721+0000 7f32fb7fe640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 0x7f32e406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.721+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32fc0aaf90 con 0x7f33041660b0 2023-11-24T08:16:11.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.722+0000 7f3302ffd640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 0x7f32e406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:11.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.722+0000 7f3302ffd640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 0x7f32e406e7d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000f60 tx=0x7f32f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:11.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.726+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32fc077af0 con 0x7f33041660b0 2023-11-24T08:16:11.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.984+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f330412c1b0 con 0x7f33041660b0 2023-11-24T08:16:11.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.989+0000 7f32fb7fe640 1 -- 172.21.15.107:0/2021536584 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f32fc0430e0 con 0x7f33041660b0 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:16:11.992 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.992+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 msgr2=0x7f32e406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.992+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 0x7f32e406e7d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000f60 tx=0x7f32f4009040 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.992+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 msgr2=0x7f3304071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.992+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f32fc036040 tx=0x7f32fc004a40 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 shutdown_connections 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32e406c310 0x7f32e406e7d0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33041660b0 0x7f3304071000 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33040ccbd0 0x7f3304163130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 --2- 172.21.15.107:0/2021536584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33040c9650 0x7f33040c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 >> 172.21.15.107:0/2021536584 conn(0x7f33040b0c50 msgr2=0x7f33040afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 shutdown_connections 2023-11-24T08:16:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:11.993+0000 7f3309301640 1 -- 172.21.15.107:0/2021536584 wait complete. 2023-11-24T08:16:12.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:16:12.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.219+0000 7f2320fa9640 1 Processor -- start 2023-11-24T08:16:12.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.219+0000 7f2320fa9640 1 -- start start 2023-11-24T08:16:12.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.220+0000 7f2320fa9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f231c0cf450 0x7f231c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.220+0000 7f2320fa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0cece0 con 0x7f231c0cdfb0 2023-11-24T08:16:12.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f2320fa9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c06d040 con 0x7f231c15c7f0 2023-11-24T08:16:12.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f2320fa9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231c0cfe40 con 0x7f231c0cf450 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f231ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f231c0cdfb0 0x7f231c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52072/0 (socket says 172.21.15.107:52072) 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f231ad9d640 1 -- 172.21.15.107:0/1348143163 learned_addr learned my addr 172.21.15.107:0/1348143163 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f2313fff640 1 --2- 172.21.15.107:0/1348143163 >> v2:172.21.15.120:3300/0 conn(0x7f231c0cf450 0x7f231c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.221+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702826031 0 0) 0x7f231c0cece0 con 0x7f231c0cdfb0 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003520 con 0x7f231c0cdfb0 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 927088557 0 0) 0x7f2308003520 con 0x7f231c0cdfb0 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 >> v2:172.21.15.120:3300/0 conn(0x7f231c0cf450 msgr2=0x7f231c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:12.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 --2- 172.21.15.107:0/1348143163 >> v2:172.21.15.120:3300/0 conn(0x7f231c0cf450 0x7f231c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 >> v1:172.21.15.120:6789/0 conn(0x7f231c15c7f0 legacy=0x7f231c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c06d320 con 0x7f231c0cdfb0 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f23140019f0 con 0x7f231c0cdfb0 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 >> v1:172.21.15.107:6789/0 conn(0x7f231c0cdfb0 legacy=0x7f231c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 0x7f2308003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080047c0 0x7f2308004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2308005180 0x7f230800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0cfe40 con 0x7f23080039c0 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230800c080 con 0x7f23080047c0 2023-11-24T08:16:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230800c240 con 0x7f2308005180 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.222+0000 7f231a59c640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 0x7f2308003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f2313fff640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080047c0 0x7f2308004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231a59c640 1 -- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2308005180 msgr2=0x7f230800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231a59c640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2308005180 0x7f230800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231a59c640 1 -- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080047c0 msgr2=0x7f2308004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231a59c640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080047c0 0x7f2308004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231a59c640 1 -- 172.21.15.107:0/1348143163 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308003520 con 0x7f23080039c0 2023-11-24T08:16:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.223+0000 7f231ad9d640 1 -- 172.21.15.107:0/1348143163 reap_dead start 2023-11-24T08:16:12.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.224+0000 7f231a59c640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 0x7f2308003de0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f230c001750 tx=0x7f23080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e992cc3ba62e77 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.224+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c0052d0 con 0x7f23080039c0 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.224+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c02e050 con 0x7f23080039c0 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.224+0000 7f2319d9b640 1 -- 172.21.15.107:0/1348143163 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c003830 con 0x7f23080039c0 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 -- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 msgr2=0x7f2308003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 0x7f2308003de0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f230c001750 tx=0x7f23080057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 -- 172.21.15.107:0/1348143163 shutdown_connections 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 --2- 172.21.15.107:0/1348143163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23080039c0 0x7f2308003de0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 -- 172.21.15.107:0/1348143163 >> 172.21.15.107:0/1348143163 conn(0x7f231c06a920 msgr2=0x7f231c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 -- 172.21.15.107:0/1348143163 shutdown_connections 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.225+0000 7f2320fa9640 1 -- 172.21.15.107:0/1348143163 wait complete. 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.227+0000 7f2320fa9640 1 Processor -- start 2023-11-24T08:16:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.227+0000 7f2320fa9640 1 -- start start 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080039c0 0x7f231c156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 0x7f231c157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c157460 con 0x7f231c154d90 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0afc80 con 0x7f23080039c0 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2320fa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c155e80 con 0x7f231c0af800 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35518/0 (socket says 172.21.15.107:35518) 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 -- 172.21.15.107:0/3767564521 learned_addr learned my addr 172.21.15.107:0/3767564521 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f231a59c640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080039c0 0x7f231c156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f231ad9d640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 0x7f231c157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 -- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 msgr2=0x7f231c157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 0x7f231c157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 -- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080039c0 msgr2=0x7f231c156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.228+0000 7f2313fff640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080039c0 0x7f231c156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2313fff640 1 -- 172.21.15.107:0/3767564521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c156160 con 0x7f231c154d90 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f231ad9d640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 0x7f231c157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2313fff640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f23040114e0 tx=0x7f23040119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304015870 con 0x7f231c154d90 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23040159d0 con 0x7f231c154d90 2023-11-24T08:16:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304015d40 con 0x7f231c154d90 2023-11-24T08:16:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c156440 con 0x7f231c154d90 2023-11-24T08:16:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.229+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c156a00 con 0x7f231c154d90 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.230+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c067470 con 0x7f231c154d90 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.231+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f2304013400 con 0x7f231c154d90 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.232+0000 7f2312ffd640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 0x7f22fc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.232+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23040885f0 con 0x7f231c154d90 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.232+0000 7f231a59c640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 0x7f22fc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.233+0000 7f231a59c640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 0x7f22fc06e7d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f230c033000 tx=0x7f230c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:12.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.236+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23040550f0 con 0x7f231c154d90 2023-11-24T08:16:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.454+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f231c15b6e0 con 0x7f231c154d90 2023-11-24T08:16:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.455+0000 7f2312ffd640 1 -- 172.21.15.107:0/3767564521 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f2304022090 con 0x7f231c154d90 2023-11-24T08:16:12.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:16:12.457 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:16:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 87 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 msgr2=0x7f22fc06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 0x7f22fc06e7d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f230c033000 tx=0x7f230c009030 comp rx=0 tx=0).stop 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 msgr2=0x7f231c0af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f23040114e0 tx=0x7f23040119a0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 shutdown_connections 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f22fc06c310 0x7f22fc06e7d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f231c0af800 0x7f231c157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f231c154d90 0x7f231c0af2c0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 --2- 172.21.15.107:0/3767564521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23080039c0 0x7f231c156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:12.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.458+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 >> 172.21.15.107:0/3767564521 conn(0x7f231c06a920 msgr2=0x7f231c072280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:12.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.459+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 shutdown_connections 2023-11-24T08:16:12.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:12.459+0000 7f2320fa9640 1 -- 172.21.15.107:0/3767564521 wait complete. 2023-11-24T08:16:12.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:16:12.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:16:12.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:16:05.904591+0000', 'modified': '2023-11-24T08:16:08.999664+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9300}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9300': {'gid': 9300, 'name': 'c', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6836/3938030686', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3938030686}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 3938030686}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:16:12.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:16:12.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:16:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.102+0000 7f14828ee640 1 Processor -- start 2023-11-24T08:16:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.102+0000 7f14828ee640 1 -- start start 2023-11-24T08:16:13.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.104+0000 7f14828ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f147c0c49f0 0x7f147c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.104+0000 7f14828ee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c15ec90 con 0x7f147c06de10 2023-11-24T08:16:13.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.104+0000 7f14828ee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c0cd530 con 0x7f147c072160 2023-11-24T08:16:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.104+0000 7f14828ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147c0c4ed0 con 0x7f147c0c49f0 2023-11-24T08:16:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.104+0000 7f1480e8b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f147c0c49f0 0x7f147c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f1480e8b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f147c0c49f0 0x7f147c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35534/0 (socket says 172.21.15.107:35534) 2023-11-24T08:16:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f1480e8b640 1 -- 172.21.15.107:0/1474484017 learned_addr learned my addr 172.21.15.107:0/1474484017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f1480e8b640 1 -- 172.21.15.107:0/1474484017 >> v1:172.21.15.120:6789/0 conn(0x7f147c072160 legacy=0x7f147c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f1480e8b640 1 -- 172.21.15.107:0/1474484017 >> v1:172.21.15.107:6789/0 conn(0x7f147c06de10 legacy=0x7f147c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f1480e8b640 1 -- 172.21.15.107:0/1474484017 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0c51b0 con 0x7f147c0c49f0 2023-11-24T08:16:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.105+0000 7f147affd640 1 -- 172.21.15.107:0/1474484017 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176136166 0 0) 0x7f1464001110 con 0x7f147c06de10 2023-11-24T08:16:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.106+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1474484017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c0c49f0 0x7f147c15e750 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f147400bb10 tx=0x7f147402e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21727c83c0cfee1e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.106+0000 7f147affd640 1 -- 172.21.15.107:0/1474484017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474036440 con 0x7f147c0c49f0 2023-11-24T08:16:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.106+0000 7f147affd640 1 -- 172.21.15.107:0/1474484017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14740365a0 con 0x7f147c0c49f0 2023-11-24T08:16:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.106+0000 7f147affd640 1 -- 172.21.15.107:0/1474484017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474036930 con 0x7f147c0c49f0 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 -- 172.21.15.107:0/1474484017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c0c49f0 msgr2=0x7f147c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 --2- 172.21.15.107:0/1474484017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c0c49f0 0x7f147c15e750 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f147400bb10 tx=0x7f147402e1f0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 -- 172.21.15.107:0/1474484017 shutdown_connections 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 --2- 172.21.15.107:0/1474484017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c0c49f0 0x7f147c15e750 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 -- 172.21.15.107:0/1474484017 >> 172.21.15.107:0/1474484017 conn(0x7f147c0b0c50 msgr2=0x7f147c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 -- 172.21.15.107:0/1474484017 shutdown_connections 2023-11-24T08:16:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.107+0000 7f14828ee640 1 -- 172.21.15.107:0/1474484017 wait complete. 2023-11-24T08:16:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.109+0000 7f14828ee640 1 Processor -- start 2023-11-24T08:16:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.109+0000 7f14828ee640 1 -- start start 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f147c06de10 0x7f147c159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f147c0c49f0 0x7f147c09c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b2ee0 con 0x7f147c072160 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b3d00 con 0x7f147c0c49f0 2023-11-24T08:16:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f14828ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b30f0 con 0x7f147c06de10 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35542/0 (socket says 172.21.15.107:35542) 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f147b7fe640 1 -- 172.21.15.107:0/4109945830 learned_addr learned my addr 172.21.15.107:0/4109945830 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.110+0000 7f1480e8b640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f147c0c49f0 0x7f147c09c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147bfff640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f147c06de10 0x7f147c159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 -- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f147c06de10 msgr2=0x7f147c159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f147c06de10 0x7f147c159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 -- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f147c0c49f0 msgr2=0x7f147c09c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f147c0c49f0 0x7f147c09c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 -- 172.21.15.107:0/4109945830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147400b770 con 0x7f147c072160 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f147b7fe640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f146400c780 tx=0x7f146400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f1478ff9640 1 -- 172.21.15.107:0/4109945830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464005270 con 0x7f147c072160 2023-11-24T08:16:13.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f1478ff9640 1 -- 172.21.15.107:0/4109945830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640053f0 con 0x7f147c072160 2023-11-24T08:16:13.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c0b33d0 con 0x7f147c072160 2023-11-24T08:16:13.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f1478ff9640 1 -- 172.21.15.107:0/4109945830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400a880 con 0x7f147c072160 2023-11-24T08:16:13.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.111+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0b3990 con 0x7f147c072160 2023-11-24T08:16:13.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.114+0000 7f1478ff9640 1 -- 172.21.15.107:0/4109945830 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f146400a9e0 con 0x7f147c072160 2023-11-24T08:16:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f1478ff9640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 0x7f145c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f1478ff9640 1 -- 172.21.15.107:0/4109945830 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14640881b0 con 0x7f147c072160 2023-11-24T08:16:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f147bfff640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 0x7f145c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f1485563740 1 Processor -- start 2023-11-24T08:16:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f1485563740 1 -- start start 2023-11-24T08:16:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.115+0000 7f147bfff640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 0x7f145c06e7d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f146c007fd0 tx=0x7f146c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe139eba0 0x557fe13a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe1302880 con 0x557fe13a4fe0 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe1311ae0 con 0x557fe13ceb80 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.116+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fe130f1d0 con 0x557fe139eba0 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35554/0 (socket says 172.21.15.107:35554) 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 -- 172.21.15.107:0/1960345672 learned_addr learned my addr 172.21.15.107:0/1960345672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 -- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe139eba0 msgr2=0x557fe13a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe139eba0 0x557fe13a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 -- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 msgr2=0x557fe13cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f147bfff640 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 -- 172.21.15.107:0/1960345672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fe12d2860 con 0x557fe13a4fe0 2023-11-24T08:16:13.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.117+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba60 tx=0x7f1474032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe7f2434a07f7c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1471ffb640 1 -- 172.21.15.107:0/1960345672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474036a50 con 0x557fe13a4fe0 2023-11-24T08:16:13.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1471ffb640 1 -- 172.21.15.107:0/1960345672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474036bb0 con 0x557fe13a4fe0 2023-11-24T08:16:13.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1471ffb640 1 -- 172.21.15.107:0/1960345672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474041420 con 0x557fe13a4fe0 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 -- 172.21.15.107:0/1960345672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 msgr2=0x557fe13e7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba60 tx=0x7f1474032000 comp rx=0 tx=0).stop 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 -- 172.21.15.107:0/1960345672 shutdown_connections 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13a4fe0 0x557fe13e7560 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe139eba0 0x557fe13a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 --2- 172.21.15.107:0/1960345672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 -- 172.21.15.107:0/1960345672 >> 172.21.15.107:0/1960345672 conn(0x557fe1365c70 msgr2=0x557fe13e9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 -- 172.21.15.107:0/1960345672 shutdown_connections 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.118+0000 7f1485563740 1 -- 172.21.15.107:0/1960345672 wait complete. 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.119+0000 7f1485563740 1 -- start start 2023-11-24T08:16:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.119+0000 7f1485563740 1 Processor -- start 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13b6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe130aac0 con 0x557fe139eba0 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe12a72f0 con 0x557fe13ceb80 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fe12c4c70 con 0x557fe13fe1b0 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.120+0000 7f1485563740 10 client.0 _ll_register_callbacks cb 0x557fe12c93b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35560/0 (socket says 172.21.15.107:35560) 2023-11-24T08:16:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 -- 172.21.15.107:0/1275580133 learned_addr learned my addr 172.21.15.107:0/1275580133 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 -- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 msgr2=0x557fe13b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 -- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 msgr2=0x557fe13fdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 -- 172.21.15.107:0/1275580133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0c51b0 con 0x557fe139eba0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f147bfff640 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f146c00db10 tx=0x7f146c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9f3b66e16e9870 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f145affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c022660 con 0x557fe139eba0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f146c01f9b0 con 0x557fe139eba0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.121+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557fe1303aa0 con 0x557fe139eba0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146c005860 con 0x557fe139eba0 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 10 client.9447 fetch_fsmap learned FSMap version 87 2023-11-24T08:16:13.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x557fe1309140 con 0x557fe139eba0 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557fe1303aa0 con 0x557fe139eba0 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f146c01fb70 con 0x557fe139eba0 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 10 client.9447 fetch_fsmap finished waiting for FSMap version 87 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 10 client.9447 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 20 client.9447 populate_metadata read hostname 'smithi107' 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13c6300 0x557fe13c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 4 client.9447 mds_command: new command op to 9300 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 --> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557fe13037c0 con 0x557fe13c6300 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.122+0000 7f147b7fe640 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13c6300 0x557fe13c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.123+0000 7f147b7fe640 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13c6300 0x557fe13c66e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccd41b46bf485878 server_cookie=683633375be1a609 in_seq=0 out_seq=0 2023-11-24T08:16:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.123+0000 7f145affd640 10 client.9447 ms_handle_connect on v2:172.21.15.107:6834/3938030686 2023-11-24T08:16:13.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f145affd640 1 -- 172.21.15.107:0/1275580133 <== mds.0 v2:172.21.15.107:6834/3938030686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557fe13037c0 con 0x557fe13c6300 2023-11-24T08:16:13.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f145affd640 10 client.9447 handle_command_reply: tid=0 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 client.9447 shutdown 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 msgr2=0x557fe13b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f146c00db10 tx=0x7f146c0014a0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 shutdown_connections 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13c6300 0x557fe13c66e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe13ceb80 0x557fe13fdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 --2- 172.21.15.107:0/1275580133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe139eba0 0x557fe13b81a0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f145affd640 0 client.9447 ms_handle_reset on v2:172.21.15.107:6834/3938030686 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 >> 172.21.15.107:0/1275580133 conn(0x557fe1365c70 msgr2=0x557fe13cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 shutdown_connections 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 1 -- 172.21.15.107:0/1275580133 wait complete. 2023-11-24T08:16:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.124+0000 7f1485563740 20 client.9447 trim_cache size 0 max 16384 2023-11-24T08:16:13.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.141+0000 7f1485563740 1 Processor -- start 2023-11-24T08:16:13.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.141+0000 7f1485563740 1 -- start start 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13ceb80 0x557fe13a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe106df20 con 0x557fe13fe1b0 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe130ca40 con 0x557fe139eba0 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fe13b80a0 con 0x557fe13ceb80 2023-11-24T08:16:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1480e8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1480e8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35568/0 (socket says 172.21.15.107:35568) 2023-11-24T08:16:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.142+0000 7f1480e8b640 1 -- 172.21.15.107:0/1771663530 learned_addr learned my addr 172.21.15.107:0/1771663530 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 -- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13ceb80 msgr2=0x557fe13a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13ceb80 0x557fe13a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 -- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 msgr2=0x557fe13fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 -- 172.21.15.107:0/1771663530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fe12a6b80 con 0x557fe13fe1b0 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f1480e8b640 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba60 tx=0x7f1474030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabfe4d7ad9681c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f14727fc640 1 -- 172.21.15.107:0/1771663530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474036790 con 0x557fe13fe1b0 2023-11-24T08:16:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f14727fc640 1 -- 172.21.15.107:0/1771663530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14740368f0 con 0x557fe13fe1b0 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.143+0000 7f14727fc640 1 -- 172.21.15.107:0/1771663530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474004260 con 0x557fe13fe1b0 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 -- 172.21.15.107:0/1771663530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 msgr2=0x557fe13c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba60 tx=0x7f1474030000 comp rx=0 tx=0).stop 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 -- 172.21.15.107:0/1771663530 shutdown_connections 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13fe1b0 0x557fe13c66e0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13ceb80 0x557fe13a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 --2- 172.21.15.107:0/1771663530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 -- 172.21.15.107:0/1771663530 >> 172.21.15.107:0/1771663530 conn(0x557fe1365c70 msgr2=0x557fe139a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 -- 172.21.15.107:0/1771663530 shutdown_connections 2023-11-24T08:16:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.144+0000 7f1485563740 1 -- 172.21.15.107:0/1771663530 wait complete. 2023-11-24T08:16:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.145+0000 7f1485563740 1 -- start start 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.145+0000 7f1485563740 1 Processor -- start 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13d5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe13d6bc0 con 0x557fe13ceb80 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fe13f4bb0 con 0x557fe139eba0 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fe13f58e0 con 0x557fe13fe1b0 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f1485563740 10 client.0 _ll_register_callbacks cb 0x7f146c001af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35582/0 (socket says 172.21.15.107:35582) 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 -- 172.21.15.107:0/3926384816 learned_addr learned my addr 172.21.15.107:0/3926384816 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 -- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 msgr2=0x557fe13d5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13d5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 -- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 msgr2=0x557fe13c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.146+0000 7f147b7fe640 1 -- 172.21.15.107:0/3926384816 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557fe12d2860 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f147b7fe640 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f1464088120 tx=0x7f1464058170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7791fbfbc4268dbd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146405c650 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1464060890 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1474036bb0 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1464054c10 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 10 client.9453 fetch_fsmap learned FSMap version 87 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x557fe12cc170 con 0x557fe13ceb80 2023-11-24T08:16:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474036bb0 con 0x557fe13ceb80 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1464019020 con 0x557fe13ceb80 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 10 client.9453 fetch_fsmap finished waiting for FSMap version 87 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 10 client.9453 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 20 client.9453 populate_metadata read hostname 'smithi107' 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13d5ff0 0x557fe13e9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 4 client.9453 mds_command: new command op to 9300 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.147+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 --> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557fe100b6c0 con 0x557fe13d5ff0 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f147bfff640 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13d5ff0 0x557fe13e9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f147bfff640 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13d5ff0 0x557fe13e9760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962094d211bd231 server_cookie=bd34be5a2eaa24ee in_seq=0 out_seq=0 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f145affd640 10 client.9453 ms_handle_connect on v2:172.21.15.107:6834/3938030686 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f145affd640 1 -- 172.21.15.107:0/3926384816 <== mds.0 v2:172.21.15.107:6834/3938030686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x557fe100b6c0 con 0x557fe13d5ff0 2023-11-24T08:16:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f145affd640 10 client.9453 handle_command_reply: tid=0 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 client.9453 shutdown 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 msgr2=0x557fe13c6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f1464088120 tx=0x7f1464058170 comp rx=0 tx=0).stop 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 shutdown_connections 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:6834/3938030686,v1:172.21.15.107:6836/3938030686] conn(0x557fe13d5ff0 0x557fe13e9760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557fe13fe1b0 0x557fe13d5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557fe13ceb80 0x557fe13c6c20 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 --2- 172.21.15.107:0/3926384816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe139eba0 0x557fe13c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f145affd640 0 client.9453 ms_handle_reset on v2:172.21.15.107:6834/3938030686 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.148+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 >> 172.21.15.107:0/3926384816 conn(0x557fe1365c70 msgr2=0x557fe13a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 shutdown_connections 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f1485563740 1 -- 172.21.15.107:0/3926384816 wait complete. 2023-11-24T08:16:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f1485563740 20 client.9453 trim_cache size 0 max 16384 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 9300, 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.660361859, 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 87, 2023-11-24T08:16:13.155 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 147, 2023-11-24T08:16:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 147, 2023-11-24T08:16:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.122646212999999 2023-11-24T08:16:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 msgr2=0x7f145c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 0x7f145c06e7d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f146c007fd0 tx=0x7f146c009040 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 msgr2=0x7f147c15a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f146400c780 tx=0x7f146400cc40 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 shutdown_connections 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f145c06c310 0x7f145c06e7d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f147c0c49f0 0x7f147c09c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147c072160 0x7f147c15a470 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 --2- 172.21.15.107:0/4109945830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f147c06de10 0x7f147c159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.149+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 >> 172.21.15.107:0/4109945830 conn(0x7f147c0b0c50 msgr2=0x7f147c0ad7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.150+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 shutdown_connections 2023-11-24T08:16:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.150+0000 7f14828ee640 1 -- 172.21.15.107:0/4109945830 wait complete. 2023-11-24T08:16:13.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:16:13.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.358+0000 7f0b1d42d640 1 Processor -- start 2023-11-24T08:16:13.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.358+0000 7f0b1d42d640 1 -- start start 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b1815c7f0 0x7f0b1806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b180cf450 0x7f0b180cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b180cdfb0 0x7f0b180ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b18156010 con 0x7f0b180cf450 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1806ce30 con 0x7f0b180cdfb0 2023-11-24T08:16:13.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.359+0000 7f0b1d42d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180cfe10 con 0x7f0b1815c7f0 2023-11-24T08:16:13.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b180cf450 0x7f0b180cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b180cf450 0x7f0b180cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35596/0 (socket says 172.21.15.107:35596) 2023-11-24T08:16:13.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 -- 172.21.15.107:0/4291036734 learned_addr learned my addr 172.21.15.107:0/4291036734 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b16ffd640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b1815c7f0 0x7f0b1806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b177fe640 1 --2- 172.21.15.107:0/4291036734 >> v2:172.21.15.120:3300/0 conn(0x7f0b180cdfb0 0x7f0b180ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 -- 172.21.15.107:0/4291036734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b1815c7f0 msgr2=0x7f0b1806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b1815c7f0 0x7f0b1806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 -- 172.21.15.107:0/4291036734 >> v2:172.21.15.120:3300/0 conn(0x7f0b180cdfb0 msgr2=0x7f0b180ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 --2- 172.21.15.107:0/4291036734 >> v2:172.21.15.120:3300/0 conn(0x7f0b180cdfb0 0x7f0b180ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.360+0000 7f0b0e7fc640 1 -- 172.21.15.107:0/4291036734 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180cece0 con 0x7f0b180cf450 2023-11-24T08:16:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b0e7fc640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b180cf450 0x7f0b180cf8d0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f0b180aebc0 tx=0x7f0b0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbac226d556d4d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b167fc640 1 -- 172.21.15.107:0/4291036734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00031870 con 0x7f0b180cf450 2023-11-24T08:16:13.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b167fc640 1 -- 172.21.15.107:0/4291036734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b000319d0 con 0x7f0b180cf450 2023-11-24T08:16:13.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b167fc640 1 -- 172.21.15.107:0/4291036734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00031d60 con 0x7f0b180cf450 2023-11-24T08:16:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b1d42d640 1 -- 172.21.15.107:0/4291036734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b180cf450 msgr2=0x7f0b180cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.362+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b180cf450 0x7f0b180cf8d0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f0b180aebc0 tx=0x7f0b0002b910 comp rx=0 tx=0).stop 2023-11-24T08:16:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 -- 172.21.15.107:0/4291036734 shutdown_connections 2023-11-24T08:16:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/4291036734 >> v2:172.21.15.120:3300/0 conn(0x7f0b180cdfb0 0x7f0b180ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b180cf450 0x7f0b180cf8d0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/4291036734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b1815c7f0 0x7f0b1806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 -- 172.21.15.107:0/4291036734 >> 172.21.15.107:0/4291036734 conn(0x7f0b1806a920 msgr2=0x7f0b1806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 -- 172.21.15.107:0/4291036734 shutdown_connections 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.363+0000 7f0b1d42d640 1 -- 172.21.15.107:0/4291036734 wait complete. 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.365+0000 7f0b1d42d640 1 Processor -- start 2023-11-24T08:16:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.365+0000 7f0b1d42d640 1 -- start start 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 0x7f0b1809b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 0x7f0b1809cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1809d150 con 0x7f0b1815c7f0 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1809a040 con 0x7f0b180cdfb0 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b1d42d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1809bb70 con 0x7f0b180cf450 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35612/0 (socket says 172.21.15.107:35612) 2023-11-24T08:16:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b177fe640 1 -- 172.21.15.107:0/1455879157 learned_addr learned my addr 172.21.15.107:0/1455879157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b0e7fc640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 0x7f0b1809cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.366+0000 7f0b16ffd640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 0x7f0b1809b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 -- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 msgr2=0x7f0b1809cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 0x7f0b1809cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 -- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 msgr2=0x7f0b1809b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 0x7f0b1809b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 -- 172.21.15.107:0/1455879157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b000094d0 con 0x7f0b1815c7f0 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b16ffd640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 0x7f0b1809b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b0e7fc640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 0x7f0b1809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b177fe640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f0b10007d10 tx=0x7f0b1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10014420 con 0x7f0b1815c7f0 2023-11-24T08:16:13.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b100145a0 con 0x7f0b1815c7f0 2023-11-24T08:16:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10014950 con 0x7f0b1815c7f0 2023-11-24T08:16:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1809be50 con 0x7f0b1815c7f0 2023-11-24T08:16:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.367+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1809c410 con 0x7f0b1815c7f0 2023-11-24T08:16:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.368+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b18067470 con 0x7f0b1815c7f0 2023-11-24T08:16:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.369+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f0b10014ab0 con 0x7f0b1815c7f0 2023-11-24T08:16:13.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.370+0000 7f0b0ffff640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 0x7f0af806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:13.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.370+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b10088fc0 con 0x7f0b1815c7f0 2023-11-24T08:16:13.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.370+0000 7f0b16ffd640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 0x7f0af806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.371+0000 7f0b16ffd640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 0x7f0af806e7d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:13.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.375+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b100556e0 con 0x7f0b1815c7f0 2023-11-24T08:16:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.597+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0b1806ad00 con 0x7f0b1815c7f0 2023-11-24T08:16:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.598+0000 7f0b0ffff640 1 -- 172.21.15.107:0/1455879157 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0b10058d00 con 0x7f0b1815c7f0 2023-11-24T08:16:13.600 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:16:13.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 msgr2=0x7f0af806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 0x7f0af806e7d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08008040 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 msgr2=0x7f0b18099b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f0b10007d10 tx=0x7f0b1000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 shutdown_connections 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0af806c310 0x7f0af806e7d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1815c7f0 0x7f0b18099b00 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b180cf450 0x7f0b1809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 --2- 172.21.15.107:0/1455879157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b180cdfb0 0x7f0b1809b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 >> 172.21.15.107:0/1455879157 conn(0x7f0b1806a920 msgr2=0x7f0b180cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 shutdown_connections 2023-11-24T08:16:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:13.600+0000 7f0b1d42d640 1 -- 172.21.15.107:0/1455879157 wait complete. 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:16:13.619 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:16:13.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:13.620 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.675 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:13.676 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41697sec preferred_lft 41697sec 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:16:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.678 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:16:13.678 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:16:13.678 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:16:13.678 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:13.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:13.678 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:16:13.765 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:16:13.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:13.769 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:16:13.818 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:16:13.819 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:16:13.819 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:16:13.819 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:16:13.819 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:16:13.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:13.820 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:16:13.820 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:16:13.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:16:13.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:16:13.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:16:13.821 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:16:13.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:13.999 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:16:14.050 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:16:14.050 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:16:14.051 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:16:14.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:14.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:16:14.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:16:14.146 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:16:14.202 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:16:14.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:16:14.203 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:16:14.256 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:16:14.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:14.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:14.397 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:14.398 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:16:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:16:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:16:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:16:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:16:14.434 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:16:14.435 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:16:14.435 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:16:14.435 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:16:14.435 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:16:14.439 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:16:14.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:14.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:16:14.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:14.528 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:16:14.586 INFO:teuthology.orchestra.run.smithi107.stdout:client9462 2023-11-24T08:16:14.586 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:14.586 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:14.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:16:14.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.887+0000 7f6890705640 1 Processor -- start 2023-11-24T08:16:14.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.888+0000 7f6890705640 1 -- start start 2023-11-24T08:16:14.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.889+0000 7f6890705640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f688815c480 0x7f68880cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.889+0000 7f6890705640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880ce2b0 0x7f688806ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.889+0000 7f6890705640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68880cf7a0 con 0x7f68880cf340 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.889+0000 7f6890705640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688815c860 con 0x7f688815c480 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.889+0000 7f6890705640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6888141830 con 0x7f68880ce2b0 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.890+0000 7f688e4a1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f688815c480 0x7f68880cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.890+0000 7f688e4a1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f688815c480 0x7f68880cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42726/0 (socket says 172.21.15.107:42726) 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.890+0000 7f688e4a1640 1 -- 172.21.15.107:0/2540884069 learned_addr learned my addr 172.21.15.107:0/2540884069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.890+0000 7f688eca2640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880ce2b0 0x7f688806ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.891+0000 7f688e4a1640 1 -- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880ce2b0 msgr2=0x7f688806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.891+0000 7f688e4a1640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880ce2b0 0x7f688806ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.891+0000 7f688e4a1640 1 -- 172.21.15.107:0/2540884069 >> v1:172.21.15.120:6789/0 conn(0x7f68880cf340 legacy=0x7f68880cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.891+0000 7f688e4a1640 1 -- 172.21.15.107:0/2540884069 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888140c20 con 0x7f688815c480 2023-11-24T08:16:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.892+0000 7f688e4a1640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f688815c480 0x7f68880cee00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6884009870 tx=0x7f688402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33f23e6a79f742de server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f688d49f640 1 -- 172.21.15.107:0/2540884069 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884030840 con 0x7f688815c480 2023-11-24T08:16:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f688d49f640 1 -- 172.21.15.107:0/2540884069 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68840309a0 con 0x7f688815c480 2023-11-24T08:16:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f688d49f640 1 -- 172.21.15.107:0/2540884069 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884030d30 con 0x7f688815c480 2023-11-24T08:16:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 -- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f688815c480 msgr2=0x7f68880cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f688815c480 0x7f68880cee00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6884009870 tx=0x7f688402b980 comp rx=0 tx=0).stop 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 -- 172.21.15.107:0/2540884069 shutdown_connections 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880ce2b0 0x7f688806ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 --2- 172.21.15.107:0/2540884069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f688815c480 0x7f68880cee00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.893+0000 7f6890705640 1 -- 172.21.15.107:0/2540884069 >> 172.21.15.107:0/2540884069 conn(0x7f68880b2e40 msgr2=0x7f68880b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.894+0000 7f6890705640 1 -- 172.21.15.107:0/2540884069 shutdown_connections 2023-11-24T08:16:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.894+0000 7f6890705640 1 -- 172.21.15.107:0/2540884069 wait complete. 2023-11-24T08:16:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.895+0000 7f6890705640 1 Processor -- start 2023-11-24T08:16:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.895+0000 7f6890705640 1 -- start start 2023-11-24T08:16:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68880ce2b0 0x7f6888155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880cf340 0x7f68880caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880c99b0 con 0x7f688815c480 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880c9c90 con 0x7f68880ce2b0 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.896+0000 7f6890705640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68880c9f70 con 0x7f68880cf340 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35626/0 (socket says 172.21.15.107:35626) 2023-11-24T08:16:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 -- 172.21.15.107:0/607506386 learned_addr learned my addr 172.21.15.107:0/607506386 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688e4a1640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68880ce2b0 0x7f6888155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688dca0640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880cf340 0x7f68880caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 -- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880cf340 msgr2=0x7f68880caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880cf340 0x7f68880caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 -- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68880ce2b0 msgr2=0x7f6888155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68880ce2b0 0x7f6888155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.897+0000 7f688eca2640 1 -- 172.21.15.107:0/607506386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68840094d0 con 0x7f688815c480 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f688eca2640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f688000efc0 tx=0x7f688000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6880009ba0 con 0x7f688815c480 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000f040 con 0x7f688815c480 2023-11-24T08:16:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68800146c0 con 0x7f688815c480 2023-11-24T08:16:14.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68880ca250 con 0x7f688815c480 2023-11-24T08:16:14.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.898+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68880c9560 con 0x7f688815c480 2023-11-24T08:16:14.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.899+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68880c9780 con 0x7f688815c480 2023-11-24T08:16:14.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.900+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f6880014820 con 0x7f688815c480 2023-11-24T08:16:14.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.901+0000 7f687effd640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 0x7f686c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:14.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.901+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6880089110 con 0x7f688815c480 2023-11-24T08:16:14.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.901+0000 7f688e4a1640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 0x7f686c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:14.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.902+0000 7f688e4a1640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 0x7f686c06e7d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6884036000 tx=0x7f6884008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:14.906+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6880055c70 con 0x7f688815c480 2023-11-24T08:16:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.135+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f68880ca770 con 0x7f688815c480 2023-11-24T08:16:15.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.136+0000 7f687effd640 1 -- 172.21.15.107:0/607506386 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7f68880ca770 con 0x7f688815c480 2023-11-24T08:16:15.138 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 msgr2=0x7f686c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 0x7f686c06e7d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6884036000 tx=0x7f6884008030 comp rx=0 tx=0).stop 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 msgr2=0x7f688806c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f688000efc0 tx=0x7f688000c490 comp rx=0 tx=0).stop 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 shutdown_connections 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f686c06c310 0x7f686c06e7d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f688815c480 0x7f688806c590 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68880cf340 0x7f68880caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 --2- 172.21.15.107:0/607506386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68880ce2b0 0x7f6888155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 >> 172.21.15.107:0/607506386 conn(0x7f68880b2e40 msgr2=0x7f68880b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 shutdown_connections 2023-11-24T08:16:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:15.139+0000 7f6890705640 1 -- 172.21.15.107:0/607506386 wait complete. 2023-11-24T08:16:15.160 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:16:15.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:16:15.161 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:16:15.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.161 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:16:15.222 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:16:15.223 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41694sec preferred_lft 41694sec 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.224 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:16:15.225 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:16:15.226 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:16:15.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.227 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:16:15.316 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:16:15.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.321 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:16:15.376 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:16:15.376 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:16:15.376 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:16:15.376 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:16:15.377 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:16:15.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:16:15.377 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:16:15.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.560 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:16:15.613 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:16:15.614 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:16:15.615 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:16:15.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:16:15.703 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:16:15.703 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:16:15.754 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:16:15.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:16:15.755 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:16:15.804 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:16:15.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.804 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:15.938 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:15.939 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:16:15.974 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:16:15.974 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:16:15.974 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:16:15.975 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:16:15.975 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:16:15.978 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:16:15.978 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:16:15.978 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:16:15.978 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:16:15.978 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:16:15.982 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:16:15.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:15.983 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:16:16.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:16.069 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:16:16.127 INFO:teuthology.orchestra.run.smithi120.stdout:client5936 2023-11-24T08:16:16.128 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:16:16.128 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:16.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:16:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.399+0000 7fe066767640 1 Processor -- start 2023-11-24T08:16:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.399+0000 7fe066767640 1 -- start start 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 0x7fe0600d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600d12e0 0x7fe0600cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe06015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0600d05b0 con 0x7fe0600d12e0 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06015ea30 con 0x7fe060160b00 2023-11-24T08:16:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe066767640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06015de20 con 0x7fe0600d0120 2023-11-24T08:16:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe064f64640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600d12e0 0x7fe0600cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe064f64640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0600d12e0 0x7fe0600cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35632/0 (socket says 172.21.15.107:35632) 2023-11-24T08:16:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.401+0000 7fe064f64640 1 -- 172.21.15.107:0/1914567719 learned_addr learned my addr 172.21.15.107:0/1914567719 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe065f66640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe06015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe065765640 1 --2- 172.21.15.107:0/1914567719 >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 0x7fe0600d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe064f64640 1 -- 172.21.15.107:0/1914567719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 msgr2=0x7fe06015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe064f64640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe06015ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe064f64640 1 -- 172.21.15.107:0/1914567719 >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 msgr2=0x7fe0600d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe064f64640 1 --2- 172.21.15.107:0/1914567719 >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 0x7fe0600d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe064f64640 1 -- 172.21.15.107:0/1914567719 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06015e100 con 0x7fe0600d12e0 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe065f66640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe06015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.402+0000 7fe065765640 1 --2- 172.21.15.107:0/1914567719 >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 0x7fe0600d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.403+0000 7fe064f64640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600d12e0 0x7fe0600cfbe0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fe06008d3a0 tx=0x7fe05002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69d9faf924a5373 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe057fff640 1 -- 172.21.15.107:0/1914567719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050031870 con 0x7fe0600d12e0 2023-11-24T08:16:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe057fff640 1 -- 172.21.15.107:0/1914567719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0500319d0 con 0x7fe0600d12e0 2023-11-24T08:16:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe057fff640 1 -- 172.21.15.107:0/1914567719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050031d60 con 0x7fe0600d12e0 2023-11-24T08:16:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 -- 172.21.15.107:0/1914567719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600d12e0 msgr2=0x7fe0600cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600d12e0 0x7fe0600cfbe0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fe06008d3a0 tx=0x7fe05002b910 comp rx=0 tx=0).stop 2023-11-24T08:16:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 -- 172.21.15.107:0/1914567719 shutdown_connections 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe06015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 --2- 172.21.15.107:0/1914567719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0600d12e0 0x7fe0600cfbe0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 --2- 172.21.15.107:0/1914567719 >> v2:172.21.15.120:3300/0 conn(0x7fe060160b00 0x7fe0600d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 -- 172.21.15.107:0/1914567719 >> 172.21.15.107:0/1914567719 conn(0x7fe06009c510 msgr2=0x7fe0600d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.404+0000 7fe066767640 1 -- 172.21.15.107:0/1914567719 shutdown_connections 2023-11-24T08:16:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.405+0000 7fe066767640 1 -- 172.21.15.107:0/1914567719 wait complete. 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.406+0000 7fe066767640 1 Processor -- start 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.406+0000 7fe066767640 1 -- start start 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe0601f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 0x7fe0600cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06015f860 con 0x7fe060160b00 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601f3ca0 con 0x7fe0600d12e0 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.407+0000 7fe066767640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601f3f80 con 0x7fe0600d0120 2023-11-24T08:16:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35638/0 (socket says 172.21.15.107:35638) 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 -- 172.21.15.107:0/1959340197 learned_addr learned my addr 172.21.15.107:0/1959340197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065765640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe0601f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 -- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 msgr2=0x7fe0601f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe0601f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe064f64640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 0x7fe0600cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 -- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 msgr2=0x7fe0600cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 0x7fe0600cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 -- 172.21.15.107:0/1959340197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0500094d0 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe064f64640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 0x7fe0600cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.408+0000 7fe065f66640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe05800cef0 tx=0x7fe05800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.409+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05800d840 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.409+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05800d9a0 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.409+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05800dd50 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.409+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0600b7f60 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.409+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0600b7b80 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.410+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0600b8480 con 0x7fe060160b00 2023-11-24T08:16:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.411+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7fe05801e3b0 con 0x7fe060160b00 2023-11-24T08:16:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.412+0000 7fe055ffb640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 0x7fe04406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.412+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe058088d10 con 0x7fe060160b00 2023-11-24T08:16:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.412+0000 7fe065765640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 0x7fe04406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.413+0000 7fe065765640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 0x7fe04406e7d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00b840 tx=0x7fe05c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.417+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0600b8480 con 0x7fe060160b00 2023-11-24T08:16:16.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.645+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe0600b8480 con 0x7fe060160b00 2023-11-24T08:16:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:16.999+0000 7fe055ffb640 1 -- 172.21.15.107:0/1959340197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v88) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe0600b8480 con 0x7fe060160b00 2023-11-24T08:16:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 msgr2=0x7fe04406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 0x7fe04406e7d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00b840 tx=0x7fe05c008040 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 msgr2=0x7fe0600cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe05800cef0 tx=0x7fe05800a710 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 shutdown_connections 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04406c310 0x7fe04406e7d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe060160b00 0x7fe0600cc800 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0600d12e0 0x7fe0600cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 --2- 172.21.15.107:0/1959340197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0600d0120 0x7fe0601f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 >> 172.21.15.107:0/1959340197 conn(0x7fe06009c510 msgr2=0x7fe060156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 shutdown_connections 2023-11-24T08:16:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.002+0000 7fe066767640 1 -- 172.21.15.107:0/1959340197 wait complete. 2023-11-24T08:16:17.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:16:17.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.230+0000 7f96c101c640 1 Processor -- start 2023-11-24T08:16:17.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.230+0000 7f96c101c640 1 -- start start 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.231+0000 7f96c101c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96bc15b440 0x7f96bc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.231+0000 7f96c101c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.231+0000 7f96c101c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc15eea0 con 0x7f96bc0cbf20 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.231+0000 7f96c101c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc0c5e00 con 0x7f96bc15b440 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.231+0000 7f96c101c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0cc3d0 con 0x7f96bc0c59b0 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96bc15b440 0x7f96bc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f96bc15b440 0x7f96bc0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35642/0 (socket says 172.21.15.107:35642) 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 -- 172.21.15.107:0/1810760923 learned_addr learned my addr 172.21.15.107:0/1810760923 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96b3fff640 1 --1- 172.21.15.107:0/1810760923 >> v1:172.21.15.107:6789/0 conn(0x7f96bc0cbf20 0x7f96bc0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52080/0 (socket says 172.21.15.107:52080) 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 -- 172.21.15.107:0/1810760923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 msgr2=0x7f96bc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 --2- 172.21.15.107:0/1810760923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 -- 172.21.15.107:0/1810760923 >> v1:172.21.15.107:6789/0 conn(0x7f96bc0cbf20 legacy=0x7f96bc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.232+0000 7f96bad9d640 1 -- 172.21.15.107:0/1810760923 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0abf30 con 0x7f96bc15b440 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.233+0000 7f96bad9d640 1 --2- 172.21.15.107:0/1810760923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc0cb9e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009870 tx=0x7f96ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4edae08ccc664ea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96ba59c640 1 -- 172.21.15.107:0/1810760923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac031640 con 0x7f96bc15b440 2023-11-24T08:16:17.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96ba59c640 1 -- 172.21.15.107:0/1810760923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac0317a0 con 0x7f96bc15b440 2023-11-24T08:16:17.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96ba59c640 1 -- 172.21.15.107:0/1810760923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac031b30 con 0x7f96bc15b440 2023-11-24T08:16:17.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 -- 172.21.15.107:0/1810760923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 msgr2=0x7f96bc0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 --2- 172.21.15.107:0/1810760923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc0cb9e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009870 tx=0x7f96ac02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 -- 172.21.15.107:0/1810760923 shutdown_connections 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 --2- 172.21.15.107:0/1810760923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 --2- 172.21.15.107:0/1810760923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc0cb9e0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 -- 172.21.15.107:0/1810760923 >> 172.21.15.107:0/1810760923 conn(0x7f96bc0b0c50 msgr2=0x7f96bc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 -- 172.21.15.107:0/1810760923 shutdown_connections 2023-11-24T08:16:17.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.234+0000 7f96c101c640 1 -- 172.21.15.107:0/1810760923 wait complete. 2023-11-24T08:16:17.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.236+0000 7f96c101c640 1 Processor -- start 2023-11-24T08:16:17.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.236+0000 7f96c101c640 1 -- start start 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96bc0cbf20 0x7f96bc0c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0c7d60 con 0x7f96bc15b440 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0c8040 con 0x7f96bc0cbf20 2023-11-24T08:16:17.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96c101c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0c8320 con 0x7f96bc0c59b0 2023-11-24T08:16:17.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.237+0000 7f96bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35648/0 (socket says 172.21.15.107:35648) 2023-11-24T08:16:17.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 -- 172.21.15.107:0/218782094 learned_addr learned my addr 172.21.15.107:0/218782094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bad9d640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 -- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 msgr2=0x7f96bc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96b3fff640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96bc0cbf20 0x7f96bc0c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 -- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96bc0cbf20 msgr2=0x7f96bc0c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96bc0cbf20 0x7f96bc0c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 -- 172.21.15.107:0/218782094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0094d0 con 0x7f96bc15b440 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96bb59e640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f96b400b970 tx=0x7f96b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b400cb40 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b400ccc0 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.238+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4014680 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.239+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc0c8600 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.239+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc157410 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.240+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc0c8b20 con 0x7f96bc15b440 2023-11-24T08:16:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.241+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f96b401d3b0 con 0x7f96bc15b440 2023-11-24T08:16:17.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.242+0000 7f96b37fe640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 0x7f969c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.242+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96b40882d0 con 0x7f96bc15b440 2023-11-24T08:16:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.242+0000 7f96bad9d640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 0x7f969c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.243+0000 7f96bad9d640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 0x7f969c06e7d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f96ac036000 tx=0x7f96ac03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.246+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96b40bee90 con 0x7f96bc15b440 2023-11-24T08:16:17.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.461+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f96bc0ccd90 con 0x7f969c06c310 2023-11-24T08:16:17.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.463+0000 7f96b37fe640 1 -- 172.21.15.107:0/218782094 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f96bc0ccd90 con 0x7f969c06c310 2023-11-24T08:16:17.465 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:16:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:16:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:16:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:17.466 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 msgr2=0x7f969c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 0x7f969c06e7d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f96ac036000 tx=0x7f96ac03c040 comp rx=0 tx=0).stop 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 msgr2=0x7f96bc15dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f96b400b970 tx=0x7f96b400be30 comp rx=0 tx=0).stop 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 shutdown_connections 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f969c06c310 0x7f969c06e7d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96bc15b440 0x7f96bc15dd00 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96bc0cbf20 0x7f96bc0c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 --2- 172.21.15.107:0/218782094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96bc0c59b0 0x7f96bc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 >> 172.21.15.107:0/218782094 conn(0x7f96bc0b0c50 msgr2=0x7f96bc0c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:17.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 shutdown_connections 2023-11-24T08:16:17.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.466+0000 7f96c101c640 1 -- 172.21.15.107:0/218782094 wait complete. 2023-11-24T08:16:17.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:16:17.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.690+0000 7f974ca0d640 1 Processor -- start 2023-11-24T08:16:17.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.690+0000 7f974ca0d640 1 -- start start 2023-11-24T08:16:17.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.692+0000 7f974ca0d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f974815d5f0 0x7f97480ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.692+0000 7f974ca0d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480cdf70 con 0x7f974806d080 2023-11-24T08:16:17.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.692+0000 7f974ca0d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480cbfa0 con 0x7f97480ac840 2023-11-24T08:16:17.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.692+0000 7f974ca0d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480cb3d0 con 0x7f974815d5f0 2023-11-24T08:16:17.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.692+0000 7f9746d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f974815d5f0 0x7f97480ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.693+0000 7f974759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f974806d080 0x7f97480ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52090/0 (socket says 172.21.15.107:52090) 2023-11-24T08:16:17.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.693+0000 7f974759e640 1 -- 172.21.15.107:0/1840773911 learned_addr learned my addr 172.21.15.107:0/1840773911 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:17.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.693+0000 7f9746d9d640 1 -- 172.21.15.107:0/1840773911 >> v1:172.21.15.120:6789/0 conn(0x7f97480ac840 legacy=0x7f974806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:17.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.693+0000 7f9746d9d640 1 -- 172.21.15.107:0/1840773911 >> v1:172.21.15.107:6789/0 conn(0x7f974806d080 legacy=0x7f97480ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.693+0000 7f9746d9d640 1 -- 172.21.15.107:0/1840773911 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480cc280 con 0x7f974815d5f0 2023-11-24T08:16:17.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.694+0000 7f9746d9d640 1 --2- 172.21.15.107:0/1840773911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f974815d5f0 0x7f97480ae020 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f97380097a0 tx=0x7f973802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10132862f334a7c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.694+0000 7f974659c640 1 -- 172.21.15.107:0/1840773911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973802e070 con 0x7f974815d5f0 2023-11-24T08:16:17.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.694+0000 7f974659c640 1 -- 172.21.15.107:0/1840773911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97380315a0 con 0x7f974815d5f0 2023-11-24T08:16:17.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.694+0000 7f974659c640 1 -- 172.21.15.107:0/1840773911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738031930 con 0x7f974815d5f0 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 -- 172.21.15.107:0/1840773911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f974815d5f0 msgr2=0x7f97480ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1840773911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f974815d5f0 0x7f97480ae020 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f97380097a0 tx=0x7f973802b840 comp rx=0 tx=0).stop 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 -- 172.21.15.107:0/1840773911 shutdown_connections 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1840773911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f974815d5f0 0x7f97480ae020 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 -- 172.21.15.107:0/1840773911 >> 172.21.15.107:0/1840773911 conn(0x7f974806a500 msgr2=0x7f974806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 -- 172.21.15.107:0/1840773911 shutdown_connections 2023-11-24T08:16:17.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.695+0000 7f974ca0d640 1 -- 172.21.15.107:0/1840773911 wait complete. 2023-11-24T08:16:17.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.697+0000 7f974ca0d640 1 Processor -- start 2023-11-24T08:16:17.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.697+0000 7f974ca0d640 1 -- start start 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f974806d080 0x7f9748155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f974815d5f0 0x7f974814e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480b0a80 con 0x7f97480ac840 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480b0d60 con 0x7f974806d080 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f974ca0d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480b1040 con 0x7f974815d5f0 2023-11-24T08:16:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35662/0 (socket says 172.21.15.107:35662) 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 -- 172.21.15.107:0/1664915278 learned_addr learned my addr 172.21.15.107:0/1664915278 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 -- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f974815d5f0 msgr2=0x7f974814e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f974815d5f0 0x7f974814e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 -- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f974806d080 msgr2=0x7f9748155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f974806d080 0x7f9748155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 -- 172.21.15.107:0/1664915278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009400 con 0x7f97480ac840 2023-11-24T08:16:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.698+0000 7f973ffff640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f973000c400 tx=0x7f973000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.699+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730011790 con 0x7f97480ac840 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.699+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9730011930 con 0x7f97480ac840 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.699+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730011ce0 con 0x7f97480ac840 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.699+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97480b1320 con 0x7f97480ac840 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.699+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9748155240 con 0x7f97480ac840 2023-11-24T08:16:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.700+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9748067990 con 0x7f97480ac840 2023-11-24T08:16:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.701+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f973001c460 con 0x7f97480ac840 2023-11-24T08:16:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.702+0000 7f973f7fe640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 0x7f972806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.702+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9730088620 con 0x7f97480ac840 2023-11-24T08:16:17.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.702+0000 7f9746d9d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 0x7f972806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:17.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.703+0000 7f9746d9d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 0x7f972806e7d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9738036000 tx=0x7f9738030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:17.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.706+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9730054da0 con 0x7f97480ac840 2023-11-24T08:16:17.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.898+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97480b1840 con 0x7f97480ac840 2023-11-24T08:16:17.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.899+0000 7f973f7fe640 1 -- 172.21.15.107:0/1664915278 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v39)=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97480b1840 con 0x7f97480ac840 2023-11-24T08:16:17.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 msgr2=0x7f972806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 0x7f972806e7d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9738036000 tx=0x7f9738030000 comp rx=0 tx=0).stop 2023-11-24T08:16:17.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 msgr2=0x7f97480affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:17.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f973000c400 tx=0x7f973000c8c0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 shutdown_connections 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f972806c310 0x7f972806e7d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f974815d5f0 0x7f974814e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97480ac840 0x7f97480affe0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 --2- 172.21.15.107:0/1664915278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f974806d080 0x7f9748155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 >> 172.21.15.107:0/1664915278 conn(0x7f974806a500 msgr2=0x7f97480723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.902+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 shutdown_connections 2023-11-24T08:16:17.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:17.903+0000 7f974ca0d640 1 -- 172.21.15.107:0/1664915278 wait complete. 2023-11-24T08:16:17.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol6903 --mode=777 2023-11-24T08:16:18.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.137+0000 7f68d0361640 1 Processor -- start 2023-11-24T08:16:18.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.137+0000 7f68d0361640 1 -- start start 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c80ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f68c80b1380 0x7f68c80b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68c806a500 0x7f68c806a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c806aed0 con 0x7f68c80b1380 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c80af420 con 0x7f68c806a500 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.138+0000 7f68d0361640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c80af700 con 0x7f68c8145030 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f68c80b1380 0x7f68c80b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f68c80b1380 0x7f68c80b17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37828/0 (socket says 172.21.15.107:37828) 2023-11-24T08:16:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 -- 172.21.15.107:0/2724603606 learned_addr learned my addr 172.21.15.107:0/2724603606 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68ce8fe640 1 --2- 172.21.15.107:0/2724603606 >> v2:172.21.15.120:3300/0 conn(0x7f68c806a500 0x7f68c806a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68ce0fd640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c80ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 -- 172.21.15.107:0/2724603606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 msgr2=0x7f68c80ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c80ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 -- 172.21.15.107:0/2724603606 >> v2:172.21.15.120:3300/0 conn(0x7f68c806a500 msgr2=0x7f68c806a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/2724603606 >> v2:172.21.15.120:3300/0 conn(0x7f68c806a500 0x7f68c806a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.139+0000 7f68cd8fc640 1 -- 172.21.15.107:0/2724603606 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c80af9e0 con 0x7f68c80b1380 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.140+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c80b17e0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f68c80b6130 tx=0x7f68b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365f32cf6e85998d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.141+0000 7f68cd0fb640 1 -- 172.21.15.107:0/2724603606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b80317e0 con 0x7f68c80b1380 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.141+0000 7f68cd0fb640 1 -- 172.21.15.107:0/2724603606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b8031940 con 0x7f68c80b1380 2023-11-24T08:16:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.141+0000 7f68cd0fb640 1 -- 172.21.15.107:0/2724603606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b8031cd0 con 0x7f68c80b1380 2023-11-24T08:16:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.141+0000 7f68d0361640 1 -- 172.21.15.107:0/2724603606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 msgr2=0x7f68c80b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.141+0000 7f68d0361640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c80b17e0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f68c80b6130 tx=0x7f68b802b880 comp rx=0 tx=0).stop 2023-11-24T08:16:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 -- 172.21.15.107:0/2724603606 shutdown_connections 2023-11-24T08:16:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 --2- 172.21.15.107:0/2724603606 >> v2:172.21.15.120:3300/0 conn(0x7f68c806a500 0x7f68c806a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c80b17e0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 --2- 172.21.15.107:0/2724603606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c80ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 -- 172.21.15.107:0/2724603606 >> 172.21.15.107:0/2724603606 conn(0x7f68c80b5580 msgr2=0x7f68c80b5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 -- 172.21.15.107:0/2724603606 shutdown_connections 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.142+0000 7f68d0361640 1 -- 172.21.15.107:0/2724603606 wait complete. 2023-11-24T08:16:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.144+0000 7f68d0361640 1 Processor -- start 2023-11-24T08:16:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.144+0000 7f68d0361640 1 -- start start 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68c806a500 0x7f68c8157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c81f6ea0 con 0x7f68c80b1380 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c8157770 con 0x7f68c806a500 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.145+0000 7f68d0361640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c81f62b0 con 0x7f68c8145030 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37840/0 (socket says 172.21.15.107:37840) 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 -- 172.21.15.107:0/1617297955 learned_addr learned my addr 172.21.15.107:0/1617297955 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 -- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 msgr2=0x7f68c813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 -- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68c806a500 msgr2=0x7f68c8157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68ce0fd640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68c806a500 0x7f68c8157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68c806a500 0x7f68c8157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 -- 172.21.15.107:0/1617297955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80094d0 con 0x7f68c80b1380 2023-11-24T08:16:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.146+0000 7f68cd8fc640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f68b802b8c0 tx=0x7f68b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.147+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b8032880 con 0x7f68c80b1380 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.147+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b80329e0 con 0x7f68c80b1380 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.147+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b8032d90 con 0x7f68c80b1380 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.147+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c815b190 con 0x7f68c80b1380 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.147+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c81f67e0 con 0x7f68c80b1380 2023-11-24T08:16:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.148+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c81f6fe0 con 0x7f68c80b1380 2023-11-24T08:16:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.149+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f68b804e450 con 0x7f68c80b1380 2023-11-24T08:16:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.150+0000 7f68beffd640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 0x7f68ac06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.150+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68b80aa3f0 con 0x7f68c80b1380 2023-11-24T08:16:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.150+0000 7f68ce0fd640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 0x7f68ac06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.151+0000 7f68ce0fd640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 0x7f68ac06e7d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f68c4006fd0 tx=0x7f68c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.154+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68c81f6fe0 con 0x7f68c80b1380 2023-11-24T08:16:18.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.370+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol6903", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c80affa0 con 0x7f68ac06c310 2023-11-24T08:16:18.385 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:18.383+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:18.385 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:18.383+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:18.386 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:18.383+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:18.386 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:18.383+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:18.386 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:18.383+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.433+0000 7f68beffd640 1 -- 172.21.15.107:0/1617297955 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68c400e4b0 con 0x7f68ac06c310 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 msgr2=0x7f68ac06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 0x7f68ac06e7d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f68c4006fd0 tx=0x7f68c4008040 comp rx=0 tx=0).stop 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 msgr2=0x7f68c813c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f68b802b8c0 tx=0x7f68b80046c0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 shutdown_connections 2023-11-24T08:16:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68ac06c310 0x7f68ac06e7d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68c8145030 0x7f68c813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68c80b1380 0x7f68c813c620 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 --2- 172.21.15.107:0/1617297955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68c806a500 0x7f68c8157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 >> 172.21.15.107:0/1617297955 conn(0x7f68c80b5580 msgr2=0x7f68c806efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 shutdown_connections 2023-11-24T08:16:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.436+0000 7f68d0361640 1 -- 172.21.15.107:0/1617297955 wait complete. 2023-11-24T08:16:18.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol6903 2023-11-24T08:16:18.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.666+0000 7f6f5858a640 1 Processor -- start 2023-11-24T08:16:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.666+0000 7f6f5858a640 1 -- start start 2023-11-24T08:16:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f5858a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6f500cc0f0 0x7f6f500c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f5858a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f500c57e0 0x7f6f5006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f5858a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5015eea0 con 0x7f6f5015b440 2023-11-24T08:16:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f5858a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5015b820 con 0x7f6f500c57e0 2023-11-24T08:16:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f5858a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f500abf30 con 0x7f6f500cc0f0 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.668+0000 7f6f56b27640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f500c57e0 0x7f6f5006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f500c57e0 0x7f6f5006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37856/0 (socket says 172.21.15.107:37856) 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 -- 172.21.15.107:0/3222410142 learned_addr learned my addr 172.21.15.107:0/3222410142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 -- 172.21.15.107:0/3222410142 >> v2:172.21.15.120:3300/0 conn(0x7f6f500cc0f0 msgr2=0x7f6f500c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 --2- 172.21.15.107:0/3222410142 >> v2:172.21.15.120:3300/0 conn(0x7f6f500cc0f0 0x7f6f500c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 -- 172.21.15.107:0/3222410142 >> v1:172.21.15.120:6789/0 conn(0x7f6f5015b440 legacy=0x7f6f500cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.669+0000 7f6f56b27640 1 -- 172.21.15.107:0/3222410142 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5015f0f0 con 0x7f6f500c57e0 2023-11-24T08:16:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.670+0000 7f6f56b27640 1 --2- 172.21.15.107:0/3222410142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500c57e0 0x7f6f5006c5f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f6f5006e720 tx=0x7f6f4802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4199ec64baa90df server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.670+0000 7f6f55324640 1 -- 172.21.15.107:0/3222410142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4802c9f0 con 0x7f6f500c57e0 2023-11-24T08:16:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f55324640 1 -- 172.21.15.107:0/3222410142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4802cb50 con 0x7f6f500c57e0 2023-11-24T08:16:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f55324640 1 -- 172.21.15.107:0/3222410142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480343f0 con 0x7f6f500c57e0 2023-11-24T08:16:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 -- 172.21.15.107:0/3222410142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500c57e0 msgr2=0x7f6f5006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 --2- 172.21.15.107:0/3222410142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500c57e0 0x7f6f5006c5f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f6f5006e720 tx=0x7f6f4802be50 comp rx=0 tx=0).stop 2023-11-24T08:16:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 -- 172.21.15.107:0/3222410142 shutdown_connections 2023-11-24T08:16:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 --2- 172.21.15.107:0/3222410142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f500c57e0 0x7f6f5006c5f0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 --2- 172.21.15.107:0/3222410142 >> v2:172.21.15.120:3300/0 conn(0x7f6f500cc0f0 0x7f6f500c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 -- 172.21.15.107:0/3222410142 >> 172.21.15.107:0/3222410142 conn(0x7f6f500b0c50 msgr2=0x7f6f500b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.671+0000 7f6f5858a640 1 -- 172.21.15.107:0/3222410142 shutdown_connections 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.672+0000 7f6f5858a640 1 -- 172.21.15.107:0/3222410142 wait complete. 2023-11-24T08:16:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.674+0000 7f6f5858a640 1 Processor -- start 2023-11-24T08:16:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.674+0000 7f6f5858a640 1 -- start start 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500c57e0 0x7f6f50154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 0x7f6f50156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c6e40 con 0x7f6f5015b440 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50154c60 con 0x7f6f500c57e0 2023-11-24T08:16:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.675+0000 7f6f5858a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c6270 con 0x7f6f500cc0f0 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37858/0 (socket says 172.21.15.107:37858) 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 -- 172.21.15.107:0/2842488404 learned_addr learned my addr 172.21.15.107:0/2842488404 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56326640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500c57e0 0x7f6f50154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f55b25640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 0x7f6f50156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 -- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 msgr2=0x7f6f50156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 0x7f6f50156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 -- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500c57e0 msgr2=0x7f6f50154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500c57e0 0x7f6f50154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 -- 172.21.15.107:0/2842488404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009b30 con 0x7f6f5015b440 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f55b25640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 0x7f6f50156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.676+0000 7f6f56b27640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f6f48036040 tx=0x7f6f48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.677+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4803e450 con 0x7f6f5015b440 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.677+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4803e5d0 con 0x7f6f5015b440 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.677+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4803e980 con 0x7f6f5015b440 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.677+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f500c6510 con 0x7f6f5015b440 2023-11-24T08:16:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.677+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f500c6ad0 con 0x7f6f5015b440 2023-11-24T08:16:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.678+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5004e8b0 con 0x7f6f5015b440 2023-11-24T08:16:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.679+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 29) v1 ==== 91118+0+0 (secure 0 0 0) 0x7f6f4803eae0 con 0x7f6f5015b440 2023-11-24T08:16:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.680+0000 7f6f46ffd640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 0x7f6f3406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.680+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f480ab150 con 0x7f6f5015b440 2023-11-24T08:16:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.680+0000 7f6f56326640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 0x7f6f3406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.681+0000 7f6f56326640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 0x7f6f3406e7d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006fd0 tx=0x7f6f4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:18.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.686+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f48077d60 con 0x7f6f5015b440 2023-11-24T08:16:18.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.906+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol6903", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f5015f990 con 0x7f6f3406c310 2023-11-24T08:16:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.915+0000 7f6f46ffd640 1 -- 172.21.15.107:0/2842488404 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f6f4c00bda0 con 0x7f6f3406c310 2023-11-24T08:16:18.917 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 msgr2=0x7f6f3406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 0x7f6f3406e7d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006fd0 tx=0x7f6f4c008040 comp rx=0 tx=0).stop 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 msgr2=0x7f6f50156f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f6f48036040 tx=0x7f6f48004a40 comp rx=0 tx=0).stop 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 shutdown_connections 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f3406c310 0x7f6f3406e7d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5015b440 0x7f6f50156f80 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f500cc0f0 0x7f6f50156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 --2- 172.21.15.107:0/2842488404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f500c57e0 0x7f6f50154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 >> 172.21.15.107:0/2842488404 conn(0x7f6f500b0c50 msgr2=0x7f6f50071010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 shutdown_connections 2023-11-24T08:16:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:18.918+0000 7f6f5858a640 1 -- 172.21.15.107:0/2842488404 wait complete. 2023-11-24T08:16:18.939 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol6903 with 64 files each 1MB size under directory volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05 2023-11-24T08:16:18.939 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:18.940 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:18.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:18.990 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314878 s, 33.3 MB/s 2023-11-24T08:16:19.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.082 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.136 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.258 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.259 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.259 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233155 s, 45.0 MB/s 2023-11-24T08:16:19.260 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.260 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.317 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.398 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.399 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.399 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277585 s, 37.8 MB/s 2023-11-24T08:16:19.400 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.400 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.461 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257221 s, 40.8 MB/s 2023-11-24T08:16:19.544 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.544 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204683 s, 51.2 MB/s 2023-11-24T08:16:19.679 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.679 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.731 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.849 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.849 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193763 s, 54.1 MB/s 2023-11-24T08:16:19.850 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:19.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:19.906 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:19.980 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:19.980 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:19.980 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225269 s, 46.5 MB/s 2023-11-24T08:16:19.982 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:19.982 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.046 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182931 s, 57.3 MB/s 2023-11-24T08:16:20.110 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.111 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.163 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247872 s, 42.3 MB/s 2023-11-24T08:16:20.289 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.290 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017513 s, 59.9 MB/s 2023-11-24T08:16:20.416 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.416 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.466 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244576 s, 42.9 MB/s 2023-11-24T08:16:20.589 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.589 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.648 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.721 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.721 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019267 s, 54.4 MB/s 2023-11-24T08:16:20.723 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.723 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.852 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.852 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.852 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193444 s, 54.2 MB/s 2023-11-24T08:16:20.854 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.854 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:20.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:20.907 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255662 s, 41.0 MB/s 2023-11-24T08:16:20.986 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:20.986 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189106 s, 55.4 MB/s 2023-11-24T08:16:21.115 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.116 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.172 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.240 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.241 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.241 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224902 s, 46.6 MB/s 2023-11-24T08:16:21.242 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.242 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.299 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.375 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.376 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.376 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238513 s, 44.0 MB/s 2023-11-24T08:16:21.377 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.377 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.433 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.508 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.508 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231058 s, 45.4 MB/s 2023-11-24T08:16:21.509 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.509 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.634 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.634 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.634 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220709 s, 47.5 MB/s 2023-11-24T08:16:21.635 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.636 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.692 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.768 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.768 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.769 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230114 s, 45.6 MB/s 2023-11-24T08:16:21.770 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.770 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:21.905 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:21.905 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:21.905 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224411 s, 46.7 MB/s 2023-11-24T08:16:21.907 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:21.907 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:21.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:21.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.042 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.043 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.043 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231916 s, 45.2 MB/s 2023-11-24T08:16:22.044 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.045 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.176 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.176 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.176 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217761 s, 48.2 MB/s 2023-11-24T08:16:22.177 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.177 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.235 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238788 s, 43.9 MB/s 2023-11-24T08:16:22.315 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.315 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.373 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.448 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.449 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.449 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245527 s, 42.7 MB/s 2023-11-24T08:16:22.450 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.450 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.510 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.582 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.583 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020841 s, 50.3 MB/s 2023-11-24T08:16:22.584 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.584 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.642 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.723 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.723 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252459 s, 41.5 MB/s 2023-11-24T08:16:22.725 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.725 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.781 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:22.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:22.850 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:22.850 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017461 s, 60.1 MB/s 2023-11-24T08:16:22.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:22.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:22.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:22.900 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247343 s, 42.4 MB/s 2023-11-24T08:16:23.027 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.028 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.081 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.212 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.212 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277555 s, 37.8 MB/s 2023-11-24T08:16:23.213 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.214 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.354 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.354 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.355 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030201 s, 34.7 MB/s 2023-11-24T08:16:23.356 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.356 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.415 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.489 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210948 s, 49.7 MB/s 2023-11-24T08:16:23.491 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.491 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202582 s, 51.8 MB/s 2023-11-24T08:16:23.627 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.627 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198093 s, 52.9 MB/s 2023-11-24T08:16:23.759 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.759 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.813 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:23.890 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:23.890 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:23.890 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244113 s, 43.0 MB/s 2023-11-24T08:16:23.892 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:23.892 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:23.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:23.949 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225636 s, 46.5 MB/s 2023-11-24T08:16:24.025 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.026 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.083 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258203 s, 40.6 MB/s 2023-11-24T08:16:24.164 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.164 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.221 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.301 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.301 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.301 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239823 s, 43.7 MB/s 2023-11-24T08:16:24.302 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.302 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.361 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196194 s, 53.4 MB/s 2023-11-24T08:16:24.437 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.437 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.493 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.570 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.570 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.570 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230369 s, 45.5 MB/s 2023-11-24T08:16:24.572 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.572 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263247 s, 39.8 MB/s 2023-11-24T08:16:24.711 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.711 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021032 s, 49.9 MB/s 2023-11-24T08:16:24.844 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:24.844 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:24.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:24.897 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.020 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.020 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.020 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204665 s, 51.2 MB/s 2023-11-24T08:16:25.021 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.021 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.077 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016443 s, 63.8 MB/s 2023-11-24T08:16:25.141 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.141 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.193 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265053 s, 39.6 MB/s 2023-11-24T08:16:25.272 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.273 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.328 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.403 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226086 s, 46.4 MB/s 2023-11-24T08:16:25.405 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.405 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.462 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221031 s, 47.4 MB/s 2023-11-24T08:16:25.541 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.541 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.598 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022212 s, 47.2 MB/s 2023-11-24T08:16:25.679 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.679 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.733 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:25.861 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:25.861 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:25.861 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02707 s, 38.7 MB/s 2023-11-24T08:16:25.862 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:25.863 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:25.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:25.917 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.003 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.003 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295513 s, 35.5 MB/s 2023-11-24T08:16:26.005 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.006 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.059 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.180 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204087 s, 51.4 MB/s 2023-11-24T08:16:26.183 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.183 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024287 s, 43.2 MB/s 2023-11-24T08:16:26.324 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.325 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.468 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257556 s, 40.7 MB/s 2023-11-24T08:16:26.469 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.469 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.526 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.597 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206652 s, 50.7 MB/s 2023-11-24T08:16:26.599 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.599 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.730 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.731 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.731 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232487 s, 45.1 MB/s 2023-11-24T08:16:26.732 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.733 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.792 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189238 s, 55.4 MB/s 2023-11-24T08:16:26.866 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.867 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:26.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:26.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179572 s, 58.4 MB/s 2023-11-24T08:16:26.996 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:26.996 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.052 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.121 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188687 s, 55.6 MB/s 2023-11-24T08:16:27.122 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.122 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.180 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.253 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.253 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.253 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201972 s, 51.9 MB/s 2023-11-24T08:16:27.255 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.255 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.313 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.383 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.383 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.383 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217205 s, 48.3 MB/s 2023-11-24T08:16:27.384 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.384 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.440 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200333 s, 52.3 MB/s 2023-11-24T08:16:27.557 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.557 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.618 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.701 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.701 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.701 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272622 s, 38.5 MB/s 2023-11-24T08:16:27.705 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.705 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.765 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208422 s, 50.3 MB/s 2023-11-24T08:16:27.840 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:16:27.840 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:16:27.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:27.897 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:16:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:16:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:16:27.976 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211851 s, 49.5 MB/s 2023-11-24T08:16:27.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol6903 subvol_snap5787 2023-11-24T08:16:28.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.198+0000 7f04afeee640 1 Processor -- start 2023-11-24T08:16:28.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.198+0000 7f04afeee640 1 -- start start 2023-11-24T08:16:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.199+0000 7f04afeee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.199+0000 7f04afeee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04a80cf450 0x7f04a80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.199+0000 7f04afeee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a80cece0 con 0x7f04a80cdfb0 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.199+0000 7f04afeee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a8156010 con 0x7f04a80cf450 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.199+0000 7f04afeee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a806ce30 con 0x7f04a815c7f0 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04a80cf450 0x7f04a80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04a80cf450 0x7f04a80cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33202/0 (socket says 172.21.15.107:33202) 2023-11-24T08:16:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 -- 172.21.15.107:0/1548670862 learned_addr learned my addr 172.21.15.107:0/1548670862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04adc8a640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 -- 172.21.15.107:0/1548670862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 msgr2=0x7f04a806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 -- 172.21.15.107:0/1548670862 >> v1:172.21.15.120:6789/0 conn(0x7f04a80cdfb0 legacy=0x7f04a80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.200+0000 7f04ad489640 1 -- 172.21.15.107:0/1548670862 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a806d040 con 0x7f04a80cf450 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.202+0000 7f04ad489640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a80cf8d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0498009870 tx=0x7f049802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dba2d70dd271b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.202+0000 7f04acc88640 1 -- 172.21.15.107:0/1548670862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498031640 con 0x7f04a80cf450 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.202+0000 7f04acc88640 1 -- 172.21.15.107:0/1548670862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04980317a0 con 0x7f04a80cf450 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.202+0000 7f04acc88640 1 -- 172.21.15.107:0/1548670862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498031b30 con 0x7f04a80cf450 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 -- 172.21.15.107:0/1548670862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 msgr2=0x7f04a80cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a80cf8d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0498009870 tx=0x7f049802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 -- 172.21.15.107:0/1548670862 shutdown_connections 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a80cf8d0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 --2- 172.21.15.107:0/1548670862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 -- 172.21.15.107:0/1548670862 >> 172.21.15.107:0/1548670862 conn(0x7f04a806a920 msgr2=0x7f04a806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 -- 172.21.15.107:0/1548670862 shutdown_connections 2023-11-24T08:16:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.203+0000 7f04afeee640 1 -- 172.21.15.107:0/1548670862 wait complete. 2023-11-24T08:16:28.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.205+0000 7f04afeee640 1 Processor -- start 2023-11-24T08:16:28.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.205+0000 7f04afeee640 1 -- start start 2023-11-24T08:16:28.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 0x7f04a8155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80c8540 con 0x7f04a80cf450 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80c87e0 con 0x7f04a80cdfb0 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04afeee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8157540 con 0x7f04a815c7f0 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04ad489640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04ad489640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33212/0 (socket says 172.21.15.107:33212) 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.206+0000 7f04ad489640 1 -- 172.21.15.107:0/2934176894 learned_addr learned my addr 172.21.15.107:0/2934176894 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:28.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 -- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 msgr2=0x7f04a815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 -- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 msgr2=0x7f04a8155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04adc8a640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 0x7f04a8155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 0x7f04a8155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 -- 172.21.15.107:0/2934176894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980094d0 con 0x7f04a80cf450 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04adc8a640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 0x7f04a8155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f04ad489640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f0498000c00 tx=0x7f04980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.207+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049803ac60 con 0x7f04a80cf450 2023-11-24T08:16:28.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.208+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049803ade0 con 0x7f04a80cf450 2023-11-24T08:16:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.208+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a8157820 con 0x7f04a80cf450 2023-11-24T08:16:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.208+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498044630 con 0x7f04a80cf450 2023-11-24T08:16:28.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.208+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a8157de0 con 0x7f04a80cf450 2023-11-24T08:16:28.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.209+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a80c8950 con 0x7f04a80cf450 2023-11-24T08:16:28.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.210+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 30) v1 ==== 91169+0+0 (secure 0 0 0) 0x7f049802f050 con 0x7f04a80cf450 2023-11-24T08:16:28.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.211+0000 7f049e7fc640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 0x7f048c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.211+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04980aa840 con 0x7f04a80cf450 2023-11-24T08:16:28.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.211+0000 7f04adc8a640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 0x7f048c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.212+0000 7f04adc8a640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 0x7f048c06e860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007eb0 tx=0x7f04a4007400 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.216+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04980770c0 con 0x7f04a80cf450 2023-11-24T08:16:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.436+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol6903", "snap_name": "subvol_snap5787", "target": ["mon-mgr", ""]}) v1 -- 0x7f04a8061ac0 con 0x7f048c06c3a0 2023-11-24T08:16:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.484+0000 7f049e7fc640 1 -- 172.21.15.107:0/2934176894 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f04a8061ac0 con 0x7f048c06c3a0 2023-11-24T08:16:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 msgr2=0x7f048c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 0x7f048c06e860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007eb0 tx=0x7f04a4007400 comp rx=0 tx=0).stop 2023-11-24T08:16:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 msgr2=0x7f04a8156b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f0498000c00 tx=0x7f04980046c0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 shutdown_connections 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f048c06c3a0 0x7f048c06e860 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a815c7f0 0x7f04a815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a80cf450 0x7f04a8156b60 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 --2- 172.21.15.107:0/2934176894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a80cdfb0 0x7f04a8155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 >> 172.21.15.107:0/2934176894 conn(0x7f04a806a920 msgr2=0x7f04a80ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 shutdown_connections 2023-11-24T08:16:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.487+0000 7f04afeee640 1 -- 172.21.15.107:0/2934176894 wait complete. 2023-11-24T08:16:28.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 2 2023-11-24T08:16:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.714+0000 7fbc19a1e640 1 Processor -- start 2023-11-24T08:16:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.714+0000 7fbc19a1e640 1 -- start start 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.715+0000 7fbc19a1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc141616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.715+0000 7fbc19a1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14161ce0 con 0x7fbc1415d270 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.715+0000 7fbc19a1e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14161fc0 con 0x7fbc14161260 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.715+0000 7fbc19a1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc140ce310 con 0x7fbc140cf220 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.716+0000 7fbc12ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbc1415d270 0x7fbc1415d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52278/0 (socket says 172.21.15.107:52278) 2023-11-24T08:16:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.716+0000 7fbc12ffd640 1 -- 172.21.15.107:0/2351785160 learned_addr learned my addr 172.21.15.107:0/2351785160 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.716+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc141616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.716+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996045450 0 0) 0x7fbc14161ce0 con 0x7fbc1415d270 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.716+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00003520 con 0x7fbc1415d270 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742112068 0 0) 0x7fbc00003520 con 0x7fbc1415d270 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 msgr2=0x7fbc141616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc141616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 >> v1:172.21.15.120:6789/0 conn(0x7fbc14161260 legacy=0x7fbc140ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc140ce5f0 con 0x7fbc1415d270 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbc040019f0 con 0x7fbc1415d270 2023-11-24T08:16:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 >> v1:172.21.15.107:6789/0 conn(0x7fbc1415d270 legacy=0x7fbc1415d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc000039a0 0x7fbc00003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc00004d00 0x7fbc00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc140ce310 con 0x7fbc00004d00 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0000ba90 con 0x7fbc000039a0 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0000bc50 con 0x7fbc140cf220 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc00004d00 0x7fbc00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc12ffd640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc000039a0 0x7fbc00003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.717+0000 7fbc127fc640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc137fe640 1 -- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 msgr2=0x7fbc000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc137fe640 1 -- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc000039a0 msgr2=0x7fbc00003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc000039a0 0x7fbc00003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc137fe640 1 -- 172.21.15.107:0/2351785160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc00003520 con 0x7fbc00004d00 2023-11-24T08:16:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.718+0000 7fbc127fc640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc140cf220 0x7fbc000047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc00004d00 0x7fbc00005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fbc140d3300 tx=0x7fbc000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b500fcda53afad84 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c02c630 con 0x7fbc00004d00 2023-11-24T08:16:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c02c790 con 0x7fbc00004d00 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc11ffb640 1 -- 172.21.15.107:0/2351785160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c02cb40 con 0x7fbc00004d00 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2351785160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc00004d00 msgr2=0x7fbc00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2351785160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc00004d00 0x7fbc00005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fbc140d3300 tx=0x7fbc000057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.719+0000 7fbc137fe640 1 -- 172.21.15.107:0/2351785160 reap_dead start 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.720+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2351785160 shutdown_connections 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.720+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2351785160 >> 172.21.15.107:0/2351785160 conn(0x7fbc1408af60 msgr2=0x7fbc1409c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.720+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2351785160 shutdown_connections 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.720+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2351785160 wait complete. 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.721+0000 7fbc19a1e640 1 Processor -- start 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.722+0000 7fbc19a1e640 1 -- start start 2023-11-24T08:16:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 0x7fbc141f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc141f7300 0x7fbc140c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc140c9e20 con 0x7fbc140c9970 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc140ca4c0 con 0x7fbc00004d00 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc19a1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc141f45d0 con 0x7fbc141f7300 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33230/0 (socket says 172.21.15.107:33230) 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 -- 172.21.15.107:0/2194010686 learned_addr learned my addr 172.21.15.107:0/2194010686 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc12ffd640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 0x7fbc141f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc127fc640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc141f7300 0x7fbc140c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 -- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc141f7300 msgr2=0x7fbc140c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc141f7300 0x7fbc140c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 -- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 msgr2=0x7fbc141f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 0x7fbc141f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc137fe640 1 -- 172.21.15.107:0/2194010686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc141f48b0 con 0x7fbc140c9970 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.723+0000 7fbc12ffd640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 0x7fbc141f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc137fe640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c037040 tx=0x7fbc0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c03e850 con 0x7fbc140c9970 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c03e9b0 con 0x7fbc140c9970 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c03ed60 con 0x7fbc140c9970 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc141f4b90 con 0x7fbc140c9970 2023-11-24T08:16:28.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.724+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1415e970 con 0x7fbc140c9970 2023-11-24T08:16:28.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.725+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc141f4d20 con 0x7fbc140c9970 2023-11-24T08:16:28.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.726+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 30) v1 ==== 91169+0+0 (secure 0 0 0) 0x7fbc0c034440 con 0x7fbc140c9970 2023-11-24T08:16:28.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.727+0000 7fbc0bfff640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 0x7fbbf406e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:28.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.727+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc0c000e40 con 0x7fbc140c9970 2023-11-24T08:16:28.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.727+0000 7fbc12ffd640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 0x7fbbf406e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:28.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.728+0000 7fbc12ffd640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 0x7fbbf406e890 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbc04002230 tx=0x7fbc040012d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:28.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.732+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc0c0766c0 con 0x7fbc140c9970 2023-11-24T08:16:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.922+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fbc1404e8b0 con 0x7fbc140c9970 2023-11-24T08:16:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.930+0000 7fbc0bfff640 1 -- 172.21.15.107:0/2194010686 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v40)=0 v40) v1 ==== 130+0+0 (secure 0 0 0) 0x7fbc0c0390c0 con 0x7fbc140c9970 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 msgr2=0x7fbbf406e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 0x7fbbf406e890 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbc04002230 tx=0x7fbc040012d0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 msgr2=0x7fbc1414c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c037040 tx=0x7fbc0c004730 comp rx=0 tx=0).stop 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 shutdown_connections 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbbf406c3d0 0x7fbbf406e890 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc140c9970 0x7fbc1414c2d0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc141f7300 0x7fbc140c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 --2- 172.21.15.107:0/2194010686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00004d00 0x7fbc141f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 >> 172.21.15.107:0/2194010686 conn(0x7fbc1408af60 msgr2=0x7fbc140b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 shutdown_connections 2023-11-24T08:16:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:28.933+0000 7fbc19a1e640 1 -- 172.21.15.107:0/2194010686 wait complete. 2023-11-24T08:16:28.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol6903 subvol_snap5787 subvol_clone9276 2023-11-24T08:16:28.965 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:28.956+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:28.965 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:28.956+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:28.965 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:28.956+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:28.965 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:28.956+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:28.966 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:28.956+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.158+0000 7fe5bee9b640 1 Processor -- start 2023-11-24T08:16:29.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.159+0000 7fe5bee9b640 1 -- start start 2023-11-24T08:16:29.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.160+0000 7fe5bee9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b80c59b0 0x7fe5b8166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.160+0000 7fe5bee9b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b80c5dc0 con 0x7fe5b815b440 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.160+0000 7fe5bee9b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b815eea0 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.160+0000 7fe5bee9b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b815b820 con 0x7fe5b80c59b0 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bcc37640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5b815b440 0x7fe5b80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52290/0 (socket says 172.21.15.107:52290) 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bcc37640 1 -- 172.21.15.107:0/1981786239 learned_addr learned my addr 172.21.15.107:0/1981786239 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bd438640 1 --2- 172.21.15.107:0/1981786239 >> v2:172.21.15.107:3300/0 conn(0x7fe5b80c59b0 0x7fe5b8166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5b77fe640 1 -- 172.21.15.107:0/1981786239 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328935942 0 0) 0x7fe5b80c5dc0 con 0x7fe5b815b440 2023-11-24T08:16:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5b77fe640 1 -- 172.21.15.107:0/1981786239 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a8003520 con 0x7fe5b815b440 2023-11-24T08:16:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bd438640 1 -- 172.21.15.107:0/1981786239 >> v1:172.21.15.120:6789/0 conn(0x7fe5b80cc0f0 legacy=0x7fe5b80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bd438640 1 -- 172.21.15.107:0/1981786239 >> v1:172.21.15.107:6789/0 conn(0x7fe5b815b440 legacy=0x7fe5b80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.161+0000 7fe5bd438640 1 -- 172.21.15.107:0/1981786239 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b806d600 con 0x7fe5b80c59b0 2023-11-24T08:16:29.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.162+0000 7fe5bd438640 1 --2- 172.21.15.107:0/1981786239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8166230 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009d90 tx=0x7fe5b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1615cb078931def9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5b77fe640 1 -- 172.21.15.107:0/1981786239 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b002cc60 con 0x7fe5b80c59b0 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5b77fe640 1 -- 172.21.15.107:0/1981786239 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b002cdc0 con 0x7fe5b80c59b0 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5b77fe640 1 -- 172.21.15.107:0/1981786239 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0034640 con 0x7fe5b80c59b0 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5bee9b640 1 -- 172.21.15.107:0/1981786239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80c59b0 msgr2=0x7fe5b8166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/1981786239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8166230 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009d90 tx=0x7fe5b002beb0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5bee9b640 1 -- 172.21.15.107:0/1981786239 shutdown_connections 2023-11-24T08:16:29.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/1981786239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8166230 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.163+0000 7fe5bee9b640 1 -- 172.21.15.107:0/1981786239 >> 172.21.15.107:0/1981786239 conn(0x7fe5b80b0c50 msgr2=0x7fe5b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:29.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.164+0000 7fe5bee9b640 1 -- 172.21.15.107:0/1981786239 shutdown_connections 2023-11-24T08:16:29.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.164+0000 7fe5bee9b640 1 -- 172.21.15.107:0/1981786239 wait complete. 2023-11-24T08:16:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.165+0000 7fe5bee9b640 1 Processor -- start 2023-11-24T08:16:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.165+0000 7fe5bee9b640 1 -- start start 2023-11-24T08:16:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815b440 0x7fe5b80ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b8152320 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b8157720 con 0x7fe5b80c59b0 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.166+0000 7fe5bee9b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b8157440 con 0x7fe5b815b440 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33278/0 (socket says 172.21.15.107:33278) 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 -- 172.21.15.107:0/204400119 learned_addr learned my addr 172.21.15.107:0/204400119 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5bcc37640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5bd438640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815b440 0x7fe5b80ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 -- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815b440 msgr2=0x7fe5b80ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815b440 0x7fe5b80ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 -- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b80c59b0 msgr2=0x7fe5b8151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.167+0000 7fe5b7fff640 1 -- 172.21.15.107:0/204400119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0009a70 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5b7fff640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c400 tx=0x7fe5a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40052d0 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4005450 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4007260 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b8158b60 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.168+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b8159120 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.169+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b804e8b0 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.170+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 30) v1 ==== 91169+0+0 (secure 0 0 0) 0x7fe5a40073c0 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.171+0000 7fe5b57fa640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 0x7fe59c06e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.171+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5a4087b10 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.171+0000 7fe5bcc37640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 0x7fe59c06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.172+0000 7fe5bcc37640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 0x7fe59c06e890 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0011e0 tx=0x7fe5ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.176+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5a4054270 con 0x7fe5b80cc0f0 2023-11-24T08:16:29.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.397+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol6903", "snap_name": "subvol_snap5787", "target_sub_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5b815f3d0 con 0x7fe59c06c3d0 2023-11-24T08:16:29.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.535+0000 7fe5b57fa640 1 -- 172.21.15.107:0/204400119 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe5ac001c40 con 0x7fe59c06c3d0 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 msgr2=0x7fe59c06e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 0x7fe59c06e890 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0011e0 tx=0x7fe5ac009040 comp rx=0 tx=0).stop 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 msgr2=0x7fe5b81569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c400 tx=0x7fe5a400c8c0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 shutdown_connections 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe59c06c3d0 0x7fe59c06e890 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5b815b440 0x7fe5b80ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b80cc0f0 0x7fe5b81569f0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 --2- 172.21.15.107:0/204400119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5b80c59b0 0x7fe5b8151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.538+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 >> 172.21.15.107:0/204400119 conn(0x7fe5b80b0c50 msgr2=0x7fe5b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.539+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 shutdown_connections 2023-11-24T08:16:29.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.539+0000 7fe5bee9b640 1 -- 172.21.15.107:0/204400119 wait complete. 2023-11-24T08:16:29.543 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.542+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.543 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.542+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.543 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.542+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.544 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.542+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.544 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.542+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol6903 subvol_snap5787 2023-11-24T08:16:29.563 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.561+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.564 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.561+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.564 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.561+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.564 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.561+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.564 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.561+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:16:29.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.748+0000 7ff2d631d640 1 Processor -- start 2023-11-24T08:16:29.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.748+0000 7ff2d631d640 1 -- start start 2023-11-24T08:16:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.749+0000 7ff2d631d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2d00cf450 0x7ff2d00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.749+0000 7ff2d631d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.749+0000 7ff2d631d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00cece0 con 0x7ff2d015c7f0 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.749+0000 7ff2d631d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d00cfe40 con 0x7ff2d00cf450 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.749+0000 7ff2d631d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d0156010 con 0x7ff2d00cdfb0 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cf7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2d00cf450 0x7ff2d00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff2d015c7f0 0x7ff2d006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52306/0 (socket says 172.21.15.107:52306) 2023-11-24T08:16:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cffff640 1 -- 172.21.15.107:0/2821055603 learned_addr learned my addr 172.21.15.107:0/2821055603 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2d48ba640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cf7fe640 1 -- 172.21.15.107:0/2821055603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 msgr2=0x7ff2d00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cf7fe640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cf7fe640 1 -- 172.21.15.107:0/2821055603 >> v1:172.21.15.107:6789/0 conn(0x7ff2d015c7f0 legacy=0x7ff2d006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2cf7fe640 1 -- 172.21.15.107:0/2821055603 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d006d040 con 0x7ff2d00cf450 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.750+0000 7ff2ceffd640 1 -- 172.21.15.107:0/2821055603 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860722168 0 0) 0x7ff2d00cece0 con 0x7ff2d015c7f0 2023-11-24T08:16:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.751+0000 7ff2cf7fe640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d00cf450 0x7ff2d00cf900 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40097a0 tx=0x7ff2c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db63ea858bfbcee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.751+0000 7ff2ceffd640 1 -- 172.21.15.107:0/2821055603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4031830 con 0x7ff2d00cf450 2023-11-24T08:16:29.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2ceffd640 1 -- 172.21.15.107:0/2821055603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c4031990 con 0x7ff2d00cf450 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2ceffd640 1 -- 172.21.15.107:0/2821055603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4031d20 con 0x7ff2d00cf450 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 -- 172.21.15.107:0/2821055603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d00cf450 msgr2=0x7ff2d00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d00cf450 0x7ff2d00cf900 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40097a0 tx=0x7ff2c402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 -- 172.21.15.107:0/2821055603 shutdown_connections 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 --2- 172.21.15.107:0/2821055603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d00cf450 0x7ff2d00cf900 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 -- 172.21.15.107:0/2821055603 >> 172.21.15.107:0/2821055603 conn(0x7ff2d006a920 msgr2=0x7ff2d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 -- 172.21.15.107:0/2821055603 shutdown_connections 2023-11-24T08:16:29.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.752+0000 7ff2d631d640 1 -- 172.21.15.107:0/2821055603 wait complete. 2023-11-24T08:16:29.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.754+0000 7ff2d631d640 1 Processor -- start 2023-11-24T08:16:29.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.754+0000 7ff2d631d640 1 -- start start 2023-11-24T08:16:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d00cf450 0x7ff2d01f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d015c7f0 0x7ff2d015df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d015e460 con 0x7ff2d015c7f0 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d00cb050 con 0x7ff2d00cf450 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d631d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d0153960 con 0x7ff2d00cdfb0 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d48ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d015c7f0 0x7ff2d015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d48ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d015c7f0 0x7ff2d015df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33346/0 (socket says 172.21.15.107:33346) 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2d48ba640 1 -- 172.21.15.107:0/3271176377 learned_addr learned my addr 172.21.15.107:0/3271176377 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.755+0000 7ff2cffff640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2cf7fe640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d00cf450 0x7ff2d01f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 -- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 msgr2=0x7ff2d00ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2d00cdfb0 0x7ff2d00ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 -- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d00cf450 msgr2=0x7ff2d01f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d00cf450 0x7ff2d01f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 -- 172.21.15.107:0/3271176377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c4009400 con 0x7ff2d015c7f0 2023-11-24T08:16:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d48ba640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d015c7f0 0x7ff2d015df20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800cd20 tx=0x7ff2c800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8003b90 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c8003cf0 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8004240 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d631d640 1 -- 172.21.15.107:0/3271176377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d0153c40 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.756+0000 7ff2d631d640 1 -- 172.21.15.107:0/3271176377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d0154200 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.757+0000 7ff2d631d640 1 -- 172.21.15.107:0/3271176377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d01543f0 con 0x7ff2d015c7f0 2023-11-24T08:16:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.758+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 30) v1 ==== 91169+0+0 (secure 0 0 0) 0x7ff2c80043a0 con 0x7ff2d015c7f0 2023-11-24T08:16:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.759+0000 7ff2ccff9640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2bc06c3a0 0x7ff2bc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.759+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2c8012070 con 0x7ff2d015c7f0 2023-11-24T08:16:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.759+0000 7ff2cffff640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2bc06c3a0 0x7ff2bc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.760+0000 7ff2cffff640 1 --2- 172.21.15.107:0/3271176377 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2bc06c3a0 0x7ff2bc06e860 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0000f60 tx=0x7ff2c0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.763+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2c8055780 con 0x7ff2d015c7f0 2023-11-24T08:16:29.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.971+0000 7ff2d631d640 1 -- 172.21.15.107:0/3271176377 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol6903", "snap_name": "subvol_snap5787", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2d00ced50 con 0x7ff2bc06c3a0 2023-11-24T08:16:29.992 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:16:29.990+0000 7f376379e640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'subvol_snap5787' has pending clones 2023-11-24T08:16:29.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:29.991+0000 7ff2ccff9640 1 -- 172.21.15.107:0/3271176377 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: -11 snapshot 'subvol_snap5787' has pending clones) v1 ==== 53+0+0 (secure 0 0 0) 0x7ff2d00ced50 con 0x7ff2bc06c3a0 2023-11-24T08:16:29.993 INFO:teuthology.orchestra.run.smithi107.stderr:Error EAGAIN: snapshot 'subvol_snap5787' has pending clones 2023-11-24T08:16:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 11 2023-11-24T08:16:30.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9276 2023-11-24T08:16:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.218+0000 7f966bfff640 1 Processor -- start 2023-11-24T08:16:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.218+0000 7f966bfff640 1 -- start start 2023-11-24T08:16:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f966c0b0b40 0x7f966c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f966c15ded0 0x7f966c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966bfff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c06d430 con 0x7f966c15a1c0 2023-11-24T08:16:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966bfff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966c15ec50 con 0x7f966c0b0b40 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966bfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966c06cb00 con 0x7f966c15ded0 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f966c0b0b40 0x7f966c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f966c0b0b40 0x7f966c0b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33358/0 (socket says 172.21.15.107:33358) 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 -- 172.21.15.107:0/2979523693 learned_addr learned my addr 172.21.15.107:0/2979523693 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966b7fe640 1 --2- 172.21.15.107:0/2979523693 >> v2:172.21.15.120:3300/0 conn(0x7f966c15ded0 0x7f966c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 -- 172.21.15.107:0/2979523693 >> v2:172.21.15.120:3300/0 conn(0x7f966c15ded0 msgr2=0x7f966c15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 --2- 172.21.15.107:0/2979523693 >> v2:172.21.15.120:3300/0 conn(0x7f966c15ded0 0x7f966c15e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 -- 172.21.15.107:0/2979523693 >> v1:172.21.15.120:6789/0 conn(0x7f966c15a1c0 legacy=0x7f966c06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.220+0000 7f966a7fc640 1 -- 172.21.15.107:0/2979523693 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0b14e0 con 0x7f966c0b0b40 2023-11-24T08:16:30.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f966a7fc640 1 --2- 172.21.15.107:0/2979523693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c0b0b40 0x7f966c0b0fa0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f966c0c9ad0 tx=0x7f966002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83ba76847eba7e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:30.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f9669ffb640 1 -- 172.21.15.107:0/2979523693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660031870 con 0x7f966c0b0b40 2023-11-24T08:16:30.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f9669ffb640 1 -- 172.21.15.107:0/2979523693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96600319d0 con 0x7f966c0b0b40 2023-11-24T08:16:30.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f9669ffb640 1 -- 172.21.15.107:0/2979523693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660031d60 con 0x7f966c0b0b40 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f966bfff640 1 -- 172.21.15.107:0/2979523693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c0b0b40 msgr2=0x7f966c0b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.222+0000 7f966bfff640 1 --2- 172.21.15.107:0/2979523693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c0b0b40 0x7f966c0b0fa0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f966c0c9ad0 tx=0x7f966002b910 comp rx=0 tx=0).stop 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 -- 172.21.15.107:0/2979523693 shutdown_connections 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 --2- 172.21.15.107:0/2979523693 >> v2:172.21.15.120:3300/0 conn(0x7f966c15ded0 0x7f966c15e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 --2- 172.21.15.107:0/2979523693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c0b0b40 0x7f966c0b0fa0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 -- 172.21.15.107:0/2979523693 >> 172.21.15.107:0/2979523693 conn(0x7f966c0cbb40 msgr2=0x7f966c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 -- 172.21.15.107:0/2979523693 shutdown_connections 2023-11-24T08:16:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.223+0000 7f966bfff640 1 -- 172.21.15.107:0/2979523693 wait complete. 2023-11-24T08:16:30.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.224+0000 7f966bfff640 1 Processor -- start 2023-11-24T08:16:30.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.225+0000 7f966bfff640 1 -- start start 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.225+0000 7f966bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 0x7f966c152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.225+0000 7f966bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c15a1c0 0x7f966c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.225+0000 7f966bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.225+0000 7f966bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c15d5c0 con 0x7f966c15ded0 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966bfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c15d130 con 0x7f966c0b0b40 2023-11-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c152bd0 con 0x7f966c15a1c0 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33362/0 (socket says 172.21.15.107:33362) 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 -- 172.21.15.107:0/1129124631 learned_addr learned my addr 172.21.15.107:0/1129124631 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966affd640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 0x7f966c152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 -- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c15a1c0 msgr2=0x7f966c15bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c15a1c0 0x7f966c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 -- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 msgr2=0x7f966c152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 0x7f966c152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966b7fe640 1 -- 172.21.15.107:0/1129124631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600094d0 con 0x7f966c15ded0 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.226+0000 7f966affd640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 0x7f966c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f966b7fe640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f9664007e60 tx=0x7f966400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664014420 con 0x7f966c15ded0 2023-11-24T08:16:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96640145a0 con 0x7f966c15ded0 2023-11-24T08:16:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664014950 con 0x7f966c15ded0 2023-11-24T08:16:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c15ea20 con 0x7f966c15ded0 2023-11-24T08:16:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.227+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c15c800 con 0x7f966c15ded0 2023-11-24T08:16:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.228+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c15d2c0 con 0x7f966c15ded0 2023-11-24T08:16:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.229+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 30) v1 ==== 91169+0+0 (secure 0 0 0) 0x7f9664014ab0 con 0x7f966c15ded0 2023-11-24T08:16:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.230+0000 7f96537fe640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 0x7f965806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.230+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9664089040 con 0x7f966c15ded0 2023-11-24T08:16:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.230+0000 7f966affd640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 0x7f965806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.231+0000 7f966affd640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 0x7f965806e890 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f965c003eb0 tx=0x7f965c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.235+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9664055740 con 0x7f966c15ded0 2023-11-24T08:16:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.448+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f966c15e8a0 con 0x7f965806c3d0 2023-11-24T08:16:30.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.460+0000 7f96537fe640 1 -- 172.21.15.107:0/1129124631 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f965c005030 con 0x7f965806c3d0 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol6903", 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5787" 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:30.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:30.465 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 msgr2=0x7f965806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 0x7f965806e890 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f965c003eb0 tx=0x7f965c009040 comp rx=0 tx=0).stop 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 msgr2=0x7f966c0c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f9664007e60 tx=0x7f966400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 shutdown_connections 2023-11-24T08:16:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f965806c3d0 0x7f965806e890 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f966c15ded0 0x7f966c0c79e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c15a1c0 0x7f966c15bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 --2- 172.21.15.107:0/1129124631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c0b0b40 0x7f966c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 >> 172.21.15.107:0/1129124631 conn(0x7f966c0cbb40 msgr2=0x7f966c0b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 shutdown_connections 2023-11-24T08:16:30.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:30.463+0000 7f966bfff640 1 -- 172.21.15.107:0/1129124631 wait complete. 2023-11-24T08:16:31.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9276 2023-11-24T08:16:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.708+0000 7f6a0e01e640 1 Processor -- start 2023-11-24T08:16:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.708+0000 7f6a0e01e640 1 -- start start 2023-11-24T08:16:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a0e01e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a081660b0 0x7f6a08162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a0e01e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a08162540 0x7f6a080c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a0e01e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a08166510 con 0x7f6a080ccbd0 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a0e01e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a081655f0 con 0x7f6a08162540 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a0e01e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a080cc380 con 0x7f6a081660b0 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a077fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6a080ccbd0 0x7f6a080ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52308/0 (socket says 172.21.15.107:52308) 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a077fe640 1 -- 172.21.15.107:0/2700583384 learned_addr learned my addr 172.21.15.107:0/2700583384 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.710+0000 7f6a07fff640 1 --2- 172.21.15.107:0/2700583384 >> v2:172.21.15.107:3300/0 conn(0x7f6a08162540 0x7f6a080c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f69fffff640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a081660b0 0x7f6a08162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a06ffd640 1 -- 172.21.15.107:0/2700583384 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834392560 0 0) 0x7f69f8001110 con 0x7f6a080ccbd0 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a06ffd640 1 -- 172.21.15.107:0/2700583384 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f4003520 con 0x7f6a080ccbd0 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a07fff640 1 -- 172.21.15.107:0/2700583384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a081660b0 msgr2=0x7f6a08162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a07fff640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a081660b0 0x7f6a08162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a07fff640 1 -- 172.21.15.107:0/2700583384 >> v1:172.21.15.107:6789/0 conn(0x7f6a080ccbd0 legacy=0x7f6a080ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.711+0000 7f6a07fff640 1 -- 172.21.15.107:0/2700583384 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08162b30 con 0x7f6a08162540 2023-11-24T08:16:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.712+0000 7f6a07fff640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a08162540 0x7f6a080c9650 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009dc0 tx=0x7f6a0002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e22cb1ec40056ff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.712+0000 7f6a06ffd640 1 -- 172.21.15.107:0/2700583384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0002cc40 con 0x7f6a08162540 2023-11-24T08:16:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.712+0000 7f6a06ffd640 1 -- 172.21.15.107:0/2700583384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0002cda0 con 0x7f6a08162540 2023-11-24T08:16:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.712+0000 7f6a06ffd640 1 -- 172.21.15.107:0/2700583384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00034660 con 0x7f6a08162540 2023-11-24T08:16:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 -- 172.21.15.107:0/2700583384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a08162540 msgr2=0x7f6a080c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a08162540 0x7f6a080c9650 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009dc0 tx=0x7f6a0002be90 comp rx=0 tx=0).stop 2023-11-24T08:16:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 -- 172.21.15.107:0/2700583384 shutdown_connections 2023-11-24T08:16:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a08162540 0x7f6a080c9650 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/2700583384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a081660b0 0x7f6a08162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 -- 172.21.15.107:0/2700583384 >> 172.21.15.107:0/2700583384 conn(0x7f6a080b0c50 msgr2=0x7f6a080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 -- 172.21.15.107:0/2700583384 shutdown_connections 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.713+0000 7f6a0e01e640 1 -- 172.21.15.107:0/2700583384 wait complete. 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.715+0000 7f6a0e01e640 1 Processor -- start 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.715+0000 7f6a0e01e640 1 -- start start 2023-11-24T08:16:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a080ccbd0 0x7f6a080b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a08162540 0x7f6a08160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a080728a0 con 0x7f6a081660b0 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08163f90 con 0x7f6a08162540 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a0e01e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08164270 con 0x7f6a080ccbd0 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33382/0 (socket says 172.21.15.107:33382) 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.716+0000 7f6a07fff640 1 -- 172.21.15.107:0/3538609194 learned_addr learned my addr 172.21.15.107:0/3538609194 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f69fffff640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a08162540 0x7f6a08160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a077fe640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a080ccbd0 0x7f6a080b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 -- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a080ccbd0 msgr2=0x7f6a080b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a080ccbd0 0x7f6a080b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 -- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a08162540 msgr2=0x7f6a08160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a08162540 0x7f6a08160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 -- 172.21.15.107:0/3538609194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00009a70 con 0x7f6a081660b0 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a07fff640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009ee0 tx=0x7f6a00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00040650 con 0x7f6a081660b0 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a000407b0 con 0x7f6a081660b0 2023-11-24T08:16:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.717+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08164550 con 0x7f6a081660b0 2023-11-24T08:16:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.718+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00040b60 con 0x7f6a081660b0 2023-11-24T08:16:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.718+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a08164b10 con 0x7f6a081660b0 2023-11-24T08:16:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.719+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0806e2a0 con 0x7f6a081660b0 2023-11-24T08:16:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.720+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f6a00040cc0 con 0x7f6a081660b0 2023-11-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.720+0000 7f6a04ff9640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 0x7f69e806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.721+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a000ab130 con 0x7f6a081660b0 2023-11-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.721+0000 7f6a077fe640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 0x7f69e806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.721+0000 7f6a077fe640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 0x7f69e806ea10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f69f8001370 tx=0x7f69f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.725+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a0806e2a0 con 0x7f6a081660b0 2023-11-24T08:16:31.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:31.950+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a08165b90 con 0x7f69e806c550 2023-11-24T08:16:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.071+0000 7f6a04ff9640 1 -- 172.21.15.107:0/3538609194 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f69f8001c50 con 0x7f69e806c550 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol6903", 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5787" 2023-11-24T08:16:35.074 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:35.075 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:35.075 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 msgr2=0x7f69e806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 0x7f69e806ea10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f69f8001370 tx=0x7f69f8009040 comp rx=0 tx=0).stop 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 msgr2=0x7f6a08157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009ee0 tx=0x7f6a00004a40 comp rx=0 tx=0).stop 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 shutdown_connections 2023-11-24T08:16:35.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f69e806c550 0x7f69e806ea10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a081660b0 0x7f6a08157710 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a08162540 0x7f6a08160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 --2- 172.21.15.107:0/3538609194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a080ccbd0 0x7f6a080b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 >> 172.21.15.107:0/3538609194 conn(0x7f6a080b0c50 msgr2=0x7f6a080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 shutdown_connections 2023-11-24T08:16:35.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:35.075+0000 7f6a0e01e640 1 -- 172.21.15.107:0/3538609194 wait complete. 2023-11-24T08:16:36.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9276 2023-11-24T08:16:36.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.321+0000 7f88e7543640 1 Processor -- start 2023-11-24T08:16:36.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.321+0000 7f88e7543640 1 -- start start 2023-11-24T08:16:36.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.322+0000 7f88e7543640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e015b440 0x7f88e00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.322+0000 7f88e7543640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f88e00c59b0 0x7f88e006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.322+0000 7f88e7543640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00c5dc0 con 0x7f88e00cbf20 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.322+0000 7f88e7543640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e00cc3d0 con 0x7f88e015b440 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.322+0000 7f88e7543640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e015b820 con 0x7f88e00c59b0 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e015b440 0x7f88e00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e015b440 0x7f88e00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33390/0 (socket says 172.21.15.107:33390) 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 -- 172.21.15.107:0/3167683262 learned_addr learned my addr 172.21.15.107:0/3167683262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e5ae0640 1 --2- 172.21.15.107:0/3167683262 >> v2:172.21.15.120:3300/0 conn(0x7f88e00c59b0 0x7f88e006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 -- 172.21.15.107:0/3167683262 >> v2:172.21.15.120:3300/0 conn(0x7f88e00c59b0 msgr2=0x7f88e006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 --2- 172.21.15.107:0/3167683262 >> v2:172.21.15.120:3300/0 conn(0x7f88e00c59b0 0x7f88e006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 -- 172.21.15.107:0/3167683262 >> v1:172.21.15.120:6789/0 conn(0x7f88e00cbf20 legacy=0x7f88e00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.323+0000 7f88e52df640 1 -- 172.21.15.107:0/3167683262 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e015eea0 con 0x7f88e015b440 2023-11-24T08:16:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.324+0000 7f88e52df640 1 --2- 172.21.15.107:0/3167683262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e015b440 0x7f88e00cb9e0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f88e00cb910 tx=0x7f88dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a47710fe7a8501 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:36.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.325+0000 7f88d7fff640 1 -- 172.21.15.107:0/3167683262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0317e0 con 0x7f88e015b440 2023-11-24T08:16:36.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.325+0000 7f88d7fff640 1 -- 172.21.15.107:0/3167683262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc031940 con 0x7f88e015b440 2023-11-24T08:16:36.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.325+0000 7f88d7fff640 1 -- 172.21.15.107:0/3167683262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc031cd0 con 0x7f88e015b440 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 -- 172.21.15.107:0/3167683262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e015b440 msgr2=0x7f88e00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 --2- 172.21.15.107:0/3167683262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e015b440 0x7f88e00cb9e0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f88e00cb910 tx=0x7f88dc02b880 comp rx=0 tx=0).stop 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 -- 172.21.15.107:0/3167683262 shutdown_connections 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 --2- 172.21.15.107:0/3167683262 >> v2:172.21.15.120:3300/0 conn(0x7f88e00c59b0 0x7f88e006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 --2- 172.21.15.107:0/3167683262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e015b440 0x7f88e00cb9e0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 -- 172.21.15.107:0/3167683262 >> 172.21.15.107:0/3167683262 conn(0x7f88e00b0c50 msgr2=0x7f88e00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 -- 172.21.15.107:0/3167683262 shutdown_connections 2023-11-24T08:16:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.326+0000 7f88e7543640 1 -- 172.21.15.107:0/3167683262 wait complete. 2023-11-24T08:16:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.328+0000 7f88e7543640 1 Processor -- start 2023-11-24T08:16:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.328+0000 7f88e7543640 1 -- start start 2023-11-24T08:16:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 0x7f88e01f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 0x7f88e01f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e015bee0 con 0x7f88e00c59b0 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e015c190 con 0x7f88e015b440 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.329+0000 7f88e7543640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e015c470 con 0x7f88e00cbf20 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33406/0 (socket says 172.21.15.107:33406) 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 -- 172.21.15.107:0/3234296342 learned_addr learned my addr 172.21.15.107:0/3234296342 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e5ae0640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 0x7f88e01f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e4ade640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 0x7f88e01f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 -- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 msgr2=0x7f88e01f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 0x7f88e01f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 -- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 msgr2=0x7f88e01f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 0x7f88e01f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e52df640 1 -- 172.21.15.107:0/3234296342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0094d0 con 0x7f88e00c59b0 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e5ae0640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 0x7f88e01f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:36.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.330+0000 7f88e4ade640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 0x7f88e01f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88e52df640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f88dc02b8c0 tx=0x7f88dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc032880 con 0x7f88e00c59b0 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc0329e0 con 0x7f88e00c59b0 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc032d90 con 0x7f88e00c59b0 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e015c750 con 0x7f88e00c59b0 2023-11-24T08:16:36.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.331+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e015cd10 con 0x7f88e00c59b0 2023-11-24T08:16:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.332+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e004e8b0 con 0x7f88e00c59b0 2023-11-24T08:16:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.333+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f88dc04e450 con 0x7f88e00c59b0 2023-11-24T08:16:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.334+0000 7f88d5ffb640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 0x7f88c406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.334+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88dc0aa440 con 0x7f88e00c59b0 2023-11-24T08:16:36.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.334+0000 7f88e4ade640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 0x7f88c406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:36.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.335+0000 7f88e4ade640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 0x7f88c406ea10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f88d0007fd0 tx=0x7f88d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:36.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.339+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88dc0770a0 con 0x7f88e00c59b0 2023-11-24T08:16:36.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.607+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f88e015f3d0 con 0x7f88c406c550 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.619+0000 7f88d5ffb640 1 -- 172.21.15.107:0/3234296342 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f88d00012c0 con 0x7f88c406c550 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:16:36.621 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:16:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol6903", 2023-11-24T08:16:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5787" 2023-11-24T08:16:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:36.622 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.621+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 msgr2=0x7f88c406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.621+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 0x7f88c406ea10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f88d0007fd0 tx=0x7f88d0009040 comp rx=0 tx=0).stop 2023-11-24T08:16:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.621+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 msgr2=0x7f88e015b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.621+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f88dc02b8c0 tx=0x7f88dc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 shutdown_connections 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88c406c550 0x7f88c406ea10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e015b440 0x7f88e01f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e00cbf20 0x7f88e01f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 --2- 172.21.15.107:0/3234296342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e00c59b0 0x7f88e015b9a0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 >> 172.21.15.107:0/3234296342 conn(0x7f88e00b0c50 msgr2=0x7f88e00cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 shutdown_connections 2023-11-24T08:16:36.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:36.622+0000 7f88e7543640 1 -- 172.21.15.107:0/3234296342 wait complete. 2023-11-24T08:16:37.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9276 2023-11-24T08:16:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.873+0000 7fcb5065c640 1 Processor -- start 2023-11-24T08:16:37.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.873+0000 7fcb5065c640 1 -- start start 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb5065c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb480c49f0 0x7fcb4815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb5065c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4815ec90 con 0x7fcb48072160 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb5065c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480cd530 con 0x7fcb4806de10 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb5065c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb480c4ed0 con 0x7fcb480c49f0 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb4ebf9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb480c49f0 0x7fcb4815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb4ebf9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb480c49f0 0x7fcb4815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33412/0 (socket says 172.21.15.107:33412) 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb4ebf9640 1 -- 172.21.15.107:0/2947505436 learned_addr learned my addr 172.21.15.107:0/2947505436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.875+0000 7fcb4e3f8640 1 --1- 172.21.15.107:0/2947505436 >> v1:172.21.15.107:6789/0 conn(0x7fcb48072160 0x7fcb4806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52324/0 (socket says 172.21.15.107:52324) 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.876+0000 7fcb4ebf9640 1 -- 172.21.15.107:0/2947505436 >> v1:172.21.15.120:6789/0 conn(0x7fcb4806de10 legacy=0x7fcb480705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:37.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.876+0000 7fcb4ebf9640 1 -- 172.21.15.107:0/2947505436 >> v1:172.21.15.107:6789/0 conn(0x7fcb48072160 legacy=0x7fcb4806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.876+0000 7fcb4ebf9640 1 -- 172.21.15.107:0/2947505436 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480c51b0 con 0x7fcb480c49f0 2023-11-24T08:16:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.876+0000 7fcb4d3f6640 1 -- 172.21.15.107:0/2947505436 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119291291 0 0) 0x7fcb44001110 con 0x7fcb48072160 2023-11-24T08:16:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.877+0000 7fcb4ebf9640 1 --2- 172.21.15.107:0/2947505436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb480c49f0 0x7fcb4815e750 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d46690d3e4a2ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.877+0000 7fcb4d3f6640 1 -- 172.21.15.107:0/2947505436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4002cbb0 con 0x7fcb480c49f0 2023-11-24T08:16:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.877+0000 7fcb4d3f6640 1 -- 172.21.15.107:0/2947505436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4002cd10 con 0x7fcb480c49f0 2023-11-24T08:16:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.877+0000 7fcb4d3f6640 1 -- 172.21.15.107:0/2947505436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40034610 con 0x7fcb480c49f0 2023-11-24T08:16:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.878+0000 7fcb5065c640 1 -- 172.21.15.107:0/2947505436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb480c49f0 msgr2=0x7fcb4815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.878+0000 7fcb5065c640 1 --2- 172.21.15.107:0/2947505436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb480c49f0 0x7fcb4815e750 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002c010 comp rx=0 tx=0).stop 2023-11-24T08:16:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.879+0000 7fcb5065c640 1 -- 172.21.15.107:0/2947505436 shutdown_connections 2023-11-24T08:16:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.879+0000 7fcb5065c640 1 --2- 172.21.15.107:0/2947505436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb480c49f0 0x7fcb4815e750 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.879+0000 7fcb5065c640 1 -- 172.21.15.107:0/2947505436 >> 172.21.15.107:0/2947505436 conn(0x7fcb480b0c50 msgr2=0x7fcb480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.879+0000 7fcb5065c640 1 -- 172.21.15.107:0/2947505436 shutdown_connections 2023-11-24T08:16:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.879+0000 7fcb5065c640 1 -- 172.21.15.107:0/2947505436 wait complete. 2023-11-24T08:16:37.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.881+0000 7fcb5065c640 1 Processor -- start 2023-11-24T08:16:37.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.882+0000 7fcb5065c640 1 -- start start 2023-11-24T08:16:37.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:37.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 0x7fcb480cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:37.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb480c49f0 0x7fcb480cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:37.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48159650 con 0x7fcb4806de10 2023-11-24T08:16:37.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815b660 con 0x7fcb480c49f0 2023-11-24T08:16:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb5065c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815b940 con 0x7fcb48072160 2023-11-24T08:16:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb4e3f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb4e3f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33428/0 (socket says 172.21.15.107:33428) 2023-11-24T08:16:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.883+0000 7fcb4e3f8640 1 -- 172.21.15.107:0/1148112922 learned_addr learned my addr 172.21.15.107:0/1148112922 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4ebf9640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb480c49f0 0x7fcb480cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 -- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 msgr2=0x7fcb480cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4dbf7640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 0x7fcb480cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 0x7fcb480cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 -- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb480c49f0 msgr2=0x7fcb480cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb480c49f0 0x7fcb480cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:37.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 -- 172.21.15.107:0/1148112922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40009b30 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4dbf7640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 0x7fcb480cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.884+0000 7fcb4e3f8640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c780 tx=0x7fcb4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.885+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440072d0 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.885+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44007450 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.885+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44001d10 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.885+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4815bc20 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.885+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4815c420 con 0x7fcb4806de10 2023-11-24T08:16:37.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.886+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4815c140 con 0x7fcb4806de10 2023-11-24T08:16:37.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.887+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7fcb440047e0 con 0x7fcb4806de10 2023-11-24T08:16:37.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.888+0000 7fcb3effd640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 0x7fcb2c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:37.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.888+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb44087310 con 0x7fcb4806de10 2023-11-24T08:16:37.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.888+0000 7fcb4dbf7640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 0x7fcb2c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:37.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.889+0000 7fcb4dbf7640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 0x7fcb2c06ea10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcb480c98f0 tx=0x7fcb38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:37.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:37.894+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb4815c140 con 0x7fcb4806de10 2023-11-24T08:16:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.130+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb4815b7d0 con 0x7fcb2c06c550 2023-11-24T08:16:38.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.938+0000 7fcb3effd640 1 -- 172.21.15.107:0/1148112922 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fcb4815b7d0 con 0x7fcb2c06c550 2023-11-24T08:16:38.941 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:16:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:16:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:16:38.941 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 msgr2=0x7fcb2c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 0x7fcb2c06ea10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcb480c98f0 tx=0x7fcb38009040 comp rx=0 tx=0).stop 2023-11-24T08:16:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 msgr2=0x7fcb48159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c780 tx=0x7fcb4400cc40 comp rx=0 tx=0).stop 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 shutdown_connections 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb2c06c550 0x7fcb2c06ea10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb480c49f0 0x7fcb480cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb48072160 0x7fcb480cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 --2- 172.21.15.107:0/1148112922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb4806de10 0x7fcb48159110 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 >> 172.21.15.107:0/1148112922 conn(0x7fcb480b0c50 msgr2=0x7fcb480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 shutdown_connections 2023-11-24T08:16:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:38.941+0000 7fcb5065c640 1 -- 172.21.15.107:0/1148112922 wait complete. 2023-11-24T08:16:38.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone9276 2023-11-24T08:16:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.175+0000 7f30749c8640 1 Processor -- start 2023-11-24T08:16:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.175+0000 7f30749c8640 1 -- start start 2023-11-24T08:16:39.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.176+0000 7f30749c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f307006e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f30749c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30700b1250 0x7f307006c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f30749c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307015ed20 con 0x7f307006e580 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f30749c8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307015b640 con 0x7f30700b1250 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f30749c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307015f8a0 con 0x7f307015b260 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30700b1250 0x7f307006c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f307006e580 0x7f30700b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56500/0 (socket says 172.21.15.107:56500) 2023-11-24T08:16:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306e59c640 1 -- 172.21.15.107:0/4186038851 learned_addr learned my addr 172.21.15.107:0/4186038851 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306ed9d640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f307006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306f59e640 1 -- 172.21.15.107:0/4186038851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 msgr2=0x7f307006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306f59e640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f307006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306f59e640 1 -- 172.21.15.107:0/4186038851 >> v1:172.21.15.107:6789/0 conn(0x7f307006e580 legacy=0x7f30700b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306f59e640 1 -- 172.21.15.107:0/4186038851 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30700abe50 con 0x7f30700b1250 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306ed9d640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f307006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:39.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.177+0000 7f306dd9b640 1 -- 172.21.15.107:0/4186038851 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780606961 0 0) 0x7f307015ed20 con 0x7f307006e580 2023-11-24T08:16:39.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.178+0000 7f306f59e640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30700b1250 0x7f307006c5b0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f3068009d90 tx=0x7f306802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8338ae066f71b1e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f306dd9b640 1 -- 172.21.15.107:0/4186038851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306802cb30 con 0x7f30700b1250 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f306dd9b640 1 -- 172.21.15.107:0/4186038851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306802cc90 con 0x7f30700b1250 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f306dd9b640 1 -- 172.21.15.107:0/4186038851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068034440 con 0x7f30700b1250 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 -- 172.21.15.107:0/4186038851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30700b1250 msgr2=0x7f307006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30700b1250 0x7f307006c5b0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f3068009d90 tx=0x7f306802bd80 comp rx=0 tx=0).stop 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 -- 172.21.15.107:0/4186038851 shutdown_connections 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30700b1250 0x7f307006c5b0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 --2- 172.21.15.107:0/4186038851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f307006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.179+0000 7f30749c8640 1 -- 172.21.15.107:0/4186038851 >> 172.21.15.107:0/4186038851 conn(0x7f30700cbb40 msgr2=0x7f30700cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.180+0000 7f30749c8640 1 -- 172.21.15.107:0/4186038851 shutdown_connections 2023-11-24T08:16:39.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.180+0000 7f30749c8640 1 -- 172.21.15.107:0/4186038851 wait complete. 2023-11-24T08:16:39.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.181+0000 7f30749c8640 1 Processor -- start 2023-11-24T08:16:39.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.181+0000 7f30749c8640 1 -- start start 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 0x7f3070099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f3070099f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307009a620 con 0x7f307006e580 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30700b2e90 con 0x7f30700b1250 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.182+0000 7f30749c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3070158bc0 con 0x7f307015b260 2023-11-24T08:16:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39212/0 (socket says 172.21.15.107:39212) 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 -- 172.21.15.107:0/996358814 learned_addr learned my addr 172.21.15.107:0/996358814 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 -- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 msgr2=0x7f3070099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f3070099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306e59c640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 0x7f3070099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 -- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 msgr2=0x7f3070099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 0x7f3070099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 -- 172.21.15.107:0/996358814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068009a70 con 0x7f307006e580 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306e59c640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 0x7f3070099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.183+0000 7f306ed9d640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f306000c820 tx=0x7f306000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.184+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060007a30 con 0x7f307006e580 2023-11-24T08:16:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.184+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3060007b90 con 0x7f307006e580 2023-11-24T08:16:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.184+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306000ce10 con 0x7f307006e580 2023-11-24T08:16:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.184+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307009a870 con 0x7f307006e580 2023-11-24T08:16:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.184+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307009ae30 con 0x7f307006e580 2023-11-24T08:16:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.185+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307004e840 con 0x7f307006e580 2023-11-24T08:16:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.186+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f30600113f0 con 0x7f307006e580 2023-11-24T08:16:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.187+0000 7f30577fe640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 0x7f305c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.187+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f30600884a0 con 0x7f307006e580 2023-11-24T08:16:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.187+0000 7f306e59c640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 0x7f305c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.188+0000 7f306e59c640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 0x7f305c06ea10 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f30700c4be0 tx=0x7f3064009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.191+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3060055050 con 0x7f307006e580 2023-11-24T08:16:39.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.412+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f307015f530 con 0x7f305c06c550 2023-11-24T08:16:39.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.418+0000 7f30577fe640 1 -- 172.21.15.107:0/996358814 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f30640012f0 con 0x7f305c06c550 2023-11-24T08:16:39.420 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7 2023-11-24T08:16:39.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 msgr2=0x7f305c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 0x7f305c06ea10 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f30700c4be0 tx=0x7f3064009040 comp rx=0 tx=0).stop 2023-11-24T08:16:39.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 msgr2=0x7f3070158680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f306000c820 tx=0x7f306000cce0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 shutdown_connections 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f305c06c550 0x7f305c06ea10 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307015b260 0x7f3070099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30700b1250 0x7f3070099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 --2- 172.21.15.107:0/996358814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f307006e580 0x7f3070158680 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 >> 172.21.15.107:0/996358814 conn(0x7f30700cbb40 msgr2=0x7f307006ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.421+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 shutdown_connections 2023-11-24T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.422+0000 7f30749c8640 1 -- 172.21.15.107:0/996358814 wait complete. 2023-11-24T08:16:39.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol6903 2023-11-24T08:16:39.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.660+0000 7fcfe95c9640 1 Processor -- start 2023-11-24T08:16:39.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.660+0000 7fcfe95c9640 1 -- start start 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe95c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcfe40b2400 0x7fcfe406ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe95c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfe406f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe95c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe41612f0 con 0x7fcfe40cfb90 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe95c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe40cff70 con 0x7fcfe40b2400 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe95c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40afb20 con 0x7fcfe40cc2b0 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.662+0000 7fcfe2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcfe40cfb90 0x7fcfe4161a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56516/0 (socket says 172.21.15.107:56516) 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/2644375979 learned_addr learned my addr 172.21.15.107:0/2644375979 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/2644375979 >> v2:172.21.15.120:3300/0 conn(0x7fcfe40b2400 0x7fcfe406ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630486578 0 0) 0x7fcfe41612f0 con 0x7fcfe40cfb90 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe37fe640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfe406f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd0003520 con 0x7fcfe40cfb90 2023-11-24T08:16:39.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198991797 0 0) 0x7fcfd0003520 con 0x7fcfe40cfb90 2023-11-24T08:16:39.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 msgr2=0x7fcfe406f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:39.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfe406f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 >> v2:172.21.15.120:3300/0 conn(0x7fcfe40b2400 msgr2=0x7fcfe406ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 --2- 172.21.15.107:0/2644375979 >> v2:172.21.15.120:3300/0 conn(0x7fcfe40b2400 0x7fcfe406ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4162170 con 0x7fcfe40cfb90 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcfd4001670 con 0x7fcfe40cfb90 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe37fe640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfe406f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 >> v1:172.21.15.107:6789/0 conn(0x7fcfe40cfb90 legacy=0x7fcfe4161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.663+0000 7fcfe27fc640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfd0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe27fc640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd00047c0 0x7fcfd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe27fc640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40cff70 con 0x7fcfd00039a0 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40afb20 con 0x7fcfd00047c0 2023-11-24T08:16:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd000b980 con 0x7fcfe40cc2b0 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfd0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 msgr2=0x7fcfd0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40cc2b0 0x7fcfd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd00047c0 0x7fcfd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd00047c0 msgr2=0x7fcfd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd00047c0 0x7fcfd0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.664+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/2644375979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0003520 con 0x7fcfd00039a0 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.665+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfd0003dc0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c270 tx=0x7fcfd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249e7c1b2f5259aa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4005ad0 con 0x7fcfd00039a0 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd4005c30 con 0x7fcfd00039a0 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe27fc640 1 -- 172.21.15.107:0/2644375979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd402c8a0 con 0x7fcfd00039a0 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 -- 172.21.15.107:0/2644375979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 msgr2=0x7fcfd0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/2644375979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfd0003dc0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c270 tx=0x7fcfd00057f0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe37fe640 1 -- 172.21.15.107:0/2644375979 reap_dead start 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 -- 172.21.15.107:0/2644375979 shutdown_connections 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 -- 172.21.15.107:0/2644375979 >> 172.21.15.107:0/2644375979 conn(0x7fcfe40999d0 msgr2=0x7fcfe40b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 -- 172.21.15.107:0/2644375979 shutdown_connections 2023-11-24T08:16:39.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.666+0000 7fcfe95c9640 1 -- 172.21.15.107:0/2644375979 wait complete. 2023-11-24T08:16:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.668+0000 7fcfe95c9640 1 Processor -- start 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.668+0000 7fcfe95c9640 1 -- start start 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 0x7fcfe40cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40b51c0 0x7fcfe4152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40b5ea0 con 0x7fcfd00039a0 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40b5bc0 con 0x7fcfe40c6940 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe95c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40c4d00 con 0x7fcfe40b51c0 2023-11-24T08:16:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39228/0 (socket says 172.21.15.107:39228) 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.669+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/752232295 learned_addr learned my addr 172.21.15.107:0/752232295 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40b51c0 msgr2=0x7fcfe4152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40b51c0 0x7fcfe4152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 msgr2=0x7fcfe40cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 0x7fcfe40cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 0x7fcfe40cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 -- 172.21.15.107:0/752232295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40c4fe0 con 0x7fcfd00039a0 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 0x7fcfe40cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.670+0000 7fcfe2ffd640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4001930 tx=0x7fcfd4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.671+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4039070 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.671+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd402ddc0 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.671+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd403c040 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.671+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe40c52c0 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.671+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe40cd120 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.672+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe40c4e40 con 0x7fcfd00039a0 2023-11-24T08:16:39.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.673+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7fcfd4032070 con 0x7fcfd00039a0 2023-11-24T08:16:39.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.674+0000 7fcfdb7fe640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 0x7fcfc406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:39.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.674+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcfd40a9420 con 0x7fcfd00039a0 2023-11-24T08:16:39.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.674+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 0x7fcfc406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:39.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.675+0000 7fcfdbfff640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 0x7fcfc406ea40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfe415c060 tx=0x7fcfcc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.679+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfe40c4e40 con 0x7fcfd00039a0 2023-11-24T08:16:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.902+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol6903", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfe406ae10 con 0x7fcfc406c580 2023-11-24T08:16:39.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.917+0000 7fcfdb7fe640 1 -- 172.21.15.107:0/752232295 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fcfcc009650 con 0x7fcfc406c580 2023-11-24T08:16:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol6903/b2d2cb3b-4626-4593-9afb-021868c54c05 2023-11-24T08:16:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.919+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 msgr2=0x7fcfc406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.919+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 0x7fcfc406ea40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfe415c060 tx=0x7fcfcc008040 comp rx=0 tx=0).stop 2023-11-24T08:16:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 msgr2=0x7fcfe415ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4001930 tx=0x7fcfd4001a00 comp rx=0 tx=0).stop 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 shutdown_connections 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfc406c580 0x7fcfc406ea40 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfe40b51c0 0x7fcfe4152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfe40c6940 0x7fcfe40cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 --2- 172.21.15.107:0/752232295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd00039a0 0x7fcfe415ddf0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 >> 172.21.15.107:0/752232295 conn(0x7fcfe40999d0 msgr2=0x7fcfe415c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 shutdown_connections 2023-11-24T08:16:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:39.920+0000 7fcfe95c9640 1 -- 172.21.15.107:0/752232295 wait complete. 2023-11-24T08:16:39.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone9276 2023-11-24T08:16:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.158+0000 7f1ee51b3640 1 Processor -- start 2023-11-24T08:16:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.158+0000 7f1ee51b3640 1 -- start start 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.159+0000 7f1ee51b3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ee015b260 0x7f1ee006e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee51b3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee006e580 0x7f1ee00b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee51b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ee00b1080 0x7f1ee00ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee51b3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00b1530 con 0x7f1ee00b1080 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee51b3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee015b640 con 0x7f1ee015b260 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee51b3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015ed20 con 0x7f1ee006e580 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ee00b1080 0x7f1ee00ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ee00b1080 0x7f1ee00ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39238/0 (socket says 172.21.15.107:39238) 2023-11-24T08:16:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 -- 172.21.15.107:0/2181629649 learned_addr learned my addr 172.21.15.107:0/2181629649 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1edffff640 1 --2- 172.21.15.107:0/2181629649 >> v2:172.21.15.120:3300/0 conn(0x7f1ee015b260 0x7f1ee006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ed7fff640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee006e580 0x7f1ee00b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 -- 172.21.15.107:0/2181629649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee006e580 msgr2=0x7f1ee00b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee006e580 0x7f1ee00b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 -- 172.21.15.107:0/2181629649 >> v2:172.21.15.120:3300/0 conn(0x7f1ee015b260 msgr2=0x7f1ee006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.160+0000 7f1ee49b2640 1 --2- 172.21.15.107:0/2181629649 >> v2:172.21.15.120:3300/0 conn(0x7f1ee015b260 0x7f1ee006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.161+0000 7f1ee49b2640 1 -- 172.21.15.107:0/2181629649 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee015f8a0 con 0x7f1ee00b1080 2023-11-24T08:16:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.162+0000 7f1ee49b2640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee00b1080 0x7f1ee00ccc90 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f1ee00d0160 tx=0x7f1ed802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27377ddebbaa888 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.162+0000 7f1edf7fe640 1 -- 172.21.15.107:0/2181629649 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed802c950 con 0x7f1ee00b1080 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.162+0000 7f1edf7fe640 1 -- 172.21.15.107:0/2181629649 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed802cab0 con 0x7f1ee00b1080 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.162+0000 7f1edf7fe640 1 -- 172.21.15.107:0/2181629649 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed802ce40 con 0x7f1ee00b1080 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 -- 172.21.15.107:0/2181629649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee00b1080 msgr2=0x7f1ee00ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee00b1080 0x7f1ee00ccc90 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f1ee00d0160 tx=0x7f1ed802bdb0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 -- 172.21.15.107:0/2181629649 shutdown_connections 2023-11-24T08:16:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee00b1080 0x7f1ee00ccc90 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/2181629649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee006e580 0x7f1ee00b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/2181629649 >> v2:172.21.15.120:3300/0 conn(0x7f1ee015b260 0x7f1ee006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 -- 172.21.15.107:0/2181629649 >> 172.21.15.107:0/2181629649 conn(0x7f1ee00cbb40 msgr2=0x7f1ee00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 -- 172.21.15.107:0/2181629649 shutdown_connections 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.163+0000 7f1ee51b3640 1 -- 172.21.15.107:0/2181629649 wait complete. 2023-11-24T08:16:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.165+0000 7f1ee51b3640 1 Processor -- start 2023-11-24T08:16:40.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.165+0000 7f1ee51b3640 1 -- start start 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee00b1080 0x7f1ee00b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee015b260 0x7f1ee00b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b1a70 con 0x7f1ee006e580 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b1d50 con 0x7f1ee015b260 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1ee51b3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b2030 con 0x7f1ee00b1080 2023-11-24T08:16:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1edffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1edffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39244/0 (socket says 172.21.15.107:39244) 2023-11-24T08:16:40.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.166+0000 7f1edffff640 1 -- 172.21.15.107:0/615960647 learned_addr learned my addr 172.21.15.107:0/615960647 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 -- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee00b1080 msgr2=0x7f1ee00b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee00b1080 0x7f1ee00b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 -- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee015b260 msgr2=0x7f1ee00b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee015b260 0x7f1ee00b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 -- 172.21.15.107:0/615960647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8009b30 con 0x7f1ee006e580 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edffff640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000d5f0 tx=0x7f1ed000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00125f0 con 0x7f1ee006e580 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0012750 con 0x7f1ee006e580 2023-11-24T08:16:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee00b2310 con 0x7f1ee006e580 2023-11-24T08:16:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.167+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0012b00 con 0x7f1ee006e580 2023-11-24T08:16:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.168+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee00b46b0 con 0x7f1ee006e580 2023-11-24T08:16:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.169+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee004e840 con 0x7f1ee006e580 2023-11-24T08:16:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.170+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f1ed0012c60 con 0x7f1ee006e580 2023-11-24T08:16:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.171+0000 7f1edd7fa640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 0x7f1ec006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.171+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1ed0088850 con 0x7f1ee006e580 2023-11-24T08:16:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.171+0000 7f1ed7fff640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 0x7f1ec006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.172+0000 7f1ed7fff640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 0x7f1ec006ea10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1ee009d170 tx=0x7f1ec8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.176+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ed0055360 con 0x7f1ee006e580 2023-11-24T08:16:40.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.398+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee00b1ec0 con 0x7f1ec006c550 2023-11-24T08:16:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.410+0000 7f1edd7fa640 1 -- 172.21.15.107:0/615960647 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f1ee00b1ec0 con 0x7f1ec006c550 2023-11-24T08:16:40.412 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7 2023-11-24T08:16:40.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 msgr2=0x7f1ec006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 0x7f1ec006ea10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1ee009d170 tx=0x7f1ec8008040 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 msgr2=0x7f1ee0157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000d5f0 tx=0x7f1ed000dab0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 shutdown_connections 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ec006c550 0x7f1ec006ea10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee015b260 0x7f1ee00b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee00b1080 0x7f1ee00b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 --2- 172.21.15.107:0/615960647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee006e580 0x7f1ee0157790 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.412+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 >> 172.21.15.107:0/615960647 conn(0x7f1ee00cbb40 msgr2=0x7f1ee00b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.413+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 shutdown_connections 2023-11-24T08:16:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.413+0000 7f1ee51b3640 1 -- 172.21.15.107:0/615960647 wait complete. 2023-11-24T08:16:40.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:40.434 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:40.491 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:16:40.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:40.492 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7) 2023-11-24T08:16:40.547 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:16:40.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone9276 2023-11-24T08:16:40.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.757+0000 7fe9bbd2c640 1 Processor -- start 2023-11-24T08:16:40.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.758+0000 7fe9bbd2c640 1 -- start start 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9bbd2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9b4161060 0x7fe9b4161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9bbd2c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe9b406b6c0 0x7fe9b415e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9bbd2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b406dcf0 con 0x7fe9b40b3330 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9bbd2c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b406ec10 con 0x7fe9b4161060 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9bbd2c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b4161980 con 0x7fe9b406b6c0 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9b9ac8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9b4161060 0x7fe9b4161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9b9ac8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9b4161060 0x7fe9b4161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39252/0 (socket says 172.21.15.107:39252) 2023-11-24T08:16:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.759+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/102605818 learned_addr learned my addr 172.21.15.107:0/102605818 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.760+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/102605818 >> v2:172.21.15.120:3300/0 conn(0x7fe9b406b6c0 msgr2=0x7fe9b415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:16:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.760+0000 7fe9b9ac8640 1 --2- 172.21.15.107:0/102605818 >> v2:172.21.15.120:3300/0 conn(0x7fe9b406b6c0 0x7fe9b415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.760+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/102605818 >> v1:172.21.15.107:6789/0 conn(0x7fe9b40b3330 legacy=0x7fe9b415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.760+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/102605818 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b4161ec0 con 0x7fe9b4161060 2023-11-24T08:16:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.761+0000 7fe9b9ac8640 1 --2- 172.21.15.107:0/102605818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b4161060 0x7fe9b4161440 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4072170 tx=0x7fe9a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f604d877182f45e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.761+0000 7fe9b8ac6640 1 -- 172.21.15.107:0/102605818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a402ca80 con 0x7fe9b4161060 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.761+0000 7fe9b8ac6640 1 -- 172.21.15.107:0/102605818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a402cbe0 con 0x7fe9b4161060 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.761+0000 7fe9b8ac6640 1 -- 172.21.15.107:0/102605818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40343f0 con 0x7fe9b4161060 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/102605818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b4161060 msgr2=0x7fe9b4161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/102605818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b4161060 0x7fe9b4161440 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4072170 tx=0x7fe9a402bee0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/102605818 shutdown_connections 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/102605818 >> v2:172.21.15.120:3300/0 conn(0x7fe9b406b6c0 0x7fe9b415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/102605818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b4161060 0x7fe9b4161440 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/102605818 >> 172.21.15.107:0/102605818 conn(0x7fe9b40cf900 msgr2=0x7fe9b40cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/102605818 shutdown_connections 2023-11-24T08:16:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.762+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/102605818 wait complete. 2023-11-24T08:16:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.763+0000 7fe9bbd2c640 1 Processor -- start 2023-11-24T08:16:40.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.764+0000 7fe9bbd2c640 1 -- start start 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.764+0000 7fe9bbd2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9bbd2c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40b3330 0x7fe9b41f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9bbd2c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 0x7fe9b414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9bbd2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b414f080 con 0x7fe9b406b6c0 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9bbd2c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b414f360 con 0x7fe9b4161060 2023-11-24T08:16:40.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9bbd2c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b414f640 con 0x7fe9b40b3330 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39260/0 (socket says 172.21.15.107:39260) 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/1364388727 learned_addr learned my addr 172.21.15.107:0/1364388727 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40b3330 msgr2=0x7fe9b41f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40b3330 0x7fe9b41f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 msgr2=0x7fe9b414e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9ba2c9640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 0x7fe9b414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 0x7fe9b414e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9b9ac8640 1 -- 172.21.15.107:0/1364388727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4009b30 con 0x7fe9b406b6c0 2023-11-24T08:16:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.765+0000 7fe9ba2c9640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 0x7fe9b414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:16:40.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9b9ac8640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4000f80 tx=0x7fe9a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a403e450 con 0x7fe9b406b6c0 2023-11-24T08:16:40.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a403e5f0 con 0x7fe9b406b6c0 2023-11-24T08:16:40.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a403e9a0 con 0x7fe9b406b6c0 2023-11-24T08:16:40.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b414f920 con 0x7fe9b406b6c0 2023-11-24T08:16:40.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.766+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b41f3a40 con 0x7fe9b406b6c0 2023-11-24T08:16:40.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.767+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b4067990 con 0x7fe9b406b6c0 2023-11-24T08:16:40.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.768+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7fe9a402cd50 con 0x7fe9b406b6c0 2023-11-24T08:16:40.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.770+0000 7fe9b27fc640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 0x7fe9a006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:16:40.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.770+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe9a40aafe0 con 0x7fe9b406b6c0 2023-11-24T08:16:40.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.770+0000 7fe9b92c7640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 0x7fe9a006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:16:40.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.770+0000 7fe9b92c7640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 0x7fe9a006ea10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8000fd0 tx=0x7fe9a8007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:16:40.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.774+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9a4077cb0 con 0x7fe9b406b6c0 2023-11-24T08:16:40.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:40.992+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9b41f3bd0 con 0x7fe9a006c550 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.011+0000 7fe9b27fc640 1 -- 172.21.15.107:0/1364388727 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7fe9b41f3bd0 con 0x7fe9a006c550 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:16:18", 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:16:41.013 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 67108864, 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:16:29", 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:16:37", 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:16:41.014 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:16:27", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:16:41.015 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:16:41.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 msgr2=0x7fe9a006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:41.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 0x7fe9a006ea10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8000fd0 tx=0x7fe9a8007930 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 msgr2=0x7fe9b4150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4000f80 tx=0x7fe9a4004a40 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 shutdown_connections 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe9a006c550 0x7fe9a006ea10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b4161060 0x7fe9b414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40b3330 0x7fe9b41f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 --2- 172.21.15.107:0/1364388727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9b406b6c0 0x7fe9b4150120 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 >> 172.21.15.107:0/1364388727 conn(0x7fe9b40cf900 msgr2=0x7fe9b415d850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 shutdown_connections 2023-11-24T08:16:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:16:41.014+0000 7fe9bbd2c640 1 -- 172.21.15.107:0/1364388727 wait complete. 2023-11-24T08:16:41.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.034 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05: ceph.quota.max_bytes: No such attribute 2023-11-24T08:16:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:16:41.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.085 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.140 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:16:41.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.252 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05 2023-11-24T08:16:41.252 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49 2023-11-24T08:16:41.252 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22 2023-11-24T08:16:41.252 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10 2023-11-24T08:16:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38 2023-11-24T08:16:41.254 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28 2023-11-24T08:16:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33 2023-11-24T08:16:41.256 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41 2023-11-24T08:16:41.257 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12 2023-11-24T08:16:41.258 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43 2023-11-24T08:16:41.258 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16 2023-11-24T08:16:41.258 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31 2023-11-24T08:16:41.258 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1 2023-11-24T08:16:41.258 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60 2023-11-24T08:16:41.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.259 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.317 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:16:41.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.321 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/) 2023-11-24T08:16:41.377 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:16:41.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.378 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.432 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:16:41.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.434 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/) 2023-11-24T08:16:41.486 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:16:41.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.487 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.541 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:16:41.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/) 2023-11-24T08:16:41.594 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:16:41.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05) 2023-11-24T08:16:41.642 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:41.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.643 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/) 2023-11-24T08:16:41.692 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:41.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.693 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49) 2023-11-24T08:16:41.742 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:41.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.743 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.49) 2023-11-24T08:16:41.796 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:41.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.797 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49) 2023-11-24T08:16:41.843 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:41.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.845 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.49) 2023-11-24T08:16:41.944 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:41.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:41.945 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49) 2023-11-24T08:16:42.001 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:42.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.002 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.49) 2023-11-24T08:16:42.055 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:42.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.49) 2023-11-24T08:16:42.110 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:42.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.112 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.49) 2023-11-24T08:16:42.163 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:42.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.164 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22) 2023-11-24T08:16:42.215 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:42.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.216 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.22) 2023-11-24T08:16:42.269 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:42.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.270 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22) 2023-11-24T08:16:42.320 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:42.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.321 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.22) 2023-11-24T08:16:42.371 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:42.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22) 2023-11-24T08:16:42.420 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:42.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.421 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.22) 2023-11-24T08:16:42.467 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:42.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.468 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.22) 2023-11-24T08:16:42.569 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:42.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.571 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.22) 2023-11-24T08:16:42.619 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:42.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.620 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20) 2023-11-24T08:16:42.669 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:42.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.671 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.20) 2023-11-24T08:16:42.723 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:42.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.725 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20) 2023-11-24T08:16:42.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:16:42.777 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:16:42.778 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:42.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.20) 2023-11-24T08:16:42.866 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:42.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20) 2023-11-24T08:16:42.991 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:42.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:42.992 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.20) 2023-11-24T08:16:43.045 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:43.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.046 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.20) 2023-11-24T08:16:43.098 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:43.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.20) 2023-11-24T08:16:43.151 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:43.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.152 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34) 2023-11-24T08:16:43.206 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:43.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.208 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.34) 2023-11-24T08:16:43.261 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:43.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34) 2023-11-24T08:16:43.315 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:43.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.317 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.34) 2023-11-24T08:16:43.375 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:43.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.377 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34) 2023-11-24T08:16:43.429 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:43.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.430 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.34) 2023-11-24T08:16:43.483 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:43.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.485 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.34) 2023-11-24T08:16:43.535 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:43.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.536 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.34) 2023-11-24T08:16:43.588 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:43.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.589 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55) 2023-11-24T08:16:43.640 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:43.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.641 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.55) 2023-11-24T08:16:43.693 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:43.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.694 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55) 2023-11-24T08:16:43.746 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:43.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.747 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.55) 2023-11-24T08:16:43.799 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:43.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.800 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55) 2023-11-24T08:16:43.852 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:43.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.854 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.55) 2023-11-24T08:16:43.904 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:43.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.55) 2023-11-24T08:16:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:43.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:43.957 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.55) 2023-11-24T08:16:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:44.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.013 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2) 2023-11-24T08:16:44.066 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.2) 2023-11-24T08:16:44.123 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.124 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2) 2023-11-24T08:16:44.177 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:44.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.179 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.2) 2023-11-24T08:16:44.230 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:44.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.231 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2) 2023-11-24T08:16:44.286 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:44.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.2) 2023-11-24T08:16:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:44.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.2) 2023-11-24T08:16:44.399 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:44.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.400 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.2) 2023-11-24T08:16:44.453 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:44.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.454 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35) 2023-11-24T08:16:44.510 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.511 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.35) 2023-11-24T08:16:44.564 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.566 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35) 2023-11-24T08:16:44.618 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:44.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.620 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.35) 2023-11-24T08:16:44.670 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:44.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.671 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35) 2023-11-24T08:16:44.723 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:44.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.35) 2023-11-24T08:16:44.777 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:44.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.779 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.35) 2023-11-24T08:16:44.827 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:44.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.829 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.35) 2023-11-24T08:16:44.878 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:44.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.880 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30) 2023-11-24T08:16:44.930 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.932 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.30) 2023-11-24T08:16:44.983 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:44.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:44.984 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30) 2023-11-24T08:16:45.030 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:45.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.032 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.30) 2023-11-24T08:16:45.126 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:45.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30) 2023-11-24T08:16:45.176 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:45.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.177 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.30) 2023-11-24T08:16:45.226 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:45.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.30) 2023-11-24T08:16:45.276 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:45.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.277 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.30) 2023-11-24T08:16:45.328 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:45.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.329 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14) 2023-11-24T08:16:45.378 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:45.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.379 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.14) 2023-11-24T08:16:45.430 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:45.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.432 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14) 2023-11-24T08:16:45.480 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:45.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.481 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.14) 2023-11-24T08:16:45.528 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:45.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.529 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14) 2023-11-24T08:16:45.627 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:45.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.629 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.14) 2023-11-24T08:16:45.682 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:45.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.683 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.14) 2023-11-24T08:16:45.738 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:45.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.739 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.14) 2023-11-24T08:16:45.793 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:45.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.795 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26) 2023-11-24T08:16:45.846 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:45.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.848 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.26) 2023-11-24T08:16:45.900 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:45.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.901 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26) 2023-11-24T08:16:45.954 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:45.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:45.955 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.26) 2023-11-24T08:16:46.009 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:46.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.010 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26) 2023-11-24T08:16:46.063 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:46.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.064 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.26) 2023-11-24T08:16:46.115 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:46.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.26) 2023-11-24T08:16:46.168 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:46.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.169 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.26) 2023-11-24T08:16:46.223 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:46.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.225 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51) 2023-11-24T08:16:46.279 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:46.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.280 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.51) 2023-11-24T08:16:46.333 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:46.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.334 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51) 2023-11-24T08:16:46.386 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:46.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.388 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.51) 2023-11-24T08:16:46.442 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:46.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51) 2023-11-24T08:16:46.494 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:46.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.495 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.51) 2023-11-24T08:16:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:46.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.51) 2023-11-24T08:16:46.596 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:46.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.51) 2023-11-24T08:16:46.696 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:46.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.697 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50) 2023-11-24T08:16:46.748 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:46.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.749 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.50) 2023-11-24T08:16:46.802 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:46.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50) 2023-11-24T08:16:46.856 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:46.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.50) 2023-11-24T08:16:46.910 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:46.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.911 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50) 2023-11-24T08:16:46.966 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:46.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:46.967 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.50) 2023-11-24T08:16:47.025 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:47.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.026 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.50) 2023-11-24T08:16:47.078 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:47.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.079 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.50) 2023-11-24T08:16:47.132 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:47.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.133 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13) 2023-11-24T08:16:47.188 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:47.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.13) 2023-11-24T08:16:47.243 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:47.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.244 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13) 2023-11-24T08:16:47.299 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:47.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.301 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.13) 2023-11-24T08:16:47.353 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:47.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.354 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13) 2023-11-24T08:16:47.406 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:47.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.407 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.13) 2023-11-24T08:16:47.459 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:47.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.461 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.13) 2023-11-24T08:16:47.512 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:47.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.514 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.13) 2023-11-24T08:16:47.565 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:47.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.567 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10) 2023-11-24T08:16:47.619 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:47.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.620 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.10) 2023-11-24T08:16:47.673 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:47.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10) 2023-11-24T08:16:47.730 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:47.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.732 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.10) 2023-11-24T08:16:47.783 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:47.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10) 2023-11-24T08:16:47.833 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:47.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.834 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.10) 2023-11-24T08:16:47.879 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:47.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:47.881 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.10) 2023-11-24T08:16:47.980 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:48.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.037 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.10) 2023-11-24T08:16:48.099 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:48.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47) 2023-11-24T08:16:48.155 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:48.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.156 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.47) 2023-11-24T08:16:48.211 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:48.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.212 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47) 2023-11-24T08:16:48.264 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:48.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.47) 2023-11-24T08:16:48.317 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:48.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47) 2023-11-24T08:16:48.371 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:48.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.47) 2023-11-24T08:16:48.422 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:48.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.423 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.47) 2023-11-24T08:16:48.473 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:48.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.474 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.47) 2023-11-24T08:16:48.523 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:48.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5) 2023-11-24T08:16:48.576 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:48.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.577 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.5) 2023-11-24T08:16:48.627 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:48.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5) 2023-11-24T08:16:48.676 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:48.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.678 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.5) 2023-11-24T08:16:48.727 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:48.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.728 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5) 2023-11-24T08:16:48.781 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:48.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.5) 2023-11-24T08:16:48.833 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:48.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.834 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.5) 2023-11-24T08:16:48.889 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:48.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.891 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.5) 2023-11-24T08:16:48.944 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:48.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.945 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37) 2023-11-24T08:16:48.997 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:48.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:48.998 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.37) 2023-11-24T08:16:49.049 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:49.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37) 2023-11-24T08:16:49.102 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.37) 2023-11-24T08:16:49.155 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.156 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37) 2023-11-24T08:16:49.211 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:49.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.213 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.37) 2023-11-24T08:16:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:49.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.265 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.37) 2023-11-24T08:16:49.316 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:49.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.317 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.37) 2023-11-24T08:16:49.370 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:49.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39) 2023-11-24T08:16:49.419 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:49.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.420 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.39) 2023-11-24T08:16:49.468 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:49.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.469 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39) 2023-11-24T08:16:49.517 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.519 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.39) 2023-11-24T08:16:49.574 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.575 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39) 2023-11-24T08:16:49.626 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:49.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.39) 2023-11-24T08:16:49.678 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:49.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.39) 2023-11-24T08:16:49.728 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:49.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.39) 2023-11-24T08:16:49.777 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:49.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54) 2023-11-24T08:16:49.828 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:49.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.829 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.54) 2023-11-24T08:16:49.882 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:49.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.884 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54) 2023-11-24T08:16:49.940 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.54) 2023-11-24T08:16:49.991 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:49.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:49.993 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54) 2023-11-24T08:16:50.044 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.045 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.54) 2023-11-24T08:16:50.097 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.098 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.54) 2023-11-24T08:16:50.149 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:50.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.54) 2023-11-24T08:16:50.202 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:50.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.203 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36) 2023-11-24T08:16:50.254 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:50.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.256 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.36) 2023-11-24T08:16:50.310 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:50.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.311 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36) 2023-11-24T08:16:50.365 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:50.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.366 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.36) 2023-11-24T08:16:50.419 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:50.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.420 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36) 2023-11-24T08:16:50.473 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.474 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.36) 2023-11-24T08:16:50.526 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.36) 2023-11-24T08:16:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:50.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.578 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.36) 2023-11-24T08:16:50.626 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18) 2023-11-24T08:16:50.679 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:50.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.681 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.18) 2023-11-24T08:16:50.735 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:50.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18) 2023-11-24T08:16:50.789 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:50.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.790 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.18) 2023-11-24T08:16:50.844 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:50.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.845 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18) 2023-11-24T08:16:50.898 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.899 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.18) 2023-11-24T08:16:50.957 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:50.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:50.958 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.18) 2023-11-24T08:16:51.011 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:51.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.013 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.18) 2023-11-24T08:16:51.065 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:51.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.067 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17) 2023-11-24T08:16:51.118 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:51.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.119 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.17) 2023-11-24T08:16:51.172 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:51.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.173 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17) 2023-11-24T08:16:51.227 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:51.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.228 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.17) 2023-11-24T08:16:51.280 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:51.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.281 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17) 2023-11-24T08:16:51.334 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:51.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.335 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.17) 2023-11-24T08:16:51.388 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:51.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.389 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.17) 2023-11-24T08:16:51.442 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:51.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.17) 2023-11-24T08:16:51.496 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:51.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.497 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4) 2023-11-24T08:16:51.549 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:51.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.4) 2023-11-24T08:16:51.605 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:51.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.606 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4) 2023-11-24T08:16:51.659 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:51.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.660 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.4) 2023-11-24T08:16:51.712 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:51.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.714 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4) 2023-11-24T08:16:51.768 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:51.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.4) 2023-11-24T08:16:51.826 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:51.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.4) 2023-11-24T08:16:51.882 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:51.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.884 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.4) 2023-11-24T08:16:51.939 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:51.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58) 2023-11-24T08:16:51.995 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:51.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:51.997 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.58) 2023-11-24T08:16:52.051 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:52.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.053 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58) 2023-11-24T08:16:52.104 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:52.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.106 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.58) 2023-11-24T08:16:52.158 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:52.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.159 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58) 2023-11-24T08:16:52.213 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:52.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.214 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.58) 2023-11-24T08:16:52.267 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:52.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.268 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.58) 2023-11-24T08:16:52.321 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:52.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.58) 2023-11-24T08:16:52.371 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:52.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29) 2023-11-24T08:16:52.425 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:52.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.426 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.29) 2023-11-24T08:16:52.480 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:52.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.481 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29) 2023-11-24T08:16:52.537 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:52.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.539 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.29) 2023-11-24T08:16:52.590 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:52.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.591 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29) 2023-11-24T08:16:52.643 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:52.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.644 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.29) 2023-11-24T08:16:52.695 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:52.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.697 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.29) 2023-11-24T08:16:52.746 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:52.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.747 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.29) 2023-11-24T08:16:52.796 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:52.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.798 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38) 2023-11-24T08:16:52.849 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:52.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.850 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.38) 2023-11-24T08:16:52.904 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:52.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.906 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38) 2023-11-24T08:16:52.961 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:52.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:52.962 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.38) 2023-11-24T08:16:53.016 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:53.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.017 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38) 2023-11-24T08:16:53.071 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.072 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.38) 2023-11-24T08:16:53.121 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.123 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.38) 2023-11-24T08:16:53.175 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:53.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.176 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.38) 2023-11-24T08:16:53.229 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:53.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.231 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19) 2023-11-24T08:16:53.285 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:53.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.287 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.19) 2023-11-24T08:16:53.339 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:53.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.341 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19) 2023-11-24T08:16:53.394 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:53.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.397 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.19) 2023-11-24T08:16:53.453 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:53.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.454 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19) 2023-11-24T08:16:53.507 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.509 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.19) 2023-11-24T08:16:53.560 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.561 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.19) 2023-11-24T08:16:53.615 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:53.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.617 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.19) 2023-11-24T08:16:53.667 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:16:53.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.669 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7) 2023-11-24T08:16:53.720 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:53.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.722 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.7) 2023-11-24T08:16:53.772 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:53.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.773 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7) 2023-11-24T08:16:53.826 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:53.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.7) 2023-11-24T08:16:53.885 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:53.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.887 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7) 2023-11-24T08:16:53.941 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.943 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.7) 2023-11-24T08:16:53.995 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:53.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:53.997 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.7) 2023-11-24T08:16:54.049 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:54.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.7) 2023-11-24T08:16:54.105 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:54.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.107 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48) 2023-11-24T08:16:54.159 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:54.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.160 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.48) 2023-11-24T08:16:54.213 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:54.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.214 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48) 2023-11-24T08:16:54.270 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:54.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.48) 2023-11-24T08:16:54.325 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:54.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.326 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48) 2023-11-24T08:16:54.381 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:54.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.382 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.48) 2023-11-24T08:16:54.435 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:54.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.436 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.48) 2023-11-24T08:16:54.489 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:54.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.48) 2023-11-24T08:16:54.545 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:54.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57) 2023-11-24T08:16:54.598 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:54.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.57) 2023-11-24T08:16:54.651 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:54.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57) 2023-11-24T08:16:54.707 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:54.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.57) 2023-11-24T08:16:54.759 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:54.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.760 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57) 2023-11-24T08:16:54.812 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:54.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.813 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.57) 2023-11-24T08:16:54.866 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:54.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.867 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.57) 2023-11-24T08:16:54.919 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:54.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.920 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.57) 2023-11-24T08:16:54.972 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:16:54.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:54.973 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6) 2023-11-24T08:16:55.023 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.024 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.6) 2023-11-24T08:16:55.073 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.074 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6) 2023-11-24T08:16:55.119 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:55.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.121 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.6) 2023-11-24T08:16:55.220 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:55.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.221 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6) 2023-11-24T08:16:55.273 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:55.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.274 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.6) 2023-11-24T08:16:55.328 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:55.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.329 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.6) 2023-11-24T08:16:55.381 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:55.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.383 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.6) 2023-11-24T08:16:55.434 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:55.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.435 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53) 2023-11-24T08:16:55.484 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.486 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.53) 2023-11-24T08:16:55.537 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53) 2023-11-24T08:16:55.588 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:55.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.590 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.53) 2023-11-24T08:16:55.640 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:55.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.642 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53) 2023-11-24T08:16:55.698 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:55.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.699 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.53) 2023-11-24T08:16:55.751 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:55.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.752 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.53) 2023-11-24T08:16:55.809 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:55.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.810 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.53) 2023-11-24T08:16:55.864 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:16:55.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.866 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40) 2023-11-24T08:16:55.919 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.921 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.40) 2023-11-24T08:16:55.979 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:55.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:55.981 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40) 2023-11-24T08:16:56.035 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.037 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.40) 2023-11-24T08:16:56.089 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.091 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40) 2023-11-24T08:16:56.143 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:56.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.144 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.40) 2023-11-24T08:16:56.197 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:56.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.198 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.40) 2023-11-24T08:16:56.248 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:56.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.249 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.40) 2023-11-24T08:16:56.299 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:16:56.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.300 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42) 2023-11-24T08:16:56.352 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:56.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.353 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.42) 2023-11-24T08:16:56.406 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:56.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.407 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42) 2023-11-24T08:16:56.457 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.458 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.42) 2023-11-24T08:16:56.512 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.513 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42) 2023-11-24T08:16:56.563 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:56.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.564 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.42) 2023-11-24T08:16:56.616 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:56.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.617 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.42) 2023-11-24T08:16:56.675 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:56.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.42) 2023-11-24T08:16:56.728 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:16:56.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.729 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0) 2023-11-24T08:16:56.780 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:56.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.781 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.0) 2023-11-24T08:16:56.837 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:56.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.839 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0) 2023-11-24T08:16:56.890 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.892 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.0) 2023-11-24T08:16:56.941 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:56.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.943 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0) 2023-11-24T08:16:56.992 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:56.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:56.994 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.0) 2023-11-24T08:16:57.044 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:57.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.046 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.0) 2023-11-24T08:16:57.146 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:57.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.0) 2023-11-24T08:16:57.205 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:16:57.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27) 2023-11-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:57.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.27) 2023-11-24T08:16:57.317 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:57.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.319 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27) 2023-11-24T08:16:57.374 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:57.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.375 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.27) 2023-11-24T08:16:57.429 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:57.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.431 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27) 2023-11-24T08:16:57.482 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:57.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.483 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.27) 2023-11-24T08:16:57.536 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:57.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.537 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.27) 2023-11-24T08:16:57.592 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:57.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.593 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.27) 2023-11-24T08:16:57.648 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:57.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.650 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25) 2023-11-24T08:16:57.705 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:57.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.707 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.25) 2023-11-24T08:16:57.764 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:57.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.765 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25) 2023-11-24T08:16:57.815 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:57.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.25) 2023-11-24T08:16:57.865 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:57.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.867 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25) 2023-11-24T08:16:57.920 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:57.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.920 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.25) 2023-11-24T08:16:57.965 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:57.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:57.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.25) 2023-11-24T08:16:58.067 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:58.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.069 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.25) 2023-11-24T08:16:58.124 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:58.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.125 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32) 2023-11-24T08:16:58.179 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:58.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.32) 2023-11-24T08:16:58.235 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:58.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32) 2023-11-24T08:16:58.288 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:58.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.32) 2023-11-24T08:16:58.338 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:58.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.341 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32) 2023-11-24T08:16:58.396 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:58.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.397 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.32) 2023-11-24T08:16:58.449 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:58.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.32) 2023-11-24T08:16:58.503 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:58.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.32) 2023-11-24T08:16:58.556 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:58.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.558 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28) 2023-11-24T08:16:58.610 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:58.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.612 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.28) 2023-11-24T08:16:58.664 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:58.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.666 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28) 2023-11-24T08:16:58.719 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:58.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.720 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.28) 2023-11-24T08:16:58.767 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:58.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.768 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28) 2023-11-24T08:16:58.867 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:58.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.869 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.28) 2023-11-24T08:16:58.921 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:58.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.28) 2023-11-24T08:16:58.981 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:58.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:58.983 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.28) 2023-11-24T08:16:59.036 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:16:59.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.037 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21) 2023-11-24T08:16:59.088 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:59.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.089 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.21) 2023-11-24T08:16:59.143 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:59.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.145 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21) 2023-11-24T08:16:59.197 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:59.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.198 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.21) 2023-11-24T08:16:59.251 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:59.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21) 2023-11-24T08:16:59.305 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:59.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.21) 2023-11-24T08:16:59.358 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:59.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.21) 2023-11-24T08:16:59.412 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:59.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.413 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.21) 2023-11-24T08:16:59.466 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:16:59.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.467 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9) 2023-11-24T08:16:59.519 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:59.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.520 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.9) 2023-11-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:16:59.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.572 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9) 2023-11-24T08:16:59.624 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:59.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.9) 2023-11-24T08:16:59.678 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:16:59.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.679 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9) 2023-11-24T08:16:59.731 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:59.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.733 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.9) 2023-11-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:16:59.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.787 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.9) 2023-11-24T08:16:59.886 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:59.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.9) 2023-11-24T08:16:59.942 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:16:59.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:16:59.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23) 2023-11-24T08:17:00.001 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.003 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.23) 2023-11-24T08:17:00.056 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.057 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23) 2023-11-24T08:17:00.108 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:00.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.109 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.23) 2023-11-24T08:17:00.161 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:00.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.162 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23) 2023-11-24T08:17:00.213 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:00.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.215 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.23) 2023-11-24T08:17:00.274 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:00.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.23) 2023-11-24T08:17:00.329 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:17:00.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.331 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.23) 2023-11-24T08:17:00.382 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:17:00.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.384 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15) 2023-11-24T08:17:00.436 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.437 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.15) 2023-11-24T08:17:00.489 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15) 2023-11-24T08:17:00.541 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:00.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.15) 2023-11-24T08:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:00.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15) 2023-11-24T08:17:00.649 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:00.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.650 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.15) 2023-11-24T08:17:00.699 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:00.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.15) 2023-11-24T08:17:00.747 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:17:00.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.748 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.15) 2023-11-24T08:17:00.797 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:17:00.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.799 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8) 2023-11-24T08:17:00.850 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.852 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.8) 2023-11-24T08:17:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:00.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.907 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8) 2023-11-24T08:17:00.959 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:00.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:00.960 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.8) 2023-11-24T08:17:01.013 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:01.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.015 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8) 2023-11-24T08:17:01.066 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.8) 2023-11-24T08:17:01.119 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.121 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.8) 2023-11-24T08:17:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:01.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.177 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.8) 2023-11-24T08:17:01.230 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:01.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.231 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46) 2023-11-24T08:17:01.283 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:01.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.284 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.46) 2023-11-24T08:17:01.338 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:01.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.339 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46) 2023-11-24T08:17:01.388 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:01.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.389 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.46) 2023-11-24T08:17:01.438 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:01.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.439 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46) 2023-11-24T08:17:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.492 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.46) 2023-11-24T08:17:01.544 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.545 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.46) 2023-11-24T08:17:01.594 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:01.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.46) 2023-11-24T08:17:01.645 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:01.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.646 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33) 2023-11-24T08:17:01.697 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:01.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.699 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.33) 2023-11-24T08:17:01.749 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:01.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.750 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33) 2023-11-24T08:17:01.800 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:01.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.801 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.33) 2023-11-24T08:17:01.852 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:01.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.854 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33) 2023-11-24T08:17:01.906 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.907 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.33) 2023-11-24T08:17:01.962 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:01.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:01.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.33) 2023-11-24T08:17:02.016 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:17:02.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.017 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.33) 2023-11-24T08:17:02.073 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:17:02.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.074 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45) 2023-11-24T08:17:02.127 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:02.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.129 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.45) 2023-11-24T08:17:02.186 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:02.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.187 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45) 2023-11-24T08:17:02.237 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:02.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.45) 2023-11-24T08:17:02.290 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:02.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45) 2023-11-24T08:17:02.343 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:02.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.344 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.45) 2023-11-24T08:17:02.392 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:02.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.45) 2023-11-24T08:17:02.445 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:02.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.447 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.45) 2023-11-24T08:17:02.497 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:02.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61) 2023-11-24T08:17:02.550 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:02.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.551 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.61) 2023-11-24T08:17:02.603 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:02.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.604 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61) 2023-11-24T08:17:02.654 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:02.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.61) 2023-11-24T08:17:02.706 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:02.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.707 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61) 2023-11-24T08:17:02.760 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:02.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.761 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.61) 2023-11-24T08:17:02.814 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:02.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.61) 2023-11-24T08:17:02.867 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:02.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.61) 2023-11-24T08:17:02.922 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:02.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62) 2023-11-24T08:17:02.977 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:02.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:02.978 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.62) 2023-11-24T08:17:03.033 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:03.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.035 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62) 2023-11-24T08:17:03.088 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:03.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.089 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.62) 2023-11-24T08:17:03.142 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:03.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.143 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62) 2023-11-24T08:17:03.195 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:03.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.196 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.62) 2023-11-24T08:17:03.249 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:03.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.62) 2023-11-24T08:17:03.303 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:03.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.304 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.62) 2023-11-24T08:17:03.360 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:03.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.361 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52) 2023-11-24T08:17:03.414 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:03.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.416 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.52) 2023-11-24T08:17:03.472 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:03.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52) 2023-11-24T08:17:03.524 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:03.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.52) 2023-11-24T08:17:03.578 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:03.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.579 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52) 2023-11-24T08:17:03.628 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:03.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.629 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.52) 2023-11-24T08:17:03.680 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:03.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.681 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.52) 2023-11-24T08:17:03.729 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:17:03.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.52) 2023-11-24T08:17:03.784 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:17:03.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11) 2023-11-24T08:17:03.836 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:03.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.838 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.11) 2023-11-24T08:17:03.892 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:03.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.894 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11) 2023-11-24T08:17:03.949 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:03.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:03.950 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.11) 2023-11-24T08:17:04.003 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:04.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11) 2023-11-24T08:17:04.059 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:04.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.060 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.11) 2023-11-24T08:17:04.111 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:04.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.113 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.11) 2023-11-24T08:17:04.164 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:04.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.165 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.11) 2023-11-24T08:17:04.214 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:04.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.216 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44) 2023-11-24T08:17:04.265 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:04.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.44) 2023-11-24T08:17:04.321 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:04.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44) 2023-11-24T08:17:04.373 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:04.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.374 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.44) 2023-11-24T08:17:04.421 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:04.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.423 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44) 2023-11-24T08:17:04.525 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:04.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.44) 2023-11-24T08:17:04.581 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:04.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.582 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.44) 2023-11-24T08:17:04.632 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:04.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.634 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.44) 2023-11-24T08:17:04.686 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:04.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3) 2023-11-24T08:17:04.741 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:04.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.743 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.3) 2023-11-24T08:17:04.792 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:04.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.794 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3) 2023-11-24T08:17:04.842 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:04.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.844 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.3) 2023-11-24T08:17:04.895 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:04.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.897 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3) 2023-11-24T08:17:04.948 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:04.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:04.949 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.3) 2023-11-24T08:17:05.000 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:05.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.3) 2023-11-24T08:17:05.050 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:17:05.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.051 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.3) 2023-11-24T08:17:05.102 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:17:05.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24) 2023-11-24T08:17:05.155 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:05.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.156 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.24) 2023-11-24T08:17:05.209 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:05.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.210 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24) 2023-11-24T08:17:05.262 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:05.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.24) 2023-11-24T08:17:05.315 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:05.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.317 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24) 2023-11-24T08:17:05.367 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:05.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.368 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.24) 2023-11-24T08:17:05.417 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:05.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.24) 2023-11-24T08:17:05.471 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:17:05.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.474 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.24) 2023-11-24T08:17:05.527 INFO:teuthology.orchestra.run.smithi107.stdout:1700813782 2023-11-24T08:17:05.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.528 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56) 2023-11-24T08:17:05.581 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:05.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.582 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.56) 2023-11-24T08:17:05.635 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:05.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.636 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56) 2023-11-24T08:17:05.690 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:05.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.692 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.56) 2023-11-24T08:17:05.746 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:05.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.748 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56) 2023-11-24T08:17:05.801 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:05.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.803 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.56) 2023-11-24T08:17:05.854 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:05.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.855 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.56) 2023-11-24T08:17:05.903 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:17:05.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.56) 2023-11-24T08:17:05.955 INFO:teuthology.orchestra.run.smithi107.stdout:1700813786 2023-11-24T08:17:05.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:05.956 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63) 2023-11-24T08:17:06.008 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.009 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.63) 2023-11-24T08:17:06.061 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.062 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63) 2023-11-24T08:17:06.110 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:06.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.111 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.63) 2023-11-24T08:17:06.158 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:06.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.159 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63) 2023-11-24T08:17:06.260 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:06.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.63) 2023-11-24T08:17:06.316 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:06.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.318 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.63) 2023-11-24T08:17:06.370 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:06.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.63) 2023-11-24T08:17:06.425 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:06.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.426 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59) 2023-11-24T08:17:06.477 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.478 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.59) 2023-11-24T08:17:06.529 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.530 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59) 2023-11-24T08:17:06.582 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:06.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.584 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.59) 2023-11-24T08:17:06.634 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:06.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.636 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59) 2023-11-24T08:17:06.690 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:06.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.691 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.59) 2023-11-24T08:17:06.741 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:06.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.743 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.59) 2023-11-24T08:17:06.793 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:06.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.795 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.59) 2023-11-24T08:17:06.845 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:06.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.846 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41) 2023-11-24T08:17:06.896 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.897 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.41) 2023-11-24T08:17:06.949 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:06.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:06.950 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41) 2023-11-24T08:17:07.002 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.003 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.41) 2023-11-24T08:17:07.057 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.058 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41) 2023-11-24T08:17:07.111 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:07.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.112 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.41) 2023-11-24T08:17:07.164 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:07.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.165 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.41) 2023-11-24T08:17:07.214 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:17:07.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.215 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.41) 2023-11-24T08:17:07.263 INFO:teuthology.orchestra.run.smithi107.stdout:1700813784 2023-11-24T08:17:07.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.264 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12) 2023-11-24T08:17:07.312 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:07.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.314 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.12) 2023-11-24T08:17:07.366 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:07.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12) 2023-11-24T08:17:07.421 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.422 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.12) 2023-11-24T08:17:07.475 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.476 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12) 2023-11-24T08:17:07.527 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:07.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.529 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.12) 2023-11-24T08:17:07.580 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:07.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.582 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.12) 2023-11-24T08:17:07.635 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:07.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.636 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.12) 2023-11-24T08:17:07.688 INFO:teuthology.orchestra.run.smithi107.stdout:1700813780 2023-11-24T08:17:07.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.689 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43) 2023-11-24T08:17:07.741 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:07.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.742 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.43) 2023-11-24T08:17:07.792 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:07.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.793 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43) 2023-11-24T08:17:07.845 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.846 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.43) 2023-11-24T08:17:07.897 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:07.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.898 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43) 2023-11-24T08:17:07.949 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:07.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:07.951 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.43) 2023-11-24T08:17:08.007 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:08.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.008 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.43) 2023-11-24T08:17:08.060 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:08.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.43) 2023-11-24T08:17:08.114 INFO:teuthology.orchestra.run.smithi107.stdout:1700813785 2023-11-24T08:17:08.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.116 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16) 2023-11-24T08:17:08.166 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:08.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.16) 2023-11-24T08:17:08.222 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:08.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.223 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16) 2023-11-24T08:17:08.274 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:08.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.16) 2023-11-24T08:17:08.329 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:08.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.330 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16) 2023-11-24T08:17:08.385 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:08.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.16) 2023-11-24T08:17:08.439 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:08.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.441 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.16) 2023-11-24T08:17:08.493 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:17:08.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.501 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.16) 2023-11-24T08:17:08.557 INFO:teuthology.orchestra.run.smithi107.stdout:1700813781 2023-11-24T08:17:08.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.559 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31) 2023-11-24T08:17:08.614 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:08.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.615 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.31) 2023-11-24T08:17:08.668 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:08.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.669 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31) 2023-11-24T08:17:08.725 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:08.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.726 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.31) 2023-11-24T08:17:08.779 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:08.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.781 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31) 2023-11-24T08:17:08.833 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:08.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.835 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.31) 2023-11-24T08:17:08.887 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:08.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.31) 2023-11-24T08:17:08.941 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:17:08.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.943 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.31) 2023-11-24T08:17:08.992 INFO:teuthology.orchestra.run.smithi107.stdout:1700813783 2023-11-24T08:17:08.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:08.993 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1) 2023-11-24T08:17:09.042 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:09.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.1) 2023-11-24T08:17:09.099 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:09.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1) 2023-11-24T08:17:09.154 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:09.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.156 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.1) 2023-11-24T08:17:09.208 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:09.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.209 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1) 2023-11-24T08:17:09.261 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:09.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.262 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.1) 2023-11-24T08:17:09.314 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:09.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.315 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.1) 2023-11-24T08:17:09.368 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:17:09.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.369 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.1) 2023-11-24T08:17:09.422 INFO:teuthology.orchestra.run.smithi107.stdout:1700813779 2023-11-24T08:17:09.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.423 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60) 2023-11-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:09.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.480 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.60) 2023-11-24T08:17:09.528 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:17:09.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.530 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60) 2023-11-24T08:17:09.585 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:09.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.586 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.60) 2023-11-24T08:17:09.637 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:17:09.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.639 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60) 2023-11-24T08:17:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:09.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.686 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.60) 2023-11-24T08:17:09.786 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:17:09.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.787 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol6903/.snap/subvol_snap5787/b2d2cb3b-4626-4593-9afb-021868c54c05/subvolume_file.60) 2023-11-24T08:17:09.835 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:09.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:09.836 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9276/32842bba-697a-4502-98c5-26e00cd3cec7/subvolume_file.60) 2023-11-24T08:17:09.882 INFO:teuthology.orchestra.run.smithi107.stdout:1700813787 2023-11-24T08:17:09.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol6903 subvol_snap5787 2023-11-24T08:17:10.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.152+0000 7fc9fbfff640 1 Processor -- start 2023-11-24T08:17:10.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.152+0000 7fc9fbfff640 1 -- start start 2023-11-24T08:17:10.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9fbfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9fc146660 0x7fc9fc160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9fc0accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9fbfff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc15de90 con 0x7fc9fc0b1560 2023-11-24T08:17:10.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9fbfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9fc0ad210 con 0x7fc9fc146660 2023-11-24T08:17:10.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9fbfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc15ea30 con 0x7fc9fc0ac840 2023-11-24T08:17:10.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9f27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc9fc0b1560 0x7fc9fc15e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34718/0 (socket says 172.21.15.107:34718) 2023-11-24T08:17:10.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.154+0000 7fc9f27fc640 1 -- 172.21.15.107:0/2630473187 learned_addr learned my addr 172.21.15.107:0/2630473187 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:10.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fb7fe640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9fc0accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9faffd640 1 --2- 172.21.15.107:0/2630473187 >> v2:172.21.15.120:3300/0 conn(0x7fc9fc146660 0x7fc9fc160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076455534 0 0) 0x7fc9fc15de90 con 0x7fc9fc0b1560 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8003520 con 0x7fc9fc0b1560 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304515748 0 0) 0x7fc9e8003520 con 0x7fc9fc0b1560 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 msgr2=0x7fc9fc0accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9fc0accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 >> v2:172.21.15.120:3300/0 conn(0x7fc9fc146660 msgr2=0x7fc9fc160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 --2- 172.21.15.107:0/2630473187 >> v2:172.21.15.120:3300/0 conn(0x7fc9fc146660 0x7fc9fc160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc146a40 con 0x7fc9fc0b1560 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc9e40017f0 con 0x7fc9fc0b1560 2023-11-24T08:17:10.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 >> v1:172.21.15.107:6789/0 conn(0x7fc9fc0b1560 legacy=0x7fc9fc15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e8003f70 0x7fc9e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e8004d00 0x7fc9e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc0ad210 con 0x7fc9e8004d00 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc15ea30 con 0x7fc9e8003f70 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.155+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e800ba30 con 0x7fc9fc0ac840 2023-11-24T08:17:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e8004d00 0x7fc9e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9faffd640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9f27fc640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e8003f70 0x7fc9e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 -- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 msgr2=0x7fc9e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 -- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e8003f70 msgr2=0x7fc9e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e8003f70 0x7fc9e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9fb7fe640 1 -- 172.21.15.107:0/2630473187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8003520 con 0x7fc9e8004d00 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.156+0000 7fc9faffd640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc0ac840 0x7fc9e80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fb7fe640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e8004d00 0x7fc9e8005180 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009fc0 tx=0x7fc9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0783b72292fdade server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f402cc20 con 0x7fc9e8004d00 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f400f040 con 0x7fc9e8004d00 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fa7fc640 1 -- 172.21.15.107:0/2630473187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4034440 con 0x7fc9e8004d00 2023-11-24T08:17:10.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fbfff640 1 -- 172.21.15.107:0/2630473187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e8004d00 msgr2=0x7fc9e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/2630473187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e8004d00 0x7fc9e8005180 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009fc0 tx=0x7fc9e80057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.158+0000 7fc9fb7fe640 1 -- 172.21.15.107:0/2630473187 reap_dead start 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.159+0000 7fc9fbfff640 1 -- 172.21.15.107:0/2630473187 shutdown_connections 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.159+0000 7fc9fbfff640 1 -- 172.21.15.107:0/2630473187 >> 172.21.15.107:0/2630473187 conn(0x7fc9fc0cb8a0 msgr2=0x7fc9fc0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.159+0000 7fc9fbfff640 1 -- 172.21.15.107:0/2630473187 shutdown_connections 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.159+0000 7fc9fbfff640 1 -- 172.21.15.107:0/2630473187 wait complete. 2023-11-24T08:17:10.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.161+0000 7fc9fbfff640 1 Processor -- start 2023-11-24T08:17:10.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.161+0000 7fc9fbfff640 1 -- start start 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc142d50 0x7fc9fc071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9fc0b1bd0 0x7fc9fc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc0b3620 con 0x7fc9fc142970 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc141240 con 0x7fc9fc0b1bd0 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.162+0000 7fc9fbfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc071eb0 con 0x7fc9fc142d50 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41332/0 (socket says 172.21.15.107:41332) 2023-11-24T08:17:10.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 -- 172.21.15.107:0/3649205304 learned_addr learned my addr 172.21.15.107:0/3649205304 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:10.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 -- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc142d50 msgr2=0x7fc9fc071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc142d50 0x7fc9fc071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 -- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9fc0b1bd0 msgr2=0x7fc9fc0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9fc0b1bd0 0x7fc9fc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 -- 172.21.15.107:0/3649205304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc0720e0 con 0x7fc9fc142970 2023-11-24T08:17:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9faffd640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0116d0 tx=0x7fc9ec011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.163+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec00f940 con 0x7fc9fc142970 2023-11-24T08:17:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.164+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec00faa0 con 0x7fc9fc142970 2023-11-24T08:17:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.164+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc0b2c80 con 0x7fc9fc142970 2023-11-24T08:17:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.164+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec00fe50 con 0x7fc9fc142970 2023-11-24T08:17:10.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.164+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc0b3240 con 0x7fc9fc142970 2023-11-24T08:17:10.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.165+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc1f3000 con 0x7fc9fc142970 2023-11-24T08:17:10.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.166+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7fc9ec015460 con 0x7fc9fc142970 2023-11-24T08:17:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.167+0000 7fc9f3fff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 0x7fc9dc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.167+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc9ec088c20 con 0x7fc9fc142970 2023-11-24T08:17:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.167+0000 7fc9f27fc640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 0x7fc9dc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.168+0000 7fc9f27fc640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 0x7fc9dc06ea10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc072750 tx=0x7fc9e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.173+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9ec0557e0 con 0x7fc9fc142970 2023-11-24T08:17:10.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.396+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol6903", "snap_name": "subvol_snap5787", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9fc15db20 con 0x7fc9dc06c550 2023-11-24T08:17:10.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.443+0000 7fc9f3fff640 1 -- 172.21.15.107:0/3649205304 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc9e4001750 con 0x7fc9dc06c550 2023-11-24T08:17:10.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 msgr2=0x7fc9dc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 0x7fc9dc06ea10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc072750 tx=0x7fc9e4009040 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 msgr2=0x7fc9fc140d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0116d0 tx=0x7fc9ec011b90 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 shutdown_connections 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9dc06c550 0x7fc9dc06ea10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9fc0b1bd0 0x7fc9fc0b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9fc142d50 0x7fc9fc071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 --2- 172.21.15.107:0/3649205304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9fc142970 0x7fc9fc140d00 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 >> 172.21.15.107:0/3649205304 conn(0x7fc9fc0cb8a0 msgr2=0x7fc9fc06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 shutdown_connections 2023-11-24T08:17:10.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.446+0000 7fc9fbfff640 1 -- 172.21.15.107:0/3649205304 wait complete. 2023-11-24T08:17:10.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol6903 2023-11-24T08:17:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.674+0000 7f09b669b640 1 Processor -- start 2023-11-24T08:17:10.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.674+0000 7f09b669b640 1 -- start start 2023-11-24T08:17:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b669b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 0x7f09b0161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b669b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f09b006b6c0 0x7f09b015e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b669b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b006dcf0 con 0x7f09b00b3330 2023-11-24T08:17:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b669b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b006ec10 con 0x7f09b006b6c0 2023-11-24T08:17:10.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b669b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b0161980 con 0x7f09b0161060 2023-11-24T08:17:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09af7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f09b00b3330 0x7f09b015da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34720/0 (socket says 172.21.15.107:34720) 2023-11-24T08:17:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09af7fe640 1 -- 172.21.15.107:0/740892996 learned_addr learned my addr 172.21.15.107:0/740892996 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.676+0000 7f09b4c38640 1 --2- 172.21.15.107:0/740892996 >> v2:172.21.15.107:3300/0 conn(0x7f09b006b6c0 0x7f09b015e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09aeffd640 1 -- 172.21.15.107:0/740892996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536402327 0 0) 0x7f09b006dcf0 con 0x7f09b00b3330 2023-11-24T08:17:10.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09affff640 1 --2- 172.21.15.107:0/740892996 >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 0x7f09b0161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09aeffd640 1 -- 172.21.15.107:0/740892996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c003520 con 0x7f09b00b3330 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09b4c38640 1 -- 172.21.15.107:0/740892996 >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 msgr2=0x7f09b0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09b4c38640 1 --2- 172.21.15.107:0/740892996 >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 0x7f09b0161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09b4c38640 1 -- 172.21.15.107:0/740892996 >> v1:172.21.15.107:6789/0 conn(0x7f09b00b3330 legacy=0x7f09b015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09b4c38640 1 -- 172.21.15.107:0/740892996 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0161ec0 con 0x7f09b006b6c0 2023-11-24T08:17:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.677+0000 7f09affff640 1 --2- 172.21.15.107:0/740892996 >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 0x7f09b0161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.678+0000 7f09b4c38640 1 --2- 172.21.15.107:0/740892996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b006b6c0 0x7f09b015e190 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f09b00611c0 tx=0x7f09a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a3458195c61721 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.678+0000 7f09aeffd640 1 -- 172.21.15.107:0/740892996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802e070 con 0x7f09b006b6c0 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.678+0000 7f09aeffd640 1 -- 172.21.15.107:0/740892996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a80315a0 con 0x7f09b006b6c0 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.678+0000 7f09aeffd640 1 -- 172.21.15.107:0/740892996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8031930 con 0x7f09b006b6c0 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 -- 172.21.15.107:0/740892996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b006b6c0 msgr2=0x7f09b015e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 --2- 172.21.15.107:0/740892996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b006b6c0 0x7f09b015e190 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f09b00611c0 tx=0x7f09a802b840 comp rx=0 tx=0).stop 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 -- 172.21.15.107:0/740892996 shutdown_connections 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 --2- 172.21.15.107:0/740892996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b006b6c0 0x7f09b015e190 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 --2- 172.21.15.107:0/740892996 >> v2:172.21.15.120:3300/0 conn(0x7f09b0161060 0x7f09b0161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 -- 172.21.15.107:0/740892996 >> 172.21.15.107:0/740892996 conn(0x7f09b00cf900 msgr2=0x7f09b00cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 -- 172.21.15.107:0/740892996 shutdown_connections 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.679+0000 7f09b669b640 1 -- 172.21.15.107:0/740892996 wait complete. 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.681+0000 7f09b669b640 1 Processor -- start 2023-11-24T08:17:10.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.681+0000 7f09b669b640 1 -- start start 2023-11-24T08:17:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b006b6c0 0x7f09b01f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b00b3330 0x7f09b01546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01553e0 con 0x7f09b0161060 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00cc280 con 0x7f09b006b6c0 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b669b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0155100 con 0x7f09b00b3330 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b4c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b4c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41354/0 (socket says 172.21.15.107:41354) 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09b4c38640 1 -- 172.21.15.107:0/904072442 learned_addr learned my addr 172.21.15.107:0/904072442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09affff640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b006b6c0 0x7f09b01f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.682+0000 7f09af7fe640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b00b3330 0x7f09b01546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 -- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b00b3330 msgr2=0x7f09b01546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b00b3330 0x7f09b01546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 -- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b006b6c0 msgr2=0x7f09b01f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b006b6c0 0x7f09b01f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 -- 172.21.15.107:0/904072442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009400 con 0x7f09b0161060 2023-11-24T08:17:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b4c38640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f09a802be50 tx=0x7f09a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8032840 con 0x7f09b0161060 2023-11-24T08:17:10.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a80329a0 con 0x7f09b0161060 2023-11-24T08:17:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b00cbc30 con 0x7f09b0161060 2023-11-24T08:17:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8032d30 con 0x7f09b0161060 2023-11-24T08:17:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.683+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0155c00 con 0x7f09b0161060 2023-11-24T08:17:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.685+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00cc410 con 0x7f09b0161060 2023-11-24T08:17:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.686+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f09a804e430 con 0x7f09b0161060 2023-11-24T08:17:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.686+0000 7f09acff9640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 0x7f099006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.687+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f09a80aa310 con 0x7f09b0161060 2023-11-24T08:17:10.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.687+0000 7f09affff640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 0x7f099006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:10.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.687+0000 7f09affff640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 0x7f099006ea10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f09a00075b0 tx=0x7f09a0007540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:10.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.691+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09a8076e20 con 0x7f09b0161060 2023-11-24T08:17:10.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.908+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol6903", "target": ["mon-mgr", ""]}) v1 -- 0x7f09b006e550 con 0x7f099006c550 2023-11-24T08:17:10.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.937+0000 7f09acff9640 1 -- 172.21.15.107:0/904072442 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f09a0007cf0 con 0x7f099006c550 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 msgr2=0x7f099006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 0x7f099006ea10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f09a00075b0 tx=0x7f09a0007540 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 msgr2=0x7f09b00cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f09a802be50 tx=0x7f09a80046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 shutdown_connections 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f099006c550 0x7f099006ea10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09b0161060 0x7f09b00cde60 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b00b3330 0x7f09b01546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 --2- 172.21.15.107:0/904072442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b006b6c0 0x7f09b01f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.940+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 >> 172.21.15.107:0/904072442 conn(0x7f09b00cf900 msgr2=0x7f09b015f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.941+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 shutdown_connections 2023-11-24T08:17:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:10.941+0000 7f09b669b640 1 -- 172.21.15.107:0/904072442 wait complete. 2023-11-24T08:17:10.957 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:10.955+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:10.957 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:10.955+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:10.957 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:10.955+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:10.957 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:10.955+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:10.957 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:10.955+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:10.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone9276 2023-11-24T08:17:11.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.166+0000 7f6fb1c7c640 1 Processor -- start 2023-11-24T08:17:11.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.166+0000 7f6fb1c7c640 1 -- start start 2023-11-24T08:17:11.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fb1c7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fac161060 0x7f6fac161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fb1c7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fac06b6c0 0x7f6fac15e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fb1c7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac06dcf0 con 0x7f6fac0b3330 2023-11-24T08:17:11.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fb1c7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac06ec10 con 0x7f6fac161060 2023-11-24T08:17:11.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fb1c7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac161980 con 0x7f6fac06b6c0 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6faaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6fac0b3330 0x7f6fac15da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34726/0 (socket says 172.21.15.107:34726) 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6faaffd640 1 -- 172.21.15.107:0/4275038438 learned_addr learned my addr 172.21.15.107:0/4275038438 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.168+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/4275038438 >> v2:172.21.15.107:3300/0 conn(0x7f6fac161060 0x7f6fac161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6fabfff640 1 --2- 172.21.15.107:0/4275038438 >> v2:172.21.15.120:3300/0 conn(0x7f6fac06b6c0 0x7f6fac15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6fab7fe640 1 -- 172.21.15.107:0/4275038438 >> v2:172.21.15.120:3300/0 conn(0x7f6fac06b6c0 msgr2=0x7f6fac15e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:11.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/4275038438 >> v2:172.21.15.120:3300/0 conn(0x7f6fac06b6c0 0x7f6fac15e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6fab7fe640 1 -- 172.21.15.107:0/4275038438 >> v1:172.21.15.107:6789/0 conn(0x7f6fac0b3330 legacy=0x7f6fac15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6faa7fc640 1 -- 172.21.15.107:0/4275038438 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663028805 0 0) 0x7f6fac06dcf0 con 0x7f6fac0b3330 2023-11-24T08:17:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.169+0000 7f6fab7fe640 1 -- 172.21.15.107:0/4275038438 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac161ec0 con 0x7f6fac161060 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.170+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/4275038438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac161060 0x7f6fac161440 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f6fac072170 tx=0x7f6f9c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87de2f45f8a1652 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.170+0000 7f6faa7fc640 1 -- 172.21.15.107:0/4275038438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c02e070 con 0x7f6fac161060 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.170+0000 7f6faa7fc640 1 -- 172.21.15.107:0/4275038438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c0315a0 con 0x7f6fac161060 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.170+0000 7f6faa7fc640 1 -- 172.21.15.107:0/4275038438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c031930 con 0x7f6fac161060 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/4275038438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac161060 msgr2=0x7f6fac161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/4275038438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac161060 0x7f6fac161440 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f6fac072170 tx=0x7f6f9c02b840 comp rx=0 tx=0).stop 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/4275038438 shutdown_connections 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/4275038438 >> v2:172.21.15.120:3300/0 conn(0x7f6fac06b6c0 0x7f6fac15e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/4275038438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac161060 0x7f6fac161440 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/4275038438 >> 172.21.15.107:0/4275038438 conn(0x7f6fac0cf900 msgr2=0x7f6fac0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/4275038438 shutdown_connections 2023-11-24T08:17:11.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.171+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/4275038438 wait complete. 2023-11-24T08:17:11.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.173+0000 7f6fb1c7c640 1 Processor -- start 2023-11-24T08:17:11.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.174+0000 7f6fb1c7c640 1 -- start start 2023-11-24T08:17:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fac0b3330 0x7f6fac0b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fac161060 0x7f6fac1f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0ac840 con 0x7f6fac06b6c0 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1f7120 con 0x7f6fac161060 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fb1c7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0aca90 con 0x7f6fac0b3330 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41400/0 (socket says 172.21.15.107:41400) 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.175+0000 7f6fab7fe640 1 -- 172.21.15.107:0/2783780910 learned_addr learned my addr 172.21.15.107:0/2783780910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6faaffd640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fac0b3330 0x7f6fac0b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fabfff640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fac161060 0x7f6fac1f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 -- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fac0b3330 msgr2=0x7f6fac0b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fac0b3330 0x7f6fac0b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 -- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fac161060 msgr2=0x7f6fac1f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fac161060 0x7f6fac1f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 -- 172.21.15.107:0/2783780910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c009400 con 0x7f6fac06b6c0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.176+0000 7f6fab7fe640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009890 tx=0x7f6f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.177+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c032840 con 0x7f6fac06b6c0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.177+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c0329a0 con 0x7f6fac06b6c0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.177+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c032d50 con 0x7f6fac06b6c0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.177+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac0acd70 con 0x7f6fac06b6c0 2023-11-24T08:17:11.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.177+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac0ad570 con 0x7f6fac06b6c0 2023-11-24T08:17:11.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.178+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac0ad290 con 0x7f6fac06b6c0 2023-11-24T08:17:11.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.180+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 31) v1 ==== 91322+0+0 (secure 0 0 0) 0x7f6f9c03b3f0 con 0x7f6fac06b6c0 2023-11-24T08:17:11.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.181+0000 7f6f93fff640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 0x7f6f9806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:11.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.181+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6f9c0aa310 con 0x7f6fac06b6c0 2023-11-24T08:17:11.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.182+0000 7f6faaffd640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 0x7f6f9806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:11.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.183+0000 7f6faaffd640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 0x7f6f9806ea10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6fac15d0a0 tx=0x7f6fa0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.186+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fac0ad290 con 0x7f6fac06b6c0 2023-11-24T08:17:11.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.404+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone9276", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fac06e550 con 0x7f6f9806c550 2023-11-24T08:17:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.444+0000 7f6f93fff640 1 -- 172.21.15.107:0/2783780910 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fa00012c0 con 0x7f6f9806c550 2023-11-24T08:17:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.447+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 msgr2=0x7f6f9806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.447+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 0x7f6f9806ea10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6fac15d0a0 tx=0x7f6fa0009040 comp rx=0 tx=0).stop 2023-11-24T08:17:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.447+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 msgr2=0x7f6fac0afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.447+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009890 tx=0x7f6f9c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 shutdown_connections 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f9806c550 0x7f6f9806ea10 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fac161060 0x7f6fac1f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fac0b3330 0x7f6fac0b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 --2- 172.21.15.107:0/2783780910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac06b6c0 0x7f6fac0afc20 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 >> 172.21.15.107:0/2783780910 conn(0x7f6fac0cf900 msgr2=0x7f6fac073740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 shutdown_connections 2023-11-24T08:17:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:11.448+0000 7f6fb1c7c640 1 -- 172.21.15.107:0/2783780910 wait complete. 2023-11-24T08:17:11.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:11.472 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:17:11.520 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:17:12.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:17:12.930 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:17:16.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:16.522 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:17:16.586 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:17:16.588 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:17:16.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:17:16.707 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:17:16.829 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:17:16.830 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:16.830 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:16.888 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:16.888 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:16.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.189+0000 7f284cbb7640 1 Processor -- start 2023-11-24T08:17:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.189+0000 7f284cbb7640 1 -- start start 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f284cbb7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 0x7f284815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f284cbb7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284815ab80 con 0x7f28480b2630 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f284cbb7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284815e680 con 0x7f28480b2a70 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f284cbb7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284815a990 con 0x7f28480b2250 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f283ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f28480b2630 0x7f28480ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34740/0 (socket says 172.21.15.107:34740) 2023-11-24T08:17:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.191+0000 7f283ffff640 1 -- 172.21.15.107:0/907331977 learned_addr learned my addr 172.21.15.107:0/907331977 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f284659c640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 0x7f284815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772498523 0 0) 0x7f284815ab80 con 0x7f28480b2630 2023-11-24T08:17:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834003520 con 0x7f28480b2630 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225462616 0 0) 0x7f2834003520 con 0x7f28480b2630 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 msgr2=0x7f284815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 0x7f284815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 >> v1:172.21.15.120:6789/0 conn(0x7f28480b2a70 legacy=0x7f284815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284815e960 con 0x7f28480b2630 2023-11-24T08:17:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2830001670 con 0x7f28480b2630 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 >> v1:172.21.15.107:6789/0 conn(0x7f28480b2630 legacy=0x7f28480ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f2834003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 0x7f28340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2834004d00 0x7f2834005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283400ba30 con 0x7f28340039a0 2023-11-24T08:17:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283400bb70 con 0x7f2834004d00 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.192+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283400df40 con 0x7f28480b2250 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f2834003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 -- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 msgr2=0x7f28340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480b2250 0x7f28340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 -- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2834004d00 msgr2=0x7f2834005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f2846d9d640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2834004d00 0x7f2834005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2834004d00 0x7f2834005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f284659c640 1 -- 172.21.15.107:0/907331977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834003520 con 0x7f28340039a0 2023-11-24T08:17:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.193+0000 7f2846d9d640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2834004d00 0x7f2834005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.194+0000 7f284659c640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f2834003dc0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f2838009a90 tx=0x7f28340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3e844571c332d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.194+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283802c6b0 con 0x7f28340039a0 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.194+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283802c810 con 0x7f28340039a0 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.194+0000 7f2845d9b640 1 -- 172.21.15.107:0/907331977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283802cba0 con 0x7f28340039a0 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 -- 172.21.15.107:0/907331977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 msgr2=0x7f2834003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 --2- 172.21.15.107:0/907331977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f2834003dc0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f2838009a90 tx=0x7f28340057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f2846d9d640 1 -- 172.21.15.107:0/907331977 reap_dead start 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 -- 172.21.15.107:0/907331977 shutdown_connections 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 -- 172.21.15.107:0/907331977 >> 172.21.15.107:0/907331977 conn(0x7f284806a920 msgr2=0x7f284806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 -- 172.21.15.107:0/907331977 shutdown_connections 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.195+0000 7f284cbb7640 1 -- 172.21.15.107:0/907331977 wait complete. 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.197+0000 7f284cbb7640 1 Processor -- start 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.197+0000 7f284cbb7640 1 -- start start 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284814b230 0x7f2848147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480c85b0 0x7f2848070a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848071830 con 0x7f28340039a0 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284814c690 con 0x7f284814b230 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284cbb7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28480abfc0 con 0x7f28480c85b0 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41414/0 (socket says 172.21.15.107:41414) 2023-11-24T08:17:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.198+0000 7f284659c640 1 -- 172.21.15.107:0/87864009 learned_addr learned my addr 172.21.15.107:0/87864009 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 -- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480c85b0 msgr2=0x7f2848070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480c85b0 0x7f2848070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 -- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284814b230 msgr2=0x7f2848147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284814b230 0x7f2848147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 -- 172.21.15.107:0/87864009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28480cd7c0 con 0x7f28340039a0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284659c640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2838000c00 tx=0x7f2838004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838003810 con 0x7f28340039a0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2838003970 con 0x7f28340039a0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838003d20 con 0x7f28340039a0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28480ac860 con 0x7f28340039a0 2023-11-24T08:17:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.199+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2848147a90 con 0x7f28340039a0 2023-11-24T08:17:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.201+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2848067470 con 0x7f28340039a0 2023-11-24T08:17:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.201+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f283803b070 con 0x7f28340039a0 2023-11-24T08:17:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.202+0000 7f283effd640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 0x7f282806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.202+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f28380aa2b0 con 0x7f28340039a0 2023-11-24T08:17:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.203+0000 7f283ffff640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 0x7f282806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.203+0000 7f283ffff640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 0x7f282806ead0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f28481f6930 tx=0x7f2830001520 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.207+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2838077010 con 0x7f28340039a0 2023-11-24T08:17:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.397+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28480cb7a0 con 0x7f28340039a0 2023-11-24T08:17:17.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.400+0000 7f283effd640 1 -- 172.21.15.107:0/87864009 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16740 (secure 0 0 0) 0x7f283807a630 con 0x7f28340039a0 2023-11-24T08:17:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:17.403 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:16.002760+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 msgr2=0x7f282806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 0x7f282806ead0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f28481f6930 tx=0x7f2830001520 comp rx=0 tx=0).stop 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 msgr2=0x7f284814c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2838000c00 tx=0x7f2838004730 comp rx=0 tx=0).stop 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 shutdown_connections 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f282806c610 0x7f282806ead0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28480c85b0 0x7f2848070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284814b230 0x7f2848147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 --2- 172.21.15.107:0/87864009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28340039a0 0x7f284814c150 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 >> 172.21.15.107:0/87864009 conn(0x7f284806a920 msgr2=0x7f284815bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:17.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 shutdown_connections 2023-11-24T08:17:17.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.403+0000 7f284cbb7640 1 -- 172.21.15.107:0/87864009 wait complete. 2023-11-24T08:17:17.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:16:03.725981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:16:04.911827+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 150, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:17:17.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:17:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.634+0000 7f0de29e2640 1 Processor -- start 2023-11-24T08:17:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.634+0000 7f0de29e2640 1 -- start start 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ddc15b260 0x7f0ddc06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ddc06e580 0x7f0ddc0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc15b640 con 0x7f0ddc06e580 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc0b1510 con 0x7f0ddc15b260 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.635+0000 7f0de29e2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc15ed20 con 0x7f0ddc0b1080 2023-11-24T08:17:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ddc06e580 0x7f0ddc0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ddc06e580 0x7f0ddc0b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41418/0 (socket says 172.21.15.107:41418) 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 -- 172.21.15.107:0/2873856799 learned_addr learned my addr 172.21.15.107:0/2873856799 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de19e0640 1 --2- 172.21.15.107:0/2873856799 >> v2:172.21.15.120:3300/0 conn(0x7f0ddc15b260 0x7f0ddc06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de21e1640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 -- 172.21.15.107:0/2873856799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 msgr2=0x7f0ddc0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 -- 172.21.15.107:0/2873856799 >> v2:172.21.15.120:3300/0 conn(0x7f0ddc15b260 msgr2=0x7f0ddc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 --2- 172.21.15.107:0/2873856799 >> v2:172.21.15.120:3300/0 conn(0x7f0ddc15b260 0x7f0ddc06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.636+0000 7f0de11df640 1 -- 172.21.15.107:0/2873856799 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc15f8a0 con 0x7f0ddc06e580 2023-11-24T08:17:17.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de11df640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc0b0b40 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0d0160 tx=0x7f0dcc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6e78549d83b00f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de09de640 1 -- 172.21.15.107:0/2873856799 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc031740 con 0x7f0ddc06e580 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de09de640 1 -- 172.21.15.107:0/2873856799 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc0318a0 con 0x7f0ddc06e580 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de09de640 1 -- 172.21.15.107:0/2873856799 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc031c30 con 0x7f0ddc06e580 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de29e2640 1 -- 172.21.15.107:0/2873856799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 msgr2=0x7f0ddc0b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.638+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc0b0b40 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0d0160 tx=0x7f0dcc02b7e0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 -- 172.21.15.107:0/2873856799 shutdown_connections 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2873856799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc0b0b40 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2873856799 >> v2:172.21.15.120:3300/0 conn(0x7f0ddc15b260 0x7f0ddc06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 -- 172.21.15.107:0/2873856799 >> 172.21.15.107:0/2873856799 conn(0x7f0ddc0cbb40 msgr2=0x7f0ddc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 -- 172.21.15.107:0/2873856799 shutdown_connections 2023-11-24T08:17:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.639+0000 7f0de29e2640 1 -- 172.21.15.107:0/2873856799 wait complete. 2023-11-24T08:17:17.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.640+0000 7f0de29e2640 1 Processor -- start 2023-11-24T08:17:17.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 -- start start 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc1e49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ddc15b260 0x7f0ddc157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc09bad0 con 0x7f0ddc06e580 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc14da80 con 0x7f0ddc15b260 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.641+0000 7f0de29e2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc09ae50 con 0x7f0ddc0b1080 2023-11-24T08:17:17.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41426/0 (socket says 172.21.15.107:41426) 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 -- 172.21.15.107:0/2612847686 learned_addr learned my addr 172.21.15.107:0/2612847686 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 -- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 msgr2=0x7f0ddc1e49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc1e49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 -- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ddc15b260 msgr2=0x7f0ddc157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ddc15b260 0x7f0ddc157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 -- 172.21.15.107:0/2612847686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0094d0 con 0x7f0ddc06e580 2023-11-24T08:17:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.642+0000 7f0de19e0640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800d5f0 tx=0x7f0dd800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.643+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80125f0 con 0x7f0ddc06e580 2023-11-24T08:17:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.643+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd8012750 con 0x7f0ddc06e580 2023-11-24T08:17:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.643+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8012b00 con 0x7f0ddc06e580 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.643+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc09b130 con 0x7f0ddc06e580 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.643+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc09b6f0 con 0x7f0ddc06e580 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.644+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc04e840 con 0x7f0ddc06e580 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.645+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f0dd8012c60 con 0x7f0ddc06e580 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.646+0000 7f0dd27fc640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 0x7f0dc006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.646+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f0dd8088850 con 0x7f0ddc06e580 2023-11-24T08:17:17.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.646+0000 7f0de11df640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 0x7f0dc006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:17.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.647+0000 7f0de11df640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 0x7f0dc006eaa0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc035000 tx=0x7f0dcc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:17.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.650+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0dd8055360 con 0x7f0ddc06e580 2023-11-24T08:17:17.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.839+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ddc15ea70 con 0x7f0ddc06e580 2023-11-24T08:17:17.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.839+0000 7f0dd27fc640 1 -- 172.21.15.107:0/2612847686 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0dd8058980 con 0x7f0ddc06e580 2023-11-24T08:17:17.841 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:17:17.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 msgr2=0x7f0dc006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 0x7f0dc006eaa0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc035000 tx=0x7f0dcc030000 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 msgr2=0x7f0ddc14d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800d5f0 tx=0x7f0dd800dab0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 shutdown_connections 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dc006c5e0 0x7f0dc006eaa0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ddc15b260 0x7f0ddc157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ddc0b1080 0x7f0ddc1e49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 --2- 172.21.15.107:0/2612847686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ddc06e580 0x7f0ddc14d540 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.842+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 >> 172.21.15.107:0/2612847686 conn(0x7f0ddc0cbb40 msgr2=0x7f0ddc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.843+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 shutdown_connections 2023-11-24T08:17:17.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:17.843+0000 7f0de29e2640 1 -- 172.21.15.107:0/2612847686 wait complete. 2023-11-24T08:17:17.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:17:18.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.082+0000 7fe43a9d5640 1 Processor -- start 2023-11-24T08:17:18.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.082+0000 7fe43a9d5640 1 -- start start 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.083+0000 7fe43a9d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe43415b440 0x7fe4340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.083+0000 7fe43a9d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4340cbf20 0x7fe4340c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe43a9d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe4340c57e0 0x7fe4340c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe43a9d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43415b820 con 0x7fe4340cbf20 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe43a9d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4340cc3a0 con 0x7fe4340c57e0 2023-11-24T08:17:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe43a9d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43415eea0 con 0x7fe43415b440 2023-11-24T08:17:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe4337fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4340cbf20 0x7fe4340c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe4337fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4340cbf20 0x7fe4340c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40436/0 (socket says 172.21.15.107:40436) 2023-11-24T08:17:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe4337fe640 1 -- 172.21.15.107:0/3295195626 learned_addr learned my addr 172.21.15.107:0/3295195626 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe433fff640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe43415b440 0x7fe4340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.084+0000 7fe438f72640 1 --2- 172.21.15.107:0/3295195626 >> v2:172.21.15.120:3300/0 conn(0x7fe4340c57e0 0x7fe4340c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.085+0000 7fe4337fe640 1 -- 172.21.15.107:0/3295195626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe43415b440 msgr2=0x7fe4340cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.085+0000 7fe4337fe640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe43415b440 0x7fe4340cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.085+0000 7fe4337fe640 1 -- 172.21.15.107:0/3295195626 >> v2:172.21.15.120:3300/0 conn(0x7fe4340c57e0 msgr2=0x7fe4340c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.085+0000 7fe4337fe640 1 --2- 172.21.15.107:0/3295195626 >> v2:172.21.15.120:3300/0 conn(0x7fe4340c57e0 0x7fe4340c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.085+0000 7fe4337fe640 1 -- 172.21.15.107:0/3295195626 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4340abf30 con 0x7fe4340cbf20 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.086+0000 7fe4337fe640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340cbf20 0x7fe4340c52a0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe4340cb910 tx=0x7fe42802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2164c28cb6aec7aa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.086+0000 7fe432ffd640 1 -- 172.21.15.107:0/3295195626 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4280317e0 con 0x7fe4340cbf20 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.086+0000 7fe432ffd640 1 -- 172.21.15.107:0/3295195626 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe428031940 con 0x7fe4340cbf20 2023-11-24T08:17:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.086+0000 7fe432ffd640 1 -- 172.21.15.107:0/3295195626 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428031cd0 con 0x7fe4340cbf20 2023-11-24T08:17:18.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 -- 172.21.15.107:0/3295195626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340cbf20 msgr2=0x7fe4340c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340cbf20 0x7fe4340c52a0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe4340cb910 tx=0x7fe42802b880 comp rx=0 tx=0).stop 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 -- 172.21.15.107:0/3295195626 shutdown_connections 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/3295195626 >> v2:172.21.15.120:3300/0 conn(0x7fe4340c57e0 0x7fe4340c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340cbf20 0x7fe4340c52a0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/3295195626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe43415b440 0x7fe4340cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 -- 172.21.15.107:0/3295195626 >> 172.21.15.107:0/3295195626 conn(0x7fe4340b0c50 msgr2=0x7fe4340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 -- 172.21.15.107:0/3295195626 shutdown_connections 2023-11-24T08:17:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.087+0000 7fe43a9d5640 1 -- 172.21.15.107:0/3295195626 wait complete. 2023-11-24T08:17:18.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.089+0000 7fe43a9d5640 1 Processor -- start 2023-11-24T08:17:18.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.089+0000 7fe43a9d5640 1 -- start start 2023-11-24T08:17:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4340cbf20 0x7fe43409b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 0x7fe4341562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43409bdf0 con 0x7fe4340c57e0 2023-11-24T08:17:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4340b2d40 con 0x7fe43415b440 2023-11-24T08:17:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe43a9d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4340b3020 con 0x7fe4340cbf20 2023-11-24T08:17:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe433fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe433fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40440/0 (socket says 172.21.15.107:40440) 2023-11-24T08:17:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.090+0000 7fe433fff640 1 -- 172.21.15.107:0/1362689263 learned_addr learned my addr 172.21.15.107:0/1362689263 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe438f72640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 0x7fe4341562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe4337fe640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4340cbf20 0x7fe43409b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 -- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4340cbf20 msgr2=0x7fe43409b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4340cbf20 0x7fe43409b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 -- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 msgr2=0x7fe4341562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 0x7fe4341562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 -- 172.21.15.107:0/1362689263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4280094d0 con 0x7fe4340c57e0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe438f72640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 0x7fe4341562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.091+0000 7fe433fff640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe42400d5f0 tx=0x7fe42400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.092+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4240125f0 con 0x7fe4340c57e0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.092+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe424012750 con 0x7fe4340c57e0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.092+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4340b3300 con 0x7fe4340c57e0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.092+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe424012b00 con 0x7fe4340c57e0 2023-11-24T08:17:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.092+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe434156800 con 0x7fe4340c57e0 2023-11-24T08:17:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.093+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4340cfe50 con 0x7fe4340c57e0 2023-11-24T08:17:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.094+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fe424012c60 con 0x7fe4340c57e0 2023-11-24T08:17:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.095+0000 7fe430ff9640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 0x7fe42006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.095+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7fe424088860 con 0x7fe4340c57e0 2023-11-24T08:17:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.095+0000 7fe4337fe640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 0x7fe42006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.096+0000 7fe4337fe640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 0x7fe42006eaa0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe428036000 tx=0x7fe42803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.099+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe424055410 con 0x7fe4340c57e0 2023-11-24T08:17:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.291+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe43404e8b0 con 0x7fe4340c57e0 2023-11-24T08:17:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.291+0000 7fe430ff9640 1 -- 172.21.15.107:0/1362689263 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe424058a30 con 0x7fe4340c57e0 2023-11-24T08:17:18.293 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 msgr2=0x7fe42006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 0x7fe42006eaa0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe428036000 tx=0x7fe42803d040 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 msgr2=0x7fe4340b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe42400d5f0 tx=0x7fe42400dab0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 shutdown_connections 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe42006c5e0 0x7fe42006eaa0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe43415b440 0x7fe4341562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4340cbf20 0x7fe43409b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 --2- 172.21.15.107:0/1362689263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4340c57e0 0x7fe4340b2720 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.294+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 >> 172.21.15.107:0/1362689263 conn(0x7fe4340b0c50 msgr2=0x7fe43406c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.295+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 shutdown_connections 2023-11-24T08:17:18.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.295+0000 7fe43a9d5640 1 -- 172.21.15.107:0/1362689263 wait complete. 2023-11-24T08:17:18.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:17:18.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.529+0000 7f7333ddc640 1 Processor -- start 2023-11-24T08:17:18.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.530+0000 7f7333ddc640 1 -- start start 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.531+0000 7f7333ddc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 0x7f732c1684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.531+0000 7f7333ddc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c161a10 con 0x7f732c06e160 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.531+0000 7f7333ddc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c161cf0 con 0x7f732c146eb0 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.531+0000 7f7333ddc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c161fd0 con 0x7f732c0cd2e0 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.531+0000 7f7331377640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f732c06e160 0x7f732c0ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39122/0 (socket says 172.21.15.107:39122) 2023-11-24T08:17:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7331377640 1 -- 172.21.15.107:0/3045435484 learned_addr learned my addr 172.21.15.107:0/3045435484 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7332379640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 0x7f732c1684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626770308 0 0) 0x7f732c161a10 con 0x7f732c06e160 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318003520 con 0x7f732c06e160 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803970960 0 0) 0x7f7318003520 con 0x7f732c06e160 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 msgr2=0x7f732c1684e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 0x7f732c1684e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 >> v1:172.21.15.120:6789/0 conn(0x7f732c146eb0 legacy=0x7f732c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c070660 con 0x7f732c06e160 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f731c001690 con 0x7f732c06e160 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 >> v1:172.21.15.107:6789/0 conn(0x7f732c06e160 legacy=0x7f732c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73180039a0 0x7f7318003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.532+0000 7f7330b76640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f7318004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7330b76640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 0x7f7318005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c161fd0 con 0x7f73180047c0 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731800bab0 con 0x7f73180039a0 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731800bc70 con 0x7f732c0cd2e0 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f7318004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 -- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 msgr2=0x7f7318005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0cd2e0 0x7f7318005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 -- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73180039a0 msgr2=0x7f7318003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73180039a0 0x7f7318003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.533+0000 7f7331377640 1 -- 172.21.15.107:0/3045435484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318003520 con 0x7f73180047c0 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.534+0000 7f7331377640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f7318004c40 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f731c00c0e0 tx=0x7f73180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ca36e8fe8f6b07 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.534+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c005ad0 con 0x7f73180047c0 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.534+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c005c50 con 0x7f73180047c0 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.534+0000 7f7330b76640 1 -- 172.21.15.107:0/3045435484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c02c7d0 con 0x7f73180047c0 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 -- 172.21.15.107:0/3045435484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 msgr2=0x7f7318004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 --2- 172.21.15.107:0/3045435484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f7318004c40 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f731c00c0e0 tx=0x7f73180057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7332379640 1 -- 172.21.15.107:0/3045435484 reap_dead start 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 -- 172.21.15.107:0/3045435484 shutdown_connections 2023-11-24T08:17:18.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 -- 172.21.15.107:0/3045435484 >> 172.21.15.107:0/3045435484 conn(0x7f732c0b0c50 msgr2=0x7f732c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 -- 172.21.15.107:0/3045435484 shutdown_connections 2023-11-24T08:17:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.535+0000 7f7333ddc640 1 -- 172.21.15.107:0/3045435484 wait complete. 2023-11-24T08:17:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.537+0000 7f7333ddc640 1 Processor -- start 2023-11-24T08:17:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.537+0000 7f7333ddc640 1 -- start start 2023-11-24T08:17:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0b4550 0x7f732c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f732c0ad490 0x7f732c0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0b4ef0 con 0x7f73180047c0 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0ade50 con 0x7f732c0ad490 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7333ddc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0b1e70 con 0x7f732c0b4550 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7331b78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7331b78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40452/0 (socket says 172.21.15.107:40452) 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.538+0000 7f7331b78640 1 -- 172.21.15.107:0/2339867419 learned_addr learned my addr 172.21.15.107:0/2339867419 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 -- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0b4550 msgr2=0x7f732c0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331377640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0b4550 0x7f732c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0b4550 0x7f732c0b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 -- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f732c0ad490 msgr2=0x7f732c0ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f732c0ad490 0x7f732c0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 -- 172.21.15.107:0/2339867419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c15f850 con 0x7f73180047c0 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7331b78640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f732800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280052d0 con 0x7f73180047c0 2023-11-24T08:17:18.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328005450 con 0x7f73180047c0 2023-11-24T08:17:18.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328012870 con 0x7f73180047c0 2023-11-24T08:17:18.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c15fae0 con 0x7f73180047c0 2023-11-24T08:17:18.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.539+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c1600a0 con 0x7f73180047c0 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.541+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c146a00 con 0x7f73180047c0 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.542+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f73280129d0 con 0x7f73180047c0 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.543+0000 7f73227fc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 0x7f731006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.543+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f7328087f20 con 0x7f73180047c0 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.543+0000 7f7331377640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 0x7f731006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.544+0000 7f7331377640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 0x7f731006eaa0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f731c034000 tx=0x7f731c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.547+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7328054a30 con 0x7f73180047c0 2023-11-24T08:17:18.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.737+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f732c04e8b0 con 0x7f73180047c0 2023-11-24T08:17:18.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.738+0000 7f73227fc640 1 -- 172.21.15.107:0/2339867419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7328058050 con 0x7f73180047c0 2023-11-24T08:17:18.741 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:17:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 msgr2=0x7f731006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 0x7f731006eaa0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f731c034000 tx=0x7f731c009030 comp rx=0 tx=0).stop 2023-11-24T08:17:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 msgr2=0x7f732c15c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f732800c960 comp rx=0 tx=0).stop 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 shutdown_connections 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f731006c5e0 0x7f731006eaa0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f732c0ad490 0x7f732c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f732c0b4550 0x7f732c0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 --2- 172.21.15.107:0/2339867419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73180047c0 0x7f732c15c8b0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 >> 172.21.15.107:0/2339867419 conn(0x7f732c0b0c50 msgr2=0x7f732c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 shutdown_connections 2023-11-24T08:17:18.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.741+0000 7f7333ddc640 1 -- 172.21.15.107:0/2339867419 wait complete. 2023-11-24T08:17:18.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:17:18.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.968+0000 7ff68a621640 1 Processor -- start 2023-11-24T08:17:18.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.969+0000 7ff68a621640 1 -- start start 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.970+0000 7ff68a621640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff68415d490 0x7ff6840cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.970+0000 7ff68a621640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b63d0 0x7ff6840d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.970+0000 7ff68a621640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6840cea30 con 0x7ff6840ce270 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.970+0000 7ff68a621640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6840d0450 con 0x7ff68415d490 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.970+0000 7ff68a621640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840d0bf0 con 0x7ff6840b63d0 2023-11-24T08:17:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff689e20640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff6840ce270 0x7ff684160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39138/0 (socket says 172.21.15.107:39138) 2023-11-24T08:17:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff689e20640 1 -- 172.21.15.107:0/4140767430 learned_addr learned my addr 172.21.15.107:0/4140767430 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff688e1e640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b63d0 0x7ff6840d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff68961f640 1 --2- 172.21.15.107:0/4140767430 >> v2:172.21.15.120:3300/0 conn(0x7ff68415d490 0x7ff6840cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186233243 0 0) 0x7ff6840cea30 con 0x7ff6840ce270 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c003520 con 0x7ff6840ce270 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3367687320 0 0) 0x7ff66c003520 con 0x7ff6840ce270 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b63d0 msgr2=0x7ff6840d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b63d0 0x7ff6840d3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 >> v2:172.21.15.120:3300/0 conn(0x7ff68415d490 msgr2=0x7ff6840cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 --2- 172.21.15.107:0/4140767430 >> v2:172.21.15.120:3300/0 conn(0x7ff68415d490 0x7ff6840cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840d2260 con 0x7ff6840ce270 2023-11-24T08:17:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff67c001670 con 0x7ff6840ce270 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 >> v1:172.21.15.107:6789/0 conn(0x7ff6840ce270 legacy=0x7ff684160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.971+0000 7ff683fff640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff66c003970 0x7ff66c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff683fff640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66c0047c0 0x7ff66c004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff683fff640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c005140 0x7ff66c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840d0450 con 0x7ff66c005140 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840d0bf0 con 0x7ff66c003970 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c004610 con 0x7ff66c0047c0 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c005140 0x7ff66c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 -- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66c0047c0 msgr2=0x7ff66c004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66c0047c0 0x7ff66c004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 -- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff66c003970 msgr2=0x7ff66c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:18.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff66c003970 0x7ff66c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.972+0000 7ff689e20640 1 -- 172.21.15.107:0/4140767430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c003520 con 0x7ff66c005140 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.973+0000 7ff689e20640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c005140 0x7ff66c00b980 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00c0e0 tx=0x7ff66c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec1ed788095ac4f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.973+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c005ad0 con 0x7ff66c005140 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c005c50 con 0x7ff66c005140 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff683fff640 1 -- 172.21.15.107:0/4140767430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c02c810 con 0x7ff66c005140 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 -- 172.21.15.107:0/4140767430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c005140 msgr2=0x7ff66c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 --2- 172.21.15.107:0/4140767430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c005140 0x7ff66c00b980 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00c0e0 tx=0x7ff66c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff689e20640 1 -- 172.21.15.107:0/4140767430 reap_dead start 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 -- 172.21.15.107:0/4140767430 shutdown_connections 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 -- 172.21.15.107:0/4140767430 >> 172.21.15.107:0/4140767430 conn(0x7ff684061340 msgr2=0x7ff684061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 -- 172.21.15.107:0/4140767430 shutdown_connections 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.974+0000 7ff68a621640 1 -- 172.21.15.107:0/4140767430 wait complete. 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.976+0000 7ff68a621640 1 Processor -- start 2023-11-24T08:17:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.976+0000 7ff68a621640 1 -- start start 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 0x7ff6840ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b7e40 0x7ff6840b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff684155ce0 con 0x7ff6840c97d0 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840b8800 con 0x7ff6840ca370 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68a621640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840caec0 con 0x7ff6840b7e40 2023-11-24T08:17:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68961f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68961f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40466/0 (socket says 172.21.15.107:40466) 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.977+0000 7ff68961f640 1 -- 172.21.15.107:0/4129851566 learned_addr learned my addr 172.21.15.107:0/4129851566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff688e1e640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 0x7ff6840ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff689e20640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b7e40 0x7ff6840b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 -- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b7e40 msgr2=0x7ff6840b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b7e40 0x7ff6840b82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 -- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 msgr2=0x7ff6840ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 0x7ff6840ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 -- 172.21.15.107:0/4129851566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840cb1a0 con 0x7ff6840c97d0 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff688e1e640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 0x7ff6840ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68961f640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7ff67400efa0 tx=0x7ff67400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674009ea0 con 0x7ff6840c97d0 2023-11-24T08:17:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67400f040 con 0x7ff6840c97d0 2023-11-24T08:17:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6840cb970 con 0x7ff6840c97d0 2023-11-24T08:17:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.978+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674003bd0 con 0x7ff6840c97d0 2023-11-24T08:17:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.979+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6840cbf30 con 0x7ff6840c97d0 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.980+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6840cc0f0 con 0x7ff6840c97d0 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.981+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7ff674003d30 con 0x7ff6840c97d0 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.982+0000 7ff681ffb640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 0x7ff67006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.982+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7ff674088d80 con 0x7ff6840c97d0 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.982+0000 7ff688e1e640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 0x7ff67006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.983+0000 7ff688e1e640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 0x7ff67006ead0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff6841515f0 tx=0x7ff678008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:18.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:18.986+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff674055890 con 0x7ff6840c97d0 2023-11-24T08:17:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.193+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff68404e840 con 0x7ff6840c97d0 2023-11-24T08:17:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.194+0000 7ff681ffb640 1 -- 172.21.15.107:0/4129851566 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v150) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff674058eb0 con 0x7ff6840c97d0 2023-11-24T08:17:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:17:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 msgr2=0x7ff67006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 0x7ff67006ead0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff6841515f0 tx=0x7ff678008040 comp rx=0 tx=0).stop 2023-11-24T08:17:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 msgr2=0x7ff6841557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7ff67400efa0 tx=0x7ff67400c490 comp rx=0 tx=0).stop 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 shutdown_connections 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff67006c610 0x7ff67006ead0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6840b7e40 0x7ff6840b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6840ca370 0x7ff6840ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 --2- 172.21.15.107:0/4129851566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6840c97d0 0x7ff6841557a0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.196+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 >> 172.21.15.107:0/4129851566 conn(0x7ff684061340 msgr2=0x7ff6840b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.197+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 shutdown_connections 2023-11-24T08:17:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.197+0000 7ff68a621640 1 -- 172.21.15.107:0/4129851566 wait complete. 2023-11-24T08:17:19.217 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:17:19.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:19.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:17:19.353 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:17:19.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:19.354 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:17:19.407 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:19.407 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:19.448 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:19.448 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:19.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.749+0000 7f71e929b640 1 Processor -- start 2023-11-24T08:17:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.749+0000 7f71e929b640 1 -- start start 2023-11-24T08:17:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e929b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71e415c7f0 0x7f71e406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e929b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71e40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e929b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40cece0 con 0x7f71e40cf450 2023-11-24T08:17:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e929b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e40cfe40 con 0x7f71e415c7f0 2023-11-24T08:17:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e929b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4156010 con 0x7f71e40ce6c0 2023-11-24T08:17:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71da7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71e40cf450 0x7f71e40cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39150/0 (socket says 172.21.15.107:39150) 2023-11-24T08:17:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71da7fc640 1 -- 172.21.15.107:0/1187590400 learned_addr learned my addr 172.21.15.107:0/1187590400 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e37fe640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71e40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.751+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/1187590400 >> v2:172.21.15.120:3300/0 conn(0x7f71e415c7f0 0x7f71e406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497853526 0 0) 0x7f71e40cece0 con 0x7f71e40cf450 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d0003520 con 0x7f71e40cf450 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976618401 0 0) 0x7f71d0003520 con 0x7f71e40cf450 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 msgr2=0x7f71e40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71e40cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 >> v2:172.21.15.120:3300/0 conn(0x7f71e415c7f0 msgr2=0x7f71e406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 --2- 172.21.15.107:0/1187590400 >> v2:172.21.15.120:3300/0 conn(0x7f71e415c7f0 0x7f71e406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e406d040 con 0x7f71e40cf450 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f71cc001670 con 0x7f71e40cf450 2023-11-24T08:17:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 >> v1:172.21.15.107:6789/0 conn(0x7f71e40cf450 legacy=0x7f71e40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d0003970 0x7f71d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40cfe40 con 0x7f71d0004d00 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4156010 con 0x7f71d0003970 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d000b980 con 0x7f71e40ce6c0 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.752+0000 7f71e37fe640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71da7fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d0003970 0x7f71d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e37fe640 1 -- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 msgr2=0x7f71d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e37fe640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e37fe640 1 -- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d0003970 msgr2=0x7f71d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e37fe640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d0003970 0x7f71d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e37fe640 1 -- 172.21.15.107:0/1187590400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d0003520 con 0x7f71d0004d00 2023-11-24T08:17:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d0003970 0x7f71d0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.753+0000 7f71da7fc640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e40ce6c0 0x7f71d00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.754+0000 7f71e37fe640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71d0005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009fc0 tx=0x7f71d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a513ae92ca1e73d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.754+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc02cbf0 con 0x7f71d0004d00 2023-11-24T08:17:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.754+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc00f040 con 0x7f71d0004d00 2023-11-24T08:17:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.754+0000 7f71e27fc640 1 -- 172.21.15.107:0/1187590400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc034440 con 0x7f71d0004d00 2023-11-24T08:17:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 -- 172.21.15.107:0/1187590400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 msgr2=0x7f71d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 --2- 172.21.15.107:0/1187590400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71d0005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009fc0 tx=0x7f71d00057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e37fe640 1 -- 172.21.15.107:0/1187590400 reap_dead start 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 -- 172.21.15.107:0/1187590400 shutdown_connections 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 -- 172.21.15.107:0/1187590400 >> 172.21.15.107:0/1187590400 conn(0x7f71e406a920 msgr2=0x7f71e406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 -- 172.21.15.107:0/1187590400 shutdown_connections 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.755+0000 7f71e929b640 1 -- 172.21.15.107:0/1187590400 wait complete. 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.757+0000 7f71e929b640 1 Processor -- start 2023-11-24T08:17:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.757+0000 7f71e929b640 1 -- start start 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 0x7f71e4158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e4156150 0x7f71e41565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4151d30 con 0x7f71d0004d00 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41f3d50 con 0x7f71e4156150 2023-11-24T08:17:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e929b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41f34b0 con 0x7f71e415b430 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.758+0000 7f71e2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40482/0 (socket says 172.21.15.107:40482) 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2827200328 learned_addr learned my addr 172.21.15.107:0/2827200328 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71da7fc640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 0x7f71e4158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e37fe640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e4156150 0x7f71e41565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 msgr2=0x7f71e4158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 0x7f71e4158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e4156150 msgr2=0x7f71e41565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e4156150 0x7f71e41565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2827200328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41f3790 con 0x7f71d0004d00 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71da7fc640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 0x7f71e4158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.759+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f71d40109d0 tx=0x7f71d4010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.760+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d400ec20 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.760+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d400ed80 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.760+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41f3a70 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.760+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d401c620 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.760+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e415b870 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.761+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e4067470 con 0x7f71d0004d00 2023-11-24T08:17:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.763+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f71d401c780 con 0x7f71d0004d00 2023-11-24T08:17:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.764+0000 7f71dbfff640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 0x7f71c406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.764+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f71d4088ab0 con 0x7f71d0004d00 2023-11-24T08:17:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.764+0000 7f71da7fc640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 0x7f71c406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.764+0000 7f71da7fc640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 0x7f71c406eaa0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f71cc001200 tx=0x7f71cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.768+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71d40555c0 con 0x7f71d0004d00 2023-11-24T08:17:19.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.959+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71e40cd730 con 0x7f71d0004d00 2023-11-24T08:17:19.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.960+0000 7f71dbfff640 1 -- 172.21.15.107:0/2827200328 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16740 (secure 0 0 0) 0x7f71d4058be0 con 0x7f71d0004d00 2023-11-24T08:17:19.963 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:19.963 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:16.002760+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:19.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 msgr2=0x7f71c406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 0x7f71c406eaa0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f71cc001200 tx=0x7f71cc009040 comp rx=0 tx=0).stop 2023-11-24T08:17:19.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 msgr2=0x7f71e41f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:19.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f71d40109d0 tx=0x7f71d4010e90 comp rx=0 tx=0).stop 2023-11-24T08:17:19.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 shutdown_connections 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f71c406c5e0 0x7f71c406eaa0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e4156150 0x7f71e41565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71e415b430 0x7f71e4158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 --2- 172.21.15.107:0/2827200328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71d0004d00 0x7f71e41f2e10 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 >> 172.21.15.107:0/2827200328 conn(0x7f71e406a920 msgr2=0x7f71e40cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 shutdown_connections 2023-11-24T08:17:19.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:19.964+0000 7f71e929b640 1 -- 172.21.15.107:0/2827200328 wait complete. 2023-11-24T08:17:19.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:16:03.725981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:16:04.911827+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 150, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:17:19.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:17:20.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.202+0000 7fd2cc1d5640 1 Processor -- start 2023-11-24T08:17:20.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.202+0000 7fd2cc1d5640 1 -- start start 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2cc1d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2c415b440 0x7fd2c40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2cc1d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2c40cbf20 0x7fd2c40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2cc1d5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c415eea0 con 0x7fd2c40c57e0 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2cc1d5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c40cc3d0 con 0x7fd2c40cbf20 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2cc1d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c415b820 con 0x7fd2c415b440 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2ca772640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd2c40c57e0 0x7fd2c40c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39156/0 (socket says 172.21.15.107:39156) 2023-11-24T08:17:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2ca772640 1 -- 172.21.15.107:0/1815251648 learned_addr learned my addr 172.21.15.107:0/1815251648 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:20.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.204+0000 7fd2c9770640 1 --2- 172.21.15.107:0/1815251648 >> v2:172.21.15.107:3300/0 conn(0x7fd2c40cbf20 0x7fd2c40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c9f71640 1 --2- 172.21.15.107:0/1815251648 >> v2:172.21.15.120:3300/0 conn(0x7fd2c415b440 0x7fd2c40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c8f6f640 1 -- 172.21.15.107:0/1815251648 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109912305 0 0) 0x7fd2c415eea0 con 0x7fd2c40c57e0 2023-11-24T08:17:20.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c8f6f640 1 -- 172.21.15.107:0/1815251648 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0003520 con 0x7fd2c40c57e0 2023-11-24T08:17:20.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c9770640 1 -- 172.21.15.107:0/1815251648 >> v2:172.21.15.120:3300/0 conn(0x7fd2c415b440 msgr2=0x7fd2c40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:20.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c9770640 1 --2- 172.21.15.107:0/1815251648 >> v2:172.21.15.120:3300/0 conn(0x7fd2c415b440 0x7fd2c40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c9770640 1 -- 172.21.15.107:0/1815251648 >> v1:172.21.15.107:6789/0 conn(0x7fd2c40c57e0 legacy=0x7fd2c40c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.205+0000 7fd2c9770640 1 -- 172.21.15.107:0/1815251648 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c406d600 con 0x7fd2c40cbf20 2023-11-24T08:17:20.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.206+0000 7fd2c9770640 1 --2- 172.21.15.107:0/1815251648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c40cbf20 0x7fd2c40c52a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fd2c406e770 tx=0x7fd2b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b864653aafb954 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.207+0000 7fd2c8f6f640 1 -- 172.21.15.107:0/1815251648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b402e070 con 0x7fd2c40cbf20 2023-11-24T08:17:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.207+0000 7fd2c8f6f640 1 -- 172.21.15.107:0/1815251648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b40315a0 con 0x7fd2c40cbf20 2023-11-24T08:17:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.207+0000 7fd2c8f6f640 1 -- 172.21.15.107:0/1815251648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4031930 con 0x7fd2c40cbf20 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.207+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/1815251648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c40cbf20 msgr2=0x7fd2c40c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.207+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/1815251648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c40cbf20 0x7fd2c40c52a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fd2c406e770 tx=0x7fd2b402b840 comp rx=0 tx=0).stop 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/1815251648 shutdown_connections 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/1815251648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c40cbf20 0x7fd2c40c52a0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/1815251648 >> v2:172.21.15.120:3300/0 conn(0x7fd2c415b440 0x7fd2c40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/1815251648 >> 172.21.15.107:0/1815251648 conn(0x7fd2c40b0c50 msgr2=0x7fd2c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/1815251648 shutdown_connections 2023-11-24T08:17:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.208+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/1815251648 wait complete. 2023-11-24T08:17:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.209+0000 7fd2cc1d5640 1 Processor -- start 2023-11-24T08:17:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.210+0000 7fd2cc1d5640 1 -- start start 2023-11-24T08:17:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.210+0000 7fd2cc1d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2c40c57e0 0x7fd2c41f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2cc1d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2c40cbf20 0x7fd2c4155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2cc1d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2cc1d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4154640 con 0x7fd2c415b440 2023-11-24T08:17:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2cc1d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41568a0 con 0x7fd2c40c57e0 2023-11-24T08:17:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2cc1d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4156b80 con 0x7fd2c40cbf20 2023-11-24T08:17:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40498/0 (socket says 172.21.15.107:40498) 2023-11-24T08:17:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 -- 172.21.15.107:0/183156898 learned_addr learned my addr 172.21.15.107:0/183156898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:20.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2c9f71640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2c40c57e0 0x7fd2c41f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2c9770640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2c40cbf20 0x7fd2c4155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 -- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2c40cbf20 msgr2=0x7fd2c4155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2c40cbf20 0x7fd2c4155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 -- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2c40c57e0 msgr2=0x7fd2c41f0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2c40c57e0 0x7fd2c41f0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.211+0000 7fd2ca772640 1 -- 172.21.15.107:0/183156898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4009400 con 0x7fd2c415b440 2023-11-24T08:17:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2ca772640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00cb40 tx=0x7fd2bc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc00aca0 con 0x7fd2c415b440 2023-11-24T08:17:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2bc00ae00 con 0x7fd2c415b440 2023-11-24T08:17:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc012620 con 0x7fd2c415b440 2023-11-24T08:17:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c4156e60 con 0x7fd2c415b440 2023-11-24T08:17:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.212+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c40c6170 con 0x7fd2c415b440 2023-11-24T08:17:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.213+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c404e8b0 con 0x7fd2c415b440 2023-11-24T08:17:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.214+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fd2bc007070 con 0x7fd2c415b440 2023-11-24T08:17:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.216+0000 7fd2ba7fc640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 0x7fd2a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.216+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7fd2bc088460 con 0x7fd2c415b440 2023-11-24T08:17:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.216+0000 7fd2c9f71640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 0x7fd2a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.216+0000 7fd2c9f71640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 0x7fd2a806eaa0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006fd0 tx=0x7fd2c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.221+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2bc054f70 con 0x7fd2c415b440 2023-11-24T08:17:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.413+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2c406dc80 con 0x7fd2c415b440 2023-11-24T08:17:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.414+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/183156898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2bc017070 con 0x7fd2c415b440 2023-11-24T08:17:20.416 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 msgr2=0x7fd2a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 0x7fd2a806eaa0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006fd0 tx=0x7fd2c0008040 comp rx=0 tx=0).stop 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 msgr2=0x7fd2c40af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00cb40 tx=0x7fd2bc00a060 comp rx=0 tx=0).stop 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 shutdown_connections 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2a806c5e0 0x7fd2a806eaa0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c415b440 0x7fd2c40af9c0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2c40cbf20 0x7fd2c4155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 --2- 172.21.15.107:0/183156898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2c40c57e0 0x7fd2c41f0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 >> 172.21.15.107:0/183156898 conn(0x7fd2c40b0c50 msgr2=0x7fd2c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 shutdown_connections 2023-11-24T08:17:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.417+0000 7fd2cc1d5640 1 -- 172.21.15.107:0/183156898 wait complete. 2023-11-24T08:17:20.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:17:20.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.644+0000 7f17e1da9640 1 Processor -- start 2023-11-24T08:17:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.645+0000 7f17e1da9640 1 -- start start 2023-11-24T08:17:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17e1da9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17dc15b440 0x7f17dc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17e1da9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f17dc0cbf20 0x7f17dc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17e1da9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc15eea0 con 0x7f17dc0c57e0 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17e1da9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc0cc3d0 con 0x7f17dc15b440 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17e1da9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc15b820 con 0x7f17dc0cbf20 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17db7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17dc15b440 0x7f17dc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17db7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17dc15b440 0x7f17dc0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40514/0 (socket says 172.21.15.107:40514) 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.646+0000 7f17db7fe640 1 -- 172.21.15.107:0/1114516164 learned_addr learned my addr 172.21.15.107:0/1114516164 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.647+0000 7f17db7fe640 1 -- 172.21.15.107:0/1114516164 >> v2:172.21.15.120:3300/0 conn(0x7f17dc0cbf20 msgr2=0x7f17dc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.647+0000 7f17db7fe640 1 --2- 172.21.15.107:0/1114516164 >> v2:172.21.15.120:3300/0 conn(0x7f17dc0cbf20 0x7f17dc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.647+0000 7f17db7fe640 1 -- 172.21.15.107:0/1114516164 >> v1:172.21.15.120:6789/0 conn(0x7f17dc0c57e0 legacy=0x7f17dc0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:20.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.647+0000 7f17db7fe640 1 -- 172.21.15.107:0/1114516164 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc06d600 con 0x7f17dc15b440 2023-11-24T08:17:20.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.648+0000 7f17db7fe640 1 --2- 172.21.15.107:0/1114516164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0cb9e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f17dc06e890 tx=0x7f17cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8c142b0cd0db8f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.648+0000 7f17daffd640 1 -- 172.21.15.107:0/1114516164 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0317e0 con 0x7f17dc15b440 2023-11-24T08:17:20.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.648+0000 7f17daffd640 1 -- 172.21.15.107:0/1114516164 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc031940 con 0x7f17dc15b440 2023-11-24T08:17:20.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.648+0000 7f17daffd640 1 -- 172.21.15.107:0/1114516164 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc031cd0 con 0x7f17dc15b440 2023-11-24T08:17:20.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 -- 172.21.15.107:0/1114516164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 msgr2=0x7f17dc0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 --2- 172.21.15.107:0/1114516164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0cb9e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f17dc06e890 tx=0x7f17cc02b880 comp rx=0 tx=0).stop 2023-11-24T08:17:20.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 -- 172.21.15.107:0/1114516164 shutdown_connections 2023-11-24T08:17:20.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 --2- 172.21.15.107:0/1114516164 >> v2:172.21.15.120:3300/0 conn(0x7f17dc0cbf20 0x7f17dc0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 --2- 172.21.15.107:0/1114516164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0cb9e0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 -- 172.21.15.107:0/1114516164 >> 172.21.15.107:0/1114516164 conn(0x7f17dc0b0c50 msgr2=0x7f17dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 -- 172.21.15.107:0/1114516164 shutdown_connections 2023-11-24T08:17:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.649+0000 7f17e1da9640 1 -- 172.21.15.107:0/1114516164 wait complete. 2023-11-24T08:17:20.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.651+0000 7f17e1da9640 1 Processor -- start 2023-11-24T08:17:20.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.651+0000 7f17e1da9640 1 -- start start 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc0c57e0 0x7f17dc146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17dc0cbf20 0x7f17dc0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc09aaa0 con 0x7f17dc15b440 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc146960 con 0x7f17dc0c57e0 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17e1da9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc099a70 con 0x7f17dc0cbf20 2023-11-24T08:17:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40526/0 (socket says 172.21.15.107:40526) 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.652+0000 7f17dbfff640 1 -- 172.21.15.107:0/4188845085 learned_addr learned my addr 172.21.15.107:0/4188845085 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17db7fe640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc0c57e0 0x7f17dc146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 -- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17dc0cbf20 msgr2=0x7f17dc0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17dc0cbf20 0x7f17dc0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 -- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc0c57e0 msgr2=0x7f17dc146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc0c57e0 0x7f17dc146420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 -- 172.21.15.107:0/4188845085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0094d0 con 0x7f17dc15b440 2023-11-24T08:17:20.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17dbfff640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f17d400c7b0 tx=0x7f17d400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.653+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4003d80 con 0x7f17dc15b440 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.654+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d4005070 con 0x7f17dc15b440 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.654+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc099ce0 con 0x7f17dc15b440 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.654+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4005420 con 0x7f17dc15b440 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.654+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc09a2a0 con 0x7f17dc15b440 2023-11-24T08:17:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.655+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc04e8b0 con 0x7f17dc15b440 2023-11-24T08:17:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.656+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f17d4005580 con 0x7f17dc15b440 2023-11-24T08:17:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.657+0000 7f17d8ff9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 0x7f17bc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:20.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.657+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f17d4087bb0 con 0x7f17dc15b440 2023-11-24T08:17:20.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.657+0000 7f17db7fe640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 0x7f17bc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:20.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.658+0000 7f17db7fe640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 0x7f17bc06eaa0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f17cc036000 tx=0x7f17cc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.661+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17d4054760 con 0x7f17dc15b440 2023-11-24T08:17:20.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.849+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f17dc0b4f70 con 0x7f17dc15b440 2023-11-24T08:17:20.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.849+0000 7f17d8ff9640 1 -- 172.21.15.107:0/4188845085 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+11 (secure 0 0 0) 0x7f17d4017070 con 0x7f17dc15b440 2023-11-24T08:17:20.851 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:17:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 msgr2=0x7f17bc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 0x7f17bc06eaa0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f17cc036000 tx=0x7f17cc03d040 comp rx=0 tx=0).stop 2023-11-24T08:17:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 msgr2=0x7f17dc0b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f17d400c7b0 tx=0x7f17d400cc70 comp rx=0 tx=0).stop 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 shutdown_connections 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f17bc06c5e0 0x7f17bc06eaa0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17dc15b440 0x7f17dc0b3760 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17dc0cbf20 0x7f17dc0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 --2- 172.21.15.107:0/4188845085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc0c57e0 0x7f17dc146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.852+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 >> 172.21.15.107:0/4188845085 conn(0x7f17dc0b0c50 msgr2=0x7f17dc0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.853+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 shutdown_connections 2023-11-24T08:17:20.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:20.853+0000 7f17e1da9640 1 -- 172.21.15.107:0/4188845085 wait complete. 2023-11-24T08:17:20.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:17:21.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.092+0000 7fe92b91d640 1 Processor -- start 2023-11-24T08:17:21.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.092+0000 7fe92b91d640 1 -- start start 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe92b91d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe924146eb0 0x7fe924161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe92b91d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe9240cd2e0 0x7fe924070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe92b91d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe924161a10 con 0x7fe92406e160 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe92b91d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe924161cf0 con 0x7fe924146eb0 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe92b91d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe924161fd0 con 0x7fe9240cd2e0 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe9296b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe924146eb0 0x7fe924161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe928eb8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe92406e160 0x7fe9240ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39162/0 (socket says 172.21.15.107:39162) 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe9296b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe924146eb0 0x7fe924161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40528/0 (socket says 172.21.15.107:40528) 2023-11-24T08:17:21.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.094+0000 7fe928eb8640 1 -- 172.21.15.107:0/2548330172 learned_addr learned my addr 172.21.15.107:0/2548330172 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.095+0000 7fe929eba640 1 --2- 172.21.15.107:0/2548330172 >> v2:172.21.15.120:3300/0 conn(0x7fe9240cd2e0 0x7fe924070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.095+0000 7fe9296b9640 1 -- 172.21.15.107:0/2548330172 >> v2:172.21.15.120:3300/0 conn(0x7fe9240cd2e0 msgr2=0x7fe924070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.095+0000 7fe9296b9640 1 --2- 172.21.15.107:0/2548330172 >> v2:172.21.15.120:3300/0 conn(0x7fe9240cd2e0 0x7fe924070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.095+0000 7fe9296b9640 1 -- 172.21.15.107:0/2548330172 >> v1:172.21.15.107:6789/0 conn(0x7fe92406e160 legacy=0x7fe9240ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.095+0000 7fe9296b9640 1 -- 172.21.15.107:0/2548330172 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe924070660 con 0x7fe924146eb0 2023-11-24T08:17:21.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.096+0000 7fe9296b9640 1 --2- 172.21.15.107:0/2548330172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe924146eb0 0x7fe924161150 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fe9240ae150 tx=0x7fe92002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a089f68b0118e2e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.096+0000 7fe91bfff640 1 -- 172.21.15.107:0/2548330172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031830 con 0x7fe924146eb0 2023-11-24T08:17:21.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.096+0000 7fe91bfff640 1 -- 172.21.15.107:0/2548330172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920031990 con 0x7fe924146eb0 2023-11-24T08:17:21.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.096+0000 7fe91bfff640 1 -- 172.21.15.107:0/2548330172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031d20 con 0x7fe924146eb0 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 -- 172.21.15.107:0/2548330172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe924146eb0 msgr2=0x7fe924161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2548330172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe924146eb0 0x7fe924161150 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fe9240ae150 tx=0x7fe92002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 -- 172.21.15.107:0/2548330172 shutdown_connections 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2548330172 >> v2:172.21.15.120:3300/0 conn(0x7fe9240cd2e0 0x7fe924070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2548330172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe924146eb0 0x7fe924161150 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 -- 172.21.15.107:0/2548330172 >> 172.21.15.107:0/2548330172 conn(0x7fe9240b0c50 msgr2=0x7fe9240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 -- 172.21.15.107:0/2548330172 shutdown_connections 2023-11-24T08:17:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.097+0000 7fe92b91d640 1 -- 172.21.15.107:0/2548330172 wait complete. 2023-11-24T08:17:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.099+0000 7fe92b91d640 1 Processor -- start 2023-11-24T08:17:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.099+0000 7fe92b91d640 1 -- start start 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe92406e160 0x7fe92415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe924146eb0 0x7fe9240b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240b26e0 con 0x7fe9240cd2e0 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924160770 con 0x7fe92406e160 2023-11-24T08:17:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe92b91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415cf40 con 0x7fe924146eb0 2023-11-24T08:17:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe928eb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe928eb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40540/0 (socket says 172.21.15.107:40540) 2023-11-24T08:17:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe928eb8640 1 -- 172.21.15.107:0/2447599879 learned_addr learned my addr 172.21.15.107:0/2447599879 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.100+0000 7fe9296b9640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe92406e160 0x7fe92415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe929eba640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe924146eb0 0x7fe9240b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 -- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe924146eb0 msgr2=0x7fe9240b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe924146eb0 0x7fe9240b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 -- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe92406e160 msgr2=0x7fe92415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe92406e160 0x7fe92415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 -- 172.21.15.107:0/2447599879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920009400 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe928eb8640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c4a0 tx=0x7fe91400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914005ce0 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914005e40 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914007270 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.101+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240b1db0 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.102+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9240b4a90 con 0x7fe9240cd2e0 2023-11-24T08:17:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.103+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240b1f40 con 0x7fe9240cd2e0 2023-11-24T08:17:21.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.104+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fe914007490 con 0x7fe9240cd2e0 2023-11-24T08:17:21.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.105+0000 7fe919ffb640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 0x7fe90806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.105+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7fe914087820 con 0x7fe9240cd2e0 2023-11-24T08:17:21.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.105+0000 7fe9296b9640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 0x7fe90806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.106+0000 7fe9296b9640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 0x7fe90806eaa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.110+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9240b1f40 con 0x7fe9240cd2e0 2023-11-24T08:17:21.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.299+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9240b2820 con 0x7fe9240cd2e0 2023-11-24T08:17:21.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.300+0000 7fe919ffb640 1 -- 172.21.15.107:0/2447599879 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe9240b2820 con 0x7fe9240cd2e0 2023-11-24T08:17:21.302 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:17:21.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 msgr2=0x7fe90806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 0x7fe90806eaa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003c040 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 msgr2=0x7fe9240b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c4a0 tx=0x7fe91400c960 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 shutdown_connections 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe90806c5e0 0x7fe90806eaa0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe924146eb0 0x7fe9240b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9240cd2e0 0x7fe9240b2b00 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 --2- 172.21.15.107:0/2447599879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe92406e160 0x7fe92415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.302+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 >> 172.21.15.107:0/2447599879 conn(0x7fe9240b0c50 msgr2=0x7fe9240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.303+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 shutdown_connections 2023-11-24T08:17:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.303+0000 7fe92b91d640 1 -- 172.21.15.107:0/2447599879 wait complete. 2023-11-24T08:17:21.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:17:21.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.532+0000 7f763caa2640 1 Processor -- start 2023-11-24T08:17:21.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.533+0000 7f763caa2640 1 -- start start 2023-11-24T08:17:21.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.534+0000 7f763caa2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76380ccbd0 0x7f76381660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.534+0000 7f763caa2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f76380ccfb0 0x7f7638162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.534+0000 7f763caa2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76381655f0 con 0x7f7638162540 2023-11-24T08:17:21.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.534+0000 7f763caa2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76380cc380 con 0x7f76380ccbd0 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.534+0000 7f763caa2640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7638162b30 con 0x7f76380ccfb0 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f763659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76380ccbd0 0x7f76381660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f7636d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7638162540 0x7f76380c9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39164/0 (socket says 172.21.15.107:39164) 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f7636d9d640 1 -- 172.21.15.107:0/2725717398 learned_addr learned my addr 172.21.15.107:0/2725717398 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f763659c640 1 -- 172.21.15.107:0/2725717398 >> v2:172.21.15.120:3300/0 conn(0x7f76380ccfb0 msgr2=0x7f7638162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f763659c640 1 --2- 172.21.15.107:0/2725717398 >> v2:172.21.15.120:3300/0 conn(0x7f76380ccfb0 0x7f7638162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f763659c640 1 -- 172.21.15.107:0/2725717398 >> v1:172.21.15.107:6789/0 conn(0x7f7638162540 legacy=0x7f76380c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f763659c640 1 -- 172.21.15.107:0/2725717398 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638162e10 con 0x7f76380ccbd0 2023-11-24T08:17:21.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.535+0000 7f7635d9b640 1 -- 172.21.15.107:0/2725717398 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633091425 0 0) 0x7f76381655f0 con 0x7f7638162540 2023-11-24T08:17:21.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.536+0000 7f763659c640 1 --2- 172.21.15.107:0/2725717398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76380ccbd0 0x7f76381660b0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f763806e700 tx=0x7f762802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0c59818d62e84 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f7635d9b640 1 -- 172.21.15.107:0/2725717398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628031830 con 0x7f76380ccbd0 2023-11-24T08:17:21.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f7635d9b640 1 -- 172.21.15.107:0/2725717398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7628031990 con 0x7f76380ccbd0 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f7635d9b640 1 -- 172.21.15.107:0/2725717398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628031d20 con 0x7f76380ccbd0 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 -- 172.21.15.107:0/2725717398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76380ccbd0 msgr2=0x7f76381660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 --2- 172.21.15.107:0/2725717398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76380ccbd0 0x7f76381660b0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f763806e700 tx=0x7f762802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 -- 172.21.15.107:0/2725717398 shutdown_connections 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 --2- 172.21.15.107:0/2725717398 >> v2:172.21.15.120:3300/0 conn(0x7f76380ccfb0 0x7f7638162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 --2- 172.21.15.107:0/2725717398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76380ccbd0 0x7f76381660b0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 -- 172.21.15.107:0/2725717398 >> 172.21.15.107:0/2725717398 conn(0x7f76380b0c50 msgr2=0x7f76380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 -- 172.21.15.107:0/2725717398 shutdown_connections 2023-11-24T08:17:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.537+0000 7f763caa2640 1 -- 172.21.15.107:0/2725717398 wait complete. 2023-11-24T08:17:21.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.539+0000 7f763caa2640 1 Processor -- start 2023-11-24T08:17:21.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.539+0000 7f763caa2640 1 -- start start 2023-11-24T08:17:21.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 0x7f7638155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 0x7f7638158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76381fba60 con 0x7f7638162540 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638070680 con 0x7f76380ccbd0 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f763caa2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76380728a0 con 0x7f76380ccfb0 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f7636d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f7636d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40550/0 (socket says 172.21.15.107:40550) 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.540+0000 7f7636d9d640 1 -- 172.21.15.107:0/4225554599 learned_addr learned my addr 172.21.15.107:0/4225554599 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f763659c640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 0x7f7638155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 -- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 msgr2=0x7f7638158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f762ffff640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 0x7f7638158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 0x7f7638158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 -- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 msgr2=0x7f7638155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 0x7f7638155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 -- 172.21.15.107:0/4225554599 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628009400 con 0x7f7638162540 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f762ffff640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 0x7f7638158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f763659c640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 0x7f7638155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f7636d9d640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f763000c7a0 tx=0x7f763000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763000a640 con 0x7f7638162540 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.541+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763000a7a0 con 0x7f7638162540 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.542+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763000abd0 con 0x7f7638162540 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.542+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76380722e0 con 0x7f7638162540 2023-11-24T08:17:21.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.542+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763815aaa0 con 0x7f7638162540 2023-11-24T08:17:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.543+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76380cd410 con 0x7f7638162540 2023-11-24T08:17:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.544+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f763000ad30 con 0x7f7638162540 2023-11-24T08:17:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.545+0000 7f762effd640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 0x7f761806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.545+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f76300879a0 con 0x7f7638162540 2023-11-24T08:17:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.545+0000 7f763659c640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 0x7f761806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.546+0000 7f763659c640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 0x7f761806eaa0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7628036000 tx=0x7f7628008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:21.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.550+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76300544b0 con 0x7f7638162540 2023-11-24T08:17:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.753+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f76380707f0 con 0x7f7638162540 2023-11-24T08:17:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.754+0000 7f762effd640 1 -- 172.21.15.107:0/4225554599 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v150) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7630018070 con 0x7f7638162540 2023-11-24T08:17:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.756+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 msgr2=0x7f761806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.756+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 0x7f761806eaa0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7628036000 tx=0x7f7628008030 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.756+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 msgr2=0x7f76381fb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.756+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f763000c7a0 tx=0x7f763000cc60 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 shutdown_connections 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f761806c5e0 0x7f761806eaa0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638162540 0x7f76381fb520 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76380ccfb0 0x7f7638158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 --2- 172.21.15.107:0/4225554599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76380ccbd0 0x7f7638155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 >> 172.21.15.107:0/4225554599 conn(0x7f76380b0c50 msgr2=0x7f76380afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 shutdown_connections 2023-11-24T08:17:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:21.757+0000 7f763caa2640 1 -- 172.21.15.107:0/4225554599 wait complete. 2023-11-24T08:17:21.779 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:17:21.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:21.779 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:17:21.891 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:17:21.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:21.891 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:17:21.946 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:21.946 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:22.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:17:22.006 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:22.006 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:22.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:17:22.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:22.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.284+0000 7f236f58f640 1 Processor -- start 2023-11-24T08:17:22.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.284+0000 7f236f58f640 1 -- start start 2023-11-24T08:17:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236f58f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f236815af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236f58f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f236806d6c0 0x7f236806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236f58f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236806e060 con 0x7f23680b0a70 2023-11-24T08:17:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236f58f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236813f9e0 con 0x7f236806d6c0 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236f58f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236813fcc0 con 0x7f236813d120 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236ed8e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f23680b0a70 0x7f23680b0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39168/0 (socket says 172.21.15.107:39168) 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.286+0000 7f236ed8e640 1 -- 172.21.15.107:0/3536474761 learned_addr learned my addr 172.21.15.107:0/3536474761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236e58d640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f236815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236dd8c640 1 --2- 172.21.15.107:0/3536474761 >> v2:172.21.15.120:3300/0 conn(0x7f236806d6c0 0x7f236806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895986452 0 0) 0x7f236806e060 con 0x7f23680b0a70 2023-11-24T08:17:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2354003520 con 0x7f23680b0a70 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910096384 0 0) 0x7f2354003520 con 0x7f23680b0a70 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 msgr2=0x7f236815af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f236815af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 >> v2:172.21.15.120:3300/0 conn(0x7f236806d6c0 msgr2=0x7f236806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 --2- 172.21.15.107:0/3536474761 >> v2:172.21.15.120:3300/0 conn(0x7f236806d6c0 0x7f236806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680cec00 con 0x7f23680b0a70 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f23600019f0 con 0x7f23680b0a70 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 >> v1:172.21.15.107:6789/0 conn(0x7f23680b0a70 legacy=0x7f23680b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f2354003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f23540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2354004d00 0x7f2354005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236813f9e0 con 0x7f2354003970 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236813fcc0 con 0x7f2354004d00 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.287+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235400b980 con 0x7f236813d120 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f2354003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236dd8c640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f23540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236ed8e640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2354004d00 0x7f2354005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 -- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 msgr2=0x7f23540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f236813d120 0x7f23540047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 -- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2354004d00 msgr2=0x7f2354005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2354004d00 0x7f2354005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.288+0000 7f236e58d640 1 -- 172.21.15.107:0/3536474761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354003520 con 0x7f2354003970 2023-11-24T08:17:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.289+0000 7f236e58d640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f2354003d90 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f2364009ac0 tx=0x7f23540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39c72a9b9e98cb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.289+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402c6d0 con 0x7f2354003970 2023-11-24T08:17:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236402c830 con 0x7f2354003970 2023-11-24T08:17:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236d58b640 1 -- 172.21.15.107:0/3536474761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402cbc0 con 0x7f2354003970 2023-11-24T08:17:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 -- 172.21.15.107:0/3536474761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 msgr2=0x7f2354003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 --2- 172.21.15.107:0/3536474761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f2354003d90 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f2364009ac0 tx=0x7f23540057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236ed8e640 1 -- 172.21.15.107:0/3536474761 reap_dead start 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 -- 172.21.15.107:0/3536474761 shutdown_connections 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 -- 172.21.15.107:0/3536474761 >> 172.21.15.107:0/3536474761 conn(0x7f23680cb830 msgr2=0x7f23680cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 -- 172.21.15.107:0/3536474761 shutdown_connections 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.290+0000 7f236f58f640 1 -- 172.21.15.107:0/3536474761 wait complete. 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.292+0000 7f236f58f640 1 Processor -- start 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.292+0000 7f236f58f640 1 -- start start 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23680b34d0 0x7f23680b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2368159210 0x7f2368159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2368142010 con 0x7f2354003970 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23680b4e10 con 0x7f2368159210 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236f58f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23680b4200 con 0x7f23680b34d0 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236e58d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236e58d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40572/0 (socket says 172.21.15.107:40572) 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.293+0000 7f236e58d640 1 -- 172.21.15.107:0/1425832305 learned_addr learned my addr 172.21.15.107:0/1425832305 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 -- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23680b34d0 msgr2=0x7f23680b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23680b34d0 0x7f23680b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 -- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2368159210 msgr2=0x7f2368159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2368159210 0x7f2368159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 -- 172.21.15.107:0/1425832305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680b44e0 con 0x7f2354003970 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236e58d640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f236402c270 tx=0x7f2364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364003810 con 0x7f2354003970 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364003970 con 0x7f2354003970 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23680b47c0 con 0x7f2354003970 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.294+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364003d00 con 0x7f2354003970 2023-11-24T08:17:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.295+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23680b3e70 con 0x7f2354003970 2023-11-24T08:17:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.296+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236804e840 con 0x7f2354003970 2023-11-24T08:17:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.296+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f2364034440 con 0x7f2354003970 2023-11-24T08:17:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.297+0000 7f235effd640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 0x7f234c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.297+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f23640aa550 con 0x7f2354003970 2023-11-24T08:17:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.298+0000 7f236dd8c640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 0x7f234c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.298+0000 7f236dd8c640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 0x7f234c06eaa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f236813f200 tx=0x7f2358008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.302+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2364077230 con 0x7f2354003970 2023-11-24T08:17:22.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.524+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23680cd7d0 con 0x7f2354003970 2023-11-24T08:17:22.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.525+0000 7f235effd640 1 -- 172.21.15.107:0/1425832305 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f236404c090 con 0x7f2354003970 2023-11-24T08:17:22.527 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:22.528 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:17:22.528 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 88 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.527+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 msgr2=0x7f234c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.527+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 0x7f234c06eaa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f236813f200 tx=0x7f2358008040 comp rx=0 tx=0).stop 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 msgr2=0x7f23680af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f236402c270 tx=0x7f2364004730 comp rx=0 tx=0).stop 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 shutdown_connections 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f234c06c5e0 0x7f234c06eaa0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2368159210 0x7f2368159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23680b34d0 0x7f23680b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 --2- 172.21.15.107:0/1425832305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2354003970 0x7f23680af260 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 >> 172.21.15.107:0/1425832305 conn(0x7f23680cb830 msgr2=0x7f236815e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 shutdown_connections 2023-11-24T08:17:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.528+0000 7f236f58f640 1 -- 172.21.15.107:0/1425832305 wait complete. 2023-11-24T08:17:22.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:22.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.750+0000 7f5adc002640 1 Processor -- start 2023-11-24T08:17:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.751+0000 7f5adc002640 1 -- start start 2023-11-24T08:17:22.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.752+0000 7f5adc002640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ad40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.752+0000 7f5adc002640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ad40cdfb0 0x7f5ad40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.752+0000 7f5adc002640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40cece0 con 0x7f5ad415c7f0 2023-11-24T08:17:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.752+0000 7f5adc002640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad4156010 con 0x7f5ad40cdfb0 2023-11-24T08:17:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.752+0000 7f5adc002640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad406ce30 con 0x7f5ad40cf450 2023-11-24T08:17:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad9d9e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5ad415c7f0 0x7f5ad406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39172/0 (socket says 172.21.15.107:39172) 2023-11-24T08:17:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/4122958258 learned_addr learned my addr 172.21.15.107:0/4122958258 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703796214 0 0) 0x7f5ad40cece0 con 0x7f5ad415c7f0 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad959d640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ad40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac0003520 con 0x7f5ad415c7f0 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ada59f640 1 --2- 172.21.15.107:0/4122958258 >> v2:172.21.15.120:3300/0 conn(0x7f5ad40cdfb0 0x7f5ad40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426020189 0 0) 0x7f5ac0003520 con 0x7f5ad415c7f0 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 msgr2=0x7f5ad40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ad40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 >> v2:172.21.15.120:3300/0 conn(0x7f5ad40cdfb0 msgr2=0x7f5ad40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 --2- 172.21.15.107:0/4122958258 >> v2:172.21.15.120:3300/0 conn(0x7f5ad40cdfb0 0x7f5ad40ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad406d040 con 0x7f5ad415c7f0 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5ad0001670 con 0x7f5ad415c7f0 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.753+0000 7f5ad959d640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ad40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 >> v1:172.21.15.107:6789/0 conn(0x7f5ad415c7f0 legacy=0x7f5ad406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ac0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ac00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ac0004d00 0x7f5ac0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4156010 con 0x7f5ac0003970 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad406ce30 con 0x7f5ac0004d00 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac000b980 con 0x7f5ad40cf450 2023-11-24T08:17:22.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ac0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad959d640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ac00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 msgr2=0x7f5ac00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad40cf450 0x7f5ac00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ac0004d00 msgr2=0x7f5ac0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ac0004d00 0x7f5ac0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.754+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/4122958258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac0003520 con 0x7f5ac0003970 2023-11-24T08:17:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.755+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ac0003d90 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c240 tx=0x7f5ac00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622d820e2ab66eff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad002cb00 con 0x7f5ac0003970 2023-11-24T08:17:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad002cc60 con 0x7f5ac0003970 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5ad8d9c640 1 -- 172.21.15.107:0/4122958258 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0005af0 con 0x7f5ac0003970 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 -- 172.21.15.107:0/4122958258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 msgr2=0x7f5ac0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 --2- 172.21.15.107:0/4122958258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ac0003d90 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c240 tx=0x7f5ac00057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5ada59f640 1 -- 172.21.15.107:0/4122958258 reap_dead start 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 -- 172.21.15.107:0/4122958258 shutdown_connections 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 -- 172.21.15.107:0/4122958258 >> 172.21.15.107:0/4122958258 conn(0x7f5ad406a920 msgr2=0x7f5ad406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 -- 172.21.15.107:0/4122958258 shutdown_connections 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.756+0000 7f5adc002640 1 -- 172.21.15.107:0/4122958258 wait complete. 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.758+0000 7f5adc002640 1 Processor -- start 2023-11-24T08:17:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.758+0000 7f5adc002640 1 -- start start 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad41f0aa0 0x7f5ad415db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad415e0b0 0x7f5ad406d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4156e70 con 0x7f5ac0003970 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4156260 con 0x7f5ad41f0aa0 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5adc002640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4156540 con 0x7f5ad415e0b0 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.759+0000 7f5ad9d9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40584/0 (socket says 172.21.15.107:40584) 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/1333811520 learned_addr learned my addr 172.21.15.107:0/1333811520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad415e0b0 msgr2=0x7f5ad406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad415e0b0 0x7f5ad406d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad41f0aa0 msgr2=0x7f5ad415db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad41f0aa0 0x7f5ad415db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 -- 172.21.15.107:0/1333811520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4156820 con 0x7f5ac0003970 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5ad9d9e640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c240 tx=0x7f5ad0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.760+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0039030 con 0x7f5ac0003970 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.761+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0004bf0 con 0x7f5ac0003970 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.761+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad003c040 con 0x7f5ac0003970 2023-11-24T08:17:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.761+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad4156b00 con 0x7f5ac0003970 2023-11-24T08:17:22.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.761+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad406deb0 con 0x7f5ac0003970 2023-11-24T08:17:22.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.762+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad4067470 con 0x7f5ac0003970 2023-11-24T08:17:22.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.763+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f5ad0032070 con 0x7f5ac0003970 2023-11-24T08:17:22.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.764+0000 7f5aca7fc640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 0x7f5ab806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:22.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.764+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f5ad00a98d0 con 0x7f5ac0003970 2023-11-24T08:17:22.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.764+0000 7f5ad959d640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 0x7f5ab806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:22.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.765+0000 7f5ad959d640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 0x7f5ab806ead0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5ad415c050 tx=0x7f5ac4006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:22.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.769+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ad00763e0 con 0x7f5ac0003970 2023-11-24T08:17:22.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.964+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ad40cd730 con 0x7f5ac0003970 2023-11-24T08:17:22.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.967+0000 7f5aca7fc640 1 -- 172.21.15.107:0/1333811520 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16740 (secure 0 0 0) 0x7f5ad0079a00 con 0x7f5ac0003970 2023-11-24T08:17:22.969 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:22.969 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:16.002760+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:22.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 msgr2=0x7f5ab806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 0x7f5ab806ead0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5ad415c050 tx=0x7f5ac4006df0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 msgr2=0x7f5ad41484d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c240 tx=0x7f5ad0001a00 comp rx=0 tx=0).stop 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 shutdown_connections 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ab806c610 0x7f5ab806ead0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ad415e0b0 0x7f5ad406d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad41f0aa0 0x7f5ad415db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 --2- 172.21.15.107:0/1333811520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac0003970 0x7f5ad41484d0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.969+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 >> 172.21.15.107:0/1333811520 conn(0x7f5ad406a920 msgr2=0x7f5ad40af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.970+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 shutdown_connections 2023-11-24T08:17:22.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:22.970+0000 7f5adc002640 1 -- 172.21.15.107:0/1333811520 wait complete. 2023-11-24T08:17:22.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:16:03.725981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:16:04.911827+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 150, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:17:22.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:17:23.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.201+0000 7f3d0161a640 1 Processor -- start 2023-11-24T08:17:23.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.201+0000 7f3d0161a640 1 -- start start 2023-11-24T08:17:23.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.202+0000 7f3d0161a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3cfc15b260 0x7f3cfc06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.202+0000 7f3d0161a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc15ed20 con 0x7f3cfc06e580 2023-11-24T08:17:23.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.202+0000 7f3d0161a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc15f8a0 con 0x7f3cfc0b1250 2023-11-24T08:17:23.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.202+0000 7f3d0161a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc15b640 con 0x7f3cfc15b260 2023-11-24T08:17:23.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3cfc06e580 0x7f3cfc0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39174/0 (socket says 172.21.15.107:39174) 2023-11-24T08:17:23.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfb7fe640 1 -- 172.21.15.107:0/3639932145 learned_addr learned my addr 172.21.15.107:0/3639932145 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:23.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/3639932145 >> v2:172.21.15.120:3300/0 conn(0x7f3cfc15b260 0x7f3cfc06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078736842 0 0) 0x7f3cfc15ed20 con 0x7f3cfc06e580 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce8003520 con 0x7f3cfc06e580 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928988105 0 0) 0x7f3ce8003520 con 0x7f3cfc06e580 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 >> v2:172.21.15.120:3300/0 conn(0x7f3cfc15b260 msgr2=0x7f3cfc06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 --2- 172.21.15.107:0/3639932145 >> v2:172.21.15.120:3300/0 conn(0x7f3cfc15b260 0x7f3cfc06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 >> v1:172.21.15.120:6789/0 conn(0x7f3cfc0b1250 legacy=0x7f3cfc06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0abe50 con 0x7f3cfc06e580 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3ce40019f0 con 0x7f3cfc06e580 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.203+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 >> v1:172.21.15.107:6789/0 conn(0x7f3cfc06e580 legacy=0x7f3cfc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3ce8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ce80047c0 0x7f3ce8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ce80052a0 0x7f3ce800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc15b640 con 0x7f3ce80039c0 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce800c1a0 con 0x7f3ce80052a0 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce800c360 con 0x7f3ce80047c0 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3ce8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 -- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ce80047c0 msgr2=0x7f3ce8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ce80047c0 0x7f3ce8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 -- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ce80052a0 msgr2=0x7f3ce800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ce80052a0 0x7f3ce800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3cfbfff640 1 -- 172.21.15.107:0/3639932145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8003520 con 0x7f3ce80039c0 2023-11-24T08:17:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.204+0000 7f3d00e19640 1 -- 172.21.15.107:0/3639932145 reap_dead start 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.205+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3ce8003dc0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009a90 tx=0x7f3ce80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3501ef71d01746a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.205+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec02c6b0 con 0x7f3ce80039c0 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.205+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec02c810 con 0x7f3ce80039c0 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.205+0000 7f3cfaffd640 1 -- 172.21.15.107:0/3639932145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec02cba0 con 0x7f3ce80039c0 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 -- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 msgr2=0x7f3ce8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3ce8003dc0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009a90 tx=0x7f3ce80057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 -- 172.21.15.107:0/3639932145 shutdown_connections 2023-11-24T08:17:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 --2- 172.21.15.107:0/3639932145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3ce8003dc0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 -- 172.21.15.107:0/3639932145 >> 172.21.15.107:0/3639932145 conn(0x7f3cfc0cbb40 msgr2=0x7f3cfc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 -- 172.21.15.107:0/3639932145 shutdown_connections 2023-11-24T08:17:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.206+0000 7f3d0161a640 1 -- 172.21.15.107:0/3639932145 wait complete. 2023-11-24T08:17:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.208+0000 7f3d0161a640 1 Processor -- start 2023-11-24T08:17:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.208+0000 7f3d0161a640 1 -- start start 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cfc0ccc50 0x7f3cfc0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cfc1570d0 0x7f3cfc157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc158930 con 0x7f3ce80039c0 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0cd610 con 0x7f3cfc1570d0 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3d0161a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc157a90 con 0x7f3cfc0ccc50 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3cfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3cfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40604/0 (socket says 172.21.15.107:40604) 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.209+0000 7f3cfbfff640 1 -- 172.21.15.107:0/2412739846 learned_addr learned my addr 172.21.15.107:0/2412739846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 -- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cfc0ccc50 msgr2=0x7f3cfc0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cfc0ccc50 0x7f3cfc0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 -- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cfc1570d0 msgr2=0x7f3cfc157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cfc1570d0 0x7f3cfc157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 -- 172.21.15.107:0/2412739846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc15e710 con 0x7f3ce80039c0 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cfbfff640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f3cec037040 tx=0x7f3cec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec003810 con 0x7f3ce80039c0 2023-11-24T08:17:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec003970 con 0x7f3ce80039c0 2023-11-24T08:17:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc158d50 con 0x7f3ce80039c0 2023-11-24T08:17:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.210+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec003d20 con 0x7f3ce80039c0 2023-11-24T08:17:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.211+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc157f90 con 0x7f3ce80039c0 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.212+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc158ba0 con 0x7f3ce80039c0 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.213+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f3cec034440 con 0x7f3ce80039c0 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.214+0000 7f3cf8ff9640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 0x7f3cdc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.214+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f3cec0aa400 con 0x7f3ce80039c0 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.214+0000 7f3cfb7fe640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 0x7f3cdc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.214+0000 7f3cfb7fe640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 0x7f3cdc06eaa0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc15e370 tx=0x7f3ce40018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.218+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cec077100 con 0x7f3ce80039c0 2023-11-24T08:17:23.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.409+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3cfc04e840 con 0x7f3ce80039c0 2023-11-24T08:17:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.410+0000 7f3cf8ff9640 1 -- 172.21.15.107:0/2412739846 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3cec07a720 con 0x7f3ce80039c0 2023-11-24T08:17:23.412 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:17:23.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 msgr2=0x7f3cdc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 0x7f3cdc06eaa0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc15e370 tx=0x7f3ce40018b0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 msgr2=0x7f3cfc14e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f3cec037040 tx=0x7f3cec004730 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 shutdown_connections 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cdc06c5e0 0x7f3cdc06eaa0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cfc1570d0 0x7f3cfc157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cfc0ccc50 0x7f3cfc0cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 --2- 172.21.15.107:0/2412739846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce80039c0 0x7f3cfc14e4c0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 >> 172.21.15.107:0/2412739846 conn(0x7f3cfc0cbb40 msgr2=0x7f3cfc0af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:23.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 shutdown_connections 2023-11-24T08:17:23.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.413+0000 7f3d0161a640 1 -- 172.21.15.107:0/2412739846 wait complete. 2023-11-24T08:17:23.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:17:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.638+0000 7f0164607640 1 Processor -- start 2023-11-24T08:17:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.639+0000 7f0164607640 1 -- start start 2023-11-24T08:17:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.640+0000 7f0164607640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 0x7f015c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.640+0000 7f0164607640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f015c0c59b0 0x7f015c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.640+0000 7f0164607640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0cc3a0 con 0x7f015c0cbf20 2023-11-24T08:17:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.640+0000 7f0164607640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c15b820 con 0x7f015c0c59b0 2023-11-24T08:17:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.640+0000 7f0164607640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c15eea0 con 0x7f015c15b440 2023-11-24T08:17:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f0161ba2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f015c0cbf20 0x7f015c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39184/0 (socket says 172.21.15.107:39184) 2023-11-24T08:17:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f0161ba2640 1 -- 172.21.15.107:0/3775236739 learned_addr learned my addr 172.21.15.107:0/3775236739 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01623a3640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 0x7f015c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900713589 0 0) 0x7f014c001110 con 0x7f015c0cbf20 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f0162ba4640 1 --2- 172.21.15.107:0/3775236739 >> v2:172.21.15.120:3300/0 conn(0x7f015c0c59b0 0x7f015c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0148003520 con 0x7f015c0cbf20 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683160770 0 0) 0x7f0148003520 con 0x7f015c0cbf20 2023-11-24T08:17:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 msgr2=0x7f015c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 0x7f015c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 >> v2:172.21.15.120:3300/0 conn(0x7f015c0c59b0 msgr2=0x7f015c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 --2- 172.21.15.107:0/3775236739 >> v2:172.21.15.120:3300/0 conn(0x7f015c0c59b0 0x7f015c06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0abf30 con 0x7f015c0cbf20 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f014c001800 con 0x7f015c0cbf20 2023-11-24T08:17:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.641+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 >> v1:172.21.15.107:6789/0 conn(0x7f015c0cbf20 legacy=0x7f015c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0148003970 0x7f0148003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01480047c0 0x7f0148004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 0x7f0148005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c15b820 con 0x7f01480047c0 2023-11-24T08:17:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c15eea0 con 0x7f0148003970 2023-11-24T08:17:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014800ba30 con 0x7f015c15b440 2023-11-24T08:17:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01480047c0 0x7f0148004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f01623a3640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0148003970 0x7f0148003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 -- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 msgr2=0x7f0148005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f015c15b440 0x7f0148005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 -- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0148003970 msgr2=0x7f0148003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0148003970 0x7f0148003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.642+0000 7f0161ba2640 1 -- 172.21.15.107:0/3775236739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148003520 con 0x7f01480047c0 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.643+0000 7f0161ba2640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01480047c0 0x7f0148004c40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f014c001eb0 tx=0x7f01480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81e3a36b089c2f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c005ad0 con 0x7f01480047c0 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c005c50 con 0x7f01480047c0 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f01613a1640 1 -- 172.21.15.107:0/3775236739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c02ca20 con 0x7f01480047c0 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 -- 172.21.15.107:0/3775236739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01480047c0 msgr2=0x7f0148004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 --2- 172.21.15.107:0/3775236739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01480047c0 0x7f0148004c40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f014c001eb0 tx=0x7f01480057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0162ba4640 1 -- 172.21.15.107:0/3775236739 reap_dead start 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 -- 172.21.15.107:0/3775236739 shutdown_connections 2023-11-24T08:17:23.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 -- 172.21.15.107:0/3775236739 >> 172.21.15.107:0/3775236739 conn(0x7f015c0b0c50 msgr2=0x7f015c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 -- 172.21.15.107:0/3775236739 shutdown_connections 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.644+0000 7f0164607640 1 -- 172.21.15.107:0/3775236739 wait complete. 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.646+0000 7f0164607640 1 Processor -- start 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.646+0000 7f0164607640 1 -- start start 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01480047c0 0x7f015c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f015c0c7210 0x7f015c0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1575b0 con 0x7f015c0b26c0 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0b3080 con 0x7f015c0c7210 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.647+0000 7f0164607640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0c6170 con 0x7f01480047c0 2023-11-24T08:17:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40634/0 (socket says 172.21.15.107:40634) 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 -- 172.21.15.107:0/2626293094 learned_addr learned my addr 172.21.15.107:0/2626293094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f01623a3640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01480047c0 0x7f015c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 -- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01480047c0 msgr2=0x7f015c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01480047c0 0x7f015c157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 -- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f015c0c7210 msgr2=0x7f015c0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0161ba2640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f015c0c7210 0x7f015c0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f015c0c7210 0x7f015c0c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 -- 172.21.15.107:0/2626293094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0c6450 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.648+0000 7f0162ba4640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f015c0ccb40 tx=0x7f015400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.649+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015400ed90 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.649+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154014420 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.649+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540147d0 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.649+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c0c6730 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.649+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c0c6cf0 con 0x7f015c0b26c0 2023-11-24T08:17:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.650+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c04e8b0 con 0x7f015c0b26c0 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.651+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f0154014930 con 0x7f015c0b26c0 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.652+0000 7f0152ffd640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 0x7f014006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.652+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f0154088900 con 0x7f015c0b26c0 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.652+0000 7f01623a3640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 0x7f014006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.653+0000 7f01623a3640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 0x7f014006eaa0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f015c15ae50 tx=0x7f0158008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.656+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0154055490 con 0x7f015c0b26c0 2023-11-24T08:17:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.846+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f015c06dc80 con 0x7f015c0b26c0 2023-11-24T08:17:23.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.847+0000 7f0152ffd640 1 -- 172.21.15.107:0/2626293094 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0154058ab0 con 0x7f015c0b26c0 2023-11-24T08:17:23.849 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 msgr2=0x7f014006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 0x7f014006eaa0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f015c15ae50 tx=0x7f0158008040 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 msgr2=0x7f015c0b2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f015c0ccb40 tx=0x7f015400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 shutdown_connections 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f014006c5e0 0x7f014006eaa0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f015c0b26c0 0x7f015c0b2b40 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f015c0c7210 0x7f015c0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 --2- 172.21.15.107:0/2626293094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01480047c0 0x7f015c157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 >> 172.21.15.107:0/2626293094 conn(0x7f015c0b0c50 msgr2=0x7f015c0cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 shutdown_connections 2023-11-24T08:17:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:23.850+0000 7f0164607640 1 -- 172.21.15.107:0/2626293094 wait complete. 2023-11-24T08:17:23.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:17:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.076+0000 7fd41101c640 1 Processor -- start 2023-11-24T08:17:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.076+0000 7fd41101c640 1 -- start start 2023-11-24T08:17:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd41101c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd40c15b6d0 0x7fd40c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd41101c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0b4eb0 con 0x7fd40c15b2f0 2023-11-24T08:17:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd41101c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0b5620 con 0x7fd40c06bc20 2023-11-24T08:17:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd41101c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c0cbf10 con 0x7fd40c15b6d0 2023-11-24T08:17:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd40ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd40c15b2f0 0x7fd40c06b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39198/0 (socket says 172.21.15.107:39198) 2023-11-24T08:17:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd40ad9d640 1 -- 172.21.15.107:0/1355406796 learned_addr learned my addr 172.21.15.107:0/1355406796 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd40b59e640 1 --2- 172.21.15.107:0/1355406796 >> v2:172.21.15.120:3300/0 conn(0x7fd40c15b6d0 0x7fd40c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191739633 0 0) 0x7fd40c0b4eb0 con 0x7fd40c15b2f0 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.078+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f4003520 con 0x7fd40c15b2f0 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2206978139 0 0) 0x7fd3f4003520 con 0x7fd40c15b2f0 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 >> v2:172.21.15.120:3300/0 conn(0x7fd40c15b6d0 msgr2=0x7fd40c0aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 --2- 172.21.15.107:0/1355406796 >> v2:172.21.15.120:3300/0 conn(0x7fd40c15b6d0 0x7fd40c0aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 >> v1:172.21.15.120:6789/0 conn(0x7fd40c06bc20 legacy=0x7fd40c06d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0cc1f0 con 0x7fd40c15b2f0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd3fc001670 con 0x7fd40c15b2f0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40b59e640 1 --2- 172.21.15.107:0/1355406796 >> v2:172.21.15.120:3300/0 conn(0x7fd40c15b6d0 0x7fd40c0aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 >> v1:172.21.15.107:6789/0 conn(0x7fd40c15b2f0 legacy=0x7fd40c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 0x7fd3f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3f40047c0 0x7fd3f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f40052a0 0x7fd3f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0cbf10 con 0x7fd3f4003970 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f400c170 con 0x7fd3f40047c0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f400c330 con 0x7fd3f40052a0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 0x7fd3f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40a59c640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3f40047c0 0x7fd3f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 -- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f40052a0 msgr2=0x7fd3f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f40052a0 0x7fd3f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 -- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3f40047c0 msgr2=0x7fd3f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3f40047c0 0x7fd3f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40ad9d640 1 -- 172.21.15.107:0/1355406796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4003520 con 0x7fd3f4003970 2023-11-24T08:17:24.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40b59e640 1 -- 172.21.15.107:0/1355406796 reap_dead start 2023-11-24T08:17:24.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.079+0000 7fd40a59c640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3f40047c0 0x7fd3f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 0x7fd3f4003d90 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c210 tx=0x7fd3f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b48e83171e63983 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc005ad0 con 0x7fd3f4003970 2023-11-24T08:17:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc005c30 con 0x7fd3f4003970 2023-11-24T08:17:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd409d9b640 1 -- 172.21.15.107:0/1355406796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc02cab0 con 0x7fd3f4003970 2023-11-24T08:17:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 -- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 msgr2=0x7fd3f4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 0x7fd3f4003d90 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c210 tx=0x7fd3f40057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 -- 172.21.15.107:0/1355406796 shutdown_connections 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 --2- 172.21.15.107:0/1355406796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3f4003970 0x7fd3f4003d90 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 -- 172.21.15.107:0/1355406796 >> 172.21.15.107:0/1355406796 conn(0x7fd40c070fa0 msgr2=0x7fd40c0713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.081+0000 7fd41101c640 1 -- 172.21.15.107:0/1355406796 shutdown_connections 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.082+0000 7fd41101c640 1 -- 172.21.15.107:0/1355406796 wait complete. 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.083+0000 7fd41101c640 1 Processor -- start 2023-11-24T08:17:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.083+0000 7fd41101c640 1 -- start start 2023-11-24T08:17:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f4003970 0x7fd40c146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd40c159e10 0x7fd40c15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c154a90 con 0x7fd40c15bae0 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c1f7890 con 0x7fd40c159e10 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.084+0000 7fd41101c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c1f75b0 con 0x7fd3f4003970 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40652/0 (socket says 172.21.15.107:40652) 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 -- 172.21.15.107:0/993936708 learned_addr learned my addr 172.21.15.107:0/993936708 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 -- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f4003970 msgr2=0x7fd40c146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f4003970 0x7fd40c146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 -- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd40c159e10 msgr2=0x7fd40c15a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd40c159e10 0x7fd40c15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 -- 172.21.15.107:0/993936708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c159330 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd40a59c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd40000d430 tx=0x7fd40000d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000b070 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4000054e0 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400003830 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.085+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c159610 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.086+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c155f10 con 0x7fd40c15bae0 2023-11-24T08:17:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.087+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c156860 con 0x7fd40c15bae0 2023-11-24T08:17:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.088+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fd400003990 con 0x7fd40c15bae0 2023-11-24T08:17:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.088+0000 7fd3f37fe640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 0x7fd3f806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.089+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7fd400087c00 con 0x7fd40c15bae0 2023-11-24T08:17:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.089+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 0x7fd3f806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.089+0000 7fd40ad9d640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 0x7fd3f806eaa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc034000 tx=0x7fd3fc039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.093+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4000547c0 con 0x7fd40c15bae0 2023-11-24T08:17:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.283+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd40c04e8b0 con 0x7fd40c15bae0 2023-11-24T08:17:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.284+0000 7fd3f37fe640 1 -- 172.21.15.107:0/993936708 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd400018030 con 0x7fd40c15bae0 2023-11-24T08:17:24.286 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:17:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 msgr2=0x7fd3f806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 0x7fd3f806eaa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc034000 tx=0x7fd3fc039040 comp rx=0 tx=0).stop 2023-11-24T08:17:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 msgr2=0x7fd40c15bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd40000d430 tx=0x7fd40000d8f0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 shutdown_connections 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd3f806c5e0 0x7fd3f806eaa0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd40c159e10 0x7fd40c15a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40c15bae0 0x7fd40c15bf60 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 --2- 172.21.15.107:0/993936708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3f4003970 0x7fd40c146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 >> 172.21.15.107:0/993936708 conn(0x7fd40c070fa0 msgr2=0x7fd40c06d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 shutdown_connections 2023-11-24T08:17:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.286+0000 7fd41101c640 1 -- 172.21.15.107:0/993936708 wait complete. 2023-11-24T08:17:24.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.511+0000 7fd7a6631640 1 Processor -- start 2023-11-24T08:17:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.511+0000 7fd7a6631640 1 -- start start 2023-11-24T08:17:24.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd7a6631640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd7a0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd7a6631640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00ce0b0 con 0x7fd7a0157c70 2023-11-24T08:17:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd7a6631640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0161960 con 0x7fd7a015dc80 2023-11-24T08:17:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd7a6631640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a015e8d0 con 0x7fd7a00cfb60 2023-11-24T08:17:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd79ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd7a0157c70 0x7fd7a00cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39206/0 (socket says 172.21.15.107:39206) 2023-11-24T08:17:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd79ffff640 1 -- 172.21.15.107:0/4057341192 learned_addr learned my addr 172.21.15.107:0/4057341192 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.513+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd7a0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141543008 0 0) 0x7fd7a00ce0b0 con 0x7fd7a0157c70 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003520 con 0x7fd7a0157c70 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258165745 0 0) 0x7fd788003520 con 0x7fd7a0157c70 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 msgr2=0x7fd7a0161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd7a0161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 >> v1:172.21.15.120:6789/0 conn(0x7fd7a015dc80 legacy=0x7fd7a00b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a015ebb0 con 0x7fd7a0157c70 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd790001670 con 0x7fd7a0157c70 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 >> v1:172.21.15.107:6789/0 conn(0x7fd7a0157c70 legacy=0x7fd7a00cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7880039a0 0x7fd788003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd788005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a015e8d0 con 0x7fd7880047c0 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800ba80 con 0x7fd7880039a0 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.514+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800bc40 con 0x7fd7a00cfb60 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd7a4bce640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd788005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79ffff640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7880039a0 0x7fd788003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 -- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 msgr2=0x7fd788005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cfb60 0x7fd788005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 -- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7880039a0 msgr2=0x7fd788003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7880039a0 0x7fd788003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79f7fe640 1 -- 172.21.15.107:0/4057341192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788003520 con 0x7fd7880047c0 2023-11-24T08:17:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.515+0000 7fd79ffff640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7880039a0 0x7fd788003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.516+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fd794009ac0 tx=0x7fd7880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3ddc56e7a389ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79402c6b0 con 0x7fd7880047c0 2023-11-24T08:17:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79402c810 con 0x7fd7880047c0 2023-11-24T08:17:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd79effd640 1 -- 172.21.15.107:0/4057341192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79402cba0 con 0x7fd7880047c0 2023-11-24T08:17:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 -- 172.21.15.107:0/4057341192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 msgr2=0x7fd788004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 --2- 172.21.15.107:0/4057341192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fd794009ac0 tx=0x7fd7880057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a4bce640 1 -- 172.21.15.107:0/4057341192 reap_dead start 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 -- 172.21.15.107:0/4057341192 shutdown_connections 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 -- 172.21.15.107:0/4057341192 >> 172.21.15.107:0/4057341192 conn(0x7fd7a006a920 msgr2=0x7fd7a006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 -- 172.21.15.107:0/4057341192 shutdown_connections 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.517+0000 7fd7a6631640 1 -- 172.21.15.107:0/4057341192 wait complete. 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.519+0000 7fd7a6631640 1 Processor -- start 2023-11-24T08:17:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.519+0000 7fd7a6631640 1 -- start start 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 0x7fd7a00cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7a00ad010 0x7fd7a00ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01f73f0 con 0x7fd7880047c0 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a00ad9d0 con 0x7fd7a00ad010 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.520+0000 7fd7a6631640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01f6770 con 0x7fd7a00cba30 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40674/0 (socket says 172.21.15.107:40674) 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 -- 172.21.15.107:0/697248496 learned_addr learned my addr 172.21.15.107:0/697248496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 0x7fd7a00cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd7a4bce640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7a00ad010 0x7fd7a00ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 -- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 msgr2=0x7fd7a00cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 0x7fd7a00cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 -- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7a00ad010 msgr2=0x7fd7a00ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7a00ad010 0x7fd7a00ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 -- 172.21.15.107:0/697248496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a01f6a50 con 0x7fd7880047c0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 0x7fd7a00cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.521+0000 7fd79ffff640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d7e0 tx=0x7fd79000dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.522+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900074e0 con 0x7fd7880047c0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.522+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790007640 con 0x7fd7880047c0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.522+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a01f6d30 con 0x7fd7880047c0 2023-11-24T08:17:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.522+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790004a20 con 0x7fd7880047c0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.522+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a0156310 con 0x7fd7880047c0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.523+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a0154d80 con 0x7fd7880047c0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.524+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fd790004b80 con 0x7fd7880047c0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.525+0000 7fd79cff9640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 0x7fd78c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.525+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7fd7900880d0 con 0x7fd7880047c0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.525+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 0x7fd78c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.526+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 0x7fd78c06ead0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd794036000 tx=0x7fd794030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.529+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd790054be0 con 0x7fd7880047c0 2023-11-24T08:17:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.755+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a0067470 con 0x7fd7880047c0 2023-11-24T08:17:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.756+0000 7fd79cff9640 1 -- 172.21.15.107:0/697248496 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4929 (secure 0 0 0) 0x7fd790019020 con 0x7fd7880047c0 2023-11-24T08:17:24.758 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:24.758 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:17:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 88 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 msgr2=0x7fd78c06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 0x7fd78c06ead0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd794036000 tx=0x7fd794030000 comp rx=0 tx=0).stop 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 msgr2=0x7fd7a0154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d7e0 tx=0x7fd79000dca0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 shutdown_connections 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd78c06c610 0x7fd78c06ead0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7a00ad010 0x7fd7a00ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7a00cba30 0x7fd7a00cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 --2- 172.21.15.107:0/697248496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7880047c0 0x7fd7a0154840 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 >> 172.21.15.107:0/697248496 conn(0x7fd7a006a920 msgr2=0x7fd7a00af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 shutdown_connections 2023-11-24T08:17:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.759+0000 7fd7a6631640 1 -- 172.21.15.107:0/697248496 wait complete. 2023-11-24T08:17:24.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:24.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.988+0000 7f8b11c90640 1 Processor -- start 2023-11-24T08:17:24.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.988+0000 7f8b11c90640 1 -- start start 2023-11-24T08:17:24.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.989+0000 7f8b11c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.989+0000 7f8b11c90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b0c0ce6c0 0x7f8b0c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.989+0000 7f8b11c90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0c0cece0 con 0x7f8b0c0cf450 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b11c90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0c156010 con 0x7f8b0c0ce6c0 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b11c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c06ce30 con 0x7f8b0c15c7f0 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b03fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8b0c0cf450 0x7f8b0c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39214/0 (socket says 172.21.15.107:39214) 2023-11-24T08:17:24.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b03fff640 1 -- 172.21.15.107:0/1941723272 learned_addr learned my addr 172.21.15.107:0/1941723272 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/1941723272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b0bfff640 1 --2- 172.21.15.107:0/1941723272 >> v2:172.21.15.120:3300/0 conn(0x7f8b0c0ce6c0 0x7f8b0c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.990+0000 7f8b0affd640 1 -- 172.21.15.107:0/1941723272 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189047463 0 0) 0x7f8b0c0cece0 con 0x7f8b0c0cf450 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.991+0000 7f8b0affd640 1 -- 172.21.15.107:0/1941723272 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af8003520 con 0x7f8b0c0cf450 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.991+0000 7f8b0b7fe640 1 -- 172.21.15.107:0/1941723272 >> v2:172.21.15.120:3300/0 conn(0x7f8b0c0ce6c0 msgr2=0x7f8b0c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.991+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/1941723272 >> v2:172.21.15.120:3300/0 conn(0x7f8b0c0ce6c0 0x7f8b0c0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.991+0000 7f8b0b7fe640 1 -- 172.21.15.107:0/1941723272 >> v1:172.21.15.107:6789/0 conn(0x7f8b0c0cf450 legacy=0x7f8b0c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.991+0000 7f8b0b7fe640 1 -- 172.21.15.107:0/1941723272 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c06d040 con 0x7f8b0c15c7f0 2023-11-24T08:17:24.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.992+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/1941723272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c06db20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59056fbb83e8e37 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:24.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.993+0000 7f8b0affd640 1 -- 172.21.15.107:0/1941723272 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc02d040 con 0x7f8b0c15c7f0 2023-11-24T08:17:24.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.993+0000 7f8b0affd640 1 -- 172.21.15.107:0/1941723272 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc031810 con 0x7f8b0c15c7f0 2023-11-24T08:17:24.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.993+0000 7f8b0affd640 1 -- 172.21.15.107:0/1941723272 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc031b80 con 0x7f8b0c15c7f0 2023-11-24T08:17:24.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.993+0000 7f8b11c90640 1 -- 172.21.15.107:0/1941723272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 msgr2=0x7f8b0c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.993+0000 7f8b11c90640 1 --2- 172.21.15.107:0/1941723272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c06db20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 -- 172.21.15.107:0/1941723272 shutdown_connections 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 --2- 172.21.15.107:0/1941723272 >> v2:172.21.15.120:3300/0 conn(0x7f8b0c0ce6c0 0x7f8b0c0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 --2- 172.21.15.107:0/1941723272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c06db20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 -- 172.21.15.107:0/1941723272 >> 172.21.15.107:0/1941723272 conn(0x7f8b0c06a920 msgr2=0x7f8b0c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 -- 172.21.15.107:0/1941723272 shutdown_connections 2023-11-24T08:17:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.994+0000 7f8b11c90640 1 -- 172.21.15.107:0/1941723272 wait complete. 2023-11-24T08:17:24.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.995+0000 7f8b11c90640 1 Processor -- start 2023-11-24T08:17:24.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.996+0000 7f8b11c90640 1 -- start start 2023-11-24T08:17:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.996+0000 7f8b11c90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 0x7f8b0c0cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.996+0000 7f8b11c90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b11c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c0adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b11c90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0aca20 con 0x7f8b0c0cf450 2023-11-24T08:17:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b11c90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0acd00 con 0x7f8b0c0ce6c0 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b11c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0acfe0 con 0x7f8b0c15c7f0 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40686/0 (socket says 172.21.15.107:40686) 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 -- 172.21.15.107:0/614906559 learned_addr learned my addr 172.21.15.107:0/614906559 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 0x7f8b0c0cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b0bfff640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c0adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 -- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 msgr2=0x7f8b0c0adac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c0adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 -- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 msgr2=0x7f8b0c0cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 0x7f8b0c0cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b03fff640 1 -- 172.21.15.107:0/614906559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009400 con 0x7f8b0c0cf450 2023-11-24T08:17:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.997+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 0x7f8b0c0cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b03fff640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c430 tx=0x7f8af400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af4005ce0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af4005e40 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af40072f0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c0ad2c0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.998+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c14c8b0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:24.999+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c0ad7e0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.000+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f8af4007450 con 0x7f8b0c0cf450 2023-11-24T08:17:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.002+0000 7f8b08ff9640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 0x7f8aec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.002+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5794+0+0 (secure 0 0 0) 0x7f8af4087a00 con 0x7f8b0c0cf450 2023-11-24T08:17:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.002+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 0x7f8aec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.003+0000 7f8b0b7fe640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 0x7f8aec06eaa0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8afc036000 tx=0x7f8afc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.007+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b0c0ad7e0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.201+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b0c0ad7e0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.204+0000 7f8b08ff9640 1 -- 172.21.15.107:0/614906559 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16696 (secure 0 0 0) 0x7f8b0c0ad7e0 con 0x7f8b0c0cf450 2023-11-24T08:17:25.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:25.207 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":151,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:25.001477+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.207+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 msgr2=0x7f8aec06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.207+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 0x7f8aec06eaa0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8afc036000 tx=0x7f8afc03c040 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.207+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 msgr2=0x7f8b0c14c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.207+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c430 tx=0x7f8af400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 shutdown_connections 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8aec06c5e0 0x7f8aec06eaa0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0c15c7f0 0x7f8b0c0adac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b0c0cf450 0x7f8b0c14c370 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 --2- 172.21.15.107:0/614906559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0c0ce6c0 0x7f8b0c0cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 >> 172.21.15.107:0/614906559 conn(0x7f8b0c06a920 msgr2=0x7f8b0c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:25.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 shutdown_connections 2023-11-24T08:17:25.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.208+0000 7f8b11c90640 1 -- 172.21.15.107:0/614906559 wait complete. 2023-11-24T08:17:25.229 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:17:25.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.438+0000 7fd1c10da640 1 Processor -- start 2023-11-24T08:17:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.438+0000 7fd1c10da640 1 -- start start 2023-11-24T08:17:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.440+0000 7fd1c10da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd1bc0703f0 0x7fd1bc162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.440+0000 7fd1c10da640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd1bc0cd160 0x7fd1bc0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.440+0000 7fd1c10da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc162eb0 con 0x7fd1bc14b100 2023-11-24T08:17:25.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.440+0000 7fd1c10da640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cdb10 con 0x7fd1bc0cd160 2023-11-24T08:17:25.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.440+0000 7fd1c10da640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc1654a0 con 0x7fd1bc0703f0 2023-11-24T08:17:25.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1bbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1bc14b100 0x7fd1bc165c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39222/0 (socket says 172.21.15.107:39222) 2023-11-24T08:17:25.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1bbfff640 1 -- 172.21.15.107:0/3117292642 learned_addr learned my addr 172.21.15.107:0/3117292642 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:25.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1c08d9640 1 --2- 172.21.15.107:0/3117292642 >> v2:172.21.15.107:3300/0 conn(0x7fd1bc0cd160 0x7fd1bc0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1bb7fe640 1 --2- 172.21.15.107:0/3117292642 >> v2:172.21.15.120:3300/0 conn(0x7fd1bc0703f0 0x7fd1bc162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1c08d9640 1 -- 172.21.15.107:0/3117292642 >> v2:172.21.15.120:3300/0 conn(0x7fd1bc0703f0 msgr2=0x7fd1bc162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1c08d9640 1 --2- 172.21.15.107:0/3117292642 >> v2:172.21.15.120:3300/0 conn(0x7fd1bc0703f0 0x7fd1bc162970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.441+0000 7fd1c08d9640 1 -- 172.21.15.107:0/3117292642 >> v1:172.21.15.107:6789/0 conn(0x7fd1bc14b100 legacy=0x7fd1bc165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.442+0000 7fd1baffd640 1 -- 172.21.15.107:0/3117292642 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618031778 0 0) 0x7fd1a4001110 con 0x7fd1bc14b100 2023-11-24T08:17:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.442+0000 7fd1c08d9640 1 -- 172.21.15.107:0/3117292642 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0ccf00 con 0x7fd1bc0cd160 2023-11-24T08:17:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.444+0000 7fd1c08d9640 1 --2- 172.21.15.107:0/3117292642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0cd5d0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc06e510 tx=0x7fd1b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70dd6c5a6e755dad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.445+0000 7fd1baffd640 1 -- 172.21.15.107:0/3117292642 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b402e070 con 0x7fd1bc0cd160 2023-11-24T08:17:25.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.445+0000 7fd1baffd640 1 -- 172.21.15.107:0/3117292642 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b40315a0 con 0x7fd1bc0cd160 2023-11-24T08:17:25.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.445+0000 7fd1baffd640 1 -- 172.21.15.107:0/3117292642 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4031930 con 0x7fd1bc0cd160 2023-11-24T08:17:25.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 -- 172.21.15.107:0/3117292642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 msgr2=0x7fd1bc0cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 --2- 172.21.15.107:0/3117292642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0cd5d0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc06e510 tx=0x7fd1b402b840 comp rx=0 tx=0).stop 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 -- 172.21.15.107:0/3117292642 shutdown_connections 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 --2- 172.21.15.107:0/3117292642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0cd5d0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 --2- 172.21.15.107:0/3117292642 >> v2:172.21.15.120:3300/0 conn(0x7fd1bc0703f0 0x7fd1bc162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 -- 172.21.15.107:0/3117292642 >> 172.21.15.107:0/3117292642 conn(0x7fd1bc0cb830 msgr2=0x7fd1bc0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 -- 172.21.15.107:0/3117292642 shutdown_connections 2023-11-24T08:17:25.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.446+0000 7fd1c10da640 1 -- 172.21.15.107:0/3117292642 wait complete. 2023-11-24T08:17:25.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.448+0000 7fd1c10da640 1 Processor -- start 2023-11-24T08:17:25.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.449+0000 7fd1c10da640 1 -- start start 2023-11-24T08:17:25.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1bc0703f0 0x7fd1bc146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1bc14b100 0x7fd1bc148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc161580 con 0x7fd1bc0cd160 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b2370 con 0x7fd1bc0703f0 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1c10da640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b4280 con 0x7fd1bc14b100 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40712/0 (socket says 172.21.15.107:40712) 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.450+0000 7fd1bb7fe640 1 -- 172.21.15.107:0/1194305551 learned_addr learned my addr 172.21.15.107:0/1194305551 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 -- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1bc14b100 msgr2=0x7fd1bc148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1bc14b100 0x7fd1bc148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 -- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1bc0703f0 msgr2=0x7fd1bc146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1bc0703f0 0x7fd1bc146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 -- 172.21.15.107:0/1194305551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4009400 con 0x7fd1bc0cd160 2023-11-24T08:17:25.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1bb7fe640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00ce70 tx=0x7fd1ac00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac003b90 con 0x7fd1bc0cd160 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac003cf0 con 0x7fd1bc0cd160 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0b1a40 con 0x7fd1bc0cd160 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac004260 con 0x7fd1bc0cd160 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.451+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc0b2000 con 0x7fd1bc0cd160 2023-11-24T08:17:25.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.453+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc04e840 con 0x7fd1bc0cd160 2023-11-24T08:17:25.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.454+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fd1ac0043c0 con 0x7fd1bc0cd160 2023-11-24T08:17:25.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.455+0000 7fd1b8ff9640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 0x7fd19c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.455+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fd1ac012070 con 0x7fd1bc0cd160 2023-11-24T08:17:25.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.456+0000 7fd1bbfff640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 0x7fd19c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.456+0000 7fd1bbfff640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 0x7fd19c06eaa0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007fd0 tx=0x7fd1a4007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.459+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1ac055950 con 0x7fd1bc0cd160 2023-11-24T08:17:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.688+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1bc0b24b0 con 0x7fd1bc0cd160 2023-11-24T08:17:25.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.690+0000 7fd1b8ff9640 1 -- 172.21.15.107:0/1194305551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4929 (secure 0 0 0) 0x7fd1bc0b24b0 con 0x7fd1bc0cd160 2023-11-24T08:17:25.692 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:25.692 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:17:25.692 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 88 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.692+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 msgr2=0x7fd19c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.692+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 0x7fd19c06eaa0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007fd0 tx=0x7fd1a4007960 comp rx=0 tx=0).stop 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.692+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 msgr2=0x7fd1bc0b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.692+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00ce70 tx=0x7fd1ac00a660 comp rx=0 tx=0).stop 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 shutdown_connections 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19c06c5e0 0x7fd19c06eaa0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1bc14b100 0x7fd1bc148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1bc0cd160 0x7fd1bc0b2800 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 --2- 172.21.15.107:0/1194305551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1bc0703f0 0x7fd1bc146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 >> 172.21.15.107:0/1194305551 conn(0x7fd1bc0cb830 msgr2=0x7fd1bc06ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:25.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 shutdown_connections 2023-11-24T08:17:25.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.693+0000 7fd1c10da640 1 -- 172.21.15.107:0/1194305551 wait complete. 2023-11-24T08:17:25.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:25.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.923+0000 7f4590c67640 1 Processor -- start 2023-11-24T08:17:25.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.923+0000 7f4590c67640 1 -- start start 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f4590c67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458c15d710 0x7f458c0d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f4590c67640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c161830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f4590c67640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458c0cdd70 con 0x7f458c0d17b0 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f4590c67640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c0b0e10 con 0x7f458c15d710 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f4590c67640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0cdff0 con 0x7f458c0d1bf0 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458c15d710 0x7f458c0d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458c15d710 0x7f458c0d3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40724/0 (socket says 172.21.15.107:40724) 2023-11-24T08:17:25.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 -- 172.21.15.107:0/2443499948 learned_addr learned my addr 172.21.15.107:0/2443499948 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 -- 172.21.15.107:0/2443499948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 msgr2=0x7f458c161830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 --2- 172.21.15.107:0/2443499948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c161830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.925+0000 7f458b7fe640 1 -- 172.21.15.107:0/2443499948 >> v1:172.21.15.120:6789/0 conn(0x7f458c0d17b0 legacy=0x7f458c161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.926+0000 7f458b7fe640 1 -- 172.21.15.107:0/2443499948 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0ce2d0 con 0x7f458c15d710 2023-11-24T08:17:25.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.927+0000 7f458b7fe640 1 --2- 172.21.15.107:0/2443499948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c15d710 0x7f458c0d3af0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f457c009870 tx=0x7f457c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcae9b2c641ff42 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.927+0000 7f458affd640 1 -- 172.21.15.107:0/2443499948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c030840 con 0x7f458c15d710 2023-11-24T08:17:25.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.927+0000 7f458affd640 1 -- 172.21.15.107:0/2443499948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c0309a0 con 0x7f458c15d710 2023-11-24T08:17:25.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.927+0000 7f458affd640 1 -- 172.21.15.107:0/2443499948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c030d30 con 0x7f458c15d710 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 -- 172.21.15.107:0/2443499948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c15d710 msgr2=0x7f458c0d3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 --2- 172.21.15.107:0/2443499948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c15d710 0x7f458c0d3af0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f457c009870 tx=0x7f457c02b980 comp rx=0 tx=0).stop 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 -- 172.21.15.107:0/2443499948 shutdown_connections 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 --2- 172.21.15.107:0/2443499948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c161830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 --2- 172.21.15.107:0/2443499948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c15d710 0x7f458c0d3af0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 -- 172.21.15.107:0/2443499948 >> 172.21.15.107:0/2443499948 conn(0x7f458c09c510 msgr2=0x7f458c0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 -- 172.21.15.107:0/2443499948 shutdown_connections 2023-11-24T08:17:25.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.928+0000 7f4590c67640 1 -- 172.21.15.107:0/2443499948 wait complete. 2023-11-24T08:17:25.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.930+0000 7f4590c67640 1 Processor -- start 2023-11-24T08:17:25.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.930+0000 7f4590c67640 1 -- start start 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c1f7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f458c15d710 0x7f458c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0b66d0 con 0x7f458c0d17b0 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c1f3740 con 0x7f458c15d710 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f4590c67640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c1f7da0 con 0x7f458c0d1bf0 2023-11-24T08:17:25.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40734/0 (socket says 172.21.15.107:40734) 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 -- 172.21.15.107:0/864151260 learned_addr learned my addr 172.21.15.107:0/864151260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 -- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 msgr2=0x7f458c1f7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c1f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 -- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f458c15d710 msgr2=0x7f458c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458bfff640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f458c15d710 0x7f458c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f458c15d710 0x7f458c0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.931+0000 7f458b7fe640 1 -- 172.21.15.107:0/864151260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0094d0 con 0x7f458c0d17b0 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f458b7fe640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f457c009990 tx=0x7f457c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c039b60 con 0x7f458c0d17b0 2023-11-24T08:17:25.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c039cc0 con 0x7f458c0d17b0 2023-11-24T08:17:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c043630 con 0x7f458c0d17b0 2023-11-24T08:17:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c146e60 con 0x7f458c0d17b0 2023-11-24T08:17:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.932+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c147100 con 0x7f458c0d17b0 2023-11-24T08:17:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.933+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c04e840 con 0x7f458c0d17b0 2023-11-24T08:17:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.934+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f457c04c3b0 con 0x7f458c0d17b0 2023-11-24T08:17:25.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.935+0000 7f4588ff9640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 0x7f456c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:25.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.935+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f457c0aaad0 con 0x7f458c0d17b0 2023-11-24T08:17:25.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.935+0000 7f4583fff640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 0x7f456c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:25.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.936+0000 7f4583fff640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 0x7f456c06eaa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f458c0b0b90 tx=0x7f4574007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:25.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:25.939+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f457c0775e0 con 0x7f458c0d17b0 2023-11-24T08:17:26.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.170+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458c0d1590 con 0x7f458c0d17b0 2023-11-24T08:17:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.171+0000 7f4588ff9640 1 -- 172.21.15.107:0/864151260 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f457c048090 con 0x7f458c0d17b0 2023-11-24T08:17:26.173 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:26.174 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:16:05.904591+0000","modified":"2023-11-24T08:16:08.999664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9300":{"gid":9300,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3938030686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3938030686},{"type":"v1","addr":"172.21.15.107:6836","nonce":3938030686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-24T08:17:26.174 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 88 2023-11-24T08:17:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.174+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 msgr2=0x7f456c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.174+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 0x7f456c06eaa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f458c0b0b90 tx=0x7f4574007990 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.174+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 msgr2=0x7f458c1f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.174+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f457c009990 tx=0x7f457c004730 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 shutdown_connections 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f456c06c5e0 0x7f456c06eaa0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f458c15d710 0x7f458c0b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f458c0d1bf0 0x7f458c1f7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 --2- 172.21.15.107:0/864151260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458c0d17b0 0x7f458c1f3200 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 >> 172.21.15.107:0/864151260 conn(0x7f458c09c510 msgr2=0x7f458c15e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 shutdown_connections 2023-11-24T08:17:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.175+0000 7f4590c67640 1 -- 172.21.15.107:0/864151260 wait complete. 2023-11-24T08:17:26.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.401+0000 7fbe6df01640 1 Processor -- start 2023-11-24T08:17:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.401+0000 7fbe6df01640 1 -- start start 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe6df01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 0x7fbe680cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe6df01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe680cdfb0 0x7fbe680ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe6df01640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680cece0 con 0x7fbe6815c7f0 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe6df01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe68156010 con 0x7fbe680cdfb0 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe6df01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6806ce30 con 0x7fbe680cf450 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe677fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbe6815c7f0 0x7fbe6806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39236/0 (socket says 172.21.15.107:39236) 2023-11-24T08:17:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe677fe640 1 -- 172.21.15.107:0/2328410149 learned_addr learned my addr 172.21.15.107:0/2328410149 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe5ffff640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 0x7fbe680cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2328410149 >> v2:172.21.15.120:3300/0 conn(0x7fbe680cdfb0 0x7fbe680ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.403+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814157439 0 0) 0x7fbe680cece0 con 0x7fbe6815c7f0 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54003520 con 0x7fbe6815c7f0 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3963592604 0 0) 0x7fbe54003520 con 0x7fbe6815c7f0 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 msgr2=0x7fbe680cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 0x7fbe680cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 >> v2:172.21.15.120:3300/0 conn(0x7fbe680cdfb0 msgr2=0x7fbe680ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 --2- 172.21.15.107:0/2328410149 >> v2:172.21.15.120:3300/0 conn(0x7fbe680cdfb0 0x7fbe680ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6806d040 con 0x7fbe6815c7f0 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbe58001670 con 0x7fbe6815c7f0 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 >> v1:172.21.15.107:6789/0 conn(0x7fbe6815c7f0 legacy=0x7fbe6806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 0x7fbe54003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe54003f40 0x7fbe540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe54004d00 0x7fbe54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68156010 con 0x7fbe54004d00 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6806ce30 con 0x7fbe54003f40 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5400b980 con 0x7fbe680cf450 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.404+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe54004d00 0x7fbe54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.405+0000 7fbe67fff640 1 -- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 msgr2=0x7fbe54003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.405+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe680cf450 0x7fbe54003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.405+0000 7fbe67fff640 1 -- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe54003f40 msgr2=0x7fbe540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:26.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.405+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe54003f40 0x7fbe540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.405+0000 7fbe67fff640 1 -- 172.21.15.107:0/2328410149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54003520 con 0x7fbe54004d00 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.406+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe54004d00 0x7fbe54005180 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000e170 tx=0x7fbe540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9731591bf5161be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.406+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6002cc20 con 0x7fbe54004d00 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.406+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6000f040 con 0x7fbe54004d00 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.406+0000 7fbe66ffd640 1 -- 172.21.15.107:0/2328410149 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60034440 con 0x7fbe54004d00 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 -- 172.21.15.107:0/2328410149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe54004d00 msgr2=0x7fbe54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2328410149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe54004d00 0x7fbe54005180 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000e170 tx=0x7fbe540057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe67fff640 1 -- 172.21.15.107:0/2328410149 reap_dead start 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 -- 172.21.15.107:0/2328410149 shutdown_connections 2023-11-24T08:17:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 -- 172.21.15.107:0/2328410149 >> 172.21.15.107:0/2328410149 conn(0x7fbe6806a920 msgr2=0x7fbe6806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 -- 172.21.15.107:0/2328410149 shutdown_connections 2023-11-24T08:17:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.407+0000 7fbe6df01640 1 -- 172.21.15.107:0/2328410149 wait complete. 2023-11-24T08:17:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.409+0000 7fbe6df01640 1 Processor -- start 2023-11-24T08:17:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.409+0000 7fbe6df01640 1 -- start start 2023-11-24T08:17:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe68147690 0x7fbe6806d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe68156b40 0x7fbe6806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681567e0 con 0x7fbe681472b0 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68159770 con 0x7fbe68156b40 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe6df01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68159a50 con 0x7fbe68147690 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40738/0 (socket says 172.21.15.107:40738) 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 -- 172.21.15.107:0/2829341781 learned_addr learned my addr 172.21.15.107:0/2829341781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 -- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe68147690 msgr2=0x7fbe6806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe68147690 0x7fbe6806d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 -- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe68156b40 msgr2=0x7fbe6806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe67fff640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe68156b40 0x7fbe6806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe68156b40 0x7fbe6806dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.410+0000 7fbe677fe640 1 -- 172.21.15.107:0/2829341781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68159d30 con 0x7fbe681472b0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe677fe640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800ca70 tx=0x7fbe5800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004cb0 con 0x7fbe681472b0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58004e30 con 0x7fbe681472b0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6815a010 con 0x7fbe681472b0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58007310 con 0x7fbe681472b0 2023-11-24T08:17:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.411+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68151ca0 con 0x7fbe681472b0 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.412+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68067470 con 0x7fbe681472b0 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.413+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fbe58007470 con 0x7fbe681472b0 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.414+0000 7fbe64ff9640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 0x7fbe4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.414+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fbe580878f0 con 0x7fbe681472b0 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.414+0000 7fbe5ffff640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 0x7fbe4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.415+0000 7fbe5ffff640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 0x7fbe4806eaa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbe68155170 tx=0x7fbe500094e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.418+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe580544a0 con 0x7fbe681472b0 2023-11-24T08:17:26.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.618+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe680cd730 con 0x7fbe681472b0 2023-11-24T08:17:26.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.620+0000 7fbe64ff9640 1 -- 172.21.15.107:0/2829341781 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16696 (secure 0 0 0) 0x7fbe58018030 con 0x7fbe681472b0 2023-11-24T08:17:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:26.622 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":151,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:25.001477+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:16:03.725981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-11-24T08:16:04.911827+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":21,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.622+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 msgr2=0x7fbe4806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.622+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 0x7fbe4806eaa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbe68155170 tx=0x7fbe500094e0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 msgr2=0x7fbe68156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800ca70 tx=0x7fbe5800cf30 comp rx=0 tx=0).stop 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 shutdown_connections 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe4806c5e0 0x7fbe4806eaa0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe68156b40 0x7fbe6806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe68147690 0x7fbe6806d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 --2- 172.21.15.107:0/2829341781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe681472b0 0x7fbe68156150 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 >> 172.21.15.107:0/2829341781 conn(0x7fbe6806a920 msgr2=0x7fbe680ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 shutdown_connections 2023-11-24T08:17:26.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.623+0000 7fbe6df01640 1 -- 172.21.15.107:0/2829341781 wait complete. 2023-11-24T08:17:26.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:17:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.859+0000 7f883bc32640 1 Processor -- start 2023-11-24T08:17:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.860+0000 7f883bc32640 1 -- start start 2023-11-24T08:17:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883bc32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f883415b440 0x7f88340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883bc32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340c5dc0 con 0x7f88340c59b0 2023-11-24T08:17:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883bc32640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883415eea0 con 0x7f88340cbf20 2023-11-24T08:17:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883bc32640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88340cc3d0 con 0x7f883415b440 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f88399ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f883415b440 0x7f88340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883a1cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f88340c59b0 0x7f883406c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39250/0 (socket says 172.21.15.107:39250) 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.861+0000 7f883a1cf640 1 -- 172.21.15.107:0/2285315960 learned_addr learned my addr 172.21.15.107:0/2285315960 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.862+0000 7f88389cc640 1 -- 172.21.15.107:0/2285315960 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258937200 0 0) 0x7f88340c5dc0 con 0x7f88340c59b0 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.862+0000 7f88389cc640 1 -- 172.21.15.107:0/2285315960 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8820003520 con 0x7f88340c59b0 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.862+0000 7f88399ce640 1 -- 172.21.15.107:0/2285315960 >> v1:172.21.15.120:6789/0 conn(0x7f88340cbf20 legacy=0x7f88340c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.862+0000 7f88399ce640 1 -- 172.21.15.107:0/2285315960 >> v1:172.21.15.107:6789/0 conn(0x7f88340c59b0 legacy=0x7f883406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.862+0000 7f88399ce640 1 -- 172.21.15.107:0/2285315960 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340abf30 con 0x7f883415b440 2023-11-24T08:17:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.863+0000 7f88399ce640 1 --2- 172.21.15.107:0/2285315960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340cb9e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8830009d90 tx=0x7f883002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6ed94b473de71f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.863+0000 7f88389cc640 1 -- 172.21.15.107:0/2285315960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883002cbd0 con 0x7f883415b440 2023-11-24T08:17:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.863+0000 7f88389cc640 1 -- 172.21.15.107:0/2285315960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883002cd30 con 0x7f883415b440 2023-11-24T08:17:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.863+0000 7f88389cc640 1 -- 172.21.15.107:0/2285315960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883002cea0 con 0x7f883415b440 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 -- 172.21.15.107:0/2285315960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 msgr2=0x7f88340cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 --2- 172.21.15.107:0/2285315960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340cb9e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8830009d90 tx=0x7f883002be20 comp rx=0 tx=0).stop 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 -- 172.21.15.107:0/2285315960 shutdown_connections 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 --2- 172.21.15.107:0/2285315960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340cb9e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 -- 172.21.15.107:0/2285315960 >> 172.21.15.107:0/2285315960 conn(0x7f88340b0c50 msgr2=0x7f88340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 -- 172.21.15.107:0/2285315960 shutdown_connections 2023-11-24T08:17:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.864+0000 7f883bc32640 1 -- 172.21.15.107:0/2285315960 wait complete. 2023-11-24T08:17:26.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.866+0000 7f883bc32640 1 Processor -- start 2023-11-24T08:17:26.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.866+0000 7f883bc32640 1 -- start start 2023-11-24T08:17:26.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88340c59b0 0x7f8834149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 0x7f88340ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341e55b0 con 0x7f883415b440 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341e5890 con 0x7f88340cbf20 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883bc32640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88341e5b70 con 0x7f88340c59b0 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883a1cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883a1cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40762/0 (socket says 172.21.15.107:40762) 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f883a1cf640 1 -- 172.21.15.107:0/2200822221 learned_addr learned my addr 172.21.15.107:0/2200822221 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f88399ce640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88340c59b0 0x7f8834149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.867+0000 7f88391cd640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 0x7f88340ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 -- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88340c59b0 msgr2=0x7f8834149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88340c59b0 0x7f8834149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 -- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 msgr2=0x7f88340ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 0x7f88340ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 -- 172.21.15.107:0/2200822221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8830009a70 con 0x7f883415b440 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f88391cd640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 0x7f88340ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:26.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883a1cf640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c430 tx=0x7f882c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c005ce0 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c005e40 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c007310 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88341e5e50 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.868+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883414a360 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.869+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88341e6370 con 0x7f883415b440 2023-11-24T08:17:26.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.870+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f882c007470 con 0x7f883415b440 2023-11-24T08:17:26.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.871+0000 7f882a7fc640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 0x7f881806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:26.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.871+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f882c087a60 con 0x7f883415b440 2023-11-24T08:17:26.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.872+0000 7f88399ce640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 0x7f881806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.872+0000 7f88399ce640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 0x7f881806ead0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8830036000 tx=0x7f8830030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:26.876+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88341e6370 con 0x7f883415b440 2023-11-24T08:17:27.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.106+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f88341e6370 con 0x7f883415b440 2023-11-24T08:17:27.109 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:17:27.107+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:17:27.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.110+0000 7f882a7fc640 1 -- 172.21.15.107:0/2200822221 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v89) v1 ==== 161+0+0 (secure 0 0 0) 0x7f88341e6370 con 0x7f883415b440 2023-11-24T08:17:27.113 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:17:27.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 msgr2=0x7f881806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 0x7f881806ead0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8830036000 tx=0x7f8830030000 comp rx=0 tx=0).stop 2023-11-24T08:17:27.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 msgr2=0x7f88340c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c430 tx=0x7f882c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 shutdown_connections 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f881806c610 0x7f881806ead0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f883415b440 0x7f88340c60d0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88340cbf20 0x7f88340ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 --2- 172.21.15.107:0/2200822221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88340c59b0 0x7f8834149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.115+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 >> 172.21.15.107:0/2200822221 conn(0x7f88340b0c50 msgr2=0x7f88340cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.116+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 shutdown_connections 2023-11-24T08:17:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.116+0000 7f883bc32640 1 -- 172.21.15.107:0/2200822221 wait complete. 2023-11-24T08:17:27.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:17:27.232 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:17:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.336+0000 7f584840e640 1 Processor -- start 2023-11-24T08:17:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.336+0000 7f584840e640 1 -- start start 2023-11-24T08:17:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.337+0000 7f584840e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58400ccbd0 0x7f58401660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.337+0000 7f584840e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 0x7f5840162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.337+0000 7f584840e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58401655f0 con 0x7f5840162540 2023-11-24T08:17:27.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.337+0000 7f584840e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400cc380 con 0x7f58400ccbd0 2023-11-24T08:17:27.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.337+0000 7f584840e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5840162b30 con 0x7f58400ccfb0 2023-11-24T08:17:27.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.338+0000 7f58461aa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58400ccbd0 0x7f58401660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.338+0000 7f58459a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 0x7f5840162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.338+0000 7f58461aa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58400ccbd0 0x7f58401660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:47270/0 (socket says 172.21.15.107:47270) 2023-11-24T08:17:27.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.338+0000 7f58459a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 0x7f5840162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:59292/0 (socket says 172.21.15.107:59292) 2023-11-24T08:17:27.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.338+0000 7f58461aa640 1 -- 172.21.15.107:0/696284597 learned_addr learned my addr 172.21.15.107:0/696284597 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:27.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.339+0000 7f58461aa640 1 -- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 msgr2=0x7f5840162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:27.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.339+0000 7f58461aa640 1 --2- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 0x7f5840162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.339+0000 7f58461aa640 1 -- 172.21.15.107:0/696284597 >> v1:172.21.15.120:6789/0 conn(0x7f5840162540 legacy=0x7f58400c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.339+0000 7f58461aa640 1 -- 172.21.15.107:0/696284597 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840162e10 con 0x7f58400ccbd0 2023-11-24T08:17:27.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.340+0000 7f58461aa640 1 --2- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccbd0 0x7f58401660b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098a0 tx=0x7f583c02b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be081d591560187d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:27.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.341+0000 7f58451a8640 1 -- 172.21.15.107:0/696284597 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c030830 con 0x7f58400ccbd0 2023-11-24T08:17:27.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.341+0000 7f58451a8640 1 -- 172.21.15.107:0/696284597 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c030990 con 0x7f58400ccbd0 2023-11-24T08:17:27.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.341+0000 7f58451a8640 1 -- 172.21.15.107:0/696284597 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c030d20 con 0x7f58400ccbd0 2023-11-24T08:17:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 -- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccbd0 msgr2=0x7f58401660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 --2- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccbd0 0x7f58401660b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098a0 tx=0x7f583c02b970 comp rx=0 tx=0).stop 2023-11-24T08:17:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 -- 172.21.15.107:0/696284597 shutdown_connections 2023-11-24T08:17:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 --2- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccfb0 0x7f5840162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 --2- 172.21.15.107:0/696284597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccbd0 0x7f58401660b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 -- 172.21.15.107:0/696284597 >> 172.21.15.107:0/696284597 conn(0x7f58400b0c50 msgr2=0x7f58400b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 -- 172.21.15.107:0/696284597 shutdown_connections 2023-11-24T08:17:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.342+0000 7f584840e640 1 -- 172.21.15.107:0/696284597 wait complete. 2023-11-24T08:17:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.344+0000 7f584840e640 1 Processor -- start 2023-11-24T08:17:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.344+0000 7f584840e640 1 -- start start 2023-11-24T08:17:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccbd0 0x7f5840157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccfb0 0x7f584015f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400abf30 con 0x7f5840162540 2023-11-24T08:17:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840157d30 con 0x7f58400ccfb0 2023-11-24T08:17:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f584840e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5840158720 con 0x7f58400ccbd0 2023-11-24T08:17:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f58469ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f58469ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40794/0 (socket says 172.21.15.107:40794) 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.345+0000 7f58469ab640 1 -- 172.21.15.107:0/4294392090 learned_addr learned my addr 172.21.15.107:0/4294392090 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58461aa640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccbd0 0x7f5840157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58459a9640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccfb0 0x7f584015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 -- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccbd0 msgr2=0x7f5840157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccbd0 0x7f5840157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 -- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccfb0 msgr2=0x7f584015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:27.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccfb0 0x7f584015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 -- 172.21.15.107:0/4294392090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0094d0 con 0x7f5840162540 2023-11-24T08:17:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f58469ab640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f583800c7a0 tx=0x7f583800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838003b70 con 0x7f5840162540 2023-11-24T08:17:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838003cf0 con 0x7f5840162540 2023-11-24T08:17:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838003ea0 con 0x7f5840162540 2023-11-24T08:17:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.346+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58400adbe0 con 0x7f5840162540 2023-11-24T08:17:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.347+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58400ac140 con 0x7f5840162540 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.348+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58400ac360 con 0x7f5840162540 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.349+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f583800aa90 con 0x7f5840162540 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.349+0000 7f5836ffd640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 0x7f582406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.350+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f58380881d0 con 0x7f5840162540 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.350+0000 7f58461aa640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 0x7f582406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.350+0000 7f58461aa640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 0x7f582406eaa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f583c036000 tx=0x7f583c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.354+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5838054d80 con 0x7f5840162540 2023-11-24T08:17:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:27.579+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f584004e8b0 con 0x7f5840162540 2023-11-24T08:17:28.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.121+0000 7f5836ffd640 1 -- 172.21.15.107:0/4294392090 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v90) v1 ==== 104+0+0 (secure 0 0 0) 0x7f58380583a0 con 0x7f5840162540 2023-11-24T08:17:28.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.125+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 msgr2=0x7f582406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.125+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 0x7f582406eaa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f583c036000 tx=0x7f583c03b040 comp rx=0 tx=0).stop 2023-11-24T08:17:28.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.125+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 msgr2=0x7f58401577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.125+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f583800c7a0 tx=0x7f583800cc60 comp rx=0 tx=0).stop 2023-11-24T08:17:28.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 shutdown_connections 2023-11-24T08:17:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f582406c5e0 0x7f582406eaa0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5840162540 0x7f58401577f0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58400ccfb0 0x7f584015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 --2- 172.21.15.107:0/4294392090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58400ccbd0 0x7f5840157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 >> 172.21.15.107:0/4294392090 conn(0x7f58400b0c50 msgr2=0x7f58400cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 shutdown_connections 2023-11-24T08:17:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.126+0000 7f584840e640 1 -- 172.21.15.107:0/4294392090 wait complete. 2023-11-24T08:17:28.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:17:28.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.356+0000 7fcbca8bb640 1 Processor -- start 2023-11-24T08:17:28.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.357+0000 7fcbca8bb640 1 -- start start 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca8bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcbc40d1850 0x7fcbc406c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca8bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc406c410 con 0x7fcbc406f1f0 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca8bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc406cde0 con 0x7fcbc40d1c30 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca8bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40b4dd0 con 0x7fcbc40d1850 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbc98b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcbc40d1850 0x7fcbc406c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca0ba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcbc406f1f0 0x7fcbc40b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47600/0 (socket says 172.21.15.107:47600) 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbca0ba640 1 -- 172.21.15.107:0/3235827938 learned_addr learned my addr 172.21.15.107:0/3235827938 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:28.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbc98b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcbc40d1850 0x7fcbc406c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36802/0 (socket says 172.21.15.107:36802) 2023-11-24T08:17:28.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbc98b9640 1 -- 172.21.15.107:0/3235827938 >> v1:172.21.15.120:6789/0 conn(0x7fcbc40d1c30 legacy=0x7fcbc406eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:28.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.358+0000 7fcbc98b9640 1 -- 172.21.15.107:0/3235827938 >> v1:172.21.15.107:6789/0 conn(0x7fcbc406f1f0 legacy=0x7fcbc40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.359+0000 7fcbc98b9640 1 -- 172.21.15.107:0/3235827938 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc406bae0 con 0x7fcbc40d1850 2023-11-24T08:17:28.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.359+0000 7fcbc88b7640 1 -- 172.21.15.107:0/3235827938 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385862222 0 0) 0x7fcbc406c410 con 0x7fcbc406f1f0 2023-11-24T08:17:28.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbc98b9640 1 --2- 172.21.15.107:0/3235827938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc406c8a0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00097d0 tx=0x7fcbc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3825eb59a290649 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbc88b7640 1 -- 172.21.15.107:0/3235827938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc002e070 con 0x7fcbc40d1850 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbc88b7640 1 -- 172.21.15.107:0/3235827938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc00315a0 con 0x7fcbc40d1850 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbc88b7640 1 -- 172.21.15.107:0/3235827938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0031930 con 0x7fcbc40d1850 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbca8bb640 1 -- 172.21.15.107:0/3235827938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 msgr2=0x7fcbc406c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/3235827938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc406c8a0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00097d0 tx=0x7fcbc002b840 comp rx=0 tx=0).stop 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbca8bb640 1 -- 172.21.15.107:0/3235827938 shutdown_connections 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/3235827938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc406c8a0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.360+0000 7fcbca8bb640 1 -- 172.21.15.107:0/3235827938 >> 172.21.15.107:0/3235827938 conn(0x7fcbc4061100 msgr2=0x7fcbc40614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.361+0000 7fcbca8bb640 1 -- 172.21.15.107:0/3235827938 shutdown_connections 2023-11-24T08:17:28.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.361+0000 7fcbca8bb640 1 -- 172.21.15.107:0/3235827938 wait complete. 2023-11-24T08:17:28.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.362+0000 7fcbca8bb640 1 Processor -- start 2023-11-24T08:17:28.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.362+0000 7fcbca8bb640 1 -- start start 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc406f1f0 0x7fcbc40d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc40d1c30 0x7fcbc415b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40ccff0 con 0x7fcbc40d1850 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40d1150 con 0x7fcbc406f1f0 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbca8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc415fd20 con 0x7fcbc40d1c30 2023-11-24T08:17:28.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbc90b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbc90b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36810/0 (socket says 172.21.15.107:36810) 2023-11-24T08:17:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.363+0000 7fcbc90b8640 1 -- 172.21.15.107:0/2419243564 learned_addr learned my addr 172.21.15.107:0/2419243564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 -- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc40d1c30 msgr2=0x7fcbc415b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc40d1c30 0x7fcbc415b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 -- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc406f1f0 msgr2=0x7fcbc40d0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc406f1f0 0x7fcbc40d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 -- 172.21.15.107:0/2419243564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc0009400 con 0x7fcbc40d1850 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbc90b8640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400b3e0 tx=0x7fcbb400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4011830 con 0x7fcbc40d1850 2023-11-24T08:17:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb40119b0 con 0x7fcbc40d1850 2023-11-24T08:17:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4011d60 con 0x7fcbc40d1850 2023-11-24T08:17:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.364+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4160000 con 0x7fcbc40d1850 2023-11-24T08:17:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.365+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc41605c0 con 0x7fcbc40d1850 2023-11-24T08:17:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.365+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc404e840 con 0x7fcbc40d1850 2023-11-24T08:17:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.367+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fcbb401a400 con 0x7fcbc40d1850 2023-11-24T08:17:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.367+0000 7fcbba7fc640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 0x7fcba806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.368+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fcbb4088540 con 0x7fcbc40d1850 2023-11-24T08:17:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.368+0000 7fcbc98b9640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 0x7fcba806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:28.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.368+0000 7fcbc98b9640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 0x7fcba806eaa0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0036000 tx=0x7fcbc0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:28.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.372+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbb4055070 con 0x7fcbc40d1850 2023-11-24T08:17:28.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:28.557+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcbc4070e20 con 0x7fcbc40d1850 2023-11-24T08:17:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.124+0000 7fcbba7fc640 1 -- 172.21.15.107:0/2419243564 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v153) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcbb4015020 con 0x7fcbc40d1850 2023-11-24T08:17:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:17:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 msgr2=0x7fcba806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 0x7fcba806eaa0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0036000 tx=0x7fcbc0030000 comp rx=0 tx=0).stop 2023-11-24T08:17:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 msgr2=0x7fcbc4161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400b3e0 tx=0x7fcbb400b8a0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 shutdown_connections 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcba806c5e0 0x7fcba806eaa0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc40d1c30 0x7fcbc415b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbc40d1850 0x7fcbc4161180 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 --2- 172.21.15.107:0/2419243564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc406f1f0 0x7fcbc40d0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 >> 172.21.15.107:0/2419243564 conn(0x7fcbc4061100 msgr2=0x7fcbc4158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 shutdown_connections 2023-11-24T08:17:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.128+0000 7fcbca8bb640 1 -- 172.21.15.107:0/2419243564 wait complete. 2023-11-24T08:17:29.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:17:29.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.363+0000 7f5c963ff640 1 Processor -- start 2023-11-24T08:17:29.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.364+0000 7f5c963ff640 1 -- start start 2023-11-24T08:17:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.365+0000 7f5c963ff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c901419f0 0x7f5c90142110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.365+0000 7f5c963ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c900cc8c0 0x7f5c900ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.365+0000 7f5c963ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9006ed10 con 0x7f5c9006a500 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.365+0000 7f5c963ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c900cd2b0 con 0x7f5c900cc8c0 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.365+0000 7f5c963ff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c90141dd0 con 0x7f5c901419f0 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c900cc8c0 0x7f5c900ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c900cc8c0 0x7f5c900ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36826/0 (socket says 172.21.15.107:36826) 2023-11-24T08:17:29.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 -- 172.21.15.107:0/1840770377 learned_addr learned my addr 172.21.15.107:0/1840770377 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c953fd640 1 --2- 172.21.15.107:0/1840770377 >> v2:172.21.15.120:3300/0 conn(0x7f5c901419f0 0x7f5c90142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 -- 172.21.15.107:0/1840770377 >> v2:172.21.15.120:3300/0 conn(0x7f5c901419f0 msgr2=0x7f5c90142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 --2- 172.21.15.107:0/1840770377 >> v2:172.21.15.120:3300/0 conn(0x7f5c901419f0 0x7f5c90142110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 -- 172.21.15.107:0/1840770377 >> v1:172.21.15.120:6789/0 conn(0x7f5c9006a500 legacy=0x7f5c90072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:29.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.366+0000 7f5c95bfe640 1 -- 172.21.15.107:0/1840770377 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9006aa70 con 0x7f5c900cc8c0 2023-11-24T08:17:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.368+0000 7f5c95bfe640 1 --2- 172.21.15.107:0/1840770377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c900cc8c0 0x7f5c900ccd70 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f5c9006c5c0 tx=0x7f5c8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece157a148fe7ca7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.368+0000 7f5c8ffff640 1 -- 172.21.15.107:0/1840770377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8802ca80 con 0x7f5c900cc8c0 2023-11-24T08:17:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.368+0000 7f5c8ffff640 1 -- 172.21.15.107:0/1840770377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8802cbe0 con 0x7f5c900cc8c0 2023-11-24T08:17:29.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.368+0000 7f5c8ffff640 1 -- 172.21.15.107:0/1840770377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880343f0 con 0x7f5c900cc8c0 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 -- 172.21.15.107:0/1840770377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c900cc8c0 msgr2=0x7f5c900ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 --2- 172.21.15.107:0/1840770377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c900cc8c0 0x7f5c900ccd70 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f5c9006c5c0 tx=0x7f5c8802bee0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 -- 172.21.15.107:0/1840770377 shutdown_connections 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 --2- 172.21.15.107:0/1840770377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c900cc8c0 0x7f5c900ccd70 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 --2- 172.21.15.107:0/1840770377 >> v2:172.21.15.120:3300/0 conn(0x7f5c901419f0 0x7f5c90142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 -- 172.21.15.107:0/1840770377 >> 172.21.15.107:0/1840770377 conn(0x7f5c900cfed0 msgr2=0x7f5c900d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 -- 172.21.15.107:0/1840770377 shutdown_connections 2023-11-24T08:17:29.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.369+0000 7f5c963ff640 1 -- 172.21.15.107:0/1840770377 wait complete. 2023-11-24T08:17:29.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.371+0000 7f5c963ff640 1 Processor -- start 2023-11-24T08:17:29.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.371+0000 7f5c963ff640 1 -- start start 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9006a500 0x7f5c9015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c900cc8c0 0x7f5c900b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9012f660 con 0x7f5c901419f0 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900c6ce0 con 0x7f5c9006a500 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.372+0000 7f5c963ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9012f900 con 0x7f5c900cc8c0 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36840/0 (socket says 172.21.15.107:36840) 2023-11-24T08:17:29.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 -- 172.21.15.107:0/620509253 learned_addr learned my addr 172.21.15.107:0/620509253 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c953fd640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9006a500 0x7f5c9015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 -- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c900cc8c0 msgr2=0x7f5c900b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c900cc8c0 0x7f5c900b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 -- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9006a500 msgr2=0x7f5c9015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9006a500 0x7f5c9015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:29.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.373+0000 7f5c95bfe640 1 -- 172.21.15.107:0/620509253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88009b30 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c95bfe640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f5c88036040 tx=0x7f5c88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88009510 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8803e450 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8803e800 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9012fbe0 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.374+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c900c6fc0 con 0x7f5c901419f0 2023-11-24T08:17:29.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.375+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9012fdd0 con 0x7f5c901419f0 2023-11-24T08:17:29.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.376+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f5c8802cd50 con 0x7f5c901419f0 2023-11-24T08:17:29.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.377+0000 7f5c8dffb640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 0x7f5c7c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:29.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.378+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f5c880aaf50 con 0x7f5c901419f0 2023-11-24T08:17:29.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.378+0000 7f5c953fd640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 0x7f5c7c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:29.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.378+0000 7f5c953fd640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 0x7f5c7c06eaa0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:29.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.382+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c88077a60 con 0x7f5c901419f0 2023-11-24T08:17:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:29.570+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5c9004e840 con 0x7f5c901419f0 2023-11-24T08:17:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.128+0000 7f5c8dffb640 1 -- 172.21.15.107:0/620509253 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v154) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5c88043020 con 0x7f5c901419f0 2023-11-24T08:17:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:17:30.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 msgr2=0x7f5c7c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 0x7f5c7c06eaa0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80008040 comp rx=0 tx=0).stop 2023-11-24T08:17:30.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 msgr2=0x7f5c900b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f5c88036040 tx=0x7f5c88004a40 comp rx=0 tx=0).stop 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 shutdown_connections 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c7c06c5e0 0x7f5c7c06eaa0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c901419f0 0x7f5c900b0d40 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c900cc8c0 0x7f5c900b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 --2- 172.21.15.107:0/620509253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9006a500 0x7f5c9015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.131+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 >> 172.21.15.107:0/620509253 conn(0x7f5c900cfed0 msgr2=0x7f5c9015e040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.132+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 shutdown_connections 2023-11-24T08:17:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.132+0000 7f5c963ff640 1 -- 172.21.15.107:0/620509253 wait complete. 2023-11-24T08:17:30.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:17:30.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.370+0000 7fc8f2044640 1 Processor -- start 2023-11-24T08:17:30.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.371+0000 7fc8f2044640 1 -- start start 2023-11-24T08:17:30.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.372+0000 7fc8f2044640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccbd0 0x7fc8ec1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.372+0000 7fc8f2044640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.372+0000 7fc8f2044640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec1655f0 con 0x7fc8ec0ccfb0 2023-11-24T08:17:30.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.372+0000 7fc8f2044640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0cc380 con 0x7fc8ec0c9650 2023-11-24T08:17:30.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.372+0000 7fc8f2044640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec162b30 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.373+0000 7fc8ebfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.373+0000 7fc8ebfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:43724/0 (socket says 172.21.15.107:43724) 2023-11-24T08:17:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.373+0000 7fc8ebfff640 1 -- 172.21.15.107:0/2335102239 learned_addr learned my addr 172.21.15.107:0/2335102239 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.373+0000 7fc8eb7fe640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccbd0 0x7fc8ec1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.374+0000 7fc8ebfff640 1 -- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccbd0 msgr2=0x7fc8ec1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:30.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.374+0000 7fc8ebfff640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccbd0 0x7fc8ec1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.374+0000 7fc8ebfff640 1 -- 172.21.15.107:0/2335102239 >> v1:172.21.15.120:6789/0 conn(0x7fc8ec0ccfb0 legacy=0x7fc8ec162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.374+0000 7fc8ebfff640 1 -- 172.21.15.107:0/2335102239 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec162e10 con 0x7fc8ec0c9650 2023-11-24T08:17:30.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.375+0000 7fc8ebfff640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009e80 tx=0x7fc8e402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8a134193c4d4c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8eaffd640 1 -- 172.21.15.107:0/2335102239 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e402cae0 con 0x7fc8ec0c9650 2023-11-24T08:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8eaffd640 1 -- 172.21.15.107:0/2335102239 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e402cc40 con 0x7fc8ec0c9650 2023-11-24T08:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8eaffd640 1 -- 172.21.15.107:0/2335102239 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40343f0 con 0x7fc8ec0c9650 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 -- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 msgr2=0x7fc8ec0c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009e80 tx=0x7fc8e402bf40 comp rx=0 tx=0).stop 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 -- 172.21.15.107:0/2335102239 shutdown_connections 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c9ae0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 --2- 172.21.15.107:0/2335102239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccbd0 0x7fc8ec1660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 -- 172.21.15.107:0/2335102239 >> 172.21.15.107:0/2335102239 conn(0x7fc8ec0b0c50 msgr2=0x7fc8ec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 -- 172.21.15.107:0/2335102239 shutdown_connections 2023-11-24T08:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.376+0000 7fc8f2044640 1 -- 172.21.15.107:0/2335102239 wait complete. 2023-11-24T08:17:30.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.378+0000 7fc8f2044640 1 Processor -- start 2023-11-24T08:17:30.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.378+0000 7fc8f2044640 1 -- start start 2023-11-24T08:17:30.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccfb0 0x7fc8ec0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c8930 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c7a80 con 0x7fc8ec0c9650 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.379+0000 7fc8f2044640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c7d20 con 0x7fc8ec0ccfb0 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36850/0 (socket says 172.21.15.107:36850) 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 -- 172.21.15.107:0/218796943 learned_addr learned my addr 172.21.15.107:0/218796943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8eb7fe640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8ebfff640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccfb0 0x7fc8ec0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 -- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccfb0 msgr2=0x7fc8ec0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccfb0 0x7fc8ec0c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 -- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 msgr2=0x7fc8ec0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 -- 172.21.15.107:0/218796943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009b30 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8eb7fe640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.380+0000 7fc8e3fff640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c450 tx=0x7fc8d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.381+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40052d0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.381+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d4005430 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.381+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4012870 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.381+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec0c8000 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.381+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec0c85c0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.382+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec0c8aa0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.383+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fc8d40129d0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.384+0000 7fc8e8ff9640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 0x7fc8cc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.384+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8d4087eb0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.384+0000 7fc8eb7fe640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 0x7fc8cc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.385+0000 7fc8eb7fe640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 0x7fc8cc06eaa0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009ac0 tx=0x7fc8dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.388+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8d4054a60 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.581+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc8ec04e8b0 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.588+0000 7fc8e8ff9640 1 -- 172.21.15.107:0/218796943 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc8d4058080 con 0x7fc8ec0ccbd0 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.590+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 msgr2=0x7fc8cc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.590+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 0x7fc8cc06eaa0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009ac0 tx=0x7fc8dc008040 comp rx=0 tx=0).stop 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.590+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 msgr2=0x7fc8ec157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.590+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c450 tx=0x7fc8d400c910 comp rx=0 tx=0).stop 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 shutdown_connections 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8cc06c5e0 0x7fc8cc06eaa0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8ec0ccfb0 0x7fc8ec0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec0ccbd0 0x7fc8ec157cb0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 --2- 172.21.15.107:0/218796943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8ec0c9650 0x7fc8ec0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 >> 172.21.15.107:0/218796943 conn(0x7fc8ec0b0c50 msgr2=0x7fc8ec0aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:30.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 shutdown_connections 2023-11-24T08:17:30.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.591+0000 7fc8f2044640 1 -- 172.21.15.107:0/218796943 wait complete. 2023-11-24T08:17:30.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:17:30.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.816+0000 7f1cf0f90640 1 Processor -- start 2023-11-24T08:17:30.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.816+0000 7f1cf0f90640 1 -- start start 2023-11-24T08:17:30.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cec15b440 0x7f1cec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 0x7f1cec0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec0cc3d0 con 0x7f1cec15b440 2023-11-24T08:17:30.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec15b820 con 0x7f1cec0cbf20 2023-11-24T08:17:30.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cf0f90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15eea0 con 0x7f1cec0c57e0 2023-11-24T08:17:30.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cea59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cec15b440 0x7f1cec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cea59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cec15b440 0x7f1cec0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36858/0 (socket says 172.21.15.107:36858) 2023-11-24T08:17:30.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.818+0000 7f1cea59c640 1 -- 172.21.15.107:0/1714988305 learned_addr learned my addr 172.21.15.107:0/1714988305 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1ce9d9b640 1 --2- 172.21.15.107:0/1714988305 >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 0x7f1cec0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cea59c640 1 -- 172.21.15.107:0/1714988305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 msgr2=0x7f1cec0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cead9d640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cea59c640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec0c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cea59c640 1 -- 172.21.15.107:0/1714988305 >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 msgr2=0x7f1cec0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cea59c640 1 --2- 172.21.15.107:0/1714988305 >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 0x7f1cec0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cea59c640 1 -- 172.21.15.107:0/1714988305 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0abf30 con 0x7f1cec15b440 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1ce9d9b640 1 --2- 172.21.15.107:0/1714988305 >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 0x7f1cec0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.819+0000 7f1cead9d640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.820+0000 7f1cea59c640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec15b440 0x7f1cec0cb9e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06e890 tx=0x7f1cdc02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdab461c186a994 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.820+0000 7f1ce959a640 1 -- 172.21.15.107:0/1714988305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0317f0 con 0x7f1cec15b440 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.820+0000 7f1ce959a640 1 -- 172.21.15.107:0/1714988305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc031950 con 0x7f1cec15b440 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.820+0000 7f1ce959a640 1 -- 172.21.15.107:0/1714988305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc031ce0 con 0x7f1cec15b440 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 -- 172.21.15.107:0/1714988305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec15b440 msgr2=0x7f1cec0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec15b440 0x7f1cec0cb9e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06e890 tx=0x7f1cdc02b890 comp rx=0 tx=0).stop 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 -- 172.21.15.107:0/1714988305 shutdown_connections 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/1714988305 >> v2:172.21.15.120:3300/0 conn(0x7f1cec0cbf20 0x7f1cec0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/1714988305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec15b440 0x7f1cec0cb9e0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 -- 172.21.15.107:0/1714988305 >> 172.21.15.107:0/1714988305 conn(0x7f1cec0b0c50 msgr2=0x7f1cec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:30.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 -- 172.21.15.107:0/1714988305 shutdown_connections 2023-11-24T08:17:30.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.821+0000 7f1cf0f90640 1 -- 172.21.15.107:0/1714988305 wait complete. 2023-11-24T08:17:30.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.823+0000 7f1cf0f90640 1 Processor -- start 2023-11-24T08:17:30.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.823+0000 7f1cf0f90640 1 -- start start 2023-11-24T08:17:30.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cec15b440 0x7f1cec1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0c76c0 con 0x7f1cec0cbf20 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15dd40 con 0x7f1cec15b440 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cf0f90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15cdf0 con 0x7f1cec0c57e0 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1ce9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1ce9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36874/0 (socket says 172.21.15.107:36874) 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1ce9d9b640 1 -- 172.21.15.107:0/2682267786 learned_addr learned my addr 172.21.15.107:0/2682267786 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.824+0000 7f1cea59c640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1cead9d640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cec15b440 0x7f1cec1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 -- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 msgr2=0x7f1cec157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 -- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cec15b440 msgr2=0x7f1cec1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:30.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cec15b440 0x7f1cec1566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 -- 172.21.15.107:0/2682267786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc0094d0 con 0x7f1cec0cbf20 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1cea59c640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce9d9b640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400c7f0 tx=0x7f1cd400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4007a30 con 0x7f1cec0cbf20 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd4007b90 con 0x7f1cec0cbf20 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd400ce40 con 0x7f1cec0cbf20 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec15df90 con 0x7f1cec0cbf20 2023-11-24T08:17:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.825+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec15e550 con 0x7f1cec0cbf20 2023-11-24T08:17:30.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.827+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec0c7870 con 0x7f1cec0cbf20 2023-11-24T08:17:30.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.827+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f1cd40113f0 con 0x7f1cec0cbf20 2023-11-24T08:17:30.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.828+0000 7f1ce2ffd640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 0x7f1ccc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:30.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.828+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1cd4088600 con 0x7f1cec0cbf20 2023-11-24T08:17:30.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.829+0000 7f1cea59c640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 0x7f1ccc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:30.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.829+0000 7f1cea59c640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 0x7f1ccc06eaa0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc036000 tx=0x7f1cdc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:30.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:30.833+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cec0c7870 con 0x7f1cec0cbf20 2023-11-24T08:17:31.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.037+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f1cec0c7870 con 0x7f1cec0cbf20 2023-11-24T08:17:31.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.043+0000 7f1ce2ffd640 1 -- 172.21.15.107:0/2682267786 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v42) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1cec0c7870 con 0x7f1cec0cbf20 2023-11-24T08:17:31.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 msgr2=0x7f1ccc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 0x7f1ccc06eaa0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc036000 tx=0x7f1cdc03d040 comp rx=0 tx=0).stop 2023-11-24T08:17:31.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 msgr2=0x7f1cec159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400c7f0 tx=0x7f1cd400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 shutdown_connections 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ccc06c5e0 0x7f1ccc06eaa0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cec15b440 0x7f1cec1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cec0cbf20 0x7f1cec159e90 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 --2- 172.21.15.107:0/2682267786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1cec0c57e0 0x7f1cec157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 >> 172.21.15.107:0/2682267786 conn(0x7f1cec0b0c50 msgr2=0x7f1cec0cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 shutdown_connections 2023-11-24T08:17:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.046+0000 7f1cf0f90640 1 -- 172.21.15.107:0/2682267786 wait complete. 2023-11-24T08:17:31.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm' 2023-11-24T08:17:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.260+0000 7fe5abee7640 1 Processor -- start 2023-11-24T08:17:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.261+0000 7fe5abee7640 1 -- start start 2023-11-24T08:17:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5abee7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5a406dc00 0x7fe5a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5abee7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a406c630 con 0x7fe5a415d520 2023-11-24T08:17:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5abee7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a406c910 con 0x7fe5a4071850 2023-11-24T08:17:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5abee7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a406e5d0 con 0x7fe5a406dc00 2023-11-24T08:17:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5a9c83640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5a415d520 0x7fe5a40ce390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47612/0 (socket says 172.21.15.107:47612) 2023-11-24T08:17:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.262+0000 7fe5a9c83640 1 -- 172.21.15.107:0/797614276 learned_addr learned my addr 172.21.15.107:0/797614276 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669385568 0 0) 0x7fe5a406c630 con 0x7fe5a415d520 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594003520 con 0x7fe5a415d520 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a9482640 1 --2- 172.21.15.107:0/797614276 >> v2:172.21.15.120:3300/0 conn(0x7fe5a406dc00 0x7fe5a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200773553 0 0) 0x7fe594003520 con 0x7fe5a415d520 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 >> v2:172.21.15.120:3300/0 conn(0x7fe5a406dc00 msgr2=0x7fe5a406e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 --2- 172.21.15.107:0/797614276 >> v2:172.21.15.120:3300/0 conn(0x7fe5a406dc00 0x7fe5a406e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 >> v1:172.21.15.120:6789/0 conn(0x7fe5a4071850 legacy=0x7fe5a4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a406cbf0 con 0x7fe5a415d520 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe5980019f0 con 0x7fe5a415d520 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 >> v1:172.21.15.107:6789/0 conn(0x7fe5a415d520 legacy=0x7fe5a40ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.263+0000 7fe5a8c81640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 0x7fe594003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a8c81640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5940047c0 0x7fe594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a8c81640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940051d0 0x7fe59400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a406e5d0 con 0x7fe5940039c0 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59400c120 con 0x7fe5940047c0 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59400c2e0 con 0x7fe5940051d0 2023-11-24T08:17:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 0x7fe594003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9482640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5940047c0 0x7fe594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 -- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940051d0 msgr2=0x7fe59400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940051d0 0x7fe59400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 -- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5940047c0 msgr2=0x7fe594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5940047c0 0x7fe594004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5a9c83640 1 -- 172.21.15.107:0/797614276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594003520 con 0x7fe5940039c0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.264+0000 7fe5aa484640 1 -- 172.21.15.107:0/797614276 reap_dead start 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.265+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 0x7fe594003de0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a760 tx=0x7fe5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2dc0e042f02732 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.265+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598034630 con 0x7fe5940039c0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.265+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59802f070 con 0x7fe5940039c0 2023-11-24T08:17:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.265+0000 7fe5a8c81640 1 -- 172.21.15.107:0/797614276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598034a50 con 0x7fe5940039c0 2023-11-24T08:17:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 -- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 msgr2=0x7fe594003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 0x7fe594003de0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a760 tx=0x7fe5940057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 -- 172.21.15.107:0/797614276 shutdown_connections 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 --2- 172.21.15.107:0/797614276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5940039c0 0x7fe594003de0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 -- 172.21.15.107:0/797614276 >> 172.21.15.107:0/797614276 conn(0x7fe5a40b0c70 msgr2=0x7fe5a40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 -- 172.21.15.107:0/797614276 shutdown_connections 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.266+0000 7fe5abee7640 1 -- 172.21.15.107:0/797614276 wait complete. 2023-11-24T08:17:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.268+0000 7fe5abee7640 1 Processor -- start 2023-11-24T08:17:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.268+0000 7fe5abee7640 1 -- start start 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940039c0 0x7fe5a40b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4145310 0x7fe5a4155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40b3200 con 0x7fe5a40b20a0 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40b25f0 con 0x7fe5a4145310 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5abee7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40b28d0 con 0x7fe5940039c0 2023-11-24T08:17:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.269+0000 7fe5a9482640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36886/0 (socket says 172.21.15.107:36886) 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 -- 172.21.15.107:0/534149700 learned_addr learned my addr 172.21.15.107:0/534149700 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 -- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940039c0 msgr2=0x7fe5a40b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940039c0 0x7fe5a40b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 -- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4145310 msgr2=0x7fe5a4155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4145310 0x7fe5a4155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 -- 172.21.15.107:0/534149700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40b2bb0 con 0x7fe5a40b20a0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a9482640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe59000e7e0 tx=0x7fe59000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59000c6d0 con 0x7fe5a40b20a0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59000c830 con 0x7fe5a40b20a0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59000cba0 con 0x7fe5a40b20a0 2023-11-24T08:17:31.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.270+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a40b2e90 con 0x7fe5a40b20a0 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.271+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a4145790 con 0x7fe5a40b20a0 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.272+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a4067990 con 0x7fe5a40b20a0 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.273+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fe59000cd00 con 0x7fe5a40b20a0 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.273+0000 7fe5a27fc640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 0x7fe58806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.274+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe590088120 con 0x7fe5a40b20a0 2023-11-24T08:17:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.274+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 0x7fe58806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.275+0000 7fe5a9c83640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 0x7fe58806eaa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe598034000 tx=0x7fe598009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.278+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe590054cd0 con 0x7fe5a40b20a0 2023-11-24T08:17:31.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:31.536+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm"]} v 0) v1 -- 0x7fe5a40b3020 con 0x7fe5a40b20a0 2023-11-24T08:17:32.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.152+0000 7fe5a27fc640 1 -- 172.21.15.107:0/534149700 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_snapshot_rm"]}]=0 v640) v1 ==== 177+0+0 (secure 0 0 0) 0x7fe590021020 con 0x7fe5a40b20a0 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.155+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 msgr2=0x7fe58806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.155+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 0x7fe58806eaa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe598034000 tx=0x7fe598009030 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.155+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 msgr2=0x7fe5a4154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.155+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe59000e7e0 tx=0x7fe59000eca0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 shutdown_connections 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe58806c5e0 0x7fe58806eaa0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4145310 0x7fe5a4155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5a40b20a0 0x7fe5a4154520 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 --2- 172.21.15.107:0/534149700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5940039c0 0x7fe5a40b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 >> 172.21.15.107:0/534149700 conn(0x7fe5a40b0c70 msgr2=0x7fe5a40accb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 shutdown_connections 2023-11-24T08:17:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.156+0000 7fe5abee7640 1 -- 172.21.15.107:0/534149700 wait complete. 2023-11-24T08:17:32.178 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_snapshot_rm (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:17:32.178 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_source (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:17:32.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:32.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.387+0000 7fc3e4168640 1 Processor -- start 2023-11-24T08:17:32.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.388+0000 7fc3e4168640 1 -- start start 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e4168640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc3dc15c7f0 0x7fc3dc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e4168640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3dc0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e4168640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc0cece0 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e4168640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc156010 con 0x7fc3dc15c7f0 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e4168640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc06ce30 con 0x7fc3dc0cf450 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e2705640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc3dc0cdfb0 0x7fc3dc0ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47622/0 (socket says 172.21.15.107:47622) 2023-11-24T08:17:32.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.389+0000 7fc3e2705640 1 -- 172.21.15.107:0/2500098995 learned_addr learned my addr 172.21.15.107:0/2500098995 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3dc0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e1f04640 1 --2- 172.21.15.107:0/2500098995 >> v2:172.21.15.120:3300/0 conn(0x7fc3dc15c7f0 0x7fc3dc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067657137 0 0) 0x7fc3dc0cece0 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8003520 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850916422 0 0) 0x7fc3c8003520 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 msgr2=0x7fc3dc0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3dc0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 >> v2:172.21.15.120:3300/0 conn(0x7fc3dc15c7f0 msgr2=0x7fc3dc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 --2- 172.21.15.107:0/2500098995 >> v2:172.21.15.120:3300/0 conn(0x7fc3dc15c7f0 0x7fc3dc06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc06d040 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc3d4001670 con 0x7fc3dc0cdfb0 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 >> v1:172.21.15.107:6789/0 conn(0x7fc3dc0cdfb0 legacy=0x7fc3dc0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3c8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8004d00 0x7fc3c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc156010 con 0x7fc3c8003ef0 2023-11-24T08:17:32.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc06ce30 con 0x7fc3c8004d00 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.390+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c800b980 con 0x7fc3dc0cf450 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1f04640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3c8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e2705640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8004d00 0x7fc3c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 -- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 msgr2=0x7fc3c8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0cf450 0x7fc3c8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 -- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8004d00 msgr2=0x7fc3c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8004d00 0x7fc3c8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.391+0000 7fc3e1703640 1 -- 172.21.15.107:0/2500098995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8003520 con 0x7fc3c8003ef0 2023-11-24T08:17:32.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.392+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3c80047c0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009ac0 tx=0x7fc3c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b28f81390ca701e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.392+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc02c6b0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.392+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc02c810 con 0x7fc3c8003ef0 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.392+0000 7fc3e0f02640 1 -- 172.21.15.107:0/2500098995 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc02cba0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 -- 172.21.15.107:0/2500098995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 msgr2=0x7fc3c80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2500098995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3c80047c0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009ac0 tx=0x7fc3c80057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e2705640 1 -- 172.21.15.107:0/2500098995 reap_dead start 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 -- 172.21.15.107:0/2500098995 shutdown_connections 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 -- 172.21.15.107:0/2500098995 >> 172.21.15.107:0/2500098995 conn(0x7fc3dc06a920 msgr2=0x7fc3dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 -- 172.21.15.107:0/2500098995 shutdown_connections 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.393+0000 7fc3e4168640 1 -- 172.21.15.107:0/2500098995 wait complete. 2023-11-24T08:17:32.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.395+0000 7fc3e4168640 1 Processor -- start 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.395+0000 7fc3e4168640 1 -- start start 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0b3130 0x7fc3dc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3dc0ad540 0x7fc3dc0ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b3ad0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc156150 con 0x7fc3dc0ad540 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e4168640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc156430 con 0x7fc3dc0b3130 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e1f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e1f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36896/0 (socket says 172.21.15.107:36896) 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.396+0000 7fc3e1f04640 1 -- 172.21.15.107:0/2348745834 learned_addr learned my addr 172.21.15.107:0/2348745834 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:32.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e2705640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3dc0ad540 0x7fc3dc0ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 -- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0b3130 msgr2=0x7fc3dc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0b3130 0x7fc3dc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 -- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3dc0ad540 msgr2=0x7fc3dc0ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3dc0ad540 0x7fc3dc0ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 -- 172.21.15.107:0/2348745834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc156710 con 0x7fc3c8003ef0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e1f04640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800f3d0 tx=0x7fc3d800f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8016620 con 0x7fc3c8003ef0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d80167a0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8016b50 con 0x7fc3c8003ef0 2023-11-24T08:17:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc1569f0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.397+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc0adf00 con 0x7fc3c8003ef0 2023-11-24T08:17:32.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.398+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc06e060 con 0x7fc3c8003ef0 2023-11-24T08:17:32.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.399+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fc3d8016cb0 con 0x7fc3c8003ef0 2023-11-24T08:17:32.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.400+0000 7fc3d27fc640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 0x7fc3c006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.400+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3d8089420 con 0x7fc3c8003ef0 2023-11-24T08:17:32.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.400+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 0x7fc3c006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.401+0000 7fc3e1703640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 0x7fc3c006eaa0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc036000 tx=0x7fc3cc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.404+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3d8055f30 con 0x7fc3c8003ef0 2023-11-24T08:17:32.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.595+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3dc067470 con 0x7fc3c8003ef0 2023-11-24T08:17:32.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.597+0000 7fc3d27fc640 1 -- 172.21.15.107:0/2348745834 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+12998 (secure 0 0 0) 0x7fc3d8059550 con 0x7fc3c8003ef0 2023-11-24T08:17:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:32.599 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":154,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:30.125442+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6836/3938030686":"2023-11-25T08:17:27.108460+0000","172.21.15.107:6834/3938030686":"2023-11-25T08:17:27.108460+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 msgr2=0x7fc3c006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 0x7fc3c006eaa0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc036000 tx=0x7fc3cc02e000 comp rx=0 tx=0).stop 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 msgr2=0x7fc3dc14d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800f3d0 tx=0x7fc3d800f890 comp rx=0 tx=0).stop 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 shutdown_connections 2023-11-24T08:17:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc3c006c5e0 0x7fc3c006eaa0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3dc0ad540 0x7fc3dc0ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3dc0b3130 0x7fc3dc0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 --2- 172.21.15.107:0/2348745834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c8003ef0 0x7fc3dc14d5b0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.599+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 >> 172.21.15.107:0/2348745834 conn(0x7fc3dc06a920 msgr2=0x7fc3dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.600+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 shutdown_connections 2023-11-24T08:17:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.600+0000 7fc3e4168640 1 -- 172.21.15.107:0/2348745834 wait complete. 2023-11-24T08:17:32.620 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:17:32.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:17:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.826+0000 7f655627c640 1 Processor -- start 2023-11-24T08:17:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.827+0000 7f655627c640 1 -- start start 2023-11-24T08:17:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f655015b440 0x7f65500cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 0x7f65500c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 0x7f65500c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65500cc3d0 con 0x7f655015b440 2023-11-24T08:17:32.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655015b820 con 0x7f65500cbf20 2023-11-24T08:17:32.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.828+0000 7f655627c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655015eea0 con 0x7f65500c57e0 2023-11-24T08:17:32.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f655015b440 0x7f65500cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f655015b440 0x7f65500cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36900/0 (socket says 172.21.15.107:36900) 2023-11-24T08:17:32.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 -- 172.21.15.107:0/576670990 learned_addr learned my addr 172.21.15.107:0/576670990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654f7fe640 1 --2- 172.21.15.107:0/576670990 >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 0x7f65500c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f6554819640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 0x7f65500c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 -- 172.21.15.107:0/576670990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 msgr2=0x7f65500c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 0x7f65500c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 -- 172.21.15.107:0/576670990 >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 msgr2=0x7f65500c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 --2- 172.21.15.107:0/576670990 >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 0x7f65500c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654ffff640 1 -- 172.21.15.107:0/576670990 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500abf30 con 0x7f655015b440 2023-11-24T08:17:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f6554819640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 0x7f65500c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.829+0000 7f654f7fe640 1 --2- 172.21.15.107:0/576670990 >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 0x7f65500c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.830+0000 7f654ffff640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f655015b440 0x7f65500cb9e0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f655006e890 tx=0x7f654002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dfcef3db9abeac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.830+0000 7f654effd640 1 -- 172.21.15.107:0/576670990 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654002c9e0 con 0x7f655015b440 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f654effd640 1 -- 172.21.15.107:0/576670990 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654002cb40 con 0x7f655015b440 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f654effd640 1 -- 172.21.15.107:0/576670990 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400343f0 con 0x7f655015b440 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 -- 172.21.15.107:0/576670990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f655015b440 msgr2=0x7f65500cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f655015b440 0x7f65500cb9e0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f655006e890 tx=0x7f654002be40 comp rx=0 tx=0).stop 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 -- 172.21.15.107:0/576670990 shutdown_connections 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500c57e0 0x7f65500c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 --2- 172.21.15.107:0/576670990 >> v2:172.21.15.120:3300/0 conn(0x7f65500cbf20 0x7f65500c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 --2- 172.21.15.107:0/576670990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f655015b440 0x7f65500cb9e0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.831+0000 7f655627c640 1 -- 172.21.15.107:0/576670990 >> 172.21.15.107:0/576670990 conn(0x7f65500b0c50 msgr2=0x7f65500b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.832+0000 7f655627c640 1 -- 172.21.15.107:0/576670990 shutdown_connections 2023-11-24T08:17:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.832+0000 7f655627c640 1 -- 172.21.15.107:0/576670990 wait complete. 2023-11-24T08:17:32.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.834+0000 7f655627c640 1 Processor -- start 2023-11-24T08:17:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.834+0000 7f655627c640 1 -- start start 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500cbf20 0x7f6550149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655015b440 0x7f65500999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500c6170 con 0x7f65500c57e0 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500b4380 con 0x7f655015b440 2023-11-24T08:17:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f655627c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65500c6410 con 0x7f65500cbf20 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.835+0000 7f654ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36908/0 (socket says 172.21.15.107:36908) 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 -- 172.21.15.107:0/2956246610 learned_addr learned my addr 172.21.15.107:0/2956246610 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 -- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500cbf20 msgr2=0x7f6550149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500cbf20 0x7f6550149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 -- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655015b440 msgr2=0x7f65500999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655015b440 0x7f65500999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 -- 172.21.15.107:0/2956246610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009b30 con 0x7f65500c57e0 2023-11-24T08:17:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.836+0000 7f654ffff640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f6540000f80 tx=0x7f6540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.837+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654003e450 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.837+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654003e5f0 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.837+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65500c66f0 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.837+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654003e9a0 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.837+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6550156f70 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.838+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6550157220 con 0x7f65500c57e0 2023-11-24T08:17:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.839+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f654003eb00 con 0x7f65500c57e0 2023-11-24T08:17:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.840+0000 7f654cff9640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 0x7f653006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.840+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f65400ab200 con 0x7f65500c57e0 2023-11-24T08:17:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.841+0000 7f654f7fe640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 0x7f653006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:32.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.841+0000 7f654f7fe640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 0x7f653006eaf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6538006fd0 tx=0x7f6538008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:32.844+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6540078060 con 0x7f65500c57e0 2023-11-24T08:17:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.034+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65500c6910 con 0x7f65500c57e0 2023-11-24T08:17:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.035+0000 7f654cff9640 1 -- 172.21.15.107:0/2956246610 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) v1 ==== 93+0+10 (secure 0 0 0) 0x7f654003edb0 con 0x7f65500c57e0 2023-11-24T08:17:33.037 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 msgr2=0x7f653006eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 0x7f653006eaf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6538006fd0 tx=0x7f6538008040 comp rx=0 tx=0).stop 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 msgr2=0x7f65501499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f6540000f80 tx=0x7f6540004a40 comp rx=0 tx=0).stop 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 shutdown_connections 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f653006c630 0x7f653006eaf0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655015b440 0x7f65500999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65500cbf20 0x7f6550149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 --2- 172.21.15.107:0/2956246610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65500c57e0 0x7f65501499d0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 >> 172.21.15.107:0/2956246610 conn(0x7f65500b0c50 msgr2=0x7f65500c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 shutdown_connections 2023-11-24T08:17:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.038+0000 7f655627c640 1 -- 172.21.15.107:0/2956246610 wait complete. 2023-11-24T08:17:33.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source' 2023-11-24T08:17:33.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.263+0000 7fc0b610f640 1 Processor -- start 2023-11-24T08:17:33.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.263+0000 7fc0b610f640 1 -- start start 2023-11-24T08:17:33.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.264+0000 7fc0b610f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b015a4f0 0x7fc0b00b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.264+0000 7fc0b610f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b00cdfb0 0x7fc0b00ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.264+0000 7fc0b610f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b00ce950 con 0x7fc0b00b20c0 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.264+0000 7fc0b610f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b00b3a40 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.264+0000 7fc0b610f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00b2e30 con 0x7fc0b015a4f0 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b00cdfb0 0x7fc0b00ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b00cdfb0 0x7fc0b00ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36910/0 (socket says 172.21.15.107:36910) 2023-11-24T08:17:33.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 -- 172.21.15.107:0/4198120279 learned_addr learned my addr 172.21.15.107:0/4198120279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:33.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0af7fe640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b015a4f0 0x7fc0b00b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 -- 172.21.15.107:0/4198120279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b015a4f0 msgr2=0x7fc0b00b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b015a4f0 0x7fc0b00b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 -- 172.21.15.107:0/4198120279 >> v1:172.21.15.107:6789/0 conn(0x7fc0b00b20c0 legacy=0x7fc0b00b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0a7fff640 1 -- 172.21.15.107:0/4198120279 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00b3110 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.265+0000 7fc0aeffd640 1 -- 172.21.15.107:0/4198120279 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259342195 0 0) 0x7fc0b00ce950 con 0x7fc0b00b20c0 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.266+0000 7fc0a7fff640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b00ce410 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fc09800bb10 tx=0x7fc09802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c577fcd97b28fb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0aeffd640 1 -- 172.21.15.107:0/4198120279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098036440 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0aeffd640 1 -- 172.21.15.107:0/4198120279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0980365a0 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0aeffd640 1 -- 172.21.15.107:0/4198120279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098036930 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 -- 172.21.15.107:0/4198120279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b00cdfb0 msgr2=0x7fc0b00ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b00ce410 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fc09800bb10 tx=0x7fc09802e150 comp rx=0 tx=0).stop 2023-11-24T08:17:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 -- 172.21.15.107:0/4198120279 shutdown_connections 2023-11-24T08:17:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b00ce410 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 --2- 172.21.15.107:0/4198120279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b015a4f0 0x7fc0b00b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.267+0000 7fc0b610f640 1 -- 172.21.15.107:0/4198120279 >> 172.21.15.107:0/4198120279 conn(0x7fc0b00b0c70 msgr2=0x7fc0b00b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.268+0000 7fc0b610f640 1 -- 172.21.15.107:0/4198120279 shutdown_connections 2023-11-24T08:17:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.268+0000 7fc0b610f640 1 -- 172.21.15.107:0/4198120279 wait complete. 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.269+0000 7fc0b610f640 1 Processor -- start 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.269+0000 7fc0b610f640 1 -- start start 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b00b20c0 0x7fc0b015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b015be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b015c380 con 0x7fc0b015a4f0 2023-11-24T08:17:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b015d730 con 0x7fc0b00cdfb0 2023-11-24T08:17:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.270+0000 7fc0b610f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b015c970 con 0x7fc0b00b20c0 2023-11-24T08:17:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36914/0 (socket says 172.21.15.107:36914) 2023-11-24T08:17:33.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 -- 172.21.15.107:0/3732456567 learned_addr learned my addr 172.21.15.107:0/3732456567 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:33.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0af7fe640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b00b20c0 0x7fc0b015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0a7fff640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 -- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b00b20c0 msgr2=0x7fc0b015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b00b20c0 0x7fc0b015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 -- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 msgr2=0x7fc0b015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0affff640 1 -- 172.21.15.107:0/3732456567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09800b770 con 0x7fc0b015a4f0 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.271+0000 7fc0a7fff640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0affff640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c790 tx=0x7fc0a800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800cea0 con 0x7fc0b015a4f0 2023-11-24T08:17:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a8003da0 con 0x7fc0b015a4f0 2023-11-24T08:17:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a80052b0 con 0x7fc0b015a4f0 2023-11-24T08:17:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b015cc50 con 0x7fc0b015a4f0 2023-11-24T08:17:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.272+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b015d210 con 0x7fc0b015a4f0 2023-11-24T08:17:33.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.273+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b015d400 con 0x7fc0b015a4f0 2023-11-24T08:17:33.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.274+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fc0a8005410 con 0x7fc0b015a4f0 2023-11-24T08:17:33.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.275+0000 7fc0acff9640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 0x7fc09006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:33.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.275+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc0a80881a0 con 0x7fc0b015a4f0 2023-11-24T08:17:33.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.275+0000 7fc0af7fe640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 0x7fc09006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:33.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.276+0000 7fc0af7fe640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 0x7fc09006eaa0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0006fd0 tx=0x7fc0a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:33.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.280+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0a8054d50 con 0x7fc0b015a4f0 2023-11-24T08:17:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:33.530+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source"]} v 0) v1 -- 0x7fc0b004e8b0 con 0x7fc0b015a4f0 2023-11-24T08:17:34.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.165+0000 7fc0acff9640 1 -- 172.21.15.107:0/3732456567 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source"]}]=0 v642) v1 ==== 175+0+0 (secure 0 0 0) 0x7fc0a8058370 con 0x7fc0b015a4f0 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 msgr2=0x7fc09006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 0x7fc09006eaa0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0006fd0 tx=0x7fc0a0008040 comp rx=0 tx=0).stop 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 msgr2=0x7fc0b0156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c790 tx=0x7fc0a800cc50 comp rx=0 tx=0).stop 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 shutdown_connections 2023-11-24T08:17:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc09006c5e0 0x7fc09006eaa0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015a4f0 0x7fc0b0156c80 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b00cdfb0 0x7fc0b015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 --2- 172.21.15.107:0/3732456567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0b00b20c0 0x7fc0b015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 >> 172.21.15.107:0/3732456567 conn(0x7fc0b00b0c70 msgr2=0x7fc0b0158990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.169+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 shutdown_connections 2023-11-24T08:17:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.170+0000 7fc0b610f640 1 -- 172.21.15.107:0/3732456567 wait complete. 2023-11-24T08:17:34.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:17:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.407+0000 7ff3cc790640 1 Processor -- start 2023-11-24T08:17:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.407+0000 7ff3cc790640 1 -- start start 2023-11-24T08:17:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cc790640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3c415b440 0x7ff3c40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cc790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cc790640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c415eea0 con 0x7ff3c40cbf20 2023-11-24T08:17:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cc790640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c40c5e00 con 0x7ff3c415b440 2023-11-24T08:17:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cc790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40cc3d0 con 0x7ff3c40c59b0 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3cad2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.409+0000 7ff3ca52c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3c415b440 0x7ff3c40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:41062/0 (socket says 172.21.15.107:41062) 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 -- 172.21.15.107:0/2462268471 learned_addr learned my addr 172.21.15.107:0/2462268471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 -- 172.21.15.107:0/2462268471 >> v2:172.21.15.120:3300/0 conn(0x7ff3c415b440 msgr2=0x7ff3c40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 --2- 172.21.15.107:0/2462268471 >> v2:172.21.15.120:3300/0 conn(0x7ff3c415b440 0x7ff3c40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 -- 172.21.15.107:0/2462268471 >> v1:172.21.15.120:6789/0 conn(0x7ff3c40cbf20 legacy=0x7ff3c40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.410+0000 7ff3cad2d640 1 -- 172.21.15.107:0/2462268471 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40abf30 con 0x7ff3c40c59b0 2023-11-24T08:17:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.412+0000 7ff3cad2d640 1 --2- 172.21.15.107:0/2462268471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009e50 tx=0x7ff3bc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e9eb18a8ebbe16e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.412+0000 7ff3c952a640 1 -- 172.21.15.107:0/2462268471 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc02ca70 con 0x7ff3c40c59b0 2023-11-24T08:17:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.412+0000 7ff3c952a640 1 -- 172.21.15.107:0/2462268471 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc02cbd0 con 0x7ff3c40c59b0 2023-11-24T08:17:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.412+0000 7ff3c952a640 1 -- 172.21.15.107:0/2462268471 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc0343f0 con 0x7ff3c40c59b0 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 -- 172.21.15.107:0/2462268471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 msgr2=0x7ff3c406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 --2- 172.21.15.107:0/2462268471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009e50 tx=0x7ff3bc02bed0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 -- 172.21.15.107:0/2462268471 shutdown_connections 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 --2- 172.21.15.107:0/2462268471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40c59b0 0x7ff3c406c5f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 --2- 172.21.15.107:0/2462268471 >> v2:172.21.15.120:3300/0 conn(0x7ff3c415b440 0x7ff3c40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 -- 172.21.15.107:0/2462268471 >> 172.21.15.107:0/2462268471 conn(0x7ff3c40b0c50 msgr2=0x7ff3c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:34.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 -- 172.21.15.107:0/2462268471 shutdown_connections 2023-11-24T08:17:34.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.413+0000 7ff3cc790640 1 -- 172.21.15.107:0/2462268471 wait complete. 2023-11-24T08:17:34.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.415+0000 7ff3cc790640 1 Processor -- start 2023-11-24T08:17:34.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.416+0000 7ff3cc790640 1 -- start start 2023-11-24T08:17:34.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 0x7ff3c414bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cbf20 0x7ff3c409b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c409b5b0 con 0x7ff3c415b440 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c409a690 con 0x7ff3c40c59b0 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cc790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c409a930 con 0x7ff3c40cbf20 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cad2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cad2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36916/0 (socket says 172.21.15.107:36916) 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3cad2d640 1 -- 172.21.15.107:0/105158603 learned_addr learned my addr 172.21.15.107:0/105158603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3c9d2b640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cbf20 0x7ff3c409b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.417+0000 7ff3ca52c640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 0x7ff3c414bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 -- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cbf20 msgr2=0x7ff3c409b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cbf20 0x7ff3c409b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 -- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 msgr2=0x7ff3c414bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 0x7ff3c414bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 -- 172.21.15.107:0/105158603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc009b30 con 0x7ff3c415b440 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3ca52c640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 0x7ff3c414bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cad2d640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009e50 tx=0x7ff3bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc009510 con 0x7ff3c415b440 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc03e450 con 0x7ff3c415b440 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c409ac10 con 0x7ff3c415b440 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.418+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc03e800 con 0x7ff3c415b440 2023-11-24T08:17:34.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.419+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c409b1d0 con 0x7ff3c415b440 2023-11-24T08:17:34.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.420+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c409b6f0 con 0x7ff3c415b440 2023-11-24T08:17:34.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.421+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7ff3bc03e960 con 0x7ff3c415b440 2023-11-24T08:17:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.422+0000 7ff3baffd640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 0x7ff3a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.422+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff3bc0ab100 con 0x7ff3c415b440 2023-11-24T08:17:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.422+0000 7ff3ca52c640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 0x7ff3a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.423+0000 7ff3ca52c640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 0x7ff3a806eaa0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009af0 tx=0x7ff3c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:34.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.428+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3c409b6f0 con 0x7ff3c415b440 2023-11-24T08:17:34.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.619+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff3c409b6f0 con 0x7ff3c415b440 2023-11-24T08:17:34.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.626+0000 7ff3baffd640 1 -- 172.21.15.107:0/105158603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v43)=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff3c409b6f0 con 0x7ff3c415b440 2023-11-24T08:17:34.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 msgr2=0x7ff3a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 0x7ff3a806eaa0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009af0 tx=0x7ff3c0008040 comp rx=0 tx=0).stop 2023-11-24T08:17:34.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 msgr2=0x7ff3c4158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:34.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009e50 tx=0x7ff3bc004a40 comp rx=0 tx=0).stop 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 shutdown_connections 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3a806c5e0 0x7ff3a806eaa0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3c415b440 0x7ff3c4158a30 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c40cbf20 0x7ff3c409b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 --2- 172.21.15.107:0/105158603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c40c59b0 0x7ff3c414bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 >> 172.21.15.107:0/105158603 conn(0x7ff3c40b0c50 msgr2=0x7ff3c40c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 shutdown_connections 2023-11-24T08:17:34.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:34.629+0000 7ff3cc790640 1 -- 172.21.15.107:0/105158603 wait complete. 2023-11-24T08:17:34.650 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:17:34.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:17:34.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:17:34.869 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:17:34.869 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:34.869 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:34.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:17:34.925 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:34.925 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:34.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:17:34.984 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:34.984 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:35.045 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:17:35.046 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:35.046 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:35.104 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:17:35.105 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:17:35.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:35.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.325+0000 7f99cce36640 1 Processor -- start 2023-11-24T08:17:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.325+0000 7f99cce36640 1 -- start start 2023-11-24T08:17:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99cce36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99c815b080 0x7f99c815b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99cce36640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c815e3f0 con 0x7f99c815df80 2023-11-24T08:17:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99cce36640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c815ab80 con 0x7f99c80b2250 2023-11-24T08:17:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99cce36640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c815a990 con 0x7f99c815b080 2023-11-24T08:17:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99c6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f99c815b080 0x7f99c815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99bffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f99c815df80 0x7f99c80ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47630/0 (socket says 172.21.15.107:47630) 2023-11-24T08:17:35.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.327+0000 7f99bffff640 1 -- 172.21.15.107:0/1266295700 learned_addr learned my addr 172.21.15.107:0/1266295700 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:35.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.328+0000 7f99c6d9d640 1 -- 172.21.15.107:0/1266295700 >> v1:172.21.15.120:6789/0 conn(0x7f99c80b2250 legacy=0x7f99c80b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:35.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.328+0000 7f99c6d9d640 1 -- 172.21.15.107:0/1266295700 >> v1:172.21.15.107:6789/0 conn(0x7f99c815df80 legacy=0x7f99c80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.328+0000 7f99c6d9d640 1 -- 172.21.15.107:0/1266295700 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c80b2d40 con 0x7f99c815b080 2023-11-24T08:17:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.328+0000 7f99c5d9b640 1 -- 172.21.15.107:0/1266295700 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194242555 0 0) 0x7f99c815e3f0 con 0x7f99c815df80 2023-11-24T08:17:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.329+0000 7f99c6d9d640 1 --2- 172.21.15.107:0/1266295700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c815b490 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009d90 tx=0x7f99c002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e641069c55317b69 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.329+0000 7f99c5d9b640 1 -- 172.21.15.107:0/1266295700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c002cc60 con 0x7f99c815b080 2023-11-24T08:17:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.329+0000 7f99c5d9b640 1 -- 172.21.15.107:0/1266295700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c002cdc0 con 0x7f99c815b080 2023-11-24T08:17:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.329+0000 7f99c5d9b640 1 -- 172.21.15.107:0/1266295700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0034660 con 0x7f99c815b080 2023-11-24T08:17:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 -- 172.21.15.107:0/1266295700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 msgr2=0x7f99c815b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 --2- 172.21.15.107:0/1266295700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c815b490 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009d90 tx=0x7f99c002beb0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 -- 172.21.15.107:0/1266295700 shutdown_connections 2023-11-24T08:17:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 --2- 172.21.15.107:0/1266295700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c815b490 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 -- 172.21.15.107:0/1266295700 >> 172.21.15.107:0/1266295700 conn(0x7f99c806a920 msgr2=0x7f99c806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 -- 172.21.15.107:0/1266295700 shutdown_connections 2023-11-24T08:17:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.330+0000 7f99cce36640 1 -- 172.21.15.107:0/1266295700 wait complete. 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.332+0000 7f99cce36640 1 Processor -- start 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.332+0000 7f99cce36640 1 -- start start 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 0x7f99c81560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 0x7f99c81577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8156740 con 0x7f99c815b080 2023-11-24T08:17:35.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8156a20 con 0x7f99c80b2250 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99cce36640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8156d00 con 0x7f99c815df80 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36930/0 (socket says 172.21.15.107:36930) 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.333+0000 7f99bffff640 1 -- 172.21.15.107:0/2417785634 learned_addr learned my addr 172.21.15.107:0/2417785634 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99c659c640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 0x7f99c81560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99c6d9d640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 0x7f99c81577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 -- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 msgr2=0x7f99c81577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 0x7f99c81577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 -- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 msgr2=0x7f99c81560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 0x7f99c81560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 -- 172.21.15.107:0/2417785634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c0009a70 con 0x7f99c815b080 2023-11-24T08:17:35.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99c659c640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 0x7f99c81560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99c6d9d640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 0x7f99c81577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99bffff640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c430 tx=0x7f99b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b00074e0 con 0x7f99c815b080 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b0007660 con 0x7f99c815b080 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b0004a20 con 0x7f99c815b080 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.334+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c8156fe0 con 0x7f99c815b080 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.335+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c8157d20 con 0x7f99c815b080 2023-11-24T08:17:35.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.336+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c8157500 con 0x7f99c815b080 2023-11-24T08:17:35.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.337+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f99b0004b80 con 0x7f99c815b080 2023-11-24T08:17:35.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.338+0000 7f99beffd640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 0x7f99a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.338+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f99b0088230 con 0x7f99c815b080 2023-11-24T08:17:35.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.338+0000 7f99c659c640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 0x7f99a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.339+0000 7f99c659c640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 0x7f99a806eaa0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f99b8003eb0 tx=0x7f99b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.342+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99c8157500 con 0x7f99c815b080 2023-11-24T08:17:35.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.571+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99c8157500 con 0x7f99c815b080 2023-11-24T08:17:35.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.572+0000 7f99beffd640 1 -- 172.21.15.107:0/2417785634 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f99c8157500 con 0x7f99c815b080 2023-11-24T08:17:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:35.575 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1913781587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":1913781587},{"type":"v1","addr":"172.21.15.107:6837","nonce":1913781587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2438561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2438561485},{"type":"v1","addr":"172.21.15.120:6833","nonce":2438561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":6592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2856253134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2856253134},{"type":"v1","addr":"172.21.15.120:6835","nonce":2856253134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/1034827820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1034827820},{"type":"v1","addr":"172.21.15.107:6836","nonce":1034827820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[]} 2023-11-24T08:17:35.575 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-11-24T08:17:35.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 msgr2=0x7f99a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 0x7f99a806eaa0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f99b8003eb0 tx=0x7f99b8009040 comp rx=0 tx=0).stop 2023-11-24T08:17:35.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 msgr2=0x7f99c80aed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c430 tx=0x7f99b000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 shutdown_connections 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f99a806c5e0 0x7f99a806eaa0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99c815df80 0x7f99c81577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c815b080 0x7f99c80aed10 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 --2- 172.21.15.107:0/2417785634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c80b2250 0x7f99c81560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 >> 172.21.15.107:0/2417785634 conn(0x7f99c806a920 msgr2=0x7f99c8154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 shutdown_connections 2023-11-24T08:17:35.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.575+0000 7f99cce36640 1 -- 172.21.15.107:0/2417785634 wait complete. 2023-11-24T08:17:35.594 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:17:35.594 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:17:35.595 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:17:35.595 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:17:35.595 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:17:35.595 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:17:35.596 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:17:35.596 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:17:35.596 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:17:35.596 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:17:35.597 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:17:35.597 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:17:35.597 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:17:35.598 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:17:35.598 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:17:35.598 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:17:35.610 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:17:35.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:17:35.611 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:17:35.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:17:35.617 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:17:35.617 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:17:35.619 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:17:35.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:17:35.661 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:17:35.662 DEBUG:teuthology.parallel:result is None 2023-11-24T08:17:35.668 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:17:35.668 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:17:35.669 DEBUG:teuthology.parallel:result is None 2023-11-24T08:17:35.669 DEBUG:teuthology.parallel:result is None 2023-11-24T08:17:35.676 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:17:35.676 DEBUG:teuthology.parallel:result is None 2023-11-24T08:17:35.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:17:35.822 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:17:35.824 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:17:35.832 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:17:35.833 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:17:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.865+0000 7fc28359e640 1 Processor -- start 2023-11-24T08:17:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.865+0000 7fc28359e640 1 -- start start 2023-11-24T08:17:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2840b63d0 0x7fc2840d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2840ce270 0x7fc2840d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2840cea30 con 0x7fc28415d490 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2840d0450 con 0x7fc2840b63d0 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840d0bf0 con 0x7fc2840ce270 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.866+0000 7fc28259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc28415d490 0x7fc2840cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47652/0 (socket says 172.21.15.107:47652) 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc28259c640 1 -- 172.21.15.107:0/1077645103 learned_addr learned my addr 172.21.15.107:0/1077645103 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc282d9d640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2840ce270 0x7fc2840d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1077645103 >> v2:172.21.15.120:3300/0 conn(0x7fc2840b63d0 0x7fc2840d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc28159a640 1 -- 172.21.15.107:0/1077645103 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510960236 0 0) 0x7fc2840cea30 con 0x7fc28415d490 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc28159a640 1 -- 172.21.15.107:0/1077645103 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c003520 con 0x7fc28415d490 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc281d9b640 1 -- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2840ce270 msgr2=0x7fc2840d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2840ce270 0x7fc2840d3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc281d9b640 1 -- 172.21.15.107:0/1077645103 >> v1:172.21.15.107:6789/0 conn(0x7fc28415d490 legacy=0x7fc2840cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.867+0000 7fc281d9b640 1 -- 172.21.15.107:0/1077645103 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840d2260 con 0x7fc2840b63d0 2023-11-24T08:17:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.869+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2840b63d0 0x7fc2840d3230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2780097a0 tx=0x7fc27802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8a094a5ea7ea425 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.869+0000 7fc28159a640 1 -- 172.21.15.107:0/1077645103 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27802d040 con 0x7fc2840b63d0 2023-11-24T08:17:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.869+0000 7fc28159a640 1 -- 172.21.15.107:0/1077645103 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc278031810 con 0x7fc2840b63d0 2023-11-24T08:17:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.869+0000 7fc28159a640 1 -- 172.21.15.107:0/1077645103 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278031ba0 con 0x7fc2840b63d0 2023-11-24T08:17:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 -- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2840b63d0 msgr2=0x7fc2840d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2840b63d0 0x7fc2840d3230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2780097a0 tx=0x7fc27802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 -- 172.21.15.107:0/1077645103 shutdown_connections 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2840ce270 0x7fc2840d3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 --2- 172.21.15.107:0/1077645103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2840b63d0 0x7fc2840d3230 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 -- 172.21.15.107:0/1077645103 >> 172.21.15.107:0/1077645103 conn(0x7fc284061340 msgr2=0x7fc284061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 -- 172.21.15.107:0/1077645103 shutdown_connections 2023-11-24T08:17:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.870+0000 7fc28359e640 1 -- 172.21.15.107:0/1077645103 wait complete. 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.872+0000 7fc28359e640 1 Processor -- start 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.872+0000 7fc28359e640 1 -- start start 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28415d490 0x7fc2840c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2841f3d20 0x7fc2841f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840cb650 con 0x7fc2840ce270 2023-11-24T08:17:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840caaa0 con 0x7fc2841f3d20 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.873+0000 7fc28359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840cad20 con 0x7fc28415d490 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36980/0 (socket says 172.21.15.107:36980) 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 -- 172.21.15.107:0/1472616004 learned_addr learned my addr 172.21.15.107:0/1472616004 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28415d490 0x7fc2840c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc282d9d640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2841f3d20 0x7fc2841f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 -- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28415d490 msgr2=0x7fc2840c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28415d490 0x7fc2840c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 -- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2841f3d20 msgr2=0x7fc2841f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2841f3d20 0x7fc2841f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.874+0000 7fc28259c640 1 -- 172.21.15.107:0/1472616004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc278009400 con 0x7fc2840ce270 2023-11-24T08:17:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc28259c640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fc27400c440 tx=0x7fc27400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274005ce0 con 0x7fc2840ce270 2023-11-24T08:17:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274005e40 con 0x7fc2840ce270 2023-11-24T08:17:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2840cb000 con 0x7fc2840ce270 2023-11-24T08:17:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2740072b0 con 0x7fc2840ce270 2023-11-24T08:17:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.875+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2840c7560 con 0x7fc2840ce270 2023-11-24T08:17:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.876+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2840c7750 con 0x7fc2840ce270 2023-11-24T08:17:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.877+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fc274007410 con 0x7fc2840ce270 2023-11-24T08:17:35.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.879+0000 7fc272ffd640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 0x7fc26406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.879+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc274087860 con 0x7fc2840ce270 2023-11-24T08:17:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.879+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 0x7fc26406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:35.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.880+0000 7fc281d9b640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 0x7fc26406eb20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc278036000 tx=0x7fc27803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:35.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:35.883+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc27408de60 con 0x7fc2840ce270 2023-11-24T08:17:36.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.084+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc28404e840 con 0x7fc2840ce270 2023-11-24T08:17:36.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.192+0000 7fc272ffd640 1 -- 172.21.15.107:0/1472616004 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v156) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc274017020 con 0x7fc2840ce270 2023-11-24T08:17:36.194 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:17:36.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 msgr2=0x7fc26406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 0x7fc26406eb20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc278036000 tx=0x7fc27803c040 comp rx=0 tx=0).stop 2023-11-24T08:17:36.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 msgr2=0x7fc2840c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fc27400c440 tx=0x7fc27400c900 comp rx=0 tx=0).stop 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 shutdown_connections 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc26406c660 0x7fc26406eb20 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2841f3d20 0x7fc2841f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28415d490 0x7fc2840c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 --2- 172.21.15.107:0/1472616004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840ce270 0x7fc2840c7020 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 >> 172.21.15.107:0/1472616004 conn(0x7fc284061340 msgr2=0x7fc2840cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.196+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 shutdown_connections 2023-11-24T08:17:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.197+0000 7fc28359e640 1 -- 172.21.15.107:0/1472616004 wait complete. 2023-11-24T08:17:36.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:17:36.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.418+0000 7f020d056640 1 Processor -- start 2023-11-24T08:17:36.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.418+0000 7f020d056640 1 -- start start 2023-11-24T08:17:36.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f020815d520 0x7f02080ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02080ce8d0 0x7f020806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 0x7f02080abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020806e5b0 con 0x7f02080ce8d0 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020806c630 con 0x7f020815d520 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020d056640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020806c910 con 0x7f020806e140 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02080ce8d0 0x7f020806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02080ce8d0 0x7f020806dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36984/0 (socket says 172.21.15.107:36984) 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020659c640 1 -- 172.21.15.107:0/4021196726 learned_addr learned my addr 172.21.15.107:0/4021196726 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:36.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f0206d9d640 1 --2- 172.21.15.107:0/4021196726 >> v2:172.21.15.120:3300/0 conn(0x7f020815d520 0x7f02080ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.420+0000 7f020759e640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 0x7f02080abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020659c640 1 -- 172.21.15.107:0/4021196726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 msgr2=0x7f02080abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:36.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020659c640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 0x7f02080abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020659c640 1 -- 172.21.15.107:0/4021196726 >> v2:172.21.15.120:3300/0 conn(0x7f020815d520 msgr2=0x7f02080ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:36.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020659c640 1 --2- 172.21.15.107:0/4021196726 >> v2:172.21.15.120:3300/0 conn(0x7f020815d520 0x7f02080ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020659c640 1 -- 172.21.15.107:0/4021196726 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020806cbf0 con 0x7f02080ce8d0 2023-11-24T08:17:36.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.421+0000 7f020759e640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 0x7f02080abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:36.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.422+0000 7f020659c640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02080ce8d0 0x7f020806dc00 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f020806c020 tx=0x7f01fc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899dc05481072ea1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.422+0000 7f0205d9b640 1 -- 172.21.15.107:0/4021196726 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc031860 con 0x7f02080ce8d0 2023-11-24T08:17:36.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.422+0000 7f0205d9b640 1 -- 172.21.15.107:0/4021196726 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc0319c0 con 0x7f02080ce8d0 2023-11-24T08:17:36.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.422+0000 7f0205d9b640 1 -- 172.21.15.107:0/4021196726 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc031d50 con 0x7f02080ce8d0 2023-11-24T08:17:36.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 -- 172.21.15.107:0/4021196726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02080ce8d0 msgr2=0x7f020806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02080ce8d0 0x7f020806dc00 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f020806c020 tx=0x7f01fc02b900 comp rx=0 tx=0).stop 2023-11-24T08:17:36.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 -- 172.21.15.107:0/4021196726 shutdown_connections 2023-11-24T08:17:36.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f020806e140 0x7f02080abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 --2- 172.21.15.107:0/4021196726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02080ce8d0 0x7f020806dc00 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 --2- 172.21.15.107:0/4021196726 >> v2:172.21.15.120:3300/0 conn(0x7f020815d520 0x7f02080ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 -- 172.21.15.107:0/4021196726 >> 172.21.15.107:0/4021196726 conn(0x7f02080b0c70 msgr2=0x7f02080b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:36.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 -- 172.21.15.107:0/4021196726 shutdown_connections 2023-11-24T08:17:36.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.423+0000 7f020d056640 1 -- 172.21.15.107:0/4021196726 wait complete. 2023-11-24T08:17:36.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.425+0000 7f020d056640 1 Processor -- start 2023-11-24T08:17:36.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.425+0000 7f020d056640 1 -- start start 2023-11-24T08:17:36.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.426+0000 7f020d056640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f020806e140 0x7f02080c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.426+0000 7f020d056640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02080ce8d0 0x7f02080c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.426+0000 7f020d056640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020d056640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020814fe50 con 0x7f020815d520 2023-11-24T08:17:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020d056640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080b2f60 con 0x7f020806e140 2023-11-24T08:17:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020d056640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02081f6800 con 0x7f02080ce8d0 2023-11-24T08:17:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36996/0 (socket says 172.21.15.107:36996) 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 -- 172.21.15.107:0/3155953763 learned_addr learned my addr 172.21.15.107:0/3155953763 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 -- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02080ce8d0 msgr2=0x7f02080c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020659c640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02080ce8d0 0x7f02080c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f0206d9d640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f020806e140 0x7f02080c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02080ce8d0 0x7f02080c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 -- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f020806e140 msgr2=0x7f02080c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f020806e140 0x7f02080c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 -- 172.21.15.107:0/3155953763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0094d0 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.427+0000 7f020759e640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f020000cd50 tx=0x7f020000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.428+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020000f680 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.428+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020000f800 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.428+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020000fbb0 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.428+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02081f6ae0 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.428+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02081f70a0 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.429+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02081f6c70 con 0x7f020815d520 2023-11-24T08:17:36.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.430+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f020000b040 con 0x7f020815d520 2023-11-24T08:17:36.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.431+0000 7f01ef7fe640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 0x7f01f406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.431+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0200088d10 con 0x7f020815d520 2023-11-24T08:17:36.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.431+0000 7f0206d9d640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 0x7f01f406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.432+0000 7f0206d9d640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 0x7f01f406ead0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.435+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02000558c0 con 0x7f020815d520 2023-11-24T08:17:36.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.694+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f02080c8890 con 0x7f020815d520 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.696+0000 7f01ef7fe640 1 -- 172.21.15.107:0/3155953763 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f020001d070 con 0x7f020815d520 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:17:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:17:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:17:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:17:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:17:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:17:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:17:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:17:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:17:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:17:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:17:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:17:36.719 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.698+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 msgr2=0x7f01f406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 0x7f01f406ead0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).stop 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 msgr2=0x7f02080cbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f020000cd50 tx=0x7f020000a650 comp rx=0 tx=0).stop 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 shutdown_connections 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f01f406c610 0x7f01f406ead0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f020815d520 0x7f02080cbb50 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02080ce8d0 0x7f02080c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 --2- 172.21.15.107:0/3155953763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f020806e140 0x7f02080c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 >> 172.21.15.107:0/3155953763 conn(0x7f02080b0c70 msgr2=0x7f02080af780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 shutdown_connections 2023-11-24T08:17:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.699+0000 7f020d056640 1 -- 172.21.15.107:0/3155953763 wait complete. 2023-11-24T08:17:36.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.921+0000 7ffbdfc6b640 1 Processor -- start 2023-11-24T08:17:36.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.922+0000 7ffbdfc6b640 1 -- start start 2023-11-24T08:17:36.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.923+0000 7ffbdfc6b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbd806c330 0x7ffbd8161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.923+0000 7ffbdfc6b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd806b8f0 con 0x7ffbd8070e20 2023-11-24T08:17:36.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.923+0000 7ffbdfc6b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd806b610 con 0x7ffbd815d830 2023-11-24T08:17:36.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.923+0000 7ffbdfc6b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd81611e0 con 0x7ffbd806c330 2023-11-24T08:17:36.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbd806c330 0x7ffbd8161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbd806c330 0x7ffbd8161cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36998/0 (socket says 172.21.15.107:36998) 2023-11-24T08:17:36.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 -- 172.21.15.107:0/1226615842 learned_addr learned my addr 172.21.15.107:0/1226615842 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:36.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 -- 172.21.15.107:0/1226615842 >> v1:172.21.15.120:6789/0 conn(0x7ffbd815d830 legacy=0x7ffbd806c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:36.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 -- 172.21.15.107:0/1226615842 >> v1:172.21.15.107:6789/0 conn(0x7ffbd8070e20 legacy=0x7ffbd80cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.924+0000 7ffbdda07640 1 -- 172.21.15.107:0/1226615842 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd81614c0 con 0x7ffbd806c330 2023-11-24T08:17:36.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.926+0000 7ffbdda07640 1 --2- 172.21.15.107:0/1226615842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd806c330 0x7ffbd8161cc0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009870 tx=0x7ffbd402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1288aefdcd0ee416 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.926+0000 7ffbdca05640 1 -- 172.21.15.107:0/1226615842 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4030850 con 0x7ffbd806c330 2023-11-24T08:17:36.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.926+0000 7ffbdca05640 1 -- 172.21.15.107:0/1226615842 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd40309b0 con 0x7ffbd806c330 2023-11-24T08:17:36.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.926+0000 7ffbdca05640 1 -- 172.21.15.107:0/1226615842 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4030d40 con 0x7ffbd806c330 2023-11-24T08:17:36.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/1226615842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd806c330 msgr2=0x7ffbd8161cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/1226615842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd806c330 0x7ffbd8161cc0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009870 tx=0x7ffbd402b990 comp rx=0 tx=0).stop 2023-11-24T08:17:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/1226615842 shutdown_connections 2023-11-24T08:17:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/1226615842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd806c330 0x7ffbd8161cc0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/1226615842 >> 172.21.15.107:0/1226615842 conn(0x7ffbd80b2eb0 msgr2=0x7ffbd80b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/1226615842 shutdown_connections 2023-11-24T08:17:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.927+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/1226615842 wait complete. 2023-11-24T08:17:36.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.928+0000 7ffbdfc6b640 1 Processor -- start 2023-11-24T08:17:36.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.929+0000 7ffbdfc6b640 1 -- start start 2023-11-24T08:17:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.929+0000 7ffbdfc6b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbd806c330 0x7ffbd81599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.929+0000 7ffbdfc6b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 0x7ffbd814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdfc6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdfc6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd815a210 con 0x7ffbd815d830 2023-11-24T08:17:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdfc6b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd815a4f0 con 0x7ffbd8070e20 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdfc6b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd815a7d0 con 0x7ffbd806c330 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37008/0 (socket says 172.21.15.107:37008) 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 -- 172.21.15.107:0/3754524376 learned_addr learned my addr 172.21.15.107:0/3754524376 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdd206640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 0x7ffbd814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdda07640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbd806c330 0x7ffbd81599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 -- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbd806c330 msgr2=0x7ffbd81599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbd806c330 0x7ffbd81599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 -- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 msgr2=0x7ffbd814e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 0x7ffbd814e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbde208640 1 -- 172.21.15.107:0/3754524376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40094d0 con 0x7ffbd815d830 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.930+0000 7ffbdd206640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 0x7ffbd814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbde208640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c7a0 tx=0x7ffbd000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0003b70 con 0x7ffbd815d830 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0003cf0 con 0x7ffbd815d830 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0003ea0 con 0x7ffbd815d830 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd815aab0 con 0x7ffbd815d830 2023-11-24T08:17:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.931+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd8159f00 con 0x7ffbd815d830 2023-11-24T08:17:36.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.932+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd815afd0 con 0x7ffbd815d830 2023-11-24T08:17:36.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.933+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7ffbd000aa90 con 0x7ffbd815d830 2023-11-24T08:17:36.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.934+0000 7ffbca7fc640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 0x7ffbbc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:36.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.934+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffbd0088270 con 0x7ffbd815d830 2023-11-24T08:17:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.934+0000 7ffbdda07640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 0x7ffbbc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.935+0000 7ffbdda07640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 0x7ffbbc06eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4036000 tx=0x7ffbd403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:36.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:36.938+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbd815afd0 con 0x7ffbd815d830 2023-11-24T08:17:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.131+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbd815afd0 con 0x7ffbd815d830 2023-11-24T08:17:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.132+0000 7ffbca7fc640 1 -- 172.21.15.107:0/3754524376 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+12867 (secure 0 0 0) 0x7ffbd815afd0 con 0x7ffbd815d830 2023-11-24T08:17:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:37.134 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":158,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:36.207236+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 msgr2=0x7ffbbc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 0x7ffbbc06eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4036000 tx=0x7ffbd403b040 comp rx=0 tx=0).stop 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 msgr2=0x7ffbd815c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c7a0 tx=0x7ffbd000cc60 comp rx=0 tx=0).stop 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 shutdown_connections 2023-11-24T08:17:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbbc06c5e0 0x7ffbbc06eaa0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbd815d830 0x7ffbd815c0c0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd8070e20 0x7ffbd814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 --2- 172.21.15.107:0/3754524376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbd806c330 0x7ffbd81599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 >> 172.21.15.107:0/3754524376 conn(0x7ffbd80b2eb0 msgr2=0x7ffbd80cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 shutdown_connections 2023-11-24T08:17:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.135+0000 7ffbdfc6b640 1 -- 172.21.15.107:0/3754524376 wait complete. 2023-11-24T08:17:37.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:17:37.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:17:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.364+0000 7f6fdde97640 1 Processor -- start 2023-11-24T08:17:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.364+0000 7f6fdde97640 1 -- start start 2023-11-24T08:17:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fdde97640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 0x7f6fd80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fdde97640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd80c59b0 0x7f6fd806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fdde97640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80c5dc0 con 0x7f6fd80cbf20 2023-11-24T08:17:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fdde97640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd80cc3d0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fdde97640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd815b820 con 0x7f6fd815b440 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.366+0000 7f6fd7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd80c59b0 0x7f6fd806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd80c59b0 0x7f6fd806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37024/0 (socket says 172.21.15.107:37024) 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 -- 172.21.15.107:0/3243827122 learned_addr learned my addr 172.21.15.107:0/3243827122 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd77fe640 1 --2- 172.21.15.107:0/3243827122 >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 0x7f6fd80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 -- 172.21.15.107:0/3243827122 >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 msgr2=0x7f6fd80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 --2- 172.21.15.107:0/3243827122 >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 0x7f6fd80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 -- 172.21.15.107:0/3243827122 >> v1:172.21.15.120:6789/0 conn(0x7f6fd80cbf20 legacy=0x7f6fd80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd7fff640 1 -- 172.21.15.107:0/3243827122 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd815eea0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.367+0000 7f6fd77fe640 1 --2- 172.21.15.107:0/3243827122 >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 0x7f6fd80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.368+0000 7f6fd7fff640 1 --2- 172.21.15.107:0/3243827122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd806c5f0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6fd806e770 tx=0x7f6fd002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362202608d0ccc38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.368+0000 7f6fd6ffd640 1 -- 172.21.15.107:0/3243827122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd002c9f0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.368+0000 7f6fd6ffd640 1 -- 172.21.15.107:0/3243827122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd002cb50 con 0x7f6fd80c59b0 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.368+0000 7f6fd6ffd640 1 -- 172.21.15.107:0/3243827122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00343f0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 -- 172.21.15.107:0/3243827122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 msgr2=0x7f6fd806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 --2- 172.21.15.107:0/3243827122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd806c5f0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6fd806e770 tx=0x7f6fd002be50 comp rx=0 tx=0).stop 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 -- 172.21.15.107:0/3243827122 shutdown_connections 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 --2- 172.21.15.107:0/3243827122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd806c5f0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 --2- 172.21.15.107:0/3243827122 >> v2:172.21.15.120:3300/0 conn(0x7f6fd815b440 0x7f6fd80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 -- 172.21.15.107:0/3243827122 >> 172.21.15.107:0/3243827122 conn(0x7f6fd80b0c50 msgr2=0x7f6fd80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:37.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 -- 172.21.15.107:0/3243827122 shutdown_connections 2023-11-24T08:17:37.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.369+0000 7f6fdde97640 1 -- 172.21.15.107:0/3243827122 wait complete. 2023-11-24T08:17:37.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.371+0000 7f6fdde97640 1 Processor -- start 2023-11-24T08:17:37.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.371+0000 7f6fdde97640 1 -- start start 2023-11-24T08:17:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80cbf20 0x7f6fd80ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd815b440 0x7f6fd80ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd815a3e0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd814c4f0 con 0x7f6fd80cbf20 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fdde97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd8159760 con 0x7f6fd815b440 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37026/0 (socket says 172.21.15.107:37026) 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.372+0000 7f6fd77fe640 1 -- 172.21.15.107:0/1224426469 learned_addr learned my addr 172.21.15.107:0/1224426469 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 -- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd815b440 msgr2=0x7f6fd80ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd815b440 0x7f6fd80ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 -- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80cbf20 msgr2=0x7f6fd80ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80cbf20 0x7f6fd80ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 -- 172.21.15.107:0/1224426469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0009b30 con 0x7f6fd80c59b0 2023-11-24T08:17:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd77fe640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800c7f0 tx=0x7f6fc800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8007820 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc8007980 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8007d10 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd8159a40 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.373+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd815a000 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.374+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd804e8b0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.375+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f6fc80113f0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.376+0000 7f6fd4ff9640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 0x7f6fb806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.376+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6fc80886f0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.377+0000 7f6fcffff640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 0x7f6fb806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.377+0000 7f6fcffff640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 0x7f6fb806eaa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81f67f0 tx=0x7f6fc0007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.380+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fc80552a0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.567+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fd806dc80 con 0x7f6fd80c59b0 2023-11-24T08:17:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.568+0000 7f6fd4ff9640 1 -- 172.21.15.107:0/1224426469 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fc801f0a0 con 0x7f6fd80c59b0 2023-11-24T08:17:37.570 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 msgr2=0x7f6fb806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 0x7f6fb806eaa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81f67f0 tx=0x7f6fc0007990 comp rx=0 tx=0).stop 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 msgr2=0x7f6fd814bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800c7f0 tx=0x7f6fc800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 shutdown_connections 2023-11-24T08:17:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fb806c5e0 0x7f6fb806eaa0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd815b440 0x7f6fd80ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80cbf20 0x7f6fd80ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 --2- 172.21.15.107:0/1224426469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd80c59b0 0x7f6fd814bfb0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 >> 172.21.15.107:0/1224426469 conn(0x7f6fd80b0c50 msgr2=0x7f6fd81549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 shutdown_connections 2023-11-24T08:17:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.571+0000 7f6fdde97640 1 -- 172.21.15.107:0/1224426469 wait complete. 2023-11-24T08:17:37.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:17:37.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.798+0000 7fa2ba052640 1 Processor -- start 2023-11-24T08:17:37.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.798+0000 7fa2ba052640 1 -- start start 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2ba052640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa2b4144b30 0x7fa2b415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2ba052640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa2b415c410 0x7fa2b40b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2ba052640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b415bf80 con 0x7fa2b40b1de0 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2ba052640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40b1560 con 0x7fa2b415c410 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2ba052640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40b0fb0 con 0x7fa2b4144b30 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2b884f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa2b415c410 0x7fa2b40b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2b884f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa2b415c410 0x7fa2b40b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37032/0 (socket says 172.21.15.107:37032) 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.800+0000 7fa2b884f640 1 -- 172.21.15.107:0/2307906363 learned_addr learned my addr 172.21.15.107:0/2307906363 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.801+0000 7fa2b9050640 1 --2- 172.21.15.107:0/2307906363 >> v2:172.21.15.120:3300/0 conn(0x7fa2b4144b30 0x7fa2b415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.801+0000 7fa2b884f640 1 -- 172.21.15.107:0/2307906363 >> v2:172.21.15.120:3300/0 conn(0x7fa2b4144b30 msgr2=0x7fa2b415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.801+0000 7fa2b884f640 1 --2- 172.21.15.107:0/2307906363 >> v2:172.21.15.120:3300/0 conn(0x7fa2b4144b30 0x7fa2b415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.801+0000 7fa2b884f640 1 -- 172.21.15.107:0/2307906363 >> v1:172.21.15.120:6789/0 conn(0x7fa2b40b1de0 legacy=0x7fa2b406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.801+0000 7fa2b884f640 1 -- 172.21.15.107:0/2307906363 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40b1280 con 0x7fa2b415c410 2023-11-24T08:17:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.802+0000 7fa2b884f640 1 --2- 172.21.15.107:0/2307906363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b415c410 0x7fa2b40b0a70 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fa2b406f690 tx=0x7fa2a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf83b13c3fc602a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.802+0000 7fa2abfff640 1 -- 172.21.15.107:0/2307906363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4031870 con 0x7fa2b415c410 2023-11-24T08:17:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.802+0000 7fa2abfff640 1 -- 172.21.15.107:0/2307906363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a40319d0 con 0x7fa2b415c410 2023-11-24T08:17:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.802+0000 7fa2abfff640 1 -- 172.21.15.107:0/2307906363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4031d60 con 0x7fa2b415c410 2023-11-24T08:17:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 -- 172.21.15.107:0/2307906363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b415c410 msgr2=0x7fa2b40b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 --2- 172.21.15.107:0/2307906363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b415c410 0x7fa2b40b0a70 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fa2b406f690 tx=0x7fa2a402b910 comp rx=0 tx=0).stop 2023-11-24T08:17:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 -- 172.21.15.107:0/2307906363 shutdown_connections 2023-11-24T08:17:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 --2- 172.21.15.107:0/2307906363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b415c410 0x7fa2b40b0a70 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 --2- 172.21.15.107:0/2307906363 >> v2:172.21.15.120:3300/0 conn(0x7fa2b4144b30 0x7fa2b415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 -- 172.21.15.107:0/2307906363 >> 172.21.15.107:0/2307906363 conn(0x7fa2b40cb830 msgr2=0x7fa2b40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 -- 172.21.15.107:0/2307906363 shutdown_connections 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.803+0000 7fa2ba052640 1 -- 172.21.15.107:0/2307906363 wait complete. 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.805+0000 7fa2ba052640 1 Processor -- start 2023-11-24T08:17:37.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.805+0000 7fa2ba052640 1 -- start start 2023-11-24T08:17:37.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 0x7fa2b413f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 0x7fa2b413feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b4130b20 con 0x7fa2b40b1de0 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b413ecf0 con 0x7fa2b4144b30 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2ba052640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b4133060 con 0x7fa2b415c410 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b9050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b9050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37038/0 (socket says 172.21.15.107:37038) 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b9050640 1 -- 172.21.15.107:0/1452778013 learned_addr learned my addr 172.21.15.107:0/1452778013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b884f640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 0x7fa2b413f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b9050640 1 -- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 msgr2=0x7fa2b413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.806+0000 7fa2b9851640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 0x7fa2b413feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9050640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 0x7fa2b413feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9050640 1 -- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 msgr2=0x7fa2b413f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9050640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 0x7fa2b413f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9050640 1 -- 172.21.15.107:0/1452778013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a40094d0 con 0x7fa2b40b1de0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b884f640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 0x7fa2b413f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9851640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 0x7fa2b413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2b9050640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c7f0 tx=0x7fa2b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004220 con 0x7fa2b40b1de0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b00043a0 con 0x7fa2b40b1de0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b4133340 con 0x7fa2b40b1de0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0007860 con 0x7fa2b40b1de0 2023-11-24T08:17:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.807+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b4130740 con 0x7fa2b40b1de0 2023-11-24T08:17:37.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.808+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b40cdca0 con 0x7fa2b40b1de0 2023-11-24T08:17:37.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.809+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fa2b0007bd0 con 0x7fa2b40b1de0 2023-11-24T08:17:37.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.810+0000 7fa2a9ffb640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 0x7fa29806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:37.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.810+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2b00883a0 con 0x7fa2b40b1de0 2023-11-24T08:17:37.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.811+0000 7fa2b884f640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 0x7fa29806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:37.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.812+0000 7fa2b884f640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 0x7fa29806ead0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4036000 tx=0x7fa2a403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:37.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:37.815+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2b0054eb0 con 0x7fa2b40b1de0 2023-11-24T08:17:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.006+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa2b404e840 con 0x7fa2b40b1de0 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.007+0000 7fa2a9ffb640 1 -- 172.21.15.107:0/1452778013 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v158) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa2b001e020 con 0x7fa2b40b1de0 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:38.009 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.009+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 msgr2=0x7fa29806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.009+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 0x7fa29806ead0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4036000 tx=0x7fa2a403c040 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.009+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 msgr2=0x7fa2b40afe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.009+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c7f0 tx=0x7fa2b000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 shutdown_connections 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa29806c610 0x7fa29806ead0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2b415c410 0x7fa2b413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2b4144b30 0x7fa2b413f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 --2- 172.21.15.107:0/1452778013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2b40b1de0 0x7fa2b40afe50 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 >> 172.21.15.107:0/1452778013 conn(0x7fa2b40cb830 msgr2=0x7fa2b40722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 shutdown_connections 2023-11-24T08:17:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.010+0000 7fa2ba052640 1 -- 172.21.15.107:0/1452778013 wait complete. 2023-11-24T08:17:38.031 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:17:38.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:17:38.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.242+0000 7faa4162d640 1 Processor -- start 2023-11-24T08:17:38.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.242+0000 7faa4162d640 1 -- start start 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa4162d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa3c0cfb60 0x7faa3c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa4162d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c0ce0b0 con 0x7faa3c157c70 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa4162d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c161960 con 0x7faa3c15dc80 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa4162d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3c15e8d0 con 0x7faa3c0cfb60 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa3affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faa3c157c70 0x7faa3c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33292/0 (socket says 172.21.15.107:33292) 2023-11-24T08:17:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.244+0000 7faa3affd640 1 -- 172.21.15.107:0/2530074976 learned_addr learned my addr 172.21.15.107:0/2530074976 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:38.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa327fc640 1 --2- 172.21.15.107:0/2530074976 >> v2:172.21.15.120:3300/0 conn(0x7faa3c0cfb60 0x7faa3c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836645332 0 0) 0x7faa3c0ce0b0 con 0x7faa3c157c70 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28003520 con 0x7faa3c157c70 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710301847 0 0) 0x7faa28003520 con 0x7faa3c157c70 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 >> v2:172.21.15.120:3300/0 conn(0x7faa3c0cfb60 msgr2=0x7faa3c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 --2- 172.21.15.107:0/2530074976 >> v2:172.21.15.120:3300/0 conn(0x7faa3c0cfb60 0x7faa3c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 >> v1:172.21.15.120:6789/0 conn(0x7faa3c15dc80 legacy=0x7faa3c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c15ebb0 con 0x7faa3c157c70 2023-11-24T08:17:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7faa2c001670 con 0x7faa3c157c70 2023-11-24T08:17:38.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 >> v1:172.21.15.107:6789/0 conn(0x7faa3c157c70 legacy=0x7faa3c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 0x7faa28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa280047c0 0x7faa28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28005180 0x7faa2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c15e8d0 con 0x7faa28003970 2023-11-24T08:17:38.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2800c120 con 0x7faa28005180 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.245+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2800c2e0 con 0x7faa280047c0 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 0x7faa28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa327fc640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa280047c0 0x7faa28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3b7fe640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28005180 0x7faa2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 -- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa280047c0 msgr2=0x7faa28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa280047c0 0x7faa28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 -- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28005180 msgr2=0x7faa2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:38.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28005180 0x7faa2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3affd640 1 -- 172.21.15.107:0/2530074976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28003520 con 0x7faa28003970 2023-11-24T08:17:38.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3b7fe640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28005180 0x7faa2800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:38.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa3b7fe640 1 -- 172.21.15.107:0/2530074976 reap_dead start 2023-11-24T08:17:38.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.246+0000 7faa327fc640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa280047c0 0x7faa28004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.247+0000 7faa3affd640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 0x7faa28003d90 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7faa2c00c190 tx=0x7faa280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe19ee5138766ed7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.247+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c005ad0 con 0x7faa28003970 2023-11-24T08:17:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.247+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c005c30 con 0x7faa28003970 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.247+0000 7faa3a7fc640 1 -- 172.21.15.107:0/2530074976 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02caf0 con 0x7faa28003970 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 -- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 msgr2=0x7faa28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 0x7faa28003d90 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7faa2c00c190 tx=0x7faa280057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 -- 172.21.15.107:0/2530074976 shutdown_connections 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 --2- 172.21.15.107:0/2530074976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa28003970 0x7faa28003d90 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 -- 172.21.15.107:0/2530074976 >> 172.21.15.107:0/2530074976 conn(0x7faa3c06a920 msgr2=0x7faa3c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 -- 172.21.15.107:0/2530074976 shutdown_connections 2023-11-24T08:17:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.248+0000 7faa4162d640 1 -- 172.21.15.107:0/2530074976 wait complete. 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.250+0000 7faa4162d640 1 Processor -- start 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.250+0000 7faa4162d640 1 -- start start 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 0x7faa3c14a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa3c0ae160 0x7faa3c0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0ae750 con 0x7faa3c0ad790 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0ac9d0 con 0x7faa28003970 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa4162d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0accb0 con 0x7faa3c0ae160 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa327fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa327fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36108/0 (socket says 172.21.15.107:36108) 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.251+0000 7faa327fc640 1 -- 172.21.15.107:0/276668279 learned_addr learned my addr 172.21.15.107:0/276668279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa3affd640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 0x7faa3c14a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa3b7fe640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa3c0ae160 0x7faa3c0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 -- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa3c0ae160 msgr2=0x7faa3c0c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa3c0ae160 0x7faa3c0c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 -- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 msgr2=0x7faa3c14a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 0x7faa3c14a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 -- 172.21.15.107:0/276668279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0acf90 con 0x7faa3c0ad790 2023-11-24T08:17:38.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa3affd640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 0x7faa3c14a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa327fc640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7faa240107c0 tx=0x7faa24010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2400eb60 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.252+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2400ecc0 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.253+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24010e70 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.253+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3c0ad270 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.253+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3c14aac0 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.254+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c067470 con 0x7faa3c0ad790 2023-11-24T08:17:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.255+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7faa2401c400 con 0x7faa3c0ad790 2023-11-24T08:17:38.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.256+0000 7faa33fff640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 0x7faa1c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:38.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.256+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faa240886e0 con 0x7faa3c0ad790 2023-11-24T08:17:38.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.256+0000 7faa3affd640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 0x7faa1c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:38.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.257+0000 7faa3affd640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 0x7faa1c06eaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faa2c034000 tx=0x7faa2c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:38.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.260+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa24055340 con 0x7faa3c0ad790 2023-11-24T08:17:38.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:38.448+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faa3c0affd0 con 0x7faa3c0ad790 2023-11-24T08:17:39.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.220+0000 7faa33fff640 1 -- 172.21.15.107:0/276668279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v159) v1 ==== 138+0+0 (secure 0 0 0) 0x7faa24058960 con 0x7faa3c0ad790 2023-11-24T08:17:39.222 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 msgr2=0x7faa1c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 0x7faa1c06eaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faa2c034000 tx=0x7faa2c009030 comp rx=0 tx=0).stop 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 msgr2=0x7faa3c0adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7faa240107c0 tx=0x7faa24010c80 comp rx=0 tx=0).stop 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 shutdown_connections 2023-11-24T08:17:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faa1c06c5e0 0x7faa1c06eaa0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa3c0ae160 0x7faa3c0c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa3c0ad790 0x7faa3c0adc20 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 --2- 172.21.15.107:0/276668279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28003970 0x7faa3c14a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.225+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 >> 172.21.15.107:0/276668279 conn(0x7faa3c06a920 msgr2=0x7faa3c0b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:39.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.226+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 shutdown_connections 2023-11-24T08:17:39.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.226+0000 7faa4162d640 1 -- 172.21.15.107:0/276668279 wait complete. 2023-11-24T08:17:39.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:17:39.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.456+0000 7f61dc2e7640 1 Processor -- start 2023-11-24T08:17:39.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.456+0000 7f61dc2e7640 1 -- start start 2023-11-24T08:17:39.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61dc2e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61d4072160 0x7f61d415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61dc2e7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f61d406de10 0x7f61d40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61dc2e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d415ec90 con 0x7f61d40c49f0 2023-11-24T08:17:39.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61dc2e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d40cd530 con 0x7f61d4072160 2023-11-24T08:17:39.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61dc2e7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d40c4ed0 con 0x7f61d406de10 2023-11-24T08:17:39.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61da083640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61d4072160 0x7f61d415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:39.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61da083640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61d4072160 0x7f61d415e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36112/0 (socket says 172.21.15.107:36112) 2023-11-24T08:17:39.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.458+0000 7f61da083640 1 -- 172.21.15.107:0/4141737701 learned_addr learned my addr 172.21.15.107:0/4141737701 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:39.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.459+0000 7f61da083640 1 -- 172.21.15.107:0/4141737701 >> v2:172.21.15.120:3300/0 conn(0x7f61d406de10 msgr2=0x7f61d40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.459+0000 7f61da083640 1 --2- 172.21.15.107:0/4141737701 >> v2:172.21.15.120:3300/0 conn(0x7f61d406de10 0x7f61d40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.459+0000 7f61da083640 1 -- 172.21.15.107:0/4141737701 >> v1:172.21.15.107:6789/0 conn(0x7f61d40c49f0 legacy=0x7f61d406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.459+0000 7f61da083640 1 -- 172.21.15.107:0/4141737701 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d40c51b0 con 0x7f61d4072160 2023-11-24T08:17:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.460+0000 7f61da083640 1 --2- 172.21.15.107:0/4141737701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415e750 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f61d406eab0 tx=0x7f61d002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5d09298c963a80 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.460+0000 7f61d9081640 1 -- 172.21.15.107:0/4141737701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d002f8e0 con 0x7f61d4072160 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.460+0000 7f61d9081640 1 -- 172.21.15.107:0/4141737701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d002fa40 con 0x7f61d4072160 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.460+0000 7f61d9081640 1 -- 172.21.15.107:0/4141737701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d002fdd0 con 0x7f61d4072160 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 -- 172.21.15.107:0/4141737701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 msgr2=0x7f61d415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/4141737701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415e750 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f61d406eab0 tx=0x7f61d002db20 comp rx=0 tx=0).stop 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 -- 172.21.15.107:0/4141737701 shutdown_connections 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/4141737701 >> v2:172.21.15.120:3300/0 conn(0x7f61d406de10 0x7f61d40705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/4141737701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415e750 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 -- 172.21.15.107:0/4141737701 >> 172.21.15.107:0/4141737701 conn(0x7f61d40b0c50 msgr2=0x7f61d40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:39.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 -- 172.21.15.107:0/4141737701 shutdown_connections 2023-11-24T08:17:39.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.461+0000 7f61dc2e7640 1 -- 172.21.15.107:0/4141737701 wait complete. 2023-11-24T08:17:39.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.463+0000 7f61dc2e7640 1 Processor -- start 2023-11-24T08:17:39.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.463+0000 7f61dc2e7640 1 -- start start 2023-11-24T08:17:39.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61d406de10 0x7f61d41f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61d40c49f0 0x7f61d40c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d41f6ed0 con 0x7f61d4072160 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d41f68f0 con 0x7f61d40c49f0 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61dc2e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40c6b90 con 0x7f61d406de10 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61d9882640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61d9882640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36122/0 (socket says 172.21.15.107:36122) 2023-11-24T08:17:39.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.464+0000 7f61d9882640 1 -- 172.21.15.107:0/964858453 learned_addr learned my addr 172.21.15.107:0/964858453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61da083640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61d406de10 0x7f61d41f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61da884640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61d40c49f0 0x7f61d40c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 -- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61d406de10 msgr2=0x7f61d41f7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61d406de10 0x7f61d41f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 -- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61d40c49f0 msgr2=0x7f61d40c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61d40c49f0 0x7f61d40c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 -- 172.21.15.107:0/964858453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00094d0 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61d9882640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f61c400b420 tx=0x7f61c400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4011830 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4011990 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d41f6b60 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4011d40 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.465+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d415d190 con 0x7f61d4072160 2023-11-24T08:17:39.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.466+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d41f7080 con 0x7f61d4072160 2023-11-24T08:17:39.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.468+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f61c401b460 con 0x7f61d4072160 2023-11-24T08:17:39.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.469+0000 7f61caffd640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 0x7f61b806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:39.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.469+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f61c40886b0 con 0x7f61d4072160 2023-11-24T08:17:39.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.469+0000 7f61da083640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 0x7f61b806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:39.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.470+0000 7f61da083640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 0x7f61b806ead0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f61d0038000 tx=0x7f61d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:39.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.473+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61c4055340 con 0x7f61d4072160 2023-11-24T08:17:39.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:39.663+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f61d41f7080 con 0x7f61d4072160 2023-11-24T08:17:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.232+0000 7f61caffd640 1 -- 172.21.15.107:0/964858453 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v160) v1 ==== 170+0+0 (secure 0 0 0) 0x7f61d41f7080 con 0x7f61d4072160 2023-11-24T08:17:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 msgr2=0x7f61b806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 0x7f61b806ead0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f61d0038000 tx=0x7f61d0008030 comp rx=0 tx=0).stop 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 msgr2=0x7f61d415cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f61c400b420 tx=0x7f61c400b8e0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 shutdown_connections 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f61b806c610 0x7f61b806ead0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61d40c49f0 0x7f61d40c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d4072160 0x7f61d415cc50 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 --2- 172.21.15.107:0/964858453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61d406de10 0x7f61d41f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 >> 172.21.15.107:0/964858453 conn(0x7f61d40b0c50 msgr2=0x7f61d40ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:40.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 shutdown_connections 2023-11-24T08:17:40.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.236+0000 7f61dc2e7640 1 -- 172.21.15.107:0/964858453 wait complete. 2023-11-24T08:17:40.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:17:40.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.449+0000 7f0ebcffc640 1 Processor -- start 2023-11-24T08:17:40.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.449+0000 7f0ebcffc640 1 -- start start 2023-11-24T08:17:40.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.450+0000 7f0ebcffc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 0x7f0eb8166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.450+0000 7f0ebcffc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb815eea0 con 0x7f0eb815b440 2023-11-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.450+0000 7f0ebcffc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb806d600 con 0x7f0eb80cc0f0 2023-11-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0ebcffc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c5e00 con 0x7f0eb80c59b0 2023-11-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0eb659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0eb815b440 0x7f0eb80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33306/0 (socket says 172.21.15.107:33306) 2023-11-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0eb659c640 1 -- 172.21.15.107:0/3920226485 learned_addr learned my addr 172.21.15.107:0/3920226485 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995671524 0 0) 0x7f0eb815eea0 con 0x7f0eb815b440 2023-11-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003520 con 0x7f0eb815b440 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.451+0000 7f0eb6d9d640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 0x7f0eb8166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3089839602 0 0) 0x7f0ea4003520 con 0x7f0eb815b440 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 msgr2=0x7f0eb8166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 0x7f0eb8166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 >> v1:172.21.15.120:6789/0 conn(0x7f0eb80cc0f0 legacy=0x7f0eb80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80abf30 con 0x7f0eb815b440 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0ea8001670 con 0x7f0eb815b440 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 >> v1:172.21.15.107:6789/0 conn(0x7f0eb815b440 legacy=0x7f0eb80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 0x7f0ea40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea4003fd0 0x7f0ea40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4004d00 0x7f0ea4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea400bb90 con 0x7f0ea4003fd0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea400bd00 con 0x7f0ea4004d00 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea400bec0 con 0x7f0eb80c59b0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea4003fd0 0x7f0ea40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 -- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 msgr2=0x7f0ea40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0eb80c59b0 0x7f0ea40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 -- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4004d00 msgr2=0x7f0ea4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eb6d9d640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4004d00 0x7f0ea4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4004d00 0x7f0ea4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.452+0000 7f0eadd9b640 1 -- 172.21.15.107:0/3920226485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4003520 con 0x7f0ea4003fd0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.453+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea4003fd0 0x7f0ea40047c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001750 tx=0x7f0ea40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0b63058ef72091 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00052d0 con 0x7f0ea4003fd0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea002e050 con 0x7f0ea4003fd0 2023-11-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0eb5d9b640 1 -- 172.21.15.107:0/3920226485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0003830 con 0x7f0ea4003fd0 2023-11-24T08:17:40.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 -- 172.21.15.107:0/3920226485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea4003fd0 msgr2=0x7f0ea40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/3920226485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea4003fd0 0x7f0ea40047c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001750 tx=0x7f0ea40057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0eb6d9d640 1 -- 172.21.15.107:0/3920226485 reap_dead start 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 -- 172.21.15.107:0/3920226485 shutdown_connections 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 -- 172.21.15.107:0/3920226485 >> 172.21.15.107:0/3920226485 conn(0x7f0eb80b0c50 msgr2=0x7f0eb80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 -- 172.21.15.107:0/3920226485 shutdown_connections 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.454+0000 7f0ebcffc640 1 -- 172.21.15.107:0/3920226485 wait complete. 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.456+0000 7f0ebcffc640 1 Processor -- start 2023-11-24T08:17:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.456+0000 7f0ebcffc640 1 -- start start 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4003fd0 0x7f0eb8154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0eb80af9c0 0x7f0eb80afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b0380 con 0x7f0eb8153df0 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8144e80 con 0x7f0eb80af9c0 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0ebcffc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb81528d0 con 0x7f0ea4003fd0 2023-11-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0eadd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0eadd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36144/0 (socket says 172.21.15.107:36144) 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.457+0000 7f0eadd9b640 1 -- 172.21.15.107:0/2118901366 learned_addr learned my addr 172.21.15.107:0/2118901366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 -- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4003fd0 msgr2=0x7f0eb8154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4003fd0 0x7f0eb8154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 -- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0eb80af9c0 msgr2=0x7f0eb80afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0eb80af9c0 0x7f0eb80afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 -- 172.21.15.107:0/2118901366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb8153170 con 0x7f0eb8153df0 2023-11-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eadd9b640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0008fd0 tx=0x7f0ea0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00076c0 con 0x7f0eb8153df0 2023-11-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea0007a30 con 0x7f0eb8153df0 2023-11-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb81542f0 con 0x7f0eb8153df0 2023-11-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.458+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0006c00 con 0x7f0eb8153df0 2023-11-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.459+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb8152e00 con 0x7f0eb8153df0 2023-11-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.460+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb804e8b0 con 0x7f0eb8153df0 2023-11-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.462+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f0ea00035f0 con 0x7f0eb8153df0 2023-11-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.463+0000 7f0eaf7fe640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 0x7f0e9806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.463+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0ea00aa720 con 0x7f0eb8153df0 2023-11-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.463+0000 7f0eb659c640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 0x7f0e9806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:40.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.464+0000 7f0eb659c640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 0x7f0e9806eaa0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80011e0 tx=0x7f0ea80014e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:40.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.467+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ea007aed0 con 0x7f0eb8153df0 2023-11-24T08:17:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:40.692+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0eb806dc80 con 0x7f0eb8153df0 2023-11-24T08:17:41.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.255+0000 7f0eaf7fe640 1 -- 172.21.15.107:0/2118901366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v93) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0ea004b070 con 0x7f0eb8153df0 2023-11-24T08:17:41.259 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 22 and data pool 23 2023-11-24T08:17:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 msgr2=0x7f0e9806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 0x7f0e9806eaa0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80011e0 tx=0x7f0ea80014e0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 msgr2=0x7f0eb8144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0008fd0 tx=0x7f0ea0001120 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 shutdown_connections 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0e9806c5e0 0x7f0e9806eaa0 secure :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80011e0 tx=0x7f0ea80014e0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0eb80af9c0 0x7f0eb80afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0eb8153df0 0x7f0eb8144940 secure :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0008fd0 tx=0x7f0ea0001120 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 --2- 172.21.15.107:0/2118901366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4003fd0 0x7f0eb8154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.265+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 >> 172.21.15.107:0/2118901366 conn(0x7f0eb80b0c50 msgr2=0x7f0eb80c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.267+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 shutdown_connections 2023-11-24T08:17:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.267+0000 7f0ebcffc640 1 -- 172.21.15.107:0/2118901366 wait complete. 2023-11-24T08:17:41.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:41.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.476+0000 7f7aecac2640 1 Processor -- start 2023-11-24T08:17:41.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.477+0000 7f7aecac2640 1 -- start start 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7aecac2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ae80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7aecac2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae80cece0 con 0x7f7ae815c7f0 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7aecac2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae806d040 con 0x7f7ae80cf450 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7aecac2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80cfe40 con 0x7f7ae80ce6c0 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7ae659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7ae815c7f0 0x7f7ae806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33312/0 (socket says 172.21.15.107:33312) 2023-11-24T08:17:41.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.478+0000 7f7ae659c640 1 -- 172.21.15.107:0/1126288227 learned_addr learned my addr 172.21.15.107:0/1126288227 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551686165 0 0) 0x7f7ae80cece0 con 0x7f7ae815c7f0 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad4003520 con 0x7f7ae815c7f0 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae6d9d640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ae80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4212780984 0 0) 0x7f7ad4003520 con 0x7f7ae815c7f0 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 msgr2=0x7f7ae80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ae80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 >> v1:172.21.15.120:6789/0 conn(0x7f7ae80cf450 legacy=0x7f7ae80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae806d320 con 0x7f7ae815c7f0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7ad8001670 con 0x7f7ae815c7f0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 >> v1:172.21.15.107:6789/0 conn(0x7f7ae815c7f0 legacy=0x7f7ae806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ad40039a0 0x7f7ad4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ad40047c0 0x7f7ad4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ad40051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80cfe40 con 0x7f7ad40047c0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad400ba90 con 0x7f7ad40039a0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.479+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad400bc50 con 0x7f7ae80ce6c0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ad40047c0 0x7f7ad4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7ae659c640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ad40039a0 0x7f7ad4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7ae6d9d640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ad40051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 -- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 msgr2=0x7f7ad40051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae80ce6c0 0x7f7ad40051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 -- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ad40039a0 msgr2=0x7f7ad4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ad40039a0 0x7f7ad4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.480+0000 7f7adffff640 1 -- 172.21.15.107:0/1126288227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4003520 con 0x7f7ad40047c0 2023-11-24T08:17:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.481+0000 7f7adffff640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ad40047c0 0x7f7ad4004c40 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0003db0 tx=0x7f7ad40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cd9f75b40ae401 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.481+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0032600 con 0x7f7ad40047c0 2023-11-24T08:17:41.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.481+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad002e050 con 0x7f7ad40047c0 2023-11-24T08:17:41.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7ae5d9b640 1 -- 172.21.15.107:0/1126288227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00329a0 con 0x7f7ad40047c0 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 -- 172.21.15.107:0/1126288227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ad40047c0 msgr2=0x7f7ad4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 --2- 172.21.15.107:0/1126288227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ad40047c0 0x7f7ad4004c40 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0003db0 tx=0x7f7ad40057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7ae6d9d640 1 -- 172.21.15.107:0/1126288227 reap_dead start 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 -- 172.21.15.107:0/1126288227 shutdown_connections 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 -- 172.21.15.107:0/1126288227 >> 172.21.15.107:0/1126288227 conn(0x7f7ae806a920 msgr2=0x7f7ae806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 -- 172.21.15.107:0/1126288227 shutdown_connections 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.482+0000 7f7aecac2640 1 -- 172.21.15.107:0/1126288227 wait complete. 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.484+0000 7f7aecac2640 1 Processor -- start 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.484+0000 7f7aecac2640 1 -- start start 2023-11-24T08:17:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ad40047c0 0x7f7ae81578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae8156320 0x7f7ae815acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8157490 con 0x7f7ae8147da0 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8156810 con 0x7f7ae8156320 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7aecac2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8156af0 con 0x7f7ad40047c0 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7adffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7adffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36166/0 (socket says 172.21.15.107:36166) 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.485+0000 7f7adffff640 1 -- 172.21.15.107:0/2497750402 learned_addr learned my addr 172.21.15.107:0/2497750402 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 -- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ad40047c0 msgr2=0x7f7ae81578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ad40047c0 0x7f7ae81578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 -- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae8156320 msgr2=0x7f7ae815acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae8156320 0x7f7ae815acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 -- 172.21.15.107:0/2497750402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae8156dd0 con 0x7f7ae8147da0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adffff640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f7ad002bb30 tx=0x7f7ad00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad003d830 con 0x7f7ae8147da0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad003d990 con 0x7f7ae8147da0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae81570b0 con 0x7f7ae8147da0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad003dd00 con 0x7f7ae8147da0 2023-11-24T08:17:41.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.486+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae81486a0 con 0x7f7ae8147da0 2023-11-24T08:17:41.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.488+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae8067470 con 0x7f7ae8147da0 2023-11-24T08:17:41.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.489+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f7ad0032400 con 0x7f7ae8147da0 2023-11-24T08:17:41.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.490+0000 7f7adeffd640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 0x7f7ac806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.490+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ad00aa0e0 con 0x7f7ae8147da0 2023-11-24T08:17:41.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.490+0000 7f7ae659c640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 0x7f7ac806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.490+0000 7f7ae659c640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 0x7f7ac806eaa0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005990 tx=0x7f7ad8001180 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.494+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ad0076d40 con 0x7f7ae8147da0 2023-11-24T08:17:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.684+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ae80cd730 con 0x7f7ae8147da0 2023-11-24T08:17:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.687+0000 7f7adeffd640 1 -- 172.21.15.107:0/2497750402 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f7ad0037090 con 0x7f7ae8147da0 2023-11-24T08:17:41.690 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:41.690 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":161,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:40.694340+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 msgr2=0x7f7ac806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 0x7f7ac806eaa0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005990 tx=0x7f7ad8001180 comp rx=0 tx=0).stop 2023-11-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 msgr2=0x7f7ae8155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f7ad002bb30 tx=0x7f7ad00036f0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 shutdown_connections 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ac806c5e0 0x7f7ac806eaa0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae8156320 0x7f7ae815acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ae8147da0 0x7f7ae8155de0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 --2- 172.21.15.107:0/2497750402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ad40047c0 0x7f7ae81578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 >> 172.21.15.107:0/2497750402 conn(0x7f7ae806a920 msgr2=0x7f7ae80b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 shutdown_connections 2023-11-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.690+0000 7f7aecac2640 1 -- 172.21.15.107:0/2497750402 wait complete. 2023-11-24T08:17:41.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:41.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.913+0000 7f60b7733640 1 Processor -- start 2023-11-24T08:17:41.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.913+0000 7f60b7733640 1 -- start start 2023-11-24T08:17:41.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b7733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f60b015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b7733640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b015ab80 con 0x7f60b00b2630 2023-11-24T08:17:41.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b7733640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b015e680 con 0x7f60b00b2a70 2023-11-24T08:17:41.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b7733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b015a990 con 0x7f60b00b2250 2023-11-24T08:17:41.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b4cce640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f60b00b2630 0x7f60b00ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33316/0 (socket says 172.21.15.107:33316) 2023-11-24T08:17:41.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b4cce640 1 -- 172.21.15.107:0/1622948228 learned_addr learned my addr 172.21.15.107:0/1622948228 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:41.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730907976 0 0) 0x7f60b015ab80 con 0x7f60b00b2630 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c005730 con 0x7f60b00b2630 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.915+0000 7f60b54cf640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f60b015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 754934727 0 0) 0x7f609c005730 con 0x7f60b00b2630 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 msgr2=0x7f60b015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f60b015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 >> v1:172.21.15.120:6789/0 conn(0x7f60b00b2a70 legacy=0x7f60b015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b015e960 con 0x7f60b00b2630 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f60a0001670 con 0x7f60b00b2630 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60b54cf640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f60b015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 >> v1:172.21.15.107:6789/0 conn(0x7f60b00b2630 legacy=0x7f60b00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f609c005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c0061a0 0x7f609c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f609c006f10 0x7f609c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c00dcc0 con 0x7f609c006f10 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c00de30 con 0x7f609c0061a0 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c00dff0 con 0x7f60b00b2250 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60b5cd0640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f609c006f10 0x7f609c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60b4cce640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c0061a0 0x7f609c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.916+0000 7f60b54cf640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f609c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.917+0000 7f60b5cd0640 1 -- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 msgr2=0x7f609c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.917+0000 7f60b5cd0640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b00b2250 0x7f609c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.917+0000 7f60b5cd0640 1 -- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c0061a0 msgr2=0x7f609c0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.917+0000 7f60b5cd0640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c0061a0 0x7f609c0069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.917+0000 7f60b5cd0640 1 -- 172.21.15.107:0/1622948228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c005730 con 0x7f609c006f10 2023-11-24T08:17:41.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60b5cd0640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f609c006f10 0x7f609c007390 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f60a80044b0 tx=0x7f609c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb0ee73f32e21f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8030620 con 0x7f609c006f10 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8030780 con 0x7f609c006f10 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60a7fff640 1 -- 172.21.15.107:0/1622948228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8030b70 con 0x7f609c006f10 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60b7733640 1 -- 172.21.15.107:0/1622948228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f609c006f10 msgr2=0x7f609c007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60b7733640 1 --2- 172.21.15.107:0/1622948228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f609c006f10 0x7f609c007390 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f60a80044b0 tx=0x7f609c007a00 comp rx=0 tx=0).stop 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.918+0000 7f60b5cd0640 1 -- 172.21.15.107:0/1622948228 reap_dead start 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.919+0000 7f60b7733640 1 -- 172.21.15.107:0/1622948228 shutdown_connections 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.919+0000 7f60b7733640 1 -- 172.21.15.107:0/1622948228 >> 172.21.15.107:0/1622948228 conn(0x7f60b006a920 msgr2=0x7f60b006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.919+0000 7f60b7733640 1 -- 172.21.15.107:0/1622948228 shutdown_connections 2023-11-24T08:17:41.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.919+0000 7f60b7733640 1 -- 172.21.15.107:0/1622948228 wait complete. 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.920+0000 7f60b7733640 1 Processor -- start 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.920+0000 7f60b7733640 1 -- start start 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c006f10 0x7f60b00c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b0153910 0x7f60b0153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01542d0 con 0x7f60b01464b0 2023-11-24T08:17:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b00ad120 con 0x7f609c006f10 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.921+0000 7f60b7733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0146930 con 0x7f60b0153910 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36184/0 (socket says 172.21.15.107:36184) 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 -- 172.21.15.107:0/1850812044 learned_addr learned my addr 172.21.15.107:0/1850812044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 -- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b0153910 msgr2=0x7f60b0153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b0153910 0x7f60b0153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 -- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c006f10 msgr2=0x7f60b00c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c006f10 0x7f60b00c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 -- 172.21.15.107:0/1850812044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00cd7c0 con 0x7f60b01464b0 2023-11-24T08:17:41.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60b4cce640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f60a000d8e0 tx=0x7f60a000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.922+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00074e0 con 0x7f60b01464b0 2023-11-24T08:17:41.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.923+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0001d70 con 0x7f60b01464b0 2023-11-24T08:17:41.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.923+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0004a00 con 0x7f60b01464b0 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.923+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b00ac510 con 0x7f60b01464b0 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.923+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b00acad0 con 0x7f60b01464b0 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.924+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b00b3940 con 0x7f60b01464b0 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.925+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f60a0004b60 con 0x7f60b01464b0 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.926+0000 7f60a5ffb640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 0x7f609406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:41.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.926+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60a0087cf0 con 0x7f60b01464b0 2023-11-24T08:17:41.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.926+0000 7f60b54cf640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 0x7f609406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:41.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.926+0000 7f60b54cf640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 0x7f609406eaa0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f60ac006f80 tx=0x7f60ac006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:41.930+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60a0058330 con 0x7f60b01464b0 2023-11-24T08:17:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.118+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60b00cd610 con 0x7f60b01464b0 2023-11-24T08:17:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.120+0000 7f60a5ffb640 1 -- 172.21.15.107:0/1850812044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f60a0019070 con 0x7f60b01464b0 2023-11-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":161,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:40.694340+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 msgr2=0x7f609406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 0x7f609406eaa0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f60ac006f80 tx=0x7f60ac006dd0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 msgr2=0x7f60b00ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f60a000d8e0 tx=0x7f60a000dda0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 shutdown_connections 2023-11-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f609406c5e0 0x7f609406eaa0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b0153910 0x7f60b0153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60b01464b0 0x7f60b00ca9a0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 --2- 172.21.15.107:0/1850812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f609c006f10 0x7f60b00c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 >> 172.21.15.107:0/1850812044 conn(0x7f60b006a920 msgr2=0x7f60b01594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 shutdown_connections 2023-11-24T08:17:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.123+0000 7f60b7733640 1 -- 172.21.15.107:0/1850812044 wait complete. 2023-11-24T08:17:42.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.357+0000 7f265cd90640 1 Processor -- start 2023-11-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.357+0000 7f265cd90640 1 -- start start 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265cd90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f265815b440 0x7f26580cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265cd90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f26580c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265cd90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265815eea0 con 0x7f26580c57e0 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265cd90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26580cc3d0 con 0x7f265815b440 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265cd90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265815b820 con 0x7f26580cbf20 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f265815b440 0x7f26580cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f265815b440 0x7f26580cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36188/0 (socket says 172.21.15.107:36188) 2023-11-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 -- 172.21.15.107:0/1537015195 learned_addr learned my addr 172.21.15.107:0/1537015195 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 -- 172.21.15.107:0/1537015195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 msgr2=0x7f26580c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f2655d9b640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f26580c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f26580c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 -- 172.21.15.107:0/1537015195 >> v1:172.21.15.120:6789/0 conn(0x7f26580c57e0 legacy=0x7f26580c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.359+0000 7f265659c640 1 -- 172.21.15.107:0/1537015195 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265806d600 con 0x7f265815b440 2023-11-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.361+0000 7f265659c640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580cb9e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f2648009870 tx=0x7f264802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3febdc342cc664 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.361+0000 7f265559a640 1 -- 172.21.15.107:0/1537015195 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26480317c0 con 0x7f265815b440 2023-11-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.361+0000 7f265559a640 1 -- 172.21.15.107:0/1537015195 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2648031920 con 0x7f265815b440 2023-11-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.361+0000 7f265559a640 1 -- 172.21.15.107:0/1537015195 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648031cb0 con 0x7f265815b440 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 -- 172.21.15.107:0/1537015195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 msgr2=0x7f26580cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580cb9e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f2648009870 tx=0x7f264802b860 comp rx=0 tx=0).stop 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 -- 172.21.15.107:0/1537015195 shutdown_connections 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f26580c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 --2- 172.21.15.107:0/1537015195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580cb9e0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 -- 172.21.15.107:0/1537015195 >> 172.21.15.107:0/1537015195 conn(0x7f26580b0c50 msgr2=0x7f26580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 -- 172.21.15.107:0/1537015195 shutdown_connections 2023-11-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.362+0000 7f265cd90640 1 -- 172.21.15.107:0/1537015195 wait complete. 2023-11-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.364+0000 7f265cd90640 1 Processor -- start 2023-11-24T08:17:42.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.364+0000 7f265cd90640 1 -- start start 2023-11-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 0x7f265815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f265814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26580b4950 con 0x7f265815b440 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265815cfb0 con 0x7f26580c57e0 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f265cd90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26580c6610 con 0x7f26580cbf20 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f2656d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f2656d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36192/0 (socket says 172.21.15.107:36192) 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.365+0000 7f2656d9d640 1 -- 172.21.15.107:0/124902622 learned_addr learned my addr 172.21.15.107:0/124902622 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f265659c640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 0x7f265815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2655d9b640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f265814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 -- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 msgr2=0x7f265814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f265814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 -- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 msgr2=0x7f265815d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 0x7f265815d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 -- 172.21.15.107:0/124902622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480094d0 con 0x7f265815b440 2023-11-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f265659c640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 0x7f265815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2656d9d640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f265000c7b0 tx=0x7f265000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650003d80 con 0x7f265815b440 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650005070 con 0x7f265815b440 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26580b4670 con 0x7f265815b440 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.366+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650005420 con 0x7f265815b440 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.367+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265815cc40 con 0x7f265815b440 2023-11-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.368+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265815d160 con 0x7f265815b440 2023-11-24T08:17:42.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.369+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f2650005580 con 0x7f265815b440 2023-11-24T08:17:42.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.370+0000 7f2646ffd640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 0x7f263806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:42.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.370+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2650087c50 con 0x7f265815b440 2023-11-24T08:17:42.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.370+0000 7f265659c640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 0x7f263806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:42.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.371+0000 7f265659c640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 0x7f263806ead0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2648036000 tx=0x7f264803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:42.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.374+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f265815d160 con 0x7f265815b440 2023-11-24T08:17:42.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:42.604+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f265815d120 con 0x7f265815b440 2023-11-24T08:17:43.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:17:43.079 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:17:43.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.269+0000 7f2646ffd640 1 -- 172.21.15.107:0/124902622 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2650017070 con 0x7f265815b440 2023-11-24T08:17:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 msgr2=0x7f263806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 0x7f263806ead0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2648036000 tx=0x7f264803d040 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 msgr2=0x7f26580c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f265000c7b0 tx=0x7f265000cc70 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 shutdown_connections 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f263806c610 0x7f263806ead0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f265815b440 0x7f26580c60d0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26580cbf20 0x7f265814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 --2- 172.21.15.107:0/124902622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26580c57e0 0x7f265815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.273+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 >> 172.21.15.107:0/124902622 conn(0x7f26580b0c50 msgr2=0x7f265806cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.274+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 shutdown_connections 2023-11-24T08:17:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.274+0000 7f265cd90640 1 -- 172.21.15.107:0/124902622 wait complete. 2023-11-24T08:17:43.296 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:17:43.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:17:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.505+0000 7f4bd5bac640 1 Processor -- start 2023-11-24T08:17:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.505+0000 7f4bd5bac640 1 -- start start 2023-11-24T08:17:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.506+0000 7f4bd5bac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bd0143440 0x7f4bd015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.506+0000 7f4bd5bac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 0x7f4bd015e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bd5bac640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd015e850 con 0x7f4bd00ce170 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bd5bac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd00d0de0 con 0x7f4bd0143440 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bd5bac640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd00d10c0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bd0143440 0x7f4bd015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bd0143440 0x7f4bd015ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36208/0 (socket says 172.21.15.107:36208) 2023-11-24T08:17:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 -- 172.21.15.107:0/964313245 learned_addr learned my addr 172.21.15.107:0/964313245 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 -- 172.21.15.107:0/964313245 >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 msgr2=0x7f4bd015e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bd53ab640 1 --2- 172.21.15.107:0/964313245 >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 0x7f4bd015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 --2- 172.21.15.107:0/964313245 >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 0x7f4bd015e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 -- 172.21.15.107:0/964313245 >> v1:172.21.15.120:6789/0 conn(0x7f4bd00ce170 legacy=0x7f4bd0142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bcffff640 1 -- 172.21.15.107:0/964313245 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00d13a0 con 0x7f4bd0143440 2023-11-24T08:17:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.507+0000 7f4bd53ab640 1 --2- 172.21.15.107:0/964313245 >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 0x7f4bd015e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.508+0000 7f4bcffff640 1 --2- 172.21.15.107:0/964313245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0143440 0x7f4bd015ddd0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0061b30 tx=0x7f4bbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc45e6f8b16a03e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.509+0000 7f4bcf7fe640 1 -- 172.21.15.107:0/964313245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0317e0 con 0x7f4bd0143440 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.509+0000 7f4bcf7fe640 1 -- 172.21.15.107:0/964313245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc031940 con 0x7f4bd0143440 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.509+0000 7f4bcf7fe640 1 -- 172.21.15.107:0/964313245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc031cd0 con 0x7f4bd0143440 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 -- 172.21.15.107:0/964313245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0143440 msgr2=0x7f4bd015ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/964313245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0143440 0x7f4bd015ddd0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0061b30 tx=0x7f4bbc02b880 comp rx=0 tx=0).stop 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 -- 172.21.15.107:0/964313245 shutdown_connections 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/964313245 >> v2:172.21.15.120:3300/0 conn(0x7f4bd00d1ba0 0x7f4bd015e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/964313245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0143440 0x7f4bd015ddd0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 -- 172.21.15.107:0/964313245 >> 172.21.15.107:0/964313245 conn(0x7f4bd0060f20 msgr2=0x7f4bd0061300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 -- 172.21.15.107:0/964313245 shutdown_connections 2023-11-24T08:17:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.510+0000 7f4bd5bac640 1 -- 172.21.15.107:0/964313245 wait complete. 2023-11-24T08:17:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.512+0000 7f4bd5bac640 1 Processor -- start 2023-11-24T08:17:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.512+0000 7f4bd5bac640 1 -- start start 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bd00ce170 0x7f4bd00ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 0x7f4bd015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c6420 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c6700 con 0x7f4bd0143440 2023-11-24T08:17:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd5bac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c69e0 con 0x7f4bd00ce170 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36214/0 (socket says 172.21.15.107:36214) 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 -- 172.21.15.107:0/1351003005 learned_addr learned my addr 172.21.15.107:0/1351003005 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd53ab640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 0x7f4bd015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd4baa640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bd00ce170 0x7f4bd00ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 -- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bd00ce170 msgr2=0x7f4bd00ad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bd00ce170 0x7f4bd00ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 -- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 msgr2=0x7f4bd015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 0x7f4bd015b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bcffff640 1 -- 172.21.15.107:0/1351003005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0094d0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.513+0000 7f4bd53ab640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 0x7f4bd015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bcffff640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0090b0 tx=0x7f4bbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc032940 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc032ac0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd00c6cc0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc032e70 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.514+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd00ad2a0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.515+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd004e840 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.516+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f4bbc04e450 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.517+0000 7f4bcd7fa640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 0x7f4bb406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.517+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bbc0aa520 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.518+0000 7f4bd4baa640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 0x7f4bb406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:43.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.518+0000 7f4bd4baa640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 0x7f4bb406eaa0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400a750 tx=0x7f4bc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.522+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bbc0771f0 con 0x7f4bd00d1ba0 2023-11-24T08:17:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:43.748+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4bd00cd810 con 0x7f4bd00d1ba0 2023-11-24T08:17:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.280+0000 7f4bcd7fa640 1 -- 172.21.15.107:0/1351003005 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4bbc07a810 con 0x7f4bd00d1ba0 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 msgr2=0x7f4bb406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 0x7f4bb406eaa0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400a750 tx=0x7f4bc4008040 comp rx=0 tx=0).stop 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 msgr2=0x7f4bd00acd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0090b0 tx=0x7f4bbc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 shutdown_connections 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4bb406c5e0 0x7f4bb406eaa0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bd0143440 0x7f4bd015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd00d1ba0 0x7f4bd00acd60 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 --2- 172.21.15.107:0/1351003005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bd00ce170 0x7f4bd00ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 >> 172.21.15.107:0/1351003005 conn(0x7f4bd0060f20 msgr2=0x7f4bd0160070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:44.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 shutdown_connections 2023-11-24T08:17:44.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.284+0000 7f4bd5bac640 1 -- 172.21.15.107:0/1351003005 wait complete. 2023-11-24T08:17:44.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:17:44.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.583+0000 7f0ca8890640 1 Processor -- start 2023-11-24T08:17:44.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.583+0000 7f0ca8890640 1 -- start start 2023-11-24T08:17:44.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca8890640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ca40cf450 0x7f0ca40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca8890640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40cece0 con 0x7f0ca415c7f0 2023-11-24T08:17:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca8890640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406d040 con 0x7f0ca40cdfb0 2023-11-24T08:17:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca8890640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40cfe40 con 0x7f0ca40cf450 2023-11-24T08:17:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0ca415c7f0 0x7f0ca406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33330/0 (socket says 172.21.15.107:33330) 2023-11-24T08:17:44.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0ca259c640 1 -- 172.21.15.107:0/3319242624 learned_addr learned my addr 172.21.15.107:0/3319242624 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:44.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.585+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/3319242624 >> v2:172.21.15.120:3300/0 conn(0x7f0ca40cf450 0x7f0ca40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751495609 0 0) 0x7f0ca40cece0 con 0x7f0ca415c7f0 2023-11-24T08:17:44.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c90003520 con 0x7f0ca415c7f0 2023-11-24T08:17:44.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274453591 0 0) 0x7f0c90003520 con 0x7f0ca415c7f0 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 >> v2:172.21.15.120:3300/0 conn(0x7f0ca40cf450 msgr2=0x7f0ca40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 --2- 172.21.15.107:0/3319242624 >> v2:172.21.15.120:3300/0 conn(0x7f0ca40cf450 0x7f0ca40cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 >> v1:172.21.15.120:6789/0 conn(0x7f0ca40cdfb0 legacy=0x7f0ca40ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca406d320 con 0x7f0ca415c7f0 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0c94001670 con 0x7f0ca415c7f0 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 >> v1:172.21.15.107:6789/0 conn(0x7f0ca415c7f0 legacy=0x7f0ca406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c90003970 0x7f0c90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0c90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c90005180 0x7f0c9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40cfe40 con 0x7f0c900047c0 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9000c020 con 0x7f0c90005180 2023-11-24T08:17:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9000c1e0 con 0x7f0c90003970 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.586+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0c90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0ca2d9d640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c90005180 0x7f0c9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0ca259c640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c90003970 0x7f0c90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0c9bfff640 1 -- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c90003970 msgr2=0x7f0c90003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c90003970 0x7f0c90003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0c9bfff640 1 -- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c90005180 msgr2=0x7f0c9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c90005180 0x7f0c9000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0c9bfff640 1 -- 172.21.15.107:0/3319242624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c90003520 con 0x7f0c900047c0 2023-11-24T08:17:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.587+0000 7f0ca2d9d640 1 -- 172.21.15.107:0/3319242624 reap_dead start 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.588+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0c90004c40 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009ac0 tx=0x7f0c900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c588c96819f904 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.588+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c02c6b0 con 0x7f0c900047c0 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.588+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c02c810 con 0x7f0c900047c0 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.588+0000 7f0ca1d9b640 1 -- 172.21.15.107:0/3319242624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c02cba0 con 0x7f0c900047c0 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 -- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 msgr2=0x7f0c90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0c90004c40 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009ac0 tx=0x7f0c900057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 -- 172.21.15.107:0/3319242624 shutdown_connections 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 --2- 172.21.15.107:0/3319242624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0c90004c40 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 -- 172.21.15.107:0/3319242624 >> 172.21.15.107:0/3319242624 conn(0x7f0ca406a920 msgr2=0x7f0ca406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 -- 172.21.15.107:0/3319242624 shutdown_connections 2023-11-24T08:17:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.589+0000 7f0ca8890640 1 -- 172.21.15.107:0/3319242624 wait complete. 2023-11-24T08:17:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.591+0000 7f0ca8890640 1 Processor -- start 2023-11-24T08:17:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.591+0000 7f0ca8890640 1 -- start start 2023-11-24T08:17:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca415ddd0 0x7f0ca415e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca41508a0 0x7f0ca4150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca415e790 con 0x7f0c900047c0 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4151b00 con 0x7f0ca415ddd0 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca8890640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40c8830 con 0x7f0ca41508a0 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36236/0 (socket says 172.21.15.107:36236) 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 -- 172.21.15.107:0/1755300740 learned_addr learned my addr 172.21.15.107:0/1755300740 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 -- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca41508a0 msgr2=0x7f0ca4150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca41508a0 0x7f0ca4150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 -- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca415ddd0 msgr2=0x7f0ca415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca415ddd0 0x7f0ca415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.592+0000 7f0ca259c640 1 -- 172.21.15.107:0/1755300740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca41499b0 con 0x7f0c900047c0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0ca259c640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400d8b0 tx=0x7f0c9400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940074e0 con 0x7f0c900047c0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c94007640 con 0x7f0c900047c0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940049e0 con 0x7f0c900047c0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca40c8500 con 0x7f0c900047c0 2023-11-24T08:17:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.593+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca4151820 con 0x7f0c900047c0 2023-11-24T08:17:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.594+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca40c8360 con 0x7f0c900047c0 2023-11-24T08:17:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.595+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f0c94004b40 con 0x7f0c900047c0 2023-11-24T08:17:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.596+0000 7f0c9affd640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 0x7f0c8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.596+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c94087e50 con 0x7f0c900047c0 2023-11-24T08:17:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.596+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 0x7f0c8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.597+0000 7f0c9bfff640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 0x7f0c8406eaa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.600+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca40c8360 con 0x7f0c900047c0 2023-11-24T08:17:44.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.826+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ca40c8360 con 0x7f0c900047c0 2023-11-24T08:17:44.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.827+0000 7f0c9affd640 1 -- 172.21.15.107:0/1755300740 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f0ca40c8360 con 0x7f0c900047c0 2023-11-24T08:17:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:44.830 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:17:44.830 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 97 2023-11-24T08:17:44.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 msgr2=0x7f0c8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 0x7f0c8406eaa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c02e000 comp rx=0 tx=0).stop 2023-11-24T08:17:44.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 msgr2=0x7f0ca4149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400d8b0 tx=0x7f0c9400dd70 comp rx=0 tx=0).stop 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 shutdown_connections 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c8406c5e0 0x7f0c8406eaa0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ca41508a0 0x7f0ca4150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca415ddd0 0x7f0ca415e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 --2- 172.21.15.107:0/1755300740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c900047c0 0x7f0ca4149470 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 >> 172.21.15.107:0/1755300740 conn(0x7f0ca406a920 msgr2=0x7f0ca415f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 shutdown_connections 2023-11-24T08:17:44.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:44.830+0000 7f0ca8890640 1 -- 172.21.15.107:0/1755300740 wait complete. 2023-11-24T08:17:44.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:17:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.050+0000 7fa922ee1640 1 Processor -- start 2023-11-24T08:17:45.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.050+0000 7fa922ee1640 1 -- start start 2023-11-24T08:17:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa922ee1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa91c15c410 0x7fa91c15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa922ee1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa922ee1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c15bf80 con 0x7fa91c144b30 2023-11-24T08:17:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa922ee1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0b27b0 con 0x7fa91c15c410 2023-11-24T08:17:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa922ee1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0b1560 con 0x7fa91c0b1de0 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa91c15c410 0x7fa91c15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa91c15c410 0x7fa91c15f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36240/0 (socket says 172.21.15.107:36240) 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 -- 172.21.15.107:0/1037778270 learned_addr learned my addr 172.21.15.107:0/1037778270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 -- 172.21.15.107:0/1037778270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 msgr2=0x7fa91c0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 --2- 172.21.15.107:0/1037778270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 -- 172.21.15.107:0/1037778270 >> v1:172.21.15.107:6789/0 conn(0x7fa91c144b30 legacy=0x7fa91c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.052+0000 7fa9216de640 1 -- 172.21.15.107:0/1037778270 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0b1280 con 0x7fa91c15c410 2023-11-24T08:17:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.053+0000 7fa9216de640 1 --2- 172.21.15.107:0/1037778270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c15c410 0x7fa91c15f920 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009870 tx=0x7fa90c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea9ad5d70c4bfb2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa920edd640 1 -- 172.21.15.107:0/1037778270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c030840 con 0x7fa91c15c410 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa920edd640 1 -- 172.21.15.107:0/1037778270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0309a0 con 0x7fa91c15c410 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa920edd640 1 -- 172.21.15.107:0/1037778270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c030d30 con 0x7fa91c15c410 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 -- 172.21.15.107:0/1037778270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c15c410 msgr2=0x7fa91c15f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 --2- 172.21.15.107:0/1037778270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c15c410 0x7fa91c15f920 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009870 tx=0x7fa90c02b980 comp rx=0 tx=0).stop 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 -- 172.21.15.107:0/1037778270 shutdown_connections 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 --2- 172.21.15.107:0/1037778270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c0b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 --2- 172.21.15.107:0/1037778270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c15c410 0x7fa91c15f920 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.054+0000 7fa922ee1640 1 -- 172.21.15.107:0/1037778270 >> 172.21.15.107:0/1037778270 conn(0x7fa91c0cb830 msgr2=0x7fa91c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.055+0000 7fa922ee1640 1 -- 172.21.15.107:0/1037778270 shutdown_connections 2023-11-24T08:17:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.055+0000 7fa922ee1640 1 -- 172.21.15.107:0/1037778270 wait complete. 2023-11-24T08:17:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.056+0000 7fa922ee1640 1 Processor -- start 2023-11-24T08:17:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.056+0000 7fa922ee1640 1 -- start start 2023-11-24T08:17:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c15a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c15c410 0x7fa91c0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0c3d50 con 0x7fa91c144b30 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0c4310 con 0x7fa91c15c410 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.057+0000 7fa922ee1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c1427b0 con 0x7fa91c0b1de0 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36246/0 (socket says 172.21.15.107:36246) 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 -- 172.21.15.107:0/2727102996 learned_addr learned my addr 172.21.15.107:0/2727102996 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa921edf640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c15a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 -- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 msgr2=0x7fa91c15a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9226e0640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c15c410 0x7fa91c0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c15a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 -- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c15c410 msgr2=0x7fa91c0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c15c410 0x7fa91c0c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 -- 172.21.15.107:0/2727102996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0094d0 con 0x7fa91c144b30 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa921edf640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c15a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.058+0000 7fa9216de640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0090b0 tx=0x7fa90c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.059+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c03e070 con 0x7fa91c144b30 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.059+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0307f0 con 0x7fa91c144b30 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.059+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039d40 con 0x7fa91c144b30 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.059+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c142a90 con 0x7fa91c144b30 2023-11-24T08:17:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.059+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c15ade0 con 0x7fa91c144b30 2023-11-24T08:17:45.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.060+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c067990 con 0x7fa91c144b30 2023-11-24T08:17:45.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.061+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fa90c042070 con 0x7fa91c144b30 2023-11-24T08:17:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.062+0000 7fa9127fc640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 0x7fa90006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.062+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa90c0aa1b0 con 0x7fa91c144b30 2023-11-24T08:17:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.063+0000 7fa921edf640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 0x7fa90006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.063+0000 7fa921edf640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 0x7fa90006eaa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa918000f60 tx=0x7fa918009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.067+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa90c076d60 con 0x7fa91c144b30 2023-11-24T08:17:45.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.245+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa91c142c50 con 0x7fa91c144b30 2023-11-24T08:17:45.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.248+0000 7fa9127fc640 1 -- 172.21.15.107:0/2727102996 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fa90c049090 con 0x7fa91c144b30 2023-11-24T08:17:45.250 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:45.251 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":162,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:17:42.255942+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:17:45.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.250+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 msgr2=0x7fa90006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.250+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 0x7fa90006eaa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa918000f60 tx=0x7fa918009040 comp rx=0 tx=0).stop 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 msgr2=0x7fa91c0c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0090b0 tx=0x7fa90c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 shutdown_connections 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa90006c5e0 0x7fa90006eaa0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c15c410 0x7fa91c0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa91c144b30 0x7fa91c0c47a0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 --2- 172.21.15.107:0/2727102996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa91c0b1de0 0x7fa91c15a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 >> 172.21.15.107:0/2727102996 conn(0x7fa91c0cb830 msgr2=0x7fa91c15dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 shutdown_connections 2023-11-24T08:17:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.251+0000 7fa922ee1640 1 -- 172.21.15.107:0/2727102996 wait complete. 2023-11-24T08:17:45.271 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:17:45.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:17:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.468+0000 7f4d4c602640 1 Processor -- start 2023-11-24T08:17:45.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.469+0000 7f4d4c602640 1 -- start start 2023-11-24T08:17:45.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.470+0000 7f4d4c602640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d440cf450 0x7f4d440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.470+0000 7f4d4c602640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440cece0 con 0x7f4d440cdfb0 2023-11-24T08:17:45.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.470+0000 7f4d4c602640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4406d040 con 0x7f4d4415c7f0 2023-11-24T08:17:45.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.470+0000 7f4d4c602640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440cfe40 con 0x7f4d440cf450 2023-11-24T08:17:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4ab9f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4d440cdfb0 0x7f4d440ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33350/0 (socket says 172.21.15.107:33350) 2023-11-24T08:17:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4ab9f640 1 -- 172.21.15.107:0/1900557622 learned_addr learned my addr 172.21.15.107:0/1900557622 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d49b9d640 1 --2- 172.21.15.107:0/1900557622 >> v2:172.21.15.120:3300/0 conn(0x7f4d440cf450 0x7f4d440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162959093 0 0) 0x7f4d440cece0 con 0x7f4d440cdfb0 2023-11-24T08:17:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d440cdfb0 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204490092 0 0) 0x7f4d30003520 con 0x7f4d440cdfb0 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 >> v2:172.21.15.120:3300/0 conn(0x7f4d440cf450 msgr2=0x7f4d440cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 --2- 172.21.15.107:0/1900557622 >> v2:172.21.15.120:3300/0 conn(0x7f4d440cf450 0x7f4d440cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 >> v1:172.21.15.120:6789/0 conn(0x7f4d4415c7f0 legacy=0x7f4d4406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4406d320 con 0x7f4d440cdfb0 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4d3c0019f0 con 0x7f4d440cdfb0 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 >> v1:172.21.15.107:6789/0 conn(0x7f4d440cdfb0 legacy=0x7f4d440ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.471+0000 7f4d4939c640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d30003970 0x7f4d30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4939c640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d300047c0 0x7f4d30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4939c640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440cfe40 con 0x7f4d30005180 2023-11-24T08:17:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c020 con 0x7f4d30003970 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c1e0 con 0x7f4d300047c0 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 -- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d300047c0 msgr2=0x7f4d30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d300047c0 0x7f4d30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 -- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d30003970 msgr2=0x7f4d30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4a39e640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d30003970 0x7f4d30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d30003970 0x7f4d30003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.472+0000 7f4d4ab9f640 1 -- 172.21.15.107:0/1900557622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30003520 con 0x7f4d30005180 2023-11-24T08:17:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.473+0000 7f4d4ab9f640 1 -- 172.21.15.107:0/1900557622 reap_dead start 2023-11-24T08:17:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.473+0000 7f4d4ab9f640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d3000b980 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00a830 tx=0x7f4d300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ef5b93510f6bf7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.473+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0072d0 con 0x7f4d30005180 2023-11-24T08:17:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.473+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c02f070 con 0x7f4d30005180 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.473+0000 7f4d4939c640 1 -- 172.21.15.107:0/1900557622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0076b0 con 0x7f4d30005180 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 -- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 msgr2=0x7f4d3000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d3000b980 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00a830 tx=0x7f4d300057f0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 -- 172.21.15.107:0/1900557622 shutdown_connections 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1900557622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d3000b980 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 -- 172.21.15.107:0/1900557622 >> 172.21.15.107:0/1900557622 conn(0x7f4d4406a920 msgr2=0x7f4d4406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 -- 172.21.15.107:0/1900557622 shutdown_connections 2023-11-24T08:17:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.474+0000 7f4d4c602640 1 -- 172.21.15.107:0/1900557622 wait complete. 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.476+0000 7f4d4c602640 1 Processor -- start 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.476+0000 7f4d4c602640 1 -- start start 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441493e0 0x7f4d4415a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4415ad70 0x7f4d440ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b2b80 con 0x7f4d30005180 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415b1f0 con 0x7f4d441493e0 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4c602640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b2270 con 0x7f4d4415ad70 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36266/0 (socket says 172.21.15.107:36266) 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 -- 172.21.15.107:0/1740160819 learned_addr learned my addr 172.21.15.107:0/1740160819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 -- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4415ad70 msgr2=0x7f4d440ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4415ad70 0x7f4d440ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 -- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441493e0 msgr2=0x7f4d4415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d49b9d640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441493e0 0x7f4d4415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.477+0000 7f4d4a39e640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441493e0 0x7f4d4415a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d4a39e640 1 -- 172.21.15.107:0/1740160819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44158d20 con 0x7f4d30005180 2023-11-24T08:17:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d4a39e640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000d4c0 tx=0x7f4d4000d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40005480 con 0x7f4d30005180 2023-11-24T08:17:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40003610 con 0x7f4d30005180 2023-11-24T08:17:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d44149ec0 con 0x7f4d30005180 2023-11-24T08:17:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400039a0 con 0x7f4d30005180 2023-11-24T08:17:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.478+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440b24c0 con 0x7f4d30005180 2023-11-24T08:17:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.479+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44067470 con 0x7f4d30005180 2023-11-24T08:17:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.480+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f4d40003b00 con 0x7f4d30005180 2023-11-24T08:17:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.481+0000 7f4d3affd640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 0x7f4d2806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.481+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d40088280 con 0x7f4d30005180 2023-11-24T08:17:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.481+0000 7f4d49b9d640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 0x7f4d2806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.482+0000 7f4d49b9d640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 0x7f4d2806eaa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4d4415b810 tx=0x7f4d34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.485+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d40054e50 con 0x7f4d30005180 2023-11-24T08:17:45.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.671+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d440cc850 con 0x7f4d2806c5e0 2023-11-24T08:17:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.694+0000 7f4d3affd640 1 -- 172.21.15.107:0/1740160819 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287420 (secure 0 0 0) 0x7f4d440cc850 con 0x7f4d2806c5e0 2023-11-24T08:17:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:17:45.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 msgr2=0x7f4d2806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 0x7f4d2806eaa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4d4415b810 tx=0x7f4d34008040 comp rx=0 tx=0).stop 2023-11-24T08:17:45.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 msgr2=0x7f4d441587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000d4c0 tx=0x7f4d4000d980 comp rx=0 tx=0).stop 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 shutdown_connections 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4d2806c5e0 0x7f4d2806eaa0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4415ad70 0x7f4d440ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441493e0 0x7f4d4415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 --2- 172.21.15.107:0/1740160819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d30005180 0x7f4d441587e0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.700+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 >> 172.21.15.107:0/1740160819 conn(0x7f4d4406a920 msgr2=0x7f4d4406c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.701+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 shutdown_connections 2023-11-24T08:17:45.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.701+0000 7f4d4c602640 1 -- 172.21.15.107:0/1740160819 wait complete. 2023-11-24T08:17:45.722 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2023-11-24T08:17:45.673544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":299780,"kb_used_data":17024,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754674940,"statfs":{"total":773094113280,"available":772787138560,"internally_reserved":0,"allocated":17432576,"data_stored":8302340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":289394238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001787"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217334+0000","last_change":"2023-11-24T08:17:41.278306+0000","last_active":"2023-11-24T08:17:43.217334+0000","last_peered":"2023-11-24T08:17:43.217334+0000","last_clean":"2023-11-24T08:17:43.217334+0000","last_became_active":"2023-11-24T08:17:41.276281+0000","last_became_peered":"2023-11-24T08:17:41.276281+0000","last_unstale":"2023-11-24T08:17:43.217334+0000","last_undegraded":"2023-11-24T08:17:43.217334+0000","last_fullsized":"2023-11-24T08:17:43.217334+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:13.172494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219008+0000","last_change":"2023-11-24T08:17:41.283094+0000","last_active":"2023-11-24T08:17:43.219008+0000","last_peered":"2023-11-24T08:17:43.219008+0000","last_clean":"2023-11-24T08:17:43.219008+0000","last_became_active":"2023-11-24T08:17:41.282784+0000","last_became_peered":"2023-11-24T08:17:41.282784+0000","last_unstale":"2023-11-24T08:17:43.219008+0000","last_undegraded":"2023-11-24T08:17:43.219008+0000","last_fullsized":"2023-11-24T08:17:43.219008+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:02.348205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276431+0000","last_change":"2023-11-24T08:17:41.277931+0000","last_active":"2023-11-24T08:17:42.276431+0000","last_peered":"2023-11-24T08:17:42.276431+0000","last_clean":"2023-11-24T08:17:42.276431+0000","last_became_active":"2023-11-24T08:17:41.277609+0000","last_became_peered":"2023-11-24T08:17:41.277609+0000","last_unstale":"2023-11-24T08:17:42.276431+0000","last_undegraded":"2023-11-24T08:17:42.276431+0000","last_fullsized":"2023-11-24T08:17:42.276431+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:46.144410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220836+0000","last_change":"2023-11-24T08:17:41.268672+0000","last_active":"2023-11-24T08:17:43.220836+0000","last_peered":"2023-11-24T08:17:43.220836+0000","last_clean":"2023-11-24T08:17:43.220836+0000","last_became_active":"2023-11-24T08:17:41.267916+0000","last_became_peered":"2023-11-24T08:17:41.267916+0000","last_unstale":"2023-11-24T08:17:43.220836+0000","last_undegraded":"2023-11-24T08:17:43.220836+0000","last_fullsized":"2023-11-24T08:17:43.220836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:32.835769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219320+0000","last_change":"2023-11-24T08:17:41.282966+0000","last_active":"2023-11-24T08:17:43.219320+0000","last_peered":"2023-11-24T08:17:43.219320+0000","last_clean":"2023-11-24T08:17:43.219320+0000","last_became_active":"2023-11-24T08:17:41.281861+0000","last_became_peered":"2023-11-24T08:17:41.281861+0000","last_unstale":"2023-11-24T08:17:43.219320+0000","last_undegraded":"2023-11-24T08:17:43.219320+0000","last_fullsized":"2023-11-24T08:17:43.219320+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:05.969690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219681+0000","last_change":"2023-11-24T08:17:41.282304+0000","last_active":"2023-11-24T08:17:43.219681+0000","last_peered":"2023-11-24T08:17:43.219681+0000","last_clean":"2023-11-24T08:17:43.219681+0000","last_became_active":"2023-11-24T08:17:41.276947+0000","last_became_peered":"2023-11-24T08:17:41.276947+0000","last_unstale":"2023-11-24T08:17:43.219681+0000","last_undegraded":"2023-11-24T08:17:43.219681+0000","last_fullsized":"2023-11-24T08:17:43.219681+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:02.011176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215693+0000","last_change":"2023-11-24T08:17:41.286873+0000","last_active":"2023-11-24T08:17:43.215693+0000","last_peered":"2023-11-24T08:17:43.215693+0000","last_clean":"2023-11-24T08:17:43.215693+0000","last_became_active":"2023-11-24T08:17:41.283757+0000","last_became_peered":"2023-11-24T08:17:41.283757+0000","last_unstale":"2023-11-24T08:17:43.215693+0000","last_undegraded":"2023-11-24T08:17:43.215693+0000","last_fullsized":"2023-11-24T08:17:43.215693+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:31.300677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218482+0000","last_change":"2023-11-24T08:17:41.279770+0000","last_active":"2023-11-24T08:17:43.218482+0000","last_peered":"2023-11-24T08:17:43.218482+0000","last_clean":"2023-11-24T08:17:43.218482+0000","last_became_active":"2023-11-24T08:17:41.278621+0000","last_became_peered":"2023-11-24T08:17:41.278621+0000","last_unstale":"2023-11-24T08:17:43.218482+0000","last_undegraded":"2023-11-24T08:17:43.218482+0000","last_fullsized":"2023-11-24T08:17:43.218482+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:37.614211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219614+0000","last_change":"2023-11-24T08:17:41.282377+0000","last_active":"2023-11-24T08:17:43.219614+0000","last_peered":"2023-11-24T08:17:43.219614+0000","last_clean":"2023-11-24T08:17:43.219614+0000","last_became_active":"2023-11-24T08:17:41.280764+0000","last_became_peered":"2023-11-24T08:17:41.280764+0000","last_unstale":"2023-11-24T08:17:43.219614+0000","last_undegraded":"2023-11-24T08:17:43.219614+0000","last_fullsized":"2023-11-24T08:17:43.219614+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219105+0000","last_change":"2023-11-24T08:17:41.272496+0000","last_active":"2023-11-24T08:17:43.219105+0000","last_peered":"2023-11-24T08:17:43.219105+0000","last_clean":"2023-11-24T08:17:43.219105+0000","last_became_active":"2023-11-24T08:17:41.272201+0000","last_became_peered":"2023-11-24T08:17:41.272201+0000","last_unstale":"2023-11-24T08:17:43.219105+0000","last_undegraded":"2023-11-24T08:17:43.219105+0000","last_fullsized":"2023-11-24T08:17:43.219105+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:55.017070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219011+0000","last_change":"2023-11-24T08:17:41.280150+0000","last_active":"2023-11-24T08:17:43.219011+0000","last_peered":"2023-11-24T08:17:43.219011+0000","last_clean":"2023-11-24T08:17:43.219011+0000","last_became_active":"2023-11-24T08:17:41.279171+0000","last_became_peered":"2023-11-24T08:17:41.279171+0000","last_unstale":"2023-11-24T08:17:43.219011+0000","last_undegraded":"2023-11-24T08:17:43.219011+0000","last_fullsized":"2023-11-24T08:17:43.219011+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:39.960967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218768+0000","last_change":"2023-11-24T08:17:41.281036+0000","last_active":"2023-11-24T08:17:43.218768+0000","last_peered":"2023-11-24T08:17:43.218768+0000","last_clean":"2023-11-24T08:17:43.218768+0000","last_became_active":"2023-11-24T08:17:41.279999+0000","last_became_peered":"2023-11-24T08:17:41.279999+0000","last_unstale":"2023-11-24T08:17:43.218768+0000","last_undegraded":"2023-11-24T08:17:43.218768+0000","last_fullsized":"2023-11-24T08:17:43.218768+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:36.836942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275484+0000","last_change":"2023-11-24T08:17:41.274253+0000","last_active":"2023-11-24T08:17:42.275484+0000","last_peered":"2023-11-24T08:17:42.275484+0000","last_clean":"2023-11-24T08:17:42.275484+0000","last_became_active":"2023-11-24T08:17:41.273958+0000","last_became_peered":"2023-11-24T08:17:41.273958+0000","last_unstale":"2023-11-24T08:17:42.275484+0000","last_undegraded":"2023-11-24T08:17:42.275484+0000","last_fullsized":"2023-11-24T08:17:42.275484+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:44:10.578605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274697+0000","last_change":"2023-11-24T08:17:41.272152+0000","last_active":"2023-11-24T08:17:42.274697+0000","last_peered":"2023-11-24T08:17:42.274697+0000","last_clean":"2023-11-24T08:17:42.274697+0000","last_became_active":"2023-11-24T08:17:41.271551+0000","last_became_peered":"2023-11-24T08:17:41.271551+0000","last_unstale":"2023-11-24T08:17:42.274697+0000","last_undegraded":"2023-11-24T08:17:42.274697+0000","last_fullsized":"2023-11-24T08:17:42.274697+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:07:04.816752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216933+0000","last_change":"2023-11-24T08:17:41.274580+0000","last_active":"2023-11-24T08:17:43.216933+0000","last_peered":"2023-11-24T08:17:43.216933+0000","last_clean":"2023-11-24T08:17:43.216933+0000","last_became_active":"2023-11-24T08:17:41.273155+0000","last_became_peered":"2023-11-24T08:17:41.273155+0000","last_unstale":"2023-11-24T08:17:43.216933+0000","last_undegraded":"2023-11-24T08:17:43.216933+0000","last_fullsized":"2023-11-24T08:17:43.216933+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:56:37.604937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215741+0000","last_change":"2023-11-24T08:17:41.278248+0000","last_active":"2023-11-24T08:17:43.215741+0000","last_peered":"2023-11-24T08:17:43.215741+0000","last_clean":"2023-11-24T08:17:43.215741+0000","last_became_active":"2023-11-24T08:17:41.277145+0000","last_became_peered":"2023-11-24T08:17:41.277145+0000","last_unstale":"2023-11-24T08:17:43.215741+0000","last_undegraded":"2023-11-24T08:17:43.215741+0000","last_fullsized":"2023-11-24T08:17:43.215741+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:47.493778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221337+0000","last_change":"2023-11-24T08:17:41.278710+0000","last_active":"2023-11-24T08:17:43.221337+0000","last_peered":"2023-11-24T08:17:43.221337+0000","last_clean":"2023-11-24T08:17:43.221337+0000","last_became_active":"2023-11-24T08:17:41.278444+0000","last_became_peered":"2023-11-24T08:17:41.278444+0000","last_unstale":"2023-11-24T08:17:43.221337+0000","last_undegraded":"2023-11-24T08:17:43.221337+0000","last_fullsized":"2023-11-24T08:17:43.221337+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:00:45.163228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218776+0000","last_change":"2023-11-24T08:17:41.271384+0000","last_active":"2023-11-24T08:17:43.218776+0000","last_peered":"2023-11-24T08:17:43.218776+0000","last_clean":"2023-11-24T08:17:43.218776+0000","last_became_active":"2023-11-24T08:17:41.268797+0000","last_became_peered":"2023-11-24T08:17:41.268797+0000","last_unstale":"2023-11-24T08:17:43.218776+0000","last_undegraded":"2023-11-24T08:17:43.218776+0000","last_fullsized":"2023-11-24T08:17:43.218776+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:42.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220467+0000","last_change":"2023-11-24T08:17:41.277705+0000","last_active":"2023-11-24T08:17:43.220467+0000","last_peered":"2023-11-24T08:17:43.220467+0000","last_clean":"2023-11-24T08:17:43.220467+0000","last_became_active":"2023-11-24T08:17:41.277049+0000","last_became_peered":"2023-11-24T08:17:41.277049+0000","last_unstale":"2023-11-24T08:17:43.220467+0000","last_undegraded":"2023-11-24T08:17:43.220467+0000","last_fullsized":"2023-11-24T08:17:43.220467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:21.903235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276077+0000","last_change":"2023-11-24T08:17:41.281322+0000","last_active":"2023-11-24T08:17:42.276077+0000","last_peered":"2023-11-24T08:17:42.276077+0000","last_clean":"2023-11-24T08:17:42.276077+0000","last_became_active":"2023-11-24T08:17:41.280414+0000","last_became_peered":"2023-11-24T08:17:41.280414+0000","last_unstale":"2023-11-24T08:17:42.276077+0000","last_undegraded":"2023-11-24T08:17:42.276077+0000","last_fullsized":"2023-11-24T08:17:42.276077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:23.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219812+0000","last_change":"2023-11-24T08:17:41.277945+0000","last_active":"2023-11-24T08:17:43.219812+0000","last_peered":"2023-11-24T08:17:43.219812+0000","last_clean":"2023-11-24T08:17:43.219812+0000","last_became_active":"2023-11-24T08:17:41.277690+0000","last_became_peered":"2023-11-24T08:17:41.277690+0000","last_unstale":"2023-11-24T08:17:43.219812+0000","last_undegraded":"2023-11-24T08:17:43.219812+0000","last_fullsized":"2023-11-24T08:17:43.219812+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:25:34.500836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218210+0000","last_change":"2023-11-24T08:17:41.274009+0000","last_active":"2023-11-24T08:17:43.218210+0000","last_peered":"2023-11-24T08:17:43.218210+0000","last_clean":"2023-11-24T08:17:43.218210+0000","last_became_active":"2023-11-24T08:17:41.273471+0000","last_became_peered":"2023-11-24T08:17:41.273471+0000","last_unstale":"2023-11-24T08:17:43.218210+0000","last_undegraded":"2023-11-24T08:17:43.218210+0000","last_fullsized":"2023-11-24T08:17:43.218210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:26.875145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219289+0000","last_change":"2023-11-24T08:17:41.274359+0000","last_active":"2023-11-24T08:17:43.219289+0000","last_peered":"2023-11-24T08:17:43.219289+0000","last_clean":"2023-11-24T08:17:43.219289+0000","last_became_active":"2023-11-24T08:17:41.274089+0000","last_became_peered":"2023-11-24T08:17:41.274089+0000","last_unstale":"2023-11-24T08:17:43.219289+0000","last_undegraded":"2023-11-24T08:17:43.219289+0000","last_fullsized":"2023-11-24T08:17:43.219289+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:58.402192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218208+0000","last_change":"2023-11-24T08:17:41.281157+0000","last_active":"2023-11-24T08:17:43.218208+0000","last_peered":"2023-11-24T08:17:43.218208+0000","last_clean":"2023-11-24T08:17:43.218208+0000","last_became_active":"2023-11-24T08:17:41.280841+0000","last_became_peered":"2023-11-24T08:17:41.280841+0000","last_unstale":"2023-11-24T08:17:43.218208+0000","last_undegraded":"2023-11-24T08:17:43.218208+0000","last_fullsized":"2023-11-24T08:17:43.218208+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:48.953924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277761+0000","last_change":"2023-11-24T08:17:41.274379+0000","last_active":"2023-11-24T08:17:42.277761+0000","last_peered":"2023-11-24T08:17:42.277761+0000","last_clean":"2023-11-24T08:17:42.277761+0000","last_became_active":"2023-11-24T08:17:41.273291+0000","last_became_peered":"2023-11-24T08:17:41.273291+0000","last_unstale":"2023-11-24T08:17:42.277761+0000","last_undegraded":"2023-11-24T08:17:42.277761+0000","last_fullsized":"2023-11-24T08:17:42.277761+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:11.968663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216694+0000","last_change":"2023-11-24T08:17:41.289045+0000","last_active":"2023-11-24T08:17:43.216694+0000","last_peered":"2023-11-24T08:17:43.216694+0000","last_clean":"2023-11-24T08:17:43.216694+0000","last_became_active":"2023-11-24T08:17:41.288003+0000","last_became_peered":"2023-11-24T08:17:41.288003+0000","last_unstale":"2023-11-24T08:17:43.216694+0000","last_undegraded":"2023-11-24T08:17:43.216694+0000","last_fullsized":"2023-11-24T08:17:43.216694+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:53.416104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276606+0000","last_change":"2023-11-24T08:17:41.275711+0000","last_active":"2023-11-24T08:17:42.276606+0000","last_peered":"2023-11-24T08:17:42.276606+0000","last_clean":"2023-11-24T08:17:42.276606+0000","last_became_active":"2023-11-24T08:17:41.275431+0000","last_became_peered":"2023-11-24T08:17:41.275431+0000","last_unstale":"2023-11-24T08:17:42.276606+0000","last_undegraded":"2023-11-24T08:17:42.276606+0000","last_fullsized":"2023-11-24T08:17:42.276606+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:28.931916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221242+0000","last_change":"2023-11-24T08:17:41.284218+0000","last_active":"2023-11-24T08:17:43.221242+0000","last_peered":"2023-11-24T08:17:43.221242+0000","last_clean":"2023-11-24T08:17:43.221242+0000","last_became_active":"2023-11-24T08:17:41.283176+0000","last_became_peered":"2023-11-24T08:17:41.283176+0000","last_unstale":"2023-11-24T08:17:43.221242+0000","last_undegraded":"2023-11-24T08:17:43.221242+0000","last_fullsized":"2023-11-24T08:17:43.221242+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:37.776952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275711+0000","last_change":"2023-11-24T08:17:41.275576+0000","last_active":"2023-11-24T08:17:42.275711+0000","last_peered":"2023-11-24T08:17:42.275711+0000","last_clean":"2023-11-24T08:17:42.275711+0000","last_became_active":"2023-11-24T08:17:41.275285+0000","last_became_peered":"2023-11-24T08:17:41.275285+0000","last_unstale":"2023-11-24T08:17:42.275711+0000","last_undegraded":"2023-11-24T08:17:42.275711+0000","last_fullsized":"2023-11-24T08:17:42.275711+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:14.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275222+0000","last_change":"2023-11-24T08:17:41.268935+0000","last_active":"2023-11-24T08:17:42.275222+0000","last_peered":"2023-11-24T08:17:42.275222+0000","last_clean":"2023-11-24T08:17:42.275222+0000","last_became_active":"2023-11-24T08:17:41.268428+0000","last_became_peered":"2023-11-24T08:17:41.268428+0000","last_unstale":"2023-11-24T08:17:42.275222+0000","last_undegraded":"2023-11-24T08:17:42.275222+0000","last_fullsized":"2023-11-24T08:17:42.275222+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275862+0000","last_change":"2023-11-24T08:17:41.275641+0000","last_active":"2023-11-24T08:17:42.275862+0000","last_peered":"2023-11-24T08:17:42.275862+0000","last_clean":"2023-11-24T08:17:42.275862+0000","last_became_active":"2023-11-24T08:17:41.275347+0000","last_became_peered":"2023-11-24T08:17:41.275347+0000","last_unstale":"2023-11-24T08:17:42.275862+0000","last_undegraded":"2023-11-24T08:17:42.275862+0000","last_fullsized":"2023-11-24T08:17:42.275862+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:16.806161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218054+0000","last_change":"2023-11-24T08:17:41.265592+0000","last_active":"2023-11-24T08:17:43.218054+0000","last_peered":"2023-11-24T08:17:43.218054+0000","last_clean":"2023-11-24T08:17:43.218054+0000","last_became_active":"2023-11-24T08:17:41.265173+0000","last_became_peered":"2023-11-24T08:17:41.265173+0000","last_unstale":"2023-11-24T08:17:43.218054+0000","last_undegraded":"2023-11-24T08:17:43.218054+0000","last_fullsized":"2023-11-24T08:17:43.218054+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:46.526325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216744+0000","last_change":"2023-11-24T08:17:41.288145+0000","last_active":"2023-11-24T08:17:43.216744+0000","last_peered":"2023-11-24T08:17:43.216744+0000","last_clean":"2023-11-24T08:17:43.216744+0000","last_became_active":"2023-11-24T08:17:41.287654+0000","last_became_peered":"2023-11-24T08:17:41.287654+0000","last_unstale":"2023-11-24T08:17:43.216744+0000","last_undegraded":"2023-11-24T08:17:43.216744+0000","last_fullsized":"2023-11-24T08:17:43.216744+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:18.585408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"161'4","reported_seq":16,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217473+0000","last_change":"2023-11-24T08:17:40.274270+0000","last_active":"2023-11-24T08:17:43.217473+0000","last_peered":"2023-11-24T08:17:43.217473+0000","last_clean":"2023-11-24T08:17:43.217473+0000","last_became_active":"2023-11-24T08:17:40.274039+0000","last_became_peered":"2023-11-24T08:17:40.274039+0000","last_unstale":"2023-11-24T08:17:43.217473+0000","last_undegraded":"2023-11-24T08:17:43.217473+0000","last_fullsized":"2023-11-24T08:17:43.217473+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:44:52.323526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277764+0000","last_change":"2023-11-24T08:17:41.261553+0000","last_active":"2023-11-24T08:17:42.277764+0000","last_peered":"2023-11-24T08:17:42.277764+0000","last_clean":"2023-11-24T08:17:42.277764+0000","last_became_active":"2023-11-24T08:17:41.260308+0000","last_became_peered":"2023-11-24T08:17:41.260308+0000","last_unstale":"2023-11-24T08:17:42.277764+0000","last_undegraded":"2023-11-24T08:17:42.277764+0000","last_fullsized":"2023-11-24T08:17:42.277764+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:32.835769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.278023+0000","last_change":"2023-11-24T08:17:40.264001+0000","last_active":"2023-11-24T08:17:42.278023+0000","last_peered":"2023-11-24T08:17:42.278023+0000","last_clean":"2023-11-24T08:17:42.278023+0000","last_became_active":"2023-11-24T08:17:40.263573+0000","last_became_peered":"2023-11-24T08:17:40.263573+0000","last_unstale":"2023-11-24T08:17:42.278023+0000","last_undegraded":"2023-11-24T08:17:42.278023+0000","last_fullsized":"2023-11-24T08:17:42.278023+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:38.944147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219939+0000","last_change":"2023-11-24T08:17:41.280993+0000","last_active":"2023-11-24T08:17:43.219939+0000","last_peered":"2023-11-24T08:17:43.219939+0000","last_clean":"2023-11-24T08:17:43.219939+0000","last_became_active":"2023-11-24T08:17:41.280581+0000","last_became_peered":"2023-11-24T08:17:41.280581+0000","last_unstale":"2023-11-24T08:17:43.219939+0000","last_undegraded":"2023-11-24T08:17:43.219939+0000","last_fullsized":"2023-11-24T08:17:43.219939+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:02.387416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219666+0000","last_change":"2023-11-24T08:17:40.268530+0000","last_active":"2023-11-24T08:17:43.219666+0000","last_peered":"2023-11-24T08:17:43.219666+0000","last_clean":"2023-11-24T08:17:43.219666+0000","last_became_active":"2023-11-24T08:17:40.267997+0000","last_became_peered":"2023-11-24T08:17:40.267997+0000","last_unstale":"2023-11-24T08:17:43.219666+0000","last_undegraded":"2023-11-24T08:17:43.219666+0000","last_fullsized":"2023-11-24T08:17:43.219666+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:47.937104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276783+0000","last_change":"2023-11-24T08:17:41.275182+0000","last_active":"2023-11-24T08:17:42.276783+0000","last_peered":"2023-11-24T08:17:42.276783+0000","last_clean":"2023-11-24T08:17:42.276783+0000","last_became_active":"2023-11-24T08:17:41.273716+0000","last_became_peered":"2023-11-24T08:17:41.273716+0000","last_unstale":"2023-11-24T08:17:42.276783+0000","last_undegraded":"2023-11-24T08:17:42.276783+0000","last_fullsized":"2023-11-24T08:17:42.276783+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:07:04.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221369+0000","last_change":"2023-11-24T08:17:40.266226+0000","last_active":"2023-11-24T08:17:43.221369+0000","last_peered":"2023-11-24T08:17:43.221369+0000","last_clean":"2023-11-24T08:17:43.221369+0000","last_became_active":"2023-11-24T08:17:40.265926+0000","last_became_peered":"2023-11-24T08:17:40.265926+0000","last_unstale":"2023-11-24T08:17:43.221369+0000","last_undegraded":"2023-11-24T08:17:43.221369+0000","last_fullsized":"2023-11-24T08:17:43.221369+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:38.944147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216819+0000","last_change":"2023-11-24T08:17:41.286252+0000","last_active":"2023-11-24T08:17:43.216819+0000","last_peered":"2023-11-24T08:17:43.216819+0000","last_clean":"2023-11-24T08:17:43.216819+0000","last_became_active":"2023-11-24T08:17:41.285964+0000","last_became_peered":"2023-11-24T08:17:41.285964+0000","last_unstale":"2023-11-24T08:17:43.216819+0000","last_undegraded":"2023-11-24T08:17:43.216819+0000","last_fullsized":"2023-11-24T08:17:43.216819+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:36:38.121400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275689+0000","last_change":"2023-11-24T08:17:40.267090+0000","last_active":"2023-11-24T08:17:42.275689+0000","last_peered":"2023-11-24T08:17:42.275689+0000","last_clean":"2023-11-24T08:17:42.275689+0000","last_became_active":"2023-11-24T08:17:40.266810+0000","last_became_peered":"2023-11-24T08:17:40.266810+0000","last_unstale":"2023-11-24T08:17:42.275689+0000","last_undegraded":"2023-11-24T08:17:42.275689+0000","last_fullsized":"2023-11-24T08:17:42.275689+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:20.886415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219835+0000","last_change":"2023-11-24T08:17:41.272717+0000","last_active":"2023-11-24T08:17:43.219835+0000","last_peered":"2023-11-24T08:17:43.219835+0000","last_clean":"2023-11-24T08:17:43.219835+0000","last_became_active":"2023-11-24T08:17:41.272016+0000","last_became_peered":"2023-11-24T08:17:41.272016+0000","last_unstale":"2023-11-24T08:17:43.219835+0000","last_undegraded":"2023-11-24T08:17:43.219835+0000","last_fullsized":"2023-11-24T08:17:43.219835+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:00.248461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218285+0000","last_change":"2023-11-24T08:17:40.264261+0000","last_active":"2023-11-24T08:17:43.218285+0000","last_peered":"2023-11-24T08:17:43.218285+0000","last_clean":"2023-11-24T08:17:43.218285+0000","last_became_active":"2023-11-24T08:17:40.263850+0000","last_became_peered":"2023-11-24T08:17:40.263850+0000","last_unstale":"2023-11-24T08:17:43.218285+0000","last_undegraded":"2023-11-24T08:17:43.218285+0000","last_fullsized":"2023-11-24T08:17:43.218285+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:35.820122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217323+0000","last_change":"2023-11-24T08:17:41.274369+0000","last_active":"2023-11-24T08:17:43.217323+0000","last_peered":"2023-11-24T08:17:43.217323+0000","last_clean":"2023-11-24T08:17:43.217323+0000","last_became_active":"2023-11-24T08:17:41.273693+0000","last_became_peered":"2023-11-24T08:17:41.273693+0000","last_unstale":"2023-11-24T08:17:43.217323+0000","last_undegraded":"2023-11-24T08:17:43.217323+0000","last_fullsized":"2023-11-24T08:17:43.217323+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218133+0000","last_change":"2023-11-24T08:17:40.266986+0000","last_active":"2023-11-24T08:17:43.218133+0000","last_peered":"2023-11-24T08:17:43.218133+0000","last_clean":"2023-11-24T08:17:43.218133+0000","last_became_active":"2023-11-24T08:17:40.266606+0000","last_became_peered":"2023-11-24T08:17:40.266606+0000","last_unstale":"2023-11-24T08:17:43.218133+0000","last_undegraded":"2023-11-24T08:17:43.218133+0000","last_fullsized":"2023-11-24T08:17:43.218133+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:45.127590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219301+0000","last_change":"2023-11-24T08:17:41.279975+0000","last_active":"2023-11-24T08:17:43.219301+0000","last_peered":"2023-11-24T08:17:43.219301+0000","last_clean":"2023-11-24T08:17:43.219301+0000","last_became_active":"2023-11-24T08:17:41.279469+0000","last_became_peered":"2023-11-24T08:17:41.279469+0000","last_unstale":"2023-11-24T08:17:43.219301+0000","last_undegraded":"2023-11-24T08:17:43.219301+0000","last_fullsized":"2023-11-24T08:17:43.219301+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:43.843570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215999+0000","last_change":"2023-11-24T08:17:40.260966+0000","last_active":"2023-11-24T08:17:43.215999+0000","last_peered":"2023-11-24T08:17:43.215999+0000","last_clean":"2023-11-24T08:17:43.215999+0000","last_became_active":"2023-11-24T08:17:40.260688+0000","last_became_peered":"2023-11-24T08:17:40.260688+0000","last_unstale":"2023-11-24T08:17:43.215999+0000","last_undegraded":"2023-11-24T08:17:43.215999+0000","last_fullsized":"2023-11-24T08:17:43.215999+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:14:16.120418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218933+0000","last_change":"2023-11-24T08:17:41.280361+0000","last_active":"2023-11-24T08:17:43.218933+0000","last_peered":"2023-11-24T08:17:43.218933+0000","last_clean":"2023-11-24T08:17:43.218933+0000","last_became_active":"2023-11-24T08:17:41.275998+0000","last_became_peered":"2023-11-24T08:17:41.275998+0000","last_unstale":"2023-11-24T08:17:43.218933+0000","last_undegraded":"2023-11-24T08:17:43.218933+0000","last_fullsized":"2023-11-24T08:17:43.218933+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:26.366513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220973+0000","last_change":"2023-11-24T08:17:40.264680+0000","last_active":"2023-11-24T08:17:43.220973+0000","last_peered":"2023-11-24T08:17:43.220973+0000","last_clean":"2023-11-24T08:17:43.220973+0000","last_became_active":"2023-11-24T08:17:40.264406+0000","last_became_peered":"2023-11-24T08:17:40.264406+0000","last_unstale":"2023-11-24T08:17:43.220973+0000","last_undegraded":"2023-11-24T08:17:43.220973+0000","last_fullsized":"2023-11-24T08:17:43.220973+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:03:41.552001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218210+0000","last_change":"2023-11-24T08:17:41.273748+0000","last_active":"2023-11-24T08:17:43.218210+0000","last_peered":"2023-11-24T08:17:43.218210+0000","last_clean":"2023-11-24T08:17:43.218210+0000","last_became_active":"2023-11-24T08:17:41.273418+0000","last_became_peered":"2023-11-24T08:17:41.273418+0000","last_unstale":"2023-11-24T08:17:43.218210+0000","last_undegraded":"2023-11-24T08:17:43.218210+0000","last_fullsized":"2023-11-24T08:17:43.218210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:44:53.340346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"161'3","reported_seq":15,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220441+0000","last_change":"2023-11-24T08:17:40.263032+0000","last_active":"2023-11-24T08:17:43.220441+0000","last_peered":"2023-11-24T08:17:43.220441+0000","last_clean":"2023-11-24T08:17:43.220441+0000","last_became_active":"2023-11-24T08:17:40.262622+0000","last_became_peered":"2023-11-24T08:17:40.262622+0000","last_unstale":"2023-11-24T08:17:43.220441+0000","last_undegraded":"2023-11-24T08:17:43.220441+0000","last_fullsized":"2023-11-24T08:17:43.220441+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:52.599545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.278430+0000","last_change":"2023-11-24T08:17:41.275011+0000","last_active":"2023-11-24T08:17:42.278430+0000","last_peered":"2023-11-24T08:17:42.278430+0000","last_clean":"2023-11-24T08:17:42.278430+0000","last_became_active":"2023-11-24T08:17:41.274717+0000","last_became_peered":"2023-11-24T08:17:41.274717+0000","last_unstale":"2023-11-24T08:17:42.278430+0000","last_undegraded":"2023-11-24T08:17:42.278430+0000","last_fullsized":"2023-11-24T08:17:42.278430+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:13.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"161'6","reported_seq":18,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277235+0000","last_change":"2023-11-24T08:17:40.267331+0000","last_active":"2023-11-24T08:17:42.277235+0000","last_peered":"2023-11-24T08:17:42.277235+0000","last_clean":"2023-11-24T08:17:42.277235+0000","last_became_active":"2023-11-24T08:17:40.265931+0000","last_became_peered":"2023-11-24T08:17:40.265931+0000","last_unstale":"2023-11-24T08:17:42.277235+0000","last_undegraded":"2023-11-24T08:17:42.277235+0000","last_fullsized":"2023-11-24T08:17:42.277235+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:35.820122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218077+0000","last_change":"2023-11-24T08:17:41.273218+0000","last_active":"2023-11-24T08:17:43.218077+0000","last_peered":"2023-11-24T08:17:43.218077+0000","last_clean":"2023-11-24T08:17:43.218077+0000","last_became_active":"2023-11-24T08:17:41.272827+0000","last_became_peered":"2023-11-24T08:17:41.272827+0000","last_unstale":"2023-11-24T08:17:43.218077+0000","last_undegraded":"2023-11-24T08:17:43.218077+0000","last_fullsized":"2023-11-24T08:17:43.218077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:29:05.900760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219942+0000","last_change":"2023-11-24T08:17:40.262554+0000","last_active":"2023-11-24T08:17:43.219942+0000","last_peered":"2023-11-24T08:17:43.219942+0000","last_clean":"2023-11-24T08:17:43.219942+0000","last_became_active":"2023-11-24T08:17:40.260612+0000","last_became_peered":"2023-11-24T08:17:40.260612+0000","last_unstale":"2023-11-24T08:17:43.219942+0000","last_undegraded":"2023-11-24T08:17:43.219942+0000","last_fullsized":"2023-11-24T08:17:43.219942+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:25.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216642+0000","last_change":"2023-11-24T08:17:41.279064+0000","last_active":"2023-11-24T08:17:43.216642+0000","last_peered":"2023-11-24T08:17:43.216642+0000","last_clean":"2023-11-24T08:17:43.216642+0000","last_became_active":"2023-11-24T08:17:41.278709+0000","last_became_peered":"2023-11-24T08:17:41.278709+0000","last_unstale":"2023-11-24T08:17:43.216642+0000","last_undegraded":"2023-11-24T08:17:43.216642+0000","last_fullsized":"2023-11-24T08:17:43.216642+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:47:14.293972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218494+0000","last_change":"2023-11-24T08:17:40.261755+0000","last_active":"2023-11-24T08:17:43.218494+0000","last_peered":"2023-11-24T08:17:43.218494+0000","last_clean":"2023-11-24T08:17:43.218494+0000","last_became_active":"2023-11-24T08:17:40.261454+0000","last_became_peered":"2023-11-24T08:17:40.261454+0000","last_unstale":"2023-11-24T08:17:43.218494+0000","last_undegraded":"2023-11-24T08:17:43.218494+0000","last_fullsized":"2023-11-24T08:17:43.218494+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:53.736584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217692+0000","last_change":"2023-11-24T08:17:41.274521+0000","last_active":"2023-11-24T08:17:43.217692+0000","last_peered":"2023-11-24T08:17:43.217692+0000","last_clean":"2023-11-24T08:17:43.217692+0000","last_became_active":"2023-11-24T08:17:41.274072+0000","last_became_peered":"2023-11-24T08:17:41.274072+0000","last_unstale":"2023-11-24T08:17:43.217692+0000","last_undegraded":"2023-11-24T08:17:43.217692+0000","last_fullsized":"2023-11-24T08:17:43.217692+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:16.806161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218288+0000","last_change":"2023-11-24T08:17:40.268091+0000","last_active":"2023-11-24T08:17:43.218288+0000","last_peered":"2023-11-24T08:17:43.218288+0000","last_clean":"2023-11-24T08:17:43.218288+0000","last_became_active":"2023-11-24T08:17:40.267732+0000","last_became_peered":"2023-11-24T08:17:40.267732+0000","last_unstale":"2023-11-24T08:17:43.218288+0000","last_undegraded":"2023-11-24T08:17:43.218288+0000","last_fullsized":"2023-11-24T08:17:43.218288+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:04.952870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274568+0000","last_change":"2023-11-24T08:17:41.271855+0000","last_active":"2023-11-24T08:17:42.274568+0000","last_peered":"2023-11-24T08:17:42.274568+0000","last_clean":"2023-11-24T08:17:42.274568+0000","last_became_active":"2023-11-24T08:17:41.271560+0000","last_became_peered":"2023-11-24T08:17:41.271560+0000","last_unstale":"2023-11-24T08:17:42.274568+0000","last_undegraded":"2023-11-24T08:17:42.274568+0000","last_fullsized":"2023-11-24T08:17:42.274568+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:25.595604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"161'8","reported_seq":20,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275084+0000","last_change":"2023-11-24T08:17:40.267502+0000","last_active":"2023-11-24T08:17:42.275084+0000","last_peered":"2023-11-24T08:17:42.275084+0000","last_clean":"2023-11-24T08:17:42.275084+0000","last_became_active":"2023-11-24T08:17:40.267104+0000","last_became_peered":"2023-11-24T08:17:40.267104+0000","last_unstale":"2023-11-24T08:17:42.275084+0000","last_undegraded":"2023-11-24T08:17:42.275084+0000","last_fullsized":"2023-11-24T08:17:42.275084+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:10.951843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220844+0000","last_change":"2023-11-24T08:17:41.273337+0000","last_active":"2023-11-24T08:17:43.220844+0000","last_peered":"2023-11-24T08:17:43.220844+0000","last_clean":"2023-11-24T08:17:43.220844+0000","last_became_active":"2023-11-24T08:17:41.273017+0000","last_became_peered":"2023-11-24T08:17:41.273017+0000","last_unstale":"2023-11-24T08:17:43.220844+0000","last_undegraded":"2023-11-24T08:17:43.220844+0000","last_fullsized":"2023-11-24T08:17:43.220844+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:13.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277815+0000","last_change":"2023-11-24T08:17:40.268858+0000","last_active":"2023-11-24T08:17:42.277815+0000","last_peered":"2023-11-24T08:17:42.277815+0000","last_clean":"2023-11-24T08:17:42.277815+0000","last_became_active":"2023-11-24T08:17:40.268026+0000","last_became_peered":"2023-11-24T08:17:40.268026+0000","last_unstale":"2023-11-24T08:17:42.277815+0000","last_undegraded":"2023-11-24T08:17:42.277815+0000","last_fullsized":"2023-11-24T08:17:42.277815+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:38.502188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219132+0000","last_change":"2023-11-24T08:17:41.257281+0000","last_active":"2023-11-24T08:17:43.219132+0000","last_peered":"2023-11-24T08:17:43.219132+0000","last_clean":"2023-11-24T08:17:43.219132+0000","last_became_active":"2023-11-24T08:17:41.256350+0000","last_became_peered":"2023-11-24T08:17:41.256350+0000","last_unstale":"2023-11-24T08:17:43.219132+0000","last_undegraded":"2023-11-24T08:17:43.219132+0000","last_fullsized":"2023-11-24T08:17:43.219132+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:21.357162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276320+0000","last_change":"2023-11-24T08:17:40.261216+0000","last_active":"2023-11-24T08:17:42.276320+0000","last_peered":"2023-11-24T08:17:42.276320+0000","last_clean":"2023-11-24T08:17:42.276320+0000","last_became_active":"2023-11-24T08:17:40.260897+0000","last_became_peered":"2023-11-24T08:17:40.260897+0000","last_unstale":"2023-11-24T08:17:42.276320+0000","last_undegraded":"2023-11-24T08:17:42.276320+0000","last_fullsized":"2023-11-24T08:17:42.276320+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:50.771442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221174+0000","last_change":"2023-11-24T08:17:41.274336+0000","last_active":"2023-11-24T08:17:43.221174+0000","last_peered":"2023-11-24T08:17:43.221174+0000","last_clean":"2023-11-24T08:17:43.221174+0000","last_became_active":"2023-11-24T08:17:41.273028+0000","last_became_peered":"2023-11-24T08:17:41.273028+0000","last_unstale":"2023-11-24T08:17:43.221174+0000","last_undegraded":"2023-11-24T08:17:43.221174+0000","last_fullsized":"2023-11-24T08:17:43.221174+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:01:41.929632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217250+0000","last_change":"2023-11-24T08:17:40.269351+0000","last_active":"2023-11-24T08:17:43.217250+0000","last_peered":"2023-11-24T08:17:43.217250+0000","last_clean":"2023-11-24T08:17:43.217250+0000","last_became_active":"2023-11-24T08:17:40.268930+0000","last_became_peered":"2023-11-24T08:17:40.268930+0000","last_unstale":"2023-11-24T08:17:43.217250+0000","last_undegraded":"2023-11-24T08:17:43.217250+0000","last_fullsized":"2023-11-24T08:17:43.217250+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:27.968787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216522+0000","last_change":"2023-11-24T08:17:41.284621+0000","last_active":"2023-11-24T08:17:43.216522+0000","last_peered":"2023-11-24T08:17:43.216522+0000","last_clean":"2023-11-24T08:17:43.216522+0000","last_became_active":"2023-11-24T08:17:41.282874+0000","last_became_peered":"2023-11-24T08:17:41.282874+0000","last_unstale":"2023-11-24T08:17:43.216522+0000","last_undegraded":"2023-11-24T08:17:43.216522+0000","last_fullsized":"2023-11-24T08:17:43.216522+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:46:29.875536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217244+0000","last_change":"2023-11-24T08:17:40.268661+0000","last_active":"2023-11-24T08:17:43.217244+0000","last_peered":"2023-11-24T08:17:43.217244+0000","last_clean":"2023-11-24T08:17:43.217244+0000","last_became_active":"2023-11-24T08:17:40.268296+0000","last_became_peered":"2023-11-24T08:17:40.268296+0000","last_unstale":"2023-11-24T08:17:43.217244+0000","last_undegraded":"2023-11-24T08:17:43.217244+0000","last_fullsized":"2023-11-24T08:17:43.217244+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:37.201642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275210+0000","last_change":"2023-11-24T08:17:41.271880+0000","last_active":"2023-11-24T08:17:42.275210+0000","last_peered":"2023-11-24T08:17:42.275210+0000","last_clean":"2023-11-24T08:17:42.275210+0000","last_became_active":"2023-11-24T08:17:41.270914+0000","last_became_peered":"2023-11-24T08:17:41.270914+0000","last_unstale":"2023-11-24T08:17:42.275210+0000","last_undegraded":"2023-11-24T08:17:42.275210+0000","last_fullsized":"2023-11-24T08:17:42.275210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:36.806072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219964+0000","last_change":"2023-11-24T08:17:40.266769+0000","last_active":"2023-11-24T08:17:43.219964+0000","last_peered":"2023-11-24T08:17:43.219964+0000","last_clean":"2023-11-24T08:17:43.219964+0000","last_became_active":"2023-11-24T08:17:40.266312+0000","last_became_peered":"2023-11-24T08:17:40.266312+0000","last_unstale":"2023-11-24T08:17:43.219964+0000","last_undegraded":"2023-11-24T08:17:43.219964+0000","last_fullsized":"2023-11-24T08:17:43.219964+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:36.597391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219141+0000","last_change":"2023-11-24T08:17:41.285206+0000","last_active":"2023-11-24T08:17:43.219141+0000","last_peered":"2023-11-24T08:17:43.219141+0000","last_clean":"2023-11-24T08:17:43.219141+0000","last_became_active":"2023-11-24T08:17:41.282438+0000","last_became_peered":"2023-11-24T08:17:41.282438+0000","last_unstale":"2023-11-24T08:17:43.219141+0000","last_undegraded":"2023-11-24T08:17:43.219141+0000","last_fullsized":"2023-11-24T08:17:43.219141+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:05.238469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216765+0000","last_change":"2023-11-24T08:17:40.268374+0000","last_active":"2023-11-24T08:17:43.216765+0000","last_peered":"2023-11-24T08:17:43.216765+0000","last_clean":"2023-11-24T08:17:43.216765+0000","last_became_active":"2023-11-24T08:17:40.268078+0000","last_became_peered":"2023-11-24T08:17:40.268078+0000","last_unstale":"2023-11-24T08:17:43.216765+0000","last_undegraded":"2023-11-24T08:17:43.216765+0000","last_fullsized":"2023-11-24T08:17:43.216765+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:46.674445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217622+0000","last_change":"2023-11-24T08:17:41.272699+0000","last_active":"2023-11-24T08:17:43.217622+0000","last_peered":"2023-11-24T08:17:43.217622+0000","last_clean":"2023-11-24T08:17:43.217622+0000","last_became_active":"2023-11-24T08:17:41.272366+0000","last_became_peered":"2023-11-24T08:17:41.272366+0000","last_unstale":"2023-11-24T08:17:43.217622+0000","last_undegraded":"2023-11-24T08:17:43.217622+0000","last_fullsized":"2023-11-24T08:17:43.217622+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:42.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218510+0000","last_change":"2023-11-24T08:17:40.263781+0000","last_active":"2023-11-24T08:17:43.218510+0000","last_peered":"2023-11-24T08:17:43.218510+0000","last_clean":"2023-11-24T08:17:43.218510+0000","last_became_active":"2023-11-24T08:17:40.263468+0000","last_became_peered":"2023-11-24T08:17:40.263468+0000","last_unstale":"2023-11-24T08:17:43.218510+0000","last_undegraded":"2023-11-24T08:17:43.218510+0000","last_fullsized":"2023-11-24T08:17:43.218510+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219564+0000","last_change":"2023-11-24T08:17:41.273505+0000","last_active":"2023-11-24T08:17:43.219564+0000","last_peered":"2023-11-24T08:17:43.219564+0000","last_clean":"2023-11-24T08:17:43.219564+0000","last_became_active":"2023-11-24T08:17:41.273251+0000","last_became_peered":"2023-11-24T08:17:41.273251+0000","last_unstale":"2023-11-24T08:17:43.219564+0000","last_undegraded":"2023-11-24T08:17:43.219564+0000","last_fullsized":"2023-11-24T08:17:43.219564+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:19:10.931535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218222+0000","last_change":"2023-11-24T08:17:40.270845+0000","last_active":"2023-11-24T08:17:43.218222+0000","last_peered":"2023-11-24T08:17:43.218222+0000","last_clean":"2023-11-24T08:17:43.218222+0000","last_became_active":"2023-11-24T08:17:40.270368+0000","last_became_peered":"2023-11-24T08:17:40.270368+0000","last_unstale":"2023-11-24T08:17:43.218222+0000","last_undegraded":"2023-11-24T08:17:43.218222+0000","last_fullsized":"2023-11-24T08:17:43.218222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:51.014940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219323+0000","last_change":"2023-11-24T08:17:41.274658+0000","last_active":"2023-11-24T08:17:43.219323+0000","last_peered":"2023-11-24T08:17:43.219323+0000","last_clean":"2023-11-24T08:17:43.219323+0000","last_became_active":"2023-11-24T08:17:41.274084+0000","last_became_peered":"2023-11-24T08:17:41.274084+0000","last_unstale":"2023-11-24T08:17:43.219323+0000","last_undegraded":"2023-11-24T08:17:43.219323+0000","last_fullsized":"2023-11-24T08:17:43.219323+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:09.576180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219310+0000","last_change":"2023-11-24T08:17:40.271571+0000","last_active":"2023-11-24T08:17:43.219310+0000","last_peered":"2023-11-24T08:17:43.219310+0000","last_clean":"2023-11-24T08:17:43.219310+0000","last_became_active":"2023-11-24T08:17:40.271345+0000","last_became_peered":"2023-11-24T08:17:40.271345+0000","last_unstale":"2023-11-24T08:17:43.219310+0000","last_undegraded":"2023-11-24T08:17:43.219310+0000","last_fullsized":"2023-11-24T08:17:43.219310+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:46.674445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216823+0000","last_change":"2023-11-24T08:17:41.274647+0000","last_active":"2023-11-24T08:17:43.216823+0000","last_peered":"2023-11-24T08:17:43.216823+0000","last_clean":"2023-11-24T08:17:43.216823+0000","last_became_active":"2023-11-24T08:17:41.274377+0000","last_became_peered":"2023-11-24T08:17:41.274377+0000","last_unstale":"2023-11-24T08:17:43.216823+0000","last_undegraded":"2023-11-24T08:17:43.216823+0000","last_fullsized":"2023-11-24T08:17:43.216823+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:06:25.162702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276564+0000","last_change":"2023-11-24T08:17:40.264532+0000","last_active":"2023-11-24T08:17:42.276564+0000","last_peered":"2023-11-24T08:17:42.276564+0000","last_clean":"2023-11-24T08:17:42.276564+0000","last_became_active":"2023-11-24T08:17:40.264219+0000","last_became_peered":"2023-11-24T08:17:40.264219+0000","last_unstale":"2023-11-24T08:17:42.276564+0000","last_undegraded":"2023-11-24T08:17:42.276564+0000","last_fullsized":"2023-11-24T08:17:42.276564+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:36.570984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220346+0000","last_change":"2023-11-24T08:17:41.280834+0000","last_active":"2023-11-24T08:17:43.220346+0000","last_peered":"2023-11-24T08:17:43.220346+0000","last_clean":"2023-11-24T08:17:43.220346+0000","last_became_active":"2023-11-24T08:17:41.280250+0000","last_became_peered":"2023-11-24T08:17:41.280250+0000","last_unstale":"2023-11-24T08:17:43.220346+0000","last_undegraded":"2023-11-24T08:17:43.220346+0000","last_fullsized":"2023-11-24T08:17:43.220346+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:16:37.412337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219108+0000","last_change":"2023-11-24T08:17:40.268586+0000","last_active":"2023-11-24T08:17:43.219108+0000","last_peered":"2023-11-24T08:17:43.219108+0000","last_clean":"2023-11-24T08:17:43.219108+0000","last_became_active":"2023-11-24T08:17:40.267562+0000","last_became_peered":"2023-11-24T08:17:40.267562+0000","last_unstale":"2023-11-24T08:17:43.219108+0000","last_undegraded":"2023-11-24T08:17:43.219108+0000","last_fullsized":"2023-11-24T08:17:43.219108+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:10.951843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218731+0000","last_change":"2023-11-24T08:17:41.264880+0000","last_active":"2023-11-24T08:17:43.218731+0000","last_peered":"2023-11-24T08:17:43.218731+0000","last_clean":"2023-11-24T08:17:43.218731+0000","last_became_active":"2023-11-24T08:17:41.264462+0000","last_became_peered":"2023-11-24T08:17:41.264462+0000","last_unstale":"2023-11-24T08:17:43.218731+0000","last_undegraded":"2023-11-24T08:17:43.218731+0000","last_fullsized":"2023-11-24T08:17:43.218731+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:13.172494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218362+0000","last_change":"2023-11-24T08:17:40.265481+0000","last_active":"2023-11-24T08:17:43.218362+0000","last_peered":"2023-11-24T08:17:43.218362+0000","last_clean":"2023-11-24T08:17:43.218362+0000","last_became_active":"2023-11-24T08:17:40.262585+0000","last_became_peered":"2023-11-24T08:17:40.262585+0000","last_unstale":"2023-11-24T08:17:43.218362+0000","last_undegraded":"2023-11-24T08:17:43.218362+0000","last_fullsized":"2023-11-24T08:17:43.218362+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:44.339007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218326+0000","last_change":"2023-11-24T08:17:41.281684+0000","last_active":"2023-11-24T08:17:43.218326+0000","last_peered":"2023-11-24T08:17:43.218326+0000","last_clean":"2023-11-24T08:17:43.218326+0000","last_became_active":"2023-11-24T08:17:41.281410+0000","last_became_peered":"2023-11-24T08:17:41.281410+0000","last_unstale":"2023-11-24T08:17:43.218326+0000","last_undegraded":"2023-11-24T08:17:43.218326+0000","last_fullsized":"2023-11-24T08:17:43.218326+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:56:37.604937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277004+0000","last_change":"2023-11-24T08:17:40.259164+0000","last_active":"2023-11-24T08:17:42.277004+0000","last_peered":"2023-11-24T08:17:42.277004+0000","last_clean":"2023-11-24T08:17:42.277004+0000","last_became_active":"2023-11-24T08:17:40.258813+0000","last_became_peered":"2023-11-24T08:17:40.258813+0000","last_unstale":"2023-11-24T08:17:42.277004+0000","last_undegraded":"2023-11-24T08:17:42.277004+0000","last_fullsized":"2023-11-24T08:17:42.277004+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:04.952870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277052+0000","last_change":"2023-11-24T08:17:41.274538+0000","last_active":"2023-11-24T08:17:42.277052+0000","last_peered":"2023-11-24T08:17:42.277052+0000","last_clean":"2023-11-24T08:17:42.277052+0000","last_became_active":"2023-11-24T08:17:41.272127+0000","last_became_peered":"2023-11-24T08:17:41.272127+0000","last_unstale":"2023-11-24T08:17:42.277052+0000","last_undegraded":"2023-11-24T08:17:42.277052+0000","last_fullsized":"2023-11-24T08:17:42.277052+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:00.248461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218510+0000","last_change":"2023-11-24T08:17:40.268292+0000","last_active":"2023-11-24T08:17:43.218510+0000","last_peered":"2023-11-24T08:17:43.218510+0000","last_clean":"2023-11-24T08:17:43.218510+0000","last_became_active":"2023-11-24T08:17:40.267973+0000","last_became_peered":"2023-11-24T08:17:40.267973+0000","last_unstale":"2023-11-24T08:17:43.218510+0000","last_undegraded":"2023-11-24T08:17:43.218510+0000","last_fullsized":"2023-11-24T08:17:43.218510+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:01.331385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215690+0000","last_change":"2023-11-24T08:17:41.281784+0000","last_active":"2023-11-24T08:17:43.215690+0000","last_peered":"2023-11-24T08:17:43.215690+0000","last_clean":"2023-11-24T08:17:43.215690+0000","last_became_active":"2023-11-24T08:17:41.281486+0000","last_became_peered":"2023-11-24T08:17:41.281486+0000","last_unstale":"2023-11-24T08:17:43.215690+0000","last_undegraded":"2023-11-24T08:17:43.215690+0000","last_fullsized":"2023-11-24T08:17:43.215690+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:55.343969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218352+0000","last_change":"2023-11-24T08:17:40.268699+0000","last_active":"2023-11-24T08:17:43.218352+0000","last_peered":"2023-11-24T08:17:43.218352+0000","last_clean":"2023-11-24T08:17:43.218352+0000","last_became_active":"2023-11-24T08:17:40.268429+0000","last_became_peered":"2023-11-24T08:17:40.268429+0000","last_unstale":"2023-11-24T08:17:43.218352+0000","last_undegraded":"2023-11-24T08:17:43.218352+0000","last_fullsized":"2023-11-24T08:17:43.218352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:27.915096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274649+0000","last_change":"2023-11-24T08:17:41.274726+0000","last_active":"2023-11-24T08:17:42.274649+0000","last_peered":"2023-11-24T08:17:42.274649+0000","last_clean":"2023-11-24T08:17:42.274649+0000","last_became_active":"2023-11-24T08:17:41.274040+0000","last_became_peered":"2023-11-24T08:17:41.274040+0000","last_unstale":"2023-11-24T08:17:42.274649+0000","last_undegraded":"2023-11-24T08:17:42.274649+0000","last_fullsized":"2023-11-24T08:17:42.274649+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:05:17.464923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275300+0000","last_change":"2023-11-24T08:17:40.254305+0000","last_active":"2023-11-24T08:17:42.275300+0000","last_peered":"2023-11-24T08:17:42.275300+0000","last_clean":"2023-11-24T08:17:42.275300+0000","last_became_active":"2023-11-24T08:17:40.253763+0000","last_became_peered":"2023-11-24T08:17:40.253763+0000","last_unstale":"2023-11-24T08:17:42.275300+0000","last_undegraded":"2023-11-24T08:17:42.275300+0000","last_fullsized":"2023-11-24T08:17:42.275300+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:25.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218945+0000","last_change":"2023-11-24T08:17:41.274813+0000","last_active":"2023-11-24T08:17:43.218945+0000","last_peered":"2023-11-24T08:17:43.218945+0000","last_clean":"2023-11-24T08:17:43.218945+0000","last_became_active":"2023-11-24T08:17:41.274416+0000","last_became_peered":"2023-11-24T08:17:41.274416+0000","last_unstale":"2023-11-24T08:17:43.218945+0000","last_undegraded":"2023-11-24T08:17:43.218945+0000","last_fullsized":"2023-11-24T08:17:43.218945+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:26:11.676844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277623+0000","last_change":"2023-11-24T08:17:40.266271+0000","last_active":"2023-11-24T08:17:42.277623+0000","last_peered":"2023-11-24T08:17:42.277623+0000","last_clean":"2023-11-24T08:17:42.277623+0000","last_became_active":"2023-11-24T08:17:40.265994+0000","last_became_peered":"2023-11-24T08:17:40.265994+0000","last_unstale":"2023-11-24T08:17:42.277623+0000","last_undegraded":"2023-11-24T08:17:42.277623+0000","last_fullsized":"2023-11-24T08:17:42.277623+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-24T08:17:41.257834+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:17:41.257834+0000","last_peered":"2023-11-24T08:17:41.257834+0000","last_clean":"2023-11-24T08:17:41.257834+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:17:41.257834+0000","last_undegraded":"2023-11-24T08:17:41.257834+0000","last_fullsized":"2023-11-24T08:17:41.257834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37924,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333916,"statfs":{"total":96636764160,"available":96597929984,"internally_reserved":0,"allocated":2637824,"data_stored":1480745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.0249999999999999,"15min":1.0349999999999999},"min":{"1min":0.60499999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6060000000000001,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.042,"5min":1.0669999999999999,"15min":1.0880000000000001},"min":{"1min":0.67100000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.4570000000000001,"5min":1.774,"15min":2.7109999999999999},"last":1.3859999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1559999999999999,"15min":1.155},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.6000000000000001,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.256,"15min":1.2150000000000001},"min":{"1min":0.71099999999999997,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":1.377,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.874}]},{"osd":2,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1220000000000001,"15min":1.1619999999999999},"min":{"1min":0.69499999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5980000000000001,"5min":1.774,"15min":4.7000000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.2889999999999999,"15min":1.3320000000000001},"min":{"1min":0.91900000000000004,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7310000000000001,"5min":1.889,"15min":4.633},"last":1.2370000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.22,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.645,"5min":1.99,"15min":3.133},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.357,"15min":1.3779999999999999},"min":{"1min":0.96899999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.7370000000000001,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.446}]},{"osd":4,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3660000000000001,"15min":1.351},"min":{"1min":0.83699999999999997,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":1.597,"5min":2.665,"15min":2.851},"last":1.6659999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.3460000000000001,"15min":1.351},"min":{"1min":0.98499999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8069999999999999,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.3220000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.462,"15min":1.4770000000000001},"min":{"1min":1.1990000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.1080000000000001,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.552},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":1.095,"5min":0.622,"15min":0.622},"max":{"1min":1.8959999999999999,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.514}]},{"osd":6,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.2030000000000001,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":1.994,"5min":2.702,"15min":2.992},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.518,"15min":1.494},"min":{"1min":0.97799999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.982,"5min":2.6259999999999999,"15min":3.798},"last":1.635}]}]},{"osd":6,"up_from":13,"seq":55834575007,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37328,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334512,"statfs":{"total":96636764160,"available":96598540288,"internally_reserved":0,"allocated":2023424,"data_stored":888045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0960000000000001,"15min":1.0860000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":1.95},"last":1.2589999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.077,"15min":1.083},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.58099999999999996},"max":{"1min":1.3420000000000001,"5min":1.804,"15min":2.2069999999999999},"last":1.575}]},{"osd":1,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.151,"15min":1.1950000000000001},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5289999999999999,"5min":1.835,"15min":2.0230000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.206,"15min":1.2170000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.5649999999999999,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.4430000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.79500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.127},{"interface":"front","average":{"1min":1.25,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.60399999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":1.609}]},{"osd":3,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.325,"15min":1.3600000000000001},"min":{"1min":0.91400000000000003,"5min":0.86399999999999999,"15min":0.71499999999999997},"max":{"1min":1.7490000000000001,"5min":2.2330000000000001,"15min":4.7880000000000003},"last":1.339},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.304,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.788,"5min":2.194,"15min":4.8319999999999999},"last":1.4750000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.452,"15min":1.4730000000000001},"min":{"1min":0.96899999999999997,"5min":0.90200000000000002,"15min":0.76500000000000001},"max":{"1min":1.9019999999999999,"5min":2.1190000000000002,"15min":2.895},"last":1.367},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5169999999999999},"min":{"1min":1.056,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.8740000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.448,"15min":1.48},"min":{"1min":1.145,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.956,"5min":2.3300000000000001,"15min":2.629},"last":1.302},{"interface":"front","average":{"1min":1.603,"5min":1.5840000000000001,"15min":1.6120000000000001},"min":{"1min":1.0009999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.996,"5min":2.165,"15min":2.7759999999999998},"last":1.72}]},{"osd":7,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.6020000000000001,"15min":1.6379999999999999},"min":{"1min":1.179,"5min":0.748,"15min":0.748},"max":{"1min":2.101,"5min":2.4100000000000001,"15min":3.0329999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.627,"15min":1.6299999999999999},"min":{"1min":1.056,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.4119999999999999,"5min":2.512,"15min":2.6890000000000001},"last":1.544}]}]},{"osd":5,"up_from":13,"seq":55834575007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37336,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334504,"statfs":{"total":96636764160,"available":96598532096,"internally_reserved":0,"allocated":2039808,"data_stored":894927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.96999999999999997,"15min":1.044},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.129,"5min":1.5169999999999999,"15min":3.1179999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.04,"15min":1.123},"min":{"1min":0.66500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.407,"5min":1.627,"15min":2.198},"last":0.86299999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.0980000000000001,"15min":1.1279999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.4059999999999999,"5min":1.8999999999999999,"15min":2.0529999999999999},"last":1.478},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.177,"15min":1.163},"min":{"1min":0.875,"5min":0.76200000000000001,"15min":0.48699999999999999},"max":{"1min":1.448,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.046}]},{"osd":2,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.1120000000000001,"15min":1.1799999999999999},"min":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.51600000000000001},"max":{"1min":1.4650000000000001,"5min":1.671,"15min":1.8520000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.64200000000000002},"max":{"1min":1.651,"5min":1.784,"15min":1.8129999999999999},"last":1.528}]},{"osd":3,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.157,"15min":1.163},"min":{"1min":0.78800000000000003,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.5620000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.298,"5min":1.2310000000000001,"15min":1.286},"min":{"1min":1.1399999999999999,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.603,"5min":1.603,"15min":1.99},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.355,"15min":1.337},"min":{"1min":1.012,"5min":0.66400000000000003,"15min":0.60799999999999998},"max":{"1min":1.6419999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.28},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.421,"15min":1.484},"min":{"1min":0.94399999999999995,"5min":0.71099999999999997,"15min":0.46700000000000003},"max":{"1min":1.6890000000000001,"5min":1.994,"15min":3.5030000000000001},"last":1.3009999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.3380000000000001,"15min":1.3740000000000001},"min":{"1min":0.878,"5min":0.63600000000000001,"15min":0.48899999999999999},"max":{"1min":1.7150000000000001,"5min":2.085,"15min":3.2949999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.492,"5min":1.4450000000000001,"15min":1.4299999999999999},"min":{"1min":1.2629999999999999,"5min":0.71299999999999997,"15min":0.60999999999999999},"max":{"1min":1.7390000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.137}]},{"osd":7,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5029999999999999,"15min":1.5229999999999999},"min":{"1min":1.1499999999999999,"5min":0.69899999999999995,"15min":0.52900000000000003},"max":{"1min":1.8400000000000001,"5min":2.1440000000000001,"15min":3.4420000000000002},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.3620000000000001,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":1.827,"5min":2.113,"15min":2.181},"last":1.3400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37344,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334496,"statfs":{"total":96636764160,"available":96598523904,"internally_reserved":0,"allocated":2048000,"data_stored":896005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0469999999999999,"15min":1.0900000000000001},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.42699999999999999},"max":{"1min":1.4139999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.099,"15min":1.0840000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.696,"5min":5.452,"15min":5.452},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.1579999999999999,"15min":1.151},"min":{"1min":0.85499999999999998,"5min":0.70199999999999996,"15min":0.64200000000000002},"max":{"1min":1.708,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.421},{"interface":"front","average":{"1min":1.147,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.68799999999999994,"15min":0.51900000000000002},"max":{"1min":1.556,"5min":1.988,"15min":1.988},"last":1.496}]},{"osd":2,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0640000000000001,"15min":1.1000000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.645,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.246,"15min":1.2829999999999999},"min":{"1min":0.76100000000000001,"5min":0.71199999999999997,"15min":0.505},"max":{"1min":1.6359999999999999,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.5389999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3640000000000001,"15min":1.369},"min":{"1min":0.85299999999999998,"5min":0.74399999999999999,"15min":0.62},"max":{"1min":1.6359999999999999,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3089999999999999,"15min":1.296},"min":{"1min":0.87,"5min":0.87,"15min":0.55800000000000005},"max":{"1min":1.6639999999999999,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.1659999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.5329999999999999,"15min":1.5009999999999999},"min":{"1min":1.276,"5min":0.71999999999999997,"15min":0.69299999999999995},"max":{"1min":1.853,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.472,"15min":1.4810000000000001},"min":{"1min":0.85799999999999998,"5min":0.78200000000000003,"15min":0.60199999999999998},"max":{"1min":1.748,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.738}]},{"osd":6,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2669999999999999,"15min":1.3240000000000001},"min":{"1min":0.93999999999999995,"5min":0.57499999999999996,"15min":0.50900000000000001},"max":{"1min":1.73,"5min":1.907,"15min":3.5899999999999999},"last":1.228},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5620000000000001,"15min":1.581},"min":{"1min":1.1319999999999999,"5min":0.79000000000000004,"15min":0.58099999999999996},"max":{"1min":1.772,"5min":2.2890000000000001,"15min":3.391},"last":1.9219999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.52,"15min":1.5449999999999999},"min":{"1min":1.0089999999999999,"5min":0.63300000000000001,"15min":0.60899999999999999},"max":{"1min":1.6950000000000001,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.4370000000000001,"15min":1.478},"min":{"1min":1.0389999999999999,"5min":0.63500000000000001,"15min":0.61799999999999999},"max":{"1min":1.78,"5min":2.1659999999999999,"15min":3.4849999999999999},"last":1.6419999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37320,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334520,"statfs":{"total":96636764160,"available":96598548480,"internally_reserved":0,"allocated":2023424,"data_stored":888057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.39200000000000002,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.3360000000000001,"5min":1.829,"15min":10.557},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94899999999999995,"15min":0.91800000000000004},"min":{"1min":0.54900000000000004,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85499999999999998,"15min":0.82499999999999996},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":1.71,"5min":1.728,"15min":2.101},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.95899999999999996,"15min":0.98999999999999999},"min":{"1min":0.52600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.355,"5min":1.8799999999999999,"15min":13.090999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.93300000000000005},"min":{"1min":0.628,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.349,"5min":1.6060000000000001,"15min":3.9489999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0780000000000001},"min":{"1min":0.55900000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.589,"5min":1.796,"15min":13.460000000000001},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.997,"5min":0.72399999999999998,"15min":0.67800000000000005},"max":{"1min":1.877,"5min":2.1840000000000002,"15min":4.367},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2030000000000001,"15min":1.25},"min":{"1min":0.73199999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.746,"5min":1.746,"15min":3.3380000000000001},"last":1.4339999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2909999999999999,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.635,"5min":2.0609999999999999,"15min":13.231},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2549999999999999,"15min":1.282},"min":{"1min":0.80500000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.744,"5min":1.992,"15min":5.1180000000000003},"last":1.0149999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.339,"15min":1.377},"min":{"1min":0.77100000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.9370000000000001,"5min":2.0449999999999999,"15min":13.359999999999999},"last":1.242},{"interface":"front","average":{"1min":1.232,"5min":1.2709999999999999,"15min":1.286},"min":{"1min":0.83999999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.0369999999999999,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.5609999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3180000000000001,"15min":1.363},"min":{"1min":0.85399999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.823,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.335},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3200000000000001,"15min":1.3640000000000001},"min":{"1min":0.81200000000000006,"5min":0.76400000000000001,"15min":0.68799999999999994},"max":{"1min":1.855,"5min":1.887,"15min":4.032},"last":1.5029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575006,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37312,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334528,"statfs":{"total":96636764160,"available":96598556672,"internally_reserved":0,"allocated":2023424,"data_stored":888113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.78200000000000003,"15min":0.82699999999999996},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.86899999999999999,"5min":1.601,"15min":1.9610000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.379,"5min":1.782,"15min":2.5779999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.91400000000000003,"15min":0.98299999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.2430000000000001,"5min":1.518,"15min":1.9890000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.95299999999999996,"15min":1.0109999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.458,"5min":1.6200000000000001,"15min":2.6240000000000001},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.84399999999999997,"15min":0.88700000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.224,"5min":1.5309999999999999,"15min":2.2090000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0489999999999999,"15min":1.119},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4650000000000001,"5min":1.927,"15min":2.9950000000000001},"last":0.85899999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2709999999999999,"15min":1.3220000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.669,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.228,"15min":1.3049999999999999},"min":{"1min":0.78300000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.704,"5min":1.921,"15min":3.3279999999999998},"last":0.88}]},{"osd":5,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.236,"15min":1.323},"min":{"1min":0.66100000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.9319999999999999,"15min":2.508},"last":1.246},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.2190000000000001,"15min":1.298},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.6679999999999999,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.048}]},{"osd":6,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1799999999999999,"15min":1.2490000000000001},"min":{"1min":0.77000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.7629999999999999,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3120000000000001,"15min":1.391},"min":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5449999999999999,"5min":2.069,"15min":2.6629999999999998},"last":1.2969999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.294,"15min":1.3839999999999999},"min":{"1min":0.81999999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.581,"5min":2.3999999999999999,"15min":2.859},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.2849999999999999,"15min":1.3560000000000001},"min":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.746},"max":{"1min":1.6140000000000001,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":0.98399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37324,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334516,"statfs":{"total":96636764160,"available":96598544384,"internally_reserved":0,"allocated":2023424,"data_stored":888057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.79500000000000004,"15min":0.78500000000000003},"min":{"1min":0.34499999999999997,"5min":0.33000000000000002,"15min":0.28399999999999997},"max":{"1min":1.5660000000000001,"5min":1.5900000000000001,"15min":1.7749999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.872,"15min":0.86499999999999999},"min":{"1min":0.505,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.413,"5min":1.5189999999999999,"15min":2.1619999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.76500000000000001},"min":{"1min":0.47299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.4790000000000001,"5min":1.679,"15min":1.861},"last":1.125},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.91000000000000003,"15min":0.89900000000000002},"min":{"1min":0.39100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.484,"5min":1.925,"15min":1.925},"last":1.0620000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.97899999999999998,"15min":0.97099999999999997},"min":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.601,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.90800000000000003,"15min":0.90000000000000002},"min":{"1min":0.48199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.117,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.22}]},{"osd":4,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2529999999999999,"15min":1.252},"min":{"1min":0.879,"5min":0.76300000000000001,"15min":0.65300000000000002},"max":{"1min":1.788,"5min":1.8380000000000001,"15min":2.7959999999999998},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.23,"15min":1.248},"min":{"1min":0.82099999999999995,"5min":0.77200000000000002,"15min":0.55900000000000005},"max":{"1min":1.5629999999999999,"5min":2.141,"15min":3.0409999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.222,"15min":1.228},"min":{"1min":0.79400000000000004,"5min":0.75900000000000001,"15min":0.69599999999999995},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":3.7639999999999998},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1739999999999999,"15min":1.1890000000000001},"min":{"1min":0.85999999999999999,"5min":0.66900000000000004,"15min":0.57299999999999995},"max":{"1min":1.6359999999999999,"5min":2.246,"15min":3.286},"last":1.6060000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.244,"15min":1.2569999999999999},"min":{"1min":0.81399999999999995,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":1.679,"5min":2.0419999999999998,"15min":4.2750000000000004},"last":1.46},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.232,"15min":1.2370000000000001},"min":{"1min":0.80500000000000005,"5min":0.71299999999999997,"15min":0.63100000000000001},"max":{"1min":1.7350000000000001,"5min":1.788,"15min":3.7040000000000002},"last":1.385}]},{"osd":7,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2849999999999999,"15min":1.3009999999999999},"min":{"1min":0.876,"5min":0.71899999999999997,"15min":0.59099999999999997},"max":{"1min":1.798,"5min":2.1429999999999998,"15min":3.1760000000000002},"last":1.52},{"interface":"front","average":{"1min":1.196,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.72299999999999998,"5min":0.624,"15min":0.624},"max":{"1min":1.841,"5min":2.3620000000000001,"15min":3.653},"last":1.7010000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37892,"kb_used_data":2552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333948,"statfs":{"total":96636764160,"available":96597962752,"internally_reserved":0,"allocated":2613248,"data_stored":1478391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.78900000000000003},"min":{"1min":0.39300000000000002,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81899999999999995,"15min":0.89800000000000002},"min":{"1min":0.51000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.4219999999999999,"5min":1.4610000000000001,"15min":2.3620000000000001},"last":1.1659999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0209999999999999,"15min":1.0309999999999999},"min":{"1min":0.58399999999999996,"5min":0.51100000000000001,"15min":0.47699999999999998},"max":{"1min":1.7529999999999999,"5min":1.833,"15min":1.891},"last":1.3169999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.88600000000000001,"15min":0.90200000000000002},"min":{"1min":0.51900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.798,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.97999999999999998,"15min":1.036},"min":{"1min":0.60999999999999999,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.667,"5min":1.679,"15min":2.0670000000000002},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89500000000000002,"15min":0.95599999999999996},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.651,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":1.24}]},{"osd":4,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1659999999999999,"15min":1.286},"min":{"1min":0.80800000000000005,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.7610000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.139,"15min":1.1699999999999999},"min":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5109999999999999,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.284}]},{"osd":5,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1679999999999999,"15min":1.2210000000000001},"min":{"1min":0.82099999999999995,"5min":0.752,"15min":0.69699999999999995},"max":{"1min":1.6080000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.091},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1579999999999999,"15min":1.2709999999999999},"min":{"1min":0.75600000000000001,"5min":0.72299999999999998,"15min":0.71099999999999997},"max":{"1min":1.595,"5min":1.595,"15min":2.7240000000000002},"last":1.341}]},{"osd":6,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2549999999999999,"15min":1.329},"min":{"1min":0.79400000000000004,"5min":0.78000000000000003,"15min":0.66800000000000004},"max":{"1min":1.7669999999999999,"5min":2.052,"15min":2.4369999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.262,"15min":1.3340000000000001},"min":{"1min":0.84399999999999997,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.7070000000000001,"5min":2.823,"15min":2.823},"last":1.377}]},{"osd":7,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3440000000000001,"15min":1.4199999999999999},"min":{"1min":0.997,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":1.8300000000000001,"5min":2.1349999999999998,"15min":2.6989999999999998},"last":1.486},{"interface":"front","average":{"1min":1.302,"5min":1.2969999999999999,"15min":1.391},"min":{"1min":0.80300000000000005,"5min":0.747,"15min":0.747},"max":{"1min":1.885,"5min":2.0499999999999998,"15min":2.3929999999999998},"last":1.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:17:45.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:17:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.974+0000 7fa1f2621640 1 Processor -- start 2023-11-24T08:17:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.975+0000 7fa1f2621640 1 -- start start 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f2621640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa1ec0703f0 0x7fa1ec162970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f2621640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 0x7fa1ec0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f2621640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec162eb0 con 0x7fa1ec14b100 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f2621640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cdb10 con 0x7fa1ec0703f0 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f2621640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec1654a0 con 0x7fa1ec0cd160 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f161f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa1ec14b100 0x7fa1ec165c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33364/0 (socket says 172.21.15.107:33364) 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f161f640 1 -- 172.21.15.107:0/2741079433 learned_addr learned my addr 172.21.15.107:0/2741079433 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.976+0000 7fa1f0e1e640 1 --2- 172.21.15.107:0/2741079433 >> v2:172.21.15.107:3300/0 conn(0x7fa1ec0703f0 0x7fa1ec162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f1e20640 1 --2- 172.21.15.107:0/2741079433 >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 0x7fa1ec0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f0e1e640 1 -- 172.21.15.107:0/2741079433 >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 msgr2=0x7fa1ec0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f0e1e640 1 --2- 172.21.15.107:0/2741079433 >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 0x7fa1ec0cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f0e1e640 1 -- 172.21.15.107:0/2741079433 >> v1:172.21.15.107:6789/0 conn(0x7fa1ec14b100 legacy=0x7fa1ec165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1e3fff640 1 -- 172.21.15.107:0/2741079433 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598700016 0 0) 0x7fa1e8001110 con 0x7fa1ec14b100 2023-11-24T08:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f0e1e640 1 -- 172.21.15.107:0/2741079433 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0ccf00 con 0x7fa1ec0703f0 2023-11-24T08:17:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.977+0000 7fa1f1e20640 1 --2- 172.21.15.107:0/2741079433 >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 0x7fa1ec0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.978+0000 7fa1f0e1e640 1 --2- 172.21.15.107:0/2741079433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec162970 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0aea00 tx=0x7fa1dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c19ec7c2383d908 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.978+0000 7fa1e3fff640 1 -- 172.21.15.107:0/2741079433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc02e070 con 0x7fa1ec0703f0 2023-11-24T08:17:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.978+0000 7fa1e3fff640 1 -- 172.21.15.107:0/2741079433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc0315a0 con 0x7fa1ec0703f0 2023-11-24T08:17:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.978+0000 7fa1e3fff640 1 -- 172.21.15.107:0/2741079433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc031930 con 0x7fa1ec0703f0 2023-11-24T08:17:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 -- 172.21.15.107:0/2741079433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 msgr2=0x7fa1ec162970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 --2- 172.21.15.107:0/2741079433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec162970 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0aea00 tx=0x7fa1dc02b840 comp rx=0 tx=0).stop 2023-11-24T08:17:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 -- 172.21.15.107:0/2741079433 shutdown_connections 2023-11-24T08:17:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 --2- 172.21.15.107:0/2741079433 >> v2:172.21.15.120:3300/0 conn(0x7fa1ec0cd160 0x7fa1ec0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 --2- 172.21.15.107:0/2741079433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec162970 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 -- 172.21.15.107:0/2741079433 >> 172.21.15.107:0/2741079433 conn(0x7fa1ec0cb830 msgr2=0x7fa1ec0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 -- 172.21.15.107:0/2741079433 shutdown_connections 2023-11-24T08:17:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.979+0000 7fa1f2621640 1 -- 172.21.15.107:0/2741079433 wait complete. 2023-11-24T08:17:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.981+0000 7fa1f2621640 1 Processor -- start 2023-11-24T08:17:45.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.981+0000 7fa1f2621640 1 -- start start 2023-11-24T08:17:45.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cd160 0x7fa1ec0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec14b100 0x7fa1ec1481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0a8a80 con 0x7fa1ec0703f0 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0a8d60 con 0x7fa1ec14b100 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f2621640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0a9040 con 0x7fa1ec0cd160 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f161f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f161f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36282/0 (socket says 172.21.15.107:36282) 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.982+0000 7fa1f161f640 1 -- 172.21.15.107:0/1544154180 learned_addr learned my addr 172.21.15.107:0/1544154180 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 -- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cd160 msgr2=0x7fa1ec0ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cd160 0x7fa1ec0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 -- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec14b100 msgr2=0x7fa1ec1481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec14b100 0x7fa1ec1481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:45.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 -- 172.21.15.107:0/1544154180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009400 con 0x7fa1ec0703f0 2023-11-24T08:17:45.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f161f640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800c430 tx=0x7fa1e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80074e0 con 0x7fa1ec0703f0 2023-11-24T08:17:45.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e8007660 con 0x7fa1ec0703f0 2023-11-24T08:17:45.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8004a20 con 0x7fa1ec0703f0 2023-11-24T08:17:45.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0a9320 con 0x7fa1ec0703f0 2023-11-24T08:17:45.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.983+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec1f8490 con 0x7fa1ec0703f0 2023-11-24T08:17:45.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.985+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec0a9840 con 0x7fa1ec0703f0 2023-11-24T08:17:45.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.986+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fa1e8004b80 con 0x7fa1ec0703f0 2023-11-24T08:17:45.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.986+0000 7fa1e1ffb640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 0x7fa1d006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:45.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.987+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1e8088110 con 0x7fa1ec0703f0 2023-11-24T08:17:45.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.987+0000 7fa1f0e1e640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 0x7fa1d006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:45.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.987+0000 7fa1f0e1e640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 0x7fa1d006eaa0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036000 tx=0x7fa1dc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:45.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:45.991+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1ec0a9840 con 0x7fa1ec0703f0 2023-11-24T08:17:46.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.178+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1ec070bd0 con 0x7fa1d006c5e0 2023-11-24T08:17:46.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.190+0000 7fa1e1ffb640 1 -- 172.21.15.107:0/1544154180 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287420 (secure 0 0 0) 0x7fa1dc031a70 con 0x7fa1d006c5e0 2023-11-24T08:17:46.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:46.198 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:17:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 msgr2=0x7fa1d006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 0x7fa1d006eaa0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036000 tx=0x7fa1dc030000 comp rx=0 tx=0).stop 2023-11-24T08:17:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 msgr2=0x7fa1ec0a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800c430 tx=0x7fa1e800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 shutdown_connections 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1d006c5e0 0x7fa1d006eaa0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec14b100 0x7fa1ec1481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cd160 0x7fa1ec0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 --2- 172.21.15.107:0/1544154180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec0703f0 0x7fa1ec0a8430 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.195+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 >> 172.21.15.107:0/1544154180 conn(0x7fa1ec0cb830 msgr2=0x7fa1ec163050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.196+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 shutdown_connections 2023-11-24T08:17:46.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.196+0000 7fa1f2621640 1 -- 172.21.15.107:0/1544154180 wait complete. 2023-11-24T08:17:46.217 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2023-11-24T08:17:45.673544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":299780,"kb_used_data":17024,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754674940,"statfs":{"total":773094113280,"available":772787138560,"internally_reserved":0,"allocated":17432576,"data_stored":8302340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":289394238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001787"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217334+0000","last_change":"2023-11-24T08:17:41.278306+0000","last_active":"2023-11-24T08:17:43.217334+0000","last_peered":"2023-11-24T08:17:43.217334+0000","last_clean":"2023-11-24T08:17:43.217334+0000","last_became_active":"2023-11-24T08:17:41.276281+0000","last_became_peered":"2023-11-24T08:17:41.276281+0000","last_unstale":"2023-11-24T08:17:43.217334+0000","last_undegraded":"2023-11-24T08:17:43.217334+0000","last_fullsized":"2023-11-24T08:17:43.217334+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:13.172494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219008+0000","last_change":"2023-11-24T08:17:41.283094+0000","last_active":"2023-11-24T08:17:43.219008+0000","last_peered":"2023-11-24T08:17:43.219008+0000","last_clean":"2023-11-24T08:17:43.219008+0000","last_became_active":"2023-11-24T08:17:41.282784+0000","last_became_peered":"2023-11-24T08:17:41.282784+0000","last_unstale":"2023-11-24T08:17:43.219008+0000","last_undegraded":"2023-11-24T08:17:43.219008+0000","last_fullsized":"2023-11-24T08:17:43.219008+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:02.348205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276431+0000","last_change":"2023-11-24T08:17:41.277931+0000","last_active":"2023-11-24T08:17:42.276431+0000","last_peered":"2023-11-24T08:17:42.276431+0000","last_clean":"2023-11-24T08:17:42.276431+0000","last_became_active":"2023-11-24T08:17:41.277609+0000","last_became_peered":"2023-11-24T08:17:41.277609+0000","last_unstale":"2023-11-24T08:17:42.276431+0000","last_undegraded":"2023-11-24T08:17:42.276431+0000","last_fullsized":"2023-11-24T08:17:42.276431+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:46.144410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220836+0000","last_change":"2023-11-24T08:17:41.268672+0000","last_active":"2023-11-24T08:17:43.220836+0000","last_peered":"2023-11-24T08:17:43.220836+0000","last_clean":"2023-11-24T08:17:43.220836+0000","last_became_active":"2023-11-24T08:17:41.267916+0000","last_became_peered":"2023-11-24T08:17:41.267916+0000","last_unstale":"2023-11-24T08:17:43.220836+0000","last_undegraded":"2023-11-24T08:17:43.220836+0000","last_fullsized":"2023-11-24T08:17:43.220836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:32.835769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219320+0000","last_change":"2023-11-24T08:17:41.282966+0000","last_active":"2023-11-24T08:17:43.219320+0000","last_peered":"2023-11-24T08:17:43.219320+0000","last_clean":"2023-11-24T08:17:43.219320+0000","last_became_active":"2023-11-24T08:17:41.281861+0000","last_became_peered":"2023-11-24T08:17:41.281861+0000","last_unstale":"2023-11-24T08:17:43.219320+0000","last_undegraded":"2023-11-24T08:17:43.219320+0000","last_fullsized":"2023-11-24T08:17:43.219320+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:05.969690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219681+0000","last_change":"2023-11-24T08:17:41.282304+0000","last_active":"2023-11-24T08:17:43.219681+0000","last_peered":"2023-11-24T08:17:43.219681+0000","last_clean":"2023-11-24T08:17:43.219681+0000","last_became_active":"2023-11-24T08:17:41.276947+0000","last_became_peered":"2023-11-24T08:17:41.276947+0000","last_unstale":"2023-11-24T08:17:43.219681+0000","last_undegraded":"2023-11-24T08:17:43.219681+0000","last_fullsized":"2023-11-24T08:17:43.219681+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:02.011176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215693+0000","last_change":"2023-11-24T08:17:41.286873+0000","last_active":"2023-11-24T08:17:43.215693+0000","last_peered":"2023-11-24T08:17:43.215693+0000","last_clean":"2023-11-24T08:17:43.215693+0000","last_became_active":"2023-11-24T08:17:41.283757+0000","last_became_peered":"2023-11-24T08:17:41.283757+0000","last_unstale":"2023-11-24T08:17:43.215693+0000","last_undegraded":"2023-11-24T08:17:43.215693+0000","last_fullsized":"2023-11-24T08:17:43.215693+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:31.300677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218482+0000","last_change":"2023-11-24T08:17:41.279770+0000","last_active":"2023-11-24T08:17:43.218482+0000","last_peered":"2023-11-24T08:17:43.218482+0000","last_clean":"2023-11-24T08:17:43.218482+0000","last_became_active":"2023-11-24T08:17:41.278621+0000","last_became_peered":"2023-11-24T08:17:41.278621+0000","last_unstale":"2023-11-24T08:17:43.218482+0000","last_undegraded":"2023-11-24T08:17:43.218482+0000","last_fullsized":"2023-11-24T08:17:43.218482+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:37.614211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219614+0000","last_change":"2023-11-24T08:17:41.282377+0000","last_active":"2023-11-24T08:17:43.219614+0000","last_peered":"2023-11-24T08:17:43.219614+0000","last_clean":"2023-11-24T08:17:43.219614+0000","last_became_active":"2023-11-24T08:17:41.280764+0000","last_became_peered":"2023-11-24T08:17:41.280764+0000","last_unstale":"2023-11-24T08:17:43.219614+0000","last_undegraded":"2023-11-24T08:17:43.219614+0000","last_fullsized":"2023-11-24T08:17:43.219614+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219105+0000","last_change":"2023-11-24T08:17:41.272496+0000","last_active":"2023-11-24T08:17:43.219105+0000","last_peered":"2023-11-24T08:17:43.219105+0000","last_clean":"2023-11-24T08:17:43.219105+0000","last_became_active":"2023-11-24T08:17:41.272201+0000","last_became_peered":"2023-11-24T08:17:41.272201+0000","last_unstale":"2023-11-24T08:17:43.219105+0000","last_undegraded":"2023-11-24T08:17:43.219105+0000","last_fullsized":"2023-11-24T08:17:43.219105+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:55.017070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219011+0000","last_change":"2023-11-24T08:17:41.280150+0000","last_active":"2023-11-24T08:17:43.219011+0000","last_peered":"2023-11-24T08:17:43.219011+0000","last_clean":"2023-11-24T08:17:43.219011+0000","last_became_active":"2023-11-24T08:17:41.279171+0000","last_became_peered":"2023-11-24T08:17:41.279171+0000","last_unstale":"2023-11-24T08:17:43.219011+0000","last_undegraded":"2023-11-24T08:17:43.219011+0000","last_fullsized":"2023-11-24T08:17:43.219011+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:39.960967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218768+0000","last_change":"2023-11-24T08:17:41.281036+0000","last_active":"2023-11-24T08:17:43.218768+0000","last_peered":"2023-11-24T08:17:43.218768+0000","last_clean":"2023-11-24T08:17:43.218768+0000","last_became_active":"2023-11-24T08:17:41.279999+0000","last_became_peered":"2023-11-24T08:17:41.279999+0000","last_unstale":"2023-11-24T08:17:43.218768+0000","last_undegraded":"2023-11-24T08:17:43.218768+0000","last_fullsized":"2023-11-24T08:17:43.218768+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:36.836942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275484+0000","last_change":"2023-11-24T08:17:41.274253+0000","last_active":"2023-11-24T08:17:42.275484+0000","last_peered":"2023-11-24T08:17:42.275484+0000","last_clean":"2023-11-24T08:17:42.275484+0000","last_became_active":"2023-11-24T08:17:41.273958+0000","last_became_peered":"2023-11-24T08:17:41.273958+0000","last_unstale":"2023-11-24T08:17:42.275484+0000","last_undegraded":"2023-11-24T08:17:42.275484+0000","last_fullsized":"2023-11-24T08:17:42.275484+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:44:10.578605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274697+0000","last_change":"2023-11-24T08:17:41.272152+0000","last_active":"2023-11-24T08:17:42.274697+0000","last_peered":"2023-11-24T08:17:42.274697+0000","last_clean":"2023-11-24T08:17:42.274697+0000","last_became_active":"2023-11-24T08:17:41.271551+0000","last_became_peered":"2023-11-24T08:17:41.271551+0000","last_unstale":"2023-11-24T08:17:42.274697+0000","last_undegraded":"2023-11-24T08:17:42.274697+0000","last_fullsized":"2023-11-24T08:17:42.274697+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:07:04.816752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216933+0000","last_change":"2023-11-24T08:17:41.274580+0000","last_active":"2023-11-24T08:17:43.216933+0000","last_peered":"2023-11-24T08:17:43.216933+0000","last_clean":"2023-11-24T08:17:43.216933+0000","last_became_active":"2023-11-24T08:17:41.273155+0000","last_became_peered":"2023-11-24T08:17:41.273155+0000","last_unstale":"2023-11-24T08:17:43.216933+0000","last_undegraded":"2023-11-24T08:17:43.216933+0000","last_fullsized":"2023-11-24T08:17:43.216933+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:56:37.604937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215741+0000","last_change":"2023-11-24T08:17:41.278248+0000","last_active":"2023-11-24T08:17:43.215741+0000","last_peered":"2023-11-24T08:17:43.215741+0000","last_clean":"2023-11-24T08:17:43.215741+0000","last_became_active":"2023-11-24T08:17:41.277145+0000","last_became_peered":"2023-11-24T08:17:41.277145+0000","last_unstale":"2023-11-24T08:17:43.215741+0000","last_undegraded":"2023-11-24T08:17:43.215741+0000","last_fullsized":"2023-11-24T08:17:43.215741+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:47.493778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221337+0000","last_change":"2023-11-24T08:17:41.278710+0000","last_active":"2023-11-24T08:17:43.221337+0000","last_peered":"2023-11-24T08:17:43.221337+0000","last_clean":"2023-11-24T08:17:43.221337+0000","last_became_active":"2023-11-24T08:17:41.278444+0000","last_became_peered":"2023-11-24T08:17:41.278444+0000","last_unstale":"2023-11-24T08:17:43.221337+0000","last_undegraded":"2023-11-24T08:17:43.221337+0000","last_fullsized":"2023-11-24T08:17:43.221337+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:00:45.163228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218776+0000","last_change":"2023-11-24T08:17:41.271384+0000","last_active":"2023-11-24T08:17:43.218776+0000","last_peered":"2023-11-24T08:17:43.218776+0000","last_clean":"2023-11-24T08:17:43.218776+0000","last_became_active":"2023-11-24T08:17:41.268797+0000","last_became_peered":"2023-11-24T08:17:41.268797+0000","last_unstale":"2023-11-24T08:17:43.218776+0000","last_undegraded":"2023-11-24T08:17:43.218776+0000","last_fullsized":"2023-11-24T08:17:43.218776+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:42.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220467+0000","last_change":"2023-11-24T08:17:41.277705+0000","last_active":"2023-11-24T08:17:43.220467+0000","last_peered":"2023-11-24T08:17:43.220467+0000","last_clean":"2023-11-24T08:17:43.220467+0000","last_became_active":"2023-11-24T08:17:41.277049+0000","last_became_peered":"2023-11-24T08:17:41.277049+0000","last_unstale":"2023-11-24T08:17:43.220467+0000","last_undegraded":"2023-11-24T08:17:43.220467+0000","last_fullsized":"2023-11-24T08:17:43.220467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:21.903235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276077+0000","last_change":"2023-11-24T08:17:41.281322+0000","last_active":"2023-11-24T08:17:42.276077+0000","last_peered":"2023-11-24T08:17:42.276077+0000","last_clean":"2023-11-24T08:17:42.276077+0000","last_became_active":"2023-11-24T08:17:41.280414+0000","last_became_peered":"2023-11-24T08:17:41.280414+0000","last_unstale":"2023-11-24T08:17:42.276077+0000","last_undegraded":"2023-11-24T08:17:42.276077+0000","last_fullsized":"2023-11-24T08:17:42.276077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:23.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219812+0000","last_change":"2023-11-24T08:17:41.277945+0000","last_active":"2023-11-24T08:17:43.219812+0000","last_peered":"2023-11-24T08:17:43.219812+0000","last_clean":"2023-11-24T08:17:43.219812+0000","last_became_active":"2023-11-24T08:17:41.277690+0000","last_became_peered":"2023-11-24T08:17:41.277690+0000","last_unstale":"2023-11-24T08:17:43.219812+0000","last_undegraded":"2023-11-24T08:17:43.219812+0000","last_fullsized":"2023-11-24T08:17:43.219812+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:25:34.500836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218210+0000","last_change":"2023-11-24T08:17:41.274009+0000","last_active":"2023-11-24T08:17:43.218210+0000","last_peered":"2023-11-24T08:17:43.218210+0000","last_clean":"2023-11-24T08:17:43.218210+0000","last_became_active":"2023-11-24T08:17:41.273471+0000","last_became_peered":"2023-11-24T08:17:41.273471+0000","last_unstale":"2023-11-24T08:17:43.218210+0000","last_undegraded":"2023-11-24T08:17:43.218210+0000","last_fullsized":"2023-11-24T08:17:43.218210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:26.875145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219289+0000","last_change":"2023-11-24T08:17:41.274359+0000","last_active":"2023-11-24T08:17:43.219289+0000","last_peered":"2023-11-24T08:17:43.219289+0000","last_clean":"2023-11-24T08:17:43.219289+0000","last_became_active":"2023-11-24T08:17:41.274089+0000","last_became_peered":"2023-11-24T08:17:41.274089+0000","last_unstale":"2023-11-24T08:17:43.219289+0000","last_undegraded":"2023-11-24T08:17:43.219289+0000","last_fullsized":"2023-11-24T08:17:43.219289+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:15:58.402192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218208+0000","last_change":"2023-11-24T08:17:41.281157+0000","last_active":"2023-11-24T08:17:43.218208+0000","last_peered":"2023-11-24T08:17:43.218208+0000","last_clean":"2023-11-24T08:17:43.218208+0000","last_became_active":"2023-11-24T08:17:41.280841+0000","last_became_peered":"2023-11-24T08:17:41.280841+0000","last_unstale":"2023-11-24T08:17:43.218208+0000","last_undegraded":"2023-11-24T08:17:43.218208+0000","last_fullsized":"2023-11-24T08:17:43.218208+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:48.953924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277761+0000","last_change":"2023-11-24T08:17:41.274379+0000","last_active":"2023-11-24T08:17:42.277761+0000","last_peered":"2023-11-24T08:17:42.277761+0000","last_clean":"2023-11-24T08:17:42.277761+0000","last_became_active":"2023-11-24T08:17:41.273291+0000","last_became_peered":"2023-11-24T08:17:41.273291+0000","last_unstale":"2023-11-24T08:17:42.277761+0000","last_undegraded":"2023-11-24T08:17:42.277761+0000","last_fullsized":"2023-11-24T08:17:42.277761+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:11.968663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216694+0000","last_change":"2023-11-24T08:17:41.289045+0000","last_active":"2023-11-24T08:17:43.216694+0000","last_peered":"2023-11-24T08:17:43.216694+0000","last_clean":"2023-11-24T08:17:43.216694+0000","last_became_active":"2023-11-24T08:17:41.288003+0000","last_became_peered":"2023-11-24T08:17:41.288003+0000","last_unstale":"2023-11-24T08:17:43.216694+0000","last_undegraded":"2023-11-24T08:17:43.216694+0000","last_fullsized":"2023-11-24T08:17:43.216694+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:53.416104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276606+0000","last_change":"2023-11-24T08:17:41.275711+0000","last_active":"2023-11-24T08:17:42.276606+0000","last_peered":"2023-11-24T08:17:42.276606+0000","last_clean":"2023-11-24T08:17:42.276606+0000","last_became_active":"2023-11-24T08:17:41.275431+0000","last_became_peered":"2023-11-24T08:17:41.275431+0000","last_unstale":"2023-11-24T08:17:42.276606+0000","last_undegraded":"2023-11-24T08:17:42.276606+0000","last_fullsized":"2023-11-24T08:17:42.276606+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:28.931916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221242+0000","last_change":"2023-11-24T08:17:41.284218+0000","last_active":"2023-11-24T08:17:43.221242+0000","last_peered":"2023-11-24T08:17:43.221242+0000","last_clean":"2023-11-24T08:17:43.221242+0000","last_became_active":"2023-11-24T08:17:41.283176+0000","last_became_peered":"2023-11-24T08:17:41.283176+0000","last_unstale":"2023-11-24T08:17:43.221242+0000","last_undegraded":"2023-11-24T08:17:43.221242+0000","last_fullsized":"2023-11-24T08:17:43.221242+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:37.776952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275711+0000","last_change":"2023-11-24T08:17:41.275576+0000","last_active":"2023-11-24T08:17:42.275711+0000","last_peered":"2023-11-24T08:17:42.275711+0000","last_clean":"2023-11-24T08:17:42.275711+0000","last_became_active":"2023-11-24T08:17:41.275285+0000","last_became_peered":"2023-11-24T08:17:41.275285+0000","last_unstale":"2023-11-24T08:17:42.275711+0000","last_undegraded":"2023-11-24T08:17:42.275711+0000","last_fullsized":"2023-11-24T08:17:42.275711+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:14.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275222+0000","last_change":"2023-11-24T08:17:41.268935+0000","last_active":"2023-11-24T08:17:42.275222+0000","last_peered":"2023-11-24T08:17:42.275222+0000","last_clean":"2023-11-24T08:17:42.275222+0000","last_became_active":"2023-11-24T08:17:41.268428+0000","last_became_peered":"2023-11-24T08:17:41.268428+0000","last_unstale":"2023-11-24T08:17:42.275222+0000","last_undegraded":"2023-11-24T08:17:42.275222+0000","last_fullsized":"2023-11-24T08:17:42.275222+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275862+0000","last_change":"2023-11-24T08:17:41.275641+0000","last_active":"2023-11-24T08:17:42.275862+0000","last_peered":"2023-11-24T08:17:42.275862+0000","last_clean":"2023-11-24T08:17:42.275862+0000","last_became_active":"2023-11-24T08:17:41.275347+0000","last_became_peered":"2023-11-24T08:17:41.275347+0000","last_unstale":"2023-11-24T08:17:42.275862+0000","last_undegraded":"2023-11-24T08:17:42.275862+0000","last_fullsized":"2023-11-24T08:17:42.275862+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:16.806161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218054+0000","last_change":"2023-11-24T08:17:41.265592+0000","last_active":"2023-11-24T08:17:43.218054+0000","last_peered":"2023-11-24T08:17:43.218054+0000","last_clean":"2023-11-24T08:17:43.218054+0000","last_became_active":"2023-11-24T08:17:41.265173+0000","last_became_peered":"2023-11-24T08:17:41.265173+0000","last_unstale":"2023-11-24T08:17:43.218054+0000","last_undegraded":"2023-11-24T08:17:43.218054+0000","last_fullsized":"2023-11-24T08:17:43.218054+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:46.526325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216744+0000","last_change":"2023-11-24T08:17:41.288145+0000","last_active":"2023-11-24T08:17:43.216744+0000","last_peered":"2023-11-24T08:17:43.216744+0000","last_clean":"2023-11-24T08:17:43.216744+0000","last_became_active":"2023-11-24T08:17:41.287654+0000","last_became_peered":"2023-11-24T08:17:41.287654+0000","last_unstale":"2023-11-24T08:17:43.216744+0000","last_undegraded":"2023-11-24T08:17:43.216744+0000","last_fullsized":"2023-11-24T08:17:43.216744+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:18:18.585408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"161'4","reported_seq":16,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217473+0000","last_change":"2023-11-24T08:17:40.274270+0000","last_active":"2023-11-24T08:17:43.217473+0000","last_peered":"2023-11-24T08:17:43.217473+0000","last_clean":"2023-11-24T08:17:43.217473+0000","last_became_active":"2023-11-24T08:17:40.274039+0000","last_became_peered":"2023-11-24T08:17:40.274039+0000","last_unstale":"2023-11-24T08:17:43.217473+0000","last_undegraded":"2023-11-24T08:17:43.217473+0000","last_fullsized":"2023-11-24T08:17:43.217473+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:44:52.323526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277764+0000","last_change":"2023-11-24T08:17:41.261553+0000","last_active":"2023-11-24T08:17:42.277764+0000","last_peered":"2023-11-24T08:17:42.277764+0000","last_clean":"2023-11-24T08:17:42.277764+0000","last_became_active":"2023-11-24T08:17:41.260308+0000","last_became_peered":"2023-11-24T08:17:41.260308+0000","last_unstale":"2023-11-24T08:17:42.277764+0000","last_undegraded":"2023-11-24T08:17:42.277764+0000","last_fullsized":"2023-11-24T08:17:42.277764+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:32.835769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.278023+0000","last_change":"2023-11-24T08:17:40.264001+0000","last_active":"2023-11-24T08:17:42.278023+0000","last_peered":"2023-11-24T08:17:42.278023+0000","last_clean":"2023-11-24T08:17:42.278023+0000","last_became_active":"2023-11-24T08:17:40.263573+0000","last_became_peered":"2023-11-24T08:17:40.263573+0000","last_unstale":"2023-11-24T08:17:42.278023+0000","last_undegraded":"2023-11-24T08:17:42.278023+0000","last_fullsized":"2023-11-24T08:17:42.278023+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:38.944147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219939+0000","last_change":"2023-11-24T08:17:41.280993+0000","last_active":"2023-11-24T08:17:43.219939+0000","last_peered":"2023-11-24T08:17:43.219939+0000","last_clean":"2023-11-24T08:17:43.219939+0000","last_became_active":"2023-11-24T08:17:41.280581+0000","last_became_peered":"2023-11-24T08:17:41.280581+0000","last_unstale":"2023-11-24T08:17:43.219939+0000","last_undegraded":"2023-11-24T08:17:43.219939+0000","last_fullsized":"2023-11-24T08:17:43.219939+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:02.387416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219666+0000","last_change":"2023-11-24T08:17:40.268530+0000","last_active":"2023-11-24T08:17:43.219666+0000","last_peered":"2023-11-24T08:17:43.219666+0000","last_clean":"2023-11-24T08:17:43.219666+0000","last_became_active":"2023-11-24T08:17:40.267997+0000","last_became_peered":"2023-11-24T08:17:40.267997+0000","last_unstale":"2023-11-24T08:17:43.219666+0000","last_undegraded":"2023-11-24T08:17:43.219666+0000","last_fullsized":"2023-11-24T08:17:43.219666+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:47.937104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276783+0000","last_change":"2023-11-24T08:17:41.275182+0000","last_active":"2023-11-24T08:17:42.276783+0000","last_peered":"2023-11-24T08:17:42.276783+0000","last_clean":"2023-11-24T08:17:42.276783+0000","last_became_active":"2023-11-24T08:17:41.273716+0000","last_became_peered":"2023-11-24T08:17:41.273716+0000","last_unstale":"2023-11-24T08:17:42.276783+0000","last_undegraded":"2023-11-24T08:17:42.276783+0000","last_fullsized":"2023-11-24T08:17:42.276783+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:07:04.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221369+0000","last_change":"2023-11-24T08:17:40.266226+0000","last_active":"2023-11-24T08:17:43.221369+0000","last_peered":"2023-11-24T08:17:43.221369+0000","last_clean":"2023-11-24T08:17:43.221369+0000","last_became_active":"2023-11-24T08:17:40.265926+0000","last_became_peered":"2023-11-24T08:17:40.265926+0000","last_unstale":"2023-11-24T08:17:43.221369+0000","last_undegraded":"2023-11-24T08:17:43.221369+0000","last_fullsized":"2023-11-24T08:17:43.221369+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:07:38.944147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216819+0000","last_change":"2023-11-24T08:17:41.286252+0000","last_active":"2023-11-24T08:17:43.216819+0000","last_peered":"2023-11-24T08:17:43.216819+0000","last_clean":"2023-11-24T08:17:43.216819+0000","last_became_active":"2023-11-24T08:17:41.285964+0000","last_became_peered":"2023-11-24T08:17:41.285964+0000","last_unstale":"2023-11-24T08:17:43.216819+0000","last_undegraded":"2023-11-24T08:17:43.216819+0000","last_fullsized":"2023-11-24T08:17:43.216819+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:36:38.121400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275689+0000","last_change":"2023-11-24T08:17:40.267090+0000","last_active":"2023-11-24T08:17:42.275689+0000","last_peered":"2023-11-24T08:17:42.275689+0000","last_clean":"2023-11-24T08:17:42.275689+0000","last_became_active":"2023-11-24T08:17:40.266810+0000","last_became_peered":"2023-11-24T08:17:40.266810+0000","last_unstale":"2023-11-24T08:17:42.275689+0000","last_undegraded":"2023-11-24T08:17:42.275689+0000","last_fullsized":"2023-11-24T08:17:42.275689+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:20.886415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219835+0000","last_change":"2023-11-24T08:17:41.272717+0000","last_active":"2023-11-24T08:17:43.219835+0000","last_peered":"2023-11-24T08:17:43.219835+0000","last_clean":"2023-11-24T08:17:43.219835+0000","last_became_active":"2023-11-24T08:17:41.272016+0000","last_became_peered":"2023-11-24T08:17:41.272016+0000","last_unstale":"2023-11-24T08:17:43.219835+0000","last_undegraded":"2023-11-24T08:17:43.219835+0000","last_fullsized":"2023-11-24T08:17:43.219835+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:00.248461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218285+0000","last_change":"2023-11-24T08:17:40.264261+0000","last_active":"2023-11-24T08:17:43.218285+0000","last_peered":"2023-11-24T08:17:43.218285+0000","last_clean":"2023-11-24T08:17:43.218285+0000","last_became_active":"2023-11-24T08:17:40.263850+0000","last_became_peered":"2023-11-24T08:17:40.263850+0000","last_unstale":"2023-11-24T08:17:43.218285+0000","last_undegraded":"2023-11-24T08:17:43.218285+0000","last_fullsized":"2023-11-24T08:17:43.218285+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:35.820122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217323+0000","last_change":"2023-11-24T08:17:41.274369+0000","last_active":"2023-11-24T08:17:43.217323+0000","last_peered":"2023-11-24T08:17:43.217323+0000","last_clean":"2023-11-24T08:17:43.217323+0000","last_became_active":"2023-11-24T08:17:41.273693+0000","last_became_peered":"2023-11-24T08:17:41.273693+0000","last_unstale":"2023-11-24T08:17:43.217323+0000","last_undegraded":"2023-11-24T08:17:43.217323+0000","last_fullsized":"2023-11-24T08:17:43.217323+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:09:10.670438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218133+0000","last_change":"2023-11-24T08:17:40.266986+0000","last_active":"2023-11-24T08:17:43.218133+0000","last_peered":"2023-11-24T08:17:43.218133+0000","last_clean":"2023-11-24T08:17:43.218133+0000","last_became_active":"2023-11-24T08:17:40.266606+0000","last_became_peered":"2023-11-24T08:17:40.266606+0000","last_unstale":"2023-11-24T08:17:43.218133+0000","last_undegraded":"2023-11-24T08:17:43.218133+0000","last_fullsized":"2023-11-24T08:17:43.218133+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:45.127590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219301+0000","last_change":"2023-11-24T08:17:41.279975+0000","last_active":"2023-11-24T08:17:43.219301+0000","last_peered":"2023-11-24T08:17:43.219301+0000","last_clean":"2023-11-24T08:17:43.219301+0000","last_became_active":"2023-11-24T08:17:41.279469+0000","last_became_peered":"2023-11-24T08:17:41.279469+0000","last_unstale":"2023-11-24T08:17:43.219301+0000","last_undegraded":"2023-11-24T08:17:43.219301+0000","last_fullsized":"2023-11-24T08:17:43.219301+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:43.843570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215999+0000","last_change":"2023-11-24T08:17:40.260966+0000","last_active":"2023-11-24T08:17:43.215999+0000","last_peered":"2023-11-24T08:17:43.215999+0000","last_clean":"2023-11-24T08:17:43.215999+0000","last_became_active":"2023-11-24T08:17:40.260688+0000","last_became_peered":"2023-11-24T08:17:40.260688+0000","last_unstale":"2023-11-24T08:17:43.215999+0000","last_undegraded":"2023-11-24T08:17:43.215999+0000","last_fullsized":"2023-11-24T08:17:43.215999+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:14:16.120418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218933+0000","last_change":"2023-11-24T08:17:41.280361+0000","last_active":"2023-11-24T08:17:43.218933+0000","last_peered":"2023-11-24T08:17:43.218933+0000","last_clean":"2023-11-24T08:17:43.218933+0000","last_became_active":"2023-11-24T08:17:41.275998+0000","last_became_peered":"2023-11-24T08:17:41.275998+0000","last_unstale":"2023-11-24T08:17:43.218933+0000","last_undegraded":"2023-11-24T08:17:43.218933+0000","last_fullsized":"2023-11-24T08:17:43.218933+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:26.366513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220973+0000","last_change":"2023-11-24T08:17:40.264680+0000","last_active":"2023-11-24T08:17:43.220973+0000","last_peered":"2023-11-24T08:17:43.220973+0000","last_clean":"2023-11-24T08:17:43.220973+0000","last_became_active":"2023-11-24T08:17:40.264406+0000","last_became_peered":"2023-11-24T08:17:40.264406+0000","last_unstale":"2023-11-24T08:17:43.220973+0000","last_undegraded":"2023-11-24T08:17:43.220973+0000","last_fullsized":"2023-11-24T08:17:43.220973+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:03:41.552001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218210+0000","last_change":"2023-11-24T08:17:41.273748+0000","last_active":"2023-11-24T08:17:43.218210+0000","last_peered":"2023-11-24T08:17:43.218210+0000","last_clean":"2023-11-24T08:17:43.218210+0000","last_became_active":"2023-11-24T08:17:41.273418+0000","last_became_peered":"2023-11-24T08:17:41.273418+0000","last_unstale":"2023-11-24T08:17:43.218210+0000","last_undegraded":"2023-11-24T08:17:43.218210+0000","last_fullsized":"2023-11-24T08:17:43.218210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:44:53.340346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"161'3","reported_seq":15,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220441+0000","last_change":"2023-11-24T08:17:40.263032+0000","last_active":"2023-11-24T08:17:43.220441+0000","last_peered":"2023-11-24T08:17:43.220441+0000","last_clean":"2023-11-24T08:17:43.220441+0000","last_became_active":"2023-11-24T08:17:40.262622+0000","last_became_peered":"2023-11-24T08:17:40.262622+0000","last_unstale":"2023-11-24T08:17:43.220441+0000","last_undegraded":"2023-11-24T08:17:43.220441+0000","last_fullsized":"2023-11-24T08:17:43.220441+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:52.599545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.278430+0000","last_change":"2023-11-24T08:17:41.275011+0000","last_active":"2023-11-24T08:17:42.278430+0000","last_peered":"2023-11-24T08:17:42.278430+0000","last_clean":"2023-11-24T08:17:42.278430+0000","last_became_active":"2023-11-24T08:17:41.274717+0000","last_became_peered":"2023-11-24T08:17:41.274717+0000","last_unstale":"2023-11-24T08:17:42.278430+0000","last_undegraded":"2023-11-24T08:17:42.278430+0000","last_fullsized":"2023-11-24T08:17:42.278430+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:13.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"161'6","reported_seq":18,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277235+0000","last_change":"2023-11-24T08:17:40.267331+0000","last_active":"2023-11-24T08:17:42.277235+0000","last_peered":"2023-11-24T08:17:42.277235+0000","last_clean":"2023-11-24T08:17:42.277235+0000","last_became_active":"2023-11-24T08:17:40.265931+0000","last_became_peered":"2023-11-24T08:17:40.265931+0000","last_unstale":"2023-11-24T08:17:42.277235+0000","last_undegraded":"2023-11-24T08:17:42.277235+0000","last_fullsized":"2023-11-24T08:17:42.277235+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:35.820122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218077+0000","last_change":"2023-11-24T08:17:41.273218+0000","last_active":"2023-11-24T08:17:43.218077+0000","last_peered":"2023-11-24T08:17:43.218077+0000","last_clean":"2023-11-24T08:17:43.218077+0000","last_became_active":"2023-11-24T08:17:41.272827+0000","last_became_peered":"2023-11-24T08:17:41.272827+0000","last_unstale":"2023-11-24T08:17:43.218077+0000","last_undegraded":"2023-11-24T08:17:43.218077+0000","last_fullsized":"2023-11-24T08:17:43.218077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:29:05.900760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219942+0000","last_change":"2023-11-24T08:17:40.262554+0000","last_active":"2023-11-24T08:17:43.219942+0000","last_peered":"2023-11-24T08:17:43.219942+0000","last_clean":"2023-11-24T08:17:43.219942+0000","last_became_active":"2023-11-24T08:17:40.260612+0000","last_became_peered":"2023-11-24T08:17:40.260612+0000","last_unstale":"2023-11-24T08:17:43.219942+0000","last_undegraded":"2023-11-24T08:17:43.219942+0000","last_fullsized":"2023-11-24T08:17:43.219942+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:25.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216642+0000","last_change":"2023-11-24T08:17:41.279064+0000","last_active":"2023-11-24T08:17:43.216642+0000","last_peered":"2023-11-24T08:17:43.216642+0000","last_clean":"2023-11-24T08:17:43.216642+0000","last_became_active":"2023-11-24T08:17:41.278709+0000","last_became_peered":"2023-11-24T08:17:41.278709+0000","last_unstale":"2023-11-24T08:17:43.216642+0000","last_undegraded":"2023-11-24T08:17:43.216642+0000","last_fullsized":"2023-11-24T08:17:43.216642+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:47:14.293972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218494+0000","last_change":"2023-11-24T08:17:40.261755+0000","last_active":"2023-11-24T08:17:43.218494+0000","last_peered":"2023-11-24T08:17:43.218494+0000","last_clean":"2023-11-24T08:17:43.218494+0000","last_became_active":"2023-11-24T08:17:40.261454+0000","last_became_peered":"2023-11-24T08:17:40.261454+0000","last_unstale":"2023-11-24T08:17:43.218494+0000","last_undegraded":"2023-11-24T08:17:43.218494+0000","last_fullsized":"2023-11-24T08:17:43.218494+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:14:53.736584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217692+0000","last_change":"2023-11-24T08:17:41.274521+0000","last_active":"2023-11-24T08:17:43.217692+0000","last_peered":"2023-11-24T08:17:43.217692+0000","last_clean":"2023-11-24T08:17:43.217692+0000","last_became_active":"2023-11-24T08:17:41.274072+0000","last_became_peered":"2023-11-24T08:17:41.274072+0000","last_unstale":"2023-11-24T08:17:43.217692+0000","last_undegraded":"2023-11-24T08:17:43.217692+0000","last_fullsized":"2023-11-24T08:17:43.217692+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:16.806161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218288+0000","last_change":"2023-11-24T08:17:40.268091+0000","last_active":"2023-11-24T08:17:43.218288+0000","last_peered":"2023-11-24T08:17:43.218288+0000","last_clean":"2023-11-24T08:17:43.218288+0000","last_became_active":"2023-11-24T08:17:40.267732+0000","last_became_peered":"2023-11-24T08:17:40.267732+0000","last_unstale":"2023-11-24T08:17:43.218288+0000","last_undegraded":"2023-11-24T08:17:43.218288+0000","last_fullsized":"2023-11-24T08:17:43.218288+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:04.952870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274568+0000","last_change":"2023-11-24T08:17:41.271855+0000","last_active":"2023-11-24T08:17:42.274568+0000","last_peered":"2023-11-24T08:17:42.274568+0000","last_clean":"2023-11-24T08:17:42.274568+0000","last_became_active":"2023-11-24T08:17:41.271560+0000","last_became_peered":"2023-11-24T08:17:41.271560+0000","last_unstale":"2023-11-24T08:17:42.274568+0000","last_undegraded":"2023-11-24T08:17:42.274568+0000","last_fullsized":"2023-11-24T08:17:42.274568+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:05:25.595604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"161'8","reported_seq":20,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275084+0000","last_change":"2023-11-24T08:17:40.267502+0000","last_active":"2023-11-24T08:17:42.275084+0000","last_peered":"2023-11-24T08:17:42.275084+0000","last_clean":"2023-11-24T08:17:42.275084+0000","last_became_active":"2023-11-24T08:17:40.267104+0000","last_became_peered":"2023-11-24T08:17:40.267104+0000","last_unstale":"2023-11-24T08:17:42.275084+0000","last_undegraded":"2023-11-24T08:17:42.275084+0000","last_fullsized":"2023-11-24T08:17:42.275084+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:10.951843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220844+0000","last_change":"2023-11-24T08:17:41.273337+0000","last_active":"2023-11-24T08:17:43.220844+0000","last_peered":"2023-11-24T08:17:43.220844+0000","last_clean":"2023-11-24T08:17:43.220844+0000","last_became_active":"2023-11-24T08:17:41.273017+0000","last_became_peered":"2023-11-24T08:17:41.273017+0000","last_unstale":"2023-11-24T08:17:43.220844+0000","last_undegraded":"2023-11-24T08:17:43.220844+0000","last_fullsized":"2023-11-24T08:17:43.220844+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:13.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277815+0000","last_change":"2023-11-24T08:17:40.268858+0000","last_active":"2023-11-24T08:17:42.277815+0000","last_peered":"2023-11-24T08:17:42.277815+0000","last_clean":"2023-11-24T08:17:42.277815+0000","last_became_active":"2023-11-24T08:17:40.268026+0000","last_became_peered":"2023-11-24T08:17:40.268026+0000","last_unstale":"2023-11-24T08:17:42.277815+0000","last_undegraded":"2023-11-24T08:17:42.277815+0000","last_fullsized":"2023-11-24T08:17:42.277815+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:32:38.502188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219132+0000","last_change":"2023-11-24T08:17:41.257281+0000","last_active":"2023-11-24T08:17:43.219132+0000","last_peered":"2023-11-24T08:17:43.219132+0000","last_clean":"2023-11-24T08:17:43.219132+0000","last_became_active":"2023-11-24T08:17:41.256350+0000","last_became_peered":"2023-11-24T08:17:41.256350+0000","last_unstale":"2023-11-24T08:17:43.219132+0000","last_undegraded":"2023-11-24T08:17:43.219132+0000","last_fullsized":"2023-11-24T08:17:43.219132+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:21.357162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276320+0000","last_change":"2023-11-24T08:17:40.261216+0000","last_active":"2023-11-24T08:17:42.276320+0000","last_peered":"2023-11-24T08:17:42.276320+0000","last_clean":"2023-11-24T08:17:42.276320+0000","last_became_active":"2023-11-24T08:17:40.260897+0000","last_became_peered":"2023-11-24T08:17:40.260897+0000","last_unstale":"2023-11-24T08:17:42.276320+0000","last_undegraded":"2023-11-24T08:17:42.276320+0000","last_fullsized":"2023-11-24T08:17:42.276320+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:50.771442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.221174+0000","last_change":"2023-11-24T08:17:41.274336+0000","last_active":"2023-11-24T08:17:43.221174+0000","last_peered":"2023-11-24T08:17:43.221174+0000","last_clean":"2023-11-24T08:17:43.221174+0000","last_became_active":"2023-11-24T08:17:41.273028+0000","last_became_peered":"2023-11-24T08:17:41.273028+0000","last_unstale":"2023-11-24T08:17:43.221174+0000","last_undegraded":"2023-11-24T08:17:43.221174+0000","last_fullsized":"2023-11-24T08:17:43.221174+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:01:41.929632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"161'1","reported_seq":13,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217250+0000","last_change":"2023-11-24T08:17:40.269351+0000","last_active":"2023-11-24T08:17:43.217250+0000","last_peered":"2023-11-24T08:17:43.217250+0000","last_clean":"2023-11-24T08:17:43.217250+0000","last_became_active":"2023-11-24T08:17:40.268930+0000","last_became_peered":"2023-11-24T08:17:40.268930+0000","last_unstale":"2023-11-24T08:17:43.217250+0000","last_undegraded":"2023-11-24T08:17:43.217250+0000","last_fullsized":"2023-11-24T08:17:43.217250+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:27.968787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216522+0000","last_change":"2023-11-24T08:17:41.284621+0000","last_active":"2023-11-24T08:17:43.216522+0000","last_peered":"2023-11-24T08:17:43.216522+0000","last_clean":"2023-11-24T08:17:43.216522+0000","last_became_active":"2023-11-24T08:17:41.282874+0000","last_became_peered":"2023-11-24T08:17:41.282874+0000","last_unstale":"2023-11-24T08:17:43.216522+0000","last_undegraded":"2023-11-24T08:17:43.216522+0000","last_fullsized":"2023-11-24T08:17:43.216522+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:46:29.875536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217244+0000","last_change":"2023-11-24T08:17:40.268661+0000","last_active":"2023-11-24T08:17:43.217244+0000","last_peered":"2023-11-24T08:17:43.217244+0000","last_clean":"2023-11-24T08:17:43.217244+0000","last_became_active":"2023-11-24T08:17:40.268296+0000","last_became_peered":"2023-11-24T08:17:40.268296+0000","last_unstale":"2023-11-24T08:17:43.217244+0000","last_undegraded":"2023-11-24T08:17:43.217244+0000","last_fullsized":"2023-11-24T08:17:43.217244+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:37.201642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275210+0000","last_change":"2023-11-24T08:17:41.271880+0000","last_active":"2023-11-24T08:17:42.275210+0000","last_peered":"2023-11-24T08:17:42.275210+0000","last_clean":"2023-11-24T08:17:42.275210+0000","last_became_active":"2023-11-24T08:17:41.270914+0000","last_became_peered":"2023-11-24T08:17:41.270914+0000","last_unstale":"2023-11-24T08:17:42.275210+0000","last_undegraded":"2023-11-24T08:17:42.275210+0000","last_fullsized":"2023-11-24T08:17:42.275210+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:36.806072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219964+0000","last_change":"2023-11-24T08:17:40.266769+0000","last_active":"2023-11-24T08:17:43.219964+0000","last_peered":"2023-11-24T08:17:43.219964+0000","last_clean":"2023-11-24T08:17:43.219964+0000","last_became_active":"2023-11-24T08:17:40.266312+0000","last_became_peered":"2023-11-24T08:17:40.266312+0000","last_unstale":"2023-11-24T08:17:43.219964+0000","last_undegraded":"2023-11-24T08:17:43.219964+0000","last_fullsized":"2023-11-24T08:17:43.219964+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:36.597391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219141+0000","last_change":"2023-11-24T08:17:41.285206+0000","last_active":"2023-11-24T08:17:43.219141+0000","last_peered":"2023-11-24T08:17:43.219141+0000","last_clean":"2023-11-24T08:17:43.219141+0000","last_became_active":"2023-11-24T08:17:41.282438+0000","last_became_peered":"2023-11-24T08:17:41.282438+0000","last_unstale":"2023-11-24T08:17:43.219141+0000","last_undegraded":"2023-11-24T08:17:43.219141+0000","last_fullsized":"2023-11-24T08:17:43.219141+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:05.238469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216765+0000","last_change":"2023-11-24T08:17:40.268374+0000","last_active":"2023-11-24T08:17:43.216765+0000","last_peered":"2023-11-24T08:17:43.216765+0000","last_clean":"2023-11-24T08:17:43.216765+0000","last_became_active":"2023-11-24T08:17:40.268078+0000","last_became_peered":"2023-11-24T08:17:40.268078+0000","last_unstale":"2023-11-24T08:17:43.216765+0000","last_undegraded":"2023-11-24T08:17:43.216765+0000","last_fullsized":"2023-11-24T08:17:43.216765+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:46.674445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.217622+0000","last_change":"2023-11-24T08:17:41.272699+0000","last_active":"2023-11-24T08:17:43.217622+0000","last_peered":"2023-11-24T08:17:43.217622+0000","last_clean":"2023-11-24T08:17:43.217622+0000","last_became_active":"2023-11-24T08:17:41.272366+0000","last_became_peered":"2023-11-24T08:17:41.272366+0000","last_unstale":"2023-11-24T08:17:43.217622+0000","last_undegraded":"2023-11-24T08:17:43.217622+0000","last_fullsized":"2023-11-24T08:17:43.217622+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:42.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218510+0000","last_change":"2023-11-24T08:17:40.263781+0000","last_active":"2023-11-24T08:17:43.218510+0000","last_peered":"2023-11-24T08:17:43.218510+0000","last_clean":"2023-11-24T08:17:43.218510+0000","last_became_active":"2023-11-24T08:17:40.263468+0000","last_became_peered":"2023-11-24T08:17:40.263468+0000","last_unstale":"2023-11-24T08:17:43.218510+0000","last_undegraded":"2023-11-24T08:17:43.218510+0000","last_fullsized":"2023-11-24T08:17:43.218510+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219564+0000","last_change":"2023-11-24T08:17:41.273505+0000","last_active":"2023-11-24T08:17:43.219564+0000","last_peered":"2023-11-24T08:17:43.219564+0000","last_clean":"2023-11-24T08:17:43.219564+0000","last_became_active":"2023-11-24T08:17:41.273251+0000","last_became_peered":"2023-11-24T08:17:41.273251+0000","last_unstale":"2023-11-24T08:17:43.219564+0000","last_undegraded":"2023-11-24T08:17:43.219564+0000","last_fullsized":"2023-11-24T08:17:43.219564+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:19:10.931535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218222+0000","last_change":"2023-11-24T08:17:40.270845+0000","last_active":"2023-11-24T08:17:43.218222+0000","last_peered":"2023-11-24T08:17:43.218222+0000","last_clean":"2023-11-24T08:17:43.218222+0000","last_became_active":"2023-11-24T08:17:40.270368+0000","last_became_peered":"2023-11-24T08:17:40.270368+0000","last_unstale":"2023-11-24T08:17:43.218222+0000","last_undegraded":"2023-11-24T08:17:43.218222+0000","last_fullsized":"2023-11-24T08:17:43.218222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:51.014940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219323+0000","last_change":"2023-11-24T08:17:41.274658+0000","last_active":"2023-11-24T08:17:43.219323+0000","last_peered":"2023-11-24T08:17:43.219323+0000","last_clean":"2023-11-24T08:17:43.219323+0000","last_became_active":"2023-11-24T08:17:41.274084+0000","last_became_peered":"2023-11-24T08:17:41.274084+0000","last_unstale":"2023-11-24T08:17:43.219323+0000","last_undegraded":"2023-11-24T08:17:43.219323+0000","last_fullsized":"2023-11-24T08:17:43.219323+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:45:09.576180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"161'2","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219310+0000","last_change":"2023-11-24T08:17:40.271571+0000","last_active":"2023-11-24T08:17:43.219310+0000","last_peered":"2023-11-24T08:17:43.219310+0000","last_clean":"2023-11-24T08:17:43.219310+0000","last_became_active":"2023-11-24T08:17:40.271345+0000","last_became_peered":"2023-11-24T08:17:40.271345+0000","last_unstale":"2023-11-24T08:17:43.219310+0000","last_undegraded":"2023-11-24T08:17:43.219310+0000","last_fullsized":"2023-11-24T08:17:43.219310+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:46.674445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.216823+0000","last_change":"2023-11-24T08:17:41.274647+0000","last_active":"2023-11-24T08:17:43.216823+0000","last_peered":"2023-11-24T08:17:43.216823+0000","last_clean":"2023-11-24T08:17:43.216823+0000","last_became_active":"2023-11-24T08:17:41.274377+0000","last_became_peered":"2023-11-24T08:17:41.274377+0000","last_unstale":"2023-11-24T08:17:43.216823+0000","last_undegraded":"2023-11-24T08:17:43.216823+0000","last_fullsized":"2023-11-24T08:17:43.216823+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:06:25.162702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.276564+0000","last_change":"2023-11-24T08:17:40.264532+0000","last_active":"2023-11-24T08:17:42.276564+0000","last_peered":"2023-11-24T08:17:42.276564+0000","last_clean":"2023-11-24T08:17:42.276564+0000","last_became_active":"2023-11-24T08:17:40.264219+0000","last_became_peered":"2023-11-24T08:17:40.264219+0000","last_unstale":"2023-11-24T08:17:42.276564+0000","last_undegraded":"2023-11-24T08:17:42.276564+0000","last_fullsized":"2023-11-24T08:17:42.276564+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:36.570984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.220346+0000","last_change":"2023-11-24T08:17:41.280834+0000","last_active":"2023-11-24T08:17:43.220346+0000","last_peered":"2023-11-24T08:17:43.220346+0000","last_clean":"2023-11-24T08:17:43.220346+0000","last_became_active":"2023-11-24T08:17:41.280250+0000","last_became_peered":"2023-11-24T08:17:41.280250+0000","last_unstale":"2023-11-24T08:17:43.220346+0000","last_undegraded":"2023-11-24T08:17:43.220346+0000","last_fullsized":"2023-11-24T08:17:43.220346+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:16:37.412337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.219108+0000","last_change":"2023-11-24T08:17:40.268586+0000","last_active":"2023-11-24T08:17:43.219108+0000","last_peered":"2023-11-24T08:17:43.219108+0000","last_clean":"2023-11-24T08:17:43.219108+0000","last_became_active":"2023-11-24T08:17:40.267562+0000","last_became_peered":"2023-11-24T08:17:40.267562+0000","last_unstale":"2023-11-24T08:17:43.219108+0000","last_undegraded":"2023-11-24T08:17:43.219108+0000","last_fullsized":"2023-11-24T08:17:43.219108+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:40:10.951843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218731+0000","last_change":"2023-11-24T08:17:41.264880+0000","last_active":"2023-11-24T08:17:43.218731+0000","last_peered":"2023-11-24T08:17:43.218731+0000","last_clean":"2023-11-24T08:17:43.218731+0000","last_became_active":"2023-11-24T08:17:41.264462+0000","last_became_peered":"2023-11-24T08:17:41.264462+0000","last_unstale":"2023-11-24T08:17:43.218731+0000","last_undegraded":"2023-11-24T08:17:43.218731+0000","last_fullsized":"2023-11-24T08:17:43.218731+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:13.172494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218362+0000","last_change":"2023-11-24T08:17:40.265481+0000","last_active":"2023-11-24T08:17:43.218362+0000","last_peered":"2023-11-24T08:17:43.218362+0000","last_clean":"2023-11-24T08:17:43.218362+0000","last_became_active":"2023-11-24T08:17:40.262585+0000","last_became_peered":"2023-11-24T08:17:40.262585+0000","last_unstale":"2023-11-24T08:17:43.218362+0000","last_undegraded":"2023-11-24T08:17:43.218362+0000","last_fullsized":"2023-11-24T08:17:43.218362+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:44.339007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218326+0000","last_change":"2023-11-24T08:17:41.281684+0000","last_active":"2023-11-24T08:17:43.218326+0000","last_peered":"2023-11-24T08:17:43.218326+0000","last_clean":"2023-11-24T08:17:43.218326+0000","last_became_active":"2023-11-24T08:17:41.281410+0000","last_became_peered":"2023-11-24T08:17:41.281410+0000","last_unstale":"2023-11-24T08:17:43.218326+0000","last_undegraded":"2023-11-24T08:17:43.218326+0000","last_fullsized":"2023-11-24T08:17:43.218326+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:56:37.604937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277004+0000","last_change":"2023-11-24T08:17:40.259164+0000","last_active":"2023-11-24T08:17:42.277004+0000","last_peered":"2023-11-24T08:17:42.277004+0000","last_clean":"2023-11-24T08:17:42.277004+0000","last_became_active":"2023-11-24T08:17:40.258813+0000","last_became_peered":"2023-11-24T08:17:40.258813+0000","last_unstale":"2023-11-24T08:17:42.277004+0000","last_undegraded":"2023-11-24T08:17:42.277004+0000","last_fullsized":"2023-11-24T08:17:42.277004+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:26:04.952870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277052+0000","last_change":"2023-11-24T08:17:41.274538+0000","last_active":"2023-11-24T08:17:42.277052+0000","last_peered":"2023-11-24T08:17:42.277052+0000","last_clean":"2023-11-24T08:17:42.277052+0000","last_became_active":"2023-11-24T08:17:41.272127+0000","last_became_peered":"2023-11-24T08:17:41.272127+0000","last_unstale":"2023-11-24T08:17:42.277052+0000","last_undegraded":"2023-11-24T08:17:42.277052+0000","last_fullsized":"2023-11-24T08:17:42.277052+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:34:00.248461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218510+0000","last_change":"2023-11-24T08:17:40.268292+0000","last_active":"2023-11-24T08:17:43.218510+0000","last_peered":"2023-11-24T08:17:43.218510+0000","last_clean":"2023-11-24T08:17:43.218510+0000","last_became_active":"2023-11-24T08:17:40.267973+0000","last_became_peered":"2023-11-24T08:17:40.267973+0000","last_unstale":"2023-11-24T08:17:43.218510+0000","last_undegraded":"2023-11-24T08:17:43.218510+0000","last_fullsized":"2023-11-24T08:17:43.218510+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:17:01.331385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.215690+0000","last_change":"2023-11-24T08:17:41.281784+0000","last_active":"2023-11-24T08:17:43.215690+0000","last_peered":"2023-11-24T08:17:43.215690+0000","last_clean":"2023-11-24T08:17:43.215690+0000","last_became_active":"2023-11-24T08:17:41.281486+0000","last_became_peered":"2023-11-24T08:17:41.281486+0000","last_unstale":"2023-11-24T08:17:43.215690+0000","last_undegraded":"2023-11-24T08:17:43.215690+0000","last_fullsized":"2023-11-24T08:17:43.215690+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:55.343969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218352+0000","last_change":"2023-11-24T08:17:40.268699+0000","last_active":"2023-11-24T08:17:43.218352+0000","last_peered":"2023-11-24T08:17:43.218352+0000","last_clean":"2023-11-24T08:17:43.218352+0000","last_became_active":"2023-11-24T08:17:40.268429+0000","last_became_peered":"2023-11-24T08:17:40.268429+0000","last_unstale":"2023-11-24T08:17:43.218352+0000","last_undegraded":"2023-11-24T08:17:43.218352+0000","last_fullsized":"2023-11-24T08:17:43.218352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:49:27.915096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.274649+0000","last_change":"2023-11-24T08:17:41.274726+0000","last_active":"2023-11-24T08:17:42.274649+0000","last_peered":"2023-11-24T08:17:42.274649+0000","last_clean":"2023-11-24T08:17:42.274649+0000","last_became_active":"2023-11-24T08:17:41.274040+0000","last_became_peered":"2023-11-24T08:17:41.274040+0000","last_unstale":"2023-11-24T08:17:42.274649+0000","last_undegraded":"2023-11-24T08:17:42.274649+0000","last_fullsized":"2023-11-24T08:17:42.274649+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:05:17.464923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.275300+0000","last_change":"2023-11-24T08:17:40.254305+0000","last_active":"2023-11-24T08:17:42.275300+0000","last_peered":"2023-11-24T08:17:42.275300+0000","last_clean":"2023-11-24T08:17:42.275300+0000","last_became_active":"2023-11-24T08:17:40.253763+0000","last_became_peered":"2023-11-24T08:17:40.253763+0000","last_unstale":"2023-11-24T08:17:42.275300+0000","last_undegraded":"2023-11-24T08:17:42.275300+0000","last_fullsized":"2023-11-24T08:17:42.275300+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:25.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:43.218945+0000","last_change":"2023-11-24T08:17:41.274813+0000","last_active":"2023-11-24T08:17:43.218945+0000","last_peered":"2023-11-24T08:17:43.218945+0000","last_clean":"2023-11-24T08:17:43.218945+0000","last_became_active":"2023-11-24T08:17:41.274416+0000","last_became_peered":"2023-11-24T08:17:41.274416+0000","last_unstale":"2023-11-24T08:17:43.218945+0000","last_undegraded":"2023-11-24T08:17:43.218945+0000","last_fullsized":"2023-11-24T08:17:43.218945+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:40.222064+0000","last_clean_scrub_stamp":"2023-11-24T08:17:40.222064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:26:11.676844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-11-24T08:17:42.277623+0000","last_change":"2023-11-24T08:17:40.266271+0000","last_active":"2023-11-24T08:17:42.277623+0000","last_peered":"2023-11-24T08:17:42.277623+0000","last_clean":"2023-11-24T08:17:42.277623+0000","last_became_active":"2023-11-24T08:17:40.265994+0000","last_became_peered":"2023-11-24T08:17:40.265994+0000","last_unstale":"2023-11-24T08:17:42.277623+0000","last_undegraded":"2023-11-24T08:17:42.277623+0000","last_fullsized":"2023-11-24T08:17:42.277623+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:17:39.205245+0000","last_clean_scrub_stamp":"2023-11-24T08:17:39.205245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-24T08:17:41.257834+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:17:41.257834+0000","last_peered":"2023-11-24T08:17:41.257834+0000","last_clean":"2023-11-24T08:17:41.257834+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:17:41.257834+0000","last_undegraded":"2023-11-24T08:17:41.257834+0000","last_fullsized":"2023-11-24T08:17:41.257834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37924,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333916,"statfs":{"total":96636764160,"available":96597929984,"internally_reserved":0,"allocated":2637824,"data_stored":1480745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.0249999999999999,"15min":1.0349999999999999},"min":{"1min":0.60499999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6060000000000001,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.042,"5min":1.0669999999999999,"15min":1.0880000000000001},"min":{"1min":0.67100000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.4570000000000001,"5min":1.774,"15min":2.7109999999999999},"last":1.3859999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1559999999999999,"15min":1.155},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.6000000000000001,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.256,"15min":1.2150000000000001},"min":{"1min":0.71099999999999997,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":1.377,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.874}]},{"osd":2,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1220000000000001,"15min":1.1619999999999999},"min":{"1min":0.69499999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5980000000000001,"5min":1.774,"15min":4.7000000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.2889999999999999,"15min":1.3320000000000001},"min":{"1min":0.91900000000000004,"5min":0.70399999999999996,"15min":0.60399999999999998},"max":{"1min":1.7310000000000001,"5min":1.889,"15min":4.633},"last":1.2370000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.22,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.645,"5min":1.99,"15min":3.133},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.357,"15min":1.3779999999999999},"min":{"1min":0.96899999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.7370000000000001,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.446}]},{"osd":4,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3660000000000001,"15min":1.351},"min":{"1min":0.83699999999999997,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":1.597,"5min":2.665,"15min":2.851},"last":1.6659999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.3460000000000001,"15min":1.351},"min":{"1min":0.98499999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8069999999999999,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.3220000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.462,"15min":1.4770000000000001},"min":{"1min":1.1990000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.1080000000000001,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.552},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":1.095,"5min":0.622,"15min":0.622},"max":{"1min":1.8959999999999999,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.514}]},{"osd":6,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.2030000000000001,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":1.994,"5min":2.702,"15min":2.992},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.518,"15min":1.494},"min":{"1min":0.97799999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.982,"5min":2.6259999999999999,"15min":3.798},"last":1.635}]}]},{"osd":6,"up_from":13,"seq":55834575007,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37328,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334512,"statfs":{"total":96636764160,"available":96598540288,"internally_reserved":0,"allocated":2023424,"data_stored":888045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0960000000000001,"15min":1.0860000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":1.95},"last":1.2589999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.077,"15min":1.083},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.58099999999999996},"max":{"1min":1.3420000000000001,"5min":1.804,"15min":2.2069999999999999},"last":1.575}]},{"osd":1,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.151,"15min":1.1950000000000001},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5289999999999999,"5min":1.835,"15min":2.0230000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.206,"15min":1.2170000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.5649999999999999,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.4430000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.79500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.6240000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.127},{"interface":"front","average":{"1min":1.25,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.60399999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":1.609}]},{"osd":3,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.325,"15min":1.3600000000000001},"min":{"1min":0.91400000000000003,"5min":0.86399999999999999,"15min":0.71499999999999997},"max":{"1min":1.7490000000000001,"5min":2.2330000000000001,"15min":4.7880000000000003},"last":1.339},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.304,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.788,"5min":2.194,"15min":4.8319999999999999},"last":1.4750000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.452,"15min":1.4730000000000001},"min":{"1min":0.96899999999999997,"5min":0.90200000000000002,"15min":0.76500000000000001},"max":{"1min":1.9019999999999999,"5min":2.1190000000000002,"15min":2.895},"last":1.367},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5169999999999999},"min":{"1min":1.056,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.8740000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.448,"15min":1.48},"min":{"1min":1.145,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.956,"5min":2.3300000000000001,"15min":2.629},"last":1.302},{"interface":"front","average":{"1min":1.603,"5min":1.5840000000000001,"15min":1.6120000000000001},"min":{"1min":1.0009999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.996,"5min":2.165,"15min":2.7759999999999998},"last":1.72}]},{"osd":7,"last update":"Fri Nov 24 08:17:10 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.6020000000000001,"15min":1.6379999999999999},"min":{"1min":1.179,"5min":0.748,"15min":0.748},"max":{"1min":2.101,"5min":2.4100000000000001,"15min":3.0329999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.627,"15min":1.6299999999999999},"min":{"1min":1.056,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.4119999999999999,"5min":2.512,"15min":2.6890000000000001},"last":1.544}]}]},{"osd":5,"up_from":13,"seq":55834575007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37336,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334504,"statfs":{"total":96636764160,"available":96598532096,"internally_reserved":0,"allocated":2039808,"data_stored":894927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.96999999999999997,"15min":1.044},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.129,"5min":1.5169999999999999,"15min":3.1179999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.04,"15min":1.123},"min":{"1min":0.66500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.407,"5min":1.627,"15min":2.198},"last":0.86299999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.0980000000000001,"15min":1.1279999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.4059999999999999,"5min":1.8999999999999999,"15min":2.0529999999999999},"last":1.478},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.177,"15min":1.163},"min":{"1min":0.875,"5min":0.76200000000000001,"15min":0.48699999999999999},"max":{"1min":1.448,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.046}]},{"osd":2,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.1120000000000001,"15min":1.1799999999999999},"min":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.51600000000000001},"max":{"1min":1.4650000000000001,"5min":1.671,"15min":1.8520000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.64200000000000002},"max":{"1min":1.651,"5min":1.784,"15min":1.8129999999999999},"last":1.528}]},{"osd":3,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.157,"15min":1.163},"min":{"1min":0.78800000000000003,"5min":0.66600000000000004,"15min":0.432},"max":{"1min":1.5620000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.298,"5min":1.2310000000000001,"15min":1.286},"min":{"1min":1.1399999999999999,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.603,"5min":1.603,"15min":1.99},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.355,"15min":1.337},"min":{"1min":1.012,"5min":0.66400000000000003,"15min":0.60799999999999998},"max":{"1min":1.6419999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.28},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.421,"15min":1.484},"min":{"1min":0.94399999999999995,"5min":0.71099999999999997,"15min":0.46700000000000003},"max":{"1min":1.6890000000000001,"5min":1.994,"15min":3.5030000000000001},"last":1.3009999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.3380000000000001,"15min":1.3740000000000001},"min":{"1min":0.878,"5min":0.63600000000000001,"15min":0.48899999999999999},"max":{"1min":1.7150000000000001,"5min":2.085,"15min":3.2949999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.492,"5min":1.4450000000000001,"15min":1.4299999999999999},"min":{"1min":1.2629999999999999,"5min":0.71299999999999997,"15min":0.60999999999999999},"max":{"1min":1.7390000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.137}]},{"osd":7,"last update":"Fri Nov 24 08:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5029999999999999,"15min":1.5229999999999999},"min":{"1min":1.1499999999999999,"5min":0.69899999999999995,"15min":0.52900000000000003},"max":{"1min":1.8400000000000001,"5min":2.1440000000000001,"15min":3.4420000000000002},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.3620000000000001,"5min":0.75600000000000001,"15min":0.71199999999999997},"max":{"1min":1.827,"5min":2.113,"15min":2.181},"last":1.3400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37344,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334496,"statfs":{"total":96636764160,"available":96598523904,"internally_reserved":0,"allocated":2048000,"data_stored":896005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0469999999999999,"15min":1.0900000000000001},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.42699999999999999},"max":{"1min":1.4139999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.099,"15min":1.0840000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.696,"5min":5.452,"15min":5.452},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.1579999999999999,"15min":1.151},"min":{"1min":0.85499999999999998,"5min":0.70199999999999996,"15min":0.64200000000000002},"max":{"1min":1.708,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.421},{"interface":"front","average":{"1min":1.147,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.68799999999999994,"15min":0.51900000000000002},"max":{"1min":1.556,"5min":1.988,"15min":1.988},"last":1.496}]},{"osd":2,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0640000000000001,"15min":1.1000000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.645,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.246,"15min":1.2829999999999999},"min":{"1min":0.76100000000000001,"5min":0.71199999999999997,"15min":0.505},"max":{"1min":1.6359999999999999,"5min":2.0470000000000002,"15min":2.7229999999999999},"last":1.5389999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3640000000000001,"15min":1.369},"min":{"1min":0.85299999999999998,"5min":0.74399999999999999,"15min":0.62},"max":{"1min":1.6359999999999999,"5min":1.9910000000000001,"15min":2.9100000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3089999999999999,"15min":1.296},"min":{"1min":0.87,"5min":0.87,"15min":0.55800000000000005},"max":{"1min":1.6639999999999999,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.1659999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.5329999999999999,"15min":1.5009999999999999},"min":{"1min":1.276,"5min":0.71999999999999997,"15min":0.69299999999999995},"max":{"1min":1.853,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.472,"15min":1.4810000000000001},"min":{"1min":0.85799999999999998,"5min":0.78200000000000003,"15min":0.60199999999999998},"max":{"1min":1.748,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.738}]},{"osd":6,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2669999999999999,"15min":1.3240000000000001},"min":{"1min":0.93999999999999995,"5min":0.57499999999999996,"15min":0.50900000000000001},"max":{"1min":1.73,"5min":1.907,"15min":3.5899999999999999},"last":1.228},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5620000000000001,"15min":1.581},"min":{"1min":1.1319999999999999,"5min":0.79000000000000004,"15min":0.58099999999999996},"max":{"1min":1.772,"5min":2.2890000000000001,"15min":3.391},"last":1.9219999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.52,"15min":1.5449999999999999},"min":{"1min":1.0089999999999999,"5min":0.63300000000000001,"15min":0.60899999999999999},"max":{"1min":1.6950000000000001,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.4370000000000001,"15min":1.478},"min":{"1min":1.0389999999999999,"5min":0.63500000000000001,"15min":0.61799999999999999},"max":{"1min":1.78,"5min":2.1659999999999999,"15min":3.4849999999999999},"last":1.6419999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37320,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334520,"statfs":{"total":96636764160,"available":96598548480,"internally_reserved":0,"allocated":2023424,"data_stored":888057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.39200000000000002,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.3360000000000001,"5min":1.829,"15min":10.557},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94899999999999995,"15min":0.91800000000000004},"min":{"1min":0.54900000000000004,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85499999999999998,"15min":0.82499999999999996},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":1.71,"5min":1.728,"15min":2.101},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.95899999999999996,"15min":0.98999999999999999},"min":{"1min":0.52600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.355,"5min":1.8799999999999999,"15min":13.090999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.93300000000000005},"min":{"1min":0.628,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.349,"5min":1.6060000000000001,"15min":3.9489999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0780000000000001},"min":{"1min":0.55900000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.589,"5min":1.796,"15min":13.460000000000001},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.997,"5min":0.72399999999999998,"15min":0.67800000000000005},"max":{"1min":1.877,"5min":2.1840000000000002,"15min":4.367},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2030000000000001,"15min":1.25},"min":{"1min":0.73199999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.746,"5min":1.746,"15min":3.3380000000000001},"last":1.4339999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2909999999999999,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.635,"5min":2.0609999999999999,"15min":13.231},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2549999999999999,"15min":1.282},"min":{"1min":0.80500000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.744,"5min":1.992,"15min":5.1180000000000003},"last":1.0149999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.339,"15min":1.377},"min":{"1min":0.77100000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.9370000000000001,"5min":2.0449999999999999,"15min":13.359999999999999},"last":1.242},{"interface":"front","average":{"1min":1.232,"5min":1.2709999999999999,"15min":1.286},"min":{"1min":0.83999999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.0369999999999999,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.5609999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3180000000000001,"15min":1.363},"min":{"1min":0.85399999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.823,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.335},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3200000000000001,"15min":1.3640000000000001},"min":{"1min":0.81200000000000006,"5min":0.76400000000000001,"15min":0.68799999999999994},"max":{"1min":1.855,"5min":1.887,"15min":4.032},"last":1.5029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575006,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37312,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334528,"statfs":{"total":96636764160,"available":96598556672,"internally_reserved":0,"allocated":2023424,"data_stored":888113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.78200000000000003,"15min":0.82699999999999996},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.86899999999999999,"5min":1.601,"15min":1.9610000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.379,"5min":1.782,"15min":2.5779999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.91400000000000003,"15min":0.98299999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.2430000000000001,"5min":1.518,"15min":1.9890000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.95299999999999996,"15min":1.0109999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.458,"5min":1.6200000000000001,"15min":2.6240000000000001},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.84399999999999997,"15min":0.88700000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.224,"5min":1.5309999999999999,"15min":2.2090000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0489999999999999,"15min":1.119},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4650000000000001,"5min":1.927,"15min":2.9950000000000001},"last":0.85899999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2709999999999999,"15min":1.3220000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.669,"5min":2.1379999999999999,"15min":2.7160000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.228,"15min":1.3049999999999999},"min":{"1min":0.78300000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.704,"5min":1.921,"15min":3.3279999999999998},"last":0.88}]},{"osd":5,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.236,"15min":1.323},"min":{"1min":0.66100000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.9319999999999999,"15min":2.508},"last":1.246},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.2190000000000001,"15min":1.298},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.6679999999999999,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.048}]},{"osd":6,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1799999999999999,"15min":1.2490000000000001},"min":{"1min":0.77000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.7629999999999999,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3120000000000001,"15min":1.391},"min":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5449999999999999,"5min":2.069,"15min":2.6629999999999998},"last":1.2969999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.294,"15min":1.3839999999999999},"min":{"1min":0.81999999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.581,"5min":2.3999999999999999,"15min":2.859},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.2849999999999999,"15min":1.3560000000000001},"min":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.746},"max":{"1min":1.6140000000000001,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":0.98399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37324,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334516,"statfs":{"total":96636764160,"available":96598544384,"internally_reserved":0,"allocated":2023424,"data_stored":888057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.79500000000000004,"15min":0.78500000000000003},"min":{"1min":0.34499999999999997,"5min":0.33000000000000002,"15min":0.28399999999999997},"max":{"1min":1.5660000000000001,"5min":1.5900000000000001,"15min":1.7749999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.872,"15min":0.86499999999999999},"min":{"1min":0.505,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.413,"5min":1.5189999999999999,"15min":2.1619999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.76500000000000001},"min":{"1min":0.47299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.4790000000000001,"5min":1.679,"15min":1.861},"last":1.125},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.91000000000000003,"15min":0.89900000000000002},"min":{"1min":0.39100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.484,"5min":1.925,"15min":1.925},"last":1.0620000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.97899999999999998,"15min":0.97099999999999997},"min":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.601,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.90800000000000003,"15min":0.90000000000000002},"min":{"1min":0.48199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.117,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.22}]},{"osd":4,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2529999999999999,"15min":1.252},"min":{"1min":0.879,"5min":0.76300000000000001,"15min":0.65300000000000002},"max":{"1min":1.788,"5min":1.8380000000000001,"15min":2.7959999999999998},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.23,"15min":1.248},"min":{"1min":0.82099999999999995,"5min":0.77200000000000002,"15min":0.55900000000000005},"max":{"1min":1.5629999999999999,"5min":2.141,"15min":3.0409999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.222,"15min":1.228},"min":{"1min":0.79400000000000004,"5min":0.75900000000000001,"15min":0.69599999999999995},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":3.7639999999999998},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1739999999999999,"15min":1.1890000000000001},"min":{"1min":0.85999999999999999,"5min":0.66900000000000004,"15min":0.57299999999999995},"max":{"1min":1.6359999999999999,"5min":2.246,"15min":3.286},"last":1.6060000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.244,"15min":1.2569999999999999},"min":{"1min":0.81399999999999995,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":1.679,"5min":2.0419999999999998,"15min":4.2750000000000004},"last":1.46},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.232,"15min":1.2370000000000001},"min":{"1min":0.80500000000000005,"5min":0.71299999999999997,"15min":0.63100000000000001},"max":{"1min":1.7350000000000001,"5min":1.788,"15min":3.7040000000000002},"last":1.385}]},{"osd":7,"last update":"Fri Nov 24 08:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2849999999999999,"15min":1.3009999999999999},"min":{"1min":0.876,"5min":0.71899999999999997,"15min":0.59099999999999997},"max":{"1min":1.798,"5min":2.1429999999999998,"15min":3.1760000000000002},"last":1.52},{"interface":"front","average":{"1min":1.196,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.72299999999999998,"5min":0.624,"15min":0.624},"max":{"1min":1.841,"5min":2.3620000000000001,"15min":3.653},"last":1.7010000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37892,"kb_used_data":2552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333948,"statfs":{"total":96636764160,"available":96597962752,"internally_reserved":0,"allocated":2613248,"data_stored":1478391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.78900000000000003},"min":{"1min":0.39300000000000002,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81899999999999995,"15min":0.89800000000000002},"min":{"1min":0.51000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.4219999999999999,"5min":1.4610000000000001,"15min":2.3620000000000001},"last":1.1659999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0209999999999999,"15min":1.0309999999999999},"min":{"1min":0.58399999999999996,"5min":0.51100000000000001,"15min":0.47699999999999998},"max":{"1min":1.7529999999999999,"5min":1.833,"15min":1.891},"last":1.3169999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.88600000000000001,"15min":0.90200000000000002},"min":{"1min":0.51900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.798,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.97999999999999998,"15min":1.036},"min":{"1min":0.60999999999999999,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.667,"5min":1.679,"15min":2.0670000000000002},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89500000000000002,"15min":0.95599999999999996},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.651,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":1.24}]},{"osd":4,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1659999999999999,"15min":1.286},"min":{"1min":0.80800000000000005,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.7610000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.139,"15min":1.1699999999999999},"min":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5109999999999999,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.284}]},{"osd":5,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1679999999999999,"15min":1.2210000000000001},"min":{"1min":0.82099999999999995,"5min":0.752,"15min":0.69699999999999995},"max":{"1min":1.6080000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.091},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1579999999999999,"15min":1.2709999999999999},"min":{"1min":0.75600000000000001,"5min":0.72299999999999998,"15min":0.71099999999999997},"max":{"1min":1.595,"5min":1.595,"15min":2.7240000000000002},"last":1.341}]},{"osd":6,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2549999999999999,"15min":1.329},"min":{"1min":0.79400000000000004,"5min":0.78000000000000003,"15min":0.66800000000000004},"max":{"1min":1.7669999999999999,"5min":2.052,"15min":2.4369999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.262,"15min":1.3340000000000001},"min":{"1min":0.84399999999999997,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.7070000000000001,"5min":2.823,"15min":2.823},"last":1.377}]},{"osd":7,"last update":"Fri Nov 24 08:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3440000000000001,"15min":1.4199999999999999},"min":{"1min":0.997,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":1.8300000000000001,"5min":2.1349999999999998,"15min":2.6989999999999998},"last":1.486},{"interface":"front","average":{"1min":1.302,"5min":1.2969999999999999,"15min":1.391},"min":{"1min":0.80300000000000005,"5min":0.747,"15min":0.747},"max":{"1min":1.885,"5min":2.0499999999999998,"15min":2.3929999999999998},"last":1.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:17:46.222 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:17:46.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:17:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.477+0000 7fe4e091d640 1 Processor -- start 2023-11-24T08:17:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.477+0000 7fe4e091d640 1 -- start start 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4e091d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe4dc15b440 0x7fe4dc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4e091d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4e091d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc0c5dc0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4e091d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc0cc3d0 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4e091d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc15b820 con 0x7fe4dc15b440 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4dad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4dad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36298/0 (socket says 172.21.15.107:36298) 2023-11-24T08:17:46.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4dad9d640 1 -- 172.21.15.107:0/2764563946 learned_addr learned my addr 172.21.15.107:0/2764563946 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.479+0000 7fe4da59c640 1 --2- 172.21.15.107:0/2764563946 >> v2:172.21.15.120:3300/0 conn(0x7fe4dc15b440 0x7fe4dc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.480+0000 7fe4dad9d640 1 -- 172.21.15.107:0/2764563946 >> v2:172.21.15.120:3300/0 conn(0x7fe4dc15b440 msgr2=0x7fe4dc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.480+0000 7fe4dad9d640 1 --2- 172.21.15.107:0/2764563946 >> v2:172.21.15.120:3300/0 conn(0x7fe4dc15b440 0x7fe4dc0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.480+0000 7fe4dad9d640 1 -- 172.21.15.107:0/2764563946 >> v1:172.21.15.120:6789/0 conn(0x7fe4dc0cbf20 legacy=0x7fe4dc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.480+0000 7fe4dad9d640 1 -- 172.21.15.107:0/2764563946 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc15eea0 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.481+0000 7fe4dad9d640 1 --2- 172.21.15.107:0/2764563946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc06e770 tx=0x7fe4d402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7134123e5e99d19 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:46.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.481+0000 7fe4d959a640 1 -- 172.21.15.107:0/2764563946 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d402c9f0 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.481+0000 7fe4d959a640 1 -- 172.21.15.107:0/2764563946 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d402cb50 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.481+0000 7fe4d959a640 1 -- 172.21.15.107:0/2764563946 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40343f0 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 -- 172.21.15.107:0/2764563946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0c59b0 msgr2=0x7fe4dc06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 --2- 172.21.15.107:0/2764563946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc06e770 tx=0x7fe4d402be50 comp rx=0 tx=0).stop 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 -- 172.21.15.107:0/2764563946 shutdown_connections 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 --2- 172.21.15.107:0/2764563946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc06c5f0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 --2- 172.21.15.107:0/2764563946 >> v2:172.21.15.120:3300/0 conn(0x7fe4dc15b440 0x7fe4dc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 -- 172.21.15.107:0/2764563946 >> 172.21.15.107:0/2764563946 conn(0x7fe4dc0b0c50 msgr2=0x7fe4dc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 -- 172.21.15.107:0/2764563946 shutdown_connections 2023-11-24T08:17:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.482+0000 7fe4e091d640 1 -- 172.21.15.107:0/2764563946 wait complete. 2023-11-24T08:17:46.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.484+0000 7fe4e091d640 1 Processor -- start 2023-11-24T08:17:46.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.484+0000 7fe4e091d640 1 -- start start 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4dc15b440 0x7fe4dc153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0c78e0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0c7b10 con 0x7fe4dc0c59b0 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4e091d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0c7df0 con 0x7fe4dc15b440 2023-11-24T08:17:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36312/0 (socket says 172.21.15.107:36312) 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 -- 172.21.15.107:0/1218971697 learned_addr learned my addr 172.21.15.107:0/1218971697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4da59c640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 -- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4dc15b440 msgr2=0x7fe4dc153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4dc15b440 0x7fe4dc153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 -- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4dc0c59b0 msgr2=0x7fe4dc0c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.485+0000 7fe4d9d9b640 1 -- 172.21.15.107:0/1218971697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d4009b30 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4d9d9b640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c410 tx=0x7fe4d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00035f0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0003770 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0003b20 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc0c80d0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.486+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc0c8690 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.487+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc04e8b0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.488+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fe4d0003c80 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.489+0000 7fe4caffd640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 0x7fe4bc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.489+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4d00874d0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.489+0000 7fe4da59c640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 0x7fe4bc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.490+0000 7fe4da59c640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 0x7fe4bc06eaa0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc006fd0 tx=0x7fe4cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:46.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.493+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d0054080 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.753+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe4dc15f980 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.759+0000 7fe4caffd640 1 -- 172.21.15.107:0/1218971697 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe4d00170b0 con 0x7fe4dc0cbf20 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:17:46.761 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.762+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 msgr2=0x7fe4bc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.762+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 0x7fe4bc06eaa0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc006fd0 tx=0x7fe4cc008040 comp rx=0 tx=0).stop 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.762+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 msgr2=0x7fe4dc0ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.762+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c410 tx=0x7fe4d000c8d0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 shutdown_connections 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe4bc06c5e0 0x7fe4bc06eaa0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4dc15b440 0x7fe4dc153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4dc0cbf20 0x7fe4dc0ad410 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 --2- 172.21.15.107:0/1218971697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4dc0c59b0 0x7fe4dc0c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:46.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 >> 172.21.15.107:0/1218971697 conn(0x7fe4dc0b0c50 msgr2=0x7fe4dc0cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:46.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 shutdown_connections 2023-11-24T08:17:46.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.763+0000 7fe4e091d640 1 -- 172.21.15.107:0/1218971697 wait complete. 2023-11-24T08:17:46.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:17:46.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.996+0000 7f5d166e1640 1 Processor -- start 2023-11-24T08:17:46.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.997+0000 7f5d166e1640 1 -- start start 2023-11-24T08:17:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.998+0000 7f5d166e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d1013d120 0x7f5d1015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.998+0000 7f5d166e1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d1006d6c0 0x7f5d1006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.998+0000 7f5d166e1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1006e060 con 0x7f5d100b0a70 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.998+0000 7f5d166e1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1013f9e0 con 0x7f5d1013d120 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.998+0000 7f5d166e1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1013fcc0 con 0x7f5d1006d6c0 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d1013d120 0x7f5d1015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d1013d120 0x7f5d1015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36316/0 (socket says 172.21.15.107:36316) 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 -- 172.21.15.107:0/4072757603 learned_addr learned my addr 172.21.15.107:0/4072757603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 -- 172.21.15.107:0/4072757603 >> v2:172.21.15.120:3300/0 conn(0x7f5d1006d6c0 msgr2=0x7f5d1006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d14ede640 1 --2- 172.21.15.107:0/4072757603 >> v2:172.21.15.120:3300/0 conn(0x7f5d1006d6c0 0x7f5d1006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 --2- 172.21.15.107:0/4072757603 >> v2:172.21.15.120:3300/0 conn(0x7f5d1006d6c0 0x7f5d1006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 -- 172.21.15.107:0/4072757603 >> v1:172.21.15.120:6789/0 conn(0x7f5d100b0a70 legacy=0x7f5d100b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:46.999+0000 7f5d156df640 1 -- 172.21.15.107:0/4072757603 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100cec00 con 0x7f5d1013d120 2023-11-24T08:17:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.000+0000 7f5d156df640 1 --2- 172.21.15.107:0/4072757603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1013d120 0x7f5d1015af40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f5d10072a70 tx=0x7f5d0c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c20604f12a69f1d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.000+0000 7f5d07fff640 1 -- 172.21.15.107:0/4072757603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c031800 con 0x7f5d1013d120 2023-11-24T08:17:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d07fff640 1 -- 172.21.15.107:0/4072757603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c031960 con 0x7f5d1013d120 2023-11-24T08:17:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d07fff640 1 -- 172.21.15.107:0/4072757603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c031cf0 con 0x7f5d1013d120 2023-11-24T08:17:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 -- 172.21.15.107:0/4072757603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1013d120 msgr2=0x7f5d1015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 --2- 172.21.15.107:0/4072757603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1013d120 0x7f5d1015af40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f5d10072a70 tx=0x7f5d0c02b8a0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 -- 172.21.15.107:0/4072757603 shutdown_connections 2023-11-24T08:17:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 --2- 172.21.15.107:0/4072757603 >> v2:172.21.15.120:3300/0 conn(0x7f5d1006d6c0 0x7f5d1006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 --2- 172.21.15.107:0/4072757603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1013d120 0x7f5d1015af40 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 -- 172.21.15.107:0/4072757603 >> 172.21.15.107:0/4072757603 conn(0x7f5d100cb830 msgr2=0x7f5d100cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 -- 172.21.15.107:0/4072757603 shutdown_connections 2023-11-24T08:17:47.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.001+0000 7f5d166e1640 1 -- 172.21.15.107:0/4072757603 wait complete. 2023-11-24T08:17:47.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.003+0000 7f5d166e1640 1 Processor -- start 2023-11-24T08:17:47.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.003+0000 7f5d166e1640 1 -- start start 2023-11-24T08:17:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d100b0a70 0x7f5d10159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d1013d120 0x7f5d10070210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100b2a20 con 0x7f5d1006d6c0 2023-11-24T08:17:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10158260 con 0x7f5d1013d120 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d166e1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1015a0d0 con 0x7f5d100b0a70 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d156df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d156df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36320/0 (socket says 172.21.15.107:36320) 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.004+0000 7f5d156df640 1 -- 172.21.15.107:0/3848813931 learned_addr learned my addr 172.21.15.107:0/3848813931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 -- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d100b0a70 msgr2=0x7f5d10159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d100b0a70 0x7f5d10159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 -- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d1013d120 msgr2=0x7f5d10070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d1013d120 0x7f5d10070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 -- 172.21.15.107:0/3848813931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0094d0 con 0x7f5d1006d6c0 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d156df640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02b8e0 tx=0x7f5d0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c03ad30 con 0x7f5d1006d6c0 2023-11-24T08:17:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c03ae90 con 0x7f5d1006d6c0 2023-11-24T08:17:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d10072110 con 0x7f5d1006d6c0 2023-11-24T08:17:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.005+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c044630 con 0x7f5d1006d6c0 2023-11-24T08:17:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.006+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d100b2c70 con 0x7f5d1006d6c0 2023-11-24T08:17:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.007+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d100b2e90 con 0x7f5d1006d6c0 2023-11-24T08:17:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.008+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f5d0c044790 con 0x7f5d1006d6c0 2023-11-24T08:17:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.009+0000 7f5d05ffb640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 0x7f5cf406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.009+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d0c02e070 con 0x7f5d1006d6c0 2023-11-24T08:17:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.009+0000 7f5d14ede640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 0x7f5cf406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.010+0000 7f5d14ede640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 0x7f5cf406eaa0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006fd0 tx=0x7f5d00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.013+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d0c077400 con 0x7f5d1006d6c0 2023-11-24T08:17:47.201 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:17:47.199+0000 7fef42daf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:17:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.241+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d1004e840 con 0x7f5d1006d6c0 2023-11-24T08:17:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.242+0000 7f5d05ffb640 1 -- 172.21.15.107:0/3848813931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f5d0c07aa20 con 0x7f5d1006d6c0 2023-11-24T08:17:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:17:47.244 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:17:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 97 2023-11-24T08:17:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.244+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 msgr2=0x7f5cf406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.244+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 0x7f5cf406eaa0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006fd0 tx=0x7f5d00008040 comp rx=0 tx=0).stop 2023-11-24T08:17:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 msgr2=0x7f5d1014e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02b8e0 tx=0x7f5d0c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 shutdown_connections 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cf406c5e0 0x7f5cf406eaa0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d1013d120 0x7f5d10070210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d100b0a70 0x7f5d10159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 --2- 172.21.15.107:0/3848813931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1006d6c0 0x7f5d1014e300 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 >> 172.21.15.107:0/3848813931 conn(0x7f5d100cb830 msgr2=0x7f5d10157a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 shutdown_connections 2023-11-24T08:17:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.245+0000 7f5d166e1640 1 -- 172.21.15.107:0/3848813931 wait complete. 2023-11-24T08:17:47.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:17:40.695113+0000', 'modified': '2023-11-24T08:17:43.749894+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9873}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9873': {'gid': 9873, 'name': 'c', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/1134531969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1134531969}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1134531969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:17:47.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:17:47.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:17:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.892+0000 7f746dbac640 1 Processor -- start 2023-11-24T08:17:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.893+0000 7f746dbac640 1 -- start start 2023-11-24T08:17:47.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f746815a1c0 0x7f7468070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74680b15e0 0x7f746815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680cd920 con 0x7f74680b15e0 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680b1a50 con 0x7f746815a1c0 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.894+0000 7f746dbac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746806df90 con 0x7f746806d640 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74680b15e0 0x7f746815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74680b15e0 0x7f746815dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37054/0 (socket says 172.21.15.107:37054) 2023-11-24T08:17:47.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 -- 172.21.15.107:0/1470606349 learned_addr learned my addr 172.21.15.107:0/1470606349 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 -- 172.21.15.107:0/1470606349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 msgr2=0x7f746806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 --2- 172.21.15.107:0/1470606349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 -- 172.21.15.107:0/1470606349 >> v2:172.21.15.120:3300/0 conn(0x7f746815a1c0 msgr2=0x7f7468070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 --2- 172.21.15.107:0/1470606349 >> v2:172.21.15.120:3300/0 conn(0x7f746815a1c0 0x7f7468070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.895+0000 7f7467fff640 1 -- 172.21.15.107:0/1470606349 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746815e2e0 con 0x7f74680b15e0 2023-11-24T08:17:47.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.896+0000 7f7467fff640 1 --2- 172.21.15.107:0/1470606349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74680b15e0 0x7f746815dda0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f74680ccc20 tx=0x7f745c02dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cbcbd8ea3cc80a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.896+0000 7f74677fe640 1 -- 172.21.15.107:0/1470606349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c02e950 con 0x7f74680b15e0 2023-11-24T08:17:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.896+0000 7f74677fe640 1 -- 172.21.15.107:0/1470606349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c02eab0 con 0x7f74680b15e0 2023-11-24T08:17:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.896+0000 7f74677fe640 1 -- 172.21.15.107:0/1470606349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c02ee40 con 0x7f74680b15e0 2023-11-24T08:17:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 -- 172.21.15.107:0/1470606349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74680b15e0 msgr2=0x7f746815dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 --2- 172.21.15.107:0/1470606349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74680b15e0 0x7f746815dda0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f74680ccc20 tx=0x7f745c02dba0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 -- 172.21.15.107:0/1470606349 shutdown_connections 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 --2- 172.21.15.107:0/1470606349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 --2- 172.21.15.107:0/1470606349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74680b15e0 0x7f746815dda0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 --2- 172.21.15.107:0/1470606349 >> v2:172.21.15.120:3300/0 conn(0x7f746815a1c0 0x7f7468070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 -- 172.21.15.107:0/1470606349 >> 172.21.15.107:0/1470606349 conn(0x7f74680cbb40 msgr2=0x7f74680cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 -- 172.21.15.107:0/1470606349 shutdown_connections 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.897+0000 7f746dbac640 1 -- 172.21.15.107:0/1470606349 wait complete. 2023-11-24T08:17:47.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.899+0000 7f746dbac640 1 Processor -- start 2023-11-24T08:17:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.899+0000 7f746dbac640 1 -- start start 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 0x7f74680b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680b4190 con 0x7f746815a1c0 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746815b690 con 0x7f74680b15e0 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746dbac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746815b970 con 0x7f746806d640 2023-11-24T08:17:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37062/0 (socket says 172.21.15.107:37062) 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.900+0000 7f746d3ab640 1 -- 172.21.15.107:0/3392448577 learned_addr learned my addr 172.21.15.107:0/3392448577 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f7467fff640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 0x7f74680b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 -- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 msgr2=0x7f746815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 -- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 msgr2=0x7f74680b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 0x7f74680b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 -- 172.21.15.107:0/3392448577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0094d0 con 0x7f746815a1c0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f7467fff640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 0x7f74680b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f746000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f74657fa640 1 -- 172.21.15.107:0/3392448577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460011830 con 0x7f746815a1c0 2023-11-24T08:17:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f74657fa640 1 -- 172.21.15.107:0/3392448577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460011990 con 0x7f746815a1c0 2023-11-24T08:17:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746815bc50 con 0x7f746815a1c0 2023-11-24T08:17:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f74657fa640 1 -- 172.21.15.107:0/3392448577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460011d20 con 0x7f746815a1c0 2023-11-24T08:17:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.901+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746815c210 con 0x7f746815a1c0 2023-11-24T08:17:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.904+0000 7f74657fa640 1 -- 172.21.15.107:0/3392448577 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f746001b460 con 0x7f746815a1c0 2023-11-24T08:17:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.905+0000 7f74657fa640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 0x7f745406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.905+0000 7f74657fa640 1 -- 172.21.15.107:0/3392448577 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74600882c0 con 0x7f746815a1c0 2023-11-24T08:17:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.906+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 0x7f745406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.906+0000 7f74719a7740 1 Processor -- start 2023-11-24T08:17:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.906+0000 7f74719a7740 1 -- start start 2023-11-24T08:17:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.906+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 0x7f745406eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7458007dc0 tx=0x7f74580072f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x560266625270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5602666257b0 0x560266669730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560266584880 con 0x560266626070 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560266593ae0 con 0x560266620d70 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602665911d0 con 0x5602666257b0 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f746cbaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f746cbaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37076/0 (socket says 172.21.15.107:37076) 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f746cbaa640 1 -- 172.21.15.107:0/2214662688 learned_addr learned my addr 172.21.15.107:0/2214662688 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f746cbaa640 1 -- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5602666257b0 msgr2=0x560266669730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.907+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5602666257b0 0x560266669730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f746cbaa640 1 -- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 msgr2=0x560266625270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f7467fff640 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x560266625270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x560266625270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f746cbaa640 1 -- 172.21.15.107:0/2214662688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560266554860 con 0x560266626070 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f745800cfc0 tx=0x7f745800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88406f765be7b18 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f744e7fc640 1 -- 172.21.15.107:0/2214662688 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458011820 con 0x560266626070 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f744e7fc640 1 -- 172.21.15.107:0/2214662688 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458011980 con 0x560266626070 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.908+0000 7f744e7fc640 1 -- 172.21.15.107:0/2214662688 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458011d10 con 0x560266626070 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 -- 172.21.15.107:0/2214662688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 msgr2=0x560266626450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f745800cfc0 tx=0x7f745800f000 comp rx=0 tx=0).stop 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 -- 172.21.15.107:0/2214662688 shutdown_connections 2023-11-24T08:17:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5602666257b0 0x560266669730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x560266625270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 --2- 172.21.15.107:0/2214662688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266626450 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 -- 172.21.15.107:0/2214662688 >> 172.21.15.107:0/2214662688 conn(0x5602665e7c70 msgr2=0x56026666bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 -- 172.21.15.107:0/2214662688 shutdown_connections 2023-11-24T08:17:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.909+0000 7f74719a7740 1 -- 172.21.15.107:0/2214662688 wait complete. 2023-11-24T08:17:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.910+0000 7f74719a7740 1 -- start start 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.910+0000 7f74719a7740 1 Processor -- start 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x560266646690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x5602666698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026658cac0 con 0x560266626070 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602665292f0 con 0x5602666257b0 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560266546c70 con 0x560266620d70 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f74719a7740 10 client.0 _ll_register_callbacks cb 0x56026654b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37078/0 (socket says 172.21.15.107:37078) 2023-11-24T08:17:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f746d3ab640 1 -- 172.21.15.107:0/3501087887 learned_addr learned my addr 172.21.15.107:0/3501087887 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x560266646690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.911+0000 7f7467fff640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x5602666698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 -- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 msgr2=0x560266646690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x560266646690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 -- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 msgr2=0x5602666698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x5602666698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 -- 172.21.15.107:0/3501087887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745800cca0 con 0x560266626070 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f746d3ab640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f746000bde0 tx=0x7f7460059240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10109551a1343030 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74477fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460070380 con 0x560266626070 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7460060d90 con 0x560266626070 2023-11-24T08:17:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560266585aa0 con 0x560266626070 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7460060900 con 0x560266626070 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74719a7740 10 client.10005 fetch_fsmap learned FSMap version 97 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.912+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x56026658b140 con 0x560266626070 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460074070 con 0x560266626070 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 97) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f746001b710 con 0x560266626070 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 10 client.10005 fetch_fsmap finished waiting for FSMap version 97 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 10 client.10005 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 20 client.10005 populate_metadata read hostname 'smithi107' 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026666d3f0 0x56026667ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 4 client.10005 mds_command: new command op to 9873 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 --> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602665857c0 con 0x56026666d3f0 2023-11-24T08:17:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.913+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026666d3f0 0x56026667ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.914+0000 7f746cbaa640 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026666d3f0 0x56026667ee20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b73e8a0778b885d2 server_cookie=cc3603852b6d7567 in_seq=0 out_seq=0 2023-11-24T08:17:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.914+0000 7f74477fe640 10 client.10005 ms_handle_connect on v2:172.21.15.107:6836/1134531969 2023-11-24T08:17:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.914+0000 7f74477fe640 1 -- 172.21.15.107:0/3501087887 <== mds.0 v2:172.21.15.107:6836/1134531969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5602665857c0 con 0x56026666d3f0 2023-11-24T08:17:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.914+0000 7f74477fe640 10 client.10005 handle_command_reply: tid=0 2023-11-24T08:17:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.914+0000 7f74719a7740 1 client.10005 shutdown 2023-11-24T08:17:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 msgr2=0x560266646f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f746000bde0 tx=0x7f7460059240 comp rx=0 tx=0).stop 2023-11-24T08:17:47.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 shutdown_connections 2023-11-24T08:17:47.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026666d3f0 0x56026667ee20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266646f90 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x5602666698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 --2- 172.21.15.107:0/3501087887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x560266646690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74477fe640 0 client.10005 ms_handle_reset on v2:172.21.15.107:6836/1134531969 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 >> 172.21.15.107:0/3501087887 conn(0x5602665e7c70 msgr2=0x56026661fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 shutdown_connections 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 1 -- 172.21.15.107:0/3501087887 wait complete. 2023-11-24T08:17:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.915+0000 7f74719a7740 20 client.10005 trim_cache size 0 max 16384 2023-11-24T08:17:47.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.932+0000 7f74719a7740 1 Processor -- start 2023-11-24T08:17:47.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.932+0000 7f74719a7740 1 -- start start 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x56026666b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266626070 0x56026666ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602662eff20 con 0x5602666257b0 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560266540e20 con 0x560266620d70 2023-11-24T08:17:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560266669b40 con 0x560266626070 2023-11-24T08:17:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f7467fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f7467fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37084/0 (socket says 172.21.15.107:37084) 2023-11-24T08:17:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.933+0000 7f7467fff640 1 -- 172.21.15.107:0/3485252404 learned_addr learned my addr 172.21.15.107:0/3485252404 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 -- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266626070 msgr2=0x56026666ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266626070 0x56026666ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 -- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 msgr2=0x56026666b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x56026666b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 -- 172.21.15.107:0/3485252404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560266528b80 con 0x5602666257b0 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f7467fff640 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f745c02dbe0 tx=0x7f745c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64b987d60e61b8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f744effd640 1 -- 172.21.15.107:0/3485252404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c040440 con 0x5602666257b0 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f744effd640 1 -- 172.21.15.107:0/3485252404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c0405a0 con 0x5602666257b0 2023-11-24T08:17:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.934+0000 7f744effd640 1 -- 172.21.15.107:0/3485252404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c040910 con 0x5602666257b0 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 -- 172.21.15.107:0/3485252404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 msgr2=0x56026666e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f745c02dbe0 tx=0x7f745c030000 comp rx=0 tx=0).stop 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 -- 172.21.15.107:0/3485252404 shutdown_connections 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266626070 0x56026666ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602666257b0 0x56026666e7e0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 --2- 172.21.15.107:0/3485252404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560266620d70 0x56026666b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 -- 172.21.15.107:0/3485252404 >> 172.21.15.107:0/3485252404 conn(0x5602665e7c70 msgr2=0x560266626450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 -- 172.21.15.107:0/3485252404 shutdown_connections 2023-11-24T08:17:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.935+0000 7f74719a7740 1 -- 172.21.15.107:0/3485252404 wait complete. 2023-11-24T08:17:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.936+0000 7f74719a7740 1 -- start start 2023-11-24T08:17:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.936+0000 7f74719a7740 1 Processor -- start 2023-11-24T08:17:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x5602666dbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x56026661fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602666df740 con 0x560266626070 2023-11-24T08:17:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560266667dc0 con 0x5602666257b0 2023-11-24T08:17:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560266667f60 con 0x560266620d70 2023-11-24T08:17:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f74719a7740 10 client.0 _ll_register_callbacks cb 0x7f7460054d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:17:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746d3ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37088/0 (socket says 172.21.15.107:37088) 2023-11-24T08:17:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746d3ab640 1 -- 172.21.15.107:0/2885736059 learned_addr learned my addr 172.21.15.107:0/2885736059 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x5602666dbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f7467fff640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x56026661fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746d3ab640 1 -- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 msgr2=0x5602666dbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.937+0000 7f746d3ab640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x5602666dbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f746d3ab640 1 -- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 msgr2=0x56026661fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f746d3ab640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x56026661fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f746d3ab640 1 -- 172.21.15.107:0/2885736059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746815e2e0 con 0x560266626070 2023-11-24T08:17:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f746d3ab640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f7460004700 tx=0x7f74600047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78cf8ef122601bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74477fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:17:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460060900 con 0x560266626070 2023-11-24T08:17:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7460060d90 con 0x560266626070 2023-11-24T08:17:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7458011980 con 0x560266626070 2023-11-24T08:17:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f746000d040 con 0x560266626070 2023-11-24T08:17:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74719a7740 10 client.10011 fetch_fsmap learned FSMap version 97 2023-11-24T08:17:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x56026654e170 con 0x560266626070 2023-11-24T08:17:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.938+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460081020 con 0x560266626070 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 97) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f746001b710 con 0x560266626070 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 10 client.10011 fetch_fsmap finished waiting for FSMap version 97 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 10 client.10011 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 20 client.10011 populate_metadata read hostname 'smithi107' 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026667a650 0x56026667aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 4 client.10011 mds_command: new command op to 9873 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 --> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56026628d6c0 con 0x56026667a650 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026667a650 0x56026667aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f746cbaa640 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026667a650 0x56026667aa30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a17118200a0c0c0a server_cookie=c61d43983e0fada4 in_seq=0 out_seq=0 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.939+0000 7f74477fe640 10 client.10011 ms_handle_connect on v2:172.21.15.107:6836/1134531969 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74477fe640 1 -- 172.21.15.107:0/2885736059 <== mds.0 v2:172.21.15.107:6836/1134531969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56026628d6c0 con 0x56026667a650 2023-11-24T08:17:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74477fe640 10 client.10011 handle_command_reply: tid=0 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 client.10011 shutdown 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 msgr2=0x560266620310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f7460004700 tx=0x7f74600047d0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 shutdown_connections 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:6836/1134531969,v1:172.21.15.107:6837/1134531969] conn(0x56026667a650 0x56026667aa30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560266626070 0x560266620310 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602666257b0 0x56026661fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 --2- 172.21.15.107:0/2885736059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560266620d70 0x5602666dbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 >> 172.21.15.107:0/2885736059 conn(0x5602665e7c70 msgr2=0x56026666b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74477fe640 0 client.10011 ms_handle_reset on v2:172.21.15.107:6836/1134531969 2023-11-24T08:17:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 shutdown_connections 2023-11-24T08:17:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 1 -- 172.21.15.107:0/2885736059 wait complete. 2023-11-24T08:17:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.940+0000 7f74719a7740 20 client.10011 trim_cache size 0 max 16384 2023-11-24T08:17:47.958 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 9873, 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:17:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6825291599999996, 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 97, 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 162, 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 162, 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.112108593 2023-11-24T08:17:47.960 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:17:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 msgr2=0x7f745406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 0x7f745406eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7458007dc0 tx=0x7f74580072f0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 msgr2=0x7f74681591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f746000b8e0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 shutdown_connections 2023-11-24T08:17:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f745406c5e0 0x7f745406eaa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f746815a1c0 0x7f74681591b0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74680b15e0 0x7f74680b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 --2- 172.21.15.107:0/3392448577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f746806d640 0x7f746815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 >> 172.21.15.107:0/3392448577 conn(0x7f74680cbb40 msgr2=0x7f74680ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 shutdown_connections 2023-11-24T08:17:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:47.941+0000 7f746dbac640 1 -- 172.21.15.107:0/3392448577 wait complete. 2023-11-24T08:17:47.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:17:48.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.179+0000 7f4f22a7e640 1 Processor -- start 2023-11-24T08:17:48.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.179+0000 7f4f22a7e640 1 -- start start 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f22a7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f1c06e400 0x7f4f1c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f22a7e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 0x7f4f1c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f22a7e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c1508e0 con 0x7f4f1c15a270 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f22a7e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c15d7f0 con 0x7f4f1c06e400 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f22a7e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c15dad0 con 0x7f4f1c150480 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f21a7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f1c06e400 0x7f4f1c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:48.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f21a7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f1c06e400 0x7f4f1c071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37104/0 (socket says 172.21.15.107:37104) 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.180+0000 7f4f21a7c640 1 -- 172.21.15.107:0/2212528829 learned_addr learned my addr 172.21.15.107:0/2212528829 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f2127b640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 0x7f4f1c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f21a7c640 1 -- 172.21.15.107:0/2212528829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 msgr2=0x7f4f1c15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f21a7c640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 0x7f4f1c15cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f21a7c640 1 -- 172.21.15.107:0/2212528829 >> v1:172.21.15.107:6789/0 conn(0x7f4f1c15a270 legacy=0x7f4f1c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f21a7c640 1 -- 172.21.15.107:0/2212528829 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c15ddb0 con 0x7f4f1c06e400 2023-11-24T08:17:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.181+0000 7f4f2127b640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 0x7f4f1c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:48.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.182+0000 7f4f21a7c640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c071130 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009870 tx=0x7f4f1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abe267c88a93292 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.182+0000 7f4f20a7a640 1 -- 172.21.15.107:0/2212528829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030840 con 0x7f4f1c06e400 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.182+0000 7f4f20a7a640 1 -- 172.21.15.107:0/2212528829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f180309a0 con 0x7f4f1c06e400 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.182+0000 7f4f20a7a640 1 -- 172.21.15.107:0/2212528829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030d30 con 0x7f4f1c06e400 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 -- 172.21.15.107:0/2212528829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 msgr2=0x7f4f1c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c071130 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009870 tx=0x7f4f1802b980 comp rx=0 tx=0).stop 2023-11-24T08:17:48.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 -- 172.21.15.107:0/2212528829 shutdown_connections 2023-11-24T08:17:48.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c150480 0x7f4f1c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/2212528829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c071130 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 -- 172.21.15.107:0/2212528829 >> 172.21.15.107:0/2212528829 conn(0x7f4f1c0cb830 msgr2=0x7f4f1c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:48.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 -- 172.21.15.107:0/2212528829 shutdown_connections 2023-11-24T08:17:48.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.183+0000 7f4f22a7e640 1 -- 172.21.15.107:0/2212528829 wait complete. 2023-11-24T08:17:48.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.185+0000 7f4f22a7e640 1 Processor -- start 2023-11-24T08:17:48.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.185+0000 7f4f22a7e640 1 -- start start 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f1c150480 0x7f4f1c1f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c15a270 0x7f4f1c0b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c14dfe0 con 0x7f4f1c06e400 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f70d0 con 0x7f4f1c150480 2023-11-24T08:17:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f22a7e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f73b0 con 0x7f4f1c15a270 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37116/0 (socket says 172.21.15.107:37116) 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 -- 172.21.15.107:0/362307646 learned_addr learned my addr 172.21.15.107:0/362307646 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 -- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c15a270 msgr2=0x7f4f1c0b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c15a270 0x7f4f1c0b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f21a7c640 1 -- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f1c150480 msgr2=0x7f4f1c1f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.186+0000 7f4f2127b640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f1c150480 0x7f4f1c1f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f21a7c640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f1c150480 0x7f4f1c1f67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f21a7c640 1 -- 172.21.15.107:0/362307646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f180094d0 con 0x7f4f1c06e400 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f21a7c640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000c00 tx=0x7f4f18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030810 con 0x7f4f1c06e400 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f18039d90 con 0x7f4f1c06e400 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18043670 con 0x7f4f1c06e400 2023-11-24T08:17:48.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c1f7690 con 0x7f4f1c06e400 2023-11-24T08:17:48.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.187+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0b1d90 con 0x7f4f1c06e400 2023-11-24T08:17:48.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.188+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c067990 con 0x7f4f1c06e400 2023-11-24T08:17:48.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.189+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f4f1802f390 con 0x7f4f1c06e400 2023-11-24T08:17:48.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.190+0000 7f4f127fc640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 0x7f4f0006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:48.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.190+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f180aa760 con 0x7f4f1c06e400 2023-11-24T08:17:48.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.191+0000 7f4f2127b640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 0x7f4f0006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:48.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.191+0000 7f4f2127b640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 0x7f4f0006eaa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c14fb50 tx=0x7f4f0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:48.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.195+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f18077310 con 0x7f4f1c06e400 2023-11-24T08:17:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.430+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4f1c04e840 con 0x7f4f1c06e400 2023-11-24T08:17:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.431+0000 7f4f127fc640 1 -- 172.21.15.107:0/362307646 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4f18048060 con 0x7f4f1c06e400 2023-11-24T08:17:48.433 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.433+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 msgr2=0x7f4f0006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.433+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 0x7f4f0006eaa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c14fb50 tx=0x7f4f0c008040 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.433+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 msgr2=0x7f4f1c14daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.433+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000c00 tx=0x7f4f18004730 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 shutdown_connections 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4f0006c5e0 0x7f4f0006eaa0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1c15a270 0x7f4f1c0b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f1c150480 0x7f4f1c1f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 --2- 172.21.15.107:0/362307646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f1c06e400 0x7f4f1c14daa0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:48.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 >> 172.21.15.107:0/362307646 conn(0x7f4f1c0cb830 msgr2=0x7f4f1c14e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:48.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 shutdown_connections 2023-11-24T08:17:48.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:48.434+0000 7f4f22a7e640 1 -- 172.21.15.107:0/362307646 wait complete. 2023-11-24T08:17:48.453 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:17:48.453 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:17:48.454 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:17:48.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.454 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:17:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:17:48.507 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:17:48.507 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:17:48.507 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.507 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:48.508 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41602sec preferred_lft 41602sec 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:17:48.509 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:48.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.511 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:17:48.597 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:17:48.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.601 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:17:48.656 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:17:48.657 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:17:48.657 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:17:48.657 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:17:48.657 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:17:48.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:17:48.657 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:17:48.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:17:48.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:17:48.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:17:48.658 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:17:48.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.841 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:17:48.893 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:17:48.893 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:17:48.893 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:17:48.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:48.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:17:48.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:17:48.981 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:17:49.038 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:17:49.038 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:17:49.038 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:17:49.091 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:17:49.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:49.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:49.237 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:49.237 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:17:49.269 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:49.270 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:17:49.270 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:17:49.270 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:17:49.270 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:17:49.271 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:17:49.271 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:17:49.271 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:17:49.272 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:17:49.272 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:17:49.276 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:17:49.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:49.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:17:49.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:49.356 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:17:49.415 INFO:teuthology.orchestra.run.smithi107.stdout:client10020 2023-11-24T08:17:49.415 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:49.415 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:49.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:17:49.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.725+0000 7f76819f5640 1 Processor -- start 2023-11-24T08:17:49.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.725+0000 7f76819f5640 1 -- start start 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f76819f5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f767c0b2250 0x7f767c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f76819f5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f767c0b2aa0 0x7f767c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f76819f5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c15ab80 con 0x7f767c0b2630 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f76819f5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c15a990 con 0x7f767c0b2250 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f76819f5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c15e680 con 0x7f767c0b2aa0 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.727+0000 7f767affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f767c0b2250 0x7f767c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f767c0b2250 0x7f767c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37132/0 (socket says 172.21.15.107:37132) 2023-11-24T08:17:49.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 -- 172.21.15.107:0/4237332418 learned_addr learned my addr 172.21.15.107:0/4237332418 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 -- 172.21.15.107:0/4237332418 >> v2:172.21.15.120:3300/0 conn(0x7f767c0b2aa0 msgr2=0x7f767c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 --2- 172.21.15.107:0/4237332418 >> v2:172.21.15.120:3300/0 conn(0x7f767c0b2aa0 0x7f767c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 -- 172.21.15.107:0/4237332418 >> v1:172.21.15.120:6789/0 conn(0x7f767c0b2630 legacy=0x7f767c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.728+0000 7f767affd640 1 -- 172.21.15.107:0/4237332418 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c15e960 con 0x7f767c0b2250 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.729+0000 7f767affd640 1 --2- 172.21.15.107:0/4237332418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c15df80 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f767c0b2fa0 tx=0x7f766c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7faf0d697ca2b69 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.729+0000 7f767a7fc640 1 -- 172.21.15.107:0/4237332418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c031870 con 0x7f767c0b2250 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.729+0000 7f767a7fc640 1 -- 172.21.15.107:0/4237332418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c0319d0 con 0x7f767c0b2250 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.729+0000 7f767a7fc640 1 -- 172.21.15.107:0/4237332418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c031d60 con 0x7f767c0b2250 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 -- 172.21.15.107:0/4237332418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 msgr2=0x7f767c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 --2- 172.21.15.107:0/4237332418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c15df80 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f767c0b2fa0 tx=0x7f766c02b910 comp rx=0 tx=0).stop 2023-11-24T08:17:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 -- 172.21.15.107:0/4237332418 shutdown_connections 2023-11-24T08:17:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 --2- 172.21.15.107:0/4237332418 >> v2:172.21.15.120:3300/0 conn(0x7f767c0b2aa0 0x7f767c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 --2- 172.21.15.107:0/4237332418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c15df80 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 -- 172.21.15.107:0/4237332418 >> 172.21.15.107:0/4237332418 conn(0x7f767c06a920 msgr2=0x7f767c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 -- 172.21.15.107:0/4237332418 shutdown_connections 2023-11-24T08:17:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.730+0000 7f76819f5640 1 -- 172.21.15.107:0/4237332418 wait complete. 2023-11-24T08:17:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.732+0000 7f76819f5640 1 Processor -- start 2023-11-24T08:17:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.732+0000 7f76819f5640 1 -- start start 2023-11-24T08:17:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f767c0b2630 0x7f767c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f767c0b2aa0 0x7f767c0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c1565d0 con 0x7f767c0b2250 2023-11-24T08:17:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c14c570 con 0x7f767c0b2aa0 2023-11-24T08:17:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f76819f5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c156a60 con 0x7f767c0b2630 2023-11-24T08:17:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f767affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f767affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37146/0 (socket says 172.21.15.107:37146) 2023-11-24T08:17:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.733+0000 7f767affd640 1 -- 172.21.15.107:0/1371577768 learned_addr learned my addr 172.21.15.107:0/1371577768 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 -- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f767c0b2630 msgr2=0x7f767c155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f7673fff640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f767c0b2630 0x7f767c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767b7fe640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f767c0b2aa0 0x7f767c0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f767c0b2630 0x7f767c155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 -- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f767c0b2aa0 msgr2=0x7f767c0b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f767c0b2aa0 0x7f767c0b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 -- 172.21.15.107:0/1371577768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c0094d0 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.734+0000 7f767affd640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f766c02b950 tx=0x7f766c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.735+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c03ad10 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.735+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c03ae70 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.735+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c044610 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.735+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c0cd7c0 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.735+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c155ca0 con 0x7f767c0b2250 2023-11-24T08:17:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.736+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c067470 con 0x7f767c0b2250 2023-11-24T08:17:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.737+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f766c044770 con 0x7f767c0b2250 2023-11-24T08:17:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.738+0000 7f7678ff9640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 0x7f765c072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.738+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f766c02d040 con 0x7f767c0b2250 2023-11-24T08:17:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.738+0000 7f7673fff640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 0x7f765c072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.739+0000 7f7673fff640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 0x7f765c072eb0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f767c09d310 tx=0x7f7664009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.742+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f766c077780 con 0x7f767c0b2250 2023-11-24T08:17:49.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.975+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f767c0cea00 con 0x7f767c0b2250 2023-11-24T08:17:49.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.976+0000 7f7678ff9640 1 -- 172.21.15.107:0/1371577768 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7f766c07ada0 con 0x7f767c0b2250 2023-11-24T08:17:49.978 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:17:49.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 msgr2=0x7f765c072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:49.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 0x7f765c072eb0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f767c09d310 tx=0x7f7664009040 comp rx=0 tx=0).stop 2023-11-24T08:17:49.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 msgr2=0x7f767c14c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:49.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f766c02b950 tx=0x7f766c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 shutdown_connections 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f765c0709f0 0x7f765c072eb0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f767c0b2aa0 0x7f767c0b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f767c0b2630 0x7f767c155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 --2- 172.21.15.107:0/1371577768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c0b2250 0x7f767c14c030 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 >> 172.21.15.107:0/1371577768 conn(0x7f767c06a920 msgr2=0x7f767c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 shutdown_connections 2023-11-24T08:17:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:49.979+0000 7f76819f5640 1 -- 172.21.15.107:0/1371577768 wait complete. 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:17:50.001 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:17:50.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.002 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:17:50.059 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:50.060 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41599sec preferred_lft 41599sec 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:17:50.061 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:17:50.063 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:17:50.063 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:17:50.063 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:17:50.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.063 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:17:50.154 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:17:50.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.158 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:17:50.214 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:17:50.214 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:17:50.214 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:17:50.214 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:17:50.214 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:17:50.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.218 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:17:50.218 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:17:50.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:17:50.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:17:50.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:17:50.219 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:17:50.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.405 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:17:50.454 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:17:50.454 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:17:50.454 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:17:50.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.455 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:17:50.597 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:17:50.597 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:17:50.653 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:17:50.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:17:50.654 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:17:50.708 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:17:50.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.708 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:50.848 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:50.848 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:17:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:17:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:17:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:17:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:17:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:17:50.887 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:17:50.887 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:17:50.887 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:17:50.887 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:17:50.887 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:17:50.890 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:17:50.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.891 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:17:50.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:50.973 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:17:51.030 INFO:teuthology.orchestra.run.smithi120.stdout:client10029 2023-11-24T08:17:51.030 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:17:51.031 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:51.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:17:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.311+0000 7fdfd7101640 1 Processor -- start 2023-11-24T08:17:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.311+0000 7fdfd7101640 1 -- start start 2023-11-24T08:17:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfd7101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd01461d0 0x7fdfd00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfd7101640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdfd00b4fe0 0x7fdfd00b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfd7101640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd006d180 con 0x7fdfd00b3890 2023-11-24T08:17:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfd7101640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd00b3d00 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfd7101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd006ba70 con 0x7fdfd01461d0 2023-11-24T08:17:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdfd00b4fe0 0x7fdfd00b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdfd00b4fe0 0x7fdfd00b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37152/0 (socket says 172.21.15.107:37152) 2023-11-24T08:17:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.313+0000 7fdfcffff640 1 -- 172.21.15.107:0/272080829 learned_addr learned my addr 172.21.15.107:0/272080829 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.314+0000 7fdfd4e9d640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd01461d0 0x7fdfd00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.314+0000 7fdfcffff640 1 -- 172.21.15.107:0/272080829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd01461d0 msgr2=0x7fdfd00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.314+0000 7fdfcffff640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd01461d0 0x7fdfd00b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.314+0000 7fdfcffff640 1 -- 172.21.15.107:0/272080829 >> v1:172.21.15.107:6789/0 conn(0x7fdfd00b3890 legacy=0x7fdfd015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.314+0000 7fdfcffff640 1 -- 172.21.15.107:0/272080829 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd006bd50 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.315+0000 7fdfcffff640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00b3350 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009870 tx=0x7fdfbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aa3264b2fc5252 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.315+0000 7fdfcf7fe640 1 -- 172.21.15.107:0/272080829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc030840 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.315+0000 7fdfcf7fe640 1 -- 172.21.15.107:0/272080829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc0309a0 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.315+0000 7fdfcf7fe640 1 -- 172.21.15.107:0/272080829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc030d30 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 -- 172.21.15.107:0/272080829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd00b4fe0 msgr2=0x7fdfd00b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00b3350 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009870 tx=0x7fdfbc02b980 comp rx=0 tx=0).stop 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 -- 172.21.15.107:0/272080829 shutdown_connections 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00b3350 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 --2- 172.21.15.107:0/272080829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd01461d0 0x7fdfd00b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 -- 172.21.15.107:0/272080829 >> 172.21.15.107:0/272080829 conn(0x7fdfd0061340 msgr2=0x7fdfd0061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 -- 172.21.15.107:0/272080829 shutdown_connections 2023-11-24T08:17:51.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.316+0000 7fdfd7101640 1 -- 172.21.15.107:0/272080829 wait complete. 2023-11-24T08:17:51.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.318+0000 7fdfd7101640 1 Processor -- start 2023-11-24T08:17:51.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.318+0000 7fdfd7101640 1 -- start start 2023-11-24T08:17:51.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd00b3890 0x7fdfd013a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd013ab10 con 0x7fdfd01461d0 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00cade0 con 0x7fdfd00b4fe0 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd7101640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd013ad40 con 0x7fdfd00b3890 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd569e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd569e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37164/0 (socket says 172.21.15.107:37164) 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.319+0000 7fdfd569e640 1 -- 172.21.15.107:0/1511895972 learned_addr learned my addr 172.21.15.107:0/1511895972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd4e9d640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd00b3890 0x7fdfd013a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfcffff640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 -- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd00b3890 msgr2=0x7fdfd013a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd00b3890 0x7fdfd013a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 -- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 msgr2=0x7fdfd00ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 -- 172.21.15.107:0/1511895972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc0094d0 con 0x7fdfd01461d0 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfcffff640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfd569e640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800c7a0 tx=0x7fdfc800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8003b70 con 0x7fdfd01461d0 2023-11-24T08:17:51.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc8003cf0 con 0x7fdfd01461d0 2023-11-24T08:17:51.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.320+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8003ea0 con 0x7fdfd01461d0 2023-11-24T08:17:51.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.321+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd013b020 con 0x7fdfd01461d0 2023-11-24T08:17:51.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.321+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd013b5e0 con 0x7fdfd01461d0 2023-11-24T08:17:51.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.322+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd0067ea0 con 0x7fdfd01461d0 2023-11-24T08:17:51.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.323+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fdfc800aa90 con 0x7fdfd01461d0 2023-11-24T08:17:51.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.324+0000 7fdfcd7fa640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 0x7fdfb406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.324+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfc8088270 con 0x7fdfd01461d0 2023-11-24T08:17:51.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.324+0000 7fdfd4e9d640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 0x7fdfb406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.325+0000 7fdfd4e9d640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 0x7fdfb406eaa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4006fd0 tx=0x7fdfc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.328+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfc8054d80 con 0x7fdfd01461d0 2023-11-24T08:17:51.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.551+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdfd004e930 con 0x7fdfd01461d0 2023-11-24T08:17:51.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.606+0000 7fdfcd7fa640 1 -- 172.21.15.107:0/1511895972 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v98) v1 ==== 136+0+0 (secure 0 0 0) 0x7fdfc80583a0 con 0x7fdfd01461d0 2023-11-24T08:17:51.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 msgr2=0x7fdfb406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 0x7fdfb406eaa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4006fd0 tx=0x7fdfc4008040 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 msgr2=0x7fdfd01437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800c7a0 tx=0x7fdfc800cc60 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 shutdown_connections 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdfb406c5e0 0x7fdfb406eaa0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01461d0 0x7fdfd01437c0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd00b4fe0 0x7fdfd00ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.609+0000 7fdfd7101640 1 --2- 172.21.15.107:0/1511895972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfd00b3890 0x7fdfd013a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.610+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 >> 172.21.15.107:0/1511895972 conn(0x7fdfd0061340 msgr2=0x7fdfd00b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:51.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.610+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 shutdown_connections 2023-11-24T08:17:51.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.610+0000 7fdfd7101640 1 -- 172.21.15.107:0/1511895972 wait complete. 2023-11-24T08:17:51.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:17:51.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.844+0000 7ff055a94640 1 Processor -- start 2023-11-24T08:17:51.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.844+0000 7ff055a94640 1 -- start start 2023-11-24T08:17:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.845+0000 7ff055a94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 0x7ff05015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.845+0000 7ff055a94640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0500c49f0 0x7ff0500705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.845+0000 7ff055a94640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05015ec90 con 0x7ff050072160 2023-11-24T08:17:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.845+0000 7ff055a94640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0500cd530 con 0x7ff0500c49f0 2023-11-24T08:17:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.845+0000 7ff055a94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0500c4ed0 con 0x7ff05006de10 2023-11-24T08:17:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0500c49f0 0x7ff0500705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 0x7ff05015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0500c49f0 0x7ff0500705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:40460/0 (socket says 172.21.15.107:40460) 2023-11-24T08:17:51.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 -- 172.21.15.107:0/3354141563 learned_addr learned my addr 172.21.15.107:0/3354141563 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:51.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 -- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 msgr2=0x7ff05015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:51.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 0x7ff05015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 -- 172.21.15.107:0/3354141563 >> v1:172.21.15.120:6789/0 conn(0x7ff050072160 legacy=0x7ff05006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04ffff640 1 -- 172.21.15.107:0/3354141563 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0500c51b0 con 0x7ff0500c49f0 2023-11-24T08:17:51.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.846+0000 7ff04effd640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 0x7ff05015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:17:51.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.848+0000 7ff04ffff640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0500c49f0 0x7ff0500705c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff04802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c91d07b600e818 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.848+0000 7ff04e7fc640 1 -- 172.21.15.107:0/3354141563 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048030840 con 0x7ff0500c49f0 2023-11-24T08:17:51.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.848+0000 7ff04e7fc640 1 -- 172.21.15.107:0/3354141563 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0480309a0 con 0x7ff0500c49f0 2023-11-24T08:17:51.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.848+0000 7ff04e7fc640 1 -- 172.21.15.107:0/3354141563 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048030d30 con 0x7ff0500c49f0 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 -- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0500c49f0 msgr2=0x7ff0500705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0500c49f0 0x7ff0500705c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff04802b980 comp rx=0 tx=0).stop 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 -- 172.21.15.107:0/3354141563 shutdown_connections 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0500c49f0 0x7ff0500705c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 --2- 172.21.15.107:0/3354141563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff05006de10 0x7ff05015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 -- 172.21.15.107:0/3354141563 >> 172.21.15.107:0/3354141563 conn(0x7ff0500b0c50 msgr2=0x7ff0500b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 -- 172.21.15.107:0/3354141563 shutdown_connections 2023-11-24T08:17:51.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.849+0000 7ff055a94640 1 -- 172.21.15.107:0/3354141563 wait complete. 2023-11-24T08:17:51.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.851+0000 7ff055a94640 1 Processor -- start 2023-11-24T08:17:51.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.851+0000 7ff055a94640 1 -- start start 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05006de10 0x7ff0500c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff050072160 0x7ff050145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05015a9e0 con 0x7ff0500c49f0 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05015acc0 con 0x7ff05006de10 2023-11-24T08:17:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff055a94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05015afa0 con 0x7ff050072160 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37180/0 (socket says 172.21.15.107:37180) 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 -- 172.21.15.107:0/138056375 learned_addr learned my addr 172.21.15.107:0/138056375 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04f7fe640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05006de10 0x7ff0500c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04effd640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff050072160 0x7ff050145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 -- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff050072160 msgr2=0x7ff050145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff050072160 0x7ff050145d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 -- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05006de10 msgr2=0x7ff0500c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05006de10 0x7ff0500c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.852+0000 7ff04ffff640 1 -- 172.21.15.107:0/138056375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff038009400 con 0x7ff0500c49f0 2023-11-24T08:17:51.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff04ffff640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff048004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048039a90 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff048039c10 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048043410 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0480094d0 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.853+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05015b560 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.855+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05004e8b0 con 0x7ff0500c49f0 2023-11-24T08:17:51.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.855+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7ff04802f390 con 0x7ff0500c49f0 2023-11-24T08:17:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.856+0000 7ff047fff640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 0x7ff03006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.856+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0480aa3e0 con 0x7ff0500c49f0 2023-11-24T08:17:51.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.856+0000 7ff04f7fe640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 0x7ff03006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:51.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.857+0000 7ff04f7fe640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 0x7ff03006ead0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff0400012e0 tx=0x7ff040009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:51.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:51.860+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff048076f90 con 0x7ff0500c49f0 2023-11-24T08:17:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.064+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0500c5a50 con 0x7ff03006c610 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.066+0000 7ff047fff640 1 -- 172.21.15.107:0/138056375 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff040001aa0 con 0x7ff03006c610 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:17:52.068 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.068+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 msgr2=0x7ff03006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.068+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 0x7ff03006ead0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff0400012e0 tx=0x7ff040009040 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.068+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 msgr2=0x7ff05015ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.068+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff048004730 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 shutdown_connections 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff03006c610 0x7ff03006ead0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0500c49f0 0x7ff05015ba80 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff050072160 0x7ff050145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 --2- 172.21.15.107:0/138056375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05006de10 0x7ff0500c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 >> 172.21.15.107:0/138056375 conn(0x7ff0500b0c50 msgr2=0x7ff0500aca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 shutdown_connections 2023-11-24T08:17:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.069+0000 7ff055a94640 1 -- 172.21.15.107:0/138056375 wait complete. 2023-11-24T08:17:52.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:17:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.298+0000 7f3432fdc640 1 Processor -- start 2023-11-24T08:17:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.298+0000 7f3432fdc640 1 -- start start 2023-11-24T08:17:52.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3432fdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 0x7f342c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3432fdc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342c1655f0 con 0x7f342c0c9650 2023-11-24T08:17:52.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3432fdc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342c0cc380 con 0x7f342c0ccfb0 2023-11-24T08:17:52.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3432fdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342c162b30 con 0x7f342c0ccbd0 2023-11-24T08:17:52.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3431579640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f342c0c9650 0x7f342c0c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51972/0 (socket says 172.21.15.107:51972) 2023-11-24T08:17:52.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.300+0000 7f3431579640 1 -- 172.21.15.107:0/1352739408 learned_addr learned my addr 172.21.15.107:0/1352739408 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:17:52.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f3430d78640 1 --2- 172.21.15.107:0/1352739408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 0x7f342c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794607182 0 0) 0x7f342c1655f0 con 0x7f342c0c9650 2023-11-24T08:17:52.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c003520 con 0x7f342c0c9650 2023-11-24T08:17:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973784187 0 0) 0x7f341c003520 con 0x7f342c0c9650 2023-11-24T08:17:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f3430d78640 1 -- 172.21.15.107:0/1352739408 >> v1:172.21.15.120:6789/0 conn(0x7f342c0ccfb0 legacy=0x7f342c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f3430d78640 1 -- 172.21.15.107:0/1352739408 >> v1:172.21.15.107:6789/0 conn(0x7f342c0c9650 legacy=0x7f342c0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.301+0000 7f3430d78640 1 -- 172.21.15.107:0/1352739408 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c162e10 con 0x7f342c0ccbd0 2023-11-24T08:17:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.303+0000 7f3430d78640 1 --2- 172.21.15.107:0/1352739408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 0x7f342c1660b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3420009800 tx=0x7f342002b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa136ee61b607c21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.303+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240019f0 con 0x7f342c0ccbd0 2023-11-24T08:17:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.303+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34200314d0 con 0x7f342c0ccbd0 2023-11-24T08:17:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.303+0000 7f342b7fe640 1 -- 172.21.15.107:0/1352739408 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420031860 con 0x7f342c0ccbd0 2023-11-24T08:17:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 -- 172.21.15.107:0/1352739408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 msgr2=0x7f342c1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 --2- 172.21.15.107:0/1352739408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 0x7f342c1660b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3420009800 tx=0x7f342002b7f0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 -- 172.21.15.107:0/1352739408 shutdown_connections 2023-11-24T08:17:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 --2- 172.21.15.107:0/1352739408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccbd0 0x7f342c1660b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 -- 172.21.15.107:0/1352739408 >> 172.21.15.107:0/1352739408 conn(0x7f342c0b0c50 msgr2=0x7f342c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 -- 172.21.15.107:0/1352739408 shutdown_connections 2023-11-24T08:17:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.304+0000 7f3432fdc640 1 -- 172.21.15.107:0/1352739408 wait complete. 2023-11-24T08:17:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.306+0000 7f3432fdc640 1 Processor -- start 2023-11-24T08:17:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.306+0000 7f3432fdc640 1 -- start start 2023-11-24T08:17:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f342c0ccbd0 0x7f342c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccfb0 0x7f342c164e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c160950 con 0x7f342c0c9650 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c15fd40 con 0x7f342c0ccbd0 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3432fdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342c160020 con 0x7f342c0ccfb0 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3430d78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3430d78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37182/0 (socket says 172.21.15.107:37182) 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.307+0000 7f3430d78640 1 -- 172.21.15.107:0/2015459049 learned_addr learned my addr 172.21.15.107:0/2015459049 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f342bfff640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f342c0ccbd0 0x7f342c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 -- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccfb0 msgr2=0x7f342c164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccfb0 0x7f342c164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 -- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f342c0ccbd0 msgr2=0x7f342c158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f342c0ccbd0 0x7f342c158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 -- 172.21.15.107:0/2015459049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420009400 con 0x7f342c0c9650 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3430d78640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3420037040 tx=0x7f34200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420004220 con 0x7f342c0c9650 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420004380 con 0x7f342c0c9650 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342003e8a0 con 0x7f342c0c9650 2023-11-24T08:17:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.308+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342c160300 con 0x7f342c0c9650 2023-11-24T08:17:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.309+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342c1588b0 con 0x7f342c0c9650 2023-11-24T08:17:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.310+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342c04e8b0 con 0x7f342c0c9650 2023-11-24T08:17:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.310+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f342003b040 con 0x7f342c0c9650 2023-11-24T08:17:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.311+0000 7f34297fa640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 0x7f341006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.311+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34200aa140 con 0x7f342c0c9650 2023-11-24T08:17:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.312+0000 7f342bfff640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 0x7f341006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.312+0000 7f342bfff640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 0x7f341006ead0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f342c0c9160 tx=0x7f3418009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.316+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3420076e10 con 0x7f342c0c9650 2023-11-24T08:17:52.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.508+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f342c12c1b0 con 0x7f342c0c9650 2023-11-24T08:17:52.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.509+0000 7f34297fa640 1 -- 172.21.15.107:0/2015459049 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v43)=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3420039030 con 0x7f342c0c9650 2023-11-24T08:17:52.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 msgr2=0x7f341006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 0x7f341006ead0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f342c0c9160 tx=0x7f3418009040 comp rx=0 tx=0).stop 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 msgr2=0x7f342c15a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3420037040 tx=0x7f34200046c0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 shutdown_connections 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f341006c610 0x7f341006ead0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.511+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f342c0ccfb0 0x7f342c164e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.512+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f342c0ccbd0 0x7f342c158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.512+0000 7f3432fdc640 1 --2- 172.21.15.107:0/2015459049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f342c0c9650 0x7f342c15a6b0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.512+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 >> 172.21.15.107:0/2015459049 conn(0x7f342c0b0c50 msgr2=0x7f342c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:52.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.512+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 shutdown_connections 2023-11-24T08:17:52.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.512+0000 7f3432fdc640 1 -- 172.21.15.107:0/2015459049 wait complete. 2023-11-24T08:17:52.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol2512 --mode=777 2023-11-24T08:17:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.754+0000 7f97daad1640 1 Processor -- start 2023-11-24T08:17:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.754+0000 7f97daad1640 1 -- start start 2023-11-24T08:17:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f97d4160b00 0x7f97d40d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97d40d12e0 0x7f97d40cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d0120 0x7f97d415ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d40d05b0 con 0x7f97d40d12e0 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d415ea30 con 0x7f97d4160b00 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97daad1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d415de20 con 0x7f97d40d0120 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97d92ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97d40d12e0 0x7f97d40cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97d92ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97d40d12e0 0x7f97d40cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37194/0 (socket says 172.21.15.107:37194) 2023-11-24T08:17:52.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.756+0000 7f97d92ce640 1 -- 172.21.15.107:0/175671423 learned_addr learned my addr 172.21.15.107:0/175671423 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97da2d0640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d0120 0x7f97d415ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d9acf640 1 --2- 172.21.15.107:0/175671423 >> v2:172.21.15.120:3300/0 conn(0x7f97d4160b00 0x7f97d40d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d92ce640 1 -- 172.21.15.107:0/175671423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d0120 msgr2=0x7f97d415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d92ce640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d0120 0x7f97d415ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d92ce640 1 -- 172.21.15.107:0/175671423 >> v2:172.21.15.120:3300/0 conn(0x7f97d4160b00 msgr2=0x7f97d40d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d92ce640 1 --2- 172.21.15.107:0/175671423 >> v2:172.21.15.120:3300/0 conn(0x7f97d4160b00 0x7f97d40d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.757+0000 7f97d92ce640 1 -- 172.21.15.107:0/175671423 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d415e100 con 0x7f97d40d12e0 2023-11-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.758+0000 7f97d92ce640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d12e0 0x7f97d40cfbe0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f97d408d3a0 tx=0x7f97c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b356010cdcb77e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.759+0000 7f97d8acd640 1 -- 172.21.15.107:0/175671423 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4031870 con 0x7f97d40d12e0 2023-11-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.759+0000 7f97d8acd640 1 -- 172.21.15.107:0/175671423 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c40319d0 con 0x7f97d40d12e0 2023-11-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.759+0000 7f97d8acd640 1 -- 172.21.15.107:0/175671423 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4031d60 con 0x7f97d40d12e0 2023-11-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 -- 172.21.15.107:0/175671423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d12e0 msgr2=0x7f97d40cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d12e0 0x7f97d40cfbe0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f97d408d3a0 tx=0x7f97c402b910 comp rx=0 tx=0).stop 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 -- 172.21.15.107:0/175671423 shutdown_connections 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d0120 0x7f97d415ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 --2- 172.21.15.107:0/175671423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d12e0 0x7f97d40cfbe0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 --2- 172.21.15.107:0/175671423 >> v2:172.21.15.120:3300/0 conn(0x7f97d4160b00 0x7f97d40d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 -- 172.21.15.107:0/175671423 >> 172.21.15.107:0/175671423 conn(0x7f97d409c510 msgr2=0x7f97d40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 -- 172.21.15.107:0/175671423 shutdown_connections 2023-11-24T08:17:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.760+0000 7f97daad1640 1 -- 172.21.15.107:0/175671423 wait complete. 2023-11-24T08:17:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.762+0000 7f97daad1640 1 Processor -- start 2023-11-24T08:17:52.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.762+0000 7f97daad1640 1 -- start start 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d12e0 0x7f97d41f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 0x7f97d41f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40caeb0 con 0x7f97d40d0120 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40ca2a0 con 0x7f97d4160b00 2023-11-24T08:17:52.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97daad1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40ca580 con 0x7f97d40d12e0 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97d9acf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97d9acf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37206/0 (socket says 172.21.15.107:37206) 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97d9acf640 1 -- 172.21.15.107:0/3575913167 learned_addr learned my addr 172.21.15.107:0/3575913167 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97d92ce640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d12e0 0x7f97d41f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.763+0000 7f97d9acf640 1 -- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d12e0 msgr2=0x7f97d41f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97d9acf640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d12e0 0x7f97d41f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97da2d0640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 0x7f97d41f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97d9acf640 1 -- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 msgr2=0x7f97d41f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97d9acf640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 0x7f97d41f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97d9acf640 1 -- 172.21.15.107:0/3575913167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40094d0 con 0x7f97d40d0120 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97da2d0640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 0x7f97d41f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97d9acf640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f97d000d5f0 tx=0x7f97d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00125f0 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0012750 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d40ca860 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012b00 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.764+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d40b4920 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.765+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d404e840 con 0x7f97d40d0120 2023-11-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.766+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7f97d0012c60 con 0x7f97d40d0120 2023-11-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.767+0000 7f97ca7fc640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 0x7f97b806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.767+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97d0088840 con 0x7f97d40d0120 2023-11-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.768+0000 7f97d92ce640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 0x7f97b806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.768+0000 7f97d92ce640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 0x7f97b806eaa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f97c4036000 tx=0x7f97c403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.772+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97d00553f0 con 0x7f97d40d0120 2023-11-24T08:17:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:52.988+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol2512", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f97d415ee40 con 0x7f97b806c5e0 2023-11-24T08:17:53.004 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:53.002+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:53.004 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:53.002+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:53.004 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:53.002+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:53.004 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:53.002+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:53.004 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:17:53.002+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:17:53.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.060+0000 7f97ca7fc640 1 -- 172.21.15.107:0/3575913167 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f97c4031cb0 con 0x7f97b806c5e0 2023-11-24T08:17:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 msgr2=0x7f97b806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 0x7f97b806eaa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f97c4036000 tx=0x7f97c403c040 comp rx=0 tx=0).stop 2023-11-24T08:17:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 msgr2=0x7f97d40b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f97d000d5f0 tx=0x7f97d000dab0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 shutdown_connections 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f97b806c5e0 0x7f97b806eaa0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97d4160b00 0x7f97d41f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97d40d12e0 0x7f97d41f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 --2- 172.21.15.107:0/3575913167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97d40d0120 0x7f97d40b43e0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 >> 172.21.15.107:0/3575913167 conn(0x7f97d409c510 msgr2=0x7f97d415c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 shutdown_connections 2023-11-24T08:17:53.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.063+0000 7f97daad1640 1 -- 172.21.15.107:0/3575913167 wait complete. 2023-11-24T08:17:53.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol2512 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.285+0000 7fe15f456640 1 Processor -- start 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.285+0000 7fe15f456640 1 -- start start 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15f456640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1580c59b0 0x7fe158166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15f456640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1580c5dc0 con 0x7fe1580cc0f0 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15f456640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15815eea0 con 0x7fe15815b440 2023-11-24T08:17:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15f456640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15815b820 con 0x7fe1580c59b0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15d9f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1580c59b0 0x7fe158166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15d9f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1580c59b0 0x7fe158166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37224/0 (socket says 172.21.15.107:37224) 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.287+0000 7fe15d9f3640 1 -- 172.21.15.107:0/4248458048 learned_addr learned my addr 172.21.15.107:0/4248458048 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.288+0000 7fe15d9f3640 1 -- 172.21.15.107:0/4248458048 >> v1:172.21.15.120:6789/0 conn(0x7fe15815b440 legacy=0x7fe1580cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.288+0000 7fe15d9f3640 1 -- 172.21.15.107:0/4248458048 >> v1:172.21.15.107:6789/0 conn(0x7fe1580cc0f0 legacy=0x7fe1580c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.288+0000 7fe15d9f3640 1 -- 172.21.15.107:0/4248458048 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15806d600 con 0x7fe1580c59b0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.289+0000 7fe15d9f3640 1 --2- 172.21.15.107:0/4248458048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1580c59b0 0x7fe158166230 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fe150009e50 tx=0x7fe15002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e839b732a98716f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.289+0000 7fe14ffff640 1 -- 172.21.15.107:0/4248458048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15002ca80 con 0x7fe1580c59b0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.289+0000 7fe14ffff640 1 -- 172.21.15.107:0/4248458048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15002cbe0 con 0x7fe1580c59b0 2023-11-24T08:17:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.289+0000 7fe14ffff640 1 -- 172.21.15.107:0/4248458048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1500343f0 con 0x7fe1580c59b0 2023-11-24T08:17:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 -- 172.21.15.107:0/4248458048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1580c59b0 msgr2=0x7fe158166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 --2- 172.21.15.107:0/4248458048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1580c59b0 0x7fe158166230 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fe150009e50 tx=0x7fe15002bee0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 -- 172.21.15.107:0/4248458048 shutdown_connections 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 --2- 172.21.15.107:0/4248458048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1580c59b0 0x7fe158166230 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 -- 172.21.15.107:0/4248458048 >> 172.21.15.107:0/4248458048 conn(0x7fe1580b0c50 msgr2=0x7fe1580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 -- 172.21.15.107:0/4248458048 shutdown_connections 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.290+0000 7fe15f456640 1 -- 172.21.15.107:0/4248458048 wait complete. 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.292+0000 7fe15f456640 1 Processor -- start 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.292+0000 7fe15f456640 1 -- start start 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1580c59b0 0x7fe158154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 0x7fe1580ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158154f20 con 0x7fe15815b440 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581554b0 con 0x7fe1580c59b0 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15f456640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe158155790 con 0x7fe1580cc0f0 2023-11-24T08:17:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15d9f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15d9f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37236/0 (socket says 172.21.15.107:37236) 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.293+0000 7fe15d9f3640 1 -- 172.21.15.107:0/2413466844 learned_addr learned my addr 172.21.15.107:0/2413466844 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d1f2640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1580c59b0 0x7fe158154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15c9f1640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 0x7fe1580ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 -- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 msgr2=0x7fe1580ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 0x7fe1580ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 -- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1580c59b0 msgr2=0x7fe158154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1580c59b0 0x7fe158154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 -- 172.21.15.107:0/2413466844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe150009b30 con 0x7fe15815b440 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15c9f1640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 0x7fe1580ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15d9f3640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe150036040 tx=0x7fe150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15003e450 con 0x7fe15815b440 2023-11-24T08:17:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15003e5d0 con 0x7fe15815b440 2023-11-24T08:17:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15003e980 con 0x7fe15815b440 2023-11-24T08:17:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.294+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe158155a70 con 0x7fe15815b440 2023-11-24T08:17:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.295+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe158156030 con 0x7fe15815b440 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.296+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe158156250 con 0x7fe15815b440 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.297+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 32) v1 ==== 91373+0+0 (secure 0 0 0) 0x7fe15002cd50 con 0x7fe15815b440 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.297+0000 7fe14dffb640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 0x7fe13c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.298+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1500ab010 con 0x7fe15815b440 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.298+0000 7fe15d1f2640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 0x7fe13c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:17:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.298+0000 7fe15d1f2640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 0x7fe13c06eaa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe154006fd0 tx=0x7fe154008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:17:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.302+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe150077cf0 con 0x7fe15815b440 2023-11-24T08:17:53.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.516+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol2512", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1580ca610 con 0x7fe13c06c5e0 2023-11-24T08:17:53.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.529+0000 7fe14dffb640 1 -- 172.21.15.107:0/2413466844 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fe1580ca610 con 0x7fe13c06c5e0 2023-11-24T08:17:53.531 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 msgr2=0x7fe13c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 0x7fe13c06eaa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe154006fd0 tx=0x7fe154008040 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 msgr2=0x7fe1580ad7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe150036040 tx=0x7fe150004a40 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 shutdown_connections 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe13c06c5e0 0x7fe13c06eaa0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe15815b440 0x7fe1580ad7d0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1580cc0f0 0x7fe1580ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 --2- 172.21.15.107:0/2413466844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1580c59b0 0x7fe158154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 >> 172.21.15.107:0/2413466844 conn(0x7fe1580b0c50 msgr2=0x7fe1580cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:17:53.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 shutdown_connections 2023-11-24T08:17:53.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:17:53.532+0000 7fe15f456640 1 -- 172.21.15.107:0/2413466844 wait complete. 2023-11-24T08:17:53.551 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol2512 with 64 files each 1MB size under directory volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793 2023-11-24T08:17:53.551 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:53.552 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:53.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:53.599 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:53.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:53.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:53.734 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283976 s, 36.9 MB/s 2023-11-24T08:17:53.735 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:53.735 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:53.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:53.791 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:53.875 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:53.875 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:53.875 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262503 s, 39.9 MB/s 2023-11-24T08:17:53.877 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:53.877 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:53.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:53.934 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.010 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.010 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.010 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227386 s, 46.1 MB/s 2023-11-24T08:17:54.011 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.012 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.138 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.138 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019011 s, 55.2 MB/s 2023-11-24T08:17:54.140 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.140 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.264 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.264 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.264 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020294 s, 51.7 MB/s 2023-11-24T08:17:54.265 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.265 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.402 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.402 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.402 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023645 s, 44.3 MB/s 2023-11-24T08:17:54.404 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.404 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.458 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.586 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.586 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.586 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274415 s, 38.2 MB/s 2023-11-24T08:17:54.587 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.587 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.731 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.732 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282274 s, 37.1 MB/s 2023-11-24T08:17:54.733 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.733 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.784 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024986 s, 42.0 MB/s 2023-11-24T08:17:54.912 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:54.912 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:54.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:54.968 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.045 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.045 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231838 s, 45.2 MB/s 2023-11-24T08:17:55.046 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.046 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234677 s, 44.7 MB/s 2023-11-24T08:17:55.178 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.178 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200898 s, 52.2 MB/s 2023-11-24T08:17:55.314 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.314 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.371 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.446 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.446 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.446 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210717 s, 49.8 MB/s 2023-11-24T08:17:55.447 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.447 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304889 s, 34.4 MB/s 2023-11-24T08:17:55.594 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.594 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.653 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258942 s, 40.5 MB/s 2023-11-24T08:17:55.735 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.735 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.791 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:55.871 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:55.872 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:55.872 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255284 s, 41.1 MB/s 2023-11-24T08:17:55.873 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:55.873 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:55.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:55.928 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.005 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279563 s, 37.5 MB/s 2023-11-24T08:17:56.007 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.007 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.061 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.127 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.127 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178489 s, 58.7 MB/s 2023-11-24T08:17:56.129 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.129 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.183 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238041 s, 44.1 MB/s 2023-11-24T08:17:56.258 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.259 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.313 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.393 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.393 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250386 s, 41.9 MB/s 2023-11-24T08:17:56.394 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.394 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.448 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.571 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.571 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.571 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228854 s, 45.8 MB/s 2023-11-24T08:17:56.573 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.573 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.630 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147061 s, 71.3 MB/s 2023-11-24T08:17:56.698 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.698 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.753 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.827 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.827 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.827 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219608 s, 47.7 MB/s 2023-11-24T08:17:56.828 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.829 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:56.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:56.882 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176521 s, 59.4 MB/s 2023-11-24T08:17:56.954 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:56.954 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.009 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249173 s, 42.1 MB/s 2023-11-24T08:17:57.135 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.135 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229805 s, 45.6 MB/s 2023-11-24T08:17:57.273 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.273 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.333 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021198 s, 49.5 MB/s 2023-11-24T08:17:57.407 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.407 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.462 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200052 s, 52.4 MB/s 2023-11-24T08:17:57.539 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.539 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.594 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221653 s, 47.3 MB/s 2023-11-24T08:17:57.669 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.670 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.726 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.803 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.803 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.803 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226722 s, 46.2 MB/s 2023-11-24T08:17:57.805 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.805 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:57.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:57.864 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:57.942 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:57.942 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:57.942 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247106 s, 42.4 MB/s 2023-11-24T08:17:57.944 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:57.944 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.002 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209491 s, 50.1 MB/s 2023-11-24T08:17:58.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.078 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.136 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190007 s, 55.2 MB/s 2023-11-24T08:17:58.207 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.207 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219069 s, 47.9 MB/s 2023-11-24T08:17:58.387 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.387 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.447 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231729 s, 45.3 MB/s 2023-11-24T08:17:58.529 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.530 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.588 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023735 s, 44.2 MB/s 2023-11-24T08:17:58.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.672 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.725 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.796 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.796 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.796 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194176 s, 54.0 MB/s 2023-11-24T08:17:58.798 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.798 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:58.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:58.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:58.978 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:58.978 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:58.978 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262757 s, 39.9 MB/s 2023-11-24T08:17:58.980 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:58.980 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.036 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.113 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.114 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022531 s, 46.5 MB/s 2023-11-24T08:17:59.115 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.115 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.172 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.248 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.248 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.248 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225049 s, 46.6 MB/s 2023-11-24T08:17:59.249 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.250 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.305 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.378 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.378 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203762 s, 51.5 MB/s 2023-11-24T08:17:59.379 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.379 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221005 s, 47.4 MB/s 2023-11-24T08:17:59.517 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.517 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.572 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229346 s, 45.7 MB/s 2023-11-24T08:17:59.699 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.700 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.763 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.843 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.843 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023151 s, 45.3 MB/s 2023-11-24T08:17:59.844 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.844 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:17:59.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:17:59.898 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:17:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:17:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:17:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180329 s, 58.1 MB/s 2023-11-24T08:17:59.968 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:17:59.969 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.102 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222912 s, 47.0 MB/s 2023-11-24T08:18:00.104 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.104 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.158 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221242 s, 47.4 MB/s 2023-11-24T08:18:00.235 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.235 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.293 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.371 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.372 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.372 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223749 s, 46.9 MB/s 2023-11-24T08:18:00.373 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.373 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.430 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.504 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221608 s, 47.3 MB/s 2023-11-24T08:18:00.506 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.506 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.565 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.644 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.644 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.644 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252264 s, 41.6 MB/s 2023-11-24T08:18:00.645 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.646 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.703 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.775 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.775 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208814 s, 50.2 MB/s 2023-11-24T08:18:00.777 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.777 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.832 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:00.907 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:00.907 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:00.907 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207868 s, 50.4 MB/s 2023-11-24T08:18:00.909 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:00.909 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:00.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:00.961 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192543 s, 54.5 MB/s 2023-11-24T08:18:01.033 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.034 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.089 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195471 s, 53.6 MB/s 2023-11-24T08:18:01.159 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.160 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.331 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.331 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.331 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230459 s, 45.5 MB/s 2023-11-24T08:18:01.332 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.332 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.392 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025399 s, 41.3 MB/s 2023-11-24T08:18:01.475 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.475 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.534 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.607 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.607 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.608 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222006 s, 47.2 MB/s 2023-11-24T08:18:01.609 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.609 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.668 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.748 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.748 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.748 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278701 s, 37.6 MB/s 2023-11-24T08:18:01.750 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.750 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.801 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017119 s, 61.3 MB/s 2023-11-24T08:18:01.867 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:01.867 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:01.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:01.919 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:02.047 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:02.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:02.048 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272468 s, 38.5 MB/s 2023-11-24T08:18:02.049 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:02.050 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:02.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:02.108 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217208 s, 48.3 MB/s 2023-11-24T08:18:02.187 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:02.187 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:02.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:02.242 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:02.318 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227112 s, 46.2 MB/s 2023-11-24T08:18:02.319 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:02.319 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:02.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:02.369 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258952 s, 40.5 MB/s 2023-11-24T08:18:02.498 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:02.499 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:02.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:02.554 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:18:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:18:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:18:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215755 s, 48.6 MB/s 2023-11-24T08:18:02.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol2512 subvol_snap5437 2023-11-24T08:18:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.852+0000 7fb311333640 1 Processor -- start 2023-11-24T08:18:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.852+0000 7fb311333640 1 -- start start 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb311333640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb30c13c920 0x7fb30c13fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb311333640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb30c15dc80 0x7fb30c0b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb311333640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30c0cea30 con 0x7fb30c0ceec0 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb311333640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30c0b59b0 con 0x7fb30c13c920 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb311333640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30c0ce100 con 0x7fb30c15dc80 2023-11-24T08:18:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb30affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb30c13c920 0x7fb30c13fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb30affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb30c13c920 0x7fb30c13fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49600/0 (socket says 172.21.15.107:49600) 2023-11-24T08:18:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb30affd640 1 -- 172.21.15.107:0/3978431130 learned_addr learned my addr 172.21.15.107:0/3978431130 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.854+0000 7fb30b7fe640 1 --2- 172.21.15.107:0/3978431130 >> v2:172.21.15.120:3300/0 conn(0x7fb30c15dc80 0x7fb30c0b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.855+0000 7fb30affd640 1 -- 172.21.15.107:0/3978431130 >> v2:172.21.15.120:3300/0 conn(0x7fb30c15dc80 msgr2=0x7fb30c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:02.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.855+0000 7fb30affd640 1 --2- 172.21.15.107:0/3978431130 >> v2:172.21.15.120:3300/0 conn(0x7fb30c15dc80 0x7fb30c0b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.855+0000 7fb30affd640 1 -- 172.21.15.107:0/3978431130 >> v1:172.21.15.107:6789/0 conn(0x7fb30c0ceec0 legacy=0x7fb30c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:02.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.855+0000 7fb30affd640 1 -- 172.21.15.107:0/3978431130 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c0ce3e0 con 0x7fb30c13c920 2023-11-24T08:18:02.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.856+0000 7fb30affd640 1 --2- 172.21.15.107:0/3978431130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c13c920 0x7fb30c13fbc0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0b5c40 tx=0x7fb2fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f399d5eb22f2e2f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.856+0000 7fb309ffb640 1 -- 172.21.15.107:0/3978431130 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc030860 con 0x7fb30c13c920 2023-11-24T08:18:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.856+0000 7fb309ffb640 1 -- 172.21.15.107:0/3978431130 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc0309c0 con 0x7fb30c13c920 2023-11-24T08:18:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.856+0000 7fb309ffb640 1 -- 172.21.15.107:0/3978431130 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc030d50 con 0x7fb30c13c920 2023-11-24T08:18:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 -- 172.21.15.107:0/3978431130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c13c920 msgr2=0x7fb30c13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 --2- 172.21.15.107:0/3978431130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c13c920 0x7fb30c13fbc0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0b5c40 tx=0x7fb2fc02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 -- 172.21.15.107:0/3978431130 shutdown_connections 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 --2- 172.21.15.107:0/3978431130 >> v2:172.21.15.120:3300/0 conn(0x7fb30c15dc80 0x7fb30c0b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 --2- 172.21.15.107:0/3978431130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c13c920 0x7fb30c13fbc0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 -- 172.21.15.107:0/3978431130 >> 172.21.15.107:0/3978431130 conn(0x7fb30c071d40 msgr2=0x7fb30c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 -- 172.21.15.107:0/3978431130 shutdown_connections 2023-11-24T08:18:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.857+0000 7fb311333640 1 -- 172.21.15.107:0/3978431130 wait complete. 2023-11-24T08:18:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.859+0000 7fb311333640 1 Processor -- start 2023-11-24T08:18:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.859+0000 7fb311333640 1 -- start start 2023-11-24T08:18:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 0x7fb30c139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb30c13c920 0x7fb30c1353b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c138a40 con 0x7fb30c15dc80 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c153b40 con 0x7fb30c0ceec0 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb311333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c139970 con 0x7fb30c13c920 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb30b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb30b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49616/0 (socket says 172.21.15.107:49616) 2023-11-24T08:18:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.860+0000 7fb30b7fe640 1 -- 172.21.15.107:0/3703797191 learned_addr learned my addr 172.21.15.107:0/3703797191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30affd640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 0x7fb30c139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 -- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb30c13c920 msgr2=0x7fb30c1353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30a7fc640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb30c13c920 0x7fb30c1353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb30c13c920 0x7fb30c1353b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 -- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 msgr2=0x7fb30c139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 0x7fb30c139430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 -- 172.21.15.107:0/3703797191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0094d0 con 0x7fb30c15dc80 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30affd640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 0x7fb30c139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.861+0000 7fb30b7fe640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fb304007e70 tx=0x7fb30400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.862+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304014620 con 0x7fb30c15dc80 2023-11-24T08:18:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.862+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3040147a0 con 0x7fb30c15dc80 2023-11-24T08:18:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.862+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30c137dc0 con 0x7fb30c15dc80 2023-11-24T08:18:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.862+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304014b50 con 0x7fb30c15dc80 2023-11-24T08:18:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.862+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30c138380 con 0x7fb30c15dc80 2023-11-24T08:18:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.863+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c138b80 con 0x7fb30c15dc80 2023-11-24T08:18:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.864+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 33) v1 ==== 91424+0+0 (secure 0 0 0) 0x7fb304014cb0 con 0x7fb30c15dc80 2023-11-24T08:18:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.865+0000 7fb2f37fe640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 0x7fb2f806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.865+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb304088d20 con 0x7fb30c15dc80 2023-11-24T08:18:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.865+0000 7fb30affd640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 0x7fb2f806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.866+0000 7fb30affd640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 0x7fb2f806eb30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc036000 tx=0x7fb2fc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:02.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:02.870+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb30c138b80 con 0x7fb30c15dc80 2023-11-24T08:18:03.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.091+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol2512", "snap_name": "subvol_snap5437", "target": ["mon-mgr", ""]}) v1 -- 0x7fb30c138b80 con 0x7fb2f806c670 2023-11-24T08:18:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.142+0000 7fb2f37fe640 1 -- 172.21.15.107:0/3703797191 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb30c138b80 con 0x7fb2f806c670 2023-11-24T08:18:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.144+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 msgr2=0x7fb2f806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.144+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 0x7fb2f806eb30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc036000 tx=0x7fb2fc03b040 comp rx=0 tx=0).stop 2023-11-24T08:18:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.144+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 msgr2=0x7fb30c152310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.144+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fb304007e70 tx=0x7fb30400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 shutdown_connections 2023-11-24T08:18:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2f806c670 0x7fb2f806eb30 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb30c15dc80 0x7fb30c152310 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb30c13c920 0x7fb30c1353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 --2- 172.21.15.107:0/3703797191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb30c0ceec0 0x7fb30c139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 >> 172.21.15.107:0/3703797191 conn(0x7fb30c071d40 msgr2=0x7fb30c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 shutdown_connections 2023-11-24T08:18:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.145+0000 7fb311333640 1 -- 172.21.15.107:0/3703797191 wait complete. 2023-11-24T08:18:03.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 2 2023-11-24T08:18:03.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.355+0000 7fb610a39640 1 Processor -- start 2023-11-24T08:18:03.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.356+0000 7fb610a39640 1 -- start start 2023-11-24T08:18:03.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb60c15b2f0 0x7fb60c0cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 0x7fb60c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 0x7fb60c099f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c06e4d0 con 0x7fb60c15b2f0 2023-11-24T08:18:03.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c15c7e0 con 0x7fb60c15b6d0 2023-11-24T08:18:03.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb610a39640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0cf370 con 0x7fb60c0b3740 2023-11-24T08:18:03.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb60b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb60c15b2f0 0x7fb60c0cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb60b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb60c15b2f0 0x7fb60c0cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49626/0 (socket says 172.21.15.107:49626) 2023-11-24T08:18:03.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.357+0000 7fb60b7fe640 1 -- 172.21.15.107:0/3541245163 learned_addr learned my addr 172.21.15.107:0/3541245163 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:03.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60bfff640 1 --2- 172.21.15.107:0/3541245163 >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 0x7fb60c099f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60affd640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 0x7fb60c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60b7fe640 1 -- 172.21.15.107:0/3541245163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 msgr2=0x7fb60c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 0x7fb60c0999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60b7fe640 1 -- 172.21.15.107:0/3541245163 >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 msgr2=0x7fb60c099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/3541245163 >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 0x7fb60c099f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60b7fe640 1 -- 172.21.15.107:0/3541245163 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c15bbd0 con 0x7fb60c15b2f0 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60affd640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 0x7fb60c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:03.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.358+0000 7fb60bfff640 1 --2- 172.21.15.107:0/3541245163 >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 0x7fb60c099f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:03.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.359+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c15b2f0 0x7fb60c0cee30 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0d1000 tx=0x7fb5fc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1897b9033f88a064 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.359+0000 7fb60a7fc640 1 -- 172.21.15.107:0/3541245163 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc030850 con 0x7fb60c15b2f0 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.359+0000 7fb60a7fc640 1 -- 172.21.15.107:0/3541245163 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc0309b0 con 0x7fb60c15b2f0 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.359+0000 7fb60a7fc640 1 -- 172.21.15.107:0/3541245163 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc030d40 con 0x7fb60c15b2f0 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 -- 172.21.15.107:0/3541245163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c15b2f0 msgr2=0x7fb60c0cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c15b2f0 0x7fb60c0cee30 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0d1000 tx=0x7fb5fc02b990 comp rx=0 tx=0).stop 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 -- 172.21.15.107:0/3541245163 shutdown_connections 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 --2- 172.21.15.107:0/3541245163 >> v2:172.21.15.120:3300/0 conn(0x7fb60c15b6d0 0x7fb60c099f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c0b3740 0x7fb60c0999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 --2- 172.21.15.107:0/3541245163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c15b2f0 0x7fb60c0cee30 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 -- 172.21.15.107:0/3541245163 >> 172.21.15.107:0/3541245163 conn(0x7fb60c0b2b40 msgr2=0x7fb60c0b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 -- 172.21.15.107:0/3541245163 shutdown_connections 2023-11-24T08:18:03.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.360+0000 7fb610a39640 1 -- 172.21.15.107:0/3541245163 wait complete. 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.362+0000 7fb610a39640 1 Processor -- start 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.362+0000 7fb610a39640 1 -- start start 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c15b2f0 0x7fb60c0b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb60c15b6d0 0x7fb60c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1f7890 con 0x7fb60c0b3740 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1f69c0 con 0x7fb60c15b6d0 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb610a39640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1f6c10 con 0x7fb60c15b2f0 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49640/0 (socket says 172.21.15.107:49640) 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 -- 172.21.15.107:0/2299913553 learned_addr learned my addr 172.21.15.107:0/2299913553 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 -- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c15b2f0 msgr2=0x7fb60c0b65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c15b2f0 0x7fb60c0b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 -- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb60c15b6d0 msgr2=0x7fb60c0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb60c15b6d0 0x7fb60c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.363+0000 7fb60b7fe640 1 -- 172.21.15.107:0/2299913553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0094d0 con 0x7fb60c0b3740 2023-11-24T08:18:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb60b7fe640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc02b9d0 tx=0x7fb5fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc030840 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc043400 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c1f6ef0 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0437b0 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.364+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c1f74b0 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.365+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c04e840 con 0x7fb60c0b3740 2023-11-24T08:18:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.366+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 33) v1 ==== 91424+0+0 (secure 0 0 0) 0x7fb5fc02e070 con 0x7fb60c0b3740 2023-11-24T08:18:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.367+0000 7fb603fff640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 0x7fb5ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.367+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5fc03e070 con 0x7fb60c0b3740 2023-11-24T08:18:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.367+0000 7fb60affd640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 0x7fb5ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.368+0000 7fb60affd640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 0x7fb5ec06eb30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb60c1f4540 tx=0x7fb5f40083d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.371+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5fc0780d0 con 0x7fb60c0b3740 2023-11-24T08:18:03.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.560+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fb60c1f7640 con 0x7fb60c0b3740 2023-11-24T08:18:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.567+0000 7fb603fff640 1 -- 172.21.15.107:0/2299913553 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v44)=0 v44) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb5fc07b6f0 con 0x7fb60c0b3740 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.570+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 msgr2=0x7fb5ec06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.570+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 0x7fb5ec06eb30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb60c1f4540 tx=0x7fb5f40083d0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 msgr2=0x7fb60c1574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc02b9d0 tx=0x7fb5fc004730 comp rx=0 tx=0).stop 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 shutdown_connections 2023-11-24T08:18:03.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb5ec06c670 0x7fb5ec06eb30 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb60c15b6d0 0x7fb60c0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb60c15b2f0 0x7fb60c0b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 --2- 172.21.15.107:0/2299913553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c0b3740 0x7fb60c1574c0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 >> 172.21.15.107:0/2299913553 conn(0x7fb60c0b2b40 msgr2=0x7fb60c0b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 shutdown_connections 2023-11-24T08:18:03.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.571+0000 7fb610a39640 1 -- 172.21.15.107:0/2299913553 wait complete. 2023-11-24T08:18:03.590 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:03.589+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:03.591 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:03.590+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:03.591 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:03.590+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:03.591 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:03.590+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:03.591 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:03.590+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:03.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol2512 subvol_snap5437 subvol_clone1612 2023-11-24T08:18:03.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.789+0000 7fde7390f640 1 Processor -- start 2023-11-24T08:18:03.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.789+0000 7fde7390f640 1 -- start start 2023-11-24T08:18:03.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde7390f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fde6c15b440 0x7fde6c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde7390f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde6c0cbf20 0x7fde6c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde7390f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c15eea0 con 0x7fde6c0c57e0 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde7390f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0cc3d0 con 0x7fde6c0cbf20 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde7390f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c15b820 con 0x7fde6c15b440 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde70eaa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde6c0cbf20 0x7fde6c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde70eaa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde6c0cbf20 0x7fde6c0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49668/0 (socket says 172.21.15.107:49668) 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde70eaa640 1 -- 172.21.15.107:0/1016718911 learned_addr learned my addr 172.21.15.107:0/1016718911 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.791+0000 7fde716ab640 1 --2- 172.21.15.107:0/1016718911 >> v2:172.21.15.120:3300/0 conn(0x7fde6c15b440 0x7fde6c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.792+0000 7fde70eaa640 1 -- 172.21.15.107:0/1016718911 >> v2:172.21.15.120:3300/0 conn(0x7fde6c15b440 msgr2=0x7fde6c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.792+0000 7fde70eaa640 1 --2- 172.21.15.107:0/1016718911 >> v2:172.21.15.120:3300/0 conn(0x7fde6c15b440 0x7fde6c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.792+0000 7fde70eaa640 1 -- 172.21.15.107:0/1016718911 >> v1:172.21.15.120:6789/0 conn(0x7fde6c0c57e0 legacy=0x7fde6c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.792+0000 7fde70eaa640 1 -- 172.21.15.107:0/1016718911 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c06d600 con 0x7fde6c0cbf20 2023-11-24T08:18:03.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde70eaa640 1 --2- 172.21.15.107:0/1016718911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c52a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06e770 tx=0x7fde5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54555e3a8fe68983 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde63fff640 1 -- 172.21.15.107:0/1016718911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0317e0 con 0x7fde6c0cbf20 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde63fff640 1 -- 172.21.15.107:0/1016718911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c031940 con 0x7fde6c0cbf20 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde63fff640 1 -- 172.21.15.107:0/1016718911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031cd0 con 0x7fde6c0cbf20 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde7390f640 1 -- 172.21.15.107:0/1016718911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c0cbf20 msgr2=0x7fde6c0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.793+0000 7fde7390f640 1 --2- 172.21.15.107:0/1016718911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c52a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06e770 tx=0x7fde5c02b880 comp rx=0 tx=0).stop 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 -- 172.21.15.107:0/1016718911 shutdown_connections 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 --2- 172.21.15.107:0/1016718911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c52a0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 --2- 172.21.15.107:0/1016718911 >> v2:172.21.15.120:3300/0 conn(0x7fde6c15b440 0x7fde6c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 -- 172.21.15.107:0/1016718911 >> 172.21.15.107:0/1016718911 conn(0x7fde6c0b0c50 msgr2=0x7fde6c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 -- 172.21.15.107:0/1016718911 shutdown_connections 2023-11-24T08:18:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.794+0000 7fde7390f640 1 -- 172.21.15.107:0/1016718911 wait complete. 2023-11-24T08:18:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.795+0000 7fde7390f640 1 Processor -- start 2023-11-24T08:18:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 -- start start 2023-11-24T08:18:03.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde6c0c57e0 0x7fde6c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b29e0 con 0x7fde6c15b440 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b2cc0 con 0x7fde6c0cbf20 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.796+0000 7fde7390f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b2fa0 con 0x7fde6c0c57e0 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49678/0 (socket says 172.21.15.107:49678) 2023-11-24T08:18:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 -- 172.21.15.107:0/1657610624 learned_addr learned my addr 172.21.15.107:0/1657610624 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde716ab640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde6c0c57e0 0x7fde6c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde70eaa640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 -- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde6c0c57e0 msgr2=0x7fde6c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde6c0c57e0 0x7fde6c15e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 -- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde6c0cbf20 msgr2=0x7fde6c0c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.797+0000 7fde71eac640 1 -- 172.21.15.107:0/1657610624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0094d0 con 0x7fde6c15b440 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde71eac640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c780 tx=0x7fde6400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64003d80 con 0x7fde6c15b440 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6400ce90 con 0x7fde6c15b440 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0b3280 con 0x7fde6c15b440 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640052b0 con 0x7fde6c15b440 2023-11-24T08:18:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.798+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c15eb40 con 0x7fde6c15b440 2023-11-24T08:18:03.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.799+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c0ae550 con 0x7fde6c15b440 2023-11-24T08:18:03.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.800+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 33) v1 ==== 91424+0+0 (secure 0 0 0) 0x7fde64005410 con 0x7fde6c15b440 2023-11-24T08:18:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.801+0000 7fde61ffb640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 0x7fde5006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.801+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde640878c0 con 0x7fde6c15b440 2023-11-24T08:18:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.801+0000 7fde716ab640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 0x7fde5006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.802+0000 7fde716ab640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 0x7fde5006eb30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fde68006fd0 tx=0x7fde68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:03.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:03.805+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde640543d0 con 0x7fde6c15b440 2023-11-24T08:18:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.029+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol2512", "snap_name": "subvol_snap5437", "target_sub_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7fde6c0b2e30 con 0x7fde5006c670 2023-11-24T08:18:04.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.144+0000 7fde61ffb640 1 -- 172.21.15.107:0/1657610624 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde6c0b2e30 con 0x7fde5006c670 2023-11-24T08:18:04.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 msgr2=0x7fde5006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 0x7fde5006eb30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fde68006fd0 tx=0x7fde68008040 comp rx=0 tx=0).stop 2023-11-24T08:18:04.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 msgr2=0x7fde6c0c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c780 tx=0x7fde6400cc40 comp rx=0 tx=0).stop 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 shutdown_connections 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde5006c670 0x7fde5006eb30 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde6c15b440 0x7fde6c0c8150 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde6c0cbf20 0x7fde6c0c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 --2- 172.21.15.107:0/1657610624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde6c0c57e0 0x7fde6c15e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 >> 172.21.15.107:0/1657610624 conn(0x7fde6c0b0c50 msgr2=0x7fde6c070780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.147+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 shutdown_connections 2023-11-24T08:18:04.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.148+0000 7fde7390f640 1 -- 172.21.15.107:0/1657610624 wait complete. 2023-11-24T08:18:04.155 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.154+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.155 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.154+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.155 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.154+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.156 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.154+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.156 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.154+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone1612 2023-11-24T08:18:04.180 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.179+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.180 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.179+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.180 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.179+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.181 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.179+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.181 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:04.179+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.361+0000 7f76a42ad640 1 Processor -- start 2023-11-24T08:18:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.362+0000 7f76a42ad640 1 -- start start 2023-11-24T08:18:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.363+0000 7f76a42ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 0x7f769c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.363+0000 7f76a42ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c166510 con 0x7f769c0ccbd0 2023-11-24T08:18:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.363+0000 7f76a42ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c1655f0 con 0x7f769c162540 2023-11-24T08:18:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.363+0000 7f76a42ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0cc380 con 0x7f769c1660b0 2023-11-24T08:18:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.364+0000 7f76a2049640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f769c0ccbd0 0x7f769c0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45992/0 (socket says 172.21.15.107:45992) 2023-11-24T08:18:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.364+0000 7f76a2049640 1 -- 172.21.15.107:0/2696648980 learned_addr learned my addr 172.21.15.107:0/2696648980 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.364+0000 7f76a1848640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 0x7f769c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.364+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000545226 0 0) 0x7f7698001110 con 0x7f769c0ccbd0 2023-11-24T08:18:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.364+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7688003520 con 0x7f769c0ccbd0 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157649917 0 0) 0x7f7688003520 con 0x7f769c0ccbd0 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 msgr2=0x7f769c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 0x7f769c162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 >> v1:172.21.15.120:6789/0 conn(0x7f769c162540 legacy=0x7f769c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c162b30 con 0x7f769c0ccbd0 2023-11-24T08:18:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7698001800 con 0x7f769c0ccbd0 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 >> v1:172.21.15.107:6789/0 conn(0x7f769c0ccbd0 legacy=0x7f769c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 0x7f76880039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7688003f40 0x7f76880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688004d00 0x7f7688005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0cc380 con 0x7f7688003f40 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768800ba90 con 0x7f7688004d00 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768800bc50 con 0x7f769c1660b0 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7688003f40 0x7f76880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a284a640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688004d00 0x7f7688005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 -- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 msgr2=0x7f76880039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f769c1660b0 0x7f76880039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 -- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688004d00 msgr2=0x7f7688005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688004d00 0x7f7688005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.365+0000 7f76a1848640 1 -- 172.21.15.107:0/2696648980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7688003520 con 0x7f7688003f40 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a1848640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7688003f40 0x7f76880047c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f768c009a90 tx=0x7f76880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5984e91263b13945 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c02c6d0 con 0x7f7688003f40 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c02c830 con 0x7f7688003f40 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a1047640 1 -- 172.21.15.107:0/2696648980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c02cbc0 con 0x7f7688003f40 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a42ad640 1 -- 172.21.15.107:0/2696648980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7688003f40 msgr2=0x7f76880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a42ad640 1 --2- 172.21.15.107:0/2696648980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7688003f40 0x7f76880047c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f768c009a90 tx=0x7f76880057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a284a640 1 -- 172.21.15.107:0/2696648980 reap_dead start 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a42ad640 1 -- 172.21.15.107:0/2696648980 shutdown_connections 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.367+0000 7f76a42ad640 1 -- 172.21.15.107:0/2696648980 >> 172.21.15.107:0/2696648980 conn(0x7f769c0b0c50 msgr2=0x7f769c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.368+0000 7f76a42ad640 1 -- 172.21.15.107:0/2696648980 shutdown_connections 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.368+0000 7f76a42ad640 1 -- 172.21.15.107:0/2696648980 wait complete. 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.369+0000 7f76a42ad640 1 Processor -- start 2023-11-24T08:18:04.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.369+0000 7f76a42ad640 1 -- start start 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7688003f40 0x7f769c1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769c1640a0 0x7f769c164520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c072270 con 0x7f769c1580a0 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c164a60 con 0x7f769c1640a0 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.370+0000 7f76a42ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c072550 con 0x7f7688003f40 2023-11-24T08:18:04.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49756/0 (socket says 172.21.15.107:49756) 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 -- 172.21.15.107:0/767481508 learned_addr learned my addr 172.21.15.107:0/767481508 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a2049640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7688003f40 0x7f769c1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a284a640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769c1640a0 0x7f769c164520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 -- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7688003f40 msgr2=0x7f769c1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7688003f40 0x7f769c1585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 -- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769c1640a0 msgr2=0x7f769c164520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769c1640a0 0x7f769c164520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.371+0000 7f76a1848640 1 -- 172.21.15.107:0/767481508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c157ec0 con 0x7f769c1580a0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f76a1848640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f768c0099e0 tx=0x7f768c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c003810 con 0x7f769c1580a0 2023-11-24T08:18:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c003970 con 0x7f769c1580a0 2023-11-24T08:18:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c003d20 con 0x7f769c1580a0 2023-11-24T08:18:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0704c0 con 0x7f769c1580a0 2023-11-24T08:18:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.372+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c070780 con 0x7f769c1580a0 2023-11-24T08:18:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.373+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c04e8b0 con 0x7f769c1580a0 2023-11-24T08:18:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.374+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 33) v1 ==== 91424+0+0 (secure 0 0 0) 0x7f768c034440 con 0x7f769c1580a0 2023-11-24T08:18:04.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.375+0000 7f7692ffd640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 0x7f768006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.376+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f768c0aa4a0 con 0x7f769c1580a0 2023-11-24T08:18:04.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.376+0000 7f76a2049640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 0x7f768006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.376+0000 7f76a2049640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 0x7f768006eb30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7698001370 tx=0x7f7698001c70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.380+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f768c077220 con 0x7f769c1580a0 2023-11-24T08:18:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.580+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 34) v1 ==== 91475+0+0 (secure 0 0 0) 0x7f768c0775f0 con 0x7f769c1580a0 2023-11-24T08:18:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.613+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f769c165b90 con 0x7f768006c670 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.622+0000 7f7692ffd640 1 -- 172.21.15.107:0/767481508 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f769800e8d0 con 0x7f768006c670 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:18:04.623 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol2512", 2023-11-24T08:18:04.624 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5437" 2023-11-24T08:18:04.624 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:04.624 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:04.624 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 msgr2=0x7f768006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 0x7f768006eb30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7698001370 tx=0x7f7698001c70 comp rx=0 tx=0).stop 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 msgr2=0x7f769c157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f768c0099e0 tx=0x7f768c004730 comp rx=0 tx=0).stop 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 shutdown_connections 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f768006c670 0x7f768006eb30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769c1640a0 0x7f769c164520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f769c1580a0 0x7f769c157980 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 --2- 172.21.15.107:0/767481508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7688003f40 0x7f769c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 >> 172.21.15.107:0/767481508 conn(0x7f769c0b0c50 msgr2=0x7f769c0ca970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 shutdown_connections 2023-11-24T08:18:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.625+0000 7f76a42ad640 1 -- 172.21.15.107:0/767481508 wait complete. 2023-11-24T08:18:04.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone1612 2023-11-24T08:18:04.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.901+0000 7f56c255a640 1 Processor -- start 2023-11-24T08:18:04.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.901+0000 7f56c255a640 1 -- start start 2023-11-24T08:18:04.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.903+0000 7f56c255a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56bc0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.903+0000 7f56c255a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc15e400 con 0x7f56bc15d7f0 2023-11-24T08:18:04.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.903+0000 7f56c255a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc0c46e0 con 0x7f56bc0c4220 2023-11-24T08:18:04.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.903+0000 7f56c255a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0ce880 con 0x7f56bc06d600 2023-11-24T08:18:04.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f56bc15d7f0 0x7f56bc0c5990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46006/0 (socket says 172.21.15.107:46006) 2023-11-24T08:18:04.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bbfff640 1 -- 172.21.15.107:0/3588604663 learned_addr learned my addr 172.21.15.107:0/3588604663 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166526698 0 0) 0x7f56bc15e400 con 0x7f56bc15d7f0 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56c0af7640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56bc0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4003520 con 0x7f56bc15d7f0 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288596389 0 0) 0x7f56a4003520 con 0x7f56bc15d7f0 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 msgr2=0x7f56bc0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56bc0cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 >> v1:172.21.15.120:6789/0 conn(0x7f56bc0c4220 legacy=0x7f56bc0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc0c49c0 con 0x7f56bc15d7f0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f56ac001a10 con 0x7f56bc15d7f0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.904+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 >> v1:172.21.15.107:6789/0 conn(0x7f56bc15d7f0 legacy=0x7f56bc0c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a4003fd0 0x7f56a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56a4004d00 0x7f56a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a400bb10 con 0x7f56a4004d00 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a400bc80 con 0x7f56a4003fd0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a400be40 con 0x7f56bc06d600 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56a4004d00 0x7f56a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bbfff640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56abfff640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a4003fd0 0x7f56a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 -- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 msgr2=0x7f56a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 -- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a4003fd0 msgr2=0x7f56a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:04.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a4003fd0 0x7f56a40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56c0af7640 1 -- 172.21.15.107:0/3588604663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4003520 con 0x7f56a4004d00 2023-11-24T08:18:04.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.905+0000 7f56bbfff640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56bc06d600 0x7f56a40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:04.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.906+0000 7f56c0af7640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56a4004d00 0x7f56a4005180 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f56b400ba30 tx=0x7f56a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780b8fa41dda34c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4032640 con 0x7f56a4004d00 2023-11-24T08:18:04.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b40327a0 con 0x7f56a4004d00 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56bb7fe640 1 -- 172.21.15.107:0/3588604663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4032b30 con 0x7f56a4004d00 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 -- 172.21.15.107:0/3588604663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56a4004d00 msgr2=0x7f56a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 --2- 172.21.15.107:0/3588604663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56a4004d00 0x7f56a4005180 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f56b400ba30 tx=0x7f56a40057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c0af7640 1 -- 172.21.15.107:0/3588604663 reap_dead start 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 -- 172.21.15.107:0/3588604663 shutdown_connections 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 -- 172.21.15.107:0/3588604663 >> 172.21.15.107:0/3588604663 conn(0x7f56bc0b0c50 msgr2=0x7f56bc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 -- 172.21.15.107:0/3588604663 shutdown_connections 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.907+0000 7f56c255a640 1 -- 172.21.15.107:0/3588604663 wait complete. 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.909+0000 7f56c255a640 1 Processor -- start 2023-11-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.909+0000 7f56c255a640 1 -- start start 2023-11-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56a4004d00 0x7f56bc0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 0x7f56bc0b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b3ab0 con 0x7f56bc0c9960 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0c9010 con 0x7f56bc0c6880 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.910+0000 7f56c255a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0c6d30 con 0x7f56a4004d00 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49778/0 (socket says 172.21.15.107:49778) 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 -- 172.21.15.107:0/4180426363 learned_addr learned my addr 172.21.15.107:0/4180426363 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56c0af7640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 0x7f56bc0b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56bbfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56a4004d00 0x7f56bc0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 -- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56a4004d00 msgr2=0x7f56bc0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56a4004d00 0x7f56bc0b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 -- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 msgr2=0x7f56bc0b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 0x7f56bc0b3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56abfff640 1 -- 172.21.15.107:0/4180426363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc158730 con 0x7f56bc0c9960 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.911+0000 7f56c0af7640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 0x7f56bc0b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56abfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d570 tx=0x7f56b000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0003610 con 0x7f56bc0c9960 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0003770 con 0x7f56bc0c9960 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc0c6fd0 con 0x7f56bc0c9960 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0003ae0 con 0x7f56bc0c9960 2023-11-24T08:18:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.912+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc0c9de0 con 0x7f56bc0c9960 2023-11-24T08:18:04.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.913+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc04e8b0 con 0x7f56bc0c9960 2023-11-24T08:18:04.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.914+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 34) v1 ==== 91475+0+0 (secure 0 0 0) 0x7f56b0003c40 con 0x7f56bc0c9960 2023-11-24T08:18:04.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.915+0000 7f56b97fa640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 0x7f569c06ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:04.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.915+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56b0088320 con 0x7f56bc0c9960 2023-11-24T08:18:04.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.915+0000 7f56bbfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 0x7f569c06ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:04.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.916+0000 7f56bbfff640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 0x7f569c06ebf0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f56ac002230 tx=0x7f56ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:04.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:04.919+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56b0054f90 con 0x7f56bc0c9960 2023-11-24T08:18:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.148+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f56bc0c5260 con 0x7f569c06c730 2023-11-24T08:18:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.159+0000 7f56b97fa640 1 -- 172.21.15.107:0/4180426363 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f56ac00d040 con 0x7f569c06c730 2023-11-24T08:18:05.160 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol2512", 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5437" 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:05.161 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 msgr2=0x7f569c06ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 0x7f569c06ebf0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f56ac002230 tx=0x7f56ac009040 comp rx=0 tx=0).stop 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 msgr2=0x7f56bc0c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d570 tx=0x7f56b000da30 comp rx=0 tx=0).stop 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 shutdown_connections 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f569c06c730 0x7f569c06ebf0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56bc0c6880 0x7f56bc0b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56bc0c9960 0x7f56bc0c6340 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 --2- 172.21.15.107:0/4180426363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56a4004d00 0x7f56bc0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 >> 172.21.15.107:0/4180426363 conn(0x7f56bc0b0c50 msgr2=0x7f56bc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 shutdown_connections 2023-11-24T08:18:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:05.162+0000 7f56c255a640 1 -- 172.21.15.107:0/4180426363 wait complete. 2023-11-24T08:18:06.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone1612 2023-11-24T08:18:06.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.415+0000 7f49a95c1640 1 Processor -- start 2023-11-24T08:18:06.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.416+0000 7f49a95c1640 1 -- start start 2023-11-24T08:18:06.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a95c1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f49a415b260 0x7f49a406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a95c1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 0x7f49a40b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a95c1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a406d5c0 con 0x7f49a40b1080 2023-11-24T08:18:06.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a95c1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a415b640 con 0x7f49a415b260 2023-11-24T08:18:06.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a95c1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a40b14d0 con 0x7f49a406e580 2023-11-24T08:18:06.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f49a415b260 0x7f49a406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:06.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f49a415b260 0x7f49a406e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49784/0 (socket says 172.21.15.107:49784) 2023-11-24T08:18:06.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.417+0000 7f49a3fff640 1 -- 172.21.15.107:0/3768653813 learned_addr learned my addr 172.21.15.107:0/3768653813 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a3fff640 1 -- 172.21.15.107:0/3768653813 >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 msgr2=0x7f49a40b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a37fe640 1 --2- 172.21.15.107:0/3768653813 >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 0x7f49a40b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a3fff640 1 --2- 172.21.15.107:0/3768653813 >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 0x7f49a40b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a3fff640 1 -- 172.21.15.107:0/3768653813 >> v1:172.21.15.120:6789/0 conn(0x7f49a40b1080 legacy=0x7f49a40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a3fff640 1 -- 172.21.15.107:0/3768653813 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a415ed20 con 0x7f49a415b260 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.418+0000 7f49a37fe640 1 --2- 172.21.15.107:0/3768653813 >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 0x7f49a40b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:06.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.419+0000 7f49a3fff640 1 --2- 172.21.15.107:0/3768653813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a406e040 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f49a40ce370 tx=0x7f499402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9acd6307d9b69e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:06.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.419+0000 7f49a2ffd640 1 -- 172.21.15.107:0/3768653813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499402c960 con 0x7f49a415b260 2023-11-24T08:18:06.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.419+0000 7f49a2ffd640 1 -- 172.21.15.107:0/3768653813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499402cac0 con 0x7f49a415b260 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.419+0000 7f49a2ffd640 1 -- 172.21.15.107:0/3768653813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499402ce50 con 0x7f49a415b260 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 -- 172.21.15.107:0/3768653813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 msgr2=0x7f49a406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 --2- 172.21.15.107:0/3768653813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a406e040 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f49a40ce370 tx=0x7f499402bdc0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 -- 172.21.15.107:0/3768653813 shutdown_connections 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 --2- 172.21.15.107:0/3768653813 >> v2:172.21.15.120:3300/0 conn(0x7f49a406e580 0x7f49a40b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 --2- 172.21.15.107:0/3768653813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a406e040 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 -- 172.21.15.107:0/3768653813 >> 172.21.15.107:0/3768653813 conn(0x7f49a40cbb40 msgr2=0x7f49a40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 -- 172.21.15.107:0/3768653813 shutdown_connections 2023-11-24T08:18:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.420+0000 7f49a95c1640 1 -- 172.21.15.107:0/3768653813 wait complete. 2023-11-24T08:18:06.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.422+0000 7f49a95c1640 1 Processor -- start 2023-11-24T08:18:06.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.422+0000 7f49a95c1640 1 -- start start 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49a406e580 0x7f49a41f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49a40b1080 0x7f49a41f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40b1a70 con 0x7f49a415b260 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40b1d50 con 0x7f49a40b1080 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.423+0000 7f49a95c1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40b2030 con 0x7f49a406e580 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49792/0 (socket says 172.21.15.107:49792) 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 -- 172.21.15.107:0/4263779658 learned_addr learned my addr 172.21.15.107:0/4263779658 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a37fe640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49a40b1080 0x7f49a41f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 -- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49a406e580 msgr2=0x7f49a41f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49a406e580 0x7f49a41f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 -- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49a40b1080 msgr2=0x7f49a41f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49a40b1080 0x7f49a41f32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 -- 172.21.15.107:0/4263779658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994009b30 con 0x7f49a415b260 2023-11-24T08:18:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.424+0000 7f49a8dc0640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f499c00b490 tx=0x7f499c00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.425+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c011830 con 0x7f49a415b260 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.425+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c0119b0 con 0x7f49a415b260 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.425+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a40b2310 con 0x7f49a415b260 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.425+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c011d60 con 0x7f49a415b260 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.425+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a40b4ac0 con 0x7f49a415b260 2023-11-24T08:18:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.426+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40b2830 con 0x7f49a415b260 2023-11-24T08:18:06.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.427+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 34) v1 ==== 91475+0+0 (secure 0 0 0) 0x7f499c01a400 con 0x7f49a415b260 2023-11-24T08:18:06.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.428+0000 7f49a0ff9640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 0x7f498406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:06.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.428+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f499c088480 con 0x7f49a415b260 2023-11-24T08:18:06.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.428+0000 7f49a3fff640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 0x7f498406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:06.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.429+0000 7f49a3fff640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 0x7f498406ebc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4994035000 tx=0x7f499402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:06.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.433+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49a40b2830 con 0x7f49a415b260 2023-11-24T08:18:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.618+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f499c055400 con 0x7f49a415b260 2023-11-24T08:18:06.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:06.668+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f49a415ef70 con 0x7f498406c700 2023-11-24T08:18:10.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.864+0000 7f49a0ff9640 1 -- 172.21.15.107:0/4263779658 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f4994009510 con 0x7f498406c700 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol2512", 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5437" 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:10.868 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 msgr2=0x7f498406ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 0x7f498406ebc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4994035000 tx=0x7f499402e000 comp rx=0 tx=0).stop 2023-11-24T08:18:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 msgr2=0x7f49a40b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f499c00b490 tx=0x7f499c00b950 comp rx=0 tx=0).stop 2023-11-24T08:18:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 shutdown_connections 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f498406c700 0x7f498406ebc0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a415b260 0x7f49a40b4580 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49a40b1080 0x7f49a41f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 --2- 172.21.15.107:0/4263779658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49a406e580 0x7f49a41f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.868+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 >> 172.21.15.107:0/4263779658 conn(0x7f49a40cbb40 msgr2=0x7f49a40af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.869+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 shutdown_connections 2023-11-24T08:18:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:10.869+0000 7f49a95c1640 1 -- 172.21.15.107:0/4263779658 wait complete. 2023-11-24T08:18:11.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone1612 2023-11-24T08:18:12.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.131+0000 7f247651d640 1 Processor -- start 2023-11-24T08:18:12.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.132+0000 7f247651d640 1 -- start start 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f247651d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f247006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f247651d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24700b1ed0 0x7f247006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f247651d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247015db00 con 0x7f247015d720 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f247651d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24700d0ce0 con 0x7f24700b1ed0 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f247651d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247006d400 con 0x7f24700d3a40 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f246ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f247015d720 0x7f24700d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37384/0 (socket says 172.21.15.107:37384) 2023-11-24T08:18:12.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.133+0000 7f246ffff640 1 -- 172.21.15.107:0/4160784318 learned_addr learned my addr 172.21.15.107:0/4160784318 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f247006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f2474aba640 1 --2- 172.21.15.107:0/4160784318 >> v2:172.21.15.120:3300/0 conn(0x7f24700b1ed0 0x7f247006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556168903 0 0) 0x7f247015db00 con 0x7f247015d720 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2458003520 con 0x7f247015d720 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275510543 0 0) 0x7f2458003520 con 0x7f247015d720 2023-11-24T08:18:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 msgr2=0x7f247006ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f247006ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 >> v2:172.21.15.120:3300/0 conn(0x7f24700b1ed0 msgr2=0x7f247006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 --2- 172.21.15.107:0/4160784318 >> v2:172.21.15.120:3300/0 conn(0x7f24700b1ed0 0x7f247006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700b0100 con 0x7f247015d720 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2460001670 con 0x7f247015d720 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f247006ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 >> v1:172.21.15.107:6789/0 conn(0x7f247015d720 legacy=0x7f24700d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f2458003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2458003f60 0x7f24580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458004d00 0x7f2458005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700d0ce0 con 0x7f2458003f60 2023-11-24T08:18:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247006d400 con 0x7f2458004d00 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.134+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245800bae0 con 0x7f24700d3a40 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2458003f60 0x7f24580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 -- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 msgr2=0x7f2458003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700d3a40 0x7f2458003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 -- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458004d00 msgr2=0x7f2458005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458004d00 0x7f2458005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.135+0000 7f246f7fe640 1 -- 172.21.15.107:0/4160784318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458003520 con 0x7f2458003f60 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.136+0000 7f246f7fe640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2458003f60 0x7f24580047c0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f24700d3e50 tx=0x7f24580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbee2255ae7f826 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.136+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464004230 con 0x7f2458003f60 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.136+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464004390 con 0x7f2458003f60 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.136+0000 7f246effd640 1 -- 172.21.15.107:0/4160784318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464003830 con 0x7f2458003f60 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.137+0000 7f247651d640 1 -- 172.21.15.107:0/4160784318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2458003f60 msgr2=0x7f24580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.137+0000 7f247651d640 1 --2- 172.21.15.107:0/4160784318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2458003f60 0x7f24580047c0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f24700d3e50 tx=0x7f24580057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.137+0000 7f2474aba640 1 -- 172.21.15.107:0/4160784318 reap_dead start 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.138+0000 7f247651d640 1 -- 172.21.15.107:0/4160784318 shutdown_connections 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.138+0000 7f247651d640 1 -- 172.21.15.107:0/4160784318 >> 172.21.15.107:0/4160784318 conn(0x7f247008af60 msgr2=0x7f247009c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.138+0000 7f247651d640 1 -- 172.21.15.107:0/4160784318 shutdown_connections 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.138+0000 7f247651d640 1 -- 172.21.15.107:0/4160784318 wait complete. 2023-11-24T08:18:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.140+0000 7f247651d640 1 Processor -- start 2023-11-24T08:18:12.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.140+0000 7f247651d640 1 -- start start 2023-11-24T08:18:12.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 0x7f24701f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700b2e70 0x7f24700b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2470155650 con 0x7f247015e790 2023-11-24T08:18:12.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700b3810 con 0x7f2458003f60 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f247651d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24700cc2e0 con 0x7f24700b2e70 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f2474aba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f2474aba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36018/0 (socket says 172.21.15.107:36018) 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f2474aba640 1 -- 172.21.15.107:0/2451543748 learned_addr learned my addr 172.21.15.107:0/2451543748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.141+0000 7f246ffff640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 0x7f24701f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f246f7fe640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700b2e70 0x7f24700b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 -- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700b2e70 msgr2=0x7f24700b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700b2e70 0x7f24700b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 -- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 msgr2=0x7f24701f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 0x7f24701f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 -- 172.21.15.107:0/2451543748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700cc5c0 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f246ffff640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 0x7f24701f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f2474aba640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f247008b620 tx=0x7f246800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.142+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246800ed90 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.143+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468014420 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.143+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680147d0 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.143+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24700cc8a0 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.143+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24700cce60 con 0x7f247015e790 2023-11-24T08:18:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.144+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24700d1f30 con 0x7f247015e790 2023-11-24T08:18:12.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.145+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f2468014930 con 0x7f247015e790 2023-11-24T08:18:12.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.146+0000 7f246cff9640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 0x7f245c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:12.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.146+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2468088840 con 0x7f247015e790 2023-11-24T08:18:12.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.146+0000 7f246ffff640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 0x7f245c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:12.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.147+0000 7f246ffff640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 0x7f245c06ece0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2460000ff0 tx=0x7f2460001760 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:12.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.151+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2468055350 con 0x7f247015e790 2023-11-24T08:18:12.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.425+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f24700b09a0 con 0x7f245c06c820 2023-11-24T08:18:12.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.440+0000 7f246cff9640 1 -- 172.21.15.107:0/2451543748 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f2460005870 con 0x7f245c06c820 2023-11-24T08:18:12.442 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:12.442 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol2512", 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5437" 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:12.443 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 msgr2=0x7f245c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 0x7f245c06ece0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2460000ff0 tx=0x7f2460001760 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 msgr2=0x7f247015ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f247008b620 tx=0x7f246800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 shutdown_connections 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245c06c820 0x7f245c06ece0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015e790 0x7f247015ebf0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700b2e70 0x7f24700b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 --2- 172.21.15.107:0/2451543748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2458003f60 0x7f24701f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.443+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 >> 172.21.15.107:0/2451543748 conn(0x7f247008af60 msgr2=0x7f24700ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.444+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 shutdown_connections 2023-11-24T08:18:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:12.444+0000 7f247651d640 1 -- 172.21.15.107:0/2451543748 wait complete. 2023-11-24T08:18:13.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:18:13.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:18:13.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone1612 2023-11-24T08:18:13.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.695+0000 7f8ff488f640 1 Processor -- start 2023-11-24T08:18:13.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.695+0000 7f8ff488f640 1 -- start start 2023-11-24T08:18:13.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8ff488f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8ff488f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8ff015dee0 0x7f8ff006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8ff488f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff015e500 con 0x7f8ff0161a40 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8ff488f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff015e7e0 con 0x7f8ff015dee0 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8ff488f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff015e2e0 con 0x7f8ff006df60 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fef59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8ff015dee0 0x7f8ff006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fee59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8ff0161a40 0x7f8ff015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37390/0 (socket says 172.21.15.107:37390) 2023-11-24T08:18:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fee59c640 1 -- 172.21.15.107:0/4022237339 learned_addr learned my addr 172.21.15.107:0/4022237339 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8feed9d640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fef59e640 1 -- 172.21.15.107:0/4022237339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 msgr2=0x7f8ff006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fef59e640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff006a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.697+0000 7f8fef59e640 1 -- 172.21.15.107:0/4022237339 >> v1:172.21.15.107:6789/0 conn(0x7f8ff0161a40 legacy=0x7f8ff015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.698+0000 7f8fef59e640 1 -- 172.21.15.107:0/4022237339 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0160f60 con 0x7f8ff015dee0 2023-11-24T08:18:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.698+0000 7f8fedd9b640 1 -- 172.21.15.107:0/4022237339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579518630 0 0) 0x7f8ff015e500 con 0x7f8ff0161a40 2023-11-24T08:18:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.699+0000 7f8fef59e640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff006aa40 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009d90 tx=0x7f8fe802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4524f09163c90a6f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.699+0000 7f8fedd9b640 1 -- 172.21.15.107:0/4022237339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe802cc50 con 0x7f8ff015dee0 2023-11-24T08:18:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.699+0000 7f8fedd9b640 1 -- 172.21.15.107:0/4022237339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe802cdb0 con 0x7f8ff015dee0 2023-11-24T08:18:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.699+0000 7f8fedd9b640 1 -- 172.21.15.107:0/4022237339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8034660 con 0x7f8ff015dee0 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 -- 172.21.15.107:0/4022237339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 msgr2=0x7f8ff006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff006aa40 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009d90 tx=0x7f8fe802bea0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 -- 172.21.15.107:0/4022237339 shutdown_connections 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff006aa40 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 --2- 172.21.15.107:0/4022237339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 -- 172.21.15.107:0/4022237339 >> 172.21.15.107:0/4022237339 conn(0x7f8ff00b1bd0 msgr2=0x7f8ff00b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 -- 172.21.15.107:0/4022237339 shutdown_connections 2023-11-24T08:18:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.700+0000 7f8ff488f640 1 -- 172.21.15.107:0/4022237339 wait complete. 2023-11-24T08:18:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.702+0000 7f8ff488f640 1 Processor -- start 2023-11-24T08:18:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.702+0000 7f8ff488f640 1 -- start start 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ff0161a40 0x7f8ff00af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00b4ac0 con 0x7f8ff015dee0 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00b4da0 con 0x7f8ff0161a40 2023-11-24T08:18:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8ff488f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00b5080 con 0x7f8ff006df60 2023-11-24T08:18:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8fee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8fee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36044/0 (socket says 172.21.15.107:36044) 2023-11-24T08:18:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8fee59c640 1 -- 172.21.15.107:0/2985652825 learned_addr learned my addr 172.21.15.107:0/2985652825 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.703+0000 7f8feed9d640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fef59e640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ff0161a40 0x7f8ff00af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 -- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 msgr2=0x7f8ff015a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff015a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 -- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ff0161a40 msgr2=0x7f8ff00af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ff0161a40 0x7f8ff00af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 -- 172.21.15.107:0/2985652825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8009a70 con 0x7f8ff015dee0 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8feed9d640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fee59c640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c430 tx=0x7f8fe400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40074e0 con 0x7f8ff015dee0 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4007660 con 0x7f8ff015dee0 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.704+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff00b5360 con 0x7f8ff015dee0 2023-11-24T08:18:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.705+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4004a20 con 0x7f8ff015dee0 2023-11-24T08:18:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.705+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff00affb0 con 0x7f8ff015dee0 2023-11-24T08:18:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.706+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff01f4050 con 0x7f8ff015dee0 2023-11-24T08:18:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.707+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f8fe4004b80 con 0x7f8ff015dee0 2023-11-24T08:18:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.708+0000 7f8fd77fe640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 0x7f8fdc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.708+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fe40882f0 con 0x7f8ff015dee0 2023-11-24T08:18:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.709+0000 7f8feed9d640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 0x7f8fdc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.709+0000 7f8feed9d640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 0x7f8fdc06ece0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000a9d0 tx=0x7f8fe0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.713+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ff01f4050 con 0x7f8ff015dee0 2023-11-24T08:18:13.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:13.943+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff00cfe40 con 0x7f8fdc06c820 2023-11-24T08:18:14.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.739+0000 7f8fd77fe640 1 -- 172.21.15.107:0/2985652825 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f8ff00cfe40 con 0x7f8fdc06c820 2023-11-24T08:18:14.742 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:18:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:18:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:18:14.742 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 msgr2=0x7f8fdc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 0x7f8fdc06ece0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000a9d0 tx=0x7f8fe0008040 comp rx=0 tx=0).stop 2023-11-24T08:18:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 msgr2=0x7f8ff015a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c430 tx=0x7f8fe400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 shutdown_connections 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fdc06c820 0x7f8fdc06ece0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ff0161a40 0x7f8ff00af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ff015dee0 0x7f8ff015a760 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 --2- 172.21.15.107:0/2985652825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ff006df60 0x7f8ff015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.743+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 >> 172.21.15.107:0/2985652825 conn(0x7f8ff00b1bd0 msgr2=0x7f8ff006a710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.744+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 shutdown_connections 2023-11-24T08:18:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.744+0000 7f8ff488f640 1 -- 172.21.15.107:0/2985652825 wait complete. 2023-11-24T08:18:14.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone1612 2023-11-24T08:18:14.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.986+0000 7f66eb121640 1 Processor -- start 2023-11-24T08:18:14.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.986+0000 7f66eb121640 1 -- start start 2023-11-24T08:18:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e415b260 0x7f66e406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66e406e580 0x7f66e40b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e40ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e415b640 con 0x7f66e415b260 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e40b1510 con 0x7f66e406e580 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66eb121640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e415ed20 con 0x7f66e40b1080 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66ea11f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e415b260 0x7f66e406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:14.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66ea11f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e415b260 0x7f66e406e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36058/0 (socket says 172.21.15.107:36058) 2023-11-24T08:18:14.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66ea11f640 1 -- 172.21.15.107:0/2497942667 learned_addr learned my addr 172.21.15.107:0/2497942667 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:14.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66ea920640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e40ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:14.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.988+0000 7f66e991e640 1 --2- 172.21.15.107:0/2497942667 >> v2:172.21.15.120:3300/0 conn(0x7f66e406e580 0x7f66e40b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.989+0000 7f66ea11f640 1 -- 172.21.15.107:0/2497942667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 msgr2=0x7f66e40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.989+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e40ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.989+0000 7f66ea11f640 1 -- 172.21.15.107:0/2497942667 >> v2:172.21.15.120:3300/0 conn(0x7f66e406e580 msgr2=0x7f66e40b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.989+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2497942667 >> v2:172.21.15.120:3300/0 conn(0x7f66e406e580 0x7f66e40b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.989+0000 7f66ea11f640 1 -- 172.21.15.107:0/2497942667 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e415f8a0 con 0x7f66e415b260 2023-11-24T08:18:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.990+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e415b260 0x7f66e406e040 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f66e40ce370 tx=0x7f66e002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88ecebced154422 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.991+0000 7f66e911d640 1 -- 172.21.15.107:0/2497942667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0031750 con 0x7f66e415b260 2023-11-24T08:18:14.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.991+0000 7f66e911d640 1 -- 172.21.15.107:0/2497942667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00318b0 con 0x7f66e415b260 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.991+0000 7f66e911d640 1 -- 172.21.15.107:0/2497942667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0031c40 con 0x7f66e415b260 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.991+0000 7f66eb121640 1 -- 172.21.15.107:0/2497942667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e415b260 msgr2=0x7f66e406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.991+0000 7f66eb121640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e415b260 0x7f66e406e040 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f66e40ce370 tx=0x7f66e002b7f0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 -- 172.21.15.107:0/2497942667 shutdown_connections 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e40ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 --2- 172.21.15.107:0/2497942667 >> v2:172.21.15.120:3300/0 conn(0x7f66e406e580 0x7f66e40b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 --2- 172.21.15.107:0/2497942667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e415b260 0x7f66e406e040 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 -- 172.21.15.107:0/2497942667 >> 172.21.15.107:0/2497942667 conn(0x7f66e40cbb40 msgr2=0x7f66e40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 -- 172.21.15.107:0/2497942667 shutdown_connections 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.992+0000 7f66eb121640 1 -- 172.21.15.107:0/2497942667 wait complete. 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.993+0000 7f66eb121640 1 Processor -- start 2023-11-24T08:18:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.994+0000 7f66eb121640 1 -- start start 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.994+0000 7f66eb121640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66eb121640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e4152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66eb121640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e415b260 0x7f66e415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66eb121640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41554c0 con 0x7f66e406e580 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66eb121640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41567a0 con 0x7f66e415b260 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66eb121640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4156a80 con 0x7f66e40b1080 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36060/0 (socket says 172.21.15.107:36060) 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 -- 172.21.15.107:0/2602846723 learned_addr learned my addr 172.21.15.107:0/2602846723 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66e991e640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e4152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 -- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 msgr2=0x7f66e4152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e4152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea920640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e415b260 0x7f66e415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 -- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e415b260 msgr2=0x7f66e415c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e415b260 0x7f66e415c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.995+0000 7f66ea11f640 1 -- 172.21.15.107:0/2602846723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e00094d0 con 0x7f66e406e580 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66ea11f640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0033c80 con 0x7f66e406e580 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0033e00 con 0x7f66e406e580 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e4156d60 con 0x7f66e406e580 2023-11-24T08:18:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e003d6b0 con 0x7f66e406e580 2023-11-24T08:18:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.996+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e4157320 con 0x7f66e406e580 2023-11-24T08:18:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.997+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e4157510 con 0x7f66e406e580 2023-11-24T08:18:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.998+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f66e003a040 con 0x7f66e406e580 2023-11-24T08:18:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.999+0000 7f66daffd640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 0x7f66c806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.999+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66e00aa4a0 con 0x7f66e406e580 2023-11-24T08:18:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:14.999+0000 7f66e991e640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 0x7f66c806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.000+0000 7f66e991e640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 0x7f66c806ed10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f66d4006fd0 tx=0x7f66d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.003+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66e00770d0 con 0x7f66e406e580 2023-11-24T08:18:15.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.225+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f66e415f250 con 0x7f66c806c850 2023-11-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.236+0000 7f66daffd640 1 -- 172.21.15.107:0/2602846723 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f66d4009480 con 0x7f66c806c850 2023-11-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 msgr2=0x7f66c806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 0x7f66c806ed10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f66d4006fd0 tx=0x7f66d4008040 comp rx=0 tx=0).stop 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 msgr2=0x7f66e4153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 shutdown_connections 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806c850 0x7f66c806ed10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e415b260 0x7f66e415c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e40b1080 0x7f66e4152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 --2- 172.21.15.107:0/2602846723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e406e580 0x7f66e4153040 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 >> 172.21.15.107:0/2602846723 conn(0x7f66e40cbb40 msgr2=0x7f66e406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 shutdown_connections 2023-11-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.239+0000 7f66eb121640 1 -- 172.21.15.107:0/2602846723 wait complete. 2023-11-24T08:18:15.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol2512 2023-11-24T08:18:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.470+0000 7f404e706640 1 Processor -- start 2023-11-24T08:18:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.470+0000 7f404e706640 1 -- start start 2023-11-24T08:18:15.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.471+0000 7f404e706640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40480b0fb0 0x7f404806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.471+0000 7f404e706640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404815ed20 con 0x7f40480b0b40 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.471+0000 7f404e706640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404815f8a0 con 0x7f404815b260 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404e706640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404815b640 con 0x7f40480b0fb0 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404cf03640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f40480b0b40 0x7f404806c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37400/0 (socket says 172.21.15.107:37400) 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404cf03640 1 -- 172.21.15.107:0/1570487810 learned_addr learned my addr 172.21.15.107:0/1570487810 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404df05640 1 --2- 172.21.15.107:0/1570487810 >> v2:172.21.15.107:3300/0 conn(0x7f40480b0fb0 0x7f404806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f403ffff640 1 -- 172.21.15.107:0/1570487810 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182388907 0 0) 0x7f404815ed20 con 0x7f40480b0b40 2023-11-24T08:18:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f403ffff640 1 -- 172.21.15.107:0/1570487810 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034003520 con 0x7f40480b0b40 2023-11-24T08:18:15.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404df05640 1 -- 172.21.15.107:0/1570487810 >> v1:172.21.15.120:6789/0 conn(0x7f404815b260 legacy=0x7f404806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:15.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404df05640 1 -- 172.21.15.107:0/1570487810 >> v1:172.21.15.107:6789/0 conn(0x7f40480b0b40 legacy=0x7f404806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.472+0000 7f404df05640 1 -- 172.21.15.107:0/1570487810 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404806e750 con 0x7f40480b0fb0 2023-11-24T08:18:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f404df05640 1 --2- 172.21.15.107:0/1570487810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404806ccc0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f4040009d90 tx=0x7f404002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6de0bf67c5b61d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f403ffff640 1 -- 172.21.15.107:0/1570487810 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404002cb40 con 0x7f40480b0fb0 2023-11-24T08:18:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f403ffff640 1 -- 172.21.15.107:0/1570487810 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404002cca0 con 0x7f40480b0fb0 2023-11-24T08:18:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f403ffff640 1 -- 172.21.15.107:0/1570487810 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040034660 con 0x7f40480b0fb0 2023-11-24T08:18:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f404e706640 1 -- 172.21.15.107:0/1570487810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 msgr2=0x7f404806ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.474+0000 7f404e706640 1 --2- 172.21.15.107:0/1570487810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404806ccc0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f4040009d90 tx=0x7f404002bd90 comp rx=0 tx=0).stop 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.475+0000 7f404e706640 1 -- 172.21.15.107:0/1570487810 shutdown_connections 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.475+0000 7f404e706640 1 --2- 172.21.15.107:0/1570487810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404806ccc0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.475+0000 7f404e706640 1 -- 172.21.15.107:0/1570487810 >> 172.21.15.107:0/1570487810 conn(0x7f40480cbb40 msgr2=0x7f40480cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.475+0000 7f404e706640 1 -- 172.21.15.107:0/1570487810 shutdown_connections 2023-11-24T08:18:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.475+0000 7f404e706640 1 -- 172.21.15.107:0/1570487810 wait complete. 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.476+0000 7f404e706640 1 Processor -- start 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.477+0000 7f404e706640 1 -- start start 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.477+0000 7f404e706640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40480b0b40 0x7f404815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.477+0000 7f404e706640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.477+0000 7f404e706640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404815b260 0x7f40480b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404e706640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404815c970 con 0x7f40480b0fb0 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404e706640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404815bd60 con 0x7f40480b0b40 2023-11-24T08:18:15.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404e706640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404815c040 con 0x7f404815b260 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36078/0 (socket says 172.21.15.107:36078) 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 -- 172.21.15.107:0/1661052906 learned_addr learned my addr 172.21.15.107:0/1661052906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 -- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404815b260 msgr2=0x7f40480b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404815b260 0x7f40480b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 -- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40480b0b40 msgr2=0x7f404815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404d704640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40480b0b40 0x7f404815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40480b0b40 0x7f404815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.478+0000 7f404cf03640 1 -- 172.21.15.107:0/1661052906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040009a70 con 0x7f40480b0fb0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f404cf03640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f403800c410 tx=0x7f403800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038005ce0 con 0x7f40480b0fb0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4038005e40 con 0x7f40480b0fb0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404815c320 con 0x7f40480b0fb0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40380072f0 con 0x7f40480b0fb0 2023-11-24T08:18:15.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.479+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40480b4170 con 0x7f40480b0fb0 2023-11-24T08:18:15.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.480+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404804e840 con 0x7f40480b0fb0 2023-11-24T08:18:15.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.481+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f4038007510 con 0x7f40480b0fb0 2023-11-24T08:18:15.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.482+0000 7f403dffb640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 0x7f402c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.482+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4038087c00 con 0x7f40480b0fb0 2023-11-24T08:18:15.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.482+0000 7f404d704640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 0x7f402c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.483+0000 7f404d704640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 0x7f402c06ece0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f404400a750 tx=0x7f4044008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.487+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4038054740 con 0x7f40480b0fb0 2023-11-24T08:18:15.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.706+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol2512", "target": ["mon-mgr", ""]}) v1 -- 0x7f40480ce3e0 con 0x7f402c06c820 2023-11-24T08:18:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.725+0000 7f403dffb640 1 -- 172.21.15.107:0/1661052906 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f40480ce3e0 con 0x7f402c06c820 2023-11-24T08:18:15.727 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol2512/8afb5694-a4f9-43a8-9186-62acb4c80793 2023-11-24T08:18:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.727+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 msgr2=0x7f402c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.727+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 0x7f402c06ece0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f404400a750 tx=0x7f4044008040 comp rx=0 tx=0).stop 2023-11-24T08:18:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.727+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 msgr2=0x7f404815ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.727+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f403800c410 tx=0x7f403800c8d0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 shutdown_connections 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f402c06c820 0x7f402c06ece0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404815b260 0x7f40480b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40480b0fb0 0x7f404815ce00 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 --2- 172.21.15.107:0/1661052906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40480b0b40 0x7f404815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 >> 172.21.15.107:0/1661052906 conn(0x7f40480cbb40 msgr2=0x7f40480ac070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 shutdown_connections 2023-11-24T08:18:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.728+0000 7f404e706640 1 -- 172.21.15.107:0/1661052906 wait complete. 2023-11-24T08:18:15.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone1612 2023-11-24T08:18:15.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.955+0000 7f4b238a6640 1 Processor -- start 2023-11-24T08:18:15.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.955+0000 7f4b238a6640 1 -- start start 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.956+0000 7f4b238a6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0ccbd0 0x7f4b1c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.956+0000 7f4b238a6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b1c0c9650 0x7f4b1c0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.956+0000 7f4b238a6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c1655f0 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.956+0000 7f4b238a6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1c0cc380 con 0x7f4b1c0c9650 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b238a6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c162b30 con 0x7f4b1c0ccbd0 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b20e41640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4b1c0ccfb0 0x7f4b1c162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37410/0 (socket says 172.21.15.107:37410) 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b20e41640 1 -- 172.21.15.107:0/255428441 learned_addr learned my addr 172.21.15.107:0/255428441 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:15.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b21e43640 1 --2- 172.21.15.107:0/255428441 >> v2:172.21.15.107:3300/0 conn(0x7f4b1c0c9650 0x7f4b1c0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b21642640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0ccbd0 0x7f4b1c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b1bfff640 1 -- 172.21.15.107:0/255428441 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745384361 0 0) 0x7f4b1c1655f0 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b1bfff640 1 -- 172.21.15.107:0/255428441 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b04003520 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b21e43640 1 -- 172.21.15.107:0/255428441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0ccbd0 msgr2=0x7f4b1c1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.957+0000 7f4b21e43640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0ccbd0 0x7f4b1c1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.958+0000 7f4b21e43640 1 -- 172.21.15.107:0/255428441 >> v1:172.21.15.107:6789/0 conn(0x7f4b1c0ccfb0 legacy=0x7f4b1c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.958+0000 7f4b21e43640 1 -- 172.21.15.107:0/255428441 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c162e10 con 0x7f4b1c0c9650 2023-11-24T08:18:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b21e43640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0c9650 0x7f4b1c0c9ae0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f4b140097d0 tx=0x7f4b1402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1465f7629311408c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b1bfff640 1 -- 172.21.15.107:0/255428441 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1402d040 con 0x7f4b1c0c9650 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b1bfff640 1 -- 172.21.15.107:0/255428441 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b14031810 con 0x7f4b1c0c9650 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b1bfff640 1 -- 172.21.15.107:0/255428441 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14031ba0 con 0x7f4b1c0c9650 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b238a6640 1 -- 172.21.15.107:0/255428441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0c9650 msgr2=0x7f4b1c0c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.959+0000 7f4b238a6640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0c9650 0x7f4b1c0c9ae0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f4b140097d0 tx=0x7f4b1402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 -- 172.21.15.107:0/255428441 shutdown_connections 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0c9650 0x7f4b1c0c9ae0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 --2- 172.21.15.107:0/255428441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0ccbd0 0x7f4b1c1660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 -- 172.21.15.107:0/255428441 >> 172.21.15.107:0/255428441 conn(0x7f4b1c0b0c50 msgr2=0x7f4b1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 -- 172.21.15.107:0/255428441 shutdown_connections 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.960+0000 7f4b238a6640 1 -- 172.21.15.107:0/255428441 wait complete. 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.961+0000 7f4b238a6640 1 Processor -- start 2023-11-24T08:18:15.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.962+0000 7f4b238a6640 1 -- start start 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.962+0000 7f4b238a6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 0x7f4b1c1ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.962+0000 7f4b238a6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 0x7f4b1c0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.962+0000 7f4b238a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b238a6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c099e90 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b238a6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c09a170 con 0x7f4b1c0ccbd0 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b238a6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c09a450 con 0x7f4b1c0c9650 2023-11-24T08:18:15.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36092/0 (socket says 172.21.15.107:36092) 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 -- 172.21.15.107:0/2036607769 learned_addr learned my addr 172.21.15.107:0/2036607769 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21642640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 0x7f4b1c1ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 -- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 msgr2=0x7f4b1c1ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b20e41640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 0x7f4b1c0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 0x7f4b1c1ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 -- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 msgr2=0x7f4b1c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 0x7f4b1c0ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21e43640 1 -- 172.21.15.107:0/2036607769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b14009400 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b21642640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 0x7f4b1c1ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.963+0000 7f4b20e41640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 0x7f4b1c0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b21e43640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f4b140096f0 tx=0x7f4b140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14006e10 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1403ace0 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14044620 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c09a730 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.964+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c1ed6f0 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.966+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c09ac50 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.966+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f4b1404d440 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.967+0000 7f4b19ffb640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 0x7f4b0806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:15.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.967+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b140aa620 con 0x7f4b1c0ccfb0 2023-11-24T08:18:15.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.968+0000 7f4b21642640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 0x7f4b0806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.968+0000 7f4b21642640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 0x7f4b0806ece0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c007330 tx=0x7f4b0c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:15.972+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b1c09ac50 con 0x7f4b1c0ccfb0 2023-11-24T08:18:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.182+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b1c0cd7a0 con 0x7f4b0806c820 2023-11-24T08:18:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.191+0000 7f4b19ffb640 1 -- 172.21.15.107:0/2036607769 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f4b1c0cd7a0 con 0x7f4b0806c820 2023-11-24T08:18:16.192 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 msgr2=0x7f4b0806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 0x7f4b0806ece0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c007330 tx=0x7f4b0c00b040 comp rx=0 tx=0).stop 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 msgr2=0x7f4b1c158360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f4b140096f0 tx=0x7f4b140046c0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 shutdown_connections 2023-11-24T08:18:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4b0806c820 0x7f4b0806ece0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b1c0ccfb0 0x7f4b1c158360 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b1c0ccbd0 0x7f4b1c0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 --2- 172.21.15.107:0/2036607769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b1c0c9650 0x7f4b1c1ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 >> 172.21.15.107:0/2036607769 conn(0x7f4b1c0b0c50 msgr2=0x7f4b1c0c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 shutdown_connections 2023-11-24T08:18:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.193+0000 7f4b238a6640 1 -- 172.21.15.107:0/2036607769 wait complete. 2023-11-24T08:18:16.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.211 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:16.262 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:18:16.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446) 2023-11-24T08:18:16.315 INFO:teuthology.orchestra.run.smithi107.stdout:65 2023-11-24T08:18:16.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone1612 2023-11-24T08:18:16.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.513+0000 7f98273be640 1 Processor -- start 2023-11-24T08:18:16.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.513+0000 7f98273be640 1 -- start start 2023-11-24T08:18:16.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.514+0000 7f98273be640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f98200c9650 0x7f98200c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f98273be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9820166510 con 0x7f98200ccbd0 2023-11-24T08:18:16.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f98273be640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98201655f0 con 0x7f98201660b0 2023-11-24T08:18:16.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f98273be640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200cc380 con 0x7f98200c9650 2023-11-24T08:18:16.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f982515a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98200ccbd0 0x7f98200ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37426/0 (socket says 172.21.15.107:37426) 2023-11-24T08:18:16.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f982515a640 1 -- 172.21.15.107:0/2402964423 learned_addr learned my addr 172.21.15.107:0/2402964423 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:16.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f982595b640 1 --2- 172.21.15.107:0/2402964423 >> v2:172.21.15.120:3300/0 conn(0x7f98200c9650 0x7f98200c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575390057 0 0) 0x7f981c001110 con 0x7f98200ccbd0 2023-11-24T08:18:16.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.515+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c003520 con 0x7f98200ccbd0 2023-11-24T08:18:16.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158472998 0 0) 0x7f980c003520 con 0x7f98200ccbd0 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 >> v2:172.21.15.120:3300/0 conn(0x7f98200c9650 msgr2=0x7f98200c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 --2- 172.21.15.107:0/2402964423 >> v2:172.21.15.120:3300/0 conn(0x7f98200c9650 0x7f98200c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 >> v1:172.21.15.120:6789/0 conn(0x7f98201660b0 legacy=0x7f9820162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820162b30 con 0x7f98200ccbd0 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f981c001800 con 0x7f98200ccbd0 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 >> v1:172.21.15.107:6789/0 conn(0x7f98200ccbd0 legacy=0x7f98200ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200cc380 con 0x7f980c0047c0 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c020 con 0x7f980c003970 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c1e0 con 0x7f980c005180 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f9824959640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.516+0000 7f982515a640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f982595b640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f9824959640 1 -- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c005180 msgr2=0x7f980c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f9824959640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f9824959640 1 -- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f980c003970 msgr2=0x7f980c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f9824959640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f9824959640 1 -- 172.21.15.107:0/2402964423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c003520 con 0x7f980c0047c0 2023-11-24T08:18:16.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f982595b640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:16.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.517+0000 7f982595b640 1 -- 172.21.15.107:0/2402964423 reap_dead start 2023-11-24T08:18:16.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.518+0000 7f9824959640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f9810001750 tx=0x7f980c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31de25c8978d3be8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:16.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.518+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100052d0 con 0x7f980c0047c0 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.518+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981002e050 con 0x7f980c0047c0 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.518+0000 7f9817fff640 1 -- 172.21.15.107:0/2402964423 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810003830 con 0x7f980c0047c0 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 -- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 msgr2=0x7f980c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f9810001750 tx=0x7f980c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 -- 172.21.15.107:0/2402964423 shutdown_connections 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 --2- 172.21.15.107:0/2402964423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 -- 172.21.15.107:0/2402964423 >> 172.21.15.107:0/2402964423 conn(0x7f98200b0c50 msgr2=0x7f98200b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 -- 172.21.15.107:0/2402964423 shutdown_connections 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.519+0000 7f98273be640 1 -- 172.21.15.107:0/2402964423 wait complete. 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.520+0000 7f98273be640 1 Processor -- start 2023-11-24T08:18:16.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 -- start start 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c0047c0 0x7f9820071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 0x7f9820071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820070e90 con 0x7f98201587a0 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201651c0 con 0x7f9820158340 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.521+0000 7f98273be640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820070560 con 0x7f980c0047c0 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36096/0 (socket says 172.21.15.107:36096) 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 -- 172.21.15.107:0/1091058897 learned_addr learned my addr 172.21.15.107:0/1091058897 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982515a640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c0047c0 0x7f9820071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f9824959640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 0x7f9820071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 -- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c0047c0 msgr2=0x7f9820071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c0047c0 0x7f9820071320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 -- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 msgr2=0x7f9820071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 0x7f9820071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 -- 172.21.15.107:0/1091058897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820070840 con 0x7f98201587a0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f9824959640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 0x7f9820071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.522+0000 7f982595b640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f982009d280 tx=0x7f981800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.523+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818016420 con 0x7f98201587a0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.523+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818016580 con 0x7f98201587a0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.523+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180168f0 con 0x7f98201587a0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.523+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820070b20 con 0x7f98201587a0 2023-11-24T08:18:16.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.523+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98200ad1e0 con 0x7f98201587a0 2023-11-24T08:18:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.524+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820071000 con 0x7f98201587a0 2023-11-24T08:18:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.525+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f9818016a50 con 0x7f98201587a0 2023-11-24T08:18:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.526+0000 7f9815ffb640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 0x7f980406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.526+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9818089350 con 0x7f98201587a0 2023-11-24T08:18:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.526+0000 7f982515a640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 0x7f980406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:16.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.527+0000 7f982515a640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 0x7f980406ece0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f981c001e20 tx=0x7f981c000f70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.531+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9818091030 con 0x7f98201587a0 2023-11-24T08:18:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.731+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7f9820165b90 con 0x7f980406c820 2023-11-24T08:18:16.746 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:18:16.746 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:17:53", 2023-11-24T08:18:16.746 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 67108864, 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:18:04", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:18:13", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:18:16.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:18:02", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:18:16.748 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:18:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.744+0000 7f9815ffb640 1 -- 172.21.15.107:0/1091058897 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7f981c0015e0 con 0x7f980406c820 2023-11-24T08:18:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.746+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 msgr2=0x7f980406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.746+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 0x7f980406ece0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f981c001e20 tx=0x7f981c000f70 comp rx=0 tx=0).stop 2023-11-24T08:18:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.746+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 msgr2=0x7f98200720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.746+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f982009d280 tx=0x7f981800f520 comp rx=0 tx=0).stop 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 shutdown_connections 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f980406c820 0x7f980406ece0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98201587a0 0x7f98200720d0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820158340 0x7f9820071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 --2- 172.21.15.107:0/1091058897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f980c0047c0 0x7f9820071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 >> 172.21.15.107:0/1091058897 conn(0x7f98200b0c50 msgr2=0x7f98200c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 shutdown_connections 2023-11-24T08:18:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:16.747+0000 7f98273be640 1 -- 172.21.15.107:0/1091058897 wait complete. 2023-11-24T08:18:16.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.766 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:16.815 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793: ceph.quota.max_bytes: No such attribute 2023-11-24T08:18:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:18:16.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:16.912 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:18:16.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.913 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:16.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:16.963 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30 2023-11-24T08:18:17.016 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36 2023-11-24T08:18:17.017 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57 2023-11-24T08:18:17.018 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23 2023-11-24T08:18:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11 2023-11-24T08:18:17.020 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1 2023-11-24T08:18:17.021 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60 2023-11-24T08:18:17.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.023 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:18:17.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.083 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/) 2023-11-24T08:18:17.137 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:18:17.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:17.189 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:18:17.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.190 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/) 2023-11-24T08:18:17.242 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:18:17.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:17.293 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:18:17.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/) 2023-11-24T08:18:17.347 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:18:17.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.348 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793) 2023-11-24T08:18:17.401 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:17.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/) 2023-11-24T08:18:17.460 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:17.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.461 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49) 2023-11-24T08:18:17.516 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:17.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.517 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.49) 2023-11-24T08:18:17.577 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:17.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.579 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49) 2023-11-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:17.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.630 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.49) 2023-11-24T08:18:17.677 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:17.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.678 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49) 2023-11-24T08:18:17.776 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:17.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.777 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.49) 2023-11-24T08:18:17.829 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:17.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.830 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.49) 2023-11-24T08:18:17.881 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:17.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.882 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.49) 2023-11-24T08:18:17.934 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:17.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.935 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22) 2023-11-24T08:18:17.986 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:17.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:17.987 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.22) 2023-11-24T08:18:18.036 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:18.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.037 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22) 2023-11-24T08:18:18.085 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:18.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.086 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.22) 2023-11-24T08:18:18.140 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:18.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22) 2023-11-24T08:18:18.194 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:18.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.195 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.22) 2023-11-24T08:18:18.243 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:18.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.244 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.22) 2023-11-24T08:18:18.292 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:18.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.292 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.22) 2023-11-24T08:18:18.393 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:18.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.394 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20) 2023-11-24T08:18:18.449 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:18.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.20) 2023-11-24T08:18:18.508 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:18.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.510 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20) 2023-11-24T08:18:18.563 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:18.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.565 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.20) 2023-11-24T08:18:18.619 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:18.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.621 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20) 2023-11-24T08:18:18.674 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:18.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.676 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.20) 2023-11-24T08:18:18.728 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:18.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.729 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.20) 2023-11-24T08:18:18.782 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:18.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.784 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.20) 2023-11-24T08:18:18.835 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:18.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.836 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34) 2023-11-24T08:18:18.887 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:18.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.34) 2023-11-24T08:18:18.946 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:18.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.948 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34) 2023-11-24T08:18:18.998 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:18.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:18.999 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.34) 2023-11-24T08:18:19.050 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:19.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.052 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34) 2023-11-24T08:18:19.103 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:19.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.105 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.34) 2023-11-24T08:18:19.156 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:19.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.157 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.34) 2023-11-24T08:18:19.205 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:19.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.34) 2023-11-24T08:18:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:19.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55) 2023-11-24T08:18:19.353 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:19.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.355 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.55) 2023-11-24T08:18:19.412 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:19.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.414 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55) 2023-11-24T08:18:19.469 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:19.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.55) 2023-11-24T08:18:19.524 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:19.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.526 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55) 2023-11-24T08:18:19.580 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:19.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.581 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.55) 2023-11-24T08:18:19.634 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:19.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.636 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.55) 2023-11-24T08:18:19.689 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:19.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.691 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.55) 2023-11-24T08:18:19.743 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:19.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.745 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2) 2023-11-24T08:18:19.798 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:19.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.800 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.2) 2023-11-24T08:18:19.852 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:19.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.854 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2) 2023-11-24T08:18:19.908 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:19.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.2) 2023-11-24T08:18:19.958 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:19.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:19.959 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2) 2023-11-24T08:18:20.009 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.011 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.2) 2023-11-24T08:18:20.064 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.2) 2023-11-24T08:18:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:20.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.120 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.2) 2023-11-24T08:18:20.174 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:20.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.176 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35) 2023-11-24T08:18:20.229 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:20.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.230 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.35) 2023-11-24T08:18:20.287 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:20.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35) 2023-11-24T08:18:20.346 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:20.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.347 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.35) 2023-11-24T08:18:20.400 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:20.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.401 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35) 2023-11-24T08:18:20.451 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.452 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.35) 2023-11-24T08:18:20.501 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.502 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.35) 2023-11-24T08:18:20.552 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:20.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.553 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.35) 2023-11-24T08:18:20.604 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:20.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.606 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30) 2023-11-24T08:18:20.658 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:20.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.660 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.30) 2023-11-24T08:18:20.713 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:20.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30) 2023-11-24T08:18:20.769 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:20.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.773 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.30) 2023-11-24T08:18:20.825 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:20.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30) 2023-11-24T08:18:20.876 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.878 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.30) 2023-11-24T08:18:20.922 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:20.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:20.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.30) 2023-11-24T08:18:21.021 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:21.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.023 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.30) 2023-11-24T08:18:21.075 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:21.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.076 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14) 2023-11-24T08:18:21.125 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:21.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.127 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.14) 2023-11-24T08:18:21.178 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:21.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.178 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14) 2023-11-24T08:18:21.227 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:21.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.228 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.14) 2023-11-24T08:18:21.282 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:21.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14) 2023-11-24T08:18:21.332 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:21.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.333 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.14) 2023-11-24T08:18:21.387 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:21.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.388 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.14) 2023-11-24T08:18:21.437 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:21.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.14) 2023-11-24T08:18:21.490 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:21.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26) 2023-11-24T08:18:21.546 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:21.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.548 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.26) 2023-11-24T08:18:21.599 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:21.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26) 2023-11-24T08:18:21.649 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:21.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.651 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.26) 2023-11-24T08:18:21.696 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:21.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.697 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26) 2023-11-24T08:18:21.798 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:21.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.799 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.26) 2023-11-24T08:18:21.853 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:21.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.854 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.26) 2023-11-24T08:18:21.907 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:21.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.908 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.26) 2023-11-24T08:18:21.956 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:21.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:21.957 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51) 2023-11-24T08:18:22.008 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.010 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.51) 2023-11-24T08:18:22.060 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.062 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51) 2023-11-24T08:18:22.109 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:22.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.111 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.51) 2023-11-24T08:18:22.210 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:22.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.211 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51) 2023-11-24T08:18:22.265 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:22.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.267 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.51) 2023-11-24T08:18:22.321 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:22.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.51) 2023-11-24T08:18:22.378 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:22.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.380 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.51) 2023-11-24T08:18:22.431 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:22.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.433 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50) 2023-11-24T08:18:22.485 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.486 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.50) 2023-11-24T08:18:22.540 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50) 2023-11-24T08:18:22.591 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:22.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.592 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.50) 2023-11-24T08:18:22.642 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:22.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.644 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50) 2023-11-24T08:18:22.696 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:22.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.698 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.50) 2023-11-24T08:18:22.750 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:22.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.50) 2023-11-24T08:18:22.803 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:22.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.50) 2023-11-24T08:18:22.855 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:22.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.856 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13) 2023-11-24T08:18:22.909 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.13) 2023-11-24T08:18:22.968 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:22.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:22.970 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13) 2023-11-24T08:18:23.020 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.13) 2023-11-24T08:18:23.073 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13) 2023-11-24T08:18:23.129 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:23.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.131 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.13) 2023-11-24T08:18:23.183 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:23.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.13) 2023-11-24T08:18:23.237 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:23.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.13) 2023-11-24T08:18:23.295 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:23.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.296 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10) 2023-11-24T08:18:23.349 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:23.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.351 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.10) 2023-11-24T08:18:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:23.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.405 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10) 2023-11-24T08:18:23.456 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.458 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.10) 2023-11-24T08:18:23.510 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.511 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10) 2023-11-24T08:18:23.564 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:23.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.566 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.10) 2023-11-24T08:18:23.618 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:23.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.620 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.10) 2023-11-24T08:18:23.672 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:23.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.673 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.10) 2023-11-24T08:18:23.728 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:23.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.729 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47) 2023-11-24T08:18:23.784 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:23.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.47) 2023-11-24T08:18:23.836 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:23.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.837 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47) 2023-11-24T08:18:23.887 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.47) 2023-11-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:23.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47) 2023-11-24T08:18:23.992 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:23.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:23.993 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.47) 2023-11-24T08:18:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:24.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.47) 2023-11-24T08:18:24.095 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:24.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.096 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.47) 2023-11-24T08:18:24.149 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:24.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.151 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5) 2023-11-24T08:18:24.205 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:24.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.5) 2023-11-24T08:18:24.260 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:24.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5) 2023-11-24T08:18:24.310 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:24.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.312 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.5) 2023-11-24T08:18:24.363 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:24.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.364 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5) 2023-11-24T08:18:24.413 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:24.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.415 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.5) 2023-11-24T08:18:24.464 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:24.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.465 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.5) 2023-11-24T08:18:24.514 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:24.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.516 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.5) 2023-11-24T08:18:24.567 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:24.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.568 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37) 2023-11-24T08:18:24.621 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:24.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.622 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.37) 2023-11-24T08:18:24.677 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:24.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.679 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37) 2023-11-24T08:18:24.730 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:24.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.732 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.37) 2023-11-24T08:18:24.783 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:24.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.784 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37) 2023-11-24T08:18:24.833 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:24.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.834 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.37) 2023-11-24T08:18:24.885 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:24.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.886 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.37) 2023-11-24T08:18:24.941 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:24.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.37) 2023-11-24T08:18:24.995 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:24.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:24.997 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39) 2023-11-24T08:18:25.050 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:25.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.051 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.39) 2023-11-24T08:18:25.100 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:25.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.100 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39) 2023-11-24T08:18:25.151 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:25.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.39) 2023-11-24T08:18:25.204 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:25.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39) 2023-11-24T08:18:25.258 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:25.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.424 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.39) 2023-11-24T08:18:25.485 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:25.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.487 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.39) 2023-11-24T08:18:25.540 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:25.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.39) 2023-11-24T08:18:25.593 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:25.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54) 2023-11-24T08:18:25.642 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:25.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.644 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.54) 2023-11-24T08:18:25.695 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:25.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.697 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54) 2023-11-24T08:18:25.749 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:25.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.54) 2023-11-24T08:18:25.800 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:25.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.802 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54) 2023-11-24T08:18:25.854 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:25.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.856 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.54) 2023-11-24T08:18:25.912 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:25.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.913 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.54) 2023-11-24T08:18:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:25.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:25.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.54) 2023-11-24T08:18:26.018 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:26.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.020 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36) 2023-11-24T08:18:26.074 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:26.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.076 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.36) 2023-11-24T08:18:26.129 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:26.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.131 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36) 2023-11-24T08:18:26.179 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:26.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.36) 2023-11-24T08:18:26.228 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:26.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36) 2023-11-24T08:18:26.327 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:26.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.329 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.36) 2023-11-24T08:18:26.381 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:26.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.382 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.36) 2023-11-24T08:18:26.434 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:26.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.435 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.36) 2023-11-24T08:18:26.488 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:26.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18) 2023-11-24T08:18:26.544 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:26.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.18) 2023-11-24T08:18:26.600 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:26.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18) 2023-11-24T08:18:26.654 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:26.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.656 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.18) 2023-11-24T08:18:26.709 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:26.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18) 2023-11-24T08:18:26.766 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:26.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.768 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.18) 2023-11-24T08:18:26.820 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:26.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.822 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.18) 2023-11-24T08:18:26.878 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:26.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.880 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.18) 2023-11-24T08:18:26.931 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:26.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.932 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17) 2023-11-24T08:18:26.979 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:26.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:26.980 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.17) 2023-11-24T08:18:27.029 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:27.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.030 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17) 2023-11-24T08:18:27.079 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.080 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.17) 2023-11-24T08:18:27.131 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.133 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17) 2023-11-24T08:18:27.185 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:27.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.186 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.17) 2023-11-24T08:18:27.238 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:27.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.17) 2023-11-24T08:18:27.292 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:27.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.293 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.17) 2023-11-24T08:18:27.344 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:27.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4) 2023-11-24T08:18:27.397 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:27.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.4) 2023-11-24T08:18:27.450 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:27.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.451 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4) 2023-11-24T08:18:27.500 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.501 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.4) 2023-11-24T08:18:27.551 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4) 2023-11-24T08:18:27.599 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:27.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.4) 2023-11-24T08:18:27.653 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:27.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.654 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.4) 2023-11-24T08:18:27.710 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:27.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.711 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.4) 2023-11-24T08:18:27.763 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:27.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.764 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58) 2023-11-24T08:18:27.815 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:27.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.58) 2023-11-24T08:18:27.866 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:27.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58) 2023-11-24T08:18:27.923 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.924 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.58) 2023-11-24T08:18:27.975 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:27.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:27.976 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58) 2023-11-24T08:18:28.025 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.026 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.58) 2023-11-24T08:18:28.073 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.58) 2023-11-24T08:18:28.124 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:28.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.125 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.58) 2023-11-24T08:18:28.173 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:28.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.174 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29) 2023-11-24T08:18:28.224 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:28.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.29) 2023-11-24T08:18:28.281 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:28.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29) 2023-11-24T08:18:28.336 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:28.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.338 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.29) 2023-11-24T08:18:28.389 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:28.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29) 2023-11-24T08:18:28.444 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.445 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.29) 2023-11-24T08:18:28.498 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.29) 2023-11-24T08:18:28.554 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:28.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.555 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.29) 2023-11-24T08:18:28.608 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:28.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.609 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38) 2023-11-24T08:18:28.660 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:28.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.38) 2023-11-24T08:18:28.711 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:28.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.712 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38) 2023-11-24T08:18:28.764 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:28.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.765 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.38) 2023-11-24T08:18:28.817 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:28.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.818 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38) 2023-11-24T08:18:28.872 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.873 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.38) 2023-11-24T08:18:28.930 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:28.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.931 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.38) 2023-11-24T08:18:28.983 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:28.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:28.984 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.38) 2023-11-24T08:18:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:29.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:29.037 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19) 2023-11-24T08:18:29.092 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:29.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:29.094 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.19) 2023-11-24T08:18:29.150 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:29.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:29.151 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19) 2023-11-24T08:18:29.206 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:31.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.545 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.19) 2023-11-24T08:18:31.607 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:31.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.609 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19) 2023-11-24T08:18:31.661 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:31.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.663 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.19) 2023-11-24T08:18:31.716 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:31.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.717 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.19) 2023-11-24T08:18:31.773 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:31.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.774 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.19) 2023-11-24T08:18:31.827 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:31.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7) 2023-11-24T08:18:31.886 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:31.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.7) 2023-11-24T08:18:31.940 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:31.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7) 2023-11-24T08:18:31.993 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:31.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:31.995 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.7) 2023-11-24T08:18:32.046 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:32.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7) 2023-11-24T08:18:32.102 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.7) 2023-11-24T08:18:32.152 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.7) 2023-11-24T08:18:32.204 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:32.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.7) 2023-11-24T08:18:32.256 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:32.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.257 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48) 2023-11-24T08:18:32.310 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:32.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.311 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.48) 2023-11-24T08:18:32.366 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:32.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48) 2023-11-24T08:18:32.421 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:32.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.422 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.48) 2023-11-24T08:18:32.474 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:32.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.476 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48) 2023-11-24T08:18:32.522 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.523 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.48) 2023-11-24T08:18:32.573 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.574 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.48) 2023-11-24T08:18:32.624 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:32.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.626 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.48) 2023-11-24T08:18:32.674 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:32.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57) 2023-11-24T08:18:32.725 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:32.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.57) 2023-11-24T08:18:32.776 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:32.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57) 2023-11-24T08:18:32.833 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:32.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.835 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.57) 2023-11-24T08:18:32.886 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:32.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57) 2023-11-24T08:18:32.941 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.943 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.57) 2023-11-24T08:18:32.994 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:32.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:32.996 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.57) 2023-11-24T08:18:33.046 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:33.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.048 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.57) 2023-11-24T08:18:33.102 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:33.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.103 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6) 2023-11-24T08:18:33.153 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:33.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.6) 2023-11-24T08:18:33.203 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:33.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6) 2023-11-24T08:18:33.257 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:33.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.259 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.6) 2023-11-24T08:18:33.308 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:33.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.309 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6) 2023-11-24T08:18:33.357 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:33.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.358 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.6) 2023-11-24T08:18:33.454 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:33.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.6) 2023-11-24T08:18:33.505 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:33.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.506 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.6) 2023-11-24T08:18:33.555 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:33.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.556 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53) 2023-11-24T08:18:33.607 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:33.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.53) 2023-11-24T08:18:33.661 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:33.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53) 2023-11-24T08:18:33.714 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:33.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.53) 2023-11-24T08:18:33.769 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:33.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.770 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53) 2023-11-24T08:18:33.819 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:33.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.821 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.53) 2023-11-24T08:18:33.874 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:33.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.875 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.53) 2023-11-24T08:18:33.925 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:33.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.53) 2023-11-24T08:18:33.977 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:33.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:33.978 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40) 2023-11-24T08:18:34.030 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.031 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.40) 2023-11-24T08:18:34.087 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.089 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40) 2023-11-24T08:18:34.136 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:34.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.137 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.40) 2023-11-24T08:18:34.187 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:34.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.188 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40) 2023-11-24T08:18:34.242 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:34.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.40) 2023-11-24T08:18:34.296 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:34.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.298 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.40) 2023-11-24T08:18:34.348 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:34.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.349 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.40) 2023-11-24T08:18:34.397 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:34.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.398 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42) 2023-11-24T08:18:34.445 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.446 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.42) 2023-11-24T08:18:34.497 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.498 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42) 2023-11-24T08:18:34.550 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:34.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.42) 2023-11-24T08:18:34.604 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:34.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.605 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42) 2023-11-24T08:18:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:34.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.42) 2023-11-24T08:18:34.710 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:34.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.714 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.42) 2023-11-24T08:18:34.771 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:34.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.773 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.42) 2023-11-24T08:18:34.825 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:34.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0) 2023-11-24T08:18:34.882 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.884 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.0) 2023-11-24T08:18:34.939 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:34.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0) 2023-11-24T08:18:34.995 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:34.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:34.996 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.0) 2023-11-24T08:18:35.051 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:35.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.052 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0) 2023-11-24T08:18:35.103 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:35.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.105 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.0) 2023-11-24T08:18:35.158 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:35.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.160 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.0) 2023-11-24T08:18:35.218 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:35.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.218 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.0) 2023-11-24T08:18:35.269 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:35.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27) 2023-11-24T08:18:35.324 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:35.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.325 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.27) 2023-11-24T08:18:35.378 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:35.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.380 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27) 2023-11-24T08:18:35.430 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:35.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.432 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.27) 2023-11-24T08:18:35.488 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:35.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.489 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27) 2023-11-24T08:18:35.541 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:35.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.27) 2023-11-24T08:18:35.595 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:35.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.27) 2023-11-24T08:18:35.645 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:35.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.646 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.27) 2023-11-24T08:18:35.695 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:35.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.696 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25) 2023-11-24T08:18:35.746 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:35.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.747 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.25) 2023-11-24T08:18:35.796 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:35.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.798 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25) 2023-11-24T08:18:35.849 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:35.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.25) 2023-11-24T08:18:35.904 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:35.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25) 2023-11-24T08:18:35.960 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:35.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:35.962 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.25) 2023-11-24T08:18:36.013 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:36.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.014 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.25) 2023-11-24T08:18:36.066 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:36.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.25) 2023-11-24T08:18:36.118 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:36.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.119 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32) 2023-11-24T08:18:36.169 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:36.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.171 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.32) 2023-11-24T08:18:36.223 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:36.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.224 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32) 2023-11-24T08:18:36.279 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:36.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.280 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.32) 2023-11-24T08:18:36.332 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:36.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.334 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32) 2023-11-24T08:18:36.387 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:36.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.388 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.32) 2023-11-24T08:18:36.441 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:36.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.32) 2023-11-24T08:18:36.495 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:36.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.496 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.32) 2023-11-24T08:18:36.547 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:36.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28) 2023-11-24T08:18:36.599 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:36.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.28) 2023-11-24T08:18:36.653 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:36.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.654 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28) 2023-11-24T08:18:36.709 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:36.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.710 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.28) 2023-11-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:36.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.767 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28) 2023-11-24T08:18:36.820 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:36.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.821 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.28) 2023-11-24T08:18:36.874 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:36.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.875 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.28) 2023-11-24T08:18:36.927 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:36.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.929 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.28) 2023-11-24T08:18:36.980 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:36.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:36.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21) 2023-11-24T08:18:37.033 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.034 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.21) 2023-11-24T08:18:37.088 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.090 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21) 2023-11-24T08:18:37.144 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:37.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.145 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.21) 2023-11-24T08:18:37.199 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:37.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.200 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21) 2023-11-24T08:18:37.253 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:37.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.254 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.21) 2023-11-24T08:18:37.307 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:37.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.309 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.21) 2023-11-24T08:18:37.356 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:37.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.357 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.21) 2023-11-24T08:18:37.410 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:37.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.412 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9) 2023-11-24T08:18:37.463 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.464 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.9) 2023-11-24T08:18:37.514 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.516 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9) 2023-11-24T08:18:37.562 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:37.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.563 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.9) 2023-11-24T08:18:37.613 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:37.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.615 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9) 2023-11-24T08:18:37.667 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:37.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.668 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.9) 2023-11-24T08:18:37.722 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:37.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.723 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.9) 2023-11-24T08:18:37.775 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:37.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.9) 2023-11-24T08:18:37.820 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:37.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.821 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23) 2023-11-24T08:18:37.921 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.922 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.23) 2023-11-24T08:18:37.981 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:37.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:37.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23) 2023-11-24T08:18:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.033 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.23) 2023-11-24T08:18:38.083 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.084 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23) 2023-11-24T08:18:38.137 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:38.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.23) 2023-11-24T08:18:38.191 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:38.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.193 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.23) 2023-11-24T08:18:38.250 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:38.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.23) 2023-11-24T08:18:38.304 INFO:teuthology.orchestra.run.smithi107.stdout:1700813876 2023-11-24T08:18:38.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.305 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15) 2023-11-24T08:18:38.359 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:38.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.361 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.15) 2023-11-24T08:18:38.418 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:38.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15) 2023-11-24T08:18:38.471 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.472 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.15) 2023-11-24T08:18:38.525 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.526 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15) 2023-11-24T08:18:38.581 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:38.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.582 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.15) 2023-11-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:38.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.635 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.15) 2023-11-24T08:18:38.687 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:38.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.688 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.15) 2023-11-24T08:18:38.744 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:38.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.746 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8) 2023-11-24T08:18:38.799 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:38.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.801 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.8) 2023-11-24T08:18:38.858 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:38.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.859 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8) 2023-11-24T08:18:38.911 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.8) 2023-11-24T08:18:38.961 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:38.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:38.962 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8) 2023-11-24T08:18:39.013 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.014 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.8) 2023-11-24T08:18:39.067 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.8) 2023-11-24T08:18:39.118 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:39.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.119 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.8) 2023-11-24T08:18:39.167 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:39.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46) 2023-11-24T08:18:39.216 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:39.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.218 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.46) 2023-11-24T08:18:39.267 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:39.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46) 2023-11-24T08:18:39.319 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:39.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.320 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.46) 2023-11-24T08:18:39.371 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:39.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.372 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46) 2023-11-24T08:18:39.425 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.427 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.46) 2023-11-24T08:18:39.479 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.481 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.46) 2023-11-24T08:18:39.533 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:39.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.535 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.46) 2023-11-24T08:18:39.587 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:39.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.589 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33) 2023-11-24T08:18:39.646 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:39.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.646 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.33) 2023-11-24T08:18:39.700 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:39.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.701 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33) 2023-11-24T08:18:39.750 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:39.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.33) 2023-11-24T08:18:39.800 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:39.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.802 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33) 2023-11-24T08:18:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.857 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.33) 2023-11-24T08:18:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:39.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.33) 2023-11-24T08:18:39.963 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:39.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:39.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.33) 2023-11-24T08:18:40.016 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:40.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.017 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45) 2023-11-24T08:18:40.072 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.074 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.45) 2023-11-24T08:18:40.127 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45) 2023-11-24T08:18:40.181 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:40.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.182 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.45) 2023-11-24T08:18:40.232 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:40.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.233 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45) 2023-11-24T08:18:40.282 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:40.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.45) 2023-11-24T08:18:40.335 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:40.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.337 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.45) 2023-11-24T08:18:40.388 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:40.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.389 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.45) 2023-11-24T08:18:40.442 INFO:teuthology.orchestra.run.smithi107.stdout:1700813880 2023-11-24T08:18:40.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61) 2023-11-24T08:18:40.500 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.501 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.61) 2023-11-24T08:18:40.553 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.555 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61) 2023-11-24T08:18:40.604 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:40.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.605 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.61) 2023-11-24T08:18:40.657 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:40.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.659 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61) 2023-11-24T08:18:40.714 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:40.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.61) 2023-11-24T08:18:40.770 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:40.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.771 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.61) 2023-11-24T08:18:40.823 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:40.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.825 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.61) 2023-11-24T08:18:40.876 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:40.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.877 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62) 2023-11-24T08:18:40.929 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.931 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.62) 2023-11-24T08:18:40.986 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:40.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:40.987 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62) 2023-11-24T08:18:41.042 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.62) 2023-11-24T08:18:41.095 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.096 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62) 2023-11-24T08:18:41.149 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:41.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.62) 2023-11-24T08:18:41.204 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:41.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.62) 2023-11-24T08:18:41.259 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:41.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.62) 2023-11-24T08:18:41.313 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:41.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.314 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52) 2023-11-24T08:18:41.366 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:41.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.52) 2023-11-24T08:18:41.418 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:41.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.420 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52) 2023-11-24T08:18:41.472 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.473 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.52) 2023-11-24T08:18:41.524 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52) 2023-11-24T08:18:41.579 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:41.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.52) 2023-11-24T08:18:41.635 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:41.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.637 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.52) 2023-11-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:41.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.694 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.52) 2023-11-24T08:18:41.751 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:41.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.752 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11) 2023-11-24T08:18:41.805 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:41.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.807 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.11) 2023-11-24T08:18:41.860 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:41.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.862 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11) 2023-11-24T08:18:41.915 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.916 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.11) 2023-11-24T08:18:41.969 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:41.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:41.970 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11) 2023-11-24T08:18:42.020 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.11) 2023-11-24T08:18:42.074 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.076 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.11) 2023-11-24T08:18:42.127 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:42.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.130 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.11) 2023-11-24T08:18:42.183 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:42.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44) 2023-11-24T08:18:42.237 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:42.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.44) 2023-11-24T08:18:42.294 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:42.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44) 2023-11-24T08:18:42.346 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:42.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.347 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.44) 2023-11-24T08:18:42.397 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:42.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44) 2023-11-24T08:18:42.450 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.452 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.44) 2023-11-24T08:18:42.504 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.506 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.44) 2023-11-24T08:18:42.562 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:42.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.563 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.44) 2023-11-24T08:18:42.617 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:42.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.619 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3) 2023-11-24T08:18:42.672 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:42.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.674 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.3) 2023-11-24T08:18:42.729 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:42.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.730 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3) 2023-11-24T08:18:42.785 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:42.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.786 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.3) 2023-11-24T08:18:42.839 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:42.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.840 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3) 2023-11-24T08:18:42.893 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.894 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.3) 2023-11-24T08:18:42.947 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:42.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:42.948 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.3) 2023-11-24T08:18:43.003 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:43.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.004 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.3) 2023-11-24T08:18:43.056 INFO:teuthology.orchestra.run.smithi107.stdout:1700813874 2023-11-24T08:18:43.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.057 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24) 2023-11-24T08:18:43.110 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:43.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.111 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.24) 2023-11-24T08:18:43.164 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:43.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.166 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24) 2023-11-24T08:18:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:43.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.218 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.24) 2023-11-24T08:18:43.268 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:43.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24) 2023-11-24T08:18:43.321 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:43.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.322 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.24) 2023-11-24T08:18:43.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:18:43.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:18:43.378 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:43.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.379 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.24) 2023-11-24T08:18:43.468 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:43.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.24) 2023-11-24T08:18:43.512 INFO:teuthology.orchestra.run.smithi107.stdout:1700813877 2023-11-24T08:18:43.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.513 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56) 2023-11-24T08:18:43.611 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:43.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.613 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.56) 2023-11-24T08:18:43.667 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:43.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.668 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56) 2023-11-24T08:18:43.720 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:43.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.721 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.56) 2023-11-24T08:18:43.777 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:43.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.778 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56) 2023-11-24T08:18:43.831 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:43.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.832 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.56) 2023-11-24T08:18:43.883 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:43.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.884 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.56) 2023-11-24T08:18:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:43.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.936 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.56) 2023-11-24T08:18:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:1700813881 2023-11-24T08:18:43.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:43.987 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63) 2023-11-24T08:18:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.63) 2023-11-24T08:18:44.097 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.098 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63) 2023-11-24T08:18:44.150 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:44.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.151 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.63) 2023-11-24T08:18:44.203 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:44.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.204 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63) 2023-11-24T08:18:44.258 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:44.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.260 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.63) 2023-11-24T08:18:44.313 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:44.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.314 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.63) 2023-11-24T08:18:44.363 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:44.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.364 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.63) 2023-11-24T08:18:44.414 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:44.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.416 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59) 2023-11-24T08:18:44.466 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.468 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.59) 2023-11-24T08:18:44.518 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.519 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59) 2023-11-24T08:18:44.569 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:44.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.571 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.59) 2023-11-24T08:18:44.624 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:44.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59) 2023-11-24T08:18:44.679 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:44.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.59) 2023-11-24T08:18:44.733 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:44.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.734 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.59) 2023-11-24T08:18:44.787 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:44.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.789 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.59) 2023-11-24T08:18:44.840 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:44.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.841 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41) 2023-11-24T08:18:44.891 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.893 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.41) 2023-11-24T08:18:44.943 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:44.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:44.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41) 2023-11-24T08:18:44.999 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.000 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.41) 2023-11-24T08:18:45.055 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41) 2023-11-24T08:18:45.108 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:45.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.109 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.41) 2023-11-24T08:18:45.157 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:45.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.160 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.41) 2023-11-24T08:18:45.206 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:45.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.41) 2023-11-24T08:18:45.308 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:45.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.309 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12) 2023-11-24T08:18:45.358 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:45.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.360 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.12) 2023-11-24T08:18:45.414 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:45.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.415 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12) 2023-11-24T08:18:45.466 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.467 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.12) 2023-11-24T08:18:45.518 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.520 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12) 2023-11-24T08:18:45.573 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:45.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.574 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.12) 2023-11-24T08:18:45.630 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:45.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.632 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.12) 2023-11-24T08:18:45.683 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:45.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.12) 2023-11-24T08:18:45.735 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:45.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43) 2023-11-24T08:18:45.790 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:45.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.791 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.43) 2023-11-24T08:18:45.844 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:45.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.846 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43) 2023-11-24T08:18:45.897 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.898 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.43) 2023-11-24T08:18:45.954 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:45.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:45.955 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43) 2023-11-24T08:18:46.009 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.010 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.43) 2023-11-24T08:18:46.064 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.43) 2023-11-24T08:18:46.117 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:46.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.118 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.43) 2023-11-24T08:18:46.168 INFO:teuthology.orchestra.run.smithi107.stdout:1700813879 2023-11-24T08:18:46.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.169 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16) 2023-11-24T08:18:46.221 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:46.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.222 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.16) 2023-11-24T08:18:46.273 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:46.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.274 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16) 2023-11-24T08:18:46.323 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:46.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.324 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.16) 2023-11-24T08:18:46.378 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:46.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.379 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16) 2023-11-24T08:18:46.430 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.431 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.16) 2023-11-24T08:18:46.483 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.485 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.16) 2023-11-24T08:18:46.536 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:46.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.16) 2023-11-24T08:18:46.590 INFO:teuthology.orchestra.run.smithi107.stdout:1700813875 2023-11-24T08:18:46.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.592 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31) 2023-11-24T08:18:46.644 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:46.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.645 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.31) 2023-11-24T08:18:46.700 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:46.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.701 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31) 2023-11-24T08:18:46.753 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:46.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.754 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.31) 2023-11-24T08:18:46.810 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:46.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.811 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31) 2023-11-24T08:18:46.862 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.31) 2023-11-24T08:18:46.915 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:46.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.916 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.31) 2023-11-24T08:18:46.968 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:46.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:46.970 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.31) 2023-11-24T08:18:47.021 INFO:teuthology.orchestra.run.smithi107.stdout:1700813878 2023-11-24T08:18:47.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1) 2023-11-24T08:18:47.075 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:47.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.076 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.1) 2023-11-24T08:18:47.126 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:47.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.127 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1) 2023-11-24T08:18:47.174 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:47.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.175 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.1) 2023-11-24T08:18:47.227 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:47.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.228 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1) 2023-11-24T08:18:47.282 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:47.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.283 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.1) 2023-11-24T08:18:47.336 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:47.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.337 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.1) 2023-11-24T08:18:47.390 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:47.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.1) 2023-11-24T08:18:47.446 INFO:teuthology.orchestra.run.smithi107.stdout:1700813873 2023-11-24T08:18:47.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.448 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60) 2023-11-24T08:18:47.499 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:47.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.500 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.60) 2023-11-24T08:18:47.552 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:18:47.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.553 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60) 2023-11-24T08:18:47.606 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:47.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.60) 2023-11-24T08:18:47.663 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:18:47.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.665 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60) 2023-11-24T08:18:47.715 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:47.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.717 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.60) 2023-11-24T08:18:47.768 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:18:47.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol2512/.snap/subvol_snap5437/8afb5694-a4f9-43a8-9186-62acb4c80793/subvolume_file.60) 2023-11-24T08:18:47.820 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:47.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:47.821 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone1612/1ffbb0cb-f1f3-48d6-ac55-00ca76d0c446/subvolume_file.60) 2023-11-24T08:18:47.872 INFO:teuthology.orchestra.run.smithi107.stdout:1700813882 2023-11-24T08:18:47.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol2512 subvol_snap5437 2023-11-24T08:18:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.085+0000 7f3ecf17a640 1 Processor -- start 2023-11-24T08:18:48.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.086+0000 7f3ecf17a640 1 -- start start 2023-11-24T08:18:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.087+0000 7f3ecf17a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3ec813e040 0x7f3ec806c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.087+0000 7f3ecf17a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3ec813e420 0x7f3ec806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.087+0000 7f3ecf17a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec813fa60 con 0x7f3ec813ecf0 2023-11-24T08:18:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.087+0000 7f3ecf17a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec813f850 con 0x7f3ec813e040 2023-11-24T08:18:48.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.087+0000 7f3ecf17a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec813fd40 con 0x7f3ec813e420 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece979640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3ec813ecf0 0x7f3ec813f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50848/0 (socket says 172.21.15.107:50848) 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece178640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3ec813e040 0x7f3ec806c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece979640 1 -- 172.21.15.107:0/2492473532 learned_addr learned my addr 172.21.15.107:0/2492473532 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ecd977640 1 --2- 172.21.15.107:0/2492473532 >> v2:172.21.15.120:3300/0 conn(0x7f3ec813e420 0x7f3ec806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece178640 1 -- 172.21.15.107:0/2492473532 >> v2:172.21.15.120:3300/0 conn(0x7f3ec813e420 msgr2=0x7f3ec806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:48.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece178640 1 --2- 172.21.15.107:0/2492473532 >> v2:172.21.15.120:3300/0 conn(0x7f3ec813e420 0x7f3ec806ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece178640 1 -- 172.21.15.107:0/2492473532 >> v1:172.21.15.107:6789/0 conn(0x7f3ec813ecf0 legacy=0x7f3ec813f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.088+0000 7f3ece178640 1 -- 172.21.15.107:0/2492473532 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8140020 con 0x7f3ec813e040 2023-11-24T08:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.089+0000 7f3ece178640 1 --2- 172.21.15.107:0/2492473532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e040 0x7f3ec806c4f0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009d90 tx=0x7f3ebc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a73e9e05631d5f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.089+0000 7f3ecd176640 1 -- 172.21.15.107:0/2492473532 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc02cc60 con 0x7f3ec813e040 2023-11-24T08:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.089+0000 7f3ecd176640 1 -- 172.21.15.107:0/2492473532 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc02cdc0 con 0x7f3ec813e040 2023-11-24T08:18:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.089+0000 7f3ecd176640 1 -- 172.21.15.107:0/2492473532 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc034660 con 0x7f3ec813e040 2023-11-24T08:18:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 -- 172.21.15.107:0/2492473532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e040 msgr2=0x7f3ec806c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/2492473532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e040 0x7f3ec806c4f0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009d90 tx=0x7f3ebc02beb0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 -- 172.21.15.107:0/2492473532 shutdown_connections 2023-11-24T08:18:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/2492473532 >> v2:172.21.15.120:3300/0 conn(0x7f3ec813e420 0x7f3ec806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/2492473532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e040 0x7f3ec806c4f0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 -- 172.21.15.107:0/2492473532 >> 172.21.15.107:0/2492473532 conn(0x7f3ec806dc30 msgr2=0x7f3ec806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 -- 172.21.15.107:0/2492473532 shutdown_connections 2023-11-24T08:18:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.090+0000 7f3ecf17a640 1 -- 172.21.15.107:0/2492473532 wait complete. 2023-11-24T08:18:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.092+0000 7f3ecf17a640 1 Processor -- start 2023-11-24T08:18:48.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.092+0000 7f3ecf17a640 1 -- start start 2023-11-24T08:18:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec813e040 0x7f3ec81546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 0x7f3ec8138460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81f0b10 con 0x7f3ec813e420 2023-11-24T08:18:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81399e0 con 0x7f3ec813ecf0 2023-11-24T08:18:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecf17a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8138dd0 con 0x7f3ec813e040 2023-11-24T08:18:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.093+0000 7f3ecd977640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55378/0 (socket says 172.21.15.107:55378) 2023-11-24T08:18:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 -- 172.21.15.107:0/4171780111 learned_addr learned my addr 172.21.15.107:0/4171780111 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ece178640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec813e040 0x7f3ec81546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 -- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec813e040 msgr2=0x7f3ec81546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ece979640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 0x7f3ec8138460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec813e040 0x7f3ec81546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 -- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 msgr2=0x7f3ec8138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 0x7f3ec8138460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 -- 172.21.15.107:0/4171780111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc009a70 con 0x7f3ec813e420 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ece979640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 0x7f3ec8138460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.094+0000 7f3ecd977640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400d800 tx=0x7f3eb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.095+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400b6c0 con 0x7f3ec813e420 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.095+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb400b820 con 0x7f3ec813e420 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.095+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400bbb0 con 0x7f3ec813e420 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.095+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec81390b0 con 0x7f3ec813e420 2023-11-24T08:18:48.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.095+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8139670 con 0x7f3ec813e420 2023-11-24T08:18:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.096+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8139800 con 0x7f3ec813e420 2023-11-24T08:18:48.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.097+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7f3eb400bd10 con 0x7f3ec813e420 2023-11-24T08:18:48.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.098+0000 7f3ec6ffd640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 0x7f3eac06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.098+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3eb4088a80 con 0x7f3ec813e420 2023-11-24T08:18:48.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.098+0000 7f3ece178640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 0x7f3eac06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.099+0000 7f3ece178640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 0x7f3eac06ece0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc036000 tx=0x7f3ebc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.103+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3eb4055590 con 0x7f3ec813e420 2023-11-24T08:18:48.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.325+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol2512", "snap_name": "subvol_snap5437", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ec81405e0 con 0x7f3eac06c820 2023-11-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.380+0000 7f3ec6ffd640 1 -- 172.21.15.107:0/4171780111 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3ebc0440e0 con 0x7f3eac06c820 2023-11-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 msgr2=0x7f3eac06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 0x7f3eac06ece0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc036000 tx=0x7f3ebc030000 comp rx=0 tx=0).stop 2023-11-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 msgr2=0x7f3ec8137f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400d800 tx=0x7f3eb400dcc0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 shutdown_connections 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3eac06c820 0x7f3eac06ece0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec813ecf0 0x7f3ec8138460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ec813e420 0x7f3ec8137f20 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 --2- 172.21.15.107:0/4171780111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec813e040 0x7f3ec81546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 >> 172.21.15.107:0/4171780111 conn(0x7f3ec806dc30 msgr2=0x7f3ec80aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 shutdown_connections 2023-11-24T08:18:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.383+0000 7f3ecf17a640 1 -- 172.21.15.107:0/4171780111 wait complete. 2023-11-24T08:18:48.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol2512 2023-11-24T08:18:48.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.617+0000 7fa641e71640 1 Processor -- start 2023-11-24T08:18:48.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.617+0000 7fa641e71640 1 -- start start 2023-11-24T08:18:48.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.618+0000 7fa641e71640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa63c0b0b40 0x7fa63c0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.618+0000 7fa641e71640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa63c06c5b0 0x7fa63c06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.618+0000 7fa641e71640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63c15ed20 con 0x7fa63c15b260 2023-11-24T08:18:48.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.618+0000 7fa641e71640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63c15b640 con 0x7fa63c0b0b40 2023-11-24T08:18:48.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.618+0000 7fa641e71640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63c0b14f0 con 0x7fa63c06c5b0 2023-11-24T08:18:48.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa63bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa63c0b0b40 0x7fa63c0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa640e6f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa63c15b260 0x7fa63c06e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50854/0 (socket says 172.21.15.107:50854) 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa640e6f640 1 -- 172.21.15.107:0/2317928741 learned_addr learned my addr 172.21.15.107:0/2317928741 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa641670640 1 --2- 172.21.15.107:0/2317928741 >> v2:172.21.15.120:3300/0 conn(0x7fa63c06c5b0 0x7fa63c06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa63bfff640 1 -- 172.21.15.107:0/2317928741 >> v2:172.21.15.120:3300/0 conn(0x7fa63c06c5b0 msgr2=0x7fa63c06ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa63bfff640 1 --2- 172.21.15.107:0/2317928741 >> v2:172.21.15.120:3300/0 conn(0x7fa63c06c5b0 0x7fa63c06ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa63bfff640 1 -- 172.21.15.107:0/2317928741 >> v1:172.21.15.107:6789/0 conn(0x7fa63c15b260 legacy=0x7fa63c06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.619+0000 7fa63bfff640 1 -- 172.21.15.107:0/2317928741 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c15f8a0 con 0x7fa63c0b0b40 2023-11-24T08:18:48.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.620+0000 7fa63bfff640 1 --2- 172.21.15.107:0/2317928741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c0b0fb0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0ccac0 tx=0x7fa62802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcaba6e6e00d1b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa63b7fe640 1 -- 172.21.15.107:0/2317928741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628031520 con 0x7fa63c0b0b40 2023-11-24T08:18:48.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa63b7fe640 1 -- 172.21.15.107:0/2317928741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628031680 con 0x7fa63c0b0b40 2023-11-24T08:18:48.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa63b7fe640 1 -- 172.21.15.107:0/2317928741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628031a10 con 0x7fa63c0b0b40 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 -- 172.21.15.107:0/2317928741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 msgr2=0x7fa63c0b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 --2- 172.21.15.107:0/2317928741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c0b0fb0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0ccac0 tx=0x7fa62802b7c0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 -- 172.21.15.107:0/2317928741 shutdown_connections 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 --2- 172.21.15.107:0/2317928741 >> v2:172.21.15.120:3300/0 conn(0x7fa63c06c5b0 0x7fa63c06ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 --2- 172.21.15.107:0/2317928741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c0b0fb0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 -- 172.21.15.107:0/2317928741 >> 172.21.15.107:0/2317928741 conn(0x7fa63c0cbb40 msgr2=0x7fa63c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 -- 172.21.15.107:0/2317928741 shutdown_connections 2023-11-24T08:18:48.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.621+0000 7fa641e71640 1 -- 172.21.15.107:0/2317928741 wait complete. 2023-11-24T08:18:48.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.623+0000 7fa641e71640 1 Processor -- start 2023-11-24T08:18:48.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.623+0000 7fa641e71640 1 -- start start 2023-11-24T08:18:48.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 0x7fa63c1464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c15b260 0x7fa63c09a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c143780 con 0x7fa63c0b0b40 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c146a10 con 0x7fa63c06c5b0 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.624+0000 7fa641e71640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c143da0 con 0x7fa63c15b260 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55406/0 (socket says 172.21.15.107:55406) 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 -- 172.21.15.107:0/1888057904 learned_addr learned my addr 172.21.15.107:0/1888057904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa640e6f640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 0x7fa63c1464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 -- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c15b260 msgr2=0x7fa63c09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c15b260 0x7fa63c09a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 -- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 msgr2=0x7fa63c1464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 0x7fa63c1464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 -- 172.21.15.107:0/1888057904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa628009400 con 0x7fa63c0b0b40 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa640e6f640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 0x7fa63c1464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.625+0000 7fa63bfff640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa6280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.626+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628033c50 con 0x7fa63c0b0b40 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.626+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628033dd0 con 0x7fa63c0b0b40 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.626+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c144080 con 0x7fa63c0b0b40 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.626+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62803c640 con 0x7fa63c0b0b40 2023-11-24T08:18:48.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.626+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c144640 con 0x7fa63c0b0b40 2023-11-24T08:18:48.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.627+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c04e840 con 0x7fa63c0b0b40 2023-11-24T08:18:48.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.628+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7fa62803a040 con 0x7fa63c0b0b40 2023-11-24T08:18:48.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.629+0000 7fa6397fa640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 0x7fa62006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.629+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6280a9e80 con 0x7fa63c0b0b40 2023-11-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.629+0000 7fa640e6f640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 0x7fa62006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:48.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.630+0000 7fa640e6f640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 0x7fa62006ece0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa630005880 tx=0x7fa630009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.634+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa628076b50 con 0x7fa63c0b0b40 2023-11-24T08:18:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.852+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol2512", "target": ["mon-mgr", ""]}) v1 -- 0x7fa63c15ef70 con 0x7fa62006c820 2023-11-24T08:18:48.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.892+0000 7fa6397fa640 1 -- 172.21.15.107:0/1888057904 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa630001c40 con 0x7fa62006c820 2023-11-24T08:18:48.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 msgr2=0x7fa62006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 0x7fa62006ece0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa630005880 tx=0x7fa630009040 comp rx=0 tx=0).stop 2023-11-24T08:18:48.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 msgr2=0x7fa63c1f2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:48.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa6280046c0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 shutdown_connections 2023-11-24T08:18:48.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa62006c820 0x7fa62006ece0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c15b260 0x7fa63c09a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa63c0b0b40 0x7fa63c1f2d90 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 --2- 172.21.15.107:0/1888057904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c06c5b0 0x7fa63c1464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:48.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 >> 172.21.15.107:0/1888057904 conn(0x7fa63c0cbb40 msgr2=0x7fa63c071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:48.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 shutdown_connections 2023-11-24T08:18:48.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:48.895+0000 7fa641e71640 1 -- 172.21.15.107:0/1888057904 wait complete. 2023-11-24T08:18:48.913 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:48.911+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:48.913 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:48.911+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:48.913 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:48.911+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:48.914 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:48.911+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:48.914 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:18:48.911+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:18:48.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone1612 2023-11-24T08:18:49.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.120+0000 7fac15652640 1 Processor -- start 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.120+0000 7fac15652640 1 -- start start 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.121+0000 7fac15652640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fac1015d7d0 0x7fac100cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.121+0000 7fac15652640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fac100d1f40 0x7fac100ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.121+0000 7fac15652640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac100d15e0 con 0x7fac1015d3f0 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac15652640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac100d0280 con 0x7fac100d1f40 2023-11-24T08:18:49.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac15652640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac100b0eb0 con 0x7fac1015d7d0 2023-11-24T08:18:49.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac14e51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fac100d1f40 0x7fac100ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:49.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac0ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fac1015d3f0 0x7fac100d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50870/0 (socket says 172.21.15.107:50870) 2023-11-24T08:18:49.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac0ffff640 1 -- 172.21.15.107:0/40961749 learned_addr learned my addr 172.21.15.107:0/40961749 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac0f7fe640 1 --2- 172.21.15.107:0/40961749 >> v2:172.21.15.120:3300/0 conn(0x7fac1015d7d0 0x7fac100cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac14e51640 1 -- 172.21.15.107:0/40961749 >> v2:172.21.15.120:3300/0 conn(0x7fac1015d7d0 msgr2=0x7fac100cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac14e51640 1 --2- 172.21.15.107:0/40961749 >> v2:172.21.15.120:3300/0 conn(0x7fac1015d7d0 0x7fac100cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac14e51640 1 -- 172.21.15.107:0/40961749 >> v1:172.21.15.107:6789/0 conn(0x7fac1015d3f0 legacy=0x7fac100d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.122+0000 7fac14e51640 1 -- 172.21.15.107:0/40961749 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac100d1300 con 0x7fac100d1f40 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.124+0000 7fac14e51640 1 --2- 172.21.15.107:0/40961749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac100d1f40 0x7fac100ce280 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fac10067bd0 tx=0x7fac0802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc9e772f9cf7da5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.124+0000 7fac0effd640 1 -- 172.21.15.107:0/40961749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0802cc70 con 0x7fac100d1f40 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.124+0000 7fac0effd640 1 -- 172.21.15.107:0/40961749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0802cdd0 con 0x7fac100d1f40 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.124+0000 7fac0effd640 1 -- 172.21.15.107:0/40961749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac08034660 con 0x7fac100d1f40 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 -- 172.21.15.107:0/40961749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac100d1f40 msgr2=0x7fac100ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 --2- 172.21.15.107:0/40961749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac100d1f40 0x7fac100ce280 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fac10067bd0 tx=0x7fac0802bec0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 -- 172.21.15.107:0/40961749 shutdown_connections 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 --2- 172.21.15.107:0/40961749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac100d1f40 0x7fac100ce280 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 --2- 172.21.15.107:0/40961749 >> v2:172.21.15.120:3300/0 conn(0x7fac1015d7d0 0x7fac100cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 -- 172.21.15.107:0/40961749 >> 172.21.15.107:0/40961749 conn(0x7fac1009c5b0 msgr2=0x7fac100d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:49.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 -- 172.21.15.107:0/40961749 shutdown_connections 2023-11-24T08:18:49.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.125+0000 7fac15652640 1 -- 172.21.15.107:0/40961749 wait complete. 2023-11-24T08:18:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.127+0000 7fac15652640 1 Processor -- start 2023-11-24T08:18:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.127+0000 7fac15652640 1 -- start start 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac100d1f40 0x7fac10159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac1015d7d0 0x7fac100ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac100b3750 con 0x7fac1015d3f0 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1015a0d0 con 0x7fac1015d7d0 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac15652640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1006b440 con 0x7fac100d1f40 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55438/0 (socket says 172.21.15.107:55438) 2023-11-24T08:18:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.128+0000 7fac0f7fe640 1 -- 172.21.15.107:0/1611405555 learned_addr learned my addr 172.21.15.107:0/1611405555 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac14e51640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac1015d7d0 0x7fac100ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 -- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac100d1f40 msgr2=0x7fac10159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac100d1f40 0x7fac10159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 -- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac1015d7d0 msgr2=0x7fac100ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac1015d7d0 0x7fac100ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 -- 172.21.15.107:0/1611405555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08009a70 con 0x7fac1015d3f0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0f7fe640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fac0400d5c0 tx=0x7fac0400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.129+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac040125f0 con 0x7fac1015d3f0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.130+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac04012750 con 0x7fac1015d3f0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.130+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04012b00 con 0x7fac1015d3f0 2023-11-24T08:18:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.130+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1006b720 con 0x7fac1015d3f0 2023-11-24T08:18:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.130+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1006bce0 con 0x7fac1015d3f0 2023-11-24T08:18:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.131+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1004e840 con 0x7fac1015d3f0 2023-11-24T08:18:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.132+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 35) v1 ==== 91577+0+0 (secure 0 0 0) 0x7fac04012c60 con 0x7fac1015d3f0 2023-11-24T08:18:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.133+0000 7fac0cff9640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 0x7fabfc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.133+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac04088990 con 0x7fac1015d3f0 2023-11-24T08:18:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.133+0000 7fac0ffff640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 0x7fabfc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.134+0000 7fac0ffff640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 0x7fabfc06ece0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fac000015f0 tx=0x7fac00007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.138+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac040554a0 con 0x7fac1015d3f0 2023-11-24T08:18:49.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.365+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone1612", "target": ["mon-mgr", ""]}) v1 -- 0x7fac100b27f0 con 0x7fabfc06c820 2023-11-24T08:18:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.402+0000 7fac0cff9640 1 -- 172.21.15.107:0/1611405555 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fac000012b0 con 0x7fabfc06c820 2023-11-24T08:18:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 msgr2=0x7fabfc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 0x7fabfc06ece0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fac000015f0 tx=0x7fac00007a40 comp rx=0 tx=0).stop 2023-11-24T08:18:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 msgr2=0x7fac100b68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fac0400d5c0 tx=0x7fac0400da80 comp rx=0 tx=0).stop 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 shutdown_connections 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fabfc06c820 0x7fabfc06ece0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac1015d7d0 0x7fac100ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac1015d3f0 0x7fac100b68f0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 --2- 172.21.15.107:0/1611405555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac100d1f40 0x7fac10159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 >> 172.21.15.107:0/1611405555 conn(0x7fac1009c5b0 msgr2=0x7fac100cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 shutdown_connections 2023-11-24T08:18:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:49.405+0000 7fac15652640 1 -- 172.21.15.107:0/1611405555 wait complete. 2023-11-24T08:18:49.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:49.429 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:18:49.479 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-11-24T08:18:54.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:54.481 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:18:54.547 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:18:54.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:18:54.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:18:54.796 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:18:54.797 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:54.797 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:54.857 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:54.857 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:54.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:18:55.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.127+0000 7fcec0eee640 1 Processor -- start 2023-11-24T08:18:55.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.127+0000 7fcec0eee640 1 -- start start 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcebc0cc470 0x7fcebc0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0ce640 0x7fcebc0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cf450 0x7fcebc0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc0cfe20 con 0x7fcebc0ce640 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc06c560 con 0x7fcebc0cc470 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fcec0eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0ce270 con 0x7fcebc0cf450 2023-11-24T08:18:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fceb9d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0ce640 0x7fcebc0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fceb9d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcebc0ce640 0x7fcebc0b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55442/0 (socket says 172.21.15.107:55442) 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.129+0000 7fceb9d9b640 1 -- 172.21.15.107:0/3622849547 learned_addr learned my addr 172.21.15.107:0/3622849547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fcebad9d640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cf450 0x7fcebc0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceba59c640 1 --2- 172.21.15.107:0/3622849547 >> v2:172.21.15.120:3300/0 conn(0x7fcebc0cc470 0x7fcebc0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceb9d9b640 1 -- 172.21.15.107:0/3622849547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cf450 msgr2=0x7fcebc0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceb9d9b640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cf450 0x7fcebc0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceb9d9b640 1 -- 172.21.15.107:0/3622849547 >> v2:172.21.15.120:3300/0 conn(0x7fcebc0cc470 msgr2=0x7fcebc0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:55.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceb9d9b640 1 --2- 172.21.15.107:0/3622849547 >> v2:172.21.15.120:3300/0 conn(0x7fcebc0cc470 0x7fcebc0abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.130+0000 7fceb9d9b640 1 -- 172.21.15.107:0/3622849547 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0b4200 con 0x7fcebc0ce640 2023-11-24T08:18:55.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.133+0000 7fceb9d9b640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0ce640 0x7fcebc0b3cc0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0b51d0 tx=0x7fceb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb13872d71d51eeb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.135+0000 7fceb959a640 1 -- 172.21.15.107:0/3622849547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0031870 con 0x7fcebc0ce640 2023-11-24T08:18:55.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.136+0000 7fceb959a640 1 -- 172.21.15.107:0/3622849547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb00319d0 con 0x7fcebc0ce640 2023-11-24T08:18:55.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.136+0000 7fceb959a640 1 -- 172.21.15.107:0/3622849547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0031d60 con 0x7fcebc0ce640 2023-11-24T08:18:55.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 -- 172.21.15.107:0/3622849547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0ce640 msgr2=0x7fcebc0b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0ce640 0x7fcebc0b3cc0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0b51d0 tx=0x7fceb002b910 comp rx=0 tx=0).stop 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 -- 172.21.15.107:0/3622849547 shutdown_connections 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cf450 0x7fcebc0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 --2- 172.21.15.107:0/3622849547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0ce640 0x7fcebc0b3cc0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 --2- 172.21.15.107:0/3622849547 >> v2:172.21.15.120:3300/0 conn(0x7fcebc0cc470 0x7fcebc0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 -- 172.21.15.107:0/3622849547 >> 172.21.15.107:0/3622849547 conn(0x7fcebc06a920 msgr2=0x7fcebc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 -- 172.21.15.107:0/3622849547 shutdown_connections 2023-11-24T08:18:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.137+0000 7fcec0eee640 1 -- 172.21.15.107:0/3622849547 wait complete. 2023-11-24T08:18:55.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.140+0000 7fcec0eee640 1 Processor -- start 2023-11-24T08:18:55.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.140+0000 7fcec0eee640 1 -- start start 2023-11-24T08:18:55.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cc470 0x7fcebc14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0ce640 0x7fcebc1f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc14c120 con 0x7fcebc0cf450 2023-11-24T08:18:55.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1f8170 con 0x7fcebc0ce640 2023-11-24T08:18:55.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcec0eee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1f8450 con 0x7fcebc0cc470 2023-11-24T08:18:55.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcebad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcebad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55452/0 (socket says 172.21.15.107:55452) 2023-11-24T08:18:55.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.141+0000 7fcebad9d640 1 -- 172.21.15.107:0/2057824506 learned_addr learned my addr 172.21.15.107:0/2057824506 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 -- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cc470 msgr2=0x7fcebc14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cc470 0x7fcebc14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 -- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0ce640 msgr2=0x7fcebc1f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0ce640 0x7fcebc1f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 -- 172.21.15.107:0/2057824506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00094d0 con 0x7fcebc0cf450 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcebad9d640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fceb4007cf0 tx=0x7fceb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4014420 con 0x7fcebc0cf450 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb40145a0 con 0x7fcebc0cf450 2023-11-24T08:18:55.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4014950 con 0x7fcebc0cf450 2023-11-24T08:18:55.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.142+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc14c330 con 0x7fcebc0cf450 2023-11-24T08:18:55.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.143+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc0993d0 con 0x7fcebc0cf450 2023-11-24T08:18:55.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.144+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc1f8670 con 0x7fcebc0cf450 2023-11-24T08:18:55.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.145+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fceb4014ab0 con 0x7fcebc0cf450 2023-11-24T08:18:55.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.146+0000 7fceaaffd640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 0x7fce9c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.146+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fceb4088de0 con 0x7fcebc0cf450 2023-11-24T08:18:55.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.146+0000 7fceba59c640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 0x7fce9c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.147+0000 7fceba59c640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 0x7fce9c06ed70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fceac009870 tx=0x7fceac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.151+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fceb4055870 con 0x7fcebc0cf450 2023-11-24T08:18:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.340+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcebc14c850 con 0x7fcebc0cf450 2023-11-24T08:18:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.342+0000 7fceaaffd640 1 -- 172.21.15.107:0/2057824506 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16638 (secure 0 0 0) 0x7fceb4058e90 con 0x7fcebc0cf450 2023-11-24T08:18:55.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:18:55.344 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:18:55.038112+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:18:55.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 msgr2=0x7fce9c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 0x7fce9c06ed70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fceac009870 tx=0x7fceac008040 comp rx=0 tx=0).stop 2023-11-24T08:18:55.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 msgr2=0x7fcebc071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fceb4007cf0 tx=0x7fceb400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 shutdown_connections 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce9c06c8b0 0x7fce9c06ed70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcebc0cf450 0x7fcebc071610 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc0ce640 0x7fcebc1f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 --2- 172.21.15.107:0/2057824506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0cc470 0x7fcebc14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.344+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 >> 172.21.15.107:0/2057824506 conn(0x7fcebc06a920 msgr2=0x7fcebc161550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.345+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 shutdown_connections 2023-11-24T08:18:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.345+0000 7fcec0eee640 1 -- 172.21.15.107:0/2057824506 wait complete. 2023-11-24T08:18:55.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:17:38.499480+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:17:39.719116+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:18:55.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:18:55.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.578+0000 7fe665b41640 1 Processor -- start 2023-11-24T08:18:55.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.578+0000 7fe665b41640 1 -- start start 2023-11-24T08:18:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe665b41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe66015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe665b41640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66015ec90 con 0x7fe6600c49f0 2023-11-24T08:18:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe665b41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6600cd530 con 0x7fe660072160 2023-11-24T08:18:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe665b41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600c4ed0 con 0x7fe66006de10 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe65ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe6600c49f0 0x7fe6600705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50874/0 (socket says 172.21.15.107:50874) 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.580+0000 7fe65ffff640 1 -- 172.21.15.107:0/3062472424 learned_addr learned my addr 172.21.15.107:0/3062472424 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe657fff640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe66015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193932832 0 0) 0x7fe658001490 con 0x7fe6600c49f0 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64c003520 con 0x7fe6600c49f0 2023-11-24T08:18:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109072216 0 0) 0x7fe64c003520 con 0x7fe6600c49f0 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 msgr2=0x7fe66015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe66015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 >> v1:172.21.15.120:6789/0 conn(0x7fe660072160 legacy=0x7fe66006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600c51b0 con 0x7fe6600c49f0 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe658001b80 con 0x7fe6600c49f0 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 >> v1:172.21.15.107:6789/0 conn(0x7fe6600c49f0 legacy=0x7fe6600705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c0039a0 0x7fe64c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0047c0 0x7fe64c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe64c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.581+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600c4ed0 con 0x7fe64c0039a0 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c00ba30 con 0x7fe64c0047c0 2023-11-24T08:18:55.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c00de00 con 0x7fe66006de10 2023-11-24T08:18:55.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c0039a0 0x7fe64c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe657fff640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0047c0 0x7fe64c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65ffff640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe64c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 -- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 msgr2=0x7fe64c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe64c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 -- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0047c0 msgr2=0x7fe64c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0047c0 0x7fe64c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65f7fe640 1 -- 172.21.15.107:0/3062472424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c003520 con 0x7fe64c0039a0 2023-11-24T08:18:55.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.582+0000 7fe65ffff640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66006de10 0x7fe64c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.583+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c0039a0 0x7fe64c003dc0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe650001720 tx=0x7fe64c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd18b4b6eda3520 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6500052d0 con 0x7fe64c0039a0 2023-11-24T08:18:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65002e050 con 0x7fe64c0039a0 2023-11-24T08:18:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe65effd640 1 -- 172.21.15.107:0/3062472424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6500035f0 con 0x7fe64c0039a0 2023-11-24T08:18:55.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 -- 172.21.15.107:0/3062472424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c0039a0 msgr2=0x7fe64c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 --2- 172.21.15.107:0/3062472424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c0039a0 0x7fe64c003dc0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe650001720 tx=0x7fe64c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe65ffff640 1 -- 172.21.15.107:0/3062472424 reap_dead start 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 -- 172.21.15.107:0/3062472424 shutdown_connections 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 -- 172.21.15.107:0/3062472424 >> 172.21.15.107:0/3062472424 conn(0x7fe6600b0c50 msgr2=0x7fe6600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 -- 172.21.15.107:0/3062472424 shutdown_connections 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.584+0000 7fe665b41640 1 -- 172.21.15.107:0/3062472424 wait complete. 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.586+0000 7fe665b41640 1 Processor -- start 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.586+0000 7fe665b41640 1 -- start start 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0039a0 0x7fe6600999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66009bfb0 0x7fe6600ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6601f36d0 con 0x7fe66015cd70 2023-11-24T08:18:55.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66009b600 con 0x7fe64c0039a0 2023-11-24T08:18:55.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe665b41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66009a020 con 0x7fe66009bfb0 2023-11-24T08:18:55.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe657fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe657fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55480/0 (socket says 172.21.15.107:55480) 2023-11-24T08:18:55.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.587+0000 7fe657fff640 1 -- 172.21.15.107:0/1347552702 learned_addr learned my addr 172.21.15.107:0/1347552702 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe65ffff640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66009bfb0 0x7fe6600ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 -- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66009bfb0 msgr2=0x7fe6600ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66009bfb0 0x7fe6600ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 -- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0039a0 msgr2=0x7fe6600999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0039a0 0x7fe6600999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0039a0 0x7fe6600999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 -- 172.21.15.107:0/1347552702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66009a300 con 0x7fe66015cd70 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe657fff640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fe648011600 tx=0x7fe648011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64800f920 con 0x7fe66015cd70 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.588+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64800fa80 con 0x7fe66015cd70 2023-11-24T08:18:55.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.589+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66009a5e0 con 0x7fe66015cd70 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.589+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64800fdf0 con 0x7fe66015cd70 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.589+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66009aba0 con 0x7fe66015cd70 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.590+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66004e8b0 con 0x7fe66015cd70 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.591+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fe648015460 con 0x7fe66015cd70 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.592+0000 7fe65cff9640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 0x7fe64006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.592+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe6480889b0 con 0x7fe66015cd70 2023-11-24T08:18:55.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.592+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 0x7fe64006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:55.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.593+0000 7fe65f7fe640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 0x7fe64006eda0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe650033000 tx=0x7fe650009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:55.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.597+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe648055430 con 0x7fe66015cd70 2023-11-24T08:18:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.786+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe66015dab0 con 0x7fe66015cd70 2023-11-24T08:18:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.787+0000 7fe65cff9640 1 -- 172.21.15.107:0/1347552702 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe648022020 con 0x7fe66015cd70 2023-11-24T08:18:55.788 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:18:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 msgr2=0x7fe64006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 0x7fe64006eda0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe650033000 tx=0x7fe650009030 comp rx=0 tx=0).stop 2023-11-24T08:18:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 msgr2=0x7fe66009b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fe648011600 tx=0x7fe648011ac0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 shutdown_connections 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe64006c8e0 0x7fe64006eda0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe66009bfb0 0x7fe6600ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66015cd70 0x7fe66009b0c0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 --2- 172.21.15.107:0/1347552702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0039a0 0x7fe6600999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.789+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 >> 172.21.15.107:0/1347552702 conn(0x7fe6600b0c50 msgr2=0x7fe660159930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.790+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 shutdown_connections 2023-11-24T08:18:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:55.790+0000 7fe665b41640 1 -- 172.21.15.107:0/1347552702 wait complete. 2023-11-24T08:18:55.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:18:56.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.018+0000 7fed736bb640 1 Processor -- start 2023-11-24T08:18:56.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.018+0000 7fed736bb640 1 -- start start 2023-11-24T08:18:56.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed736bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed736bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c15eea0 con 0x7fed6c0c57e0 2023-11-24T08:18:56.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed736bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c0abf30 con 0x7fed6c15b440 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed736bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c15b820 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed70c56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed70c56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55482/0 (socket says 172.21.15.107:55482) 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.020+0000 7fed70c56640 1 -- 172.21.15.107:0/1651409327 learned_addr learned my addr 172.21.15.107:0/1651409327 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.021+0000 7fed70c56640 1 -- 172.21.15.107:0/1651409327 >> v1:172.21.15.120:6789/0 conn(0x7fed6c15b440 legacy=0x7fed6c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.021+0000 7fed70c56640 1 -- 172.21.15.107:0/1651409327 >> v1:172.21.15.107:6789/0 conn(0x7fed6c0c57e0 legacy=0x7fed6c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.021+0000 7fed70c56640 1 -- 172.21.15.107:0/1651409327 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c15f0f0 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.021+0000 7fed63fff640 1 -- 172.21.15.107:0/1651409327 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606858815 0 0) 0x7fed6c15eea0 con 0x7fed6c0c57e0 2023-11-24T08:18:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.022+0000 7fed70c56640 1 --2- 172.21.15.107:0/1651409327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a744d3dc947132 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.022+0000 7fed63fff640 1 -- 172.21.15.107:0/1651409327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c031870 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.022+0000 7fed63fff640 1 -- 172.21.15.107:0/1651409327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c0319d0 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.022+0000 7fed63fff640 1 -- 172.21.15.107:0/1651409327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c031d60 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 -- 172.21.15.107:0/1651409327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 msgr2=0x7fed6c0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 --2- 172.21.15.107:0/1651409327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c02b910 comp rx=0 tx=0).stop 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 -- 172.21.15.107:0/1651409327 shutdown_connections 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 --2- 172.21.15.107:0/1651409327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c52a0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 -- 172.21.15.107:0/1651409327 >> 172.21.15.107:0/1651409327 conn(0x7fed6c0b0c50 msgr2=0x7fed6c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 -- 172.21.15.107:0/1651409327 shutdown_connections 2023-11-24T08:18:56.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.023+0000 7fed736bb640 1 -- 172.21.15.107:0/1651409327 wait complete. 2023-11-24T08:18:56.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.024+0000 7fed736bb640 1 Processor -- start 2023-11-24T08:18:56.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.025+0000 7fed736bb640 1 -- start start 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.025+0000 7fed736bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 0x7fed6c156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.025+0000 7fed736bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed736bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed6c15b440 0x7fed6c0c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed736bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0af670 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed736bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0af950 con 0x7fed6c0c57e0 2023-11-24T08:18:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed736bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0afc30 con 0x7fed6c15b440 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55486/0 (socket says 172.21.15.107:55486) 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 -- 172.21.15.107:0/102359311 learned_addr learned my addr 172.21.15.107:0/102359311 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed71457640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 0x7fed6c156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 -- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed6c15b440 msgr2=0x7fed6c0c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed71c58640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed6c15b440 0x7fed6c0c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed6c15b440 0x7fed6c0c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 -- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 msgr2=0x7fed6c156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 0x7fed6c156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed70c56640 1 -- 172.21.15.107:0/102359311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0094d0 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.026+0000 7fed71457640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 0x7fed6c156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed70c56640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c004230 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c0043b0 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c004520 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c0aff10 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.027+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c157030 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.028+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c04e8b0 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.029+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fed5c043070 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.030+0000 7fed61ffb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 0x7fed5006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.030+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fed5c02d040 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.030+0000 7fed71457640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 0x7fed5006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.031+0000 7fed71457640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 0x7fed5006eda0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0c4e80 tx=0x7fed68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.034+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed5c077270 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.225+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed6c06dc80 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.226+0000 7fed61ffb640 1 -- 172.21.15.107:0/102359311 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed5c07a890 con 0x7fed6c0cc0f0 2023-11-24T08:18:56.228 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:18:56.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 msgr2=0x7fed5006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 0x7fed5006eda0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0c4e80 tx=0x7fed68008040 comp rx=0 tx=0).stop 2023-11-24T08:18:56.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 msgr2=0x7fed6c0c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 shutdown_connections 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed5006c8e0 0x7fed5006eda0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed6c15b440 0x7fed6c0c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed6c0cc0f0 0x7fed6c0c42f0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 --2- 172.21.15.107:0/102359311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed6c0c57e0 0x7fed6c156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 >> 172.21.15.107:0/102359311 conn(0x7fed6c0b0c50 msgr2=0x7fed6c154900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 shutdown_connections 2023-11-24T08:18:56.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.229+0000 7fed736bb640 1 -- 172.21.15.107:0/102359311 wait complete. 2023-11-24T08:18:56.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:18:56.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.463+0000 7f8304814640 1 Processor -- start 2023-11-24T08:18:56.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.464+0000 7f8304814640 1 -- start start 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f8304814640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 0x7f83000d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f8304814640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83000d15e0 con 0x7f83000d0e40 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f8304814640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83000d0280 con 0x7f830015d7d0 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f8304814640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83000d1300 con 0x7f830015d3f0 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f82ff59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f83000d0e40 0x7f83000b05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50900/0 (socket says 172.21.15.107:50900) 2023-11-24T08:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.465+0000 7f82ff59e640 1 -- 172.21.15.107:0/863534549 learned_addr learned my addr 172.21.15.107:0/863534549 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fed9d640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 0x7f83000d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478817068 0 0) 0x7f83000d15e0 con 0x7f83000d0e40 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e8003520 con 0x7f83000d0e40 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471503595 0 0) 0x7f82e8003520 con 0x7f83000d0e40 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 msgr2=0x7f83000d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 0x7f83000d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 >> v1:172.21.15.120:6789/0 conn(0x7f830015d7d0 legacy=0x7f83000afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83000cfc30 con 0x7f83000d0e40 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f82f80019f0 con 0x7f83000d0e40 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 >> v1:172.21.15.107:6789/0 conn(0x7f83000d0e40 legacy=0x7f83000b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 0x7f82e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.466+0000 7f82fdd9b640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e8003f20 0x7f82e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fdd9b640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8004d00 0x7f82e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83000d1300 con 0x7f82e8003f20 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e800ba60 con 0x7f82e8004d00 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e800bc20 con 0x7f830015d3f0 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e8003f20 0x7f82e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 -- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 msgr2=0x7f82e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f830015d3f0 0x7f82e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 -- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8004d00 msgr2=0x7f82e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8004d00 0x7f82e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.467+0000 7f82fe59c640 1 -- 172.21.15.107:0/863534549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e8003520 con 0x7f82e8003f20 2023-11-24T08:18:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.468+0000 7f82fe59c640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e8003f20 0x7f82e80047c0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001720 tx=0x7f82e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbfb152712a4fc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.468+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f40052d0 con 0x7f82e8003f20 2023-11-24T08:18:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.468+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f402e050 con 0x7f82e8003f20 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.468+0000 7f82fdd9b640 1 -- 172.21.15.107:0/863534549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4003830 con 0x7f82e8003f20 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 -- 172.21.15.107:0/863534549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e8003f20 msgr2=0x7f82e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 --2- 172.21.15.107:0/863534549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82e8003f20 0x7f82e80047c0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001720 tx=0x7f82e80057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f82ff59e640 1 -- 172.21.15.107:0/863534549 reap_dead start 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 -- 172.21.15.107:0/863534549 shutdown_connections 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 -- 172.21.15.107:0/863534549 >> 172.21.15.107:0/863534549 conn(0x7f830009c5b0 msgr2=0x7f83000d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 -- 172.21.15.107:0/863534549 shutdown_connections 2023-11-24T08:18:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.469+0000 7f8304814640 1 -- 172.21.15.107:0/863534549 wait complete. 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.471+0000 7f8304814640 1 Processor -- start 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.471+0000 7f8304814640 1 -- start start 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8003f20 0x7f830014ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83000b3940 0x7f83000b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830014c740 con 0x7f830014bb30 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83000b2970 con 0x7f82e8003f20 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f8304814640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830015b4b0 con 0x7f83000b3940 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f82fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f82fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55502/0 (socket says 172.21.15.107:55502) 2023-11-24T08:18:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.472+0000 7f82fe59c640 1 -- 172.21.15.107:0/1267135138 learned_addr learned my addr 172.21.15.107:0/1267135138 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 -- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83000b3940 msgr2=0x7f83000b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83000b3940 0x7f83000b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 -- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8003f20 msgr2=0x7f830014ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8003f20 0x7f830014ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 -- 172.21.15.107:0/1267135138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830014bf50 con 0x7f830014bb30 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82fe59c640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f82f4008fd0 tx=0x7f82f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f40076c0 con 0x7f830014bb30 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f4007a30 con 0x7f830014bb30 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4006be0 con 0x7f830014bb30 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.473+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83000b2b80 con 0x7f830014bb30 2023-11-24T08:18:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.474+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83000b3140 con 0x7f830014bb30 2023-11-24T08:18:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.475+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830004e840 con 0x7f830014bb30 2023-11-24T08:18:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.476+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f82f4006d40 con 0x7f830014bb30 2023-11-24T08:18:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.476+0000 7f82e77fe640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 0x7f82ec06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.476+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f82f40aa560 con 0x7f830014bb30 2023-11-24T08:18:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.477+0000 7f82fed9d640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 0x7f82ec06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.477+0000 7f82fed9d640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 0x7f82ec06eda0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006a90 tx=0x7f82f0006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.481+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82f40e1e90 con 0x7f830014bb30 2023-11-24T08:18:56.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.657+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f830006c7a0 con 0x7f830014bb30 2023-11-24T08:18:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.658+0000 7f82e77fe640 1 -- 172.21.15.107:0/1267135138 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7f82f4077110 con 0x7f830014bb30 2023-11-24T08:18:56.660 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.660+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 msgr2=0x7f82ec06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.660+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 0x7f82ec06eda0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006a90 tx=0x7f82f0006a20 comp rx=0 tx=0).stop 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.660+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 msgr2=0x7f830014c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.660+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f82f4008fd0 tx=0x7f82f4001120 comp rx=0 tx=0).stop 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 shutdown_connections 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82ec06c8e0 0x7f82ec06eda0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83000b3940 0x7f83000b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f830014bb30 0x7f830014c200 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 --2- 172.21.15.107:0/1267135138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82e8003f20 0x7f830014ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 >> 172.21.15.107:0/1267135138 conn(0x7f830009c5b0 msgr2=0x7f83000cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 shutdown_connections 2023-11-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.661+0000 7f8304814640 1 -- 172.21.15.107:0/1267135138 wait complete. 2023-11-24T08:18:56.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:18:56.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.884+0000 7fd301716640 1 Processor -- start 2023-11-24T08:18:56.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.884+0000 7fd301716640 1 -- start start 2023-11-24T08:18:56.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.886+0000 7fd301716640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2fc06ac10 0x7fd2fc0cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.886+0000 7fd301716640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0cf540 0x7fd2fc161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.886+0000 7fd301716640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0cd400 con 0x7fd2fc15d6e0 2023-11-24T08:18:56.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.886+0000 7fd301716640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc0cd890 con 0x7fd2fc0cf540 2023-11-24T08:18:56.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.886+0000 7fd301716640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc0cfdf0 con 0x7fd2fc06ac10 2023-11-24T08:18:56.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0cf540 0x7fd2fc161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2fc0cf540 0x7fd2fc161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55508/0 (socket says 172.21.15.107:55508) 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 -- 172.21.15.107:0/209701639 learned_addr learned my addr 172.21.15.107:0/209701639 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd2fb7fe640 1 --2- 172.21.15.107:0/209701639 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc06ac10 0x7fd2fc0cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 -- 172.21.15.107:0/209701639 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc06ac10 msgr2=0x7fd2fc0cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 --2- 172.21.15.107:0/209701639 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc06ac10 0x7fd2fc0cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 -- 172.21.15.107:0/209701639 >> v1:172.21.15.120:6789/0 conn(0x7fd2fc15d6e0 legacy=0x7fd2fc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:56.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 -- 172.21.15.107:0/209701639 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0ccad0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.887+0000 7fd300f15640 1 --2- 172.21.15.107:0/209701639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc161130 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0ce7b0 tx=0x7fd2f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7263997a8855ca7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd2faffd640 1 -- 172.21.15.107:0/209701639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4031870 con 0x7fd2fc0cf540 2023-11-24T08:18:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd2faffd640 1 -- 172.21.15.107:0/209701639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f40319d0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd2faffd640 1 -- 172.21.15.107:0/209701639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4031d60 con 0x7fd2fc0cf540 2023-11-24T08:18:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 -- 172.21.15.107:0/209701639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 msgr2=0x7fd2fc161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 --2- 172.21.15.107:0/209701639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc161130 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0ce7b0 tx=0x7fd2f402b910 comp rx=0 tx=0).stop 2023-11-24T08:18:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 -- 172.21.15.107:0/209701639 shutdown_connections 2023-11-24T08:18:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 --2- 172.21.15.107:0/209701639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc161130 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 --2- 172.21.15.107:0/209701639 >> v2:172.21.15.120:3300/0 conn(0x7fd2fc06ac10 0x7fd2fc0cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.888+0000 7fd301716640 1 -- 172.21.15.107:0/209701639 >> 172.21.15.107:0/209701639 conn(0x7fd2fc0b1cd0 msgr2=0x7fd2fc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.889+0000 7fd301716640 1 -- 172.21.15.107:0/209701639 shutdown_connections 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.889+0000 7fd301716640 1 -- 172.21.15.107:0/209701639 wait complete. 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.890+0000 7fd301716640 1 Processor -- start 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.891+0000 7fd301716640 1 -- start start 2023-11-24T08:18:56.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.891+0000 7fd301716640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc06ac10 0x7fd2fc1e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.891+0000 7fd301716640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.891+0000 7fd301716640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d6e0 0x7fd2fc0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.891+0000 7fd301716640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0acdb0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd301716640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0ad090 con 0x7fd2fc15d6e0 2023-11-24T08:18:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd301716640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0ad370 con 0x7fd2fc06ac10 2023-11-24T08:18:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55522/0 (socket says 172.21.15.107:55522) 2023-11-24T08:18:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 -- 172.21.15.107:0/3638332466 learned_addr learned my addr 172.21.15.107:0/3638332466 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 -- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc06ac10 msgr2=0x7fd2fc1e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc06ac10 0x7fd2fc1e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 -- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d6e0 msgr2=0x7fd2fc0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d6e0 0x7fd2fc0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 -- 172.21.15.107:0/3638332466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f40094d0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.892+0000 7fd2fb7fe640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400d5d0 tx=0x7fd2e400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.893+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40125f0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.893+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4012750 con 0x7fd2fc0cf540 2023-11-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.893+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4012ae0 con 0x7fd2fc0cf540 2023-11-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.893+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc15dc40 con 0x7fd2fc0cf540 2023-11-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.893+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc0ae040 con 0x7fd2fc0cf540 2023-11-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.894+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc04e840 con 0x7fd2fc0cf540 2023-11-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.895+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fd2e4012c40 con 0x7fd2fc0cf540 2023-11-24T08:18:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.896+0000 7fd2f8ff9640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 0x7fd2dc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.896+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd2e4088a10 con 0x7fd2fc0cf540 2023-11-24T08:18:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.896+0000 7fd2fbfff640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 0x7fd2dc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.897+0000 7fd2fbfff640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 0x7fd2dc06ed70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec001c60 tx=0x7fd2ec007d10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:56.900+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2e40554a0 con 0x7fd2fc0cf540 2023-11-24T08:18:57.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.106+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd2fc0b2890 con 0x7fd2fc0cf540 2023-11-24T08:18:57.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.108+0000 7fd2f8ff9640 1 -- 172.21.15.107:0/3638332466 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd2e4058ac0 con 0x7fd2fc0cf540 2023-11-24T08:18:57.109 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 msgr2=0x7fd2dc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 0x7fd2dc06ed70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec001c60 tx=0x7fd2ec007d10 comp rx=0 tx=0).stop 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 msgr2=0x7fd2fc1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400d5d0 tx=0x7fd2e400da90 comp rx=0 tx=0).stop 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 shutdown_connections 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd2dc06c8b0 0x7fd2dc06ed70 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2fc15d6e0 0x7fd2fc0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0cf540 0x7fd2fc1599a0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 --2- 172.21.15.107:0/3638332466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2fc06ac10 0x7fd2fc1e7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 >> 172.21.15.107:0/3638332466 conn(0x7fd2fc0b1cd0 msgr2=0x7fd2fc06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 shutdown_connections 2023-11-24T08:18:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.110+0000 7fd301716640 1 -- 172.21.15.107:0/3638332466 wait complete. 2023-11-24T08:18:57.131 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:18:57.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:57.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:18:57.256 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:18:57.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:57.257 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:18:57.317 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:18:57.317 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:57.359 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:18:57.359 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:57.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:18:57.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.663+0000 7f0cfa26b640 1 Processor -- start 2023-11-24T08:18:57.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.664+0000 7f0cfa26b640 1 -- start start 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cfa26b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cf415c7f0 0x7f0cf406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cfa26b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cf450 0x7f0cf40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cfa26b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40cece0 con 0x7f0cf40cdfb0 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cfa26b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf4156010 con 0x7f0cf415c7f0 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cfa26b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf406ce30 con 0x7f0cf40cf450 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.665+0000 7f0cf3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cf415c7f0 0x7f0cf406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0cf415c7f0 0x7f0cf406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55524/0 (socket says 172.21.15.107:55524) 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 -- 172.21.15.107:0/2261625599 learned_addr learned my addr 172.21.15.107:0/2261625599 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 -- 172.21.15.107:0/2261625599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cf450 msgr2=0x7f0cf40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 --2- 172.21.15.107:0/2261625599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cf450 0x7f0cf40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 -- 172.21.15.107:0/2261625599 >> v1:172.21.15.120:6789/0 conn(0x7f0cf40cdfb0 legacy=0x7f0cf40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.666+0000 7f0cf3fff640 1 -- 172.21.15.107:0/2261625599 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf406d040 con 0x7f0cf415c7f0 2023-11-24T08:18:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.667+0000 7f0cf3fff640 1 --2- 172.21.15.107:0/2261625599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf406db20 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62aa7de3c3b57bb6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.667+0000 7f0cf37fe640 1 -- 172.21.15.107:0/2261625599 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4031640 con 0x7f0cf415c7f0 2023-11-24T08:18:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.667+0000 7f0cf37fe640 1 -- 172.21.15.107:0/2261625599 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce40317a0 con 0x7f0cf415c7f0 2023-11-24T08:18:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.667+0000 7f0cf37fe640 1 -- 172.21.15.107:0/2261625599 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4031b30 con 0x7f0cf415c7f0 2023-11-24T08:18:57.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 -- 172.21.15.107:0/2261625599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 msgr2=0x7f0cf406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/2261625599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf406db20 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 -- 172.21.15.107:0/2261625599 shutdown_connections 2023-11-24T08:18:57.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/2261625599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cf450 0x7f0cf40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/2261625599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf406db20 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 -- 172.21.15.107:0/2261625599 >> 172.21.15.107:0/2261625599 conn(0x7f0cf406a920 msgr2=0x7f0cf406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:57.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 -- 172.21.15.107:0/2261625599 shutdown_connections 2023-11-24T08:18:57.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.668+0000 7f0cfa26b640 1 -- 172.21.15.107:0/2261625599 wait complete. 2023-11-24T08:18:57.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.670+0000 7f0cfa26b640 1 Processor -- start 2023-11-24T08:18:57.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.670+0000 7f0cfa26b640 1 -- start start 2023-11-24T08:18:57.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cdfb0 0x7f0cf41e7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf40cf450 0x7f0cf4151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41e7cb0 con 0x7f0cf415c7f0 2023-11-24T08:18:57.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4151010 con 0x7f0cf40cf450 2023-11-24T08:18:57.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cfa26b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41e79d0 con 0x7f0cf40cdfb0 2023-11-24T08:18:57.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.671+0000 7f0cf8808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:57.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55528/0 (socket says 172.21.15.107:55528) 2023-11-24T08:18:57.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 -- 172.21.15.107:0/3996202494 learned_addr learned my addr 172.21.15.107:0/3996202494 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 -- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cdfb0 msgr2=0x7f0cf41e7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cdfb0 0x7f0cf41e7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 -- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf40cf450 msgr2=0x7f0cf4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf40cf450 0x7f0cf4151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 -- 172.21.15.107:0/3996202494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40094d0 con 0x7f0cf415c7f0 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf8808640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c780 tx=0x7f0cec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:57.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec003d80 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec00ce30 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec005290 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.672+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf41512b0 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.673+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf4151870 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.674+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf4151a50 con 0x7f0cf415c7f0 2023-11-24T08:18:57.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.675+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f0cec0053f0 con 0x7f0cf415c7f0 2023-11-24T08:18:57.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.676+0000 7f0cf17fa640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 0x7f0cd406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:57.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.676+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0cec087940 con 0x7f0cf415c7f0 2023-11-24T08:18:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.676+0000 7f0cf3fff640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 0x7f0cd406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.677+0000 7f0cf3fff640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 0x7f0cd406ed70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4036000 tx=0x7f0ce4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:57.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.680+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cec054450 con 0x7f0cf415c7f0 2023-11-24T08:18:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.869+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf4067470 con 0x7f0cf415c7f0 2023-11-24T08:18:57.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.872+0000 7f0cf17fa640 1 -- 172.21.15.107:0/3996202494 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f0cec017020 con 0x7f0cf415c7f0 2023-11-24T08:18:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:18:57.874 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:18:55.038112+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 msgr2=0x7f0cd406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 0x7f0cd406ed70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4036000 tx=0x7f0ce4030000 comp rx=0 tx=0).stop 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 msgr2=0x7f0cf41e53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c780 tx=0x7f0cec00cc40 comp rx=0 tx=0).stop 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 shutdown_connections 2023-11-24T08:18:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0cd406c8b0 0x7f0cd406ed70 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cf415c7f0 0x7f0cf41e53b0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf40cf450 0x7f0cf4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 --2- 172.21.15.107:0/3996202494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cf40cdfb0 0x7f0cf41e7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 >> 172.21.15.107:0/3996202494 conn(0x7f0cf406a920 msgr2=0x7f0cf40af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 shutdown_connections 2023-11-24T08:18:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:57.875+0000 7f0cfa26b640 1 -- 172.21.15.107:0/3996202494 wait complete. 2023-11-24T08:18:57.894 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:17:38.499480+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:17:39.719116+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:18:57.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:18:58.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.110+0000 7fce97d0f640 1 Processor -- start 2023-11-24T08:18:58.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.111+0000 7fce97d0f640 1 -- start start 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce97d0f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce9015d720 0x7fce900d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce97d0f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fce900d3870 0x7fce900b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce97d0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9015db00 con 0x7fce900b1400 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce97d0f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce900d3d20 con 0x7fce9015d720 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce97d0f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce900d0ce0 con 0x7fce900d3870 2023-11-24T08:18:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.112+0000 7fce95aab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce9015d720 0x7fce900d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce9015d720 0x7fce900d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49702/0 (socket says 172.21.15.107:49702) 2023-11-24T08:18:58.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 -- 172.21.15.107:0/924264771 learned_addr learned my addr 172.21.15.107:0/924264771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:58.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce952aa640 1 --2- 172.21.15.107:0/924264771 >> v2:172.21.15.120:3300/0 conn(0x7fce900d3870 0x7fce900b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 -- 172.21.15.107:0/924264771 >> v2:172.21.15.120:3300/0 conn(0x7fce900d3870 msgr2=0x7fce900b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:58.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 --2- 172.21.15.107:0/924264771 >> v2:172.21.15.120:3300/0 conn(0x7fce900d3870 0x7fce900b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 -- 172.21.15.107:0/924264771 >> v1:172.21.15.107:6789/0 conn(0x7fce900b1400 legacy=0x7fce900b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.113+0000 7fce95aab640 1 -- 172.21.15.107:0/924264771 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9006d400 con 0x7fce9015d720 2023-11-24T08:18:58.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.114+0000 7fce95aab640 1 --2- 172.21.15.107:0/924264771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9015d720 0x7fce900d3330 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fce9008d370 tx=0x7fce8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a57470fe698695b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.114+0000 7fce94aa9640 1 -- 172.21.15.107:0/924264771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8402ca80 con 0x7fce9015d720 2023-11-24T08:18:58.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.114+0000 7fce94aa9640 1 -- 172.21.15.107:0/924264771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8402cbe0 con 0x7fce9015d720 2023-11-24T08:18:58.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce94aa9640 1 -- 172.21.15.107:0/924264771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840343f0 con 0x7fce9015d720 2023-11-24T08:18:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 -- 172.21.15.107:0/924264771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9015d720 msgr2=0x7fce900d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 --2- 172.21.15.107:0/924264771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9015d720 0x7fce900d3330 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fce9008d370 tx=0x7fce8402bee0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 -- 172.21.15.107:0/924264771 shutdown_connections 2023-11-24T08:18:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 --2- 172.21.15.107:0/924264771 >> v2:172.21.15.120:3300/0 conn(0x7fce900d3870 0x7fce900b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 --2- 172.21.15.107:0/924264771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9015d720 0x7fce900d3330 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 -- 172.21.15.107:0/924264771 >> 172.21.15.107:0/924264771 conn(0x7fce9008af60 msgr2=0x7fce9009c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 -- 172.21.15.107:0/924264771 shutdown_connections 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.115+0000 7fce97d0f640 1 -- 172.21.15.107:0/924264771 wait complete. 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.117+0000 7fce97d0f640 1 Processor -- start 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.117+0000 7fce97d0f640 1 -- start start 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900d3870 0x7fce900c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9015d720 0x7fce900cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce900b6a30 con 0x7fce900b1400 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce901f75c0 con 0x7fce9015d720 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce97d0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce900ccf50 con 0x7fce900d3870 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce95aab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce95aab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49718/0 (socket says 172.21.15.107:49718) 2023-11-24T08:18:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.118+0000 7fce95aab640 1 -- 172.21.15.107:0/3379630233 learned_addr learned my addr 172.21.15.107:0/3379630233 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce962ac640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9015d720 0x7fce900cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 -- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900d3870 msgr2=0x7fce900c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900d3870 0x7fce900c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 -- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9015d720 msgr2=0x7fce900cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9015d720 0x7fce900cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 -- 172.21.15.107:0/3379630233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84009b30 con 0x7fce900b1400 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.119+0000 7fce95aab640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7fce84000f80 tx=0x7fce84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.120+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84009510 con 0x7fce900b1400 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.120+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8403e450 con 0x7fce900b1400 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.120+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8403e800 con 0x7fce900b1400 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.120+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce900b6090 con 0x7fce900b1400 2023-11-24T08:18:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.120+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce900b6650 con 0x7fce900b1400 2023-11-24T08:18:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.121+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce900b3be0 con 0x7fce900b1400 2023-11-24T08:18:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.122+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fce8402cd50 con 0x7fce900b1400 2023-11-24T08:18:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.123+0000 7fce8e7fc640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 0x7fce7406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.123+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fce840aaf80 con 0x7fce900b1400 2023-11-24T08:18:58.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.123+0000 7fce952aa640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 0x7fce7406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.124+0000 7fce952aa640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 0x7fce7406ed70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fce900c79d0 tx=0x7fce7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.127+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce84077b30 con 0x7fce900b1400 2023-11-24T08:18:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.315+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce9004e8b0 con 0x7fce900b1400 2023-11-24T08:18:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.316+0000 7fce8e7fc640 1 -- 172.21.15.107:0/3379630233 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce8407b150 con 0x7fce900b1400 2023-11-24T08:18:58.317 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.318+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 msgr2=0x7fce7406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 0x7fce7406ed70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fce900c79d0 tx=0x7fce7c008040 comp rx=0 tx=0).stop 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 msgr2=0x7fce900b8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7fce84000f80 tx=0x7fce84004a40 comp rx=0 tx=0).stop 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 shutdown_connections 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fce7406c8b0 0x7fce7406ed70 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9015d720 0x7fce900cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce900d3870 0x7fce900c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 --2- 172.21.15.107:0/3379630233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce900b1400 0x7fce900b8700 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 >> 172.21.15.107:0/3379630233 conn(0x7fce9008af60 msgr2=0x7fce900b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 shutdown_connections 2023-11-24T08:18:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.319+0000 7fce97d0f640 1 -- 172.21.15.107:0/3379630233 wait complete. 2023-11-24T08:18:58.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:18:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.552+0000 7ffb9e701640 1 Processor -- start 2023-11-24T08:18:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.553+0000 7ffb9e701640 1 -- start start 2023-11-24T08:18:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 0x7ffb980cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980cbf20 0x7ffb980c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb980c57e0 0x7ffb980c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb980cc3d0 con 0x7ffb980c57e0 2023-11-24T08:18:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9815b820 con 0x7ffb9815b440 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.554+0000 7ffb9e701640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9815eea0 con 0x7ffb980cbf20 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb980c57e0 0x7ffb980c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb980c57e0 0x7ffb980c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49726/0 (socket says 172.21.15.107:49726) 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 -- 172.21.15.107:0/608872606 learned_addr learned my addr 172.21.15.107:0/608872606 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb977fe640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980cbf20 0x7ffb980c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb97fff640 1 --2- 172.21.15.107:0/608872606 >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 0x7ffb980cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 -- 172.21.15.107:0/608872606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980cbf20 msgr2=0x7ffb980c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980cbf20 0x7ffb980c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 -- 172.21.15.107:0/608872606 >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 msgr2=0x7ffb980cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 --2- 172.21.15.107:0/608872606 >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 0x7ffb980cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb9cc9e640 1 -- 172.21.15.107:0/608872606 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb980abf30 con 0x7ffb980c57e0 2023-11-24T08:18:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.555+0000 7ffb97fff640 1 --2- 172.21.15.107:0/608872606 >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 0x7ffb980cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.556+0000 7ffb9cc9e640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980c57e0 0x7ffb980c3ec0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7ffb9806e770 tx=0x7ffb9002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368c7999a6d5afc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.556+0000 7ffb96ffd640 1 -- 172.21.15.107:0/608872606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb900317d0 con 0x7ffb980c57e0 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb96ffd640 1 -- 172.21.15.107:0/608872606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90031930 con 0x7ffb980c57e0 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb96ffd640 1 -- 172.21.15.107:0/608872606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90031cc0 con 0x7ffb980c57e0 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 -- 172.21.15.107:0/608872606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980c57e0 msgr2=0x7ffb980c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980c57e0 0x7ffb980c3ec0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7ffb9806e770 tx=0x7ffb9002b870 comp rx=0 tx=0).stop 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 -- 172.21.15.107:0/608872606 shutdown_connections 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980c57e0 0x7ffb980c3ec0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 --2- 172.21.15.107:0/608872606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980cbf20 0x7ffb980c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 --2- 172.21.15.107:0/608872606 >> v2:172.21.15.120:3300/0 conn(0x7ffb9815b440 0x7ffb980cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 -- 172.21.15.107:0/608872606 >> 172.21.15.107:0/608872606 conn(0x7ffb980b0c50 msgr2=0x7ffb980b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 -- 172.21.15.107:0/608872606 shutdown_connections 2023-11-24T08:18:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.557+0000 7ffb9e701640 1 -- 172.21.15.107:0/608872606 wait complete. 2023-11-24T08:18:58.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.559+0000 7ffb9e701640 1 Processor -- start 2023-11-24T08:18:58.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.559+0000 7ffb9e701640 1 -- start start 2023-11-24T08:18:58.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980c57e0 0x7ffb98159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb9815b440 0x7ffb980ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb98159f60 con 0x7ffb980cbf20 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980cd7e0 con 0x7ffb9815b440 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb9e701640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980ccbd0 con 0x7ffb980c57e0 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.560+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49732/0 (socket says 172.21.15.107:49732) 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 -- 172.21.15.107:0/260165246 learned_addr learned my addr 172.21.15.107:0/260165246 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 -- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980c57e0 msgr2=0x7ffb98159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980c57e0 0x7ffb98159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 -- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb9815b440 msgr2=0x7ffb980ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb9815b440 0x7ffb980ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 -- 172.21.15.107:0/260165246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb900094d0 con 0x7ffb980cbf20 2023-11-24T08:18:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb977fe640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000c7f0 tx=0x7ffb8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80004220 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb80004380 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80007860 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.561+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb980cceb0 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.562+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb980cd470 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.563+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9804e8b0 con 0x7ffb980cbf20 2023-11-24T08:18:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.564+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7ffb80007bd0 con 0x7ffb980cbf20 2023-11-24T08:18:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.565+0000 7ffb94ff9640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 0x7ffb7806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.565+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ffb800884d0 con 0x7ffb980cbf20 2023-11-24T08:18:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.565+0000 7ffb97fff640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 0x7ffb7806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.566+0000 7ffb97fff640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 0x7ffb7806eda0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffb88008e00 tx=0x7ffb88008330 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.569+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb80054f50 con 0x7ffb980cbf20 2023-11-24T08:18:58.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.759+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb980cd920 con 0x7ffb980cbf20 2023-11-24T08:18:58.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.760+0000 7ffb94ff9640 1 -- 172.21.15.107:0/260165246 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb980cd920 con 0x7ffb980cbf20 2023-11-24T08:18:58.761 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 msgr2=0x7ffb7806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 0x7ffb7806eda0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffb88008e00 tx=0x7ffb88008330 comp rx=0 tx=0).stop 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 msgr2=0x7ffb980cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000c7f0 tx=0x7ffb8000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 shutdown_connections 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffb7806c8e0 0x7ffb7806eda0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb9815b440 0x7ffb980ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb980cbf20 0x7ffb980cdd00 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 --2- 172.21.15.107:0/260165246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb980c57e0 0x7ffb98159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.763+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 >> 172.21.15.107:0/260165246 conn(0x7ffb980b0c50 msgr2=0x7ffb980719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:58.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.764+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 shutdown_connections 2023-11-24T08:18:58.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.764+0000 7ffb9e701640 1 -- 172.21.15.107:0/260165246 wait complete. 2023-11-24T08:18:58.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:18:58.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.989+0000 7fd171e09640 1 Processor -- start 2023-11-24T08:18:58.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.989+0000 7fd171e09640 1 -- start start 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd171e09640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd16c06de30 0x7fd16c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd171e09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd16c0b4730 0x7fd16c0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd171e09640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0ccdf0 con 0x7fd16c0b4bc0 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd171e09640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c06e210 con 0x7fd16c0b4730 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd171e09640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c0cd0d0 con 0x7fd16c06de30 2023-11-24T08:18:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd16bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd16c0b4730 0x7fd16c0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.991+0000 7fd16bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd16c0b4730 0x7fd16c0b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49738/0 (socket says 172.21.15.107:49738) 2023-11-24T08:18:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd16bfff640 1 -- 172.21.15.107:0/4065774534 learned_addr learned my addr 172.21.15.107:0/4065774534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd170e07640 1 --2- 172.21.15.107:0/4065774534 >> v2:172.21.15.120:3300/0 conn(0x7fd16c06de30 0x7fd16c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd16bfff640 1 -- 172.21.15.107:0/4065774534 >> v2:172.21.15.120:3300/0 conn(0x7fd16c06de30 msgr2=0x7fd16c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd16bfff640 1 --2- 172.21.15.107:0/4065774534 >> v2:172.21.15.120:3300/0 conn(0x7fd16c06de30 0x7fd16c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd16bfff640 1 -- 172.21.15.107:0/4065774534 >> v1:172.21.15.120:6789/0 conn(0x7fd16c0b4bc0 legacy=0x7fd16c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:58.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.992+0000 7fd16bfff640 1 -- 172.21.15.107:0/4065774534 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0cd3b0 con 0x7fd16c0b4730 2023-11-24T08:18:58.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.993+0000 7fd16bfff640 1 --2- 172.21.15.107:0/4065774534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4730 0x7fd16c0b1de0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fd16c068740 tx=0x7fd15802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2ff6763cfe5bde server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:58.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.994+0000 7fd16b7fe640 1 -- 172.21.15.107:0/4065774534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158031870 con 0x7fd16c0b4730 2023-11-24T08:18:58.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.994+0000 7fd16b7fe640 1 -- 172.21.15.107:0/4065774534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580319d0 con 0x7fd16c0b4730 2023-11-24T08:18:58.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.994+0000 7fd16b7fe640 1 -- 172.21.15.107:0/4065774534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158031d60 con 0x7fd16c0b4730 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 -- 172.21.15.107:0/4065774534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4730 msgr2=0x7fd16c0b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 --2- 172.21.15.107:0/4065774534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4730 0x7fd16c0b1de0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fd16c068740 tx=0x7fd15802b910 comp rx=0 tx=0).stop 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 -- 172.21.15.107:0/4065774534 shutdown_connections 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 --2- 172.21.15.107:0/4065774534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4730 0x7fd16c0b1de0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 --2- 172.21.15.107:0/4065774534 >> v2:172.21.15.120:3300/0 conn(0x7fd16c06de30 0x7fd16c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 -- 172.21.15.107:0/4065774534 >> 172.21.15.107:0/4065774534 conn(0x7fd16c0ced50 msgr2=0x7fd16c0cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 -- 172.21.15.107:0/4065774534 shutdown_connections 2023-11-24T08:18:58.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.995+0000 7fd171e09640 1 -- 172.21.15.107:0/4065774534 wait complete. 2023-11-24T08:18:58.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.997+0000 7fd171e09640 1 Processor -- start 2023-11-24T08:18:58.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.997+0000 7fd171e09640 1 -- start start 2023-11-24T08:18:59.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 0x7fd16c070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd16c0b4730 0x7fd16c13f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c13f830 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c071070 con 0x7fd16c06de30 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171e09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c071350 con 0x7fd16c0b4730 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171608640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171608640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49752/0 (socket says 172.21.15.107:49752) 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.998+0000 7fd171608640 1 -- 172.21.15.107:0/3153826947 learned_addr learned my addr 172.21.15.107:0/3153826947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd16bfff640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd16c0b4730 0x7fd16c13f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd170e07640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 0x7fd16c070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 -- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd16c0b4730 msgr2=0x7fd16c13f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd16c0b4730 0x7fd16c13f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 -- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 msgr2=0x7fd16c070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 0x7fd16c070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 -- 172.21.15.107:0/3153826947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580094d0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd170e07640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 0x7fd16c070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171608640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fd16400b9e0 tx=0x7fd16400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164003b70 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164003cd0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:58.999+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c071630 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.000+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16400cae0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.000+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c071bf0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.001+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c071de0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.002+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fd16400cc40 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.002+0000 7fd1697fa640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 0x7fd15006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.003+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd1640882c0 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.003+0000 7fd170e07640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 0x7fd15006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.003+0000 7fd170e07640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 0x7fd15006ed70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd160006fd0 tx=0x7fd160008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:59.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.007+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd164093080 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.196+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd16c04e840 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.197+0000 7fd1697fa640 1 -- 172.21.15.107:0/3153826947 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd164054d40 con 0x7fd16c0b4bc0 2023-11-24T08:18:59.198 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 msgr2=0x7fd15006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 0x7fd15006ed70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd160006fd0 tx=0x7fd160008040 comp rx=0 tx=0).stop 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 msgr2=0x7fd16c153b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fd16400b9e0 tx=0x7fd16400bea0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 shutdown_connections 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd15006c8b0 0x7fd15006ed70 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd16c0b4bc0 0x7fd16c153b40 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd16c0b4730 0x7fd16c13f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 --2- 172.21.15.107:0/3153826947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd16c06de30 0x7fd16c070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 >> 172.21.15.107:0/3153826947 conn(0x7fd16c0ced50 msgr2=0x7fd16c15e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:59.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 shutdown_connections 2023-11-24T08:18:59.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.200+0000 7fd171e09640 1 -- 172.21.15.107:0/3153826947 wait complete. 2023-11-24T08:18:59.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:18:59.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.425+0000 7f983c5c4640 1 Processor -- start 2023-11-24T08:18:59.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.425+0000 7f983c5c4640 1 -- start start 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983c5c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f98340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983c5c4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f98340cbf20 0x7f98340c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983c5c4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340cc3a0 con 0x7f98340c57e0 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983c5c4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983415b820 con 0x7f98340cbf20 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983c5c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983415eea0 con 0x7f983415b440 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983ab61640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98340c57e0 0x7f98340c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51336/0 (socket says 172.21.15.107:51336) 2023-11-24T08:18:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.427+0000 7f983ab61640 1 -- 172.21.15.107:0/2010390012 learned_addr learned my addr 172.21.15.107:0/2010390012 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f9839b5f640 1 --2- 172.21.15.107:0/2010390012 >> v2:172.21.15.120:3300/0 conn(0x7f98340cbf20 0x7f98340c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983a360640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f98340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231288303 0 0) 0x7f982c001490 con 0x7f98340c57e0 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003520 con 0x7f98340c57e0 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797292042 0 0) 0x7f9820003520 con 0x7f98340c57e0 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 msgr2=0x7f98340cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:59.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f98340cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 >> v2:172.21.15.120:3300/0 conn(0x7f98340cbf20 msgr2=0x7f98340c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 --2- 172.21.15.107:0/2010390012 >> v2:172.21.15.120:3300/0 conn(0x7f98340cbf20 0x7f98340c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983406d600 con 0x7f98340c57e0 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f982c001b80 con 0x7f98340c57e0 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 >> v1:172.21.15.107:6789/0 conn(0x7f98340c57e0 legacy=0x7f98340c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f9820003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820003ef0 0x7f98200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9820004d00 0x7f9820005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983415b820 con 0x7f9820004d00 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983415eea0 con 0x7f9820003ef0 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982000ba80 con 0x7f983415b440 2023-11-24T08:18:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.428+0000 7f983ab61640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9820004d00 0x7f9820005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983a360640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f9820003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983ab61640 1 -- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 msgr2=0x7f9820003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983ab61640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f983415b440 0x7f9820003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983ab61640 1 -- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820003ef0 msgr2=0x7f98200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983ab61640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9820003ef0 0x7f98200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.429+0000 7f983ab61640 1 -- 172.21.15.107:0/2010390012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820003520 con 0x7f9820004d00 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.430+0000 7f983ab61640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9820004d00 0x7f9820005180 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f982c00a870 tx=0x7f98200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadaf1ff6d237565 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.430+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c034630 con 0x7f9820004d00 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.430+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c02f070 con 0x7f9820004d00 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.430+0000 7f983935e640 1 -- 172.21.15.107:0/2010390012 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c034a30 con 0x7f9820004d00 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 -- 172.21.15.107:0/2010390012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9820004d00 msgr2=0x7f9820005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 --2- 172.21.15.107:0/2010390012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9820004d00 0x7f9820005180 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f982c00a870 tx=0x7f98200057f0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983ab61640 1 -- 172.21.15.107:0/2010390012 reap_dead start 2023-11-24T08:18:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 -- 172.21.15.107:0/2010390012 shutdown_connections 2023-11-24T08:18:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 -- 172.21.15.107:0/2010390012 >> 172.21.15.107:0/2010390012 conn(0x7f98340b0c50 msgr2=0x7f98340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 -- 172.21.15.107:0/2010390012 shutdown_connections 2023-11-24T08:18:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.431+0000 7f983c5c4640 1 -- 172.21.15.107:0/2010390012 wait complete. 2023-11-24T08:18:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.433+0000 7f983c5c4640 1 Processor -- start 2023-11-24T08:18:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.433+0000 7f983c5c4640 1 -- start start 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9820004d00 0x7f9834152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 0x7f98340b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98340c8fb0 con 0x7f98340c6d50 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98340b3f40 con 0x7f98340c71b0 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f983c5c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98340c8cd0 con 0x7f9820004d00 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f9839b5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f9839b5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49774/0 (socket says 172.21.15.107:49774) 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.434+0000 7f9839b5f640 1 -- 172.21.15.107:0/3051266932 learned_addr learned my addr 172.21.15.107:0/3051266932 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f983a360640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9820004d00 0x7f9834152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f983ab61640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 0x7f98340b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 -- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9820004d00 msgr2=0x7f9834152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9820004d00 0x7f9834152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 -- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 msgr2=0x7f98340b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 0x7f98340b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 -- 172.21.15.107:0/3051266932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340c6270 con 0x7f98340c6d50 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f983ab61640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 0x7f98340b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:18:59.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f9839b5f640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f982400e7c0 tx=0x7f982400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400eed0 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982400c940 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98340c6550 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400ccf0 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.435+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9834153010 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.436+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98340c60b0 con 0x7f98340c6d50 2023-11-24T08:18:59.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.437+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f9824013470 con 0x7f98340c6d50 2023-11-24T08:18:59.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.439+0000 7f982affd640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 0x7f981806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:18:59.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.439+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9824088600 con 0x7f98340c6d50 2023-11-24T08:18:59.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.439+0000 7f983a360640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 0x7f981806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:18:59.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.440+0000 7f983a360640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 0x7f981806eda0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9830006fd0 tx=0x7f9830008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:18:59.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.443+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9824055110 con 0x7f98340c6d50 2023-11-24T08:18:59.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.647+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f983404e8b0 con 0x7f98340c6d50 2023-11-24T08:18:59.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.648+0000 7f982affd640 1 -- 172.21.15.107:0/3051266932 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9824017080 con 0x7f98340c6d50 2023-11-24T08:18:59.649 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:18:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.650+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 msgr2=0x7f981806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.650+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 0x7f981806eda0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9830006fd0 tx=0x7f9830008040 comp rx=0 tx=0).stop 2023-11-24T08:18:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.650+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 msgr2=0x7f98340afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:18:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.650+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f982400e7c0 tx=0x7f982400ec80 comp rx=0 tx=0).stop 2023-11-24T08:18:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.650+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 shutdown_connections 2023-11-24T08:18:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f981806c8e0 0x7f981806eda0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98340c71b0 0x7f98340b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98340c6d50 0x7f98340afc20 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 --2- 172.21.15.107:0/3051266932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9820004d00 0x7f9834152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:18:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 >> 172.21.15.107:0/3051266932 conn(0x7f98340b0c50 msgr2=0x7f9834071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:18:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 shutdown_connections 2023-11-24T08:18:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:18:59.651+0000 7f983c5c4640 1 -- 172.21.15.107:0/3051266932 wait complete. 2023-11-24T08:18:59.671 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:18:59.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:59.671 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:18:59.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:18:59.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:18:59.789 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:18:59.845 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:18:59.845 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:59.906 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:18:59.906 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:18:59.906 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:18:59.966 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:18:59.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.189+0000 7f7e81615640 1 Processor -- start 2023-11-24T08:19:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.189+0000 7f7e81615640 1 -- start start 2023-11-24T08:19:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 0x7f7e7c0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e7c06ed10 0x7f7e7c0d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c15eac0 con 0x7f7e7c06ed10 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c15deb0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.191+0000 7f7e81615640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c15e190 con 0x7f7e7c15d9b0 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e7c06ed10 0x7f7e7c0d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e7c06ed10 0x7f7e7c0d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49776/0 (socket says 172.21.15.107:49776) 2023-11-24T08:19:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 -- 172.21.15.107:0/974325335 learned_addr learned my addr 172.21.15.107:0/974325335 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 -- 172.21.15.107:0/974325335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 msgr2=0x7f7e7c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7affd640 1 --2- 172.21.15.107:0/974325335 >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 0x7f7e7c0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7a7fc640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 -- 172.21.15.107:0/974325335 >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 msgr2=0x7f7e7c0c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 --2- 172.21.15.107:0/974325335 >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 0x7f7e7c0c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7b7fe640 1 -- 172.21.15.107:0/974325335 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c15e470 con 0x7f7e7c06ed10 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.192+0000 7f7e7affd640 1 --2- 172.21.15.107:0/974325335 >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 0x7f7e7c0c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.193+0000 7f7e7b7fe640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0d0580 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c061ed0 tx=0x7f7e7402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab7f966bdcede5e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e79ffb640 1 -- 172.21.15.107:0/974325335 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74031750 con 0x7f7e7c06ed10 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e79ffb640 1 -- 172.21.15.107:0/974325335 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e740318b0 con 0x7f7e7c06ed10 2023-11-24T08:19:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e79ffb640 1 -- 172.21.15.107:0/974325335 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74031c40 con 0x7f7e7c06ed10 2023-11-24T08:19:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 -- 172.21.15.107:0/974325335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c06ed10 msgr2=0x7f7e7c0d0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0d0580 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c061ed0 tx=0x7f7e7402b7f0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 -- 172.21.15.107:0/974325335 shutdown_connections 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0d0580 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 --2- 172.21.15.107:0/974325335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 --2- 172.21.15.107:0/974325335 >> v2:172.21.15.120:3300/0 conn(0x7f7e7c15d5d0 0x7f7e7c0c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 -- 172.21.15.107:0/974325335 >> 172.21.15.107:0/974325335 conn(0x7f7e7c0d12d0 msgr2=0x7f7e7c0d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.194+0000 7f7e81615640 1 -- 172.21.15.107:0/974325335 shutdown_connections 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.195+0000 7f7e81615640 1 -- 172.21.15.107:0/974325335 wait complete. 2023-11-24T08:19:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.196+0000 7f7e81615640 1 Processor -- start 2023-11-24T08:19:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.196+0000 7f7e81615640 1 -- start start 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1465f0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c157030 con 0x7f7e7c06ed10 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.197+0000 7f7e81615640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c157310 con 0x7f7e7c15d9b0 2023-11-24T08:19:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49794/0 (socket says 172.21.15.107:49794) 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 -- 172.21.15.107:0/1927220709 learned_addr learned my addr 172.21.15.107:0/1927220709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7affd640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7b7fe640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 -- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 msgr2=0x7f7e7c1f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c1f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 -- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e7c06ed10 msgr2=0x7f7e7c0cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 -- 172.21.15.107:0/1927220709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740094d0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7b7fe640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.198+0000 7f7e7a7fc640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c7f0 tx=0x7f7e6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.199+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64004220 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.199+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e640043a0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.199+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64007860 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.199+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c15af30 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.199+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c0cb0b0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.200+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c067b80 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.201+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f7e64007bd0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.202+0000 7f7e737fe640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 0x7f7e5c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.202+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7e640883f0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.202+0000 7f7e7affd640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 0x7f7e5c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.203+0000 7f7e7affd640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 0x7f7e5c06ed70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c007fc0 tx=0x7f7e6c007550 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.207+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e64054fa0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.434+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7c06f2c0 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.436+0000 7f7e737fe640 1 -- 172.21.15.107:0/1927220709 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f7e6401e070 con 0x7f7e7c15d5d0 2023-11-24T08:19:00.437 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:00.437 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":98,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:19:00.437 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 98 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.438+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 msgr2=0x7f7e5c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.438+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 0x7f7e5c06ed70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c007fc0 tx=0x7f7e6c007550 comp rx=0 tx=0).stop 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.438+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 msgr2=0x7f7e7c15b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.438+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c7f0 tx=0x7f7e6400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 shutdown_connections 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7e5c06c8b0 0x7f7e5c06ed70 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e7c15d9b0 0x7f7e7c1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e7c15d5d0 0x7f7e7c15b730 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 --2- 172.21.15.107:0/1927220709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e7c06ed10 0x7f7e7c0cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 >> 172.21.15.107:0/1927220709 conn(0x7f7e7c0d12d0 msgr2=0x7f7e7c06b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 shutdown_connections 2023-11-24T08:19:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.439+0000 7f7e81615640 1 -- 172.21.15.107:0/1927220709 wait complete. 2023-11-24T08:19:00.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:00.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.663+0000 7f48feaad640 1 Processor -- start 2023-11-24T08:19:00.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.663+0000 7f48feaad640 1 -- start start 2023-11-24T08:19:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.664+0000 7f48feaad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48f80b2250 0x7f48f815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.664+0000 7f48feaad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48f80b2630 0x7f48f80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.664+0000 7f48feaad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f815ab80 con 0x7f48f80b2aa0 2023-11-24T08:19:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.664+0000 7f48feaad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f815a990 con 0x7f48f80b2630 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.664+0000 7f48feaad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f815e680 con 0x7f48f80b2250 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48f80b2630 0x7f48f80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48f80b2630 0x7f48f80ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49796/0 (socket says 172.21.15.107:49796) 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 -- 172.21.15.107:0/1326833507 learned_addr learned my addr 172.21.15.107:0/1326833507 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48fd04a640 1 --1- 172.21.15.107:0/1326833507 >> v1:172.21.15.107:6789/0 conn(0x7f48f80b2aa0 0x7f48f815b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51346/0 (socket says 172.21.15.107:51346) 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48fc849640 1 --2- 172.21.15.107:0/1326833507 >> v2:172.21.15.120:3300/0 conn(0x7f48f80b2250 0x7f48f815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 -- 172.21.15.107:0/1326833507 >> v2:172.21.15.120:3300/0 conn(0x7f48f80b2250 msgr2=0x7f48f815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 --2- 172.21.15.107:0/1326833507 >> v2:172.21.15.120:3300/0 conn(0x7f48f80b2250 0x7f48f815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 -- 172.21.15.107:0/1326833507 >> v1:172.21.15.107:6789/0 conn(0x7f48f80b2aa0 legacy=0x7f48f815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.665+0000 7f48f7fff640 1 -- 172.21.15.107:0/1326833507 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f815e960 con 0x7f48f80b2630 2023-11-24T08:19:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48f7fff640 1 --2- 172.21.15.107:0/1326833507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f80ccf30 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f48f8072a70 tx=0x7f48e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bb8d4983841aac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48f77fe640 1 -- 172.21.15.107:0/1326833507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4030860 con 0x7f48f80b2630 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48f77fe640 1 -- 172.21.15.107:0/1326833507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e40309c0 con 0x7f48f80b2630 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48f77fe640 1 -- 172.21.15.107:0/1326833507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4030d50 con 0x7f48f80b2630 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48feaad640 1 -- 172.21.15.107:0/1326833507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 msgr2=0x7f48f80ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48feaad640 1 --2- 172.21.15.107:0/1326833507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f80ccf30 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f48f8072a70 tx=0x7f48e402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48feaad640 1 -- 172.21.15.107:0/1326833507 shutdown_connections 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.667+0000 7f48feaad640 1 --2- 172.21.15.107:0/1326833507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f80ccf30 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.668+0000 7f48feaad640 1 --2- 172.21.15.107:0/1326833507 >> v2:172.21.15.120:3300/0 conn(0x7f48f80b2250 0x7f48f815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.668+0000 7f48feaad640 1 -- 172.21.15.107:0/1326833507 >> 172.21.15.107:0/1326833507 conn(0x7f48f806a920 msgr2=0x7f48f806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.668+0000 7f48feaad640 1 -- 172.21.15.107:0/1326833507 shutdown_connections 2023-11-24T08:19:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.668+0000 7f48feaad640 1 -- 172.21.15.107:0/1326833507 wait complete. 2023-11-24T08:19:00.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.669+0000 7f48feaad640 1 Processor -- start 2023-11-24T08:19:00.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.669+0000 7f48feaad640 1 -- start start 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48f80b2250 0x7f48f8167ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48f80b2aa0 0x7f48f80aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f814da20 con 0x7f48f80b2630 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f814dcc0 con 0x7f48f80b2250 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48feaad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f814dfa0 con 0x7f48f80b2aa0 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.670+0000 7f48f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49800/0 (socket says 172.21.15.107:49800) 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 -- 172.21.15.107:0/3817955923 learned_addr learned my addr 172.21.15.107:0/3817955923 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 -- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48f80b2aa0 msgr2=0x7f48f80aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48f80b2aa0 0x7f48f80aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 -- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48f80b2250 msgr2=0x7f48f8167ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48f80b2250 0x7f48f8167ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 -- 172.21.15.107:0/3817955923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e40094d0 con 0x7f48f80b2630 2023-11-24T08:19:00.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f7fff640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f48e402b9e0 tx=0x7f48e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4039d60 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e4043400 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e40437b0 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.671+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f814e280 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.672+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f815d8c0 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.673+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f8153e70 con 0x7f48f80b2630 2023-11-24T08:19:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.673+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f48e4043910 con 0x7f48f80b2630 2023-11-24T08:19:00.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.674+0000 7f48f57fa640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 0x7f48dc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:00.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.674+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f48e403e070 con 0x7f48f80b2630 2023-11-24T08:19:00.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.675+0000 7f48fc849640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 0x7f48dc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:00.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.675+0000 7f48fc849640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 0x7f48dc06ed70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0098f0 tx=0x7f48ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:00.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.679+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48f8153e70 con 0x7f48f80b2630 2023-11-24T08:19:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.870+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48f8153e70 con 0x7f48f80b2630 2023-11-24T08:19:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.873+0000 7f48f57fa640 1 -- 172.21.15.107:0/3817955923 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f48f8153e70 con 0x7f48f80b2630 2023-11-24T08:19:00.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:00.876 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:18:55.038112+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 msgr2=0x7f48dc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 0x7f48dc06ed70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0098f0 tx=0x7f48ec008040 comp rx=0 tx=0).stop 2023-11-24T08:19:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 msgr2=0x7f48f815d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f48e402b9e0 tx=0x7f48e4004730 comp rx=0 tx=0).stop 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 shutdown_connections 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48dc06c8b0 0x7f48dc06ed70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48f80b2aa0 0x7f48f80aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48f80b2630 0x7f48f815d380 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 --2- 172.21.15.107:0/3817955923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48f80b2250 0x7f48f8167ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 >> 172.21.15.107:0/3817955923 conn(0x7f48f806a920 msgr2=0x7f48f80cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.876+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 shutdown_connections 2023-11-24T08:19:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:00.877+0000 7f48feaad640 1 -- 172.21.15.107:0/3817955923 wait complete. 2023-11-24T08:19:00.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:17:38.499480+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:17:39.719116+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:19:00.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:19:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.101+0000 7f0ac2541640 1 Processor -- start 2023-11-24T08:19:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.101+0000 7f0ac2541640 1 -- start start 2023-11-24T08:19:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0ac2541640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0ac2541640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc15eea0 con 0x7f0abc15b440 2023-11-24T08:19:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0ac2541640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc0abf30 con 0x7f0abc0c57e0 2023-11-24T08:19:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0ac2541640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc15b820 con 0x7f0abc0cc0f0 2023-11-24T08:19:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0abbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0abc15b440 0x7f0abc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51358/0 (socket says 172.21.15.107:51358) 2023-11-24T08:19:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0abbfff640 1 -- 172.21.15.107:0/843451021 learned_addr learned my addr 172.21.15.107:0/843451021 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/843451021 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417867063 0 0) 0x7f0abc15eea0 con 0x7f0abc15b440 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.103+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa4003520 con 0x7f0abc15b440 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 52109603 0 0) 0x7f0aa4003520 con 0x7f0abc15b440 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 msgr2=0x7f0abc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 --2- 172.21.15.107:0/843451021 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 >> v1:172.21.15.120:6789/0 conn(0x7f0abc0c57e0 legacy=0x7f0abc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc15f0f0 con 0x7f0abc15b440 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0aac001670 con 0x7f0abc15b440 2023-11-24T08:19:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 >> v1:172.21.15.107:6789/0 conn(0x7f0abc15b440 legacy=0x7f0abc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40039c0 0x7f0aa4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 0x7f0aa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa40052a0 0x7f0aa400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15b820 con 0x7f0aa40047c0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa400c1c0 con 0x7f0aa40052a0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa400c380 con 0x7f0aa40039c0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.104+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 0x7f0aa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abbfff640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40039c0 0x7f0aa4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0ac0ade640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa40052a0 0x7f0aa400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abb7fe640 1 -- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40039c0 msgr2=0x7f0aa4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40039c0 0x7f0aa4003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abb7fe640 1 -- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa40052a0 msgr2=0x7f0aa400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa40052a0 0x7f0aa400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0abb7fe640 1 -- 172.21.15.107:0/843451021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4003520 con 0x7f0aa40047c0 2023-11-24T08:19:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0ac0ade640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa40052a0 0x7f0aa400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.105+0000 7f0ac0ade640 1 -- 172.21.15.107:0/843451021 reap_dead start 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.106+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 0x7f0aa4004c40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000ba30 tx=0x7f0aa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7727cd8b4fb18f4b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.106+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0032640 con 0x7f0aa40047c0 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.106+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab00327a0 con 0x7f0aa40047c0 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.106+0000 7f0abaffd640 1 -- 172.21.15.107:0/843451021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0032b30 con 0x7f0aa40047c0 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 -- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 msgr2=0x7f0aa4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 0x7f0aa4004c40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000ba30 tx=0x7f0aa40057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 -- 172.21.15.107:0/843451021 shutdown_connections 2023-11-24T08:19:01.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 --2- 172.21.15.107:0/843451021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0aa40047c0 0x7f0aa4004c40 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 -- 172.21.15.107:0/843451021 >> 172.21.15.107:0/843451021 conn(0x7f0abc0b0c50 msgr2=0x7f0abc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:01.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 -- 172.21.15.107:0/843451021 shutdown_connections 2023-11-24T08:19:01.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.107+0000 7f0ac2541640 1 -- 172.21.15.107:0/843451021 wait complete. 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.109+0000 7f0ac2541640 1 Processor -- start 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.109+0000 7f0ac2541640 1 -- start start 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40047c0 0x7f0abc15b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc06c860 0x7f0abc09a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0c7480 con 0x7f0abc06cce0 2023-11-24T08:19:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc156d00 con 0x7f0abc06c860 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac2541640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc156fa0 con 0x7f0aa40047c0 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49824/0 (socket says 172.21.15.107:49824) 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 -- 172.21.15.107:0/2077097936 learned_addr learned my addr 172.21.15.107:0/2077097936 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0abb7fe640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc06c860 0x7f0abc09a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0abbfff640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40047c0 0x7f0abc15b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 -- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40047c0 msgr2=0x7f0abc15b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40047c0 0x7f0abc15b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 -- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc06c860 msgr2=0x7f0abc09a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc06c860 0x7f0abc09a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.110+0000 7f0ac0ade640 1 -- 172.21.15.107:0/2077097936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc157280 con 0x7f0abc06cce0 2023-11-24T08:19:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ac0ade640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f0abc06e890 tx=0x7f0ab400d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab400b070 con 0x7f0abc06cce0 2023-11-24T08:19:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab4003da0 con 0x7f0abc06cce0 2023-11-24T08:19:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc0711f0 con 0x7f0abc06cce0 2023-11-24T08:19:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4005310 con 0x7f0abc06cce0 2023-11-24T08:19:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.111+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc09a850 con 0x7f0abc06cce0 2023-11-24T08:19:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.112+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc157410 con 0x7f0abc06cce0 2023-11-24T08:19:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.113+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f0ab4005470 con 0x7f0abc06cce0 2023-11-24T08:19:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.114+0000 7f0ab8ff9640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 0x7f0aa806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.114+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0ab4087c40 con 0x7f0abc06cce0 2023-11-24T08:19:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.115+0000 7f0abbfff640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 0x7f0aa806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.115+0000 7f0abbfff640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 0x7f0aa806ed70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0aac000c00 tx=0x7f0aac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.119+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ab40547f0 con 0x7f0abc06cce0 2023-11-24T08:19:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.313+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0abc04e8b0 con 0x7f0abc06cce0 2023-11-24T08:19:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.314+0000 7f0ab8ff9640 1 -- 172.21.15.107:0/2077097936 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ab40180a0 con 0x7f0abc06cce0 2023-11-24T08:19:01.314 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 msgr2=0x7f0aa806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 0x7f0aa806ed70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0aac000c00 tx=0x7f0aac009040 comp rx=0 tx=0).stop 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 msgr2=0x7f0abc156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f0abc06e890 tx=0x7f0ab400d900 comp rx=0 tx=0).stop 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 shutdown_connections 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa806c8b0 0x7f0aa806ed70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc06cce0 0x7f0abc156080 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc06c860 0x7f0abc09a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 --2- 172.21.15.107:0/2077097936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40047c0 0x7f0abc15b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 >> 172.21.15.107:0/2077097936 conn(0x7f0abc0b0c50 msgr2=0x7f0abc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 shutdown_connections 2023-11-24T08:19:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.317+0000 7f0ac2541640 1 -- 172.21.15.107:0/2077097936 wait complete. 2023-11-24T08:19:01.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:19:01.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.547+0000 7fe3cdead640 1 Processor -- start 2023-11-24T08:19:01.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.547+0000 7fe3cdead640 1 -- start start 2023-11-24T08:19:01.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3cdead640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3c80cc0f0 0x7fe3c80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3cdead640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3c80c57e0 0x7fe3c806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3cdead640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c815eea0 con 0x7fe3c815b440 2023-11-24T08:19:01.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3cdead640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c815b820 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3cdead640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c80abf30 con 0x7fe3c80c57e0 2023-11-24T08:19:01.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3c77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3c815b440 0x7fe3c80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51372/0 (socket says 172.21.15.107:51372) 2023-11-24T08:19:01.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3c77fe640 1 -- 172.21.15.107:0/1936466561 learned_addr learned my addr 172.21.15.107:0/1936466561 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:01.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.549+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/1936466561 >> v2:172.21.15.107:3300/0 conn(0x7fe3c80cc0f0 0x7fe3c80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.550+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/1936466561 >> v2:172.21.15.120:3300/0 conn(0x7fe3c80c57e0 msgr2=0x7fe3c806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.550+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/1936466561 >> v2:172.21.15.120:3300/0 conn(0x7fe3c80c57e0 0x7fe3c806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.550+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/1936466561 >> v2:172.21.15.120:3300/0 conn(0x7fe3c80c57e0 0x7fe3c806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.550+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/1936466561 >> v1:172.21.15.107:6789/0 conn(0x7fe3c815b440 legacy=0x7fe3c80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.550+0000 7fe3c6ffd640 1 -- 172.21.15.107:0/1936466561 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c815f0f0 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.551+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/1936466561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c80c52a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80c95e0 tx=0x7fe3bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9864fd3fbacb5c5d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.551+0000 7fe3c67fc640 1 -- 172.21.15.107:0/1936466561 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc02e070 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.551+0000 7fe3c67fc640 1 -- 172.21.15.107:0/1936466561 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3bc0315a0 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.551+0000 7fe3c67fc640 1 -- 172.21.15.107:0/1936466561 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc031930 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 -- 172.21.15.107:0/1936466561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c80cc0f0 msgr2=0x7fe3c80c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 --2- 172.21.15.107:0/1936466561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c80c52a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80c95e0 tx=0x7fe3bc02b840 comp rx=0 tx=0).stop 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 -- 172.21.15.107:0/1936466561 shutdown_connections 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 --2- 172.21.15.107:0/1936466561 >> v2:172.21.15.120:3300/0 conn(0x7fe3c80c57e0 0x7fe3c806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 --2- 172.21.15.107:0/1936466561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c80c52a0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 -- 172.21.15.107:0/1936466561 >> 172.21.15.107:0/1936466561 conn(0x7fe3c80b0c50 msgr2=0x7fe3c80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 -- 172.21.15.107:0/1936466561 shutdown_connections 2023-11-24T08:19:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.552+0000 7fe3cdead640 1 -- 172.21.15.107:0/1936466561 wait complete. 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.553+0000 7fe3cdead640 1 Processor -- start 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 -- start start 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c80c57e0 0x7fe3c81554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c81566c0 con 0x7fe3c815b440 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c8155ab0 con 0x7fe3c80cc0f0 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.554+0000 7fe3cdead640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c8155d90 con 0x7fe3c80c57e0 2023-11-24T08:19:01.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49848/0 (socket says 172.21.15.107:49848) 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2648067075 learned_addr learned my addr 172.21.15.107:0/2648067075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c6ffd640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c80c57e0 msgr2=0x7fe3c81554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c80c57e0 0x7fe3c81554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c80cc0f0 msgr2=0x7fe3c8156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c8156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 -- 172.21.15.107:0/2648067075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc009400 con 0x7fe3c815b440 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.555+0000 7fe3c7fff640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000cd70 tx=0x7fe3c000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.556+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0003b90 con 0x7fe3c815b440 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.556+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c0003cf0 con 0x7fe3c815b440 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.556+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0004240 con 0x7fe3c815b440 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.556+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c8156070 con 0x7fe3c815b440 2023-11-24T08:19:01.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.556+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c8150e40 con 0x7fe3c815b440 2023-11-24T08:19:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.557+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c804e8b0 con 0x7fe3c815b440 2023-11-24T08:19:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.558+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fe3c00043a0 con 0x7fe3c815b440 2023-11-24T08:19:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.559+0000 7fe3affff640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 0x7fe3b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.559+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe3c0012070 con 0x7fe3c815b440 2023-11-24T08:19:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.559+0000 7fe3c77fe640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 0x7fe3b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.560+0000 7fe3c77fe640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 0x7fe3b406ed70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001550 tx=0x7fe3b8007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.563+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3c0055a50 con 0x7fe3c815b440 2023-11-24T08:19:01.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.738+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe3c8156800 con 0x7fe3c815b440 2023-11-24T08:19:01.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.739+0000 7fe3affff640 1 -- 172.21.15.107:0/2648067075 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe3c0059070 con 0x7fe3c815b440 2023-11-24T08:19:01.740 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.741+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 msgr2=0x7fe3b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.741+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 0x7fe3b406ed70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001550 tx=0x7fe3b8007a40 comp rx=0 tx=0).stop 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.741+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 msgr2=0x7fe3c8150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.741+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000cd70 tx=0x7fe3c000a660 comp rx=0 tx=0).stop 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 shutdown_connections 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b406c8b0 0x7fe3b406ed70 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3c815b440 0x7fe3c8150900 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c80cc0f0 0x7fe3c8156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 --2- 172.21.15.107:0/2648067075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3c80c57e0 0x7fe3c81554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 >> 172.21.15.107:0/2648067075 conn(0x7fe3c80b0c50 msgr2=0x7fe3c8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:01.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 shutdown_connections 2023-11-24T08:19:01.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.742+0000 7fe3cdead640 1 -- 172.21.15.107:0/2648067075 wait complete. 2023-11-24T08:19:01.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:19:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.967+0000 7f4e833cf640 1 Processor -- start 2023-11-24T08:19:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.967+0000 7f4e833cf640 1 -- start start 2023-11-24T08:19:01.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.968+0000 7f4e833cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 0x7f4e7c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.968+0000 7f4e833cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.968+0000 7f4e833cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c15ed20 con 0x7f4e7c06e580 2023-11-24T08:19:01.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.968+0000 7f4e833cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7c15b640 con 0x7f4e7c0b1250 2023-11-24T08:19:01.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.968+0000 7f4e833cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c15f8a0 con 0x7f4e7c15b260 2023-11-24T08:19:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.969+0000 7f4e82bce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.969+0000 7f4e823cd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 0x7f4e7c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.969+0000 7f4e82bce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42268/0 (socket says 172.21.15.107:42268) 2023-11-24T08:19:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.969+0000 7f4e82bce640 1 -- 172.21.15.107:0/1578673498 learned_addr learned my addr 172.21.15.107:0/1578673498 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.969+0000 7f4e81bcc640 1 --1- 172.21.15.107:0/1578673498 >> v1:172.21.15.120:6789/0 conn(0x7f4e7c06e580 0x7f4e7c0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:45740/0 (socket says 172.21.15.107:45740) 2023-11-24T08:19:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.970+0000 7f4e82bce640 1 -- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 msgr2=0x7f4e7c06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.970+0000 7f4e82bce640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 0x7f4e7c06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.970+0000 7f4e82bce640 1 -- 172.21.15.107:0/1578673498 >> v1:172.21.15.120:6789/0 conn(0x7f4e7c06e580 legacy=0x7f4e7c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.970+0000 7f4e82bce640 1 -- 172.21.15.107:0/1578673498 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c0abe50 con 0x7f4e7c0b1250 2023-11-24T08:19:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.970+0000 7f4e823cd640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 0x7f4e7c06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:19:01.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.972+0000 7f4e82bce640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009e50 tx=0x7f4e7402be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=354b46abeced2baf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.972+0000 7f4e813cb640 1 -- 172.21.15.107:0/1578673498 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7402c9d0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.972+0000 7f4e813cb640 1 -- 172.21.15.107:0/1578673498 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7402cb30 con 0x7f4e7c0b1250 2023-11-24T08:19:01.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.972+0000 7f4e813cb640 1 -- 172.21.15.107:0/1578673498 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e740343f0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 -- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c0b1250 msgr2=0x7f4e7c06c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009e50 tx=0x7f4e7402be30 comp rx=0 tx=0).stop 2023-11-24T08:19:01.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 -- 172.21.15.107:0/1578673498 shutdown_connections 2023-11-24T08:19:01.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c06c5b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 --2- 172.21.15.107:0/1578673498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c15b260 0x7f4e7c06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 -- 172.21.15.107:0/1578673498 >> 172.21.15.107:0/1578673498 conn(0x7f4e7c0cbb40 msgr2=0x7f4e7c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:01.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 -- 172.21.15.107:0/1578673498 shutdown_connections 2023-11-24T08:19:01.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.973+0000 7f4e833cf640 1 -- 172.21.15.107:0/1578673498 wait complete. 2023-11-24T08:19:01.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.975+0000 7f4e833cf640 1 Processor -- start 2023-11-24T08:19:01.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.976+0000 7f4e833cf640 1 -- start start 2023-11-24T08:19:01.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c06e580 0x7f4e7c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c15b260 0x7f4e7c155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c155fe0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c5a90 con 0x7f4e7c15b260 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e833cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0b3ce0 con 0x7f4e7c06e580 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e81bcc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e81bcc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49854/0 (socket says 172.21.15.107:49854) 2023-11-24T08:19:01.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e81bcc640 1 -- 172.21.15.107:0/3785949914 learned_addr learned my addr 172.21.15.107:0/3785949914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.977+0000 7f4e823cd640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c06e580 0x7f4e7c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e82bce640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c15b260 0x7f4e7c155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 -- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c06e580 msgr2=0x7f4e7c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c06e580 0x7f4e7c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 -- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c15b260 msgr2=0x7f4e7c155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c15b260 0x7f4e7c155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 -- 172.21.15.107:0/3785949914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78009400 con 0x7f4e7c0b1250 2023-11-24T08:19:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e81bcc640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00c3e0 tx=0x7f4e6c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0074e0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c007640 con 0x7f4e7c0b1250 2023-11-24T08:19:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74009b30 con 0x7f4e7c0b1250 2023-11-24T08:19:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.978+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0049e0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.979+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c0b42a0 con 0x7f4e7c0b1250 2023-11-24T08:19:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.980+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c067d70 con 0x7f4e7c0b1250 2023-11-24T08:19:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.981+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f4e6c004b40 con 0x7f4e7c0b1250 2023-11-24T08:19:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.981+0000 7f4e72ffd640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 0x7f4e6006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.982+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4e6c087e40 con 0x7f4e7c0b1250 2023-11-24T08:19:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.982+0000 7f4e823cd640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 0x7f4e6006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:01.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.983+0000 7f4e823cd640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 0x7f4e6006ed70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009bc0 tx=0x7f4e78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:01.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:01.987+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e6c0549f0 con 0x7f4e7c0b1250 2023-11-24T08:19:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.169+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e7c04e840 con 0x7f4e7c0b1250 2023-11-24T08:19:02.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.171+0000 7f4e72ffd640 1 -- 172.21.15.107:0/3785949914 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e6c058010 con 0x7f4e7c0b1250 2023-11-24T08:19:02.172 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:19:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 msgr2=0x7f4e6006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 0x7f4e6006ed70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009bc0 tx=0x7f4e78008040 comp rx=0 tx=0).stop 2023-11-24T08:19:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 msgr2=0x7f4e7c0c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00c3e0 tx=0x7f4e6c00c8a0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 shutdown_connections 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4e6006c8b0 0x7f4e6006ed70 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e7c15b260 0x7f4e7c155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e7c0b1250 0x7f4e7c0c53b0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 --2- 172.21.15.107:0/3785949914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7c06e580 0x7f4e7c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 >> 172.21.15.107:0/3785949914 conn(0x7f4e7c0cbb40 msgr2=0x7f4e7c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 shutdown_connections 2023-11-24T08:19:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.173+0000 7f4e833cf640 1 -- 172.21.15.107:0/3785949914 wait complete. 2023-11-24T08:19:02.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:02.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.394+0000 7efe97733640 1 Processor -- start 2023-11-24T08:19:02.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.395+0000 7efe97733640 1 -- start start 2023-11-24T08:19:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.396+0000 7efe97733640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe9015c7f0 0x7efe9006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.396+0000 7efe97733640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe900ce6c0 0x7efe900cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.396+0000 7efe97733640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe900cece0 con 0x7efe900cf450 2023-11-24T08:19:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.396+0000 7efe97733640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe900cfe40 con 0x7efe9015c7f0 2023-11-24T08:19:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.396+0000 7efe97733640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe900ceaf0 con 0x7efe900ce6c0 2023-11-24T08:19:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe9015c7f0 0x7efe9006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe9015c7f0 0x7efe9006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49856/0 (socket says 172.21.15.107:49856) 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 -- 172.21.15.107:0/3822167155 learned_addr learned my addr 172.21.15.107:0/3822167155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe95cd0640 1 --2- 172.21.15.107:0/3822167155 >> v2:172.21.15.120:3300/0 conn(0x7efe900ce6c0 0x7efe900cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 -- 172.21.15.107:0/3822167155 >> v2:172.21.15.120:3300/0 conn(0x7efe900ce6c0 msgr2=0x7efe900cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 --2- 172.21.15.107:0/3822167155 >> v2:172.21.15.120:3300/0 conn(0x7efe900ce6c0 0x7efe900cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 -- 172.21.15.107:0/3822167155 >> v1:172.21.15.120:6789/0 conn(0x7efe900cf450 legacy=0x7efe900cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.397+0000 7efe954cf640 1 -- 172.21.15.107:0/3822167155 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9006d040 con 0x7efe9015c7f0 2023-11-24T08:19:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.398+0000 7efe954cf640 1 --2- 172.21.15.107:0/3822167155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe9015c7f0 0x7efe9006db20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7efe900d00e0 tx=0x7efe8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7f96015ca6de0a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.398+0000 7efe8ffff640 1 -- 172.21.15.107:0/3822167155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8402ca80 con 0x7efe9015c7f0 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.398+0000 7efe8ffff640 1 -- 172.21.15.107:0/3822167155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8402cbe0 con 0x7efe9015c7f0 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.398+0000 7efe8ffff640 1 -- 172.21.15.107:0/3822167155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840343f0 con 0x7efe9015c7f0 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 -- 172.21.15.107:0/3822167155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe9015c7f0 msgr2=0x7efe9006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 --2- 172.21.15.107:0/3822167155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe9015c7f0 0x7efe9006db20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7efe900d00e0 tx=0x7efe8402bee0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 -- 172.21.15.107:0/3822167155 shutdown_connections 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 --2- 172.21.15.107:0/3822167155 >> v2:172.21.15.120:3300/0 conn(0x7efe900ce6c0 0x7efe900cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 --2- 172.21.15.107:0/3822167155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe9015c7f0 0x7efe9006db20 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 -- 172.21.15.107:0/3822167155 >> 172.21.15.107:0/3822167155 conn(0x7efe9006a920 msgr2=0x7efe9006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 -- 172.21.15.107:0/3822167155 shutdown_connections 2023-11-24T08:19:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.399+0000 7efe97733640 1 -- 172.21.15.107:0/3822167155 wait complete. 2023-11-24T08:19:02.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.401+0000 7efe97733640 1 Processor -- start 2023-11-24T08:19:02.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.401+0000 7efe97733640 1 -- start start 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe900cf450 0x7efe900b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9015c7f0 0x7efe90071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90149da0 con 0x7efe900ce6c0 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90155f10 con 0x7efe9015c7f0 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe97733640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe900b3020 con 0x7efe900cf450 2023-11-24T08:19:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe954cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe954cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49872/0 (socket says 172.21.15.107:49872) 2023-11-24T08:19:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.402+0000 7efe954cf640 1 -- 172.21.15.107:0/746918973 learned_addr learned my addr 172.21.15.107:0/746918973 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe94cce640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe900cf450 0x7efe900b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 -- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe900cf450 msgr2=0x7efe900b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe900cf450 0x7efe900b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 -- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9015c7f0 msgr2=0x7efe90071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9015c7f0 0x7efe90071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 -- 172.21.15.107:0/746918973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84009b30 con 0x7efe900ce6c0 2023-11-24T08:19:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe954cf640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7efe84000f80 tx=0x7efe84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8403e450 con 0x7efe900ce6c0 2023-11-24T08:19:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8403e5f0 con 0x7efe900ce6c0 2023-11-24T08:19:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8403e9a0 con 0x7efe900ce6c0 2023-11-24T08:19:02.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.403+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe900b2780 con 0x7efe900ce6c0 2023-11-24T08:19:02.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.404+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe900b2d40 con 0x7efe900ce6c0 2023-11-24T08:19:02.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.405+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe900b0360 con 0x7efe900ce6c0 2023-11-24T08:19:02.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.406+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7efe8402cd50 con 0x7efe900ce6c0 2023-11-24T08:19:02.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.406+0000 7efe8dffb640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 0x7efe7406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.407+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efe840ab0c0 con 0x7efe900ce6c0 2023-11-24T08:19:02.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.407+0000 7efe94cce640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 0x7efe7406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.407+0000 7efe94cce640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 0x7efe7406ed70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efe9015c130 tx=0x7efe7c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.411+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe84077c70 con 0x7efe900ce6c0 2023-11-24T08:19:02.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.636+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe90067470 con 0x7efe900ce6c0 2023-11-24T08:19:02.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.637+0000 7efe8dffb640 1 -- 172.21.15.107:0/746918973 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4929 (secure 0 0 0) 0x7efe8407b290 con 0x7efe900ce6c0 2023-11-24T08:19:02.638 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:02.638 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":98,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:19:02.639 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 98 2023-11-24T08:19:02.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 msgr2=0x7efe7406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 0x7efe7406ed70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efe9015c130 tx=0x7efe7c009040 comp rx=0 tx=0).stop 2023-11-24T08:19:02.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 msgr2=0x7efe900b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7efe84000f80 tx=0x7efe84004a40 comp rx=0 tx=0).stop 2023-11-24T08:19:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 shutdown_connections 2023-11-24T08:19:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efe7406c8b0 0x7efe7406ed70 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9015c7f0 0x7efe90071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe900cf450 0x7efe900b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 --2- 172.21.15.107:0/746918973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900ce6c0 0x7efe900b20e0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 >> 172.21.15.107:0/746918973 conn(0x7efe9006a920 msgr2=0x7efe900afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 shutdown_connections 2023-11-24T08:19:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.640+0000 7efe97733640 1 -- 172.21.15.107:0/746918973 wait complete. 2023-11-24T08:19:02.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:02.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.871+0000 7f0f818b4640 1 Processor -- start 2023-11-24T08:19:02.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.872+0000 7f0f818b4640 1 -- start start 2023-11-24T08:19:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f7c0d0130 0x7f0f7c161040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c0d2150 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c0cdde0 con 0x7f0f7c0d0130 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f818b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0cec20 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f808b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f808b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49888/0 (socket says 172.21.15.107:49888) 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.873+0000 7f0f808b2640 1 -- 172.21.15.107:0/2623740751 learned_addr learned my addr 172.21.15.107:0/2623740751 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f810b3640 1 --2- 172.21.15.107:0/2623740751 >> v2:172.21.15.120:3300/0 conn(0x7f0f7c0d0130 0x7f0f7c161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f808b2640 1 -- 172.21.15.107:0/2623740751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c0d1cd0 msgr2=0x7f0f7c0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f808b2640 1 --2- 172.21.15.107:0/2623740751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f808b2640 1 -- 172.21.15.107:0/2623740751 >> v2:172.21.15.120:3300/0 conn(0x7f0f7c0d0130 msgr2=0x7f0f7c161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f808b2640 1 --2- 172.21.15.107:0/2623740751 >> v2:172.21.15.120:3300/0 conn(0x7f0f7c0d0130 0x7f0f7c161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.874+0000 7f0f808b2640 1 -- 172.21.15.107:0/2623740751 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c0ce010 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.875+0000 7f0f808b2640 1 --2- 172.21.15.107:0/2623740751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0a11b0 tx=0x7f0f6402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23ffea765d26903 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.875+0000 7f0f7b7fe640 1 -- 172.21.15.107:0/2623740751 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64031860 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.875+0000 7f0f7b7fe640 1 -- 172.21.15.107:0/2623740751 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f640319c0 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.875+0000 7f0f7b7fe640 1 -- 172.21.15.107:0/2623740751 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64031d50 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 -- 172.21.15.107:0/2623740751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c15d4e0 msgr2=0x7f0f7c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 --2- 172.21.15.107:0/2623740751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0a11b0 tx=0x7f0f6402b900 comp rx=0 tx=0).stop 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 -- 172.21.15.107:0/2623740751 shutdown_connections 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 --2- 172.21.15.107:0/2623740751 >> v2:172.21.15.120:3300/0 conn(0x7f0f7c0d0130 0x7f0f7c161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 --2- 172.21.15.107:0/2623740751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 --2- 172.21.15.107:0/2623740751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c15d4e0 0x7f0f7c0d1790 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 -- 172.21.15.107:0/2623740751 >> 172.21.15.107:0/2623740751 conn(0x7f0f7c09c500 msgr2=0x7f0f7c0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 -- 172.21.15.107:0/2623740751 shutdown_connections 2023-11-24T08:19:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.876+0000 7f0f818b4640 1 -- 172.21.15.107:0/2623740751 wait complete. 2023-11-24T08:19:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.878+0000 7f0f818b4640 1 Processor -- start 2023-11-24T08:19:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.878+0000 7f0f818b4640 1 -- start start 2023-11-24T08:19:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f7c0d0130 0x7f0f7c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c15d4e0 0x7f0f7c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0c84a0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0c8010 con 0x7f0f7c0d0130 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f818b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0c7400 con 0x7f0f7c15d4e0 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49900/0 (socket says 172.21.15.107:49900) 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.879+0000 7f0f7bfff640 1 -- 172.21.15.107:0/486096224 learned_addr learned my addr 172.21.15.107:0/486096224 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f808b2640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f7c0d0130 0x7f0f7c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f810b3640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c15d4e0 0x7f0f7c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 -- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c15d4e0 msgr2=0x7f0f7c153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c15d4e0 0x7f0f7c153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 -- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f7c0d0130 msgr2=0x7f0f7c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f7c0d0130 0x7f0f7c0c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 -- 172.21.15.107:0/486096224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640094d0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f7bfff640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b410 tx=0x7f0f6c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c011830 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c0119b0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c011d60 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c0c76e0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.880+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c0c7ca0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.882+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7c0c81c0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.882+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f0f6c01a400 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.884+0000 7f0f797fa640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 0x7f0f5c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:02.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.884+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0f6c088610 con 0x7f0f7c0d1cd0 2023-11-24T08:19:02.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.884+0000 7f0f808b2640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 0x7f0f5c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:02.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.885+0000 7f0f808b2640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 0x7f0f5c06eda0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0f64036000 tx=0x7f0f6403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:02.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:02.889+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f7c0c81c0 con 0x7f0f7c0d1cd0 2023-11-24T08:19:03.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.080+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f7c0c8180 con 0x7f0f7c0d1cd0 2023-11-24T08:19:03.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.083+0000 7f0f797fa640 1 -- 172.21.15.107:0/486096224 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f0f6c015020 con 0x7f0f7c0d1cd0 2023-11-24T08:19:03.084 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:03.084 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:18:55.038112+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 msgr2=0x7f0f5c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 0x7f0f5c06eda0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0f64036000 tx=0x7f0f6403c040 comp rx=0 tx=0).stop 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 msgr2=0x7f0f7c0c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b410 tx=0x7f0f6c00b8d0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 shutdown_connections 2023-11-24T08:19:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5c06c8e0 0x7f0f5c06eda0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7c15d4e0 0x7f0f7c153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7c0d1cd0 0x7f0f7c0c6980 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 --2- 172.21.15.107:0/486096224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f7c0d0130 0x7f0f7c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 >> 172.21.15.107:0/486096224 conn(0x7f0f7c09c500 msgr2=0x7f0f7c0b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 shutdown_connections 2023-11-24T08:19:03.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.086+0000 7f0f818b4640 1 -- 172.21.15.107:0/486096224 wait complete. 2023-11-24T08:19:03.105 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:19:03.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:03.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.306+0000 7f3b4837c640 1 Processor -- start 2023-11-24T08:19:03.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.307+0000 7f3b4837c640 1 -- start start 2023-11-24T08:19:03.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.308+0000 7f3b4837c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 0x7f3b4006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.308+0000 7f3b4837c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.308+0000 7f3b4837c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400cece0 con 0x7f3b400cdfb0 2023-11-24T08:19:03.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.308+0000 7f3b4837c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b40156010 con 0x7f3b400cf450 2023-11-24T08:19:03.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.308+0000 7f3b4837c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4006ce30 con 0x7f3b4015c7f0 2023-11-24T08:19:03.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.309+0000 7f3b45917640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.309+0000 7f3b46118640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 0x7f3b4006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.309+0000 7f3b45917640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42320/0 (socket says 172.21.15.107:42320) 2023-11-24T08:19:03.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.309+0000 7f3b45917640 1 -- 172.21.15.107:0/4129770517 learned_addr learned my addr 172.21.15.107:0/4129770517 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:03.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.310+0000 7f3b46118640 1 -- 172.21.15.107:0/4129770517 >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 msgr2=0x7f3b400cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:03.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.310+0000 7f3b46118640 1 --2- 172.21.15.107:0/4129770517 >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.310+0000 7f3b46118640 1 -- 172.21.15.107:0/4129770517 >> v1:172.21.15.120:6789/0 conn(0x7f3b400cdfb0 legacy=0x7f3b400ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.310+0000 7f3b46118640 1 -- 172.21.15.107:0/4129770517 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4006d040 con 0x7f3b4015c7f0 2023-11-24T08:19:03.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.310+0000 7f3b45917640 1 --2- 172.21.15.107:0/4129770517 >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:19:03.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.311+0000 7f3b46118640 1 --2- 172.21.15.107:0/4129770517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 0x7f3b4006db20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0097a0 tx=0x7f3b3c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e6be6fe4265c465 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b45116640 1 -- 172.21.15.107:0/4129770517 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c02d040 con 0x7f3b4015c7f0 2023-11-24T08:19:03.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b45116640 1 -- 172.21.15.107:0/4129770517 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c031810 con 0x7f3b4015c7f0 2023-11-24T08:19:03.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b45116640 1 -- 172.21.15.107:0/4129770517 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c031ba0 con 0x7f3b4015c7f0 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b4837c640 1 -- 172.21.15.107:0/4129770517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 msgr2=0x7f3b4006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b4837c640 1 --2- 172.21.15.107:0/4129770517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 0x7f3b4006db20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0097a0 tx=0x7f3b3c02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b4837c640 1 -- 172.21.15.107:0/4129770517 shutdown_connections 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b4837c640 1 --2- 172.21.15.107:0/4129770517 >> v2:172.21.15.120:3300/0 conn(0x7f3b400cf450 0x7f3b400cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.312+0000 7f3b4837c640 1 --2- 172.21.15.107:0/4129770517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b4015c7f0 0x7f3b4006db20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.313+0000 7f3b4837c640 1 -- 172.21.15.107:0/4129770517 >> 172.21.15.107:0/4129770517 conn(0x7f3b4006a920 msgr2=0x7f3b4006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.313+0000 7f3b4837c640 1 -- 172.21.15.107:0/4129770517 shutdown_connections 2023-11-24T08:19:03.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.313+0000 7f3b4837c640 1 -- 172.21.15.107:0/4129770517 wait complete. 2023-11-24T08:19:03.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.314+0000 7f3b4837c640 1 Processor -- start 2023-11-24T08:19:03.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.314+0000 7f3b4837c640 1 -- start start 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b400cdfb0 0x7f3b40148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b400cf450 0x7f3b401f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401f1c40 con 0x7f3b4015c7f0 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4015e610 con 0x7f3b400cdfb0 2023-11-24T08:19:03.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.315+0000 7f3b4837c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4015e8f0 con 0x7f3b400cf450 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49906/0 (socket says 172.21.15.107:49906) 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 -- 172.21.15.107:0/1718194573 learned_addr learned my addr 172.21.15.107:0/1718194573 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46118640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b400cdfb0 0x7f3b40148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b45917640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b400cf450 0x7f3b401f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 -- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b400cf450 msgr2=0x7f3b401f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b400cf450 0x7f3b401f1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 -- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b400cdfb0 msgr2=0x7f3b40148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b400cdfb0 0x7f3b40148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 -- 172.21.15.107:0/1718194573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300094d0 con 0x7f3b4015c7f0 2023-11-24T08:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.316+0000 7f3b46919640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c730 tx=0x7f3b3800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.317+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38003b70 con 0x7f3b4015c7f0 2023-11-24T08:19:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.317+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38003cf0 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.317+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38003ea0 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.317+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c009400 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.317+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b401f11e0 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.318+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b40067470 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.319+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f3b38005520 con 0x7f3b4015c7f0 2023-11-24T08:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.320+0000 7f3b36ffd640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 0x7f3b2406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.320+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3b38088460 con 0x7f3b4015c7f0 2023-11-24T08:19:03.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.320+0000 7f3b46118640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 0x7f3b2406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.321+0000 7f3b46118640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 0x7f3b2406eda0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036000 tx=0x7f3b3c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.324+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b38054f70 con 0x7f3b4015c7f0 2023-11-24T08:19:03.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.554+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b400cd730 con 0x7f3b4015c7f0 2023-11-24T08:19:03.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.555+0000 7f3b36ffd640 1 -- 172.21.15.107:0/1718194573 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f3b38058590 con 0x7f3b4015c7f0 2023-11-24T08:19:03.556 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:03.556 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":98,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:19:03.556 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 98 2023-11-24T08:19:03.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 msgr2=0x7f3b2406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 0x7f3b2406eda0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036000 tx=0x7f3b3c03c040 comp rx=0 tx=0).stop 2023-11-24T08:19:03.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 msgr2=0x7f3b4015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c730 tx=0x7f3b3800cbf0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 shutdown_connections 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b2406c8e0 0x7f3b2406eda0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b4015c7f0 0x7f3b4015dca0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b400cf450 0x7f3b401f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.557+0000 7f3b4837c640 1 --2- 172.21.15.107:0/1718194573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b400cdfb0 0x7f3b40148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.558+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 >> 172.21.15.107:0/1718194573 conn(0x7f3b4006a920 msgr2=0x7f3b400af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.558+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 shutdown_connections 2023-11-24T08:19:03.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.558+0000 7f3b4837c640 1 -- 172.21.15.107:0/1718194573 wait complete. 2023-11-24T08:19:03.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.792+0000 7f3546dc4640 1 Processor -- start 2023-11-24T08:19:03.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.792+0000 7f3546dc4640 1 -- start start 2023-11-24T08:19:03.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.793+0000 7f3546dc4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3540157c70 0x7f3540161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.793+0000 7f3546dc4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f3546dc4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35400ce0b0 con 0x7f354015dc80 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f3546dc4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35400cfe10 con 0x7f3540157c70 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f3546dc4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3540161960 con 0x7f35400cf450 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f353ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f353ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:53972/0 (socket says 172.21.15.107:53972) 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f3544b60640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3540157c70 0x7f3540161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.794+0000 7f353ffff640 1 -- 172.21.15.107:0/442785356 learned_addr learned my addr 172.21.15.107:0/442785356 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.795+0000 7f353ffff640 1 -- 172.21.15.107:0/442785356 >> v2:172.21.15.120:3300/0 conn(0x7f3540157c70 msgr2=0x7f3540161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.795+0000 7f353ffff640 1 --2- 172.21.15.107:0/442785356 >> v2:172.21.15.120:3300/0 conn(0x7f3540157c70 0x7f3540161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.795+0000 7f353ffff640 1 -- 172.21.15.107:0/442785356 >> v1:172.21.15.120:6789/0 conn(0x7f354015dc80 legacy=0x7f354015e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.795+0000 7f353ffff640 1 -- 172.21.15.107:0/442785356 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354015e8d0 con 0x7f35400cf450 2023-11-24T08:19:03.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.796+0000 7f353ffff640 1 --2- 172.21.15.107:0/442785356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3530009870 tx=0x7f353002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351d5c9ae0e0cf96 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f353f7fe640 1 -- 172.21.15.107:0/442785356 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530031640 con 0x7f35400cf450 2023-11-24T08:19:03.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f353f7fe640 1 -- 172.21.15.107:0/442785356 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35300317a0 con 0x7f35400cf450 2023-11-24T08:19:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f353f7fe640 1 -- 172.21.15.107:0/442785356 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530031b30 con 0x7f35400cf450 2023-11-24T08:19:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 -- 172.21.15.107:0/442785356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 msgr2=0x7f35400cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 --2- 172.21.15.107:0/442785356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3530009870 tx=0x7f353002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 -- 172.21.15.107:0/442785356 shutdown_connections 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 --2- 172.21.15.107:0/442785356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35400cf450 0x7f35400cf8d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 --2- 172.21.15.107:0/442785356 >> v2:172.21.15.120:3300/0 conn(0x7f3540157c70 0x7f3540161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 -- 172.21.15.107:0/442785356 >> 172.21.15.107:0/442785356 conn(0x7f354006a920 msgr2=0x7f354006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 -- 172.21.15.107:0/442785356 shutdown_connections 2023-11-24T08:19:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.797+0000 7f3546dc4640 1 -- 172.21.15.107:0/442785356 wait complete. 2023-11-24T08:19:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.799+0000 7f3546dc4640 1 Processor -- start 2023-11-24T08:19:03.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.799+0000 7f3546dc4640 1 -- start start 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35400cf450 0x7f354014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f354015dc80 0x7f3540154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3540152040 con 0x7f3540157c70 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354014c360 con 0x7f35400cf450 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.800+0000 7f3546dc4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3540151430 con 0x7f354015dc80 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49922/0 (socket says 172.21.15.107:49922) 2023-11-24T08:19:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 -- 172.21.15.107:0/2254837551 learned_addr learned my addr 172.21.15.107:0/2254837551 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f3545361640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f354015dc80 0x7f3540154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f3544b60640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35400cf450 0x7f354014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 -- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f354015dc80 msgr2=0x7f3540154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f354015dc80 0x7f3540154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 -- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35400cf450 msgr2=0x7f354014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35400cf450 0x7f354014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.801+0000 7f353ffff640 1 -- 172.21.15.107:0/2254837551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35300094d0 con 0x7f3540157c70 2023-11-24T08:19:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f353ffff640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f3530000c00 tx=0x7f35300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353003a930 con 0x7f3540157c70 2023-11-24T08:19:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353003aa90 con 0x7f3540157c70 2023-11-24T08:19:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3540151710 con 0x7f3540157c70 2023-11-24T08:19:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353003ae40 con 0x7f3540157c70 2023-11-24T08:19:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.802+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3540151cd0 con 0x7f3540157c70 2023-11-24T08:19:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.803+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3540067470 con 0x7f3540157c70 2023-11-24T08:19:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.804+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f353002f050 con 0x7f3540157c70 2023-11-24T08:19:03.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.805+0000 7f353d7fa640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 0x7f352c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:03.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.806+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f35300aa260 con 0x7f3540157c70 2023-11-24T08:19:03.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.806+0000 7f3544b60640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 0x7f352c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:03.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.806+0000 7f3544b60640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 0x7f352c06eda0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f35400b0a80 tx=0x7f3534008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:03.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:03.810+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3530076d70 con 0x7f3540157c70 2023-11-24T08:19:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.036+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35400cd910 con 0x7f3540157c70 2023-11-24T08:19:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.037+0000 7f353d7fa640 1 -- 172.21.15.107:0/2254837551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f3530035090 con 0x7f3540157c70 2023-11-24T08:19:04.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:04.038 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":98,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:17:40.695113+0000","modified":"2023-11-24T08:17:43.749894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9873},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9873":{"gid":9873,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1134531969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1134531969},{"type":"v1","addr":"172.21.15.107:6837","nonce":1134531969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-24T08:19:04.038 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 98 2023-11-24T08:19:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 msgr2=0x7f352c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 0x7f352c06eda0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f35400b0a80 tx=0x7f3534008040 comp rx=0 tx=0).stop 2023-11-24T08:19:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 msgr2=0x7f3540150be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f3530000c00 tx=0x7f35300046c0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 shutdown_connections 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f352c06c8e0 0x7f352c06eda0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f354015dc80 0x7f3540154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3540157c70 0x7f3540150be0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 --2- 172.21.15.107:0/2254837551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35400cf450 0x7f354014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 >> 172.21.15.107:0/2254837551 conn(0x7f354006a920 msgr2=0x7f35400b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 shutdown_connections 2023-11-24T08:19:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.039+0000 7f3546dc4640 1 -- 172.21.15.107:0/2254837551 wait complete. 2023-11-24T08:19:04.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:04.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.266+0000 7fdf23fc9640 1 Processor -- start 2023-11-24T08:19:04.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.266+0000 7fdf23fc9640 1 -- start start 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0cc470 0x7fdf1c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 0x7fdf1c0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c0cfe20 con 0x7fdf1c0ce640 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c06c560 con 0x7fdf1c0cf450 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf23fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0ce270 con 0x7fdf1c0cc470 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf21564640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf21564640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49938/0 (socket says 172.21.15.107:49938) 2023-11-24T08:19:04.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf21564640 1 -- 172.21.15.107:0/2668994336 learned_addr learned my addr 172.21.15.107:0/2668994336 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.268+0000 7fdf22566640 1 --2- 172.21.15.107:0/2668994336 >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 0x7fdf1c0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21d65640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0cc470 0x7fdf1c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21564640 1 -- 172.21.15.107:0/2668994336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0cc470 msgr2=0x7fdf1c0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21564640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0cc470 0x7fdf1c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21564640 1 -- 172.21.15.107:0/2668994336 >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 msgr2=0x7fdf1c0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21564640 1 --2- 172.21.15.107:0/2668994336 >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 0x7fdf1c0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf21564640 1 -- 172.21.15.107:0/2668994336 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0b4200 con 0x7fdf1c0ce640 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.269+0000 7fdf22566640 1 --2- 172.21.15.107:0/2668994336 >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 0x7fdf1c0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.270+0000 7fdf21564640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0cd610 tx=0x7fdf0c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f2f12637a28735 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.270+0000 7fdf20d63640 1 -- 172.21.15.107:0/2668994336 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c031640 con 0x7fdf1c0ce640 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.270+0000 7fdf20d63640 1 -- 172.21.15.107:0/2668994336 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0c0317a0 con 0x7fdf1c0ce640 2023-11-24T08:19:04.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.270+0000 7fdf20d63640 1 -- 172.21.15.107:0/2668994336 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c031b30 con 0x7fdf1c0ce640 2023-11-24T08:19:04.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 -- 172.21.15.107:0/2668994336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0ce640 msgr2=0x7fdf1c0b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0cd610 tx=0x7fdf0c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 -- 172.21.15.107:0/2668994336 shutdown_connections 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/2668994336 >> v2:172.21.15.120:3300/0 conn(0x7fdf1c0cf450 0x7fdf1c0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0ce640 0x7fdf1c0b3cc0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/2668994336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0cc470 0x7fdf1c0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 -- 172.21.15.107:0/2668994336 >> 172.21.15.107:0/2668994336 conn(0x7fdf1c06a920 msgr2=0x7fdf1c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 -- 172.21.15.107:0/2668994336 shutdown_connections 2023-11-24T08:19:04.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.271+0000 7fdf23fc9640 1 -- 172.21.15.107:0/2668994336 wait complete. 2023-11-24T08:19:04.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.273+0000 7fdf23fc9640 1 Processor -- start 2023-11-24T08:19:04.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.273+0000 7fdf23fc9640 1 -- start start 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf1c0cc470 0x7fdf1c093770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0ce640 0x7fdf1c154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c163180 con 0x7fdf1c0cf450 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c163a20 con 0x7fdf1c0cc470 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf23fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c09a4c0 con 0x7fdf1c0ce640 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49942/0 (socket says 172.21.15.107:49942) 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 -- 172.21.15.107:0/4130003630 learned_addr learned my addr 172.21.15.107:0/4130003630 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf21d65640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf1c0cc470 0x7fdf1c093770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 -- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0ce640 msgr2=0x7fdf1c154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0ce640 0x7fdf1c154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 -- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf1c0cc470 msgr2=0x7fdf1c093770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf1c0cc470 0x7fdf1c093770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.274+0000 7fdf22566640 1 -- 172.21.15.107:0/4130003630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0094d0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf22566640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400ccd0 tx=0x7fdf1400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400f680 con 0x7fdf1c0cf450 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1400f800 con 0x7fdf1c0cf450 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400fbb0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c1633d0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.275+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c154ab0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.276+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c067470 con 0x7fdf1c0cf450 2023-11-24T08:19:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.277+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fdf1400b040 con 0x7fdf1c0cf450 2023-11-24T08:19:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.278+0000 7fdf127fc640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 0x7fdf0006edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.278+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdf14088d70 con 0x7fdf1c0cf450 2023-11-24T08:19:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.278+0000 7fdf21d65640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 0x7fdf0006edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.279+0000 7fdf21d65640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 0x7fdf0006edc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b7e0 tx=0x7fdf18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.282+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf140559a0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.462+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf1c163b60 con 0x7fdf1c0cf450 2023-11-24T08:19:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.463+0000 7fdf127fc640 1 -- 172.21.15.107:0/4130003630 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16638 (secure 0 0 0) 0x7fdf14058fc0 con 0x7fdf1c0cf450 2023-11-24T08:19:04.466 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:04.466 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:18:55.038112+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:17:38.499480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-11-24T08:17:39.719116+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 msgr2=0x7fdf0006edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 0x7fdf0006edc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b7e0 tx=0x7fdf18008040 comp rx=0 tx=0).stop 2023-11-24T08:19:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 msgr2=0x7fdf1c099b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400ccd0 tx=0x7fdf1400a650 comp rx=0 tx=0).stop 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 shutdown_connections 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdf0006c900 0x7fdf0006edc0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c0cf450 0x7fdf1c099b00 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf1c0ce640 0x7fdf1c154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 --2- 172.21.15.107:0/4130003630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf1c0cc470 0x7fdf1c093770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.466+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 >> 172.21.15.107:0/4130003630 conn(0x7fdf1c06a920 msgr2=0x7fdf1c06d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.467+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 shutdown_connections 2023-11-24T08:19:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.467+0000 7fdf23fc9640 1 -- 172.21.15.107:0/4130003630 wait complete. 2023-11-24T08:19:04.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:19:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.690+0000 7fe2bc153640 1 Processor -- start 2023-11-24T08:19:04.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.691+0000 7fe2bc153640 1 -- start start 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2bc153640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2b41684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2bc153640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4161a10 con 0x7fe2b406e160 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2bc153640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4161cf0 con 0x7fe2b4146eb0 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2bc153640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4161fd0 con 0x7fe2b40cd2e0 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2b96ee640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe2b406e160 0x7fe2b40ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51386/0 (socket says 172.21.15.107:51386) 2023-11-24T08:19:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.692+0000 7fe2b96ee640 1 -- 172.21.15.107:0/1618350624 learned_addr learned my addr 172.21.15.107:0/1618350624 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813062775 0 0) 0x7fe2b4161a10 con 0x7fe2b406e160 2023-11-24T08:19:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2ba6f0640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2b41684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0003520 con 0x7fe2b406e160 2023-11-24T08:19:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268096505 0 0) 0x7fe2a0003520 con 0x7fe2b406e160 2023-11-24T08:19:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 msgr2=0x7fe2b41684e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2b41684e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 >> v1:172.21.15.120:6789/0 conn(0x7fe2b4146eb0 legacy=0x7fe2b4161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4070660 con 0x7fe2b406e160 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe2a4001690 con 0x7fe2b406e160 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 >> v1:172.21.15.107:6789/0 conn(0x7fe2b406e160 legacy=0x7fe2b40ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00039a0 0x7fe2a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.693+0000 7fe2b8eed640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b8eed640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4161fd0 con 0x7fe2a0004d00 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a000ba60 con 0x7fe2a00039a0 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a000bc20 con 0x7fe2b40cd2e0 2023-11-24T08:19:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b96ee640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2b9eef640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00039a0 0x7fe2a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 -- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 msgr2=0x7fe2a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b40cd2e0 0x7fe2a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 -- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00039a0 msgr2=0x7fe2a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a00039a0 0x7fe2a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.694+0000 7fe2ba6f0640 1 -- 172.21.15.107:0/1618350624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0003520 con 0x7fe2a0004d00 2023-11-24T08:19:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.695+0000 7fe2ba6f0640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2a0005180 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009fc0 tx=0x7fe2a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4442344f9e6b7c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac02cc20 con 0x7fe2a0004d00 2023-11-24T08:19:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac00f040 con 0x7fe2a0004d00 2023-11-24T08:19:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2b8eed640 1 -- 172.21.15.107:0/1618350624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac034440 con 0x7fe2a0004d00 2023-11-24T08:19:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2bc153640 1 -- 172.21.15.107:0/1618350624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 msgr2=0x7fe2a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2bc153640 1 --2- 172.21.15.107:0/1618350624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2a0005180 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009fc0 tx=0x7fe2a00057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2ba6f0640 1 -- 172.21.15.107:0/1618350624 reap_dead start 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2bc153640 1 -- 172.21.15.107:0/1618350624 shutdown_connections 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2bc153640 1 -- 172.21.15.107:0/1618350624 >> 172.21.15.107:0/1618350624 conn(0x7fe2b40b0c50 msgr2=0x7fe2b40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.696+0000 7fe2bc153640 1 -- 172.21.15.107:0/1618350624 shutdown_connections 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.697+0000 7fe2bc153640 1 -- 172.21.15.107:0/1618350624 wait complete. 2023-11-24T08:19:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.698+0000 7fe2bc153640 1 Processor -- start 2023-11-24T08:19:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.698+0000 7fe2bc153640 1 -- start start 2023-11-24T08:19:04.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b40b4190 0x7fe2b4141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b4142240 0x7fe2b415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4142b50 con 0x7fe2a0004d00 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4160490 con 0x7fe2b40b4190 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.699+0000 7fe2bc153640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4142870 con 0x7fe2b4142240 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49958/0 (socket says 172.21.15.107:49958) 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 -- 172.21.15.107:0/101373536 learned_addr learned my addr 172.21.15.107:0/101373536 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 -- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b4142240 msgr2=0x7fe2b415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b4142240 0x7fe2b415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 -- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b40b4190 msgr2=0x7fe2b4141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b40b4190 0x7fe2b4141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 -- 172.21.15.107:0/101373536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40b36b0 con 0x7fe2a0004d00 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2b9eef640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e550 tx=0x7fe2b000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.700+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00052e0 con 0x7fe2a0004d00 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.701+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0005440 con 0x7fe2a0004d00 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.701+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b40b3990 con 0x7fe2a0004d00 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.701+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0003810 con 0x7fe2a0004d00 2023-11-24T08:19:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.701+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b40c56c0 con 0x7fe2a0004d00 2023-11-24T08:19:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.702+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b40cb560 con 0x7fe2a0004d00 2023-11-24T08:19:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.703+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fe2b0003970 con 0x7fe2a0004d00 2023-11-24T08:19:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.704+0000 7fe2aa7fc640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 0x7fe29806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.704+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe2b0088230 con 0x7fe2a0004d00 2023-11-24T08:19:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.704+0000 7fe2b96ee640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 0x7fe29806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.705+0000 7fe2b96ee640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 0x7fe29806eda0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4001220 tx=0x7fe2a4001d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.709+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2b0054d40 con 0x7fe2a0004d00 2023-11-24T08:19:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.934+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe2b40c5850 con 0x7fe2a0004d00 2023-11-24T08:19:04.936 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:19:04.935+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:19:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.939+0000 7fe2aa7fc640 1 -- 172.21.15.107:0/101373536 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe2b0058360 con 0x7fe2a0004d00 2023-11-24T08:19:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:19:04.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 msgr2=0x7fe29806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 0x7fe29806eda0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4001220 tx=0x7fe2a4001d50 comp rx=0 tx=0).stop 2023-11-24T08:19:04.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 msgr2=0x7fe2b40c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:04.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e550 tx=0x7fe2b000ea10 comp rx=0 tx=0).stop 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 shutdown_connections 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe29806c8e0 0x7fe29806eda0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2b4142240 0x7fe2b415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2b40b4190 0x7fe2b4141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 --2- 172.21.15.107:0/101373536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a0004d00 0x7fe2b40c5180 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.942+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 >> 172.21.15.107:0/101373536 conn(0x7fe2b40b0c50 msgr2=0x7fe2b40af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.943+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 shutdown_connections 2023-11-24T08:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:04.943+0000 7fe2bc153640 1 -- 172.21.15.107:0/101373536 wait complete. 2023-11-24T08:19:04.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:19:05.069 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:19:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.151+0000 7f20f9d06640 1 Processor -- start 2023-11-24T08:19:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.151+0000 7f20f9d06640 1 -- start start 2023-11-24T08:19:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f9d06640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20f4072620 0x7f20f40ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f9d06640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20f4162490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f9d06640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f4071eb0 con 0x7f20f40b39b0 2023-11-24T08:19:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f9d06640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f40b0b50 con 0x7f20f4072620 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f9d06640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f40b0e30 con 0x7f20f4162000 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f8d04640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f20f40b39b0 0x7f20f40b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51402/0 (socket says 172.21.15.107:51402) 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.152+0000 7f20f8d04640 1 -- 172.21.15.107:0/1988406525 learned_addr learned my addr 172.21.15.107:0/1988406525 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165300825 0 0) 0x7f20f4071eb0 con 0x7f20f40b39b0 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4003520 con 0x7f20f40b39b0 2023-11-24T08:19:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f9505640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20f4162490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f3fff640 1 --2- 172.21.15.107:0/1988406525 >> v2:172.21.15.120:3300/0 conn(0x7f20f4072620 0x7f20f40ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 652806567 0 0) 0x7f20e4003520 con 0x7f20f40b39b0 2023-11-24T08:19:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 msgr2=0x7f20f4162490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20f4162490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 >> v2:172.21.15.120:3300/0 conn(0x7f20f4072620 msgr2=0x7f20f40ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 --2- 172.21.15.107:0/1988406525 >> v2:172.21.15.120:3300/0 conn(0x7f20f4072620 0x7f20f40ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f40b1110 con 0x7f20f40b39b0 2023-11-24T08:19:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f20e8001670 con 0x7f20f40b39b0 2023-11-24T08:19:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 >> v1:172.21.15.107:6789/0 conn(0x7f20f40b39b0 legacy=0x7f20f40b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20e4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e4003fa0 0x7f20e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20e4004d00 0x7f20e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f40b0b50 con 0x7f20e4003fa0 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f40b0e30 con 0x7f20e4004d00 2023-11-24T08:19:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.153+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e400bb40 con 0x7f20f4162000 2023-11-24T08:19:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e4003fa0 0x7f20e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 -- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 msgr2=0x7f20e4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f8d04640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20e4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20f4162000 0x7f20e4003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 -- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20e4004d00 msgr2=0x7f20e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f9505640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20e4004d00 0x7f20e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20e4004d00 0x7f20e4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.154+0000 7f20f3fff640 1 -- 172.21.15.107:0/1988406525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e4003520 con 0x7f20e4003fa0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.155+0000 7f20f3fff640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e4003fa0 0x7f20e40047c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f20e000ba30 tx=0x7f20e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fc60bb4e77b207 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e0032680 con 0x7f20e4003fa0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e00327e0 con 0x7f20e4003fa0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f37fe640 1 -- 172.21.15.107:0/1988406525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e0032b70 con 0x7f20e4003fa0 2023-11-24T08:19:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 -- 172.21.15.107:0/1988406525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e4003fa0 msgr2=0x7f20e40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 --2- 172.21.15.107:0/1988406525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e4003fa0 0x7f20e40047c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f20e000ba30 tx=0x7f20e40057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9505640 1 -- 172.21.15.107:0/1988406525 reap_dead start 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 -- 172.21.15.107:0/1988406525 shutdown_connections 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 -- 172.21.15.107:0/1988406525 >> 172.21.15.107:0/1988406525 conn(0x7f20f406a500 msgr2=0x7f20f406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 -- 172.21.15.107:0/1988406525 shutdown_connections 2023-11-24T08:19:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.156+0000 7f20f9d06640 1 -- 172.21.15.107:0/1988406525 wait complete. 2023-11-24T08:19:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.158+0000 7f20f9d06640 1 Processor -- start 2023-11-24T08:19:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.158+0000 7f20f9d06640 1 -- start start 2023-11-24T08:19:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20e4003fa0 0x7f20f4159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f4163040 0x7f20f41634f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4163a30 con 0x7f20f4146680 2023-11-24T08:19:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f415b1d0 con 0x7f20f4163040 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f9d06640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f415b660 con 0x7f20e4003fa0 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49984/0 (socket says 172.21.15.107:49984) 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.159+0000 7f20f3fff640 1 -- 172.21.15.107:0/3100320989 learned_addr learned my addr 172.21.15.107:0/3100320989 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f8d04640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20e4003fa0 0x7f20f4159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 -- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20e4003fa0 msgr2=0x7f20f4159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20e4003fa0 0x7f20f4159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 -- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f4163040 msgr2=0x7f20f41634f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f9505640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f4163040 0x7f20f41634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f4163040 0x7f20f41634f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 -- 172.21.15.107:0/3100320989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f415a5c0 con 0x7f20f4146680 2023-11-24T08:19:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f3fff640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f20e0035040 tx=0x7f20e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e0030b20 con 0x7f20f4146680 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e0030c80 con 0x7f20f4146680 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e003b420 con 0x7f20f4146680 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.160+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f415a8a0 con 0x7f20f4146680 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.161+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f415ae60 con 0x7f20f4146680 2023-11-24T08:19:05.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.162+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f404e840 con 0x7f20f4146680 2023-11-24T08:19:05.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.162+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f20e003b5a0 con 0x7f20f4146680 2023-11-24T08:19:05.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.164+0000 7f20f17fa640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 0x7f20d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:05.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.164+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f20e00aaab0 con 0x7f20f4146680 2023-11-24T08:19:05.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.164+0000 7f20f8d04640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 0x7f20d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:05.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.165+0000 7f20f8d04640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 0x7f20d806ed70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20e80017a0 tx=0x7f20e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:05.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.169+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20e0077660 con 0x7f20f4146680 2023-11-24T08:19:05.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:05.398+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f20f415b310 con 0x7f20f4146680 2023-11-24T08:19:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.048+0000 7f20f17fa640 1 -- 172.21.15.107:0/3100320989 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7f20e00360a0 con 0x7f20f4146680 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.051+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 msgr2=0x7f20d806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.051+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 0x7f20d806ed70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20e80017a0 tx=0x7f20e8009040 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.051+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 msgr2=0x7f20f4145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.051+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f20e0035040 tx=0x7f20e0004730 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 shutdown_connections 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20d806c8b0 0x7f20d806ed70 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f4163040 0x7f20f41634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f4146680 0x7f20f4145d60 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 --2- 172.21.15.107:0/3100320989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20e4003fa0 0x7f20f4159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 >> 172.21.15.107:0/3100320989 conn(0x7f20f406a500 msgr2=0x7f20f406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:06.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 shutdown_connections 2023-11-24T08:19:06.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.052+0000 7f20f9d06640 1 -- 172.21.15.107:0/3100320989 wait complete. 2023-11-24T08:19:06.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:19:06.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.282+0000 7f769a978640 1 Processor -- start 2023-11-24T08:19:06.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.282+0000 7f769a978640 1 -- start start 2023-11-24T08:19:06.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.283+0000 7f769a978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76940b2400 0x7f769406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.283+0000 7f769a978640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76940cfed0 con 0x7f76940af400 2023-11-24T08:19:06.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.283+0000 7f769a978640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76940caae0 con 0x7f769415d920 2023-11-24T08:19:06.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.283+0000 7f769a978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940cadc0 con 0x7f76940b2400 2023-11-24T08:19:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.284+0000 7f7698f15640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76940af400 0x7f76940ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51418/0 (socket says 172.21.15.107:51418) 2023-11-24T08:19:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.284+0000 7f7698f15640 1 -- 172.21.15.107:0/99248751 learned_addr learned my addr 172.21.15.107:0/99248751 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.284+0000 7f76937fe640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76940b2400 0x7f769406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.284+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737398878 0 0) 0x7f76940cfed0 con 0x7f76940af400 2023-11-24T08:19:06.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.284+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7680003520 con 0x7f76940af400 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390489887 0 0) 0x7f7680003520 con 0x7f76940af400 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76940b2400 msgr2=0x7f769406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76940b2400 0x7f769406a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 >> v1:172.21.15.120:6789/0 conn(0x7f769415d920 legacy=0x7f76940acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940cb0a0 con 0x7f76940af400 2023-11-24T08:19:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f768c001670 con 0x7f76940af400 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 >> v1:172.21.15.107:6789/0 conn(0x7f76940af400 legacy=0x7f76940ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76800039a0 0x7f7680003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800047c0 0x7f7680004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7680005180 0x7f768000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940cadc0 con 0x7f76800039a0 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768000c030 con 0x7f7680005180 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768000c1f0 con 0x7f76800047c0 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f76937fe640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76940b2400 0x7f769406a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:19:06.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.285+0000 7f7693fff640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76800039a0 0x7f7680003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7698f15640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7680005180 0x7f768000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7693fff640 1 -- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800047c0 msgr2=0x7f7680004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7693fff640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800047c0 0x7f7680004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7693fff640 1 -- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7680005180 msgr2=0x7f768000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7693fff640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7680005180 0x7f768000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.286+0000 7f7693fff640 1 -- 172.21.15.107:0/99248751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7680003520 con 0x7f76800039a0 2023-11-24T08:19:06.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.287+0000 7f7693fff640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76800039a0 0x7f7680003dc0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f7684001750 tx=0x7f76800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b21b7be78c6ec3b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:06.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.287+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840052d0 con 0x7f76800039a0 2023-11-24T08:19:06.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.287+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768402e050 con 0x7f76800039a0 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.287+0000 7f7692ffd640 1 -- 172.21.15.107:0/99248751 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840035f0 con 0x7f76800039a0 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 -- 172.21.15.107:0/99248751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76800039a0 msgr2=0x7f7680003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 --2- 172.21.15.107:0/99248751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76800039a0 0x7f7680003dc0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f7684001750 tx=0x7f76800057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f7698f15640 1 -- 172.21.15.107:0/99248751 reap_dead start 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 -- 172.21.15.107:0/99248751 shutdown_connections 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 -- 172.21.15.107:0/99248751 >> 172.21.15.107:0/99248751 conn(0x7f76940999d0 msgr2=0x7f76940b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 -- 172.21.15.107:0/99248751 shutdown_connections 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.288+0000 7f769a978640 1 -- 172.21.15.107:0/99248751 wait complete. 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.290+0000 7f769a978640 1 Processor -- start 2023-11-24T08:19:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.290+0000 7f769a978640 1 -- start start 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800039a0 0x7f76940b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769415ff50 0x7f76941603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b1b40 con 0x7f76940c4180 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7694160920 con 0x7f769415ff50 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f769a978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b1d30 con 0x7f76800039a0 2023-11-24T08:19:06.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50004/0 (socket says 172.21.15.107:50004) 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 -- 172.21.15.107:0/1500625109 learned_addr learned my addr 172.21.15.107:0/1500625109 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 -- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800039a0 msgr2=0x7f76940b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800039a0 0x7f76940b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 -- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769415ff50 msgr2=0x7f76941603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769415ff50 0x7f76941603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.291+0000 7f76937fe640 1 -- 172.21.15.107:0/1500625109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940c7810 con 0x7f76940c4180 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f76937fe640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f767c007e20 tx=0x7f767c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c003990 con 0x7f76940c4180 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c00f040 con 0x7f76940c4180 2023-11-24T08:19:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c003d70 con 0x7f76940c4180 2023-11-24T08:19:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76940c7af0 con 0x7f76940c4180 2023-11-24T08:19:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.292+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76940c4600 con 0x7f76940c4180 2023-11-24T08:19:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.293+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694067470 con 0x7f76940c4180 2023-11-24T08:19:06.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.294+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f767c004030 con 0x7f76940c4180 2023-11-24T08:19:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.295+0000 7f7690ff9640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 0x7f767406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.295+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f767c0885b0 con 0x7f76940c4180 2023-11-24T08:19:06.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.295+0000 7f7693fff640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 0x7f767406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:06.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.296+0000 7f7693fff640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 0x7f767406eda0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f7684033000 tx=0x7f768402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:06.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.299+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f767c0bfe90 con 0x7f76940c4180 2023-11-24T08:19:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:06.476+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76940cd270 con 0x7f76940c4180 2023-11-24T08:19:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.132+0000 7f7690ff9640 1 -- 172.21.15.107:0/1500625109 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v166) v1 ==== 181+0+0 (secure 0 0 0) 0x7f767c01a020 con 0x7f76940c4180 2023-11-24T08:19:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:19:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.136+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 msgr2=0x7f767406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.136+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 0x7f767406eda0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f7684033000 tx=0x7f768402f000 comp rx=0 tx=0).stop 2023-11-24T08:19:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.136+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 msgr2=0x7f769415b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.136+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f767c007e20 tx=0x7f767c00d4a0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 shutdown_connections 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f767406c8e0 0x7f767406eda0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769415ff50 0x7f76941603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76940c4180 0x7f769415b930 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 --2- 172.21.15.107:0/1500625109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76800039a0 0x7f76940b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 >> 172.21.15.107:0/1500625109 conn(0x7f76940999d0 msgr2=0x7f769406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 shutdown_connections 2023-11-24T08:19:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.138+0000 7f769a978640 1 -- 172.21.15.107:0/1500625109 wait complete. 2023-11-24T08:19:07.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:19:07.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.368+0000 7faba893c640 1 Processor -- start 2023-11-24T08:19:07.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.369+0000 7faba893c640 1 -- start start 2023-11-24T08:19:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba893c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba415e8f0 0x7faba40cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba893c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406ecb0 0x7faba40cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba893c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba40cebc0 con 0x7faba40cf8f0 2023-11-24T08:19:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba893c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40b3a50 con 0x7faba415e8f0 2023-11-24T08:19:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba893c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40ccc00 con 0x7faba406ecb0 2023-11-24T08:19:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba415e8f0 0x7faba40cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faba415e8f0 0x7faba40cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50012/0 (socket says 172.21.15.107:50012) 2023-11-24T08:19:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba37fe640 1 -- 172.21.15.107:0/1623601054 learned_addr learned my addr 172.21.15.107:0/1623601054 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.370+0000 7faba3fff640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406ecb0 0x7faba40cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.371+0000 7faba37fe640 1 -- 172.21.15.107:0/1623601054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406ecb0 msgr2=0x7faba40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.371+0000 7faba37fe640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406ecb0 0x7faba40cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.371+0000 7faba37fe640 1 -- 172.21.15.107:0/1623601054 >> v1:172.21.15.120:6789/0 conn(0x7faba40cf8f0 legacy=0x7faba406e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.371+0000 7faba37fe640 1 -- 172.21.15.107:0/1623601054 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40cce80 con 0x7faba415e8f0 2023-11-24T08:19:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.372+0000 7faba37fe640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba40cc6c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fab94009e50 tx=0x7fab9402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d40a767ca5b5e31 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.372+0000 7faba27fc640 1 -- 172.21.15.107:0/1623601054 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9402cb90 con 0x7faba415e8f0 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.372+0000 7faba27fc640 1 -- 172.21.15.107:0/1623601054 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9402ccf0 con 0x7faba415e8f0 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.372+0000 7faba27fc640 1 -- 172.21.15.107:0/1623601054 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94034610 con 0x7faba415e8f0 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 -- 172.21.15.107:0/1623601054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 msgr2=0x7faba40cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba40cc6c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fab94009e50 tx=0x7fab9402bff0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 -- 172.21.15.107:0/1623601054 shutdown_connections 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba406ecb0 0x7faba40cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 --2- 172.21.15.107:0/1623601054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba40cc6c0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 -- 172.21.15.107:0/1623601054 >> 172.21.15.107:0/1623601054 conn(0x7faba40b2ae0 msgr2=0x7faba40b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 -- 172.21.15.107:0/1623601054 shutdown_connections 2023-11-24T08:19:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.373+0000 7faba893c640 1 -- 172.21.15.107:0/1623601054 wait complete. 2023-11-24T08:19:07.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.375+0000 7faba893c640 1 Processor -- start 2023-11-24T08:19:07.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.375+0000 7faba893c640 1 -- start start 2023-11-24T08:19:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406ecb0 0x7faba40c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 0x7faba414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40b48e0 con 0x7faba415e8f0 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba414ea40 con 0x7faba406ecb0 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba893c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40b3c60 con 0x7faba40cf8f0 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50018/0 (socket says 172.21.15.107:50018) 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.376+0000 7faba3fff640 1 -- 172.21.15.107:0/2434240812 learned_addr learned my addr 172.21.15.107:0/2434240812 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba2ffd640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 0x7faba414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 -- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 msgr2=0x7faba414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba37fe640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406ecb0 0x7faba40c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 0x7faba414dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 -- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406ecb0 msgr2=0x7faba40c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406ecb0 0x7faba40c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 -- 172.21.15.107:0/2434240812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab94009b30 con 0x7faba415e8f0 2023-11-24T08:19:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba2ffd640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 0x7faba414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba3fff640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00d800 tx=0x7fab9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00b6c0 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c00b840 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00bbf0 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.377+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba40b3f40 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.378+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba40b4500 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.379+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba40cd490 con 0x7faba415e8f0 2023-11-24T08:19:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.380+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fab9c00bd50 con 0x7faba415e8f0 2023-11-24T08:19:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.380+0000 7fab9bfff640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 0x7fab8406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.381+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fab9c088a00 con 0x7faba415e8f0 2023-11-24T08:19:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.381+0000 7faba37fe640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 0x7fab8406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.381+0000 7faba37fe640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 0x7fab8406ed70 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fab94035000 tx=0x7fab94030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.385+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab9c0555b0 con 0x7faba415e8f0 2023-11-24T08:19:07.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:07.575+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faba40b4a20 con 0x7faba415e8f0 2023-11-24T08:19:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.134+0000 7fab9bfff640 1 -- 172.21.15.107:0/2434240812 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v167) v1 ==== 169+0+0 (secure 0 0 0) 0x7faba40b4a20 con 0x7faba415e8f0 2023-11-24T08:19:08.137 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:19:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 msgr2=0x7fab8406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 0x7fab8406ed70 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fab94035000 tx=0x7fab94030000 comp rx=0 tx=0).stop 2023-11-24T08:19:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 msgr2=0x7faba414e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00d800 tx=0x7fab9c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 shutdown_connections 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab8406c8b0 0x7fab8406ed70 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba415e8f0 0x7faba414e500 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faba40cf8f0 0x7faba414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 --2- 172.21.15.107:0/2434240812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba406ecb0 0x7faba40c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.138+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 >> 172.21.15.107:0/2434240812 conn(0x7faba40b2ae0 msgr2=0x7faba40aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.139+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 shutdown_connections 2023-11-24T08:19:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.139+0000 7faba893c640 1 -- 172.21.15.107:0/2434240812 wait complete. 2023-11-24T08:19:08.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:19:08.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.377+0000 7f7c13b16640 1 Processor -- start 2023-11-24T08:19:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.378+0000 7f7c13b16640 1 -- start start 2023-11-24T08:19:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.379+0000 7f7c13b16640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c0c13e3e0 0x7f7c0c0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.379+0000 7f7c13b16640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.379+0000 7f7c13b16640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c15c630 con 0x7f7c0c13ea00 2023-11-24T08:19:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.379+0000 7f7c13b16640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c13e7c0 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.379+0000 7f7c13b16640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c15b810 con 0x7f7c0c13e3e0 2023-11-24T08:19:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32822/0 (socket says 172.21.15.107:32822) 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 -- 172.21.15.107:0/3953815985 learned_addr learned my addr 172.21.15.107:0/3953815985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c118b2640 1 --2- 172.21.15.107:0/3953815985 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c13e3e0 0x7f7c0c0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 -- 172.21.15.107:0/3953815985 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c13e3e0 msgr2=0x7f7c0c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 --2- 172.21.15.107:0/3953815985 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c13e3e0 0x7f7c0c0ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 -- 172.21.15.107:0/3953815985 >> v1:172.21.15.120:6789/0 conn(0x7f7c0c13ea00 legacy=0x7f7c0c0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.380+0000 7f7c120b3640 1 -- 172.21.15.107:0/3953815985 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c15ba20 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.382+0000 7f7c120b3640 1 --2- 172.21.15.107:0/3953815985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009870 tx=0x7f7c0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7640180cfad75c4c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.382+0000 7f7c108b0640 1 -- 172.21.15.107:0/3953815985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040317e0 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.382+0000 7f7c108b0640 1 -- 172.21.15.107:0/3953815985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c04031940 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.382+0000 7f7c108b0640 1 -- 172.21.15.107:0/3953815985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04031cd0 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 -- 172.21.15.107:0/3953815985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c0cf4f0 msgr2=0x7f7c0c0700b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 --2- 172.21.15.107:0/3953815985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009870 tx=0x7f7c0402b880 comp rx=0 tx=0).stop 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 -- 172.21.15.107:0/3953815985 shutdown_connections 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 --2- 172.21.15.107:0/3953815985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c0cf4f0 0x7f7c0c0700b0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 --2- 172.21.15.107:0/3953815985 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c13e3e0 0x7f7c0c0ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 -- 172.21.15.107:0/3953815985 >> 172.21.15.107:0/3953815985 conn(0x7f7c0c0b2e40 msgr2=0x7f7c0c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 -- 172.21.15.107:0/3953815985 shutdown_connections 2023-11-24T08:19:08.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.383+0000 7f7c13b16640 1 -- 172.21.15.107:0/3953815985 wait complete. 2023-11-24T08:19:08.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.385+0000 7f7c13b16640 1 Processor -- start 2023-11-24T08:19:08.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.385+0000 7f7c13b16640 1 -- start start 2023-11-24T08:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 0x7f7c0c1f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 0x7f7c0c13bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c13a730 con 0x7f7c0c13ea00 2023-11-24T08:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c13bc00 con 0x7f7c0c13e3e0 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.386+0000 7f7c13b16640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c13a910 con 0x7f7c0c0cf4f0 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32834/0 (socket says 172.21.15.107:32834) 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 -- 172.21.15.107:0/2054197831 learned_addr learned my addr 172.21.15.107:0/2054197831 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c118b2640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 0x7f7c0c1f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c110b1640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 0x7f7c0c13bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 -- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 msgr2=0x7f7c0c1f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 0x7f7c0c1f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 -- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 msgr2=0x7f7c0c13bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 0x7f7c0c13bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c120b3640 1 -- 172.21.15.107:0/2054197831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c040094d0 con 0x7f7c0c13ea00 2023-11-24T08:19:08.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c118b2640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 0x7f7c0c1f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.387+0000 7f7c110b1640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 0x7f7c0c13bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c120b3640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009870 tx=0x7f7c040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04032880 con 0x7f7c0c13ea00 2023-11-24T08:19:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c040329e0 con 0x7f7c0c13ea00 2023-11-24T08:19:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04032d90 con 0x7f7c0c13ea00 2023-11-24T08:19:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c13abf0 con 0x7f7c0c13ea00 2023-11-24T08:19:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.388+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c13b1b0 con 0x7f7c0c13ea00 2023-11-24T08:19:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.389+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c04e8b0 con 0x7f7c0c13ea00 2023-11-24T08:19:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.391+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f7c0404e450 con 0x7f7c0c13ea00 2023-11-24T08:19:08.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.391+0000 7f7c0a7fc640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 0x7f7bf006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.392+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c040aa510 con 0x7f7c0c13ea00 2023-11-24T08:19:08.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.392+0000 7f7c118b2640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 0x7f7bf006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.392+0000 7f7c118b2640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 0x7f7bf006eda0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7c00007330 tx=0x7f7c0000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.398+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c040b4020 con 0x7f7c0c13ea00 2023-11-24T08:19:08.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.591+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7c0c15d370 con 0x7f7c0c13ea00 2023-11-24T08:19:08.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.597+0000 7f7c0a7fc640 1 -- 172.21.15.107:0/2054197831 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c04077170 con 0x7f7c0c13ea00 2023-11-24T08:19:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 msgr2=0x7f7bf006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 0x7f7bf006eda0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7c00007330 tx=0x7f7c0000b040 comp rx=0 tx=0).stop 2023-11-24T08:19:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 msgr2=0x7f7c0c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009870 tx=0x7f7c040046c0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 shutdown_connections 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7bf006c8e0 0x7f7bf006eda0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c13ea00 0x7f7c0c0c73a0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c13e3e0 0x7f7c0c13bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 --2- 172.21.15.107:0/2054197831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c0cf4f0 0x7f7c0c1f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 >> 172.21.15.107:0/2054197831 conn(0x7f7c0c0b2e40 msgr2=0x7f7c0c15ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 shutdown_connections 2023-11-24T08:19:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.600+0000 7f7c13b16640 1 -- 172.21.15.107:0/2054197831 wait complete. 2023-11-24T08:19:08.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2023-11-24T08:19:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.834+0000 7f4cbf09b640 1 Processor -- start 2023-11-24T08:19:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.834+0000 7f4cbf09b640 1 -- start start 2023-11-24T08:19:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cbf09b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4cb80c49f0 0x7f4cb815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cbf09b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb815ec90 con 0x7f4cb806de10 2023-11-24T08:19:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cbf09b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80cd530 con 0x7f4cb8072160 2023-11-24T08:19:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cbf09b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb80c4ed0 con 0x7f4cb80c49f0 2023-11-24T08:19:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cb7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4cb806de10 0x7f4cb80705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43154/0 (socket says 172.21.15.107:43154) 2023-11-24T08:19:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cb7fff640 1 -- 172.21.15.107:0/3267335243 learned_addr learned my addr 172.21.15.107:0/3267335243 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3267335243 >> v2:172.21.15.120:3300/0 conn(0x7f4cb80c49f0 0x7f4cb815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.836+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413315885 0 0) 0x7f4ca4001110 con 0x7f4cb806de10 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca8003520 con 0x7f4cb806de10 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735583548 0 0) 0x7f4ca8003520 con 0x7f4cb806de10 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 >> v2:172.21.15.120:3300/0 conn(0x7f4cb80c49f0 msgr2=0x7f4cb815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 --2- 172.21.15.107:0/3267335243 >> v2:172.21.15.120:3300/0 conn(0x7f4cb80c49f0 0x7f4cb815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 >> v1:172.21.15.120:6789/0 conn(0x7f4cb8072160 legacy=0x7f4cb806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb80c51b0 con 0x7f4cb806de10 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4ca4001800 con 0x7f4cb806de10 2023-11-24T08:19:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 >> v1:172.21.15.107:6789/0 conn(0x7f4cb806de10 legacy=0x7f4cb80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8003970 0x7f4ca8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ca80047c0 0x7f4ca8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 0x7f4ca800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb80c4ed0 con 0x7f4ca8005180 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca800c020 con 0x7f4ca80047c0 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca800c1e0 con 0x7f4ca8003970 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.837+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 0x7f4ca800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cb7fff640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ca80047c0 0x7f4ca8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbce37640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8003970 0x7f4ca8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbd638640 1 -- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8003970 msgr2=0x7f4ca8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8003970 0x7f4ca8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbd638640 1 -- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ca80047c0 msgr2=0x7f4ca8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:08.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ca80047c0 0x7f4ca8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.838+0000 7f4cbd638640 1 -- 172.21.15.107:0/3267335243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8003520 con 0x7f4ca8005180 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.839+0000 7f4cbd638640 1 -- 172.21.15.107:0/3267335243 reap_dead start 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.839+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 0x7f4ca800b980 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009f90 tx=0x7f4ca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe96e0024c41df81 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.839+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb002cc20 con 0x7f4ca8005180 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.839+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb000f040 con 0x7f4ca8005180 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.839+0000 7f4cb77fe640 1 -- 172.21.15.107:0/3267335243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0003990 con 0x7f4ca8005180 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 msgr2=0x7f4ca800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 0x7f4ca800b980 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009f90 tx=0x7f4ca80057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3267335243 shutdown_connections 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3267335243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005180 0x7f4ca800b980 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3267335243 >> 172.21.15.107:0/3267335243 conn(0x7f4cb80b0c50 msgr2=0x7f4cb80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3267335243 shutdown_connections 2023-11-24T08:19:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.840+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3267335243 wait complete. 2023-11-24T08:19:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.842+0000 7f4cbf09b640 1 Processor -- start 2023-11-24T08:19:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.842+0000 7f4cbf09b640 1 -- start start 2023-11-24T08:19:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8005180 0x7f4cb815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb809b960 0x7f4cb80af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb809a8e0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb809abc0 con 0x7f4cb809b960 2023-11-24T08:19:08.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbf09b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb809aea0 con 0x7f4ca8005180 2023-11-24T08:19:08.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32852/0 (socket says 172.21.15.107:32852) 2023-11-24T08:19:08.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 -- 172.21.15.107:0/3162320540 learned_addr learned my addr 172.21.15.107:0/3162320540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:08.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 -- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8005180 msgr2=0x7f4cb815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8005180 0x7f4cb815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 -- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb809b960 msgr2=0x7f4cb80af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb809b960 0x7f4cb80af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.843+0000 7f4cbd638640 1 -- 172.21.15.107:0/3162320540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb809b180 con 0x7f4cb809a3e0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cbd638640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002c690 tx=0x7f4cb0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0003da0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb00358d0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb809b460 con 0x7f4cb809a3e0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0035c60 con 0x7f4cb809a3e0 2023-11-24T08:19:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.844+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb80afe00 con 0x7f4cb809a3e0 2023-11-24T08:19:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.845+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb80afff0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.846+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f4cb0035dc0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.847+0000 7f4cb57fa640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 0x7f4c9c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.847+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4cb00ab0e0 con 0x7f4cb809a3e0 2023-11-24T08:19:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.847+0000 7f4cbce37640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 0x7f4c9c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.848+0000 7f4cbce37640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 0x7f4c9c06eda0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4cac000f60 tx=0x7f4cac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:08.851+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cb0077d20 con 0x7f4cb809a3e0 2023-11-24T08:19:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.040+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f4cb80ce660 con 0x7f4cb809a3e0 2023-11-24T08:19:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.046+0000 7f4cb57fa640 1 -- 172.21.15.107:0/3162320540 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v46) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4cb007b340 con 0x7f4cb809a3e0 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.048+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 msgr2=0x7f4c9c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.048+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 0x7f4c9c06eda0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4cac000f60 tx=0x7f4cac009040 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.048+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 msgr2=0x7f4cb815a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.048+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002c690 tx=0x7f4cb0009750 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 shutdown_connections 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4c9c06c8e0 0x7f4c9c06eda0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cb809a3e0 0x7f4cb815a470 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb809b960 0x7f4cb80af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 --2- 172.21.15.107:0/3162320540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ca8005180 0x7f4cb815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 >> 172.21.15.107:0/3162320540 conn(0x7f4cb80b0c50 msgr2=0x7f4cb80cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:09.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 shutdown_connections 2023-11-24T08:19:09.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.049+0000 7f4cbf09b640 1 -- 172.21.15.107:0/3162320540 wait complete. 2023-11-24T08:19:09.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source' 2023-11-24T08:19:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.281+0000 7f68c2c7c640 1 Processor -- start 2023-11-24T08:19:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.282+0000 7f68c2c7c640 1 -- start start 2023-11-24T08:19:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.283+0000 7f68c2c7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.283+0000 7f68c2c7c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68bc0b2a70 0x7f68bc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.283+0000 7f68c2c7c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc15ab80 con 0x7f68bc0b2630 2023-11-24T08:19:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.283+0000 7f68c2c7c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68bc15a990 con 0x7f68bc0b2a70 2023-11-24T08:19:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.283+0000 7f68c2c7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc15e680 con 0x7f68bc0b2250 2023-11-24T08:19:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.284+0000 7f68c0a18640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.284+0000 7f68c0a18640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:37258/0 (socket says 172.21.15.107:37258) 2023-11-24T08:19:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.284+0000 7f68c0a18640 1 -- 172.21.15.107:0/312857317 learned_addr learned my addr 172.21.15.107:0/312857317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.284+0000 7f68c1219640 1 --2- 172.21.15.107:0/312857317 >> v2:172.21.15.120:3300/0 conn(0x7f68bc0b2a70 0x7f68bc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.285+0000 7f68c0a18640 1 -- 172.21.15.107:0/312857317 >> v2:172.21.15.120:3300/0 conn(0x7f68bc0b2a70 msgr2=0x7f68bc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.285+0000 7f68c0a18640 1 --2- 172.21.15.107:0/312857317 >> v2:172.21.15.120:3300/0 conn(0x7f68bc0b2a70 0x7f68bc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.285+0000 7f68c0a18640 1 -- 172.21.15.107:0/312857317 >> v1:172.21.15.120:6789/0 conn(0x7f68bc0b2630 legacy=0x7f68bc0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.285+0000 7f68c0a18640 1 -- 172.21.15.107:0/312857317 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc15e960 con 0x7f68bc0b2250 2023-11-24T08:19:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.286+0000 7f68c0a18640 1 --2- 172.21.15.107:0/312857317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009870 tx=0x7f68b002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65767068eff7751 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68bb7fe640 1 -- 172.21.15.107:0/312857317 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0030840 con 0x7f68bc0b2250 2023-11-24T08:19:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68bb7fe640 1 -- 172.21.15.107:0/312857317 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b00309a0 con 0x7f68bc0b2250 2023-11-24T08:19:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68bb7fe640 1 -- 172.21.15.107:0/312857317 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0030d30 con 0x7f68bc0b2250 2023-11-24T08:19:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 -- 172.21.15.107:0/312857317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 msgr2=0x7f68bc15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/312857317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009870 tx=0x7f68b002b980 comp rx=0 tx=0).stop 2023-11-24T08:19:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 -- 172.21.15.107:0/312857317 shutdown_connections 2023-11-24T08:19:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/312857317 >> v2:172.21.15.120:3300/0 conn(0x7f68bc0b2a70 0x7f68bc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/312857317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2250 0x7f68bc15df80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 -- 172.21.15.107:0/312857317 >> 172.21.15.107:0/312857317 conn(0x7f68bc06a920 msgr2=0x7f68bc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 -- 172.21.15.107:0/312857317 shutdown_connections 2023-11-24T08:19:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.287+0000 7f68c2c7c640 1 -- 172.21.15.107:0/312857317 wait complete. 2023-11-24T08:19:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.289+0000 7f68c2c7c640 1 Processor -- start 2023-11-24T08:19:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.289+0000 7f68c2c7c640 1 -- start start 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68bc0b2630 0x7f68bc1f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2a70 0x7f68bc1ecf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0ac560 con 0x7f68bc0b2250 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0ac840 con 0x7f68bc0b2630 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c2c7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0acb20 con 0x7f68bc0b2a70 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32866/0 (socket says 172.21.15.107:32866) 2023-11-24T08:19:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.290+0000 7f68c0a18640 1 -- 172.21.15.107:0/4186532374 learned_addr learned my addr 172.21.15.107:0/4186532374 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68bbfff640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68bc0b2630 0x7f68bc1f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c1219640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2a70 0x7f68bc1ecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 -- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2a70 msgr2=0x7f68bc1ecf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2a70 0x7f68bc1ecf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 -- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68bc0b2630 msgr2=0x7f68bc1f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68bc0b2630 0x7f68bc1f6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 -- 172.21.15.107:0/4186532374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b00094d0 con 0x7f68bc0b2250 2023-11-24T08:19:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68c0a18640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f68b0000c00 tx=0x7f68b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.291+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0039c90 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.292+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0039df0 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.292+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0043640 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.292+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc0cd7c0 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.292+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc0ad0e0 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.293+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68bc067470 con 0x7f68bc0b2250 2023-11-24T08:19:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.294+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f68b00437a0 con 0x7f68bc0b2250 2023-11-24T08:19:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.295+0000 7f68b97fa640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 0x7f68a006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:09.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.295+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68b003e070 con 0x7f68bc0b2250 2023-11-24T08:19:09.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.295+0000 7f68bbfff640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 0x7f68a006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:09.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.296+0000 7f68bbfff640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 0x7f68a006ed70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b1d00 tx=0x7f68a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:09.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.299+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68b0078020 con 0x7f68bc0b2250 2023-11-24T08:19:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:09.560+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source"]} v 0) v1 -- 0x7f68bc159320 con 0x7f68bc0b2250 2023-11-24T08:19:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.164+0000 7f68b97fa640 1 -- 172.21.15.107:0/4186532374 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_in_progress_source"]}]=0 v715) v1 ==== 172+0+0 (secure 0 0 0) 0x7f68b00340f0 con 0x7f68bc0b2250 2023-11-24T08:19:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 msgr2=0x7f68a006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 0x7f68a006ed70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b1d00 tx=0x7f68a8009040 comp rx=0 tx=0).stop 2023-11-24T08:19:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 msgr2=0x7f68bc0c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f68b0000c00 tx=0x7f68b0004730 comp rx=0 tx=0).stop 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 shutdown_connections 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f68a006c8b0 0x7f68a006ed70 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68bc0b2a70 0x7f68bc1ecf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68bc0b2630 0x7f68bc1f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 --2- 172.21.15.107:0/4186532374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68bc0b2250 0x7f68bc0c9580 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 >> 172.21.15.107:0/4186532374 conn(0x7f68bc06a920 msgr2=0x7f68bc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 shutdown_connections 2023-11-24T08:19:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.168+0000 7f68c2c7c640 1 -- 172.21.15.107:0/4186532374 wait complete. 2023-11-24T08:19:10.191 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_source (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:19:10.192 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_inherit_quota_attrs (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:19:10.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:10.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.400+0000 7fa1f1490640 1 Processor -- start 2023-11-24T08:19:10.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.400+0000 7fa1f1490640 1 -- start start 2023-11-24T08:19:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.402+0000 7fa1f1490640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1ec157c70 0x7fa1ec161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.402+0000 7fa1f1490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cf450 0x7fa1ec0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.402+0000 7fa1f1490640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0ce0b0 con 0x7fa1ec15dc80 2023-11-24T08:19:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.402+0000 7fa1f1490640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cfe10 con 0x7fa1ec157c70 2023-11-24T08:19:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.402+0000 7fa1f1490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec161960 con 0x7fa1ec0cf450 2023-11-24T08:19:10.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.403+0000 7fa1eaffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1ec157c70 0x7fa1ec161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.403+0000 7fa1ea7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cf450 0x7fa1ec0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.403+0000 7fa1eaffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1ec157c70 0x7fa1ec161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:41542/0 (socket says 172.21.15.107:41542) 2023-11-24T08:19:10.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.403+0000 7fa1eaffd640 1 -- 172.21.15.107:0/1745141024 learned_addr learned my addr 172.21.15.107:0/1745141024 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:10.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.404+0000 7fa1eaffd640 1 -- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cf450 msgr2=0x7fa1ec0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:10.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.404+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cf450 0x7fa1ec0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.404+0000 7fa1eaffd640 1 -- 172.21.15.107:0/1745141024 >> v1:172.21.15.120:6789/0 conn(0x7fa1ec15dc80 legacy=0x7fa1ec15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.404+0000 7fa1eaffd640 1 -- 172.21.15.107:0/1745141024 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec15e8d0 con 0x7fa1ec157c70 2023-11-24T08:19:10.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.406+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec157c70 0x7fa1ec161df0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009870 tx=0x7fa1dc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa817c6b2c1cccab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.406+0000 7fa1e9ffb640 1 -- 172.21.15.107:0/1745141024 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc031640 con 0x7fa1ec157c70 2023-11-24T08:19:10.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.406+0000 7fa1e9ffb640 1 -- 172.21.15.107:0/1745141024 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc0317a0 con 0x7fa1ec157c70 2023-11-24T08:19:10.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.406+0000 7fa1e9ffb640 1 -- 172.21.15.107:0/1745141024 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc031b30 con 0x7fa1ec157c70 2023-11-24T08:19:10.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 -- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec157c70 msgr2=0x7fa1ec161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 --2- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec157c70 0x7fa1ec161df0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009870 tx=0x7fa1dc02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 -- 172.21.15.107:0/1745141024 shutdown_connections 2023-11-24T08:19:10.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 --2- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec0cf450 0x7fa1ec0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 --2- 172.21.15.107:0/1745141024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec157c70 0x7fa1ec161df0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 -- 172.21.15.107:0/1745141024 >> 172.21.15.107:0/1745141024 conn(0x7fa1ec06a920 msgr2=0x7fa1ec06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:10.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 -- 172.21.15.107:0/1745141024 shutdown_connections 2023-11-24T08:19:10.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.407+0000 7fa1f1490640 1 -- 172.21.15.107:0/1745141024 wait complete. 2023-11-24T08:19:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.409+0000 7fa1f1490640 1 Processor -- start 2023-11-24T08:19:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.410+0000 7fa1f1490640 1 -- start start 2023-11-24T08:19:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 0x7fa1ec1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 0x7fa1ec09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cc890 con 0x7fa1ec157c70 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1451b0 con 0x7fa1ec0cf450 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1f1490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec145470 con 0x7fa1ec15dc80 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32880/0 (socket says 172.21.15.107:32880) 2023-11-24T08:19:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1ea7fc640 1 -- 172.21.15.107:0/3110574766 learned_addr learned my addr 172.21.15.107:0/3110574766 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 0x7fa1ec1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.411+0000 7fa1eb7fe640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 0x7fa1ec09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 -- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 msgr2=0x7fa1ec09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 0x7fa1ec09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 -- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 msgr2=0x7fa1ec1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 0x7fa1ec1f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 -- 172.21.15.107:0/3110574766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0094d0 con 0x7fa1ec157c70 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1eb7fe640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 0x7fa1ec09bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 0x7fa1ec1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1ea7fc640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400efc0 tx=0x7fa1d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4009b80 con 0x7fa1ec157c70 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d400f040 con 0x7fa1ec157c70 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec145750 con 0x7fa1ec157c70 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40146a0 con 0x7fa1ec157c70 2023-11-24T08:19:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.412+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec145d10 con 0x7fa1ec157c70 2023-11-24T08:19:10.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.413+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec067470 con 0x7fa1ec157c70 2023-11-24T08:19:10.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.414+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fa1d4014800 con 0x7fa1ec157c70 2023-11-24T08:19:10.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.415+0000 7fa1e37fe640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 0x7fa1cc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.415+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa1d40892f0 con 0x7fa1ec157c70 2023-11-24T08:19:10.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.416+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 0x7fa1cc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.416+0000 7fa1eaffd640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 0x7fa1cc06eda0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036000 tx=0x7fa1dc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.420+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1d4055e00 con 0x7fa1ec157c70 2023-11-24T08:19:10.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.610+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1ec0cd910 con 0x7fa1ec157c70 2023-11-24T08:19:10.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.612+0000 7fa1e37fe640 1 -- 172.21.15.107:0/3110574766 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+12998 (secure 0 0 0) 0x7fa1d4059420 con 0x7fa1ec157c70 2023-11-24T08:19:10.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:10.615 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":167,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:08.132046+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/1134531969":"2023-11-25T08:19:04.935528+0000","172.21.15.107:6836/1134531969":"2023-11-25T08:19:04.935528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 msgr2=0x7fa1cc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 0x7fa1cc06eda0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc036000 tx=0x7fa1dc03c040 comp rx=0 tx=0).stop 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 msgr2=0x7fa1ec0cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400efc0 tx=0x7fa1d400c490 comp rx=0 tx=0).stop 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 shutdown_connections 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1cc06c8e0 0x7fa1cc06eda0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1ec15dc80 0x7fa1ec09bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec157c70 0x7fa1ec0cc350 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 --2- 172.21.15.107:0/3110574766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1ec0cf450 0x7fa1ec1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 >> 172.21.15.107:0/3110574766 conn(0x7fa1ec06a920 msgr2=0x7fa1ec160420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 shutdown_connections 2023-11-24T08:19:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.615+0000 7fa1f1490640 1 -- 172.21.15.107:0/3110574766 wait complete. 2023-11-24T08:19:10.635 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:19:10.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:19:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.851+0000 7f4744de4640 1 Processor -- start 2023-11-24T08:19:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.851+0000 7f4744de4640 1 -- start start 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f474015d3f0 0x7f47400d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f474015d7d0 0x7f47400d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47400d0e40 0x7f47400afea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47400d15e0 con 0x7f47400d0e40 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47400d0280 con 0x7f474015d7d0 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f4744de4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47400d1300 con 0x7f474015d3f0 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47400d0e40 0x7f47400afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47400d0e40 0x7f47400afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32890/0 (socket says 172.21.15.107:32890) 2023-11-24T08:19:10.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 -- 172.21.15.107:0/1718025456 learned_addr learned my addr 172.21.15.107:0/1718025456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:10.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 -- 172.21.15.107:0/1718025456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f474015d3f0 msgr2=0x7f47400d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:10.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 --2- 172.21.15.107:0/1718025456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f474015d3f0 0x7f47400d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 -- 172.21.15.107:0/1718025456 >> v2:172.21.15.120:3300/0 conn(0x7f474015d7d0 msgr2=0x7f47400d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:10.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 --2- 172.21.15.107:0/1718025456 >> v2:172.21.15.120:3300/0 conn(0x7f474015d7d0 0x7f47400d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.853+0000 7f473ffff640 1 -- 172.21.15.107:0/1718025456 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47400cfc30 con 0x7f47400d0e40 2023-11-24T08:19:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.854+0000 7f473ffff640 1 --2- 172.21.15.107:0/1718025456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47400d0e40 0x7f47400afea0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f474008bbc0 tx=0x7f473802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f846ddfd081872f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f473effd640 1 -- 172.21.15.107:0/1718025456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738031870 con 0x7f47400d0e40 2023-11-24T08:19:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f473effd640 1 -- 172.21.15.107:0/1718025456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47380319d0 con 0x7f47400d0e40 2023-11-24T08:19:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f473effd640 1 -- 172.21.15.107:0/1718025456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738031d60 con 0x7f47400d0e40 2023-11-24T08:19:10.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 -- 172.21.15.107:0/1718025456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47400d0e40 msgr2=0x7f47400afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 --2- 172.21.15.107:0/1718025456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47400d0e40 0x7f47400afea0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f474008bbc0 tx=0x7f473802b910 comp rx=0 tx=0).stop 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 -- 172.21.15.107:0/1718025456 shutdown_connections 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 --2- 172.21.15.107:0/1718025456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47400d0e40 0x7f47400afea0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 --2- 172.21.15.107:0/1718025456 >> v2:172.21.15.120:3300/0 conn(0x7f474015d7d0 0x7f47400d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 --2- 172.21.15.107:0/1718025456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f474015d3f0 0x7f47400d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 -- 172.21.15.107:0/1718025456 >> 172.21.15.107:0/1718025456 conn(0x7f474009c5b0 msgr2=0x7f47400d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 -- 172.21.15.107:0/1718025456 shutdown_connections 2023-11-24T08:19:10.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.855+0000 7f4744de4640 1 -- 172.21.15.107:0/1718025456 wait complete. 2023-11-24T08:19:10.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.857+0000 7f4744de4640 1 Processor -- start 2023-11-24T08:19:10.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.857+0000 7f4744de4640 1 -- start start 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47400d0e40 0x7f47400b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474015d7d0 0x7f4740155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47400ca340 con 0x7f474015d3f0 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47400ca590 con 0x7f474015d7d0 2023-11-24T08:19:10.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4744de4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47400ca870 con 0x7f47400d0e40 2023-11-24T08:19:10.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.858+0000 7f4736ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32900/0 (socket says 172.21.15.107:32900) 2023-11-24T08:19:10.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 -- 172.21.15.107:0/4146601148 learned_addr learned my addr 172.21.15.107:0/4146601148 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 -- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47400d0e40 msgr2=0x7f47400b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f473f7fe640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47400d0e40 0x7f47400b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47400d0e40 0x7f47400b6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 -- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474015d7d0 msgr2=0x7f4740155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474015d7d0 0x7f4740155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 -- 172.21.15.107:0/4146601148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47380094d0 con 0x7f474015d3f0 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f4736ffd640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f472800b410 tx=0x7f472800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.859+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728011830 con 0x7f474015d3f0 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.860+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47280119b0 con 0x7f474015d3f0 2023-11-24T08:19:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.860+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728011d60 con 0x7f474015d3f0 2023-11-24T08:19:10.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.860+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47400cab50 con 0x7f474015d3f0 2023-11-24T08:19:10.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.860+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4740159460 con 0x7f474015d3f0 2023-11-24T08:19:10.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.861+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47400cb070 con 0x7f474015d3f0 2023-11-24T08:19:10.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.862+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f472801a400 con 0x7f474015d3f0 2023-11-24T08:19:10.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.863+0000 7f473cff9640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 0x7f472006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:10.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.863+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4728088550 con 0x7f474015d3f0 2023-11-24T08:19:10.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.863+0000 7f473f7fe640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 0x7f472006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:10.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.864+0000 7f473f7fe640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 0x7f472006ed70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4730007dc0 tx=0x7f47300072f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:10.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:10.867+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47400cb070 con 0x7f474015d3f0 2023-11-24T08:19:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.043+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47400cb070 con 0x7f474015d3f0 2023-11-24T08:19:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.044+0000 7f473cff9640 1 -- 172.21.15.107:0/4146601148 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47400cb070 con 0x7f474015d3f0 2023-11-24T08:19:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:19:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 msgr2=0x7f472006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 0x7f472006ed70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4730007dc0 tx=0x7f47300072f0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 msgr2=0x7f47400b7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f472800b410 tx=0x7f472800b8d0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 shutdown_connections 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f472006c8b0 0x7f472006ed70 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474015d7d0 0x7f4740155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f474015d3f0 0x7f47400b7da0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 --2- 172.21.15.107:0/4146601148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47400d0e40 0x7f47400b6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 >> 172.21.15.107:0/4146601148 conn(0x7f474009c5b0 msgr2=0x7f47400b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 shutdown_connections 2023-11-24T08:19:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.047+0000 7f4744de4640 1 -- 172.21.15.107:0/4146601148 wait complete. 2023-11-24T08:19:11.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs' 2023-11-24T08:19:11.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.278+0000 7f5ab718a640 1 Processor -- start 2023-11-24T08:19:11.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.278+0000 7f5ab718a640 1 -- start start 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab718a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab718a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ab0070a80 0x7f5ab013e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab718a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab015b7a0 con 0x7f5ab006cee0 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab718a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab006d360 con 0x7f5ab0070a80 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab718a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab015ba60 con 0x7f5ab013e0e0 2023-11-24T08:19:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab6989640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ab0070a80 0x7f5ab013e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab6989640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ab0070a80 0x7f5ab013e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32916/0 (socket says 172.21.15.107:32916) 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.280+0000 7f5ab6989640 1 -- 172.21.15.107:0/751434807 learned_addr learned my addr 172.21.15.107:0/751434807 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.281+0000 7f5ab6188640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.281+0000 7f5ab6989640 1 -- 172.21.15.107:0/751434807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 msgr2=0x7f5ab006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.281+0000 7f5ab6989640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.281+0000 7f5ab6989640 1 -- 172.21.15.107:0/751434807 >> v1:172.21.15.120:6789/0 conn(0x7f5ab006cee0 legacy=0x7f5ab0070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.281+0000 7f5ab6989640 1 -- 172.21.15.107:0/751434807 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab015bd40 con 0x7f5ab0070a80 2023-11-24T08:19:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.282+0000 7f5ab6989640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab013e810 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009e50 tx=0x7f5aa802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ce26294921c3c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.282+0000 7f5ab5186640 1 -- 172.21.15.107:0/751434807 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa802ca70 con 0x7f5ab0070a80 2023-11-24T08:19:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.282+0000 7f5ab5186640 1 -- 172.21.15.107:0/751434807 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa802cbd0 con 0x7f5ab0070a80 2023-11-24T08:19:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.282+0000 7f5ab5186640 1 -- 172.21.15.107:0/751434807 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80343f0 con 0x7f5ab0070a80 2023-11-24T08:19:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 -- 172.21.15.107:0/751434807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 msgr2=0x7f5ab013e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab013e810 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009e50 tx=0x7f5aa802bed0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 -- 172.21.15.107:0/751434807 shutdown_connections 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab013e810 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 --2- 172.21.15.107:0/751434807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 -- 172.21.15.107:0/751434807 >> 172.21.15.107:0/751434807 conn(0x7f5ab00b4630 msgr2=0x7f5ab00b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 -- 172.21.15.107:0/751434807 shutdown_connections 2023-11-24T08:19:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.283+0000 7f5ab718a640 1 -- 172.21.15.107:0/751434807 wait complete. 2023-11-24T08:19:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.285+0000 7f5ab718a640 1 Processor -- start 2023-11-24T08:19:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.286+0000 7f5ab718a640 1 -- start start 2023-11-24T08:19:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab006cee0 0x7f5ab00adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab01f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00ac840 con 0x7f5ab0070a80 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00ad6a0 con 0x7f5ab006cee0 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab718a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00aca90 con 0x7f5ab013e0e0 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab5987640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab5987640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32926/0 (socket says 172.21.15.107:32926) 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab5987640 1 -- 172.21.15.107:0/4065324381 learned_addr learned my addr 172.21.15.107:0/4065324381 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab6188640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab006cee0 0x7f5ab00adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.287+0000 7f5ab6989640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab01f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 -- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 msgr2=0x7f5ab01f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab01f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 -- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab006cee0 msgr2=0x7f5ab00adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab006cee0 0x7f5ab00adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 -- 172.21.15.107:0/4065324381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8009b30 con 0x7f5ab0070a80 2023-11-24T08:19:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab5987640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c430 tx=0x7f5aa000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0011790 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa00118f0 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0011ce0 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.288+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab00acd70 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.289+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab00ad330 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.290+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab00ad4c0 con 0x7f5ab0070a80 2023-11-24T08:19:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.291+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f5aa001c460 con 0x7f5ab0070a80 2023-11-24T08:19:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.291+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 0x7f5a9406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.292+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5aa0088430 con 0x7f5ab0070a80 2023-11-24T08:19:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.292+0000 7f5ab6188640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 0x7f5a9406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:11.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.292+0000 7f5ab6188640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 0x7f5a9406ed70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006fd0 tx=0x7f5aac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.296+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5aa0054fe0 con 0x7f5ab0070a80 2023-11-24T08:19:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:11.559+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs"]} v 0) v1 -- 0x7f5ab00ad810 con 0x7f5ab0070a80 2023-11-24T08:19:12.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.180+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/4065324381 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs"]}]=0 v717) v1 ==== 176+0+0 (secure 0 0 0) 0x7f5aa0058600 con 0x7f5ab0070a80 2023-11-24T08:19:14.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 msgr2=0x7f5a9406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 0x7f5a9406ed70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006fd0 tx=0x7f5aac008040 comp rx=0 tx=0).stop 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 msgr2=0x7f5ab0159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c430 tx=0x7f5aa000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 shutdown_connections 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a9406c8b0 0x7f5a9406ed70 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ab013e0e0 0x7f5ab01f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ab0070a80 0x7f5ab0159110 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 --2- 172.21.15.107:0/4065324381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ab006cee0 0x7f5ab00adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 >> 172.21.15.107:0/4065324381 conn(0x7f5ab00b4630 msgr2=0x7f5ab00b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 shutdown_connections 2023-11-24T08:19:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:12.184+0000 7f5ab718a640 1 -- 172.21.15.107:0/4065324381 wait complete. 2023-11-24T08:19:15.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:19:15.050 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:19:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.256+0000 7faf4519e640 1 Processor -- start 2023-11-24T08:19:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.256+0000 7faf4519e640 1 -- start start 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf4519e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf4015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf4519e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faf400c49f0 0x7faf4006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf4519e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4015ec90 con 0x7faf4006dc40 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf4519e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4006e0f0 con 0x7faf400c49f0 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf4519e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400cd530 con 0x7faf40072160 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf3f59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faf4006dc40 0x7faf400cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43162/0 (socket says 172.21.15.107:43162) 2023-11-24T08:19:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.258+0000 7faf3f59e640 1 -- 172.21.15.107:0/3887314880 learned_addr learned my addr 172.21.15.107:0/3887314880 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:15.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf4015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf37fff640 1 --2- 172.21.15.107:0/3887314880 >> v2:172.21.15.120:3300/0 conn(0x7faf400c49f0 0x7faf4006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218070571 0 0) 0x7faf38001490 con 0x7faf4006dc40 2023-11-24T08:19:15.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c003520 con 0x7faf4006dc40 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457316281 0 0) 0x7faf2c003520 con 0x7faf4006dc40 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 msgr2=0x7faf4015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf4015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 >> v2:172.21.15.120:3300/0 conn(0x7faf400c49f0 msgr2=0x7faf4006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 --2- 172.21.15.107:0/3887314880 >> v2:172.21.15.120:3300/0 conn(0x7faf400c49f0 0x7faf4006d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400c4ed0 con 0x7faf4006dc40 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7faf38001b80 con 0x7faf4006dc40 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 >> v1:172.21.15.107:6789/0 conn(0x7faf4006dc40 legacy=0x7faf400cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf2c003970 0x7faf2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4006e0f0 con 0x7faf2c0047c0 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400cd530 con 0x7faf2c003970 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.259+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c00b980 con 0x7faf40072160 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf3f59e640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf2c003970 0x7faf2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 -- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 msgr2=0x7faf2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf40072160 0x7faf2c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 -- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf2c003970 msgr2=0x7faf2c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf2c003970 0x7faf2c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf37fff640 1 -- 172.21.15.107:0/3887314880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c003520 con 0x7faf2c0047c0 2023-11-24T08:19:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.260+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf2c003970 0x7faf2c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.261+0000 7faf37fff640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf2c004c40 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7faf28009ac0 tx=0x7faf2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849760f24a602f80 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2802c6d0 con 0x7faf2c0047c0 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2802c830 con 0x7faf2c0047c0 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf3e59c640 1 -- 172.21.15.107:0/3887314880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2802cbc0 con 0x7faf2c0047c0 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 -- 172.21.15.107:0/3887314880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 msgr2=0x7faf2c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 --2- 172.21.15.107:0/3887314880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf2c004c40 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7faf28009ac0 tx=0x7faf2c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf3f59e640 1 -- 172.21.15.107:0/3887314880 reap_dead start 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 -- 172.21.15.107:0/3887314880 shutdown_connections 2023-11-24T08:19:15.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 -- 172.21.15.107:0/3887314880 >> 172.21.15.107:0/3887314880 conn(0x7faf400b0c50 msgr2=0x7faf400b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:15.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 -- 172.21.15.107:0/3887314880 shutdown_connections 2023-11-24T08:19:15.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.262+0000 7faf4519e640 1 -- 172.21.15.107:0/3887314880 wait complete. 2023-11-24T08:19:15.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.264+0000 7faf4519e640 1 Processor -- start 2023-11-24T08:19:15.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.264+0000 7faf4519e640 1 -- start start 2023-11-24T08:19:15.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf401e5650 0x7faf40154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf400afb40 0x7faf400ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf401e45f0 con 0x7faf2c0047c0 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf401e4890 con 0x7faf401e5650 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf4519e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf401e4b70 con 0x7faf400afb40 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.265+0000 7faf3ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32950/0 (socket says 172.21.15.107:32950) 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 -- 172.21.15.107:0/2810375473 learned_addr learned my addr 172.21.15.107:0/2810375473 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 -- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf400afb40 msgr2=0x7faf400ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf400afb40 0x7faf400ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 -- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf401e5650 msgr2=0x7faf40154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf401e5650 0x7faf40154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 -- 172.21.15.107:0/2810375473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf401e4e50 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf3ed9d640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7faf30010890 tx=0x7faf30010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.266+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000eb00 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.267+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000ec60 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.267+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3001c400 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.267+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf401e5130 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.267+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf400ad3b0 con 0x7faf2c0047c0 2023-11-24T08:19:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.268+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf401e49d0 con 0x7faf2c0047c0 2023-11-24T08:19:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.269+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7faf3001c560 con 0x7faf2c0047c0 2023-11-24T08:19:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.270+0000 7faf377fe640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 0x7faf2006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.270+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faf30088970 con 0x7faf2c0047c0 2023-11-24T08:19:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.270+0000 7faf37fff640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 0x7faf2006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.271+0000 7faf37fff640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 0x7faf2006ed00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7faf28036000 tx=0x7faf2802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:15.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.275+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf401e49d0 con 0x7faf2c0047c0 2023-11-24T08:19:15.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.465+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7faf401e49d0 con 0x7faf2c0047c0 2023-11-24T08:19:15.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.473+0000 7faf377fe640 1 -- 172.21.15.107:0/2810375473 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v47)=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7faf401e49d0 con 0x7faf2c0047c0 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 msgr2=0x7faf2006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 0x7faf2006ed00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7faf28036000 tx=0x7faf2802e000 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 msgr2=0x7faf400af390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7faf30010890 tx=0x7faf30010d50 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 shutdown_connections 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faf2006c840 0x7faf2006ed00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf400afb40 0x7faf400ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf401e5650 0x7faf40154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 --2- 172.21.15.107:0/2810375473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c0047c0 0x7faf400af390 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.475+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 >> 172.21.15.107:0/2810375473 conn(0x7faf400b0c50 msgr2=0x7faf400cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.476+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 shutdown_connections 2023-11-24T08:19:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:15.476+0000 7faf4519e640 1 -- 172.21.15.107:0/2810375473 wait complete. 2023-11-24T08:19:15.497 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:19:15.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:19:15.606 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:19:15.736 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:19:15.737 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:15.737 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:15.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:19:15.800 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:19:15.800 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:15.862 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:19:15.862 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:15.863 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:15.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:19:15.923 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:19:15.923 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:15.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:19:15.983 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:19:15.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:16.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.202+0000 7fbe7ff8a640 1 Processor -- start 2023-11-24T08:19:16.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.203+0000 7fbe7ff8a640 1 -- start start 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7ff8a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815c7f0 0x7fbe7806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7ff8a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe780cf450 0x7fbe780cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7ff8a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe780cece0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7ff8a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe780cfe40 con 0x7fbe7815c7f0 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7ff8a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe780ceaf0 con 0x7fbe780cf450 2023-11-24T08:19:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7dd26640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815c7f0 0x7fbe7806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7dd26640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815c7f0 0x7fbe7806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32958/0 (socket says 172.21.15.107:32958) 2023-11-24T08:19:16.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.204+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1386821161 learned_addr learned my addr 172.21.15.107:0/1386821161 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.205+0000 7fbe7d525640 1 --2- 172.21.15.107:0/1386821161 >> v2:172.21.15.120:3300/0 conn(0x7fbe780cf450 0x7fbe780cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.205+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1386821161 >> v2:172.21.15.120:3300/0 conn(0x7fbe780cf450 msgr2=0x7fbe780cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.205+0000 7fbe7dd26640 1 --2- 172.21.15.107:0/1386821161 >> v2:172.21.15.120:3300/0 conn(0x7fbe780cf450 0x7fbe780cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.205+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1386821161 >> v1:172.21.15.107:6789/0 conn(0x7fbe780cdfb0 legacy=0x7fbe780ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.205+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1386821161 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7806d040 con 0x7fbe7815c7f0 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.206+0000 7fbe7dd26640 1 --2- 172.21.15.107:0/1386821161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815c7f0 0x7fbe7806db20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fbe78070410 tx=0x7fbe7402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3881f0b3d3bc6d4b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.206+0000 7fbe7cd24640 1 -- 172.21.15.107:0/1386821161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74030860 con 0x7fbe7815c7f0 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.206+0000 7fbe7cd24640 1 -- 172.21.15.107:0/1386821161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe740309c0 con 0x7fbe7815c7f0 2023-11-24T08:19:16.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.206+0000 7fbe7cd24640 1 -- 172.21.15.107:0/1386821161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74030d50 con 0x7fbe7815c7f0 2023-11-24T08:19:16.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1386821161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815c7f0 msgr2=0x7fbe7806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1386821161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815c7f0 0x7fbe7806db20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fbe78070410 tx=0x7fbe7402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1386821161 shutdown_connections 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1386821161 >> v2:172.21.15.120:3300/0 conn(0x7fbe780cf450 0x7fbe780cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1386821161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815c7f0 0x7fbe7806db20 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1386821161 >> 172.21.15.107:0/1386821161 conn(0x7fbe7806a920 msgr2=0x7fbe7806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1386821161 shutdown_connections 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.207+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1386821161 wait complete. 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.208+0000 7fbe7ff8a640 1 Processor -- start 2023-11-24T08:19:16.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.209+0000 7fbe7ff8a640 1 -- start start 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.209+0000 7fbe7ff8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.209+0000 7fbe7ff8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cf450 0x7fbe780b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7ff8a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815c7f0 0x7fbe780b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7ff8a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781f4240 con 0x7fbe780cdfb0 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7ff8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781f3c80 con 0x7fbe780cf450 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7ff8a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781f3f60 con 0x7fbe7815c7f0 2023-11-24T08:19:16.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32960/0 (socket says 172.21.15.107:32960) 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1587416900 learned_addr learned my addr 172.21.15.107:0/1587416900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7d525640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cf450 0x7fbe780b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815c7f0 msgr2=0x7fbe780b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815c7f0 0x7fbe780b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cf450 msgr2=0x7fbe780b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cf450 0x7fbe780b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.210+0000 7fbe7dd26640 1 -- 172.21.15.107:0/1587416900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe740094d0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe7dd26640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fbe7402b9e0 tx=0x7fbe74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74030850 con 0x7fbe780cdfb0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74039d90 con 0x7fbe780cdfb0 2023-11-24T08:19:16.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74043650 con 0x7fbe780cdfb0 2023-11-24T08:19:16.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe780ac4d0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.211+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe781f37b0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.212+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe780ac9f0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.213+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fbe7402f3b0 con 0x7fbe780cdfb0 2023-11-24T08:19:16.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.214+0000 7fbe6e7fc640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 0x7fbe5c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.214+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbe740aa960 con 0x7fbe780cdfb0 2023-11-24T08:19:16.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.214+0000 7fbe7d525640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 0x7fbe5c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.215+0000 7fbe7d525640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 0x7fbe5c06ed70 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.218+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe74077470 con 0x7fbe780cdfb0 2023-11-24T08:19:16.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.443+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe78067470 con 0x7fbe780cdfb0 2023-11-24T08:19:16.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.445+0000 7fbe6e7fc640 1 -- 172.21.15.107:0/1587416900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3714 (secure 0 0 0) 0x7fbe74034090 con 0x7fbe780cdfb0 2023-11-24T08:19:16.447 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:16.447 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":100,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3236579861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3236579861},{"type":"v1","addr":"172.21.15.107:6835","nonce":3236579861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782981811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2782981811},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782981811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":6841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2998091637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2998091637},{"type":"v1","addr":"172.21.15.120:6834","nonce":2998091637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":10332,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3813355022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3813355022},{"type":"v1","addr":"172.21.15.107:6837","nonce":3813355022}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[]} 2023-11-24T08:19:16.447 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 100 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.447+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 msgr2=0x7fbe5c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.447+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 0x7fbe5c06ed70 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).stop 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 msgr2=0x7fbe781f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fbe7402b9e0 tx=0x7fbe74004730 comp rx=0 tx=0).stop 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 shutdown_connections 2023-11-24T08:19:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5c06c8b0 0x7fbe5c06ed70 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815c7f0 0x7fbe780b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cf450 0x7fbe780b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 --2- 172.21.15.107:0/1587416900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780cdfb0 0x7fbe781f3270 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 >> 172.21.15.107:0/1587416900 conn(0x7fbe7806a920 msgr2=0x7fbe780cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 shutdown_connections 2023-11-24T08:19:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.448+0000 7fbe7ff8a640 1 -- 172.21.15.107:0/1587416900 wait complete. 2023-11-24T08:19:16.469 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:19:16.522 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:19:16.522 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:19:16.523 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:19:16.523 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:19:16.523 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:19:16.523 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:19:16.523 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:19:16.523 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:19:16.523 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:19:16.524 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:19:16.524 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:19:16.524 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:19:16.524 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:19:16.524 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:19:16.524 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:19:16.543 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:19:16.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:19:16.544 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:19:16.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:19:16.548 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:19:16.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:19:16.549 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:19:16.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:19:16.596 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:19:16.597 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:19:16.597 DEBUG:teuthology.parallel:result is None 2023-11-24T08:19:16.597 DEBUG:teuthology.parallel:result is None 2023-11-24T08:19:16.604 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:19:16.604 DEBUG:teuthology.parallel:result is None 2023-11-24T08:19:16.604 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:19:16.605 DEBUG:teuthology.parallel:result is None 2023-11-24T08:19:16.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:19:16.750 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:19:16.751 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:19:16.760 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:19:16.764 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:19:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.784+0000 7f0c8f333640 1 Processor -- start 2023-11-24T08:19:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.784+0000 7f0c8f333640 1 -- start start 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8f333640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0c88140520 0x7f0c880c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8f333640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c88161b10 0x7f0c8806ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8f333640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c88161f60 con 0x7f0c880c65f0 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8f333640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c880c6a70 con 0x7f0c88161b10 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8f333640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8815de00 con 0x7f0c88140520 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c88161b10 0x7f0c8806ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c88161b10 0x7f0c8806ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32998/0 (socket says 172.21.15.107:32998) 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/2994620242 learned_addr learned my addr 172.21.15.107:0/2994620242 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d0cf640 1 --2- 172.21.15.107:0/2994620242 >> v2:172.21.15.120:3300/0 conn(0x7f0c88140520 0x7f0c880c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/2994620242 >> v2:172.21.15.120:3300/0 conn(0x7f0c88140520 msgr2=0x7f0c880c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 --2- 172.21.15.107:0/2994620242 >> v2:172.21.15.120:3300/0 conn(0x7f0c88140520 0x7f0c880c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/2994620242 >> v1:172.21.15.107:6789/0 conn(0x7f0c880c65f0 legacy=0x7f0c88161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.786+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/2994620242 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8815e0e0 con 0x7f0c88161b10 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.787+0000 7f0c8d8d0640 1 --2- 172.21.15.107:0/2994620242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806ff70 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0c8806e0b0 tx=0x7f0c8002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a532caecbfeb0d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c7ffff640 1 -- 172.21.15.107:0/2994620242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8002cb20 con 0x7f0c88161b10 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c7ffff640 1 -- 172.21.15.107:0/2994620242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8002cc80 con 0x7f0c88161b10 2023-11-24T08:19:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c7ffff640 1 -- 172.21.15.107:0/2994620242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800343f0 con 0x7f0c88161b10 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 -- 172.21.15.107:0/2994620242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 msgr2=0x7f0c8806ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 --2- 172.21.15.107:0/2994620242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806ff70 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0c8806e0b0 tx=0x7f0c8002bf80 comp rx=0 tx=0).stop 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 -- 172.21.15.107:0/2994620242 shutdown_connections 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 --2- 172.21.15.107:0/2994620242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806ff70 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 --2- 172.21.15.107:0/2994620242 >> v2:172.21.15.120:3300/0 conn(0x7f0c88140520 0x7f0c880c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 -- 172.21.15.107:0/2994620242 >> 172.21.15.107:0/2994620242 conn(0x7f0c880b2e40 msgr2=0x7f0c880b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.788+0000 7f0c8f333640 1 -- 172.21.15.107:0/2994620242 shutdown_connections 2023-11-24T08:19:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.789+0000 7f0c8f333640 1 -- 172.21.15.107:0/2994620242 wait complete. 2023-11-24T08:19:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.790+0000 7f0c8f333640 1 Processor -- start 2023-11-24T08:19:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.790+0000 7f0c8f333640 1 -- start start 2023-11-24T08:19:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 0x7f0c8815d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 0x7f0c881f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c881f2d50 con 0x7f0c88161b10 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8815c820 con 0x7f0c88140520 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8f333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8815ccb0 con 0x7f0c880c65f0 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.791+0000 7f0c8d8d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33004/0 (socket says 172.21.15.107:33004) 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/1559107675 learned_addr learned my addr 172.21.15.107:0/1559107675 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 msgr2=0x7f0c8815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d0cf640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 0x7f0c8815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8c8ce640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 0x7f0c881f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 0x7f0c8815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 msgr2=0x7f0c881f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 0x7f0c881f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 -- 172.21.15.107:0/1559107675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80009b30 con 0x7f0c88161b10 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8c8ce640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 0x7f0c881f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d0cf640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 0x7f0c8815d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c8d8d0640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0c80036040 tx=0x7f0c80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.792+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80009510 con 0x7f0c88161b10 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.793+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8002c920 con 0x7f0c88161b10 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.793+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8003e690 con 0x7f0c88161b10 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.793+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8815bc10 con 0x7f0c88161b10 2023-11-24T08:19:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.793+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8815c1d0 con 0x7f0c88161b10 2023-11-24T08:19:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.794+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8815c9d0 con 0x7f0c88161b10 2023-11-24T08:19:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.795+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f0c8003e7f0 con 0x7f0c88161b10 2023-11-24T08:19:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.795+0000 7f0c7dffb640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 0x7f0c6c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.796+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c800aaf50 con 0x7f0c88161b10 2023-11-24T08:19:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.796+0000 7f0c8d0cf640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 0x7f0c6c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.796+0000 7f0c8d0cf640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 0x7f0c6c06ed70 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009ac0 tx=0x7f0c84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.800+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c8815c9d0 con 0x7f0c88161b10 2023-11-24T08:19:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:16.987+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0c8815dac0 con 0x7f0c88161b10 2023-11-24T08:19:17.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.505+0000 7f0c7dffb640 1 -- 172.21.15.107:0/1559107675 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v169) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0c80077b00 con 0x7f0c88161b10 2023-11-24T08:19:17.507 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:19:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 msgr2=0x7f0c6c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 0x7f0c6c06ed70 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009ac0 tx=0x7f0c84008040 comp rx=0 tx=0).stop 2023-11-24T08:19:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 msgr2=0x7f0c8806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0c80036040 tx=0x7f0c80004a40 comp rx=0 tx=0).stop 2023-11-24T08:19:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 shutdown_connections 2023-11-24T08:19:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0c6c06c8b0 0x7f0c6c06ed70 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c88161b10 0x7f0c8806c2a0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c88140520 0x7f0c881f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 --2- 172.21.15.107:0/1559107675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c880c65f0 0x7f0c8815d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.509+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 >> 172.21.15.107:0/1559107675 conn(0x7f0c880b2e40 msgr2=0x7f0c8815a140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:17.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.510+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 shutdown_connections 2023-11-24T08:19:17.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.510+0000 7f0c8f333640 1 -- 172.21.15.107:0/1559107675 wait complete. 2023-11-24T08:19:17.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:19:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.724+0000 7f95fa3a9640 1 Processor -- start 2023-11-24T08:19:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.725+0000 7f95fa3a9640 1 -- start start 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95fa3a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f95f40cdfb0 0x7f95f40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95fa3a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f40b2c00 con 0x7f95f415a4f0 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95fa3a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f40b3a40 con 0x7f95f40b20c0 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95fa3a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f40ce980 con 0x7f95f40cdfb0 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95f3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f95f415a4f0 0x7f95f40b3ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43192/0 (socket says 172.21.15.107:43192) 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95f3fff640 1 -- 172.21.15.107:0/253434103 learned_addr learned my addr 172.21.15.107:0/253434103 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.726+0000 7f95f37fe640 1 --2- 172.21.15.107:0/253434103 >> v2:172.21.15.107:3300/0 conn(0x7f95f40cdfb0 0x7f95f40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.727+0000 7f95f37fe640 1 -- 172.21.15.107:0/253434103 >> v1:172.21.15.120:6789/0 conn(0x7f95f40b20c0 legacy=0x7f95f40b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.727+0000 7f95f37fe640 1 -- 172.21.15.107:0/253434103 >> v1:172.21.15.107:6789/0 conn(0x7f95f415a4f0 legacy=0x7f95f40b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.727+0000 7f95f37fe640 1 -- 172.21.15.107:0/253434103 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40b2e30 con 0x7f95f40cdfb0 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.728+0000 7f95f37fe640 1 --2- 172.21.15.107:0/253434103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f40cdfb0 0x7f95f40ce440 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0165ba3b773a55 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.728+0000 7f95f2ffd640 1 -- 172.21.15.107:0/253434103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc031830 con 0x7f95f40cdfb0 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.728+0000 7f95f2ffd640 1 -- 172.21.15.107:0/253434103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc031990 con 0x7f95f40cdfb0 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.728+0000 7f95f2ffd640 1 -- 172.21.15.107:0/253434103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc031d20 con 0x7f95f40cdfb0 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 -- 172.21.15.107:0/253434103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f40cdfb0 msgr2=0x7f95f40ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/253434103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f40cdfb0 0x7f95f40ce440 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 -- 172.21.15.107:0/253434103 shutdown_connections 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/253434103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f40cdfb0 0x7f95f40ce440 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 -- 172.21.15.107:0/253434103 >> 172.21.15.107:0/253434103 conn(0x7f95f40b0c70 msgr2=0x7f95f40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 -- 172.21.15.107:0/253434103 shutdown_connections 2023-11-24T08:19:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.729+0000 7f95fa3a9640 1 -- 172.21.15.107:0/253434103 wait complete. 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.730+0000 7f95fa3a9640 1 Processor -- start 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.731+0000 7f95fa3a9640 1 -- start start 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.731+0000 7f95fa3a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f40b20c0 0x7f95f4150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95fa3a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f40cdfb0 0x7f95f4156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95fa3a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95fa3a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4152ed0 con 0x7f95f415a4f0 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95fa3a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41522e0 con 0x7f95f40b20c0 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95fa3a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41525a0 con 0x7f95f40cdfb0 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33028/0 (socket says 172.21.15.107:33028) 2023-11-24T08:19:17.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 -- 172.21.15.107:0/3309945750 learned_addr learned my addr 172.21.15.107:0/3309945750 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 -- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f40cdfb0 msgr2=0x7f95f4156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f40cdfb0 0x7f95f4156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 -- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f40b20c0 msgr2=0x7f95f4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f40b20c0 0x7f95f4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 -- 172.21.15.107:0/3309945750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc009400 con 0x7f95f415a4f0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.732+0000 7f95f8946640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c390 tx=0x7f95ec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.733+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0035f0 con 0x7f95f415a4f0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.733+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ec003750 con 0x7f95f415a4f0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.733+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec003b00 con 0x7f95f415a4f0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.733+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f4152880 con 0x7f95f415a4f0 2023-11-24T08:19:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.733+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f4150540 con 0x7f95f415a4f0 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.734+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f41506d0 con 0x7f95f415a4f0 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.735+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f95ec003c60 con 0x7f95f415a4f0 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.736+0000 7f95f0ff9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 0x7f95d406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.736+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f95ec0876c0 con 0x7f95f415a4f0 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.736+0000 7f95f3fff640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 0x7f95d406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.737+0000 7f95f3fff640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 0x7f95d406ed70 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f95e4001fd0 tx=0x7f95e40015c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.741+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95ec054270 con 0x7f95f415a4f0 2023-11-24T08:19:18.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:17.999+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f95f404e8b0 con 0x7f95f415a4f0 2023-11-24T08:19:18.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.000+0000 7f95f0ff9640 1 -- 172.21.15.107:0/3309945750 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f95ec017090 con 0x7f95f415a4f0 2023-11-24T08:19:18.002 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:18.002 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:19:18.005 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:19:18.007 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.010 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:19:18.010 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:19:18.010 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.011 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.013 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.017 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:19:18.018 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:19:18.019 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:19:18.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.021 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:19:18.023 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 msgr2=0x7f95d406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 0x7f95d406ed70 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f95e4001fd0 tx=0x7f95e40015c0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 msgr2=0x7f95f41570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c390 tx=0x7f95ec00c850 comp rx=0 tx=0).stop 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 shutdown_connections 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f95d406c8b0 0x7f95d406ed70 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95f415a4f0 0x7f95f41570f0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f40cdfb0 0x7f95f4156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 --2- 172.21.15.107:0/3309945750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f40b20c0 0x7f95f4150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 >> 172.21.15.107:0/3309945750 conn(0x7f95f40b0c70 msgr2=0x7f95f40cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 shutdown_connections 2023-11-24T08:19:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.003+0000 7f95fa3a9640 1 -- 172.21.15.107:0/3309945750 wait complete. 2023-11-24T08:19:18.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:18.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.236+0000 7fe331c32640 1 Processor -- start 2023-11-24T08:19:18.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.236+0000 7fe331c32640 1 -- start start 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe331c32640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe32c0ce6c0 0x7fe32c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe331c32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0cece0 con 0x7fe32c0cf450 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe331c32640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c06d040 con 0x7fe32c15c7f0 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe331c32640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c0cfe40 con 0x7fe32c0ce6c0 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe32affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe32c0cf450 0x7fe32c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43602/0 (socket says 172.21.15.107:43602) 2023-11-24T08:19:18.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe32affd640 1 -- 172.21.15.107:0/3112933342 learned_addr learned my addr 172.21.15.107:0/3112933342 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:18.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe32bfff640 1 --2- 172.21.15.107:0/3112933342 >> v2:172.21.15.120:3300/0 conn(0x7fe32c0ce6c0 0x7fe32c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.238+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955843222 0 0) 0x7fe32c0cece0 con 0x7fe32c0cf450 2023-11-24T08:19:18.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314003520 con 0x7fe32c0cf450 2023-11-24T08:19:18.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1691935391 0 0) 0x7fe314003520 con 0x7fe32c0cf450 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 >> v2:172.21.15.120:3300/0 conn(0x7fe32c0ce6c0 msgr2=0x7fe32c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 --2- 172.21.15.107:0/3112933342 >> v2:172.21.15.120:3300/0 conn(0x7fe32c0ce6c0 0x7fe32c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 >> v1:172.21.15.120:6789/0 conn(0x7fe32c15c7f0 legacy=0x7fe32c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c06d320 con 0x7fe32c0cf450 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe320001670 con 0x7fe32c0cf450 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 >> v1:172.21.15.107:6789/0 conn(0x7fe32c0cf450 legacy=0x7fe32c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3140039c0 0x7fe314003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 0x7fe314004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314005180 0x7fe31400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0cfe40 con 0x7fe3140047c0 2023-11-24T08:19:18.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31400c080 con 0x7fe314005180 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31400c240 con 0x7fe3140039c0 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.239+0000 7fe32affd640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 0x7fe314004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32affd640 1 -- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3140039c0 msgr2=0x7fe314003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32affd640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3140039c0 0x7fe314003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32affd640 1 -- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314005180 msgr2=0x7fe31400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32affd640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314005180 0x7fe31400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32affd640 1 -- 172.21.15.107:0/3112933342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314003520 con 0x7fe3140047c0 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.240+0000 7fe32bfff640 1 -- 172.21.15.107:0/3112933342 reap_dead start 2023-11-24T08:19:18.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe32affd640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 0x7fe314004c40 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c0e0 tx=0x7fe3140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54bba88285a4a77 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320005ad0 con 0x7fe3140047c0 2023-11-24T08:19:18.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe320005c30 con 0x7fe3140047c0 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe32a7fc640 1 -- 172.21.15.107:0/3112933342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32002ca50 con 0x7fe3140047c0 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe331c32640 1 -- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 msgr2=0x7fe314004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.241+0000 7fe331c32640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 0x7fe314004c40 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c0e0 tx=0x7fe3140057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.242+0000 7fe331c32640 1 -- 172.21.15.107:0/3112933342 shutdown_connections 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.242+0000 7fe331c32640 1 --2- 172.21.15.107:0/3112933342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3140047c0 0x7fe314004c40 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.242+0000 7fe331c32640 1 -- 172.21.15.107:0/3112933342 >> 172.21.15.107:0/3112933342 conn(0x7fe32c06a920 msgr2=0x7fe32c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:18.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.242+0000 7fe331c32640 1 -- 172.21.15.107:0/3112933342 shutdown_connections 2023-11-24T08:19:18.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.242+0000 7fe331c32640 1 -- 172.21.15.107:0/3112933342 wait complete. 2023-11-24T08:19:18.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.243+0000 7fe331c32640 1 Processor -- start 2023-11-24T08:19:18.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 -- start start 2023-11-24T08:19:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3140047c0 0x7fe32c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32c0ad4c0 0x7fe32c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c156320 con 0x7fe32c0c89d0 2023-11-24T08:19:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c1565c0 con 0x7fe3140047c0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.244+0000 7fe331c32640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c1568a0 con 0x7fe32c0ad4c0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48466/0 (socket says 172.21.15.107:48466) 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 -- 172.21.15.107:0/3311232998 learned_addr learned my addr 172.21.15.107:0/3311232998 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 -- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32c0ad4c0 msgr2=0x7fe32c15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32c0ad4c0 0x7fe32c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32b7fe640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3140047c0 0x7fe32c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 -- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3140047c0 msgr2=0x7fe32c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3140047c0 0x7fe32c155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 -- 172.21.15.107:0/3311232998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c156b80 con 0x7fe32c0c89d0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.245+0000 7fe32bfff640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0ccd80 tx=0x7fe32400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.246+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32400ca60 con 0x7fe32c0c89d0 2023-11-24T08:19:18.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.246+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32400f040 con 0x7fe32c0c89d0 2023-11-24T08:19:18.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.246+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c156e60 con 0x7fe32c0c89d0 2023-11-24T08:19:18.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.246+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32400ce00 con 0x7fe32c0c89d0 2023-11-24T08:19:18.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.246+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c15aed0 con 0x7fe32c0c89d0 2023-11-24T08:19:18.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.247+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c157380 con 0x7fe32c0c89d0 2023-11-24T08:19:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.248+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fe324014480 con 0x7fe32c0c89d0 2023-11-24T08:19:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.249+0000 7fe313fff640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 0x7fe31806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.249+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe324089080 con 0x7fe32c0c89d0 2023-11-24T08:19:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.250+0000 7fe32b7fe640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 0x7fe31806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.250+0000 7fe32b7fe640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 0x7fe31806eda0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0681f0 tx=0x7fe31c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.254+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe32c157380 con 0x7fe32c0c89d0 2023-11-24T08:19:18.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.445+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe32c157380 con 0x7fe32c0c89d0 2023-11-24T08:19:18.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.446+0000 7fe313fff640 1 -- 172.21.15.107:0/3311232998 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fe32c157380 con 0x7fe32c0c89d0 2023-11-24T08:19:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:18.449 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":171,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:17.521252+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 msgr2=0x7fe31806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 0x7fe31806eda0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0681f0 tx=0x7fe31c009040 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 msgr2=0x7fe32c0c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0ccd80 tx=0x7fe32400ee70 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 shutdown_connections 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe31806c8e0 0x7fe31806eda0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe32c0c89d0 0x7fe32c0c8e80 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32c0ad4c0 0x7fe32c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 --2- 172.21.15.107:0/3311232998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3140047c0 0x7fe32c155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.449+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 >> 172.21.15.107:0/3311232998 conn(0x7fe32c06a920 msgr2=0x7fe32c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.450+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 shutdown_connections 2023-11-24T08:19:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.450+0000 7fe331c32640 1 -- 172.21.15.107:0/3311232998 wait complete. 2023-11-24T08:19:18.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:19:18.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:19:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.669+0000 7f929edef640 1 Processor -- start 2023-11-24T08:19:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.669+0000 7f929edef640 1 -- start start 2023-11-24T08:19:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929815b260 0x7f929806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 0x7f92980b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 0x7f92980ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92980b1530 con 0x7f929815b260 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929815b640 con 0x7f92980b1080 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.670+0000 7f929edef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929815ed20 con 0x7f929806e580 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929815b260 0x7f929806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929815b260 0x7f929806e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48472/0 (socket says 172.21.15.107:48472) 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 -- 172.21.15.107:0/55076205 learned_addr learned my addr 172.21.15.107:0/55076205 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929d5ec640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 0x7f92980b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 -- 172.21.15.107:0/55076205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 msgr2=0x7f92980b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 0x7f92980b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 -- 172.21.15.107:0/55076205 >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 msgr2=0x7f92980ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929e5ee640 1 --2- 172.21.15.107:0/55076205 >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 0x7f92980ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 --2- 172.21.15.107:0/55076205 >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 0x7f92980ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929dded640 1 -- 172.21.15.107:0/55076205 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929815f8a0 con 0x7f929815b260 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929d5ec640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 0x7f92980b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.671+0000 7f929e5ee640 1 --2- 172.21.15.107:0/55076205 >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 0x7f92980ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.672+0000 7f929dded640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f929806e040 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f929806dfd0 tx=0x7f929402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4e248515495c67 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929cdeb640 1 -- 172.21.15.107:0/55076205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294031750 con 0x7f929815b260 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929cdeb640 1 -- 172.21.15.107:0/55076205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92940318b0 con 0x7f929815b260 2023-11-24T08:19:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929cdeb640 1 -- 172.21.15.107:0/55076205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294031c40 con 0x7f929815b260 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 -- 172.21.15.107:0/55076205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 msgr2=0x7f929806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f929806e040 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f929806dfd0 tx=0x7f929402b7f0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 -- 172.21.15.107:0/55076205 shutdown_connections 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 --2- 172.21.15.107:0/55076205 >> v2:172.21.15.120:3300/0 conn(0x7f92980b1080 0x7f92980ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f929806e580 0x7f92980b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 --2- 172.21.15.107:0/55076205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f929806e040 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 -- 172.21.15.107:0/55076205 >> 172.21.15.107:0/55076205 conn(0x7f92980cbb40 msgr2=0x7f92980cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 -- 172.21.15.107:0/55076205 shutdown_connections 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.673+0000 7f929edef640 1 -- 172.21.15.107:0/55076205 wait complete. 2023-11-24T08:19:18.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.675+0000 7f929edef640 1 Processor -- start 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.675+0000 7f929edef640 1 -- start start 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929806e580 0x7f92981567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92980b1080 0x7f92980b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92980b2cf0 con 0x7f929815b260 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298099c00 con 0x7f929806e580 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.676+0000 7f929edef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92980b2450 con 0x7f92980b1080 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48478/0 (socket says 172.21.15.107:48478) 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 -- 172.21.15.107:0/1769219020 learned_addr learned my addr 172.21.15.107:0/1769219020 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929dded640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929806e580 0x7f92981567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 -- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92980b1080 msgr2=0x7f92980b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92980b1080 0x7f92980b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 -- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929806e580 msgr2=0x7f92981567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929806e580 0x7f92981567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 -- 172.21.15.107:0/1769219020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940094d0 con 0x7f929815b260 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.677+0000 7f929e5ee640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f929000cd50 tx=0x7f929000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.678+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929000f680 con 0x7f929815b260 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.678+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929000f800 con 0x7f929815b260 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.678+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929000fbb0 con 0x7f929815b260 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.678+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92980b2730 con 0x7f929815b260 2023-11-24T08:19:18.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.678+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92980b2fa0 con 0x7f929815b260 2023-11-24T08:19:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.679+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92980999d0 con 0x7f929815b260 2023-11-24T08:19:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.680+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f929000b040 con 0x7f929815b260 2023-11-24T08:19:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.681+0000 7f928e7fc640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 0x7f927c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.681+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9290088ed0 con 0x7f929815b260 2023-11-24T08:19:18.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.682+0000 7f929dded640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 0x7f927c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.682+0000 7f929dded640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 0x7f927c06ed70 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9294035000 tx=0x7f9294008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:18.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.686+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92980999d0 con 0x7f929815b260 2023-11-24T08:19:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.873+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92980999d0 con 0x7f929815b260 2023-11-24T08:19:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.874+0000 7f928e7fc640 1 -- 172.21.15.107:0/1769219020 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92980999d0 con 0x7f929815b260 2023-11-24T08:19:18.876 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 msgr2=0x7f927c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 0x7f927c06ed70 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9294035000 tx=0x7f9294008030 comp rx=0 tx=0).stop 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 msgr2=0x7f92980b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f929000cd50 tx=0x7f929000a650 comp rx=0 tx=0).stop 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 shutdown_connections 2023-11-24T08:19:18.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f927c06c8b0 0x7f927c06ed70 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929815b260 0x7f92980b3730 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92980b1080 0x7f92980b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 --2- 172.21.15.107:0/1769219020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929806e580 0x7f92981567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 >> 172.21.15.107:0/1769219020 conn(0x7f92980cbb40 msgr2=0x7f92980c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 shutdown_connections 2023-11-24T08:19:18.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:18.876+0000 7f929edef640 1 -- 172.21.15.107:0/1769219020 wait complete. 2023-11-24T08:19:18.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.108+0000 7fa64a2bb640 1 Processor -- start 2023-11-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.108+0000 7fa64a2bb640 1 -- start start 2023-11-24T08:19:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.109+0000 7fa64a2bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa644157c70 0x7fa644161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.109+0000 7fa64a2bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa6440cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa64a2bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6440ce0b0 con 0x7fa64415dc80 2023-11-24T08:19:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa64a2bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6440cfe10 con 0x7fa644157c70 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa64a2bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa644161960 con 0x7fa6440cf450 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa643fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa644157c70 0x7fa644161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa643fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa644157c70 0x7fa644161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48488/0 (socket says 172.21.15.107:48488) 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa643fff640 1 -- 172.21.15.107:0/2023690116 learned_addr learned my addr 172.21.15.107:0/2023690116 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa6437fe640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa6440cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa643fff640 1 -- 172.21.15.107:0/2023690116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 msgr2=0x7fa6440cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.110+0000 7fa643fff640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa6440cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.111+0000 7fa643fff640 1 -- 172.21.15.107:0/2023690116 >> v1:172.21.15.107:6789/0 conn(0x7fa64415dc80 legacy=0x7fa64415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.111+0000 7fa643fff640 1 -- 172.21.15.107:0/2023690116 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64415e8d0 con 0x7fa644157c70 2023-11-24T08:19:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa643fff640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644157c70 0x7fa644161df0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fa634009870 tx=0x7fa63402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3cc51b3871e04b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa642ffd640 1 -- 172.21.15.107:0/2023690116 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634031640 con 0x7fa644157c70 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa642ffd640 1 -- 172.21.15.107:0/2023690116 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6340317a0 con 0x7fa644157c70 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa642ffd640 1 -- 172.21.15.107:0/2023690116 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634031b30 con 0x7fa644157c70 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa64a2bb640 1 -- 172.21.15.107:0/2023690116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644157c70 msgr2=0x7fa644161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.112+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644157c70 0x7fa644161df0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fa634009870 tx=0x7fa63402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 -- 172.21.15.107:0/2023690116 shutdown_connections 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa6440cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/2023690116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644157c70 0x7fa644161df0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 -- 172.21.15.107:0/2023690116 >> 172.21.15.107:0/2023690116 conn(0x7fa64406a920 msgr2=0x7fa64406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 -- 172.21.15.107:0/2023690116 shutdown_connections 2023-11-24T08:19:19.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.113+0000 7fa64a2bb640 1 -- 172.21.15.107:0/2023690116 wait complete. 2023-11-24T08:19:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.114+0000 7fa64a2bb640 1 Processor -- start 2023-11-24T08:19:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.115+0000 7fa64a2bb640 1 -- start start 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.115+0000 7fa64a2bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa644146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.115+0000 7fa64a2bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa644157c70 0x7fa6440b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa64a2bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa64a2bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa644146fc0 con 0x7fa64415dc80 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa64a2bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6441465f0 con 0x7fa644157c70 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa64a2bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6440b0790 con 0x7fa6440cf450 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48500/0 (socket says 172.21.15.107:48500) 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 -- 172.21.15.107:0/1790672618 learned_addr learned my addr 172.21.15.107:0/1790672618 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 -- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 msgr2=0x7fa644146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa644146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 -- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa644157c70 msgr2=0x7fa6440b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa644157c70 0x7fa6440b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 -- 172.21.15.107:0/1790672618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340094d0 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.116+0000 7fa648858640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c730 tx=0x7fa63c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.117+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c003b70 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.117+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c003cf0 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.117+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6441459e0 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.117+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c003ea0 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.117+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa644145fa0 con 0x7fa64415dc80 2023-11-24T08:19:19.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.118+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa644067470 con 0x7fa64415dc80 2023-11-24T08:19:19.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.119+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fa63c005520 con 0x7fa64415dc80 2023-11-24T08:19:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.120+0000 7fa640ff9640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 0x7fa63006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.121+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa63c088460 con 0x7fa64415dc80 2023-11-24T08:19:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.121+0000 7fa643fff640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 0x7fa63006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.121+0000 7fa643fff640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 0x7fa63006eda0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa634036000 tx=0x7fa634030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.125+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa63c054f70 con 0x7fa64415dc80 2023-11-24T08:19:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.319+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa644146730 con 0x7fa64415dc80 2023-11-24T08:19:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.320+0000 7fa640ff9640 1 -- 172.21.15.107:0/1790672618 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v171) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa644146730 con 0x7fa64415dc80 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:19.322 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:19:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 msgr2=0x7fa63006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 0x7fa63006eda0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa634036000 tx=0x7fa634030000 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 msgr2=0x7fa644145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c730 tx=0x7fa63c00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 shutdown_connections 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63006c8e0 0x7fa63006eda0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64415dc80 0x7fa644145140 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa644157c70 0x7fa6440b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 --2- 172.21.15.107:0/1790672618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6440cf450 0x7fa644146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 >> 172.21.15.107:0/1790672618 conn(0x7fa64406a920 msgr2=0x7fa6440b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 shutdown_connections 2023-11-24T08:19:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.323+0000 7fa64a2bb640 1 -- 172.21.15.107:0/1790672618 wait complete. 2023-11-24T08:19:19.344 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:19:19.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:19:19.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.564+0000 7fcc97051640 1 Processor -- start 2023-11-24T08:19:19.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.565+0000 7fcc97051640 1 -- start start 2023-11-24T08:19:19.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc97051640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc9015d920 0x7fcc900acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc97051640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc9006ac10 0x7fcc900ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc97051640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc900cfed0 con 0x7fcc900b2400 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc97051640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900caae0 con 0x7fcc9006ac10 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc97051640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900cadc0 con 0x7fcc9015d920 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc955ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc9006ac10 0x7fcc900ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc955ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc9006ac10 0x7fcc900ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48512/0 (socket says 172.21.15.107:48512) 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.566+0000 7fcc955ee640 1 -- 172.21.15.107:0/1919853264 learned_addr learned my addr 172.21.15.107:0/1919853264 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.567+0000 7fcc94ded640 1 --2- 172.21.15.107:0/1919853264 >> v2:172.21.15.120:3300/0 conn(0x7fcc9015d920 0x7fcc900acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.567+0000 7fcc955ee640 1 -- 172.21.15.107:0/1919853264 >> v2:172.21.15.120:3300/0 conn(0x7fcc9015d920 msgr2=0x7fcc900acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.567+0000 7fcc955ee640 1 --2- 172.21.15.107:0/1919853264 >> v2:172.21.15.120:3300/0 conn(0x7fcc9015d920 0x7fcc900acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.567+0000 7fcc955ee640 1 -- 172.21.15.107:0/1919853264 >> v1:172.21.15.107:6789/0 conn(0x7fcc900b2400 legacy=0x7fcc9006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.567+0000 7fcc955ee640 1 -- 172.21.15.107:0/1919853264 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900cb0a0 con 0x7fcc9006ac10 2023-11-24T08:19:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.568+0000 7fcc955ee640 1 --2- 172.21.15.107:0/1919853264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9006ac10 0x7fcc900ccb90 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fcc900d1200 tx=0x7fcc8802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4068d59643c6e39 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.568+0000 7fcc8f7fe640 1 -- 172.21.15.107:0/1919853264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8802f850 con 0x7fcc9006ac10 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.568+0000 7fcc8f7fe640 1 -- 172.21.15.107:0/1919853264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8802f9b0 con 0x7fcc9006ac10 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.568+0000 7fcc8f7fe640 1 -- 172.21.15.107:0/1919853264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8802fd40 con 0x7fcc9006ac10 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 -- 172.21.15.107:0/1919853264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9006ac10 msgr2=0x7fcc900ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 --2- 172.21.15.107:0/1919853264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9006ac10 0x7fcc900ccb90 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fcc900d1200 tx=0x7fcc8802ba30 comp rx=0 tx=0).stop 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 -- 172.21.15.107:0/1919853264 shutdown_connections 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 --2- 172.21.15.107:0/1919853264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9006ac10 0x7fcc900ccb90 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 --2- 172.21.15.107:0/1919853264 >> v2:172.21.15.120:3300/0 conn(0x7fcc9015d920 0x7fcc900acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 -- 172.21.15.107:0/1919853264 >> 172.21.15.107:0/1919853264 conn(0x7fcc900999d0 msgr2=0x7fcc900b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 -- 172.21.15.107:0/1919853264 shutdown_connections 2023-11-24T08:19:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.569+0000 7fcc97051640 1 -- 172.21.15.107:0/1919853264 wait complete. 2023-11-24T08:19:19.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.571+0000 7fcc97051640 1 Processor -- start 2023-11-24T08:19:19.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.571+0000 7fcc97051640 1 -- start start 2023-11-24T08:19:19.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc9006ac10 0x7fcc901596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc9015d920 0x7fcc900b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900c4c00 con 0x7fcc900b2400 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900c4ee0 con 0x7fcc9006ac10 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc97051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900c51c0 con 0x7fcc9015d920 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48516/0 (socket says 172.21.15.107:48516) 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 -- 172.21.15.107:0/1145535805 learned_addr learned my addr 172.21.15.107:0/1145535805 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 -- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc9015d920 msgr2=0x7fcc900b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc9015d920 0x7fcc900b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 -- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc9006ac10 msgr2=0x7fcc901596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc9006ac10 0x7fcc901596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.572+0000 7fcc8ffff640 1 -- 172.21.15.107:0/1145535805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880094d0 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc8ffff640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c450 tx=0x7fcc7800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc780052d0 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78005430 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc900c54a0 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78012870 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.573+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc900c3e70 con 0x7fcc900b2400 2023-11-24T08:19:19.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.574+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc900ceb90 con 0x7fcc900b2400 2023-11-24T08:19:19.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.575+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fcc780129d0 con 0x7fcc900b2400 2023-11-24T08:19:19.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.576+0000 7fcc8d7fa640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 0x7fcc7006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:19.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.576+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcc78087fb0 con 0x7fcc900b2400 2023-11-24T08:19:19.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.576+0000 7fcc94ded640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 0x7fcc7006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:19.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.577+0000 7fcc94ded640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 0x7fcc7006ed70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b5be0 tx=0x7fcc800072f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.581+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc78054b60 con 0x7fcc900b2400 2023-11-24T08:19:19.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:19.771+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcc90067470 con 0x7fcc900b2400 2023-11-24T08:19:20.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.515+0000 7fcc8d7fa640 1 -- 172.21.15.107:0/1145535805 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v172) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcc78058180 con 0x7fcc900b2400 2023-11-24T08:19:20.517 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 msgr2=0x7fcc7006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 0x7fcc7006ed70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b5be0 tx=0x7fcc800072f0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 msgr2=0x7fcc900b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c450 tx=0x7fcc7800c910 comp rx=0 tx=0).stop 2023-11-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 shutdown_connections 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcc7006c8b0 0x7fcc7006ed70 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc9015d920 0x7fcc900b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc900b2400 0x7fcc900b44a0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 --2- 172.21.15.107:0/1145535805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc9006ac10 0x7fcc901596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 >> 172.21.15.107:0/1145535805 conn(0x7fcc900999d0 msgr2=0x7fcc9015c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.519+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 shutdown_connections 2023-11-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.520+0000 7fcc97051640 1 -- 172.21.15.107:0/1145535805 wait complete. 2023-11-24T08:19:20.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:19:20.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.751+0000 7f446d67c640 1 Processor -- start 2023-11-24T08:19:20.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.751+0000 7f446d67c640 1 -- start start 2023-11-24T08:19:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f446d67c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4468072160 0x7f446815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f446d67c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 0x7f446806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f446d67c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446815ec90 con 0x7f446806dc40 2023-11-24T08:19:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f446d67c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446806e090 con 0x7f4468072160 2023-11-24T08:19:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f446d67c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44680cd530 con 0x7f44680c49f0 2023-11-24T08:19:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f4466ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4468072160 0x7f446815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f4466ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4468072160 0x7f446815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48526/0 (socket says 172.21.15.107:48526) 2023-11-24T08:19:20.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.753+0000 7f4466ffd640 1 -- 172.21.15.107:0/3863492000 learned_addr learned my addr 172.21.15.107:0/3863492000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f445ffff640 1 --2- 172.21.15.107:0/3863492000 >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 0x7f446806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f4466ffd640 1 -- 172.21.15.107:0/3863492000 >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 msgr2=0x7f446806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3863492000 >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 0x7f446806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f4466ffd640 1 -- 172.21.15.107:0/3863492000 >> v1:172.21.15.107:6789/0 conn(0x7f446806dc40 legacy=0x7f44680cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f4466ffd640 1 -- 172.21.15.107:0/3863492000 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44680c4ed0 con 0x7f4468072160 2023-11-24T08:19:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.754+0000 7f445ffff640 1 --2- 172.21.15.107:0/3863492000 >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 0x7f446806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.755+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3863492000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f446815e750 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f44680cef60 tx=0x7f445802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d5450a47fcc928 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.755+0000 7f44667fc640 1 -- 172.21.15.107:0/3863492000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458030860 con 0x7f4468072160 2023-11-24T08:19:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.755+0000 7f44667fc640 1 -- 172.21.15.107:0/3863492000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44580309c0 con 0x7f4468072160 2023-11-24T08:19:20.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.755+0000 7f44667fc640 1 -- 172.21.15.107:0/3863492000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458030d50 con 0x7f4468072160 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 -- 172.21.15.107:0/3863492000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 msgr2=0x7f446815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 --2- 172.21.15.107:0/3863492000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f446815e750 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f44680cef60 tx=0x7f445802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 -- 172.21.15.107:0/3863492000 shutdown_connections 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 --2- 172.21.15.107:0/3863492000 >> v2:172.21.15.120:3300/0 conn(0x7f44680c49f0 0x7f446806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 --2- 172.21.15.107:0/3863492000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f446815e750 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 -- 172.21.15.107:0/3863492000 >> 172.21.15.107:0/3863492000 conn(0x7f44680b0c50 msgr2=0x7f44680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 -- 172.21.15.107:0/3863492000 shutdown_connections 2023-11-24T08:19:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.756+0000 7f446d67c640 1 -- 172.21.15.107:0/3863492000 wait complete. 2023-11-24T08:19:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.758+0000 7f446d67c640 1 Processor -- start 2023-11-24T08:19:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.758+0000 7f446d67c640 1 -- start start 2023-11-24T08:19:20.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 0x7f446815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 0x7f44680acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680ae040 con 0x7f4468072160 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446815b160 con 0x7f44680c49f0 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f446d67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446815b440 con 0x7f446806dc40 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f445ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f445ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48528/0 (socket says 172.21.15.107:48528) 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f445ffff640 1 -- 172.21.15.107:0/3612213931 learned_addr learned my addr 172.21.15.107:0/3612213931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.759+0000 7f44677fe640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 0x7f44680acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 0x7f446815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 -- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 msgr2=0x7f446815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 0x7f446815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 -- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 msgr2=0x7f44680acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 0x7f44680acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 -- 172.21.15.107:0/3612213931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580094d0 con 0x7f4468072160 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f44677fe640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 0x7f44680acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 0x7f446815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445ffff640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f445000c7f0 tx=0x7f445000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450007a30 con 0x7f4468072160 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450007b90 con 0x7f4468072160 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446815b720 con 0x7f4468072160 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000ce40 con 0x7f4468072160 2023-11-24T08:19:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.760+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44680adbb0 con 0x7f4468072160 2023-11-24T08:19:20.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.761+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446815b960 con 0x7f4468072160 2023-11-24T08:19:20.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.763+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f44500113f0 con 0x7f4468072160 2023-11-24T08:19:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.764+0000 7f445f7fe640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 0x7f444806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.764+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f44500886b0 con 0x7f4468072160 2023-11-24T08:19:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.764+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 0x7f444806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.765+0000 7f4466ffd640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 0x7f444806ed70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4458036000 tx=0x7f445803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.769+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44500551c0 con 0x7f4468072160 2023-11-24T08:19:20.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:20.959+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f446804e8b0 con 0x7f4468072160 2023-11-24T08:19:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.521+0000 7f445f7fe640 1 -- 172.21.15.107:0/3612213931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v173) v1 ==== 170+0+0 (secure 0 0 0) 0x7f445001f0d0 con 0x7f4468072160 2023-11-24T08:19:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:19:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 msgr2=0x7f444806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 0x7f444806ed70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4458036000 tx=0x7f445803b040 comp rx=0 tx=0).stop 2023-11-24T08:19:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 msgr2=0x7f4468154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f445000c7f0 tx=0x7f445000cbf0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 shutdown_connections 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444806c8b0 0x7f444806ed70 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44680c49f0 0x7f44680acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4468072160 0x7f4468154410 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 --2- 172.21.15.107:0/3612213931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f446806dc40 0x7f446815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 >> 172.21.15.107:0/3612213931 conn(0x7f44680b0c50 msgr2=0x7f446806d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.525+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 shutdown_connections 2023-11-24T08:19:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.526+0000 7f446d67c640 1 -- 172.21.15.107:0/3612213931 wait complete. 2023-11-24T08:19:21.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:19:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.743+0000 7fec75b5c640 1 Processor -- start 2023-11-24T08:19:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.744+0000 7fec75b5c640 1 -- start start 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec75b5c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec7015a1c0 0x7fec7006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec75b5c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 0x7fec700b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec75b5c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7006d430 con 0x7fec700b0b40 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec75b5c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7015ea70 con 0x7fec7015a1c0 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec75b5c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7015ec50 con 0x7fec7015e5e0 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec74b5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec7015a1c0 0x7fec7006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec74b5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec7015a1c0 0x7fec7006d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48544/0 (socket says 172.21.15.107:48544) 2023-11-24T08:19:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.745+0000 7fec74b5a640 1 -- 172.21.15.107:0/124283733 learned_addr learned my addr 172.21.15.107:0/124283733 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec7535b640 1 --2- 172.21.15.107:0/124283733 >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 0x7fec700b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec74b5a640 1 -- 172.21.15.107:0/124283733 >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 msgr2=0x7fec700b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec74b5a640 1 --2- 172.21.15.107:0/124283733 >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 0x7fec700b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec74b5a640 1 -- 172.21.15.107:0/124283733 >> v1:172.21.15.107:6789/0 conn(0x7fec700b0b40 legacy=0x7fec7015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec74b5a640 1 -- 172.21.15.107:0/124283733 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700b14e0 con 0x7fec7015a1c0 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.746+0000 7fec7535b640 1 --2- 172.21.15.107:0/124283733 >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 0x7fec700b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.747+0000 7fec74b5a640 1 --2- 172.21.15.107:0/124283733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec7015a1c0 0x7fec7006d8c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fec700c9ad0 tx=0x7fec6402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34cdb3ca86a1a18 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.747+0000 7fec6f7fe640 1 -- 172.21.15.107:0/124283733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64030870 con 0x7fec7015a1c0 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.747+0000 7fec6f7fe640 1 -- 172.21.15.107:0/124283733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec640309d0 con 0x7fec7015a1c0 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.747+0000 7fec6f7fe640 1 -- 172.21.15.107:0/124283733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64030d60 con 0x7fec7015a1c0 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 -- 172.21.15.107:0/124283733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec7015a1c0 msgr2=0x7fec7006d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 --2- 172.21.15.107:0/124283733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec7015a1c0 0x7fec7006d8c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fec700c9ad0 tx=0x7fec6402b9b0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 -- 172.21.15.107:0/124283733 shutdown_connections 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 --2- 172.21.15.107:0/124283733 >> v2:172.21.15.120:3300/0 conn(0x7fec7015e5e0 0x7fec700b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 --2- 172.21.15.107:0/124283733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec7015a1c0 0x7fec7006d8c0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 -- 172.21.15.107:0/124283733 >> 172.21.15.107:0/124283733 conn(0x7fec700cbb40 msgr2=0x7fec700cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 -- 172.21.15.107:0/124283733 shutdown_connections 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.748+0000 7fec75b5c640 1 -- 172.21.15.107:0/124283733 wait complete. 2023-11-24T08:19:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.750+0000 7fec75b5c640 1 Processor -- start 2023-11-24T08:19:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.750+0000 7fec75b5c640 1 -- start start 2023-11-24T08:19:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec7015a1c0 0x7fec700c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec7015e5e0 0x7fec700c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec700b3310 con 0x7fec700b0b40 2023-11-24T08:19:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70159b40 con 0x7fec7015e5e0 2023-11-24T08:19:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec75b5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700b2790 con 0x7fec7015a1c0 2023-11-24T08:19:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.751+0000 7fec74b5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48556/0 (socket says 172.21.15.107:48556) 2023-11-24T08:19:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 -- 172.21.15.107:0/3448549169 learned_addr learned my addr 172.21.15.107:0/3448549169 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 -- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec7015a1c0 msgr2=0x7fec700c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec7015a1c0 0x7fec700c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 -- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec7015e5e0 msgr2=0x7fec700c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec7015e5e0 0x7fec700c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 -- 172.21.15.107:0/3448549169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640094d0 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.752+0000 7fec74b5a640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fec6402b9f0 tx=0x7fec64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.753+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64030860 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.753+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec64043400 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.753+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec700b2a70 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.753+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec640437b0 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.753+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec700b3030 con 0x7fec700b0b40 2023-11-24T08:19:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.754+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec700c27d0 con 0x7fec700b0b40 2023-11-24T08:19:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.755+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fec64043910 con 0x7fec700b0b40 2023-11-24T08:19:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.756+0000 7fec6d7fa640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 0x7fec5406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.756+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fec6403e070 con 0x7fec700b0b40 2023-11-24T08:19:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.757+0000 7fec6ffff640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 0x7fec5406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.757+0000 7fec6ffff640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 0x7fec5406eda0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fec5c000f60 tx=0x7fec5c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:21.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.760+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec64078c90 con 0x7fec700b0b40 2023-11-24T08:19:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:21.990+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec700c27d0 con 0x7fec700b0b40 2023-11-24T08:19:22.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.571+0000 7fec6d7fa640 1 -- 172.21.15.107:0/3448549169 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v103) v1 ==== 174+0+0 (secure 0 0 0) 0x7fec64034070 con 0x7fec700b0b40 2023-11-24T08:19:22.581 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 24 and data pool 25 2023-11-24T08:19:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 msgr2=0x7fec5406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 0x7fec5406eda0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fec5c000f60 tx=0x7fec5c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 msgr2=0x7fec70159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fec6402b9f0 tx=0x7fec64004730 comp rx=0 tx=0).stop 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 shutdown_connections 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec5406c8e0 0x7fec5406eda0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec7015e5e0 0x7fec700c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec7015a1c0 0x7fec700c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 --2- 172.21.15.107:0/3448549169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec700b0b40 0x7fec70159600 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 >> 172.21.15.107:0/3448549169 conn(0x7fec700cbb40 msgr2=0x7fec70099be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.590+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 shutdown_connections 2023-11-24T08:19:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.591+0000 7fec75b5c640 1 -- 172.21.15.107:0/3448549169 wait complete. 2023-11-24T08:19:22.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.800+0000 7f2d45fe5640 1 Processor -- start 2023-11-24T08:19:22.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.801+0000 7f2d45fe5640 1 -- start start 2023-11-24T08:19:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d45fe5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d400cfb60 0x7f2d40161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d45fe5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400ce0b0 con 0x7f2d40157c70 2023-11-24T08:19:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d45fe5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d40161960 con 0x7f2d4015dc80 2023-11-24T08:19:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d45fe5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4015e8d0 con 0x7f2d400cfb60 2023-11-24T08:19:22.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d3f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2d40157c70 0x7f2d400cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43656/0 (socket says 172.21.15.107:43656) 2023-11-24T08:19:22.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.802+0000 7f2d3f7fe640 1 -- 172.21.15.107:0/4209380097 learned_addr learned my addr 172.21.15.107:0/4209380097 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:22.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d37fff640 1 --2- 172.21.15.107:0/4209380097 >> v2:172.21.15.120:3300/0 conn(0x7f2d400cfb60 0x7f2d40161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264192197 0 0) 0x7f2d400ce0b0 con 0x7f2d40157c70 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d2c003520 con 0x7f2d40157c70 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771179464 0 0) 0x7f2d2c003520 con 0x7f2d40157c70 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 >> v2:172.21.15.120:3300/0 conn(0x7f2d400cfb60 msgr2=0x7f2d40161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 --2- 172.21.15.107:0/4209380097 >> v2:172.21.15.120:3300/0 conn(0x7f2d400cfb60 0x7f2d40161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 >> v1:172.21.15.120:6789/0 conn(0x7f2d4015dc80 legacy=0x7f2d400b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4015ebb0 con 0x7f2d40157c70 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2d30001670 con 0x7f2d40157c70 2023-11-24T08:19:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.803+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 >> v1:172.21.15.107:6789/0 conn(0x7f2d40157c70 legacy=0x7f2d400cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c003970 0x7f2d2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d2c0047c0 0x7f2d2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 0x7f2d2c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4015e8d0 con 0x7f2d2c005180 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c00c120 con 0x7f2d2c003970 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c00c2e0 con 0x7f2d2c0047c0 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 0x7f2d2c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3f7fe640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c003970 0x7f2d2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 -- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d2c0047c0 msgr2=0x7f2d2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d2c0047c0 0x7f2d2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 -- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c003970 msgr2=0x7f2d2c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c003970 0x7f2d2c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3ffff640 1 -- 172.21.15.107:0/4209380097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c003520 con 0x7f2d2c005180 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.804+0000 7f2d3f7fe640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c003970 0x7f2d2c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.805+0000 7f2d3ffff640 1 -- 172.21.15.107:0/4209380097 reap_dead start 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.805+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 0x7f2d2c00b980 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f2d38001e50 tx=0x7f2d2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51518372380f26d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38003b70 con 0x7f2d2c005180 2023-11-24T08:19:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d38003cf0 con 0x7f2d2c005180 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d3effd640 1 -- 172.21.15.107:0/4209380097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38005310 con 0x7f2d2c005180 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 -- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 msgr2=0x7f2d2c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 0x7f2d2c00b980 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f2d38001e50 tx=0x7f2d2c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 -- 172.21.15.107:0/4209380097 shutdown_connections 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/4209380097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d2c005180 0x7f2d2c00b980 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 -- 172.21.15.107:0/4209380097 >> 172.21.15.107:0/4209380097 conn(0x7f2d4006a920 msgr2=0x7f2d4006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 -- 172.21.15.107:0/4209380097 shutdown_connections 2023-11-24T08:19:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.806+0000 7f2d45fe5640 1 -- 172.21.15.107:0/4209380097 wait complete. 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.808+0000 7f2d45fe5640 1 Processor -- start 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.808+0000 7f2d45fe5640 1 -- start start 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c005180 0x7f2d401f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d4015ff10 0x7f2d40160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d401608d0 con 0x7f2d400abfc0 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d400accf0 con 0x7f2d2c005180 2023-11-24T08:19:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.809+0000 7f2d45fe5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d400acfd0 con 0x7f2d4015ff10 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48578/0 (socket says 172.21.15.107:48578) 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 -- 172.21.15.107:0/2937288091 learned_addr learned my addr 172.21.15.107:0/2937288091 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3f7fe640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c005180 0x7f2d401f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 -- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d4015ff10 msgr2=0x7f2d40160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d37fff640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d4015ff10 0x7f2d40160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d4015ff10 0x7f2d40160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 -- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c005180 msgr2=0x7f2d401f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c005180 0x7f2d401f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 -- 172.21.15.107:0/2937288091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400ad2b0 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.810+0000 7f2d3ffff640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2d38001f70 tx=0x7f2d380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.811+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38006d50 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.811+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d38039480 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.811+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d400ad590 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.811+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38039830 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.811+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d401f0c60 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.812+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d400ad7b0 con 0x7f2d400abfc0 2023-11-24T08:19:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.813+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f2d38039990 con 0x7f2d400abfc0 2023-11-24T08:19:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.814+0000 7f2d3cff9640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 0x7f2d2006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.814+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d380aa170 con 0x7f2d400abfc0 2023-11-24T08:19:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.814+0000 7f2d3f7fe640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 0x7f2d2006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:22.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.815+0000 7f2d3f7fe640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 0x7f2d2006eda0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f2d401f79a0 tx=0x7f2d30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:22.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:22.818+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d38076d30 con 0x7f2d400abfc0 2023-11-24T08:19:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.007+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d40067470 con 0x7f2d400abfc0 2023-11-24T08:19:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.010+0000 7f2d3cff9640 1 -- 172.21.15.107:0/2937288091 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f2d38048020 con 0x7f2d400abfc0 2023-11-24T08:19:23.012 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:23.012 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":174,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:21.992723+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.012+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 msgr2=0x7f2d2006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.012+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 0x7f2d2006eda0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f2d401f79a0 tx=0x7f2d30009040 comp rx=0 tx=0).stop 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.012+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 msgr2=0x7f2d400ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.012+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2d38001f70 tx=0x7f2d380014a0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 shutdown_connections 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d2006c8e0 0x7f2d2006eda0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d400abfc0 0x7f2d400ac470 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d4015ff10 0x7f2d40160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 --2- 172.21.15.107:0/2937288091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d2c005180 0x7f2d401f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 >> 172.21.15.107:0/2937288091 conn(0x7f2d4006a920 msgr2=0x7f2d4015e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:23.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 shutdown_connections 2023-11-24T08:19:23.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.013+0000 7f2d45fe5640 1 -- 172.21.15.107:0/2937288091 wait complete. 2023-11-24T08:19:23.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.235+0000 7fc8d711d640 1 Processor -- start 2023-11-24T08:19:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.235+0000 7fc8d711d640 1 -- start start 2023-11-24T08:19:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d711d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8d015c7f0 0x7fc8d006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d711d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00ce6c0 0x7fc8d00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d711d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00cece0 con 0x7fc8d00cf450 2023-11-24T08:19:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d711d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d00cfe40 con 0x7fc8d015c7f0 2023-11-24T08:19:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d711d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0156010 con 0x7fc8d00ce6c0 2023-11-24T08:19:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d4eb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8d015c7f0 0x7fc8d006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d4eb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8d015c7f0 0x7fc8d006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48592/0 (socket says 172.21.15.107:48592) 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d4eb9640 1 -- 172.21.15.107:0/753058199 learned_addr learned my addr 172.21.15.107:0/753058199 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.237+0000 7fc8d56ba640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00ce6c0 0x7fc8d00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.238+0000 7fc8d4eb9640 1 -- 172.21.15.107:0/753058199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00ce6c0 msgr2=0x7fc8d00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.238+0000 7fc8d4eb9640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00ce6c0 0x7fc8d00cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.238+0000 7fc8d4eb9640 1 -- 172.21.15.107:0/753058199 >> v1:172.21.15.120:6789/0 conn(0x7fc8d00cf450 legacy=0x7fc8d00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.238+0000 7fc8d4eb9640 1 -- 172.21.15.107:0/753058199 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d006d040 con 0x7fc8d015c7f0 2023-11-24T08:19:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.239+0000 7fc8d4eb9640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d006db20 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009870 tx=0x7fc8c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b37b0aadd80f000 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.240+0000 7fc8cf7fe640 1 -- 172.21.15.107:0/753058199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4031640 con 0x7fc8d015c7f0 2023-11-24T08:19:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.240+0000 7fc8cf7fe640 1 -- 172.21.15.107:0/753058199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c40317a0 con 0x7fc8d015c7f0 2023-11-24T08:19:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.240+0000 7fc8cf7fe640 1 -- 172.21.15.107:0/753058199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4031b30 con 0x7fc8d015c7f0 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.240+0000 7fc8d711d640 1 -- 172.21.15.107:0/753058199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 msgr2=0x7fc8d006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.240+0000 7fc8d711d640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d006db20 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009870 tx=0x7fc8c402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 -- 172.21.15.107:0/753058199 shutdown_connections 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00ce6c0 0x7fc8d00cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 --2- 172.21.15.107:0/753058199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d006db20 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 -- 172.21.15.107:0/753058199 >> 172.21.15.107:0/753058199 conn(0x7fc8d006a920 msgr2=0x7fc8d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 -- 172.21.15.107:0/753058199 shutdown_connections 2023-11-24T08:19:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.241+0000 7fc8d711d640 1 -- 172.21.15.107:0/753058199 wait complete. 2023-11-24T08:19:23.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.242+0000 7fc8d711d640 1 Processor -- start 2023-11-24T08:19:23.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 -- start start 2023-11-24T08:19:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8d00ce6c0 0x7fc8d0149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00cf450 0x7fc8d0156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0157940 con 0x7fc8d015c7f0 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0156cc0 con 0x7fc8d00ce6c0 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.243+0000 7fc8d711d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0156fa0 con 0x7fc8d00cf450 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48600/0 (socket says 172.21.15.107:48600) 2023-11-24T08:19:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 -- 172.21.15.107:0/3239547678 learned_addr learned my addr 172.21.15.107:0/3239547678 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:23.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d4eb9640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8d00ce6c0 0x7fc8d0149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 -- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00cf450 msgr2=0x7fc8d0156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00cf450 0x7fc8d0156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 -- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8d00ce6c0 msgr2=0x7fc8d0149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8d00ce6c0 0x7fc8d0149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 -- 172.21.15.107:0/3239547678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40094d0 con 0x7fc8d015c7f0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.244+0000 7fc8d56ba640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8007d90 tx=0x7fc8c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.245+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8014420 con 0x7fc8d015c7f0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.245+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c80145a0 con 0x7fc8d015c7f0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.245+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8014950 con 0x7fc8d015c7f0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.245+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0157280 con 0x7fc8d015c7f0 2023-11-24T08:19:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.245+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0149940 con 0x7fc8d015c7f0 2023-11-24T08:19:23.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.246+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d00cc890 con 0x7fc8d015c7f0 2023-11-24T08:19:23.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.247+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fc8c8014ab0 con 0x7fc8d015c7f0 2023-11-24T08:19:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.248+0000 7fc8cd7fa640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 0x7fc8b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.248+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8c8088e60 con 0x7fc8d015c7f0 2023-11-24T08:19:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.248+0000 7fc8d4eb9640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 0x7fc8b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.249+0000 7fc8d4eb9640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 0x7fc8b406ed70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4036000 tx=0x7fc8c403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.252+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8c8055970 con 0x7fc8d015c7f0 2023-11-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.441+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8d0067470 con 0x7fc8d015c7f0 2023-11-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.444+0000 7fc8cd7fa640 1 -- 172.21.15.107:0/3239547678 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16592 (secure 0 0 0) 0x7fc8c8058f90 con 0x7fc8d015c7f0 2023-11-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":174,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:21.992723+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 msgr2=0x7fc8b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 0x7fc8b406ed70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4036000 tx=0x7fc8c403c040 comp rx=0 tx=0).stop 2023-11-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 msgr2=0x7fc8d0156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8007d90 tx=0x7fc8c800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 shutdown_connections 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8b406c8b0 0x7fc8b406ed70 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8d015c7f0 0x7fc8d0156690 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8d00cf450 0x7fc8d0156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 --2- 172.21.15.107:0/3239547678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8d00ce6c0 0x7fc8d0149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.447+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 >> 172.21.15.107:0/3239547678 conn(0x7fc8d006a920 msgr2=0x7fc8d006ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.448+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 shutdown_connections 2023-11-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.448+0000 7fc8d711d640 1 -- 172.21.15.107:0/3239547678 wait complete. 2023-11-24T08:19:23.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:19:23.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.673+0000 7f6b1a52c640 1 Processor -- start 2023-11-24T08:19:23.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.673+0000 7f6b1a52c640 1 -- start start 2023-11-24T08:19:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b1a52c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 0x7f6b14161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b1a52c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b14161a10 con 0x7f6b1406e160 2023-11-24T08:19:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b1a52c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b14161cf0 con 0x7f6b140cd2e0 2023-11-24T08:19:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b1a52c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14161fd0 con 0x7f6b14146eb0 2023-11-24T08:19:23.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b0bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6b1406e160 0x7f6b140ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43666/0 (socket says 172.21.15.107:43666) 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.675+0000 7f6b0bfff640 1 -- 172.21.15.107:0/3146208743 learned_addr learned my addr 172.21.15.107:0/3146208743 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 0x7f6b14161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099635871 0 0) 0x7f6b14161a10 con 0x7f6b1406e160 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b00003520 con 0x7f6b1406e160 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 966435300 0 0) 0x7f6b00003520 con 0x7f6b1406e160 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 msgr2=0x7f6b14161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 0x7f6b14161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 >> v1:172.21.15.120:6789/0 conn(0x7f6b140cd2e0 legacy=0x7f6b14070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b14070660 con 0x7f6b1406e160 2023-11-24T08:19:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6afc001670 con 0x7f6b1406e160 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 >> v1:172.21.15.107:6789/0 conn(0x7f6b1406e160 legacy=0x7f6b140ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b000039a0 0x7f6b00003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000047c0 0x7f6b00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 0x7f6b00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14161fd0 con 0x7f6b000039a0 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0000ba30 con 0x7f6b000047c0 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.676+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0000bbf0 con 0x7f6b14146eb0 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b000039a0 0x7f6b00003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 -- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 msgr2=0x7f6b00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b14146eb0 0x7f6b00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 -- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000047c0 msgr2=0x7f6b00004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:23.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000047c0 0x7f6b00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.677+0000 7f6b13fff640 1 -- 172.21.15.107:0/3146208743 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b00003520 con 0x7f6b000039a0 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.678+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b000039a0 0x7f6b00003dc0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009a90 tx=0x7f6b000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4081ee78a297e119 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.678+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0402c6b0 con 0x7f6b000039a0 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.678+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0402c810 con 0x7f6b000039a0 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.678+0000 7f6b137fe640 1 -- 172.21.15.107:0/3146208743 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0402cba0 con 0x7f6b000039a0 2023-11-24T08:19:23.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3146208743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b000039a0 msgr2=0x7f6b00003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3146208743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b000039a0 0x7f6b00003dc0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009a90 tx=0x7f6b000057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b18ac9640 1 -- 172.21.15.107:0/3146208743 reap_dead start 2023-11-24T08:19:23.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3146208743 shutdown_connections 2023-11-24T08:19:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3146208743 >> 172.21.15.107:0/3146208743 conn(0x7f6b140b0c50 msgr2=0x7f6b140b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3146208743 shutdown_connections 2023-11-24T08:19:23.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.679+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3146208743 wait complete. 2023-11-24T08:19:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.681+0000 7f6b1a52c640 1 Processor -- start 2023-11-24T08:19:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.681+0000 7f6b1a52c640 1 -- start start 2023-11-24T08:19:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000039a0 0x7f6b1415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b140adc60 0x7f6b1409b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1409bf90 con 0x7f6b14142fd0 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14160860 con 0x7f6b000039a0 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b1a52c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141439a0 con 0x7f6b140adc60 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b18ac9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b18ac9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48622/0 (socket says 172.21.15.107:48622) 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b18ac9640 1 -- 172.21.15.107:0/3156122121 learned_addr learned my addr 172.21.15.107:0/3156122121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000039a0 0x7f6b1415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.682+0000 7f6b0bfff640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b140adc60 0x7f6b1409b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 -- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b140adc60 msgr2=0x7f6b1409b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b140adc60 0x7f6b1409b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 -- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000039a0 msgr2=0x7f6b1415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000039a0 0x7f6b1415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 -- 172.21.15.107:0/3156122121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1415cf40 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b18ac9640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f6b1406ea50 tx=0x7f6b0c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c005d40 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c00ced0 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00a850 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1415dc30 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.683+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1415e1f0 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.684+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1404e8b0 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.685+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f6b0c00a9b0 con 0x7f6b14142fd0 2023-11-24T08:19:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.686+0000 7f6b117fa640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 0x7f6af406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:23.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.686+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b0c087590 con 0x7f6b14142fd0 2023-11-24T08:19:23.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.686+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 0x7f6af406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:23.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.687+0000 7f6b13fff640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 0x7f6af406ed70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6b04036000 tx=0x7f6b04030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:23.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.691+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b0c054140 con 0x7f6b14142fd0 2023-11-24T08:19:23.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:23.919+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6b140cf7b0 con 0x7f6b14142fd0 2023-11-24T08:19:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.591+0000 7f6b117fa640 1 -- 172.21.15.107:0/3156122121 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v106) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6b0c0170b0 con 0x7f6b14142fd0 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.594+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 msgr2=0x7f6af406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.594+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 0x7f6af406ed70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6b04036000 tx=0x7f6b04030000 comp rx=0 tx=0).stop 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.594+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 msgr2=0x7f6b14143460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.594+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f6b1406ea50 tx=0x7f6b0c00cb90 comp rx=0 tx=0).stop 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 shutdown_connections 2023-11-24T08:19:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6af406c8b0 0x7f6af406ed70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b14142fd0 0x7f6b14143460 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b140adc60 0x7f6b1409b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 --2- 172.21.15.107:0/3156122121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b000039a0 0x7f6b1415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 >> 172.21.15.107:0/3156122121 conn(0x7f6b140b0c50 msgr2=0x7f6b140aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 shutdown_connections 2023-11-24T08:19:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.595+0000 7f6b1a52c640 1 -- 172.21.15.107:0/3156122121 wait complete. 2023-11-24T08:19:24.615 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:19:24.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:19:24.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.833+0000 7fd35a4a6640 1 Processor -- start 2023-11-24T08:19:24.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.833+0000 7fd35a4a6640 1 -- start start 2023-11-24T08:19:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.834+0000 7fd35a4a6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd3541684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.834+0000 7fd35a4a6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354161a10 con 0x7fd35406e160 2023-11-24T08:19:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.834+0000 7fd35a4a6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354161cf0 con 0x7fd354146eb0 2023-11-24T08:19:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.834+0000 7fd35a4a6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd354161fd0 con 0x7fd3540cd2e0 2023-11-24T08:19:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.834+0000 7fd3537fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd35406e160 0x7fd3540ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43668/0 (socket says 172.21.15.107:43668) 2023-11-24T08:19:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd3537fe640 1 -- 172.21.15.107:0/3665795773 learned_addr learned my addr 172.21.15.107:0/3665795773 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd358a43640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd3541684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058317527 0 0) 0x7fd354161a10 con 0x7fd35406e160 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003520 con 0x7fd35406e160 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438334172 0 0) 0x7fd340003520 con 0x7fd35406e160 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 msgr2=0x7fd3541684e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd3541684e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 >> v1:172.21.15.120:6789/0 conn(0x7fd354146eb0 legacy=0x7fd354161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354070660 con 0x7fd35406e160 2023-11-24T08:19:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd33c001a10 con 0x7fd35406e160 2023-11-24T08:19:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.835+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 >> v1:172.21.15.107:6789/0 conn(0x7fd35406e160 legacy=0x7fd3540ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd340003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3400047c0 0x7fd340004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd340005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354161fd0 con 0x7fd3400039a0 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34000bab0 con 0x7fd3400047c0 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34000bc70 con 0x7fd3540cd2e0 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd340003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd358a43640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd340005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd3537fe640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3400047c0 0x7fd340004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 -- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 msgr2=0x7fd340005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3540cd2e0 0x7fd340005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 -- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3400047c0 msgr2=0x7fd340004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3400047c0 0x7fd340004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.836+0000 7fd353fff640 1 -- 172.21.15.107:0/3665795773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340003520 con 0x7fd3400039a0 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.837+0000 7fd353fff640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd340003dc0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fd344001750 tx=0x7fd3400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57919baed2c2ee02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440052d0 con 0x7fd3400039a0 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34402e050 con 0x7fd3400039a0 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd352ffd640 1 -- 172.21.15.107:0/3665795773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440035f0 con 0x7fd3400039a0 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 -- 172.21.15.107:0/3665795773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 msgr2=0x7fd340003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/3665795773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd340003dc0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fd344001750 tx=0x7fd3400057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd358a43640 1 -- 172.21.15.107:0/3665795773 reap_dead start 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 -- 172.21.15.107:0/3665795773 shutdown_connections 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 -- 172.21.15.107:0/3665795773 >> 172.21.15.107:0/3665795773 conn(0x7fd3540b0c50 msgr2=0x7fd3540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 -- 172.21.15.107:0/3665795773 shutdown_connections 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.838+0000 7fd35a4a6640 1 -- 172.21.15.107:0/3665795773 wait complete. 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.840+0000 7fd35a4a6640 1 Processor -- start 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.840+0000 7fd35a4a6640 1 -- start start 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd354144010 0x7fd35415f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd35415fdd0 0x7fd3540b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354160250 con 0x7fd3400039a0 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3541f4040 con 0x7fd354144010 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd35a4a6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35415df40 con 0x7fd35415fdd0 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd353fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd353fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48640/0 (socket says 172.21.15.107:48640) 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.841+0000 7fd353fff640 1 -- 172.21.15.107:0/1294508873 learned_addr learned my addr 172.21.15.107:0/1294508873 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 -- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd35415fdd0 msgr2=0x7fd3540b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd35415fdd0 0x7fd3540b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 -- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd354144010 msgr2=0x7fd35415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd354144010 0x7fd35415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 -- 172.21.15.107:0/1294508873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35415e220 con 0x7fd3400039a0 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd353fff640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd344008fd0 tx=0x7fd344001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.842+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344007a30 con 0x7fd3400039a0 2023-11-24T08:19:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.843+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344007640 con 0x7fd3400039a0 2023-11-24T08:19:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.843+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344006be0 con 0x7fd3400039a0 2023-11-24T08:19:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.843+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35415e500 con 0x7fd3400039a0 2023-11-24T08:19:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.843+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35415eac0 con 0x7fd3400039a0 2023-11-24T08:19:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.844+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3541601f0 con 0x7fd3400039a0 2023-11-24T08:19:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.845+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fd344006d40 con 0x7fd3400039a0 2023-11-24T08:19:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.846+0000 7fd350ff9640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 0x7fd33406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.846+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3440aa1b0 con 0x7fd3400039a0 2023-11-24T08:19:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.846+0000 7fd3537fe640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 0x7fd33406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.847+0000 7fd3537fe640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 0x7fd33406ed70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd33c007c40 tx=0x7fd33c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:24.850+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3541601f0 con 0x7fd3400039a0 2023-11-24T08:19:25.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:25.080+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd3541601f0 con 0x7fd3400039a0 2023-11-24T08:19:26.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.059+0000 7fd350ff9640 1 -- 172.21.15.107:0/1294508873 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v107) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd3541601f0 con 0x7fd3400039a0 2023-11-24T08:19:26.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.062+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 msgr2=0x7fd33406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.062+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 0x7fd33406ed70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd33c007c40 tx=0x7fd33c009040 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 msgr2=0x7fd3541f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd344008fd0 tx=0x7fd344001120 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 shutdown_connections 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd33406c8b0 0x7fd33406ed70 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd35415fdd0 0x7fd3540b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd354144010 0x7fd35415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 --2- 172.21.15.107:0/1294508873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3400039a0 0x7fd3541f3b00 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 >> 172.21.15.107:0/1294508873 conn(0x7fd3540b0c50 msgr2=0x7fd3540aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:26.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 shutdown_connections 2023-11-24T08:19:26.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.063+0000 7fd35a4a6640 1 -- 172.21.15.107:0/1294508873 wait complete. 2023-11-24T08:19:26.086 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:19:26.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:26.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.300+0000 7f2dc6341640 1 Processor -- start 2023-11-24T08:19:26.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.301+0000 7f2dc6341640 1 -- start start 2023-11-24T08:19:26.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.302+0000 7f2dc6341640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2dc00b2250 0x7f2dc015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.302+0000 7f2dc6341640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00b2630 0x7f2dc00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.302+0000 7f2dc6341640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc015ab80 con 0x7f2dc00cd470 2023-11-24T08:19:26.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.302+0000 7f2dc6341640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc015a990 con 0x7f2dc00b2250 2023-11-24T08:19:26.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.302+0000 7f2dc6341640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc015e680 con 0x7f2dc00b2630 2023-11-24T08:19:26.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.303+0000 7f2dbffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2dc00b2250 0x7f2dc015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.303+0000 7f2dbf7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00b2630 0x7f2dc00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.303+0000 7f2dc48de640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2dc00cd470 0x7f2dc015b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:45180/0 (socket says 172.21.15.107:45180) 2023-11-24T08:19:26.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.303+0000 7f2dc48de640 1 -- 172.21.15.107:0/758376282 learned_addr learned my addr 172.21.15.107:0/758376282 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:26.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.304+0000 7f2dbffff640 1 -- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00b2630 msgr2=0x7f2dc00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:26.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.304+0000 7f2dbffff640 1 --2- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00b2630 0x7f2dc00ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.304+0000 7f2dbffff640 1 -- 172.21.15.107:0/758376282 >> v1:172.21.15.120:6789/0 conn(0x7f2dc00cd470 legacy=0x7f2dc015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.304+0000 7f2dbffff640 1 -- 172.21.15.107:0/758376282 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc015e960 con 0x7f2dc00b2250 2023-11-24T08:19:26.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.305+0000 7f2dbffff640 1 --2- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc015df80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2db00097a0 tx=0x7f2db002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361c94a64454e66f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.306+0000 7f2dbeffd640 1 -- 172.21.15.107:0/758376282 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db002d040 con 0x7f2dc00b2250 2023-11-24T08:19:26.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.306+0000 7f2dbeffd640 1 -- 172.21.15.107:0/758376282 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0031810 con 0x7f2dc00b2250 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.306+0000 7f2dbeffd640 1 -- 172.21.15.107:0/758376282 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0031ba0 con 0x7f2dc00b2250 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 -- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 msgr2=0x7f2dc015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 --2- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc015df80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2db00097a0 tx=0x7f2db002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 -- 172.21.15.107:0/758376282 shutdown_connections 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 --2- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00b2630 0x7f2dc00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 --2- 172.21.15.107:0/758376282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc015df80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 -- 172.21.15.107:0/758376282 >> 172.21.15.107:0/758376282 conn(0x7f2dc006a920 msgr2=0x7f2dc006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:26.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 -- 172.21.15.107:0/758376282 shutdown_connections 2023-11-24T08:19:26.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.307+0000 7f2dc6341640 1 -- 172.21.15.107:0/758376282 wait complete. 2023-11-24T08:19:26.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.309+0000 7f2dc6341640 1 Processor -- start 2023-11-24T08:19:26.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.309+0000 7f2dc6341640 1 -- start start 2023-11-24T08:19:26.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc00abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00cd470 0x7f2dc014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00afb90 con 0x7f2dc00b2630 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00ac8c0 con 0x7f2dc00b2250 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc6341640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00acba0 con 0x7f2dc00cd470 2023-11-24T08:19:26.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48656/0 (socket says 172.21.15.107:48656) 2023-11-24T08:19:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 -- 172.21.15.107:0/798970107 learned_addr learned my addr 172.21.15.107:0/798970107 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbffff640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc00abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 -- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00cd470 msgr2=0x7f2dc014d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dc48de640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00cd470 0x7f2dc014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00cd470 0x7f2dc014d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 -- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 msgr2=0x7f2dc00abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc00abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.311+0000 7f2dbf7fe640 1 -- 172.21.15.107:0/798970107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0009400 con 0x7f2dc00b2630 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dbf7fe640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f2db4010dc0 tx=0x7f2db400e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db400b910 con 0x7f2dc00b2630 2023-11-24T08:19:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db4011860 con 0x7f2dc00b2630 2023-11-24T08:19:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db4011c10 con 0x7f2dc00b2630 2023-11-24T08:19:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc00ace80 con 0x7f2dc00b2630 2023-11-24T08:19:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.312+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc00ac500 con 0x7f2dc00b2630 2023-11-24T08:19:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.313+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc00ad3a0 con 0x7f2dc00b2630 2023-11-24T08:19:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.314+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f2db4019440 con 0x7f2dc00b2630 2023-11-24T08:19:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.315+0000 7f2dbcff9640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 0x7f2dac06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.315+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2db408b3c0 con 0x7f2dc00b2630 2023-11-24T08:19:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.316+0000 7f2dbffff640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 0x7f2dac06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.316+0000 7f2dbffff640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 0x7f2dac06ed70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2db0036000 tx=0x7f2db003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.320+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dc00ad3a0 con 0x7f2dc00b2630 2023-11-24T08:19:26.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.549+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dc00ad3a0 con 0x7f2dc00b2630 2023-11-24T08:19:26.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.551+0000 7f2dbcff9640 1 -- 172.21.15.107:0/798970107 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f2dc00ad3a0 con 0x7f2dc00b2630 2023-11-24T08:19:26.553 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:26.553 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:19:26.553 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 107 2023-11-24T08:19:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 msgr2=0x7f2dac06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 0x7f2dac06ed70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2db0036000 tx=0x7f2db003c040 comp rx=0 tx=0).stop 2023-11-24T08:19:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 msgr2=0x7f2dc0071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f2db4010dc0 tx=0x7f2db400e730 comp rx=0 tx=0).stop 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 shutdown_connections 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2dac06c8b0 0x7f2dac06ed70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dc00cd470 0x7f2dc014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2dc00b2630 0x7f2dc0071880 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 --2- 172.21.15.107:0/798970107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dc00b2250 0x7f2dc00abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.553+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 >> 172.21.15.107:0/798970107 conn(0x7f2dc006a920 msgr2=0x7f2dc00cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.554+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 shutdown_connections 2023-11-24T08:19:26.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.554+0000 7f2dc6341640 1 -- 172.21.15.107:0/798970107 wait complete. 2023-11-24T08:19:26.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:19:26.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.783+0000 7f367809b640 1 Processor -- start 2023-11-24T08:19:26.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.783+0000 7f367809b640 1 -- start start 2023-11-24T08:19:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f367809b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f367015af40 0x7f367006d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f367809b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015b320 0x7f36700ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f367809b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367015e8b0 con 0x7f3670070c40 2023-11-24T08:19:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f367809b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367015eb90 con 0x7f367015af40 2023-11-24T08:19:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f367809b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367015e6a0 con 0x7f367015b320 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f3675e37640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f367015af40 0x7f367006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f3675636640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015b320 0x7f36700ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f3675e37640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f367015af40 0x7f367006d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:43404/0 (socket says 172.21.15.107:43404) 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.785+0000 7f3675e37640 1 -- 172.21.15.107:0/763026496 learned_addr learned my addr 172.21.15.107:0/763026496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.786+0000 7f3675e37640 1 -- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015b320 msgr2=0x7f36700ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.786+0000 7f3675e37640 1 --2- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015b320 0x7f36700ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.786+0000 7f3675e37640 1 -- 172.21.15.107:0/763026496 >> v1:172.21.15.120:6789/0 conn(0x7f3670070c40 legacy=0x7f3670161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.786+0000 7f3675e37640 1 -- 172.21.15.107:0/763026496 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670070180 con 0x7f367015af40 2023-11-24T08:19:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.787+0000 7f3675e37640 1 --2- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015af40 0x7f367006d220 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f366c009870 tx=0x7f366c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e9e2eb0cba08069 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.788+0000 7f3674e35640 1 -- 172.21.15.107:0/763026496 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c030840 con 0x7f367015af40 2023-11-24T08:19:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.788+0000 7f3674e35640 1 -- 172.21.15.107:0/763026496 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c0309a0 con 0x7f367015af40 2023-11-24T08:19:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.788+0000 7f3674e35640 1 -- 172.21.15.107:0/763026496 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c030d30 con 0x7f367015af40 2023-11-24T08:19:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 -- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015af40 msgr2=0x7f367006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 --2- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015af40 0x7f367006d220 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f366c009870 tx=0x7f366c02b980 comp rx=0 tx=0).stop 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 -- 172.21.15.107:0/763026496 shutdown_connections 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 --2- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015b320 0x7f36700ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 --2- 172.21.15.107:0/763026496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015af40 0x7f367006d220 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 -- 172.21.15.107:0/763026496 >> 172.21.15.107:0/763026496 conn(0x7f367006e7f0 msgr2=0x7f367006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 -- 172.21.15.107:0/763026496 shutdown_connections 2023-11-24T08:19:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.789+0000 7f367809b640 1 -- 172.21.15.107:0/763026496 wait complete. 2023-11-24T08:19:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.790+0000 7f367809b640 1 Processor -- start 2023-11-24T08:19:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.791+0000 7f367809b640 1 -- start start 2023-11-24T08:19:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015af40 0x7f36700b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 0x7f367006b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670151860 con 0x7f3670070c40 2023-11-24T08:19:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36700b5c30 con 0x7f367015b320 2023-11-24T08:19:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f367809b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36700b5e80 con 0x7f367015af40 2023-11-24T08:19:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48670/0 (socket says 172.21.15.107:48670) 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 -- 172.21.15.107:0/3859831305 learned_addr learned my addr 172.21.15.107:0/3859831305 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3676638640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 0x7f367006b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 -- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015af40 msgr2=0x7f36700b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675636640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015af40 0x7f36700b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015af40 0x7f36700b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 -- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 msgr2=0x7f367006b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 0x7f367006b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3675e37640 1 -- 172.21.15.107:0/3859831305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0094d0 con 0x7f3670070c40 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.792+0000 7f3676638640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 0x7f367006b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f3675e37640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c039a90 con 0x7f3670070c40 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c039bf0 con 0x7f3670070c40 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c043410 con 0x7f3670070c40 2023-11-24T08:19:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36700b6160 con 0x7f3670070c40 2023-11-24T08:19:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.793+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36700b6720 con 0x7f3670070c40 2023-11-24T08:19:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.794+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36700d0a50 con 0x7f3670070c40 2023-11-24T08:19:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.795+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f366c02f390 con 0x7f3670070c40 2023-11-24T08:19:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.796+0000 7f36667fc640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 0x7f365406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.796+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f366c0aa580 con 0x7f3670070c40 2023-11-24T08:19:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.796+0000 7f3675636640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 0x7f365406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.797+0000 7f3675636640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 0x7f365406ed70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f36700b0b50 tx=0x7f3660008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.800+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f366c077250 con 0x7f3670070c40 2023-11-24T08:19:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.992+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3670067470 con 0x7f3670070c40 2023-11-24T08:19:26.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.995+0000 7f36667fc640 1 -- 172.21.15.107:0/3859831305 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f366c034090 con 0x7f3670070c40 2023-11-24T08:19:26.997 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:26.997 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:23.570363+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:19:26.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 msgr2=0x7f365406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 0x7f365406ed70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f36700b0b50 tx=0x7f3660008040 comp rx=0 tx=0).stop 2023-11-24T08:19:27.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 msgr2=0x7f367014b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c004730 comp rx=0 tx=0).stop 2023-11-24T08:19:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 shutdown_connections 2023-11-24T08:19:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f365406c8b0 0x7f365406ed70 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f367015b320 0x7f367006b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f367015af40 0x7f36700b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 --2- 172.21.15.107:0/3859831305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670070c40 0x7f367014b2e0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.997+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 >> 172.21.15.107:0/3859831305 conn(0x7f367006e7f0 msgr2=0x7f36700b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.998+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 shutdown_connections 2023-11-24T08:19:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:26.998+0000 7f367809b640 1 -- 172.21.15.107:0/3859831305 wait complete. 2023-11-24T08:19:27.020 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:19:27.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:19:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.235+0000 7f3c09af1640 1 Processor -- start 2023-11-24T08:19:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.235+0000 7f3c09af1640 1 -- start start 2023-11-24T08:19:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c09af1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3c0415c410 0x7f3c0415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c09af1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0415bf80 con 0x7f3c04144b30 2023-11-24T08:19:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c09af1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040b1560 con 0x7f3c040b1de0 2023-11-24T08:19:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c09af1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c040b1280 con 0x7f3c0415c410 2023-11-24T08:19:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c08aef640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3c04144b30 0x7f3c040b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43674/0 (socket says 172.21.15.107:43674) 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c08aef640 1 -- 172.21.15.107:0/980969469 learned_addr learned my addr 172.21.15.107:0/980969469 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c03fff640 1 --2- 172.21.15.107:0/980969469 >> v2:172.21.15.107:3300/0 conn(0x7f3c0415c410 0x7f3c0415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.237+0000 7f3c037fe640 1 -- 172.21.15.107:0/980969469 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803013166 0 0) 0x7f3c0415bf80 con 0x7f3c04144b30 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.238+0000 7f3c037fe640 1 -- 172.21.15.107:0/980969469 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf0003520 con 0x7f3c04144b30 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.238+0000 7f3c03fff640 1 -- 172.21.15.107:0/980969469 >> v1:172.21.15.120:6789/0 conn(0x7f3c040b1de0 legacy=0x7f3c040b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.238+0000 7f3c03fff640 1 -- 172.21.15.107:0/980969469 >> v1:172.21.15.107:6789/0 conn(0x7f3c04144b30 legacy=0x7f3c040b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.238+0000 7f3c03fff640 1 -- 172.21.15.107:0/980969469 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0415b930 con 0x7f3c0415c410 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.239+0000 7f3c03fff640 1 --2- 172.21.15.107:0/980969469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c0415c410 0x7f3c0415f920 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6c18b453e492f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.239+0000 7f3c037fe640 1 -- 172.21.15.107:0/980969469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec031830 con 0x7f3c0415c410 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.239+0000 7f3c037fe640 1 -- 172.21.15.107:0/980969469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec031990 con 0x7f3c0415c410 2023-11-24T08:19:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.239+0000 7f3c037fe640 1 -- 172.21.15.107:0/980969469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec031d20 con 0x7f3c0415c410 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 -- 172.21.15.107:0/980969469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c0415c410 msgr2=0x7f3c0415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 --2- 172.21.15.107:0/980969469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c0415c410 0x7f3c0415f920 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 -- 172.21.15.107:0/980969469 shutdown_connections 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 --2- 172.21.15.107:0/980969469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c0415c410 0x7f3c0415f920 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 -- 172.21.15.107:0/980969469 >> 172.21.15.107:0/980969469 conn(0x7f3c040cb830 msgr2=0x7f3c040cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 -- 172.21.15.107:0/980969469 shutdown_connections 2023-11-24T08:19:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.240+0000 7f3c09af1640 1 -- 172.21.15.107:0/980969469 wait complete. 2023-11-24T08:19:27.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.242+0000 7f3c09af1640 1 Processor -- start 2023-11-24T08:19:27.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.242+0000 7f3c09af1640 1 -- start start 2023-11-24T08:19:27.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c04144b30 0x7f3c0415d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 0x7f3c0412f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041f4240 con 0x7f3c040b1de0 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c04133820 con 0x7f3c0415c410 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c09af1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041300c0 con 0x7f3c04144b30 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c08aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c08aef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48692/0 (socket says 172.21.15.107:48692) 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c08aef640 1 -- 172.21.15.107:0/2429195788 learned_addr learned my addr 172.21.15.107:0/2429195788 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.243+0000 7f3c092f0640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 0x7f3c0412f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 -- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c04144b30 msgr2=0x7f3c0415d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c04144b30 0x7f3c0415d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 -- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 msgr2=0x7f3c0412f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 0x7f3c0412f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 -- 172.21.15.107:0/2429195788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec009400 con 0x7f3c040b1de0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c092f0640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 0x7f3c0412f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c08aef640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400cbc0 tx=0x7f3bf400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400ad40 con 0x7f3c040b1de0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4012400 con 0x7f3c040b1de0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400aea0 con 0x7f3c040b1de0 2023-11-24T08:19:27.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c041303a0 con 0x7f3c040b1de0 2023-11-24T08:19:27.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.244+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c041f47e0 con 0x7f3c040b1de0 2023-11-24T08:19:27.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.246+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0404e840 con 0x7f3c040b1de0 2023-11-24T08:19:27.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.246+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f3bf4007070 con 0x7f3c040b1de0 2023-11-24T08:19:27.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.247+0000 7f3c017fa640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 0x7f3be406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.247+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bf4088510 con 0x7f3c040b1de0 2023-11-24T08:19:27.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.248+0000 7f3c03fff640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 0x7f3be406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.248+0000 7f3c03fff640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 0x7f3be406ed70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3bec036000 tx=0x7f3bec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.252+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bf40550c0 con 0x7f3c040b1de0 2023-11-24T08:19:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.437+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c0406b510 con 0x7f3be406c8b0 2023-11-24T08:19:27.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.449+0000 7f3c017fa640 1 -- 172.21.15.107:0/2429195788 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287509 (secure 0 0 0) 0x7f3c0406b510 con 0x7f3be406c8b0 2023-11-24T08:19:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:27.456 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 msgr2=0x7f3be406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 0x7f3be406ed70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3bec036000 tx=0x7f3bec030000 comp rx=0 tx=0).stop 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 msgr2=0x7f3c041f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400cbc0 tx=0x7f3bf400a430 comp rx=0 tx=0).stop 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 shutdown_connections 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3be406c8b0 0x7f3be406ed70 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c0415c410 0x7f3c0412f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c04144b30 0x7f3c0415d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 --2- 172.21.15.107:0/2429195788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040b1de0 0x7f3c041f4d00 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 >> 172.21.15.107:0/2429195788 conn(0x7f3c040cb830 msgr2=0x7f3c0415c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 shutdown_connections 2023-11-24T08:19:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.456+0000 7f3c09af1640 1 -- 172.21.15.107:0/2429195788 wait complete. 2023-11-24T08:19:27.479 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2023-11-24T08:19:25.707309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309384,"kb_used_data":18312,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754665336,"statfs":{"total":773094113280,"available":772777304064,"internally_reserved":0,"allocated":18751488,"data_stored":8894132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":297913918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.188899"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.595245+0000","last_change":"2023-11-24T08:19:22.588643+0000","last_active":"2023-11-24T08:19:22.595245+0000","last_peered":"2023-11-24T08:19:22.595245+0000","last_clean":"2023-11-24T08:19:22.595245+0000","last_became_active":"2023-11-24T08:19:22.588118+0000","last_became_peered":"2023-11-24T08:19:22.588118+0000","last_unstale":"2023-11-24T08:19:22.595245+0000","last_undegraded":"2023-11-24T08:19:22.595245+0000","last_fullsized":"2023-11-24T08:19:22.595245+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:23:17.425447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586717+0000","last_change":"2023-11-24T08:19:22.580198+0000","last_active":"2023-11-24T08:19:22.586717+0000","last_peered":"2023-11-24T08:19:22.586717+0000","last_clean":"2023-11-24T08:19:22.586717+0000","last_became_active":"2023-11-24T08:19:22.579897+0000","last_became_peered":"2023-11-24T08:19:22.579897+0000","last_unstale":"2023-11-24T08:19:22.586717+0000","last_undegraded":"2023-11-24T08:19:22.586717+0000","last_fullsized":"2023-11-24T08:19:22.586717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:34.933597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591052+0000","last_change":"2023-11-24T08:19:22.587251+0000","last_active":"2023-11-24T08:19:22.591052+0000","last_peered":"2023-11-24T08:19:22.591052+0000","last_clean":"2023-11-24T08:19:22.591052+0000","last_became_active":"2023-11-24T08:19:22.586943+0000","last_became_peered":"2023-11-24T08:19:22.586943+0000","last_unstale":"2023-11-24T08:19:22.591052+0000","last_undegraded":"2023-11-24T08:19:22.591052+0000","last_fullsized":"2023-11-24T08:19:22.591052+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:27.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586841+0000","last_change":"2023-11-24T08:19:22.580053+0000","last_active":"2023-11-24T08:19:22.586841+0000","last_peered":"2023-11-24T08:19:22.586841+0000","last_clean":"2023-11-24T08:19:22.586841+0000","last_became_active":"2023-11-24T08:19:22.579739+0000","last_became_peered":"2023-11-24T08:19:22.579739+0000","last_unstale":"2023-11-24T08:19:22.586841+0000","last_undegraded":"2023-11-24T08:19:22.586841+0000","last_fullsized":"2023-11-24T08:19:22.586841+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:52.970487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.601987+0000","last_change":"2023-11-24T08:19:22.592309+0000","last_active":"2023-11-24T08:19:22.601987+0000","last_peered":"2023-11-24T08:19:22.601987+0000","last_clean":"2023-11-24T08:19:22.601987+0000","last_became_active":"2023-11-24T08:19:22.592024+0000","last_became_peered":"2023-11-24T08:19:22.592024+0000","last_unstale":"2023-11-24T08:19:22.601987+0000","last_undegraded":"2023-11-24T08:19:22.601987+0000","last_fullsized":"2023-11-24T08:19:22.601987+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:13:43.146970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591974+0000","last_change":"2023-11-24T08:19:22.586770+0000","last_active":"2023-11-24T08:19:22.591974+0000","last_peered":"2023-11-24T08:19:22.591974+0000","last_clean":"2023-11-24T08:19:22.591974+0000","last_became_active":"2023-11-24T08:19:22.585338+0000","last_became_peered":"2023-11-24T08:19:22.585338+0000","last_unstale":"2023-11-24T08:19:22.591974+0000","last_undegraded":"2023-11-24T08:19:22.591974+0000","last_fullsized":"2023-11-24T08:19:22.591974+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590027+0000","last_change":"2023-11-24T08:19:22.592042+0000","last_active":"2023-11-24T08:19:23.590027+0000","last_peered":"2023-11-24T08:19:23.590027+0000","last_clean":"2023-11-24T08:19:23.590027+0000","last_became_active":"2023-11-24T08:19:22.591732+0000","last_became_peered":"2023-11-24T08:19:22.591732+0000","last_unstale":"2023-11-24T08:19:23.590027+0000","last_undegraded":"2023-11-24T08:19:23.590027+0000","last_fullsized":"2023-11-24T08:19:23.590027+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.579673+0000","last_change":"2023-11-24T08:19:22.578021+0000","last_active":"2023-11-24T08:19:22.579673+0000","last_peered":"2023-11-24T08:19:22.579673+0000","last_clean":"2023-11-24T08:19:22.579673+0000","last_became_active":"2023-11-24T08:19:22.577653+0000","last_became_peered":"2023-11-24T08:19:22.577653+0000","last_unstale":"2023-11-24T08:19:22.579673+0000","last_undegraded":"2023-11-24T08:19:22.579673+0000","last_fullsized":"2023-11-24T08:19:22.579673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:04:32.767118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588509+0000","last_change":"2023-11-24T08:19:22.581067+0000","last_active":"2023-11-24T08:19:23.588509+0000","last_peered":"2023-11-24T08:19:23.588509+0000","last_clean":"2023-11-24T08:19:23.588509+0000","last_became_active":"2023-11-24T08:19:22.580040+0000","last_became_peered":"2023-11-24T08:19:22.580040+0000","last_unstale":"2023-11-24T08:19:23.588509+0000","last_undegraded":"2023-11-24T08:19:23.588509+0000","last_fullsized":"2023-11-24T08:19:23.588509+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.763256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602451+0000","last_change":"2023-11-24T08:19:22.597984+0000","last_active":"2023-11-24T08:19:22.602451+0000","last_peered":"2023-11-24T08:19:22.602451+0000","last_clean":"2023-11-24T08:19:22.602451+0000","last_became_active":"2023-11-24T08:19:22.597712+0000","last_became_peered":"2023-11-24T08:19:22.597712+0000","last_unstale":"2023-11-24T08:19:22.602451+0000","last_undegraded":"2023-11-24T08:19:22.602451+0000","last_fullsized":"2023-11-24T08:19:22.602451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:02.003634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590679+0000","last_change":"2023-11-24T08:19:22.582432+0000","last_active":"2023-11-24T08:19:23.590679+0000","last_peered":"2023-11-24T08:19:23.590679+0000","last_clean":"2023-11-24T08:19:23.590679+0000","last_became_active":"2023-11-24T08:19:22.582138+0000","last_became_peered":"2023-11-24T08:19:22.582138+0000","last_unstale":"2023-11-24T08:19:23.590679+0000","last_undegraded":"2023-11-24T08:19:23.590679+0000","last_fullsized":"2023-11-24T08:19:23.590679+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.763256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588618+0000","last_change":"2023-11-24T08:19:22.583314+0000","last_active":"2023-11-24T08:19:22.588618+0000","last_peered":"2023-11-24T08:19:22.588618+0000","last_clean":"2023-11-24T08:19:22.588618+0000","last_became_active":"2023-11-24T08:19:22.582659+0000","last_became_peered":"2023-11-24T08:19:22.582659+0000","last_unstale":"2023-11-24T08:19:22.588618+0000","last_undegraded":"2023-11-24T08:19:22.588618+0000","last_fullsized":"2023-11-24T08:19:22.588618+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:54.466137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.582488+0000","last_change":"2023-11-24T08:19:22.578217+0000","last_active":"2023-11-24T08:19:22.582488+0000","last_peered":"2023-11-24T08:19:22.582488+0000","last_clean":"2023-11-24T08:19:22.582488+0000","last_became_active":"2023-11-24T08:19:22.577724+0000","last_became_peered":"2023-11-24T08:19:22.577724+0000","last_unstale":"2023-11-24T08:19:22.582488+0000","last_undegraded":"2023-11-24T08:19:22.582488+0000","last_fullsized":"2023-11-24T08:19:22.582488+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:27.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586925+0000","last_change":"2023-11-24T08:19:22.586219+0000","last_active":"2023-11-24T08:19:23.586925+0000","last_peered":"2023-11-24T08:19:23.586925+0000","last_clean":"2023-11-24T08:19:23.586925+0000","last_became_active":"2023-11-24T08:19:22.585896+0000","last_became_peered":"2023-11-24T08:19:22.585896+0000","last_unstale":"2023-11-24T08:19:23.586925+0000","last_undegraded":"2023-11-24T08:19:23.586925+0000","last_fullsized":"2023-11-24T08:19:23.586925+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:49.298452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.587207+0000","last_change":"2023-11-24T08:19:22.581015+0000","last_active":"2023-11-24T08:19:22.587207+0000","last_peered":"2023-11-24T08:19:22.587207+0000","last_clean":"2023-11-24T08:19:22.587207+0000","last_became_active":"2023-11-24T08:19:22.580379+0000","last_became_peered":"2023-11-24T08:19:22.580379+0000","last_unstale":"2023-11-24T08:19:22.587207+0000","last_undegraded":"2023-11-24T08:19:22.587207+0000","last_fullsized":"2023-11-24T08:19:22.587207+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:49.298452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.576574+0000","last_change":"2023-11-24T08:19:22.573391+0000","last_active":"2023-11-24T08:19:22.576574+0000","last_peered":"2023-11-24T08:19:22.576574+0000","last_clean":"2023-11-24T08:19:22.576574+0000","last_became_active":"2023-11-24T08:19:22.572874+0000","last_became_peered":"2023-11-24T08:19:22.572874+0000","last_unstale":"2023-11-24T08:19:22.576574+0000","last_undegraded":"2023-11-24T08:19:22.576574+0000","last_fullsized":"2023-11-24T08:19:22.576574+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:54.912422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591445+0000","last_change":"2023-11-24T08:19:22.587690+0000","last_active":"2023-11-24T08:19:22.591445+0000","last_peered":"2023-11-24T08:19:22.591445+0000","last_clean":"2023-11-24T08:19:22.591445+0000","last_became_active":"2023-11-24T08:19:22.587362+0000","last_became_peered":"2023-11-24T08:19:22.587362+0000","last_unstale":"2023-11-24T08:19:22.591445+0000","last_undegraded":"2023-11-24T08:19:22.591445+0000","last_fullsized":"2023-11-24T08:19:22.591445+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:03.587373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584517+0000","last_change":"2023-11-24T08:19:22.579492+0000","last_active":"2023-11-24T08:19:22.584517+0000","last_peered":"2023-11-24T08:19:22.584517+0000","last_clean":"2023-11-24T08:19:22.584517+0000","last_became_active":"2023-11-24T08:19:22.579152+0000","last_became_peered":"2023-11-24T08:19:22.579152+0000","last_unstale":"2023-11-24T08:19:22.584517+0000","last_undegraded":"2023-11-24T08:19:22.584517+0000","last_fullsized":"2023-11-24T08:19:22.584517+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:33.628765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600210+0000","last_change":"2023-11-24T08:19:22.594982+0000","last_active":"2023-11-24T08:19:22.600210+0000","last_peered":"2023-11-24T08:19:22.600210+0000","last_clean":"2023-11-24T08:19:22.600210+0000","last_became_active":"2023-11-24T08:19:22.594362+0000","last_became_peered":"2023-11-24T08:19:22.594362+0000","last_unstale":"2023-11-24T08:19:22.600210+0000","last_undegraded":"2023-11-24T08:19:22.600210+0000","last_fullsized":"2023-11-24T08:19:22.600210+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:03.374509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.601175+0000","last_change":"2023-11-24T08:19:22.589325+0000","last_active":"2023-11-24T08:19:22.601175+0000","last_peered":"2023-11-24T08:19:22.601175+0000","last_clean":"2023-11-24T08:19:22.601175+0000","last_became_active":"2023-11-24T08:19:22.588243+0000","last_became_peered":"2023-11-24T08:19:22.588243+0000","last_unstale":"2023-11-24T08:19:22.601175+0000","last_undegraded":"2023-11-24T08:19:22.601175+0000","last_fullsized":"2023-11-24T08:19:22.601175+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:19.415043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599410+0000","last_change":"2023-11-24T08:19:22.589823+0000","last_active":"2023-11-24T08:19:22.599410+0000","last_peered":"2023-11-24T08:19:22.599410+0000","last_clean":"2023-11-24T08:19:22.599410+0000","last_became_active":"2023-11-24T08:19:22.589536+0000","last_became_peered":"2023-11-24T08:19:22.589536+0000","last_unstale":"2023-11-24T08:19:22.599410+0000","last_undegraded":"2023-11-24T08:19:22.599410+0000","last_fullsized":"2023-11-24T08:19:22.599410+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:12:03.848491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.594324+0000","last_change":"2023-11-24T08:19:22.586931+0000","last_active":"2023-11-24T08:19:22.594324+0000","last_peered":"2023-11-24T08:19:22.594324+0000","last_clean":"2023-11-24T08:19:22.594324+0000","last_became_active":"2023-11-24T08:19:22.586390+0000","last_became_peered":"2023-11-24T08:19:22.586390+0000","last_unstale":"2023-11-24T08:19:22.594324+0000","last_undegraded":"2023-11-24T08:19:22.594324+0000","last_fullsized":"2023-11-24T08:19:22.594324+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:56.999101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599320+0000","last_change":"2023-11-24T08:19:22.591765+0000","last_active":"2023-11-24T08:19:22.599320+0000","last_peered":"2023-11-24T08:19:22.599320+0000","last_clean":"2023-11-24T08:19:22.599320+0000","last_became_active":"2023-11-24T08:19:22.591459+0000","last_became_peered":"2023-11-24T08:19:22.591459+0000","last_unstale":"2023-11-24T08:19:22.599320+0000","last_undegraded":"2023-11-24T08:19:22.599320+0000","last_fullsized":"2023-11-24T08:19:22.599320+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:57.500978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.587383+0000","last_change":"2023-11-24T08:19:22.581944+0000","last_active":"2023-11-24T08:19:22.587383+0000","last_peered":"2023-11-24T08:19:22.587383+0000","last_clean":"2023-11-24T08:19:22.587383+0000","last_became_active":"2023-11-24T08:19:22.581661+0000","last_became_peered":"2023-11-24T08:19:22.581661+0000","last_unstale":"2023-11-24T08:19:22.587383+0000","last_undegraded":"2023-11-24T08:19:22.587383+0000","last_fullsized":"2023-11-24T08:19:22.587383+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:05.125474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.595458+0000","last_change":"2023-11-24T08:19:22.592067+0000","last_active":"2023-11-24T08:19:22.595458+0000","last_peered":"2023-11-24T08:19:22.595458+0000","last_clean":"2023-11-24T08:19:22.595458+0000","last_became_active":"2023-11-24T08:19:22.591756+0000","last_became_peered":"2023-11-24T08:19:22.591756+0000","last_unstale":"2023-11-24T08:19:22.595458+0000","last_undegraded":"2023-11-24T08:19:22.595458+0000","last_fullsized":"2023-11-24T08:19:22.595458+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:15:24.287481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590967+0000","last_change":"2023-11-24T08:19:22.591985+0000","last_active":"2023-11-24T08:19:23.590967+0000","last_peered":"2023-11-24T08:19:23.590967+0000","last_clean":"2023-11-24T08:19:23.590967+0000","last_became_active":"2023-11-24T08:19:22.591697+0000","last_became_peered":"2023-11-24T08:19:22.591697+0000","last_unstale":"2023-11-24T08:19:23.590967+0000","last_undegraded":"2023-11-24T08:19:23.590967+0000","last_fullsized":"2023-11-24T08:19:23.590967+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:51:38.735044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.596280+0000","last_change":"2023-11-24T08:19:22.588307+0000","last_active":"2023-11-24T08:19:22.596280+0000","last_peered":"2023-11-24T08:19:22.596280+0000","last_clean":"2023-11-24T08:19:22.596280+0000","last_became_active":"2023-11-24T08:19:22.586440+0000","last_became_peered":"2023-11-24T08:19:22.586440+0000","last_unstale":"2023-11-24T08:19:22.596280+0000","last_undegraded":"2023-11-24T08:19:22.596280+0000","last_fullsized":"2023-11-24T08:19:22.596280+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:50.869823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604696+0000","last_change":"2023-11-24T08:19:22.597233+0000","last_active":"2023-11-24T08:19:22.604696+0000","last_peered":"2023-11-24T08:19:22.604696+0000","last_clean":"2023-11-24T08:19:22.604696+0000","last_became_active":"2023-11-24T08:19:22.596960+0000","last_became_peered":"2023-11-24T08:19:22.596960+0000","last_unstale":"2023-11-24T08:19:22.604696+0000","last_undegraded":"2023-11-24T08:19:22.604696+0000","last_fullsized":"2023-11-24T08:19:22.604696+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:55.587615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.589882+0000","last_change":"2023-11-24T08:19:22.586967+0000","last_active":"2023-11-24T08:19:22.589882+0000","last_peered":"2023-11-24T08:19:22.589882+0000","last_clean":"2023-11-24T08:19:22.589882+0000","last_became_active":"2023-11-24T08:19:22.586710+0000","last_became_peered":"2023-11-24T08:19:22.586710+0000","last_unstale":"2023-11-24T08:19:22.589882+0000","last_undegraded":"2023-11-24T08:19:22.589882+0000","last_fullsized":"2023-11-24T08:19:22.589882+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:16:02.650805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587217+0000","last_change":"2023-11-24T08:19:22.578847+0000","last_active":"2023-11-24T08:19:23.587217+0000","last_peered":"2023-11-24T08:19:23.587217+0000","last_clean":"2023-11-24T08:19:23.587217+0000","last_became_active":"2023-11-24T08:19:22.578532+0000","last_became_peered":"2023-11-24T08:19:22.578532+0000","last_unstale":"2023-11-24T08:19:23.587217+0000","last_undegraded":"2023-11-24T08:19:23.587217+0000","last_fullsized":"2023-11-24T08:19:23.587217+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591671+0000","last_change":"2023-11-24T08:19:22.581063+0000","last_active":"2023-11-24T08:19:22.591671+0000","last_peered":"2023-11-24T08:19:22.591671+0000","last_clean":"2023-11-24T08:19:22.591671+0000","last_became_active":"2023-11-24T08:19:22.580698+0000","last_became_peered":"2023-11-24T08:19:22.580698+0000","last_unstale":"2023-11-24T08:19:22.591671+0000","last_undegraded":"2023-11-24T08:19:22.591671+0000","last_fullsized":"2023-11-24T08:19:22.591671+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:52.600814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586129+0000","last_change":"2023-11-24T08:19:22.582313+0000","last_active":"2023-11-24T08:19:22.586129+0000","last_peered":"2023-11-24T08:19:22.586129+0000","last_clean":"2023-11-24T08:19:22.586129+0000","last_became_active":"2023-11-24T08:19:22.581987+0000","last_became_peered":"2023-11-24T08:19:22.581987+0000","last_unstale":"2023-11-24T08:19:22.586129+0000","last_undegraded":"2023-11-24T08:19:22.586129+0000","last_fullsized":"2023-11-24T08:19:22.586129+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:34.709747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.585564+0000","last_change":"2023-11-24T08:19:22.581629+0000","last_active":"2023-11-24T08:19:22.585564+0000","last_peered":"2023-11-24T08:19:22.585564+0000","last_clean":"2023-11-24T08:19:22.585564+0000","last_became_active":"2023-11-24T08:19:22.580906+0000","last_became_peered":"2023-11-24T08:19:22.580906+0000","last_unstale":"2023-11-24T08:19:22.585564+0000","last_undegraded":"2023-11-24T08:19:22.585564+0000","last_fullsized":"2023-11-24T08:19:22.585564+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:06.713002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"174'4","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604836+0000","last_change":"2023-11-24T08:19:21.551814+0000","last_active":"2023-11-24T08:19:22.604836+0000","last_peered":"2023-11-24T08:19:22.604836+0000","last_clean":"2023-11-24T08:19:22.604836+0000","last_became_active":"2023-11-24T08:19:21.551310+0000","last_became_peered":"2023-11-24T08:19:21.551310+0000","last_unstale":"2023-11-24T08:19:22.604836+0000","last_undegraded":"2023-11-24T08:19:22.604836+0000","last_fullsized":"2023-11-24T08:19:22.604836+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:53:33.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602360+0000","last_change":"2023-11-24T08:19:22.597227+0000","last_active":"2023-11-24T08:19:22.602360+0000","last_peered":"2023-11-24T08:19:22.602360+0000","last_clean":"2023-11-24T08:19:22.602360+0000","last_became_active":"2023-11-24T08:19:22.596614+0000","last_became_peered":"2023-11-24T08:19:22.596614+0000","last_unstale":"2023-11-24T08:19:22.602360+0000","last_undegraded":"2023-11-24T08:19:22.602360+0000","last_fullsized":"2023-11-24T08:19:22.602360+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:16:55.461611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588421+0000","last_change":"2023-11-24T08:19:21.552633+0000","last_active":"2023-11-24T08:19:23.588421+0000","last_peered":"2023-11-24T08:19:23.588421+0000","last_clean":"2023-11-24T08:19:23.588421+0000","last_became_active":"2023-11-24T08:19:21.551810+0000","last_became_peered":"2023-11-24T08:19:21.551810+0000","last_unstale":"2023-11-24T08:19:23.588421+0000","last_undegraded":"2023-11-24T08:19:23.588421+0000","last_fullsized":"2023-11-24T08:19:23.588421+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:49.859364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.591113+0000","last_change":"2023-11-24T08:19:22.585074+0000","last_active":"2023-11-24T08:19:23.591113+0000","last_peered":"2023-11-24T08:19:23.591113+0000","last_clean":"2023-11-24T08:19:23.591113+0000","last_became_active":"2023-11-24T08:19:22.584764+0000","last_became_peered":"2023-11-24T08:19:22.584764+0000","last_unstale":"2023-11-24T08:19:23.591113+0000","last_undegraded":"2023-11-24T08:19:23.591113+0000","last_fullsized":"2023-11-24T08:19:23.591113+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:34.659045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.608099+0000","last_change":"2023-11-24T08:19:21.559329+0000","last_active":"2023-11-24T08:19:22.608099+0000","last_peered":"2023-11-24T08:19:22.608099+0000","last_clean":"2023-11-24T08:19:22.608099+0000","last_became_active":"2023-11-24T08:19:21.558165+0000","last_became_peered":"2023-11-24T08:19:21.558165+0000","last_unstale":"2023-11-24T08:19:22.608099+0000","last_undegraded":"2023-11-24T08:19:22.608099+0000","last_fullsized":"2023-11-24T08:19:22.608099+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:33.923138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588267+0000","last_change":"2023-11-24T08:19:22.571466+0000","last_active":"2023-11-24T08:19:23.588267+0000","last_peered":"2023-11-24T08:19:23.588267+0000","last_clean":"2023-11-24T08:19:23.588267+0000","last_became_active":"2023-11-24T08:19:22.570217+0000","last_became_peered":"2023-11-24T08:19:22.570217+0000","last_unstale":"2023-11-24T08:19:23.588267+0000","last_undegraded":"2023-11-24T08:19:23.588267+0000","last_fullsized":"2023-11-24T08:19:23.588267+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607786+0000","last_change":"2023-11-24T08:19:21.559519+0000","last_active":"2023-11-24T08:19:22.607786+0000","last_peered":"2023-11-24T08:19:22.607786+0000","last_clean":"2023-11-24T08:19:22.607786+0000","last_became_active":"2023-11-24T08:19:21.558830+0000","last_became_peered":"2023-11-24T08:19:21.558830+0000","last_unstale":"2023-11-24T08:19:22.607786+0000","last_undegraded":"2023-11-24T08:19:22.607786+0000","last_fullsized":"2023-11-24T08:19:22.607786+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:49.859364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.594303+0000","last_change":"2023-11-24T08:19:22.586739+0000","last_active":"2023-11-24T08:19:22.594303+0000","last_peered":"2023-11-24T08:19:22.594303+0000","last_clean":"2023-11-24T08:19:22.594303+0000","last_became_active":"2023-11-24T08:19:22.586009+0000","last_became_peered":"2023-11-24T08:19:22.586009+0000","last_unstale":"2023-11-24T08:19:22.594303+0000","last_undegraded":"2023-11-24T08:19:22.594303+0000","last_fullsized":"2023-11-24T08:19:22.594303+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:21:59.002683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.605850+0000","last_change":"2023-11-24T08:19:21.551868+0000","last_active":"2023-11-24T08:19:22.605850+0000","last_peered":"2023-11-24T08:19:22.605850+0000","last_clean":"2023-11-24T08:19:22.605850+0000","last_became_active":"2023-11-24T08:19:21.551442+0000","last_became_peered":"2023-11-24T08:19:21.551442+0000","last_unstale":"2023-11-24T08:19:22.605850+0000","last_undegraded":"2023-11-24T08:19:22.605850+0000","last_fullsized":"2023-11-24T08:19:22.605850+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:51:49.262584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.575866+0000","last_change":"2023-11-24T08:19:22.572998+0000","last_active":"2023-11-24T08:19:22.575866+0000","last_peered":"2023-11-24T08:19:22.575866+0000","last_clean":"2023-11-24T08:19:22.575866+0000","last_became_active":"2023-11-24T08:19:22.572702+0000","last_became_peered":"2023-11-24T08:19:22.572702+0000","last_unstale":"2023-11-24T08:19:22.575866+0000","last_undegraded":"2023-11-24T08:19:22.575866+0000","last_fullsized":"2023-11-24T08:19:22.575866+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:38.041778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602762+0000","last_change":"2023-11-24T08:19:21.556229+0000","last_active":"2023-11-24T08:19:22.602762+0000","last_peered":"2023-11-24T08:19:22.602762+0000","last_clean":"2023-11-24T08:19:22.602762+0000","last_became_active":"2023-11-24T08:19:21.555580+0000","last_became_peered":"2023-11-24T08:19:21.555580+0000","last_unstale":"2023-11-24T08:19:22.602762+0000","last_undegraded":"2023-11-24T08:19:22.602762+0000","last_fullsized":"2023-11-24T08:19:22.602762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:52.380299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590191+0000","last_change":"2023-11-24T08:19:22.587093+0000","last_active":"2023-11-24T08:19:23.590191+0000","last_peered":"2023-11-24T08:19:23.590191+0000","last_clean":"2023-11-24T08:19:23.590191+0000","last_became_active":"2023-11-24T08:19:22.586307+0000","last_became_peered":"2023-11-24T08:19:22.586307+0000","last_unstale":"2023-11-24T08:19:23.590191+0000","last_undegraded":"2023-11-24T08:19:23.590191+0000","last_fullsized":"2023-11-24T08:19:23.590191+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:17.005243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.570086+0000","last_change":"2023-11-24T08:19:21.561585+0000","last_active":"2023-11-24T08:19:22.570086+0000","last_peered":"2023-11-24T08:19:22.570086+0000","last_clean":"2023-11-24T08:19:22.570086+0000","last_became_active":"2023-11-24T08:19:21.561358+0000","last_became_peered":"2023-11-24T08:19:21.561358+0000","last_unstale":"2023-11-24T08:19:22.570086+0000","last_undegraded":"2023-11-24T08:19:22.570086+0000","last_fullsized":"2023-11-24T08:19:22.570086+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:46.397173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.590793+0000","last_change":"2023-11-24T08:19:22.581938+0000","last_active":"2023-11-24T08:19:22.590793+0000","last_peered":"2023-11-24T08:19:22.590793+0000","last_clean":"2023-11-24T08:19:22.590793+0000","last_became_active":"2023-11-24T08:19:22.580866+0000","last_became_peered":"2023-11-24T08:19:22.580866+0000","last_unstale":"2023-11-24T08:19:22.590793+0000","last_undegraded":"2023-11-24T08:19:22.590793+0000","last_fullsized":"2023-11-24T08:19:22.590793+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:51:38.735044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590986+0000","last_change":"2023-11-24T08:19:21.555108+0000","last_active":"2023-11-24T08:19:23.590986+0000","last_peered":"2023-11-24T08:19:23.590986+0000","last_clean":"2023-11-24T08:19:23.590986+0000","last_became_active":"2023-11-24T08:19:21.553150+0000","last_became_peered":"2023-11-24T08:19:21.553150+0000","last_unstale":"2023-11-24T08:19:23.590986+0000","last_undegraded":"2023-11-24T08:19:23.590986+0000","last_fullsized":"2023-11-24T08:19:23.590986+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:26.656730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599277+0000","last_change":"2023-11-24T08:19:22.597246+0000","last_active":"2023-11-24T08:19:22.599277+0000","last_peered":"2023-11-24T08:19:22.599277+0000","last_clean":"2023-11-24T08:19:22.599277+0000","last_became_active":"2023-11-24T08:19:22.596938+0000","last_became_peered":"2023-11-24T08:19:22.596938+0000","last_unstale":"2023-11-24T08:19:22.599277+0000","last_undegraded":"2023-11-24T08:19:22.599277+0000","last_fullsized":"2023-11-24T08:19:22.599277+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:46.486016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.575241+0000","last_change":"2023-11-24T08:19:21.561530+0000","last_active":"2023-11-24T08:19:22.575241+0000","last_peered":"2023-11-24T08:19:22.575241+0000","last_clean":"2023-11-24T08:19:22.575241+0000","last_became_active":"2023-11-24T08:19:21.560527+0000","last_became_peered":"2023-11-24T08:19:21.560527+0000","last_unstale":"2023-11-24T08:19:22.575241+0000","last_undegraded":"2023-11-24T08:19:22.575241+0000","last_fullsized":"2023-11-24T08:19:22.575241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:00:15.010190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590686+0000","last_change":"2023-11-24T08:19:22.580321+0000","last_active":"2023-11-24T08:19:23.590686+0000","last_peered":"2023-11-24T08:19:23.590686+0000","last_clean":"2023-11-24T08:19:23.590686+0000","last_became_active":"2023-11-24T08:19:22.580033+0000","last_became_peered":"2023-11-24T08:19:22.580033+0000","last_unstale":"2023-11-24T08:19:23.590686+0000","last_undegraded":"2023-11-24T08:19:23.590686+0000","last_fullsized":"2023-11-24T08:19:23.590686+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:29.525302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"174'3","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607283+0000","last_change":"2023-11-24T08:19:21.552871+0000","last_active":"2023-11-24T08:19:22.607283+0000","last_peered":"2023-11-24T08:19:22.607283+0000","last_clean":"2023-11-24T08:19:22.607283+0000","last_became_active":"2023-11-24T08:19:21.551875+0000","last_became_peered":"2023-11-24T08:19:21.551875+0000","last_unstale":"2023-11-24T08:19:22.607283+0000","last_undegraded":"2023-11-24T08:19:22.607283+0000","last_fullsized":"2023-11-24T08:19:22.607283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:27.136424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600230+0000","last_change":"2023-11-24T08:19:22.595477+0000","last_active":"2023-11-24T08:19:22.600230+0000","last_peered":"2023-11-24T08:19:22.600230+0000","last_clean":"2023-11-24T08:19:22.600230+0000","last_became_active":"2023-11-24T08:19:22.594719+0000","last_became_peered":"2023-11-24T08:19:22.594719+0000","last_unstale":"2023-11-24T08:19:22.600230+0000","last_undegraded":"2023-11-24T08:19:22.600230+0000","last_fullsized":"2023-11-24T08:19:22.600230+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:49.157351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"174'6","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.611931+0000","last_change":"2023-11-24T08:19:21.556481+0000","last_active":"2023-11-24T08:19:22.611931+0000","last_peered":"2023-11-24T08:19:22.611931+0000","last_clean":"2023-11-24T08:19:22.611931+0000","last_became_active":"2023-11-24T08:19:21.556146+0000","last_became_peered":"2023-11-24T08:19:21.556146+0000","last_unstale":"2023-11-24T08:19:22.611931+0000","last_undegraded":"2023-11-24T08:19:22.611931+0000","last_fullsized":"2023-11-24T08:19:22.611931+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:28.514843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599718+0000","last_change":"2023-11-24T08:19:22.591889+0000","last_active":"2023-11-24T08:19:22.599718+0000","last_peered":"2023-11-24T08:19:22.599718+0000","last_clean":"2023-11-24T08:19:22.599718+0000","last_became_active":"2023-11-24T08:19:22.591486+0000","last_became_peered":"2023-11-24T08:19:22.591486+0000","last_unstale":"2023-11-24T08:19:22.599718+0000","last_undegraded":"2023-11-24T08:19:22.599718+0000","last_fullsized":"2023-11-24T08:19:22.599718+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607593+0000","last_change":"2023-11-24T08:19:21.559312+0000","last_active":"2023-11-24T08:19:22.607593+0000","last_peered":"2023-11-24T08:19:22.607593+0000","last_clean":"2023-11-24T08:19:22.607593+0000","last_became_active":"2023-11-24T08:19:21.559033+0000","last_became_peered":"2023-11-24T08:19:21.559033+0000","last_unstale":"2023-11-24T08:19:22.607593+0000","last_undegraded":"2023-11-24T08:19:22.607593+0000","last_fullsized":"2023-11-24T08:19:22.607593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:04.115015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590586+0000","last_change":"2023-11-24T08:19:22.582930+0000","last_active":"2023-11-24T08:19:23.590586+0000","last_peered":"2023-11-24T08:19:23.590586+0000","last_clean":"2023-11-24T08:19:23.590586+0000","last_became_active":"2023-11-24T08:19:22.582593+0000","last_became_peered":"2023-11-24T08:19:22.582593+0000","last_unstale":"2023-11-24T08:19:23.590586+0000","last_undegraded":"2023-11-24T08:19:23.590586+0000","last_fullsized":"2023-11-24T08:19:23.590586+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.581475+0000","last_change":"2023-11-24T08:19:21.547030+0000","last_active":"2023-11-24T08:19:22.581475+0000","last_peered":"2023-11-24T08:19:22.581475+0000","last_clean":"2023-11-24T08:19:22.581475+0000","last_became_active":"2023-11-24T08:19:21.546748+0000","last_became_peered":"2023-11-24T08:19:21.546748+0000","last_unstale":"2023-11-24T08:19:22.581475+0000","last_undegraded":"2023-11-24T08:19:22.581475+0000","last_fullsized":"2023-11-24T08:19:22.581475+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:11.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586163+0000","last_change":"2023-11-24T08:19:22.581222+0000","last_active":"2023-11-24T08:19:23.586163+0000","last_peered":"2023-11-24T08:19:23.586163+0000","last_clean":"2023-11-24T08:19:23.586163+0000","last_became_active":"2023-11-24T08:19:22.580220+0000","last_became_peered":"2023-11-24T08:19:22.580220+0000","last_unstale":"2023-11-24T08:19:23.586163+0000","last_undegraded":"2023-11-24T08:19:23.586163+0000","last_fullsized":"2023-11-24T08:19:23.586163+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:34.933597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.561566+0000","last_change":"2023-11-24T08:19:21.560650+0000","last_active":"2023-11-24T08:19:22.561566+0000","last_peered":"2023-11-24T08:19:22.561566+0000","last_clean":"2023-11-24T08:19:22.561566+0000","last_became_active":"2023-11-24T08:19:21.560390+0000","last_became_peered":"2023-11-24T08:19:21.560390+0000","last_unstale":"2023-11-24T08:19:22.561566+0000","last_undegraded":"2023-11-24T08:19:22.561566+0000","last_fullsized":"2023-11-24T08:19:22.561566+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:27.334358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599077+0000","last_change":"2023-11-24T08:19:22.593347+0000","last_active":"2023-11-24T08:19:22.599077+0000","last_peered":"2023-11-24T08:19:22.599077+0000","last_clean":"2023-11-24T08:19:22.599077+0000","last_became_active":"2023-11-24T08:19:22.590877+0000","last_became_peered":"2023-11-24T08:19:22.590877+0000","last_unstale":"2023-11-24T08:19:22.599077+0000","last_undegraded":"2023-11-24T08:19:22.599077+0000","last_fullsized":"2023-11-24T08:19:22.599077+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:52.844484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"174'8","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607798+0000","last_change":"2023-11-24T08:19:21.557576+0000","last_active":"2023-11-24T08:19:22.607798+0000","last_peered":"2023-11-24T08:19:22.607798+0000","last_clean":"2023-11-24T08:19:22.607798+0000","last_became_active":"2023-11-24T08:19:21.557271+0000","last_became_peered":"2023-11-24T08:19:21.557271+0000","last_unstale":"2023-11-24T08:19:22.607798+0000","last_undegraded":"2023-11-24T08:19:22.607798+0000","last_fullsized":"2023-11-24T08:19:22.607798+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:52.380299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602431+0000","last_change":"2023-11-24T08:19:22.595705+0000","last_active":"2023-11-24T08:19:22.602431+0000","last_peered":"2023-11-24T08:19:22.602431+0000","last_clean":"2023-11-24T08:19:22.602431+0000","last_became_active":"2023-11-24T08:19:22.592584+0000","last_became_peered":"2023-11-24T08:19:22.592584+0000","last_unstale":"2023-11-24T08:19:22.602431+0000","last_undegraded":"2023-11-24T08:19:22.602431+0000","last_fullsized":"2023-11-24T08:19:22.602431+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:10.117362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604734+0000","last_change":"2023-11-24T08:19:21.556089+0000","last_active":"2023-11-24T08:19:22.604734+0000","last_peered":"2023-11-24T08:19:22.604734+0000","last_clean":"2023-11-24T08:19:22.604734+0000","last_became_active":"2023-11-24T08:19:21.555222+0000","last_became_peered":"2023-11-24T08:19:21.555222+0000","last_unstale":"2023-11-24T08:19:22.604734+0000","last_undegraded":"2023-11-24T08:19:22.604734+0000","last_fullsized":"2023-11-24T08:19:22.604734+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:33.515593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588717+0000","last_change":"2023-11-24T08:19:22.587097+0000","last_active":"2023-11-24T08:19:23.588717+0000","last_peered":"2023-11-24T08:19:23.588717+0000","last_clean":"2023-11-24T08:19:23.588717+0000","last_became_active":"2023-11-24T08:19:22.585891+0000","last_became_peered":"2023-11-24T08:19:22.585891+0000","last_unstale":"2023-11-24T08:19:23.588717+0000","last_undegraded":"2023-11-24T08:19:23.588717+0000","last_fullsized":"2023-11-24T08:19:23.588717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:29.525302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.603237+0000","last_change":"2023-11-24T08:19:21.545458+0000","last_active":"2023-11-24T08:19:22.603237+0000","last_peered":"2023-11-24T08:19:22.603237+0000","last_clean":"2023-11-24T08:19:22.603237+0000","last_became_active":"2023-11-24T08:19:21.544961+0000","last_became_peered":"2023-11-24T08:19:21.544961+0000","last_unstale":"2023-11-24T08:19:22.603237+0000","last_undegraded":"2023-11-24T08:19:22.603237+0000","last_fullsized":"2023-11-24T08:19:22.603237+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:46:17.987011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587785+0000","last_change":"2023-11-24T08:19:22.581976+0000","last_active":"2023-11-24T08:19:23.587785+0000","last_peered":"2023-11-24T08:19:23.587785+0000","last_clean":"2023-11-24T08:19:23.587785+0000","last_became_active":"2023-11-24T08:19:22.581721+0000","last_became_peered":"2023-11-24T08:19:22.581721+0000","last_unstale":"2023-11-24T08:19:23.587785+0000","last_undegraded":"2023-11-24T08:19:23.587785+0000","last_fullsized":"2023-11-24T08:19:23.587785+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:17.005243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607151+0000","last_change":"2023-11-24T08:19:21.556884+0000","last_active":"2023-11-24T08:19:22.607151+0000","last_peered":"2023-11-24T08:19:22.607151+0000","last_clean":"2023-11-24T08:19:22.607151+0000","last_became_active":"2023-11-24T08:19:21.556621+0000","last_became_peered":"2023-11-24T08:19:21.556621+0000","last_unstale":"2023-11-24T08:19:22.607151+0000","last_undegraded":"2023-11-24T08:19:22.607151+0000","last_fullsized":"2023-11-24T08:19:22.607151+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:53.901963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588648+0000","last_change":"2023-11-24T08:19:22.584792+0000","last_active":"2023-11-24T08:19:22.588648+0000","last_peered":"2023-11-24T08:19:22.588648+0000","last_clean":"2023-11-24T08:19:22.588648+0000","last_became_active":"2023-11-24T08:19:22.584517+0000","last_became_peered":"2023-11-24T08:19:22.584517+0000","last_unstale":"2023-11-24T08:19:22.588648+0000","last_undegraded":"2023-11-24T08:19:22.588648+0000","last_fullsized":"2023-11-24T08:19:22.588648+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:57.500978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590920+0000","last_change":"2023-11-24T08:19:21.547542+0000","last_active":"2023-11-24T08:19:23.590920+0000","last_peered":"2023-11-24T08:19:23.590920+0000","last_clean":"2023-11-24T08:19:23.590920+0000","last_became_active":"2023-11-24T08:19:21.545070+0000","last_became_peered":"2023-11-24T08:19:21.545070+0000","last_unstale":"2023-11-24T08:19:23.590920+0000","last_undegraded":"2023-11-24T08:19:23.590920+0000","last_fullsized":"2023-11-24T08:19:23.590920+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:33.923138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600019+0000","last_change":"2023-11-24T08:19:22.594630+0000","last_active":"2023-11-24T08:19:22.600019+0000","last_peered":"2023-11-24T08:19:22.600019+0000","last_clean":"2023-11-24T08:19:22.600019+0000","last_became_active":"2023-11-24T08:19:22.594333+0000","last_became_peered":"2023-11-24T08:19:22.594333+0000","last_unstale":"2023-11-24T08:19:22.600019+0000","last_undegraded":"2023-11-24T08:19:22.600019+0000","last_fullsized":"2023-11-24T08:19:22.600019+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:01:14.610966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607860+0000","last_change":"2023-11-24T08:19:21.559079+0000","last_active":"2023-11-24T08:19:22.607860+0000","last_peered":"2023-11-24T08:19:22.607860+0000","last_clean":"2023-11-24T08:19:22.607860+0000","last_became_active":"2023-11-24T08:19:21.558434+0000","last_became_peered":"2023-11-24T08:19:21.558434+0000","last_unstale":"2023-11-24T08:19:22.607860+0000","last_undegraded":"2023-11-24T08:19:22.607860+0000","last_fullsized":"2023-11-24T08:19:22.607860+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:19.752797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.591048+0000","last_change":"2023-11-24T08:19:22.577393+0000","last_active":"2023-11-24T08:19:23.591048+0000","last_peered":"2023-11-24T08:19:23.591048+0000","last_clean":"2023-11-24T08:19:23.591048+0000","last_became_active":"2023-11-24T08:19:22.577079+0000","last_became_peered":"2023-11-24T08:19:22.577079+0000","last_unstale":"2023-11-24T08:19:23.591048+0000","last_undegraded":"2023-11-24T08:19:23.591048+0000","last_fullsized":"2023-11-24T08:19:23.591048+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:30:35.294595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.566369+0000","last_change":"2023-11-24T08:19:21.561403+0000","last_active":"2023-11-24T08:19:22.566369+0000","last_peered":"2023-11-24T08:19:22.566369+0000","last_clean":"2023-11-24T08:19:22.566369+0000","last_became_active":"2023-11-24T08:19:21.561088+0000","last_became_peered":"2023-11-24T08:19:21.561088+0000","last_unstale":"2023-11-24T08:19:22.566369+0000","last_undegraded":"2023-11-24T08:19:22.566369+0000","last_fullsized":"2023-11-24T08:19:22.566369+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:22:45.521653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588561+0000","last_change":"2023-11-24T08:19:22.582302+0000","last_active":"2023-11-24T08:19:22.588561+0000","last_peered":"2023-11-24T08:19:22.588561+0000","last_clean":"2023-11-24T08:19:22.588561+0000","last_became_active":"2023-11-24T08:19:22.582030+0000","last_became_peered":"2023-11-24T08:19:22.582030+0000","last_unstale":"2023-11-24T08:19:22.588561+0000","last_undegraded":"2023-11-24T08:19:22.588561+0000","last_fullsized":"2023-11-24T08:19:22.588561+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:52.600814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.567383+0000","last_change":"2023-11-24T08:19:21.559637+0000","last_active":"2023-11-24T08:19:22.567383+0000","last_peered":"2023-11-24T08:19:22.567383+0000","last_clean":"2023-11-24T08:19:22.567383+0000","last_became_active":"2023-11-24T08:19:21.559383+0000","last_became_peered":"2023-11-24T08:19:21.559383+0000","last_unstale":"2023-11-24T08:19:22.567383+0000","last_undegraded":"2023-11-24T08:19:22.567383+0000","last_fullsized":"2023-11-24T08:19:22.567383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:48.287993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588170+0000","last_change":"2023-11-24T08:19:22.584963+0000","last_active":"2023-11-24T08:19:22.588170+0000","last_peered":"2023-11-24T08:19:22.588170+0000","last_clean":"2023-11-24T08:19:22.588170+0000","last_became_active":"2023-11-24T08:19:22.584709+0000","last_became_peered":"2023-11-24T08:19:22.584709+0000","last_unstale":"2023-11-24T08:19:22.588170+0000","last_undegraded":"2023-11-24T08:19:22.588170+0000","last_fullsized":"2023-11-24T08:19:22.588170+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:06.456345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586334+0000","last_change":"2023-11-24T08:19:21.557719+0000","last_active":"2023-11-24T08:19:23.586334+0000","last_peered":"2023-11-24T08:19:23.586334+0000","last_clean":"2023-11-24T08:19:23.586334+0000","last_became_active":"2023-11-24T08:19:21.557450+0000","last_became_peered":"2023-11-24T08:19:21.557450+0000","last_unstale":"2023-11-24T08:19:23.586334+0000","last_undegraded":"2023-11-24T08:19:23.586334+0000","last_fullsized":"2023-11-24T08:19:23.586334+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:33.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586041+0000","last_change":"2023-11-24T08:19:22.582505+0000","last_active":"2023-11-24T08:19:22.586041+0000","last_peered":"2023-11-24T08:19:22.586041+0000","last_clean":"2023-11-24T08:19:22.586041+0000","last_became_active":"2023-11-24T08:19:22.582227+0000","last_became_peered":"2023-11-24T08:19:22.582227+0000","last_unstale":"2023-11-24T08:19:22.586041+0000","last_undegraded":"2023-11-24T08:19:22.586041+0000","last_fullsized":"2023-11-24T08:19:22.586041+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:19:59.879051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.606515+0000","last_change":"2023-11-24T08:19:21.554785+0000","last_active":"2023-11-24T08:19:22.606515+0000","last_peered":"2023-11-24T08:19:22.606515+0000","last_clean":"2023-11-24T08:19:22.606515+0000","last_became_active":"2023-11-24T08:19:21.553819+0000","last_became_peered":"2023-11-24T08:19:21.553819+0000","last_unstale":"2023-11-24T08:19:22.606515+0000","last_undegraded":"2023-11-24T08:19:22.606515+0000","last_fullsized":"2023-11-24T08:19:22.606515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:58:48.283067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602725+0000","last_change":"2023-11-24T08:19:22.598143+0000","last_active":"2023-11-24T08:19:22.602725+0000","last_peered":"2023-11-24T08:19:22.602725+0000","last_clean":"2023-11-24T08:19:22.602725+0000","last_became_active":"2023-11-24T08:19:22.597872+0000","last_became_peered":"2023-11-24T08:19:22.597872+0000","last_unstale":"2023-11-24T08:19:22.602725+0000","last_undegraded":"2023-11-24T08:19:22.602725+0000","last_fullsized":"2023-11-24T08:19:22.602725+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:27:29.053204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584046+0000","last_change":"2023-11-24T08:19:21.560805+0000","last_active":"2023-11-24T08:19:22.584046+0000","last_peered":"2023-11-24T08:19:22.584046+0000","last_clean":"2023-11-24T08:19:22.584046+0000","last_became_active":"2023-11-24T08:19:21.560053+0000","last_became_peered":"2023-11-24T08:19:21.560053+0000","last_unstale":"2023-11-24T08:19:22.584046+0000","last_undegraded":"2023-11-24T08:19:22.584046+0000","last_fullsized":"2023-11-24T08:19:22.584046+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:47.137624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588540+0000","last_change":"2023-11-24T08:19:22.584089+0000","last_active":"2023-11-24T08:19:22.588540+0000","last_peered":"2023-11-24T08:19:22.588540+0000","last_clean":"2023-11-24T08:19:22.588540+0000","last_became_active":"2023-11-24T08:19:22.583819+0000","last_became_peered":"2023-11-24T08:19:22.583819+0000","last_unstale":"2023-11-24T08:19:22.588540+0000","last_undegraded":"2023-11-24T08:19:22.588540+0000","last_fullsized":"2023-11-24T08:19:22.588540+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:30.395795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.582352+0000","last_change":"2023-11-24T08:19:21.559892+0000","last_active":"2023-11-24T08:19:22.582352+0000","last_peered":"2023-11-24T08:19:22.582352+0000","last_clean":"2023-11-24T08:19:22.582352+0000","last_became_active":"2023-11-24T08:19:21.558640+0000","last_became_peered":"2023-11-24T08:19:21.558640+0000","last_unstale":"2023-11-24T08:19:22.582352+0000","last_undegraded":"2023-11-24T08:19:22.582352+0000","last_fullsized":"2023-11-24T08:19:22.582352+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:23.288804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.585624+0000","last_change":"2023-11-24T08:19:22.581025+0000","last_active":"2023-11-24T08:19:22.585624+0000","last_peered":"2023-11-24T08:19:22.585624+0000","last_clean":"2023-11-24T08:19:22.585624+0000","last_became_active":"2023-11-24T08:19:22.580698+0000","last_became_peered":"2023-11-24T08:19:22.580698+0000","last_unstale":"2023-11-24T08:19:22.585624+0000","last_undegraded":"2023-11-24T08:19:22.585624+0000","last_fullsized":"2023-11-24T08:19:22.585624+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:23:17.425447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.563255+0000","last_change":"2023-11-24T08:19:21.550954+0000","last_active":"2023-11-24T08:19:22.563255+0000","last_peered":"2023-11-24T08:19:22.563255+0000","last_clean":"2023-11-24T08:19:22.563255+0000","last_became_active":"2023-11-24T08:19:21.550695+0000","last_became_peered":"2023-11-24T08:19:21.550695+0000","last_unstale":"2023-11-24T08:19:22.563255+0000","last_undegraded":"2023-11-24T08:19:22.563255+0000","last_fullsized":"2023-11-24T08:19:22.563255+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:34.871855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584658+0000","last_change":"2023-11-24T08:19:22.581831+0000","last_active":"2023-11-24T08:19:22.584658+0000","last_peered":"2023-11-24T08:19:22.584658+0000","last_clean":"2023-11-24T08:19:22.584658+0000","last_became_active":"2023-11-24T08:19:22.581556+0000","last_became_peered":"2023-11-24T08:19:22.581556+0000","last_unstale":"2023-11-24T08:19:22.584658+0000","last_undegraded":"2023-11-24T08:19:22.584658+0000","last_fullsized":"2023-11-24T08:19:22.584658+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:49.386749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587555+0000","last_change":"2023-11-24T08:19:21.556655+0000","last_active":"2023-11-24T08:19:23.587555+0000","last_peered":"2023-11-24T08:19:23.587555+0000","last_clean":"2023-11-24T08:19:23.587555+0000","last_became_active":"2023-11-24T08:19:21.556351+0000","last_became_peered":"2023-11-24T08:19:21.556351+0000","last_unstale":"2023-11-24T08:19:23.587555+0000","last_undegraded":"2023-11-24T08:19:23.587555+0000","last_fullsized":"2023-11-24T08:19:23.587555+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:06.649697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584584+0000","last_change":"2023-11-24T08:19:22.581463+0000","last_active":"2023-11-24T08:19:22.584584+0000","last_peered":"2023-11-24T08:19:22.584584+0000","last_clean":"2023-11-24T08:19:22.584584+0000","last_became_active":"2023-11-24T08:19:22.581163+0000","last_became_peered":"2023-11-24T08:19:22.581163+0000","last_unstale":"2023-11-24T08:19:22.584584+0000","last_undegraded":"2023-11-24T08:19:22.584584+0000","last_fullsized":"2023-11-24T08:19:22.584584+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:46.486016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587552+0000","last_change":"2023-11-24T08:19:21.557321+0000","last_active":"2023-11-24T08:19:23.587552+0000","last_peered":"2023-11-24T08:19:23.587552+0000","last_clean":"2023-11-24T08:19:23.587552+0000","last_became_active":"2023-11-24T08:19:21.556639+0000","last_became_peered":"2023-11-24T08:19:21.556639+0000","last_unstale":"2023-11-24T08:19:23.587552+0000","last_undegraded":"2023-11-24T08:19:23.587552+0000","last_fullsized":"2023-11-24T08:19:23.587552+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:16:01.640346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588581+0000","last_change":"2023-11-24T08:19:22.583991+0000","last_active":"2023-11-24T08:19:22.588581+0000","last_peered":"2023-11-24T08:19:22.588581+0000","last_clean":"2023-11-24T08:19:22.588581+0000","last_became_active":"2023-11-24T08:19:22.583601+0000","last_became_peered":"2023-11-24T08:19:22.583601+0000","last_unstale":"2023-11-24T08:19:22.588581+0000","last_undegraded":"2023-11-24T08:19:22.588581+0000","last_fullsized":"2023-11-24T08:19:22.588581+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:07.660157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.572098+0000","last_change":"2023-11-24T08:19:21.558779+0000","last_active":"2023-11-24T08:19:22.572098+0000","last_peered":"2023-11-24T08:19:22.572098+0000","last_clean":"2023-11-24T08:19:22.572098+0000","last_became_active":"2023-11-24T08:19:21.558494+0000","last_became_peered":"2023-11-24T08:19:21.558494+0000","last_unstale":"2023-11-24T08:19:22.572098+0000","last_undegraded":"2023-11-24T08:19:22.572098+0000","last_fullsized":"2023-11-24T08:19:22.572098+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:25.247835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.596152+0000","last_change":"2023-11-24T08:19:22.589444+0000","last_active":"2023-11-24T08:19:22.596152+0000","last_peered":"2023-11-24T08:19:22.596152+0000","last_clean":"2023-11-24T08:19:22.596152+0000","last_became_active":"2023-11-24T08:19:22.589124+0000","last_became_peered":"2023-11-24T08:19:22.589124+0000","last_unstale":"2023-11-24T08:19:22.596152+0000","last_undegraded":"2023-11-24T08:19:22.596152+0000","last_fullsized":"2023-11-24T08:19:22.596152+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:57.941975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.568326+0000","last_change":"2023-11-24T08:19:21.555464+0000","last_active":"2023-11-24T08:19:22.568326+0000","last_peered":"2023-11-24T08:19:22.568326+0000","last_clean":"2023-11-24T08:19:22.568326+0000","last_became_active":"2023-11-24T08:19:21.555164+0000","last_became_peered":"2023-11-24T08:19:21.555164+0000","last_unstale":"2023-11-24T08:19:22.568326+0000","last_undegraded":"2023-11-24T08:19:22.568326+0000","last_fullsized":"2023-11-24T08:19:22.568326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:15.994784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599037+0000","last_change":"2023-11-24T08:19:22.593765+0000","last_active":"2023-11-24T08:19:22.599037+0000","last_peered":"2023-11-24T08:19:22.599037+0000","last_clean":"2023-11-24T08:19:22.599037+0000","last_became_active":"2023-11-24T08:19:22.593449+0000","last_became_peered":"2023-11-24T08:19:22.593449+0000","last_unstale":"2023-11-24T08:19:22.599037+0000","last_undegraded":"2023-11-24T08:19:22.599037+0000","last_fullsized":"2023-11-24T08:19:22.599037+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:30:37.347675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.568201+0000","last_change":"2023-11-24T08:19:21.560369+0000","last_active":"2023-11-24T08:19:22.568201+0000","last_peered":"2023-11-24T08:19:22.568201+0000","last_clean":"2023-11-24T08:19:22.568201+0000","last_became_active":"2023-11-24T08:19:21.560074+0000","last_became_peered":"2023-11-24T08:19:21.560074+0000","last_unstale":"2023-11-24T08:19:22.568201+0000","last_undegraded":"2023-11-24T08:19:22.568201+0000","last_fullsized":"2023-11-24T08:19:22.568201+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:35.139730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":302,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-11-24T08:19:21.546680+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:19:21.546680+0000","last_peered":"2023-11-24T08:19:21.546680+0000","last_clean":"2023-11-24T08:19:21.546680+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:19:21.546680+0000","last_undegraded":"2023-11-24T08:19:21.546680+0000","last_fullsized":"2023-11-24T08:19:21.546680+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575028,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38068,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333772,"statfs":{"total":96636764160,"available":96597782528,"internally_reserved":0,"allocated":2789376,"data_stored":1554044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.042,"15min":1.0529999999999999},"min":{"1min":0.70499999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.601,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.208,"5min":1.0940000000000001,"15min":1.097},"min":{"1min":0.91500000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.625,"5min":1.774,"15min":2.7109999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1739999999999999,"15min":1.1579999999999999},"min":{"1min":0.77000000000000002,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.595,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":1.135},{"interface":"front","average":{"1min":1.23,"5min":1.21,"15min":1.216},"min":{"1min":0.77600000000000002,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":1.653,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":1.0389999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.111,"15min":1.139},"min":{"1min":0.67500000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4690000000000001,"5min":1.698,"15min":4.7000000000000002},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.28,"5min":1.302,"15min":1.319},"min":{"1min":0.94799999999999995,"5min":0.71799999999999997,"15min":0.60399999999999998},"max":{"1min":1.5509999999999999,"5min":1.889,"15min":4.633},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2290000000000001,"15min":1.2070000000000001},"min":{"1min":0.79100000000000004,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.5960000000000001,"5min":1.99,"15min":3.133},"last":1.083},{"interface":"front","average":{"1min":1.333,"5min":1.3640000000000001,"15min":1.375},"min":{"1min":0.879,"5min":0.622,"15min":0.622},"max":{"1min":1.696,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.3799999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.381,"15min":1.3560000000000001},"min":{"1min":0.95399999999999996,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":1.657,"5min":2.665,"15min":2.851},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":0.89900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.0419999999999998,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.181}]},{"osd":5,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.486,"15min":1.48},"min":{"1min":0.88300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7929999999999999,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.522},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.508,"15min":1.498},"min":{"1min":0.94799999999999995,"5min":0.622,"15min":0.622},"max":{"1min":1.988,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.575}]},{"osd":6,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.579,"15min":1.5720000000000001},"min":{"1min":1.0649999999999999,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":1.863,"5min":2.702,"15min":2.992},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.5429999999999999,"15min":1.506},"min":{"1min":1.2110000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4159999999999999,"5min":2.6259999999999999,"15min":3.798},"last":1.466}]}]},{"osd":6,"up_from":13,"seq":55834575027,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37480,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334360,"statfs":{"total":96636764160,"available":96598384640,"internally_reserved":0,"allocated":2187264,"data_stored":963572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0580000000000001,"15min":1.0760000000000001},"min":{"1min":0.65500000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.95},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.085,"15min":1.083},"min":{"1min":0.68400000000000005,"5min":0.63900000000000001,"15min":0.58099999999999996},"max":{"1min":1.494,"5min":1.7370000000000001,"15min":2.2069999999999999},"last":1.0900000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.135,"15min":1.1699999999999999},"min":{"1min":0.72899999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6080000000000001,"5min":1.835,"15min":2.0230000000000001},"last":1.218},{"interface":"front","average":{"1min":1.252,"5min":1.242,"15min":1.2070000000000001},"min":{"1min":0.79100000000000004,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.867,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.1930000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0900000000000001,"15min":1.1000000000000001},"min":{"1min":0.71699999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4990000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.284},"min":{"1min":0.86899999999999999,"5min":0.746,"15min":0.60399999999999998},"max":{"1min":1.7,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.359,"15min":1.3500000000000001},"min":{"1min":0.92400000000000004,"5min":0.86199999999999999,"15min":0.71499999999999997},"max":{"1min":1.9650000000000001,"5min":2.2330000000000001,"15min":4.7880000000000003},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2789999999999999,"15min":1.3140000000000001},"min":{"1min":0.94199999999999995,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.681,"5min":2.194,"15min":4.8319999999999999},"last":1.319}]},{"osd":4,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4450000000000001,"15min":1.4470000000000001},"min":{"1min":1.0469999999999999,"5min":0.89100000000000001,"15min":0.76500000000000001},"max":{"1min":1.857,"5min":2.702,"15min":2.895},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.492,"15min":1.5009999999999999},"min":{"1min":1.0429999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0190000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":1.266}]},{"osd":5,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.413,"15min":1.4490000000000001},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7450000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.165},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5680000000000001,"15min":1.587},"min":{"1min":1.274,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9590000000000001,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":1.5629999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.615,"15min":1.617},"min":{"1min":1.198,"5min":0.748,"15min":0.748},"max":{"1min":2.0779999999999998,"5min":2.903,"15min":3.0329999999999999},"last":1.466},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5940000000000001,"15min":1.6080000000000001},"min":{"1min":1.292,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.032,"5min":2.649,"15min":2.6890000000000001},"last":1.5049999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575027,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37484,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334356,"statfs":{"total":96636764160,"available":96598380544,"internally_reserved":0,"allocated":2187264,"data_stored":963572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0209999999999999,"15min":1.038},"min":{"1min":0.61599999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5309999999999999,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0780000000000001,"15min":1.1160000000000001},"min":{"1min":0.82799999999999996,"5min":0.66500000000000004,"15min":0.60599999999999998},"max":{"1min":1.5269999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":0.998}]},{"osd":1,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1399999999999999,"15min":1.1339999999999999},"min":{"1min":0.78800000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7350000000000001,"5min":7.258,"15min":7.258},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.76100000000000001,"5min":0.71799999999999997,"15min":0.48699999999999999},"max":{"1min":1.796,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.70699999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1859999999999999,"15min":1.1899999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.51600000000000001},"max":{"1min":1.579,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.214,"15min":1.175},"min":{"1min":0.82299999999999995,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.5089999999999999,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.155}]},{"osd":3,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.179},"min":{"1min":0.74099999999999999,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":2.242,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.3120000000000001,"15min":1.2909999999999999},"min":{"1min":0.91500000000000004,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.71,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.101}]},{"osd":4,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.472,"15min":1.3660000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":2.359,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.53,"15min":1.494},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":1.9610000000000001,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.228}]},{"osd":6,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.4139999999999999,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":2.0830000000000002,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.304},{"interface":"front","average":{"1min":1.498,"5min":1.534,"15min":1.452},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":2.452,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.522}]},{"osd":7,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5580000000000001,"15min":1.5169999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":1.8220000000000001,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.7589999999999999,"15min":1.6439999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.7240000000000002,"5min":19.663,"15min":19.663},"last":1.5860000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2203648,"data_stored":965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0129999999999999,"15min":1.0680000000000001},"min":{"1min":0.70799999999999996,"5min":0.626,"15min":0.42699999999999999},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.52},{"interface":"front","average":{"1min":1.034,"5min":1.0900000000000001,"15min":1.079},"min":{"1min":0.75900000000000001,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.591,"5min":5.452,"15min":5.452},"last":0.83999999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.127,"15min":1.1359999999999999},"min":{"1min":0.67600000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6399999999999999,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.1060000000000001,"15min":1.153},"min":{"1min":0.71499999999999997,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.3979999999999999,"5min":1.988,"15min":1.988},"last":1.056}]},{"osd":2,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0489999999999999,"15min":1.079},"min":{"1min":0.73699999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.6519999999999999,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.244,"15min":1.256},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.827,"5min":1.827,"15min":2.7229999999999999},"last":1.2030000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.76800000000000002,"5min":0.74399999999999999,"15min":0.62},"max":{"1min":1.7370000000000001,"5min":1.9750000000000001,"15min":2.9100000000000001},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2789999999999999,"15min":1.276},"min":{"1min":0.93999999999999995,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.7290000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.575}]},{"osd":5,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.54,"15min":1.496},"min":{"1min":1.159,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.893,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":0.998},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4159999999999999,"15min":1.4399999999999999},"min":{"1min":0.79200000000000004,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":2.0419999999999998,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.718}]},{"osd":6,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.246,"15min":1.278},"min":{"1min":0.83899999999999997,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.8540000000000001,"5min":1.907,"15min":3.5899999999999999},"last":1.47},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5700000000000001,"15min":1.5529999999999999},"min":{"1min":1.3280000000000001,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.2890000000000001,"15min":3.391},"last":1.7589999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.486,"15min":1.504},"min":{"1min":0.72899999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.948,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.403},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.425,"15min":1.4339999999999999},"min":{"1min":0.98299999999999998,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":3.4849999999999999},"last":1.627}]}]},{"osd":3,"up_from":13,"seq":55834575028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37488,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334352,"statfs":{"total":96636764160,"available":96598376448,"internally_reserved":0,"allocated":2199552,"data_stored":963766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.75700000000000001,"15min":0.84699999999999998},"min":{"1min":0.45400000000000001,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":0.97499999999999998,"5min":1.655,"15min":10.557},"last":1.5820000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.873,"15min":0.91200000000000003},"min":{"1min":0.56899999999999995,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2210000000000001,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.96799999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0109999999999999,"5min":1.728,"15min":2.101},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.84899999999999998,"15min":0.96499999999999997},"min":{"1min":0.52600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.236,"5min":1.8799999999999999,"15min":13.090999999999999},"last":1.1180000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81799999999999995,"15min":0.90700000000000003},"min":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3260000000000001,"5min":1.4750000000000001,"15min":3.9489999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":1.0529999999999999},"min":{"1min":0.54400000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.518,"5min":1.796,"15min":13.460000000000001},"last":1.665}]},{"osd":4,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2090000000000001,"15min":1.266},"min":{"1min":0.874,"5min":0.73999999999999999,"15min":0.67800000000000005},"max":{"1min":1.5680000000000001,"5min":1.877,"15min":4.367},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1899999999999999,"15min":1.224},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.609,"5min":1.9630000000000001,"15min":3.3380000000000001},"last":0.871}]},{"osd":5,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2110000000000001,"15min":1.294},"min":{"1min":0.74099999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":13.231},"last":1.726},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1699999999999999,"15min":1.25},"min":{"1min":0.84999999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6910000000000001,"5min":1.992,"15min":5.1180000000000003},"last":1.399}]},{"osd":6,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.258,"15min":1.357},"min":{"1min":0.89900000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7689999999999999,"5min":2.0449999999999999,"15min":13.359999999999999},"last":1.762},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2050000000000001,"15min":1.254},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3980000000000001,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.8420000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2370000000000001,"15min":1.331},"min":{"1min":0.85899999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8260000000000001,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.246,"15min":1.3169999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":3.0030000000000001,"5min":3.0030000000000001,"15min":4.032},"last":1.49}]}]},{"osd":2,"up_from":13,"seq":55834575026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2199552,"data_stored":964266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68999999999999995,"15min":0.78600000000000003},"min":{"1min":0.34200000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.075,"5min":1.601,"15min":1.9610000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80400000000000005,"15min":0.88200000000000001},"min":{"1min":0.437,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.361,"5min":1.7390000000000001,"15min":2.5779999999999998},"last":0.91700000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.82599999999999996,"15min":0.92500000000000004},"min":{"1min":0.47299999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.3120000000000001,"5min":1.518,"15min":1.9890000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.89000000000000001,"15min":0.96699999999999997},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.373,"5min":1.6200000000000001,"15min":2.6240000000000001},"last":0.95499999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84299999999999997,"15min":0.872},"min":{"1min":0.51300000000000001,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.633,"5min":1.6850000000000001,"15min":2.2090000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.95299999999999996,"15min":1.0549999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5820000000000001,"5min":1.927,"15min":2.9950000000000001},"last":0.85199999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.22,"15min":1.2889999999999999},"min":{"1min":0.72799999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.782,"5min":1.968,"15min":2.7160000000000002},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1839999999999999,"15min":1.268},"min":{"1min":0.78700000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.6639999999999999,"5min":1.8839999999999999,"15min":3.3279999999999998},"last":1.5209999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1599999999999999,"15min":1.272},"min":{"1min":0.753,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.8360000000000001,"5min":1.857,"15min":2.508},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.1850000000000001,"15min":1.26},"min":{"1min":0.752,"5min":0.499,"15min":0.499},"max":{"1min":1.677,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.218}]},{"osd":6,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1539999999999999,"15min":1.2170000000000001},"min":{"1min":0.71899999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.694,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2290000000000001,"15min":1.339},"min":{"1min":0.83599999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.72,"5min":1.911,"15min":2.6629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.258,"15min":1.3420000000000001},"min":{"1min":0.77400000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8779999999999999,"5min":2.3999999999999999,"15min":2.859},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2430000000000001,"15min":1.3109999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.764,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":1.4750000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2199552,"data_stored":965184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77500000000000002,"15min":0.79700000000000004},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.879,"15min":0.88},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3540000000000001,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.2410000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78900000000000003,"15min":0.78600000000000003},"min":{"1min":0.436,"5min":0.374,"15min":0.37},"max":{"1min":1.242,"5min":1.5229999999999999,"15min":1.861},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.90800000000000003},"min":{"1min":0.54500000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.7490000000000001,"5min":1.925,"15min":1.925},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.63300000000000001,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.42899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5669999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.5980000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.256,"15min":1.266},"min":{"1min":0.77200000000000002,"5min":0.76300000000000001,"15min":0.65300000000000002},"max":{"1min":1.984,"5min":1.984,"15min":2.7959999999999998},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.2509999999999999,"15min":1.2490000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.518,"5min":2.141,"15min":3.0409999999999999},"last":1.5149999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2210000000000001,"15min":1.244},"min":{"1min":0.748,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.8879999999999999,"5min":1.9079999999999999,"15min":3.7639999999999998},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1870000000000001,"15min":1.2},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.643,"5min":1.8340000000000001,"15min":3.286},"last":1.302}]},{"osd":6,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.252,"15min":1.268},"min":{"1min":0.76400000000000001,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":1.6739999999999999,"5min":2.0419999999999998,"15min":4.2750000000000004},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.236,"15min":1.254},"min":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.63100000000000001},"max":{"1min":1.8029999999999999,"5min":1.9690000000000001,"15min":3.7040000000000002},"last":1.5509999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3129999999999999,"15min":1.3109999999999999},"min":{"1min":0.82299999999999995,"5min":0.80200000000000005,"15min":0.59099999999999997},"max":{"1min":1.7969999999999999,"5min":2.1429999999999998,"15min":3.1760000000000002},"last":1.45},{"interface":"front","average":{"1min":1.208,"5min":1.2789999999999999,"15min":1.3129999999999999},"min":{"1min":0.78800000000000003,"5min":0.624,"15min":0.624},"max":{"1min":1.829,"5min":1.929,"15min":3.653},"last":1.6259999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575028,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2785280,"data_stored":1553962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.1819999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78100000000000003,"15min":0.85399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.302,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":1.081}]},{"osd":2,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98899999999999999,"15min":1.0149999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.627,"5min":1.893,"15min":1.893},"last":1.149},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.874},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.54,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.0129999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.92800000000000005,"15min":0.99099999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"max":{"1min":1.4430000000000001,"5min":1.679,"15min":2.0670000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.87,"15min":0.91700000000000004},"min":{"1min":0.53200000000000003,"5min":0.48899999999999999,"15min":0.40100000000000002},"max":{"1min":1.4970000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.169,"15min":1.244},"min":{"1min":0.82899999999999996,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":1.6539999999999999,"5min":1.778,"15min":3.7610000000000001},"last":1.278},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.1080000000000001,"15min":1.1319999999999999},"min":{"1min":0.80200000000000005,"5min":0.72299999999999998,"15min":0.67800000000000005},"max":{"1min":1.4399999999999999,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4390000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.80800000000000005,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.4410000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.232},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.71099999999999997},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":2.7240000000000002},"last":1.1160000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2310000000000001,"15min":1.2869999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.8560000000000001,"5min":2.052,"15min":2.4369999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.2370000000000001,"15min":1.286},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.5489999999999999,"5min":2.823,"15min":2.823},"last":1.3400000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3109999999999999,"15min":1.3740000000000001},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.6989999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2729999999999999,"15min":1.3440000000000001},"min":{"1min":0.88100000000000001,"5min":0.747,"15min":0.747},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3929999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:19:27.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:19:27.515 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:19:27.513+0000 7fb6bf5f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:19:27.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.733+0000 7f13a8bb0640 1 Processor -- start 2023-11-24T08:19:27.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.733+0000 7f13a8bb0640 1 -- start start 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.734+0000 7f13a8bb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13a415c7f0 0x7f13a406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a8bb0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13a40ce6c0 0x7f13a40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a8bb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40cece0 con 0x7f13a40cf450 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a8bb0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40cfe40 con 0x7f13a415c7f0 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a8bb0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40ceaf0 con 0x7f13a40ce6c0 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13a415c7f0 0x7f13a406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13a415c7f0 0x7f13a406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48706/0 (socket says 172.21.15.107:48706) 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a259c640 1 -- 172.21.15.107:0/333864848 learned_addr learned my addr 172.21.15.107:0/333864848 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.735+0000 7f13a2d9d640 1 --2- 172.21.15.107:0/333864848 >> v2:172.21.15.120:3300/0 conn(0x7f13a40ce6c0 0x7f13a40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.736+0000 7f13a259c640 1 -- 172.21.15.107:0/333864848 >> v2:172.21.15.120:3300/0 conn(0x7f13a40ce6c0 msgr2=0x7f13a40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.736+0000 7f13a259c640 1 --2- 172.21.15.107:0/333864848 >> v2:172.21.15.120:3300/0 conn(0x7f13a40ce6c0 0x7f13a40cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.736+0000 7f13a259c640 1 -- 172.21.15.107:0/333864848 >> v1:172.21.15.120:6789/0 conn(0x7f13a40cf450 legacy=0x7f13a40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.736+0000 7f13a259c640 1 -- 172.21.15.107:0/333864848 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a406d040 con 0x7f13a415c7f0 2023-11-24T08:19:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.737+0000 7f13a259c640 1 --2- 172.21.15.107:0/333864848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a415c7f0 0x7f13a406db20 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f13a40d00e0 tx=0x7f139402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f16d0904953f577 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a1d9b640 1 -- 172.21.15.107:0/333864848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394030860 con 0x7f13a415c7f0 2023-11-24T08:19:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a1d9b640 1 -- 172.21.15.107:0/333864848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13940309c0 con 0x7f13a415c7f0 2023-11-24T08:19:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a1d9b640 1 -- 172.21.15.107:0/333864848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394030d50 con 0x7f13a415c7f0 2023-11-24T08:19:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 -- 172.21.15.107:0/333864848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a415c7f0 msgr2=0x7f13a406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/333864848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a415c7f0 0x7f13a406db20 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f13a40d00e0 tx=0x7f139402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 -- 172.21.15.107:0/333864848 shutdown_connections 2023-11-24T08:19:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/333864848 >> v2:172.21.15.120:3300/0 conn(0x7f13a40ce6c0 0x7f13a40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/333864848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a415c7f0 0x7f13a406db20 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 -- 172.21.15.107:0/333864848 >> 172.21.15.107:0/333864848 conn(0x7f13a406a920 msgr2=0x7f13a406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:27.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 -- 172.21.15.107:0/333864848 shutdown_connections 2023-11-24T08:19:27.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.738+0000 7f13a8bb0640 1 -- 172.21.15.107:0/333864848 wait complete. 2023-11-24T08:19:27.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.740+0000 7f13a8bb0640 1 Processor -- start 2023-11-24T08:19:27.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.740+0000 7f13a8bb0640 1 -- start start 2023-11-24T08:19:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13a40ce6c0 0x7f13a4152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 0x7f13a4154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a4154280 con 0x7f13a40cf450 2023-11-24T08:19:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a4154c50 con 0x7f13a415c7f0 2023-11-24T08:19:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f13a8bb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a4153670 con 0x7f13a40ce6c0 2023-11-24T08:19:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.741+0000 7f139bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48716/0 (socket says 172.21.15.107:48716) 2023-11-24T08:19:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 -- 172.21.15.107:0/3240561126 learned_addr learned my addr 172.21.15.107:0/3240561126 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 -- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13a40ce6c0 msgr2=0x7f13a4152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13a40ce6c0 0x7f13a4152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 -- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 msgr2=0x7f13a4154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f13a2d9d640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 0x7f13a4154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 0x7f13a4154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 -- 172.21.15.107:0/3240561126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940094d0 con 0x7f13a40cf450 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f13a2d9d640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 0x7f13a4154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139bfff640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c450 tx=0x7f138c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.742+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c0052d0 con 0x7f13a40cf450 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.743+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c005430 con 0x7f13a40cf450 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.743+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4153950 con 0x7f13a40cf450 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.743+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c012870 con 0x7f13a40cf450 2023-11-24T08:19:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.743+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a4153f10 con 0x7f13a40cf450 2023-11-24T08:19:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.744+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a41540a0 con 0x7f13a40cf450 2023-11-24T08:19:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.745+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f138c0129d0 con 0x7f13a40cf450 2023-11-24T08:19:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.746+0000 7f139affd640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 0x7f138406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:27.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.746+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f138c088040 con 0x7f13a40cf450 2023-11-24T08:19:27.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.746+0000 7f13a259c640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 0x7f138406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:27.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.747+0000 7f13a259c640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 0x7f138406ed70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f139403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:27.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.750+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f138c054b50 con 0x7f13a40cf450 2023-11-24T08:19:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.935+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13a406efa0 con 0x7f138406c8b0 2023-11-24T08:19:27.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.948+0000 7f139affd640 1 -- 172.21.15.107:0/3240561126 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287509 (secure 0 0 0) 0x7f13a406efa0 con 0x7f138406c8b0 2023-11-24T08:19:27.953 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:27.954 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:19:27.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 msgr2=0x7f138406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 0x7f138406ed70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f139403b040 comp rx=0 tx=0).stop 2023-11-24T08:19:27.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 msgr2=0x7f13a4145530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:27.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c450 tx=0x7f138c00c910 comp rx=0 tx=0).stop 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 shutdown_connections 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f138406c8b0 0x7f138406ed70 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13a415c7f0 0x7f13a4154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13a40cf450 0x7f13a4145530 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 --2- 172.21.15.107:0/3240561126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13a40ce6c0 0x7f13a4152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 >> 172.21.15.107:0/3240561126 conn(0x7f13a406a920 msgr2=0x7f13a415eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 shutdown_connections 2023-11-24T08:19:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:27.953+0000 7f13a8bb0640 1 -- 172.21.15.107:0/3240561126 wait complete. 2023-11-24T08:19:27.975 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2023-11-24T08:19:27.707899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309384,"kb_used_data":18312,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754665336,"statfs":{"total":773094113280,"available":772777304064,"internally_reserved":0,"allocated":18751488,"data_stored":8894132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":297913918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.183250"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.595245+0000","last_change":"2023-11-24T08:19:22.588643+0000","last_active":"2023-11-24T08:19:22.595245+0000","last_peered":"2023-11-24T08:19:22.595245+0000","last_clean":"2023-11-24T08:19:22.595245+0000","last_became_active":"2023-11-24T08:19:22.588118+0000","last_became_peered":"2023-11-24T08:19:22.588118+0000","last_unstale":"2023-11-24T08:19:22.595245+0000","last_undegraded":"2023-11-24T08:19:22.595245+0000","last_fullsized":"2023-11-24T08:19:22.595245+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:23:17.425447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586717+0000","last_change":"2023-11-24T08:19:22.580198+0000","last_active":"2023-11-24T08:19:22.586717+0000","last_peered":"2023-11-24T08:19:22.586717+0000","last_clean":"2023-11-24T08:19:22.586717+0000","last_became_active":"2023-11-24T08:19:22.579897+0000","last_became_peered":"2023-11-24T08:19:22.579897+0000","last_unstale":"2023-11-24T08:19:22.586717+0000","last_undegraded":"2023-11-24T08:19:22.586717+0000","last_fullsized":"2023-11-24T08:19:22.586717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:34.933597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591052+0000","last_change":"2023-11-24T08:19:22.587251+0000","last_active":"2023-11-24T08:19:22.591052+0000","last_peered":"2023-11-24T08:19:22.591052+0000","last_clean":"2023-11-24T08:19:22.591052+0000","last_became_active":"2023-11-24T08:19:22.586943+0000","last_became_peered":"2023-11-24T08:19:22.586943+0000","last_unstale":"2023-11-24T08:19:22.591052+0000","last_undegraded":"2023-11-24T08:19:22.591052+0000","last_fullsized":"2023-11-24T08:19:22.591052+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:27.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586841+0000","last_change":"2023-11-24T08:19:22.580053+0000","last_active":"2023-11-24T08:19:22.586841+0000","last_peered":"2023-11-24T08:19:22.586841+0000","last_clean":"2023-11-24T08:19:22.586841+0000","last_became_active":"2023-11-24T08:19:22.579739+0000","last_became_peered":"2023-11-24T08:19:22.579739+0000","last_unstale":"2023-11-24T08:19:22.586841+0000","last_undegraded":"2023-11-24T08:19:22.586841+0000","last_fullsized":"2023-11-24T08:19:22.586841+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:27:52.970487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.601987+0000","last_change":"2023-11-24T08:19:22.592309+0000","last_active":"2023-11-24T08:19:22.601987+0000","last_peered":"2023-11-24T08:19:22.601987+0000","last_clean":"2023-11-24T08:19:22.601987+0000","last_became_active":"2023-11-24T08:19:22.592024+0000","last_became_peered":"2023-11-24T08:19:22.592024+0000","last_unstale":"2023-11-24T08:19:22.601987+0000","last_undegraded":"2023-11-24T08:19:22.601987+0000","last_fullsized":"2023-11-24T08:19:22.601987+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:13:43.146970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591974+0000","last_change":"2023-11-24T08:19:22.586770+0000","last_active":"2023-11-24T08:19:22.591974+0000","last_peered":"2023-11-24T08:19:22.591974+0000","last_clean":"2023-11-24T08:19:22.591974+0000","last_became_active":"2023-11-24T08:19:22.585338+0000","last_became_peered":"2023-11-24T08:19:22.585338+0000","last_unstale":"2023-11-24T08:19:22.591974+0000","last_undegraded":"2023-11-24T08:19:22.591974+0000","last_fullsized":"2023-11-24T08:19:22.591974+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590027+0000","last_change":"2023-11-24T08:19:22.592042+0000","last_active":"2023-11-24T08:19:23.590027+0000","last_peered":"2023-11-24T08:19:23.590027+0000","last_clean":"2023-11-24T08:19:23.590027+0000","last_became_active":"2023-11-24T08:19:22.591732+0000","last_became_peered":"2023-11-24T08:19:22.591732+0000","last_unstale":"2023-11-24T08:19:23.590027+0000","last_undegraded":"2023-11-24T08:19:23.590027+0000","last_fullsized":"2023-11-24T08:19:23.590027+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.579673+0000","last_change":"2023-11-24T08:19:22.578021+0000","last_active":"2023-11-24T08:19:22.579673+0000","last_peered":"2023-11-24T08:19:22.579673+0000","last_clean":"2023-11-24T08:19:22.579673+0000","last_became_active":"2023-11-24T08:19:22.577653+0000","last_became_peered":"2023-11-24T08:19:22.577653+0000","last_unstale":"2023-11-24T08:19:22.579673+0000","last_undegraded":"2023-11-24T08:19:22.579673+0000","last_fullsized":"2023-11-24T08:19:22.579673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:04:32.767118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588509+0000","last_change":"2023-11-24T08:19:22.581067+0000","last_active":"2023-11-24T08:19:23.588509+0000","last_peered":"2023-11-24T08:19:23.588509+0000","last_clean":"2023-11-24T08:19:23.588509+0000","last_became_active":"2023-11-24T08:19:22.580040+0000","last_became_peered":"2023-11-24T08:19:22.580040+0000","last_unstale":"2023-11-24T08:19:23.588509+0000","last_undegraded":"2023-11-24T08:19:23.588509+0000","last_fullsized":"2023-11-24T08:19:23.588509+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.763256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602451+0000","last_change":"2023-11-24T08:19:22.597984+0000","last_active":"2023-11-24T08:19:22.602451+0000","last_peered":"2023-11-24T08:19:22.602451+0000","last_clean":"2023-11-24T08:19:22.602451+0000","last_became_active":"2023-11-24T08:19:22.597712+0000","last_became_peered":"2023-11-24T08:19:22.597712+0000","last_unstale":"2023-11-24T08:19:22.602451+0000","last_undegraded":"2023-11-24T08:19:22.602451+0000","last_fullsized":"2023-11-24T08:19:22.602451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:02.003634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590679+0000","last_change":"2023-11-24T08:19:22.582432+0000","last_active":"2023-11-24T08:19:23.590679+0000","last_peered":"2023-11-24T08:19:23.590679+0000","last_clean":"2023-11-24T08:19:23.590679+0000","last_became_active":"2023-11-24T08:19:22.582138+0000","last_became_peered":"2023-11-24T08:19:22.582138+0000","last_unstale":"2023-11-24T08:19:23.590679+0000","last_undegraded":"2023-11-24T08:19:23.590679+0000","last_fullsized":"2023-11-24T08:19:23.590679+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.763256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588618+0000","last_change":"2023-11-24T08:19:22.583314+0000","last_active":"2023-11-24T08:19:22.588618+0000","last_peered":"2023-11-24T08:19:22.588618+0000","last_clean":"2023-11-24T08:19:22.588618+0000","last_became_active":"2023-11-24T08:19:22.582659+0000","last_became_peered":"2023-11-24T08:19:22.582659+0000","last_unstale":"2023-11-24T08:19:22.588618+0000","last_undegraded":"2023-11-24T08:19:22.588618+0000","last_fullsized":"2023-11-24T08:19:22.588618+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:54.466137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.582488+0000","last_change":"2023-11-24T08:19:22.578217+0000","last_active":"2023-11-24T08:19:22.582488+0000","last_peered":"2023-11-24T08:19:22.582488+0000","last_clean":"2023-11-24T08:19:22.582488+0000","last_became_active":"2023-11-24T08:19:22.577724+0000","last_became_peered":"2023-11-24T08:19:22.577724+0000","last_unstale":"2023-11-24T08:19:22.582488+0000","last_undegraded":"2023-11-24T08:19:22.582488+0000","last_fullsized":"2023-11-24T08:19:22.582488+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:27.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586925+0000","last_change":"2023-11-24T08:19:22.586219+0000","last_active":"2023-11-24T08:19:23.586925+0000","last_peered":"2023-11-24T08:19:23.586925+0000","last_clean":"2023-11-24T08:19:23.586925+0000","last_became_active":"2023-11-24T08:19:22.585896+0000","last_became_peered":"2023-11-24T08:19:22.585896+0000","last_unstale":"2023-11-24T08:19:23.586925+0000","last_undegraded":"2023-11-24T08:19:23.586925+0000","last_fullsized":"2023-11-24T08:19:23.586925+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:49.298452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.587207+0000","last_change":"2023-11-24T08:19:22.581015+0000","last_active":"2023-11-24T08:19:22.587207+0000","last_peered":"2023-11-24T08:19:22.587207+0000","last_clean":"2023-11-24T08:19:22.587207+0000","last_became_active":"2023-11-24T08:19:22.580379+0000","last_became_peered":"2023-11-24T08:19:22.580379+0000","last_unstale":"2023-11-24T08:19:22.587207+0000","last_undegraded":"2023-11-24T08:19:22.587207+0000","last_fullsized":"2023-11-24T08:19:22.587207+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:49.298452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.576574+0000","last_change":"2023-11-24T08:19:22.573391+0000","last_active":"2023-11-24T08:19:22.576574+0000","last_peered":"2023-11-24T08:19:22.576574+0000","last_clean":"2023-11-24T08:19:22.576574+0000","last_became_active":"2023-11-24T08:19:22.572874+0000","last_became_peered":"2023-11-24T08:19:22.572874+0000","last_unstale":"2023-11-24T08:19:22.576574+0000","last_undegraded":"2023-11-24T08:19:22.576574+0000","last_fullsized":"2023-11-24T08:19:22.576574+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:54.912422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591445+0000","last_change":"2023-11-24T08:19:22.587690+0000","last_active":"2023-11-24T08:19:22.591445+0000","last_peered":"2023-11-24T08:19:22.591445+0000","last_clean":"2023-11-24T08:19:22.591445+0000","last_became_active":"2023-11-24T08:19:22.587362+0000","last_became_peered":"2023-11-24T08:19:22.587362+0000","last_unstale":"2023-11-24T08:19:22.591445+0000","last_undegraded":"2023-11-24T08:19:22.591445+0000","last_fullsized":"2023-11-24T08:19:22.591445+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:03.587373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584517+0000","last_change":"2023-11-24T08:19:22.579492+0000","last_active":"2023-11-24T08:19:22.584517+0000","last_peered":"2023-11-24T08:19:22.584517+0000","last_clean":"2023-11-24T08:19:22.584517+0000","last_became_active":"2023-11-24T08:19:22.579152+0000","last_became_peered":"2023-11-24T08:19:22.579152+0000","last_unstale":"2023-11-24T08:19:22.584517+0000","last_undegraded":"2023-11-24T08:19:22.584517+0000","last_fullsized":"2023-11-24T08:19:22.584517+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:33.628765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600210+0000","last_change":"2023-11-24T08:19:22.594982+0000","last_active":"2023-11-24T08:19:22.600210+0000","last_peered":"2023-11-24T08:19:22.600210+0000","last_clean":"2023-11-24T08:19:22.600210+0000","last_became_active":"2023-11-24T08:19:22.594362+0000","last_became_peered":"2023-11-24T08:19:22.594362+0000","last_unstale":"2023-11-24T08:19:22.600210+0000","last_undegraded":"2023-11-24T08:19:22.600210+0000","last_fullsized":"2023-11-24T08:19:22.600210+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:03.374509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.601175+0000","last_change":"2023-11-24T08:19:22.589325+0000","last_active":"2023-11-24T08:19:22.601175+0000","last_peered":"2023-11-24T08:19:22.601175+0000","last_clean":"2023-11-24T08:19:22.601175+0000","last_became_active":"2023-11-24T08:19:22.588243+0000","last_became_peered":"2023-11-24T08:19:22.588243+0000","last_unstale":"2023-11-24T08:19:22.601175+0000","last_undegraded":"2023-11-24T08:19:22.601175+0000","last_fullsized":"2023-11-24T08:19:22.601175+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:19.415043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599410+0000","last_change":"2023-11-24T08:19:22.589823+0000","last_active":"2023-11-24T08:19:22.599410+0000","last_peered":"2023-11-24T08:19:22.599410+0000","last_clean":"2023-11-24T08:19:22.599410+0000","last_became_active":"2023-11-24T08:19:22.589536+0000","last_became_peered":"2023-11-24T08:19:22.589536+0000","last_unstale":"2023-11-24T08:19:22.599410+0000","last_undegraded":"2023-11-24T08:19:22.599410+0000","last_fullsized":"2023-11-24T08:19:22.599410+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:12:03.848491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.594324+0000","last_change":"2023-11-24T08:19:22.586931+0000","last_active":"2023-11-24T08:19:22.594324+0000","last_peered":"2023-11-24T08:19:22.594324+0000","last_clean":"2023-11-24T08:19:22.594324+0000","last_became_active":"2023-11-24T08:19:22.586390+0000","last_became_peered":"2023-11-24T08:19:22.586390+0000","last_unstale":"2023-11-24T08:19:22.594324+0000","last_undegraded":"2023-11-24T08:19:22.594324+0000","last_fullsized":"2023-11-24T08:19:22.594324+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:56.999101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599320+0000","last_change":"2023-11-24T08:19:22.591765+0000","last_active":"2023-11-24T08:19:22.599320+0000","last_peered":"2023-11-24T08:19:22.599320+0000","last_clean":"2023-11-24T08:19:22.599320+0000","last_became_active":"2023-11-24T08:19:22.591459+0000","last_became_peered":"2023-11-24T08:19:22.591459+0000","last_unstale":"2023-11-24T08:19:22.599320+0000","last_undegraded":"2023-11-24T08:19:22.599320+0000","last_fullsized":"2023-11-24T08:19:22.599320+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:57.500978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.587383+0000","last_change":"2023-11-24T08:19:22.581944+0000","last_active":"2023-11-24T08:19:22.587383+0000","last_peered":"2023-11-24T08:19:22.587383+0000","last_clean":"2023-11-24T08:19:22.587383+0000","last_became_active":"2023-11-24T08:19:22.581661+0000","last_became_peered":"2023-11-24T08:19:22.581661+0000","last_unstale":"2023-11-24T08:19:22.587383+0000","last_undegraded":"2023-11-24T08:19:22.587383+0000","last_fullsized":"2023-11-24T08:19:22.587383+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:05.125474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.595458+0000","last_change":"2023-11-24T08:19:22.592067+0000","last_active":"2023-11-24T08:19:22.595458+0000","last_peered":"2023-11-24T08:19:22.595458+0000","last_clean":"2023-11-24T08:19:22.595458+0000","last_became_active":"2023-11-24T08:19:22.591756+0000","last_became_peered":"2023-11-24T08:19:22.591756+0000","last_unstale":"2023-11-24T08:19:22.595458+0000","last_undegraded":"2023-11-24T08:19:22.595458+0000","last_fullsized":"2023-11-24T08:19:22.595458+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:15:24.287481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590967+0000","last_change":"2023-11-24T08:19:22.591985+0000","last_active":"2023-11-24T08:19:23.590967+0000","last_peered":"2023-11-24T08:19:23.590967+0000","last_clean":"2023-11-24T08:19:23.590967+0000","last_became_active":"2023-11-24T08:19:22.591697+0000","last_became_peered":"2023-11-24T08:19:22.591697+0000","last_unstale":"2023-11-24T08:19:23.590967+0000","last_undegraded":"2023-11-24T08:19:23.590967+0000","last_fullsized":"2023-11-24T08:19:23.590967+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:51:38.735044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.596280+0000","last_change":"2023-11-24T08:19:22.588307+0000","last_active":"2023-11-24T08:19:22.596280+0000","last_peered":"2023-11-24T08:19:22.596280+0000","last_clean":"2023-11-24T08:19:22.596280+0000","last_became_active":"2023-11-24T08:19:22.586440+0000","last_became_peered":"2023-11-24T08:19:22.586440+0000","last_unstale":"2023-11-24T08:19:22.596280+0000","last_undegraded":"2023-11-24T08:19:22.596280+0000","last_fullsized":"2023-11-24T08:19:22.596280+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:50.869823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604696+0000","last_change":"2023-11-24T08:19:22.597233+0000","last_active":"2023-11-24T08:19:22.604696+0000","last_peered":"2023-11-24T08:19:22.604696+0000","last_clean":"2023-11-24T08:19:22.604696+0000","last_became_active":"2023-11-24T08:19:22.596960+0000","last_became_peered":"2023-11-24T08:19:22.596960+0000","last_unstale":"2023-11-24T08:19:22.604696+0000","last_undegraded":"2023-11-24T08:19:22.604696+0000","last_fullsized":"2023-11-24T08:19:22.604696+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:55.587615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.589882+0000","last_change":"2023-11-24T08:19:22.586967+0000","last_active":"2023-11-24T08:19:22.589882+0000","last_peered":"2023-11-24T08:19:22.589882+0000","last_clean":"2023-11-24T08:19:22.589882+0000","last_became_active":"2023-11-24T08:19:22.586710+0000","last_became_peered":"2023-11-24T08:19:22.586710+0000","last_unstale":"2023-11-24T08:19:22.589882+0000","last_undegraded":"2023-11-24T08:19:22.589882+0000","last_fullsized":"2023-11-24T08:19:22.589882+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:16:02.650805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587217+0000","last_change":"2023-11-24T08:19:22.578847+0000","last_active":"2023-11-24T08:19:23.587217+0000","last_peered":"2023-11-24T08:19:23.587217+0000","last_clean":"2023-11-24T08:19:23.587217+0000","last_became_active":"2023-11-24T08:19:22.578532+0000","last_became_peered":"2023-11-24T08:19:22.578532+0000","last_unstale":"2023-11-24T08:19:23.587217+0000","last_undegraded":"2023-11-24T08:19:23.587217+0000","last_fullsized":"2023-11-24T08:19:23.587217+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.591671+0000","last_change":"2023-11-24T08:19:22.581063+0000","last_active":"2023-11-24T08:19:22.591671+0000","last_peered":"2023-11-24T08:19:22.591671+0000","last_clean":"2023-11-24T08:19:22.591671+0000","last_became_active":"2023-11-24T08:19:22.580698+0000","last_became_peered":"2023-11-24T08:19:22.580698+0000","last_unstale":"2023-11-24T08:19:22.591671+0000","last_undegraded":"2023-11-24T08:19:22.591671+0000","last_fullsized":"2023-11-24T08:19:22.591671+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:52.600814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586129+0000","last_change":"2023-11-24T08:19:22.582313+0000","last_active":"2023-11-24T08:19:22.586129+0000","last_peered":"2023-11-24T08:19:22.586129+0000","last_clean":"2023-11-24T08:19:22.586129+0000","last_became_active":"2023-11-24T08:19:22.581987+0000","last_became_peered":"2023-11-24T08:19:22.581987+0000","last_unstale":"2023-11-24T08:19:22.586129+0000","last_undegraded":"2023-11-24T08:19:22.586129+0000","last_fullsized":"2023-11-24T08:19:22.586129+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:45:34.709747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.585564+0000","last_change":"2023-11-24T08:19:22.581629+0000","last_active":"2023-11-24T08:19:22.585564+0000","last_peered":"2023-11-24T08:19:22.585564+0000","last_clean":"2023-11-24T08:19:22.585564+0000","last_became_active":"2023-11-24T08:19:22.580906+0000","last_became_peered":"2023-11-24T08:19:22.580906+0000","last_unstale":"2023-11-24T08:19:22.585564+0000","last_undegraded":"2023-11-24T08:19:22.585564+0000","last_fullsized":"2023-11-24T08:19:22.585564+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:02:06.713002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"174'4","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604836+0000","last_change":"2023-11-24T08:19:21.551814+0000","last_active":"2023-11-24T08:19:22.604836+0000","last_peered":"2023-11-24T08:19:22.604836+0000","last_clean":"2023-11-24T08:19:22.604836+0000","last_became_active":"2023-11-24T08:19:21.551310+0000","last_became_peered":"2023-11-24T08:19:21.551310+0000","last_unstale":"2023-11-24T08:19:22.604836+0000","last_undegraded":"2023-11-24T08:19:22.604836+0000","last_fullsized":"2023-11-24T08:19:22.604836+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:53:33.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602360+0000","last_change":"2023-11-24T08:19:22.597227+0000","last_active":"2023-11-24T08:19:22.602360+0000","last_peered":"2023-11-24T08:19:22.602360+0000","last_clean":"2023-11-24T08:19:22.602360+0000","last_became_active":"2023-11-24T08:19:22.596614+0000","last_became_peered":"2023-11-24T08:19:22.596614+0000","last_unstale":"2023-11-24T08:19:22.602360+0000","last_undegraded":"2023-11-24T08:19:22.602360+0000","last_fullsized":"2023-11-24T08:19:22.602360+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:16:55.461611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588421+0000","last_change":"2023-11-24T08:19:21.552633+0000","last_active":"2023-11-24T08:19:23.588421+0000","last_peered":"2023-11-24T08:19:23.588421+0000","last_clean":"2023-11-24T08:19:23.588421+0000","last_became_active":"2023-11-24T08:19:21.551810+0000","last_became_peered":"2023-11-24T08:19:21.551810+0000","last_unstale":"2023-11-24T08:19:23.588421+0000","last_undegraded":"2023-11-24T08:19:23.588421+0000","last_fullsized":"2023-11-24T08:19:23.588421+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:49.859364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.591113+0000","last_change":"2023-11-24T08:19:22.585074+0000","last_active":"2023-11-24T08:19:23.591113+0000","last_peered":"2023-11-24T08:19:23.591113+0000","last_clean":"2023-11-24T08:19:23.591113+0000","last_became_active":"2023-11-24T08:19:22.584764+0000","last_became_peered":"2023-11-24T08:19:22.584764+0000","last_unstale":"2023-11-24T08:19:23.591113+0000","last_undegraded":"2023-11-24T08:19:23.591113+0000","last_fullsized":"2023-11-24T08:19:23.591113+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:34.659045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.608099+0000","last_change":"2023-11-24T08:19:21.559329+0000","last_active":"2023-11-24T08:19:22.608099+0000","last_peered":"2023-11-24T08:19:22.608099+0000","last_clean":"2023-11-24T08:19:22.608099+0000","last_became_active":"2023-11-24T08:19:21.558165+0000","last_became_peered":"2023-11-24T08:19:21.558165+0000","last_unstale":"2023-11-24T08:19:22.608099+0000","last_undegraded":"2023-11-24T08:19:22.608099+0000","last_fullsized":"2023-11-24T08:19:22.608099+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:33.923138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588267+0000","last_change":"2023-11-24T08:19:22.571466+0000","last_active":"2023-11-24T08:19:23.588267+0000","last_peered":"2023-11-24T08:19:23.588267+0000","last_clean":"2023-11-24T08:19:23.588267+0000","last_became_active":"2023-11-24T08:19:22.570217+0000","last_became_peered":"2023-11-24T08:19:22.570217+0000","last_unstale":"2023-11-24T08:19:23.588267+0000","last_undegraded":"2023-11-24T08:19:23.588267+0000","last_fullsized":"2023-11-24T08:19:23.588267+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607786+0000","last_change":"2023-11-24T08:19:21.559519+0000","last_active":"2023-11-24T08:19:22.607786+0000","last_peered":"2023-11-24T08:19:22.607786+0000","last_clean":"2023-11-24T08:19:22.607786+0000","last_became_active":"2023-11-24T08:19:21.558830+0000","last_became_peered":"2023-11-24T08:19:21.558830+0000","last_unstale":"2023-11-24T08:19:22.607786+0000","last_undegraded":"2023-11-24T08:19:22.607786+0000","last_fullsized":"2023-11-24T08:19:22.607786+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:46:49.859364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.594303+0000","last_change":"2023-11-24T08:19:22.586739+0000","last_active":"2023-11-24T08:19:22.594303+0000","last_peered":"2023-11-24T08:19:22.594303+0000","last_clean":"2023-11-24T08:19:22.594303+0000","last_became_active":"2023-11-24T08:19:22.586009+0000","last_became_peered":"2023-11-24T08:19:22.586009+0000","last_unstale":"2023-11-24T08:19:22.594303+0000","last_undegraded":"2023-11-24T08:19:22.594303+0000","last_fullsized":"2023-11-24T08:19:22.594303+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:21:59.002683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.605850+0000","last_change":"2023-11-24T08:19:21.551868+0000","last_active":"2023-11-24T08:19:22.605850+0000","last_peered":"2023-11-24T08:19:22.605850+0000","last_clean":"2023-11-24T08:19:22.605850+0000","last_became_active":"2023-11-24T08:19:21.551442+0000","last_became_peered":"2023-11-24T08:19:21.551442+0000","last_unstale":"2023-11-24T08:19:22.605850+0000","last_undegraded":"2023-11-24T08:19:22.605850+0000","last_fullsized":"2023-11-24T08:19:22.605850+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:51:49.262584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.575866+0000","last_change":"2023-11-24T08:19:22.572998+0000","last_active":"2023-11-24T08:19:22.575866+0000","last_peered":"2023-11-24T08:19:22.575866+0000","last_clean":"2023-11-24T08:19:22.575866+0000","last_became_active":"2023-11-24T08:19:22.572702+0000","last_became_peered":"2023-11-24T08:19:22.572702+0000","last_unstale":"2023-11-24T08:19:22.575866+0000","last_undegraded":"2023-11-24T08:19:22.575866+0000","last_fullsized":"2023-11-24T08:19:22.575866+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:38.041778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602762+0000","last_change":"2023-11-24T08:19:21.556229+0000","last_active":"2023-11-24T08:19:22.602762+0000","last_peered":"2023-11-24T08:19:22.602762+0000","last_clean":"2023-11-24T08:19:22.602762+0000","last_became_active":"2023-11-24T08:19:21.555580+0000","last_became_peered":"2023-11-24T08:19:21.555580+0000","last_unstale":"2023-11-24T08:19:22.602762+0000","last_undegraded":"2023-11-24T08:19:22.602762+0000","last_fullsized":"2023-11-24T08:19:22.602762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:52.380299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590191+0000","last_change":"2023-11-24T08:19:22.587093+0000","last_active":"2023-11-24T08:19:23.590191+0000","last_peered":"2023-11-24T08:19:23.590191+0000","last_clean":"2023-11-24T08:19:23.590191+0000","last_became_active":"2023-11-24T08:19:22.586307+0000","last_became_peered":"2023-11-24T08:19:22.586307+0000","last_unstale":"2023-11-24T08:19:23.590191+0000","last_undegraded":"2023-11-24T08:19:23.590191+0000","last_fullsized":"2023-11-24T08:19:23.590191+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:17.005243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.570086+0000","last_change":"2023-11-24T08:19:21.561585+0000","last_active":"2023-11-24T08:19:22.570086+0000","last_peered":"2023-11-24T08:19:22.570086+0000","last_clean":"2023-11-24T08:19:22.570086+0000","last_became_active":"2023-11-24T08:19:21.561358+0000","last_became_peered":"2023-11-24T08:19:21.561358+0000","last_unstale":"2023-11-24T08:19:22.570086+0000","last_undegraded":"2023-11-24T08:19:22.570086+0000","last_fullsized":"2023-11-24T08:19:22.570086+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:46.397173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.590793+0000","last_change":"2023-11-24T08:19:22.581938+0000","last_active":"2023-11-24T08:19:22.590793+0000","last_peered":"2023-11-24T08:19:22.590793+0000","last_clean":"2023-11-24T08:19:22.590793+0000","last_became_active":"2023-11-24T08:19:22.580866+0000","last_became_peered":"2023-11-24T08:19:22.580866+0000","last_unstale":"2023-11-24T08:19:22.590793+0000","last_undegraded":"2023-11-24T08:19:22.590793+0000","last_fullsized":"2023-11-24T08:19:22.590793+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:51:38.735044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590986+0000","last_change":"2023-11-24T08:19:21.555108+0000","last_active":"2023-11-24T08:19:23.590986+0000","last_peered":"2023-11-24T08:19:23.590986+0000","last_clean":"2023-11-24T08:19:23.590986+0000","last_became_active":"2023-11-24T08:19:21.553150+0000","last_became_peered":"2023-11-24T08:19:21.553150+0000","last_unstale":"2023-11-24T08:19:23.590986+0000","last_undegraded":"2023-11-24T08:19:23.590986+0000","last_fullsized":"2023-11-24T08:19:23.590986+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:26.656730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599277+0000","last_change":"2023-11-24T08:19:22.597246+0000","last_active":"2023-11-24T08:19:22.599277+0000","last_peered":"2023-11-24T08:19:22.599277+0000","last_clean":"2023-11-24T08:19:22.599277+0000","last_became_active":"2023-11-24T08:19:22.596938+0000","last_became_peered":"2023-11-24T08:19:22.596938+0000","last_unstale":"2023-11-24T08:19:22.599277+0000","last_undegraded":"2023-11-24T08:19:22.599277+0000","last_fullsized":"2023-11-24T08:19:22.599277+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:46.486016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.575241+0000","last_change":"2023-11-24T08:19:21.561530+0000","last_active":"2023-11-24T08:19:22.575241+0000","last_peered":"2023-11-24T08:19:22.575241+0000","last_clean":"2023-11-24T08:19:22.575241+0000","last_became_active":"2023-11-24T08:19:21.560527+0000","last_became_peered":"2023-11-24T08:19:21.560527+0000","last_unstale":"2023-11-24T08:19:22.575241+0000","last_undegraded":"2023-11-24T08:19:22.575241+0000","last_fullsized":"2023-11-24T08:19:22.575241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:00:15.010190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590686+0000","last_change":"2023-11-24T08:19:22.580321+0000","last_active":"2023-11-24T08:19:23.590686+0000","last_peered":"2023-11-24T08:19:23.590686+0000","last_clean":"2023-11-24T08:19:23.590686+0000","last_became_active":"2023-11-24T08:19:22.580033+0000","last_became_peered":"2023-11-24T08:19:22.580033+0000","last_unstale":"2023-11-24T08:19:23.590686+0000","last_undegraded":"2023-11-24T08:19:23.590686+0000","last_fullsized":"2023-11-24T08:19:23.590686+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:29.525302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"174'3","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607283+0000","last_change":"2023-11-24T08:19:21.552871+0000","last_active":"2023-11-24T08:19:22.607283+0000","last_peered":"2023-11-24T08:19:22.607283+0000","last_clean":"2023-11-24T08:19:22.607283+0000","last_became_active":"2023-11-24T08:19:21.551875+0000","last_became_peered":"2023-11-24T08:19:21.551875+0000","last_unstale":"2023-11-24T08:19:22.607283+0000","last_undegraded":"2023-11-24T08:19:22.607283+0000","last_fullsized":"2023-11-24T08:19:22.607283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:27.136424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600230+0000","last_change":"2023-11-24T08:19:22.595477+0000","last_active":"2023-11-24T08:19:22.600230+0000","last_peered":"2023-11-24T08:19:22.600230+0000","last_clean":"2023-11-24T08:19:22.600230+0000","last_became_active":"2023-11-24T08:19:22.594719+0000","last_became_peered":"2023-11-24T08:19:22.594719+0000","last_unstale":"2023-11-24T08:19:22.600230+0000","last_undegraded":"2023-11-24T08:19:22.600230+0000","last_fullsized":"2023-11-24T08:19:22.600230+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:49.157351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"174'6","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.611931+0000","last_change":"2023-11-24T08:19:21.556481+0000","last_active":"2023-11-24T08:19:22.611931+0000","last_peered":"2023-11-24T08:19:22.611931+0000","last_clean":"2023-11-24T08:19:22.611931+0000","last_became_active":"2023-11-24T08:19:21.556146+0000","last_became_peered":"2023-11-24T08:19:21.556146+0000","last_unstale":"2023-11-24T08:19:22.611931+0000","last_undegraded":"2023-11-24T08:19:22.611931+0000","last_fullsized":"2023-11-24T08:19:22.611931+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:28.514843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599718+0000","last_change":"2023-11-24T08:19:22.591889+0000","last_active":"2023-11-24T08:19:22.599718+0000","last_peered":"2023-11-24T08:19:22.599718+0000","last_clean":"2023-11-24T08:19:22.599718+0000","last_became_active":"2023-11-24T08:19:22.591486+0000","last_became_peered":"2023-11-24T08:19:22.591486+0000","last_unstale":"2023-11-24T08:19:22.599718+0000","last_undegraded":"2023-11-24T08:19:22.599718+0000","last_fullsized":"2023-11-24T08:19:22.599718+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:26.258294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607593+0000","last_change":"2023-11-24T08:19:21.559312+0000","last_active":"2023-11-24T08:19:22.607593+0000","last_peered":"2023-11-24T08:19:22.607593+0000","last_clean":"2023-11-24T08:19:22.607593+0000","last_became_active":"2023-11-24T08:19:21.559033+0000","last_became_peered":"2023-11-24T08:19:21.559033+0000","last_unstale":"2023-11-24T08:19:22.607593+0000","last_undegraded":"2023-11-24T08:19:22.607593+0000","last_fullsized":"2023-11-24T08:19:22.607593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:04.115015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590586+0000","last_change":"2023-11-24T08:19:22.582930+0000","last_active":"2023-11-24T08:19:23.590586+0000","last_peered":"2023-11-24T08:19:23.590586+0000","last_clean":"2023-11-24T08:19:23.590586+0000","last_became_active":"2023-11-24T08:19:22.582593+0000","last_became_peered":"2023-11-24T08:19:22.582593+0000","last_unstale":"2023-11-24T08:19:23.590586+0000","last_undegraded":"2023-11-24T08:19:23.590586+0000","last_fullsized":"2023-11-24T08:19:23.590586+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:57:25.724700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.581475+0000","last_change":"2023-11-24T08:19:21.547030+0000","last_active":"2023-11-24T08:19:22.581475+0000","last_peered":"2023-11-24T08:19:22.581475+0000","last_clean":"2023-11-24T08:19:22.581475+0000","last_became_active":"2023-11-24T08:19:21.546748+0000","last_became_peered":"2023-11-24T08:19:21.546748+0000","last_unstale":"2023-11-24T08:19:22.581475+0000","last_undegraded":"2023-11-24T08:19:22.581475+0000","last_fullsized":"2023-11-24T08:19:22.581475+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:11.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586163+0000","last_change":"2023-11-24T08:19:22.581222+0000","last_active":"2023-11-24T08:19:23.586163+0000","last_peered":"2023-11-24T08:19:23.586163+0000","last_clean":"2023-11-24T08:19:23.586163+0000","last_became_active":"2023-11-24T08:19:22.580220+0000","last_became_peered":"2023-11-24T08:19:22.580220+0000","last_unstale":"2023-11-24T08:19:23.586163+0000","last_undegraded":"2023-11-24T08:19:23.586163+0000","last_fullsized":"2023-11-24T08:19:23.586163+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:34.933597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.561566+0000","last_change":"2023-11-24T08:19:21.560650+0000","last_active":"2023-11-24T08:19:22.561566+0000","last_peered":"2023-11-24T08:19:22.561566+0000","last_clean":"2023-11-24T08:19:22.561566+0000","last_became_active":"2023-11-24T08:19:21.560390+0000","last_became_peered":"2023-11-24T08:19:21.560390+0000","last_unstale":"2023-11-24T08:19:22.561566+0000","last_undegraded":"2023-11-24T08:19:22.561566+0000","last_fullsized":"2023-11-24T08:19:22.561566+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:27.334358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599077+0000","last_change":"2023-11-24T08:19:22.593347+0000","last_active":"2023-11-24T08:19:22.599077+0000","last_peered":"2023-11-24T08:19:22.599077+0000","last_clean":"2023-11-24T08:19:22.599077+0000","last_became_active":"2023-11-24T08:19:22.590877+0000","last_became_peered":"2023-11-24T08:19:22.590877+0000","last_unstale":"2023-11-24T08:19:22.599077+0000","last_undegraded":"2023-11-24T08:19:22.599077+0000","last_fullsized":"2023-11-24T08:19:22.599077+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:52.844484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"174'8","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607798+0000","last_change":"2023-11-24T08:19:21.557576+0000","last_active":"2023-11-24T08:19:22.607798+0000","last_peered":"2023-11-24T08:19:22.607798+0000","last_clean":"2023-11-24T08:19:22.607798+0000","last_became_active":"2023-11-24T08:19:21.557271+0000","last_became_peered":"2023-11-24T08:19:21.557271+0000","last_unstale":"2023-11-24T08:19:22.607798+0000","last_undegraded":"2023-11-24T08:19:22.607798+0000","last_fullsized":"2023-11-24T08:19:22.607798+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:52.380299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602431+0000","last_change":"2023-11-24T08:19:22.595705+0000","last_active":"2023-11-24T08:19:22.602431+0000","last_peered":"2023-11-24T08:19:22.602431+0000","last_clean":"2023-11-24T08:19:22.602431+0000","last_became_active":"2023-11-24T08:19:22.592584+0000","last_became_peered":"2023-11-24T08:19:22.592584+0000","last_unstale":"2023-11-24T08:19:22.602431+0000","last_undegraded":"2023-11-24T08:19:22.602431+0000","last_fullsized":"2023-11-24T08:19:22.602431+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:10.117362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.604734+0000","last_change":"2023-11-24T08:19:21.556089+0000","last_active":"2023-11-24T08:19:22.604734+0000","last_peered":"2023-11-24T08:19:22.604734+0000","last_clean":"2023-11-24T08:19:22.604734+0000","last_became_active":"2023-11-24T08:19:21.555222+0000","last_became_peered":"2023-11-24T08:19:21.555222+0000","last_unstale":"2023-11-24T08:19:22.604734+0000","last_undegraded":"2023-11-24T08:19:22.604734+0000","last_fullsized":"2023-11-24T08:19:22.604734+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:33.515593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.588717+0000","last_change":"2023-11-24T08:19:22.587097+0000","last_active":"2023-11-24T08:19:23.588717+0000","last_peered":"2023-11-24T08:19:23.588717+0000","last_clean":"2023-11-24T08:19:23.588717+0000","last_became_active":"2023-11-24T08:19:22.585891+0000","last_became_peered":"2023-11-24T08:19:22.585891+0000","last_unstale":"2023-11-24T08:19:23.588717+0000","last_undegraded":"2023-11-24T08:19:23.588717+0000","last_fullsized":"2023-11-24T08:19:23.588717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:18:29.525302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.603237+0000","last_change":"2023-11-24T08:19:21.545458+0000","last_active":"2023-11-24T08:19:22.603237+0000","last_peered":"2023-11-24T08:19:22.603237+0000","last_clean":"2023-11-24T08:19:22.603237+0000","last_became_active":"2023-11-24T08:19:21.544961+0000","last_became_peered":"2023-11-24T08:19:21.544961+0000","last_unstale":"2023-11-24T08:19:22.603237+0000","last_undegraded":"2023-11-24T08:19:22.603237+0000","last_fullsized":"2023-11-24T08:19:22.603237+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:46:17.987011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587785+0000","last_change":"2023-11-24T08:19:22.581976+0000","last_active":"2023-11-24T08:19:23.587785+0000","last_peered":"2023-11-24T08:19:23.587785+0000","last_clean":"2023-11-24T08:19:23.587785+0000","last_became_active":"2023-11-24T08:19:22.581721+0000","last_became_peered":"2023-11-24T08:19:22.581721+0000","last_unstale":"2023-11-24T08:19:23.587785+0000","last_undegraded":"2023-11-24T08:19:23.587785+0000","last_fullsized":"2023-11-24T08:19:23.587785+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:17.005243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"174'1","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607151+0000","last_change":"2023-11-24T08:19:21.556884+0000","last_active":"2023-11-24T08:19:22.607151+0000","last_peered":"2023-11-24T08:19:22.607151+0000","last_clean":"2023-11-24T08:19:22.607151+0000","last_became_active":"2023-11-24T08:19:21.556621+0000","last_became_peered":"2023-11-24T08:19:21.556621+0000","last_unstale":"2023-11-24T08:19:22.607151+0000","last_undegraded":"2023-11-24T08:19:22.607151+0000","last_fullsized":"2023-11-24T08:19:22.607151+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:53.901963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588648+0000","last_change":"2023-11-24T08:19:22.584792+0000","last_active":"2023-11-24T08:19:22.588648+0000","last_peered":"2023-11-24T08:19:22.588648+0000","last_clean":"2023-11-24T08:19:22.588648+0000","last_became_active":"2023-11-24T08:19:22.584517+0000","last_became_peered":"2023-11-24T08:19:22.584517+0000","last_unstale":"2023-11-24T08:19:22.588648+0000","last_undegraded":"2023-11-24T08:19:22.588648+0000","last_fullsized":"2023-11-24T08:19:22.588648+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:19:57.500978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.590920+0000","last_change":"2023-11-24T08:19:21.547542+0000","last_active":"2023-11-24T08:19:23.590920+0000","last_peered":"2023-11-24T08:19:23.590920+0000","last_clean":"2023-11-24T08:19:23.590920+0000","last_became_active":"2023-11-24T08:19:21.545070+0000","last_became_peered":"2023-11-24T08:19:21.545070+0000","last_unstale":"2023-11-24T08:19:23.590920+0000","last_undegraded":"2023-11-24T08:19:23.590920+0000","last_fullsized":"2023-11-24T08:19:23.590920+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:01:33.923138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.600019+0000","last_change":"2023-11-24T08:19:22.594630+0000","last_active":"2023-11-24T08:19:22.600019+0000","last_peered":"2023-11-24T08:19:22.600019+0000","last_clean":"2023-11-24T08:19:22.600019+0000","last_became_active":"2023-11-24T08:19:22.594333+0000","last_became_peered":"2023-11-24T08:19:22.594333+0000","last_unstale":"2023-11-24T08:19:22.600019+0000","last_undegraded":"2023-11-24T08:19:22.600019+0000","last_fullsized":"2023-11-24T08:19:22.600019+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:01:14.610966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.607860+0000","last_change":"2023-11-24T08:19:21.559079+0000","last_active":"2023-11-24T08:19:22.607860+0000","last_peered":"2023-11-24T08:19:22.607860+0000","last_clean":"2023-11-24T08:19:22.607860+0000","last_became_active":"2023-11-24T08:19:21.558434+0000","last_became_peered":"2023-11-24T08:19:21.558434+0000","last_unstale":"2023-11-24T08:19:22.607860+0000","last_undegraded":"2023-11-24T08:19:22.607860+0000","last_fullsized":"2023-11-24T08:19:22.607860+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:19.752797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.591048+0000","last_change":"2023-11-24T08:19:22.577393+0000","last_active":"2023-11-24T08:19:23.591048+0000","last_peered":"2023-11-24T08:19:23.591048+0000","last_clean":"2023-11-24T08:19:23.591048+0000","last_became_active":"2023-11-24T08:19:22.577079+0000","last_became_peered":"2023-11-24T08:19:22.577079+0000","last_unstale":"2023-11-24T08:19:23.591048+0000","last_undegraded":"2023-11-24T08:19:23.591048+0000","last_fullsized":"2023-11-24T08:19:23.591048+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:30:35.294595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.566369+0000","last_change":"2023-11-24T08:19:21.561403+0000","last_active":"2023-11-24T08:19:22.566369+0000","last_peered":"2023-11-24T08:19:22.566369+0000","last_clean":"2023-11-24T08:19:22.566369+0000","last_became_active":"2023-11-24T08:19:21.561088+0000","last_became_peered":"2023-11-24T08:19:21.561088+0000","last_unstale":"2023-11-24T08:19:22.566369+0000","last_undegraded":"2023-11-24T08:19:22.566369+0000","last_fullsized":"2023-11-24T08:19:22.566369+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:22:45.521653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588561+0000","last_change":"2023-11-24T08:19:22.582302+0000","last_active":"2023-11-24T08:19:22.588561+0000","last_peered":"2023-11-24T08:19:22.588561+0000","last_clean":"2023-11-24T08:19:22.588561+0000","last_became_active":"2023-11-24T08:19:22.582030+0000","last_became_peered":"2023-11-24T08:19:22.582030+0000","last_unstale":"2023-11-24T08:19:22.588561+0000","last_undegraded":"2023-11-24T08:19:22.588561+0000","last_fullsized":"2023-11-24T08:19:22.588561+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:52.600814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.567383+0000","last_change":"2023-11-24T08:19:21.559637+0000","last_active":"2023-11-24T08:19:22.567383+0000","last_peered":"2023-11-24T08:19:22.567383+0000","last_clean":"2023-11-24T08:19:22.567383+0000","last_became_active":"2023-11-24T08:19:21.559383+0000","last_became_peered":"2023-11-24T08:19:21.559383+0000","last_unstale":"2023-11-24T08:19:22.567383+0000","last_undegraded":"2023-11-24T08:19:22.567383+0000","last_fullsized":"2023-11-24T08:19:22.567383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:48.287993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588170+0000","last_change":"2023-11-24T08:19:22.584963+0000","last_active":"2023-11-24T08:19:22.588170+0000","last_peered":"2023-11-24T08:19:22.588170+0000","last_clean":"2023-11-24T08:19:22.588170+0000","last_became_active":"2023-11-24T08:19:22.584709+0000","last_became_peered":"2023-11-24T08:19:22.584709+0000","last_unstale":"2023-11-24T08:19:22.588170+0000","last_undegraded":"2023-11-24T08:19:22.588170+0000","last_fullsized":"2023-11-24T08:19:22.588170+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:08:06.456345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.586334+0000","last_change":"2023-11-24T08:19:21.557719+0000","last_active":"2023-11-24T08:19:23.586334+0000","last_peered":"2023-11-24T08:19:23.586334+0000","last_clean":"2023-11-24T08:19:23.586334+0000","last_became_active":"2023-11-24T08:19:21.557450+0000","last_became_peered":"2023-11-24T08:19:21.557450+0000","last_unstale":"2023-11-24T08:19:23.586334+0000","last_undegraded":"2023-11-24T08:19:23.586334+0000","last_fullsized":"2023-11-24T08:19:23.586334+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:33.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.586041+0000","last_change":"2023-11-24T08:19:22.582505+0000","last_active":"2023-11-24T08:19:22.586041+0000","last_peered":"2023-11-24T08:19:22.586041+0000","last_clean":"2023-11-24T08:19:22.586041+0000","last_became_active":"2023-11-24T08:19:22.582227+0000","last_became_peered":"2023-11-24T08:19:22.582227+0000","last_unstale":"2023-11-24T08:19:22.586041+0000","last_undegraded":"2023-11-24T08:19:22.586041+0000","last_fullsized":"2023-11-24T08:19:22.586041+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:19:59.879051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"174'2","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.606515+0000","last_change":"2023-11-24T08:19:21.554785+0000","last_active":"2023-11-24T08:19:22.606515+0000","last_peered":"2023-11-24T08:19:22.606515+0000","last_clean":"2023-11-24T08:19:22.606515+0000","last_became_active":"2023-11-24T08:19:21.553819+0000","last_became_peered":"2023-11-24T08:19:21.553819+0000","last_unstale":"2023-11-24T08:19:22.606515+0000","last_undegraded":"2023-11-24T08:19:22.606515+0000","last_fullsized":"2023-11-24T08:19:22.606515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:58:48.283067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.602725+0000","last_change":"2023-11-24T08:19:22.598143+0000","last_active":"2023-11-24T08:19:22.602725+0000","last_peered":"2023-11-24T08:19:22.602725+0000","last_clean":"2023-11-24T08:19:22.602725+0000","last_became_active":"2023-11-24T08:19:22.597872+0000","last_became_peered":"2023-11-24T08:19:22.597872+0000","last_unstale":"2023-11-24T08:19:22.602725+0000","last_undegraded":"2023-11-24T08:19:22.602725+0000","last_fullsized":"2023-11-24T08:19:22.602725+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:27:29.053204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584046+0000","last_change":"2023-11-24T08:19:21.560805+0000","last_active":"2023-11-24T08:19:22.584046+0000","last_peered":"2023-11-24T08:19:22.584046+0000","last_clean":"2023-11-24T08:19:22.584046+0000","last_became_active":"2023-11-24T08:19:21.560053+0000","last_became_peered":"2023-11-24T08:19:21.560053+0000","last_unstale":"2023-11-24T08:19:22.584046+0000","last_undegraded":"2023-11-24T08:19:22.584046+0000","last_fullsized":"2023-11-24T08:19:22.584046+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:47.137624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588540+0000","last_change":"2023-11-24T08:19:22.584089+0000","last_active":"2023-11-24T08:19:22.588540+0000","last_peered":"2023-11-24T08:19:22.588540+0000","last_clean":"2023-11-24T08:19:22.588540+0000","last_became_active":"2023-11-24T08:19:22.583819+0000","last_became_peered":"2023-11-24T08:19:22.583819+0000","last_unstale":"2023-11-24T08:19:22.588540+0000","last_undegraded":"2023-11-24T08:19:22.588540+0000","last_fullsized":"2023-11-24T08:19:22.588540+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:22:30.395795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.582352+0000","last_change":"2023-11-24T08:19:21.559892+0000","last_active":"2023-11-24T08:19:22.582352+0000","last_peered":"2023-11-24T08:19:22.582352+0000","last_clean":"2023-11-24T08:19:22.582352+0000","last_became_active":"2023-11-24T08:19:21.558640+0000","last_became_peered":"2023-11-24T08:19:21.558640+0000","last_unstale":"2023-11-24T08:19:22.582352+0000","last_undegraded":"2023-11-24T08:19:22.582352+0000","last_fullsized":"2023-11-24T08:19:22.582352+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:04:23.288804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.585624+0000","last_change":"2023-11-24T08:19:22.581025+0000","last_active":"2023-11-24T08:19:22.585624+0000","last_peered":"2023-11-24T08:19:22.585624+0000","last_clean":"2023-11-24T08:19:22.585624+0000","last_became_active":"2023-11-24T08:19:22.580698+0000","last_became_peered":"2023-11-24T08:19:22.580698+0000","last_unstale":"2023-11-24T08:19:22.585624+0000","last_undegraded":"2023-11-24T08:19:22.585624+0000","last_fullsized":"2023-11-24T08:19:22.585624+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:23:17.425447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.563255+0000","last_change":"2023-11-24T08:19:21.550954+0000","last_active":"2023-11-24T08:19:22.563255+0000","last_peered":"2023-11-24T08:19:22.563255+0000","last_clean":"2023-11-24T08:19:22.563255+0000","last_became_active":"2023-11-24T08:19:21.550695+0000","last_became_peered":"2023-11-24T08:19:21.550695+0000","last_unstale":"2023-11-24T08:19:22.563255+0000","last_undegraded":"2023-11-24T08:19:22.563255+0000","last_fullsized":"2023-11-24T08:19:22.563255+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:34.871855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584658+0000","last_change":"2023-11-24T08:19:22.581831+0000","last_active":"2023-11-24T08:19:22.584658+0000","last_peered":"2023-11-24T08:19:22.584658+0000","last_clean":"2023-11-24T08:19:22.584658+0000","last_became_active":"2023-11-24T08:19:22.581556+0000","last_became_peered":"2023-11-24T08:19:22.581556+0000","last_unstale":"2023-11-24T08:19:22.584658+0000","last_undegraded":"2023-11-24T08:19:22.584658+0000","last_fullsized":"2023-11-24T08:19:22.584658+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:49.386749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587555+0000","last_change":"2023-11-24T08:19:21.556655+0000","last_active":"2023-11-24T08:19:23.587555+0000","last_peered":"2023-11-24T08:19:23.587555+0000","last_clean":"2023-11-24T08:19:23.587555+0000","last_became_active":"2023-11-24T08:19:21.556351+0000","last_became_peered":"2023-11-24T08:19:21.556351+0000","last_unstale":"2023-11-24T08:19:23.587555+0000","last_undegraded":"2023-11-24T08:19:23.587555+0000","last_fullsized":"2023-11-24T08:19:23.587555+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:06.649697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.584584+0000","last_change":"2023-11-24T08:19:22.581463+0000","last_active":"2023-11-24T08:19:22.584584+0000","last_peered":"2023-11-24T08:19:22.584584+0000","last_clean":"2023-11-24T08:19:22.584584+0000","last_became_active":"2023-11-24T08:19:22.581163+0000","last_became_peered":"2023-11-24T08:19:22.581163+0000","last_unstale":"2023-11-24T08:19:22.584584+0000","last_undegraded":"2023-11-24T08:19:22.584584+0000","last_fullsized":"2023-11-24T08:19:22.584584+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:46.486016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-11-24T08:19:23.587552+0000","last_change":"2023-11-24T08:19:21.557321+0000","last_active":"2023-11-24T08:19:23.587552+0000","last_peered":"2023-11-24T08:19:23.587552+0000","last_clean":"2023-11-24T08:19:23.587552+0000","last_became_active":"2023-11-24T08:19:21.556639+0000","last_became_peered":"2023-11-24T08:19:21.556639+0000","last_unstale":"2023-11-24T08:19:23.587552+0000","last_undegraded":"2023-11-24T08:19:23.587552+0000","last_fullsized":"2023-11-24T08:19:23.587552+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:16:01.640346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.588581+0000","last_change":"2023-11-24T08:19:22.583991+0000","last_active":"2023-11-24T08:19:22.588581+0000","last_peered":"2023-11-24T08:19:22.588581+0000","last_clean":"2023-11-24T08:19:22.588581+0000","last_became_active":"2023-11-24T08:19:22.583601+0000","last_became_peered":"2023-11-24T08:19:22.583601+0000","last_unstale":"2023-11-24T08:19:22.588581+0000","last_undegraded":"2023-11-24T08:19:22.588581+0000","last_fullsized":"2023-11-24T08:19:22.588581+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:20:07.660157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.572098+0000","last_change":"2023-11-24T08:19:21.558779+0000","last_active":"2023-11-24T08:19:22.572098+0000","last_peered":"2023-11-24T08:19:22.572098+0000","last_clean":"2023-11-24T08:19:22.572098+0000","last_became_active":"2023-11-24T08:19:21.558494+0000","last_became_peered":"2023-11-24T08:19:21.558494+0000","last_unstale":"2023-11-24T08:19:22.572098+0000","last_undegraded":"2023-11-24T08:19:22.572098+0000","last_fullsized":"2023-11-24T08:19:22.572098+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:25.247835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.596152+0000","last_change":"2023-11-24T08:19:22.589444+0000","last_active":"2023-11-24T08:19:22.596152+0000","last_peered":"2023-11-24T08:19:22.596152+0000","last_clean":"2023-11-24T08:19:22.596152+0000","last_became_active":"2023-11-24T08:19:22.589124+0000","last_became_peered":"2023-11-24T08:19:22.589124+0000","last_unstale":"2023-11-24T08:19:22.596152+0000","last_undegraded":"2023-11-24T08:19:22.596152+0000","last_fullsized":"2023-11-24T08:19:22.596152+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:02:57.941975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.568326+0000","last_change":"2023-11-24T08:19:21.555464+0000","last_active":"2023-11-24T08:19:22.568326+0000","last_peered":"2023-11-24T08:19:22.568326+0000","last_clean":"2023-11-24T08:19:22.568326+0000","last_became_active":"2023-11-24T08:19:21.555164+0000","last_became_peered":"2023-11-24T08:19:21.555164+0000","last_unstale":"2023-11-24T08:19:22.568326+0000","last_undegraded":"2023-11-24T08:19:22.568326+0000","last_fullsized":"2023-11-24T08:19:22.568326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:44:15.994784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.599037+0000","last_change":"2023-11-24T08:19:22.593765+0000","last_active":"2023-11-24T08:19:22.599037+0000","last_peered":"2023-11-24T08:19:22.599037+0000","last_clean":"2023-11-24T08:19:22.599037+0000","last_became_active":"2023-11-24T08:19:22.593449+0000","last_became_peered":"2023-11-24T08:19:22.593449+0000","last_unstale":"2023-11-24T08:19:22.599037+0000","last_undegraded":"2023-11-24T08:19:22.599037+0000","last_fullsized":"2023-11-24T08:19:22.599037+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:21.515708+0000","last_clean_scrub_stamp":"2023-11-24T08:19:21.515708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:30:37.347675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-11-24T08:19:22.568201+0000","last_change":"2023-11-24T08:19:21.560369+0000","last_active":"2023-11-24T08:19:22.568201+0000","last_peered":"2023-11-24T08:19:22.568201+0000","last_clean":"2023-11-24T08:19:22.568201+0000","last_became_active":"2023-11-24T08:19:21.560074+0000","last_became_peered":"2023-11-24T08:19:21.560074+0000","last_unstale":"2023-11-24T08:19:22.568201+0000","last_undegraded":"2023-11-24T08:19:22.568201+0000","last_fullsized":"2023-11-24T08:19:22.568201+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:19:20.505248+0000","last_clean_scrub_stamp":"2023-11-24T08:19:20.505248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:35.139730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":302,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-11-24T08:19:21.546680+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:19:21.546680+0000","last_peered":"2023-11-24T08:19:21.546680+0000","last_clean":"2023-11-24T08:19:21.546680+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:19:21.546680+0000","last_undegraded":"2023-11-24T08:19:21.546680+0000","last_fullsized":"2023-11-24T08:19:21.546680+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575028,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38068,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333772,"statfs":{"total":96636764160,"available":96597782528,"internally_reserved":0,"allocated":2789376,"data_stored":1554044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36174284},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.042,"15min":1.0529999999999999},"min":{"1min":0.70499999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.601,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.208,"5min":1.0940000000000001,"15min":1.097},"min":{"1min":0.91500000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.625,"5min":1.774,"15min":2.7109999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1739999999999999,"15min":1.1579999999999999},"min":{"1min":0.77000000000000002,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.595,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":1.135},{"interface":"front","average":{"1min":1.23,"5min":1.21,"15min":1.216},"min":{"1min":0.77600000000000002,"5min":0.66600000000000004,"15min":0.628},"max":{"1min":1.653,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":1.0389999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.111,"15min":1.139},"min":{"1min":0.67500000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4690000000000001,"5min":1.698,"15min":4.7000000000000002},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.28,"5min":1.302,"15min":1.319},"min":{"1min":0.94799999999999995,"5min":0.71799999999999997,"15min":0.60399999999999998},"max":{"1min":1.5509999999999999,"5min":1.889,"15min":4.633},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2290000000000001,"15min":1.2070000000000001},"min":{"1min":0.79100000000000004,"5min":0.60199999999999998,"15min":0.504},"max":{"1min":1.5960000000000001,"5min":1.99,"15min":3.133},"last":1.083},{"interface":"front","average":{"1min":1.333,"5min":1.3640000000000001,"15min":1.375},"min":{"1min":0.879,"5min":0.622,"15min":0.622},"max":{"1min":1.696,"5min":1.9890000000000001,"15min":3.9209999999999998},"last":1.3799999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.381,"15min":1.3560000000000001},"min":{"1min":0.95399999999999996,"5min":0.58799999999999997,"15min":0.54700000000000004},"max":{"1min":1.657,"5min":2.665,"15min":2.851},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":0.89900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.0419999999999998,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.181}]},{"osd":5,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.486,"15min":1.48},"min":{"1min":0.88300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7929999999999999,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":1.522},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.508,"15min":1.498},"min":{"1min":0.94799999999999995,"5min":0.622,"15min":0.622},"max":{"1min":1.988,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.575}]},{"osd":6,"last update":"Fri Nov 24 08:19:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.579,"15min":1.5720000000000001},"min":{"1min":1.0649999999999999,"5min":0.71899999999999997,"15min":0.60199999999999998},"max":{"1min":1.863,"5min":2.702,"15min":2.992},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.5429999999999999,"15min":1.506},"min":{"1min":1.2110000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.4159999999999999,"5min":2.6259999999999999,"15min":3.798},"last":1.466}]}]},{"osd":6,"up_from":13,"seq":55834575027,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37480,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334360,"statfs":{"total":96636764160,"available":96598384640,"internally_reserved":0,"allocated":2187264,"data_stored":963572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0580000000000001,"15min":1.0760000000000001},"min":{"1min":0.65500000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.95},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.085,"15min":1.083},"min":{"1min":0.68400000000000005,"5min":0.63900000000000001,"15min":0.58099999999999996},"max":{"1min":1.494,"5min":1.7370000000000001,"15min":2.2069999999999999},"last":1.0900000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.135,"15min":1.1699999999999999},"min":{"1min":0.72899999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6080000000000001,"5min":1.835,"15min":2.0230000000000001},"last":1.218},{"interface":"front","average":{"1min":1.252,"5min":1.242,"15min":1.2070000000000001},"min":{"1min":0.79100000000000004,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.867,"5min":2.1150000000000002,"15min":2.4039999999999999},"last":1.1930000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0900000000000001,"15min":1.1000000000000001},"min":{"1min":0.71699999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4990000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.284},"min":{"1min":0.86899999999999999,"5min":0.746,"15min":0.60399999999999998},"max":{"1min":1.7,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.359,"15min":1.3500000000000001},"min":{"1min":0.92400000000000004,"5min":0.86199999999999999,"15min":0.71499999999999997},"max":{"1min":1.9650000000000001,"5min":2.2330000000000001,"15min":4.7880000000000003},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2789999999999999,"15min":1.3140000000000001},"min":{"1min":0.94199999999999995,"5min":0.78300000000000003,"15min":0.77900000000000003},"max":{"1min":1.681,"5min":2.194,"15min":4.8319999999999999},"last":1.319}]},{"osd":4,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4450000000000001,"15min":1.4470000000000001},"min":{"1min":1.0469999999999999,"5min":0.89100000000000001,"15min":0.76500000000000001},"max":{"1min":1.857,"5min":2.702,"15min":2.895},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.492,"15min":1.5009999999999999},"min":{"1min":1.0429999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0190000000000001,"5min":2.3210000000000002,"15min":2.8820000000000001},"last":1.266}]},{"osd":5,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.413,"15min":1.4490000000000001},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7450000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.165},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5680000000000001,"15min":1.587},"min":{"1min":1.274,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9590000000000001,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":1.5629999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.615,"15min":1.617},"min":{"1min":1.198,"5min":0.748,"15min":0.748},"max":{"1min":2.0779999999999998,"5min":2.903,"15min":3.0329999999999999},"last":1.466},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5940000000000001,"15min":1.6080000000000001},"min":{"1min":1.292,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.032,"5min":2.649,"15min":2.6890000000000001},"last":1.5049999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575027,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37484,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334356,"statfs":{"total":96636764160,"available":96598380544,"internally_reserved":0,"allocated":2187264,"data_stored":963572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0209999999999999,"15min":1.038},"min":{"1min":0.61599999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5309999999999999,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0780000000000001,"15min":1.1160000000000001},"min":{"1min":0.82799999999999996,"5min":0.66500000000000004,"15min":0.60599999999999998},"max":{"1min":1.5269999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":0.998}]},{"osd":1,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1399999999999999,"15min":1.1339999999999999},"min":{"1min":0.78800000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7350000000000001,"5min":7.258,"15min":7.258},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.76100000000000001,"5min":0.71799999999999997,"15min":0.48699999999999999},"max":{"1min":1.796,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.70699999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1859999999999999,"15min":1.1899999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.51600000000000001},"max":{"1min":1.579,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.214,"15min":1.175},"min":{"1min":0.82299999999999995,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.5089999999999999,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.155}]},{"osd":3,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.179},"min":{"1min":0.74099999999999999,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":2.242,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.3120000000000001,"15min":1.2909999999999999},"min":{"1min":0.91500000000000004,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.71,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.101}]},{"osd":4,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.472,"15min":1.3660000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":2.359,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.53,"15min":1.494},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":1.9610000000000001,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.228}]},{"osd":6,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.4139999999999999,"15min":1.3700000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":2.0830000000000002,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.304},{"interface":"front","average":{"1min":1.498,"5min":1.534,"15min":1.452},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":2.452,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.522}]},{"osd":7,"last update":"Fri Nov 24 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5580000000000001,"15min":1.5169999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":1.8220000000000001,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.7589999999999999,"15min":1.6439999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.7240000000000002,"5min":19.663,"15min":19.663},"last":1.5860000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2203648,"data_stored":965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0129999999999999,"15min":1.0680000000000001},"min":{"1min":0.70799999999999996,"5min":0.626,"15min":0.42699999999999999},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.52},{"interface":"front","average":{"1min":1.034,"5min":1.0900000000000001,"15min":1.079},"min":{"1min":0.75900000000000001,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.591,"5min":5.452,"15min":5.452},"last":0.83999999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.127,"15min":1.1359999999999999},"min":{"1min":0.67600000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6399999999999999,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.1060000000000001,"15min":1.153},"min":{"1min":0.71499999999999997,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.3979999999999999,"5min":1.988,"15min":1.988},"last":1.056}]},{"osd":2,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0489999999999999,"15min":1.079},"min":{"1min":0.73699999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.6519999999999999,"5min":1.7929999999999999,"15min":2.1419999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.244,"15min":1.256},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.827,"5min":1.827,"15min":2.7229999999999999},"last":1.2030000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.76800000000000002,"5min":0.74399999999999999,"15min":0.62},"max":{"1min":1.7370000000000001,"5min":1.9750000000000001,"15min":2.9100000000000001},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2789999999999999,"15min":1.276},"min":{"1min":0.93999999999999995,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.7290000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.575}]},{"osd":5,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.54,"15min":1.496},"min":{"1min":1.159,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.893,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":0.998},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4159999999999999,"15min":1.4399999999999999},"min":{"1min":0.79200000000000004,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":2.0419999999999998,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.718}]},{"osd":6,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.246,"15min":1.278},"min":{"1min":0.83899999999999997,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.8540000000000001,"5min":1.907,"15min":3.5899999999999999},"last":1.47},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5700000000000001,"15min":1.5529999999999999},"min":{"1min":1.3280000000000001,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.2890000000000001,"15min":3.391},"last":1.7589999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.486,"15min":1.504},"min":{"1min":0.72899999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.948,"5min":3.1219999999999999,"15min":3.1230000000000002},"last":1.403},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.425,"15min":1.4339999999999999},"min":{"1min":0.98299999999999998,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":3.4849999999999999},"last":1.627}]}]},{"osd":3,"up_from":13,"seq":55834575028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37488,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334352,"statfs":{"total":96636764160,"available":96598376448,"internally_reserved":0,"allocated":2199552,"data_stored":963766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.75700000000000001,"15min":0.84699999999999998},"min":{"1min":0.45400000000000001,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":0.97499999999999998,"5min":1.655,"15min":10.557},"last":1.5820000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.873,"15min":0.91200000000000003},"min":{"1min":0.56899999999999995,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2210000000000001,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.96799999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0109999999999999,"5min":1.728,"15min":2.101},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.84899999999999998,"15min":0.96499999999999997},"min":{"1min":0.52600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.236,"5min":1.8799999999999999,"15min":13.090999999999999},"last":1.1180000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81799999999999995,"15min":0.90700000000000003},"min":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3260000000000001,"5min":1.4750000000000001,"15min":3.9489999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":1.0529999999999999},"min":{"1min":0.54400000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.518,"5min":1.796,"15min":13.460000000000001},"last":1.665}]},{"osd":4,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2090000000000001,"15min":1.266},"min":{"1min":0.874,"5min":0.73999999999999999,"15min":0.67800000000000005},"max":{"1min":1.5680000000000001,"5min":1.877,"15min":4.367},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1899999999999999,"15min":1.224},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.609,"5min":1.9630000000000001,"15min":3.3380000000000001},"last":0.871}]},{"osd":5,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2110000000000001,"15min":1.294},"min":{"1min":0.74099999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":13.231},"last":1.726},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1699999999999999,"15min":1.25},"min":{"1min":0.84999999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6910000000000001,"5min":1.992,"15min":5.1180000000000003},"last":1.399}]},{"osd":6,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.258,"15min":1.357},"min":{"1min":0.89900000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7689999999999999,"5min":2.0449999999999999,"15min":13.359999999999999},"last":1.762},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2050000000000001,"15min":1.254},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3980000000000001,"5min":2.4060000000000001,"15min":3.4409999999999998},"last":1.8420000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:19:06 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2370000000000001,"15min":1.331},"min":{"1min":0.85899999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8260000000000001,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.246,"15min":1.3169999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":3.0030000000000001,"5min":3.0030000000000001,"15min":4.032},"last":1.49}]}]},{"osd":2,"up_from":13,"seq":55834575026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2199552,"data_stored":964266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68999999999999995,"15min":0.78600000000000003},"min":{"1min":0.34200000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.075,"5min":1.601,"15min":1.9610000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80400000000000005,"15min":0.88200000000000001},"min":{"1min":0.437,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.361,"5min":1.7390000000000001,"15min":2.5779999999999998},"last":0.91700000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.82599999999999996,"15min":0.92500000000000004},"min":{"1min":0.47299999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.3120000000000001,"5min":1.518,"15min":1.9890000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.89000000000000001,"15min":0.96699999999999997},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.373,"5min":1.6200000000000001,"15min":2.6240000000000001},"last":0.95499999999999996}]},{"osd":3,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84299999999999997,"15min":0.872},"min":{"1min":0.51300000000000001,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.633,"5min":1.6850000000000001,"15min":2.2090000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.95299999999999996,"15min":1.0549999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5820000000000001,"5min":1.927,"15min":2.9950000000000001},"last":0.85199999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.22,"15min":1.2889999999999999},"min":{"1min":0.72799999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.782,"5min":1.968,"15min":2.7160000000000002},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1839999999999999,"15min":1.268},"min":{"1min":0.78700000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.6639999999999999,"5min":1.8839999999999999,"15min":3.3279999999999998},"last":1.5209999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1599999999999999,"15min":1.272},"min":{"1min":0.753,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.8360000000000001,"5min":1.857,"15min":2.508},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.1850000000000001,"15min":1.26},"min":{"1min":0.752,"5min":0.499,"15min":0.499},"max":{"1min":1.677,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.218}]},{"osd":6,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1539999999999999,"15min":1.2170000000000001},"min":{"1min":0.71899999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.694,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2290000000000001,"15min":1.339},"min":{"1min":0.83599999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.72,"5min":1.911,"15min":2.6629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:15 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.258,"15min":1.3420000000000001},"min":{"1min":0.77400000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8779999999999999,"5min":2.3999999999999999,"15min":2.859},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2430000000000001,"15min":1.3109999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.764,"5min":2.0409999999999999,"15min":3.3809999999999998},"last":1.4750000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2199552,"data_stored":965184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77500000000000002,"15min":0.79700000000000004},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.879,"15min":0.88},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3540000000000001,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.2410000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78900000000000003,"15min":0.78600000000000003},"min":{"1min":0.436,"5min":0.374,"15min":0.37},"max":{"1min":1.242,"5min":1.5229999999999999,"15min":1.861},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.90800000000000003},"min":{"1min":0.54500000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.7490000000000001,"5min":1.925,"15min":1.925},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.63300000000000001,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.42899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5669999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.5980000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.256,"15min":1.266},"min":{"1min":0.77200000000000002,"5min":0.76300000000000001,"15min":0.65300000000000002},"max":{"1min":1.984,"5min":1.984,"15min":2.7959999999999998},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.2509999999999999,"15min":1.2490000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.518,"5min":2.141,"15min":3.0409999999999999},"last":1.5149999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2210000000000001,"15min":1.244},"min":{"1min":0.748,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.8879999999999999,"5min":1.9079999999999999,"15min":3.7639999999999998},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1870000000000001,"15min":1.2},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.643,"5min":1.8340000000000001,"15min":3.286},"last":1.302}]},{"osd":6,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.252,"15min":1.268},"min":{"1min":0.76400000000000001,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":1.6739999999999999,"5min":2.0419999999999998,"15min":4.2750000000000004},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.236,"15min":1.254},"min":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.63100000000000001},"max":{"1min":1.8029999999999999,"5min":1.9690000000000001,"15min":3.7040000000000002},"last":1.5509999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:19:12 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3129999999999999,"15min":1.3109999999999999},"min":{"1min":0.82299999999999995,"5min":0.80200000000000005,"15min":0.59099999999999997},"max":{"1min":1.7969999999999999,"5min":2.1429999999999998,"15min":3.1760000000000002},"last":1.45},{"interface":"front","average":{"1min":1.208,"5min":1.2789999999999999,"15min":1.3129999999999999},"min":{"1min":0.78800000000000003,"5min":0.624,"15min":0.624},"max":{"1min":1.829,"5min":1.929,"15min":3.653},"last":1.6259999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575028,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2785280,"data_stored":1553962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.1819999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78100000000000003,"15min":0.85399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.302,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":1.081}]},{"osd":2,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98899999999999999,"15min":1.0149999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.627,"5min":1.893,"15min":1.893},"last":1.149},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.874},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.54,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.0129999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.92800000000000005,"15min":0.99099999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"max":{"1min":1.4430000000000001,"5min":1.679,"15min":2.0670000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.87,"15min":0.91700000000000004},"min":{"1min":0.53200000000000003,"5min":0.48899999999999999,"15min":0.40100000000000002},"max":{"1min":1.4970000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.169,"15min":1.244},"min":{"1min":0.82899999999999996,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":1.6539999999999999,"5min":1.778,"15min":3.7610000000000001},"last":1.278},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.1080000000000001,"15min":1.1319999999999999},"min":{"1min":0.80200000000000005,"5min":0.72299999999999998,"15min":0.67800000000000005},"max":{"1min":1.4399999999999999,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4390000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.80800000000000005,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.4410000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.232},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.145,"15min":1.2170000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.71099999999999997},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":2.7240000000000002},"last":1.1160000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2310000000000001,"15min":1.2869999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.8560000000000001,"5min":2.052,"15min":2.4369999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.2370000000000001,"15min":1.286},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.5489999999999999,"5min":2.823,"15min":2.823},"last":1.3400000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:19:01 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3109999999999999,"15min":1.3740000000000001},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.81200000000000006},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.6989999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2729999999999999,"15min":1.3440000000000001},"min":{"1min":0.88100000000000001,"5min":0.747,"15min":0.747},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3929999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:19:27.981 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:19:27.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:19:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.240+0000 7f36eeae5640 1 Processor -- start 2023-11-24T08:19:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.240+0000 7f36eeae5640 1 -- start start 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36eeae5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36e815b440 0x7f36e80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36eeae5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80c5dc0 con 0x7f36e80c59b0 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36eeae5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e815eea0 con 0x7f36e80cbf20 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36eeae5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e80cc3d0 con 0x7f36e815b440 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36ec881640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36e815b440 0x7f36e80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36ec881640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36e815b440 0x7f36e80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52868/0 (socket says 172.21.15.107:52868) 2023-11-24T08:19:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.242+0000 7f36ec881640 1 -- 172.21.15.107:0/3210876209 learned_addr learned my addr 172.21.15.107:0/3210876209 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.243+0000 7f36ec881640 1 -- 172.21.15.107:0/3210876209 >> v1:172.21.15.120:6789/0 conn(0x7f36e80cbf20 legacy=0x7f36e80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.243+0000 7f36ec881640 1 -- 172.21.15.107:0/3210876209 >> v1:172.21.15.107:6789/0 conn(0x7f36e80c59b0 legacy=0x7f36e806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.243+0000 7f36ec881640 1 -- 172.21.15.107:0/3210876209 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80abf30 con 0x7f36e815b440 2023-11-24T08:19:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.244+0000 7f36ec881640 1 --2- 172.21.15.107:0/3210876209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e815b440 0x7f36e80cb9e0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d3046ec2478d37 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.244+0000 7f36e77fe640 1 -- 172.21.15.107:0/3210876209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc031870 con 0x7f36e815b440 2023-11-24T08:19:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.244+0000 7f36e77fe640 1 -- 172.21.15.107:0/3210876209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0319d0 con 0x7f36e815b440 2023-11-24T08:19:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.244+0000 7f36e77fe640 1 -- 172.21.15.107:0/3210876209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc031d60 con 0x7f36e815b440 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 -- 172.21.15.107:0/3210876209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e815b440 msgr2=0x7f36e80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 --2- 172.21.15.107:0/3210876209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e815b440 0x7f36e80cb9e0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc02b910 comp rx=0 tx=0).stop 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 -- 172.21.15.107:0/3210876209 shutdown_connections 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 --2- 172.21.15.107:0/3210876209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e815b440 0x7f36e80cb9e0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 -- 172.21.15.107:0/3210876209 >> 172.21.15.107:0/3210876209 conn(0x7f36e80b0c50 msgr2=0x7f36e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 -- 172.21.15.107:0/3210876209 shutdown_connections 2023-11-24T08:19:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.245+0000 7f36eeae5640 1 -- 172.21.15.107:0/3210876209 wait complete. 2023-11-24T08:19:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.247+0000 7f36eeae5640 1 Processor -- start 2023-11-24T08:19:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.247+0000 7f36eeae5640 1 -- start start 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 0x7f36e80b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e815b440 0x7f36e806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80b2d00 con 0x7f36e80c59b0 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80ce9d0 con 0x7f36e80cbf20 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36eeae5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80b2080 con 0x7f36e815b440 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ec881640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ec881640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52880/0 (socket says 172.21.15.107:52880) 2023-11-24T08:19:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ec881640 1 -- 172.21.15.107:0/205582940 learned_addr learned my addr 172.21.15.107:0/205582940 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36e7fff640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 0x7f36e80b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ec881640 1 -- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e815b440 msgr2=0x7f36e806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ed082640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e815b440 0x7f36e806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.248+0000 7f36ec881640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e815b440 0x7f36e806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36ec881640 1 -- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 msgr2=0x7f36e80b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36ec881640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 0x7f36e80b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36ec881640 1 -- 172.21.15.107:0/205582940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0094d0 con 0x7f36e80c59b0 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36e7fff640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 0x7f36e80b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36ec881640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc031820 con 0x7f36e80c59b0 2023-11-24T08:19:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0443f0 con 0x7f36e80c59b0 2023-11-24T08:19:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0447a0 con 0x7f36e80c59b0 2023-11-24T08:19:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e80b2360 con 0x7f36e80c59b0 2023-11-24T08:19:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.249+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e80b2920 con 0x7f36e80c59b0 2023-11-24T08:19:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.250+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e804e8b0 con 0x7f36e80c59b0 2023-11-24T08:19:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.251+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f36dc04d450 con 0x7f36e80c59b0 2023-11-24T08:19:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.253+0000 7f36e57fa640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 0x7f36cc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.253+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36dc02d040 con 0x7f36e80c59b0 2023-11-24T08:19:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.253+0000 7f36e7fff640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 0x7f36cc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.254+0000 7f36e7fff640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 0x7f36cc06ed70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f36e80c9a60 tx=0x7f36d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.257+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36dc077740 con 0x7f36e80c59b0 2023-11-24T08:19:28.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.512+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f36e806dc80 con 0x7f36e80c59b0 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.517+0000 7f36e57fa640 1 -- 172.21.15.107:0/205582940 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f36dc035070 con 0x7f36e80c59b0 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:19:28.519 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:19:28.520 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.519+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 msgr2=0x7f36cc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.519+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 0x7f36cc06ed70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f36e80c9a60 tx=0x7f36d4009040 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.519+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 msgr2=0x7f36e8156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.519+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 shutdown_connections 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f36cc06c8b0 0x7f36cc06ed70 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e815b440 0x7f36e806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e80cbf20 0x7f36e80b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 --2- 172.21.15.107:0/205582940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36e80c59b0 0x7f36e8156670 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 >> 172.21.15.107:0/205582940 conn(0x7f36e80b0c50 msgr2=0x7f36e80ad110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:28.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 shutdown_connections 2023-11-24T08:19:28.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.520+0000 7f36eeae5640 1 -- 172.21.15.107:0/205582940 wait complete. 2023-11-24T08:19:28.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:19:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.748+0000 7f1a882e7640 1 Processor -- start 2023-11-24T08:19:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.749+0000 7f1a882e7640 1 -- start start 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a882e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a800cf5d0 0x7f1a8006d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a882e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8015e8b0 con 0x7f1a8015af40 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a882e7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a80070180 con 0x7f1a8015b320 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a882e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8015eb90 con 0x7f1a800cf5d0 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a85882640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a800cf5d0 0x7f1a8006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a85882640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a800cf5d0 0x7f1a8006d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52892/0 (socket says 172.21.15.107:52892) 2023-11-24T08:19:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.750+0000 7f1a85882640 1 -- 172.21.15.107:0/3067065411 learned_addr learned my addr 172.21.15.107:0/3067065411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.751+0000 7f1a85882640 1 -- 172.21.15.107:0/3067065411 >> v1:172.21.15.120:6789/0 conn(0x7f1a8015b320 legacy=0x7f1a8006b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.751+0000 7f1a85882640 1 -- 172.21.15.107:0/3067065411 >> v1:172.21.15.107:6789/0 conn(0x7f1a8015af40 legacy=0x7f1a800ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.751+0000 7f1a85882640 1 -- 172.21.15.107:0/3067065411 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80070460 con 0x7f1a800cf5d0 2023-11-24T08:19:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.752+0000 7f1a85882640 1 --2- 172.21.15.107:0/3067065411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a800cf5d0 0x7f1a8006d220 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a7402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a1afb03fa47fb2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.752+0000 7f1a85081640 1 -- 172.21.15.107:0/3067065411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74030860 con 0x7f1a800cf5d0 2023-11-24T08:19:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.752+0000 7f1a85081640 1 -- 172.21.15.107:0/3067065411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a740309c0 con 0x7f1a800cf5d0 2023-11-24T08:19:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.752+0000 7f1a85081640 1 -- 172.21.15.107:0/3067065411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74030d50 con 0x7f1a800cf5d0 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 -- 172.21.15.107:0/3067065411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a800cf5d0 msgr2=0x7f1a8006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 --2- 172.21.15.107:0/3067065411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a800cf5d0 0x7f1a8006d220 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a7402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 -- 172.21.15.107:0/3067065411 shutdown_connections 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 --2- 172.21.15.107:0/3067065411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a800cf5d0 0x7f1a8006d220 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 -- 172.21.15.107:0/3067065411 >> 172.21.15.107:0/3067065411 conn(0x7f1a8006e7f0 msgr2=0x7f1a8006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 -- 172.21.15.107:0/3067065411 shutdown_connections 2023-11-24T08:19:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.753+0000 7f1a882e7640 1 -- 172.21.15.107:0/3067065411 wait complete. 2023-11-24T08:19:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.755+0000 7f1a882e7640 1 Processor -- start 2023-11-24T08:19:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.755+0000 7f1a882e7640 1 -- start start 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a800cf5d0 0x7f1a80159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8015b320 0x7f1a80152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80152650 con 0x7f1a8015af40 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8014fd20 con 0x7f1a8015b320 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a882e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80151a40 con 0x7f1a800cf5d0 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52906/0 (socket says 172.21.15.107:52906) 2023-11-24T08:19:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 -- 172.21.15.107:0/1155915459 learned_addr learned my addr 172.21.15.107:0/1155915459 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:28.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 -- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a800cf5d0 msgr2=0x7f1a80159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a800cf5d0 0x7f1a80159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 -- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8015b320 msgr2=0x7f1a80152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8015b320 0x7f1a80152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.756+0000 7f1a85882640 1 -- 172.21.15.107:0/1155915459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a740094d0 con 0x7f1a8015af40 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a85882640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74030810 con 0x7f1a8015af40 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74043400 con 0x7f1a8015af40 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a80151d20 con 0x7f1a8015af40 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a740437b0 con 0x7f1a8015af40 2023-11-24T08:19:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.757+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a801522e0 con 0x7f1a8015af40 2023-11-24T08:19:28.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.758+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a80067470 con 0x7f1a8015af40 2023-11-24T08:19:28.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.759+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f1a74043910 con 0x7f1a8015af40 2023-11-24T08:19:28.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.760+0000 7f1a7effd640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 0x7f1a6c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:28.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.760+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a7403e070 con 0x7f1a8015af40 2023-11-24T08:19:28.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.760+0000 7f1a86083640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 0x7f1a6c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:28.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.761+0000 7f1a86083640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 0x7f1a6c06ed70 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1a700012e0 tx=0x7f1a70007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.765+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a74078480 con 0x7f1a8015af40 2023-11-24T08:19:28.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.989+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a800b7270 con 0x7f1a8015af40 2023-11-24T08:19:28.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.990+0000 7f1a7effd640 1 -- 172.21.15.107:0/1155915459 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f1a74034070 con 0x7f1a8015af40 2023-11-24T08:19:28.992 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:19:28.992 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:19:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 107 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 msgr2=0x7f1a6c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 0x7f1a6c06ed70 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1a700012e0 tx=0x7f1a70007d30 comp rx=0 tx=0).stop 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 msgr2=0x7f1a8014f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a74004730 comp rx=0 tx=0).stop 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 shutdown_connections 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a6c06c8b0 0x7f1a6c06ed70 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a8015b320 0x7f1a80152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a8015af40 0x7f1a8014f7e0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 --2- 172.21.15.107:0/1155915459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a800cf5d0 0x7f1a80159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:28.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 >> 172.21.15.107:0/1155915459 conn(0x7f1a8006e7f0 msgr2=0x7f1a8015bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:28.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 shutdown_connections 2023-11-24T08:19:28.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:28.993+0000 7f1a882e7640 1 -- 172.21.15.107:0/1155915459 wait complete. 2023-11-24T08:19:29.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:19:21.993451+0000', 'modified': '2023-11-24T08:19:25.082775+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10410}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10410': {'gid': 10410, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6836/3208934992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3208934992}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 3208934992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:19:29.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:19:29.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:19:29.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.646+0000 7f82e1a1a640 1 Processor -- start 2023-11-24T08:19:29.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.646+0000 7f82e1a1a640 1 -- start start 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e1a1a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82dc15b1d0 0x7f82dc071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e1a1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc0b1250 0x7f82dc15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e1a1a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc0ca0e0 con 0x7f82dc070550 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e1a1a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc072800 con 0x7f82dc15b1d0 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e1a1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0c97b0 con 0x7f82dc0b1250 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82dc15b1d0 0x7f82dc071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82dc15b1d0 0x7f82dc071f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52910/0 (socket says 172.21.15.107:52910) 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 -- 172.21.15.107:0/177383921 learned_addr learned my addr 172.21.15.107:0/177383921 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 -- 172.21.15.107:0/177383921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc0b1250 msgr2=0x7f82dc15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 --2- 172.21.15.107:0/177383921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc0b1250 0x7f82dc15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 -- 172.21.15.107:0/177383921 >> v1:172.21.15.120:6789/0 conn(0x7f82dc070550 legacy=0x7f82dc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.648+0000 7f82e0a18640 1 -- 172.21.15.107:0/177383921 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc0c9a90 con 0x7f82dc15b1d0 2023-11-24T08:19:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.649+0000 7f82e0a18640 1 --2- 172.21.15.107:0/177383921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc15b1d0 0x7f82dc071f40 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009870 tx=0x7f82d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb894ca357eb124 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82db7fe640 1 -- 172.21.15.107:0/177383921 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031640 con 0x7f82dc15b1d0 2023-11-24T08:19:29.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82db7fe640 1 -- 172.21.15.107:0/177383921 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00317a0 con 0x7f82dc15b1d0 2023-11-24T08:19:29.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82db7fe640 1 -- 172.21.15.107:0/177383921 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031b30 con 0x7f82dc15b1d0 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 -- 172.21.15.107:0/177383921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc15b1d0 msgr2=0x7f82dc071f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/177383921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc15b1d0 0x7f82dc071f40 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009870 tx=0x7f82d002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 -- 172.21.15.107:0/177383921 shutdown_connections 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/177383921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc0b1250 0x7f82dc15edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/177383921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc15b1d0 0x7f82dc071f40 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.650+0000 7f82e1a1a640 1 -- 172.21.15.107:0/177383921 >> 172.21.15.107:0/177383921 conn(0x7f82dc0cbb40 msgr2=0x7f82dc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.651+0000 7f82e1a1a640 1 -- 172.21.15.107:0/177383921 shutdown_connections 2023-11-24T08:19:29.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.651+0000 7f82e1a1a640 1 -- 172.21.15.107:0/177383921 wait complete. 2023-11-24T08:19:29.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.652+0000 7f82e1a1a640 1 Processor -- start 2023-11-24T08:19:29.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.652+0000 7f82e1a1a640 1 -- start start 2023-11-24T08:19:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82dc0b1250 0x7f82dc0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc15b1d0 0x7f82dc0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0c7de0 con 0x7f82dc070550 2023-11-24T08:19:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1e5890 con 0x7f82dc0b1250 2023-11-24T08:19:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.653+0000 7f82e1a1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0c7440 con 0x7f82dc15b1d0 2023-11-24T08:19:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52918/0 (socket says 172.21.15.107:52918) 2023-11-24T08:19:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 -- 172.21.15.107:0/575982346 learned_addr learned my addr 172.21.15.107:0/575982346 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82dbfff640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82dc0b1250 0x7f82dc0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 -- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc15b1d0 msgr2=0x7f82dc0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc15b1d0 0x7f82dc0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 -- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82dc0b1250 msgr2=0x7f82dc0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82dc0b1250 0x7f82dc0cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 -- 172.21.15.107:0/575982346 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00094d0 con 0x7f82dc070550 2023-11-24T08:19:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.654+0000 7f82e0a18640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009870 tx=0x7f82d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.655+0000 7f82d97fa640 1 -- 172.21.15.107:0/575982346 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d003ac60 con 0x7f82dc070550 2023-11-24T08:19:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.655+0000 7f82d97fa640 1 -- 172.21.15.107:0/575982346 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d003adc0 con 0x7f82dc070550 2023-11-24T08:19:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.655+0000 7f82d97fa640 1 -- 172.21.15.107:0/575982346 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0044630 con 0x7f82dc070550 2023-11-24T08:19:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.655+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc0c7720 con 0x7f82dc070550 2023-11-24T08:19:29.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.655+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc148e50 con 0x7f82dc070550 2023-11-24T08:19:29.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.657+0000 7f82d97fa640 1 -- 172.21.15.107:0/575982346 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f82d002f050 con 0x7f82dc070550 2023-11-24T08:19:29.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.658+0000 7f82d97fa640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 0x7f82c006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.658+0000 7f82d97fa640 1 -- 172.21.15.107:0/575982346 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82d00aa530 con 0x7f82dc070550 2023-11-24T08:19:29.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.658+0000 7f82dbfff640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 0x7f82c006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.659+0000 7f82e580b740 1 Processor -- start 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.659+0000 7f82e580b740 1 -- start start 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.659+0000 7f82dbfff640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 0x7f82c006ed70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0cfc50 tx=0x7f82c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f6f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987e8a880 con 0x555987f26d70 2023-11-24T08:19:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987e99ae0 con 0x555987f2b7b0 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555987e971d0 con 0x555987f2c070 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52930/0 (socket says 172.21.15.107:52930) 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.660+0000 7f82dbfff640 1 -- 172.21.15.107:0/2310911292 learned_addr learned my addr 172.21.15.107:0/2310911292 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82e0a18640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 -- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 msgr2=0x555987f2c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 -- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 msgr2=0x555987f6f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82e1219640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f6f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f6f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 -- 172.21.15.107:0/2310911292 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555987e5a860 con 0x555987f26d70 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82e1219640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f6f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:29.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82dbfff640 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c970 tx=0x7f82c800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28717e98c4974e5d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82c67fc640 1 -- 172.21.15.107:0/2310911292 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8013b80 con 0x555987f26d70 2023-11-24T08:19:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82c67fc640 1 -- 172.21.15.107:0/2310911292 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8013ce0 con 0x555987f26d70 2023-11-24T08:19:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.661+0000 7f82c67fc640 1 -- 172.21.15.107:0/2310911292 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c801d630 con 0x555987f26d70 2023-11-24T08:19:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 -- 172.21.15.107:0/2310911292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 msgr2=0x555987f2b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f82c800c970 tx=0x7f82c800f000 comp rx=0 tx=0).stop 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 -- 172.21.15.107:0/2310911292 shutdown_connections 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f6f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f2b270 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 --2- 172.21.15.107:0/2310911292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 -- 172.21.15.107:0/2310911292 >> 172.21.15.107:0/2310911292 conn(0x555987eedc70 msgr2=0x555987f71b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 -- 172.21.15.107:0/2310911292 shutdown_connections 2023-11-24T08:19:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.662+0000 7f82e580b740 1 -- 172.21.15.107:0/2310911292 wait complete. 2023-11-24T08:19:29.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.663+0000 7f82e580b740 1 -- start start 2023-11-24T08:19:29.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.663+0000 7f82e580b740 1 Processor -- start 2023-11-24T08:19:29.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f797c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f78ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987e92ac0 con 0x555987f26d70 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987e2f2f0 con 0x555987f2b7b0 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555987e4cc70 con 0x555987f2c070 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e580b740 10 client.0 _ll_register_callbacks cb 0x555987e513b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.664+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52938/0 (socket says 172.21.15.107:52938) 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 -- 172.21.15.107:0/3961226589 learned_addr learned my addr 172.21.15.107:0/3961226589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 -- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 msgr2=0x555987f78ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f78ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 -- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 msgr2=0x555987f797c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 -- 172.21.15.107:0/3961226589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c800c5f0 con 0x555987f26d70 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f82d002b930 tx=0x7f82d007a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c22b50e64b026cc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00aa210 con 0x555987f26d70 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82d007ad00 con 0x555987f26d70 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.665+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555987e8baa0 con 0x555987f26d70 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d007ee50 con 0x555987f26d70 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 10 client.10551 fetch_fsmap learned FSMap version 107 2023-11-24T08:19:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x555987e91140 con 0x555987f26d70 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555987e8baa0 con 0x555987f26d70 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f82d007a1f0 con 0x555987f26d70 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 10 client.10551 fetch_fsmap finished waiting for FSMap version 107 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 10 client.10551 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 20 client.10551 populate_metadata read hostname 'smithi107' 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f51af0 0x555987f8f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 4 client.10551 mds_command: new command op to 10410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 --> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555987e8b7c0 con 0x555987f51af0 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.666+0000 7f82dbfff640 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f51af0 0x555987f8f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.667+0000 7f82dbfff640 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f51af0 0x555987f8f140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420f8520099d259c server_cookie=bee9d4ccd77f3b47 in_seq=0 out_seq=0 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.667+0000 7f82bb7fe640 10 client.10551 ms_handle_connect on v2:172.21.15.107:6834/3208934992 2023-11-24T08:19:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3961226589 <== mds.0 v2:172.21.15.107:6834/3208934992 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555987e8b7c0 con 0x555987f51af0 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82bb7fe640 10 client.10551 handle_command_reply: tid=0 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 client.10551 shutdown 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 msgr2=0x555987f6e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f82d002b930 tx=0x7f82d007a660 comp rx=0 tx=0).stop 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 shutdown_connections 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f51af0 0x555987f8f140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f78ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 --2- 172.21.15.107:0/3961226589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f6e0c0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82bb7fe640 0 client.10551 ms_handle_reset on v2:172.21.15.107:6834/3208934992 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 >> 172.21.15.107:0/3961226589 conn(0x555987eedc70 msgr2=0x555987f2c450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 shutdown_connections 2023-11-24T08:19:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 1 -- 172.21.15.107:0/3961226589 wait complete. 2023-11-24T08:19:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.668+0000 7f82e580b740 20 client.10551 trim_cache size 0 max 16384 2023-11-24T08:19:29.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.685+0000 7f82e580b740 1 Processor -- start 2023-11-24T08:19:29.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.685+0000 7f82e580b740 1 -- start start 2023-11-24T08:19:29.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f70900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987bf5f20 con 0x555987f26d70 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987f78b70 con 0x555987f2b7b0 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555987f528a0 con 0x555987f2c070 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52950/0 (socket says 172.21.15.107:52950) 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.686+0000 7f82e0a18640 1 -- 172.21.15.107:0/1577965901 learned_addr learned my addr 172.21.15.107:0/1577965901 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 -- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 msgr2=0x555987f2add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 -- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 msgr2=0x555987f70900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f70900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 -- 172.21.15.107:0/1577965901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555987e2eb80 con 0x555987f26d70 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82e0a18640 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f82d007ac90 tx=0x7f82d0091000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dd0898354bf8c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82c6ffd640 1 -- 172.21.15.107:0/1577965901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d007aad0 con 0x555987f26d70 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82c6ffd640 1 -- 172.21.15.107:0/1577965901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d0031e60 con 0x555987f26d70 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.687+0000 7f82c6ffd640 1 -- 172.21.15.107:0/1577965901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d007ad00 con 0x555987f26d70 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 -- 172.21.15.107:0/1577965901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 msgr2=0x555987f55de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f82d007ac90 tx=0x7f82d0091000 comp rx=0 tx=0).stop 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 -- 172.21.15.107:0/1577965901 shutdown_connections 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987f2c070 0x555987f2add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2b7b0 0x555987f70900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 --2- 172.21.15.107:0/1577965901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f26d70 0x555987f55de0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 -- 172.21.15.107:0/1577965901 >> 172.21.15.107:0/1577965901 conn(0x555987eedc70 msgr2=0x555987f2c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 -- 172.21.15.107:0/1577965901 shutdown_connections 2023-11-24T08:19:29.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.688+0000 7f82e580b740 1 -- 172.21.15.107:0/1577965901 wait complete. 2023-11-24T08:19:29.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.689+0000 7f82e580b740 1 -- start start 2023-11-24T08:19:29.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.689+0000 7f82e580b740 1 Processor -- start 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2c070 0x555987f27740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987fdb390 0x555987f594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987f5a600 con 0x555987f2b7b0 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555987f26260 con 0x555987f2c070 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555987f52d90 con 0x555987fdb390 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e580b740 10 client.0 _ll_register_callbacks cb 0x7f82d003af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:19:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52966/0 (socket says 172.21.15.107:52966) 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 -- 172.21.15.107:0/3456957102 learned_addr learned my addr 172.21.15.107:0/3456957102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 -- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987fdb390 msgr2=0x555987f594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987fdb390 0x555987f594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 -- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2c070 msgr2=0x555987f27740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.690+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2c070 0x555987f27740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82e0a18640 1 -- 172.21.15.107:0/3456957102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c800c5f0 con 0x555987f2b7b0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82e0a18640 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f82d007aa00 tx=0x7f82d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=decf4edd7d4b9990 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d007aad0 con 0x555987f2b7b0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82d007ad00 con 0x555987f2b7b0 2023-11-24T08:19:29.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82c8013ce0 con 0x555987f2b7b0 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d007ee50 con 0x555987f2b7b0 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82e580b740 10 client.10557 fetch_fsmap learned FSMap version 107 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x555987e54170 con 0x555987f2b7b0 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8013ce0 con 0x555987f2b7b0 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.691+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f82d00a0030 con 0x555987f2b7b0 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 10 client.10557 fetch_fsmap finished waiting for FSMap version 107 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 10 client.10557 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 20 client.10557 populate_metadata read hostname 'smithi107' 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f59a30 0x555987f74ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 4 client.10557 mds_command: new command op to 10410 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:19:29.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 --> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555987b936c0 con 0x555987f59a30 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82dbfff640 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f59a30 0x555987f74ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82dbfff640 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f59a30 0x555987f74ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9d3210a60f34c0c server_cookie=da24e2dfeed37f in_seq=0 out_seq=0 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.692+0000 7f82bb7fe640 10 client.10557 ms_handle_connect on v2:172.21.15.107:6834/3208934992 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82bb7fe640 1 -- 172.21.15.107:0/3456957102 <== mds.0 v2:172.21.15.107:6834/3208934992 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x555987b936c0 con 0x555987f59a30 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82bb7fe640 10 client.10557 handle_command_reply: tid=0 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 client.10557 shutdown 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 msgr2=0x555987f27200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f82d007aa00 tx=0x7f82d0000bc0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 shutdown_connections 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:6834/3208934992,v1:172.21.15.107:6836/3208934992] conn(0x555987f59a30 0x555987f74ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555987fdb390 0x555987f594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555987f2c070 0x555987f27740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 --2- 172.21.15.107:0/3456957102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555987f2b7b0 0x555987f27200 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 >> 172.21.15.107:0/3456957102 conn(0x555987eedc70 msgr2=0x555987f2c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82bb7fe640 0 client.10557 ms_handle_reset on v2:172.21.15.107:6834/3208934992 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 shutdown_connections 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 1 -- 172.21.15.107:0/3456957102 wait complete. 2023-11-24T08:19:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.693+0000 7f82e580b740 20 client.10557 trim_cache size 0 max 16384 2023-11-24T08:19:29.701 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:29.701 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:19:29.701 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:19:29.701 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 10410, 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 7.1193933869999997, 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 107, 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 175, 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 175, 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.940222691000001 2023-11-24T08:19:29.702 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 msgr2=0x7f82c006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 0x7f82c006ed70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0cfc50 tx=0x7f82c8008040 comp rx=0 tx=0).stop 2023-11-24T08:19:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 msgr2=0x7f82dc148910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009870 tx=0x7f82d00046c0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 shutdown_connections 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82c006c8b0 0x7f82c006ed70 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82dc15b1d0 0x7f82dc0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82dc0b1250 0x7f82dc0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 --2- 172.21.15.107:0/575982346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82dc070550 0x7f82dc148910 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 >> 172.21.15.107:0/575982346 conn(0x7f82dc0cbb40 msgr2=0x7f82dc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 shutdown_connections 2023-11-24T08:19:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.694+0000 7f82e1a1a640 1 -- 172.21.15.107:0/575982346 wait complete. 2023-11-24T08:19:29.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:19:29.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.914+0000 7f5d873c9640 1 Processor -- start 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.914+0000 7f5d873c9640 1 -- start start 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.915+0000 7f5d873c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d8013e110 0x7f5d800b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.915+0000 7f5d873c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800b2320 con 0x7f5d8006ce90 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.915+0000 7f5d873c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8015fb30 con 0x7f5d800cf410 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.915+0000 7f5d873c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d800b2550 con 0x7f5d8013e110 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d85bc6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5d8006ce90 0x7f5d800ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58500/0 (socket says 172.21.15.107:58500) 2023-11-24T08:19:29.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d85bc6640 1 -- 172.21.15.107:0/1469225639 learned_addr learned my addr 172.21.15.107:0/1469225639 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d863c7640 1 --2- 172.21.15.107:0/1469225639 >> v2:172.21.15.120:3300/0 conn(0x7f5d8013e110 0x7f5d800b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873481154 0 0) 0x7f5d800b2320 con 0x7f5d8006ce90 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c003520 con 0x7f5d8006ce90 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271793020 0 0) 0x7f5d6c003520 con 0x7f5d8006ce90 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 >> v2:172.21.15.120:3300/0 conn(0x7f5d8013e110 msgr2=0x7f5d800b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 --2- 172.21.15.107:0/1469225639 >> v2:172.21.15.120:3300/0 conn(0x7f5d8013e110 0x7f5d800b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 >> v1:172.21.15.120:6789/0 conn(0x7f5d800cf410 legacy=0x7f5d8015b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8015ed70 con 0x7f5d8006ce90 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5d74001670 con 0x7f5d8006ce90 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 >> v1:172.21.15.107:6789/0 conn(0x7f5d8006ce90 legacy=0x7f5d800ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.916+0000 7f5d853c5640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0039c0 0x7f5d6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d853c5640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 0x7f5d6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d853c5640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d6c0051d0 0x7f5d6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800b2550 con 0x7f5d6c0047c0 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c00c100 con 0x7f5d6c0039c0 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c00c2c0 con 0x7f5d6c0051d0 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 0x7f5d6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 -- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d6c0051d0 msgr2=0x7f5d6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d6c0051d0 0x7f5d6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 -- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0039c0 msgr2=0x7f5d6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0039c0 0x7f5d6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d85bc6640 1 -- 172.21.15.107:0/1469225639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c003520 con 0x7f5d6c0047c0 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.917+0000 7f5d86bc8640 1 -- 172.21.15.107:0/1469225639 reap_dead start 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.918+0000 7f5d85bc6640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 0x7f5d6c004c40 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400c190 tx=0x7f5d6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11be64158e404291 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.918+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74005ad0 con 0x7f5d6c0047c0 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.918+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d74005c30 con 0x7f5d6c0047c0 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.918+0000 7f5d853c5640 1 -- 172.21.15.107:0/1469225639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7402c810 con 0x7f5d6c0047c0 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 -- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 msgr2=0x7f5d6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 0x7f5d6c004c40 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400c190 tx=0x7f5d6c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 -- 172.21.15.107:0/1469225639 shutdown_connections 2023-11-24T08:19:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 --2- 172.21.15.107:0/1469225639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d6c0047c0 0x7f5d6c004c40 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 -- 172.21.15.107:0/1469225639 >> 172.21.15.107:0/1469225639 conn(0x7f5d800cba10 msgr2=0x7f5d800cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 -- 172.21.15.107:0/1469225639 shutdown_connections 2023-11-24T08:19:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.919+0000 7f5d873c9640 1 -- 172.21.15.107:0/1469225639 wait complete. 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.921+0000 7f5d873c9640 1 Processor -- start 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.921+0000 7f5d873c9640 1 -- start start 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0047c0 0x7f5d8013d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8013e4f0 0x7f5d8013e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8013fbd0 con 0x7f5d800afbf0 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800b05b0 con 0x7f5d6c0047c0 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d873c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8013d990 con 0x7f5d8013e4f0 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d86bc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d86bc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52990/0 (socket says 172.21.15.107:52990) 2023-11-24T08:19:29.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d86bc8640 1 -- 172.21.15.107:0/3865828430 learned_addr learned my addr 172.21.15.107:0/3865828430 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.922+0000 7f5d863c7640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0047c0 0x7f5d8013d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 -- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8013e4f0 msgr2=0x7f5d8013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d85bc6640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8013e4f0 0x7f5d8013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8013e4f0 0x7f5d8013e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 -- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0047c0 msgr2=0x7f5d8013d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0047c0 0x7f5d8013d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 -- 172.21.15.107:0/3865828430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8013ef50 con 0x7f5d800afbf0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d86bc8640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f5d800cfa20 tx=0x7f5d7800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d780070d0 con 0x7f5d800afbf0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78007230 con 0x7f5d800afbf0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d780075e0 con 0x7f5d800afbf0 2023-11-24T08:19:29.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8013f230 con 0x7f5d800afbf0 2023-11-24T08:19:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.923+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8013f7f0 con 0x7f5d800afbf0 2023-11-24T08:19:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.924+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d800b4f40 con 0x7f5d800afbf0 2023-11-24T08:19:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.925+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f5d78007740 con 0x7f5d800afbf0 2023-11-24T08:19:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.927+0000 7f5d72ffd640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 0x7f5d6406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.927+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d78087950 con 0x7f5d800afbf0 2023-11-24T08:19:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.927+0000 7f5d863c7640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 0x7f5d6406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.928+0000 7f5d863c7640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 0x7f5d6406ed70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f5d80064320 tx=0x7f5d7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:29.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:29.932+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d78054460 con 0x7f5d800afbf0 2023-11-24T08:19:30.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.166+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5d8013f980 con 0x7f5d800afbf0 2023-11-24T08:19:30.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.166+0000 7f5d72ffd640 1 -- 172.21.15.107:0/3865828430 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5d78017070 con 0x7f5d800afbf0 2023-11-24T08:19:30.168 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:19:30.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 msgr2=0x7f5d6406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:30.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 0x7f5d6406ed70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f5d80064320 tx=0x7f5d7c008040 comp rx=0 tx=0).stop 2023-11-24T08:19:30.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 msgr2=0x7f5d800b0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:30.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f5d800cfa20 tx=0x7f5d7800c980 comp rx=0 tx=0).stop 2023-11-24T08:19:30.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 shutdown_connections 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d6406c8b0 0x7f5d6406ed70 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d800afbf0 0x7f5d800b0070 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8013e4f0 0x7f5d8013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 --2- 172.21.15.107:0/3865828430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c0047c0 0x7f5d8013d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 >> 172.21.15.107:0/3865828430 conn(0x7f5d800cba10 msgr2=0x7f5d800ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 shutdown_connections 2023-11-24T08:19:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:30.169+0000 7f5d873c9640 1 -- 172.21.15.107:0/3865828430 wait complete. 2023-11-24T08:19:30.190 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:19:30.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:19:30.191 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:19:30.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.191 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:19:30.245 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:19:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:19:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:19:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:19:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41500sec preferred_lft 41500sec 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:19:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:30.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.248 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:19:30.338 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:19:30.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.342 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:19:30.398 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:19:30.398 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:19:30.398 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:19:30.399 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:19:30.399 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:19:30.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:19:30.400 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:19:30.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:19:30.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:19:30.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:19:30.400 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:19:30.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.583 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:19:30.639 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:19:30.639 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:19:30.640 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:19:30.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:19:30.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:19:30.729 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:19:30.780 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:19:30.780 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:19:30.780 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:19:30.832 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:19:30.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:30.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:30.969 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:30.969 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:19:31.003 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:31.003 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:19:31.003 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:19:31.003 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:19:31.004 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:19:31.005 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:19:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:19:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:19:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:19:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:19:31.010 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:19:31.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:19:31.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.100 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:19:31.167 INFO:teuthology.orchestra.run.smithi107.stdout:client7135 2023-11-24T08:19:31.167 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:31.167 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:31.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:19:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.425+0000 7fa21e9ac640 1 Processor -- start 2023-11-24T08:19:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.425+0000 7fa21e9ac640 1 -- start start 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21e9ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa21813d120 0x7fa21815af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21e9ac640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa21806d6c0 0x7fa21806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21e9ac640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806e060 con 0x7fa2180b0a70 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21e9ac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21813f9e0 con 0x7fa21813d120 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21e9ac640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21813fcc0 con 0x7fa21806d6c0 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d9aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa21813d120 0x7fa21815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d9aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa21813d120 0x7fa21815af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52994/0 (socket says 172.21.15.107:52994) 2023-11-24T08:19:31.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d9aa640 1 -- 172.21.15.107:0/280815682 learned_addr learned my addr 172.21.15.107:0/280815682 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d9aa640 1 -- 172.21.15.107:0/280815682 >> v2:172.21.15.120:3300/0 conn(0x7fa21806d6c0 msgr2=0x7fa21806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d1a9640 1 --2- 172.21.15.107:0/280815682 >> v2:172.21.15.120:3300/0 conn(0x7fa21806d6c0 0x7fa21806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.427+0000 7fa21d9aa640 1 --2- 172.21.15.107:0/280815682 >> v2:172.21.15.120:3300/0 conn(0x7fa21806d6c0 0x7fa21806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.428+0000 7fa21d9aa640 1 -- 172.21.15.107:0/280815682 >> v1:172.21.15.120:6789/0 conn(0x7fa2180b0a70 legacy=0x7fa2180b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.428+0000 7fa21d9aa640 1 -- 172.21.15.107:0/280815682 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180cec00 con 0x7fa21813d120 2023-11-24T08:19:31.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21d9aa640 1 --2- 172.21.15.107:0/280815682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21813d120 0x7fa21815af40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fa218072a70 tx=0x7fa21402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b109d1ed336b290 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:31.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21c9a8640 1 -- 172.21.15.107:0/280815682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140317e0 con 0x7fa21813d120 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21c9a8640 1 -- 172.21.15.107:0/280815682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214031940 con 0x7fa21813d120 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21c9a8640 1 -- 172.21.15.107:0/280815682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214031cd0 con 0x7fa21813d120 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21e9ac640 1 -- 172.21.15.107:0/280815682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21813d120 msgr2=0x7fa21815af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.429+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/280815682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21813d120 0x7fa21815af40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fa218072a70 tx=0x7fa21402b880 comp rx=0 tx=0).stop 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 -- 172.21.15.107:0/280815682 shutdown_connections 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/280815682 >> v2:172.21.15.120:3300/0 conn(0x7fa21806d6c0 0x7fa21806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/280815682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21813d120 0x7fa21815af40 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 -- 172.21.15.107:0/280815682 >> 172.21.15.107:0/280815682 conn(0x7fa2180cb830 msgr2=0x7fa2180cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 -- 172.21.15.107:0/280815682 shutdown_connections 2023-11-24T08:19:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.430+0000 7fa21e9ac640 1 -- 172.21.15.107:0/280815682 wait complete. 2023-11-24T08:19:31.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.431+0000 7fa21e9ac640 1 Processor -- start 2023-11-24T08:19:31.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 -- start start 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa21806d6c0 0x7fa2180b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa21813d120 0x7fa218070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b3110 con 0x7fa2180b0a70 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b4ac0 con 0x7fa21813d120 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.432+0000 7fa21e9ac640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b2b50 con 0x7fa21806d6c0 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52996/0 (socket says 172.21.15.107:52996) 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 -- 172.21.15.107:0/733142456 learned_addr learned my addr 172.21.15.107:0/733142456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 -- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa21806d6c0 msgr2=0x7fa2180b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa21806d6c0 0x7fa2180b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 -- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa21813d120 msgr2=0x7fa218070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:31.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa21813d120 0x7fa218070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 -- 172.21.15.107:0/733142456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140094d0 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.433+0000 7fa21d1a9640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c7f0 tx=0x7fa20800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.434+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208004220 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.434+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa208004380 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.434+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208007840 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.434+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180b2e30 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.434+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218070690 con 0x7fa2180b0a70 2023-11-24T08:19:31.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.435+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa218070210 con 0x7fa2180b0a70 2023-11-24T08:19:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.436+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fa208007bb0 con 0x7fa2180b0a70 2023-11-24T08:19:31.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.437+0000 7fa20e7fc640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 0x7fa1fc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:31.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.437+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa208088470 con 0x7fa2180b0a70 2023-11-24T08:19:31.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.437+0000 7fa21d9aa640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 0x7fa1fc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:31.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.438+0000 7fa21d9aa640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 0x7fa1fc06ed70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa214036000 tx=0x7fa214030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.442+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa218070210 con 0x7fa2180b0a70 2023-11-24T08:19:31.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.669+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa218070210 con 0x7fa2180b0a70 2023-11-24T08:19:31.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.670+0000 7fa20e7fc640 1 -- 172.21.15.107:0/733142456 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa218070210 con 0x7fa2180b0a70 2023-11-24T08:19:31.672 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:19:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 msgr2=0x7fa1fc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 0x7fa1fc06ed70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa214036000 tx=0x7fa214030000 comp rx=0 tx=0).stop 2023-11-24T08:19:31.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 msgr2=0x7fa2180cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c7f0 tx=0x7fa20800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 shutdown_connections 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa1fc06c8b0 0x7fa1fc06ed70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa21813d120 0x7fa218070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2180b0a70 0x7fa2180cf000 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 --2- 172.21.15.107:0/733142456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa21806d6c0 0x7fa2180b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:31.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 >> 172.21.15.107:0/733142456 conn(0x7fa2180cb830 msgr2=0x7fa2180acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:31.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 shutdown_connections 2023-11-24T08:19:31.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:31.673+0000 7fa21e9ac640 1 -- 172.21.15.107:0/733142456 wait complete. 2023-11-24T08:19:31.692 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:19:31.693 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:19:31.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.693 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:19:31.754 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:19:31.755 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41497sec preferred_lft 41497sec 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.756 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:19:31.757 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:19:31.757 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:19:31.757 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:19:31.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.758 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:19:31.851 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:19:31.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.856 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:19:31.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:31.909 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:19:31.910 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:19:31.910 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:19:31.910 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:19:31.910 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:19:31.910 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:19:32.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:32.093 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:19:32.143 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:19:32.143 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:19:32.144 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:19:32.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:32.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:19:32.235 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:19:32.235 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:19:32.286 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:19:32.286 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:19:32.286 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:19:32.336 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:19:32.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:32.336 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:32.474 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:32.474 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:19:32.508 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:19:32.508 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:19:32.508 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:19:32.508 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:19:32.508 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:19:32.510 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:19:32.510 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:19:32.510 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:19:32.510 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:19:32.510 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:19:32.513 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:19:32.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:32.514 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:19:32.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:32.590 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:19:32.643 INFO:teuthology.orchestra.run.smithi120.stdout:client6251 2023-11-24T08:19:32.644 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:19:32.644 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:32.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:19:32.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.917+0000 7f95059ce640 1 Processor -- start 2023-11-24T08:19:32.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.917+0000 7f95059ce640 1 -- start start 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.918+0000 7f95059ce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 0x7f950006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.918+0000 7f95059ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f95000ce6c0 0x7f95000cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.918+0000 7f95059ce640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000cece0 con 0x7f95000cf450 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.918+0000 7f95059ce640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95000cfe40 con 0x7f95000ce6c0 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.918+0000 7f95059ce640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95000ceaf0 con 0x7f950015c7f0 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94fe7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f95000cf450 0x7f95000cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58514/0 (socket says 172.21.15.107:58514) 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94fe7fc640 1 -- 172.21.15.107:0/2080241075 learned_addr learned my addr 172.21.15.107:0/2080241075 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94ff7fe640 1 --2- 172.21.15.107:0/2080241075 >> v2:172.21.15.107:3300/0 conn(0x7f95000ce6c0 0x7f95000cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94feffd640 1 --2- 172.21.15.107:0/2080241075 >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 0x7f950006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94fdffb640 1 -- 172.21.15.107:0/2080241075 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204227867 0 0) 0x7f95000cece0 con 0x7f95000cf450 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94fdffb640 1 -- 172.21.15.107:0/2080241075 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003520 con 0x7f95000cf450 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94ff7fe640 1 -- 172.21.15.107:0/2080241075 >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 msgr2=0x7f950006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94ff7fe640 1 --2- 172.21.15.107:0/2080241075 >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 0x7f950006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94ff7fe640 1 -- 172.21.15.107:0/2080241075 >> v1:172.21.15.107:6789/0 conn(0x7f95000cf450 legacy=0x7f95000cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94ff7fe640 1 -- 172.21.15.107:0/2080241075 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950006d040 con 0x7f95000ce6c0 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.919+0000 7f94feffd640 1 --2- 172.21.15.107:0/2080241075 >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 0x7f950006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f94ff7fe640 1 --2- 172.21.15.107:0/2080241075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000ce6c0 0x7f95000cf900 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f95000d0130 tx=0x7f94f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a108eb501f7fb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f94fdffb640 1 -- 172.21.15.107:0/2080241075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f802cc80 con 0x7f95000ce6c0 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f94fdffb640 1 -- 172.21.15.107:0/2080241075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f802cde0 con 0x7f95000ce6c0 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f94fdffb640 1 -- 172.21.15.107:0/2080241075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8034660 con 0x7f95000ce6c0 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f95059ce640 1 -- 172.21.15.107:0/2080241075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000ce6c0 msgr2=0x7f95000cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.921+0000 7f95059ce640 1 --2- 172.21.15.107:0/2080241075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000ce6c0 0x7f95000cf900 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f95000d0130 tx=0x7f94f802bed0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 -- 172.21.15.107:0/2080241075 shutdown_connections 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 --2- 172.21.15.107:0/2080241075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000ce6c0 0x7f95000cf900 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 --2- 172.21.15.107:0/2080241075 >> v2:172.21.15.120:3300/0 conn(0x7f950015c7f0 0x7f950006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 -- 172.21.15.107:0/2080241075 >> 172.21.15.107:0/2080241075 conn(0x7f950006a920 msgr2=0x7f950006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:32.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 -- 172.21.15.107:0/2080241075 shutdown_connections 2023-11-24T08:19:32.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.922+0000 7f95059ce640 1 -- 172.21.15.107:0/2080241075 wait complete. 2023-11-24T08:19:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.923+0000 7f95059ce640 1 Processor -- start 2023-11-24T08:19:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.924+0000 7f95059ce640 1 -- start start 2023-11-24T08:19:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95000ce6c0 0x7f9500145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 0x7f95000af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95001569f0 con 0x7f95000cf450 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500142c20 con 0x7f950015c7f0 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f95059ce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9500155de0 con 0x7f95000ce6c0 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53022/0 (socket says 172.21.15.107:53022) 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 -- 172.21.15.107:0/3021635641 learned_addr learned my addr 172.21.15.107:0/3021635641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94feffd640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95000ce6c0 0x7f9500145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94ff7fe640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 0x7f95000af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 -- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95000ce6c0 msgr2=0x7f9500145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95000ce6c0 0x7f9500145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 -- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 msgr2=0x7f95000af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 0x7f95000af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94fe7fc640 1 -- 172.21.15.107:0/3021635641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8009a70 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.925+0000 7f94ff7fe640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 0x7f95000af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f94fe7fc640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f94e800c500 tx=0x7f94e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8005ce0 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e8005e40 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8007330 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95001560c0 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.926+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9500156680 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.927+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9500156ba0 con 0x7f95000cf450 2023-11-24T08:19:32.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.928+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f94e8007490 con 0x7f95000cf450 2023-11-24T08:19:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.929+0000 7f94f7fff640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 0x7f94e006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.929+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94e8087b40 con 0x7f95000cf450 2023-11-24T08:19:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.929+0000 7f94feffd640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 0x7f94e006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:32.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.930+0000 7f94feffd640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 0x7f94e006eda0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f94f0006fd0 tx=0x7f94f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:32.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:32.933+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9500156ba0 con 0x7f95000cf450 2023-11-24T08:19:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.159+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9500067470 con 0x7f95000cf450 2023-11-24T08:19:33.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.776+0000 7f94f7fff640 1 -- 172.21.15.107:0/3021635641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 136+0+0 (secure 0 0 0) 0x7f94e8017020 con 0x7f95000cf450 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 msgr2=0x7f94e006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 0x7f94e006eda0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f94f0006fd0 tx=0x7f94f0008040 comp rx=0 tx=0).stop 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 msgr2=0x7f95000ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f94e800c500 tx=0x7f94e800c9c0 comp rx=0 tx=0).stop 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 shutdown_connections 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94e006c8e0 0x7f94e006eda0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950015c7f0 0x7f95000af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:33.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95000cf450 0x7f95000ada20 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:33.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 --2- 172.21.15.107:0/3021635641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95000ce6c0 0x7f9500145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:33.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 >> 172.21.15.107:0/3021635641 conn(0x7f950006a920 msgr2=0x7f95000cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:33.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 shutdown_connections 2023-11-24T08:19:33.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:33.780+0000 7f95059ce640 1 -- 172.21.15.107:0/3021635641 wait complete. 2023-11-24T08:19:33.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:19:34.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.015+0000 7f8d7af41640 1 Processor -- start 2023-11-24T08:19:34.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.015+0000 7f8d7af41640 1 -- start start 2023-11-24T08:19:34.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.017+0000 7f8d7af41640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8d740af000 0x7f8d740b3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.017+0000 7f8d7af41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.017+0000 7f8d7af41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7406dca0 con 0x7f8d74161060 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.017+0000 7f8d7af41640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d740af3e0 con 0x7f8d740af000 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.017+0000 7f8d7af41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74161a10 con 0x7f8d7406b680 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.018+0000 7f8d794de640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.018+0000 7f8d78cdd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8d740af000 0x7f8d740b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.018+0000 7f8d794de640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:46224/0 (socket says 172.21.15.107:46224) 2023-11-24T08:19:34.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.018+0000 7f8d794de640 1 -- 172.21.15.107:0/1416505789 learned_addr learned my addr 172.21.15.107:0/1416505789 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.019+0000 7f8d794de640 1 -- 172.21.15.107:0/1416505789 >> v2:172.21.15.120:3300/0 conn(0x7f8d740af000 msgr2=0x7f8d740b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.019+0000 7f8d794de640 1 --2- 172.21.15.107:0/1416505789 >> v2:172.21.15.120:3300/0 conn(0x7f8d740af000 0x7f8d740b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.019+0000 7f8d794de640 1 -- 172.21.15.107:0/1416505789 >> v1:172.21.15.120:6789/0 conn(0x7f8d74161060 legacy=0x7f8d7415da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.019+0000 7f8d794de640 1 -- 172.21.15.107:0/1416505789 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7406df40 con 0x7f8d7406b680 2023-11-24T08:19:34.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.020+0000 7f8d794de640 1 --2- 172.21.15.107:0/1416505789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0098a0 tx=0x7f8d6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aa49baf6f4132ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.021+0000 7f8d6b7fe640 1 -- 172.21.15.107:0/1416505789 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c030840 con 0x7f8d7406b680 2023-11-24T08:19:34.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.021+0000 7f8d6b7fe640 1 -- 172.21.15.107:0/1416505789 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c0309a0 con 0x7f8d7406b680 2023-11-24T08:19:34.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.021+0000 7f8d6b7fe640 1 -- 172.21.15.107:0/1416505789 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c030d30 con 0x7f8d7406b680 2023-11-24T08:19:34.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.021+0000 7f8d7af41640 1 -- 172.21.15.107:0/1416505789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 msgr2=0x7f8d741614d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.021+0000 7f8d7af41640 1 --2- 172.21.15.107:0/1416505789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0098a0 tx=0x7f8d6c02b980 comp rx=0 tx=0).stop 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 -- 172.21.15.107:0/1416505789 shutdown_connections 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 --2- 172.21.15.107:0/1416505789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d741614d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 --2- 172.21.15.107:0/1416505789 >> v2:172.21.15.120:3300/0 conn(0x7f8d740af000 0x7f8d740b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 -- 172.21.15.107:0/1416505789 >> 172.21.15.107:0/1416505789 conn(0x7f8d740cf900 msgr2=0x7f8d740cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 -- 172.21.15.107:0/1416505789 shutdown_connections 2023-11-24T08:19:34.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.022+0000 7f8d7af41640 1 -- 172.21.15.107:0/1416505789 wait complete. 2023-11-24T08:19:34.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.023+0000 7f8d7af41640 1 Processor -- start 2023-11-24T08:19:34.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.024+0000 7f8d7af41640 1 -- start start 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.024+0000 7f8d7af41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d7414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.024+0000 7f8d7af41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.024+0000 7f8d7af41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 0x7f8d740adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d7af41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74150bc0 con 0x7f8d740af000 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d7af41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7415d4c0 con 0x7f8d74161060 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d7af41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d741532c0 con 0x7f8d7406b680 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53030/0 (socket says 172.21.15.107:53030) 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 -- 172.21.15.107:0/4119605087 learned_addr learned my addr 172.21.15.107:0/4119605087 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d78cdd640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d7414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d794de640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 0x7f8d740adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 -- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 msgr2=0x7f8d7414fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d7414fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 -- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 msgr2=0x7f8d740adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 0x7f8d740adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d6bfff640 1 -- 172.21.15.107:0/4119605087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c0094d0 con 0x7f8d740af000 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d78cdd640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d7414fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.025+0000 7f8d794de640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 0x7f8d740adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d6bfff640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400c450 tx=0x7f8d6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d640052d0 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d64005450 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d64003880 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d74153560 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.026+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d74153ae0 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.027+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7404e8b0 con 0x7f8d740af000 2023-11-24T08:19:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.028+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f8d640039e0 con 0x7f8d740af000 2023-11-24T08:19:34.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.029+0000 7f8d697fa640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 0x7f8d6006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.029+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d640880c0 con 0x7f8d740af000 2023-11-24T08:19:34.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.029+0000 7f8d78cdd640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 0x7f8d6006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.030+0000 7f8d78cdd640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 0x7f8d6006ed70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8d700074c0 tx=0x7f8d7000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.034+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d64054bd0 con 0x7f8d740af000 2023-11-24T08:19:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.250+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d7406e7e0 con 0x7f8d6006c8b0 2023-11-24T08:19:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.253+0000 7f8d697fa640 1 -- 172.21.15.107:0/4119605087 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8d7000f050 con 0x7f8d6006c8b0 2023-11-24T08:19:34.255 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:19:34.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:19:34.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:19:34.255 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:34.255 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:19:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 msgr2=0x7f8d6006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 0x7f8d6006ed70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8d700074c0 tx=0x7f8d7000b040 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 msgr2=0x7f8d740ada30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400c450 tx=0x7f8d6400c910 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 shutdown_connections 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8d6006c8b0 0x7f8d6006ed70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d74161060 0x7f8d740adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d740af000 0x7f8d740ada30 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 --2- 172.21.15.107:0/4119605087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7406b680 0x7f8d7414fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.255+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 >> 172.21.15.107:0/4119605087 conn(0x7f8d740cf900 msgr2=0x7f8d741567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.256+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 shutdown_connections 2023-11-24T08:19:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.256+0000 7f8d7af41640 1 -- 172.21.15.107:0/4119605087 wait complete. 2023-11-24T08:19:34.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:19:34.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.490+0000 7fd0b4f78640 1 Processor -- start 2023-11-24T08:19:34.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.491+0000 7fd0b4f78640 1 -- start start 2023-11-24T08:19:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.492+0000 7fd0b4f78640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0b00cc0f0 0x7fd0b00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.492+0000 7fd0b4f78640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b015eea0 con 0x7fd0b00c57e0 2023-11-24T08:19:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.492+0000 7fd0b4f78640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00abf30 con 0x7fd0b015b440 2023-11-24T08:19:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.492+0000 7fd0b4f78640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b015b820 con 0x7fd0b00cc0f0 2023-11-24T08:19:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0aed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd0b00c57e0 0x7fd0b006c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58520/0 (socket says 172.21.15.107:58520) 2023-11-24T08:19:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0aed9d640 1 -- 172.21.15.107:0/1170338512 learned_addr learned my addr 172.21.15.107:0/1170338512 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/1170338512 >> v2:172.21.15.120:3300/0 conn(0x7fd0b00cc0f0 0x7fd0b00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194773715 0 0) 0x7fd0b015eea0 con 0x7fd0b00c57e0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd09c003520 con 0x7fd0b00c57e0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288642405 0 0) 0x7fd09c003520 con 0x7fd0b00c57e0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 >> v2:172.21.15.120:3300/0 conn(0x7fd0b00cc0f0 msgr2=0x7fd0b00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 --2- 172.21.15.107:0/1170338512 >> v2:172.21.15.120:3300/0 conn(0x7fd0b00cc0f0 0x7fd0b00c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 >> v1:172.21.15.120:6789/0 conn(0x7fd0b015b440 legacy=0x7fd0b00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b015f0f0 con 0x7fd0b00c57e0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.493+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd0a80019f0 con 0x7fd0b00c57e0 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 >> v1:172.21.15.107:6789/0 conn(0x7fd0b00c57e0 legacy=0x7fd0b006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd09c0039c0 0x7fd09c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 0x7fd09c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0051d0 0x7fd09c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015b820 con 0x7fd09c0047c0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c00c120 con 0x7fd09c0051d0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c00c2e0 con 0x7fd09c0039c0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 0x7fd09c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0aed9d640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0051d0 0x7fd09c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd09c0039c0 0x7fd09c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 -- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd09c0039c0 msgr2=0x7fd09c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd09c0039c0 0x7fd09c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 -- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0051d0 msgr2=0x7fd09c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0051d0 0x7fd09c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0a7fff640 1 -- 172.21.15.107:0/1170338512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c003520 con 0x7fd09c0047c0 2023-11-24T08:19:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0aed9d640 1 -- 172.21.15.107:0/1170338512 reap_dead start 2023-11-24T08:19:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.494+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd09c0039c0 0x7fd09c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.496+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 0x7fd09c004c40 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fd098009a90 tx=0x7fd09c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d47dc1cdc70575 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.496+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09802c680 con 0x7fd09c0047c0 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.496+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09802c7e0 con 0x7fd09c0047c0 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.496+0000 7fd0add9b640 1 -- 172.21.15.107:0/1170338512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09802cb70 con 0x7fd09c0047c0 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 -- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 msgr2=0x7fd09c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 0x7fd09c004c40 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fd098009a90 tx=0x7fd09c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 -- 172.21.15.107:0/1170338512 shutdown_connections 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/1170338512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd09c0047c0 0x7fd09c004c40 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 -- 172.21.15.107:0/1170338512 >> 172.21.15.107:0/1170338512 conn(0x7fd0b00b0c50 msgr2=0x7fd0b00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 -- 172.21.15.107:0/1170338512 shutdown_connections 2023-11-24T08:19:34.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.497+0000 7fd0b4f78640 1 -- 172.21.15.107:0/1170338512 wait complete. 2023-11-24T08:19:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.499+0000 7fd0b4f78640 1 Processor -- start 2023-11-24T08:19:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.499+0000 7fd0b4f78640 1 -- start start 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 0x7fd0b014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b00c8270 0x7fd0b00c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00c8c30 con 0x7fd0b00b3270 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00b21d0 con 0x7fd09c0047c0 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0b4f78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00b24b0 con 0x7fd0b00c8270 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53048/0 (socket says 172.21.15.107:53048) 2023-11-24T08:19:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0aed9d640 1 -- 172.21.15.107:0/735602540 learned_addr learned my addr 172.21.15.107:0/735602540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 0x7fd0b014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.500+0000 7fd0a7fff640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b00c8270 0x7fd0b00c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 -- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b00c8270 msgr2=0x7fd0b00c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b00c8270 0x7fd0b00c86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 -- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 msgr2=0x7fd0b014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 0x7fd0b014ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 -- 172.21.15.107:0/735602540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00b2790 con 0x7fd0b00b3270 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 0x7fd0b014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0aed9d640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006e890 tx=0x7fd0a8007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800eaa0 con 0x7fd0b00b3270 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a800ec00 con 0x7fd0b00b3270 2023-11-24T08:19:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8013420 con 0x7fd0b00b3270 2023-11-24T08:19:34.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b00b2a70 con 0x7fd0b00b3270 2023-11-24T08:19:34.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.501+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b014f490 con 0x7fd0b00b3270 2023-11-24T08:19:34.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.503+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b004e8b0 con 0x7fd0b00b3270 2023-11-24T08:19:34.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.503+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fd0a8013580 con 0x7fd0b00b3270 2023-11-24T08:19:34.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.504+0000 7fd0a6ffd640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 0x7fd09006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.504+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0a80887f0 con 0x7fd0b00b3270 2023-11-24T08:19:34.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.505+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 0x7fd09006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.505+0000 7fd0ae59c640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 0x7fd09006ed70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0064320 tx=0x7fd0a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.509+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0a80553a0 con 0x7fd0b00b3270 2023-11-24T08:19:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.699+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd0b006dc80 con 0x7fd0b00b3270 2023-11-24T08:19:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.700+0000 7fd0a6ffd640 1 -- 172.21.15.107:0/735602540 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v47)=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd0a80589c0 con 0x7fd0b00b3270 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 msgr2=0x7fd09006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 0x7fd09006ed70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0064320 tx=0x7fd0a0009040 comp rx=0 tx=0).stop 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 msgr2=0x7fd0b00b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006e890 tx=0x7fd0a8007bc0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 shutdown_connections 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd09006c8b0 0x7fd09006ed70 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0b00b3270 0x7fd0b00b36f0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b00c8270 0x7fd0b00c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 --2- 172.21.15.107:0/735602540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd09c0047c0 0x7fd0b014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 >> 172.21.15.107:0/735602540 conn(0x7fd0b00b0c50 msgr2=0x7fd0b0071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:34.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 shutdown_connections 2023-11-24T08:19:34.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.703+0000 7fd0b4f78640 1 -- 172.21.15.107:0/735602540 wait complete. 2023-11-24T08:19:34.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol4786 --mode=777 --size 12582912 2023-11-24T08:19:34.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.928+0000 7f351591d640 1 Processor -- start 2023-11-24T08:19:34.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.928+0000 7f351591d640 1 -- start start 2023-11-24T08:19:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f351591d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35100c59b0 0x7f3510166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f351591d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100c5dc0 con 0x7f351015b440 2023-11-24T08:19:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f351591d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351015eea0 con 0x7f35100cc0f0 2023-11-24T08:19:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f351591d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351015b820 con 0x7f35100c59b0 2023-11-24T08:19:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f350effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f351015b440 0x7f35100cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58532/0 (socket says 172.21.15.107:58532) 2023-11-24T08:19:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.930+0000 7f350effd640 1 -- 172.21.15.107:0/3265681515 learned_addr learned my addr 172.21.15.107:0/3265681515 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350f7fe640 1 --2- 172.21.15.107:0/3265681515 >> v2:172.21.15.120:3300/0 conn(0x7f35100c59b0 0x7f3510166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812484373 0 0) 0x7f35100c5dc0 con 0x7f351015b440 2023-11-24T08:19:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34fc003520 con 0x7f351015b440 2023-11-24T08:19:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721590609 0 0) 0x7f34fc003520 con 0x7f351015b440 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 >> v2:172.21.15.120:3300/0 conn(0x7f35100c59b0 msgr2=0x7f3510166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 --2- 172.21.15.107:0/3265681515 >> v2:172.21.15.120:3300/0 conn(0x7f35100c59b0 0x7f3510166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 >> v1:172.21.15.120:6789/0 conn(0x7f35100cc0f0 legacy=0x7f35100c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351006d600 con 0x7f351015b440 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3500001670 con 0x7f351015b440 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 >> v1:172.21.15.107:6789/0 conn(0x7f351015b440 legacy=0x7f35100cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34fc003970 0x7f34fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 0x7f34fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc005180 0x7f34fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351015b820 con 0x7f34fc0047c0 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc00c120 con 0x7f34fc003970 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.931+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc00c2e0 con 0x7f34fc005180 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 0x7f34fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f350effd640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34fc003970 0x7f34fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 -- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc005180 msgr2=0x7f34fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc005180 0x7f34fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 -- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34fc003970 msgr2=0x7f34fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34fc003970 0x7f34fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f3507fff640 1 -- 172.21.15.107:0/3265681515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc003520 con 0x7f34fc0047c0 2023-11-24T08:19:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.932+0000 7f350f7fe640 1 -- 172.21.15.107:0/3265681515 reap_dead start 2023-11-24T08:19:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f3507fff640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 0x7f34fc004c40 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f34f8001750 tx=0x7f34fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c4a9ba9bb63182 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80052d0 con 0x7f34fc0047c0 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f802e050 con 0x7f34fc0047c0 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f350e7fc640 1 -- 172.21.15.107:0/3265681515 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8003830 con 0x7f34fc0047c0 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f351591d640 1 -- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 msgr2=0x7f34fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.934+0000 7f351591d640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 0x7f34fc004c40 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f34f8001750 tx=0x7f34fc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.935+0000 7f351591d640 1 -- 172.21.15.107:0/3265681515 shutdown_connections 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.935+0000 7f351591d640 1 --2- 172.21.15.107:0/3265681515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34fc0047c0 0x7f34fc004c40 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.935+0000 7f351591d640 1 -- 172.21.15.107:0/3265681515 >> 172.21.15.107:0/3265681515 conn(0x7f35100b0c50 msgr2=0x7f35100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.935+0000 7f351591d640 1 -- 172.21.15.107:0/3265681515 shutdown_connections 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.935+0000 7f351591d640 1 -- 172.21.15.107:0/3265681515 wait complete. 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.936+0000 7f351591d640 1 Processor -- start 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.937+0000 7f351591d640 1 -- start start 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.937+0000 7f351591d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc0047c0 0x7f3510150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.937+0000 7f351591d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35100af9c0 0x7f35100afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.937+0000 7f351591d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f351591d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100b0380 con 0x7f35101555c0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f351591d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510155f80 con 0x7f35100af9c0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f351591d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3510156cb0 con 0x7f34fc0047c0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53062/0 (socket says 172.21.15.107:53062) 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 -- 172.21.15.107:0/1756499979 learned_addr learned my addr 172.21.15.107:0/1756499979 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f3507fff640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35100af9c0 0x7f35100afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350effd640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc0047c0 0x7f3510150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 -- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc0047c0 msgr2=0x7f3510150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc0047c0 0x7f3510150e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 -- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35100af9c0 msgr2=0x7f35100afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35100af9c0 0x7f35100afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.938+0000 7f350f7fe640 1 -- 172.21.15.107:0/1756499979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510156760 con 0x7f35101555c0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f350f7fe640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f351006dc00 tx=0x7f350800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508010040 con 0x7f35101555c0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350800bcb0 con 0x7f35101555c0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508014640 con 0x7f35101555c0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101561a0 con 0x7f35101555c0 2023-11-24T08:19:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.939+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351015ba90 con 0x7f35101555c0 2023-11-24T08:19:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.940+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351006ef60 con 0x7f35101555c0 2023-11-24T08:19:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.941+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f35080147a0 con 0x7f35101555c0 2023-11-24T08:19:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.942+0000 7f350cff9640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 0x7f34f006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.942+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3508088a40 con 0x7f35101555c0 2023-11-24T08:19:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.942+0000 7f350effd640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 0x7f34f006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:34.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.943+0000 7f350effd640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 0x7f34f006eda0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3500001eb0 tx=0x7f3500000f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:34.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:34.947+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3508055550 con 0x7f35101555c0 2023-11-24T08:19:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.169+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol4786", "mode": "777", "size": 12582912, "target": ["mon-mgr", ""]}) v1 -- 0x7f351015f3d0 con 0x7f34f006c8e0 2023-11-24T08:19:35.184 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:35.182+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:35.184 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:35.182+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:35.184 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:35.182+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:35.185 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:35.182+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:35.185 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:35.182+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:35.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.246+0000 7f350cff9640 1 -- 172.21.15.107:0/1756499979 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3500005da0 con 0x7f34f006c8e0 2023-11-24T08:19:35.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.249+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 msgr2=0x7f34f006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.249+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 0x7f34f006eda0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3500001eb0 tx=0x7f3500000f10 comp rx=0 tx=0).stop 2023-11-24T08:19:35.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.249+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 msgr2=0x7f3510155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.249+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f351006dc00 tx=0x7f350800b480 comp rx=0 tx=0).stop 2023-11-24T08:19:35.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 shutdown_connections 2023-11-24T08:19:35.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f34f006c8e0 0x7f34f006eda0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101555c0 0x7f3510155a40 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35100af9c0 0x7f35100afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 --2- 172.21.15.107:0/1756499979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc0047c0 0x7f3510150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 >> 172.21.15.107:0/1756499979 conn(0x7f35100b0c50 msgr2=0x7f35100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 shutdown_connections 2023-11-24T08:19:35.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.250+0000 7f351591d640 1 -- 172.21.15.107:0/1756499979 wait complete. 2023-11-24T08:19:35.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4786 2023-11-24T08:19:35.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.481+0000 7f2b0d6ec640 1 Processor -- start 2023-11-24T08:19:35.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.481+0000 7f2b0d6ec640 1 -- start start 2023-11-24T08:19:35.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.482+0000 7f2b0d6ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2b080aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b0d6ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0815aef0 con 0x7f2b0806e6c0 2023-11-24T08:19:35.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b0d6ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08159d70 con 0x7f2b080cf430 2023-11-24T08:19:35.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b0d6ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0806e2a0 con 0x7f2b080b3c00 2023-11-24T08:19:35.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b077fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2b0806e6c0 0x7f2b0806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58538/0 (socket says 172.21.15.107:58538) 2023-11-24T08:19:35.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b077fe640 1 -- 172.21.15.107:0/3896284413 learned_addr learned my addr 172.21.15.107:0/3896284413 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b067fc640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2b080aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280739893 0 0) 0x7f2b0815aef0 con 0x7f2b0806e6c0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.483+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af4003520 con 0x7f2b0806e6c0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012656287 0 0) 0x7f2af4003520 con 0x7f2b0806e6c0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 msgr2=0x7f2b080aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2b080aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 >> v1:172.21.15.120:6789/0 conn(0x7f2b080cf430 legacy=0x7f2b080b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0806d900 con 0x7f2b0806e6c0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2b00001670 con 0x7f2b0806e6c0 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 >> v1:172.21.15.107:6789/0 conn(0x7f2b0806e6c0 legacy=0x7f2b0806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2af4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2af40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af4004d00 0x7f2af4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0806e2a0 con 0x7f2af40039a0 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af400bb90 con 0x7f2af4004d00 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af400df60 con 0x7f2b080b3c00 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2af4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.484+0000 7f2b067fc640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2af40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b077fe640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af4004d00 0x7f2af4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b06ffd640 1 -- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 msgr2=0x7f2af40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b080b3c00 0x7f2af40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b06ffd640 1 -- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af4004d00 msgr2=0x7f2af4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2af4004d00 0x7f2af4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.485+0000 7f2b06ffd640 1 -- 172.21.15.107:0/3896284413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4003520 con 0x7f2af40039a0 2023-11-24T08:19:35.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.486+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2af4003dc0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f2af8001750 tx=0x7f2af40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0c47626416f8f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:35.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.486+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80052d0 con 0x7f2af40039a0 2023-11-24T08:19:35.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.486+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af802e050 con 0x7f2af40039a0 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.486+0000 7f2b05ffb640 1 -- 172.21.15.107:0/3896284413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80035f0 con 0x7f2af40039a0 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/3896284413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 msgr2=0x7f2af4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/3896284413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2af4003dc0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f2af8001750 tx=0x7f2af40057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b077fe640 1 -- 172.21.15.107:0/3896284413 reap_dead start 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/3896284413 shutdown_connections 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/3896284413 >> 172.21.15.107:0/3896284413 conn(0x7f2b080cf900 msgr2=0x7f2b080cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/3896284413 shutdown_connections 2023-11-24T08:19:35.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.487+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/3896284413 wait complete. 2023-11-24T08:19:35.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.489+0000 7f2b0d6ec640 1 Processor -- start 2023-11-24T08:19:35.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.489+0000 7f2b0d6ec640 1 -- start start 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b0815cc80 0x7f2b08154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 0x7f2b080ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08154d40 con 0x7f2af40039a0 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08156310 con 0x7f2b080ce2e0 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b0d6ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081565f0 con 0x7f2b0815cc80 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b06ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b06ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53092/0 (socket says 172.21.15.107:53092) 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.490+0000 7f2b06ffd640 1 -- 172.21.15.107:0/618053698 learned_addr learned my addr 172.21.15.107:0/618053698 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b067fc640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b0815cc80 0x7f2b08154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b077fe640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 0x7f2b080ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 -- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b0815cc80 msgr2=0x7f2b08154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b0815cc80 0x7f2b08154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 -- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 msgr2=0x7f2b080ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 0x7f2b080ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 -- 172.21.15.107:0/618053698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b081568d0 con 0x7f2af40039a0 2023-11-24T08:19:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b077fe640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 0x7f2b080ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.491+0000 7f2b06ffd640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f2af8008fd0 tx=0x7f2af8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.492+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80076c0 con 0x7f2af40039a0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.492+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af8007a30 con 0x7f2af40039a0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.492+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b08156bb0 con 0x7f2af40039a0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.492+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8006c00 con 0x7f2af40039a0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.492+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b080addd0 con 0x7f2af40039a0 2023-11-24T08:19:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.493+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0804e8b0 con 0x7f2af40039a0 2023-11-24T08:19:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.494+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 36) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f2af80035f0 con 0x7f2af40039a0 2023-11-24T08:19:35.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.495+0000 7f2aff7fe640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 0x7f2ae806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:35.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.496+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2af80aa510 con 0x7f2af40039a0 2023-11-24T08:19:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.496+0000 7f2b067fc640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 0x7f2ae806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.496+0000 7f2b067fc640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 0x7f2ae806eda0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2b0815d080 tx=0x7f2af000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:35.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.501+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2af8077020 con 0x7f2af40039a0 2023-11-24T08:19:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.720+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4786", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b08156760 con 0x7f2ae806c8e0 2023-11-24T08:19:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.729+0000 7f2aff7fe640 1 -- 172.21.15.107:0/618053698 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f2b08156760 con 0x7f2ae806c8e0 2023-11-24T08:19:35.731 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 msgr2=0x7f2ae806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 0x7f2ae806eda0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2b0815d080 tx=0x7f2af000b040 comp rx=0 tx=0).stop 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 msgr2=0x7f2b081573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f2af8008fd0 tx=0x7f2af8001120 comp rx=0 tx=0).stop 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 shutdown_connections 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ae806c8e0 0x7f2ae806eda0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b080ce2e0 0x7f2b080ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b0815cc80 0x7f2b08154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 --2- 172.21.15.107:0/618053698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af40039a0 0x7f2b081573b0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 >> 172.21.15.107:0/618053698 conn(0x7f2b080cf900 msgr2=0x7f2b08158430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 shutdown_connections 2023-11-24T08:19:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:35.732+0000 7f2b0d6ec640 1 -- 172.21.15.107:0/618053698 wait complete. 2023-11-24T08:19:35.752 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol4786 with 8 files each 1MB size under directory volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411 2023-11-24T08:19:35.752 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:35.752 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:35.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:35.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:35.945 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:35.945 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:35.945 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032971 s, 31.8 MB/s 2023-11-24T08:19:35.947 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:35.947 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352822 s, 29.7 MB/s 2023-11-24T08:19:36.090 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.090 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249373 s, 42.0 MB/s 2023-11-24T08:19:36.234 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.235 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.363 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022969 s, 45.7 MB/s 2023-11-24T08:19:36.364 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.364 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.419 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021014 s, 49.9 MB/s 2023-11-24T08:19:36.492 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.492 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.548 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.623 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.624 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230512 s, 45.5 MB/s 2023-11-24T08:19:36.625 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.625 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.682 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264192 s, 39.7 MB/s 2023-11-24T08:19:36.761 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:19:36.761 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:19:36.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:36.822 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:19:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:19:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:19:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215851 s, 48.6 MB/s 2023-11-24T08:19:36.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4786 2023-11-24T08:19:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.108+0000 7f7cef3db640 1 Processor -- start 2023-11-24T08:19:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.108+0000 7f7cef3db640 1 -- start start 2023-11-24T08:19:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.109+0000 7f7cef3db640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ce80b0b40 0x7f7ce80b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.109+0000 7f7cef3db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce80b14d0 con 0x7f7ce80cce40 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cef3db640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce80ae5d0 con 0x7f7ce815d310 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cef3db640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce806cd10 con 0x7f7ce80b0b40 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ce80b0b40 0x7f7ce80b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ce80b0b40 0x7f7ce80b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53102/0 (socket says 172.21.15.107:53102) 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1825559370 learned_addr learned my addr 172.21.15.107:0/1825559370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1825559370 >> v1:172.21.15.120:6789/0 conn(0x7f7ce815d310 legacy=0x7f7ce80ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1825559370 >> v1:172.21.15.107:6789/0 conn(0x7f7ce80cce40 legacy=0x7f7ce80cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.110+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1825559370 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce80ae8b0 con 0x7f7ce80b0b40 2023-11-24T08:19:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.111+0000 7f7cedbd8640 1 --2- 172.21.15.107:0/1825559370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80b0b40 0x7f7ce80b0f90 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6efb9b36d8d283 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7ced3d7640 1 -- 172.21.15.107:0/1825559370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc030850 con 0x7f7ce80b0b40 2023-11-24T08:19:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7ced3d7640 1 -- 172.21.15.107:0/1825559370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc0309b0 con 0x7f7ce80b0b40 2023-11-24T08:19:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7ced3d7640 1 -- 172.21.15.107:0/1825559370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc030d40 con 0x7f7ce80b0b40 2023-11-24T08:19:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7cef3db640 1 -- 172.21.15.107:0/1825559370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80b0b40 msgr2=0x7f7ce80b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1825559370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80b0b40 0x7f7ce80b0f90 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b990 comp rx=0 tx=0).stop 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7cef3db640 1 -- 172.21.15.107:0/1825559370 shutdown_connections 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1825559370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80b0b40 0x7f7ce80b0f90 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.112+0000 7f7cef3db640 1 -- 172.21.15.107:0/1825559370 >> 172.21.15.107:0/1825559370 conn(0x7f7ce80cbb40 msgr2=0x7f7ce80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.113+0000 7f7cef3db640 1 -- 172.21.15.107:0/1825559370 shutdown_connections 2023-11-24T08:19:37.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.113+0000 7f7cef3db640 1 -- 172.21.15.107:0/1825559370 wait complete. 2023-11-24T08:19:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.114+0000 7f7cef3db640 1 Processor -- start 2023-11-24T08:19:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.114+0000 7f7cef3db640 1 -- start start 2023-11-24T08:19:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 0x7f7ce8099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ce815d310 0x7f7ce80c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce81560c0 con 0x7f7ce80cce40 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce80c4310 con 0x7f7ce80b0b40 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.115+0000 7f7cef3db640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce80c7860 con 0x7f7ce815d310 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53118/0 (socket says 172.21.15.107:53118) 2023-11-24T08:19:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1778242628 learned_addr learned my addr 172.21.15.107:0/1778242628 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7ceebda640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ce815d310 0x7f7ce80c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ce815d310 msgr2=0x7f7ce80c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ce815d310 0x7f7ce80c39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cee3d9640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 0x7f7ce8099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 msgr2=0x7f7ce8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 0x7f7ce8099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cedbd8640 1 -- 172.21.15.107:0/1778242628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0094d0 con 0x7f7ce80cce40 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.116+0000 7f7cee3d9640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 0x7f7ce8099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cedbd8640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc030800 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc043400 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0437b0 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce80c7b40 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.117+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce80c3f20 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.118+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce80c8060 con 0x7f7ce80cce40 2023-11-24T08:19:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.119+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 37) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f7cdc02e070 con 0x7f7ce80cce40 2023-11-24T08:19:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.120+0000 7f7cdaffd640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 0x7f7ccc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.120+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cdc03e070 con 0x7f7ce80cce40 2023-11-24T08:19:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.120+0000 7f7cee3d9640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 0x7f7ccc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.121+0000 7f7cee3d9640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 0x7f7ccc06ed70 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400a830 tx=0x7f7ce4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.124+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7cdc078380 con 0x7f7ce80cce40 2023-11-24T08:19:37.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.343+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4786", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ce806c930 con 0x7f7ccc06c8b0 2023-11-24T08:19:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.363+0000 7f7cdaffd640 1 -- 172.21.15.107:0/1778242628 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f7ce400adb0 con 0x7f7ccc06c8b0 2023-11-24T08:19:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.365+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 msgr2=0x7f7ccc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.365+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 0x7f7ccc06ed70 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400a830 tx=0x7f7ce4008040 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.365+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 msgr2=0x7f7ce8155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.365+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc004730 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 shutdown_connections 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7ccc06c8b0 0x7f7ccc06ed70 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ce815d310 0x7f7ce80c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce80cce40 0x7f7ce8155b80 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 --2- 172.21.15.107:0/1778242628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ce80b0b40 0x7f7ce8099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 >> 172.21.15.107:0/1778242628 conn(0x7f7ce80cbb40 msgr2=0x7f7ce80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 shutdown_connections 2023-11-24T08:19:37.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.366+0000 7f7cef3db640 1 -- 172.21.15.107:0/1778242628 wait complete. 2023-11-24T08:19:37.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:19:37.387 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.quota.max_files -v 20 volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:19:37.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol4786 subvol_snap8897 2023-11-24T08:19:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.714+0000 7f9756032640 1 Processor -- start 2023-11-24T08:19:37.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.715+0000 7f9756032640 1 -- start start 2023-11-24T08:19:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.716+0000 7f9756032640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f975015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.716+0000 7f9756032640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975015ab80 con 0x7f97500b2630 2023-11-24T08:19:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.716+0000 7f9756032640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975015e680 con 0x7f97500b2a70 2023-11-24T08:19:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.716+0000 7f9756032640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975015a990 con 0x7f97500b2250 2023-11-24T08:19:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f9747fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f97500b2630 0x7f97500ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58548/0 (socket says 172.21.15.107:58548) 2023-11-24T08:19:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f9747fff640 1 -- 172.21.15.107:0/3323903972 learned_addr learned my addr 172.21.15.107:0/3323903972 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974f7fe640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f975015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085489196 0 0) 0x7f975015ab80 con 0x7f97500b2630 2023-11-24T08:19:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c003520 con 0x7f97500b2630 2023-11-24T08:19:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047615323 0 0) 0x7f973c003520 con 0x7f97500b2630 2023-11-24T08:19:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 msgr2=0x7f975015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f975015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 >> v1:172.21.15.120:6789/0 conn(0x7f97500b2a70 legacy=0x7f975015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975015e960 con 0x7f97500b2630 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.717+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9738001670 con 0x7f97500b2630 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 >> v1:172.21.15.107:6789/0 conn(0x7f97500b2630 legacy=0x7f97500ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c0039a0 0x7f973c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f973c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c004d00 0x7f973c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c00ba30 con 0x7f973c004d00 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c00bb70 con 0x7f973c0039a0 2023-11-24T08:19:37.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c00bd30 con 0x7f97500b2250 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c004d00 0x7f973c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f9747fff640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f973c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 -- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 msgr2=0x7f973c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97500b2250 0x7f973c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 -- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c0039a0 msgr2=0x7f973c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c0039a0 0x7f973c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.718+0000 7f974ffff640 1 -- 172.21.15.107:0/3323903972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c003520 con 0x7f973c004d00 2023-11-24T08:19:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.719+0000 7f974ffff640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c004d00 0x7f973c005180 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f9748001a30 tx=0x7f973c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fbc174aec52306 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748003b70 con 0x7f973c004d00 2023-11-24T08:19:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748003cd0 con 0x7f973c004d00 2023-11-24T08:19:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f974effd640 1 -- 172.21.15.107:0/3323903972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748005310 con 0x7f973c004d00 2023-11-24T08:19:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 -- 172.21.15.107:0/3323903972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c004d00 msgr2=0x7f973c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 --2- 172.21.15.107:0/3323903972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c004d00 0x7f973c005180 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f9748001a30 tx=0x7f973c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f974ffff640 1 -- 172.21.15.107:0/3323903972 reap_dead start 2023-11-24T08:19:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 -- 172.21.15.107:0/3323903972 shutdown_connections 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 -- 172.21.15.107:0/3323903972 >> 172.21.15.107:0/3323903972 conn(0x7f975006a920 msgr2=0x7f975006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 -- 172.21.15.107:0/3323903972 shutdown_connections 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.720+0000 7f9756032640 1 -- 172.21.15.107:0/3323903972 wait complete. 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.722+0000 7f9756032640 1 Processor -- start 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.722+0000 7f9756032640 1 -- start start 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c004d00 0x7f97500b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f975014c460 0x7f97500cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500b1aa0 con 0x7f975014c8e0 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750152010 con 0x7f973c004d00 2023-11-24T08:19:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f9756032640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97501516e0 con 0x7f975014c460 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.723+0000 7f974ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53138/0 (socket says 172.21.15.107:53138) 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 -- 172.21.15.107:0/2999502434 learned_addr learned my addr 172.21.15.107:0/2999502434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 -- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f975014c460 msgr2=0x7f97500cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f975014c460 0x7f97500cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 -- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c004d00 msgr2=0x7f97500b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c004d00 0x7f97500b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 -- 172.21.15.107:0/2999502434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500cd7c0 con 0x7f975014c8e0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974ffff640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f9748008fd0 tx=0x7f97480014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748005600 con 0x7f975014c8e0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748005760 con 0x7f975014c8e0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974802e7e0 con 0x7f975014c8e0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.724+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97501519c0 con 0x7f975014c8e0 2023-11-24T08:19:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.725+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97501524a0 con 0x7f975014c8e0 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.726+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9750067470 con 0x7f975014c8e0 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.727+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 37) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f974802e940 con 0x7f975014c8e0 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.727+0000 7f974cff9640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 0x7f973006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.728+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97480ab040 con 0x7f975014c8e0 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.728+0000 7f974f7fe640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 0x7f973006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:37.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.728+0000 7f974f7fe640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 0x7f973006ed70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f97501f2aa0 tx=0x7f97400094e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:37.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.732+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9748077cd0 con 0x7f975014c8e0 2023-11-24T08:19:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.960+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol4786", "snap_name": "subvol_snap8897", "target": ["mon-mgr", ""]}) v1 -- 0x7f97500cd640 con 0x7f973006c8b0 2023-11-24T08:19:37.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.989+0000 7f974cff9640 1 -- 172.21.15.107:0/2999502434 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9740009670 con 0x7f973006c8b0 2023-11-24T08:19:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 msgr2=0x7f973006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 0x7f973006ed70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f97501f2aa0 tx=0x7f97400094e0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 msgr2=0x7f97500cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:37.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f9748008fd0 tx=0x7f97480014a0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 shutdown_connections 2023-11-24T08:19:37.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f973006c8b0 0x7f973006ed70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f975014c8e0 0x7f97500cf960 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f975014c460 0x7f97500cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 --2- 172.21.15.107:0/2999502434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f973c004d00 0x7f97500b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 >> 172.21.15.107:0/2999502434 conn(0x7f975006a920 msgr2=0x7f975015bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 shutdown_connections 2023-11-24T08:19:37.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:37.992+0000 7f9756032640 1 -- 172.21.15.107:0/2999502434 wait complete. 2023-11-24T08:19:38.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol4786 subvol_snap8897 subvol_clone8559 2023-11-24T08:19:38.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.230+0000 7fc5e5be5640 1 Processor -- start 2023-11-24T08:19:38.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.231+0000 7fc5e5be5640 1 -- start start 2023-11-24T08:19:38.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.232+0000 7fc5e5be5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e00d2000 0x7fc5e00cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.232+0000 7fc5e5be5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00ce410 con 0x7fc5e00d0200 2023-11-24T08:19:38.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.232+0000 7fc5e5be5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00d06f0 con 0x7fc5e00d1c20 2023-11-24T08:19:38.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.232+0000 7fc5e5be5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e006c270 con 0x7fc5e00d2000 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.232+0000 7fc5deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e00d2000 0x7fc5e00cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.233+0000 7fc5deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e00d2000 0x7fc5e00cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52194/0 (socket says 172.21.15.107:52194) 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.233+0000 7fc5deffd640 1 -- 172.21.15.107:0/2825684662 learned_addr learned my addr 172.21.15.107:0/2825684662 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.233+0000 7fc5deffd640 1 -- 172.21.15.107:0/2825684662 >> v1:172.21.15.120:6789/0 conn(0x7fc5e00d1c20 legacy=0x7fc5e006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.233+0000 7fc5deffd640 1 -- 172.21.15.107:0/2825684662 >> v1:172.21.15.107:6789/0 conn(0x7fc5e00d0200 legacy=0x7fc5e01605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.233+0000 7fc5deffd640 1 -- 172.21.15.107:0/2825684662 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00d09d0 con 0x7fc5e00d2000 2023-11-24T08:19:38.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.234+0000 7fc5deffd640 1 --2- 172.21.15.107:0/2825684662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d2000 0x7fc5e00cfcc0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009870 tx=0x7fc5c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb83cf1447147906 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.234+0000 7fc5de7fc640 1 -- 172.21.15.107:0/2825684662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8030860 con 0x7fc5e00d2000 2023-11-24T08:19:38.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.234+0000 7fc5de7fc640 1 -- 172.21.15.107:0/2825684662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80309c0 con 0x7fc5e00d2000 2023-11-24T08:19:38.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.234+0000 7fc5de7fc640 1 -- 172.21.15.107:0/2825684662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8030d50 con 0x7fc5e00d2000 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 -- 172.21.15.107:0/2825684662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d2000 msgr2=0x7fc5e00cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/2825684662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d2000 0x7fc5e00cfcc0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009870 tx=0x7fc5c802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 -- 172.21.15.107:0/2825684662 shutdown_connections 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/2825684662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d2000 0x7fc5e00cfcc0 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 -- 172.21.15.107:0/2825684662 >> 172.21.15.107:0/2825684662 conn(0x7fc5e008cea0 msgr2=0x7fc5e009c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:38.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 -- 172.21.15.107:0/2825684662 shutdown_connections 2023-11-24T08:19:38.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.235+0000 7fc5e5be5640 1 -- 172.21.15.107:0/2825684662 wait complete. 2023-11-24T08:19:38.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.237+0000 7fc5e5be5640 1 Processor -- start 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.237+0000 7fc5e5be5640 1 -- start start 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5e00d1c20 0x7fc5e00a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 0x7fc5e015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00a0f70 con 0x7fc5e00d0200 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e015c090 con 0x7fc5e00d1c20 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5e5be5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e015c330 con 0x7fc5e00d2000 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52210/0 (socket says 172.21.15.107:52210) 2023-11-24T08:19:38.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.238+0000 7fc5df7fe640 1 -- 172.21.15.107:0/349688312 learned_addr learned my addr 172.21.15.107:0/349688312 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 -- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 msgr2=0x7fc5e015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5dffff640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 0x7fc5e015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5deffd640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5e00d1c20 0x7fc5e00a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 0x7fc5e015bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 -- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5e00d1c20 msgr2=0x7fc5e00a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5e00d1c20 0x7fc5e00a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 -- 172.21.15.107:0/349688312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80094d0 con 0x7fc5e00d0200 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5dffff640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 0x7fc5e015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5df7fe640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000c7c0 tx=0x7fc5d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d000aa50 con 0x7fc5e00d0200 2023-11-24T08:19:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d000abb0 con 0x7fc5e00d0200 2023-11-24T08:19:38.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0012420 con 0x7fc5e00d0200 2023-11-24T08:19:38.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e015c610 con 0x7fc5e00d0200 2023-11-24T08:19:38.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.239+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e015cbd0 con 0x7fc5e00d0200 2023-11-24T08:19:38.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.240+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e004e8b0 con 0x7fc5e00d0200 2023-11-24T08:19:38.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.241+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 37) v1 ==== 91628+0+0 (secure 0 0 0) 0x7fc5d0012580 con 0x7fc5e00d0200 2023-11-24T08:19:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.242+0000 7fc5d7fff640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 0x7fc5c006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.242+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5d0088510 con 0x7fc5e00d0200 2023-11-24T08:19:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.243+0000 7fc5deffd640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 0x7fc5c006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.243+0000 7fc5deffd640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 0x7fc5c006ed70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8036000 tx=0x7fc5c803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.247+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5d0054f90 con 0x7fc5e00d0200 2023-11-24T08:19:38.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.470+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol4786", "snap_name": "subvol_snap8897", "target_sub_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e00d0cb0 con 0x7fc5c006c8b0 2023-11-24T08:19:38.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.629+0000 7fc5d7fff640 1 -- 172.21.15.107:0/349688312 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc5c8030e30 con 0x7fc5c006c8b0 2023-11-24T08:19:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 msgr2=0x7fc5c006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 0x7fc5c006ed70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8036000 tx=0x7fc5c803b040 comp rx=0 tx=0).stop 2023-11-24T08:19:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 msgr2=0x7fc5e00ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000c7c0 tx=0x7fc5d000cc80 comp rx=0 tx=0).stop 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 shutdown_connections 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc5c006c8b0 0x7fc5c006ed70 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5e00d2000 0x7fc5e015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5e00d1c20 0x7fc5e00a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 --2- 172.21.15.107:0/349688312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e00d0200 0x7fc5e00ccab0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 >> 172.21.15.107:0/349688312 conn(0x7fc5e008cea0 msgr2=0x7fc5e00c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 shutdown_connections 2023-11-24T08:19:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.633+0000 7fc5e5be5640 1 -- 172.21.15.107:0/349688312 wait complete. 2023-11-24T08:19:38.642 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.641+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.642 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.641+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.642 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.641+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.642 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.641+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.643 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.641+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:38.664 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.662+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.664 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.662+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.664 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.662+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.664 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.662+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.664 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:19:38.662+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:19:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.847+0000 7f1251ebb640 1 Processor -- start 2023-11-24T08:19:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.847+0000 7f1251ebb640 1 -- start start 2023-11-24T08:19:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.848+0000 7f1251ebb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f124c0cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.848+0000 7f1251ebb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0ae5d0 con 0x7f124c0b0b40 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.848+0000 7f1251ebb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0ae8b0 con 0x7f124c15d310 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.848+0000 7f1251ebb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c06cd10 con 0x7f124c0b0f90 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.848+0000 7f124bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f124c0b0b40 0x7f124c0cce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35018/0 (socket says 172.21.15.107:35018) 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124bfff640 1 -- 172.21.15.107:0/1462311371 learned_addr learned my addr 172.21.15.107:0/1462311371 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f12516ba640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f124c0cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439511594 0 0) 0x7f124c0ae5d0 con 0x7f124c0b0b40 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1234003520 con 0x7f124c0b0b40 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345068081 0 0) 0x7f1234003520 con 0x7f124c0b0b40 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 msgr2=0x7f124c0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f124c0cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 >> v1:172.21.15.120:6789/0 conn(0x7f124c15d310 legacy=0x7f124c0ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c06c650 con 0x7f124c0b0b40 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f123c001690 con 0x7f124c0b0b40 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 >> v1:172.21.15.107:6789/0 conn(0x7f124c0b0b40 legacy=0x7f124c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.849+0000 7f124b7fe640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f12340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124b7fe640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1234003fd0 0x7f12340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124b7fe640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1234004d00 0x7f1234005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c06cd10 con 0x7f1234004d00 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123400bb40 con 0x7f1234003fd0 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123400bd00 con 0x7f124c0b0f90 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1234004d00 0x7f1234005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f1250eb9640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f12340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124bfff640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1234003fd0 0x7f12340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 -- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 msgr2=0x7f12340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f124c0b0f90 0x7f12340039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 -- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1234003fd0 msgr2=0x7f12340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1234003fd0 0x7f12340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f12516ba640 1 -- 172.21.15.107:0/1462311371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234003520 con 0x7f1234004d00 2023-11-24T08:19:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.850+0000 7f124bfff640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1234003fd0 0x7f12340047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.851+0000 7f12516ba640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1234004d00 0x7f1234005180 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f1244007360 tx=0x7f12340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480ea9aed512a873 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124402ccd0 con 0x7f1234004d00 2023-11-24T08:19:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124402ce30 con 0x7f1234004d00 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f124b7fe640 1 -- 172.21.15.107:0/1462311371 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244034660 con 0x7f1234004d00 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 -- 172.21.15.107:0/1462311371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1234004d00 msgr2=0x7f1234005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 --2- 172.21.15.107:0/1462311371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1234004d00 0x7f1234005180 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f1244007360 tx=0x7f12340057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f12516ba640 1 -- 172.21.15.107:0/1462311371 reap_dead start 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 -- 172.21.15.107:0/1462311371 shutdown_connections 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 -- 172.21.15.107:0/1462311371 >> 172.21.15.107:0/1462311371 conn(0x7f124c0cbb40 msgr2=0x7f124c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 -- 172.21.15.107:0/1462311371 shutdown_connections 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.852+0000 7f1251ebb640 1 -- 172.21.15.107:0/1462311371 wait complete. 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.854+0000 7f1251ebb640 1 Processor -- start 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.854+0000 7f1251ebb640 1 -- start start 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 0x7f124c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 0x7f124c159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c1f7da0 con 0x7f124c09ab30 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c0c6c70 con 0x7f124c09b710 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.855+0000 7f1251ebb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c1f7220 con 0x7f1234004d00 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52270/0 (socket says 172.21.15.107:52270) 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 -- 172.21.15.107:0/2791376417 learned_addr learned my addr 172.21.15.107:0/2791376417 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f12516ba640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 0x7f124c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f1250eb9640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 0x7f124c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 -- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 msgr2=0x7f124c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 0x7f124c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 -- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 msgr2=0x7f124c159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 0x7f124c159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 -- 172.21.15.107:0/2791376417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c1f7500 con 0x7f124c09ab30 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f12516ba640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 0x7f124c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f1250eb9640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 0x7f124c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.856+0000 7f124bfff640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f123c00d8d0 tx=0x7f123c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.857+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0074e0 con 0x7f124c09ab30 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.857+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c007640 con 0x7f124c09ab30 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.857+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0049e0 con 0x7f124c09ab30 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.857+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c1f77e0 con 0x7f124c09ab30 2023-11-24T08:19:38.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.857+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1f6e40 con 0x7f124c09ab30 2023-11-24T08:19:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.858+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c04e840 con 0x7f124c09ab30 2023-11-24T08:19:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.859+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 37) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f123c004b40 con 0x7f124c09ab30 2023-11-24T08:19:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.860+0000 7f12497fa640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 0x7f123806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.860+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f123c087fc0 con 0x7f124c09ab30 2023-11-24T08:19:38.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.860+0000 7f1250eb9640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 0x7f123806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:38.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.861+0000 7f1250eb9640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 0x7f123806ed70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f12400016b0 tx=0x7f1240009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:38.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:38.864+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f123c054ad0 con 0x7f124c09ab30 2023-11-24T08:19:39.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.084+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f124c161ea0 con 0x7f123806c8b0 2023-11-24T08:19:39.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.095+0000 7f12497fa640 1 -- 172.21.15.107:0/2791376417 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f124c161ea0 con 0x7f123806c8b0 2023-11-24T08:19:39.098 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:19:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:39.099 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 msgr2=0x7f123806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:39.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 0x7f123806ed70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f12400016b0 tx=0x7f1240009040 comp rx=0 tx=0).stop 2023-11-24T08:19:39.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 msgr2=0x7f124c0affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f123c00d8d0 tx=0x7f123c00dd90 comp rx=0 tx=0).stop 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 shutdown_connections 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f123806c8b0 0x7f123806ed70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f124c09b710 0x7f124c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f124c09ab30 0x7f124c0affa0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 --2- 172.21.15.107:0/2791376417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1234004d00 0x7f124c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 >> 172.21.15.107:0/2791376417 conn(0x7f124c0cbb40 msgr2=0x7f124c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 shutdown_connections 2023-11-24T08:19:39.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:39.098+0000 7f1251ebb640 1 -- 172.21.15.107:0/2791376417 wait complete. 2023-11-24T08:19:40.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:40.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.340+0000 7f4e0879e640 1 Processor -- start 2023-11-24T08:19:40.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.340+0000 7f4e0879e640 1 -- start start 2023-11-24T08:19:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e0879e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e00072160 0x7f4e0015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e0879e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0015ec90 con 0x7f4e0006de10 2023-11-24T08:19:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e0879e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000cd530 con 0x7f4e000c49f0 2023-11-24T08:19:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e0879e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e000c4ed0 con 0x7f4e00072160 2023-11-24T08:19:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e06d3b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4e0006de10 0x7f4e000705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35022/0 (socket says 172.21.15.107:35022) 2023-11-24T08:19:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e06d3b640 1 -- 172.21.15.107:0/223653643 learned_addr learned my addr 172.21.15.107:0/223653643 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:40.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.342+0000 7f4e0653a640 1 --2- 172.21.15.107:0/223653643 >> v2:172.21.15.120:3300/0 conn(0x7f4e00072160 0x7f4e0015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287361951 0 0) 0x7f4df8001490 con 0x7f4e0006de10 2023-11-24T08:19:40.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec003520 con 0x7f4e0006de10 2023-11-24T08:19:40.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551003568 0 0) 0x7f4dec003520 con 0x7f4e0006de10 2023-11-24T08:19:40.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 >> v2:172.21.15.120:3300/0 conn(0x7f4e00072160 msgr2=0x7f4e0015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:40.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 --2- 172.21.15.107:0/223653643 >> v2:172.21.15.120:3300/0 conn(0x7f4e00072160 0x7f4e0015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 >> v1:172.21.15.120:6789/0 conn(0x7f4e000c49f0 legacy=0x7f4e0006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000c51b0 con 0x7f4e0006de10 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4df8001b80 con 0x7f4e0006de10 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 >> v1:172.21.15.107:6789/0 conn(0x7f4e0006de10 legacy=0x7f4e000705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec003970 0x7f4dec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dec0047c0 0x7f4dec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 0x7f4dec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000c4ed0 con 0x7f4dec005180 2023-11-24T08:19:40.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00c020 con 0x7f4dec003970 2023-11-24T08:19:40.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00c1e0 con 0x7f4dec0047c0 2023-11-24T08:19:40.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.343+0000 7f4e06d3b640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 0x7f4dec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e0653a640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec003970 0x7f4dec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e06d3b640 1 -- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dec0047c0 msgr2=0x7f4dec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e06d3b640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dec0047c0 0x7f4dec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e06d3b640 1 -- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec003970 msgr2=0x7f4dec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e06d3b640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec003970 0x7f4dec003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.344+0000 7f4e06d3b640 1 -- 172.21.15.107:0/223653643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec003520 con 0x7f4dec005180 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.345+0000 7f4e06d3b640 1 -- 172.21.15.107:0/223653643 reap_dead start 2023-11-24T08:19:40.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.345+0000 7f4e06d3b640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 0x7f4dec00b980 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f4df800a7e0 tx=0x7f4dec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75b8b2018ab04a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:40.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.345+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8034630 con 0x7f4dec005180 2023-11-24T08:19:40.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.345+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df802f070 con 0x7f4dec005180 2023-11-24T08:19:40.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.345+0000 7f4e05538640 1 -- 172.21.15.107:0/223653643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df80349f0 con 0x7f4dec005180 2023-11-24T08:19:40.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 -- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 msgr2=0x7f4dec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 0x7f4dec00b980 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f4df800a7e0 tx=0x7f4dec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 -- 172.21.15.107:0/223653643 shutdown_connections 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 --2- 172.21.15.107:0/223653643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec005180 0x7f4dec00b980 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 -- 172.21.15.107:0/223653643 >> 172.21.15.107:0/223653643 conn(0x7f4e000b0c50 msgr2=0x7f4e000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 -- 172.21.15.107:0/223653643 shutdown_connections 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.346+0000 7f4e0879e640 1 -- 172.21.15.107:0/223653643 wait complete. 2023-11-24T08:19:40.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.348+0000 7f4e0879e640 1 Processor -- start 2023-11-24T08:19:40.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.348+0000 7f4e0879e640 1 -- start start 2023-11-24T08:19:40.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec005180 0x7f4e00159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e000cefe0 0x7f4e000cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000cf9d0 con 0x7f4e00159320 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e001597a0 con 0x7f4dec005180 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0879e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000c6ce0 con 0x7f4e000cefe0 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52292/0 (socket says 172.21.15.107:52292) 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 -- 172.21.15.107:0/972258332 learned_addr learned my addr 172.21.15.107:0/972258332 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e0653a640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec005180 0x7f4e00159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 -- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e000cefe0 msgr2=0x7f4e000cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e000cefe0 0x7f4e000cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 -- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec005180 msgr2=0x7f4e00159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec005180 0x7f4e00159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.349+0000 7f4e05d39640 1 -- 172.21.15.107:0/972258332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000c60d0 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4e05d39640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c430 tx=0x7f4df000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df00052d0 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0005450 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e000c63b0 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0003830 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.350+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e000c6970 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.351+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e000c6e50 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.352+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 37) v1 ==== 91628+0+0 (secure 0 0 0) 0x7f4df0003990 con 0x7f4e00159320 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.353+0000 7f4df6ffd640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 0x7f4de406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:40.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.353+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4df0087aa0 con 0x7f4e00159320 2023-11-24T08:19:40.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.353+0000 7f4e0653a640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 0x7f4de406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:40.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.354+0000 7f4e0653a640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 0x7f4de406ed70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f4e000b40d0 tx=0x7f4dfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:40.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.358+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4df00545b0 con 0x7f4e00159320 2023-11-24T08:19:40.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.591+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e000c5a50 con 0x7f4de406c8b0 2023-11-24T08:19:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.602+0000 7f4df6ffd640 1 -- 172.21.15.107:0/972258332 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f4dfc0095c0 con 0x7f4de406c8b0 2023-11-24T08:19:40.604 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:40.604 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:40.605 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 msgr2=0x7f4de406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 0x7f4de406ed70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f4e000b40d0 tx=0x7f4dfc008040 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 msgr2=0x7f4e000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c430 tx=0x7f4df000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 shutdown_connections 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4de406c8b0 0x7f4de406ed70 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e000cefe0 0x7f4e000cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00159320 0x7f4e000b4140 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 --2- 172.21.15.107:0/972258332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dec005180 0x7f4e00159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 >> 172.21.15.107:0/972258332 conn(0x7f4e000b0c50 msgr2=0x7f4e000aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 shutdown_connections 2023-11-24T08:19:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:40.605+0000 7f4e0879e640 1 -- 172.21.15.107:0/972258332 wait complete. 2023-11-24T08:19:41.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.861+0000 7f4bfe8bb640 1 Processor -- start 2023-11-24T08:19:41.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.862+0000 7f4bfe8bb640 1 -- start start 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfe8bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bf815d3f0 0x7f4bf80d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfe8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf80d0e40 0x7f4bf80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfe8bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80d15e0 con 0x7f4bf815d7d0 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfe8bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf80d0280 con 0x7f4bf815d3f0 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfe8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80d1300 con 0x7f4bf80d0e40 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfd8b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4bf815d3f0 0x7f4bf80d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfd0b8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4bf815d7d0 0x7f4bf80afea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35026/0 (socket says 172.21.15.107:35026) 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.863+0000 7f4bfd0b8640 1 -- 172.21.15.107:0/3426230086 learned_addr learned my addr 172.21.15.107:0/3426230086 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.864+0000 7f4bfc8b7640 1 -- 172.21.15.107:0/3426230086 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024865645 0 0) 0x7f4bf80d15e0 con 0x7f4bf815d7d0 2023-11-24T08:19:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.864+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/3426230086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf80d0e40 msgr2=0x7f4bf80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.864+0000 7f4bfd8b9640 1 --2- 172.21.15.107:0/3426230086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf80d0e40 0x7f4bf80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.864+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/3426230086 >> v1:172.21.15.107:6789/0 conn(0x7f4bf815d7d0 legacy=0x7f4bf80afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.864+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/3426230086 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80cfc30 con 0x7f4bf815d3f0 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.865+0000 7f4bfd8b9640 1 --2- 172.21.15.107:0/3426230086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80d1830 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009d90 tx=0x7f4be802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3beea84df31d5152 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.865+0000 7f4bfc8b7640 1 -- 172.21.15.107:0/3426230086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be802cc80 con 0x7f4bf815d3f0 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.865+0000 7f4bfc8b7640 1 -- 172.21.15.107:0/3426230086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be802cde0 con 0x7f4bf815d3f0 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.865+0000 7f4bfc8b7640 1 -- 172.21.15.107:0/3426230086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8034660 con 0x7f4bf815d3f0 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/3426230086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf815d3f0 msgr2=0x7f4bf80d1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/3426230086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80d1830 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009d90 tx=0x7f4be802bed0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/3426230086 shutdown_connections 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/3426230086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf80d0e40 0x7f4bf80d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/3426230086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80d1830 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/3426230086 >> 172.21.15.107:0/3426230086 conn(0x7f4bf809c5b0 msgr2=0x7f4bf80d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/3426230086 shutdown_connections 2023-11-24T08:19:41.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.866+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/3426230086 wait complete. 2023-11-24T08:19:41.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.868+0000 7f4bfe8bb640 1 Processor -- start 2023-11-24T08:19:41.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.868+0000 7f4bfe8bb640 1 -- start start 2023-11-24T08:19:41.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf815d7d0 0x7f4bf80b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf814a770 con 0x7f4bf80d0e40 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80c7f20 con 0x7f4bf815d3f0 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfe8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80c8200 con 0x7f4bf815d7d0 2023-11-24T08:19:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfd8b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfd8b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52310/0 (socket says 172.21.15.107:52310) 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.869+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/1529580813 learned_addr learned my addr 172.21.15.107:0/1529580813 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf815d7d0 msgr2=0x7f4bf80b88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf815d7d0 0x7f4bf80b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bf815d3f0 msgr2=0x7f4bf80c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:41.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 -- 172.21.15.107:0/1529580813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8009a70 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfd8b9640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f4be8037040 tx=0x7f4be8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be803e610 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be803e770 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be803eb20 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.870+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf80c84e0 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.871+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf80c8aa0 con 0x7f4bf80d0e40 2023-11-24T08:19:41.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.872+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf804e840 con 0x7f4bf80d0e40 2023-11-24T08:19:41.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.873+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f4be803b050 con 0x7f4bf80d0e40 2023-11-24T08:19:41.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.874+0000 7f4bf67fc640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 0x7f4be406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:41.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.874+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4be80aaf00 con 0x7f4bf80d0e40 2023-11-24T08:19:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.874+0000 7f4bfd0b8640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 0x7f4be406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.874+0000 7f4bfd0b8640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 0x7f4be406ee90 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f4bec007fd0 tx=0x7f4bec007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:41.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:41.878+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4be8077ab0 con 0x7f4bf80d0e40 2023-11-24T08:19:42.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.104+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bf80d0710 con 0x7f4be406c9d0 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.124+0000 7f4bf67fc640 1 -- 172.21.15.107:0/1529580813 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f4bec001220 con 0x7f4be406c9d0 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:42.126 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:42.127 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:42.127 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:42.127 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:42.127 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:42.127 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:42.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.126+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 msgr2=0x7f4be406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.126+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 0x7f4be406ee90 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f4bec007fd0 tx=0x7f4bec007960 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.126+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 msgr2=0x7f4bf814a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.126+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f4be8037040 tx=0x7f4be8004a40 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 shutdown_connections 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be406c9d0 0x7f4be406ee90 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bf815d7d0 0x7f4bf80b88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bf815d3f0 0x7f4bf80c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 --2- 172.21.15.107:0/1529580813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf80d0e40 0x7f4bf814a230 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 >> 172.21.15.107:0/1529580813 conn(0x7f4bf809c5b0 msgr2=0x7f4bf806c120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 shutdown_connections 2023-11-24T08:19:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:42.127+0000 7f4bfe8bb640 1 -- 172.21.15.107:0/1529580813 wait complete. 2023-11-24T08:19:43.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.373+0000 7f791d04b640 1 Processor -- start 2023-11-24T08:19:43.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.373+0000 7f791d04b640 1 -- start start 2023-11-24T08:19:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f791d04b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f791815b260 0x7f791806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f791d04b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f791806c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f791d04b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791815ed20 con 0x7f791806e580 2023-11-24T08:19:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f791d04b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791815b640 con 0x7f791815b260 2023-11-24T08:19:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f791d04b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791815f8a0 con 0x7f79180b1250 2023-11-24T08:19:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f7917fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f791815b260 0x7f791806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.374+0000 7f79177fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f791806e580 0x7f79180b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35028/0 (socket says 172.21.15.107:35028) 2023-11-24T08:19:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f79177fe640 1 -- 172.21.15.107:0/2242518815 learned_addr learned my addr 172.21.15.107:0/2242518815 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f7917fff640 1 -- 172.21.15.107:0/2242518815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 msgr2=0x7f791806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f791c84a640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f791806c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f7917fff640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f791806c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f7917fff640 1 -- 172.21.15.107:0/2242518815 >> v1:172.21.15.107:6789/0 conn(0x7f791806e580 legacy=0x7f79180b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f7917fff640 1 -- 172.21.15.107:0/2242518815 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180abe50 con 0x7f791815b260 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f7916ffd640 1 -- 172.21.15.107:0/2242518815 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813166803 0 0) 0x7f791815ed20 con 0x7f791806e580 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.375+0000 7f791c84a640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f791806c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:43.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.376+0000 7f7917fff640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791806e040 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f79000097a0 tx=0x7f790002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7a1f7e4f6cef08 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:43.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.376+0000 7f7916ffd640 1 -- 172.21.15.107:0/2242518815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79000316f0 con 0x7f791815b260 2023-11-24T08:19:43.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.376+0000 7f7916ffd640 1 -- 172.21.15.107:0/2242518815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7900031850 con 0x7f791815b260 2023-11-24T08:19:43.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.376+0000 7f7916ffd640 1 -- 172.21.15.107:0/2242518815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900031be0 con 0x7f791815b260 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 -- 172.21.15.107:0/2242518815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 msgr2=0x7f791806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791806e040 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f79000097a0 tx=0x7f790002b790 comp rx=0 tx=0).stop 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 -- 172.21.15.107:0/2242518815 shutdown_connections 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f791806c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 --2- 172.21.15.107:0/2242518815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791806e040 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 -- 172.21.15.107:0/2242518815 >> 172.21.15.107:0/2242518815 conn(0x7f79180cbb40 msgr2=0x7f79180cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 -- 172.21.15.107:0/2242518815 shutdown_connections 2023-11-24T08:19:43.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.377+0000 7f791d04b640 1 -- 172.21.15.107:0/2242518815 wait complete. 2023-11-24T08:19:43.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.379+0000 7f791d04b640 1 Processor -- start 2023-11-24T08:19:43.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.379+0000 7f791d04b640 1 -- start start 2023-11-24T08:19:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 0x7f7918158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f79181505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7918150b00 con 0x7f791815b260 2023-11-24T08:19:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791815ca90 con 0x7f791806e580 2023-11-24T08:19:43.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791d04b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791815cd70 con 0x7f79180b1250 2023-11-24T08:19:43.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.380+0000 7f791c84a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52314/0 (socket says 172.21.15.107:52314) 2023-11-24T08:19:43.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 -- 172.21.15.107:0/3403093756 learned_addr learned my addr 172.21.15.107:0/3403093756 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f7917fff640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 0x7f7918158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f79177fe640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f79181505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 -- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 msgr2=0x7f79181505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f79181505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 -- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 msgr2=0x7f7918158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 0x7f7918158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 -- 172.21.15.107:0/3403093756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7900009400 con 0x7f791815b260 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f7917fff640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 0x7f7918158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f79177fe640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f79181505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f791c84a640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f791000c590 tx=0x7f791000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.381+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910011800 con 0x7f791815b260 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.382+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7910011980 con 0x7f791815b260 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.382+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910011d30 con 0x7f791815b260 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.382+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791815d050 con 0x7f791815b260 2023-11-24T08:19:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.382+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791815d610 con 0x7f791815b260 2023-11-24T08:19:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.383+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791804e840 con 0x7f791815b260 2023-11-24T08:19:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.384+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f7910022350 con 0x7f791815b260 2023-11-24T08:19:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.385+0000 7f7914ff9640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 0x7f78f806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.385+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f79100885a0 con 0x7f791815b260 2023-11-24T08:19:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.385+0000 7f7917fff640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 0x7f78f806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.386+0000 7f7917fff640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 0x7f78f806ee90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7900036000 tx=0x7f7900008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:43.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.389+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79100550b0 con 0x7f791815b260 2023-11-24T08:19:43.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.621+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f79180adeb0 con 0x7f78f806c9d0 2023-11-24T08:19:43.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.631+0000 7f7914ff9640 1 -- 172.21.15.107:0/3403093756 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f79180adeb0 con 0x7f78f806c9d0 2023-11-24T08:19:43.634 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:43.634 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:43.634 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:43.635 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:43.635 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:43.635 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:43.635 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:43.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:43.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:43.636 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 msgr2=0x7f78f806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 0x7f78f806ee90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7900036000 tx=0x7f7900008030 comp rx=0 tx=0).stop 2023-11-24T08:19:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 msgr2=0x7f791809c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f791000c590 tx=0x7f791000ca50 comp rx=0 tx=0).stop 2023-11-24T08:19:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 shutdown_connections 2023-11-24T08:19:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f78f806c9d0 0x7f78f806ee90 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f791815b260 0x7f791809c410 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79180b1250 0x7f79181505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 --2- 172.21.15.107:0/3403093756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f791806e580 0x7f7918158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 >> 172.21.15.107:0/3403093756 conn(0x7f79180cbb40 msgr2=0x7f79180cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:43.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 shutdown_connections 2023-11-24T08:19:43.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:43.635+0000 7f791d04b640 1 -- 172.21.15.107:0/3403093756 wait complete. 2023-11-24T08:19:44.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:44.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.884+0000 7f70948df640 1 Processor -- start 2023-11-24T08:19:44.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.884+0000 7f70948df640 1 -- start start 2023-11-24T08:19:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f70948df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f709015a1c0 0x7f709006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f70948df640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709006d430 con 0x7f709015e5e0 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f70948df640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709015ec50 con 0x7f70900b0b40 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f70948df640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709015ea70 con 0x7f709015a1c0 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f709015a1c0 0x7f709006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f709015e5e0 0x7f7090070e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35034/0 (socket says 172.21.15.107:35034) 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708ffff640 1 -- 172.21.15.107:0/2149636208 learned_addr learned my addr 172.21.15.107:0/2149636208 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f709015a1c0 0x7f709006d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52330/0 (socket says 172.21.15.107:52330) 2023-11-24T08:19:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708f7fe640 1 -- 172.21.15.107:0/2149636208 >> v1:172.21.15.120:6789/0 conn(0x7f70900b0b40 legacy=0x7f709015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.886+0000 7f708f7fe640 1 -- 172.21.15.107:0/2149636208 >> v1:172.21.15.107:6789/0 conn(0x7f709015e5e0 legacy=0x7f7090070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.887+0000 7f708f7fe640 1 -- 172.21.15.107:0/2149636208 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709006cb00 con 0x7f709015a1c0 2023-11-24T08:19:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f708f7fe640 1 --2- 172.21.15.107:0/2149636208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f709006d8c0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f70800097a0 tx=0x7f708002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c4c055a751a27 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f708e7fc640 1 -- 172.21.15.107:0/2149636208 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080031830 con 0x7f709015a1c0 2023-11-24T08:19:44.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f708e7fc640 1 -- 172.21.15.107:0/2149636208 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080031990 con 0x7f709015a1c0 2023-11-24T08:19:44.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f708e7fc640 1 -- 172.21.15.107:0/2149636208 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080031d20 con 0x7f709015a1c0 2023-11-24T08:19:44.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f70948df640 1 -- 172.21.15.107:0/2149636208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 msgr2=0x7f709006d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:44.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.888+0000 7f70948df640 1 --2- 172.21.15.107:0/2149636208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f709006d8c0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f70800097a0 tx=0x7f708002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:19:44.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.889+0000 7f70948df640 1 -- 172.21.15.107:0/2149636208 shutdown_connections 2023-11-24T08:19:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.889+0000 7f70948df640 1 --2- 172.21.15.107:0/2149636208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f709006d8c0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.889+0000 7f70948df640 1 -- 172.21.15.107:0/2149636208 >> 172.21.15.107:0/2149636208 conn(0x7f70900cbb40 msgr2=0x7f70900cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.889+0000 7f70948df640 1 -- 172.21.15.107:0/2149636208 shutdown_connections 2023-11-24T08:19:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.889+0000 7f70948df640 1 -- 172.21.15.107:0/2149636208 wait complete. 2023-11-24T08:19:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.890+0000 7f70948df640 1 Processor -- start 2023-11-24T08:19:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 -- start start 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900b0b40 0x7f7090150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709015e5e0 0x7f70900acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70900c7a70 con 0x7f709015a1c0 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090159ae0 con 0x7f70900b0b40 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.891+0000 7f70948df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7090150ab0 con 0x7f709015e5e0 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52346/0 (socket says 172.21.15.107:52346) 2023-11-24T08:19:44.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 -- 172.21.15.107:0/2226379665 learned_addr learned my addr 172.21.15.107:0/2226379665 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 -- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709015e5e0 msgr2=0x7f70900acca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709015e5e0 0x7f70900acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 -- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900b0b40 msgr2=0x7f7090150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708f7fe640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900b0b40 0x7f7090150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900b0b40 0x7f7090150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 -- 172.21.15.107:0/2226379665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080009400 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.892+0000 7f708effd640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f708400b3f0 tx=0x7f708400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.893+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084011830 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.893+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7084011990 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.893+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70900af450 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.893+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084011d40 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.893+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70900afa10 con 0x7f709015a1c0 2023-11-24T08:19:44.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.894+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709004e840 con 0x7f709015a1c0 2023-11-24T08:19:44.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.895+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f708401a400 con 0x7f709015a1c0 2023-11-24T08:19:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.896+0000 7f7077fff640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 0x7f707c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.896+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f70840883c0 con 0x7f709015a1c0 2023-11-24T08:19:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.896+0000 7f708f7fe640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 0x7f707c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.897+0000 7f708f7fe640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 0x7f707c06ee90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7080036000 tx=0x7f7080008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:44.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:44.900+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7084054ed0 con 0x7f709015a1c0 2023-11-24T08:19:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.129+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f709006de00 con 0x7f707c06c9d0 2023-11-24T08:19:45.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.141+0000 7f7077fff640 1 -- 172.21.15.107:0/2226379665 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f7080031b80 con 0x7f707c06c9d0 2023-11-24T08:19:45.143 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:45.143 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:45.143 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:45.143 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:45.143 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:45.144 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:45.144 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:45.144 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:45.144 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:45.144 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 msgr2=0x7f707c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 0x7f707c06ee90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7080036000 tx=0x7f7080008030 comp rx=0 tx=0).stop 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 msgr2=0x7f70900ac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f708400b3f0 tx=0x7f708400b8b0 comp rx=0 tx=0).stop 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 shutdown_connections 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f707c06c9d0 0x7f707c06ee90 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.143+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f709015e5e0 0x7f70900acca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:45.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.144+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f709015a1c0 0x7f70900ac760 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:45.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.144+0000 7f70948df640 1 --2- 172.21.15.107:0/2226379665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900b0b40 0x7f7090150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:45.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.144+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 >> 172.21.15.107:0/2226379665 conn(0x7f70900cbb40 msgr2=0x7f709015b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:45.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.144+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 shutdown_connections 2023-11-24T08:19:45.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:45.144+0000 7f70948df640 1 -- 172.21.15.107:0/2226379665 wait complete. 2023-11-24T08:19:45.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:19:45.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:19:46.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:46.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.389+0000 7f0e012d3640 1 Processor -- start 2023-11-24T08:19:46.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.390+0000 7f0e012d3640 1 -- start start 2023-11-24T08:19:46.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.391+0000 7f0e012d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0dfc15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.391+0000 7f0e012d3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0dfc0c49f0 0x7f0dfc06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0e012d3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc15ec90 con 0x7f0dfc06dc40 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0e012d3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc06e0f0 con 0x7f0dfc0c49f0 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0e012d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0cd530 con 0x7f0dfc072160 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0dfb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0dfc06dc40 0x7f0dfc0cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35040/0 (socket says 172.21.15.107:35040) 2023-11-24T08:19:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3781953793 learned_addr learned my addr 172.21.15.107:0/3781953793 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:46.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0dfc15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0df27fc640 1 --2- 172.21.15.107:0/3781953793 >> v2:172.21.15.120:3300/0 conn(0x7f0dfc0c49f0 0x7f0dfc06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809025928 0 0) 0x7f0df4001490 con 0x7f0dfc06dc40 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.392+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de8003520 con 0x7f0dfc06dc40 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859358598 0 0) 0x7f0de8003520 con 0x7f0dfc06dc40 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 msgr2=0x7f0dfc15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0dfc15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 >> v2:172.21.15.120:3300/0 conn(0x7f0dfc0c49f0 msgr2=0x7f0dfc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 --2- 172.21.15.107:0/3781953793 >> v2:172.21.15.120:3300/0 conn(0x7f0dfc0c49f0 0x7f0dfc06d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc0c4ed0 con 0x7f0dfc06dc40 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0df4001b80 con 0x7f0dfc06dc40 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 >> v1:172.21.15.107:6789/0 conn(0x7f0dfc06dc40 legacy=0x7f0dfc0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0de8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8003ef0 0x7f0de80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0de8004d00 0x7f0de8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc06e0f0 con 0x7f0de8004d00 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0cd530 con 0x7f0de8003ef0 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de800b980 con 0x7f0dfc072160 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.393+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0de8004d00 0x7f0de8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0df27fc640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8003ef0 0x7f0de80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0de8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 msgr2=0x7f0de8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc072160 0x7f0de8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8003ef0 msgr2=0x7f0de80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8003ef0 0x7f0de80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.394+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3781953793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8003520 con 0x7f0de8004d00 2023-11-24T08:19:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.395+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0de8004d00 0x7f0de8005180 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f0df400a830 tx=0x7f0de80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54a8f85ef01c579 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.395+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df402cea0 con 0x7f0de8004d00 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.395+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df402f070 con 0x7f0de8004d00 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.395+0000 7f0dfa7fc640 1 -- 172.21.15.107:0/3781953793 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4034870 con 0x7f0de8004d00 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 -- 172.21.15.107:0/3781953793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0de8004d00 msgr2=0x7f0de8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3781953793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0de8004d00 0x7f0de8005180 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f0df400a830 tx=0x7f0de80057f0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3781953793 reap_dead start 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 -- 172.21.15.107:0/3781953793 shutdown_connections 2023-11-24T08:19:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 -- 172.21.15.107:0/3781953793 >> 172.21.15.107:0/3781953793 conn(0x7f0dfc0b0c50 msgr2=0x7f0dfc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 -- 172.21.15.107:0/3781953793 shutdown_connections 2023-11-24T08:19:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.396+0000 7f0e012d3640 1 -- 172.21.15.107:0/3781953793 wait complete. 2023-11-24T08:19:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.398+0000 7f0e012d3640 1 Processor -- start 2023-11-24T08:19:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.398+0000 7f0e012d3640 1 -- start start 2023-11-24T08:19:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 0x7f0dfc09a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc15c4a0 0x7f0dfc09b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1524d0 con 0x7f0dfc151ae0 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc15c920 con 0x7f0de8004d00 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0e012d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc15cbf0 con 0x7f0dfc15c4a0 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52356/0 (socket says 172.21.15.107:52356) 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3691058774 learned_addr learned my addr 172.21.15.107:0/3691058774 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 0x7f0dfc09a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0df27fc640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc15c4a0 0x7f0dfc09b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc15c4a0 msgr2=0x7f0dfc09b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc15c4a0 0x7f0dfc09b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 msgr2=0x7f0dfc09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 0x7f0dfc09a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfb7fe640 1 -- 172.21.15.107:0/3691058774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc15ced0 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.399+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 0x7f0dfc09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0dfb7fe640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f0df4000f80 tx=0x7f0df4001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4034630 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df4034790 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc15a610 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4034b20 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.400+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc09bc00 con 0x7f0dfc151ae0 2023-11-24T08:19:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.401+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc15a470 con 0x7f0dfc151ae0 2023-11-24T08:19:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.402+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f0df4031030 con 0x7f0dfc151ae0 2023-11-24T08:19:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.403+0000 7f0df3fff640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 0x7f0ddc06eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.403+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0df40aa150 con 0x7f0dfc151ae0 2023-11-24T08:19:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.403+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 0x7f0ddc06eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.404+0000 7f0dfaffd640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 0x7f0ddc06eec0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0099e0 tx=0x7f0dec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:46.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.408+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0dfc15a470 con 0x7f0dfc151ae0 2023-11-24T08:19:46.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.639+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dfc0c5770 con 0x7f0ddc06ca00 2023-11-24T08:19:46.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.651+0000 7f0df3fff640 1 -- 172.21.15.107:0/3691058774 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f0dec009650 con 0x7f0ddc06ca00 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:46.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:46.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:46.655 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 msgr2=0x7f0ddc06eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 0x7f0ddc06eec0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0099e0 tx=0x7f0dec008040 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 msgr2=0x7f0dfc151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f0df4000f80 tx=0x7f0df4001360 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 shutdown_connections 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0ddc06ca00 0x7f0ddc06eec0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dfc151ae0 0x7f0dfc151f90 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dfc15c4a0 0x7f0dfc09b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 --2- 172.21.15.107:0/3691058774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de8004d00 0x7f0dfc09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:46.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 >> 172.21.15.107:0/3691058774 conn(0x7f0dfc0b0c50 msgr2=0x7f0dfc0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:46.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 shutdown_connections 2023-11-24T08:19:46.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:46.654+0000 7f0e012d3640 1 -- 172.21.15.107:0/3691058774 wait complete. 2023-11-24T08:19:47.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.899+0000 7f50c9a94640 1 Processor -- start 2023-11-24T08:19:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.899+0000 7f50c9a94640 1 -- start start 2023-11-24T08:19:47.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c9a94640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50c4072160 0x7f50c415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c9a94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c406de10 0x7f50c40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c9a94640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c415ec90 con 0x7f50c40c49f0 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c9a94640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40cd530 con 0x7f50c4072160 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c9a94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c40c4ed0 con 0x7f50c406de10 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50c4072160 0x7f50c415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.901+0000 7f50c3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c406de10 0x7f50c40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f50c4072160 0x7f50c415e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:49594/0 (socket says 172.21.15.107:49594) 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 -- 172.21.15.107:0/4288029268 learned_addr learned my addr 172.21.15.107:0/4288029268 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 -- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c406de10 msgr2=0x7f50c40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 --2- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c406de10 0x7f50c40705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 -- 172.21.15.107:0/4288029268 >> v1:172.21.15.120:6789/0 conn(0x7f50c40c49f0 legacy=0x7f50c406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:47.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.902+0000 7f50c37fe640 1 -- 172.21.15.107:0/4288029268 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40c51b0 con 0x7f50c4072160 2023-11-24T08:19:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.904+0000 7f50c37fe640 1 --2- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c415e750 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009870 tx=0x7f50b402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42d981cba04bf27e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.904+0000 7f50c2ffd640 1 -- 172.21.15.107:0/4288029268 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4030840 con 0x7f50c4072160 2023-11-24T08:19:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.904+0000 7f50c2ffd640 1 -- 172.21.15.107:0/4288029268 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b40309a0 con 0x7f50c4072160 2023-11-24T08:19:47.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.904+0000 7f50c2ffd640 1 -- 172.21.15.107:0/4288029268 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4030d30 con 0x7f50c4072160 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 -- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 msgr2=0x7f50c415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 --2- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c415e750 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009870 tx=0x7f50b402b980 comp rx=0 tx=0).stop 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 -- 172.21.15.107:0/4288029268 shutdown_connections 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 --2- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c406de10 0x7f50c40705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 --2- 172.21.15.107:0/4288029268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c415e750 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 -- 172.21.15.107:0/4288029268 >> 172.21.15.107:0/4288029268 conn(0x7f50c40b0c50 msgr2=0x7f50c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 -- 172.21.15.107:0/4288029268 shutdown_connections 2023-11-24T08:19:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.905+0000 7f50c9a94640 1 -- 172.21.15.107:0/4288029268 wait complete. 2023-11-24T08:19:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.907+0000 7f50c9a94640 1 Processor -- start 2023-11-24T08:19:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.907+0000 7f50c9a94640 1 -- start start 2023-11-24T08:19:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 0x7f50c406cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c415a7c0 con 0x7f50c406de10 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c40ca8d0 con 0x7f50c4072160 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c9a94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c415bf70 con 0x7f50c40c49f0 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50652/0 (socket says 172.21.15.107:50652) 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c37fe640 1 -- 172.21.15.107:0/681211759 learned_addr learned my addr 172.21.15.107:0/681211759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50bbfff640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c3fff640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 0x7f50c406cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c37fe640 1 -- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 msgr2=0x7f50c406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.908+0000 7f50c37fe640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 0x7f50c406cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c37fe640 1 -- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 msgr2=0x7f50c406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c37fe640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c37fe640 1 -- 172.21.15.107:0/681211759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b40094d0 con 0x7f50c406de10 2023-11-24T08:19:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c3fff640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 0x7f50c406cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c37fe640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f50b4000c00 tx=0x7f50b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4039c90 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b4039df0 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4043640 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c415c250 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.909+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c40ca560 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.910+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c404e8b0 con 0x7f50c406de10 2023-11-24T08:19:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.911+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f50b40437a0 con 0x7f50c406de10 2023-11-24T08:19:47.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.912+0000 7f50c0ff9640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 0x7f50a406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:47.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.912+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50b403e070 con 0x7f50c406de10 2023-11-24T08:19:47.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.913+0000 7f50bbfff640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 0x7f50a406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.913+0000 7f50bbfff640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 0x7f50a406ee90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f50c414aa40 tx=0x7f50ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:47.916+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50b4078090 con 0x7f50c406de10 2023-11-24T08:19:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.145+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f50c40715e0 con 0x7f50a406c9d0 2023-11-24T08:19:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.161+0000 7f50c0ff9640 1 -- 172.21.15.107:0/681211759 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f50c40715e0 con 0x7f50a406c9d0 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:48.164 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.163+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 msgr2=0x7f50a406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.163+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 0x7f50a406ee90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f50c414aa40 tx=0x7f50ac009040 comp rx=0 tx=0).stop 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 msgr2=0x7f50c41f30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f50b4000c00 tx=0x7f50b4004730 comp rx=0 tx=0).stop 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 shutdown_connections 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f50a406c9d0 0x7f50a406ee90 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50c40c49f0 0x7f50c406cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c4072160 0x7f50c406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 --2- 172.21.15.107:0/681211759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50c406de10 0x7f50c41f30d0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 >> 172.21.15.107:0/681211759 conn(0x7f50c40b0c50 msgr2=0x7f50c40acd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 shutdown_connections 2023-11-24T08:19:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:48.164+0000 7f50c9a94640 1 -- 172.21.15.107:0/681211759 wait complete. 2023-11-24T08:19:49.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.413+0000 7f3d768ad640 1 Processor -- start 2023-11-24T08:19:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.413+0000 7f3d768ad640 1 -- start start 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d768ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d700c59b0 0x7f3d70166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d768ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700c5dc0 con 0x7f3d700cc0f0 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d768ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7015eea0 con 0x7f3d7015b440 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d768ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7015b820 con 0x7f3d700c59b0 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d74e4a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d700c59b0 0x7f3d70166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d67fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3d700cc0f0 0x7f3d700c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46030/0 (socket says 172.21.15.107:46030) 2023-11-24T08:19:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.415+0000 7f3d67fff640 1 -- 172.21.15.107:0/3309874771 learned_addr learned my addr 172.21.15.107:0/3309874771 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.416+0000 7f3d74e4a640 1 -- 172.21.15.107:0/3309874771 >> v1:172.21.15.120:6789/0 conn(0x7f3d7015b440 legacy=0x7f3d700cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.416+0000 7f3d74e4a640 1 -- 172.21.15.107:0/3309874771 >> v1:172.21.15.107:6789/0 conn(0x7f3d700cc0f0 legacy=0x7f3d700c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.416+0000 7f3d74e4a640 1 -- 172.21.15.107:0/3309874771 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7006d600 con 0x7f3d700c59b0 2023-11-24T08:19:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.417+0000 7f3d74e4a640 1 --2- 172.21.15.107:0/3309874771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700c59b0 0x7f3d70166230 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009d90 tx=0x7f3d6802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9be5638b756a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.417+0000 7f3d6f7fe640 1 -- 172.21.15.107:0/3309874771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6802cbd0 con 0x7f3d700c59b0 2023-11-24T08:19:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.417+0000 7f3d6f7fe640 1 -- 172.21.15.107:0/3309874771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6802cd30 con 0x7f3d700c59b0 2023-11-24T08:19:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.417+0000 7f3d6f7fe640 1 -- 172.21.15.107:0/3309874771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6802cea0 con 0x7f3d700c59b0 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 -- 172.21.15.107:0/3309874771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700c59b0 msgr2=0x7f3d70166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3309874771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700c59b0 0x7f3d70166230 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009d90 tx=0x7f3d6802be20 comp rx=0 tx=0).stop 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 -- 172.21.15.107:0/3309874771 shutdown_connections 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3309874771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700c59b0 0x7f3d70166230 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 -- 172.21.15.107:0/3309874771 >> 172.21.15.107:0/3309874771 conn(0x7f3d700b0c50 msgr2=0x7f3d700b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 -- 172.21.15.107:0/3309874771 shutdown_connections 2023-11-24T08:19:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.418+0000 7f3d768ad640 1 -- 172.21.15.107:0/3309874771 wait complete. 2023-11-24T08:19:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.420+0000 7f3d768ad640 1 Processor -- start 2023-11-24T08:19:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.420+0000 7f3d768ad640 1 -- start start 2023-11-24T08:19:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d700c59b0 0x7f3d700c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 0x7f3d700cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7015a160 con 0x7f3d700cc0f0 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d70152970 con 0x7f3d7015b440 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.421+0000 7f3d768ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700ce280 con 0x7f3d700c59b0 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50670/0 (socket says 172.21.15.107:50670) 2023-11-24T08:19:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 -- 172.21.15.107:0/3220905183 learned_addr learned my addr 172.21.15.107:0/3220905183 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 -- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d700c59b0 msgr2=0x7f3d700c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d700c59b0 0x7f3d700c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 -- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 msgr2=0x7f3d700cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d74e4a640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 0x7f3d700cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 0x7f3d700cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 -- 172.21.15.107:0/3220905183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68009a70 con 0x7f3d700cc0f0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d74e4a640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 0x7f3d700cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.422+0000 7f3d67fff640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800c4a0 tx=0x7f3d5800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.423+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58005ce0 con 0x7f3d700cc0f0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.423+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58005e40 con 0x7f3d700cc0f0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.423+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7015a370 con 0x7f3d700cc0f0 2023-11-24T08:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.423+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58007270 con 0x7f3d700cc0f0 2023-11-24T08:19:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.423+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7015a930 con 0x7f3d700cc0f0 2023-11-24T08:19:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.424+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7004e8b0 con 0x7f3d700cc0f0 2023-11-24T08:19:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.425+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f3d580073d0 con 0x7f3d700cc0f0 2023-11-24T08:19:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.426+0000 7f3d6d7fa640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 0x7f3d5006eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.427+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d580878d0 con 0x7f3d700cc0f0 2023-11-24T08:19:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.427+0000 7f3d6ffff640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 0x7f3d5006eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.427+0000 7f3d6ffff640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 0x7f3d5006eec0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009650 tx=0x7f3d60006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.431+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d58054530 con 0x7f3d700cc0f0 2023-11-24T08:19:49.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.658+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d700c5840 con 0x7f3d5006ca00 2023-11-24T08:19:49.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.671+0000 7f3d6d7fa640 1 -- 172.21.15.107:0/3220905183 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f3d700c5840 con 0x7f3d5006ca00 2023-11-24T08:19:49.673 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:49.675 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:49.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.673+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 msgr2=0x7f3d5006eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.673+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 0x7f3d5006eec0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009650 tx=0x7f3d60006ba0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.673+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 msgr2=0x7f3d70159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:49.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.673+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800c4a0 tx=0x7f3d5800c960 comp rx=0 tx=0).stop 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 shutdown_connections 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d5006ca00 0x7f3d5006eec0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d7015b440 0x7f3d700cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d700cc0f0 0x7f3d70159c20 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 --2- 172.21.15.107:0/3220905183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d700c59b0 0x7f3d700c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 >> 172.21.15.107:0/3220905183 conn(0x7f3d700b0c50 msgr2=0x7f3d700cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:49.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 shutdown_connections 2023-11-24T08:19:49.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:49.674+0000 7f3d768ad640 1 -- 172.21.15.107:0/3220905183 wait complete. 2023-11-24T08:19:50.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.922+0000 7ff53c70f640 1 Processor -- start 2023-11-24T08:19:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.923+0000 7ff53c70f640 1 -- start start 2023-11-24T08:19:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.924+0000 7ff53c70f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff53415b440 0x7ff5340cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.924+0000 7ff53c70f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5340c5dc0 con 0x7ff5340cbf20 2023-11-24T08:19:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.924+0000 7ff53c70f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53415eea0 con 0x7ff5340c59b0 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.924+0000 7ff53c70f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5340cc3d0 con 0x7ff53415b440 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff53415b440 0x7ff5340cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff53415b440 0x7ff5340cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50674/0 (socket says 172.21.15.107:50674) 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 -- 172.21.15.107:0/1972568483 learned_addr learned my addr 172.21.15.107:0/1972568483 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 -- 172.21.15.107:0/1972568483 >> v1:172.21.15.120:6789/0 conn(0x7ff5340c59b0 legacy=0x7ff53406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 -- 172.21.15.107:0/1972568483 >> v1:172.21.15.107:6789/0 conn(0x7ff5340cbf20 legacy=0x7ff5340c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:50.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.925+0000 7ff53a4ab640 1 -- 172.21.15.107:0/1972568483 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5340abf30 con 0x7ff53415b440 2023-11-24T08:19:50.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.926+0000 7ff53a4ab640 1 --2- 172.21.15.107:0/1972568483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53415b440 0x7ff5340cb9e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7ff524009e50 tx=0x7ff52402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7f09801c96e933 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:50.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.926+0000 7ff5394a9640 1 -- 172.21.15.107:0/1972568483 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52402ca80 con 0x7ff53415b440 2023-11-24T08:19:50.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff5394a9640 1 -- 172.21.15.107:0/1972568483 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52402cbe0 con 0x7ff53415b440 2023-11-24T08:19:50.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff5394a9640 1 -- 172.21.15.107:0/1972568483 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240343f0 con 0x7ff53415b440 2023-11-24T08:19:50.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 -- 172.21.15.107:0/1972568483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53415b440 msgr2=0x7ff5340cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:50.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 --2- 172.21.15.107:0/1972568483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53415b440 0x7ff5340cb9e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7ff524009e50 tx=0x7ff52402bee0 comp rx=0 tx=0).stop 2023-11-24T08:19:50.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 -- 172.21.15.107:0/1972568483 shutdown_connections 2023-11-24T08:19:50.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 --2- 172.21.15.107:0/1972568483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53415b440 0x7ff5340cb9e0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:50.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 -- 172.21.15.107:0/1972568483 >> 172.21.15.107:0/1972568483 conn(0x7ff5340b0c50 msgr2=0x7ff5340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:50.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 -- 172.21.15.107:0/1972568483 shutdown_connections 2023-11-24T08:19:50.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.927+0000 7ff53c70f640 1 -- 172.21.15.107:0/1972568483 wait complete. 2023-11-24T08:19:50.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.929+0000 7ff53c70f640 1 Processor -- start 2023-11-24T08:19:50.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.929+0000 7ff53c70f640 1 -- start start 2023-11-24T08:19:50.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5340c59b0 0x7ff53415cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53415b440 0x7ff53409c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53415b9a0 con 0x7ff5340cbf20 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5341571e0 con 0x7ff53415b440 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff53c70f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53415d170 con 0x7ff5340c59b0 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff539caa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff539caa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50682/0 (socket says 172.21.15.107:50682) 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.930+0000 7ff539caa640 1 -- 172.21.15.107:0/931543784 learned_addr learned my addr 172.21.15.107:0/931543784 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff53a4ab640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5340c59b0 0x7ff53415cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff53acac640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53415b440 0x7ff53409c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 -- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5340c59b0 msgr2=0x7ff53415cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5340c59b0 0x7ff53415cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 -- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53415b440 msgr2=0x7ff53409c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53415b440 0x7ff53409c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 -- 172.21.15.107:0/931543784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009b30 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff539caa640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c450 tx=0x7ff52800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.931+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5280052d0 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.932+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528005430 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.932+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528003830 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.932+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53415be90 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.932+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53415c450 con 0x7ff5340cbf20 2023-11-24T08:19:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.933+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53404e8b0 con 0x7ff5340cbf20 2023-11-24T08:19:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.934+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7ff528003990 con 0x7ff5340cbf20 2023-11-24T08:19:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.935+0000 7ff532ffd640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 0x7ff52006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.935+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff528087ab0 con 0x7ff5340cbf20 2023-11-24T08:19:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.935+0000 7ff53a4ab640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 0x7ff52006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.936+0000 7ff53a4ab640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 0x7ff52006ee90 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff524035000 tx=0x7ff524030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:50.939+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff528054660 con 0x7ff5340cbf20 2023-11-24T08:19:51.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.161+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7ff53415f3d0 con 0x7ff52006c9d0 2023-11-24T08:19:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.177+0000 7ff532ffd640 1 -- 172.21.15.107:0/931543784 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7ff5240440c0 con 0x7ff52006c9d0 2023-11-24T08:19:51.179 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:51.180 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.180+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 msgr2=0x7ff52006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.180+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 0x7ff52006ee90 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff524035000 tx=0x7ff524030000 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.180+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 msgr2=0x7ff53415db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.180+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c450 tx=0x7ff52800c910 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 shutdown_connections 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff52006c9d0 0x7ff52006ee90 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53415b440 0x7ff53409c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5340cbf20 0x7ff53415db60 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 --2- 172.21.15.107:0/931543784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5340c59b0 0x7ff53415cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 >> 172.21.15.107:0/931543784 conn(0x7ff5340b0c50 msgr2=0x7ff5340cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 shutdown_connections 2023-11-24T08:19:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:51.181+0000 7ff53c70f640 1 -- 172.21.15.107:0/931543784 wait complete. 2023-11-24T08:19:52.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.426+0000 7f3970d9e640 1 Processor -- start 2023-11-24T08:19:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.426+0000 7f3970d9e640 1 -- start start 2023-11-24T08:19:52.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.428+0000 7f3970d9e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.428+0000 7f3970d9e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f396c0cd110 0x7f396c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.428+0000 7f3970d9e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0cd510 con 0x7f396c146eb0 2023-11-24T08:19:52.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.428+0000 7f3970d9e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c161a10 con 0x7f396c0cd110 2023-11-24T08:19:52.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.428+0000 7f3970d9e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c161cf0 con 0x7f396c06e160 2023-11-24T08:19:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.429+0000 7f3963fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.429+0000 7f396a59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f396c146eb0 0x7f396c161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:56990/0 (socket says 172.21.15.107:56990) 2023-11-24T08:19:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.429+0000 7f396ad9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f396c0cd110 0x7f396c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.429+0000 7f396a59c640 1 -- 172.21.15.107:0/893240075 learned_addr learned my addr 172.21.15.107:0/893240075 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:52.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.430+0000 7f3963fff640 1 -- 172.21.15.107:0/893240075 >> v2:172.21.15.120:3300/0 conn(0x7f396c0cd110 msgr2=0x7f396c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.430+0000 7f3969d9b640 1 -- 172.21.15.107:0/893240075 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240841920 0 0) 0x7f396c0cd510 con 0x7f396c146eb0 2023-11-24T08:19:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.430+0000 7f3963fff640 1 --2- 172.21.15.107:0/893240075 >> v2:172.21.15.120:3300/0 conn(0x7f396c0cd110 0x7f396c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.430+0000 7f3963fff640 1 -- 172.21.15.107:0/893240075 >> v1:172.21.15.120:6789/0 conn(0x7f396c146eb0 legacy=0x7f396c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.430+0000 7f3963fff640 1 -- 172.21.15.107:0/893240075 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c161fd0 con 0x7f396c06e160 2023-11-24T08:19:52.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.431+0000 7f3963fff640 1 --2- 172.21.15.107:0/893240075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c0ccbd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39540097a0 tx=0x7f395402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9859c12377c96694 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:52.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.432+0000 7f3969d9b640 1 -- 172.21.15.107:0/893240075 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395402d040 con 0x7f396c06e160 2023-11-24T08:19:52.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.432+0000 7f3969d9b640 1 -- 172.21.15.107:0/893240075 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954031810 con 0x7f396c06e160 2023-11-24T08:19:52.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.432+0000 7f3969d9b640 1 -- 172.21.15.107:0/893240075 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954031ba0 con 0x7f396c06e160 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 -- 172.21.15.107:0/893240075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 msgr2=0x7f396c0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 --2- 172.21.15.107:0/893240075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c0ccbd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39540097a0 tx=0x7f395402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 -- 172.21.15.107:0/893240075 shutdown_connections 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 --2- 172.21.15.107:0/893240075 >> v2:172.21.15.120:3300/0 conn(0x7f396c0cd110 0x7f396c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 --2- 172.21.15.107:0/893240075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c0ccbd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 -- 172.21.15.107:0/893240075 >> 172.21.15.107:0/893240075 conn(0x7f396c0b0c50 msgr2=0x7f396c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 -- 172.21.15.107:0/893240075 shutdown_connections 2023-11-24T08:19:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.433+0000 7f3970d9e640 1 -- 172.21.15.107:0/893240075 wait complete. 2023-11-24T08:19:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.435+0000 7f3970d9e640 1 Processor -- start 2023-11-24T08:19:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.436+0000 7f3970d9e640 1 -- start start 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c1430a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f396c0cd110 0x7f396c15cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c09a9a0 con 0x7f396c146eb0 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c09ac80 con 0x7f396c0cd110 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.437+0000 7f3970d9e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c09af60 con 0x7f396c06e160 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50684/0 (socket says 172.21.15.107:50684) 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 -- 172.21.15.107:0/716853269 learned_addr learned my addr 172.21.15.107:0/716853269 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396a59c640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c1430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f3963fff640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f396c0cd110 0x7f396c15cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 -- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 msgr2=0x7f396c1430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c1430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 -- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f396c0cd110 msgr2=0x7f396c15cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f396c0cd110 0x7f396c15cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 -- 172.21.15.107:0/716853269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954009400 con 0x7f396c146eb0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.438+0000 7f396ad9d640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f396400efc0 tx=0x7f396400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.439+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964014480 con 0x7f396c146eb0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.439+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396400f040 con 0x7f396c146eb0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.439+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964014820 con 0x7f396c146eb0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.439+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c09b240 con 0x7f396c146eb0 2023-11-24T08:19:52.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.439+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c09ba40 con 0x7f396c146eb0 2023-11-24T08:19:52.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.440+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c09b760 con 0x7f396c146eb0 2023-11-24T08:19:52.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.441+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f3964014980 con 0x7f396c146eb0 2023-11-24T08:19:52.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.442+0000 7f3962ffd640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 0x7f394c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:52.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.442+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f39640890f0 con 0x7f396c146eb0 2023-11-24T08:19:52.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.442+0000 7f396a59c640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 0x7f394c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:52.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.443+0000 7f396a59c640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 0x7f394c06ee90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f395c007fd0 tx=0x7f395c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:52.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.446+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f396c09b760 con 0x7f396c146eb0 2023-11-24T08:19:52.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.672+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f396c147290 con 0x7f394c06c9d0 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.681+0000 7f3962ffd640 1 -- 172.21.15.107:0/716853269 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f395c004c70 con 0x7f394c06c9d0 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:52.683 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:52.684 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:52.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:52.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:52.684 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.683+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 msgr2=0x7f394c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.683+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 0x7f394c06ee90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f395c007fd0 tx=0x7f395c007de0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.683+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 msgr2=0x7f396c15cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.683+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f396400efc0 tx=0x7f396400c490 comp rx=0 tx=0).stop 2023-11-24T08:19:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 shutdown_connections 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f394c06c9d0 0x7f394c06ee90 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396c146eb0 0x7f396c15cff0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f396c0cd110 0x7f396c15cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 --2- 172.21.15.107:0/716853269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f396c06e160 0x7f396c1430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 >> 172.21.15.107:0/716853269 conn(0x7f396c0b0c50 msgr2=0x7f396c0cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 shutdown_connections 2023-11-24T08:19:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:52.684+0000 7f3970d9e640 1 -- 172.21.15.107:0/716853269 wait complete. 2023-11-24T08:19:53.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.932+0000 7ff3d51cb640 1 Processor -- start 2023-11-24T08:19:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.932+0000 7ff3d51cb640 1 -- start start 2023-11-24T08:19:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3d51cb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3d015d3c0 0x7ff3d0160f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3d51cb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3d00cb9d0 0x7ff3d0161480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3d51cb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d0161df0 con 0x7ff3d00c63d0 2023-11-24T08:19:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3d51cb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d0161b10 con 0x7ff3d015d3c0 2023-11-24T08:19:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3d51cb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d00cada0 con 0x7ff3d00cb9d0 2023-11-24T08:19:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.934+0000 7ff3cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3d015d3c0 0x7ff3d0160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3d015d3c0 0x7ff3d0160f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50700/0 (socket says 172.21.15.107:50700) 2023-11-24T08:19:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 -- 172.21.15.107:0/852059620 learned_addr learned my addr 172.21.15.107:0/852059620 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3c7fff640 1 --2- 172.21.15.107:0/852059620 >> v2:172.21.15.120:3300/0 conn(0x7ff3d00cb9d0 0x7ff3d0161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 -- 172.21.15.107:0/852059620 >> v2:172.21.15.120:3300/0 conn(0x7ff3d00cb9d0 msgr2=0x7ff3d0161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 --2- 172.21.15.107:0/852059620 >> v2:172.21.15.120:3300/0 conn(0x7ff3d00cb9d0 0x7ff3d0161480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 -- 172.21.15.107:0/852059620 >> v1:172.21.15.120:6789/0 conn(0x7ff3d00c63d0 legacy=0x7ff3d00cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.935+0000 7ff3cffff640 1 -- 172.21.15.107:0/852059620 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00cb080 con 0x7ff3d015d3c0 2023-11-24T08:19:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.937+0000 7ff3cffff640 1 --2- 172.21.15.107:0/852059620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0160f40 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0072160 tx=0x7ff3c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbedf75e9861ec6c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:53.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.937+0000 7ff3cf7fe640 1 -- 172.21.15.107:0/852059620 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0030860 con 0x7ff3d015d3c0 2023-11-24T08:19:53.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.937+0000 7ff3cf7fe640 1 -- 172.21.15.107:0/852059620 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c00309c0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.937+0000 7ff3cf7fe640 1 -- 172.21.15.107:0/852059620 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0030d50 con 0x7ff3d015d3c0 2023-11-24T08:19:53.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 -- 172.21.15.107:0/852059620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 msgr2=0x7ff3d0160f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:53.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/852059620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0160f40 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0072160 tx=0x7ff3c002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 -- 172.21.15.107:0/852059620 shutdown_connections 2023-11-24T08:19:53.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/852059620 >> v2:172.21.15.120:3300/0 conn(0x7ff3d00cb9d0 0x7ff3d0161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/852059620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0160f40 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 -- 172.21.15.107:0/852059620 >> 172.21.15.107:0/852059620 conn(0x7ff3d00cc340 msgr2=0x7ff3d00cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:53.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 -- 172.21.15.107:0/852059620 shutdown_connections 2023-11-24T08:19:53.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.938+0000 7ff3d51cb640 1 -- 172.21.15.107:0/852059620 wait complete. 2023-11-24T08:19:53.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.940+0000 7ff3d51cb640 1 Processor -- start 2023-11-24T08:19:53.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.940+0000 7ff3d51cb640 1 -- start start 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 0x7ff3d00c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d00cb9d0 0x7ff3d006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006dfd0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006f280 con 0x7ff3d00c63d0 2023-11-24T08:19:53.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d51cb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006e1e0 con 0x7ff3d00cb9d0 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.941+0000 7ff3d49ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50714/0 (socket says 172.21.15.107:50714) 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 -- 172.21.15.107:0/3076617389 learned_addr learned my addr 172.21.15.107:0/3076617389 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3cffff640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 0x7ff3d00c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3c7fff640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d00cb9d0 0x7ff3d006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 -- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d00cb9d0 msgr2=0x7ff3d006da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d00cb9d0 0x7ff3d006da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 -- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 msgr2=0x7ff3d00c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 0x7ff3d00c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:53.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 -- 172.21.15.107:0/3076617389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00094d0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3cffff640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 0x7ff3d00c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.942+0000 7ff3d49ca640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c7a0 tx=0x7ff3c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.943+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8003b70 con 0x7ff3d015d3c0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.943+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8003cf0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.943+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8003ea0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.943+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d006e4c0 con 0x7ff3d015d3c0 2023-11-24T08:19:53.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.943+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d006ea80 con 0x7ff3d015d3c0 2023-11-24T08:19:53.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.944+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d004e840 con 0x7ff3d015d3c0 2023-11-24T08:19:53.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.945+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7ff3c800aa90 con 0x7ff3d015d3c0 2023-11-24T08:19:53.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.946+0000 7ff3cd7fa640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 0x7ff3b006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:53.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.946+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff3c8088470 con 0x7ff3d015d3c0 2023-11-24T08:19:53.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.946+0000 7ff3cffff640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 0x7ff3b006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:53.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.947+0000 7ff3cffff640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 0x7ff3b006ee90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0036000 tx=0x7ff3c003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:53.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:53.950+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3c80bfe90 con 0x7ff3d015d3c0 2023-11-24T08:19:54.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.180+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3d00af1c0 con 0x7ff3b006c9d0 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.189+0000 7ff3cd7fa640 1 -- 172.21.15.107:0/3076617389 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7ff3c0030c30 con 0x7ff3b006c9d0 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:54.192 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.191+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 msgr2=0x7ff3b006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.191+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 0x7ff3b006ee90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0036000 tx=0x7ff3c003b040 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.191+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 msgr2=0x7ff3d0157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.191+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c7a0 tx=0x7ff3c800cc60 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 shutdown_connections 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff3b006c9d0 0x7ff3b006ee90 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d015d3c0 0x7ff3d0157780 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d00cb9d0 0x7ff3d006da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 --2- 172.21.15.107:0/3076617389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d00c63d0 0x7ff3d00c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 >> 172.21.15.107:0/3076617389 conn(0x7ff3d00cc340 msgr2=0x7ff3d00ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 shutdown_connections 2023-11-24T08:19:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:54.192+0000 7ff3d51cb640 1 -- 172.21.15.107:0/3076617389 wait complete. 2023-11-24T08:19:55.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:55.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.444+0000 7fd87dae7640 1 Processor -- start 2023-11-24T08:19:55.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.444+0000 7fd87dae7640 1 -- start start 2023-11-24T08:19:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd87dae7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd87815b440 0x7fd8780cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd87dae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 0x7fd87806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd87dae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87815eea0 con 0x7fd8780cbf20 2023-11-24T08:19:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd87dae7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8780c5e00 con 0x7fd87815b440 2023-11-24T08:19:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd87dae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780cc3d0 con 0x7fd8780c59b0 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd8777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd87815b440 0x7fd8780cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd86ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd8780cbf20 0x7fd8780c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46050/0 (socket says 172.21.15.107:46050) 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.446+0000 7fd86ffff640 1 -- 172.21.15.107:0/4267673128 learned_addr learned my addr 172.21.15.107:0/4267673128 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd877fff640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 0x7fd87806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd8777fe640 1 -- 172.21.15.107:0/4267673128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 msgr2=0x7fd87806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd8777fe640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 0x7fd87806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd8777fe640 1 -- 172.21.15.107:0/4267673128 >> v1:172.21.15.107:6789/0 conn(0x7fd8780cbf20 legacy=0x7fd8780c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd876ffd640 1 -- 172.21.15.107:0/4267673128 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292738594 0 0) 0x7fd87815eea0 con 0x7fd8780cbf20 2023-11-24T08:19:55.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd8777fe640 1 -- 172.21.15.107:0/4267673128 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780abf30 con 0x7fd87815b440 2023-11-24T08:19:55.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.447+0000 7fd877fff640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 0x7fd87806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:55.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.448+0000 7fd8777fe640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd8780cb9e0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fd8680097a0 tx=0x7fd86802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6eb8ed74a671cd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:55.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.449+0000 7fd876ffd640 1 -- 172.21.15.107:0/4267673128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868031780 con 0x7fd87815b440 2023-11-24T08:19:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.449+0000 7fd876ffd640 1 -- 172.21.15.107:0/4267673128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8680318e0 con 0x7fd87815b440 2023-11-24T08:19:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.449+0000 7fd876ffd640 1 -- 172.21.15.107:0/4267673128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868031c70 con 0x7fd87815b440 2023-11-24T08:19:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.449+0000 7fd87dae7640 1 -- 172.21.15.107:0/4267673128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 msgr2=0x7fd8780cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.449+0000 7fd87dae7640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd8780cb9e0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fd8680097a0 tx=0x7fd86802b820 comp rx=0 tx=0).stop 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 -- 172.21.15.107:0/4267673128 shutdown_connections 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780c59b0 0x7fd87806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 --2- 172.21.15.107:0/4267673128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd8780cb9e0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 -- 172.21.15.107:0/4267673128 >> 172.21.15.107:0/4267673128 conn(0x7fd8780b0c50 msgr2=0x7fd8780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 -- 172.21.15.107:0/4267673128 shutdown_connections 2023-11-24T08:19:55.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.450+0000 7fd87dae7640 1 -- 172.21.15.107:0/4267673128 wait complete. 2023-11-24T08:19:55.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.451+0000 7fd87dae7640 1 Processor -- start 2023-11-24T08:19:55.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.451+0000 7fd87dae7640 1 -- start start 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8780c59b0 0x7fd878155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780cbf20 0x7fd87815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878155a60 con 0x7fd87815b440 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87815b9a0 con 0x7fd8780c59b0 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.452+0000 7fd87dae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780b3ed0 con 0x7fd8780cbf20 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50736/0 (socket says 172.21.15.107:50736) 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 -- 172.21.15.107:0/3374653411 learned_addr learned my addr 172.21.15.107:0/3374653411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd86ffff640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780cbf20 0x7fd87815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 -- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780cbf20 msgr2=0x7fd87815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780cbf20 0x7fd87815dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 -- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8780c59b0 msgr2=0x7fd878155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8780c59b0 0x7fd878155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 -- 172.21.15.107:0/3374653411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868009400 con 0x7fd87815b440 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.453+0000 7fd877fff640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fd87000cef0 tx=0x7fd87000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.454+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87000d840 con 0x7fd87815b440 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.454+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87000d9c0 con 0x7fd87815b440 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.454+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87000dd70 con 0x7fd87815b440 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.454+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8780b41b0 con 0x7fd87815b440 2023-11-24T08:19:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.454+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87815bf20 con 0x7fd87815b440 2023-11-24T08:19:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.455+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8780b43d0 con 0x7fd87815b440 2023-11-24T08:19:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.456+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7fd87001e3b0 con 0x7fd87815b440 2023-11-24T08:19:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.457+0000 7fd874ff9640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 0x7fd85806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.457+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd870088fc0 con 0x7fd87815b440 2023-11-24T08:19:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.457+0000 7fd8777fe640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 0x7fd85806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:55.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.458+0000 7fd8777fe640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 0x7fd85806ee90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd868036000 tx=0x7fd868030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:55.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.462+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8780b43d0 con 0x7fd87815b440 2023-11-24T08:19:55.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.690+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8780c9630 con 0x7fd85806c9d0 2023-11-24T08:19:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.708+0000 7fd874ff9640 1 -- 172.21.15.107:0/3374653411 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fd8780c9630 con 0x7fd85806c9d0 2023-11-24T08:19:55.710 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:19:55.710 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4786", 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap8897" 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:19:55.711 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:19:55.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 msgr2=0x7fd85806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 0x7fd85806ee90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd868036000 tx=0x7fd868030000 comp rx=0 tx=0).stop 2023-11-24T08:19:55.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 msgr2=0x7fd87815e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fd87000cef0 tx=0x7fd87000a710 comp rx=0 tx=0).stop 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 shutdown_connections 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd85806c9d0 0x7fd85806ee90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd87815b440 0x7fd87815e180 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8780cbf20 0x7fd87815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 --2- 172.21.15.107:0/3374653411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8780c59b0 0x7fd878155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.711+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 >> 172.21.15.107:0/3374653411 conn(0x7fd8780b0c50 msgr2=0x7fd8780cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.712+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 shutdown_connections 2023-11-24T08:19:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:55.712+0000 7fd87dae7640 1 -- 172.21.15.107:0/3374653411 wait complete. 2023-11-24T08:19:56.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone8559 2023-11-24T08:19:56.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.957+0000 7f06e11cf640 1 Processor -- start 2023-11-24T08:19:56.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.957+0000 7f06e11cf640 1 -- start start 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06e11cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f06dc15b260 0x7f06dc06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06e11cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f06dc06e580 0x7f06dc0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06e11cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06dc06d5c0 con 0x7f06dc0b1080 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06e11cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06dc15b640 con 0x7f06dc15b260 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06e11cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06dc0b14d0 con 0x7f06dc06e580 2023-11-24T08:19:56.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06dbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f06dc15b260 0x7f06dc06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06dbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f06dc15b260 0x7f06dc06e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50748/0 (socket says 172.21.15.107:50748) 2023-11-24T08:19:56.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.959+0000 7f06dbfff640 1 -- 172.21.15.107:0/167107996 learned_addr learned my addr 172.21.15.107:0/167107996 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:56.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.960+0000 7f06dbfff640 1 -- 172.21.15.107:0/167107996 >> v2:172.21.15.120:3300/0 conn(0x7f06dc06e580 msgr2=0x7f06dc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:56.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.960+0000 7f06d3fff640 1 --2- 172.21.15.107:0/167107996 >> v2:172.21.15.120:3300/0 conn(0x7f06dc06e580 0x7f06dc0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.960+0000 7f06dbfff640 1 --2- 172.21.15.107:0/167107996 >> v2:172.21.15.120:3300/0 conn(0x7f06dc06e580 0x7f06dc0b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.960+0000 7f06dbfff640 1 -- 172.21.15.107:0/167107996 >> v1:172.21.15.120:6789/0 conn(0x7f06dc0b1080 legacy=0x7f06dc0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:19:56.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.960+0000 7f06dbfff640 1 -- 172.21.15.107:0/167107996 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc15ed20 con 0x7f06dc15b260 2023-11-24T08:19:56.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.961+0000 7f06dbfff640 1 --2- 172.21.15.107:0/167107996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc15b260 0x7f06dc06e040 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0ce370 tx=0x7f06cc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0cac26cdc52974 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.961+0000 7f06db7fe640 1 -- 172.21.15.107:0/167107996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc031750 con 0x7f06dc15b260 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.961+0000 7f06db7fe640 1 -- 172.21.15.107:0/167107996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc0318b0 con 0x7f06dc15b260 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.961+0000 7f06db7fe640 1 -- 172.21.15.107:0/167107996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc031c40 con 0x7f06dc15b260 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 -- 172.21.15.107:0/167107996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc15b260 msgr2=0x7f06dc06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 --2- 172.21.15.107:0/167107996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc15b260 0x7f06dc06e040 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0ce370 tx=0x7f06cc02b7f0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 -- 172.21.15.107:0/167107996 shutdown_connections 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 --2- 172.21.15.107:0/167107996 >> v2:172.21.15.120:3300/0 conn(0x7f06dc06e580 0x7f06dc0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 --2- 172.21.15.107:0/167107996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc15b260 0x7f06dc06e040 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 -- 172.21.15.107:0/167107996 >> 172.21.15.107:0/167107996 conn(0x7f06dc0cbb40 msgr2=0x7f06dc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 -- 172.21.15.107:0/167107996 shutdown_connections 2023-11-24T08:19:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.962+0000 7f06e11cf640 1 -- 172.21.15.107:0/167107996 wait complete. 2023-11-24T08:19:56.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.964+0000 7f06e11cf640 1 Processor -- start 2023-11-24T08:19:56.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.964+0000 7f06e11cf640 1 -- start start 2023-11-24T08:19:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06dc06e580 0x7f06dc158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 0x7f06dc158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0b1fb0 con 0x7f06dc0b1080 2023-11-24T08:19:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0b2df0 con 0x7f06dc15b260 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06e11cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0b21e0 con 0x7f06dc06e580 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50756/0 (socket says 172.21.15.107:50756) 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.965+0000 7f06d3fff640 1 -- 172.21.15.107:0/866168216 learned_addr learned my addr 172.21.15.107:0/866168216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06dbfff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06dc06e580 0x7f06dc158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 -- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06dc06e580 msgr2=0x7f06dc158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06e09ce640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 0x7f06dc158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06dc06e580 0x7f06dc158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 -- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 msgr2=0x7f06dc158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 0x7f06dc158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 -- 172.21.15.107:0/866168216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0094d0 con 0x7f06dc0b1080 2023-11-24T08:19:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06e09ce640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 0x7f06dc158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d3fff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7f0 tx=0x7f06c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4004220 con 0x7f06dc0b1080 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4004380 con 0x7f06dc0b1080 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4007840 con 0x7f06dc0b1080 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06dc0b24c0 con 0x7f06dc0b1080 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.966+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc0b2a80 con 0x7f06dc0b1080 2023-11-24T08:19:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.968+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06dc0b2fa0 con 0x7f06dc0b1080 2023-11-24T08:19:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.968+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f06c4007bb0 con 0x7f06dc0b1080 2023-11-24T08:19:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.969+0000 7f06d97fa640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 0x7f06bc06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:19:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.969+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f06c4088470 con 0x7f06dc0b1080 2023-11-24T08:19:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.970+0000 7f06dbfff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 0x7f06bc06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:19:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.970+0000 7f06dbfff640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 0x7f06bc06ee90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f06cc035000 tx=0x7f06cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:19:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:56.974+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06dc0b2fa0 con 0x7f06dc0b1080 2023-11-24T08:19:57.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:19:57.198+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f06dc15ef70 con 0x7f06bc06c9d0 2023-11-24T08:20:01.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.783+0000 7f06d97fa640 1 -- 172.21.15.107:0/866168216 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f06cc0430c0 con 0x7f06bc06c9d0 2023-11-24T08:20:01.785 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:20:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:20:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:01.785 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:01.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.786+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 msgr2=0x7f06bc06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:01.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.786+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 0x7f06bc06ee90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f06cc035000 tx=0x7f06cc008030 comp rx=0 tx=0).stop 2023-11-24T08:20:01.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.786+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 msgr2=0x7f06dc0b1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7f0 tx=0x7f06c400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 shutdown_connections 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f06bc06c9d0 0x7f06bc06ee90 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06dc15b260 0x7f06dc158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06dc0b1080 0x7f06dc0b1a70 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:01.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 --2- 172.21.15.107:0/866168216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06dc06e580 0x7f06dc158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:01.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 >> 172.21.15.107:0/866168216 conn(0x7f06dc0cbb40 msgr2=0x7f06dc0af600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:01.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 shutdown_connections 2023-11-24T08:20:01.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:01.787+0000 7f06e11cf640 1 -- 172.21.15.107:0/866168216 wait complete. 2023-11-24T08:20:01.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4786 2023-11-24T08:20:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.025+0000 7fb691431640 1 Processor -- start 2023-11-24T08:20:02.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.026+0000 7fb691431640 1 -- start start 2023-11-24T08:20:02.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.027+0000 7fb691431640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb68c146eb0 0x7fb68c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb691431640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c161a10 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb691431640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c161cf0 con 0x7fb68c06e160 2023-11-24T08:20:02.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb691431640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c161fd0 con 0x7fb68c146eb0 2023-11-24T08:20:02.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb68b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb68c0cd2e0 0x7fb68c070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53684/0 (socket says 172.21.15.107:53684) 2023-11-24T08:20:02.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb68b7fe640 1 -- 172.21.15.107:0/446248966 learned_addr learned my addr 172.21.15.107:0/446248966 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb68affd640 1 --2- 172.21.15.107:0/446248966 >> v2:172.21.15.120:3300/0 conn(0x7fb68c146eb0 0x7fb68c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239658013 0 0) 0x7fb68c161a10 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.028+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678003520 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905576663 0 0) 0x7fb678003520 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 >> v2:172.21.15.120:3300/0 conn(0x7fb68c146eb0 msgr2=0x7fb68c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 --2- 172.21.15.107:0/446248966 >> v2:172.21.15.120:3300/0 conn(0x7fb68c146eb0 0x7fb68c161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 >> v1:172.21.15.120:6789/0 conn(0x7fb68c06e160 legacy=0x7fb68c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c070660 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb684001670 con 0x7fb68c0cd2e0 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 >> v1:172.21.15.107:6789/0 conn(0x7fb68c0cd2e0 legacy=0x7fb68c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678003970 0x7fb678003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6780047c0 0x7fb678004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb67800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c161fd0 con 0x7fb678005180 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67800c020 con 0x7fb678003970 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67800c1e0 con 0x7fb6780047c0 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.029+0000 7fb68b7fe640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb67800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68affd640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678003970 0x7fb678003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb6827fc640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6780047c0 0x7fb678004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68b7fe640 1 -- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6780047c0 msgr2=0x7fb678004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68b7fe640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6780047c0 0x7fb678004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68b7fe640 1 -- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678003970 msgr2=0x7fb678003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68b7fe640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678003970 0x7fb678003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.030+0000 7fb68b7fe640 1 -- 172.21.15.107:0/446248966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb678003520 con 0x7fb678005180 2023-11-24T08:20:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.031+0000 7fb68b7fe640 1 -- 172.21.15.107:0/446248966 reap_dead start 2023-11-24T08:20:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.031+0000 7fb68b7fe640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb67800b980 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c240 tx=0x7fb6780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc890b114072496 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.031+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684005ad0 con 0x7fb678005180 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.031+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684005c70 con 0x7fb678005180 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.031+0000 7fb68a7fc640 1 -- 172.21.15.107:0/446248966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68402c8b0 con 0x7fb678005180 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 -- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 msgr2=0x7fb67800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb67800b980 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c240 tx=0x7fb6780057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 -- 172.21.15.107:0/446248966 shutdown_connections 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 --2- 172.21.15.107:0/446248966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb67800b980 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 -- 172.21.15.107:0/446248966 >> 172.21.15.107:0/446248966 conn(0x7fb68c0b0c50 msgr2=0x7fb68c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 -- 172.21.15.107:0/446248966 shutdown_connections 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.032+0000 7fb691431640 1 -- 172.21.15.107:0/446248966 wait complete. 2023-11-24T08:20:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.034+0000 7fb691431640 1 Processor -- start 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.034+0000 7fb691431640 1 -- start start 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb68c15ed40 0x7fb68c143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb68c15fb70 0x7fb68c15fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c1e87d0 con 0x7fb678005180 2023-11-24T08:20:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c160530 con 0x7fb68c15ed40 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb691431640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0b2cc0 con 0x7fb68c15fb70 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb68affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb68affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50334/0 (socket says 172.21.15.107:50334) 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb68affd640 1 -- 172.21.15.107:0/333194801 learned_addr learned my addr 172.21.15.107:0/333194801 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.035+0000 7fb68affd640 1 -- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb68c15fb70 msgr2=0x7fb68c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb68affd640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb68c15fb70 0x7fb68c15fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb68affd640 1 -- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb68c15ed40 msgr2=0x7fb68c143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb68affd640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb68c15ed40 0x7fb68c143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb68affd640 1 -- 172.21.15.107:0/333194801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c15f1c0 con 0x7fb678005180 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb68affd640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fb67c011540 tx=0x7fb67c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c015930 con 0x7fb678005180 2023-11-24T08:20:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c015a90 con 0x7fb678005180 2023-11-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c1e7f30 con 0x7fb678005180 2023-11-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c015e60 con 0x7fb678005180 2023-11-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.036+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c1e84f0 con 0x7fb678005180 2023-11-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.038+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c04e8b0 con 0x7fb678005180 2023-11-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.038+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7fb67c00f460 con 0x7fb678005180 2023-11-24T08:20:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.039+0000 7fb683fff640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 0x7fb66c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.040+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb67c088ca0 con 0x7fb678005180 2023-11-24T08:20:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.040+0000 7fb6827fc640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 0x7fb66c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.040+0000 7fb6827fc640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 0x7fb66c06ee90 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb68c09d0e0 tx=0x7fb674009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.045+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb67c055850 con 0x7fb678005180 2023-11-24T08:20:02.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.261+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4786", "target": ["mon-mgr", ""]}) v1 -- 0x7fb68c147290 con 0x7fb66c06c9d0 2023-11-24T08:20:02.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.281+0000 7fb683fff640 1 -- 172.21.15.107:0/333194801 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb674007e30 con 0x7fb66c06c9d0 2023-11-24T08:20:02.282 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 msgr2=0x7fb66c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 0x7fb66c06ee90 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb68c09d0e0 tx=0x7fb674009040 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 msgr2=0x7fb68c1e7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fb67c011540 tx=0x7fb67c011a00 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 shutdown_connections 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb66c06c9d0 0x7fb66c06ee90 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb68c15fb70 0x7fb68c15fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb68c15ed40 0x7fb68c143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 --2- 172.21.15.107:0/333194801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb678005180 0x7fb68c1e7830 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.283+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 >> 172.21.15.107:0/333194801 conn(0x7fb68c0b0c50 msgr2=0x7fb68c070b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.284+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 shutdown_connections 2023-11-24T08:20:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.284+0000 7fb691431640 1 -- 172.21.15.107:0/333194801 wait complete. 2023-11-24T08:20:02.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone8559 2023-11-24T08:20:02.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.509+0000 7f1368232640 1 Processor -- start 2023-11-24T08:20:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.510+0000 7f1368232640 1 -- start start 2023-11-24T08:20:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1368232640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1360072160 0x7f136015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1368232640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 0x7f136006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1368232640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136015ec90 con 0x7f136006dc40 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1368232640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136006e090 con 0x7f1360072160 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1368232640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13600cd530 con 0x7f13600c49f0 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1365fce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1360072160 0x7f136015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1365fce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1360072160 0x7f136015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50336/0 (socket says 172.21.15.107:50336) 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.511+0000 7f1365fce640 1 -- 172.21.15.107:0/2196949438 learned_addr learned my addr 172.21.15.107:0/2196949438 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f13657cd640 1 --2- 172.21.15.107:0/2196949438 >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 0x7f136006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f1365fce640 1 -- 172.21.15.107:0/2196949438 >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 msgr2=0x7f136006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f1365fce640 1 --2- 172.21.15.107:0/2196949438 >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 0x7f136006d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f1365fce640 1 -- 172.21.15.107:0/2196949438 >> v1:172.21.15.120:6789/0 conn(0x7f136006dc40 legacy=0x7f13600cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f1365fce640 1 -- 172.21.15.107:0/2196949438 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600c4ed0 con 0x7f1360072160 2023-11-24T08:20:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.512+0000 7f13657cd640 1 --2- 172.21.15.107:0/2196949438 >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 0x7f136006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.513+0000 7f1365fce640 1 --2- 172.21.15.107:0/2196949438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015e750 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f13600cef60 tx=0x7f135002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83903ad8d603685 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.513+0000 7f1364fcc640 1 -- 172.21.15.107:0/2196949438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350031870 con 0x7f1360072160 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.513+0000 7f1364fcc640 1 -- 172.21.15.107:0/2196949438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13500319d0 con 0x7f1360072160 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.513+0000 7f1364fcc640 1 -- 172.21.15.107:0/2196949438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350031d60 con 0x7f1360072160 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 -- 172.21.15.107:0/2196949438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 msgr2=0x7f136015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 --2- 172.21.15.107:0/2196949438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015e750 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f13600cef60 tx=0x7f135002b910 comp rx=0 tx=0).stop 2023-11-24T08:20:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 -- 172.21.15.107:0/2196949438 shutdown_connections 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 --2- 172.21.15.107:0/2196949438 >> v2:172.21.15.120:3300/0 conn(0x7f13600c49f0 0x7f136006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 --2- 172.21.15.107:0/2196949438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015e750 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 -- 172.21.15.107:0/2196949438 >> 172.21.15.107:0/2196949438 conn(0x7f13600b0c50 msgr2=0x7f13600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 -- 172.21.15.107:0/2196949438 shutdown_connections 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.514+0000 7f1368232640 1 -- 172.21.15.107:0/2196949438 wait complete. 2023-11-24T08:20:02.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.516+0000 7f1368232640 1 Processor -- start 2023-11-24T08:20:02.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.516+0000 7f1368232640 1 -- start start 2023-11-24T08:20:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f136006dc40 0x7f136014bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 0x7f13600c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136014c480 con 0x7f1360072160 2023-11-24T08:20:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136015ba90 con 0x7f136006dc40 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f1368232640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136015ae80 con 0x7f13600c49f0 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f13657cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f13657cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50340/0 (socket says 172.21.15.107:50340) 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.517+0000 7f13657cd640 1 -- 172.21.15.107:0/3461974161 learned_addr learned my addr 172.21.15.107:0/3461974161 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13667cf640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 0x7f13600c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f1365fce640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f136006dc40 0x7f136014bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 -- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 msgr2=0x7f13600c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 0x7f13600c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 -- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f136006dc40 msgr2=0x7f136014bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f136006dc40 0x7f136014bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 -- 172.21.15.107:0/3461974161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500094d0 con 0x7f1360072160 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13667cf640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 0x7f13600c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13657cd640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7f0 tx=0x7f135c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.518+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c007a30 con 0x7f1360072160 2023-11-24T08:20:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.519+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c007b90 con 0x7f1360072160 2023-11-24T08:20:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.519+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136015b160 con 0x7f1360072160 2023-11-24T08:20:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.519+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c00ce40 con 0x7f1360072160 2023-11-24T08:20:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.519+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136015b720 con 0x7f1360072160 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.520+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136004e8b0 con 0x7f1360072160 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.521+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f135c0113f0 con 0x7f1360072160 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.521+0000 7f13567fc640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 0x7f134c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.522+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f135c0887e0 con 0x7f1360072160 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.522+0000 7f1365fce640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 0x7f134c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.523+0000 7f1365fce640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 0x7f134c06ee90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1350036000 tx=0x7f135003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.526+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f135c055260 con 0x7f1360072160 2023-11-24T08:20:02.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.752+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f13600c8bb0 con 0x7f134c06c9d0 2023-11-24T08:20:02.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.763+0000 7f13567fc640 1 -- 172.21.15.107:0/3461974161 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f13600c8bb0 con 0x7f134c06c9d0 2023-11-24T08:20:02.764 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b 2023-11-24T08:20:02.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.765+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 msgr2=0x7f134c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.765+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 0x7f134c06ee90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1350036000 tx=0x7f135003c040 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.765+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 msgr2=0x7f136015c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.765+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7f0 tx=0x7f135c00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 shutdown_connections 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f134c06c9d0 0x7f134c06ee90 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13600c49f0 0x7f13600c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1360072160 0x7f136015c960 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 --2- 172.21.15.107:0/3461974161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f136006dc40 0x7f136014bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 >> 172.21.15.107:0/3461974161 conn(0x7f13600b0c50 msgr2=0x7f136006ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 shutdown_connections 2023-11-24T08:20:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:02.766+0000 7f1368232640 1 -- 172.21.15.107:0/3461974161 wait complete. 2023-11-24T08:20:02.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:02.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:02.836 INFO:teuthology.orchestra.run.smithi107.stdout:9 2023-11-24T08:20:02.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:02.837 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b) 2023-11-24T08:20:02.889 INFO:teuthology.orchestra.run.smithi107.stdout:9 2023-11-24T08:20:02.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone8559 2023-11-24T08:20:03.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.105+0000 7fefedcee640 1 Processor -- start 2023-11-24T08:20:03.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.105+0000 7fefedcee640 1 -- start start 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.106+0000 7fefedcee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefe8161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.106+0000 7fefedcee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe806dcf0 con 0x7fefe80b3330 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.106+0000 7fefedcee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe8161980 con 0x7fefe806b6c0 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.106+0000 7fefedcee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe806ec10 con 0x7fefe8161060 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.107+0000 7fefdffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fefe80b3330 0x7fefe815da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53700/0 (socket says 172.21.15.107:53700) 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.107+0000 7fefdffff640 1 -- 172.21.15.107:0/473898993 learned_addr learned my addr 172.21.15.107:0/473898993 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.107+0000 7fefe77fe640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefe8161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.107+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690268505 0 0) 0x7fefe806dcf0 con 0x7fefe80b3330 2023-11-24T08:20:03.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.107+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd4003520 con 0x7fefe80b3330 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708769260 0 0) 0x7fefd4003520 con 0x7fefe80b3330 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 msgr2=0x7fefe8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefe8161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 >> v1:172.21.15.120:6789/0 conn(0x7fefe806b6c0 legacy=0x7fefe815e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8161ec0 con 0x7fefe80b3330 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fefd00017f0 con 0x7fefe80b3330 2023-11-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 >> v1:172.21.15.107:6789/0 conn(0x7fefe80b3330 legacy=0x7fefe815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefd40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefd4003f20 0x7fefd40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefd4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe806ec10 con 0x7fefd4004d00 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd400bb10 con 0x7fefd4003f20 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd400bcd0 con 0x7fefe8161060 2023-11-24T08:20:03.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe7fff640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefd4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefe77fe640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefd40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.108+0000 7fefdffff640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefd4003f20 0x7fefd40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.109+0000 7fefe7fff640 1 -- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 msgr2=0x7fefd40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.109+0000 7fefe7fff640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8161060 0x7fefd40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.109+0000 7fefe7fff640 1 -- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefd4003f20 msgr2=0x7fefd40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.109+0000 7fefe7fff640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefd4003f20 0x7fefd40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.109+0000 7fefe7fff640 1 -- 172.21.15.107:0/473898993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4003520 con 0x7fefd4004d00 2023-11-24T08:20:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.110+0000 7fefe7fff640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefd4005180 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fefe0001e50 tx=0x7fefd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc285f0e4c8e118 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.110+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0003b70 con 0x7fefd4004d00 2023-11-24T08:20:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.110+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0003cd0 con 0x7fefd4004d00 2023-11-24T08:20:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.110+0000 7fefe6ffd640 1 -- 172.21.15.107:0/473898993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0005310 con 0x7fefd4004d00 2023-11-24T08:20:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 -- 172.21.15.107:0/473898993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 msgr2=0x7fefd4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 --2- 172.21.15.107:0/473898993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefd4005180 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fefe0001e50 tx=0x7fefd40057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefe7fff640 1 -- 172.21.15.107:0/473898993 reap_dead start 2023-11-24T08:20:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 -- 172.21.15.107:0/473898993 shutdown_connections 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 -- 172.21.15.107:0/473898993 >> 172.21.15.107:0/473898993 conn(0x7fefe80cf900 msgr2=0x7fefe80cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 -- 172.21.15.107:0/473898993 shutdown_connections 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.111+0000 7fefedcee640 1 -- 172.21.15.107:0/473898993 wait complete. 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.113+0000 7fefedcee640 1 Processor -- start 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.113+0000 7fefedcee640 1 -- start start 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8143820 0x7fefe8159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe8143db0 0x7fefe8144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe81598a0 con 0x7fefd4004d00 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8159ad0 con 0x7fefe8143db0 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefedcee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8159db0 con 0x7fefe8143820 2023-11-24T08:20:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50364/0 (socket says 172.21.15.107:50364) 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 -- 172.21.15.107:0/2108846668 learned_addr learned my addr 172.21.15.107:0/2108846668 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 -- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8143820 msgr2=0x7fefe8159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8143820 0x7fefe8159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 -- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe8143db0 msgr2=0x7fefe8144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe8143db0 0x7fefe8144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.114+0000 7fefe77fe640 1 -- 172.21.15.107:0/2108846668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe815a090 con 0x7fefd4004d00 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefe77fe640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fefd8011600 tx=0x7fefd8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8015870 con 0x7fefd4004d00 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd80159d0 con 0x7fefd4004d00 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8015d40 con 0x7fefd4004d00 2023-11-24T08:20:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe815a370 con 0x7fefd4004d00 2023-11-24T08:20:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.115+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe8144780 con 0x7fefd4004d00 2023-11-24T08:20:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.116+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe804e8b0 con 0x7fefd4004d00 2023-11-24T08:20:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.117+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7fefd800f490 con 0x7fefd4004d00 2023-11-24T08:20:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.118+0000 7fefe4ff9640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 0x7fefc806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.118+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fefd8088a30 con 0x7fefd4004d00 2023-11-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.118+0000 7fefdffff640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 0x7fefc806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.119+0000 7fefdffff640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 0x7fefc806ee90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000ef0 tx=0x7fefd0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.123+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefd8055540 con 0x7fefd4004d00 2023-11-24T08:20:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.340+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7fefe8159f20 con 0x7fefc806c9d0 2023-11-24T08:20:03.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.361+0000 7fefe4ff9640 1 -- 172.21.15.107:0/2108846668 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+701 (secure 0 0 0) 0x7fefe8159f20 con 0x7fefc806c9d0 2023-11-24T08:20:03.363 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:03.363 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:19:35", 2023-11-24T08:20:03.363 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "66.67", 2023-11-24T08:20:03.363 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": 12582912, 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 8388608, 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:19:38", 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:19:55", 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:20:03.364 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:20:03.365 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:19:36", 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b", 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:20:03.366 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:20:03.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:20:03.367 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:20:03.367 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 msgr2=0x7fefc806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 0x7fefc806ee90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000ef0 tx=0x7fefd0009040 comp rx=0 tx=0).stop 2023-11-24T08:20:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 msgr2=0x7fefe8158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fefd8011600 tx=0x7fefd8011ac0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 shutdown_connections 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fefc806c9d0 0x7fefc806ee90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe8143db0 0x7fefe8144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8143820 0x7fefe8159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 --2- 172.21.15.107:0/2108846668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefd4004d00 0x7fefe8158b20 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 >> 172.21.15.107:0/2108846668 conn(0x7fefe80cf900 msgr2=0x7fefe806a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 shutdown_connections 2023-11-24T08:20:03.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:03.364+0000 7fefedcee640 1 -- 172.21.15.107:0/2108846668 wait complete. 2023-11-24T08:20:03.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.383 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.436 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411: ceph.quota.max_bytes: No such attribute 2023-11-24T08:20:03.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:20:03.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.437 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.489 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:20:03.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.489 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.595 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411 2023-11-24T08:20:03.595 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2 2023-11-24T08:20:03.595 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3 2023-11-24T08:20:03.596 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1 2023-11-24T08:20:03.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.646 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:20:03.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.647 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/) 2023-11-24T08:20:03.698 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:20:03.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.752 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:20:03.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.755 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/) 2023-11-24T08:20:03.810 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:20:03.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.811 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.867 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:20:03.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.868 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/) 2023-11-24T08:20:03.923 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:20:03.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.924 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:03.978 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:03.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:03.979 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/) 2023-11-24T08:20:04.033 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:04.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.035 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2) 2023-11-24T08:20:04.087 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.088 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.2) 2023-11-24T08:20:04.143 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.144 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2) 2023-11-24T08:20:04.195 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:04.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.196 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.2) 2023-11-24T08:20:04.246 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:04.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.247 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2) 2023-11-24T08:20:04.298 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:04.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.300 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.2) 2023-11-24T08:20:04.350 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:04.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.351 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.2) 2023-11-24T08:20:04.401 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:04.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.2) 2023-11-24T08:20:04.453 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:04.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.454 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5) 2023-11-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.509 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.5) 2023-11-24T08:20:04.561 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.563 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5) 2023-11-24T08:20:04.617 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:04.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.618 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.5) 2023-11-24T08:20:04.671 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:04.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.672 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5) 2023-11-24T08:20:04.723 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:04.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.5) 2023-11-24T08:20:04.775 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:04.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.5) 2023-11-24T08:20:04.827 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:04.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.829 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.5) 2023-11-24T08:20:04.881 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:04.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.883 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4) 2023-11-24T08:20:04.936 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.938 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.4) 2023-11-24T08:20:04.988 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:04.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:04.989 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4) 2023-11-24T08:20:05.040 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.041 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.4) 2023-11-24T08:20:05.089 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.090 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4) 2023-11-24T08:20:05.142 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:05.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.143 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.4) 2023-11-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:05.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.194 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.4) 2023-11-24T08:20:05.250 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:05.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.4) 2023-11-24T08:20:05.305 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:05.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.306 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7) 2023-11-24T08:20:05.358 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:05.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.7) 2023-11-24T08:20:05.409 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:05.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.410 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7) 2023-11-24T08:20:05.459 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.461 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.7) 2023-11-24T08:20:05.510 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.511 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7) 2023-11-24T08:20:05.561 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:05.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.562 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.7) 2023-11-24T08:20:05.614 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:05.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.615 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.7) 2023-11-24T08:20:05.671 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:05.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.672 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.7) 2023-11-24T08:20:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:05.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.726 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6) 2023-11-24T08:20:05.779 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:05.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.6) 2023-11-24T08:20:05.832 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:05.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.833 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6) 2023-11-24T08:20:05.885 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.886 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.6) 2023-11-24T08:20:05.939 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:05.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.941 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6) 2023-11-24T08:20:05.991 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:05.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:05.992 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.6) 2023-11-24T08:20:06.042 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:06.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.6) 2023-11-24T08:20:06.096 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:06.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.097 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.6) 2023-11-24T08:20:06.148 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:06.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.149 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0) 2023-11-24T08:20:06.206 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:06.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.207 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.0) 2023-11-24T08:20:06.262 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:06.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.263 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0) 2023-11-24T08:20:06.314 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:06.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.315 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.0) 2023-11-24T08:20:06.367 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:06.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.368 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0) 2023-11-24T08:20:06.422 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:06.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.423 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.0) 2023-11-24T08:20:06.475 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:06.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.476 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.0) 2023-11-24T08:20:06.526 INFO:teuthology.orchestra.run.smithi107.stdout:1700813975 2023-11-24T08:20:06.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.528 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.0) 2023-11-24T08:20:06.580 INFO:teuthology.orchestra.run.smithi107.stdout:1700813975 2023-11-24T08:20:06.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.581 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3) 2023-11-24T08:20:06.635 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:06.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.637 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.3) 2023-11-24T08:20:06.690 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:06.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.692 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3) 2023-11-24T08:20:06.747 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:06.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.748 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.3) 2023-11-24T08:20:06.799 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:06.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.800 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3) 2023-11-24T08:20:06.851 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:06.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.853 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.3) 2023-11-24T08:20:06.902 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:06.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.904 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.3) 2023-11-24T08:20:06.952 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:06.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:06.954 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.3) 2023-11-24T08:20:07.004 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:07.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.005 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1) 2023-11-24T08:20:07.056 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:07.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.057 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.1) 2023-11-24T08:20:07.107 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:20:07.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.109 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1) 2023-11-24T08:20:07.157 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:07.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.158 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.1) 2023-11-24T08:20:07.208 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:20:07.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.209 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1) 2023-11-24T08:20:07.260 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:07.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.261 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.1) 2023-11-24T08:20:07.311 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:20:07.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.312 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4786/.snap/subvol_snap8897/08732b8d-ca3c-4370-a65d-45d6feb63411/subvolume_file.1) 2023-11-24T08:20:07.363 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:07.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.364 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b/subvolume_file.1) 2023-11-24T08:20:07.416 INFO:teuthology.orchestra.run.smithi107.stdout:1700813976 2023-11-24T08:20:07.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone8559 2023-11-24T08:20:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.631+0000 7f0688d6c640 1 Processor -- start 2023-11-24T08:20:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.631+0000 7f0688d6c640 1 -- start start 2023-11-24T08:20:07.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0688d6c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f0684071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0688d6c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0684071620 0x7f06840ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0688d6c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684070860 con 0x7f068415aa80 2023-11-24T08:20:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0688d6c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0684159f40 con 0x7f0684071620 2023-11-24T08:20:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0688d6c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068415b020 con 0x7f068406c7c0 2023-11-24T08:20:07.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f06837fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f0684071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.633+0000 7f0683fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0684071620 0x7f06840ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0684071620 0x7f06840ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:57570/0 (socket says 172.21.15.107:57570) 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 -- 172.21.15.107:0/842326208 learned_addr learned my addr 172.21.15.107:0/842326208 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 -- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 msgr2=0x7f0684071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f0684071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 -- 172.21.15.107:0/842326208 >> v1:172.21.15.120:6789/0 conn(0x7f068415aa80 legacy=0x7f068406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f0683fff640 1 -- 172.21.15.107:0/842326208 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684070b40 con 0x7f0684071620 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.634+0000 7f06837fe640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f0684071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:20:07.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.636+0000 7f0683fff640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0684071620 0x7f06840ccc90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f067c009e50 tx=0x7f067c02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5d9f57279a42e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:07.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.636+0000 7f06827fc640 1 -- 172.21.15.107:0/842326208 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c02cb90 con 0x7f0684071620 2023-11-24T08:20:07.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.636+0000 7f06827fc640 1 -- 172.21.15.107:0/842326208 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c02ccf0 con 0x7f0684071620 2023-11-24T08:20:07.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.636+0000 7f06827fc640 1 -- 172.21.15.107:0/842326208 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c034610 con 0x7f0684071620 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 -- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0684071620 msgr2=0x7f06840ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0684071620 0x7f06840ccc90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f067c009e50 tx=0x7f067c02bff0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 -- 172.21.15.107:0/842326208 shutdown_connections 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0684071620 0x7f06840ccc90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 --2- 172.21.15.107:0/842326208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f0684071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 -- 172.21.15.107:0/842326208 >> 172.21.15.107:0/842326208 conn(0x7f06840cbb40 msgr2=0x7f06840cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:07.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 -- 172.21.15.107:0/842326208 shutdown_connections 2023-11-24T08:20:07.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.637+0000 7f0688d6c640 1 -- 172.21.15.107:0/842326208 wait complete. 2023-11-24T08:20:07.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.639+0000 7f0688d6c640 1 Processor -- start 2023-11-24T08:20:07.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.639+0000 7f0688d6c640 1 -- start start 2023-11-24T08:20:07.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f06841544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068415aa80 0x7f06840c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0684153140 con 0x7f0684071620 2023-11-24T08:20:07.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b0a00 con 0x7f068415aa80 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0688d6c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b2ec0 con 0x7f068406c7c0 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0682ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0682ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50380/0 (socket says 172.21.15.107:50380) 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.640+0000 7f0682ffd640 1 -- 172.21.15.107:0/2171393666 learned_addr learned my addr 172.21.15.107:0/2171393666 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f06837fe640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f06841544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0683fff640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068415aa80 0x7f06840c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 -- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 msgr2=0x7f06841544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f06841544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 -- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068415aa80 msgr2=0x7f06840c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068415aa80 0x7f06840c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 -- 172.21.15.107:0/2171393666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674009400 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0682ffd640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f067800c400 tx=0x7f067800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06780052d0 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0678005430 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c009b30 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.641+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0678003830 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.642+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06840b3480 con 0x7f0684071620 2023-11-24T08:20:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.643+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068404e840 con 0x7f0684071620 2023-11-24T08:20:07.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.644+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f0678003990 con 0x7f0684071620 2023-11-24T08:20:07.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.645+0000 7f066bfff640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 0x7f067006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:07.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.645+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0678087b00 con 0x7f0684071620 2023-11-24T08:20:07.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.645+0000 7f06837fe640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 0x7f067006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:07.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.646+0000 7f06837fe640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 0x7f067006ee90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0684153ab0 tx=0x7f0674008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:07.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.649+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0678054610 con 0x7f0684071620 2023-11-24T08:20:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.869+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f0684071100 con 0x7f067006c9d0 2023-11-24T08:20:07.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.880+0000 7f066bfff640 1 -- 172.21.15.107:0/2171393666 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f06740098d0 con 0x7f067006c9d0 2023-11-24T08:20:07.881 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b 2023-11-24T08:20:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.882+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 msgr2=0x7f067006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.882+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 0x7f067006ee90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0684153ab0 tx=0x7f0674008040 comp rx=0 tx=0).stop 2023-11-24T08:20:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.882+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 msgr2=0x7f06840ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.882+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f067800c400 tx=0x7f067800c8c0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 shutdown_connections 2023-11-24T08:20:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f067006c9d0 0x7f067006ee90 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068415aa80 0x7f06840c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0684071620 0x7f06840ca0e0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 --2- 172.21.15.107:0/2171393666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f068406c7c0 0x7f06841544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 >> 172.21.15.107:0/2171393666 conn(0x7f06840cbb40 msgr2=0x7f06840abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 shutdown_connections 2023-11-24T08:20:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:07.883+0000 7f0688d6c640 1 -- 172.21.15.107:0/2171393666 wait complete. 2023-11-24T08:20:07.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.901 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files volumes/_nogroup/subvol4786/08732b8d-ca3c-4370-a65d-45d6feb63411) 2023-11-24T08:20:07.949 INFO:teuthology.orchestra.run.smithi107.stdout:20 2023-11-24T08:20:07.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:07.950 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files volumes/_nogroup/subvol_clone8559/2e6118fa-4503-44cb-8981-d8a720d0dd7b) 2023-11-24T08:20:07.999 INFO:teuthology.orchestra.run.smithi107.stdout:20 2023-11-24T08:20:07.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol4786 subvol_snap8897 2023-11-24T08:20:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.208+0000 7f7428bef640 1 Processor -- start 2023-11-24T08:20:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.209+0000 7f7428bef640 1 -- start start 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7428bef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f74240b2ad0 0x7f742406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7428bef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74240b2f60 0x7f742406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7428bef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7424143c10 con 0x7f74241431f0 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7428bef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74240b4fd0 con 0x7f74240b2f60 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7428bef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74240705b0 con 0x7f74240b2ad0 2023-11-24T08:20:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f74237fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f74241431f0 0x7f742415e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38274/0 (socket says 172.21.15.107:38274) 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f7423fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74240b2f60 0x7f742406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.210+0000 7f74237fe640 1 -- 172.21.15.107:0/3894843727 learned_addr learned my addr 172.21.15.107:0/3894843727 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.211+0000 7f7423fff640 1 -- 172.21.15.107:0/3894843727 >> v2:172.21.15.120:3300/0 conn(0x7f74240b2ad0 msgr2=0x7f742406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.211+0000 7f7423fff640 1 --2- 172.21.15.107:0/3894843727 >> v2:172.21.15.120:3300/0 conn(0x7f74240b2ad0 0x7f742406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.211+0000 7f7423fff640 1 -- 172.21.15.107:0/3894843727 >> v1:172.21.15.107:6789/0 conn(0x7f74241431f0 legacy=0x7f742415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.211+0000 7f7423fff640 1 -- 172.21.15.107:0/3894843727 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424070890 con 0x7f74240b2f60 2023-11-24T08:20:08.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.212+0000 7f7423fff640 1 --2- 172.21.15.107:0/3894843727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2f60 0x7f742406dbe0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f74240cd7e0 tx=0x7f741c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accf78a39eb74836 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.212+0000 7f7422ffd640 1 -- 172.21.15.107:0/3894843727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c02cbe0 con 0x7f74240b2f60 2023-11-24T08:20:08.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.212+0000 7f7422ffd640 1 -- 172.21.15.107:0/3894843727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c02cd40 con 0x7f74240b2f60 2023-11-24T08:20:08.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.212+0000 7f7422ffd640 1 -- 172.21.15.107:0/3894843727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c034660 con 0x7f74240b2f60 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 -- 172.21.15.107:0/3894843727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2f60 msgr2=0x7f742406dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 --2- 172.21.15.107:0/3894843727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2f60 0x7f742406dbe0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f74240cd7e0 tx=0x7f741c02be30 comp rx=0 tx=0).stop 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 -- 172.21.15.107:0/3894843727 shutdown_connections 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 --2- 172.21.15.107:0/3894843727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2f60 0x7f742406dbe0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 --2- 172.21.15.107:0/3894843727 >> v2:172.21.15.120:3300/0 conn(0x7f74240b2ad0 0x7f742406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 -- 172.21.15.107:0/3894843727 >> 172.21.15.107:0/3894843727 conn(0x7f74240cb8a0 msgr2=0x7f74240cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 -- 172.21.15.107:0/3894843727 shutdown_connections 2023-11-24T08:20:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.213+0000 7f7428bef640 1 -- 172.21.15.107:0/3894843727 wait complete. 2023-11-24T08:20:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.215+0000 7f7428bef640 1 Processor -- start 2023-11-24T08:20:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.215+0000 7f7428bef640 1 -- start start 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74240b2f60 0x7f742415b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74241431f0 0x7f74240cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74240c8300 con 0x7f74240b2ad0 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742415bbe0 con 0x7f74241431f0 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7428bef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742415bec0 con 0x7f74240b2f60 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f74237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f74237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55914/0 (socket says 172.21.15.107:55914) 2023-11-24T08:20:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f74237fe640 1 -- 172.21.15.107:0/320949773 learned_addr learned my addr 172.21.15.107:0/320949773 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.216+0000 7f7423fff640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74241431f0 0x7f74240cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f741bfff640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74240b2f60 0x7f742415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 -- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74240b2f60 msgr2=0x7f742415b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74240b2f60 0x7f742415b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 -- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74241431f0 msgr2=0x7f74240cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74241431f0 0x7f74240cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 -- 172.21.15.107:0/320949773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c009a70 con 0x7f74240b2ad0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f74237fe640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f741400c480 tx=0x7f741400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414005ce0 con 0x7f74240b2ad0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414005e40 con 0x7f74240b2ad0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742415c1a0 con 0x7f74240b2ad0 2023-11-24T08:20:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.217+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74140072f0 con 0x7f74240b2ad0 2023-11-24T08:20:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.218+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7424135fa0 con 0x7f74240b2ad0 2023-11-24T08:20:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.219+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742404e840 con 0x7f74240b2ad0 2023-11-24T08:20:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.220+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f7414007450 con 0x7f74240b2ad0 2023-11-24T08:20:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.221+0000 7f7420ff9640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 0x7f740406eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.221+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7414087900 con 0x7f74240b2ad0 2023-11-24T08:20:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.221+0000 7f741bfff640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 0x7f740406eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.221+0000 7f741bfff640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 0x7f740406eec0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f74241429e0 tx=0x7f740c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.225+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7414054560 con 0x7f74240b2ad0 2023-11-24T08:20:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.448+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol4786", "snap_name": "subvol_snap8897", "target": ["mon-mgr", ""]}) v1 -- 0x7f7424071130 con 0x7f740406ca00 2023-11-24T08:20:08.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.500+0000 7f7420ff9640 1 -- 172.21.15.107:0/320949773 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f740c00adc0 con 0x7f740406ca00 2023-11-24T08:20:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 msgr2=0x7f740406eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 0x7f740406eec0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f74241429e0 tx=0x7f740c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 msgr2=0x7f7424135400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f741400c480 tx=0x7f741400c940 comp rx=0 tx=0).stop 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 shutdown_connections 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f740406ca00 0x7f740406eec0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74241431f0 0x7f74240cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74240b2f60 0x7f742415b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 --2- 172.21.15.107:0/320949773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240b2ad0 0x7f7424135400 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 >> 172.21.15.107:0/320949773 conn(0x7f74240cb8a0 msgr2=0x7f7424159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.503+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 shutdown_connections 2023-11-24T08:20:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.504+0000 7f7428bef640 1 -- 172.21.15.107:0/320949773 wait complete. 2023-11-24T08:20:08.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol4786 2023-11-24T08:20:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.734+0000 7fa3e52ff640 1 Processor -- start 2023-11-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.734+0000 7fa3e52ff640 1 -- start start 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.735+0000 7fa3e52ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3e00b0b40 0x7fa3e00b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.735+0000 7fa3e52ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e006ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.735+0000 7fa3e52ff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e015ed20 con 0x7fa3e015b260 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.735+0000 7fa3e52ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e015b640 con 0x7fa3e00b0b40 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3e52ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e00b14f0 con 0x7fa3e006c5b0 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3dffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3e015b260 0x7fa3e006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38278/0 (socket says 172.21.15.107:38278) 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3dffff640 1 -- 172.21.15.107:0/1905119595 learned_addr learned my addr 172.21.15.107:0/1905119595 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1905119595 >> v2:172.21.15.107:3300/0 conn(0x7fa3e00b0b40 0x7fa3e00b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3e4afe640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e006ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3df7fe640 1 -- 172.21.15.107:0/1905119595 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648162896 0 0) 0x7fa3e015ed20 con 0x7fa3e015b260 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3df7fe640 1 -- 172.21.15.107:0/1905119595 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc003520 con 0x7fa3e015b260 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1905119595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 msgr2=0x7fa3e006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e006ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1905119595 >> v1:172.21.15.107:6789/0 conn(0x7fa3e015b260 legacy=0x7fa3e006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.736+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1905119595 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e015f8a0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.737+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e00b0fb0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097d0 tx=0x7fa3c802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fdb5d71964408 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.738+0000 7fa3df7fe640 1 -- 172.21.15.107:0/1905119595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80316f0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.738+0000 7fa3df7fe640 1 -- 172.21.15.107:0/1905119595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8031850 con 0x7fa3e00b0b40 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.738+0000 7fa3df7fe640 1 -- 172.21.15.107:0/1905119595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8031be0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.738+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1905119595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 msgr2=0x7fa3e00b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.738+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e00b0fb0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097d0 tx=0x7fa3c802b790 comp rx=0 tx=0).stop 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1905119595 shutdown_connections 2023-11-24T08:20:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e006ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1905119595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e00b0fb0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1905119595 >> 172.21.15.107:0/1905119595 conn(0x7fa3e00cbb40 msgr2=0x7fa3e00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1905119595 shutdown_connections 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.739+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1905119595 wait complete. 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.740+0000 7fa3e52ff640 1 Processor -- start 2023-11-24T08:20:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.741+0000 7fa3e52ff640 1 -- start start 2023-11-24T08:20:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 0x7fa3e01f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e00b36e0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e00b3b00 con 0x7fa3e015b260 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e52ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e00b2a60 con 0x7fa3e006c5b0 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55942/0 (socket says 172.21.15.107:55942) 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1764311806 learned_addr learned my addr 172.21.15.107:0/1764311806 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3dffff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e4afe640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 0x7fa3e01f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 msgr2=0x7fa3e00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e00b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 msgr2=0x7fa3e01f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 0x7fa3e01f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3d7fff640 1 -- 172.21.15.107:0/1764311806 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8009400 con 0x7fa3e00b0b40 2023-11-24T08:20:08.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.742+0000 7fa3e4afe640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 0x7fa3e01f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:08.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3d7fff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097d0 tx=0x7fa3c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8034680 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c80347e0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e00b2d40 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8034b90 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.743+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e00b3300 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.745+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e00b3820 con 0x7fa3e00b0b40 2023-11-24T08:20:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.745+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7fa3c803b040 con 0x7fa3e00b0b40 2023-11-24T08:20:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.746+0000 7fa3dd7fa640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 0x7fa3c006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.746+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa3c80aa0b0 con 0x7fa3e00b0b40 2023-11-24T08:20:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.747+0000 7fa3dffff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 0x7fa3c006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.747+0000 7fa3dffff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 0x7fa3c006ee90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00011e0 tx=0x7fa3d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.751+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3e00b3820 con 0x7fa3e00b0b40 2023-11-24T08:20:08.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.967+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol4786", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e015ef70 con 0x7fa3c006c9d0 2023-11-24T08:20:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.992+0000 7fa3dd7fa640 1 -- 172.21.15.107:0/1764311806 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3d0001c40 con 0x7fa3c006c9d0 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 msgr2=0x7fa3c006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 0x7fa3c006ee90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00011e0 tx=0x7fa3d0009040 comp rx=0 tx=0).stop 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 msgr2=0x7fa3e015cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097d0 tx=0x7fa3c80046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 shutdown_connections 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3c006c9d0 0x7fa3c006ee90 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3e015b260 0x7fa3e01f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3e00b0b40 0x7fa3e015cd30 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 --2- 172.21.15.107:0/1764311806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3e006c5b0 0x7fa3e00b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 >> 172.21.15.107:0/1764311806 conn(0x7fa3e00cbb40 msgr2=0x7fa3e00af630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 shutdown_connections 2023-11-24T08:20:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:08.995+0000 7fa3e52ff640 1 -- 172.21.15.107:0/1764311806 wait complete. 2023-11-24T08:20:08.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:08.998+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:08.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:08.998+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:08.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:08.998+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:08.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:08.998+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:08.998 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:08.998+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone8559 2023-11-24T08:20:09.020 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:09.016+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.020 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:09.016+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.020 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:09.016+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.020 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:09.016+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.020 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:09.016+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:09.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.202+0000 7f52a8133640 1 Processor -- start 2023-11-24T08:20:09.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.202+0000 7f52a8133640 1 -- start start 2023-11-24T08:20:09.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.203+0000 7f52a8133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 0x7f52a0161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.203+0000 7f52a8133640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52a006e160 0x7f52a00ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.203+0000 7f52a8133640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a00cd530 con 0x7f52a00cd110 2023-11-24T08:20:09.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.203+0000 7f52a8133640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a0161a10 con 0x7f52a006e160 2023-11-24T08:20:09.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.203+0000 7f52a8133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0161cf0 con 0x7f52a0146eb0 2023-11-24T08:20:09.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.204+0000 7f52a5ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 0x7f52a0161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.204+0000 7f52a56ce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52a006e160 0x7f52a00ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.204+0000 7f52a5ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 0x7f52a0161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:33154/0 (socket says 172.21.15.107:33154) 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.204+0000 7f52a56ce640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52a006e160 0x7f52a00ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:50440/0 (socket says 172.21.15.107:50440) 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.204+0000 7f52a5ecf640 1 -- 172.21.15.107:0/376456016 learned_addr learned my addr 172.21.15.107:0/376456016 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.205+0000 7f52a56ce640 1 -- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 msgr2=0x7f52a0161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.205+0000 7f52a56ce640 1 --2- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 0x7f52a0161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.205+0000 7f52a56ce640 1 -- 172.21.15.107:0/376456016 >> v1:172.21.15.120:6789/0 conn(0x7f52a00cd110 legacy=0x7f52a006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.205+0000 7f52a56ce640 1 -- 172.21.15.107:0/376456016 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0161fd0 con 0x7f52a006e160 2023-11-24T08:20:09.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.206+0000 7f52a56ce640 1 --2- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00ccbd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f52900097a0 tx=0x7f529002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1651f5ae64a18fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.207+0000 7f52a4ecd640 1 -- 172.21.15.107:0/376456016 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290031830 con 0x7f52a006e160 2023-11-24T08:20:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.207+0000 7f52a4ecd640 1 -- 172.21.15.107:0/376456016 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5290031990 con 0x7f52a006e160 2023-11-24T08:20:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.207+0000 7f52a4ecd640 1 -- 172.21.15.107:0/376456016 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290031d20 con 0x7f52a006e160 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 -- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 msgr2=0x7f52a00ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 --2- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00ccbd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f52900097a0 tx=0x7f529002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 -- 172.21.15.107:0/376456016 shutdown_connections 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 --2- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00ccbd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 --2- 172.21.15.107:0/376456016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a0146eb0 0x7f52a0161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 -- 172.21.15.107:0/376456016 >> 172.21.15.107:0/376456016 conn(0x7f52a00b0c50 msgr2=0x7f52a00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 -- 172.21.15.107:0/376456016 shutdown_connections 2023-11-24T08:20:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.208+0000 7f52a8133640 1 -- 172.21.15.107:0/376456016 wait complete. 2023-11-24T08:20:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.210+0000 7f52a8133640 1 Processor -- start 2023-11-24T08:20:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.210+0000 7f52a8133640 1 -- start start 2023-11-24T08:20:09.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a00cd110 0x7f52a01601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00b2920 con 0x7f52a0146eb0 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00b1d10 con 0x7f52a006e160 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a8133640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00b1ff0 con 0x7f52a00cd110 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56008/0 (socket says 172.21.15.107:56008) 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 -- 172.21.15.107:0/3663360933 learned_addr learned my addr 172.21.15.107:0/3663360933 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a5ecf640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a56ce640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a00cd110 0x7f52a01601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 -- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a00cd110 msgr2=0x7f52a01601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a00cd110 0x7f52a01601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 -- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 msgr2=0x7f52a00714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:09.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.211+0000 7f52a66d0640 1 -- 172.21.15.107:0/3663360933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290009400 con 0x7f52a0146eb0 2023-11-24T08:20:09.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52a66d0640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f529800c7a0 tx=0x7f529800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003b70 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298003cf0 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003ea0 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a00b22d0 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.212+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a00b2db0 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.213+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a00b2a90 con 0x7f52a0146eb0 2023-11-24T08:20:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.214+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f529800aa90 con 0x7f52a0146eb0 2023-11-24T08:20:09.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.215+0000 7f52967fc640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 0x7f528406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:09.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.215+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5298088470 con 0x7f52a0146eb0 2023-11-24T08:20:09.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.216+0000 7f52a5ecf640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 0x7f528406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:09.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.216+0000 7f52a5ecf640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 0x7f528406ee90 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f529c009ac0 tx=0x7f529c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:09.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.220+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52980bfe90 con 0x7f52a0146eb0 2023-11-24T08:20:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.420+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone8559", "target": ["mon-mgr", ""]}) v1 -- 0x7f52a0161690 con 0x7f528406c9d0 2023-11-24T08:20:09.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.459+0000 7f52967fc640 1 -- 172.21.15.107:0/3663360933 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f529c009710 con 0x7f528406c9d0 2023-11-24T08:20:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 msgr2=0x7f528406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 0x7f528406ee90 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f529c009ac0 tx=0x7f529c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 msgr2=0x7f52a0160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f529800c7a0 tx=0x7f529800cc60 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 shutdown_connections 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f528406c9d0 0x7f528406ee90 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a0146eb0 0x7f52a0160720 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52a00cd110 0x7f52a01601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 --2- 172.21.15.107:0/3663360933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52a006e160 0x7f52a00714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 >> 172.21.15.107:0/3663360933 conn(0x7f52a00b0c50 msgr2=0x7f52a0145250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 shutdown_connections 2023-11-24T08:20:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:09.462+0000 7f52a8133640 1 -- 172.21.15.107:0/3663360933 wait complete. 2023-11-24T08:20:09.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:09.487 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:20:09.543 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:20:09.544 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:20:09.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:20:09.640 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:20:09.766 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:20:09.766 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:09.766 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:09.822 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:09.823 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:09.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:10.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.131+0000 7f9fb6941640 1 Processor -- start 2023-11-24T08:20:10.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.132+0000 7f9fb6941640 1 -- start start 2023-11-24T08:20:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.133+0000 7f9fb6941640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9fb00ce6c0 0x7f9fb00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.133+0000 7f9fb6941640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb00cece0 con 0x7f9fb015c7f0 2023-11-24T08:20:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.133+0000 7f9fb6941640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb006d040 con 0x7f9fb00cf450 2023-11-24T08:20:10.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.133+0000 7f9fb6941640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb00cfe40 con 0x7f9fb00ce6c0 2023-11-24T08:20:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9fb015c7f0 0x7f9fb006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38288/0 (socket says 172.21.15.107:38288) 2023-11-24T08:20:10.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faffff640 1 -- 172.21.15.107:0/370183274 learned_addr learned my addr 172.21.15.107:0/370183274 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:10.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9fb4ede640 1 --2- 172.21.15.107:0/370183274 >> v2:172.21.15.120:3300/0 conn(0x7f9fb00ce6c0 0x7f9fb00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863624524 0 0) 0x7f9fb00cece0 con 0x7f9fb015c7f0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f98003520 con 0x7f9fb015c7f0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3785064935 0 0) 0x7f9f98003520 con 0x7f9fb015c7f0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 >> v2:172.21.15.120:3300/0 conn(0x7f9fb00ce6c0 msgr2=0x7f9fb00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 --2- 172.21.15.107:0/370183274 >> v2:172.21.15.120:3300/0 conn(0x7f9fb00ce6c0 0x7f9fb00cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 >> v1:172.21.15.120:6789/0 conn(0x7f9fb00cf450 legacy=0x7f9fb00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb006d320 con 0x7f9fb015c7f0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9fa0001670 con 0x7f9fb015c7f0 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.134+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 >> v1:172.21.15.107:6789/0 conn(0x7f9fb015c7f0 legacy=0x7f9fb006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 0x7f9f98003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f980047c0 0x7f9f98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98005180 0x7f9f9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00cfe40 con 0x7f9f98003970 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9800c000 con 0x7f9f980047c0 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9800c1c0 con 0x7f9f98005180 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 0x7f9f98003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faf7fe640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f980047c0 0x7f9f98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 -- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98005180 msgr2=0x7f9f9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98005180 0x7f9f9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 -- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f980047c0 msgr2=0x7f9f98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f980047c0 0x7f9f98004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9faffff640 1 -- 172.21.15.107:0/370183274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f98003520 con 0x7f9f98003970 2023-11-24T08:20:10.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.135+0000 7f9fb4ede640 1 -- 172.21.15.107:0/370183274 reap_dead start 2023-11-24T08:20:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.136+0000 7f9faffff640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 0x7f9f98003d70 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c210 tx=0x7f9f980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b506eb42de7d25a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.136+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0005ad0 con 0x7f9f98003970 2023-11-24T08:20:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.136+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa0005c50 con 0x7f9f98003970 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.136+0000 7f9faeffd640 1 -- 172.21.15.107:0/370183274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa002c8a0 con 0x7f9f98003970 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 -- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 msgr2=0x7f9f98003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 0x7f9f98003d70 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c210 tx=0x7f9f980057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 -- 172.21.15.107:0/370183274 shutdown_connections 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 --2- 172.21.15.107:0/370183274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f98003970 0x7f9f98003d70 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 -- 172.21.15.107:0/370183274 >> 172.21.15.107:0/370183274 conn(0x7f9fb006a920 msgr2=0x7f9fb006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 -- 172.21.15.107:0/370183274 shutdown_connections 2023-11-24T08:20:10.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.137+0000 7f9fb6941640 1 -- 172.21.15.107:0/370183274 wait complete. 2023-11-24T08:20:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.139+0000 7f9fb6941640 1 Processor -- start 2023-11-24T08:20:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.139+0000 7f9fb6941640 1 -- start start 2023-11-24T08:20:10.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98003970 0x7f9fb00ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0152910 0x7f9fb015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0156550 con 0x7f9fb0155de0 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0156270 con 0x7f9fb0152910 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9fb6941640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0159800 con 0x7f9f98003970 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9faf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9faf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56026/0 (socket says 172.21.15.107:56026) 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.140+0000 7f9faf7fe640 1 -- 172.21.15.107:0/3634411747 learned_addr learned my addr 172.21.15.107:0/3634411747 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9fb4ede640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0152910 0x7f9fb015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faffff640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98003970 0x7f9fb00ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 -- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98003970 msgr2=0x7f9fb00ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98003970 0x7f9fb00ac6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 -- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0152910 msgr2=0x7f9fb015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0152910 0x7f9fb015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 -- 172.21.15.107:0/3634411747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0159ae0 con 0x7f9fb0155de0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9faf7fe640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400d3b0 tx=0x7f9fa400d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4012800 con 0x7f9fb0155de0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa4012980 con 0x7f9fb0155de0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4012d30 con 0x7f9fb0155de0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.141+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb0159dc0 con 0x7f9fb0155de0 2023-11-24T08:20:10.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.142+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb015a820 con 0x7f9fb0155de0 2023-11-24T08:20:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.143+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb0159fb0 con 0x7f9fb0155de0 2023-11-24T08:20:10.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.144+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7f9fa401b400 con 0x7f9fb0155de0 2023-11-24T08:20:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.145+0000 7f9facff9640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 0x7f9f9c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.145+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9fa4088760 con 0x7f9fb0155de0 2023-11-24T08:20:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.145+0000 7f9faffff640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 0x7f9f9c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.146+0000 7f9faffff640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 0x7f9f9c06ee90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0034000 tx=0x7f9fa0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.150+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fa4055310 con 0x7f9fb0155de0 2023-11-24T08:20:10.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.339+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fb0067470 con 0x7f9fb0155de0 2023-11-24T08:20:10.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.342+0000 7f9facff9640 1 -- 172.21.15.107:0/3634411747 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f9fa4016020 con 0x7f9fb0155de0 2023-11-24T08:20:10.345 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:10.345 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":177,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:19:41.062664+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":2}]},{"pgid":"25.17","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 msgr2=0x7f9f9c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 0x7f9f9c06ee90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0034000 tx=0x7f9fa0009030 comp rx=0 tx=0).stop 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 msgr2=0x7f9fb0158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400d3b0 tx=0x7f9fa400d870 comp rx=0 tx=0).stop 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 shutdown_connections 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f9c06c9d0 0x7f9f9c06ee90 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0152910 0x7f9fb015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fb0155de0 0x7f9fb0158db0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 --2- 172.21.15.107:0/3634411747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f98003970 0x7f9fb00ac6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 >> 172.21.15.107:0/3634411747 conn(0x7f9fb006a920 msgr2=0x7f9fb00aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 shutdown_connections 2023-11-24T08:20:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.345+0000 7f9fb6941640 1 -- 172.21.15.107:0/3634411747 wait complete. 2023-11-24T08:20:10.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:19:19.824499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:19:21.010791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:20:10.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.591+0000 7fa1770cf640 1 Processor -- start 2023-11-24T08:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.591+0000 7fa1770cf640 1 -- start start 2023-11-24T08:20:10.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.592+0000 7fa1770cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa170161060 0x7fa170161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.592+0000 7fa1770cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa17015e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa1770cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17006dcf0 con 0x7fa1700b3330 2023-11-24T08:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa1770cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006ec10 con 0x7fa170161060 2023-11-24T08:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa1770cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa170161980 con 0x7fa17006b6c0 2023-11-24T08:20:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa167fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa1700b3330 0x7fa17015da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38292/0 (socket says 172.21.15.107:38292) 2023-11-24T08:20:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa167fff640 1 -- 172.21.15.107:0/3006937834 learned_addr learned my addr 172.21.15.107:0/3006937834 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa17566c640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa17015e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa174e6b640 1 --2- 172.21.15.107:0/3006937834 >> v2:172.21.15.120:3300/0 conn(0x7fa170161060 0x7fa170161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216322846 0 0) 0x7fa17006dcf0 con 0x7fa1700b3330 2023-11-24T08:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.593+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa158003520 con 0x7fa1700b3330 2023-11-24T08:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704841169 0 0) 0x7fa158003520 con 0x7fa1700b3330 2023-11-24T08:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 msgr2=0x7fa17015e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa17015e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 >> v2:172.21.15.120:3300/0 conn(0x7fa170161060 msgr2=0x7fa170161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 --2- 172.21.15.107:0/3006937834 >> v2:172.21.15.120:3300/0 conn(0x7fa170161060 0x7fa170161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170161ec0 con 0x7fa1700b3330 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa1600017f0 con 0x7fa1700b3330 2023-11-24T08:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 >> v1:172.21.15.107:6789/0 conn(0x7fa1700b3330 legacy=0x7fa17015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa158003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa158005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17006ec10 con 0x7fa158003970 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170161980 con 0x7fa1580047c0 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15800b980 con 0x7fa17006b6c0 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.594+0000 7fa174e6b640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa158003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa167fff640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa17566c640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa158005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa174e6b640 1 -- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 msgr2=0x7fa158005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa174e6b640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa17006b6c0 0x7fa158005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa174e6b640 1 -- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1580047c0 msgr2=0x7fa158004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa174e6b640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1580047c0 0x7fa158004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.595+0000 7fa174e6b640 1 -- 172.21.15.107:0/3006937834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158003520 con 0x7fa158003970 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.596+0000 7fa174e6b640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa158003d90 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009af0 tx=0x7fa1580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa8645920078dd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.596+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02c710 con 0x7fa158003970 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c02c870 con 0x7fa158003970 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1677fe640 1 -- 172.21.15.107:0/3006937834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02cc00 con 0x7fa158003970 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 -- 172.21.15.107:0/3006937834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 msgr2=0x7fa158003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 --2- 172.21.15.107:0/3006937834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa158003d90 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009af0 tx=0x7fa1580057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa17566c640 1 -- 172.21.15.107:0/3006937834 reap_dead start 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 -- 172.21.15.107:0/3006937834 shutdown_connections 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 -- 172.21.15.107:0/3006937834 >> 172.21.15.107:0/3006937834 conn(0x7fa1700cf900 msgr2=0x7fa1700cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 -- 172.21.15.107:0/3006937834 shutdown_connections 2023-11-24T08:20:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.597+0000 7fa1770cf640 1 -- 172.21.15.107:0/3006937834 wait complete. 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.599+0000 7fa1770cf640 1 Processor -- start 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.600+0000 7fa1770cf640 1 -- start start 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.600+0000 7fa1770cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.600+0000 7fa1770cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa170145e60 0x7fa1700b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.600+0000 7fa1770cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1701462e0 0x7fa1700b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa1770cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701459c0 con 0x7fa158003970 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa1770cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700cbba0 con 0x7fa170145e60 2023-11-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa1770cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700cbe80 con 0x7fa1701462e0 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56046/0 (socket says 172.21.15.107:56046) 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 -- 172.21.15.107:0/2436425721 learned_addr learned my addr 172.21.15.107:0/2436425721 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 -- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1701462e0 msgr2=0x7fa1700b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1701462e0 0x7fa1700b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 -- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa170145e60 msgr2=0x7fa1700b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa170145e60 0x7fa1700b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 -- 172.21.15.107:0/2436425721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1700cc160 con 0x7fa158003970 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.601+0000 7fa174e6b640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fa16c037040 tx=0x7fa16c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.602+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c003810 con 0x7fa158003970 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.602+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c02de90 con 0x7fa158003970 2023-11-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.602+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c003b70 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.602+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1700cc440 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.602+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1700ccc40 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.603+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1700cbfc0 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.604+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 38) v1 ==== 91730+0+0 (secure 0 0 0) 0x7fa16c003cd0 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.605+0000 7fa1657fa640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 0x7fa15c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.606+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa16c0aa3e0 con 0x7fa158003970 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.606+0000 7fa167fff640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 0x7fa15c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.606+0000 7fa167fff640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 0x7fa15c06ee90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa160000ef0 tx=0x7fa160009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:10.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.610+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1700cbfc0 con 0x7fa158003970 2023-11-24T08:20:10.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.807+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1700cbfc0 con 0x7fa158003970 2023-11-24T08:20:10.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.807+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fa16c0772c0 con 0x7fa158003970 2023-11-24T08:20:10.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.810+0000 7fa1657fa640 1 -- 172.21.15.107:0/2436425721 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1700cbfc0 con 0x7fa158003970 2023-11-24T08:20:10.812 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 msgr2=0x7fa15c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 0x7fa15c06ee90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa160000ef0 tx=0x7fa160009040 comp rx=0 tx=0).stop 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 msgr2=0x7fa170145480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fa16c037040 tx=0x7fa16c004730 comp rx=0 tx=0).stop 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 shutdown_connections 2023-11-24T08:20:10.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa15c06c9d0 0x7fa15c06ee90 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1701462e0 0x7fa1700b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa170145e60 0x7fa1700b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 --2- 172.21.15.107:0/2436425721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa158003970 0x7fa170145480 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 >> 172.21.15.107:0/2436425721 conn(0x7fa1700cf900 msgr2=0x7fa1700b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 shutdown_connections 2023-11-24T08:20:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:10.813+0000 7fa1770cf640 1 -- 172.21.15.107:0/2436425721 wait complete. 2023-11-24T08:20:10.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:20:11.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.039+0000 7f08ae12c640 1 Processor -- start 2023-11-24T08:20:11.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.039+0000 7f08ae12c640 1 -- start start 2023-11-24T08:20:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08ae12c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 0x7f08a80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08ae12c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f08a80cd110 0x7f08a8070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08ae12c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80cd510 con 0x7f08a8146eb0 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08ae12c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a8161a10 con 0x7f08a80cd110 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08ae12c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8161cf0 con 0x7f08a806e160 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08a77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f08a8146eb0 0x7f08a8161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38300/0 (socket says 172.21.15.107:38300) 2023-11-24T08:20:11.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08a77fe640 1 -- 172.21.15.107:0/3388425609 learned_addr learned my addr 172.21.15.107:0/3388425609 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f089effd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 0x7f08a80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08a7fff640 1 --2- 172.21.15.107:0/3388425609 >> v2:172.21.15.120:3300/0 conn(0x7f08a80cd110 0x7f08a8070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580222732 0 0) 0x7f08a80cd510 con 0x7f08a8146eb0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.041+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0894003520 con 0x7f08a8146eb0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032103641 0 0) 0x7f0894003520 con 0x7f08a8146eb0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 msgr2=0x7f08a80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 0x7f08a80ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 >> v2:172.21.15.120:3300/0 conn(0x7f08a80cd110 msgr2=0x7f08a8070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 --2- 172.21.15.107:0/3388425609 >> v2:172.21.15.120:3300/0 conn(0x7f08a80cd110 0x7f08a8070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8161fd0 con 0x7f08a8146eb0 2023-11-24T08:20:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f08980019f0 con 0x7f08a8146eb0 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 >> v1:172.21.15.107:6789/0 conn(0x7f08a8146eb0 legacy=0x7f08a8161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0894003970 0x7f0894003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f0894004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 0x7f0894005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8161a10 con 0x7f08940047c0 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8161cf0 con 0x7f0894003970 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089400b980 con 0x7f08a806e160 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f08a7fff640 1 --2- 172.21.15.107:0/3388425609 >> v2:172.21.15.120:3300/0 conn(0x7f08a80cd110 0x7f08a8070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.042+0000 7f089effd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f0894004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f089effd640 1 -- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 msgr2=0x7f0894005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f08a77fe640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0894003970 0x7f0894003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f089effd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a806e160 0x7f0894005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f089effd640 1 -- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0894003970 msgr2=0x7f0894003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f089effd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0894003970 0x7f0894003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.043+0000 7f089effd640 1 -- 172.21.15.107:0/3388425609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894003520 con 0x7f08940047c0 2023-11-24T08:20:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.044+0000 7f089effd640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f0894004c40 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ac0 tx=0x7f08940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a393b957e46b25 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.044+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089002c710 con 0x7f08940047c0 2023-11-24T08:20:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.044+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089002c870 con 0x7f08940047c0 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.044+0000 7f08a6ffd640 1 -- 172.21.15.107:0/3388425609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089002cc00 con 0x7f08940047c0 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 -- 172.21.15.107:0/3388425609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 msgr2=0x7f0894004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 --2- 172.21.15.107:0/3388425609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f0894004c40 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ac0 tx=0x7f08940057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08a7fff640 1 -- 172.21.15.107:0/3388425609 reap_dead start 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 -- 172.21.15.107:0/3388425609 shutdown_connections 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 -- 172.21.15.107:0/3388425609 >> 172.21.15.107:0/3388425609 conn(0x7f08a80b0c50 msgr2=0x7f08a80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 -- 172.21.15.107:0/3388425609 shutdown_connections 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.045+0000 7f08ae12c640 1 -- 172.21.15.107:0/3388425609 wait complete. 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.047+0000 7f08ae12c640 1 Processor -- start 2023-11-24T08:20:11.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.047+0000 7f08ae12c640 1 -- start start 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a815db00 0x7f08a8139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a815f0f0 0x7f08a815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8099e40 con 0x7f08940047c0 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8139e80 con 0x7f08a815f0f0 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08ae12c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a815ec60 con 0x7f08a815db00 2023-11-24T08:20:11.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56062/0 (socket says 172.21.15.107:56062) 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 -- 172.21.15.107:0/1756733117 learned_addr learned my addr 172.21.15.107:0/1756733117 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 -- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a815db00 msgr2=0x7f08a8139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a815db00 0x7f08a8139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 -- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a815f0f0 msgr2=0x7f08a815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a815f0f0 0x7f08a815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.048+0000 7f08a77fe640 1 -- 172.21.15.107:0/1756733117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a815e050 con 0x7f08940047c0 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08a77fe640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f089800b0d0 tx=0x7f089800b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898004d60 con 0x7f08940047c0 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08980070d0 con 0x7f08940047c0 2023-11-24T08:20:11.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898007440 con 0x7f08940047c0 2023-11-24T08:20:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a815e330 con 0x7f08940047c0 2023-11-24T08:20:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.049+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a815e8f0 con 0x7f08940047c0 2023-11-24T08:20:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.050+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a80adf40 con 0x7f08940047c0 2023-11-24T08:20:11.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.051+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f08980075a0 con 0x7f08940047c0 2023-11-24T08:20:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.052+0000 7f08a4ff9640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 0x7f088806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.052+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f08980886e0 con 0x7f08940047c0 2023-11-24T08:20:11.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.052+0000 7f089effd640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 0x7f088806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.053+0000 7f089effd640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 0x7f088806efe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0890036000 tx=0x7f089002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.056+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08980551f0 con 0x7f08940047c0 2023-11-24T08:20:11.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.247+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08a8061a50 con 0x7f08940047c0 2023-11-24T08:20:11.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.248+0000 7f08a4ff9640 1 -- 172.21.15.107:0/1756733117 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v177) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0898019020 con 0x7f08940047c0 2023-11-24T08:20:11.250 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:20:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 msgr2=0x7f088806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 0x7f088806efe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0890036000 tx=0x7f089002e000 comp rx=0 tx=0).stop 2023-11-24T08:20:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 msgr2=0x7f08a80b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f089800b0d0 tx=0x7f089800b590 comp rx=0 tx=0).stop 2023-11-24T08:20:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 shutdown_connections 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f088806cb20 0x7f088806efe0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a815f0f0 0x7f08a815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08a815db00 0x7f08a8139940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 --2- 172.21.15.107:0/1756733117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08940047c0 0x7f08a80b4880 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 >> 172.21.15.107:0/1756733117 conn(0x7f08a80b0c50 msgr2=0x7f08a80cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 shutdown_connections 2023-11-24T08:20:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.250+0000 7f08ae12c640 1 -- 172.21.15.107:0/1756733117 wait complete. 2023-11-24T08:20:11.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:20:11.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.484+0000 7f162759e640 1 Processor -- start 2023-11-24T08:20:11.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.484+0000 7f162759e640 1 -- start start 2023-11-24T08:20:11.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.485+0000 7f162759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f16280b0b40 0x7f16280b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.485+0000 7f162759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162806d430 con 0x7f162815ded0 2023-11-24T08:20:11.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.485+0000 7f162759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16280b14e0 con 0x7f162815a1c0 2023-11-24T08:20:11.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.485+0000 7f162759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162815ea70 con 0x7f16280b0b40 2023-11-24T08:20:11.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f1626d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f162815ded0 0x7f162815e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38316/0 (socket says 172.21.15.107:38316) 2023-11-24T08:20:11.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f1626d9d640 1 -- 172.21.15.107:0/2685795012 learned_addr learned my addr 172.21.15.107:0/2685795012 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:11.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f1625d9b640 1 --2- 172.21.15.107:0/2685795012 >> v2:172.21.15.120:3300/0 conn(0x7f16280b0b40 0x7f16280b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215209551 0 0) 0x7f162806d430 con 0x7f162815ded0 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1610003520 con 0x7f162815ded0 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681851416 0 0) 0x7f1610003520 con 0x7f162815ded0 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 >> v2:172.21.15.120:3300/0 conn(0x7f16280b0b40 msgr2=0x7f16280b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 --2- 172.21.15.107:0/2685795012 >> v2:172.21.15.120:3300/0 conn(0x7f16280b0b40 0x7f16280b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 >> v1:172.21.15.120:6789/0 conn(0x7f162815a1c0 legacy=0x7f162806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162815ec50 con 0x7f162815ded0 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1620001670 con 0x7f162815ded0 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.486+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 >> v1:172.21.15.107:6789/0 conn(0x7f162815ded0 legacy=0x7f162815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1610003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16100047c0 0x7f1610004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1610005180 0x7f161000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162815ea70 con 0x7f1610003970 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161000c0d0 con 0x7f16100047c0 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161000e4a0 con 0x7f1610005180 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1610003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f1625d9b640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16100047c0 0x7f1610004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f1626d9d640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1610005180 0x7f161000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 -- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1610005180 msgr2=0x7f161000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1610005180 0x7f161000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 -- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16100047c0 msgr2=0x7f1610004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16100047c0 0x7f1610004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f162659c640 1 -- 172.21.15.107:0/2685795012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610003520 con 0x7f1610003970 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f1626d9d640 1 -- 172.21.15.107:0/2685795012 reap_dead start 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.487+0000 7f1625d9b640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16100047c0 0x7f1610004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.488+0000 7f162659c640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1610003d90 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f1618001e50 tx=0x7f16100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c97c536eddfcec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618003b70 con 0x7f1610003970 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618003cd0 con 0x7f1610003970 2023-11-24T08:20:11.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162559a640 1 -- 172.21.15.107:0/2685795012 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618005310 con 0x7f1610003970 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 -- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 msgr2=0x7f1610003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1610003d90 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f1618001e50 tx=0x7f16100057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 -- 172.21.15.107:0/2685795012 shutdown_connections 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 --2- 172.21.15.107:0/2685795012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1610003d90 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 -- 172.21.15.107:0/2685795012 >> 172.21.15.107:0/2685795012 conn(0x7f16280cbb40 msgr2=0x7f16280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 -- 172.21.15.107:0/2685795012 shutdown_connections 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.489+0000 7f162759e640 1 -- 172.21.15.107:0/2685795012 wait complete. 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.491+0000 7f162759e640 1 Processor -- start 2023-11-24T08:20:11.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.491+0000 7f162759e640 1 -- start start 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280c83a0 0x7f16280c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16280c9030 0x7f1628070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16280b3b30 con 0x7f1610003970 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16280b2f20 con 0x7f16280c83a0 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16280b3200 con 0x7f16280c9030 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.492+0000 7f162659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56080/0 (socket says 172.21.15.107:56080) 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 -- 172.21.15.107:0/124539672 learned_addr learned my addr 172.21.15.107:0/124539672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 -- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16280c9030 msgr2=0x7f1628070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16280c9030 0x7f1628070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 -- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280c83a0 msgr2=0x7f16280c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280c83a0 0x7f16280c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 -- 172.21.15.107:0/124539672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16280b34e0 con 0x7f1610003970 2023-11-24T08:20:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f162659c640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f161802bf70 tx=0x7f16180014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.493+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161803e070 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.494+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618006b50 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.494+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618007020 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.494+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16280b37c0 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.494+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1628071530 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.495+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1628071780 con 0x7f1610003970 2023-11-24T08:20:11.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.496+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f1618007180 con 0x7f1610003970 2023-11-24T08:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.497+0000 7f1616ffd640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 0x7f160806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.497+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f16180a9c60 con 0x7f1610003970 2023-11-24T08:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.497+0000 7f1625d9b640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 0x7f160806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.498+0000 7f1625d9b640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 0x7f160806efb0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f16280c5740 tx=0x7f161c009260 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.501+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16180b0e60 con 0x7f1610003970 2023-11-24T08:20:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.701+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16280b3950 con 0x7f1610003970 2023-11-24T08:20:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.702+0000 7f1616ffd640 1 -- 172.21.15.107:0/124539672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v177) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1618076810 con 0x7f1610003970 2023-11-24T08:20:11.704 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.705+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 msgr2=0x7f160806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.705+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 0x7f160806efb0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f16280c5740 tx=0x7f161c009260 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.705+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 msgr2=0x7f1628159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.705+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f161802bf70 tx=0x7f16180014a0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 shutdown_connections 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f160806caf0 0x7f160806efb0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16280c9030 0x7f1628070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16280c83a0 0x7f16280c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 --2- 172.21.15.107:0/124539672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1610003970 0x7f1628159850 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 >> 172.21.15.107:0/124539672 conn(0x7f16280cbb40 msgr2=0x7f162815e360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:11.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 shutdown_connections 2023-11-24T08:20:11.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.706+0000 7f162759e640 1 -- 172.21.15.107:0/124539672 wait complete. 2023-11-24T08:20:11.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:20:11.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.929+0000 7f25105a9640 1 Processor -- start 2023-11-24T08:20:11.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.930+0000 7f25105a9640 1 -- start start 2023-11-24T08:20:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.931+0000 7f25105a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.931+0000 7f25105a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25080c59b0 0x7f250806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.931+0000 7f25105a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080cc3a0 con 0x7f25080cbf20 2023-11-24T08:20:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.931+0000 7f25105a9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250815b820 con 0x7f25080c59b0 2023-11-24T08:20:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.931+0000 7f25105a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250815eea0 con 0x7f250815b440 2023-11-24T08:20:11.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.932+0000 7f250eb46640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25080c59b0 0x7f250806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.932+0000 7f250e345640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.932+0000 7f250e345640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:33242/0 (socket says 172.21.15.107:33242) 2023-11-24T08:20:11.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.932+0000 7f250eb46640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25080c59b0 0x7f250806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:50572/0 (socket says 172.21.15.107:50572) 2023-11-24T08:20:11.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.932+0000 7f250e345640 1 -- 172.21.15.107:0/3907327176 learned_addr learned my addr 172.21.15.107:0/3907327176 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.933+0000 7f250eb46640 1 -- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 msgr2=0x7f25080cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.933+0000 7f250eb46640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.933+0000 7f250eb46640 1 -- 172.21.15.107:0/3907327176 >> v1:172.21.15.120:6789/0 conn(0x7f25080cbf20 legacy=0x7f25080c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.933+0000 7f250eb46640 1 -- 172.21.15.107:0/3907327176 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080abf30 con 0x7f25080c59b0 2023-11-24T08:20:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.933+0000 7f250e345640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:20:11.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.934+0000 7f250eb46640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f250806c5f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f250002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ff75e2812c57bb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.935+0000 7f250d343640 1 -- 172.21.15.107:0/3907327176 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250002cbc0 con 0x7f25080c59b0 2023-11-24T08:20:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.935+0000 7f250d343640 1 -- 172.21.15.107:0/3907327176 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250002cd20 con 0x7f25080c59b0 2023-11-24T08:20:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.935+0000 7f250d343640 1 -- 172.21.15.107:0/3907327176 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500034660 con 0x7f25080c59b0 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 -- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 msgr2=0x7f250806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f250806c5f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f250002be10 comp rx=0 tx=0).stop 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 -- 172.21.15.107:0/3907327176 shutdown_connections 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f250806c5f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 --2- 172.21.15.107:0/3907327176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250815b440 0x7f25080cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 -- 172.21.15.107:0/3907327176 >> 172.21.15.107:0/3907327176 conn(0x7f25080b0c50 msgr2=0x7f25080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 -- 172.21.15.107:0/3907327176 shutdown_connections 2023-11-24T08:20:11.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.936+0000 7f25105a9640 1 -- 172.21.15.107:0/3907327176 wait complete. 2023-11-24T08:20:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.938+0000 7f25105a9640 1 Processor -- start 2023-11-24T08:20:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.938+0000 7f25105a9640 1 -- start start 2023-11-24T08:20:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f2508154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25080cbf20 0x7f25081f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25081f3a90 con 0x7f250815b440 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25081f3d70 con 0x7f25080c59b0 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.939+0000 7f25105a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25081f4050 con 0x7f25080cbf20 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56088/0 (socket says 172.21.15.107:56088) 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 -- 172.21.15.107:0/974210354 learned_addr learned my addr 172.21.15.107:0/974210354 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250db44640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25080cbf20 0x7f25081f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250e345640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f2508154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 -- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25080cbf20 msgr2=0x7f25081f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25080cbf20 0x7f25081f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 -- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 msgr2=0x7f2508154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f2508154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.940+0000 7f250eb46640 1 -- 172.21.15.107:0/974210354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25040094d0 con 0x7f250815b440 2023-11-24T08:20:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f250eb46640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f2500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500040650 con 0x7f250815b440 2023-11-24T08:20:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250002ce90 con 0x7f250815b440 2023-11-24T08:20:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2500009a70 con 0x7f250815b440 2023-11-24T08:20:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000409f0 con 0x7f250815b440 2023-11-24T08:20:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.941+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25080ac4d0 con 0x7f250815b440 2023-11-24T08:20:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.942+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250804e8b0 con 0x7f250815b440 2023-11-24T08:20:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.943+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f2500040b50 con 0x7f250815b440 2023-11-24T08:20:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.944+0000 7f24faffd640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 0x7f24ec06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.944+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f25000ab170 con 0x7f250815b440 2023-11-24T08:20:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.944+0000 7f250e345640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 0x7f24ec06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.945+0000 7f250e345640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 0x7f24ec06efb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2508070ed0 tx=0x7f2504008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:11.948+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2500077d00 con 0x7f250815b440 2023-11-24T08:20:12.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.151+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f250806dc80 con 0x7f250815b440 2023-11-24T08:20:12.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.152+0000 7f24faffd640 1 -- 172.21.15.107:0/974210354 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v178) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2500009450 con 0x7f250815b440 2023-11-24T08:20:12.154 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:20:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 msgr2=0x7f24ec06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 0x7f24ec06efb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2508070ed0 tx=0x7f2504008040 comp rx=0 tx=0).stop 2023-11-24T08:20:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 msgr2=0x7f25080ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f2500004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 shutdown_connections 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24ec06caf0 0x7f24ec06efb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250815b440 0x7f25080ac9f0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25080cbf20 0x7f25081f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 --2- 172.21.15.107:0/974210354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25080c59b0 0x7f2508154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 >> 172.21.15.107:0/974210354 conn(0x7f25080b0c50 msgr2=0x7f25080c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 shutdown_connections 2023-11-24T08:20:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.155+0000 7f25105a9640 1 -- 172.21.15.107:0/974210354 wait complete. 2023-11-24T08:20:12.178 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:20:12.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:12.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:20:12.293 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:20:12.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:12.293 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:20:12.350 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:12.351 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:12.392 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:12.392 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:12.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.686+0000 7f214c802640 1 Processor -- start 2023-11-24T08:20:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.687+0000 7f214c802640 1 -- start start 2023-11-24T08:20:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214c802640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2148161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214c802640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480ce0b0 con 0x7f2148157c70 2023-11-24T08:20:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214c802640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2148161960 con 0x7f214815dc80 2023-11-24T08:20:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214c802640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214815e8d0 con 0x7f21480cfb60 2023-11-24T08:20:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2148157c70 0x7f21480cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38324/0 (socket says 172.21.15.107:38324) 2023-11-24T08:20:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.688+0000 7f214659c640 1 -- 172.21.15.107:0/4072386400 learned_addr learned my addr 172.21.15.107:0/4072386400 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f213ffff640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2148161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391589664 0 0) 0x7f21480ce0b0 con 0x7f2148157c70 2023-11-24T08:20:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2134003520 con 0x7f2148157c70 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360583184 0 0) 0x7f2134003520 con 0x7f2148157c70 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 msgr2=0x7f2148161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2148161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 >> v1:172.21.15.120:6789/0 conn(0x7f214815dc80 legacy=0x7f21480b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214815ebb0 con 0x7f2148157c70 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2138001670 con 0x7f2148157c70 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.689+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 >> v1:172.21.15.107:6789/0 conn(0x7f2148157c70 legacy=0x7f21480cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21340039a0 0x7f2134003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21340047c0 0x7f2134004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2134005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214815e8d0 con 0x7f21340039a0 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213400bb10 con 0x7f21340047c0 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213400bcd0 con 0x7f21480cfb60 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f214659c640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21340039a0 0x7f2134003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f2146d9d640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2134005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.690+0000 7f213ffff640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21340047c0 0x7f2134004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f214659c640 1 -- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 msgr2=0x7f2134005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f214659c640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21480cfb60 0x7f2134005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f214659c640 1 -- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21340047c0 msgr2=0x7f2134004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f214659c640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21340047c0 0x7f2134004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f214659c640 1 -- 172.21.15.107:0/4072386400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134003520 con 0x7f21340039a0 2023-11-24T08:20:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.691+0000 7f213ffff640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21340047c0 0x7f2134004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.692+0000 7f214659c640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21340039a0 0x7f2134003dc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f213800c0e0 tx=0x7f21340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c2fb4768f5a0d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.692+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138005ad0 con 0x7f21340039a0 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.692+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138005c30 con 0x7f21340039a0 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.692+0000 7f2145d9b640 1 -- 172.21.15.107:0/4072386400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213802c810 con 0x7f21340039a0 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 -- 172.21.15.107:0/4072386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21340039a0 msgr2=0x7f2134003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 --2- 172.21.15.107:0/4072386400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21340039a0 0x7f2134003dc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f213800c0e0 tx=0x7f21340057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f2146d9d640 1 -- 172.21.15.107:0/4072386400 reap_dead start 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 -- 172.21.15.107:0/4072386400 shutdown_connections 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 -- 172.21.15.107:0/4072386400 >> 172.21.15.107:0/4072386400 conn(0x7f214806a920 msgr2=0x7f214806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 -- 172.21.15.107:0/4072386400 shutdown_connections 2023-11-24T08:20:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.693+0000 7f214c802640 1 -- 172.21.15.107:0/4072386400 wait complete. 2023-11-24T08:20:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.695+0000 7f214c802640 1 Processor -- start 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.695+0000 7f214c802640 1 -- start start 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21340039a0 0x7f21480b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21481554e0 0x7f214813e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2148154c80 con 0x7f21481550a0 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214814d340 con 0x7f21481554e0 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f214c802640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21481545c0 con 0x7f21340039a0 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56120/0 (socket says 172.21.15.107:56120) 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 -- 172.21.15.107:0/1685745267 learned_addr learned my addr 172.21.15.107:0/1685745267 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 -- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21340039a0 msgr2=0x7f21480b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21340039a0 0x7f21480b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 -- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21481554e0 msgr2=0x7f214813e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f2146d9d640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21481554e0 0x7f214813e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.696+0000 7f213ffff640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21481554e0 0x7f214813e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f213ffff640 1 -- 172.21.15.107:0/1685745267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21481548a0 con 0x7f21481550a0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f213ffff640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f21300115b0 tx=0x7f2130011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130015870 con 0x7f21481550a0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300159d0 con 0x7f21481550a0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214814cac0 con 0x7f21481550a0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130015d60 con 0x7f21481550a0 2023-11-24T08:20:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.697+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214814d620 con 0x7f21481550a0 2023-11-24T08:20:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.698+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148154a30 con 0x7f21481550a0 2023-11-24T08:20:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.699+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f213000f480 con 0x7f21481550a0 2023-11-24T08:20:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.701+0000 7f213effd640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 0x7f212806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.701+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f2130088df0 con 0x7f21481550a0 2023-11-24T08:20:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.701+0000 7f214659c640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 0x7f212806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.702+0000 7f214659c640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 0x7f212806efe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f213800c0e0 tx=0x7f2138007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:12.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.705+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2130055870 con 0x7f21481550a0 2023-11-24T08:20:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.903+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2148154dc0 con 0x7f21481550a0 2023-11-24T08:20:12.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.907+0000 7f213effd640 1 -- 172.21.15.107:0/1685745267 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f2148154dc0 con 0x7f21481550a0 2023-11-24T08:20:12.909 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:12.909 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":178,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:11.814699+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":178,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":2}]},{"pgid":"25.17","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":25,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.909+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 msgr2=0x7f212806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.909+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 0x7f212806efe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f213800c0e0 tx=0x7f2138007000 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.909+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 msgr2=0x7f214813e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.909+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f21300115b0 tx=0x7f2130011a70 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 shutdown_connections 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f212806cb20 0x7f212806efe0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21481554e0 0x7f214813e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21481550a0 0x7f214813e230 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 --2- 172.21.15.107:0/1685745267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21340039a0 0x7f21480b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 >> 172.21.15.107:0/1685745267 conn(0x7f214806a920 msgr2=0x7f214806d190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:12.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 shutdown_connections 2023-11-24T08:20:12.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:12.910+0000 7f214c802640 1 -- 172.21.15.107:0/1685745267 wait complete. 2023-11-24T08:20:12.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:19:19.824499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:19:21.010791+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 178, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:20:12.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:20:13.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.139+0000 7fd774648640 1 Processor -- start 2023-11-24T08:20:13.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.140+0000 7fd774648640 1 -- start start 2023-11-24T08:20:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.141+0000 7fd774648640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd76c15b440 0x7fd76c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.141+0000 7fd774648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c0cbf20 0x7fd76c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.141+0000 7fd774648640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c15eea0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.141+0000 7fd774648640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c0cc3d0 con 0x7fd76c15b440 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.141+0000 7fd774648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15b820 con 0x7fd76c0cbf20 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd76c15b440 0x7fd76c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd76c15b440 0x7fd76c0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56124/0 (socket says 172.21.15.107:56124) 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 -- 172.21.15.107:0/2877514024 learned_addr learned my addr 172.21.15.107:0/2877514024 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd772be5640 1 --1- 172.21.15.107:0/2877514024 >> v1:172.21.15.107:6789/0 conn(0x7fd76c0c57e0 0x7fd76c0c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38340/0 (socket says 172.21.15.107:38340) 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 -- 172.21.15.107:0/2877514024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c0cbf20 msgr2=0x7fd76c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 --2- 172.21.15.107:0/2877514024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c0cbf20 0x7fd76c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 -- 172.21.15.107:0/2877514024 >> v1:172.21.15.107:6789/0 conn(0x7fd76c0c57e0 legacy=0x7fd76c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.142+0000 7fd7723e4640 1 -- 172.21.15.107:0/2877514024 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c06d600 con 0x7fd76c15b440 2023-11-24T08:20:13.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.143+0000 7fd7723e4640 1 --2- 172.21.15.107:0/2877514024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c15b440 0x7fd76c0cb9e0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd76802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c192edd824319f7c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.143+0000 7fd7713e2640 1 -- 172.21.15.107:0/2877514024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768031640 con 0x7fd76c15b440 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.143+0000 7fd7713e2640 1 -- 172.21.15.107:0/2877514024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7680317a0 con 0x7fd76c15b440 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.143+0000 7fd7713e2640 1 -- 172.21.15.107:0/2877514024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768031b30 con 0x7fd76c15b440 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 -- 172.21.15.107:0/2877514024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c15b440 msgr2=0x7fd76c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 --2- 172.21.15.107:0/2877514024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c15b440 0x7fd76c0cb9e0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd76802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 -- 172.21.15.107:0/2877514024 shutdown_connections 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 --2- 172.21.15.107:0/2877514024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c0cbf20 0x7fd76c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 --2- 172.21.15.107:0/2877514024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c15b440 0x7fd76c0cb9e0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 -- 172.21.15.107:0/2877514024 >> 172.21.15.107:0/2877514024 conn(0x7fd76c0b0c50 msgr2=0x7fd76c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 -- 172.21.15.107:0/2877514024 shutdown_connections 2023-11-24T08:20:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.144+0000 7fd774648640 1 -- 172.21.15.107:0/2877514024 wait complete. 2023-11-24T08:20:13.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.146+0000 7fd774648640 1 Processor -- start 2023-11-24T08:20:13.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.146+0000 7fd774648640 1 -- start start 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd76c0cbf20 0x7fd76c156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c15b440 0x7fd76c09b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15c8e0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15cbc0 con 0x7fd76c0cbf20 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd774648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15cea0 con 0x7fd76c15b440 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56134/0 (socket says 172.21.15.107:56134) 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 -- 172.21.15.107:0/4262005531 learned_addr learned my addr 172.21.15.107:0/4262005531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 -- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c15b440 msgr2=0x7fd76c09b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c15b440 0x7fd76c09b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 -- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd76c0cbf20 msgr2=0x7fd76c156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd76c0cbf20 0x7fd76c156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.147+0000 7fd7723e4640 1 -- 172.21.15.107:0/4262005531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680094d0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd7723e4640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd7680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76803ac60 con 0x7fd76c0c57e0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76803adc0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768044630 con 0x7fd76c0c57e0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c15d180 con 0x7fd76c0c57e0 2023-11-24T08:20:13.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.148+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c15e6c0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.149+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c0c7050 con 0x7fd76c0c57e0 2023-11-24T08:20:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.150+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fd76802f050 con 0x7fd76c0c57e0 2023-11-24T08:20:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.151+0000 7fd762ffd640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 0x7fd75006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.151+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd7680aa650 con 0x7fd76c0c57e0 2023-11-24T08:20:13.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.151+0000 7fd771be3640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 0x7fd75006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.152+0000 7fd771be3640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 0x7fd75006efb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd76c09aec0 tx=0x7fd75c006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.155+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd768077200 con 0x7fd76c0c57e0 2023-11-24T08:20:13.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.343+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd76c04e8b0 con 0x7fd76c0c57e0 2023-11-24T08:20:13.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.344+0000 7fd762ffd640 1 -- 172.21.15.107:0/4262005531 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd76807a820 con 0x7fd76c0c57e0 2023-11-24T08:20:13.346 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:20:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.346+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 msgr2=0x7fd75006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.346+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 0x7fd75006efb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd76c09aec0 tx=0x7fd75c006c90 comp rx=0 tx=0).stop 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.346+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 msgr2=0x7fd76c15e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.346+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd7680046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 shutdown_connections 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd75006caf0 0x7fd75006efb0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd76c15b440 0x7fd76c09b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd76c0cbf20 0x7fd76c156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 --2- 172.21.15.107:0/4262005531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd76c0c57e0 0x7fd76c15e180 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 >> 172.21.15.107:0/4262005531 conn(0x7fd76c0b0c50 msgr2=0x7fd76c06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 shutdown_connections 2023-11-24T08:20:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.347+0000 7fd774648640 1 -- 172.21.15.107:0/4262005531 wait complete. 2023-11-24T08:20:13.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:20:13.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.578+0000 7fd1d1f68640 1 Processor -- start 2023-11-24T08:20:13.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.578+0000 7fd1d1f68640 1 -- start start 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1d1f68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1cc166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1d1f68640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc15eea0 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1d1f68640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc06d600 con 0x7fd1cc15b440 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1d1f68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0c5e00 con 0x7fd1cc0c59b0 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1c2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1cc0cc0f0 0x7fd1cc0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38350/0 (socket says 172.21.15.107:38350) 2023-11-24T08:20:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1c2ffd640 1 -- 172.21.15.107:0/4201156560 learned_addr learned my addr 172.21.15.107:0/4201156560 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218035634 0 0) 0x7fd1cc15eea0 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.580+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b8003520 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1cc166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616420672 0 0) 0x7fd1b8003520 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 msgr2=0x7fd1cc166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:13.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1cc166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 >> v1:172.21.15.120:6789/0 conn(0x7fd1cc15b440 legacy=0x7fd1cc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0abf30 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd1b4001670 con 0x7fd1cc0cc0f0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 >> v1:172.21.15.107:6789/0 conn(0x7fd1cc0cc0f0 legacy=0x7fd1cc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1b80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b8003fd0 0x7fd1b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b8004d00 0x7fd1b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b800bb90 con 0x7fd1b8003fd0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b800bd00 con 0x7fd1b8004d00 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b800bec0 con 0x7fd1cc0c59b0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.581+0000 7fd1c2ffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b8003fd0 0x7fd1b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1cb7fe640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1b80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b8004d00 0x7fd1b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1c2ffd640 1 -- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 msgr2=0x7fd1b80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1c2ffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1cc0c59b0 0x7fd1b80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1c2ffd640 1 -- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b8004d00 msgr2=0x7fd1b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1c2ffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b8004d00 0x7fd1b8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1c2ffd640 1 -- 172.21.15.107:0/4201156560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8003520 con 0x7fd1b8003fd0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.582+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b8004d00 0x7fd1b8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.583+0000 7fd1c2ffd640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b8003fd0 0x7fd1b80047c0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c240 tx=0x7fd1b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377674bf25fc7a0c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.583+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4005ad0 con 0x7fd1b8003fd0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.583+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4005c30 con 0x7fd1b8003fd0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.583+0000 7fd1caffd640 1 -- 172.21.15.107:0/4201156560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b402caf0 con 0x7fd1b8003fd0 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 -- 172.21.15.107:0/4201156560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b8003fd0 msgr2=0x7fd1b80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/4201156560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b8003fd0 0x7fd1b80047c0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c240 tx=0x7fd1b80057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1cbfff640 1 -- 172.21.15.107:0/4201156560 reap_dead start 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 -- 172.21.15.107:0/4201156560 shutdown_connections 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 -- 172.21.15.107:0/4201156560 >> 172.21.15.107:0/4201156560 conn(0x7fd1cc0b0c50 msgr2=0x7fd1cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 -- 172.21.15.107:0/4201156560 shutdown_connections 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.584+0000 7fd1d1f68640 1 -- 172.21.15.107:0/4201156560 wait complete. 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.586+0000 7fd1d1f68640 1 Processor -- start 2023-11-24T08:20:13.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.586+0000 7fd1d1f68640 1 -- start start 2023-11-24T08:20:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b8003fd0 0x7fd1cc0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1cc0b3920 0x7fd1cc0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0c6fa0 con 0x7fd1cc0c7650 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0b42e0 con 0x7fd1cc0b3920 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1d1f68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0c60d0 con 0x7fd1b8003fd0 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56162/0 (socket says 172.21.15.107:56162) 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 -- 172.21.15.107:0/2177559801 learned_addr learned my addr 172.21.15.107:0/2177559801 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cb7fe640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b8003fd0 0x7fd1cc0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1c2ffd640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1cc0b3920 0x7fd1cc0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 -- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b8003fd0 msgr2=0x7fd1cc0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b8003fd0 0x7fd1cc0acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 -- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1cc0b3920 msgr2=0x7fd1cc0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1cc0b3920 0x7fd1cc0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.587+0000 7fd1cbfff640 1 -- 172.21.15.107:0/2177559801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0c8010 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1cbfff640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400e420 tx=0x7fd1c400e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400c070 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c4013630 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40139e0 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc0c6320 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.588+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc0c68e0 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.589+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc1f1310 con 0x7fd1cc0c7650 2023-11-24T08:20:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.590+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fd1c4013b40 con 0x7fd1cc0c7650 2023-11-24T08:20:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.591+0000 7fd1c8ff9640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 0x7fd1ac06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.591+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd1c4087960 con 0x7fd1cc0c7650 2023-11-24T08:20:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.591+0000 7fd1cb7fe640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 0x7fd1ac06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.592+0000 7fd1cb7fe640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 0x7fd1ac06efe0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc001c90 tx=0x7fd1bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.596+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1c4054510 con 0x7fd1cc0c7650 2023-11-24T08:20:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.785+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1cc04e8b0 con 0x7fd1cc0c7650 2023-11-24T08:20:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.786+0000 7fd1c8ff9640 1 -- 172.21.15.107:0/2177559801 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v178) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1c4057b30 con 0x7fd1cc0c7650 2023-11-24T08:20:13.788 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:20:13.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 msgr2=0x7fd1ac06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 0x7fd1ac06efe0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc001c90 tx=0x7fd1bc009040 comp rx=0 tx=0).stop 2023-11-24T08:20:13.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 msgr2=0x7fd1cc0c7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:13.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400e420 tx=0x7fd1c400e8e0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 shutdown_connections 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd1ac06cb20 0x7fd1ac06efe0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1cc0c7650 0x7fd1cc0c7ad0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1cc0b3920 0x7fd1cc0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 --2- 172.21.15.107:0/2177559801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b8003fd0 0x7fd1cc0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 >> 172.21.15.107:0/2177559801 conn(0x7fd1cc0b0c50 msgr2=0x7fd1cc06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.788+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 shutdown_connections 2023-11-24T08:20:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:13.789+0000 7fd1d1f68640 1 -- 172.21.15.107:0/2177559801 wait complete. 2023-11-24T08:20:13.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:20:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.024+0000 7f4af53d5640 1 Processor -- start 2023-11-24T08:20:14.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.025+0000 7f4af53d5640 1 -- start start 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4af53d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4af00c59b0 0x7f4af0166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4af53d5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af00c5dc0 con 0x7f4af015b440 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4af53d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af015eea0 con 0x7f4af00cc0f0 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4af53d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af015b820 con 0x7f4af00c59b0 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4aeeffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4af015b440 0x7f4af00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38366/0 (socket says 172.21.15.107:38366) 2023-11-24T08:20:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.027+0000 7f4aeeffd640 1 -- 172.21.15.107:0/747133389 learned_addr learned my addr 172.21.15.107:0/747133389 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982518246 0 0) 0x7f4af00c5dc0 con 0x7f4af015b440 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/747133389 >> v2:172.21.15.120:3300/0 conn(0x7f4af00c59b0 0x7f4af0166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc003520 con 0x7f4af015b440 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345972341 0 0) 0x7f4adc003520 con 0x7f4af015b440 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 >> v2:172.21.15.120:3300/0 conn(0x7f4af00c59b0 msgr2=0x7f4af0166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 --2- 172.21.15.107:0/747133389 >> v2:172.21.15.120:3300/0 conn(0x7f4af00c59b0 0x7f4af0166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 >> v1:172.21.15.120:6789/0 conn(0x7f4af00cc0f0 legacy=0x7f4af00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af006d600 con 0x7f4af015b440 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4ae0001670 con 0x7f4af015b440 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 >> v1:172.21.15.107:6789/0 conn(0x7f4af015b440 legacy=0x7f4af00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4adc0039c0 0x7f4adc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc0047c0 0x7f4adc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 0x7f4adc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af015b820 con 0x7f4adc005250 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc00c1a0 con 0x7f4adc0047c0 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.028+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc00c360 con 0x7f4adc0039c0 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 0x7f4adc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aeeffd640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4adc0039c0 0x7f4adc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 -- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4adc0039c0 msgr2=0x7f4adc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4adc0039c0 0x7f4adc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 -- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc0047c0 msgr2=0x7f4adc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc0047c0 0x7f4adc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aef7fe640 1 -- 172.21.15.107:0/747133389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc003520 con 0x7f4adc005250 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.029+0000 7f4aeeffd640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4adc0039c0 0x7f4adc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.030+0000 7f4aef7fe640 1 -- 172.21.15.107:0/747133389 reap_dead start 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.030+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 0x7f4adc00b980 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8007360 tx=0x7f4adc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c060e9db33b20020 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.030+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae802ccf0 con 0x7f4adc005250 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.030+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae802ce90 con 0x7f4adc005250 2023-11-24T08:20:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.030+0000 7f4aee7fc640 1 -- 172.21.15.107:0/747133389 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8003bd0 con 0x7f4adc005250 2023-11-24T08:20:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 -- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 msgr2=0x7f4adc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 0x7f4adc00b980 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8007360 tx=0x7f4adc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 -- 172.21.15.107:0/747133389 shutdown_connections 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 --2- 172.21.15.107:0/747133389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4adc005250 0x7f4adc00b980 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 -- 172.21.15.107:0/747133389 >> 172.21.15.107:0/747133389 conn(0x7f4af00b0c50 msgr2=0x7f4af00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 -- 172.21.15.107:0/747133389 shutdown_connections 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.031+0000 7f4af53d5640 1 -- 172.21.15.107:0/747133389 wait complete. 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.033+0000 7f4af53d5640 1 Processor -- start 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.033+0000 7f4af53d5640 1 -- start start 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc005250 0x7f4af014aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4af01f65f0 0x7f4af01f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0152360 con 0x7f4af0152d70 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af01528e0 con 0x7f4adc005250 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4af53d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af014b3e0 con 0x7f4af01f65f0 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4ae67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4ae67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56170/0 (socket says 172.21.15.107:56170) 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4ae67fc640 1 -- 172.21.15.107:0/3450579852 learned_addr learned my addr 172.21.15.107:0/3450579852 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.034+0000 7f4aeeffd640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc005250 0x7f4af014aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4aef7fe640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4af01f65f0 0x7f4af01f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 -- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4af01f65f0 msgr2=0x7f4af01f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4af01f65f0 0x7f4af01f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 -- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc005250 msgr2=0x7f4af014aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc005250 0x7f4af014aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 -- 172.21.15.107:0/3450579852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af01f7070 con 0x7f4af0152d70 2023-11-24T08:20:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae67fc640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800c480 tx=0x7f4ad800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8012830 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8012990 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af01f7350 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8012d80 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.035+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af00c6a70 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.036+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af004e8b0 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.037+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f4ad801b400 con 0x7f4af0152d70 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.038+0000 7f4ae7fff640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 0x7f4ad006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.038+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4ad8088520 con 0x7f4af0152d70 2023-11-24T08:20:14.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.039+0000 7f4aeeffd640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 0x7f4ad006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.039+0000 7f4aeeffd640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 0x7f4ad006efb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0001eb0 tx=0x7f4ae0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.043+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ad8055030 con 0x7f4af0152d70 2023-11-24T08:20:14.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.233+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4af006dc80 con 0x7f4af0152d70 2023-11-24T08:20:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.234+0000 7f4ae7fff640 1 -- 172.21.15.107:0/3450579852 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ad8016020 con 0x7f4af0152d70 2023-11-24T08:20:14.236 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 msgr2=0x7f4ad006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 0x7f4ad006efb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0001eb0 tx=0x7f4ae0009040 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 msgr2=0x7f4af01531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800c480 tx=0x7f4ad800c940 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 shutdown_connections 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4ad006caf0 0x7f4ad006efb0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4af01f65f0 0x7f4af01f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4af0152d70 0x7f4af01531f0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 --2- 172.21.15.107:0/3450579852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4adc005250 0x7f4af014aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 >> 172.21.15.107:0/3450579852 conn(0x7f4af00b0c50 msgr2=0x7f4af00c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 shutdown_connections 2023-11-24T08:20:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.237+0000 7f4af53d5640 1 -- 172.21.15.107:0/3450579852 wait complete. 2023-11-24T08:20:14.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:20:14.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.462+0000 7fae9a47c640 1 Processor -- start 2023-11-24T08:20:14.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.462+0000 7fae9a47c640 1 -- start start 2023-11-24T08:20:14.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae9a47c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 0x7fae940c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae9a47c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae94166510 con 0x7fae940ccbd0 2023-11-24T08:20:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae9a47c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae941655f0 con 0x7fae941660b0 2023-11-24T08:20:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae9a47c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cc380 con 0x7fae940c9650 2023-11-24T08:20:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae93fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fae940ccbd0 0x7fae940ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38368/0 (socket says 172.21.15.107:38368) 2023-11-24T08:20:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae93fff640 1 -- 172.21.15.107:0/219395146 learned_addr learned my addr 172.21.15.107:0/219395146 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929015734 0 0) 0x7fae84001110 con 0x7fae940ccbd0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae98a19640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 0x7fae940c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.464+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c003520 con 0x7fae940ccbd0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 382742363 0 0) 0x7fae7c003520 con 0x7fae940ccbd0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 msgr2=0x7fae940c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 0x7fae940c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 >> v1:172.21.15.120:6789/0 conn(0x7fae941660b0 legacy=0x7fae94162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94162b30 con 0x7fae940ccbd0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fae84001800 con 0x7fae940ccbd0 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 >> v1:172.21.15.107:6789/0 conn(0x7fae940ccbd0 legacy=0x7fae940ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7c0039a0 0x7fae7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 0x7fae7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae7c004d00 0x7fae7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cc380 con 0x7fae7c004d00 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c00bb10 con 0x7fae7c0039a0 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c00bcd0 con 0x7fae940c9650 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae98a19640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae7c004d00 0x7fae7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae93fff640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7c0039a0 0x7fae7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae98a19640 1 -- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 msgr2=0x7fae7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae98a19640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae940c9650 0x7fae7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae98a19640 1 -- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7c0039a0 msgr2=0x7fae7c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.465+0000 7fae98a19640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7c0039a0 0x7fae7c003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.466+0000 7fae98a19640 1 -- 172.21.15.107:0/219395146 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c003520 con 0x7fae7c004d00 2023-11-24T08:20:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae98a19640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae7c004d00 0x7fae7c005180 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fae8c007360 tx=0x7fae7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19581f916ba0f035 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c02ccf0 con 0x7fae7c004d00 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c02ce50 con 0x7fae7c004d00 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae92ffd640 1 -- 172.21.15.107:0/219395146 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c034660 con 0x7fae7c004d00 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae9a47c640 1 -- 172.21.15.107:0/219395146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae7c004d00 msgr2=0x7fae7c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae9a47c640 1 --2- 172.21.15.107:0/219395146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae7c004d00 0x7fae7c005180 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fae8c007360 tx=0x7fae7c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.467+0000 7fae98a19640 1 -- 172.21.15.107:0/219395146 reap_dead start 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.468+0000 7fae9a47c640 1 -- 172.21.15.107:0/219395146 shutdown_connections 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.468+0000 7fae9a47c640 1 -- 172.21.15.107:0/219395146 >> 172.21.15.107:0/219395146 conn(0x7fae940b0c50 msgr2=0x7fae940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.468+0000 7fae9a47c640 1 -- 172.21.15.107:0/219395146 shutdown_connections 2023-11-24T08:20:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.468+0000 7fae9a47c640 1 -- 172.21.15.107:0/219395146 wait complete. 2023-11-24T08:20:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.469+0000 7fae9a47c640 1 Processor -- start 2023-11-24T08:20:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.469+0000 7fae9a47c640 1 -- start start 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 0x7fae94072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 0x7fae94157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415c400 con 0x7fae940adbd0 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415cbd0 con 0x7fae940acdd0 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.470+0000 7fae9a47c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940ac940 con 0x7fae7c004d00 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56186/0 (socket says 172.21.15.107:56186) 2023-11-24T08:20:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 -- 172.21.15.107:0/913331535 learned_addr learned my addr 172.21.15.107:0/913331535 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae93fff640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 0x7fae94072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae937fe640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 0x7fae94157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 -- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 msgr2=0x7fae94072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 0x7fae94072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 -- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 msgr2=0x7fae94157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 0x7fae94157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae98a19640 1 -- 172.21.15.107:0/913331535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94072880 con 0x7fae940adbd0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae93fff640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 0x7fae94072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.471+0000 7fae937fe640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 0x7fae94157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae98a19640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02dfd0 tx=0x7fae8c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c03e600 con 0x7fae940adbd0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c03e760 con 0x7fae940adbd0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c03eaf0 con 0x7fae940adbd0 2023-11-24T08:20:14.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae940ae0d0 con 0x7fae940adbd0 2023-11-24T08:20:14.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.472+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9415c8f0 con 0x7fae940adbd0 2023-11-24T08:20:14.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.473+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae940acab0 con 0x7fae940adbd0 2023-11-24T08:20:14.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.474+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fae8c03ec50 con 0x7fae940adbd0 2023-11-24T08:20:14.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.475+0000 7fae90ff9640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 0x7fae8006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:14.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.475+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fae8c0aabb0 con 0x7fae940adbd0 2023-11-24T08:20:14.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.475+0000 7fae93fff640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 0x7fae8006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.476+0000 7fae93fff640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 0x7fae8006efb0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fae84001930 tx=0x7fae84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.479+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae8c0776c0 con 0x7fae940adbd0 2023-11-24T08:20:14.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.683+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae9404e8b0 con 0x7fae940adbd0 2023-11-24T08:20:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.684+0000 7fae90ff9640 1 -- 172.21.15.107:0/913331535 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v178) v1 ==== 80+0+0 (secure 0 0 0) 0x7fae8c039030 con 0x7fae940adbd0 2023-11-24T08:20:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:20:14.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 msgr2=0x7fae8006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 0x7fae8006efb0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fae84001930 tx=0x7fae84009040 comp rx=0 tx=0).stop 2023-11-24T08:20:14.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 msgr2=0x7fae94160c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:14.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02dfd0 tx=0x7fae8c004ab0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 shutdown_connections 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fae8006caf0 0x7fae8006efb0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae940adbd0 0x7fae94160c70 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae940acdd0 0x7fae94157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 --2- 172.21.15.107:0/913331535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae7c004d00 0x7fae94072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 >> 172.21.15.107:0/913331535 conn(0x7fae940b0c50 msgr2=0x7fae9406c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 shutdown_connections 2023-11-24T08:20:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:14.687+0000 7fae9a47c640 1 -- 172.21.15.107:0/913331535 wait complete. 2023-11-24T08:20:14.706 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:20:14.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:14.706 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:20:14.822 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:20:14.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:14.823 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:20:14.877 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:14.877 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:14.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:20:14.938 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:14.938 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:14.999 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:20:14.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.216+0000 7fc9b2fbe640 1 Processor -- start 2023-11-24T08:20:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.216+0000 7fc9b2fbe640 1 -- start start 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b2fbe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9ac159ce0 0x7fc9ac06de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b2fbe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac0b3a40 con 0x7fc9ac0cc150 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b2fbe640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac0cdfa0 con 0x7fc9ac0b21d0 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b2fbe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ac0cd9e0 con 0x7fc9ac159ce0 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b0d5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9ac159ce0 0x7fc9ac06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b155b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc9ac0cc150 0x7fc9ac06c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38382/0 (socket says 172.21.15.107:38382) 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b155b640 1 -- 172.21.15.107:0/3160268298 learned_addr learned my addr 172.21.15.107:0/3160268298 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b0d5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9ac159ce0 0x7fc9ac06de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56198/0 (socket says 172.21.15.107:56198) 2023-11-24T08:20:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b0d5a640 1 -- 172.21.15.107:0/3160268298 >> v1:172.21.15.120:6789/0 conn(0x7fc9ac0b21d0 legacy=0x7fc9ac0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b0d5a640 1 -- 172.21.15.107:0/3160268298 >> v1:172.21.15.107:6789/0 conn(0x7fc9ac0cc150 legacy=0x7fc9ac06c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9b0d5a640 1 -- 172.21.15.107:0/3160268298 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac0cdcc0 con 0x7fc9ac159ce0 2023-11-24T08:20:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.218+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/3160268298 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133520186 0 0) 0x7fc9ac0b3a40 con 0x7fc9ac0cc150 2023-11-24T08:20:15.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9b0d5a640 1 --2- 172.21.15.107:0/3160268298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac159ce0 0x7fc9ac06de80 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0097a0 tx=0x7fc99c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31ba7d4cce3fc04 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/3160268298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c031830 con 0x7fc9ac159ce0 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/3160268298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c031990 con 0x7fc9ac159ce0 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/3160268298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c031d20 con 0x7fc9ac159ce0 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/3160268298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac159ce0 msgr2=0x7fc9ac06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.220+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/3160268298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac159ce0 0x7fc9ac06de80 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0097a0 tx=0x7fc99c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.221+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/3160268298 shutdown_connections 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.221+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/3160268298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac159ce0 0x7fc9ac06de80 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.221+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/3160268298 >> 172.21.15.107:0/3160268298 conn(0x7fc9ac06a920 msgr2=0x7fc9ac06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.221+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/3160268298 shutdown_connections 2023-11-24T08:20:15.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.221+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/3160268298 wait complete. 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.222+0000 7fc9b2fbe640 1 Processor -- start 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.223+0000 7fc9b2fbe640 1 -- start start 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.223+0000 7fc9b2fbe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 0x7fc9ac147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b2fbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b2fbe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9ac159ce0 0x7fc9ac1448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b2fbe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac144e20 con 0x7fc9ac0cc150 2023-11-24T08:20:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b2fbe640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac155b80 con 0x7fc9ac0b21d0 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b2fbe640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac155e60 con 0x7fc9ac159ce0 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56210/0 (socket says 172.21.15.107:56210) 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 -- 172.21.15.107:0/1597236303 learned_addr learned my addr 172.21.15.107:0/1597236303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b0d5a640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 0x7fc9ac147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 -- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9ac159ce0 msgr2=0x7fc9ac1448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b155b640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9ac159ce0 0x7fc9ac1448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9ac159ce0 0x7fc9ac1448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 -- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 msgr2=0x7fc9ac147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 0x7fc9ac147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9abfff640 1 -- 172.21.15.107:0/1597236303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c009400 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.224+0000 7fc9b0d5a640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 0x7fc9ac147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9abfff640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c430 tx=0x7fc99400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994011790 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9940118f0 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994011ce0 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac156140 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.225+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac156700 con 0x7fc9ac0cc150 2023-11-24T08:20:15.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.226+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac067470 con 0x7fc9ac0cc150 2023-11-24T08:20:15.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.227+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fc99401c460 con 0x7fc9ac0cc150 2023-11-24T08:20:15.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.228+0000 7fc9a97fa640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 0x7fc98c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.228+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc994088500 con 0x7fc9ac0cc150 2023-11-24T08:20:15.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.228+0000 7fc9b0d5a640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 0x7fc98c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.229+0000 7fc9b0d5a640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 0x7fc98c06efb0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc99c036000 tx=0x7fc99c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.233+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9940550b0 con 0x7fc9ac0cc150 2023-11-24T08:20:15.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:20:15.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:20:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.460+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9ac0ce180 con 0x7fc9ac0cc150 2023-11-24T08:20:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.461+0000 7fc9a97fa640 1 -- 172.21.15.107:0/1597236303 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4938 (secure 0 0 0) 0x7fc9940586d0 con 0x7fc9ac0cc150 2023-11-24T08:20:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:15.463 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:20:15.464 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 msgr2=0x7fc98c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 0x7fc98c06efb0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc99c036000 tx=0x7fc99c03c040 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 msgr2=0x7fc9ac156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c430 tx=0x7fc99400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 shutdown_connections 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc98c06caf0 0x7fc98c06efb0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9ac159ce0 0x7fc9ac1448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ac0cc150 0x7fc9ac156c20 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 --2- 172.21.15.107:0/1597236303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9ac0b21d0 0x7fc9ac147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 >> 172.21.15.107:0/1597236303 conn(0x7fc9ac06a920 msgr2=0x7fc9ac158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:15.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 shutdown_connections 2023-11-24T08:20:15.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.464+0000 7fc9b2fbe640 1 -- 172.21.15.107:0/1597236303 wait complete. 2023-11-24T08:20:15.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:15.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.680+0000 7fdc9107c640 1 Processor -- start 2023-11-24T08:20:15.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.681+0000 7fdc9107c640 1 -- start start 2023-11-24T08:20:15.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.682+0000 7fdc9107c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc8c15e390 0x7fdc8c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.682+0000 7fdc9107c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c0ce0b0 con 0x7fdc8c157c70 2023-11-24T08:20:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.682+0000 7fdc9107c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c161960 con 0x7fdc8c0cfb60 2023-11-24T08:20:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.682+0000 7fdc9107c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8c15e8d0 con 0x7fdc8c15e390 2023-11-24T08:20:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdc8c157c70 0x7fdc8c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38392/0 (socket says 172.21.15.107:38392) 2023-11-24T08:20:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8ad9d640 1 -- 172.21.15.107:0/1126763687 learned_addr learned my addr 172.21.15.107:0/1126763687 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096267505 0 0) 0x7fdc8c0ce0b0 con 0x7fdc8c157c70 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc78003520 con 0x7fdc8c157c70 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/1126763687 >> v2:172.21.15.120:3300/0 conn(0x7fdc8c15e390 0x7fdc8c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203098318 0 0) 0x7fdc78003520 con 0x7fdc8c157c70 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 >> v2:172.21.15.120:3300/0 conn(0x7fdc8c15e390 msgr2=0x7fdc8c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 --2- 172.21.15.107:0/1126763687 >> v2:172.21.15.120:3300/0 conn(0x7fdc8c15e390 0x7fdc8c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 >> v1:172.21.15.120:6789/0 conn(0x7fdc8c0cfb60 legacy=0x7fdc8c15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c15ebb0 con 0x7fdc8c157c70 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdc7c001670 con 0x7fdc8c157c70 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 >> v1:172.21.15.107:6789/0 conn(0x7fdc8c157c70 legacy=0x7fdc8c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.683+0000 7fdc8a59c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc78003970 0x7fdc78003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8a59c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc780047c0 0x7fdc78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8a59c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 0x7fdc7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c15e8d0 con 0x7fdc780052a0 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800c150 con 0x7fdc780047c0 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800c310 con 0x7fdc78003970 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 0x7fdc7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8ad9d640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc78003970 0x7fdc78003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8259c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc780047c0 0x7fdc78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 -- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc78003970 msgr2=0x7fdc78003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc78003970 0x7fdc78003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 -- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc780047c0 msgr2=0x7fdc78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc780047c0 0x7fdc78004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8b59e640 1 -- 172.21.15.107:0/1126763687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78003520 con 0x7fdc780052a0 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.684+0000 7fdc8259c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc780047c0 0x7fdc78004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.685+0000 7fdc8b59e640 1 -- 172.21.15.107:0/1126763687 reap_dead start 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.685+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 0x7fdc7800b980 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c15e800 tx=0x7fdc780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48745d8efd848941 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8402cc70 con 0x7fdc780052a0 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8402cdd0 con 0x7fdc780052a0 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc8a59c640 1 -- 172.21.15.107:0/1126763687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84034660 con 0x7fdc780052a0 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 -- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 msgr2=0x7fdc7800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 0x7fdc7800b980 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c15e800 tx=0x7fdc780057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 -- 172.21.15.107:0/1126763687 shutdown_connections 2023-11-24T08:20:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 --2- 172.21.15.107:0/1126763687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc780052a0 0x7fdc7800b980 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 -- 172.21.15.107:0/1126763687 >> 172.21.15.107:0/1126763687 conn(0x7fdc8c06a920 msgr2=0x7fdc8c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 -- 172.21.15.107:0/1126763687 shutdown_connections 2023-11-24T08:20:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.686+0000 7fdc9107c640 1 -- 172.21.15.107:0/1126763687 wait complete. 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.688+0000 7fdc9107c640 1 Processor -- start 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.689+0000 7fdc9107c640 1 -- start start 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 0x7fdc8c0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc8c148240 0x7fdc8c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0acdb0 con 0x7fdc8c1486c0 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0ad090 con 0x7fdc8c148240 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc9107c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0ad370 con 0x7fdc780052a0 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56224/0 (socket says 172.21.15.107:56224) 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 -- 172.21.15.107:0/3192197091 learned_addr learned my addr 172.21.15.107:0/3192197091 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8ad9d640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 0x7fdc8c0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8259c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc8c148240 0x7fdc8c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 -- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 msgr2=0x7fdc8c0ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 0x7fdc8c0ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 -- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc8c148240 msgr2=0x7fdc8c153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc8c148240 0x7fdc8c153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.690+0000 7fdc8b59e640 1 -- 172.21.15.107:0/3192197091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c0ad650 con 0x7fdc8c1486c0 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc8ad9d640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 0x7fdc8c0ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc8b59e640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fdc8402c6e0 tx=0x7fdc84004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84044070 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8403f440 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c0ad930 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8403f7f0 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.691+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c071b50 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.693+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8c067470 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.693+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fdc8403b040 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.694+0000 7fdc83fff640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 0x7fdc6c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.694+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdc840a9680 con 0x7fdc8c1486c0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.695+0000 7fdc8ad9d640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 0x7fdc6c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:15.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.695+0000 7fdc8ad9d640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 0x7fdc6c06efe0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001750 tx=0x7fdc7c000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:15.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.699+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc840762e0 con 0x7fdc8c1486c0 2023-11-24T08:20:15.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.889+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc8c0b0590 con 0x7fdc8c1486c0 2023-11-24T08:20:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.891+0000 7fdc83fff640 1 -- 172.21.15.107:0/3192197091 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16688 (secure 0 0 0) 0x7fdc84079900 con 0x7fdc8c1486c0 2023-11-24T08:20:15.893 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:15.894 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":179,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:15.077624+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":178,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":2}]},{"pgid":"25.17","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":25,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.893+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 msgr2=0x7fdc6c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.893+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 0x7fdc6c06efe0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001750 tx=0x7fdc7c000ef0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 msgr2=0x7fdc8c071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fdc8402c6e0 tx=0x7fdc84004ab0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 shutdown_connections 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdc6c06cb20 0x7fdc6c06efe0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc8c1486c0 0x7fdc8c071610 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc8c148240 0x7fdc8c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 --2- 172.21.15.107:0/3192197091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc780052a0 0x7fdc8c0ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 >> 172.21.15.107:0/3192197091 conn(0x7fdc8c06a920 msgr2=0x7fdc8c0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 shutdown_connections 2023-11-24T08:20:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:15.894+0000 7fdc9107c640 1 -- 172.21.15.107:0/3192197091 wait complete. 2023-11-24T08:20:15.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:19:19.824499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:19:21.010791+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 178, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:20:15.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:20:16.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.114+0000 7f24d759e640 1 Processor -- start 2023-11-24T08:20:16.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.115+0000 7f24d759e640 1 -- start start 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24d759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24d80b0b40 0x7f24d80b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24d759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24d80cce40 0x7f24d80cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24d759e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80b14d0 con 0x7f24d815d310 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24d759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d806cd10 con 0x7f24d80b0b40 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24d759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d80ae5d0 con 0x7f24d80cce40 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24d80b0b40 0x7f24d80b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24d80b0b40 0x7f24d80b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56236/0 (socket says 172.21.15.107:56236) 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.116+0000 7f24cffff640 1 -- 172.21.15.107:0/1839451480 learned_addr learned my addr 172.21.15.107:0/1839451480 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.117+0000 7f24cffff640 1 -- 172.21.15.107:0/1839451480 >> v2:172.21.15.120:3300/0 conn(0x7f24d80cce40 msgr2=0x7f24d80cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.117+0000 7f24cffff640 1 --2- 172.21.15.107:0/1839451480 >> v2:172.21.15.120:3300/0 conn(0x7f24d80cce40 0x7f24d80cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.117+0000 7f24cffff640 1 -- 172.21.15.107:0/1839451480 >> v1:172.21.15.120:6789/0 conn(0x7f24d815d310 legacy=0x7f24d80ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.117+0000 7f24cffff640 1 -- 172.21.15.107:0/1839451480 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80ae8b0 con 0x7f24d80b0b40 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24cffff640 1 --2- 172.21.15.107:0/1839451480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d80b0f90 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f24d80cb240 tx=0x7f24c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1724e1048ffc57a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24d5d9b640 1 -- 172.21.15.107:0/1839451480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0031870 con 0x7f24d80b0b40 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24d5d9b640 1 -- 172.21.15.107:0/1839451480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00319d0 con 0x7f24d80b0b40 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24d5d9b640 1 -- 172.21.15.107:0/1839451480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0031d60 con 0x7f24d80b0b40 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24d759e640 1 -- 172.21.15.107:0/1839451480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 msgr2=0x7f24d80b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.118+0000 7f24d759e640 1 --2- 172.21.15.107:0/1839451480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d80b0f90 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f24d80cb240 tx=0x7f24c002b910 comp rx=0 tx=0).stop 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 -- 172.21.15.107:0/1839451480 shutdown_connections 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 --2- 172.21.15.107:0/1839451480 >> v2:172.21.15.120:3300/0 conn(0x7f24d80cce40 0x7f24d80cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 --2- 172.21.15.107:0/1839451480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d80b0f90 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 -- 172.21.15.107:0/1839451480 >> 172.21.15.107:0/1839451480 conn(0x7f24d80cbb40 msgr2=0x7f24d80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 -- 172.21.15.107:0/1839451480 shutdown_connections 2023-11-24T08:20:16.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.119+0000 7f24d759e640 1 -- 172.21.15.107:0/1839451480 wait complete. 2023-11-24T08:20:16.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.120+0000 7f24d759e640 1 Processor -- start 2023-11-24T08:20:16.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.120+0000 7f24d759e640 1 -- start start 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24d80cce40 0x7f24d80c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24d815d310 0x7f24d814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d81e8c80 con 0x7f24d80b0b40 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d81e80b0 con 0x7f24d80cce40 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d81e8350 con 0x7f24d815d310 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.121+0000 7f24d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56240/0 (socket says 172.21.15.107:56240) 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 -- 172.21.15.107:0/55090844 learned_addr learned my addr 172.21.15.107:0/55090844 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 -- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24d815d310 msgr2=0x7f24d814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24d815d310 0x7f24d814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24cffff640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24d80cce40 0x7f24d80c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 -- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24d80cce40 msgr2=0x7f24d80c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24d80cce40 0x7f24d80c2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 -- 172.21.15.107:0/55090844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c00094d0 con 0x7f24d80b0b40 2023-11-24T08:20:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d659c640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f24c800b3e0 tx=0x7f24c800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8011830 con 0x7f24d80b0b40 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c80119b0 con 0x7f24d80b0b40 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8011d60 con 0x7f24d80b0b40 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.122+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d81e8630 con 0x7f24d80b0b40 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.123+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d8099bb0 con 0x7f24d80b0b40 2023-11-24T08:20:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.124+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d804e840 con 0x7f24d80b0b40 2023-11-24T08:20:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.125+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f24c801a400 con 0x7f24d80b0b40 2023-11-24T08:20:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.125+0000 7f24ceffd640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 0x7f24b806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.126+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f24c80887e0 con 0x7f24d80b0b40 2023-11-24T08:20:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.126+0000 7f24cffff640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 0x7f24b806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.126+0000 7f24cffff640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 0x7f24b806efe0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f24c0036000 tx=0x7f24c003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.130+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24c8055260 con 0x7f24d80b0b40 2023-11-24T08:20:16.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.304+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24d809cfb0 con 0x7f24d80b0b40 2023-11-24T08:20:16.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.305+0000 7f24ceffd640 1 -- 172.21.15.107:0/55090844 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24c8015070 con 0x7f24d80b0b40 2023-11-24T08:20:16.307 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 msgr2=0x7f24b806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 0x7f24b806efe0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f24c0036000 tx=0x7f24c003c040 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 msgr2=0x7f24d809c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f24c800b3e0 tx=0x7f24c800b8a0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 shutdown_connections 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24b806cb20 0x7f24b806efe0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24d815d310 0x7f24d814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24d80cce40 0x7f24d80c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 --2- 172.21.15.107:0/55090844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d80b0b40 0x7f24d809c4f0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 >> 172.21.15.107:0/55090844 conn(0x7f24d80cbb40 msgr2=0x7f24d80cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 shutdown_connections 2023-11-24T08:20:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.308+0000 7f24d759e640 1 -- 172.21.15.107:0/55090844 wait complete. 2023-11-24T08:20:16.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:20:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.542+0000 7fb3db31d640 1 Processor -- start 2023-11-24T08:20:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.542+0000 7fb3db31d640 1 -- start start 2023-11-24T08:20:16.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.543+0000 7fb3db31d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3d415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.543+0000 7fb3db31d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d415ec90 con 0x7fb3d4072160 2023-11-24T08:20:16.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3db31d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d40cd530 con 0x7fb3d406de10 2023-11-24T08:20:16.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3db31d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40c4ed0 con 0x7fb3d40c49f0 2023-11-24T08:20:16.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3d90b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb3d4072160 0x7fb3d406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38402/0 (socket says 172.21.15.107:38402) 2023-11-24T08:20:16.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3d90b9640 1 -- 172.21.15.107:0/1707981228 learned_addr learned my addr 172.21.15.107:0/1707981228 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553724006 0 0) 0x7fb3c8001110 con 0x7fb3d4072160 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3d98ba640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3d415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.544+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c0005730 con 0x7fb3d4072160 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044855669 0 0) 0x7fb3c0005730 con 0x7fb3d4072160 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 msgr2=0x7fb3d415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3d415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 >> v1:172.21.15.120:6789/0 conn(0x7fb3d406de10 legacy=0x7fb3d40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d40c51b0 con 0x7fb3d4072160 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb3c8001800 con 0x7fb3d4072160 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 >> v1:172.21.15.107:6789/0 conn(0x7fb3d4072160 legacy=0x7fb3d406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c0005bb0 0x7fb3c0005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c00069d0 0x7fb3c0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3c0007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40c4ed0 con 0x7fb3c0005bb0 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000dd20 con 0x7fb3c00069d0 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000dee0 con 0x7fb3d40c49f0 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c0005bb0 0x7fb3c0005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.545+0000 7fb3d98ba640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3c0007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d90b9640 1 -- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 msgr2=0x7fb3c0007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d88b8640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c00069d0 0x7fb3c0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40c49f0 0x7fb3c0007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d90b9640 1 -- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c00069d0 msgr2=0x7fb3c0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c00069d0 0x7fb3c0006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d90b9640 1 -- 172.21.15.107:0/1707981228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c0005730 con 0x7fb3c0005bb0 2023-11-24T08:20:16.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.546+0000 7fb3d88b8640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c00069d0 0x7fb3c0006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.547+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c0005bb0 0x7fb3c0005fd0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb8e4a37e7222ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.547+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8005ad0 con 0x7fb3c0005bb0 2023-11-24T08:20:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.547+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8005c30 con 0x7fb3c0005bb0 2023-11-24T08:20:16.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.547+0000 7fb3c7fff640 1 -- 172.21.15.107:0/1707981228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c802c7c0 con 0x7fb3c0005bb0 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 -- 172.21.15.107:0/1707981228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c0005bb0 msgr2=0x7fb3c0005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1707981228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c0005bb0 0x7fb3c0005fd0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c0007a00 comp rx=0 tx=0).stop 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3d98ba640 1 -- 172.21.15.107:0/1707981228 reap_dead start 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 -- 172.21.15.107:0/1707981228 shutdown_connections 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 -- 172.21.15.107:0/1707981228 >> 172.21.15.107:0/1707981228 conn(0x7fb3d40b0c50 msgr2=0x7fb3d40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:16.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 -- 172.21.15.107:0/1707981228 shutdown_connections 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.548+0000 7fb3db31d640 1 -- 172.21.15.107:0/1707981228 wait complete. 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.549+0000 7fb3db31d640 1 Processor -- start 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.550+0000 7fb3db31d640 1 -- start start 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c0005bb0 0x7fb3d4152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40ae8e0 0x7fb3d415c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40aed80 con 0x7fb3d40b03e0 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40af060 con 0x7fb3c0005bb0 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3db31d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40af340 con 0x7fb3d40ae8e0 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56246/0 (socket says 172.21.15.107:56246) 2023-11-24T08:20:16.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 -- 172.21.15.107:0/1759202590 learned_addr learned my addr 172.21.15.107:0/1759202590 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c0005bb0 0x7fb3d4152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d98ba640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40ae8e0 0x7fb3d415c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 -- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40ae8e0 msgr2=0x7fb3d415c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40ae8e0 0x7fb3d415c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 -- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c0005bb0 msgr2=0x7fb3d4152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c0005bb0 0x7fb3d4152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.551+0000 7fb3d88b8640 1 -- 172.21.15.107:0/1759202590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d40af620 con 0x7fb3d40b03e0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3d88b8640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000d770 tx=0x7fb3d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0005270 con 0x7fb3d40b03e0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d00053d0 con 0x7fb3d40b03e0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d40af900 con 0x7fb3d40b03e0 2023-11-24T08:20:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d000b8d0 con 0x7fb3d40b03e0 2023-11-24T08:20:16.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.552+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d415ce90 con 0x7fb3d40b03e0 2023-11-24T08:20:16.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.553+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d415d110 con 0x7fb3d40b03e0 2023-11-24T08:20:16.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.554+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb3d000ba30 con 0x7fb3d40b03e0 2023-11-24T08:20:16.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.555+0000 7fb3c5ffb640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 0x7fb3b806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.555+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3d00884d0 con 0x7fb3d40b03e0 2023-11-24T08:20:16.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.556+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 0x7fb3b806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.556+0000 7fb3d90b9640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 0x7fb3b806efb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c8037040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.559+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3d0055080 con 0x7fb3d40b03e0 2023-11-24T08:20:16.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.749+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3d404e8b0 con 0x7fb3d40b03e0 2023-11-24T08:20:16.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.750+0000 7fb3c5ffb640 1 -- 172.21.15.107:0/1759202590 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v179) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb3d0019310 con 0x7fb3d40b03e0 2023-11-24T08:20:16.752 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.752+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 msgr2=0x7fb3b806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.752+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 0x7fb3b806efb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c8037040 comp rx=0 tx=0).stop 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.752+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 msgr2=0x7fb3d40ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.752+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000d770 tx=0x7fb3d000dc30 comp rx=0 tx=0).stop 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 shutdown_connections 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb3b806caf0 0x7fb3b806efb0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3d40ae8e0 0x7fb3d415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3d40b03e0 0x7fb3d40ae3a0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 --2- 172.21.15.107:0/1759202590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c0005bb0 0x7fb3d4152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 >> 172.21.15.107:0/1759202590 conn(0x7fb3d40b0c50 msgr2=0x7fb3d40ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 shutdown_connections 2023-11-24T08:20:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.753+0000 7fb3db31d640 1 -- 172.21.15.107:0/1759202590 wait complete. 2023-11-24T08:20:16.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:20:16.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.977+0000 7f017f290640 1 Processor -- start 2023-11-24T08:20:16.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.977+0000 7f017f290640 1 -- start start 2023-11-24T08:20:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017f290640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f01780cc0f0 0x7f01780c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017f290640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017815eea0 con 0x7f01780c57e0 2023-11-24T08:20:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017f290640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780abf30 con 0x7f017815b440 2023-11-24T08:20:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017f290640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017815b820 con 0x7f01780cc0f0 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017d82d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f01780c57e0 0x7f017806c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38418/0 (socket says 172.21.15.107:38418) 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.979+0000 7f017d82d640 1 -- 172.21.15.107:0/3964703272 learned_addr learned my addr 172.21.15.107:0/3964703272 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f017c82b640 1 --2- 172.21.15.107:0/3964703272 >> v2:172.21.15.120:3300/0 conn(0x7f01780cc0f0 0x7f01780c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482435397 0 0) 0x7f017815eea0 con 0x7f01780c57e0 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0164003520 con 0x7f01780c57e0 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338036411 0 0) 0x7f0164003520 con 0x7f01780c57e0 2023-11-24T08:20:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 >> v2:172.21.15.120:3300/0 conn(0x7f01780cc0f0 msgr2=0x7f01780c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 --2- 172.21.15.107:0/3964703272 >> v2:172.21.15.120:3300/0 conn(0x7f01780cc0f0 0x7f01780c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 >> v1:172.21.15.120:6789/0 conn(0x7f017815b440 legacy=0x7f01780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017815f0f0 con 0x7f01780c57e0 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f01700019f0 con 0x7f01780c57e0 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 >> v1:172.21.15.107:6789/0 conn(0x7f01780c57e0 legacy=0x7f017806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 0x7f0164003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01640047c0 0x7f0164004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164005180 0x7f016400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017815b820 con 0x7f0164003970 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016400c120 con 0x7f0164005180 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.980+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016400c2e0 con 0x7f01640047c0 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 0x7f0164003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d82d640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164005180 0x7f016400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 -- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01640047c0 msgr2=0x7f0164004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01640047c0 0x7f0164004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 -- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164005180 msgr2=0x7f016400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164005180 0x7f016400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d02c640 1 -- 172.21.15.107:0/3964703272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0164003520 con 0x7f0164003970 2023-11-24T08:20:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d82d640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164005180 0x7f016400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.981+0000 7f017d82d640 1 -- 172.21.15.107:0/3964703272 reap_dead start 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.982+0000 7f017d02c640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 0x7f0164003d90 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0174001750 tx=0x7f01640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c449ca9082bdb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.982+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01740052d0 con 0x7f0164003970 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.982+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017402e050 con 0x7f0164003970 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.982+0000 7f016ffff640 1 -- 172.21.15.107:0/3964703272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174003830 con 0x7f0164003970 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 -- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 msgr2=0x7f0164003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 0x7f0164003d90 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0174001750 tx=0x7f01640057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 -- 172.21.15.107:0/3964703272 shutdown_connections 2023-11-24T08:20:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 --2- 172.21.15.107:0/3964703272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164003970 0x7f0164003d90 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 -- 172.21.15.107:0/3964703272 >> 172.21.15.107:0/3964703272 conn(0x7f01780b0c50 msgr2=0x7f01780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 -- 172.21.15.107:0/3964703272 shutdown_connections 2023-11-24T08:20:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.983+0000 7f017f290640 1 -- 172.21.15.107:0/3964703272 wait complete. 2023-11-24T08:20:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.985+0000 7f017f290640 1 Processor -- start 2023-11-24T08:20:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.985+0000 7f017f290640 1 -- start start 2023-11-24T08:20:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164003970 0x7f01780ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01780b3e80 0x7f01780b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01780b4840 con 0x7f0178156b10 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01780c8420 con 0x7f0164003970 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017f290640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0178156f90 con 0x7f01780b3e80 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017c82b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017c82b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56268/0 (socket says 172.21.15.107:56268) 2023-11-24T08:20:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.986+0000 7f017c82b640 1 -- 172.21.15.107:0/2534027686 learned_addr learned my addr 172.21.15.107:0/2534027686 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 -- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01780b3e80 msgr2=0x7f01780b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01780b3e80 0x7f01780b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 -- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164003970 msgr2=0x7f01780ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164003970 0x7f01780ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 -- 172.21.15.107:0/2534027686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780c7810 con 0x7f0178156b10 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017c82b640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f016800e890 tx=0x7f016800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800c940 con 0x7f0178156b10 2023-11-24T08:20:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016800cac0 con 0x7f0178156b10 2023-11-24T08:20:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800ce70 con 0x7f0178156b10 2023-11-24T08:20:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01780c7af0 con 0x7f0178156b10 2023-11-24T08:20:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.987+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01780c80b0 con 0x7f0178156b10 2023-11-24T08:20:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.988+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017804e8b0 con 0x7f0178156b10 2023-11-24T08:20:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.990+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f0168013470 con 0x7f0178156b10 2023-11-24T08:20:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.990+0000 7f016dffb640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 0x7f015c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.990+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0168088560 con 0x7f0178156b10 2023-11-24T08:20:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.991+0000 7f017d02c640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 0x7f015c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.991+0000 7f017d02c640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 0x7f015c06efe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0174033000 tx=0x7f017402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:16.995+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0168055110 con 0x7f0178156b10 2023-11-24T08:20:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.189+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f017806dc80 con 0x7f0178156b10 2023-11-24T08:20:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.190+0000 7f016dffb640 1 -- 172.21.15.107:0/2534027686 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v179) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0168058730 con 0x7f0178156b10 2023-11-24T08:20:17.192 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:20:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 msgr2=0x7f015c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 0x7f015c06efe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0174033000 tx=0x7f017402f000 comp rx=0 tx=0).stop 2023-11-24T08:20:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 msgr2=0x7f01780c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f016800e890 tx=0x7f016800ed50 comp rx=0 tx=0).stop 2023-11-24T08:20:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 shutdown_connections 2023-11-24T08:20:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f015c06cb20 0x7f015c06efe0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01780b3e80 0x7f01780b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178156b10 0x7f01780c6fa0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 --2- 172.21.15.107:0/2534027686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0164003970 0x7f01780ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.193+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 >> 172.21.15.107:0/2534027686 conn(0x7f01780b0c50 msgr2=0x7f01780ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.194+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 shutdown_connections 2023-11-24T08:20:17.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.194+0000 7f017f290640 1 -- 172.21.15.107:0/2534027686 wait complete. 2023-11-24T08:20:17.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:17.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.429+0000 7f8793fc9640 1 Processor -- start 2023-11-24T08:20:17.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.429+0000 7f8793fc9640 1 -- start start 2023-11-24T08:20:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.430+0000 7f8793fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c15c7f0 0x7f878c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.430+0000 7f8793fc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f878c0ce6c0 0x7f878c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.430+0000 7f8793fc9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0cece0 con 0x7f878c0cf450 2023-11-24T08:20:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.430+0000 7f8793fc9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c156010 con 0x7f878c0ce6c0 2023-11-24T08:20:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.430+0000 7f8793fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c06ce30 con 0x7f878c15c7f0 2023-11-24T08:20:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f878c0ce6c0 0x7f878c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f878c0ce6c0 0x7f878c0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56274/0 (socket says 172.21.15.107:56274) 2023-11-24T08:20:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 -- 172.21.15.107:0/3087796728 learned_addr learned my addr 172.21.15.107:0/3087796728 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8791d65640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c15c7f0 0x7f878c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 -- 172.21.15.107:0/3087796728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c15c7f0 msgr2=0x7f878c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:17.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c15c7f0 0x7f878c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 -- 172.21.15.107:0/3087796728 >> v1:172.21.15.120:6789/0 conn(0x7f878c0cf450 legacy=0x7f878c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:17.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.431+0000 7f8792566640 1 -- 172.21.15.107:0/3087796728 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c06d040 con 0x7f878c0ce6c0 2023-11-24T08:20:17.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.432+0000 7f8792566640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c0ce6c0 0x7f878c0cf8d0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f8784009e50 tx=0x7f878402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54742af35b90621d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.432+0000 7f8790d63640 1 -- 172.21.15.107:0/3087796728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878402ca70 con 0x7f878c0ce6c0 2023-11-24T08:20:17.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8790d63640 1 -- 172.21.15.107:0/3087796728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878402cbd0 con 0x7f878c0ce6c0 2023-11-24T08:20:17.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8790d63640 1 -- 172.21.15.107:0/3087796728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87840343f0 con 0x7f878c0ce6c0 2023-11-24T08:20:17.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 -- 172.21.15.107:0/3087796728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c0ce6c0 msgr2=0x7f878c0cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c0ce6c0 0x7f878c0cf8d0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f8784009e50 tx=0x7f878402bed0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 -- 172.21.15.107:0/3087796728 shutdown_connections 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c0ce6c0 0x7f878c0cf8d0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 --2- 172.21.15.107:0/3087796728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c15c7f0 0x7f878c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 -- 172.21.15.107:0/3087796728 >> 172.21.15.107:0/3087796728 conn(0x7f878c06a920 msgr2=0x7f878c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 -- 172.21.15.107:0/3087796728 shutdown_connections 2023-11-24T08:20:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.433+0000 7f8793fc9640 1 -- 172.21.15.107:0/3087796728 wait complete. 2023-11-24T08:20:17.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.435+0000 7f8793fc9640 1 Processor -- start 2023-11-24T08:20:17.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.435+0000 7f8793fc9640 1 -- start start 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 0x7f878c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c0cf450 0x7f878c1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c1f4240 con 0x7f878c15c7f0 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b2860 con 0x7f878c0ce6c0 2023-11-24T08:20:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.436+0000 7f8793fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b3f10 con 0x7f878c0cf450 2023-11-24T08:20:17.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56290/0 (socket says 172.21.15.107:56290) 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 -- 172.21.15.107:0/2575111100 learned_addr learned my addr 172.21.15.107:0/2575111100 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8791d65640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 0x7f878c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8791564640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c0cf450 0x7f878c1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 -- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c0cf450 msgr2=0x7f878c1f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c0cf450 0x7f878c1f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 -- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 msgr2=0x7f878c153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 0x7f878c153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8792566640 1 -- 172.21.15.107:0/2575111100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009b30 con 0x7f878c15c7f0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.437+0000 7f8791d65640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 0x7f878c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f8792566640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f8784009f70 tx=0x7f8784004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784009510 con 0x7f878c15c7f0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878403e450 con 0x7f878c15c7f0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c156680 con 0x7f878c15c7f0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878403e800 con 0x7f878c15c7f0 2023-11-24T08:20:17.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.438+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c0b24f0 con 0x7f878c15c7f0 2023-11-24T08:20:17.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.439+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c0b29f0 con 0x7f878c15c7f0 2023-11-24T08:20:17.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.440+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f878403e960 con 0x7f878c15c7f0 2023-11-24T08:20:17.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.441+0000 7f87827fc640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 0x7f877006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.441+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f87840ab5b0 con 0x7f878c15c7f0 2023-11-24T08:20:17.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.441+0000 7f8791d65640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 0x7f877006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.442+0000 7f8791d65640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 0x7f877006efb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.445+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8784078280 con 0x7f878c15c7f0 2023-11-24T08:20:17.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.675+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f878c06db20 con 0x7f878c15c7f0 2023-11-24T08:20:17.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.677+0000 7f87827fc640 1 -- 172.21.15.107:0/2575111100 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f878407b8a0 con 0x7f878c15c7f0 2023-11-24T08:20:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:17.679 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:20:17.680 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-11-24T08:20:17.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.679+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 msgr2=0x7f877006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 0x7f877006efb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788008040 comp rx=0 tx=0).stop 2023-11-24T08:20:17.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 msgr2=0x7f878c0718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f8784009f70 tx=0x7f8784004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 shutdown_connections 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f877006caf0 0x7f877006efb0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f878c15c7f0 0x7f878c0718b0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f878c0cf450 0x7f878c1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 --2- 172.21.15.107:0/2575111100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f878c0ce6c0 0x7f878c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 >> 172.21.15.107:0/2575111100 conn(0x7f878c06a920 msgr2=0x7f878c0cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:17.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 shutdown_connections 2023-11-24T08:20:17.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.680+0000 7f8793fc9640 1 -- 172.21.15.107:0/2575111100 wait complete. 2023-11-24T08:20:17.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:17.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.906+0000 7f7b4be46640 1 Processor -- start 2023-11-24T08:20:17.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.906+0000 7f7b4be46640 1 -- start start 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.907+0000 7f7b4be46640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b4415c7f0 0x7f7b4406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b4be46640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 0x7f7b440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b4be46640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440cece0 con 0x7f7b440cf450 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b4be46640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b440cfe40 con 0x7f7b4415c7f0 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b4be46640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44156010 con 0x7f7b440ce6c0 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b49be2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b4415c7f0 0x7f7b4406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.908+0000 7f7b4a3e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 0x7f7b440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b49be2640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b4415c7f0 0x7f7b4406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:36472/0 (socket says 172.21.15.107:36472) 2023-11-24T08:20:17.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b49be2640 1 -- 172.21.15.107:0/29503276 learned_addr learned my addr 172.21.15.107:0/29503276 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b4a3e3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 0x7f7b440cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40816/0 (socket says 172.21.15.107:40816) 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b493e1640 1 --1- 172.21.15.107:0/29503276 >> v1:172.21.15.120:6789/0 conn(0x7f7b440cf450 0x7f7b440cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:50726/0 (socket says 172.21.15.107:50726) 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b49be2640 1 -- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 msgr2=0x7f7b440cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b49be2640 1 --2- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 0x7f7b440cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b49be2640 1 -- 172.21.15.107:0/29503276 >> v1:172.21.15.120:6789/0 conn(0x7f7b440cf450 legacy=0x7f7b440cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.909+0000 7f7b48be0640 1 -- 172.21.15.107:0/29503276 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412839524 0 0) 0x7f7b440cece0 con 0x7f7b440cf450 2023-11-24T08:20:17.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.910+0000 7f7b49be2640 1 -- 172.21.15.107:0/29503276 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4406d040 con 0x7f7b4415c7f0 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.911+0000 7f7b49be2640 1 --2- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b4415c7f0 0x7f7b4406db20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009e50 tx=0x7f7b3802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=783f19a88c6067cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.911+0000 7f7b48be0640 1 -- 172.21.15.107:0/29503276 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802cb00 con 0x7f7b4415c7f0 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.911+0000 7f7b48be0640 1 -- 172.21.15.107:0/29503276 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3802cc60 con 0x7f7b4415c7f0 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.911+0000 7f7b48be0640 1 -- 172.21.15.107:0/29503276 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380343f0 con 0x7f7b4415c7f0 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 -- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b4415c7f0 msgr2=0x7f7b4406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 --2- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b4415c7f0 0x7f7b4406db20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009e50 tx=0x7f7b3802bf60 comp rx=0 tx=0).stop 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 -- 172.21.15.107:0/29503276 shutdown_connections 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 --2- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440ce6c0 0x7f7b440cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 --2- 172.21.15.107:0/29503276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b4415c7f0 0x7f7b4406db20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 -- 172.21.15.107:0/29503276 >> 172.21.15.107:0/29503276 conn(0x7f7b4406a920 msgr2=0x7f7b4406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 -- 172.21.15.107:0/29503276 shutdown_connections 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.912+0000 7f7b4be46640 1 -- 172.21.15.107:0/29503276 wait complete. 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.914+0000 7f7b4be46640 1 Processor -- start 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.914+0000 7f7b4be46640 1 -- start start 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b440ce6c0 0x7f7b44144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440cf450 0x7f7b44147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44148330 con 0x7f7b4415c7f0 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441564f0 con 0x7f7b440ce6c0 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4be46640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44157f60 con 0x7f7b440cf450 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4a3e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4a3e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40464/0 (socket says 172.21.15.107:40464) 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.915+0000 7f7b4a3e3640 1 -- 172.21.15.107:0/1839487995 learned_addr learned my addr 172.21.15.107:0/1839487995 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b49be2640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b440ce6c0 0x7f7b44144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b493e1640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440cf450 0x7f7b44147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 -- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440cf450 msgr2=0x7f7b44147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440cf450 0x7f7b44147df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 -- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b440ce6c0 msgr2=0x7f7b44144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b440ce6c0 0x7f7b44144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 -- 172.21.15.107:0/1839487995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38009b30 con 0x7f7b4415c7f0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4a3e3640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e860 tx=0x7f7b3c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00c6d0 con 0x7f7b4415c7f0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c00c830 con 0x7f7b4415c7f0 2023-11-24T08:20:17.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00cbe0 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b44158240 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.916+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44156150 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.918+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44157dc0 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.919+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f7b3c013470 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.919+0000 7f7b427fc640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 0x7f7b2806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.920+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7b3c0889b0 con 0x7f7b4415c7f0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.920+0000 7f7b49be2640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 0x7f7b2806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.920+0000 7f7b49be2640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 0x7f7b2806efb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7b38035000 tx=0x7f7b38030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:17.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:17.924+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b44157dc0 con 0x7f7b4415c7f0 2023-11-24T08:20:18.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.115+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b44157dc0 con 0x7f7b4415c7f0 2023-11-24T08:20:18.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.116+0000 7f7b427fc640 1 -- 172.21.15.107:0/1839487995 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16688 (secure 0 0 0) 0x7f7b44157dc0 con 0x7f7b4415c7f0 2023-11-24T08:20:18.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:18.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":179,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:15.077624+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":178,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":2}]},{"pgid":"25.17","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":25,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:18.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 msgr2=0x7f7b2806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 0x7f7b2806efb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7b38035000 tx=0x7f7b38030000 comp rx=0 tx=0).stop 2023-11-24T08:20:18.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 msgr2=0x7f7b44145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e860 tx=0x7f7b3c00ed20 comp rx=0 tx=0).stop 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 shutdown_connections 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2806caf0 0x7f7b2806efb0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.119+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b4415c7f0 0x7f7b44145350 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.120+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b440cf450 0x7f7b44147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.120+0000 7f7b4be46640 1 --2- 172.21.15.107:0/1839487995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b440ce6c0 0x7f7b44144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.120+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 >> 172.21.15.107:0/1839487995 conn(0x7f7b4406a920 msgr2=0x7f7b44155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.120+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 shutdown_connections 2023-11-24T08:20:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.120+0000 7f7b4be46640 1 -- 172.21.15.107:0/1839487995 wait complete. 2023-11-24T08:20:18.139 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:20:18.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.348+0000 7fb481cbb640 1 Processor -- start 2023-11-24T08:20:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.348+0000 7fb481cbb640 1 -- start start 2023-11-24T08:20:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.349+0000 7fb481cbb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb47c144b30 0x7fb47c15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.349+0000 7fb481cbb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c15bf80 con 0x7fb47c0b1de0 2023-11-24T08:20:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.349+0000 7fb481cbb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0b1560 con 0x7fb47c15c410 2023-11-24T08:20:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.349+0000 7fb481cbb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0b1280 con 0x7fb47c144b30 2023-11-24T08:20:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.350+0000 7fb4814ba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb47c0b1de0 0x7fb47c0b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43724/0 (socket says 172.21.15.107:43724) 2023-11-24T08:20:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.350+0000 7fb4814ba640 1 -- 172.21.15.107:0/789639465 learned_addr learned my addr 172.21.15.107:0/789639465 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.350+0000 7fb480cb9640 1 --2- 172.21.15.107:0/789639465 >> v2:172.21.15.120:3300/0 conn(0x7fb47c144b30 0x7fb47c15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.350+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815012776 0 0) 0x7fb47c15bf80 con 0x7fb47c0b1de0 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.350+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c003520 con 0x7fb47c0b1de0 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741727488 0 0) 0x7fb46c003520 con 0x7fb47c0b1de0 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 >> v2:172.21.15.120:3300/0 conn(0x7fb47c144b30 msgr2=0x7fb47c15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 --2- 172.21.15.107:0/789639465 >> v2:172.21.15.120:3300/0 conn(0x7fb47c144b30 0x7fb47c15f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 >> v1:172.21.15.120:6789/0 conn(0x7fb47c15c410 legacy=0x7fb47c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c15b930 con 0x7fb47c0b1de0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb4740019f0 con 0x7fb47c0b1de0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 >> v1:172.21.15.107:6789/0 conn(0x7fb47c0b1de0 legacy=0x7fb47c0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0039c0 0x7fb46c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 0x7fb46c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb46c005180 0x7fb46c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b1280 con 0x7fb46c0047c0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c080 con 0x7fb46c0039c0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c240 con 0x7fb46c005180 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb47bfff640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 0x7fb46c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.351+0000 7fb480cb9640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0039c0 0x7fb46c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb4814ba640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb46c005180 0x7fb46c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb47bfff640 1 -- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb46c005180 msgr2=0x7fb46c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb47bfff640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb46c005180 0x7fb46c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb47bfff640 1 -- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0039c0 msgr2=0x7fb46c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb47bfff640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0039c0 0x7fb46c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb47bfff640 1 -- 172.21.15.107:0/789639465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c003520 con 0x7fb46c0047c0 2023-11-24T08:20:18.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.352+0000 7fb4814ba640 1 -- 172.21.15.107:0/789639465 reap_dead start 2023-11-24T08:20:18.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.353+0000 7fb47bfff640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 0x7fb46c004c40 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb468001750 tx=0x7fb46c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e7551ded6f03d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.353+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4680052d0 con 0x7fb46c0047c0 2023-11-24T08:20:18.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.353+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46802e050 con 0x7fb46c0047c0 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.353+0000 7fb47b7fe640 1 -- 172.21.15.107:0/789639465 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468003830 con 0x7fb46c0047c0 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 -- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 msgr2=0x7fb46c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 0x7fb46c004c40 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb468001750 tx=0x7fb46c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 -- 172.21.15.107:0/789639465 shutdown_connections 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 --2- 172.21.15.107:0/789639465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb46c0047c0 0x7fb46c004c40 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 -- 172.21.15.107:0/789639465 >> 172.21.15.107:0/789639465 conn(0x7fb47c0cb830 msgr2=0x7fb47c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 -- 172.21.15.107:0/789639465 shutdown_connections 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.354+0000 7fb481cbb640 1 -- 172.21.15.107:0/789639465 wait complete. 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.356+0000 7fb481cbb640 1 Processor -- start 2023-11-24T08:20:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.356+0000 7fb481cbb640 1 -- start start 2023-11-24T08:20:18.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0047c0 0x7fb47c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47c0c4af0 0x7fb47c0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c142470 con 0x7fb47c141ad0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0c3d50 con 0x7fb46c0047c0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb481cbb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c141790 con 0x7fb47c0c4af0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40496/0 (socket says 172.21.15.107:40496) 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 -- 172.21.15.107:0/1269618216 learned_addr learned my addr 172.21.15.107:0/1269618216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb480cb9640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0047c0 0x7fb47c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb47bfff640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47c0c4af0 0x7fb47c0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 -- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47c0c4af0 msgr2=0x7fb47c0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47c0c4af0 0x7fb47c0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 -- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0047c0 msgr2=0x7fb47c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.357+0000 7fb4814ba640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0047c0 0x7fb47c0714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb4814ba640 1 -- 172.21.15.107:0/1269618216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0c4010 con 0x7fb47c141ad0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb4814ba640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0b50a0 tx=0x7fb474007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47400eb00 con 0x7fb47c141ad0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47400eca0 con 0x7fb47c141ad0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0c42f0 con 0x7fb47c141ad0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474013420 con 0x7fb47c141ad0 2023-11-24T08:20:18.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.358+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c071a10 con 0x7fb47c141ad0 2023-11-24T08:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.359+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0c4810 con 0x7fb47c141ad0 2023-11-24T08:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.360+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb474013580 con 0x7fb47c141ad0 2023-11-24T08:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.361+0000 7fb4797fa640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 0x7fb46006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.361+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb4740889d0 con 0x7fb47c141ad0 2023-11-24T08:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.361+0000 7fb480cb9640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 0x7fb46006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.362+0000 7fb480cb9640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 0x7fb46006ef40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb47c064320 tx=0x7fb470008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.365+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb47c0c4810 con 0x7fb47c141ad0 2023-11-24T08:20:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.595+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47c0c4810 con 0x7fb47c141ad0 2023-11-24T08:20:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.596+0000 7fb4797fa640 1 -- 172.21.15.107:0/1269618216 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4938 (secure 0 0 0) 0x7fb47c0c4810 con 0x7fb47c141ad0 2023-11-24T08:20:18.598 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:18.599 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:20:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.598+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 msgr2=0x7fb46006ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.598+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 0x7fb46006ef40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb47c064320 tx=0x7fb470008040 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.598+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 msgr2=0x7fb47c141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.598+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0b50a0 tx=0x7fb474007bc0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 shutdown_connections 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb46006ca80 0x7fb46006ef40 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb47c141ad0 0x7fb47c141f30 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47c0c4af0 0x7fb47c0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 --2- 172.21.15.107:0/1269618216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb46c0047c0 0x7fb47c0714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 >> 172.21.15.107:0/1269618216 conn(0x7fb47c0cb830 msgr2=0x7fb47c15c820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 shutdown_connections 2023-11-24T08:20:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.599+0000 7fb481cbb640 1 -- 172.21.15.107:0/1269618216 wait complete. 2023-11-24T08:20:18.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.825+0000 7f5c3a6bb640 1 Processor -- start 2023-11-24T08:20:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.825+0000 7f5c3a6bb640 1 -- start start 2023-11-24T08:20:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.826+0000 7f5c3a6bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 0x7f5c3406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c3a6bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c340cdbf0 0x7f5c340ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c3a6bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3406e060 con 0x7f5c3413d120 2023-11-24T08:20:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c3a6bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3413f9e0 con 0x7f5c340cdbf0 2023-11-24T08:20:18.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c3a6bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3413fcc0 con 0x7f5c3406d6c0 2023-11-24T08:20:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c396b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c3413d120 0x7f5c3415af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43740/0 (socket says 172.21.15.107:43740) 2023-11-24T08:20:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c396b9640 1 -- 172.21.15.107:0/47989494 learned_addr learned my addr 172.21.15.107:0/47989494 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c39eba640 1 --2- 172.21.15.107:0/47989494 >> v2:172.21.15.107:3300/0 conn(0x7f5c340cdbf0 0x7f5c340ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c38eb8640 1 --2- 172.21.15.107:0/47989494 >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 0x7f5c3406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c39eba640 1 -- 172.21.15.107:0/47989494 >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 msgr2=0x7f5c3406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c2bfff640 1 -- 172.21.15.107:0/47989494 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821211038 0 0) 0x7f5c3406e060 con 0x7f5c3413d120 2023-11-24T08:20:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c39eba640 1 --2- 172.21.15.107:0/47989494 >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 0x7f5c3406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c39eba640 1 -- 172.21.15.107:0/47989494 >> v1:172.21.15.107:6789/0 conn(0x7f5c3413d120 legacy=0x7f5c3415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.827+0000 7f5c39eba640 1 -- 172.21.15.107:0/47989494 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3413ffa0 con 0x7f5c340cdbf0 2023-11-24T08:20:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.828+0000 7f5c38eb8640 1 --2- 172.21.15.107:0/47989494 >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 0x7f5c3406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c39eba640 1 --2- 172.21.15.107:0/47989494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c340cdbf0 0x7f5c340ce020 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f5c34072a70 tx=0x7f5c2c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b057d518eb72264d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c2bfff640 1 -- 172.21.15.107:0/47989494 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c02cbe0 con 0x7f5c340cdbf0 2023-11-24T08:20:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c2bfff640 1 -- 172.21.15.107:0/47989494 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c02cd40 con 0x7f5c340cdbf0 2023-11-24T08:20:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c2bfff640 1 -- 172.21.15.107:0/47989494 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c034660 con 0x7f5c340cdbf0 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/47989494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c340cdbf0 msgr2=0x7f5c340ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/47989494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c340cdbf0 0x7f5c340ce020 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f5c34072a70 tx=0x7f5c2c02be30 comp rx=0 tx=0).stop 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/47989494 shutdown_connections 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/47989494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c340cdbf0 0x7f5c340ce020 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/47989494 >> v2:172.21.15.120:3300/0 conn(0x7f5c3406d6c0 0x7f5c3406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.829+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/47989494 >> 172.21.15.107:0/47989494 conn(0x7f5c340cb830 msgr2=0x7f5c340cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.830+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/47989494 shutdown_connections 2023-11-24T08:20:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.830+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/47989494 wait complete. 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.831+0000 7f5c3a6bb640 1 Processor -- start 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.831+0000 7f5c3a6bb640 1 -- start start 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c3406d6c0 0x7f5c34138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c340cdbf0 0x7f5c34139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34139040 con 0x7f5c3413d120 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340afe10 con 0x7f5c340cdbf0 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.832+0000 7f5c3a6bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3413a930 con 0x7f5c3406d6c0 2023-11-24T08:20:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40522/0 (socket says 172.21.15.107:40522) 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 -- 172.21.15.107:0/2690748538 learned_addr learned my addr 172.21.15.107:0/2690748538 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 -- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c3406d6c0 msgr2=0x7f5c34138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c3406d6c0 0x7f5c34138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 -- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c340cdbf0 msgr2=0x7f5c34139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c340cdbf0 0x7f5c34139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 -- 172.21.15.107:0/2690748538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c009a70 con 0x7f5c3413d120 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.833+0000 7f5c39eba640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009e80 tx=0x7f5c2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.834+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c040650 con 0x7f5c3413d120 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.834+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c0407d0 con 0x7f5c3413d120 2023-11-24T08:20:18.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.834+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c040b80 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.834+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3415a9d0 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.834+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3413aef0 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.835+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3413b0e0 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.836+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f5c2c040ce0 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.837+0000 7f5c29ffb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 0x7f5c1806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.837+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5c2c0ab4a0 con 0x7f5c3413d120 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.837+0000 7f5c396b9640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 0x7f5c1806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:18.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.838+0000 7f5c396b9640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 0x7f5c1806efb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5c30007fd0 tx=0x7f5c30007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:18.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:18.841+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c2c078050 con 0x7f5c3413d120 2023-11-24T08:20:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.070+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c3404e840 con 0x7f5c3413d120 2023-11-24T08:20:19.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.071+0000 7f5c29ffb640 1 -- 172.21.15.107:0/2690748538 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4938 (secure 0 0 0) 0x7f5c2c07b670 con 0x7f5c3413d120 2023-11-24T08:20:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:19.074 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:19:21.993451+0000","modified":"2023-11-24T08:19:25.082775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10410":{"gid":10410,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6836/3208934992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3208934992},{"type":"v1","addr":"172.21.15.107:6836","nonce":3208934992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-24T08:20:19.074 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-11-24T08:20:19.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 msgr2=0x7f5c1806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 0x7f5c1806efb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5c30007fd0 tx=0x7f5c30007960 comp rx=0 tx=0).stop 2023-11-24T08:20:19.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 msgr2=0x7f5c34159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009e80 tx=0x7f5c2c004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:19.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 shutdown_connections 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c1806caf0 0x7f5c1806efb0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c3413d120 0x7f5c34159070 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c340cdbf0 0x7f5c34139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 --2- 172.21.15.107:0/2690748538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c3406d6c0 0x7f5c34138b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 >> 172.21.15.107:0/2690748538 conn(0x7f5c340cb830 msgr2=0x7f5c340cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 shutdown_connections 2023-11-24T08:20:19.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.074+0000 7f5c3a6bb640 1 -- 172.21.15.107:0/2690748538 wait complete. 2023-11-24T08:20:19.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.296+0000 7f45a2978640 1 Processor -- start 2023-11-24T08:20:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.297+0000 7f45a2978640 1 -- start start 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.298+0000 7f45a2978640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f459c0cc470 0x7f459c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.298+0000 7f45a2978640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f459c0cfb60 0x7f459c0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.298+0000 7f45a2978640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0ce270 con 0x7f459c0ce640 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.298+0000 7f45a2978640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c06c560 con 0x7f459c0cc470 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.298+0000 7f45a2978640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c06d0a0 con 0x7f459c0cfb60 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f459c0cc470 0x7f459c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f459c0cc470 0x7f459c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40532/0 (socket says 172.21.15.107:40532) 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 -- 172.21.15.107:0/2596207819 learned_addr learned my addr 172.21.15.107:0/2596207819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 -- 172.21.15.107:0/2596207819 >> v2:172.21.15.120:3300/0 conn(0x7f459c0cfb60 msgr2=0x7f459c0b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 --2- 172.21.15.107:0/2596207819 >> v2:172.21.15.120:3300/0 conn(0x7f459c0cfb60 0x7f459c0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 -- 172.21.15.107:0/2596207819 >> v1:172.21.15.120:6789/0 conn(0x7f459c0ce640 legacy=0x7f459c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.299+0000 7f459bfff640 1 -- 172.21.15.107:0/2596207819 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0b4200 con 0x7f459c0cc470 2023-11-24T08:20:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.300+0000 7f459bfff640 1 --2- 172.21.15.107:0/2596207819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cc470 0x7f459c0abfc0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f459c0b2170 tx=0x7f458c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c342906984ef23 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.300+0000 7f459affd640 1 -- 172.21.15.107:0/2596207819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c02ca80 con 0x7f459c0cc470 2023-11-24T08:20:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.300+0000 7f459affd640 1 -- 172.21.15.107:0/2596207819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c02cbe0 con 0x7f459c0cc470 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.300+0000 7f459affd640 1 -- 172.21.15.107:0/2596207819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0343f0 con 0x7f459c0cc470 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 -- 172.21.15.107:0/2596207819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cc470 msgr2=0x7f459c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 --2- 172.21.15.107:0/2596207819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cc470 0x7f459c0abfc0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f459c0b2170 tx=0x7f458c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 -- 172.21.15.107:0/2596207819 shutdown_connections 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 --2- 172.21.15.107:0/2596207819 >> v2:172.21.15.120:3300/0 conn(0x7f459c0cfb60 0x7f459c0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 --2- 172.21.15.107:0/2596207819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cc470 0x7f459c0abfc0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 -- 172.21.15.107:0/2596207819 >> 172.21.15.107:0/2596207819 conn(0x7f459c06a920 msgr2=0x7f459c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 -- 172.21.15.107:0/2596207819 shutdown_connections 2023-11-24T08:20:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.301+0000 7f45a2978640 1 -- 172.21.15.107:0/2596207819 wait complete. 2023-11-24T08:20:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.303+0000 7f45a2978640 1 Processor -- start 2023-11-24T08:20:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.303+0000 7f45a2978640 1 -- start start 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f459c0cc470 0x7f459c1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 0x7f459c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c15fe50 con 0x7f459c0cfb60 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c150d20 con 0x7f459c0ce640 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a2978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c15f1d0 con 0x7f459c0cc470 2023-11-24T08:20:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a0f15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a0f15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40534/0 (socket says 172.21.15.107:40534) 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.304+0000 7f45a0f15640 1 -- 172.21.15.107:0/10106681 learned_addr learned my addr 172.21.15.107:0/10106681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 -- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f459c0cc470 msgr2=0x7f459c1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f459c0cc470 0x7f459c1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 -- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 msgr2=0x7f459c157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f459b7fe640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 0x7f459c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 0x7f459c157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 -- 172.21.15.107:0/10106681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c009b30 con 0x7f459c0cfb60 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f459b7fe640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 0x7f459c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a0f15640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f459400b410 tx=0x7f459400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594011830 con 0x7f459c0cfb60 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45940119b0 con 0x7f459c0cfb60 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594011d60 con 0x7f459c0cfb60 2023-11-24T08:20:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c15f4b0 con 0x7f459c0cfb60 2023-11-24T08:20:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.305+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c15fa70 con 0x7f459c0cfb60 2023-11-24T08:20:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.306+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c15fc30 con 0x7f459c0cfb60 2023-11-24T08:20:19.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.307+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f459401a400 con 0x7f459c0cfb60 2023-11-24T08:20:19.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.309+0000 7f4598ff9640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 0x7f457c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.309+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4594088670 con 0x7f459c0cfb60 2023-11-24T08:20:19.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.309+0000 7f459bfff640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 0x7f457c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.310+0000 7f459bfff640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 0x7f457c06efb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f458c035000 tx=0x7f458c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.313+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4594055180 con 0x7f459c0cfb60 2023-11-24T08:20:19.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.505+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f459c15ff90 con 0x7f459c0cfb60 2023-11-24T08:20:19.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.508+0000 7f4598ff9640 1 -- 172.21.15.107:0/10106681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16688 (secure 0 0 0) 0x7f459c15ff90 con 0x7f459c0cfb60 2023-11-24T08:20:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:19.510 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":179,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:15.077624+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:19:19.824499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-11-24T08:19:21.010791+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":178,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":2}]},{"pgid":"25.17","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":25,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 msgr2=0x7f457c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 0x7f457c06efb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f458c035000 tx=0x7f458c02e000 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 msgr2=0x7f459c160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f459400b410 tx=0x7f459400b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 shutdown_connections 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f457c06caf0 0x7f457c06efb0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c0cfb60 0x7f459c160270 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f459c0ce640 0x7f459c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 --2- 172.21.15.107:0/10106681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f459c0cc470 0x7f459c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.510+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 >> 172.21.15.107:0/10106681 conn(0x7f459c06a920 msgr2=0x7f459c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:19.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.511+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 shutdown_connections 2023-11-24T08:20:19.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.511+0000 7f45a2978640 1 -- 172.21.15.107:0/10106681 wait complete. 2023-11-24T08:20:19.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:20:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.739+0000 7ff50d701640 1 Processor -- start 2023-11-24T08:20:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.739+0000 7ff50d701640 1 -- start start 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff50d701640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff50815b440 0x7ff5080cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff50d701640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5080cbf20 0x7ff5080c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff50d701640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50815eea0 con 0x7ff5080c57e0 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff50d701640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5080cc3d0 con 0x7ff5080cbf20 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff50d701640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50815b820 con 0x7ff50815b440 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff5067fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5080cbf20 0x7ff5080c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff5067fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5080cbf20 0x7ff5080c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40538/0 (socket says 172.21.15.107:40538) 2023-11-24T08:20:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff5067fc640 1 -- 172.21.15.107:0/4222093172 learned_addr learned my addr 172.21.15.107:0/4222093172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:19.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.741+0000 7ff506ffd640 1 --2- 172.21.15.107:0/4222093172 >> v2:172.21.15.120:3300/0 conn(0x7ff50815b440 0x7ff5080cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.742+0000 7ff5067fc640 1 -- 172.21.15.107:0/4222093172 >> v2:172.21.15.120:3300/0 conn(0x7ff50815b440 msgr2=0x7ff5080cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.742+0000 7ff5067fc640 1 --2- 172.21.15.107:0/4222093172 >> v2:172.21.15.120:3300/0 conn(0x7ff50815b440 0x7ff5080cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.742+0000 7ff5067fc640 1 -- 172.21.15.107:0/4222093172 >> v1:172.21.15.120:6789/0 conn(0x7ff5080c57e0 legacy=0x7ff5080c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.742+0000 7ff5067fc640 1 -- 172.21.15.107:0/4222093172 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50806d600 con 0x7ff5080cbf20 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.743+0000 7ff5067fc640 1 --2- 172.21.15.107:0/4222093172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080cbf20 0x7ff5080c52a0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7ff50806e770 tx=0x7ff4fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2ee1d2a31fb256 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.743+0000 7ff505ffb640 1 -- 172.21.15.107:0/4222093172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0317e0 con 0x7ff5080cbf20 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff505ffb640 1 -- 172.21.15.107:0/4222093172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc031940 con 0x7ff5080cbf20 2023-11-24T08:20:19.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff505ffb640 1 -- 172.21.15.107:0/4222093172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc031cd0 con 0x7ff5080cbf20 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 -- 172.21.15.107:0/4222093172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080cbf20 msgr2=0x7ff5080c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 --2- 172.21.15.107:0/4222093172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080cbf20 0x7ff5080c52a0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7ff50806e770 tx=0x7ff4fc02b880 comp rx=0 tx=0).stop 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 -- 172.21.15.107:0/4222093172 shutdown_connections 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 --2- 172.21.15.107:0/4222093172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080cbf20 0x7ff5080c52a0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 --2- 172.21.15.107:0/4222093172 >> v2:172.21.15.120:3300/0 conn(0x7ff50815b440 0x7ff5080cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 -- 172.21.15.107:0/4222093172 >> 172.21.15.107:0/4222093172 conn(0x7ff5080b0c50 msgr2=0x7ff5080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 -- 172.21.15.107:0/4222093172 shutdown_connections 2023-11-24T08:20:19.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.744+0000 7ff50d701640 1 -- 172.21.15.107:0/4222093172 wait complete. 2023-11-24T08:20:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.746+0000 7ff50d701640 1 Processor -- start 2023-11-24T08:20:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.746+0000 7ff50d701640 1 -- start start 2023-11-24T08:20:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 0x7ff50814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff50815b440 0x7ff5080b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff508099a70 con 0x7ff5080c57e0 2023-11-24T08:20:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff508099d50 con 0x7ff50815b440 2023-11-24T08:20:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff50d701640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50809a030 con 0x7ff5080cbf20 2023-11-24T08:20:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.747+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40550/0 (socket says 172.21.15.107:40550) 2023-11-24T08:20:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 -- 172.21.15.107:0/3315904919 learned_addr learned my addr 172.21.15.107:0/3315904919 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff5067fc640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 0x7ff50814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 -- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 msgr2=0x7ff50814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff5077fe640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff50815b440 0x7ff5080b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 0x7ff50814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 -- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff50815b440 msgr2=0x7ff5080b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff50815b440 0x7ff5080b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 -- 172.21.15.107:0/3315904919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc0094d0 con 0x7ff5080c57e0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff5067fc640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 0x7ff50814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.748+0000 7ff506ffd640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800d5f0 tx=0x7ff4f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.749+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80125f0 con 0x7ff5080c57e0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.749+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8012750 con 0x7ff5080c57e0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.749+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8012b00 con 0x7ff5080c57e0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.749+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50809a310 con 0x7ff5080c57e0 2023-11-24T08:20:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.749+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080b08b0 con 0x7ff5080c57e0 2023-11-24T08:20:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.750+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50804e8b0 con 0x7ff5080c57e0 2023-11-24T08:20:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.751+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7ff4f8012c60 con 0x7ff5080c57e0 2023-11-24T08:20:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.752+0000 7ff4ef7fe640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 0x7ff4f406f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.752+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4f8088a30 con 0x7ff5080c57e0 2023-11-24T08:20:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.752+0000 7ff5067fc640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 0x7ff4f406f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.753+0000 7ff5067fc640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 0x7ff4f406f000 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.757+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4f80555c0 con 0x7ff5080c57e0 2023-11-24T08:20:19.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.983+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff50806dc80 con 0x7ff5080c57e0 2023-11-24T08:20:19.986 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:20:19.984+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:20:19.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.988+0000 7ff4ef7fe640 1 -- 172.21.15.107:0/3315904919 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff4f8058be0 con 0x7ff5080c57e0 2023-11-24T08:20:19.990 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.991+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 msgr2=0x7ff4f406f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.991+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 0x7ff4f406f000 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03d040 comp rx=0 tx=0).stop 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 msgr2=0x7ff50809ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800d5f0 tx=0x7ff4f800dab0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 shutdown_connections 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4f406cb40 0x7ff4f406f000 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff50815b440 0x7ff5080b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5080cbf20 0x7ff50814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 --2- 172.21.15.107:0/3315904919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5080c57e0 0x7ff50809ab10 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 >> 172.21.15.107:0/3315904919 conn(0x7ff5080b0c50 msgr2=0x7ff508070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 shutdown_connections 2023-11-24T08:20:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:19.992+0000 7ff50d701640 1 -- 172.21.15.107:0/3315904919 wait complete. 2023-11-24T08:20:20.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:20:20.108 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:20:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.207+0000 7fcdf2461640 1 Processor -- start 2023-11-24T08:20:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.207+0000 7fcdf2461640 1 -- start start 2023-11-24T08:20:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdf2461640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcdec0ccbd0 0x7fcdec1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdf2461640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcdec0ccfb0 0x7fcdec162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdf2461640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec1655f0 con 0x7fcdec162540 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdf2461640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec0cc380 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdf2461640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec162b30 con 0x7fcdec0ccbd0 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdeb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcdec0ccfb0 0x7fcdec162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdeb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcdec0ccfb0 0x7fcdec162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40566/0 (socket says 172.21.15.107:40566) 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/2746294445 learned_addr learned my addr 172.21.15.107:0/2746294445 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.209+0000 7fcdebfff640 1 --2- 172.21.15.107:0/2746294445 >> v2:172.21.15.120:3300/0 conn(0x7fcdec0ccbd0 0x7fcdec1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.210+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/2746294445 >> v2:172.21.15.120:3300/0 conn(0x7fcdec0ccbd0 msgr2=0x7fcdec1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.210+0000 7fcdeb7fe640 1 --2- 172.21.15.107:0/2746294445 >> v2:172.21.15.120:3300/0 conn(0x7fcdec0ccbd0 0x7fcdec1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.210+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/2746294445 >> v1:172.21.15.120:6789/0 conn(0x7fcdec162540 legacy=0x7fcdec0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.210+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/2746294445 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec162e10 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.211+0000 7fcdeb7fe640 1 --2- 172.21.15.107:0/2746294445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec162000 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fcdec06ea50 tx=0x7fcdd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d03b7f04a339c6f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.211+0000 7fcdeaffd640 1 -- 172.21.15.107:0/2746294445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4031870 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.211+0000 7fcdeaffd640 1 -- 172.21.15.107:0/2746294445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd40319d0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.211+0000 7fcdeaffd640 1 -- 172.21.15.107:0/2746294445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4031d60 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 -- 172.21.15.107:0/2746294445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 msgr2=0x7fcdec162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 --2- 172.21.15.107:0/2746294445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec162000 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fcdec06ea50 tx=0x7fcdd402b910 comp rx=0 tx=0).stop 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 -- 172.21.15.107:0/2746294445 shutdown_connections 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 --2- 172.21.15.107:0/2746294445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec162000 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 --2- 172.21.15.107:0/2746294445 >> v2:172.21.15.120:3300/0 conn(0x7fcdec0ccbd0 0x7fcdec1660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 -- 172.21.15.107:0/2746294445 >> 172.21.15.107:0/2746294445 conn(0x7fcdec0b0c50 msgr2=0x7fcdec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:20.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 -- 172.21.15.107:0/2746294445 shutdown_connections 2023-11-24T08:20:20.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.212+0000 7fcdf2461640 1 -- 172.21.15.107:0/2746294445 wait complete. 2023-11-24T08:20:20.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.214+0000 7fcdf2461640 1 Processor -- start 2023-11-24T08:20:20.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.214+0000 7fcdf2461640 1 -- start start 2023-11-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdec0ccbd0 0x7fcdec157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdec162540 0x7fcdec1ed100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec072360 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec163b90 con 0x7fcdec0ccbd0 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdf2461640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1631f0 con 0x7fcdec162540 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40576/0 (socket says 172.21.15.107:40576) 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/1385119262 learned_addr learned my addr 172.21.15.107:0/1385119262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdebfff640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdec0ccbd0 0x7fcdec157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdec162540 msgr2=0x7fcdec1ed100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdec162540 0x7fcdec1ed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdec0ccbd0 msgr2=0x7fcdec157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdec0ccbd0 0x7fcdec157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:20.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.215+0000 7fcdeb7fe640 1 -- 172.21.15.107:0/1385119262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd40094d0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcdeb7fe640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4000c00 tx=0x7fcdd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd403ad30 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd403ae90 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4044630 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec1634d0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.216+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec163fb0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.217+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec163cd0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.218+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fcdd404d450 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.219+0000 7fcde8ff9640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 0x7fcdcc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.219+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcdd402d040 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.219+0000 7fcdebfff640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 0x7fcdcc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:20.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.220+0000 7fcdebfff640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 0x7fcdcc06efb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e90 tx=0x7fcde400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:20.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.224+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdec163cd0 con 0x7fcdec0ccfb0 2023-11-24T08:20:20.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:20.450+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcdec163cd0 con 0x7fcdec0ccfb0 2023-11-24T08:20:21.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.084+0000 7fcde8ff9640 1 -- 172.21.15.107:0/1385119262 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcdec163cd0 con 0x7fcdec0ccfb0 2023-11-24T08:20:21.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 msgr2=0x7fcdcc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 0x7fcdcc06efb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e90 tx=0x7fcde400b040 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 msgr2=0x7fcdec1ecbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4000c00 tx=0x7fcdd40046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 shutdown_connections 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcdcc06caf0 0x7fcdcc06efb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdec162540 0x7fcdec1ed100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdec0ccfb0 0x7fcdec1ecbc0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 --2- 172.21.15.107:0/1385119262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdec0ccbd0 0x7fcdec157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.087+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 >> 172.21.15.107:0/1385119262 conn(0x7fcdec0b0c50 msgr2=0x7fcdec0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.088+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 shutdown_connections 2023-11-24T08:20:21.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.088+0000 7fcdf2461640 1 -- 172.21.15.107:0/1385119262 wait complete. 2023-11-24T08:20:21.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:20:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.313+0000 7f9130b53640 1 Processor -- start 2023-11-24T08:20:21.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.313+0000 7f9130b53640 1 -- start start 2023-11-24T08:20:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.314+0000 7f9130b53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c157c70 0x7f912c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.314+0000 7f9130b53640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f912c0cf450 0x7f912c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.314+0000 7f9130b53640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c0ce0b0 con 0x7f912c15dc80 2023-11-24T08:20:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.314+0000 7f9130b53640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912c0cfe10 con 0x7f912c0cf450 2023-11-24T08:20:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.314+0000 7f9130b53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c161960 con 0x7f912c157c70 2023-11-24T08:20:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.315+0000 7f9123fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f912c0cf450 0x7f912c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.315+0000 7f912a59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c157c70 0x7f912c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.315+0000 7f9123fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f912c0cf450 0x7f912c0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:36590/0 (socket says 172.21.15.107:36590) 2023-11-24T08:20:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.315+0000 7f9123fff640 1 -- 172.21.15.107:0/1285410710 learned_addr learned my addr 172.21.15.107:0/1285410710 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.316+0000 7f9123fff640 1 -- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c157c70 msgr2=0x7f912c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.316+0000 7f9123fff640 1 --2- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c157c70 0x7f912c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.316+0000 7f9123fff640 1 -- 172.21.15.107:0/1285410710 >> v1:172.21.15.120:6789/0 conn(0x7f912c15dc80 legacy=0x7f912c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.316+0000 7f9123fff640 1 -- 172.21.15.107:0/1285410710 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c15e8d0 con 0x7f912c0cf450 2023-11-24T08:20:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.317+0000 7f9123fff640 1 --2- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c0cf450 0x7f912c0cf8d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9114009870 tx=0x7f911402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96ede4b3dcce9e64 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.318+0000 7f9129d9b640 1 -- 172.21.15.107:0/1285410710 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114031640 con 0x7f912c0cf450 2023-11-24T08:20:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.318+0000 7f9129d9b640 1 -- 172.21.15.107:0/1285410710 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91140317a0 con 0x7f912c0cf450 2023-11-24T08:20:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.318+0000 7f9129d9b640 1 -- 172.21.15.107:0/1285410710 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114031b30 con 0x7f912c0cf450 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 -- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c0cf450 msgr2=0x7f912c0cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 --2- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c0cf450 0x7f912c0cf8d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9114009870 tx=0x7f911402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 -- 172.21.15.107:0/1285410710 shutdown_connections 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 --2- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c0cf450 0x7f912c0cf8d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 --2- 172.21.15.107:0/1285410710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c157c70 0x7f912c161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 -- 172.21.15.107:0/1285410710 >> 172.21.15.107:0/1285410710 conn(0x7f912c06a920 msgr2=0x7f912c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 -- 172.21.15.107:0/1285410710 shutdown_connections 2023-11-24T08:20:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.319+0000 7f9130b53640 1 -- 172.21.15.107:0/1285410710 wait complete. 2023-11-24T08:20:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.321+0000 7f9130b53640 1 Processor -- start 2023-11-24T08:20:21.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.321+0000 7f9130b53640 1 -- start start 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c0cf450 0x7f912c1f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c157c70 0x7f912c160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1f4400 con 0x7f912c15dc80 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c09bd30 con 0x7f912c157c70 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9130b53640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c099640 con 0x7f912c0cf450 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f912ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f912ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40582/0 (socket says 172.21.15.107:40582) 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f912ad9d640 1 -- 172.21.15.107:0/3672436647 learned_addr learned my addr 172.21.15.107:0/3672436647 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:21.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f912a59c640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c0cf450 0x7f912c1f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.323+0000 7f9123fff640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c157c70 0x7f912c160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 -- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c0cf450 msgr2=0x7f912c1f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c0cf450 0x7f912c1f3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 -- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c157c70 msgr2=0x7f912c160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c157c70 0x7f912c160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 -- 172.21.15.107:0/3672436647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91140094d0 con 0x7f912c15dc80 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f912ad9d640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f912400c730 tx=0x7f912400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124003b70 con 0x7f912c15dc80 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124003cf0 con 0x7f912c15dc80 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c099920 con 0x7f912c15dc80 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124003ea0 con 0x7f912c15dc80 2023-11-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.324+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c099ee0 con 0x7f912c15dc80 2023-11-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.325+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912c067470 con 0x7f912c15dc80 2023-11-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.326+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f9124005520 con 0x7f912c15dc80 2023-11-24T08:20:21.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.328+0000 7f9122ffd640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 0x7f910c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:21.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.328+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9124088590 con 0x7f912c15dc80 2023-11-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.328+0000 7f912a59c640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 0x7f910c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.329+0000 7f912a59c640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 0x7f910c06efb0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f911c009ad0 tx=0x7f911c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.332+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91240550a0 con 0x7f912c15dc80 2023-11-24T08:20:21.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:21.522+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f912c072880 con 0x7f912c15dc80 2023-11-24T08:20:22.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.137+0000 7f9122ffd640 1 -- 172.21.15.107:0/3672436647 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v181) v1 ==== 181+0+0 (secure 0 0 0) 0x7f91240586c0 con 0x7f912c15dc80 2023-11-24T08:20:22.139 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:20:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 msgr2=0x7f910c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 0x7f910c06efb0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f911c009ad0 tx=0x7f911c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 msgr2=0x7f912c09a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f912400c730 tx=0x7f912400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 shutdown_connections 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f910c06caf0 0x7f910c06efb0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f912c15dc80 0x7f912c09a6e0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f912c157c70 0x7f912c160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 --2- 172.21.15.107:0/3672436647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f912c0cf450 0x7f912c1f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 >> 172.21.15.107:0/3672436647 conn(0x7f912c06a920 msgr2=0x7f912c1574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 shutdown_connections 2023-11-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.141+0000 7f9130b53640 1 -- 172.21.15.107:0/3672436647 wait complete. 2023-11-24T08:20:22.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:20:22.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.381+0000 7f8e62494640 1 Processor -- start 2023-11-24T08:20:22.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.381+0000 7f8e62494640 1 -- start start 2023-11-24T08:20:22.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.382+0000 7f8e62494640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e5c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.382+0000 7f8e62494640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e5c0b2630 0x7f8e5c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.382+0000 7f8e62494640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c15ab80 con 0x7f8e5c0cd470 2023-11-24T08:20:22.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e62494640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c15a990 con 0x7f8e5c0b2630 2023-11-24T08:20:22.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e62494640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c15e680 con 0x7f8e5c0b2250 2023-11-24T08:20:22.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e60a31640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8e5c0cd470 0x7f8e5c15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43754/0 (socket says 172.21.15.107:43754) 2023-11-24T08:20:22.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e60a31640 1 -- 172.21.15.107:0/1232797569 learned_addr learned my addr 172.21.15.107:0/1232797569 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e5bfff640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e5c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e53fff640 1 --2- 172.21.15.107:0/1232797569 >> v2:172.21.15.120:3300/0 conn(0x7f8e5c0b2630 0x7f8e5c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606813757 0 0) 0x7f8e5c15ab80 con 0x7f8e5c0cd470 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.383+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e48003520 con 0x7f8e5c0cd470 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305293122 0 0) 0x7f8e48003520 con 0x7f8e5c0cd470 2023-11-24T08:20:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 msgr2=0x7f8e5c15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e5c15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 >> v2:172.21.15.120:3300/0 conn(0x7f8e5c0b2630 msgr2=0x7f8e5c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 --2- 172.21.15.107:0/1232797569 >> v2:172.21.15.120:3300/0 conn(0x7f8e5c0b2630 0x7f8e5c0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c15e960 con 0x7f8e5c0cd470 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8e540019f0 con 0x7f8e5c0cd470 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 >> v1:172.21.15.107:6789/0 conn(0x7f8e5c0cd470 legacy=0x7f8e5c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48003970 0x7f8e48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e48004d00 0x7f8e48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c15e680 con 0x7f8e48004d00 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4800b980 con 0x7f8e48003970 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4800bac0 con 0x7f8e5c0b2250 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.384+0000 7f8e60a31640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e48004d00 0x7f8e48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e5bfff640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48003970 0x7f8e48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e53fff640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e60a31640 1 -- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 msgr2=0x7f8e480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e60a31640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c0b2250 0x7f8e480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e60a31640 1 -- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48003970 msgr2=0x7f8e48003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e60a31640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48003970 0x7f8e48003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.385+0000 7f8e60a31640 1 -- 172.21.15.107:0/1232797569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e48003520 con 0x7f8e48004d00 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.386+0000 7f8e60a31640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e48004d00 0x7f8e48005180 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400a440 tx=0x7f8e480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247abc9859fcf4a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.386+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54035640 con 0x7f8e48004d00 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.386+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e540357a0 con 0x7f8e48004d00 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.386+0000 7f8e5b7fe640 1 -- 172.21.15.107:0/1232797569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54035b70 con 0x7f8e48004d00 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 -- 172.21.15.107:0/1232797569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e48004d00 msgr2=0x7f8e48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 --2- 172.21.15.107:0/1232797569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e48004d00 0x7f8e48005180 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400a440 tx=0x7f8e480057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e60a31640 1 -- 172.21.15.107:0/1232797569 reap_dead start 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 -- 172.21.15.107:0/1232797569 shutdown_connections 2023-11-24T08:20:22.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 -- 172.21.15.107:0/1232797569 >> 172.21.15.107:0/1232797569 conn(0x7f8e5c06a920 msgr2=0x7f8e5c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:22.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 -- 172.21.15.107:0/1232797569 shutdown_connections 2023-11-24T08:20:22.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.387+0000 7f8e62494640 1 -- 172.21.15.107:0/1232797569 wait complete. 2023-11-24T08:20:22.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.389+0000 7f8e62494640 1 Processor -- start 2023-11-24T08:20:22.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.389+0000 7f8e62494640 1 -- start start 2023-11-24T08:20:22.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48004d00 0x7f8e5c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c154490 0x7f8e5c147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0ca240 con 0x7f8e5c0c8360 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c8c60 con 0x7f8e48004d00 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e62494640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c8f40 con 0x7f8e5c154490 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e60a31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e60a31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40600/0 (socket says 172.21.15.107:40600) 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.390+0000 7f8e60a31640 1 -- 172.21.15.107:0/567211552 learned_addr learned my addr 172.21.15.107:0/567211552 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e5bfff640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48004d00 0x7f8e5c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e53fff640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c154490 0x7f8e5c147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 -- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c154490 msgr2=0x7f8e5c147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c154490 0x7f8e5c147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 -- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48004d00 msgr2=0x7f8e5c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48004d00 0x7f8e5c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 -- 172.21.15.107:0/567211552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0c9220 con 0x7f8e5c0c8360 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e60a31640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402edb0 tx=0x7f8e54001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54035d20 con 0x7f8e5c0c8360 2023-11-24T08:20:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54035e80 con 0x7f8e5c0c8360 2023-11-24T08:20:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54040620 con 0x7f8e5c0c8360 2023-11-24T08:20:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.391+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c0c9500 con 0x7f8e5c0c8360 2023-11-24T08:20:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.392+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c147860 con 0x7f8e5c0c8360 2023-11-24T08:20:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.393+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c067470 con 0x7f8e5c0c8360 2023-11-24T08:20:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.394+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f8e54040780 con 0x7f8e5c0c8360 2023-11-24T08:20:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.395+0000 7f8e597fa640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 0x7f8e3c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.395+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f8e540aa380 con 0x7f8e5c0c8360 2023-11-24T08:20:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.395+0000 7f8e5bfff640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 0x7f8e3c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.396+0000 7f8e5bfff640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 0x7f8e3c06efe0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009990 tx=0x7f8e4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:22.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.399+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e54076eb0 con 0x7f8e5c0c8360 2023-11-24T08:20:22.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:22.583+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e5c0cb3f0 con 0x7f8e5c0c8360 2023-11-24T08:20:23.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.140+0000 7f8e597fa640 1 -- 172.21.15.107:0/567211552 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v182) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8e54048030 con 0x7f8e5c0c8360 2023-11-24T08:20:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:20:23.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 msgr2=0x7f8e3c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 0x7f8e3c06efe0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009990 tx=0x7f8e4c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:23.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 msgr2=0x7f8e5c158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402edb0 tx=0x7f8e54001d80 comp rx=0 tx=0).stop 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 shutdown_connections 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3c06cb20 0x7f8e3c06efe0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e5c0c8360 0x7f8e5c158b10 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e5c154490 0x7f8e5c147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 --2- 172.21.15.107:0/567211552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e48004d00 0x7f8e5c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.144+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 >> 172.21.15.107:0/567211552 conn(0x7f8e5c06a920 msgr2=0x7f8e5c15c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.145+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 shutdown_connections 2023-11-24T08:20:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.145+0000 7f8e62494640 1 -- 172.21.15.107:0/567211552 wait complete. 2023-11-24T08:20:23.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:20:23.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.379+0000 7f73474e1640 1 Processor -- start 2023-11-24T08:20:23.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.380+0000 7f73474e1640 1 -- start start 2023-11-24T08:20:23.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.382+0000 7f73474e1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f73401609f0 0x7f73400d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.382+0000 7f73474e1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b41b0 con 0x7f73400b2050 2023-11-24T08:20:23.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.382+0000 7f73474e1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b4490 con 0x7f7340160dd0 2023-11-24T08:20:23.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.382+0000 7f73474e1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400b0090 con 0x7f73401609f0 2023-11-24T08:20:23.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.383+0000 7f7346ce0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f73400b2050 0x7f73400b0fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43764/0 (socket says 172.21.15.107:43764) 2023-11-24T08:20:23.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.383+0000 7f7346ce0640 1 -- 172.21.15.107:0/3921424918 learned_addr learned my addr 172.21.15.107:0/3921424918 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:23.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.383+0000 7f73464df640 1 --2- 172.21.15.107:0/3921424918 >> v2:172.21.15.120:3300/0 conn(0x7f73401609f0 0x7f73400d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.385+0000 7f73464df640 1 -- 172.21.15.107:0/3921424918 >> v1:172.21.15.120:6789/0 conn(0x7f7340160dd0 legacy=0x7f73400b0890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T08:20:23.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.385+0000 7f73464df640 1 -- 172.21.15.107:0/3921424918 >> v1:172.21.15.107:6789/0 conn(0x7f73400b2050 legacy=0x7f73400b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.385+0000 7f73464df640 1 -- 172.21.15.107:0/3921424918 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400b0370 con 0x7f73401609f0 2023-11-24T08:20:23.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.389+0000 7f73464df640 1 --2- 172.21.15.107:0/3921424918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73401609f0 0x7f73400d12f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2432dd947e505f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.390+0000 7f73454dd640 1 -- 172.21.15.107:0/3921424918 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c030710 con 0x7f73401609f0 2023-11-24T08:20:23.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.390+0000 7f73454dd640 1 -- 172.21.15.107:0/3921424918 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c02e050 con 0x7f73401609f0 2023-11-24T08:20:23.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.390+0000 7f73454dd640 1 -- 172.21.15.107:0/3921424918 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c030a90 con 0x7f73401609f0 2023-11-24T08:20:23.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.392+0000 7f73474e1640 1 -- 172.21.15.107:0/3921424918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73401609f0 msgr2=0x7f73400d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.392+0000 7f73474e1640 1 --2- 172.21.15.107:0/3921424918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73401609f0 0x7f73400d12f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02b710 comp rx=0 tx=0).stop 2023-11-24T08:20:23.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.394+0000 7f73474e1640 1 -- 172.21.15.107:0/3921424918 shutdown_connections 2023-11-24T08:20:23.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.395+0000 7f73474e1640 1 --2- 172.21.15.107:0/3921424918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73401609f0 0x7f73400d12f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.395+0000 7f73474e1640 1 -- 172.21.15.107:0/3921424918 >> 172.21.15.107:0/3921424918 conn(0x7f734009c5b0 msgr2=0x7f73400d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:23.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.395+0000 7f73474e1640 1 -- 172.21.15.107:0/3921424918 shutdown_connections 2023-11-24T08:20:23.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.396+0000 7f73474e1640 1 -- 172.21.15.107:0/3921424918 wait complete. 2023-11-24T08:20:23.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.399+0000 7f73474e1640 1 Processor -- start 2023-11-24T08:20:23.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.399+0000 7f73474e1640 1 -- start start 2023-11-24T08:20:23.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73401609f0 0x7f73400cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7340160dd0 0x7f73400ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7340154d00 con 0x7f73401609f0 2023-11-24T08:20:23.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400c8d10 con 0x7f7340160dd0 2023-11-24T08:20:23.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.400+0000 7f73474e1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400c8100 con 0x7f73400b2050 2023-11-24T08:20:23.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.402+0000 7f73464df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.402+0000 7f73464df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:40948/0 (socket says 172.21.15.107:40948) 2023-11-24T08:20:23.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.402+0000 7f73464df640 1 -- 172.21.15.107:0/2877165314 learned_addr learned my addr 172.21.15.107:0/2877165314 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.402+0000 7f7346ce0640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7340160dd0 0x7f73400ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.403+0000 7f73464df640 1 -- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7340160dd0 msgr2=0x7f73400ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.403+0000 7f73464df640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7340160dd0 0x7f73400ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.403+0000 7f73464df640 1 -- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73401609f0 msgr2=0x7f73400cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.403+0000 7f73464df640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73401609f0 0x7f73400cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.403+0000 7f73464df640 1 -- 172.21.15.107:0/2877165314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009400 con 0x7f73400b2050 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.404+0000 7f73464df640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.404+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c032c40 con 0x7f73400b2050 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.405+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c032dc0 con 0x7f73400b2050 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.405+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c03c6b0 con 0x7f73400b2050 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.405+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73400c83e0 con 0x7f73400b2050 2023-11-24T08:20:23.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.405+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73400c89a0 con 0x7f73400b2050 2023-11-24T08:20:23.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.406+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734004e840 con 0x7f73400b2050 2023-11-24T08:20:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.410+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f733c039040 con 0x7f73400b2050 2023-11-24T08:20:23.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.412+0000 7f7336ffd640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 0x7f732406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.412+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f733c0aaa20 con 0x7f73400b2050 2023-11-24T08:20:23.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.412+0000 7f7345cde640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 0x7f732406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.413+0000 7f7345cde640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 0x7f732406efe0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7340072170 tx=0x7f7330007a50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.425+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f733c077660 con 0x7f73400b2050 2023-11-24T08:20:23.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.609+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f734006b2c0 con 0x7f73400b2050 2023-11-24T08:20:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.623+0000 7f7336ffd640 1 -- 172.21.15.107:0/2877165314 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v48) v1 ==== 104+0+0 (secure 0 0 0) 0x7f733c037020 con 0x7f73400b2050 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 msgr2=0x7f732406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 0x7f732406efe0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7340072170 tx=0x7f7330007a50 comp rx=0 tx=0).stop 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 msgr2=0x7f73401547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 shutdown_connections 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f732406cb20 0x7f732406efe0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7340160dd0 0x7f73400ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73401609f0 0x7f73400cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 --2- 172.21.15.107:0/2877165314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73400b2050 0x7f73401547c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 >> 172.21.15.107:0/2877165314 conn(0x7f734009c5b0 msgr2=0x7f73400d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 shutdown_connections 2023-11-24T08:20:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.626+0000 7f73474e1640 1 -- 172.21.15.107:0/2877165314 wait complete. 2023-11-24T08:20:23.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs' 2023-11-24T08:20:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.848+0000 7f648f59e640 1 Processor -- start 2023-11-24T08:20:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.848+0000 7f648f59e640 1 -- start start 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f649013d120 0x7f649015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900cec00 con 0x7f649006db20 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648f59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900b1890 con 0x7f649006d6c0 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648f59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649013f9e0 con 0x7f649013d120 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f649013d120 0x7f649015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f649013d120 0x7f649015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40628/0 (socket says 172.21.15.107:40628) 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648e59c640 1 -- 172.21.15.107:0/1644628763 learned_addr learned my addr 172.21.15.107:0/1644628763 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.850+0000 7f648ed9d640 1 --1- 172.21.15.107:0/1644628763 >> v1:172.21.15.107:6789/0 conn(0x7f649006db20 0x7f64900b1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43766/0 (socket says 172.21.15.107:43766) 2023-11-24T08:20:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.851+0000 7f648e59c640 1 -- 172.21.15.107:0/1644628763 >> v1:172.21.15.120:6789/0 conn(0x7f649006d6c0 legacy=0x7f64900b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.851+0000 7f648e59c640 1 -- 172.21.15.107:0/1644628763 >> v1:172.21.15.107:6789/0 conn(0x7f649006db20 legacy=0x7f64900b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.851+0000 7f648e59c640 1 -- 172.21.15.107:0/1644628763 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649013fcc0 con 0x7f649013d120 2023-11-24T08:20:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.851+0000 7f648dd9b640 1 -- 172.21.15.107:0/1644628763 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716755211 0 0) 0x7f64900cec00 con 0x7f649006db20 2023-11-24T08:20:23.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648e59c640 1 --2- 172.21.15.107:0/1644628763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649013d120 0x7f649015af40 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f6480009870 tx=0x7f648002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b3eacecd2b8cca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648dd9b640 1 -- 172.21.15.107:0/1644628763 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480031870 con 0x7f649013d120 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648dd9b640 1 -- 172.21.15.107:0/1644628763 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64800319d0 con 0x7f649013d120 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648dd9b640 1 -- 172.21.15.107:0/1644628763 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480031d60 con 0x7f649013d120 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648f59e640 1 -- 172.21.15.107:0/1644628763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649013d120 msgr2=0x7f649015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.852+0000 7f648f59e640 1 --2- 172.21.15.107:0/1644628763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649013d120 0x7f649015af40 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f6480009870 tx=0x7f648002b910 comp rx=0 tx=0).stop 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.853+0000 7f648f59e640 1 -- 172.21.15.107:0/1644628763 shutdown_connections 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.853+0000 7f648f59e640 1 --2- 172.21.15.107:0/1644628763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649013d120 0x7f649015af40 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.853+0000 7f648f59e640 1 -- 172.21.15.107:0/1644628763 >> 172.21.15.107:0/1644628763 conn(0x7f64900cb830 msgr2=0x7f64900cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.853+0000 7f648f59e640 1 -- 172.21.15.107:0/1644628763 shutdown_connections 2023-11-24T08:20:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.853+0000 7f648f59e640 1 -- 172.21.15.107:0/1644628763 wait complete. 2023-11-24T08:20:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.855+0000 7f648f59e640 1 Processor -- start 2023-11-24T08:20:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.855+0000 7f648f59e640 1 -- start start 2023-11-24T08:20:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649006db20 0x7f649015bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649013d120 0x7f649015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649015d550 con 0x7f649006d6c0 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649015b1b0 con 0x7f649006db20 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649015c940 con 0x7f649013d120 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40640/0 (socket says 172.21.15.107:40640) 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 -- 172.21.15.107:0/3965593339 learned_addr learned my addr 172.21.15.107:0/3965593339 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 -- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649013d120 msgr2=0x7f649015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649013d120 0x7f649015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 -- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649006db20 msgr2=0x7f649015bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f6485d9b640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649006db20 0x7f649015bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649006db20 0x7f649015bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.856+0000 7f648e59c640 1 -- 172.21.15.107:0/3965593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800094d0 con 0x7f649006d6c0 2023-11-24T08:20:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f648e59c640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f6480009870 tx=0x7f64800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648003ad30 con 0x7f649006d6c0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648003ae90 con 0x7f649006d6c0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649015cc20 con 0x7f649006d6c0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480044630 con 0x7f649006d6c0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.857+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649015d1e0 con 0x7f649006d6c0 2023-11-24T08:20:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.858+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649004e840 con 0x7f649006d6c0 2023-11-24T08:20:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.859+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f648004d450 con 0x7f649006d6c0 2023-11-24T08:20:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.860+0000 7f64877fe640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 0x7f647006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.860+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f648002d040 con 0x7f649006d6c0 2023-11-24T08:20:23.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.860+0000 7f6485d9b640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 0x7f647006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:23.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.861+0000 7f6485d9b640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 0x7f647006f000 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f64900cd400 tx=0x7f6478008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:23.864+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64800778a0 con 0x7f649006d6c0 2023-11-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.123+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs"]} v 0) v1 -- 0x7f6490155400 con 0x7f649006d6c0 2023-11-24T08:20:24.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.156+0000 7f64877fe640 1 -- 172.21.15.107:0/3965593339 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_quota_attrs"]}]=0 v779) v1 ==== 173+0+0 (secure 0 0 0) 0x7f648007aec0 con 0x7f649006d6c0 2023-11-24T08:20:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 msgr2=0x7f647006f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 0x7f647006f000 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f64900cd400 tx=0x7f6478008040 comp rx=0 tx=0).stop 2023-11-24T08:20:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 msgr2=0x7f6490133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f6480009870 tx=0x7f64800046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 shutdown_connections 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f647006cb40 0x7f647006f000 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f649013d120 0x7f649015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649006db20 0x7f649015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 --2- 172.21.15.107:0/3965593339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f649006d6c0 0x7f6490133700 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 >> 172.21.15.107:0/3965593339 conn(0x7f64900cb830 msgr2=0x7f6490159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 shutdown_connections 2023-11-24T08:20:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.160+0000 7f648f59e640 1 -- 172.21.15.107:0/3965593339 wait complete. 2023-11-24T08:20:24.181 INFO:tasks.cephfs_test_runner:test_subvolume_clone_inherit_quota_attrs (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:20:24.181 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_inherit_snapshot_namespace_and_size (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:20:24.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.387+0000 7f916771d640 1 Processor -- start 2023-11-24T08:20:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.387+0000 7f916771d640 1 -- start start 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f916771d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91600b2250 0x7f916015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f916771d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2aa0 0x7f916015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f916771d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916015ab80 con 0x7f91600b2630 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f916771d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916015a990 con 0x7f91600b2250 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f916771d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916015e680 con 0x7f91600b2aa0 2023-11-24T08:20:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f91654b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91600b2250 0x7f916015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f91654b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91600b2250 0x7f916015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40656/0 (socket says 172.21.15.107:40656) 2023-11-24T08:20:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.389+0000 7f91654b9640 1 -- 172.21.15.107:0/133844822 learned_addr learned my addr 172.21.15.107:0/133844822 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.390+0000 7f9165cba640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2aa0 0x7f916015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.390+0000 7f91654b9640 1 -- 172.21.15.107:0/133844822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2aa0 msgr2=0x7f916015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.390+0000 7f91654b9640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2aa0 0x7f916015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.390+0000 7f91654b9640 1 -- 172.21.15.107:0/133844822 >> v1:172.21.15.120:6789/0 conn(0x7f91600b2630 legacy=0x7f91600ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.390+0000 7f91654b9640 1 -- 172.21.15.107:0/133844822 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916015e960 con 0x7f91600b2250 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.391+0000 7f91654b9640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2250 0x7f916015df80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f915c00bb10 tx=0x7f915c02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0db9e9381d591f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.391+0000 7f9157fff640 1 -- 172.21.15.107:0/133844822 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c02ee70 con 0x7f91600b2250 2023-11-24T08:20:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.391+0000 7f9157fff640 1 -- 172.21.15.107:0/133844822 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c036440 con 0x7f91600b2250 2023-11-24T08:20:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.391+0000 7f9157fff640 1 -- 172.21.15.107:0/133844822 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0367d0 con 0x7f91600b2250 2023-11-24T08:20:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 -- 172.21.15.107:0/133844822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2250 msgr2=0x7f916015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2250 0x7f916015df80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f915c00bb10 tx=0x7f915c02e0c0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 -- 172.21.15.107:0/133844822 shutdown_connections 2023-11-24T08:20:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2aa0 0x7f916015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 --2- 172.21.15.107:0/133844822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2250 0x7f916015df80 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 -- 172.21.15.107:0/133844822 >> 172.21.15.107:0/133844822 conn(0x7f916006a920 msgr2=0x7f916006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 -- 172.21.15.107:0/133844822 shutdown_connections 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.392+0000 7f916771d640 1 -- 172.21.15.107:0/133844822 wait complete. 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.394+0000 7f916771d640 1 Processor -- start 2023-11-24T08:20:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.394+0000 7f916771d640 1 -- start start 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2250 0x7f91600c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 0x7f9160159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9160156710 con 0x7f91600b2aa0 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916015bee0 con 0x7f91600b2630 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f916771d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916015c1c0 con 0x7f91600b2250 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f9165cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f9165cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40666/0 (socket says 172.21.15.107:40666) 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f9165cba640 1 -- 172.21.15.107:0/586203690 learned_addr learned my addr 172.21.15.107:0/586203690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f91654b9640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2250 0x7f91600c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.395+0000 7f9164cb8640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 0x7f9160159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 -- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2250 msgr2=0x7f91600c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2250 0x7f91600c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 -- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 msgr2=0x7f9160159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 0x7f9160159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 -- 172.21.15.107:0/586203690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c00b770 con 0x7f91600b2aa0 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9164cb8640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 0x7f9160159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9165cba640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f915800cc80 tx=0x7f915800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915800ce30 con 0x7f91600b2aa0 2023-11-24T08:20:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158003990 con 0x7f91600b2aa0 2023-11-24T08:20:24.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158003d20 con 0x7f91600b2aa0 2023-11-24T08:20:24.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91600cd7c0 con 0x7f91600b2aa0 2023-11-24T08:20:24.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.396+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916015c780 con 0x7f91600b2aa0 2023-11-24T08:20:24.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.397+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9160067470 con 0x7f91600b2aa0 2023-11-24T08:20:24.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.398+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f9158003e80 con 0x7f91600b2aa0 2023-11-24T08:20:24.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.399+0000 7f9155ffb640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 0x7f914406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.399+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9158088930 con 0x7f91600b2aa0 2023-11-24T08:20:24.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.400+0000 7f91654b9640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 0x7f914406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.400+0000 7f91654b9640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 0x7f914406efb0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f915c038000 tx=0x7f915c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.403+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9158055440 con 0x7f91600b2aa0 2023-11-24T08:20:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.597+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91600cea00 con 0x7f91600b2aa0 2023-11-24T08:20:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.599+0000 7f9155ffb640 1 -- 172.21.15.107:0/586203690 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+12998 (secure 0 0 0) 0x7f915801d070 con 0x7f91600b2aa0 2023-11-24T08:20:24.601 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:24.601 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":182,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:23.138296+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6836/3208934992":"2023-11-25T08:20:19.985510+0000","172.21.15.107:6834/3208934992":"2023-11-25T08:20:19.985510+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.601+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 msgr2=0x7f914406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.601+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 0x7f914406efb0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f915c038000 tx=0x7f915c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.601+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 msgr2=0x7f91600c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.601+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f915800cc80 tx=0x7f915800a5c0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 shutdown_connections 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f914406caf0 0x7f914406efb0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91600b2aa0 0x7f91600c8360 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91600b2630 0x7f9160159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 --2- 172.21.15.107:0/586203690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91600b2250 0x7f91600c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 >> 172.21.15.107:0/586203690 conn(0x7f916006a920 msgr2=0x7f91600abfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 shutdown_connections 2023-11-24T08:20:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.602+0000 7f916771d640 1 -- 172.21.15.107:0/586203690 wait complete. 2023-11-24T08:20:24.624 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:20:24.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:20:24.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.842+0000 7f58026ad640 1 Processor -- start 2023-11-24T08:20:24.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.842+0000 7f58026ad640 1 -- start start 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f58026ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f57fc0cc0f0 0x7f57fc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f58026ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57fc15eea0 con 0x7f57fc15b440 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f58026ad640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57fc0abf30 con 0x7f57fc0c57e0 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f58026ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc15b820 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f57fbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f57fc15b440 0x7f57fc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43780/0 (socket says 172.21.15.107:43780) 2023-11-24T08:20:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f57fbfff640 1 -- 172.21.15.107:0/1705570927 learned_addr learned my addr 172.21.15.107:0/1705570927 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.844+0000 7f57fb7fe640 1 --2- 172.21.15.107:0/1705570927 >> v2:172.21.15.107:3300/0 conn(0x7f57fc0cc0f0 0x7f57fc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.845+0000 7f57faffd640 1 -- 172.21.15.107:0/1705570927 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506785812 0 0) 0x7f57fc15eea0 con 0x7f57fc15b440 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.845+0000 7f57faffd640 1 -- 172.21.15.107:0/1705570927 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4003520 con 0x7f57fc15b440 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.845+0000 7f57fb7fe640 1 -- 172.21.15.107:0/1705570927 >> v1:172.21.15.120:6789/0 conn(0x7f57fc0c57e0 legacy=0x7f57fc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.845+0000 7f57fb7fe640 1 -- 172.21.15.107:0/1705570927 >> v1:172.21.15.107:6789/0 conn(0x7f57fc15b440 legacy=0x7f57fc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.845+0000 7f57fb7fe640 1 -- 172.21.15.107:0/1705570927 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc15f0f0 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.846+0000 7f57fb7fe640 1 --2- 172.21.15.107:0/1705570927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc0cc0f0 0x7f57fc0c52a0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321b23525eec240b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.846+0000 7f57faffd640 1 -- 172.21.15.107:0/1705570927 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f002e070 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.846+0000 7f57faffd640 1 -- 172.21.15.107:0/1705570927 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f00315a0 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.846+0000 7f57faffd640 1 -- 172.21.15.107:0/1705570927 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0031930 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 -- 172.21.15.107:0/1705570927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc0cc0f0 msgr2=0x7f57fc0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 --2- 172.21.15.107:0/1705570927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc0cc0f0 0x7f57fc0c52a0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002b840 comp rx=0 tx=0).stop 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 -- 172.21.15.107:0/1705570927 shutdown_connections 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 --2- 172.21.15.107:0/1705570927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc0cc0f0 0x7f57fc0c52a0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 -- 172.21.15.107:0/1705570927 >> 172.21.15.107:0/1705570927 conn(0x7f57fc0b0c50 msgr2=0x7f57fc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 -- 172.21.15.107:0/1705570927 shutdown_connections 2023-11-24T08:20:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.847+0000 7f58026ad640 1 -- 172.21.15.107:0/1705570927 wait complete. 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.849+0000 7f58026ad640 1 Processor -- start 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.849+0000 7f58026ad640 1 -- start start 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc0c57e0 0x7f57fc1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 0x7f57fc157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc154810 con 0x7f57fc15b440 2023-11-24T08:20:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc15ba00 con 0x7f57fc0c57e0 2023-11-24T08:20:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f58026ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc15bce0 con 0x7f57fc0cc0f0 2023-11-24T08:20:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40682/0 (socket says 172.21.15.107:40682) 2023-11-24T08:20:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 -- 172.21.15.107:0/1192928304 learned_addr learned my addr 172.21.15.107:0/1192928304 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:24.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 -- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 msgr2=0x7f57fc157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f57fb7fe640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 0x7f57fc157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f57fbfff640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc0c57e0 0x7f57fc1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 0x7f57fc157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 -- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc0c57e0 msgr2=0x7f57fc1542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc0c57e0 0x7f57fc1542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.850+0000 7f5800c4a640 1 -- 172.21.15.107:0/1192928304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f0009400 con 0x7f57fc15b440 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f57fb7fe640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 0x7f57fc157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f5800c4a640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f57f400c410 tx=0x7f57f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4011790 con 0x7f57fc15b440 2023-11-24T08:20:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f40118f0 con 0x7f57fc15b440 2023-11-24T08:20:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4011ce0 con 0x7f57fc15b440 2023-11-24T08:20:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc15bfc0 con 0x7f57fc15b440 2023-11-24T08:20:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.851+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc15c580 con 0x7f57fc15b440 2023-11-24T08:20:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.852+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc15c7a0 con 0x7f57fc15b440 2023-11-24T08:20:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.853+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f57f401c460 con 0x7f57fc15b440 2023-11-24T08:20:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.854+0000 7f57f8ff9640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 0x7f57e806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.854+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f57f40882b0 con 0x7f57fc15b440 2023-11-24T08:20:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.855+0000 7f57fbfff640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 0x7f57e806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.855+0000 7f57fbfff640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 0x7f57e806efb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f57ec001560 tx=0x7f57ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:24.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:24.858+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57f4054dc0 con 0x7f57fc15b440 2023-11-24T08:20:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.046+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57fc04e8b0 con 0x7f57fc15b440 2023-11-24T08:20:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.047+0000 7f57f8ff9640 1 -- 172.21.15.107:0/1192928304 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57f40583e0 con 0x7f57fc15b440 2023-11-24T08:20:25.049 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 msgr2=0x7f57e806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 0x7f57e806efb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f57ec001560 tx=0x7f57ec009040 comp rx=0 tx=0).stop 2023-11-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 msgr2=0x7f57fc0afa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f57f400c410 tx=0x7f57f400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 shutdown_connections 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f57e806caf0 0x7f57e806efb0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57fc15b440 0x7f57fc0afa00 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57fc0cc0f0 0x7f57fc157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 --2- 172.21.15.107:0/1192928304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc0c57e0 0x7f57fc1542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 >> 172.21.15.107:0/1192928304 conn(0x7f57fc0b0c50 msgr2=0x7f57fc070990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.049+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 shutdown_connections 2023-11-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.050+0000 7f58026ad640 1 -- 172.21.15.107:0/1192928304 wait complete. 2023-11-24T08:20:25.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size' 2023-11-24T08:20:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.273+0000 7f52b8eec640 1 Processor -- start 2023-11-24T08:20:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.274+0000 7f52b8eec640 1 -- start start 2023-11-24T08:20:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.275+0000 7f52b8eec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52b415d520 0x7f52b40ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.275+0000 7f52b8eec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f52b406e310 0x7f52b4071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.275+0000 7f52b8eec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b406c630 con 0x7f52b40ce8d0 2023-11-24T08:20:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.275+0000 7f52b8eec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b406c910 con 0x7f52b406e310 2023-11-24T08:20:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.275+0000 7f52b8eec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b406cbf0 con 0x7f52b415d520 2023-11-24T08:20:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f52b406e310 0x7f52b4071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f52b406e310 0x7f52b4071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40698/0 (socket says 172.21.15.107:40698) 2023-11-24T08:20:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 -- 172.21.15.107:0/3091387802 learned_addr learned my addr 172.21.15.107:0/3091387802 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52abfff640 1 --1- 172.21.15.107:0/3091387802 >> v1:172.21.15.107:6789/0 conn(0x7f52b40ce8d0 0x7f52b406dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43788/0 (socket says 172.21.15.107:43788) 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b259c640 1 --2- 172.21.15.107:0/3091387802 >> v2:172.21.15.120:3300/0 conn(0x7f52b415d520 0x7f52b40ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 -- 172.21.15.107:0/3091387802 >> v2:172.21.15.120:3300/0 conn(0x7f52b415d520 msgr2=0x7f52b40ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 --2- 172.21.15.107:0/3091387802 >> v2:172.21.15.120:3300/0 conn(0x7f52b415d520 0x7f52b40ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 -- 172.21.15.107:0/3091387802 >> v1:172.21.15.107:6789/0 conn(0x7f52b40ce8d0 legacy=0x7f52b406dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b2d9d640 1 -- 172.21.15.107:0/3091387802 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b406ced0 con 0x7f52b406e310 2023-11-24T08:20:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.276+0000 7f52b1d9b640 1 -- 172.21.15.107:0/3091387802 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137499949 0 0) 0x7f52b406c630 con 0x7f52b40ce8d0 2023-11-24T08:20:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.277+0000 7f52b2d9d640 1 --2- 172.21.15.107:0/3091387802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4071850 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f52b406c020 tx=0x7f52ac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e4caafc5dd0f40 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.277+0000 7f52b1d9b640 1 -- 172.21.15.107:0/3091387802 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac02cb20 con 0x7f52b406e310 2023-11-24T08:20:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.277+0000 7f52b1d9b640 1 -- 172.21.15.107:0/3091387802 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac02cc80 con 0x7f52b406e310 2023-11-24T08:20:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.277+0000 7f52b1d9b640 1 -- 172.21.15.107:0/3091387802 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac0343f0 con 0x7f52b406e310 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 -- 172.21.15.107:0/3091387802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 msgr2=0x7f52b4071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 --2- 172.21.15.107:0/3091387802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4071850 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f52b406c020 tx=0x7f52ac02bf80 comp rx=0 tx=0).stop 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 -- 172.21.15.107:0/3091387802 shutdown_connections 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 --2- 172.21.15.107:0/3091387802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4071850 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 --2- 172.21.15.107:0/3091387802 >> v2:172.21.15.120:3300/0 conn(0x7f52b415d520 0x7f52b40ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 -- 172.21.15.107:0/3091387802 >> 172.21.15.107:0/3091387802 conn(0x7f52b40b0c70 msgr2=0x7f52b40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 -- 172.21.15.107:0/3091387802 shutdown_connections 2023-11-24T08:20:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.278+0000 7f52b8eec640 1 -- 172.21.15.107:0/3091387802 wait complete. 2023-11-24T08:20:25.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.280+0000 7f52b8eec640 1 Processor -- start 2023-11-24T08:20:25.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.280+0000 7f52b8eec640 1 -- start start 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52b40ce8d0 0x7f52b41505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 0x7f52b4160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b415ff40 con 0x7f52b406e310 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b415f070 con 0x7f52b40ce8d0 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b8eec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41550d0 con 0x7f52b415d520 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40712/0 (socket says 172.21.15.107:40712) 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.281+0000 7f52b259c640 1 -- 172.21.15.107:0/4017444407 learned_addr learned my addr 172.21.15.107:0/4017444407 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b2d9d640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 0x7f52b4160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 -- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 msgr2=0x7f52b4160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 0x7f52b4160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52abfff640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52b40ce8d0 0x7f52b41505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 -- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52b40ce8d0 msgr2=0x7f52b41505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52b40ce8d0 0x7f52b41505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 -- 172.21.15.107:0/4017444407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac009b30 con 0x7f52b406e310 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b2d9d640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 0x7f52b4160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.282+0000 7f52b259c640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f52a400d800 tx=0x7f52a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.283+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a400b660 con 0x7f52b406e310 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.283+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a400b7c0 con 0x7f52b406e310 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.283+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a400bb50 con 0x7f52b406e310 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.283+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b415f2c0 con 0x7f52b406e310 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.283+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b415f880 con 0x7f52b406e310 2023-11-24T08:20:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.284+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b404e8b0 con 0x7f52b406e310 2023-11-24T08:20:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.285+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f52a400bcb0 con 0x7f52b406e310 2023-11-24T08:20:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.286+0000 7f52aaffd640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 0x7f529406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.286+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f52a4088b00 con 0x7f52b406e310 2023-11-24T08:20:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.286+0000 7f52abfff640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 0x7f529406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.287+0000 7f52abfff640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 0x7f529406efb0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f529c007fd0 tx=0x7f529c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.291+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52a4055610 con 0x7f52b406e310 2023-11-24T08:20:25.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:25.550+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size"]} v 0) v1 -- 0x7f52b41601b0 con 0x7f52b406e310 2023-11-24T08:20:26.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.169+0000 7f52aaffd640 1 -- 172.21.15.107:0/4017444407 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size"]}]=0 v781) v1 ==== 192+0+0 (secure 0 0 0) 0x7f52a4058c30 con 0x7f52b406e310 2023-11-24T08:20:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 msgr2=0x7f529406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 0x7f529406efb0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f529c007fd0 tx=0x7f529c009040 comp rx=0 tx=0).stop 2023-11-24T08:20:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 msgr2=0x7f52b4154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f52a400d800 tx=0x7f52a400dcc0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 shutdown_connections 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f529406caf0 0x7f529406efb0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52b415d520 0x7f52b4160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52b40ce8d0 0x7f52b41505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 --2- 172.21.15.107:0/4017444407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b406e310 0x7f52b4154b90 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 >> 172.21.15.107:0/4017444407 conn(0x7f52b40b0c70 msgr2=0x7f52b415c850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 shutdown_connections 2023-11-24T08:20:26.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.173+0000 7f52b8eec640 1 -- 172.21.15.107:0/4017444407 wait complete. 2023-11-24T08:20:26.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:20:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.406+0000 7f1ddf2ad640 1 Processor -- start 2023-11-24T08:20:26.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.407+0000 7f1ddf2ad640 1 -- start start 2023-11-24T08:20:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.408+0000 7f1ddf2ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1dd8141b30 0x7f1dd806c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.408+0000 7f1ddf2ad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 0x7f1dd80cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.408+0000 7f1ddf2ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd80ccdf0 con 0x7f1dd815ef00 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.408+0000 7f1ddf2ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd80cf730 con 0x7f1dd8141b30 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.408+0000 7f1ddf2ad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd815f470 con 0x7f1dd80cede0 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1dd8141b30 0x7f1dd806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dddaaa640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1dd815ef00 0x7f1dd8071700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43804/0 (socket says 172.21.15.107:43804) 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1dd8141b30 0x7f1dd806c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40716/0 (socket says 172.21.15.107:40716) 2023-11-24T08:20:26.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dddaaa640 1 -- 172.21.15.107:0/834669290 learned_addr learned my addr 172.21.15.107:0/834669290 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1ddeaac640 1 --2- 172.21.15.107:0/834669290 >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 0x7f1dd80cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 -- 172.21.15.107:0/834669290 >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 msgr2=0x7f1dd80cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 --2- 172.21.15.107:0/834669290 >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 0x7f1dd80cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 -- 172.21.15.107:0/834669290 >> v1:172.21.15.107:6789/0 conn(0x7f1dd815ef00 legacy=0x7f1dd8071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1dde2ab640 1 -- 172.21.15.107:0/834669290 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd815f750 con 0x7f1dd8141b30 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1ddeaac640 1 --2- 172.21.15.107:0/834669290 >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 0x7f1dd80cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.409+0000 7f1ddd2a9640 1 -- 172.21.15.107:0/834669290 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256366393 0 0) 0x7f1dd80ccdf0 con 0x7f1dd815ef00 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.410+0000 7f1dde2ab640 1 --2- 172.21.15.107:0/834669290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd806c770 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80cff30 tx=0x7f1dd402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3079f38beae1bf7e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.410+0000 7f1ddd2a9640 1 -- 172.21.15.107:0/834669290 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4031710 con 0x7f1dd8141b30 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddd2a9640 1 -- 172.21.15.107:0/834669290 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd402f050 con 0x7f1dd8141b30 2023-11-24T08:20:26.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddd2a9640 1 -- 172.21.15.107:0/834669290 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4031a90 con 0x7f1dd8141b30 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/834669290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 msgr2=0x7f1dd806c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/834669290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd806c770 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80cff30 tx=0x7f1dd402b7b0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/834669290 shutdown_connections 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/834669290 >> v2:172.21.15.120:3300/0 conn(0x7f1dd80cede0 0x7f1dd80cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/834669290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd806c770 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/834669290 >> 172.21.15.107:0/834669290 conn(0x7f1dd806a500 msgr2=0x7f1dd806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/834669290 shutdown_connections 2023-11-24T08:20:26.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.411+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/834669290 wait complete. 2023-11-24T08:20:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.413+0000 7f1ddf2ad640 1 Processor -- start 2023-11-24T08:20:26.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.413+0000 7f1ddf2ad640 1 -- start start 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80cede0 0x7f1dd81f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd815ef00 0x7f1dd80a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8135e60 con 0x7f1dd8141b30 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81f0960 con 0x7f1dd815ef00 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.414+0000 7f1ddf2ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80aa480 con 0x7f1dd80cede0 2023-11-24T08:20:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40728/0 (socket says 172.21.15.107:40728) 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 -- 172.21.15.107:0/468742245 learned_addr learned my addr 172.21.15.107:0/468742245 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dde2ab640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80cede0 0x7f1dd81f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 -- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80cede0 msgr2=0x7f1dd81f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80cede0 0x7f1dd81f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 -- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd815ef00 msgr2=0x7f1dd80a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd815ef00 0x7f1dd80a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 -- 172.21.15.107:0/468742245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4009400 con 0x7f1dd8141b30 2023-11-24T08:20:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.415+0000 7f1dddaaa640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800c4d0 tx=0x7f1dc800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.416+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc80074e0 con 0x7f1dd8141b30 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.416+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8007660 con 0x7f1dd8141b30 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.416+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8004a20 con 0x7f1dd8141b30 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.416+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd80aa760 con 0x7f1dd8141b30 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.416+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd80aad20 con 0x7f1dd8141b30 2023-11-24T08:20:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.417+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd804e840 con 0x7f1dd8141b30 2023-11-24T08:20:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.418+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f1dc8004b80 con 0x7f1dd8141b30 2023-11-24T08:20:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.419+0000 7f1dceffd640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 0x7f1dbc06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.419+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1dc8087780 con 0x7f1dd8141b30 2023-11-24T08:20:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.419+0000 7f1dde2ab640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 0x7f1dbc06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.420+0000 7f1dde2ab640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 0x7f1dbc06efe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4035000 tx=0x7f1dd4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:26.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.424+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dc80542a0 con 0x7f1dd8141b30 2023-11-24T08:20:26.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.615+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1dd815b020 con 0x7f1dd8141b30 2023-11-24T08:20:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.623+0000 7f1dceffd640 1 -- 172.21.15.107:0/468742245 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v49)=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1dc8018030 con 0x7f1dd8141b30 2023-11-24T08:20:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 msgr2=0x7f1dbc06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 0x7f1dbc06efe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4035000 tx=0x7f1dd4008030 comp rx=0 tx=0).stop 2023-11-24T08:20:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 msgr2=0x7f1dd80ab520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800c4d0 tx=0x7f1dc800c990 comp rx=0 tx=0).stop 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 shutdown_connections 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dbc06cb20 0x7f1dbc06efe0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd815ef00 0x7f1dd80a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd8141b30 0x7f1dd80ab520 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 --2- 172.21.15.107:0/468742245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80cede0 0x7f1dd81f0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 >> 172.21.15.107:0/468742245 conn(0x7f1dd806a500 msgr2=0x7f1dd8140d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 shutdown_connections 2023-11-24T08:20:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:26.626+0000 7f1ddf2ad640 1 -- 172.21.15.107:0/468742245 wait complete. 2023-11-24T08:20:26.646 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:20:26.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:20:26.762 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:20:26.887 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:20:26.888 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:26.888 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:26.947 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:20:26.947 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:26.947 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:27.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:20:27.006 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:27.007 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:27.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:20:27.068 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:27.068 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:27.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:20:27.130 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:20:27.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.349+0000 7f73beac9640 1 Processor -- start 2023-11-24T08:20:27.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.350+0000 7f73beac9640 1 -- start start 2023-11-24T08:20:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.351+0000 7f73beac9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.351+0000 7f73beac9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73b80cf450 0x7f73b80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.351+0000 7f73beac9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b80cece0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.351+0000 7f73beac9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b8156010 con 0x7f73b80cf450 2023-11-24T08:20:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.351+0000 7f73beac9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806ce30 con 0x7f73b815c7f0 2023-11-24T08:20:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73b80cf450 0x7f73b80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73b80cf450 0x7f73b80cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40730/0 (socket says 172.21.15.107:40730) 2023-11-24T08:20:27.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 -- 172.21.15.107:0/203677460 learned_addr learned my addr 172.21.15.107:0/203677460 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 -- 172.21.15.107:0/203677460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 msgr2=0x7f73b806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73bc865640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 -- 172.21.15.107:0/203677460 >> v1:172.21.15.120:6789/0 conn(0x7f73b80cdfb0 legacy=0x7f73b80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73b7fff640 1 -- 172.21.15.107:0/203677460 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b806d040 con 0x7f73b80cf450 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.352+0000 7f73bc865640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.353+0000 7f73b7fff640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cf450 0x7f73b80cf8d0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70da148cb78f921d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.353+0000 7f73b77fe640 1 -- 172.21.15.107:0/203677460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031640 con 0x7f73b80cf450 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73b77fe640 1 -- 172.21.15.107:0/203677460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a40317a0 con 0x7f73b80cf450 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73b77fe640 1 -- 172.21.15.107:0/203677460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031b30 con 0x7f73b80cf450 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 -- 172.21.15.107:0/203677460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cf450 msgr2=0x7f73b80cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cf450 0x7f73b80cf8d0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 -- 172.21.15.107:0/203677460 shutdown_connections 2023-11-24T08:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cf450 0x7f73b80cf8d0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 --2- 172.21.15.107:0/203677460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 -- 172.21.15.107:0/203677460 >> 172.21.15.107:0/203677460 conn(0x7f73b806a920 msgr2=0x7f73b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 -- 172.21.15.107:0/203677460 shutdown_connections 2023-11-24T08:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.354+0000 7f73beac9640 1 -- 172.21.15.107:0/203677460 wait complete. 2023-11-24T08:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.356+0000 7f73beac9640 1 Processor -- start 2023-11-24T08:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.356+0000 7f73beac9640 1 -- start start 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b80cf450 0x7f73b81e7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b81537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8153380 con 0x7f73b80cdfb0 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8151620 con 0x7f73b80cf450 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73beac9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b81e7ad0 con 0x7f73b815c7f0 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73bc865640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73bc865640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40740/0 (socket says 172.21.15.107:40740) 2023-11-24T08:20:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.357+0000 7f73bc865640 1 -- 172.21.15.107:0/4218295279 learned_addr learned my addr 172.21.15.107:0/4218295279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bd066640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b81537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 -- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 msgr2=0x7f73b81537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b81537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 -- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b80cf450 msgr2=0x7f73b81e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73b7fff640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b80cf450 0x7f73b81e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b80cf450 0x7f73b81e7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 -- 172.21.15.107:0/4218295279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40094d0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73bc865640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c7f0 tx=0x7f73ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac007820 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac007980 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac007cf0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.358+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b8152700 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.359+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b8152cc0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.360+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b81535f0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.361+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f73ac0113f0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.362+0000 7f73b57fa640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 0x7f739c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.362+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73ac0888f0 con 0x7f73b80cdfb0 2023-11-24T08:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.362+0000 7f73b7fff640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 0x7f739c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.363+0000 7f73b7fff640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 0x7f739c06efb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f73a4036000 tx=0x7f73a403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.367+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73ac055400 con 0x7f73b80cdfb0 2023-11-24T08:20:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.594+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b8067470 con 0x7f73b80cdfb0 2023-11-24T08:20:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.595+0000 7f73b57fa640 1 -- 172.21.15.107:0/4218295279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f73ac01f090 con 0x7f73b80cdfb0 2023-11-24T08:20:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:27.597 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":110,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2579647412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2579647412},{"type":"v1","addr":"172.21.15.107:6837","nonce":2579647412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4175306234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4175306234},{"type":"v1","addr":"172.21.15.120:6833","nonce":4175306234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2689665464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2689665464},{"type":"v1","addr":"172.21.15.120:6835","nonce":2689665464}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/3159726670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3159726670},{"type":"v1","addr":"172.21.15.107:6836","nonce":3159726670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":110}],"filesystems":[]} 2023-11-24T08:20:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 110 2023-11-24T08:20:27.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 msgr2=0x7f739c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 0x7f739c06efb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f73a4036000 tx=0x7f73a403c040 comp rx=0 tx=0).stop 2023-11-24T08:20:27.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 msgr2=0x7f73b8152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c7f0 tx=0x7f73ac00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 shutdown_connections 2023-11-24T08:20:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f739c06caf0 0x7f739c06efb0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b815c7f0 0x7f73b81537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b80cf450 0x7f73b81e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 --2- 172.21.15.107:0/4218295279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73b80cdfb0 0x7f73b8152020 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 >> 172.21.15.107:0/4218295279 conn(0x7f73b806a920 msgr2=0x7f73b80cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 shutdown_connections 2023-11-24T08:20:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.598+0000 7f73beac9640 1 -- 172.21.15.107:0/4218295279 wait complete. 2023-11-24T08:20:27.620 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:20:27.620 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:20:27.620 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:20:27.620 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:20:27.621 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:20:27.621 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:20:27.621 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:20:27.621 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:20:27.622 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:20:27.622 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:20:27.622 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:20:27.622 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:20:27.622 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:20:27.622 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:20:27.623 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:20:27.623 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:20:27.636 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:20:27.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:20:27.643 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:20:27.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:20:27.645 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:20:27.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:20:27.646 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:20:27.646 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:20:27.690 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:20:27.691 DEBUG:teuthology.parallel:result is None 2023-11-24T08:20:27.694 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:20:27.694 DEBUG:teuthology.parallel:result is None 2023-11-24T08:20:27.698 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:20:27.698 DEBUG:teuthology.parallel:result is None 2023-11-24T08:20:27.700 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:20:27.701 DEBUG:teuthology.parallel:result is None 2023-11-24T08:20:27.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:20:27.851 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:20:27.851 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:20:27.852 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:20:27.854 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:20:27.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.895+0000 7f8e84501640 1 Processor -- start 2023-11-24T08:20:27.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.895+0000 7f8e84501640 1 -- start start 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e84501640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 0x7f8e7c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e84501640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8e7c161080 0x7f8e7c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e84501640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c161600 con 0x7f8e7c140720 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e84501640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7c0cbf70 con 0x7f8e7c161080 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e84501640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c1618e0 con 0x7f8e7c140340 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8e7c161080 0x7f8e7c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8e7c161080 0x7f8e7c0c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59438/0 (socket says 172.21.15.107:59438) 2023-11-24T08:20:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 -- 172.21.15.107:0/1400529345 learned_addr learned my addr 172.21.15.107:0/1400529345 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e8229d640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 0x7f8e7c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 -- 172.21.15.107:0/1400529345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 msgr2=0x7f8e7c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 0x7f8e7c0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.897+0000 7f8e82a9e640 1 -- 172.21.15.107:0/1400529345 >> v1:172.21.15.120:6789/0 conn(0x7f8e7c140720 legacy=0x7f8e7c0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.898+0000 7f8e82a9e640 1 -- 172.21.15.107:0/1400529345 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c161bc0 con 0x7f8e7c161080 2023-11-24T08:20:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.898+0000 7f8e8229d640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 0x7f8e7c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e82a9e640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c60b0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009e50 tx=0x7f8e7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb03e69fae5bb80 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e8129b640 1 -- 172.21.15.107:0/1400529345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7402cb00 con 0x7f8e7c161080 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e8129b640 1 -- 172.21.15.107:0/1400529345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7402cc60 con 0x7f8e7c161080 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e8129b640 1 -- 172.21.15.107:0/1400529345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e740343f0 con 0x7f8e7c161080 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e84501640 1 -- 172.21.15.107:0/1400529345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 msgr2=0x7f8e7c0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.899+0000 7f8e84501640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c60b0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009e50 tx=0x7f8e7402bf60 comp rx=0 tx=0).stop 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 -- 172.21.15.107:0/1400529345 shutdown_connections 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c60b0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 --2- 172.21.15.107:0/1400529345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140340 0x7f8e7c0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 -- 172.21.15.107:0/1400529345 >> 172.21.15.107:0/1400529345 conn(0x7f8e7c0b2e40 msgr2=0x7f8e7c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 -- 172.21.15.107:0/1400529345 shutdown_connections 2023-11-24T08:20:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.900+0000 7f8e84501640 1 -- 172.21.15.107:0/1400529345 wait complete. 2023-11-24T08:20:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.901+0000 7f8e84501640 1 Processor -- start 2023-11-24T08:20:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.902+0000 7f8e84501640 1 -- start start 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.902+0000 7f8e84501640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e7c140340 0x7f8e7c0cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e84501640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140720 0x7f8e7c0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e84501640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e84501640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0b5260 con 0x7f8e7c161080 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e84501640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0c8e50 con 0x7f8e7c140340 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e84501640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0c9130 con 0x7f8e7c140720 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59440/0 (socket says 172.21.15.107:59440) 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 -- 172.21.15.107:0/3089567364 learned_addr learned my addr 172.21.15.107:0/3089567364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:27.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e8229d640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e7c140340 0x7f8e7c0cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e81a9c640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140720 0x7f8e7c0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 -- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140720 msgr2=0x7f8e7c0c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140720 0x7f8e7c0c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 -- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e7c140340 msgr2=0x7f8e7c0cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e7c140340 0x7f8e7c0cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.903+0000 7f8e82a9e640 1 -- 172.21.15.107:0/3089567364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74009b30 con 0x7f8e7c161080 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e82a9e640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009e50 tx=0x7f8e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74040670 con 0x7f8e7c161080 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e740407d0 con 0x7f8e7c161080 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c0c9410 con 0x7f8e7c161080 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74040b80 con 0x7f8e7c161080 2023-11-24T08:20:27.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.904+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c0c99d0 con 0x7f8e7c161080 2023-11-24T08:20:27.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.905+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c04e8b0 con 0x7f8e7c161080 2023-11-24T08:20:27.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.906+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f8e74040ce0 con 0x7f8e7c161080 2023-11-24T08:20:27.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.907+0000 7f8e72ffd640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 0x7f8e6006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:27.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.907+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e740ab520 con 0x7f8e7c161080 2023-11-24T08:20:27.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.907+0000 7f8e8229d640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 0x7f8e6006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:27.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.908+0000 7f8e8229d640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 0x7f8e6006efb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800ab60 tx=0x7f8e78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:27.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:27.911+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e740782a0 con 0x7f8e7c161080 2023-11-24T08:20:28.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.110+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8e7c0d1090 con 0x7f8e7c161080 2023-11-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.193+0000 7f8e72ffd640 1 -- 172.21.15.107:0/3089567364 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v184) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8e7407b8c0 con 0x7f8e7c161080 2023-11-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:20:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 msgr2=0x7f8e6006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 0x7f8e6006efb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800ab60 tx=0x7f8e78008040 comp rx=0 tx=0).stop 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 msgr2=0x7f8e7c0c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009e50 tx=0x7f8e74004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 shutdown_connections 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e6006caf0 0x7f8e6006efb0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e7c161080 0x7f8e7c0c8820 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e7c140720 0x7f8e7c0c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 --2- 172.21.15.107:0/3089567364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e7c140340 0x7f8e7c0cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 >> 172.21.15.107:0/3089567364 conn(0x7f8e7c0b2e40 msgr2=0x7f8e7c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 shutdown_connections 2023-11-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.196+0000 7f8e84501640 1 -- 172.21.15.107:0/3089567364 wait complete. 2023-11-24T08:20:28.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:20:28.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.408+0000 7fd664461640 1 Processor -- start 2023-11-24T08:20:28.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.408+0000 7fd664461640 1 -- start start 2023-11-24T08:20:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.409+0000 7fd664461640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd65c0cd5b0 0x7fd65c0b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd664461640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 0x7fd65c0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd664461640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0711e0 con 0x7fd65c15b5e0 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd664461640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65c0b4540 con 0x7fd65c0cd5b0 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd664461640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0cdc80 con 0x7fd65c0abf50 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6621fd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd65c15b5e0 0x7fd65c0ccea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47500/0 (socket says 172.21.15.107:47500) 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6621fd640 1 -- 172.21.15.107:0/3850091279 learned_addr learned my addr 172.21.15.107:0/3850091279 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270709831 0 0) 0x7fd65c0711e0 con 0x7fd65c15b5e0 2023-11-24T08:20:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003520 con 0x7fd65c15b5e0 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6619fc640 1 --2- 172.21.15.107:0/3850091279 >> v2:172.21.15.120:3300/0 conn(0x7fd65c0cd5b0 0x7fd65c0b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.410+0000 7fd6629fe640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 0x7fd65c0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067583685 0 0) 0x7fd648003520 con 0x7fd65c15b5e0 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 msgr2=0x7fd65c0b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 0x7fd65c0b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 >> v2:172.21.15.120:3300/0 conn(0x7fd65c0cd5b0 msgr2=0x7fd65c0b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 --2- 172.21.15.107:0/3850091279 >> v2:172.21.15.120:3300/0 conn(0x7fd65c0cd5b0 0x7fd65c0b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c0705d0 con 0x7fd65c15b5e0 2023-11-24T08:20:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd658001670 con 0x7fd65c15b5e0 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 >> v1:172.21.15.107:6789/0 conn(0x7fd65c15b5e0 legacy=0x7fd65c0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd648003970 0x7fd648003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 0x7fd6480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648004d00 0x7fd648005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0cdc80 con 0x7fd648003970 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800ba30 con 0x7fd648004d00 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800bb70 con 0x7fd65c0abf50 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd648003970 0x7fd648003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 -- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 msgr2=0x7fd6480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c0abf50 0x7fd6480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 -- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648004d00 msgr2=0x7fd648005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648004d00 0x7fd648005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.411+0000 7fd6621fd640 1 -- 172.21.15.107:0/3850091279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648003520 con 0x7fd648003970 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.412+0000 7fd6621fd640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd648003970 0x7fd648003d90 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c270 tx=0x7fd6480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af33eead382845e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65802cae0 con 0x7fd648003970 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65802cc40 con 0x7fd648003970 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd6611fb640 1 -- 172.21.15.107:0/3850091279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6580058d0 con 0x7fd648003970 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 -- 172.21.15.107:0/3850091279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd648003970 msgr2=0x7fd648003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 --2- 172.21.15.107:0/3850091279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd648003970 0x7fd648003d90 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c270 tx=0x7fd6480057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd6629fe640 1 -- 172.21.15.107:0/3850091279 reap_dead start 2023-11-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 -- 172.21.15.107:0/3850091279 shutdown_connections 2023-11-24T08:20:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 -- 172.21.15.107:0/3850091279 >> 172.21.15.107:0/3850091279 conn(0x7fd65c0b0c70 msgr2=0x7fd65c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 -- 172.21.15.107:0/3850091279 shutdown_connections 2023-11-24T08:20:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.413+0000 7fd664461640 1 -- 172.21.15.107:0/3850091279 wait complete. 2023-11-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.415+0000 7fd664461640 1 Processor -- start 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.415+0000 7fd664461640 1 -- start start 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648003970 0x7fd65c1473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c1f33d0 0x7fd65c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c157b60 con 0x7fd65c15a600 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c15aa80 con 0x7fd648003970 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd664461640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c15ac80 con 0x7fd65c1f33d0 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd6629fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd6629fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59458/0 (socket says 172.21.15.107:59458) 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.416+0000 7fd6629fe640 1 -- 172.21.15.107:0/32218136 learned_addr learned my addr 172.21.15.107:0/32218136 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6621fd640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648003970 0x7fd65c1473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 -- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c1f33d0 msgr2=0x7fd65c15a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c1f33d0 0x7fd65c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 -- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648003970 msgr2=0x7fd65c1473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648003970 0x7fd65c1473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 -- 172.21.15.107:0/32218136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c147900 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd6629fe640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0aea30 tx=0x7fd65400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65400edc0 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654014420 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6540147d0 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c1571c0 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.417+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65c157780 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.418+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c157ca0 con 0x7fd65c15a600 2023-11-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.419+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fd654014930 con 0x7fd65c15a600 2023-11-24T08:20:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.420+0000 7fd652ffd640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 0x7fd64006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.420+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd654088b90 con 0x7fd65c15a600 2023-11-24T08:20:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.421+0000 7fd6621fd640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 0x7fd64006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.421+0000 7fd6621fd640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 0x7fd64006efb0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd658034000 tx=0x7fd658007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.425+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd65c157ca0 con 0x7fd65c15a600 2023-11-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.665+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd65c157ca0 con 0x7fd65c15a600 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.666+0000 7fd652ffd640 1 -- 172.21.15.107:0/32218136 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd65c157ca0 con 0x7fd65c15a600 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.672 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.673 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.681 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:20:28.682 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:20:28.684 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:20:28.685 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.668+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 msgr2=0x7fd64006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.668+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 0x7fd64006efb0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd658034000 tx=0x7fd658007000 comp rx=0 tx=0).stop 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.668+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 msgr2=0x7fd65c1554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.668+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0aea30 tx=0x7fd65400e4b0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 shutdown_connections 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd64006caf0 0x7fd64006efb0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd65c15a600 0x7fd65c1554d0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd65c1f33d0 0x7fd65c15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 --2- 172.21.15.107:0/32218136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd648003970 0x7fd65c1473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 >> 172.21.15.107:0/32218136 conn(0x7fd65c0b0c70 msgr2=0x7fd65c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 shutdown_connections 2023-11-24T08:20:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.669+0000 7fd664461640 1 -- 172.21.15.107:0/32218136 wait complete. 2023-11-24T08:20:28.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.891+0000 7f0919997640 1 Processor -- start 2023-11-24T08:20:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.891+0000 7f0919997640 1 -- start start 2023-11-24T08:20:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0919997640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f091415c7f0 0x7f091406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0919997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f09140cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0919997640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140cece0 con 0x7f09140cf450 2023-11-24T08:20:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0919997640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09140cfe40 con 0x7f091415c7f0 2023-11-24T08:20:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0919997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0914156010 con 0x7f09140ce6c0 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0912ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f091415c7f0 0x7f091406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0912ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f091415c7f0 0x7f091406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59472/0 (socket says 172.21.15.107:59472) 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f0912ffd640 1 -- 172.21.15.107:0/3626349056 learned_addr learned my addr 172.21.15.107:0/3626349056 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.893+0000 7f09137fe640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f09140cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.894+0000 7f0912ffd640 1 -- 172.21.15.107:0/3626349056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 msgr2=0x7f09140cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.894+0000 7f0912ffd640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f09140cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.894+0000 7f0912ffd640 1 -- 172.21.15.107:0/3626349056 >> v1:172.21.15.120:6789/0 conn(0x7f09140cf450 legacy=0x7f09140cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.894+0000 7f0912ffd640 1 -- 172.21.15.107:0/3626349056 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091406d040 con 0x7f091415c7f0 2023-11-24T08:20:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.895+0000 7f0912ffd640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f091415c7f0 0x7f091406db20 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f0904009870 tx=0x7f090402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8235042744757960 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0911ffb640 1 -- 172.21.15.107:0/3626349056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904030840 con 0x7f091415c7f0 2023-11-24T08:20:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0911ffb640 1 -- 172.21.15.107:0/3626349056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09040309a0 con 0x7f091415c7f0 2023-11-24T08:20:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0911ffb640 1 -- 172.21.15.107:0/3626349056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904030d30 con 0x7f091415c7f0 2023-11-24T08:20:28.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 -- 172.21.15.107:0/3626349056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f091415c7f0 msgr2=0x7f091406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f091415c7f0 0x7f091406db20 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f0904009870 tx=0x7f090402b980 comp rx=0 tx=0).stop 2023-11-24T08:20:28.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 -- 172.21.15.107:0/3626349056 shutdown_connections 2023-11-24T08:20:28.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f09140cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 --2- 172.21.15.107:0/3626349056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f091415c7f0 0x7f091406db20 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.896+0000 7f0919997640 1 -- 172.21.15.107:0/3626349056 >> 172.21.15.107:0/3626349056 conn(0x7f091406a920 msgr2=0x7f091406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:28.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.897+0000 7f0919997640 1 -- 172.21.15.107:0/3626349056 shutdown_connections 2023-11-24T08:20:28.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.897+0000 7f0919997640 1 -- 172.21.15.107:0/3626349056 wait complete. 2023-11-24T08:20:28.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.898+0000 7f0919997640 1 Processor -- start 2023-11-24T08:20:28.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.898+0000 7f0919997640 1 -- start start 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f0914152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 0x7f091415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09141538c0 con 0x7f09140cf450 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09141f66f0 con 0x7f091415c7f0 2023-11-24T08:20:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.899+0000 7f0919997640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09141f69d0 con 0x7f09140ce6c0 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59476/0 (socket says 172.21.15.107:59476) 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 -- 172.21.15.107:0/1563318868 learned_addr learned my addr 172.21.15.107:0/1563318868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09137fe640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 0x7f091415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f0912ffd640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f0914152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 -- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 msgr2=0x7f0914152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f0914152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 -- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 msgr2=0x7f091415eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 0x7f091415eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 -- 172.21.15.107:0/1563318868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040094d0 con 0x7f09140cf450 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09137fe640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 0x7f091415eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.900+0000 7f09127fc640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f090800c450 tx=0x7f090800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.901+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09080052d0 con 0x7f09140cf450 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.901+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908005430 con 0x7f09140cf450 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.901+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09141f6cb0 con 0x7f09140cf450 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.901+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908012870 con 0x7f09140cf450 2023-11-24T08:20:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.901+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09141f7270 con 0x7f09140cf450 2023-11-24T08:20:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.902+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0914067470 con 0x7f09140cf450 2023-11-24T08:20:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.903+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f09080129d0 con 0x7f09140cf450 2023-11-24T08:20:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.904+0000 7f08fbfff640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 0x7f090006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.904+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0908088120 con 0x7f09140cf450 2023-11-24T08:20:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.904+0000 7f0912ffd640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 0x7f090006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.905+0000 7f0912ffd640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 0x7f090006efb0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0904036000 tx=0x7f090403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:28.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:28.908+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0908054c30 con 0x7f09140cf450 2023-11-24T08:20:29.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.097+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09140cd730 con 0x7f09140cf450 2023-11-24T08:20:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.099+0000 7f08fbfff640 1 -- 172.21.15.107:0/1563318868 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f0908058250 con 0x7f09140cf450 2023-11-24T08:20:29.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:29.101 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":186,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:28.207475+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 msgr2=0x7f090006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 0x7f090006efb0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0904036000 tx=0x7f090403b040 comp rx=0 tx=0).stop 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 msgr2=0x7f0914152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f090800c450 tx=0x7f090800c910 comp rx=0 tx=0).stop 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 shutdown_connections 2023-11-24T08:20:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f090006caf0 0x7f090006efb0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f091415c7f0 0x7f091415eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09140cf450 0x7f0914152cb0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 --2- 172.21.15.107:0/1563318868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09140ce6c0 0x7f0914152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 >> 172.21.15.107:0/1563318868 conn(0x7f091406a920 msgr2=0x7f09140ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 shutdown_connections 2023-11-24T08:20:29.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.102+0000 7f0919997640 1 -- 172.21.15.107:0/1563318868 wait complete. 2023-11-24T08:20:29.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:20:29.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:20:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.324+0000 7f3d21dc1640 1 Processor -- start 2023-11-24T08:20:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.324+0000 7f3d21dc1640 1 -- start start 2023-11-24T08:20:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.325+0000 7f3d21dc1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 0x7f3d1c071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.325+0000 7f3d21dc1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.325+0000 7f3d21dc1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0b4db0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.325+0000 7f3d21dc1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c15b6f0 con 0x7f3d1c06c5b0 2023-11-24T08:20:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.325+0000 7f3d21dc1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0b3f50 con 0x7f3d1c157ec0 2023-11-24T08:20:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.326+0000 7f3d1bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.326+0000 7f3d20dbf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 0x7f3d1c071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.326+0000 7f3d1bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:49212/0 (socket says 172.21.15.107:49212) 2023-11-24T08:20:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.326+0000 7f3d1bfff640 1 -- 172.21.15.107:0/981097990 learned_addr learned my addr 172.21.15.107:0/981097990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.327+0000 7f3d20dbf640 1 -- 172.21.15.107:0/981097990 >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 msgr2=0x7f3d1c0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.327+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/981097990 >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.327+0000 7f3d20dbf640 1 -- 172.21.15.107:0/981097990 >> v1:172.21.15.120:6789/0 conn(0x7f3d1c0b4950 legacy=0x7f3d1c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.327+0000 7f3d20dbf640 1 -- 172.21.15.107:0/981097990 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0ccc50 con 0x7f3d1c157ec0 2023-11-24T08:20:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.327+0000 7f3d1bfff640 1 --2- 172.21.15.107:0/981097990 >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:20:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.328+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/981097990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 0x7f3d1c071ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d100097a0 tx=0x7f3d1002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58342f3a336bc43 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.329+0000 7f3d1b7fe640 1 -- 172.21.15.107:0/981097990 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1002d040 con 0x7f3d1c157ec0 2023-11-24T08:20:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.329+0000 7f3d1b7fe640 1 -- 172.21.15.107:0/981097990 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d10031810 con 0x7f3d1c157ec0 2023-11-24T08:20:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.329+0000 7f3d1b7fe640 1 -- 172.21.15.107:0/981097990 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10031ba0 con 0x7f3d1c157ec0 2023-11-24T08:20:29.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 -- 172.21.15.107:0/981097990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 msgr2=0x7f3d1c071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/981097990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 0x7f3d1c071ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d100097a0 tx=0x7f3d1002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 -- 172.21.15.107:0/981097990 shutdown_connections 2023-11-24T08:20:29.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/981097990 >> v2:172.21.15.120:3300/0 conn(0x7f3d1c06c5b0 0x7f3d1c0b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/981097990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c157ec0 0x7f3d1c071ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 -- 172.21.15.107:0/981097990 >> 172.21.15.107:0/981097990 conn(0x7f3d1c0ced50 msgr2=0x7f3d1c0cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 -- 172.21.15.107:0/981097990 shutdown_connections 2023-11-24T08:20:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.330+0000 7f3d21dc1640 1 -- 172.21.15.107:0/981097990 wait complete. 2023-11-24T08:20:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.332+0000 7f3d21dc1640 1 Processor -- start 2023-11-24T08:20:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.333+0000 7f3d21dc1640 1 -- start start 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 0x7f3d1c13c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 0x7f3d1c13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0702b0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c070590 con 0x7f3d1c157ec0 2023-11-24T08:20:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d21dc1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c070870 con 0x7f3d1c06c5b0 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59482/0 (socket says 172.21.15.107:59482) 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 -- 172.21.15.107:0/494275124 learned_addr learned my addr 172.21.15.107:0/494275124 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 0x7f3d1c13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d215c0640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 0x7f3d1c13cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 -- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 msgr2=0x7f3d1c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 0x7f3d1c13c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 -- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 msgr2=0x7f3d1c13cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.334+0000 7f3d1bfff640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 0x7f3d1c13cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d1bfff640 1 -- 172.21.15.107:0/494275124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080094d0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 0x7f3d1c13c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d215c0640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 0x7f3d1c13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d1bfff640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800e9b0 tx=0x7f3d0800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0800c9a0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0800cb00 con 0x7f3d1c0b4950 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0800ce70 con 0x7f3d1c0b4950 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d10009400 con 0x7f3d1c0b4950 2023-11-24T08:20:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.335+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c070e30 con 0x7f3d1c0b4950 2023-11-24T08:20:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.336+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c04e840 con 0x7f3d1c0b4950 2023-11-24T08:20:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.337+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f3d08013470 con 0x7f3d1c0b4950 2023-11-24T08:20:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.338+0000 7f3d197fa640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 0x7f3d0006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.338+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d08088bc0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.339+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 0x7f3d0006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.339+0000 7f3d20dbf640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 0x7f3d0006efb0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3d10036000 tx=0x7f3d1003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.342+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d080556d0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.534+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d1c15c1c0 con 0x7f3d1c0b4950 2023-11-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.535+0000 7f3d197fa640 1 -- 172.21.15.107:0/494275124 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v186) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d08021020 con 0x7f3d1c0b4950 2023-11-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 msgr2=0x7f3d0006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 0x7f3d0006efb0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3d10036000 tx=0x7f3d1003c040 comp rx=0 tx=0).stop 2023-11-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 msgr2=0x7f3d1c071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800e9b0 tx=0x7f3d0800ee70 comp rx=0 tx=0).stop 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 shutdown_connections 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3d0006caf0 0x7f3d0006efb0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d1c157ec0 0x7f3d1c13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c0b4950 0x7f3d1c071350 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 --2- 172.21.15.107:0/494275124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1c06c5b0 0x7f3d1c13c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 >> 172.21.15.107:0/494275124 conn(0x7f3d1c0ced50 msgr2=0x7f3d1c1582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 shutdown_connections 2023-11-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.538+0000 7f3d21dc1640 1 -- 172.21.15.107:0/494275124 wait complete. 2023-11-24T08:20:29.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.769+0000 7f0b8d01c640 1 Processor -- start 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.769+0000 7f0b8d01c640 1 -- start start 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8d01c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b88160d40 0x7f0b880b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8d01c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8d01c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b880cdf90 con 0x7f0b880b2740 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8d01c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b880ce230 con 0x7f0b88160d40 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8d01c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880d2430 con 0x7f0b880d04f0 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b86d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b88160d40 0x7f0b880b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b86d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b88160d40 0x7f0b880b21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59498/0 (socket says 172.21.15.107:59498) 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b86d9d640 1 -- 172.21.15.107:0/1178120002 learned_addr learned my addr 172.21.15.107:0/1178120002 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.771+0000 7f0b8759e640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.772+0000 7f0b86d9d640 1 -- 172.21.15.107:0/1178120002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 msgr2=0x7f0b880ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.772+0000 7f0b86d9d640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.772+0000 7f0b86d9d640 1 -- 172.21.15.107:0/1178120002 >> v1:172.21.15.120:6789/0 conn(0x7f0b880b2740 legacy=0x7f0b880cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.772+0000 7f0b86d9d640 1 -- 172.21.15.107:0/1178120002 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880d1b90 con 0x7f0b88160d40 2023-11-24T08:20:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.773+0000 7f0b86d9d640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b88160d40 0x7f0b880b21d0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b7802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f318e6fe042269 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.773+0000 7f0b8659c640 1 -- 172.21.15.107:0/1178120002 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7802ca80 con 0x7f0b88160d40 2023-11-24T08:20:29.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.773+0000 7f0b8659c640 1 -- 172.21.15.107:0/1178120002 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7802cbe0 con 0x7f0b88160d40 2023-11-24T08:20:29.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.773+0000 7f0b8659c640 1 -- 172.21.15.107:0/1178120002 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b780343f0 con 0x7f0b88160d40 2023-11-24T08:20:29.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 -- 172.21.15.107:0/1178120002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b88160d40 msgr2=0x7f0b880b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b88160d40 0x7f0b880b21d0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b7802bee0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 -- 172.21.15.107:0/1178120002 shutdown_connections 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/1178120002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b88160d40 0x7f0b880b21d0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 -- 172.21.15.107:0/1178120002 >> 172.21.15.107:0/1178120002 conn(0x7f0b8808af60 msgr2=0x7f0b880d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 -- 172.21.15.107:0/1178120002 shutdown_connections 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.774+0000 7f0b8d01c640 1 -- 172.21.15.107:0/1178120002 wait complete. 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.776+0000 7f0b8d01c640 1 Processor -- start 2023-11-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.776+0000 7f0b8d01c640 1 -- start start 2023-11-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b88160d40 0x7f0b880b8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880c85f0 con 0x7f0b880b2740 2023-11-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880c7970 con 0x7f0b88160d40 2023-11-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b8d01c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880c7c50 con 0x7f0b880d04f0 2023-11-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b86d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b86d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59502/0 (socket says 172.21.15.107:59502) 2023-11-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.777+0000 7f0b86d9d640 1 -- 172.21.15.107:0/2543506587 learned_addr learned my addr 172.21.15.107:0/2543506587 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 -- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 msgr2=0x7f0b880c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 -- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b88160d40 msgr2=0x7f0b880b8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b88160d40 0x7f0b880b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 -- 172.21.15.107:0/2543506587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009b30 con 0x7f0b880b2740 2023-11-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b86d9d640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7803e450 con 0x7f0b880b2740 2023-11-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7803e5f0 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7803e9a0 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b880c7f30 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.778+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b880b8de0 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.779+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88067470 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.780+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f0b7802cd50 con 0x7f0b880b2740 2023-11-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.781+0000 7f0b7ffff640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 0x7f0b6806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.781+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0b780ab270 con 0x7f0b880b2740 2023-11-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.781+0000 7f0b7e59c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 0x7f0b6806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.782+0000 7f0b7e59c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 0x7f0b6806efb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0b88160290 tx=0x7f0b700079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.785+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b78077d80 con 0x7f0b880b2740 2023-11-24T08:20:29.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.965+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0b8808b400 con 0x7f0b880b2740 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.966+0000 7f0b7ffff640 1 -- 172.21.15.107:0/2543506587 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v186) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0b7807b3a0 con 0x7f0b880b2740 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:20:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:20:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:29.969 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.968+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 msgr2=0x7f0b6806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 0x7f0b6806efb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0b88160290 tx=0x7f0b700079b0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 msgr2=0x7f0b8815f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b78004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 shutdown_connections 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0b6806caf0 0x7f0b6806efb0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b88160d40 0x7f0b880b8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880d04f0 0x7f0b880c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 --2- 172.21.15.107:0/2543506587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880b2740 0x7f0b8815f6a0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:29.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 >> 172.21.15.107:0/2543506587 conn(0x7f0b8808af60 msgr2=0x7f0b880b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:29.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 shutdown_connections 2023-11-24T08:20:29.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:29.969+0000 7f0b8d01c640 1 -- 172.21.15.107:0/2543506587 wait complete. 2023-11-24T08:20:29.989 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:20:29.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:20:30.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.204+0000 7ffab6ba5640 1 Processor -- start 2023-11-24T08:20:30.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.204+0000 7ffab6ba5640 1 -- start start 2023-11-24T08:20:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.205+0000 7ffab6ba5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffab015d740 0x7ffab00d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.205+0000 7ffab6ba5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00ce2c0 con 0x7ffab00cf080 2023-11-24T08:20:30.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffab6ba5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00ce5a0 con 0x7ffab00d2100 2023-11-24T08:20:30.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffab6ba5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab00cec60 con 0x7ffab015d740 2023-11-24T08:20:30.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffaa7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffab00cf080 0x7ffab00d19f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47504/0 (socket says 172.21.15.107:47504) 2023-11-24T08:20:30.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffaa7fff640 1 -- 172.21.15.107:0/3540748553 learned_addr learned my addr 172.21.15.107:0/3540748553 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:30.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffab4941640 1 --2- 172.21.15.107:0/3540748553 >> v2:172.21.15.120:3300/0 conn(0x7ffab015d740 0x7ffab00d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513932780 0 0) 0x7ffab00ce2c0 con 0x7ffab00cf080 2023-11-24T08:20:30.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98003520 con 0x7ffab00cf080 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.206+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 326924979 0 0) 0x7ffa98003520 con 0x7ffab00cf080 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 >> v2:172.21.15.120:3300/0 conn(0x7ffab015d740 msgr2=0x7ffab00d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/3540748553 >> v2:172.21.15.120:3300/0 conn(0x7ffab015d740 0x7ffab00d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 >> v1:172.21.15.120:6789/0 conn(0x7ffab00d2100 legacy=0x7ffab00b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00ce880 con 0x7ffab00cf080 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ffaa0001670 con 0x7ffab00cf080 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 >> v1:172.21.15.107:6789/0 conn(0x7ffab00cf080 legacy=0x7ffab00d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 0x7ffa98003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980047c0 0x7ffa98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa98005180 0x7ffa9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00cec60 con 0x7ffa980039c0 2023-11-24T08:20:30.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800c080 con 0x7ffa980047c0 2023-11-24T08:20:30.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800c240 con 0x7ffa98005180 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 0x7ffa98003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 -- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa98005180 msgr2=0x7ffa9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa98005180 0x7ffa9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 -- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980047c0 msgr2=0x7ffa98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980047c0 0x7ffa98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab4941640 1 -- 172.21.15.107:0/3540748553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98003520 con 0x7ffa980039c0 2023-11-24T08:20:30.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.207+0000 7ffab5142640 1 -- 172.21.15.107:0/3540748553 reap_dead start 2023-11-24T08:20:30.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.209+0000 7ffab4941640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 0x7ffa98003de0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009ac0 tx=0x7ffa980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145f2a3f96210b67 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:30.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.209+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac02c710 con 0x7ffa980039c0 2023-11-24T08:20:30.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.209+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac02c870 con 0x7ffa980039c0 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.209+0000 7ffaa77fe640 1 -- 172.21.15.107:0/3540748553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac02cc00 con 0x7ffa980039c0 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 -- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 msgr2=0x7ffa98003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 0x7ffa98003de0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009ac0 tx=0x7ffa980057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 -- 172.21.15.107:0/3540748553 shutdown_connections 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/3540748553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa980039c0 0x7ffa98003de0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 -- 172.21.15.107:0/3540748553 >> 172.21.15.107:0/3540748553 conn(0x7ffab008af60 msgr2=0x7ffab00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 -- 172.21.15.107:0/3540748553 shutdown_connections 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.210+0000 7ffab6ba5640 1 -- 172.21.15.107:0/3540748553 wait complete. 2023-11-24T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.212+0000 7ffab6ba5640 1 Processor -- start 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.212+0000 7ffab6ba5640 1 -- start start 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980039c0 0x7ffab015dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00b85e0 0x7ffab00b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00b59e0 con 0x7ffab00b48f0 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00b8f80 con 0x7ffa980039c0 2023-11-24T08:20:30.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab6ba5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00b5700 con 0x7ffab00b85e0 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab5142640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab5142640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59532/0 (socket says 172.21.15.107:59532) 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffab5142640 1 -- 172.21.15.107:0/4153977945 learned_addr learned my addr 172.21.15.107:0/4153977945 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.213+0000 7ffaa7fff640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00b85e0 0x7ffab00b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab4941640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980039c0 0x7ffab015dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 -- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00b85e0 msgr2=0x7ffab00b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:30.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00b85e0 0x7ffab00b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 -- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980039c0 msgr2=0x7ffab015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980039c0 0x7ffab015dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 -- 172.21.15.107:0/4153977945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab01472d0 con 0x7ffab00b48f0 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab5142640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ffab00b5fe0 tx=0x7ffaa800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8005d40 con 0x7ffab00b48f0 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa800ced0 con 0x7ffab00b48f0 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab01475b0 con 0x7ffab00b48f0 2023-11-24T08:20:30.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800a850 con 0x7ffab00b48f0 2023-11-24T08:20:30.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.214+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00b3400 con 0x7ffab00b48f0 2023-11-24T08:20:30.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.215+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab0147130 con 0x7ffab00b48f0 2023-11-24T08:20:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.216+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7ffaa800a9b0 con 0x7ffab00b48f0 2023-11-24T08:20:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.217+0000 7ffaa57fa640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 0x7ffa9c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.217+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffaa8087420 con 0x7ffab00b48f0 2023-11-24T08:20:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.217+0000 7ffab4941640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 0x7ffa9c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.218+0000 7ffab4941640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 0x7ffa9c06efe0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ffaac036000 tx=0x7ffaac030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.222+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffab0147130 con 0x7ffab00b48f0 2023-11-24T08:20:30.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:30.412+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffab0147ad0 con 0x7ffab00b48f0 2023-11-24T08:20:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.228+0000 7ffaa57fa640 1 -- 172.21.15.107:0/4153977945 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v187) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffab0147ad0 con 0x7ffab00b48f0 2023-11-24T08:20:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.231+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 msgr2=0x7ffa9c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.231+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 0x7ffa9c06efe0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ffaac036000 tx=0x7ffaac030000 comp rx=0 tx=0).stop 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.231+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 msgr2=0x7ffab00b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.231+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ffab00b5fe0 tx=0x7ffaa800cb90 comp rx=0 tx=0).stop 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 shutdown_connections 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa9c06cb20 0x7ffa9c06efe0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffab00b48f0 0x7ffab00b4d70 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00b85e0 0x7ffab00b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 --2- 172.21.15.107:0/4153977945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa980039c0 0x7ffab015dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 >> 172.21.15.107:0/4153977945 conn(0x7ffab008af60 msgr2=0x7ffab0159970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 shutdown_connections 2023-11-24T08:20:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.232+0000 7ffab6ba5640 1 -- 172.21.15.107:0/4153977945 wait complete. 2023-11-24T08:20:31.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:20:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.455+0000 7f79daea3640 1 Processor -- start 2023-11-24T08:20:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.455+0000 7f79daea3640 1 -- start start 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.456+0000 7f79daea3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f79d40cf390 0x7f79d40ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.456+0000 7f79daea3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d40ce000 con 0x7f79d40cb700 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.456+0000 7f79daea3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d40cfab0 con 0x7f79d4158ce0 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.456+0000 7f79daea3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d40cd6d0 con 0x7f79d40cf390 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.457+0000 7f79da6a2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79d40cb700 0x7f79d40d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47520/0 (socket says 172.21.15.107:47520) 2023-11-24T08:20:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.457+0000 7f79da6a2640 1 -- 172.21.15.107:0/3423524301 learned_addr learned my addr 172.21.15.107:0/3423524301 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.457+0000 7f79d96a0640 1 --2- 172.21.15.107:0/3423524301 >> v2:172.21.15.120:3300/0 conn(0x7f79d40cf390 0x7f79d40ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.457+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765971247 0 0) 0x7f79d40ce000 con 0x7f79d40cb700 2023-11-24T08:20:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.457+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c0003520 con 0x7f79d40cb700 2023-11-24T08:20:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4191104531 0 0) 0x7f79c0003520 con 0x7f79d40cb700 2023-11-24T08:20:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 >> v2:172.21.15.120:3300/0 conn(0x7f79d40cf390 msgr2=0x7f79d40ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 --2- 172.21.15.107:0/3423524301 >> v2:172.21.15.120:3300/0 conn(0x7f79d40cf390 0x7f79d40ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 >> v1:172.21.15.120:6789/0 conn(0x7f79d4158ce0 legacy=0x7f79d40cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40cd9b0 con 0x7f79d40cb700 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f79cc001670 con 0x7f79d40cb700 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 >> v1:172.21.15.107:6789/0 conn(0x7f79d40cb700 legacy=0x7f79d40d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79c0003970 0x7f79c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79c00047c0 0x7f79c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40cd6d0 con 0x7f79c0005180 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000c020 con 0x7f79c0003970 2023-11-24T08:20:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000c1e0 con 0x7f79c00047c0 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.458+0000 7f79da6a2640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79d96a0640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79c00047c0 0x7f79c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 -- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79c00047c0 msgr2=0x7f79c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79c00047c0 0x7f79c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 -- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79c0003970 msgr2=0x7f79c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79c0003970 0x7f79c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 -- 172.21.15.107:0/3423524301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0003520 con 0x7f79c0005180 2023-11-24T08:20:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.459+0000 7f79da6a2640 1 -- 172.21.15.107:0/3423524301 reap_dead start 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79da6a2640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79c000b980 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c240 tx=0x7f79c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8966f5196b39ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc005ad0 con 0x7f79c0005180 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc005c70 con 0x7f79c0005180 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79d8e9f640 1 -- 172.21.15.107:0/3423524301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc02c8b0 con 0x7f79c0005180 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79daea3640 1 -- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 msgr2=0x7f79c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79daea3640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79c000b980 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c240 tx=0x7f79c00057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79daea3640 1 -- 172.21.15.107:0/3423524301 shutdown_connections 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79daea3640 1 --2- 172.21.15.107:0/3423524301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79c000b980 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.460+0000 7f79daea3640 1 -- 172.21.15.107:0/3423524301 >> 172.21.15.107:0/3423524301 conn(0x7f79d4060f20 msgr2=0x7f79d4061300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.461+0000 7f79daea3640 1 -- 172.21.15.107:0/3423524301 shutdown_connections 2023-11-24T08:20:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.461+0000 7f79daea3640 1 -- 172.21.15.107:0/3423524301 wait complete. 2023-11-24T08:20:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.462+0000 7f79daea3640 1 Processor -- start 2023-11-24T08:20:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.462+0000 7f79daea3640 1 -- start start 2023-11-24T08:20:31.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 0x7f79d4158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 0x7f79d40c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c8e50 con 0x7f79c0005180 2023-11-24T08:20:31.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c7450 con 0x7f79d40c6fd0 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.463+0000 7f79daea3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4158740 con 0x7f79d40c9270 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59538/0 (socket says 172.21.15.107:59538) 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 -- 172.21.15.107:0/1978168317 learned_addr learned my addr 172.21.15.107:0/1978168317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79da6a2640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 0x7f79d40c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d96a0640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 0x7f79d4158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 -- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 msgr2=0x7f79d40c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 0x7f79d40c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 -- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 msgr2=0x7f79d4158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 0x7f79d4158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d9ea1640 1 -- 172.21.15.107:0/1978168317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40c8820 con 0x7f79c0005180 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79da6a2640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 0x7f79d40c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.464+0000 7f79d96a0640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 0x7f79d4158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79d9ea1640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f79c800c500 tx=0x7f79c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c80052d0 con 0x7f79c0005180 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8005450 con 0x7f79c0005180 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8003830 con 0x7f79c0005180 2023-11-24T08:20:31.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d40b0510 con 0x7f79c0005180 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.465+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d40c7970 con 0x7f79c0005180 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.466+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d404e840 con 0x7f79c0005180 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.467+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f79c8003990 con 0x7f79c0005180 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.468+0000 7f79c67fc640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 0x7f79b806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.468+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f79c8087bd0 con 0x7f79c0005180 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.469+0000 7f79d96a0640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 0x7f79b806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:31.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.469+0000 7f79d96a0640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 0x7f79b806efb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f79d406f810 tx=0x7f79d000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:31.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.472+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79c8054810 con 0x7f79c0005180 2023-11-24T08:20:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:31.665+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f79d40c8600 con 0x7f79c0005180 2023-11-24T08:20:32.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.243+0000 7f79c67fc640 1 -- 172.21.15.107:0/1978168317 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v188) v1 ==== 170+0+0 (secure 0 0 0) 0x7f79c80200d0 con 0x7f79c0005180 2023-11-24T08:20:32.246 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:20:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 msgr2=0x7f79b806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 0x7f79b806efb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f79d406f810 tx=0x7f79d000d040 comp rx=0 tx=0).stop 2023-11-24T08:20:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 msgr2=0x7f79d40c5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f79c800c500 tx=0x7f79c800c9c0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 shutdown_connections 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79b806caf0 0x7f79b806efb0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79d40c9270 0x7f79d40c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79d40c6fd0 0x7f79d4158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 --2- 172.21.15.107:0/1978168317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79c0005180 0x7f79d40c5fe0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 >> 172.21.15.107:0/1978168317 conn(0x7f79d4060f20 msgr2=0x7f79d415b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 shutdown_connections 2023-11-24T08:20:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.248+0000 7f79daea3640 1 -- 172.21.15.107:0/1978168317 wait complete. 2023-11-24T08:20:32.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:20:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.469+0000 7fdd62b73640 1 Processor -- start 2023-11-24T08:20:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.470+0000 7fdd62b73640 1 -- start start 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd62b73640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd5c072160 0x7fdd5c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd62b73640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd5c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd62b73640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c15ec90 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd62b73640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cd530 con 0x7fdd5c072160 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd62b73640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0c4ed0 con 0x7fdd5c06de10 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd5bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdd5c0c49f0 0x7fdd5c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47530/0 (socket says 172.21.15.107:47530) 2023-11-24T08:20:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.471+0000 7fdd5bfff640 1 -- 172.21.15.107:0/1037973665 learned_addr learned my addr 172.21.15.107:0/1037973665 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd6090f640 1 --2- 172.21.15.107:0/1037973665 >> v2:172.21.15.120:3300/0 conn(0x7fdd5c072160 0x7fdd5c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd61110640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd5c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106680737 0 0) 0x7fdd44001110 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd48003520 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162071732 0 0) 0x7fdd48003520 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 msgr2=0x7fdd5c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd5c0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 >> v2:172.21.15.120:3300/0 conn(0x7fdd5c072160 msgr2=0x7fdd5c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 --2- 172.21.15.107:0/1037973665 >> v2:172.21.15.120:3300/0 conn(0x7fdd5c072160 0x7fdd5c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0c51b0 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdd44001800 con 0x7fdd5c0c49f0 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 >> v1:172.21.15.107:6789/0 conn(0x7fdd5c0c49f0 legacy=0x7fdd5c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.472+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cd530 con 0x7fdd480047c0 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0c4ed0 con 0x7fdd48003970 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4800b980 con 0x7fdd5c06de10 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd61110640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 -- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 msgr2=0x7fdd48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd5c06de10 0x7fdd48005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 -- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd48003970 msgr2=0x7fdd48003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.473+0000 7fdd5bfff640 1 -- 172.21.15.107:0/1037973665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd48003520 con 0x7fdd480047c0 2023-11-24T08:20:32.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.474+0000 7fdd5bfff640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c110 tx=0x7fdd480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fce67d730f4974 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.474+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44005ad0 con 0x7fdd480047c0 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.474+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44005c50 con 0x7fdd480047c0 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.474+0000 7fdd5b7fe640 1 -- 172.21.15.107:0/1037973665 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4402c810 con 0x7fdd480047c0 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 -- 172.21.15.107:0/1037973665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd480047c0 msgr2=0x7fdd48004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 --2- 172.21.15.107:0/1037973665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c110 tx=0x7fdd480057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd61110640 1 -- 172.21.15.107:0/1037973665 reap_dead start 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 -- 172.21.15.107:0/1037973665 shutdown_connections 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 -- 172.21.15.107:0/1037973665 >> 172.21.15.107:0/1037973665 conn(0x7fdd5c0b0c50 msgr2=0x7fdd5c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 -- 172.21.15.107:0/1037973665 shutdown_connections 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.475+0000 7fdd62b73640 1 -- 172.21.15.107:0/1037973665 wait complete. 2023-11-24T08:20:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.477+0000 7fdd62b73640 1 Processor -- start 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.477+0000 7fdd62b73640 1 -- start start 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd480047c0 0x7fdd5c159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd5c0c6370 0x7fdd5c148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c149860 con 0x7fdd5c1493e0 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c146b50 con 0x7fdd5c0c6370 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd62b73640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cadf0 con 0x7fdd480047c0 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.478+0000 7fdd61110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59550/0 (socket says 172.21.15.107:59550) 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 -- 172.21.15.107:0/83005722 learned_addr learned my addr 172.21.15.107:0/83005722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd6090f640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd480047c0 0x7fdd5c159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd5bfff640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd5c0c6370 0x7fdd5c148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 -- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd480047c0 msgr2=0x7fdd5c159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd480047c0 0x7fdd5c159e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 -- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd5c0c6370 msgr2=0x7fdd5c148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd5c0c6370 0x7fdd5c148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 -- 172.21.15.107:0/83005722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0cb0d0 con 0x7fdd5c1493e0 2023-11-24T08:20:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd61110640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c06ead0 tx=0x7fdd5400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.479+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54017400 con 0x7fdd5c1493e0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.480+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd54011040 con 0x7fdd5c1493e0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.480+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd540177c0 con 0x7fdd5c1493e0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.480+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0cb3b0 con 0x7fdd5c1493e0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.480+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0cb970 con 0x7fdd5c1493e0 2023-11-24T08:20:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.481+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c0cbb90 con 0x7fdd5c1493e0 2023-11-24T08:20:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.482+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fdd54017920 con 0x7fdd5c1493e0 2023-11-24T08:20:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.483+0000 7fdd597fa640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 0x7fdd3c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.483+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fdd540889d0 con 0x7fdd5c1493e0 2023-11-24T08:20:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.483+0000 7fdd6090f640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 0x7fdd3c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.484+0000 7fdd6090f640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 0x7fdd3c06efb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c004770 tx=0x7fdd4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.488+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd540554e0 con 0x7fdd5c1493e0 2023-11-24T08:20:32.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:32.716+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdd5c04e8b0 con 0x7fdd5c1493e0 2023-11-24T08:20:33.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.261+0000 7fdd597fa640 1 -- 172.21.15.107:0/83005722 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v113) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdd54058b00 con 0x7fdd5c1493e0 2023-11-24T08:20:33.265 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 26 and data pool 27 2023-11-24T08:20:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.270+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 msgr2=0x7fdd3c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.270+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 0x7fdd3c06efb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c004770 tx=0x7fdd4c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.270+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 msgr2=0x7fdd5c09bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.270+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c06ead0 tx=0x7fdd5400f4a0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 shutdown_connections 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdd3c06caf0 0x7fdd3c06efb0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd5c1493e0 0x7fdd5c09bb10 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd5c0c6370 0x7fdd5c148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 --2- 172.21.15.107:0/83005722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd480047c0 0x7fdd5c159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.273+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 >> 172.21.15.107:0/83005722 conn(0x7fdd5c0b0c50 msgr2=0x7fdd5c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:33.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.274+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 shutdown_connections 2023-11-24T08:20:33.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.274+0000 7fdd62b73640 1 -- 172.21.15.107:0/83005722 wait complete. 2023-11-24T08:20:33.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:33.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.477+0000 7fa659c90640 1 Processor -- start 2023-11-24T08:20:33.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.477+0000 7fa659c90640 1 -- start start 2023-11-24T08:20:33.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa659c90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa65415c7f0 0x7fa65406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa659c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa6540cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa659c90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540cece0 con 0x7fa6540cf450 2023-11-24T08:20:33.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa659c90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6540cfe40 con 0x7fa65415c7f0 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa659c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa654156010 con 0x7fa6540ce6c0 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa6537fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa65415c7f0 0x7fa65406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa6537fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa65415c7f0 0x7fa65406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59554/0 (socket says 172.21.15.107:59554) 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa6537fe640 1 -- 172.21.15.107:0/2901578988 learned_addr learned my addr 172.21.15.107:0/2901578988 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa653fff640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa6540cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa6537fe640 1 -- 172.21.15.107:0/2901578988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 msgr2=0x7fa6540cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.479+0000 7fa6537fe640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa6540cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.480+0000 7fa6537fe640 1 -- 172.21.15.107:0/2901578988 >> v1:172.21.15.120:6789/0 conn(0x7fa6540cf450 legacy=0x7fa6540cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.480+0000 7fa6537fe640 1 -- 172.21.15.107:0/2901578988 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65406d040 con 0x7fa65415c7f0 2023-11-24T08:20:33.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa6537fe640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa65406db20 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7fa644009870 tx=0x7fa64402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0466f903fa91c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa652ffd640 1 -- 172.21.15.107:0/2901578988 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644031640 con 0x7fa65415c7f0 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa652ffd640 1 -- 172.21.15.107:0/2901578988 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6440317a0 con 0x7fa65415c7f0 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa652ffd640 1 -- 172.21.15.107:0/2901578988 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644031b30 con 0x7fa65415c7f0 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa659c90640 1 -- 172.21.15.107:0/2901578988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 msgr2=0x7fa65406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.481+0000 7fa659c90640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa65406db20 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7fa644009870 tx=0x7fa64402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 -- 172.21.15.107:0/2901578988 shutdown_connections 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa6540cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 --2- 172.21.15.107:0/2901578988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa65406db20 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 -- 172.21.15.107:0/2901578988 >> 172.21.15.107:0/2901578988 conn(0x7fa65406a920 msgr2=0x7fa65406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 -- 172.21.15.107:0/2901578988 shutdown_connections 2023-11-24T08:20:33.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.482+0000 7fa659c90640 1 -- 172.21.15.107:0/2901578988 wait complete. 2023-11-24T08:20:33.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.483+0000 7fa659c90640 1 Processor -- start 2023-11-24T08:20:33.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.483+0000 7fa659c90640 1 -- start start 2023-11-24T08:20:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa654152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6540cf450 0x7fa6541442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540ac810 con 0x7fa65415c7f0 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540b3db0 con 0x7fa6540cf450 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa659c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540ac530 con 0x7fa6540ce6c0 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.484+0000 7fa653fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59560/0 (socket says 172.21.15.107:59560) 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 -- 172.21.15.107:0/26614407 learned_addr learned my addr 172.21.15.107:0/26614407 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 -- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 msgr2=0x7fa654152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa654152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 -- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6540cf450 msgr2=0x7fa6541442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6540cf450 0x7fa6541442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 -- 172.21.15.107:0/26614407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6440094d0 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa653fff640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00cd20 tx=0x7fa64c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c003b90 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c003cf0 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c004240 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.485+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6540b37f0 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.486+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa654153100 con 0x7fa65415c7f0 2023-11-24T08:20:33.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.486+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6540b3650 con 0x7fa65415c7f0 2023-11-24T08:20:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.488+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fa64c0043a0 con 0x7fa65415c7f0 2023-11-24T08:20:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.489+0000 7fa650ff9640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 0x7fa63406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.489+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa64c012070 con 0x7fa65415c7f0 2023-11-24T08:20:33.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.489+0000 7fa6537fe640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 0x7fa63406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.490+0000 7fa6537fe640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 0x7fa63406efb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa644036000 tx=0x7fa644030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.493+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6540b3650 con 0x7fa65415c7f0 2023-11-24T08:20:33.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.684+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa654151930 con 0x7fa65415c7f0 2023-11-24T08:20:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.687+0000 7fa650ff9640 1 -- 172.21.15.107:0/26614407 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+16596 (secure 0 0 0) 0x7fa654151930 con 0x7fa65415c7f0 2023-11-24T08:20:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:33.690 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":189,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:32.718895+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 msgr2=0x7fa63406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 0x7fa63406efb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa644036000 tx=0x7fa644030000 comp rx=0 tx=0).stop 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 msgr2=0x7fa654151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00cd20 tx=0x7fa64c00a660 comp rx=0 tx=0).stop 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 shutdown_connections 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63406caf0 0x7fa63406efb0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65415c7f0 0x7fa654151c10 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6540cf450 0x7fa6541442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 --2- 172.21.15.107:0/26614407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6540ce6c0 0x7fa654152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 >> 172.21.15.107:0/26614407 conn(0x7fa65406a920 msgr2=0x7fa6540cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 shutdown_connections 2023-11-24T08:20:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.690+0000 7fa659c90640 1 -- 172.21.15.107:0/26614407 wait complete. 2023-11-24T08:20:33.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.908+0000 7fb2d3490640 1 Processor -- start 2023-11-24T08:20:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.908+0000 7fb2d3490640 1 -- start start 2023-11-24T08:20:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d3490640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb2cc0b2250 0x7fb2cc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d3490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 0x7fb2cc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d3490640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc15ab80 con 0x7fb2cc0b2630 2023-11-24T08:20:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d3490640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc15a990 con 0x7fb2cc0b2250 2023-11-24T08:20:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d3490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc15e680 con 0x7fb2cc0b2aa0 2023-11-24T08:20:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.910+0000 7fb2d122c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb2cc0b2250 0x7fb2cc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb2cc0b2250 0x7fb2cc15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:49298/0 (socket says 172.21.15.107:49298) 2023-11-24T08:20:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d1a2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 0x7fb2cc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 -- 172.21.15.107:0/3241698760 learned_addr learned my addr 172.21.15.107:0/3241698760 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 -- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 msgr2=0x7fb2cc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 0x7fb2cc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 -- 172.21.15.107:0/3241698760 >> v1:172.21.15.120:6789/0 conn(0x7fb2cc0b2630 legacy=0x7fb2cc0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d122c640 1 -- 172.21.15.107:0/3241698760 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc15e960 con 0x7fb2cc0b2250 2023-11-24T08:20:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.911+0000 7fb2d1a2d640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 0x7fb2cc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:20:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.913+0000 7fb2d122c640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2250 0x7fb2cc15df80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009870 tx=0x7fb2c802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab4cbfdd4a6049c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.913+0000 7fb2c3fff640 1 -- 172.21.15.107:0/3241698760 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8030840 con 0x7fb2cc0b2250 2023-11-24T08:20:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.913+0000 7fb2c3fff640 1 -- 172.21.15.107:0/3241698760 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c80309a0 con 0x7fb2cc0b2250 2023-11-24T08:20:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.913+0000 7fb2c3fff640 1 -- 172.21.15.107:0/3241698760 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8030d30 con 0x7fb2cc0b2250 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 -- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2250 msgr2=0x7fb2cc15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2250 0x7fb2cc15df80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009870 tx=0x7fb2c802b980 comp rx=0 tx=0).stop 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 -- 172.21.15.107:0/3241698760 shutdown_connections 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2aa0 0x7fb2cc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 --2- 172.21.15.107:0/3241698760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2250 0x7fb2cc15df80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 -- 172.21.15.107:0/3241698760 >> 172.21.15.107:0/3241698760 conn(0x7fb2cc06a920 msgr2=0x7fb2cc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 -- 172.21.15.107:0/3241698760 shutdown_connections 2023-11-24T08:20:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.914+0000 7fb2d3490640 1 -- 172.21.15.107:0/3241698760 wait complete. 2023-11-24T08:20:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.915+0000 7fb2d3490640 1 Processor -- start 2023-11-24T08:20:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 -- start start 2023-11-24T08:20:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2250 0x7fb2cc144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2aa0 0x7fb2cc099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc09ca50 con 0x7fb2cc0b2630 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc156230 con 0x7fb2cc0b2aa0 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.916+0000 7fb2d3490640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc144b80 con 0x7fb2cc0b2250 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59576/0 (socket says 172.21.15.107:59576) 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 -- 172.21.15.107:0/1545763970 learned_addr learned my addr 172.21.15.107:0/1545763970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d1a2d640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2aa0 0x7fb2cc099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d122c640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2250 0x7fb2cc144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 -- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2250 msgr2=0x7fb2cc144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2250 0x7fb2cc144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 -- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2aa0 msgr2=0x7fb2cc099680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2aa0 0x7fb2cc099680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.917+0000 7fb2d0a2b640 1 -- 172.21.15.107:0/1545763970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4009a70 con 0x7fb2cc0b2630 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2d0a2b640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c450 tx=0x7fb2bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0052d0 con 0x7fb2cc0b2630 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc005430 con 0x7fb2cc0b2630 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c80094d0 con 0x7fb2cc0b2630 2023-11-24T08:20:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc012870 con 0x7fb2cc0b2630 2023-11-24T08:20:33.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.918+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0cd7c0 con 0x7fb2cc0b2630 2023-11-24T08:20:33.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.919+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc09cb90 con 0x7fb2cc0b2630 2023-11-24T08:20:33.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.921+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb2bc0129d0 con 0x7fb2cc0b2630 2023-11-24T08:20:33.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.922+0000 7fb2c1ffb640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 0x7fb2b006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:33.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.922+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2bc088120 con 0x7fb2cc0b2630 2023-11-24T08:20:33.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.922+0000 7fb2d122c640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 0x7fb2b006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:33.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.923+0000 7fb2d122c640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 0x7fb2b006efb0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8036000 tx=0x7fb2c803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:33.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:33.926+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2cc09cb90 con 0x7fb2cc0b2630 2023-11-24T08:20:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.102+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2cc09cb90 con 0x7fb2cc0b2630 2023-11-24T08:20:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.104+0000 7fb2c1ffb640 1 -- 172.21.15.107:0/1545763970 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+16596 (secure 0 0 0) 0x7fb2cc09cb90 con 0x7fb2cc0b2630 2023-11-24T08:20:34.107 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:34.107 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":189,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:32.718895+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 msgr2=0x7fb2b006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 0x7fb2b006efb0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8036000 tx=0x7fb2c803b040 comp rx=0 tx=0).stop 2023-11-24T08:20:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 msgr2=0x7fb2cc099140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c450 tx=0x7fb2bc00c910 comp rx=0 tx=0).stop 2023-11-24T08:20:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 shutdown_connections 2023-11-24T08:20:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb2b006caf0 0x7fb2b006efb0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2cc0b2aa0 0x7fb2cc099680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2cc0b2630 0x7fb2cc099140 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 --2- 172.21.15.107:0/1545763970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2cc0b2250 0x7fb2cc144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 >> 172.21.15.107:0/1545763970 conn(0x7fb2cc06a920 msgr2=0x7fb2cc0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.107+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 shutdown_connections 2023-11-24T08:20:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.108+0000 7fb2d3490640 1 -- 172.21.15.107:0/1545763970 wait complete. 2023-11-24T08:20:34.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:20:34.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.339+0000 7fb8f47fb640 1 Processor -- start 2023-11-24T08:20:34.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.339+0000 7fb8f47fb640 1 -- start start 2023-11-24T08:20:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f47fb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8ec06e160 0x7fb8ec0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f47fb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8ec0cd110 0x7fb8ec070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f47fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec161a10 con 0x7fb8ec146eb0 2023-11-24T08:20:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f47fb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec0cd580 con 0x7fb8ec0cd110 2023-11-24T08:20:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f47fb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec161cf0 con 0x7fb8ec06e160 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f2d98640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8ec0cd110 0x7fb8ec070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f2597640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8ec146eb0 0x7fb8ec161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47542/0 (socket says 172.21.15.107:47542) 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.341+0000 7fb8f2597640 1 -- 172.21.15.107:0/972791183 learned_addr learned my addr 172.21.15.107:0/972791183 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f1d96640 1 --2- 172.21.15.107:0/972791183 >> v2:172.21.15.120:3300/0 conn(0x7fb8ec06e160 0x7fb8ec0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f2d98640 1 -- 172.21.15.107:0/972791183 >> v2:172.21.15.120:3300/0 conn(0x7fb8ec06e160 msgr2=0x7fb8ec0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f2d98640 1 --2- 172.21.15.107:0/972791183 >> v2:172.21.15.120:3300/0 conn(0x7fb8ec06e160 0x7fb8ec0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f2d98640 1 -- 172.21.15.107:0/972791183 >> v1:172.21.15.107:6789/0 conn(0x7fb8ec146eb0 legacy=0x7fb8ec161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f2d98640 1 -- 172.21.15.107:0/972791183 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec161fd0 con 0x7fb8ec0cd110 2023-11-24T08:20:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.342+0000 7fb8f1595640 1 -- 172.21.15.107:0/972791183 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810623386 0 0) 0x7fb8ec161a10 con 0x7fb8ec146eb0 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.343+0000 7fb8f2d98640 1 --2- 172.21.15.107:0/972791183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec0cd110 0x7fb8ec070e40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec06eab0 tx=0x7fb8e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8f47c41d271bef server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.343+0000 7fb8f1595640 1 -- 172.21.15.107:0/972791183 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e402cc80 con 0x7fb8ec0cd110 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.343+0000 7fb8f1595640 1 -- 172.21.15.107:0/972791183 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e402cde0 con 0x7fb8ec0cd110 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.343+0000 7fb8f1595640 1 -- 172.21.15.107:0/972791183 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4034660 con 0x7fb8ec0cd110 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 -- 172.21.15.107:0/972791183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec0cd110 msgr2=0x7fb8ec070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/972791183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec0cd110 0x7fb8ec070e40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec06eab0 tx=0x7fb8e402bed0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 -- 172.21.15.107:0/972791183 shutdown_connections 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/972791183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec0cd110 0x7fb8ec070e40 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/972791183 >> v2:172.21.15.120:3300/0 conn(0x7fb8ec06e160 0x7fb8ec0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 -- 172.21.15.107:0/972791183 >> 172.21.15.107:0/972791183 conn(0x7fb8ec0b0c50 msgr2=0x7fb8ec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 -- 172.21.15.107:0/972791183 shutdown_connections 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.344+0000 7fb8f47fb640 1 -- 172.21.15.107:0/972791183 wait complete. 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.346+0000 7fb8f47fb640 1 Processor -- start 2023-11-24T08:20:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.346+0000 7fb8f47fb640 1 -- start start 2023-11-24T08:20:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec06e160 0x7fb8ec1f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0cd110 0x7fb8ec0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec141ae0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec15fea0 con 0x7fb8ec06e160 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f47fb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec1f71f0 con 0x7fb8ec0cd110 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59596/0 (socket says 172.21.15.107:59596) 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 -- 172.21.15.107:0/4182318877 learned_addr learned my addr 172.21.15.107:0/4182318877 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2597640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec06e160 0x7fb8ec1f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 -- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0cd110 msgr2=0x7fb8ec0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0cd110 0x7fb8ec0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 -- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec06e160 msgr2=0x7fb8ec1f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec06e160 0x7fb8ec1f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.347+0000 7fb8f2d98640 1 -- 172.21.15.107:0/4182318877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4009a70 con 0x7fb8ec146eb0 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8f2d98640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009e80 tx=0x7fb8e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4040650 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40407d0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4040b80 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec141cf0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.348+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec1422b0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.349+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec04e8b0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.350+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb8e4040ce0 con 0x7fb8ec146eb0 2023-11-24T08:20:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.351+0000 7fb8e2ffd640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 0x7fb8d006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.351+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8e40ab340 con 0x7fb8ec146eb0 2023-11-24T08:20:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.351+0000 7fb8f2597640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 0x7fb8d006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.352+0000 7fb8f2597640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 0x7fb8d006efe0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000f60 tx=0x7fb8e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.355+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8e40e2e90 con 0x7fb8ec146eb0 2023-11-24T08:20:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:34.581+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb8ec06ded0 con 0x7fb8ec146eb0 2023-11-24T08:20:35.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.277+0000 7fb8e2ffd640 1 -- 172.21.15.107:0/4182318877 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v116) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb8e4077e50 con 0x7fb8ec146eb0 2023-11-24T08:20:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 msgr2=0x7fb8d006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 0x7fb8d006efe0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000f60 tx=0x7fb8e8009040 comp rx=0 tx=0).stop 2023-11-24T08:20:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 msgr2=0x7fb8ec0ad4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009e80 tx=0x7fb8e4004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 shutdown_connections 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8d006cb20 0x7fb8d006efe0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec146eb0 0x7fb8ec0ad4c0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8ec0cd110 0x7fb8ec0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 --2- 172.21.15.107:0/4182318877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec06e160 0x7fb8ec1f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 >> 172.21.15.107:0/4182318877 conn(0x7fb8ec0b0c50 msgr2=0x7fb8ec145ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.281+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 shutdown_connections 2023-11-24T08:20:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.282+0000 7fb8f47fb640 1 -- 172.21.15.107:0/4182318877 wait complete. 2023-11-24T08:20:35.304 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:20:35.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:20:35.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.510+0000 7fb21190f640 1 Processor -- start 2023-11-24T08:20:35.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.510+0000 7fb21190f640 1 -- start start 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.512+0000 7fb21190f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb20c15b440 0x7fb20c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.512+0000 7fb21190f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb20c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.512+0000 7fb21190f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c15eea0 con 0x7fb20c0cbf20 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.512+0000 7fb21190f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20c0c5e00 con 0x7fb20c15b440 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.512+0000 7fb21190f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0cc3d0 con 0x7fb20c0c59b0 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.513+0000 7fb203fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb20c0cbf20 0x7fb20c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47548/0 (socket says 172.21.15.107:47548) 2023-11-24T08:20:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.513+0000 7fb203fff640 1 -- 172.21.15.107:0/3927525809 learned_addr learned my addr 172.21.15.107:0/3927525809 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.513+0000 7fb20affd640 1 --2- 172.21.15.107:0/3927525809 >> v2:172.21.15.120:3300/0 conn(0x7fb20c15b440 0x7fb20c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.513+0000 7fb20b7fe640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb20c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886554879 0 0) 0x7fb20c15eea0 con 0x7fb20c0cbf20 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f8003520 con 0x7fb20c0cbf20 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12362065 0 0) 0x7fb1f8003520 con 0x7fb20c0cbf20 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 msgr2=0x7fb20c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb20c06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 >> v2:172.21.15.120:3300/0 conn(0x7fb20c15b440 msgr2=0x7fb20c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 --2- 172.21.15.107:0/3927525809 >> v2:172.21.15.120:3300/0 conn(0x7fb20c15b440 0x7fb20c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0abf30 con 0x7fb20c0cbf20 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb1f4001670 con 0x7fb20c0cbf20 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 >> v1:172.21.15.107:6789/0 conn(0x7fb20c0cbf20 legacy=0x7fb20c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb1f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb1f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f8004d00 0x7fb1f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0cc3d0 con 0x7fb1f8003970 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f800b980 con 0x7fb1f8004d00 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f800bac0 con 0x7fb20c0c59b0 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.514+0000 7fb20affd640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb1f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb203fff640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb1f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb20affd640 1 -- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 msgr2=0x7fb1f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb20affd640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c0c59b0 0x7fb1f80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb20affd640 1 -- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f8004d00 msgr2=0x7fb1f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb20affd640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f8004d00 0x7fb1f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.515+0000 7fb20affd640 1 -- 172.21.15.107:0/3927525809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8003520 con 0x7fb1f8003970 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.516+0000 7fb20affd640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb1f8003d90 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009ac0 tx=0x7fb1f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35805172a0c5dcc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.516+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc02c710 con 0x7fb1f8003970 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.516+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc02c870 con 0x7fb1f8003970 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.516+0000 7fb20a7fc640 1 -- 172.21.15.107:0/3927525809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc02cc00 con 0x7fb1f8003970 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 -- 172.21.15.107:0/3927525809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 msgr2=0x7fb1f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 --2- 172.21.15.107:0/3927525809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb1f8003d90 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009ac0 tx=0x7fb1f80057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb20b7fe640 1 -- 172.21.15.107:0/3927525809 reap_dead start 2023-11-24T08:20:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 -- 172.21.15.107:0/3927525809 shutdown_connections 2023-11-24T08:20:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 -- 172.21.15.107:0/3927525809 >> 172.21.15.107:0/3927525809 conn(0x7fb20c0b0c50 msgr2=0x7fb20c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 -- 172.21.15.107:0/3927525809 shutdown_connections 2023-11-24T08:20:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.517+0000 7fb21190f640 1 -- 172.21.15.107:0/3927525809 wait complete. 2023-11-24T08:20:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.519+0000 7fb21190f640 1 Processor -- start 2023-11-24T08:20:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.519+0000 7fb21190f640 1 -- start start 2023-11-24T08:20:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c15b9a0 0x7fb20c0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb20c1598c0 0x7fb20c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c15a260 con 0x7fb1f8003970 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0b2a40 con 0x7fb20c1598c0 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb21190f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c1549a0 con 0x7fb20c15b9a0 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59616/0 (socket says 172.21.15.107:59616) 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 -- 172.21.15.107:0/2301968394 learned_addr learned my addr 172.21.15.107:0/2301968394 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 -- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c15b9a0 msgr2=0x7fb20c0ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c15b9a0 0x7fb20c0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 -- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb20c1598c0 msgr2=0x7fb20c159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb20c1598c0 0x7fb20c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.520+0000 7fb20affd640 1 -- 172.21.15.107:0/2301968394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0b1e30 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb20affd640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c2b0 tx=0x7fb1fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc003810 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc02de90 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c0b2110 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc003b70 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.521+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c0b26d0 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.522+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c0ae230 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.523+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb1fc003cd0 con 0x7fb1f8003970 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.524+0000 7fb208ff9640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 0x7fb1ec06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:35.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.524+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1fc0aa910 con 0x7fb1f8003970 2023-11-24T08:20:35.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.524+0000 7fb203fff640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 0x7fb1ec06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:35.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.525+0000 7fb203fff640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 0x7fb1ec06efb0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40017a0 tx=0x7fb1f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:35.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.528+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1fc0775a0 con 0x7fb1f8003970 2023-11-24T08:20:35.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:35.759+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb20c04e8b0 con 0x7fb1f8003970 2023-11-24T08:20:36.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.289+0000 7fb208ff9640 1 -- 172.21.15.107:0/2301968394 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v117) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb1fc0390c0 con 0x7fb1f8003970 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 msgr2=0x7fb1ec06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 0x7fb1ec06efb0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40017a0 tx=0x7fb1f4009040 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 msgr2=0x7fb20c154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c2b0 tx=0x7fb1fc004730 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 shutdown_connections 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb1ec06caf0 0x7fb1ec06efb0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb20c1598c0 0x7fb20c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20c15b9a0 0x7fb20c0ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 --2- 172.21.15.107:0/2301968394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f8003970 0x7fb20c154460 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 >> 172.21.15.107:0/2301968394 conn(0x7fb20c0b0c50 msgr2=0x7fb20c071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.293+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 shutdown_connections 2023-11-24T08:20:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.294+0000 7fb21190f640 1 -- 172.21.15.107:0/2301968394 wait complete. 2023-11-24T08:20:36.319 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:20:36.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:36.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.525+0000 7fef5c3fb640 1 Processor -- start 2023-11-24T08:20:36.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.525+0000 7fef5c3fb640 1 -- start start 2023-11-24T08:20:36.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5c3fb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 0x7fef5415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5c3fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5415ab80 con 0x7fef540b2a70 2023-11-24T08:20:36.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5c3fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5415e680 con 0x7fef540b2630 2023-11-24T08:20:36.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5c3fb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5415a990 con 0x7fef540b2250 2023-11-24T08:20:36.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5a998640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fef540b2a70 0x7fef5415b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47550/0 (socket says 172.21.15.107:47550) 2023-11-24T08:20:36.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5a998640 1 -- 172.21.15.107:0/1388691399 learned_addr learned my addr 172.21.15.107:0/1388691399 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235933636 0 0) 0x7fef5415ab80 con 0x7fef540b2a70 2023-11-24T08:20:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.527+0000 7fef5a197640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 0x7fef5415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40005730 con 0x7fef540b2a70 2023-11-24T08:20:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166856007 0 0) 0x7fef40005730 con 0x7fef540b2a70 2023-11-24T08:20:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 msgr2=0x7fef5415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 0x7fef5415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 >> v1:172.21.15.120:6789/0 conn(0x7fef540b2630 legacy=0x7fef540ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5415e960 con 0x7fef540b2a70 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fef4c0019f0 con 0x7fef540b2a70 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 >> v1:172.21.15.107:6789/0 conn(0x7fef540b2a70 legacy=0x7fef5415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef40005bb0 0x7fef40005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef40006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 0x7fef40007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4000dd20 con 0x7fef400069d0 2023-11-24T08:20:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4000de60 con 0x7fef40005bb0 2023-11-24T08:20:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4000e020 con 0x7fef540b2250 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.528+0000 7fef59996640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef40006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef5a197640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef40005bb0 0x7fef40005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef59996640 1 -- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 msgr2=0x7fef40007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef59996640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b2250 0x7fef40007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef59996640 1 -- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef40005bb0 msgr2=0x7fef40005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef59996640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef40005bb0 0x7fef40005fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.529+0000 7fef59996640 1 -- 172.21.15.107:0/1388691399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef40005730 con 0x7fef400069d0 2023-11-24T08:20:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.530+0000 7fef59996640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef40006e50 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fef44001750 tx=0x7fef40007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18305615ba43c8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.530+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef440052d0 con 0x7fef400069d0 2023-11-24T08:20:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.530+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4402e050 con 0x7fef400069d0 2023-11-24T08:20:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.530+0000 7fef59195640 1 -- 172.21.15.107:0/1388691399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef440035f0 con 0x7fef400069d0 2023-11-24T08:20:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 -- 172.21.15.107:0/1388691399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 msgr2=0x7fef40006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/1388691399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef40006e50 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fef44001750 tx=0x7fef40007a00 comp rx=0 tx=0).stop 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5a998640 1 -- 172.21.15.107:0/1388691399 reap_dead start 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 -- 172.21.15.107:0/1388691399 shutdown_connections 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 -- 172.21.15.107:0/1388691399 >> 172.21.15.107:0/1388691399 conn(0x7fef5406a920 msgr2=0x7fef5406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 -- 172.21.15.107:0/1388691399 shutdown_connections 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.531+0000 7fef5c3fb640 1 -- 172.21.15.107:0/1388691399 wait complete. 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.533+0000 7fef5c3fb640 1 Processor -- start 2023-11-24T08:20:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.533+0000 7fef5c3fb640 1 -- start start 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef54071740 0x7fef54071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b0a50 0x7fef540b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5414cb00 con 0x7fef400069d0 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef540b1410 con 0x7fef54071740 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5c3fb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef540af570 con 0x7fef540b0a50 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59636/0 (socket says 172.21.15.107:59636) 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 -- 172.21.15.107:0/317821698 learned_addr learned my addr 172.21.15.107:0/317821698 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 -- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b0a50 msgr2=0x7fef540b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b0a50 0x7fef540b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 -- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef54071740 msgr2=0x7fef54071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef54071740 0x7fef54071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.534+0000 7fef5a197640 1 -- 172.21.15.107:0/317821698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540cd7c0 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef5a197640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fef5000e7e0 tx=0x7fef5000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5000c670 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5000c7d0 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef540722b0 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5000cb40 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.535+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5414d4e0 con 0x7fef400069d0 2023-11-24T08:20:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.536+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef54067470 con 0x7fef400069d0 2023-11-24T08:20:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.537+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fef5000cca0 con 0x7fef400069d0 2023-11-24T08:20:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.538+0000 7fef4affd640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 0x7fef3806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.538+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef50088210 con 0x7fef400069d0 2023-11-24T08:20:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.538+0000 7fef59996640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 0x7fef3806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.539+0000 7fef59996640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 0x7fef3806efe0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fef44001750 tx=0x7fef4402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.542+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef50054dc0 con 0x7fef400069d0 2023-11-24T08:20:36.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.769+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef540cea00 con 0x7fef400069d0 2023-11-24T08:20:36.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.770+0000 7fef4affd640 1 -- 172.21.15.107:0/317821698 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fef50021020 con 0x7fef400069d0 2023-11-24T08:20:36.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:36.773 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:35.761395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:20:36.773 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 117 2023-11-24T08:20:36.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 msgr2=0x7fef3806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 0x7fef3806efe0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fef44001750 tx=0x7fef4402f000 comp rx=0 tx=0).stop 2023-11-24T08:20:36.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 msgr2=0x7fef5414c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fef5000e7e0 tx=0x7fef5000eca0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 shutdown_connections 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef3806cb20 0x7fef3806efe0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540b0a50 0x7fef540b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef54071740 0x7fef54071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 --2- 172.21.15.107:0/317821698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef400069d0 0x7fef5414c5c0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 >> 172.21.15.107:0/317821698 conn(0x7fef5406a920 msgr2=0x7fef540cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 shutdown_connections 2023-11-24T08:20:36.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:36.773+0000 7fef5c3fb640 1 -- 172.21.15.107:0/317821698 wait complete. 2023-11-24T08:20:36.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:37.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.005+0000 7fd897b9e640 1 Processor -- start 2023-11-24T08:20:37.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.005+0000 7fd897b9e640 1 -- start start 2023-11-24T08:20:37.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.006+0000 7fd897b9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8900cfb60 0x7fd890161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.006+0000 7fd897b9e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8900ce0b0 con 0x7fd89015dc80 2023-11-24T08:20:37.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.006+0000 7fd897b9e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd890161960 con 0x7fd890157c70 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.006+0000 7fd897b9e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89015e8d0 con 0x7fd8900cfb60 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd895139640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8900cfb60 0x7fd890161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd89613b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd89015dc80 0x7fd8900b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47554/0 (socket says 172.21.15.107:47554) 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd89613b640 1 -- 172.21.15.107:0/1920126322 learned_addr learned my addr 172.21.15.107:0/1920126322 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd895139640 1 -- 172.21.15.107:0/1920126322 >> v1:172.21.15.120:6789/0 conn(0x7fd890157c70 legacy=0x7fd8900cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd895139640 1 -- 172.21.15.107:0/1920126322 >> v1:172.21.15.107:6789/0 conn(0x7fd89015dc80 legacy=0x7fd8900b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd895139640 1 -- 172.21.15.107:0/1920126322 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89015ebb0 con 0x7fd8900cfb60 2023-11-24T08:20:37.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.007+0000 7fd894938640 1 -- 172.21.15.107:0/1920126322 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394458579 0 0) 0x7fd8900ce0b0 con 0x7fd89015dc80 2023-11-24T08:20:37.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.008+0000 7fd895139640 1 --2- 172.21.15.107:0/1920126322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8900cfb60 0x7fd890161df0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fd8840097a0 tx=0x7fd88402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2deb64caba9f746f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd894938640 1 -- 172.21.15.107:0/1920126322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031830 con 0x7fd8900cfb60 2023-11-24T08:20:37.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd894938640 1 -- 172.21.15.107:0/1920126322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884031990 con 0x7fd8900cfb60 2023-11-24T08:20:37.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd894938640 1 -- 172.21.15.107:0/1920126322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031d20 con 0x7fd8900cfb60 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd897b9e640 1 -- 172.21.15.107:0/1920126322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8900cfb60 msgr2=0x7fd890161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd897b9e640 1 --2- 172.21.15.107:0/1920126322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8900cfb60 0x7fd890161df0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fd8840097a0 tx=0x7fd88402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd897b9e640 1 -- 172.21.15.107:0/1920126322 shutdown_connections 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd897b9e640 1 --2- 172.21.15.107:0/1920126322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8900cfb60 0x7fd890161df0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.009+0000 7fd897b9e640 1 -- 172.21.15.107:0/1920126322 >> 172.21.15.107:0/1920126322 conn(0x7fd89006a920 msgr2=0x7fd89006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.010+0000 7fd897b9e640 1 -- 172.21.15.107:0/1920126322 shutdown_connections 2023-11-24T08:20:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.010+0000 7fd897b9e640 1 -- 172.21.15.107:0/1920126322 wait complete. 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.011+0000 7fd897b9e640 1 Processor -- start 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.011+0000 7fd897b9e640 1 -- start start 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8900cfb60 0x7fd8900cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd89015dc80 0x7fd8900af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900c8360 con 0x7fd890157c70 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900cbc30 con 0x7fd8900cfb60 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.012+0000 7fd897b9e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900cbf10 con 0x7fd89015dc80 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59652/0 (socket says 172.21.15.107:59652) 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 -- 172.21.15.107:0/697843336 learned_addr learned my addr 172.21.15.107:0/697843336 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 -- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd89015dc80 msgr2=0x7fd8900af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd89015dc80 0x7fd8900af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 -- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8900cfb60 msgr2=0x7fd8900cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8900cfb60 0x7fd8900cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 -- 172.21.15.107:0/697843336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd884009400 con 0x7fd890157c70 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.013+0000 7fd895139640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.014+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88403aa60 con 0x7fd890157c70 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.014+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88403abc0 con 0x7fd890157c70 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.014+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8840443f0 con 0x7fd890157c70 2023-11-24T08:20:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.014+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8900cc1f0 con 0x7fd890157c70 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.014+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89009bcd0 con 0x7fd890157c70 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.015+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8900c87b0 con 0x7fd890157c70 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.016+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fd88404d450 con 0x7fd890157c70 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.017+0000 7fd8827fc640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 0x7fd87406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.017+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8840aa860 con 0x7fd890157c70 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.017+0000 7fd89593a640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 0x7fd87406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.018+0000 7fd89593a640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 0x7fd87406efe0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd88c000f60 tx=0x7fd88c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.021+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8900c87b0 con 0x7fd890157c70 2023-11-24T08:20:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.210+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8900c87b0 con 0x7fd890157c70 2023-11-24T08:20:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.213+0000 7fd8827fc640 1 -- 172.21.15.107:0/697843336 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+16583 (secure 0 0 0) 0x7fd8900c87b0 con 0x7fd890157c70 2023-11-24T08:20:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:37.216 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:34.257377+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 msgr2=0x7fd87406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 0x7fd87406efe0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd88c000f60 tx=0x7fd88c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 msgr2=0x7fd890154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd8840046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 shutdown_connections 2023-11-24T08:20:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd87406cb20 0x7fd87406efe0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd89015dc80 0x7fd8900af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd890157c70 0x7fd890154720 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 --2- 172.21.15.107:0/697843336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8900cfb60 0x7fd8900cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 >> 172.21.15.107:0/697843336 conn(0x7fd89006a920 msgr2=0x7fd8900ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 shutdown_connections 2023-11-24T08:20:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.216+0000 7fd897b9e640 1 -- 172.21.15.107:0/697843336 wait complete. 2023-11-24T08:20:37.237 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:20:37.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:20:37.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.446+0000 7f8ea9ac2640 1 Processor -- start 2023-11-24T08:20:37.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.446+0000 7f8ea9ac2640 1 -- start start 2023-11-24T08:20:37.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.447+0000 7f8ea9ac2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8ea4157c70 0x7f8ea4161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.447+0000 7f8ea9ac2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea40cf450 0x7f8ea40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.447+0000 7f8ea9ac2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ce0b0 con 0x7f8ea415dc80 2023-11-24T08:20:37.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.447+0000 7f8ea9ac2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea40cfe10 con 0x7f8ea4157c70 2023-11-24T08:20:37.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.447+0000 7f8ea9ac2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4161960 con 0x7f8ea40cf450 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8ea4157c70 0x7f8ea4161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8ea4157c70 0x7f8ea4161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59666/0 (socket says 172.21.15.107:59666) 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1780443783 learned_addr learned my addr 172.21.15.107:0/1780443783 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1780443783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea40cf450 msgr2=0x7f8ea40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/1780443783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea40cf450 0x7f8ea40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1780443783 >> v1:172.21.15.120:6789/0 conn(0x7f8ea415dc80 legacy=0x7f8ea415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:37.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.448+0000 7f8ea37fe640 1 -- 172.21.15.107:0/1780443783 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea415e8d0 con 0x7f8ea4157c70 2023-11-24T08:20:37.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.449+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/1780443783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea4157c70 0x7f8ea4161df0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009870 tx=0x7f8e9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f092a3fa9465e95 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.449+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1780443783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e940317c0 con 0x7f8ea4157c70 2023-11-24T08:20:37.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1780443783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94031920 con 0x7f8ea4157c70 2023-11-24T08:20:37.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea2ffd640 1 -- 172.21.15.107:0/1780443783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94031cb0 con 0x7f8ea4157c70 2023-11-24T08:20:37.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/1780443783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea4157c70 msgr2=0x7f8ea4161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/1780443783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea4157c70 0x7f8ea4161df0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009870 tx=0x7f8e9402b860 comp rx=0 tx=0).stop 2023-11-24T08:20:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/1780443783 shutdown_connections 2023-11-24T08:20:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/1780443783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea40cf450 0x7f8ea40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/1780443783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea4157c70 0x7f8ea4161df0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/1780443783 >> 172.21.15.107:0/1780443783 conn(0x7f8ea406a920 msgr2=0x7f8ea406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/1780443783 shutdown_connections 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.450+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/1780443783 wait complete. 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.452+0000 7f8ea9ac2640 1 Processor -- start 2023-11-24T08:20:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.452+0000 7f8ea9ac2640 1 -- start start 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40cf450 0x7f8ea41f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea4157c70 0x7f8ea40ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40cb750 con 0x7f8ea415dc80 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40ca8b0 con 0x7f8ea40cf450 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea9ac2640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40cab40 con 0x7f8ea4157c70 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59674/0 (socket says 172.21.15.107:59674) 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.453+0000 7f8ea3fff640 1 -- 172.21.15.107:0/4230068172 learned_addr learned my addr 172.21.15.107:0/4230068172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40cf450 0x7f8ea41f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8e9bfff640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea4157c70 0x7f8ea40ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 -- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea4157c70 msgr2=0x7f8ea40ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea4157c70 0x7f8ea40ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 -- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40cf450 msgr2=0x7f8ea41f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40cf450 0x7f8ea41f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 -- 172.21.15.107:0/4230068172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940094d0 con 0x7f8ea415dc80 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea3fff640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00c780 tx=0x7f8e9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c003d80 con 0x7f8ea415dc80 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00ce30 con 0x7f8ea415dc80 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c005290 con 0x7f8ea415dc80 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.454+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea40cae20 con 0x7f8ea415dc80 2023-11-24T08:20:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.455+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea40cb3e0 con 0x7f8ea415dc80 2023-11-24T08:20:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.456+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea40cb900 con 0x7f8ea415dc80 2023-11-24T08:20:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.457+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f8e9c0053f0 con 0x7f8ea415dc80 2023-11-24T08:20:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.458+0000 7f8ea0ff9640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 0x7f8e8406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.458+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e9c0878c0 con 0x7f8ea415dc80 2023-11-24T08:20:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.458+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 0x7f8e8406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.459+0000 7f8ea37fe640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 0x7f8e8406efe0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.462+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ea40cb900 con 0x7f8ea415dc80 2023-11-24T08:20:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.649+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ea41615f0 con 0x7f8e8406cb20 2023-11-24T08:20:37.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.658+0000 7f8ea0ff9640 1 -- 172.21.15.107:0/4230068172 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287590 (secure 0 0 0) 0x7f8e94031c10 con 0x7f8e8406cb20 2023-11-24T08:20:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:37.665 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:20:37.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 msgr2=0x7f8e8406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 0x7f8e8406efe0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403d040 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 msgr2=0x7f8ea40afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00c780 tx=0x7f8e9c00cc40 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 shutdown_connections 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e8406cb20 0x7f8e8406efe0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ea415dc80 0x7f8ea40afc70 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ea4157c70 0x7f8ea40ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 --2- 172.21.15.107:0/4230068172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ea40cf450 0x7f8ea41f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.664+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 >> 172.21.15.107:0/4230068172 conn(0x7f8ea406a920 msgr2=0x7f8ea4160060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.665+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 shutdown_connections 2023-11-24T08:20:37.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.665+0000 7f8ea9ac2640 1 -- 172.21.15.107:0/4230068172 wait complete. 2023-11-24T08:20:37.686 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2023-11-24T08:20:35.731015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319160,"kb_used_data":19788,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754655560,"statfs":{"total":773094113280,"available":772767293440,"internally_reserved":0,"allocated":20262912,"data_stored":9589725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":306433598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.500642"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282910+0000","last_change":"2023-11-24T08:20:33.275015+0000","last_active":"2023-11-24T08:20:33.282910+0000","last_peered":"2023-11-24T08:20:33.282910+0000","last_clean":"2023-11-24T08:20:33.282910+0000","last_became_active":"2023-11-24T08:20:33.274163+0000","last_became_peered":"2023-11-24T08:20:33.274163+0000","last_unstale":"2023-11-24T08:20:33.282910+0000","last_undegraded":"2023-11-24T08:20:33.282910+0000","last_fullsized":"2023-11-24T08:20:33.282910+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:53:46.467461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.264264+0000","last_change":"2023-11-24T08:20:33.257095+0000","last_active":"2023-11-24T08:20:33.264264+0000","last_peered":"2023-11-24T08:20:33.264264+0000","last_clean":"2023-11-24T08:20:33.264264+0000","last_became_active":"2023-11-24T08:20:33.256632+0000","last_became_peered":"2023-11-24T08:20:33.256632+0000","last_unstale":"2023-11-24T08:20:33.264264+0000","last_undegraded":"2023-11-24T08:20:33.264264+0000","last_fullsized":"2023-11-24T08:20:33.264264+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:24:28.139211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287773+0000","last_change":"2023-11-24T08:20:33.279680+0000","last_active":"2023-11-24T08:20:33.287773+0000","last_peered":"2023-11-24T08:20:33.287773+0000","last_clean":"2023-11-24T08:20:33.287773+0000","last_became_active":"2023-11-24T08:20:33.276732+0000","last_became_peered":"2023-11-24T08:20:33.276732+0000","last_unstale":"2023-11-24T08:20:33.287773+0000","last_undegraded":"2023-11-24T08:20:33.287773+0000","last_fullsized":"2023-11-24T08:20:33.287773+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:00:38.854360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278671+0000","last_change":"2023-11-24T08:20:33.274066+0000","last_active":"2023-11-24T08:20:33.278671+0000","last_peered":"2023-11-24T08:20:33.278671+0000","last_clean":"2023-11-24T08:20:33.278671+0000","last_became_active":"2023-11-24T08:20:33.273762+0000","last_became_peered":"2023-11-24T08:20:33.273762+0000","last_unstale":"2023-11-24T08:20:33.278671+0000","last_undegraded":"2023-11-24T08:20:33.278671+0000","last_fullsized":"2023-11-24T08:20:33.278671+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:35.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284652+0000","last_change":"2023-11-24T08:20:33.281609+0000","last_active":"2023-11-24T08:20:33.284652+0000","last_peered":"2023-11-24T08:20:33.284652+0000","last_clean":"2023-11-24T08:20:33.284652+0000","last_became_active":"2023-11-24T08:20:33.281306+0000","last_became_peered":"2023-11-24T08:20:33.281306+0000","last_unstale":"2023-11-24T08:20:33.284652+0000","last_undegraded":"2023-11-24T08:20:33.284652+0000","last_fullsized":"2023-11-24T08:20:33.284652+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:51.433824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.269870+0000","last_change":"2023-11-24T08:20:33.266900+0000","last_active":"2023-11-24T08:20:33.269870+0000","last_peered":"2023-11-24T08:20:33.269870+0000","last_clean":"2023-11-24T08:20:33.269870+0000","last_became_active":"2023-11-24T08:20:33.266584+0000","last_became_peered":"2023-11-24T08:20:33.266584+0000","last_unstale":"2023-11-24T08:20:33.269870+0000","last_undegraded":"2023-11-24T08:20:33.269870+0000","last_fullsized":"2023-11-24T08:20:33.269870+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:51.539078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287100+0000","last_change":"2023-11-24T08:20:33.282620+0000","last_active":"2023-11-24T08:20:33.287100+0000","last_peered":"2023-11-24T08:20:33.287100+0000","last_clean":"2023-11-24T08:20:33.287100+0000","last_became_active":"2023-11-24T08:20:33.282107+0000","last_became_peered":"2023-11-24T08:20:33.282107+0000","last_unstale":"2023-11-24T08:20:33.287100+0000","last_undegraded":"2023-11-24T08:20:33.287100+0000","last_fullsized":"2023-11-24T08:20:33.287100+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:58.338405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284689+0000","last_change":"2023-11-24T08:20:33.282341+0000","last_active":"2023-11-24T08:20:33.284689+0000","last_peered":"2023-11-24T08:20:33.284689+0000","last_clean":"2023-11-24T08:20:33.284689+0000","last_became_active":"2023-11-24T08:20:33.280592+0000","last_became_peered":"2023-11-24T08:20:33.280592+0000","last_unstale":"2023-11-24T08:20:33.284689+0000","last_undegraded":"2023-11-24T08:20:33.284689+0000","last_fullsized":"2023-11-24T08:20:33.284689+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.280885+0000","last_change":"2023-11-24T08:20:33.279063+0000","last_active":"2023-11-24T08:20:33.280885+0000","last_peered":"2023-11-24T08:20:33.280885+0000","last_clean":"2023-11-24T08:20:33.280885+0000","last_became_active":"2023-11-24T08:20:33.276192+0000","last_became_peered":"2023-11-24T08:20:33.276192+0000","last_unstale":"2023-11-24T08:20:33.280885+0000","last_undegraded":"2023-11-24T08:20:33.280885+0000","last_fullsized":"2023-11-24T08:20:33.280885+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:39.058582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285689+0000","last_change":"2023-11-24T08:20:33.281007+0000","last_active":"2023-11-24T08:20:33.285689+0000","last_peered":"2023-11-24T08:20:33.285689+0000","last_clean":"2023-11-24T08:20:33.285689+0000","last_became_active":"2023-11-24T08:20:33.280740+0000","last_became_peered":"2023-11-24T08:20:33.280740+0000","last_unstale":"2023-11-24T08:20:33.285689+0000","last_undegraded":"2023-11-24T08:20:33.285689+0000","last_fullsized":"2023-11-24T08:20:33.285689+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:33:46.310173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287936+0000","last_change":"2023-11-24T08:20:33.282159+0000","last_active":"2023-11-24T08:20:33.287936+0000","last_peered":"2023-11-24T08:20:33.287936+0000","last_clean":"2023-11-24T08:20:33.287936+0000","last_became_active":"2023-11-24T08:20:33.281885+0000","last_became_peered":"2023-11-24T08:20:33.281885+0000","last_unstale":"2023-11-24T08:20:33.287936+0000","last_undegraded":"2023-11-24T08:20:33.287936+0000","last_fullsized":"2023-11-24T08:20:33.287936+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:35:43.364804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274499+0000","last_change":"2023-11-24T08:20:33.267260+0000","last_active":"2023-11-24T08:20:33.274499+0000","last_peered":"2023-11-24T08:20:33.274499+0000","last_clean":"2023-11-24T08:20:33.274499+0000","last_became_active":"2023-11-24T08:20:33.266985+0000","last_became_peered":"2023-11-24T08:20:33.266985+0000","last_unstale":"2023-11-24T08:20:33.274499+0000","last_undegraded":"2023-11-24T08:20:33.274499+0000","last_fullsized":"2023-11-24T08:20:33.274499+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:27.083986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286038+0000","last_change":"2023-11-24T08:20:33.281678+0000","last_active":"2023-11-24T08:20:33.286038+0000","last_peered":"2023-11-24T08:20:33.286038+0000","last_clean":"2023-11-24T08:20:33.286038+0000","last_became_active":"2023-11-24T08:20:33.281380+0000","last_became_peered":"2023-11-24T08:20:33.281380+0000","last_unstale":"2023-11-24T08:20:33.286038+0000","last_undegraded":"2023-11-24T08:20:33.286038+0000","last_fullsized":"2023-11-24T08:20:33.286038+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:51.539078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281690+0000","last_change":"2023-11-24T08:20:33.272998+0000","last_active":"2023-11-24T08:20:33.281690+0000","last_peered":"2023-11-24T08:20:33.281690+0000","last_clean":"2023-11-24T08:20:33.281690+0000","last_became_active":"2023-11-24T08:20:33.272749+0000","last_became_peered":"2023-11-24T08:20:33.272749+0000","last_unstale":"2023-11-24T08:20:33.281690+0000","last_undegraded":"2023-11-24T08:20:33.281690+0000","last_fullsized":"2023-11-24T08:20:33.281690+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:02:00.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284979+0000","last_change":"2023-11-24T08:20:33.282410+0000","last_active":"2023-11-24T08:20:33.284979+0000","last_peered":"2023-11-24T08:20:33.284979+0000","last_clean":"2023-11-24T08:20:33.284979+0000","last_became_active":"2023-11-24T08:20:33.281756+0000","last_became_peered":"2023-11-24T08:20:33.281756+0000","last_unstale":"2023-11-24T08:20:33.284979+0000","last_undegraded":"2023-11-24T08:20:33.284979+0000","last_fullsized":"2023-11-24T08:20:33.284979+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:48.190825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285565+0000","last_change":"2023-11-24T08:20:33.278589+0000","last_active":"2023-11-24T08:20:33.285565+0000","last_peered":"2023-11-24T08:20:33.285565+0000","last_clean":"2023-11-24T08:20:33.285565+0000","last_became_active":"2023-11-24T08:20:33.278299+0000","last_became_peered":"2023-11-24T08:20:33.278299+0000","last_unstale":"2023-11-24T08:20:33.285565+0000","last_undegraded":"2023-11-24T08:20:33.285565+0000","last_fullsized":"2023-11-24T08:20:33.285565+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:57:18.109701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282930+0000","last_change":"2023-11-24T08:20:33.275217+0000","last_active":"2023-11-24T08:20:33.282930+0000","last_peered":"2023-11-24T08:20:33.282930+0000","last_clean":"2023-11-24T08:20:33.282930+0000","last_became_active":"2023-11-24T08:20:33.274731+0000","last_became_peered":"2023-11-24T08:20:33.274731+0000","last_unstale":"2023-11-24T08:20:33.282930+0000","last_undegraded":"2023-11-24T08:20:33.282930+0000","last_fullsized":"2023-11-24T08:20:33.282930+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:20.893125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285505+0000","last_change":"2023-11-24T08:20:33.277350+0000","last_active":"2023-11-24T08:20:33.285505+0000","last_peered":"2023-11-24T08:20:33.285505+0000","last_clean":"2023-11-24T08:20:33.285505+0000","last_became_active":"2023-11-24T08:20:33.275259+0000","last_became_peered":"2023-11-24T08:20:33.275259+0000","last_unstale":"2023-11-24T08:20:33.285505+0000","last_undegraded":"2023-11-24T08:20:33.285505+0000","last_fullsized":"2023-11-24T08:20:33.285505+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:12:29.085586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286116+0000","last_change":"2023-11-24T08:20:33.282360+0000","last_active":"2023-11-24T08:20:33.286116+0000","last_peered":"2023-11-24T08:20:33.286116+0000","last_clean":"2023-11-24T08:20:33.286116+0000","last_became_active":"2023-11-24T08:20:33.281856+0000","last_became_peered":"2023-11-24T08:20:33.281856+0000","last_unstale":"2023-11-24T08:20:33.286116+0000","last_undegraded":"2023-11-24T08:20:33.286116+0000","last_fullsized":"2023-11-24T08:20:33.286116+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:35.502297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284499+0000","last_change":"2023-11-24T08:20:33.280157+0000","last_active":"2023-11-24T08:20:33.284499+0000","last_peered":"2023-11-24T08:20:33.284499+0000","last_clean":"2023-11-24T08:20:33.284499+0000","last_became_active":"2023-11-24T08:20:33.279873+0000","last_became_peered":"2023-11-24T08:20:33.279873+0000","last_unstale":"2023-11-24T08:20:33.284499+0000","last_undegraded":"2023-11-24T08:20:33.284499+0000","last_fullsized":"2023-11-24T08:20:33.284499+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:32.253157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278717+0000","last_change":"2023-11-24T08:20:33.274875+0000","last_active":"2023-11-24T08:20:33.278717+0000","last_peered":"2023-11-24T08:20:33.278717+0000","last_clean":"2023-11-24T08:20:33.278717+0000","last_became_active":"2023-11-24T08:20:33.274544+0000","last_became_peered":"2023-11-24T08:20:33.274544+0000","last_unstale":"2023-11-24T08:20:33.278717+0000","last_undegraded":"2023-11-24T08:20:33.278717+0000","last_fullsized":"2023-11-24T08:20:33.278717+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:31.664775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274577+0000","last_change":"2023-11-24T08:20:33.263590+0000","last_active":"2023-11-24T08:20:33.274577+0000","last_peered":"2023-11-24T08:20:33.274577+0000","last_clean":"2023-11-24T08:20:33.274577+0000","last_became_active":"2023-11-24T08:20:33.263307+0000","last_became_peered":"2023-11-24T08:20:33.263307+0000","last_unstale":"2023-11-24T08:20:33.274577+0000","last_undegraded":"2023-11-24T08:20:33.274577+0000","last_fullsized":"2023-11-24T08:20:33.274577+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:43.480882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278101+0000","last_change":"2023-11-24T08:20:33.275880+0000","last_active":"2023-11-24T08:20:33.278101+0000","last_peered":"2023-11-24T08:20:33.278101+0000","last_clean":"2023-11-24T08:20:33.278101+0000","last_became_active":"2023-11-24T08:20:33.275611+0000","last_became_peered":"2023-11-24T08:20:33.275611+0000","last_unstale":"2023-11-24T08:20:33.278101+0000","last_undegraded":"2023-11-24T08:20:33.278101+0000","last_fullsized":"2023-11-24T08:20:33.278101+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278727+0000","last_change":"2023-11-24T08:20:33.276152+0000","last_active":"2023-11-24T08:20:33.278727+0000","last_peered":"2023-11-24T08:20:33.278727+0000","last_clean":"2023-11-24T08:20:33.278727+0000","last_became_active":"2023-11-24T08:20:33.275832+0000","last_became_peered":"2023-11-24T08:20:33.275832+0000","last_unstale":"2023-11-24T08:20:33.278727+0000","last_undegraded":"2023-11-24T08:20:33.278727+0000","last_fullsized":"2023-11-24T08:20:33.278727+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:47.193596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.270051+0000","last_change":"2023-11-24T08:20:33.267060+0000","last_active":"2023-11-24T08:20:33.270051+0000","last_peered":"2023-11-24T08:20:33.270051+0000","last_clean":"2023-11-24T08:20:33.270051+0000","last_became_active":"2023-11-24T08:20:33.264117+0000","last_became_peered":"2023-11-24T08:20:33.264117+0000","last_unstale":"2023-11-24T08:20:33.270051+0000","last_undegraded":"2023-11-24T08:20:33.270051+0000","last_fullsized":"2023-11-24T08:20:33.270051+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:41.109560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282251+0000","last_change":"2023-11-24T08:20:33.274992+0000","last_active":"2023-11-24T08:20:33.282251+0000","last_peered":"2023-11-24T08:20:33.282251+0000","last_clean":"2023-11-24T08:20:33.282251+0000","last_became_active":"2023-11-24T08:20:33.274315+0000","last_became_peered":"2023-11-24T08:20:33.274315+0000","last_unstale":"2023-11-24T08:20:33.282251+0000","last_undegraded":"2023-11-24T08:20:33.282251+0000","last_fullsized":"2023-11-24T08:20:33.282251+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:51.606324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.276290+0000","last_change":"2023-11-24T08:20:33.273248+0000","last_active":"2023-11-24T08:20:33.276290+0000","last_peered":"2023-11-24T08:20:33.276290+0000","last_clean":"2023-11-24T08:20:33.276290+0000","last_became_active":"2023-11-24T08:20:33.268914+0000","last_became_peered":"2023-11-24T08:20:33.268914+0000","last_unstale":"2023-11-24T08:20:33.276290+0000","last_undegraded":"2023-11-24T08:20:33.276290+0000","last_fullsized":"2023-11-24T08:20:33.276290+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:48.755542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287644+0000","last_change":"2023-11-24T08:20:33.277091+0000","last_active":"2023-11-24T08:20:33.287644+0000","last_peered":"2023-11-24T08:20:33.287644+0000","last_clean":"2023-11-24T08:20:33.287644+0000","last_became_active":"2023-11-24T08:20:33.274190+0000","last_became_peered":"2023-11-24T08:20:33.274190+0000","last_unstale":"2023-11-24T08:20:33.287644+0000","last_undegraded":"2023-11-24T08:20:33.287644+0000","last_fullsized":"2023-11-24T08:20:33.287644+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:09:42.881197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275925+0000","last_change":"2023-11-24T08:20:33.271677+0000","last_active":"2023-11-24T08:20:33.275925+0000","last_peered":"2023-11-24T08:20:33.275925+0000","last_clean":"2023-11-24T08:20:33.275925+0000","last_became_active":"2023-11-24T08:20:33.271370+0000","last_became_peered":"2023-11-24T08:20:33.271370+0000","last_unstale":"2023-11-24T08:20:33.275925+0000","last_undegraded":"2023-11-24T08:20:33.275925+0000","last_fullsized":"2023-11-24T08:20:33.275925+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:21.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.269402+0000","last_change":"2023-11-24T08:20:33.264978+0000","last_active":"2023-11-24T08:20:33.269402+0000","last_peered":"2023-11-24T08:20:33.269402+0000","last_clean":"2023-11-24T08:20:33.269402+0000","last_became_active":"2023-11-24T08:20:33.264406+0000","last_became_peered":"2023-11-24T08:20:33.264406+0000","last_unstale":"2023-11-24T08:20:33.269402+0000","last_undegraded":"2023-11-24T08:20:33.269402+0000","last_fullsized":"2023-11-24T08:20:33.269402+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:58.121397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279325+0000","last_change":"2023-11-24T08:20:33.275187+0000","last_active":"2023-11-24T08:20:33.279325+0000","last_peered":"2023-11-24T08:20:33.279325+0000","last_clean":"2023-11-24T08:20:33.279325+0000","last_became_active":"2023-11-24T08:20:33.274300+0000","last_became_peered":"2023-11-24T08:20:33.274300+0000","last_unstale":"2023-11-24T08:20:33.279325+0000","last_undegraded":"2023-11-24T08:20:33.279325+0000","last_fullsized":"2023-11-24T08:20:33.279325+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:00:32.072184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282043+0000","last_change":"2023-11-24T08:20:33.274878+0000","last_active":"2023-11-24T08:20:33.282043+0000","last_peered":"2023-11-24T08:20:33.282043+0000","last_clean":"2023-11-24T08:20:33.282043+0000","last_became_active":"2023-11-24T08:20:33.274556+0000","last_became_peered":"2023-11-24T08:20:33.274556+0000","last_unstale":"2023-11-24T08:20:33.282043+0000","last_undegraded":"2023-11-24T08:20:33.282043+0000","last_fullsized":"2023-11-24T08:20:33.282043+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:59:49.093706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274409+0000","last_change":"2023-11-24T08:20:33.271552+0000","last_active":"2023-11-24T08:20:33.274409+0000","last_peered":"2023-11-24T08:20:33.274409+0000","last_clean":"2023-11-24T08:20:33.274409+0000","last_became_active":"2023-11-24T08:20:33.270431+0000","last_became_peered":"2023-11-24T08:20:33.270431+0000","last_unstale":"2023-11-24T08:20:33.274409+0000","last_undegraded":"2023-11-24T08:20:33.274409+0000","last_fullsized":"2023-11-24T08:20:33.274409+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:57.199781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"189'4","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.290884+0000","last_change":"2023-11-24T08:20:32.280779+0000","last_active":"2023-11-24T08:20:33.290884+0000","last_peered":"2023-11-24T08:20:33.290884+0000","last_clean":"2023-11-24T08:20:33.290884+0000","last_became_active":"2023-11-24T08:20:32.280482+0000","last_became_peered":"2023-11-24T08:20:32.280482+0000","last_unstale":"2023-11-24T08:20:33.290884+0000","last_undegraded":"2023-11-24T08:20:33.290884+0000","last_fullsized":"2023-11-24T08:20:33.290884+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:22.660075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278170+0000","last_change":"2023-11-24T08:20:33.274492+0000","last_active":"2023-11-24T08:20:33.278170+0000","last_peered":"2023-11-24T08:20:33.278170+0000","last_clean":"2023-11-24T08:20:33.278170+0000","last_became_active":"2023-11-24T08:20:33.274193+0000","last_became_peered":"2023-11-24T08:20:33.274193+0000","last_unstale":"2023-11-24T08:20:33.278170+0000","last_undegraded":"2023-11-24T08:20:33.278170+0000","last_fullsized":"2023-11-24T08:20:33.278170+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:41.109560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.289234+0000","last_change":"2023-11-24T08:20:32.278574+0000","last_active":"2023-11-24T08:20:33.289234+0000","last_peered":"2023-11-24T08:20:33.289234+0000","last_clean":"2023-11-24T08:20:33.289234+0000","last_became_active":"2023-11-24T08:20:32.278127+0000","last_became_peered":"2023-11-24T08:20:32.278127+0000","last_unstale":"2023-11-24T08:20:33.289234+0000","last_undegraded":"2023-11-24T08:20:33.289234+0000","last_fullsized":"2023-11-24T08:20:33.289234+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:04.609533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282890+0000","last_change":"2023-11-24T08:20:33.280061+0000","last_active":"2023-11-24T08:20:33.282890+0000","last_peered":"2023-11-24T08:20:33.282890+0000","last_clean":"2023-11-24T08:20:33.282890+0000","last_became_active":"2023-11-24T08:20:33.279799+0000","last_became_peered":"2023-11-24T08:20:33.279799+0000","last_unstale":"2023-11-24T08:20:33.282890+0000","last_undegraded":"2023-11-24T08:20:33.282890+0000","last_fullsized":"2023-11-24T08:20:33.282890+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:35.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.296770+0000","last_change":"2023-11-24T08:20:32.278740+0000","last_active":"2023-11-24T08:20:33.296770+0000","last_peered":"2023-11-24T08:20:33.296770+0000","last_clean":"2023-11-24T08:20:33.296770+0000","last_became_active":"2023-11-24T08:20:32.278202+0000","last_became_peered":"2023-11-24T08:20:32.278202+0000","last_unstale":"2023-11-24T08:20:33.296770+0000","last_undegraded":"2023-11-24T08:20:33.296770+0000","last_fullsized":"2023-11-24T08:20:33.296770+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:45.847300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281627+0000","last_change":"2023-11-24T08:20:33.276379+0000","last_active":"2023-11-24T08:20:33.281627+0000","last_peered":"2023-11-24T08:20:33.281627+0000","last_clean":"2023-11-24T08:20:33.281627+0000","last_became_active":"2023-11-24T08:20:33.276077+0000","last_became_peered":"2023-11-24T08:20:33.276077+0000","last_unstale":"2023-11-24T08:20:33.281627+0000","last_undegraded":"2023-11-24T08:20:33.281627+0000","last_fullsized":"2023-11-24T08:20:33.281627+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:12:17.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.293472+0000","last_change":"2023-11-24T08:20:32.279533+0000","last_active":"2023-11-24T08:20:33.293472+0000","last_peered":"2023-11-24T08:20:33.293472+0000","last_clean":"2023-11-24T08:20:33.293472+0000","last_became_active":"2023-11-24T08:20:32.279174+0000","last_became_peered":"2023-11-24T08:20:32.279174+0000","last_unstale":"2023-11-24T08:20:33.293472+0000","last_undegraded":"2023-11-24T08:20:33.293472+0000","last_fullsized":"2023-11-24T08:20:33.293472+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:57.321751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.277791+0000","last_change":"2023-11-24T08:20:33.273172+0000","last_active":"2023-11-24T08:20:33.277791+0000","last_peered":"2023-11-24T08:20:33.277791+0000","last_clean":"2023-11-24T08:20:33.277791+0000","last_became_active":"2023-11-24T08:20:33.272818+0000","last_became_peered":"2023-11-24T08:20:33.272818+0000","last_unstale":"2023-11-24T08:20:33.277791+0000","last_undegraded":"2023-11-24T08:20:33.277791+0000","last_fullsized":"2023-11-24T08:20:33.277791+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:02:00.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.289580+0000","last_change":"2023-11-24T08:20:32.272345+0000","last_active":"2023-11-24T08:20:33.289580+0000","last_peered":"2023-11-24T08:20:33.289580+0000","last_clean":"2023-11-24T08:20:33.289580+0000","last_became_active":"2023-11-24T08:20:32.272002+0000","last_became_peered":"2023-11-24T08:20:32.272002+0000","last_unstale":"2023-11-24T08:20:33.289580+0000","last_undegraded":"2023-11-24T08:20:33.289580+0000","last_fullsized":"2023-11-24T08:20:33.289580+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:52.699687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283575+0000","last_change":"2023-11-24T08:20:33.272726+0000","last_active":"2023-11-24T08:20:33.283575+0000","last_peered":"2023-11-24T08:20:33.283575+0000","last_clean":"2023-11-24T08:20:33.283575+0000","last_became_active":"2023-11-24T08:20:33.269024+0000","last_became_peered":"2023-11-24T08:20:33.269024+0000","last_unstale":"2023-11-24T08:20:33.283575+0000","last_undegraded":"2023-11-24T08:20:33.283575+0000","last_fullsized":"2023-11-24T08:20:33.283575+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:17.426766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.249719+0000","last_change":"2023-11-24T08:20:32.279187+0000","last_active":"2023-11-24T08:20:33.249719+0000","last_peered":"2023-11-24T08:20:33.249719+0000","last_clean":"2023-11-24T08:20:33.249719+0000","last_became_active":"2023-11-24T08:20:32.278221+0000","last_became_peered":"2023-11-24T08:20:32.278221+0000","last_unstale":"2023-11-24T08:20:33.249719+0000","last_undegraded":"2023-11-24T08:20:33.249719+0000","last_fullsized":"2023-11-24T08:20:33.249719+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:40.092906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278668+0000","last_change":"2023-11-24T08:20:33.275984+0000","last_active":"2023-11-24T08:20:33.278668+0000","last_peered":"2023-11-24T08:20:33.278668+0000","last_clean":"2023-11-24T08:20:33.278668+0000","last_became_active":"2023-11-24T08:20:33.275275+0000","last_became_peered":"2023-11-24T08:20:33.275275+0000","last_unstale":"2023-11-24T08:20:33.278668+0000","last_undegraded":"2023-11-24T08:20:33.278668+0000","last_fullsized":"2023-11-24T08:20:33.278668+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:58.947708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.247816+0000","last_change":"2023-11-24T08:20:32.258971+0000","last_active":"2023-11-24T08:20:33.247816+0000","last_peered":"2023-11-24T08:20:33.247816+0000","last_clean":"2023-11-24T08:20:33.247816+0000","last_became_active":"2023-11-24T08:20:32.257755+0000","last_became_peered":"2023-11-24T08:20:32.257755+0000","last_unstale":"2023-11-24T08:20:33.247816+0000","last_undegraded":"2023-11-24T08:20:33.247816+0000","last_fullsized":"2023-11-24T08:20:33.247816+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:16.410112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275148+0000","last_change":"2023-11-24T08:20:33.270490+0000","last_active":"2023-11-24T08:20:33.275148+0000","last_peered":"2023-11-24T08:20:33.275148+0000","last_clean":"2023-11-24T08:20:33.275148+0000","last_became_active":"2023-11-24T08:20:33.270194+0000","last_became_peered":"2023-11-24T08:20:33.270194+0000","last_unstale":"2023-11-24T08:20:33.275148+0000","last_undegraded":"2023-11-24T08:20:33.275148+0000","last_fullsized":"2023-11-24T08:20:33.275148+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:51.601095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.251009+0000","last_change":"2023-11-24T08:20:32.278673+0000","last_active":"2023-11-24T08:20:33.251009+0000","last_peered":"2023-11-24T08:20:33.251009+0000","last_clean":"2023-11-24T08:20:33.251009+0000","last_became_active":"2023-11-24T08:20:32.278372+0000","last_became_peered":"2023-11-24T08:20:32.278372+0000","last_unstale":"2023-11-24T08:20:33.251009+0000","last_undegraded":"2023-11-24T08:20:33.251009+0000","last_fullsized":"2023-11-24T08:20:33.251009+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:45.579425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287842+0000","last_change":"2023-11-24T08:20:33.281071+0000","last_active":"2023-11-24T08:20:33.287842+0000","last_peered":"2023-11-24T08:20:33.287842+0000","last_clean":"2023-11-24T08:20:33.287842+0000","last_became_active":"2023-11-24T08:20:33.278156+0000","last_became_peered":"2023-11-24T08:20:33.278156+0000","last_unstale":"2023-11-24T08:20:33.287842+0000","last_undegraded":"2023-11-24T08:20:33.287842+0000","last_fullsized":"2023-11-24T08:20:33.287842+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:04.104523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.246602+0000","last_change":"2023-11-24T08:20:32.270934+0000","last_active":"2023-11-24T08:20:33.246602+0000","last_peered":"2023-11-24T08:20:33.246602+0000","last_clean":"2023-11-24T08:20:33.246602+0000","last_became_active":"2023-11-24T08:20:32.270056+0000","last_became_peered":"2023-11-24T08:20:32.270056+0000","last_unstale":"2023-11-24T08:20:33.246602+0000","last_undegraded":"2023-11-24T08:20:33.246602+0000","last_fullsized":"2023-11-24T08:20:33.246602+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:38.041928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283225+0000","last_change":"2023-11-24T08:20:33.278863+0000","last_active":"2023-11-24T08:20:33.283225+0000","last_peered":"2023-11-24T08:20:33.283225+0000","last_clean":"2023-11-24T08:20:33.283225+0000","last_became_active":"2023-11-24T08:20:33.278398+0000","last_became_peered":"2023-11-24T08:20:33.278398+0000","last_unstale":"2023-11-24T08:20:33.283225+0000","last_undegraded":"2023-11-24T08:20:33.283225+0000","last_fullsized":"2023-11-24T08:20:33.283225+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:48.190825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"189'3","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.298245+0000","last_change":"2023-11-24T08:20:32.270893+0000","last_active":"2023-11-24T08:20:33.298245+0000","last_peered":"2023-11-24T08:20:33.298245+0000","last_clean":"2023-11-24T08:20:33.298245+0000","last_became_active":"2023-11-24T08:20:32.270121+0000","last_became_peered":"2023-11-24T08:20:32.270121+0000","last_unstale":"2023-11-24T08:20:33.298245+0000","last_undegraded":"2023-11-24T08:20:33.298245+0000","last_fullsized":"2023-11-24T08:20:33.298245+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:51:33.002261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.280243+0000","last_change":"2023-11-24T08:20:33.271132+0000","last_active":"2023-11-24T08:20:33.280243+0000","last_peered":"2023-11-24T08:20:33.280243+0000","last_clean":"2023-11-24T08:20:33.280243+0000","last_became_active":"2023-11-24T08:20:33.270772+0000","last_became_peered":"2023-11-24T08:20:33.270772+0000","last_unstale":"2023-11-24T08:20:33.280243+0000","last_undegraded":"2023-11-24T08:20:33.280243+0000","last_fullsized":"2023-11-24T08:20:33.280243+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:08:35.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"189'6","reported_seq":17,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.302170+0000","last_change":"2023-11-24T08:20:32.283480+0000","last_active":"2023-11-24T08:20:33.302170+0000","last_peered":"2023-11-24T08:20:33.302170+0000","last_clean":"2023-11-24T08:20:33.302170+0000","last_became_active":"2023-11-24T08:20:32.283095+0000","last_became_peered":"2023-11-24T08:20:32.283095+0000","last_unstale":"2023-11-24T08:20:33.302170+0000","last_undegraded":"2023-11-24T08:20:33.302170+0000","last_fullsized":"2023-11-24T08:20:33.302170+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:29.097591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284533+0000","last_change":"2023-11-24T08:20:33.281637+0000","last_active":"2023-11-24T08:20:33.284533+0000","last_peered":"2023-11-24T08:20:33.284533+0000","last_clean":"2023-11-24T08:20:33.284533+0000","last_became_active":"2023-11-24T08:20:33.281357+0000","last_became_peered":"2023-11-24T08:20:33.281357+0000","last_unstale":"2023-11-24T08:20:33.284533+0000","last_undegraded":"2023-11-24T08:20:33.284533+0000","last_fullsized":"2023-11-24T08:20:33.284533+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:21.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.296514+0000","last_change":"2023-11-24T08:20:32.279998+0000","last_active":"2023-11-24T08:20:33.296514+0000","last_peered":"2023-11-24T08:20:33.296514+0000","last_clean":"2023-11-24T08:20:33.296514+0000","last_became_active":"2023-11-24T08:20:32.279608+0000","last_became_peered":"2023-11-24T08:20:32.279608+0000","last_unstale":"2023-11-24T08:20:33.296514+0000","last_undegraded":"2023-11-24T08:20:33.296514+0000","last_fullsized":"2023-11-24T08:20:33.296514+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:45.579425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279679+0000","last_change":"2023-11-24T08:20:33.274597+0000","last_active":"2023-11-24T08:20:33.279679+0000","last_peered":"2023-11-24T08:20:33.279679+0000","last_clean":"2023-11-24T08:20:33.279679+0000","last_became_active":"2023-11-24T08:20:33.273854+0000","last_became_peered":"2023-11-24T08:20:33.273854+0000","last_unstale":"2023-11-24T08:20:33.279679+0000","last_undegraded":"2023-11-24T08:20:33.279679+0000","last_fullsized":"2023-11-24T08:20:33.279679+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:47:25.942966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.248432+0000","last_change":"2023-11-24T08:20:32.274102+0000","last_active":"2023-11-24T08:20:33.248432+0000","last_peered":"2023-11-24T08:20:33.248432+0000","last_clean":"2023-11-24T08:20:33.248432+0000","last_became_active":"2023-11-24T08:20:32.273818+0000","last_became_peered":"2023-11-24T08:20:32.273818+0000","last_unstale":"2023-11-24T08:20:33.248432+0000","last_undegraded":"2023-11-24T08:20:33.248432+0000","last_fullsized":"2023-11-24T08:20:33.248432+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:47.738888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275613+0000","last_change":"2023-11-24T08:20:33.272907+0000","last_active":"2023-11-24T08:20:33.275613+0000","last_peered":"2023-11-24T08:20:33.275613+0000","last_clean":"2023-11-24T08:20:33.275613+0000","last_became_active":"2023-11-24T08:20:33.272598+0000","last_became_peered":"2023-11-24T08:20:33.272598+0000","last_unstale":"2023-11-24T08:20:33.275613+0000","last_undegraded":"2023-11-24T08:20:33.275613+0000","last_fullsized":"2023-11-24T08:20:33.275613+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:40.352394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.267699+0000","last_change":"2023-11-24T08:20:32.274484+0000","last_active":"2023-11-24T08:20:33.267699+0000","last_peered":"2023-11-24T08:20:33.267699+0000","last_clean":"2023-11-24T08:20:33.267699+0000","last_became_active":"2023-11-24T08:20:32.274169+0000","last_became_peered":"2023-11-24T08:20:32.274169+0000","last_unstale":"2023-11-24T08:20:33.267699+0000","last_undegraded":"2023-11-24T08:20:33.267699+0000","last_fullsized":"2023-11-24T08:20:33.267699+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:31.023782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284158+0000","last_change":"2023-11-24T08:20:33.278265+0000","last_active":"2023-11-24T08:20:33.284158+0000","last_peered":"2023-11-24T08:20:33.284158+0000","last_clean":"2023-11-24T08:20:33.284158+0000","last_became_active":"2023-11-24T08:20:33.277800+0000","last_became_peered":"2023-11-24T08:20:33.277800+0000","last_unstale":"2023-11-24T08:20:33.284158+0000","last_undegraded":"2023-11-24T08:20:33.284158+0000","last_fullsized":"2023-11-24T08:20:33.284158+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:31.664775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"189'8","reported_seq":19,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.299899+0000","last_change":"2023-11-24T08:20:32.273022+0000","last_active":"2023-11-24T08:20:33.299899+0000","last_peered":"2023-11-24T08:20:33.299899+0000","last_clean":"2023-11-24T08:20:33.299899+0000","last_became_active":"2023-11-24T08:20:32.272736+0000","last_became_peered":"2023-11-24T08:20:32.272736+0000","last_unstale":"2023-11-24T08:20:33.299899+0000","last_undegraded":"2023-11-24T08:20:33.299899+0000","last_fullsized":"2023-11-24T08:20:33.299899+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:59:58.990637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275725+0000","last_change":"2023-11-24T08:20:33.272986+0000","last_active":"2023-11-24T08:20:33.275725+0000","last_peered":"2023-11-24T08:20:33.275725+0000","last_clean":"2023-11-24T08:20:33.275725+0000","last_became_active":"2023-11-24T08:20:33.272493+0000","last_became_peered":"2023-11-24T08:20:33.272493+0000","last_unstale":"2023-11-24T08:20:33.275725+0000","last_undegraded":"2023-11-24T08:20:33.275725+0000","last_fullsized":"2023-11-24T08:20:33.275725+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:46.596079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295584+0000","last_change":"2023-11-24T08:20:32.282307+0000","last_active":"2023-11-24T08:20:33.295584+0000","last_peered":"2023-11-24T08:20:33.295584+0000","last_clean":"2023-11-24T08:20:33.295584+0000","last_became_active":"2023-11-24T08:20:32.282041+0000","last_became_peered":"2023-11-24T08:20:32.282041+0000","last_unstale":"2023-11-24T08:20:33.295584+0000","last_undegraded":"2023-11-24T08:20:33.295584+0000","last_fullsized":"2023-11-24T08:20:33.295584+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:47.738888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.270072+0000","last_change":"2023-11-24T08:20:33.267202+0000","last_active":"2023-11-24T08:20:33.270072+0000","last_peered":"2023-11-24T08:20:33.270072+0000","last_clean":"2023-11-24T08:20:33.270072+0000","last_became_active":"2023-11-24T08:20:33.266908+0000","last_became_peered":"2023-11-24T08:20:33.266908+0000","last_unstale":"2023-11-24T08:20:33.270072+0000","last_undegraded":"2023-11-24T08:20:33.270072+0000","last_fullsized":"2023-11-24T08:20:33.270072+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:17.426766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295476+0000","last_change":"2023-11-24T08:20:32.280310+0000","last_active":"2023-11-24T08:20:33.295476+0000","last_peered":"2023-11-24T08:20:33.295476+0000","last_clean":"2023-11-24T08:20:33.295476+0000","last_became_active":"2023-11-24T08:20:32.278557+0000","last_became_peered":"2023-11-24T08:20:32.278557+0000","last_unstale":"2023-11-24T08:20:33.295476+0000","last_undegraded":"2023-11-24T08:20:33.295476+0000","last_fullsized":"2023-11-24T08:20:33.295476+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:42.464228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287598+0000","last_change":"2023-11-24T08:20:33.276526+0000","last_active":"2023-11-24T08:20:33.287598+0000","last_peered":"2023-11-24T08:20:33.287598+0000","last_clean":"2023-11-24T08:20:33.287598+0000","last_became_active":"2023-11-24T08:20:33.271579+0000","last_became_peered":"2023-11-24T08:20:33.271579+0000","last_unstale":"2023-11-24T08:20:33.287598+0000","last_undegraded":"2023-11-24T08:20:33.287598+0000","last_fullsized":"2023-11-24T08:20:33.287598+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:31:38.672205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.297851+0000","last_change":"2023-11-24T08:20:32.271070+0000","last_active":"2023-11-24T08:20:33.297851+0000","last_peered":"2023-11-24T08:20:33.297851+0000","last_clean":"2023-11-24T08:20:33.297851+0000","last_became_active":"2023-11-24T08:20:32.270561+0000","last_became_peered":"2023-11-24T08:20:32.270561+0000","last_unstale":"2023-11-24T08:20:33.297851+0000","last_undegraded":"2023-11-24T08:20:33.297851+0000","last_fullsized":"2023-11-24T08:20:33.297851+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:19:11.276699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279698+0000","last_change":"2023-11-24T08:20:33.275857+0000","last_active":"2023-11-24T08:20:33.279698+0000","last_peered":"2023-11-24T08:20:33.279698+0000","last_clean":"2023-11-24T08:20:33.279698+0000","last_became_active":"2023-11-24T08:20:33.275493+0000","last_became_peered":"2023-11-24T08:20:33.275493+0000","last_unstale":"2023-11-24T08:20:33.279698+0000","last_undegraded":"2023-11-24T08:20:33.279698+0000","last_fullsized":"2023-11-24T08:20:33.279698+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:15:09.708754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.254663+0000","last_change":"2023-11-24T08:20:32.270086+0000","last_active":"2023-11-24T08:20:33.254663+0000","last_peered":"2023-11-24T08:20:33.254663+0000","last_clean":"2023-11-24T08:20:33.254663+0000","last_became_active":"2023-11-24T08:20:32.269791+0000","last_became_peered":"2023-11-24T08:20:32.269791+0000","last_unstale":"2023-11-24T08:20:33.254663+0000","last_undegraded":"2023-11-24T08:20:33.254663+0000","last_fullsized":"2023-11-24T08:20:33.254663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:20:43.325875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279518+0000","last_change":"2023-11-24T08:20:33.270636+0000","last_active":"2023-11-24T08:20:33.279518+0000","last_peered":"2023-11-24T08:20:33.279518+0000","last_clean":"2023-11-24T08:20:33.279518+0000","last_became_active":"2023-11-24T08:20:33.270352+0000","last_became_peered":"2023-11-24T08:20:33.270352+0000","last_unstale":"2023-11-24T08:20:33.279518+0000","last_undegraded":"2023-11-24T08:20:33.279518+0000","last_fullsized":"2023-11-24T08:20:33.279518+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:32.165518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295483+0000","last_change":"2023-11-24T08:20:32.267544+0000","last_active":"2023-11-24T08:20:33.295483+0000","last_peered":"2023-11-24T08:20:33.295483+0000","last_clean":"2023-11-24T08:20:33.295483+0000","last_became_active":"2023-11-24T08:20:32.267323+0000","last_became_peered":"2023-11-24T08:20:32.267323+0000","last_unstale":"2023-11-24T08:20:33.295483+0000","last_undegraded":"2023-11-24T08:20:33.295483+0000","last_fullsized":"2023-11-24T08:20:33.295483+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:14.836804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279575+0000","last_change":"2023-11-24T08:20:33.273030+0000","last_active":"2023-11-24T08:20:33.279575+0000","last_peered":"2023-11-24T08:20:33.279575+0000","last_clean":"2023-11-24T08:20:33.279575+0000","last_became_active":"2023-11-24T08:20:33.272730+0000","last_became_peered":"2023-11-24T08:20:33.272730+0000","last_unstale":"2023-11-24T08:20:33.279575+0000","last_undegraded":"2023-11-24T08:20:33.279575+0000","last_fullsized":"2023-11-24T08:20:33.279575+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:58.121397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.260039+0000","last_change":"2023-11-24T08:20:32.272923+0000","last_active":"2023-11-24T08:20:33.260039+0000","last_peered":"2023-11-24T08:20:33.260039+0000","last_clean":"2023-11-24T08:20:33.260039+0000","last_became_active":"2023-11-24T08:20:32.269689+0000","last_became_peered":"2023-11-24T08:20:32.269689+0000","last_unstale":"2023-11-24T08:20:33.260039+0000","last_undegraded":"2023-11-24T08:20:33.260039+0000","last_fullsized":"2023-11-24T08:20:33.260039+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:29.097591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.266005+0000","last_change":"2023-11-24T08:20:33.260600+0000","last_active":"2023-11-24T08:20:33.266005+0000","last_peered":"2023-11-24T08:20:33.266005+0000","last_clean":"2023-11-24T08:20:33.266005+0000","last_became_active":"2023-11-24T08:20:33.260207+0000","last_became_peered":"2023-11-24T08:20:33.260207+0000","last_unstale":"2023-11-24T08:20:33.266005+0000","last_undegraded":"2023-11-24T08:20:33.266005+0000","last_fullsized":"2023-11-24T08:20:33.266005+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:00.007291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.260102+0000","last_change":"2023-11-24T08:20:32.271411+0000","last_active":"2023-11-24T08:20:33.260102+0000","last_peered":"2023-11-24T08:20:33.260102+0000","last_clean":"2023-11-24T08:20:33.260102+0000","last_became_active":"2023-11-24T08:20:32.271061+0000","last_became_peered":"2023-11-24T08:20:32.271061+0000","last_unstale":"2023-11-24T08:20:33.260102+0000","last_undegraded":"2023-11-24T08:20:33.260102+0000","last_fullsized":"2023-11-24T08:20:33.260102+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:46.176942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.272541+0000","last_change":"2023-11-24T08:20:33.270519+0000","last_active":"2023-11-24T08:20:33.272541+0000","last_peered":"2023-11-24T08:20:33.272541+0000","last_clean":"2023-11-24T08:20:33.272541+0000","last_became_active":"2023-11-24T08:20:33.270216+0000","last_became_peered":"2023-11-24T08:20:33.270216+0000","last_unstale":"2023-11-24T08:20:33.272541+0000","last_undegraded":"2023-11-24T08:20:33.272541+0000","last_fullsized":"2023-11-24T08:20:33.272541+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:05.626187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.252024+0000","last_change":"2023-11-24T08:20:32.281739+0000","last_active":"2023-11-24T08:20:33.252024+0000","last_peered":"2023-11-24T08:20:33.252024+0000","last_clean":"2023-11-24T08:20:33.252024+0000","last_became_active":"2023-11-24T08:20:32.281454+0000","last_became_peered":"2023-11-24T08:20:32.281454+0000","last_unstale":"2023-11-24T08:20:33.252024+0000","last_undegraded":"2023-11-24T08:20:33.252024+0000","last_fullsized":"2023-11-24T08:20:33.252024+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:26.067331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274319+0000","last_change":"2023-11-24T08:20:33.266517+0000","last_active":"2023-11-24T08:20:33.274319+0000","last_peered":"2023-11-24T08:20:33.274319+0000","last_clean":"2023-11-24T08:20:33.274319+0000","last_became_active":"2023-11-24T08:20:33.266222+0000","last_became_peered":"2023-11-24T08:20:33.266222+0000","last_unstale":"2023-11-24T08:20:33.274319+0000","last_undegraded":"2023-11-24T08:20:33.274319+0000","last_fullsized":"2023-11-24T08:20:33.274319+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:46.596079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295928+0000","last_change":"2023-11-24T08:20:32.282694+0000","last_active":"2023-11-24T08:20:33.295928+0000","last_peered":"2023-11-24T08:20:33.295928+0000","last_clean":"2023-11-24T08:20:33.295928+0000","last_became_active":"2023-11-24T08:20:32.282433+0000","last_became_peered":"2023-11-24T08:20:32.282433+0000","last_unstale":"2023-11-24T08:20:33.295928+0000","last_undegraded":"2023-11-24T08:20:33.295928+0000","last_fullsized":"2023-11-24T08:20:33.295928+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:24:27.122557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281300+0000","last_change":"2023-11-24T08:20:33.274461+0000","last_active":"2023-11-24T08:20:33.281300+0000","last_peered":"2023-11-24T08:20:33.281300+0000","last_clean":"2023-11-24T08:20:33.281300+0000","last_became_active":"2023-11-24T08:20:33.273798+0000","last_became_peered":"2023-11-24T08:20:33.273798+0000","last_unstale":"2023-11-24T08:20:33.281300+0000","last_undegraded":"2023-11-24T08:20:33.281300+0000","last_fullsized":"2023-11-24T08:20:33.281300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:18.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261668+0000","last_change":"2023-11-24T08:20:32.281655+0000","last_active":"2023-11-24T08:20:33.261668+0000","last_peered":"2023-11-24T08:20:33.261668+0000","last_clean":"2023-11-24T08:20:33.261668+0000","last_became_active":"2023-11-24T08:20:32.281141+0000","last_became_peered":"2023-11-24T08:20:32.281141+0000","last_unstale":"2023-11-24T08:20:33.261668+0000","last_undegraded":"2023-11-24T08:20:33.261668+0000","last_fullsized":"2023-11-24T08:20:33.261668+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:19:07.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275384+0000","last_change":"2023-11-24T08:20:33.272189+0000","last_active":"2023-11-24T08:20:33.275384+0000","last_peered":"2023-11-24T08:20:33.275384+0000","last_clean":"2023-11-24T08:20:33.275384+0000","last_became_active":"2023-11-24T08:20:33.271875+0000","last_became_peered":"2023-11-24T08:20:33.271875+0000","last_unstale":"2023-11-24T08:20:33.275384+0000","last_undegraded":"2023-11-24T08:20:33.275384+0000","last_fullsized":"2023-11-24T08:20:33.275384+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:11:15.839239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.256434+0000","last_change":"2023-11-24T08:20:32.278559+0000","last_active":"2023-11-24T08:20:33.256434+0000","last_peered":"2023-11-24T08:20:33.256434+0000","last_clean":"2023-11-24T08:20:33.256434+0000","last_became_active":"2023-11-24T08:20:32.278255+0000","last_became_peered":"2023-11-24T08:20:32.278255+0000","last_unstale":"2023-11-24T08:20:33.256434+0000","last_undegraded":"2023-11-24T08:20:33.256434+0000","last_fullsized":"2023-11-24T08:20:33.256434+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:23:52.576515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.272890+0000","last_change":"2023-11-24T08:20:33.269511+0000","last_active":"2023-11-24T08:20:33.272890+0000","last_peered":"2023-11-24T08:20:33.272890+0000","last_clean":"2023-11-24T08:20:33.272890+0000","last_became_active":"2023-11-24T08:20:33.269217+0000","last_became_peered":"2023-11-24T08:20:33.269217+0000","last_unstale":"2023-11-24T08:20:33.272890+0000","last_undegraded":"2023-11-24T08:20:33.272890+0000","last_fullsized":"2023-11-24T08:20:33.272890+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:57.199781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.254956+0000","last_change":"2023-11-24T08:20:32.279337+0000","last_active":"2023-11-24T08:20:33.254956+0000","last_peered":"2023-11-24T08:20:33.254956+0000","last_clean":"2023-11-24T08:20:33.254956+0000","last_became_active":"2023-11-24T08:20:32.278997+0000","last_became_peered":"2023-11-24T08:20:32.278997+0000","last_unstale":"2023-11-24T08:20:33.254956+0000","last_undegraded":"2023-11-24T08:20:33.254956+0000","last_fullsized":"2023-11-24T08:20:33.254956+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:59.083859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283220+0000","last_change":"2023-11-24T08:20:33.277788+0000","last_active":"2023-11-24T08:20:33.283220+0000","last_peered":"2023-11-24T08:20:33.283220+0000","last_clean":"2023-11-24T08:20:33.283220+0000","last_became_active":"2023-11-24T08:20:33.270945+0000","last_became_peered":"2023-11-24T08:20:33.270945+0000","last_unstale":"2023-11-24T08:20:33.283220+0000","last_undegraded":"2023-11-24T08:20:33.283220+0000","last_fullsized":"2023-11-24T08:20:33.283220+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:14.064035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261387+0000","last_change":"2023-11-24T08:20:32.283547+0000","last_active":"2023-11-24T08:20:33.261387+0000","last_peered":"2023-11-24T08:20:33.261387+0000","last_clean":"2023-11-24T08:20:33.261387+0000","last_became_active":"2023-11-24T08:20:32.282341+0000","last_became_peered":"2023-11-24T08:20:32.282341+0000","last_unstale":"2023-11-24T08:20:33.261387+0000","last_undegraded":"2023-11-24T08:20:33.261387+0000","last_fullsized":"2023-11-24T08:20:33.261387+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:06.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286058+0000","last_change":"2023-11-24T08:20:33.281715+0000","last_active":"2023-11-24T08:20:33.286058+0000","last_peered":"2023-11-24T08:20:33.286058+0000","last_clean":"2023-11-24T08:20:33.286058+0000","last_became_active":"2023-11-24T08:20:33.281391+0000","last_became_peered":"2023-11-24T08:20:33.281391+0000","last_unstale":"2023-11-24T08:20:33.286058+0000","last_undegraded":"2023-11-24T08:20:33.286058+0000","last_fullsized":"2023-11-24T08:20:33.286058+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.256709+0000","last_change":"2023-11-24T08:20:32.281836+0000","last_active":"2023-11-24T08:20:33.256709+0000","last_peered":"2023-11-24T08:20:33.256709+0000","last_clean":"2023-11-24T08:20:33.256709+0000","last_became_active":"2023-11-24T08:20:32.281567+0000","last_became_peered":"2023-11-24T08:20:32.281567+0000","last_unstale":"2023-11-24T08:20:33.256709+0000","last_undegraded":"2023-11-24T08:20:33.256709+0000","last_fullsized":"2023-11-24T08:20:33.256709+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:22.660075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287716+0000","last_change":"2023-11-24T08:20:33.277816+0000","last_active":"2023-11-24T08:20:33.287716+0000","last_peered":"2023-11-24T08:20:33.287716+0000","last_clean":"2023-11-24T08:20:33.287716+0000","last_became_active":"2023-11-24T08:20:33.277463+0000","last_became_peered":"2023-11-24T08:20:33.277463+0000","last_unstale":"2023-11-24T08:20:33.287716+0000","last_undegraded":"2023-11-24T08:20:33.287716+0000","last_fullsized":"2023-11-24T08:20:33.287716+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:45.094980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.249151+0000","last_change":"2023-11-24T08:20:32.274305+0000","last_active":"2023-11-24T08:20:33.249151+0000","last_peered":"2023-11-24T08:20:33.249151+0000","last_clean":"2023-11-24T08:20:33.249151+0000","last_became_active":"2023-11-24T08:20:32.272767+0000","last_became_peered":"2023-11-24T08:20:32.272767+0000","last_unstale":"2023-11-24T08:20:33.249151+0000","last_undegraded":"2023-11-24T08:20:33.249151+0000","last_fullsized":"2023-11-24T08:20:33.249151+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:26.067331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.271448+0000","last_change":"2023-11-24T08:20:33.270073+0000","last_active":"2023-11-24T08:20:33.271448+0000","last_peered":"2023-11-24T08:20:33.271448+0000","last_clean":"2023-11-24T08:20:33.271448+0000","last_became_active":"2023-11-24T08:20:33.269791+0000","last_became_peered":"2023-11-24T08:20:33.269791+0000","last_unstale":"2023-11-24T08:20:33.271448+0000","last_undegraded":"2023-11-24T08:20:33.271448+0000","last_fullsized":"2023-11-24T08:20:33.271448+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:04.104523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261532+0000","last_change":"2023-11-24T08:20:32.274909+0000","last_active":"2023-11-24T08:20:33.261532+0000","last_peered":"2023-11-24T08:20:33.261532+0000","last_clean":"2023-11-24T08:20:33.261532+0000","last_became_active":"2023-11-24T08:20:32.274632+0000","last_became_peered":"2023-11-24T08:20:32.274632+0000","last_unstale":"2023-11-24T08:20:33.261532+0000","last_undegraded":"2023-11-24T08:20:33.261532+0000","last_fullsized":"2023-11-24T08:20:33.261532+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:52.699687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283238+0000","last_change":"2023-11-24T08:20:33.278315+0000","last_active":"2023-11-24T08:20:33.283238+0000","last_peered":"2023-11-24T08:20:33.283238+0000","last_clean":"2023-11-24T08:20:33.283238+0000","last_became_active":"2023-11-24T08:20:33.278051+0000","last_became_peered":"2023-11-24T08:20:33.278051+0000","last_unstale":"2023-11-24T08:20:33.283238+0000","last_undegraded":"2023-11-24T08:20:33.283238+0000","last_fullsized":"2023-11-24T08:20:33.283238+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:58:51.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.292195+0000","last_change":"2023-11-24T08:20:32.280816+0000","last_active":"2023-11-24T08:20:33.292195+0000","last_peered":"2023-11-24T08:20:33.292195+0000","last_clean":"2023-11-24T08:20:33.292195+0000","last_became_active":"2023-11-24T08:20:32.280455+0000","last_became_peered":"2023-11-24T08:20:32.280455+0000","last_unstale":"2023-11-24T08:20:33.292195+0000","last_undegraded":"2023-11-24T08:20:33.292195+0000","last_fullsized":"2023-11-24T08:20:33.292195+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:23:52.576515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":313,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-11-24T08:20:32.270163+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:20:32.270163+0000","last_peered":"2023-11-24T08:20:32.270163+0000","last_clean":"2023-11-24T08:20:32.270163+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:20:32.270163+0000","last_undegraded":"2023-11-24T08:20:32.270163+0000","last_fullsized":"2023-11-24T08:20:32.270163+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42416,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329424,"statfs":{"total":96636764160,"available":96593330176,"internally_reserved":0,"allocated":2985984,"data_stored":1643375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0880000000000001,"15min":1.0660000000000001},"min":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":1.7030000000000001,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.1259999999999999,"15min":1.105},"min":{"1min":0.67200000000000004,"5min":0.67100000000000004,"15min":0.52900000000000003},"max":{"1min":1.782,"5min":1.782,"15min":2.7109999999999999},"last":1.417}]},{"osd":1,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1719999999999999,"15min":1.1599999999999999},"min":{"1min":0.83899999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.532,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":0.88},{"interface":"front","average":{"1min":1.274,"5min":1.2330000000000001,"15min":1.2230000000000001},"min":{"1min":0.94299999999999995,"5min":0.71099999999999997,"15min":0.628},"max":{"1min":1.6459999999999999,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.1299999999999999},"min":{"1min":0.80800000000000005,"5min":0.67500000000000004,"15min":0.55700000000000005},"max":{"1min":1.6719999999999999,"5min":1.698,"15min":4.7000000000000002},"last":1.135},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.93500000000000005,"5min":0.71799999999999997,"15min":0.60399999999999998},"max":{"1min":1.591,"5min":1.8480000000000001,"15min":4.633},"last":1.3859999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.504},"max":{"1min":1.6240000000000001,"5min":1.99,"15min":3.133},"last":1.548},{"interface":"front","average":{"1min":1.327,"5min":1.373,"15min":1.373},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":2.173,"5min":2.173,"15min":3.9209999999999998},"last":1.252}]},{"osd":4,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3959999999999999,"15min":1.357},"min":{"1min":0.95199999999999996,"5min":0.83699999999999997,"15min":0.54700000000000004},"max":{"1min":1.879,"5min":2.665,"15min":2.851},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.399,"15min":1.3560000000000001},"min":{"1min":1.04,"5min":0.82999999999999996,"15min":0.45400000000000001},"max":{"1min":1.653,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.3520000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.4950000000000001,"15min":1.476},"min":{"1min":1.006,"5min":0.64100000000000001,"15min":0.42699999999999999},"max":{"1min":1.762,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":0.998},{"interface":"front","average":{"1min":1.464,"5min":1.5209999999999999,"15min":1.498},"min":{"1min":1.1200000000000001,"5min":0.92700000000000005,"15min":0.622},"max":{"1min":1.98,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.288}]},{"osd":6,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6080000000000001,"15min":1.5760000000000001},"min":{"1min":1.319,"5min":1.0649999999999999,"15min":0.60199999999999998},"max":{"1min":1.9810000000000001,"5min":2.702,"15min":2.992},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.544,"15min":1.508},"min":{"1min":1.179,"5min":0.97799999999999998,"15min":0.68700000000000006},"max":{"1min":1.788,"5min":2.6259999999999999,"15min":3.798},"last":1.454}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37668,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334172,"statfs":{"total":96636764160,"available":96598192128,"internally_reserved":0,"allocated":2383872,"data_stored":1051977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0569999999999999,"15min":1.077},"min":{"1min":0.73699999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.516,"5min":1.7929999999999999,"15min":1.95},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.0880000000000001,"15min":1.085},"min":{"1min":0.83599999999999997,"5min":0.63900000000000001,"15min":0.58099999999999996},"max":{"1min":2.008,"5min":2.008,"15min":2.2069999999999999},"last":0.624}]},{"osd":1,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1020000000000001,"15min":1.1559999999999999},"min":{"1min":0.75600000000000001,"5min":0.65200000000000002,"15min":0.61599999999999999},"max":{"1min":1.5669999999999999,"5min":1.835,"15min":2.0230000000000001},"last":1.026},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.236,"15min":1.202},"min":{"1min":0.86899999999999999,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":2.206,"5min":2.206,"15min":2.4039999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.093,"15min":1.1020000000000001},"min":{"1min":0.79900000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7090000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.419},{"interface":"front","average":{"1min":1.216,"5min":1.238,"15min":1.2689999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.629,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":1.1659999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3400000000000001,"15min":1.3440000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.274,"5min":1.2589999999999999,"15min":1.3069999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.907,"5min":1.907,"15min":4.8319999999999999},"last":1.216}]},{"osd":4,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.427,"15min":1.431},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.673,"5min":3.673,"15min":3.673},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.48,"15min":1.492},"min":{"1min":0.88,"5min":0.88,"15min":0.63800000000000001},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8820000000000001},"last":1.1160000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.377,"15min":1.4259999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.5960000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.325},{"interface":"front","average":{"1min":1.587,"5min":1.573,"15min":1.5760000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.83299999999999996},"max":{"1min":3.8029999999999999,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.2569999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.603,"15min":1.601},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.748},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.601,"5min":1.591,"15min":1.599},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.498}]}]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37672,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334168,"statfs":{"total":96636764160,"available":96598188032,"internally_reserved":0,"allocated":2383872,"data_stored":1051977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0209999999999999,"15min":1.0289999999999999},"min":{"1min":0.55700000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.671,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.0880000000000001,"15min":1.109},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6559999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":1.6319999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.149,"15min":1.129},"min":{"1min":0.69999999999999996,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.518,"5min":7.258,"15min":7.258},"last":1.585},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.202,"15min":1.181},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.52,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":1.659}]},{"osd":2,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.7829999999999999,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":1.337},{"interface":"front","average":{"1min":1.081,"5min":1.194,"15min":1.1679999999999999},"min":{"1min":0.72399999999999998,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.411,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.8149999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2450000000000001,"15min":1.179},"min":{"1min":0.70799999999999996,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.754,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3220000000000001,"15min":1.2849999999999999},"min":{"1min":0.85399999999999998,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.956,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.5289999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.47,"15min":1.369},"min":{"1min":1.0069999999999999,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.905,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.5449999999999999,"15min":1.488},"min":{"1min":1.1819999999999999,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":2.0579999999999998,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.7450000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3979999999999999,"15min":1.355},"min":{"1min":0.94999999999999996,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.611,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":2.0129999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.5229999999999999,"15min":1.4510000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":1.794,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.8959999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5369999999999999,"15min":1.5029999999999999},"min":{"1min":1.117,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":1.7110000000000001,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":2.0939999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.7529999999999999,"15min":1.6419999999999999},"min":{"1min":1.099,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.0070000000000001,"5min":19.663,"15min":19.663},"last":1.78}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41844,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329996,"statfs":{"total":96636764160,"available":96593915904,"internally_reserved":0,"allocated":2400256,"data_stored":1053111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.022,"15min":1.0600000000000001},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.42699999999999999},"max":{"1min":1.5449999999999999,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.022},{"interface":"front","average":{"1min":1.125,"5min":1.048,"15min":1.0800000000000001},"min":{"1min":0.81299999999999994,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.544,"5min":1.927,"15min":5.452},"last":1.47}]},{"osd":1,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1200000000000001,"15min":1.133},"min":{"1min":0.84999999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5009999999999999,"5min":1.7809999999999999,"15min":1.8100000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.119,"15min":1.1559999999999999},"min":{"1min":1.026,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.6839999999999999,"5min":1.7150000000000001,"15min":1.988},"last":1.3720000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0489999999999999,"15min":1.073},"min":{"1min":0.76800000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.482,"5min":1.6519999999999999,"15min":2.1419999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.238,"15min":1.2430000000000001},"min":{"1min":0.754,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.726,"5min":1.827,"15min":2.7229999999999999},"last":1.417}]},{"osd":3,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.341,"15min":1.3380000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.62},"max":{"1min":1.7869999999999999,"5min":1.9750000000000001,"15min":2.9100000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.2789999999999999,"15min":1.278},"min":{"1min":1.177,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":2.3340000000000001},"last":1.2709999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5209999999999999,"15min":1.4930000000000001},"min":{"1min":0.998,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.7549999999999999,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.397,"15min":1.431},"min":{"1min":0.89500000000000002,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":1.718,"5min":2.1469999999999998,"15min":2.8090000000000002},"last":1.1299999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.242,"15min":1.2629999999999999},"min":{"1min":0.94299999999999995,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.7110000000000001,"5min":1.8540000000000001,"15min":3.5899999999999999},"last":1.075},{"interface":"front","average":{"1min":1.524,"5min":1.534,"15min":1.5389999999999999},"min":{"1min":1.1259999999999999,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.8979999999999999,"5min":2.1190000000000002,"15min":3.391},"last":1.321}]},{"osd":7,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.4710000000000001,"15min":1.4950000000000001},"min":{"1min":1.3100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.861,"5min":2.2440000000000002,"15min":3.1230000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.46,"5min":1.4079999999999999,"15min":1.4179999999999999},"min":{"1min":1.1259999999999999,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":1.8100000000000001,"5min":2.3100000000000001,"15min":3.4849999999999999},"last":1.1970000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2367488,"data_stored":1044620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.78400000000000003,"15min":0.85899999999999999},"min":{"1min":0.58299999999999996,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":10.557},"last":0.747},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.90500000000000003,"15min":0.92300000000000004},"min":{"1min":0.66700000000000004,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5349999999999999,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.54900000000000004,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.165,"5min":1.71,"15min":2.101},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.85799999999999998,"15min":0.96899999999999997},"min":{"1min":0.55100000000000005,"5min":0.46200000000000002,"15min":0.379},"max":{"1min":1.452,"5min":1.452,"15min":13.090999999999999},"last":0.86499999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.83599999999999997,"15min":0.90600000000000003},"min":{"1min":0.46400000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.976,"5min":1.976,"15min":3.9489999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.95399999999999996,"15min":1.0580000000000001},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.45200000000000001},"max":{"1min":1.665,"5min":1.665,"15min":13.460000000000001},"last":0.90500000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.238,"15min":1.2709999999999999},"min":{"1min":0.878,"5min":0.77000000000000002,"15min":0.67800000000000005},"max":{"1min":1.8109999999999999,"5min":1.877,"15min":4.367},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2030000000000001,"15min":1.2170000000000001},"min":{"1min":0.76400000000000001,"5min":0.72399999999999998,"15min":0.57799999999999996},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.3380000000000001},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.2150000000000001,"15min":1.298},"min":{"1min":0.79500000000000004,"5min":0.68000000000000005,"15min":0.60499999999999998},"max":{"1min":2.0539999999999998,"5min":2.1230000000000002,"15min":13.231},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.1910000000000001,"15min":1.2470000000000001},"min":{"1min":0.95199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8620000000000001,"5min":1.992,"15min":5.1180000000000003},"last":1.496}]},{"osd":6,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.2629999999999999,"15min":1.355},"min":{"1min":0.76800000000000002,"5min":0.69499999999999995,"15min":0.68899999999999995},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.2190000000000001,"15min":1.2490000000000001},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.61399999999999999},"max":{"1min":1.8420000000000001,"5min":2.3980000000000001,"15min":3.4409999999999998},"last":1.3160000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.266,"15min":1.323},"min":{"1min":0.91500000000000004,"5min":0.79400000000000004,"15min":0.63200000000000001},"max":{"1min":1.8959999999999999,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.54},{"interface":"front","average":{"1min":1.371,"5min":1.252,"15min":1.3049999999999999},"min":{"1min":0.91500000000000004,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":1.9279999999999999,"5min":3.0030000000000001,"15min":4.032},"last":1.4650000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":2379776,"data_stored":1051395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69299999999999995,"15min":0.77200000000000002},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1459999999999999,"5min":1.601,"15min":1.9610000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.78700000000000003,"15min":0.86899999999999999},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.4810000000000001,"5min":1.7390000000000001,"15min":2.5779999999999998},"last":0.97899999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82299999999999995,"15min":0.90400000000000003},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4510000000000001,"5min":1.4850000000000001,"15min":1.9890000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.91000000000000003,"15min":0.95699999999999996},"min":{"1min":0.57799999999999996,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":2.6240000000000001},"last":1.518}]},{"osd":3,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.84999999999999998,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.4530000000000001,"5min":1.6850000000000001,"15min":2.2090000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.95399999999999996,"15min":1.034},"min":{"1min":0.57199999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.948,"5min":1.948,"15min":2.9950000000000001},"last":1.6659999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.276},"min":{"1min":0.70899999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.6220000000000001,"5min":1.782,"15min":2.7160000000000002},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.196,"15min":1.258},"min":{"1min":0.82299999999999995,"5min":0.745,"15min":0.745},"max":{"1min":1.8200000000000001,"5min":1.863,"15min":3.3279999999999998},"last":1.4670000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1679999999999999,"15min":1.2509999999999999},"min":{"1min":0.78200000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.74,"5min":1.857,"15min":2.508},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.2010000000000001,"15min":1.2509999999999999},"min":{"1min":0.80700000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.7689999999999999,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.1399999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1719999999999999,"15min":1.208},"min":{"1min":0.90000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.7370000000000001,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":1.165},{"interface":"front","average":{"1min":1.405,"5min":1.242,"15min":1.325},"min":{"1min":1.0349999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.6629999999999998},"last":1.4139999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.26,"15min":1.327},"min":{"1min":0.79600000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.7809999999999999,"5min":2.3999999999999999,"15min":2.859},"last":1.373},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.248,"15min":1.296},"min":{"1min":0.89500000000000002,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":3.3809999999999998},"last":1.321}]}]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41836,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330004,"statfs":{"total":96636764160,"available":96593924096,"internally_reserved":0,"allocated":2383872,"data_stored":1051417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78700000000000003,"15min":0.80300000000000005},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.333,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.88300000000000001,"15min":0.88600000000000001},"min":{"1min":0.46800000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.4039999999999999,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.1739999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.80500000000000005,"15min":0.79800000000000004},"min":{"1min":0.436,"5min":0.374,"15min":0.37},"max":{"1min":1.823,"5min":1.823,"15min":1.861},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.90400000000000003,"15min":0.91800000000000004},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.7110000000000001,"5min":1.7490000000000001,"15min":1.925},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.014,"15min":1},"min":{"1min":0.49199999999999999,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":1.6439999999999999,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.88700000000000001,"15min":0.91200000000000003},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5980000000000001,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":0.73199999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.272,"15min":1.2749999999999999},"min":{"1min":0.85299999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.7989999999999999,"5min":1.984,"15min":2.7959999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.264,"15min":1.254},"min":{"1min":0.84999999999999998,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":3.0409999999999999},"last":1.123}]},{"osd":5,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.236,"15min":1.2529999999999999},"min":{"1min":0.79500000000000004,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2110000000000001,"15min":1.21},"min":{"1min":0.91200000000000003,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":3.286},"last":1.1950000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.244,"15min":1.2709999999999999},"min":{"1min":0.81299999999999994,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.365},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2509999999999999,"15min":1.264},"min":{"1min":1.0209999999999999,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3180000000000001,"15min":1.3140000000000001},"min":{"1min":0.86799999999999999,"5min":0.80200000000000005,"15min":0.59099999999999997},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":3.1760000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.29,"15min":1.3240000000000001},"min":{"1min":0.89300000000000002,"5min":0.624,"15min":0.624},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":3.653},"last":1.036}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38248,"kb_used_data":2908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333592,"statfs":{"total":96636764160,"available":96597598208,"internally_reserved":0,"allocated":2977792,"data_stored":1641853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.745},"min":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.5189999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.79800000000000004,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.3360000000000001,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":1.4490000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.004,"15min":1.0209999999999999},"min":{"1min":0.60799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5660000000000001,"5min":1.893,"15min":1.893},"last":1.089},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.58899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.671,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":0.92900000000000005,"15min":0.98799999999999999},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.44400000000000001},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0670000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.40100000000000002},"max":{"1min":1.7190000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":0.88400000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.1830000000000001,"15min":1.2350000000000001},"min":{"1min":0.77500000000000002,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.387},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.109,"15min":1.1240000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.417}]},{"osd":5,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1419999999999999,"15min":1.1679999999999999},"min":{"1min":0.86899999999999999,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.667,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.135},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.71099999999999997},"max":{"1min":1.532,"5min":1.7889999999999999,"15min":2.7240000000000002},"last":1.01}]},{"osd":6,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.2450000000000001,"15min":1.2749999999999999},"min":{"1min":0.91700000000000004,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.6539999999999999,"5min":2.052,"15min":2.4369999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.2370000000000001,"15min":1.274},"min":{"1min":0.94099999999999995,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.589,"5min":2.823,"15min":2.823},"last":1.323}]},{"osd":7,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.325,"15min":1.363},"min":{"1min":0.90100000000000002,"5min":0.86799999999999999,"15min":0.81200000000000006},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.512},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.2989999999999999,"15min":1.333},"min":{"1min":0.90200000000000002,"5min":0.747,"15min":0.747},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3929999999999998},"last":1.3560000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:20:37.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:20:37.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.945+0000 7fb18dd01640 1 Processor -- start 2023-11-24T08:20:37.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.945+0000 7fb18dd01640 1 -- start start 2023-11-24T08:20:37.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb18815c7f0 0x7fb18806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1880cf450 0x7fb1880cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1880cdfb0 0x7fb1880ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb188156010 con 0x7fb18815c7f0 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18806ce30 con 0x7fb1880cdfb0 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb18dd01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880cfe10 con 0x7fb1880cf450 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb1877fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb18815c7f0 0x7fb18806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb1877fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb18815c7f0 0x7fb18806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56754/0 (socket says 172.21.15.107:56754) 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.947+0000 7fb1877fe640 1 -- 172.21.15.107:0/28467824 learned_addr learned my addr 172.21.15.107:0/28467824 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb187fff640 1 --2- 172.21.15.107:0/28467824 >> v2:172.21.15.120:3300/0 conn(0x7fb1880cdfb0 0x7fb1880ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb186ffd640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1880cf450 0x7fb1880cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb1877fe640 1 -- 172.21.15.107:0/28467824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1880cf450 msgr2=0x7fb1880cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb1877fe640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1880cf450 0x7fb1880cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb1877fe640 1 -- 172.21.15.107:0/28467824 >> v2:172.21.15.120:3300/0 conn(0x7fb1880cdfb0 msgr2=0x7fb1880ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb1877fe640 1 --2- 172.21.15.107:0/28467824 >> v2:172.21.15.120:3300/0 conn(0x7fb1880cdfb0 0x7fb1880ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.948+0000 7fb1877fe640 1 -- 172.21.15.107:0/28467824 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880cece0 con 0x7fb18815c7f0 2023-11-24T08:20:37.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.949+0000 7fb1877fe640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb18815c7f0 0x7fb18806db20 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fb1880aebc0 tx=0x7fb17802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684084482658c7fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.949+0000 7fb1867fc640 1 -- 172.21.15.107:0/28467824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178031860 con 0x7fb18815c7f0 2023-11-24T08:20:37.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.949+0000 7fb1867fc640 1 -- 172.21.15.107:0/28467824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1780319c0 con 0x7fb18815c7f0 2023-11-24T08:20:37.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.949+0000 7fb1867fc640 1 -- 172.21.15.107:0/28467824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178031d50 con 0x7fb18815c7f0 2023-11-24T08:20:37.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 -- 172.21.15.107:0/28467824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb18815c7f0 msgr2=0x7fb18806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb18815c7f0 0x7fb18806db20 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fb1880aebc0 tx=0x7fb17802b900 comp rx=0 tx=0).stop 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 -- 172.21.15.107:0/28467824 shutdown_connections 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 --2- 172.21.15.107:0/28467824 >> v2:172.21.15.120:3300/0 conn(0x7fb1880cdfb0 0x7fb1880ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1880cf450 0x7fb1880cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 --2- 172.21.15.107:0/28467824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb18815c7f0 0x7fb18806db20 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 -- 172.21.15.107:0/28467824 >> 172.21.15.107:0/28467824 conn(0x7fb18806a920 msgr2=0x7fb18806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 -- 172.21.15.107:0/28467824 shutdown_connections 2023-11-24T08:20:37.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.950+0000 7fb18dd01640 1 -- 172.21.15.107:0/28467824 wait complete. 2023-11-24T08:20:37.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.952+0000 7fb18dd01640 1 Processor -- start 2023-11-24T08:20:37.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.952+0000 7fb18dd01640 1 -- start start 2023-11-24T08:20:37.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1880cf450 0x7fb188156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb18815c7f0 0x7fb1880ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18814ace0 con 0x7fb1880cdfb0 2023-11-24T08:20:37.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880b25c0 con 0x7fb1880cf450 2023-11-24T08:20:37.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb18dd01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880b28a0 con 0x7fb18815c7f0 2023-11-24T08:20:37.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56766/0 (socket says 172.21.15.107:56766) 2023-11-24T08:20:37.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 -- 172.21.15.107:0/3275478642 learned_addr learned my addr 172.21.15.107:0/3275478642 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 -- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb18815c7f0 msgr2=0x7fb1880ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb18815c7f0 0x7fb1880ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 -- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1880cf450 msgr2=0x7fb188156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1880cf450 0x7fb188156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.954+0000 7fb1877fe640 1 -- 172.21.15.107:0/3275478642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1780094d0 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb1877fe640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fb178000c00 tx=0x7fb1780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17803ac60 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17803adc0 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1880b2b80 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178044610 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.955+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1880b3140 con 0x7fb1880cdfb0 2023-11-24T08:20:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.956+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188067470 con 0x7fb1880cdfb0 2023-11-24T08:20:37.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.957+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb178044770 con 0x7fb1880cdfb0 2023-11-24T08:20:37.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.958+0000 7fb16ffff640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 0x7fb17406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:37.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.959+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1780aacc0 con 0x7fb1880cdfb0 2023-11-24T08:20:37.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.959+0000 7fb186ffd640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 0x7fb17406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:37.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.959+0000 7fb186ffd640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 0x7fb17406efb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cbbc0 tx=0x7fb17c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:37.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:37.963+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1780778f0 con 0x7fb1880cdfb0 2023-11-24T08:20:38.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.148+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb18806d600 con 0x7fb17406caf0 2023-11-24T08:20:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.170+0000 7fb16ffff640 1 -- 172.21.15.107:0/3275478642 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287590 (secure 0 0 0) 0x7fb17c009e40 con 0x7fb17406caf0 2023-11-24T08:20:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:38.180 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:20:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 msgr2=0x7fb17406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 0x7fb17406efb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cbbc0 tx=0x7fb17c008040 comp rx=0 tx=0).stop 2023-11-24T08:20:38.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 msgr2=0x7fb18814a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fb178000c00 tx=0x7fb1780046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 shutdown_connections 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb17406caf0 0x7fb17406efb0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb18815c7f0 0x7fb1880ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1880cf450 0x7fb188156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 --2- 172.21.15.107:0/3275478642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1880cdfb0 0x7fb18814a7a0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.177+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 >> 172.21.15.107:0/3275478642 conn(0x7fb18806a920 msgr2=0x7fb188154950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.178+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 shutdown_connections 2023-11-24T08:20:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.178+0000 7fb18dd01640 1 -- 172.21.15.107:0/3275478642 wait complete. 2023-11-24T08:20:38.197 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2023-11-24T08:20:37.731618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319160,"kb_used_data":19788,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754655560,"statfs":{"total":773094113280,"available":772767293440,"internally_reserved":0,"allocated":20262912,"data_stored":9589725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":306433598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001811"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282910+0000","last_change":"2023-11-24T08:20:33.275015+0000","last_active":"2023-11-24T08:20:33.282910+0000","last_peered":"2023-11-24T08:20:33.282910+0000","last_clean":"2023-11-24T08:20:33.282910+0000","last_became_active":"2023-11-24T08:20:33.274163+0000","last_became_peered":"2023-11-24T08:20:33.274163+0000","last_unstale":"2023-11-24T08:20:33.282910+0000","last_undegraded":"2023-11-24T08:20:33.282910+0000","last_fullsized":"2023-11-24T08:20:33.282910+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:53:46.467461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.264264+0000","last_change":"2023-11-24T08:20:33.257095+0000","last_active":"2023-11-24T08:20:33.264264+0000","last_peered":"2023-11-24T08:20:33.264264+0000","last_clean":"2023-11-24T08:20:33.264264+0000","last_became_active":"2023-11-24T08:20:33.256632+0000","last_became_peered":"2023-11-24T08:20:33.256632+0000","last_unstale":"2023-11-24T08:20:33.264264+0000","last_undegraded":"2023-11-24T08:20:33.264264+0000","last_fullsized":"2023-11-24T08:20:33.264264+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:24:28.139211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287773+0000","last_change":"2023-11-24T08:20:33.279680+0000","last_active":"2023-11-24T08:20:33.287773+0000","last_peered":"2023-11-24T08:20:33.287773+0000","last_clean":"2023-11-24T08:20:33.287773+0000","last_became_active":"2023-11-24T08:20:33.276732+0000","last_became_peered":"2023-11-24T08:20:33.276732+0000","last_unstale":"2023-11-24T08:20:33.287773+0000","last_undegraded":"2023-11-24T08:20:33.287773+0000","last_fullsized":"2023-11-24T08:20:33.287773+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:00:38.854360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278671+0000","last_change":"2023-11-24T08:20:33.274066+0000","last_active":"2023-11-24T08:20:33.278671+0000","last_peered":"2023-11-24T08:20:33.278671+0000","last_clean":"2023-11-24T08:20:33.278671+0000","last_became_active":"2023-11-24T08:20:33.273762+0000","last_became_peered":"2023-11-24T08:20:33.273762+0000","last_unstale":"2023-11-24T08:20:33.278671+0000","last_undegraded":"2023-11-24T08:20:33.278671+0000","last_fullsized":"2023-11-24T08:20:33.278671+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:35.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284652+0000","last_change":"2023-11-24T08:20:33.281609+0000","last_active":"2023-11-24T08:20:33.284652+0000","last_peered":"2023-11-24T08:20:33.284652+0000","last_clean":"2023-11-24T08:20:33.284652+0000","last_became_active":"2023-11-24T08:20:33.281306+0000","last_became_peered":"2023-11-24T08:20:33.281306+0000","last_unstale":"2023-11-24T08:20:33.284652+0000","last_undegraded":"2023-11-24T08:20:33.284652+0000","last_fullsized":"2023-11-24T08:20:33.284652+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:51.433824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.269870+0000","last_change":"2023-11-24T08:20:33.266900+0000","last_active":"2023-11-24T08:20:33.269870+0000","last_peered":"2023-11-24T08:20:33.269870+0000","last_clean":"2023-11-24T08:20:33.269870+0000","last_became_active":"2023-11-24T08:20:33.266584+0000","last_became_peered":"2023-11-24T08:20:33.266584+0000","last_unstale":"2023-11-24T08:20:33.269870+0000","last_undegraded":"2023-11-24T08:20:33.269870+0000","last_fullsized":"2023-11-24T08:20:33.269870+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:51.539078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287100+0000","last_change":"2023-11-24T08:20:33.282620+0000","last_active":"2023-11-24T08:20:33.287100+0000","last_peered":"2023-11-24T08:20:33.287100+0000","last_clean":"2023-11-24T08:20:33.287100+0000","last_became_active":"2023-11-24T08:20:33.282107+0000","last_became_peered":"2023-11-24T08:20:33.282107+0000","last_unstale":"2023-11-24T08:20:33.287100+0000","last_undegraded":"2023-11-24T08:20:33.287100+0000","last_fullsized":"2023-11-24T08:20:33.287100+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:58.338405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284689+0000","last_change":"2023-11-24T08:20:33.282341+0000","last_active":"2023-11-24T08:20:33.284689+0000","last_peered":"2023-11-24T08:20:33.284689+0000","last_clean":"2023-11-24T08:20:33.284689+0000","last_became_active":"2023-11-24T08:20:33.280592+0000","last_became_peered":"2023-11-24T08:20:33.280592+0000","last_unstale":"2023-11-24T08:20:33.284689+0000","last_undegraded":"2023-11-24T08:20:33.284689+0000","last_fullsized":"2023-11-24T08:20:33.284689+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.280885+0000","last_change":"2023-11-24T08:20:33.279063+0000","last_active":"2023-11-24T08:20:33.280885+0000","last_peered":"2023-11-24T08:20:33.280885+0000","last_clean":"2023-11-24T08:20:33.280885+0000","last_became_active":"2023-11-24T08:20:33.276192+0000","last_became_peered":"2023-11-24T08:20:33.276192+0000","last_unstale":"2023-11-24T08:20:33.280885+0000","last_undegraded":"2023-11-24T08:20:33.280885+0000","last_fullsized":"2023-11-24T08:20:33.280885+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:39.058582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285689+0000","last_change":"2023-11-24T08:20:33.281007+0000","last_active":"2023-11-24T08:20:33.285689+0000","last_peered":"2023-11-24T08:20:33.285689+0000","last_clean":"2023-11-24T08:20:33.285689+0000","last_became_active":"2023-11-24T08:20:33.280740+0000","last_became_peered":"2023-11-24T08:20:33.280740+0000","last_unstale":"2023-11-24T08:20:33.285689+0000","last_undegraded":"2023-11-24T08:20:33.285689+0000","last_fullsized":"2023-11-24T08:20:33.285689+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:33:46.310173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287936+0000","last_change":"2023-11-24T08:20:33.282159+0000","last_active":"2023-11-24T08:20:33.287936+0000","last_peered":"2023-11-24T08:20:33.287936+0000","last_clean":"2023-11-24T08:20:33.287936+0000","last_became_active":"2023-11-24T08:20:33.281885+0000","last_became_peered":"2023-11-24T08:20:33.281885+0000","last_unstale":"2023-11-24T08:20:33.287936+0000","last_undegraded":"2023-11-24T08:20:33.287936+0000","last_fullsized":"2023-11-24T08:20:33.287936+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:35:43.364804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274499+0000","last_change":"2023-11-24T08:20:33.267260+0000","last_active":"2023-11-24T08:20:33.274499+0000","last_peered":"2023-11-24T08:20:33.274499+0000","last_clean":"2023-11-24T08:20:33.274499+0000","last_became_active":"2023-11-24T08:20:33.266985+0000","last_became_peered":"2023-11-24T08:20:33.266985+0000","last_unstale":"2023-11-24T08:20:33.274499+0000","last_undegraded":"2023-11-24T08:20:33.274499+0000","last_fullsized":"2023-11-24T08:20:33.274499+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:27.083986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286038+0000","last_change":"2023-11-24T08:20:33.281678+0000","last_active":"2023-11-24T08:20:33.286038+0000","last_peered":"2023-11-24T08:20:33.286038+0000","last_clean":"2023-11-24T08:20:33.286038+0000","last_became_active":"2023-11-24T08:20:33.281380+0000","last_became_peered":"2023-11-24T08:20:33.281380+0000","last_unstale":"2023-11-24T08:20:33.286038+0000","last_undegraded":"2023-11-24T08:20:33.286038+0000","last_fullsized":"2023-11-24T08:20:33.286038+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:51.539078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281690+0000","last_change":"2023-11-24T08:20:33.272998+0000","last_active":"2023-11-24T08:20:33.281690+0000","last_peered":"2023-11-24T08:20:33.281690+0000","last_clean":"2023-11-24T08:20:33.281690+0000","last_became_active":"2023-11-24T08:20:33.272749+0000","last_became_peered":"2023-11-24T08:20:33.272749+0000","last_unstale":"2023-11-24T08:20:33.281690+0000","last_undegraded":"2023-11-24T08:20:33.281690+0000","last_fullsized":"2023-11-24T08:20:33.281690+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:02:00.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284979+0000","last_change":"2023-11-24T08:20:33.282410+0000","last_active":"2023-11-24T08:20:33.284979+0000","last_peered":"2023-11-24T08:20:33.284979+0000","last_clean":"2023-11-24T08:20:33.284979+0000","last_became_active":"2023-11-24T08:20:33.281756+0000","last_became_peered":"2023-11-24T08:20:33.281756+0000","last_unstale":"2023-11-24T08:20:33.284979+0000","last_undegraded":"2023-11-24T08:20:33.284979+0000","last_fullsized":"2023-11-24T08:20:33.284979+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:48.190825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285565+0000","last_change":"2023-11-24T08:20:33.278589+0000","last_active":"2023-11-24T08:20:33.285565+0000","last_peered":"2023-11-24T08:20:33.285565+0000","last_clean":"2023-11-24T08:20:33.285565+0000","last_became_active":"2023-11-24T08:20:33.278299+0000","last_became_peered":"2023-11-24T08:20:33.278299+0000","last_unstale":"2023-11-24T08:20:33.285565+0000","last_undegraded":"2023-11-24T08:20:33.285565+0000","last_fullsized":"2023-11-24T08:20:33.285565+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:57:18.109701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282930+0000","last_change":"2023-11-24T08:20:33.275217+0000","last_active":"2023-11-24T08:20:33.282930+0000","last_peered":"2023-11-24T08:20:33.282930+0000","last_clean":"2023-11-24T08:20:33.282930+0000","last_became_active":"2023-11-24T08:20:33.274731+0000","last_became_peered":"2023-11-24T08:20:33.274731+0000","last_unstale":"2023-11-24T08:20:33.282930+0000","last_undegraded":"2023-11-24T08:20:33.282930+0000","last_fullsized":"2023-11-24T08:20:33.282930+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:15:20.893125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.285505+0000","last_change":"2023-11-24T08:20:33.277350+0000","last_active":"2023-11-24T08:20:33.285505+0000","last_peered":"2023-11-24T08:20:33.285505+0000","last_clean":"2023-11-24T08:20:33.285505+0000","last_became_active":"2023-11-24T08:20:33.275259+0000","last_became_peered":"2023-11-24T08:20:33.275259+0000","last_unstale":"2023-11-24T08:20:33.285505+0000","last_undegraded":"2023-11-24T08:20:33.285505+0000","last_fullsized":"2023-11-24T08:20:33.285505+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:12:29.085586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286116+0000","last_change":"2023-11-24T08:20:33.282360+0000","last_active":"2023-11-24T08:20:33.286116+0000","last_peered":"2023-11-24T08:20:33.286116+0000","last_clean":"2023-11-24T08:20:33.286116+0000","last_became_active":"2023-11-24T08:20:33.281856+0000","last_became_peered":"2023-11-24T08:20:33.281856+0000","last_unstale":"2023-11-24T08:20:33.286116+0000","last_undegraded":"2023-11-24T08:20:33.286116+0000","last_fullsized":"2023-11-24T08:20:33.286116+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:35.502297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284499+0000","last_change":"2023-11-24T08:20:33.280157+0000","last_active":"2023-11-24T08:20:33.284499+0000","last_peered":"2023-11-24T08:20:33.284499+0000","last_clean":"2023-11-24T08:20:33.284499+0000","last_became_active":"2023-11-24T08:20:33.279873+0000","last_became_peered":"2023-11-24T08:20:33.279873+0000","last_unstale":"2023-11-24T08:20:33.284499+0000","last_undegraded":"2023-11-24T08:20:33.284499+0000","last_fullsized":"2023-11-24T08:20:33.284499+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:32.253157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278717+0000","last_change":"2023-11-24T08:20:33.274875+0000","last_active":"2023-11-24T08:20:33.278717+0000","last_peered":"2023-11-24T08:20:33.278717+0000","last_clean":"2023-11-24T08:20:33.278717+0000","last_became_active":"2023-11-24T08:20:33.274544+0000","last_became_peered":"2023-11-24T08:20:33.274544+0000","last_unstale":"2023-11-24T08:20:33.278717+0000","last_undegraded":"2023-11-24T08:20:33.278717+0000","last_fullsized":"2023-11-24T08:20:33.278717+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:31.664775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274577+0000","last_change":"2023-11-24T08:20:33.263590+0000","last_active":"2023-11-24T08:20:33.274577+0000","last_peered":"2023-11-24T08:20:33.274577+0000","last_clean":"2023-11-24T08:20:33.274577+0000","last_became_active":"2023-11-24T08:20:33.263307+0000","last_became_peered":"2023-11-24T08:20:33.263307+0000","last_unstale":"2023-11-24T08:20:33.274577+0000","last_undegraded":"2023-11-24T08:20:33.274577+0000","last_fullsized":"2023-11-24T08:20:33.274577+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:43.480882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278101+0000","last_change":"2023-11-24T08:20:33.275880+0000","last_active":"2023-11-24T08:20:33.278101+0000","last_peered":"2023-11-24T08:20:33.278101+0000","last_clean":"2023-11-24T08:20:33.278101+0000","last_became_active":"2023-11-24T08:20:33.275611+0000","last_became_peered":"2023-11-24T08:20:33.275611+0000","last_unstale":"2023-11-24T08:20:33.278101+0000","last_undegraded":"2023-11-24T08:20:33.278101+0000","last_fullsized":"2023-11-24T08:20:33.278101+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278727+0000","last_change":"2023-11-24T08:20:33.276152+0000","last_active":"2023-11-24T08:20:33.278727+0000","last_peered":"2023-11-24T08:20:33.278727+0000","last_clean":"2023-11-24T08:20:33.278727+0000","last_became_active":"2023-11-24T08:20:33.275832+0000","last_became_peered":"2023-11-24T08:20:33.275832+0000","last_unstale":"2023-11-24T08:20:33.278727+0000","last_undegraded":"2023-11-24T08:20:33.278727+0000","last_fullsized":"2023-11-24T08:20:33.278727+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:47.193596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.270051+0000","last_change":"2023-11-24T08:20:33.267060+0000","last_active":"2023-11-24T08:20:33.270051+0000","last_peered":"2023-11-24T08:20:33.270051+0000","last_clean":"2023-11-24T08:20:33.270051+0000","last_became_active":"2023-11-24T08:20:33.264117+0000","last_became_peered":"2023-11-24T08:20:33.264117+0000","last_unstale":"2023-11-24T08:20:33.270051+0000","last_undegraded":"2023-11-24T08:20:33.270051+0000","last_fullsized":"2023-11-24T08:20:33.270051+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:41.109560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282251+0000","last_change":"2023-11-24T08:20:33.274992+0000","last_active":"2023-11-24T08:20:33.282251+0000","last_peered":"2023-11-24T08:20:33.282251+0000","last_clean":"2023-11-24T08:20:33.282251+0000","last_became_active":"2023-11-24T08:20:33.274315+0000","last_became_peered":"2023-11-24T08:20:33.274315+0000","last_unstale":"2023-11-24T08:20:33.282251+0000","last_undegraded":"2023-11-24T08:20:33.282251+0000","last_fullsized":"2023-11-24T08:20:33.282251+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:51.606324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.276290+0000","last_change":"2023-11-24T08:20:33.273248+0000","last_active":"2023-11-24T08:20:33.276290+0000","last_peered":"2023-11-24T08:20:33.276290+0000","last_clean":"2023-11-24T08:20:33.276290+0000","last_became_active":"2023-11-24T08:20:33.268914+0000","last_became_peered":"2023-11-24T08:20:33.268914+0000","last_unstale":"2023-11-24T08:20:33.276290+0000","last_undegraded":"2023-11-24T08:20:33.276290+0000","last_fullsized":"2023-11-24T08:20:33.276290+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:48.755542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287644+0000","last_change":"2023-11-24T08:20:33.277091+0000","last_active":"2023-11-24T08:20:33.287644+0000","last_peered":"2023-11-24T08:20:33.287644+0000","last_clean":"2023-11-24T08:20:33.287644+0000","last_became_active":"2023-11-24T08:20:33.274190+0000","last_became_peered":"2023-11-24T08:20:33.274190+0000","last_unstale":"2023-11-24T08:20:33.287644+0000","last_undegraded":"2023-11-24T08:20:33.287644+0000","last_fullsized":"2023-11-24T08:20:33.287644+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:09:42.881197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275925+0000","last_change":"2023-11-24T08:20:33.271677+0000","last_active":"2023-11-24T08:20:33.275925+0000","last_peered":"2023-11-24T08:20:33.275925+0000","last_clean":"2023-11-24T08:20:33.275925+0000","last_became_active":"2023-11-24T08:20:33.271370+0000","last_became_peered":"2023-11-24T08:20:33.271370+0000","last_unstale":"2023-11-24T08:20:33.275925+0000","last_undegraded":"2023-11-24T08:20:33.275925+0000","last_fullsized":"2023-11-24T08:20:33.275925+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:21.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.269402+0000","last_change":"2023-11-24T08:20:33.264978+0000","last_active":"2023-11-24T08:20:33.269402+0000","last_peered":"2023-11-24T08:20:33.269402+0000","last_clean":"2023-11-24T08:20:33.269402+0000","last_became_active":"2023-11-24T08:20:33.264406+0000","last_became_peered":"2023-11-24T08:20:33.264406+0000","last_unstale":"2023-11-24T08:20:33.269402+0000","last_undegraded":"2023-11-24T08:20:33.269402+0000","last_fullsized":"2023-11-24T08:20:33.269402+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:58.121397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279325+0000","last_change":"2023-11-24T08:20:33.275187+0000","last_active":"2023-11-24T08:20:33.279325+0000","last_peered":"2023-11-24T08:20:33.279325+0000","last_clean":"2023-11-24T08:20:33.279325+0000","last_became_active":"2023-11-24T08:20:33.274300+0000","last_became_peered":"2023-11-24T08:20:33.274300+0000","last_unstale":"2023-11-24T08:20:33.279325+0000","last_undegraded":"2023-11-24T08:20:33.279325+0000","last_fullsized":"2023-11-24T08:20:33.279325+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:00:32.072184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282043+0000","last_change":"2023-11-24T08:20:33.274878+0000","last_active":"2023-11-24T08:20:33.282043+0000","last_peered":"2023-11-24T08:20:33.282043+0000","last_clean":"2023-11-24T08:20:33.282043+0000","last_became_active":"2023-11-24T08:20:33.274556+0000","last_became_peered":"2023-11-24T08:20:33.274556+0000","last_unstale":"2023-11-24T08:20:33.282043+0000","last_undegraded":"2023-11-24T08:20:33.282043+0000","last_fullsized":"2023-11-24T08:20:33.282043+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:59:49.093706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274409+0000","last_change":"2023-11-24T08:20:33.271552+0000","last_active":"2023-11-24T08:20:33.274409+0000","last_peered":"2023-11-24T08:20:33.274409+0000","last_clean":"2023-11-24T08:20:33.274409+0000","last_became_active":"2023-11-24T08:20:33.270431+0000","last_became_peered":"2023-11-24T08:20:33.270431+0000","last_unstale":"2023-11-24T08:20:33.274409+0000","last_undegraded":"2023-11-24T08:20:33.274409+0000","last_fullsized":"2023-11-24T08:20:33.274409+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:57.199781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"189'4","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.290884+0000","last_change":"2023-11-24T08:20:32.280779+0000","last_active":"2023-11-24T08:20:33.290884+0000","last_peered":"2023-11-24T08:20:33.290884+0000","last_clean":"2023-11-24T08:20:33.290884+0000","last_became_active":"2023-11-24T08:20:32.280482+0000","last_became_peered":"2023-11-24T08:20:32.280482+0000","last_unstale":"2023-11-24T08:20:33.290884+0000","last_undegraded":"2023-11-24T08:20:33.290884+0000","last_fullsized":"2023-11-24T08:20:33.290884+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:22.660075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278170+0000","last_change":"2023-11-24T08:20:33.274492+0000","last_active":"2023-11-24T08:20:33.278170+0000","last_peered":"2023-11-24T08:20:33.278170+0000","last_clean":"2023-11-24T08:20:33.278170+0000","last_became_active":"2023-11-24T08:20:33.274193+0000","last_became_peered":"2023-11-24T08:20:33.274193+0000","last_unstale":"2023-11-24T08:20:33.278170+0000","last_undegraded":"2023-11-24T08:20:33.278170+0000","last_fullsized":"2023-11-24T08:20:33.278170+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:41.109560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.289234+0000","last_change":"2023-11-24T08:20:32.278574+0000","last_active":"2023-11-24T08:20:33.289234+0000","last_peered":"2023-11-24T08:20:33.289234+0000","last_clean":"2023-11-24T08:20:33.289234+0000","last_became_active":"2023-11-24T08:20:32.278127+0000","last_became_peered":"2023-11-24T08:20:32.278127+0000","last_unstale":"2023-11-24T08:20:33.289234+0000","last_undegraded":"2023-11-24T08:20:33.289234+0000","last_fullsized":"2023-11-24T08:20:33.289234+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:04.609533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.282890+0000","last_change":"2023-11-24T08:20:33.280061+0000","last_active":"2023-11-24T08:20:33.282890+0000","last_peered":"2023-11-24T08:20:33.282890+0000","last_clean":"2023-11-24T08:20:33.282890+0000","last_became_active":"2023-11-24T08:20:33.279799+0000","last_became_peered":"2023-11-24T08:20:33.279799+0000","last_unstale":"2023-11-24T08:20:33.282890+0000","last_undegraded":"2023-11-24T08:20:33.282890+0000","last_fullsized":"2023-11-24T08:20:33.282890+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:35.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.296770+0000","last_change":"2023-11-24T08:20:32.278740+0000","last_active":"2023-11-24T08:20:33.296770+0000","last_peered":"2023-11-24T08:20:33.296770+0000","last_clean":"2023-11-24T08:20:33.296770+0000","last_became_active":"2023-11-24T08:20:32.278202+0000","last_became_peered":"2023-11-24T08:20:32.278202+0000","last_unstale":"2023-11-24T08:20:33.296770+0000","last_undegraded":"2023-11-24T08:20:33.296770+0000","last_fullsized":"2023-11-24T08:20:33.296770+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:45.847300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281627+0000","last_change":"2023-11-24T08:20:33.276379+0000","last_active":"2023-11-24T08:20:33.281627+0000","last_peered":"2023-11-24T08:20:33.281627+0000","last_clean":"2023-11-24T08:20:33.281627+0000","last_became_active":"2023-11-24T08:20:33.276077+0000","last_became_peered":"2023-11-24T08:20:33.276077+0000","last_unstale":"2023-11-24T08:20:33.281627+0000","last_undegraded":"2023-11-24T08:20:33.281627+0000","last_fullsized":"2023-11-24T08:20:33.281627+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:12:17.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.293472+0000","last_change":"2023-11-24T08:20:32.279533+0000","last_active":"2023-11-24T08:20:33.293472+0000","last_peered":"2023-11-24T08:20:33.293472+0000","last_clean":"2023-11-24T08:20:33.293472+0000","last_became_active":"2023-11-24T08:20:32.279174+0000","last_became_peered":"2023-11-24T08:20:32.279174+0000","last_unstale":"2023-11-24T08:20:33.293472+0000","last_undegraded":"2023-11-24T08:20:33.293472+0000","last_fullsized":"2023-11-24T08:20:33.293472+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:57.321751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.277791+0000","last_change":"2023-11-24T08:20:33.273172+0000","last_active":"2023-11-24T08:20:33.277791+0000","last_peered":"2023-11-24T08:20:33.277791+0000","last_clean":"2023-11-24T08:20:33.277791+0000","last_became_active":"2023-11-24T08:20:33.272818+0000","last_became_peered":"2023-11-24T08:20:33.272818+0000","last_unstale":"2023-11-24T08:20:33.277791+0000","last_undegraded":"2023-11-24T08:20:33.277791+0000","last_fullsized":"2023-11-24T08:20:33.277791+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:02:00.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.289580+0000","last_change":"2023-11-24T08:20:32.272345+0000","last_active":"2023-11-24T08:20:33.289580+0000","last_peered":"2023-11-24T08:20:33.289580+0000","last_clean":"2023-11-24T08:20:33.289580+0000","last_became_active":"2023-11-24T08:20:32.272002+0000","last_became_peered":"2023-11-24T08:20:32.272002+0000","last_unstale":"2023-11-24T08:20:33.289580+0000","last_undegraded":"2023-11-24T08:20:33.289580+0000","last_fullsized":"2023-11-24T08:20:33.289580+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:52.699687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283575+0000","last_change":"2023-11-24T08:20:33.272726+0000","last_active":"2023-11-24T08:20:33.283575+0000","last_peered":"2023-11-24T08:20:33.283575+0000","last_clean":"2023-11-24T08:20:33.283575+0000","last_became_active":"2023-11-24T08:20:33.269024+0000","last_became_peered":"2023-11-24T08:20:33.269024+0000","last_unstale":"2023-11-24T08:20:33.283575+0000","last_undegraded":"2023-11-24T08:20:33.283575+0000","last_fullsized":"2023-11-24T08:20:33.283575+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:17.426766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.249719+0000","last_change":"2023-11-24T08:20:32.279187+0000","last_active":"2023-11-24T08:20:33.249719+0000","last_peered":"2023-11-24T08:20:33.249719+0000","last_clean":"2023-11-24T08:20:33.249719+0000","last_became_active":"2023-11-24T08:20:32.278221+0000","last_became_peered":"2023-11-24T08:20:32.278221+0000","last_unstale":"2023-11-24T08:20:33.249719+0000","last_undegraded":"2023-11-24T08:20:33.249719+0000","last_fullsized":"2023-11-24T08:20:33.249719+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:23:40.092906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.278668+0000","last_change":"2023-11-24T08:20:33.275984+0000","last_active":"2023-11-24T08:20:33.278668+0000","last_peered":"2023-11-24T08:20:33.278668+0000","last_clean":"2023-11-24T08:20:33.278668+0000","last_became_active":"2023-11-24T08:20:33.275275+0000","last_became_peered":"2023-11-24T08:20:33.275275+0000","last_unstale":"2023-11-24T08:20:33.278668+0000","last_undegraded":"2023-11-24T08:20:33.278668+0000","last_fullsized":"2023-11-24T08:20:33.278668+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:58.947708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.247816+0000","last_change":"2023-11-24T08:20:32.258971+0000","last_active":"2023-11-24T08:20:33.247816+0000","last_peered":"2023-11-24T08:20:33.247816+0000","last_clean":"2023-11-24T08:20:33.247816+0000","last_became_active":"2023-11-24T08:20:32.257755+0000","last_became_peered":"2023-11-24T08:20:32.257755+0000","last_unstale":"2023-11-24T08:20:33.247816+0000","last_undegraded":"2023-11-24T08:20:33.247816+0000","last_fullsized":"2023-11-24T08:20:33.247816+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:16.410112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275148+0000","last_change":"2023-11-24T08:20:33.270490+0000","last_active":"2023-11-24T08:20:33.275148+0000","last_peered":"2023-11-24T08:20:33.275148+0000","last_clean":"2023-11-24T08:20:33.275148+0000","last_became_active":"2023-11-24T08:20:33.270194+0000","last_became_peered":"2023-11-24T08:20:33.270194+0000","last_unstale":"2023-11-24T08:20:33.275148+0000","last_undegraded":"2023-11-24T08:20:33.275148+0000","last_fullsized":"2023-11-24T08:20:33.275148+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:51.601095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.251009+0000","last_change":"2023-11-24T08:20:32.278673+0000","last_active":"2023-11-24T08:20:33.251009+0000","last_peered":"2023-11-24T08:20:33.251009+0000","last_clean":"2023-11-24T08:20:33.251009+0000","last_became_active":"2023-11-24T08:20:32.278372+0000","last_became_peered":"2023-11-24T08:20:32.278372+0000","last_unstale":"2023-11-24T08:20:33.251009+0000","last_undegraded":"2023-11-24T08:20:33.251009+0000","last_fullsized":"2023-11-24T08:20:33.251009+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:45.579425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287842+0000","last_change":"2023-11-24T08:20:33.281071+0000","last_active":"2023-11-24T08:20:33.287842+0000","last_peered":"2023-11-24T08:20:33.287842+0000","last_clean":"2023-11-24T08:20:33.287842+0000","last_became_active":"2023-11-24T08:20:33.278156+0000","last_became_peered":"2023-11-24T08:20:33.278156+0000","last_unstale":"2023-11-24T08:20:33.287842+0000","last_undegraded":"2023-11-24T08:20:33.287842+0000","last_fullsized":"2023-11-24T08:20:33.287842+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:04.104523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.246602+0000","last_change":"2023-11-24T08:20:32.270934+0000","last_active":"2023-11-24T08:20:33.246602+0000","last_peered":"2023-11-24T08:20:33.246602+0000","last_clean":"2023-11-24T08:20:33.246602+0000","last_became_active":"2023-11-24T08:20:32.270056+0000","last_became_peered":"2023-11-24T08:20:32.270056+0000","last_unstale":"2023-11-24T08:20:33.246602+0000","last_undegraded":"2023-11-24T08:20:33.246602+0000","last_fullsized":"2023-11-24T08:20:33.246602+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:38.041928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283225+0000","last_change":"2023-11-24T08:20:33.278863+0000","last_active":"2023-11-24T08:20:33.283225+0000","last_peered":"2023-11-24T08:20:33.283225+0000","last_clean":"2023-11-24T08:20:33.283225+0000","last_became_active":"2023-11-24T08:20:33.278398+0000","last_became_peered":"2023-11-24T08:20:33.278398+0000","last_unstale":"2023-11-24T08:20:33.283225+0000","last_undegraded":"2023-11-24T08:20:33.283225+0000","last_fullsized":"2023-11-24T08:20:33.283225+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:55:48.190825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"189'3","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.298245+0000","last_change":"2023-11-24T08:20:32.270893+0000","last_active":"2023-11-24T08:20:33.298245+0000","last_peered":"2023-11-24T08:20:33.298245+0000","last_clean":"2023-11-24T08:20:33.298245+0000","last_became_active":"2023-11-24T08:20:32.270121+0000","last_became_peered":"2023-11-24T08:20:32.270121+0000","last_unstale":"2023-11-24T08:20:33.298245+0000","last_undegraded":"2023-11-24T08:20:33.298245+0000","last_fullsized":"2023-11-24T08:20:33.298245+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:51:33.002261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.280243+0000","last_change":"2023-11-24T08:20:33.271132+0000","last_active":"2023-11-24T08:20:33.280243+0000","last_peered":"2023-11-24T08:20:33.280243+0000","last_clean":"2023-11-24T08:20:33.280243+0000","last_became_active":"2023-11-24T08:20:33.270772+0000","last_became_peered":"2023-11-24T08:20:33.270772+0000","last_unstale":"2023-11-24T08:20:33.280243+0000","last_undegraded":"2023-11-24T08:20:33.280243+0000","last_fullsized":"2023-11-24T08:20:33.280243+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:08:35.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"189'6","reported_seq":17,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.302170+0000","last_change":"2023-11-24T08:20:32.283480+0000","last_active":"2023-11-24T08:20:33.302170+0000","last_peered":"2023-11-24T08:20:33.302170+0000","last_clean":"2023-11-24T08:20:33.302170+0000","last_became_active":"2023-11-24T08:20:32.283095+0000","last_became_peered":"2023-11-24T08:20:32.283095+0000","last_unstale":"2023-11-24T08:20:33.302170+0000","last_undegraded":"2023-11-24T08:20:33.302170+0000","last_fullsized":"2023-11-24T08:20:33.302170+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:29.097591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284533+0000","last_change":"2023-11-24T08:20:33.281637+0000","last_active":"2023-11-24T08:20:33.284533+0000","last_peered":"2023-11-24T08:20:33.284533+0000","last_clean":"2023-11-24T08:20:33.284533+0000","last_became_active":"2023-11-24T08:20:33.281357+0000","last_became_peered":"2023-11-24T08:20:33.281357+0000","last_unstale":"2023-11-24T08:20:33.284533+0000","last_undegraded":"2023-11-24T08:20:33.284533+0000","last_fullsized":"2023-11-24T08:20:33.284533+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:21.084643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.296514+0000","last_change":"2023-11-24T08:20:32.279998+0000","last_active":"2023-11-24T08:20:33.296514+0000","last_peered":"2023-11-24T08:20:33.296514+0000","last_clean":"2023-11-24T08:20:33.296514+0000","last_became_active":"2023-11-24T08:20:32.279608+0000","last_became_peered":"2023-11-24T08:20:32.279608+0000","last_unstale":"2023-11-24T08:20:33.296514+0000","last_undegraded":"2023-11-24T08:20:33.296514+0000","last_fullsized":"2023-11-24T08:20:33.296514+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:45.579425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279679+0000","last_change":"2023-11-24T08:20:33.274597+0000","last_active":"2023-11-24T08:20:33.279679+0000","last_peered":"2023-11-24T08:20:33.279679+0000","last_clean":"2023-11-24T08:20:33.279679+0000","last_became_active":"2023-11-24T08:20:33.273854+0000","last_became_peered":"2023-11-24T08:20:33.273854+0000","last_unstale":"2023-11-24T08:20:33.279679+0000","last_undegraded":"2023-11-24T08:20:33.279679+0000","last_fullsized":"2023-11-24T08:20:33.279679+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:47:25.942966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.248432+0000","last_change":"2023-11-24T08:20:32.274102+0000","last_active":"2023-11-24T08:20:33.248432+0000","last_peered":"2023-11-24T08:20:33.248432+0000","last_clean":"2023-11-24T08:20:33.248432+0000","last_became_active":"2023-11-24T08:20:32.273818+0000","last_became_peered":"2023-11-24T08:20:32.273818+0000","last_unstale":"2023-11-24T08:20:33.248432+0000","last_undegraded":"2023-11-24T08:20:33.248432+0000","last_fullsized":"2023-11-24T08:20:33.248432+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:47.738888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275613+0000","last_change":"2023-11-24T08:20:33.272907+0000","last_active":"2023-11-24T08:20:33.275613+0000","last_peered":"2023-11-24T08:20:33.275613+0000","last_clean":"2023-11-24T08:20:33.275613+0000","last_became_active":"2023-11-24T08:20:33.272598+0000","last_became_peered":"2023-11-24T08:20:33.272598+0000","last_unstale":"2023-11-24T08:20:33.275613+0000","last_undegraded":"2023-11-24T08:20:33.275613+0000","last_fullsized":"2023-11-24T08:20:33.275613+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:40.352394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.267699+0000","last_change":"2023-11-24T08:20:32.274484+0000","last_active":"2023-11-24T08:20:33.267699+0000","last_peered":"2023-11-24T08:20:33.267699+0000","last_clean":"2023-11-24T08:20:33.267699+0000","last_became_active":"2023-11-24T08:20:32.274169+0000","last_became_peered":"2023-11-24T08:20:32.274169+0000","last_unstale":"2023-11-24T08:20:33.267699+0000","last_undegraded":"2023-11-24T08:20:33.267699+0000","last_fullsized":"2023-11-24T08:20:33.267699+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:31.023782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.284158+0000","last_change":"2023-11-24T08:20:33.278265+0000","last_active":"2023-11-24T08:20:33.284158+0000","last_peered":"2023-11-24T08:20:33.284158+0000","last_clean":"2023-11-24T08:20:33.284158+0000","last_became_active":"2023-11-24T08:20:33.277800+0000","last_became_peered":"2023-11-24T08:20:33.277800+0000","last_unstale":"2023-11-24T08:20:33.284158+0000","last_undegraded":"2023-11-24T08:20:33.284158+0000","last_fullsized":"2023-11-24T08:20:33.284158+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:36:31.664775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"189'8","reported_seq":19,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.299899+0000","last_change":"2023-11-24T08:20:32.273022+0000","last_active":"2023-11-24T08:20:33.299899+0000","last_peered":"2023-11-24T08:20:33.299899+0000","last_clean":"2023-11-24T08:20:33.299899+0000","last_became_active":"2023-11-24T08:20:32.272736+0000","last_became_peered":"2023-11-24T08:20:32.272736+0000","last_unstale":"2023-11-24T08:20:33.299899+0000","last_undegraded":"2023-11-24T08:20:33.299899+0000","last_fullsized":"2023-11-24T08:20:33.299899+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:59:58.990637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275725+0000","last_change":"2023-11-24T08:20:33.272986+0000","last_active":"2023-11-24T08:20:33.275725+0000","last_peered":"2023-11-24T08:20:33.275725+0000","last_clean":"2023-11-24T08:20:33.275725+0000","last_became_active":"2023-11-24T08:20:33.272493+0000","last_became_peered":"2023-11-24T08:20:33.272493+0000","last_unstale":"2023-11-24T08:20:33.275725+0000","last_undegraded":"2023-11-24T08:20:33.275725+0000","last_fullsized":"2023-11-24T08:20:33.275725+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:46.596079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295584+0000","last_change":"2023-11-24T08:20:32.282307+0000","last_active":"2023-11-24T08:20:33.295584+0000","last_peered":"2023-11-24T08:20:33.295584+0000","last_clean":"2023-11-24T08:20:33.295584+0000","last_became_active":"2023-11-24T08:20:32.282041+0000","last_became_peered":"2023-11-24T08:20:32.282041+0000","last_unstale":"2023-11-24T08:20:33.295584+0000","last_undegraded":"2023-11-24T08:20:33.295584+0000","last_fullsized":"2023-11-24T08:20:33.295584+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:47.738888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.270072+0000","last_change":"2023-11-24T08:20:33.267202+0000","last_active":"2023-11-24T08:20:33.270072+0000","last_peered":"2023-11-24T08:20:33.270072+0000","last_clean":"2023-11-24T08:20:33.270072+0000","last_became_active":"2023-11-24T08:20:33.266908+0000","last_became_peered":"2023-11-24T08:20:33.266908+0000","last_unstale":"2023-11-24T08:20:33.270072+0000","last_undegraded":"2023-11-24T08:20:33.270072+0000","last_fullsized":"2023-11-24T08:20:33.270072+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:17.426766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295476+0000","last_change":"2023-11-24T08:20:32.280310+0000","last_active":"2023-11-24T08:20:33.295476+0000","last_peered":"2023-11-24T08:20:33.295476+0000","last_clean":"2023-11-24T08:20:33.295476+0000","last_became_active":"2023-11-24T08:20:32.278557+0000","last_became_peered":"2023-11-24T08:20:32.278557+0000","last_unstale":"2023-11-24T08:20:33.295476+0000","last_undegraded":"2023-11-24T08:20:33.295476+0000","last_fullsized":"2023-11-24T08:20:33.295476+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:42.464228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287598+0000","last_change":"2023-11-24T08:20:33.276526+0000","last_active":"2023-11-24T08:20:33.287598+0000","last_peered":"2023-11-24T08:20:33.287598+0000","last_clean":"2023-11-24T08:20:33.287598+0000","last_became_active":"2023-11-24T08:20:33.271579+0000","last_became_peered":"2023-11-24T08:20:33.271579+0000","last_unstale":"2023-11-24T08:20:33.287598+0000","last_undegraded":"2023-11-24T08:20:33.287598+0000","last_fullsized":"2023-11-24T08:20:33.287598+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:31:38.672205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.297851+0000","last_change":"2023-11-24T08:20:32.271070+0000","last_active":"2023-11-24T08:20:33.297851+0000","last_peered":"2023-11-24T08:20:33.297851+0000","last_clean":"2023-11-24T08:20:33.297851+0000","last_became_active":"2023-11-24T08:20:32.270561+0000","last_became_peered":"2023-11-24T08:20:32.270561+0000","last_unstale":"2023-11-24T08:20:33.297851+0000","last_undegraded":"2023-11-24T08:20:33.297851+0000","last_fullsized":"2023-11-24T08:20:33.297851+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:19:11.276699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279698+0000","last_change":"2023-11-24T08:20:33.275857+0000","last_active":"2023-11-24T08:20:33.279698+0000","last_peered":"2023-11-24T08:20:33.279698+0000","last_clean":"2023-11-24T08:20:33.279698+0000","last_became_active":"2023-11-24T08:20:33.275493+0000","last_became_peered":"2023-11-24T08:20:33.275493+0000","last_unstale":"2023-11-24T08:20:33.279698+0000","last_undegraded":"2023-11-24T08:20:33.279698+0000","last_fullsized":"2023-11-24T08:20:33.279698+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:15:09.708754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.254663+0000","last_change":"2023-11-24T08:20:32.270086+0000","last_active":"2023-11-24T08:20:33.254663+0000","last_peered":"2023-11-24T08:20:33.254663+0000","last_clean":"2023-11-24T08:20:33.254663+0000","last_became_active":"2023-11-24T08:20:32.269791+0000","last_became_peered":"2023-11-24T08:20:32.269791+0000","last_unstale":"2023-11-24T08:20:33.254663+0000","last_undegraded":"2023-11-24T08:20:33.254663+0000","last_fullsized":"2023-11-24T08:20:33.254663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:20:43.325875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279518+0000","last_change":"2023-11-24T08:20:33.270636+0000","last_active":"2023-11-24T08:20:33.279518+0000","last_peered":"2023-11-24T08:20:33.279518+0000","last_clean":"2023-11-24T08:20:33.279518+0000","last_became_active":"2023-11-24T08:20:33.270352+0000","last_became_peered":"2023-11-24T08:20:33.270352+0000","last_unstale":"2023-11-24T08:20:33.279518+0000","last_undegraded":"2023-11-24T08:20:33.279518+0000","last_fullsized":"2023-11-24T08:20:33.279518+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:32.165518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295483+0000","last_change":"2023-11-24T08:20:32.267544+0000","last_active":"2023-11-24T08:20:33.295483+0000","last_peered":"2023-11-24T08:20:33.295483+0000","last_clean":"2023-11-24T08:20:33.295483+0000","last_became_active":"2023-11-24T08:20:32.267323+0000","last_became_peered":"2023-11-24T08:20:32.267323+0000","last_unstale":"2023-11-24T08:20:33.295483+0000","last_undegraded":"2023-11-24T08:20:33.295483+0000","last_fullsized":"2023-11-24T08:20:33.295483+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:14.836804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.279575+0000","last_change":"2023-11-24T08:20:33.273030+0000","last_active":"2023-11-24T08:20:33.279575+0000","last_peered":"2023-11-24T08:20:33.279575+0000","last_clean":"2023-11-24T08:20:33.279575+0000","last_became_active":"2023-11-24T08:20:33.272730+0000","last_became_peered":"2023-11-24T08:20:33.272730+0000","last_unstale":"2023-11-24T08:20:33.279575+0000","last_undegraded":"2023-11-24T08:20:33.279575+0000","last_fullsized":"2023-11-24T08:20:33.279575+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:31:58.121397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.260039+0000","last_change":"2023-11-24T08:20:32.272923+0000","last_active":"2023-11-24T08:20:33.260039+0000","last_peered":"2023-11-24T08:20:33.260039+0000","last_clean":"2023-11-24T08:20:33.260039+0000","last_became_active":"2023-11-24T08:20:32.269689+0000","last_became_peered":"2023-11-24T08:20:32.269689+0000","last_unstale":"2023-11-24T08:20:33.260039+0000","last_undegraded":"2023-11-24T08:20:33.260039+0000","last_fullsized":"2023-11-24T08:20:33.260039+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:26:29.097591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.266005+0000","last_change":"2023-11-24T08:20:33.260600+0000","last_active":"2023-11-24T08:20:33.266005+0000","last_peered":"2023-11-24T08:20:33.266005+0000","last_clean":"2023-11-24T08:20:33.266005+0000","last_became_active":"2023-11-24T08:20:33.260207+0000","last_became_peered":"2023-11-24T08:20:33.260207+0000","last_unstale":"2023-11-24T08:20:33.266005+0000","last_undegraded":"2023-11-24T08:20:33.266005+0000","last_fullsized":"2023-11-24T08:20:33.266005+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:00.007291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.260102+0000","last_change":"2023-11-24T08:20:32.271411+0000","last_active":"2023-11-24T08:20:33.260102+0000","last_peered":"2023-11-24T08:20:33.260102+0000","last_clean":"2023-11-24T08:20:33.260102+0000","last_became_active":"2023-11-24T08:20:32.271061+0000","last_became_peered":"2023-11-24T08:20:32.271061+0000","last_unstale":"2023-11-24T08:20:33.260102+0000","last_undegraded":"2023-11-24T08:20:33.260102+0000","last_fullsized":"2023-11-24T08:20:33.260102+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:15:46.176942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.272541+0000","last_change":"2023-11-24T08:20:33.270519+0000","last_active":"2023-11-24T08:20:33.272541+0000","last_peered":"2023-11-24T08:20:33.272541+0000","last_clean":"2023-11-24T08:20:33.272541+0000","last_became_active":"2023-11-24T08:20:33.270216+0000","last_became_peered":"2023-11-24T08:20:33.270216+0000","last_unstale":"2023-11-24T08:20:33.272541+0000","last_undegraded":"2023-11-24T08:20:33.272541+0000","last_fullsized":"2023-11-24T08:20:33.272541+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:27:05.626187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.252024+0000","last_change":"2023-11-24T08:20:32.281739+0000","last_active":"2023-11-24T08:20:33.252024+0000","last_peered":"2023-11-24T08:20:33.252024+0000","last_clean":"2023-11-24T08:20:33.252024+0000","last_became_active":"2023-11-24T08:20:32.281454+0000","last_became_peered":"2023-11-24T08:20:32.281454+0000","last_unstale":"2023-11-24T08:20:33.252024+0000","last_undegraded":"2023-11-24T08:20:33.252024+0000","last_fullsized":"2023-11-24T08:20:33.252024+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:26.067331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.274319+0000","last_change":"2023-11-24T08:20:33.266517+0000","last_active":"2023-11-24T08:20:33.274319+0000","last_peered":"2023-11-24T08:20:33.274319+0000","last_clean":"2023-11-24T08:20:33.274319+0000","last_became_active":"2023-11-24T08:20:33.266222+0000","last_became_peered":"2023-11-24T08:20:33.266222+0000","last_unstale":"2023-11-24T08:20:33.274319+0000","last_undegraded":"2023-11-24T08:20:33.274319+0000","last_fullsized":"2023-11-24T08:20:33.274319+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:04:46.596079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"189'2","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.295928+0000","last_change":"2023-11-24T08:20:32.282694+0000","last_active":"2023-11-24T08:20:33.295928+0000","last_peered":"2023-11-24T08:20:33.295928+0000","last_clean":"2023-11-24T08:20:33.295928+0000","last_became_active":"2023-11-24T08:20:32.282433+0000","last_became_peered":"2023-11-24T08:20:32.282433+0000","last_unstale":"2023-11-24T08:20:33.295928+0000","last_undegraded":"2023-11-24T08:20:33.295928+0000","last_fullsized":"2023-11-24T08:20:33.295928+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:24:27.122557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.281300+0000","last_change":"2023-11-24T08:20:33.274461+0000","last_active":"2023-11-24T08:20:33.281300+0000","last_peered":"2023-11-24T08:20:33.281300+0000","last_clean":"2023-11-24T08:20:33.281300+0000","last_became_active":"2023-11-24T08:20:33.273798+0000","last_became_peered":"2023-11-24T08:20:33.273798+0000","last_unstale":"2023-11-24T08:20:33.281300+0000","last_undegraded":"2023-11-24T08:20:33.281300+0000","last_fullsized":"2023-11-24T08:20:33.281300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:18.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261668+0000","last_change":"2023-11-24T08:20:32.281655+0000","last_active":"2023-11-24T08:20:33.261668+0000","last_peered":"2023-11-24T08:20:33.261668+0000","last_clean":"2023-11-24T08:20:33.261668+0000","last_became_active":"2023-11-24T08:20:32.281141+0000","last_became_peered":"2023-11-24T08:20:32.281141+0000","last_unstale":"2023-11-24T08:20:33.261668+0000","last_undegraded":"2023-11-24T08:20:33.261668+0000","last_fullsized":"2023-11-24T08:20:33.261668+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:19:07.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.275384+0000","last_change":"2023-11-24T08:20:33.272189+0000","last_active":"2023-11-24T08:20:33.275384+0000","last_peered":"2023-11-24T08:20:33.275384+0000","last_clean":"2023-11-24T08:20:33.275384+0000","last_became_active":"2023-11-24T08:20:33.271875+0000","last_became_peered":"2023-11-24T08:20:33.271875+0000","last_unstale":"2023-11-24T08:20:33.275384+0000","last_undegraded":"2023-11-24T08:20:33.275384+0000","last_fullsized":"2023-11-24T08:20:33.275384+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:11:15.839239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.256434+0000","last_change":"2023-11-24T08:20:32.278559+0000","last_active":"2023-11-24T08:20:33.256434+0000","last_peered":"2023-11-24T08:20:33.256434+0000","last_clean":"2023-11-24T08:20:33.256434+0000","last_became_active":"2023-11-24T08:20:32.278255+0000","last_became_peered":"2023-11-24T08:20:32.278255+0000","last_unstale":"2023-11-24T08:20:33.256434+0000","last_undegraded":"2023-11-24T08:20:33.256434+0000","last_fullsized":"2023-11-24T08:20:33.256434+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:23:52.576515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.272890+0000","last_change":"2023-11-24T08:20:33.269511+0000","last_active":"2023-11-24T08:20:33.272890+0000","last_peered":"2023-11-24T08:20:33.272890+0000","last_clean":"2023-11-24T08:20:33.272890+0000","last_became_active":"2023-11-24T08:20:33.269217+0000","last_became_peered":"2023-11-24T08:20:33.269217+0000","last_unstale":"2023-11-24T08:20:33.272890+0000","last_undegraded":"2023-11-24T08:20:33.272890+0000","last_fullsized":"2023-11-24T08:20:33.272890+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:57.199781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.254956+0000","last_change":"2023-11-24T08:20:32.279337+0000","last_active":"2023-11-24T08:20:33.254956+0000","last_peered":"2023-11-24T08:20:33.254956+0000","last_clean":"2023-11-24T08:20:33.254956+0000","last_became_active":"2023-11-24T08:20:32.278997+0000","last_became_peered":"2023-11-24T08:20:32.278997+0000","last_unstale":"2023-11-24T08:20:33.254956+0000","last_undegraded":"2023-11-24T08:20:33.254956+0000","last_fullsized":"2023-11-24T08:20:33.254956+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:59.083859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283220+0000","last_change":"2023-11-24T08:20:33.277788+0000","last_active":"2023-11-24T08:20:33.283220+0000","last_peered":"2023-11-24T08:20:33.283220+0000","last_clean":"2023-11-24T08:20:33.283220+0000","last_became_active":"2023-11-24T08:20:33.270945+0000","last_became_peered":"2023-11-24T08:20:33.270945+0000","last_unstale":"2023-11-24T08:20:33.283220+0000","last_undegraded":"2023-11-24T08:20:33.283220+0000","last_fullsized":"2023-11-24T08:20:33.283220+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:39:14.064035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261387+0000","last_change":"2023-11-24T08:20:32.283547+0000","last_active":"2023-11-24T08:20:33.261387+0000","last_peered":"2023-11-24T08:20:33.261387+0000","last_clean":"2023-11-24T08:20:33.261387+0000","last_became_active":"2023-11-24T08:20:32.282341+0000","last_became_peered":"2023-11-24T08:20:32.282341+0000","last_unstale":"2023-11-24T08:20:33.261387+0000","last_undegraded":"2023-11-24T08:20:33.261387+0000","last_fullsized":"2023-11-24T08:20:33.261387+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:06.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.286058+0000","last_change":"2023-11-24T08:20:33.281715+0000","last_active":"2023-11-24T08:20:33.286058+0000","last_peered":"2023-11-24T08:20:33.286058+0000","last_clean":"2023-11-24T08:20:33.286058+0000","last_became_active":"2023-11-24T08:20:33.281391+0000","last_became_peered":"2023-11-24T08:20:33.281391+0000","last_unstale":"2023-11-24T08:20:33.286058+0000","last_undegraded":"2023-11-24T08:20:33.286058+0000","last_fullsized":"2023-11-24T08:20:33.286058+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:29.711235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.256709+0000","last_change":"2023-11-24T08:20:32.281836+0000","last_active":"2023-11-24T08:20:33.256709+0000","last_peered":"2023-11-24T08:20:33.256709+0000","last_clean":"2023-11-24T08:20:33.256709+0000","last_became_active":"2023-11-24T08:20:32.281567+0000","last_became_peered":"2023-11-24T08:20:32.281567+0000","last_unstale":"2023-11-24T08:20:33.256709+0000","last_undegraded":"2023-11-24T08:20:33.256709+0000","last_fullsized":"2023-11-24T08:20:33.256709+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:22.660075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.287716+0000","last_change":"2023-11-24T08:20:33.277816+0000","last_active":"2023-11-24T08:20:33.287716+0000","last_peered":"2023-11-24T08:20:33.287716+0000","last_clean":"2023-11-24T08:20:33.287716+0000","last_became_active":"2023-11-24T08:20:33.277463+0000","last_became_peered":"2023-11-24T08:20:33.277463+0000","last_unstale":"2023-11-24T08:20:33.287716+0000","last_undegraded":"2023-11-24T08:20:33.287716+0000","last_fullsized":"2023-11-24T08:20:33.287716+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:45.094980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.249151+0000","last_change":"2023-11-24T08:20:32.274305+0000","last_active":"2023-11-24T08:20:33.249151+0000","last_peered":"2023-11-24T08:20:33.249151+0000","last_clean":"2023-11-24T08:20:33.249151+0000","last_became_active":"2023-11-24T08:20:32.272767+0000","last_became_peered":"2023-11-24T08:20:32.272767+0000","last_unstale":"2023-11-24T08:20:33.249151+0000","last_undegraded":"2023-11-24T08:20:33.249151+0000","last_fullsized":"2023-11-24T08:20:33.249151+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:26.067331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.271448+0000","last_change":"2023-11-24T08:20:33.270073+0000","last_active":"2023-11-24T08:20:33.271448+0000","last_peered":"2023-11-24T08:20:33.271448+0000","last_clean":"2023-11-24T08:20:33.271448+0000","last_became_active":"2023-11-24T08:20:33.269791+0000","last_became_peered":"2023-11-24T08:20:33.269791+0000","last_unstale":"2023-11-24T08:20:33.271448+0000","last_undegraded":"2023-11-24T08:20:33.271448+0000","last_fullsized":"2023-11-24T08:20:33.271448+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:04.104523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.261532+0000","last_change":"2023-11-24T08:20:32.274909+0000","last_active":"2023-11-24T08:20:33.261532+0000","last_peered":"2023-11-24T08:20:33.261532+0000","last_clean":"2023-11-24T08:20:33.261532+0000","last_became_active":"2023-11-24T08:20:32.274632+0000","last_became_peered":"2023-11-24T08:20:32.274632+0000","last_unstale":"2023-11-24T08:20:33.261532+0000","last_undegraded":"2023-11-24T08:20:33.261532+0000","last_fullsized":"2023-11-24T08:20:33.261532+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:52.699687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.283238+0000","last_change":"2023-11-24T08:20:33.278315+0000","last_active":"2023-11-24T08:20:33.283238+0000","last_peered":"2023-11-24T08:20:33.283238+0000","last_clean":"2023-11-24T08:20:33.283238+0000","last_became_active":"2023-11-24T08:20:33.278051+0000","last_became_peered":"2023-11-24T08:20:33.278051+0000","last_unstale":"2023-11-24T08:20:33.283238+0000","last_undegraded":"2023-11-24T08:20:33.283238+0000","last_fullsized":"2023-11-24T08:20:33.283238+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:32.229472+0000","last_clean_scrub_stamp":"2023-11-24T08:20:32.229472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:58:51.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"189'1","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-11-24T08:20:33.292195+0000","last_change":"2023-11-24T08:20:32.280816+0000","last_active":"2023-11-24T08:20:33.292195+0000","last_peered":"2023-11-24T08:20:33.292195+0000","last_clean":"2023-11-24T08:20:33.292195+0000","last_became_active":"2023-11-24T08:20:32.280455+0000","last_became_peered":"2023-11-24T08:20:32.280455+0000","last_unstale":"2023-11-24T08:20:33.292195+0000","last_undegraded":"2023-11-24T08:20:33.292195+0000","last_fullsized":"2023-11-24T08:20:33.292195+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:20:31.212818+0000","last_clean_scrub_stamp":"2023-11-24T08:20:31.212818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:23:52.576515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":313,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-11-24T08:20:32.270163+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:20:32.270163+0000","last_peered":"2023-11-24T08:20:32.270163+0000","last_clean":"2023-11-24T08:20:32.270163+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:20:32.270163+0000","last_undegraded":"2023-11-24T08:20:32.270163+0000","last_fullsized":"2023-11-24T08:20:32.270163+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42416,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329424,"statfs":{"total":96636764160,"available":96593330176,"internally_reserved":0,"allocated":2985984,"data_stored":1643375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0880000000000001,"15min":1.0660000000000001},"min":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":1.7030000000000001,"5min":1.8029999999999999,"15min":2.7240000000000002},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.1259999999999999,"15min":1.105},"min":{"1min":0.67200000000000004,"5min":0.67100000000000004,"15min":0.52900000000000003},"max":{"1min":1.782,"5min":1.782,"15min":2.7109999999999999},"last":1.417}]},{"osd":1,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1719999999999999,"15min":1.1599999999999999},"min":{"1min":0.83899999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.532,"5min":1.6870000000000001,"15min":3.8519999999999999},"last":0.88},{"interface":"front","average":{"1min":1.274,"5min":1.2330000000000001,"15min":1.2230000000000001},"min":{"1min":0.94299999999999995,"5min":0.71099999999999997,"15min":0.628},"max":{"1min":1.6459999999999999,"5min":2.2400000000000002,"15min":2.8570000000000002},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.1299999999999999},"min":{"1min":0.80800000000000005,"5min":0.67500000000000004,"15min":0.55700000000000005},"max":{"1min":1.6719999999999999,"5min":1.698,"15min":4.7000000000000002},"last":1.135},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.93500000000000005,"5min":0.71799999999999997,"15min":0.60399999999999998},"max":{"1min":1.591,"5min":1.8480000000000001,"15min":4.633},"last":1.3859999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.504},"max":{"1min":1.6240000000000001,"5min":1.99,"15min":3.133},"last":1.548},{"interface":"front","average":{"1min":1.327,"5min":1.373,"15min":1.373},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":2.173,"5min":2.173,"15min":3.9209999999999998},"last":1.252}]},{"osd":4,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3959999999999999,"15min":1.357},"min":{"1min":0.95199999999999996,"5min":0.83699999999999997,"15min":0.54700000000000004},"max":{"1min":1.879,"5min":2.665,"15min":2.851},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.399,"15min":1.3560000000000001},"min":{"1min":1.04,"5min":0.82999999999999996,"15min":0.45400000000000001},"max":{"1min":1.653,"5min":2.3119999999999998,"15min":3.4870000000000001},"last":1.3520000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.4950000000000001,"15min":1.476},"min":{"1min":1.006,"5min":0.64100000000000001,"15min":0.42699999999999999},"max":{"1min":1.762,"5min":2.3199999999999998,"15min":4.0010000000000003},"last":0.998},{"interface":"front","average":{"1min":1.464,"5min":1.5209999999999999,"15min":1.498},"min":{"1min":1.1200000000000001,"5min":0.92700000000000005,"15min":0.622},"max":{"1min":1.98,"5min":2.5990000000000002,"15min":3.4239999999999999},"last":1.288}]},{"osd":6,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6080000000000001,"15min":1.5760000000000001},"min":{"1min":1.319,"5min":1.0649999999999999,"15min":0.60199999999999998},"max":{"1min":1.9810000000000001,"5min":2.702,"15min":2.992},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.544,"15min":1.508},"min":{"1min":1.179,"5min":0.97799999999999998,"15min":0.68700000000000006},"max":{"1min":1.788,"5min":2.6259999999999999,"15min":3.798},"last":1.454}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37668,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334172,"statfs":{"total":96636764160,"available":96598192128,"internally_reserved":0,"allocated":2383872,"data_stored":1051977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0569999999999999,"15min":1.077},"min":{"1min":0.73699999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.516,"5min":1.7929999999999999,"15min":1.95},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.0880000000000001,"15min":1.085},"min":{"1min":0.83599999999999997,"5min":0.63900000000000001,"15min":0.58099999999999996},"max":{"1min":2.008,"5min":2.008,"15min":2.2069999999999999},"last":0.624}]},{"osd":1,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1020000000000001,"15min":1.1559999999999999},"min":{"1min":0.75600000000000001,"5min":0.65200000000000002,"15min":0.61599999999999999},"max":{"1min":1.5669999999999999,"5min":1.835,"15min":2.0230000000000001},"last":1.026},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.236,"15min":1.202},"min":{"1min":0.86899999999999999,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":2.206,"5min":2.206,"15min":2.4039999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.093,"15min":1.1020000000000001},"min":{"1min":0.79900000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7090000000000001,"5min":1.8680000000000001,"15min":1.9330000000000001},"last":1.419},{"interface":"front","average":{"1min":1.216,"5min":1.238,"15min":1.2689999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.629,"5min":1.7849999999999999,"15min":2.3239999999999998},"last":1.1659999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3400000000000001,"15min":1.3440000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.274,"5min":1.2589999999999999,"15min":1.3069999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.907,"5min":1.907,"15min":4.8319999999999999},"last":1.216}]},{"osd":4,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.427,"15min":1.431},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.673,"5min":3.673,"15min":3.673},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.48,"15min":1.492},"min":{"1min":0.88,"5min":0.88,"15min":0.63800000000000001},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8820000000000001},"last":1.1160000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.377,"15min":1.4259999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.5960000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.325},{"interface":"front","average":{"1min":1.587,"5min":1.573,"15min":1.5760000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.83299999999999996},"max":{"1min":3.8029999999999999,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.2569999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.603,"15min":1.601},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.748},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.601,"5min":1.591,"15min":1.599},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.498}]}]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37672,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334168,"statfs":{"total":96636764160,"available":96598188032,"internally_reserved":0,"allocated":2383872,"data_stored":1051977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0209999999999999,"15min":1.0289999999999999},"min":{"1min":0.55700000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.671,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.0880000000000001,"15min":1.109},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6559999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":1.6319999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.149,"15min":1.129},"min":{"1min":0.69999999999999996,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.518,"5min":7.258,"15min":7.258},"last":1.585},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.202,"15min":1.181},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.52,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":1.659}]},{"osd":2,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.7829999999999999,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":1.337},{"interface":"front","average":{"1min":1.081,"5min":1.194,"15min":1.1679999999999999},"min":{"1min":0.72399999999999998,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.411,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.8149999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2450000000000001,"15min":1.179},"min":{"1min":0.70799999999999996,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.754,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3220000000000001,"15min":1.2849999999999999},"min":{"1min":0.85399999999999998,"5min":0.66800000000000004,"15min":0.55200000000000005},"max":{"1min":1.956,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.5289999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.47,"15min":1.369},"min":{"1min":1.0069999999999999,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.905,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.5449999999999999,"15min":1.488},"min":{"1min":1.1819999999999999,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":2.0579999999999998,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.7450000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3979999999999999,"15min":1.355},"min":{"1min":0.94999999999999996,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.611,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":2.0129999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.5229999999999999,"15min":1.4510000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":1.794,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.8959999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5369999999999999,"15min":1.5029999999999999},"min":{"1min":1.117,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":1.7110000000000001,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":2.0939999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.7529999999999999,"15min":1.6419999999999999},"min":{"1min":1.099,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.0070000000000001,"5min":19.663,"15min":19.663},"last":1.78}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41844,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329996,"statfs":{"total":96636764160,"available":96593915904,"internally_reserved":0,"allocated":2400256,"data_stored":1053111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.022,"15min":1.0600000000000001},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.42699999999999999},"max":{"1min":1.5449999999999999,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.022},{"interface":"front","average":{"1min":1.125,"5min":1.048,"15min":1.0800000000000001},"min":{"1min":0.81299999999999994,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.544,"5min":1.927,"15min":5.452},"last":1.47}]},{"osd":1,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1200000000000001,"15min":1.133},"min":{"1min":0.84999999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5009999999999999,"5min":1.7809999999999999,"15min":1.8100000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.119,"15min":1.1559999999999999},"min":{"1min":1.026,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.6839999999999999,"5min":1.7150000000000001,"15min":1.988},"last":1.3720000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0489999999999999,"15min":1.073},"min":{"1min":0.76800000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.482,"5min":1.6519999999999999,"15min":2.1419999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.238,"15min":1.2430000000000001},"min":{"1min":0.754,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.726,"5min":1.827,"15min":2.7229999999999999},"last":1.417}]},{"osd":3,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.341,"15min":1.3380000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.62},"max":{"1min":1.7869999999999999,"5min":1.9750000000000001,"15min":2.9100000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.2789999999999999,"15min":1.278},"min":{"1min":1.177,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":2.3340000000000001},"last":1.2709999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5209999999999999,"15min":1.4930000000000001},"min":{"1min":0.998,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.7549999999999999,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.397,"15min":1.431},"min":{"1min":0.89500000000000002,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":1.718,"5min":2.1469999999999998,"15min":2.8090000000000002},"last":1.1299999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.242,"15min":1.2629999999999999},"min":{"1min":0.94299999999999995,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.7110000000000001,"5min":1.8540000000000001,"15min":3.5899999999999999},"last":1.075},{"interface":"front","average":{"1min":1.524,"5min":1.534,"15min":1.5389999999999999},"min":{"1min":1.1259999999999999,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.8979999999999999,"5min":2.1190000000000002,"15min":3.391},"last":1.321}]},{"osd":7,"last update":"Fri Nov 24 08:20:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.4710000000000001,"15min":1.4950000000000001},"min":{"1min":1.3100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.861,"5min":2.2440000000000002,"15min":3.1230000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.46,"5min":1.4079999999999999,"15min":1.4179999999999999},"min":{"1min":1.1259999999999999,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":1.8100000000000001,"5min":2.3100000000000001,"15min":3.4849999999999999},"last":1.1970000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2367488,"data_stored":1044620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.78400000000000003,"15min":0.85899999999999999},"min":{"1min":0.58299999999999996,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":10.557},"last":0.747},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.90500000000000003,"15min":0.92300000000000004},"min":{"1min":0.66700000000000004,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5349999999999999,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.54900000000000004,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.165,"5min":1.71,"15min":2.101},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.85799999999999998,"15min":0.96899999999999997},"min":{"1min":0.55100000000000005,"5min":0.46200000000000002,"15min":0.379},"max":{"1min":1.452,"5min":1.452,"15min":13.090999999999999},"last":0.86499999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.83599999999999997,"15min":0.90600000000000003},"min":{"1min":0.46400000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.976,"5min":1.976,"15min":3.9489999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.95399999999999996,"15min":1.0580000000000001},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.45200000000000001},"max":{"1min":1.665,"5min":1.665,"15min":13.460000000000001},"last":0.90500000000000003}]},{"osd":4,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.238,"15min":1.2709999999999999},"min":{"1min":0.878,"5min":0.77000000000000002,"15min":0.67800000000000005},"max":{"1min":1.8109999999999999,"5min":1.877,"15min":4.367},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2030000000000001,"15min":1.2170000000000001},"min":{"1min":0.76400000000000001,"5min":0.72399999999999998,"15min":0.57799999999999996},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.3380000000000001},"last":1.4119999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.2150000000000001,"15min":1.298},"min":{"1min":0.79500000000000004,"5min":0.68000000000000005,"15min":0.60499999999999998},"max":{"1min":2.0539999999999998,"5min":2.1230000000000002,"15min":13.231},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.1910000000000001,"15min":1.2470000000000001},"min":{"1min":0.95199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8620000000000001,"5min":1.992,"15min":5.1180000000000003},"last":1.496}]},{"osd":6,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.2629999999999999,"15min":1.355},"min":{"1min":0.76800000000000002,"5min":0.69499999999999995,"15min":0.68899999999999995},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.2190000000000001,"15min":1.2490000000000001},"min":{"1min":0.84599999999999997,"5min":0.70899999999999996,"15min":0.61399999999999999},"max":{"1min":1.8420000000000001,"5min":2.3980000000000001,"15min":3.4409999999999998},"last":1.3160000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:20:08 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.266,"15min":1.323},"min":{"1min":0.91500000000000004,"5min":0.79400000000000004,"15min":0.63200000000000001},"max":{"1min":1.8959999999999999,"5min":2.1030000000000002,"15min":4.5060000000000002},"last":1.54},{"interface":"front","average":{"1min":1.371,"5min":1.252,"15min":1.3049999999999999},"min":{"1min":0.91500000000000004,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":1.9279999999999999,"5min":3.0030000000000001,"15min":4.032},"last":1.4650000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":2379776,"data_stored":1051395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69299999999999995,"15min":0.77200000000000002},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1459999999999999,"5min":1.601,"15min":1.9610000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.78700000000000003,"15min":0.86899999999999999},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.4810000000000001,"5min":1.7390000000000001,"15min":2.5779999999999998},"last":0.97899999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82299999999999995,"15min":0.90400000000000003},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4510000000000001,"5min":1.4850000000000001,"15min":1.9890000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.91000000000000003,"15min":0.95699999999999996},"min":{"1min":0.57799999999999996,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":2.6240000000000001},"last":1.518}]},{"osd":3,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.84999999999999998,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.4530000000000001,"5min":1.6850000000000001,"15min":2.2090000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.95399999999999996,"15min":1.034},"min":{"1min":0.57199999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.948,"5min":1.948,"15min":2.9950000000000001},"last":1.6659999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.276},"min":{"1min":0.70899999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.6220000000000001,"5min":1.782,"15min":2.7160000000000002},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.196,"15min":1.258},"min":{"1min":0.82299999999999995,"5min":0.745,"15min":0.745},"max":{"1min":1.8200000000000001,"5min":1.863,"15min":3.3279999999999998},"last":1.4670000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1679999999999999,"15min":1.2509999999999999},"min":{"1min":0.78200000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.74,"5min":1.857,"15min":2.508},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.2010000000000001,"15min":1.2509999999999999},"min":{"1min":0.80700000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.7689999999999999,"5min":2.0910000000000002,"15min":2.7839999999999998},"last":1.1399999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1719999999999999,"15min":1.208},"min":{"1min":0.90000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.7370000000000001,"5min":2.2879999999999998,"15min":3.2210000000000001},"last":1.165},{"interface":"front","average":{"1min":1.405,"5min":1.242,"15min":1.325},"min":{"1min":1.0349999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.6629999999999998},"last":1.4139999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.26,"15min":1.327},"min":{"1min":0.79600000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.7809999999999999,"5min":2.3999999999999999,"15min":2.859},"last":1.373},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.248,"15min":1.296},"min":{"1min":0.89500000000000002,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":3.3809999999999998},"last":1.321}]}]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41836,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330004,"statfs":{"total":96636764160,"available":96593924096,"internally_reserved":0,"allocated":2383872,"data_stored":1051417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78700000000000003,"15min":0.80300000000000005},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.333,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.88300000000000001,"15min":0.88600000000000001},"min":{"1min":0.46800000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.4039999999999999,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.1739999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.80500000000000005,"15min":0.79800000000000004},"min":{"1min":0.436,"5min":0.374,"15min":0.37},"max":{"1min":1.823,"5min":1.823,"15min":1.861},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.90400000000000003,"15min":0.91800000000000004},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.7110000000000001,"5min":1.7490000000000001,"15min":1.925},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.014,"15min":1},"min":{"1min":0.49199999999999999,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":1.6439999999999999,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.88700000000000001,"15min":0.91200000000000003},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5980000000000001,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":0.73199999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.272,"15min":1.2749999999999999},"min":{"1min":0.85299999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.7989999999999999,"5min":1.984,"15min":2.7959999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.264,"15min":1.254},"min":{"1min":0.84999999999999998,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":3.0409999999999999},"last":1.123}]},{"osd":5,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.236,"15min":1.2529999999999999},"min":{"1min":0.79500000000000004,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2110000000000001,"15min":1.21},"min":{"1min":0.91200000000000003,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":3.286},"last":1.1950000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.244,"15min":1.2709999999999999},"min":{"1min":0.81299999999999994,"5min":0.69299999999999995,"15min":0.52400000000000002},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.365},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2509999999999999,"15min":1.264},"min":{"1min":1.0209999999999999,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Nov 24 08:20:13 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3180000000000001,"15min":1.3140000000000001},"min":{"1min":0.86799999999999999,"5min":0.80200000000000005,"15min":0.59099999999999997},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":3.1760000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.29,"15min":1.3240000000000001},"min":{"1min":0.89300000000000002,"5min":0.624,"15min":0.624},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":3.653},"last":1.036}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38248,"kb_used_data":2908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333592,"statfs":{"total":96636764160,"available":96597598208,"internally_reserved":0,"allocated":2977792,"data_stored":1641853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.745},"min":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.5189999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.79800000000000004,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.3360000000000001,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":1.4490000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.004,"15min":1.0209999999999999},"min":{"1min":0.60799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5660000000000001,"5min":1.893,"15min":1.893},"last":1.089},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.58899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.671,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":0.92900000000000005,"15min":0.98799999999999999},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.44400000000000001},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0670000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.40100000000000002},"max":{"1min":1.7190000000000001,"5min":1.9710000000000001,"15min":2.1789999999999998},"last":0.88400000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.1830000000000001,"15min":1.2350000000000001},"min":{"1min":0.77500000000000002,"5min":0.73799999999999999,"15min":0.72399999999999998},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.387},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.109,"15min":1.1240000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.417}]},{"osd":5,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1419999999999999,"15min":1.1679999999999999},"min":{"1min":0.86899999999999999,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.667,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.135},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.71099999999999997},"max":{"1min":1.532,"5min":1.7889999999999999,"15min":2.7240000000000002},"last":1.01}]},{"osd":6,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.2450000000000001,"15min":1.2749999999999999},"min":{"1min":0.91700000000000004,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.6539999999999999,"5min":2.052,"15min":2.4369999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.2370000000000001,"15min":1.274},"min":{"1min":0.94099999999999995,"5min":0.76400000000000001,"15min":0.70899999999999996},"max":{"1min":1.589,"5min":2.823,"15min":2.823},"last":1.323}]},{"osd":7,"last update":"Fri Nov 24 08:20:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.325,"15min":1.363},"min":{"1min":0.90100000000000002,"5min":0.86799999999999999,"15min":0.81200000000000006},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.512},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.2989999999999999,"15min":1.333},"min":{"1min":0.90200000000000002,"5min":0.747,"15min":0.747},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3929999999999998},"last":1.3560000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:20:38.203 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:20:38.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:20:38.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.472+0000 7f31d00e0640 1 Processor -- start 2023-11-24T08:20:38.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.472+0000 7f31d00e0640 1 -- start start 2023-11-24T08:20:38.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31d00e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31c8072160 0x7f31c815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31d00e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31d00e0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c815ec90 con 0x7f31c806dc40 2023-11-24T08:20:38.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31d00e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c806e0f0 con 0x7f31c8072160 2023-11-24T08:20:38.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31d00e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80cd530 con 0x7f31c80c49f0 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31cde7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31c8072160 0x7f31c815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31cde7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31c8072160 0x7f31c815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56776/0 (socket says 172.21.15.107:56776) 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.473+0000 7f31cde7c640 1 -- 172.21.15.107:0/1762965131 learned_addr learned my addr 172.21.15.107:0/1762965131 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.474+0000 7f31cde7c640 1 -- 172.21.15.107:0/1762965131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 msgr2=0x7f31c806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.474+0000 7f31cd67b640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.474+0000 7f31cde7c640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.474+0000 7f31cde7c640 1 -- 172.21.15.107:0/1762965131 >> v1:172.21.15.120:6789/0 conn(0x7f31c806dc40 legacy=0x7f31c80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.474+0000 7f31cde7c640 1 -- 172.21.15.107:0/1762965131 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c80c4ed0 con 0x7f31c8072160 2023-11-24T08:20:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.475+0000 7f31cde7c640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c815e750 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009e50 tx=0x7f31b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139b6f4bb4e265bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.475+0000 7f31cce7a640 1 -- 172.21.15.107:0/1762965131 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b802ca70 con 0x7f31c8072160 2023-11-24T08:20:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31cce7a640 1 -- 172.21.15.107:0/1762965131 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b802cbd0 con 0x7f31c8072160 2023-11-24T08:20:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31cce7a640 1 -- 172.21.15.107:0/1762965131 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80343f0 con 0x7f31c8072160 2023-11-24T08:20:38.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 -- 172.21.15.107:0/1762965131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 msgr2=0x7f31c815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c815e750 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009e50 tx=0x7f31b802bed0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 -- 172.21.15.107:0/1762965131 shutdown_connections 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 --2- 172.21.15.107:0/1762965131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c815e750 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 -- 172.21.15.107:0/1762965131 >> 172.21.15.107:0/1762965131 conn(0x7f31c80b0c50 msgr2=0x7f31c80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 -- 172.21.15.107:0/1762965131 shutdown_connections 2023-11-24T08:20:38.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.476+0000 7f31d00e0640 1 -- 172.21.15.107:0/1762965131 wait complete. 2023-11-24T08:20:38.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.478+0000 7f31d00e0640 1 Processor -- start 2023-11-24T08:20:38.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.478+0000 7f31d00e0640 1 -- start start 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c806dc40 0x7f31c80b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c80ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c815c560 con 0x7f31c8072160 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c815b950 con 0x7f31c806dc40 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31d00e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c815bc30 con 0x7f31c80c49f0 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31cd67b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31cd67b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56790/0 (socket says 172.21.15.107:56790) 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.479+0000 7f31cd67b640 1 -- 172.21.15.107:0/2779342123 learned_addr learned my addr 172.21.15.107:0/2779342123 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31ce67d640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c80ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cde7c640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c806dc40 0x7f31c80b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 -- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 msgr2=0x7f31c80ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c80ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 -- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c806dc40 msgr2=0x7f31c80b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c806dc40 0x7f31c80b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 -- 172.21.15.107:0/2779342123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8009b30 con 0x7f31c8072160 2023-11-24T08:20:38.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31cd67b640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c7f0 tx=0x7f31bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc004220 con 0x7f31c8072160 2023-11-24T08:20:38.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc004380 con 0x7f31c8072160 2023-11-24T08:20:38.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.480+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc007840 con 0x7f31c8072160 2023-11-24T08:20:38.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.481+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c815bf10 con 0x7f31c8072160 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.481+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c815cb60 con 0x7f31c8072160 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.482+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c80b34b0 con 0x7f31c8072160 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.483+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f31bc007bb0 con 0x7f31c8072160 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.484+0000 7f31c67fc640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 0x7f31b406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.484+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31bc088540 con 0x7f31c8072160 2023-11-24T08:20:38.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.484+0000 7f31cde7c640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 0x7f31b406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.485+0000 7f31cde7c640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 0x7f31b406efb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f31b8035000 tx=0x7f31b8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.489+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31c80b34b0 con 0x7f31c8072160 2023-11-24T08:20:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.749+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f31c80b34b0 con 0x7f31c8072160 2023-11-24T08:20:38.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.752+0000 7f31c67fc640 1 -- 172.21.15.107:0/2779342123 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+147 (secure 0 0 0) 0x7f31c80b34b0 con 0x7f31c8072160 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:20:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.755+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 msgr2=0x7f31b406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.755+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 0x7f31b406efb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f31b8035000 tx=0x7f31b8030000 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.755+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 msgr2=0x7f31c8158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.755+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c7f0 tx=0x7f31bc00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 shutdown_connections 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31b406caf0 0x7f31b406efb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c80c49f0 0x7f31c80ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31c8072160 0x7f31c8158d50 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 --2- 172.21.15.107:0/2779342123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c806dc40 0x7f31c80b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 >> 172.21.15.107:0/2779342123 conn(0x7f31c80b0c50 msgr2=0x7f31c80cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 shutdown_connections 2023-11-24T08:20:38.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.756+0000 7f31d00e0640 1 -- 172.21.15.107:0/2779342123 wait complete. 2023-11-24T08:20:38.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.977+0000 7f61a08bb640 1 Processor -- start 2023-11-24T08:20:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.977+0000 7f61a08bb640 1 -- start start 2023-11-24T08:20:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.979+0000 7f61a08bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f619c15c410 0x7f619c15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.979+0000 7f61a08bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 0x7f619c0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.979+0000 7f61a08bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c15bf80 con 0x7f619c144b30 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.979+0000 7f61a08bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0b27b0 con 0x7f619c15c410 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.979+0000 7f61a08bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0b1560 con 0x7f619c0b1de0 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.980+0000 7f6192ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f619c15c410 0x7f619c15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.980+0000 7f619bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 0x7f619c0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.980+0000 7f6192ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f619c15c410 0x7f619c15f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:51524/0 (socket says 172.21.15.107:51524) 2023-11-24T08:20:38.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.980+0000 7f6192ffd640 1 -- 172.21.15.107:0/2525499777 learned_addr learned my addr 172.21.15.107:0/2525499777 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:38.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.980+0000 7f619bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 0x7f619c0b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:38674/0 (socket says 172.21.15.107:38674) 2023-11-24T08:20:38.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.981+0000 7f6192ffd640 1 -- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 msgr2=0x7f619c0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:38.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.981+0000 7f6192ffd640 1 --2- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 0x7f619c0b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.981+0000 7f6192ffd640 1 -- 172.21.15.107:0/2525499777 >> v1:172.21.15.120:6789/0 conn(0x7f619c144b30 legacy=0x7f619c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.981+0000 7f6192ffd640 1 -- 172.21.15.107:0/2525499777 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0b1280 con 0x7f619c15c410 2023-11-24T08:20:38.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.982+0000 7f6192ffd640 1 --2- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c15c410 0x7f619c15f920 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6184009870 tx=0x7f618402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75edce66c788700 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.982+0000 7f619affd640 1 -- 172.21.15.107:0/2525499777 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184030840 con 0x7f619c15c410 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.982+0000 7f619affd640 1 -- 172.21.15.107:0/2525499777 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61840309a0 con 0x7f619c15c410 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.982+0000 7f619affd640 1 -- 172.21.15.107:0/2525499777 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184030d30 con 0x7f619c15c410 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 -- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c15c410 msgr2=0x7f619c15f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c15c410 0x7f619c15f920 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6184009870 tx=0x7f618402b980 comp rx=0 tx=0).stop 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 -- 172.21.15.107:0/2525499777 shutdown_connections 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b1de0 0x7f619c0b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2525499777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c15c410 0x7f619c15f920 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 -- 172.21.15.107:0/2525499777 >> 172.21.15.107:0/2525499777 conn(0x7f619c0cb830 msgr2=0x7f619c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 -- 172.21.15.107:0/2525499777 shutdown_connections 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.983+0000 7f61a08bb640 1 -- 172.21.15.107:0/2525499777 wait complete. 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.985+0000 7f61a08bb640 1 Processor -- start 2023-11-24T08:20:38.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.985+0000 7f61a08bb640 1 -- start start 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0b1de0 0x7f619c15e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15c410 0x7f619c0caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1f6540 con 0x7f619c144b30 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1f6790 con 0x7f619c0b1de0 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f61a08bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1f6a70 con 0x7f619c15c410 2023-11-24T08:20:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56804/0 (socket says 172.21.15.107:56804) 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 -- 172.21.15.107:0/2940956820 learned_addr learned my addr 172.21.15.107:0/2940956820 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 -- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15c410 msgr2=0x7f619c0caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15c410 0x7f619c0caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 -- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0b1de0 msgr2=0x7f619c15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0b1de0 0x7f619c15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.986+0000 7f6192ffd640 1 -- 172.21.15.107:0/2940956820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61840094d0 con 0x7f619c144b30 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f6192ffd640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f618402b9c0 tx=0x7f61840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618403e070 con 0x7f619c144b30 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184039a10 con 0x7f619c144b30 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c1f6d50 con 0x7f619c144b30 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184039de0 con 0x7f619c144b30 2023-11-24T08:20:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.987+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c0c91b0 con 0x7f619c144b30 2023-11-24T08:20:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.988+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c1f7270 con 0x7f619c144b30 2023-11-24T08:20:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.989+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f618404b400 con 0x7f619c144b30 2023-11-24T08:20:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.990+0000 7f6198ff9640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 0x7f617c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.990+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61840aab50 con 0x7f619c144b30 2023-11-24T08:20:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.990+0000 7f619b7fe640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 0x7f617c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.991+0000 7f619b7fe640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 0x7f617c06efb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f618c0012e0 tx=0x7f618c007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:38.995+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f619c1f7270 con 0x7f619c144b30 2023-11-24T08:20:39.209 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-11-24T08:20:39.207+0000 7f54dacb8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:20:39.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.225+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f619c1f7270 con 0x7f619c144b30 2023-11-24T08:20:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.226+0000 7f6198ff9640 1 -- 172.21.15.107:0/2940956820 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f619c1f7270 con 0x7f619c144b30 2023-11-24T08:20:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:39.228 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:35.761395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:20:39.228 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 117 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 msgr2=0x7f617c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 0x7f617c06efb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f618c0012e0 tx=0x7f618c007d30 comp rx=0 tx=0).stop 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 msgr2=0x7f619c0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f618402b9c0 tx=0x7f61840046c0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 shutdown_connections 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f617c06caf0 0x7f617c06efb0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15c410 0x7f619c0caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c144b30 0x7f619c0caa20 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 --2- 172.21.15.107:0/2940956820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0b1de0 0x7f619c15e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 >> 172.21.15.107:0/2940956820 conn(0x7f619c0cb830 msgr2=0x7f619c15c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 shutdown_connections 2023-11-24T08:20:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.229+0000 7f61a08bb640 1 -- 172.21.15.107:0/2940956820 wait complete. 2023-11-24T08:20:39.650 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:20:32.719679+0000', 'modified': '2023-11-24T08:20:35.761395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11007}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11007': {'gid': 11007, 'name': 'a', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/2859392170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 2859392170}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2859392170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:20:39.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:20:39.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-24T08:20:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.871+0000 7ff2e359e640 1 Processor -- start 2023-11-24T08:20:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.872+0000 7ff2e359e640 1 -- start start 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff2e40b63d0 0x7ff2e40d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2e40d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40cea30 con 0x7ff2e415d490 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40d0450 con 0x7ff2e40b63d0 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40d0bf0 con 0x7ff2e40ce270 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff2e415d490 0x7ff2e40cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57884/0 (socket says 172.21.15.107:57884) 2023-11-24T08:20:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.873+0000 7ff2e259c640 1 -- 172.21.15.107:0/3597202255 learned_addr learned my addr 172.21.15.107:0/3597202255 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232134520 0 0) 0x7ff2e40cea30 con 0x7ff2e415d490 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d0003520 con 0x7ff2e415d490 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/3597202255 >> v2:172.21.15.120:3300/0 conn(0x7ff2e40b63d0 0x7ff2e40d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2e40d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321764699 0 0) 0x7ff2d0003520 con 0x7ff2e415d490 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 msgr2=0x7ff2e40d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2e40d3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 >> v2:172.21.15.120:3300/0 conn(0x7ff2e40b63d0 msgr2=0x7ff2e40d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 --2- 172.21.15.107:0/3597202255 >> v2:172.21.15.120:3300/0 conn(0x7ff2e40b63d0 0x7ff2e40d3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40d2260 con 0x7ff2e415d490 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff2d40019f0 con 0x7ff2e415d490 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 >> v1:172.21.15.107:6789/0 conn(0x7ff2e415d490 legacy=0x7ff2e40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d0004d00 0x7ff2d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40d0450 con 0x7ff2d0003970 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40d0bf0 con 0x7ff2d0004d00 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.874+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d000ba30 con 0x7ff2e40ce270 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 -- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 msgr2=0x7ff2d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40ce270 0x7ff2d00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 -- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d0004d00 msgr2=0x7ff2d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2d0004d00 0x7ff2d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.875+0000 7ff2e259c640 1 -- 172.21.15.107:0/3597202255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0003520 con 0x7ff2d0003970 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.876+0000 7ff2e259c640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2d0003d90 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400a890 tx=0x7ff2d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117487632cf71f2c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.876+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4034630 con 0x7ff2d0003970 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.876+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d402f070 con 0x7ff2d0003970 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.876+0000 7ff2e159a640 1 -- 172.21.15.107:0/3597202255 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d40349f0 con 0x7ff2d0003970 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 -- 172.21.15.107:0/3597202255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 msgr2=0x7ff2d0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 --2- 172.21.15.107:0/3597202255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2d0003d90 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400a890 tx=0x7ff2d00057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/3597202255 reap_dead start 2023-11-24T08:20:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 -- 172.21.15.107:0/3597202255 shutdown_connections 2023-11-24T08:20:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 -- 172.21.15.107:0/3597202255 >> 172.21.15.107:0/3597202255 conn(0x7ff2e4061340 msgr2=0x7ff2e4061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 -- 172.21.15.107:0/3597202255 shutdown_connections 2023-11-24T08:20:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.877+0000 7ff2e359e640 1 -- 172.21.15.107:0/3597202255 wait complete. 2023-11-24T08:20:39.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.879+0000 7ff2e359e640 1 Processor -- start 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.879+0000 7ff2e359e640 1 -- start start 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40c9240 0x7ff2e40cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 0x7ff2e40b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40b8330 con 0x7ff2d0003970 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40cd1d0 con 0x7ff2e40b7940 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40cc130 con 0x7ff2e40c9240 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56832/0 (socket says 172.21.15.107:56832) 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.880+0000 7ff2e259c640 1 -- 172.21.15.107:0/1766758876 learned_addr learned my addr 172.21.15.107:0/1766758876 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:39.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 0x7ff2e40b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40c9240 0x7ff2e40cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 -- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40c9240 msgr2=0x7ff2e40cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40c9240 0x7ff2e40cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 -- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 msgr2=0x7ff2e40b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 0x7ff2e40b7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 -- 172.21.15.107:0/1766758876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40cc410 con 0x7ff2d0003970 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 0x7ff2e40b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e259c640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400a9b0 tx=0x7ff2d4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2daffd640 1 -- 172.21.15.107:0/1766758876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4034900 con 0x7ff2d0003970 2023-11-24T08:20:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2daffd640 1 -- 172.21.15.107:0/1766758876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d4034a60 con 0x7ff2d0003970 2023-11-24T08:20:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2daffd640 1 -- 172.21.15.107:0/1766758876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4034dd0 con 0x7ff2d0003970 2023-11-24T08:20:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.881+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e40cc6f0 con 0x7ff2d0003970 2023-11-24T08:20:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.882+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e40cccb0 con 0x7ff2d0003970 2023-11-24T08:20:39.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.885+0000 7ff2daffd640 1 -- 172.21.15.107:0/1766758876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7ff2d4031030 con 0x7ff2d0003970 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.886+0000 7ff2daffd640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 0x7ff2c406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.886+0000 7ff2daffd640 1 -- 172.21.15.107:0/1766758876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2d40aa970 con 0x7ff2d0003970 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.886+0000 7ff2eb9a2740 1 Processor -- start 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.886+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 0x7ff2c406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.886+0000 7ff2eb9a2740 1 -- start start 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 0x7ff2c406efe0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40b5580 tx=0x7ff2cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3ac7b0 0x55acbd3f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd30b880 con 0x55acbd3a7d70 2023-11-24T08:20:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd31aae0 con 0x55acbd3ad070 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acbd3181d0 con 0x55acbd3ac7b0 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56844/0 (socket says 172.21.15.107:56844) 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.887+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/458329480 learned_addr learned my addr 172.21.15.107:0/458329480 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3ac7b0 msgr2=0x55acbd3f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3ac7b0 0x55acbd3f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 msgr2=0x55acbd3ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/458329480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acbd2db860 con 0x55acbd3a7d70 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f8e0 tx=0x7ff2cc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2f4aca42f4c1d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2c3fff640 1 -- 172.21.15.107:0/458329480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc024070 con 0x55acbd3a7d70 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2c3fff640 1 -- 172.21.15.107:0/458329480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc019070 con 0x55acbd3a7d70 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.888+0000 7ff2c3fff640 1 -- 172.21.15.107:0/458329480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00dda0 con 0x55acbd3a7d70 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/458329480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 msgr2=0x55acbd3ac270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f8e0 tx=0x7ff2cc012000 comp rx=0 tx=0).stop 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/458329480 shutdown_connections 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3ac7b0 0x55acbd3f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3a7d70 0x55acbd3ac270 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/458329480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/458329480 >> 172.21.15.107:0/458329480 conn(0x55acbd36ec70 msgr2=0x55acbd3f2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/458329480 shutdown_connections 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.889+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/458329480 wait complete. 2023-11-24T08:20:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.890+0000 7ff2eb9a2740 1 -- start start 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.890+0000 7ff2eb9a2740 1 Processor -- start 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd3d5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd313ac0 con 0x55acbd3ad070 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd2b02f0 con 0x55acbd3ac7b0 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acbd2cdc70 con 0x55acbd3a7d70 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2eb9a2740 10 client.0 _ll_register_callbacks cb 0x55acbd2d23b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2e2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2e2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56846/0 (socket says 172.21.15.107:56846) 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.891+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/1398606110 learned_addr learned my addr 172.21.15.107:0/1398606110 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 msgr2=0x55acbd3ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 msgr2=0x55acbd3d5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd3d5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/1398606110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2cc00e040 con 0x55acbd3ad070 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40b5dc0 tx=0x7ff2dc00e540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50778da27044a7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00c070 con 0x55acbd3ad070 2023-11-24T08:20:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2dc00eba0 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.892+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acbd30caa0 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2dc00ed60 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 10 client.11124 fetch_fsmap learned FSMap version 117 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55acbd312140 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55acbd30caa0 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 117) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff2dc004020 con 0x55acbd3ad070 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 10 client.11124 fetch_fsmap finished waiting for FSMap version 117 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 10 client.11124 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 20 client.11124 populate_metadata read hostname 'smithi107' 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3f1420 0x55acbd406000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 4 client.11124 mds_command: new command op to 11007 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:20:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 --> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acbd30c7c0 con 0x55acbd3f1420 2023-11-24T08:20:39.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.893+0000 7ff2e259c640 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3f1420 0x55acbd406000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.894+0000 7ff2c17fa640 10 client.11124 ms_handle_connect on v2:172.21.15.107:6836/2859392170 2023-11-24T08:20:39.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.894+0000 7ff2e259c640 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3f1420 0x55acbd406000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f3bdb036861e444 server_cookie=b655e5eca97f85bb in_seq=0 out_seq=0 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.894+0000 7ff2c17fa640 1 -- 172.21.15.107:0/1398606110 <== mds.0 v2:172.21.15.107:6836/2859392170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55acbd30c7c0 con 0x55acbd3f1420 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.894+0000 7ff2c17fa640 10 client.11124 handle_command_reply: tid=0 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 client.11124 shutdown 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 msgr2=0x55acbd3d2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40b5dc0 tx=0x7ff2dc00e540 comp rx=0 tx=0).stop 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 shutdown_connections 2023-11-24T08:20:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3f1420 0x55acbd406000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3d2f70 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd3d5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1398606110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3ef1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2c17fa640 0 client.11124 ms_handle_reset on v2:172.21.15.107:6836/2859392170 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 >> 172.21.15.107:0/1398606110 conn(0x55acbd36ec70 msgr2=0x55acbd3a6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 shutdown_connections 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1398606110 wait complete. 2023-11-24T08:20:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.895+0000 7ff2eb9a2740 20 client.11124 trim_cache size 0 max 16384 2023-11-24T08:20:39.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.912+0000 7ff2eb9a2740 1 Processor -- start 2023-11-24T08:20:39.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.912+0000 7ff2eb9a2740 1 -- start start 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3eff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3a8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd076f20 con 0x55acbd3ac7b0 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd2c7e30 con 0x55acbd3ad070 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acbd3a92d0 con 0x55acbd3a7d70 2023-11-24T08:20:39.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56850/0 (socket says 172.21.15.107:56850) 2023-11-24T08:20:39.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.913+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/1700844800 learned_addr learned my addr 172.21.15.107:0/1700844800 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 msgr2=0x55acbd3eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 msgr2=0x55acbd3a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 -- 172.21.15.107:0/1700844800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acbd2afb80 con 0x55acbd3ac7b0 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2e1d9b640 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc014830 tx=0x7ff2cc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97331110d6bb3ec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2d8ff9640 1 -- 172.21.15.107:0/1700844800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc024070 con 0x55acbd3ac7b0 2023-11-24T08:20:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2d8ff9640 1 -- 172.21.15.107:0/1700844800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc025040 con 0x55acbd3ac7b0 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.914+0000 7ff2d8ff9640 1 -- 172.21.15.107:0/1700844800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc014c20 con 0x55acbd3ac7b0 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1700844800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 msgr2=0x55acbd3a8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc014830 tx=0x7ff2cc016000 comp rx=0 tx=0).stop 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1700844800 shutdown_connections 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ad070 0x55acbd3a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ac7b0 0x55acbd3a8200 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/1700844800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1700844800 >> 172.21.15.107:0/1700844800 conn(0x55acbd36ec70 msgr2=0x55acbd3d7410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1700844800 shutdown_connections 2023-11-24T08:20:39.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.915+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/1700844800 wait complete. 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.916+0000 7ff2eb9a2740 1 -- start start 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.916+0000 7ff2eb9a2740 1 Processor -- start 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3d72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd4658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd4674d0 con 0x55acbd3ad070 2023-11-24T08:20:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acbd3d78b0 con 0x55acbd3ac7b0 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acbd3a8150 con 0x55acbd3a7d70 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2eb9a2740 10 client.0 _ll_register_callbacks cb 0x7ff2dc006e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2e2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2e2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56860/0 (socket says 172.21.15.107:56860) 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.917+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/797457132 learned_addr learned my addr 172.21.15.107:0/797457132 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 msgr2=0x55acbd3d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3d72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 msgr2=0x55acbd4658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd4658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 -- 172.21.15.107:0/797457132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acbd2db860 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2e2d9d640 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e580 tx=0x7ff2dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a88335f1dad91e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc009a90 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2dc02eae0 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2cc019070 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2dc02ecc0 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2eb9a2740 10 client.11130 fetch_fsmap learned FSMap version 117 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.918+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55acbd2d5170 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc019070 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 117) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff2dc003610 con 0x55acbd3ad070 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 10 client.11130 fetch_fsmap finished waiting for FSMap version 117 2023-11-24T08:20:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 10 client.11130 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:20:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 20 client.11130 populate_metadata read hostname 'smithi107' 2023-11-24T08:20:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3daeb0 0x55acbd466190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 4 client.11130 mds_command: new command op to 11007 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:20:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 --> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55acbd0146c0 con 0x55acbd3daeb0 2023-11-24T08:20:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.919+0000 7ff2e259c640 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3daeb0 0x55acbd466190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2e259c640 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3daeb0 0x55acbd466190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=825227ba3615fafc server_cookie=e7437fcd63ecd942 in_seq=0 out_seq=0 2023-11-24T08:20:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2c17fa640 10 client.11130 ms_handle_connect on v2:172.21.15.107:6836/2859392170 2023-11-24T08:20:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2c17fa640 1 -- 172.21.15.107:0/797457132 <== mds.0 v2:172.21.15.107:6836/2859392170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55acbd0146c0 con 0x55acbd3daeb0 2023-11-24T08:20:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2c17fa640 10 client.11130 handle_command_reply: tid=0 2023-11-24T08:20:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 client.11130 shutdown 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 msgr2=0x55acbd3eff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e580 tx=0x7ff2dc004730 comp rx=0 tx=0).stop 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 shutdown_connections 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:6836/2859392170,v1:172.21.15.107:6837/2859392170] conn(0x55acbd3daeb0 0x55acbd466190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acbd3ad070 0x55acbd3eff40 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2c17fa640 0 client.11130 ms_handle_reset on v2:172.21.15.107:6836/2859392170 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acbd3ac7b0 0x55acbd4658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 --2- 172.21.15.107:0/797457132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55acbd3a7d70 0x55acbd3d72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 >> 172.21.15.107:0/797457132 conn(0x55acbd36ec70 msgr2=0x55acbd40cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 shutdown_connections 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 1 -- 172.21.15.107:0/797457132 wait complete. 2023-11-24T08:20:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.920+0000 7ff2eb9a2740 20 client.11130 trim_cache size 0 max 16384 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 11007, 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6603920710000004, 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 117, 2023-11-24T08:20:39.927 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 190, 2023-11-24T08:20:39.928 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 190, 2023-11-24T08:20:39.928 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.061253793000001 2023-11-24T08:20:39.928 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:39.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 msgr2=0x7ff2c406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 0x7ff2c406efe0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40b5580 tx=0x7ff2cc008040 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 msgr2=0x7ff2e415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400a9b0 tx=0x7ff2d4001d80 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 shutdown_connections 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2c406cb20 0x7ff2c406efe0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2e40b7940 0x7ff2e40b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2e40c9240 0x7ff2e40cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 --2- 172.21.15.107:0/1766758876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0003970 0x7ff2e415dca0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 >> 172.21.15.107:0/1766758876 conn(0x7ff2e4061340 msgr2=0x7ff2e40d1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 shutdown_connections 2023-11-24T08:20:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:39.921+0000 7ff2e359e640 1 -- 172.21.15.107:0/1766758876 wait complete. 2023-11-24T08:20:39.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:20:40.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.139+0000 7f8882b0a640 1 Processor -- start 2023-11-24T08:20:40.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.139+0000 7f8882b0a640 1 -- start start 2023-11-24T08:20:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.140+0000 7f8882b0a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f887c15c7f0 0x7f887c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.140+0000 7f8882b0a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c0cece0 con 0x7f887c0cf450 2023-11-24T08:20:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f8882b0a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c06d040 con 0x7f887c0ce6c0 2023-11-24T08:20:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f8882b0a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c0ceaf0 con 0x7f887c15c7f0 2023-11-24T08:20:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f887bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f887c0cf450 0x7f887c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57896/0 (socket says 172.21.15.107:57896) 2023-11-24T08:20:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f887bfff640 1 -- 172.21.15.107:0/1518016118 learned_addr learned my addr 172.21.15.107:0/1518016118 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f88808a6640 1 --2- 172.21.15.107:0/1518016118 >> v2:172.21.15.120:3300/0 conn(0x7f887c15c7f0 0x7f887c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251770282 0 0) 0x7f887c0cece0 con 0x7f887c0cf450 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.141+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8864003520 con 0x7f887c0cf450 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447500446 0 0) 0x7f8864003520 con 0x7f887c0cf450 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 >> v2:172.21.15.120:3300/0 conn(0x7f887c15c7f0 msgr2=0x7f887c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 --2- 172.21.15.107:0/1518016118 >> v2:172.21.15.120:3300/0 conn(0x7f887c15c7f0 0x7f887c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 >> v1:172.21.15.120:6789/0 conn(0x7f887c0ce6c0 legacy=0x7f887c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c06d320 con 0x7f887c0cf450 2023-11-24T08:20:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8870001670 con 0x7f887c0cf450 2023-11-24T08:20:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 >> v1:172.21.15.107:6789/0 conn(0x7f887c0cf450 legacy=0x7f887c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640039c0 0x7f8864003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 0x7f8864004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8864005180 0x7f886400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886400c050 con 0x7f88640047c0 2023-11-24T08:20:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886400c1c0 con 0x7f8864005180 2023-11-24T08:20:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886400c380 con 0x7f88640039c0 2023-11-24T08:20:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.142+0000 7f887bfff640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 0x7f8864004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f88808a6640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640039c0 0x7f8864003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f887bfff640 1 -- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640039c0 msgr2=0x7f8864003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f88810a7640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8864005180 0x7f886400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f887bfff640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640039c0 0x7f8864003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f887bfff640 1 -- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8864005180 msgr2=0x7f886400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f887bfff640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8864005180 0x7f886400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f887bfff640 1 -- 172.21.15.107:0/1518016118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864003520 con 0x7f88640047c0 2023-11-24T08:20:40.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.143+0000 7f88810a7640 1 -- 172.21.15.107:0/1518016118 reap_dead start 2023-11-24T08:20:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.144+0000 7f887bfff640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 0x7f8864004c40 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f887000c0e0 tx=0x7f88640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c31dbe4e9f0803d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.144+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870005ad0 con 0x7f88640047c0 2023-11-24T08:20:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.144+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8870005c30 con 0x7f88640047c0 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.144+0000 7f887b7fe640 1 -- 172.21.15.107:0/1518016118 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887002ca50 con 0x7f88640047c0 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 -- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 msgr2=0x7f8864004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 0x7f8864004c40 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f887000c0e0 tx=0x7f88640057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 -- 172.21.15.107:0/1518016118 shutdown_connections 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 --2- 172.21.15.107:0/1518016118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88640047c0 0x7f8864004c40 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 -- 172.21.15.107:0/1518016118 >> 172.21.15.107:0/1518016118 conn(0x7f887c06a920 msgr2=0x7f887c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 -- 172.21.15.107:0/1518016118 shutdown_connections 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.145+0000 7f8882b0a640 1 -- 172.21.15.107:0/1518016118 wait complete. 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.147+0000 7f8882b0a640 1 Processor -- start 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.147+0000 7f8882b0a640 1 -- start start 2023-11-24T08:20:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640047c0 0x7f887c0b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887c156d30 0x7f887c0ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1571e0 con 0x7f887c1561a0 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0ac8c0 con 0x7f887c156d30 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f8882b0a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0b3340 con 0x7f88640047c0 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56872/0 (socket says 172.21.15.107:56872) 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 -- 172.21.15.107:0/3542073117 learned_addr learned my addr 172.21.15.107:0/3542073117 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f88810a7640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887c156d30 0x7f887c0ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f88808a6640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640047c0 0x7f887c0b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 -- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640047c0 msgr2=0x7f887c0b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640047c0 0x7f887c0b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 -- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887c156d30 msgr2=0x7f887c0ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887c156d30 0x7f887c0ad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.148+0000 7f887bfff640 1 -- 172.21.15.107:0/3542073117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0b3620 con 0x7f887c1561a0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f887bfff640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f8870000c00 tx=0x7f887002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887003c040 con 0x7f887c1561a0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887002cd00 con 0x7f887c1561a0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88700072b0 con 0x7f887c1561a0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c0b3900 con 0x7f887c1561a0 2023-11-24T08:20:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.149+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c0b3030 con 0x7f887c1561a0 2023-11-24T08:20:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.150+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c067470 con 0x7f887c1561a0 2023-11-24T08:20:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.151+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f8870032070 con 0x7f887c1561a0 2023-11-24T08:20:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.152+0000 7f88797fa640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 0x7f886806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.152+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88700a9ba0 con 0x7f887c1561a0 2023-11-24T08:20:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.152+0000 7f88808a6640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 0x7f886806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.153+0000 7f88808a6640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 0x7f886806efb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cc130 tx=0x7f886c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.157+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88700766b0 con 0x7f887c1561a0 2023-11-24T08:20:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.389+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f887c0cd730 con 0x7f887c1561a0 2023-11-24T08:20:40.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.390+0000 7f88797fa640 1 -- 172.21.15.107:0/3542073117 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8870079cd0 con 0x7f887c1561a0 2023-11-24T08:20:40.392 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:20:40.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 msgr2=0x7f886806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 0x7f886806efb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cc130 tx=0x7f886c00b040 comp rx=0 tx=0).stop 2023-11-24T08:20:40.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 msgr2=0x7f887c1567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:40.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f8870000c00 tx=0x7f887002e070 comp rx=0 tx=0).stop 2023-11-24T08:20:40.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 shutdown_connections 2023-11-24T08:20:40.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f886806caf0 0x7f886806efb0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887c156d30 0x7f887c0ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887c1561a0 0x7f887c1567f0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 --2- 172.21.15.107:0/3542073117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88640047c0 0x7f887c0b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:40.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 >> 172.21.15.107:0/3542073117 conn(0x7f887c06a920 msgr2=0x7f887c06da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:40.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 shutdown_connections 2023-11-24T08:20:40.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:40.393+0000 7f8882b0a640 1 -- 172.21.15.107:0/3542073117 wait complete. 2023-11-24T08:20:40.413 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:20:40.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:20:40.414 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:20:40.414 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:20:40.414 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:20:40.414 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:20:40.414 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:20:40.415 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:20:40.415 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:20:40.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.415 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:20:40.470 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:20:40.470 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:20:40.471 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:20:40.471 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.471 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:20:40.471 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:20:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:20:40.473 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:40.473 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:40.473 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:20:40.473 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:40.473 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:20:40.474 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41430sec preferred_lft 41430sec 2023-11-24T08:20:40.474 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:20:40.474 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.474 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:20:40.474 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:20:40.475 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:40.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.477 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:20:40.567 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:20:40.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.571 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:20:40.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:20:40.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.809 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:20:40.859 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:20:40.859 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:20:40.860 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:20:40.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:40.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:20:40.955 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:20:40.955 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:20:41.012 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:20:41.012 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:20:41.013 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:20:41.063 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:20:41.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:41.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:41.202 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:41.202 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:20:41.237 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:41.237 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:20:41.237 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:20:41.237 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:20:41.238 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:20:41.238 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:20:41.239 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:20:41.239 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:20:41.239 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:20:41.239 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:20:41.242 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:20:41.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:41.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:20:41.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:41.324 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:20:41.390 INFO:teuthology.orchestra.run.smithi107.stdout:client11139 2023-11-24T08:20:41.392 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:41.393 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:41.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:20:41.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.709+0000 7f48ddc2d640 1 Processor -- start 2023-11-24T08:20:41.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.710+0000 7f48ddc2d640 1 -- start start 2023-11-24T08:20:41.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.711+0000 7f48ddc2d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.711+0000 7f48ddc2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.711+0000 7f48ddc2d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80cece0 con 0x7f48d815c7f0 2023-11-24T08:20:41.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.711+0000 7f48ddc2d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d80cfe40 con 0x7f48d80cf450 2023-11-24T08:20:41.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.711+0000 7f48ddc2d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8156010 con 0x7f48d80cdfb0 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.712+0000 7f48cffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.712+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.712+0000 7f48cffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:51620/0 (socket says 172.21.15.107:51620) 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.712+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:38752/0 (socket says 172.21.15.107:38752) 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.712+0000 7f48cffff640 1 -- 172.21.15.107:0/2185385038 learned_addr learned my addr 172.21.15.107:0/2185385038 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.713+0000 7f48d7fff640 1 -- 172.21.15.107:0/2185385038 >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 msgr2=0x7f48d80cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:41.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.713+0000 7f48d7fff640 1 --2- 172.21.15.107:0/2185385038 >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.713+0000 7f48d7fff640 1 -- 172.21.15.107:0/2185385038 >> v1:172.21.15.120:6789/0 conn(0x7f48d815c7f0 legacy=0x7f48d806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.713+0000 7f48d7fff640 1 -- 172.21.15.107:0/2185385038 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d806d040 con 0x7f48d80cdfb0 2023-11-24T08:20:41.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.713+0000 7f48cffff640 1 --2- 172.21.15.107:0/2185385038 >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:20:41.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.714+0000 7f48d7fff640 1 --2- 172.21.15.107:0/2185385038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009d90 tx=0x7f48d002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be212a39ee35b9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:41.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.715+0000 7f48d6ffd640 1 -- 172.21.15.107:0/2185385038 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d002cc60 con 0x7f48d80cdfb0 2023-11-24T08:20:41.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.715+0000 7f48d6ffd640 1 -- 172.21.15.107:0/2185385038 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d002cdc0 con 0x7f48d80cdfb0 2023-11-24T08:20:41.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.715+0000 7f48d6ffd640 1 -- 172.21.15.107:0/2185385038 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0034660 con 0x7f48d80cdfb0 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 -- 172.21.15.107:0/2185385038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 msgr2=0x7f48d80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/2185385038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009d90 tx=0x7f48d002beb0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 -- 172.21.15.107:0/2185385038 shutdown_connections 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/2185385038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cdfb0 0x7f48d80ce420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/2185385038 >> v2:172.21.15.120:3300/0 conn(0x7f48d80cf450 0x7f48d80cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 -- 172.21.15.107:0/2185385038 >> 172.21.15.107:0/2185385038 conn(0x7f48d806a920 msgr2=0x7f48d806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 -- 172.21.15.107:0/2185385038 shutdown_connections 2023-11-24T08:20:41.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.716+0000 7f48ddc2d640 1 -- 172.21.15.107:0/2185385038 wait complete. 2023-11-24T08:20:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.718+0000 7f48ddc2d640 1 Processor -- start 2023-11-24T08:20:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.718+0000 7f48ddc2d640 1 -- start start 2023-11-24T08:20:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cf450 0x7f48d8158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 0x7f48d8071930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8151b50 con 0x7f48d80cdfb0 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8158ff0 con 0x7f48d815c7f0 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.719+0000 7f48ddc2d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8156310 con 0x7f48d80cf450 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56876/0 (socket says 172.21.15.107:56876) 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 -- 172.21.15.107:0/482245383 learned_addr learned my addr 172.21.15.107:0/482245383 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48cffff640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cf450 0x7f48d8158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d7fff640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 0x7f48d8071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 -- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cf450 msgr2=0x7f48d8158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cf450 0x7f48d8158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 -- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 msgr2=0x7f48d8071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 0x7f48d8071930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d77fe640 1 -- 172.21.15.107:0/482245383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c00094d0 con 0x7f48d80cdfb0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.720+0000 7f48d7fff640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 0x7f48d8071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48d77fe640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f48c800c430 tx=0x7f48c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c80052d0 con 0x7f48d80cdfb0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c8005430 con 0x7f48d80cdfb0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d0009a70 con 0x7f48d80cdfb0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8003830 con 0x7f48d80cdfb0 2023-11-24T08:20:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.721+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d80ac680 con 0x7f48d80cdfb0 2023-11-24T08:20:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.722+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d8067470 con 0x7f48d80cdfb0 2023-11-24T08:20:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.723+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f48c8003990 con 0x7f48d80cdfb0 2023-11-24T08:20:41.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.724+0000 7f48d4ff9640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 0x7f48b806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:41.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.724+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48c8087b40 con 0x7f48d80cdfb0 2023-11-24T08:20:41.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.724+0000 7f48cffff640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 0x7f48b806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:41.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.725+0000 7f48cffff640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 0x7f48b806efb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f48d806cdc0 tx=0x7f48c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:41.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.730+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48c8054650 con 0x7f48d80cdfb0 2023-11-24T08:20:41.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.958+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f48d80cd730 con 0x7f48d80cdfb0 2023-11-24T08:20:41.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.959+0000 7f48d4ff9640 1 -- 172.21.15.107:0/482245383 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) v1 ==== 53+0+73 (secure 0 0 0) 0x7f48c8017070 con 0x7f48d80cdfb0 2023-11-24T08:20:41.961 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.961+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 msgr2=0x7f48b806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.961+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 0x7f48b806efb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f48d806cdc0 tx=0x7f48c0008040 comp rx=0 tx=0).stop 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.961+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 msgr2=0x7f48d8151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.961+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f48c800c430 tx=0x7f48c800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 shutdown_connections 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b806caf0 0x7f48b806efb0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d815c7f0 0x7f48d8071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d80cf450 0x7f48d8158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 --2- 172.21.15.107:0/482245383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d80cdfb0 0x7f48d8151610 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 >> 172.21.15.107:0/482245383 conn(0x7f48d806a920 msgr2=0x7f48d80aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 shutdown_connections 2023-11-24T08:20:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:41.962+0000 7f48ddc2d640 1 -- 172.21.15.107:0/482245383 wait complete. 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:20:41.982 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:20:41.983 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:20:41.983 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:20:41.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:41.983 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:42.043 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41427sec preferred_lft 41427sec 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:20:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:20:42.045 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:20:42.046 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:20:42.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.047 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:20:42.143 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:20:42.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.148 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:20:42.205 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:20:42.205 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:20:42.205 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:20:42.206 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:20:42.206 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:20:42.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.206 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:20:42.207 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:20:42.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:20:42.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:20:42.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:20:42.207 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:20:42.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.372 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:20:42.423 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:20:42.423 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:20:42.424 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:20:42.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:20:42.511 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:20:42.511 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:20:42.563 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:20:42.563 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:20:42.563 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:20:42.617 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:20:42.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.618 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:42.760 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:42.760 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:20:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:20:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:20:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:20:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:20:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:20:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:20:42.797 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:20:42.797 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:20:42.797 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:20:42.797 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:20:42.800 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:20:42.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.800 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:20:42.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:42.876 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:20:42.931 INFO:teuthology.orchestra.run.smithi120.stdout:client6449 2023-11-24T08:20:42.931 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:20:42.931 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:42.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:20:43.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.211+0000 7f65d517c640 1 Processor -- start 2023-11-24T08:20:43.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.211+0000 7f65d517c640 1 -- start start 2023-11-24T08:20:43.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.212+0000 7f65d517c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f65d00d0a50 0x7f65d00d30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.212+0000 7f65d517c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00cdf30 con 0x7f65d015d4f0 2023-11-24T08:20:43.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.212+0000 7f65d517c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00d3b50 con 0x7f65d00d3630 2023-11-24T08:20:43.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.212+0000 7f65d517c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00d3e30 con 0x7f65d00d0a50 2023-11-24T08:20:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.213+0000 7f65ced9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f65d015d4f0 0x7f65d00cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57906/0 (socket says 172.21.15.107:57906) 2023-11-24T08:20:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.213+0000 7f65ced9d640 1 -- 172.21.15.107:0/1950708270 learned_addr learned my addr 172.21.15.107:0/1950708270 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:43.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.213+0000 7f65ce59c640 1 --2- 172.21.15.107:0/1950708270 >> v2:172.21.15.120:3300/0 conn(0x7f65d00d0a50 0x7f65d00d30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.213+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486106716 0 0) 0x7f65d00cdf30 con 0x7f65d015d4f0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.213+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc003520 con 0x7f65d015d4f0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108697743 0 0) 0x7f65bc003520 con 0x7f65d015d4f0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 >> v2:172.21.15.120:3300/0 conn(0x7f65d00d0a50 msgr2=0x7f65d00d30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 --2- 172.21.15.107:0/1950708270 >> v2:172.21.15.120:3300/0 conn(0x7f65d00d0a50 0x7f65d00d30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 >> v1:172.21.15.120:6789/0 conn(0x7f65d00d3630 legacy=0x7f65d00d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00ce1d0 con 0x7f65d015d4f0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f65c00019f0 con 0x7f65d015d4f0 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 >> v1:172.21.15.107:6789/0 conn(0x7f65d015d4f0 legacy=0x7f65d00cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65bc0039c0 0x7f65bc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc005180 0x7f65bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00d3e30 con 0x7f65bc0047c0 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc00c080 con 0x7f65bc005180 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc00c240 con 0x7f65bc0039c0 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 -- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65bc0039c0 msgr2=0x7f65bc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65bc0039c0 0x7f65bc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 -- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc005180 msgr2=0x7f65bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc005180 0x7f65bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65ce59c640 1 -- 172.21.15.107:0/1950708270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc003520 con 0x7f65bc0047c0 2023-11-24T08:20:43.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.214+0000 7f65cf59e640 1 -- 172.21.15.107:0/1950708270 reap_dead start 2023-11-24T08:20:43.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.215+0000 7f65ce59c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009ac0 tx=0x7f65bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39548b52d5f5ee6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b802c6d0 con 0x7f65bc0047c0 2023-11-24T08:20:43.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b802c830 con 0x7f65bc0047c0 2023-11-24T08:20:43.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65cdd9b640 1 -- 172.21.15.107:0/1950708270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b802cbc0 con 0x7f65bc0047c0 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 -- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 msgr2=0x7f65bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009ac0 tx=0x7f65bc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 -- 172.21.15.107:0/1950708270 shutdown_connections 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 --2- 172.21.15.107:0/1950708270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 -- 172.21.15.107:0/1950708270 >> 172.21.15.107:0/1950708270 conn(0x7f65d0061950 msgr2=0x7f65d008af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 -- 172.21.15.107:0/1950708270 shutdown_connections 2023-11-24T08:20:43.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.216+0000 7f65d517c640 1 -- 172.21.15.107:0/1950708270 wait complete. 2023-11-24T08:20:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.218+0000 7f65d517c640 1 Processor -- start 2023-11-24T08:20:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.218+0000 7f65d517c640 1 -- start start 2023-11-24T08:20:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc0047c0 0x7f65d015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65d015f480 0x7f65d01e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00cf3c0 con 0x7f65d01e7630 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00cb670 con 0x7f65bc0047c0 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.219+0000 7f65d517c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d0160220 con 0x7f65d015f480 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56888/0 (socket says 172.21.15.107:56888) 2023-11-24T08:20:43.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 -- 172.21.15.107:0/2017426517 learned_addr learned my addr 172.21.15.107:0/2017426517 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 -- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65d015f480 msgr2=0x7f65d01e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65d015f480 0x7f65d01e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 -- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc0047c0 msgr2=0x7f65d015a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc0047c0 0x7f65d015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 -- 172.21.15.107:0/2017426517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d015aca0 con 0x7f65d01e7630 2023-11-24T08:20:43.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65cf59e640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f65d00ca480 tx=0x7f65c800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80072d0 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.220+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8007450 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.221+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80057c0 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.221+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d015f980 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.221+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d015ff40 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.222+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d00d02d0 con 0x7f65d01e7630 2023-11-24T08:20:43.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.223+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f65c8005920 con 0x7f65d01e7630 2023-11-24T08:20:43.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.224+0000 7f65c77fe640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 0x7f65b006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.224+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65c8087820 con 0x7f65d01e7630 2023-11-24T08:20:43.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.224+0000 7f65ced9d640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 0x7f65b006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.225+0000 7f65ced9d640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 0x7f65b006efe0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f65c0007fd0 tx=0x7f65c0001980 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.228+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65c8054330 con 0x7f65d01e7630 2023-11-24T08:20:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.456+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f65d00cb7b0 con 0x7f65d01e7630 2023-11-24T08:20:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.510+0000 7f65c77fe640 1 -- 172.21.15.107:0/2017426517 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v118) v1 ==== 136+0+0 (secure 0 0 0) 0x7f65d00cb7b0 con 0x7f65d01e7630 2023-11-24T08:20:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 msgr2=0x7f65b006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 0x7f65b006efe0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f65c0007fd0 tx=0x7f65c0001980 comp rx=0 tx=0).stop 2023-11-24T08:20:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 msgr2=0x7f65d01e7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f65d00ca480 tx=0x7f65c800c820 comp rx=0 tx=0).stop 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 shutdown_connections 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f65b006cb20 0x7f65b006efe0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65d01e7630 0x7f65d01e7ab0 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65d015f480 0x7f65d01e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 --2- 172.21.15.107:0/2017426517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65bc0047c0 0x7f65d015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.513+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 >> 172.21.15.107:0/2017426517 conn(0x7f65d0061950 msgr2=0x7f65d00b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.514+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 shutdown_connections 2023-11-24T08:20:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.514+0000 7f65d517c640 1 -- 172.21.15.107:0/2017426517 wait complete. 2023-11-24T08:20:43.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:20:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.749+0000 7f91f622d640 1 Processor -- start 2023-11-24T08:20:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.750+0000 7f91f622d640 1 -- start start 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.751+0000 7f91f622d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 0x7f91f0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.751+0000 7f91f622d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91f0162540 0x7f91f00c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.751+0000 7f91f622d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f0166510 con 0x7f91f00ccbd0 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.751+0000 7f91f622d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f01655f0 con 0x7f91f0162540 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.751+0000 7f91f622d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00cc380 con 0x7f91f01660b0 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91f0162540 0x7f91f00c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91f0162540 0x7f91f00c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56902/0 (socket says 172.21.15.107:56902) 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 -- 172.21.15.107:0/2001629959 learned_addr learned my addr 172.21.15.107:0/2001629959 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 -- 172.21.15.107:0/2001629959 >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 msgr2=0x7f91f0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91e7fff640 1 --2- 172.21.15.107:0/2001629959 >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 0x7f91f0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 --2- 172.21.15.107:0/2001629959 >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 0x7f91f0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 -- 172.21.15.107:0/2001629959 >> v1:172.21.15.120:6789/0 conn(0x7f91f00ccbd0 legacy=0x7f91f00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91effff640 1 -- 172.21.15.107:0/2001629959 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0162b30 con 0x7f91f0162540 2023-11-24T08:20:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.752+0000 7f91e7fff640 1 --2- 172.21.15.107:0/2001629959 >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 0x7f91f0162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.753+0000 7f91effff640 1 --2- 172.21.15.107:0/2001629959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f0162540 0x7f91f00c9650 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f91f006e700 tx=0x7f91e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5af54d5bd1acf6a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.753+0000 7f91eeffd640 1 -- 172.21.15.107:0/2001629959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e802ca80 con 0x7f91f0162540 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91eeffd640 1 -- 172.21.15.107:0/2001629959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e802cbe0 con 0x7f91f0162540 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91eeffd640 1 -- 172.21.15.107:0/2001629959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e80343f0 con 0x7f91f0162540 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 -- 172.21.15.107:0/2001629959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f0162540 msgr2=0x7f91f00c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 --2- 172.21.15.107:0/2001629959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f0162540 0x7f91f00c9650 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f91f006e700 tx=0x7f91e802bee0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 -- 172.21.15.107:0/2001629959 shutdown_connections 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 --2- 172.21.15.107:0/2001629959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f0162540 0x7f91f00c9650 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 --2- 172.21.15.107:0/2001629959 >> v2:172.21.15.120:3300/0 conn(0x7f91f01660b0 0x7f91f0162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 -- 172.21.15.107:0/2001629959 >> 172.21.15.107:0/2001629959 conn(0x7f91f00b0c50 msgr2=0x7f91f00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 -- 172.21.15.107:0/2001629959 shutdown_connections 2023-11-24T08:20:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.754+0000 7f91f622d640 1 -- 172.21.15.107:0/2001629959 wait complete. 2023-11-24T08:20:43.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.756+0000 7f91f622d640 1 Processor -- start 2023-11-24T08:20:43.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.756+0000 7f91f622d640 1 -- start start 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00ccbd0 0x7f91f014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0162540 0x7f91f00acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f014ebf0 con 0x7f91f01660b0 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f0165360 con 0x7f91f0162540 2023-11-24T08:20:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91f622d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f01630f0 con 0x7f91f00ccbd0 2023-11-24T08:20:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56916/0 (socket says 172.21.15.107:56916) 2023-11-24T08:20:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.757+0000 7f91effff640 1 -- 172.21.15.107:0/3236154013 learned_addr learned my addr 172.21.15.107:0/3236154013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 -- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00ccbd0 msgr2=0x7f91f014e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00ccbd0 0x7f91f014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 -- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0162540 msgr2=0x7f91f00acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0162540 0x7f91f00acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 -- 172.21.15.107:0/3236154013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8009b30 con 0x7f91f01660b0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91effff640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f91e8036040 tx=0x7f91e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8009510 con 0x7f91f01660b0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e803e450 con 0x7f91f01660b0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e803e800 con 0x7f91f01660b0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.758+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f01633d0 con 0x7f91f01660b0 2023-11-24T08:20:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.759+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f0163990 con 0x7f91f01660b0 2023-11-24T08:20:43.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.760+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f0163b80 con 0x7f91f01660b0 2023-11-24T08:20:43.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.761+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f91e802cd50 con 0x7f91f01660b0 2023-11-24T08:20:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.761+0000 7f91ecff9640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 0x7f91d006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.762+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91e80ab010 con 0x7f91f01660b0 2023-11-24T08:20:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.762+0000 7f91ef7fe640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 0x7f91d006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.763+0000 7f91ef7fe640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 0x7f91d006efe0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f91f01655c0 tx=0x7f91e0007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:43.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.766+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91e8077b60 con 0x7f91f01660b0 2023-11-24T08:20:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.986+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f91f0165b90 con 0x7f91d006cb20 2023-11-24T08:20:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.989+0000 7f91ecff9640 1 -- 172.21.15.107:0/3236154013 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f91e00050d0 con 0x7f91d006cb20 2023-11-24T08:20:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:20:43.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:20:43.992 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:20:43.992 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.992+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 msgr2=0x7f91d006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.992+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 0x7f91d006efe0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f91f01655c0 tx=0x7f91e0007990 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.992+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 msgr2=0x7f91f00ad4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.992+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f91e8036040 tx=0x7f91e8004a40 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 shutdown_connections 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91d006cb20 0x7f91d006efe0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91f01660b0 0x7f91f00ad4c0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0162540 0x7f91f00acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 --2- 172.21.15.107:0/3236154013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00ccbd0 0x7f91f014e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 >> 172.21.15.107:0/3236154013 conn(0x7f91f00b0c50 msgr2=0x7f91f00c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:43.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 shutdown_connections 2023-11-24T08:20:43.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:43.993+0000 7f91f622d640 1 -- 172.21.15.107:0/3236154013 wait complete. 2023-11-24T08:20:44.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:20:44.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.227+0000 7ff5ff1a0640 1 Processor -- start 2023-11-24T08:20:44.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.227+0000 7ff5ff1a0640 1 -- start start 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.228+0000 7ff5ff1a0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5f8141b30 0x7ff5f806c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.228+0000 7ff5ff1a0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f815ef00 0x7ff5f8071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.228+0000 7ff5ff1a0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5f8071c40 0x7ff5f80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5ff1a0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f80ccdf0 con 0x7ff5f8141b30 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5ff1a0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f815f470 con 0x7ff5f8071c40 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5ff1a0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f815f750 con 0x7ff5f815ef00 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5f8141b30 0x7ff5f806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5f8141b30 0x7ff5f806c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56930/0 (socket says 172.21.15.107:56930) 2023-11-24T08:20:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 -- 172.21.15.107:0/2145229156 learned_addr learned my addr 172.21.15.107:0/2145229156 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 -- 172.21.15.107:0/2145229156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f815ef00 msgr2=0x7ff5f8071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/2145229156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f815ef00 0x7ff5f8071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 -- 172.21.15.107:0/2145229156 >> v2:172.21.15.120:3300/0 conn(0x7ff5f8071c40 msgr2=0x7ff5f80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/2145229156 >> v2:172.21.15.120:3300/0 conn(0x7ff5f8071c40 0x7ff5f80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.229+0000 7ff5fe19e640 1 -- 172.21.15.107:0/2145229156 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f815fa30 con 0x7ff5f8141b30 2023-11-24T08:20:44.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.230+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/2145229156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f8141b30 0x7ff5f806c770 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8070370 tx=0x7ff5f402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e48b490f5ae38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.231+0000 7ff5fd19c640 1 -- 172.21.15.107:0/2145229156 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4031740 con 0x7ff5f8141b30 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.231+0000 7ff5fd19c640 1 -- 172.21.15.107:0/2145229156 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f40318a0 con 0x7ff5f8141b30 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.231+0000 7ff5fd19c640 1 -- 172.21.15.107:0/2145229156 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4031c30 con 0x7ff5f8141b30 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.231+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/2145229156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f8141b30 msgr2=0x7ff5f806c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.231+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/2145229156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f8141b30 0x7ff5f806c770 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8070370 tx=0x7ff5f402b7e0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/2145229156 shutdown_connections 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/2145229156 >> v2:172.21.15.120:3300/0 conn(0x7ff5f8071c40 0x7ff5f80cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/2145229156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f815ef00 0x7ff5f8071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/2145229156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f8141b30 0x7ff5f806c770 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/2145229156 >> 172.21.15.107:0/2145229156 conn(0x7ff5f806a500 msgr2=0x7ff5f806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:44.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/2145229156 shutdown_connections 2023-11-24T08:20:44.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.232+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/2145229156 wait complete. 2023-11-24T08:20:44.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.234+0000 7ff5ff1a0640 1 Processor -- start 2023-11-24T08:20:44.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.234+0000 7ff5ff1a0640 1 -- start start 2023-11-24T08:20:44.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f8071c40 0x7ff5f813aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 0x7ff5f8143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8144900 con 0x7ff5f815ef00 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8143a50 con 0x7ff5f8141b30 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5ff1a0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8143cf0 con 0x7ff5f8071c40 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56940/0 (socket says 172.21.15.107:56940) 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 -- 172.21.15.107:0/622556922 learned_addr learned my addr 172.21.15.107:0/622556922 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fd99d640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 0x7ff5f8143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f8071c40 0x7ff5f813aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 -- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f8071c40 msgr2=0x7ff5f813aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f8071c40 0x7ff5f813aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 -- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 msgr2=0x7ff5f8143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 0x7ff5f8143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fe99f640 1 -- 172.21.15.107:0/622556922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40094d0 con 0x7ff5f815ef00 2023-11-24T08:20:44.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.236+0000 7ff5fd99d640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 0x7ff5f8143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:44.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5fe99f640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b9e0 tx=0x7ff5f000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000cb20 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f000cc80 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8143fd0 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0014680 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.237+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f8144590 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.238+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8067990 con 0x7ff5f815ef00 2023-11-24T08:20:44.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.240+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7ff5f001d3b0 con 0x7ff5f815ef00 2023-11-24T08:20:44.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.240+0000 7ff5eeffd640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 0x7ff5dc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.241+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5f0088570 con 0x7ff5f815ef00 2023-11-24T08:20:44.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.241+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 0x7ff5dc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.242+0000 7ff5fe19e640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 0x7ff5dc06efb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4035000 tx=0x7ff5f4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.247+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5f0055010 con 0x7ff5f815ef00 2023-11-24T08:20:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.443+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff5f804e840 con 0x7ff5f815ef00 2023-11-24T08:20:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.444+0000 7ff5eeffd640 1 -- 172.21.15.107:0/622556922 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v49)=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff5f0019070 con 0x7ff5f815ef00 2023-11-24T08:20:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 msgr2=0x7ff5dc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 0x7ff5dc06efb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4035000 tx=0x7ff5f4008030 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 msgr2=0x7ff5f813fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b9e0 tx=0x7ff5f000bea0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 shutdown_connections 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5dc06caf0 0x7ff5dc06efb0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5f815ef00 0x7ff5f813fce0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5f8141b30 0x7ff5f8143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 --2- 172.21.15.107:0/622556922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5f8071c40 0x7ff5f813aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.447+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 >> 172.21.15.107:0/622556922 conn(0x7ff5f806a500 msgr2=0x7ff5f815ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.448+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 shutdown_connections 2023-11-24T08:20:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.448+0000 7ff5ff1a0640 1 -- 172.21.15.107:0/622556922 wait complete. 2023-11-24T08:20:44.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol7968 --namespace-isolated --size 12582912 --mode=777 2023-11-24T08:20:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.686+0000 7fc8db59e640 1 Processor -- start 2023-11-24T08:20:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.686+0000 7fc8db59e640 1 -- start start 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8db59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8dc070550 0x7fc8dc0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8db59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0ca0e0 con 0x7fc8dc0b1080 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8db59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0b14a0 con 0x7fc8dc15b1d0 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8db59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc072800 con 0x7fc8dc070550 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8dad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc8dc0b1080 0x7fc8dc15edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57912/0 (socket says 172.21.15.107:57912) 2023-11-24T08:20:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.688+0000 7fc8dad9d640 1 -- 172.21.15.107:0/2211664425 learned_addr learned my addr 172.21.15.107:0/2211664425 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d3fff640 1 --2- 172.21.15.107:0/2211664425 >> v2:172.21.15.120:3300/0 conn(0x7fc8dc070550 0x7fc8dc0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785106662 0 0) 0x7fc8dc0ca0e0 con 0x7fc8dc0b1080 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c8003520 con 0x7fc8dc0b1080 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830561291 0 0) 0x7fc8c8003520 con 0x7fc8dc0b1080 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 >> v2:172.21.15.120:3300/0 conn(0x7fc8dc070550 msgr2=0x7fc8dc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 --2- 172.21.15.107:0/2211664425 >> v2:172.21.15.120:3300/0 conn(0x7fc8dc070550 0x7fc8dc0b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 >> v1:172.21.15.120:6789/0 conn(0x7fc8dc15b1d0 legacy=0x7fc8dc071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0c97b0 con 0x7fc8dc0b1080 2023-11-24T08:20:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc8d40019f0 con 0x7fc8dc0b1080 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 >> v1:172.21.15.107:6789/0 conn(0x7fc8dc0b1080 legacy=0x7fc8dc15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c80039c0 0x7fc8c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8c80047c0 0x7fc8c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 0x7fc8c800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc072800 con 0x7fc8c8005180 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8010490 con 0x7fc8c80039c0 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.689+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8010650 con 0x7fc8c80047c0 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 0x7fc8c800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8da59c640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c80039c0 0x7fc8c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8d3fff640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8c80047c0 0x7fc8c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 -- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8c80047c0 msgr2=0x7fc8c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8c80047c0 0x7fc8c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 -- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c80039c0 msgr2=0x7fc8c8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c80039c0 0x7fc8c8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8dad9d640 1 -- 172.21.15.107:0/2211664425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8003520 con 0x7fc8c8005180 2023-11-24T08:20:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.690+0000 7fc8da59c640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c80039c0 0x7fc8c8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:20:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.692+0000 7fc8dad9d640 1 -- 172.21.15.107:0/2211664425 reap_dead start 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.692+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 0x7fc8c800fd90 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400a760 tx=0x7fc8c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c1d46af1bb92cd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.692+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4034630 con 0x7fc8c8005180 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.692+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d402f070 con 0x7fc8c8005180 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.692+0000 7fc8d9d9b640 1 -- 172.21.15.107:0/2211664425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4034a10 con 0x7fc8c8005180 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 -- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 msgr2=0x7fc8c800fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 0x7fc8c800fd90 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400a760 tx=0x7fc8c80057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 -- 172.21.15.107:0/2211664425 shutdown_connections 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 --2- 172.21.15.107:0/2211664425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8c8005180 0x7fc8c800fd90 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 -- 172.21.15.107:0/2211664425 >> 172.21.15.107:0/2211664425 conn(0x7fc8dc0cbb40 msgr2=0x7fc8dc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 -- 172.21.15.107:0/2211664425 shutdown_connections 2023-11-24T08:20:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.693+0000 7fc8db59e640 1 -- 172.21.15.107:0/2211664425 wait complete. 2023-11-24T08:20:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.695+0000 7fc8db59e640 1 Processor -- start 2023-11-24T08:20:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.695+0000 7fc8db59e640 1 -- start start 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c8005180 0x7fc8dc155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 0x7fc8dc15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc15c740 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0c33e0 con 0x7fc8c8005180 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8db59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc156bf0 con 0x7fc8dc156770 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8dad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56958/0 (socket says 172.21.15.107:56958) 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.696+0000 7fc8dad9d640 1 -- 172.21.15.107:0/322510482 learned_addr learned my addr 172.21.15.107:0/322510482 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:44.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8d3fff640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 0x7fc8dc15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8da59c640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c8005180 0x7fc8dc155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 -- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 msgr2=0x7fc8dc15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 0x7fc8dc15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 -- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c8005180 msgr2=0x7fc8dc155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c8005180 0x7fc8dc155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 -- 172.21.15.107:0/322510482 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc15c380 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8d3fff640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 0x7fc8dc15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8dad9d640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4002230 tx=0x7fc8d4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40019f0 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d40345f0 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc156e80 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.697+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4004da0 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.698+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc157440 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.698+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc04e840 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.700+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fc8d4031030 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.701+0000 7fc8d2ffd640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 0x7fc8bc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:44.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.701+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8d40a9a70 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.701+0000 7fc8da59c640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 0x7fc8bc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:44.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.702+0000 7fc8da59c640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 0x7fc8bc06efb0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc007fd0 tx=0x7fc8cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.705+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8d4076580 con 0x7fc8dc0c2a20 2023-11-24T08:20:44.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.920+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol7968", "namespace_isolated": true, "size": 12582912, "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8dc0ca570 con 0x7fc8bc06caf0 2023-11-24T08:20:44.932 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:44.930+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:44.932 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:44.930+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:44.932 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:44.930+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:44.933 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:44.930+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:44.933 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:44.930+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:44.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.988+0000 7fc8d2ffd640 1 -- 172.21.15.107:0/322510482 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc8cc00d050 con 0x7fc8bc06caf0 2023-11-24T08:20:44.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 msgr2=0x7fc8bc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 0x7fc8bc06efb0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc007fd0 tx=0x7fc8cc009040 comp rx=0 tx=0).stop 2023-11-24T08:20:44.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 msgr2=0x7fc8dc0c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:44.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4002230 tx=0x7fc8d4001cb0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 shutdown_connections 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc8bc06caf0 0x7fc8bc06efb0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8dc0c2a20 0x7fc8dc0c2ea0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8dc156770 0x7fc8dc15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 --2- 172.21.15.107:0/322510482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8c8005180 0x7fc8dc155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 >> 172.21.15.107:0/322510482 conn(0x7fc8dc0cbb40 msgr2=0x7fc8dc0ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.991+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 shutdown_connections 2023-11-24T08:20:44.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:44.992+0000 7fc8db59e640 1 -- 172.21.15.107:0/322510482 wait complete. 2023-11-24T08:20:45.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol7968 2023-11-24T08:20:45.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.215+0000 7fb65ee75640 1 Processor -- start 2023-11-24T08:20:45.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.215+0000 7fb65ee75640 1 -- start start 2023-11-24T08:20:45.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65ee75640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 0x7fb6580ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65ee75640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6580cd110 0x7fb658070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65ee75640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb658161a10 con 0x7fb658146eb0 2023-11-24T08:20:45.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65ee75640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580cd580 con 0x7fb6580cd110 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65ee75640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb658161cf0 con 0x7fb65806e160 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6580cd110 0x7fb658070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6580cd110 0x7fb658070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56980/0 (socket says 172.21.15.107:56980) 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 -- 172.21.15.107:0/809284858 learned_addr learned my addr 172.21.15.107:0/809284858 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb657fff640 1 --2- 172.21.15.107:0/809284858 >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 0x7fb6580ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 -- 172.21.15.107:0/809284858 >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 msgr2=0x7fb6580ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 --2- 172.21.15.107:0/809284858 >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 0x7fb6580ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 -- 172.21.15.107:0/809284858 >> v1:172.21.15.120:6789/0 conn(0x7fb658146eb0 legacy=0x7fb658161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb65d412640 1 -- 172.21.15.107:0/809284858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb658161fd0 con 0x7fb6580cd110 2023-11-24T08:20:45.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.217+0000 7fb657fff640 1 --2- 172.21.15.107:0/809284858 >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 0x7fb6580ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:45.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.219+0000 7fb65d412640 1 --2- 172.21.15.107:0/809284858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6580cd110 0x7fb658070e40 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fb65806eab0 tx=0x7fb65002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e91206579ba64e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:45.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.219+0000 7fb6577fe640 1 -- 172.21.15.107:0/809284858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65002ca80 con 0x7fb6580cd110 2023-11-24T08:20:45.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.219+0000 7fb6577fe640 1 -- 172.21.15.107:0/809284858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65002cbe0 con 0x7fb6580cd110 2023-11-24T08:20:45.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.219+0000 7fb6577fe640 1 -- 172.21.15.107:0/809284858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500343f0 con 0x7fb6580cd110 2023-11-24T08:20:45.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 -- 172.21.15.107:0/809284858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6580cd110 msgr2=0x7fb658070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:45.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 --2- 172.21.15.107:0/809284858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6580cd110 0x7fb658070e40 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fb65806eab0 tx=0x7fb65002bee0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 -- 172.21.15.107:0/809284858 shutdown_connections 2023-11-24T08:20:45.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 --2- 172.21.15.107:0/809284858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6580cd110 0x7fb658070e40 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 --2- 172.21.15.107:0/809284858 >> v2:172.21.15.120:3300/0 conn(0x7fb65806e160 0x7fb6580ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 -- 172.21.15.107:0/809284858 >> 172.21.15.107:0/809284858 conn(0x7fb6580b0c50 msgr2=0x7fb6580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:45.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 -- 172.21.15.107:0/809284858 shutdown_connections 2023-11-24T08:20:45.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.220+0000 7fb65ee75640 1 -- 172.21.15.107:0/809284858 wait complete. 2023-11-24T08:20:45.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.222+0000 7fb65ee75640 1 Processor -- start 2023-11-24T08:20:45.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.222+0000 7fb65ee75640 1 -- start start 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 0x7fb65809c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658146eb0 0x7fb6580acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581f4160 con 0x7fb65806e160 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb658142560 con 0x7fb658146eb0 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65ee75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb658141c30 con 0x7fb6580cd110 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56990/0 (socket says 172.21.15.107:56990) 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 -- 172.21.15.107:0/736629379 learned_addr learned my addr 172.21.15.107:0/736629379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb657fff640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 0x7fb65809c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 -- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 msgr2=0x7fb65809c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65d412640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658146eb0 0x7fb6580acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 0x7fb65809c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 -- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658146eb0 msgr2=0x7fb6580acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658146eb0 0x7fb6580acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb65cc11640 1 -- 172.21.15.107:0/736629379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650009b30 con 0x7fb65806e160 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.223+0000 7fb657fff640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 0x7fb65809c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb65cc11640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b410 tx=0x7fb64c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c011830 con 0x7fb65806e160 2023-11-24T08:20:45.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c011990 con 0x7fb65806e160 2023-11-24T08:20:45.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c011d40 con 0x7fb65806e160 2023-11-24T08:20:45.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb658141f10 con 0x7fb65806e160 2023-11-24T08:20:45.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.224+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65809c620 con 0x7fb65806e160 2023-11-24T08:20:45.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.225+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65804e8b0 con 0x7fb65806e160 2023-11-24T08:20:45.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.226+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 39) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb64c01a400 con 0x7fb65806e160 2023-11-24T08:20:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.228+0000 7fb6557fa640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 0x7fb63c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.228+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb64c088790 con 0x7fb65806e160 2023-11-24T08:20:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.228+0000 7fb657fff640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 0x7fb63c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.229+0000 7fb657fff640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 0x7fb63c06efb0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb644006fd0 tx=0x7fb644008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.233+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb64c0552a0 con 0x7fb65806e160 2023-11-24T08:20:45.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.434+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol7968", "target": ["mon-mgr", ""]}) v1 -- 0x7fb658147290 con 0x7fb63c06caf0 2023-11-24T08:20:45.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.441+0000 7fb6557fa640 1 -- 172.21.15.107:0/736629379 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb64400c040 con 0x7fb63c06caf0 2023-11-24T08:20:45.443 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 msgr2=0x7fb63c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 0x7fb63c06efb0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb644006fd0 tx=0x7fb644008040 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 msgr2=0x7fb6581429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b410 tx=0x7fb64c00b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 shutdown_connections 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb63c06caf0 0x7fb63c06efb0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658146eb0 0x7fb6580acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6580cd110 0x7fb65809c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 --2- 172.21.15.107:0/736629379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65806e160 0x7fb6581429f0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 >> 172.21.15.107:0/736629379 conn(0x7fb6580b0c50 msgr2=0x7fb6580cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 shutdown_connections 2023-11-24T08:20:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:45.444+0000 7fb65ee75640 1 -- 172.21.15.107:0/736629379 wait complete. 2023-11-24T08:20:45.464 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol7968 with 8 files each 1MB size under directory volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:20:45.464 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:45.465 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:45.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:45.515 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:45.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:20:45.570 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:20:45.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:45.635 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:45.635 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190774 s, 55.0 MB/s 2023-11-24T08:20:45.636 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:45.636 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:45.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:45.731 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:45.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:45.857 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:45.857 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256898 s, 40.8 MB/s 2023-11-24T08:20:45.859 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:45.859 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:45.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:45.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:45.997 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:45.998 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:45.998 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322356 s, 32.5 MB/s 2023-11-24T08:20:45.999 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:45.999 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:46.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:46.058 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283978 s, 36.9 MB/s 2023-11-24T08:20:46.142 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:46.142 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:46.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:46.200 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:46.286 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:46.287 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:46.287 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032826 s, 31.9 MB/s 2023-11-24T08:20:46.288 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:46.288 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:46.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:46.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:46.420 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232135 s, 45.2 MB/s 2023-11-24T08:20:46.422 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:46.422 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:46.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:46.479 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297608 s, 35.2 MB/s 2023-11-24T08:20:46.568 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:20:46.568 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:20:46.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:46.624 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:20:46.695 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:20:46.696 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:20:46.696 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202911 s, 51.7 MB/s 2023-11-24T08:20:46.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol7968 subvol_snap3729 2023-11-24T08:20:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.913+0000 7fa01284b640 1 Processor -- start 2023-11-24T08:20:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.913+0000 7fa01284b640 1 -- start start 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa01284b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa00c0d1ec0 0x7fa00c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa01284b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c0d2750 0x7fa00c0d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa01284b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00c0ce280 con 0x7fa00c15d260 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa01284b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00c0ce4d0 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa01284b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c0ce7b0 con 0x7fa00c0d2750 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa011048640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa00c0d1ec0 0x7fa00c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa011849640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa00c15d260 0x7fa00c0d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57914/0 (socket says 172.21.15.107:57914) 2023-11-24T08:20:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.915+0000 7fa011849640 1 -- 172.21.15.107:0/929357016 learned_addr learned my addr 172.21.15.107:0/929357016 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa01204a640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c0d2750 0x7fa00c0d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa011048640 1 -- 172.21.15.107:0/929357016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c0d2750 msgr2=0x7fa00c0d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa011048640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c0d2750 0x7fa00c0d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa011048640 1 -- 172.21.15.107:0/929357016 >> v1:172.21.15.107:6789/0 conn(0x7fa00c15d260 legacy=0x7fa00c0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:46.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa011048640 1 -- 172.21.15.107:0/929357016 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c0b8b40 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.916+0000 7fa010847640 1 -- 172.21.15.107:0/929357016 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472419261 0 0) 0x7fa00c0ce280 con 0x7fa00c15d260 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.917+0000 7fa011048640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0cdb50 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0097a0 tx=0x7f9ffc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d94c50ccdfe8b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.917+0000 7fa010847640 1 -- 172.21.15.107:0/929357016 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc031830 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.917+0000 7fa010847640 1 -- 172.21.15.107:0/929357016 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc031990 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.917+0000 7fa010847640 1 -- 172.21.15.107:0/929357016 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc031d20 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 -- 172.21.15.107:0/929357016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 msgr2=0x7fa00c0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0cdb50 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0097a0 tx=0x7f9ffc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 -- 172.21.15.107:0/929357016 shutdown_connections 2023-11-24T08:20:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c0d2750 0x7fa00c0d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 --2- 172.21.15.107:0/929357016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0cdb50 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 -- 172.21.15.107:0/929357016 >> 172.21.15.107:0/929357016 conn(0x7fa00c061340 msgr2=0x7fa00c061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 -- 172.21.15.107:0/929357016 shutdown_connections 2023-11-24T08:20:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.918+0000 7fa01284b640 1 -- 172.21.15.107:0/929357016 wait complete. 2023-11-24T08:20:46.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.920+0000 7fa01284b640 1 Processor -- start 2023-11-24T08:20:46.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.920+0000 7fa01284b640 1 -- start start 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa00c0d2750 0x7fa00c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c15d260 0x7fa00c1f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c0c9150 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c153bb0 con 0x7fa00c0d2750 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa01284b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c0c9900 con 0x7fa00c15d260 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa011849640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa011849640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57004/0 (socket says 172.21.15.107:57004) 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.921+0000 7fa011849640 1 -- 172.21.15.107:0/1569527167 learned_addr learned my addr 172.21.15.107:0/1569527167 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 -- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c15d260 msgr2=0x7fa00c1f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c15d260 0x7fa00c1f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 -- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa00c0d2750 msgr2=0x7fa00c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa00c0d2750 0x7fa00c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 -- 172.21.15.107:0/1569527167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc009400 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa011849640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fa00800c4d0 tx=0x7fa00800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0080074e0 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa008007660 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c0c9340 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.922+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008004a20 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.923+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c153e50 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.924+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c04e840 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.924+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7fa008004b80 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.925+0000 7fa0027fc640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 0x7f9ff006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.925+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa008087ad0 con 0x7fa00c0d1ec0 2023-11-24T08:20:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.926+0000 7fa011048640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 0x7f9ff006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.926+0000 7fa011048640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 0x7f9ff006f040 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc036000 tx=0x7f9ffc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:46.930+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa008054680 con 0x7fa00c0d1ec0 2023-11-24T08:20:47.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.150+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol7968", "snap_name": "subvol_snap3729", "target": ["mon-mgr", ""]}) v1 -- 0x7fa00c0b9100 con 0x7f9ff006cb80 2023-11-24T08:20:47.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.180+0000 7fa0027fc640 1 -- 172.21.15.107:0/1569527167 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ffc031b00 con 0x7f9ff006cb80 2023-11-24T08:20:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 msgr2=0x7f9ff006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 0x7f9ff006f040 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc036000 tx=0x7f9ffc030000 comp rx=0 tx=0).stop 2023-11-24T08:20:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 msgr2=0x7fa00c0c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fa00800c4d0 tx=0x7fa00800c990 comp rx=0 tx=0).stop 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 shutdown_connections 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9ff006cb80 0x7f9ff006f040 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa00c15d260 0x7fa00c1f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa00c0d2750 0x7fa00c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 --2- 172.21.15.107:0/1569527167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00c0d1ec0 0x7fa00c0c8c10 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 >> 172.21.15.107:0/1569527167 conn(0x7fa00c061340 msgr2=0x7fa00c15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 shutdown_connections 2023-11-24T08:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.183+0000 7fa01284b640 1 -- 172.21.15.107:0/1569527167 wait complete. 2023-11-24T08:20:47.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol7968 2023-11-24T08:20:47.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.411+0000 7f32cffff640 1 Processor -- start 2023-11-24T08:20:47.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.412+0000 7f32cffff640 1 -- start start 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32cffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f32d00ca940 0x7f32d00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32cffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00cb960 con 0x7f32d00cb580 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32cffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d0161e40 con 0x7f32d00c27a0 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32cffff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d0162120 con 0x7f32d00ca940 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32ceffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f32d00cb580 0x7f32d00c2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57920/0 (socket says 172.21.15.107:57920) 2023-11-24T08:20:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.413+0000 7f32ceffd640 1 -- 172.21.15.107:0/1698175680 learned_addr learned my addr 172.21.15.107:0/1698175680 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32cf7fe640 1 --2- 172.21.15.107:0/1698175680 >> v2:172.21.15.120:3300/0 conn(0x7f32d00ca940 0x7f32d00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208408482 0 0) 0x7f32d00cb960 con 0x7f32d00cb580 2023-11-24T08:20:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc003520 con 0x7f32d00cb580 2023-11-24T08:20:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 985173873 0 0) 0x7f32bc003520 con 0x7f32d00cb580 2023-11-24T08:20:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 >> v2:172.21.15.120:3300/0 conn(0x7f32d00ca940 msgr2=0x7f32d00b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 --2- 172.21.15.107:0/1698175680 >> v2:172.21.15.120:3300/0 conn(0x7f32d00ca940 0x7f32d00b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 >> v1:172.21.15.120:6789/0 conn(0x7f32d00c27a0 legacy=0x7f32d00b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0162400 con 0x7f32d00cb580 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f32c0001670 con 0x7f32d00cb580 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 >> v1:172.21.15.107:6789/0 conn(0x7f32d00cb580 legacy=0x7f32d00c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32bc0039c0 0x7f32bc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32bc005180 0x7f32bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d0162120 con 0x7f32bc0047c0 2023-11-24T08:20:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc00c080 con 0x7f32bc0039c0 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.414+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc00c240 con 0x7f32bc005180 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32ceffd640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32bc0039c0 0x7f32bc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 -- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32bc005180 msgr2=0x7f32bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32bc005180 0x7f32bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 -- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32bc0039c0 msgr2=0x7f32bc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32bc0039c0 0x7f32bc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32c67fc640 1 -- 172.21.15.107:0/1698175680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc003520 con 0x7f32bc0047c0 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32ceffd640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32bc0039c0 0x7f32bc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.415+0000 7f32cf7fe640 1 -- 172.21.15.107:0/1698175680 reap_dead start 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.416+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f32b8001750 tx=0x7f32bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5a75307a1e6564 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.416+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80052d0 con 0x7f32bc0047c0 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.416+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b802e050 con 0x7f32bc0047c0 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.416+0000 7f32ce7fc640 1 -- 172.21.15.107:0/1698175680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8003830 con 0x7f32bc0047c0 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 -- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 msgr2=0x7f32bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f32b8001750 tx=0x7f32bc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 -- 172.21.15.107:0/1698175680 shutdown_connections 2023-11-24T08:20:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 --2- 172.21.15.107:0/1698175680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 -- 172.21.15.107:0/1698175680 >> 172.21.15.107:0/1698175680 conn(0x7f32d00cbb40 msgr2=0x7f32d00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 -- 172.21.15.107:0/1698175680 shutdown_connections 2023-11-24T08:20:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.417+0000 7f32cffff640 1 -- 172.21.15.107:0/1698175680 wait complete. 2023-11-24T08:20:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.419+0000 7f32cffff640 1 Processor -- start 2023-11-24T08:20:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.419+0000 7f32cffff640 1 -- start start 2023-11-24T08:20:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32d00c89f0 0x7f32d00c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32d00c70f0 0x7f32d00c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c93b0 con 0x7f32bc0047c0 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00999d0 con 0x7f32d00c70f0 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32cffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c7f00 con 0x7f32d00c89f0 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32ceffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32ceffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57012/0 (socket says 172.21.15.107:57012) 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.420+0000 7f32ceffd640 1 -- 172.21.15.107:0/1747390902 learned_addr learned my addr 172.21.15.107:0/1747390902 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 -- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32d00c89f0 msgr2=0x7f32d00c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32d00c89f0 0x7f32d00c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 -- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32d00c70f0 msgr2=0x7f32d00c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32d00c70f0 0x7f32d00c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 -- 172.21.15.107:0/1747390902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00c81e0 con 0x7f32bc0047c0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32ceffd640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f32c000d7a0 tx=0x7f32c000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c00074e0 con 0x7f32bc0047c0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0007640 con 0x7f32bc0047c0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c00049e0 con 0x7f32bc0047c0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d00c84c0 con 0x7f32bc0047c0 2023-11-24T08:20:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.421+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d015dfa0 con 0x7f32bc0047c0 2023-11-24T08:20:47.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.422+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d004e840 con 0x7f32bc0047c0 2023-11-24T08:20:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.423+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f32c0004b40 con 0x7f32bc0047c0 2023-11-24T08:20:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.424+0000 7f32c7fff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 0x7f32b006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.424+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32c0087e80 con 0x7f32bc0047c0 2023-11-24T08:20:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.424+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 0x7f32b006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.425+0000 7f32c67fc640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 0x7f32b006f040 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f32b8033000 tx=0x7f32b802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.429+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32c0054a30 con 0x7f32bc0047c0 2023-11-24T08:20:47.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.648+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol7968", "target": ["mon-mgr", ""]}) v1 -- 0x7f32d01629c0 con 0x7f32b006cb80 2023-11-24T08:20:47.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.656+0000 7f32c7fff640 1 -- 172.21.15.107:0/1747390902 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f32b80052d0 con 0x7f32b006cb80 2023-11-24T08:20:47.658 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 msgr2=0x7f32b006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 0x7f32b006f040 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f32b8033000 tx=0x7f32b802f000 comp rx=0 tx=0).stop 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 msgr2=0x7f32d015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f32c000d7a0 tx=0x7f32c000dc60 comp rx=0 tx=0).stop 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 shutdown_connections 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f32b006cb80 0x7f32b006f040 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32d00c70f0 0x7f32d00c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32d00c89f0 0x7f32d00c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 --2- 172.21.15.107:0/1747390902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc0047c0 0x7f32d015da60 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 >> 172.21.15.107:0/1747390902 conn(0x7f32d00cbb40 msgr2=0x7f32d00cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:47.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 shutdown_connections 2023-11-24T08:20:47.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.659+0000 7f32cffff640 1 -- 172.21.15.107:0/1747390902 wait complete. 2023-11-24T08:20:47.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:47.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:20:47.734 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:20:47.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create new_pool --pg_num_min 64 2023-11-24T08:20:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.951+0000 7f82bd61c640 1 Processor -- start 2023-11-24T08:20:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.951+0000 7f82bd61c640 1 -- start start 2023-11-24T08:20:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.953+0000 7f82bd61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f82b80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.953+0000 7f82bd61c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82b80cbf80 0x7f82b806c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.953+0000 7f82bd61c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80b3a40 con 0x7f82b8159ce0 2023-11-24T08:20:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.953+0000 7f82bd61c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b80cc410 con 0x7f82b80cbf80 2023-11-24T08:20:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.953+0000 7f82bd61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80cdfa0 con 0x7f82b80b21d0 2023-11-24T08:20:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82b8159ce0 0x7f82b806de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55132/0 (socket says 172.21.15.107:55132) 2023-11-24T08:20:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b6ffd640 1 -- 172.21.15.107:0/1780170767 learned_addr learned my addr 172.21.15.107:0/1780170767 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b67fc640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f82b80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b77fe640 1 --2- 172.21.15.107:0/1780170767 >> v2:172.21.15.120:3300/0 conn(0x7f82b80cbf80 0x7f82b806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191645749 0 0) 0x7f82b80b3a40 con 0x7f82b8159ce0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298003440 con 0x7f82b8159ce0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637572893 0 0) 0x7f8298003440 con 0x7f82b8159ce0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 msgr2=0x7f82b80cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f82b80cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 >> v2:172.21.15.120:3300/0 conn(0x7f82b80cbf80 msgr2=0x7f82b806c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.954+0000 7f82b5ffb640 1 --2- 172.21.15.107:0/1780170767 >> v2:172.21.15.120:3300/0 conn(0x7f82b80cbf80 0x7f82b806c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80cd9e0 con 0x7f82b8159ce0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f82a8001670 con 0x7f82b8159ce0 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 >> v1:172.21.15.107:6789/0 conn(0x7f82b8159ce0 legacy=0x7f82b806de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f8298003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8298008850 0x7f829800ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f829800d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80cc410 con 0x7f8298003a60 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80cdfa0 con 0x7f8298008850 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829800dae0 con 0x7f82b80b21d0 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f8298003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b67fc640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8298008850 0x7f829800ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.955+0000 7f82b77fe640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f829800d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b6ffd640 1 -- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 msgr2=0x7f829800d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f829800d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b6ffd640 1 -- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8298008850 msgr2=0x7f829800ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8298008850 0x7f829800ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b6ffd640 1 -- 172.21.15.107:0/1780170767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298003440 con 0x7f8298003a60 2023-11-24T08:20:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.956+0000 7f82b77fe640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b80b21d0 0x7f829800d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f8298003e80 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c240 tx=0x7f82980036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b171eaa69603cad5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a802cb00 con 0x7f8298003a60 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a802cc60 con 0x7f8298003a60 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82b5ffb640 1 -- 172.21.15.107:0/1780170767 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8005af0 con 0x7f8298003a60 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82bd61c640 1 -- 172.21.15.107:0/1780170767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 msgr2=0x7f8298003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82bd61c640 1 --2- 172.21.15.107:0/1780170767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f8298003e80 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c240 tx=0x7f82980036f0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.957+0000 7f82b77fe640 1 -- 172.21.15.107:0/1780170767 reap_dead start 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.958+0000 7f82bd61c640 1 -- 172.21.15.107:0/1780170767 shutdown_connections 2023-11-24T08:20:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.958+0000 7f82bd61c640 1 -- 172.21.15.107:0/1780170767 >> 172.21.15.107:0/1780170767 conn(0x7f82b806a920 msgr2=0x7f82b806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.958+0000 7f82bd61c640 1 -- 172.21.15.107:0/1780170767 shutdown_connections 2023-11-24T08:20:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.958+0000 7f82bd61c640 1 -- 172.21.15.107:0/1780170767 wait complete. 2023-11-24T08:20:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.959+0000 7f82bd61c640 1 Processor -- start 2023-11-24T08:20:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 -- start start 2023-11-24T08:20:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82b809c970 0x7f82b809cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b8155140 0x7f82b81555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8144e70 con 0x7f8298003a60 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.960+0000 7f82bd61c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8155b00 con 0x7f82b809c970 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82bd61c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80b19f0 con 0x7f82b8155140 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52770/0 (socket says 172.21.15.107:52770) 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 -- 172.21.15.107:0/2969411366 learned_addr learned my addr 172.21.15.107:0/2969411366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b77fe640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b8155140 0x7f82b81555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b67fc640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82b809c970 0x7f82b809cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 -- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b8155140 msgr2=0x7f82b81555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b8155140 0x7f82b81555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 -- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82b809c970 msgr2=0x7f82b809cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82b809c970 0x7f82b809cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.961+0000 7f82b6ffd640 1 -- 172.21.15.107:0/2969411366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8144960 con 0x7f8298003a60 2023-11-24T08:20:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82b6ffd640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c360 tx=0x7f82a8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8005d00 con 0x7f8298003a60 2023-11-24T08:20:47.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8005e60 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b8147cb0 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a80072b0 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.962+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b80b0de0 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.963+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b80cb9c0 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.964+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f82a8032070 con 0x7f8298003a60 2023-11-24T08:20:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.965+0000 7f82af7fe640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 0x7f829c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:47.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.965+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82a80a9bc0 con 0x7f8298003a60 2023-11-24T08:20:47.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.965+0000 7f82b67fc640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 0x7f829c06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:47.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.966+0000 7f82b67fc640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 0x7f829c06f040 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f82b81593e0 tx=0x7f82a000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:47.969+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82a8076640 con 0x7f8298003a60 2023-11-24T08:20:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.156+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "new_pool", "pg_num_min": 64} v 0) v1 -- 0x7f82b8067470 con 0x7f8298003a60 2023-11-24T08:20:48.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.557+0000 7f82af7fe640 1 -- 172.21.15.107:0/2969411366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "new_pool", "pg_num_min": 64}]=0 pool 'new_pool' created v191) v1 ==== 124+0+0 (secure 0 0 0) 0x7f82a8045020 con 0x7f8298003a60 2023-11-24T08:20:48.560 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'new_pool' created 2023-11-24T08:20:48.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 msgr2=0x7f829c06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:48.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 0x7f829c06f040 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f82b81593e0 tx=0x7f82a000b040 comp rx=0 tx=0).stop 2023-11-24T08:20:48.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 msgr2=0x7f82b8144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:48.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c360 tx=0x7f82a8001a00 comp rx=0 tx=0).stop 2023-11-24T08:20:48.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 shutdown_connections 2023-11-24T08:20:48.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f829c06cb80 0x7f829c06f040 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82b8155140 0x7f82b81555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82b809c970 0x7f82b809cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 --2- 172.21.15.107:0/2969411366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8298003a60 0x7f82b8144420 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 >> 172.21.15.107:0/2969411366 conn(0x7f82b806a920 msgr2=0x7f82b8071720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.563+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 shutdown_connections 2023-11-24T08:20:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.564+0000 7f82bd61c640 1 -- 172.21.15.107:0/2969411366 wait complete. 2023-11-24T08:20:48.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs new_pool 2023-11-24T08:20:48.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.802+0000 7f77002d4640 1 Processor -- start 2023-11-24T08:20:48.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.803+0000 7f77002d4640 1 -- start start 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f77002d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 0x7f76f815f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f77002d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76f806ff70 0x7f76f80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f77002d4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f813eb00 con 0x7f76f80cf320 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f77002d4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f80cf7b0 con 0x7f76f806ff70 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f77002d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f813ede0 con 0x7f76f815f530 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f76fd86f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76f806ff70 0x7f76f80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f76fd86f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76f806ff70 0x7f76f80cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52782/0 (socket says 172.21.15.107:52782) 2023-11-24T08:20:48.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.804+0000 7f76fd86f640 1 -- 172.21.15.107:0/3110637240 learned_addr learned my addr 172.21.15.107:0/3110637240 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fd86f640 1 -- 172.21.15.107:0/3110637240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 msgr2=0x7f76f815f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fe070640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 0x7f76f815f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fd86f640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 0x7f76f815f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fd86f640 1 -- 172.21.15.107:0/3110637240 >> v1:172.21.15.120:6789/0 conn(0x7f76f80cf320 legacy=0x7f76f8140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fd86f640 1 -- 172.21.15.107:0/3110637240 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f815fe50 con 0x7f76f806ff70 2023-11-24T08:20:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.805+0000 7f76fe070640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 0x7f76f815f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.806+0000 7f76fd86f640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f806ff70 0x7f76f80cede0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009870 tx=0x7f76ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4725548b2b0ee0c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.806+0000 7f76fd06e640 1 -- 172.21.15.107:0/3110637240 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0317c0 con 0x7f76f806ff70 2023-11-24T08:20:48.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.806+0000 7f76fd06e640 1 -- 172.21.15.107:0/3110637240 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec031920 con 0x7f76f806ff70 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.806+0000 7f76fd06e640 1 -- 172.21.15.107:0/3110637240 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec031cb0 con 0x7f76f806ff70 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 -- 172.21.15.107:0/3110637240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f806ff70 msgr2=0x7f76f80cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f806ff70 0x7f76f80cede0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009870 tx=0x7f76ec02b860 comp rx=0 tx=0).stop 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 -- 172.21.15.107:0/3110637240 shutdown_connections 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f806ff70 0x7f76f80cede0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 --2- 172.21.15.107:0/3110637240 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f815f530 0x7f76f815f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 -- 172.21.15.107:0/3110637240 >> 172.21.15.107:0/3110637240 conn(0x7f76f80b2e40 msgr2=0x7f76f80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 -- 172.21.15.107:0/3110637240 shutdown_connections 2023-11-24T08:20:48.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.807+0000 7f77002d4640 1 -- 172.21.15.107:0/3110637240 wait complete. 2023-11-24T08:20:48.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.809+0000 7f77002d4640 1 Processor -- start 2023-11-24T08:20:48.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.809+0000 7f77002d4640 1 -- start start 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f806ff70 0x7f76f815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f80cf320 0x7f76f8144310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f815ccb0 con 0x7f76f815f530 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f815cf90 con 0x7f76f80cf320 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f77002d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f815d270 con 0x7f76f806ff70 2023-11-24T08:20:48.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f76fe871640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f76fe871640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52794/0 (socket says 172.21.15.107:52794) 2023-11-24T08:20:48.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.810+0000 7f76fe871640 1 -- 172.21.15.107:0/3640527954 learned_addr learned my addr 172.21.15.107:0/3640527954 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe070640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f806ff70 0x7f76f815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fd86f640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f80cf320 0x7f76f8144310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 -- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f806ff70 msgr2=0x7f76f815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f806ff70 0x7f76f815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 -- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f80cf320 msgr2=0x7f76f8144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f80cf320 0x7f76f8144310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 -- 172.21.15.107:0/3640527954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0094d0 con 0x7f76f815f530 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76fe871640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c780 tx=0x7f76f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0004b60 con 0x7f76f815f530 2023-11-24T08:20:48.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f000ce90 con 0x7f76f815f530 2023-11-24T08:20:48.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f000a850 con 0x7f76f815f530 2023-11-24T08:20:48.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.811+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f815d550 con 0x7f76f815f530 2023-11-24T08:20:48.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.812+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f8144850 con 0x7f76f815f530 2023-11-24T08:20:48.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.813+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f815da70 con 0x7f76f815f530 2023-11-24T08:20:48.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.814+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f76f000a9b0 con 0x7f76f815f530 2023-11-24T08:20:48.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.815+0000 7f76f6ffd640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 0x7f76e406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:48.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.815+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f76f0088750 con 0x7f76f815f530 2023-11-24T08:20:48.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.815+0000 7f76fe070640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 0x7f76e406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:48.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.816+0000 7f76fe070640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 0x7f76e406f040 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f76e800a990 tx=0x7f76e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:48.820+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76f815da70 con 0x7f76f815f530 2023-11-24T08:20:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.050+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "new_pool"} v 0) v1 -- 0x7f76f815da70 con 0x7f76f815f530 2023-11-24T08:20:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.562+0000 7f76f6ffd640 1 -- 172.21.15.107:0/3640527954 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "new_pool"}]=0 added data pool 28 to fsmap v119) v1 ==== 132+0+0 (secure 0 0 0) 0x7f76f815da70 con 0x7f76f815f530 2023-11-24T08:20:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:added data pool 28 to fsmap 2023-11-24T08:20:49.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 msgr2=0x7f76e406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 0x7f76e406f040 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f76e800a990 tx=0x7f76e8008040 comp rx=0 tx=0).stop 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 msgr2=0x7f76f80b5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c780 tx=0x7f76f000cc40 comp rx=0 tx=0).stop 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 shutdown_connections 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f76e406cb80 0x7f76e406f040 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f815f530 0x7f76f80b5d20 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f80cf320 0x7f76f8144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 --2- 172.21.15.107:0/3640527954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f806ff70 0x7f76f815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 >> 172.21.15.107:0/3640527954 conn(0x7f76f80b2e40 msgr2=0x7f76f80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:49.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 shutdown_connections 2023-11-24T08:20:49.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.566+0000 7f77002d4640 1 -- 172.21.15.107:0/3640527954 wait complete. 2023-11-24T08:20:49.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:20:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.783+0000 7f4ac63e9640 1 Processor -- start 2023-11-24T08:20:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.783+0000 7f4ac63e9640 1 -- start start 2023-11-24T08:20:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac63e9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4ac00b1180 0x7f4ac01683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac63e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac0161510 con 0x7f4ac0161080 2023-11-24T08:20:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac63e9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac01617f0 con 0x7f4ac0143870 2023-11-24T08:20:49.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac63e9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac0070460 con 0x7f4ac00b1180 2023-11-24T08:20:49.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac4be6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4ac0161080 0x7f4ac00b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55148/0 (socket says 172.21.15.107:55148) 2023-11-24T08:20:49.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac4be6640 1 -- 172.21.15.107:0/3772681733 learned_addr learned my addr 172.21.15.107:0/3772681733 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ac5be8640 1 --2- 172.21.15.107:0/3772681733 >> v2:172.21.15.120:3300/0 conn(0x7f4ac00b1180 0x7f4ac01683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953348929 0 0) 0x7f4ac0161510 con 0x7f4ac0161080 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.785+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aac003520 con 0x7f4ac0161080 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549903137 0 0) 0x7f4aac003520 con 0x7f4ac0161080 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 >> v2:172.21.15.120:3300/0 conn(0x7f4ac00b1180 msgr2=0x7f4ac01683f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 --2- 172.21.15.107:0/3772681733 >> v2:172.21.15.120:3300/0 conn(0x7f4ac00b1180 0x7f4ac01683f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 >> v1:172.21.15.120:6789/0 conn(0x7f4ac0143870 legacy=0x7f4ac0146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0161ad0 con 0x7f4ac0161080 2023-11-24T08:20:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4ab0001670 con 0x7f4ac0161080 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 >> v1:172.21.15.107:6789/0 conn(0x7f4ac0161080 legacy=0x7f4ac00b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4aac003970 0x7f4aac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4aac0047c0 0x7f4aac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4aac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0070460 con 0x7f4aac005180 2023-11-24T08:20:49.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac00c020 con 0x7f4aac0047c0 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac00c1e0 con 0x7f4aac003970 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.786+0000 7f4ac5be8640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4aac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac4be6640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4aac0047c0 0x7f4aac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac5be8640 1 -- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4aac003970 msgr2=0x7f4aac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac5be8640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4aac003970 0x7f4aac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac5be8640 1 -- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4aac0047c0 msgr2=0x7f4aac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:49.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac5be8640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4aac0047c0 0x7f4aac004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.787+0000 7f4ac5be8640 1 -- 172.21.15.107:0/3772681733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac003520 con 0x7f4aac005180 2023-11-24T08:20:49.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.788+0000 7f4ac5be8640 1 -- 172.21.15.107:0/3772681733 reap_dead start 2023-11-24T08:20:49.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.788+0000 7f4ac5be8640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4aac00b980 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f4ac006e250 tx=0x7f4aac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68494bfda71462fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:49.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.788+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab802cbd0 con 0x7f4aac005180 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.788+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab800f040 con 0x7f4aac005180 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.788+0000 7f4ab7fff640 1 -- 172.21.15.107:0/3772681733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8034440 con 0x7f4aac005180 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 -- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 msgr2=0x7f4aac00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4aac00b980 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f4ac006e250 tx=0x7f4aac0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 -- 172.21.15.107:0/3772681733 shutdown_connections 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/3772681733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4aac00b980 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 -- 172.21.15.107:0/3772681733 >> 172.21.15.107:0/3772681733 conn(0x7f4ac00cb830 msgr2=0x7f4ac00cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 -- 172.21.15.107:0/3772681733 shutdown_connections 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.789+0000 7f4ac63e9640 1 -- 172.21.15.107:0/3772681733 wait complete. 2023-11-24T08:20:49.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.791+0000 7f4ac63e9640 1 Processor -- start 2023-11-24T08:20:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.791+0000 7f4ac63e9640 1 -- start start 2023-11-24T08:20:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac013ea60 0x7f4ac015fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac015eb30 0x7f4ac0141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac01603b0 con 0x7f4aac005180 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac01605f0 con 0x7f4ac013ea60 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac63e9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac01608d0 con 0x7f4ac015eb30 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac53e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac53e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52802/0 (socket says 172.21.15.107:52802) 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.792+0000 7f4ac53e7640 1 -- 172.21.15.107:0/4139195916 learned_addr learned my addr 172.21.15.107:0/4139195916 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 -- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac015eb30 msgr2=0x7f4ac0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac015eb30 0x7f4ac0141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 -- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac013ea60 msgr2=0x7f4ac015fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac013ea60 0x7f4ac015fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 -- 172.21.15.107:0/4139195916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac015e050 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac53e7640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c450 tx=0x7f4abc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0052d0 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc005430 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc012870 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac015e330 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.793+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac015efb0 con 0x7f4aac005180 2023-11-24T08:20:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.794+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac0067990 con 0x7f4aac005180 2023-11-24T08:20:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.795+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f4abc0129d0 con 0x7f4aac005180 2023-11-24T08:20:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.796+0000 7f4ab5ffb640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 0x7f4aa406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.796+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f4abc088040 con 0x7f4aac005180 2023-11-24T08:20:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.797+0000 7f4ac4be6640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 0x7f4aa406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.797+0000 7f4ac4be6640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 0x7f4aa406f070 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00017a0 tx=0x7f4ab0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:49.801+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4abc054bf0 con 0x7f4aac005180 2023-11-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.030+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ac004e840 con 0x7f4aac005180 2023-11-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.031+0000 7f4ab5ffb640 1 -- 172.21.15.107:0/4139195916 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f4abc058210 con 0x7f4aac005180 2023-11-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:49.053270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27,28],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.033+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 msgr2=0x7f4aa406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.033+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 0x7f4aa406f070 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00017a0 tx=0x7f4ab0009040 comp rx=0 tx=0).stop 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.033+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 msgr2=0x7f4ac01f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.033+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c450 tx=0x7f4abc00c910 comp rx=0 tx=0).stop 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.033+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 shutdown_connections 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4aa406cbb0 0x7f4aa406f070 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac015eb30 0x7f4ac0141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac013ea60 0x7f4ac015fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 --2- 172.21.15.107:0/4139195916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aac005180 0x7f4ac01f3c40 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 >> 172.21.15.107:0/4139195916 conn(0x7f4ac00cb830 msgr2=0x7f4ac00affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:50.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 shutdown_connections 2023-11-24T08:20:50.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.034+0000 7f4ac63e9640 1 -- 172.21.15.107:0/4139195916 wait complete. 2023-11-24T08:20:50.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:20:50.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.261+0000 7f414ed0a640 1 Processor -- start 2023-11-24T08:20:50.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.261+0000 7f414ed0a640 1 -- start start 2023-11-24T08:20:50.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.262+0000 7f414ed0a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41480b2250 0x7f414815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.262+0000 7f414ed0a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 0x7f414815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.262+0000 7f414ed0a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414815ab80 con 0x7f41480b2630 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.262+0000 7f414ed0a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414815a990 con 0x7f41480b2250 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.262+0000 7f414ed0a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414815e680 con 0x7f41480b2aa0 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41480b2250 0x7f414815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41480b2250 0x7f414815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52814/0 (socket says 172.21.15.107:52814) 2023-11-24T08:20:50.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 -- 172.21.15.107:0/2836260431 learned_addr learned my addr 172.21.15.107:0/2836260431 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414d2a7640 1 --2- 172.21.15.107:0/2836260431 >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 0x7f414815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 -- 172.21.15.107:0/2836260431 >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 msgr2=0x7f414815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 --2- 172.21.15.107:0/2836260431 >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 0x7f414815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 -- 172.21.15.107:0/2836260431 >> v1:172.21.15.107:6789/0 conn(0x7f41480b2630 legacy=0x7f41480ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414caa6640 1 -- 172.21.15.107:0/2836260431 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414815e960 con 0x7f41480b2250 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f41477fe640 1 -- 172.21.15.107:0/2836260431 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126214673 0 0) 0x7f414815ab80 con 0x7f41480b2630 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.263+0000 7f414d2a7640 1 --2- 172.21.15.107:0/2836260431 >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 0x7f414815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:50.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.264+0000 7f414caa6640 1 --2- 172.21.15.107:0/2836260431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2250 0x7f414815df80 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f41480b2fa0 tx=0x7f413802e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e261a46ae75fc0a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f41477fe640 1 -- 172.21.15.107:0/2836260431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138036440 con 0x7f41480b2250 2023-11-24T08:20:50.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f41477fe640 1 -- 172.21.15.107:0/2836260431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41380365a0 con 0x7f41480b2250 2023-11-24T08:20:50.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f41477fe640 1 -- 172.21.15.107:0/2836260431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138036930 con 0x7f41480b2250 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 -- 172.21.15.107:0/2836260431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2250 msgr2=0x7f414815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 --2- 172.21.15.107:0/2836260431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2250 0x7f414815df80 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f41480b2fa0 tx=0x7f413802e180 comp rx=0 tx=0).stop 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 -- 172.21.15.107:0/2836260431 shutdown_connections 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 --2- 172.21.15.107:0/2836260431 >> v2:172.21.15.120:3300/0 conn(0x7f41480b2aa0 0x7f414815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 --2- 172.21.15.107:0/2836260431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2250 0x7f414815df80 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 -- 172.21.15.107:0/2836260431 >> 172.21.15.107:0/2836260431 conn(0x7f414806a920 msgr2=0x7f414806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 -- 172.21.15.107:0/2836260431 shutdown_connections 2023-11-24T08:20:50.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.265+0000 7f414ed0a640 1 -- 172.21.15.107:0/2836260431 wait complete. 2023-11-24T08:20:50.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.267+0000 7f414ed0a640 1 Processor -- start 2023-11-24T08:20:50.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.267+0000 7f414ed0a640 1 -- start start 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 0x7f41481f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 0x7f4148071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41481560b0 con 0x7f41480b2630 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480ca6d0 con 0x7f41480b2250 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f414ed0a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4148155430 con 0x7f41480b2aa0 2023-11-24T08:20:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f4147fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f4147fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52818/0 (socket says 172.21.15.107:52818) 2023-11-24T08:20:50.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.268+0000 7f4147fff640 1 -- 172.21.15.107:0/1133791580 learned_addr learned my addr 172.21.15.107:0/1133791580 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f414d2a7640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 0x7f4148071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f414caa6640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 0x7f41481f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 -- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 msgr2=0x7f4148071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 0x7f4148071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 -- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 msgr2=0x7f41481f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 0x7f41481f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 -- 172.21.15.107:0/1133791580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413800b770 con 0x7f41480b2630 2023-11-24T08:20:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f414d2a7640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 0x7f4148071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f414caa6640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 0x7f41481f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f4147fff640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f413c00c4b0 tx=0x7f413c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c005480 con 0x7f41480b2630 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c0035f0 con 0x7f41480b2630 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.269+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c0039e0 con 0x7f41480b2630 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.270+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41480cd7c0 con 0x7f41480b2630 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.270+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41481559f0 con 0x7f41480b2630 2023-11-24T08:20:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.271+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41481561f0 con 0x7f41480b2630 2023-11-24T08:20:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.272+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f413c003b40 con 0x7f41480b2630 2023-11-24T08:20:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.272+0000 7f41457fa640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 0x7f413406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.273+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f413c087bf0 con 0x7f41480b2630 2023-11-24T08:20:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.273+0000 7f414caa6640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 0x7f413406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.273+0000 7f414caa6640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 0x7f413406f040 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f4138038000 tx=0x7f4138008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.277+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41481561f0 con 0x7f41480b2630 2023-11-24T08:20:50.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.471+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41481561f0 con 0x7f41480b2630 2023-11-24T08:20:50.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.474+0000 7f41457fa640 1 -- 172.21.15.107:0/1133791580 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+18373 (secure 0 0 0) 0x7f41481561f0 con 0x7f41480b2630 2023-11-24T08:20:50.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:20:50.476 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":192,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:49.052895+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.476+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 msgr2=0x7f413406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.476+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 0x7f413406f040 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f4138038000 tx=0x7f4138008040 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.476+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 msgr2=0x7f41480c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.476+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f413c00c4b0 tx=0x7f413c00c970 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 shutdown_connections 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f413406cb80 0x7f413406f040 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41480b2aa0 0x7f4148071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41480b2630 0x7f41480c9990 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 --2- 172.21.15.107:0/1133791580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41480b2250 0x7f41481f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 >> 172.21.15.107:0/1133791580 conn(0x7f414806a920 msgr2=0x7f41480cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 shutdown_connections 2023-11-24T08:20:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.477+0000 7f414ed0a640 1 -- 172.21.15.107:0/1133791580 wait complete. 2023-11-24T08:20:50.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol7968 2023-11-24T08:20:50.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.708+0000 7fef6dc9b640 1 Processor -- start 2023-11-24T08:20:50.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.708+0000 7fef6dc9b640 1 -- start start 2023-11-24T08:20:50.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef6dc9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef6806de10 0x7fef6815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef6dc9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef680c49f0 0x7fef680705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef6dc9b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6815ec90 con 0x7fef68072160 2023-11-24T08:20:50.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef6dc9b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef680cd530 con 0x7fef6806de10 2023-11-24T08:20:50.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef6dc9b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef680c4ed0 con 0x7fef680c49f0 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef5ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef6806de10 0x7fef6815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef5ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef6806de10 0x7fef6815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52830/0 (socket says 172.21.15.107:52830) 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef5ffff640 1 -- 172.21.15.107:0/3844468127 learned_addr learned my addr 172.21.15.107:0/3844468127 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef67fff640 1 --2- 172.21.15.107:0/3844468127 >> v2:172.21.15.120:3300/0 conn(0x7fef680c49f0 0x7fef680705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.710+0000 7fef5ffff640 1 -- 172.21.15.107:0/3844468127 >> v2:172.21.15.120:3300/0 conn(0x7fef680c49f0 msgr2=0x7fef680705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.711+0000 7fef5ffff640 1 --2- 172.21.15.107:0/3844468127 >> v2:172.21.15.120:3300/0 conn(0x7fef680c49f0 0x7fef680705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.711+0000 7fef5ffff640 1 -- 172.21.15.107:0/3844468127 >> v1:172.21.15.120:6789/0 conn(0x7fef68072160 legacy=0x7fef6806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.711+0000 7fef5ffff640 1 -- 172.21.15.107:0/3844468127 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef680c51b0 con 0x7fef6806de10 2023-11-24T08:20:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.712+0000 7fef5ffff640 1 --2- 172.21.15.107:0/3844468127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6815e750 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fef68072a30 tx=0x7fef5002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e25dd2e1732086 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.712+0000 7fef66ffd640 1 -- 172.21.15.107:0/3844468127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5002ee90 con 0x7fef6806de10 2023-11-24T08:20:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.712+0000 7fef66ffd640 1 -- 172.21.15.107:0/3844468127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50036440 con 0x7fef6806de10 2023-11-24T08:20:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.712+0000 7fef66ffd640 1 -- 172.21.15.107:0/3844468127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500367d0 con 0x7fef6806de10 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3844468127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 msgr2=0x7fef6815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3844468127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6815e750 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fef68072a30 tx=0x7fef5002e0e0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3844468127 shutdown_connections 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3844468127 >> v2:172.21.15.120:3300/0 conn(0x7fef680c49f0 0x7fef680705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3844468127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6815e750 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3844468127 >> 172.21.15.107:0/3844468127 conn(0x7fef680b0c50 msgr2=0x7fef680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3844468127 shutdown_connections 2023-11-24T08:20:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.713+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3844468127 wait complete. 2023-11-24T08:20:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.714+0000 7fef6dc9b640 1 Processor -- start 2023-11-24T08:20:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.715+0000 7fef6dc9b640 1 -- start start 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68072160 0x7fef680cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef680c49f0 0x7fef680cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815c9a0 con 0x7fef6806de10 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815cc80 con 0x7fef68072160 2023-11-24T08:20:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef6dc9b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815cf60 con 0x7fef680c49f0 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52834/0 (socket says 172.21.15.107:52834) 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 -- 172.21.15.107:0/3084411633 learned_addr learned my addr 172.21.15.107:0/3084411633 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 -- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef680c49f0 msgr2=0x7fef680cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef680c49f0 0x7fef680cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 -- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68072160 msgr2=0x7fef680cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef5ffff640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68072160 0x7fef680cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68072160 0x7fef680cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.716+0000 7fef677fe640 1 -- 172.21.15.107:0/3084411633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5000b770 con 0x7fef6806de10 2023-11-24T08:20:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef677fe640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fef5800c430 tx=0x7fef5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58011790 con 0x7fef6806de10 2023-11-24T08:20:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef580118f0 con 0x7fef6806de10 2023-11-24T08:20:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58011ce0 con 0x7fef6806de10 2023-11-24T08:20:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6815d240 con 0x7fef6806de10 2023-11-24T08:20:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.717+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef68150060 con 0x7fef6806de10 2023-11-24T08:20:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.718+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6815a170 con 0x7fef6806de10 2023-11-24T08:20:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.719+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7fef5801c460 con 0x7fef6806de10 2023-11-24T08:20:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.720+0000 7fef64ff9640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 0x7fef4806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.720+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fef58088530 con 0x7fef6806de10 2023-11-24T08:20:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.721+0000 7fef5ffff640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 0x7fef4806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:50.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.721+0000 7fef5ffff640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 0x7fef4806f040 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fef50038000 tx=0x7fef50032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.725+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef580550e0 con 0x7fef6806de10 2023-11-24T08:20:50.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.945+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol7968", "target": ["mon-mgr", ""]}) v1 -- 0x7fef6815a810 con 0x7fef4806cb80 2023-11-24T08:20:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.955+0000 7fef64ff9640 1 -- 172.21.15.107:0/3084411633 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fef6815a810 con 0x7fef4806cb80 2023-11-24T08:20:50.958 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:20:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 msgr2=0x7fef4806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 0x7fef4806f040 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fef50038000 tx=0x7fef50032000 comp rx=0 tx=0).stop 2023-11-24T08:20:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 msgr2=0x7fef6814fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fef5800c430 tx=0x7fef5800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 shutdown_connections 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4806cb80 0x7fef4806f040 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef680c49f0 0x7fef680cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68072160 0x7fef680cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 --2- 172.21.15.107:0/3084411633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef6806de10 0x7fef6814fb20 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 >> 172.21.15.107:0/3084411633 conn(0x7fef680b0c50 msgr2=0x7fef680b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.958+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 shutdown_connections 2023-11-24T08:20:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:50.959+0000 7fef6dc9b640 1 -- 172.21.15.107:0/3084411633 wait complete. 2023-11-24T08:20:50.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:50.978 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v new_pool volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:20:51.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:20:51.067 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool_namespace -v '' volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:20:51.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol7968 subvol_snap3729 subvol_clone404 2023-11-24T08:20:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.373+0000 7f1dedebb640 1 Processor -- start 2023-11-24T08:20:51.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.373+0000 7f1dedebb640 1 -- start start 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1dedebb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1de806dd50 0x7f1de80b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1dedebb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de815e580 con 0x7f1de8070ea0 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1dedebb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de815e860 con 0x7f1de815a1c0 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1dedebb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80cd920 con 0x7f1de806dd50 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1de7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1de8070ea0 0x7f1de806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55168/0 (socket says 172.21.15.107:55168) 2023-11-24T08:20:51.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.375+0000 7f1de7fff640 1 -- 172.21.15.107:0/742859618 learned_addr learned my addr 172.21.15.107:0/742859618 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1ded6ba640 1 --2- 172.21.15.107:0/742859618 >> v2:172.21.15.120:3300/0 conn(0x7f1de806dd50 0x7f1de80b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261950150 0 0) 0x7f1de815e580 con 0x7f1de8070ea0 2023-11-24T08:20:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd8003520 con 0x7f1de8070ea0 2023-11-24T08:20:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267500213 0 0) 0x7f1dd8003520 con 0x7f1de8070ea0 2023-11-24T08:20:51.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 >> v2:172.21.15.120:3300/0 conn(0x7f1de806dd50 msgr2=0x7f1de80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 --2- 172.21.15.107:0/742859618 >> v2:172.21.15.120:3300/0 conn(0x7f1de806dd50 0x7f1de80b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 >> v1:172.21.15.120:6789/0 conn(0x7f1de815a1c0 legacy=0x7f1de815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de815eb40 con 0x7f1de8070ea0 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1dd4001670 con 0x7f1de8070ea0 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 >> v1:172.21.15.107:6789/0 conn(0x7f1de8070ea0 legacy=0x7f1de806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 0x7f1dd8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd80047c0 0x7f1dd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd8005180 0x7f1dd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80cd920 con 0x7f1dd80039c0 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd800c060 con 0x7f1dd80047c0 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd800c220 con 0x7f1dd8005180 2023-11-24T08:20:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.376+0000 7f1deceb9640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 0x7f1dd8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1de7fff640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd80047c0 0x7f1dd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1deceb9640 1 -- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd8005180 msgr2=0x7f1dd800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1deceb9640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd8005180 0x7f1dd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1deceb9640 1 -- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd80047c0 msgr2=0x7f1dd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1deceb9640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dd80047c0 0x7f1dd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1deceb9640 1 -- 172.21.15.107:0/742859618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8003520 con 0x7f1dd80039c0 2023-11-24T08:20:51.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.377+0000 7f1ded6ba640 1 -- 172.21.15.107:0/742859618 reap_dead start 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.378+0000 7f1deceb9640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 0x7f1dd8003de0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc001750 tx=0x7f1dd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd380ba0e03181f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.378+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0052d0 con 0x7f1dd80039c0 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.378+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc02e050 con 0x7f1dd80039c0 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.378+0000 7f1de77fe640 1 -- 172.21.15.107:0/742859618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc003830 con 0x7f1dd80039c0 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 -- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 msgr2=0x7f1dd8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 0x7f1dd8003de0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc001750 tx=0x7f1dd80057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 -- 172.21.15.107:0/742859618 shutdown_connections 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 --2- 172.21.15.107:0/742859618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dd80039c0 0x7f1dd8003de0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 -- 172.21.15.107:0/742859618 >> 172.21.15.107:0/742859618 conn(0x7f1de80cbb40 msgr2=0x7f1de80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 -- 172.21.15.107:0/742859618 shutdown_connections 2023-11-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.379+0000 7f1dedebb640 1 -- 172.21.15.107:0/742859618 wait complete. 2023-11-24T08:20:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.381+0000 7f1dedebb640 1 Processor -- start 2023-11-24T08:20:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.381+0000 7f1dedebb640 1 -- start start 2023-11-24T08:20:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80039c0 0x7f1de80c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81582c0 0x7f1de8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8158cb0 con 0x7f1de809b520 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8146e20 con 0x7f1de81582c0 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1dedebb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de809b9b0 con 0x7f1dd80039c0 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52860/0 (socket says 172.21.15.107:52860) 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 -- 172.21.15.107:0/3158813962 learned_addr learned my addr 172.21.15.107:0/3158813962 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 -- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80039c0 msgr2=0x7f1de80c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80039c0 0x7f1de80c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 -- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81582c0 msgr2=0x7f1de8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81582c0 0x7f1de8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 -- 172.21.15.107:0/3158813962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de809bc90 con 0x7f1de809b520 2023-11-24T08:20:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.382+0000 7f1de7fff640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400e730 tx=0x7f1dd400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.383+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400c640 con 0x7f1de809b520 2023-11-24T08:20:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.383+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd400c7c0 con 0x7f1de809b520 2023-11-24T08:20:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.383+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400cb70 con 0x7f1de809b520 2023-11-24T08:20:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.383+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de809bf70 con 0x7f1de809b520 2023-11-24T08:20:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.383+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de80c5620 con 0x7f1de809b520 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.384+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8067d70 con 0x7f1de809b520 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.385+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f1dd401d460 con 0x7f1de809b520 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.386+0000 7f1de57fa640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 0x7f1dcc06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.386+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f1dd40886e0 con 0x7f1de809b520 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.386+0000 7f1deceb9640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 0x7f1dcc06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.387+0000 7f1deceb9640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 0x7f1dcc06f040 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc033000 tx=0x7f1ddc02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.391+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dd40551f0 con 0x7f1de809b520 2023-11-24T08:20:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.619+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol7968", "snap_name": "subvol_snap3729", "target_sub_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f1de8146f90 con 0x7f1dcc06cb80 2023-11-24T08:20:51.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.746+0000 7f1de57fa640 1 -- 172.21.15.107:0/3158813962 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1de8146f90 con 0x7f1dcc06cb80 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.749+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 msgr2=0x7f1dcc06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.749+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 0x7f1dcc06f040 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc033000 tx=0x7f1ddc02f000 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.749+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 msgr2=0x7f1de8155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.749+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400e730 tx=0x7f1dd400ebf0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 shutdown_connections 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1dcc06cb80 0x7f1dcc06f040 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de81582c0 0x7f1de8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de809b520 0x7f1de8155f90 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 --2- 172.21.15.107:0/3158813962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dd80039c0 0x7f1de80c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 >> 172.21.15.107:0/3158813962 conn(0x7f1de80cbb40 msgr2=0x7f1de80ca990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:51.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 shutdown_connections 2023-11-24T08:20:51.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.750+0000 7f1dedebb640 1 -- 172.21.15.107:0/3158813962 wait complete. 2023-11-24T08:20:51.758 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.756+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.758 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.756+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.758 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.756+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.758 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.756+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.759 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.756+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:20:51.781 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.780+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.781 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.780+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.782 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.780+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.782 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.780+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.782 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:20:51.780+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:20:51.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.963+0000 7f9deaabb640 1 Processor -- start 2023-11-24T08:20:51.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.963+0000 7f9deaabb640 1 -- start start 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9deaabb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9de40cc0f0 0x7f9de40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9deaabb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9de40c57e0 0x7f9de406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9deaabb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de415eea0 con 0x7f9de415b440 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9deaabb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de415b820 con 0x7f9de40c57e0 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9deaabb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de40abf30 con 0x7f9de40cc0f0 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9de9058640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9de40c57e0 0x7f9de406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9de8857640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9de415b440 0x7f9de40cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55174/0 (socket says 172.21.15.107:55174) 2023-11-24T08:20:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9de8857640 1 -- 172.21.15.107:0/2078159194 learned_addr learned my addr 172.21.15.107:0/2078159194 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:51.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.965+0000 7f9de3fff640 1 --2- 172.21.15.107:0/2078159194 >> v2:172.21.15.120:3300/0 conn(0x7f9de40cc0f0 0x7f9de40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.966+0000 7f9de9058640 1 -- 172.21.15.107:0/2078159194 >> v2:172.21.15.120:3300/0 conn(0x7f9de40cc0f0 msgr2=0x7f9de40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:51.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.966+0000 7f9de9058640 1 --2- 172.21.15.107:0/2078159194 >> v2:172.21.15.120:3300/0 conn(0x7f9de40cc0f0 0x7f9de40c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.966+0000 7f9de9058640 1 -- 172.21.15.107:0/2078159194 >> v1:172.21.15.107:6789/0 conn(0x7f9de415b440 legacy=0x7f9de40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.966+0000 7f9de9058640 1 -- 172.21.15.107:0/2078159194 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de415f0f0 con 0x7f9de40c57e0 2023-11-24T08:20:51.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.966+0000 7f9de37fe640 1 -- 172.21.15.107:0/2078159194 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169554992 0 0) 0x7f9de415eea0 con 0x7f9de415b440 2023-11-24T08:20:51.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.967+0000 7f9de9058640 1 --2- 172.21.15.107:0/2078159194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40c57e0 0x7f9de406c5f0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9de406e720 tx=0x7f9ddc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10dcc19a56e2591c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.967+0000 7f9de37fe640 1 -- 172.21.15.107:0/2078159194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc02cbf0 con 0x7f9de40c57e0 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.967+0000 7f9de37fe640 1 -- 172.21.15.107:0/2078159194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc02cd50 con 0x7f9de40c57e0 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.967+0000 7f9de37fe640 1 -- 172.21.15.107:0/2078159194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc034660 con 0x7f9de40c57e0 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 -- 172.21.15.107:0/2078159194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40c57e0 msgr2=0x7f9de406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 --2- 172.21.15.107:0/2078159194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40c57e0 0x7f9de406c5f0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9de406e720 tx=0x7f9ddc02be40 comp rx=0 tx=0).stop 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 -- 172.21.15.107:0/2078159194 shutdown_connections 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 --2- 172.21.15.107:0/2078159194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40c57e0 0x7f9de406c5f0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 --2- 172.21.15.107:0/2078159194 >> v2:172.21.15.120:3300/0 conn(0x7f9de40cc0f0 0x7f9de40c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 -- 172.21.15.107:0/2078159194 >> 172.21.15.107:0/2078159194 conn(0x7f9de40b0c50 msgr2=0x7f9de40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 -- 172.21.15.107:0/2078159194 shutdown_connections 2023-11-24T08:20:51.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.968+0000 7f9deaabb640 1 -- 172.21.15.107:0/2078159194 wait complete. 2023-11-24T08:20:51.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.969+0000 7f9deaabb640 1 Processor -- start 2023-11-24T08:20:51.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.970+0000 7f9deaabb640 1 -- start start 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.970+0000 7f9deaabb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 0x7f9de4159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.970+0000 7f9deaabb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9deaabb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9de415b440 0x7f9de4156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9deaabb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de4156ab0 con 0x7f9de40cc0f0 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9deaabb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de4155c20 con 0x7f9de40c57e0 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9deaabb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de4155ea0 con 0x7f9de415b440 2023-11-24T08:20:51.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52912/0 (socket says 172.21.15.107:52912) 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 -- 172.21.15.107:0/1763746861 learned_addr learned my addr 172.21.15.107:0/1763746861 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de8857640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 0x7f9de4159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 -- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9de415b440 msgr2=0x7f9de4156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9de415b440 0x7f9de4156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 -- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 msgr2=0x7f9de4159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 0x7f9de4159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de3fff640 1 -- 172.21.15.107:0/1763746861 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc009a70 con 0x7f9de40cc0f0 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.971+0000 7f9de8857640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 0x7f9de4159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:51.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9de3fff640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000d830 tx=0x7f9dd000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd000b6c0 con 0x7f9de40cc0f0 2023-11-24T08:20:51.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd000b820 con 0x7f9de40cc0f0 2023-11-24T08:20:51.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de4156180 con 0x7f9de40cc0f0 2023-11-24T08:20:51.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd000bbb0 con 0x7f9de40cc0f0 2023-11-24T08:20:51.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.972+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de4156740 con 0x7f9de40cc0f0 2023-11-24T08:20:51.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.973+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de41568d0 con 0x7f9de40cc0f0 2023-11-24T08:20:51.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.974+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 40) v1 ==== 91883+0+0 (secure 0 0 0) 0x7f9dd001e460 con 0x7f9de40cc0f0 2023-11-24T08:20:51.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.975+0000 7f9de17fa640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 0x7f9dc806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:51.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.975+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f9dd0088d80 con 0x7f9de40cc0f0 2023-11-24T08:20:51.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.975+0000 7f9de8857640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 0x7f9dc806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:51.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.976+0000 7f9de8857640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 0x7f9dc806f040 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8000f60 tx=0x7f9dd8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:51.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:51.979+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dd00558a0 con 0x7f9de40cc0f0 2023-11-24T08:20:52.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.207+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f9de415f990 con 0x7f9dc806cb80 2023-11-24T08:20:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.218+0000 7f9de17fa640 1 -- 172.21.15.107:0/1763746861 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f9dd80012f0 con 0x7f9dc806cb80 2023-11-24T08:20:52.221 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:20:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:52.223 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 msgr2=0x7f9dc806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 0x7f9dc806f040 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8000f60 tx=0x7f9dd8009040 comp rx=0 tx=0).stop 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 msgr2=0x7f9de4151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000d830 tx=0x7f9dd000dcf0 comp rx=0 tx=0).stop 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 shutdown_connections 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9dc806cb80 0x7f9dc806f040 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9de415b440 0x7f9de4156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9de40cc0f0 0x7f9de4151f40 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 --2- 172.21.15.107:0/1763746861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de40c57e0 0x7f9de4159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 >> 172.21.15.107:0/1763746861 conn(0x7f9de40b0c50 msgr2=0x7f9de40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 shutdown_connections 2023-11-24T08:20:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:52.221+0000 7f9deaabb640 1 -- 172.21.15.107:0/1763746861 wait complete. 2023-11-24T08:20:53.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:20:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.458+0000 7f445d831640 1 Processor -- start 2023-11-24T08:20:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.458+0000 7f445d831640 1 -- start start 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f445d831640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f44580b3790 0x7f445806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f445d831640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445806dcf0 con 0x7f44580b3330 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f445d831640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4458161ec0 con 0x7f4458161060 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f445d831640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445806ec10 con 0x7f44580b3790 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f44577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f44580b3790 0x7f445806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f44577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f44580b3790 0x7f445806adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52928/0 (socket says 172.21.15.107:52928) 2023-11-24T08:20:53.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.460+0000 7f44577fe640 1 -- 172.21.15.107:0/826256617 learned_addr learned my addr 172.21.15.107:0/826256617 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.461+0000 7f44577fe640 1 -- 172.21.15.107:0/826256617 >> v1:172.21.15.120:6789/0 conn(0x7f4458161060 legacy=0x7f4458161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.461+0000 7f44577fe640 1 -- 172.21.15.107:0/826256617 >> v1:172.21.15.107:6789/0 conn(0x7f44580b3330 legacy=0x7f445806a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.461+0000 7f44577fe640 1 -- 172.21.15.107:0/826256617 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445806df90 con 0x7f44580b3790 2023-11-24T08:20:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.462+0000 7f44577fe640 1 --2- 172.21.15.107:0/826256617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f445806adc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f4450009e50 tx=0x7f445002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71a4238bd7dcd05 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.462+0000 7f4455ffb640 1 -- 172.21.15.107:0/826256617 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445002ca80 con 0x7f44580b3790 2023-11-24T08:20:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.462+0000 7f4455ffb640 1 -- 172.21.15.107:0/826256617 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445002cbe0 con 0x7f44580b3790 2023-11-24T08:20:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.462+0000 7f4455ffb640 1 -- 172.21.15.107:0/826256617 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44500343f0 con 0x7f44580b3790 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 -- 172.21.15.107:0/826256617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 msgr2=0x7f445806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 --2- 172.21.15.107:0/826256617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f445806adc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f4450009e50 tx=0x7f445002bee0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 -- 172.21.15.107:0/826256617 shutdown_connections 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 --2- 172.21.15.107:0/826256617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f445806adc0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 -- 172.21.15.107:0/826256617 >> 172.21.15.107:0/826256617 conn(0x7f44580cf900 msgr2=0x7f44580cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 -- 172.21.15.107:0/826256617 shutdown_connections 2023-11-24T08:20:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.463+0000 7f445d831640 1 -- 172.21.15.107:0/826256617 wait complete. 2023-11-24T08:20:53.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.465+0000 7f445d831640 1 Processor -- start 2023-11-24T08:20:53.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.465+0000 7f445d831640 1 -- start start 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44580b3330 0x7f4458141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 0x7f4458152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4458141910 con 0x7f44580b3790 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4458142370 con 0x7f4458161060 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f445d831640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4458142650 con 0x7f44580b3330 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f44567fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f44567fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52934/0 (socket says 172.21.15.107:52934) 2023-11-24T08:20:53.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.466+0000 7f44567fc640 1 -- 172.21.15.107:0/1221442718 learned_addr learned my addr 172.21.15.107:0/1221442718 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44577fe640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 0x7f4458152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f4456ffd640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44580b3330 0x7f4458141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 -- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44580b3330 msgr2=0x7f4458141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44580b3330 0x7f4458141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 -- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 msgr2=0x7f4458152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 0x7f4458152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 -- 172.21.15.107:0/1221442718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009b30 con 0x7f44580b3790 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44577fe640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 0x7f4458152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f44567fc640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c400 tx=0x7f444c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c0035f0 con 0x7f44580b3790 2023-11-24T08:20:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c003750 con 0x7f44580b3790 2023-11-24T08:20:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c003ac0 con 0x7f44580b3790 2023-11-24T08:20:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4458142930 con 0x7f44580b3790 2023-11-24T08:20:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.467+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4458142ef0 con 0x7f44580b3790 2023-11-24T08:20:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.469+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4458067990 con 0x7f44580b3790 2023-11-24T08:20:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.470+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f444c003c20 con 0x7f44580b3790 2023-11-24T08:20:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.470+0000 7f443ffff640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 0x7f444406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.471+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f444c087ad0 con 0x7f44580b3790 2023-11-24T08:20:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.471+0000 7f4456ffd640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 0x7f444406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.471+0000 7f4456ffd640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 0x7f444406f160 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f444800a750 tx=0x7f4448008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.475+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f444c054690 con 0x7f44580b3790 2023-11-24T08:20:53.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.704+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f445806e830 con 0x7f444406cca0 2023-11-24T08:20:53.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.711+0000 7f443ffff640 1 -- 172.21.15.107:0/1221442718 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f444800a380 con 0x7f444406cca0 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:20:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:20:53.714 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:20:53.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:53.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:53.714 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 msgr2=0x7f444406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 0x7f444406f160 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f444800a750 tx=0x7f4448008040 comp rx=0 tx=0).stop 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 msgr2=0x7f44581413d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c400 tx=0x7f444c00c8c0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 shutdown_connections 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f444406cca0 0x7f444406f160 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4458161060 0x7f4458152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44580b3790 0x7f44581413d0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 --2- 172.21.15.107:0/1221442718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44580b3330 0x7f4458141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.713+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 >> 172.21.15.107:0/1221442718 conn(0x7f44580cf900 msgr2=0x7f445815f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.714+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 shutdown_connections 2023-11-24T08:20:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:53.714+0000 7f445d831640 1 -- 172.21.15.107:0/1221442718 wait complete. 2023-11-24T08:20:54.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:20:54.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.954+0000 7f6328c83640 1 Processor -- start 2023-11-24T08:20:54.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.954+0000 7f6328c83640 1 -- start start 2023-11-24T08:20:54.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.955+0000 7f6328c83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f632415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.955+0000 7f6328c83640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632415ec90 con 0x7f6324072160 2023-11-24T08:20:54.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.955+0000 7f6328c83640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240cd530 con 0x7f63240c49f0 2023-11-24T08:20:54.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.955+0000 7f6328c83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240c4ed0 con 0x7f632406de10 2023-11-24T08:20:54.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.956+0000 7f632259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6324072160 0x7f632406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55202/0 (socket says 172.21.15.107:55202) 2023-11-24T08:20:54.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.956+0000 7f632259c640 1 -- 172.21.15.107:0/1081635893 learned_addr learned my addr 172.21.15.107:0/1081635893 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.956+0000 7f6321d9b640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f632415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.956+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305959818 0 0) 0x7f6314001110 con 0x7f6324072160 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.956+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f630c003520 con 0x7f6324072160 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1519950857 0 0) 0x7f630c003520 con 0x7f6324072160 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 msgr2=0x7f632415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f632415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 >> v1:172.21.15.120:6789/0 conn(0x7f63240c49f0 legacy=0x7f63240705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240c51b0 con 0x7f6324072160 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6314001800 con 0x7f6324072160 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 >> v1:172.21.15.107:6789/0 conn(0x7f6324072160 legacy=0x7f632406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:54.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630c0039a0 0x7f630c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f630c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f630c004d00 0x7f630c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63240c4ed0 con 0x7f630c0039a0 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c00dc70 con 0x7f630c004d00 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c00de30 con 0x7f632406de10 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.957+0000 7f632259c640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630c0039a0 0x7f630c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f6321d9b640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f630c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f632259c640 1 -- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 msgr2=0x7f630c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f632259c640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f632406de10 0x7f630c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f632259c640 1 -- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f630c004d00 msgr2=0x7f630c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f6322d9d640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f630c004d00 0x7f630c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f632259c640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f630c004d00 0x7f630c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f632259c640 1 -- 172.21.15.107:0/1081635893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c003520 con 0x7f630c0039a0 2023-11-24T08:20:54.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.958+0000 7f6322d9d640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f630c004d00 0x7f630c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.959+0000 7f632259c640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630c0039a0 0x7f630c003dc0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f6314007fa0 tx=0x7f630c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90878e0325151eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.959+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314005ad0 con 0x7f630c0039a0 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.959+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314005c30 con 0x7f630c0039a0 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.959+0000 7f632159a640 1 -- 172.21.15.107:0/1081635893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631402c790 con 0x7f630c0039a0 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 -- 172.21.15.107:0/1081635893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630c0039a0 msgr2=0x7f630c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 --2- 172.21.15.107:0/1081635893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630c0039a0 0x7f630c003dc0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f6314007fa0 tx=0x7f630c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6322d9d640 1 -- 172.21.15.107:0/1081635893 reap_dead start 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 -- 172.21.15.107:0/1081635893 shutdown_connections 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 -- 172.21.15.107:0/1081635893 >> 172.21.15.107:0/1081635893 conn(0x7f63240b0c50 msgr2=0x7f63240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 -- 172.21.15.107:0/1081635893 shutdown_connections 2023-11-24T08:20:54.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.960+0000 7f6328c83640 1 -- 172.21.15.107:0/1081635893 wait complete. 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.961+0000 7f6328c83640 1 Processor -- start 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.962+0000 7f6328c83640 1 -- start start 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.962+0000 7f6328c83640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f630c0039a0 0x7f632414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6328c83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6328c83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63240c8930 0x7f63240aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6328c83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632414d0c0 con 0x7f6324159aa0 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6328c83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632415c1c0 con 0x7f63240c8930 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6328c83640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632415b5b0 con 0x7f630c0039a0 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52946/0 (socket says 172.21.15.107:52946) 2023-11-24T08:20:54.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 -- 172.21.15.107:0/1146584877 learned_addr learned my addr 172.21.15.107:0/1146584877 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 -- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f630c0039a0 msgr2=0x7f632414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f632259c640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f630c0039a0 0x7f632414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6322d9d640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63240c8930 0x7f63240aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f630c0039a0 0x7f632414cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 -- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63240c8930 msgr2=0x7f63240aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63240c8930 0x7f63240aead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.963+0000 7f6321d9b640 1 -- 172.21.15.107:0/1146584877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632415b890 con 0x7f6324159aa0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6321d9b640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f631800f350 tx=0x7f631800f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318017600 con 0x7f6324159aa0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6318011040 con 0x7f6324159aa0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632415bb70 con 0x7f6324159aa0 2023-11-24T08:20:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63180179c0 con 0x7f6324159aa0 2023-11-24T08:20:54.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.964+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324099f10 con 0x7f6324159aa0 2023-11-24T08:20:54.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.965+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632415c330 con 0x7f6324159aa0 2023-11-24T08:20:54.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.966+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f6318017b20 con 0x7f6324159aa0 2023-11-24T08:20:54.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.968+0000 7f6312ffd640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 0x7f630406f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:54.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.968+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f6318088b80 con 0x7f6324159aa0 2023-11-24T08:20:54.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.968+0000 7f632259c640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 0x7f630406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:54.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.969+0000 7f632259c640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 0x7f630406f190 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6314034000 tx=0x7f6314037040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:54.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:54.972+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63180557f0 con 0x7f6324159aa0 2023-11-24T08:20:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:55.191+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f63240ad030 con 0x7f630406ccd0 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.895+0000 7f6312ffd640 1 -- 172.21.15.107:0/1146584877 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f63240ad030 con 0x7f630406ccd0 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:20:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:20:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:57.898 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 msgr2=0x7f630406f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 0x7f630406f190 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6314034000 tx=0x7f6314037040 comp rx=0 tx=0).stop 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 msgr2=0x7f63240999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f631800f350 tx=0x7f631800f810 comp rx=0 tx=0).stop 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 shutdown_connections 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.898+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f630406ccd0 0x7f630406f190 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63240c8930 0x7f63240aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6324159aa0 0x7f63240999d0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 --2- 172.21.15.107:0/1146584877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f630c0039a0 0x7f632414cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:57.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 >> 172.21.15.107:0/1146584877 conn(0x7f63240b0c50 msgr2=0x7f63240ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:57.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 shutdown_connections 2023-11-24T08:20:57.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:57.899+0000 7f6328c83640 1 -- 172.21.15.107:0/1146584877 wait complete. 2023-11-24T08:20:58.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:20:59.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.146+0000 7f24eb6bb640 1 Processor -- start 2023-11-24T08:20:59.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.146+0000 7f24eb6bb640 1 -- start start 2023-11-24T08:20:59.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24eb6bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24e415b440 0x7f24e40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24eb6bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24e40c59b0 0x7f24e406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24eb6bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e40c5dc0 con 0x7f24e40cbf20 2023-11-24T08:20:59.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24eb6bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e40cc3d0 con 0x7f24e415b440 2023-11-24T08:20:59.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24eb6bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e415b820 con 0x7f24e40c59b0 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24e9457640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24e415b440 0x7f24e40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24e9457640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f24e415b440 0x7f24e40cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56238/0 (socket says 172.21.15.107:56238) 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.149+0000 7f24e9457640 1 -- 172.21.15.107:0/3494925767 learned_addr learned my addr 172.21.15.107:0/3494925767 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.150+0000 7f24e9c58640 1 --2- 172.21.15.107:0/3494925767 >> v2:172.21.15.120:3300/0 conn(0x7f24e40c59b0 0x7f24e406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.150+0000 7f24e9457640 1 -- 172.21.15.107:0/3494925767 >> v2:172.21.15.120:3300/0 conn(0x7f24e40c59b0 msgr2=0x7f24e406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.150+0000 7f24e9457640 1 --2- 172.21.15.107:0/3494925767 >> v2:172.21.15.120:3300/0 conn(0x7f24e40c59b0 0x7f24e406c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.150+0000 7f24e9457640 1 -- 172.21.15.107:0/3494925767 >> v1:172.21.15.120:6789/0 conn(0x7f24e40cbf20 legacy=0x7f24e40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:20:59.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.150+0000 7f24e9457640 1 -- 172.21.15.107:0/3494925767 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e415eea0 con 0x7f24e415b440 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.151+0000 7f24e9457640 1 --2- 172.21.15.107:0/3494925767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e415b440 0x7f24e40cb9e0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f24e40cb910 tx=0x7f24e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc518e436b7e8ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.151+0000 7f24dbfff640 1 -- 172.21.15.107:0/3494925767 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e00317e0 con 0x7f24e415b440 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.151+0000 7f24dbfff640 1 -- 172.21.15.107:0/3494925767 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0031940 con 0x7f24e415b440 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.151+0000 7f24dbfff640 1 -- 172.21.15.107:0/3494925767 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0031cd0 con 0x7f24e415b440 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 -- 172.21.15.107:0/3494925767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e415b440 msgr2=0x7f24e40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/3494925767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e415b440 0x7f24e40cb9e0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f24e40cb910 tx=0x7f24e002b880 comp rx=0 tx=0).stop 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 -- 172.21.15.107:0/3494925767 shutdown_connections 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/3494925767 >> v2:172.21.15.120:3300/0 conn(0x7f24e40c59b0 0x7f24e406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/3494925767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e415b440 0x7f24e40cb9e0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 -- 172.21.15.107:0/3494925767 >> 172.21.15.107:0/3494925767 conn(0x7f24e40b0c50 msgr2=0x7f24e40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:59.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 -- 172.21.15.107:0/3494925767 shutdown_connections 2023-11-24T08:20:59.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.152+0000 7f24eb6bb640 1 -- 172.21.15.107:0/3494925767 wait complete. 2023-11-24T08:20:59.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.153+0000 7f24eb6bb640 1 Processor -- start 2023-11-24T08:20:59.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.154+0000 7f24eb6bb640 1 -- start start 2023-11-24T08:20:59.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.154+0000 7f24eb6bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24e40c59b0 0x7f24e40c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.154+0000 7f24eb6bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.154+0000 7f24eb6bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e415b440 0x7f24e409be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24eb6bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e40c78c0 con 0x7f24e40cbf20 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24eb6bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e415db60 con 0x7f24e40c59b0 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24eb6bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e40c6610 con 0x7f24e415b440 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56244/0 (socket says 172.21.15.107:56244) 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 -- 172.21.15.107:0/1906275697 learned_addr learned my addr 172.21.15.107:0/1906275697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e9457640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24e40c59b0 0x7f24e40c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 -- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e415b440 msgr2=0x7f24e409be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e415b440 0x7f24e409be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 -- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24e40c59b0 msgr2=0x7f24e40c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24e40c59b0 0x7f24e40c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 -- 172.21.15.107:0/1906275697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00094d0 con 0x7f24e40cbf20 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.155+0000 7f24e8c56640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c410 tx=0x7f24d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.156+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d40035f0 con 0x7f24e40cbf20 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.156+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d4003750 con 0x7f24e40cbf20 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.156+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4003b40 con 0x7f24e40cbf20 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.156+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e40c6820 con 0x7f24e40cbf20 2023-11-24T08:20:59.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.156+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e40c6de0 con 0x7f24e40cbf20 2023-11-24T08:20:59.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.157+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e404e8b0 con 0x7f24e40cbf20 2023-11-24T08:20:59.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.158+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f24d4003ca0 con 0x7f24e40cbf20 2023-11-24T08:20:59.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.159+0000 7f24d9ffb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 0x7f24c806f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:20:59.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.159+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f24d40877f0 con 0x7f24e40cbf20 2023-11-24T08:20:59.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.159+0000 7f24e9457640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 0x7f24c806f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:20:59.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.160+0000 7f24e9457640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 0x7f24c806f160 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f24e0036000 tx=0x7f24e003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:20:59.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.164+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24d40543a0 con 0x7f24e40cbf20 2023-11-24T08:20:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.392+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f24e40ccd90 con 0x7f24c806cca0 2023-11-24T08:20:59.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.403+0000 7f24d9ffb640 1 -- 172.21.15.107:0/1906275697 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f24e40ccd90 con 0x7f24c806cca0 2023-11-24T08:20:59.404 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:20:59.404 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:20:59.404 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:20:59.405 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:20:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 msgr2=0x7f24c806f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 0x7f24c806f160 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f24e0036000 tx=0x7f24e003d040 comp rx=0 tx=0).stop 2023-11-24T08:20:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 msgr2=0x7f24e4156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:20:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c410 tx=0x7f24d400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 shutdown_connections 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f24c806cca0 0x7f24c806f160 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24e415b440 0x7f24e409be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24e40cbf20 0x7f24e4156d10 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 --2- 172.21.15.107:0/1906275697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24e40c59b0 0x7f24e40c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 >> 172.21.15.107:0/1906275697 conn(0x7f24e40b0c50 msgr2=0x7f24e40cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.406+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 shutdown_connections 2023-11-24T08:20:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:20:59.407+0000 7f24eb6bb640 1 -- 172.21.15.107:0/1906275697 wait complete. 2023-11-24T08:21:00.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:00.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.656+0000 7fcf29e51640 1 Processor -- start 2023-11-24T08:21:00.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.656+0000 7fcf29e51640 1 -- start start 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.657+0000 7fcf29e51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf2415b440 0x7fcf240cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf29e51640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf240c5dc0 con 0x7fcf240cbf20 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf29e51640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2415eea0 con 0x7fcf240c59b0 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf29e51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240cc3d0 con 0x7fcf2415b440 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf237fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf2415b440 0x7fcf240cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf237fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf2415b440 0x7fcf240cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56254/0 (socket says 172.21.15.107:56254) 2023-11-24T08:21:00.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.658+0000 7fcf237fe640 1 -- 172.21.15.107:0/1795746924 learned_addr learned my addr 172.21.15.107:0/1795746924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.659+0000 7fcf227fc640 1 -- 172.21.15.107:0/1795746924 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312838582 0 0) 0x7fcf240c5dc0 con 0x7fcf240cbf20 2023-11-24T08:21:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.659+0000 7fcf237fe640 1 -- 172.21.15.107:0/1795746924 >> v1:172.21.15.120:6789/0 conn(0x7fcf240c59b0 legacy=0x7fcf2406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.659+0000 7fcf237fe640 1 -- 172.21.15.107:0/1795746924 >> v1:172.21.15.107:6789/0 conn(0x7fcf240cbf20 legacy=0x7fcf240c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.659+0000 7fcf237fe640 1 -- 172.21.15.107:0/1795746924 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240abf30 con 0x7fcf2415b440 2023-11-24T08:21:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.660+0000 7fcf237fe640 1 --2- 172.21.15.107:0/1795746924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240cb9e0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009870 tx=0x7fcf1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120780c18d0e9094 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:00.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.660+0000 7fcf227fc640 1 -- 172.21.15.107:0/1795746924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14031870 con 0x7fcf2415b440 2023-11-24T08:21:00.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.660+0000 7fcf227fc640 1 -- 172.21.15.107:0/1795746924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf140319d0 con 0x7fcf2415b440 2023-11-24T08:21:00.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.660+0000 7fcf227fc640 1 -- 172.21.15.107:0/1795746924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14031d60 con 0x7fcf2415b440 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 -- 172.21.15.107:0/1795746924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 msgr2=0x7fcf240cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 --2- 172.21.15.107:0/1795746924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240cb9e0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009870 tx=0x7fcf1402b910 comp rx=0 tx=0).stop 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 -- 172.21.15.107:0/1795746924 shutdown_connections 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 --2- 172.21.15.107:0/1795746924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240cb9e0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 -- 172.21.15.107:0/1795746924 >> 172.21.15.107:0/1795746924 conn(0x7fcf240b0c50 msgr2=0x7fcf240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 -- 172.21.15.107:0/1795746924 shutdown_connections 2023-11-24T08:21:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.661+0000 7fcf29e51640 1 -- 172.21.15.107:0/1795746924 wait complete. 2023-11-24T08:21:00.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.663+0000 7fcf29e51640 1 Processor -- start 2023-11-24T08:21:00.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.663+0000 7fcf29e51640 1 -- start start 2023-11-24T08:21:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240c59b0 0x7fcf241565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240cbf20 0x7fcf2415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2415c5a0 con 0x7fcf2415b440 2023-11-24T08:21:00.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240c8b40 con 0x7fcf240cbf20 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf29e51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240c8e20 con 0x7fcf240c59b0 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56270/0 (socket says 172.21.15.107:56270) 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 -- 172.21.15.107:0/3389908611 learned_addr learned my addr 172.21.15.107:0/3389908611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf237fe640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240c59b0 0x7fcf241565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 -- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240c59b0 msgr2=0x7fcf241565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240c59b0 0x7fcf241565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 -- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240cbf20 msgr2=0x7fcf2415c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240cbf20 0x7fcf2415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.664+0000 7fcf23fff640 1 -- 172.21.15.107:0/3389908611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140094d0 con 0x7fcf2415b440 2023-11-24T08:21:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf23fff640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00c780 tx=0x7fcf1c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c004b60 con 0x7fcf2415b440 2023-11-24T08:21:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c00ce90 con 0x7fcf2415b440 2023-11-24T08:21:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00a850 con 0x7fcf2415b440 2023-11-24T08:21:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf240b1e90 con 0x7fcf2415b440 2023-11-24T08:21:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.665+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf24156dd0 con 0x7fcf2415b440 2023-11-24T08:21:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.667+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf240b23b0 con 0x7fcf2415b440 2023-11-24T08:21:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.667+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fcf1c00a9b0 con 0x7fcf2415b440 2023-11-24T08:21:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.668+0000 7fcf0bfff640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 0x7fcf1006f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.668+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fcf1c088700 con 0x7fcf2415b440 2023-11-24T08:21:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.669+0000 7fcf237fe640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 0x7fcf1006f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:00.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.669+0000 7fcf237fe640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 0x7fcf1006f190 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fcf14036000 tx=0x7fcf1403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.673+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf240b23b0 con 0x7fcf2415b440 2023-11-24T08:21:00.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.899+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf240b1d10 con 0x7fcf1006ccd0 2023-11-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.911+0000 7fcf0bfff640 1 -- 172.21.15.107:0/3389908611 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fcf14031cc0 con 0x7fcf1006ccd0 2023-11-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.913+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 msgr2=0x7fcf1006f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.913+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 0x7fcf1006f190 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fcf14036000 tx=0x7fcf1403c040 comp rx=0 tx=0).stop 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.913+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 msgr2=0x7fcf240c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.913+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00c780 tx=0x7fcf1c00cc40 comp rx=0 tx=0).stop 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 shutdown_connections 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcf1006ccd0 0x7fcf1006f190 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf2415b440 0x7fcf240c80f0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240cbf20 0x7fcf2415c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 --2- 172.21.15.107:0/3389908611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240c59b0 0x7fcf241565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 >> 172.21.15.107:0/3389908611 conn(0x7fcf240b0c50 msgr2=0x7fcf240cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 shutdown_connections 2023-11-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:00.914+0000 7fcf29e51640 1 -- 172.21.15.107:0/3389908611 wait complete. 2023-11-24T08:21:01.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:02.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.158+0000 7f0dc9161640 1 Processor -- start 2023-11-24T08:21:02.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.159+0000 7f0dc9161640 1 -- start start 2023-11-24T08:21:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.160+0000 7f0dc9161640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0dc415b440 0x7f0dc40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.160+0000 7f0dc9161640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40c5dc0 con 0x7f0dc40cbf20 2023-11-24T08:21:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.160+0000 7f0dc9161640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc415eea0 con 0x7f0dc40c59b0 2023-11-24T08:21:02.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.160+0000 7f0dc9161640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc40cc3d0 con 0x7f0dc415b440 2023-11-24T08:21:02.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0dc40cbf20 0x7f0dc40c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45030/0 (socket says 172.21.15.107:45030) 2023-11-24T08:21:02.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc259c640 1 -- 172.21.15.107:0/876781849 learned_addr learned my addr 172.21.15.107:0/876781849 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc2d9d640 1 --2- 172.21.15.107:0/876781849 >> v2:172.21.15.120:3300/0 conn(0x7f0dc415b440 0x7f0dc40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853764091 0 0) 0x7f0dc40c5dc0 con 0x7f0dc40cbf20 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0003520 con 0x7f0dc40cbf20 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3862024708 0 0) 0x7f0db0003520 con 0x7f0dc40cbf20 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 >> v2:172.21.15.120:3300/0 conn(0x7f0dc415b440 msgr2=0x7f0dc40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 --2- 172.21.15.107:0/876781849 >> v2:172.21.15.120:3300/0 conn(0x7f0dc415b440 0x7f0dc40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.161+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 >> v1:172.21.15.120:6789/0 conn(0x7f0dc40c59b0 legacy=0x7f0dc406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40abf30 con 0x7f0dc40cbf20 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0db40019f0 con 0x7f0dc40cbf20 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 >> v1:172.21.15.107:6789/0 conn(0x7f0dc40cbf20 legacy=0x7f0dc40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db00039c0 0x7f0db0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00047c0 0x7f0db0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 0x7f0db000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40cc3d0 con 0x7f0db00051d0 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db000c120 con 0x7f0db00039c0 2023-11-24T08:21:02.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db000c2e0 con 0x7f0db00047c0 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 0x7f0db000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc2d9d640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db00039c0 0x7f0db0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 -- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00047c0 msgr2=0x7f0db0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00047c0 0x7f0db0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 -- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db00039c0 msgr2=0x7f0db0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:02.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db00039c0 0x7f0db0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.162+0000 7f0dc359e640 1 -- 172.21.15.107:0/876781849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0003520 con 0x7f0db00051d0 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.163+0000 7f0dc359e640 1 -- 172.21.15.107:0/876781849 reap_dead start 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.163+0000 7f0dc359e640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 0x7f0db000b980 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc001720 tx=0x7f0db00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5998463851f49855 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0052d0 con 0x7f0db00051d0 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc02e050 con 0x7f0db00051d0 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc1d9b640 1 -- 172.21.15.107:0/876781849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0035f0 con 0x7f0db00051d0 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 -- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 msgr2=0x7f0db000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 0x7f0db000b980 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc001720 tx=0x7f0db00057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 -- 172.21.15.107:0/876781849 shutdown_connections 2023-11-24T08:21:02.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 --2- 172.21.15.107:0/876781849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0db00051d0 0x7f0db000b980 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 -- 172.21.15.107:0/876781849 >> 172.21.15.107:0/876781849 conn(0x7f0dc40b0c50 msgr2=0x7f0dc40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:02.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 -- 172.21.15.107:0/876781849 shutdown_connections 2023-11-24T08:21:02.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.164+0000 7f0dc9161640 1 -- 172.21.15.107:0/876781849 wait complete. 2023-11-24T08:21:02.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.166+0000 7f0dc9161640 1 Processor -- start 2023-11-24T08:21:02.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.166+0000 7f0dc9161640 1 -- start start 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00051d0 0x7f0dc409b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc415bda0 0x7f0dc40afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c60d0 con 0x7f0dc41f65c0 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41f6aa0 con 0x7f0dc415bda0 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc9161640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41f6d80 con 0x7f0db00051d0 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.167+0000 7f0dc259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56276/0 (socket says 172.21.15.107:56276) 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 -- 172.21.15.107:0/1568469145 learned_addr learned my addr 172.21.15.107:0/1568469145 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 -- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00051d0 msgr2=0x7f0dc409b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00051d0 0x7f0dc409b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 -- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc415bda0 msgr2=0x7f0dc40afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc415bda0 0x7f0dc40afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 -- 172.21.15.107:0/1568469145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc41f7060 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc259c640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f0db400ca70 tx=0x7f0db400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400a6d0 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db400a850 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400ac80 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.168+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc41f7340 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.169+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4153460 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.170+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc40c6240 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.171+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f0db40070d0 con 0x7f0dc41f65c0 2023-11-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.172+0000 7f0dbb7fe640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 0x7f0da406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.172+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f0db4087dc0 con 0x7f0dc41f65c0 2023-11-24T08:21:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.172+0000 7f0dc2d9d640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 0x7f0da406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.173+0000 7f0dc2d9d640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 0x7f0da406f160 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009890 tx=0x7f0dac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:02.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.177+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0dc40c6240 con 0x7f0dc41f65c0 2023-11-24T08:21:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.405+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dc415f3d0 con 0x7f0da406cca0 2023-11-24T08:21:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.417+0000 7f0dbb7fe640 1 -- 172.21.15.107:0/1568469145 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f0dac009e70 con 0x7f0da406cca0 2023-11-24T08:21:02.418 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:02.418 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:02.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:02.420 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 msgr2=0x7f0da406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 0x7f0da406f160 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009890 tx=0x7f0dac008040 comp rx=0 tx=0).stop 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 msgr2=0x7f0dc4152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f0db400ca70 tx=0x7f0db400cf30 comp rx=0 tx=0).stop 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 shutdown_connections 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0da406cca0 0x7f0da406f160 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc415bda0 0x7f0dc40afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0dc41f65c0 0x7f0dc4152f20 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 --2- 172.21.15.107:0/1568469145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0db00051d0 0x7f0dc409b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 >> 172.21.15.107:0/1568469145 conn(0x7f0dc40b0c50 msgr2=0x7f0dc4071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 shutdown_connections 2023-11-24T08:21:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:02.420+0000 7f0dc9161640 1 -- 172.21.15.107:0/1568469145 wait complete. 2023-11-24T08:21:03.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.665+0000 7fd1b666b640 1 Processor -- start 2023-11-24T08:21:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.665+0000 7fd1b666b640 1 -- start start 2023-11-24T08:21:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b666b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd1b01660b0 0x7fd1b0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b666b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b0166510 con 0x7fd1b0162540 2023-11-24T08:21:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b666b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b01655f0 con 0x7fd1b00ccbd0 2023-11-24T08:21:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b666b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b00cc380 con 0x7fd1b01660b0 2023-11-24T08:21:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b4c08640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1b0162540 0x7fd1b00c9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45040/0 (socket says 172.21.15.107:45040) 2023-11-24T08:21:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.667+0000 7fd1b4c08640 1 -- 172.21.15.107:0/824620003 learned_addr learned my addr 172.21.15.107:0/824620003 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/824620003 >> v2:172.21.15.120:3300/0 conn(0x7fd1b01660b0 0x7fd1b0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769477767 0 0) 0x7fd1a8001490 con 0x7fd1b0162540 2023-11-24T08:21:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c003520 con 0x7fd1b0162540 2023-11-24T08:21:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353900442 0 0) 0x7fd19c003520 con 0x7fd1b0162540 2023-11-24T08:21:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 >> v2:172.21.15.120:3300/0 conn(0x7fd1b01660b0 msgr2=0x7fd1b0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 --2- 172.21.15.107:0/824620003 >> v2:172.21.15.120:3300/0 conn(0x7fd1b01660b0 0x7fd1b0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 >> v1:172.21.15.120:6789/0 conn(0x7fd1b00ccbd0 legacy=0x7fd1b00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0162b30 con 0x7fd1b0162540 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd1a8001b80 con 0x7fd1b0162540 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 >> v1:172.21.15.107:6789/0 conn(0x7fd1b0162540 legacy=0x7fd1b00c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd19c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd19c005180 0x7fd19c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00cc380 con 0x7fd19c0047c0 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c00c080 con 0x7fd19c0039c0 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.668+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c00c240 con 0x7fd19c005180 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd19c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1affff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 -- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd19c005180 msgr2=0x7fd19c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd19c005180 0x7fd19c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 -- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd19c0039c0 msgr2=0x7fd19c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:03.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1a7fff640 1 -- 172.21.15.107:0/824620003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c003520 con 0x7fd19c0047c0 2023-11-24T08:21:03.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1b4c08640 1 -- 172.21.15.107:0/824620003 reap_dead start 2023-11-24T08:21:03.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.669+0000 7fd1affff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.670+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd19c004c40 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fd198009ac0 tx=0x7fd19c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c3486cae8683b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.670+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19802c6d0 con 0x7fd19c0047c0 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.670+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19802c830 con 0x7fd19c0047c0 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.670+0000 7fd1af7fe640 1 -- 172.21.15.107:0/824620003 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19802cbc0 con 0x7fd19c0047c0 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 -- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 msgr2=0x7fd19c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd19c004c40 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fd198009ac0 tx=0x7fd19c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 -- 172.21.15.107:0/824620003 shutdown_connections 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 --2- 172.21.15.107:0/824620003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd19c004c40 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 -- 172.21.15.107:0/824620003 >> 172.21.15.107:0/824620003 conn(0x7fd1b00b0c50 msgr2=0x7fd1b00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 -- 172.21.15.107:0/824620003 shutdown_connections 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.671+0000 7fd1b666b640 1 -- 172.21.15.107:0/824620003 wait complete. 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.673+0000 7fd1b666b640 1 Processor -- start 2023-11-24T08:21:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.673+0000 7fd1b666b640 1 -- start start 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b014b630 0x7fd1b01645a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 0x7fd1b00ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01ec0d0 con 0x7fd19c0047c0 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00ad940 con 0x7fd1b00acf80 2023-11-24T08:21:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1b666b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01eb450 con 0x7fd1b014b630 2023-11-24T08:21:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56294/0 (socket says 172.21.15.107:56294) 2023-11-24T08:21:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.674+0000 7fd1affff640 1 -- 172.21.15.107:0/3219574165 learned_addr learned my addr 172.21.15.107:0/3219574165 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 -- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b014b630 msgr2=0x7fd1b01645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b014b630 0x7fd1b01645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1b4c08640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 0x7fd1b00ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 -- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 msgr2=0x7fd1b00ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 0x7fd1b00ad400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 -- 172.21.15.107:0/3219574165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b01eb730 con 0x7fd19c0047c0 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1b4c08640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 0x7fd1b00ad400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1affff640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000d480 tx=0x7fd1a000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.675+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0005480 con 0x7fd19c0047c0 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.676+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0003610 con 0x7fd19c0047c0 2023-11-24T08:21:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.676+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00039c0 con 0x7fd19c0047c0 2023-11-24T08:21:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.676+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b01eba10 con 0x7fd19c0047c0 2023-11-24T08:21:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.676+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b0164ae0 con 0x7fd19c0047c0 2023-11-24T08:21:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.677+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004e8b0 con 0x7fd19c0047c0 2023-11-24T08:21:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.678+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fd1a0003b20 con 0x7fd19c0047c0 2023-11-24T08:21:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.678+0000 7fd1ad7fa640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 0x7fd19006f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.679+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fd1a00884b0 con 0x7fd19c0047c0 2023-11-24T08:21:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.679+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 0x7fd19006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:03.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.680+0000 7fd1a7fff640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 0x7fd19006f160 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd198036000 tx=0x7fd19802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:03.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.683+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1a0055120 con 0x7fd19c0047c0 2023-11-24T08:21:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.912+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1b0165b90 con 0x7fd19006cca0 2023-11-24T08:21:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.923+0000 7fd1ad7fa640 1 -- 172.21.15.107:0/3219574165 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fd1980040c0 con 0x7fd19006cca0 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:03.924 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:03.925 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:03.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:03.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:03.925 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.925+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 msgr2=0x7fd19006f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.925+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 0x7fd19006f160 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd198036000 tx=0x7fd19802e000 comp rx=0 tx=0).stop 2023-11-24T08:21:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.925+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 msgr2=0x7fd1b01ed580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.925+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000d480 tx=0x7fd1a000d940 comp rx=0 tx=0).stop 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 shutdown_connections 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd19006cca0 0x7fd19006f160 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b00acf80 0x7fd1b00ad400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b014b630 0x7fd1b01645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 --2- 172.21.15.107:0/3219574165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd19c0047c0 0x7fd1b01ed580 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 >> 172.21.15.107:0/3219574165 conn(0x7fd1b00b0c50 msgr2=0x7fd1b00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 shutdown_connections 2023-11-24T08:21:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:03.926+0000 7fd1b666b640 1 -- 172.21.15.107:0/3219574165 wait complete. 2023-11-24T08:21:04.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.176+0000 7fa49e1bc640 1 Processor -- start 2023-11-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.176+0000 7fa49e1bc640 1 -- start start 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 0x7fa4980c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4980ca940 0x7fa4980c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa4980b0e90 0x7fa4980b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4980cb960 con 0x7fa4980ca940 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa498161e40 con 0x7fa4980b0e90 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49e1bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa498162120 con 0x7fa4980cb580 2023-11-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49c9b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4980ca940 0x7fa4980c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49c9b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4980ca940 0x7fa4980c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56306/0 (socket says 172.21.15.107:56306) 2023-11-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49c9b9640 1 -- 172.21.15.107:0/261230611 learned_addr learned my addr 172.21.15.107:0/261230611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49d1ba640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 0x7fa4980c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49c9b9640 1 -- 172.21.15.107:0/261230611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 msgr2=0x7fa4980c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.178+0000 7fa49d9bb640 1 --2- 172.21.15.107:0/261230611 >> v2:172.21.15.120:3300/0 conn(0x7fa4980b0e90 0x7fa4980b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.179+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 0x7fa4980c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.179+0000 7fa49c9b9640 1 -- 172.21.15.107:0/261230611 >> v2:172.21.15.120:3300/0 conn(0x7fa4980b0e90 msgr2=0x7fa4980b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.179+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/261230611 >> v2:172.21.15.120:3300/0 conn(0x7fa4980b0e90 0x7fa4980b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.179+0000 7fa49c9b9640 1 -- 172.21.15.107:0/261230611 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498162400 con 0x7fa4980ca940 2023-11-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.179+0000 7fa49d1ba640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 0x7fa4980c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.180+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980ca940 0x7fa4980c25d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fa498070310 tx=0x7fa48802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6187fa48c68ca9d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.180+0000 7fa48ffff640 1 -- 172.21.15.107:0/261230611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488031870 con 0x7fa4980ca940 2023-11-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.180+0000 7fa48ffff640 1 -- 172.21.15.107:0/261230611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4880319d0 con 0x7fa4980ca940 2023-11-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.180+0000 7fa48ffff640 1 -- 172.21.15.107:0/261230611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488031d60 con 0x7fa4980ca940 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 -- 172.21.15.107:0/261230611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980ca940 msgr2=0x7fa4980c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980ca940 0x7fa4980c25d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fa498070310 tx=0x7fa48802b910 comp rx=0 tx=0).stop 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 -- 172.21.15.107:0/261230611 shutdown_connections 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/261230611 >> v2:172.21.15.120:3300/0 conn(0x7fa4980b0e90 0x7fa4980b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980ca940 0x7fa4980c25d0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/261230611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980cb580 0x7fa4980c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 -- 172.21.15.107:0/261230611 >> 172.21.15.107:0/261230611 conn(0x7fa4980cbb40 msgr2=0x7fa4980cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 -- 172.21.15.107:0/261230611 shutdown_connections 2023-11-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.181+0000 7fa49e1bc640 1 -- 172.21.15.107:0/261230611 wait complete. 2023-11-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.183+0000 7fa49e1bc640 1 Processor -- start 2023-11-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.183+0000 7fa49e1bc640 1 -- start start 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 0x7fa4980b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4980cb580 0x7fa498157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c62c0 con 0x7fa4980b0e90 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c65a0 con 0x7fa4980cb580 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49e1bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c6880 con 0x7fa4980ca940 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49d1ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49d1ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56310/0 (socket says 172.21.15.107:56310) 2023-11-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.184+0000 7fa49d1ba640 1 -- 172.21.15.107:0/3608206006 learned_addr learned my addr 172.21.15.107:0/3608206006 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 0x7fa4980b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d9bb640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4980cb580 0x7fa498157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 -- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 msgr2=0x7fa4980b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 0x7fa4980b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 -- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4980cb580 msgr2=0x7fa498157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4980cb580 0x7fa498157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 -- 172.21.15.107:0/3608206006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880094d0 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 0x7fa4980b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49d1ba640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c7f0 tx=0x7fa49400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494007820 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494007980 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494007cf0 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4980c6b60 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.185+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4980c5570 con 0x7fa4980b0e90 2023-11-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.187+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4980c5760 con 0x7fa4980b0e90 2023-11-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.188+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fa4940113f0 con 0x7fa4980b0e90 2023-11-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.188+0000 7fa48dffb640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 0x7fa47c06f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.189+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fa494088990 con 0x7fa4980b0e90 2023-11-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.189+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 0x7fa47c06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.189+0000 7fa49c9b9640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 0x7fa47c06f160 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.193+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4940554a0 con 0x7fa4980b0e90 2023-11-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.424+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4981629c0 con 0x7fa47c06cca0 2023-11-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.439+0000 7fa48dffb640 1 -- 172.21.15.107:0/3608206006 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fa48803a420 con 0x7fa47c06cca0 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.442+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 msgr2=0x7fa47c06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.442+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 0x7fa47c06f160 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).stop 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.442+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 msgr2=0x7fa4980c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.442+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c7f0 tx=0x7fa49400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 shutdown_connections 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa47c06cca0 0x7fa47c06f160 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4980cb580 0x7fa498157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4980ca940 0x7fa4980b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 --2- 172.21.15.107:0/3608206006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4980b0e90 0x7fa4980c5030 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 >> 172.21.15.107:0/3608206006 conn(0x7fa4980cbb40 msgr2=0x7fa4980ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 shutdown_connections 2023-11-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:05.443+0000 7fa49e1bc640 1 -- 172.21.15.107:0/3608206006 wait complete. 2023-11-24T08:21:06.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.691+0000 7f3f9fc61640 1 Processor -- start 2023-11-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.692+0000 7f3f9fc61640 1 -- start start 2023-11-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.693+0000 7f3f9fc61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f980af000 0x7f3f980b3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.693+0000 7f3f9fc61640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f9806b680 0x7f3f981614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.693+0000 7f3f9fc61640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f981619e0 con 0x7f3f98161060 2023-11-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.693+0000 7f3f9fc61640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980af3e0 con 0x7f3f9806b680 2023-11-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.693+0000 7f3f9fc61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9806dca0 con 0x7f3f980af000 2023-11-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.694+0000 7f3f9e1fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f9806b680 0x7f3f981614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.694+0000 7f3f9d9fd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f980af000 0x7f3f980b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.694+0000 7f3f9e1fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f9806b680 0x7f3f981614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:38882/0 (socket says 172.21.15.107:38882) 2023-11-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.694+0000 7f3f9e1fe640 1 -- 172.21.15.107:0/2269132762 learned_addr learned my addr 172.21.15.107:0/2269132762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.695+0000 7f3f9e1fe640 1 -- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f980af000 msgr2=0x7f3f980b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.695+0000 7f3f9e1fe640 1 --2- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f980af000 0x7f3f980b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.695+0000 7f3f9e1fe640 1 -- 172.21.15.107:0/2269132762 >> v1:172.21.15.120:6789/0 conn(0x7f3f98161060 legacy=0x7f3f9815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.695+0000 7f3f9e1fe640 1 -- 172.21.15.107:0/2269132762 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9806df40 con 0x7f3f9806b680 2023-11-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.696+0000 7f3f9e1fe640 1 --2- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9806b680 0x7f3f981614a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009870 tx=0x7f3f9002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8f918fb8fcd49d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.696+0000 7f3f9c9fb640 1 -- 172.21.15.107:0/2269132762 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90030840 con 0x7f3f9806b680 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9c9fb640 1 -- 172.21.15.107:0/2269132762 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f900309a0 con 0x7f3f9806b680 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9c9fb640 1 -- 172.21.15.107:0/2269132762 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90030d30 con 0x7f3f9806b680 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 -- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9806b680 msgr2=0x7f3f981614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9806b680 0x7f3f981614a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009870 tx=0x7f3f9002b980 comp rx=0 tx=0).stop 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 -- 172.21.15.107:0/2269132762 shutdown_connections 2023-11-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9806b680 0x7f3f981614a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/2269132762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f980af000 0x7f3f980b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 -- 172.21.15.107:0/2269132762 >> 172.21.15.107:0/2269132762 conn(0x7f3f980cf900 msgr2=0x7f3f980cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.697+0000 7f3f9fc61640 1 -- 172.21.15.107:0/2269132762 shutdown_connections 2023-11-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.698+0000 7f3f9fc61640 1 -- 172.21.15.107:0/2269132762 wait complete. 2023-11-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.700+0000 7f3f9fc61640 1 Processor -- start 2023-11-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.700+0000 7f3f9fc61640 1 -- start start 2023-11-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f980af000 0x7f3f98152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f98161060 0x7f3f98155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f981f3440 con 0x7f3f9806b680 2023-11-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f981536b0 con 0x7f3f980af000 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9fc61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f98153990 con 0x7f3f98161060 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.701+0000 7f3f9d9fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56318/0 (socket says 172.21.15.107:56318) 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 -- 172.21.15.107:0/382815662 learned_addr learned my addr 172.21.15.107:0/382815662 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d1fc640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f980af000 0x7f3f98152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9e1fe640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f98161060 0x7f3f98155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 -- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f98161060 msgr2=0x7f3f98155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f98161060 0x7f3f98155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 -- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f980af000 msgr2=0x7f3f98152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f980af000 0x7f3f98152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 -- 172.21.15.107:0/382815662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f900094d0 con 0x7f3f9806b680 2023-11-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.702+0000 7f3f9d9fd640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800efc0 tx=0x7f3f8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.703+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88014480 con 0x7f3f9806b680 2023-11-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.703+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8800f040 con 0x7f3f9806b680 2023-11-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.703+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f98153c70 con 0x7f3f9806b680 2023-11-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.703+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88014800 con 0x7f3f9806b680 2023-11-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.703+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f98151f60 con 0x7f3f9806b680 2023-11-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.704+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9804e8b0 con 0x7f3f9806b680 2023-11-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.705+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f3f88014960 con 0x7f3f9806b680 2023-11-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.706+0000 7f3f967fc640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 0x7f3f8406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.706+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f3f88089210 con 0x7f3f9806b680 2023-11-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.706+0000 7f3f9d1fc640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 0x7f3f8406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.707+0000 7f3f9d1fc640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 0x7f3f8406f160 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f3f9815cea0 tx=0x7f3f8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.710+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f88055dc0 con 0x7f3f9806b680 2023-11-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.932+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f9806e7e0 con 0x7f3f8406cca0 2023-11-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.945+0000 7f3f967fc640 1 -- 172.21.15.107:0/382815662 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f3f8c001330 con 0x7f3f8406cca0 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 msgr2=0x7f3f8406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 0x7f3f8406f160 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f3f9815cea0 tx=0x7f3f8c009040 comp rx=0 tx=0).stop 2023-11-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 msgr2=0x7f3f98151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800efc0 tx=0x7f3f8800c490 comp rx=0 tx=0).stop 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 shutdown_connections 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f8406cca0 0x7f3f8406f160 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f98161060 0x7f3f98155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f980af000 0x7f3f98152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 --2- 172.21.15.107:0/382815662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f9806b680 0x7f3f98151a20 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 >> 172.21.15.107:0/382815662 conn(0x7f3f980cf900 msgr2=0x7f3f9815f580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 shutdown_connections 2023-11-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:06.948+0000 7f3f9fc61640 1 -- 172.21.15.107:0/382815662 wait complete. 2023-11-24T08:21:07.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.187+0000 7fb1a359e640 1 Processor -- start 2023-11-24T08:21:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.187+0000 7fb1a359e640 1 -- start start 2023-11-24T08:21:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1a415d3f0 0x7fb1a40d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1a415d7d0 0x7fb1a40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40d15e0 con 0x7fb1a40d0e40 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a40d0280 con 0x7fb1a415d3f0 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a40d1300 con 0x7fb1a415d7d0 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1a415d3f0 0x7fb1a40d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1a415d3f0 0x7fb1a40d1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60278/0 (socket says 172.21.15.107:60278) 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.189+0000 7fb1a259c640 1 -- 172.21.15.107:0/3667592748 learned_addr learned my addr 172.21.15.107:0/3667592748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.190+0000 7fb1a259c640 1 -- 172.21.15.107:0/3667592748 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415d7d0 msgr2=0x7fb1a40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.190+0000 7fb1a259c640 1 --2- 172.21.15.107:0/3667592748 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415d7d0 0x7fb1a40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.190+0000 7fb1a259c640 1 -- 172.21.15.107:0/3667592748 >> v1:172.21.15.120:6789/0 conn(0x7fb1a40d0e40 legacy=0x7fb1a40afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.190+0000 7fb1a259c640 1 -- 172.21.15.107:0/3667592748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40cfc30 con 0x7fb1a415d3f0 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.191+0000 7fb1a259c640 1 --2- 172.21.15.107:0/3667592748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40d1830 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4067bd0 tx=0x7fb19402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7831a85727813354 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.191+0000 7fb1a159a640 1 -- 172.21.15.107:0/3667592748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19402ca80 con 0x7fb1a415d3f0 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.191+0000 7fb1a159a640 1 -- 172.21.15.107:0/3667592748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19402cbe0 con 0x7fb1a415d3f0 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.191+0000 7fb1a159a640 1 -- 172.21.15.107:0/3667592748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1940343f0 con 0x7fb1a415d3f0 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 -- 172.21.15.107:0/3667592748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 msgr2=0x7fb1a40d1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3667592748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40d1830 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4067bd0 tx=0x7fb19402bee0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 -- 172.21.15.107:0/3667592748 shutdown_connections 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3667592748 >> v2:172.21.15.120:3300/0 conn(0x7fb1a415d7d0 0x7fb1a40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3667592748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40d1830 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 -- 172.21.15.107:0/3667592748 >> 172.21.15.107:0/3667592748 conn(0x7fb1a409c5b0 msgr2=0x7fb1a40d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:08.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 -- 172.21.15.107:0/3667592748 shutdown_connections 2023-11-24T08:21:08.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.192+0000 7fb1a359e640 1 -- 172.21.15.107:0/3667592748 wait complete. 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.194+0000 7fb1a359e640 1 Processor -- start 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.194+0000 7fb1a359e640 1 -- start start 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40d0e40 0x7fb1a4152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1a415d7d0 0x7fb1a40c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4152c40 con 0x7fb1a415d3f0 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40c9250 con 0x7fb1a415d7d0 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40c9530 con 0x7fb1a40d0e40 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60284/0 (socket says 172.21.15.107:60284) 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 -- 172.21.15.107:0/3122941161 learned_addr learned my addr 172.21.15.107:0/3122941161 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 -- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40d0e40 msgr2=0x7fb1a4152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40d0e40 0x7fb1a4152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 -- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1a415d7d0 msgr2=0x7fb1a40c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.195+0000 7fb1a1d9b640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1a415d7d0 0x7fb1a40c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb1a1d9b640 1 -- 172.21.15.107:0/3122941161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb194009b30 con 0x7fb1a415d3f0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb1a1d9b640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00b410 tx=0x7fb18c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c011830 con 0x7fb1a415d3f0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c0119b0 con 0x7fb1a415d3f0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c011d60 con 0x7fb1a415d3f0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a40c9810 con 0x7fb1a415d3f0 2023-11-24T08:21:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.196+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a40c8f40 con 0x7fb1a415d3f0 2023-11-24T08:21:08.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.197+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a40d06f0 con 0x7fb1a415d3f0 2023-11-24T08:21:08.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.198+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fb18c01a400 con 0x7fb1a415d3f0 2023-11-24T08:21:08.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.200+0000 7fb19affd640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 0x7fb18406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:08.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.200+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb18c088710 con 0x7fb1a415d3f0 2023-11-24T08:21:08.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.200+0000 7fb1a259c640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 0x7fb18406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:08.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.201+0000 7fb1a259c640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 0x7fb18406f160 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb194035000 tx=0x7fb19402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.204+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1a40d06f0 con 0x7fb1a415d3f0 2023-11-24T08:21:08.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.431+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1a40c93c0 con 0x7fb18406cca0 2023-11-24T08:21:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.444+0000 7fb19affd640 1 -- 172.21.15.107:0/3122941161 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7fb1a40c93c0 con 0x7fb18406cca0 2023-11-24T08:21:08.445 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:08.446 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 msgr2=0x7fb18406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 0x7fb18406f160 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb194035000 tx=0x7fb19402e000 comp rx=0 tx=0).stop 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 msgr2=0x7fb1a40c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00b410 tx=0x7fb18c00b8d0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 shutdown_connections 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb18406cca0 0x7fb18406f160 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1a415d7d0 0x7fb1a40c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1a415d3f0 0x7fb1a40c6da0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 --2- 172.21.15.107:0/3122941161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1a40d0e40 0x7fb1a4152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 >> 172.21.15.107:0/3122941161 conn(0x7fb1a409c5b0 msgr2=0x7fb1a40b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 shutdown_connections 2023-11-24T08:21:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:08.447+0000 7fb1a359e640 1 -- 172.21.15.107:0/3122941161 wait complete. 2023-11-24T08:21:09.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.694+0000 7f88edba0640 1 Processor -- start 2023-11-24T08:21:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.694+0000 7f88edba0640 1 -- start start 2023-11-24T08:21:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 0x7f88e806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e80b0b40 0x7f88e80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815ded0 0x7f88e815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e806d430 con 0x7f88e80b0b40 2023-11-24T08:21:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e80b14e0 con 0x7f88e815a1c0 2023-11-24T08:21:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88edba0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e815ec50 con 0x7f88e815ded0 2023-11-24T08:21:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.695+0000 7f88e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e80b0b40 0x7f88e80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88e80b0b40 0x7f88e80b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60292/0 (socket says 172.21.15.107:60292) 2023-11-24T08:21:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 -- 172.21.15.107:0/1514229712 learned_addr learned my addr 172.21.15.107:0/1514229712 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88ed39f640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815ded0 0x7f88e815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88ecb9e640 1 --2- 172.21.15.107:0/1514229712 >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 0x7f88e806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 -- 172.21.15.107:0/1514229712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815ded0 msgr2=0x7f88e815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815ded0 0x7f88e815e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 -- 172.21.15.107:0/1514229712 >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 msgr2=0x7f88e806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 --2- 172.21.15.107:0/1514229712 >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 0x7f88e806d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88e7fff640 1 -- 172.21.15.107:0/1514229712 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e806cb00 con 0x7f88e80b0b40 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.696+0000 7f88ecb9e640 1 --2- 172.21.15.107:0/1514229712 >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 0x7f88e806d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.697+0000 7f88e7fff640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e80b0fa0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f88e80cca80 tx=0x7f88d402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8ab81a124f357a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.697+0000 7f88e77fe640 1 -- 172.21.15.107:0/1514229712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4031860 con 0x7f88e80b0b40 2023-11-24T08:21:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88e77fe640 1 -- 172.21.15.107:0/1514229712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d40319c0 con 0x7f88e80b0b40 2023-11-24T08:21:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88e77fe640 1 -- 172.21.15.107:0/1514229712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4031d50 con 0x7f88e80b0b40 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 -- 172.21.15.107:0/1514229712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 msgr2=0x7f88e80b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e80b0fa0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f88e80cca80 tx=0x7f88d402b900 comp rx=0 tx=0).stop 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 -- 172.21.15.107:0/1514229712 shutdown_connections 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815ded0 0x7f88e815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 --2- 172.21.15.107:0/1514229712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e80b0fa0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 --2- 172.21.15.107:0/1514229712 >> v2:172.21.15.120:3300/0 conn(0x7f88e815a1c0 0x7f88e806d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 -- 172.21.15.107:0/1514229712 >> 172.21.15.107:0/1514229712 conn(0x7f88e80cbb40 msgr2=0x7f88e80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 -- 172.21.15.107:0/1514229712 shutdown_connections 2023-11-24T08:21:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.698+0000 7f88edba0640 1 -- 172.21.15.107:0/1514229712 wait complete. 2023-11-24T08:21:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.700+0000 7f88edba0640 1 Processor -- start 2023-11-24T08:21:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.700+0000 7f88edba0640 1 -- start start 2023-11-24T08:21:09.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815a1c0 0x7f88e80b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e815ded0 0x7f88e80b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80c8ff0 con 0x7f88e80b0b40 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e814bbf0 con 0x7f88e815ded0 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88edba0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80c8370 con 0x7f88e815a1c0 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88ecb9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88ecb9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60308/0 (socket says 172.21.15.107:60308) 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.701+0000 7f88ecb9e640 1 -- 172.21.15.107:0/3414892396 learned_addr learned my addr 172.21.15.107:0/3414892396 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 -- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815a1c0 msgr2=0x7f88e80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88e7fff640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815a1c0 0x7f88e80b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815a1c0 0x7f88e80b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 -- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e815ded0 msgr2=0x7f88e80b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ed39f640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e815ded0 0x7f88e80b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e815ded0 0x7f88e80b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 -- 172.21.15.107:0/3414892396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d40094d0 con 0x7f88e80b0b40 2023-11-24T08:21:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88ecb9e640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c7f0 tx=0x7f88dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00cea0 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc007a30 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e80c8650 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.702+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc007dc0 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.703+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e80c8c10 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.704+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e804e840 con 0x7f88e80b0b40 2023-11-24T08:21:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.705+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f88dc0113f0 con 0x7f88e80b0b40 2023-11-24T08:21:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.706+0000 7f88e57fa640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 0x7f88cc06f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.706+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f88dc088810 con 0x7f88e80b0b40 2023-11-24T08:21:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.706+0000 7f88e7fff640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 0x7f88cc06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.707+0000 7f88e7fff640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 0x7f88cc06f160 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f88d4036000 tx=0x7f88d403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.710+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88dc055320 con 0x7f88e80b0b40 2023-11-24T08:21:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.938+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f88e80cec00 con 0x7f88cc06cca0 2023-11-24T08:21:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.946+0000 7f88e57fa640 1 -- 172.21.15.107:0/3414892396 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f88e80cec00 con 0x7f88cc06cca0 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:09.948 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 msgr2=0x7f88cc06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:09.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 0x7f88cc06f160 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f88d4036000 tx=0x7f88d403c040 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 msgr2=0x7f88e814b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c7f0 tx=0x7f88dc00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 shutdown_connections 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f88cc06cca0 0x7f88cc06f160 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e815ded0 0x7f88e80b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e815a1c0 0x7f88e80b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 --2- 172.21.15.107:0/3414892396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88e80b0b40 0x7f88e814b6b0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 >> 172.21.15.107:0/3414892396 conn(0x7f88e80cbb40 msgr2=0x7f88e80ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.949+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 shutdown_connections 2023-11-24T08:21:09.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:09.950+0000 7f88edba0640 1 -- 172.21.15.107:0/3414892396 wait complete. 2023-11-24T08:21:10.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.186+0000 7f14dbd41640 1 Processor -- start 2023-11-24T08:21:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.186+0000 7f14dbd41640 1 -- start start 2023-11-24T08:21:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14dbd41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14d415b440 0x7f14d40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14dbd41640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 0x7f14d406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14dbd41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40c5dc0 con 0x7f14d40cbf20 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14dbd41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40cc3d0 con 0x7f14d415b440 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14dbd41640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d415b820 con 0x7f14d40c59b0 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14d9add640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14d415b440 0x7f14d40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14d9add640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14d415b440 0x7f14d40cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60318/0 (socket says 172.21.15.107:60318) 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14d9add640 1 -- 172.21.15.107:0/1605399737 learned_addr learned my addr 172.21.15.107:0/1605399737 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.188+0000 7f14da2de640 1 --2- 172.21.15.107:0/1605399737 >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 0x7f14d406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.189+0000 7f14d9add640 1 -- 172.21.15.107:0/1605399737 >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 msgr2=0x7f14d406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.189+0000 7f14d9add640 1 --2- 172.21.15.107:0/1605399737 >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 0x7f14d406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.189+0000 7f14d9add640 1 -- 172.21.15.107:0/1605399737 >> v1:172.21.15.120:6789/0 conn(0x7f14d40cbf20 legacy=0x7f14d40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.189+0000 7f14d9add640 1 -- 172.21.15.107:0/1605399737 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d415eea0 con 0x7f14d415b440 2023-11-24T08:21:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.189+0000 7f14da2de640 1 --2- 172.21.15.107:0/1605399737 >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 0x7f14d406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.190+0000 7f14d9add640 1 --2- 172.21.15.107:0/1605399737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d40cb9e0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f14d40cb910 tx=0x7f14d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2fd823ff6f480c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.190+0000 7f14d8adb640 1 -- 172.21.15.107:0/1605399737 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00317e0 con 0x7f14d415b440 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.190+0000 7f14d8adb640 1 -- 172.21.15.107:0/1605399737 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d0031940 con 0x7f14d415b440 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.190+0000 7f14d8adb640 1 -- 172.21.15.107:0/1605399737 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0031cd0 con 0x7f14d415b440 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 -- 172.21.15.107:0/1605399737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 msgr2=0x7f14d40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 --2- 172.21.15.107:0/1605399737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d40cb9e0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f14d40cb910 tx=0x7f14d002b880 comp rx=0 tx=0).stop 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 -- 172.21.15.107:0/1605399737 shutdown_connections 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 --2- 172.21.15.107:0/1605399737 >> v2:172.21.15.120:3300/0 conn(0x7f14d40c59b0 0x7f14d406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 --2- 172.21.15.107:0/1605399737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d40cb9e0 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 -- 172.21.15.107:0/1605399737 >> 172.21.15.107:0/1605399737 conn(0x7f14d40b0c50 msgr2=0x7f14d40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:11.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 -- 172.21.15.107:0/1605399737 shutdown_connections 2023-11-24T08:21:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.191+0000 7f14dbd41640 1 -- 172.21.15.107:0/1605399737 wait complete. 2023-11-24T08:21:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.193+0000 7f14dbd41640 1 Processor -- start 2023-11-24T08:21:11.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.193+0000 7f14dbd41640 1 -- start start 2023-11-24T08:21:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14d40c59b0 0x7f14d40b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 0x7f14d4150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40b4510 con 0x7f14d415b440 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4150dc0 con 0x7f14d40cbf20 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14dbd41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d415bb40 con 0x7f14d40c59b0 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14da2de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14da2de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60326/0 (socket says 172.21.15.107:60326) 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.194+0000 7f14da2de640 1 -- 172.21.15.107:0/2038460404 learned_addr learned my addr 172.21.15.107:0/2038460404 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14d92dc640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 0x7f14d4150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14d9add640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14d40c59b0 0x7f14d40b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 -- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14d40c59b0 msgr2=0x7f14d40b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14d40c59b0 0x7f14d40b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 -- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 msgr2=0x7f14d4150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 0x7f14d4150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 -- 172.21.15.107:0/2038460404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00094d0 con 0x7f14d415b440 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14d92dc640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 0x7f14d4150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14da2de640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00cef0 tx=0x7f14cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00d840 con 0x7f14d415b440 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc00d9c0 con 0x7f14d415b440 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00dd70 con 0x7f14d415b440 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.195+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d415be20 con 0x7f14d415b440 2023-11-24T08:21:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.196+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d415c3e0 con 0x7f14d415b440 2023-11-24T08:21:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.197+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d415c600 con 0x7f14d415b440 2023-11-24T08:21:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.198+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f14cc01e3b0 con 0x7f14d415b440 2023-11-24T08:21:11.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.199+0000 7f14ca7fc640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 0x7f14b806f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:11.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.199+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f14cc089160 con 0x7f14d415b440 2023-11-24T08:21:11.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.199+0000 7f14d9add640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 0x7f14b806f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:11.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.200+0000 7f14d9add640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 0x7f14b806f160 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f14d0036000 tx=0x7f14d003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:11.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.204+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14cc055d30 con 0x7f14d415b440 2023-11-24T08:21:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.429+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f14d415f3d0 con 0x7f14b806cca0 2023-11-24T08:21:11.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.442+0000 7f14ca7fc640 1 -- 172.21.15.107:0/2038460404 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+167 (secure 0 0 0) 0x7f14d0031c30 con 0x7f14b806cca0 2023-11-24T08:21:11.444 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "in-progress", 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol7968", 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap3729" 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:11.445 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 msgr2=0x7f14b806f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 0x7f14b806f160 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f14d0036000 tx=0x7f14d003d040 comp rx=0 tx=0).stop 2023-11-24T08:21:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 msgr2=0x7f14d4155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00cef0 tx=0x7f14cc00a710 comp rx=0 tx=0).stop 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 shutdown_connections 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14b806cca0 0x7f14b806f160 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14d415b440 0x7f14d4155b30 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d40cbf20 0x7f14d4150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 --2- 172.21.15.107:0/2038460404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14d40c59b0 0x7f14d40b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 >> 172.21.15.107:0/2038460404 conn(0x7f14d40b0c50 msgr2=0x7f14d40cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.446+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 shutdown_connections 2023-11-24T08:21:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:11.447+0000 7f14dbd41640 1 -- 172.21.15.107:0/2038460404 wait complete. 2023-11-24T08:21:12.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone404 2023-11-24T08:21:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.691+0000 7ff4cc54c640 1 Processor -- start 2023-11-24T08:21:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.691+0000 7ff4cc54c640 1 -- start start 2023-11-24T08:21:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c4072160 0x7ff4c415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff4c40c49f0 0x7ff4c406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c40cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c406e0d0 con 0x7ff4c4072160 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c415ec90 con 0x7ff4c40c49f0 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.692+0000 7ff4cc54c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c40cd530 con 0x7ff4c406dc40 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c4072160 0x7ff4c415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c4072160 0x7ff4c415e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60342/0 (socket says 172.21.15.107:60342) 2023-11-24T08:21:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 -- 172.21.15.107:0/3286177112 learned_addr learned my addr 172.21.15.107:0/3286177112 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4c9ae7640 1 --2- 172.21.15.107:0/3286177112 >> v2:172.21.15.120:3300/0 conn(0x7ff4c40c49f0 0x7ff4c406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4caae9640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c40cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 -- 172.21.15.107:0/3286177112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 msgr2=0x7ff4c40cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c40cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 -- 172.21.15.107:0/3286177112 >> v2:172.21.15.120:3300/0 conn(0x7ff4c40c49f0 msgr2=0x7ff4c406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/3286177112 >> v2:172.21.15.120:3300/0 conn(0x7ff4c40c49f0 0x7ff4c406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.693+0000 7ff4ca2e8640 1 -- 172.21.15.107:0/3286177112 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40c4ed0 con 0x7ff4c4072160 2023-11-24T08:21:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.695+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c4072160 0x7ff4c415e750 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40cef60 tx=0x7ff4c002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a040b0416d0a4ae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.695+0000 7ff4c92e6640 1 -- 172.21.15.107:0/3286177112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0031860 con 0x7ff4c4072160 2023-11-24T08:21:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.695+0000 7ff4c92e6640 1 -- 172.21.15.107:0/3286177112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c00319c0 con 0x7ff4c4072160 2023-11-24T08:21:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.695+0000 7ff4c92e6640 1 -- 172.21.15.107:0/3286177112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0031d50 con 0x7ff4c4072160 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 -- 172.21.15.107:0/3286177112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c4072160 msgr2=0x7ff4c415e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c4072160 0x7ff4c415e750 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40cef60 tx=0x7ff4c002b900 comp rx=0 tx=0).stop 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 -- 172.21.15.107:0/3286177112 shutdown_connections 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c40cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/3286177112 >> v2:172.21.15.120:3300/0 conn(0x7ff4c40c49f0 0x7ff4c406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/3286177112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c4072160 0x7ff4c415e750 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 -- 172.21.15.107:0/3286177112 >> 172.21.15.107:0/3286177112 conn(0x7ff4c40b0c50 msgr2=0x7ff4c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 -- 172.21.15.107:0/3286177112 shutdown_connections 2023-11-24T08:21:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.696+0000 7ff4cc54c640 1 -- 172.21.15.107:0/3286177112 wait complete. 2023-11-24T08:21:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.698+0000 7ff4cc54c640 1 Processor -- start 2023-11-24T08:21:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.698+0000 7ff4cc54c640 1 -- start start 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c4155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c4072160 0x7ff4c41574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c409c760 con 0x7ff4c40c49f0 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4156430 con 0x7ff4c4072160 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4cc54c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4156710 con 0x7ff4c406dc40 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4caae9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4caae9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60348/0 (socket says 172.21.15.107:60348) 2023-11-24T08:21:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.699+0000 7ff4caae9640 1 -- 172.21.15.107:0/2605899549 learned_addr learned my addr 172.21.15.107:0/2605899549 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4c9ae7640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c4072160 0x7ff4c41574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c4155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 -- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 msgr2=0x7ff4c4155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c4155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 -- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c4072160 msgr2=0x7ff4c41574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c4072160 0x7ff4c41574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 -- 172.21.15.107:0/2605899549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00094d0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4caae9640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc007e60 tx=0x7ff4bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc014620 con 0x7ff4c40c49f0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc00ee90 con 0x7ff4c40c49f0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0149e0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c41569f0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.700+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c4156fb0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.702+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c404e8b0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.703+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7ff4bc014b40 con 0x7ff4c40c49f0 2023-11-24T08:21:12.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.704+0000 7ff4baffd640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 0x7ff4a806f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:12.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.704+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff4bc088f40 con 0x7ff4c40c49f0 2023-11-24T08:21:12.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.704+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 0x7ff4a806f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:12.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.705+0000 7ff4ca2e8640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 0x7ff4a806f160 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0036000 tx=0x7ff4c003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.708+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4bc055af0 con 0x7ff4c40c49f0 2023-11-24T08:21:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:12.937+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4c40c5770 con 0x7ff4a806cca0 2023-11-24T08:21:15.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:21:15.713 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.892+0000 7ff4baffd640 1 -- 172.21.15.107:0/2605899549 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7ff4c003a420 con 0x7ff4a806cca0 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:17.894 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:17.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 msgr2=0x7ff4a806f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:17.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 0x7ff4a806f160 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0036000 tx=0x7ff4c003c040 comp rx=0 tx=0).stop 2023-11-24T08:21:17.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 msgr2=0x7ff4c409c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:17.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc007e60 tx=0x7ff4bc00e4b0 comp rx=0 tx=0).stop 2023-11-24T08:21:17.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 shutdown_connections 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4a806cca0 0x7ff4a806f160 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c49f0 0x7ff4c409c220 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4c4072160 0x7ff4c41574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 --2- 172.21.15.107:0/2605899549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4c406dc40 0x7ff4c4155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 >> 172.21.15.107:0/2605899549 conn(0x7ff4c40b0c50 msgr2=0x7ff4c40adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.895+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 shutdown_connections 2023-11-24T08:21:17.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:17.896+0000 7ff4cc54c640 1 -- 172.21.15.107:0/2605899549 wait complete. 2023-11-24T08:21:17.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol7968 2023-11-24T08:21:18.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.136+0000 7f7fbf3dc640 1 Processor -- start 2023-11-24T08:21:18.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.136+0000 7f7fbf3dc640 1 -- start start 2023-11-24T08:21:18.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.138+0000 7f7fbf3dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7fb806de10 0x7f7fb815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.138+0000 7f7fbf3dc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7fb80c49f0 0x7f7fb80705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.138+0000 7f7fbf3dc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb815ec90 con 0x7f7fb8072160 2023-11-24T08:21:18.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.138+0000 7f7fbf3dc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80cd530 con 0x7f7fb806de10 2023-11-24T08:21:18.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.138+0000 7f7fbf3dc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80c4ed0 con 0x7f7fb80c49f0 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7fb806de10 0x7f7fb815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbd979640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7fb80c49f0 0x7f7fb80705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7fb806de10 0x7f7fb815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44628/0 (socket says 172.21.15.107:44628) 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 -- 172.21.15.107:0/2283333153 learned_addr learned my addr 172.21.15.107:0/2283333153 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 -- 172.21.15.107:0/2283333153 >> v2:172.21.15.120:3300/0 conn(0x7f7fb80c49f0 msgr2=0x7f7fb80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 --2- 172.21.15.107:0/2283333153 >> v2:172.21.15.120:3300/0 conn(0x7f7fb80c49f0 0x7f7fb80705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 -- 172.21.15.107:0/2283333153 >> v1:172.21.15.120:6789/0 conn(0x7f7fb8072160 legacy=0x7f7fb806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.139+0000 7f7fbc977640 1 -- 172.21.15.107:0/2283333153 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80c51b0 con 0x7f7fb806de10 2023-11-24T08:21:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.140+0000 7f7fbc977640 1 --2- 172.21.15.107:0/2283333153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb815e750 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8072a30 tx=0x7f7fa802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb22f09f79571b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.140+0000 7f7faffff640 1 -- 172.21.15.107:0/2283333153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8030860 con 0x7f7fb806de10 2023-11-24T08:21:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7faffff640 1 -- 172.21.15.107:0/2283333153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa80309c0 con 0x7f7fb806de10 2023-11-24T08:21:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7faffff640 1 -- 172.21.15.107:0/2283333153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8030d50 con 0x7f7fb806de10 2023-11-24T08:21:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/2283333153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 msgr2=0x7f7fb815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/2283333153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb815e750 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8072a30 tx=0x7f7fa802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/2283333153 shutdown_connections 2023-11-24T08:21:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/2283333153 >> v2:172.21.15.120:3300/0 conn(0x7f7fb80c49f0 0x7f7fb80705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/2283333153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb815e750 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/2283333153 >> 172.21.15.107:0/2283333153 conn(0x7f7fb80b0c50 msgr2=0x7f7fb80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/2283333153 shutdown_connections 2023-11-24T08:21:18.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.141+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/2283333153 wait complete. 2023-11-24T08:21:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.143+0000 7f7fbf3dc640 1 Processor -- start 2023-11-24T08:21:18.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.143+0000 7f7fbf3dc640 1 -- start start 2023-11-24T08:21:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb8072160 0x7f7fb80ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb80c49f0 0x7f7fb80ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80b4980 con 0x7f7fb806de10 2023-11-24T08:21:18.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815ac20 con 0x7f7fb80c49f0 2023-11-24T08:21:18.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbf3dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815a610 con 0x7f7fb8072160 2023-11-24T08:21:18.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbd178640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbd178640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44630/0 (socket says 172.21.15.107:44630) 2023-11-24T08:21:18.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.144+0000 7f7fbd178640 1 -- 172.21.15.107:0/3091318938 learned_addr learned my addr 172.21.15.107:0/3091318938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 -- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb8072160 msgr2=0x7f7fb80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb8072160 0x7f7fb80ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 -- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb80c49f0 msgr2=0x7f7fb80ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd979640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb80c49f0 0x7f7fb80ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb80c49f0 0x7f7fb80ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 -- 172.21.15.107:0/3091318938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa80094d0 con 0x7f7fb806de10 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fbd178640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c450 tx=0x7f7fb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40052d0 con 0x7f7fb806de10 2023-11-24T08:21:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4005430 con 0x7f7fb806de10 2023-11-24T08:21:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.145+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4012870 con 0x7f7fb806de10 2023-11-24T08:21:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.146+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb80b46a0 con 0x7f7fb806de10 2023-11-24T08:21:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.146+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81563a0 con 0x7f7fb806de10 2023-11-24T08:21:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.147+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb815a470 con 0x7f7fb806de10 2023-11-24T08:21:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.148+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f7fb40129d0 con 0x7f7fb806de10 2023-11-24T08:21:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.149+0000 7f7fadffb640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 0x7f7f9c06f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.149+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7fb4088030 con 0x7f7fb806de10 2023-11-24T08:21:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.149+0000 7f7fbc977640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 0x7f7f9c06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.150+0000 7f7fbc977640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 0x7f7f9c06f160 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8036000 tx=0x7f7fa803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.154+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fb815a470 con 0x7f7fb806de10 2023-11-24T08:21:18.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.372+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol7968", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fb80c5a50 con 0x7f7f9c06cca0 2023-11-24T08:21:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.389+0000 7f7fadffb640 1 -- 172.21.15.107:0/3091318938 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f7fa8030bb0 con 0x7f7f9c06cca0 2023-11-24T08:21:18.391 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol7968/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:21:18.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 msgr2=0x7f7f9c06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 0x7f7f9c06f160 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8036000 tx=0x7f7fa803b040 comp rx=0 tx=0).stop 2023-11-24T08:21:18.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 msgr2=0x7f7fb814ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c450 tx=0x7f7fb400c910 comp rx=0 tx=0).stop 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 shutdown_connections 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f9c06cca0 0x7f7f9c06f160 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb80c49f0 0x7f7fb80ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb8072160 0x7f7fb80ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 --2- 172.21.15.107:0/3091318938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb806de10 0x7f7fb814ee50 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 >> 172.21.15.107:0/3091318938 conn(0x7f7fb80b0c50 msgr2=0x7f7fb80c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 shutdown_connections 2023-11-24T08:21:18.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.392+0000 7f7fbf3dc640 1 -- 172.21.15.107:0/3091318938 wait complete. 2023-11-24T08:21:18.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone404 2023-11-24T08:21:18.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.616+0000 7f04f413c640 1 Processor -- start 2023-11-24T08:21:18.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.617+0000 7f04f413c640 1 -- start start 2023-11-24T08:21:18.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.618+0000 7f04f413c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04ec0ae9b0 0x7f04ec161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.618+0000 7f04f413c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec164440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.618+0000 7f04f413c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec15a810 con 0x7f04ec0b38a0 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.618+0000 7f04f413c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec06eb90 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.618+0000 7f04f413c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec06ee70 con 0x7f04ec06d900 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04ec0ae9b0 0x7f04ec161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04ec0ae9b0 0x7f04ec161ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44632/0 (socket says 172.21.15.107:44632) 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f1ed8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04ec0b38a0 0x7f04ec0b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55176/0 (socket says 172.21.15.107:55176) 2023-11-24T08:21:18.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 -- 172.21.15.107:0/4008874673 learned_addr learned my addr 172.21.15.107:0/4008874673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f26d9640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 -- 172.21.15.107:0/4008874673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 msgr2=0x7f04ec164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 -- 172.21.15.107:0/4008874673 >> v1:172.21.15.107:6789/0 conn(0x7f04ec0b38a0 legacy=0x7f04ec0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f16d7640 1 -- 172.21.15.107:0/4008874673 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec06f150 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.619+0000 7f04f0ed6640 1 -- 172.21.15.107:0/4008874673 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171154310 0 0) 0x7f04ec15a810 con 0x7f04ec0b38a0 2023-11-24T08:21:18.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.621+0000 7f04f16d7640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec161ef0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009870 tx=0x7f04d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fb53f7209ef9ce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.621+0000 7f04f0ed6640 1 -- 172.21.15.107:0/4008874673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8031640 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.621+0000 7f04f0ed6640 1 -- 172.21.15.107:0/4008874673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d80317a0 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.621+0000 7f04f0ed6640 1 -- 172.21.15.107:0/4008874673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8031b30 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 -- 172.21.15.107:0/4008874673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 msgr2=0x7f04ec161ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec161ef0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009870 tx=0x7f04d802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 -- 172.21.15.107:0/4008874673 shutdown_connections 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec164440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 --2- 172.21.15.107:0/4008874673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec161ef0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 -- 172.21.15.107:0/4008874673 >> 172.21.15.107:0/4008874673 conn(0x7f04ec06c090 msgr2=0x7f04ec0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 -- 172.21.15.107:0/4008874673 shutdown_connections 2023-11-24T08:21:18.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.622+0000 7f04f413c640 1 -- 172.21.15.107:0/4008874673 wait complete. 2023-11-24T08:21:18.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.624+0000 7f04f413c640 1 Processor -- start 2023-11-24T08:21:18.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.624+0000 7f04f413c640 1 -- start start 2023-11-24T08:21:18.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04ec0b38a0 0x7f04ec1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec09ab40 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec147420 con 0x7f04ec0b38a0 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f413c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec146810 con 0x7f04ec06d900 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f16d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f16d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44646/0 (socket says 172.21.15.107:44646) 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f16d7640 1 -- 172.21.15.107:0/3527104918 learned_addr learned my addr 172.21.15.107:0/3527104918 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f26d9640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04ec0b38a0 0x7f04ec1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.625+0000 7f04f1ed8640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 -- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 msgr2=0x7f04ec146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 -- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04ec0b38a0 msgr2=0x7f04ec1478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04ec0b38a0 0x7f04ec1478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 -- 172.21.15.107:0/3527104918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d80094d0 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f1ed8640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:18.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f16d7640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f04d8000c00 tx=0x7f04d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d803ac60 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d803adc0 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8044630 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.626+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec146af0 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.627+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1470b0 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.628+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec147590 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.629+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f04d802f050 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.630+0000 7f04ea7fc640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 0x7f04d006f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:18.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.630+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f04d80aa550 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.630+0000 7f04f1ed8640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 0x7f04d006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.631+0000 7f04f1ed8640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 0x7f04d006f160 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f04e0007fd0 tx=0x7f04e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:18.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.635+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04d8077100 con 0x7f04ec0ae9b0 2023-11-24T08:21:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.855+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f04ec06e820 con 0x7f04d006cca0 2023-11-24T08:21:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.863+0000 7f04ea7fc640 1 -- 172.21.15.107:0/3527104918 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+71 (secure 0 0 0) 0x7f04e0005b90 con 0x7f04d006cca0 2023-11-24T08:21:18.866 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.865+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 msgr2=0x7f04d006f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.865+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 0x7f04d006f160 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f04e0007fd0 tx=0x7f04e0009040 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.865+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 msgr2=0x7f04ec09a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.865+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f04d8000c00 tx=0x7f04d80046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 shutdown_connections 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04d006cca0 0x7f04d006f160 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04ec0b38a0 0x7f04ec1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04ec0ae9b0 0x7f04ec09a600 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 --2- 172.21.15.107:0/3527104918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04ec06d900 0x7f04ec146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 >> 172.21.15.107:0/3527104918 conn(0x7f04ec06c090 msgr2=0x7f04ec1573c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 shutdown_connections 2023-11-24T08:21:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:18.866+0000 7f04f413c640 1 -- 172.21.15.107:0/3527104918 wait complete. 2023-11-24T08:21:18.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:18.887 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:18.940 INFO:teuthology.orchestra.run.smithi107.stdout:9 2023-11-24T08:21:18.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:18.940 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a) 2023-11-24T08:21:18.999 INFO:teuthology.orchestra.run.smithi107.stdout:9 2023-11-24T08:21:18.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone404 2023-11-24T08:21:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.214+0000 7f33d1f1d640 1 Processor -- start 2023-11-24T08:21:19.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.215+0000 7f33d1f1d640 1 -- start start 2023-11-24T08:21:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.216+0000 7f33d1f1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f33cc0ccbd0 0x7f33cc1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.216+0000 7f33d1f1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc1655f0 con 0x7f33cc0c9650 2023-11-24T08:21:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.216+0000 7f33d1f1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0cc380 con 0x7f33cc0ccfb0 2023-11-24T08:21:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.216+0000 7f33d1f1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc162b30 con 0x7f33cc0ccbd0 2023-11-24T08:21:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33cbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f33cc0c9650 0x7f33cc0c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55182/0 (socket says 172.21.15.107:55182) 2023-11-24T08:21:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33cbfff640 1 -- 172.21.15.107:0/89808755 learned_addr learned my addr 172.21.15.107:0/89808755 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33cb7fe640 1 --2- 172.21.15.107:0/89808755 >> v2:172.21.15.120:3300/0 conn(0x7f33cc0ccbd0 0x7f33cc1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439023900 0 0) 0x7f33cc1655f0 con 0x7f33cc0c9650 2023-11-24T08:21:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8003520 con 0x7f33cc0c9650 2023-11-24T08:21:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56296212 0 0) 0x7f33b8003520 con 0x7f33cc0c9650 2023-11-24T08:21:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 >> v2:172.21.15.120:3300/0 conn(0x7f33cc0ccbd0 msgr2=0x7f33cc1660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 --2- 172.21.15.107:0/89808755 >> v2:172.21.15.120:3300/0 conn(0x7f33cc0ccbd0 0x7f33cc1660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 >> v1:172.21.15.120:6789/0 conn(0x7f33cc0ccfb0 legacy=0x7f33cc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.217+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc162e10 con 0x7f33cc0c9650 2023-11-24T08:21:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f33c4001670 con 0x7f33cc0c9650 2023-11-24T08:21:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 >> v1:172.21.15.107:6789/0 conn(0x7f33cc0c9650 legacy=0x7f33cc0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b80039c0 0x7f33b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b80047c0 0x7f33b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 0x7f33b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc162b30 con 0x7f33b8005180 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c080 con 0x7f33b80039c0 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c240 con 0x7f33b80047c0 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33cbfff640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 0x7f33b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33caffd640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b80047c0 0x7f33b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33cb7fe640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b80039c0 0x7f33b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.218+0000 7f33cbfff640 1 -- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b80047c0 msgr2=0x7f33b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cbfff640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b80047c0 0x7f33b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cbfff640 1 -- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b80039c0 msgr2=0x7f33b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cbfff640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b80039c0 0x7f33b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cbfff640 1 -- 172.21.15.107:0/89808755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b8003520 con 0x7f33b8005180 2023-11-24T08:21:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cb7fe640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b80039c0 0x7f33b8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.219+0000 7f33cbfff640 1 -- 172.21.15.107:0/89808755 reap_dead start 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33cbfff640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 0x7f33b800b980 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c0e0 tx=0x7f33b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c86e46305dd6f00 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4005ad0 con 0x7f33b8005180 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c4005c70 con 0x7f33b8005180 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33ca7fc640 1 -- 172.21.15.107:0/89808755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c402c810 con 0x7f33b8005180 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33d1f1d640 1 -- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 msgr2=0x7f33b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.220+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 0x7f33b800b980 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c0e0 tx=0x7f33b80057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.221+0000 7f33d1f1d640 1 -- 172.21.15.107:0/89808755 shutdown_connections 2023-11-24T08:21:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.221+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/89808755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33b8005180 0x7f33b800b980 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.221+0000 7f33d1f1d640 1 -- 172.21.15.107:0/89808755 >> 172.21.15.107:0/89808755 conn(0x7f33cc0b0c50 msgr2=0x7f33cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.221+0000 7f33d1f1d640 1 -- 172.21.15.107:0/89808755 shutdown_connections 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.221+0000 7f33d1f1d640 1 -- 172.21.15.107:0/89808755 wait complete. 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.222+0000 7f33d1f1d640 1 Processor -- start 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 -- start start 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b8005180 0x7f33cc0c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33cc158580 0x7f33cc0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c8570 con 0x7f33cc1650e0 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0b3b40 con 0x7f33b8005180 2023-11-24T08:21:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.223+0000 7f33d1f1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c8290 con 0x7f33cc158580 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44660/0 (socket says 172.21.15.107:44660) 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 -- 172.21.15.107:0/2081134929 learned_addr learned my addr 172.21.15.107:0/2081134929 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 -- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33cc158580 msgr2=0x7f33cc0ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33cc158580 0x7f33cc0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 -- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b8005180 msgr2=0x7f33cc0c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b8005180 0x7f33cc0c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 -- 172.21.15.107:0/2081134929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b1db0 con 0x7f33cc1650e0 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.224+0000 7f33caffd640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f33b400e410 tx=0x7f33b400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.225+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40052e0 con 0x7f33cc1650e0 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.225+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b4005440 con 0x7f33cc1650e0 2023-11-24T08:21:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.225+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4003810 con 0x7f33cc1650e0 2023-11-24T08:21:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.225+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc0b2090 con 0x7f33cc1650e0 2023-11-24T08:21:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.225+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc0c90b0 con 0x7f33cc1650e0 2023-11-24T08:21:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.226+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc04e8b0 con 0x7f33cc1650e0 2023-11-24T08:21:19.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.227+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f33b4003970 con 0x7f33cc1650e0 2023-11-24T08:21:19.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.228+0000 7f33c3fff640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 0x7f33ac06f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:19.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.228+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f33b4087da0 con 0x7f33cc1650e0 2023-11-24T08:21:19.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.228+0000 7f33cb7fe640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 0x7f33ac06f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:19.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.229+0000 7f33cb7fe640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 0x7f33ac06f190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0c8740 tx=0x7f33bc009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.233+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33b408ee60 con 0x7f33cc1650e0 2023-11-24T08:21:19.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.451+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7f33cc0cd410 con 0x7f33ac06ccd0 2023-11-24T08:21:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.468+0000 7f33c3fff640 1 -- 172.21.15.107:0/2081134929 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+721 (secure 0 0 0) 0x7f33bc011040 con 0x7f33ac06ccd0 2023-11-24T08:21:19.471 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:19.471 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:20:44", 2023-11-24T08:21:19.471 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "66.67", 2023-11-24T08:21:19.471 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": 12582912, 2023-11-24T08:21:19.471 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 8388608, 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:20:51", 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:21:11", 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:21:19.472 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:21:19.473 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:21:19.473 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:21:19.473 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:21:19.473 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:21:19.473 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:20:46", 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a", 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "fsvolumens_subvol7968", 2023-11-24T08:21:19.474 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:21:19.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:21:19.475 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:21:19.475 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 msgr2=0x7f33ac06f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 0x7f33ac06f190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0c8740 tx=0x7f33bc009750 comp rx=0 tx=0).stop 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 msgr2=0x7f33cc158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f33b400e410 tx=0x7f33b400e8d0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 shutdown_connections 2023-11-24T08:21:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f33ac06ccd0 0x7f33ac06f190 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33cc158580 0x7f33cc0ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33cc1650e0 0x7f33cc158040 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 --2- 172.21.15.107:0/2081134929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b8005180 0x7f33cc0c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:19.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 >> 172.21.15.107:0/2081134929 conn(0x7f33cc0b0c50 msgr2=0x7f33cc0aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:19.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 shutdown_connections 2023-11-24T08:21:19.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:19.471+0000 7f33d1f1d640 1 -- 172.21.15.107:0/2081134929 wait complete. 2023-11-24T08:21:19.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d: ceph.quota.max_bytes: No such attribute 2023-11-24T08:21:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:21:19.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:21:19.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.596 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.650 INFO:teuthology.orchestra.run.smithi107.stdout:fsvolumens_subvol7968 2023-11-24T08:21:19.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.650 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.702 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3 2023-11-24T08:21:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1 2023-11-24T08:21:19.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.704 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.754 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:21:19.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.755 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/) 2023-11-24T08:21:19.812 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:21:19.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.814 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.864 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:21:19.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.866 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/) 2023-11-24T08:21:19.916 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:21:19.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.917 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:19.968 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:21:19.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:19.969 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/) 2023-11-24T08:21:20.020 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:21:20.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.021 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d) 2023-11-24T08:21:20.073 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:20.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/) 2023-11-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:20.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.129 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2) 2023-11-24T08:21:20.182 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:20.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.183 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.2) 2023-11-24T08:21:20.233 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:20.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.234 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2) 2023-11-24T08:21:20.286 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:20.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.287 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.2) 2023-11-24T08:21:20.338 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:20.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.339 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2) 2023-11-24T08:21:20.390 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:20.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.392 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.2) 2023-11-24T08:21:20.442 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:20.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.443 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.2) 2023-11-24T08:21:20.494 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:20.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.495 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.2) 2023-11-24T08:21:20.548 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:20.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.550 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5) 2023-11-24T08:21:20.602 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:20.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.5) 2023-11-24T08:21:20.654 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:20.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5) 2023-11-24T08:21:20.706 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:20.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.5) 2023-11-24T08:21:20.761 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:20.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.762 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5) 2023-11-24T08:21:20.818 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:20.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.820 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.5) 2023-11-24T08:21:20.873 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:20.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.874 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.5) 2023-11-24T08:21:20.924 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:20.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.5) 2023-11-24T08:21:20.974 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:20.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:20.975 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4) 2023-11-24T08:21:21.028 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.029 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.4) 2023-11-24T08:21:21.082 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.083 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4) 2023-11-24T08:21:21.137 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:21.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.4) 2023-11-24T08:21:21.188 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:21.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.189 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4) 2023-11-24T08:21:21.241 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:21.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.242 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.4) 2023-11-24T08:21:21.297 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:21.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.298 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.4) 2023-11-24T08:21:21.346 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:21.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.347 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.4) 2023-11-24T08:21:21.396 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:21.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.398 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7) 2023-11-24T08:21:21.448 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.450 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.7) 2023-11-24T08:21:21.502 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.503 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7) 2023-11-24T08:21:21.554 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:21.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.556 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.7) 2023-11-24T08:21:21.609 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:21.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.611 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7) 2023-11-24T08:21:21.662 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:21.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.663 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.7) 2023-11-24T08:21:21.712 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:21.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.713 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.7) 2023-11-24T08:21:21.763 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:21.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.764 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.7) 2023-11-24T08:21:21.818 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:21.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.820 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6) 2023-11-24T08:21:21.872 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.874 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.6) 2023-11-24T08:21:21.925 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:21.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6) 2023-11-24T08:21:21.975 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:21.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:21.976 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.6) 2023-11-24T08:21:22.025 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:22.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6) 2023-11-24T08:21:22.077 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.078 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.6) 2023-11-24T08:21:22.126 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.128 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.6) 2023-11-24T08:21:22.177 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:22.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.178 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.6) 2023-11-24T08:21:22.228 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:22.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0) 2023-11-24T08:21:22.283 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:22.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.284 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.0) 2023-11-24T08:21:22.338 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:22.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.339 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0) 2023-11-24T08:21:22.389 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:22.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.390 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.0) 2023-11-24T08:21:22.442 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:22.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0) 2023-11-24T08:21:22.495 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.496 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.0) 2023-11-24T08:21:22.548 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.0) 2023-11-24T08:21:22.601 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:22.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.602 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.0) 2023-11-24T08:21:22.655 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:22.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.656 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3) 2023-11-24T08:21:22.711 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:22.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.713 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.3) 2023-11-24T08:21:22.767 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:22.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.768 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3) 2023-11-24T08:21:22.818 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:22.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.819 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.3) 2023-11-24T08:21:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:22.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.872 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3) 2023-11-24T08:21:22.921 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.922 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.3) 2023-11-24T08:21:22.971 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:22.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:22.973 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.3) 2023-11-24T08:21:23.024 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:23.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.026 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.3) 2023-11-24T08:21:23.077 INFO:teuthology.orchestra.run.smithi107.stdout:1700814046 2023-11-24T08:21:23.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.079 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1) 2023-11-24T08:21:23.130 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:23.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.132 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.1) 2023-11-24T08:21:23.182 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:21:23.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1) 2023-11-24T08:21:23.235 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:23.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.1) 2023-11-24T08:21:23.288 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:21:23.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1) 2023-11-24T08:21:23.340 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:23.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.341 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.1) 2023-11-24T08:21:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:21:23.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.394 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol7968/.snap/subvol_snap3729/cfcbbc53-bad1-417e-a9b6-7fd52356b58d/subvolume_file.1) 2023-11-24T08:21:23.444 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:23.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:23.445 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone404/594f41b2-1a90-416b-85cc-42ebe6ec5c6a/subvolume_file.1) 2023-11-24T08:21:23.496 INFO:teuthology.orchestra.run.smithi107.stdout:1700814045 2023-11-24T08:21:23.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol7968 subvol_snap3729 2023-11-24T08:21:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.710+0000 7f464a105640 1 Processor -- start 2023-11-24T08:21:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.710+0000 7f464a105640 1 -- start start 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f464a105640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46440cf450 0x7f46440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f464a105640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46440cdfb0 0x7f46440ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f464a105640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440cece0 con 0x7f464415c7f0 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f464a105640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46440cfe40 con 0x7f46440cf450 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f464a105640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644156010 con 0x7f46440cdfb0 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.712+0000 7f4642ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46440cf450 0x7f46440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.713+0000 7f4642ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46440cf450 0x7f46440cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:36456/0 (socket says 172.21.15.107:36456) 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.713+0000 7f4642ffd640 1 -- 172.21.15.107:0/1985766406 learned_addr learned my addr 172.21.15.107:0/1985766406 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.713+0000 7f46437fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f464415c7f0 0x7f464406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:39772/0 (socket says 172.21.15.107:39772) 2023-11-24T08:21:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.713+0000 7f4643fff640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46440cdfb0 0x7f46440ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.713+0000 7f4642ffd640 1 -- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46440cdfb0 msgr2=0x7f46440ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.714+0000 7f4642ffd640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46440cdfb0 0x7f46440ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.714+0000 7f4642ffd640 1 -- 172.21.15.107:0/1985766406 >> v1:172.21.15.120:6789/0 conn(0x7f464415c7f0 legacy=0x7f464406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.714+0000 7f4642ffd640 1 -- 172.21.15.107:0/1985766406 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464406d040 con 0x7f46440cf450 2023-11-24T08:21:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.715+0000 7f4642ffd640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cf450 0x7f46440cf900 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4638009870 tx=0x7f463802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace06395cbd5083b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.715+0000 7f46427fc640 1 -- 172.21.15.107:0/1985766406 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638030840 con 0x7f46440cf450 2023-11-24T08:21:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.715+0000 7f46427fc640 1 -- 172.21.15.107:0/1985766406 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46380309a0 con 0x7f46440cf450 2023-11-24T08:21:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.715+0000 7f46427fc640 1 -- 172.21.15.107:0/1985766406 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638030d30 con 0x7f46440cf450 2023-11-24T08:21:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 -- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cf450 msgr2=0x7f46440cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cf450 0x7f46440cf900 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4638009870 tx=0x7f463802b980 comp rx=0 tx=0).stop 2023-11-24T08:21:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 -- 172.21.15.107:0/1985766406 shutdown_connections 2023-11-24T08:21:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46440cdfb0 0x7f46440ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 --2- 172.21.15.107:0/1985766406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cf450 0x7f46440cf900 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 -- 172.21.15.107:0/1985766406 >> 172.21.15.107:0/1985766406 conn(0x7f464406a920 msgr2=0x7f464406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 -- 172.21.15.107:0/1985766406 shutdown_connections 2023-11-24T08:21:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.716+0000 7f464a105640 1 -- 172.21.15.107:0/1985766406 wait complete. 2023-11-24T08:21:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.718+0000 7f464a105640 1 Processor -- start 2023-11-24T08:21:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.719+0000 7f464a105640 1 -- start start 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cdfb0 0x7f4644151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464415c7f0 0x7f4644156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46440ad4d0 con 0x7f46440cf450 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644151b50 con 0x7f46440cdfb0 2023-11-24T08:21:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f464a105640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46440ac8c0 con 0x7f464415c7f0 2023-11-24T08:21:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44664/0 (socket says 172.21.15.107:44664) 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.720+0000 7f4642ffd640 1 -- 172.21.15.107:0/2645005260 learned_addr learned my addr 172.21.15.107:0/2645005260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f46437fe640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cdfb0 0x7f4644151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4643fff640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464415c7f0 0x7f4644156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 -- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464415c7f0 msgr2=0x7f4644156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464415c7f0 0x7f4644156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 -- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cdfb0 msgr2=0x7f4644151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cdfb0 0x7f4644151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 -- 172.21.15.107:0/2645005260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380094d0 con 0x7f46440cf450 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f4642ffd640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638039c90 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.721+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638039df0 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.722+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638043640 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.722+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46440acba0 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.722+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46440ad160 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.723+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46440ad640 con 0x7f46440cf450 2023-11-24T08:21:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.724+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f46380437a0 con 0x7f46440cf450 2023-11-24T08:21:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.725+0000 7f462bfff640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 0x7f463006f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.725+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f463803e070 con 0x7f46440cf450 2023-11-24T08:21:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.725+0000 7f46437fe640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 0x7f463006f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.726+0000 7f46437fe640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 0x7f463006f190 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4634000f60 tx=0x7f4634007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.730+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46380781b0 con 0x7f46440cf450 2023-11-24T08:21:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:23.955+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol7968", "snap_name": "subvol_snap3729", "target": ["mon-mgr", ""]}) v1 -- 0x7f46440ce960 con 0x7f463006ccd0 2023-11-24T08:21:24.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.019+0000 7f462bfff640 1 -- 172.21.15.107:0/2645005260 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4634005a70 con 0x7f463006ccd0 2023-11-24T08:21:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 msgr2=0x7f463006f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 0x7f463006f190 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4634000f60 tx=0x7f4634007ca0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 msgr2=0x7f46440ad960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638004730 comp rx=0 tx=0).stop 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 shutdown_connections 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f463006ccd0 0x7f463006f190 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464415c7f0 0x7f4644156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440cf450 0x7f46440ad960 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 --2- 172.21.15.107:0/2645005260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46440cdfb0 0x7f4644151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 >> 172.21.15.107:0/2645005260 conn(0x7f464406a920 msgr2=0x7f46440b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 shutdown_connections 2023-11-24T08:21:24.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.022+0000 7f464a105640 1 -- 172.21.15.107:0/2645005260 wait complete. 2023-11-24T08:21:24.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol7968 2023-11-24T08:21:24.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.256+0000 7f533f5c5640 1 Processor -- start 2023-11-24T08:21:24.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.256+0000 7f533f5c5640 1 -- start start 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815b440 0x7f53380cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f53380cbf20 0x7f53380c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53380c57e0 0x7f53380c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533815b820 con 0x7f53380c57e0 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53380cc3a0 con 0x7f53380cbf20 2023-11-24T08:21:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.258+0000 7f533f5c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533815eea0 con 0x7f533815b440 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53380c57e0 0x7f53380c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53380c57e0 0x7f53380c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44674/0 (socket says 172.21.15.107:44674) 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 -- 172.21.15.107:0/2292347114 learned_addr learned my addr 172.21.15.107:0/2292347114 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533cb60640 1 --2- 172.21.15.107:0/2292347114 >> v2:172.21.15.120:3300/0 conn(0x7f53380cbf20 0x7f53380c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533d361640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815b440 0x7f53380cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 -- 172.21.15.107:0/2292347114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815b440 msgr2=0x7f53380cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815b440 0x7f53380cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 -- 172.21.15.107:0/2292347114 >> v2:172.21.15.120:3300/0 conn(0x7f53380cbf20 msgr2=0x7f53380c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 --2- 172.21.15.107:0/2292347114 >> v2:172.21.15.120:3300/0 conn(0x7f53380cbf20 0x7f53380c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.259+0000 7f533db62640 1 -- 172.21.15.107:0/2292347114 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380abf30 con 0x7f53380c57e0 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.260+0000 7f533db62640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53380c57e0 0x7f53380c3ec0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f53380cee60 tx=0x7f533002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad45c2dcc914e31 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.260+0000 7f532ffff640 1 -- 172.21.15.107:0/2292347114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533002c9e0 con 0x7f53380c57e0 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.260+0000 7f532ffff640 1 -- 172.21.15.107:0/2292347114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533002cb40 con 0x7f53380c57e0 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f532ffff640 1 -- 172.21.15.107:0/2292347114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53300343f0 con 0x7f53380c57e0 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 -- 172.21.15.107:0/2292347114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53380c57e0 msgr2=0x7f53380c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53380c57e0 0x7f53380c3ec0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f53380cee60 tx=0x7f533002be40 comp rx=0 tx=0).stop 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 -- 172.21.15.107:0/2292347114 shutdown_connections 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53380c57e0 0x7f53380c3ec0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 --2- 172.21.15.107:0/2292347114 >> v2:172.21.15.120:3300/0 conn(0x7f53380cbf20 0x7f53380c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 --2- 172.21.15.107:0/2292347114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815b440 0x7f53380cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 -- 172.21.15.107:0/2292347114 >> 172.21.15.107:0/2292347114 conn(0x7f53380b0c50 msgr2=0x7f53380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:24.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 -- 172.21.15.107:0/2292347114 shutdown_connections 2023-11-24T08:21:24.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.261+0000 7f533f5c5640 1 -- 172.21.15.107:0/2292347114 wait complete. 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.263+0000 7f533f5c5640 1 Processor -- start 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.263+0000 7f533f5c5640 1 -- start start 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380c57e0 0x7f5338156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 0x7f53380999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338156ea0 con 0x7f533815b440 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533809a400 con 0x7f53380c57e0 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533f5c5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533809a6e0 con 0x7f53380cbf20 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.264+0000 7f533db62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44680/0 (socket says 172.21.15.107:44680) 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 -- 172.21.15.107:0/3698680007 learned_addr learned my addr 172.21.15.107:0/3698680007 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533cb60640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 0x7f53380999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533d361640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380c57e0 0x7f5338156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 -- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 msgr2=0x7f53380999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 0x7f53380999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 -- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380c57e0 msgr2=0x7f5338156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380c57e0 0x7f5338156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 -- 172.21.15.107:0/3698680007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330009b30 con 0x7f533815b440 2023-11-24T08:21:24.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533cb60640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 0x7f53380999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.265+0000 7f533db62640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f5330036040 tx=0x7f5330004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.266+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330009510 con 0x7f533815b440 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.266+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533003e450 con 0x7f533815b440 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.266+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533003e7e0 con 0x7f533815b440 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.266+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533809a9c0 con 0x7f533815b440 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.266+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533809af80 con 0x7f533815b440 2023-11-24T08:21:24.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.267+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533809b1a0 con 0x7f533815b440 2023-11-24T08:21:24.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.268+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f533003e940 con 0x7f533815b440 2023-11-24T08:21:24.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.269+0000 7f532dffb640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 0x7f531c06f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.269+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f53300ab0c0 con 0x7f533815b440 2023-11-24T08:21:24.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.269+0000 7f533d361640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 0x7f531c06f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.270+0000 7f533d361640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 0x7f531c06f190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5334006fd0 tx=0x7f5334008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.273+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5330077c70 con 0x7f533815b440 2023-11-24T08:21:24.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.489+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol7968", "target": ["mon-mgr", ""]}) v1 -- 0x7f533815f6b0 con 0x7f531c06ccd0 2023-11-24T08:21:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.534+0000 7f532dffb640 1 -- 172.21.15.107:0/3698680007 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f533400e4c0 con 0x7f531c06ccd0 2023-11-24T08:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.537+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 msgr2=0x7f531c06f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 0x7f531c06f190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5334006fd0 tx=0x7f5334008040 comp rx=0 tx=0).stop 2023-11-24T08:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 msgr2=0x7f533815d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f5330036040 tx=0x7f5330004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 shutdown_connections 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f531c06ccd0 0x7f531c06f190 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f533815b440 0x7f533815d470 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53380cbf20 0x7f53380999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 --2- 172.21.15.107:0/3698680007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380c57e0 0x7f5338156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 >> 172.21.15.107:0/3698680007 conn(0x7f53380b0c50 msgr2=0x7f53380ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 shutdown_connections 2023-11-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.538+0000 7f533f5c5640 1 -- 172.21.15.107:0/3698680007 wait complete. 2023-11-24T08:21:24.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.550+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.550+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.550+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.550+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.550+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone404 2023-11-24T08:21:24.574 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.572+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.574 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.572+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.574 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.572+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.574 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.572+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.574 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:21:24.572+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:21:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.754+0000 7fcfdd48c640 1 Processor -- start 2023-11-24T08:21:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.755+0000 7fcfdd48c640 1 -- start start 2023-11-24T08:21:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfdd48c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfd815d490 0x7fcfd80cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfdd48c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80d3940 0x7fcfd80d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfdd48c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd80cea30 con 0x7fcfd80b63d0 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfdd48c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd80d0450 con 0x7fcfd815d490 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfdd48c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80d0bf0 con 0x7fcfd80d3940 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfd7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfd815d490 0x7fcfd80cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfd7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcfd815d490 0x7fcfd80cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44714/0 (socket says 172.21.15.107:44714) 2023-11-24T08:21:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.756+0000 7fcfd7fff640 1 -- 172.21.15.107:0/1656672403 learned_addr learned my addr 172.21.15.107:0/1656672403 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.757+0000 7fcfd7fff640 1 -- 172.21.15.107:0/1656672403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80d3940 msgr2=0x7fcfd80d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.757+0000 7fcfdcc8b640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80d3940 0x7fcfd80d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.757+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80d3940 0x7fcfd80d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.757+0000 7fcfd7fff640 1 -- 172.21.15.107:0/1656672403 >> v1:172.21.15.120:6789/0 conn(0x7fcfd80b63d0 legacy=0x7fcfd80d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.757+0000 7fcfd7fff640 1 -- 172.21.15.107:0/1656672403 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd80ce750 con 0x7fcfd815d490 2023-11-24T08:21:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.758+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd815d490 0x7fcfd80cd8f0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e50 tx=0x7fcfc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8370dcdebfd31b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.758+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/1656672403 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802ca70 con 0x7fcfd815d490 2023-11-24T08:21:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.758+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/1656672403 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc802cbd0 con 0x7fcfd815d490 2023-11-24T08:21:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.758+0000 7fcfd6ffd640 1 -- 172.21.15.107:0/1656672403 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80343f0 con 0x7fcfd815d490 2023-11-24T08:21:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 -- 172.21.15.107:0/1656672403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd815d490 msgr2=0x7fcfd80cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd815d490 0x7fcfd80cd8f0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e50 tx=0x7fcfc802bed0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 -- 172.21.15.107:0/1656672403 shutdown_connections 2023-11-24T08:21:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80d3940 0x7fcfd80d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/1656672403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd815d490 0x7fcfd80cd8f0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 -- 172.21.15.107:0/1656672403 >> 172.21.15.107:0/1656672403 conn(0x7fcfd8061340 msgr2=0x7fcfd8061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:24.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 -- 172.21.15.107:0/1656672403 shutdown_connections 2023-11-24T08:21:24.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.759+0000 7fcfdd48c640 1 -- 172.21.15.107:0/1656672403 wait complete. 2023-11-24T08:21:24.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.761+0000 7fcfdd48c640 1 Processor -- start 2023-11-24T08:21:24.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.761+0000 7fcfdd48c640 1 -- start start 2023-11-24T08:21:24.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd815d490 0x7fcfd80cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd81f6260 0x7fcfd80cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd815ded0 con 0x7fcfd80d3940 2023-11-24T08:21:24.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80c76a0 con 0x7fcfd81f6260 2023-11-24T08:21:24.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfdd48c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80ccd40 con 0x7fcfd815d490 2023-11-24T08:21:24.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44726/0 (socket says 172.21.15.107:44726) 2023-11-24T08:21:24.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 -- 172.21.15.107:0/658854431 learned_addr learned my addr 172.21.15.107:0/658854431 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:24.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 -- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd815d490 msgr2=0x7fcfd80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd815d490 0x7fcfd80cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.762+0000 7fcfd7fff640 1 -- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd81f6260 msgr2=0x7fcfd80cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:24.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd81f6260 0x7fcfd80cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:24.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd7fff640 1 -- 172.21.15.107:0/658854431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8009b30 con 0x7fcfd80d3940 2023-11-24T08:21:24.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd7fff640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e50 tx=0x7fcfc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803e450 con 0x7fcfd80d3940 2023-11-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc803e5f0 con 0x7fcfd80d3940 2023-11-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803e9a0 con 0x7fcfd80d3940 2023-11-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd8149970 con 0x7fcfd80d3940 2023-11-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.763+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd815d910 con 0x7fcfd80d3940 2023-11-24T08:21:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.764+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd80ce230 con 0x7fcfd80d3940 2023-11-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.765+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fcfc803eb00 con 0x7fcfd80d3940 2023-11-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.766+0000 7fcfd4ff9640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 0x7fcfb806f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.766+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fcfc80ab2a0 con 0x7fcfd80d3940 2023-11-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.766+0000 7fcfd77fe640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 0x7fcfb806f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.767+0000 7fcfd77fe640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 0x7fcfb806f190 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80b2100 tx=0x7fcfc00079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.770+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfc8077e50 con 0x7fcfd80d3940 2023-11-24T08:21:24.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:24.976+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone404", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfd80cd860 con 0x7fcfb806ccd0 2023-11-24T08:21:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.000+0000 7fcfd4ff9640 1 -- 172.21.15.107:0/658854431 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcfd80cd860 con 0x7fcfb806ccd0 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 msgr2=0x7fcfb806f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 0x7fcfb806f190 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80b2100 tx=0x7fcfc00079b0 comp rx=0 tx=0).stop 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 msgr2=0x7fcfd80cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e50 tx=0x7fcfc8004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 shutdown_connections 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcfb806ccd0 0x7fcfb806f190 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd81f6260 0x7fcfd80cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd815d490 0x7fcfd80cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 --2- 172.21.15.107:0/658854431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd80d3940 0x7fcfd80cbd80 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 >> 172.21.15.107:0/658854431 conn(0x7fcfd8061340 msgr2=0x7fcfd8160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 shutdown_connections 2023-11-24T08:21:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:25.003+0000 7fcfdd48c640 1 -- 172.21.15.107:0/658854431 wait complete. 2023-11-24T08:21:25.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:25.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:21:25.634 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:21:25.636 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:21:25.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:21:25.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:21:25.888 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:21:25.889 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:25.889 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:25.951 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:25.952 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:26.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:26.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.220+0000 7fd4dcd1d640 1 Processor -- start 2023-11-24T08:21:26.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.220+0000 7fd4dcd1d640 1 -- start start 2023-11-24T08:21:26.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4dcd1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4d81609e0 0x7fd4d806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4dcd1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d80ceb70 con 0x7fd4d80ce010 2023-11-24T08:21:26.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4dcd1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d806c1f0 con 0x7fd4d80d31a0 2023-11-24T08:21:26.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4dcd1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d80d1070 con 0x7fd4d81609e0 2023-11-24T08:21:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4d659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4d81609e0 0x7fd4d806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4d6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd4d80ce010 0x7fd4d80ce460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55194/0 (socket says 172.21.15.107:55194) 2023-11-24T08:21:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.221+0000 7fd4d6d9d640 1 -- 172.21.15.107:0/3921157343 learned_addr learned my addr 172.21.15.107:0/3921157343 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.222+0000 7fd4d659c640 1 -- 172.21.15.107:0/3921157343 >> v1:172.21.15.120:6789/0 conn(0x7fd4d80d31a0 legacy=0x7fd4d80d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.222+0000 7fd4d659c640 1 -- 172.21.15.107:0/3921157343 >> v1:172.21.15.107:6789/0 conn(0x7fd4d80ce010 legacy=0x7fd4d80ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.222+0000 7fd4d659c640 1 -- 172.21.15.107:0/3921157343 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80cedc0 con 0x7fd4d81609e0 2023-11-24T08:21:26.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.222+0000 7fd4d559a640 1 -- 172.21.15.107:0/3921157343 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971181626 0 0) 0x7fd4d80ceb70 con 0x7fd4d80ce010 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.223+0000 7fd4d659c640 1 --2- 172.21.15.107:0/3921157343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d81609e0 0x7fd4d806bcb0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00097a0 tx=0x7fd4c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6126186a93eb58 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.223+0000 7fd4d559a640 1 -- 172.21.15.107:0/3921157343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0031830 con 0x7fd4d81609e0 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.223+0000 7fd4d559a640 1 -- 172.21.15.107:0/3921157343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c0031990 con 0x7fd4d81609e0 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.223+0000 7fd4d559a640 1 -- 172.21.15.107:0/3921157343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0031d20 con 0x7fd4d81609e0 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3921157343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d81609e0 msgr2=0x7fd4d806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3921157343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d81609e0 0x7fd4d806bcb0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00097a0 tx=0x7fd4c002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3921157343 shutdown_connections 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3921157343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d81609e0 0x7fd4d806bcb0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3921157343 >> 172.21.15.107:0/3921157343 conn(0x7fd4d808af60 msgr2=0x7fd4d809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3921157343 shutdown_connections 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.224+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3921157343 wait complete. 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.226+0000 7fd4dcd1d640 1 Processor -- start 2023-11-24T08:21:26.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.226+0000 7fd4dcd1d640 1 -- start start 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4d80d31a0 0x7fd4d80b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4d81609e0 0x7fd4d81e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d815f830 con 0x7fd4d80ce010 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d80b66d0 con 0x7fd4d81609e0 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4dcd1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d80b69b0 con 0x7fd4d80d31a0 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44742/0 (socket says 172.21.15.107:44742) 2023-11-24T08:21:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4d659c640 1 -- 172.21.15.107:0/3248099367 learned_addr learned my addr 172.21.15.107:0/3248099367 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.227+0000 7fd4d5d9b640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4d80d31a0 0x7fd4d80b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d6d9d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4d81609e0 0x7fd4d81e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 -- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4d80d31a0 msgr2=0x7fd4d80b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4d80d31a0 0x7fd4d80b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 -- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4d81609e0 msgr2=0x7fd4d81e7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4d81609e0 0x7fd4d81e7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 -- 172.21.15.107:0/3248099367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c0009400 con 0x7fd4d80ce010 2023-11-24T08:21:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4d659c640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00098c0 tx=0x7fd4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c003ba70 con 0x7fd4d80ce010 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c003bbf0 con 0x7fd4d80ce010 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c00453f0 con 0x7fd4d80ce010 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d80b6c90 con 0x7fd4d80ce010 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.228+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d80b7250 con 0x7fd4d80ce010 2023-11-24T08:21:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.230+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d8067470 con 0x7fd4d80ce010 2023-11-24T08:21:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.231+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7fd4c004e450 con 0x7fd4d80ce010 2023-11-24T08:21:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.231+0000 7fd4ceffd640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 0x7fd4b806f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.232+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fd4c00aa8c0 con 0x7fd4d80ce010 2023-11-24T08:21:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.232+0000 7fd4d5d9b640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 0x7fd4b806f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.232+0000 7fd4d5d9b640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 0x7fd4b806f160 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80d3d10 tx=0x7fd4c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.236+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4c0077470 con 0x7fd4d80ce010 2023-11-24T08:21:26.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.429+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4d80d39b0 con 0x7fd4d80ce010 2023-11-24T08:21:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.433+0000 7fd4ceffd640 1 -- 172.21.15.107:0/3248099367 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+18360 (secure 0 0 0) 0x7fd4c007aa90 con 0x7fd4d80ce010 2023-11-24T08:21:26.436 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:26.436 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":193,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:20:50.576938+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:26.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.436+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 msgr2=0x7fd4b806f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.436+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 0x7fd4b806f160 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80d3d10 tx=0x7fd4c8009040 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.436+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 msgr2=0x7fd4d80b7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.436+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00098c0 tx=0x7fd4c00046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 shutdown_connections 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd4b806cca0 0x7fd4b806f160 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4d81609e0 0x7fd4d81e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4d80d31a0 0x7fd4d80b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 --2- 172.21.15.107:0/3248099367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4d80ce010 0x7fd4d80b7770 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 >> 172.21.15.107:0/3248099367 conn(0x7fd4d808af60 msgr2=0x7fd4d80d1830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 shutdown_connections 2023-11-24T08:21:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.437+0000 7fd4dcd1d640 1 -- 172.21.15.107:0/3248099367 wait complete. 2023-11-24T08:21:26.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:20:30.467990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:20:31.721268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '190', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'new_pool', 'create_time': '2023-11-24T08:20:48.209202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '193', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:21:26.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:21:26.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.668+0000 7f7b46d3c640 1 Processor -- start 2023-11-24T08:21:26.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.668+0000 7f7b46d3c640 1 -- start start 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b46d3c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400b0b40 0x7f7b400b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b46d3c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b400cce40 0x7f7b400cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b46d3c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400b14d0 con 0x7f7b4015d310 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b46d3c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4006cd10 con 0x7f7b400b0b40 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b46d3c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400ae5d0 con 0x7f7b400cce40 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b45539640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400b0b40 0x7f7b400b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b45539640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400b0b40 0x7f7b400b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44750/0 (socket says 172.21.15.107:44750) 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b45539640 1 -- 172.21.15.107:0/880216283 learned_addr learned my addr 172.21.15.107:0/880216283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:26.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b45d3a640 1 --1- 172.21.15.107:0/880216283 >> v1:172.21.15.107:6789/0 conn(0x7f7b4015d310 0x7f7b400ca120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55206/0 (socket says 172.21.15.107:55206) 2023-11-24T08:21:26.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.670+0000 7f7b4653b640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b400cce40 0x7f7b400cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.671+0000 7f7b45539640 1 -- 172.21.15.107:0/880216283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b400cce40 msgr2=0x7f7b400cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:26.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.671+0000 7f7b45539640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b400cce40 0x7f7b400cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.671+0000 7f7b45539640 1 -- 172.21.15.107:0/880216283 >> v1:172.21.15.107:6789/0 conn(0x7f7b4015d310 legacy=0x7f7b400ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.671+0000 7f7b45539640 1 -- 172.21.15.107:0/880216283 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400ae8b0 con 0x7f7b400b0b40 2023-11-24T08:21:26.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.671+0000 7f7b44d38640 1 -- 172.21.15.107:0/880216283 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855025525 0 0) 0x7f7b3c001110 con 0x7f7b4015d310 2023-11-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.672+0000 7f7b45539640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400b0b40 0x7f7b400b0f90 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9806dde5785360 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.672+0000 7f7b44d38640 1 -- 172.21.15.107:0/880216283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30030840 con 0x7f7b400b0b40 2023-11-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.672+0000 7f7b44d38640 1 -- 172.21.15.107:0/880216283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b300309a0 con 0x7f7b400b0b40 2023-11-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.672+0000 7f7b44d38640 1 -- 172.21.15.107:0/880216283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30030d30 con 0x7f7b400b0b40 2023-11-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 -- 172.21.15.107:0/880216283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400b0b40 msgr2=0x7f7b400b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400b0b40 0x7f7b400b0f90 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b3002b980 comp rx=0 tx=0).stop 2023-11-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 -- 172.21.15.107:0/880216283 shutdown_connections 2023-11-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b400cce40 0x7f7b400cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/880216283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400b0b40 0x7f7b400b0f90 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 -- 172.21.15.107:0/880216283 >> 172.21.15.107:0/880216283 conn(0x7f7b400cbb40 msgr2=0x7f7b400cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:26.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 -- 172.21.15.107:0/880216283 shutdown_connections 2023-11-24T08:21:26.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.673+0000 7f7b46d3c640 1 -- 172.21.15.107:0/880216283 wait complete. 2023-11-24T08:21:26.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.675+0000 7f7b46d3c640 1 Processor -- start 2023-11-24T08:21:26.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.675+0000 7f7b46d3c640 1 -- start start 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400b0b40 0x7f7b400c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 0x7f7b400c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4015bdf0 con 0x7f7b400cce40 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4015f060 con 0x7f7b400b0b40 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b46d3c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4015f340 con 0x7f7b4015d310 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44766/0 (socket says 172.21.15.107:44766) 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 -- 172.21.15.107:0/721299582 learned_addr learned my addr 172.21.15.107:0/721299582 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:26.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 -- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 msgr2=0x7f7b400c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b4653b640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 0x7f7b400c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45d3a640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400b0b40 0x7f7b400c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 0x7f7b400c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 -- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400b0b40 msgr2=0x7f7b400c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400b0b40 0x7f7b400c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b45539640 1 -- 172.21.15.107:0/721299582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300094d0 con 0x7f7b400cce40 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.676+0000 7f7b4653b640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 0x7f7b400c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b45539640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7b30000c00 tx=0x7f7b30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3003e040 con 0x7f7b400cce40 2023-11-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b30039c90 con 0x7f7b400cce40 2023-11-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4015f620 con 0x7f7b400cce40 2023-11-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30004030 con 0x7f7b400cce40 2023-11-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.677+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4015fbe0 con 0x7f7b400cce40 2023-11-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.678+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4004e840 con 0x7f7b400cce40 2023-11-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.680+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 41) v1 ==== 91985+0+0 (secure 0 0 0) 0x7f7b30004190 con 0x7f7b400cce40 2023-11-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.681+0000 7f7b367fc640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 0x7f7b2406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.681+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7b300aaa60 con 0x7f7b400cce40 2023-11-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.681+0000 7f7b45d3a640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 0x7f7b2406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.682+0000 7f7b45d3a640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 0x7f7b2406f160 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c000f60 tx=0x7f7b3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.685+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b30077730 con 0x7f7b400cce40 2023-11-24T08:21:26.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.837+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7b30077b00 con 0x7f7b400cce40 2023-11-24T08:21:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.881+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b4015fdd0 con 0x7f7b400cce40 2023-11-24T08:21:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.882+0000 7f7b367fc640 1 -- 172.21.15.107:0/721299582 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b300921e0 con 0x7f7b400cce40 2023-11-24T08:21:26.884 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 msgr2=0x7f7b2406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 0x7f7b2406f160 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c000f60 tx=0x7f7b3c009040 comp rx=0 tx=0).stop 2023-11-24T08:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 msgr2=0x7f7b4015b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7b30000c00 tx=0x7f7b30004730 comp rx=0 tx=0).stop 2023-11-24T08:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 shutdown_connections 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b2406cca0 0x7f7b2406f160 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b4015d310 0x7f7b400c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400cce40 0x7f7b4015b8b0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 --2- 172.21.15.107:0/721299582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b400b0b40 0x7f7b400c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 >> 172.21.15.107:0/721299582 conn(0x7f7b400cbb40 msgr2=0x7f7b400cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 shutdown_connections 2023-11-24T08:21:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:26.885+0000 7f7b46d3c640 1 -- 172.21.15.107:0/721299582 wait complete. 2023-11-24T08:21:26.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:21:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.118+0000 7ff73a8bb640 1 Processor -- start 2023-11-24T08:21:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.118+0000 7ff73a8bb640 1 -- start start 2023-11-24T08:21:27.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.119+0000 7ff73a8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff7340b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.119+0000 7ff73a8bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff73406c5b0 0x7ff73406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff73a8bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73415ed20 con 0x7ff73415b260 2023-11-24T08:21:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff73a8bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73415b640 con 0x7ff73406c5b0 2023-11-24T08:21:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff73a8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73415f8a0 con 0x7ff7340b0b40 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff7398b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff73415b260 0x7ff73406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55212/0 (socket says 172.21.15.107:55212) 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff7398b9640 1 -- 172.21.15.107:0/2991457981 learned_addr learned my addr 172.21.15.107:0/2991457981 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff7390b8640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff7340b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff73a0ba640 1 --2- 172.21.15.107:0/2991457981 >> v2:172.21.15.120:3300/0 conn(0x7ff73406c5b0 0x7ff73406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.120+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655133976 0 0) 0x7ff73415ed20 con 0x7ff73415b260 2023-11-24T08:21:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff720003520 con 0x7ff73415b260 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2980615652 0 0) 0x7ff720003520 con 0x7ff73415b260 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 msgr2=0x7ff7340b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff7340b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 >> v2:172.21.15.120:3300/0 conn(0x7ff73406c5b0 msgr2=0x7ff73406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 --2- 172.21.15.107:0/2991457981 >> v2:172.21.15.120:3300/0 conn(0x7ff73406c5b0 0x7ff73406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7340b14c0 con 0x7ff73415b260 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff730001670 con 0x7ff73415b260 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 >> v1:172.21.15.107:6789/0 conn(0x7ff73415b260 legacy=0x7ff73406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff720003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff720005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73415b640 con 0x7ff720003970 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73415f8a0 con 0x7ff7200047c0 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72000ba00 con 0x7ff7340b0b40 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.121+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff720003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7390b8640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff73a0ba640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff720005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7398b9640 1 -- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 msgr2=0x7ff720005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b0b40 0x7ff720005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7398b9640 1 -- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7200047c0 msgr2=0x7ff720004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.122+0000 7ff7398b9640 1 -- 172.21.15.107:0/2991457981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720003520 con 0x7ff720003970 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.123+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff720003d90 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7ff73000c240 tx=0x7ff7200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46b137771ed93e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.123+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73002cb00 con 0x7ff720003970 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.123+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73002cc60 con 0x7ff720003970 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.123+0000 7ff7388b7640 1 -- 172.21.15.107:0/2991457981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730005af0 con 0x7ff720003970 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2991457981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 msgr2=0x7ff720003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2991457981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff720003d90 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7ff73000c240 tx=0x7ff7200057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a0ba640 1 -- 172.21.15.107:0/2991457981 reap_dead start 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2991457981 shutdown_connections 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2991457981 >> 172.21.15.107:0/2991457981 conn(0x7ff7340cbb40 msgr2=0x7ff7340cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2991457981 shutdown_connections 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.124+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2991457981 wait complete. 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.126+0000 7ff73a8bb640 1 Processor -- start 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.126+0000 7ff73a8bb640 1 -- start start 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73415e540 0x7ff7341f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7341f1920 0x7ff7340c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341f09c0 con 0x7ff720003970 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341f4f60 con 0x7ff73415e540 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341f49a0 con 0x7ff7341f1920 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44776/0 (socket says 172.21.15.107:44776) 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 -- 172.21.15.107:0/2742704557 learned_addr learned my addr 172.21.15.107:0/2742704557 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7390b8640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73415e540 0x7ff7341f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 -- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7341f1920 msgr2=0x7ff7340c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff73a0ba640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7341f1920 0x7ff7340c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7341f1920 0x7ff7340c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 -- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73415e540 msgr2=0x7ff7341f13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73415e540 0x7ff7341f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.127+0000 7ff7398b9640 1 -- 172.21.15.107:0/2742704557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7341f4c80 con 0x7ff720003970 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff7398b9640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7ff73002c4a0 tx=0x7ff730001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730039030 con 0x7ff720003970 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff730004bf0 con 0x7ff720003970 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73003c040 con 0x7ff720003970 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7341f0be0 con 0x7ff720003970 2023-11-24T08:21:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.128+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7341f4750 con 0x7ff720003970 2023-11-24T08:21:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.129+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73409d040 con 0x7ff720003970 2023-11-24T08:21:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.130+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ff730032070 con 0x7ff720003970 2023-11-24T08:21:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.131+0000 7ff72a7fc640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 0x7ff71806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.131+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff7300a9ac0 con 0x7ff720003970 2023-11-24T08:21:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.132+0000 7ff7390b8640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 0x7ff71806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.132+0000 7ff7390b8640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 0x7ff71806f280 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff73415e960 tx=0x7ff724008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.136+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7300765d0 con 0x7ff720003970 2023-11-24T08:21:27.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.326+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff73404e840 con 0x7ff720003970 2023-11-24T08:21:27.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.327+0000 7ff72a7fc640 1 -- 172.21.15.107:0/2742704557 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v193) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff730079bf0 con 0x7ff720003970 2023-11-24T08:21:27.329 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 msgr2=0x7ff71806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 0x7ff71806f280 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff73415e960 tx=0x7ff724008040 comp rx=0 tx=0).stop 2023-11-24T08:21:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 msgr2=0x7ff7341f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7ff73002c4a0 tx=0x7ff730001a00 comp rx=0 tx=0).stop 2023-11-24T08:21:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 shutdown_connections 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff71806cdc0 0x7ff71806f280 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7341f1920 0x7ff7340c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73415e540 0x7ff7341f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 --2- 172.21.15.107:0/2742704557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff720003970 0x7ff7341f4030 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 >> 172.21.15.107:0/2742704557 conn(0x7ff7340cbb40 msgr2=0x7ff7340aca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 shutdown_connections 2023-11-24T08:21:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.330+0000 7ff73a8bb640 1 -- 172.21.15.107:0/2742704557 wait complete. 2023-11-24T08:21:27.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:21:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.549+0000 7f2f74094640 1 Processor -- start 2023-11-24T08:21:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.549+0000 7f2f74094640 1 -- start start 2023-11-24T08:21:27.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f74094640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 0x7f2f6c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f74094640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f6c0c65f0 0x7f2f6c161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f74094640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c15de00 con 0x7f2f6c161940 2023-11-24T08:21:27.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f74094640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c6a70 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f74094640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c15e0e0 con 0x7f2f6c140520 2023-11-24T08:21:27.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f7162f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f6c0c65f0 0x7f2f6c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f7162f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2f6c0c65f0 0x7f2f6c161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44780/0 (socket says 172.21.15.107:44780) 2023-11-24T08:21:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f7162f640 1 -- 172.21.15.107:0/685307272 learned_addr learned my addr 172.21.15.107:0/685307272 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f72631640 1 --1- 172.21.15.107:0/685307272 >> v1:172.21.15.107:6789/0 conn(0x7f2f6c161940 0x7f2f6c142d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55218/0 (socket says 172.21.15.107:55218) 2023-11-24T08:21:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.551+0000 7f2f71e30640 1 --2- 172.21.15.107:0/685307272 >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 0x7f2f6c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.552+0000 7f2f7162f640 1 -- 172.21.15.107:0/685307272 >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 msgr2=0x7f2f6c0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.552+0000 7f2f7162f640 1 --2- 172.21.15.107:0/685307272 >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 0x7f2f6c0c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.552+0000 7f2f7162f640 1 -- 172.21.15.107:0/685307272 >> v1:172.21.15.107:6789/0 conn(0x7f2f6c161940 legacy=0x7f2f6c142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.552+0000 7f2f7162f640 1 -- 172.21.15.107:0/685307272 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c15e3c0 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.552+0000 7f2f71e30640 1 --2- 172.21.15.107:0/685307272 >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 0x7f2f6c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.553+0000 7f2f7162f640 1 --2- 172.21.15.107:0/685307272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c161400 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06e0b0 tx=0x7f2f5c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8875b626ae3eb225 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.553+0000 7f2f70e2e640 1 -- 172.21.15.107:0/685307272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c030870 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.553+0000 7f2f70e2e640 1 -- 172.21.15.107:0/685307272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0309d0 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.553+0000 7f2f70e2e640 1 -- 172.21.15.107:0/685307272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c030d60 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 -- 172.21.15.107:0/685307272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c0c65f0 msgr2=0x7f2f6c161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 --2- 172.21.15.107:0/685307272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c161400 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c06e0b0 tx=0x7f2f5c02b9b0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 -- 172.21.15.107:0/685307272 shutdown_connections 2023-11-24T08:21:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 --2- 172.21.15.107:0/685307272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c161400 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 --2- 172.21.15.107:0/685307272 >> v2:172.21.15.120:3300/0 conn(0x7f2f6c140520 0x7f2f6c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 -- 172.21.15.107:0/685307272 >> 172.21.15.107:0/685307272 conn(0x7f2f6c0b2e40 msgr2=0x7f2f6c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 -- 172.21.15.107:0/685307272 shutdown_connections 2023-11-24T08:21:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.554+0000 7f2f74094640 1 -- 172.21.15.107:0/685307272 wait complete. 2023-11-24T08:21:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.555+0000 7f2f74094640 1 Processor -- start 2023-11-24T08:21:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.556+0000 7f2f74094640 1 -- start start 2023-11-24T08:21:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.556+0000 7f2f74094640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c1f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f74094640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f6c140520 0x7f2f6c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f74094640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f74094640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0ce470 con 0x7f2f6c161940 2023-11-24T08:21:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f74094640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0ce750 con 0x7f2f6c0c65f0 2023-11-24T08:21:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f74094640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0cea30 con 0x7f2f6c140520 2023-11-24T08:21:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44796/0 (socket says 172.21.15.107:44796) 2023-11-24T08:21:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 -- 172.21.15.107:0/3282487478 learned_addr learned my addr 172.21.15.107:0/3282487478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 -- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f6c140520 msgr2=0x7f2f6c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f6c140520 0x7f2f6c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 -- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f6c0c65f0 msgr2=0x7f2f6c1f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c1f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.557+0000 7f2f72631640 1 -- 172.21.15.107:0/3282487478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0094d0 con 0x7f2f6c161940 2023-11-24T08:21:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f72631640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c730 tx=0x7f2f6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400ce40 con 0x7f2f6c161940 2023-11-24T08:21:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400a640 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c0ced10 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400a9b0 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.558+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c0d06c0 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.559+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c0cf230 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.560+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f2f6400ab10 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.561+0000 7f2f627fc640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 0x7f2f5006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.561+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2f64087d10 con 0x7f2f6c161940 2023-11-24T08:21:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.561+0000 7f2f71e30640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 0x7f2f5006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.562+0000 7f2f71e30640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 0x7f2f5006f280 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2f68006fd0 tx=0x7f2f68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.565+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f640bfe90 con 0x7f2f6c161940 2023-11-24T08:21:27.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.754+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f6c04e8b0 con 0x7f2f6c161940 2023-11-24T08:21:27.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.755+0000 7f2f627fc640 1 -- 172.21.15.107:0/3282487478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v193) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f64017070 con 0x7f2f6c161940 2023-11-24T08:21:27.757 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.757+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 msgr2=0x7f2f5006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.757+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 0x7f2f5006f280 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2f68006fd0 tx=0x7f2f68008040 comp rx=0 tx=0).stop 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.757+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 msgr2=0x7f2f6c0cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.757+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c730 tx=0x7f2f6400cbf0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 shutdown_connections 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2f5006cdc0 0x7f2f5006f280 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f6c161940 0x7f2f6c0cda90 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f6c140520 0x7f2f6c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 --2- 172.21.15.107:0/3282487478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f6c0c65f0 0x7f2f6c1f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 >> 172.21.15.107:0/3282487478 conn(0x7f2f6c0b2e40 msgr2=0x7f2f6c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 shutdown_connections 2023-11-24T08:21:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.758+0000 7f2f74094640 1 -- 172.21.15.107:0/3282487478 wait complete. 2023-11-24T08:21:27.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get new_pool pg_num 2023-11-24T08:21:27.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.983+0000 7f7dcc368640 1 Processor -- start 2023-11-24T08:21:27.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.984+0000 7f7dcc368640 1 -- start start 2023-11-24T08:21:27.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.985+0000 7f7dcc368640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc415b440 0x7f7dc40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.985+0000 7f7dcc368640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7dc40cbf20 0x7f7dc40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.985+0000 7f7dcc368640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40cc3a0 con 0x7f7dc40c57e0 2023-11-24T08:21:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.985+0000 7f7dcc368640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc415b820 con 0x7f7dc40cbf20 2023-11-24T08:21:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.985+0000 7f7dcc368640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc415eea0 con 0x7f7dc415b440 2023-11-24T08:21:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9903640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7dc40cbf20 0x7f7dc40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dca905640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7dc40c57e0 0x7f7dc40c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41574/0 (socket says 172.21.15.107:41574) 2023-11-24T08:21:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dca905640 1 -- 172.21.15.107:0/634084214 learned_addr learned my addr 172.21.15.107:0/634084214 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dca104640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc415b440 0x7f7dc40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9102640 1 -- 172.21.15.107:0/634084214 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733307247 0 0) 0x7f7dbc001490 con 0x7f7dc40c57e0 2023-11-24T08:21:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9102640 1 -- 172.21.15.107:0/634084214 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db0003520 con 0x7f7dc40c57e0 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9903640 1 -- 172.21.15.107:0/634084214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc415b440 msgr2=0x7f7dc40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9903640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc415b440 0x7f7dc40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9903640 1 -- 172.21.15.107:0/634084214 >> v1:172.21.15.107:6789/0 conn(0x7f7dc40c57e0 legacy=0x7f7dc40c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.986+0000 7f7dc9903640 1 -- 172.21.15.107:0/634084214 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc406d600 con 0x7f7dc40cbf20 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.987+0000 7f7dc9903640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40c52a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f7db40097a0 tx=0x7f7db402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae5502a3e5166a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dc9102640 1 -- 172.21.15.107:0/634084214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4031780 con 0x7f7dc40cbf20 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dc9102640 1 -- 172.21.15.107:0/634084214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db40318e0 con 0x7f7dc40cbf20 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dc9102640 1 -- 172.21.15.107:0/634084214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4031c70 con 0x7f7dc40cbf20 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 -- 172.21.15.107:0/634084214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc40cbf20 msgr2=0x7f7dc40c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40c52a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f7db40097a0 tx=0x7f7db402b820 comp rx=0 tx=0).stop 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 -- 172.21.15.107:0/634084214 shutdown_connections 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40c52a0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 --2- 172.21.15.107:0/634084214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc415b440 0x7f7dc40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 -- 172.21.15.107:0/634084214 >> 172.21.15.107:0/634084214 conn(0x7f7dc40b0c50 msgr2=0x7f7dc40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 -- 172.21.15.107:0/634084214 shutdown_connections 2023-11-24T08:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.988+0000 7f7dcc368640 1 -- 172.21.15.107:0/634084214 wait complete. 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.990+0000 7f7dcc368640 1 Processor -- start 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.990+0000 7f7dcc368640 1 -- start start 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40c57e0 0x7f7dc4154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40b1d10 con 0x7f7dc415b440 2023-11-24T08:21:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc41549c0 con 0x7f7dc40cbf20 2023-11-24T08:21:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dcc368640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40ceb80 con 0x7f7dc40c57e0 2023-11-24T08:21:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dca905640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dca905640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37402/0 (socket says 172.21.15.107:37402) 2023-11-24T08:21:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.991+0000 7f7dca905640 1 -- 172.21.15.107:0/563072752 learned_addr learned my addr 172.21.15.107:0/563072752 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca104640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40c57e0 0x7f7dc4154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dc9903640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 -- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40c57e0 msgr2=0x7f7dc4154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40c57e0 0x7f7dc4154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 -- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc40cbf20 msgr2=0x7f7dc40ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 -- 172.21.15.107:0/563072752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009400 con 0x7f7dc415b440 2023-11-24T08:21:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dca905640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00cb40 tx=0x7f7dbc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc00aca0 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc00ae00 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc012620 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.992+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc40ae970 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.993+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40cd400 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.994+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc40aeb60 con 0x7f7dc415b440 2023-11-24T08:21:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.995+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7dbc007070 con 0x7f7dc415b440 2023-11-24T08:21:27.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.996+0000 7f7dbaffd640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 0x7f7da806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:27.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.996+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7dbc0885b0 con 0x7f7dc415b440 2023-11-24T08:21:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.996+0000 7f7dca104640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 0x7f7da806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:27.997+0000 7f7dca104640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 0x7f7da806f280 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0006fd0 tx=0x7f7dc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.000+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7dbc0550c0 con 0x7f7dc415b440 2023-11-24T08:21:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.190+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"} v 0) v1 -- 0x7f7dc40cc3a0 con 0x7f7dc415b440 2023-11-24T08:21:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.191+0000 7f7dbaffd640 1 -- 172.21.15.107:0/563072752 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"}]=0 v193) v1 ==== 97+0+11 (secure 0 0 0) 0x7f7dc40cc3a0 con 0x7f7dc415b440 2023-11-24T08:21:28.193 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 msgr2=0x7f7da806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 0x7f7da806f280 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0006fd0 tx=0x7f7dc0008040 comp rx=0 tx=0).stop 2023-11-24T08:21:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 msgr2=0x7f7dc40cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00cb40 tx=0x7f7dbc00a060 comp rx=0 tx=0).stop 2023-11-24T08:21:28.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 shutdown_connections 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7da806cdc0 0x7f7da806f280 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7dc415b440 0x7f7dc40cef30 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc40cbf20 0x7f7dc40ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 --2- 172.21.15.107:0/563072752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40c57e0 0x7f7dc4154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 >> 172.21.15.107:0/563072752 conn(0x7f7dc40b0c50 msgr2=0x7f7dc40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 shutdown_connections 2023-11-24T08:21:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.194+0000 7f7dcc368640 1 -- 172.21.15.107:0/563072752 wait complete. 2023-11-24T08:21:28.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:21:28.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.430+0000 7fef67f43640 1 Processor -- start 2023-11-24T08:21:28.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.430+0000 7fef67f43640 1 -- start start 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef67f43640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef67f43640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef600c59b0 0x7fef6006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef67f43640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600cc3a0 con 0x7fef600cbf20 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef67f43640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6015b820 con 0x7fef600c59b0 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef67f43640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015eea0 con 0x7fef6015b440 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef664e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef600c59b0 0x7fef6006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef664e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef600c59b0 0x7fef6006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37410/0 (socket says 172.21.15.107:37410) 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.432+0000 7fef664e0640 1 -- 172.21.15.107:0/1036433609 learned_addr learned my addr 172.21.15.107:0/1036433609 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.433+0000 7fef65cdf640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.433+0000 7fef664e0640 1 -- 172.21.15.107:0/1036433609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 msgr2=0x7fef600cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.433+0000 7fef664e0640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.433+0000 7fef664e0640 1 -- 172.21.15.107:0/1036433609 >> v1:172.21.15.120:6789/0 conn(0x7fef600cbf20 legacy=0x7fef600c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.433+0000 7fef664e0640 1 -- 172.21.15.107:0/1036433609 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600abf30 con 0x7fef600c59b0 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.434+0000 7fef664e0640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6006c5f0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fef58009e50 tx=0x7fef5802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc18ac299f7bac5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.434+0000 7fef64cdd640 1 -- 172.21.15.107:0/1036433609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5802c9e0 con 0x7fef600c59b0 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.434+0000 7fef64cdd640 1 -- 172.21.15.107:0/1036433609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5802cb40 con 0x7fef600c59b0 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.434+0000 7fef64cdd640 1 -- 172.21.15.107:0/1036433609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef580343f0 con 0x7fef600c59b0 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 -- 172.21.15.107:0/1036433609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 msgr2=0x7fef6006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6006c5f0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fef58009e50 tx=0x7fef5802be40 comp rx=0 tx=0).stop 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 -- 172.21.15.107:0/1036433609 shutdown_connections 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6006c5f0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 --2- 172.21.15.107:0/1036433609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 -- 172.21.15.107:0/1036433609 >> 172.21.15.107:0/1036433609 conn(0x7fef600b0c50 msgr2=0x7fef600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 -- 172.21.15.107:0/1036433609 shutdown_connections 2023-11-24T08:21:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.435+0000 7fef67f43640 1 -- 172.21.15.107:0/1036433609 wait complete. 2023-11-24T08:21:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.437+0000 7fef67f43640 1 Processor -- start 2023-11-24T08:21:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.437+0000 7fef67f43640 1 -- start start 2023-11-24T08:21:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 0x7fef600acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6014aff0 con 0x7fef600c59b0 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600c60d0 con 0x7fef600cbf20 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef67f43640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600c63b0 con 0x7fef6015b440 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37416/0 (socket says 172.21.15.107:37416) 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 -- 172.21.15.107:0/3330737362 learned_addr learned my addr 172.21.15.107:0/3330737362 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef654de640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 0x7fef600acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef664e0640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 -- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 msgr2=0x7fef600ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 -- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 msgr2=0x7fef600acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 0x7fef600acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef65cdf640 1 -- 172.21.15.107:0/3330737362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef58009b30 con 0x7fef600c59b0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.438+0000 7fef654de640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 0x7fef600acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef65cdf640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00d5f0 tx=0x7fef5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0125f0 con 0x7fef600c59b0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c012750 con 0x7fef600c59b0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c012b00 con 0x7fef600c59b0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef600c6690 con 0x7fef600c59b0 2023-11-24T08:21:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.439+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef600c6c50 con 0x7fef600c59b0 2023-11-24T08:21:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.440+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6006ef60 con 0x7fef600c59b0 2023-11-24T08:21:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.441+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fef5c012c60 con 0x7fef600c59b0 2023-11-24T08:21:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.442+0000 7fef567fc640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 0x7fef4406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.442+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fef5c088b80 con 0x7fef600c59b0 2023-11-24T08:21:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.443+0000 7fef654de640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 0x7fef4406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.443+0000 7fef654de640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 0x7fef4406f280 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fef6015ee70 tx=0x7fef50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.447+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef5c055690 con 0x7fef600c59b0 2023-11-24T08:21:28.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.651+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fef6004e8b0 con 0x7fef600c59b0 2023-11-24T08:21:28.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.652+0000 7fef567fc640 1 -- 172.21.15.107:0/3330737362 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v193) v1 ==== 80+0+0 (secure 0 0 0) 0x7fef5c058cb0 con 0x7fef600c59b0 2023-11-24T08:21:28.654 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:21:28.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 msgr2=0x7fef4406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 0x7fef4406f280 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fef6015ee70 tx=0x7fef50009040 comp rx=0 tx=0).stop 2023-11-24T08:21:28.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 msgr2=0x7fef6014aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00d5f0 tx=0x7fef5c00dab0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 shutdown_connections 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fef4406cdc0 0x7fef4406f280 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6015b440 0x7fef600ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef600cbf20 0x7fef600acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 --2- 172.21.15.107:0/3330737362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef600c59b0 0x7fef6014aab0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:28.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 >> 172.21.15.107:0/3330737362 conn(0x7fef600b0c50 msgr2=0x7fef600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:28.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 shutdown_connections 2023-11-24T08:21:28.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:28.655+0000 7fef67f43640 1 -- 172.21.15.107:0/3330737362 wait complete. 2023-11-24T08:21:28.676 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:21:28.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:28.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:21:28.788 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:21:28.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:28.789 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:21:28.855 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:21:28.855 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:28.896 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:21:28.897 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:28.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:29.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.196+0000 7f658e6ad640 1 Processor -- start 2023-11-24T08:21:29.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.196+0000 7f658e6ad640 1 -- start start 2023-11-24T08:21:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658e6ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f658815c7f0 0x7f658806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658e6ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f65880cf450 0x7f65880cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658e6ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65880cece0 con 0x7f65880cdfb0 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658e6ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588156010 con 0x7f65880cf450 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658e6ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658806ce30 con 0x7f658815c7f0 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f657f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f65880cf450 0x7f65880cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658cc4a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f65880cdfb0 0x7f65880ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41588/0 (socket says 172.21.15.107:41588) 2023-11-24T08:21:29.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.198+0000 7f658cc4a640 1 -- 172.21.15.107:0/1688018968 learned_addr learned my addr 172.21.15.107:0/1688018968 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:29.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.199+0000 7f6587fff640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f658815c7f0 0x7f658806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.199+0000 7f657f7fe640 1 -- 172.21.15.107:0/1688018968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f658815c7f0 msgr2=0x7f658806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:29.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.199+0000 7f657f7fe640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f658815c7f0 0x7f658806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.199+0000 7f657f7fe640 1 -- 172.21.15.107:0/1688018968 >> v1:172.21.15.107:6789/0 conn(0x7f65880cdfb0 legacy=0x7f65880ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.199+0000 7f657f7fe640 1 -- 172.21.15.107:0/1688018968 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658806d040 con 0x7f65880cf450 2023-11-24T08:21:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.200+0000 7f657f7fe640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cf450 0x7f65880cf8d0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f65700097a0 tx=0x7f657002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e718690dcad40595 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.200+0000 7f65877fe640 1 -- 172.21.15.107:0/1688018968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657002d040 con 0x7f65880cf450 2023-11-24T08:21:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.200+0000 7f65877fe640 1 -- 172.21.15.107:0/1688018968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570031810 con 0x7f65880cf450 2023-11-24T08:21:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.200+0000 7f65877fe640 1 -- 172.21.15.107:0/1688018968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570031ba0 con 0x7f65880cf450 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 -- 172.21.15.107:0/1688018968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cf450 msgr2=0x7f65880cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cf450 0x7f65880cf8d0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f65700097a0 tx=0x7f657002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 -- 172.21.15.107:0/1688018968 shutdown_connections 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cf450 0x7f65880cf8d0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 --2- 172.21.15.107:0/1688018968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f658815c7f0 0x7f658806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 -- 172.21.15.107:0/1688018968 >> 172.21.15.107:0/1688018968 conn(0x7f658806a920 msgr2=0x7f658806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 -- 172.21.15.107:0/1688018968 shutdown_connections 2023-11-24T08:21:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.201+0000 7f658e6ad640 1 -- 172.21.15.107:0/1688018968 wait complete. 2023-11-24T08:21:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.203+0000 7f658e6ad640 1 Processor -- start 2023-11-24T08:21:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.203+0000 7f658e6ad640 1 -- start start 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65880cf450 0x7f6588156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 0x7f65880ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880ad9f0 con 0x7f65880cdfb0 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880ace50 con 0x7f658815c7f0 2023-11-24T08:21:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658e6ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6588151aa0 con 0x7f65880cf450 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37438/0 (socket says 172.21.15.107:37438) 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 -- 172.21.15.107:0/3979045935 learned_addr learned my addr 172.21.15.107:0/3979045935 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f657f7fe640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65880cf450 0x7f6588156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f658cc4a640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 0x7f65880ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 -- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65880cf450 msgr2=0x7f6588156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65880cf450 0x7f6588156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.204+0000 7f6587fff640 1 -- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 msgr2=0x7f65880ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f6587fff640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 0x7f65880ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f6587fff640 1 -- 172.21.15.107:0/3979045935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570009400 con 0x7f65880cdfb0 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f658cc4a640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 0x7f65880ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f6587fff640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f657800cd30 tx=0x7f657800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578003990 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578003af0 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578003e80 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65880ad0c0 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.205+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65880ad680 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.206+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6588067470 con 0x7f65880cdfb0 2023-11-24T08:21:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.207+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f6578004020 con 0x7f65880cdfb0 2023-11-24T08:21:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.208+0000 7f65857fa640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 0x7f656806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.208+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f6578088a90 con 0x7f65880cdfb0 2023-11-24T08:21:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.209+0000 7f657f7fe640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 0x7f656806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.210+0000 7f657f7fe640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 0x7f656806f280 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6570036000 tx=0x7f657003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.213+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65780555a0 con 0x7f65880cdfb0 2023-11-24T08:21:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.408+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6588155aa0 con 0x7f65880cdfb0 2023-11-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.410+0000 7f65857fa640 1 -- 172.21.15.107:0/3979045935 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+18584 (secure 0 0 0) 0x7f657801d030 con 0x7f65880cdfb0 2023-11-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":194,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:28.833824+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.412+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 msgr2=0x7f656806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.412+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 0x7f656806f280 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6570036000 tx=0x7f657003c040 comp rx=0 tx=0).stop 2023-11-24T08:21:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 msgr2=0x7f6588151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f657800cd30 tx=0x7f657800a5c0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 shutdown_connections 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f656806cdc0 0x7f656806f280 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f658815c7f0 0x7f65880ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65880cf450 0x7f6588156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 --2- 172.21.15.107:0/3979045935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65880cdfb0 0x7f6588151560 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 >> 172.21.15.107:0/3979045935 conn(0x7f658806a920 msgr2=0x7f65880cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 shutdown_connections 2023-11-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.413+0000 7f658e6ad640 1 -- 172.21.15.107:0/3979045935 wait complete. 2023-11-24T08:21:29.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:20:30.467990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:20:31.721268+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'new_pool', 'create_time': '2023-11-24T08:20:48.209202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:21:29.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.642+0000 7ffaf2f21640 1 Processor -- start 2023-11-24T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.643+0000 7ffaf2f21640 1 -- start start 2023-11-24T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.644+0000 7ffaf2f21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffaec0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.644+0000 7ffaf2f21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec06d430 con 0x7ffaec15a1c0 2023-11-24T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.644+0000 7ffaf2f21640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b14e0 con 0x7ffaec15ded0 2023-11-24T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.644+0000 7ffaf2f21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15ea70 con 0x7ffaec0b0b40 2023-11-24T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.644+0000 7ffaf1f1f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffaec15a1c0 0x7ffaec06d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41602/0 (socket says 172.21.15.107:41602) 2023-11-24T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf1f1f640 1 -- 172.21.15.107:0/2377345060 learned_addr learned my addr 172.21.15.107:0/2377345060 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777123411 0 0) 0x7ffaec06d430 con 0x7ffaec15a1c0 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf171e640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffaec0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8005730 con 0x7ffaec15a1c0 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128956818 0 0) 0x7ffad8005730 con 0x7ffaec15a1c0 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 msgr2=0x7ffaec0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffaec0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 >> v1:172.21.15.120:6789/0 conn(0x7ffaec15ded0 legacy=0x7ffaec15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec15ec50 con 0x7ffaec15a1c0 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ffae8001670 con 0x7ffaec15a1c0 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.645+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 >> v1:172.21.15.107:6789/0 conn(0x7ffaec15a1c0 legacy=0x7ffaec06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8005bb0 0x7ffad8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffad80069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad8006f10 0x7ffad8007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15ea70 con 0x7ffad8006f10 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800dc40 con 0x7ffad8005bb0 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800de00 con 0x7ffaec0b0b40 2023-11-24T08:21:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad8006f10 0x7ffad8007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8005bb0 0x7ffad8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf171e640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffad80069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 -- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 msgr2=0x7ffad80069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffad80069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 -- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8005bb0 msgr2=0x7ffad8005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8005bb0 0x7ffad8005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf2720640 1 -- 172.21.15.107:0/2377345060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad8005730 con 0x7ffad8006f10 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf171e640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b0b40 0x7ffad80069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:29.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.646+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8005bb0 0x7ffad8005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:29.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.647+0000 7ffaf2720640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad8006f10 0x7ffad8007390 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7ffae4001e20 tx=0x7ffad8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b188f83668b2a8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4003b70 con 0x7ffad8006f10 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae4003cd0 con 0x7ffad8006f10 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf0f1d640 1 -- 172.21.15.107:0/2377345060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4005310 con 0x7ffad8006f10 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf2f21640 1 -- 172.21.15.107:0/2377345060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad8006f10 msgr2=0x7ffad8007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/2377345060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad8006f10 0x7ffad8007390 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7ffae4001e20 tx=0x7ffad8007a00 comp rx=0 tx=0).stop 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf2720640 1 -- 172.21.15.107:0/2377345060 reap_dead start 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf2f21640 1 -- 172.21.15.107:0/2377345060 shutdown_connections 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.648+0000 7ffaf2f21640 1 -- 172.21.15.107:0/2377345060 >> 172.21.15.107:0/2377345060 conn(0x7ffaec0cbb40 msgr2=0x7ffaec0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.649+0000 7ffaf2f21640 1 -- 172.21.15.107:0/2377345060 shutdown_connections 2023-11-24T08:21:29.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.649+0000 7ffaf2f21640 1 -- 172.21.15.107:0/2377345060 wait complete. 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.650+0000 7ffaf2f21640 1 Processor -- start 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.650+0000 7ffaf2f21640 1 -- start start 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 0x7ffaec1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 0x7ffaec0b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec1ee9a0 con 0x7ffaec0b4000 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec158a20 con 0x7ffad8006f10 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.651+0000 7ffaf2f21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec158d00 con 0x7ffaec0b4480 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37454/0 (socket says 172.21.15.107:37454) 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 -- 172.21.15.107:0/3738411141 learned_addr learned my addr 172.21.15.107:0/3738411141 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:29.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 0x7ffaec1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 -- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 msgr2=0x7ffaec0b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf2720640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 0x7ffaec0b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 0x7ffaec0b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 -- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 msgr2=0x7ffaec1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 0x7ffaec1581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 -- 172.21.15.107:0/3738411141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec158fe0 con 0x7ffaec0b4000 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 0x7ffaec1581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf2720640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 0x7ffaec0b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.652+0000 7ffaf171e640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0108b0 tx=0x7ffadc010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.653+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc00eb60 con 0x7ffaec0b4000 2023-11-24T08:21:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.653+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc00ecc0 con 0x7ffaec0b4000 2023-11-24T08:21:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.653+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec1592c0 con 0x7ffaec0b4000 2023-11-24T08:21:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.653+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc01c400 con 0x7ffaec0b4000 2023-11-24T08:21:29.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.653+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec159ac0 con 0x7ffaec0b4000 2023-11-24T08:21:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.654+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec04e840 con 0x7ffaec0b4000 2023-11-24T08:21:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.655+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ffadc01c560 con 0x7ffaec0b4000 2023-11-24T08:21:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.656+0000 7ffae27fc640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 0x7ffad006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.656+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7ffadc088cb0 con 0x7ffaec0b4000 2023-11-24T08:21:29.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.656+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 0x7ffad006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.657+0000 7ffaf1f1f640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 0x7ffad006f280 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ffae8001eb0 tx=0x7ffae8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:29.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.660+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffadc055870 con 0x7ffaec0b4000 2023-11-24T08:21:29.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.855+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaec0ca020 con 0x7ffaec0b4000 2023-11-24T08:21:29.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.856+0000 7ffae27fc640 1 -- 172.21.15.107:0/3738411141 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffadc058e90 con 0x7ffaec0b4000 2023-11-24T08:21:29.858 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.858+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 msgr2=0x7ffad006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.858+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 0x7ffad006f280 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ffae8001eb0 tx=0x7ffae8009040 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.858+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 msgr2=0x7ffaec0b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.858+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0108b0 tx=0x7ffadc010d70 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 shutdown_connections 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffad006cdc0 0x7ffad006f280 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaec0b4480 0x7ffaec0b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec0b4000 0x7ffaec0b1bf0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 --2- 172.21.15.107:0/3738411141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffad8006f10 0x7ffaec1581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 >> 172.21.15.107:0/3738411141 conn(0x7ffaec0cbb40 msgr2=0x7ffaec0ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 shutdown_connections 2023-11-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:29.859+0000 7ffaf2f21640 1 -- 172.21.15.107:0/3738411141 wait complete. 2023-11-24T08:21:29.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:21:30.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.097+0000 7f4c0699e640 1 Processor -- start 2023-11-24T08:21:30.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.098+0000 7f4c0699e640 1 -- start start 2023-11-24T08:21:30.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.099+0000 7f4c0699e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c0015b440 0x7f4c000cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.099+0000 7f4c0699e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000cbf20 0x7f4c000c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.099+0000 7f4c0699e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0015eea0 con 0x7f4c000c57e0 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.099+0000 7f4c0699e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c000cc3d0 con 0x7f4c0015b440 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.099+0000 7f4c0699e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0015b820 con 0x7f4c000cbf20 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c0015b440 0x7f4c000cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c0015b440 0x7f4c000cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37468/0 (socket says 172.21.15.107:37468) 2023-11-24T08:21:30.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 -- 172.21.15.107:0/2176185181 learned_addr learned my addr 172.21.15.107:0/2176185181 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bf7fff640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000cbf20 0x7f4c000c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 -- 172.21.15.107:0/2176185181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000cbf20 msgr2=0x7f4c000c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000cbf20 0x7f4c000c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 -- 172.21.15.107:0/2176185181 >> v1:172.21.15.120:6789/0 conn(0x7f4c000c57e0 legacy=0x7f4c000c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.100+0000 7f4bfffff640 1 -- 172.21.15.107:0/2176185181 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0006d600 con 0x7f4c0015b440 2023-11-24T08:21:30.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.102+0000 7f4bfffff640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0015b440 0x7f4c000cb9e0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009870 tx=0x7f4bf002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342db04b766c5125 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.102+0000 7f4bff7fe640 1 -- 172.21.15.107:0/2176185181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00317c0 con 0x7f4c0015b440 2023-11-24T08:21:30.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.102+0000 7f4bff7fe640 1 -- 172.21.15.107:0/2176185181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf0031920 con 0x7f4c0015b440 2023-11-24T08:21:30.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.102+0000 7f4bff7fe640 1 -- 172.21.15.107:0/2176185181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0031cb0 con 0x7f4c0015b440 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 -- 172.21.15.107:0/2176185181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0015b440 msgr2=0x7f4c000cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0015b440 0x7f4c000cb9e0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009870 tx=0x7f4bf002b860 comp rx=0 tx=0).stop 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 -- 172.21.15.107:0/2176185181 shutdown_connections 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000cbf20 0x7f4c000c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 --2- 172.21.15.107:0/2176185181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0015b440 0x7f4c000cb9e0 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 -- 172.21.15.107:0/2176185181 >> 172.21.15.107:0/2176185181 conn(0x7f4c000b0c50 msgr2=0x7f4c000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 -- 172.21.15.107:0/2176185181 shutdown_connections 2023-11-24T08:21:30.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.103+0000 7f4c0699e640 1 -- 172.21.15.107:0/2176185181 wait complete. 2023-11-24T08:21:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.105+0000 7f4c0699e640 1 Processor -- start 2023-11-24T08:21:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.105+0000 7f4c0699e640 1 -- start start 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 0x7f4c000ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c0015b440 0x7f4c00156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000b4a50 con 0x7f4c000cbf20 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0015c390 con 0x7f4c0015b440 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4c0699e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000b4770 con 0x7f4c000c57e0 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37482/0 (socket says 172.21.15.107:37482) 2023-11-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.106+0000 7f4bf7fff640 1 -- 172.21.15.107:0/718013432 learned_addr learned my addr 172.21.15.107:0/718013432 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bfffff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 0x7f4c000ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4c04f3b640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c0015b440 0x7f4c00156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 -- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 msgr2=0x7f4c000ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 0x7f4c000ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 -- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c0015b440 msgr2=0x7f4c00156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c0015b440 0x7f4c00156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 -- 172.21.15.107:0/718013432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00094d0 con 0x7f4c000cbf20 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bfffff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 0x7f4c000ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.107+0000 7f4bf7fff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d830 tx=0x7f4be800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.108+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be800b6c0 con 0x7f4c000cbf20 2023-11-24T08:21:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.108+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be800b820 con 0x7f4c000cbf20 2023-11-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.108+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be800bbb0 con 0x7f4c000cbf20 2023-11-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.108+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0015bcd0 con 0x7f4c000cbf20 2023-11-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.108+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c000ce830 con 0x7f4c000cbf20 2023-11-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.109+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0015bb30 con 0x7f4c000cbf20 2023-11-24T08:21:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.110+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f4be801e460 con 0x7f4c000cbf20 2023-11-24T08:21:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.111+0000 7f4bfd7fa640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 0x7f4be006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.111+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4be8088e60 con 0x7f4c000cbf20 2023-11-24T08:21:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.111+0000 7f4bfffff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 0x7f4be006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.112+0000 7f4bfffff640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 0x7f4be006f280 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0036000 tx=0x7f4bf003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.116+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c0015bb30 con 0x7f4c000cbf20 2023-11-24T08:21:30.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.305+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c0015bb30 con 0x7f4c000cbf20 2023-11-24T08:21:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.306+0000 7f4bfd7fa640 1 -- 172.21.15.107:0/718013432 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v194) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c0015bb30 con 0x7f4c000cbf20 2023-11-24T08:21:30.308 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 msgr2=0x7f4be006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 0x7f4be006f280 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0036000 tx=0x7f4bf003d040 comp rx=0 tx=0).stop 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 msgr2=0x7f4c000b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d830 tx=0x7f4be800dcf0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 shutdown_connections 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4be006cdc0 0x7f4be006f280 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c0015b440 0x7f4c00156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c000cbf20 0x7f4c000b3e00 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 --2- 172.21.15.107:0/718013432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c000c57e0 0x7f4c000ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 >> 172.21.15.107:0/718013432 conn(0x7f4c000b0c50 msgr2=0x7f4c000cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 shutdown_connections 2023-11-24T08:21:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.309+0000 7f4c0699e640 1 -- 172.21.15.107:0/718013432 wait complete. 2023-11-24T08:21:30.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:21:30.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.533+0000 7f85de639640 1 Processor -- start 2023-11-24T08:21:30.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.534+0000 7f85de639640 1 -- start start 2023-11-24T08:21:30.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85de639640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85de639640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85d80d1f40 0x7f85d80ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85de639640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80d15e0 con 0x7f85d815d3f0 2023-11-24T08:21:30.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85de639640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80d0280 con 0x7f85d80d1f40 2023-11-24T08:21:30.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85de639640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80b0eb0 con 0x7f85d815d7d0 2023-11-24T08:21:30.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85dd637640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85d815d3f0 0x7f85d80d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41606/0 (socket says 172.21.15.107:41606) 2023-11-24T08:21:30.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85dd637640 1 -- 172.21.15.107:0/283494749 learned_addr learned my addr 172.21.15.107:0/283494749 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:30.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.535+0000 7f85dde38640 1 --2- 172.21.15.107:0/283494749 >> v2:172.21.15.107:3300/0 conn(0x7f85d80d1f40 0x7f85d80ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dce36640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85d7fff640 1 -- 172.21.15.107:0/283494749 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709690541 0 0) 0x7f85d80d15e0 con 0x7f85d815d3f0 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85d7fff640 1 -- 172.21.15.107:0/283494749 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c0003520 con 0x7f85d815d3f0 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dde38640 1 -- 172.21.15.107:0/283494749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 msgr2=0x7f85d80cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dde38640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dde38640 1 -- 172.21.15.107:0/283494749 >> v1:172.21.15.107:6789/0 conn(0x7f85d815d3f0 legacy=0x7f85d80d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dde38640 1 -- 172.21.15.107:0/283494749 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80d1300 con 0x7f85d80d1f40 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.536+0000 7f85dce36640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.537+0000 7f85dde38640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80ce280 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097a0 tx=0x7f85d002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d1dc709278983c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.537+0000 7f85d7fff640 1 -- 172.21.15.107:0/283494749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d002d040 con 0x7f85d80d1f40 2023-11-24T08:21:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.537+0000 7f85d7fff640 1 -- 172.21.15.107:0/283494749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0031810 con 0x7f85d80d1f40 2023-11-24T08:21:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.537+0000 7f85d7fff640 1 -- 172.21.15.107:0/283494749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0031ba0 con 0x7f85d80d1f40 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 -- 172.21.15.107:0/283494749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 msgr2=0x7f85d80ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80ce280 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097a0 tx=0x7f85d002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 -- 172.21.15.107:0/283494749 shutdown_connections 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80ce280 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 --2- 172.21.15.107:0/283494749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 -- 172.21.15.107:0/283494749 >> 172.21.15.107:0/283494749 conn(0x7f85d809c5b0 msgr2=0x7f85d80d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 -- 172.21.15.107:0/283494749 shutdown_connections 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.538+0000 7f85de639640 1 -- 172.21.15.107:0/283494749 wait complete. 2023-11-24T08:21:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.540+0000 7f85de639640 1 Processor -- start 2023-11-24T08:21:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.540+0000 7f85de639640 1 -- start start 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85d815d3f0 0x7f85d8151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80b4d40 con 0x7f85d80d1f40 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80b3e30 con 0x7f85d815d3f0 2023-11-24T08:21:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85de639640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80cc740 con 0x7f85d815d7d0 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85dd637640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85dd637640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37502/0 (socket says 172.21.15.107:37502) 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.541+0000 7f85dd637640 1 -- 172.21.15.107:0/1342420262 learned_addr learned my addr 172.21.15.107:0/1342420262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 -- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 msgr2=0x7f85d80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 -- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85d815d3f0 msgr2=0x7f85d8151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85d815d3f0 0x7f85d8151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 -- 172.21.15.107:0/1342420262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d0009400 con 0x7f85d80d1f40 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85dd637640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f85c800cbf0 tx=0x7f85c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800acf0 con 0x7f85d80d1f40 2023-11-24T08:21:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800ae50 con 0x7f85d80d1f40 2023-11-24T08:21:30.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d8151a90 con 0x7f85d80d1f40 2023-11-24T08:21:30.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.542+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8012620 con 0x7f85d80d1f40 2023-11-24T08:21:30.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.543+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d80cc0f0 con 0x7f85d80d1f40 2023-11-24T08:21:30.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.544+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d804e840 con 0x7f85d80d1f40 2023-11-24T08:21:30.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.545+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f85c8007070 con 0x7f85d80d1f40 2023-11-24T08:21:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.546+0000 7f85d5ffb640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 0x7f85c406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.546+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f85c8088700 con 0x7f85d80d1f40 2023-11-24T08:21:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.546+0000 7f85dce36640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 0x7f85c406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.547+0000 7f85dce36640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 0x7f85c406f280 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.551+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85c8055210 con 0x7f85d80d1f40 2023-11-24T08:21:30.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.740+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85d815be20 con 0x7f85d80d1f40 2023-11-24T08:21:30.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.741+0000 7f85d5ffb640 1 -- 172.21.15.107:0/1342420262 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v194) v1 ==== 100+0+11 (secure 0 0 0) 0x7f85c8017020 con 0x7f85d80d1f40 2023-11-24T08:21:30.743 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 msgr2=0x7f85c406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 0x7f85c406f280 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc008040 comp rx=0 tx=0).stop 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 msgr2=0x7f85d80b5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f85c800cbf0 tx=0x7f85c800a430 comp rx=0 tx=0).stop 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 shutdown_connections 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f85c406cdc0 0x7f85c406f280 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85d815d7d0 0x7f85d80ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85d815d3f0 0x7f85d8151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 --2- 172.21.15.107:0/1342420262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85d80d1f40 0x7f85d80b5250 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 >> 172.21.15.107:0/1342420262 conn(0x7f85d809c5b0 msgr2=0x7f85d8160240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:30.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 shutdown_connections 2023-11-24T08:21:30.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.744+0000 7f85de639640 1 -- 172.21.15.107:0/1342420262 wait complete. 2023-11-24T08:21:30.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get new_pool pg_num 2023-11-24T08:21:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.976+0000 7fc183a9b640 1 Processor -- start 2023-11-24T08:21:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.976+0000 7fc183a9b640 1 -- start start 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc183a9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc17c0ccbd0 0x7fc17c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc183a9b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c1655f0 con 0x7fc17c0c9650 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc183a9b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c0cc380 con 0x7fc17c0ccfb0 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc183a9b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c162b30 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc181837640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc17c0ccbd0 0x7fc17c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc182038640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc17c0c9650 0x7fc17c0c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41616/0 (socket says 172.21.15.107:41616) 2023-11-24T08:21:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.978+0000 7fc182038640 1 -- 172.21.15.107:0/3779838403 learned_addr learned my addr 172.21.15.107:0/3779838403 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:30.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.979+0000 7fc181837640 1 -- 172.21.15.107:0/3779838403 >> v1:172.21.15.120:6789/0 conn(0x7fc17c0ccfb0 legacy=0x7fc17c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:30.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.979+0000 7fc181837640 1 -- 172.21.15.107:0/3779838403 >> v1:172.21.15.107:6789/0 conn(0x7fc17c0c9650 legacy=0x7fc17c0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.979+0000 7fc181837640 1 -- 172.21.15.107:0/3779838403 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c162e10 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.979+0000 7fc180835640 1 -- 172.21.15.107:0/3779838403 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641992011 0 0) 0x7fc17c1655f0 con 0x7fc17c0c9650 2023-11-24T08:21:30.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.980+0000 7fc181837640 1 --2- 172.21.15.107:0/3779838403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c1660b0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fc178009d90 tx=0x7fc17802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcaa7759049a093 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc180835640 1 -- 172.21.15.107:0/3779838403 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17802cc60 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc180835640 1 -- 172.21.15.107:0/3779838403 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17802cdc0 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc180835640 1 -- 172.21.15.107:0/3779838403 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178034660 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 -- 172.21.15.107:0/3779838403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0ccbd0 msgr2=0x7fc17c1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 --2- 172.21.15.107:0/3779838403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c1660b0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fc178009d90 tx=0x7fc17802beb0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 -- 172.21.15.107:0/3779838403 shutdown_connections 2023-11-24T08:21:30.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 --2- 172.21.15.107:0/3779838403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c1660b0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 -- 172.21.15.107:0/3779838403 >> 172.21.15.107:0/3779838403 conn(0x7fc17c0b0c50 msgr2=0x7fc17c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:30.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 -- 172.21.15.107:0/3779838403 shutdown_connections 2023-11-24T08:21:30.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.981+0000 7fc183a9b640 1 -- 172.21.15.107:0/3779838403 wait complete. 2023-11-24T08:21:30.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.983+0000 7fc183a9b640 1 Processor -- start 2023-11-24T08:21:30.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.983+0000 7fc183a9b640 1 -- start start 2023-11-24T08:21:30.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc17c0ccfb0 0x7fc17c1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c158350 con 0x7fc17c0c9650 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0ad800 con 0x7fc17c0ccbd0 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.984+0000 7fc183a9b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c15b100 con 0x7fc17c0ccfb0 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37534/0 (socket says 172.21.15.107:37534) 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 -- 172.21.15.107:0/2789865316 learned_addr learned my addr 172.21.15.107:0/2789865316 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 -- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc17c0ccfb0 msgr2=0x7fc17c1594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc17c0ccfb0 0x7fc17c1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 -- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc17c0ccbd0 msgr2=0x7fc17c158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181036640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 -- 172.21.15.107:0/2789865316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178009a70 con 0x7fc17c0c9650 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.985+0000 7fc181837640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fc178037040 tx=0x7fc178004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.986+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178040650 con 0x7fc17c0c9650 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.986+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1780407b0 con 0x7fc17c0c9650 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.986+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178040b60 con 0x7fc17c0c9650 2023-11-24T08:21:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.986+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c15b3e0 con 0x7fc17c0c9650 2023-11-24T08:21:30.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.986+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c15b9a0 con 0x7fc17c0c9650 2023-11-24T08:21:30.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.987+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c15bb30 con 0x7fc17c0c9650 2023-11-24T08:21:30.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.988+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fc178040cc0 con 0x7fc17c0c9650 2023-11-24T08:21:30.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.989+0000 7fc1727fc640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 0x7fc16006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:30.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.990+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fc1780ab6e0 con 0x7fc17c0c9650 2023-11-24T08:21:30.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.990+0000 7fc181036640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 0x7fc16006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:30.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.990+0000 7fc181036640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 0x7fc16006f280 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0b0410 tx=0x7fc16c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:30.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:30.994+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1780781f0 con 0x7fc17c0c9650 2023-11-24T08:21:31.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.183+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"} v 0) v1 -- 0x7fc17c0cd3f0 con 0x7fc17c0c9650 2023-11-24T08:21:31.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.184+0000 7fc1727fc640 1 -- 172.21.15.107:0/2789865316 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"}]=0 v194) v1 ==== 97+0+11 (secure 0 0 0) 0x7fc17c0cd3f0 con 0x7fc17c0c9650 2023-11-24T08:21:31.186 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 msgr2=0x7fc16006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 0x7fc16006f280 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0b0410 tx=0x7fc16c009040 comp rx=0 tx=0).stop 2023-11-24T08:21:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 msgr2=0x7fc17c157e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fc178037040 tx=0x7fc178004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 shutdown_connections 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc16006cdc0 0x7fc16006f280 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc17c0ccfb0 0x7fc17c1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc17c0ccbd0 0x7fc17c158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 --2- 172.21.15.107:0/2789865316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc17c0c9650 0x7fc17c157e10 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 >> 172.21.15.107:0/2789865316 conn(0x7fc17c0b0c50 msgr2=0x7fc17c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 shutdown_connections 2023-11-24T08:21:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.186+0000 7fc183a9b640 1 -- 172.21.15.107:0/2789865316 wait complete. 2023-11-24T08:21:31.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:21:31.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.407+0000 7f1196a0e640 1 Processor -- start 2023-11-24T08:21:31.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.407+0000 7f1196a0e640 1 -- start start 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1190072160 0x7f119015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11900c49f0 0x7f119006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f119006dc40 0x7f11900cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119006e0d0 con 0x7f11900c49f0 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119015ec90 con 0x7f1190072160 2023-11-24T08:21:31.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f1196a0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900cd530 con 0x7f119006dc40 2023-11-24T08:21:31.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f11877fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11900c49f0 0x7f119006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f11877fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11900c49f0 0x7f119006d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37546/0 (socket says 172.21.15.107:37546) 2023-11-24T08:21:31.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.409+0000 7f11877fe640 1 -- 172.21.15.107:0/3737829250 learned_addr learned my addr 172.21.15.107:0/3737829250 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:31.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f118ffff640 1 --2- 172.21.15.107:0/3737829250 >> v2:172.21.15.120:3300/0 conn(0x7f1190072160 0x7f119015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f1194fab640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f119006dc40 0x7f11900cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f11877fe640 1 -- 172.21.15.107:0/3737829250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f119006dc40 msgr2=0x7f11900cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f11877fe640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f119006dc40 0x7f11900cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f11877fe640 1 -- 172.21.15.107:0/3737829250 >> v2:172.21.15.120:3300/0 conn(0x7f1190072160 msgr2=0x7f119015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f11877fe640 1 --2- 172.21.15.107:0/3737829250 >> v2:172.21.15.120:3300/0 conn(0x7f1190072160 0x7f119015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.410+0000 7f11877fe640 1 -- 172.21.15.107:0/3737829250 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900c4ed0 con 0x7f11900c49f0 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.411+0000 7f11877fe640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11900c49f0 0x7f119006d700 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f11900ccae0 tx=0x7f117802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e39bb55a7a3c89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.411+0000 7f118f7fe640 1 -- 172.21.15.107:0/3737829250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178031860 con 0x7f11900c49f0 2023-11-24T08:21:31.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.411+0000 7f118f7fe640 1 -- 172.21.15.107:0/3737829250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11780319c0 con 0x7f11900c49f0 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.411+0000 7f118f7fe640 1 -- 172.21.15.107:0/3737829250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178031d50 con 0x7f11900c49f0 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 -- 172.21.15.107:0/3737829250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11900c49f0 msgr2=0x7f119006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11900c49f0 0x7f119006d700 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f11900ccae0 tx=0x7f117802b900 comp rx=0 tx=0).stop 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 -- 172.21.15.107:0/3737829250 shutdown_connections 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f119006dc40 0x7f11900cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 --2- 172.21.15.107:0/3737829250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11900c49f0 0x7f119006d700 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 --2- 172.21.15.107:0/3737829250 >> v2:172.21.15.120:3300/0 conn(0x7f1190072160 0x7f119015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 -- 172.21.15.107:0/3737829250 >> 172.21.15.107:0/3737829250 conn(0x7f11900b0c50 msgr2=0x7f11900b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 -- 172.21.15.107:0/3737829250 shutdown_connections 2023-11-24T08:21:31.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.412+0000 7f1196a0e640 1 -- 172.21.15.107:0/3737829250 wait complete. 2023-11-24T08:21:31.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.414+0000 7f1196a0e640 1 Processor -- start 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.414+0000 7f1196a0e640 1 -- start start 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1190072160 0x7f1190159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 0x7f11900ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11901f3e00 con 0x7f119006dc40 2023-11-24T08:21:31.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119015d640 con 0x7f11900c49f0 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1196a0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11901f3040 con 0x7f1190072160 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f118ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f118ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37554/0 (socket says 172.21.15.107:37554) 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f118ffff640 1 -- 172.21.15.107:0/2812841671 learned_addr learned my addr 172.21.15.107:0/2812841671 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f1194fab640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 0x7f11900ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.415+0000 7f11877fe640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1190072160 0x7f1190159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 -- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1190072160 msgr2=0x7f1190159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1190072160 0x7f1190159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 -- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 msgr2=0x7f11900ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 0x7f11900ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 -- 172.21.15.107:0/2812841671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780094d0 con 0x7f119006dc40 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f1194fab640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 0x7f11900ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:31.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118ffff640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f118000c7f0 tx=0x7f118000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180004220 con 0x7f119006dc40 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11800043a0 con 0x7f119006dc40 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11901f3320 con 0x7f119006dc40 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180007860 con 0x7f119006dc40 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.416+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11901f38e0 con 0x7f119006dc40 2023-11-24T08:21:31.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.417+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119015dab0 con 0x7f119006dc40 2023-11-24T08:21:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.418+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f1180007bd0 con 0x7f119006dc40 2023-11-24T08:21:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.419+0000 7f118d7fa640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 0x7f117006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.419+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f1180088670 con 0x7f119006dc40 2023-11-24T08:21:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.420+0000 7f11877fe640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 0x7f117006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:31.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.420+0000 7f11877fe640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 0x7f117006f280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f1178036000 tx=0x7f117803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:31.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.424+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1180055220 con 0x7f119006dc40 2023-11-24T08:21:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.625+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f11900c47d0 con 0x7f119006dc40 2023-11-24T08:21:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.626+0000 7f118d7fa640 1 -- 172.21.15.107:0/2812841671 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v194) v1 ==== 80+0+0 (secure 0 0 0) 0x7f118001e020 con 0x7f119006dc40 2023-11-24T08:21:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:21:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 msgr2=0x7f117006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 0x7f117006f280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f1178036000 tx=0x7f117803c040 comp rx=0 tx=0).stop 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 msgr2=0x7f119009a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f118000c7f0 tx=0x7f118000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 shutdown_connections 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f117006cdc0 0x7f117006f280 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11900c49f0 0x7f11900ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1190072160 0x7f1190159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 --2- 172.21.15.107:0/2812841671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119006dc40 0x7f119009a340 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.629+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 >> 172.21.15.107:0/2812841671 conn(0x7f11900b0c50 msgr2=0x7f119015e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.630+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 shutdown_connections 2023-11-24T08:21:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:31.630+0000 7f1196a0e640 1 -- 172.21.15.107:0/2812841671 wait complete. 2023-11-24T08:21:31.650 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:21:31.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:31.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:21:31.766 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:21:31.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:31.767 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:21:31.822 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:31.822 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:31.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:21:31.882 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:21:31.882 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:31.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:21:31.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:32.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.163+0000 7fb920c64640 1 Processor -- start 2023-11-24T08:21:32.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.163+0000 7fb920c64640 1 -- start start 2023-11-24T08:21:32.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb920c64640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb91c15c7f0 0x7fb91c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb920c64640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91c0cece0 con 0x7fb91c0cf450 2023-11-24T08:21:32.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb920c64640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91c06d040 con 0x7fb91c0ce6c0 2023-11-24T08:21:32.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb920c64640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91c0ceaf0 con 0x7fb91c15c7f0 2023-11-24T08:21:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb919d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb91c0cf450 0x7fb91c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41626/0 (socket says 172.21.15.107:41626) 2023-11-24T08:21:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.164+0000 7fb919d9b640 1 -- 172.21.15.107:0/1492240004 learned_addr learned my addr 172.21.15.107:0/1492240004 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593640494 0 0) 0x7fb91c0cece0 con 0x7fb91c0cf450 2023-11-24T08:21:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb904003520 con 0x7fb91c0cf450 2023-11-24T08:21:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91a59c640 1 --2- 172.21.15.107:0/1492240004 >> v2:172.21.15.120:3300/0 conn(0x7fb91c15c7f0 0x7fb91c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2336858964 0 0) 0x7fb904003520 con 0x7fb91c0cf450 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 >> v2:172.21.15.120:3300/0 conn(0x7fb91c15c7f0 msgr2=0x7fb91c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 --2- 172.21.15.107:0/1492240004 >> v2:172.21.15.120:3300/0 conn(0x7fb91c15c7f0 0x7fb91c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 >> v1:172.21.15.120:6789/0 conn(0x7fb91c0ce6c0 legacy=0x7fb91c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c06d320 con 0x7fb91c0cf450 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb910001670 con 0x7fb91c0cf450 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 >> v1:172.21.15.107:6789/0 conn(0x7fb91c0cf450 legacy=0x7fb91c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb904003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9040047c0 0x7fb904004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.165+0000 7fb91959a640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb904005180 0x7fb90400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90400c080 con 0x7fb904003970 2023-11-24T08:21:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90400c1f0 con 0x7fb904005180 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90400c3b0 con 0x7fb9040047c0 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb904003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 -- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9040047c0 msgr2=0x7fb904004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9040047c0 0x7fb904004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 -- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb904005180 msgr2=0x7fb90400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb904005180 0x7fb90400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91ad9d640 1 -- 172.21.15.107:0/1492240004 reap_dead start 2023-11-24T08:21:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.166+0000 7fb91a59c640 1 -- 172.21.15.107:0/1492240004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904003520 con 0x7fb904003970 2023-11-24T08:21:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.167+0000 7fb91a59c640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb904003d70 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00ba00 tx=0x7fb9040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f233abc7f3daca63 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.167+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c032640 con 0x7fb904003970 2023-11-24T08:21:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.167+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c0327a0 con 0x7fb904003970 2023-11-24T08:21:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.167+0000 7fb91959a640 1 -- 172.21.15.107:0/1492240004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c032b30 con 0x7fb904003970 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 -- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 msgr2=0x7fb904003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb904003d70 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00ba00 tx=0x7fb9040057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 -- 172.21.15.107:0/1492240004 shutdown_connections 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 --2- 172.21.15.107:0/1492240004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb904003d70 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 -- 172.21.15.107:0/1492240004 >> 172.21.15.107:0/1492240004 conn(0x7fb91c06a920 msgr2=0x7fb91c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 -- 172.21.15.107:0/1492240004 shutdown_connections 2023-11-24T08:21:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.168+0000 7fb920c64640 1 -- 172.21.15.107:0/1492240004 wait complete. 2023-11-24T08:21:32.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.170+0000 7fb920c64640 1 Processor -- start 2023-11-24T08:21:32.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.170+0000 7fb920c64640 1 -- start start 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91c0b3860 0x7fb91c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 0x7fb91c158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0b3440 con 0x7fb904003970 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c153ad0 con 0x7fb91c1f6490 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.171+0000 7fb920c64640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0b27c0 con 0x7fb91c0b3860 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37566/0 (socket says 172.21.15.107:37566) 2023-11-24T08:21:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 -- 172.21.15.107:0/985017357 learned_addr learned my addr 172.21.15.107:0/985017357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91ad9d640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 0x7fb91c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb919d9b640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91c0b3860 0x7fb91c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 -- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91c0b3860 msgr2=0x7fb91c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91c0b3860 0x7fb91c0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 -- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 msgr2=0x7fb91c158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 0x7fb91c158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 -- 172.21.15.107:0/985017357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0b2aa0 con 0x7fb904003970 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91ad9d640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 0x7fb91c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.172+0000 7fb91a59c640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fb90c02bec0 tx=0x7fb90c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.173+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c004220 con 0x7fb904003970 2023-11-24T08:21:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.173+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c004380 con 0x7fb904003970 2023-11-24T08:21:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.173+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c0b2d80 con 0x7fb904003970 2023-11-24T08:21:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.173+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c03ba70 con 0x7fb904003970 2023-11-24T08:21:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.173+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c158580 con 0x7fb904003970 2023-11-24T08:21:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.174+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c067470 con 0x7fb904003970 2023-11-24T08:21:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.175+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fb90c03bbd0 con 0x7fb904003970 2023-11-24T08:21:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.176+0000 7fb90affd640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 0x7fb8fc06f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.176+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fb90c0aaef0 con 0x7fb904003970 2023-11-24T08:21:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.176+0000 7fb919d9b640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 0x7fb8fc06f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.177+0000 7fb919d9b640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 0x7fb8fc06f2b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0b3c40 tx=0x7fb910009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.180+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb90c077a00 con 0x7fb904003970 2023-11-24T08:21:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.405+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb91c067470 con 0x7fb904003970 2023-11-24T08:21:32.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.407+0000 7fb90affd640 1 -- 172.21.15.107:0/985017357 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4939 (secure 0 0 0) 0x7fb90c006c30 con 0x7fb904003970 2023-11-24T08:21:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:32.409 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:49.053270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27,28],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:21:32.409 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-11-24T08:21:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 msgr2=0x7fb8fc06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 0x7fb8fc06f2b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0b3c40 tx=0x7fb910009040 comp rx=0 tx=0).stop 2023-11-24T08:21:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 msgr2=0x7fb91c1567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fb90c02bec0 tx=0x7fb90c004730 comp rx=0 tx=0).stop 2023-11-24T08:21:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 shutdown_connections 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8fc06cdf0 0x7fb8fc06f2b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb91c1f6490 0x7fb91c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb91c0b3860 0x7fb91c0b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 --2- 172.21.15.107:0/985017357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb904003970 0x7fb91c1567f0 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 >> 172.21.15.107:0/985017357 conn(0x7fb91c06a920 msgr2=0x7fb91c0afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.409+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 shutdown_connections 2023-11-24T08:21:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.410+0000 7fb920c64640 1 -- 172.21.15.107:0/985017357 wait complete. 2023-11-24T08:21:32.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.639+0000 7f8e4e2ee640 1 Processor -- start 2023-11-24T08:21:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.639+0000 7f8e4e2ee640 1 -- start start 2023-11-24T08:21:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.640+0000 7f8e4e2ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 0x7f8e4806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.640+0000 7f8e4e2ee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e480cf450 0x7f8e480cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.640+0000 7f8e4e2ee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e480cece0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.640+0000 7f8e4e2ee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e48156010 con 0x7f8e480cf450 2023-11-24T08:21:32.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.640+0000 7f8e4e2ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806ce30 con 0x7f8e4815c7f0 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.641+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 0x7f8e4806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.641+0000 7f8e477fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e480cf450 0x7f8e480cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.641+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 0x7f8e4806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56844/0 (socket says 172.21.15.107:56844) 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.641+0000 7f8e47fff640 1 -- 172.21.15.107:0/4033498810 learned_addr learned my addr 172.21.15.107:0/4033498810 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.642+0000 7f8e477fe640 1 -- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 msgr2=0x7f8e4806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:32.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.642+0000 7f8e477fe640 1 --2- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 0x7f8e4806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.642+0000 7f8e477fe640 1 -- 172.21.15.107:0/4033498810 >> v1:172.21.15.120:6789/0 conn(0x7f8e480cdfb0 legacy=0x7f8e480ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.642+0000 7f8e477fe640 1 -- 172.21.15.107:0/4033498810 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4806d040 con 0x7f8e480cf450 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.643+0000 7f8e477fe640 1 --2- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480cf450 0x7f8e480cf8d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0097a0 tx=0x7f8e3c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c534e7b113fa9f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e46ffd640 1 -- 172.21.15.107:0/4033498810 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c02d040 con 0x7f8e480cf450 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e46ffd640 1 -- 172.21.15.107:0/4033498810 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c031810 con 0x7f8e480cf450 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e46ffd640 1 -- 172.21.15.107:0/4033498810 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c031ba0 con 0x7f8e480cf450 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480cf450 msgr2=0x7f8e480cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480cf450 0x7f8e480cf8d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0097a0 tx=0x7f8e3c02b8b0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/4033498810 shutdown_connections 2023-11-24T08:21:32.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e480cf450 0x7f8e480cf8d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/4033498810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4815c7f0 0x7f8e4806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.644+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/4033498810 >> 172.21.15.107:0/4033498810 conn(0x7f8e4806a920 msgr2=0x7f8e4806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:32.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.645+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/4033498810 shutdown_connections 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.645+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/4033498810 wait complete. 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.646+0000 7f8e4e2ee640 1 Processor -- start 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.646+0000 7f8e4e2ee640 1 -- start start 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480cf450 0x7f8e480ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 0x7f8e481542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e481579e0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48157cc0 con 0x7f8e4815c7f0 2023-11-24T08:21:32.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.647+0000 7f8e4e2ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48157fa0 con 0x7f8e480cf450 2023-11-24T08:21:32.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37582/0 (socket says 172.21.15.107:37582) 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 -- 172.21.15.107:0/3894779326 learned_addr learned my addr 172.21.15.107:0/3894779326 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e477fe640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480cf450 0x7f8e480ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 -- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480cf450 msgr2=0x7f8e480ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480cf450 0x7f8e480ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 -- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 msgr2=0x7f8e481542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e4c88b640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 0x7f8e481542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 0x7f8e481542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 -- 172.21.15.107:0/3894779326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c009400 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e4c88b640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 0x7f8e481542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.648+0000 7f8e47fff640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e840 tx=0x7f8e3800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.649+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800c8e0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.649+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3800ca40 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.649+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800cdf0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.649+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e48158280 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.649+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e480ad7c0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.650+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e48067470 con 0x7f8e480cdfb0 2023-11-24T08:21:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.651+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f8e38013470 con 0x7f8e480cdfb0 2023-11-24T08:21:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.652+0000 7f8e44ff9640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 0x7f8e3406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.652+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f8e38088730 con 0x7f8e480cdfb0 2023-11-24T08:21:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.652+0000 7f8e477fe640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 0x7f8e3406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.653+0000 7f8e477fe640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 0x7f8e3406f280 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c036000 tx=0x7f8e3c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:32.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.656+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e380552e0 con 0x7f8e480cdfb0 2023-11-24T08:21:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.847+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e48157b20 con 0x7f8e480cdfb0 2023-11-24T08:21:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.849+0000 7f8e44ff9640 1 -- 172.21.15.107:0/3894779326 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+18584 (secure 0 0 0) 0x7f8e48157b20 con 0x7f8e480cdfb0 2023-11-24T08:21:32.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:32.852 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":194,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:28.833824+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.851+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 msgr2=0x7f8e3406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.851+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 0x7f8e3406f280 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c036000 tx=0x7f8e3c03c040 comp rx=0 tx=0).stop 2023-11-24T08:21:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.851+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 msgr2=0x7f8e48150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.851+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e840 tx=0x7f8e3800ed00 comp rx=0 tx=0).stop 2023-11-24T08:21:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 shutdown_connections 2023-11-24T08:21:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8e3406cdc0 0x7f8e3406f280 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e4815c7f0 0x7f8e481542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e480cf450 0x7f8e480ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 --2- 172.21.15.107:0/3894779326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e480cdfb0 0x7f8e48150400 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 >> 172.21.15.107:0/3894779326 conn(0x7f8e4806a920 msgr2=0x7f8e480cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 shutdown_connections 2023-11-24T08:21:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:32.852+0000 7f8e4e2ee640 1 -- 172.21.15.107:0/3894779326 wait complete. 2023-11-24T08:21:32.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:20:30.467990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:20:31.721268+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'new_pool', 'create_time': '2023-11-24T08:20:48.209202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:21:32.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:21:33.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.083+0000 7f3636fc5640 1 Processor -- start 2023-11-24T08:21:33.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.083+0000 7f3636fc5640 1 -- start start 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.084+0000 7f3636fc5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 0x7f36300cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.084+0000 7f3636fc5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36300cbf20 0x7f36300c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3636fc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 0x7f36300c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3636fc5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300cc3d0 con 0x7f36300cbf20 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3636fc5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363015b820 con 0x7f363015b440 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3636fc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363015eea0 con 0x7f36300c57e0 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36300cbf20 0x7f36300c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36300cbf20 0x7f36300c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37586/0 (socket says 172.21.15.107:37586) 2023-11-24T08:21:33.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 -- 172.21.15.107:0/2074971902 learned_addr learned my addr 172.21.15.107:0/2074971902 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:33.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3635562640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 0x7f36300c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 -- 172.21.15.107:0/2074971902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 msgr2=0x7f36300c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3634d61640 1 --2- 172.21.15.107:0/2074971902 >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 0x7f36300cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 0x7f36300c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 -- 172.21.15.107:0/2074971902 >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 msgr2=0x7f36300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 --2- 172.21.15.107:0/2074971902 >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 0x7f36300cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f362ffff640 1 -- 172.21.15.107:0/2074971902 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300abf30 con 0x7f36300cbf20 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3635562640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 0x7f36300c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.085+0000 7f3634d61640 1 --2- 172.21.15.107:0/2074971902 >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 0x7f36300cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:33.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.087+0000 7f362ffff640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300cbf20 0x7f36300c52a0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f363006e770 tx=0x7f361c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0dabda6ee85364 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.087+0000 7f362f7fe640 1 -- 172.21.15.107:0/2074971902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0317f0 con 0x7f36300cbf20 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.087+0000 7f362f7fe640 1 -- 172.21.15.107:0/2074971902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c031950 con 0x7f36300cbf20 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.087+0000 7f362f7fe640 1 -- 172.21.15.107:0/2074971902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c031ce0 con 0x7f36300cbf20 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 -- 172.21.15.107:0/2074971902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300cbf20 msgr2=0x7f36300c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300cbf20 0x7f36300c52a0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f363006e770 tx=0x7f361c02b890 comp rx=0 tx=0).stop 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 -- 172.21.15.107:0/2074971902 shutdown_connections 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300c57e0 0x7f36300c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 --2- 172.21.15.107:0/2074971902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300cbf20 0x7f36300c52a0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 --2- 172.21.15.107:0/2074971902 >> v2:172.21.15.120:3300/0 conn(0x7f363015b440 0x7f36300cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 -- 172.21.15.107:0/2074971902 >> 172.21.15.107:0/2074971902 conn(0x7f36300b0c50 msgr2=0x7f36300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 -- 172.21.15.107:0/2074971902 shutdown_connections 2023-11-24T08:21:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.088+0000 7f3636fc5640 1 -- 172.21.15.107:0/2074971902 wait complete. 2023-11-24T08:21:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.090+0000 7f3636fc5640 1 Processor -- start 2023-11-24T08:21:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.090+0000 7f3636fc5640 1 -- start start 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300cbf20 0x7f36300cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 0x7f36300ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300c8f50 con 0x7f36300c57e0 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300c83d0 con 0x7f363015b440 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.091+0000 7f3636fc5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36300c86b0 con 0x7f36300cbf20 2023-11-24T08:21:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37602/0 (socket says 172.21.15.107:37602) 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 -- 172.21.15.107:0/1719588066 learned_addr learned my addr 172.21.15.107:0/1719588066 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f362ffff640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300cbf20 0x7f36300cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 -- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300cbf20 msgr2=0x7f36300cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300cbf20 0x7f36300cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3635562640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 0x7f36300ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 -- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 msgr2=0x7f36300ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 0x7f36300ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 -- 172.21.15.107:0/1719588066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0094d0 con 0x7f36300c57e0 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3635562640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 0x7f36300ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.092+0000 7f3634d61640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f362400c7f0 tx=0x7f362400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.093+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624007820 con 0x7f36300c57e0 2023-11-24T08:21:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.093+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624007980 con 0x7f36300c57e0 2023-11-24T08:21:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.093+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624007cf0 con 0x7f36300c57e0 2023-11-24T08:21:33.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.093+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36300c8990 con 0x7f36300c57e0 2023-11-24T08:21:33.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.093+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36300c7610 con 0x7f36300c57e0 2023-11-24T08:21:33.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.094+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363004e8b0 con 0x7f36300c57e0 2023-11-24T08:21:33.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.095+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f36240113f0 con 0x7f36300c57e0 2023-11-24T08:21:33.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.096+0000 7f362d7fa640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 0x7f361406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.096+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f3624088960 con 0x7f36300c57e0 2023-11-24T08:21:33.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.096+0000 7f362ffff640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 0x7f361406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.097+0000 7f362ffff640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 0x7f361406f2b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f361c036000 tx=0x7f361c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.101+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3624055510 con 0x7f36300c57e0 2023-11-24T08:21:33.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.290+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3630159110 con 0x7f36300c57e0 2023-11-24T08:21:33.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.291+0000 7f362d7fa640 1 -- 172.21.15.107:0/1719588066 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7f362401f020 con 0x7f36300c57e0 2023-11-24T08:21:33.293 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:21:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 msgr2=0x7f361406f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 0x7f361406f2b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f361c036000 tx=0x7f361c03d040 comp rx=0 tx=0).stop 2023-11-24T08:21:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 msgr2=0x7f36301548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f362400c7f0 tx=0x7f362400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 shutdown_connections 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f361406cdf0 0x7f361406f2b0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f363015b440 0x7f36300ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36300cbf20 0x7f36300cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 --2- 172.21.15.107:0/1719588066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36300c57e0 0x7f36301548a0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.294+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 >> 172.21.15.107:0/1719588066 conn(0x7f36300b0c50 msgr2=0x7f3630070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.295+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 shutdown_connections 2023-11-24T08:21:33.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.295+0000 7f3636fc5640 1 -- 172.21.15.107:0/1719588066 wait complete. 2023-11-24T08:21:33.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:21:33.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.520+0000 7f6b00631640 1 Processor -- start 2023-11-24T08:21:33.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.520+0000 7f6b00631640 1 -- start start 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6af815b440 0x7f6af80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 0x7f6af80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af80c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af80cc3d0 con 0x7f6af815b440 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af815b820 con 0x7f6af80cbf20 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6b00631640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815eea0 con 0x7f6af80c57e0 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6afe3cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6af815b440 0x7f6af80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6afe3cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6af815b440 0x7f6af80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37610/0 (socket says 172.21.15.107:37610) 2023-11-24T08:21:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.522+0000 7f6afe3cd640 1 -- 172.21.15.107:0/1446773678 learned_addr learned my addr 172.21.15.107:0/1446773678 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afdbcc640 1 --2- 172.21.15.107:0/1446773678 >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 0x7f6af80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afebce640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af80c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afe3cd640 1 -- 172.21.15.107:0/1446773678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 msgr2=0x7f6af80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af80c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afe3cd640 1 -- 172.21.15.107:0/1446773678 >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 msgr2=0x7f6af80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1446773678 >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 0x7f6af80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afe3cd640 1 -- 172.21.15.107:0/1446773678 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af80abf30 con 0x7f6af815b440 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afdbcc640 1 --2- 172.21.15.107:0/1446773678 >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 0x7f6af80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.523+0000 7f6afebce640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af80c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.524+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af80cb9e0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f6af806e890 tx=0x7f6af402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5c91e0fb68ffde server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.524+0000 7f6afd3cb640 1 -- 172.21.15.107:0/1446773678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af40317e0 con 0x7f6af815b440 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.524+0000 7f6afd3cb640 1 -- 172.21.15.107:0/1446773678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4031940 con 0x7f6af815b440 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.524+0000 7f6afd3cb640 1 -- 172.21.15.107:0/1446773678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4031cd0 con 0x7f6af815b440 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 -- 172.21.15.107:0/1446773678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 msgr2=0x7f6af80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af80cb9e0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f6af806e890 tx=0x7f6af402b880 comp rx=0 tx=0).stop 2023-11-24T08:21:33.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 -- 172.21.15.107:0/1446773678 shutdown_connections 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af80c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 --2- 172.21.15.107:0/1446773678 >> v2:172.21.15.120:3300/0 conn(0x7f6af80cbf20 0x7f6af80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 --2- 172.21.15.107:0/1446773678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af80cb9e0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 -- 172.21.15.107:0/1446773678 >> 172.21.15.107:0/1446773678 conn(0x7f6af80b0c50 msgr2=0x7f6af80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 -- 172.21.15.107:0/1446773678 shutdown_connections 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.525+0000 7f6b00631640 1 -- 172.21.15.107:0/1446773678 wait complete. 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.527+0000 7f6b00631640 1 Processor -- start 2023-11-24T08:21:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.527+0000 7f6b00631640 1 -- start start 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af8157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80cbf20 0x7f6af8159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80c76c0 con 0x7f6af815b440 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815dd40 con 0x7f6af80cbf20 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6b00631640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815cdf0 con 0x7f6af80c57e0 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37616/0 (socket says 172.21.15.107:37616) 2023-11-24T08:21:33.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 -- 172.21.15.107:0/1186334465 learned_addr learned my addr 172.21.15.107:0/1186334465 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 -- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 msgr2=0x7f6af8157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af8157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afdbcc640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80cbf20 0x7f6af8159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af8157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 -- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80cbf20 msgr2=0x7f6af8159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80cbf20 0x7f6af8159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.528+0000 7f6afebce640 1 -- 172.21.15.107:0/1186334465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40094d0 con 0x7f6af815b440 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6afebce640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f6af000cef0 tx=0x7f6af000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000d840 con 0x7f6af815b440 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000d9a0 con 0x7f6af815b440 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000dd50 con 0x7f6af815b440 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af815df90 con 0x7f6af815b440 2023-11-24T08:21:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.529+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af815e550 con 0x7f6af815b440 2023-11-24T08:21:33.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.530+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af804e8b0 con 0x7f6af815b440 2023-11-24T08:21:33.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.531+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f6af001e3b0 con 0x7f6af815b440 2023-11-24T08:21:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.532+0000 7f6aeeffd640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 0x7f6adc06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.532+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f6af0089090 con 0x7f6af815b440 2023-11-24T08:21:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.532+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 0x7f6adc06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.533+0000 7f6afe3cd640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 0x7f6adc06f280 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036000 tx=0x7f6af403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.536+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6af0055c40 con 0x7f6af815b440 2023-11-24T08:21:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.715+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6af815e8f0 con 0x7f6af815b440 2023-11-24T08:21:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.715+0000 7f6aeeffd640 1 -- 172.21.15.107:0/1186334465 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v194) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6af001a020 con 0x7f6af815b440 2023-11-24T08:21:33.718 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 msgr2=0x7f6adc06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 0x7f6adc06f280 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036000 tx=0x7f6af403d040 comp rx=0 tx=0).stop 2023-11-24T08:21:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 msgr2=0x7f6af81566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f6af000cef0 tx=0x7f6af000a710 comp rx=0 tx=0).stop 2023-11-24T08:21:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 shutdown_connections 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6adc06cdc0 0x7f6adc06f280 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6af815b440 0x7f6af81566d0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80cbf20 0x7f6af8159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 --2- 172.21.15.107:0/1186334465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af80c57e0 0x7f6af8157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 >> 172.21.15.107:0/1186334465 conn(0x7f6af80b0c50 msgr2=0x7f6af80cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 shutdown_connections 2023-11-24T08:21:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.718+0000 7f6b00631640 1 -- 172.21.15.107:0/1186334465 wait complete. 2023-11-24T08:21:33.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:21:33.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.951+0000 7fd45aabb640 1 Processor -- start 2023-11-24T08:21:33.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.951+0000 7fd45aabb640 1 -- start start 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd45aabb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4540c59b0 0x7fd454166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd45aabb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4540c5dc0 con 0x7fd45415b440 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd45aabb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45415eea0 con 0x7fd4540cc0f0 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd45aabb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45415b820 con 0x7fd4540c59b0 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd458857640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd45415b440 0x7fd4540cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41642/0 (socket says 172.21.15.107:41642) 2023-11-24T08:21:33.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd458857640 1 -- 172.21.15.107:0/3685486059 learned_addr learned my addr 172.21.15.107:0/3685486059 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.953+0000 7fd459058640 1 --2- 172.21.15.107:0/3685486059 >> v2:172.21.15.120:3300/0 conn(0x7fd4540c59b0 0x7fd454166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678835717 0 0) 0x7fd4540c5dc0 con 0x7fd45415b440 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd43c003520 con 0x7fd45415b440 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3830179699 0 0) 0x7fd43c003520 con 0x7fd45415b440 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 >> v2:172.21.15.120:3300/0 conn(0x7fd4540c59b0 msgr2=0x7fd454166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 --2- 172.21.15.107:0/3685486059 >> v2:172.21.15.120:3300/0 conn(0x7fd4540c59b0 0x7fd454166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 >> v1:172.21.15.120:6789/0 conn(0x7fd4540cc0f0 legacy=0x7fd4540c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45406d600 con 0x7fd45415b440 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd448001670 con 0x7fd45415b440 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 >> v1:172.21.15.107:6789/0 conn(0x7fd45415b440 legacy=0x7fd4540cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd43c003970 0x7fd43c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd43c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43c005180 0x7fd43c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45415b820 con 0x7fd43c0047c0 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c00c120 con 0x7fd43c005180 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c00c2e0 con 0x7fd43c003970 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.954+0000 7fd453fff640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd43c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd453fff640 1 -- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd43c003970 msgr2=0x7fd43c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd453fff640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd43c003970 0x7fd43c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd453fff640 1 -- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43c005180 msgr2=0x7fd43c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd453fff640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43c005180 0x7fd43c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd453fff640 1 -- 172.21.15.107:0/3685486059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c003520 con 0x7fd43c0047c0 2023-11-24T08:21:33.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.955+0000 7fd459058640 1 -- 172.21.15.107:0/3685486059 reap_dead start 2023-11-24T08:21:33.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.956+0000 7fd453fff640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd43c004c40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fd444001750 tx=0x7fd43c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9398de91e418b77 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.956+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440052d0 con 0x7fd43c0047c0 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.956+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44402e050 con 0x7fd43c0047c0 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.956+0000 7fd4537fe640 1 -- 172.21.15.107:0/3685486059 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444003830 con 0x7fd43c0047c0 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 -- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 msgr2=0x7fd43c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd43c004c40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fd444001750 tx=0x7fd43c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 -- 172.21.15.107:0/3685486059 shutdown_connections 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 --2- 172.21.15.107:0/3685486059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd43c004c40 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 -- 172.21.15.107:0/3685486059 >> 172.21.15.107:0/3685486059 conn(0x7fd4540b0c50 msgr2=0x7fd4540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 -- 172.21.15.107:0/3685486059 shutdown_connections 2023-11-24T08:21:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.957+0000 7fd45aabb640 1 -- 172.21.15.107:0/3685486059 wait complete. 2023-11-24T08:21:33.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.958+0000 7fd45aabb640 1 Processor -- start 2023-11-24T08:21:33.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.959+0000 7fd45aabb640 1 -- start start 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454150ed0 0x7fd4540ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd45414ffe0 0x7fd4540ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540b0870 con 0x7fd43c0047c0 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c7980 con 0x7fd45414ffe0 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd45aabb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c7c60 con 0x7fd454150ed0 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37632/0 (socket says 172.21.15.107:37632) 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 -- 172.21.15.107:0/2219589603 learned_addr learned my addr 172.21.15.107:0/2219589603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd453fff640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454150ed0 0x7fd4540ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 -- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454150ed0 msgr2=0x7fd4540ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454150ed0 0x7fd4540ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd459058640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd45414ffe0 0x7fd4540ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 -- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd45414ffe0 msgr2=0x7fd4540ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd45414ffe0 0x7fd4540ad6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.960+0000 7fd458857640 1 -- 172.21.15.107:0/2219589603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4540c7f40 con 0x7fd43c0047c0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd458857640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fd44800c700 tx=0x7fd44800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4480074e0 con 0x7fd43c0047c0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd448007660 con 0x7fd43c0047c0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448004a20 con 0x7fd43c0047c0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4540c8220 con 0x7fd43c0047c0 2023-11-24T08:21:33.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.961+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4540c7650 con 0x7fd43c0047c0 2023-11-24T08:21:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.962+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4541548f0 con 0x7fd43c0047c0 2023-11-24T08:21:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.963+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fd44800a5e0 con 0x7fd43c0047c0 2023-11-24T08:21:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.964+0000 7fd4517fa640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 0x7fd44006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.964+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fd448087960 con 0x7fd43c0047c0 2023-11-24T08:21:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.965+0000 7fd453fff640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 0x7fd44006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.965+0000 7fd453fff640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 0x7fd44006f280 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd444033000 tx=0x7fd444009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:33.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:33.968+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4541548f0 con 0x7fd43c0047c0 2023-11-24T08:21:34.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.157+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4541548f0 con 0x7fd43c0047c0 2023-11-24T08:21:34.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.158+0000 7fd4517fa640 1 -- 172.21.15.107:0/2219589603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v194) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4541548f0 con 0x7fd43c0047c0 2023-11-24T08:21:34.160 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.160+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 msgr2=0x7fd44006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.160+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 0x7fd44006f280 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd444033000 tx=0x7fd444009030 comp rx=0 tx=0).stop 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.160+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 msgr2=0x7fd4540c7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.160+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fd44800c700 tx=0x7fd44800cbc0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 shutdown_connections 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd44006cdc0 0x7fd44006f280 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd45414ffe0 0x7fd4540ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454150ed0 0x7fd4540ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 --2- 172.21.15.107:0/2219589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd43c0047c0 0x7fd4540c7110 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 >> 172.21.15.107:0/2219589603 conn(0x7fd4540b0c50 msgr2=0x7fd4540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:34.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 shutdown_connections 2023-11-24T08:21:34.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.161+0000 7fd45aabb640 1 -- 172.21.15.107:0/2219589603 wait complete. 2023-11-24T08:21:34.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get new_pool pg_num 2023-11-24T08:21:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.387+0000 7f7a50f43640 1 Processor -- start 2023-11-24T08:21:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.387+0000 7f7a50f43640 1 -- start start 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a50f43640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a4c15b440 0x7f7a4c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a50f43640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a50f43640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c15eea0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a50f43640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c0cc3d0 con 0x7f7a4c15b440 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a50f43640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c15b820 con 0x7f7a4c0cbf20 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a4a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a4c15b440 0x7f7a4c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a4ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7a4c0c57e0 0x7f7a4c0c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41652/0 (socket says 172.21.15.107:41652) 2023-11-24T08:21:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a4ad9d640 1 -- 172.21.15.107:0/3613787141 learned_addr learned my addr 172.21.15.107:0/3613787141 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.389+0000 7f7a49d9b640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4959a640 1 -- 172.21.15.107:0/3613787141 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911627097 0 0) 0x7f7a4c15eea0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4959a640 1 -- 172.21.15.107:0/3613787141 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a34003520 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4a59c640 1 -- 172.21.15.107:0/3613787141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 msgr2=0x7f7a4c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4a59c640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4a59c640 1 -- 172.21.15.107:0/3613787141 >> v1:172.21.15.107:6789/0 conn(0x7f7a4c0c57e0 legacy=0x7f7a4c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.390+0000 7f7a4a59c640 1 -- 172.21.15.107:0/3613787141 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c06d600 con 0x7f7a4c15b440 2023-11-24T08:21:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.391+0000 7f7a4a59c640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c15b440 0x7f7a4c0cb9e0 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cd25a685947417 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.391+0000 7f7a4959a640 1 -- 172.21.15.107:0/3613787141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c031780 con 0x7f7a4c15b440 2023-11-24T08:21:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.391+0000 7f7a4959a640 1 -- 172.21.15.107:0/3613787141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c0318e0 con 0x7f7a4c15b440 2023-11-24T08:21:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.391+0000 7f7a4959a640 1 -- 172.21.15.107:0/3613787141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c031c70 con 0x7f7a4c15b440 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 -- 172.21.15.107:0/3613787141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c15b440 msgr2=0x7f7a4c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c15b440 0x7f7a4c0cb9e0 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02b820 comp rx=0 tx=0).stop 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 -- 172.21.15.107:0/3613787141 shutdown_connections 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 --2- 172.21.15.107:0/3613787141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c15b440 0x7f7a4c0cb9e0 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 -- 172.21.15.107:0/3613787141 >> 172.21.15.107:0/3613787141 conn(0x7f7a4c0b0c50 msgr2=0x7f7a4c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 -- 172.21.15.107:0/3613787141 shutdown_connections 2023-11-24T08:21:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.392+0000 7f7a50f43640 1 -- 172.21.15.107:0/3613787141 wait complete. 2023-11-24T08:21:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.394+0000 7f7a50f43640 1 Processor -- start 2023-11-24T08:21:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.394+0000 7f7a50f43640 1 -- start start 2023-11-24T08:21:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a4c15b440 0x7f7a4c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c1569f0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c15caa0 con 0x7f7a4c15b440 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a50f43640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c15ba00 con 0x7f7a4c0cbf20 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37660/0 (socket says 172.21.15.107:37660) 2023-11-24T08:21:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.395+0000 7f7a4a59c640 1 -- 172.21.15.107:0/1520234660 learned_addr learned my addr 172.21.15.107:0/1520234660 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 -- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 msgr2=0x7f7a4c155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 -- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a4c15b440 msgr2=0x7f7a4c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a4c15b440 0x7f7a4c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 -- 172.21.15.107:0/1520234660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c009400 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a4a59c640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c000c00 tx=0x7f7a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c032820 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c0329c0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c032d70 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c15bce0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.396+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c15c2a0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.398+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c04e8b0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.398+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7a3c04e450 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.399+0000 7f7a3affd640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 0x7f7a2c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.399+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f7a3c0aa6e0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.400+0000 7f7a49d9b640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 0x7f7a2c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.400+0000 7f7a49d9b640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 0x7f7a2c06f280 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7a40006a90 tx=0x7f7a40006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.403+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a3c0771f0 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.592+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"} v 0) v1 -- 0x7f7a4c06dc80 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.593+0000 7f7a3affd640 1 -- 172.21.15.107:0/1520234660 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "new_pool", "var": "pg_num"}]=0 v194) v1 ==== 97+0+11 (secure 0 0 0) 0x7f7a3c07a810 con 0x7f7a4c0c57e0 2023-11-24T08:21:34.595 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:21:34.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 msgr2=0x7f7a2c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 0x7f7a2c06f280 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7a40006a90 tx=0x7f7a40006a20 comp rx=0 tx=0).stop 2023-11-24T08:21:34.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 msgr2=0x7f7a4c0ad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c000c00 tx=0x7f7a3c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 shutdown_connections 2023-11-24T08:21:34.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7a2c06cdc0 0x7f7a2c06f280 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a4c15b440 0x7f7a4c1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a4c0cbf20 0x7f7a4c155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 --2- 172.21.15.107:0/1520234660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a4c0c57e0 0x7f7a4c0ad3b0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 >> 172.21.15.107:0/1520234660 conn(0x7f7a4c0b0c50 msgr2=0x7f7a4c0c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:34.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 shutdown_connections 2023-11-24T08:21:34.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.596+0000 7f7a50f43640 1 -- 172.21.15.107:0/1520234660 wait complete. 2023-11-24T08:21:34.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.822+0000 7f814a36f640 1 Processor -- start 2023-11-24T08:21:34.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.822+0000 7f814a36f640 1 -- start start 2023-11-24T08:21:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.823+0000 7f814a36f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8144157c70 0x7f8144161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.823+0000 7f814a36f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.823+0000 7f814a36f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440ce0b0 con 0x7f81440cf450 2023-11-24T08:21:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.823+0000 7f814a36f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81440cfe40 con 0x7f8144157c70 2023-11-24T08:21:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.823+0000 7f814a36f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8144161960 con 0x7f814415e390 2023-11-24T08:21:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.824+0000 7f814890c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.824+0000 7f814890c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56964/0 (socket says 172.21.15.107:56964) 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.824+0000 7f814890c640 1 -- 172.21.15.107:0/3715227889 learned_addr learned my addr 172.21.15.107:0/3715227889 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.824+0000 7f8143fff640 1 --2- 172.21.15.107:0/3715227889 >> v2:172.21.15.120:3300/0 conn(0x7f8144157c70 0x7f8144161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.825+0000 7f814890c640 1 -- 172.21.15.107:0/3715227889 >> v2:172.21.15.120:3300/0 conn(0x7f8144157c70 msgr2=0x7f8144161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.825+0000 7f814890c640 1 --2- 172.21.15.107:0/3715227889 >> v2:172.21.15.120:3300/0 conn(0x7f8144157c70 0x7f8144161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.825+0000 7f814890c640 1 -- 172.21.15.107:0/3715227889 >> v1:172.21.15.120:6789/0 conn(0x7f81440cf450 legacy=0x7f814415dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.825+0000 7f814890c640 1 -- 172.21.15.107:0/3715227889 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814415e8d0 con 0x7f814415e390 2023-11-24T08:21:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.826+0000 7f814890c640 1 --2- 172.21.15.107:0/3715227889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f813c009870 tx=0x7f813c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ccacc93f987984 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.826+0000 7f8142ffd640 1 -- 172.21.15.107:0/3715227889 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c031640 con 0x7f814415e390 2023-11-24T08:21:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f8142ffd640 1 -- 172.21.15.107:0/3715227889 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c0317a0 con 0x7f814415e390 2023-11-24T08:21:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f8142ffd640 1 -- 172.21.15.107:0/3715227889 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c031b30 con 0x7f814415e390 2023-11-24T08:21:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 -- 172.21.15.107:0/3715227889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 msgr2=0x7f81440cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 --2- 172.21.15.107:0/3715227889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f813c009870 tx=0x7f813c02b8f0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 -- 172.21.15.107:0/3715227889 shutdown_connections 2023-11-24T08:21:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 --2- 172.21.15.107:0/3715227889 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814415e390 0x7f81440cf900 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 --2- 172.21.15.107:0/3715227889 >> v2:172.21.15.120:3300/0 conn(0x7f8144157c70 0x7f8144161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 -- 172.21.15.107:0/3715227889 >> 172.21.15.107:0/3715227889 conn(0x7f814406a920 msgr2=0x7f814406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 -- 172.21.15.107:0/3715227889 shutdown_connections 2023-11-24T08:21:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.827+0000 7f814a36f640 1 -- 172.21.15.107:0/3715227889 wait complete. 2023-11-24T08:21:34.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.829+0000 7f814a36f640 1 Processor -- start 2023-11-24T08:21:34.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.829+0000 7f814a36f640 1 -- start start 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81440cf450 0x7f81440cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 0x7f814409cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81440cc340 con 0x7f814415e390 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81440caac0 con 0x7f8144157c70 2023-11-24T08:21:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814a36f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814409d130 con 0x7f81440cf450 2023-11-24T08:21:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814890c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814890c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37666/0 (socket says 172.21.15.107:37666) 2023-11-24T08:21:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.830+0000 7f814890c640 1 -- 172.21.15.107:0/157618558 learned_addr learned my addr 172.21.15.107:0/157618558 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f8143fff640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81440cf450 0x7f81440cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 -- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81440cf450 msgr2=0x7f81440cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81440cf450 0x7f81440cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f81437fe640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 0x7f814409cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 -- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 msgr2=0x7f814409cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 0x7f814409cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 -- 172.21.15.107:0/157618558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0094d0 con 0x7f814415e390 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f81437fe640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 0x7f814409cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.831+0000 7f814890c640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f813c0090b0 tx=0x7f813c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.832+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c03a6f0 con 0x7f814415e390 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.832+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c03a870 con 0x7f814415e390 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.832+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c03ac20 con 0x7f814415e390 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.832+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81440cad60 con 0x7f814415e390 2023-11-24T08:21:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.832+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81440cb320 con 0x7f814415e390 2023-11-24T08:21:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.833+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8144067470 con 0x7f814415e390 2023-11-24T08:21:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.834+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f813c02f050 con 0x7f814415e390 2023-11-24T08:21:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.835+0000 7f8140ff9640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 0x7f812406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.835+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f813c0a9fd0 con 0x7f814415e390 2023-11-24T08:21:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.835+0000 7f8143fff640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 0x7f812406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.836+0000 7f8143fff640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 0x7f812406f280 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8134007380 tx=0x7f813400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:34.839+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f813c076b80 con 0x7f814415e390 2023-11-24T08:21:35.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.066+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f814415fdf0 con 0x7f814415e390 2023-11-24T08:21:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.068+0000 7f8140ff9640 1 -- 172.21.15.107:0/157618558 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f813c048090 con 0x7f814415e390 2023-11-24T08:21:35.070 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:35.070 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:49.053270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27,28],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:21:35.070 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-11-24T08:21:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 msgr2=0x7f812406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 0x7f812406f280 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8134007380 tx=0x7f813400b040 comp rx=0 tx=0).stop 2023-11-24T08:21:35.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 msgr2=0x7f814409a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f813c0090b0 tx=0x7f813c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 shutdown_connections 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406cdc0 0x7f812406f280 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814415e390 0x7f814409a3f0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8144157c70 0x7f814409cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 --2- 172.21.15.107:0/157618558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81440cf450 0x7f81440cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.070+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 >> 172.21.15.107:0/157618558 conn(0x7f814406a920 msgr2=0x7f8144160010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.071+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 shutdown_connections 2023-11-24T08:21:35.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.071+0000 7f814a36f640 1 -- 172.21.15.107:0/157618558 wait complete. 2023-11-24T08:21:35.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.294+0000 7f724209b640 1 Processor -- start 2023-11-24T08:21:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.294+0000 7f724209b640 1 -- start start 2023-11-24T08:21:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.295+0000 7f724209b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.295+0000 7f724209b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 0x7f723c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.295+0000 7f724209b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c15ab80 con 0x7f723c0b2630 2023-11-24T08:21:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.295+0000 7f724209b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c15a990 con 0x7f723c0b2a70 2023-11-24T08:21:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.295+0000 7f724209b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c15e680 con 0x7f723c0b2250 2023-11-24T08:21:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.296+0000 7f723b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.296+0000 7f723bfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 0x7f723c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.296+0000 7f723b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56984/0 (socket says 172.21.15.107:56984) 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.296+0000 7f723b7fe640 1 -- 172.21.15.107:0/1423808920 learned_addr learned my addr 172.21.15.107:0/1423808920 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.297+0000 7f723b7fe640 1 -- 172.21.15.107:0/1423808920 >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 msgr2=0x7f723c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.297+0000 7f723b7fe640 1 --2- 172.21.15.107:0/1423808920 >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 0x7f723c15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.297+0000 7f723b7fe640 1 -- 172.21.15.107:0/1423808920 >> v1:172.21.15.120:6789/0 conn(0x7f723c0b2630 legacy=0x7f723c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.297+0000 7f723b7fe640 1 -- 172.21.15.107:0/1423808920 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c15e960 con 0x7f723c0b2250 2023-11-24T08:21:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.297+0000 7f723bfff640 1 --2- 172.21.15.107:0/1423808920 >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 0x7f723c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:21:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.298+0000 7f723b7fe640 1 --2- 172.21.15.107:0/1423808920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f722c009870 tx=0x7f722c02db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16a0a90040ec7482 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.299+0000 7f723affd640 1 -- 172.21.15.107:0/1423808920 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c02e940 con 0x7f723c0b2250 2023-11-24T08:21:35.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.299+0000 7f723affd640 1 -- 172.21.15.107:0/1423808920 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c02eaa0 con 0x7f723c0b2250 2023-11-24T08:21:35.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.299+0000 7f723affd640 1 -- 172.21.15.107:0/1423808920 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c02ee30 con 0x7f723c0b2250 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.299+0000 7f724209b640 1 -- 172.21.15.107:0/1423808920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 msgr2=0x7f723c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.299+0000 7f724209b640 1 --2- 172.21.15.107:0/1423808920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f722c009870 tx=0x7f722c02db90 comp rx=0 tx=0).stop 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 -- 172.21.15.107:0/1423808920 shutdown_connections 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 --2- 172.21.15.107:0/1423808920 >> v2:172.21.15.120:3300/0 conn(0x7f723c0b2a70 0x7f723c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 --2- 172.21.15.107:0/1423808920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2250 0x7f723c15df80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 -- 172.21.15.107:0/1423808920 >> 172.21.15.107:0/1423808920 conn(0x7f723c06a920 msgr2=0x7f723c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:35.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 -- 172.21.15.107:0/1423808920 shutdown_connections 2023-11-24T08:21:35.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.300+0000 7f724209b640 1 -- 172.21.15.107:0/1423808920 wait complete. 2023-11-24T08:21:35.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.301+0000 7f724209b640 1 Processor -- start 2023-11-24T08:21:35.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 -- start start 2023-11-24T08:21:35.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 0x7f723c0ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 0x7f723c0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c1f7970 con 0x7f723c0b2a70 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c0adee0 con 0x7f723c0b2250 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.302+0000 7f724209b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c1f6d60 con 0x7f723c0b2630 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37672/0 (socket says 172.21.15.107:37672) 2023-11-24T08:21:35.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 -- 172.21.15.107:0/492958408 learned_addr learned my addr 172.21.15.107:0/492958408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723b7fe640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 0x7f723c0ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f7233fff640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 0x7f723c0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 -- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 msgr2=0x7f723c0cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 0x7f723c0cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 -- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 msgr2=0x7f723c0ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 0x7f723c0ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723bfff640 1 -- 172.21.15.107:0/492958408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0094d0 con 0x7f723c0b2a70 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f723b7fe640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 0x7f723c0ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.303+0000 7f7233fff640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 0x7f723c0cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f723bfff640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f7234007d40 tx=0x7f723400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234003990 con 0x7f723c0b2a70 2023-11-24T08:21:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723400f040 con 0x7f723c0b2a70 2023-11-24T08:21:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234003d30 con 0x7f723c0b2a70 2023-11-24T08:21:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c0cd7c0 con 0x7f723c0b2a70 2023-11-24T08:21:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.304+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c1f7320 con 0x7f723c0b2a70 2023-11-24T08:21:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.305+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c1f7ae0 con 0x7f723c0b2a70 2023-11-24T08:21:35.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.306+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7234004030 con 0x7f723c0b2a70 2023-11-24T08:21:35.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.307+0000 7f7238ff9640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 0x7f721c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.308+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f72340888a0 con 0x7f723c0b2a70 2023-11-24T08:21:35.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.308+0000 7f723b7fe640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 0x7f721c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.308+0000 7f723b7fe640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 0x7f721c06f280 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f722c038000 tx=0x7f722c032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.312+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7234055450 con 0x7f723c0b2a70 2023-11-24T08:21:35.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.507+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f723c067470 con 0x7f723c0b2a70 2023-11-24T08:21:35.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.510+0000 7f7238ff9640 1 -- 172.21.15.107:0/492958408 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+18495 (secure 0 0 0) 0x7f723401a090 con 0x7f723c0b2a70 2023-11-24T08:21:35.512 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:35.512 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:35.109558+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 msgr2=0x7f721c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 0x7f721c06f280 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f722c038000 tx=0x7f722c032000 comp rx=0 tx=0).stop 2023-11-24T08:21:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 msgr2=0x7f723c0cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f7234007d40 tx=0x7f723400d4a0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 shutdown_connections 2023-11-24T08:21:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f721c06cdc0 0x7f721c06f280 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f723c0b2a70 0x7f723c0cf960 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f723c0b2630 0x7f723c0cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 --2- 172.21.15.107:0/492958408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f723c0b2250 0x7f723c0ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 >> 172.21.15.107:0/492958408 conn(0x7f723c06a920 msgr2=0x7f723c1585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 shutdown_connections 2023-11-24T08:21:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.513+0000 7f724209b640 1 -- 172.21.15.107:0/492958408 wait complete. 2023-11-24T08:21:35.534 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:21:35.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:35.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.746+0000 7f8403dd5640 1 Processor -- start 2023-11-24T08:21:35.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.746+0000 7f8403dd5640 1 -- start start 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83fc0b2250 0x7f83fc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0b2630 0x7f83fc0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 0x7f83fc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83fc15a990 con 0x7f83fc0b2250 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83fc15ab80 con 0x7f83fc0cd470 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.747+0000 7f8403dd5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc15e680 con 0x7f83fc0b2630 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83fc0b2250 0x7f83fc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83fc0b2250 0x7f83fc15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37686/0 (socket says 172.21.15.107:37686) 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 -- 172.21.15.107:0/3990922258 learned_addr learned my addr 172.21.15.107:0/3990922258 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8402372640 1 --2- 172.21.15.107:0/3990922258 >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 0x7f83fc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401370640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0b2630 0x7f83fc0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 -- 172.21.15.107:0/3990922258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0b2630 msgr2=0x7f83fc0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0b2630 0x7f83fc0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 -- 172.21.15.107:0/3990922258 >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 msgr2=0x7f83fc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 --2- 172.21.15.107:0/3990922258 >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 0x7f83fc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8401b71640 1 -- 172.21.15.107:0/3990922258 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc15e960 con 0x7f83fc0b2250 2023-11-24T08:21:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.748+0000 7f8402372640 1 --2- 172.21.15.107:0/3990922258 >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 0x7f83fc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.749+0000 7f8401b71640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2250 0x7f83fc15df80 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f83fc158560 tx=0x7f83ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ed27bc7fe5b4a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8400b6f640 1 -- 172.21.15.107:0/3990922258 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec02ca70 con 0x7f83fc0b2250 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8400b6f640 1 -- 172.21.15.107:0/3990922258 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ec02cbd0 con 0x7f83fc0b2250 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8400b6f640 1 -- 172.21.15.107:0/3990922258 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0343f0 con 0x7f83fc0b2250 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 -- 172.21.15.107:0/3990922258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2250 msgr2=0x7f83fc15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2250 0x7f83fc15df80 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f83fc158560 tx=0x7f83ec02bed0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 -- 172.21.15.107:0/3990922258 shutdown_connections 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 --2- 172.21.15.107:0/3990922258 >> v2:172.21.15.120:3300/0 conn(0x7f83fc0cd470 0x7f83fc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0b2630 0x7f83fc0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 --2- 172.21.15.107:0/3990922258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2250 0x7f83fc15df80 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.750+0000 7f8403dd5640 1 -- 172.21.15.107:0/3990922258 >> 172.21.15.107:0/3990922258 conn(0x7f83fc06a920 msgr2=0x7f83fc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:35.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.751+0000 7f8403dd5640 1 -- 172.21.15.107:0/3990922258 shutdown_connections 2023-11-24T08:21:35.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.751+0000 7f8403dd5640 1 -- 172.21.15.107:0/3990922258 wait complete. 2023-11-24T08:21:35.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.752+0000 7f8403dd5640 1 Processor -- start 2023-11-24T08:21:35.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.752+0000 7f8403dd5640 1 -- start start 2023-11-24T08:21:35.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83fc0b2250 0x7f83fc14c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0cd470 0x7f83fc1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc14f2c0 con 0x7f83fc0b2630 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc0c9770 con 0x7f83fc0b2250 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.753+0000 7f8403dd5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc0c86d0 con 0x7f83fc0cd470 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37702/0 (socket says 172.21.15.107:37702) 2023-11-24T08:21:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 -- 172.21.15.107:0/2887514160 learned_addr learned my addr 172.21.15.107:0/2887514160 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 -- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0cd470 msgr2=0x7f83fc1577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0cd470 0x7f83fc1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 -- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83fc0b2250 msgr2=0x7f83fc14c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401b71640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83fc0b2250 0x7f83fc14c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83fc0b2250 0x7f83fc14c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 -- 172.21.15.107:0/2887514160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec009b30 con 0x7f83fc0b2630 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.754+0000 7f8401370640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c7f0 tx=0x7f83f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.755+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0004220 con 0x7f83fc0b2630 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.755+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0004380 con 0x7f83fc0b2630 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.755+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0007860 con 0x7f83fc0b2630 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.755+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83fc0c89b0 con 0x7f83fc0b2630 2023-11-24T08:21:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.755+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83fc0c8f70 con 0x7f83fc0b2630 2023-11-24T08:21:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.756+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc067470 con 0x7f83fc0b2630 2023-11-24T08:21:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.757+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f83f0007bd0 con 0x7f83fc0b2630 2023-11-24T08:21:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.758+0000 7f83fa7fc640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 0x7f83e806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:35.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.758+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f83f0088680 con 0x7f83fc0b2630 2023-11-24T08:21:35.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.758+0000 7f8401b71640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 0x7f83e806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:35.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.759+0000 7f8401b71640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 0x7f83e806f280 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f83ec035000 tx=0x7f83ec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:35.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.762+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83f0055120 con 0x7f83fc0b2630 2023-11-24T08:21:35.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.993+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83fc0cea00 con 0x7f83fc0b2630 2023-11-24T08:21:35.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.994+0000 7f83fa7fc640 1 -- 172.21.15.107:0/2887514160 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f83f001e090 con 0x7f83fc0b2630 2023-11-24T08:21:35.997 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:35.997 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:49.053270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27,28],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:21:35.997 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-11-24T08:21:36.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.997+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 msgr2=0x7f83e806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.997+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 0x7f83e806f280 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f83ec035000 tx=0x7f83ec030000 comp rx=0 tx=0).stop 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 msgr2=0x7f83fc1547e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c7f0 tx=0x7f83f000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 shutdown_connections 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f83e806cdc0 0x7f83e806f280 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83fc0cd470 0x7f83fc1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83fc0b2630 0x7f83fc1547e0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 --2- 172.21.15.107:0/2887514160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83fc0b2250 0x7f83fc14c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 >> 172.21.15.107:0/2887514160 conn(0x7f83fc06a920 msgr2=0x7f83fc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:36.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 shutdown_connections 2023-11-24T08:21:36.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:35.998+0000 7f8403dd5640 1 -- 172.21.15.107:0/2887514160 wait complete. 2023-11-24T08:21:36.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:36.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.226+0000 7fa4cc09b640 1 Processor -- start 2023-11-24T08:21:36.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.226+0000 7fa4cc09b640 1 -- start start 2023-11-24T08:21:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4cc09b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa4c40cf450 0x7fa4c40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4cc09b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c40cece0 con 0x7fa4c40cdfb0 2023-11-24T08:21:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4cc09b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c406d040 con 0x7fa4c415c7f0 2023-11-24T08:21:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4cc09b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c40cfe40 con 0x7fa4c40cf450 2023-11-24T08:21:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4ca638640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4c40cdfb0 0x7fa4c40ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41660/0 (socket says 172.21.15.107:41660) 2023-11-24T08:21:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4ca638640 1 -- 172.21.15.107:0/869469829 learned_addr learned my addr 172.21.15.107:0/869469829 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.228+0000 7fa4c9636640 1 --2- 172.21.15.107:0/869469829 >> v2:172.21.15.107:3300/0 conn(0x7fa4c40cf450 0x7fa4c40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.229+0000 7fa4c8e35640 1 -- 172.21.15.107:0/869469829 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118167441 0 0) 0x7fa4c40cece0 con 0x7fa4c40cdfb0 2023-11-24T08:21:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.229+0000 7fa4c8e35640 1 -- 172.21.15.107:0/869469829 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ac003520 con 0x7fa4c40cdfb0 2023-11-24T08:21:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.229+0000 7fa4c9636640 1 -- 172.21.15.107:0/869469829 >> v1:172.21.15.120:6789/0 conn(0x7fa4c415c7f0 legacy=0x7fa4c406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.229+0000 7fa4c9636640 1 -- 172.21.15.107:0/869469829 >> v1:172.21.15.107:6789/0 conn(0x7fa4c40cdfb0 legacy=0x7fa4c40ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.229+0000 7fa4c9636640 1 -- 172.21.15.107:0/869469829 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c406d320 con 0x7fa4c40cf450 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.230+0000 7fa4c9636640 1 --2- 172.21.15.107:0/869469829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c40cf900 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80097a0 tx=0x7fa4b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7590b1b106c2b100 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.230+0000 7fa4c8e35640 1 -- 172.21.15.107:0/869469829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8030610 con 0x7fa4c40cf450 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4c8e35640 1 -- 172.21.15.107:0/869469829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b8030770 con 0x7fa4c40cf450 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4c8e35640 1 -- 172.21.15.107:0/869469829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8030b00 con 0x7fa4c40cf450 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 -- 172.21.15.107:0/869469829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 msgr2=0x7fa4c40cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/869469829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c40cf900 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80097a0 tx=0x7fa4b802b960 comp rx=0 tx=0).stop 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 -- 172.21.15.107:0/869469829 shutdown_connections 2023-11-24T08:21:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/869469829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c40cf900 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 -- 172.21.15.107:0/869469829 >> 172.21.15.107:0/869469829 conn(0x7fa4c406a920 msgr2=0x7fa4c406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 -- 172.21.15.107:0/869469829 shutdown_connections 2023-11-24T08:21:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.231+0000 7fa4cc09b640 1 -- 172.21.15.107:0/869469829 wait complete. 2023-11-24T08:21:36.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.233+0000 7fa4cc09b640 1 Processor -- start 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.233+0000 7fa4cc09b640 1 -- start start 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4c40cdfb0 0x7fa4c414e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4c415c7f0 0x7fa4c415ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4150fe0 con 0x7fa4c40cf450 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40ad640 con 0x7fa4c415c7f0 2023-11-24T08:21:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4cc09b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c414f240 con 0x7fa4c40cdfb0 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4c9636640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4c9636640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37710/0 (socket says 172.21.15.107:37710) 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.234+0000 7fa4c9636640 1 -- 172.21.15.107:0/4277740904 learned_addr learned my addr 172.21.15.107:0/4277740904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4ca638640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4c415c7f0 0x7fa4c415ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 -- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4c40cdfb0 msgr2=0x7fa4c414e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4c40cdfb0 0x7fa4c414e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 -- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4c415c7f0 msgr2=0x7fa4c415ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4c415c7f0 0x7fa4c415ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 -- 172.21.15.107:0/4277740904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b8009400 con 0x7fa4c40cf450 2023-11-24T08:21:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c9636640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fa4b802bf70 tx=0x7fa4b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b803e040 con 0x7fa4c40cf450 2023-11-24T08:21:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b8039cb0 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c4150d00 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.235+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8044630 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.236+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c40b3df0 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.237+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c40b3970 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.237+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fa4b804d3b0 con 0x7fa4c40cf450 2023-11-24T08:21:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.238+0000 7fa4c27fc640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 0x7fa4b006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.238+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6160+0+0 (secure 0 0 0) 0x7fa4b80aab10 con 0x7fa4c40cf450 2023-11-24T08:21:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.239+0000 7fa4c9e37640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 0x7fa4b006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.239+0000 7fa4c9e37640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 0x7fa4b006f280 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40012e0 tx=0x7fa4b4007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.243+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4c40b3970 con 0x7fa4c40cf450 2023-11-24T08:21:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.473+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4c40b20e0 con 0x7fa4c40cf450 2023-11-24T08:21:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.474+0000 7fa4c27fc640 1 -- 172.21.15.107:0/4277740904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4939 (secure 0 0 0) 0x7fa4c40b20e0 con 0x7fa4c40cf450 2023-11-24T08:21:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:36.476 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:20:32.719679+0000","modified":"2023-11-24T08:20:49.053270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11007},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11007":{"gid":11007,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2859392170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2859392170},{"type":"v1","addr":"172.21.15.107:6837","nonce":2859392170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27,28],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-11-24T08:21:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-11-24T08:21:36.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 msgr2=0x7fa4b006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 0x7fa4b006f280 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40012e0 tx=0x7fa4b4007d30 comp rx=0 tx=0).stop 2023-11-24T08:21:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 msgr2=0x7fa4c414ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fa4b802bf70 tx=0x7fa4b8004730 comp rx=0 tx=0).stop 2023-11-24T08:21:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 shutdown_connections 2023-11-24T08:21:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa4b006cdc0 0x7fa4b006f280 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4c415c7f0 0x7fa4c415ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4c40cf450 0x7fa4c414ed00 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 --2- 172.21.15.107:0/4277740904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4c40cdfb0 0x7fa4c414e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.476+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 >> 172.21.15.107:0/4277740904 conn(0x7fa4c406a920 msgr2=0x7fa4c406c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.477+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 shutdown_connections 2023-11-24T08:21:36.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.477+0000 7fa4cc09b640 1 -- 172.21.15.107:0/4277740904 wait complete. 2023-11-24T08:21:36.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:36.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.700+0000 7f60dffb0640 1 Processor -- start 2023-11-24T08:21:36.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.701+0000 7f60dffb0640 1 -- start start 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60dffb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60d806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60dffb0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d80cece0 con 0x7f60d80ce6c0 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60dffb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d806d040 con 0x7f60d80cf450 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60dffb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8156010 con 0x7f60d815c7f0 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60de54d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f60d80ce6c0 0x7f60d80714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41666/0 (socket says 172.21.15.107:41666) 2023-11-24T08:21:36.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.702+0000 7f60de54d640 1 -- 172.21.15.107:0/2411937305 learned_addr learned my addr 172.21.15.107:0/2411937305 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169236936 0 0) 0x7f60d80cece0 con 0x7f60d80ce6c0 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c4003520 con 0x7f60d80ce6c0 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60d806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662136867 0 0) 0x7f60c4003520 con 0x7f60d80ce6c0 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 msgr2=0x7f60d806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:36.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60d806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 >> v1:172.21.15.120:6789/0 conn(0x7f60d80cf450 legacy=0x7f60d80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d806d320 con 0x7f60d80ce6c0 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f60d00019f0 con 0x7f60d80ce6c0 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60d806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 >> v1:172.21.15.107:6789/0 conn(0x7f60d80ce6c0 legacy=0x7f60d80714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c40039a0 0x7f60c4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8156010 con 0x7f60c40047c0 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c400bb10 con 0x7f60c40039a0 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c400bc80 con 0x7f60d815c7f0 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.703+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c40039a0 0x7f60c4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60de54d640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60dd54b640 1 -- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 msgr2=0x7f60c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d815c7f0 0x7f60c40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60dd54b640 1 -- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c40039a0 msgr2=0x7f60c4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c40039a0 0x7f60c4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.704+0000 7f60dd54b640 1 -- 172.21.15.107:0/2411937305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4003520 con 0x7f60c40047c0 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.705+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60c4004c40 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f60c8001750 tx=0x7f60c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23d2a95f27c2771 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.705+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80052d0 con 0x7f60c40047c0 2023-11-24T08:21:36.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.705+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c802e050 con 0x7f60c40047c0 2023-11-24T08:21:36.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.705+0000 7f60dcd4a640 1 -- 172.21.15.107:0/2411937305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8003830 con 0x7f60c40047c0 2023-11-24T08:21:36.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 -- 172.21.15.107:0/2411937305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 msgr2=0x7f60c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2411937305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60c4004c40 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f60c8001750 tx=0x7f60c40057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60de54d640 1 -- 172.21.15.107:0/2411937305 reap_dead start 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 -- 172.21.15.107:0/2411937305 shutdown_connections 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 -- 172.21.15.107:0/2411937305 >> 172.21.15.107:0/2411937305 conn(0x7f60d806a920 msgr2=0x7f60d806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 -- 172.21.15.107:0/2411937305 shutdown_connections 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.706+0000 7f60dffb0640 1 -- 172.21.15.107:0/2411937305 wait complete. 2023-11-24T08:21:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.708+0000 7f60dffb0640 1 Processor -- start 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.708+0000 7f60dffb0640 1 -- start start 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d8151ca0 0x7f60d81f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d8152100 0x7f60d81f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80b2280 con 0x7f60c40047c0 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80b2560 con 0x7f60d8152100 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dffb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80b2840 con 0x7f60d8151ca0 2023-11-24T08:21:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37716/0 (socket says 172.21.15.107:37716) 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 -- 172.21.15.107:0/2559648381 learned_addr learned my addr 172.21.15.107:0/2559648381 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d8151ca0 0x7f60d81f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 -- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d8151ca0 msgr2=0x7f60d81f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d8151ca0 0x7f60d81f69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 -- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d8152100 msgr2=0x7f60d81f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d8152100 0x7f60d81f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.709+0000 7f60ddd4c640 1 -- 172.21.15.107:0/2559648381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d80b2b20 con 0x7f60c40047c0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60ddd4c640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010860 tx=0x7f60d4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400cb40 con 0x7f60c40047c0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400cca0 con 0x7f60c40047c0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4015690 con 0x7f60c40047c0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d80b2e00 con 0x7f60c40047c0 2023-11-24T08:21:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.710+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d815b180 con 0x7f60c40047c0 2023-11-24T08:21:36.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.711+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d8067470 con 0x7f60c40047c0 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.712+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f60d40157f0 con 0x7f60c40047c0 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.713+0000 7f60ce7fc640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 0x7f60bc06f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.713+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f60d4088b90 con 0x7f60c40047c0 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.713+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 0x7f60bc06f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.714+0000 7f60dd54b640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 0x7f60bc06f2b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f60c8033000 tx=0x7f60c8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.717+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60d4055740 con 0x7f60c40047c0 2023-11-24T08:21:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.909+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60d806ca90 con 0x7f60c40047c0 2023-11-24T08:21:36.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.913+0000 7f60ce7fc640 1 -- 172.21.15.107:0/2559648381 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+18495 (secure 0 0 0) 0x7f60d4058d60 con 0x7f60c40047c0 2023-11-24T08:21:36.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:36.915 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:35.109558+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:20:30.467990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-11-24T08:20:31.721268+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"new_pool","create_time":"2023-11-24T08:20:48.209202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":27,"snaps":[{"begin":2,"length":2}]},{"pool":28,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 msgr2=0x7f60bc06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 0x7f60bc06f2b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f60c8033000 tx=0x7f60c8009030 comp rx=0 tx=0).stop 2023-11-24T08:21:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 msgr2=0x7f60d81f45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010860 tx=0x7f60d4010d20 comp rx=0 tx=0).stop 2023-11-24T08:21:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 shutdown_connections 2023-11-24T08:21:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f60bc06cdf0 0x7f60bc06f2b0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d8152100 0x7f60d81f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d8151ca0 0x7f60d81f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 --2- 172.21.15.107:0/2559648381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60c40047c0 0x7f60d81f45a0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.915+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 >> 172.21.15.107:0/2559648381 conn(0x7f60d806a920 msgr2=0x7f60d81557e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.916+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 shutdown_connections 2023-11-24T08:21:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:36.916+0000 7f60dffb0640 1 -- 172.21.15.107:0/2559648381 wait complete. 2023-11-24T08:21:36.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:21:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.146+0000 7f00c31fc640 1 Processor -- start 2023-11-24T08:21:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.146+0000 7f00c31fc640 1 -- start start 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c31fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00bc0cff90 0x7f00bc15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c31fc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc0d13f0 con 0x7f00bc0cfb50 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c31fc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc0d16d0 con 0x7f00bc160dd0 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c31fc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc0d11e0 con 0x7f00bc0cff90 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00bbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f00bc0cfb50 0x7f00bc15f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41682/0 (socket says 172.21.15.107:41682) 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00bbfff640 1 -- 172.21.15.107:0/700484636 learned_addr learned my addr 172.21.15.107:0/700484636 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c1799640 1 --2- 172.21.15.107:0/700484636 >> v2:172.21.15.107:3300/0 conn(0x7f00bc0cff90 0x7f00bc15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c1799640 1 -- 172.21.15.107:0/700484636 >> v1:172.21.15.120:6789/0 conn(0x7f00bc160dd0 legacy=0x7f00bc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:37.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c1799640 1 -- 172.21.15.107:0/700484636 >> v1:172.21.15.107:6789/0 conn(0x7f00bc0cfb50 legacy=0x7f00bc15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.148+0000 7f00c1799640 1 -- 172.21.15.107:0/700484636 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc0d19b0 con 0x7f00bc0cff90 2023-11-24T08:21:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00c1799640 1 --2- 172.21.15.107:0/700484636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc15fa00 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009d90 tx=0x7f00b402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb35a746f21f72ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00bb7fe640 1 -- 172.21.15.107:0/700484636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b402ccf0 con 0x7f00bc0cff90 2023-11-24T08:21:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00bb7fe640 1 -- 172.21.15.107:0/700484636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b402ce50 con 0x7f00bc0cff90 2023-11-24T08:21:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00bb7fe640 1 -- 172.21.15.107:0/700484636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4034660 con 0x7f00bc0cff90 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00c31fc640 1 -- 172.21.15.107:0/700484636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 msgr2=0x7f00bc15fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00c31fc640 1 --2- 172.21.15.107:0/700484636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc15fa00 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009d90 tx=0x7f00b402bf40 comp rx=0 tx=0).stop 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.150+0000 7f00c31fc640 1 -- 172.21.15.107:0/700484636 shutdown_connections 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.151+0000 7f00c31fc640 1 --2- 172.21.15.107:0/700484636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc15fa00 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.151+0000 7f00c31fc640 1 -- 172.21.15.107:0/700484636 >> 172.21.15.107:0/700484636 conn(0x7f00bc08af60 msgr2=0x7f00bc0d3400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.151+0000 7f00c31fc640 1 -- 172.21.15.107:0/700484636 shutdown_connections 2023-11-24T08:21:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.151+0000 7f00c31fc640 1 -- 172.21.15.107:0/700484636 wait complete. 2023-11-24T08:21:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.152+0000 7f00c31fc640 1 Processor -- start 2023-11-24T08:21:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.152+0000 7f00c31fc640 1 -- start start 2023-11-24T08:21:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00bc0cfb50 0x7f00bc153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00bc160dd0 0x7f00bc156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc0cb650 con 0x7f00bc0cff90 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc0cb930 con 0x7f00bc160dd0 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.153+0000 7f00c31fc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc0cbc10 con 0x7f00bc0cfb50 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37746/0 (socket says 172.21.15.107:37746) 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 -- 172.21.15.107:0/546709370 learned_addr learned my addr 172.21.15.107:0/546709370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 -- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00bc0cfb50 msgr2=0x7f00bc153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00bc0cfb50 0x7f00bc153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 -- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00bc160dd0 msgr2=0x7f00bc156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00bc160dd0 0x7f00bc156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 -- 172.21.15.107:0/546709370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4009a70 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00bbfff640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f00a800c4a0 tx=0x7f00a800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8005ce0 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a8005e40 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8007270 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.154+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc0cbef0 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.155+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc153cb0 con 0x7f00bc0cff90 2023-11-24T08:21:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.156+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc0cc410 con 0x7f00bc0cff90 2023-11-24T08:21:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.157+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f00a8007490 con 0x7f00bc0cff90 2023-11-24T08:21:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.158+0000 7f00b97fa640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 0x7f00a006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.158+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f00a8087b00 con 0x7f00bc0cff90 2023-11-24T08:21:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.158+0000 7f00c0f98640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 0x7f00a006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.158+0000 7f00c0f98640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 0x7f00a006f280 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f00b0006fd0 tx=0x7f00b0006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.162+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00bc0cc410 con 0x7f00bc0cff90 2023-11-24T08:21:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.402+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f00bc0cc410 con 0x7f00bc0cff90 2023-11-24T08:21:37.404 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:21:37.403+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:21:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.406+0000 7f00b97fa640 1 -- 172.21.15.107:0/546709370 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) v1 ==== 161+0+0 (secure 0 0 0) 0x7f00bc0cc410 con 0x7f00bc0cff90 2023-11-24T08:21:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:21:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 msgr2=0x7f00a006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 0x7f00a006f280 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f00b0006fd0 tx=0x7f00b0006bc0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 msgr2=0x7f00bc0cc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f00a800c4a0 tx=0x7f00a800c960 comp rx=0 tx=0).stop 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 shutdown_connections 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00a006cdc0 0x7f00a006f280 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00bc160dd0 0x7f00bc156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00bc0cff90 0x7f00bc0cc6f0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 --2- 172.21.15.107:0/546709370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00bc0cfb50 0x7f00bc153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 >> 172.21.15.107:0/546709370 conn(0x7f00bc08af60 msgr2=0x7f00bc161df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:37.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.409+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 shutdown_connections 2023-11-24T08:21:37.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.410+0000 7f00c31fc640 1 -- 172.21.15.107:0/546709370 wait complete. 2023-11-24T08:21:37.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:21:37.512 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:21:37.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.626+0000 7ff4ebcc9640 1 Processor -- start 2023-11-24T08:21:37.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.626+0000 7ff4ebcc9640 1 -- start start 2023-11-24T08:21:37.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.627+0000 7ff4ebcc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.627+0000 7ff4ebcc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4e406de10 0x7ff4e40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.627+0000 7ff4ebcc9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e415ec90 con 0x7ff4e40c49f0 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.627+0000 7ff4ebcc9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40cd530 con 0x7ff4e406de10 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.627+0000 7ff4ebcc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c4ed0 con 0x7ff4e4072160 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4e406de10 0x7ff4e40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4e406de10 0x7ff4e40705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37766/0 (socket says 172.21.15.107:37766) 2023-11-24T08:21:37.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 -- 172.21.15.107:0/535986025 learned_addr learned my addr 172.21.15.107:0/535986025 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:37.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4e9a65640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 -- 172.21.15.107:0/535986025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 msgr2=0x7ff4e415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 -- 172.21.15.107:0/535986025 >> v1:172.21.15.120:6789/0 conn(0x7ff4e40c49f0 legacy=0x7ff4e406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4ea266640 1 -- 172.21.15.107:0/535986025 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40c51b0 con 0x7ff4e406de10 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.628+0000 7ff4e9a65640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.629+0000 7ff4ea266640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e406de10 0x7ff4e40705c0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00d150 tx=0x7ff4dc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e58248fc035b30 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.629+0000 7ff4e8a63640 1 -- 172.21.15.107:0/535986025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc02ea80 con 0x7ff4e406de10 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4e8a63640 1 -- 172.21.15.107:0/535986025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc02ebe0 con 0x7ff4e406de10 2023-11-24T08:21:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4e8a63640 1 -- 172.21.15.107:0/535986025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc009600 con 0x7ff4e406de10 2023-11-24T08:21:37.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/535986025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e406de10 msgr2=0x7ff4e40705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e406de10 0x7ff4e40705c0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00d150 tx=0x7ff4dc02e0e0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/535986025 shutdown_connections 2023-11-24T08:21:37.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e406de10 0x7ff4e40705c0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/535986025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/535986025 >> 172.21.15.107:0/535986025 conn(0x7ff4e40b0c50 msgr2=0x7ff4e40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:37.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/535986025 shutdown_connections 2023-11-24T08:21:37.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.630+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/535986025 wait complete. 2023-11-24T08:21:37.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.632+0000 7ff4ebcc9640 1 Processor -- start 2023-11-24T08:21:37.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.632+0000 7ff4ebcc9640 1 -- start start 2023-11-24T08:21:37.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4e406de10 0x7ff4e414f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40b2720 con 0x7ff4e40c49f0 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40b3580 con 0x7ff4e406de10 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ebcc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40b2970 con 0x7ff4e4072160 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ea266640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ea266640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37782/0 (socket says 172.21.15.107:37782) 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.633+0000 7ff4ea266640 1 -- 172.21.15.107:0/1767213428 learned_addr learned my addr 172.21.15.107:0/1767213428 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 -- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 msgr2=0x7ff4e415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 -- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4e406de10 msgr2=0x7ff4e414f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4e406de10 0x7ff4e414f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 -- 172.21.15.107:0/1767213428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc00bd40 con 0x7ff4e40c49f0 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ea266640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00d150 tx=0x7ff4dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc02ea80 con 0x7ff4e40c49f0 2023-11-24T08:21:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc036820 con 0x7ff4e40c49f0 2023-11-24T08:21:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc036c10 con 0x7ff4e40c49f0 2023-11-24T08:21:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e40b2c50 con 0x7ff4e40c49f0 2023-11-24T08:21:37.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.634+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e40b3210 con 0x7ff4e40c49f0 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.635+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e40b33a0 con 0x7ff4e40c49f0 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.636+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ff4dc02ed50 con 0x7ff4e40c49f0 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.637+0000 7ff4da7fc640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 0x7ff4c806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.637+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6190+0+0 (secure 0 0 0) 0x7ff4dc0ab050 con 0x7ff4e40c49f0 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.638+0000 7ff4e9a65640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 0x7ff4c806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:37.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.638+0000 7ff4e9a65640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 0x7ff4c806f2b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006fd0 tx=0x7ff4e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:37.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.642+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4dc077ae0 con 0x7ff4e40c49f0 2023-11-24T08:21:37.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:37.855+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff4e415ec90 con 0x7ff4e40c49f0 2023-11-24T08:21:38.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.409+0000 7ff4da7fc640 1 -- 172.21.15.107:0/1767213428 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff4e415ec90 con 0x7ff4e40c49f0 2023-11-24T08:21:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 msgr2=0x7ff4c806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 0x7ff4c806f2b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006fd0 tx=0x7ff4e0008040 comp rx=0 tx=0).stop 2023-11-24T08:21:38.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 msgr2=0x7ff4e415c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00d150 tx=0x7ff4dc004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 shutdown_connections 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff4c806cdf0 0x7ff4c806f2b0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e40c49f0 0x7ff4e415c4d0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4e4072160 0x7ff4e415bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 --2- 172.21.15.107:0/1767213428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4e406de10 0x7ff4e414f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 >> 172.21.15.107:0/1767213428 conn(0x7ff4e40b0c50 msgr2=0x7ff4e40ae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 shutdown_connections 2023-11-24T08:21:38.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.414+0000 7ff4ebcc9640 1 -- 172.21.15.107:0/1767213428 wait complete. 2023-11-24T08:21:38.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:21:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.639+0000 7f76b4872640 1 Processor -- start 2023-11-24T08:21:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.639+0000 7f76b4872640 1 -- start start 2023-11-24T08:21:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.640+0000 7f76b4872640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f76b0157c70 0x7f76b0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.640+0000 7f76b4872640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76b015e390 0x7f76b00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76b4872640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b00ce0b0 con 0x7f76b00cf450 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76b4872640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b00cfe40 con 0x7f76b015e390 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76b4872640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b0161960 con 0x7f76b0157c70 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76a7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76b00cf450 0x7f76b015dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45430/0 (socket says 172.21.15.107:45430) 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76a7fff640 1 -- 172.21.15.107:0/3811176397 learned_addr learned my addr 172.21.15.107:0/3811176397 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76aed9d640 1 --2- 172.21.15.107:0/3811176397 >> v2:172.21.15.107:3300/0 conn(0x7f76b015e390 0x7f76b00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76add9b640 1 -- 172.21.15.107:0/3811176397 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761232121 0 0) 0x7f76b00ce0b0 con 0x7f76b00cf450 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76add9b640 1 -- 172.21.15.107:0/3811176397 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f769c003520 con 0x7f76b00cf450 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76ae59c640 1 --2- 172.21.15.107:0/3811176397 >> v2:172.21.15.120:3300/0 conn(0x7f76b0157c70 0x7f76b0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76aed9d640 1 -- 172.21.15.107:0/3811176397 >> v2:172.21.15.120:3300/0 conn(0x7f76b0157c70 msgr2=0x7f76b0161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76aed9d640 1 --2- 172.21.15.107:0/3811176397 >> v2:172.21.15.120:3300/0 conn(0x7f76b0157c70 0x7f76b0161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76aed9d640 1 -- 172.21.15.107:0/3811176397 >> v1:172.21.15.107:6789/0 conn(0x7f76b00cf450 legacy=0x7f76b015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.641+0000 7f76aed9d640 1 -- 172.21.15.107:0/3811176397 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b015e8d0 con 0x7f76b015e390 2023-11-24T08:21:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.642+0000 7f76aed9d640 1 --2- 172.21.15.107:0/3811176397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00cf900 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f76b006d5e0 tx=0x7f76a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4ae49b56a35dc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76add9b640 1 -- 172.21.15.107:0/3811176397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a802cbf0 con 0x7f76b015e390 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76add9b640 1 -- 172.21.15.107:0/3811176397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a802cd50 con 0x7f76b015e390 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76add9b640 1 -- 172.21.15.107:0/3811176397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8034660 con 0x7f76b015e390 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 -- 172.21.15.107:0/3811176397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 msgr2=0x7f76b00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 --2- 172.21.15.107:0/3811176397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00cf900 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f76b006d5e0 tx=0x7f76a802be40 comp rx=0 tx=0).stop 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 -- 172.21.15.107:0/3811176397 shutdown_connections 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 --2- 172.21.15.107:0/3811176397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00cf900 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 --2- 172.21.15.107:0/3811176397 >> v2:172.21.15.120:3300/0 conn(0x7f76b0157c70 0x7f76b0161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 -- 172.21.15.107:0/3811176397 >> 172.21.15.107:0/3811176397 conn(0x7f76b006a920 msgr2=0x7f76b006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 -- 172.21.15.107:0/3811176397 shutdown_connections 2023-11-24T08:21:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.643+0000 7f76b4872640 1 -- 172.21.15.107:0/3811176397 wait complete. 2023-11-24T08:21:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.645+0000 7f76b4872640 1 Processor -- start 2023-11-24T08:21:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.645+0000 7f76b4872640 1 -- start start 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b00cf450 0x7f76b01f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76b0157c70 0x7f76b0155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b01f7430 con 0x7f76b015e390 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b00ae000 con 0x7f76b00cf450 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76b4872640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0155a70 con 0x7f76b0157c70 2023-11-24T08:21:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:38.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58508/0 (socket says 172.21.15.107:58508) 2023-11-24T08:21:38.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.646+0000 7f76aed9d640 1 -- 172.21.15.107:0/2808061896 learned_addr learned my addr 172.21.15.107:0/2808061896 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:38.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 -- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76b0157c70 msgr2=0x7f76b0155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76b0157c70 0x7f76b0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 -- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b00cf450 msgr2=0x7f76b01f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:38.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b00cf450 0x7f76b01f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 -- 172.21.15.107:0/2808061896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8009a70 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76aed9d640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009e80 tx=0x7f76a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8040650 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a80407b0 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b00ad3f0 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8040b60 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.647+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b00ad9b0 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.648+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b0067470 con 0x7f76b015e390 2023-11-24T08:21:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.649+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f76a8040cc0 con 0x7f76b015e390 2023-11-24T08:21:38.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.650+0000 7f76a6ffd640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 0x7f769006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:38.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.650+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f76a80ab340 con 0x7f76b015e390 2023-11-24T08:21:38.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.651+0000 7f76ae59c640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 0x7f769006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:38.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.651+0000 7f76ae59c640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 0x7f769006f280 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006fd0 tx=0x7f76a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:38.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.654+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76a8077ef0 con 0x7f76b015e390 2023-11-24T08:21:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:38.845+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76b00ae190 con 0x7f76b015e390 2023-11-24T08:21:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.422+0000 7f76a6ffd640 1 -- 172.21.15.107:0/2808061896 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v197) v1 ==== 181+0+0 (secure 0 0 0) 0x7f76a80440d0 con 0x7f76b015e390 2023-11-24T08:21:39.424 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:21:39.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.425+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 msgr2=0x7f769006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.425+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 0x7f769006f280 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006fd0 tx=0x7f76a0008040 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 msgr2=0x7f76b00ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009e80 tx=0x7f76a8004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 shutdown_connections 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f769006cdc0 0x7f769006f280 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b015e390 0x7f76b00ae490 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76b0157c70 0x7f76b0155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 --2- 172.21.15.107:0/2808061896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b00cf450 0x7f76b01f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 >> 172.21.15.107:0/2808061896 conn(0x7f76b006a920 msgr2=0x7f76b00b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:39.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 shutdown_connections 2023-11-24T08:21:39.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.426+0000 7f76b4872640 1 -- 172.21.15.107:0/2808061896 wait complete. 2023-11-24T08:21:39.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:21:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.667+0000 7f0429397640 1 Processor -- start 2023-11-24T08:21:39.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.667+0000 7f0429397640 1 -- start start 2023-11-24T08:21:39.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.668+0000 7f0429397640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 0x7f042406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f0429397640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04240cece0 con 0x7f04240ce6c0 2023-11-24T08:21:39.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f0429397640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042406d040 con 0x7f04240cf450 2023-11-24T08:21:39.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f0429397640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0424156010 con 0x7f042415c7f0 2023-11-24T08:21:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f04237fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04240ce6c0 0x7f04240714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45438/0 (socket says 172.21.15.107:45438) 2023-11-24T08:21:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f04237fe640 1 -- 172.21.15.107:0/2885906217 learned_addr learned my addr 172.21.15.107:0/2885906217 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f0422ffd640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 0x7f042406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697317696 0 0) 0x7f04240cece0 con 0x7f04240ce6c0 2023-11-24T08:21:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.669+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003520 con 0x7f04240ce6c0 2023-11-24T08:21:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848365704 0 0) 0x7f0410003520 con 0x7f04240ce6c0 2023-11-24T08:21:39.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 msgr2=0x7f042406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 0x7f042406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 >> v1:172.21.15.120:6789/0 conn(0x7f04240cf450 legacy=0x7f04240cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042406d320 con 0x7f04240ce6c0 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f041c0019f0 con 0x7f04240ce6c0 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 >> v1:172.21.15.107:6789/0 conn(0x7f04240ce6c0 legacy=0x7f04240714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100039a0 0x7f0410003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100047c0 0x7f0410004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 0x7f0410005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0424156010 con 0x7f04100039a0 2023-11-24T08:21:39.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041000ba30 con 0x7f04100047c0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041000bbf0 con 0x7f042415c7f0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.670+0000 7f0422ffd640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100039a0 0x7f0410003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f041a7fc640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100047c0 0x7f0410004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f0422ffd640 1 -- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 msgr2=0x7f0410005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f0422ffd640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042415c7f0 0x7f0410005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f0422ffd640 1 -- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100047c0 msgr2=0x7f0410004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f0422ffd640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100047c0 0x7f0410004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.671+0000 7f0422ffd640 1 -- 172.21.15.107:0/2885906217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410003520 con 0x7f04100039a0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.672+0000 7f0422ffd640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100039a0 0x7f0410003dc0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f0414009a90 tx=0x7f04100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed5f890880c7e9c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.672+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041402c6b0 con 0x7f04100039a0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.672+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041402c810 con 0x7f04100039a0 2023-11-24T08:21:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.672+0000 7f04227fc640 1 -- 172.21.15.107:0/2885906217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041402cba0 con 0x7f04100039a0 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 -- 172.21.15.107:0/2885906217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100039a0 msgr2=0x7f0410003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 --2- 172.21.15.107:0/2885906217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100039a0 0x7f0410003dc0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f0414009a90 tx=0x7f04100057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f04237fe640 1 -- 172.21.15.107:0/2885906217 reap_dead start 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 -- 172.21.15.107:0/2885906217 shutdown_connections 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 -- 172.21.15.107:0/2885906217 >> 172.21.15.107:0/2885906217 conn(0x7f042406a920 msgr2=0x7f042406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 -- 172.21.15.107:0/2885906217 shutdown_connections 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.673+0000 7f0429397640 1 -- 172.21.15.107:0/2885906217 wait complete. 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.674+0000 7f0429397640 1 Processor -- start 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.675+0000 7f0429397640 1 -- start start 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.675+0000 7f0429397640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100039a0 0x7f04240b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.675+0000 7f0429397640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f0429397640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042414a110 0x7f042414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f0429397640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042414aad0 con 0x7f04241573c0 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f0429397640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0424157930 con 0x7f04100039a0 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f0429397640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0424157c10 con 0x7f042414a110 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58520/0 (socket says 172.21.15.107:58520) 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 -- 172.21.15.107:0/3980727338 learned_addr learned my addr 172.21.15.107:0/3980727338 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f0422ffd640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100039a0 0x7f04240b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 -- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042414a110 msgr2=0x7f042414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042414a110 0x7f042414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 -- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100039a0 msgr2=0x7f04240b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:39.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100039a0 0x7f04240b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.676+0000 7f041a7fc640 1 -- 172.21.15.107:0/3980727338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0424157ef0 con 0x7f04241573c0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f041a7fc640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d570 tx=0x7f040c00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c005480 con 0x7f04241573c0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c003610 con 0x7f04241573c0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04241581d0 con 0x7f04241573c0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c003980 con 0x7f04241573c0 2023-11-24T08:21:39.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.677+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04240b1aa0 con 0x7f04241573c0 2023-11-24T08:21:39.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.678+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0424067470 con 0x7f04241573c0 2023-11-24T08:21:39.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.679+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f040c003ae0 con 0x7f04241573c0 2023-11-24T08:21:39.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.680+0000 7f041bfff640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 0x7f040406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:39.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.680+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f040c088570 con 0x7f04241573c0 2023-11-24T08:21:39.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.680+0000 7f0422ffd640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 0x7f040406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:39.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.681+0000 7f0422ffd640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 0x7f040406f280 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f0414036000 tx=0x7f0414030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.684+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f040c055120 con 0x7f04241573c0 2023-11-24T08:21:39.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:39.882+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f04240cd730 con 0x7f04241573c0 2023-11-24T08:21:40.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.435+0000 7f041bfff640 1 -- 172.21.15.107:0/3980727338 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v198) v1 ==== 169+0+0 (secure 0 0 0) 0x7f040c001230 con 0x7f04241573c0 2023-11-24T08:21:40.437 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 msgr2=0x7f040406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 0x7f040406f280 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f0414036000 tx=0x7f0414030000 comp rx=0 tx=0).stop 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 msgr2=0x7f042415a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d570 tx=0x7f040c00da30 comp rx=0 tx=0).stop 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 shutdown_connections 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f040406cdc0 0x7f040406f280 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f042414a110 0x7f042414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04241573c0 0x7f042415a0e0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 --2- 172.21.15.107:0/3980727338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04100039a0 0x7f04240b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 >> 172.21.15.107:0/3980727338 conn(0x7f042406a920 msgr2=0x7f042415ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 shutdown_connections 2023-11-24T08:21:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.439+0000 7f0429397640 1 -- 172.21.15.107:0/3980727338 wait complete. 2023-11-24T08:21:40.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm new_pool new_pool --yes-i-really-really-mean-it 2023-11-24T08:21:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.676+0000 7fdd0927c640 1 Processor -- start 2023-11-24T08:21:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.676+0000 7fdd0927c640 1 -- start start 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd0927c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd0415df80 0x7fdd040ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd0927c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0415e3f0 con 0x7fdd040b2250 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd0927c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0415ab80 con 0x7fdd040cd470 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd0927c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0415a990 con 0x7fdd0415df80 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd02ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdd040b2250 0x7fdd040b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45444/0 (socket says 172.21.15.107:45444) 2023-11-24T08:21:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.678+0000 7fdd02ffd640 1 -- 172.21.15.107:0/2695746193 learned_addr learned my addr 172.21.15.107:0/2695746193 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907368091 0 0) 0x7fdd0415e3f0 con 0x7fdd040b2250 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0005730 con 0x7fdd040b2250 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdcfa7fc640 1 --2- 172.21.15.107:0/2695746193 >> v2:172.21.15.120:3300/0 conn(0x7fdd0415df80 0x7fdd040ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467932811 0 0) 0x7fdcf0005730 con 0x7fdd040b2250 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 >> v2:172.21.15.120:3300/0 conn(0x7fdd0415df80 msgr2=0x7fdd040ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 --2- 172.21.15.107:0/2695746193 >> v2:172.21.15.120:3300/0 conn(0x7fdd0415df80 0x7fdd040ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 >> v1:172.21.15.120:6789/0 conn(0x7fdd040cd470 legacy=0x7fdd0415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd040b2d40 con 0x7fdd040b2250 2023-11-24T08:21:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdcf40019f0 con 0x7fdd040b2250 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 >> v1:172.21.15.107:6789/0 conn(0x7fdd040b2250 legacy=0x7fdd040b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0005b80 0x7fdcf0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcf00069d0 0x7fdcf0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 0x7fdcf000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000e290 con 0x7fdcf0007390 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000e400 con 0x7fdcf00069d0 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000e5c0 con 0x7fdcf0005b80 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.679+0000 7fdd037fe640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 0x7fdcf000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdcfa7fc640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcf00069d0 0x7fdcf0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd02ffd640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0005b80 0x7fdcf0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd037fe640 1 -- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0005b80 msgr2=0x7fdcf0005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd037fe640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0005b80 0x7fdcf0005f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd037fe640 1 -- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcf00069d0 msgr2=0x7fdcf0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd037fe640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcf00069d0 0x7fdcf0006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdd037fe640 1 -- 172.21.15.107:0/2695746193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf0005730 con 0x7fdcf0007390 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.680+0000 7fdcfa7fc640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcf00069d0 0x7fdcf0006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.681+0000 7fdd037fe640 1 -- 172.21.15.107:0/2695746193 reap_dead start 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.681+0000 7fdd037fe640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 0x7fdcf000db90 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc001620 tx=0x7fdcf0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab2f6a3abf4562b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.681+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc005270 con 0x7fdcf0007390 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.681+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc02e070 con 0x7fdcf0007390 2023-11-24T08:21:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.681+0000 7fdd027fc640 1 -- 172.21.15.107:0/2695746193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0035f0 con 0x7fdcf0007390 2023-11-24T08:21:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 -- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 msgr2=0x7fdcf000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 0x7fdcf000db90 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc001620 tx=0x7fdcf0007a00 comp rx=0 tx=0).stop 2023-11-24T08:21:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 -- 172.21.15.107:0/2695746193 shutdown_connections 2023-11-24T08:21:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 --2- 172.21.15.107:0/2695746193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf0007390 0x7fdcf000db90 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 -- 172.21.15.107:0/2695746193 >> 172.21.15.107:0/2695746193 conn(0x7fdd0406a920 msgr2=0x7fdd0406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 -- 172.21.15.107:0/2695746193 shutdown_connections 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.682+0000 7fdd0927c640 1 -- 172.21.15.107:0/2695746193 wait complete. 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.684+0000 7fdd0927c640 1 Processor -- start 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.684+0000 7fdd0927c640 1 -- start start 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0007390 0x7fdd040aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd040af270 0x7fdd040abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040ad740 con 0x7fdd040ac500 2023-11-24T08:21:40.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040acb30 con 0x7fdd040af270 2023-11-24T08:21:40.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd0927c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040ace10 con 0x7fdcf0007390 2023-11-24T08:21:40.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58534/0 (socket says 172.21.15.107:58534) 2023-11-24T08:21:40.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.685+0000 7fdd037fe640 1 -- 172.21.15.107:0/824031131 learned_addr learned my addr 172.21.15.107:0/824031131 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdcfa7fc640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd040af270 0x7fdd040abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd02ffd640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0007390 0x7fdd040aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 -- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0007390 msgr2=0x7fdd040aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0007390 0x7fdd040aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 -- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd040af270 msgr2=0x7fdd040abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd040af270 0x7fdd040abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 -- 172.21.15.107:0/824031131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd040ad0f0 con 0x7fdd040ac500 2023-11-24T08:21:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd037fe640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc02bbb0 tx=0x7fdcfc000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc007040 con 0x7fdd040ac500 2023-11-24T08:21:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc0071c0 con 0x7fdd040ac500 2023-11-24T08:21:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc007570 con 0x7fdd040ac500 2023-11-24T08:21:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd040ad3d0 con 0x7fdd040ac500 2023-11-24T08:21:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.686+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0414b010 con 0x7fdd040ac500 2023-11-24T08:21:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.687+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd040ad8f0 con 0x7fdd040ac500 2023-11-24T08:21:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.688+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fdcfc030030 con 0x7fdd040ac500 2023-11-24T08:21:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.689+0000 7fdcfbfff640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 0x7fdce406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.689+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5338+0+0 (secure 0 0 0) 0x7fdcfc0a8a60 con 0x7fdd040ac500 2023-11-24T08:21:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.690+0000 7fdd02ffd640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 0x7fdce406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.690+0000 7fdd02ffd640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 0x7fdce406f2b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4007c40 tx=0x7fdcf4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:40.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.694+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd040ad8f0 con 0x7fdd040ac500 2023-11-24T08:21:40.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:40.891+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "new_pool", "pool2": "new_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd0415d1b0 con 0x7fdd040ac500 2023-11-24T08:21:41.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.439+0000 7fdcfbfff640 1 -- 172.21.15.107:0/824031131 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "new_pool", "pool2": "new_pool", "yes_i_really_really_mean_it": true}]=0 pool 'new_pool' removed v199) v1 ==== 160+0+0 (secure 0 0 0) 0x7fdcfc042070 con 0x7fdd040ac500 2023-11-24T08:21:41.441 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'new_pool' removed 2023-11-24T08:21:41.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.442+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 msgr2=0x7fdce406f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.442+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 0x7fdce406f2b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4007c40 tx=0x7fdcf4009040 comp rx=0 tx=0).stop 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.442+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 msgr2=0x7fdd0414a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.442+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc02bbb0 tx=0x7fdcfc000fb0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 shutdown_connections 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdce406cdf0 0x7fdce406f2b0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd040ac500 0x7fdd0414a8f0 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd040af270 0x7fdd040abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 --2- 172.21.15.107:0/824031131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcf0007390 0x7fdd040aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 >> 172.21.15.107:0/824031131 conn(0x7fdd0406a920 msgr2=0x7fdd041599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 shutdown_connections 2023-11-24T08:21:41.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.443+0000 7fdd0927c640 1 -- 172.21.15.107:0/824031131 wait complete. 2023-11-24T08:21:41.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:21:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.679+0000 7f9e8c9af640 1 Processor -- start 2023-11-24T08:21:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.679+0000 7f9e8c9af640 1 -- start start 2023-11-24T08:21:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.680+0000 7f9e8c9af640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9e88072710 0x7f9e880ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8c9af640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9e880b3090 0x7f9e8815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8c9af640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880709c0 con 0x7f9e88154980 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8c9af640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e880b0df0 con 0x7f9e88072710 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8c9af640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8815af40 con 0x7f9e880b3090 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9e88072710 0x7f9e880ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9e88072710 0x7f9e880ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58542/0 (socket says 172.21.15.107:58542) 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 -- 172.21.15.107:0/2919877303 learned_addr learned my addr 172.21.15.107:0/2919877303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8759e640 1 --2- 172.21.15.107:0/2919877303 >> v2:172.21.15.120:3300/0 conn(0x7f9e880b3090 0x7f9e8815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 -- 172.21.15.107:0/2919877303 >> v2:172.21.15.120:3300/0 conn(0x7f9e880b3090 msgr2=0x7f9e8815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 --2- 172.21.15.107:0/2919877303 >> v2:172.21.15.120:3300/0 conn(0x7f9e880b3090 0x7f9e8815a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 -- 172.21.15.107:0/2919877303 >> v1:172.21.15.120:6789/0 conn(0x7f9e88154980 legacy=0x7f9e88072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.681+0000 7f9e8659c640 1 -- 172.21.15.107:0/2919877303 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8815b220 con 0x7f9e88072710 2023-11-24T08:21:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.683+0000 7f9e8659c640 1 --2- 172.21.15.107:0/2919877303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88072710 0x7f9e880ccfa0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f9e880ccc60 tx=0x7f9e7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa0ecea9dc33ca0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.683+0000 7f9e85d9b640 1 -- 172.21.15.107:0/2919877303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c031870 con 0x7f9e88072710 2023-11-24T08:21:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.683+0000 7f9e85d9b640 1 -- 172.21.15.107:0/2919877303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c0319d0 con 0x7f9e88072710 2023-11-24T08:21:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.683+0000 7f9e85d9b640 1 -- 172.21.15.107:0/2919877303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c031d60 con 0x7f9e88072710 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 -- 172.21.15.107:0/2919877303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88072710 msgr2=0x7f9e880ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/2919877303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88072710 0x7f9e880ccfa0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f9e880ccc60 tx=0x7f9e7c02b910 comp rx=0 tx=0).stop 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 -- 172.21.15.107:0/2919877303 shutdown_connections 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/2919877303 >> v2:172.21.15.120:3300/0 conn(0x7f9e880b3090 0x7f9e8815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/2919877303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88072710 0x7f9e880ccfa0 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 -- 172.21.15.107:0/2919877303 >> 172.21.15.107:0/2919877303 conn(0x7f9e8806a500 msgr2=0x7f9e8806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 -- 172.21.15.107:0/2919877303 shutdown_connections 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.684+0000 7f9e8c9af640 1 -- 172.21.15.107:0/2919877303 wait complete. 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.686+0000 7f9e8c9af640 1 Processor -- start 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.686+0000 7f9e8c9af640 1 -- start start 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 0x7f9e881596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 0x7f9e8814a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e88165b60 con 0x7f9e88154980 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e881657c0 con 0x7f9e880b3090 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8c9af640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8814aa50 con 0x7f9e88072710 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58548/0 (socket says 172.21.15.107:58548) 2023-11-24T08:21:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 -- 172.21.15.107:0/1604656558 learned_addr learned my addr 172.21.15.107:0/1604656558 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 -- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 msgr2=0x7f9e881596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8659c640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 0x7f9e8814a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e86d9d640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 0x7f9e881596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 0x7f9e881596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 -- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 msgr2=0x7f9e8814a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 0x7f9e8814a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8759e640 1 -- 172.21.15.107:0/1604656558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0094d0 con 0x7f9e88154980 2023-11-24T08:21:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e8659c640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 0x7f9e8814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.687+0000 7f9e86d9d640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 0x7f9e881596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e8759e640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f9e80007e40 tx=0x7f9e8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e80014620 con 0x7f9e88154980 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8000ee90 con 0x7f9e88154980 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e800149e0 con 0x7f9e88154980 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e88159da0 con 0x7f9e88154980 2023-11-24T08:21:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.688+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8815a360 con 0x7f9e88154980 2023-11-24T08:21:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.689+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e88165ca0 con 0x7f9e88154980 2023-11-24T08:21:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.690+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f9e80014b40 con 0x7f9e88154980 2023-11-24T08:21:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.691+0000 7f9e6f7fe640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 0x7f9e7406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.691+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9e80089090 con 0x7f9e88154980 2023-11-24T08:21:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.692+0000 7f9e86d9d640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 0x7f9e7406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.692+0000 7f9e86d9d640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 0x7f9e7406f280 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800a830 tx=0x7f9e78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.696+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e80055ba0 con 0x7f9e88154980 2023-11-24T08:21:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.890+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9e8804e840 con 0x7f9e88154980 2023-11-24T08:21:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.895+0000 7f9e6f7fe640 1 -- 172.21.15.107:0/1604656558 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e800591c0 con 0x7f9e88154980 2023-11-24T08:21:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 msgr2=0x7f9e7406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 0x7f9e7406f280 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800a830 tx=0x7f9e78008040 comp rx=0 tx=0).stop 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 msgr2=0x7f9e88149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f9e80007e40 tx=0x7f9e8000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 shutdown_connections 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e7406cdc0 0x7f9e7406f280 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e88154980 0x7f9e88149a20 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e880b3090 0x7f9e8814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 --2- 172.21.15.107:0/1604656558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e88072710 0x7f9e881596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 >> 172.21.15.107:0/1604656558 conn(0x7f9e8806a500 msgr2=0x7f9e8806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.898+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 shutdown_connections 2023-11-24T08:21:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:41.899+0000 7f9e8c9af640 1 -- 172.21.15.107:0/1604656558 wait complete. 2023-11-24T08:21:41.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size' 2023-11-24T08:21:42.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.132+0000 7f7ac11a9640 1 Processor -- start 2023-11-24T08:21:42.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.132+0000 7f7ac11a9640 1 -- start start 2023-11-24T08:21:42.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7ac11a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7abc157c70 0x7f7abc161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7ac11a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc0ce0b0 con 0x7f7abc15e390 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7ac11a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc161960 con 0x7f7abc0cf450 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7ac11a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc15e8d0 con 0x7f7abc157c70 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7abb59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7abc15e390 0x7f7abc0b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45460/0 (socket says 172.21.15.107:45460) 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7abad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7abc157c70 0x7f7abc161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7abb59e640 1 -- 172.21.15.107:0/1155594904 learned_addr learned my addr 172.21.15.107:0/1155594904 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.135+0000 7f7abad9d640 1 -- 172.21.15.107:0/1155594904 >> v1:172.21.15.120:6789/0 conn(0x7f7abc0cf450 legacy=0x7f7abc15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:42.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.135+0000 7f7abad9d640 1 -- 172.21.15.107:0/1155594904 >> v1:172.21.15.107:6789/0 conn(0x7f7abc15e390 legacy=0x7f7abc0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.135+0000 7f7abad9d640 1 -- 172.21.15.107:0/1155594904 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc15ebb0 con 0x7f7abc157c70 2023-11-24T08:21:42.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.134+0000 7f7ab9d9b640 1 -- 172.21.15.107:0/1155594904 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754596634 0 0) 0x7f7abc0ce0b0 con 0x7f7abc15e390 2023-11-24T08:21:42.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.136+0000 7f7abad9d640 1 --2- 172.21.15.107:0/1155594904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc157c70 0x7f7abc161df0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0097a0 tx=0x7f7aac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e83cbd5437ea19 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.136+0000 7f7ab9d9b640 1 -- 172.21.15.107:0/1155594904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac02e070 con 0x7f7abc157c70 2023-11-24T08:21:42.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.136+0000 7f7ab9d9b640 1 -- 172.21.15.107:0/1155594904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac0315a0 con 0x7f7abc157c70 2023-11-24T08:21:42.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.136+0000 7f7ab9d9b640 1 -- 172.21.15.107:0/1155594904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac031930 con 0x7f7abc157c70 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 -- 172.21.15.107:0/1155594904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc157c70 msgr2=0x7f7abc161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/1155594904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc157c70 0x7f7abc161df0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0097a0 tx=0x7f7aac02b840 comp rx=0 tx=0).stop 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 -- 172.21.15.107:0/1155594904 shutdown_connections 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/1155594904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc157c70 0x7f7abc161df0 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 -- 172.21.15.107:0/1155594904 >> 172.21.15.107:0/1155594904 conn(0x7f7abc06a920 msgr2=0x7f7abc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 -- 172.21.15.107:0/1155594904 shutdown_connections 2023-11-24T08:21:42.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.137+0000 7f7ac11a9640 1 -- 172.21.15.107:0/1155594904 wait complete. 2023-11-24T08:21:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.139+0000 7f7ac11a9640 1 Processor -- start 2023-11-24T08:21:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.139+0000 7f7ac11a9640 1 -- start start 2023-11-24T08:21:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc157c70 0x7f7abc0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc15e390 0x7f7abc1561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c9180 con 0x7f7abc0cf450 2023-11-24T08:21:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c8500 con 0x7f7abc15e390 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7ac11a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c87e0 con 0x7f7abc157c70 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58570/0 (socket says 172.21.15.107:58570) 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 -- 172.21.15.107:0/2545592716 learned_addr learned my addr 172.21.15.107:0/2545592716 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7aba59c640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc157c70 0x7f7abc0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 -- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc157c70 msgr2=0x7f7abc0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc157c70 0x7f7abc0cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 -- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc15e390 msgr2=0x7f7abc1561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc15e390 0x7f7abc1561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.140+0000 7f7abad9d640 1 -- 172.21.15.107:0/2545592716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac009400 con 0x7f7abc0cf450 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7abad9d640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f7aac034040 tx=0x7f7aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac032900 con 0x7f7abc0cf450 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac032a60 con 0x7f7abc0cf450 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac032e10 con 0x7f7abc0cf450 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc0c8ac0 con 0x7f7abc0cf450 2023-11-24T08:21:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.141+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc14aa10 con 0x7f7abc0cf450 2023-11-24T08:21:42.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.142+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc067470 con 0x7f7abc0cf450 2023-11-24T08:21:42.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.143+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7aac03b3f0 con 0x7f7abc0cf450 2023-11-24T08:21:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.144+0000 7f7aa37fe640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 0x7f7aa806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.145+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7aac0aa810 con 0x7f7abc0cf450 2023-11-24T08:21:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.145+0000 7f7aba59c640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 0x7f7aa806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.145+0000 7f7aba59c640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 0x7f7aa806f280 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0ae980 tx=0x7f7ab0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.149+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7aac077320 con 0x7f7abc0cf450 2023-11-24T08:21:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.407+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size"]} v 0) v1 -- 0x7f7abc162110 con 0x7f7abc0cf450 2023-11-24T08:21:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.455+0000 7f7aa37fe640 1 -- 172.21.15.107:0/2545592716 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_inherit_snapshot_namespace_and_size"]}]=0 v845) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7aac07a940 con 0x7f7abc0cf450 2023-11-24T08:21:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 msgr2=0x7f7aa806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 0x7f7aa806f280 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0ae980 tx=0x7f7ab0008040 comp rx=0 tx=0).stop 2023-11-24T08:21:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 msgr2=0x7f7abc14a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f7aac034040 tx=0x7f7aac0046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 shutdown_connections 2023-11-24T08:21:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7aa806cdc0 0x7f7aa806f280 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7abc15e390 0x7f7abc1561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7abc157c70 0x7f7abc0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 --2- 172.21.15.107:0/2545592716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7abc0cf450 0x7f7abc14a4d0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 >> 172.21.15.107:0/2545592716 conn(0x7f7abc06a920 msgr2=0x7f7abc0afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 shutdown_connections 2023-11-24T08:21:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.458+0000 7f7ac11a9640 1 -- 172.21.15.107:0/2545592716 wait complete. 2023-11-24T08:21:42.481 INFO:tasks.cephfs_test_runner:test_subvolume_clone_inherit_snapshot_namespace_and_size (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... ok 2023-11-24T08:21:42.481 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_retain_snapshot_with_snapshots (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) retain snapshots of a cloned subvolume and check disallowed operations 2023-11-24T08:21:42.481 INFO:tasks.cephfs_test_runner:test_subvolume_clone_retain_snapshot_with_snapshots (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:21:42.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:42.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.689+0000 7fb9cce0e640 1 Processor -- start 2023-11-24T08:21:42.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.689+0000 7fb9cce0e640 1 -- start start 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.690+0000 7fb9cce0e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9c815c7f0 0x7fb9c806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.690+0000 7fb9cce0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cf450 0x7fb9c80cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9cce0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80cece0 con 0x7fb9c80cdfb0 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9cce0e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c8156010 con 0x7fb9c815c7f0 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9cce0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c806ce30 con 0x7fb9c80cf450 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9c815c7f0 0x7fb9c806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9c815c7f0 0x7fb9c806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58574/0 (socket says 172.21.15.107:58574) 2023-11-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 -- 172.21.15.107:0/1874534708 learned_addr learned my addr 172.21.15.107:0/1874534708 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:42.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c5d9b640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cf450 0x7fb9c80cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 -- 172.21.15.107:0/1874534708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cf450 msgr2=0x7fb9c80cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:42.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cf450 0x7fb9c80cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.691+0000 7fb9c659c640 1 -- 172.21.15.107:0/1874534708 >> v1:172.21.15.120:6789/0 conn(0x7fb9c80cdfb0 legacy=0x7fb9c80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:42.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.692+0000 7fb9c659c640 1 -- 172.21.15.107:0/1874534708 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c806d040 con 0x7fb9c815c7f0 2023-11-24T08:21:42.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.693+0000 7fb9c659c640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c815c7f0 0x7fb9c806db20 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009870 tx=0x7fb9b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1d6996798be8d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.693+0000 7fb9c559a640 1 -- 172.21.15.107:0/1874534708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8031640 con 0x7fb9c815c7f0 2023-11-24T08:21:42.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.693+0000 7fb9c559a640 1 -- 172.21.15.107:0/1874534708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b80317a0 con 0x7fb9c815c7f0 2023-11-24T08:21:42.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.693+0000 7fb9c559a640 1 -- 172.21.15.107:0/1874534708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8031b30 con 0x7fb9c815c7f0 2023-11-24T08:21:42.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 -- 172.21.15.107:0/1874534708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c815c7f0 msgr2=0x7fb9c806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c815c7f0 0x7fb9c806db20 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009870 tx=0x7fb9b802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 -- 172.21.15.107:0/1874534708 shutdown_connections 2023-11-24T08:21:42.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cf450 0x7fb9c80cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/1874534708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c815c7f0 0x7fb9c806db20 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 -- 172.21.15.107:0/1874534708 >> 172.21.15.107:0/1874534708 conn(0x7fb9c806a920 msgr2=0x7fb9c806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:42.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 -- 172.21.15.107:0/1874534708 shutdown_connections 2023-11-24T08:21:42.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.694+0000 7fb9cce0e640 1 -- 172.21.15.107:0/1874534708 wait complete. 2023-11-24T08:21:42.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.696+0000 7fb9cce0e640 1 Processor -- start 2023-11-24T08:21:42.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.696+0000 7fb9cce0e640 1 -- start start 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cdfb0 0x7fb9c8156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c815c7f0 0x7fb9c80ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8155590 con 0x7fb9c80cf450 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8157de0 con 0x7fb9c815c7f0 2023-11-24T08:21:42.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.697+0000 7fb9cce0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80ada50 con 0x7fb9c80cdfb0 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58576/0 (socket says 172.21.15.107:58576) 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 -- 172.21.15.107:0/3192696007 learned_addr learned my addr 172.21.15.107:0/3192696007 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c659c640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cdfb0 0x7fb9c8156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c6d9d640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c815c7f0 0x7fb9c80ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 -- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cdfb0 msgr2=0x7fb9c8156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cdfb0 0x7fb9c8156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 -- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c815c7f0 msgr2=0x7fb9c80ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c815c7f0 0x7fb9c80ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.698+0000 7fb9c5d9b640 1 -- 172.21.15.107:0/3192696007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b80094d0 con 0x7fb9c80cf450 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9c5d9b640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c820 tx=0x7fb9bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc004220 con 0x7fb9c80cf450 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc0043a0 con 0x7fb9c80cf450 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc007a70 con 0x7fb9c80cf450 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c81571d0 con 0x7fb9c80cf450 2023-11-24T08:21:42.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.699+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c8157790 con 0x7fb9c80cf450 2023-11-24T08:21:42.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.700+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8067470 con 0x7fb9c80cf450 2023-11-24T08:21:42.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.702+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fb9bc007bd0 con 0x7fb9c80cf450 2023-11-24T08:21:42.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.702+0000 7fb9b6ffd640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 0x7fb9a806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:42.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.702+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb9bc088380 con 0x7fb9c80cf450 2023-11-24T08:21:42.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.703+0000 7fb9c659c640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 0x7fb9a806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:42.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.703+0000 7fb9c659c640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 0x7fb9a806f2b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8036000 tx=0x7fb9b803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:42.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.707+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9bc054f30 con 0x7fb9c80cf450 2023-11-24T08:21:42.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.899+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9c80cc3b0 con 0x7fb9c80cf450 2023-11-24T08:21:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.900+0000 7fb9b6ffd640 1 -- 172.21.15.107:0/3192696007 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+12998 (secure 0 0 0) 0x7fb9bc01e070 con 0x7fb9c80cf450 2023-11-24T08:21:42.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:42.902 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":199,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:41.436310+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/2859392170":"2023-11-25T08:21:37.404054+0000","172.21.15.107:6836/2859392170":"2023-11-25T08:21:37.404054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 msgr2=0x7fb9a806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 0x7fb9a806f2b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8036000 tx=0x7fb9b803c040 comp rx=0 tx=0).stop 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 msgr2=0x7fb9c8155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c820 tx=0x7fb9bc00cce0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 shutdown_connections 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9a806cdf0 0x7fb9a806f2b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c815c7f0 0x7fb9c80ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9c80cf450 0x7fb9c8155050 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 --2- 172.21.15.107:0/3192696007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9c80cdfb0 0x7fb9c8156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 >> 172.21.15.107:0/3192696007 conn(0x7fb9c806a920 msgr2=0x7fb9c80aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 shutdown_connections 2023-11-24T08:21:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:42.903+0000 7fb9cce0e640 1 -- 172.21.15.107:0/3192696007 wait complete. 2023-11-24T08:21:42.924 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:21:42.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:21:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.140+0000 7f802430c640 1 Processor -- start 2023-11-24T08:21:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.141+0000 7f802430c640 1 -- start start 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f802430c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 0x7f801c0d4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f802430c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f801c0cfc80 0x7f801c0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f802430c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0d05e0 con 0x7f801c0d1e30 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f802430c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c0cf300 con 0x7f801c0cfc80 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f802430c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c0d1090 con 0x7f801c15d580 2023-11-24T08:21:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f80228a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f801c0cfc80 0x7f801c0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f80228a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f801c0cfc80 0x7f801c0d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58588/0 (socket says 172.21.15.107:58588) 2023-11-24T08:21:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.142+0000 7f80228a9640 1 -- 172.21.15.107:0/38101544 learned_addr learned my addr 172.21.15.107:0/38101544 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80220a8640 1 --2- 172.21.15.107:0/38101544 >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 0x7f801c0d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80228a9640 1 -- 172.21.15.107:0/38101544 >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 msgr2=0x7f801c0d4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80228a9640 1 --2- 172.21.15.107:0/38101544 >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 0x7f801c0d4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80228a9640 1 -- 172.21.15.107:0/38101544 >> v1:172.21.15.107:6789/0 conn(0x7f801c0d1e30 legacy=0x7f801c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80228a9640 1 -- 172.21.15.107:0/38101544 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0d1370 con 0x7f801c0cfc80 2023-11-24T08:21:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.143+0000 7f80220a8640 1 --2- 172.21.15.107:0/38101544 >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 0x7f801c0d4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.144+0000 7f80228a9640 1 --2- 172.21.15.107:0/38101544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c0cfc80 0x7f801c0d00a0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f801c072b90 tx=0x7f801402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e79a5ae0ef49e0b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.144+0000 7f80210a6640 1 -- 172.21.15.107:0/38101544 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014030860 con 0x7f801c0cfc80 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.144+0000 7f80210a6640 1 -- 172.21.15.107:0/38101544 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80140309c0 con 0x7f801c0cfc80 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.144+0000 7f80210a6640 1 -- 172.21.15.107:0/38101544 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014030d50 con 0x7f801c0cfc80 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 -- 172.21.15.107:0/38101544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c0cfc80 msgr2=0x7f801c0d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 --2- 172.21.15.107:0/38101544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c0cfc80 0x7f801c0d00a0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f801c072b90 tx=0x7f801402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 -- 172.21.15.107:0/38101544 shutdown_connections 2023-11-24T08:21:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 --2- 172.21.15.107:0/38101544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c0cfc80 0x7f801c0d00a0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 --2- 172.21.15.107:0/38101544 >> v2:172.21.15.120:3300/0 conn(0x7f801c15d580 0x7f801c0d4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 -- 172.21.15.107:0/38101544 >> 172.21.15.107:0/38101544 conn(0x7f801c09c770 msgr2=0x7f801c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 -- 172.21.15.107:0/38101544 shutdown_connections 2023-11-24T08:21:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.145+0000 7f802430c640 1 -- 172.21.15.107:0/38101544 wait complete. 2023-11-24T08:21:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.147+0000 7f802430c640 1 Processor -- start 2023-11-24T08:21:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.147+0000 7f802430c640 1 -- start start 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0cfc80 0x7f801c14e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0d1e30 0x7f801c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0b60b0 con 0x7f801c15d580 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0b6320 con 0x7f801c0d1e30 2023-11-24T08:21:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f802430c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0b6600 con 0x7f801c0cfc80 2023-11-24T08:21:43.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f80228a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f80228a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58602/0 (socket says 172.21.15.107:58602) 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.148+0000 7f80228a9640 1 -- 172.21.15.107:0/1939348129 learned_addr learned my addr 172.21.15.107:0/1939348129 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80218a7640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0d1e30 0x7f801c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80220a8640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0cfc80 0x7f801c14e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 -- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0cfc80 msgr2=0x7f801c14e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0cfc80 0x7f801c14e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 -- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0d1e30 msgr2=0x7f801c0c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0d1e30 0x7f801c0c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 -- 172.21.15.107:0/1939348129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80140094d0 con 0x7f801c15d580 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f80228a9640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f801402b9e0 tx=0x7f8014004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.149+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140307f0 con 0x7f801c15d580 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.150+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8014043400 con 0x7f801c15d580 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.150+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140437b0 con 0x7f801c15d580 2023-11-24T08:21:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.150+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c0b68e0 con 0x7f801c15d580 2023-11-24T08:21:43.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.150+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c14ed20 con 0x7f801c15d580 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.151+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c0b6e00 con 0x7f801c15d580 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.152+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f8014043910 con 0x7f801c15d580 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.153+0000 7f801affd640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 0x7f800006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.153+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f801403e070 con 0x7f801c15d580 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.153+0000 7f80220a8640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 0x7f800006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.154+0000 7f80220a8640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 0x7f800006f280 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f8010008ee0 tx=0x7f8010008430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.157+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f801c0b6e00 con 0x7f801c15d580 2023-11-24T08:21:43.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.349+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f801c0b6e00 con 0x7f801c15d580 2023-11-24T08:21:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.350+0000 7f801affd640 1 -- 172.21.15.107:0/1939348129 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v199) v1 ==== 93+0+10 (secure 0 0 0) 0x7f801c0b6e00 con 0x7f801c15d580 2023-11-24T08:21:43.352 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 msgr2=0x7f800006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 0x7f800006f280 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f8010008ee0 tx=0x7f8010008430 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 msgr2=0x7f801c0cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f801402b9e0 tx=0x7f8014004730 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 shutdown_connections 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f800006cdc0 0x7f800006f280 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f801c15d580 0x7f801c0cb0e0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0d1e30 0x7f801c0c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 --2- 172.21.15.107:0/1939348129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0cfc80 0x7f801c14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 >> 172.21.15.107:0/1939348129 conn(0x7f801c09c770 msgr2=0x7f801c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 shutdown_connections 2023-11-24T08:21:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.353+0000 7f802430c640 1 -- 172.21.15.107:0/1939348129 wait complete. 2023-11-24T08:21:43.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots' 2023-11-24T08:21:43.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.582+0000 7ffa274df640 1 Processor -- start 2023-11-24T08:21:43.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.582+0000 7ffa274df640 1 -- start start 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa274df640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa20144b30 0x7ffa2015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa274df640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffa2015c410 0x7ffa200b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa274df640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2015bf80 con 0x7ffa200b1de0 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa274df640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200b1560 con 0x7ffa2015c410 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa274df640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200b0fb0 con 0x7ffa20144b30 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa25cdc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffa2015c410 0x7ffa200b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa26cde640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffa200b1de0 0x7ffa2006c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45484/0 (socket says 172.21.15.107:45484) 2023-11-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa26cde640 1 -- 172.21.15.107:0/2059543586 learned_addr learned my addr 172.21.15.107:0/2059543586 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa264dd640 1 --2- 172.21.15.107:0/2059543586 >> v2:172.21.15.120:3300/0 conn(0x7ffa20144b30 0x7ffa2015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa25cdc640 1 -- 172.21.15.107:0/2059543586 >> v2:172.21.15.120:3300/0 conn(0x7ffa20144b30 msgr2=0x7ffa2015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa25cdc640 1 --2- 172.21.15.107:0/2059543586 >> v2:172.21.15.120:3300/0 conn(0x7ffa20144b30 0x7ffa2015f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa25cdc640 1 -- 172.21.15.107:0/2059543586 >> v1:172.21.15.107:6789/0 conn(0x7ffa200b1de0 legacy=0x7ffa2006c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.584+0000 7ffa25cdc640 1 -- 172.21.15.107:0/2059543586 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200b1280 con 0x7ffa2015c410 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.585+0000 7ffa25cdc640 1 --2- 172.21.15.107:0/2059543586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa2015c410 0x7ffa200b0a70 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7ffa2006f690 tx=0x7ffa1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f8e3a05e6f5760 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.586+0000 7ffa254db640 1 -- 172.21.15.107:0/2059543586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10031830 con 0x7ffa2015c410 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.586+0000 7ffa254db640 1 -- 172.21.15.107:0/2059543586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa10031990 con 0x7ffa2015c410 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.586+0000 7ffa254db640 1 -- 172.21.15.107:0/2059543586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10031d20 con 0x7ffa2015c410 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.586+0000 7ffa274df640 1 -- 172.21.15.107:0/2059543586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa2015c410 msgr2=0x7ffa200b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.586+0000 7ffa274df640 1 --2- 172.21.15.107:0/2059543586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa2015c410 0x7ffa200b0a70 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7ffa2006f690 tx=0x7ffa1002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 -- 172.21.15.107:0/2059543586 shutdown_connections 2023-11-24T08:21:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 --2- 172.21.15.107:0/2059543586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa2015c410 0x7ffa200b0a70 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 --2- 172.21.15.107:0/2059543586 >> v2:172.21.15.120:3300/0 conn(0x7ffa20144b30 0x7ffa2015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 -- 172.21.15.107:0/2059543586 >> 172.21.15.107:0/2059543586 conn(0x7ffa200cb830 msgr2=0x7ffa200cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 -- 172.21.15.107:0/2059543586 shutdown_connections 2023-11-24T08:21:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.587+0000 7ffa274df640 1 -- 172.21.15.107:0/2059543586 wait complete. 2023-11-24T08:21:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.588+0000 7ffa274df640 1 Processor -- start 2023-11-24T08:21:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 -- start start 2023-11-24T08:21:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20144b30 0x7ffa20071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2015c410 0x7ffa20071780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20071cc0 con 0x7ffa200b1de0 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20139370 con 0x7ffa20144b30 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.589+0000 7ffa274df640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200c8160 con 0x7ffa2015c410 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58624/0 (socket says 172.21.15.107:58624) 2023-11-24T08:21:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 -- 172.21.15.107:0/2282529718 learned_addr learned my addr 172.21.15.107:0/2282529718 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 -- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2015c410 msgr2=0x7ffa20071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2015c410 0x7ffa20071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 -- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20144b30 msgr2=0x7ffa20071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20144b30 0x7ffa20071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 -- 172.21.15.107:0/2282529718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009400 con 0x7ffa200b1de0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.590+0000 7ffa264dd640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c820 tx=0x7ffa1c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.591+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c004220 con 0x7ffa200b1de0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.591+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c0043a0 con 0x7ffa200b1de0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.591+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c007a70 con 0x7ffa200b1de0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.591+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20136450 con 0x7ffa200b1de0 2023-11-24T08:21:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.591+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa200c83b0 con 0x7ffa200b1de0 2023-11-24T08:21:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.592+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20136970 con 0x7ffa200b1de0 2023-11-24T08:21:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.593+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ffa1c007bd0 con 0x7ffa200b1de0 2023-11-24T08:21:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.594+0000 7ffa16ffd640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 0x7ffa0406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:43.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.594+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffa1c0883c0 con 0x7ffa200b1de0 2023-11-24T08:21:43.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.594+0000 7ffa25cdc640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 0x7ffa0406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:43.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.595+0000 7ffa25cdc640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 0x7ffa0406f280 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ffa10036000 tx=0x7ffa10030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:43.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.598+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa1c054f70 con 0x7ffa200b1de0 2023-11-24T08:21:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:43.859+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots"]} v 0) v1 -- 0x7ffa2004e840 con 0x7ffa200b1de0 2023-11-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.468+0000 7ffa16ffd640 1 -- 172.21.15.107:0/2282529718 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots"]}]=0 v847) v1 ==== 187+0+0 (secure 0 0 0) 0x7ffa1c01e070 con 0x7ffa200b1de0 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.471+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 msgr2=0x7ffa0406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 0x7ffa0406f280 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ffa10036000 tx=0x7ffa10030000 comp rx=0 tx=0).stop 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 msgr2=0x7ffa20070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c820 tx=0x7ffa1c00cce0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 shutdown_connections 2023-11-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffa0406cdc0 0x7ffa0406f280 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2015c410 0x7ffa20071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20144b30 0x7ffa20071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 --2- 172.21.15.107:0/2282529718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa200b1de0 0x7ffa20070d00 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 >> 172.21.15.107:0/2282529718 conn(0x7ffa200cb830 msgr2=0x7ffa2015cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 shutdown_connections 2023-11-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.472+0000 7ffa274df640 1 -- 172.21.15.107:0/2282529718 wait complete. 2023-11-24T08:21:44.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:21:44.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.712+0000 7f1c9893d640 1 Processor -- start 2023-11-24T08:21:44.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.712+0000 7f1c9893d640 1 -- start start 2023-11-24T08:21:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c9893d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9415d810 0x7f1c94071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c9893d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 0x7f1c94072330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c9893d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9415e9b0 con 0x7f1c9415de90 2023-11-24T08:21:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c9893d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c94072870 con 0x7f1c9415d810 2023-11-24T08:21:44.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c9893d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9415ebe0 con 0x7f1c94071ea0 2023-11-24T08:21:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c92d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9415d810 0x7f1c94071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c92d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9415d810 0x7f1c94071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58632/0 (socket says 172.21.15.107:58632) 2023-11-24T08:21:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.714+0000 7f1c92d9d640 1 -- 172.21.15.107:0/1500049800 learned_addr learned my addr 172.21.15.107:0/1500049800 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c92d9d640 1 -- 172.21.15.107:0/1500049800 >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 msgr2=0x7f1c94072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1500049800 >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 0x7f1c94072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c92d9d640 1 --2- 172.21.15.107:0/1500049800 >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 0x7f1c94072330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c92d9d640 1 -- 172.21.15.107:0/1500049800 >> v1:172.21.15.120:6789/0 conn(0x7f1c9415de90 legacy=0x7f1c9415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c92d9d640 1 -- 172.21.15.107:0/1500049800 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94070860 con 0x7f1c9415d810 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.715+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1500049800 >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 0x7f1c94072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:44.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.716+0000 7f1c92d9d640 1 --2- 172.21.15.107:0/1500049800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c94071620 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f1c940ccbc0 tx=0x7f1c8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7ed195cdea1d41 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:44.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.716+0000 7f1c9259c640 1 -- 172.21.15.107:0/1500049800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84031870 con 0x7f1c9415d810 2023-11-24T08:21:44.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9259c640 1 -- 172.21.15.107:0/1500049800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c840319d0 con 0x7f1c9415d810 2023-11-24T08:21:44.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9259c640 1 -- 172.21.15.107:0/1500049800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84031d60 con 0x7f1c9415d810 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 -- 172.21.15.107:0/1500049800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 msgr2=0x7f1c94071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1500049800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c94071620 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f1c940ccbc0 tx=0x7f1c8402b910 comp rx=0 tx=0).stop 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 -- 172.21.15.107:0/1500049800 shutdown_connections 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1500049800 >> v2:172.21.15.120:3300/0 conn(0x7f1c94071ea0 0x7f1c94072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1500049800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c94071620 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 -- 172.21.15.107:0/1500049800 >> 172.21.15.107:0/1500049800 conn(0x7f1c9406a500 msgr2=0x7f1c9406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 -- 172.21.15.107:0/1500049800 shutdown_connections 2023-11-24T08:21:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.717+0000 7f1c9893d640 1 -- 172.21.15.107:0/1500049800 wait complete. 2023-11-24T08:21:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.719+0000 7f1c9893d640 1 Processor -- start 2023-11-24T08:21:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.719+0000 7f1c9893d640 1 -- start start 2023-11-24T08:21:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c94071ea0 0x7f1c941f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9415de90 0x7f1c9415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c94151790 con 0x7f1c9415d810 2023-11-24T08:21:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c941519c0 con 0x7f1c9415de90 2023-11-24T08:21:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.720+0000 7f1c9893d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c94151ca0 con 0x7f1c94071ea0 2023-11-24T08:21:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58634/0 (socket says 172.21.15.107:58634) 2023-11-24T08:21:44.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1004884122 learned_addr learned my addr 172.21.15.107:0/1004884122 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c94071ea0 msgr2=0x7f1c941f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c94071ea0 0x7f1c941f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9415de90 msgr2=0x7f1c9415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9415de90 0x7f1c9415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1004884122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c840094d0 con 0x7f1c9415d810 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c7f0 tx=0x7f1c7c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.721+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c007a30 con 0x7f1c9415d810 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.722+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c007b90 con 0x7f1c9415d810 2023-11-24T08:21:44.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.722+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00ce40 con 0x7f1c9415d810 2023-11-24T08:21:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.722+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c94151f80 con 0x7f1c9415d810 2023-11-24T08:21:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.722+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c94152540 con 0x7f1c9415d810 2023-11-24T08:21:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.723+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c94152730 con 0x7f1c9415d810 2023-11-24T08:21:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.724+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f1c7c0113f0 con 0x7f1c9415d810 2023-11-24T08:21:44.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.725+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 0x7f1c7406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:44.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.725+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1c7c088880 con 0x7f1c9415d810 2023-11-24T08:21:44.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.725+0000 7f1c92d9d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 0x7f1c7406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:44.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.726+0000 7f1c92d9d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 0x7f1c7406f280 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f1c84036000 tx=0x7f1c84030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:44.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.729+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c7c055390 con 0x7f1c9415d810 2023-11-24T08:21:44.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.917+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1c9404e840 con 0x7f1c9415d810 2023-11-24T08:21:44.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.924+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/1004884122 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1c7c01f0c0 con 0x7f1c9415d810 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.926+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 msgr2=0x7f1c7406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.926+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 0x7f1c7406f280 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f1c84036000 tx=0x7f1c84030000 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.926+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 msgr2=0x7f1c9414f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.926+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c7f0 tx=0x7f1c7c00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 shutdown_connections 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1c7406cdc0 0x7f1c7406f280 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9415de90 0x7f1c9415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9415d810 0x7f1c9414f750 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 --2- 172.21.15.107:0/1004884122 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c94071ea0 0x7f1c941f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 >> 172.21.15.107:0/1004884122 conn(0x7f1c9406a500 msgr2=0x7f1c940cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 shutdown_connections 2023-11-24T08:21:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:44.927+0000 7f1c9893d640 1 -- 172.21.15.107:0/1004884122 wait complete. 2023-11-24T08:21:44.948 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:21:44.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:21:45.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:21:45.178 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:21:45.178 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:45.179 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:45.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:21:45.238 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:21:45.238 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:45.297 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:21:45.297 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:45.297 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:45.358 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:21:45.358 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:21:45.359 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:45.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:21:45.420 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:21:45.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.639+0000 7fced460e640 1 Processor -- start 2023-11-24T08:21:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.639+0000 7fced460e640 1 -- start start 2023-11-24T08:21:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced460e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcecc0cf450 0x7fcecc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced460e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 0x7fcecc0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced460e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc0cece0 con 0x7fcecc15c7f0 2023-11-24T08:21:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced460e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcecc0cfe40 con 0x7fcecc0cf450 2023-11-24T08:21:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced460e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc156010 con 0x7fcecc0cdfb0 2023-11-24T08:21:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced23aa640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcecc15c7f0 0x7fcecc06db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45490/0 (socket says 172.21.15.107:45490) 2023-11-24T08:21:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.640+0000 7fced23aa640 1 -- 172.21.15.107:0/2937851045 learned_addr learned my addr 172.21.15.107:0/2937851045 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260789480 0 0) 0x7fcecc0cece0 con 0x7fcecc15c7f0 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced1ba9640 1 --2- 172.21.15.107:0/2937851045 >> v2:172.21.15.120:3300/0 conn(0x7fcecc0cf450 0x7fcecc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8003520 con 0x7fcecc15c7f0 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced2bab640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 0x7fcecc0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516807701 0 0) 0x7fceb8003520 con 0x7fcecc15c7f0 2023-11-24T08:21:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 msgr2=0x7fcecc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 0x7fcecc0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 >> v2:172.21.15.120:3300/0 conn(0x7fcecc0cf450 msgr2=0x7fcecc0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 --2- 172.21.15.107:0/2937851045 >> v2:172.21.15.120:3300/0 conn(0x7fcecc0cf450 0x7fcecc0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc06d040 con 0x7fcecc15c7f0 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcec8001670 con 0x7fcecc15c7f0 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 >> v1:172.21.15.107:6789/0 conn(0x7fcecc15c7f0 legacy=0x7fcecc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.641+0000 7fced13a8640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb8003970 0x7fceb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced13a8640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 0x7fceb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced13a8640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fceb8004d00 0x7fceb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0cfe40 con 0x7fceb8004d00 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc156010 con 0x7fceb8003970 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb800ba30 con 0x7fcecc0cdfb0 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced1ba9640 1 --2- 172.21.15.107:0/2937851045 >> v2:172.21.15.120:3300/0 conn(0x7fcecc0cf450 0x7fcecc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fceb8004d00 0x7fceb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced23aa640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb8003970 0x7fceb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 -- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 msgr2=0x7fceb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcecc0cdfb0 0x7fceb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 -- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb8003970 msgr2=0x7fceb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb8003970 0x7fceb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.642+0000 7fced2bab640 1 -- 172.21.15.107:0/2937851045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8003520 con 0x7fceb8004d00 2023-11-24T08:21:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.643+0000 7fced2bab640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fceb8004d00 0x7fceb8005180 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fcec4007390 tx=0x7fceb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0d224af45a01c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec402ccf0 con 0x7fceb8004d00 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec402ce50 con 0x7fceb8004d00 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced13a8640 1 -- 172.21.15.107:0/2937851045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4034660 con 0x7fceb8004d00 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 -- 172.21.15.107:0/2937851045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fceb8004d00 msgr2=0x7fceb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 --2- 172.21.15.107:0/2937851045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fceb8004d00 0x7fceb8005180 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fcec4007390 tx=0x7fceb80057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced2bab640 1 -- 172.21.15.107:0/2937851045 reap_dead start 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 -- 172.21.15.107:0/2937851045 shutdown_connections 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 -- 172.21.15.107:0/2937851045 >> 172.21.15.107:0/2937851045 conn(0x7fcecc06a920 msgr2=0x7fcecc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 -- 172.21.15.107:0/2937851045 shutdown_connections 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.644+0000 7fced460e640 1 -- 172.21.15.107:0/2937851045 wait complete. 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.646+0000 7fced460e640 1 Processor -- start 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.646+0000 7fced460e640 1 -- start start 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb8004d00 0x7fcecc151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcecc0abfc0 0x7fcecc0ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc152e60 con 0x7fcecc0ac440 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc153140 con 0x7fcecc0abfc0 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.647+0000 7fced460e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc153420 con 0x7fceb8004d00 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58646/0 (socket says 172.21.15.107:58646) 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 -- 172.21.15.107:0/180732718 learned_addr learned my addr 172.21.15.107:0/180732718 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced23aa640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb8004d00 0x7fcecc151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced1ba9640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcecc0abfc0 0x7fcecc0ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 -- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb8004d00 msgr2=0x7fcecc151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb8004d00 0x7fcecc151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 -- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcecc0abfc0 msgr2=0x7fcecc0ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcecc0abfc0 0x7fcecc0ac990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 -- 172.21.15.107:0/180732718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0b20e0 con 0x7fcecc0ac440 2023-11-24T08:21:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.648+0000 7fced2bab640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fcec4007390 tx=0x7fcec4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.649+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4045070 con 0x7fcecc0ac440 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.649+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec4034e20 con 0x7fcecc0ac440 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.649+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec403d640 con 0x7fcecc0ac440 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.649+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc0b23c0 con 0x7fcecc0ac440 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.649+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc1f68e0 con 0x7fcecc0ac440 2023-11-24T08:21:45.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.650+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc153560 con 0x7fcecc0ac440 2023-11-24T08:21:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.651+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fcec403b040 con 0x7fcecc0ac440 2023-11-24T08:21:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.652+0000 7fcec2ffd640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 0x7fceb006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.652+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcec40a9a60 con 0x7fcecc0ac440 2023-11-24T08:21:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.652+0000 7fced23aa640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 0x7fceb006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.653+0000 7fced23aa640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 0x7fceb006f2b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fcec8001050 tx=0x7fcec8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.657+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcecc153560 con 0x7fcecc0ac440 2023-11-24T08:21:45.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:21:45.877 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:21:45.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.883+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcecc153560 con 0x7fcecc0ac440 2023-11-24T08:21:45.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.884+0000 7fcec2ffd640 1 -- 172.21.15.107:0/180732718 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3715 (secure 0 0 0) 0x7fcecc153560 con 0x7fcecc0ac440 2023-11-24T08:21:45.886 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:45.886 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3194991706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3194991706},{"type":"v1","addr":"172.21.15.120:6835","nonce":3194991706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/486810951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":486810951},{"type":"v1","addr":"172.21.15.120:6834","nonce":486810951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1320425624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1320425624},{"type":"v1","addr":"172.21.15.107:6835","nonce":1320425624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":11547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2046182169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2046182169},{"type":"v1","addr":"172.21.15.107:6837","nonce":2046182169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[]} 2023-11-24T08:21:45.886 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 121 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 msgr2=0x7fceb006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 0x7fceb006f2b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fcec8001050 tx=0x7fcec8009040 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 msgr2=0x7fcecc0aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fcec4007390 tx=0x7fcec4004ab0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 shutdown_connections 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fceb006cdf0 0x7fceb006f2b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.886+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcecc0ac440 0x7fcecc0aced0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.887+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcecc0abfc0 0x7fcecc0ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.887+0000 7fced460e640 1 --2- 172.21.15.107:0/180732718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb8004d00 0x7fcecc151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.887+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 >> 172.21.15.107:0/180732718 conn(0x7fcecc06a920 msgr2=0x7fcecc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.887+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 shutdown_connections 2023-11-24T08:21:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:45.887+0000 7fced460e640 1 -- 172.21.15.107:0/180732718 wait complete. 2023-11-24T08:21:45.906 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:21:45.907 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:21:45.907 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:21:45.907 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:21:45.907 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:21:45.907 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:21:45.908 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:21:45.908 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:21:45.908 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:21:45.908 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:21:45.908 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:21:45.908 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:21:45.908 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:21:45.909 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:21:45.909 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:21:45.909 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:21:45.951 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:21:45.951 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:21:45.952 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:21:45.952 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:21:45.964 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:21:45.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:21:45.966 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:21:45.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:21:45.990 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:21:45.990 DEBUG:teuthology.parallel:result is None 2023-11-24T08:21:45.997 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:21:45.998 DEBUG:teuthology.parallel:result is None 2023-11-24T08:21:46.004 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:21:46.004 DEBUG:teuthology.parallel:result is None 2023-11-24T08:21:46.034 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:21:46.034 DEBUG:teuthology.parallel:result is None 2023-11-24T08:21:46.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:21:46.155 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:21:46.156 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:21:46.157 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:21:46.194 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:21:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.219+0000 7f48b5b97640 1 Processor -- start 2023-11-24T08:21:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.219+0000 7f48b5b97640 1 -- start start 2023-11-24T08:21:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.220+0000 7f48b5b97640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48b00cc0f0 0x7f48b00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.220+0000 7f48b5b97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00c57e0 0x7f48b006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.220+0000 7f48b5b97640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b015eea0 con 0x7f48b015b440 2023-11-24T08:21:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.220+0000 7f48b5b97640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b015b820 con 0x7f48b00cc0f0 2023-11-24T08:21:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.220+0000 7f48b5b97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00abf30 con 0x7f48b00c57e0 2023-11-24T08:21:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48af7fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f48b015b440 0x7f48b00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:48796/0 (socket says 172.21.15.107:48796) 2023-11-24T08:21:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48af7fe640 1 -- 172.21.15.107:0/4064737667 learned_addr learned my addr 172.21.15.107:0/4064737667 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48affff640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00c57e0 0x7f48b006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48aeffd640 1 --2- 172.21.15.107:0/4064737667 >> v2:172.21.15.120:3300/0 conn(0x7f48b00cc0f0 0x7f48b00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48aeffd640 1 -- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00c57e0 msgr2=0x7f48b006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48aeffd640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00c57e0 0x7f48b006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48ae7fc640 1 -- 172.21.15.107:0/4064737667 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692350333 0 0) 0x7f48b015eea0 con 0x7f48b015b440 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48aeffd640 1 -- 172.21.15.107:0/4064737667 >> v1:172.21.15.120:6789/0 conn(0x7f48b015b440 legacy=0x7f48b00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.221+0000 7f48aeffd640 1 -- 172.21.15.107:0/4064737667 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b015f0f0 con 0x7f48b00cc0f0 2023-11-24T08:21:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.222+0000 7f48aeffd640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00cc0f0 0x7f48b00c52a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48980097a0 tx=0x7f489802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84c73476516fdfd9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.223+0000 7f48ae7fc640 1 -- 172.21.15.107:0/4064737667 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898031780 con 0x7f48b00cc0f0 2023-11-24T08:21:46.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.223+0000 7f48ae7fc640 1 -- 172.21.15.107:0/4064737667 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48980318e0 con 0x7f48b00cc0f0 2023-11-24T08:21:46.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.223+0000 7f48ae7fc640 1 -- 172.21.15.107:0/4064737667 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898031c70 con 0x7f48b00cc0f0 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.223+0000 7f48b5b97640 1 -- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00cc0f0 msgr2=0x7f48b00c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.223+0000 7f48b5b97640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00cc0f0 0x7f48b00c52a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48980097a0 tx=0x7f489802b820 comp rx=0 tx=0).stop 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 -- 172.21.15.107:0/4064737667 shutdown_connections 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00c57e0 0x7f48b006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 --2- 172.21.15.107:0/4064737667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00cc0f0 0x7f48b00c52a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 -- 172.21.15.107:0/4064737667 >> 172.21.15.107:0/4064737667 conn(0x7f48b00b0c50 msgr2=0x7f48b00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 -- 172.21.15.107:0/4064737667 shutdown_connections 2023-11-24T08:21:46.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.224+0000 7f48b5b97640 1 -- 172.21.15.107:0/4064737667 wait complete. 2023-11-24T08:21:46.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.225+0000 7f48b5b97640 1 Processor -- start 2023-11-24T08:21:46.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 -- start start 2023-11-24T08:21:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00c57e0 0x7f48b0159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00cc0f0 0x7f48b01f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b01f3e30 con 0x7f48b015b440 2023-11-24T08:21:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b01f3b80 con 0x7f48b00c57e0 2023-11-24T08:21:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.226+0000 7f48b5b97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b3040 con 0x7f48b00cc0f0 2023-11-24T08:21:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58686/0 (socket says 172.21.15.107:58686) 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 -- 172.21.15.107:0/3054017079 learned_addr learned my addr 172.21.15.107:0/3054017079 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48aeffd640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00cc0f0 0x7f48b01f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48af7fe640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00c57e0 0x7f48b0159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 -- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00cc0f0 msgr2=0x7f48b01f3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00cc0f0 0x7f48b01f3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 -- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00c57e0 msgr2=0x7f48b0159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00c57e0 0x7f48b0159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.227+0000 7f48affff640 1 -- 172.21.15.107:0/3054017079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4898009400 con 0x7f48b015b440 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48affff640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e930 tx=0x7f48a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0009690 con 0x7f48b015b440 2023-11-24T08:21:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a000c980 con 0x7f48b015b440 2023-11-24T08:21:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a000cd30 con 0x7f48b015b440 2023-11-24T08:21:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00b3320 con 0x7f48b015b440 2023-11-24T08:21:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.228+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00b38e0 con 0x7f48b015b440 2023-11-24T08:21:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.229+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b004e8b0 con 0x7f48b015b440 2023-11-24T08:21:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.230+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f48a0013470 con 0x7f48b015b440 2023-11-24T08:21:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.232+0000 7f48a7fff640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 0x7f489006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.232+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f48a00888d0 con 0x7f48b015b440 2023-11-24T08:21:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.232+0000 7f48af7fe640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 0x7f489006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.232+0000 7f48af7fe640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 0x7f489006f2b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f48a8007ed0 tx=0x7f48a8007e60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.236+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48a00554a0 con 0x7f48b015b440 2023-11-24T08:21:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.439+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f48b006dc80 con 0x7f48b015b440 2023-11-24T08:21:46.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.491+0000 7f48a7fff640 1 -- 172.21.15.107:0/3054017079 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v201) v1 ==== 97+0+0 (secure 0 0 0) 0x7f48a0021080 con 0x7f48b015b440 2023-11-24T08:21:46.494 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:21:46.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 msgr2=0x7f489006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 0x7f489006f2b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f48a8007ed0 tx=0x7f48a8007e60 comp rx=0 tx=0).stop 2023-11-24T08:21:46.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 msgr2=0x7f48b01f3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e930 tx=0x7f48a000edf0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 shutdown_connections 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f489006cdf0 0x7f489006f2b0 secure :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f48a8007ed0 tx=0x7f48a8007e60 comp rx=0 tx=0).stop 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b015b440 0x7f48b01f3640 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48b00cc0f0 0x7f48b01f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 --2- 172.21.15.107:0/3054017079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48b00c57e0 0x7f48b0159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 >> 172.21.15.107:0/3054017079 conn(0x7f48b00b0c50 msgr2=0x7f48b0070550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.495+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 shutdown_connections 2023-11-24T08:21:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.496+0000 7f48b5b97640 1 -- 172.21.15.107:0/3054017079 wait complete. 2023-11-24T08:21:46.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:21:46.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.732+0000 7f882db0c640 1 Processor -- start 2023-11-24T08:21:46.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.733+0000 7f882db0c640 1 -- start start 2023-11-24T08:21:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.734+0000 7f882db0c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 0x7f88280ca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.734+0000 7f882db0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280cf070 con 0x7f88280af7d0 2023-11-24T08:21:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.734+0000 7f882db0c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882806f120 con 0x7f882806ec40 2023-11-24T08:21:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.734+0000 7f882db0c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882806f400 con 0x7f8828071640 2023-11-24T08:21:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.735+0000 7f8826ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f88280af7d0 0x7f88280cf500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45518/0 (socket says 172.21.15.107:45518) 2023-11-24T08:21:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.735+0000 7f8826ffd640 1 -- 172.21.15.107:0/3440086993 learned_addr learned my addr 172.21.15.107:0/3440086993 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.735+0000 7f88277fe640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 0x7f88280ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.735+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088966393 0 0) 0x7f88280cf070 con 0x7f88280af7d0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.735+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8814003520 con 0x7f88280af7d0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729307192 0 0) 0x7f8814003520 con 0x7f88280af7d0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 msgr2=0x7f88280ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 0x7f88280ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 >> v1:172.21.15.120:6789/0 conn(0x7f882806ec40 legacy=0x7f8828071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280cea20 con 0x7f88280af7d0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8810001670 con 0x7f88280af7d0 2023-11-24T08:21:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 >> v1:172.21.15.107:6789/0 conn(0x7f88280af7d0 legacy=0x7f88280cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 0x7f88140039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8814004d00 0x7f8814005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882806f400 con 0x7f8814003f20 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881400ba60 con 0x7f8814004d00 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881400bc20 con 0x7f8828071640 2023-11-24T08:21:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f8826ffd640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.736+0000 7f8826ffd640 1 -- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 msgr2=0x7f88140039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8826ffd640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8828071640 0x7f88140039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8826ffd640 1 -- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8814004d00 msgr2=0x7f8814005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8827fff640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8814004d00 0x7f8814005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8826ffd640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8814004d00 0x7f8814005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8826ffd640 1 -- 172.21.15.107:0/3440086993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814003520 con 0x7f8814003f20 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.737+0000 7f8827fff640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8814004d00 0x7f8814005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.738+0000 7f8826ffd640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88140047c0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f881000c240 tx=0x7f88140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8f8c72f6614771 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.738+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810005ad0 con 0x7f8814003f20 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.738+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8810005c30 con 0x7f8814003f20 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.738+0000 7f88267fc640 1 -- 172.21.15.107:0/3440086993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881002caf0 con 0x7f8814003f20 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 -- 172.21.15.107:0/3440086993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 msgr2=0x7f88140047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 --2- 172.21.15.107:0/3440086993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88140047c0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f881000c240 tx=0x7f88140057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f8827fff640 1 -- 172.21.15.107:0/3440086993 reap_dead start 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 -- 172.21.15.107:0/3440086993 shutdown_connections 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 -- 172.21.15.107:0/3440086993 >> 172.21.15.107:0/3440086993 conn(0x7f88280d0110 msgr2=0x7f88280d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 -- 172.21.15.107:0/3440086993 shutdown_connections 2023-11-24T08:21:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.739+0000 7f882db0c640 1 -- 172.21.15.107:0/3440086993 wait complete. 2023-11-24T08:21:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.741+0000 7f882db0c640 1 Processor -- start 2023-11-24T08:21:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.741+0000 7f882db0c640 1 -- start start 2023-11-24T08:21:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88280b2d30 0x7f88280b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88280ae230 0x7f88280ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280aebf0 con 0x7f8814003f20 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b37b0 con 0x7f88280b2d30 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f882db0c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b3a90 con 0x7f88280ae230 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58704/0 (socket says 172.21.15.107:58704) 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 -- 172.21.15.107:0/887016987 learned_addr learned my addr 172.21.15.107:0/887016987 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 -- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88280ae230 msgr2=0x7f88280ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88280ae230 0x7f88280ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f8826ffd640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88280b2d30 0x7f88280b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 -- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88280b2d30 msgr2=0x7f88280b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88280b2d30 0x7f88280b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.742+0000 7f88277fe640 1 -- 172.21.15.107:0/887016987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280b3d70 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f88277fe640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f881800ef10 tx=0x7f881800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818010040 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8818014400 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818014790 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88280b4050 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.743+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8828150870 con 0x7f8814003f20 2023-11-24T08:21:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.744+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88280d10d0 con 0x7f8814003f20 2023-11-24T08:21:46.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.745+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f88180148f0 con 0x7f8814003f20 2023-11-24T08:21:46.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.746+0000 7f881ffff640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 0x7f880806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:46.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.746+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8818089100 con 0x7f8814003f20 2023-11-24T08:21:46.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.747+0000 7f8826ffd640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 0x7f880806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:46.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.747+0000 7f8826ffd640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 0x7f880806f280 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f8810034000 tx=0x7f8810009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:46.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:46.751+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88180c0e90 con 0x7f8814003f20 2023-11-24T08:21:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.008+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f882815ebb0 con 0x7f8814003f20 2023-11-24T08:21:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.009+0000 7f881ffff640 1 -- 172.21.15.107:0/887016987 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8818055c10 con 0x7f8814003f20 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.013 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.014 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.016 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.017 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:21:47.019 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:21:47.020 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.021 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:21:47.022 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:21:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:21:47.025 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.011+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 msgr2=0x7f880806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.011+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 0x7f880806f280 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f8810034000 tx=0x7f8810009030 comp rx=0 tx=0).stop 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.011+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 msgr2=0x7f88280c5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.011+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f881800ef10 tx=0x7f881800b480 comp rx=0 tx=0).stop 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 shutdown_connections 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f880806cdc0 0x7f880806f280 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88280ae230 0x7f88280ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88280b2d30 0x7f88280b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 --2- 172.21.15.107:0/887016987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8814003f20 0x7f88280c5fe0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 >> 172.21.15.107:0/887016987 conn(0x7f88280d0110 msgr2=0x7f8828071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 shutdown_connections 2023-11-24T08:21:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.012+0000 7f882db0c640 1 -- 172.21.15.107:0/887016987 wait complete. 2023-11-24T08:21:47.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.237+0000 7fd8529d5640 1 Processor -- start 2023-11-24T08:21:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.237+0000 7fd8529d5640 1 -- start start 2023-11-24T08:21:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.238+0000 7fd8529d5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd84c0cf450 0x7fd84c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.238+0000 7fd8529d5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c0cece0 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.238+0000 7fd8529d5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c06d040 con 0x7fd84c15c7f0 2023-11-24T08:21:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.238+0000 7fd8529d5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c0cfe40 con 0x7fd84c0cf450 2023-11-24T08:21:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.239+0000 7fd850f72640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd84c0cdfb0 0x7fd84c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45520/0 (socket says 172.21.15.107:45520) 2023-11-24T08:21:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.239+0000 7fd850f72640 1 -- 172.21.15.107:0/2048095286 learned_addr learned my addr 172.21.15.107:0/2048095286 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.239+0000 7fd843fff640 1 --2- 172.21.15.107:0/2048095286 >> v2:172.21.15.120:3300/0 conn(0x7fd84c0cf450 0x7fd84c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.239+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198665402 0 0) 0x7fd84c0cece0 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.239+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838003520 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42308131 0 0) 0x7fd838003520 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 >> v2:172.21.15.120:3300/0 conn(0x7fd84c0cf450 msgr2=0x7fd84c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/2048095286 >> v2:172.21.15.120:3300/0 conn(0x7fd84c0cf450 0x7fd84c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 >> v1:172.21.15.120:6789/0 conn(0x7fd84c15c7f0 legacy=0x7fd84c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c06d320 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd8440019f0 con 0x7fd84c0cdfb0 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 >> v1:172.21.15.107:6789/0 conn(0x7fd84c0cdfb0 legacy=0x7fd84c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 0x7fd838003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8380047c0 0x7fd838004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd838005180 0x7fd83800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0cfe40 con 0x7fd838003970 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83800c020 con 0x7fd8380047c0 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83800c1e0 con 0x7fd838005180 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.240+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 0x7fd838003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd843fff640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8380047c0 0x7fd838004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd84bfff640 1 -- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd838005180 msgr2=0x7fd83800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd850f72640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd838005180 0x7fd83800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd838005180 0x7fd83800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd84bfff640 1 -- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8380047c0 msgr2=0x7fd838004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8380047c0 0x7fd838004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd84bfff640 1 -- 172.21.15.107:0/2048095286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838003520 con 0x7fd838003970 2023-11-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.241+0000 7fd850f72640 1 -- 172.21.15.107:0/2048095286 reap_dead start 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.242+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 0x7fd838003d90 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fd83c001750 tx=0x7fd8380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338e82f65436b6f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.242+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0052d0 con 0x7fd838003970 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.242+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c02e050 con 0x7fd838003970 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.242+0000 7fd84b7fe640 1 -- 172.21.15.107:0/2048095286 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c003830 con 0x7fd838003970 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 -- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 msgr2=0x7fd838003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 0x7fd838003d90 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fd83c001750 tx=0x7fd8380057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 -- 172.21.15.107:0/2048095286 shutdown_connections 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2048095286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd838003970 0x7fd838003d90 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 -- 172.21.15.107:0/2048095286 >> 172.21.15.107:0/2048095286 conn(0x7fd84c06a920 msgr2=0x7fd84c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 -- 172.21.15.107:0/2048095286 shutdown_connections 2023-11-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.243+0000 7fd8529d5640 1 -- 172.21.15.107:0/2048095286 wait complete. 2023-11-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.245+0000 7fd8529d5640 1 Processor -- start 2023-11-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.245+0000 7fd8529d5640 1 -- start start 2023-11-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd838003970 0x7fd84c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c0b36d0 0x7fd84c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c1538d0 con 0x7fd84c144260 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0b4090 con 0x7fd838003970 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd8529d5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c1f3410 con 0x7fd84c0b36d0 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58728/0 (socket says 172.21.15.107:58728) 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd843fff640 1 -- 172.21.15.107:0/2754154654 learned_addr learned my addr 172.21.15.107:0/2754154654 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.246+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd838003970 0x7fd84c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd850f72640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c0b36d0 0x7fd84c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 -- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c0b36d0 msgr2=0x7fd84c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c0b36d0 0x7fd84c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 -- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd838003970 msgr2=0x7fd84c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd838003970 0x7fd84c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 -- 172.21.15.107:0/2754154654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c153440 con 0x7fd84c144260 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd843fff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7fd834011510 tx=0x7fd8340119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834015870 con 0x7fd84c144260 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8340159d0 con 0x7fd84c144260 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c152830 con 0x7fd84c144260 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834015d40 con 0x7fd84c144260 2023-11-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.247+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c1530d0 con 0x7fd84c144260 2023-11-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.248+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c1535d0 con 0x7fd84c144260 2023-11-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.250+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fd834013400 con 0x7fd84c144260 2023-11-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.250+0000 7fd8497fa640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 0x7fd82c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.251+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd834088a50 con 0x7fd84c144260 2023-11-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.251+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 0x7fd82c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.252+0000 7fd84bfff640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 0x7fd82c06f280 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd83c033000 tx=0x7fd83c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.255+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd834055560 con 0x7fd84c144260 2023-11-24T08:21:47.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.447+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd84c067470 con 0x7fd84c144260 2023-11-24T08:21:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.449+0000 7fd8497fa640 1 -- 172.21.15.107:0/2754154654 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fd834022020 con 0x7fd84c144260 2023-11-24T08:21:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:47.451 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":203,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:46.507908+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 msgr2=0x7fd82c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 0x7fd82c06f280 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd83c033000 tx=0x7fd83c009030 comp rx=0 tx=0).stop 2023-11-24T08:21:47.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 msgr2=0x7fd84c151f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7fd834011510 tx=0x7fd8340119d0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 shutdown_connections 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd82c06cdc0 0x7fd82c06f280 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c0b36d0 0x7fd84c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c144260 0x7fd84c151f30 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 --2- 172.21.15.107:0/2754154654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd838003970 0x7fd84c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 >> 172.21.15.107:0/2754154654 conn(0x7fd84c06a920 msgr2=0x7fd84c071490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 shutdown_connections 2023-11-24T08:21:47.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.452+0000 7fd8529d5640 1 -- 172.21.15.107:0/2754154654 wait complete. 2023-11-24T08:21:47.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:21:47.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:21:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.679+0000 7ff62e478640 1 Processor -- start 2023-11-24T08:21:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.679+0000 7ff62e478640 1 -- start start 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62e478640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6280b0fb0 0x7ff62806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62e478640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62815ed20 con 0x7ff62815b260 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62e478640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62815f8a0 con 0x7ff6280b0b40 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62e478640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62815b640 con 0x7ff6280b0fb0 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62dc77640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6280b0fb0 0x7ff62806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62dc77640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6280b0fb0 0x7ff62806ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58744/0 (socket says 172.21.15.107:58744) 2023-11-24T08:21:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62dc77640 1 -- 172.21.15.107:0/1528835872 learned_addr learned my addr 172.21.15.107:0/1528835872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:47.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.681+0000 7ff62dc77640 1 -- 172.21.15.107:0/1528835872 >> v1:172.21.15.120:6789/0 conn(0x7ff6280b0b40 legacy=0x7ff62806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:47.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.682+0000 7ff62dc77640 1 -- 172.21.15.107:0/1528835872 >> v1:172.21.15.107:6789/0 conn(0x7ff62815b260 legacy=0x7ff62806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.682+0000 7ff62dc77640 1 -- 172.21.15.107:0/1528835872 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62806e750 con 0x7ff6280b0fb0 2023-11-24T08:21:47.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.682+0000 7ff61ffff640 1 -- 172.21.15.107:0/1528835872 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332042094 0 0) 0x7ff62815ed20 con 0x7ff62815b260 2023-11-24T08:21:47.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff62dc77640 1 --2- 172.21.15.107:0/1528835872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6280b0fb0 0x7ff62806ccc0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7ff620009e50 tx=0x7ff62002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cf23781b096b3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff61ffff640 1 -- 172.21.15.107:0/1528835872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62002c9f0 con 0x7ff6280b0fb0 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff61ffff640 1 -- 172.21.15.107:0/1528835872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62002cb50 con 0x7ff6280b0fb0 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff61ffff640 1 -- 172.21.15.107:0/1528835872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200343f0 con 0x7ff6280b0fb0 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff62e478640 1 -- 172.21.15.107:0/1528835872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6280b0fb0 msgr2=0x7ff62806ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.683+0000 7ff62e478640 1 --2- 172.21.15.107:0/1528835872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6280b0fb0 0x7ff62806ccc0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7ff620009e50 tx=0x7ff62002be50 comp rx=0 tx=0).stop 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.684+0000 7ff62e478640 1 -- 172.21.15.107:0/1528835872 shutdown_connections 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.684+0000 7ff62e478640 1 --2- 172.21.15.107:0/1528835872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6280b0fb0 0x7ff62806ccc0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.684+0000 7ff62e478640 1 -- 172.21.15.107:0/1528835872 >> 172.21.15.107:0/1528835872 conn(0x7ff6280cbb40 msgr2=0x7ff6280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.684+0000 7ff62e478640 1 -- 172.21.15.107:0/1528835872 shutdown_connections 2023-11-24T08:21:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.684+0000 7ff62e478640 1 -- 172.21.15.107:0/1528835872 wait complete. 2023-11-24T08:21:47.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.685+0000 7ff62e478640 1 Processor -- start 2023-11-24T08:21:47.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 -- start start 2023-11-24T08:21:47.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6280b0b40 0x7ff628158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6280b0fb0 0x7ff6281f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff628158bc0 con 0x7ff62815b260 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281f52b0 con 0x7ff6280b0fb0 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.686+0000 7ff62e478640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280b3870 con 0x7ff6280b0b40 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58754/0 (socket says 172.21.15.107:58754) 2023-11-24T08:21:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 -- 172.21.15.107:0/2292245671 learned_addr learned my addr 172.21.15.107:0/2292245671 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:47.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 -- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6280b0b40 msgr2=0x7ff628158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6280b0b40 0x7ff628158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 -- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6280b0fb0 msgr2=0x7ff6281f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6280b0fb0 0x7ff6281f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 -- 172.21.15.107:0/2292245671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620009b30 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.687+0000 7ff62dc77640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7ff620036040 tx=0x7ff620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.688+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62003e450 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.688+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62003e5d0 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.688+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62003e980 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.688+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6280c2b90 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.688+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6280b3df0 con 0x7ff62815b260 2023-11-24T08:21:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.689+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62804e840 con 0x7ff62815b260 2023-11-24T08:21:47.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.690+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ff62003eae0 con 0x7ff62815b260 2023-11-24T08:21:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.691+0000 7ff61dffb640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 0x7ff60c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.691+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff6200ab600 con 0x7ff62815b260 2023-11-24T08:21:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.691+0000 7ff62d476640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 0x7ff60c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.692+0000 7ff62d476640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 0x7ff60c06f280 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff624001560 tx=0x7ff624007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:47.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.695+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff620078110 con 0x7ff62815b260 2023-11-24T08:21:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.885+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6280ce640 con 0x7ff62815b260 2023-11-24T08:21:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.886+0000 7ff61dffb640 1 -- 172.21.15.107:0/2292245671 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v203) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff620043020 con 0x7ff62815b260 2023-11-24T08:21:47.888 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.888+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 msgr2=0x7ff60c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.888+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 0x7ff60c06f280 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff624001560 tx=0x7ff624007930 comp rx=0 tx=0).stop 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.888+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 msgr2=0x7ff6280ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.888+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7ff620036040 tx=0x7ff620004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 shutdown_connections 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff60c06cdc0 0x7ff60c06f280 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62815b260 0x7ff6280ccc50 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6280b0fb0 0x7ff6281f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 --2- 172.21.15.107:0/2292245671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6280b0b40 0x7ff628158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 >> 172.21.15.107:0/2292245671 conn(0x7ff6280cbb40 msgr2=0x7ff628159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 shutdown_connections 2023-11-24T08:21:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:47.889+0000 7ff62e478640 1 -- 172.21.15.107:0/2292245671 wait complete. 2023-11-24T08:21:47.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:21:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.119+0000 7fe4ae86d640 1 Processor -- start 2023-11-24T08:21:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.119+0000 7fe4ae86d640 1 -- start start 2023-11-24T08:21:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.120+0000 7fe4ae86d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4a813d120 0x7fe4a815af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.120+0000 7fe4ae86d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.120+0000 7fe4ae86d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80cec00 con 0x7fe4a806d6c0 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.120+0000 7fe4ae86d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a813f9e0 con 0x7fe4a813d120 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.120+0000 7fe4ae86d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a813fcc0 con 0x7fe4a806db20 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad86b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4a813d120 0x7fe4a815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad06a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe4a806d6c0 0x7fe4a80b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51742/0 (socket says 172.21.15.107:51742) 2023-11-24T08:21:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad06a640 1 -- 172.21.15.107:0/2247123039 learned_addr learned my addr 172.21.15.107:0/2247123039 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ae06c640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad86b640 1 -- 172.21.15.107:0/2247123039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 msgr2=0x7fe4a80b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad86b640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad86b640 1 -- 172.21.15.107:0/2247123039 >> v1:172.21.15.107:6789/0 conn(0x7fe4a806d6c0 legacy=0x7fe4a80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ad86b640 1 -- 172.21.15.107:0/2247123039 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a813ffa0 con 0x7fe4a813d120 2023-11-24T08:21:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.121+0000 7fe4ac869640 1 -- 172.21.15.107:0/2247123039 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652449713 0 0) 0x7fe4a80cec00 con 0x7fe4a806d6c0 2023-11-24T08:21:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ad86b640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a813d120 0x7fe4a815af40 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40097a0 tx=0x7fe4a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e85c7da47f302cf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ac869640 1 -- 172.21.15.107:0/2247123039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a402e070 con 0x7fe4a813d120 2023-11-24T08:21:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ac869640 1 -- 172.21.15.107:0/2247123039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a40315a0 con 0x7fe4a813d120 2023-11-24T08:21:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ac869640 1 -- 172.21.15.107:0/2247123039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4031930 con 0x7fe4a813d120 2023-11-24T08:21:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ae86d640 1 -- 172.21.15.107:0/2247123039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a813d120 msgr2=0x7fe4a815af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.123+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a813d120 0x7fe4a815af40 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40097a0 tx=0x7fe4a402b840 comp rx=0 tx=0).stop 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 -- 172.21.15.107:0/2247123039 shutdown_connections 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/2247123039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a813d120 0x7fe4a815af40 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 -- 172.21.15.107:0/2247123039 >> 172.21.15.107:0/2247123039 conn(0x7fe4a80cb830 msgr2=0x7fe4a80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 -- 172.21.15.107:0/2247123039 shutdown_connections 2023-11-24T08:21:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.124+0000 7fe4ae86d640 1 -- 172.21.15.107:0/2247123039 wait complete. 2023-11-24T08:21:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.125+0000 7fe4ae86d640 1 Processor -- start 2023-11-24T08:21:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 -- start start 2023-11-24T08:21:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a813d120 0x7fe4a80b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cda80 con 0x7fe4a806d6c0 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.126+0000 7fe4ae86d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80b17b0 con 0x7fe4a813d120 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ae86d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80ad700 con 0x7fe4a806db20 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49306/0 (socket says 172.21.15.107:49306) 2023-11-24T08:21:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 -- 172.21.15.107:0/728907486 learned_addr learned my addr 172.21.15.107:0/728907486 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 -- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 msgr2=0x7fe4a80b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 -- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a813d120 msgr2=0x7fe4a80b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a813d120 0x7fe4a80b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.127+0000 7fe4ad86b640 1 -- 172.21.15.107:0/728907486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4009400 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe4ad86b640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098c0 tx=0x7fe4a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4032840 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a40329a0 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a80cdcd0 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4032d50 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.128+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a80ce290 con 0x7fe4a806d6c0 2023-11-24T08:21:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.129+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80ce480 con 0x7fe4a806d6c0 2023-11-24T08:21:48.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.130+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fe4a404e430 con 0x7fe4a806d6c0 2023-11-24T08:21:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.131+0000 7fe49e7fc640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 0x7fe48c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.131+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe4a40aa6b0 con 0x7fe4a806d6c0 2023-11-24T08:21:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.131+0000 7fe4ad06a640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 0x7fe48c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.132+0000 7fe4ad06a640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 0x7fe48c06f280 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fe498007fd0 tx=0x7fe498007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.136+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4a4077260 con 0x7fe4a806d6c0 2023-11-24T08:21:48.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.330+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe4a815b400 con 0x7fe4a806d6c0 2023-11-24T08:21:48.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.331+0000 7fe49e7fc640 1 -- 172.21.15.107:0/728907486 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v203) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe4a407a880 con 0x7fe4a806d6c0 2023-11-24T08:21:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:48.333 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:21:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:21:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:21:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:21:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:21:48.334 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 msgr2=0x7fe48c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 0x7fe48c06f280 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fe498007fd0 tx=0x7fe498007960 comp rx=0 tx=0).stop 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 msgr2=0x7fe4a80b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098c0 tx=0x7fe4a40046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 shutdown_connections 2023-11-24T08:21:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe48c06cdc0 0x7fe48c06f280 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a813d120 0x7fe4a80b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4a806db20 0x7fe4a80b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 --2- 172.21.15.107:0/728907486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4a806d6c0 0x7fe4a80b3ce0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 >> 172.21.15.107:0/728907486 conn(0x7fe4a80cb830 msgr2=0x7fe4a80cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.334+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 shutdown_connections 2023-11-24T08:21:48.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.335+0000 7fe4ae86d640 1 -- 172.21.15.107:0/728907486 wait complete. 2023-11-24T08:21:48.356 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:21:48.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:21:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.572+0000 7f317ffff640 1 Processor -- start 2023-11-24T08:21:48.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.573+0000 7f317ffff640 1 -- start start 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f317ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f318006cc00 0x7f318013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f317ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e420 0x7f318013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f317ffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318013fa60 con 0x7f318013e040 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f317ffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318013fd40 con 0x7f318006cc00 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f317ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180140020 con 0x7f318013e420 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f31767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f318006cc00 0x7f318013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f31767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f318006cc00 0x7f318013ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49314/0 (socket says 172.21.15.107:49314) 2023-11-24T08:21:48.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f31767fc640 1 -- 172.21.15.107:0/3540568748 learned_addr learned my addr 172.21.15.107:0/3540568748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:48.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f31767fc640 1 -- 172.21.15.107:0/3540568748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e420 msgr2=0x7f318013f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:48.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.575+0000 7f31767fc640 1 --2- 172.21.15.107:0/3540568748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e420 0x7f318013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.576+0000 7f31767fc640 1 -- 172.21.15.107:0/3540568748 >> v1:172.21.15.120:6789/0 conn(0x7f318013e040 legacy=0x7f318006c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:48.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.576+0000 7f31767fc640 1 -- 172.21.15.107:0/3540568748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3180140300 con 0x7f318006cc00 2023-11-24T08:21:48.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.577+0000 7f31767fc640 1 --2- 172.21.15.107:0/3540568748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318006cc00 0x7f318013ecf0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3168009870 tx=0x7f316802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340b114bb50d9ca7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.577+0000 7f317e7fc640 1 -- 172.21.15.107:0/3540568748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168031640 con 0x7f318006cc00 2023-11-24T08:21:48.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.577+0000 7f317e7fc640 1 -- 172.21.15.107:0/3540568748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31680317a0 con 0x7f318006cc00 2023-11-24T08:21:48.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.577+0000 7f317e7fc640 1 -- 172.21.15.107:0/3540568748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168031b30 con 0x7f318006cc00 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 -- 172.21.15.107:0/3540568748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318006cc00 msgr2=0x7f318013ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 --2- 172.21.15.107:0/3540568748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318006cc00 0x7f318013ecf0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3168009870 tx=0x7f316802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 -- 172.21.15.107:0/3540568748 shutdown_connections 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 --2- 172.21.15.107:0/3540568748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e420 0x7f318013f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 --2- 172.21.15.107:0/3540568748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318006cc00 0x7f318013ecf0 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 -- 172.21.15.107:0/3540568748 >> 172.21.15.107:0/3540568748 conn(0x7f318006dc30 msgr2=0x7f318006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:48.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 -- 172.21.15.107:0/3540568748 shutdown_connections 2023-11-24T08:21:48.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.578+0000 7f317ffff640 1 -- 172.21.15.107:0/3540568748 wait complete. 2023-11-24T08:21:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.580+0000 7f317ffff640 1 Processor -- start 2023-11-24T08:21:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.580+0000 7f317ffff640 1 -- start start 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.581+0000 7f317ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f318006cc00 0x7f3180135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.581+0000 7f317ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e040 0x7f31800ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.581+0000 7f317ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800ae860 con 0x7f318013e420 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317ffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800adfc0 con 0x7f318006cc00 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31800ae2a0 con 0x7f318013e040 2023-11-24T08:21:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49330/0 (socket says 172.21.15.107:49330) 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 -- 172.21.15.107:0/685883598 learned_addr learned my addr 172.21.15.107:0/685883598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f31767fc640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e040 0x7f31800ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317effd640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f318006cc00 0x7f3180135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 -- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e040 msgr2=0x7f31800ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e040 0x7f31800ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 -- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f318006cc00 msgr2=0x7f3180135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f318006cc00 0x7f3180135930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.582+0000 7f317f7fe640 1 -- 172.21.15.107:0/685883598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31680094d0 con 0x7f318013e420 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f317f7fe640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3178009fd0 tx=0x7f317800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178011830 con 0x7f318013e420 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178011990 con 0x7f318013e420 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178011d20 con 0x7f318013e420 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31800ae580 con 0x7f318013e420 2023-11-24T08:21:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.583+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31800ad290 con 0x7f318013e420 2023-11-24T08:21:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.584+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31800a88a0 con 0x7f318013e420 2023-11-24T08:21:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.585+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f317801b460 con 0x7f318013e420 2023-11-24T08:21:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.586+0000 7f3177fff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 0x7f316006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.586+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3178088500 con 0x7f318013e420 2023-11-24T08:21:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.586+0000 7f317effd640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 0x7f316006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:48.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.587+0000 7f317effd640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 0x7f316006f280 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3170007fd0 tx=0x7f3170009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:48.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.590+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31800a88a0 con 0x7f318013e420 2023-11-24T08:21:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:48.777+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f31800a88a0 con 0x7f318013e420 2023-11-24T08:21:49.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.529+0000 7f3177fff640 1 -- 172.21.15.107:0/685883598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v204) v1 ==== 138+0+0 (secure 0 0 0) 0x7f31800a88a0 con 0x7f318013e420 2023-11-24T08:21:49.531 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.533+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 msgr2=0x7f316006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.533+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 0x7f316006f280 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3170007fd0 tx=0x7f3170009040 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.533+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 msgr2=0x7f31800ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.533+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3178009fd0 tx=0x7f317800b8e0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 shutdown_connections 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006cdc0 0x7f316006f280 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318013e420 0x7f31800ce840 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f318013e040 0x7f31800ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 --2- 172.21.15.107:0/685883598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f318006cc00 0x7f3180135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 >> 172.21.15.107:0/685883598 conn(0x7f318006dc30 msgr2=0x7f31800af380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:49.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 shutdown_connections 2023-11-24T08:21:49.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.534+0000 7f317ffff640 1 -- 172.21.15.107:0/685883598 wait complete. 2023-11-24T08:21:49.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:21:49.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.772+0000 7f6656cf1640 1 Processor -- start 2023-11-24T08:21:49.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.772+0000 7f6656cf1640 1 -- start start 2023-11-24T08:21:49.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.773+0000 7f6656cf1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665015d490 0x7f66500cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.773+0000 7f6656cf1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500b63d0 0x7f66500d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.773+0000 7f6656cf1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500cea30 con 0x7f66500ce270 2023-11-24T08:21:49.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.773+0000 7f6656cf1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500d0450 con 0x7f665015d490 2023-11-24T08:21:49.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.773+0000 7f6656cf1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500d0bf0 con 0x7f66500b63d0 2023-11-24T08:21:49.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.774+0000 7f6655cef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665015d490 0x7f66500cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.774+0000 7f66564f0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f66500ce270 0x7f6650160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:52936/0 (socket says 172.21.15.107:52936) 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.774+0000 7f66564f0640 1 -- 172.21.15.107:0/3906117677 learned_addr learned my addr 172.21.15.107:0/3906117677 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.774+0000 7f6655cef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665015d490 0x7f66500cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49332/0 (socket says 172.21.15.107:49332) 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.774+0000 7f66554ee640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500b63d0 0x7f66500d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.775+0000 7f6655cef640 1 -- 172.21.15.107:0/3906117677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500b63d0 msgr2=0x7f66500d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.775+0000 7f6655cef640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500b63d0 0x7f66500d3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.775+0000 7f6655cef640 1 -- 172.21.15.107:0/3906117677 >> v1:172.21.15.120:6789/0 conn(0x7f66500ce270 legacy=0x7f6650160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.775+0000 7f6655cef640 1 -- 172.21.15.107:0/3906117677 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500d2260 con 0x7f665015d490 2023-11-24T08:21:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.776+0000 7f6655cef640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665015d490 0x7f66500cd8f0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f664002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf10da457a940d7a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6654ced640 1 -- 172.21.15.107:0/3906117677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664002cc50 con 0x7f665015d490 2023-11-24T08:21:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6654ced640 1 -- 172.21.15.107:0/3906117677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664002cdb0 con 0x7f665015d490 2023-11-24T08:21:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6654ced640 1 -- 172.21.15.107:0/3906117677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640034660 con 0x7f665015d490 2023-11-24T08:21:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 -- 172.21.15.107:0/3906117677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665015d490 msgr2=0x7f66500cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665015d490 0x7f66500cd8f0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f664002bea0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 -- 172.21.15.107:0/3906117677 shutdown_connections 2023-11-24T08:21:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500b63d0 0x7f66500d3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 --2- 172.21.15.107:0/3906117677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665015d490 0x7f66500cd8f0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 -- 172.21.15.107:0/3906117677 >> 172.21.15.107:0/3906117677 conn(0x7f6650061340 msgr2=0x7f6650061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 -- 172.21.15.107:0/3906117677 shutdown_connections 2023-11-24T08:21:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.777+0000 7f6656cf1640 1 -- 172.21.15.107:0/3906117677 wait complete. 2023-11-24T08:21:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.779+0000 7f6656cf1640 1 Processor -- start 2023-11-24T08:21:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.779+0000 7f6656cf1640 1 -- start start 2023-11-24T08:21:49.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f665015d490 0x7f665014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500ca370 0x7f66500b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665014ac80 con 0x7f66500ce270 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500cb7b0 con 0x7f665015d490 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6656cf1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500caba0 con 0x7f66500ca370 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6655cef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6655cef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49340/0 (socket says 172.21.15.107:49340) 2023-11-24T08:21:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.780+0000 7f6655cef640 1 -- 172.21.15.107:0/2754846192 learned_addr learned my addr 172.21.15.107:0/2754846192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f66554ee640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f665015d490 0x7f665014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f66564f0640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500ca370 0x7f66500b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 -- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500ca370 msgr2=0x7f66500b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500ca370 0x7f66500b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 -- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f665015d490 msgr2=0x7f665014a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f665015d490 0x7f665014a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 -- 172.21.15.107:0/2754846192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009a70 con 0x7f66500ce270 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6655cef640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f6640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640040650 con 0x7f66500ce270 2023-11-24T08:21:49.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66400407b0 con 0x7f66500ce270 2023-11-24T08:21:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66500cae80 con 0x7f66500ce270 2023-11-24T08:21:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.781+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640040b60 con 0x7f66500ce270 2023-11-24T08:21:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.782+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66500cb440 con 0x7f66500ce270 2023-11-24T08:21:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.783+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665004e840 con 0x7f66500ce270 2023-11-24T08:21:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.784+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f6640040cc0 con 0x7f66500ce270 2023-11-24T08:21:49.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.786+0000 7f664e7fc640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 0x7f663c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:49.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.786+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f66400ab9e0 con 0x7f66500ce270 2023-11-24T08:21:49.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.786+0000 7f66554ee640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 0x7f663c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:49.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.786+0000 7f66554ee640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 0x7f663c06f280 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f66500cd800 tx=0x7f6644008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:49.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.789+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6640078620 con 0x7f66500ce270 2023-11-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:49.982+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6650160e00 con 0x7f66500ce270 2023-11-24T08:21:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.545+0000 7f664e7fc640 1 -- 172.21.15.107:0/2754846192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v205) v1 ==== 170+0+0 (secure 0 0 0) 0x7f66400440d0 con 0x7f66500ce270 2023-11-24T08:21:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:21:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 msgr2=0x7f663c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 0x7f663c06f280 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f66500cd800 tx=0x7f6644008040 comp rx=0 tx=0).stop 2023-11-24T08:21:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 msgr2=0x7f665015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f6640004a40 comp rx=0 tx=0).stop 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 shutdown_connections 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f663c06cdc0 0x7f663c06f280 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66500ca370 0x7f66500b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f665015d490 0x7f665014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 --2- 172.21.15.107:0/2754846192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66500ce270 0x7f665015e870 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.548+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 >> 172.21.15.107:0/2754846192 conn(0x7f6650061340 msgr2=0x7f665015b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.549+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 shutdown_connections 2023-11-24T08:21:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.549+0000 7f6656cf1640 1 -- 172.21.15.107:0/2754846192 wait complete. 2023-11-24T08:21:50.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:21:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.765+0000 7f9008e85640 1 Processor -- start 2023-11-24T08:21:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.765+0000 7f9008e85640 1 -- start start 2023-11-24T08:21:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f900415b260 0x7f900406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f90040b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f90040b1080 0x7f90040ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90040b1530 con 0x7f90040b1080 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900415b640 con 0x7f900415b260 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.766+0000 7f9008e85640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900415ed20 con 0x7f900406e580 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f90040b1080 0x7f90040ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f90040b1080 0x7f90040ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49342/0 (socket says 172.21.15.107:49342) 2023-11-24T08:21:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 -- 172.21.15.107:0/2041556725 learned_addr learned my addr 172.21.15.107:0/2041556725 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f90037fe640 1 --2- 172.21.15.107:0/2041556725 >> v2:172.21.15.120:3300/0 conn(0x7f900415b260 0x7f900406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f8ffaffd640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f90040b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 -- 172.21.15.107:0/2041556725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 msgr2=0x7f90040b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f90040b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 -- 172.21.15.107:0/2041556725 >> v2:172.21.15.120:3300/0 conn(0x7f900415b260 msgr2=0x7f900406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 --2- 172.21.15.107:0/2041556725 >> v2:172.21.15.120:3300/0 conn(0x7f900415b260 0x7f900406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.767+0000 7f9003fff640 1 -- 172.21.15.107:0/2041556725 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900415f8a0 con 0x7f90040b1080 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.768+0000 7f9003fff640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040ccc90 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f90040d0160 tx=0x7f8ffc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36efa5568318695 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.769+0000 7f9002ffd640 1 -- 172.21.15.107:0/2041556725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc02c950 con 0x7f90040b1080 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.769+0000 7f9002ffd640 1 -- 172.21.15.107:0/2041556725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc02cab0 con 0x7f90040b1080 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.769+0000 7f9002ffd640 1 -- 172.21.15.107:0/2041556725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc02ce40 con 0x7f90040b1080 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.769+0000 7f9008e85640 1 -- 172.21.15.107:0/2041556725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 msgr2=0x7f90040ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.769+0000 7f9008e85640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040ccc90 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f90040d0160 tx=0x7f8ffc02bdb0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 -- 172.21.15.107:0/2041556725 shutdown_connections 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040ccc90 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 --2- 172.21.15.107:0/2041556725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f90040b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 --2- 172.21.15.107:0/2041556725 >> v2:172.21.15.120:3300/0 conn(0x7f900415b260 0x7f900406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 -- 172.21.15.107:0/2041556725 >> 172.21.15.107:0/2041556725 conn(0x7f90040cbb40 msgr2=0x7f90040cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 -- 172.21.15.107:0/2041556725 shutdown_connections 2023-11-24T08:21:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.770+0000 7f9008e85640 1 -- 172.21.15.107:0/2041556725 wait complete. 2023-11-24T08:21:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.771+0000 7f9008e85640 1 Processor -- start 2023-11-24T08:21:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.772+0000 7f9008e85640 1 -- start start 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f900414a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900415b260 0x7f9004155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b1ba0 con 0x7f90040b1080 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b1e80 con 0x7f900415b260 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9008e85640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b2160 con 0x7f900406e580 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49358/0 (socket says 172.21.15.107:49358) 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 -- 172.21.15.107:0/2994853508 learned_addr learned my addr 172.21.15.107:0/2994853508 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f90037fe640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f900414a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f9003fff640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900415b260 0x7f9004155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 -- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 msgr2=0x7f900414a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f900414a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 -- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900415b260 msgr2=0x7f9004155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900415b260 0x7f9004155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.773+0000 7f8ffaffd640 1 -- 172.21.15.107:0/2994853508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009b30 con 0x7f90040b1080 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f8ffaffd640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00d5f0 tx=0x7f8fec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0125f0 con 0x7f90040b1080 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec012750 con 0x7f90040b1080 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec012b00 con 0x7f90040b1080 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040b2440 con 0x7f90040b1080 2023-11-24T08:21:50.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.774+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900414aac0 con 0x7f90040b1080 2023-11-24T08:21:50.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.775+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900404e840 con 0x7f90040b1080 2023-11-24T08:21:50.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.776+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f8fec012c60 con 0x7f90040b1080 2023-11-24T08:21:50.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.778+0000 7f9000ff9640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 0x7f8fe406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:50.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.778+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8fec088bd0 con 0x7f90040b1080 2023-11-24T08:21:50.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.778+0000 7f90037fe640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 0x7f8fe406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.778+0000 7f90037fe640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 0x7f8fe406f280 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4006fd0 tx=0x7f8ff4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:50.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:50.783+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8fec0556e0 con 0x7f90040b1080 2023-11-24T08:21:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.014+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f90040ce640 con 0x7f90040b1080 2023-11-24T08:21:51.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.548+0000 7f9000ff9640 1 -- 172.21.15.107:0/2994853508 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 29 and data pool 30 v124) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8fec058d00 con 0x7f90040b1080 2023-11-24T08:21:51.551 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 29 and data pool 30 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 msgr2=0x7f8fe406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 0x7f8fe406f280 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4006fd0 tx=0x7f8ff4008040 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 msgr2=0x7f90040b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00d5f0 tx=0x7f8fec00dab0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 shutdown_connections 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f8fe406cdc0 0x7f8fe406f280 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900415b260 0x7f9004155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90040b1080 0x7f90040b3e80 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 --2- 172.21.15.107:0/2994853508 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f900406e580 0x7f900414a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.554+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 >> 172.21.15.107:0/2994853508 conn(0x7f90040cbb40 msgr2=0x7f90040adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.555+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 shutdown_connections 2023-11-24T08:21:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.555+0000 7f9008e85640 1 -- 172.21.15.107:0/2994853508 wait complete. 2023-11-24T08:21:51.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:51.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.776+0000 7f35ea1a0640 1 Processor -- start 2023-11-24T08:21:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.777+0000 7f35ea1a0640 1 -- start start 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35ea1a0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e415d710 0x7f35e40d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35ea1a0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35e40d17b0 0x7f35e40d1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35ea1a0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40d2130 con 0x7f35e4161120 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35ea1a0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e40cdd70 con 0x7f35e40d17b0 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35ea1a0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40b0e10 con 0x7f35e415d710 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35e899d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35e40d17b0 0x7f35e40d1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35e899d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35e40d17b0 0x7f35e40d1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49370/0 (socket says 172.21.15.107:49370) 2023-11-24T08:21:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35e899d640 1 -- 172.21.15.107:0/4270981073 learned_addr learned my addr 172.21.15.107:0/4270981073 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.778+0000 7f35e999f640 1 --1- 172.21.15.107:0/4270981073 >> v1:172.21.15.107:6789/0 conn(0x7f35e4161120 0x7f35e4161580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51744/0 (socket says 172.21.15.107:51744) 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e919e640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e415d710 0x7f35e40d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e899d640 1 -- 172.21.15.107:0/4270981073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e415d710 msgr2=0x7f35e40d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e899d640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e415d710 0x7f35e40d3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e899d640 1 -- 172.21.15.107:0/4270981073 >> v1:172.21.15.107:6789/0 conn(0x7f35e4161120 legacy=0x7f35e4161580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e3fff640 1 -- 172.21.15.107:0/4270981073 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533029034 0 0) 0x7f35e40d2130 con 0x7f35e4161120 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.779+0000 7f35e899d640 1 -- 172.21.15.107:0/4270981073 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40cdff0 con 0x7f35e40d17b0 2023-11-24T08:21:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.780+0000 7f35e899d640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e40d17b0 0x7f35e40d1bf0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009870 tx=0x7f35d002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0ed3703e7f3098 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:51.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.780+0000 7f35e3fff640 1 -- 172.21.15.107:0/4270981073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d002f830 con 0x7f35e40d17b0 2023-11-24T08:21:51.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.780+0000 7f35e3fff640 1 -- 172.21.15.107:0/4270981073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d002f990 con 0x7f35e40d17b0 2023-11-24T08:21:51.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.780+0000 7f35e3fff640 1 -- 172.21.15.107:0/4270981073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d002fd20 con 0x7f35e40d17b0 2023-11-24T08:21:51.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 -- 172.21.15.107:0/4270981073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e40d17b0 msgr2=0x7f35e40d1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e40d17b0 0x7f35e40d1bf0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009870 tx=0x7f35d002ba10 comp rx=0 tx=0).stop 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 -- 172.21.15.107:0/4270981073 shutdown_connections 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e40d17b0 0x7f35e40d1bf0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/4270981073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e415d710 0x7f35e40d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 -- 172.21.15.107:0/4270981073 >> 172.21.15.107:0/4270981073 conn(0x7f35e409c510 msgr2=0x7f35e40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 -- 172.21.15.107:0/4270981073 shutdown_connections 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.781+0000 7f35ea1a0640 1 -- 172.21.15.107:0/4270981073 wait complete. 2023-11-24T08:21:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.783+0000 7f35ea1a0640 1 Processor -- start 2023-11-24T08:21:51.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.783+0000 7f35ea1a0640 1 -- start start 2023-11-24T08:21:51.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e40d17b0 0x7f35e415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e4161120 0x7f35e4156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e4159b40 con 0x7f35e415d710 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e4157ef0 con 0x7f35e40d17b0 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35ea1a0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e41581d0 con 0x7f35e4161120 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35e899d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35e899d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49382/0 (socket says 172.21.15.107:49382) 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.784+0000 7f35e899d640 1 -- 172.21.15.107:0/3095372601 learned_addr learned my addr 172.21.15.107:0/3095372601 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e999f640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e4161120 0x7f35e4156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 -- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e4161120 msgr2=0x7f35e4156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e919e640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e40d17b0 0x7f35e415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e4161120 0x7f35e4156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 -- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e40d17b0 msgr2=0x7f35e415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e40d17b0 0x7f35e415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 -- 172.21.15.107:0/3095372601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00094d0 con 0x7f35e415d710 2023-11-24T08:21:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e899d640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f35d0000c00 tx=0x7f35d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d002f800 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d0041400 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e41584b0 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.785+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d00417d0 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.786+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e4157a20 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.787+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e41589d0 con 0x7f35e415d710 2023-11-24T08:21:51.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.788+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f35d0041930 con 0x7f35e415d710 2023-11-24T08:21:51.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.789+0000 7f35e1ffb640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 0x7f35c806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:51.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.789+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35d003c070 con 0x7f35e415d710 2023-11-24T08:21:51.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.789+0000 7f35e919e640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 0x7f35c806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:51.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.790+0000 7f35e919e640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 0x7f35c806f280 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f35d8008ef0 tx=0x7f35d8008440 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:51.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.793+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35e41589d0 con 0x7f35e415d710 2023-11-24T08:21:51.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.984+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35e41589d0 con 0x7f35e415d710 2023-11-24T08:21:51.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.986+0000 7f35e1ffb640 1 -- 172.21.15.107:0/3095372601 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+16524 (secure 0 0 0) 0x7f35e41589d0 con 0x7f35e415d710 2023-11-24T08:21:51.989 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:51.989 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":206,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:51.016583+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 msgr2=0x7f35c806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 0x7f35c806f280 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f35d8008ef0 tx=0x7f35d8008440 comp rx=0 tx=0).stop 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 msgr2=0x7f35e4156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f35d0000c00 tx=0x7f35d0004730 comp rx=0 tx=0).stop 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 shutdown_connections 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f35c806cdc0 0x7f35c806f280 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e4161120 0x7f35e4156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e415d710 0x7f35e4156750 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 --2- 172.21.15.107:0/3095372601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e40d17b0 0x7f35e415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 >> 172.21.15.107:0/3095372601 conn(0x7f35e409c510 msgr2=0x7f35e40d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 shutdown_connections 2023-11-24T08:21:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:51.989+0000 7f35ea1a0640 1 -- 172.21.15.107:0/3095372601 wait complete. 2023-11-24T08:21:52.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:52.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.221+0000 7f47606d3640 1 Processor -- start 2023-11-24T08:21:52.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.221+0000 7f47606d3640 1 -- start start 2023-11-24T08:21:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.222+0000 7f47606d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 0x7f475806de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.222+0000 7f47606d3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47580b21d0 0x7f47580cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.222+0000 7f47606d3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47580b3a40 con 0x7f47580cbf80 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f47606d3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580cc410 con 0x7f47580b21d0 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f47606d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580cdfa0 con 0x7f4758159ce0 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475dc6e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47580b21d0 0x7f47580cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475ec70640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47580cbf80 0x7f475815d760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51748/0 (socket says 172.21.15.107:51748) 2023-11-24T08:21:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475ec70640 1 -- 172.21.15.107:0/876212906 learned_addr learned my addr 172.21.15.107:0/876212906 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475e46f640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 0x7f475806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475dc6e640 1 -- 172.21.15.107:0/876212906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 msgr2=0x7f475806de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475dc6e640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 0x7f475806de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475d46d640 1 -- 172.21.15.107:0/876212906 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540662865 0 0) 0x7f47580b3a40 con 0x7f47580cbf80 2023-11-24T08:21:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475dc6e640 1 -- 172.21.15.107:0/876212906 >> v1:172.21.15.107:6789/0 conn(0x7f47580cbf80 legacy=0x7f475815d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475dc6e640 1 -- 172.21.15.107:0/876212906 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47580cd9e0 con 0x7f47580b21d0 2023-11-24T08:21:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.223+0000 7f475e46f640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 0x7f475806de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f475dc6e640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f47580cba40 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f47440097a0 tx=0x7f474402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b175c2cd0ebbf97a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f475d46d640 1 -- 172.21.15.107:0/876212906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744031830 con 0x7f47580b21d0 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f475d46d640 1 -- 172.21.15.107:0/876212906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744031990 con 0x7f47580b21d0 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f475d46d640 1 -- 172.21.15.107:0/876212906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744031d20 con 0x7f47580b21d0 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f47606d3640 1 -- 172.21.15.107:0/876212906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 msgr2=0x7f47580cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.225+0000 7f47606d3640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f47580cba40 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f47440097a0 tx=0x7f474402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 -- 172.21.15.107:0/876212906 shutdown_connections 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f47580cba40 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 --2- 172.21.15.107:0/876212906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4758159ce0 0x7f475806de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 -- 172.21.15.107:0/876212906 >> 172.21.15.107:0/876212906 conn(0x7f475806a920 msgr2=0x7f475806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 -- 172.21.15.107:0/876212906 shutdown_connections 2023-11-24T08:21:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.226+0000 7f47606d3640 1 -- 172.21.15.107:0/876212906 wait complete. 2023-11-24T08:21:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.227+0000 7f47606d3640 1 Processor -- start 2023-11-24T08:21:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.227+0000 7f47606d3640 1 -- start start 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580cbf80 0x7f4758099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4758159ce0 0x7f4758157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475809c4a0 con 0x7f47580b21d0 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475809aec0 con 0x7f4758159ce0 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.228+0000 7f47606d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475809b1a0 con 0x7f47580cbf80 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49414/0 (socket says 172.21.15.107:49414) 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 -- 172.21.15.107:0/3518363591 learned_addr learned my addr 172.21.15.107:0/3518363591 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 -- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580cbf80 msgr2=0x7f4758099140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580cbf80 0x7f4758099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 -- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4758159ce0 msgr2=0x7f4758157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4758159ce0 0x7f4758157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 -- 172.21.15.107:0/3518363591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4744009400 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.229+0000 7f475e46f640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f474c00ce00 tx=0x7f474c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.230+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00f890 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.230+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c00f9f0 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.230+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00fdc0 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.230+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475809b480 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.230+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475809ba40 con 0x7f47580b21d0 2023-11-24T08:21:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.231+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4758067470 con 0x7f47580b21d0 2023-11-24T08:21:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.232+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f474c00b040 con 0x7f47580b21d0 2023-11-24T08:21:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.233+0000 7f4756ffd640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 0x7f473c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.233+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f474c088fd0 con 0x7f47580b21d0 2023-11-24T08:21:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.233+0000 7f475dc6e640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 0x7f473c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.234+0000 7f475dc6e640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 0x7f473c06f280 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4744036000 tx=0x7f4744030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.237+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f474c055b80 con 0x7f47580b21d0 2023-11-24T08:21:52.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.427+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47580736a0 con 0x7f47580b21d0 2023-11-24T08:21:52.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.430+0000 7f4756ffd640 1 -- 172.21.15.107:0/3518363591 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+16524 (secure 0 0 0) 0x7f474c01d030 con 0x7f47580b21d0 2023-11-24T08:21:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:52.433 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":206,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:51.016583+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 msgr2=0x7f473c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 0x7f473c06f280 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4744036000 tx=0x7f4744030000 comp rx=0 tx=0).stop 2023-11-24T08:21:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 msgr2=0x7f475809bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f474c00ce00 tx=0x7f474c00a650 comp rx=0 tx=0).stop 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 shutdown_connections 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f473c06cdc0 0x7f473c06f280 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4758159ce0 0x7f4758157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580cbf80 0x7f4758099140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 --2- 172.21.15.107:0/3518363591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47580b21d0 0x7f475809bf60 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.433+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 >> 172.21.15.107:0/3518363591 conn(0x7f475806a920 msgr2=0x7f47580adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.434+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 shutdown_connections 2023-11-24T08:21:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.434+0000 7f47606d3640 1 -- 172.21.15.107:0/3518363591 wait complete. 2023-11-24T08:21:52.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:21:52.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.674+0000 7f4dee851640 1 Processor -- start 2023-11-24T08:21:52.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.674+0000 7f4dee851640 1 -- start start 2023-11-24T08:21:52.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.675+0000 7f4dee851640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 0x7f4de80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4dee851640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4de80cbf20 0x7f4de80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4dee851640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de80cc3a0 con 0x7f4de80c57e0 2023-11-24T08:21:52.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4dee851640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de815b820 con 0x7f4de80cbf20 2023-11-24T08:21:52.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4dee851640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de815eea0 con 0x7f4de815b440 2023-11-24T08:21:52.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4de7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 0x7f4de80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4de77fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4de80cbf20 0x7f4de80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4decdee640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4de80c57e0 0x7f4de80c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:52944/0 (socket says 172.21.15.107:52944) 2023-11-24T08:21:52.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.676+0000 7f4decdee640 1 -- 172.21.15.107:0/1072761116 learned_addr learned my addr 172.21.15.107:0/1072761116 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:52.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de77fe640 1 -- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 msgr2=0x7f4de80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:52.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de77fe640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 0x7f4de80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de77fe640 1 -- 172.21.15.107:0/1072761116 >> v1:172.21.15.120:6789/0 conn(0x7f4de80c57e0 legacy=0x7f4de80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de77fe640 1 -- 172.21.15.107:0/1072761116 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de806d600 con 0x7f4de80cbf20 2023-11-24T08:21:52.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de7fff640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 0x7f4de80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:21:52.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.677+0000 7f4de6ffd640 1 -- 172.21.15.107:0/1072761116 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703250707 0 0) 0x7f4de0001490 con 0x7f4de80c57e0 2023-11-24T08:21:52.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.679+0000 7f4de77fe640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de80c52a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0097a0 tx=0x7f4ddc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8cf01c159d623f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4de6ffd640 1 -- 172.21.15.107:0/1072761116 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc031780 con 0x7f4de80cbf20 2023-11-24T08:21:52.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4de6ffd640 1 -- 172.21.15.107:0/1072761116 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc0318e0 con 0x7f4de80cbf20 2023-11-24T08:21:52.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4de6ffd640 1 -- 172.21.15.107:0/1072761116 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc031c70 con 0x7f4de80cbf20 2023-11-24T08:21:52.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 -- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 msgr2=0x7f4de80c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de80c52a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0097a0 tx=0x7f4ddc02b820 comp rx=0 tx=0).stop 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 -- 172.21.15.107:0/1072761116 shutdown_connections 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de80c52a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 --2- 172.21.15.107:0/1072761116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de815b440 0x7f4de80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 -- 172.21.15.107:0/1072761116 >> 172.21.15.107:0/1072761116 conn(0x7f4de80b0c50 msgr2=0x7f4de80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 -- 172.21.15.107:0/1072761116 shutdown_connections 2023-11-24T08:21:52.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.680+0000 7f4dee851640 1 -- 172.21.15.107:0/1072761116 wait complete. 2023-11-24T08:21:52.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.682+0000 7f4dee851640 1 Processor -- start 2023-11-24T08:21:52.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.682+0000 7f4dee851640 1 -- start start 2023-11-24T08:21:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de80c57e0 0x7f4de80cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de8154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80cd330 con 0x7f4de815b440 2023-11-24T08:21:52.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80cdc90 con 0x7f4de80cbf20 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.683+0000 7f4dee851640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8155150 con 0x7f4de80c57e0 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49426/0 (socket says 172.21.15.107:49426) 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 -- 172.21.15.107:0/234035197 learned_addr learned my addr 172.21.15.107:0/234035197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4de77fe640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de8154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4de7fff640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de80c57e0 0x7f4de80cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 -- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de80c57e0 msgr2=0x7f4de80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de80c57e0 0x7f4de80cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 -- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 msgr2=0x7f4de8154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de8154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 -- 172.21.15.107:0/234035197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8009400 con 0x7f4de815b440 2023-11-24T08:21:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.684+0000 7f4decdee640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c820 tx=0x7f4de000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.685+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000a5e0 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.685+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de000a760 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.685+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000ab10 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.685+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc009400 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.685+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de80ceb40 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.686+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de80cd470 con 0x7f4de815b440 2023-11-24T08:21:52.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.687+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f4de000ac70 con 0x7f4de815b440 2023-11-24T08:21:52.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.689+0000 7f4de4ff9640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 0x7f4dd406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:52.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.689+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4de0087900 con 0x7f4de815b440 2023-11-24T08:21:52.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.689+0000 7f4de7fff640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 0x7f4dd406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:52.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.690+0000 7f4de7fff640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 0x7f4dd406f280 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4de80ce850 tx=0x7f4dd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:52.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.693+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4de80cd470 con 0x7f4de815b440 2023-11-24T08:21:52.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:52.920+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4de80cc3a0 con 0x7f4de815b440 2023-11-24T08:21:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.579+0000 7f4de4ff9640 1 -- 172.21.15.107:0/234035197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v127) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4de80cc3a0 con 0x7f4de815b440 2023-11-24T08:21:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 msgr2=0x7f4dd406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 0x7f4dd406f280 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4de80ce850 tx=0x7f4dd8008040 comp rx=0 tx=0).stop 2023-11-24T08:21:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 msgr2=0x7f4de81f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c820 tx=0x7f4de000cce0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 shutdown_connections 2023-11-24T08:21:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4dd406cdc0 0x7f4dd406f280 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4de815b440 0x7f4de81f39d0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de80cbf20 0x7f4de8154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 --2- 172.21.15.107:0/234035197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de80c57e0 0x7f4de80cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.583+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 >> 172.21.15.107:0/234035197 conn(0x7f4de80b0c50 msgr2=0x7f4de80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.584+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 shutdown_connections 2023-11-24T08:21:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.584+0000 7f4dee851640 1 -- 172.21.15.107:0/234035197 wait complete. 2023-11-24T08:21:53.608 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:21:53.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:21:53.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.822+0000 7f6e9de2b640 1 Processor -- start 2023-11-24T08:21:53.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.822+0000 7f6e9de2b640 1 -- start start 2023-11-24T08:21:53.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e9de2b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e980c59b0 0x7f6e98166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e9de2b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980c5dc0 con 0x7f6e980cc0f0 2023-11-24T08:21:53.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e9de2b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9815eea0 con 0x7f6e9815b440 2023-11-24T08:21:53.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e9de2b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9815b820 con 0x7f6e980c59b0 2023-11-24T08:21:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e96ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e980cc0f0 0x7f6e980c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51758/0 (socket says 172.21.15.107:51758) 2023-11-24T08:21:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.824+0000 7f6e96ffd640 1 -- 172.21.15.107:0/738482228 learned_addr learned my addr 172.21.15.107:0/738482228 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:53.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557069851 0 0) 0x7f6e980c5dc0 con 0x7f6e980cc0f0 2023-11-24T08:21:53.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84003520 con 0x7f6e980cc0f0 2023-11-24T08:21:53.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e97fff640 1 --2- 172.21.15.107:0/738482228 >> v2:172.21.15.120:3300/0 conn(0x7f6e980c59b0 0x7f6e98166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 771287782 0 0) 0x7f6e84003520 con 0x7f6e980cc0f0 2023-11-24T08:21:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 >> v2:172.21.15.120:3300/0 conn(0x7f6e980c59b0 msgr2=0x7f6e98166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 --2- 172.21.15.107:0/738482228 >> v2:172.21.15.120:3300/0 conn(0x7f6e980c59b0 0x7f6e98166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 >> v1:172.21.15.120:6789/0 conn(0x7f6e9815b440 legacy=0x7f6e980cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9806d600 con 0x7f6e980cc0f0 2023-11-24T08:21:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6e80001670 con 0x7f6e980cc0f0 2023-11-24T08:21:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 >> v1:172.21.15.107:6789/0 conn(0x7f6e980cc0f0 legacy=0x7f6e980c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 0x7f6e84003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e840047c0 0x7f6e84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840052a0 0x7f6e8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9815b820 con 0x7f6e840039c0 2023-11-24T08:21:53.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400c1c0 con 0x7f6e840052a0 2023-11-24T08:21:53.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.825+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400c380 con 0x7f6e840047c0 2023-11-24T08:21:53.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 0x7f6e84003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e96ffd640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e840047c0 0x7f6e84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e97fff640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840052a0 0x7f6e8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 -- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e840047c0 msgr2=0x7f6e84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e840047c0 0x7f6e84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 -- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840052a0 msgr2=0x7f6e8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840052a0 0x7f6e8400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e977fe640 1 -- 172.21.15.107:0/738482228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84003520 con 0x7f6e840039c0 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.826+0000 7f6e97fff640 1 -- 172.21.15.107:0/738482228 reap_dead start 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.827+0000 7f6e977fe640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 0x7f6e84003de0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001e50 tx=0x7f6e840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fecab9216375b50 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88003b70 con 0x7f6e840039c0 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88003cd0 con 0x7f6e840039c0 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e967fc640 1 -- 172.21.15.107:0/738482228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88005310 con 0x7f6e840039c0 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 -- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 msgr2=0x7f6e84003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 0x7f6e84003de0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001e50 tx=0x7f6e840057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 -- 172.21.15.107:0/738482228 shutdown_connections 2023-11-24T08:21:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/738482228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e840039c0 0x7f6e84003de0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 -- 172.21.15.107:0/738482228 >> 172.21.15.107:0/738482228 conn(0x7f6e980b0c50 msgr2=0x7f6e980b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 -- 172.21.15.107:0/738482228 shutdown_connections 2023-11-24T08:21:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.828+0000 7f6e9de2b640 1 -- 172.21.15.107:0/738482228 wait complete. 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.830+0000 7f6e9de2b640 1 Processor -- start 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.830+0000 7f6e9de2b640 1 -- start start 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840039c0 0x7f6e980afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981f35e0 0x7f6e981f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980ccf70 con 0x7f6e981f3fa0 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980ccc70 con 0x7f6e840039c0 2023-11-24T08:21:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.831+0000 7f6e9de2b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980c7c60 con 0x7f6e981f35e0 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49444/0 (socket says 172.21.15.107:49444) 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 -- 172.21.15.107:0/1194924379 learned_addr learned my addr 172.21.15.107:0/1194924379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e96ffd640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981f35e0 0x7f6e981f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 -- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981f35e0 msgr2=0x7f6e981f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981f35e0 0x7f6e981f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e977fe640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840039c0 0x7f6e980afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 -- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840039c0 msgr2=0x7f6e980afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840039c0 0x7f6e980afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 -- 172.21.15.107:0/1194924379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e981f44c0 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e97fff640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f6e9806e890 tx=0x7f6e9000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.832+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90004230 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.833+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e900043b0 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.833+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90004520 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.833+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980c7f00 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.833+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e980c84c0 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.834+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e980cd0e0 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.835+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f6e90003850 con 0x7f6e981f3fa0 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.836+0000 7f6e8ffff640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 0x7f6e7806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:53.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.836+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e90087ea0 con 0x7f6e981f3fa0 2023-11-24T08:21:53.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.836+0000 7f6e977fe640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 0x7f6e7806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:53.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.837+0000 7f6e977fe640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 0x7f6e7806f280 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6e88034000 tx=0x7f6e88009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:53.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:53.840+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e900549b0 con 0x7f6e981f3fa0 2023-11-24T08:21:54.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.071+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6e9804e8b0 con 0x7f6e981f3fa0 2023-11-24T08:21:54.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.602+0000 7f6e8ffff640 1 -- 172.21.15.107:0/1194924379 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v128) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6e9000c020 con 0x7f6e981f3fa0 2023-11-24T08:21:54.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 msgr2=0x7f6e7806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:54.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 0x7f6e7806f280 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6e88034000 tx=0x7f6e88009030 comp rx=0 tx=0).stop 2023-11-24T08:21:54.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 msgr2=0x7f6e980cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:54.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f6e9806e890 tx=0x7f6e9000e830 comp rx=0 tx=0).stop 2023-11-24T08:21:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 shutdown_connections 2023-11-24T08:21:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e7806cdc0 0x7f6e7806f280 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e981f3fa0 0x7f6e980cd400 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981f35e0 0x7f6e981f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 --2- 172.21.15.107:0/1194924379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e840039c0 0x7f6e980afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 >> 172.21.15.107:0/1194924379 conn(0x7f6e980b0c50 msgr2=0x7f6e980ac150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:54.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 shutdown_connections 2023-11-24T08:21:54.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.606+0000 7f6e9de2b640 1 -- 172.21.15.107:0/1194924379 wait complete. 2023-11-24T08:21:54.629 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:21:54.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:54.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.835+0000 7f3f2b59e640 1 Processor -- start 2023-11-24T08:21:54.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.835+0000 7f3f2b59e640 1 -- start start 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.836+0000 7f3f2b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f2c13d120 0x7f3f2c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2b59e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f2c06d6c0 0x7f3f2c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2b59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c06e060 con 0x7f3f2c0b0a70 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2b59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c13f9e0 con 0x7f3f2c13d120 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2b59e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c13fcc0 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f2c13d120 0x7f3f2c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3f2c13d120 0x7f3f2c15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49454/0 (socket says 172.21.15.107:49454) 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 -- 172.21.15.107:0/2523733689 learned_addr learned my addr 172.21.15.107:0/2523733689 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 -- 172.21.15.107:0/2523733689 >> v2:172.21.15.120:3300/0 conn(0x7f3f2c06d6c0 msgr2=0x7f3f2c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 --2- 172.21.15.107:0/2523733689 >> v2:172.21.15.120:3300/0 conn(0x7f3f2c06d6c0 0x7f3f2c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 -- 172.21.15.107:0/2523733689 >> v1:172.21.15.120:6789/0 conn(0x7f3f2c0b0a70 legacy=0x7f3f2c0b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.837+0000 7f3f2a59c640 1 -- 172.21.15.107:0/2523733689 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c0cec00 con 0x7f3f2c13d120 2023-11-24T08:21:54.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.838+0000 7f3f2a59c640 1 --2- 172.21.15.107:0/2523733689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c13d120 0x7f3f2c15af40 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c072a70 tx=0x7f3f1c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e5c5e326afadfd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:54.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f29d9b640 1 -- 172.21.15.107:0/2523733689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c02ee00 con 0x7f3f2c13d120 2023-11-24T08:21:54.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f29d9b640 1 -- 172.21.15.107:0/2523733689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c036440 con 0x7f3f2c13d120 2023-11-24T08:21:54.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f29d9b640 1 -- 172.21.15.107:0/2523733689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c0367d0 con 0x7f3f2c13d120 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 -- 172.21.15.107:0/2523733689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c13d120 msgr2=0x7f3f2c15af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/2523733689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c13d120 0x7f3f2c15af40 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c072a70 tx=0x7f3f1c02e050 comp rx=0 tx=0).stop 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 -- 172.21.15.107:0/2523733689 shutdown_connections 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/2523733689 >> v2:172.21.15.120:3300/0 conn(0x7f3f2c06d6c0 0x7f3f2c06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/2523733689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c13d120 0x7f3f2c15af40 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.839+0000 7f3f2b59e640 1 -- 172.21.15.107:0/2523733689 >> 172.21.15.107:0/2523733689 conn(0x7f3f2c0cb830 msgr2=0x7f3f2c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.840+0000 7f3f2b59e640 1 -- 172.21.15.107:0/2523733689 shutdown_connections 2023-11-24T08:21:54.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.840+0000 7f3f2b59e640 1 -- 172.21.15.107:0/2523733689 wait complete. 2023-11-24T08:21:54.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.841+0000 7f3f2b59e640 1 Processor -- start 2023-11-24T08:21:54.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.842+0000 7f3f2b59e640 1 -- start start 2023-11-24T08:21:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.842+0000 7f3f2b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2b59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f2c0b0a70 0x7f3f2c0b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f2c13d120 0x7f3f2c13ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0ac8b0 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2b59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c13b0d0 con 0x7f3f2c13d120 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2b59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1f6780 con 0x7f3f2c0b0a70 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49458/0 (socket says 172.21.15.107:49458) 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 -- 172.21.15.107:0/259641615 learned_addr learned my addr 172.21.15.107:0/259641615 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 -- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f2c0b0a70 msgr2=0x7f3f2c0b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f2c0b0a70 0x7f3f2c0b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 -- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f2c13d120 msgr2=0x7f3f2c13ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:54.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f2c13d120 0x7f3f2c13ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:54.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.843+0000 7f3f2a59c640 1 -- 172.21.15.107:0/259641615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c00b770 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f2a59c640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c02e090 tx=0x7f3f1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c034950 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c034ad0 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c1f6a60 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c034e80 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.844+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c1f7020 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.845+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c04e840 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.846+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f3f1c041480 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.848+0000 7f3f22ffd640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 0x7f3f0c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.848+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f1c0ab9b0 con 0x7f3f2c06d6c0 2023-11-24T08:21:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.848+0000 7f3f23fff640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 0x7f3f0c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.849+0000 7f3f23fff640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 0x7f3f0c06f280 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c143d70 tx=0x7f3f14006cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:54.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:54.852+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f1c0784c0 con 0x7f3f2c06d6c0 2023-11-24T08:21:55.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.068+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2c04e840 con 0x7f3f2c06d6c0 2023-11-24T08:21:55.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.069+0000 7f3f22ffd640 1 -- 172.21.15.107:0/259641615 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f3f1c03b090 con 0x7f3f2c06d6c0 2023-11-24T08:21:55.071 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:55.072 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":128,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:21:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 128 2023-11-24T08:21:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 msgr2=0x7f3f0c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 0x7f3f0c06f280 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c143d70 tx=0x7f3f14006cb0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 msgr2=0x7f3f2c0b00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c02e090 tx=0x7f3f1c004730 comp rx=0 tx=0).stop 2023-11-24T08:21:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 shutdown_connections 2023-11-24T08:21:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3f0c06cdc0 0x7f3f0c06f280 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f2c13d120 0x7f3f2c13ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f2c0b0a70 0x7f3f2c0b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 --2- 172.21.15.107:0/259641615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f2c06d6c0 0x7f3f2c0b00c0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 >> 172.21.15.107:0/259641615 conn(0x7f3f2c0cb830 msgr2=0x7f3f2c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 shutdown_connections 2023-11-24T08:21:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.072+0000 7f3f2b59e640 1 -- 172.21.15.107:0/259641615 wait complete. 2023-11-24T08:21:55.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:21:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.301+0000 7fef08648640 1 Processor -- start 2023-11-24T08:21:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.301+0000 7fef08648640 1 -- start start 2023-11-24T08:21:55.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef08648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7fef00161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef08648640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef000cf450 0x7fef000cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef08648640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000ce0b0 con 0x7fef0015dc80 2023-11-24T08:21:55.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef08648640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef000cfe10 con 0x7fef000cf450 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef08648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef00161960 con 0x7fef00157c70 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef06be5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fef0015dc80 0x7fef0015e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51760/0 (socket says 172.21.15.107:51760) 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef06be5640 1 -- 172.21.15.107:0/532385958 learned_addr learned my addr 172.21.15.107:0/532385958 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef063e4640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7fef00161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef05be3640 1 --2- 172.21.15.107:0/532385958 >> v2:172.21.15.120:3300/0 conn(0x7fef000cf450 0x7fef000cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.303+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527332917 0 0) 0x7fef000ce0b0 con 0x7fef0015dc80 2023-11-24T08:21:55.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec003520 con 0x7fef0015dc80 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230655522 0 0) 0x7feeec003520 con 0x7fef0015dc80 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 msgr2=0x7fef00161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7fef00161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 >> v2:172.21.15.120:3300/0 conn(0x7fef000cf450 msgr2=0x7fef000cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 --2- 172.21.15.107:0/532385958 >> v2:172.21.15.120:3300/0 conn(0x7fef000cf450 0x7fef000cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0015e8d0 con 0x7fef0015dc80 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7feef80019f0 con 0x7fef0015dc80 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 >> v1:172.21.15.107:6789/0 conn(0x7fef0015dc80 legacy=0x7fef0015e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7feeec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeec003ef0 0x7feeec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7feeec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000cfe10 con 0x7feeec004d00 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00161960 con 0x7feeec003ef0 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00b9d0 con 0x7fef00157c70 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.304+0000 7fef06be5640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7feeec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef063e4640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7feeec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef05be3640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeec003ef0 0x7feeec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef06be5640 1 -- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 msgr2=0x7feeec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef06be5640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00157c70 0x7feeec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef06be5640 1 -- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeec003ef0 msgr2=0x7feeec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef06be5640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeec003ef0 0x7feeec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.305+0000 7fef06be5640 1 -- 172.21.15.107:0/532385958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec003520 con 0x7feeec004d00 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef06be5640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7feeec005180 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7feef800a890 tx=0x7feeec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcceda53ad878b44 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8034630 con 0x7feeec004d00 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef802f070 con 0x7feeec004d00 2023-11-24T08:21:55.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef053e2640 1 -- 172.21.15.107:0/532385958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8034a30 con 0x7feeec004d00 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef08648640 1 -- 172.21.15.107:0/532385958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 msgr2=0x7feeec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef08648640 1 --2- 172.21.15.107:0/532385958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7feeec005180 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7feef800a890 tx=0x7feeec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.306+0000 7fef06be5640 1 -- 172.21.15.107:0/532385958 reap_dead start 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.307+0000 7fef08648640 1 -- 172.21.15.107:0/532385958 shutdown_connections 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.307+0000 7fef08648640 1 -- 172.21.15.107:0/532385958 >> 172.21.15.107:0/532385958 conn(0x7fef0006a920 msgr2=0x7fef0006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.307+0000 7fef08648640 1 -- 172.21.15.107:0/532385958 shutdown_connections 2023-11-24T08:21:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.307+0000 7fef08648640 1 -- 172.21.15.107:0/532385958 wait complete. 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.308+0000 7fef08648640 1 Processor -- start 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 -- start start 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef00099140 0x7fef0009b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00152b80 0x7fef0009a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00148910 con 0x7feeec004d00 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000995d0 con 0x7fef00099140 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.309+0000 7fef08648640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef000998b0 con 0x7fef00152b80 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49484/0 (socket says 172.21.15.107:49484) 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 -- 172.21.15.107:0/3850587424 learned_addr learned my addr 172.21.15.107:0/3850587424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef06be5640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00152b80 0x7fef0009a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 -- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00152b80 msgr2=0x7fef0009a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00152b80 0x7fef0009a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef05be3640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef00099140 0x7fef0009b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 -- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef00099140 msgr2=0x7fef0009b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef00099140 0x7fef0009b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 -- 172.21.15.107:0/3850587424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00099b90 con 0x7feeec004d00 2023-11-24T08:21:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.310+0000 7fef063e4640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7feefc011600 tx=0x7feefc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.311+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00f920 con 0x7feeec004d00 2023-11-24T08:21:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.311+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc00fa80 con 0x7feeec004d00 2023-11-24T08:21:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.311+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00fdf0 con 0x7feeec004d00 2023-11-24T08:21:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.311+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00099e70 con 0x7feeec004d00 2023-11-24T08:21:55.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.311+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0009b690 con 0x7feeec004d00 2023-11-24T08:21:55.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.312+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00067470 con 0x7feeec004d00 2023-11-24T08:21:55.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.313+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7feefc015460 con 0x7feeec004d00 2023-11-24T08:21:55.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.314+0000 7feef6ffd640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 0x7feee406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.314+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feefc088af0 con 0x7feeec004d00 2023-11-24T08:21:55.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.314+0000 7fef05be3640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 0x7feee406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.315+0000 7fef05be3640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 0x7feee406f2b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fef000cef10 tx=0x7feef0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.319+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feefc0556a0 con 0x7feeec004d00 2023-11-24T08:21:55.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.510+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef000cc860 con 0x7feeec004d00 2023-11-24T08:21:55.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.511+0000 7feef6ffd640 1 -- 172.21.15.107:0/3850587424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16511 (secure 0 0 0) 0x7feefc022020 con 0x7feeec004d00 2023-11-24T08:21:55.514 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:55.514 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":207,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:52.567075+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:21:55.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 msgr2=0x7feee406f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 0x7feee406f2b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fef000cef10 tx=0x7feef0008040 comp rx=0 tx=0).stop 2023-11-24T08:21:55.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 msgr2=0x7fef001483d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7feefc011600 tx=0x7feefc011ac0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 shutdown_connections 2023-11-24T08:21:55.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feee406cdf0 0x7feee406f2b0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef00152b80 0x7fef0009a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef00099140 0x7fef0009b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 --2- 172.21.15.107:0/3850587424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeec004d00 0x7fef001483d0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.514+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 >> 172.21.15.107:0/3850587424 conn(0x7fef0006a920 msgr2=0x7fef000ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.515+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 shutdown_connections 2023-11-24T08:21:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.515+0000 7fef08648640 1 -- 172.21.15.107:0/3850587424 wait complete. 2023-11-24T08:21:55.535 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:21:55.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:21:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.735+0000 7f1e5ea00640 1 Processor -- start 2023-11-24T08:21:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.735+0000 7f1e5ea00640 1 -- start start 2023-11-24T08:21:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.736+0000 7f1e5ea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1e580b2250 0x7f1e5815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e5ea00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5815ab80 con 0x7f1e580b2630 2023-11-24T08:21:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e5ea00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5815e680 con 0x7f1e580b2aa0 2023-11-24T08:21:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e5ea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5815a990 con 0x7f1e580b2250 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e4f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e580b2630 0x7f1e580ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51768/0 (socket says 172.21.15.107:51768) 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e4f7fe640 1 -- 172.21.15.107:0/3580014663 learned_addr learned my addr 172.21.15.107:0/3580014663 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e57fff640 1 --2- 172.21.15.107:0/3580014663 >> v2:172.21.15.120:3300/0 conn(0x7f1e580b2250 0x7f1e5815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.737+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374651345 0 0) 0x7f1e5815ab80 con 0x7f1e580b2630 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e44003520 con 0x7f1e580b2630 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799112468 0 0) 0x7f1e44003520 con 0x7f1e580b2630 2023-11-24T08:21:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 >> v2:172.21.15.120:3300/0 conn(0x7f1e580b2250 msgr2=0x7f1e5815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 --2- 172.21.15.107:0/3580014663 >> v2:172.21.15.120:3300/0 conn(0x7f1e580b2250 0x7f1e5815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 >> v1:172.21.15.120:6789/0 conn(0x7f1e580b2aa0 legacy=0x7f1e5815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5815e960 con 0x7f1e580b2630 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1e40001670 con 0x7f1e580b2630 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 >> v1:172.21.15.107:6789/0 conn(0x7f1e580b2630 legacy=0x7f1e580ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e44003970 0x7f1e44003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 0x7f1e44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e44005180 0x7f1e4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4400c050 con 0x7f1e440047c0 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4400c190 con 0x7f1e44005180 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4400c350 con 0x7f1e44003970 2023-11-24T08:21:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.738+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 0x7f1e44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e57fff640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e44003970 0x7f1e44003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e4f7fe640 1 -- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e44003970 msgr2=0x7f1e44003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e44003970 0x7f1e44003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e4f7fe640 1 -- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e44005180 msgr2=0x7f1e4400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e44005180 0x7f1e4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e5cf9d640 1 -- 172.21.15.107:0/3580014663 reap_dead start 2023-11-24T08:21:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.739+0000 7f1e4f7fe640 1 -- 172.21.15.107:0/3580014663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44003520 con 0x7f1e440047c0 2023-11-24T08:21:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.740+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 0x7f1e44004c40 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000c240 tx=0x7f1e440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8b81d68fc4a862 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.740+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40005ad0 con 0x7f1e440047c0 2023-11-24T08:21:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.740+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e40005c30 con 0x7f1e440047c0 2023-11-24T08:21:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.740+0000 7f1e577fe640 1 -- 172.21.15.107:0/3580014663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4002c8b0 con 0x7f1e440047c0 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 msgr2=0x7f1e44004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 0x7f1e44004c40 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000c240 tx=0x7f1e440057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3580014663 shutdown_connections 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3580014663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440047c0 0x7f1e44004c40 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3580014663 >> 172.21.15.107:0/3580014663 conn(0x7f1e5806a920 msgr2=0x7f1e5806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:55.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3580014663 shutdown_connections 2023-11-24T08:21:55.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.741+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3580014663 wait complete. 2023-11-24T08:21:55.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.743+0000 7f1e5ea00640 1 Processor -- start 2023-11-24T08:21:55.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.743+0000 7f1e5ea00640 1 -- start start 2023-11-24T08:21:55.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440047c0 0x7f1e5815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 0x7f1e580c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580afa60 con 0x7f1e58157200 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580afd40 con 0x7f1e580af560 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580b0020 con 0x7f1e440047c0 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5cf9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5cf9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49502/0 (socket says 172.21.15.107:49502) 2023-11-24T08:21:55.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.744+0000 7f1e5cf9d640 1 -- 172.21.15.107:0/3942851944 learned_addr learned my addr 172.21.15.107:0/3942851944 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:55.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 0x7f1e580c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e57fff640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440047c0 0x7f1e5815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 -- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440047c0 msgr2=0x7f1e5815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440047c0 0x7f1e5815c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 -- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 msgr2=0x7f1e580c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 0x7f1e580c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 -- 172.21.15.107:0/3942851944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580b0300 con 0x7f1e58157200 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e4f7fe640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 0x7f1e580c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5cf9d640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f1e58158560 tx=0x7f1e5000c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e500072d0 con 0x7f1e58157200 2023-11-24T08:21:55.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e50007430 con 0x7f1e58157200 2023-11-24T08:21:55.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e580b05e0 con 0x7f1e58157200 2023-11-24T08:21:55.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e500057c0 con 0x7f1e58157200 2023-11-24T08:21:55.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.745+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5815c7a0 con 0x7f1e58157200 2023-11-24T08:21:55.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.747+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e580b07d0 con 0x7f1e58157200 2023-11-24T08:21:55.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.748+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f1e50005920 con 0x7f1e58157200 2023-11-24T08:21:55.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.749+0000 7f1e557fa640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 0x7f1e3806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:55.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.749+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e50087790 con 0x7f1e58157200 2023-11-24T08:21:55.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.749+0000 7f1e57fff640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 0x7f1e3806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:55.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.750+0000 7f1e57fff640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 0x7f1e3806f2b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b1d00 tx=0x7f1e4800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:55.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.753+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e50054340 con 0x7f1e58157200 2023-11-24T08:21:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.940+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e580afeb0 con 0x7f1e3806cdf0 2023-11-24T08:21:55.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.962+0000 7f1e557fa640 1 -- 172.21.15.107:0/3942851944 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287275 (secure 0 0 0) 0x7f1e580afeb0 con 0x7f1e3806cdf0 2023-11-24T08:21:55.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:55.970 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:21:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 msgr2=0x7f1e3806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 0x7f1e3806f2b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b1d00 tx=0x7f1e4800b040 comp rx=0 tx=0).stop 2023-11-24T08:21:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 msgr2=0x7f1e580c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f1e58158560 tx=0x7f1e5000c570 comp rx=0 tx=0).stop 2023-11-24T08:21:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 shutdown_connections 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1e3806cdf0 0x7f1e3806f2b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e58157200 0x7f1e580c9e00 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e580af560 0x7f1e580c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 --2- 172.21.15.107:0/3942851944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440047c0 0x7f1e5815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 >> 172.21.15.107:0/3942851944 conn(0x7f1e5806a920 msgr2=0x7f1e58159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 shutdown_connections 2023-11-24T08:21:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:55.970+0000 7f1e5ea00640 1 -- 172.21.15.107:0/3942851944 wait complete. 2023-11-24T08:21:55.992 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2023-11-24T08:21:55.758868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329276,"kb_used_data":21568,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754645444,"statfs":{"total":773094113280,"available":772756934656,"internally_reserved":0,"allocated":22085632,"data_stored":10438612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":314953278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001828"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598347+0000","last_change":"2023-11-24T08:21:51.590343+0000","last_active":"2023-11-24T08:21:51.598347+0000","last_peered":"2023-11-24T08:21:51.598347+0000","last_clean":"2023-11-24T08:21:51.598347+0000","last_became_active":"2023-11-24T08:21:51.590064+0000","last_became_peered":"2023-11-24T08:21:51.590064+0000","last_unstale":"2023-11-24T08:21:51.598347+0000","last_undegraded":"2023-11-24T08:21:51.598347+0000","last_fullsized":"2023-11-24T08:21:51.598347+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:39.783416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584357+0000","last_change":"2023-11-24T08:21:51.580536+0000","last_active":"2023-11-24T08:21:51.584357+0000","last_peered":"2023-11-24T08:21:51.584357+0000","last_clean":"2023-11-24T08:21:51.584357+0000","last_became_active":"2023-11-24T08:21:51.579046+0000","last_became_peered":"2023-11-24T08:21:51.579046+0000","last_unstale":"2023-11-24T08:21:51.584357+0000","last_undegraded":"2023-11-24T08:21:51.584357+0000","last_fullsized":"2023-11-24T08:21:51.584357+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:36.358037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.579780+0000","last_change":"2023-11-24T08:21:51.573236+0000","last_active":"2023-11-24T08:21:51.579780+0000","last_peered":"2023-11-24T08:21:51.579780+0000","last_clean":"2023-11-24T08:21:51.579780+0000","last_became_active":"2023-11-24T08:21:51.572949+0000","last_became_peered":"2023-11-24T08:21:51.572949+0000","last_unstale":"2023-11-24T08:21:51.579780+0000","last_undegraded":"2023-11-24T08:21:51.579780+0000","last_fullsized":"2023-11-24T08:21:51.579780+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:11.160812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598416+0000","last_change":"2023-11-24T08:21:51.595111+0000","last_active":"2023-11-24T08:21:51.598416+0000","last_peered":"2023-11-24T08:21:51.598416+0000","last_clean":"2023-11-24T08:21:51.598416+0000","last_became_active":"2023-11-24T08:21:51.589089+0000","last_became_peered":"2023-11-24T08:21:51.589089+0000","last_unstale":"2023-11-24T08:21:51.598416+0000","last_undegraded":"2023-11-24T08:21:51.598416+0000","last_fullsized":"2023-11-24T08:21:51.598416+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:35.455602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588841+0000","last_change":"2023-11-24T08:21:51.583718+0000","last_active":"2023-11-24T08:21:51.588841+0000","last_peered":"2023-11-24T08:21:51.588841+0000","last_clean":"2023-11-24T08:21:51.588841+0000","last_became_active":"2023-11-24T08:21:51.583398+0000","last_became_peered":"2023-11-24T08:21:51.583398+0000","last_unstale":"2023-11-24T08:21:51.588841+0000","last_undegraded":"2023-11-24T08:21:51.588841+0000","last_fullsized":"2023-11-24T08:21:51.588841+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:41:22.205969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584271+0000","last_change":"2023-11-24T08:21:51.586321+0000","last_active":"2023-11-24T08:21:52.584271+0000","last_peered":"2023-11-24T08:21:52.584271+0000","last_clean":"2023-11-24T08:21:52.584271+0000","last_became_active":"2023-11-24T08:21:51.584951+0000","last_became_peered":"2023-11-24T08:21:51.584951+0000","last_unstale":"2023-11-24T08:21:52.584271+0000","last_undegraded":"2023-11-24T08:21:52.584271+0000","last_fullsized":"2023-11-24T08:21:52.584271+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:23.995739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581027+0000","last_change":"2023-11-24T08:21:51.575848+0000","last_active":"2023-11-24T08:21:51.581027+0000","last_peered":"2023-11-24T08:21:51.581027+0000","last_clean":"2023-11-24T08:21:51.581027+0000","last_became_active":"2023-11-24T08:21:51.575592+0000","last_became_peered":"2023-11-24T08:21:51.575592+0000","last_unstale":"2023-11-24T08:21:51.581027+0000","last_undegraded":"2023-11-24T08:21:51.581027+0000","last_fullsized":"2023-11-24T08:21:51.581027+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:58:17.711093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583729+0000","last_change":"2023-11-24T08:21:51.592099+0000","last_active":"2023-11-24T08:21:52.583729+0000","last_peered":"2023-11-24T08:21:52.583729+0000","last_clean":"2023-11-24T08:21:52.583729+0000","last_became_active":"2023-11-24T08:21:51.591780+0000","last_became_peered":"2023-11-24T08:21:51.591780+0000","last_unstale":"2023-11-24T08:21:52.583729+0000","last_undegraded":"2023-11-24T08:21:52.583729+0000","last_fullsized":"2023-11-24T08:21:52.583729+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:53.955792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.574534+0000","last_change":"2023-11-24T08:21:51.567654+0000","last_active":"2023-11-24T08:21:51.574534+0000","last_peered":"2023-11-24T08:21:51.574534+0000","last_clean":"2023-11-24T08:21:51.574534+0000","last_became_active":"2023-11-24T08:21:51.567346+0000","last_became_peered":"2023-11-24T08:21:51.567346+0000","last_unstale":"2023-11-24T08:21:51.574534+0000","last_undegraded":"2023-11-24T08:21:51.574534+0000","last_fullsized":"2023-11-24T08:21:51.574534+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:53.803218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584340+0000","last_change":"2023-11-24T08:21:51.579885+0000","last_active":"2023-11-24T08:21:51.584340+0000","last_peered":"2023-11-24T08:21:51.584340+0000","last_clean":"2023-11-24T08:21:51.584340+0000","last_became_active":"2023-11-24T08:21:51.579643+0000","last_became_peered":"2023-11-24T08:21:51.579643+0000","last_unstale":"2023-11-24T08:21:51.584340+0000","last_undegraded":"2023-11-24T08:21:51.584340+0000","last_fullsized":"2023-11-24T08:21:51.584340+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:12:56.367676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.596212+0000","last_change":"2023-11-24T08:21:51.579110+0000","last_active":"2023-11-24T08:21:51.596212+0000","last_peered":"2023-11-24T08:21:51.596212+0000","last_clean":"2023-11-24T08:21:51.596212+0000","last_became_active":"2023-11-24T08:21:51.572264+0000","last_became_peered":"2023-11-24T08:21:51.572264+0000","last_unstale":"2023-11-24T08:21:51.596212+0000","last_undegraded":"2023-11-24T08:21:51.596212+0000","last_fullsized":"2023-11-24T08:21:51.596212+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:47.386507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585287+0000","last_change":"2023-11-24T08:21:51.602686+0000","last_active":"2023-11-24T08:21:52.585287+0000","last_peered":"2023-11-24T08:21:52.585287+0000","last_clean":"2023-11-24T08:21:52.585287+0000","last_became_active":"2023-11-24T08:21:51.601606+0000","last_became_peered":"2023-11-24T08:21:51.601606+0000","last_unstale":"2023-11-24T08:21:52.585287+0000","last_undegraded":"2023-11-24T08:21:52.585287+0000","last_fullsized":"2023-11-24T08:21:52.585287+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:26.855786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583641+0000","last_change":"2023-11-24T08:21:51.592472+0000","last_active":"2023-11-24T08:21:52.583641+0000","last_peered":"2023-11-24T08:21:52.583641+0000","last_clean":"2023-11-24T08:21:52.583641+0000","last_became_active":"2023-11-24T08:21:51.592136+0000","last_became_peered":"2023-11-24T08:21:51.592136+0000","last_unstale":"2023-11-24T08:21:52.583641+0000","last_undegraded":"2023-11-24T08:21:52.583641+0000","last_fullsized":"2023-11-24T08:21:52.583641+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:00:55.474780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586279+0000","last_change":"2023-11-24T08:21:51.593723+0000","last_active":"2023-11-24T08:21:52.586279+0000","last_peered":"2023-11-24T08:21:52.586279+0000","last_clean":"2023-11-24T08:21:52.586279+0000","last_became_active":"2023-11-24T08:21:51.589852+0000","last_became_peered":"2023-11-24T08:21:51.589852+0000","last_unstale":"2023-11-24T08:21:52.586279+0000","last_undegraded":"2023-11-24T08:21:52.586279+0000","last_fullsized":"2023-11-24T08:21:52.586279+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:36.366997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422698+0000","last_change":"2023-11-24T08:21:51.575331+0000","last_active":"2023-11-24T08:21:53.422698+0000","last_peered":"2023-11-24T08:21:53.422698+0000","last_clean":"2023-11-24T08:21:53.422698+0000","last_became_active":"2023-11-24T08:21:51.575033+0000","last_became_peered":"2023-11-24T08:21:51.575033+0000","last_unstale":"2023-11-24T08:21:53.422698+0000","last_undegraded":"2023-11-24T08:21:53.422698+0000","last_fullsized":"2023-11-24T08:21:53.422698+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:30.282265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583041+0000","last_change":"2023-11-24T08:21:51.582490+0000","last_active":"2023-11-24T08:21:52.583041+0000","last_peered":"2023-11-24T08:21:52.583041+0000","last_clean":"2023-11-24T08:21:52.583041+0000","last_became_active":"2023-11-24T08:21:51.578823+0000","last_became_peered":"2023-11-24T08:21:51.578823+0000","last_unstale":"2023-11-24T08:21:52.583041+0000","last_undegraded":"2023-11-24T08:21:52.583041+0000","last_fullsized":"2023-11-24T08:21:52.583041+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:27.009420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587190+0000","last_change":"2023-11-24T08:21:51.597212+0000","last_active":"2023-11-24T08:21:52.587190+0000","last_peered":"2023-11-24T08:21:52.587190+0000","last_clean":"2023-11-24T08:21:52.587190+0000","last_became_active":"2023-11-24T08:21:51.596627+0000","last_became_peered":"2023-11-24T08:21:51.596627+0000","last_unstale":"2023-11-24T08:21:52.587190+0000","last_undegraded":"2023-11-24T08:21:52.587190+0000","last_fullsized":"2023-11-24T08:21:52.587190+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:39:52.476877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.587872+0000","last_change":"2023-11-24T08:21:51.585577+0000","last_active":"2023-11-24T08:21:51.587872+0000","last_peered":"2023-11-24T08:21:51.587872+0000","last_clean":"2023-11-24T08:21:51.587872+0000","last_became_active":"2023-11-24T08:21:51.585287+0000","last_became_peered":"2023-11-24T08:21:51.585287+0000","last_unstale":"2023-11-24T08:21:51.587872+0000","last_undegraded":"2023-11-24T08:21:51.587872+0000","last_fullsized":"2023-11-24T08:21:51.587872+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:20.042283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598428+0000","last_change":"2023-11-24T08:21:51.589152+0000","last_active":"2023-11-24T08:21:51.598428+0000","last_peered":"2023-11-24T08:21:51.598428+0000","last_clean":"2023-11-24T08:21:51.598428+0000","last_became_active":"2023-11-24T08:21:51.588665+0000","last_became_peered":"2023-11-24T08:21:51.588665+0000","last_unstale":"2023-11-24T08:21:51.598428+0000","last_undegraded":"2023-11-24T08:21:51.598428+0000","last_fullsized":"2023-11-24T08:21:51.598428+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:10.411920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584250+0000","last_change":"2023-11-24T08:21:51.581669+0000","last_active":"2023-11-24T08:21:52.584250+0000","last_peered":"2023-11-24T08:21:52.584250+0000","last_clean":"2023-11-24T08:21:52.584250+0000","last_became_active":"2023-11-24T08:21:51.581333+0000","last_became_peered":"2023-11-24T08:21:51.581333+0000","last_unstale":"2023-11-24T08:21:52.584250+0000","last_undegraded":"2023-11-24T08:21:52.584250+0000","last_fullsized":"2023-11-24T08:21:52.584250+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:40.399989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576386+0000","last_change":"2023-11-24T08:21:51.572575+0000","last_active":"2023-11-24T08:21:51.576386+0000","last_peered":"2023-11-24T08:21:51.576386+0000","last_clean":"2023-11-24T08:21:51.576386+0000","last_became_active":"2023-11-24T08:21:51.572322+0000","last_became_peered":"2023-11-24T08:21:51.572322+0000","last_unstale":"2023-11-24T08:21:51.576386+0000","last_undegraded":"2023-11-24T08:21:51.576386+0000","last_fullsized":"2023-11-24T08:21:51.576386+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:01.619958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422851+0000","last_change":"2023-11-24T08:21:51.579005+0000","last_active":"2023-11-24T08:21:53.422851+0000","last_peered":"2023-11-24T08:21:53.422851+0000","last_clean":"2023-11-24T08:21:53.422851+0000","last_became_active":"2023-11-24T08:21:51.578676+0000","last_became_peered":"2023-11-24T08:21:51.578676+0000","last_unstale":"2023-11-24T08:21:53.422851+0000","last_undegraded":"2023-11-24T08:21:53.422851+0000","last_fullsized":"2023-11-24T08:21:53.422851+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:53.955792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576713+0000","last_change":"2023-11-24T08:21:51.568542+0000","last_active":"2023-11-24T08:21:51.576713+0000","last_peered":"2023-11-24T08:21:51.576713+0000","last_clean":"2023-11-24T08:21:51.576713+0000","last_became_active":"2023-11-24T08:21:51.568270+0000","last_became_peered":"2023-11-24T08:21:51.568270+0000","last_unstale":"2023-11-24T08:21:51.576713+0000","last_undegraded":"2023-11-24T08:21:51.576713+0000","last_fullsized":"2023-11-24T08:21:51.576713+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:28.009675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.585091+0000","last_change":"2023-11-24T08:21:51.576721+0000","last_active":"2023-11-24T08:21:51.585091+0000","last_peered":"2023-11-24T08:21:51.585091+0000","last_clean":"2023-11-24T08:21:51.585091+0000","last_became_active":"2023-11-24T08:21:51.576411+0000","last_became_peered":"2023-11-24T08:21:51.576411+0000","last_unstale":"2023-11-24T08:21:51.585091+0000","last_undegraded":"2023-11-24T08:21:51.585091+0000","last_fullsized":"2023-11-24T08:21:51.585091+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:59.195803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.597345+0000","last_change":"2023-11-24T08:21:51.586077+0000","last_active":"2023-11-24T08:21:51.597345+0000","last_peered":"2023-11-24T08:21:51.597345+0000","last_clean":"2023-11-24T08:21:51.597345+0000","last_became_active":"2023-11-24T08:21:51.583309+0000","last_became_peered":"2023-11-24T08:21:51.583309+0000","last_unstale":"2023-11-24T08:21:51.597345+0000","last_undegraded":"2023-11-24T08:21:51.597345+0000","last_fullsized":"2023-11-24T08:21:51.597345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:21.698335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.595218+0000","last_change":"2023-11-24T08:21:51.594658+0000","last_active":"2023-11-24T08:21:51.595218+0000","last_peered":"2023-11-24T08:21:51.595218+0000","last_clean":"2023-11-24T08:21:51.595218+0000","last_became_active":"2023-11-24T08:21:51.594157+0000","last_became_peered":"2023-11-24T08:21:51.594157+0000","last_unstale":"2023-11-24T08:21:51.595218+0000","last_undegraded":"2023-11-24T08:21:51.595218+0000","last_fullsized":"2023-11-24T08:21:51.595218+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:35:54.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.596190+0000","last_change":"2023-11-24T08:21:51.578705+0000","last_active":"2023-11-24T08:21:51.596190+0000","last_peered":"2023-11-24T08:21:51.596190+0000","last_clean":"2023-11-24T08:21:51.596190+0000","last_became_active":"2023-11-24T08:21:51.578423+0000","last_became_peered":"2023-11-24T08:21:51.578423+0000","last_unstale":"2023-11-24T08:21:51.596190+0000","last_undegraded":"2023-11-24T08:21:51.596190+0000","last_fullsized":"2023-11-24T08:21:51.596190+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:44.243887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585569+0000","last_change":"2023-11-24T08:21:51.594477+0000","last_active":"2023-11-24T08:21:52.585569+0000","last_peered":"2023-11-24T08:21:52.585569+0000","last_clean":"2023-11-24T08:21:52.585569+0000","last_became_active":"2023-11-24T08:21:51.594101+0000","last_became_peered":"2023-11-24T08:21:51.594101+0000","last_unstale":"2023-11-24T08:21:52.585569+0000","last_undegraded":"2023-11-24T08:21:52.585569+0000","last_fullsized":"2023-11-24T08:21:52.585569+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:55:47.728601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588928+0000","last_change":"2023-11-24T08:21:51.582719+0000","last_active":"2023-11-24T08:21:51.588928+0000","last_peered":"2023-11-24T08:21:51.588928+0000","last_clean":"2023-11-24T08:21:51.588928+0000","last_became_active":"2023-11-24T08:21:51.582435+0000","last_became_peered":"2023-11-24T08:21:51.582435+0000","last_unstale":"2023-11-24T08:21:51.588928+0000","last_undegraded":"2023-11-24T08:21:51.588928+0000","last_fullsized":"2023-11-24T08:21:51.588928+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:30.346032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583720+0000","last_change":"2023-11-24T08:21:51.598869+0000","last_active":"2023-11-24T08:21:52.583720+0000","last_peered":"2023-11-24T08:21:52.583720+0000","last_clean":"2023-11-24T08:21:52.583720+0000","last_became_active":"2023-11-24T08:21:51.598598+0000","last_became_peered":"2023-11-24T08:21:51.598598+0000","last_unstale":"2023-11-24T08:21:52.583720+0000","last_undegraded":"2023-11-24T08:21:52.583720+0000","last_fullsized":"2023-11-24T08:21:52.583720+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:30.282265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576735+0000","last_change":"2023-11-24T08:21:51.571213+0000","last_active":"2023-11-24T08:21:51.576735+0000","last_peered":"2023-11-24T08:21:51.576735+0000","last_clean":"2023-11-24T08:21:51.576735+0000","last_became_active":"2023-11-24T08:21:51.570835+0000","last_became_peered":"2023-11-24T08:21:51.570835+0000","last_unstale":"2023-11-24T08:21:51.576735+0000","last_undegraded":"2023-11-24T08:21:51.576735+0000","last_fullsized":"2023-11-24T08:21:51.576735+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:19:59.446051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583423+0000","last_change":"2023-11-24T08:21:51.596009+0000","last_active":"2023-11-24T08:21:52.583423+0000","last_peered":"2023-11-24T08:21:52.583423+0000","last_clean":"2023-11-24T08:21:52.583423+0000","last_became_active":"2023-11-24T08:21:51.595710+0000","last_became_peered":"2023-11-24T08:21:51.595710+0000","last_unstale":"2023-11-24T08:21:52.583423+0000","last_undegraded":"2023-11-24T08:21:52.583423+0000","last_fullsized":"2023-11-24T08:21:52.583423+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:16.476510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589125+0000","last_change":"2023-11-24T08:21:51.587432+0000","last_active":"2023-11-24T08:21:51.589125+0000","last_peered":"2023-11-24T08:21:51.589125+0000","last_clean":"2023-11-24T08:21:51.589125+0000","last_became_active":"2023-11-24T08:21:51.587146+0000","last_became_peered":"2023-11-24T08:21:51.587146+0000","last_unstale":"2023-11-24T08:21:51.589125+0000","last_undegraded":"2023-11-24T08:21:51.589125+0000","last_fullsized":"2023-11-24T08:21:51.589125+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:22.114873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"206'4","reported_seq":15,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598591+0000","last_change":"2023-11-24T08:21:50.580627+0000","last_active":"2023-11-24T08:21:51.598591+0000","last_peered":"2023-11-24T08:21:51.598591+0000","last_clean":"2023-11-24T08:21:51.598591+0000","last_became_active":"2023-11-24T08:21:50.580329+0000","last_became_peered":"2023-11-24T08:21:50.580329+0000","last_unstale":"2023-11-24T08:21:51.598591+0000","last_undegraded":"2023-11-24T08:21:51.598591+0000","last_fullsized":"2023-11-24T08:21:51.598591+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:34.438903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576137+0000","last_change":"2023-11-24T08:21:51.572049+0000","last_active":"2023-11-24T08:21:51.576137+0000","last_peered":"2023-11-24T08:21:51.576137+0000","last_clean":"2023-11-24T08:21:51.576137+0000","last_became_active":"2023-11-24T08:21:51.571793+0000","last_became_peered":"2023-11-24T08:21:51.571793+0000","last_unstale":"2023-11-24T08:21:51.576137+0000","last_undegraded":"2023-11-24T08:21:51.576137+0000","last_fullsized":"2023-11-24T08:21:51.576137+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:24.923787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587569+0000","last_change":"2023-11-24T08:21:50.587199+0000","last_active":"2023-11-24T08:21:52.587569+0000","last_peered":"2023-11-24T08:21:52.587569+0000","last_clean":"2023-11-24T08:21:52.587569+0000","last_became_active":"2023-11-24T08:21:50.586820+0000","last_became_peered":"2023-11-24T08:21:50.586820+0000","last_unstale":"2023-11-24T08:21:52.587569+0000","last_undegraded":"2023-11-24T08:21:52.587569+0000","last_fullsized":"2023-11-24T08:21:52.587569+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:31:31.292020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587560+0000","last_change":"2023-11-24T08:21:51.598051+0000","last_active":"2023-11-24T08:21:52.587560+0000","last_peered":"2023-11-24T08:21:52.587560+0000","last_clean":"2023-11-24T08:21:52.587560+0000","last_became_active":"2023-11-24T08:21:51.597671+0000","last_became_peered":"2023-11-24T08:21:51.597671+0000","last_unstale":"2023-11-24T08:21:52.587560+0000","last_undegraded":"2023-11-24T08:21:52.587560+0000","last_fullsized":"2023-11-24T08:21:52.587560+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:30.346032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584588+0000","last_change":"2023-11-24T08:21:50.587346+0000","last_active":"2023-11-24T08:21:52.584588+0000","last_peered":"2023-11-24T08:21:52.584588+0000","last_clean":"2023-11-24T08:21:52.584588+0000","last_became_active":"2023-11-24T08:21:50.587054+0000","last_became_peered":"2023-11-24T08:21:50.587054+0000","last_unstale":"2023-11-24T08:21:52.584588+0000","last_undegraded":"2023-11-24T08:21:52.584588+0000","last_fullsized":"2023-11-24T08:21:52.584588+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:47.268620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581612+0000","last_change":"2023-11-24T08:21:51.579104+0000","last_active":"2023-11-24T08:21:51.581612+0000","last_peered":"2023-11-24T08:21:51.581612+0000","last_clean":"2023-11-24T08:21:51.581612+0000","last_became_active":"2023-11-24T08:21:51.578833+0000","last_became_peered":"2023-11-24T08:21:51.578833+0000","last_unstale":"2023-11-24T08:21:51.581612+0000","last_undegraded":"2023-11-24T08:21:51.581612+0000","last_fullsized":"2023-11-24T08:21:51.581612+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:44.218789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423056+0000","last_change":"2023-11-24T08:21:50.585753+0000","last_active":"2023-11-24T08:21:53.423056+0000","last_peered":"2023-11-24T08:21:53.423056+0000","last_clean":"2023-11-24T08:21:53.423056+0000","last_became_active":"2023-11-24T08:21:50.585515+0000","last_became_peered":"2023-11-24T08:21:50.585515+0000","last_unstale":"2023-11-24T08:21:53.423056+0000","last_undegraded":"2023-11-24T08:21:53.423056+0000","last_fullsized":"2023-11-24T08:21:53.423056+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:47.268620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586073+0000","last_change":"2023-11-24T08:21:51.594807+0000","last_active":"2023-11-24T08:21:52.586073+0000","last_peered":"2023-11-24T08:21:52.586073+0000","last_clean":"2023-11-24T08:21:52.586073+0000","last_became_active":"2023-11-24T08:21:51.594122+0000","last_became_peered":"2023-11-24T08:21:51.594122+0000","last_unstale":"2023-11-24T08:21:52.586073+0000","last_undegraded":"2023-11-24T08:21:52.586073+0000","last_fullsized":"2023-11-24T08:21:52.586073+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:06.621548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"206'1","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594306+0000","last_change":"2023-11-24T08:21:50.587203+0000","last_active":"2023-11-24T08:21:51.594306+0000","last_peered":"2023-11-24T08:21:51.594306+0000","last_clean":"2023-11-24T08:21:51.594306+0000","last_became_active":"2023-11-24T08:21:50.586924+0000","last_became_peered":"2023-11-24T08:21:50.586924+0000","last_unstale":"2023-11-24T08:21:51.594306+0000","last_undegraded":"2023-11-24T08:21:51.594306+0000","last_fullsized":"2023-11-24T08:21:51.594306+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:48.388108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422916+0000","last_change":"2023-11-24T08:21:51.579729+0000","last_active":"2023-11-24T08:21:53.422916+0000","last_peered":"2023-11-24T08:21:53.422916+0000","last_clean":"2023-11-24T08:21:53.422916+0000","last_became_active":"2023-11-24T08:21:51.579463+0000","last_became_peered":"2023-11-24T08:21:51.579463+0000","last_unstale":"2023-11-24T08:21:53.422916+0000","last_undegraded":"2023-11-24T08:21:53.422916+0000","last_fullsized":"2023-11-24T08:21:53.422916+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:40.399989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.563788+0000","last_change":"2023-11-24T08:21:50.585026+0000","last_active":"2023-11-24T08:21:51.563788+0000","last_peered":"2023-11-24T08:21:51.563788+0000","last_clean":"2023-11-24T08:21:51.563788+0000","last_became_active":"2023-11-24T08:21:50.584718+0000","last_became_peered":"2023-11-24T08:21:50.584718+0000","last_unstale":"2023-11-24T08:21:51.563788+0000","last_undegraded":"2023-11-24T08:21:51.563788+0000","last_fullsized":"2023-11-24T08:21:51.563788+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:11:33.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589480+0000","last_change":"2023-11-24T08:21:51.585963+0000","last_active":"2023-11-24T08:21:51.589480+0000","last_peered":"2023-11-24T08:21:51.589480+0000","last_clean":"2023-11-24T08:21:51.589480+0000","last_became_active":"2023-11-24T08:21:51.585718+0000","last_became_peered":"2023-11-24T08:21:51.585718+0000","last_unstale":"2023-11-24T08:21:51.589480+0000","last_undegraded":"2023-11-24T08:21:51.589480+0000","last_fullsized":"2023-11-24T08:21:51.589480+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:36.410622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.564062+0000","last_change":"2023-11-24T08:21:50.571703+0000","last_active":"2023-11-24T08:21:51.564062+0000","last_peered":"2023-11-24T08:21:51.564062+0000","last_clean":"2023-11-24T08:21:51.564062+0000","last_became_active":"2023-11-24T08:21:50.571411+0000","last_became_peered":"2023-11-24T08:21:50.571411+0000","last_unstale":"2023-11-24T08:21:51.564062+0000","last_undegraded":"2023-11-24T08:21:51.564062+0000","last_fullsized":"2023-11-24T08:21:51.564062+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:11:41.187561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423129+0000","last_change":"2023-11-24T08:21:51.588944+0000","last_active":"2023-11-24T08:21:53.423129+0000","last_peered":"2023-11-24T08:21:53.423129+0000","last_clean":"2023-11-24T08:21:53.423129+0000","last_became_active":"2023-11-24T08:21:51.588628+0000","last_became_peered":"2023-11-24T08:21:51.588628+0000","last_unstale":"2023-11-24T08:21:53.423129+0000","last_undegraded":"2023-11-24T08:21:53.423129+0000","last_fullsized":"2023-11-24T08:21:53.423129+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:23.995739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.575396+0000","last_change":"2023-11-24T08:21:50.579938+0000","last_active":"2023-11-24T08:21:51.575396+0000","last_peered":"2023-11-24T08:21:51.575396+0000","last_clean":"2023-11-24T08:21:51.575396+0000","last_became_active":"2023-11-24T08:21:50.578966+0000","last_became_peered":"2023-11-24T08:21:50.578966+0000","last_unstale":"2023-11-24T08:21:51.575396+0000","last_undegraded":"2023-11-24T08:21:51.575396+0000","last_fullsized":"2023-11-24T08:21:51.575396+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:36.912515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423699+0000","last_change":"2023-11-24T08:21:51.571316+0000","last_active":"2023-11-24T08:21:53.423699+0000","last_peered":"2023-11-24T08:21:53.423699+0000","last_clean":"2023-11-24T08:21:53.423699+0000","last_became_active":"2023-11-24T08:21:51.571020+0000","last_became_peered":"2023-11-24T08:21:51.571020+0000","last_unstale":"2023-11-24T08:21:53.423699+0000","last_undegraded":"2023-11-24T08:21:53.423699+0000","last_fullsized":"2023-11-24T08:21:53.423699+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:42.189432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.563211+0000","last_change":"2023-11-24T08:21:50.584651+0000","last_active":"2023-11-24T08:21:51.563211+0000","last_peered":"2023-11-24T08:21:51.563211+0000","last_clean":"2023-11-24T08:21:51.563211+0000","last_became_active":"2023-11-24T08:21:50.584354+0000","last_became_peered":"2023-11-24T08:21:50.584354+0000","last_unstale":"2023-11-24T08:21:51.563211+0000","last_undegraded":"2023-11-24T08:21:51.563211+0000","last_fullsized":"2023-11-24T08:21:51.563211+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:01:49.356407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.580413+0000","last_change":"2023-11-24T08:21:51.573829+0000","last_active":"2023-11-24T08:21:51.580413+0000","last_peered":"2023-11-24T08:21:51.580413+0000","last_clean":"2023-11-24T08:21:51.580413+0000","last_became_active":"2023-11-24T08:21:51.573562+0000","last_became_peered":"2023-11-24T08:21:51.573562+0000","last_unstale":"2023-11-24T08:21:51.580413+0000","last_undegraded":"2023-11-24T08:21:51.580413+0000","last_fullsized":"2023-11-24T08:21:51.580413+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:35:40.189115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"206'3","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598530+0000","last_change":"2023-11-24T08:21:50.583516+0000","last_active":"2023-11-24T08:21:51.598530+0000","last_peered":"2023-11-24T08:21:51.598530+0000","last_clean":"2023-11-24T08:21:51.598530+0000","last_became_active":"2023-11-24T08:21:50.582753+0000","last_became_peered":"2023-11-24T08:21:50.582753+0000","last_unstale":"2023-11-24T08:21:51.598530+0000","last_undegraded":"2023-11-24T08:21:51.598530+0000","last_fullsized":"2023-11-24T08:21:51.598530+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:28:53.332475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588536+0000","last_change":"2023-11-24T08:21:51.576091+0000","last_active":"2023-11-24T08:21:51.588536+0000","last_peered":"2023-11-24T08:21:51.588536+0000","last_clean":"2023-11-24T08:21:51.588536+0000","last_became_active":"2023-11-24T08:21:51.575740+0000","last_became_peered":"2023-11-24T08:21:51.575740+0000","last_unstale":"2023-11-24T08:21:51.588536+0000","last_undegraded":"2023-11-24T08:21:51.588536+0000","last_fullsized":"2023-11-24T08:21:51.588536+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.319836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"206'6","reported_seq":18,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584193+0000","last_change":"2023-11-24T08:21:50.589155+0000","last_active":"2023-11-24T08:21:52.584193+0000","last_peered":"2023-11-24T08:21:52.584193+0000","last_clean":"2023-11-24T08:21:52.584193+0000","last_became_active":"2023-11-24T08:21:50.588733+0000","last_became_peered":"2023-11-24T08:21:50.588733+0000","last_unstale":"2023-11-24T08:21:52.584193+0000","last_undegraded":"2023-11-24T08:21:52.584193+0000","last_fullsized":"2023-11-24T08:21:52.584193+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:29.579582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.590849+0000","last_change":"2023-11-24T08:21:51.588326+0000","last_active":"2023-11-24T08:21:51.590849+0000","last_peered":"2023-11-24T08:21:51.590849+0000","last_clean":"2023-11-24T08:21:51.590849+0000","last_became_active":"2023-11-24T08:21:51.588025+0000","last_became_peered":"2023-11-24T08:21:51.588025+0000","last_unstale":"2023-11-24T08:21:51.590849+0000","last_undegraded":"2023-11-24T08:21:51.590849+0000","last_fullsized":"2023-11-24T08:21:51.590849+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:44.396426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587076+0000","last_change":"2023-11-24T08:21:50.586145+0000","last_active":"2023-11-24T08:21:52.587076+0000","last_peered":"2023-11-24T08:21:52.587076+0000","last_clean":"2023-11-24T08:21:52.587076+0000","last_became_active":"2023-11-24T08:21:50.583351+0000","last_became_peered":"2023-11-24T08:21:50.583351+0000","last_unstale":"2023-11-24T08:21:52.587076+0000","last_undegraded":"2023-11-24T08:21:52.587076+0000","last_fullsized":"2023-11-24T08:21:52.587076+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:24.796699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587303+0000","last_change":"2023-11-24T08:21:51.593188+0000","last_active":"2023-11-24T08:21:52.587303+0000","last_peered":"2023-11-24T08:21:52.587303+0000","last_clean":"2023-11-24T08:21:52.587303+0000","last_became_active":"2023-11-24T08:21:51.592808+0000","last_became_peered":"2023-11-24T08:21:51.592808+0000","last_unstale":"2023-11-24T08:21:52.587303+0000","last_undegraded":"2023-11-24T08:21:52.587303+0000","last_fullsized":"2023-11-24T08:21:52.587303+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:59.195803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.568391+0000","last_change":"2023-11-24T08:21:50.564949+0000","last_active":"2023-11-24T08:21:51.568391+0000","last_peered":"2023-11-24T08:21:51.568391+0000","last_clean":"2023-11-24T08:21:51.568391+0000","last_became_active":"2023-11-24T08:21:50.564690+0000","last_became_peered":"2023-11-24T08:21:50.564690+0000","last_unstale":"2023-11-24T08:21:51.568391+0000","last_undegraded":"2023-11-24T08:21:51.568391+0000","last_fullsized":"2023-11-24T08:21:51.568391+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:57.160663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.580670+0000","last_change":"2023-11-24T08:21:51.574274+0000","last_active":"2023-11-24T08:21:51.580670+0000","last_peered":"2023-11-24T08:21:51.580670+0000","last_clean":"2023-11-24T08:21:51.580670+0000","last_became_active":"2023-11-24T08:21:51.573985+0000","last_became_peered":"2023-11-24T08:21:51.573985+0000","last_unstale":"2023-11-24T08:21:51.580670+0000","last_undegraded":"2023-11-24T08:21:51.580670+0000","last_fullsized":"2023-11-24T08:21:51.580670+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:26:38.263958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.582926+0000","last_change":"2023-11-24T08:21:50.576886+0000","last_active":"2023-11-24T08:21:52.582926+0000","last_peered":"2023-11-24T08:21:52.582926+0000","last_clean":"2023-11-24T08:21:52.582926+0000","last_became_active":"2023-11-24T08:21:50.576591+0000","last_became_peered":"2023-11-24T08:21:50.576591+0000","last_unstale":"2023-11-24T08:21:52.582926+0000","last_undegraded":"2023-11-24T08:21:52.582926+0000","last_fullsized":"2023-11-24T08:21:52.582926+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:54.522055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423168+0000","last_change":"2023-11-24T08:21:51.574026+0000","last_active":"2023-11-24T08:21:53.423168+0000","last_peered":"2023-11-24T08:21:53.423168+0000","last_clean":"2023-11-24T08:21:53.423168+0000","last_became_active":"2023-11-24T08:21:51.573776+0000","last_became_peered":"2023-11-24T08:21:51.573776+0000","last_unstale":"2023-11-24T08:21:53.423168+0000","last_undegraded":"2023-11-24T08:21:53.423168+0000","last_fullsized":"2023-11-24T08:21:53.423168+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.811886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"206'8","reported_seq":20,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585430+0000","last_change":"2023-11-24T08:21:50.587706+0000","last_active":"2023-11-24T08:21:52.585430+0000","last_peered":"2023-11-24T08:21:52.585430+0000","last_clean":"2023-11-24T08:21:52.585430+0000","last_became_active":"2023-11-24T08:21:50.587370+0000","last_became_peered":"2023-11-24T08:21:50.587370+0000","last_unstale":"2023-11-24T08:21:52.585430+0000","last_undegraded":"2023-11-24T08:21:52.585430+0000","last_fullsized":"2023-11-24T08:21:52.585430+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:49.795187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.595541+0000","last_change":"2023-11-24T08:21:51.592785+0000","last_active":"2023-11-24T08:21:51.595541+0000","last_peered":"2023-11-24T08:21:51.595541+0000","last_clean":"2023-11-24T08:21:51.595541+0000","last_became_active":"2023-11-24T08:21:51.592504+0000","last_became_peered":"2023-11-24T08:21:51.592504+0000","last_unstale":"2023-11-24T08:21:51.595541+0000","last_undegraded":"2023-11-24T08:21:51.595541+0000","last_fullsized":"2023-11-24T08:21:51.595541+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:10.411920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"206'1","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.577493+0000","last_change":"2023-11-24T08:21:50.584442+0000","last_active":"2023-11-24T08:21:51.577493+0000","last_peered":"2023-11-24T08:21:51.577493+0000","last_clean":"2023-11-24T08:21:51.577493+0000","last_became_active":"2023-11-24T08:21:50.584093+0000","last_became_peered":"2023-11-24T08:21:50.584093+0000","last_unstale":"2023-11-24T08:21:51.577493+0000","last_undegraded":"2023-11-24T08:21:51.577493+0000","last_fullsized":"2023-11-24T08:21:51.577493+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.242866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.601662+0000","last_change":"2023-11-24T08:21:51.599031+0000","last_active":"2023-11-24T08:21:51.601662+0000","last_peered":"2023-11-24T08:21:51.601662+0000","last_clean":"2023-11-24T08:21:51.601662+0000","last_became_active":"2023-11-24T08:21:51.598747+0000","last_became_peered":"2023-11-24T08:21:51.598747+0000","last_unstale":"2023-11-24T08:21:51.601662+0000","last_undegraded":"2023-11-24T08:21:51.601662+0000","last_fullsized":"2023-11-24T08:21:51.601662+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:58.177362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"206'2","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.587864+0000","last_change":"2023-11-24T08:21:50.570680+0000","last_active":"2023-11-24T08:21:51.587864+0000","last_peered":"2023-11-24T08:21:51.587864+0000","last_clean":"2023-11-24T08:21:51.587864+0000","last_became_active":"2023-11-24T08:21:50.570430+0000","last_became_peered":"2023-11-24T08:21:50.570430+0000","last_unstale":"2023-11-24T08:21:51.587864+0000","last_undegraded":"2023-11-24T08:21:51.587864+0000","last_fullsized":"2023-11-24T08:21:51.587864+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:28.237795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581312+0000","last_change":"2023-11-24T08:21:51.577355+0000","last_active":"2023-11-24T08:21:51.581312+0000","last_peered":"2023-11-24T08:21:51.581312+0000","last_clean":"2023-11-24T08:21:51.581312+0000","last_became_active":"2023-11-24T08:21:51.577071+0000","last_became_peered":"2023-11-24T08:21:51.577071+0000","last_unstale":"2023-11-24T08:21:51.581312+0000","last_undegraded":"2023-11-24T08:21:51.581312+0000","last_fullsized":"2023-11-24T08:21:51.581312+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:01:35.877808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586831+0000","last_change":"2023-11-24T08:21:50.582670+0000","last_active":"2023-11-24T08:21:52.586831+0000","last_peered":"2023-11-24T08:21:52.586831+0000","last_clean":"2023-11-24T08:21:52.586831+0000","last_became_active":"2023-11-24T08:21:50.582396+0000","last_became_peered":"2023-11-24T08:21:50.582396+0000","last_unstale":"2023-11-24T08:21:52.586831+0000","last_undegraded":"2023-11-24T08:21:52.586831+0000","last_fullsized":"2023-11-24T08:21:52.586831+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:57:57.084338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587345+0000","last_change":"2023-11-24T08:21:51.604541+0000","last_active":"2023-11-24T08:21:52.587345+0000","last_peered":"2023-11-24T08:21:52.587345+0000","last_clean":"2023-11-24T08:21:52.587345+0000","last_became_active":"2023-11-24T08:21:51.604123+0000","last_became_peered":"2023-11-24T08:21:51.604123+0000","last_unstale":"2023-11-24T08:21:52.587345+0000","last_undegraded":"2023-11-24T08:21:52.587345+0000","last_fullsized":"2023-11-24T08:21:52.587345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:41:22.205969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585418+0000","last_change":"2023-11-24T08:21:50.584277+0000","last_active":"2023-11-24T08:21:52.585418+0000","last_peered":"2023-11-24T08:21:52.585418+0000","last_clean":"2023-11-24T08:21:52.585418+0000","last_became_active":"2023-11-24T08:21:50.583971+0000","last_became_peered":"2023-11-24T08:21:50.583971+0000","last_unstale":"2023-11-24T08:21:52.585418+0000","last_undegraded":"2023-11-24T08:21:52.585418+0000","last_fullsized":"2023-11-24T08:21:52.585418+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:36:03.042901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585158+0000","last_change":"2023-11-24T08:21:51.590865+0000","last_active":"2023-11-24T08:21:52.585158+0000","last_peered":"2023-11-24T08:21:52.585158+0000","last_clean":"2023-11-24T08:21:52.585158+0000","last_became_active":"2023-11-24T08:21:51.589755+0000","last_became_peered":"2023-11-24T08:21:51.589755+0000","last_unstale":"2023-11-24T08:21:52.585158+0000","last_undegraded":"2023-11-24T08:21:52.585158+0000","last_fullsized":"2023-11-24T08:21:52.585158+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:21:33.418895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587331+0000","last_change":"2023-11-24T08:21:50.583979+0000","last_active":"2023-11-24T08:21:52.587331+0000","last_peered":"2023-11-24T08:21:52.587331+0000","last_clean":"2023-11-24T08:21:52.587331+0000","last_became_active":"2023-11-24T08:21:50.583017+0000","last_became_peered":"2023-11-24T08:21:50.583017+0000","last_unstale":"2023-11-24T08:21:52.587331+0000","last_undegraded":"2023-11-24T08:21:52.587331+0000","last_fullsized":"2023-11-24T08:21:52.587331+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:02.853886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594548+0000","last_change":"2023-11-24T08:21:51.584499+0000","last_active":"2023-11-24T08:21:51.594548+0000","last_peered":"2023-11-24T08:21:51.594548+0000","last_clean":"2023-11-24T08:21:51.594548+0000","last_became_active":"2023-11-24T08:21:51.583368+0000","last_became_peered":"2023-11-24T08:21:51.583368+0000","last_unstale":"2023-11-24T08:21:51.594548+0000","last_undegraded":"2023-11-24T08:21:51.594548+0000","last_fullsized":"2023-11-24T08:21:51.594548+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:23.943409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423085+0000","last_change":"2023-11-24T08:21:50.575845+0000","last_active":"2023-11-24T08:21:53.423085+0000","last_peered":"2023-11-24T08:21:53.423085+0000","last_clean":"2023-11-24T08:21:53.423085+0000","last_became_active":"2023-11-24T08:21:50.575535+0000","last_became_peered":"2023-11-24T08:21:50.575535+0000","last_unstale":"2023-11-24T08:21:53.423085+0000","last_undegraded":"2023-11-24T08:21:53.423085+0000","last_fullsized":"2023-11-24T08:21:53.423085+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:54.522055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594906+0000","last_change":"2023-11-24T08:21:51.591447+0000","last_active":"2023-11-24T08:21:51.594906+0000","last_peered":"2023-11-24T08:21:51.594906+0000","last_clean":"2023-11-24T08:21:51.594906+0000","last_became_active":"2023-11-24T08:21:51.590758+0000","last_became_peered":"2023-11-24T08:21:51.590758+0000","last_unstale":"2023-11-24T08:21:51.594906+0000","last_undegraded":"2023-11-24T08:21:51.594906+0000","last_fullsized":"2023-11-24T08:21:51.594906+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:49.404807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422778+0000","last_change":"2023-11-24T08:21:50.565008+0000","last_active":"2023-11-24T08:21:53.422778+0000","last_peered":"2023-11-24T08:21:53.422778+0000","last_clean":"2023-11-24T08:21:53.422778+0000","last_became_active":"2023-11-24T08:21:50.564591+0000","last_became_peered":"2023-11-24T08:21:50.564591+0000","last_unstale":"2023-11-24T08:21:53.422778+0000","last_undegraded":"2023-11-24T08:21:53.422778+0000","last_fullsized":"2023-11-24T08:21:53.422778+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:20.980206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.592165+0000","last_change":"2023-11-24T08:21:51.590026+0000","last_active":"2023-11-24T08:21:51.592165+0000","last_peered":"2023-11-24T08:21:51.592165+0000","last_clean":"2023-11-24T08:21:51.592165+0000","last_became_active":"2023-11-24T08:21:51.589768+0000","last_became_peered":"2023-11-24T08:21:51.589768+0000","last_unstale":"2023-11-24T08:21:51.592165+0000","last_undegraded":"2023-11-24T08:21:51.592165+0000","last_fullsized":"2023-11-24T08:21:51.592165+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:18.731436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.555060+0000","last_change":"2023-11-24T08:21:50.571240+0000","last_active":"2023-11-24T08:21:51.555060+0000","last_peered":"2023-11-24T08:21:51.555060+0000","last_clean":"2023-11-24T08:21:51.555060+0000","last_became_active":"2023-11-24T08:21:50.570965+0000","last_became_peered":"2023-11-24T08:21:50.570965+0000","last_unstale":"2023-11-24T08:21:51.555060+0000","last_undegraded":"2023-11-24T08:21:51.555060+0000","last_fullsized":"2023-11-24T08:21:51.555060+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:11:32.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584492+0000","last_change":"2023-11-24T08:21:51.580901+0000","last_active":"2023-11-24T08:21:51.584492+0000","last_peered":"2023-11-24T08:21:51.584492+0000","last_clean":"2023-11-24T08:21:51.584492+0000","last_became_active":"2023-11-24T08:21:51.580559+0000","last_became_peered":"2023-11-24T08:21:51.580559+0000","last_unstale":"2023-11-24T08:21:51.584492+0000","last_undegraded":"2023-11-24T08:21:51.584492+0000","last_fullsized":"2023-11-24T08:21:51.584492+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:21.698335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587627+0000","last_change":"2023-11-24T08:21:50.583636+0000","last_active":"2023-11-24T08:21:52.587627+0000","last_peered":"2023-11-24T08:21:52.587627+0000","last_clean":"2023-11-24T08:21:52.587627+0000","last_became_active":"2023-11-24T08:21:50.583281+0000","last_became_peered":"2023-11-24T08:21:50.583281+0000","last_unstale":"2023-11-24T08:21:52.587627+0000","last_undegraded":"2023-11-24T08:21:52.587627+0000","last_fullsized":"2023-11-24T08:21:52.587627+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:51:17.234565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598408+0000","last_change":"2023-11-24T08:21:51.588934+0000","last_active":"2023-11-24T08:21:51.598408+0000","last_peered":"2023-11-24T08:21:51.598408+0000","last_clean":"2023-11-24T08:21:51.598408+0000","last_became_active":"2023-11-24T08:21:51.588357+0000","last_became_peered":"2023-11-24T08:21:51.588357+0000","last_unstale":"2023-11-24T08:21:51.598408+0000","last_undegraded":"2023-11-24T08:21:51.598408+0000","last_fullsized":"2023-11-24T08:21:51.598408+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:36.358037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423808+0000","last_change":"2023-11-24T08:21:50.567569+0000","last_active":"2023-11-24T08:21:53.423808+0000","last_peered":"2023-11-24T08:21:53.423808+0000","last_clean":"2023-11-24T08:21:53.423808+0000","last_became_active":"2023-11-24T08:21:50.566563+0000","last_became_peered":"2023-11-24T08:21:50.566563+0000","last_unstale":"2023-11-24T08:21:53.423808+0000","last_undegraded":"2023-11-24T08:21:53.423808+0000","last_fullsized":"2023-11-24T08:21:53.423808+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:25.992721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586302+0000","last_change":"2023-11-24T08:21:51.597897+0000","last_active":"2023-11-24T08:21:52.586302+0000","last_peered":"2023-11-24T08:21:52.586302+0000","last_clean":"2023-11-24T08:21:52.586302+0000","last_became_active":"2023-11-24T08:21:51.592609+0000","last_became_peered":"2023-11-24T08:21:51.592609+0000","last_unstale":"2023-11-24T08:21:52.586302+0000","last_undegraded":"2023-11-24T08:21:52.586302+0000","last_fullsized":"2023-11-24T08:21:52.586302+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.811886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423707+0000","last_change":"2023-11-24T08:21:50.568397+0000","last_active":"2023-11-24T08:21:53.423707+0000","last_peered":"2023-11-24T08:21:53.423707+0000","last_clean":"2023-11-24T08:21:53.423707+0000","last_became_active":"2023-11-24T08:21:50.565995+0000","last_became_peered":"2023-11-24T08:21:50.565995+0000","last_unstale":"2023-11-24T08:21:53.423707+0000","last_undegraded":"2023-11-24T08:21:53.423707+0000","last_fullsized":"2023-11-24T08:21:53.423707+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:29.579582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.579591+0000","last_change":"2023-11-24T08:21:51.574685+0000","last_active":"2023-11-24T08:21:51.579591+0000","last_peered":"2023-11-24T08:21:51.579591+0000","last_clean":"2023-11-24T08:21:51.579591+0000","last_became_active":"2023-11-24T08:21:51.572557+0000","last_became_peered":"2023-11-24T08:21:51.572557+0000","last_unstale":"2023-11-24T08:21:51.579591+0000","last_undegraded":"2023-11-24T08:21:51.579591+0000","last_fullsized":"2023-11-24T08:21:51.579591+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:50.466439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585598+0000","last_change":"2023-11-24T08:21:50.584238+0000","last_active":"2023-11-24T08:21:52.585598+0000","last_peered":"2023-11-24T08:21:52.585598+0000","last_clean":"2023-11-24T08:21:52.585598+0000","last_became_active":"2023-11-24T08:21:50.583939+0000","last_became_peered":"2023-11-24T08:21:50.583939+0000","last_unstale":"2023-11-24T08:21:52.585598+0000","last_undegraded":"2023-11-24T08:21:52.585598+0000","last_fullsized":"2023-11-24T08:21:52.585598+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:08:49.637900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.592453+0000","last_change":"2023-11-24T08:21:51.590953+0000","last_active":"2023-11-24T08:21:51.592453+0000","last_peered":"2023-11-24T08:21:51.592453+0000","last_clean":"2023-11-24T08:21:51.592453+0000","last_became_active":"2023-11-24T08:21:51.590410+0000","last_became_peered":"2023-11-24T08:21:51.590410+0000","last_unstale":"2023-11-24T08:21:51.592453+0000","last_undegraded":"2023-11-24T08:21:51.592453+0000","last_fullsized":"2023-11-24T08:21:51.592453+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:21:11.394560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587574+0000","last_change":"2023-11-24T08:21:50.585948+0000","last_active":"2023-11-24T08:21:52.587574+0000","last_peered":"2023-11-24T08:21:52.587574+0000","last_clean":"2023-11-24T08:21:52.587574+0000","last_became_active":"2023-11-24T08:21:50.582684+0000","last_became_peered":"2023-11-24T08:21:50.582684+0000","last_unstale":"2023-11-24T08:21:52.587574+0000","last_undegraded":"2023-11-24T08:21:52.587574+0000","last_fullsized":"2023-11-24T08:21:52.587574+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:22:03.718601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589912+0000","last_change":"2023-11-24T08:21:51.588761+0000","last_active":"2023-11-24T08:21:51.589912+0000","last_peered":"2023-11-24T08:21:51.589912+0000","last_clean":"2023-11-24T08:21:51.589912+0000","last_became_active":"2023-11-24T08:21:51.588490+0000","last_became_peered":"2023-11-24T08:21:51.588490+0000","last_unstale":"2023-11-24T08:21:51.589912+0000","last_undegraded":"2023-11-24T08:21:51.589912+0000","last_fullsized":"2023-11-24T08:21:51.589912+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:40.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.562621+0000","last_change":"2023-11-24T08:21:50.572673+0000","last_active":"2023-11-24T08:21:51.562621+0000","last_peered":"2023-11-24T08:21:51.562621+0000","last_clean":"2023-11-24T08:21:51.562621+0000","last_became_active":"2023-11-24T08:21:50.572373+0000","last_became_peered":"2023-11-24T08:21:50.572373+0000","last_unstale":"2023-11-24T08:21:51.562621+0000","last_undegraded":"2023-11-24T08:21:51.562621+0000","last_fullsized":"2023-11-24T08:21:51.562621+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:20:29.577576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576033+0000","last_change":"2023-11-24T08:21:51.572024+0000","last_active":"2023-11-24T08:21:51.576033+0000","last_peered":"2023-11-24T08:21:51.576033+0000","last_clean":"2023-11-24T08:21:51.576033+0000","last_became_active":"2023-11-24T08:21:51.571757+0000","last_became_peered":"2023-11-24T08:21:51.571757+0000","last_unstale":"2023-11-24T08:21:51.576033+0000","last_undegraded":"2023-11-24T08:21:51.576033+0000","last_fullsized":"2023-11-24T08:21:51.576033+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:16.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.571395+0000","last_change":"2023-11-24T08:21:50.579323+0000","last_active":"2023-11-24T08:21:51.571395+0000","last_peered":"2023-11-24T08:21:51.571395+0000","last_clean":"2023-11-24T08:21:51.571395+0000","last_became_active":"2023-11-24T08:21:50.578991+0000","last_became_peered":"2023-11-24T08:21:50.578991+0000","last_unstale":"2023-11-24T08:21:51.571395+0000","last_undegraded":"2023-11-24T08:21:51.571395+0000","last_fullsized":"2023-11-24T08:21:51.571395+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:43.379727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587398+0000","last_change":"2023-11-24T08:21:51.603204+0000","last_active":"2023-11-24T08:21:52.587398+0000","last_peered":"2023-11-24T08:21:52.587398+0000","last_clean":"2023-11-24T08:21:52.587398+0000","last_became_active":"2023-11-24T08:21:51.602933+0000","last_became_peered":"2023-11-24T08:21:51.602933+0000","last_unstale":"2023-11-24T08:21:52.587398+0000","last_undegraded":"2023-11-24T08:21:52.587398+0000","last_fullsized":"2023-11-24T08:21:52.587398+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:23.943409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585683+0000","last_change":"2023-11-24T08:21:50.583512+0000","last_active":"2023-11-24T08:21:52.585683+0000","last_peered":"2023-11-24T08:21:52.585683+0000","last_clean":"2023-11-24T08:21:52.585683+0000","last_became_active":"2023-11-24T08:21:50.583180+0000","last_became_peered":"2023-11-24T08:21:50.583180+0000","last_unstale":"2023-11-24T08:21:52.585683+0000","last_undegraded":"2023-11-24T08:21:52.585683+0000","last_fullsized":"2023-11-24T08:21:52.585683+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:41.172733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.591202+0000","last_change":"2023-11-24T08:21:51.589982+0000","last_active":"2023-11-24T08:21:51.591202+0000","last_peered":"2023-11-24T08:21:51.591202+0000","last_clean":"2023-11-24T08:21:51.591202+0000","last_became_active":"2023-11-24T08:21:51.589694+0000","last_became_peered":"2023-11-24T08:21:51.589694+0000","last_unstale":"2023-11-24T08:21:51.591202+0000","last_undegraded":"2023-11-24T08:21:51.591202+0000","last_fullsized":"2023-11-24T08:21:51.591202+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:39:19.313081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"206'2","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598053+0000","last_change":"2023-11-24T08:21:50.580872+0000","last_active":"2023-11-24T08:21:51.598053+0000","last_peered":"2023-11-24T08:21:51.598053+0000","last_clean":"2023-11-24T08:21:51.598053+0000","last_became_active":"2023-11-24T08:21:50.580534+0000","last_became_peered":"2023-11-24T08:21:50.580534+0000","last_unstale":"2023-11-24T08:21:51.598053+0000","last_undegraded":"2023-11-24T08:21:51.598053+0000","last_fullsized":"2023-11-24T08:21:51.598053+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:29.329334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":329,"reported_epoch":205,"state":"active+clean","last_fresh":"2023-11-24T08:21:50.570895+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:21:50.570895+0000","last_peered":"2023-11-24T08:21:50.570895+0000","last_clean":"2023-11-24T08:21:50.570895+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:21:50.570895+0000","last_undegraded":"2023-11-24T08:21:50.570895+0000","last_fullsized":"2023-11-24T08:21:50.570895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575058,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42628,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329212,"statfs":{"total":96636764160,"available":96593113088,"internally_reserved":0,"allocated":3203072,"data_stored":1746097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.071,"15min":1.069},"min":{"1min":0.622,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":2.048,"5min":2.048,"15min":2.7240000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.1220000000000001,"15min":1.1100000000000001},"min":{"1min":0.71699999999999997,"5min":0.67100000000000004,"15min":0.53700000000000003},"max":{"1min":1.899,"5min":1.899,"15min":2.7109999999999999},"last":0.87}]},{"osd":1,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.127,"15min":1.159},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.681,"5min":1.681,"15min":3.8519999999999999},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.1950000000000001,"15min":1.2250000000000001},"min":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.628},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.2569999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1060000000000001,"15min":1.121},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.55700000000000005},"max":{"1min":1.762,"5min":1.762,"15min":2.4430000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2609999999999999,"15min":1.302},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.60399999999999998},"max":{"1min":1.5629999999999999,"5min":1.7310000000000001,"15min":4.0650000000000004},"last":1.2949999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1910000000000001,"15min":1.2170000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.504},"max":{"1min":1.611,"5min":1.645,"15min":3.133},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.329,"15min":1.369},"min":{"1min":0.97299999999999998,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.77,"5min":2.173,"15min":3.9209999999999998},"last":1.1759999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.323,"15min":1.3540000000000001},"min":{"1min":0.84099999999999997,"5min":0.83699999999999997,"15min":0.54700000000000004},"max":{"1min":1.752,"5min":1.879,"15min":2.851},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.355,"15min":1.357},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.7130000000000001,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.117}]},{"osd":5,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4450000000000001,"15min":1.4750000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4470000000000001,"15min":1.4950000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":1.845,"5min":1.988,"15min":3.4239999999999999},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.548,"15min":1.5760000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.9059999999999999,"5min":1.994,"15min":2.992},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.484,"15min":1.51},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.827,"5min":2.4159999999999999,"15min":3.798},"last":1.2210000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37920,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333920,"statfs":{"total":96636764160,"available":96597934080,"internally_reserved":0,"allocated":2637824,"data_stored":1163095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.68300000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.756,"5min":1.7929999999999999,"15min":1.95},"last":1.155},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.089,"15min":1.083},"min":{"1min":0.624,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.619,"5min":2.008,"15min":2.008},"last":1.379}]},{"osd":1,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.097,"15min":1.1419999999999999},"min":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.61599999999999999},"max":{"1min":1.625,"5min":1.8100000000000001,"15min":2.0230000000000001},"last":1.327},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.2,"15min":1.1819999999999999},"min":{"1min":0.745,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.5109999999999999,"5min":2.206,"15min":2.4039999999999999},"last":1.6699999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1060000000000001,"15min":1.101},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.6699999999999999,"5min":1.7090000000000001,"15min":1.9330000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2350000000000001,"15min":1.254},"min":{"1min":0.90100000000000002,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.577,"5min":1.7849999999999999,"15min":2.0369999999999999},"last":1.518}]},{"osd":3,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3340000000000001,"15min":1.337},"min":{"1min":0.83799999999999997,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":1.823,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2769999999999999,"15min":1.302},"min":{"1min":0.92300000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8740000000000001,"5min":1.907,"15min":4.8319999999999999},"last":1.462}]},{"osd":4,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4099999999999999,"15min":1.413},"min":{"1min":0.79300000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.6870000000000001,"5min":3.673,"15min":3.673},"last":1.641},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4810000000000001,"15min":1.4830000000000001},"min":{"1min":1.1160000000000001,"5min":0.88,"15min":0.63800000000000001},"max":{"1min":1.913,"5min":2.8690000000000002,"15min":2.8820000000000001},"last":1.9850000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.379,"15min":1.409},"min":{"1min":1.109,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.782,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.22},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5720000000000001,"15min":1.5589999999999999},"min":{"1min":1.05,"5min":0.90100000000000002,"15min":0.83299999999999996},"max":{"1min":1.8500000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.73}]},{"osd":7,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.605,"15min":1.5860000000000001},"min":{"1min":0.94899999999999995,"5min":0.93500000000000005,"15min":0.748},"max":{"1min":1.986,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":1.552,"5min":1.5940000000000001,"15min":1.5860000000000001},"min":{"1min":1.1930000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.0409999999999999,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.613}]}]},{"osd":5,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37916,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333924,"statfs":{"total":96636764160,"available":96597938176,"internally_reserved":0,"allocated":2629632,"data_stored":1163039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.02,"15min":1.014},"min":{"1min":0.65300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.357,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.143},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.115,"15min":1.105},"min":{"1min":0.66100000000000003,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6319999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":0.92000000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1519999999999999,"15min":1.123},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.585,"5min":7.258,"15min":7.258},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.216,"15min":1.1870000000000001},"min":{"1min":0.79600000000000004,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":1.383}]},{"osd":2,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2130000000000001,"15min":1.1830000000000001},"min":{"1min":0.88900000000000001,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.7,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.2050000000000001,"15min":1.1719999999999999},"min":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.923,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.2649999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2549999999999999,"15min":1.179},"min":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.9670000000000001,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.335,"15min":1.276},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.55200000000000005},"max":{"1min":1.8580000000000001,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.347}]},{"osd":4,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4770000000000001,"15min":1.3779999999999999},"min":{"1min":0.92200000000000004,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":2.536,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.552,"15min":1.478},"min":{"1min":0.88100000000000001,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":2.4550000000000001,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.4370000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3740000000000001,"15min":1.341},"min":{"1min":0.81899999999999995,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":2.246,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.53,"15min":1.452},"min":{"1min":0.92600000000000005,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":2.3519999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.508}]},{"osd":7,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.536,"15min":1.494},"min":{"1min":0.94299999999999995,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":2.802,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":0.877},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.762,"15min":1.645},"min":{"1min":1.325,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.9489999999999998,"5min":19.663,"15min":19.663},"last":1.5940000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575059,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42044,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329796,"statfs":{"total":96636764160,"available":96593711104,"internally_reserved":0,"allocated":2605056,"data_stored":1155147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.044,"15min":1.0549999999999999},"min":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.42699999999999999},"max":{"1min":1.5600000000000001,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0609999999999999,"15min":1.079},"min":{"1min":0.79000000000000004,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.47,"5min":1.7070000000000001,"15min":5.452},"last":1.282}]},{"osd":1,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.119,"15min":1.1319999999999999},"min":{"1min":0.72199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6080000000000001,"5min":1.708,"15min":1.8100000000000001},"last":1.419},{"interface":"front","average":{"1min":1.155,"5min":1.123,"15min":1.153},"min":{"1min":0.751,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.468,"5min":1.7150000000000001,"15min":1.988},"last":1.157}]},{"osd":2,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.66100000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5209999999999999,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":1.107},{"interface":"front","average":{"1min":1.294,"5min":1.2490000000000001,"15min":1.2330000000000001},"min":{"1min":0.88500000000000001,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.714,"5min":1.827,"15min":2.0470000000000002},"last":1.4750000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3360000000000001,"15min":1.3280000000000001},"min":{"1min":1.159,"5min":0.754,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.925,"15min":2.121},"last":1.349},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.266,"15min":1.2709999999999999},"min":{"1min":0.873,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.538,"5min":1.7290000000000001,"15min":2.3340000000000001},"last":1.3779999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.5069999999999999,"15min":1.4890000000000001},"min":{"1min":1.266,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.8640000000000001,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.218},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.4139999999999999},"min":{"1min":0.93999999999999995,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":1.6759999999999999,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.5049999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.238,"15min":1.2410000000000001},"min":{"1min":0.81399999999999995,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.6779999999999999,"5min":1.8540000000000001,"15min":1.907},"last":1.252},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5189999999999999,"15min":1.5229999999999999},"min":{"1min":0.98399999999999999,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.9810000000000001,"5min":2.1190000000000002,"15min":2.2890000000000001},"last":1.5269999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4399999999999999,"15min":1.4750000000000001},"min":{"1min":0.76800000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.9199999999999999,"5min":1.948,"15min":3.1219999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.3939999999999999,"15min":1.3959999999999999},"min":{"1min":0.89700000000000002,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.1190000000000002,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.321}]}]},{"osd":3,"up_from":13,"seq":55834575058,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42048,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329792,"statfs":{"total":96636764160,"available":96593707008,"internally_reserved":0,"allocated":2605056,"data_stored":1155079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.2609999999999999,"5min":1.5820000000000001,"15min":10.557},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.92500000000000004,"15min":0.92600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.5149999999999999,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":1.097}]},{"osd":1,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.2010000000000001,"5min":1.71,"15min":2.101},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.872,"15min":0.96599999999999997},"min":{"1min":0.51500000000000001,"5min":0.5,"15min":0.379},"max":{"1min":1.4199999999999999,"5min":1.452,"15min":13.090999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.875,"15min":0.90600000000000003},"min":{"1min":0.497,"5min":0.433,"15min":0.39900000000000002},"max":{"1min":1.2989999999999999,"5min":1.976,"15min":3.9489999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.099,"5min":0.96999999999999997,"15min":1.0629999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":1.6319999999999999,"5min":1.665,"15min":13.460000000000001},"last":1.1699999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.2769999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.845,"5min":1.877,"15min":4.367},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.2270000000000001,"15min":1.21},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.76,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.353}]},{"osd":5,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2250000000000001,"15min":1.298},"min":{"1min":0.626,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.8149999999999999,"5min":2.1230000000000002,"15min":13.231},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.2150000000000001,"15min":1.244},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.583,"5min":1.8620000000000001,"15min":5.1180000000000003},"last":1.333}]},{"osd":6,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.284,"15min":1.3540000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.923,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.385},{"interface":"front","average":{"1min":1.244,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8140000000000001,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.2929999999999999,"15min":1.3129999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":1.8029999999999999,"5min":2.0720000000000001,"15min":4.5060000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.2849999999999999,"15min":1.2929999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.75,"5min":3.0030000000000001,"15min":3.331},"last":1.004}]}]},{"osd":2,"up_from":13,"seq":55834575056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42048,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329792,"statfs":{"total":96636764160,"available":96593707008,"internally_reserved":0,"allocated":2609152,"data_stored":1155673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.72699999999999998,"15min":0.76800000000000002},"min":{"1min":0.503,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.601},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.80800000000000005,"15min":0.85499999999999998},"min":{"1min":0.58899999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.2689999999999999,"5min":1.4810000000000001,"15min":1.782},"last":1.042}]},{"osd":1,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.86299999999999999,"15min":0.89200000000000002},"min":{"1min":0.61399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.57799999999999996,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.8360000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.86699999999999999,"15min":0.86299999999999999},"min":{"1min":0.53100000000000003,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.228,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.996,"15min":1.0229999999999999},"min":{"1min":0.63900000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.8109999999999999,"5min":1.948,"15min":1.948},"last":1.089}]},{"osd":4,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2430000000000001,"15min":1.2649999999999999},"min":{"1min":0.90600000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.7350000000000001,"5min":1.782,"15min":2.1379999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.262,"5min":1.2050000000000001,"15min":1.2450000000000001},"min":{"1min":0.89200000000000002,"5min":0.78300000000000003,"15min":0.745},"max":{"1min":1.48,"5min":1.8200000000000001,"15min":3.3279999999999998},"last":1.4179999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.2,"15min":1.2390000000000001},"min":{"1min":0.77200000000000002,"5min":0.66100000000000003,"15min":0.61499999999999999},"max":{"1min":1.7649999999999999,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.282},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.234,"15min":1.2410000000000001},"min":{"1min":1.0820000000000001,"5min":0.752,"15min":0.499},"max":{"1min":1.841,"5min":1.841,"15min":2.0910000000000002},"last":1.538}]},{"osd":6,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.1830000000000001,"15min":1.196},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":3.2210000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.28,"15min":1.3140000000000001},"min":{"1min":0.84099999999999997,"5min":0.70099999999999996,"15min":0.65600000000000003},"max":{"1min":1.8300000000000001,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.347}]},{"osd":7,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.29,"15min":1.3140000000000001},"min":{"1min":0.89100000000000001,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.381},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.27,"15min":1.2849999999999999},"min":{"1min":0.78300000000000003,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.994,"5min":2.1539999999999999,"15min":3.2730000000000001},"last":1.321}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42052,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329788,"statfs":{"total":96636764160,"available":96593702912,"internally_reserved":0,"allocated":2600960,"data_stored":1155057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.383,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.01,"5min":0.92700000000000005,"15min":0.89900000000000002},"min":{"1min":0.53000000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.637,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.7350000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.82599999999999996,"15min":0.81299999999999994},"min":{"1min":0.436,"5min":0.436,"15min":0.37},"max":{"1min":1.2350000000000001,"5min":1.823,"15min":1.861},"last":0.877},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.92400000000000004,"15min":0.92500000000000004},"min":{"1min":0.51500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.357,"5min":1.7490000000000001,"15min":1.925},"last":1.3180000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.046,"15min":1.012},"min":{"1min":0.55400000000000005,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":1.528,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.91200000000000003,"15min":0.91900000000000004},"min":{"1min":0.5,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.3959999999999999,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":1.444}]},{"osd":4,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.304,"15min":1.286},"min":{"1min":0.92700000000000005,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.6719999999999999,"5min":1.984,"15min":2.7959999999999998},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.284,"15min":1.262},"min":{"1min":0.98099999999999998,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":1.9410000000000001,"15min":2.8300000000000001},"last":1.8420000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2490000000000001,"15min":1.2589999999999999},"min":{"1min":0.84799999999999998,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.7030000000000001,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":1.524},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2390000000000001,"15min":1.2210000000000001},"min":{"1min":1.073,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":3.286},"last":1.6919999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2609999999999999,"15min":1.28},"min":{"1min":0.95299999999999996,"5min":0.76400000000000001,"15min":0.52400000000000002},"max":{"1min":1.762,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.631},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.284,"15min":1.278},"min":{"1min":0.81299999999999994,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.706,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.774}]},{"osd":7,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.323,"15min":1.3180000000000001},"min":{"1min":0.89400000000000002,"5min":0.82299999999999995,"15min":0.59099999999999997},"max":{"1min":1.865,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.177},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.327,"15min":1.337},"min":{"1min":0.96799999999999997,"5min":0.72299999999999998,"15min":0.624},"max":{"1min":1.8140000000000001,"5min":1.9930000000000001,"15min":3.653},"last":1.8260000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42620,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329220,"statfs":{"total":96636764160,"available":96593121280,"internally_reserved":0,"allocated":3194880,"data_stored":1745425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.73299999999999998},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.4910000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.4490000000000001,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.0209999999999999,"15min":1.012},"min":{"1min":0.54900000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.893,"15min":1.893},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.86699999999999999},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.571,"5min":1.798,"15min":2.4990000000000001},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93899999999999995,"15min":0.97199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86199999999999999,"15min":0.88500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.40100000000000002},"max":{"1min":1.429,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":0.60099999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.212,"15min":1.216},"min":{"1min":0.84799999999999998,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.722,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.331},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.081,"15min":1.109},"min":{"1min":0.76600000000000001,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.417,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1399999999999999,"15min":1.151},"min":{"1min":0.81899999999999995,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.55,"5min":1.6910000000000001,"15min":2.6850000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.129,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.4890000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1040000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.256},"min":{"1min":0.94299999999999995,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.587,"5min":1.8560000000000001,"15min":2.052},"last":1.077},{"interface":"front","average":{"1min":1.222,"5min":1.2290000000000001,"15min":1.2529999999999999},"min":{"1min":0.84799999999999998,"5min":0.76700000000000002,"15min":0.70899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8440000000000001,"15min":2.823},"last":1.139}]},{"osd":7,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.319,"15min":1.3400000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6379999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2889999999999999,"15min":1.306},"min":{"1min":0.95099999999999996,"5min":0.79600000000000004,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:21:55.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:21:56.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.247+0000 7f795c231640 1 Processor -- start 2023-11-24T08:21:56.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.247+0000 7f795c231640 1 -- start start 2023-11-24T08:21:56.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.248+0000 7f795c231640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f79540b2250 0x7f795415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.248+0000 7f795c231640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795415b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.248+0000 7f795c231640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795415ab80 con 0x7f79540b2630 2023-11-24T08:21:56.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f795c231640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795415a990 con 0x7f79540b2250 2023-11-24T08:21:56.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f795c231640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795415e680 con 0x7f79540b2aa0 2023-11-24T08:21:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f795a7ce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f79597cc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f79540b2630 0x7f79540ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:52984/0 (socket says 172.21.15.107:52984) 2023-11-24T08:21:56.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f79597cc640 1 -- 172.21.15.107:0/127519406 learned_addr learned my addr 172.21.15.107:0/127519406 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:56.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.249+0000 7f7959fcd640 1 --2- 172.21.15.107:0/127519406 >> v2:172.21.15.120:3300/0 conn(0x7f79540b2250 0x7f795415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.250+0000 7f7958fcb640 1 -- 172.21.15.107:0/127519406 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443407367 0 0) 0x7f795415ab80 con 0x7f79540b2630 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.250+0000 7f795a7ce640 1 -- 172.21.15.107:0/127519406 >> v2:172.21.15.120:3300/0 conn(0x7f79540b2250 msgr2=0x7f795415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.250+0000 7f795a7ce640 1 --2- 172.21.15.107:0/127519406 >> v2:172.21.15.120:3300/0 conn(0x7f79540b2250 0x7f795415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.250+0000 7f795a7ce640 1 -- 172.21.15.107:0/127519406 >> v1:172.21.15.120:6789/0 conn(0x7f79540b2630 legacy=0x7f79540ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.250+0000 7f795a7ce640 1 -- 172.21.15.107:0/127519406 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795415e960 con 0x7f79540b2aa0 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.252+0000 7f795a7ce640 1 --2- 172.21.15.107:0/127519406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795415b080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f794c00bfa0 tx=0x7f794c02e0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ec173b5225f60a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.252+0000 7f7958fcb640 1 -- 172.21.15.107:0/127519406 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c02ea40 con 0x7f79540b2aa0 2023-11-24T08:21:56.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.252+0000 7f7958fcb640 1 -- 172.21.15.107:0/127519406 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c02eba0 con 0x7f79540b2aa0 2023-11-24T08:21:56.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.252+0000 7f7958fcb640 1 -- 172.21.15.107:0/127519406 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c009540 con 0x7f79540b2aa0 2023-11-24T08:21:56.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 -- 172.21.15.107:0/127519406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 msgr2=0x7f795415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 --2- 172.21.15.107:0/127519406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795415b080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f794c00bfa0 tx=0x7f794c02e0a0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 -- 172.21.15.107:0/127519406 shutdown_connections 2023-11-24T08:21:56.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 --2- 172.21.15.107:0/127519406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795415b080 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 --2- 172.21.15.107:0/127519406 >> v2:172.21.15.120:3300/0 conn(0x7f79540b2250 0x7f795415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 -- 172.21.15.107:0/127519406 >> 172.21.15.107:0/127519406 conn(0x7f795406a920 msgr2=0x7f795406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:56.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 -- 172.21.15.107:0/127519406 shutdown_connections 2023-11-24T08:21:56.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.253+0000 7f795c231640 1 -- 172.21.15.107:0/127519406 wait complete. 2023-11-24T08:21:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.255+0000 7f795c231640 1 Processor -- start 2023-11-24T08:21:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.255+0000 7f795c231640 1 -- start start 2023-11-24T08:21:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 0x7f795409cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795414dfd0 con 0x7f79540b2630 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795414e1e0 con 0x7f79540b2250 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f795c231640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795414e4c0 con 0x7f79540b2aa0 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f79597cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f79597cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49508/0 (socket says 172.21.15.107:49508) 2023-11-24T08:21:56.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.256+0000 7f79597cc640 1 -- 172.21.15.107:0/217204948 learned_addr learned my addr 172.21.15.107:0/217204948 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:56.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f7959fcd640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 0x7f795409cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 -- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 msgr2=0x7f795414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f795a7ce640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795414da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 -- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 msgr2=0x7f795409cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 0x7f795409cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 -- 172.21.15.107:0/217204948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c00bc80 con 0x7f79540b2630 2023-11-24T08:21:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f7959fcd640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 0x7f795409cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f79597cc640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f794400c4d0 tx=0x7f794400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.257+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79440074e0 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.258+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7944007660 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.258+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944004a20 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.258+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79540cd7c0 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.258+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79541502b0 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.259+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7954067470 con 0x7f79540b2630 2023-11-24T08:21:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.260+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f7944004b80 con 0x7f79540b2630 2023-11-24T08:21:56.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.261+0000 7f794a7fc640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 0x7f793806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.261+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7944087860 con 0x7f79540b2630 2023-11-24T08:21:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.261+0000 7f7959fcd640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 0x7f793806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.262+0000 7f7959fcd640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 0x7f793806f280 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7950009ac0 tx=0x7f7950008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.265+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7944054380 con 0x7f79540b2630 2023-11-24T08:21:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.450+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79540cd640 con 0x7f793806cdc0 2023-11-24T08:21:56.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.463+0000 7f794a7fc640 1 -- 172.21.15.107:0/217204948 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287275 (secure 0 0 0) 0x7f7950010910 con 0x7f793806cdc0 2023-11-24T08:21:56.468 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:21:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 msgr2=0x7f793806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 0x7f793806f280 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7950009ac0 tx=0x7f7950008040 comp rx=0 tx=0).stop 2023-11-24T08:21:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 msgr2=0x7f7954150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f794400c4d0 tx=0x7f794400c990 comp rx=0 tx=0).stop 2023-11-24T08:21:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 shutdown_connections 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f793806cdc0 0x7f793806f280 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79540b2aa0 0x7f795414da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79540b2630 0x7f7954150a90 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 --2- 172.21.15.107:0/217204948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79540b2250 0x7f795409cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 >> 172.21.15.107:0/217204948 conn(0x7f795406a920 msgr2=0x7f79540ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 shutdown_connections 2023-11-24T08:21:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.470+0000 7f795c231640 1 -- 172.21.15.107:0/217204948 wait complete. 2023-11-24T08:21:56.492 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2023-11-24T08:21:55.758868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329276,"kb_used_data":21568,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754645444,"statfs":{"total":773094113280,"available":772756934656,"internally_reserved":0,"allocated":22085632,"data_stored":10438612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":314953278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001828"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598347+0000","last_change":"2023-11-24T08:21:51.590343+0000","last_active":"2023-11-24T08:21:51.598347+0000","last_peered":"2023-11-24T08:21:51.598347+0000","last_clean":"2023-11-24T08:21:51.598347+0000","last_became_active":"2023-11-24T08:21:51.590064+0000","last_became_peered":"2023-11-24T08:21:51.590064+0000","last_unstale":"2023-11-24T08:21:51.598347+0000","last_undegraded":"2023-11-24T08:21:51.598347+0000","last_fullsized":"2023-11-24T08:21:51.598347+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:39.783416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584357+0000","last_change":"2023-11-24T08:21:51.580536+0000","last_active":"2023-11-24T08:21:51.584357+0000","last_peered":"2023-11-24T08:21:51.584357+0000","last_clean":"2023-11-24T08:21:51.584357+0000","last_became_active":"2023-11-24T08:21:51.579046+0000","last_became_peered":"2023-11-24T08:21:51.579046+0000","last_unstale":"2023-11-24T08:21:51.584357+0000","last_undegraded":"2023-11-24T08:21:51.584357+0000","last_fullsized":"2023-11-24T08:21:51.584357+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:36.358037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.579780+0000","last_change":"2023-11-24T08:21:51.573236+0000","last_active":"2023-11-24T08:21:51.579780+0000","last_peered":"2023-11-24T08:21:51.579780+0000","last_clean":"2023-11-24T08:21:51.579780+0000","last_became_active":"2023-11-24T08:21:51.572949+0000","last_became_peered":"2023-11-24T08:21:51.572949+0000","last_unstale":"2023-11-24T08:21:51.579780+0000","last_undegraded":"2023-11-24T08:21:51.579780+0000","last_fullsized":"2023-11-24T08:21:51.579780+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:53:11.160812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598416+0000","last_change":"2023-11-24T08:21:51.595111+0000","last_active":"2023-11-24T08:21:51.598416+0000","last_peered":"2023-11-24T08:21:51.598416+0000","last_clean":"2023-11-24T08:21:51.598416+0000","last_became_active":"2023-11-24T08:21:51.589089+0000","last_became_peered":"2023-11-24T08:21:51.589089+0000","last_unstale":"2023-11-24T08:21:51.598416+0000","last_undegraded":"2023-11-24T08:21:51.598416+0000","last_fullsized":"2023-11-24T08:21:51.598416+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:35.455602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588841+0000","last_change":"2023-11-24T08:21:51.583718+0000","last_active":"2023-11-24T08:21:51.588841+0000","last_peered":"2023-11-24T08:21:51.588841+0000","last_clean":"2023-11-24T08:21:51.588841+0000","last_became_active":"2023-11-24T08:21:51.583398+0000","last_became_peered":"2023-11-24T08:21:51.583398+0000","last_unstale":"2023-11-24T08:21:51.588841+0000","last_undegraded":"2023-11-24T08:21:51.588841+0000","last_fullsized":"2023-11-24T08:21:51.588841+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:41:22.205969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584271+0000","last_change":"2023-11-24T08:21:51.586321+0000","last_active":"2023-11-24T08:21:52.584271+0000","last_peered":"2023-11-24T08:21:52.584271+0000","last_clean":"2023-11-24T08:21:52.584271+0000","last_became_active":"2023-11-24T08:21:51.584951+0000","last_became_peered":"2023-11-24T08:21:51.584951+0000","last_unstale":"2023-11-24T08:21:52.584271+0000","last_undegraded":"2023-11-24T08:21:52.584271+0000","last_fullsized":"2023-11-24T08:21:52.584271+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:23.995739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581027+0000","last_change":"2023-11-24T08:21:51.575848+0000","last_active":"2023-11-24T08:21:51.581027+0000","last_peered":"2023-11-24T08:21:51.581027+0000","last_clean":"2023-11-24T08:21:51.581027+0000","last_became_active":"2023-11-24T08:21:51.575592+0000","last_became_peered":"2023-11-24T08:21:51.575592+0000","last_unstale":"2023-11-24T08:21:51.581027+0000","last_undegraded":"2023-11-24T08:21:51.581027+0000","last_fullsized":"2023-11-24T08:21:51.581027+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:58:17.711093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583729+0000","last_change":"2023-11-24T08:21:51.592099+0000","last_active":"2023-11-24T08:21:52.583729+0000","last_peered":"2023-11-24T08:21:52.583729+0000","last_clean":"2023-11-24T08:21:52.583729+0000","last_became_active":"2023-11-24T08:21:51.591780+0000","last_became_peered":"2023-11-24T08:21:51.591780+0000","last_unstale":"2023-11-24T08:21:52.583729+0000","last_undegraded":"2023-11-24T08:21:52.583729+0000","last_fullsized":"2023-11-24T08:21:52.583729+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:53.955792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.574534+0000","last_change":"2023-11-24T08:21:51.567654+0000","last_active":"2023-11-24T08:21:51.574534+0000","last_peered":"2023-11-24T08:21:51.574534+0000","last_clean":"2023-11-24T08:21:51.574534+0000","last_became_active":"2023-11-24T08:21:51.567346+0000","last_became_peered":"2023-11-24T08:21:51.567346+0000","last_unstale":"2023-11-24T08:21:51.574534+0000","last_undegraded":"2023-11-24T08:21:51.574534+0000","last_fullsized":"2023-11-24T08:21:51.574534+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:53.803218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584340+0000","last_change":"2023-11-24T08:21:51.579885+0000","last_active":"2023-11-24T08:21:51.584340+0000","last_peered":"2023-11-24T08:21:51.584340+0000","last_clean":"2023-11-24T08:21:51.584340+0000","last_became_active":"2023-11-24T08:21:51.579643+0000","last_became_peered":"2023-11-24T08:21:51.579643+0000","last_unstale":"2023-11-24T08:21:51.584340+0000","last_undegraded":"2023-11-24T08:21:51.584340+0000","last_fullsized":"2023-11-24T08:21:51.584340+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:12:56.367676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.596212+0000","last_change":"2023-11-24T08:21:51.579110+0000","last_active":"2023-11-24T08:21:51.596212+0000","last_peered":"2023-11-24T08:21:51.596212+0000","last_clean":"2023-11-24T08:21:51.596212+0000","last_became_active":"2023-11-24T08:21:51.572264+0000","last_became_peered":"2023-11-24T08:21:51.572264+0000","last_unstale":"2023-11-24T08:21:51.596212+0000","last_undegraded":"2023-11-24T08:21:51.596212+0000","last_fullsized":"2023-11-24T08:21:51.596212+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:47.386507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585287+0000","last_change":"2023-11-24T08:21:51.602686+0000","last_active":"2023-11-24T08:21:52.585287+0000","last_peered":"2023-11-24T08:21:52.585287+0000","last_clean":"2023-11-24T08:21:52.585287+0000","last_became_active":"2023-11-24T08:21:51.601606+0000","last_became_peered":"2023-11-24T08:21:51.601606+0000","last_unstale":"2023-11-24T08:21:52.585287+0000","last_undegraded":"2023-11-24T08:21:52.585287+0000","last_fullsized":"2023-11-24T08:21:52.585287+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:26.855786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583641+0000","last_change":"2023-11-24T08:21:51.592472+0000","last_active":"2023-11-24T08:21:52.583641+0000","last_peered":"2023-11-24T08:21:52.583641+0000","last_clean":"2023-11-24T08:21:52.583641+0000","last_became_active":"2023-11-24T08:21:51.592136+0000","last_became_peered":"2023-11-24T08:21:51.592136+0000","last_unstale":"2023-11-24T08:21:52.583641+0000","last_undegraded":"2023-11-24T08:21:52.583641+0000","last_fullsized":"2023-11-24T08:21:52.583641+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:00:55.474780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586279+0000","last_change":"2023-11-24T08:21:51.593723+0000","last_active":"2023-11-24T08:21:52.586279+0000","last_peered":"2023-11-24T08:21:52.586279+0000","last_clean":"2023-11-24T08:21:52.586279+0000","last_became_active":"2023-11-24T08:21:51.589852+0000","last_became_peered":"2023-11-24T08:21:51.589852+0000","last_unstale":"2023-11-24T08:21:52.586279+0000","last_undegraded":"2023-11-24T08:21:52.586279+0000","last_fullsized":"2023-11-24T08:21:52.586279+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:20:36.366997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422698+0000","last_change":"2023-11-24T08:21:51.575331+0000","last_active":"2023-11-24T08:21:53.422698+0000","last_peered":"2023-11-24T08:21:53.422698+0000","last_clean":"2023-11-24T08:21:53.422698+0000","last_became_active":"2023-11-24T08:21:51.575033+0000","last_became_peered":"2023-11-24T08:21:51.575033+0000","last_unstale":"2023-11-24T08:21:53.422698+0000","last_undegraded":"2023-11-24T08:21:53.422698+0000","last_fullsized":"2023-11-24T08:21:53.422698+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:30.282265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583041+0000","last_change":"2023-11-24T08:21:51.582490+0000","last_active":"2023-11-24T08:21:52.583041+0000","last_peered":"2023-11-24T08:21:52.583041+0000","last_clean":"2023-11-24T08:21:52.583041+0000","last_became_active":"2023-11-24T08:21:51.578823+0000","last_became_peered":"2023-11-24T08:21:51.578823+0000","last_unstale":"2023-11-24T08:21:52.583041+0000","last_undegraded":"2023-11-24T08:21:52.583041+0000","last_fullsized":"2023-11-24T08:21:52.583041+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:27.009420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587190+0000","last_change":"2023-11-24T08:21:51.597212+0000","last_active":"2023-11-24T08:21:52.587190+0000","last_peered":"2023-11-24T08:21:52.587190+0000","last_clean":"2023-11-24T08:21:52.587190+0000","last_became_active":"2023-11-24T08:21:51.596627+0000","last_became_peered":"2023-11-24T08:21:51.596627+0000","last_unstale":"2023-11-24T08:21:52.587190+0000","last_undegraded":"2023-11-24T08:21:52.587190+0000","last_fullsized":"2023-11-24T08:21:52.587190+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:39:52.476877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.587872+0000","last_change":"2023-11-24T08:21:51.585577+0000","last_active":"2023-11-24T08:21:51.587872+0000","last_peered":"2023-11-24T08:21:51.587872+0000","last_clean":"2023-11-24T08:21:51.587872+0000","last_became_active":"2023-11-24T08:21:51.585287+0000","last_became_peered":"2023-11-24T08:21:51.585287+0000","last_unstale":"2023-11-24T08:21:51.587872+0000","last_undegraded":"2023-11-24T08:21:51.587872+0000","last_fullsized":"2023-11-24T08:21:51.587872+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:20.042283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598428+0000","last_change":"2023-11-24T08:21:51.589152+0000","last_active":"2023-11-24T08:21:51.598428+0000","last_peered":"2023-11-24T08:21:51.598428+0000","last_clean":"2023-11-24T08:21:51.598428+0000","last_became_active":"2023-11-24T08:21:51.588665+0000","last_became_peered":"2023-11-24T08:21:51.588665+0000","last_unstale":"2023-11-24T08:21:51.598428+0000","last_undegraded":"2023-11-24T08:21:51.598428+0000","last_fullsized":"2023-11-24T08:21:51.598428+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:10.411920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584250+0000","last_change":"2023-11-24T08:21:51.581669+0000","last_active":"2023-11-24T08:21:52.584250+0000","last_peered":"2023-11-24T08:21:52.584250+0000","last_clean":"2023-11-24T08:21:52.584250+0000","last_became_active":"2023-11-24T08:21:51.581333+0000","last_became_peered":"2023-11-24T08:21:51.581333+0000","last_unstale":"2023-11-24T08:21:52.584250+0000","last_undegraded":"2023-11-24T08:21:52.584250+0000","last_fullsized":"2023-11-24T08:21:52.584250+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:40.399989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576386+0000","last_change":"2023-11-24T08:21:51.572575+0000","last_active":"2023-11-24T08:21:51.576386+0000","last_peered":"2023-11-24T08:21:51.576386+0000","last_clean":"2023-11-24T08:21:51.576386+0000","last_became_active":"2023-11-24T08:21:51.572322+0000","last_became_peered":"2023-11-24T08:21:51.572322+0000","last_unstale":"2023-11-24T08:21:51.576386+0000","last_undegraded":"2023-11-24T08:21:51.576386+0000","last_fullsized":"2023-11-24T08:21:51.576386+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:29:01.619958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422851+0000","last_change":"2023-11-24T08:21:51.579005+0000","last_active":"2023-11-24T08:21:53.422851+0000","last_peered":"2023-11-24T08:21:53.422851+0000","last_clean":"2023-11-24T08:21:53.422851+0000","last_became_active":"2023-11-24T08:21:51.578676+0000","last_became_peered":"2023-11-24T08:21:51.578676+0000","last_unstale":"2023-11-24T08:21:53.422851+0000","last_undegraded":"2023-11-24T08:21:53.422851+0000","last_fullsized":"2023-11-24T08:21:53.422851+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:53.955792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576713+0000","last_change":"2023-11-24T08:21:51.568542+0000","last_active":"2023-11-24T08:21:51.576713+0000","last_peered":"2023-11-24T08:21:51.576713+0000","last_clean":"2023-11-24T08:21:51.576713+0000","last_became_active":"2023-11-24T08:21:51.568270+0000","last_became_peered":"2023-11-24T08:21:51.568270+0000","last_unstale":"2023-11-24T08:21:51.576713+0000","last_undegraded":"2023-11-24T08:21:51.576713+0000","last_fullsized":"2023-11-24T08:21:51.576713+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:28.009675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.585091+0000","last_change":"2023-11-24T08:21:51.576721+0000","last_active":"2023-11-24T08:21:51.585091+0000","last_peered":"2023-11-24T08:21:51.585091+0000","last_clean":"2023-11-24T08:21:51.585091+0000","last_became_active":"2023-11-24T08:21:51.576411+0000","last_became_peered":"2023-11-24T08:21:51.576411+0000","last_unstale":"2023-11-24T08:21:51.585091+0000","last_undegraded":"2023-11-24T08:21:51.585091+0000","last_fullsized":"2023-11-24T08:21:51.585091+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:59.195803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.597345+0000","last_change":"2023-11-24T08:21:51.586077+0000","last_active":"2023-11-24T08:21:51.597345+0000","last_peered":"2023-11-24T08:21:51.597345+0000","last_clean":"2023-11-24T08:21:51.597345+0000","last_became_active":"2023-11-24T08:21:51.583309+0000","last_became_peered":"2023-11-24T08:21:51.583309+0000","last_unstale":"2023-11-24T08:21:51.597345+0000","last_undegraded":"2023-11-24T08:21:51.597345+0000","last_fullsized":"2023-11-24T08:21:51.597345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:21.698335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.595218+0000","last_change":"2023-11-24T08:21:51.594658+0000","last_active":"2023-11-24T08:21:51.595218+0000","last_peered":"2023-11-24T08:21:51.595218+0000","last_clean":"2023-11-24T08:21:51.595218+0000","last_became_active":"2023-11-24T08:21:51.594157+0000","last_became_peered":"2023-11-24T08:21:51.594157+0000","last_unstale":"2023-11-24T08:21:51.595218+0000","last_undegraded":"2023-11-24T08:21:51.595218+0000","last_fullsized":"2023-11-24T08:21:51.595218+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:35:54.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.596190+0000","last_change":"2023-11-24T08:21:51.578705+0000","last_active":"2023-11-24T08:21:51.596190+0000","last_peered":"2023-11-24T08:21:51.596190+0000","last_clean":"2023-11-24T08:21:51.596190+0000","last_became_active":"2023-11-24T08:21:51.578423+0000","last_became_peered":"2023-11-24T08:21:51.578423+0000","last_unstale":"2023-11-24T08:21:51.596190+0000","last_undegraded":"2023-11-24T08:21:51.596190+0000","last_fullsized":"2023-11-24T08:21:51.596190+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:48:44.243887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585569+0000","last_change":"2023-11-24T08:21:51.594477+0000","last_active":"2023-11-24T08:21:52.585569+0000","last_peered":"2023-11-24T08:21:52.585569+0000","last_clean":"2023-11-24T08:21:52.585569+0000","last_became_active":"2023-11-24T08:21:51.594101+0000","last_became_peered":"2023-11-24T08:21:51.594101+0000","last_unstale":"2023-11-24T08:21:52.585569+0000","last_undegraded":"2023-11-24T08:21:52.585569+0000","last_fullsized":"2023-11-24T08:21:52.585569+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:55:47.728601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588928+0000","last_change":"2023-11-24T08:21:51.582719+0000","last_active":"2023-11-24T08:21:51.588928+0000","last_peered":"2023-11-24T08:21:51.588928+0000","last_clean":"2023-11-24T08:21:51.588928+0000","last_became_active":"2023-11-24T08:21:51.582435+0000","last_became_peered":"2023-11-24T08:21:51.582435+0000","last_unstale":"2023-11-24T08:21:51.588928+0000","last_undegraded":"2023-11-24T08:21:51.588928+0000","last_fullsized":"2023-11-24T08:21:51.588928+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:30.346032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583720+0000","last_change":"2023-11-24T08:21:51.598869+0000","last_active":"2023-11-24T08:21:52.583720+0000","last_peered":"2023-11-24T08:21:52.583720+0000","last_clean":"2023-11-24T08:21:52.583720+0000","last_became_active":"2023-11-24T08:21:51.598598+0000","last_became_peered":"2023-11-24T08:21:51.598598+0000","last_unstale":"2023-11-24T08:21:52.583720+0000","last_undegraded":"2023-11-24T08:21:52.583720+0000","last_fullsized":"2023-11-24T08:21:52.583720+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:10:30.282265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576735+0000","last_change":"2023-11-24T08:21:51.571213+0000","last_active":"2023-11-24T08:21:51.576735+0000","last_peered":"2023-11-24T08:21:51.576735+0000","last_clean":"2023-11-24T08:21:51.576735+0000","last_became_active":"2023-11-24T08:21:51.570835+0000","last_became_peered":"2023-11-24T08:21:51.570835+0000","last_unstale":"2023-11-24T08:21:51.576735+0000","last_undegraded":"2023-11-24T08:21:51.576735+0000","last_fullsized":"2023-11-24T08:21:51.576735+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:19:59.446051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.583423+0000","last_change":"2023-11-24T08:21:51.596009+0000","last_active":"2023-11-24T08:21:52.583423+0000","last_peered":"2023-11-24T08:21:52.583423+0000","last_clean":"2023-11-24T08:21:52.583423+0000","last_became_active":"2023-11-24T08:21:51.595710+0000","last_became_peered":"2023-11-24T08:21:51.595710+0000","last_unstale":"2023-11-24T08:21:52.583423+0000","last_undegraded":"2023-11-24T08:21:52.583423+0000","last_fullsized":"2023-11-24T08:21:52.583423+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:16.476510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589125+0000","last_change":"2023-11-24T08:21:51.587432+0000","last_active":"2023-11-24T08:21:51.589125+0000","last_peered":"2023-11-24T08:21:51.589125+0000","last_clean":"2023-11-24T08:21:51.589125+0000","last_became_active":"2023-11-24T08:21:51.587146+0000","last_became_peered":"2023-11-24T08:21:51.587146+0000","last_unstale":"2023-11-24T08:21:51.589125+0000","last_undegraded":"2023-11-24T08:21:51.589125+0000","last_fullsized":"2023-11-24T08:21:51.589125+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:05:22.114873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"206'4","reported_seq":15,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598591+0000","last_change":"2023-11-24T08:21:50.580627+0000","last_active":"2023-11-24T08:21:51.598591+0000","last_peered":"2023-11-24T08:21:51.598591+0000","last_clean":"2023-11-24T08:21:51.598591+0000","last_became_active":"2023-11-24T08:21:50.580329+0000","last_became_peered":"2023-11-24T08:21:50.580329+0000","last_unstale":"2023-11-24T08:21:51.598591+0000","last_undegraded":"2023-11-24T08:21:51.598591+0000","last_fullsized":"2023-11-24T08:21:51.598591+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:34.438903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576137+0000","last_change":"2023-11-24T08:21:51.572049+0000","last_active":"2023-11-24T08:21:51.576137+0000","last_peered":"2023-11-24T08:21:51.576137+0000","last_clean":"2023-11-24T08:21:51.576137+0000","last_became_active":"2023-11-24T08:21:51.571793+0000","last_became_peered":"2023-11-24T08:21:51.571793+0000","last_unstale":"2023-11-24T08:21:51.576137+0000","last_undegraded":"2023-11-24T08:21:51.576137+0000","last_fullsized":"2023-11-24T08:21:51.576137+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:35:24.923787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587569+0000","last_change":"2023-11-24T08:21:50.587199+0000","last_active":"2023-11-24T08:21:52.587569+0000","last_peered":"2023-11-24T08:21:52.587569+0000","last_clean":"2023-11-24T08:21:52.587569+0000","last_became_active":"2023-11-24T08:21:50.586820+0000","last_became_peered":"2023-11-24T08:21:50.586820+0000","last_unstale":"2023-11-24T08:21:52.587569+0000","last_undegraded":"2023-11-24T08:21:52.587569+0000","last_fullsized":"2023-11-24T08:21:52.587569+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:31:31.292020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587560+0000","last_change":"2023-11-24T08:21:51.598051+0000","last_active":"2023-11-24T08:21:52.587560+0000","last_peered":"2023-11-24T08:21:52.587560+0000","last_clean":"2023-11-24T08:21:52.587560+0000","last_became_active":"2023-11-24T08:21:51.597671+0000","last_became_peered":"2023-11-24T08:21:51.597671+0000","last_unstale":"2023-11-24T08:21:52.587560+0000","last_undegraded":"2023-11-24T08:21:52.587560+0000","last_fullsized":"2023-11-24T08:21:52.587560+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:30.346032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584588+0000","last_change":"2023-11-24T08:21:50.587346+0000","last_active":"2023-11-24T08:21:52.584588+0000","last_peered":"2023-11-24T08:21:52.584588+0000","last_clean":"2023-11-24T08:21:52.584588+0000","last_became_active":"2023-11-24T08:21:50.587054+0000","last_became_peered":"2023-11-24T08:21:50.587054+0000","last_unstale":"2023-11-24T08:21:52.584588+0000","last_undegraded":"2023-11-24T08:21:52.584588+0000","last_fullsized":"2023-11-24T08:21:52.584588+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:47.268620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581612+0000","last_change":"2023-11-24T08:21:51.579104+0000","last_active":"2023-11-24T08:21:51.581612+0000","last_peered":"2023-11-24T08:21:51.581612+0000","last_clean":"2023-11-24T08:21:51.581612+0000","last_became_active":"2023-11-24T08:21:51.578833+0000","last_became_peered":"2023-11-24T08:21:51.578833+0000","last_unstale":"2023-11-24T08:21:51.581612+0000","last_undegraded":"2023-11-24T08:21:51.581612+0000","last_fullsized":"2023-11-24T08:21:51.581612+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:44.218789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423056+0000","last_change":"2023-11-24T08:21:50.585753+0000","last_active":"2023-11-24T08:21:53.423056+0000","last_peered":"2023-11-24T08:21:53.423056+0000","last_clean":"2023-11-24T08:21:53.423056+0000","last_became_active":"2023-11-24T08:21:50.585515+0000","last_became_peered":"2023-11-24T08:21:50.585515+0000","last_unstale":"2023-11-24T08:21:53.423056+0000","last_undegraded":"2023-11-24T08:21:53.423056+0000","last_fullsized":"2023-11-24T08:21:53.423056+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:28:47.268620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586073+0000","last_change":"2023-11-24T08:21:51.594807+0000","last_active":"2023-11-24T08:21:52.586073+0000","last_peered":"2023-11-24T08:21:52.586073+0000","last_clean":"2023-11-24T08:21:52.586073+0000","last_became_active":"2023-11-24T08:21:51.594122+0000","last_became_peered":"2023-11-24T08:21:51.594122+0000","last_unstale":"2023-11-24T08:21:52.586073+0000","last_undegraded":"2023-11-24T08:21:52.586073+0000","last_fullsized":"2023-11-24T08:21:52.586073+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:06.621548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"206'1","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594306+0000","last_change":"2023-11-24T08:21:50.587203+0000","last_active":"2023-11-24T08:21:51.594306+0000","last_peered":"2023-11-24T08:21:51.594306+0000","last_clean":"2023-11-24T08:21:51.594306+0000","last_became_active":"2023-11-24T08:21:50.586924+0000","last_became_peered":"2023-11-24T08:21:50.586924+0000","last_unstale":"2023-11-24T08:21:51.594306+0000","last_undegraded":"2023-11-24T08:21:51.594306+0000","last_fullsized":"2023-11-24T08:21:51.594306+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:48.388108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422916+0000","last_change":"2023-11-24T08:21:51.579729+0000","last_active":"2023-11-24T08:21:53.422916+0000","last_peered":"2023-11-24T08:21:53.422916+0000","last_clean":"2023-11-24T08:21:53.422916+0000","last_became_active":"2023-11-24T08:21:51.579463+0000","last_became_peered":"2023-11-24T08:21:51.579463+0000","last_unstale":"2023-11-24T08:21:53.422916+0000","last_undegraded":"2023-11-24T08:21:53.422916+0000","last_fullsized":"2023-11-24T08:21:53.422916+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:52:40.399989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.563788+0000","last_change":"2023-11-24T08:21:50.585026+0000","last_active":"2023-11-24T08:21:51.563788+0000","last_peered":"2023-11-24T08:21:51.563788+0000","last_clean":"2023-11-24T08:21:51.563788+0000","last_became_active":"2023-11-24T08:21:50.584718+0000","last_became_peered":"2023-11-24T08:21:50.584718+0000","last_unstale":"2023-11-24T08:21:51.563788+0000","last_undegraded":"2023-11-24T08:21:51.563788+0000","last_fullsized":"2023-11-24T08:21:51.563788+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:11:33.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589480+0000","last_change":"2023-11-24T08:21:51.585963+0000","last_active":"2023-11-24T08:21:51.589480+0000","last_peered":"2023-11-24T08:21:51.589480+0000","last_clean":"2023-11-24T08:21:51.589480+0000","last_became_active":"2023-11-24T08:21:51.585718+0000","last_became_peered":"2023-11-24T08:21:51.585718+0000","last_unstale":"2023-11-24T08:21:51.589480+0000","last_undegraded":"2023-11-24T08:21:51.589480+0000","last_fullsized":"2023-11-24T08:21:51.589480+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:36.410622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.564062+0000","last_change":"2023-11-24T08:21:50.571703+0000","last_active":"2023-11-24T08:21:51.564062+0000","last_peered":"2023-11-24T08:21:51.564062+0000","last_clean":"2023-11-24T08:21:51.564062+0000","last_became_active":"2023-11-24T08:21:50.571411+0000","last_became_peered":"2023-11-24T08:21:50.571411+0000","last_unstale":"2023-11-24T08:21:51.564062+0000","last_undegraded":"2023-11-24T08:21:51.564062+0000","last_fullsized":"2023-11-24T08:21:51.564062+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:11:41.187561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423129+0000","last_change":"2023-11-24T08:21:51.588944+0000","last_active":"2023-11-24T08:21:53.423129+0000","last_peered":"2023-11-24T08:21:53.423129+0000","last_clean":"2023-11-24T08:21:53.423129+0000","last_became_active":"2023-11-24T08:21:51.588628+0000","last_became_peered":"2023-11-24T08:21:51.588628+0000","last_unstale":"2023-11-24T08:21:53.423129+0000","last_undegraded":"2023-11-24T08:21:53.423129+0000","last_fullsized":"2023-11-24T08:21:53.423129+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:23.995739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.575396+0000","last_change":"2023-11-24T08:21:50.579938+0000","last_active":"2023-11-24T08:21:51.575396+0000","last_peered":"2023-11-24T08:21:51.575396+0000","last_clean":"2023-11-24T08:21:51.575396+0000","last_became_active":"2023-11-24T08:21:50.578966+0000","last_became_peered":"2023-11-24T08:21:50.578966+0000","last_unstale":"2023-11-24T08:21:51.575396+0000","last_undegraded":"2023-11-24T08:21:51.575396+0000","last_fullsized":"2023-11-24T08:21:51.575396+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:36.912515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423699+0000","last_change":"2023-11-24T08:21:51.571316+0000","last_active":"2023-11-24T08:21:53.423699+0000","last_peered":"2023-11-24T08:21:53.423699+0000","last_clean":"2023-11-24T08:21:53.423699+0000","last_became_active":"2023-11-24T08:21:51.571020+0000","last_became_peered":"2023-11-24T08:21:51.571020+0000","last_unstale":"2023-11-24T08:21:53.423699+0000","last_undegraded":"2023-11-24T08:21:53.423699+0000","last_fullsized":"2023-11-24T08:21:53.423699+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:42.189432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.563211+0000","last_change":"2023-11-24T08:21:50.584651+0000","last_active":"2023-11-24T08:21:51.563211+0000","last_peered":"2023-11-24T08:21:51.563211+0000","last_clean":"2023-11-24T08:21:51.563211+0000","last_became_active":"2023-11-24T08:21:50.584354+0000","last_became_peered":"2023-11-24T08:21:50.584354+0000","last_unstale":"2023-11-24T08:21:51.563211+0000","last_undegraded":"2023-11-24T08:21:51.563211+0000","last_fullsized":"2023-11-24T08:21:51.563211+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:01:49.356407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.580413+0000","last_change":"2023-11-24T08:21:51.573829+0000","last_active":"2023-11-24T08:21:51.580413+0000","last_peered":"2023-11-24T08:21:51.580413+0000","last_clean":"2023-11-24T08:21:51.580413+0000","last_became_active":"2023-11-24T08:21:51.573562+0000","last_became_peered":"2023-11-24T08:21:51.573562+0000","last_unstale":"2023-11-24T08:21:51.580413+0000","last_undegraded":"2023-11-24T08:21:51.580413+0000","last_fullsized":"2023-11-24T08:21:51.580413+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:35:40.189115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"206'3","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598530+0000","last_change":"2023-11-24T08:21:50.583516+0000","last_active":"2023-11-24T08:21:51.598530+0000","last_peered":"2023-11-24T08:21:51.598530+0000","last_clean":"2023-11-24T08:21:51.598530+0000","last_became_active":"2023-11-24T08:21:50.582753+0000","last_became_peered":"2023-11-24T08:21:50.582753+0000","last_unstale":"2023-11-24T08:21:51.598530+0000","last_undegraded":"2023-11-24T08:21:51.598530+0000","last_fullsized":"2023-11-24T08:21:51.598530+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:28:53.332475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.588536+0000","last_change":"2023-11-24T08:21:51.576091+0000","last_active":"2023-11-24T08:21:51.588536+0000","last_peered":"2023-11-24T08:21:51.588536+0000","last_clean":"2023-11-24T08:21:51.588536+0000","last_became_active":"2023-11-24T08:21:51.575740+0000","last_became_peered":"2023-11-24T08:21:51.575740+0000","last_unstale":"2023-11-24T08:21:51.588536+0000","last_undegraded":"2023-11-24T08:21:51.588536+0000","last_fullsized":"2023-11-24T08:21:51.588536+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:52:52.319836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"206'6","reported_seq":18,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.584193+0000","last_change":"2023-11-24T08:21:50.589155+0000","last_active":"2023-11-24T08:21:52.584193+0000","last_peered":"2023-11-24T08:21:52.584193+0000","last_clean":"2023-11-24T08:21:52.584193+0000","last_became_active":"2023-11-24T08:21:50.588733+0000","last_became_peered":"2023-11-24T08:21:50.588733+0000","last_unstale":"2023-11-24T08:21:52.584193+0000","last_undegraded":"2023-11-24T08:21:52.584193+0000","last_fullsized":"2023-11-24T08:21:52.584193+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:29.579582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.590849+0000","last_change":"2023-11-24T08:21:51.588326+0000","last_active":"2023-11-24T08:21:51.590849+0000","last_peered":"2023-11-24T08:21:51.590849+0000","last_clean":"2023-11-24T08:21:51.590849+0000","last_became_active":"2023-11-24T08:21:51.588025+0000","last_became_peered":"2023-11-24T08:21:51.588025+0000","last_unstale":"2023-11-24T08:21:51.590849+0000","last_undegraded":"2023-11-24T08:21:51.590849+0000","last_fullsized":"2023-11-24T08:21:51.590849+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:44.396426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587076+0000","last_change":"2023-11-24T08:21:50.586145+0000","last_active":"2023-11-24T08:21:52.587076+0000","last_peered":"2023-11-24T08:21:52.587076+0000","last_clean":"2023-11-24T08:21:52.587076+0000","last_became_active":"2023-11-24T08:21:50.583351+0000","last_became_peered":"2023-11-24T08:21:50.583351+0000","last_unstale":"2023-11-24T08:21:52.587076+0000","last_undegraded":"2023-11-24T08:21:52.587076+0000","last_fullsized":"2023-11-24T08:21:52.587076+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:24.796699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587303+0000","last_change":"2023-11-24T08:21:51.593188+0000","last_active":"2023-11-24T08:21:52.587303+0000","last_peered":"2023-11-24T08:21:52.587303+0000","last_clean":"2023-11-24T08:21:52.587303+0000","last_became_active":"2023-11-24T08:21:51.592808+0000","last_became_peered":"2023-11-24T08:21:51.592808+0000","last_unstale":"2023-11-24T08:21:52.587303+0000","last_undegraded":"2023-11-24T08:21:52.587303+0000","last_fullsized":"2023-11-24T08:21:52.587303+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:59.195803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.568391+0000","last_change":"2023-11-24T08:21:50.564949+0000","last_active":"2023-11-24T08:21:51.568391+0000","last_peered":"2023-11-24T08:21:51.568391+0000","last_clean":"2023-11-24T08:21:51.568391+0000","last_became_active":"2023-11-24T08:21:50.564690+0000","last_became_peered":"2023-11-24T08:21:50.564690+0000","last_unstale":"2023-11-24T08:21:51.568391+0000","last_undegraded":"2023-11-24T08:21:51.568391+0000","last_fullsized":"2023-11-24T08:21:51.568391+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:57.160663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.580670+0000","last_change":"2023-11-24T08:21:51.574274+0000","last_active":"2023-11-24T08:21:51.580670+0000","last_peered":"2023-11-24T08:21:51.580670+0000","last_clean":"2023-11-24T08:21:51.580670+0000","last_became_active":"2023-11-24T08:21:51.573985+0000","last_became_peered":"2023-11-24T08:21:51.573985+0000","last_unstale":"2023-11-24T08:21:51.580670+0000","last_undegraded":"2023-11-24T08:21:51.580670+0000","last_fullsized":"2023-11-24T08:21:51.580670+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:26:38.263958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.582926+0000","last_change":"2023-11-24T08:21:50.576886+0000","last_active":"2023-11-24T08:21:52.582926+0000","last_peered":"2023-11-24T08:21:52.582926+0000","last_clean":"2023-11-24T08:21:52.582926+0000","last_became_active":"2023-11-24T08:21:50.576591+0000","last_became_peered":"2023-11-24T08:21:50.576591+0000","last_unstale":"2023-11-24T08:21:52.582926+0000","last_undegraded":"2023-11-24T08:21:52.582926+0000","last_fullsized":"2023-11-24T08:21:52.582926+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:54.522055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423168+0000","last_change":"2023-11-24T08:21:51.574026+0000","last_active":"2023-11-24T08:21:53.423168+0000","last_peered":"2023-11-24T08:21:53.423168+0000","last_clean":"2023-11-24T08:21:53.423168+0000","last_became_active":"2023-11-24T08:21:51.573776+0000","last_became_peered":"2023-11-24T08:21:51.573776+0000","last_unstale":"2023-11-24T08:21:53.423168+0000","last_undegraded":"2023-11-24T08:21:53.423168+0000","last_fullsized":"2023-11-24T08:21:53.423168+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.811886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"206'8","reported_seq":20,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585430+0000","last_change":"2023-11-24T08:21:50.587706+0000","last_active":"2023-11-24T08:21:52.585430+0000","last_peered":"2023-11-24T08:21:52.585430+0000","last_clean":"2023-11-24T08:21:52.585430+0000","last_became_active":"2023-11-24T08:21:50.587370+0000","last_became_peered":"2023-11-24T08:21:50.587370+0000","last_unstale":"2023-11-24T08:21:52.585430+0000","last_undegraded":"2023-11-24T08:21:52.585430+0000","last_fullsized":"2023-11-24T08:21:52.585430+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:49.795187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.595541+0000","last_change":"2023-11-24T08:21:51.592785+0000","last_active":"2023-11-24T08:21:51.595541+0000","last_peered":"2023-11-24T08:21:51.595541+0000","last_clean":"2023-11-24T08:21:51.595541+0000","last_became_active":"2023-11-24T08:21:51.592504+0000","last_became_peered":"2023-11-24T08:21:51.592504+0000","last_unstale":"2023-11-24T08:21:51.595541+0000","last_undegraded":"2023-11-24T08:21:51.595541+0000","last_fullsized":"2023-11-24T08:21:51.595541+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:06:10.411920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"206'1","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.577493+0000","last_change":"2023-11-24T08:21:50.584442+0000","last_active":"2023-11-24T08:21:51.577493+0000","last_peered":"2023-11-24T08:21:51.577493+0000","last_clean":"2023-11-24T08:21:51.577493+0000","last_became_active":"2023-11-24T08:21:50.584093+0000","last_became_peered":"2023-11-24T08:21:50.584093+0000","last_unstale":"2023-11-24T08:21:51.577493+0000","last_undegraded":"2023-11-24T08:21:51.577493+0000","last_fullsized":"2023-11-24T08:21:51.577493+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.242866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.601662+0000","last_change":"2023-11-24T08:21:51.599031+0000","last_active":"2023-11-24T08:21:51.601662+0000","last_peered":"2023-11-24T08:21:51.601662+0000","last_clean":"2023-11-24T08:21:51.601662+0000","last_became_active":"2023-11-24T08:21:51.598747+0000","last_became_peered":"2023-11-24T08:21:51.598747+0000","last_unstale":"2023-11-24T08:21:51.601662+0000","last_undegraded":"2023-11-24T08:21:51.601662+0000","last_fullsized":"2023-11-24T08:21:51.601662+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:58.177362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"206'2","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.587864+0000","last_change":"2023-11-24T08:21:50.570680+0000","last_active":"2023-11-24T08:21:51.587864+0000","last_peered":"2023-11-24T08:21:51.587864+0000","last_clean":"2023-11-24T08:21:51.587864+0000","last_became_active":"2023-11-24T08:21:50.570430+0000","last_became_peered":"2023-11-24T08:21:50.570430+0000","last_unstale":"2023-11-24T08:21:51.587864+0000","last_undegraded":"2023-11-24T08:21:51.587864+0000","last_fullsized":"2023-11-24T08:21:51.587864+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:28.237795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.581312+0000","last_change":"2023-11-24T08:21:51.577355+0000","last_active":"2023-11-24T08:21:51.581312+0000","last_peered":"2023-11-24T08:21:51.581312+0000","last_clean":"2023-11-24T08:21:51.581312+0000","last_became_active":"2023-11-24T08:21:51.577071+0000","last_became_peered":"2023-11-24T08:21:51.577071+0000","last_unstale":"2023-11-24T08:21:51.581312+0000","last_undegraded":"2023-11-24T08:21:51.581312+0000","last_fullsized":"2023-11-24T08:21:51.581312+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:01:35.877808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"206'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586831+0000","last_change":"2023-11-24T08:21:50.582670+0000","last_active":"2023-11-24T08:21:52.586831+0000","last_peered":"2023-11-24T08:21:52.586831+0000","last_clean":"2023-11-24T08:21:52.586831+0000","last_became_active":"2023-11-24T08:21:50.582396+0000","last_became_peered":"2023-11-24T08:21:50.582396+0000","last_unstale":"2023-11-24T08:21:52.586831+0000","last_undegraded":"2023-11-24T08:21:52.586831+0000","last_fullsized":"2023-11-24T08:21:52.586831+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:57:57.084338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587345+0000","last_change":"2023-11-24T08:21:51.604541+0000","last_active":"2023-11-24T08:21:52.587345+0000","last_peered":"2023-11-24T08:21:52.587345+0000","last_clean":"2023-11-24T08:21:52.587345+0000","last_became_active":"2023-11-24T08:21:51.604123+0000","last_became_peered":"2023-11-24T08:21:51.604123+0000","last_unstale":"2023-11-24T08:21:52.587345+0000","last_undegraded":"2023-11-24T08:21:52.587345+0000","last_fullsized":"2023-11-24T08:21:52.587345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:41:22.205969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585418+0000","last_change":"2023-11-24T08:21:50.584277+0000","last_active":"2023-11-24T08:21:52.585418+0000","last_peered":"2023-11-24T08:21:52.585418+0000","last_clean":"2023-11-24T08:21:52.585418+0000","last_became_active":"2023-11-24T08:21:50.583971+0000","last_became_peered":"2023-11-24T08:21:50.583971+0000","last_unstale":"2023-11-24T08:21:52.585418+0000","last_undegraded":"2023-11-24T08:21:52.585418+0000","last_fullsized":"2023-11-24T08:21:52.585418+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:36:03.042901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585158+0000","last_change":"2023-11-24T08:21:51.590865+0000","last_active":"2023-11-24T08:21:52.585158+0000","last_peered":"2023-11-24T08:21:52.585158+0000","last_clean":"2023-11-24T08:21:52.585158+0000","last_became_active":"2023-11-24T08:21:51.589755+0000","last_became_peered":"2023-11-24T08:21:51.589755+0000","last_unstale":"2023-11-24T08:21:52.585158+0000","last_undegraded":"2023-11-24T08:21:52.585158+0000","last_fullsized":"2023-11-24T08:21:52.585158+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:21:33.418895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587331+0000","last_change":"2023-11-24T08:21:50.583979+0000","last_active":"2023-11-24T08:21:52.587331+0000","last_peered":"2023-11-24T08:21:52.587331+0000","last_clean":"2023-11-24T08:21:52.587331+0000","last_became_active":"2023-11-24T08:21:50.583017+0000","last_became_peered":"2023-11-24T08:21:50.583017+0000","last_unstale":"2023-11-24T08:21:52.587331+0000","last_undegraded":"2023-11-24T08:21:52.587331+0000","last_fullsized":"2023-11-24T08:21:52.587331+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:02.853886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594548+0000","last_change":"2023-11-24T08:21:51.584499+0000","last_active":"2023-11-24T08:21:51.594548+0000","last_peered":"2023-11-24T08:21:51.594548+0000","last_clean":"2023-11-24T08:21:51.594548+0000","last_became_active":"2023-11-24T08:21:51.583368+0000","last_became_peered":"2023-11-24T08:21:51.583368+0000","last_unstale":"2023-11-24T08:21:51.594548+0000","last_undegraded":"2023-11-24T08:21:51.594548+0000","last_fullsized":"2023-11-24T08:21:51.594548+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:23.943409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423085+0000","last_change":"2023-11-24T08:21:50.575845+0000","last_active":"2023-11-24T08:21:53.423085+0000","last_peered":"2023-11-24T08:21:53.423085+0000","last_clean":"2023-11-24T08:21:53.423085+0000","last_became_active":"2023-11-24T08:21:50.575535+0000","last_became_peered":"2023-11-24T08:21:50.575535+0000","last_unstale":"2023-11-24T08:21:53.423085+0000","last_undegraded":"2023-11-24T08:21:53.423085+0000","last_fullsized":"2023-11-24T08:21:53.423085+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:50:54.522055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.594906+0000","last_change":"2023-11-24T08:21:51.591447+0000","last_active":"2023-11-24T08:21:51.594906+0000","last_peered":"2023-11-24T08:21:51.594906+0000","last_clean":"2023-11-24T08:21:51.594906+0000","last_became_active":"2023-11-24T08:21:51.590758+0000","last_became_peered":"2023-11-24T08:21:51.590758+0000","last_unstale":"2023-11-24T08:21:51.594906+0000","last_undegraded":"2023-11-24T08:21:51.594906+0000","last_fullsized":"2023-11-24T08:21:51.594906+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:49.404807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.422778+0000","last_change":"2023-11-24T08:21:50.565008+0000","last_active":"2023-11-24T08:21:53.422778+0000","last_peered":"2023-11-24T08:21:53.422778+0000","last_clean":"2023-11-24T08:21:53.422778+0000","last_became_active":"2023-11-24T08:21:50.564591+0000","last_became_peered":"2023-11-24T08:21:50.564591+0000","last_unstale":"2023-11-24T08:21:53.422778+0000","last_undegraded":"2023-11-24T08:21:53.422778+0000","last_fullsized":"2023-11-24T08:21:53.422778+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:20.980206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.592165+0000","last_change":"2023-11-24T08:21:51.590026+0000","last_active":"2023-11-24T08:21:51.592165+0000","last_peered":"2023-11-24T08:21:51.592165+0000","last_clean":"2023-11-24T08:21:51.592165+0000","last_became_active":"2023-11-24T08:21:51.589768+0000","last_became_peered":"2023-11-24T08:21:51.589768+0000","last_unstale":"2023-11-24T08:21:51.592165+0000","last_undegraded":"2023-11-24T08:21:51.592165+0000","last_fullsized":"2023-11-24T08:21:51.592165+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:29:18.731436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.555060+0000","last_change":"2023-11-24T08:21:50.571240+0000","last_active":"2023-11-24T08:21:51.555060+0000","last_peered":"2023-11-24T08:21:51.555060+0000","last_clean":"2023-11-24T08:21:51.555060+0000","last_became_active":"2023-11-24T08:21:50.570965+0000","last_became_peered":"2023-11-24T08:21:50.570965+0000","last_unstale":"2023-11-24T08:21:51.555060+0000","last_undegraded":"2023-11-24T08:21:51.555060+0000","last_fullsized":"2023-11-24T08:21:51.555060+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:11:32.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.584492+0000","last_change":"2023-11-24T08:21:51.580901+0000","last_active":"2023-11-24T08:21:51.584492+0000","last_peered":"2023-11-24T08:21:51.584492+0000","last_clean":"2023-11-24T08:21:51.584492+0000","last_became_active":"2023-11-24T08:21:51.580559+0000","last_became_peered":"2023-11-24T08:21:51.580559+0000","last_unstale":"2023-11-24T08:21:51.584492+0000","last_undegraded":"2023-11-24T08:21:51.584492+0000","last_fullsized":"2023-11-24T08:21:51.584492+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:27:21.698335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"206'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587627+0000","last_change":"2023-11-24T08:21:50.583636+0000","last_active":"2023-11-24T08:21:52.587627+0000","last_peered":"2023-11-24T08:21:52.587627+0000","last_clean":"2023-11-24T08:21:52.587627+0000","last_became_active":"2023-11-24T08:21:50.583281+0000","last_became_peered":"2023-11-24T08:21:50.583281+0000","last_unstale":"2023-11-24T08:21:52.587627+0000","last_undegraded":"2023-11-24T08:21:52.587627+0000","last_fullsized":"2023-11-24T08:21:52.587627+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:51:17.234565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598408+0000","last_change":"2023-11-24T08:21:51.588934+0000","last_active":"2023-11-24T08:21:51.598408+0000","last_peered":"2023-11-24T08:21:51.598408+0000","last_clean":"2023-11-24T08:21:51.598408+0000","last_became_active":"2023-11-24T08:21:51.588357+0000","last_became_peered":"2023-11-24T08:21:51.588357+0000","last_unstale":"2023-11-24T08:21:51.598408+0000","last_undegraded":"2023-11-24T08:21:51.598408+0000","last_fullsized":"2023-11-24T08:21:51.598408+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:26:36.358037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423808+0000","last_change":"2023-11-24T08:21:50.567569+0000","last_active":"2023-11-24T08:21:53.423808+0000","last_peered":"2023-11-24T08:21:53.423808+0000","last_clean":"2023-11-24T08:21:53.423808+0000","last_became_active":"2023-11-24T08:21:50.566563+0000","last_became_peered":"2023-11-24T08:21:50.566563+0000","last_unstale":"2023-11-24T08:21:53.423808+0000","last_undegraded":"2023-11-24T08:21:53.423808+0000","last_fullsized":"2023-11-24T08:21:53.423808+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:25.992721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.586302+0000","last_change":"2023-11-24T08:21:51.597897+0000","last_active":"2023-11-24T08:21:52.586302+0000","last_peered":"2023-11-24T08:21:52.586302+0000","last_clean":"2023-11-24T08:21:52.586302+0000","last_became_active":"2023-11-24T08:21:51.592609+0000","last_became_peered":"2023-11-24T08:21:51.592609+0000","last_unstale":"2023-11-24T08:21:52.586302+0000","last_undegraded":"2023-11-24T08:21:52.586302+0000","last_fullsized":"2023-11-24T08:21:52.586302+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:50.811886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:53.423707+0000","last_change":"2023-11-24T08:21:50.568397+0000","last_active":"2023-11-24T08:21:53.423707+0000","last_peered":"2023-11-24T08:21:53.423707+0000","last_clean":"2023-11-24T08:21:53.423707+0000","last_became_active":"2023-11-24T08:21:50.565995+0000","last_became_peered":"2023-11-24T08:21:50.565995+0000","last_unstale":"2023-11-24T08:21:53.423707+0000","last_undegraded":"2023-11-24T08:21:53.423707+0000","last_fullsized":"2023-11-24T08:21:53.423707+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:29.579582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.579591+0000","last_change":"2023-11-24T08:21:51.574685+0000","last_active":"2023-11-24T08:21:51.579591+0000","last_peered":"2023-11-24T08:21:51.579591+0000","last_clean":"2023-11-24T08:21:51.579591+0000","last_became_active":"2023-11-24T08:21:51.572557+0000","last_became_peered":"2023-11-24T08:21:51.572557+0000","last_unstale":"2023-11-24T08:21:51.579591+0000","last_undegraded":"2023-11-24T08:21:51.579591+0000","last_fullsized":"2023-11-24T08:21:51.579591+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:31:50.466439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585598+0000","last_change":"2023-11-24T08:21:50.584238+0000","last_active":"2023-11-24T08:21:52.585598+0000","last_peered":"2023-11-24T08:21:52.585598+0000","last_clean":"2023-11-24T08:21:52.585598+0000","last_became_active":"2023-11-24T08:21:50.583939+0000","last_became_peered":"2023-11-24T08:21:50.583939+0000","last_unstale":"2023-11-24T08:21:52.585598+0000","last_undegraded":"2023-11-24T08:21:52.585598+0000","last_fullsized":"2023-11-24T08:21:52.585598+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:08:49.637900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.592453+0000","last_change":"2023-11-24T08:21:51.590953+0000","last_active":"2023-11-24T08:21:51.592453+0000","last_peered":"2023-11-24T08:21:51.592453+0000","last_clean":"2023-11-24T08:21:51.592453+0000","last_became_active":"2023-11-24T08:21:51.590410+0000","last_became_peered":"2023-11-24T08:21:51.590410+0000","last_unstale":"2023-11-24T08:21:51.592453+0000","last_undegraded":"2023-11-24T08:21:51.592453+0000","last_fullsized":"2023-11-24T08:21:51.592453+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:21:11.394560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587574+0000","last_change":"2023-11-24T08:21:50.585948+0000","last_active":"2023-11-24T08:21:52.587574+0000","last_peered":"2023-11-24T08:21:52.587574+0000","last_clean":"2023-11-24T08:21:52.587574+0000","last_became_active":"2023-11-24T08:21:50.582684+0000","last_became_peered":"2023-11-24T08:21:50.582684+0000","last_unstale":"2023-11-24T08:21:52.587574+0000","last_undegraded":"2023-11-24T08:21:52.587574+0000","last_fullsized":"2023-11-24T08:21:52.587574+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:22:03.718601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.589912+0000","last_change":"2023-11-24T08:21:51.588761+0000","last_active":"2023-11-24T08:21:51.589912+0000","last_peered":"2023-11-24T08:21:51.589912+0000","last_clean":"2023-11-24T08:21:51.589912+0000","last_became_active":"2023-11-24T08:21:51.588490+0000","last_became_peered":"2023-11-24T08:21:51.588490+0000","last_unstale":"2023-11-24T08:21:51.589912+0000","last_undegraded":"2023-11-24T08:21:51.589912+0000","last_fullsized":"2023-11-24T08:21:51.589912+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:40.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.562621+0000","last_change":"2023-11-24T08:21:50.572673+0000","last_active":"2023-11-24T08:21:51.562621+0000","last_peered":"2023-11-24T08:21:51.562621+0000","last_clean":"2023-11-24T08:21:51.562621+0000","last_became_active":"2023-11-24T08:21:50.572373+0000","last_became_peered":"2023-11-24T08:21:50.572373+0000","last_unstale":"2023-11-24T08:21:51.562621+0000","last_undegraded":"2023-11-24T08:21:51.562621+0000","last_fullsized":"2023-11-24T08:21:51.562621+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:20:29.577576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.576033+0000","last_change":"2023-11-24T08:21:51.572024+0000","last_active":"2023-11-24T08:21:51.576033+0000","last_peered":"2023-11-24T08:21:51.576033+0000","last_clean":"2023-11-24T08:21:51.576033+0000","last_became_active":"2023-11-24T08:21:51.571757+0000","last_became_peered":"2023-11-24T08:21:51.571757+0000","last_unstale":"2023-11-24T08:21:51.576033+0000","last_undegraded":"2023-11-24T08:21:51.576033+0000","last_fullsized":"2023-11-24T08:21:51.576033+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:16.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.571395+0000","last_change":"2023-11-24T08:21:50.579323+0000","last_active":"2023-11-24T08:21:51.571395+0000","last_peered":"2023-11-24T08:21:51.571395+0000","last_clean":"2023-11-24T08:21:51.571395+0000","last_became_active":"2023-11-24T08:21:50.578991+0000","last_became_peered":"2023-11-24T08:21:50.578991+0000","last_unstale":"2023-11-24T08:21:51.571395+0000","last_undegraded":"2023-11-24T08:21:51.571395+0000","last_fullsized":"2023-11-24T08:21:51.571395+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:43.379727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.587398+0000","last_change":"2023-11-24T08:21:51.603204+0000","last_active":"2023-11-24T08:21:52.587398+0000","last_peered":"2023-11-24T08:21:52.587398+0000","last_clean":"2023-11-24T08:21:52.587398+0000","last_became_active":"2023-11-24T08:21:51.602933+0000","last_became_peered":"2023-11-24T08:21:51.602933+0000","last_unstale":"2023-11-24T08:21:52.587398+0000","last_undegraded":"2023-11-24T08:21:52.587398+0000","last_fullsized":"2023-11-24T08:21:52.587398+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:20:23.943409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-11-24T08:21:52.585683+0000","last_change":"2023-11-24T08:21:50.583512+0000","last_active":"2023-11-24T08:21:52.585683+0000","last_peered":"2023-11-24T08:21:52.585683+0000","last_clean":"2023-11-24T08:21:52.585683+0000","last_became_active":"2023-11-24T08:21:50.583180+0000","last_became_peered":"2023-11-24T08:21:50.583180+0000","last_unstale":"2023-11-24T08:21:52.585683+0000","last_undegraded":"2023-11-24T08:21:52.585683+0000","last_fullsized":"2023-11-24T08:21:52.585683+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:41.172733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.591202+0000","last_change":"2023-11-24T08:21:51.589982+0000","last_active":"2023-11-24T08:21:51.591202+0000","last_peered":"2023-11-24T08:21:51.591202+0000","last_clean":"2023-11-24T08:21:51.591202+0000","last_became_active":"2023-11-24T08:21:51.589694+0000","last_became_peered":"2023-11-24T08:21:51.589694+0000","last_unstale":"2023-11-24T08:21:51.591202+0000","last_undegraded":"2023-11-24T08:21:51.591202+0000","last_fullsized":"2023-11-24T08:21:51.591202+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:50.530393+0000","last_clean_scrub_stamp":"2023-11-24T08:21:50.530393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:39:19.313081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"206'2","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-11-24T08:21:51.598053+0000","last_change":"2023-11-24T08:21:50.580872+0000","last_active":"2023-11-24T08:21:51.598053+0000","last_peered":"2023-11-24T08:21:51.598053+0000","last_clean":"2023-11-24T08:21:51.598053+0000","last_became_active":"2023-11-24T08:21:50.580534+0000","last_became_peered":"2023-11-24T08:21:50.580534+0000","last_unstale":"2023-11-24T08:21:51.598053+0000","last_undegraded":"2023-11-24T08:21:51.598053+0000","last_fullsized":"2023-11-24T08:21:51.598053+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:21:49.513694+0000","last_clean_scrub_stamp":"2023-11-24T08:21:49.513694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:22:29.329334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":329,"reported_epoch":205,"state":"active+clean","last_fresh":"2023-11-24T08:21:50.570895+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:21:50.570895+0000","last_peered":"2023-11-24T08:21:50.570895+0000","last_clean":"2023-11-24T08:21:50.570895+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:21:50.570895+0000","last_undegraded":"2023-11-24T08:21:50.570895+0000","last_fullsized":"2023-11-24T08:21:50.570895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575058,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42628,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329212,"statfs":{"total":96636764160,"available":96593113088,"internally_reserved":0,"allocated":3203072,"data_stored":1746097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.071,"15min":1.069},"min":{"1min":0.622,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":2.048,"5min":2.048,"15min":2.7240000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.1220000000000001,"15min":1.1100000000000001},"min":{"1min":0.71699999999999997,"5min":0.67100000000000004,"15min":0.53700000000000003},"max":{"1min":1.899,"5min":1.899,"15min":2.7109999999999999},"last":0.87}]},{"osd":1,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.127,"15min":1.159},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.681,"5min":1.681,"15min":3.8519999999999999},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.1950000000000001,"15min":1.2250000000000001},"min":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.628},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.2569999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1060000000000001,"15min":1.121},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.55700000000000005},"max":{"1min":1.762,"5min":1.762,"15min":2.4430000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2609999999999999,"15min":1.302},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.60399999999999998},"max":{"1min":1.5629999999999999,"5min":1.7310000000000001,"15min":4.0650000000000004},"last":1.2949999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1910000000000001,"15min":1.2170000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.504},"max":{"1min":1.611,"5min":1.645,"15min":3.133},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.329,"15min":1.369},"min":{"1min":0.97299999999999998,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.77,"5min":2.173,"15min":3.9209999999999998},"last":1.1759999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.323,"15min":1.3540000000000001},"min":{"1min":0.84099999999999997,"5min":0.83699999999999997,"15min":0.54700000000000004},"max":{"1min":1.752,"5min":1.879,"15min":2.851},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.355,"15min":1.357},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.7130000000000001,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.117}]},{"osd":5,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4450000000000001,"15min":1.4750000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4470000000000001,"15min":1.4950000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":1.845,"5min":1.988,"15min":3.4239999999999999},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.548,"15min":1.5760000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.9059999999999999,"5min":1.994,"15min":2.992},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.484,"15min":1.51},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.827,"5min":2.4159999999999999,"15min":3.798},"last":1.2210000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37920,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333920,"statfs":{"total":96636764160,"available":96597934080,"internally_reserved":0,"allocated":2637824,"data_stored":1163095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.68300000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.756,"5min":1.7929999999999999,"15min":1.95},"last":1.155},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.089,"15min":1.083},"min":{"1min":0.624,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.619,"5min":2.008,"15min":2.008},"last":1.379}]},{"osd":1,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.097,"15min":1.1419999999999999},"min":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.61599999999999999},"max":{"1min":1.625,"5min":1.8100000000000001,"15min":2.0230000000000001},"last":1.327},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.2,"15min":1.1819999999999999},"min":{"1min":0.745,"5min":0.70999999999999996,"15min":0.65000000000000002},"max":{"1min":1.5109999999999999,"5min":2.206,"15min":2.4039999999999999},"last":1.6699999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1060000000000001,"15min":1.101},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.6699999999999999,"5min":1.7090000000000001,"15min":1.9330000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2350000000000001,"15min":1.254},"min":{"1min":0.90100000000000002,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.577,"5min":1.7849999999999999,"15min":2.0369999999999999},"last":1.518}]},{"osd":3,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3340000000000001,"15min":1.337},"min":{"1min":0.83799999999999997,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":1.823,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2769999999999999,"15min":1.302},"min":{"1min":0.92300000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8740000000000001,"5min":1.907,"15min":4.8319999999999999},"last":1.462}]},{"osd":4,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4099999999999999,"15min":1.413},"min":{"1min":0.79300000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.6870000000000001,"5min":3.673,"15min":3.673},"last":1.641},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4810000000000001,"15min":1.4830000000000001},"min":{"1min":1.1160000000000001,"5min":0.88,"15min":0.63800000000000001},"max":{"1min":1.913,"5min":2.8690000000000002,"15min":2.8820000000000001},"last":1.9850000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.379,"15min":1.409},"min":{"1min":1.109,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.782,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.22},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5720000000000001,"15min":1.5589999999999999},"min":{"1min":1.05,"5min":0.90100000000000002,"15min":0.83299999999999996},"max":{"1min":1.8500000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.73}]},{"osd":7,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.605,"15min":1.5860000000000001},"min":{"1min":0.94899999999999995,"5min":0.93500000000000005,"15min":0.748},"max":{"1min":1.986,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":1.552,"5min":1.5940000000000001,"15min":1.5860000000000001},"min":{"1min":1.1930000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.0409999999999999,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.613}]}]},{"osd":5,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37916,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333924,"statfs":{"total":96636764160,"available":96597938176,"internally_reserved":0,"allocated":2629632,"data_stored":1163039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.02,"15min":1.014},"min":{"1min":0.65300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.357,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.143},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.115,"15min":1.105},"min":{"1min":0.66100000000000003,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6319999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":0.92000000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1519999999999999,"15min":1.123},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.585,"5min":7.258,"15min":7.258},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.216,"15min":1.1870000000000001},"min":{"1min":0.79600000000000004,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":1.383}]},{"osd":2,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2130000000000001,"15min":1.1830000000000001},"min":{"1min":0.88900000000000001,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.7,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.2050000000000001,"15min":1.1719999999999999},"min":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.923,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.2649999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2549999999999999,"15min":1.179},"min":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.9670000000000001,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.335,"15min":1.276},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.55200000000000005},"max":{"1min":1.8580000000000001,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.347}]},{"osd":4,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4770000000000001,"15min":1.3779999999999999},"min":{"1min":0.92200000000000004,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":2.536,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.552,"15min":1.478},"min":{"1min":0.88100000000000001,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":2.4550000000000001,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.4370000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3740000000000001,"15min":1.341},"min":{"1min":0.81899999999999995,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":2.246,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.53,"15min":1.452},"min":{"1min":0.92600000000000005,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":2.3519999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.508}]},{"osd":7,"last update":"Fri Nov 24 08:21:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.536,"15min":1.494},"min":{"1min":0.94299999999999995,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":2.802,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":0.877},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.762,"15min":1.645},"min":{"1min":1.325,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.9489999999999998,"5min":19.663,"15min":19.663},"last":1.5940000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575059,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42044,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329796,"statfs":{"total":96636764160,"available":96593711104,"internally_reserved":0,"allocated":2605056,"data_stored":1155147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.044,"15min":1.0549999999999999},"min":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.42699999999999999},"max":{"1min":1.5600000000000001,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0609999999999999,"15min":1.079},"min":{"1min":0.79000000000000004,"5min":0.60599999999999998,"15min":0.46100000000000002},"max":{"1min":1.47,"5min":1.7070000000000001,"15min":5.452},"last":1.282}]},{"osd":1,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.119,"15min":1.1319999999999999},"min":{"1min":0.72199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6080000000000001,"5min":1.708,"15min":1.8100000000000001},"last":1.419},{"interface":"front","average":{"1min":1.155,"5min":1.123,"15min":1.153},"min":{"1min":0.751,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.468,"5min":1.7150000000000001,"15min":1.988},"last":1.157}]},{"osd":2,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.66100000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5209999999999999,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":1.107},{"interface":"front","average":{"1min":1.294,"5min":1.2490000000000001,"15min":1.2330000000000001},"min":{"1min":0.88500000000000001,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.714,"5min":1.827,"15min":2.0470000000000002},"last":1.4750000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3360000000000001,"15min":1.3280000000000001},"min":{"1min":1.159,"5min":0.754,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.925,"15min":2.121},"last":1.349},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.266,"15min":1.2709999999999999},"min":{"1min":0.873,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.538,"5min":1.7290000000000001,"15min":2.3340000000000001},"last":1.3779999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.5069999999999999,"15min":1.4890000000000001},"min":{"1min":1.266,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.8640000000000001,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.218},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.4139999999999999},"min":{"1min":0.93999999999999995,"5min":0.66200000000000003,"15min":0.60199999999999998},"max":{"1min":1.6759999999999999,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.5049999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.238,"15min":1.2410000000000001},"min":{"1min":0.81399999999999995,"5min":0.629,"15min":0.50900000000000001},"max":{"1min":1.6779999999999999,"5min":1.8540000000000001,"15min":1.907},"last":1.252},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5189999999999999,"15min":1.5229999999999999},"min":{"1min":0.98399999999999999,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.9810000000000001,"5min":2.1190000000000002,"15min":2.2890000000000001},"last":1.5269999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:21:22 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4399999999999999,"15min":1.4750000000000001},"min":{"1min":0.76800000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.9199999999999999,"5min":1.948,"15min":3.1219999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.3939999999999999,"15min":1.3959999999999999},"min":{"1min":0.89700000000000002,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.1190000000000002,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.321}]}]},{"osd":3,"up_from":13,"seq":55834575058,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42048,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329792,"statfs":{"total":96636764160,"available":96593707008,"internally_reserved":0,"allocated":2605056,"data_stored":1155079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.2609999999999999,"5min":1.5820000000000001,"15min":10.557},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.92500000000000004,"15min":0.92600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.5149999999999999,"5min":1.7310000000000001,"15min":3.8260000000000001},"last":1.097}]},{"osd":1,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.2010000000000001,"5min":1.71,"15min":2.101},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.872,"15min":0.96599999999999997},"min":{"1min":0.51500000000000001,"5min":0.5,"15min":0.379},"max":{"1min":1.4199999999999999,"5min":1.452,"15min":13.090999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.875,"15min":0.90600000000000003},"min":{"1min":0.497,"5min":0.433,"15min":0.39900000000000002},"max":{"1min":1.2989999999999999,"5min":1.976,"15min":3.9489999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.099,"5min":0.96999999999999997,"15min":1.0629999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":1.6319999999999999,"5min":1.665,"15min":13.460000000000001},"last":1.1699999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.2769999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.845,"5min":1.877,"15min":4.367},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.2270000000000001,"15min":1.21},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.76,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.353}]},{"osd":5,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2250000000000001,"15min":1.298},"min":{"1min":0.626,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.8149999999999999,"5min":2.1230000000000002,"15min":13.231},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.2150000000000001,"15min":1.244},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.583,"5min":1.8620000000000001,"15min":5.1180000000000003},"last":1.333}]},{"osd":6,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.284,"15min":1.3540000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.923,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.385},{"interface":"front","average":{"1min":1.244,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8140000000000001,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:21:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.2929999999999999,"15min":1.3129999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":1.8029999999999999,"5min":2.0720000000000001,"15min":4.5060000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.2849999999999999,"15min":1.2929999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.75,"5min":3.0030000000000001,"15min":3.331},"last":1.004}]}]},{"osd":2,"up_from":13,"seq":55834575056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42048,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329792,"statfs":{"total":96636764160,"available":96593707008,"internally_reserved":0,"allocated":2609152,"data_stored":1155673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.72699999999999998,"15min":0.76800000000000002},"min":{"1min":0.503,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.601},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.80800000000000005,"15min":0.85499999999999998},"min":{"1min":0.58899999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.2689999999999999,"5min":1.4810000000000001,"15min":1.782},"last":1.042}]},{"osd":1,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.86299999999999999,"15min":0.89200000000000002},"min":{"1min":0.61399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.57799999999999996,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.8360000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.86699999999999999,"15min":0.86299999999999999},"min":{"1min":0.53100000000000003,"5min":0.45000000000000001,"15min":0.42699999999999999},"max":{"1min":1.228,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.996,"15min":1.0229999999999999},"min":{"1min":0.63900000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.8109999999999999,"5min":1.948,"15min":1.948},"last":1.089}]},{"osd":4,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2430000000000001,"15min":1.2649999999999999},"min":{"1min":0.90600000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.7350000000000001,"5min":1.782,"15min":2.1379999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.262,"5min":1.2050000000000001,"15min":1.2450000000000001},"min":{"1min":0.89200000000000002,"5min":0.78300000000000003,"15min":0.745},"max":{"1min":1.48,"5min":1.8200000000000001,"15min":3.3279999999999998},"last":1.4179999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.2,"15min":1.2390000000000001},"min":{"1min":0.77200000000000002,"5min":0.66100000000000003,"15min":0.61499999999999999},"max":{"1min":1.7649999999999999,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.282},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.234,"15min":1.2410000000000001},"min":{"1min":1.0820000000000001,"5min":0.752,"15min":0.499},"max":{"1min":1.841,"5min":1.841,"15min":2.0910000000000002},"last":1.538}]},{"osd":6,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.1830000000000001,"15min":1.196},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":3.2210000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.28,"15min":1.3140000000000001},"min":{"1min":0.84099999999999997,"5min":0.70099999999999996,"15min":0.65600000000000003},"max":{"1min":1.8300000000000001,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.347}]},{"osd":7,"last update":"Fri Nov 24 08:21:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.29,"15min":1.3140000000000001},"min":{"1min":0.89100000000000001,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.381},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.27,"15min":1.2849999999999999},"min":{"1min":0.78300000000000003,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.994,"5min":2.1539999999999999,"15min":3.2730000000000001},"last":1.321}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42052,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329788,"statfs":{"total":96636764160,"available":96593702912,"internally_reserved":0,"allocated":2600960,"data_stored":1155057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.383,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.01,"5min":0.92700000000000005,"15min":0.89900000000000002},"min":{"1min":0.53000000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.637,"5min":1.6919999999999999,"15min":2.1619999999999999},"last":1.7350000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.82599999999999996,"15min":0.81299999999999994},"min":{"1min":0.436,"5min":0.436,"15min":0.37},"max":{"1min":1.2350000000000001,"5min":1.823,"15min":1.861},"last":0.877},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.92400000000000004,"15min":0.92500000000000004},"min":{"1min":0.51500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.357,"5min":1.7490000000000001,"15min":1.925},"last":1.3180000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.046,"15min":1.012},"min":{"1min":0.55400000000000005,"5min":0.41199999999999998,"15min":0.40699999999999997},"max":{"1min":1.528,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.91200000000000003,"15min":0.91900000000000004},"min":{"1min":0.5,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.3959999999999999,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":1.444}]},{"osd":4,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.304,"15min":1.286},"min":{"1min":0.92700000000000005,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.6719999999999999,"5min":1.984,"15min":2.7959999999999998},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.284,"15min":1.262},"min":{"1min":0.98099999999999998,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":1.9410000000000001,"15min":2.8300000000000001},"last":1.8420000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2490000000000001,"15min":1.2589999999999999},"min":{"1min":0.84799999999999998,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.7030000000000001,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":1.524},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2390000000000001,"15min":1.2210000000000001},"min":{"1min":1.073,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":3.286},"last":1.6919999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2609999999999999,"15min":1.28},"min":{"1min":0.95299999999999996,"5min":0.76400000000000001,"15min":0.52400000000000002},"max":{"1min":1.762,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.631},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.284,"15min":1.278},"min":{"1min":0.81299999999999994,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.706,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.774}]},{"osd":7,"last update":"Fri Nov 24 08:21:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.323,"15min":1.3180000000000001},"min":{"1min":0.89400000000000002,"5min":0.82299999999999995,"15min":0.59099999999999997},"max":{"1min":1.865,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.177},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.327,"15min":1.337},"min":{"1min":0.96799999999999997,"5min":0.72299999999999998,"15min":0.624},"max":{"1min":1.8140000000000001,"5min":1.9930000000000001,"15min":3.653},"last":1.8260000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42620,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329220,"statfs":{"total":96636764160,"available":96593121280,"internally_reserved":0,"allocated":3194880,"data_stored":1745425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.73299999999999998},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.4910000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.4490000000000001,"5min":1.5189999999999999,"15min":2.3620000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.0209999999999999,"15min":1.012},"min":{"1min":0.54900000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.893,"15min":1.893},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.86699999999999999},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.571,"5min":1.798,"15min":2.4990000000000001},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93899999999999995,"15min":0.97199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86199999999999999,"15min":0.88500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.40100000000000002},"max":{"1min":1.429,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":0.60099999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.212,"15min":1.216},"min":{"1min":0.84799999999999998,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.722,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.331},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.081,"15min":1.109},"min":{"1min":0.76600000000000001,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.417,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1399999999999999,"15min":1.151},"min":{"1min":0.81899999999999995,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.55,"5min":1.6910000000000001,"15min":2.6850000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.129,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.4890000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1040000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.256},"min":{"1min":0.94299999999999995,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.587,"5min":1.8560000000000001,"15min":2.052},"last":1.077},{"interface":"front","average":{"1min":1.222,"5min":1.2290000000000001,"15min":1.2529999999999999},"min":{"1min":0.84799999999999998,"5min":0.76700000000000002,"15min":0.70899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8440000000000001,"15min":2.823},"last":1.139}]},{"osd":7,"last update":"Fri Nov 24 08:21:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.319,"15min":1.3400000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6379999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2889999999999999,"15min":1.306},"min":{"1min":0.95099999999999996,"5min":0.79600000000000004,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:21:56.498 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:21:56.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:21:56.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.756+0000 7f212bd78640 1 Processor -- start 2023-11-24T08:21:56.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.756+0000 7f212bd78640 1 -- start start 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212bd78640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f21240cc0f0 0x7f21240c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212bd78640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21240c57e0 0x7f212406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212bd78640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212415eea0 con 0x7f212415b440 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212bd78640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212415b820 con 0x7f21240c57e0 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212bd78640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21240abf30 con 0x7f21240cc0f0 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212a315640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21240c57e0 0x7f212406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212a315640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21240c57e0 0x7f212406c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49512/0 (socket says 172.21.15.107:49512) 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.758+0000 7f212a315640 1 -- 172.21.15.107:0/3835100759 learned_addr learned my addr 172.21.15.107:0/3835100759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.759+0000 7f212a315640 1 -- 172.21.15.107:0/3835100759 >> v2:172.21.15.120:3300/0 conn(0x7f21240cc0f0 msgr2=0x7f21240c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.759+0000 7f212a315640 1 --2- 172.21.15.107:0/3835100759 >> v2:172.21.15.120:3300/0 conn(0x7f21240cc0f0 0x7f21240c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.759+0000 7f212a315640 1 -- 172.21.15.107:0/3835100759 >> v1:172.21.15.120:6789/0 conn(0x7f212415b440 legacy=0x7f21240cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.759+0000 7f212a315640 1 -- 172.21.15.107:0/3835100759 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212415f0f0 con 0x7f21240c57e0 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.760+0000 7f212a315640 1 --2- 172.21.15.107:0/3835100759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f212406c5f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f212406e720 tx=0x7f211c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea88785ba10d5fa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.760+0000 7f2128b12640 1 -- 172.21.15.107:0/3835100759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c02c9f0 con 0x7f21240c57e0 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.760+0000 7f2128b12640 1 -- 172.21.15.107:0/3835100759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c02cb50 con 0x7f21240c57e0 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.760+0000 7f2128b12640 1 -- 172.21.15.107:0/3835100759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c0343f0 con 0x7f21240c57e0 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 -- 172.21.15.107:0/3835100759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 msgr2=0x7f212406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 --2- 172.21.15.107:0/3835100759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f212406c5f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f212406e720 tx=0x7f211c02be50 comp rx=0 tx=0).stop 2023-11-24T08:21:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 -- 172.21.15.107:0/3835100759 shutdown_connections 2023-11-24T08:21:56.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 --2- 172.21.15.107:0/3835100759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f212406c5f0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 --2- 172.21.15.107:0/3835100759 >> v2:172.21.15.120:3300/0 conn(0x7f21240cc0f0 0x7f21240c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 -- 172.21.15.107:0/3835100759 >> 172.21.15.107:0/3835100759 conn(0x7f21240b0c50 msgr2=0x7f21240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:56.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 -- 172.21.15.107:0/3835100759 shutdown_connections 2023-11-24T08:21:56.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.761+0000 7f212bd78640 1 -- 172.21.15.107:0/3835100759 wait complete. 2023-11-24T08:21:56.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.763+0000 7f212bd78640 1 Processor -- start 2023-11-24T08:21:56.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.763+0000 7f212bd78640 1 -- start start 2023-11-24T08:21:56.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21240cc0f0 0x7f2124155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212415b440 0x7f21240ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2124155180 con 0x7f21240c57e0 2023-11-24T08:21:56.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21240ce5d0 con 0x7f21240cc0f0 2023-11-24T08:21:56.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f212bd78640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21240ce820 con 0x7f212415b440 2023-11-24T08:21:56.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f2129b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f2129b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49518/0 (socket says 172.21.15.107:49518) 2023-11-24T08:21:56.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.764+0000 7f2129b14640 1 -- 172.21.15.107:0/2835820115 learned_addr learned my addr 172.21.15.107:0/2835820115 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129313640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21240cc0f0 0x7f2124155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f212a315640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212415b440 0x7f21240ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 -- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212415b440 msgr2=0x7f21240ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212415b440 0x7f21240ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 -- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21240cc0f0 msgr2=0x7f2124155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:56.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21240cc0f0 0x7f2124155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 -- 172.21.15.107:0/2835820115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c009b30 con 0x7f21240c57e0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f2129b14640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f212000b3e0 tx=0x7f212000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120011830 con 0x7f21240c57e0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21200119b0 con 0x7f21240c57e0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.765+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21240ceb00 con 0x7f21240c57e0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.766+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120011d60 con 0x7f21240c57e0 2023-11-24T08:21:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.766+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2124155790 con 0x7f21240c57e0 2023-11-24T08:21:56.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.767+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212404e8b0 con 0x7f21240c57e0 2023-11-24T08:21:56.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.768+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f212001a400 con 0x7f21240c57e0 2023-11-24T08:21:56.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.769+0000 7f211a7fc640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 0x7f210806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:56.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.769+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21200887e0 con 0x7f21240c57e0 2023-11-24T08:21:56.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.769+0000 7f2129313640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 0x7f210806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:56.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.770+0000 7f2129313640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 0x7f210806f280 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f21240cbe30 tx=0x7f2114008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:56.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:56.773+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2120055390 con 0x7f21240c57e0 2023-11-24T08:21:57.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.032+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f212406dc80 con 0x7f21240c57e0 2023-11-24T08:21:57.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.035+0000 7f211a7fc640 1 -- 172.21.15.107:0/2835820115 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2120015020 con 0x7f21240c57e0 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:21:57.038 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:21:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 msgr2=0x7f210806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 0x7f210806f280 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f21240cbe30 tx=0x7f2114008040 comp rx=0 tx=0).stop 2023-11-24T08:21:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 msgr2=0x7f2124153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f212000b3e0 tx=0x7f212000b8a0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 shutdown_connections 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f210806cdc0 0x7f210806f280 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212415b440 0x7f21240ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21240cc0f0 0x7f2124155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 --2- 172.21.15.107:0/2835820115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21240c57e0 0x7f2124153ec0 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 >> 172.21.15.107:0/2835820115 conn(0x7f21240b0c50 msgr2=0x7f2124070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 shutdown_connections 2023-11-24T08:21:57.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.039+0000 7f212bd78640 1 -- 172.21.15.107:0/2835820115 wait complete. 2023-11-24T08:21:57.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:21:57.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.273+0000 7efdcbdfb640 1 Processor -- start 2023-11-24T08:21:57.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.274+0000 7efdcbdfb640 1 -- start start 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdc415c7f0 0x7efdc406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efdc40cf450 0x7efdc40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc40cdfb0 0x7efdc40ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc40cfe40 con 0x7efdc415c7f0 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc4156010 con 0x7efdc40cf450 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdcbdfb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc406ce30 con 0x7efdc40cdfb0 2023-11-24T08:21:57.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdc9b97640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdc415c7f0 0x7efdc406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdc9b97640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdc415c7f0 0x7efdc406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49524/0 (socket says 172.21.15.107:49524) 2023-11-24T08:21:57.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.275+0000 7efdc9b97640 1 -- 172.21.15.107:0/3547052748 learned_addr learned my addr 172.21.15.107:0/3547052748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:57.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9b97640 1 -- 172.21.15.107:0/3547052748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc40cdfb0 msgr2=0x7efdc40ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:57.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9396640 1 --2- 172.21.15.107:0/3547052748 >> v2:172.21.15.120:3300/0 conn(0x7efdc40cf450 0x7efdc40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9b97640 1 --2- 172.21.15.107:0/3547052748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc40cdfb0 0x7efdc40ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9b97640 1 -- 172.21.15.107:0/3547052748 >> v2:172.21.15.120:3300/0 conn(0x7efdc40cf450 msgr2=0x7efdc40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:57.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9b97640 1 --2- 172.21.15.107:0/3547052748 >> v2:172.21.15.120:3300/0 conn(0x7efdc40cf450 0x7efdc40cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.276+0000 7efdc9b97640 1 -- 172.21.15.107:0/3547052748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc40cece0 con 0x7efdc415c7f0 2023-11-24T08:21:57.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.277+0000 7efdc9b97640 1 --2- 172.21.15.107:0/3547052748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc415c7f0 0x7efdc406db20 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7efdc4070410 tx=0x7efdc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae877e7b48e6826a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.277+0000 7efdc8b95640 1 -- 172.21.15.107:0/3547052748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0031870 con 0x7efdc415c7f0 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.277+0000 7efdc8b95640 1 -- 172.21.15.107:0/3547052748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc00319d0 con 0x7efdc415c7f0 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.277+0000 7efdc8b95640 1 -- 172.21.15.107:0/3547052748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0031d60 con 0x7efdc415c7f0 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 -- 172.21.15.107:0/3547052748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc415c7f0 msgr2=0x7efdc406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/3547052748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc415c7f0 0x7efdc406db20 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7efdc4070410 tx=0x7efdc002b910 comp rx=0 tx=0).stop 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 -- 172.21.15.107:0/3547052748 shutdown_connections 2023-11-24T08:21:57.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/3547052748 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc40cdfb0 0x7efdc40ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/3547052748 >> v2:172.21.15.120:3300/0 conn(0x7efdc40cf450 0x7efdc40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/3547052748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc415c7f0 0x7efdc406db20 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 -- 172.21.15.107:0/3547052748 >> 172.21.15.107:0/3547052748 conn(0x7efdc406a920 msgr2=0x7efdc406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:57.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 -- 172.21.15.107:0/3547052748 shutdown_connections 2023-11-24T08:21:57.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.278+0000 7efdcbdfb640 1 -- 172.21.15.107:0/3547052748 wait complete. 2023-11-24T08:21:57.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.280+0000 7efdcbdfb640 1 Processor -- start 2023-11-24T08:21:57.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.280+0000 7efdcbdfb640 1 -- start start 2023-11-24T08:21:57.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdc40cdfb0 0x7efdc40cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc415c7f0 0x7efdc4156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4156690 con 0x7efdc40cf450 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4156930 con 0x7efdc40cdfb0 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdcbdfb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4156c10 con 0x7efdc415c7f0 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.281+0000 7efdc9396640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49538/0 (socket says 172.21.15.107:49538) 2023-11-24T08:21:57.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 -- 172.21.15.107:0/4212688762 learned_addr learned my addr 172.21.15.107:0/4212688762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 -- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc415c7f0 msgr2=0x7efdc4156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdca398640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc415c7f0 0x7efdc4156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc415c7f0 0x7efdc4156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 -- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdc40cdfb0 msgr2=0x7efdc40cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9b97640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdc40cdfb0 0x7efdc40cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdc40cdfb0 0x7efdc40cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 -- 172.21.15.107:0/4212688762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00094d0 con 0x7efdc40cf450 2023-11-24T08:21:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.282+0000 7efdc9396640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c7f0 tx=0x7efdb400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.283+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4004220 con 0x7efdc40cf450 2023-11-24T08:21:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.283+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb40043a0 con 0x7efdc40cf450 2023-11-24T08:21:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.283+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc4156ef0 con 0x7efdc40cf450 2023-11-24T08:21:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.283+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4007860 con 0x7efdc40cf450 2023-11-24T08:21:57.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.283+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4071490 con 0x7efdc40cf450 2023-11-24T08:21:57.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.284+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc41f4350 con 0x7efdc40cf450 2023-11-24T08:21:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.285+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7efdb4007bd0 con 0x7efdc40cf450 2023-11-24T08:21:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.286+0000 7efdba7fc640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 0x7efda806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.286+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdb4088670 con 0x7efdc40cf450 2023-11-24T08:21:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.286+0000 7efdc9b97640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 0x7efda806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.287+0000 7efdc9b97640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 0x7efda806f280 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7efdc0036000 tx=0x7efdc003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.290+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdc41f4350 con 0x7efdc40cf450 2023-11-24T08:21:57.507 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:21:57.505+0000 7f38b5f3a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:21:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.516+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdc41f4350 con 0x7efdc40cf450 2023-11-24T08:21:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.517+0000 7efdba7fc640 1 -- 172.21.15.107:0/4212688762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4939 (secure 0 0 0) 0x7efdc41f4350 con 0x7efdc40cf450 2023-11-24T08:21:57.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:21:57.520 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":128,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:21:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 128 2023-11-24T08:21:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 msgr2=0x7efda806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 0x7efda806f280 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7efdc0036000 tx=0x7efdc003c040 comp rx=0 tx=0).stop 2023-11-24T08:21:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 msgr2=0x7efdc4157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c7f0 tx=0x7efdb400ccb0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 shutdown_connections 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efda806cdc0 0x7efda806f280 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdc415c7f0 0x7efdc4156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdc40cf450 0x7efdc4157cd0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 --2- 172.21.15.107:0/4212688762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdc40cdfb0 0x7efdc40cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 >> 172.21.15.107:0/4212688762 conn(0x7efdc406a920 msgr2=0x7efdc40ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 shutdown_connections 2023-11-24T08:21:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:57.520+0000 7efdcbdfb640 1 -- 172.21.15.107:0/4212688762 wait complete. 2023-11-24T08:21:57.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:21:51.017422+0000', 'modified': '2023-11-24T08:21:54.073750+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11649}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11649': {'gid': 11649, 'name': 'c', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/2018281423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 2018281423}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2018281423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:21:57.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:21:57.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:21:58.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.167+0000 7f132c69b640 1 Processor -- start 2023-11-24T08:21:58.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.167+0000 7f132c69b640 1 -- start start 2023-11-24T08:21:58.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.168+0000 7f132c69b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13240cc0f0 0x7f13240c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.168+0000 7f132c69b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13240c57e0 0x7f132406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.168+0000 7f132c69b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132415eea0 con 0x7f132415b440 2023-11-24T08:21:58.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f132c69b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132415b820 con 0x7f13240cc0f0 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f132c69b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13240abf30 con 0x7f13240c57e0 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f132a437640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f132415b440 0x7f13240cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40834/0 (socket says 172.21.15.107:40834) 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f132a437640 1 -- 172.21.15.107:0/1191674424 learned_addr learned my addr 172.21.15.107:0/1191674424 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329c36640 1 --2- 172.21.15.107:0/1191674424 >> v2:172.21.15.107:3300/0 conn(0x7f13240cc0f0 0x7f13240c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329435640 1 -- 172.21.15.107:0/1191674424 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202152433 0 0) 0x7f132415eea0 con 0x7f132415b440 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329435640 1 -- 172.21.15.107:0/1191674424 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310003520 con 0x7f132415b440 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329c36640 1 -- 172.21.15.107:0/1191674424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13240c57e0 msgr2=0x7f132406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f132ac38640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13240c57e0 0x7f132406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329c36640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13240c57e0 0x7f132406c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329c36640 1 -- 172.21.15.107:0/1191674424 >> v1:172.21.15.107:6789/0 conn(0x7f132415b440 legacy=0x7f13240cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.169+0000 7f1329c36640 1 -- 172.21.15.107:0/1191674424 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132415f0f0 con 0x7f13240cc0f0 2023-11-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.171+0000 7f1329c36640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c52a0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f13140097a0 tx=0x7f131402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b88fba8ae61aec1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.171+0000 7f1329435640 1 -- 172.21.15.107:0/1191674424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131402e070 con 0x7f13240cc0f0 2023-11-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.171+0000 7f1329435640 1 -- 172.21.15.107:0/1191674424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13140315a0 con 0x7f13240cc0f0 2023-11-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.171+0000 7f1329435640 1 -- 172.21.15.107:0/1191674424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314031930 con 0x7f13240cc0f0 2023-11-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 -- 172.21.15.107:0/1191674424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 msgr2=0x7f13240c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c52a0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f13140097a0 tx=0x7f131402b840 comp rx=0 tx=0).stop 2023-11-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 -- 172.21.15.107:0/1191674424 shutdown_connections 2023-11-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13240c57e0 0x7f132406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 --2- 172.21.15.107:0/1191674424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c52a0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 -- 172.21.15.107:0/1191674424 >> 172.21.15.107:0/1191674424 conn(0x7f13240b0c50 msgr2=0x7f13240b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 -- 172.21.15.107:0/1191674424 shutdown_connections 2023-11-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.172+0000 7f132c69b640 1 -- 172.21.15.107:0/1191674424 wait complete. 2023-11-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.174+0000 7f132c69b640 1 Processor -- start 2023-11-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.174+0000 7f132c69b640 1 -- start start 2023-11-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13240c57e0 0x7f13240c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f132415b440 0x7f13240ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240ccc60 con 0x7f13240cc0f0 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241f6ae0 con 0x7f13240c57e0 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f132c69b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13241f6dc0 con 0x7f132415b440 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46280/0 (socket says 172.21.15.107:46280) 2023-11-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 -- 172.21.15.107:0/1640722196 learned_addr learned my addr 172.21.15.107:0/1640722196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 -- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f132415b440 msgr2=0x7f13240ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f132415b440 0x7f13240ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 -- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13240c57e0 msgr2=0x7f13240c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13240c57e0 0x7f13240c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.175+0000 7f1329c36640 1 -- 172.21.15.107:0/1640722196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314009400 con 0x7f13240cc0f0 2023-11-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f1329c36640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f1314034040 tx=0x7f13140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f131affd640 1 -- 172.21.15.107:0/1640722196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314032840 con 0x7f13240cc0f0 2023-11-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f131affd640 1 -- 172.21.15.107:0/1640722196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1314031f00 con 0x7f13240cc0f0 2023-11-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f131affd640 1 -- 172.21.15.107:0/1640722196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314032be0 con 0x7f13240cc0f0 2023-11-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13241f70a0 con 0x7f13240cc0f0 2023-11-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.176+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240c6680 con 0x7f13240cc0f0 2023-11-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.178+0000 7f131affd640 1 -- 172.21.15.107:0/1640722196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f1314032d40 con 0x7f13240cc0f0 2023-11-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.179+0000 7f131affd640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 0x7f130806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.179+0000 7f131affd640 1 -- 172.21.15.107:0/1640722196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13140aa640 con 0x7f13240cc0f0 2023-11-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.179+0000 7f132a437640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 0x7f130806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.180+0000 7f132f23c740 1 Processor -- start 2023-11-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.180+0000 7f132f23c740 1 -- start start 2023-11-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.180+0000 7f132a437640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 0x7f130806f2b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f1320001570 tx=0x7f1320007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b44460a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b44465e0 0x55b6b448a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b43a5880 con 0x55b6b4446ea0 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b43b4ae0 con 0x55b6b4441ba0 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6b43b21d0 con 0x55b6b44465e0 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132a437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132a437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46290/0 (socket says 172.21.15.107:46290) 2023-11-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.181+0000 7f132a437640 1 -- 172.21.15.107:0/3082962096 learned_addr learned my addr 172.21.15.107:0/3082962096 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 -- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b44465e0 msgr2=0x55b6b448a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b44465e0 0x55b6b448a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 -- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 msgr2=0x55b6b44460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b44460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 -- 172.21.15.107:0/3082962096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6b4375860 con 0x55b6b4446ea0 2023-11-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f132a437640 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f1320005c30 tx=0x7f132000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30a2917f3faf45f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f1307fff640 1 -- 172.21.15.107:0/3082962096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132001b070 con 0x55b6b4446ea0 2023-11-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f1307fff640 1 -- 172.21.15.107:0/3082962096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1320016110 con 0x55b6b4446ea0 2023-11-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.182+0000 7f1307fff640 1 -- 172.21.15.107:0/3082962096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320005e80 con 0x55b6b4446ea0 2023-11-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 -- 172.21.15.107:0/3082962096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 msgr2=0x55b6b4447280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f1320005c30 tx=0x7f132000f000 comp rx=0 tx=0).stop 2023-11-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 -- 172.21.15.107:0/3082962096 shutdown_connections 2023-11-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b44465e0 0x55b6b448a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b44460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 --2- 172.21.15.107:0/3082962096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b4447280 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 -- 172.21.15.107:0/3082962096 >> 172.21.15.107:0/3082962096 conn(0x55b6b4408c70 msgr2=0x55b6b448c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 -- 172.21.15.107:0/3082962096 shutdown_connections 2023-11-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.183+0000 7f132f23c740 1 -- 172.21.15.107:0/3082962096 wait complete. 2023-11-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.184+0000 7f132f23c740 1 -- start start 2023-11-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.184+0000 7f132f23c740 1 Processor -- start 2023-11-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b446b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4446ea0 0x55b6b4484730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b43adac0 con 0x55b6b44465e0 2023-11-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b434a2f0 con 0x55b6b4446ea0 2023-11-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6b4367c70 con 0x55b6b4441ba0 2023-11-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132f23c740 10 client.0 _ll_register_callbacks cb 0x55b6b436c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46296/0 (socket says 172.21.15.107:46296) 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f1329c36640 1 -- 172.21.15.107:0/3331813193 learned_addr learned my addr 172.21.15.107:0/3331813193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f1329c36640 1 -- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 msgr2=0x55b6b446b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.185+0000 7f132a437640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b446b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f132ac38640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4446ea0 0x55b6b4484730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f1329c36640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b446b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f1329c36640 1 -- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4446ea0 msgr2=0x55b6b4484730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f1329c36640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4446ea0 0x55b6b4484730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f1329c36640 1 -- 172.21.15.107:0/3331813193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1310003520 con 0x55b6b44465e0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f1329c36640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f131407ab10 tx=0x7f131407aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96b999588b4a0b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f13057fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131407ae10 con 0x55b6b44465e0 2023-11-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f131407ef60 con 0x55b6b44465e0 2023-11-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6b43a6aa0 con 0x55b6b44465e0 2023-11-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1314083110 con 0x55b6b44465e0 2023-11-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f132f23c740 10 client.11775 fetch_fsmap learned FSMap version 128 2023-11-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x55b6b43ac140 con 0x55b6b44465e0 2023-11-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.186+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6b43a6aa0 con 0x55b6b44465e0 2023-11-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 128) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f131407a300 con 0x55b6b44465e0 2023-11-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 10 client.11775 fetch_fsmap finished waiting for FSMap version 128 2023-11-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 10 client.11775 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 20 client.11775 populate_metadata read hostname 'smithi107' 2023-11-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b446c590 0x55b6b4491990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 4 client.11775 mds_command: new command op to 11649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 --> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6b43a67c0 con 0x55b6b446c590 2023-11-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132a437640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b446c590 0x55b6b4491990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f132a437640 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b446c590 0x55b6b4491990 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f78b8b07f9cfa8 server_cookie=ab4e67a0a3b3d2cd in_seq=0 out_seq=0 2023-11-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.187+0000 7f13057fa640 10 client.11775 ms_handle_connect on v2:172.21.15.107:6836/2018281423 2023-11-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.188+0000 7f13057fa640 1 -- 172.21.15.107:0/3331813193 <== mds.0 v2:172.21.15.107:6836/2018281423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b6b43a67c0 con 0x55b6b446c590 2023-11-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.188+0000 7f13057fa640 10 client.11775 handle_command_reply: tid=0 2023-11-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.188+0000 7f132f23c740 1 client.11775 shutdown 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 msgr2=0x55b6b4493fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f131407ab10 tx=0x7f131407aa80 comp rx=0 tx=0).stop 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 shutdown_connections 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b446c590 0x55b6b4491990 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4446ea0 0x55b6b4484730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b4493fa0 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 --2- 172.21.15.107:0/3331813193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b446b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 >> 172.21.15.107:0/3331813193 conn(0x55b6b4408c70 msgr2=0x55b6b4440b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f13057fa640 0 client.11775 ms_handle_reset on v2:172.21.15.107:6836/2018281423 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 shutdown_connections 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 1 -- 172.21.15.107:0/3331813193 wait complete. 2023-11-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.189+0000 7f132f23c740 20 client.11775 trim_cache size 0 max 16384 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.206+0000 7f132f23c740 1 Processor -- start 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.206+0000 7f132f23c740 1 -- start start 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b44af350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b44465e0 0x55b6b4447f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b4110f20 con 0x55b6b4446ea0 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b43b62f0 con 0x55b6b44465e0 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6b4447280 con 0x55b6b4441ba0 2023-11-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132ac38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132ac38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46312/0 (socket says 172.21.15.107:46312) 2023-11-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.207+0000 7f132ac38640 1 -- 172.21.15.107:0/2038578878 learned_addr learned my addr 172.21.15.107:0/2038578878 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 -- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 msgr2=0x55b6b44af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b44af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 -- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b44465e0 msgr2=0x55b6b4447f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b44465e0 0x55b6b4447f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 -- 172.21.15.107:0/2038578878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6b4349b80 con 0x55b6b4446ea0 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f132ac38640 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x55b6b4442000 tx=0x7f131c00d750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74fbf7f2c0f171d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f1318ff9640 1 -- 172.21.15.107:0/2038578878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0165f0 con 0x55b6b4446ea0 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f1318ff9640 1 -- 172.21.15.107:0/2038578878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c016750 con 0x55b6b4446ea0 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.208+0000 7f1318ff9640 1 -- 172.21.15.107:0/2038578878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c016b00 con 0x55b6b4446ea0 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- 172.21.15.107:0/2038578878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 msgr2=0x55b6b446e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x55b6b4442000 tx=0x7f131c00d750 comp rx=0 tx=0).stop 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- 172.21.15.107:0/2038578878 shutdown_connections 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b4446ea0 0x55b6b446e8d0 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b44465e0 0x55b6b4447f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 --2- 172.21.15.107:0/2038578878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4441ba0 0x55b6b44af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- 172.21.15.107:0/2038578878 >> 172.21.15.107:0/2038578878 conn(0x55b6b4408c70 msgr2=0x55b6b446b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- 172.21.15.107:0/2038578878 shutdown_connections 2023-11-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- 172.21.15.107:0/2038578878 wait complete. 2023-11-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.209+0000 7f132f23c740 1 -- start start 2023-11-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.210+0000 7f132f23c740 1 Processor -- start 2023-11-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.210+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b446e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.210+0000 7f132f23c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f132f23c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4446ea0 0x55b6b449e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f132f23c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b4489690 con 0x55b6b44465e0 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6b44925a0 con 0x55b6b4441ba0 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f132f23c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6b4445ed0 con 0x55b6b4446ea0 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f132f23c740 10 client.0 _ll_register_callbacks cb 0x7f131407acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46314/0 (socket says 172.21.15.107:46314) 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 -- 172.21.15.107:0/3049359547 learned_addr learned my addr 172.21.15.107:0/3049359547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 -- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4446ea0 msgr2=0x55b6b449e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4446ea0 0x55b6b449e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 -- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 msgr2=0x55b6b446e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b446e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 -- 172.21.15.107:0/3049359547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c0089a0 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f1329c36640 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f1314079160 tx=0x7f131402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bbebb245f207c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.211+0000 7f13057fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314083110 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f131407ef60 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1320016110 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7f131407ae10 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 10 client.11781 fetch_fsmap learned FSMap version 128 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x55b6b436f170 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1320016110 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 128) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f13140a2070 con 0x55b6b44465e0 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 10 client.11781 fetch_fsmap finished waiting for FSMap version 128 2023-11-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 10 client.11781 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 20 client.11781 populate_metadata read hostname 'smithi107' 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b4492a40 0x55b6b4492e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 4 client.11781 mds_command: new command op to 11649 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 --> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6b40ae6c0 con 0x55b6b4492a40 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.212+0000 7f132a437640 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b4492a40 0x55b6b4492e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132a437640 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b4492a40 0x55b6b4492e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acdb7ae8c7593f34 server_cookie=2b207f9798b017d4 in_seq=0 out_seq=0 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f13057fa640 10 client.11781 ms_handle_connect on v2:172.21.15.107:6836/2018281423 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f13057fa640 1 -- 172.21.15.107:0/3049359547 <== mds.0 v2:172.21.15.107:6836/2018281423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55b6b40ae6c0 con 0x55b6b4492a40 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f13057fa640 10 client.11781 handle_command_reply: tid=0 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 client.11781 shutdown 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 msgr2=0x55b6b44f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f1314079160 tx=0x7f131402b840 comp rx=0 tx=0).stop 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 shutdown_connections 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:6836/2018281423,v1:172.21.15.107:6837/2018281423] conn(0x55b6b4492a40 0x55b6b4492e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6b4446ea0 0x55b6b449e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b6b44465e0 0x55b6b44f5f30 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 --2- 172.21.15.107:0/3049359547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b4441ba0 0x55b6b446e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f13057fa640 0 client.11781 ms_handle_reset on v2:172.21.15.107:6836/2018281423 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.213+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 >> 172.21.15.107:0/3049359547 conn(0x55b6b4408c70 msgr2=0x55b6b4440d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 shutdown_connections 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132f23c740 1 -- 172.21.15.107:0/3049359547 wait complete. 2023-11-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132f23c740 20 client.11781 trim_cache size 0 max 16384 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 11649, 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6629882790000003, 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 128, 2023-11-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 207, 2023-11-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 207, 2023-11-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.01699208 2023-11-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 msgr2=0x7f130806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 0x7f130806f2b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f1320001570 tx=0x7f1320007930 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 msgr2=0x7f13240c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f1314034040 tx=0x7f13140046c0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 shutdown_connections 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f130806cdf0 0x7f130806f2b0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f132415b440 0x7f13240ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13240cc0f0 0x7f13240c7bc0 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 --2- 172.21.15.107:0/1640722196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13240c57e0 0x7f13240c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.214+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 >> 172.21.15.107:0/1640722196 conn(0x7f13240b0c50 msgr2=0x7f13240707f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.215+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 shutdown_connections 2023-11-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.215+0000 7f132c69b640 1 -- 172.21.15.107:0/1640722196 wait complete. 2023-11-24T08:21:58.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:21:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.430+0000 7f104fc0e640 1 Processor -- start 2023-11-24T08:21:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.430+0000 7f104fc0e640 1 -- start start 2023-11-24T08:21:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.431+0000 7f104fc0e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1048159ce0 0x7f104806de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.431+0000 7f104fc0e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480b3a40 con 0x7f10480b21d0 2023-11-24T08:21:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104fc0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480cdfa0 con 0x7f10480cba40 2023-11-24T08:21:58.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104fc0e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10480cfb60 con 0x7f1048159ce0 2023-11-24T08:21:58.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104d1a9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10480b21d0 0x7f10480cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40838/0 (socket says 172.21.15.107:40838) 2023-11-24T08:21:58.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104d1a9640 1 -- 172.21.15.107:0/730030735 learned_addr learned my addr 172.21.15.107:0/730030735 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:21:58.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104d9aa640 1 --2- 172.21.15.107:0/730030735 >> v2:172.21.15.120:3300/0 conn(0x7f1048159ce0 0x7f104806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885001899 0 0) 0x7f10480b3a40 con 0x7f10480b21d0 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.432+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1030003520 con 0x7f10480b21d0 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35149446 0 0) 0x7f1030003520 con 0x7f10480b21d0 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 >> v2:172.21.15.120:3300/0 conn(0x7f1048159ce0 msgr2=0x7f104806de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 --2- 172.21.15.107:0/730030735 >> v2:172.21.15.120:3300/0 conn(0x7f1048159ce0 0x7f104806de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 >> v1:172.21.15.120:6789/0 conn(0x7f10480cba40 legacy=0x7f1048164af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480cd9e0 con 0x7f10480b21d0 2023-11-24T08:21:58.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f103c001670 con 0x7f10480b21d0 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 >> v1:172.21.15.107:6789/0 conn(0x7f10480b21d0 legacy=0x7f10480cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10300039c0 0x7f1030003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 0x7f1030004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1030005180 0x7f103000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10480cfb60 con 0x7f10300047c0 2023-11-24T08:21:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103000e290 con 0x7f1030005180 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103000e450 con 0x7f10300039c0 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104d1a9640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 0x7f1030004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104d9aa640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10300039c0 0x7f1030003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104e1ab640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1030005180 0x7f103000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104d1a9640 1 -- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10300039c0 msgr2=0x7f1030003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104d1a9640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10300039c0 0x7f1030003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.433+0000 7f104d1a9640 1 -- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1030005180 msgr2=0x7f103000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.434+0000 7f104d1a9640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1030005180 0x7f103000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.434+0000 7f104d1a9640 1 -- 172.21.15.107:0/730030735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1030003520 con 0x7f10300047c0 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.434+0000 7f104d9aa640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10300039c0 0x7f1030003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.434+0000 7f104e1ab640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1030005180 0x7f103000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:58.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.434+0000 7f104e1ab640 1 -- 172.21.15.107:0/730030735 reap_dead start 2023-11-24T08:21:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104d1a9640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 0x7f1030004c40 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c0e0 tx=0x7f10300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9103ec8bed3064c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c005ad0 con 0x7f10300047c0 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c005c30 con 0x7f10300047c0 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104c9a8640 1 -- 172.21.15.107:0/730030735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c02c7f0 con 0x7f10300047c0 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104fc0e640 1 -- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 msgr2=0x7f1030004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104fc0e640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 0x7f1030004c40 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c0e0 tx=0x7f10300057f0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104fc0e640 1 -- 172.21.15.107:0/730030735 shutdown_connections 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104fc0e640 1 --2- 172.21.15.107:0/730030735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10300047c0 0x7f1030004c40 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.435+0000 7f104fc0e640 1 -- 172.21.15.107:0/730030735 >> 172.21.15.107:0/730030735 conn(0x7f104806a920 msgr2=0x7f104806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.436+0000 7f104fc0e640 1 -- 172.21.15.107:0/730030735 shutdown_connections 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.436+0000 7f104fc0e640 1 -- 172.21.15.107:0/730030735 wait complete. 2023-11-24T08:21:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.437+0000 7f104fc0e640 1 Processor -- start 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.437+0000 7f104fc0e640 1 -- start start 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10300047c0 0x7f104813ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1048150d30 0x7f10481511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10481508a0 con 0x7f1048158ad0 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104814e660 con 0x7f10300047c0 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.438+0000 7f104fc0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104814ff70 con 0x7f1048150d30 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46328/0 (socket says 172.21.15.107:46328) 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 -- 172.21.15.107:0/2525223227 learned_addr learned my addr 172.21.15.107:0/2525223227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 -- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1048150d30 msgr2=0x7f10481511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1048150d30 0x7f10481511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 -- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10300047c0 msgr2=0x7f104813ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10300047c0 0x7f104813ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 -- 172.21.15.107:0/2525223227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048150250 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f104d1a9640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f103c000c00 tx=0x7f103c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.439+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c03c040 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.440+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c02cc20 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.440+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c007070 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.440+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1048150530 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.440+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1048098d30 con 0x7f1048158ad0 2023-11-24T08:21:58.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.441+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1048067470 con 0x7f1048158ad0 2023-11-24T08:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.442+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f103c032070 con 0x7f1048158ad0 2023-11-24T08:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.443+0000 7f10467fc640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 0x7f103406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.443+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f103c0a9c20 con 0x7f1048158ad0 2023-11-24T08:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.443+0000 7f104d9aa640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 0x7f103406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.443+0000 7f104d9aa640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 0x7f103406f300 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f10480681f0 tx=0x7f1038009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:58.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.447+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f103c076870 con 0x7f1048158ad0 2023-11-24T08:21:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.677+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f10481500b0 con 0x7f1048158ad0 2023-11-24T08:21:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.678+0000 7f10467fc640 1 -- 172.21.15.107:0/2525223227 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v128) v1 ==== 53+0+73 (secure 0 0 0) 0x7f10481500b0 con 0x7f1048158ad0 2023-11-24T08:21:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:21:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 msgr2=0x7f103406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 0x7f103406f300 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f10480681f0 tx=0x7f1038009750 comp rx=0 tx=0).stop 2023-11-24T08:21:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 msgr2=0x7f10480987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f103c000c00 tx=0x7f103c02e070 comp rx=0 tx=0).stop 2023-11-24T08:21:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 shutdown_connections 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f103406ce40 0x7f103406f300 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1048150d30 0x7f10481511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1048158ad0 0x7f10480987f0 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 --2- 172.21.15.107:0/2525223227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10300047c0 0x7f104813ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 >> 172.21.15.107:0/2525223227 conn(0x7f104806a920 msgr2=0x7f1048155400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 shutdown_connections 2023-11-24T08:21:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:58.681+0000 7f104fc0e640 1 -- 172.21.15.107:0/2525223227 wait complete. 2023-11-24T08:21:58.701 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:21:58.702 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:21:58.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:58.703 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:21:58.750 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:21:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:21:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:21:58.751 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:21:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41352sec preferred_lft 41352sec 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:21:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:21:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.754 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:21:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:21:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:21:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.755 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:21:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:21:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:21:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:21:58.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:58.756 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:21:58.884 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:21:58.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:58.888 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:21:58.943 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:21:58.943 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:21:58.943 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:21:58.944 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:21:58.944 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:21:58.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:58.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:21:58.944 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:21:58.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:21:58.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:21:58.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:21:58.945 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:21:59.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:59.124 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:21:59.175 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:21:59.175 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:21:59.175 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:21:59.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:59.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:21:59.262 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:21:59.263 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:21:59.315 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:21:59.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:21:59.317 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:21:59.370 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:21:59.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:59.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:21:59.505 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:21:59.506 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:21:59.540 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:21:59.540 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:21:59.541 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:21:59.541 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:21:59.541 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:21:59.543 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:21:59.543 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:21:59.544 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:21:59.544 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:21:59.544 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:21:59.547 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:21:59.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:59.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:21:59.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:21:59.635 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:21:59.702 INFO:teuthology.orchestra.run.smithi107.stdout:client6602 2023-11-24T08:21:59.702 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:21:59.702 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:21:59.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:21:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.960+0000 7fb8bb189640 1 Processor -- start 2023-11-24T08:21:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.960+0000 7fb8bb189640 1 -- start start 2023-11-24T08:21:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8bb189640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 0x7fb8b415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8bb189640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8b40b2630 0x7fb8b40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8bb189640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b415ab80 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8bb189640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b415a990 con 0x7fb8b40b2630 2023-11-24T08:21:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8bb189640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b415e680 con 0x7fb8b40b2250 2023-11-24T08:21:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8b3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8b40b2630 0x7fb8b40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8b3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8b40b2630 0x7fb8b40ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46344/0 (socket says 172.21.15.107:46344) 2023-11-24T08:21:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.963+0000 7fb8b3fff640 1 -- 172.21.15.107:0/2392131891 learned_addr learned my addr 172.21.15.107:0/2392131891 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/2392131891 >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 0x7fb8b415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b3fff640 1 -- 172.21.15.107:0/2392131891 >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 msgr2=0x7fb8b415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b3fff640 1 --2- 172.21.15.107:0/2392131891 >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 0x7fb8b415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b3fff640 1 -- 172.21.15.107:0/2392131891 >> v1:172.21.15.120:6789/0 conn(0x7fb8b40b2aa0 legacy=0x7fb8b415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:21:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b3fff640 1 -- 172.21.15.107:0/2392131891 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b415e960 con 0x7fb8b40b2630 2023-11-24T08:21:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/2392131891 >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 0x7fb8b415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.964+0000 7fb8b3fff640 1 --2- 172.21.15.107:0/2392131891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2630 0x7fb8b40ccf30 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4072a70 tx=0x7fb8a402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14e9f11fc23fadc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.965+0000 7fb8b37fe640 1 -- 172.21.15.107:0/2392131891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4036440 con 0x7fb8b40b2630 2023-11-24T08:21:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.965+0000 7fb8b37fe640 1 -- 172.21.15.107:0/2392131891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a40365a0 con 0x7fb8b40b2630 2023-11-24T08:21:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.965+0000 7fb8b37fe640 1 -- 172.21.15.107:0/2392131891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4036930 con 0x7fb8b40b2630 2023-11-24T08:21:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 -- 172.21.15.107:0/2392131891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2630 msgr2=0x7fb8b40ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 --2- 172.21.15.107:0/2392131891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2630 0x7fb8b40ccf30 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4072a70 tx=0x7fb8a402e170 comp rx=0 tx=0).stop 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 -- 172.21.15.107:0/2392131891 shutdown_connections 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 --2- 172.21.15.107:0/2392131891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2630 0x7fb8b40ccf30 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 --2- 172.21.15.107:0/2392131891 >> v2:172.21.15.120:3300/0 conn(0x7fb8b40b2250 0x7fb8b415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 -- 172.21.15.107:0/2392131891 >> 172.21.15.107:0/2392131891 conn(0x7fb8b406a920 msgr2=0x7fb8b406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:21:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 -- 172.21.15.107:0/2392131891 shutdown_connections 2023-11-24T08:21:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.966+0000 7fb8bb189640 1 -- 172.21.15.107:0/2392131891 wait complete. 2023-11-24T08:21:59.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.967+0000 7fb8bb189640 1 Processor -- start 2023-11-24T08:21:59.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.968+0000 7fb8bb189640 1 -- start start 2023-11-24T08:21:59.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.968+0000 7fb8bb189640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 0x7fb8b414d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.968+0000 7fb8bb189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8b40b2630 0x7fb8b40aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8bb189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8bb189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40acf30 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8bb189640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40ad210 con 0x7fb8b40b2250 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8bb189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40ad4f0 con 0x7fb8b40b2630 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46348/0 (socket says 172.21.15.107:46348) 2023-11-24T08:21:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 -- 172.21.15.107:0/3695515413 learned_addr learned my addr 172.21.15.107:0/3695515413 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 -- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8b40b2630 msgr2=0x7fb8b40aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8b40b2630 0x7fb8b40aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 0x7fb8b414d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 -- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 msgr2=0x7fb8b414d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 0x7fb8b414d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:21:59.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b9726640 1 -- 172.21.15.107:0/3695515413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a400b770 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.969+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 0x7fb8b414d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8b9726640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c730 tx=0x7fb8ac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003b70 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac003cf0 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003ea0 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b40ad7d0 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.970+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b414dd80 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.971+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b40adcf0 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.972+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fb8ac005310 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.973+0000 7fb8b17fa640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 0x7fb8a006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:21:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.973+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8ac088400 con 0x7fb8b40b2aa0 2023-11-24T08:21:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.973+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 0x7fb8a006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:21:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.974+0000 7fb8b8f25640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 0x7fb8a006f280 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:21:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:21:59.978+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8b40adcf0 con 0x7fb8b40b2aa0 2023-11-24T08:22:00.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.210+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb8b40adcf0 con 0x7fb8b40b2aa0 2023-11-24T08:22:00.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.211+0000 7fb8b17fa640 1 -- 172.21.15.107:0/3695515413 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v128) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8b40adcf0 con 0x7fb8b40b2aa0 2023-11-24T08:22:00.212 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:22:00.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 msgr2=0x7fb8a006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:00.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 0x7fb8a006f280 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).stop 2023-11-24T08:22:00.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 msgr2=0x7fb8b40adfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c730 tx=0x7fb8ac00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 shutdown_connections 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb8a006cdc0 0x7fb8a006f280 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8b40b2aa0 0x7fb8b40adfd0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8b40b2630 0x7fb8b40aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.213+0000 7fb8bb189640 1 --2- 172.21.15.107:0/3695515413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8b40b2250 0x7fb8b414d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.214+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 >> 172.21.15.107:0/3695515413 conn(0x7fb8b406a920 msgr2=0x7fb8b4159170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.214+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 shutdown_connections 2023-11-24T08:22:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:00.214+0000 7fb8bb189640 1 -- 172.21.15.107:0/3695515413 wait complete. 2023-11-24T08:22:00.233 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:22:00.233 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:22:00.234 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:22:00.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.234 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:22:00.296 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:22:00.297 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41349sec preferred_lft 41349sec 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:22:00.298 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.299 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:22:00.299 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:22:00.299 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:22:00.299 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:00.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.299 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:22:00.392 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:22:00.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.397 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:22:00.450 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:22:00.450 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:22:00.450 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:22:00.450 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:22:00.451 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:22:00.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:22:00.451 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:22:00.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.629 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:22:00.677 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:22:00.678 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:22:00.680 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:22:00.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:22:00.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:22:00.776 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:22:00.833 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:22:00.833 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:22:00.834 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:22:00.885 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:22:00.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:00.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:01.022 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:01.022 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:22:01.056 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:01.057 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:22:01.057 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:22:01.057 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:22:01.057 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:22:01.058 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:22:01.059 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:22:01.059 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:22:01.059 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:22:01.059 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:22:01.062 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:22:01.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:01.062 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:22:01.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:01.148 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:22:01.206 INFO:teuthology.orchestra.run.smithi120.stdout:client6605 2023-11-24T08:22:01.206 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:01.206 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:01.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:22:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.479+0000 7f7536147640 1 Processor -- start 2023-11-24T08:22:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.480+0000 7f7536147640 1 -- start start 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f753015d4e0 0x7f75300d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75300d38e0 0x7f75300cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75300d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75300d4110 con 0x7f753015d4e0 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75300d04e0 con 0x7f75300d38e0 2023-11-24T08:22:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7536147640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75300ce990 con 0x7f75300d0070 2023-11-24T08:22:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7535145640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f753015d4e0 0x7f75300d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7535145640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f753015d4e0 0x7f75300d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46356/0 (socket says 172.21.15.107:46356) 2023-11-24T08:22:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.481+0000 7f7535145640 1 -- 172.21.15.107:0/1501506609 learned_addr learned my addr 172.21.15.107:0/1501506609 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.482+0000 7f7535145640 1 -- 172.21.15.107:0/1501506609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 msgr2=0x7f75300d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.482+0000 7f7535145640 1 --2- 172.21.15.107:0/1501506609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75300d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.482+0000 7f7535145640 1 -- 172.21.15.107:0/1501506609 >> v2:172.21.15.120:3300/0 conn(0x7f75300d38e0 msgr2=0x7f75300cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.482+0000 7f7535145640 1 --2- 172.21.15.107:0/1501506609 >> v2:172.21.15.120:3300/0 conn(0x7f75300d38e0 0x7f75300cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.482+0000 7f7535145640 1 -- 172.21.15.107:0/1501506609 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75300cec70 con 0x7f753015d4e0 2023-11-24T08:22:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.483+0000 7f7535145640 1 --2- 172.21.15.107:0/1501506609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300d1e20 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f75300b9280 tx=0x7f752002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376f4673f73875b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.483+0000 7f752ffff640 1 -- 172.21.15.107:0/1501506609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752002ca80 con 0x7f753015d4e0 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.483+0000 7f752ffff640 1 -- 172.21.15.107:0/1501506609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752002cbe0 con 0x7f753015d4e0 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.483+0000 7f752ffff640 1 -- 172.21.15.107:0/1501506609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75200343f0 con 0x7f753015d4e0 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 -- 172.21.15.107:0/1501506609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 msgr2=0x7f75300d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 --2- 172.21.15.107:0/1501506609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300d1e20 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f75300b9280 tx=0x7f752002bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 -- 172.21.15.107:0/1501506609 shutdown_connections 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 --2- 172.21.15.107:0/1501506609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75300d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 --2- 172.21.15.107:0/1501506609 >> v2:172.21.15.120:3300/0 conn(0x7f75300d38e0 0x7f75300cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 --2- 172.21.15.107:0/1501506609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300d1e20 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 -- 172.21.15.107:0/1501506609 >> 172.21.15.107:0/1501506609 conn(0x7f753009c510 msgr2=0x7f75300d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 -- 172.21.15.107:0/1501506609 shutdown_connections 2023-11-24T08:22:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.484+0000 7f7536147640 1 -- 172.21.15.107:0/1501506609 wait complete. 2023-11-24T08:22:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.486+0000 7f7536147640 1 Processor -- start 2023-11-24T08:22:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.486+0000 7f7536147640 1 -- start start 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75301f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75300d38e0 0x7f75300c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75301f3790 con 0x7f753015d4e0 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75300b72f0 con 0x7f75300d38e0 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7536147640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75300b75d0 con 0x7f75300d0070 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7535946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7535946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46358/0 (socket says 172.21.15.107:46358) 2023-11-24T08:22:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.487+0000 7f7535946640 1 -- 172.21.15.107:0/3221820130 learned_addr learned my addr 172.21.15.107:0/3221820130 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535145640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75301f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7534944640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75300d38e0 0x7f75300c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 -- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 msgr2=0x7f75301f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75301f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 -- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75300d38e0 msgr2=0x7f75300c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75300d38e0 0x7f75300c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 -- 172.21.15.107:0/3221820130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520009b30 con 0x7f753015d4e0 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535145640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75301f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.488+0000 7f7535946640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f752800b410 tx=0x7f752800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.489+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528011830 con 0x7f753015d4e0 2023-11-24T08:22:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.489+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75280119b0 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.489+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75300b78b0 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.489+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528011d60 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.489+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75300c9920 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.490+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75300c7a50 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.491+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f752801a400 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.492+0000 7f752dffb640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 0x7f751c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.492+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75280886a0 con 0x7f753015d4e0 2023-11-24T08:22:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.492+0000 7f7535145640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 0x7f751c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.493+0000 7f7535145640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 0x7f751c06f280 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7520035000 tx=0x7f7520030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.496+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75300c7a50 con 0x7f753015d4e0 2023-11-24T08:22:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.724+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f75300c7a50 con 0x7f753015d4e0 2023-11-24T08:22:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.778+0000 7f752dffb640 1 -- 172.21.15.107:0/3221820130 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v129) v1 ==== 136+0+0 (secure 0 0 0) 0x7f75300c7a50 con 0x7f753015d4e0 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 msgr2=0x7f751c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 0x7f751c06f280 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7520035000 tx=0x7f7520030000 comp rx=0 tx=0).stop 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 msgr2=0x7f75300cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f752800b410 tx=0x7f752800b810 comp rx=0 tx=0).stop 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 shutdown_connections 2023-11-24T08:22:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f751c06cdc0 0x7f751c06f280 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f753015d4e0 0x7f75300cbc30 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75300d38e0 0x7f75300c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 --2- 172.21.15.107:0/3221820130 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75300d0070 0x7f75301f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 >> 172.21.15.107:0/3221820130 conn(0x7f753009c510 msgr2=0x7f75300d30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.781+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 shutdown_connections 2023-11-24T08:22:01.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:01.782+0000 7f7536147640 1 -- 172.21.15.107:0/3221820130 wait complete. 2023-11-24T08:22:01.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:22:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.014+0000 7f077e93e640 1 Processor -- start 2023-11-24T08:22:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.015+0000 7f077e93e640 1 -- start start 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077e93e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07780cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077e93e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07780c59b0 0x7f077806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077e93e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780cc3a0 con 0x7f07780cbf20 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077e93e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077815b820 con 0x7f07780c59b0 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077e93e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077815eea0 con 0x7f077815b440 2023-11-24T08:22:02.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.016+0000 7f077cedb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07780c59b0 0x7f077806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07780c59b0 0x7f077806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46372/0 (socket says 172.21.15.107:46372) 2023-11-24T08:22:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 -- 172.21.15.107:0/2394722504 learned_addr learned my addr 172.21.15.107:0/2394722504 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f0777fff640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07780cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 -- 172.21.15.107:0/2394722504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 msgr2=0x7f07780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07780cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 -- 172.21.15.107:0/2394722504 >> v1:172.21.15.120:6789/0 conn(0x7f07780cbf20 legacy=0x7f07780c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.017+0000 7f077cedb640 1 -- 172.21.15.107:0/2394722504 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780abf30 con 0x7f07780c59b0 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.018+0000 7f077cedb640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780c59b0 0x7f077806c5f0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f0770009e50 tx=0x7f077002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9745e418819243 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.018+0000 7f07777fe640 1 -- 172.21.15.107:0/2394722504 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077002c9e0 con 0x7f07780c59b0 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.018+0000 7f07777fe640 1 -- 172.21.15.107:0/2394722504 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077002cb40 con 0x7f07780c59b0 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f07777fe640 1 -- 172.21.15.107:0/2394722504 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07700343f0 con 0x7f07780c59b0 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 -- 172.21.15.107:0/2394722504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780c59b0 msgr2=0x7f077806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780c59b0 0x7f077806c5f0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f0770009e50 tx=0x7f077002be40 comp rx=0 tx=0).stop 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 -- 172.21.15.107:0/2394722504 shutdown_connections 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780c59b0 0x7f077806c5f0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 --2- 172.21.15.107:0/2394722504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07780cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 -- 172.21.15.107:0/2394722504 >> 172.21.15.107:0/2394722504 conn(0x7f07780b0c50 msgr2=0x7f07780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 -- 172.21.15.107:0/2394722504 shutdown_connections 2023-11-24T08:22:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.019+0000 7f077e93e640 1 -- 172.21.15.107:0/2394722504 wait complete. 2023-11-24T08:22:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.021+0000 7f077e93e640 1 Processor -- start 2023-11-24T08:22:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.021+0000 7f077e93e640 1 -- start start 2023-11-24T08:22:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07780c59b0 0x7f0778154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07781561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07781554b0 con 0x7f07780cbf20 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780cea70 con 0x7f07780c59b0 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f077e93e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0778155700 con 0x7f077815b440 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f076ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f076ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46374/0 (socket says 172.21.15.107:46374) 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.022+0000 7f076ffff640 1 -- 172.21.15.107:0/4060801819 learned_addr learned my addr 172.21.15.107:0/4060801819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 -- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 msgr2=0x7f07781561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07781561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 -- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07780c59b0 msgr2=0x7f0778154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07780c59b0 0x7f0778154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 -- 172.21.15.107:0/4060801819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770009b30 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f076ffff640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f076000c430 tx=0x7f076000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760011790 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07600118f0 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760011ce0 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07781559e0 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.023+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07780ce290 con 0x7f07780cbf20 2023-11-24T08:22:02.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.024+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0778155f00 con 0x7f07780cbf20 2023-11-24T08:22:02.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.025+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f076001c460 con 0x7f07780cbf20 2023-11-24T08:22:02.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.026+0000 7f07757fa640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 0x7f075806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.027+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0760088600 con 0x7f07780cbf20 2023-11-24T08:22:02.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.027+0000 7f0777fff640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 0x7f075806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.027+0000 7f0777fff640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 0x7f075806f280 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f076800a810 tx=0x7f0768008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.031+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0778155f00 con 0x7f07780cbf20 2023-11-24T08:22:02.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.245+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f077815f6b0 con 0x7f075806cdc0 2023-11-24T08:22:02.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.248+0000 7f07757fa640 1 -- 172.21.15.107:0/4060801819 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f076800a400 con 0x7f075806cdc0 2023-11-24T08:22:02.250 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:22:02.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:02.250 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:22:02.250 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:02.251 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:22:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 msgr2=0x7f075806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 0x7f075806f280 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f076800a810 tx=0x7f0768008040 comp rx=0 tx=0).stop 2023-11-24T08:22:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 msgr2=0x7f07780c6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f076000c430 tx=0x7f076000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 shutdown_connections 2023-11-24T08:22:02.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f075806cdc0 0x7f075806f280 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f077815b440 0x7f07781561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07780cbf20 0x7f07780c6300 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 --2- 172.21.15.107:0/4060801819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07780c59b0 0x7f0778154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 >> 172.21.15.107:0/4060801819 conn(0x7f07780b0c50 msgr2=0x7f07780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:02.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 shutdown_connections 2023-11-24T08:22:02.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.251+0000 7f077e93e640 1 -- 172.21.15.107:0/4060801819 wait complete. 2023-11-24T08:22:02.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:22:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.487+0000 7f919359e640 1 Processor -- start 2023-11-24T08:22:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.488+0000 7f919359e640 1 -- start start 2023-11-24T08:22:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.489+0000 7f919359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f919406d080 0x7f91940b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.489+0000 7f919359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940b4410 con 0x7f91940b3ce0 2023-11-24T08:22:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.489+0000 7f919359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940cf840 con 0x7f9194161a00 2023-11-24T08:22:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.489+0000 7f919359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91940b2160 con 0x7f919406d080 2023-11-24T08:22:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f919406d080 0x7f91940b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f919406d080 0x7f91940b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46384/0 (socket says 172.21.15.107:46384) 2023-11-24T08:22:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 -- 172.21.15.107:0/4225930139 learned_addr learned my addr 172.21.15.107:0/4225930139 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f9192d9d640 1 --1- 172.21.15.107:0/4225930139 >> v1:172.21.15.107:6789/0 conn(0x7f91940b3ce0 0x7f91940704c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40842/0 (socket says 172.21.15.107:40842) 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 -- 172.21.15.107:0/4225930139 >> v1:172.21.15.120:6789/0 conn(0x7f9194161a00 legacy=0x7f919406c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 -- 172.21.15.107:0/4225930139 >> v1:172.21.15.107:6789/0 conn(0x7f91940b3ce0 legacy=0x7f91940704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f918bfff640 1 -- 172.21.15.107:0/4225930139 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940b4150 con 0x7f919406d080 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.490+0000 7f9191d9b640 1 -- 172.21.15.107:0/4225930139 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692682664 0 0) 0x7f91940b4410 con 0x7f91940b3ce0 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.491+0000 7f918bfff640 1 --2- 172.21.15.107:0/4225930139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f91940b37a0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f917c009870 tx=0x7f917c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85e9fef77353829 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f9191d9b640 1 -- 172.21.15.107:0/4225930139 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c030850 con 0x7f919406d080 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f9191d9b640 1 -- 172.21.15.107:0/4225930139 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c0309b0 con 0x7f919406d080 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f9191d9b640 1 -- 172.21.15.107:0/4225930139 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c030d40 con 0x7f919406d080 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 -- 172.21.15.107:0/4225930139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 msgr2=0x7f91940b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 --2- 172.21.15.107:0/4225930139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f91940b37a0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f917c009870 tx=0x7f917c02b990 comp rx=0 tx=0).stop 2023-11-24T08:22:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 -- 172.21.15.107:0/4225930139 shutdown_connections 2023-11-24T08:22:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 --2- 172.21.15.107:0/4225930139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f91940b37a0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 -- 172.21.15.107:0/4225930139 >> 172.21.15.107:0/4225930139 conn(0x7f919406a500 msgr2=0x7f919406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 -- 172.21.15.107:0/4225930139 shutdown_connections 2023-11-24T08:22:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.492+0000 7f919359e640 1 -- 172.21.15.107:0/4225930139 wait complete. 2023-11-24T08:22:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.494+0000 7f919359e640 1 Processor -- start 2023-11-24T08:22:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.494+0000 7f919359e640 1 -- start start 2023-11-24T08:22:02.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91940b3ce0 0x7f919415f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9194161a00 0x7f9194156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919415f470 con 0x7f919406d080 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194143130 con 0x7f91940b3ce0 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91941599f0 con 0x7f9194161a00 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46388/0 (socket says 172.21.15.107:46388) 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.495+0000 7f919259c640 1 -- 172.21.15.107:0/2720978684 learned_addr learned my addr 172.21.15.107:0/2720978684 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 -- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9194161a00 msgr2=0x7f9194156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9194161a00 0x7f9194156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 -- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91940b3ce0 msgr2=0x7f919415f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91940b3ce0 0x7f919415f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 -- 172.21.15.107:0/2720978684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c0094d0 con 0x7f919406d080 2023-11-24T08:22:02.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919259c640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f918400c430 tx=0x7f918400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91840052d0 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9184005430 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184003830 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.496+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9194159cd0 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.497+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919415f100 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.498+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919415f290 con 0x7f919406d080 2023-11-24T08:22:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.499+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f9184003990 con 0x7f919406d080 2023-11-24T08:22:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.499+0000 7f918affd640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 0x7f917406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.500+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9184087ba0 con 0x7f919406d080 2023-11-24T08:22:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.500+0000 7f918bfff640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 0x7f917406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.500+0000 7f918bfff640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 0x7f917406f280 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.504+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9184054750 con 0x7f919406d080 2023-11-24T08:22:02.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.692+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f919404e840 con 0x7f919406d080 2023-11-24T08:22:02.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.693+0000 7f918affd640 1 -- 172.21.15.107:0/2720978684 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9184017020 con 0x7f919406d080 2023-11-24T08:22:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 msgr2=0x7f917406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 0x7f917406f280 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c03b040 comp rx=0 tx=0).stop 2023-11-24T08:22:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 msgr2=0x7f9194142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f918400c430 tx=0x7f918400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 shutdown_connections 2023-11-24T08:22:02.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f917406cdc0 0x7f917406f280 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9194161a00 0x7f9194156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91940b3ce0 0x7f919415f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 --2- 172.21.15.107:0/2720978684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f919406d080 0x7f9194142bf0 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 >> 172.21.15.107:0/2720978684 conn(0x7f919406a500 msgr2=0x7f9194071020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:02.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 shutdown_connections 2023-11-24T08:22:02.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.696+0000 7f919359e640 1 -- 172.21.15.107:0/2720978684 wait complete. 2023-11-24T08:22:02.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol239 --mode=777 2023-11-24T08:22:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.937+0000 7fb80e685640 1 Processor -- start 2023-11-24T08:22:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.938+0000 7fb80e685640 1 -- start start 2023-11-24T08:22:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80e685640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb80815d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80e685640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80815ea30 con 0x7fb8080cfbe0 2023-11-24T08:22:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80e685640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080d04e0 con 0x7fb808160b00 2023-11-24T08:22:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80e685640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80815de20 con 0x7fb8080d0050 2023-11-24T08:22:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80ce82640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8080cfbe0 0x7fb80815ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40858/0 (socket says 172.21.15.107:40858) 2023-11-24T08:22:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.939+0000 7fb80ce82640 1 -- 172.21.15.107:0/109461336 learned_addr learned my addr 172.21.15.107:0/109461336 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:02.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182751274 0 0) 0x7fb80815ea30 con 0x7fb8080cfbe0 2023-11-24T08:22:02.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb80de84640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb80815d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003520 con 0x7fb8080cfbe0 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040074467 0 0) 0x7fb7f4003520 con 0x7fb8080cfbe0 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 msgr2=0x7fb80815d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb80815d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 >> v1:172.21.15.120:6789/0 conn(0x7fb808160b00 legacy=0x7fb8080d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80815e100 con 0x7fb8080cfbe0 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb7f8001670 con 0x7fb8080cfbe0 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 >> v1:172.21.15.107:6789/0 conn(0x7fb8080cfbe0 legacy=0x7fb80815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40039a0 0x7fb7f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40047c0 0x7fb7f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb7f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80815de20 con 0x7fb7f40039a0 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400bb10 con 0x7fb7f40047c0 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.940+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400bcd0 con 0x7fb8080d0050 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40039a0 0x7fb7f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80de84640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb7f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 -- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 msgr2=0x7fb7f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80ce82640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40047c0 0x7fb7f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8080d0050 0x7fb7f4005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 -- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40047c0 msgr2=0x7fb7f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40047c0 0x7fb7f4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.941+0000 7fb80d683640 1 -- 172.21.15.107:0/109461336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4003520 con 0x7fb7f40039a0 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.942+0000 7fb80d683640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40039a0 0x7fb7f4003dc0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fb804001750 tx=0x7fb7f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29777eaca25d9132 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.942+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040052d0 con 0x7fb7f40039a0 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.942+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80402e050 con 0x7fb7f40039a0 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.942+0000 7fb7fffff640 1 -- 172.21.15.107:0/109461336 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040035f0 con 0x7fb7f40039a0 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 -- 172.21.15.107:0/109461336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40039a0 msgr2=0x7fb7f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 --2- 172.21.15.107:0/109461336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40039a0 0x7fb7f4003dc0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fb804001750 tx=0x7fb7f40057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80de84640 1 -- 172.21.15.107:0/109461336 reap_dead start 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 -- 172.21.15.107:0/109461336 shutdown_connections 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 -- 172.21.15.107:0/109461336 >> 172.21.15.107:0/109461336 conn(0x7fb80809c510 msgr2=0x7fb8080d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:02.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 -- 172.21.15.107:0/109461336 shutdown_connections 2023-11-24T08:22:02.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.943+0000 7fb80e685640 1 -- 172.21.15.107:0/109461336 wait complete. 2023-11-24T08:22:02.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.945+0000 7fb80e685640 1 Processor -- start 2023-11-24T08:22:02.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.945+0000 7fb80e685640 1 -- start start 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40039a0 0x7fb8080c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8081f7160 0x7fb808153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080c8820 con 0x7fb8080b7e00 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080cd440 con 0x7fb7f40039a0 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80e685640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080b87c0 con 0x7fb8081f7160 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80de84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80de84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46398/0 (socket says 172.21.15.107:46398) 2023-11-24T08:22:02.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.946+0000 7fb80de84640 1 -- 172.21.15.107:0/3173345786 learned_addr learned my addr 172.21.15.107:0/3173345786 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80ce82640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8081f7160 0x7fb808153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 -- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8081f7160 msgr2=0x7fb808153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8081f7160 0x7fb808153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 -- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40039a0 msgr2=0x7fb8080c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40039a0 0x7fb8080c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 -- 172.21.15.107:0/3173345786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080c7c30 con 0x7fb8080b7e00 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80de84640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fb8080b9280 tx=0x7fb800009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800007cc0 con 0x7fb8080b7e00 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800007e40 con 0x7fb8080b7e00 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800014640 con 0x7fb8080b7e00 2023-11-24T08:22:02.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.947+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8081f7700 con 0x7fb8080b7e00 2023-11-24T08:22:02.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.948+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8081f7cc0 con 0x7fb8080b7e00 2023-11-24T08:22:02.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.949+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8080c8960 con 0x7fb8080b7e00 2023-11-24T08:22:02.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.950+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7fb80001d3b0 con 0x7fb8080b7e00 2023-11-24T08:22:02.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.951+0000 7fb7fdffb640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 0x7fb7ec06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:02.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.951+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8000880b0 con 0x7fb8080b7e00 2023-11-24T08:22:02.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.951+0000 7fb80d683640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 0x7fb7ec06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:02.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.952+0000 7fb80d683640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 0x7fb7ec06f280 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb804001750 tx=0x7fb80402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:02.956+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8080c8960 con 0x7fb8080b7e00 2023-11-24T08:22:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.173+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol239", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8080cf140 con 0x7fb7ec06cdc0 2023-11-24T08:22:03.189 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:03.188+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:03.190 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:03.189+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:03.190 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:03.189+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:03.190 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:03.189+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:03.190 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:03.189+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:03.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.245+0000 7fb7fdffb640 1 -- 172.21.15.107:0/3173345786 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb804007a30 con 0x7fb7ec06cdc0 2023-11-24T08:22:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 msgr2=0x7fb7ec06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 0x7fb7ec06f280 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb804001750 tx=0x7fb80402f000 comp rx=0 tx=0).stop 2023-11-24T08:22:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 msgr2=0x7fb8080b8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fb8080b9280 tx=0x7fb800009d30 comp rx=0 tx=0).stop 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 shutdown_connections 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb7ec06cdc0 0x7fb7ec06f280 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8080b7e00 0x7fb8080b8280 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8081f7160 0x7fb808153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 --2- 172.21.15.107:0/3173345786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f40039a0 0x7fb8080c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.248+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 >> 172.21.15.107:0/3173345786 conn(0x7fb80809c510 msgr2=0x7fb80815c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.249+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 shutdown_connections 2023-11-24T08:22:03.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.249+0000 7fb80e685640 1 -- 172.21.15.107:0/3173345786 wait complete. 2023-11-24T08:22:03.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol239 2023-11-24T08:22:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.484+0000 7ff3112e0640 1 Processor -- start 2023-11-24T08:22:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.484+0000 7ff3112e0640 1 -- start start 2023-11-24T08:22:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.486+0000 7ff3112e0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff30c15b440 0x7ff30c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.486+0000 7ff3112e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c0c59b0 0x7ff30c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.486+0000 7ff3112e0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c15eea0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.486+0000 7ff3112e0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30c0c5e00 con 0x7ff30c15b440 2023-11-24T08:22:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.486+0000 7ff3112e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0cc3d0 con 0x7ff30c0c59b0 2023-11-24T08:22:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.487+0000 7ff30b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c0c59b0 0x7ff30c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.487+0000 7ff30affd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff30c15b440 0x7ff30c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.487+0000 7ff30a7fc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff30c0cbf20 0x7ff30c0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:33726/0 (socket says 172.21.15.107:33726) 2023-11-24T08:22:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.487+0000 7ff30a7fc640 1 -- 172.21.15.107:0/134780671 learned_addr learned my addr 172.21.15.107:0/134780671 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.487+0000 7ff309ffb640 1 -- 172.21.15.107:0/134780671 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440170158 0 0) 0x7ff30c15eea0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.488+0000 7ff309ffb640 1 -- 172.21.15.107:0/134780671 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f4003520 con 0x7ff30c0cbf20 2023-11-24T08:22:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.488+0000 7ff30affd640 1 -- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c0c59b0 msgr2=0x7ff30c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.488+0000 7ff30affd640 1 --2- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c0c59b0 0x7ff30c06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.488+0000 7ff30affd640 1 -- 172.21.15.107:0/134780671 >> v1:172.21.15.120:6789/0 conn(0x7ff30c0cbf20 legacy=0x7ff30c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.488+0000 7ff30affd640 1 -- 172.21.15.107:0/134780671 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0abf30 con 0x7ff30c15b440 2023-11-24T08:22:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.489+0000 7ff30affd640 1 --2- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c15b440 0x7ff30c0cb9e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc0097a0 tx=0x7ff2fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42dbafdd3e80380f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.489+0000 7ff309ffb640 1 -- 172.21.15.107:0/134780671 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc02e070 con 0x7ff30c15b440 2023-11-24T08:22:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff309ffb640 1 -- 172.21.15.107:0/134780671 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc0315a0 con 0x7ff30c15b440 2023-11-24T08:22:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff309ffb640 1 -- 172.21.15.107:0/134780671 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc031930 con 0x7ff30c15b440 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 -- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c15b440 msgr2=0x7ff30c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 --2- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c15b440 0x7ff30c0cb9e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc0097a0 tx=0x7ff2fc02b840 comp rx=0 tx=0).stop 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 -- 172.21.15.107:0/134780671 shutdown_connections 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 --2- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c0c59b0 0x7ff30c06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 --2- 172.21.15.107:0/134780671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c15b440 0x7ff30c0cb9e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 -- 172.21.15.107:0/134780671 >> 172.21.15.107:0/134780671 conn(0x7ff30c0b0c50 msgr2=0x7ff30c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 -- 172.21.15.107:0/134780671 shutdown_connections 2023-11-24T08:22:03.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.490+0000 7ff3112e0640 1 -- 172.21.15.107:0/134780671 wait complete. 2023-11-24T08:22:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.492+0000 7ff3112e0640 1 Processor -- start 2023-11-24T08:22:03.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.492+0000 7ff3112e0640 1 -- start start 2023-11-24T08:22:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c0c59b0 0x7ff30c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c15b440 0x7ff30c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0cec40 con 0x7ff30c0cbf20 2023-11-24T08:22:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1f3350 con 0x7ff30c0c59b0 2023-11-24T08:22:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff3112e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1f3630 con 0x7ff30c15b440 2023-11-24T08:22:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.493+0000 7ff30a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46414/0 (socket says 172.21.15.107:46414) 2023-11-24T08:22:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 -- 172.21.15.107:0/1724398959 learned_addr learned my addr 172.21.15.107:0/1724398959 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:03.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30affd640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c0c59b0 0x7ff30c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30b7fe640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c15b440 0x7ff30c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 -- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c15b440 msgr2=0x7ff30c0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c15b440 0x7ff30c0b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 -- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c0c59b0 msgr2=0x7ff30c155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c0c59b0 0x7ff30c155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 -- 172.21.15.107:0/1724398959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc009400 con 0x7ff30c0cbf20 2023-11-24T08:22:03.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.494+0000 7ff30a7fc640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c430 tx=0x7ff30000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.495+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300005ce0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.495+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300005e40 con 0x7ff30c0cbf20 2023-11-24T08:22:03.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.495+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c1f3910 con 0x7ff30c0cbf20 2023-11-24T08:22:03.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.495+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000072f0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.495+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c0ce2b0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.496+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c0ce440 con 0x7ff30c0cbf20 2023-11-24T08:22:03.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.497+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7ff300007450 con 0x7ff30c0cbf20 2023-11-24T08:22:03.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.498+0000 7ff2f37fe640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 0x7ff2f806f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.498+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff300087cd0 con 0x7ff30c0cbf20 2023-11-24T08:22:03.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.498+0000 7ff30affd640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 0x7ff2f806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.499+0000 7ff30affd640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 0x7ff2f806f280 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc036000 tx=0x7ff2fc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.502+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff30c0ce440 con 0x7ff30c0cbf20 2023-11-24T08:22:03.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.723+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol239", "target": ["mon-mgr", ""]}) v1 -- 0x7ff30c0c50b0 con 0x7ff2f806cdc0 2023-11-24T08:22:03.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.730+0000 7ff2f37fe640 1 -- 172.21.15.107:0/1724398959 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+65 (secure 0 0 0) 0x7ff30c0c50b0 con 0x7ff2f806cdc0 2023-11-24T08:22:03.731 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.732+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 msgr2=0x7ff2f806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.732+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 0x7ff2f806f280 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc036000 tx=0x7ff2fc03d040 comp rx=0 tx=0).stop 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.732+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 msgr2=0x7ff30c0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.732+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c430 tx=0x7ff30000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 shutdown_connections 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff2f806cdc0 0x7ff2f806f280 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff30c15b440 0x7ff30c0b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0cbf20 0x7ff30c0ce700 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 --2- 172.21.15.107:0/1724398959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff30c0c59b0 0x7ff30c155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 >> 172.21.15.107:0/1724398959 conn(0x7ff30c0b0c50 msgr2=0x7ff30c071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 shutdown_connections 2023-11-24T08:22:03.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.733+0000 7ff3112e0640 1 -- 172.21.15.107:0/1724398959 wait complete. 2023-11-24T08:22:03.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol239 2023-11-24T08:22:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.958+0000 7f33232bb640 1 Processor -- start 2023-11-24T08:22:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.958+0000 7f33232bb640 1 -- start start 2023-11-24T08:22:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f33232bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f331c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f33232bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c15ec90 con 0x7f331c06de10 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f33232bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c0cd530 con 0x7f331c0c49f0 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f33232bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0c4ed0 con 0x7f331c072160 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f3321858640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f331c06de10 0x7f331c0705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40868/0 (socket says 172.21.15.107:40868) 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f3321858640 1 -- 172.21.15.107:0/2505254833 learned_addr learned my addr 172.21.15.107:0/2505254833 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f3321057640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f331c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.960+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842468356 0 0) 0x7f3314001490 con 0x7f331c06de10 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3308005730 con 0x7f331c06de10 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136028439 0 0) 0x7f3308005730 con 0x7f331c06de10 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 msgr2=0x7f331c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f331c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 >> v1:172.21.15.120:6789/0 conn(0x7f331c0c49f0 legacy=0x7f331c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0c51b0 con 0x7f331c06de10 2023-11-24T08:22:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3314001b80 con 0x7f331c06de10 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 >> v1:172.21.15.107:6789/0 conn(0x7f331c06de10 legacy=0x7f331c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3308005bb0 0x7f3308005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33080069d0 0x7f3308006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f3308007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0c4ed0 con 0x7f33080069d0 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330800dd20 con 0x7f3308005bb0 2023-11-24T08:22:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330800dee0 con 0x7f331c072160 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.961+0000 7f3320856640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33080069d0 0x7f3308006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3321057640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3308005bb0 0x7f3308005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3321858640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f3308007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3320856640 1 -- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 msgr2=0x7f3308007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3320856640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c072160 0x7f3308007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3320856640 1 -- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3308005bb0 msgr2=0x7f3308005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3320856640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3308005bb0 0x7f3308005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.962+0000 7f3320856640 1 -- 172.21.15.107:0/2505254833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308005730 con 0x7f33080069d0 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.963+0000 7f3320856640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33080069d0 0x7f3308006e50 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f330c001750 tx=0x7f3308007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240f45f1bd357498 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.963+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0052d0 con 0x7f33080069d0 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.963+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c02e050 con 0x7f33080069d0 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.963+0000 7f3313fff640 1 -- 172.21.15.107:0/2505254833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0035f0 con 0x7f33080069d0 2023-11-24T08:22:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 -- 172.21.15.107:0/2505254833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33080069d0 msgr2=0x7f3308006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 --2- 172.21.15.107:0/2505254833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33080069d0 0x7f3308006e50 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f330c001750 tx=0x7f3308007a00 comp rx=0 tx=0).stop 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f3321858640 1 -- 172.21.15.107:0/2505254833 reap_dead start 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 -- 172.21.15.107:0/2505254833 shutdown_connections 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 -- 172.21.15.107:0/2505254833 >> 172.21.15.107:0/2505254833 conn(0x7f331c0b0c50 msgr2=0x7f331c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 -- 172.21.15.107:0/2505254833 shutdown_connections 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.964+0000 7f33232bb640 1 -- 172.21.15.107:0/2505254833 wait complete. 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.966+0000 7f33232bb640 1 Processor -- start 2023-11-24T08:22:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.966+0000 7f33232bb640 1 -- start start 2023-11-24T08:22:03.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 0x7f331c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c1f3c10 0x7f331c1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c15a700 con 0x7f331c0c8a20 2023-11-24T08:22:03.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c1f45d0 con 0x7f33080069d0 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f33232bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c15cc30 con 0x7f331c1f3c10 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f3320856640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f3320856640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46436/0 (socket says 172.21.15.107:46436) 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.967+0000 7f3320856640 1 -- 172.21.15.107:0/2153524943 learned_addr learned my addr 172.21.15.107:0/2153524943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3321057640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 0x7f331c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 -- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c1f3c10 msgr2=0x7f331c1f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c1f3c10 0x7f331c1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 -- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 msgr2=0x7f331c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 0x7f331c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 -- 172.21.15.107:0/2153524943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0b3c00 con 0x7f331c0c8a20 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3321057640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 0x7f331c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3320856640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f330c001870 tx=0x7f330c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.968+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0076c0 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.969+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c007a30 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.969+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c15ce60 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.969+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c006be0 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.969+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c15d420 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.970+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c15d640 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.971+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 42) v1 ==== 92087+0+0 (secure 0 0 0) 0x7f330c006d40 con 0x7f331c0c8a20 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.972+0000 7f3311ffb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 0x7f330006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:03.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.973+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f330c0aa290 con 0x7f331c0c8a20 2023-11-24T08:22:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.973+0000 7f3321057640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 0x7f330006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.973+0000 7f3321057640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 0x7f330006f2b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3318006fd0 tx=0x7f3318008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:03.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:03.977+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f330c076f60 con 0x7f331c0c8a20 2023-11-24T08:22:04.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.196+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol239", "target": ["mon-mgr", ""]}) v1 -- 0x7f331c15a700 con 0x7f330006cdf0 2023-11-24T08:22:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.205+0000 7f3311ffb640 1 -- 172.21.15.107:0/2153524943 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+65 (secure 0 0 0) 0x7f331c15a700 con 0x7f330006cdf0 2023-11-24T08:22:04.207 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41 2023-11-24T08:22:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 msgr2=0x7f330006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 0x7f330006f2b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3318006fd0 tx=0x7f3318008040 comp rx=0 tx=0).stop 2023-11-24T08:22:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 msgr2=0x7f331c1f36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f330c001870 tx=0x7f330c001120 comp rx=0 tx=0).stop 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 shutdown_connections 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f330006cdf0 0x7f330006f2b0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f331c1f3c10 0x7f331c1f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f331c0c8a20 0x7f331c1f36d0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 --2- 172.21.15.107:0/2153524943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33080069d0 0x7f331c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 >> 172.21.15.107:0/2153524943 conn(0x7f331c0b0c50 msgr2=0x7f331c0cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 shutdown_connections 2023-11-24T08:22:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:04.208+0000 7f33232bb640 1 -- 172.21.15.107:0/2153524943 wait complete. 2023-11-24T08:22:04.228 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol239 with 16 files each 1MB size under directory volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41 2023-11-24T08:22:04.229 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:04.229 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:04.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:04.281 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:04.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:04.426 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340829 s, 30.8 MB/s 2023-11-24T08:22:04.428 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:04.429 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:04.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:04.482 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242855 s, 43.2 MB/s 2023-11-24T08:22:04.558 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:04.558 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:04.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:04.614 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237454 s, 44.2 MB/s 2023-11-24T08:22:04.690 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:04.690 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:04.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:04.744 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367106 s, 28.6 MB/s 2023-11-24T08:22:04.833 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:04.834 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:04.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:04.883 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.020 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.020 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.020 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355618 s, 29.5 MB/s 2023-11-24T08:22:05.022 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.022 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.084 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189401 s, 55.4 MB/s 2023-11-24T08:22:05.159 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.159 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.215 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228981 s, 45.8 MB/s 2023-11-24T08:22:05.294 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.294 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.352 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.427 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.539 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.540 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209173 s, 50.1 MB/s 2023-11-24T08:22:05.541 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.542 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.692 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.693 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028444 s, 36.9 MB/s 2023-11-24T08:22:05.693 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.693 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.750 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.832 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.832 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263053 s, 39.9 MB/s 2023-11-24T08:22:05.834 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.834 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:05.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:05.892 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:05.960 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:05.960 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:05.960 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170099 s, 61.6 MB/s 2023-11-24T08:22:05.961 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:05.961 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:06.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:06.009 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:06.129 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213137 s, 49.2 MB/s 2023-11-24T08:22:06.131 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:06.131 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:06.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:06.182 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:06.252 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:06.252 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:06.252 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193708 s, 54.1 MB/s 2023-11-24T08:22:06.253 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:06.253 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:06.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:06.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:06.380 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:06.380 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:06.380 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224042 s, 46.8 MB/s 2023-11-24T08:22:06.382 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:06.382 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:06.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:06.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275915 s, 38.0 MB/s 2023-11-24T08:22:06.522 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:06.522 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:06.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:06.578 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol239/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:22:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:22:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:22:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228807 s, 45.8 MB/s 2023-11-24T08:22:06.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol239 subvol_snap2348 2023-11-24T08:22:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.933+0000 7eff82f01640 1 Processor -- start 2023-11-24T08:22:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.933+0000 7eff82f01640 1 -- start start 2023-11-24T08:22:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.934+0000 7eff82f01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7eff7c0cf450 0x7eff7c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.934+0000 7eff82f01640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c0cece0 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.934+0000 7eff82f01640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c06d040 con 0x7eff7c15c7f0 2023-11-24T08:22:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.934+0000 7eff82f01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c0cfe40 con 0x7eff7c0cf450 2023-11-24T08:22:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff8149e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7eff7c0cdfb0 0x7eff7c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40874/0 (socket says 172.21.15.107:40874) 2023-11-24T08:22:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff8149e640 1 -- 172.21.15.107:0/629631596 learned_addr learned my addr 172.21.15.107:0/629631596 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362919937 0 0) 0x7eff7c0cece0 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7bfff640 1 --2- 172.21.15.107:0/629631596 >> v2:172.21.15.120:3300/0 conn(0x7eff7c0cf450 0x7eff7c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff68003520 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657573702 0 0) 0x7eff68003520 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 >> v2:172.21.15.120:3300/0 conn(0x7eff7c0cf450 msgr2=0x7eff7c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 --2- 172.21.15.107:0/629631596 >> v2:172.21.15.120:3300/0 conn(0x7eff7c0cf450 0x7eff7c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 >> v1:172.21.15.120:6789/0 conn(0x7eff7c15c7f0 legacy=0x7eff7c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.935+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c06d320 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7eff74001670 con 0x7eff7c0cdfb0 2023-11-24T08:22:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 >> v1:172.21.15.107:6789/0 conn(0x7eff7c0cdfb0 legacy=0x7eff7c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff680039c0 0x7eff68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff680047c0 0x7eff68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 0x7eff6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0cfe40 con 0x7eff68005180 2023-11-24T08:22:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6800c080 con 0x7eff680047c0 2023-11-24T08:22:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6800c240 con 0x7eff680039c0 2023-11-24T08:22:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 0x7eff6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 -- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff680039c0 msgr2=0x7eff68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7bfff640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff680047c0 0x7eff68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff80c9d640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff680039c0 0x7eff68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff680039c0 0x7eff68003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 -- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff680047c0 msgr2=0x7eff68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff680047c0 0x7eff68004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff8149e640 1 -- 172.21.15.107:0/629631596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68003520 con 0x7eff68005180 2023-11-24T08:22:06.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.936+0000 7eff7bfff640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff680047c0 0x7eff68004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.937+0000 7eff8149e640 1 -- 172.21.15.107:0/629631596 reap_dead start 2023-11-24T08:22:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.937+0000 7eff8149e640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 0x7eff6800b980 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7eff7400c0e0 tx=0x7eff680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92744253d3c0084 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74005ad0 con 0x7eff68005180 2023-11-24T08:22:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74005c70 con 0x7eff68005180 2023-11-24T08:22:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff7b7fe640 1 -- 172.21.15.107:0/629631596 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7402c810 con 0x7eff68005180 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 -- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 msgr2=0x7eff6800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 0x7eff6800b980 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7eff7400c0e0 tx=0x7eff680057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 -- 172.21.15.107:0/629631596 shutdown_connections 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 --2- 172.21.15.107:0/629631596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68005180 0x7eff6800b980 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 -- 172.21.15.107:0/629631596 >> 172.21.15.107:0/629631596 conn(0x7eff7c06a920 msgr2=0x7eff7c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 -- 172.21.15.107:0/629631596 shutdown_connections 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.938+0000 7eff82f01640 1 -- 172.21.15.107:0/629631596 wait complete. 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.940+0000 7eff82f01640 1 Processor -- start 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.940+0000 7eff82f01640 1 -- start start 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff68005180 0x7eff7c154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c15d960 0x7eff7c15ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c15e330 con 0x7eff7c1f2400 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0ae250 con 0x7eff68005180 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.941+0000 7eff82f01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c155040 con 0x7eff7c15d960 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46450/0 (socket says 172.21.15.107:46450) 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 -- 172.21.15.107:0/3062571664 learned_addr learned my addr 172.21.15.107:0/3062571664 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff80c9d640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff68005180 0x7eff7c154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff8149e640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c15d960 0x7eff7c15ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 -- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c15d960 msgr2=0x7eff7c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c15d960 0x7eff7c15ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 -- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff68005180 msgr2=0x7eff7c154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff68005180 0x7eff7c154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 -- 172.21.15.107:0/3062571664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c0ad1b0 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.942+0000 7eff7bfff640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7eff64010900 tx=0x7eff64010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.943+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6400ebc0 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.943+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6400ed20 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.943+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6401c620 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.943+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c0ad490 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.943+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c0ada50 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.944+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c067470 con 0x7eff7c1f2400 2023-11-24T08:22:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.945+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 43) v1 ==== 92138+0+0 (secure 0 0 0) 0x7eff6401c780 con 0x7eff7c1f2400 2023-11-24T08:22:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.946+0000 7eff797fa640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 0x7eff5c06f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.946+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff64088ce0 con 0x7eff7c1f2400 2023-11-24T08:22:06.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.946+0000 7eff80c9d640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 0x7eff5c06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:06.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.947+0000 7eff80c9d640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 0x7eff5c06f310 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7eff6c007fd0 tx=0x7eff6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:06.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:06.951+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff64055890 con 0x7eff7c1f2400 2023-11-24T08:22:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.175+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol239", "snap_name": "subvol_snap2348", "target": ["mon-mgr", ""]}) v1 -- 0x7eff7c15b900 con 0x7eff5c06ce50 2023-11-24T08:22:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.205+0000 7eff797fa640 1 -- 172.21.15.107:0/3062571664 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7eff7c15b900 con 0x7eff5c06ce50 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.208+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 msgr2=0x7eff5c06f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.208+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 0x7eff5c06f310 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7eff6c007fd0 tx=0x7eff6c009040 comp rx=0 tx=0).stop 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.208+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 msgr2=0x7eff7c1f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.208+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7eff64010900 tx=0x7eff64010dc0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 shutdown_connections 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7eff5c06ce50 0x7eff5c06f310 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c15d960 0x7eff7c15ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff7c1f2400 0x7eff7c1f2b10 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 --2- 172.21.15.107:0/3062571664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff68005180 0x7eff7c154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 >> 172.21.15.107:0/3062571664 conn(0x7eff7c06a920 msgr2=0x7eff7c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 shutdown_connections 2023-11-24T08:22:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.209+0000 7eff82f01640 1 -- 172.21.15.107:0/3062571664 wait complete. 2023-11-24T08:22:07.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol239 --retain-snapshots 2023-11-24T08:22:07.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.446+0000 7feba9f41640 1 Processor -- start 2023-11-24T08:22:07.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.446+0000 7feba9f41640 1 -- start start 2023-11-24T08:22:07.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.447+0000 7feba9f41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba40cfb60 0x7feba4161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.447+0000 7feba9f41640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba40ce0b0 con 0x7feba415dc80 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.447+0000 7feba9f41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba4161960 con 0x7feba4157c70 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.447+0000 7feba9f41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba415e8d0 con 0x7feba40cfb60 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba40cfb60 0x7feba4161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba40cfb60 0x7feba4161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46454/0 (socket says 172.21.15.107:46454) 2023-11-24T08:22:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 -- 172.21.15.107:0/1948629521 learned_addr learned my addr 172.21.15.107:0/1948629521 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 -- 172.21.15.107:0/1948629521 >> v1:172.21.15.120:6789/0 conn(0x7feba4157c70 legacy=0x7feba40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 -- 172.21.15.107:0/1948629521 >> v1:172.21.15.107:6789/0 conn(0x7feba415dc80 legacy=0x7feba40b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.448+0000 7feba2ffd640 1 -- 172.21.15.107:0/1948629521 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba415ebb0 con 0x7feba40cfb60 2023-11-24T08:22:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.450+0000 7feba2ffd640 1 --2- 172.21.15.107:0/1948629521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba40cfb60 0x7feba4161df0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3dbe4b2c3d282a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.450+0000 7feba27fc640 1 -- 172.21.15.107:0/1948629521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98031870 con 0x7feba40cfb60 2023-11-24T08:22:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.450+0000 7feba27fc640 1 -- 172.21.15.107:0/1948629521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb980319d0 con 0x7feba40cfb60 2023-11-24T08:22:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.450+0000 7feba27fc640 1 -- 172.21.15.107:0/1948629521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98031d60 con 0x7feba40cfb60 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 -- 172.21.15.107:0/1948629521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba40cfb60 msgr2=0x7feba4161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 --2- 172.21.15.107:0/1948629521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba40cfb60 0x7feba4161df0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9802b910 comp rx=0 tx=0).stop 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 -- 172.21.15.107:0/1948629521 shutdown_connections 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 --2- 172.21.15.107:0/1948629521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba40cfb60 0x7feba4161df0 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 -- 172.21.15.107:0/1948629521 >> 172.21.15.107:0/1948629521 conn(0x7feba406a920 msgr2=0x7feba406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 -- 172.21.15.107:0/1948629521 shutdown_connections 2023-11-24T08:22:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.451+0000 7feba9f41640 1 -- 172.21.15.107:0/1948629521 wait complete. 2023-11-24T08:22:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.453+0000 7feba9f41640 1 Processor -- start 2023-11-24T08:22:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.453+0000 7feba9f41640 1 -- start start 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feba40cfb60 0x7feba4099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feba4157c70 0x7feba4154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba40c8e30 con 0x7feba415dc80 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba40c8500 con 0x7feba4157c70 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.454+0000 7feba9f41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba40c87e0 con 0x7feba40cfb60 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46460/0 (socket says 172.21.15.107:46460) 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 -- 172.21.15.107:0/3168577244 learned_addr learned my addr 172.21.15.107:0/3168577244 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 -- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feba40cfb60 msgr2=0x7feba4099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feba40cfb60 0x7feba4099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 -- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feba4157c70 msgr2=0x7feba4154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feba4157c70 0x7feba4154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 -- 172.21.15.107:0/3168577244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980094d0 con 0x7feba415dc80 2023-11-24T08:22:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feba3fff640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c730 tx=0x7feb9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c003b70 con 0x7feba415dc80 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c003cf0 con 0x7feba415dc80 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.455+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c003ea0 con 0x7feba415dc80 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.456+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba40c8ac0 con 0x7feba415dc80 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.456+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba40c92c0 con 0x7feba415dc80 2023-11-24T08:22:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.457+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba4067470 con 0x7feba415dc80 2023-11-24T08:22:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.458+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 43) v1 ==== 92138+0+0 (secure 0 0 0) 0x7feb9c005520 con 0x7feba415dc80 2023-11-24T08:22:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.459+0000 7feb8bfff640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 0x7feb9006f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.459+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb9c088620 con 0x7feba415dc80 2023-11-24T08:22:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.459+0000 7feba37fe640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 0x7feb9006f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.460+0000 7feba37fe640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 0x7feb9006f310 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7feba409b120 tx=0x7feb940079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.463+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb9c0551d0 con 0x7feba415dc80 2023-11-24T08:22:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.684+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol239", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7feba4158050 con 0x7feb9006ce50 2023-11-24T08:22:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.753+0000 7feb8bfff640 1 -- 172.21.15.107:0/3168577244 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feb94005030 con 0x7feb9006ce50 2023-11-24T08:22:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 msgr2=0x7feb9006f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 0x7feb9006f310 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7feba409b120 tx=0x7feb940079b0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 msgr2=0x7feba4155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c730 tx=0x7feb9c00cbf0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 shutdown_connections 2023-11-24T08:22:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb9006ce50 0x7feb9006f310 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba415dc80 0x7feba4155350 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feba4157c70 0x7feba4154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 --2- 172.21.15.107:0/3168577244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feba40cfb60 0x7feba4099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 >> 172.21.15.107:0/3168577244 conn(0x7feba406a920 msgr2=0x7feba4160600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.756+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 shutdown_connections 2023-11-24T08:22:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.757+0000 7feba9f41640 1 -- 172.21.15.107:0/3168577244 wait complete. 2023-11-24T08:22:07.765 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.764+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.765 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.764+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.765 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.764+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.765 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.764+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.765 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.764+0000 7f3765fa3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol239 subvol_snap2348 subvol_clone7628 2023-11-24T08:22:07.788 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.787+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.788 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.787+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.788 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.787+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.788 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.787+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.788 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:07.787+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:07.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.971+0000 7f887da61640 1 Processor -- start 2023-11-24T08:22:07.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.971+0000 7f887da61640 1 -- start start 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f887da61640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8878072160 0x7f887815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f887da61640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f887806de10 0x7f88780705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f887da61640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887815ec90 con 0x7f88780c49f0 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f887da61640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780cd530 con 0x7f887806de10 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f887da61640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780c4ed0 con 0x7f8878072160 2023-11-24T08:22:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f887806de10 0x7f88780705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f887806de10 0x7f88780705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34040/0 (socket says 172.21.15.107:34040) 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 -- 172.21.15.107:0/3712934238 learned_addr learned my addr 172.21.15.107:0/3712934238 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f8876ffd640 1 --2- 172.21.15.107:0/3712934238 >> v2:172.21.15.120:3300/0 conn(0x7f8878072160 0x7f887815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 -- 172.21.15.107:0/3712934238 >> v2:172.21.15.120:3300/0 conn(0x7f8878072160 msgr2=0x7f887815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 --2- 172.21.15.107:0/3712934238 >> v2:172.21.15.120:3300/0 conn(0x7f8878072160 0x7f887815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 -- 172.21.15.107:0/3712934238 >> v1:172.21.15.120:6789/0 conn(0x7f88780c49f0 legacy=0x7f887806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:07.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.973+0000 7f88777fe640 1 -- 172.21.15.107:0/3712934238 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780c51b0 con 0x7f887806de10 2023-11-24T08:22:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.974+0000 7f88777fe640 1 --2- 172.21.15.107:0/3712934238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887806de10 0x7f88780705c0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f88780ccae0 tx=0x7f887002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ff435f5d47237f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f88767fc640 1 -- 172.21.15.107:0/3712934238 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887002ca80 con 0x7f887806de10 2023-11-24T08:22:07.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f88767fc640 1 -- 172.21.15.107:0/3712934238 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887002cbe0 con 0x7f887806de10 2023-11-24T08:22:07.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f88767fc640 1 -- 172.21.15.107:0/3712934238 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88700343f0 con 0x7f887806de10 2023-11-24T08:22:07.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 -- 172.21.15.107:0/3712934238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887806de10 msgr2=0x7f88780705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 --2- 172.21.15.107:0/3712934238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887806de10 0x7f88780705c0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f88780ccae0 tx=0x7f887002bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 -- 172.21.15.107:0/3712934238 shutdown_connections 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 --2- 172.21.15.107:0/3712934238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f887806de10 0x7f88780705c0 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 --2- 172.21.15.107:0/3712934238 >> v2:172.21.15.120:3300/0 conn(0x7f8878072160 0x7f887815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.975+0000 7f887da61640 1 -- 172.21.15.107:0/3712934238 >> 172.21.15.107:0/3712934238 conn(0x7f88780b0c50 msgr2=0x7f88780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.976+0000 7f887da61640 1 -- 172.21.15.107:0/3712934238 shutdown_connections 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.976+0000 7f887da61640 1 -- 172.21.15.107:0/3712934238 wait complete. 2023-11-24T08:22:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.977+0000 7f887da61640 1 Processor -- start 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.977+0000 7f887da61640 1 -- start start 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887806de10 0x7f8878158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8878072160 0x7f887815c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8878158a00 con 0x7f88780c49f0 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887815d0a0 con 0x7f887806de10 2023-11-24T08:22:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f887da61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88781f4620 con 0x7f8878072160 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.978+0000 7f88777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34050/0 (socket says 172.21.15.107:34050) 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 -- 172.21.15.107:0/1892580868 learned_addr learned my addr 172.21.15.107:0/1892580868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 -- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8878072160 msgr2=0x7f887815c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f8876ffd640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887806de10 0x7f8878158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8878072160 0x7f887815c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 -- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887806de10 msgr2=0x7f8878158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887806de10 0x7f8878158b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 -- 172.21.15.107:0/1892580868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8870009b30 con 0x7f88780c49f0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f88777fe640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f8870009f40 tx=0x7f8870004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.979+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887003e450 con 0x7f88780c49f0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.980+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887003e5d0 con 0x7f88780c49f0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.980+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887003e980 con 0x7f88780c49f0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.980+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88781593f0 con 0x7f88780c49f0 2023-11-24T08:22:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.980+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88781f4340 con 0x7f88780c49f0 2023-11-24T08:22:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.981+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887804e8b0 con 0x7f88780c49f0 2023-11-24T08:22:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.982+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 43) v1 ==== 92138+0+0 (secure 0 0 0) 0x7f887002cd50 con 0x7f88780c49f0 2023-11-24T08:22:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.982+0000 7f8874ff9640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 0x7f885806f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.983+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88700ab180 con 0x7f88780c49f0 2023-11-24T08:22:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.983+0000 7f8876ffd640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 0x7f885806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.983+0000 7f8876ffd640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 0x7f885806f310 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f8868006fd0 tx=0x7f8868008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:07.987+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8870077d40 con 0x7f88780c49f0 2023-11-24T08:22:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:08.211+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol239", "snap_name": "subvol_snap2348", "target_sub_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f88780c5a50 con 0x7f885806ce50 2023-11-24T08:22:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:08.849+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 44) v1 ==== 92240+0+0 (secure 0 0 0) 0x7f8870078110 con 0x7f88780c49f0 2023-11-24T08:22:11.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.401+0000 7f8874ff9640 1 -- 172.21.15.107:0/1892580868 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f886800c040 con 0x7f885806ce50 2023-11-24T08:22:11.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 msgr2=0x7f885806f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 0x7f885806f310 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f8868006fd0 tx=0x7f8868008040 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 msgr2=0x7f88781584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f8870009f40 tx=0x7f8870004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 shutdown_connections 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f885806ce50 0x7f885806f310 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88780c49f0 0x7f88781584c0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8878072160 0x7f887815c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.404+0000 7f887da61640 1 --2- 172.21.15.107:0/1892580868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f887806de10 0x7f8878158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.405+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 >> 172.21.15.107:0/1892580868 conn(0x7f88780b0c50 msgr2=0x7f88780cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.405+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 shutdown_connections 2023-11-24T08:22:11.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.405+0000 7f887da61640 1 -- 172.21.15.107:0/1892580868 wait complete. 2023-11-24T08:22:11.411 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.410+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.412 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.410+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.412 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.410+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.412 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.410+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.412 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.410+0000 7f37687a8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7628 2023-11-24T08:22:11.434 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.432+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.434 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.432+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.434 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.432+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.434 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.432+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.434 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:11.432+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:22:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.619+0000 7f0e184a6640 1 Processor -- start 2023-11-24T08:22:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.619+0000 7f0e184a6640 1 -- start start 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e184a6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e100c59b0 0x7f0e10166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e184a6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100c5dc0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e184a6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1015eea0 con 0x7f0e1015b440 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e184a6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1015b820 con 0x7f0e100c59b0 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e15a41640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e100cc0f0 0x7f0e100c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42262/0 (socket says 172.21.15.107:42262) 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e15a41640 1 -- 172.21.15.107:0/592614939 learned_addr learned my addr 172.21.15.107:0/592614939 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:11.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.621+0000 7f0e16a43640 1 --2- 172.21.15.107:0/592614939 >> v2:172.21.15.107:3300/0 conn(0x7f0e100c59b0 0x7f0e10166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:11.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.622+0000 7f0e15240640 1 -- 172.21.15.107:0/592614939 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32588805 0 0) 0x7f0e100c5dc0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.622+0000 7f0e16a43640 1 -- 172.21.15.107:0/592614939 >> v1:172.21.15.120:6789/0 conn(0x7f0e1015b440 legacy=0x7f0e100cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:11.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.622+0000 7f0e16a43640 1 -- 172.21.15.107:0/592614939 >> v1:172.21.15.107:6789/0 conn(0x7f0e100cc0f0 legacy=0x7f0e100c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.622+0000 7f0e16a43640 1 -- 172.21.15.107:0/592614939 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1006d600 con 0x7f0e100c59b0 2023-11-24T08:22:11.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.623+0000 7f0e16a43640 1 --2- 172.21.15.107:0/592614939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100c59b0 0x7f0e10166230 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f0e080097a0 tx=0x7f0e0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5700c2948cb21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:11.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.623+0000 7f0e15240640 1 -- 172.21.15.107:0/592614939 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0802e070 con 0x7f0e100c59b0 2023-11-24T08:22:11.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.623+0000 7f0e15240640 1 -- 172.21.15.107:0/592614939 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e080315a0 con 0x7f0e100c59b0 2023-11-24T08:22:11.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.623+0000 7f0e15240640 1 -- 172.21.15.107:0/592614939 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08031930 con 0x7f0e100c59b0 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 -- 172.21.15.107:0/592614939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100c59b0 msgr2=0x7f0e10166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 --2- 172.21.15.107:0/592614939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100c59b0 0x7f0e10166230 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f0e080097a0 tx=0x7f0e0802b840 comp rx=0 tx=0).stop 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 -- 172.21.15.107:0/592614939 shutdown_connections 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 --2- 172.21.15.107:0/592614939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100c59b0 0x7f0e10166230 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 -- 172.21.15.107:0/592614939 >> 172.21.15.107:0/592614939 conn(0x7f0e100b0c50 msgr2=0x7f0e100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 -- 172.21.15.107:0/592614939 shutdown_connections 2023-11-24T08:22:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.624+0000 7f0e184a6640 1 -- 172.21.15.107:0/592614939 wait complete. 2023-11-24T08:22:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.626+0000 7f0e184a6640 1 Processor -- start 2023-11-24T08:22:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.626+0000 7f0e184a6640 1 -- start start 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e100c59b0 0x7f0e100b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1015b440 0x7f0e101596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100b49f0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10099ea0 con 0x7f0e100c59b0 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e184a6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100b3b30 con 0x7f0e1015b440 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e15a41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e15a41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34102/0 (socket says 172.21.15.107:34102) 2023-11-24T08:22:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.627+0000 7f0e15a41640 1 -- 172.21.15.107:0/588704456 learned_addr learned my addr 172.21.15.107:0/588704456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e16a43640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1015b440 0x7f0e101596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e16242640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e100c59b0 0x7f0e100b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 -- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1015b440 msgr2=0x7f0e101596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1015b440 0x7f0e101596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 -- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e100c59b0 msgr2=0x7f0e100b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e100c59b0 0x7f0e100b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 -- 172.21.15.107:0/588704456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08009400 con 0x7f0e100cc0f0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e15a41640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400c4d0 tx=0x7f0e0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040074e0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04007660 con 0x7f0e100cc0f0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04004a20 con 0x7f0e100cc0f0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e10159c00 con 0x7f0e100cc0f0 2023-11-24T08:22:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.628+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1015a720 con 0x7f0e100cc0f0 2023-11-24T08:22:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.629+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1004e8b0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.631+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 44) v1 ==== 92240+0+0 (secure 0 0 0) 0x7f0e04004b80 con 0x7f0e100cc0f0 2023-11-24T08:22:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.631+0000 7f0e0effd640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 0x7f0dfc06f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.632+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e040878d0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.632+0000 7f0e16242640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 0x7f0dfc06f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.632+0000 7f0e16242640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 0x7f0dfc06f430 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0e00007bf0 tx=0x7f0e00007240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.636+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e040543e0 con 0x7f0e100cc0f0 2023-11-24T08:22:11.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.863+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e100cbf80 con 0x7f0dfc06cf70 2023-11-24T08:22:11.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.876+0000 7f0e0effd640 1 -- 172.21.15.107:0/588704456 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7f0e100cbf80 con 0x7f0dfc06cf70 2023-11-24T08:22:11.878 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:11.878 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol239", 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap2348" 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:11.879 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 msgr2=0x7f0dfc06f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 0x7f0dfc06f430 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0e00007bf0 tx=0x7f0e00007240 comp rx=0 tx=0).stop 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 msgr2=0x7f0e10154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400c4d0 tx=0x7f0e0400c990 comp rx=0 tx=0).stop 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 shutdown_connections 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0dfc06cf70 0x7f0dfc06f430 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1015b440 0x7f0e101596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e100cc0f0 0x7f0e10154c70 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 --2- 172.21.15.107:0/588704456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e100c59b0 0x7f0e100b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:11.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 >> 172.21.15.107:0/588704456 conn(0x7f0e100b0c50 msgr2=0x7f0e100c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:11.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 shutdown_connections 2023-11-24T08:22:11.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:11.879+0000 7f0e184a6640 1 -- 172.21.15.107:0/588704456 wait complete. 2023-11-24T08:22:12.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7628 2023-11-24T08:22:13.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.104+0000 7f1405f7c640 1 Processor -- start 2023-11-24T08:22:13.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.104+0000 7f1405f7c640 1 -- start start 2023-11-24T08:22:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.105+0000 7f1405f7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14000b0b40 0x7f14000b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.105+0000 7f1405f7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f140006c5b0 0x7f140006ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.105+0000 7f1405f7c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140015ed20 con 0x7f140015b260 2023-11-24T08:22:13.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.105+0000 7f1405f7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140015b640 con 0x7f140006c5b0 2023-11-24T08:22:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.105+0000 7f1405f7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140015f8a0 con 0x7f14000b0b40 2023-11-24T08:22:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f1404f7a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f140015b260 0x7f140006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42274/0 (socket says 172.21.15.107:42274) 2023-11-24T08:22:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f1404f7a640 1 -- 172.21.15.107:0/927167367 learned_addr learned my addr 172.21.15.107:0/927167367 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:13.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f140577b640 1 --2- 172.21.15.107:0/927167367 >> v2:172.21.15.107:3300/0 conn(0x7f140006c5b0 0x7f140006ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f13ff7fe640 1 -- 172.21.15.107:0/927167367 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517182029 0 0) 0x7f140015ed20 con 0x7f140015b260 2023-11-24T08:22:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f13ff7fe640 1 -- 172.21.15.107:0/927167367 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f0003520 con 0x7f140015b260 2023-11-24T08:22:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f13fffff640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14000b0b40 0x7f14000b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f140577b640 1 -- 172.21.15.107:0/927167367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14000b0b40 msgr2=0x7f14000b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f140577b640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14000b0b40 0x7f14000b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f140577b640 1 -- 172.21.15.107:0/927167367 >> v1:172.21.15.107:6789/0 conn(0x7f140015b260 legacy=0x7f140006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.106+0000 7f140577b640 1 -- 172.21.15.107:0/927167367 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14000b14c0 con 0x7f140006c5b0 2023-11-24T08:22:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.107+0000 7f140577b640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f140006c5b0 0x7f140006ca40 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009d90 tx=0x7f13f802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b2204802048078 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.107+0000 7f13ff7fe640 1 -- 172.21.15.107:0/927167367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f802cb30 con 0x7f140006c5b0 2023-11-24T08:22:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f13ff7fe640 1 -- 172.21.15.107:0/927167367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f802cc90 con 0x7f140006c5b0 2023-11-24T08:22:13.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f13ff7fe640 1 -- 172.21.15.107:0/927167367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8034440 con 0x7f140006c5b0 2023-11-24T08:22:13.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 -- 172.21.15.107:0/927167367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f140006c5b0 msgr2=0x7f140006ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f140006c5b0 0x7f140006ca40 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009d90 tx=0x7f13f802bd80 comp rx=0 tx=0).stop 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 -- 172.21.15.107:0/927167367 shutdown_connections 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f140006c5b0 0x7f140006ca40 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 --2- 172.21.15.107:0/927167367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14000b0b40 0x7f14000b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 -- 172.21.15.107:0/927167367 >> 172.21.15.107:0/927167367 conn(0x7f14000cbb40 msgr2=0x7f14000cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 -- 172.21.15.107:0/927167367 shutdown_connections 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.108+0000 7f1405f7c640 1 -- 172.21.15.107:0/927167367 wait complete. 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.110+0000 7f1405f7c640 1 Processor -- start 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.110+0000 7f1405f7c640 1 -- start start 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f140006c5b0 0x7f14001f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f140015b260 0x7f14000c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140009d020 con 0x7f14000b0b40 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14000c6190 con 0x7f140015b260 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f1405f7c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14000c6450 con 0x7f140006c5b0 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.111+0000 7f13fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34118/0 (socket says 172.21.15.107:34118) 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 -- 172.21.15.107:0/676780108 learned_addr learned my addr 172.21.15.107:0/676780108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f1404f7a640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f140006c5b0 0x7f14001f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 -- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f140006c5b0 msgr2=0x7f14001f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f140006c5b0 0x7f14001f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 -- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f140015b260 msgr2=0x7f14000c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f140015b260 0x7f14000c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 -- 172.21.15.107:0/676780108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8009a70 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fffff640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00c7f0 tx=0x7f13ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec004220 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec004380 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec007840 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.112+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14000c6730 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.113+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14000c6cf0 con 0x7f14000b0b40 2023-11-24T08:22:13.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.114+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14000c6f10 con 0x7f14000b0b40 2023-11-24T08:22:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.115+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f13ec007bb0 con 0x7f14000b0b40 2023-11-24T08:22:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.115+0000 7f13fd7fa640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 0x7f13e406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.116+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13ec0886e0 con 0x7f14000b0b40 2023-11-24T08:22:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.116+0000 7f1404f7a640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 0x7f13e406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:13.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.117+0000 7f1404f7a640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 0x7f13e406f550 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f13f4000f60 tx=0x7f13f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.120+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13ec0551f0 con 0x7f14000b0b40 2023-11-24T08:22:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.348+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f14000c65c0 con 0x7f13e406d090 2023-11-24T08:22:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.361+0000 7f13fd7fa640 1 -- 172.21.15.107:0/676780108 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7f14000c65c0 con 0x7f13e406d090 2023-11-24T08:22:13.363 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:13.363 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:22:13.363 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:22:13.363 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol239", 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap2348" 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:13.364 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 msgr2=0x7f13e406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 0x7f13e406f550 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f13f4000f60 tx=0x7f13f4009040 comp rx=0 tx=0).stop 2023-11-24T08:22:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 msgr2=0x7f1400159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:13.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00c7f0 tx=0x7f13ec00ccb0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 shutdown_connections 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13e406d090 0x7f13e406f550 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f140015b260 0x7f14000c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14000b0b40 0x7f1400159790 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 --2- 172.21.15.107:0/676780108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f140006c5b0 0x7f14001f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 >> 172.21.15.107:0/676780108 conn(0x7f14000cbb40 msgr2=0x7f14000af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 shutdown_connections 2023-11-24T08:22:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:13.364+0000 7f1405f7c640 1 -- 172.21.15.107:0/676780108 wait complete. 2023-11-24T08:22:14.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7628 2023-11-24T08:22:14.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.603+0000 7ff1a3bdc640 1 Processor -- start 2023-11-24T08:22:14.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.604+0000 7ff1a3bdc640 1 -- start start 2023-11-24T08:22:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.605+0000 7ff1a3bdc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff19c1660b0 0x7ff19c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.605+0000 7ff1a3bdc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff19c162540 0x7ff19c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.605+0000 7ff1a3bdc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c166510 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.605+0000 7ff1a3bdc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c1655f0 con 0x7ff19c162540 2023-11-24T08:22:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.605+0000 7ff1a3bdc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c0cc380 con 0x7ff19c1660b0 2023-11-24T08:22:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff19c162540 0x7ff19c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff19c162540 0x7ff19c0c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34130/0 (socket says 172.21.15.107:34130) 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 -- 172.21.15.107:0/460105552 learned_addr learned my addr 172.21.15.107:0/460105552 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a1177640 1 --2- 172.21.15.107:0/460105552 >> v2:172.21.15.120:3300/0 conn(0x7ff19c1660b0 0x7ff19c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 -- 172.21.15.107:0/460105552 >> v2:172.21.15.120:3300/0 conn(0x7ff19c1660b0 msgr2=0x7ff19c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 --2- 172.21.15.107:0/460105552 >> v2:172.21.15.120:3300/0 conn(0x7ff19c1660b0 0x7ff19c162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 -- 172.21.15.107:0/460105552 >> v1:172.21.15.120:6789/0 conn(0x7ff19c0ccbd0 legacy=0x7ff19c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.606+0000 7ff1a2179640 1 -- 172.21.15.107:0/460105552 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c162b30 con 0x7ff19c162540 2023-11-24T08:22:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.608+0000 7ff1a2179640 1 --2- 172.21.15.107:0/460105552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c162540 0x7ff19c0c9650 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff19c06e700 tx=0x7ff19402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34865f6ff1ae9f8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.608+0000 7ff1a0976640 1 -- 172.21.15.107:0/460105552 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19402ca80 con 0x7ff19c162540 2023-11-24T08:22:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.608+0000 7ff1a0976640 1 -- 172.21.15.107:0/460105552 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19402cbe0 con 0x7ff19c162540 2023-11-24T08:22:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.608+0000 7ff1a0976640 1 -- 172.21.15.107:0/460105552 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1940343f0 con 0x7ff19c162540 2023-11-24T08:22:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/460105552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c162540 msgr2=0x7ff19c0c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/460105552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c162540 0x7ff19c0c9650 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff19c06e700 tx=0x7ff19402bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/460105552 shutdown_connections 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/460105552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c162540 0x7ff19c0c9650 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/460105552 >> v2:172.21.15.120:3300/0 conn(0x7ff19c1660b0 0x7ff19c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/460105552 >> 172.21.15.107:0/460105552 conn(0x7ff19c0b0c50 msgr2=0x7ff19c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/460105552 shutdown_connections 2023-11-24T08:22:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.609+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/460105552 wait complete. 2023-11-24T08:22:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.611+0000 7ff1a3bdc640 1 Processor -- start 2023-11-24T08:22:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.611+0000 7ff1a3bdc640 1 -- start start 2023-11-24T08:22:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.612+0000 7ff1a3bdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.612+0000 7ff1a3bdc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff19c162540 0x7ff19c1f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a3bdc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff19c1660b0 0x7ff19c154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a3bdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c1f85f0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a3bdc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c071370 con 0x7ff19c1660b0 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a3bdc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c1fac30 con 0x7ff19c162540 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34142/0 (socket says 172.21.15.107:34142) 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 -- 172.21.15.107:0/133650189 learned_addr learned my addr 172.21.15.107:0/133650189 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 -- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff19c162540 msgr2=0x7ff19c1f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff19c162540 0x7ff19c1f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 -- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff19c1660b0 msgr2=0x7ff19c154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff19c1660b0 0x7ff19c154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.613+0000 7ff1a1978640 1 -- 172.21.15.107:0/133650189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff194009b30 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff1a1978640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c430 tx=0x7ff19800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011790 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1980118f0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c1faf10 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011ca0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.614+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c1fb4d0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.615+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c04e8b0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.616+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff19801c460 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.618+0000 7ff18e7fc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 0x7ff18006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.618+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1980886f0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.618+0000 7ff1a1177640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 0x7ff18006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.619+0000 7ff1a1177640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 0x7ff18006f550 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff190006fd0 tx=0x7ff190008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:14.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.622+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1980552a0 con 0x7ff19c0ccbd0 2023-11-24T08:22:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:14.847+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7ff19c165b90 con 0x7ff18006d090 2023-11-24T08:22:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.386+0000 7ff18e7fc640 1 -- 172.21.15.107:0/133650189 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7ff19000be20 con 0x7ff18006d090 2023-11-24T08:22:15.388 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:15.389 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:22:15.389 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:22:15.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:15.389 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.389+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 msgr2=0x7ff18006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.389+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 0x7ff18006f550 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff190006fd0 tx=0x7ff190008040 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.389+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 msgr2=0x7ff19c070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.389+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c430 tx=0x7ff19800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 shutdown_connections 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff18006d090 0x7ff18006f550 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff19c1660b0 0x7ff19c154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff19c162540 0x7ff19c1f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 --2- 172.21.15.107:0/133650189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff19c0ccbd0 0x7ff19c070d40 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 >> 172.21.15.107:0/133650189 conn(0x7ff19c0b0c50 msgr2=0x7ff19c0c9630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 shutdown_connections 2023-11-24T08:22:15.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.390+0000 7ff1a3bdc640 1 -- 172.21.15.107:0/133650189 wait complete. 2023-11-24T08:22:15.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone7628 2023-11-24T08:22:15.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.624+0000 7fe3d08cf640 1 Processor -- start 2023-11-24T08:22:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.625+0000 7fe3d08cf640 1 -- start start 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3d08cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3d08cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3cc0ccfb0 0x7fe3cc162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3d08cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc1655f0 con 0x7fe3cc162540 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3d08cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0cc380 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3d08cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc162b30 con 0x7fe3cc0ccfb0 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.626+0000 7fe3ca59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34152/0 (socket says 172.21.15.107:34152) 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 -- 172.21.15.107:0/3507834641 learned_addr learned my addr 172.21.15.107:0/3507834641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3cad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3cc162540 0x7fe3cc0c9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42284/0 (socket says 172.21.15.107:42284) 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 -- 172.21.15.107:0/3507834641 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc0ccfb0 msgr2=0x7fe3cc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:15.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 --2- 172.21.15.107:0/3507834641 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc0ccfb0 0x7fe3cc162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 -- 172.21.15.107:0/3507834641 >> v1:172.21.15.107:6789/0 conn(0x7fe3cc162540 legacy=0x7fe3cc0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.627+0000 7fe3ca59c640 1 -- 172.21.15.107:0/3507834641 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc162e10 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.628+0000 7fe3ca59c640 1 --2- 172.21.15.107:0/3507834641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06e700 tx=0x7fe3bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d699fb5ed793b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3c9d9b640 1 -- 172.21.15.107:0/3507834641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc030860 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3c9d9b640 1 -- 172.21.15.107:0/3507834641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3bc0309c0 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3c9d9b640 1 -- 172.21.15.107:0/3507834641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc030d50 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 -- 172.21.15.107:0/3507834641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 msgr2=0x7fe3cc1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/3507834641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06e700 tx=0x7fe3bc02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 -- 172.21.15.107:0/3507834641 shutdown_connections 2023-11-24T08:22:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/3507834641 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc0ccfb0 0x7fe3cc162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/3507834641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc1660b0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 -- 172.21.15.107:0/3507834641 >> 172.21.15.107:0/3507834641 conn(0x7fe3cc0b0c50 msgr2=0x7fe3cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:15.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 -- 172.21.15.107:0/3507834641 shutdown_connections 2023-11-24T08:22:15.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.629+0000 7fe3d08cf640 1 -- 172.21.15.107:0/3507834641 wait complete. 2023-11-24T08:22:15.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.631+0000 7fe3d08cf640 1 Processor -- start 2023-11-24T08:22:15.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.631+0000 7fe3d08cf640 1 -- start start 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc0ccfb0 0x7fe3cc1644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc162540 0x7fe3cc164a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0c6ab0 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0c8ce0 con 0x7fe3cc162540 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.632+0000 7fe3d08cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0c8fc0 con 0x7fe3cc0ccfb0 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34162/0 (socket says 172.21.15.107:34162) 2023-11-24T08:22:15.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 -- 172.21.15.107:0/1735263800 learned_addr learned my addr 172.21.15.107:0/1735263800 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 -- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc0ccfb0 msgr2=0x7fe3cc1644e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc0ccfb0 0x7fe3cc1644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 -- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc162540 msgr2=0x7fe3cc164a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc162540 0x7fe3cc164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 -- 172.21.15.107:0/1735263800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0094d0 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.633+0000 7fe3ca59c640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc000c00 tx=0x7fe3bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.634+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc039b40 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.634+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3bc039cc0 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.634+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc043630 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.634+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc070560 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.634+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc070b20 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.635+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc0cbf30 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.636+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe3bc02f3b0 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.637+0000 7fe3c2ffd640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 0x7fe3ac06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:15.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.637+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3bc0aab50 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.637+0000 7fe3c3fff640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 0x7fe3ac06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:15.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.638+0000 7fe3c3fff640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 0x7fe3ac06f550 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc1fb490 tx=0x7fe3b4006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:15.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.642+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3bc077700 con 0x7fe3cc0ccbd0 2023-11-24T08:22:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.865+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3cc0cd410 con 0x7fe3ac06d090 2023-11-24T08:22:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.877+0000 7fe3c2ffd640 1 -- 172.21.15.107:0/1735263800 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fe3b4013400 con 0x7fe3ac06d090 2023-11-24T08:22:15.879 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d 2023-11-24T08:22:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.879+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 msgr2=0x7fe3ac06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.879+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 0x7fe3ac06f550 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc1fb490 tx=0x7fe3b4006c90 comp rx=0 tx=0).stop 2023-11-24T08:22:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.879+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 msgr2=0x7fe3cc0c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.879+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc000c00 tx=0x7fe3bc004730 comp rx=0 tx=0).stop 2023-11-24T08:22:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 shutdown_connections 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3ac06d090 0x7fe3ac06f550 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc162540 0x7fe3cc164a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc0ccfb0 0x7fe3cc1644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 --2- 172.21.15.107:0/1735263800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0ccbd0 0x7fe3cc0c6570 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 >> 172.21.15.107:0/1735263800 conn(0x7fe3cc0b0c50 msgr2=0x7fe3cc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 shutdown_connections 2023-11-24T08:22:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:15.880+0000 7fe3d08cf640 1 -- 172.21.15.107:0/1735263800 wait complete. 2023-11-24T08:22:15.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone7628 2023-11-24T08:22:16.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:22:16.019 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:22:16.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.117+0000 7f320a1ef640 1 Processor -- start 2023-11-24T08:22:16.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.118+0000 7f320a1ef640 1 -- start start 2023-11-24T08:22:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f320a1ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f32040ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f320a1ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040d15e0 con 0x7f320415d7d0 2023-11-24T08:22:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f320a1ef640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040d0280 con 0x7f320415d3f0 2023-11-24T08:22:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f320a1ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b0eb0 con 0x7f32040d1f40 2023-11-24T08:22:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f32089ec640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f320415d7d0 0x7f32040cdd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42288/0 (socket says 172.21.15.107:42288) 2023-11-24T08:22:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.119+0000 7f32089ec640 1 -- 172.21.15.107:0/2249526003 learned_addr learned my addr 172.21.15.107:0/2249526003 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f32099ee640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f32040ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241037134 0 0) 0x7f32040d15e0 con 0x7f320415d7d0 2023-11-24T08:22:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f0003520 con 0x7f320415d7d0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523959390 0 0) 0x7f32040d0280 con 0x7f320415d3f0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32040d15e0 con 0x7f320415d3f0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 774534454 0 0) 0x7f31f0003520 con 0x7f320415d7d0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 msgr2=0x7f32040ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f32040ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 >> v1:172.21.15.120:6789/0 conn(0x7f320415d3f0 legacy=0x7f32040d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040d1300 con 0x7f320415d7d0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f31f4001670 con 0x7f320415d7d0 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 >> v1:172.21.15.107:6789/0 conn(0x7f320415d7d0 legacy=0x7f32040cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f31f0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0004090 0x7f31f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0004d00 0x7f31f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.120+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b0eb0 con 0x7f31f0004d00 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f000ba60 con 0x7f31f0004090 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f000bc20 con 0x7f32040d1f40 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0004d00 0x7f31f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32089ec640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0004090 0x7f31f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 -- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 msgr2=0x7f31f0003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32091ed640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f31f0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040d1f40 0x7f31f0003b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 -- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0004090 msgr2=0x7f31f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0004090 0x7f31f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.121+0000 7f32099ee640 1 -- 172.21.15.107:0/2249526003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0003520 con 0x7f31f0004d00 2023-11-24T08:22:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.122+0000 7f32099ee640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0004d00 0x7f31f0005180 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f320408bb90 tx=0x7f31f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bc60f81013a707 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc02cb90 con 0x7f31f0004d00 2023-11-24T08:22:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc00f040 con 0x7f31f0004d00 2023-11-24T08:22:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f31fbfff640 1 -- 172.21.15.107:0/2249526003 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc034440 con 0x7f31f0004d00 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 -- 172.21.15.107:0/2249526003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0004d00 msgr2=0x7f31f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 --2- 172.21.15.107:0/2249526003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0004d00 0x7f31f0005180 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f320408bb90 tx=0x7f31f00057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f32099ee640 1 -- 172.21.15.107:0/2249526003 reap_dead start 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 -- 172.21.15.107:0/2249526003 shutdown_connections 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 -- 172.21.15.107:0/2249526003 >> 172.21.15.107:0/2249526003 conn(0x7f320409c5b0 msgr2=0x7f32040d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 -- 172.21.15.107:0/2249526003 shutdown_connections 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.123+0000 7f320a1ef640 1 -- 172.21.15.107:0/2249526003 wait complete. 2023-11-24T08:22:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.125+0000 7f320a1ef640 1 Processor -- start 2023-11-24T08:22:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.125+0000 7f320a1ef640 1 -- start start 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 0x7f32041f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040b2970 0x7f32040b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b4330 con 0x7f32040c8f70 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040c9930 con 0x7f320415b640 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.126+0000 7f320a1ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b3360 con 0x7f32040b2970 2023-11-24T08:22:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34176/0 (socket says 172.21.15.107:34176) 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 -- 172.21.15.107:0/3708418783 learned_addr learned my addr 172.21.15.107:0/3708418783 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32091ed640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 0x7f32041f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32099ee640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040b2970 0x7f32040b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 -- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040b2970 msgr2=0x7f32040b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040b2970 0x7f32040b2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 -- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 msgr2=0x7f32041f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 0x7f32041f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 -- 172.21.15.107:0/3708418783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320415ba20 con 0x7f32040c8f70 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32091ed640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 0x7f32041f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.127+0000 7f32089ec640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f31f400eaf0 tx=0x7f31f4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.128+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400c840 con 0x7f32040c8f70 2023-11-24T08:22:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.128+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f400c9a0 con 0x7f32040c8f70 2023-11-24T08:22:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.128+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400cd50 con 0x7f32040c8f70 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.128+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32040cac60 con 0x7f32040c8f70 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.128+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32040b35c0 con 0x7f32040c8f70 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.129+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320404e840 con 0x7f32040c8f70 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.130+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f31f401d460 con 0x7f32040c8f70 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.131+0000 7f31f9ffb640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 0x7f31e806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.131+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31f4088820 con 0x7f32040c8f70 2023-11-24T08:22:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.131+0000 7f32091ed640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 0x7f31e806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.132+0000 7f32091ed640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 0x7f31e806f550 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3200007fd0 tx=0x7f3200009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:16.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.135+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31f4055330 con 0x7f32040c8f70 2023-11-24T08:22:16.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.360+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f32040b27f0 con 0x7f31e806d090 2023-11-24T08:22:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.378+0000 7f31f9ffb640 1 -- 172.21.15.107:0/3708418783 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7f320000d040 con 0x7f31e806d090 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:22:03", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 16777216, 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:22:11", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:22:14", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:22:16.381 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:22:06", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:22:16.382 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:22:16.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:22:16.383 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:22:16.383 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 msgr2=0x7f31e806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 0x7f31e806f550 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3200007fd0 tx=0x7f3200009040 comp rx=0 tx=0).stop 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 msgr2=0x7f32040c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f31f400eaf0 tx=0x7f31f4001670 comp rx=0 tx=0).stop 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 shutdown_connections 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31e806d090 0x7f31e806f550 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32040b2970 0x7f32040b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32040c8f70 0x7f32040c93f0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 --2- 172.21.15.107:0/3708418783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f320415b640 0x7f32041f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.381+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 >> 172.21.15.107:0/3708418783 conn(0x7f320409c5b0 msgr2=0x7f320406c890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.382+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 shutdown_connections 2023-11-24T08:22:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:16.382+0000 7f320a1ef640 1 -- 172.21.15.107:0/3708418783 wait complete. 2023-11-24T08:22:16.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.402 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41: ceph.quota.max_bytes: No such attribute 2023-11-24T08:22:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:22:16.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.452 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.502 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:22:16.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.503 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.555 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.609 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41 2023-11-24T08:22:16.609 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2 2023-11-24T08:22:16.609 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14 2023-11-24T08:22:16.609 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13 2023-11-24T08:22:16.609 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12 2023-11-24T08:22:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1 2023-11-24T08:22:16.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.611 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.661 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:22:16.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/) 2023-11-24T08:22:16.714 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:22:16.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.766 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:22:16.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.767 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/) 2023-11-24T08:22:16.820 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:22:16.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.822 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.872 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:22:16.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.874 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/) 2023-11-24T08:22:16.926 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:22:16.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.930 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41) 2023-11-24T08:22:16.987 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:16.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:16.989 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/) 2023-11-24T08:22:17.040 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:17.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.042 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2) 2023-11-24T08:22:17.092 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:17.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.093 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.2) 2023-11-24T08:22:17.152 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:17.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2) 2023-11-24T08:22:17.207 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:17.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.208 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.2) 2023-11-24T08:22:17.264 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:17.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.266 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2) 2023-11-24T08:22:17.317 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:17.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.319 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.2) 2023-11-24T08:22:17.372 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:17.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.374 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.2) 2023-11-24T08:22:17.427 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:17.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.429 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.2) 2023-11-24T08:22:17.484 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:17.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.486 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14) 2023-11-24T08:22:17.538 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:17.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.540 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.14) 2023-11-24T08:22:17.599 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:17.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14) 2023-11-24T08:22:17.654 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:17.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.14) 2023-11-24T08:22:17.708 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:17.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.709 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14) 2023-11-24T08:22:17.761 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:17.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.762 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.14) 2023-11-24T08:22:17.815 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:17.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.817 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.14) 2023-11-24T08:22:17.872 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:17.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.873 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.14) 2023-11-24T08:22:17.927 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:17.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.928 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13) 2023-11-24T08:22:17.981 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:17.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:17.982 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.13) 2023-11-24T08:22:18.041 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:18.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13) 2023-11-24T08:22:18.094 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:18.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.095 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.13) 2023-11-24T08:22:18.147 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:18.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.149 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13) 2023-11-24T08:22:18.202 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:18.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.204 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.13) 2023-11-24T08:22:18.251 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:18.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.13) 2023-11-24T08:22:18.352 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:18.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.353 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.13) 2023-11-24T08:22:18.411 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:18.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.412 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10) 2023-11-24T08:22:18.468 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:18.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.10) 2023-11-24T08:22:18.529 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:18.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.531 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10) 2023-11-24T08:22:18.584 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:18.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.585 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.10) 2023-11-24T08:22:18.637 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:18.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.639 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10) 2023-11-24T08:22:18.689 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:18.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.690 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.10) 2023-11-24T08:22:18.742 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:18.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.744 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.10) 2023-11-24T08:22:18.799 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:18.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.801 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.10) 2023-11-24T08:22:18.854 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:18.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.855 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5) 2023-11-24T08:22:18.907 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:18.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.908 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.5) 2023-11-24T08:22:18.965 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:18.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:18.967 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5) 2023-11-24T08:22:19.019 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.020 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.5) 2023-11-24T08:22:19.073 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.075 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5) 2023-11-24T08:22:19.126 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:19.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.127 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.5) 2023-11-24T08:22:19.179 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:19.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.181 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.5) 2023-11-24T08:22:19.233 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:19.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.235 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.5) 2023-11-24T08:22:19.287 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:19.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4) 2023-11-24T08:22:19.340 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:19.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.341 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.4) 2023-11-24T08:22:19.402 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:19.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.403 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4) 2023-11-24T08:22:19.453 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.4) 2023-11-24T08:22:19.503 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4) 2023-11-24T08:22:19.557 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:19.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.559 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.4) 2023-11-24T08:22:19.612 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:19.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.613 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.4) 2023-11-24T08:22:19.671 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:19.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.672 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.4) 2023-11-24T08:22:19.726 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:19.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7) 2023-11-24T08:22:19.781 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:19.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.783 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.7) 2023-11-24T08:22:19.844 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:19.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.845 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7) 2023-11-24T08:22:19.898 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.900 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.7) 2023-11-24T08:22:19.948 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:19.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:19.949 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7) 2023-11-24T08:22:20.001 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.002 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.7) 2023-11-24T08:22:20.049 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.7) 2023-11-24T08:22:20.100 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:20.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.102 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.7) 2023-11-24T08:22:20.149 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:20.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6) 2023-11-24T08:22:20.197 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:20.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.199 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.6) 2023-11-24T08:22:20.251 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:20.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6) 2023-11-24T08:22:20.301 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:20.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.303 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.6) 2023-11-24T08:22:20.354 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:20.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.355 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6) 2023-11-24T08:22:20.410 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.410 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.6) 2023-11-24T08:22:20.461 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.463 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.6) 2023-11-24T08:22:20.516 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:20.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.518 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.6) 2023-11-24T08:22:20.570 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:20.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.572 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0) 2023-11-24T08:22:20.624 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:20.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.0) 2023-11-24T08:22:20.681 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:20.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.682 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0) 2023-11-24T08:22:20.731 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:20.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.732 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.0) 2023-11-24T08:22:20.781 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:20.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0) 2023-11-24T08:22:20.833 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.834 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.0) 2023-11-24T08:22:20.886 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:20.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.888 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.0) 2023-11-24T08:22:20.941 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:20.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.0) 2023-11-24T08:22:20.996 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:20.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:20.997 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9) 2023-11-24T08:22:21.046 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.048 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.9) 2023-11-24T08:22:21.098 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.099 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9) 2023-11-24T08:22:21.152 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:21.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.9) 2023-11-24T08:22:21.205 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:21.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.206 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9) 2023-11-24T08:22:21.256 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:21.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.257 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.9) 2023-11-24T08:22:21.308 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:21.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.309 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.9) 2023-11-24T08:22:21.359 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:21.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.361 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.9) 2023-11-24T08:22:21.411 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:21.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.412 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15) 2023-11-24T08:22:21.461 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.462 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.15) 2023-11-24T08:22:21.513 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.514 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15) 2023-11-24T08:22:21.565 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:21.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.566 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.15) 2023-11-24T08:22:21.618 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:21.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.619 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15) 2023-11-24T08:22:21.670 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:21.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.672 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.15) 2023-11-24T08:22:21.722 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:21.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.724 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.15) 2023-11-24T08:22:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:21.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.15) 2023-11-24T08:22:21.829 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:21.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.831 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8) 2023-11-24T08:22:21.879 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.881 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.8) 2023-11-24T08:22:21.979 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:21.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:21.981 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8) 2023-11-24T08:22:22.034 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.035 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.8) 2023-11-24T08:22:22.084 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.085 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8) 2023-11-24T08:22:22.132 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:22.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.133 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.8) 2023-11-24T08:22:22.183 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:22.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.8) 2023-11-24T08:22:22.236 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:22.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.237 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.8) 2023-11-24T08:22:22.287 INFO:teuthology.orchestra.run.smithi107.stdout:1700814125 2023-11-24T08:22:22.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11) 2023-11-24T08:22:22.337 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:22.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.339 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.11) 2023-11-24T08:22:22.389 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:22.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11) 2023-11-24T08:22:22.440 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.11) 2023-11-24T08:22:22.492 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.493 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11) 2023-11-24T08:22:22.542 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:22.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.11) 2023-11-24T08:22:22.597 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:22.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.11) 2023-11-24T08:22:22.652 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:22.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.653 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.11) 2023-11-24T08:22:22.705 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:22.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.707 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3) 2023-11-24T08:22:22.759 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:22.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.760 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.3) 2023-11-24T08:22:22.816 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:22.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.818 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3) 2023-11-24T08:22:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.871 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.3) 2023-11-24T08:22:22.921 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:22.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3) 2023-11-24T08:22:22.972 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:22.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:22.974 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.3) 2023-11-24T08:22:23.024 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:23.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.026 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.3) 2023-11-24T08:22:23.078 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:23.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.078 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.3) 2023-11-24T08:22:23.128 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:23.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.129 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12) 2023-11-24T08:22:23.180 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:23.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.182 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.12) 2023-11-24T08:22:23.234 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:23.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.235 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12) 2023-11-24T08:22:23.287 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:23.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.12) 2023-11-24T08:22:23.340 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:23.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12) 2023-11-24T08:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:23.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.395 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.12) 2023-11-24T08:22:23.445 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:23.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.447 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.12) 2023-11-24T08:22:23.498 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:23.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.12) 2023-11-24T08:22:23.552 INFO:teuthology.orchestra.run.smithi107.stdout:1700814126 2023-11-24T08:22:23.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.553 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1) 2023-11-24T08:22:23.607 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:23.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.608 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.1) 2023-11-24T08:22:23.658 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:22:23.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.659 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1) 2023-11-24T08:22:23.711 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:23.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.712 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.1) 2023-11-24T08:22:23.763 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:22:23.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.764 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1) 2023-11-24T08:22:23.817 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:23.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.819 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.1) 2023-11-24T08:22:23.871 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:22:23.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.872 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol239/.snap/subvol_snap2348/54c63ee5-cdaa-444a-9462-1f074f563b41/subvolume_file.1) 2023-11-24T08:22:23.923 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:23.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:23.924 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7628/a08f77b0-cabb-4a02-ab8c-06483ede5f0d/subvolume_file.1) 2023-11-24T08:22:23.975 INFO:teuthology.orchestra.run.smithi107.stdout:1700814124 2023-11-24T08:22:23.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol_clone7628 subvol_snap450 2023-11-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.193+0000 7efc161a9640 1 Processor -- start 2023-11-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.193+0000 7efc161a9640 1 -- start start 2023-11-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc161a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efc100cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc161a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc100cece0 con 0x7efc1015c7f0 2023-11-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc161a9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1006d040 con 0x7efc100cdfb0 2023-11-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc161a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc10156010 con 0x7efc100cf450 2023-11-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc0f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efc1015c7f0 0x7efc1006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41600/0 (socket says 172.21.15.107:41600) 2023-11-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.195+0000 7efc0f7fe640 1 -- 172.21.15.107:0/3964542994 learned_addr learned my addr 172.21.15.107:0/3964542994 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc07fff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efc100cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392674372 0 0) 0x7efc100cece0 con 0x7efc1015c7f0 2023-11-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbfc003520 con 0x7efc1015c7f0 2023-11-24T08:22:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487442071 0 0) 0x7efbfc003520 con 0x7efc1015c7f0 2023-11-24T08:22:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 msgr2=0x7efc100cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efc100cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 >> v1:172.21.15.120:6789/0 conn(0x7efc100cdfb0 legacy=0x7efc100ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1006d320 con 0x7efc1015c7f0 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7efc00001670 con 0x7efc1015c7f0 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 >> v1:172.21.15.107:6789/0 conn(0x7efc1015c7f0 legacy=0x7efc1006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efbfc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc003fd0 0x7efbfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10156010 con 0x7efbfc004d00 2023-11-24T08:22:24.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc00bb40 con 0x7efbfc003fd0 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.196+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc00bd00 con 0x7efc100cf450 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0f7fe640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efbfc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc07fff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc003fd0 0x7efbfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 -- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 msgr2=0x7efbfc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efbfc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 -- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc003fd0 msgr2=0x7efbfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc003fd0 0x7efbfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0ffff640 1 -- 172.21.15.107:0/3964542994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc003520 con 0x7efbfc004d00 2023-11-24T08:22:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.197+0000 7efc0f7fe640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc100cf450 0x7efbfc0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.198+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7efc08001e50 tx=0x7efbfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa777a8e6a67c08 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.198+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08003b70 con 0x7efbfc004d00 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08003cd0 con 0x7efbfc004d00 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc0effd640 1 -- 172.21.15.107:0/3964542994 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08005310 con 0x7efbfc004d00 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 -- 172.21.15.107:0/3964542994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 msgr2=0x7efbfc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 --2- 172.21.15.107:0/3964542994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7efc08001e50 tx=0x7efbfc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc0ffff640 1 -- 172.21.15.107:0/3964542994 reap_dead start 2023-11-24T08:22:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 -- 172.21.15.107:0/3964542994 shutdown_connections 2023-11-24T08:22:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 -- 172.21.15.107:0/3964542994 >> 172.21.15.107:0/3964542994 conn(0x7efc1006a920 msgr2=0x7efc1006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 -- 172.21.15.107:0/3964542994 shutdown_connections 2023-11-24T08:22:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.199+0000 7efc161a9640 1 -- 172.21.15.107:0/3964542994 wait complete. 2023-11-24T08:22:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.201+0000 7efc161a9640 1 Processor -- start 2023-11-24T08:22:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.201+0000 7efc161a9640 1 -- start start 2023-11-24T08:22:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc10156af0 0x7efc1006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 0x7efc10144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014e9d0 con 0x7efbfc004d00 2023-11-24T08:22:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014f250 con 0x7efc10156f70 2023-11-24T08:22:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc161a9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014f530 con 0x7efc10156af0 2023-11-24T08:22:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.202+0000 7efc0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41510/0 (socket says 172.21.15.107:41510) 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 -- 172.21.15.107:0/3873630480 learned_addr learned my addr 172.21.15.107:0/3873630480 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 -- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc10156af0 msgr2=0x7efc1006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc07fff640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc10156af0 0x7efc1006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 0x7efc10144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc10156af0 0x7efc1006dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 -- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 msgr2=0x7efc10144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 0x7efc10144140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 -- 172.21.15.107:0/3873630480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1014f810 con 0x7efbfc004d00 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0ffff640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 0x7efc10144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0f7fe640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7efc0000d8c0 tx=0x7efc0000dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000074e0 con 0x7efbfc004d00 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00007640 con 0x7efbfc004d00 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000049e0 con 0x7efbfc004d00 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.203+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1014faf0 con 0x7efbfc004d00 2023-11-24T08:22:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.204+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1014ee70 con 0x7efbfc004d00 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.205+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc100728c0 con 0x7efbfc004d00 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.206+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7efc00004b40 con 0x7efbfc004d00 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.206+0000 7efc0cff9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 0x7efbf006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.207+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc000880b0 con 0x7efbfc004d00 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.207+0000 7efc07fff640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 0x7efbf006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.207+0000 7efc07fff640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 0x7efbf006f550 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7efc10156500 tx=0x7efbf8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.211+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc00054d10 con 0x7efbfc004d00 2023-11-24T08:22:24.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.432+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "snap_name": "subvol_snap450", "target": ["mon-mgr", ""]}) v1 -- 0x7efc1014f6a0 con 0x7efbf006d090 2023-11-24T08:22:24.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.461+0000 7efc0cff9640 1 -- 172.21.15.107:0/3873630480 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efc1014f6a0 con 0x7efbf006d090 2023-11-24T08:22:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 msgr2=0x7efbf006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 0x7efbf006f550 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7efc10156500 tx=0x7efbf8008040 comp rx=0 tx=0).stop 2023-11-24T08:22:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 msgr2=0x7efc100aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7efc0000d8c0 tx=0x7efc0000dd80 comp rx=0 tx=0).stop 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 shutdown_connections 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efbf006d090 0x7efbf006f550 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc10156f70 0x7efc10144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc10156af0 0x7efc1006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 --2- 172.21.15.107:0/3873630480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efbfc004d00 0x7efc100aec50 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 >> 172.21.15.107:0/3873630480 conn(0x7efc1006a920 msgr2=0x7efc1006c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 shutdown_connections 2023-11-24T08:22:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.464+0000 7efc161a9640 1 -- 172.21.15.107:0/3873630480 wait complete. 2023-11-24T08:22:24.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone7628 --retain-snapshots 2023-11-24T08:22:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.697+0000 7ff1b2c67640 1 Processor -- start 2023-11-24T08:22:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.697+0000 7ff1b2c67640 1 -- start start 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b2c67640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1ac1431f0 0x7ff1ac15e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b2c67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b2c67640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac15ec20 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b2c67640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac0b4fd0 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b2c67640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac143c10 con 0x7ff1ac1431f0 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b1464640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b1464640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41512/0 (socket says 172.21.15.107:41512) 2023-11-24T08:22:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.699+0000 7ff1b1464640 1 -- 172.21.15.107:0/2201510693 learned_addr learned my addr 172.21.15.107:0/2201510693 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.700+0000 7ff1b1c65640 1 --2- 172.21.15.107:0/2201510693 >> v2:172.21.15.120:3300/0 conn(0x7ff1ac1431f0 0x7ff1ac15e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.700+0000 7ff1b1464640 1 -- 172.21.15.107:0/2201510693 >> v2:172.21.15.120:3300/0 conn(0x7ff1ac1431f0 msgr2=0x7ff1ac15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.700+0000 7ff1b1464640 1 --2- 172.21.15.107:0/2201510693 >> v2:172.21.15.120:3300/0 conn(0x7ff1ac1431f0 0x7ff1ac15e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.700+0000 7ff1b1464640 1 -- 172.21.15.107:0/2201510693 >> v1:172.21.15.120:6789/0 conn(0x7ff1ac0b2f60 legacy=0x7ff1ac071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.700+0000 7ff1b1464640 1 -- 172.21.15.107:0/2201510693 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0705b0 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.701+0000 7ff1b1464640 1 --2- 172.21.15.107:0/2201510693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cd540 tx=0x7ff19c02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523630fe7ebdbffe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.701+0000 7ff1b0c63640 1 -- 172.21.15.107:0/2201510693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c02ede0 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b0c63640 1 -- 172.21.15.107:0/2201510693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c004030 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b0c63640 1 -- 172.21.15.107:0/2201510693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c004420 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2201510693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2ad0 msgr2=0x7ff1ac06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2201510693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cd540 tx=0x7ff19c02e030 comp rx=0 tx=0).stop 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2201510693 shutdown_connections 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2201510693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2ad0 0x7ff1ac06d6a0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2201510693 >> v2:172.21.15.120:3300/0 conn(0x7ff1ac1431f0 0x7ff1ac15e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2201510693 >> 172.21.15.107:0/2201510693 conn(0x7ff1ac0cb8a0 msgr2=0x7ff1ac0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2201510693 shutdown_connections 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.702+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2201510693 wait complete. 2023-11-24T08:22:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.704+0000 7ff1b2c67640 1 Processor -- start 2023-11-24T08:22:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.704+0000 7ff1b2c67640 1 -- start start 2023-11-24T08:22:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1ac0b2ad0 0x7ff1ac0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1ac1431f0 0x7ff1ac1f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac0a8810 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac1f3680 con 0x7ff1ac1431f0 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b2c67640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac1f3960 con 0x7ff1ac0b2ad0 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.705+0000 7ff1b1464640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41524/0 (socket says 172.21.15.107:41524) 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 -- 172.21.15.107:0/2027273234 learned_addr learned my addr 172.21.15.107:0/2027273234 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1c65640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1ac0b2ad0 0x7ff1ac0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 -- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1ac0b2ad0 msgr2=0x7ff1ac0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1ac0b2ad0 0x7ff1ac0cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 -- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1ac1431f0 msgr2=0x7ff1ac1f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1ac1431f0 0x7ff1ac1f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 -- 172.21.15.107:0/2027273234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c00b700 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.706+0000 7ff1b1464640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7ff19c038040 tx=0x7ff19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.707+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c003810 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.707+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c003970 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.707+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c003d20 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.707+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac1f3c40 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.707+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac1f32c0 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.708+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac0a8950 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.709+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff19c006040 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.710+0000 7ff1a27fc640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 0x7ff19006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.710+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff19c0ab480 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.710+0000 7ff1b1c65640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 0x7ff19006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.711+0000 7ff1b1c65640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 0x7ff19006f550 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8006fd0 tx=0x7ff1a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.714+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1ac0a8950 con 0x7ff1ac0b2f60 2023-11-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.932+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff1ac070e50 con 0x7ff19006d090 2023-11-24T08:22:24.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.994+0000 7ff1a27fc640 1 -- 172.21.15.107:0/2027273234 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff1a800e4b0 con 0x7ff19006d090 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 msgr2=0x7ff19006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 0x7ff19006f550 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8006fd0 tx=0x7ff1a8008040 comp rx=0 tx=0).stop 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 msgr2=0x7ff1ac13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7ff19c038040 tx=0x7ff19c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 shutdown_connections 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff19006d090 0x7ff19006f550 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1ac1431f0 0x7ff1ac1f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1ac0b2f60 0x7ff1ac13f950 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 --2- 172.21.15.107:0/2027273234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1ac0b2ad0 0x7ff1ac0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 >> 172.21.15.107:0/2027273234 conn(0x7ff1ac0cb8a0 msgr2=0x7ff1ac06d620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 shutdown_connections 2023-11-24T08:22:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:24.997+0000 7ff1b2c67640 1 -- 172.21.15.107:0/2027273234 wait complete. 2023-11-24T08:22:25.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs subvol_clone7628 2023-11-24T08:22:25.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.224+0000 7f3bc189b640 1 Processor -- start 2023-11-24T08:22:25.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.224+0000 7f3bc189b640 1 -- start start 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bc189b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3bbc0d30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bc189b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc0cdf30 con 0x7f3bbc0d3630 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bc189b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc0d3b50 con 0x7f3bbc15d4f0 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bc189b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0d3e30 con 0x7f3bbc0d0a50 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bbb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3bbc0d3630 0x7f3bbc0d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41608/0 (socket says 172.21.15.107:41608) 2023-11-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.226+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2693082509 learned_addr learned my addr 172.21.15.107:0/2693082509 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:25.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3bbc0d30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203429213 0 0) 0x7f3bbc0cdf30 con 0x7f3bbc0d3630 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba8003520 con 0x7f3bbc0d3630 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1430563330 0 0) 0x7f3ba8003520 con 0x7f3bbc0d3630 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 msgr2=0x7f3bbc0d30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3bbc0d30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 >> v1:172.21.15.120:6789/0 conn(0x7f3bbc15d4f0 legacy=0x7f3bbc0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0ce1d0 con 0x7f3bbc0d3630 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3bb40019f0 con 0x7f3bbc0d3630 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 >> v1:172.21.15.107:6789/0 conn(0x7f3bbc0d3630 legacy=0x7f3bbc0d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.227+0000 7f3bba7fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3ba8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bba7fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ba8004350 0x7f3ba8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bba7fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3ba8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0d3e30 con 0x7f3ba80039a0 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba800b980 con 0x7f3ba8004350 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba800bb40 con 0x7f3bbc0d0a50 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3ba8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3ba8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ba8004350 0x7f3ba8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 -- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 msgr2=0x7f3ba8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3ba8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 -- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ba8004350 msgr2=0x7f3ba8004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ba8004350 0x7f3ba8004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbaffd640 1 -- 172.21.15.107:0/2693082509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8003520 con 0x7f3ba80039a0 2023-11-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ba8004350 0x7f3ba8004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.228+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc0d0a50 0x7f3ba8005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.229+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3ba8003d80 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f3bac001860 tx=0x7f3ba80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abec3eba96664ef3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac033630 con 0x7f3ba80039a0 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac0337b0 con 0x7f3ba80039a0 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bba7fc640 1 -- 172.21.15.107:0/2693082509 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac033b60 con 0x7f3ba80039a0 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 -- 172.21.15.107:0/2693082509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 msgr2=0x7f3ba8003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 --2- 172.21.15.107:0/2693082509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3ba8003d80 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f3bac001860 tx=0x7f3ba80057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2693082509 reap_dead start 2023-11-24T08:22:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 -- 172.21.15.107:0/2693082509 shutdown_connections 2023-11-24T08:22:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 -- 172.21.15.107:0/2693082509 >> 172.21.15.107:0/2693082509 conn(0x7f3bbc061950 msgr2=0x7f3bbc08af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 -- 172.21.15.107:0/2693082509 shutdown_connections 2023-11-24T08:22:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.230+0000 7f3bc189b640 1 -- 172.21.15.107:0/2693082509 wait complete. 2023-11-24T08:22:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.232+0000 7f3bc189b640 1 Processor -- start 2023-11-24T08:22:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.232+0000 7f3bc189b640 1 -- start start 2023-11-24T08:22:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15bdf0 0x7f3bbc14bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc0cbc20 0x7f3bbc0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cf3c0 con 0x7f3ba80039a0 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cb790 con 0x7f3bbc0cbc20 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.233+0000 7f3bc189b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cab80 con 0x7f3bbc15bdf0 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41544/0 (socket says 172.21.15.107:41544) 2023-11-24T08:22:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4007013134 learned_addr learned my addr 172.21.15.107:0/4007013134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15bdf0 0x7f3bbc14bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc0cbc20 0x7f3bbc0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15bdf0 msgr2=0x7f3bbc14bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15bdf0 0x7f3bbc14bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc0cbc20 msgr2=0x7f3bbc0cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc0cbc20 0x7f3bbc0cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4007013134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0cae60 con 0x7f3ba80039a0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.234+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f3bac02bd20 tx=0x7f3bac001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.235+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac007050 con 0x7f3ba80039a0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.235+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac0071b0 con 0x7f3ba80039a0 2023-11-24T08:22:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.235+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac007520 con 0x7f3ba80039a0 2023-11-24T08:22:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.235+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc0cb140 con 0x7f3ba80039a0 2023-11-24T08:22:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.235+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc15c5e0 con 0x7f3ba80039a0 2023-11-24T08:22:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.236+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc067470 con 0x7f3ba80039a0 2023-11-24T08:22:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.237+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f3bac0067d0 con 0x7f3ba80039a0 2023-11-24T08:22:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.238+0000 7f3bb8ff9640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 0x7f3b9c06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.238+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bac0aa890 con 0x7f3ba80039a0 2023-11-24T08:22:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.238+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 0x7f3b9c06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.239+0000 7f3bb27fc640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 0x7f3b9c06f5a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b8ab0 tx=0x7f3ba4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.243+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bac077540 con 0x7f3ba80039a0 2023-11-24T08:22:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.467+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bbc0ce790 con 0x7f3b9c06d0e0 2023-11-24T08:22:25.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.488+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4007013134 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3ba40118e0 con 0x7f3b9c06d0e0 2023-11-24T08:22:25.490 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:22:25.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:25.491 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subvol_snap450" 2023-11-24T08:22:25.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:25.491 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 msgr2=0x7f3b9c06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 0x7f3b9c06f5a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b8ab0 tx=0x7f3ba4008040 comp rx=0 tx=0).stop 2023-11-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 msgr2=0x7f3bbc0ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f3bac02bd20 tx=0x7f3bac001230 comp rx=0 tx=0).stop 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 shutdown_connections 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3b9c06d0e0 0x7f3b9c06f5a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc0cbc20 0x7f3bbc0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15bdf0 0x7f3bbc14bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 --2- 172.21.15.107:0/4007013134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba80039a0 0x7f3bbc0ca4f0 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 >> 172.21.15.107:0/4007013134 conn(0x7f3bbc061950 msgr2=0x7f3bbc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 shutdown_connections 2023-11-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.491+0000 7f3bc189b640 1 -- 172.21.15.107:0/4007013134 wait complete. 2023-11-24T08:22:25.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7628 2023-11-24T08:22:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.717+0000 7fad71883640 1 Processor -- start 2023-11-24T08:22:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.717+0000 7fad71883640 1 -- start start 2023-11-24T08:22:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.719+0000 7fad71883640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.719+0000 7fad71883640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 0x7fad6c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.719+0000 7fad71883640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c0cd510 con 0x7fad6c146eb0 2023-11-24T08:22:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.719+0000 7fad71883640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c161a10 con 0x7fad6c0cd110 2023-11-24T08:22:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.719+0000 7fad71883640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c161cf0 con 0x7fad6c06e160 2023-11-24T08:22:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.720+0000 7fad63fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.720+0000 7fad63fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:60104/0 (socket says 172.21.15.107:60104) 2023-11-24T08:22:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.720+0000 7fad63fff640 1 -- 172.21.15.107:0/2222832853 learned_addr learned my addr 172.21.15.107:0/2222832853 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.720+0000 7fad6b7fe640 1 --2- 172.21.15.107:0/2222832853 >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 0x7fad6c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad63fff640 1 -- 172.21.15.107:0/2222832853 >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 msgr2=0x7fad6c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad63fff640 1 --2- 172.21.15.107:0/2222832853 >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 0x7fad6c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad63fff640 1 -- 172.21.15.107:0/2222832853 >> v1:172.21.15.120:6789/0 conn(0x7fad6c146eb0 legacy=0x7fad6c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad63fff640 1 -- 172.21.15.107:0/2222832853 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c161fd0 con 0x7fad6c06e160 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad6b7fe640 1 --2- 172.21.15.107:0/2222832853 >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 0x7fad6c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:22:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.721+0000 7fad6a7fc640 1 -- 172.21.15.107:0/2222832853 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882737572 0 0) 0x7fad6c0cd510 con 0x7fad6c146eb0 2023-11-24T08:22:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.722+0000 7fad63fff640 1 --2- 172.21.15.107:0/2222832853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9208380c6b10fa73 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.723+0000 7fad6a7fc640 1 -- 172.21.15.107:0/2222832853 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54030840 con 0x7fad6c06e160 2023-11-24T08:22:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.723+0000 7fad6a7fc640 1 -- 172.21.15.107:0/2222832853 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540309a0 con 0x7fad6c06e160 2023-11-24T08:22:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.723+0000 7fad6a7fc640 1 -- 172.21.15.107:0/2222832853 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54030d30 con 0x7fad6c06e160 2023-11-24T08:22:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.723+0000 7fad71883640 1 -- 172.21.15.107:0/2222832853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 msgr2=0x7fad6c0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.723+0000 7fad71883640 1 --2- 172.21.15.107:0/2222832853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5402b980 comp rx=0 tx=0).stop 2023-11-24T08:22:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 -- 172.21.15.107:0/2222832853 shutdown_connections 2023-11-24T08:22:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 --2- 172.21.15.107:0/2222832853 >> v2:172.21.15.120:3300/0 conn(0x7fad6c0cd110 0x7fad6c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 --2- 172.21.15.107:0/2222832853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c06e160 0x7fad6c0ccbd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 -- 172.21.15.107:0/2222832853 >> 172.21.15.107:0/2222832853 conn(0x7fad6c0b0c50 msgr2=0x7fad6c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 -- 172.21.15.107:0/2222832853 shutdown_connections 2023-11-24T08:22:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.724+0000 7fad71883640 1 -- 172.21.15.107:0/2222832853 wait complete. 2023-11-24T08:22:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.726+0000 7fad71883640 1 Processor -- start 2023-11-24T08:22:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.726+0000 7fad71883640 1 -- start start 2023-11-24T08:22:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.727+0000 7fad71883640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c06e160 0x7fad6c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad71883640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c0cd110 0x7fad6c15fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad71883640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad6c146eb0 0x7fad6c0ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad71883640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b1d10 con 0x7fad6c06e160 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad71883640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b2890 con 0x7fad6c146eb0 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad71883640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b4df0 con 0x7fad6c0cd110 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c06e160 0x7fad6c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c06e160 0x7fad6c0b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41546/0 (socket says 172.21.15.107:41546) 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 -- 172.21.15.107:0/1594914854 learned_addr learned my addr 172.21.15.107:0/1594914854 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad63fff640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c0cd110 0x7fad6c15fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6b7fe640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad6c146eb0 0x7fad6c0ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 -- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c0cd110 msgr2=0x7fad6c15fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad6c0cd110 0x7fad6c15fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 -- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad6c146eb0 msgr2=0x7fad6c0ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad6c146eb0 0x7fad6c0ad0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.728+0000 7fad6affd640 1 -- 172.21.15.107:0/1594914854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64009a70 con 0x7fad6c06e160 2023-11-24T08:22:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad6affd640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c06e160 0x7fad6c0b48b0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c410 tx=0x7fad5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c0035f0 con 0x7fad6c06e160 2023-11-24T08:22:25.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c003770 con 0x7fad6c06e160 2023-11-24T08:22:25.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c003b20 con 0x7fad6c06e160 2023-11-24T08:22:25.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad71883640 1 -- 172.21.15.107:0/1594914854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad540094d0 con 0x7fad6c06e160 2023-11-24T08:22:25.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.729+0000 7fad71883640 1 -- 172.21.15.107:0/1594914854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c0b2240 con 0x7fad6c06e160 2023-11-24T08:22:25.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.730+0000 7fad71883640 1 -- 172.21.15.107:0/1594914854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c04e8b0 con 0x7fad6c06e160 2023-11-24T08:22:25.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.731+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fad5c003c80 con 0x7fad6c06e160 2023-11-24T08:22:25.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.732+0000 7fad68ff9640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad4c06d090 0x7fad4c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:25.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.732+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad5c087d70 con 0x7fad6c06e160 2023-11-24T08:22:25.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.733+0000 7fad63fff640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad4c06d090 0x7fad4c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:25.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.733+0000 7fad63fff640 1 --2- 172.21.15.107:0/1594914854 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad4c06d090 0x7fad4c06f550 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fad54036000 tx=0x7fad54008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:25.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.737+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad5c054920 con 0x7fad6c06e160 2023-11-24T08:22:25.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.963+0000 7fad71883640 1 -- 172.21.15.107:0/1594914854 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7fad6c147290 con 0x7fad4c06d090 2023-11-24T08:22:25.973 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:25.971+0000 7f376379e640 -1 mgr.server reply reply (2) No such file or directory subvolume "subvol_clone7628" is removed and has only snapshots retained 2023-11-24T08:22:25.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:25.972+0000 7fad68ff9640 1 -- 172.21.15.107:0/1594914854 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: -2 subvolume "subvol_clone7628" is removed and has only snapshots retained) v1 ==== 79+0+0 (secure 0 0 0) 0x7fad54030b90 con 0x7fad4c06d090 2023-11-24T08:22:25.974 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENOENT: subvolume "subvol_clone7628" is removed and has only snapshots retained 2023-11-24T08:22:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-24T08:22:25.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7628 2023-11-24T08:22:26.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.199+0000 7f0da91cf640 1 Processor -- start 2023-11-24T08:22:26.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.200+0000 7f0da91cf640 1 -- start start 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da91cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0da4160870 0x7f0da40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da91cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0da40b4330 0x7f0da40b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da91cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da40d4050 con 0x7f0da4160c50 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da91cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da40b3760 con 0x7f0da4160870 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da91cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da40b3a40 con 0x7f0da40b4330 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0da4160870 0x7f0da40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.201+0000 7f0da3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0da4160870 0x7f0da40b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41560/0 (socket says 172.21.15.107:41560) 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.202+0000 7f0da3fff640 1 -- 172.21.15.107:0/786297126 learned_addr learned my addr 172.21.15.107:0/786297126 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:26.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.202+0000 7f0da3fff640 1 -- 172.21.15.107:0/786297126 >> v2:172.21.15.120:3300/0 conn(0x7f0da40b4330 msgr2=0x7f0da40b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:26.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.202+0000 7f0da3fff640 1 --2- 172.21.15.107:0/786297126 >> v2:172.21.15.120:3300/0 conn(0x7f0da40b4330 0x7f0da40b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.202+0000 7f0da3fff640 1 -- 172.21.15.107:0/786297126 >> v1:172.21.15.120:6789/0 conn(0x7f0da4160c50 legacy=0x7f0da40b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:26.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.202+0000 7f0da3fff640 1 -- 172.21.15.107:0/786297126 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da40b3d20 con 0x7f0da4160870 2023-11-24T08:22:26.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.203+0000 7f0da3fff640 1 --2- 172.21.15.107:0/786297126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da4160870 0x7f0da40b47f0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f0da406b130 tx=0x7f0d9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718e5d8ead08b36a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.203+0000 7f0da2ffd640 1 -- 172.21.15.107:0/786297126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9402ca80 con 0x7f0da4160870 2023-11-24T08:22:26.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.203+0000 7f0da2ffd640 1 -- 172.21.15.107:0/786297126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9402cbe0 con 0x7f0da4160870 2023-11-24T08:22:26.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.203+0000 7f0da2ffd640 1 -- 172.21.15.107:0/786297126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d940343f0 con 0x7f0da4160870 2023-11-24T08:22:26.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 -- 172.21.15.107:0/786297126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da4160870 msgr2=0x7f0da40b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 --2- 172.21.15.107:0/786297126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da4160870 0x7f0da40b47f0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f0da406b130 tx=0x7f0d9402bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 -- 172.21.15.107:0/786297126 shutdown_connections 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 --2- 172.21.15.107:0/786297126 >> v2:172.21.15.120:3300/0 conn(0x7f0da40b4330 0x7f0da40b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 --2- 172.21.15.107:0/786297126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da4160870 0x7f0da40b47f0 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 -- 172.21.15.107:0/786297126 >> 172.21.15.107:0/786297126 conn(0x7f0da409c5c0 msgr2=0x7f0da40d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 -- 172.21.15.107:0/786297126 shutdown_connections 2023-11-24T08:22:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.204+0000 7f0da91cf640 1 -- 172.21.15.107:0/786297126 wait complete. 2023-11-24T08:22:26.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.206+0000 7f0da91cf640 1 Processor -- start 2023-11-24T08:22:26.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.206+0000 7f0da91cf640 1 -- start start 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da40b4330 0x7f0da40d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0da4160870 0x7f0da4152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0da4160c50 0x7f0da40c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40c8a60 con 0x7f0da40b4330 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40cf210 con 0x7f0da4160c50 2023-11-24T08:22:26.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da91cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40cec00 con 0x7f0da4160870 2023-11-24T08:22:26.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da40b4330 0x7f0da40d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da40b4330 0x7f0da40d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41576/0 (socket says 172.21.15.107:41576) 2023-11-24T08:22:26.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.207+0000 7f0da3fff640 1 -- 172.21.15.107:0/1452140079 learned_addr learned my addr 172.21.15.107:0/1452140079 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da37fe640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0da4160870 0x7f0da4152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 -- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0da4160870 msgr2=0x7f0da4152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0da4160870 0x7f0da4152de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 -- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0da4160c50 msgr2=0x7f0da40c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da89ce640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0da4160c50 0x7f0da40c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0da4160c50 0x7f0da40c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 -- 172.21.15.107:0/1452140079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d94009b30 con 0x7f0da40b4330 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da89ce640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0da4160c50 0x7f0da40c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da3fff640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0da40b4330 0x7f0da40d08a0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f0d940093e0 tx=0x7f0d94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9403e450 con 0x7f0da40b4330 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9403e5b0 con 0x7f0da40b4330 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9403e960 con 0x7f0da40b4330 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.208+0000 7f0da91cf640 1 -- 172.21.15.107:0/1452140079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da40c79c0 con 0x7f0da40b4330 2023-11-24T08:22:26.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.209+0000 7f0da91cf640 1 -- 172.21.15.107:0/1452140079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da4153320 con 0x7f0da40b4330 2023-11-24T08:22:26.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.210+0000 7f0da91cf640 1 -- 172.21.15.107:0/1452140079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da404e840 con 0x7f0da40b4330 2023-11-24T08:22:26.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.211+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f0d9402cd50 con 0x7f0da40b4330 2023-11-24T08:22:26.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.212+0000 7f0da0ff9640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d8406d090 0x7f0d8406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.212+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d940ab410 con 0x7f0da40b4330 2023-11-24T08:22:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.212+0000 7f0da37fe640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d8406d090 0x7f0d8406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.213+0000 7f0da37fe640 1 --2- 172.21.15.107:0/1452140079 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d8406d090 0x7f0d8406f550 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0da40ceb10 tx=0x7f0d8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.216+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d94077fc0 con 0x7f0da40b4330 2023-11-24T08:22:26.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.466+0000 7f0da91cf640 1 -- 172.21.15.107:0/1452140079 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7628", "target": ["mon-mgr", ""]}) v1 -- 0x7f0da40b5890 con 0x7f0d8406d090 2023-11-24T08:22:26.482 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:22:26.480+0000 7f376379e640 -1 mgr.server reply reply (2) No such file or directory subvolume "subvol_clone7628" is removed and has only snapshots retained 2023-11-24T08:22:26.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.481+0000 7f0da0ff9640 1 -- 172.21.15.107:0/1452140079 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: -2 subvolume "subvol_clone7628" is removed and has only snapshots retained) v1 ==== 79+0+0 (secure 0 0 0) 0x7f0da40b5890 con 0x7f0d8406d090 2023-11-24T08:22:26.483 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENOENT: subvolume "subvol_clone7628" is removed and has only snapshots retained 2023-11-24T08:22:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-24T08:22:26.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol239 subvol_snap2348 2023-11-24T08:22:26.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.709+0000 7fe950fa9640 1 Processor -- start 2023-11-24T08:22:26.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.709+0000 7fe950fa9640 1 -- start start 2023-11-24T08:22:26.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.710+0000 7fe950fa9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe94c15c7f0 0x7fe94c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.710+0000 7fe950fa9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe94c0cf450 0x7fe94c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe950fa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0cece0 con 0x7fe94c0cdfb0 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe950fa9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94c0cfe40 con 0x7fe94c15c7f0 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe950fa9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94c0ceaf0 con 0x7fe94c0cf450 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe94ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe94c0cdfb0 0x7fe94c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41618/0 (socket says 172.21.15.107:41618) 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe94ad9d640 1 -- 172.21.15.107:0/4267020195 learned_addr learned my addr 172.21.15.107:0/4267020195 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:26.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.711+0000 7fe94a59c640 1 --2- 172.21.15.107:0/4267020195 >> v2:172.21.15.107:3300/0 conn(0x7fe94c15c7f0 0x7fe94c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe943fff640 1 --2- 172.21.15.107:0/4267020195 >> v2:172.21.15.120:3300/0 conn(0x7fe94c0cf450 0x7fe94c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe949d9b640 1 -- 172.21.15.107:0/4267020195 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215567962 0 0) 0x7fe94c0cece0 con 0x7fe94c0cdfb0 2023-11-24T08:22:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe94a59c640 1 -- 172.21.15.107:0/4267020195 >> v2:172.21.15.120:3300/0 conn(0x7fe94c0cf450 msgr2=0x7fe94c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe94a59c640 1 --2- 172.21.15.107:0/4267020195 >> v2:172.21.15.120:3300/0 conn(0x7fe94c0cf450 0x7fe94c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe94a59c640 1 -- 172.21.15.107:0/4267020195 >> v1:172.21.15.107:6789/0 conn(0x7fe94c0cdfb0 legacy=0x7fe94c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.712+0000 7fe94a59c640 1 -- 172.21.15.107:0/4267020195 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c06d040 con 0x7fe94c15c7f0 2023-11-24T08:22:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.713+0000 7fe94a59c640 1 --2- 172.21.15.107:0/4267020195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c06db20 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe94c070410 tx=0x7fe93c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f0bf195363b03e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.713+0000 7fe949d9b640 1 -- 172.21.15.107:0/4267020195 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c031830 con 0x7fe94c15c7f0 2023-11-24T08:22:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.713+0000 7fe949d9b640 1 -- 172.21.15.107:0/4267020195 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c031990 con 0x7fe94c15c7f0 2023-11-24T08:22:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.713+0000 7fe949d9b640 1 -- 172.21.15.107:0/4267020195 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c031d20 con 0x7fe94c15c7f0 2023-11-24T08:22:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 -- 172.21.15.107:0/4267020195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 msgr2=0x7fe94c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 --2- 172.21.15.107:0/4267020195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c06db20 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe94c070410 tx=0x7fe93c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 -- 172.21.15.107:0/4267020195 shutdown_connections 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 --2- 172.21.15.107:0/4267020195 >> v2:172.21.15.120:3300/0 conn(0x7fe94c0cf450 0x7fe94c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 --2- 172.21.15.107:0/4267020195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c06db20 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 -- 172.21.15.107:0/4267020195 >> 172.21.15.107:0/4267020195 conn(0x7fe94c06a920 msgr2=0x7fe94c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 -- 172.21.15.107:0/4267020195 shutdown_connections 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.714+0000 7fe950fa9640 1 -- 172.21.15.107:0/4267020195 wait complete. 2023-11-24T08:22:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.716+0000 7fe950fa9640 1 Processor -- start 2023-11-24T08:22:26.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.716+0000 7fe950fa9640 1 -- start start 2023-11-24T08:22:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe94c0cdfb0 0x7fe94c1473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe94c0cf450 0x7fe94c1f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c09b9c0 con 0x7fe94c15c7f0 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c09bc40 con 0x7fe94c0cdfb0 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe950fa9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c1f19d0 con 0x7fe94c0cf450 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41594/0 (socket says 172.21.15.107:41594) 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94ad9d640 1 -- 172.21.15.107:0/2750018271 learned_addr learned my addr 172.21.15.107:0/2750018271 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe943fff640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe94c0cf450 0x7fe94c1f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94a59c640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe94c0cdfb0 0x7fe94c1473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94ad9d640 1 -- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe94c0cf450 msgr2=0x7fe94c1f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.717+0000 7fe94ad9d640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe94c0cf450 0x7fe94c1f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe94ad9d640 1 -- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe94c0cdfb0 msgr2=0x7fe94c1473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:26.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe94ad9d640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe94c0cdfb0 0x7fe94c1473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:26.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe94ad9d640 1 -- 172.21.15.107:0/2750018271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c009400 con 0x7fe94c15c7f0 2023-11-24T08:22:26.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe94ad9d640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c800 tx=0x7fe94400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944004d60 con 0x7fe94c15c7f0 2023-11-24T08:22:26.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94400a640 con 0x7fe94c15c7f0 2023-11-24T08:22:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400aa30 con 0x7fe94c15c7f0 2023-11-24T08:22:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c09bf00 con 0x7fe94c15c7f0 2023-11-24T08:22:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.718+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c1478e0 con 0x7fe94c15c7f0 2023-11-24T08:22:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.720+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c09c420 con 0x7fe94c15c7f0 2023-11-24T08:22:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.720+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe94400ab90 con 0x7fe94c15c7f0 2023-11-24T08:22:26.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.722+0000 7fe942ffd640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 0x7fe92c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:26.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.722+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe944087be0 con 0x7fe94c15c7f0 2023-11-24T08:22:26.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.722+0000 7fe94a59c640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 0x7fe92c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:26.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.723+0000 7fe94a59c640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 0x7fe92c06f550 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe93c036000 tx=0x7fe93c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:26.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.726+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe94c09c420 con 0x7fe94c15c7f0 2023-11-24T08:22:26.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:26.954+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol239", "snap_name": "subvol_snap2348", "target": ["mon-mgr", ""]}) v1 -- 0x7fe94c06e060 con 0x7fe92c06d090 2023-11-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.039+0000 7fe942ffd640 1 -- 172.21.15.107:0/2750018271 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe93c031b00 con 0x7fe92c06d090 2023-11-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 msgr2=0x7fe92c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 0x7fe92c06f550 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe93c036000 tx=0x7fe93c008030 comp rx=0 tx=0).stop 2023-11-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 msgr2=0x7fe94c099b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c800 tx=0x7fe94400ccc0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 shutdown_connections 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe92c06d090 0x7fe92c06f550 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe94c15c7f0 0x7fe94c099b00 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe94c0cf450 0x7fe94c1f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 --2- 172.21.15.107:0/2750018271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe94c0cdfb0 0x7fe94c1473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.042+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 >> 172.21.15.107:0/2750018271 conn(0x7fe94c06a920 msgr2=0x7fe94c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.043+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 shutdown_connections 2023-11-24T08:22:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.043+0000 7fe950fa9640 1 -- 172.21.15.107:0/2750018271 wait complete. 2023-11-24T08:22:27.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol_clone7628 subvol_snap450 2023-11-24T08:22:27.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.266+0000 7f7cf7a75640 1 Processor -- start 2023-11-24T08:22:27.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.267+0000 7f7cf7a75640 1 -- start start 2023-11-24T08:22:27.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf7a75640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cf0157c70 0x7f7cf0161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf7a75640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7cf015e390 0x7f7cf00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf7a75640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf00ce0b0 con 0x7f7cf00cf450 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf7a75640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf00cfe40 con 0x7f7cf0157c70 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf7a75640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf0161960 con 0x7f7cf015e390 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf5811640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cf0157c70 0x7f7cf0161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf5811640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cf0157c70 0x7f7cf0161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41600/0 (socket says 172.21.15.107:41600) 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.268+0000 7f7cf5811640 1 -- 172.21.15.107:0/614865747 learned_addr learned my addr 172.21.15.107:0/614865747 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:27.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.269+0000 7f7cf5811640 1 -- 172.21.15.107:0/614865747 >> v2:172.21.15.120:3300/0 conn(0x7f7cf015e390 msgr2=0x7f7cf00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:27.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.269+0000 7f7cf5811640 1 --2- 172.21.15.107:0/614865747 >> v2:172.21.15.120:3300/0 conn(0x7f7cf015e390 0x7f7cf00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.269+0000 7f7cf5811640 1 -- 172.21.15.107:0/614865747 >> v1:172.21.15.120:6789/0 conn(0x7f7cf00cf450 legacy=0x7f7cf015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:27.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.269+0000 7f7cf5811640 1 -- 172.21.15.107:0/614865747 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf015e8d0 con 0x7f7cf0157c70 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.270+0000 7f7cf5811640 1 --2- 172.21.15.107:0/614865747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf0157c70 0x7f7cf0161df0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00d0100 tx=0x7f7cec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b36eedf7a53614 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.270+0000 7f7cf480f640 1 -- 172.21.15.107:0/614865747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0317e0 con 0x7f7cf0157c70 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.270+0000 7f7cf480f640 1 -- 172.21.15.107:0/614865747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec031940 con 0x7f7cf0157c70 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.270+0000 7f7cf480f640 1 -- 172.21.15.107:0/614865747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec031cd0 con 0x7f7cf0157c70 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 -- 172.21.15.107:0/614865747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf0157c70 msgr2=0x7f7cf0161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/614865747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf0157c70 0x7f7cf0161df0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00d0100 tx=0x7f7cec02b880 comp rx=0 tx=0).stop 2023-11-24T08:22:27.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 -- 172.21.15.107:0/614865747 shutdown_connections 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/614865747 >> v2:172.21.15.120:3300/0 conn(0x7f7cf015e390 0x7f7cf00cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/614865747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf0157c70 0x7f7cf0161df0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 -- 172.21.15.107:0/614865747 >> 172.21.15.107:0/614865747 conn(0x7f7cf006a920 msgr2=0x7f7cf006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 -- 172.21.15.107:0/614865747 shutdown_connections 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.271+0000 7f7cf7a75640 1 -- 172.21.15.107:0/614865747 wait complete. 2023-11-24T08:22:27.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.273+0000 7f7cf7a75640 1 Processor -- start 2023-11-24T08:22:27.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.273+0000 7f7cf7a75640 1 -- start start 2023-11-24T08:22:27.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf00cf450 0x7f7cf0098920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf0157c70 0x7f7cf014d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf014fec0 con 0x7f7cf015e390 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0098e60 con 0x7f7cf00cf450 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf7a75640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf014f590 con 0x7f7cf0157c70 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41602/0 (socket says 172.21.15.107:41602) 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 -- 172.21.15.107:0/2293892400 learned_addr learned my addr 172.21.15.107:0/2293892400 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf5811640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf00cf450 0x7f7cf0098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 -- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf0157c70 msgr2=0x7f7cf014d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf0157c70 0x7f7cf014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 -- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf00cf450 msgr2=0x7f7cf0098920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf00cf450 0x7f7cf0098920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.274+0000 7f7cf6012640 1 -- 172.21.15.107:0/2293892400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec0094d0 con 0x7f7cf015e390 2023-11-24T08:22:27.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7cf6012640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b9e0 tx=0x7f7ce800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800caa0 con 0x7f7cf015e390 2023-11-24T08:22:27.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce800cc20 con 0x7f7cf015e390 2023-11-24T08:22:27.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8014440 con 0x7f7cf015e390 2023-11-24T08:22:27.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf014f870 con 0x7f7cf015e390 2023-11-24T08:22:27.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.275+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf014f1b0 con 0x7f7cf015e390 2023-11-24T08:22:27.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.276+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf0150030 con 0x7f7cf015e390 2023-11-24T08:22:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.277+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f7ce801d3b0 con 0x7f7cf015e390 2023-11-24T08:22:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.278+0000 7f7ce27fc640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 0x7f7cd406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.278+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ce80885e0 con 0x7f7cf015e390 2023-11-24T08:22:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.278+0000 7f7cf5811640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 0x7f7cd406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.279+0000 7f7cf5811640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 0x7f7cd406f550 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7cec036000 tx=0x7f7cec03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.282+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ce8055190 con 0x7f7cf015e390 2023-11-24T08:22:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.506+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol_clone7628", "snap_name": "subvol_snap450", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cf014f700 con 0x7f7cd406d090 2023-11-24T08:22:27.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.609+0000 7f7ce27fc640 1 -- 172.21.15.107:0/2293892400 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7cf014f700 con 0x7f7cd406d090 2023-11-24T08:22:27.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 msgr2=0x7f7cd406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 0x7f7cd406f550 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7cec036000 tx=0x7f7cec03d040 comp rx=0 tx=0).stop 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 msgr2=0x7f7cf006ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b9e0 tx=0x7f7ce800bea0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 shutdown_connections 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7cd406d090 0x7f7cd406f550 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf015e390 0x7f7cf006ccd0 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf0157c70 0x7f7cf014d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 --2- 172.21.15.107:0/2293892400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf00cf450 0x7f7cf0098920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.613+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 >> 172.21.15.107:0/2293892400 conn(0x7f7cf006a920 msgr2=0x7f7cf01568d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.614+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 shutdown_connections 2023-11-24T08:22:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.614+0000 7f7cf7a75640 1 -- 172.21.15.107:0/2293892400 wait complete. 2023-11-24T08:22:27.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2023-11-24T08:22:27.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.828+0000 7fb35ee2b640 1 Processor -- start 2023-11-24T08:22:27.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.828+0000 7fb35ee2b640 1 -- start start 2023-11-24T08:22:27.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb35ee2b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb35806e160 0x7fb3580ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb35ee2b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb3580cd110 0x7fb358070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb35ee2b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358161a10 con 0x7fb358146eb0 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb35ee2b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580cd580 con 0x7fb35806e160 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb35ee2b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb358161cf0 con 0x7fb3580cd110 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb35806e160 0x7fb3580ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb35806e160 0x7fb3580ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41616/0 (socket says 172.21.15.107:41616) 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 -- 172.21.15.107:0/144107843 learned_addr learned my addr 172.21.15.107:0/144107843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:27.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 -- 172.21.15.107:0/144107843 >> v2:172.21.15.120:3300/0 conn(0x7fb3580cd110 msgr2=0x7fb358070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:27.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 --2- 172.21.15.107:0/144107843 >> v2:172.21.15.120:3300/0 conn(0x7fb3580cd110 0x7fb358070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 -- 172.21.15.107:0/144107843 >> v1:172.21.15.120:6789/0 conn(0x7fb358146eb0 legacy=0x7fb358161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:27.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.830+0000 7fb357fff640 1 -- 172.21.15.107:0/144107843 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358161fd0 con 0x7fb35806e160 2023-11-24T08:22:27.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.832+0000 7fb357fff640 1 --2- 172.21.15.107:0/144107843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35806e160 0x7fb3580ccbd0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fb3580ae150 tx=0x7fb34002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74d3ade61765d90 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.832+0000 7fb3577fe640 1 -- 172.21.15.107:0/144107843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340031870 con 0x7fb35806e160 2023-11-24T08:22:27.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.832+0000 7fb3577fe640 1 -- 172.21.15.107:0/144107843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3400319d0 con 0x7fb35806e160 2023-11-24T08:22:27.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.832+0000 7fb3577fe640 1 -- 172.21.15.107:0/144107843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340031d60 con 0x7fb35806e160 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 -- 172.21.15.107:0/144107843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35806e160 msgr2=0x7fb3580ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/144107843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35806e160 0x7fb3580ccbd0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fb3580ae150 tx=0x7fb34002b910 comp rx=0 tx=0).stop 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 -- 172.21.15.107:0/144107843 shutdown_connections 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/144107843 >> v2:172.21.15.120:3300/0 conn(0x7fb3580cd110 0x7fb358070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/144107843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35806e160 0x7fb3580ccbd0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 -- 172.21.15.107:0/144107843 >> 172.21.15.107:0/144107843 conn(0x7fb3580b0c50 msgr2=0x7fb3580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:27.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 -- 172.21.15.107:0/144107843 shutdown_connections 2023-11-24T08:22:27.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.833+0000 7fb35ee2b640 1 -- 172.21.15.107:0/144107843 wait complete. 2023-11-24T08:22:27.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.835+0000 7fb35ee2b640 1 Processor -- start 2023-11-24T08:22:27.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.836+0000 7fb35ee2b640 1 -- start start 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 0x7fb35815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb358146eb0 0x7fb3581450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35815cf40 con 0x7fb3580cd110 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3581600b0 con 0x7fb35806e160 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35ee2b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b1e40 con 0x7fb358146eb0 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41630/0 (socket says 172.21.15.107:41630) 2023-11-24T08:22:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 -- 172.21.15.107:0/1900876780 learned_addr learned my addr 172.21.15.107:0/1900876780 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:27.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35cbc7640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 0x7fb35815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 -- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb358146eb0 msgr2=0x7fb3581450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb358146eb0 0x7fb3581450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 -- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 msgr2=0x7fb35815ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 0x7fb35815ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb357fff640 1 -- 172.21.15.107:0/1900876780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400094d0 con 0x7fb3580cd110 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.837+0000 7fb35cbc7640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 0x7fb35815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb357fff640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb3400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34003ad10 con 0x7fb3580cd110 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34003ae70 con 0x7fb3580cd110 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3580b2120 con 0x7fb3580cd110 2023-11-24T08:22:27.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340044610 con 0x7fb3580cd110 2023-11-24T08:22:27.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.838+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3580b26e0 con 0x7fb3580cd110 2023-11-24T08:22:27.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.839+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3580cb560 con 0x7fb3580cd110 2023-11-24T08:22:27.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.841+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fb34004d450 con 0x7fb3580cd110 2023-11-24T08:22:27.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.842+0000 7fb3557fa640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 0x7fb33806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:27.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.842+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb34002d040 con 0x7fb3580cd110 2023-11-24T08:22:27.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.842+0000 7fb35cbc7640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 0x7fb33806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:27.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.843+0000 7fb35cbc7640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 0x7fb33806f550 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb348007eb0 tx=0x7fb3480073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:27.847+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3400779f0 con 0x7fb3580cd110 2023-11-24T08:22:28.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.061+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7fb358147290 con 0x7fb33806d090 2023-11-24T08:22:28.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.069+0000 7fb3557fa640 1 -- 172.21.15.107:0/1900876780 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fb34800cdd0 con 0x7fb33806d090 2023-11-24T08:22:28.071 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2023-11-24T08:22:28.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 msgr2=0x7fb33806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 0x7fb33806f550 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb348007eb0 tx=0x7fb3480073c0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 msgr2=0x7fb35815f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb3400046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 shutdown_connections 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb33806d090 0x7fb33806f550 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb358146eb0 0x7fb3581450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3580cd110 0x7fb35815f7b0 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 --2- 172.21.15.107:0/1900876780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb35806e160 0x7fb35815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 >> 172.21.15.107:0/1900876780 conn(0x7fb3580b0c50 msgr2=0x7fb3580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:28.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 shutdown_connections 2023-11-24T08:22:28.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.072+0000 7fb35ee2b640 1 -- 172.21.15.107:0/1900876780 wait complete. 2023-11-24T08:22:28.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:28.092 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:22:28.146 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:22:28.147 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:22:28.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:22:28.268 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:22:28.394 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:22:28.394 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:28.394 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:28.452 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:28.452 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:28.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.769+0000 7fdb4fda9640 1 Processor -- start 2023-11-24T08:22:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.769+0000 7fdb4fda9640 1 -- start start 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4fda9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb480cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4fda9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb480cece0 con 0x7fdb4815c7f0 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4fda9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4806d040 con 0x7fdb480cdfb0 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4fda9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48156010 con 0x7fdb480cf450 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4db45640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdb4815c7f0 0x7fdb4806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60198/0 (socket says 172.21.15.107:60198) 2023-11-24T08:22:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4db45640 1 -- 172.21.15.107:0/1018294574 learned_addr learned my addr 172.21.15.107:0/1018294574 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.771+0000 7fdb4d344640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb480cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808124550 0 0) 0x7fdb480cece0 con 0x7fdb4815c7f0 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb34003520 con 0x7fdb4815c7f0 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426000433 0 0) 0x7fdb34003520 con 0x7fdb4815c7f0 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 msgr2=0x7fdb480cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb480cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 >> v1:172.21.15.120:6789/0 conn(0x7fdb480cdfb0 legacy=0x7fdb480ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4806d320 con 0x7fdb4815c7f0 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fdb44001670 con 0x7fdb4815c7f0 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 >> v1:172.21.15.107:6789/0 conn(0x7fdb4815c7f0 legacy=0x7fdb4806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb34003fd0 0x7fdb340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d00 0x7fdb34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48156010 con 0x7fdb34004d00 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3400ff50 con 0x7fdb34003fd0 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.772+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb34010110 con 0x7fdb480cf450 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d00 0x7fdb34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4db45640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4d344640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb34003fd0 0x7fdb340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 -- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 msgr2=0x7fdb340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:28.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb480cf450 0x7fdb340039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 -- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb34003fd0 msgr2=0x7fdb340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:28.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb34003fd0 0x7fdb340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.773+0000 7fdb4e346640 1 -- 172.21.15.107:0/1018294574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34003520 con 0x7fdb34004d00 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.774+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d00 0x7fdb34005180 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fdb40001e50 tx=0x7fdb340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93cbada1f2e9ddc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.774+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40003b70 con 0x7fdb34004d00 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.774+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb40003cd0 con 0x7fdb34004d00 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.774+0000 7fdb4cb43640 1 -- 172.21.15.107:0/1018294574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40005310 con 0x7fdb34004d00 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1018294574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d00 msgr2=0x7fdb34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1018294574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d00 0x7fdb34005180 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fdb40001e50 tx=0x7fdb340057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4e346640 1 -- 172.21.15.107:0/1018294574 reap_dead start 2023-11-24T08:22:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1018294574 shutdown_connections 2023-11-24T08:22:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1018294574 >> 172.21.15.107:0/1018294574 conn(0x7fdb4806a920 msgr2=0x7fdb4806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1018294574 shutdown_connections 2023-11-24T08:22:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.775+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1018294574 wait complete. 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.777+0000 7fdb4fda9640 1 Processor -- start 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.777+0000 7fdb4fda9640 1 -- start start 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 0x7fdb480b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4814bf10 0x7fdb480b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480af910 con 0x7fdb4814c370 2023-11-24T08:22:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480afbf0 con 0x7fdb4814bf10 2023-11-24T08:22:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4fda9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480afed0 con 0x7fdb34004d00 2023-11-24T08:22:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4e346640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4e346640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56790/0 (socket says 172.21.15.107:56790) 2023-11-24T08:22:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.778+0000 7fdb4e346640 1 -- 172.21.15.107:0/1295019402 learned_addr learned my addr 172.21.15.107:0/1295019402 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4db45640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 0x7fdb480b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 -- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 msgr2=0x7fdb480b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 0x7fdb480b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 -- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4814bf10 msgr2=0x7fdb480b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4814bf10 0x7fdb480b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 -- 172.21.15.107:0/1295019402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb480b01b0 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4db45640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 0x7fdb480b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4e346640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7fdb40001f70 tx=0x7fdb400014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4003e070 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb40003e40 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40006d90 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.779+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb480b0490 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.780+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb480aec50 con 0x7fdb4814c370 2023-11-24T08:22:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.781+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb480b0680 con 0x7fdb4814c370 2023-11-24T08:22:28.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.782+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fdb40007030 con 0x7fdb4814c370 2023-11-24T08:22:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.783+0000 7fdb3e7fc640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 0x7fdb2c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.783+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb400a9a60 con 0x7fdb4814c370 2023-11-24T08:22:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.784+0000 7fdb4db45640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 0x7fdb2c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.784+0000 7fdb4db45640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 0x7fdb2c06f550 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdb440017a0 tx=0x7fdb44001d80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.788+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb40076630 con 0x7fdb4814c370 2023-11-24T08:22:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.981+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb48067470 con 0x7fdb4814c370 2023-11-24T08:22:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.984+0000 7fdb3e7fc640 1 -- 172.21.15.107:0/1295019402 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16511 (secure 0 0 0) 0x7fdb40048030 con 0x7fdb4814c370 2023-11-24T08:22:28.986 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:28.986 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":207,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:52.567075+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.986+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 msgr2=0x7fdb2c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.986+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 0x7fdb2c06f550 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdb440017a0 tx=0x7fdb44001d80 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.986+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 msgr2=0x7fdb480b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.986+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7fdb40001f70 tx=0x7fdb400014a0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 shutdown_connections 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb2c06d090 0x7fdb2c06f550 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb4814c370 0x7fdb480b3100 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4814bf10 0x7fdb480b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 --2- 172.21.15.107:0/1295019402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb34004d00 0x7fdb480b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 >> 172.21.15.107:0/1295019402 conn(0x7fdb4806a920 msgr2=0x7fdb480b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 shutdown_connections 2023-11-24T08:22:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:28.987+0000 7fdb4fda9640 1 -- 172.21.15.107:0/1295019402 wait complete. 2023-11-24T08:22:29.009 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:21:48.833467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '206', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:21:50.037981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:22:29.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:22:29.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.218+0000 7f9e6086b640 1 Processor -- start 2023-11-24T08:22:29.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.218+0000 7f9e6086b640 1 -- start start 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e6086b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e5c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e6086b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c15eea0 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e6086b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0abf30 con 0x7f9e5c15b440 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e6086b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15b820 con 0x7f9e5c0cc0f0 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e5ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9e5c0c57e0 0x7f9e5c06c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60202/0 (socket says 172.21.15.107:60202) 2023-11-24T08:22:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.220+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/2671491737 learned_addr learned my addr 172.21.15.107:0/2671491737 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e5c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180367196 0 0) 0x7f9e5c15eea0 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e44003520 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431435824 0 0) 0x7f9e44003520 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 msgr2=0x7f9e5c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e5c0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 >> v1:172.21.15.120:6789/0 conn(0x7f9e5c15b440 legacy=0x7f9e5c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c15f0f0 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f9e540019f0 con 0x7f9e5c0c57e0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 >> v1:172.21.15.107:6789/0 conn(0x7f9e5c0c57e0 legacy=0x7f9e5c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e440039a0 0x7f9e44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e440047c0 0x7f9e44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15b820 con 0x7f9e440047c0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4400bae0 con 0x7f9e440039a0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.221+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e440100b0 con 0x7f9e5c0cc0f0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e440047c0 0x7f9e44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 -- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 msgr2=0x7f9e440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e5ad9d640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e5c0cc0f0 0x7f9e440051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 -- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e440039a0 msgr2=0x7f9e44003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e440039a0 0x7f9e44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.222+0000 7f9e59d9b640 1 -- 172.21.15.107:0/2671491737 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44003520 con 0x7f9e440047c0 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.223+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e440047c0 0x7f9e44004c40 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009ac0 tx=0x7f9e440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a67ddb9c8b6557 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.223+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5002c6b0 con 0x7f9e440047c0 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.223+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5002c810 con 0x7f9e440047c0 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.223+0000 7f9e5959a640 1 -- 172.21.15.107:0/2671491737 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5002cba0 con 0x7f9e440047c0 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 -- 172.21.15.107:0/2671491737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e440047c0 msgr2=0x7f9e44004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 --2- 172.21.15.107:0/2671491737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e440047c0 0x7f9e44004c40 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009ac0 tx=0x7f9e440057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/2671491737 reap_dead start 2023-11-24T08:22:29.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 -- 172.21.15.107:0/2671491737 shutdown_connections 2023-11-24T08:22:29.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 -- 172.21.15.107:0/2671491737 >> 172.21.15.107:0/2671491737 conn(0x7f9e5c0b0c50 msgr2=0x7f9e5c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:29.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 -- 172.21.15.107:0/2671491737 shutdown_connections 2023-11-24T08:22:29.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.224+0000 7f9e6086b640 1 -- 172.21.15.107:0/2671491737 wait complete. 2023-11-24T08:22:29.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.226+0000 7f9e6086b640 1 Processor -- start 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.226+0000 7f9e6086b640 1 -- start start 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e440047c0 0x7f9e5c145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c0b4620 0x7f9e5c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c157ee0 con 0x7f9e5c158b40 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0c6610 con 0x7f9e5c0b4620 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e6086b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0af380 con 0x7f9e440047c0 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e5ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e5ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56812/0 (socket says 172.21.15.107:56812) 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/785110197 learned_addr learned my addr 172.21.15.107:0/785110197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e5a59c640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e440047c0 0x7f9e5c145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.227+0000 7f9e59d9b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c0b4620 0x7f9e5c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e440047c0 msgr2=0x7f9e5c145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e440047c0 0x7f9e5c145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c0b4620 msgr2=0x7f9e5c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c0b4620 0x7f9e5c0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 -- 172.21.15.107:0/785110197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0af5d0 con 0x7f9e5c158b40 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e5ad9d640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f9e54000f80 tx=0x7f9e5400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5400bb80 con 0x7f9e5c158b40 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5400bce0 con 0x7f9e5c158b40 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540072b0 con 0x7f9e5c158b40 2023-11-24T08:22:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c0c7bd0 con 0x7f9e5c158b40 2023-11-24T08:22:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.228+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c0af8b0 con 0x7f9e5c158b40 2023-11-24T08:22:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.230+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c0c7a30 con 0x7f9e5c158b40 2023-11-24T08:22:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.231+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f9e54007410 con 0x7f9e5c158b40 2023-11-24T08:22:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.231+0000 7f9e4affd640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 0x7f9e3c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.232+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e54088cb0 con 0x7f9e5c158b40 2023-11-24T08:22:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.232+0000 7f9e5a59c640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 0x7f9e3c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.232+0000 7f9e5a59c640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 0x7f9e3c06f550 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c158090 tx=0x7f9e4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.236+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e5c0c7a30 con 0x7f9e5c158b40 2023-11-24T08:22:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.424+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e5c0c80f0 con 0x7f9e5c158b40 2023-11-24T08:22:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.425+0000 7f9e4affd640 1 -- 172.21.15.107:0/785110197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v207) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e5c0c80f0 con 0x7f9e5c158b40 2023-11-24T08:22:29.427 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 msgr2=0x7f9e3c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 0x7f9e3c06f550 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c158090 tx=0x7f9e4c009040 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 msgr2=0x7f9e5c0c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f9e54000f80 tx=0x7f9e5400b4a0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 shutdown_connections 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9e3c06d090 0x7f9e3c06f550 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e5c158b40 0x7f9e5c0c83d0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c0b4620 0x7f9e5c0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 --2- 172.21.15.107:0/785110197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e440047c0 0x7f9e5c145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 >> 172.21.15.107:0/785110197 conn(0x7f9e5c0b0c50 msgr2=0x7f9e5c071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 shutdown_connections 2023-11-24T08:22:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.428+0000 7f9e6086b640 1 -- 172.21.15.107:0/785110197 wait complete. 2023-11-24T08:22:29.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:22:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.667+0000 7f163722d640 1 Processor -- start 2023-11-24T08:22:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.668+0000 7f163722d640 1 -- start start 2023-11-24T08:22:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.669+0000 7f163722d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16300cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.669+0000 7f163722d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16300cc3a0 con 0x7f16300c59b0 2023-11-24T08:22:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.669+0000 7f163722d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163015eea0 con 0x7f16300cbf20 2023-11-24T08:22:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.669+0000 7f163722d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163015b820 con 0x7f163015b440 2023-11-24T08:22:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f16357ca640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f16300c59b0 0x7f163006c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60204/0 (socket says 172.21.15.107:60204) 2023-11-24T08:22:29.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f16357ca640 1 -- 172.21.15.107:0/487674682 learned_addr learned my addr 172.21.15.107:0/487674682 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f1634fc9640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16300cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579394163 0 0) 0x7f1628001110 con 0x7f16300c59b0 2023-11-24T08:22:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618003520 con 0x7f16300c59b0 2023-11-24T08:22:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282251638 0 0) 0x7f1618003520 con 0x7f16300c59b0 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 msgr2=0x7f16300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16300cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 >> v1:172.21.15.120:6789/0 conn(0x7f16300cbf20 legacy=0x7f16300c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300abf30 con 0x7f16300c59b0 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.670+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f1628001800 con 0x7f16300c59b0 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 >> v1:172.21.15.107:6789/0 conn(0x7f16300c59b0 legacy=0x7f163006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16180039a0 0x7f1618003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163015b820 con 0x7f16180039a0 2023-11-24T08:22:29.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161800bab0 con 0x7f16180047c0 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161800bc70 con 0x7f163015b440 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16180039a0 0x7f1618003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f16357ca640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f162ffff640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 -- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 msgr2=0x7f16180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f163015b440 0x7f16180051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 -- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16180047c0 msgr2=0x7f1618004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.671+0000 7f1634fc9640 1 -- 172.21.15.107:0/487674682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618003520 con 0x7f16180039a0 2023-11-24T08:22:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.672+0000 7f1634fc9640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16180039a0 0x7f1618003dc0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f1620009fc0 tx=0x7f16180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0a66f62db4b4d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162002cc20 con 0x7f16180039a0 2023-11-24T08:22:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162000f040 con 0x7f16180039a0 2023-11-24T08:22:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f162f7fe640 1 -- 172.21.15.107:0/487674682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620034440 con 0x7f16180039a0 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f163722d640 1 -- 172.21.15.107:0/487674682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16180039a0 msgr2=0x7f1618003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f163722d640 1 --2- 172.21.15.107:0/487674682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16180039a0 0x7f1618003dc0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f1620009fc0 tx=0x7f16180057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f16357ca640 1 -- 172.21.15.107:0/487674682 reap_dead start 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f163722d640 1 -- 172.21.15.107:0/487674682 shutdown_connections 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.673+0000 7f163722d640 1 -- 172.21.15.107:0/487674682 >> 172.21.15.107:0/487674682 conn(0x7f16300b0c50 msgr2=0x7f16300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.674+0000 7f163722d640 1 -- 172.21.15.107:0/487674682 shutdown_connections 2023-11-24T08:22:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.674+0000 7f163722d640 1 -- 172.21.15.107:0/487674682 wait complete. 2023-11-24T08:22:29.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.675+0000 7f163722d640 1 Processor -- start 2023-11-24T08:22:29.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.676+0000 7f163722d640 1 -- start start 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.676+0000 7f163722d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16180039a0 0x7f16300c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f163722d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163014a3d0 0x7f16300c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f163722d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f163722d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c8610 con 0x7f1630099c60 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f163722d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c88f0 con 0x7f163014a3d0 2023-11-24T08:22:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f163722d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c8bd0 con 0x7f16180039a0 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56826/0 (socket says 172.21.15.107:56826) 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 -- 172.21.15.107:0/3333390467 learned_addr learned my addr 172.21.15.107:0/3333390467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f1634fc9640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16180039a0 0x7f16300c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 -- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16180039a0 msgr2=0x7f16300c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16180039a0 0x7f16300c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 -- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163014a3d0 msgr2=0x7f16300c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163014a3d0 0x7f16300c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.677+0000 7f16357ca640 1 -- 172.21.15.107:0/3333390467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300c8eb0 con 0x7f1630099c60 2023-11-24T08:22:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f16357ca640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f1628000f60 tx=0x7f162800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16280072d0 con 0x7f1630099c60 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628007450 con 0x7f1630099c60 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16280047e0 con 0x7f1630099c60 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16300c8210 con 0x7f1630099c60 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.678+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163009a800 con 0x7f1630099c60 2023-11-24T08:22:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.679+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163004e8b0 con 0x7f1630099c60 2023-11-24T08:22:29.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.680+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f1628004960 con 0x7f1630099c60 2023-11-24T08:22:29.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.681+0000 7f162d7fa640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 0x7f161c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:29.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.681+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1628087d80 con 0x7f1630099c60 2023-11-24T08:22:29.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.681+0000 7f1634fc9640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 0x7f161c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:29.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.682+0000 7f1634fc9640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 0x7f161c06f580 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1620036000 tx=0x7f162002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:29.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.685+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1628054930 con 0x7f1630099c60 2023-11-24T08:22:29.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.868+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16300cc3a0 con 0x7f1630099c60 2023-11-24T08:22:29.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.869+0000 7f162d7fa640 1 -- 172.21.15.107:0/3333390467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v207) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16300cc3a0 con 0x7f1630099c60 2023-11-24T08:22:29.871 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:22:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 msgr2=0x7f161c06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 0x7f161c06f580 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1620036000 tx=0x7f162002e000 comp rx=0 tx=0).stop 2023-11-24T08:22:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 msgr2=0x7f163009a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f1628000f60 tx=0x7f162800dc00 comp rx=0 tx=0).stop 2023-11-24T08:22:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 shutdown_connections 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f161c06d0c0 0x7f161c06f580 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1630099c60 0x7f163009a0e0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163014a3d0 0x7f16300c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 --2- 172.21.15.107:0/3333390467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16180039a0 0x7f16300c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 >> 172.21.15.107:0/3333390467 conn(0x7f16300b0c50 msgr2=0x7f1630071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.871+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 shutdown_connections 2023-11-24T08:22:29.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:29.872+0000 7f163722d640 1 -- 172.21.15.107:0/3333390467 wait complete. 2023-11-24T08:22:29.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:22:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.108+0000 7ffbd4389640 1 Processor -- start 2023-11-24T08:22:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.108+0000 7ffbd4389640 1 -- start start 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffbcc161060 0x7ffbcc161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbcc0b3330 0x7ffbcc15da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc0b3790 0x7ffbcc15dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc06ec10 con 0x7ffbcc0b3330 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc06dcf0 con 0x7ffbcc161060 2023-11-24T08:22:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.110+0000 7ffbd4389640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc161980 con 0x7ffbcc0b3790 2023-11-24T08:22:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbcc0b3330 0x7ffbcc15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbcc0b3330 0x7ffbcc15da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56838/0 (socket says 172.21.15.107:56838) 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 -- 172.21.15.107:0/3970825362 learned_addr learned my addr 172.21.15.107:0/3970825362 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd2125640 1 --2- 172.21.15.107:0/3970825362 >> v2:172.21.15.120:3300/0 conn(0x7ffbcc161060 0x7ffbcc161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd2926640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc0b3790 0x7ffbcc15dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 -- 172.21.15.107:0/3970825362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc0b3790 msgr2=0x7ffbcc15dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc0b3790 0x7ffbcc15dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 -- 172.21.15.107:0/3970825362 >> v2:172.21.15.120:3300/0 conn(0x7ffbcc161060 msgr2=0x7ffbcc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 --2- 172.21.15.107:0/3970825362 >> v2:172.21.15.120:3300/0 conn(0x7ffbcc161060 0x7ffbcc161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.111+0000 7ffbd1924640 1 -- 172.21.15.107:0/3970825362 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc161ec0 con 0x7ffbcc0b3330 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.112+0000 7ffbd1924640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc15da80 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0611c0 tx=0x7ffbbc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e46c85779276db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd1123640 1 -- 172.21.15.107:0/3970825362 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc004230 con 0x7ffbcc0b3330 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd1123640 1 -- 172.21.15.107:0/3970825362 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc004390 con 0x7ffbcc0b3330 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd1123640 1 -- 172.21.15.107:0/3970825362 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc003810 con 0x7ffbcc0b3330 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 -- 172.21.15.107:0/3970825362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3330 msgr2=0x7ffbcc15da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc15da80 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0611c0 tx=0x7ffbbc02b860 comp rx=0 tx=0).stop 2023-11-24T08:22:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 -- 172.21.15.107:0/3970825362 shutdown_connections 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc0b3790 0x7ffbcc15dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 --2- 172.21.15.107:0/3970825362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc15da80 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 --2- 172.21.15.107:0/3970825362 >> v2:172.21.15.120:3300/0 conn(0x7ffbcc161060 0x7ffbcc161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 -- 172.21.15.107:0/3970825362 >> 172.21.15.107:0/3970825362 conn(0x7ffbcc0cf900 msgr2=0x7ffbcc0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 -- 172.21.15.107:0/3970825362 shutdown_connections 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.113+0000 7ffbd4389640 1 -- 172.21.15.107:0/3970825362 wait complete. 2023-11-24T08:22:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.115+0000 7ffbd4389640 1 Processor -- start 2023-11-24T08:22:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.115+0000 7ffbd4389640 1 -- start start 2023-11-24T08:22:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc1e85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc161060 0x7ffbcc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc1e7510 con 0x7ffbcc0b3790 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc1e77f0 con 0x7ffbcc0b3330 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.116+0000 7ffbd4389640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc1e7ad0 con 0x7ffbcc161060 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56844/0 (socket says 172.21.15.107:56844) 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 -- 172.21.15.107:0/4224004960 learned_addr learned my addr 172.21.15.107:0/4224004960 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd2926640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc161060 0x7ffbcc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 -- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc161060 msgr2=0x7ffbcc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc161060 0x7ffbcc0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 -- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbcc0b3330 msgr2=0x7ffbcc1e85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc1e85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 -- 172.21.15.107:0/4224004960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc009460 con 0x7ffbcc0b3790 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.117+0000 7ffbd1924640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc02b8a0 tx=0x7ffbbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.118+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc003e50 con 0x7ffbcc0b3790 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.118+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc004500 con 0x7ffbcc0b3790 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.118+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc006260 con 0x7ffbcc0b3790 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.118+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc1e7db0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.118+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc1481b0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.119+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc1e8af0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.120+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ffbbc0063c0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.121+0000 7ffbc2ffd640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 0x7ffbb006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.121+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbbc0aab40 con 0x7ffbcc0b3790 2023-11-24T08:22:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.121+0000 7ffbd2125640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 0x7ffbb006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.122+0000 7ffbd2125640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 0x7ffbb006f550 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800b7f0 tx=0x7ffbc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.125+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbbc077650 con 0x7ffbcc0b3790 2023-11-24T08:22:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.317+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbcc1e82d0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.318+0000 7ffbc2ffd640 1 -- 172.21.15.107:0/4224004960 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v207) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbcc1e82d0 con 0x7ffbcc0b3790 2023-11-24T08:22:30.320 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 msgr2=0x7ffbb006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 0x7ffbb006f550 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800b7f0 tx=0x7ffbc8008040 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 msgr2=0x7ffbcc1e6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc02b8a0 tx=0x7ffbbc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 shutdown_connections 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ffbb006d090 0x7ffbb006f550 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffbcc161060 0x7ffbcc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc0b3790 0x7ffbcc1e6e10 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.320+0000 7ffbd4389640 1 --2- 172.21.15.107:0/4224004960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbcc0b3330 0x7ffbcc1e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.321+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 >> 172.21.15.107:0/4224004960 conn(0x7ffbcc0cf900 msgr2=0x7ffbcc15d850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.321+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 shutdown_connections 2023-11-24T08:22:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.321+0000 7ffbd4389640 1 -- 172.21.15.107:0/4224004960 wait complete. 2023-11-24T08:22:30.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:22:30.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.556+0000 7f6f82be5640 1 Processor -- start 2023-11-24T08:22:30.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.557+0000 7f6f82be5640 1 -- start start 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f82be5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6f7c0cfb90 0x7f6f7c161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f82be5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c161fa0 0x7f6f7c06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f82be5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c1612f0 con 0x7f6f7c0b2400 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f82be5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0cff70 con 0x7f6f7c0cfb90 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f82be5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0afb20 con 0x7f6f7c161fa0 2023-11-24T08:22:30.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.558+0000 7f6f81182640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f7c0b2400 0x7f6f7c0cc940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60218/0 (socket says 172.21.15.107:60218) 2023-11-24T08:22:30.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f81182640 1 -- 172.21.15.107:0/613672668 learned_addr learned my addr 172.21.15.107:0/613672668 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c161fa0 0x7f6f7c06ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f80981640 1 --2- 172.21.15.107:0/613672668 >> v2:172.21.15.120:3300/0 conn(0x7f6f7c0cfb90 0x7f6f7c161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234409822 0 0) 0x7f6f7c1612f0 con 0x7f6f7c0b2400 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f68003520 con 0x7f6f7c0b2400 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130886433 0 0) 0x7f6f68003520 con 0x7f6f7c0b2400 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c161fa0 msgr2=0x7f6f7c06ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c161fa0 0x7f6f7c06ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 >> v2:172.21.15.120:3300/0 conn(0x7f6f7c0cfb90 msgr2=0x7f6f7c161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 --2- 172.21.15.107:0/613672668 >> v2:172.21.15.120:3300/0 conn(0x7f6f7c0cfb90 0x7f6f7c161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c06a850 con 0x7f6f7c0b2400 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6f74001670 con 0x7f6f7c0b2400 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 >> v1:172.21.15.107:6789/0 conn(0x7f6f7c0b2400 legacy=0x7f6f7c0cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.559+0000 7f6f7b7fe640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f680039a0 0x7f6f68003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7b7fe640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f680047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7b7fe640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f68004f00 0x7f6f6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0cff70 con 0x7f6f68004300 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0afb20 con 0x7f6f68004f00 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6800bec0 con 0x7f6f680039a0 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f680047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f80981640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f680039a0 0x7f6f68003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 -- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f680039a0 msgr2=0x7f6f68003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f680039a0 0x7f6f68003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 -- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f68004f00 msgr2=0x7f6f6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:30.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f68004f00 0x7f6f6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.560+0000 7f6f7bfff640 1 -- 172.21.15.107:0/613672668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f68003520 con 0x7f6f68004300 2023-11-24T08:22:30.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.561+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f680047e0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400ba60 tx=0x7f6f680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4149a12f8cc52fcc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64032630 con 0x7f6f68004300 2023-11-24T08:22:30.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f64032790 con 0x7f6f68004300 2023-11-24T08:22:30.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f7b7fe640 1 -- 172.21.15.107:0/613672668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64032b20 con 0x7f6f68004300 2023-11-24T08:22:30.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 -- 172.21.15.107:0/613672668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 msgr2=0x7f6f680047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 --2- 172.21.15.107:0/613672668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f680047e0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400ba60 tx=0x7f6f680057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f81182640 1 -- 172.21.15.107:0/613672668 reap_dead start 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 -- 172.21.15.107:0/613672668 shutdown_connections 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 -- 172.21.15.107:0/613672668 >> 172.21.15.107:0/613672668 conn(0x7f6f7c0999d0 msgr2=0x7f6f7c0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 -- 172.21.15.107:0/613672668 shutdown_connections 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.562+0000 7f6f82be5640 1 -- 172.21.15.107:0/613672668 wait complete. 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.564+0000 7f6f82be5640 1 Processor -- start 2023-11-24T08:22:30.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.564+0000 7f6f82be5640 1 -- start start 2023-11-24T08:22:30.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7c15ddf0 0x7f6f7c0c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c15e270 0x7f6f7c0ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b3f10 con 0x7f6f68004300 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b41f0 con 0x7f6f7c15ddf0 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.565+0000 7f6f82be5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b44d0 con 0x7f6f7c15e270 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56858/0 (socket says 172.21.15.107:56858) 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 -- 172.21.15.107:0/2740929248 learned_addr learned my addr 172.21.15.107:0/2740929248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f81182640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c15e270 0x7f6f7c0ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 -- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c15e270 msgr2=0x7f6f7c0ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c15e270 0x7f6f7c0ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 -- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7c15ddf0 msgr2=0x7f6f7c0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7c15ddf0 0x7f6f7c0c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7c15ddf0 0x7f6f7c0c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 -- 172.21.15.107:0/2740929248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b47b0 con 0x7f6f68004300 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.566+0000 7f6f80981640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00f3d0 tx=0x7f6f6c00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.567+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c016620 con 0x7f6f68004300 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.567+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c0167a0 con 0x7f6f68004300 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.567+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c016b50 con 0x7f6f68004300 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.567+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c0b4a90 con 0x7f6f68004300 2023-11-24T08:22:30.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.567+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c159260 con 0x7f6f68004300 2023-11-24T08:22:30.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.568+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c04e8b0 con 0x7f6f68004300 2023-11-24T08:22:30.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.569+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f6f6c016cb0 con 0x7f6f68004300 2023-11-24T08:22:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.570+0000 7f6f797fa640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 0x7f6f5c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.570+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f6c089730 con 0x7f6f68004300 2023-11-24T08:22:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.571+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 0x7f6f5c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.571+0000 7f6f7bfff640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 0x7f6f5c06f580 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400ba30 tx=0x7f6f6403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:30.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.575+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f6c0562e0 con 0x7f6f68004300 2023-11-24T08:22:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.779+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6f7c071940 con 0x7f6f68004300 2023-11-24T08:22:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.780+0000 7f6f797fa640 1 -- 172.21.15.107:0/2740929248 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v207) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6f6c059900 con 0x7f6f68004300 2023-11-24T08:22:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:22:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 msgr2=0x7f6f5c06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 0x7f6f5c06f580 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400ba30 tx=0x7f6f6403d040 comp rx=0 tx=0).stop 2023-11-24T08:22:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 msgr2=0x7f6f7c158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00f3d0 tx=0x7f6f6c00f890 comp rx=0 tx=0).stop 2023-11-24T08:22:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 shutdown_connections 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5c06d0c0 0x7f6f5c06f580 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f7c15e270 0x7f6f7c0ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7c15ddf0 0x7f6f7c0c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 --2- 172.21.15.107:0/2740929248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f68004300 0x7f6f7c158d20 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 >> 172.21.15.107:0/2740929248 conn(0x7f6f7c0999d0 msgr2=0x7f6f7c159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 shutdown_connections 2023-11-24T08:22:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:30.782+0000 7f6f82be5640 1 -- 172.21.15.107:0/2740929248 wait complete. 2023-11-24T08:22:30.803 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:22:30.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:30.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:22:30.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:22:30.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:30.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:22:30.978 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:30.978 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:31.019 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:31.020 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:31.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:31.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.320+0000 7fe17263d640 1 Processor -- start 2023-11-24T08:22:31.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.320+0000 7fe17263d640 1 -- start start 2023-11-24T08:22:31.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.321+0000 7fe17263d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe16c0cfb60 0x7fe16c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.321+0000 7fe17263d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0ce270 con 0x7fe16c0cc470 2023-11-24T08:22:31.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.321+0000 7fe17263d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c06d0a0 con 0x7fe16c0ce640 2023-11-24T08:22:31.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.321+0000 7fe17263d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c06c560 con 0x7fe16c0cfb60 2023-11-24T08:22:31.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe163fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe16c0cfb60 0x7fe16c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe16bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe16c0cc470 0x7fe16c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60228/0 (socket says 172.21.15.107:60228) 2023-11-24T08:22:31.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe163fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe16c0cfb60 0x7fe16c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56870/0 (socket says 172.21.15.107:56870) 2023-11-24T08:22:31.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe16bfff640 1 -- 172.21.15.107:0/3932674308 learned_addr learned my addr 172.21.15.107:0/3932674308 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:31.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe163fff640 1 -- 172.21.15.107:0/3932674308 >> v1:172.21.15.120:6789/0 conn(0x7fe16c0ce640 legacy=0x7fe16c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:31.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe163fff640 1 -- 172.21.15.107:0/3932674308 >> v1:172.21.15.107:6789/0 conn(0x7fe16c0cc470 legacy=0x7fe16c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.322+0000 7fe163fff640 1 -- 172.21.15.107:0/3932674308 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c06c770 con 0x7fe16c0cfb60 2023-11-24T08:22:31.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.323+0000 7fe163fff640 1 --2- 172.21.15.107:0/3932674308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0cfb60 0x7fe16c0abfc0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fe1540097a0 tx=0x7fe15402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c5e6f890695919 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.324+0000 7fe16b7fe640 1 -- 172.21.15.107:0/3932674308 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154031830 con 0x7fe16c0cfb60 2023-11-24T08:22:31.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.324+0000 7fe16b7fe640 1 -- 172.21.15.107:0/3932674308 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe154031990 con 0x7fe16c0cfb60 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.324+0000 7fe16b7fe640 1 -- 172.21.15.107:0/3932674308 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154031d20 con 0x7fe16c0cfb60 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.324+0000 7fe17263d640 1 -- 172.21.15.107:0/3932674308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0cfb60 msgr2=0x7fe16c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.324+0000 7fe17263d640 1 --2- 172.21.15.107:0/3932674308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0cfb60 0x7fe16c0abfc0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fe1540097a0 tx=0x7fe15402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.325+0000 7fe17263d640 1 -- 172.21.15.107:0/3932674308 shutdown_connections 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.325+0000 7fe17263d640 1 --2- 172.21.15.107:0/3932674308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0cfb60 0x7fe16c0abfc0 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.325+0000 7fe17263d640 1 -- 172.21.15.107:0/3932674308 >> 172.21.15.107:0/3932674308 conn(0x7fe16c06a920 msgr2=0x7fe16c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.325+0000 7fe17263d640 1 -- 172.21.15.107:0/3932674308 shutdown_connections 2023-11-24T08:22:31.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.325+0000 7fe17263d640 1 -- 172.21.15.107:0/3932674308 wait complete. 2023-11-24T08:22:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.327+0000 7fe17263d640 1 Processor -- start 2023-11-24T08:22:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.327+0000 7fe17263d640 1 -- start start 2023-11-24T08:22:31.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 0x7fe16c162b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c0cfb60 0x7fe16c15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c15c660 con 0x7fe16c0ce640 2023-11-24T08:22:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c163050 con 0x7fe16c0cc470 2023-11-24T08:22:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe17263d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c15b9e0 con 0x7fe16c0cfb60 2023-11-24T08:22:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.328+0000 7fe163fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56878/0 (socket says 172.21.15.107:56878) 2023-11-24T08:22:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 -- 172.21.15.107:0/1062540954 learned_addr learned my addr 172.21.15.107:0/1062540954 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 -- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c0cfb60 msgr2=0x7fe16c15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe16bfff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 0x7fe16c162b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe170bda640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c0cfb60 0x7fe16c15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c0cfb60 0x7fe16c15ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 -- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 msgr2=0x7fe16c162b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 0x7fe16c162b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 -- 172.21.15.107:0/1062540954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009400 con 0x7fe16c0ce640 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe16bfff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 0x7fe16c162b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.329+0000 7fe163fff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fe154034040 tx=0x7fe1540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.330+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15403ba70 con 0x7fe16c0ce640 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.330+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15403bc10 con 0x7fe16c0ce640 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.330+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1540453f0 con 0x7fe16c0ce640 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.330+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c15bcc0 con 0x7fe16c0ce640 2023-11-24T08:22:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.330+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c15c280 con 0x7fe16c0ce640 2023-11-24T08:22:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.331+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c15c410 con 0x7fe16c0ce640 2023-11-24T08:22:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.332+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe15404e450 con 0x7fe16c0ce640 2023-11-24T08:22:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.333+0000 7fe1697fa640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 0x7fe14c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.334+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1540aa960 con 0x7fe16c0ce640 2023-11-24T08:22:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.334+0000 7fe16bfff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 0x7fe14c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.335+0000 7fe16bfff640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 0x7fe14c06f550 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe15c005880 tx=0x7fe15c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.338+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe154077630 con 0x7fe16c0ce640 2023-11-24T08:22:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.528+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe16c15c7a0 con 0x7fe16c0ce640 2023-11-24T08:22:31.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.530+0000 7fe1697fa640 1 -- 172.21.15.107:0/1062540954 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16511 (secure 0 0 0) 0x7fe16c15c7a0 con 0x7fe16c0ce640 2023-11-24T08:22:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:31.533 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":207,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:21:52.567075+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 msgr2=0x7fe14c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 0x7fe14c06f550 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe15c005880 tx=0x7fe15c009040 comp rx=0 tx=0).stop 2023-11-24T08:22:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 msgr2=0x7fe16c14d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fe154034040 tx=0x7fe1540046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 shutdown_connections 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe14c06d090 0x7fe14c06f550 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c0cfb60 0x7fe16c15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe16c0ce640 0x7fe16c14d380 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 --2- 172.21.15.107:0/1062540954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0cc470 0x7fe16c162b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 >> 172.21.15.107:0/1062540954 conn(0x7fe16c06a920 msgr2=0x7fe16c165680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 shutdown_connections 2023-11-24T08:22:31.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.533+0000 7fe17263d640 1 -- 172.21.15.107:0/1062540954 wait complete. 2023-11-24T08:22:31.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:21:48.833467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '206', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:21:50.037981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:22:31.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:22:31.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.766+0000 7ff55c4e7640 1 Processor -- start 2023-11-24T08:22:31.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.766+0000 7ff55c4e7640 1 -- start start 2023-11-24T08:22:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55c4e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff55415b440 0x7ff5540cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55c4e7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5540c59b0 0x7ff55406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55c4e7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5540c5dc0 con 0x7ff5540cbf20 2023-11-24T08:22:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55c4e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5540cc3d0 con 0x7ff55415b440 2023-11-24T08:22:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55c4e7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55415b820 con 0x7ff5540c59b0 2023-11-24T08:22:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff55415b440 0x7ff5540cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff55415b440 0x7ff5540cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56886/0 (socket says 172.21.15.107:56886) 2023-11-24T08:22:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 -- 172.21.15.107:0/2574825880 learned_addr learned my addr 172.21.15.107:0/2574825880 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 -- 172.21.15.107:0/2574825880 >> v2:172.21.15.120:3300/0 conn(0x7ff5540c59b0 msgr2=0x7ff55406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 --2- 172.21.15.107:0/2574825880 >> v2:172.21.15.120:3300/0 conn(0x7ff5540c59b0 0x7ff55406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 -- 172.21.15.107:0/2574825880 >> v1:172.21.15.120:6789/0 conn(0x7ff5540cbf20 legacy=0x7ff5540c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.768+0000 7ff55a283640 1 -- 172.21.15.107:0/2574825880 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55415eea0 con 0x7ff55415b440 2023-11-24T08:22:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.770+0000 7ff55a283640 1 --2- 172.21.15.107:0/2574825880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55415b440 0x7ff5540cb9e0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7ff5540cb910 tx=0x7ff55002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f93b37215ba189 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.770+0000 7ff559281640 1 -- 172.21.15.107:0/2574825880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5500317e0 con 0x7ff55415b440 2023-11-24T08:22:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.770+0000 7ff559281640 1 -- 172.21.15.107:0/2574825880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550031940 con 0x7ff55415b440 2023-11-24T08:22:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.770+0000 7ff559281640 1 -- 172.21.15.107:0/2574825880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550031cd0 con 0x7ff55415b440 2023-11-24T08:22:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 -- 172.21.15.107:0/2574825880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55415b440 msgr2=0x7ff5540cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/2574825880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55415b440 0x7ff5540cb9e0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7ff5540cb910 tx=0x7ff55002b880 comp rx=0 tx=0).stop 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 -- 172.21.15.107:0/2574825880 shutdown_connections 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/2574825880 >> v2:172.21.15.120:3300/0 conn(0x7ff5540c59b0 0x7ff55406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/2574825880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55415b440 0x7ff5540cb9e0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 -- 172.21.15.107:0/2574825880 >> 172.21.15.107:0/2574825880 conn(0x7ff5540b0c50 msgr2=0x7ff5540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 -- 172.21.15.107:0/2574825880 shutdown_connections 2023-11-24T08:22:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.771+0000 7ff55c4e7640 1 -- 172.21.15.107:0/2574825880 wait complete. 2023-11-24T08:22:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.773+0000 7ff55c4e7640 1 Processor -- start 2023-11-24T08:22:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.773+0000 7ff55c4e7640 1 -- start start 2023-11-24T08:22:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5540c59b0 0x7ff55406c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55415b440 0x7ff554157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540c8830 con 0x7ff5540cbf20 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540c7f90 con 0x7ff55415b440 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55c4e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540c8270 con 0x7ff5540c59b0 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56890/0 (socket says 172.21.15.107:56890) 2023-11-24T08:22:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 -- 172.21.15.107:0/3358288207 learned_addr learned my addr 172.21.15.107:0/3358288207 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 -- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5540c59b0 msgr2=0x7ff55406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5540c59b0 0x7ff55406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff559a82640 1 -- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55415b440 msgr2=0x7ff554157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.774+0000 7ff55aa84640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55415b440 0x7ff554157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff559a82640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55415b440 0x7ff554157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff559a82640 1 -- 172.21.15.107:0/3358288207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5500094d0 con 0x7ff5540cbf20 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff559a82640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c410 tx=0x7ff54400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440035f0 con 0x7ff5540cbf20 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544003750 con 0x7ff5540cbf20 2023-11-24T08:22:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544003b40 con 0x7ff5540cbf20 2023-11-24T08:22:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5540c8550 con 0x7ff5540cbf20 2023-11-24T08:22:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.775+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55406cca0 con 0x7ff5540cbf20 2023-11-24T08:22:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.777+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5540999d0 con 0x7ff5540cbf20 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.778+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff544003ca0 con 0x7ff5540cbf20 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.778+0000 7ff54affd640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 0x7ff53806f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.778+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5440879d0 con 0x7ff5540cbf20 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.779+0000 7ff55a283640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 0x7ff53806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.779+0000 7ff55a283640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 0x7ff53806f580 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff550030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.783+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5540999d0 con 0x7ff5540cbf20 2023-11-24T08:22:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.975+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff554099db0 con 0x7ff5540cbf20 2023-11-24T08:22:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.976+0000 7ff54affd640 1 -- 172.21.15.107:0/3358288207 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v207) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff554099db0 con 0x7ff5540cbf20 2023-11-24T08:22:31.978 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:22:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 msgr2=0x7ff53806f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 0x7ff53806f580 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff550030000 comp rx=0 tx=0).stop 2023-11-24T08:22:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 msgr2=0x7ff554156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c410 tx=0x7ff54400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 shutdown_connections 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff53806d0c0 0x7ff53806f580 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55415b440 0x7ff554157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5540cbf20 0x7ff554156bc0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 --2- 172.21.15.107:0/3358288207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5540c59b0 0x7ff55406c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 >> 172.21.15.107:0/3358288207 conn(0x7ff5540b0c50 msgr2=0x7ff5540cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.979+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 shutdown_connections 2023-11-24T08:22:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:31.980+0000 7ff55c4e7640 1 -- 172.21.15.107:0/3358288207 wait complete. 2023-11-24T08:22:32.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:22:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.213+0000 7fb11714e640 1 Processor -- start 2023-11-24T08:22:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.214+0000 7fb11714e640 1 -- start start 2023-11-24T08:22:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.215+0000 7fb11714e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb11015d3f0 0x7fb1100d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.216+0000 7fb11714e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb11015d7d0 0x7fb1100d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.216+0000 7fb11714e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1100d15e0 con 0x7fb1100d0e40 2023-11-24T08:22:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.216+0000 7fb11714e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1100d0280 con 0x7fb11015d7d0 2023-11-24T08:22:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.216+0000 7fb11714e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1100d1300 con 0x7fb11015d3f0 2023-11-24T08:22:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.217+0000 7fb11614c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb11015d3f0 0x7fb1100d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.217+0000 7fb11594b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb11015d7d0 0x7fb1100d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.217+0000 7fb11594b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb11015d7d0 0x7fb1100d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56906/0 (socket says 172.21.15.107:56906) 2023-11-24T08:22:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.217+0000 7fb11594b640 1 -- 172.21.15.107:0/1093574830 learned_addr learned my addr 172.21.15.107:0/1093574830 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.218+0000 7fb11594b640 1 -- 172.21.15.107:0/1093574830 >> v2:172.21.15.120:3300/0 conn(0x7fb11015d3f0 msgr2=0x7fb1100d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.218+0000 7fb11594b640 1 --2- 172.21.15.107:0/1093574830 >> v2:172.21.15.120:3300/0 conn(0x7fb11015d3f0 0x7fb1100d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.218+0000 7fb11594b640 1 -- 172.21.15.107:0/1093574830 >> v1:172.21.15.120:6789/0 conn(0x7fb1100d0e40 legacy=0x7fb1100afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.218+0000 7fb11594b640 1 -- 172.21.15.107:0/1093574830 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100cfc30 con 0x7fb11015d7d0 2023-11-24T08:22:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.220+0000 7fb11594b640 1 --2- 172.21.15.107:0/1093574830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d7d0 0x7fb1100d1d70 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fb110067c60 tx=0x7fb10c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9732e62ec0acc9f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11514a640 1 -- 172.21.15.107:0/1093574830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c030870 con 0x7fb11015d7d0 2023-11-24T08:22:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11514a640 1 -- 172.21.15.107:0/1093574830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c0309d0 con 0x7fb11015d7d0 2023-11-24T08:22:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11514a640 1 -- 172.21.15.107:0/1093574830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c030d60 con 0x7fb11015d7d0 2023-11-24T08:22:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 -- 172.21.15.107:0/1093574830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d7d0 msgr2=0x7fb1100d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 --2- 172.21.15.107:0/1093574830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d7d0 0x7fb1100d1d70 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fb110067c60 tx=0x7fb10c02b9b0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 -- 172.21.15.107:0/1093574830 shutdown_connections 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 --2- 172.21.15.107:0/1093574830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d7d0 0x7fb1100d1d70 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 --2- 172.21.15.107:0/1093574830 >> v2:172.21.15.120:3300/0 conn(0x7fb11015d3f0 0x7fb1100d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.221+0000 7fb11714e640 1 -- 172.21.15.107:0/1093574830 >> 172.21.15.107:0/1093574830 conn(0x7fb11009c5b0 msgr2=0x7fb1100d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.222+0000 7fb11714e640 1 -- 172.21.15.107:0/1093574830 shutdown_connections 2023-11-24T08:22:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.222+0000 7fb11714e640 1 -- 172.21.15.107:0/1093574830 wait complete. 2023-11-24T08:22:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.224+0000 7fb11714e640 1 Processor -- start 2023-11-24T08:22:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.224+0000 7fb11714e640 1 -- start start 2023-11-24T08:22:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1100d0e40 0x7fb1100b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb11015d7d0 0x7fb1100b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100ca3e0 con 0x7fb11015d3f0 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100ca6c0 con 0x7fb1100d0e40 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11714e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100ca9a0 con 0x7fb11015d7d0 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11594b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11594b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56916/0 (socket says 172.21.15.107:56916) 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11594b640 1 -- 172.21.15.107:0/23457751 learned_addr learned my addr 172.21.15.107:0/23457751 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11614c640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1100d0e40 0x7fb1100b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.225+0000 7fb11694d640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb11015d7d0 0x7fb1100b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 -- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb11015d7d0 msgr2=0x7fb1100b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb11015d7d0 0x7fb1100b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 -- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1100d0e40 msgr2=0x7fb1100b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1100d0e40 0x7fb1100b6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 -- 172.21.15.107:0/23457751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0094d0 con 0x7fb11015d3f0 2023-11-24T08:22:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11594b640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fb10c02b9f0 tx=0x7fb10c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c039b60 con 0x7fb11015d3f0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c039ce0 con 0x7fb11015d3f0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c043650 con 0x7fb11015d3f0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1100cac80 con 0x7fb11015d3f0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.226+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1100b6640 con 0x7fb11015d3f0 2023-11-24T08:22:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.228+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1100cb1a0 con 0x7fb11015d3f0 2023-11-24T08:22:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.229+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fb10c02f3c0 con 0x7fb11015d3f0 2023-11-24T08:22:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.230+0000 7fb106ffd640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 0x7fb0f406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.230+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb10c0aaab0 con 0x7fb11015d3f0 2023-11-24T08:22:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.230+0000 7fb11614c640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 0x7fb0f406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.231+0000 7fb11614c640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 0x7fb0f406f580 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fb100006fd0 tx=0x7fb100008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.234+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1100cb1a0 con 0x7fb11015d3f0 2023-11-24T08:22:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.425+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1100cb1a0 con 0x7fb11015d3f0 2023-11-24T08:22:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.426+0000 7fb106ffd640 1 -- 172.21.15.107:0/23457751 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v208) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1100cb1a0 con 0x7fb11015d3f0 2023-11-24T08:22:32.428 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 msgr2=0x7fb0f406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 0x7fb0f406f580 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fb100006fd0 tx=0x7fb100008040 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 msgr2=0x7fb1100b7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fb10c02b9f0 tx=0x7fb10c004730 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 shutdown_connections 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0f406d0c0 0x7fb0f406f580 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb11015d7d0 0x7fb1100b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.428+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb11015d3f0 0x7fb1100b7e10 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.429+0000 7fb11714e640 1 --2- 172.21.15.107:0/23457751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1100d0e40 0x7fb1100b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.429+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 >> 172.21.15.107:0/23457751 conn(0x7fb11009c5b0 msgr2=0x7fb1100afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:32.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.429+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 shutdown_connections 2023-11-24T08:22:32.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.429+0000 7fb11714e640 1 -- 172.21.15.107:0/23457751 wait complete. 2023-11-24T08:22:32.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:22:32.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.667+0000 7ff80b402640 1 Processor -- start 2023-11-24T08:22:32.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.667+0000 7ff80b402640 1 -- start start 2023-11-24T08:22:32.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.668+0000 7ff80b402640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff804161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.668+0000 7ff80b402640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff80406d900 0x7ff804164440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.668+0000 7ff80b402640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80415a810 con 0x7ff8040b38a0 2023-11-24T08:22:32.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.668+0000 7ff80b402640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80406eb90 con 0x7ff80406d900 2023-11-24T08:22:32.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff80b402640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80406ee70 con 0x7ff8040ae9b0 2023-11-24T08:22:32.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff80919e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8040b38a0 0x7ff8040b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60240/0 (socket says 172.21.15.107:60240) 2023-11-24T08:22:32.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff80919e640 1 -- 172.21.15.107:0/1003420731 learned_addr learned my addr 172.21.15.107:0/1003420731 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff80899d640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff804161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128839807 0 0) 0x7ff80415a810 con 0x7ff8040b38a0 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff80999f640 1 --2- 172.21.15.107:0/1003420731 >> v2:172.21.15.120:3300/0 conn(0x7ff80406d900 0x7ff804164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.669+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f0003520 con 0x7ff8040b38a0 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835962300 0 0) 0x7ff7f0003520 con 0x7ff8040b38a0 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 msgr2=0x7ff804161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff804161ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 >> v2:172.21.15.120:3300/0 conn(0x7ff80406d900 msgr2=0x7ff804164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/1003420731 >> v2:172.21.15.120:3300/0 conn(0x7ff80406d900 0x7ff804164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80406f150 con 0x7ff8040b38a0 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff800001670 con 0x7ff8040b38a0 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 >> v1:172.21.15.107:6789/0 conn(0x7ff8040b38a0 legacy=0x7ff8040b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f0003970 0x7ff7f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7f00047c0 0x7ff7f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff7f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80406eb90 con 0x7ff7f00047c0 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80406ee70 con 0x7ff7f0003970 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f000db90 con 0x7ff8040ae9b0 2023-11-24T08:22:32.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff80899d640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7f00047c0 0x7ff7f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.670+0000 7ff80999f640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff7f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80899d640 1 -- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 msgr2=0x7ff7f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80899d640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff7f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80899d640 1 -- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f0003970 msgr2=0x7ff7f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80899d640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f0003970 0x7ff7f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80899d640 1 -- 172.21.15.107:0/1003420731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0003520 con 0x7ff7f00047c0 2023-11-24T08:22:32.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.671+0000 7ff80999f640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8040ae9b0 0x7ff7f0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff80899d640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7f00047c0 0x7ff7f0004c40 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009ac0 tx=0x7ff7f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede9565b2ca74d7a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f402c6b0 con 0x7ff7f00047c0 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f402c810 con 0x7ff7f00047c0 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff7fbfff640 1 -- 172.21.15.107:0/1003420731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f402cba0 con 0x7ff7f00047c0 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff80b402640 1 -- 172.21.15.107:0/1003420731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7f00047c0 msgr2=0x7ff7f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff80b402640 1 --2- 172.21.15.107:0/1003420731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7f00047c0 0x7ff7f0004c40 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009ac0 tx=0x7ff7f00057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.672+0000 7ff80999f640 1 -- 172.21.15.107:0/1003420731 reap_dead start 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.673+0000 7ff80b402640 1 -- 172.21.15.107:0/1003420731 shutdown_connections 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.673+0000 7ff80b402640 1 -- 172.21.15.107:0/1003420731 >> 172.21.15.107:0/1003420731 conn(0x7ff80406c090 msgr2=0x7ff8040b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.673+0000 7ff80b402640 1 -- 172.21.15.107:0/1003420731 shutdown_connections 2023-11-24T08:22:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.673+0000 7ff80b402640 1 -- 172.21.15.107:0/1003420731 wait complete. 2023-11-24T08:22:32.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.674+0000 7ff80b402640 1 Processor -- start 2023-11-24T08:22:32.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 -- start start 2023-11-24T08:22:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f00047c0 0x7ff80414f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff804156c30 0x7ff80413edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80414c200 con 0x7ff804158720 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041593f0 con 0x7ff7f00047c0 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.675+0000 7ff80b402640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80413f310 con 0x7ff804156c30 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56938/0 (socket says 172.21.15.107:56938) 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 -- 172.21.15.107:0/2648320054 learned_addr learned my addr 172.21.15.107:0/2648320054 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80899d640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff804156c30 0x7ff80413edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 -- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff804156c30 msgr2=0x7ff80413edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff804156c30 0x7ff80413edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 -- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f00047c0 msgr2=0x7ff80414f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f00047c0 0x7ff80414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 -- 172.21.15.107:0/2648320054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804159690 con 0x7ff804158720 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.676+0000 7ff80999f640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7ff804071810 tx=0x7ff7fc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.677+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc017400 con 0x7ff804158720 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.677+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc011040 con 0x7ff804158720 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.677+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc017780 con 0x7ff804158720 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.677+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff804159970 con 0x7ff804158720 2023-11-24T08:22:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.677+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8041590e0 con 0x7ff804158720 2023-11-24T08:22:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.678+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80404e8b0 con 0x7ff804158720 2023-11-24T08:22:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.679+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff7fc0203b0 con 0x7ff804158720 2023-11-24T08:22:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.680+0000 7ff7f9ffb640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 0x7ff7e806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.680+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7ff7fc0889a0 con 0x7ff804158720 2023-11-24T08:22:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.680+0000 7ff80919e640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 0x7ff7e806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.681+0000 7ff80919e640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 0x7ff7e806f550 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff8000078d0 tx=0x7ff800001550 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.684+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7fc055550 con 0x7ff804158720 2023-11-24T08:22:32.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.872+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8040ae610 con 0x7ff804158720 2023-11-24T08:22:32.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.873+0000 7ff7f9ffb640 1 -- 172.21.15.107:0/2648320054 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v208) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7fc01c0a0 con 0x7ff804158720 2023-11-24T08:22:32.875 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.876+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 msgr2=0x7ff7e806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.876+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 0x7ff7e806f550 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff8000078d0 tx=0x7ff800001550 comp rx=0 tx=0).stop 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.876+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 msgr2=0x7ff804158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.876+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7ff804071810 tx=0x7ff7fc00f4a0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 shutdown_connections 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff7e806d090 0x7ff7e806f550 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff804158720 0x7ff804158ba0 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff804156c30 0x7ff80413edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 --2- 172.21.15.107:0/2648320054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f00047c0 0x7ff80414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:32.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 >> 172.21.15.107:0/2648320054 conn(0x7ff80406c090 msgr2=0x7ff8040ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:32.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 shutdown_connections 2023-11-24T08:22:32.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:32.877+0000 7ff80b402640 1 -- 172.21.15.107:0/2648320054 wait complete. 2023-11-24T08:22:32.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:22:33.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.108+0000 7f79c2dce640 1 Processor -- start 2023-11-24T08:22:33.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.108+0000 7f79c2dce640 1 -- start start 2023-11-24T08:22:33.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c2dce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 0x7f79bc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c2dce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79bc0c59b0 0x7f79bc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c2dce640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79bc0cc3a0 con 0x7f79bc0cbf20 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c2dce640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79bc15b820 con 0x7f79bc0c59b0 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c2dce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc15eea0 con 0x7f79bc15b440 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c136b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79bc0c59b0 0x7f79bc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c136b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79bc0c59b0 0x7f79bc06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56948/0 (socket says 172.21.15.107:56948) 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.110+0000 7f79c136b640 1 -- 172.21.15.107:0/2369949855 learned_addr learned my addr 172.21.15.107:0/2369949855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c0b6a640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 0x7f79bc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c136b640 1 -- 172.21.15.107:0/2369949855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 msgr2=0x7f79bc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:33.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c136b640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 0x7f79bc0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c136b640 1 -- 172.21.15.107:0/2369949855 >> v1:172.21.15.107:6789/0 conn(0x7f79bc0cbf20 legacy=0x7f79bc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c136b640 1 -- 172.21.15.107:0/2369949855 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc0abf30 con 0x7f79bc0c59b0 2023-11-24T08:22:33.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.111+0000 7f79c0b6a640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 0x7f79bc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:33.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.113+0000 7f79c136b640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc0c59b0 0x7f79bc06c5f0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009e50 tx=0x7f79b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea848feda04852 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.113+0000 7f79bb7fe640 1 -- 172.21.15.107:0/2369949855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b402ca70 con 0x7f79bc0c59b0 2023-11-24T08:22:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.113+0000 7f79bb7fe640 1 -- 172.21.15.107:0/2369949855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b402cbd0 con 0x7f79bc0c59b0 2023-11-24T08:22:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.113+0000 7f79bb7fe640 1 -- 172.21.15.107:0/2369949855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b40343f0 con 0x7f79bc0c59b0 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 -- 172.21.15.107:0/2369949855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc0c59b0 msgr2=0x7f79bc06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc0c59b0 0x7f79bc06c5f0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009e50 tx=0x7f79b402bed0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 -- 172.21.15.107:0/2369949855 shutdown_connections 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc0c59b0 0x7f79bc06c5f0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2369949855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc15b440 0x7f79bc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 -- 172.21.15.107:0/2369949855 >> 172.21.15.107:0/2369949855 conn(0x7f79bc0b0c50 msgr2=0x7f79bc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:33.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 -- 172.21.15.107:0/2369949855 shutdown_connections 2023-11-24T08:22:33.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.114+0000 7f79c2dce640 1 -- 172.21.15.107:0/2369949855 wait complete. 2023-11-24T08:22:33.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.116+0000 7f79c2dce640 1 Processor -- start 2023-11-24T08:22:33.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.116+0000 7f79c2dce640 1 -- start start 2023-11-24T08:22:33.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79bc0c59b0 0x7f79bc154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 0x7f79bc15d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc15d240 con 0x7f79bc15b440 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc154f80 con 0x7f79bc0c59b0 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c2dce640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc15c8e0 con 0x7f79bc0cbf20 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56954/0 (socket says 172.21.15.107:56954) 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 -- 172.21.15.107:0/2662268443 learned_addr learned my addr 172.21.15.107:0/2662268443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79bbfff640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 0x7f79bc15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 -- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 msgr2=0x7f79bc15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 0x7f79bc15d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c0b6a640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79bc0c59b0 0x7f79bc154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 -- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79bc0c59b0 msgr2=0x7f79bc154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.117+0000 7f79c136b640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79bc0c59b0 0x7f79bc154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79c136b640 1 -- 172.21.15.107:0/2662268443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b4009b30 con 0x7f79bc15b440 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79bbfff640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 0x7f79bc15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79c136b640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009e50 tx=0x7f79b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4009510 con 0x7f79bc15b440 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b403e450 con 0x7f79bc15b440 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b403e800 con 0x7f79bc15b440 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79bc156290 con 0x7f79bc15b440 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.118+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79bc15ce60 con 0x7f79bc15b440 2023-11-24T08:22:33.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.120+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79bc15d380 con 0x7f79bc15b440 2023-11-24T08:22:33.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.121+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f79b403e960 con 0x7f79bc15b440 2023-11-24T08:22:33.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.122+0000 7f79b97fa640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 0x7f79a006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.122+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f79b40ab600 con 0x7f79bc15b440 2023-11-24T08:22:33.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.122+0000 7f79c0b6a640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 0x7f79a006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.123+0000 7f79c0b6a640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 0x7f79a006f550 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f79bc1f4950 tx=0x7f79b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.128+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79bc15d380 con 0x7f79bc15b440 2023-11-24T08:22:33.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.333+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f79bc15d380 con 0x7f79bc15b440 2023-11-24T08:22:33.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.334+0000 7f79b97fa640 1 -- 172.21.15.107:0/2662268443 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v208) v1 ==== 80+0+0 (secure 0 0 0) 0x7f79bc15d380 con 0x7f79bc15b440 2023-11-24T08:22:33.336 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:22:33.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 msgr2=0x7f79a006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 0x7f79a006f550 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f79bc1f4950 tx=0x7f79b0008040 comp rx=0 tx=0).stop 2023-11-24T08:22:33.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 msgr2=0x7f79bc159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009e50 tx=0x7f79b4004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 shutdown_connections 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f79a006d090 0x7f79a006f550 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79bc15b440 0x7f79bc159af0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79bc0cbf20 0x7f79bc15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 --2- 172.21.15.107:0/2662268443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79bc0c59b0 0x7f79bc154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 >> 172.21.15.107:0/2662268443 conn(0x7f79bc0b0c50 msgr2=0x7f79bc0c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 shutdown_connections 2023-11-24T08:22:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.337+0000 7f79c2dce640 1 -- 172.21.15.107:0/2662268443 wait complete. 2023-11-24T08:22:33.356 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:22:33.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:33.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:22:33.472 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:22:33.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:33.472 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:22:33.524 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:33.524 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:33.585 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:22:33.585 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:33.585 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:33.644 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:22:33.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.863+0000 7ff11d8a3640 1 Processor -- start 2023-11-24T08:22:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.864+0000 7ff11d8a3640 1 -- start start 2023-11-24T08:22:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.865+0000 7ff11d8a3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff11815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.865+0000 7ff11d8a3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1180b1de0 0x7ff1180b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.865+0000 7ff11d8a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11815bf80 con 0x7ff11815c410 2023-11-24T08:22:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.865+0000 7ff11d8a3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1180b27b0 con 0x7ff1180b1de0 2023-11-24T08:22:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.865+0000 7ff11d8a3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180b1560 con 0x7ff118144b30 2023-11-24T08:22:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff117fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff11815c410 0x7ff1180b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60248/0 (socket says 172.21.15.107:60248) 2023-11-24T08:22:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff117fff640 1 -- 172.21.15.107:0/1272377951 learned_addr learned my addr 172.21.15.107:0/1272377951 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff11815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff11d0a2640 1 --2- 172.21.15.107:0/1272377951 >> v2:172.21.15.120:3300/0 conn(0x7ff1180b1de0 0x7ff1180b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649908527 0 0) 0x7ff11815bf80 con 0x7ff11815c410 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.866+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff108003520 con 0x7ff11815c410 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797403080 0 0) 0x7ff108003520 con 0x7ff11815c410 2023-11-24T08:22:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 msgr2=0x7ff11815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff11815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 >> v2:172.21.15.120:3300/0 conn(0x7ff1180b1de0 msgr2=0x7ff1180b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 --2- 172.21.15.107:0/1272377951 >> v2:172.21.15.120:3300/0 conn(0x7ff1180b1de0 0x7ff1180b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180b1280 con 0x7ff11815c410 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff104001670 con 0x7ff11815c410 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 >> v1:172.21.15.107:6789/0 conn(0x7ff11815c410 legacy=0x7ff1180b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff108003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080047c0 0x7ff108004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff108005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180b27b0 con 0x7ff108003970 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180b1560 con 0x7ff1080047c0 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10800b980 con 0x7ff118144b30 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff108003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11d0a2640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff108005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 -- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 msgr2=0x7ff108005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff118144b30 0x7ff108005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 -- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080047c0 msgr2=0x7ff108004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080047c0 0x7ff108004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.867+0000 7ff11c8a1640 1 -- 172.21.15.107:0/1272377951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108003520 con 0x7ff108003970 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff108003d90 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009ac0 tx=0x7ff1080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4870463464627994 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c02c6b0 con 0x7ff108003970 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c02c810 con 0x7ff108003970 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff1177fe640 1 -- 172.21.15.107:0/1272377951 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c02cba0 con 0x7ff108003970 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff11d8a3640 1 -- 172.21.15.107:0/1272377951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 msgr2=0x7ff108003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/1272377951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff108003d90 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009ac0 tx=0x7ff1080057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff11d0a2640 1 -- 172.21.15.107:0/1272377951 reap_dead start 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.869+0000 7ff11d8a3640 1 -- 172.21.15.107:0/1272377951 shutdown_connections 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.870+0000 7ff11d8a3640 1 -- 172.21.15.107:0/1272377951 >> 172.21.15.107:0/1272377951 conn(0x7ff1180cb830 msgr2=0x7ff1180cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.870+0000 7ff11d8a3640 1 -- 172.21.15.107:0/1272377951 shutdown_connections 2023-11-24T08:22:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.870+0000 7ff11d8a3640 1 -- 172.21.15.107:0/1272377951 wait complete. 2023-11-24T08:22:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.871+0000 7ff11d8a3640 1 Processor -- start 2023-11-24T08:22:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.872+0000 7ff11d8a3640 1 -- start start 2023-11-24T08:22:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.872+0000 7ff11d8a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.872+0000 7ff11d8a3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1180ca070 0x7ff1180ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11d8a3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11812e4a0 0x7ff11812e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11d8a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180c4940 con 0x7ff108003970 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11d8a3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180c4dd0 con 0x7ff11812e4a0 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11d8a3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180c3d50 con 0x7ff1180ca070 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56974/0 (socket says 172.21.15.107:56974) 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 -- 172.21.15.107:0/756763002 learned_addr learned my addr 172.21.15.107:0/756763002 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 -- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1180ca070 msgr2=0x7ff1180ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1180ca070 0x7ff1180ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 -- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11812e4a0 msgr2=0x7ff11812e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11812e4a0 0x7ff11812e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 -- 172.21.15.107:0/756763002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180c4010 con 0x7ff108003970 2023-11-24T08:22:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.873+0000 7ff11c8a1640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7ff10c037040 tx=0x7ff10c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.874+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c003810 con 0x7ff108003970 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.874+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c003970 con 0x7ff108003970 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.874+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c003d20 con 0x7ff108003970 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.874+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1180c42f0 con 0x7ff108003970 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.874+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1180caa10 con 0x7ff108003970 2023-11-24T08:22:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.875+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11804e840 con 0x7ff108003970 2023-11-24T08:22:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.876+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff10c034440 con 0x7ff108003970 2023-11-24T08:22:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.877+0000 7ff1157fa640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 0x7ff0fc06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.877+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7ff10c0aa640 con 0x7ff108003970 2023-11-24T08:22:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.878+0000 7ff117fff640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 0x7ff0fc06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.878+0000 7ff117fff640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 0x7ff0fc06f550 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff104000f60 tx=0x7ff1040015c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:33.882+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff10c0773c0 con 0x7ff108003970 2023-11-24T08:22:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.108+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff11806d860 con 0x7ff108003970 2023-11-24T08:22:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.110+0000 7ff1157fa640 1 -- 172.21.15.107:0/756763002 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4939 (secure 0 0 0) 0x7ff10c07fe60 con 0x7ff108003970 2023-11-24T08:22:34.112 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:34.112 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-11-24T08:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 msgr2=0x7ff0fc06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 0x7ff0fc06f550 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff104000f60 tx=0x7ff1040015c0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 msgr2=0x7ff11813f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7ff10c037040 tx=0x7ff10c004730 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 shutdown_connections 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff0fc06d090 0x7ff0fc06f550 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11812e4a0 0x7ff11812e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1180ca070 0x7ff1180ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 --2- 172.21.15.107:0/756763002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff108003970 0x7ff11813f810 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.112+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 >> 172.21.15.107:0/756763002 conn(0x7ff1180cb830 msgr2=0x7ff1180cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.113+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 shutdown_connections 2023-11-24T08:22:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.113+0000 7ff11d8a3640 1 -- 172.21.15.107:0/756763002 wait complete. 2023-11-24T08:22:34.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:34.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.348+0000 7fbf116c9640 1 Processor -- start 2023-11-24T08:22:34.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.348+0000 7fbf116c9640 1 -- start start 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.349+0000 7fbf116c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbf0c15c7f0 0x7fbf0c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.349+0000 7fbf116c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 0x7fbf0c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.349+0000 7fbf116c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0cece0 con 0x7fbf0c0cf450 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.349+0000 7fbf116c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c0cfe40 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.349+0000 7fbf116c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c0ceaf0 con 0x7fbf0c0ce6c0 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbf0c15c7f0 0x7fbf0c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbf0c15c7f0 0x7fbf0c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56980/0 (socket says 172.21.15.107:56980) 2023-11-24T08:22:34.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 -- 172.21.15.107:0/4258253886 learned_addr learned my addr 172.21.15.107:0/4258253886 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0a7fc640 1 --1- 172.21.15.107:0/4258253886 >> v1:172.21.15.107:6789/0 conn(0x7fbf0c0cf450 0x7fbf0c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60262/0 (socket says 172.21.15.107:60262) 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0b7fe640 1 --2- 172.21.15.107:0/4258253886 >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 0x7fbf0c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 -- 172.21.15.107:0/4258253886 >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 msgr2=0x7fbf0c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 --2- 172.21.15.107:0/4258253886 >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 0x7fbf0c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 -- 172.21.15.107:0/4258253886 >> v1:172.21.15.107:6789/0 conn(0x7fbf0c0cf450 legacy=0x7fbf0c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0affd640 1 -- 172.21.15.107:0/4258253886 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c06d040 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf09ffb640 1 -- 172.21.15.107:0/4258253886 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953878517 0 0) 0x7fbf0c0cece0 con 0x7fbf0c0cf450 2023-11-24T08:22:34.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.350+0000 7fbf0b7fe640 1 --2- 172.21.15.107:0/4258253886 >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 0x7fbf0c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.351+0000 7fbf0affd640 1 --2- 172.21.15.107:0/4258253886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c06db20 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0d00e0 tx=0x7fbefc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab19503f69646d8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf09ffb640 1 -- 172.21.15.107:0/4258253886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc02cb20 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf09ffb640 1 -- 172.21.15.107:0/4258253886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc02cc80 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf09ffb640 1 -- 172.21.15.107:0/4258253886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0343f0 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 -- 172.21.15.107:0/4258253886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 msgr2=0x7fbf0c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 --2- 172.21.15.107:0/4258253886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c06db20 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0d00e0 tx=0x7fbefc02bf80 comp rx=0 tx=0).stop 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 -- 172.21.15.107:0/4258253886 shutdown_connections 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 --2- 172.21.15.107:0/4258253886 >> v2:172.21.15.120:3300/0 conn(0x7fbf0c0ce6c0 0x7fbf0c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 --2- 172.21.15.107:0/4258253886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c06db20 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 -- 172.21.15.107:0/4258253886 >> 172.21.15.107:0/4258253886 conn(0x7fbf0c06a920 msgr2=0x7fbf0c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 -- 172.21.15.107:0/4258253886 shutdown_connections 2023-11-24T08:22:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.352+0000 7fbf116c9640 1 -- 172.21.15.107:0/4258253886 wait complete. 2023-11-24T08:22:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.354+0000 7fbf116c9640 1 Processor -- start 2023-11-24T08:22:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.354+0000 7fbf116c9640 1 -- start start 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf0c0ce6c0 0x7fbf0c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf0c0cf450 0x7fbf0c071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c157fa0 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c158280 con 0x7fbf0c0ce6c0 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf116c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c158560 con 0x7fbf0c0cf450 2023-11-24T08:22:34.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56986/0 (socket says 172.21.15.107:56986) 2023-11-24T08:22:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.355+0000 7fbf0b7fe640 1 -- 172.21.15.107:0/3070036471 learned_addr learned my addr 172.21.15.107:0/3070036471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0a7fc640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf0c0cf450 0x7fbf0c071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 -- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf0c0cf450 msgr2=0x7fbf0c071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf0c0cf450 0x7fbf0c071860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0affd640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf0c0ce6c0 0x7fbf0c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 -- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf0c0ce6c0 msgr2=0x7fbf0c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf0c0ce6c0 0x7fbf0c152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 -- 172.21.15.107:0/3070036471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc009b30 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf0b7fe640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c5e0 tx=0x7fbf0400caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.356+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04011800 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.357+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf040119a0 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.357+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c158840 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.357+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04011d50 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.357+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c159040 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.358+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c067470 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.359+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fbf04022350 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.360+0000 7fbf037fe640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 0x7fbeec06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.360+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fbf040888d0 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.360+0000 7fbf0affd640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 0x7fbeec06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.361+0000 7fbf0affd640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 0x7fbeec06f550 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbefc035000 tx=0x7fbefc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.364+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf040553e0 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.554+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf0c0cd730 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.555+0000 7fbf037fe640 1 -- 172.21.15.107:0/3070036471 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16622 (secure 0 0 0) 0x7fbf0401f090 con 0x7fbf0c15c7f0 2023-11-24T08:22:34.557 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:34.558 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":208,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:32.173766+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":208,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":30,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":30,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 msgr2=0x7fbeec06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 0x7fbeec06f550 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbefc035000 tx=0x7fbefc030000 comp rx=0 tx=0).stop 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 msgr2=0x7fbf0c071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c5e0 tx=0x7fbf0400caa0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 shutdown_connections 2023-11-24T08:22:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbeec06d090 0x7fbeec06f550 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf0c15c7f0 0x7fbf0c071da0 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf0c0cf450 0x7fbf0c071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 --2- 172.21.15.107:0/3070036471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf0c0ce6c0 0x7fbf0c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 >> 172.21.15.107:0/3070036471 conn(0x7fbf0c06a920 msgr2=0x7fbf0c15f750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 shutdown_connections 2023-11-24T08:22:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.558+0000 7fbf116c9640 1 -- 172.21.15.107:0/3070036471 wait complete. 2023-11-24T08:22:34.579 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:21:48.833467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '206', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:21:50.037981+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 208, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:22:34.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:22:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.783+0000 7f6696a07640 1 Processor -- start 2023-11-24T08:22:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.783+0000 7f6696a07640 1 -- start start 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f6696a07640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66900cc0f0 0x7f66900c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f6696a07640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669015b800 con 0x7f669015b420 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f6696a07640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669015eea0 con 0x7f66900c57e0 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f6696a07640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66900abf30 con 0x7f66900cc0f0 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66900cc0f0 0x7f66900c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66900cc0f0 0x7f66900c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56992/0 (socket says 172.21.15.107:56992) 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 -- 172.21.15.107:0/4039856383 learned_addr learned my addr 172.21.15.107:0/4039856383 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668ffff640 1 --1- 172.21.15.107:0/4039856383 >> v1:172.21.15.107:6789/0 conn(0x7f669015b420 0x7f66900cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60266/0 (socket says 172.21.15.107:60266) 2023-11-24T08:22:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 -- 172.21.15.107:0/4039856383 >> v1:172.21.15.120:6789/0 conn(0x7f66900c57e0 legacy=0x7f669006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 -- 172.21.15.107:0/4039856383 >> v1:172.21.15.107:6789/0 conn(0x7f669015b420 legacy=0x7f66900cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.785+0000 7f668f7fe640 1 -- 172.21.15.107:0/4039856383 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669015f0f0 con 0x7f66900cc0f0 2023-11-24T08:22:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.786+0000 7f668f7fe640 1 --2- 172.21.15.107:0/4039856383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900cc0f0 0x7f66900c52a0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f6684009870 tx=0x7f668402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd24ec3d11939814 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f668effd640 1 -- 172.21.15.107:0/4039856383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684031870 con 0x7f66900cc0f0 2023-11-24T08:22:34.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f668effd640 1 -- 172.21.15.107:0/4039856383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66840319d0 con 0x7f66900cc0f0 2023-11-24T08:22:34.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f668effd640 1 -- 172.21.15.107:0/4039856383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684031d60 con 0x7f66900cc0f0 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f6696a07640 1 -- 172.21.15.107:0/4039856383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900cc0f0 msgr2=0x7f66900c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f6696a07640 1 --2- 172.21.15.107:0/4039856383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900cc0f0 0x7f66900c52a0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f6684009870 tx=0x7f668402b910 comp rx=0 tx=0).stop 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f6696a07640 1 -- 172.21.15.107:0/4039856383 shutdown_connections 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f6696a07640 1 --2- 172.21.15.107:0/4039856383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900cc0f0 0x7f66900c52a0 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.787+0000 7f6696a07640 1 -- 172.21.15.107:0/4039856383 >> 172.21.15.107:0/4039856383 conn(0x7f66900b0c50 msgr2=0x7f66900b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.788+0000 7f6696a07640 1 -- 172.21.15.107:0/4039856383 shutdown_connections 2023-11-24T08:22:34.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.788+0000 7f6696a07640 1 -- 172.21.15.107:0/4039856383 wait complete. 2023-11-24T08:22:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.789+0000 7f6696a07640 1 Processor -- start 2023-11-24T08:22:34.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.789+0000 7f6696a07640 1 -- start start 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66900cc0f0 0x7f6690152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f669015b420 0x7f6690152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900aeb30 con 0x7f66900c57e0 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6690156210 con 0x7f669015b420 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.790+0000 7f6696a07640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66901564f0 con 0x7f66900cc0f0 2023-11-24T08:22:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57006/0 (socket says 172.21.15.107:57006) 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 -- 172.21.15.107:0/3980433906 learned_addr learned my addr 172.21.15.107:0/3980433906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 -- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66900cc0f0 msgr2=0x7f6690152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66900cc0f0 0x7f6690152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 -- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f669015b420 msgr2=0x7f6690152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f669015b420 0x7f6690152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 -- 172.21.15.107:0/3980433906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66840094d0 con 0x7f66900c57e0 2023-11-24T08:22:34.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.791+0000 7f668ffff640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f668000c3e0 tx=0x7f668000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.792+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800074e0 con 0x7f66900c57e0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.792+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680007640 con 0x7f66900c57e0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.792+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800049e0 con 0x7f66900c57e0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.792+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66901567d0 con 0x7f66900c57e0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.792+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6690155ea0 con 0x7f66900c57e0 2023-11-24T08:22:34.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.793+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66900c9b30 con 0x7f66900c57e0 2023-11-24T08:22:34.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.794+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f6680004b40 con 0x7f66900c57e0 2023-11-24T08:22:34.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.795+0000 7f668cff9640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 0x7f667c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:34.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.795+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6680087fa0 con 0x7f66900c57e0 2023-11-24T08:22:34.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.795+0000 7f668f7fe640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 0x7f667c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:34.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.795+0000 7f668f7fe640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 0x7f667c06f550 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6684036000 tx=0x7f6684030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.799+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6680054b50 con 0x7f66900c57e0 2023-11-24T08:22:34.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.990+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f669015b800 con 0x7f66900c57e0 2023-11-24T08:22:34.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.991+0000 7f668cff9640 1 -- 172.21.15.107:0/3980433906 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v208) v1 ==== 93+0+10 (secure 0 0 0) 0x7f669015b800 con 0x7f66900c57e0 2023-11-24T08:22:34.993 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.993+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 msgr2=0x7f667c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.993+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 0x7f667c06f550 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6684036000 tx=0x7f6684030000 comp rx=0 tx=0).stop 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 msgr2=0x7f669014db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f668000c3e0 tx=0x7f668000c8a0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 shutdown_connections 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f667c06d090 0x7f667c06f550 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f669015b420 0x7f6690152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66900cc0f0 0x7f6690152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 --2- 172.21.15.107:0/3980433906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66900c57e0 0x7f669014db10 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 >> 172.21.15.107:0/3980433906 conn(0x7f66900b0c50 msgr2=0x7f66900c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 shutdown_connections 2023-11-24T08:22:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:34.994+0000 7f6696a07640 1 -- 172.21.15.107:0/3980433906 wait complete. 2023-11-24T08:22:35.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:22:35.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.231+0000 7fd611f68640 1 Processor -- start 2023-11-24T08:22:35.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.232+0000 7fd611f68640 1 -- start start 2023-11-24T08:22:35.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.233+0000 7fd611f68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 0x7fd60c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.233+0000 7fd611f68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd60c0c59b0 0x7fd60c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.233+0000 7fd611f68640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0cc3a0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.233+0000 7fd611f68640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c15b820 con 0x7fd60c0c59b0 2023-11-24T08:22:35.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.233+0000 7fd611f68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c15eea0 con 0x7fd60c15b440 2023-11-24T08:22:35.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd60c0c59b0 0x7fd60c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd60c0c59b0 0x7fd60c06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57022/0 (socket says 172.21.15.107:57022) 2023-11-24T08:22:35.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 -- 172.21.15.107:0/1661470140 learned_addr learned my addr 172.21.15.107:0/1661470140 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60b7fe640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 0x7fd60c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 -- 172.21.15.107:0/1661470140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 msgr2=0x7fd60c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 0x7fd60c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 -- 172.21.15.107:0/1661470140 >> v1:172.21.15.120:6789/0 conn(0x7fd60c0cbf20 legacy=0x7fd60c0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60bfff640 1 -- 172.21.15.107:0/1661470140 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0abf30 con 0x7fd60c0c59b0 2023-11-24T08:22:35.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.234+0000 7fd60b7fe640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 0x7fd60c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:35.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.235+0000 7fd60bfff640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0c59b0 0x7fd60c06c5f0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e50 tx=0x7fd60402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d3e5b17dcaf452 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.235+0000 7fd60affd640 1 -- 172.21.15.107:0/1661470140 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60402c9f0 con 0x7fd60c0c59b0 2023-11-24T08:22:35.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.235+0000 7fd60affd640 1 -- 172.21.15.107:0/1661470140 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60402cb50 con 0x7fd60c0c59b0 2023-11-24T08:22:35.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.235+0000 7fd60affd640 1 -- 172.21.15.107:0/1661470140 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040343f0 con 0x7fd60c0c59b0 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 -- 172.21.15.107:0/1661470140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0c59b0 msgr2=0x7fd60c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0c59b0 0x7fd60c06c5f0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e50 tx=0x7fd60402be50 comp rx=0 tx=0).stop 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 -- 172.21.15.107:0/1661470140 shutdown_connections 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0c59b0 0x7fd60c06c5f0 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 --2- 172.21.15.107:0/1661470140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c15b440 0x7fd60c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 -- 172.21.15.107:0/1661470140 >> 172.21.15.107:0/1661470140 conn(0x7fd60c0b0c50 msgr2=0x7fd60c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:35.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 -- 172.21.15.107:0/1661470140 shutdown_connections 2023-11-24T08:22:35.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.236+0000 7fd611f68640 1 -- 172.21.15.107:0/1661470140 wait complete. 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.238+0000 7fd611f68640 1 Processor -- start 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.238+0000 7fd611f68640 1 -- start start 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c0c59b0 0x7fd60c156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd60c15b440 0x7fd60c153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c156670 con 0x7fd60c0cbf20 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c157740 con 0x7fd60c15b440 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd611f68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0c6420 con 0x7fd60c0c59b0 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd603fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd603fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57034/0 (socket says 172.21.15.107:57034) 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd603fff640 1 -- 172.21.15.107:0/717273738 learned_addr learned my addr 172.21.15.107:0/717273738 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.239+0000 7fd60b7fe640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c0c59b0 0x7fd60c156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 -- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c0c59b0 msgr2=0x7fd60c156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c0c59b0 0x7fd60c156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 -- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd60c15b440 msgr2=0x7fd60c153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd60c15b440 0x7fd60c153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 -- 172.21.15.107:0/717273738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604009b30 con 0x7fd60c0cbf20 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd603fff640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400c430 tx=0x7fd5f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4011790 con 0x7fd60c0cbf20 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f40118f0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4011ce0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c0c6700 con 0x7fd60c0cbf20 2023-11-24T08:22:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.240+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c155930 con 0x7fd60c0cbf20 2023-11-24T08:22:35.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.241+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c1554b0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.242+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fd5f401c460 con 0x7fd60c0cbf20 2023-11-24T08:22:35.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.243+0000 7fd608ff9640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 0x7fd5ec06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.243+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd5f4088780 con 0x7fd60c0cbf20 2023-11-24T08:22:35.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.243+0000 7fd60b7fe640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 0x7fd5ec06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.244+0000 7fd60b7fe640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 0x7fd5ec06f550 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b840 tx=0x7fd5fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.248+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd60c1554b0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.429+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd60c1554b0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.430+0000 7fd608ff9640 1 -- 172.21.15.107:0/717273738 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v209) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd60c1554b0 con 0x7fd60c0cbf20 2023-11-24T08:22:35.432 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 msgr2=0x7fd5ec06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 0x7fd5ec06f550 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b840 tx=0x7fd5fc008040 comp rx=0 tx=0).stop 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 msgr2=0x7fd60c157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400c430 tx=0x7fd5f400c8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 shutdown_connections 2023-11-24T08:22:35.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd5ec06d090 0x7fd5ec06f550 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd60c15b440 0x7fd60c153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd60c0cbf20 0x7fd60c157200 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 --2- 172.21.15.107:0/717273738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd60c0c59b0 0x7fd60c156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 >> 172.21.15.107:0/717273738 conn(0x7fd60c0b0c50 msgr2=0x7fd60c0c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 shutdown_connections 2023-11-24T08:22:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.433+0000 7fd611f68640 1 -- 172.21.15.107:0/717273738 wait complete. 2023-11-24T08:22:35.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:22:35.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.661+0000 7ff8c2e36640 1 Processor -- start 2023-11-24T08:22:35.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.662+0000 7ff8c2e36640 1 -- start start 2023-11-24T08:22:35.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.663+0000 7ff8c2e36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8bc0c59b0 0x7ff8bc166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.663+0000 7ff8c2e36640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc0c5dc0 con 0x7ff8bc15b440 2023-11-24T08:22:35.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.663+0000 7ff8c2e36640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc15eea0 con 0x7ff8bc0cc0f0 2023-11-24T08:22:35.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.663+0000 7ff8c2e36640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc15b820 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c0bd2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8bc15b440 0x7ff8bc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60278/0 (socket says 172.21.15.107:60278) 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c0bd2640 1 -- 172.21.15.107:0/2118341456 learned_addr learned my addr 172.21.15.107:0/2118341456 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c13d3640 1 --2- 172.21.15.107:0/2118341456 >> v2:172.21.15.107:3300/0 conn(0x7ff8bc0c59b0 0x7ff8bc166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c13d3640 1 -- 172.21.15.107:0/2118341456 >> v1:172.21.15.120:6789/0 conn(0x7ff8bc0cc0f0 legacy=0x7ff8bc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8bb7fe640 1 -- 172.21.15.107:0/2118341456 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401280269 0 0) 0x7ff8bc0c5dc0 con 0x7ff8bc15b440 2023-11-24T08:22:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c13d3640 1 -- 172.21.15.107:0/2118341456 >> v1:172.21.15.107:6789/0 conn(0x7ff8bc15b440 legacy=0x7ff8bc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.664+0000 7ff8c13d3640 1 -- 172.21.15.107:0/2118341456 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc06d600 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.665+0000 7ff8c13d3640 1 --2- 172.21.15.107:0/2118341456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc166230 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bff3dc214a2488 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.665+0000 7ff8bb7fe640 1 -- 172.21.15.107:0/2118341456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b402cbd0 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8bb7fe640 1 -- 172.21.15.107:0/2118341456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b402cd30 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8bb7fe640 1 -- 172.21.15.107:0/2118341456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b402cea0 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 -- 172.21.15.107:0/2118341456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc0c59b0 msgr2=0x7ff8bc166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/2118341456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc166230 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402be20 comp rx=0 tx=0).stop 2023-11-24T08:22:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 -- 172.21.15.107:0/2118341456 shutdown_connections 2023-11-24T08:22:35.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/2118341456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc166230 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 -- 172.21.15.107:0/2118341456 >> 172.21.15.107:0/2118341456 conn(0x7ff8bc0b0c50 msgr2=0x7ff8bc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:35.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 -- 172.21.15.107:0/2118341456 shutdown_connections 2023-11-24T08:22:35.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.666+0000 7ff8c2e36640 1 -- 172.21.15.107:0/2118341456 wait complete. 2023-11-24T08:22:35.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.668+0000 7ff8c2e36640 1 Processor -- start 2023-11-24T08:22:35.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.668+0000 7ff8c2e36640 1 -- start start 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc0b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 0x7ff8bc0b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc14c740 con 0x7ff8bc15b440 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0b4770 con 0x7ff8bc0c59b0 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c2e36640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc14c420 con 0x7ff8bc0cc0f0 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c13d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c13d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57040/0 (socket says 172.21.15.107:57040) 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.669+0000 7ff8c13d3640 1 -- 172.21.15.107:0/98560849 learned_addr learned my addr 172.21.15.107:0/98560849 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c0bd2640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc0b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 -- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 msgr2=0x7ff8bc0b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8bbfff640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 0x7ff8bc0b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 0x7ff8bc0b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 -- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 msgr2=0x7ff8bc0b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc0b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 -- 172.21.15.107:0/98560849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4009a70 con 0x7ff8bc15b440 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c0bd2640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8bbfff640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 0x7ff8bc0b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:35.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c13d3640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b402cea0 con 0x7ff8bc15b440 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b403e630 con 0x7ff8bc15b440 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc0b2250 con 0x7ff8bc15b440 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.670+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b403e9e0 con 0x7ff8bc15b440 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.671+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc0b2760 con 0x7ff8bc15b440 2023-11-24T08:22:35.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.672+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc14c880 con 0x7ff8bc15b440 2023-11-24T08:22:35.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.673+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff8b403b050 con 0x7ff8bc15b440 2023-11-24T08:22:35.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.673+0000 7ff8b97fa640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 0x7ff89c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:35.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.674+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff8b40ab030 con 0x7ff8bc15b440 2023-11-24T08:22:35.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.674+0000 7ff8c0bd2640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 0x7ff89c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:35.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.674+0000 7ff8c0bd2640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 0x7ff89c06f550 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000f60 tx=0x7ff8ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:35.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.678+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8bc14c880 con 0x7ff8bc15b440 2023-11-24T08:22:35.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.853+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8bc14c880 con 0x7ff8bc15b440 2023-11-24T08:22:35.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.854+0000 7ff8b97fa640 1 -- 172.21.15.107:0/98560849 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v209) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8b404e020 con 0x7ff8bc15b440 2023-11-24T08:22:35.855 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:22:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 msgr2=0x7ff89c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 0x7ff89c06f550 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000f60 tx=0x7ff8ac009040 comp rx=0 tx=0).stop 2023-11-24T08:22:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 msgr2=0x7ff8bc0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b4004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 shutdown_connections 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff89c06d090 0x7ff89c06f550 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8bc15b440 0x7ff8bc0ac6a0 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8bc0cc0f0 0x7ff8bc0b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 --2- 172.21.15.107:0/98560849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8bc0c59b0 0x7ff8bc0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.856+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 >> 172.21.15.107:0/98560849 conn(0x7ff8bc0b0c50 msgr2=0x7ff8bc0cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.857+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 shutdown_connections 2023-11-24T08:22:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:35.857+0000 7ff8c2e36640 1 -- 172.21.15.107:0/98560849 wait complete. 2023-11-24T08:22:35.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.081+0000 7fc01ad4e640 1 Processor -- start 2023-11-24T08:22:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.081+0000 7fc01ad4e640 1 -- start start 2023-11-24T08:22:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.082+0000 7fc01ad4e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc0140b24f0 0x7fc01415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.082+0000 7fc01ad4e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01415bf80 con 0x7fc014144b30 2023-11-24T08:22:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.082+0000 7fc01ad4e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140b1560 con 0x7fc01415c410 2023-11-24T08:22:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.082+0000 7fc01ad4e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0140b1280 con 0x7fc0140b24f0 2023-11-24T08:22:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.083+0000 7fc019d4c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc014144b30 0x7fc0140b0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60294/0 (socket says 172.21.15.107:60294) 2023-11-24T08:22:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.083+0000 7fc019d4c640 1 -- 172.21.15.107:0/2228521331 learned_addr learned my addr 172.21.15.107:0/2228521331 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.083+0000 7fc01a54d640 1 --2- 172.21.15.107:0/2228521331 >> v2:172.21.15.120:3300/0 conn(0x7fc0140b24f0 0x7fc01415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.083+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766832655 0 0) 0x7fc01415bf80 con 0x7fc014144b30 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000003520 con 0x7fc014144b30 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217023153 0 0) 0x7fc000003520 con 0x7fc014144b30 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 >> v2:172.21.15.120:3300/0 conn(0x7fc0140b24f0 msgr2=0x7fc01415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 --2- 172.21.15.107:0/2228521331 >> v2:172.21.15.120:3300/0 conn(0x7fc0140b24f0 0x7fc01415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 >> v1:172.21.15.120:6789/0 conn(0x7fc01415c410 legacy=0x7fc0140b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01415b930 con 0x7fc014144b30 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc010001670 con 0x7fc014144b30 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 >> v1:172.21.15.107:6789/0 conn(0x7fc014144b30 legacy=0x7fc0140b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc000005180 0x7fc00000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140b1280 con 0x7fc0000047c0 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00000c020 con 0x7fc000005180 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00000c1e0 con 0x7fc000003970 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.084+0000 7fc01954b640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01a54d640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc000005180 0x7fc00000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01954b640 1 -- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc000003970 msgr2=0x7fc000003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01954b640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01954b640 1 -- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc000005180 msgr2=0x7fc00000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01954b640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc000005180 0x7fc00000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01954b640 1 -- 172.21.15.107:0/2228521331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000003520 con 0x7fc0000047c0 2023-11-24T08:22:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.085+0000 7fc01a54d640 1 -- 172.21.15.107:0/2228521331 reap_dead start 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.086+0000 7fc01954b640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7fc008001750 tx=0x7fc0000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fad16a6baa2709b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.086+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0080052d0 con 0x7fc0000047c0 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.086+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00802e050 con 0x7fc0000047c0 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.086+0000 7fc018d4a640 1 -- 172.21.15.107:0/2228521331 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008003830 con 0x7fc0000047c0 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 msgr2=0x7fc000004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7fc008001750 tx=0x7fc0000057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2228521331 shutdown_connections 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2228521331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2228521331 >> 172.21.15.107:0/2228521331 conn(0x7fc0140cb830 msgr2=0x7fc0140cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2228521331 shutdown_connections 2023-11-24T08:22:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.087+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2228521331 wait complete. 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.089+0000 7fc01ad4e640 1 Processor -- start 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.089+0000 7fc01ad4e640 1 -- start start 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0000047c0 0x7fc01415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc01413a950 0x7fc01413ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc014142e40 con 0x7fc0141f3fa0 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01413b320 con 0x7fc0000047c0 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01ad4e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0141421c0 con 0x7fc01413a950 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57072/0 (socket says 172.21.15.107:57072) 2023-11-24T08:22:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 -- 172.21.15.107:0/2280505934 learned_addr learned my addr 172.21.15.107:0/2280505934 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc019d4c640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0000047c0 0x7fc01415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 -- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc01413a950 msgr2=0x7fc01413ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc01413a950 0x7fc01413ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 -- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0000047c0 msgr2=0x7fc01415a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0000047c0 0x7fc01415a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.090+0000 7fc01954b640 1 -- 172.21.15.107:0/2280505934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0141424a0 con 0x7fc0141f3fa0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc01954b640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fc008034040 tx=0x7fc008001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008007640 con 0x7fc0141f3fa0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0080069e0 con 0x7fc0141f3fa0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008006d70 con 0x7fc0141f3fa0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc014142780 con 0x7fc0141f3fa0 2023-11-24T08:22:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.091+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01415a710 con 0x7fc0141f3fa0 2023-11-24T08:22:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.092+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc014067990 con 0x7fc0141f3fa0 2023-11-24T08:22:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.094+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fc0080035f0 con 0x7fc0141f3fa0 2023-11-24T08:22:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.094+0000 7fc0067fc640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 0x7fbff806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.094+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc0080aa790 con 0x7fc0141f3fa0 2023-11-24T08:22:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.095+0000 7fc019d4c640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 0x7fbff806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.095+0000 7fc019d4c640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 0x7fbff806f550 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc010001eb0 tx=0x7fc010009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.099+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc008077340 con 0x7fc0141f3fa0 2023-11-24T08:22:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.313+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc014142e40 con 0x7fc0141f3fa0 2023-11-24T08:22:36.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.315+0000 7fc0067fc640 1 -- 172.21.15.107:0/2280505934 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4939 (secure 0 0 0) 0x7fc00804b090 con 0x7fc0141f3fa0 2023-11-24T08:22:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:36.317 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:22:36.317 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-11-24T08:22:36.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 msgr2=0x7fbff806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 0x7fbff806f550 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc010001eb0 tx=0x7fc010009040 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 msgr2=0x7fc014141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fc008034040 tx=0x7fc008001120 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 shutdown_connections 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbff806d090 0x7fbff806f550 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc01413a950 0x7fc01413ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0141f3fa0 0x7fc014141870 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 --2- 172.21.15.107:0/2280505934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0000047c0 0x7fc01415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.317+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 >> 172.21.15.107:0/2280505934 conn(0x7fc0140cb830 msgr2=0x7fc01406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:36.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.318+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 shutdown_connections 2023-11-24T08:22:36.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.318+0000 7fc01ad4e640 1 -- 172.21.15.107:0/2280505934 wait complete. 2023-11-24T08:22:36.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:36.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.531+0000 7f4142e5c640 1 Processor -- start 2023-11-24T08:22:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.531+0000 7f4142e5c640 1 -- start start 2023-11-24T08:22:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f4142e5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f4142e5c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413c0cdfb0 0x7f413c0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f4142e5c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c0cece0 con 0x7f413c15c7f0 2023-11-24T08:22:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f4142e5c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c156010 con 0x7f413c0cdfb0 2023-11-24T08:22:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f4142e5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c06ce30 con 0x7f413c0cf450 2023-11-24T08:22:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f41413f9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413c0cdfb0 0x7f413c0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f41413f9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f413c0cdfb0 0x7f413c0ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57076/0 (socket says 172.21.15.107:57076) 2023-11-24T08:22:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.533+0000 7f41413f9640 1 -- 172.21.15.107:0/3522256091 learned_addr learned my addr 172.21.15.107:0/3522256091 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.534+0000 7f413bfff640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.534+0000 7f41413f9640 1 -- 172.21.15.107:0/3522256091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 msgr2=0x7f413c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.534+0000 7f41413f9640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.534+0000 7f41413f9640 1 -- 172.21.15.107:0/3522256091 >> v1:172.21.15.120:6789/0 conn(0x7f413c15c7f0 legacy=0x7f413c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.534+0000 7f41413f9640 1 -- 172.21.15.107:0/3522256091 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c06d040 con 0x7f413c0cdfb0 2023-11-24T08:22:36.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f41413f9640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c0ce440 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f4134009e50 tx=0x7f413402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc35a7a65c4f77e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f413b7fe640 1 -- 172.21.15.107:0/3522256091 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413402ca70 con 0x7f413c0cdfb0 2023-11-24T08:22:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f413b7fe640 1 -- 172.21.15.107:0/3522256091 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413402cbd0 con 0x7f413c0cdfb0 2023-11-24T08:22:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f413b7fe640 1 -- 172.21.15.107:0/3522256091 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41340343f0 con 0x7f413c0cdfb0 2023-11-24T08:22:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f4142e5c640 1 -- 172.21.15.107:0/3522256091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 msgr2=0x7f413c0ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.535+0000 7f4142e5c640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c0ce440 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f4134009e50 tx=0x7f413402bed0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 -- 172.21.15.107:0/3522256091 shutdown_connections 2023-11-24T08:22:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c0ce440 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 --2- 172.21.15.107:0/3522256091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 -- 172.21.15.107:0/3522256091 >> 172.21.15.107:0/3522256091 conn(0x7f413c06a920 msgr2=0x7f413c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 -- 172.21.15.107:0/3522256091 shutdown_connections 2023-11-24T08:22:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.536+0000 7f4142e5c640 1 -- 172.21.15.107:0/3522256091 wait complete. 2023-11-24T08:22:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.537+0000 7f4142e5c640 1 Processor -- start 2023-11-24T08:22:36.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.537+0000 7f4142e5c640 1 -- start start 2023-11-24T08:22:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c15c7f0 0x7f413c149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c157770 con 0x7f413c0cdfb0 2023-11-24T08:22:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c156bf0 con 0x7f413c15c7f0 2023-11-24T08:22:36.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.538+0000 7f4142e5c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c156ed0 con 0x7f413c0cf450 2023-11-24T08:22:36.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57092/0 (socket says 172.21.15.107:57092) 2023-11-24T08:22:36.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 -- 172.21.15.107:0/1573211618 learned_addr learned my addr 172.21.15.107:0/1573211618 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:36.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 -- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 msgr2=0x7f413c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 -- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c15c7f0 msgr2=0x7f413c149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c15c7f0 0x7f413c149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 -- 172.21.15.107:0/1573211618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4134009b30 con 0x7f413c0cdfb0 2023-11-24T08:22:36.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4140bf8640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f413000b3f0 tx=0x7f413000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130011830 con 0x7f413c0cdfb0 2023-11-24T08:22:36.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130011990 con 0x7f413c0cdfb0 2023-11-24T08:22:36.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130011d40 con 0x7f413c0cdfb0 2023-11-24T08:22:36.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.539+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c1571b0 con 0x7f413c0cdfb0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.540+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c0adf70 con 0x7f413c0cdfb0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.541+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c067470 con 0x7f413c0cdfb0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.541+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f413001a400 con 0x7f413c0cdfb0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.542+0000 7f41397fa640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 0x7f412006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.542+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f41300885b0 con 0x7f413c0cdfb0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.543+0000 7f413bfff640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 0x7f412006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.543+0000 7f413bfff640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 0x7f412006f550 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f413c1557e0 tx=0x7f4128008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.547+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4130055160 con 0x7f413c0cdfb0 2023-11-24T08:22:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.742+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f413c0ae630 con 0x7f413c0cdfb0 2023-11-24T08:22:36.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.745+0000 7f41397fa640 1 -- 172.21.15.107:0/1573211618 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f4130015090 con 0x7f413c0cdfb0 2023-11-24T08:22:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:36.748 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":209,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:35.135447+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":208,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":30,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 msgr2=0x7f412006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 0x7f412006f550 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f413c1557e0 tx=0x7f4128008040 comp rx=0 tx=0).stop 2023-11-24T08:22:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 msgr2=0x7f413c15df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f413000b3f0 tx=0x7f413000b8b0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 shutdown_connections 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f412006d090 0x7f412006f550 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c15c7f0 0x7f413c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cf450 0x7f413c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 --2- 172.21.15.107:0/1573211618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f413c0cdfb0 0x7f413c15df10 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 >> 172.21.15.107:0/1573211618 conn(0x7f413c06a920 msgr2=0x7f413c06d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 shutdown_connections 2023-11-24T08:22:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.748+0000 7f4142e5c640 1 -- 172.21.15.107:0/1573211618 wait complete. 2023-11-24T08:22:36.768 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:22:36.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.980+0000 7fbe3801e640 1 Processor -- start 2023-11-24T08:22:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.980+0000 7fbe3801e640 1 -- start start 2023-11-24T08:22:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.981+0000 7fbe3801e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe3015c7f0 0x7fbe3006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.981+0000 7fbe3801e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 0x7fbe300cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.981+0000 7fbe3801e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300cece0 con 0x7fbe300cdfb0 2023-11-24T08:22:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.981+0000 7fbe3801e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe30156010 con 0x7fbe3015c7f0 2023-11-24T08:22:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.981+0000 7fbe3801e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3006ce30 con 0x7fbe300cf450 2023-11-24T08:22:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe365bb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbe300cdfb0 0x7fbe300ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60302/0 (socket says 172.21.15.107:60302) 2023-11-24T08:22:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe365bb640 1 -- 172.21.15.107:0/3624911840 learned_addr learned my addr 172.21.15.107:0/3624911840 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe35dba640 1 --2- 172.21.15.107:0/3624911840 >> v2:172.21.15.120:3300/0 conn(0x7fbe3015c7f0 0x7fbe3006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308646876 0 0) 0x7fbe300cece0 con 0x7fbe300cdfb0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c003520 con 0x7fbe300cdfb0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe355b9640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 0x7fbe300cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384305480 0 0) 0x7fbe1c003520 con 0x7fbe300cdfb0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 msgr2=0x7fbe300cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 0x7fbe300cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 >> v2:172.21.15.120:3300/0 conn(0x7fbe3015c7f0 msgr2=0x7fbe3006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 --2- 172.21.15.107:0/3624911840 >> v2:172.21.15.120:3300/0 conn(0x7fbe3015c7f0 0x7fbe3006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3006d040 con 0x7fbe300cdfb0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbe280019f0 con 0x7fbe300cdfb0 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.982+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 >> v1:172.21.15.107:6789/0 conn(0x7fbe300cdfb0 legacy=0x7fbe300ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe1c003970 0x7fbe1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 0x7fbe1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30156010 con 0x7fbe1c003970 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3006ce30 con 0x7fbe1c004d00 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c00b9b0 con 0x7fbe300cf450 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe1c003970 0x7fbe1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 -- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 msgr2=0x7fbe1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe300cf450 0x7fbe1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 -- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe1c004d00 msgr2=0x7fbe1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.983+0000 7fbe35dba640 1 -- 172.21.15.107:0/3624911840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c003520 con 0x7fbe1c003970 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.984+0000 7fbe35dba640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe1c003970 0x7fbe1c003d90 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ac0 tx=0x7fbe1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd43d5338bb85e6d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.984+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c02c710 con 0x7fbe1c003970 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.984+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c02c870 con 0x7fbe1c003970 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.984+0000 7fbe34db8640 1 -- 172.21.15.107:0/3624911840 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c02cc00 con 0x7fbe1c003970 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 -- 172.21.15.107:0/3624911840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe1c003970 msgr2=0x7fbe1c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 --2- 172.21.15.107:0/3624911840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe1c003970 0x7fbe1c003d90 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ac0 tx=0x7fbe1c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe365bb640 1 -- 172.21.15.107:0/3624911840 reap_dead start 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 -- 172.21.15.107:0/3624911840 shutdown_connections 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 -- 172.21.15.107:0/3624911840 >> 172.21.15.107:0/3624911840 conn(0x7fbe3006a920 msgr2=0x7fbe3006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 -- 172.21.15.107:0/3624911840 shutdown_connections 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.985+0000 7fbe3801e640 1 -- 172.21.15.107:0/3624911840 wait complete. 2023-11-24T08:22:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.987+0000 7fbe3801e640 1 Processor -- start 2023-11-24T08:22:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.987+0000 7fbe3801e640 1 -- start start 2023-11-24T08:22:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe1c003970 0x7fbe30149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe30157020 0x7fbe30149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30156c00 con 0x7fbe30157480 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300ae440 con 0x7fbe30157020 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe3801e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30156260 con 0x7fbe1c003970 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe365bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe365bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57106/0 (socket says 172.21.15.107:57106) 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.988+0000 7fbe365bb640 1 -- 172.21.15.107:0/1652208317 learned_addr learned my addr 172.21.15.107:0/1652208317 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 -- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe1c003970 msgr2=0x7fbe30149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe1c003970 0x7fbe30149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 -- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe30157020 msgr2=0x7fbe30149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe30157020 0x7fbe30149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 -- 172.21.15.107:0/1652208317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30156540 con 0x7fbe30157480 2023-11-24T08:22:36.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe365bb640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b5270 tx=0x7fbe2800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800a610 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2800a770 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe30156820 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800ab20 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.989+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3015a720 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.991+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30156d40 con 0x7fbe30157480 2023-11-24T08:22:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.991+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fbe2800ac80 con 0x7fbe30157480 2023-11-24T08:22:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.993+0000 7fbe267fc640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 0x7fbe1406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.993+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbe28087d70 con 0x7fbe30157480 2023-11-24T08:22:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.993+0000 7fbe35dba640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 0x7fbe1406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.994+0000 7fbe35dba640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 0x7fbe1406f550 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ac0 tx=0x7fbe2c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:36.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:36.997+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe30156d40 con 0x7fbe30157480 2023-11-24T08:22:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.228+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe30156d40 con 0x7fbe30157480 2023-11-24T08:22:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.229+0000 7fbe267fc640 1 -- 172.21.15.107:0/1652208317 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4939 (secure 0 0 0) 0x7fbe30156d40 con 0x7fbe30157480 2023-11-24T08:22:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:37.233 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:22:37.234 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-11-24T08:22:37.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 msgr2=0x7fbe1406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 0x7fbe1406f550 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ac0 tx=0x7fbe2c02e000 comp rx=0 tx=0).stop 2023-11-24T08:22:37.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 msgr2=0x7fbe3014a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b5270 tx=0x7fbe2800ccd0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 shutdown_connections 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe1406d090 0x7fbe1406f550 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe30157480 0x7fbe3014a500 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe30157020 0x7fbe30149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 --2- 172.21.15.107:0/1652208317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe1c003970 0x7fbe30149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 >> 172.21.15.107:0/1652208317 conn(0x7fbe3006a920 msgr2=0x7fbe300734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 shutdown_connections 2023-11-24T08:22:37.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.232+0000 7fbe3801e640 1 -- 172.21.15.107:0/1652208317 wait complete. 2023-11-24T08:22:37.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.454+0000 7f023f036640 1 Processor -- start 2023-11-24T08:22:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.454+0000 7f023f036640 1 -- start start 2023-11-24T08:22:37.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023f036640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02380cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023f036640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02380cece0 con 0x7f02380cdfb0 2023-11-24T08:22:37.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023f036640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023806d040 con 0x7f023815c7f0 2023-11-24T08:22:37.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023f036640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0238156010 con 0x7f02380cf450 2023-11-24T08:22:37.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023d5d3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02380cdfb0 0x7f02380ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60308/0 (socket says 172.21.15.107:60308) 2023-11-24T08:22:37.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.456+0000 7f023d5d3640 1 -- 172.21.15.107:0/3738306486 learned_addr learned my addr 172.21.15.107:0/3738306486 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:37.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02380cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726261290 0 0) 0x7f02380cece0 con 0x7f02380cdfb0 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220003520 con 0x7f02380cdfb0 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064702410 0 0) 0x7f0220003520 con 0x7f02380cdfb0 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 msgr2=0x7f02380cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02380cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 >> v1:172.21.15.120:6789/0 conn(0x7f023815c7f0 legacy=0x7f023806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023806d320 con 0x7f02380cdfb0 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f02300019f0 con 0x7f02380cdfb0 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02380cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 >> v1:172.21.15.107:6789/0 conn(0x7f02380cdfb0 legacy=0x7f02380ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0220004d00 0x7f0220005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238156010 con 0x7f0220003f90 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022000bae0 con 0x7f0220004d00 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.457+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022000bca0 con 0x7f02380cf450 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f023cdd2640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f023d5d3640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0220004d00 0x7f0220005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 -- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 msgr2=0x7f02200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02380cf450 0x7f02200039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 -- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0220004d00 msgr2=0x7f0220005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0220004d00 0x7f0220005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f022ffff640 1 -- 172.21.15.107:0/3738306486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220003520 con 0x7f0220003f90 2023-11-24T08:22:37.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.458+0000 7f023d5d3640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0220004d00 0x7f0220005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.459+0000 7f022ffff640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02200047c0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f02280098a0 tx=0x7f02200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07b8bd71c49298c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.459+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228004230 con 0x7f0220003f90 2023-11-24T08:22:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.459+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0228004390 con 0x7f0220003f90 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.459+0000 7f022f7fe640 1 -- 172.21.15.107:0/3738306486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228003830 con 0x7f0220003f90 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 -- 172.21.15.107:0/3738306486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 msgr2=0x7f02200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 --2- 172.21.15.107:0/3738306486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02200047c0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f02280098a0 tx=0x7f02200057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023d5d3640 1 -- 172.21.15.107:0/3738306486 reap_dead start 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 -- 172.21.15.107:0/3738306486 shutdown_connections 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 -- 172.21.15.107:0/3738306486 >> 172.21.15.107:0/3738306486 conn(0x7f023806a920 msgr2=0x7f023806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 -- 172.21.15.107:0/3738306486 shutdown_connections 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.460+0000 7f023f036640 1 -- 172.21.15.107:0/3738306486 wait complete. 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.462+0000 7f023f036640 1 Processor -- start 2023-11-24T08:22:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.462+0000 7f023f036640 1 -- start start 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0238098920 0x7f02381505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0238150b10 0x7f02380abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02380ad890 con 0x7f0220003f90 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238150f90 con 0x7f0238150b10 2023-11-24T08:22:37.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023f036640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380acc10 con 0x7f0238098920 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023cdd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023cdd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57126/0 (socket says 172.21.15.107:57126) 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.463+0000 7f023cdd2640 1 -- 172.21.15.107:0/1708089564 learned_addr learned my addr 172.21.15.107:0/1708089564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 -- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0238098920 msgr2=0x7f02381505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0238098920 0x7f02381505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 -- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0238150b10 msgr2=0x7f02380abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0238150b10 0x7f02380abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 -- 172.21.15.107:0/1708089564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02380acef0 con 0x7f0220003f90 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023cdd2640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f023400e490 tx=0x7f023400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02340052e0 con 0x7f0220003f90 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0234005440 con 0x7f0220003f90 2023-11-24T08:22:37.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234003830 con 0x7f0220003f90 2023-11-24T08:22:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.464+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02380ad1d0 con 0x7f0220003f90 2023-11-24T08:22:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.465+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0238098d80 con 0x7f0220003f90 2023-11-24T08:22:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.466+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0238067470 con 0x7f0220003f90 2023-11-24T08:22:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.467+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f0234003990 con 0x7f0220003f90 2023-11-24T08:22:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.468+0000 7f022d7fa640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 0x7f022406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.468+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0234088550 con 0x7f0220003f90 2023-11-24T08:22:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.468+0000 7f022ffff640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 0x7f022406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.469+0000 7f022ffff640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 0x7f022406f550 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0228035000 tx=0x7f0228030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.472+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0234055130 con 0x7f0220003f90 2023-11-24T08:22:37.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.699+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02380cd730 con 0x7f0220003f90 2023-11-24T08:22:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.699+0000 7f022d7fa640 1 -- 172.21.15.107:0/1708089564 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f0234058750 con 0x7f0220003f90 2023-11-24T08:22:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:37.702 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:21:51.017422+0000","modified":"2023-11-24T08:21:54.073750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11649":{"gid":11649,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/2018281423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2018281423},{"type":"v1","addr":"172.21.15.107:6837","nonce":2018281423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-11-24T08:22:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-11-24T08:22:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 msgr2=0x7f022406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 0x7f022406f550 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0228035000 tx=0x7f0228030000 comp rx=0 tx=0).stop 2023-11-24T08:22:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 msgr2=0x7f02380adcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f023400e490 tx=0x7f023400e950 comp rx=0 tx=0).stop 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 shutdown_connections 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f022406d090 0x7f022406f550 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0238150b10 0x7f02380abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0238098920 0x7f02381505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 --2- 172.21.15.107:0/1708089564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0220003f90 0x7f02380adcb0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 >> 172.21.15.107:0/1708089564 conn(0x7f023806a920 msgr2=0x7f02381555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 shutdown_connections 2023-11-24T08:22:37.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.702+0000 7f023f036640 1 -- 172.21.15.107:0/1708089564 wait complete. 2023-11-24T08:22:37.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.931+0000 7f7f65f27640 1 Processor -- start 2023-11-24T08:22:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.931+0000 7f7f65f27640 1 -- start start 2023-11-24T08:22:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65f27640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7f6015c410 0x7f7f6015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65f27640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f600b1de0 0x7f7f600b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65f27640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6015bf80 con 0x7f7f60144b30 2023-11-24T08:22:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65f27640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600b27b0 con 0x7f7f600b1de0 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65f27640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600b1560 con 0x7f7f6015c410 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65726640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f600b1de0 0x7f7f600b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65726640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f600b1de0 0x7f7f600b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49090/0 (socket says 172.21.15.107:49090) 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.933+0000 7f7f65726640 1 -- 172.21.15.107:0/1016025698 learned_addr learned my addr 172.21.15.107:0/1016025698 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.934+0000 7f7f65726640 1 -- 172.21.15.107:0/1016025698 >> v2:172.21.15.120:3300/0 conn(0x7f7f6015c410 msgr2=0x7f7f6015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.934+0000 7f7f65726640 1 --2- 172.21.15.107:0/1016025698 >> v2:172.21.15.120:3300/0 conn(0x7f7f6015c410 0x7f7f6015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.934+0000 7f7f65726640 1 -- 172.21.15.107:0/1016025698 >> v1:172.21.15.120:6789/0 conn(0x7f7f60144b30 legacy=0x7f7f600b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.934+0000 7f7f65726640 1 -- 172.21.15.107:0/1016025698 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f600b1280 con 0x7f7f600b1de0 2023-11-24T08:22:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.935+0000 7f7f65726640 1 --2- 172.21.15.107:0/1016025698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f600b2270 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f7f600cc7a0 tx=0x7f7f5802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b2a16cd63dcc85 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.935+0000 7f7f5f7fe640 1 -- 172.21.15.107:0/1016025698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5802ca80 con 0x7f7f600b1de0 2023-11-24T08:22:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.935+0000 7f7f5f7fe640 1 -- 172.21.15.107:0/1016025698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5802cbe0 con 0x7f7f600b1de0 2023-11-24T08:22:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.935+0000 7f7f5f7fe640 1 -- 172.21.15.107:0/1016025698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f580343f0 con 0x7f7f600b1de0 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 -- 172.21.15.107:0/1016025698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 msgr2=0x7f7f600b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 --2- 172.21.15.107:0/1016025698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f600b2270 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f7f600cc7a0 tx=0x7f7f5802bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 -- 172.21.15.107:0/1016025698 shutdown_connections 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 --2- 172.21.15.107:0/1016025698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f600b2270 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 --2- 172.21.15.107:0/1016025698 >> v2:172.21.15.120:3300/0 conn(0x7f7f6015c410 0x7f7f6015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 -- 172.21.15.107:0/1016025698 >> 172.21.15.107:0/1016025698 conn(0x7f7f600cb830 msgr2=0x7f7f600cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:37.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 -- 172.21.15.107:0/1016025698 shutdown_connections 2023-11-24T08:22:37.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.936+0000 7f7f65f27640 1 -- 172.21.15.107:0/1016025698 wait complete. 2023-11-24T08:22:37.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.938+0000 7f7f65f27640 1 Processor -- start 2023-11-24T08:22:37.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.938+0000 7f7f65f27640 1 -- start start 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 0x7f7f600ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f6015c410 0x7f7f600c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c5d60 con 0x7f7f600b1de0 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c6040 con 0x7f7f60144b30 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f65f27640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c6320 con 0x7f7f6015c410 2023-11-24T08:22:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49102/0 (socket says 172.21.15.107:49102) 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 -- 172.21.15.107:0/2303937155 learned_addr learned my addr 172.21.15.107:0/2303937155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f5ffff640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 0x7f7f600ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 -- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f6015c410 msgr2=0x7f7f600c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f6015c410 0x7f7f600c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 -- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 msgr2=0x7f7f600ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 0x7f7f600ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f64f25640 1 -- 172.21.15.107:0/2303937155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58009b30 con 0x7f7f600b1de0 2023-11-24T08:22:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.939+0000 7f7f5ffff640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 0x7f7f600ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:37.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f64f25640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c410 tx=0x7f7f5400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540035f0 con 0x7f7f600b1de0 2023-11-24T08:22:37.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54003750 con 0x7f7f600b1de0 2023-11-24T08:22:37.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54003b40 con 0x7f7f600b1de0 2023-11-24T08:22:37.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600c6600 con 0x7f7f600b1de0 2023-11-24T08:22:37.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.940+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f60134a20 con 0x7f7f600b1de0 2023-11-24T08:22:37.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.941+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f60134bb0 con 0x7f7f600b1de0 2023-11-24T08:22:37.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.942+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f7f54003ca0 con 0x7f7f600b1de0 2023-11-24T08:22:37.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.944+0000 7f7f5d7fa640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 0x7f7f4406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:37.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.944+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7f54087950 con 0x7f7f600b1de0 2023-11-24T08:22:37.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.944+0000 7f7f5ffff640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 0x7f7f4406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:37.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.945+0000 7f7f5ffff640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 0x7f7f4406f550 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7f600cae90 tx=0x7f7f4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:37.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:37.948+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f60134bb0 con 0x7f7f600b1de0 2023-11-24T08:22:38.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.140+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f60134bb0 con 0x7f7f600b1de0 2023-11-24T08:22:38.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.143+0000 7f7f5d7fa640 1 -- 172.21.15.107:0/2303937155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f7f60134bb0 con 0x7f7f600b1de0 2023-11-24T08:22:38.145 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:38.145 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":209,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:35.135447+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:21:48.833467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2023-11-24T08:21:50.037981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":208,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":30,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:38.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 msgr2=0x7f7f4406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 0x7f7f4406f550 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7f600cae90 tx=0x7f7f4c008040 comp rx=0 tx=0).stop 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 msgr2=0x7f7f601344e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c410 tx=0x7f7f5400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 shutdown_connections 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7f4406d090 0x7f7f4406f550 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f6015c410 0x7f7f600c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60144b30 0x7f7f600ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.145+0000 7f7f65f27640 1 --2- 172.21.15.107:0/2303937155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f600b1de0 0x7f7f601344e0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.146+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 >> 172.21.15.107:0/2303937155 conn(0x7f7f600cb830 msgr2=0x7f7f6015c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:38.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.146+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 shutdown_connections 2023-11-24T08:22:38.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.146+0000 7f7f65f27640 1 -- 172.21.15.107:0/2303937155 wait complete. 2023-11-24T08:22:38.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:22:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.376+0000 7f7e0eb90640 1 Processor -- start 2023-11-24T08:22:38.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.377+0000 7f7e0eb90640 1 -- start start 2023-11-24T08:22:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e0eb90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e0eb90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e080c59b0 0x7f7e0806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e0eb90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e080cc3a0 con 0x7f7e080cbf20 2023-11-24T08:22:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e0eb90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0815b820 con 0x7f7e080c59b0 2023-11-24T08:22:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e0eb90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0815eea0 con 0x7f7e0815b440 2023-11-24T08:22:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.378+0000 7f7e07fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7e080cbf20 0x7f7e080c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52928/0 (socket says 172.21.15.107:52928) 2023-11-24T08:22:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e07fff640 1 -- 172.21.15.107:0/997854196 learned_addr learned my addr 172.21.15.107:0/997854196 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0d12d640 1 --2- 172.21.15.107:0/997854196 >> v2:172.21.15.107:3300/0 conn(0x7f7e080c59b0 0x7f7e0806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0c92c640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e077fe640 1 -- 172.21.15.107:0/997854196 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816389035 0 0) 0x7f7df8001490 con 0x7f7e080cbf20 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e077fe640 1 -- 172.21.15.107:0/997854196 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df4003520 con 0x7f7e080cbf20 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0d12d640 1 -- 172.21.15.107:0/997854196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 msgr2=0x7f7e080cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0d12d640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0d12d640 1 -- 172.21.15.107:0/997854196 >> v1:172.21.15.107:6789/0 conn(0x7f7e080cbf20 legacy=0x7f7e080c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.379+0000 7f7e0d12d640 1 -- 172.21.15.107:0/997854196 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e080abf30 con 0x7f7e080c59b0 2023-11-24T08:22:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.380+0000 7f7e0d12d640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080c59b0 0x7f7e0806c5f0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f7e000097a0 tx=0x7f7e0002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1e54ebf23cb008 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e077fe640 1 -- 172.21.15.107:0/997854196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00031780 con 0x7f7e080c59b0 2023-11-24T08:22:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e077fe640 1 -- 172.21.15.107:0/997854196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e000318e0 con 0x7f7e080c59b0 2023-11-24T08:22:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e077fe640 1 -- 172.21.15.107:0/997854196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00031c70 con 0x7f7e080c59b0 2023-11-24T08:22:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 -- 172.21.15.107:0/997854196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080c59b0 msgr2=0x7f7e0806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080c59b0 0x7f7e0806c5f0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f7e000097a0 tx=0x7f7e0002b820 comp rx=0 tx=0).stop 2023-11-24T08:22:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 -- 172.21.15.107:0/997854196 shutdown_connections 2023-11-24T08:22:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080c59b0 0x7f7e0806c5f0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/997854196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 -- 172.21.15.107:0/997854196 >> 172.21.15.107:0/997854196 conn(0x7f7e080b0c50 msgr2=0x7f7e080b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:38.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 -- 172.21.15.107:0/997854196 shutdown_connections 2023-11-24T08:22:38.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.381+0000 7f7e0eb90640 1 -- 172.21.15.107:0/997854196 wait complete. 2023-11-24T08:22:38.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.383+0000 7f7e0eb90640 1 Processor -- start 2023-11-24T08:22:38.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.383+0000 7f7e0eb90640 1 -- start start 2023-11-24T08:22:38.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e080c59b0 0x7f7e08153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e08156230 con 0x7f7e080cbf20 2023-11-24T08:22:38.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e08156510 con 0x7f7e080c59b0 2023-11-24T08:22:38.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e0eb90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e081567f0 con 0x7f7e0815b440 2023-11-24T08:22:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.384+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49132/0 (socket says 172.21.15.107:49132) 2023-11-24T08:22:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 -- 172.21.15.107:0/3183905704 learned_addr learned my addr 172.21.15.107:0/3183905704 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e0c92c640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e080c59b0 0x7f7e08153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 -- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 msgr2=0x7f7e080b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 -- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e080c59b0 msgr2=0x7f7e08153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e080c59b0 0x7f7e08153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 -- 172.21.15.107:0/3183905704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009400 con 0x7f7e080cbf20 2023-11-24T08:22:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e07fff640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f7df800cbc0 tx=0x7f7df800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800ad40 con 0x7f7e080cbf20 2023-11-24T08:22:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8012400 con 0x7f7e080cbf20 2023-11-24T08:22:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.385+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800aea0 con 0x7f7e080cbf20 2023-11-24T08:22:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.386+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e08156ad0 con 0x7f7e080cbf20 2023-11-24T08:22:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.386+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e08154400 con 0x7f7e080cbf20 2023-11-24T08:22:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.387+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e08158a30 con 0x7f7e080cbf20 2023-11-24T08:22:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.388+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f7df8007070 con 0x7f7e080cbf20 2023-11-24T08:22:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.389+0000 7f7e057fa640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 0x7f7de806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.389+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7df80885d0 con 0x7f7e080cbf20 2023-11-24T08:22:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.389+0000 7f7e0c92c640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 0x7f7de806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.390+0000 7f7e0c92c640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 0x7f7de806f550 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7df0006fd0 tx=0x7f7df0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.393+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e08158a30 con 0x7f7e080cbf20 2023-11-24T08:22:38.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.624+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7e080cc3a0 con 0x7f7e080cbf20 2023-11-24T08:22:38.626 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:22:38.625+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:22:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.628+0000 7f7e057fa640 1 -- 172.21.15.107:0/3183905704 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v130) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7e080cc3a0 con 0x7f7e080cbf20 2023-11-24T08:22:38.630 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:22:38.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 msgr2=0x7f7de806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 0x7f7de806f550 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7df0006fd0 tx=0x7f7df0008040 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 msgr2=0x7f7e080af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f7df800cbc0 tx=0x7f7df800a430 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 shutdown_connections 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7de806d090 0x7f7de806f550 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e0815b440 0x7f7e080b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e080cbf20 0x7f7e080af4b0 secure :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f7df800cbc0 tx=0x7f7df800a430 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 --2- 172.21.15.107:0/3183905704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e080c59b0 0x7f7e08153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.631+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 >> 172.21.15.107:0/3183905704 conn(0x7f7e080b0c50 msgr2=0x7f7e08154870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.632+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 shutdown_connections 2023-11-24T08:22:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.632+0000 7f7e0eb90640 1 -- 172.21.15.107:0/3183905704 wait complete. 2023-11-24T08:22:38.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:22:38.727 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:22:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.839+0000 7fe067fff640 1 Processor -- start 2023-11-24T08:22:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.839+0000 7fe067fff640 1 -- start start 2023-11-24T08:22:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe067fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe068143d30 0x7fe068071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe067fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0681450d0 con 0x7fe068141b90 2023-11-24T08:22:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe067fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0681422b0 con 0x7fe0680b1260 2023-11-24T08:22:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe067fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0680725e0 con 0x7fe068143d30 2023-11-24T08:22:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe066ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe068141b90 0x7fe0680b0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52942/0 (socket says 172.21.15.107:52942) 2023-11-24T08:22:38.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.841+0000 7fe066ffd640 1 -- 172.21.15.107:0/1315273404 learned_addr learned my addr 172.21.15.107:0/1315273404 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0677fe640 1 --2- 172.21.15.107:0/1315273404 >> v2:172.21.15.120:3300/0 conn(0x7fe068143d30 0x7fe068071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791868432 0 0) 0x7fe0681450d0 con 0x7fe068141b90 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe054003520 con 0x7fe068141b90 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 375731965 0 0) 0x7fe054003520 con 0x7fe068141b90 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 >> v2:172.21.15.120:3300/0 conn(0x7fe068143d30 msgr2=0x7fe068071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 --2- 172.21.15.107:0/1315273404 >> v2:172.21.15.120:3300/0 conn(0x7fe068143d30 0x7fe068071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 >> v1:172.21.15.120:6789/0 conn(0x7fe0680b1260 legacy=0x7fe0680718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068142520 con 0x7fe068141b90 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.842+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe058001670 con 0x7fe068141b90 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 >> v1:172.21.15.107:6789/0 conn(0x7fe068141b90 legacy=0x7fe0680b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540039c0 0x7fe054003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 0x7fe05400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680725e0 con 0x7fe0540051d0 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05400c120 con 0x7fe0540047c0 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05400c2e0 con 0x7fe0540039c0 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 0x7fe05400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe05ffff640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe066ffd640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540039c0 0x7fe054003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 -- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540039c0 msgr2=0x7fe054003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540039c0 0x7fe054003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 -- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0540047c0 msgr2=0x7fe054004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe0677fe640 1 -- 172.21.15.107:0/1315273404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054003520 con 0x7fe0540051d0 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe05ffff640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.843+0000 7fe066ffd640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540039c0 0x7fe054003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.844+0000 7fe0677fe640 1 -- 172.21.15.107:0/1315273404 reap_dead start 2023-11-24T08:22:38.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.844+0000 7fe0677fe640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 0x7fe05400b980 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fe060009f90 tx=0x7fe0540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4911c592b73e6565 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06002cc20 con 0x7fe0540051d0 2023-11-24T08:22:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06000f040 con 0x7fe0540051d0 2023-11-24T08:22:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe0667fc640 1 -- 172.21.15.107:0/1315273404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060003990 con 0x7fe0540051d0 2023-11-24T08:22:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 -- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 msgr2=0x7fe05400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 0x7fe05400b980 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fe060009f90 tx=0x7fe0540057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 -- 172.21.15.107:0/1315273404 shutdown_connections 2023-11-24T08:22:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 --2- 172.21.15.107:0/1315273404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0540051d0 0x7fe05400b980 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 -- 172.21.15.107:0/1315273404 >> 172.21.15.107:0/1315273404 conn(0x7fe06806a500 msgr2=0x7fe06806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 -- 172.21.15.107:0/1315273404 shutdown_connections 2023-11-24T08:22:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.845+0000 7fe067fff640 1 -- 172.21.15.107:0/1315273404 wait complete. 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.847+0000 7fe067fff640 1 Processor -- start 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.847+0000 7fe067fff640 1 -- start start 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540051d0 0x7fe06815e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 0x7fe068157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cdde0 con 0x7fe068157fe0 2023-11-24T08:22:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680ce080 con 0x7fe0680cf120 2023-11-24T08:22:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe067fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680ce360 con 0x7fe0540051d0 2023-11-24T08:22:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe0677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe0677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49186/0 (socket says 172.21.15.107:49186) 2023-11-24T08:22:38.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.848+0000 7fe0677fe640 1 -- 172.21.15.107:0/4015575966 learned_addr learned my addr 172.21.15.107:0/4015575966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe05ffff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 0x7fe068157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe066ffd640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540051d0 0x7fe06815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 -- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540051d0 msgr2=0x7fe06815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540051d0 0x7fe06815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 -- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 msgr2=0x7fe068157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 0x7fe068157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 -- 172.21.15.107:0/4015575966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680ce640 con 0x7fe068157fe0 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe05ffff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 0x7fe068157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe0677fe640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7fe06002c690 tx=0x7fe060009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.849+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060003ba0 con 0x7fe068157fe0 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.850+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe060003d00 con 0x7fe068157fe0 2023-11-24T08:22:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.850+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060035b70 con 0x7fe068157fe0 2023-11-24T08:22:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.850+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0680ce920 con 0x7fe068157fe0 2023-11-24T08:22:38.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.850+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0680cf5a0 con 0x7fe068157fe0 2023-11-24T08:22:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.851+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0680cee40 con 0x7fe068157fe0 2023-11-24T08:22:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.852+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe060035cd0 con 0x7fe068157fe0 2023-11-24T08:22:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.853+0000 7fe05f7fe640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 0x7fe04806f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.853+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe0600ab390 con 0x7fe068157fe0 2023-11-24T08:22:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.853+0000 7fe066ffd640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 0x7fe04806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.854+0000 7fe066ffd640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 0x7fe04806f580 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fe0580017a0 tx=0x7fe058009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:38.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:38.858+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0680cee40 con 0x7fe068157fe0 2023-11-24T08:22:39.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.148+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe0680cee40 con 0x7fe068157fe0 2023-11-24T08:22:39.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.632+0000 7fe05f7fe640 1 -- 172.21.15.107:0/4015575966 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe0680cee40 con 0x7fe068157fe0 2023-11-24T08:22:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 msgr2=0x7fe04806f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 0x7fe04806f580 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fe0580017a0 tx=0x7fe058009040 comp rx=0 tx=0).stop 2023-11-24T08:22:39.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 msgr2=0x7fe068155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7fe06002c690 tx=0x7fe060009750 comp rx=0 tx=0).stop 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 shutdown_connections 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe04806d0c0 0x7fe04806f580 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe068157fe0 0x7fe068155020 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0680cf120 0x7fe068157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 --2- 172.21.15.107:0/4015575966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0540051d0 0x7fe06815e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 >> 172.21.15.107:0/4015575966 conn(0x7fe06806a500 msgr2=0x7fe06806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 shutdown_connections 2023-11-24T08:22:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.635+0000 7fe067fff640 1 -- 172.21.15.107:0/4015575966 wait complete. 2023-11-24T08:22:39.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:22:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.857+0000 7fd789f88640 1 Processor -- start 2023-11-24T08:22:39.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.857+0000 7fd789f88640 1 -- start start 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd78413e230 0x7fd78406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78415fa40 0x7fd78406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd78415b840 0x7fd78415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7841417c0 con 0x7fd78415b840 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78415ed50 con 0x7fd78413e230 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789f88640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78415f5b0 con 0x7fd78415fa40 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789787640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd78415b840 0x7fd78415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789787640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd78415b840 0x7fd78415bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49192/0 (socket says 172.21.15.107:49192) 2023-11-24T08:22:39.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd789787640 1 -- 172.21.15.107:0/576688530 learned_addr learned my addr 172.21.15.107:0/576688530 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:39.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd783fff640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78415fa40 0x7fd78406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.859+0000 7fd788f86640 1 --2- 172.21.15.107:0/576688530 >> v2:172.21.15.120:3300/0 conn(0x7fd78413e230 0x7fd78406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.860+0000 7fd789787640 1 -- 172.21.15.107:0/576688530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78415fa40 msgr2=0x7fd78406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:39.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.860+0000 7fd789787640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78415fa40 0x7fd78406ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.860+0000 7fd789787640 1 -- 172.21.15.107:0/576688530 >> v2:172.21.15.120:3300/0 conn(0x7fd78413e230 msgr2=0x7fd78406c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:39.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.860+0000 7fd789787640 1 --2- 172.21.15.107:0/576688530 >> v2:172.21.15.120:3300/0 conn(0x7fd78413e230 0x7fd78406c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.860+0000 7fd789787640 1 -- 172.21.15.107:0/576688530 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78415c570 con 0x7fd78415b840 2023-11-24T08:22:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.861+0000 7fd789787640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd78415bc70 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fd7840cff30 tx=0x7fd77c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fde3c794f3a3b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.862+0000 7fd7837fe640 1 -- 172.21.15.107:0/576688530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c02c9e0 con 0x7fd78415b840 2023-11-24T08:22:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.862+0000 7fd7837fe640 1 -- 172.21.15.107:0/576688530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c02cb40 con 0x7fd78415b840 2023-11-24T08:22:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.862+0000 7fd7837fe640 1 -- 172.21.15.107:0/576688530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c0343f0 con 0x7fd78415b840 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.862+0000 7fd789f88640 1 -- 172.21.15.107:0/576688530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 msgr2=0x7fd78415bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.862+0000 7fd789f88640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd78415bc70 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fd7840cff30 tx=0x7fd77c02be40 comp rx=0 tx=0).stop 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 -- 172.21.15.107:0/576688530 shutdown_connections 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd78415bc70 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 --2- 172.21.15.107:0/576688530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78415fa40 0x7fd78406ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 --2- 172.21.15.107:0/576688530 >> v2:172.21.15.120:3300/0 conn(0x7fd78413e230 0x7fd78406c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 -- 172.21.15.107:0/576688530 >> 172.21.15.107:0/576688530 conn(0x7fd78406dc30 msgr2=0x7fd78406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 -- 172.21.15.107:0/576688530 shutdown_connections 2023-11-24T08:22:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.863+0000 7fd789f88640 1 -- 172.21.15.107:0/576688530 wait complete. 2023-11-24T08:22:39.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.865+0000 7fd789f88640 1 Processor -- start 2023-11-24T08:22:39.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.865+0000 7fd789f88640 1 -- start start 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78413e230 0x7fd7841f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd78415fa40 0x7fd78412e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7841f3a40 con 0x7fd78415b840 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784151cb0 con 0x7fd78415fa40 2023-11-24T08:22:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.866+0000 7fd789f88640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78412ee40 con 0x7fd78413e230 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49202/0 (socket says 172.21.15.107:49202) 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 -- 172.21.15.107:0/1672420429 learned_addr learned my addr 172.21.15.107:0/1672420429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd788f86640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78413e230 0x7fd7841f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd789787640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd78415fa40 0x7fd78412e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 -- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78413e230 msgr2=0x7fd7841f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78413e230 0x7fd7841f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 -- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd78415fa40 msgr2=0x7fd78412e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd78415fa40 0x7fd78412e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 -- 172.21.15.107:0/1672420429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c009b30 con 0x7fd78415b840 2023-11-24T08:22:39.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.867+0000 7fd783fff640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fd77000d5f0 tx=0x7fd77000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.868+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7700125f0 con 0x7fd78415b840 2023-11-24T08:22:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.868+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770012750 con 0x7fd78415b840 2023-11-24T08:22:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.868+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770012b00 con 0x7fd78415b840 2023-11-24T08:22:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.868+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78412f120 con 0x7fd78415b840 2023-11-24T08:22:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.868+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7841f2fe0 con 0x7fd78415b840 2023-11-24T08:22:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.869+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78404e840 con 0x7fd78415b840 2023-11-24T08:22:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.871+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fd770012c60 con 0x7fd78415b840 2023-11-24T08:22:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.872+0000 7fd7817fa640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 0x7fd76806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.872+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd770088bb0 con 0x7fd78415b840 2023-11-24T08:22:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.872+0000 7fd788f86640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 0x7fd76806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.873+0000 7fd788f86640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 0x7fd76806f550 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd778006fd0 tx=0x7fd778008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:39.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:39.877+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd770055740 con 0x7fd78415b840 2023-11-24T08:22:40.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.069+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd78406b1d0 con 0x7fd78415b840 2023-11-24T08:22:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.636+0000 7fd7817fa640 1 -- 172.21.15.107:0/1672420429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v211) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd770058d60 con 0x7fd78415b840 2023-11-24T08:22:40.639 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:22:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.640+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 msgr2=0x7fd76806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.640+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 0x7fd76806f550 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd778006fd0 tx=0x7fd778008040 comp rx=0 tx=0).stop 2023-11-24T08:22:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.640+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 msgr2=0x7fd7841f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.640+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fd77000d5f0 tx=0x7fd77000dab0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 shutdown_connections 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd76806d090 0x7fd76806f550 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd78415fa40 0x7fd78412e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78415b840 0x7fd7841f2430 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 --2- 172.21.15.107:0/1672420429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd78413e230 0x7fd7841f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 >> 172.21.15.107:0/1672420429 conn(0x7fd78406dc30 msgr2=0x7fd78415c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.641+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 shutdown_connections 2023-11-24T08:22:40.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.642+0000 7fd789f88640 1 -- 172.21.15.107:0/1672420429 wait complete. 2023-11-24T08:22:40.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:22:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.879+0000 7f04ca7b0640 1 Processor -- start 2023-11-24T08:22:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.880+0000 7f04ca7b0640 1 -- start start 2023-11-24T08:22:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04ca7b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c415c7f0 0x7f04c406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04ca7b0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04c40ce6c0 0x7f04c40cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04ca7b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40cece0 con 0x7f04c40cf450 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04ca7b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c40cfe40 con 0x7f04c415c7f0 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04ca7b0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c40ceaf0 con 0x7f04c40ce6c0 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.881+0000 7f04c3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c415c7f0 0x7f04c406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c415c7f0 0x7f04c406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49214/0 (socket says 172.21.15.107:49214) 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 -- 172.21.15.107:0/4027280378 learned_addr learned my addr 172.21.15.107:0/4027280378 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c37fe640 1 --1- 172.21.15.107:0/4027280378 >> v1:172.21.15.107:6789/0 conn(0x7f04c40cf450 0x7f04c40cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52950/0 (socket says 172.21.15.107:52950) 2023-11-24T08:22:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c8d4d640 1 --2- 172.21.15.107:0/4027280378 >> v2:172.21.15.120:3300/0 conn(0x7f04c40ce6c0 0x7f04c40cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 -- 172.21.15.107:0/4027280378 >> v2:172.21.15.120:3300/0 conn(0x7f04c40ce6c0 msgr2=0x7f04c40cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 --2- 172.21.15.107:0/4027280378 >> v2:172.21.15.120:3300/0 conn(0x7f04c40ce6c0 0x7f04c40cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 -- 172.21.15.107:0/4027280378 >> v1:172.21.15.107:6789/0 conn(0x7f04c40cf450 legacy=0x7f04c40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c3fff640 1 -- 172.21.15.107:0/4027280378 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c406d040 con 0x7f04c415c7f0 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.882+0000 7f04c2ffd640 1 -- 172.21.15.107:0/4027280378 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443249003 0 0) 0x7f04c40cece0 con 0x7f04c40cf450 2023-11-24T08:22:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.883+0000 7f04c3fff640 1 --2- 172.21.15.107:0/4027280378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c415c7f0 0x7f04c406db20 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f04c40d00e0 tx=0x7f04b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9638fbd41558acea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04c2ffd640 1 -- 172.21.15.107:0/4027280378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4030860 con 0x7f04c415c7f0 2023-11-24T08:22:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04c2ffd640 1 -- 172.21.15.107:0/4027280378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40309c0 con 0x7f04c415c7f0 2023-11-24T08:22:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04c2ffd640 1 -- 172.21.15.107:0/4027280378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4030d50 con 0x7f04c415c7f0 2023-11-24T08:22:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 -- 172.21.15.107:0/4027280378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c415c7f0 msgr2=0x7f04c406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/4027280378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c415c7f0 0x7f04c406db20 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f04c40d00e0 tx=0x7f04b402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 -- 172.21.15.107:0/4027280378 shutdown_connections 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/4027280378 >> v2:172.21.15.120:3300/0 conn(0x7f04c40ce6c0 0x7f04c40cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/4027280378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c415c7f0 0x7f04c406db20 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 -- 172.21.15.107:0/4027280378 >> 172.21.15.107:0/4027280378 conn(0x7f04c406a920 msgr2=0x7f04c406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.884+0000 7f04ca7b0640 1 -- 172.21.15.107:0/4027280378 shutdown_connections 2023-11-24T08:22:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.885+0000 7f04ca7b0640 1 -- 172.21.15.107:0/4027280378 wait complete. 2023-11-24T08:22:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.886+0000 7f04ca7b0640 1 Processor -- start 2023-11-24T08:22:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.886+0000 7f04ca7b0640 1 -- start start 2023-11-24T08:22:40.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c40ce6c0 0x7f04c41514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c415c7f0 0x7f04c40c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40ad720 con 0x7f04c40cf450 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40ada00 con 0x7f04c40ce6c0 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.887+0000 7f04ca7b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40adce0 con 0x7f04c415c7f0 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49218/0 (socket says 172.21.15.107:49218) 2023-11-24T08:22:40.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 -- 172.21.15.107:0/642501243 learned_addr learned my addr 172.21.15.107:0/642501243 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:40.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c3fff640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c40ce6c0 0x7f04c41514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:40.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 -- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c415c7f0 msgr2=0x7f04c40c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c415c7f0 0x7f04c40c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 -- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c40ce6c0 msgr2=0x7f04c41514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c40ce6c0 0x7f04c41514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.888+0000 7f04c37fe640 1 -- 172.21.15.107:0/642501243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40094d0 con 0x7f04c40cf450 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04c37fe640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c890 tx=0x7f04ac00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00a640 con 0x7f04c40cf450 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac00a7c0 con 0x7f04c40cf450 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00ab70 con 0x7f04c40cf450 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c40adfc0 con 0x7f04c40cf450 2023-11-24T08:22:40.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.889+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c4151a30 con 0x7f04c40cf450 2023-11-24T08:22:40.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.890+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c4067470 con 0x7f04c40cf450 2023-11-24T08:22:40.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.891+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f04ac00acd0 con 0x7f04c40cf450 2023-11-24T08:22:40.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.892+0000 7f04c0ff9640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 0x7f04a406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:40.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.892+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f04ac087e00 con 0x7f04c40cf450 2023-11-24T08:22:40.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.892+0000 7f04c3fff640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 0x7f04a406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:40.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.893+0000 7f04c3fff640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 0x7f04a406f550 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036000 tx=0x7f04b403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:40.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:40.897+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04ac054880 con 0x7f04c40cf450 2023-11-24T08:22:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.087+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f04c41f33b0 con 0x7f04c40cf450 2023-11-24T08:22:41.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.650+0000 7f04c0ff9640 1 -- 172.21.15.107:0/642501243 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v212) v1 ==== 169+0+0 (secure 0 0 0) 0x7f04ac018030 con 0x7f04c40cf450 2023-11-24T08:22:41.652 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:22:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.653+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 msgr2=0x7f04a406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.653+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 0x7f04a406f550 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036000 tx=0x7f04b403b040 comp rx=0 tx=0).stop 2023-11-24T08:22:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.653+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 msgr2=0x7f04c40c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.653+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c890 tx=0x7f04ac00cd50 comp rx=0 tx=0).stop 2023-11-24T08:22:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.653+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 shutdown_connections 2023-11-24T08:22:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f04a406d090 0x7f04a406f550 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04c415c7f0 0x7f04c40c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c40cf450 0x7f04c40c9510 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 --2- 172.21.15.107:0/642501243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c40ce6c0 0x7f04c41514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 >> 172.21.15.107:0/642501243 conn(0x7f04c406a920 msgr2=0x7f04c40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 shutdown_connections 2023-11-24T08:22:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.654+0000 7f04ca7b0640 1 -- 172.21.15.107:0/642501243 wait complete. 2023-11-24T08:22:41.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:22:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.890+0000 7f82dab8f640 1 Processor -- start 2023-11-24T08:22:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.891+0000 7f82dab8f640 1 -- start start 2023-11-24T08:22:41.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82dab8f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82d4141b90 0x7f82d40b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82dab8f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82d40b1090 0x7f82d40718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82dab8f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d40b14f0 con 0x7f82d4071e10 2023-11-24T08:22:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82dab8f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d41450d0 con 0x7f82d4141b90 2023-11-24T08:22:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82dab8f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d41422b0 con 0x7f82d40b1090 2023-11-24T08:22:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82d9b8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f82d4141b90 0x7f82d40b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82da38e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82d4071e10 0x7f82d415f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52956/0 (socket says 172.21.15.107:52956) 2023-11-24T08:22:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.892+0000 7f82da38e640 1 -- 172.21.15.107:0/92294792 learned_addr learned my addr 172.21.15.107:0/92294792 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d938c640 1 --2- 172.21.15.107:0/92294792 >> v2:172.21.15.120:3300/0 conn(0x7f82d40b1090 0x7f82d40718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d9b8d640 1 -- 172.21.15.107:0/92294792 >> v2:172.21.15.120:3300/0 conn(0x7f82d40b1090 msgr2=0x7f82d40718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d9b8d640 1 --2- 172.21.15.107:0/92294792 >> v2:172.21.15.120:3300/0 conn(0x7f82d40b1090 0x7f82d40718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d9b8d640 1 -- 172.21.15.107:0/92294792 >> v1:172.21.15.107:6789/0 conn(0x7f82d4071e10 legacy=0x7f82d415f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d9b8d640 1 -- 172.21.15.107:0/92294792 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d40725e0 con 0x7f82d4141b90 2023-11-24T08:22:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.893+0000 7f82d8b8b640 1 -- 172.21.15.107:0/92294792 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846449593 0 0) 0x7f82d40b14f0 con 0x7f82d4071e10 2023-11-24T08:22:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.894+0000 7f82d9b8d640 1 --2- 172.21.15.107:0/92294792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4141b90 0x7f82d40b0b50 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b19a0 tx=0x7f82d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7429f22b0eaa38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:41.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.894+0000 7f82d8b8b640 1 -- 172.21.15.107:0/92294792 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d002e070 con 0x7f82d4141b90 2023-11-24T08:22:41.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.894+0000 7f82d8b8b640 1 -- 172.21.15.107:0/92294792 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00315a0 con 0x7f82d4141b90 2023-11-24T08:22:41.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.894+0000 7f82d8b8b640 1 -- 172.21.15.107:0/92294792 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031930 con 0x7f82d4141b90 2023-11-24T08:22:41.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 -- 172.21.15.107:0/92294792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4141b90 msgr2=0x7f82d40b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 --2- 172.21.15.107:0/92294792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4141b90 0x7f82d40b0b50 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b19a0 tx=0x7f82d002b840 comp rx=0 tx=0).stop 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 -- 172.21.15.107:0/92294792 shutdown_connections 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 --2- 172.21.15.107:0/92294792 >> v2:172.21.15.120:3300/0 conn(0x7f82d40b1090 0x7f82d40718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 --2- 172.21.15.107:0/92294792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4141b90 0x7f82d40b0b50 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 -- 172.21.15.107:0/92294792 >> 172.21.15.107:0/92294792 conn(0x7f82d406a500 msgr2=0x7f82d406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 -- 172.21.15.107:0/92294792 shutdown_connections 2023-11-24T08:22:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.895+0000 7f82dab8f640 1 -- 172.21.15.107:0/92294792 wait complete. 2023-11-24T08:22:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.897+0000 7f82dab8f640 1 Processor -- start 2023-11-24T08:22:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.897+0000 7f82dab8f640 1 -- start start 2023-11-24T08:22:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 0x7f82d40cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d4141b90 0x7f82d40aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d40b39f0 con 0x7f82d4071e10 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d40cf510 con 0x7f82d4141b90 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82dab8f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d40b2ad0 con 0x7f82d40b1090 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82d9b8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82d9b8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49242/0 (socket says 172.21.15.107:49242) 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82d9b8d640 1 -- 172.21.15.107:0/623797308 learned_addr learned my addr 172.21.15.107:0/623797308 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.898+0000 7f82da38e640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d4141b90 0x7f82d40aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d938c640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 0x7f82d40cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 -- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 msgr2=0x7f82d40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 0x7f82d40cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 -- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d4141b90 msgr2=0x7f82d40aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d4141b90 0x7f82d40aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 -- 172.21.15.107:0/623797308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0009400 con 0x7f82d4071e10 2023-11-24T08:22:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d938c640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 0x7f82d40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82d9b8d640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f82d0034040 tx=0x7f82d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0032840 con 0x7f82d4071e10 2023-11-24T08:22:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00329a0 con 0x7f82d4071e10 2023-11-24T08:22:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0032d50 con 0x7f82d4071e10 2023-11-24T08:22:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d40b2d70 con 0x7f82d4071e10 2023-11-24T08:22:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.899+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d40b3330 con 0x7f82d4071e10 2023-11-24T08:22:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.901+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d404e840 con 0x7f82d4071e10 2023-11-24T08:22:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.902+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f82d003b3f0 con 0x7f82d4071e10 2023-11-24T08:22:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.903+0000 7f82ca7fc640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 0x7f82b806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.903+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82d00aa7b0 con 0x7f82d4071e10 2023-11-24T08:22:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.903+0000 7f82d938c640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 0x7f82b806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.904+0000 7f82d938c640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 0x7f82b806f550 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006fd0 tx=0x7f82c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:41.907+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82d0077340 con 0x7f82d4071e10 2023-11-24T08:22:42.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.113+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f82d406b510 con 0x7f82d4071e10 2023-11-24T08:22:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.118+0000 7f82ca7fc640 1 -- 172.21.15.107:0/623797308 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f82d007a960 con 0x7f82d4071e10 2023-11-24T08:22:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 msgr2=0x7f82b806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 0x7f82b806f550 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006fd0 tx=0x7f82c4008040 comp rx=0 tx=0).stop 2023-11-24T08:22:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 msgr2=0x7f82d40cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f82d0034040 tx=0x7f82d00046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 shutdown_connections 2023-11-24T08:22:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f82b806d090 0x7f82b806f550 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82d4141b90 0x7f82d40aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82d40b1090 0x7f82d40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 --2- 172.21.15.107:0/623797308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d4071e10 0x7f82d40cefd0 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 >> 172.21.15.107:0/623797308 conn(0x7f82d406a500 msgr2=0x7f82d4140a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 shutdown_connections 2023-11-24T08:22:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.121+0000 7f82dab8f640 1 -- 172.21.15.107:0/623797308 wait complete. 2023-11-24T08:22:42.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots' 2023-11-24T08:22:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.341+0000 7f531db97640 1 Processor -- start 2023-11-24T08:22:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.342+0000 7f531db97640 1 -- start start 2023-11-24T08:22:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f531db97640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53180b2250 0x7f531815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f531db97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 0x7f531815b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f531db97640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531815ab80 con 0x7f53180b2630 2023-11-24T08:22:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f531db97640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531815a990 con 0x7f53180b2250 2023-11-24T08:22:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f531db97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531815e680 con 0x7f53180b2aa0 2023-11-24T08:22:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f53177fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53180b2250 0x7f531815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f53177fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53180b2250 0x7f531815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49246/0 (socket says 172.21.15.107:49246) 2023-11-24T08:22:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.343+0000 7f53177fe640 1 -- 172.21.15.107:0/985157113 learned_addr learned my addr 172.21.15.107:0/985157113 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f5317fff640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 0x7f531815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f53177fe640 1 -- 172.21.15.107:0/985157113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 msgr2=0x7f531815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f53177fe640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 0x7f531815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f53177fe640 1 -- 172.21.15.107:0/985157113 >> v1:172.21.15.120:6789/0 conn(0x7f53180b2630 legacy=0x7f53180ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f53177fe640 1 -- 172.21.15.107:0/985157113 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531815e960 con 0x7f53180b2250 2023-11-24T08:22:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.344+0000 7f5317fff640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 0x7f531815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.345+0000 7f53177fe640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f531815df80 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f530800d150 tx=0x7f530802e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6220eb74b6025a13 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.345+0000 7f5316ffd640 1 -- 172.21.15.107:0/985157113 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530802ea70 con 0x7f53180b2250 2023-11-24T08:22:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.345+0000 7f5316ffd640 1 -- 172.21.15.107:0/985157113 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530802ebd0 con 0x7f53180b2250 2023-11-24T08:22:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.345+0000 7f5316ffd640 1 -- 172.21.15.107:0/985157113 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308009600 con 0x7f53180b2250 2023-11-24T08:22:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 -- 172.21.15.107:0/985157113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 msgr2=0x7f531815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f531815df80 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f530800d150 tx=0x7f530802e0d0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 -- 172.21.15.107:0/985157113 shutdown_connections 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2aa0 0x7f531815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 --2- 172.21.15.107:0/985157113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f531815df80 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 -- 172.21.15.107:0/985157113 >> 172.21.15.107:0/985157113 conn(0x7f531806a920 msgr2=0x7f531806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 -- 172.21.15.107:0/985157113 shutdown_connections 2023-11-24T08:22:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.346+0000 7f531db97640 1 -- 172.21.15.107:0/985157113 wait complete. 2023-11-24T08:22:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.348+0000 7f531db97640 1 Processor -- start 2023-11-24T08:22:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.348+0000 7f531db97640 1 -- start start 2023-11-24T08:22:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2630 0x7f53181e9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53180b2aa0 0x7f5318070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318154130 con 0x7f53180b2250 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318156430 con 0x7f53180b2aa0 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f531db97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53181e97b0 con 0x7f53180b2630 2023-11-24T08:22:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49256/0 (socket says 172.21.15.107:49256) 2023-11-24T08:22:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 -- 172.21.15.107:0/1104403159 learned_addr learned my addr 172.21.15.107:0/1104403159 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 -- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2630 msgr2=0x7f53181e9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2630 0x7f53181e9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 -- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53180b2aa0 msgr2=0x7f5318070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53180b2aa0 0x7f5318070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.349+0000 7f53177fe640 1 -- 172.21.15.107:0/1104403159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530800bd40 con 0x7f53180b2250 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f53177fe640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f5308000f80 tx=0x7f5308004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530802ea70 con 0x7f53180b2250 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308036820 con 0x7f53180b2250 2023-11-24T08:22:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308036c10 con 0x7f53180b2250 2023-11-24T08:22:42.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180cd7c0 con 0x7f53180b2250 2023-11-24T08:22:42.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.350+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53180c9690 con 0x7f53180b2250 2023-11-24T08:22:42.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.351+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318067470 con 0x7f53180b2250 2023-11-24T08:22:42.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.352+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f530804a460 con 0x7f53180b2250 2023-11-24T08:22:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.353+0000 7f5314ff9640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 0x7f52f806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.353+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53080abc80 con 0x7f53180b2250 2023-11-24T08:22:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.353+0000 7f530effd640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 0x7f52f806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.354+0000 7f530effd640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 0x7f52f806f550 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f5318071740 tx=0x7f53000079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.357+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5308078710 con 0x7f53180b2250 2023-11-24T08:22:42.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.618+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots"]} v 0) v1 -- 0x7f5318067470 con 0x7f53180b2250 2023-11-24T08:22:42.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.660+0000 7f5314ff9640 1 -- 172.21.15.107:0/1104403159 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_clone_retain_snapshot_with_snapshots"]}]=0 v899) v1 ==== 184+0+0 (secure 0 0 0) 0x7f530800b720 con 0x7f53180b2250 2023-11-24T08:22:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.663+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 msgr2=0x7f52f806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.663+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 0x7f52f806f550 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f5318071740 tx=0x7f53000079b0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.663+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 msgr2=0x7f5318153bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.663+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f5308000f80 tx=0x7f5308004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 shutdown_connections 2023-11-24T08:22:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f52f806d090 0x7f52f806f550 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53180b2aa0 0x7f5318070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53180b2630 0x7f53181e9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 --2- 172.21.15.107:0/1104403159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53180b2250 0x7f5318153bf0 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 >> 172.21.15.107:0/1104403159 conn(0x7f531806a920 msgr2=0x7f5318071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 shutdown_connections 2023-11-24T08:22:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.664+0000 7f531db97640 1 -- 172.21.15.107:0/1104403159 wait complete. 2023-11-24T08:22:42.684 INFO:tasks.cephfs_test_runner:retain snapshots of a cloned subvolume and check disallowed operations ... ok 2023-11-24T08:22:42.685 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_retain_snapshot_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) clone a snapshot from a snapshot retained subvolume 2023-11-24T08:22:42.685 INFO:tasks.cephfs_test_runner:test_subvolume_retain_snapshot_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:22:42.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.890+0000 7f2ab97fb640 1 Processor -- start 2023-11-24T08:22:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.890+0000 7f2ab97fb640 1 -- start start 2023-11-24T08:22:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab97fb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ab40b2250 0x7f2ab415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab97fb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2ab40b2630 0x7f2ab40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab97fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab415ab80 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab97fb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab415a990 con 0x7f2ab40b2250 2023-11-24T08:22:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab97fb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab415e680 con 0x7f2ab40b2630 2023-11-24T08:22:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ab40b2250 0x7f2ab415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ab40b2250 0x7f2ab415df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49268/0 (socket says 172.21.15.107:49268) 2023-11-24T08:22:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.892+0000 7f2ab2ffd640 1 -- 172.21.15.107:0/606746552 learned_addr learned my addr 172.21.15.107:0/606746552 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab27fc640 1 --2- 172.21.15.107:0/606746552 >> v2:172.21.15.120:3300/0 conn(0x7f2ab40b2630 0x7f2ab40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab2ffd640 1 -- 172.21.15.107:0/606746552 >> v2:172.21.15.120:3300/0 conn(0x7f2ab40b2630 msgr2=0x7f2ab40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/606746552 >> v2:172.21.15.120:3300/0 conn(0x7f2ab40b2630 0x7f2ab40ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab2ffd640 1 -- 172.21.15.107:0/606746552 >> v1:172.21.15.107:6789/0 conn(0x7f2ab40b2aa0 legacy=0x7f2ab415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab2ffd640 1 -- 172.21.15.107:0/606746552 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab415e960 con 0x7f2ab40b2250 2023-11-24T08:22:42.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.893+0000 7f2ab1ffb640 1 -- 172.21.15.107:0/606746552 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863020353 0 0) 0x7f2ab415ab80 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.894+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/606746552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2250 0x7f2ab415df80 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40ae5c0 tx=0x7f2aa402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7db1d0565339195 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.894+0000 7f2ab1ffb640 1 -- 172.21.15.107:0/606746552 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4036440 con 0x7f2ab40b2250 2023-11-24T08:22:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.894+0000 7f2ab1ffb640 1 -- 172.21.15.107:0/606746552 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa40365a0 con 0x7f2ab40b2250 2023-11-24T08:22:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.894+0000 7f2ab1ffb640 1 -- 172.21.15.107:0/606746552 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4036930 con 0x7f2ab40b2250 2023-11-24T08:22:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 -- 172.21.15.107:0/606746552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2250 msgr2=0x7f2ab415df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/606746552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2250 0x7f2ab415df80 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40ae5c0 tx=0x7f2aa402e170 comp rx=0 tx=0).stop 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 -- 172.21.15.107:0/606746552 shutdown_connections 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/606746552 >> v2:172.21.15.120:3300/0 conn(0x7f2ab40b2630 0x7f2ab40ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/606746552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2250 0x7f2ab415df80 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 -- 172.21.15.107:0/606746552 >> 172.21.15.107:0/606746552 conn(0x7f2ab406a920 msgr2=0x7f2ab406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 -- 172.21.15.107:0/606746552 shutdown_connections 2023-11-24T08:22:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.895+0000 7f2ab97fb640 1 -- 172.21.15.107:0/606746552 wait complete. 2023-11-24T08:22:42.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.897+0000 7f2ab97fb640 1 Processor -- start 2023-11-24T08:22:42.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.897+0000 7f2ab97fb640 1 -- start start 2023-11-24T08:22:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 0x7f2ab4155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ab40b2630 0x7f2ab409cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40ad490 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab409d130 con 0x7f2ab40b2250 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab97fb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40acb60 con 0x7f2ab40b2630 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49272/0 (socket says 172.21.15.107:49272) 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 -- 172.21.15.107:0/983460478 learned_addr learned my addr 172.21.15.107:0/983460478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 0x7f2ab4155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 -- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ab40b2630 msgr2=0x7f2ab409cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab27fc640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ab40b2630 0x7f2ab409cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ab40b2630 0x7f2ab409cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 -- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 msgr2=0x7f2ab4155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 0x7f2ab4155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab37fe640 1 -- 172.21.15.107:0/983460478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa400b770 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.898+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 0x7f2ab4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2ab37fe640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c780 tx=0x7f2aac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac003b70 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac003cd0 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0052b0 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab40ace40 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.899+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab40b0300 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.900+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab40ad600 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.901+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f2aac005410 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.902+0000 7f2a9b7fe640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 0x7f2aa006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.902+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2aac087ff0 con 0x7f2ab40b2aa0 2023-11-24T08:22:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.902+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 0x7f2aa006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.903+0000 7f2ab2ffd640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 0x7f2aa006f550 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4038000 tx=0x7f2aa4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:42.906+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aac054a70 con 0x7f2ab40b2aa0 2023-11-24T08:22:43.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.097+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ab4067470 con 0x7f2ab40b2aa0 2023-11-24T08:22:43.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.099+0000 7f2a9b7fe640 1 -- 172.21.15.107:0/983460478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+12998 (secure 0 0 0) 0x7f2aac017020 con 0x7f2ab40b2aa0 2023-11-24T08:22:43.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:43.101 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":212,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:41.647289+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/2018281423":"2023-11-25T08:22:38.625865+0000","172.21.15.107:6836/2018281423":"2023-11-25T08:22:38.625865+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:43.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 msgr2=0x7f2aa006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 0x7f2aa006f550 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4038000 tx=0x7f2aa4032000 comp rx=0 tx=0).stop 2023-11-24T08:22:43.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 msgr2=0x7f2ab40af460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c780 tx=0x7f2aac00cc40 comp rx=0 tx=0).stop 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 shutdown_connections 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2aa006d090 0x7f2aa006f550 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40b2aa0 0x7f2ab40af460 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ab40b2630 0x7f2ab409cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 --2- 172.21.15.107:0/983460478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ab40b2250 0x7f2ab4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 >> 172.21.15.107:0/983460478 conn(0x7f2ab406a920 msgr2=0x7f2ab41585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:43.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 shutdown_connections 2023-11-24T08:22:43.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.101+0000 7f2ab97fb640 1 -- 172.21.15.107:0/983460478 wait complete. 2023-11-24T08:22:43.122 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:22:43.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:22:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.329+0000 7f1a96609640 1 Processor -- start 2023-11-24T08:22:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.329+0000 7f1a96609640 1 -- start start 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.330+0000 7f1a96609640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a900b0b40 0x7f1a900b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.330+0000 7f1a96609640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a9006c5b0 0x7f1a9006ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.330+0000 7f1a96609640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9015ed20 con 0x7f1a9015b260 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.330+0000 7f1a96609640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9015b640 con 0x7f1a9006c5b0 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a96609640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9015f8a0 con 0x7f1a900b0b40 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a9006c5b0 0x7f1a9006ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1a9006c5b0 0x7f1a9006ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49276/0 (socket says 172.21.15.107:49276) 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 -- 172.21.15.107:0/2465908035 learned_addr learned my addr 172.21.15.107:0/2465908035 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 -- 172.21.15.107:0/2465908035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a900b0b40 msgr2=0x7f1a900b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 --2- 172.21.15.107:0/2465908035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a900b0b40 0x7f1a900b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 -- 172.21.15.107:0/2465908035 >> v1:172.21.15.120:6789/0 conn(0x7f1a9015b260 legacy=0x7f1a9006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.331+0000 7f1a95e08640 1 -- 172.21.15.107:0/2465908035 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a900b14c0 con 0x7f1a9006c5b0 2023-11-24T08:22:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.332+0000 7f1a95e08640 1 --2- 172.21.15.107:0/2465908035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9006c5b0 0x7f1a9006ca40 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a8802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27c11c9d4faeac9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.333+0000 7f1a87fff640 1 -- 172.21.15.107:0/2465908035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8802c950 con 0x7f1a9006c5b0 2023-11-24T08:22:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.333+0000 7f1a87fff640 1 -- 172.21.15.107:0/2465908035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8802cab0 con 0x7f1a9006c5b0 2023-11-24T08:22:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.333+0000 7f1a87fff640 1 -- 172.21.15.107:0/2465908035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8802ce40 con 0x7f1a9006c5b0 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.333+0000 7f1a96609640 1 -- 172.21.15.107:0/2465908035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9006c5b0 msgr2=0x7f1a9006ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.333+0000 7f1a96609640 1 --2- 172.21.15.107:0/2465908035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9006c5b0 0x7f1a9006ca40 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a8802bdb0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 -- 172.21.15.107:0/2465908035 shutdown_connections 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 --2- 172.21.15.107:0/2465908035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9006c5b0 0x7f1a9006ca40 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 --2- 172.21.15.107:0/2465908035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a900b0b40 0x7f1a900b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 -- 172.21.15.107:0/2465908035 >> 172.21.15.107:0/2465908035 conn(0x7f1a900cbb40 msgr2=0x7f1a900cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 -- 172.21.15.107:0/2465908035 shutdown_connections 2023-11-24T08:22:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.334+0000 7f1a96609640 1 -- 172.21.15.107:0/2465908035 wait complete. 2023-11-24T08:22:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.335+0000 7f1a96609640 1 Processor -- start 2023-11-24T08:22:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 -- start start 2023-11-24T08:22:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a9006c5b0 0x7f1a9015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a9015b260 0x7f1a901e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a901e6030 con 0x7f1a900b0b40 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90159f40 con 0x7f1a9006c5b0 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.336+0000 7f1a96609640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a900b3000 con 0x7f1a9015b260 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49278/0 (socket says 172.21.15.107:49278) 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 -- 172.21.15.107:0/4247374193 learned_addr learned my addr 172.21.15.107:0/4247374193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 -- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a9015b260 msgr2=0x7f1a901e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a9015b260 0x7f1a901e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 -- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a9006c5b0 msgr2=0x7f1a9015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a95607640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a9006c5b0 0x7f1a9015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a9006c5b0 0x7f1a9015e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 -- 172.21.15.107:0/4247374193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a88009b30 con 0x7f1a900b0b40 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.337+0000 7f1a94e06640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b410 tx=0x7f1a8000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.338+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80011830 con 0x7f1a900b0b40 2023-11-24T08:22:43.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.338+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a800119b0 con 0x7f1a900b0b40 2023-11-24T08:22:43.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.338+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80011d60 con 0x7f1a900b0b40 2023-11-24T08:22:43.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.338+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a901e5a70 con 0x7f1a900b0b40 2023-11-24T08:22:43.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.338+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a900b32a0 con 0x7f1a900b0b40 2023-11-24T08:22:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.339+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9004e840 con 0x7f1a900b0b40 2023-11-24T08:22:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.340+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f1a8001a400 con 0x7f1a900b0b40 2023-11-24T08:22:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.341+0000 7f1a85ffb640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 0x7f1a7406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.341+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a800888c0 con 0x7f1a900b0b40 2023-11-24T08:22:43.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.341+0000 7f1a95607640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 0x7f1a7406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.342+0000 7f1a95607640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 0x7f1a7406f550 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00a750 tx=0x7f1a8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.345+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a80055470 con 0x7f1a900b0b40 2023-11-24T08:22:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.534+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a900ce640 con 0x7f1a900b0b40 2023-11-24T08:22:43.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.534+0000 7f1a85ffb640 1 -- 172.21.15.107:0/4247374193 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v212) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a80015020 con 0x7f1a900b0b40 2023-11-24T08:22:43.536 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:22:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 msgr2=0x7f1a7406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 0x7f1a7406f550 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00a750 tx=0x7f1a8c008040 comp rx=0 tx=0).stop 2023-11-24T08:22:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 msgr2=0x7f1a90159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b410 tx=0x7f1a8000b8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 shutdown_connections 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1a7406d090 0x7f1a7406f550 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a9015b260 0x7f1a901e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a900b0b40 0x7f1a90159a00 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 --2- 172.21.15.107:0/4247374193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a9006c5b0 0x7f1a9015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 >> 172.21.15.107:0/4247374193 conn(0x7f1a900cbb40 msgr2=0x7f1a900ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 shutdown_connections 2023-11-24T08:22:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.537+0000 7f1a96609640 1 -- 172.21.15.107:0/4247374193 wait complete. 2023-11-24T08:22:43.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone' 2023-11-24T08:22:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.771+0000 7fe39dae7640 1 Processor -- start 2023-11-24T08:22:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.772+0000 7fe39dae7640 1 -- start start 2023-11-24T08:22:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe39dae7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe39815df80 0x7fe3980ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe39dae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39815e3f0 con 0x7fe3980b2250 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe39dae7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39815ab80 con 0x7fe3980cd470 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe39dae7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39815a990 con 0x7fe39815df80 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe38ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe39815df80 0x7fe3980ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe3977fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3980b2250 0x7fe3980b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52984/0 (socket says 172.21.15.107:52984) 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.773+0000 7fe3977fe640 1 -- 172.21.15.107:0/276906684 learned_addr learned my addr 172.21.15.107:0/276906684 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.774+0000 7fe38ffff640 1 -- 172.21.15.107:0/276906684 >> v1:172.21.15.120:6789/0 conn(0x7fe3980cd470 legacy=0x7fe39815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.774+0000 7fe38ffff640 1 -- 172.21.15.107:0/276906684 >> v1:172.21.15.107:6789/0 conn(0x7fe3980b2250 legacy=0x7fe3980b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.774+0000 7fe38ffff640 1 -- 172.21.15.107:0/276906684 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980b2d40 con 0x7fe39815df80 2023-11-24T08:22:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.775+0000 7fe38ffff640 1 --2- 172.21.15.107:0/276906684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe3980ccf30 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc8776d6511490f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.775+0000 7fe396ffd640 1 -- 172.21.15.107:0/276906684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38002f8a0 con 0x7fe39815df80 2023-11-24T08:22:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.775+0000 7fe396ffd640 1 -- 172.21.15.107:0/276906684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38002fa00 con 0x7fe39815df80 2023-11-24T08:22:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.775+0000 7fe396ffd640 1 -- 172.21.15.107:0/276906684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38002fd90 con 0x7fe39815df80 2023-11-24T08:22:43.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 -- 172.21.15.107:0/276906684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 msgr2=0x7fe3980ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 --2- 172.21.15.107:0/276906684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe3980ccf30 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002dae0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 -- 172.21.15.107:0/276906684 shutdown_connections 2023-11-24T08:22:43.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 --2- 172.21.15.107:0/276906684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe3980ccf30 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 -- 172.21.15.107:0/276906684 >> 172.21.15.107:0/276906684 conn(0x7fe39806a920 msgr2=0x7fe39806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 -- 172.21.15.107:0/276906684 shutdown_connections 2023-11-24T08:22:43.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.776+0000 7fe39dae7640 1 -- 172.21.15.107:0/276906684 wait complete. 2023-11-24T08:22:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.778+0000 7fe39dae7640 1 Processor -- start 2023-11-24T08:22:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.778+0000 7fe39dae7640 1 -- start start 2023-11-24T08:22:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3980b2250 0x7fe39814b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3980cd470 0x7fe3980ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3981525a0 con 0x7fe39815df80 2023-11-24T08:22:43.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe398151fe0 con 0x7fe3980b2250 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe39dae7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3981522c0 con 0x7fe3980cd470 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49300/0 (socket says 172.21.15.107:49300) 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.779+0000 7fe397fff640 1 -- 172.21.15.107:0/3443068570 learned_addr learned my addr 172.21.15.107:0/3443068570 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe3977fe640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3980b2250 0x7fe39814b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 -- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3980cd470 msgr2=0x7fe3980ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3980cd470 0x7fe3980ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 -- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3980b2250 msgr2=0x7fe39814b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3980b2250 0x7fe39814b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 -- 172.21.15.107:0/3443068570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380009400 con 0x7fe39815df80 2023-11-24T08:22:43.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe397fff640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c780 tx=0x7fe39000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390003d80 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39000ce30 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900052b0 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.780+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3980aec80 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.781+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39814b810 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.782+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3980af1a0 con 0x7fe39815df80 2023-11-24T08:22:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.783+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe390005410 con 0x7fe39815df80 2023-11-24T08:22:43.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.784+0000 7fe394ff9640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 0x7fe37806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:43.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.784+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe390087c20 con 0x7fe39815df80 2023-11-24T08:22:43.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.784+0000 7fe3977fe640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 0x7fe37806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:43.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.785+0000 7fe3977fe640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 0x7fe37806f550 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe388000f60 tx=0x7fe388009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:43.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:43.788+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3980af1a0 con 0x7fe39815df80 2023-11-24T08:22:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.048+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone"]} v 0) v1 -- 0x7fe3980af1a0 con 0x7fe39815df80 2023-11-24T08:22:44.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.672+0000 7fe394ff9640 1 -- 172.21.15.107:0/3443068570 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone"]}]=0 v901) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe3980af1a0 con 0x7fe39815df80 2023-11-24T08:22:44.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.675+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 msgr2=0x7fe37806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.675+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 0x7fe37806f550 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe388000f60 tx=0x7fe388009040 comp rx=0 tx=0).stop 2023-11-24T08:22:44.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.675+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 msgr2=0x7fe39814c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.675+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c780 tx=0x7fe39000cc40 comp rx=0 tx=0).stop 2023-11-24T08:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 shutdown_connections 2023-11-24T08:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe37806d090 0x7fe37806f550 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe39815df80 0x7fe39814c230 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3980cd470 0x7fe3980ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 --2- 172.21.15.107:0/3443068570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3980b2250 0x7fe39814b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 >> 172.21.15.107:0/3443068570 conn(0x7fe39806a920 msgr2=0x7fe3980cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 shutdown_connections 2023-11-24T08:22:44.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.676+0000 7fe39dae7640 1 -- 172.21.15.107:0/3443068570 wait complete. 2023-11-24T08:22:44.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:22:44.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.915+0000 7f4348fb3640 1 Processor -- start 2023-11-24T08:22:44.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.915+0000 7f4348fb3640 1 -- start start 2023-11-24T08:22:44.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4348fb3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f434415d5f0 0x7f43440ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4348fb3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f434406d080 0x7f43440ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4348fb3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43440cdf70 con 0x7f43440ac840 2023-11-24T08:22:44.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4348fb3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43440cb3d0 con 0x7f434406d080 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4348fb3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43440cbfa0 con 0x7f434415d5f0 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f433bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f43440ac840 0x7f434406c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52998/0 (socket says 172.21.15.107:52998) 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f433bfff640 1 -- 172.21.15.107:0/3272795853 learned_addr learned my addr 172.21.15.107:0/3272795853 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.917+0000 7f4343fff640 1 --2- 172.21.15.107:0/3272795853 >> v2:172.21.15.107:3300/0 conn(0x7f434406d080 0x7f43440ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f43437fe640 1 --2- 172.21.15.107:0/3272795853 >> v2:172.21.15.120:3300/0 conn(0x7f434415d5f0 0x7f43440ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f4342ffd640 1 -- 172.21.15.107:0/3272795853 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216145206 0 0) 0x7f43440cdf70 con 0x7f43440ac840 2023-11-24T08:22:44.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f4343fff640 1 -- 172.21.15.107:0/3272795853 >> v2:172.21.15.120:3300/0 conn(0x7f434415d5f0 msgr2=0x7f43440ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:44.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f4343fff640 1 --2- 172.21.15.107:0/3272795853 >> v2:172.21.15.120:3300/0 conn(0x7f434415d5f0 0x7f43440ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f4343fff640 1 -- 172.21.15.107:0/3272795853 >> v1:172.21.15.107:6789/0 conn(0x7f43440ac840 legacy=0x7f434406c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.918+0000 7f4343fff640 1 -- 172.21.15.107:0/3272795853 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43440cc280 con 0x7f434406d080 2023-11-24T08:22:44.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.919+0000 7f4343fff640 1 --2- 172.21.15.107:0/3272795853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434406d080 0x7f43440ce9c0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f43440aea00 tx=0x7f433c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3f13864739e664 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:44.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.919+0000 7f4342ffd640 1 -- 172.21.15.107:0/3272795853 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c02cbf0 con 0x7f434406d080 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.919+0000 7f4342ffd640 1 -- 172.21.15.107:0/3272795853 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c02cd50 con 0x7f434406d080 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.919+0000 7f4342ffd640 1 -- 172.21.15.107:0/3272795853 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c034660 con 0x7f434406d080 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 -- 172.21.15.107:0/3272795853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434406d080 msgr2=0x7f43440ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 --2- 172.21.15.107:0/3272795853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434406d080 0x7f43440ce9c0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f43440aea00 tx=0x7f433c02be40 comp rx=0 tx=0).stop 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 -- 172.21.15.107:0/3272795853 shutdown_connections 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 --2- 172.21.15.107:0/3272795853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434406d080 0x7f43440ce9c0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 --2- 172.21.15.107:0/3272795853 >> v2:172.21.15.120:3300/0 conn(0x7f434415d5f0 0x7f43440ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 -- 172.21.15.107:0/3272795853 >> 172.21.15.107:0/3272795853 conn(0x7f434406a500 msgr2=0x7f434406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 -- 172.21.15.107:0/3272795853 shutdown_connections 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.920+0000 7f4348fb3640 1 -- 172.21.15.107:0/3272795853 wait complete. 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.922+0000 7f4348fb3640 1 Processor -- start 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.922+0000 7f4348fb3640 1 -- start start 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f434406d080 0x7f434414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 0x7f43440af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43440afb60 con 0x7f434415d5f0 2023-11-24T08:22:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434414f8e0 con 0x7f43440ac840 2023-11-24T08:22:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4348fb3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434414fbc0 con 0x7f434406d080 2023-11-24T08:22:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4343fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4343fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49316/0 (socket says 172.21.15.107:49316) 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4343fff640 1 -- 172.21.15.107:0/1541501643 learned_addr learned my addr 172.21.15.107:0/1541501643 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f43437fe640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f434406d080 0x7f434414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f433bfff640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 0x7f43440af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4343fff640 1 -- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f434406d080 msgr2=0x7f434414f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.923+0000 7f4343fff640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f434406d080 0x7f434414f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4343fff640 1 -- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 msgr2=0x7f43440af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4343fff640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 0x7f43440af620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4343fff640 1 -- 172.21.15.107:0/1541501643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c009a70 con 0x7f434415d5f0 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f433bfff640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 0x7f43440af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4343fff640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f433c009e80 tx=0x7f433c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c040650 con 0x7f434415d5f0 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c0407d0 con 0x7f434415d5f0 2023-11-24T08:22:44.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c040b80 con 0x7f434415d5f0 2023-11-24T08:22:44.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434414fea0 con 0x7f434415d5f0 2023-11-24T08:22:44.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.924+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43440b03f0 con 0x7f434415d5f0 2023-11-24T08:22:44.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.926+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43440ae7a0 con 0x7f434415d5f0 2023-11-24T08:22:44.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.926+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f433c040ce0 con 0x7f434415d5f0 2023-11-24T08:22:44.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.927+0000 7f4340ff9640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 0x7f432406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:44.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.927+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f433c0ab560 con 0x7f434415d5f0 2023-11-24T08:22:44.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.928+0000 7f43437fe640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 0x7f432406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:44.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.928+0000 7f43437fe640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 0x7f432406f550 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:44.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:44.931+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f433c078110 con 0x7f434415d5f0 2023-11-24T08:22:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.121+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f434406c1a0 con 0x7f434415d5f0 2023-11-24T08:22:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.128+0000 7f4340ff9640 1 -- 172.21.15.107:0/1541501643 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v53)=0 v53) v1 ==== 122+0+0 (secure 0 0 0) 0x7f433c07b730 con 0x7f434415d5f0 2023-11-24T08:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 msgr2=0x7f432406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 0x7f432406f550 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334008040 comp rx=0 tx=0).stop 2023-11-24T08:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 msgr2=0x7f434414d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f433c009e80 tx=0x7f433c004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 shutdown_connections 2023-11-24T08:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f432406d090 0x7f432406f550 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f434415d5f0 0x7f434414d960 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43440ac840 0x7f43440af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 --2- 172.21.15.107:0/1541501643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f434406d080 0x7f434414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 >> 172.21.15.107:0/1541501643 conn(0x7f434406a500 msgr2=0x7f4344157eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.131+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 shutdown_connections 2023-11-24T08:22:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.132+0000 7f4348fb3640 1 -- 172.21.15.107:0/1541501643 wait complete. 2023-11-24T08:22:45.154 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:22:45.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:22:45.253 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:22:45.375 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:22:45.375 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:45.375 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:45.431 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:22:45.431 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:45.432 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:45.490 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:22:45.490 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:45.490 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:45.550 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:22:45.550 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:22:45.550 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:45.613 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:22:45.614 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:22:45.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:45.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.837+0000 7ff07d797640 1 Processor -- start 2023-11-24T08:22:45.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.837+0000 7ff07d797640 1 -- start start 2023-11-24T08:22:45.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff07d797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff07815b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff07d797640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07815e3f0 con 0x7ff07815df80 2023-11-24T08:22:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff07d797640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07815ab80 con 0x7ff0780b2250 2023-11-24T08:22:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff07d797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07815a990 con 0x7ff07815b080 2023-11-24T08:22:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff06e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff07815df80 0x7ff0780ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53006/0 (socket says 172.21.15.107:53006) 2023-11-24T08:22:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff06e7fc640 1 -- 172.21.15.107:0/1089993239 learned_addr learned my addr 172.21.15.107:0/1089993239 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff0777fe640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff07815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771909406 0 0) 0x7ff07815e3f0 con 0x7ff07815df80 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.839+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff064003520 con 0x7ff07815df80 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3031276683 0 0) 0x7ff064003520 con 0x7ff07815df80 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 msgr2=0x7ff07815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff07815b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 >> v1:172.21.15.120:6789/0 conn(0x7ff0780b2250 legacy=0x7ff0780b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0780b2d40 con 0x7ff07815df80 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7ff060001670 con 0x7ff07815df80 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 >> v1:172.21.15.107:6789/0 conn(0x7ff07815df80 legacy=0x7ff0780ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640039a0 0x7ff064003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0640047c0 0x7ff064004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff064005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06400ba30 con 0x7ff0640047c0 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06400bba0 con 0x7ff0640039a0 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06400bd60 con 0x7ff07815b080 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.840+0000 7ff06e7fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0640047c0 0x7ff064004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff0777fe640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff064005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff06e7fc640 1 -- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 msgr2=0x7ff064005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff06e7fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07815b080 0x7ff064005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff06e7fc640 1 -- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640039a0 msgr2=0x7ff064003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff06e7fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640039a0 0x7ff064003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.841+0000 7ff06e7fc640 1 -- 172.21.15.107:0/1089993239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064003520 con 0x7ff0640047c0 2023-11-24T08:22:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.842+0000 7ff06e7fc640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0640047c0 0x7ff064004c40 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fb0 tx=0x7ff0640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7c4570ff77b224 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.842+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060005ad0 con 0x7ff0640047c0 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.842+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff060005c50 con 0x7ff0640047c0 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.842+0000 7ff0767fc640 1 -- 172.21.15.107:0/1089993239 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06002c790 con 0x7ff0640047c0 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 -- 172.21.15.107:0/1089993239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0640047c0 msgr2=0x7ff064004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 --2- 172.21.15.107:0/1089993239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0640047c0 0x7ff064004c40 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fb0 tx=0x7ff0640057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff0777fe640 1 -- 172.21.15.107:0/1089993239 reap_dead start 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 -- 172.21.15.107:0/1089993239 shutdown_connections 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 -- 172.21.15.107:0/1089993239 >> 172.21.15.107:0/1089993239 conn(0x7ff07806a920 msgr2=0x7ff07806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 -- 172.21.15.107:0/1089993239 shutdown_connections 2023-11-24T08:22:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.843+0000 7ff07d797640 1 -- 172.21.15.107:0/1089993239 wait complete. 2023-11-24T08:22:45.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.844+0000 7ff07d797640 1 Processor -- start 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.845+0000 7ff07d797640 1 -- start start 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 0x7ff078157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07814f080 0x7ff0781f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0781f7c90 con 0x7ff0781f72d0 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0780afd30 con 0x7ff0640047c0 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff07d797640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0780718b0 con 0x7ff07814f080 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49338/0 (socket says 172.21.15.107:49338) 2023-11-24T08:22:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 -- 172.21.15.107:0/24418879 learned_addr learned my addr 172.21.15.107:0/24418879 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff06e7fc640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07814f080 0x7ff0781f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff076ffd640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 0x7ff078157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 -- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07814f080 msgr2=0x7ff0781f6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07814f080 0x7ff0781f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 -- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 msgr2=0x7ff078157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 0x7ff078157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff0777fe640 1 -- 172.21.15.107:0/24418879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07814faa0 con 0x7ff0781f72d0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.846+0000 7ff076ffd640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 0x7ff078157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff0777fe640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7ff070007d10 tx=0x7ff0700115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000e070 con 0x7ff0781f72d0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070011e40 con 0x7ff0781f72d0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070003990 con 0x7ff0781f72d0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0780af4d0 con 0x7ff0781f72d0 2023-11-24T08:22:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.847+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0780afa50 con 0x7ff0781f72d0 2023-11-24T08:22:45.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.848+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0780ae960 con 0x7ff0781f72d0 2023-11-24T08:22:45.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.849+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7ff070003af0 con 0x7ff0781f72d0 2023-11-24T08:22:45.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.850+0000 7ff06ffff640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 0x7ff05806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:45.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.850+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff070088e60 con 0x7ff0781f72d0 2023-11-24T08:22:45.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.851+0000 7ff076ffd640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 0x7ff05806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:45.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.851+0000 7ff076ffd640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 0x7ff05806f550 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff0680016b0 tx=0x7ff068009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:45.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:45.855+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff070059400 con 0x7ff0781f72d0 2023-11-24T08:22:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.082+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff078067470 con 0x7ff0781f72d0 2023-11-24T08:22:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.083+0000 7ff06ffff640 1 -- 172.21.15.107:0/24418879 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+3718 (secure 0 0 0) 0x7ff070058f90 con 0x7ff0781f72d0 2023-11-24T08:22:46.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:46.085 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":131,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/2655001231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2655001231},{"type":"v1","addr":"172.21.15.120:6834","nonce":2655001231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7594,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3450009024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3450009024},{"type":"v1","addr":"172.21.15.107:6835","nonce":3450009024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":11643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2876713573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":2876713573},{"type":"v1","addr":"172.21.15.120:6835","nonce":2876713573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":12114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1217949304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1217949304},{"type":"v1","addr":"172.21.15.107:6837","nonce":1217949304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[]} 2023-11-24T08:22:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 131 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 msgr2=0x7ff05806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 0x7ff05806f550 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff0680016b0 tx=0x7ff068009040 comp rx=0 tx=0).stop 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 msgr2=0x7ff0781f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7ff070007d10 tx=0x7ff0700115a0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 shutdown_connections 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff05806d090 0x7ff05806f550 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0781f72d0 0x7ff0781f7750 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff07814f080 0x7ff0781f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 --2- 172.21.15.107:0/24418879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0640047c0 0x7ff078157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 >> 172.21.15.107:0/24418879 conn(0x7ff07806a920 msgr2=0x7ff0780adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 shutdown_connections 2023-11-24T08:22:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.086+0000 7ff07d797640 1 -- 172.21.15.107:0/24418879 wait complete. 2023-11-24T08:22:46.108 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:22:46.108 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:22:46.108 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:22:46.108 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:22:46.108 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:22:46.109 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:22:46.109 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:22:46.109 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:22:46.109 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:22:46.109 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:22:46.109 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:22:46.110 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:22:46.110 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:22:46.110 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:22:46.110 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:22:46.110 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:22:46.127 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:22:46.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:22:46.129 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:22:46.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:22:46.138 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:22:46.138 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:22:46.139 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:22:46.139 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:22:46.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:22:46.178 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:22:46.179 DEBUG:teuthology.parallel:result is None 2023-11-24T08:22:46.186 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:22:46.186 DEBUG:teuthology.parallel:result is None 2023-11-24T08:22:46.186 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:22:46.186 DEBUG:teuthology.parallel:result is None 2023-11-24T08:22:46.192 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:22:46.193 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:22:46.194 DEBUG:teuthology.parallel:result is None 2023-11-24T08:22:46.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:22:46.335 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:22:46.346 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:22:46.347 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:22:46.348 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:22:46.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.423+0000 7f9200e7e640 1 Processor -- start 2023-11-24T08:22:46.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.423+0000 7f9200e7e640 1 -- start start 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.424+0000 7f9200e7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91fc0b63d0 0x7f91fc0d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.424+0000 7f9200e7e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91fc0ce270 0x7f91fc0d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.424+0000 7f9200e7e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91fc0cea30 con 0x7f91fc15d490 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.424+0000 7f9200e7e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc0d0450 con 0x7f91fc0b63d0 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.424+0000 7f9200e7e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc0d0bf0 con 0x7f91fc0ce270 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91fc0b63d0 0x7f91fc0d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91fc0b63d0 0x7f91fc0d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49368/0 (socket says 172.21.15.107:49368) 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 -- 172.21.15.107:0/2852002637 learned_addr learned my addr 172.21.15.107:0/2852002637 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91fbfff640 1 --2- 172.21.15.107:0/2852002637 >> v2:172.21.15.120:3300/0 conn(0x7f91fc0ce270 0x7f91fc0d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 -- 172.21.15.107:0/2852002637 >> v2:172.21.15.120:3300/0 conn(0x7f91fc0ce270 msgr2=0x7f91fc0d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 --2- 172.21.15.107:0/2852002637 >> v2:172.21.15.120:3300/0 conn(0x7f91fc0ce270 0x7f91fc0d3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 -- 172.21.15.107:0/2852002637 >> v1:172.21.15.120:6789/0 conn(0x7f91fc15d490 legacy=0x7f91fc0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:46.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.425+0000 7f91f3fff640 1 -- 172.21.15.107:0/2852002637 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc0d2260 con 0x7f91fc0b63d0 2023-11-24T08:22:46.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.426+0000 7f91f3fff640 1 --2- 172.21.15.107:0/2852002637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc0b63d0 0x7f91fc0d3230 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f91fc161f30 tx=0x7f91e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a44306c5c7fde3c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.426+0000 7f91faffd640 1 -- 172.21.15.107:0/2852002637 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4031870 con 0x7f91fc0b63d0 2023-11-24T08:22:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f91faffd640 1 -- 172.21.15.107:0/2852002637 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e40319d0 con 0x7f91fc0b63d0 2023-11-24T08:22:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f91faffd640 1 -- 172.21.15.107:0/2852002637 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4031d60 con 0x7f91fc0b63d0 2023-11-24T08:22:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 -- 172.21.15.107:0/2852002637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc0b63d0 msgr2=0x7f91fc0d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2852002637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc0b63d0 0x7f91fc0d3230 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f91fc161f30 tx=0x7f91e402b910 comp rx=0 tx=0).stop 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 -- 172.21.15.107:0/2852002637 shutdown_connections 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2852002637 >> v2:172.21.15.120:3300/0 conn(0x7f91fc0ce270 0x7f91fc0d3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2852002637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc0b63d0 0x7f91fc0d3230 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 -- 172.21.15.107:0/2852002637 >> 172.21.15.107:0/2852002637 conn(0x7f91fc061340 msgr2=0x7f91fc061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 -- 172.21.15.107:0/2852002637 shutdown_connections 2023-11-24T08:22:46.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.427+0000 7f9200e7e640 1 -- 172.21.15.107:0/2852002637 wait complete. 2023-11-24T08:22:46.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.429+0000 7f9200e7e640 1 Processor -- start 2023-11-24T08:22:46.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.429+0000 7f9200e7e640 1 -- start start 2023-11-24T08:22:46.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91fc0ce270 0x7f91fc160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91fc15d490 0x7f91fc0b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0b8ce0 con 0x7f91fc155a80 2023-11-24T08:22:46.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0b84a0 con 0x7f91fc0ce270 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f9200e7e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0b8720 con 0x7f91fc15d490 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f91fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f91fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49372/0 (socket says 172.21.15.107:49372) 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f91fbfff640 1 -- 172.21.15.107:0/2881459710 learned_addr learned my addr 172.21.15.107:0/2881459710 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.430+0000 7f91fb7fe640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91fc0ce270 0x7f91fc160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91f3fff640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91fc15d490 0x7f91fc0b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 -- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91fc15d490 msgr2=0x7f91fc0b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91fc15d490 0x7f91fc0b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 -- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91fc0ce270 msgr2=0x7f91fc160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91fc0ce270 0x7f91fc160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 -- 172.21.15.107:0/2881459710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40094d0 con 0x7f91fc155a80 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91fbfff640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f91f400ef90 tx=0x7f91f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4009ca0 con 0x7f91fc155a80 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f400f040 con 0x7f91fc155a80 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f40146c0 con 0x7f91fc155a80 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91fc0b8a00 con 0x7f91fc155a80 2023-11-24T08:22:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.431+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91fc0ca7f0 con 0x7f91fc155a80 2023-11-24T08:22:46.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.433+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91fc0ca370 con 0x7f91fc155a80 2023-11-24T08:22:46.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.434+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f91f4014820 con 0x7f91fc155a80 2023-11-24T08:22:46.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.435+0000 7f91f8ff9640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 0x7f91dc06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.435+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f91f40896a0 con 0x7f91fc155a80 2023-11-24T08:22:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.435+0000 7f91fb7fe640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 0x7f91dc06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.436+0000 7f91fb7fe640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 0x7f91dc06f550 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006fd0 tx=0x7f91ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.439+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91fc0ca370 con 0x7f91fc155a80 2023-11-24T08:22:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.644+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f91fc0ca370 con 0x7f91fc155a80 2023-11-24T08:22:46.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.696+0000 7f91f8ff9640 1 -- 172.21.15.107:0/2881459710 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v214) v1 ==== 97+0+0 (secure 0 0 0) 0x7f91fc0ca370 con 0x7f91fc155a80 2023-11-24T08:22:46.699 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.700+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 msgr2=0x7f91dc06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.700+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 0x7f91dc06f550 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006fd0 tx=0x7f91ec008040 comp rx=0 tx=0).stop 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.700+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 msgr2=0x7f91fc0cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.700+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f91f400ef90 tx=0x7f91f400c490 comp rx=0 tx=0).stop 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 shutdown_connections 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f91dc06d090 0x7f91dc06f550 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91fc155a80 0x7f91fc0cad10 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91fc15d490 0x7f91fc0b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 --2- 172.21.15.107:0/2881459710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91fc0ce270 0x7f91fc160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 >> 172.21.15.107:0/2881459710 conn(0x7f91fc061340 msgr2=0x7f91fc0d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 shutdown_connections 2023-11-24T08:22:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.701+0000 7f9200e7e640 1 -- 172.21.15.107:0/2881459710 wait complete. 2023-11-24T08:22:46.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:22:46.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.921+0000 7f3e753b0640 1 Processor -- start 2023-11-24T08:22:46.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.921+0000 7f3e753b0640 1 -- start start 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e753b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e700d27a0 0x7f3e700d2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e753b0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 0x7f3e700b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e753b0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e700d3120 con 0x7f3e7015d520 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e753b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e700b6e50 con 0x7f3e700d27a0 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e753b0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e700b2930 con 0x7f3e700b64d0 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e667fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e700d27a0 0x7f3e700d2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e667fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e700d27a0 0x7f3e700d2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49376/0 (socket says 172.21.15.107:49376) 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.923+0000 7f3e667fc640 1 -- 172.21.15.107:0/695795856 learned_addr learned my addr 172.21.15.107:0/695795856 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e6f7fe640 1 --2- 172.21.15.107:0/695795856 >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 0x7f3e700b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e667fc640 1 -- 172.21.15.107:0/695795856 >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 msgr2=0x7f3e700b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e667fc640 1 --2- 172.21.15.107:0/695795856 >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 0x7f3e700b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e667fc640 1 -- 172.21.15.107:0/695795856 >> v1:172.21.15.120:6789/0 conn(0x7f3e7015d520 legacy=0x7f3e700b8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e667fc640 1 -- 172.21.15.107:0/695795856 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700d1420 con 0x7f3e700d27a0 2023-11-24T08:22:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.924+0000 7f3e6f7fe640 1 --2- 172.21.15.107:0/695795856 >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 0x7f3e700b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.925+0000 7f3e667fc640 1 --2- 172.21.15.107:0/695795856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e700d27a0 0x7f3e700d2be0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f3e700d0eb0 tx=0x7f3e5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3242d694bb20fced server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.925+0000 7f3e6e7fc640 1 -- 172.21.15.107:0/695795856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58030860 con 0x7f3e700d27a0 2023-11-24T08:22:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.925+0000 7f3e6e7fc640 1 -- 172.21.15.107:0/695795856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e580309c0 con 0x7f3e700d27a0 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.925+0000 7f3e6e7fc640 1 -- 172.21.15.107:0/695795856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58030d50 con 0x7f3e700d27a0 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 -- 172.21.15.107:0/695795856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e700d27a0 msgr2=0x7f3e700d2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 --2- 172.21.15.107:0/695795856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e700d27a0 0x7f3e700d2be0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f3e700d0eb0 tx=0x7f3e5802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 -- 172.21.15.107:0/695795856 shutdown_connections 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 --2- 172.21.15.107:0/695795856 >> v2:172.21.15.120:3300/0 conn(0x7f3e700b64d0 0x7f3e700b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 --2- 172.21.15.107:0/695795856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e700d27a0 0x7f3e700d2be0 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 -- 172.21.15.107:0/695795856 >> 172.21.15.107:0/695795856 conn(0x7f3e70061340 msgr2=0x7f3e70061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 -- 172.21.15.107:0/695795856 shutdown_connections 2023-11-24T08:22:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.926+0000 7f3e753b0640 1 -- 172.21.15.107:0/695795856 wait complete. 2023-11-24T08:22:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.928+0000 7f3e753b0640 1 Processor -- start 2023-11-24T08:22:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.928+0000 7f3e753b0640 1 -- start start 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e700b64d0 0x7f3e701e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 0x7f3e7014b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7015d900 con 0x7f3e7015d520 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7015c550 con 0x7f3e700d27a0 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e753b0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7014bd00 con 0x7f3e700b64d0 2023-11-24T08:22:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e6f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e6f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49384/0 (socket says 172.21.15.107:49384) 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.929+0000 7f3e6f7fe640 1 -- 172.21.15.107:0/3159352759 learned_addr learned my addr 172.21.15.107:0/3159352759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e667fc640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 0x7f3e7014b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 -- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e700b64d0 msgr2=0x7f3e701e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e700b64d0 0x7f3e701e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 -- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 msgr2=0x7f3e7014b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 0x7f3e7014b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 -- 172.21.15.107:0/3159352759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580094d0 con 0x7f3e7015d520 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e667fc640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 0x7f3e7014b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e6f7fe640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800cf20 tx=0x7f3e6800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800d840 con 0x7f3e7015d520 2023-11-24T08:22:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6800d9a0 con 0x7f3e7015d520 2023-11-24T08:22:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e70160350 con 0x7f3e7015d520 2023-11-24T08:22:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800dd50 con 0x7f3e7015d520 2023-11-24T08:22:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.930+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7015db50 con 0x7f3e7015d520 2023-11-24T08:22:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.931+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7004e8b0 con 0x7f3e7015d520 2023-11-24T08:22:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.932+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f3e6801e3b0 con 0x7f3e7015d520 2023-11-24T08:22:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.933+0000 7f3e67fff640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 0x7f3e5006f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.933+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3e68089260 con 0x7f3e7015d520 2023-11-24T08:22:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.933+0000 7f3e6effd640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 0x7f3e5006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.934+0000 7f3e6effd640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 0x7f3e5006f580 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:46.937+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e68055d70 con 0x7f3e7015d520 2023-11-24T08:22:47.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.198+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3e700b7620 con 0x7f3e7015d520 2023-11-24T08:22:47.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.199+0000 7f3e67fff640 1 -- 172.21.15.107:0/3159352759 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3e6801a070 con 0x7f3e7015d520 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:22:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:22:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:22:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:22:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:22:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:22:47.216 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.201+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 msgr2=0x7f3e5006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.201+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 0x7f3e5006f580 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006bc0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.201+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 msgr2=0x7f3e7006b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.201+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800cf20 tx=0x7f3e6800a7d0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 shutdown_connections 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3e5006d0c0 0x7f3e5006f580 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e7015d520 0x7f3e7006b4c0 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e700d27a0 0x7f3e7014b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 --2- 172.21.15.107:0/3159352759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e700b64d0 0x7f3e701e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 >> 172.21.15.107:0/3159352759 conn(0x7f3e70061340 msgr2=0x7f3e70061720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 shutdown_connections 2023-11-24T08:22:47.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.202+0000 7f3e753b0640 1 -- 172.21.15.107:0/3159352759 wait complete. 2023-11-24T08:22:47.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.425+0000 7fc06759e640 1 Processor -- start 2023-11-24T08:22:47.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.425+0000 7fc06759e640 1 -- start start 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.426+0000 7fc06759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc0680d1e20 0x7fc0680cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.426+0000 7fc06759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0680d0070 0x7fc0680b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.426+0000 7fc06759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680d04b0 con 0x7fc06815d4e0 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.426+0000 7fc06759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680d4110 con 0x7fc0680d0070 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.426+0000 7fc06759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680ce990 con 0x7fc0680d1e20 2023-11-24T08:22:47.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc06659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc06815d4e0 0x7fc0680d38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53024/0 (socket says 172.21.15.107:53024) 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc06659c640 1 -- 172.21.15.107:0/2308579479 learned_addr learned my addr 172.21.15.107:0/2308579479 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc066d9d640 1 --2- 172.21.15.107:0/2308579479 >> v2:172.21.15.107:3300/0 conn(0x7fc0680d0070 0x7fc0680b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc065d9b640 1 --2- 172.21.15.107:0/2308579479 >> v2:172.21.15.120:3300/0 conn(0x7fc0680d1e20 0x7fc0680cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc06559a640 1 -- 172.21.15.107:0/2308579479 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918261407 0 0) 0x7fc0680d04b0 con 0x7fc06815d4e0 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc06559a640 1 -- 172.21.15.107:0/2308579479 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc054003520 con 0x7fc06815d4e0 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc066d9d640 1 -- 172.21.15.107:0/2308579479 >> v2:172.21.15.120:3300/0 conn(0x7fc0680d1e20 msgr2=0x7fc0680cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc066d9d640 1 --2- 172.21.15.107:0/2308579479 >> v2:172.21.15.120:3300/0 conn(0x7fc0680d1e20 0x7fc0680cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc066d9d640 1 -- 172.21.15.107:0/2308579479 >> v1:172.21.15.107:6789/0 conn(0x7fc06815d4e0 legacy=0x7fc0680d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.427+0000 7fc066d9d640 1 -- 172.21.15.107:0/2308579479 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680cec70 con 0x7fc0680d0070 2023-11-24T08:22:47.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.428+0000 7fc066d9d640 1 --2- 172.21.15.107:0/2308579479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d0070 0x7fc0680b2e50 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fc0680b9280 tx=0x7fc06002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e9cce25170473 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06559a640 1 -- 172.21.15.107:0/2308579479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060031830 con 0x7fc0680d0070 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06559a640 1 -- 172.21.15.107:0/2308579479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060031990 con 0x7fc0680d0070 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06559a640 1 -- 172.21.15.107:0/2308579479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060031d20 con 0x7fc0680d0070 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 -- 172.21.15.107:0/2308579479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d0070 msgr2=0x7fc0680b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 --2- 172.21.15.107:0/2308579479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d0070 0x7fc0680b2e50 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fc0680b9280 tx=0x7fc06002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 -- 172.21.15.107:0/2308579479 shutdown_connections 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 --2- 172.21.15.107:0/2308579479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d0070 0x7fc0680b2e50 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 --2- 172.21.15.107:0/2308579479 >> v2:172.21.15.120:3300/0 conn(0x7fc0680d1e20 0x7fc0680cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.429+0000 7fc06759e640 1 -- 172.21.15.107:0/2308579479 >> 172.21.15.107:0/2308579479 conn(0x7fc06809c510 msgr2=0x7fc0680d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.430+0000 7fc06759e640 1 -- 172.21.15.107:0/2308579479 shutdown_connections 2023-11-24T08:22:47.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.430+0000 7fc06759e640 1 -- 172.21.15.107:0/2308579479 wait complete. 2023-11-24T08:22:47.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.431+0000 7fc06759e640 1 Processor -- start 2023-11-24T08:22:47.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.431+0000 7fc06759e640 1 -- start start 2023-11-24T08:22:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0680d0070 0x7fc0680c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 0x7fc06815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815eef0 con 0x7fc0680d1e20 2023-11-24T08:22:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680cb490 con 0x7fc06815d4e0 2023-11-24T08:22:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.432+0000 7fc06759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680cb770 con 0x7fc0680d0070 2023-11-24T08:22:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49406/0 (socket says 172.21.15.107:49406) 2023-11-24T08:22:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 -- 172.21.15.107:0/1521614824 learned_addr learned my addr 172.21.15.107:0/1521614824 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc066d9d640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 0x7fc06815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 -- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0680d0070 msgr2=0x7fc0680c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0680d0070 0x7fc0680c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 -- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 msgr2=0x7fc06815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 0x7fc06815fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 -- 172.21.15.107:0/1521614824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc060009400 con 0x7fc0680d1e20 2023-11-24T08:22:47.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc066d9d640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 0x7fc06815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc065d9b640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fc05000d800 tx=0x7fc05000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.433+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05000b6c0 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.434+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05000b820 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.434+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05000bbb0 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.434+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0680cba50 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.434+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0680b22e0 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.435+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc068067680 con 0x7fc0680d1e20 2023-11-24T08:22:47.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.436+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fc05000bd10 con 0x7fc0680d1e20 2023-11-24T08:22:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.437+0000 7fc05effd640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 0x7fc04806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.437+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc050088d80 con 0x7fc0680d1e20 2023-11-24T08:22:47.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.437+0000 7fc06659c640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 0x7fc04806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.438+0000 7fc06659c640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 0x7fc04806f550 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc058002000 tx=0x7fc0580015f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.441+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc050055890 con 0x7fc0680d1e20 2023-11-24T08:22:47.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.631+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc06804e840 con 0x7fc0680d1e20 2023-11-24T08:22:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.632+0000 7fc05effd640 1 -- 172.21.15.107:0/1521614824 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fc050058eb0 con 0x7fc0680d1e20 2023-11-24T08:22:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:47.635 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":216,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:46.708962+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 msgr2=0x7fc04806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 0x7fc04806f550 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc058002000 tx=0x7fc0580015f0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 msgr2=0x7fc06815e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fc05000d800 tx=0x7fc05000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 shutdown_connections 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc04806d090 0x7fc04806f550 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc06815d4e0 0x7fc06815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0680d1e20 0x7fc06815e450 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 --2- 172.21.15.107:0/1521614824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0680d0070 0x7fc0680c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 >> 172.21.15.107:0/1521614824 conn(0x7fc06809c510 msgr2=0x7fc0680d30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 shutdown_connections 2023-11-24T08:22:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.635+0000 7fc06759e640 1 -- 172.21.15.107:0/1521614824 wait complete. 2023-11-24T08:22:47.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:22:47.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:22:47.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.873+0000 7fedac63d640 1 Processor -- start 2023-11-24T08:22:47.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.873+0000 7fedac63d640 1 -- start start 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7fedac63d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feda415b440 0x7feda40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7fedac63d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feda40cbf20 0x7feda40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7fedac63d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda415eea0 con 0x7feda40c57e0 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7fedac63d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda40cc3d0 con 0x7feda40cbf20 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7fedac63d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda415b820 con 0x7feda415b440 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7feda9bd8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feda40cbf20 0x7feda40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7feda9bd8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feda40cbf20 0x7feda40c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49408/0 (socket says 172.21.15.107:49408) 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.875+0000 7feda9bd8640 1 -- 172.21.15.107:0/1232770429 learned_addr learned my addr 172.21.15.107:0/1232770429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:47.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.876+0000 7fedaa3d9640 1 --2- 172.21.15.107:0/1232770429 >> v2:172.21.15.120:3300/0 conn(0x7feda415b440 0x7feda40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.876+0000 7feda9bd8640 1 -- 172.21.15.107:0/1232770429 >> v2:172.21.15.120:3300/0 conn(0x7feda415b440 msgr2=0x7feda40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.876+0000 7feda9bd8640 1 --2- 172.21.15.107:0/1232770429 >> v2:172.21.15.120:3300/0 conn(0x7feda415b440 0x7feda40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.876+0000 7feda9bd8640 1 -- 172.21.15.107:0/1232770429 >> v1:172.21.15.107:6789/0 conn(0x7feda40c57e0 legacy=0x7feda40c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.876+0000 7feda9bd8640 1 -- 172.21.15.107:0/1232770429 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda406d600 con 0x7feda40cbf20 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.877+0000 7feda9bd8640 1 --2- 172.21.15.107:0/1232770429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40cbf20 0x7feda40c52a0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7feda406e770 tx=0x7fed9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f83cd296e1a382 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.877+0000 7feda93d7640 1 -- 172.21.15.107:0/1232770429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed94031870 con 0x7feda40cbf20 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.877+0000 7feda93d7640 1 -- 172.21.15.107:0/1232770429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed940319d0 con 0x7feda40cbf20 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.877+0000 7feda93d7640 1 -- 172.21.15.107:0/1232770429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed94031d60 con 0x7feda40cbf20 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 -- 172.21.15.107:0/1232770429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40cbf20 msgr2=0x7feda40c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 --2- 172.21.15.107:0/1232770429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40cbf20 0x7feda40c52a0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7feda406e770 tx=0x7fed9402b910 comp rx=0 tx=0).stop 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 -- 172.21.15.107:0/1232770429 shutdown_connections 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 --2- 172.21.15.107:0/1232770429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40cbf20 0x7feda40c52a0 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 --2- 172.21.15.107:0/1232770429 >> v2:172.21.15.120:3300/0 conn(0x7feda415b440 0x7feda40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 -- 172.21.15.107:0/1232770429 >> 172.21.15.107:0/1232770429 conn(0x7feda40b0c50 msgr2=0x7feda40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 -- 172.21.15.107:0/1232770429 shutdown_connections 2023-11-24T08:22:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.878+0000 7fedac63d640 1 -- 172.21.15.107:0/1232770429 wait complete. 2023-11-24T08:22:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.880+0000 7fedac63d640 1 Processor -- start 2023-11-24T08:22:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.880+0000 7fedac63d640 1 -- start start 2023-11-24T08:22:47.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feda40cbf20 0x7feda40c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda415b440 0x7feda40c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda409cb30 con 0x7feda40c57e0 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda40ad920 con 0x7feda40cbf20 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedac63d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda40c8530 con 0x7feda415b440 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedaa3d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedaa3d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49416/0 (socket says 172.21.15.107:49416) 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.881+0000 7fedaa3d9640 1 -- 172.21.15.107:0/1997671687 learned_addr learned my addr 172.21.15.107:0/1997671687 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7feda9bd8640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feda40cbf20 0x7feda40c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 -- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda415b440 msgr2=0x7feda40c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda415b440 0x7feda40c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 -- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feda40cbf20 msgr2=0x7feda40c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feda40cbf20 0x7feda40c76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:47.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 -- 172.21.15.107:0/1997671687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed940094d0 con 0x7feda40c57e0 2023-11-24T08:22:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedaa3d9640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7feda000d5f0 tx=0x7feda000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00125f0 con 0x7feda40c57e0 2023-11-24T08:22:47.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda0012750 con 0x7feda40c57e0 2023-11-24T08:22:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda0012b00 con 0x7feda40c57e0 2023-11-24T08:22:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda40c8810 con 0x7feda40c57e0 2023-11-24T08:22:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.882+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda40c8130 con 0x7feda40c57e0 2023-11-24T08:22:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.884+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda404e8b0 con 0x7feda40c57e0 2023-11-24T08:22:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.885+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7feda0012c60 con 0x7feda40c57e0 2023-11-24T08:22:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.885+0000 7fed9affd640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 0x7fed8806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.886+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feda0088cd0 con 0x7feda40c57e0 2023-11-24T08:22:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.886+0000 7feda9bd8640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 0x7fed8806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.886+0000 7feda9bd8640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 0x7fed8806f550 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed9403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:47.890+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feda00557e0 con 0x7feda40c57e0 2023-11-24T08:22:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.079+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feda406dc80 con 0x7feda40c57e0 2023-11-24T08:22:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.080+0000 7fed9affd640 1 -- 172.21.15.107:0/1997671687 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v216) v1 ==== 93+0+10 (secure 0 0 0) 0x7feda0058e00 con 0x7feda40c57e0 2023-11-24T08:22:48.082 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.082+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 msgr2=0x7fed8806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.082+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 0x7fed8806f550 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed9403c040 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.082+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 msgr2=0x7feda414bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.082+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7feda000d5f0 tx=0x7feda000dab0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 shutdown_connections 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fed8806d090 0x7fed8806f550 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda415b440 0x7feda40c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feda40cbf20 0x7feda40c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 --2- 172.21.15.107:0/1997671687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feda40c57e0 0x7feda414bb60 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 >> 172.21.15.107:0/1997671687 conn(0x7feda40b0c50 msgr2=0x7feda40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 shutdown_connections 2023-11-24T08:22:48.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.083+0000 7fedac63d640 1 -- 172.21.15.107:0/1997671687 wait complete. 2023-11-24T08:22:48.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:22:48.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.307+0000 7fec4a978640 1 Processor -- start 2023-11-24T08:22:48.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.307+0000 7fec4a978640 1 -- start start 2023-11-24T08:22:48.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.308+0000 7fec4a978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 0x7fec44161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.308+0000 7fec4a978640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440ce0b0 con 0x7fec4415e390 2023-11-24T08:22:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.308+0000 7fec4a978640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec44161960 con 0x7fec440cf450 2023-11-24T08:22:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.308+0000 7fec4a978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4415e8d0 con 0x7fec44157c70 2023-11-24T08:22:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec48f15640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fec4415e390 0x7fec440b31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52526/0 (socket says 172.21.15.107:52526) 2023-11-24T08:22:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec48f15640 1 -- 172.21.15.107:0/327775484 learned_addr learned my addr 172.21.15.107:0/327775484 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:48.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec43fff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 0x7fec44161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768151860 0 0) 0x7fec440ce0b0 con 0x7fec4415e390 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30003520 con 0x7fec4415e390 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970172244 0 0) 0x7fec30003520 con 0x7fec4415e390 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 msgr2=0x7fec44161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 0x7fec44161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 >> v1:172.21.15.120:6789/0 conn(0x7fec440cf450 legacy=0x7fec4415dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4415ebb0 con 0x7fec4415e390 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.309+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fec3c0019f0 con 0x7fec4415e390 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 >> v1:172.21.15.107:6789/0 conn(0x7fec4415e390 legacy=0x7fec440b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec300047c0 0x7fec30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 0x7fec300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4415e8d0 con 0x7fec300039a0 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3000bab0 con 0x7fec300047c0 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3000bc70 con 0x7fec44157c70 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec3bfff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec300047c0 0x7fec30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 -- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 msgr2=0x7fec300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec44157c70 0x7fec300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 -- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec300047c0 msgr2=0x7fec30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec300047c0 0x7fec30004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.310+0000 7fec43fff640 1 -- 172.21.15.107:0/327775484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30003520 con 0x7fec300039a0 2023-11-24T08:22:48.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.311+0000 7fec43fff640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec30003dc0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7fec34009ac0 tx=0x7fec300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c953c7de12fcc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3402c710 con 0x7fec300039a0 2023-11-24T08:22:48.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3402c870 con 0x7fec300039a0 2023-11-24T08:22:48.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec437fe640 1 -- 172.21.15.107:0/327775484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3402cc00 con 0x7fec300039a0 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 -- 172.21.15.107:0/327775484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 msgr2=0x7fec30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 --2- 172.21.15.107:0/327775484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec30003dc0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7fec34009ac0 tx=0x7fec300057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec48f15640 1 -- 172.21.15.107:0/327775484 reap_dead start 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 -- 172.21.15.107:0/327775484 shutdown_connections 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 -- 172.21.15.107:0/327775484 >> 172.21.15.107:0/327775484 conn(0x7fec4406a920 msgr2=0x7fec4406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 -- 172.21.15.107:0/327775484 shutdown_connections 2023-11-24T08:22:48.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.312+0000 7fec4a978640 1 -- 172.21.15.107:0/327775484 wait complete. 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.314+0000 7fec4a978640 1 Processor -- start 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.314+0000 7fec4a978640 1 -- start start 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec441e8940 0x7fec44155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec441e8dc0 0x7fec44155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440cb2a0 con 0x7fec300039a0 2023-11-24T08:22:48.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440ca900 con 0x7fec441e8940 2023-11-24T08:22:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec4a978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440cabe0 con 0x7fec441e8dc0 2023-11-24T08:22:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec43fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45814/0 (socket says 172.21.15.107:45814) 2023-11-24T08:22:48.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.315+0000 7fec43fff640 1 -- 172.21.15.107:0/3520584129 learned_addr learned my addr 172.21.15.107:0/3520584129 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec3bfff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec441e8940 0x7fec44155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 -- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec441e8dc0 msgr2=0x7fec44155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec441e8dc0 0x7fec44155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 -- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec441e8940 msgr2=0x7fec44155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec441e8940 0x7fec44155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 -- 172.21.15.107:0/3520584129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440caec0 con 0x7fec300039a0 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec43fff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fec3402c2b0 tx=0x7fec34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.316+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34003810 con 0x7fec300039a0 2023-11-24T08:22:48.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.317+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3402de90 con 0x7fec300039a0 2023-11-24T08:22:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.317+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec440ca5f0 con 0x7fec300039a0 2023-11-24T08:22:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.317+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34003bb0 con 0x7fec300039a0 2023-11-24T08:22:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.317+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec44156980 con 0x7fec300039a0 2023-11-24T08:22:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.318+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec44067470 con 0x7fec300039a0 2023-11-24T08:22:48.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.319+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fec34003d10 con 0x7fec300039a0 2023-11-24T08:22:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.320+0000 7fec417fa640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 0x7fec2406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.320+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec340aa4b0 con 0x7fec300039a0 2023-11-24T08:22:48.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.321+0000 7fec3bfff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 0x7fec2406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.321+0000 7fec3bfff640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 0x7fec2406f550 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001590 tx=0x7fec2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.325+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec34077060 con 0x7fec300039a0 2023-11-24T08:22:48.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.522+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fec44156980 con 0x7fec300039a0 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.523+0000 7fec417fa640 1 -- 172.21.15.107:0/3520584129 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v216) v1 ==== 84+0+65 (secure 0 0 0) 0x7fec44156980 con 0x7fec300039a0 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:22:48.525 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 msgr2=0x7fec2406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 0x7fec2406f550 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001590 tx=0x7fec2c009040 comp rx=0 tx=0).stop 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 msgr2=0x7fec44156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fec3402c2b0 tx=0x7fec34004730 comp rx=0 tx=0).stop 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 shutdown_connections 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fec2406d090 0x7fec2406f550 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec441e8dc0 0x7fec44155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec441e8940 0x7fec44155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 --2- 172.21.15.107:0/3520584129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec300039a0 0x7fec44156260 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 >> 172.21.15.107:0/3520584129 conn(0x7fec4406a920 msgr2=0x7fec440b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:48.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 shutdown_connections 2023-11-24T08:22:48.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.526+0000 7fec4a978640 1 -- 172.21.15.107:0/3520584129 wait complete. 2023-11-24T08:22:48.547 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:22:48.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:22:48.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.769+0000 7f3aa7fff640 1 Processor -- start 2023-11-24T08:22:48.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.769+0000 7f3aa7fff640 1 -- start start 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3aa80cfbe0 0x7f3aa80d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa815ce20 0x7f3aa815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa7fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa815ea30 con 0x7f3aa8160b00 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa7fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa80d0590 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa815de20 con 0x7f3aa815ce20 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3aa8160b00 0x7f3aa80d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52540/0 (socket says 172.21.15.107:52540) 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3aa6ffd640 1 -- 172.21.15.107:0/2961202064 learned_addr learned my addr 172.21.15.107:0/2961202064 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:48.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.771+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/2961202064 >> v2:172.21.15.107:3300/0 conn(0x7f3aa80cfbe0 0x7f3aa80d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3aa77fe640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa815ce20 0x7f3aa815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3aa67fc640 1 -- 172.21.15.107:0/2961202064 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168734494 0 0) 0x7f3aa815ea30 con 0x7f3aa8160b00 2023-11-24T08:22:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3aa67fc640 1 -- 172.21.15.107:0/2961202064 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a94003520 con 0x7f3aa8160b00 2023-11-24T08:22:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/2961202064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa815ce20 msgr2=0x7f3aa815d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa815ce20 0x7f3aa815d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/2961202064 >> v1:172.21.15.107:6789/0 conn(0x7f3aa8160b00 legacy=0x7f3aa80d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.772+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/2961202064 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa815e100 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.773+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80d0050 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f3a900097a0 tx=0x7f3a9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7e93ae8029aff5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa67fc640 1 -- 172.21.15.107:0/2961202064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90031830 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa67fc640 1 -- 172.21.15.107:0/2961202064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a90031990 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa67fc640 1 -- 172.21.15.107:0/2961202064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90031d20 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 -- 172.21.15.107:0/2961202064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa80cfbe0 msgr2=0x7f3aa80d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80d0050 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f3a900097a0 tx=0x7f3a9002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 -- 172.21.15.107:0/2961202064 shutdown_connections 2023-11-24T08:22:48.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa815ce20 0x7f3aa815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/2961202064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80d0050 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 -- 172.21.15.107:0/2961202064 >> 172.21.15.107:0/2961202064 conn(0x7f3aa809c510 msgr2=0x7f3aa80d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 -- 172.21.15.107:0/2961202064 shutdown_connections 2023-11-24T08:22:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.774+0000 7f3aa7fff640 1 -- 172.21.15.107:0/2961202064 wait complete. 2023-11-24T08:22:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.776+0000 7f3aa7fff640 1 Processor -- start 2023-11-24T08:22:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.776+0000 7f3aa7fff640 1 -- start start 2023-11-24T08:22:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa8160b00 0x7f3aa80c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80c7dd0 con 0x7f3aa815ce20 2023-11-24T08:22:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80c80b0 con 0x7f3aa80cfbe0 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3aa7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80c8390 con 0x7f3aa8160b00 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3a9e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.777+0000 7f3a9e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45838/0 (socket says 172.21.15.107:45838) 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/1354839102 learned_addr learned my addr 172.21.15.107:0/1354839102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3aa77fe640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa8160b00 0x7f3aa80c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3aa6ffd640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa8160b00 msgr2=0x7f3aa80c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa8160b00 0x7f3aa80c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 msgr2=0x7f3aa80ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 -- 172.21.15.107:0/1354839102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90009400 con 0x7f3aa815ce20 2023-11-24T08:22:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3aa6ffd640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.778+0000 7f3a9e7fc640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f3a90034040 tx=0x7f3a900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.779+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9003a8f0 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.779+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9003aa70 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.779+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9003ae20 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.779+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa80c8670 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.779+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa8156af0 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.780+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa80c8b30 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.781+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f3a9004d440 con 0x7f3aa815ce20 2023-11-24T08:22:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.782+0000 7f3a9ffff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 0x7f3a8806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.782+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3a900aa770 con 0x7f3aa815ce20 2023-11-24T08:22:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.782+0000 7f3aa6ffd640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 0x7f3a8806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.783+0000 7f3aa6ffd640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 0x7f3a8806f5a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f3a980011e0 tx=0x7f3a98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.786+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a90077400 con 0x7f3aa815ce20 2023-11-24T08:22:48.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:48.975+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3aa815ee10 con 0x7f3aa815ce20 2023-11-24T08:22:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.729+0000 7f3a9ffff640 1 -- 172.21.15.107:0/1354839102 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v217) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3a90035020 con 0x7f3aa815ce20 2023-11-24T08:22:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 msgr2=0x7f3a8806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 0x7f3a8806f5a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f3a980011e0 tx=0x7f3a98009040 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 msgr2=0x7f3aa81e70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f3a90034040 tx=0x7f3a900046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 shutdown_connections 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3a8806d0e0 0x7f3a8806f5a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3aa8160b00 0x7f3aa80c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3aa815ce20 0x7f3aa81e70b0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 --2- 172.21.15.107:0/1354839102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3aa80cfbe0 0x7f3aa80ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 >> 172.21.15.107:0/1354839102 conn(0x7f3aa809c510 msgr2=0x7f3aa80d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.733+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 shutdown_connections 2023-11-24T08:22:49.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.734+0000 7f3aa7fff640 1 -- 172.21.15.107:0/1354839102 wait complete. 2023-11-24T08:22:49.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:22:49.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.960+0000 7fb0c9a2e640 1 Processor -- start 2023-11-24T08:22:49.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.960+0000 7fb0c9a2e640 1 -- start start 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c9a2e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb0c415d490 0x7fb0c40cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c9a2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c40d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c9a2e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40cea30 con 0x7fb0c40b63d0 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c9a2e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40d0450 con 0x7fb0c415d490 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c9a2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40d0bf0 con 0x7fb0c40d3940 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c8a2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb0c415d490 0x7fb0c40cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:49.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c8a2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb0c415d490 0x7fb0c40cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45854/0 (socket says 172.21.15.107:45854) 2023-11-24T08:22:49.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.962+0000 7fb0c8a2c640 1 -- 172.21.15.107:0/2808925159 learned_addr learned my addr 172.21.15.107:0/2808925159 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:49.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.963+0000 7fb0c8a2c640 1 -- 172.21.15.107:0/2808925159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 msgr2=0x7fb0c40d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:49.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.963+0000 7fb0c8a2c640 1 --2- 172.21.15.107:0/2808925159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c40d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.963+0000 7fb0c8a2c640 1 -- 172.21.15.107:0/2808925159 >> v1:172.21.15.120:6789/0 conn(0x7fb0c40b63d0 legacy=0x7fb0c40d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:49.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.963+0000 7fb0c8a2c640 1 -- 172.21.15.107:0/2808925159 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40ce750 con 0x7fb0c415d490 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.964+0000 7fb0c8a2c640 1 --2- 172.21.15.107:0/2808925159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415d490 0x7fb0c40cd8f0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009870 tx=0x7fb0b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18e4bc9c3ed5bc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.964+0000 7fb0c37fe640 1 -- 172.21.15.107:0/2808925159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8031870 con 0x7fb0c415d490 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.964+0000 7fb0c37fe640 1 -- 172.21.15.107:0/2808925159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b80319d0 con 0x7fb0c415d490 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.964+0000 7fb0c37fe640 1 -- 172.21.15.107:0/2808925159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8031d60 con 0x7fb0c415d490 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/2808925159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415d490 msgr2=0x7fb0c40cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/2808925159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415d490 0x7fb0c40cd8f0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009870 tx=0x7fb0b802b910 comp rx=0 tx=0).stop 2023-11-24T08:22:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/2808925159 shutdown_connections 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/2808925159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c40d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/2808925159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415d490 0x7fb0c40cd8f0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/2808925159 >> 172.21.15.107:0/2808925159 conn(0x7fb0c4061340 msgr2=0x7fb0c4061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/2808925159 shutdown_connections 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.965+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/2808925159 wait complete. 2023-11-24T08:22:49.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.967+0000 7fb0c9a2e640 1 Processor -- start 2023-11-24T08:22:49.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.967+0000 7fb0c9a2e640 1 -- start start 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c415fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c415d490 0x7fb0c414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c415f1b0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c414ed40 con 0x7fb0c415d490 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c9a2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c415e530 con 0x7fb0c40d3940 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c922d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c922d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45868/0 (socket says 172.21.15.107:45868) 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.968+0000 7fb0c922d640 1 -- 172.21.15.107:0/1070934391 learned_addr learned my addr 172.21.15.107:0/1070934391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c8a2c640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c415fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:49.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c3fff640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c415d490 0x7fb0c414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 -- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 msgr2=0x7fb0c415fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c415fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 -- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c415d490 msgr2=0x7fb0c414e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c415d490 0x7fb0c414e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 -- 172.21.15.107:0/1070934391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80094d0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c922d640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00be50 tx=0x7fb0bc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc014440 con 0x7fb0c415f5d0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0145a0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c415e810 con 0x7fb0c415f5d0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.969+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc014950 con 0x7fb0c415f5d0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.970+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c415edd0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.971+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c415f2f0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.973+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fb0bc01d3b0 con 0x7fb0c415f5d0 2023-11-24T08:22:49.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.974+0000 7fb0c17fa640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 0x7fb0a806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:49.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.974+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb0bc088e80 con 0x7fb0c415f5d0 2023-11-24T08:22:49.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.974+0000 7fb0c8a2c640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 0x7fb0a806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:49.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.975+0000 7fb0c8a2c640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 0x7fb0a806f550 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8036000 tx=0x7fb0b803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:49.978+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0bc055a10 con 0x7fb0c415f5d0 2023-11-24T08:22:50.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.169+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb0c404e840 con 0x7fb0c415f5d0 2023-11-24T08:22:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.747+0000 7fb0c17fa640 1 -- 172.21.15.107:0/1070934391 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v218) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb0bc019070 con 0x7fb0c415f5d0 2023-11-24T08:22:50.750 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:22:50.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 msgr2=0x7fb0a806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 0x7fb0a806f550 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8036000 tx=0x7fb0b803c040 comp rx=0 tx=0).stop 2023-11-24T08:22:50.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 msgr2=0x7fb0c414e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00be50 tx=0x7fb0bc00bf40 comp rx=0 tx=0).stop 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 shutdown_connections 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb0a806d090 0x7fb0a806f550 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c415f5d0 0x7fb0c414e800 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0c415d490 0x7fb0c414e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 --2- 172.21.15.107:0/1070934391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0c40d3940 0x7fb0c415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 >> 172.21.15.107:0/1070934391 conn(0x7fb0c4061340 msgr2=0x7fb0c4160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 shutdown_connections 2023-11-24T08:22:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.752+0000 7fb0c9a2e640 1 -- 172.21.15.107:0/1070934391 wait complete. 2023-11-24T08:22:50.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:22:50.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.970+0000 7f893ea1e640 1 Processor -- start 2023-11-24T08:22:50.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.970+0000 7f893ea1e640 1 -- start start 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f893ea1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f893815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f893ea1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893815ec90 con 0x7f89380c49f0 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f893ea1e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380cd530 con 0x7f893806de10 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f893ea1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89380c4ed0 con 0x7f8938072160 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f892ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f89380c49f0 0x7f893806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52542/0 (socket says 172.21.15.107:52542) 2023-11-24T08:22:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.972+0000 7f892ffff640 1 -- 172.21.15.107:0/8269997 learned_addr learned my addr 172.21.15.107:0/8269997 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:50.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f8937fff640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f893815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022120341 0 0) 0x7f8920001110 con 0x7f89380c49f0 2023-11-24T08:22:50.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f89380c49f0 2023-11-24T08:22:50.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867246892 0 0) 0x7f8918003520 con 0x7f89380c49f0 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 msgr2=0x7f893815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f893815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 >> v1:172.21.15.120:6789/0 conn(0x7f893806de10 legacy=0x7f89380705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380c51b0 con 0x7f89380c49f0 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f8920001800 con 0x7f89380c49f0 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 >> v1:172.21.15.107:6789/0 conn(0x7f89380c49f0 legacy=0x7f893806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f89180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8918003f20 0x7f89180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.973+0000 7f89377fe640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f8918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89380c4ed0 con 0x7f8918004d00 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891800ba60 con 0x7f8918003f20 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891800de30 con 0x7f8938072160 2023-11-24T08:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f8918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f8937fff640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f89180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f892ffff640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8918003f20 0x7f89180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 -- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 msgr2=0x7f89180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8938072160 0x7f89180039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 -- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8918003f20 msgr2=0x7f89180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8918003f20 0x7f89180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.974+0000 7f893cfbb640 1 -- 172.21.15.107:0/8269997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918003520 con 0x7f8918004d00 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.975+0000 7f893cfbb640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f8918005180 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f89380aeb90 tx=0x7f89180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aafa7ff99c0fa37 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930003b70 con 0x7f8918004d00 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8930003cd0 con 0x7f8918004d00 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f89377fe640 1 -- 172.21.15.107:0/8269997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930005310 con 0x7f8918004d00 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 -- 172.21.15.107:0/8269997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 msgr2=0x7f8918005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 --2- 172.21.15.107:0/8269997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f8918005180 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f89380aeb90 tx=0x7f89180057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893cfbb640 1 -- 172.21.15.107:0/8269997 reap_dead start 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 -- 172.21.15.107:0/8269997 shutdown_connections 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 -- 172.21.15.107:0/8269997 >> 172.21.15.107:0/8269997 conn(0x7f89380b0c50 msgr2=0x7f89380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 -- 172.21.15.107:0/8269997 shutdown_connections 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.976+0000 7f893ea1e640 1 -- 172.21.15.107:0/8269997 wait complete. 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.978+0000 7f893ea1e640 1 Processor -- start 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.978+0000 7f893ea1e640 1 -- start start 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f893809bc20 0x7f893809c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89380999d0 0x7f8938099e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893809b260 con 0x7f8918004d00 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893809c5e0 con 0x7f89380999d0 2023-11-24T08:22:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.979+0000 7f893ea1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893809a370 con 0x7f893809bc20 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45878/0 (socket says 172.21.15.107:45878) 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 -- 172.21.15.107:0/2510972000 learned_addr learned my addr 172.21.15.107:0/2510972000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f892ffff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f893809bc20 0x7f893809c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f893cfbb640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89380999d0 0x7f8938099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 -- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f893809bc20 msgr2=0x7f893809c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f893809bc20 0x7f893809c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 -- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89380999d0 msgr2=0x7f8938099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89380999d0 0x7f8938099e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 -- 172.21.15.107:0/2510972000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380c7d00 con 0x7f8918004d00 2023-11-24T08:22:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.980+0000 7f8937fff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f8928011630 tx=0x7f8928011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.981+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892800f920 con 0x7f8918004d00 2023-11-24T08:22:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.981+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892800fa80 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.981+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892800fdf0 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.981+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893809a5e0 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.981+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893809aba0 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.982+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893804e8b0 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.984+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f8928015460 con 0x7f8918004d00 2023-11-24T08:22:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.985+0000 7f89357fa640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 0x7f891c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.985+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8928089010 con 0x7f8918004d00 2023-11-24T08:22:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.985+0000 7f892ffff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 0x7f891c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.985+0000 7f892ffff640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 0x7f891c06f580 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f8920001e10 tx=0x7f8920001110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:50.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:50.988+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8928055ba0 con 0x7f8918004d00 2023-11-24T08:22:51.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.210+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f893809cec0 con 0x7f8918004d00 2023-11-24T08:22:51.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.752+0000 7f89357fa640 1 -- 172.21.15.107:0/2510972000 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v134) v1 ==== 174+0+0 (secure 0 0 0) 0x7f89280591c0 con 0x7f8918004d00 2023-11-24T08:22:51.754 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 31 and data pool 32 2023-11-24T08:22:51.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.755+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 msgr2=0x7f891c06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.755+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 0x7f891c06f580 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f8920001e10 tx=0x7f8920001110 comp rx=0 tx=0).stop 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.755+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 msgr2=0x7f89381590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.755+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f8928011630 tx=0x7f8928011af0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 shutdown_connections 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f891c06d0c0 0x7f891c06f580 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89380999d0 0x7f8938099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f893809bc20 0x7f893809c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 --2- 172.21.15.107:0/2510972000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8918004d00 0x7f89381590a0 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 >> 172.21.15.107:0/2510972000 conn(0x7f89380b0c50 msgr2=0x7f89380ac440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:51.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.756+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 shutdown_connections 2023-11-24T08:22:51.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.758+0000 7f893ea1e640 1 -- 172.21.15.107:0/2510972000 wait complete. 2023-11-24T08:22:51.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:51.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.984+0000 7f633646b640 1 Processor -- start 2023-11-24T08:22:51.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.984+0000 7f633646b640 1 -- start start 2023-11-24T08:22:51.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f633646b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6330159ce0 0x7f633006de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:51.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f633646b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63300b3a40 con 0x7f63300cc150 2023-11-24T08:22:51.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f633646b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63300cdfa0 con 0x7f63300b21d0 2023-11-24T08:22:51.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f633646b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63300cd9e0 con 0x7f6330159ce0 2023-11-24T08:22:51.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f6334a08640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f63300cc150 0x7f633006c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52544/0 (socket says 172.21.15.107:52544) 2023-11-24T08:22:51.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f6334a08640 1 -- 172.21.15.107:0/3413979459 learned_addr learned my addr 172.21.15.107:0/3413979459 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:51.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.986+0000 7f632ffff640 1 --2- 172.21.15.107:0/3413979459 >> v2:172.21.15.120:3300/0 conn(0x7f6330159ce0 0x7f633006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:51.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676869401 0 0) 0x7f63300b3a40 con 0x7f63300cc150 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631c003440 con 0x7f63300cc150 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972563319 0 0) 0x7f631c003440 con 0x7f63300cc150 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 >> v2:172.21.15.120:3300/0 conn(0x7f6330159ce0 msgr2=0x7f633006de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 --2- 172.21.15.107:0/3413979459 >> v2:172.21.15.120:3300/0 conn(0x7f6330159ce0 0x7f633006de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 >> v1:172.21.15.120:6789/0 conn(0x7f63300b21d0 legacy=0x7f63300cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63300cdcc0 con 0x7f63300cc150 2023-11-24T08:22:51.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6320001670 con 0x7f63300cc150 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 >> v1:172.21.15.107:6789/0 conn(0x7f63300cc150 legacy=0x7f633006c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f631c003a60 0x7f631c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 0x7f631c00ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c00b480 0x7f631c00d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63300cd9e0 con 0x7f631c008850 2023-11-24T08:22:51.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c00e220 con 0x7f631c00b480 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631c00e3e0 con 0x7f631c003a60 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.987+0000 7f632f7fe640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 0x7f631c00ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632ffff640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f631c003a60 0x7f631c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632f7fe640 1 -- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f631c003a60 msgr2=0x7f631c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632f7fe640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f631c003a60 0x7f631c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632f7fe640 1 -- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c00b480 msgr2=0x7f631c00d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632f7fe640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c00b480 0x7f631c00d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f632f7fe640 1 -- 172.21.15.107:0/3413979459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c003440 con 0x7f631c008850 2023-11-24T08:22:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.988+0000 7f6334a08640 1 -- 172.21.15.107:0/3413979459 reap_dead start 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.989+0000 7f632f7fe640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 0x7f631c00ac80 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f6318001750 tx=0x7f631c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf355ce43408d13 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.989+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63180052d0 con 0x7f631c008850 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.989+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631802e050 con 0x7f631c008850 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.989+0000 7f632effd640 1 -- 172.21.15.107:0/3413979459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318003830 con 0x7f631c008850 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 -- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 msgr2=0x7f631c00ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 0x7f631c00ac80 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f6318001750 tx=0x7f631c0036f0 comp rx=0 tx=0).stop 2023-11-24T08:22:51.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 -- 172.21.15.107:0/3413979459 shutdown_connections 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 --2- 172.21.15.107:0/3413979459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f631c008850 0x7f631c00ac80 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 -- 172.21.15.107:0/3413979459 >> 172.21.15.107:0/3413979459 conn(0x7f633006a920 msgr2=0x7f633006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 -- 172.21.15.107:0/3413979459 shutdown_connections 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.990+0000 7f633646b640 1 -- 172.21.15.107:0/3413979459 wait complete. 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.992+0000 7f633646b640 1 Processor -- start 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.992+0000 7f633646b640 1 -- start start 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c008850 0x7f633014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63300989b0 0x7f6330151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633014f4e0 con 0x7f6330151d70 2023-11-24T08:22:52.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633014eed0 con 0x7f631c008850 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f633646b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6330098e30 con 0x7f63300989b0 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45902/0 (socket says 172.21.15.107:45902) 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 -- 172.21.15.107:0/1708716181 learned_addr learned my addr 172.21.15.107:0/1708716181 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f632ffff640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c008850 0x7f633014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 -- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63300989b0 msgr2=0x7f6330151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f632f7fe640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63300989b0 0x7f6330151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63300989b0 0x7f6330151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 -- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c008850 msgr2=0x7f633014e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c008850 0x7f633014e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.993+0000 7f6334a08640 1 -- 172.21.15.107:0/1708716181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633014d720 con 0x7f6330151d70 2023-11-24T08:22:52.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f6334a08640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f6330072700 tx=0x7f632000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63200074e0 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320007660 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633014f170 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320004a20 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.994+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633014c3f0 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.995+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633014ed20 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.996+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f6320004b80 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.997+0000 7f632cff9640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 0x7f631006f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.997+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63200878e0 con 0x7f6330151d70 2023-11-24T08:22:52.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.997+0000 7f632ffff640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 0x7f631006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:51.998+0000 7f632ffff640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 0x7f631006f580 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f63300681f0 tx=0x7f632800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.001+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6320054490 con 0x7f6330151d70 2023-11-24T08:22:52.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.179+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6330067470 con 0x7f6330151d70 2023-11-24T08:22:52.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.181+0000 7f632cff9640 1 -- 172.21.15.107:0/1708716181 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f6320057ab0 con 0x7f6330151d70 2023-11-24T08:22:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:52.184 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":219,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:51.212021+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:52.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 msgr2=0x7f631006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 0x7f631006f580 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f63300681f0 tx=0x7f632800b040 comp rx=0 tx=0).stop 2023-11-24T08:22:52.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 msgr2=0x7f63301521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f6330072700 tx=0x7f632000c820 comp rx=0 tx=0).stop 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 shutdown_connections 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f631006d0c0 0x7f631006f580 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6330151d70 0x7f63301521f0 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63300989b0 0x7f6330151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 --2- 172.21.15.107:0/1708716181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f631c008850 0x7f633014e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 >> 172.21.15.107:0/1708716181 conn(0x7f633006a920 msgr2=0x7f6330070720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 shutdown_connections 2023-11-24T08:22:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.184+0000 7f633646b640 1 -- 172.21.15.107:0/1708716181 wait complete. 2023-11-24T08:22:52.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.403+0000 7fcd83c94640 1 Processor -- start 2023-11-24T08:22:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.403+0000 7fcd83c94640 1 -- start start 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.404+0000 7fcd83c94640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd7c0b2250 0x7fcd7c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.404+0000 7fcd83c94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2630 0x7fcd7c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.404+0000 7fcd83c94640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c15ab80 con 0x7fcd7c0cd470 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.404+0000 7fcd83c94640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c15a990 con 0x7fcd7c0b2250 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.404+0000 7fcd83c94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c15e680 con 0x7fcd7c0b2630 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd7c0b2250 0x7fcd7c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd7c0b2250 0x7fcd7c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45906/0 (socket says 172.21.15.107:45906) 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 -- 172.21.15.107:0/3091835860 learned_addr learned my addr 172.21.15.107:0/3091835860 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd8122f640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2630 0x7fcd7c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 -- 172.21.15.107:0/3091835860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2630 msgr2=0x7fcd7c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:52.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2630 0x7fcd7c0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.405+0000 7fcd81a30640 1 -- 172.21.15.107:0/3091835860 >> v1:172.21.15.120:6789/0 conn(0x7fcd7c0cd470 legacy=0x7fcd7c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.406+0000 7fcd81a30640 1 -- 172.21.15.107:0/3091835860 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c15e960 con 0x7fcd7c0b2250 2023-11-24T08:22:52.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.407+0000 7fcd81a30640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2250 0x7fcd7c15df80 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800bb10 tx=0x7fcd7802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98661ff45008bca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.407+0000 7fcd80a2e640 1 -- 172.21.15.107:0/3091835860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd78036440 con 0x7fcd7c0b2250 2023-11-24T08:22:52.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.407+0000 7fcd80a2e640 1 -- 172.21.15.107:0/3091835860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd780365a0 con 0x7fcd7c0b2250 2023-11-24T08:22:52.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.407+0000 7fcd80a2e640 1 -- 172.21.15.107:0/3091835860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd78036930 con 0x7fcd7c0b2250 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 -- 172.21.15.107:0/3091835860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2250 msgr2=0x7fcd7c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2250 0x7fcd7c15df80 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800bb10 tx=0x7fcd7802e150 comp rx=0 tx=0).stop 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 -- 172.21.15.107:0/3091835860 shutdown_connections 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2630 0x7fcd7c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 --2- 172.21.15.107:0/3091835860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2250 0x7fcd7c15df80 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 -- 172.21.15.107:0/3091835860 >> 172.21.15.107:0/3091835860 conn(0x7fcd7c06a920 msgr2=0x7fcd7c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 -- 172.21.15.107:0/3091835860 shutdown_connections 2023-11-24T08:22:52.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.408+0000 7fcd83c94640 1 -- 172.21.15.107:0/3091835860 wait complete. 2023-11-24T08:22:52.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.410+0000 7fcd83c94640 1 Processor -- start 2023-11-24T08:22:52.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.410+0000 7fcd83c94640 1 -- start start 2023-11-24T08:22:52.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.410+0000 7fcd83c94640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2250 0x7fcd7c150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd83c94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd83c94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd7c0cd470 0x7fcd7c156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd83c94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0ac8d0 con 0x7fcd7c0b2630 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd83c94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0b1640 con 0x7fcd7c0cd470 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd83c94640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0b18e0 con 0x7fcd7c0b2250 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45916/0 (socket says 172.21.15.107:45916) 2023-11-24T08:22:52.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 -- 172.21.15.107:0/1963648048 learned_addr learned my addr 172.21.15.107:0/1963648048 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 -- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2250 msgr2=0x7fcd7c150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2250 0x7fcd7c150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 -- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd7c0cd470 msgr2=0x7fcd7c156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd7c0cd470 0x7fcd7c156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 -- 172.21.15.107:0/1963648048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7800b770 con 0x7fcd7c0b2630 2023-11-24T08:22:52.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.411+0000 7fcd8122f640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00d800 tx=0x7fcd6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.412+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00b6c0 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.412+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c00b840 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.412+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00bbf0 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.412+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c0acb20 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.412+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c150650 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.413+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c0ad040 con 0x7fcd7c0b2630 2023-11-24T08:22:52.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.414+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fcd6c00bd50 con 0x7fcd7c0b2630 2023-11-24T08:22:52.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.415+0000 7fcd727fc640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 0x7fcd6006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.415+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd6c088de0 con 0x7fcd7c0b2630 2023-11-24T08:22:52.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.415+0000 7fcd81a30640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 0x7fcd6006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.416+0000 7fcd81a30640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 0x7fcd6006f550 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fcd78038000 tx=0x7fcd78030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.419+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd7c0ad040 con 0x7fcd7c0b2630 2023-11-24T08:22:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.603+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd7c0ad040 con 0x7fcd7c0b2630 2023-11-24T08:22:52.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.606+0000 7fcd727fc640 1 -- 172.21.15.107:0/1963648048 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+16592 (secure 0 0 0) 0x7fcd7c0ad040 con 0x7fcd7c0b2630 2023-11-24T08:22:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:52.608 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":219,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:51.212021+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:52.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 msgr2=0x7fcd6006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 0x7fcd6006f550 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fcd78038000 tx=0x7fcd78030000 comp rx=0 tx=0).stop 2023-11-24T08:22:52.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 msgr2=0x7fcd7c1452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00d800 tx=0x7fcd6c00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 shutdown_connections 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd6006d090 0x7fcd6006f550 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd7c0cd470 0x7fcd7c156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd7c0b2630 0x7fcd7c1452e0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 --2- 172.21.15.107:0/1963648048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd7c0b2250 0x7fcd7c150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 >> 172.21.15.107:0/1963648048 conn(0x7fcd7c06a920 msgr2=0x7fcd7c1591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.608+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 shutdown_connections 2023-11-24T08:22:52.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.609+0000 7fcd83c94640 1 -- 172.21.15.107:0/1963648048 wait complete. 2023-11-24T08:22:52.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:22:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.831+0000 7feb3a91d640 1 Processor -- start 2023-11-24T08:22:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.831+0000 7feb3a91d640 1 -- start start 2023-11-24T08:22:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb3a91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb3a91d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb340ccfb0 0x7feb34162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb3a91d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb341655f0 con 0x7feb34162540 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb3a91d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb340cc380 con 0x7feb340ccfb0 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb3a91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb34162b30 con 0x7feb340ccbd0 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb33fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb337fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb340ccfb0 0x7feb34162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb33fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:57568/0 (socket says 172.21.15.107:57568) 2023-11-24T08:22:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.833+0000 7feb33fff640 1 -- 172.21.15.107:0/2486901214 learned_addr learned my addr 172.21.15.107:0/2486901214 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.834+0000 7feb33fff640 1 -- 172.21.15.107:0/2486901214 >> v2:172.21.15.120:3300/0 conn(0x7feb340ccfb0 msgr2=0x7feb34162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.834+0000 7feb33fff640 1 --2- 172.21.15.107:0/2486901214 >> v2:172.21.15.120:3300/0 conn(0x7feb340ccfb0 0x7feb34162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.834+0000 7feb33fff640 1 -- 172.21.15.107:0/2486901214 >> v1:172.21.15.120:6789/0 conn(0x7feb34162540 legacy=0x7feb340c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.834+0000 7feb33fff640 1 -- 172.21.15.107:0/2486901214 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34162e10 con 0x7feb340ccbd0 2023-11-24T08:22:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.835+0000 7feb33fff640 1 --2- 172.21.15.107:0/2486901214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb240098a0 tx=0x7feb2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be65b67995c9fdc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb32ffd640 1 -- 172.21.15.107:0/2486901214 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24030840 con 0x7feb340ccbd0 2023-11-24T08:22:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb32ffd640 1 -- 172.21.15.107:0/2486901214 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb240309a0 con 0x7feb340ccbd0 2023-11-24T08:22:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb32ffd640 1 -- 172.21.15.107:0/2486901214 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24030d30 con 0x7feb340ccbd0 2023-11-24T08:22:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 -- 172.21.15.107:0/2486901214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 msgr2=0x7feb341660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2486901214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb240098a0 tx=0x7feb2402b980 comp rx=0 tx=0).stop 2023-11-24T08:22:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 -- 172.21.15.107:0/2486901214 shutdown_connections 2023-11-24T08:22:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2486901214 >> v2:172.21.15.120:3300/0 conn(0x7feb340ccfb0 0x7feb34162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2486901214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccbd0 0x7feb341660b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 -- 172.21.15.107:0/2486901214 >> 172.21.15.107:0/2486901214 conn(0x7feb340b0c50 msgr2=0x7feb340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 -- 172.21.15.107:0/2486901214 shutdown_connections 2023-11-24T08:22:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.836+0000 7feb3a91d640 1 -- 172.21.15.107:0/2486901214 wait complete. 2023-11-24T08:22:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.838+0000 7feb3a91d640 1 Processor -- start 2023-11-24T08:22:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.838+0000 7feb3a91d640 1 -- start start 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb340ccbd0 0x7feb340abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccfb0 0x7feb340adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340b3700 con 0x7feb34162540 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340ae120 con 0x7feb340ccbd0 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb3a91d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340aced0 con 0x7feb340ccfb0 2023-11-24T08:22:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb38eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb38eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45928/0 (socket says 172.21.15.107:45928) 2023-11-24T08:22:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.839+0000 7feb38eba640 1 -- 172.21.15.107:0/2517017229 learned_addr learned my addr 172.21.15.107:0/2517017229 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb337fe640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccfb0 0x7feb340adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb33fff640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb340ccbd0 0x7feb340abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 -- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccfb0 msgr2=0x7feb340adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccfb0 0x7feb340adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 -- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb340ccbd0 msgr2=0x7feb340abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb340ccbd0 0x7feb340abf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 -- 172.21.15.107:0/2517017229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240094d0 con 0x7feb34162540 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb38eba640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c7a0 tx=0x7feb2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c003b70 con 0x7feb34162540 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c003cf0 con 0x7feb34162540 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c003ea0 con 0x7feb34162540 2023-11-24T08:22:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.840+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb340b3360 con 0x7feb34162540 2023-11-24T08:22:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.841+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb340ac910 con 0x7feb34162540 2023-11-24T08:22:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.842+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb340b3840 con 0x7feb34162540 2023-11-24T08:22:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.843+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7feb2c00aa90 con 0x7feb34162540 2023-11-24T08:22:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.844+0000 7feb30ff9640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 0x7feb2006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.844+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb2c088570 con 0x7feb34162540 2023-11-24T08:22:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.844+0000 7feb33fff640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 0x7feb2006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:52.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.844+0000 7feb33fff640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 0x7feb2006f550 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:52.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:52.848+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb340b3840 con 0x7feb34162540 2023-11-24T08:22:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.063+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7feb340b3840 con 0x7feb34162540 2023-11-24T08:22:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.783+0000 7feb30ff9640 1 -- 172.21.15.107:0/2517017229 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v137) v1 ==== 118+0+0 (secure 0 0 0) 0x7feb340b3840 con 0x7feb34162540 2023-11-24T08:22:53.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 msgr2=0x7feb2006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:53.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 0x7feb2006f550 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2403b040 comp rx=0 tx=0).stop 2023-11-24T08:22:53.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 msgr2=0x7feb34158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:53.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c7a0 tx=0x7feb2c00cc60 comp rx=0 tx=0).stop 2023-11-24T08:22:53.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 shutdown_connections 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb2006d090 0x7feb2006f550 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb34162540 0x7feb34158700 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb340ccfb0 0x7feb340adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 --2- 172.21.15.107:0/2517017229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb340ccbd0 0x7feb340abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 >> 172.21.15.107:0/2517017229 conn(0x7feb340b0c50 msgr2=0x7feb340cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 shutdown_connections 2023-11-24T08:22:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:53.786+0000 7feb3a91d640 1 -- 172.21.15.107:0/2517017229 wait complete. 2023-11-24T08:22:53.810 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:22:53.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:22:54.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.025+0000 7fc453bbe640 1 Processor -- start 2023-11-24T08:22:54.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.025+0000 7fc453bbe640 1 -- start start 2023-11-24T08:22:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc453bbe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 0x7fc44c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc453bbe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc44c0c49f0 0x7fc44c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc453bbe640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44c15ec90 con 0x7fc44c072160 2023-11-24T08:22:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc453bbe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c0cd530 con 0x7fc44c0c49f0 2023-11-24T08:22:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc453bbe640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c0c4ed0 con 0x7fc44c06de10 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc44c0c49f0 0x7fc44c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc44c0c49f0 0x7fc44c0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45936/0 (socket says 172.21.15.107:45936) 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 -- 172.21.15.107:0/3868579342 learned_addr learned my addr 172.21.15.107:0/3868579342 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc451159640 1 --2- 172.21.15.107:0/3868579342 >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 0x7fc44c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 -- 172.21.15.107:0/3868579342 >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 msgr2=0x7fc44c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 --2- 172.21.15.107:0/3868579342 >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 0x7fc44c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.027+0000 7fc45215b640 1 -- 172.21.15.107:0/3868579342 >> v1:172.21.15.120:6789/0 conn(0x7fc44c072160 legacy=0x7fc44c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.028+0000 7fc45215b640 1 -- 172.21.15.107:0/3868579342 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0c51b0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.028+0000 7fc451159640 1 --2- 172.21.15.107:0/3868579342 >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 0x7fc44c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc45215b640 1 --2- 172.21.15.107:0/3868579342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0705c0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0cef60 tx=0x7fc44402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259213dc1c63fb6e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc450958640 1 -- 172.21.15.107:0/3868579342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44402ca80 con 0x7fc44c0c49f0 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc450958640 1 -- 172.21.15.107:0/3868579342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44402cbe0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc450958640 1 -- 172.21.15.107:0/3868579342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4440343f0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc453bbe640 1 -- 172.21.15.107:0/3868579342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 msgr2=0x7fc44c0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.029+0000 7fc453bbe640 1 --2- 172.21.15.107:0/3868579342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0705c0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0cef60 tx=0x7fc44402bee0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 -- 172.21.15.107:0/3868579342 shutdown_connections 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 --2- 172.21.15.107:0/3868579342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0705c0 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 --2- 172.21.15.107:0/3868579342 >> v2:172.21.15.120:3300/0 conn(0x7fc44c06de10 0x7fc44c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 -- 172.21.15.107:0/3868579342 >> 172.21.15.107:0/3868579342 conn(0x7fc44c0b0c50 msgr2=0x7fc44c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 -- 172.21.15.107:0/3868579342 shutdown_connections 2023-11-24T08:22:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.030+0000 7fc453bbe640 1 -- 172.21.15.107:0/3868579342 wait complete. 2023-11-24T08:22:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.031+0000 7fc453bbe640 1 Processor -- start 2023-11-24T08:22:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 -- start start 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06de10 0x7fc44c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c072160 0x7fc44c0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c15a1e0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0c7dd0 con 0x7fc44c072160 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.032+0000 7fc453bbe640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0c71c0 con 0x7fc44c06de10 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:54.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45950/0 (socket says 172.21.15.107:45950) 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 -- 172.21.15.107:0/223589685 learned_addr learned my addr 172.21.15.107:0/223589685 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 -- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06de10 msgr2=0x7fc44c159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06de10 0x7fc44c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 -- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c072160 msgr2=0x7fc44c0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc451159640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c072160 0x7fc44c0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c072160 0x7fc44c0c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 -- 172.21.15.107:0/223589685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009b30 con 0x7fc44c0c49f0 2023-11-24T08:22:54.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.033+0000 7fc45215b640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fc444036040 tx=0x7fc444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:54.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.034+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44403e450 con 0x7fc44c0c49f0 2023-11-24T08:22:54.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.034+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44403e5d0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.034+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44403e980 con 0x7fc44c0c49f0 2023-11-24T08:22:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.034+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c0c74a0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.034+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c0c7a60 con 0x7fc44c0c49f0 2023-11-24T08:22:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.035+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c04e8b0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.036+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fc44402cd50 con 0x7fc44c0c49f0 2023-11-24T08:22:54.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.037+0000 7fc4427fc640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 0x7fc43006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:54.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.037+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4440ab1c0 con 0x7fc44c0c49f0 2023-11-24T08:22:54.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.037+0000 7fc45195a640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 0x7fc43006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:54.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.038+0000 7fc45195a640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 0x7fc43006f550 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0ca3d0 tx=0x7fc448007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.042+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc444077e20 con 0x7fc44c0c49f0 2023-11-24T08:22:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.269+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc44c15ec90 con 0x7fc44c0c49f0 2023-11-24T08:22:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.793+0000 7fc4427fc640 1 -- 172.21.15.107:0/223589685 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v138) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc44c15ec90 con 0x7fc44c0c49f0 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 msgr2=0x7fc43006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 0x7fc43006f550 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0ca3d0 tx=0x7fc448007990 comp rx=0 tx=0).stop 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 msgr2=0x7fc44c0b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fc444036040 tx=0x7fc444004a40 comp rx=0 tx=0).stop 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 shutdown_connections 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43006d090 0x7fc43006f550 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc44c0c49f0 0x7fc44c0b27b0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc44c072160 0x7fc44c0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 --2- 172.21.15.107:0/223589685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc44c06de10 0x7fc44c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:54.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 >> 172.21.15.107:0/223589685 conn(0x7fc44c0b0c50 msgr2=0x7fc44c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:54.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 shutdown_connections 2023-11-24T08:22:54.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:54.797+0000 7fc453bbe640 1 -- 172.21.15.107:0/223589685 wait complete. 2023-11-24T08:22:54.822 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:22:54.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.035+0000 7f6f7359e640 1 Processor -- start 2023-11-24T08:22:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.035+0000 7f6f7359e640 1 -- start start 2023-11-24T08:22:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.036+0000 7f6f7359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f740d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.036+0000 7f6f7359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f740d2060 con 0x7f6f740b3880 2023-11-24T08:22:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.036+0000 7f6f7359e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f740cef60 con 0x7f6f740d1740 2023-11-24T08:22:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.036+0000 7f6f7359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740b2ac0 con 0x7f6f740d0f40 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.037+0000 7f6f72d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f740b3880 0x7f6f740b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52548/0 (socket says 172.21.15.107:52548) 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.037+0000 7f6f72d9d640 1 -- 172.21.15.107:0/275211722 learned_addr learned my addr 172.21.15.107:0/275211722 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.037+0000 7f6f71d9b640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f740d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.037+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683808403 0 0) 0x7f6f740d2060 con 0x7f6f740b3880 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.037+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f60003520 con 0x7f6f740b3880 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545710262 0 0) 0x7f6f60003520 con 0x7f6f740b3880 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 msgr2=0x7f6f740d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f740d1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 >> v1:172.21.15.120:6789/0 conn(0x7f6f740d1740 legacy=0x7f6f740ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740b2da0 con 0x7f6f740b3880 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6f6c001670 con 0x7f6f740b3880 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 >> v1:172.21.15.107:6789/0 conn(0x7f6f740b3880 legacy=0x7f6f740b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f600039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f60003f70 0x7f6f600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f60004d00 0x7f6f60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740b2ac0 con 0x7f6f60004d00 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6000bb50 con 0x7f6f60003f70 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6000bd10 con 0x7f6f740d0f40 2023-11-24T08:22:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f60004d00 0x7f6f60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f71d9b640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f60003f70 0x7f6f600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.038+0000 7f6f7259c640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f600039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.039+0000 7f6f72d9d640 1 -- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 msgr2=0x7f6f600039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.039+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f740d0f40 0x7f6f600039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.039+0000 7f6f72d9d640 1 -- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f60003f70 msgr2=0x7f6f600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.039+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f60003f70 0x7f6f600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.039+0000 7f6f72d9d640 1 -- 172.21.15.107:0/275211722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60003520 con 0x7f6f60004d00 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f60004d00 0x7f6f60005180 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c0e0 tx=0x7f6f600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f04a8063954fb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c005ad0 con 0x7f6f60004d00 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c005c30 con 0x7f6f60004d00 2023-11-24T08:22:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f7159a640 1 -- 172.21.15.107:0/275211722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c02ca50 con 0x7f6f60004d00 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f7359e640 1 -- 172.21.15.107:0/275211722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f60004d00 msgr2=0x7f6f60005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.040+0000 7f6f7359e640 1 --2- 172.21.15.107:0/275211722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f60004d00 0x7f6f60005180 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c0e0 tx=0x7f6f600057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.041+0000 7f6f72d9d640 1 -- 172.21.15.107:0/275211722 reap_dead start 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.041+0000 7f6f7359e640 1 -- 172.21.15.107:0/275211722 shutdown_connections 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.041+0000 7f6f7359e640 1 -- 172.21.15.107:0/275211722 >> 172.21.15.107:0/275211722 conn(0x7f6f7409c500 msgr2=0x7f6f740d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.041+0000 7f6f7359e640 1 -- 172.21.15.107:0/275211722 shutdown_connections 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.041+0000 7f6f7359e640 1 -- 172.21.15.107:0/275211722 wait complete. 2023-11-24T08:22:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.042+0000 7f6f7359e640 1 Processor -- start 2023-11-24T08:22:55.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.043+0000 7f6f7359e640 1 -- start start 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60004d00 0x7f6f74160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7415dc30 0x7f6f741f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74160de0 con 0x7f6f741f69f0 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74160820 con 0x7f6f7415dc30 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74160b00 con 0x7f6f60004d00 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45972/0 (socket says 172.21.15.107:45972) 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 -- 172.21.15.107:0/730204531 learned_addr learned my addr 172.21.15.107:0/730204531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f7259c640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60004d00 0x7f6f74160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f71d9b640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7415dc30 0x7f6f741f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 -- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60004d00 msgr2=0x7f6f74160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60004d00 0x7f6f74160240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 -- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7415dc30 msgr2=0x7f6f741f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7415dc30 0x7f6f741f64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.044+0000 7f6f72d9d640 1 -- 172.21.15.107:0/730204531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740cb3b0 con 0x7f6f741f69f0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f72d9d640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c0e0 tx=0x7f6f6c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c03c040 con 0x7f6f741f69f0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c02cc20 con 0x7f6f741f69f0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c007070 con 0x7f6f741f69f0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f740cb690 con 0x7f6f741f69f0 2023-11-24T08:22:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.045+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7415e0b0 con 0x7f6f741f69f0 2023-11-24T08:22:55.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.046+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f740cb880 con 0x7f6f741f69f0 2023-11-24T08:22:55.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.047+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f6f6c032070 con 0x7f6f741f69f0 2023-11-24T08:22:55.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.048+0000 7f6f6affd640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 0x7f6f5406f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.048+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f6c0a9b00 con 0x7f6f741f69f0 2023-11-24T08:22:55.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.048+0000 7f6f7259c640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 0x7f6f5406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.049+0000 7f6f7259c640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 0x7f6f5406f550 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6f64007fd0 tx=0x7f6f64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.052+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f6c076610 con 0x7f6f741f69f0 2023-11-24T08:22:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.285+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f7404e840 con 0x7f6f741f69f0 2023-11-24T08:22:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.286+0000 7f6f6affd640 1 -- 172.21.15.107:0/730204531 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f6f6c079c30 con 0x7f6f741f69f0 2023-11-24T08:22:55.289 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:55.289 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:22:55.289 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 138 2023-11-24T08:22:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 msgr2=0x7f6f5406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 0x7f6f5406f550 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6f64007fd0 tx=0x7f6f64009040 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 msgr2=0x7f6f741f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c0e0 tx=0x7f6f6c02e070 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 shutdown_connections 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6f5406d090 0x7f6f5406f550 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f741f69f0 0x7f6f741f6e70 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7415dc30 0x7f6f741f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 --2- 172.21.15.107:0/730204531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60004d00 0x7f6f74160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.289+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 >> 172.21.15.107:0/730204531 conn(0x7f6f7409c500 msgr2=0x7f6f740d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.290+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 shutdown_connections 2023-11-24T08:22:55.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.290+0000 7f6f7359e640 1 -- 172.21.15.107:0/730204531 wait complete. 2023-11-24T08:22:55.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:22:55.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.518+0000 7fadd2b41640 1 Processor -- start 2023-11-24T08:22:55.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.518+0000 7fadd2b41640 1 -- start start 2023-11-24T08:22:55.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd2b41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadcc15c7f0 0x7fadcc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd2b41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 0x7fadcc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd2b41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc0cece0 con 0x7fadcc0cf450 2023-11-24T08:22:55.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd2b41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc0cfe40 con 0x7fadcc15c7f0 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd2b41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc156010 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd08dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadcc15c7f0 0x7fadcc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd08dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadcc15c7f0 0x7fadcc06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45982/0 (socket says 172.21.15.107:45982) 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd08dd640 1 -- 172.21.15.107:0/2205140480 learned_addr learned my addr 172.21.15.107:0/2205140480 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd10de640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 0x7fadcc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.520+0000 7fadd08dd640 1 -- 172.21.15.107:0/2205140480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 msgr2=0x7fadcc0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.521+0000 7fadd08dd640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 0x7fadcc0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.521+0000 7fadd08dd640 1 -- 172.21.15.107:0/2205140480 >> v1:172.21.15.120:6789/0 conn(0x7fadcc0cf450 legacy=0x7fadcc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.521+0000 7fadd08dd640 1 -- 172.21.15.107:0/2205140480 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc06d040 con 0x7fadcc15c7f0 2023-11-24T08:22:55.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.521+0000 7fadd10de640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 0x7fadcc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:55.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.522+0000 7fadd08dd640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc15c7f0 0x7fadcc06db20 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1d3d0376af4c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.522+0000 7fadcb7fe640 1 -- 172.21.15.107:0/2205140480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0031640 con 0x7fadcc15c7f0 2023-11-24T08:22:55.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.522+0000 7fadcb7fe640 1 -- 172.21.15.107:0/2205140480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc00317a0 con 0x7fadcc15c7f0 2023-11-24T08:22:55.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.522+0000 7fadcb7fe640 1 -- 172.21.15.107:0/2205140480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0031b30 con 0x7fadcc15c7f0 2023-11-24T08:22:55.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 -- 172.21.15.107:0/2205140480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc15c7f0 msgr2=0x7fadcc06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc15c7f0 0x7fadcc06db20 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 -- 172.21.15.107:0/2205140480 shutdown_connections 2023-11-24T08:22:55.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0ce6c0 0x7fadcc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 --2- 172.21.15.107:0/2205140480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc15c7f0 0x7fadcc06db20 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 -- 172.21.15.107:0/2205140480 >> 172.21.15.107:0/2205140480 conn(0x7fadcc06a920 msgr2=0x7fadcc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:55.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 -- 172.21.15.107:0/2205140480 shutdown_connections 2023-11-24T08:22:55.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.523+0000 7fadd2b41640 1 -- 172.21.15.107:0/2205140480 wait complete. 2023-11-24T08:22:55.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.525+0000 7fadd2b41640 1 Processor -- start 2023-11-24T08:22:55.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.525+0000 7fadd2b41640 1 -- start start 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0cf450 0x7fadcc0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadcc15c7f0 0x7fadcc156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1f6bd0 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1f6eb0 con 0x7fadcc15c7f0 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd2b41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1f7190 con 0x7fadcc0cf450 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45984/0 (socket says 172.21.15.107:45984) 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 -- 172.21.15.107:0/4093048285 learned_addr learned my addr 172.21.15.107:0/4093048285 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 -- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0cf450 msgr2=0x7fadcc0b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0cf450 0x7fadcc0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 -- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadcc15c7f0 msgr2=0x7fadcc156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadcc15c7f0 0x7fadcc156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.527+0000 7fadd08dd640 1 -- 172.21.15.107:0/4093048285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00094d0 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadd08dd640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc003ac60 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc003adc0 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc1f7470 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0044630 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.528+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc1f6860 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.529+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc067470 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.530+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fadc002f050 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.531+0000 7fadc97fa640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 0x7fadb006f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.531+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadc00aa850 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.531+0000 7fadcbfff640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 0x7fadb006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.532+0000 7fadcbfff640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 0x7fadb006f580 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ae2e0 tx=0x7fadb80079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.536+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadc0077400 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.725+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadcc0cd730 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.727+0000 7fadc97fa640 1 -- 172.21.15.107:0/4093048285 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+16579 (secure 0 0 0) 0x7fadc007aa20 con 0x7fadcc0ce6c0 2023-11-24T08:22:55.730 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:55.730 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:52.770317+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 msgr2=0x7fadb006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 0x7fadb006f580 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ae2e0 tx=0x7fadb80079d0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 msgr2=0x7fadcc0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc00046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 shutdown_connections 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fadb006d0c0 0x7fadb006f580 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.730+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadcc15c7f0 0x7fadcc156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.731+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadcc0cf450 0x7fadcc0b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.731+0000 7fadd2b41640 1 --2- 172.21.15.107:0/4093048285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadcc0ce6c0 0x7fadcc0abfc0 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.731+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 >> 172.21.15.107:0/4093048285 conn(0x7fadcc06a920 msgr2=0x7fadcc15eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.731+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 shutdown_connections 2023-11-24T08:22:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.731+0000 7fadd2b41640 1 -- 172.21.15.107:0/4093048285 wait complete. 2023-11-24T08:22:55.753 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:22:55.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:22:55.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.967+0000 7f750afb0640 1 Processor -- start 2023-11-24T08:22:55.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.968+0000 7f750afb0640 1 -- start start 2023-11-24T08:22:55.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.969+0000 7f750afb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.969+0000 7f750afb0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75040cf450 0x7f75040cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.969+0000 7f750afb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040cece0 con 0x7f75040cdfb0 2023-11-24T08:22:55.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.969+0000 7f750afb0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7504156010 con 0x7f75040cf450 2023-11-24T08:22:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.969+0000 7f750afb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750406ce30 con 0x7f750415c7f0 2023-11-24T08:22:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.970+0000 7f7508d4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.970+0000 7f7503fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75040cf450 0x7f75040cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.970+0000 7f7508d4c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:57634/0 (socket says 172.21.15.107:57634) 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.970+0000 7f7508d4c640 1 -- 172.21.15.107:0/2002709921 learned_addr learned my addr 172.21.15.107:0/2002709921 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.971+0000 7f7508d4c640 1 -- 172.21.15.107:0/2002709921 >> v2:172.21.15.120:3300/0 conn(0x7f75040cf450 msgr2=0x7f75040cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.971+0000 7f7508d4c640 1 --2- 172.21.15.107:0/2002709921 >> v2:172.21.15.120:3300/0 conn(0x7f75040cf450 0x7f75040cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.971+0000 7f7508d4c640 1 -- 172.21.15.107:0/2002709921 >> v1:172.21.15.120:6789/0 conn(0x7f75040cdfb0 legacy=0x7f75040ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.971+0000 7f7508d4c640 1 -- 172.21.15.107:0/2002709921 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750406d040 con 0x7f750415c7f0 2023-11-24T08:22:55.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.972+0000 7f7508d4c640 1 --2- 172.21.15.107:0/2002709921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009870 tx=0x7f74f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a001b4871ff4ef6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.972+0000 7f75037fe640 1 -- 172.21.15.107:0/2002709921 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8030840 con 0x7f750415c7f0 2023-11-24T08:22:55.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.972+0000 7f75037fe640 1 -- 172.21.15.107:0/2002709921 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f80309a0 con 0x7f750415c7f0 2023-11-24T08:22:55.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.972+0000 7f75037fe640 1 -- 172.21.15.107:0/2002709921 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8030d30 con 0x7f750415c7f0 2023-11-24T08:22:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 -- 172.21.15.107:0/2002709921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 msgr2=0x7f750406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 --2- 172.21.15.107:0/2002709921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009870 tx=0x7f74f802b980 comp rx=0 tx=0).stop 2023-11-24T08:22:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 -- 172.21.15.107:0/2002709921 shutdown_connections 2023-11-24T08:22:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 --2- 172.21.15.107:0/2002709921 >> v2:172.21.15.120:3300/0 conn(0x7f75040cf450 0x7f75040cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 --2- 172.21.15.107:0/2002709921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f750406db20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 -- 172.21.15.107:0/2002709921 >> 172.21.15.107:0/2002709921 conn(0x7f750406a920 msgr2=0x7f750406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:55.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 -- 172.21.15.107:0/2002709921 shutdown_connections 2023-11-24T08:22:55.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.973+0000 7f750afb0640 1 -- 172.21.15.107:0/2002709921 wait complete. 2023-11-24T08:22:55.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.975+0000 7f750afb0640 1 Processor -- start 2023-11-24T08:22:55.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.975+0000 7f750afb0640 1 -- start start 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.976+0000 7f750afb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 0x7f7504157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750afb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750afb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f75040af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750afb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504151470 con 0x7f75040cf450 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750afb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504151750 con 0x7f75040cdfb0 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750afb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7504158490 con 0x7f750415c7f0 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45988/0 (socket says 172.21.15.107:45988) 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 -- 172.21.15.107:0/3407546758 learned_addr learned my addr 172.21.15.107:0/3407546758 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7508d4c640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 0x7f7504157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750954d640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f75040af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 -- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 msgr2=0x7f75040af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f75040af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 -- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 msgr2=0x7f7504157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 0x7f7504157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7503fff640 1 -- 172.21.15.107:0/3407546758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80094d0 con 0x7f75040cf450 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f750954d640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f75040af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.977+0000 7f7508d4c640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 0x7f7504157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f7503fff640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9e0 tx=0x7f74f000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0009670 con 0x7f75040cf450 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f000c9d0 con 0x7f75040cf450 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504158770 con 0x7f75040cf450 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000cd80 con 0x7f75040cf450 2023-11-24T08:22:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.978+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7504071b00 con 0x7f75040cf450 2023-11-24T08:22:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.979+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7504067470 con 0x7f75040cf450 2023-11-24T08:22:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.980+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f74f0013470 con 0x7f75040cf450 2023-11-24T08:22:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.981+0000 7f75017fa640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 0x7f74e806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.982+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74f0088a90 con 0x7f75040cf450 2023-11-24T08:22:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.982+0000 7f7508d4c640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 0x7f74e806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:55.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.982+0000 7f7508d4c640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 0x7f74e806f550 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f74f8036000 tx=0x7f74f803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:55.986+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74f0055520 con 0x7f75040cf450 2023-11-24T08:22:56.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.175+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f750406e060 con 0x7f74e806d090 2023-11-24T08:22:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.183+0000 7f75017fa640 1 -- 172.21.15.107:0/3407546758 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287587 (secure 0 0 0) 0x7f74f8030b90 con 0x7f74e806d090 2023-11-24T08:22:56.190 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:22:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 msgr2=0x7f74e806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 0x7f74e806f550 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f74f8036000 tx=0x7f74f803b040 comp rx=0 tx=0).stop 2023-11-24T08:22:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 msgr2=0x7f75041466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9e0 tx=0x7f74f000eea0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 shutdown_connections 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f74e806d090 0x7f74e806f550 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f750415c7f0 0x7f75040af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75040cf450 0x7f75041466d0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 --2- 172.21.15.107:0/3407546758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040cdfb0 0x7f7504157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 >> 172.21.15.107:0/3407546758 conn(0x7f750406a920 msgr2=0x7f75040aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 shutdown_connections 2023-11-24T08:22:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.190+0000 7f750afb0640 1 -- 172.21.15.107:0/3407546758 wait complete. 2023-11-24T08:22:56.210 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2023-11-24T08:22:55.777425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":334812,"kb_used_data":22940,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754639908,"statfs":{"total":773094113280,"available":772751265792,"internally_reserved":0,"allocated":23490560,"data_stored":11074504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":319213118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001659"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925468+0000","last_change":"2023-11-24T08:22:51.777727+0000","last_active":"2023-11-24T08:22:52.925468+0000","last_peered":"2023-11-24T08:22:52.925468+0000","last_clean":"2023-11-24T08:22:52.925468+0000","last_became_active":"2023-11-24T08:22:51.777448+0000","last_became_peered":"2023-11-24T08:22:51.777448+0000","last_unstale":"2023-11-24T08:22:52.925468+0000","last_undegraded":"2023-11-24T08:22:52.925468+0000","last_fullsized":"2023-11-24T08:22:52.925468+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:59.480275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924301+0000","last_change":"2023-11-24T08:22:51.776411+0000","last_active":"2023-11-24T08:22:52.924301+0000","last_peered":"2023-11-24T08:22:52.924301+0000","last_clean":"2023-11-24T08:22:52.924301+0000","last_became_active":"2023-11-24T08:22:51.776107+0000","last_became_peered":"2023-11-24T08:22:51.776107+0000","last_unstale":"2023-11-24T08:22:52.924301+0000","last_undegraded":"2023-11-24T08:22:52.924301+0000","last_fullsized":"2023-11-24T08:22:52.924301+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:22.407095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922399+0000","last_change":"2023-11-24T08:22:51.794496+0000","last_active":"2023-11-24T08:22:52.922399+0000","last_peered":"2023-11-24T08:22:52.922399+0000","last_clean":"2023-11-24T08:22:52.922399+0000","last_became_active":"2023-11-24T08:22:51.794173+0000","last_became_peered":"2023-11-24T08:22:51.794173+0000","last_unstale":"2023-11-24T08:22:52.922399+0000","last_undegraded":"2023-11-24T08:22:52.922399+0000","last_fullsized":"2023-11-24T08:22:52.922399+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:26.274775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922986+0000","last_change":"2023-11-24T08:22:51.804163+0000","last_active":"2023-11-24T08:22:52.922986+0000","last_peered":"2023-11-24T08:22:52.922986+0000","last_clean":"2023-11-24T08:22:52.922986+0000","last_became_active":"2023-11-24T08:22:51.803902+0000","last_became_peered":"2023-11-24T08:22:51.803902+0000","last_unstale":"2023-11-24T08:22:52.922986+0000","last_undegraded":"2023-11-24T08:22:52.922986+0000","last_fullsized":"2023-11-24T08:22:52.922986+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:04:12.102399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925564+0000","last_change":"2023-11-24T08:22:51.797757+0000","last_active":"2023-11-24T08:22:52.925564+0000","last_peered":"2023-11-24T08:22:52.925564+0000","last_clean":"2023-11-24T08:22:52.925564+0000","last_became_active":"2023-11-24T08:22:51.793587+0000","last_became_peered":"2023-11-24T08:22:51.793587+0000","last_unstale":"2023-11-24T08:22:52.925564+0000","last_undegraded":"2023-11-24T08:22:52.925564+0000","last_fullsized":"2023-11-24T08:22:52.925564+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:38.507421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925027+0000","last_change":"2023-11-24T08:22:51.786858+0000","last_active":"2023-11-24T08:22:52.925027+0000","last_peered":"2023-11-24T08:22:52.925027+0000","last_clean":"2023-11-24T08:22:52.925027+0000","last_became_active":"2023-11-24T08:22:51.786553+0000","last_became_peered":"2023-11-24T08:22:51.786553+0000","last_unstale":"2023-11-24T08:22:52.925027+0000","last_undegraded":"2023-11-24T08:22:52.925027+0000","last_fullsized":"2023-11-24T08:22:52.925027+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:52.058301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923599+0000","last_change":"2023-11-24T08:22:51.802586+0000","last_active":"2023-11-24T08:22:52.923599+0000","last_peered":"2023-11-24T08:22:52.923599+0000","last_clean":"2023-11-24T08:22:52.923599+0000","last_became_active":"2023-11-24T08:22:51.802020+0000","last_became_peered":"2023-11-24T08:22:51.802020+0000","last_unstale":"2023-11-24T08:22:52.923599+0000","last_undegraded":"2023-11-24T08:22:52.923599+0000","last_fullsized":"2023-11-24T08:22:52.923599+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:30:18.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924681+0000","last_change":"2023-11-24T08:22:51.803602+0000","last_active":"2023-11-24T08:22:52.924681+0000","last_peered":"2023-11-24T08:22:52.924681+0000","last_clean":"2023-11-24T08:22:52.924681+0000","last_became_active":"2023-11-24T08:22:51.803301+0000","last_became_peered":"2023-11-24T08:22:51.803301+0000","last_unstale":"2023-11-24T08:22:52.924681+0000","last_undegraded":"2023-11-24T08:22:52.924681+0000","last_fullsized":"2023-11-24T08:22:52.924681+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:40.548660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925917+0000","last_change":"2023-11-24T08:22:51.793706+0000","last_active":"2023-11-24T08:22:52.925917+0000","last_peered":"2023-11-24T08:22:52.925917+0000","last_clean":"2023-11-24T08:22:52.925917+0000","last_became_active":"2023-11-24T08:22:51.792710+0000","last_became_peered":"2023-11-24T08:22:51.792710+0000","last_unstale":"2023-11-24T08:22:52.925917+0000","last_undegraded":"2023-11-24T08:22:52.925917+0000","last_fullsized":"2023-11-24T08:22:52.925917+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:06:59.522187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922228+0000","last_change":"2023-11-24T08:22:51.801479+0000","last_active":"2023-11-24T08:22:52.922228+0000","last_peered":"2023-11-24T08:22:52.922228+0000","last_clean":"2023-11-24T08:22:52.922228+0000","last_became_active":"2023-11-24T08:22:51.799551+0000","last_became_peered":"2023-11-24T08:22:51.799551+0000","last_unstale":"2023-11-24T08:22:52.922228+0000","last_undegraded":"2023-11-24T08:22:52.922228+0000","last_fullsized":"2023-11-24T08:22:52.922228+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:07.946995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922633+0000","last_change":"2023-11-24T08:22:51.800282+0000","last_active":"2023-11-24T08:22:52.922633+0000","last_peered":"2023-11-24T08:22:52.922633+0000","last_clean":"2023-11-24T08:22:52.922633+0000","last_became_active":"2023-11-24T08:22:51.799894+0000","last_became_peered":"2023-11-24T08:22:51.799894+0000","last_unstale":"2023-11-24T08:22:52.922633+0000","last_undegraded":"2023-11-24T08:22:52.922633+0000","last_fullsized":"2023-11-24T08:22:52.922633+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:06.821388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925262+0000","last_change":"2023-11-24T08:22:51.775736+0000","last_active":"2023-11-24T08:22:52.925262+0000","last_peered":"2023-11-24T08:22:52.925262+0000","last_clean":"2023-11-24T08:22:52.925262+0000","last_became_active":"2023-11-24T08:22:51.775470+0000","last_became_peered":"2023-11-24T08:22:51.775470+0000","last_unstale":"2023-11-24T08:22:52.925262+0000","last_undegraded":"2023-11-24T08:22:52.925262+0000","last_fullsized":"2023-11-24T08:22:52.925262+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:03.904749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788333+0000","last_change":"2023-11-24T08:22:51.781032+0000","last_active":"2023-11-24T08:22:52.788333+0000","last_peered":"2023-11-24T08:22:52.788333+0000","last_clean":"2023-11-24T08:22:52.788333+0000","last_became_active":"2023-11-24T08:22:51.780696+0000","last_became_peered":"2023-11-24T08:22:51.780696+0000","last_unstale":"2023-11-24T08:22:52.788333+0000","last_undegraded":"2023-11-24T08:22:52.788333+0000","last_fullsized":"2023-11-24T08:22:52.788333+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:10.613046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784870+0000","last_change":"2023-11-24T08:22:51.771728+0000","last_active":"2023-11-24T08:22:52.784870+0000","last_peered":"2023-11-24T08:22:52.784870+0000","last_clean":"2023-11-24T08:22:52.784870+0000","last_became_active":"2023-11-24T08:22:51.771420+0000","last_became_peered":"2023-11-24T08:22:51.771420+0000","last_unstale":"2023-11-24T08:22:52.784870+0000","last_undegraded":"2023-11-24T08:22:52.784870+0000","last_fullsized":"2023-11-24T08:22:52.784870+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:37:52.623774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925867+0000","last_change":"2023-11-24T08:22:51.789133+0000","last_active":"2023-11-24T08:22:52.925867+0000","last_peered":"2023-11-24T08:22:52.925867+0000","last_clean":"2023-11-24T08:22:52.925867+0000","last_became_active":"2023-11-24T08:22:51.788839+0000","last_became_peered":"2023-11-24T08:22:51.788839+0000","last_unstale":"2023-11-24T08:22:52.925867+0000","last_undegraded":"2023-11-24T08:22:52.925867+0000","last_fullsized":"2023-11-24T08:22:52.925867+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:46:16.777665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921682+0000","last_change":"2023-11-24T08:22:51.775096+0000","last_active":"2023-11-24T08:22:52.921682+0000","last_peered":"2023-11-24T08:22:52.921682+0000","last_clean":"2023-11-24T08:22:52.921682+0000","last_became_active":"2023-11-24T08:22:51.774737+0000","last_became_peered":"2023-11-24T08:22:51.774737+0000","last_unstale":"2023-11-24T08:22:52.921682+0000","last_undegraded":"2023-11-24T08:22:52.921682+0000","last_fullsized":"2023-11-24T08:22:52.921682+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:14:26.577628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789366+0000","last_change":"2023-11-24T08:22:51.794566+0000","last_active":"2023-11-24T08:22:52.789366+0000","last_peered":"2023-11-24T08:22:52.789366+0000","last_clean":"2023-11-24T08:22:52.789366+0000","last_became_active":"2023-11-24T08:22:51.794242+0000","last_became_peered":"2023-11-24T08:22:51.794242+0000","last_unstale":"2023-11-24T08:22:52.789366+0000","last_undegraded":"2023-11-24T08:22:52.789366+0000","last_fullsized":"2023-11-24T08:22:52.789366+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:29:54.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922364+0000","last_change":"2023-11-24T08:22:51.784286+0000","last_active":"2023-11-24T08:22:52.922364+0000","last_peered":"2023-11-24T08:22:52.922364+0000","last_clean":"2023-11-24T08:22:52.922364+0000","last_became_active":"2023-11-24T08:22:51.783974+0000","last_became_peered":"2023-11-24T08:22:51.783974+0000","last_unstale":"2023-11-24T08:22:52.922364+0000","last_undegraded":"2023-11-24T08:22:52.922364+0000","last_fullsized":"2023-11-24T08:22:52.922364+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:02:26.041628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926522+0000","last_change":"2023-11-24T08:22:51.799070+0000","last_active":"2023-11-24T08:22:52.926522+0000","last_peered":"2023-11-24T08:22:52.926522+0000","last_clean":"2023-11-24T08:22:52.926522+0000","last_became_active":"2023-11-24T08:22:51.798655+0000","last_became_peered":"2023-11-24T08:22:51.798655+0000","last_unstale":"2023-11-24T08:22:52.926522+0000","last_undegraded":"2023-11-24T08:22:52.926522+0000","last_fullsized":"2023-11-24T08:22:52.926522+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:37:52.623774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923554+0000","last_change":"2023-11-24T08:22:51.793936+0000","last_active":"2023-11-24T08:22:52.923554+0000","last_peered":"2023-11-24T08:22:52.923554+0000","last_clean":"2023-11-24T08:22:52.923554+0000","last_became_active":"2023-11-24T08:22:51.793685+0000","last_became_peered":"2023-11-24T08:22:51.793685+0000","last_unstale":"2023-11-24T08:22:52.923554+0000","last_undegraded":"2023-11-24T08:22:52.923554+0000","last_fullsized":"2023-11-24T08:22:52.923554+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:36.568123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923269+0000","last_change":"2023-11-24T08:22:51.792098+0000","last_active":"2023-11-24T08:22:52.923269+0000","last_peered":"2023-11-24T08:22:52.923269+0000","last_clean":"2023-11-24T08:22:52.923269+0000","last_became_active":"2023-11-24T08:22:51.791788+0000","last_became_peered":"2023-11-24T08:22:51.791788+0000","last_unstale":"2023-11-24T08:22:52.923269+0000","last_undegraded":"2023-11-24T08:22:52.923269+0000","last_fullsized":"2023-11-24T08:22:52.923269+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:54.595128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788099+0000","last_change":"2023-11-24T08:22:51.778589+0000","last_active":"2023-11-24T08:22:52.788099+0000","last_peered":"2023-11-24T08:22:52.788099+0000","last_clean":"2023-11-24T08:22:52.788099+0000","last_became_active":"2023-11-24T08:22:51.773828+0000","last_became_peered":"2023-11-24T08:22:51.773828+0000","last_unstale":"2023-11-24T08:22:52.788099+0000","last_undegraded":"2023-11-24T08:22:52.788099+0000","last_fullsized":"2023-11-24T08:22:52.788099+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:21.899461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926193+0000","last_change":"2023-11-24T08:22:51.793063+0000","last_active":"2023-11-24T08:22:52.926193+0000","last_peered":"2023-11-24T08:22:52.926193+0000","last_clean":"2023-11-24T08:22:52.926193+0000","last_became_active":"2023-11-24T08:22:51.792275+0000","last_became_peered":"2023-11-24T08:22:51.792275+0000","last_unstale":"2023-11-24T08:22:52.926193+0000","last_undegraded":"2023-11-24T08:22:52.926193+0000","last_fullsized":"2023-11-24T08:22:52.926193+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:31.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787304+0000","last_change":"2023-11-24T08:22:51.791377+0000","last_active":"2023-11-24T08:22:52.787304+0000","last_peered":"2023-11-24T08:22:52.787304+0000","last_clean":"2023-11-24T08:22:52.787304+0000","last_became_active":"2023-11-24T08:22:51.789372+0000","last_became_peered":"2023-11-24T08:22:51.789372+0000","last_unstale":"2023-11-24T08:22:52.787304+0000","last_undegraded":"2023-11-24T08:22:52.787304+0000","last_fullsized":"2023-11-24T08:22:52.787304+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:30.547159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924042+0000","last_change":"2023-11-24T08:22:51.770004+0000","last_active":"2023-11-24T08:22:52.924042+0000","last_peered":"2023-11-24T08:22:52.924042+0000","last_clean":"2023-11-24T08:22:52.924042+0000","last_became_active":"2023-11-24T08:22:51.769733+0000","last_became_peered":"2023-11-24T08:22:51.769733+0000","last_unstale":"2023-11-24T08:22:52.924042+0000","last_undegraded":"2023-11-24T08:22:52.924042+0000","last_fullsized":"2023-11-24T08:22:52.924042+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.396929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923893+0000","last_change":"2023-11-24T08:22:51.799243+0000","last_active":"2023-11-24T08:22:52.923893+0000","last_peered":"2023-11-24T08:22:52.923893+0000","last_clean":"2023-11-24T08:22:52.923893+0000","last_became_active":"2023-11-24T08:22:51.798774+0000","last_became_peered":"2023-11-24T08:22:51.798774+0000","last_unstale":"2023-11-24T08:22:52.923893+0000","last_undegraded":"2023-11-24T08:22:52.923893+0000","last_fullsized":"2023-11-24T08:22:52.923893+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:22.315999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925547+0000","last_change":"2023-11-24T08:22:51.797353+0000","last_active":"2023-11-24T08:22:52.925547+0000","last_peered":"2023-11-24T08:22:52.925547+0000","last_clean":"2023-11-24T08:22:52.925547+0000","last_became_active":"2023-11-24T08:22:51.797000+0000","last_became_peered":"2023-11-24T08:22:51.797000+0000","last_unstale":"2023-11-24T08:22:52.925547+0000","last_undegraded":"2023-11-24T08:22:52.925547+0000","last_fullsized":"2023-11-24T08:22:52.925547+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:44.412249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925514+0000","last_change":"2023-11-24T08:22:51.798827+0000","last_active":"2023-11-24T08:22:52.925514+0000","last_peered":"2023-11-24T08:22:52.925514+0000","last_clean":"2023-11-24T08:22:52.925514+0000","last_became_active":"2023-11-24T08:22:51.798565+0000","last_became_peered":"2023-11-24T08:22:51.798565+0000","last_unstale":"2023-11-24T08:22:52.925514+0000","last_undegraded":"2023-11-24T08:22:52.925514+0000","last_fullsized":"2023-11-24T08:22:52.925514+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:48.991722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.785175+0000","last_change":"2023-11-24T08:22:51.774282+0000","last_active":"2023-11-24T08:22:52.785175+0000","last_peered":"2023-11-24T08:22:52.785175+0000","last_clean":"2023-11-24T08:22:52.785175+0000","last_became_active":"2023-11-24T08:22:51.773842+0000","last_became_peered":"2023-11-24T08:22:51.773842+0000","last_unstale":"2023-11-24T08:22:52.785175+0000","last_undegraded":"2023-11-24T08:22:52.785175+0000","last_fullsized":"2023-11-24T08:22:52.785175+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:29:19.650715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784876+0000","last_change":"2023-11-24T08:22:51.793901+0000","last_active":"2023-11-24T08:22:52.784876+0000","last_peered":"2023-11-24T08:22:52.784876+0000","last_clean":"2023-11-24T08:22:52.784876+0000","last_became_active":"2023-11-24T08:22:51.793589+0000","last_became_peered":"2023-11-24T08:22:51.793589+0000","last_unstale":"2023-11-24T08:22:52.784876+0000","last_undegraded":"2023-11-24T08:22:52.784876+0000","last_fullsized":"2023-11-24T08:22:52.784876+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:03:54.450966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922825+0000","last_change":"2023-11-24T08:22:51.792690+0000","last_active":"2023-11-24T08:22:52.922825+0000","last_peered":"2023-11-24T08:22:52.922825+0000","last_clean":"2023-11-24T08:22:52.922825+0000","last_became_active":"2023-11-24T08:22:51.792440+0000","last_became_peered":"2023-11-24T08:22:51.792440+0000","last_unstale":"2023-11-24T08:22:52.922825+0000","last_undegraded":"2023-11-24T08:22:52.922825+0000","last_fullsized":"2023-11-24T08:22:52.922825+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:59:27.056912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924643+0000","last_change":"2023-11-24T08:22:51.799593+0000","last_active":"2023-11-24T08:22:52.924643+0000","last_peered":"2023-11-24T08:22:52.924643+0000","last_clean":"2023-11-24T08:22:52.924643+0000","last_became_active":"2023-11-24T08:22:51.799310+0000","last_became_peered":"2023-11-24T08:22:51.799310+0000","last_unstale":"2023-11-24T08:22:52.924643+0000","last_undegraded":"2023-11-24T08:22:52.924643+0000","last_fullsized":"2023-11-24T08:22:52.924643+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:22:45.012196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921984+0000","last_change":"2023-11-24T08:22:51.773425+0000","last_active":"2023-11-24T08:22:52.921984+0000","last_peered":"2023-11-24T08:22:52.921984+0000","last_clean":"2023-11-24T08:22:52.921984+0000","last_became_active":"2023-11-24T08:22:51.773104+0000","last_became_peered":"2023-11-24T08:22:51.773104+0000","last_unstale":"2023-11-24T08:22:52.921984+0000","last_undegraded":"2023-11-24T08:22:52.921984+0000","last_fullsized":"2023-11-24T08:22:52.921984+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:22:33.620021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"219'4","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925639+0000","last_change":"2023-11-24T08:22:50.781535+0000","last_active":"2023-11-24T08:22:52.925639+0000","last_peered":"2023-11-24T08:22:52.925639+0000","last_clean":"2023-11-24T08:22:52.925639+0000","last_became_active":"2023-11-24T08:22:50.781248+0000","last_became_peered":"2023-11-24T08:22:50.781248+0000","last_unstale":"2023-11-24T08:22:52.925639+0000","last_undegraded":"2023-11-24T08:22:52.925639+0000","last_fullsized":"2023-11-24T08:22:52.925639+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:33.342667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926371+0000","last_change":"2023-11-24T08:22:51.799930+0000","last_active":"2023-11-24T08:22:52.926371+0000","last_peered":"2023-11-24T08:22:52.926371+0000","last_clean":"2023-11-24T08:22:52.926371+0000","last_became_active":"2023-11-24T08:22:51.799619+0000","last_became_peered":"2023-11-24T08:22:51.799619+0000","last_unstale":"2023-11-24T08:22:52.926371+0000","last_undegraded":"2023-11-24T08:22:52.926371+0000","last_fullsized":"2023-11-24T08:22:52.926371+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788781+0000","last_change":"2023-11-24T08:22:50.778878+0000","last_active":"2023-11-24T08:22:52.788781+0000","last_peered":"2023-11-24T08:22:52.788781+0000","last_clean":"2023-11-24T08:22:52.788781+0000","last_became_active":"2023-11-24T08:22:50.776758+0000","last_became_peered":"2023-11-24T08:22:50.776758+0000","last_unstale":"2023-11-24T08:22:52.788781+0000","last_undegraded":"2023-11-24T08:22:52.788781+0000","last_fullsized":"2023-11-24T08:22:52.788781+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:51.659657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925713+0000","last_change":"2023-11-24T08:22:51.787267+0000","last_active":"2023-11-24T08:22:52.925713+0000","last_peered":"2023-11-24T08:22:52.925713+0000","last_clean":"2023-11-24T08:22:52.925713+0000","last_became_active":"2023-11-24T08:22:51.785985+0000","last_became_peered":"2023-11-24T08:22:51.785985+0000","last_unstale":"2023-11-24T08:22:52.925713+0000","last_undegraded":"2023-11-24T08:22:52.925713+0000","last_fullsized":"2023-11-24T08:22:52.925713+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:24.572297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784670+0000","last_change":"2023-11-24T08:22:50.779628+0000","last_active":"2023-11-24T08:22:52.784670+0000","last_peered":"2023-11-24T08:22:52.784670+0000","last_clean":"2023-11-24T08:22:52.784670+0000","last_became_active":"2023-11-24T08:22:50.779381+0000","last_became_peered":"2023-11-24T08:22:50.779381+0000","last_unstale":"2023-11-24T08:22:52.784670+0000","last_undegraded":"2023-11-24T08:22:52.784670+0000","last_fullsized":"2023-11-24T08:22:52.784670+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:46:15.759319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784583+0000","last_change":"2023-11-24T08:22:51.779434+0000","last_active":"2023-11-24T08:22:52.784583+0000","last_peered":"2023-11-24T08:22:52.784583+0000","last_clean":"2023-11-24T08:22:52.784583+0000","last_became_active":"2023-11-24T08:22:51.777412+0000","last_became_peered":"2023-11-24T08:22:51.777412+0000","last_unstale":"2023-11-24T08:22:52.784583+0000","last_undegraded":"2023-11-24T08:22:52.784583+0000","last_fullsized":"2023-11-24T08:22:52.784583+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924005+0000","last_change":"2023-11-24T08:22:50.778954+0000","last_active":"2023-11-24T08:22:52.924005+0000","last_peered":"2023-11-24T08:22:52.924005+0000","last_clean":"2023-11-24T08:22:52.924005+0000","last_became_active":"2023-11-24T08:22:50.778634+0000","last_became_peered":"2023-11-24T08:22:50.778634+0000","last_unstale":"2023-11-24T08:22:52.924005+0000","last_undegraded":"2023-11-24T08:22:52.924005+0000","last_fullsized":"2023-11-24T08:22:52.924005+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:43.393903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924200+0000","last_change":"2023-11-24T08:22:51.792720+0000","last_active":"2023-11-24T08:22:52.924200+0000","last_peered":"2023-11-24T08:22:52.924200+0000","last_clean":"2023-11-24T08:22:52.924200+0000","last_became_active":"2023-11-24T08:22:51.790380+0000","last_became_peered":"2023-11-24T08:22:51.790380+0000","last_unstale":"2023-11-24T08:22:52.924200+0000","last_undegraded":"2023-11-24T08:22:52.924200+0000","last_fullsized":"2023-11-24T08:22:52.924200+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:01.821084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784976+0000","last_change":"2023-11-24T08:22:50.774941+0000","last_active":"2023-11-24T08:22:52.784976+0000","last_peered":"2023-11-24T08:22:52.784976+0000","last_clean":"2023-11-24T08:22:52.784976+0000","last_became_active":"2023-11-24T08:22:50.774679+0000","last_became_peered":"2023-11-24T08:22:50.774679+0000","last_unstale":"2023-11-24T08:22:52.784976+0000","last_undegraded":"2023-11-24T08:22:52.784976+0000","last_fullsized":"2023-11-24T08:22:52.784976+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:14.978362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922087+0000","last_change":"2023-11-24T08:22:51.804145+0000","last_active":"2023-11-24T08:22:52.922087+0000","last_peered":"2023-11-24T08:22:52.922087+0000","last_clean":"2023-11-24T08:22:52.922087+0000","last_became_active":"2023-11-24T08:22:51.803860+0000","last_became_peered":"2023-11-24T08:22:51.803860+0000","last_unstale":"2023-11-24T08:22:52.922087+0000","last_undegraded":"2023-11-24T08:22:52.922087+0000","last_fullsized":"2023-11-24T08:22:52.922087+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:06.393248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921290+0000","last_change":"2023-11-24T08:22:50.775057+0000","last_active":"2023-11-24T08:22:52.921290+0000","last_peered":"2023-11-24T08:22:52.921290+0000","last_clean":"2023-11-24T08:22:52.921290+0000","last_became_active":"2023-11-24T08:22:50.774786+0000","last_became_peered":"2023-11-24T08:22:50.774786+0000","last_unstale":"2023-11-24T08:22:52.921290+0000","last_undegraded":"2023-11-24T08:22:52.921290+0000","last_fullsized":"2023-11-24T08:22:52.921290+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:16.893872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926232+0000","last_change":"2023-11-24T08:22:51.796536+0000","last_active":"2023-11-24T08:22:52.926232+0000","last_peered":"2023-11-24T08:22:52.926232+0000","last_clean":"2023-11-24T08:22:52.926232+0000","last_became_active":"2023-11-24T08:22:51.793407+0000","last_became_peered":"2023-11-24T08:22:51.793407+0000","last_unstale":"2023-11-24T08:22:52.926232+0000","last_undegraded":"2023-11-24T08:22:52.926232+0000","last_fullsized":"2023-11-24T08:22:52.926232+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:33.590373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924665+0000","last_change":"2023-11-24T08:22:50.773854+0000","last_active":"2023-11-24T08:22:52.924665+0000","last_peered":"2023-11-24T08:22:52.924665+0000","last_clean":"2023-11-24T08:22:52.924665+0000","last_became_active":"2023-11-24T08:22:50.773253+0000","last_became_peered":"2023-11-24T08:22:50.773253+0000","last_unstale":"2023-11-24T08:22:52.924665+0000","last_undegraded":"2023-11-24T08:22:52.924665+0000","last_fullsized":"2023-11-24T08:22:52.924665+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:49.360836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923617+0000","last_change":"2023-11-24T08:22:51.785163+0000","last_active":"2023-11-24T08:22:52.923617+0000","last_peered":"2023-11-24T08:22:52.923617+0000","last_clean":"2023-11-24T08:22:52.923617+0000","last_became_active":"2023-11-24T08:22:51.784863+0000","last_became_peered":"2023-11-24T08:22:51.784863+0000","last_unstale":"2023-11-24T08:22:52.923617+0000","last_undegraded":"2023-11-24T08:22:52.923617+0000","last_fullsized":"2023-11-24T08:22:52.923617+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:20.243410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784429+0000","last_change":"2023-11-24T08:22:50.781824+0000","last_active":"2023-11-24T08:22:52.784429+0000","last_peered":"2023-11-24T08:22:52.784429+0000","last_clean":"2023-11-24T08:22:52.784429+0000","last_became_active":"2023-11-24T08:22:50.781414+0000","last_became_peered":"2023-11-24T08:22:50.781414+0000","last_unstale":"2023-11-24T08:22:52.784429+0000","last_undegraded":"2023-11-24T08:22:52.784429+0000","last_fullsized":"2023-11-24T08:22:52.784429+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:48.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787394+0000","last_change":"2023-11-24T08:22:51.776040+0000","last_active":"2023-11-24T08:22:52.787394+0000","last_peered":"2023-11-24T08:22:52.787394+0000","last_clean":"2023-11-24T08:22:52.787394+0000","last_became_active":"2023-11-24T08:22:51.773527+0000","last_became_peered":"2023-11-24T08:22:51.773527+0000","last_unstale":"2023-11-24T08:22:52.787394+0000","last_undegraded":"2023-11-24T08:22:52.787394+0000","last_fullsized":"2023-11-24T08:22:52.787394+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:44.597552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922466+0000","last_change":"2023-11-24T08:22:50.781407+0000","last_active":"2023-11-24T08:22:52.922466+0000","last_peered":"2023-11-24T08:22:52.922466+0000","last_clean":"2023-11-24T08:22:52.922466+0000","last_became_active":"2023-11-24T08:22:50.781148+0000","last_became_peered":"2023-11-24T08:22:50.781148+0000","last_unstale":"2023-11-24T08:22:52.922466+0000","last_undegraded":"2023-11-24T08:22:52.922466+0000","last_fullsized":"2023-11-24T08:22:52.922466+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:13:55.550456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921681+0000","last_change":"2023-11-24T08:22:51.799971+0000","last_active":"2023-11-24T08:22:52.921681+0000","last_peered":"2023-11-24T08:22:52.921681+0000","last_clean":"2023-11-24T08:22:52.921681+0000","last_became_active":"2023-11-24T08:22:51.799676+0000","last_became_peered":"2023-11-24T08:22:51.799676+0000","last_unstale":"2023-11-24T08:22:52.921681+0000","last_undegraded":"2023-11-24T08:22:52.921681+0000","last_fullsized":"2023-11-24T08:22:52.921681+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:14.080962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"219'3","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925521+0000","last_change":"2023-11-24T08:22:50.783856+0000","last_active":"2023-11-24T08:22:52.925521+0000","last_peered":"2023-11-24T08:22:52.925521+0000","last_clean":"2023-11-24T08:22:52.925521+0000","last_became_active":"2023-11-24T08:22:50.783314+0000","last_became_peered":"2023-11-24T08:22:50.783314+0000","last_unstale":"2023-11-24T08:22:52.925521+0000","last_undegraded":"2023-11-24T08:22:52.925521+0000","last_fullsized":"2023-11-24T08:22:52.925521+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:23:48.657613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922946+0000","last_change":"2023-11-24T08:22:51.797312+0000","last_active":"2023-11-24T08:22:52.922946+0000","last_peered":"2023-11-24T08:22:52.922946+0000","last_clean":"2023-11-24T08:22:52.922946+0000","last_became_active":"2023-11-24T08:22:51.795064+0000","last_became_peered":"2023-11-24T08:22:51.795064+0000","last_unstale":"2023-11-24T08:22:52.922946+0000","last_undegraded":"2023-11-24T08:22:52.922946+0000","last_fullsized":"2023-11-24T08:22:52.922946+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:27:38.465084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"219'6","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925408+0000","last_change":"2023-11-24T08:22:50.781012+0000","last_active":"2023-11-24T08:22:52.925408+0000","last_peered":"2023-11-24T08:22:52.925408+0000","last_clean":"2023-11-24T08:22:52.925408+0000","last_became_active":"2023-11-24T08:22:50.780110+0000","last_became_peered":"2023-11-24T08:22:50.780110+0000","last_unstale":"2023-11-24T08:22:52.925408+0000","last_undegraded":"2023-11-24T08:22:52.925408+0000","last_fullsized":"2023-11-24T08:22:52.925408+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:38:50.896631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923426+0000","last_change":"2023-11-24T08:22:51.782867+0000","last_active":"2023-11-24T08:22:52.923426+0000","last_peered":"2023-11-24T08:22:52.923426+0000","last_clean":"2023-11-24T08:22:52.923426+0000","last_became_active":"2023-11-24T08:22:51.782516+0000","last_became_peered":"2023-11-24T08:22:51.782516+0000","last_unstale":"2023-11-24T08:22:52.923426+0000","last_undegraded":"2023-11-24T08:22:52.923426+0000","last_fullsized":"2023-11-24T08:22:52.923426+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:06.822674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925789+0000","last_change":"2023-11-24T08:22:50.776490+0000","last_active":"2023-11-24T08:22:52.925789+0000","last_peered":"2023-11-24T08:22:52.925789+0000","last_clean":"2023-11-24T08:22:52.925789+0000","last_became_active":"2023-11-24T08:22:50.776208+0000","last_became_peered":"2023-11-24T08:22:50.776208+0000","last_unstale":"2023-11-24T08:22:52.925789+0000","last_undegraded":"2023-11-24T08:22:52.925789+0000","last_fullsized":"2023-11-24T08:22:52.925789+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:37.489075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924304+0000","last_change":"2023-11-24T08:22:51.791619+0000","last_active":"2023-11-24T08:22:52.924304+0000","last_peered":"2023-11-24T08:22:52.924304+0000","last_clean":"2023-11-24T08:22:52.924304+0000","last_became_active":"2023-11-24T08:22:51.790781+0000","last_became_peered":"2023-11-24T08:22:51.790781+0000","last_unstale":"2023-11-24T08:22:52.924304+0000","last_undegraded":"2023-11-24T08:22:52.924304+0000","last_fullsized":"2023-11-24T08:22:52.924304+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:13:56.568802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925247+0000","last_change":"2023-11-24T08:22:50.786246+0000","last_active":"2023-11-24T08:22:52.925247+0000","last_peered":"2023-11-24T08:22:52.925247+0000","last_clean":"2023-11-24T08:22:52.925247+0000","last_became_active":"2023-11-24T08:22:50.785939+0000","last_became_peered":"2023-11-24T08:22:50.785939+0000","last_unstale":"2023-11-24T08:22:52.925247+0000","last_undegraded":"2023-11-24T08:22:52.925247+0000","last_fullsized":"2023-11-24T08:22:52.925247+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:03.310218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926642+0000","last_change":"2023-11-24T08:22:51.796637+0000","last_active":"2023-11-24T08:22:52.926642+0000","last_peered":"2023-11-24T08:22:52.926642+0000","last_clean":"2023-11-24T08:22:52.926642+0000","last_became_active":"2023-11-24T08:22:51.796304+0000","last_became_peered":"2023-11-24T08:22:51.796304+0000","last_unstale":"2023-11-24T08:22:52.926642+0000","last_undegraded":"2023-11-24T08:22:52.926642+0000","last_fullsized":"2023-11-24T08:22:52.926642+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:03:54.450966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924743+0000","last_change":"2023-11-24T08:22:50.779231+0000","last_active":"2023-11-24T08:22:52.924743+0000","last_peered":"2023-11-24T08:22:52.924743+0000","last_clean":"2023-11-24T08:22:52.924743+0000","last_became_active":"2023-11-24T08:22:50.778964+0000","last_became_peered":"2023-11-24T08:22:50.778964+0000","last_unstale":"2023-11-24T08:22:52.924743+0000","last_undegraded":"2023-11-24T08:22:52.924743+0000","last_fullsized":"2023-11-24T08:22:52.924743+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:24:00.337076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926313+0000","last_change":"2023-11-24T08:22:51.806877+0000","last_active":"2023-11-24T08:22:52.926313+0000","last_peered":"2023-11-24T08:22:52.926313+0000","last_clean":"2023-11-24T08:22:52.926313+0000","last_became_active":"2023-11-24T08:22:51.806471+0000","last_became_peered":"2023-11-24T08:22:51.806471+0000","last_unstale":"2023-11-24T08:22:52.926313+0000","last_undegraded":"2023-11-24T08:22:52.926313+0000","last_fullsized":"2023-11-24T08:22:52.926313+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:15.996708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"219'8","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922264+0000","last_change":"2023-11-24T08:22:50.775105+0000","last_active":"2023-11-24T08:22:52.922264+0000","last_peered":"2023-11-24T08:22:52.922264+0000","last_clean":"2023-11-24T08:22:52.922264+0000","last_became_active":"2023-11-24T08:22:50.774763+0000","last_became_peered":"2023-11-24T08:22:50.774763+0000","last_unstale":"2023-11-24T08:22:52.922264+0000","last_undegraded":"2023-11-24T08:22:52.922264+0000","last_fullsized":"2023-11-24T08:22:52.922264+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:24.996177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923293+0000","last_change":"2023-11-24T08:22:51.806834+0000","last_active":"2023-11-24T08:22:52.923293+0000","last_peered":"2023-11-24T08:22:52.923293+0000","last_clean":"2023-11-24T08:22:52.923293+0000","last_became_active":"2023-11-24T08:22:51.806537+0000","last_became_peered":"2023-11-24T08:22:51.806537+0000","last_unstale":"2023-11-24T08:22:52.923293+0000","last_undegraded":"2023-11-24T08:22:52.923293+0000","last_fullsized":"2023-11-24T08:22:52.923293+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:54:16.344938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925902+0000","last_change":"2023-11-24T08:22:50.785703+0000","last_active":"2023-11-24T08:22:52.925902+0000","last_peered":"2023-11-24T08:22:52.925902+0000","last_clean":"2023-11-24T08:22:52.925902+0000","last_became_active":"2023-11-24T08:22:50.785376+0000","last_became_peered":"2023-11-24T08:22:50.785376+0000","last_unstale":"2023-11-24T08:22:52.925902+0000","last_undegraded":"2023-11-24T08:22:52.925902+0000","last_fullsized":"2023-11-24T08:22:52.925902+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:12:53.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925406+0000","last_change":"2023-11-24T08:22:51.799712+0000","last_active":"2023-11-24T08:22:52.925406+0000","last_peered":"2023-11-24T08:22:52.925406+0000","last_clean":"2023-11-24T08:22:52.925406+0000","last_became_active":"2023-11-24T08:22:51.797643+0000","last_became_peered":"2023-11-24T08:22:51.797643+0000","last_unstale":"2023-11-24T08:22:52.925406+0000","last_undegraded":"2023-11-24T08:22:52.925406+0000","last_fullsized":"2023-11-24T08:22:52.925406+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:59:04.078503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924858+0000","last_change":"2023-11-24T08:22:50.780404+0000","last_active":"2023-11-24T08:22:52.924858+0000","last_peered":"2023-11-24T08:22:52.924858+0000","last_clean":"2023-11-24T08:22:52.924858+0000","last_became_active":"2023-11-24T08:22:50.780135+0000","last_became_peered":"2023-11-24T08:22:50.780135+0000","last_unstale":"2023-11-24T08:22:52.924858+0000","last_undegraded":"2023-11-24T08:22:52.924858+0000","last_fullsized":"2023-11-24T08:22:52.924858+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:32:31.491498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923228+0000","last_change":"2023-11-24T08:22:51.800842+0000","last_active":"2023-11-24T08:22:52.923228+0000","last_peered":"2023-11-24T08:22:52.923228+0000","last_clean":"2023-11-24T08:22:52.923228+0000","last_became_active":"2023-11-24T08:22:51.800557+0000","last_became_peered":"2023-11-24T08:22:51.800557+0000","last_unstale":"2023-11-24T08:22:52.923228+0000","last_undegraded":"2023-11-24T08:22:52.923228+0000","last_fullsized":"2023-11-24T08:22:52.923228+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:47.929727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789148+0000","last_change":"2023-11-24T08:22:50.778673+0000","last_active":"2023-11-24T08:22:52.789148+0000","last_peered":"2023-11-24T08:22:52.789148+0000","last_clean":"2023-11-24T08:22:52.789148+0000","last_became_active":"2023-11-24T08:22:50.776374+0000","last_became_peered":"2023-11-24T08:22:50.776374+0000","last_unstale":"2023-11-24T08:22:52.789148+0000","last_undegraded":"2023-11-24T08:22:52.789148+0000","last_fullsized":"2023-11-24T08:22:52.789148+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:58.378583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921938+0000","last_change":"2023-11-24T08:22:51.800647+0000","last_active":"2023-11-24T08:22:52.921938+0000","last_peered":"2023-11-24T08:22:52.921938+0000","last_clean":"2023-11-24T08:22:52.921938+0000","last_became_active":"2023-11-24T08:22:51.800376+0000","last_became_peered":"2023-11-24T08:22:51.800376+0000","last_unstale":"2023-11-24T08:22:52.921938+0000","last_undegraded":"2023-11-24T08:22:52.921938+0000","last_fullsized":"2023-11-24T08:22:52.921938+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:54.873686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922755+0000","last_change":"2023-11-24T08:22:50.779373+0000","last_active":"2023-11-24T08:22:52.922755+0000","last_peered":"2023-11-24T08:22:52.922755+0000","last_clean":"2023-11-24T08:22:52.922755+0000","last_became_active":"2023-11-24T08:22:50.779083+0000","last_became_peered":"2023-11-24T08:22:50.779083+0000","last_unstale":"2023-11-24T08:22:52.922755+0000","last_undegraded":"2023-11-24T08:22:52.922755+0000","last_fullsized":"2023-11-24T08:22:52.922755+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:14.978362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922647+0000","last_change":"2023-11-24T08:22:51.789865+0000","last_active":"2023-11-24T08:22:52.922647+0000","last_peered":"2023-11-24T08:22:52.922647+0000","last_clean":"2023-11-24T08:22:52.922647+0000","last_became_active":"2023-11-24T08:22:51.787758+0000","last_became_peered":"2023-11-24T08:22:51.787758+0000","last_unstale":"2023-11-24T08:22:52.922647+0000","last_undegraded":"2023-11-24T08:22:52.922647+0000","last_fullsized":"2023-11-24T08:22:52.922647+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:57:33.688597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921945+0000","last_change":"2023-11-24T08:22:50.784445+0000","last_active":"2023-11-24T08:22:52.921945+0000","last_peered":"2023-11-24T08:22:52.921945+0000","last_clean":"2023-11-24T08:22:52.921945+0000","last_became_active":"2023-11-24T08:22:50.784160+0000","last_became_peered":"2023-11-24T08:22:50.784160+0000","last_unstale":"2023-11-24T08:22:52.921945+0000","last_undegraded":"2023-11-24T08:22:52.921945+0000","last_fullsized":"2023-11-24T08:22:52.921945+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:27:37.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923915+0000","last_change":"2023-11-24T08:22:51.780327+0000","last_active":"2023-11-24T08:22:52.923915+0000","last_peered":"2023-11-24T08:22:52.923915+0000","last_clean":"2023-11-24T08:22:52.923915+0000","last_became_active":"2023-11-24T08:22:51.779991+0000","last_became_peered":"2023-11-24T08:22:51.779991+0000","last_unstale":"2023-11-24T08:22:52.923915+0000","last_undegraded":"2023-11-24T08:22:52.923915+0000","last_fullsized":"2023-11-24T08:22:52.923915+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:52.678003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787970+0000","last_change":"2023-11-24T08:22:50.777143+0000","last_active":"2023-11-24T08:22:52.787970+0000","last_peered":"2023-11-24T08:22:52.787970+0000","last_clean":"2023-11-24T08:22:52.787970+0000","last_became_active":"2023-11-24T08:22:50.776825+0000","last_became_peered":"2023-11-24T08:22:50.776825+0000","last_unstale":"2023-11-24T08:22:52.787970+0000","last_undegraded":"2023-11-24T08:22:52.787970+0000","last_fullsized":"2023-11-24T08:22:52.787970+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:59:26.038566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789041+0000","last_change":"2023-11-24T08:22:51.801047+0000","last_active":"2023-11-24T08:22:52.789041+0000","last_peered":"2023-11-24T08:22:52.789041+0000","last_clean":"2023-11-24T08:22:52.789041+0000","last_became_active":"2023-11-24T08:22:51.800767+0000","last_became_peered":"2023-11-24T08:22:51.800767+0000","last_unstale":"2023-11-24T08:22:52.789041+0000","last_undegraded":"2023-11-24T08:22:52.789041+0000","last_fullsized":"2023-11-24T08:22:52.789041+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:36.559163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787428+0000","last_change":"2023-11-24T08:22:50.778643+0000","last_active":"2023-11-24T08:22:52.787428+0000","last_peered":"2023-11-24T08:22:52.787428+0000","last_clean":"2023-11-24T08:22:52.787428+0000","last_became_active":"2023-11-24T08:22:50.776369+0000","last_became_peered":"2023-11-24T08:22:50.776369+0000","last_unstale":"2023-11-24T08:22:52.787428+0000","last_undegraded":"2023-11-24T08:22:52.787428+0000","last_fullsized":"2023-11-24T08:22:52.787428+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:21.388748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789037+0000","last_change":"2023-11-24T08:22:51.789812+0000","last_active":"2023-11-24T08:22:52.789037+0000","last_peered":"2023-11-24T08:22:52.789037+0000","last_clean":"2023-11-24T08:22:52.789037+0000","last_became_active":"2023-11-24T08:22:51.789556+0000","last_became_peered":"2023-11-24T08:22:51.789556+0000","last_unstale":"2023-11-24T08:22:52.789037+0000","last_undegraded":"2023-11-24T08:22:52.789037+0000","last_fullsized":"2023-11-24T08:22:52.789037+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:58.378488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784151+0000","last_change":"2023-11-24T08:22:50.781615+0000","last_active":"2023-11-24T08:22:52.784151+0000","last_peered":"2023-11-24T08:22:52.784151+0000","last_clean":"2023-11-24T08:22:52.784151+0000","last_became_active":"2023-11-24T08:22:50.781317+0000","last_became_peered":"2023-11-24T08:22:50.781317+0000","last_unstale":"2023-11-24T08:22:52.784151+0000","last_undegraded":"2023-11-24T08:22:52.784151+0000","last_fullsized":"2023-11-24T08:22:52.784151+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:39.530314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922552+0000","last_change":"2023-11-24T08:22:51.792983+0000","last_active":"2023-11-24T08:22:52.922552+0000","last_peered":"2023-11-24T08:22:52.922552+0000","last_clean":"2023-11-24T08:22:52.922552+0000","last_became_active":"2023-11-24T08:22:51.790323+0000","last_became_peered":"2023-11-24T08:22:51.790323+0000","last_unstale":"2023-11-24T08:22:52.922552+0000","last_undegraded":"2023-11-24T08:22:52.922552+0000","last_fullsized":"2023-11-24T08:22:52.922552+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:54:11.361938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923460+0000","last_change":"2023-11-24T08:22:50.778404+0000","last_active":"2023-11-24T08:22:52.923460+0000","last_peered":"2023-11-24T08:22:52.923460+0000","last_clean":"2023-11-24T08:22:52.923460+0000","last_became_active":"2023-11-24T08:22:50.778115+0000","last_became_peered":"2023-11-24T08:22:50.778115+0000","last_unstale":"2023-11-24T08:22:52.923460+0000","last_undegraded":"2023-11-24T08:22:52.923460+0000","last_fullsized":"2023-11-24T08:22:52.923460+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:40:18.495861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924050+0000","last_change":"2023-11-24T08:22:51.792274+0000","last_active":"2023-11-24T08:22:52.924050+0000","last_peered":"2023-11-24T08:22:52.924050+0000","last_clean":"2023-11-24T08:22:52.924050+0000","last_became_active":"2023-11-24T08:22:51.791963+0000","last_became_peered":"2023-11-24T08:22:51.791963+0000","last_unstale":"2023-11-24T08:22:52.924050+0000","last_undegraded":"2023-11-24T08:22:52.924050+0000","last_fullsized":"2023-11-24T08:22:52.924050+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:44.779376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926004+0000","last_change":"2023-11-24T08:22:50.784469+0000","last_active":"2023-11-24T08:22:52.926004+0000","last_peered":"2023-11-24T08:22:52.926004+0000","last_clean":"2023-11-24T08:22:52.926004+0000","last_became_active":"2023-11-24T08:22:50.784101+0000","last_became_peered":"2023-11-24T08:22:50.784101+0000","last_unstale":"2023-11-24T08:22:52.926004+0000","last_undegraded":"2023-11-24T08:22:52.926004+0000","last_fullsized":"2023-11-24T08:22:52.926004+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:40.933218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921443+0000","last_change":"2023-11-24T08:22:51.800590+0000","last_active":"2023-11-24T08:22:52.921443+0000","last_peered":"2023-11-24T08:22:52.921443+0000","last_clean":"2023-11-24T08:22:52.921443+0000","last_became_active":"2023-11-24T08:22:51.799524+0000","last_became_peered":"2023-11-24T08:22:51.799524+0000","last_unstale":"2023-11-24T08:22:52.921443+0000","last_undegraded":"2023-11-24T08:22:52.921443+0000","last_fullsized":"2023-11-24T08:22:52.921443+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:58:21.460691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784827+0000","last_change":"2023-11-24T08:22:50.785737+0000","last_active":"2023-11-24T08:22:52.784827+0000","last_peered":"2023-11-24T08:22:52.784827+0000","last_clean":"2023-11-24T08:22:52.784827+0000","last_became_active":"2023-11-24T08:22:50.784848+0000","last_became_peered":"2023-11-24T08:22:50.784848+0000","last_unstale":"2023-11-24T08:22:52.784827+0000","last_undegraded":"2023-11-24T08:22:52.784827+0000","last_fullsized":"2023-11-24T08:22:52.784827+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:43.393903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925624+0000","last_change":"2023-11-24T08:22:51.806833+0000","last_active":"2023-11-24T08:22:52.925624+0000","last_peered":"2023-11-24T08:22:52.925624+0000","last_clean":"2023-11-24T08:22:52.925624+0000","last_became_active":"2023-11-24T08:22:51.806542+0000","last_became_peered":"2023-11-24T08:22:51.806542+0000","last_unstale":"2023-11-24T08:22:52.925624+0000","last_undegraded":"2023-11-24T08:22:52.925624+0000","last_fullsized":"2023-11-24T08:22:52.925624+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:55:10.306378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784685+0000","last_change":"2023-11-24T08:22:50.785247+0000","last_active":"2023-11-24T08:22:52.784685+0000","last_peered":"2023-11-24T08:22:52.784685+0000","last_clean":"2023-11-24T08:22:52.784685+0000","last_became_active":"2023-11-24T08:22:50.783812+0000","last_became_peered":"2023-11-24T08:22:50.783812+0000","last_unstale":"2023-11-24T08:22:52.784685+0000","last_undegraded":"2023-11-24T08:22:52.784685+0000","last_fullsized":"2023-11-24T08:22:52.784685+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:43.761030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784537+0000","last_change":"2023-11-24T08:22:51.800739+0000","last_active":"2023-11-24T08:22:52.784537+0000","last_peered":"2023-11-24T08:22:52.784537+0000","last_clean":"2023-11-24T08:22:52.784537+0000","last_became_active":"2023-11-24T08:22:51.792824+0000","last_became_peered":"2023-11-24T08:22:51.792824+0000","last_unstale":"2023-11-24T08:22:52.784537+0000","last_undegraded":"2023-11-24T08:22:52.784537+0000","last_fullsized":"2023-11-24T08:22:52.784537+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:31.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788240+0000","last_change":"2023-11-24T08:22:50.778327+0000","last_active":"2023-11-24T08:22:52.788240+0000","last_peered":"2023-11-24T08:22:52.788240+0000","last_clean":"2023-11-24T08:22:52.788240+0000","last_became_active":"2023-11-24T08:22:50.778023+0000","last_became_peered":"2023-11-24T08:22:50.778023+0000","last_unstale":"2023-11-24T08:22:52.788240+0000","last_undegraded":"2023-11-24T08:22:52.788240+0000","last_fullsized":"2023-11-24T08:22:52.788240+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:05.804327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925577+0000","last_change":"2023-11-24T08:22:51.780715+0000","last_active":"2023-11-24T08:22:52.925577+0000","last_peered":"2023-11-24T08:22:52.925577+0000","last_clean":"2023-11-24T08:22:52.925577+0000","last_became_active":"2023-11-24T08:22:51.780439+0000","last_became_peered":"2023-11-24T08:22:51.780439+0000","last_unstale":"2023-11-24T08:22:52.925577+0000","last_undegraded":"2023-11-24T08:22:52.925577+0000","last_fullsized":"2023-11-24T08:22:52.925577+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:21.447010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925507+0000","last_change":"2023-11-24T08:22:50.787361+0000","last_active":"2023-11-24T08:22:52.925507+0000","last_peered":"2023-11-24T08:22:52.925507+0000","last_clean":"2023-11-24T08:22:52.925507+0000","last_became_active":"2023-11-24T08:22:50.786928+0000","last_became_peered":"2023-11-24T08:22:50.786928+0000","last_unstale":"2023-11-24T08:22:52.925507+0000","last_undegraded":"2023-11-24T08:22:52.925507+0000","last_fullsized":"2023-11-24T08:22:52.925507+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:49.871592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924631+0000","last_change":"2023-11-24T08:22:51.777572+0000","last_active":"2023-11-24T08:22:52.924631+0000","last_peered":"2023-11-24T08:22:52.924631+0000","last_clean":"2023-11-24T08:22:52.924631+0000","last_became_active":"2023-11-24T08:22:51.777330+0000","last_became_peered":"2023-11-24T08:22:51.777330+0000","last_unstale":"2023-11-24T08:22:52.924631+0000","last_undegraded":"2023-11-24T08:22:52.924631+0000","last_fullsized":"2023-11-24T08:22:52.924631+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:24.144535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921373+0000","last_change":"2023-11-24T08:22:50.776498+0000","last_active":"2023-11-24T08:22:52.921373+0000","last_peered":"2023-11-24T08:22:52.921373+0000","last_clean":"2023-11-24T08:22:52.921373+0000","last_became_active":"2023-11-24T08:22:50.775902+0000","last_became_peered":"2023-11-24T08:22:50.775902+0000","last_unstale":"2023-11-24T08:22:52.921373+0000","last_undegraded":"2023-11-24T08:22:52.921373+0000","last_fullsized":"2023-11-24T08:22:52.921373+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:12:41.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788719+0000","last_change":"2023-11-24T08:22:51.774749+0000","last_active":"2023-11-24T08:22:52.788719+0000","last_peered":"2023-11-24T08:22:52.788719+0000","last_clean":"2023-11-24T08:22:52.788719+0000","last_became_active":"2023-11-24T08:22:51.773418+0000","last_became_peered":"2023-11-24T08:22:51.773418+0000","last_unstale":"2023-11-24T08:22:52.788719+0000","last_undegraded":"2023-11-24T08:22:52.788719+0000","last_fullsized":"2023-11-24T08:22:52.788719+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:49.605933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787479+0000","last_change":"2023-11-24T08:22:50.779076+0000","last_active":"2023-11-24T08:22:52.787479+0000","last_peered":"2023-11-24T08:22:52.787479+0000","last_clean":"2023-11-24T08:22:52.787479+0000","last_became_active":"2023-11-24T08:22:50.777061+0000","last_became_peered":"2023-11-24T08:22:50.777061+0000","last_unstale":"2023-11-24T08:22:52.787479+0000","last_undegraded":"2023-11-24T08:22:52.787479+0000","last_fullsized":"2023-11-24T08:22:52.787479+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:23.126189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924603+0000","last_change":"2023-11-24T08:22:51.772269+0000","last_active":"2023-11-24T08:22:52.924603+0000","last_peered":"2023-11-24T08:22:52.924603+0000","last_clean":"2023-11-24T08:22:52.924603+0000","last_became_active":"2023-11-24T08:22:51.771935+0000","last_became_peered":"2023-11-24T08:22:51.771935+0000","last_unstale":"2023-11-24T08:22:52.924603+0000","last_undegraded":"2023-11-24T08:22:52.924603+0000","last_fullsized":"2023-11-24T08:22:52.924603+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:16.718366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923896+0000","last_change":"2023-11-24T08:22:50.775624+0000","last_active":"2023-11-24T08:22:52.923896+0000","last_peered":"2023-11-24T08:22:52.923896+0000","last_clean":"2023-11-24T08:22:52.923896+0000","last_became_active":"2023-11-24T08:22:50.775271+0000","last_became_peered":"2023-11-24T08:22:50.775271+0000","last_unstale":"2023-11-24T08:22:52.923896+0000","last_undegraded":"2023-11-24T08:22:52.923896+0000","last_fullsized":"2023-11-24T08:22:52.923896+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:24.106567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":342,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-11-24T08:22:51.778949+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:22:51.778949+0000","last_peered":"2023-11-24T08:22:51.778949+0000","last_clean":"2023-11-24T08:22:51.778949+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:22:51.778949+0000","last_undegraded":"2023-11-24T08:22:51.778949+0000","last_fullsized":"2023-11-24T08:22:51.778949+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42808,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329032,"statfs":{"total":96636764160,"available":96592928768,"internally_reserved":0,"allocated":3383296,"data_stored":1827398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0920000000000001,"15min":1.0620000000000001},"min":{"1min":0.61199999999999999,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":1.4790000000000001,"5min":2.048,"15min":2.7240000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1180000000000001},"min":{"1min":0.86199999999999999,"5min":0.67200000000000004,"15min":0.53700000000000003},"max":{"1min":1.5109999999999999,"5min":1.899,"15min":2.7109999999999999},"last":0.85299999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.139,"15min":1.1519999999999999},"min":{"1min":0.748,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.4079999999999999,"5min":1.681,"15min":3.8519999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.628},"max":{"1min":1.6519999999999999,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":0.97199999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0940000000000001,"15min":1.1220000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.478,"5min":1.762,"15min":2.4430000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.246,"15min":1.2889999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.60399999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":4.0650000000000004},"last":1.282}]},{"osd":3,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1910000000000001,"15min":1.2090000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.504},"max":{"1min":1.5760000000000001,"5min":1.6240000000000001,"15min":3.133},"last":1.363},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3160000000000001,"15min":1.359},"min":{"1min":0.89200000000000002,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.9710000000000001,"5min":2.173,"15min":3.9209999999999998},"last":1.4319999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3240000000000001,"15min":1.3480000000000001},"min":{"1min":0.94399999999999995,"5min":0.84099999999999997,"15min":0.54700000000000004},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":2.851},"last":1.319},{"interface":"front","average":{"1min":1.248,"5min":1.325,"15min":1.351},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.675,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.5129999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.423,"15min":1.464},"min":{"1min":0.89600000000000002,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":1.8180000000000001,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.4490000000000001,"15min":1.4910000000000001},"min":{"1min":1.0760000000000001,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.722}]},{"osd":6,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5569999999999999,"15min":1.573},"min":{"1min":1.1499999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.992},"last":1.627},{"interface":"front","average":{"1min":1.401,"5min":1.46,"15min":1.498},"min":{"1min":1.115,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.7030000000000001,"5min":2.4159999999999999,"15min":3.798},"last":1.6060000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2789376,"data_stored":1237018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.028,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.59099999999999997},"max":{"1min":1.552,"5min":1.7929999999999999,"15min":1.95},"last":1.3029999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.095,"15min":1.081},"min":{"1min":0.67600000000000005,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.379,"5min":2.008,"15min":2.008},"last":1.456}]},{"osd":1,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.61599999999999999},"max":{"1min":1.581,"5min":1.8100000000000001,"15min":2.0230000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.1779999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.65000000000000002},"max":{"1min":1.6699999999999999,"5min":2.206,"15min":2.206},"last":1.3460000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.115,"15min":1.1080000000000001},"min":{"1min":0.78900000000000003,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2150000000000001,"15min":1.2430000000000001},"min":{"1min":0.72999999999999998,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.518,"5min":1.7,"15min":2.0369999999999999},"last":1.409}]},{"osd":3,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.343,"15min":1.333},"min":{"1min":0.874,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":1.7989999999999999,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2669999999999999,"15min":1.298},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.794,"5min":1.907,"15min":4.8319999999999999},"last":1.367}]},{"osd":4,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.381,"15min":1.4019999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0209999999999999,"5min":3.673,"15min":3.673},"last":1.117},{"interface":"front","average":{"1min":1.397,"5min":1.4590000000000001,"15min":1.4710000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9850000000000001,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":1.198}]},{"osd":5,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.345,"15min":1.3999999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.8180000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5389999999999999,"15min":1.546},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8080000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.54}]},{"osd":7,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5800000000000001,"15min":1.5740000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.2599999999999998,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.395},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5449999999999999,"15min":1.571},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1400000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4330000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38052,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333788,"statfs":{"total":96636764160,"available":96597798912,"internally_reserved":0,"allocated":2768896,"data_stored":1230606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.016},"min":{"1min":0.61599999999999999,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.696,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.135,"15min":1.1020000000000001},"min":{"1min":0.70099999999999996,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6639999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":1.4179999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.177,"15min":1.131},"min":{"1min":0.65000000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.665,"5min":7.258,"15min":7.258},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.159,"5min":1.2190000000000001,"15min":1.1899999999999999},"min":{"1min":0.80200000000000005,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.653,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2210000000000001,"15min":1.1830000000000001},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.556,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2390000000000001,"15min":1.177},"min":{"1min":0.96099999999999997,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.806,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.0449999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.252,"15min":1.181},"min":{"1min":0.76200000000000001,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.5069999999999999,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.121},{"interface":"front","average":{"1min":1.282,"5min":1.3320000000000001,"15min":1.278},"min":{"1min":0.90800000000000003,"5min":0.82799999999999996,"15min":0.55200000000000005},"max":{"1min":1.694,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.581}]},{"osd":4,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.4850000000000001,"15min":1.383},"min":{"1min":0.98199999999999998,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.8240000000000001,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5640000000000001,"15min":1.48},"min":{"1min":1.258,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":1.98,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.667}]},{"osd":6,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3859999999999999,"15min":1.3420000000000001},"min":{"1min":0.97999999999999998,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.917,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.202},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.5069999999999999,"15min":1.448},"min":{"1min":0.88300000000000001,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":1.7669999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.369}]},{"osd":7,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.546,"15min":1.4970000000000001},"min":{"1min":0.877,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":2.0379999999999998,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.7649999999999999,"15min":1.647},"min":{"1min":1.2390000000000001,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.008,"5min":19.663,"15min":19.663},"last":1.627}]}]},{"osd":4,"up_from":13,"seq":55834575071,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2785280,"data_stored":1236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0469999999999999,"15min":1.0549999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.26,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.0840000000000001,"15min":1.0800000000000001},"min":{"1min":0.78300000000000003,"5min":0.75900000000000001,"15min":0.46100000000000002},"max":{"1min":1.5009999999999999,"5min":1.7070000000000001,"15min":5.452},"last":0.95899999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.115,"15min":1.1419999999999999},"min":{"1min":0.69899999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6359999999999999,"5min":1.6399999999999999,"15min":1.8100000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.103,"15min":1.1439999999999999},"min":{"1min":0.70799999999999996,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.329,"5min":1.7150000000000001,"15min":1.988},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0549999999999999,"15min":1.0600000000000001},"min":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.5149999999999999,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.244,"15min":1.24},"min":{"1min":0.84199999999999997,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.494,"5min":1.827,"15min":2.0470000000000002},"last":1.3200000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.337,"15min":1.331},"min":{"1min":1.012,"5min":0.754,"15min":0.62},"max":{"1min":1.6699999999999999,"5min":1.925,"15min":2.121},"last":1.431},{"interface":"front","average":{"1min":1.276,"5min":1.272,"15min":1.274},"min":{"1min":0.88700000000000001,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.744,"5min":1.744,"15min":2.3340000000000001},"last":0.85499999999999998}]},{"osd":5,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4890000000000001,"15min":1.4910000000000001},"min":{"1min":1.02,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.7390000000000001,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.375,"15min":1.4139999999999999},"min":{"1min":0.98199999999999998,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8520000000000001,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.2210000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2509999999999999,"15min":1.2450000000000001},"min":{"1min":0.81399999999999995,"5min":0.629,"15min":0.55400000000000005},"max":{"1min":1.659,"5min":1.8540000000000001,"15min":1.907},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.538,"15min":1.5349999999999999},"min":{"1min":1.242,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.897,"5min":2.1190000000000002,"15min":2.2890000000000001},"last":1.631}]},{"osd":7,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.458,"15min":1.478},"min":{"1min":1.1399999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8,"5min":1.948,"15min":3.1219999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.415,"15min":1.403},"min":{"1min":1.159,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.077,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.2769999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575070,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":2805760,"data_stored":1239156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.82199999999999995,"15min":0.872},"min":{"1min":0.45700000000000002,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.2490000000000001,"5min":1.5820000000000001,"15min":10.557},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92600000000000005,"15min":0.92300000000000004},"min":{"1min":0.60699999999999998,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.234,"5min":1.653,"15min":3.8260000000000001},"last":1.089}]},{"osd":1,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":0.79600000000000004},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.95699999999999996,"5min":1.407,"15min":2.101},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89400000000000002,"15min":0.97499999999999998},"min":{"1min":0.58199999999999996,"5min":0.5,"15min":0.379},"max":{"1min":1.667,"5min":1.667,"15min":13.090999999999999},"last":1.1519999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.871,"15min":0.89400000000000002},"min":{"1min":0.54000000000000004,"5min":0.433,"15min":0.39900000000000002},"max":{"1min":1.4650000000000001,"5min":1.976,"15min":3.9489999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.97899999999999998,"15min":1.073},"min":{"1min":0.64400000000000002,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":1.5369999999999999,"5min":1.665,"15min":13.460000000000001},"last":0.755}]},{"osd":4,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2649999999999999,"15min":1.2709999999999999},"min":{"1min":1.0069999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.621,"5min":1.845,"15min":4.367},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.242,"15min":1.222},"min":{"1min":0.96999999999999997,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.4159999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.248,"15min":1.306},"min":{"1min":0.86599999999999999,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.5649999999999999,"5min":2.1230000000000002,"15min":13.231},"last":1.282},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2070000000000001,"15min":1.236},"min":{"1min":0.80000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.617,"5min":1.8620000000000001,"15min":5.1180000000000003},"last":1.6319999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.304,"15min":1.3660000000000001},"min":{"1min":0.81399999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.621,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.232},{"interface":"front","average":{"1min":1.266,"5min":1.24,"15min":1.244},"min":{"1min":0.89300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.736,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":1.4630000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2949999999999999,"15min":1.3089999999999999},"min":{"1min":0.93000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":1.73,"5min":2.0720000000000001,"15min":4.266},"last":1.734},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2949999999999999,"15min":1.3009999999999999},"min":{"1min":0.78500000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.744,"5min":3.0030000000000001,"15min":3.331},"last":1.669}]}]},{"osd":2,"up_from":13,"seq":55834575068,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2789376,"data_stored":1237558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.38200000000000001,"5min":0.34200000000000003,"15min":0.29999999999999999},"max":{"1min":1.468,"5min":1.5429999999999999,"15min":1.601},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.85499999999999998},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.123,"5min":1.4810000000000001,"15min":1.782},"last":0.95799999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.54300000000000004,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.526,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.97199999999999998,"15min":0.95199999999999996},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":1.4019999999999999,"5min":1.6699999999999999,"15min":1.8360000000000001},"last":1.1990000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.89100000000000001,"15min":0.85999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":1.397,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.008,"15min":1.022},"min":{"1min":0.59399999999999997,"5min":0.52500000000000002,"15min":0.41399999999999998},"max":{"1min":1.395,"5min":1.948,"15min":1.948},"last":1.093}]},{"osd":4,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.268,"15min":1.2689999999999999},"min":{"1min":0.747,"5min":0.70699999999999996,"15min":0.53700000000000003},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.248,"15min":1.244},"min":{"1min":0.93899999999999995,"5min":0.78700000000000003,"15min":0.745},"max":{"1min":1.7609999999999999,"5min":1.8200000000000001,"15min":1.929},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.2190000000000001,"15min":1.2350000000000001},"min":{"1min":0.83699999999999997,"5min":0.753,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2569999999999999,"15min":1.238},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1919999999999999,"15min":1.1879999999999999},"min":{"1min":0.77200000000000002,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.7709999999999999,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":1.458},{"interface":"front","average":{"1min":1.28,"5min":1.3080000000000001,"15min":1.3140000000000001},"min":{"1min":0.92300000000000004,"5min":0.78800000000000003,"15min":0.65600000000000003},"max":{"1min":1.625,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.355}]},{"osd":7,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.319,"15min":1.3100000000000001},"min":{"1min":0.79900000000000004,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2929999999999999,"15min":1.282},"min":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.877,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.6910000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42216,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329624,"statfs":{"total":96636764160,"available":96593534976,"internally_reserved":0,"allocated":2781184,"data_stored":1236906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85599999999999998,"15min":0.81999999999999995},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.2969999999999999,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.94999999999999996,"15min":0.90900000000000003},"min":{"1min":0.55200000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":0.94999999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.83599999999999997,"15min":0.81799999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.37},"max":{"1min":1.3069999999999999,"5min":1.823,"15min":1.861},"last":0.998},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.95199999999999996,"15min":0.93000000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.6950000000000001,"5min":1.7490000000000001,"15min":1.925},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.069,"15min":1.016},"min":{"1min":0.53400000000000003,"5min":0.436,"15min":0.40699999999999997},"max":{"1min":1.4470000000000001,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93799999999999994,"15min":0.93000000000000005},"min":{"1min":0.623,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.444,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":0.84599999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3460000000000001,"15min":1.2929999999999999},"min":{"1min":0.97699999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.8200000000000001,"5min":1.984,"15min":2.7959999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3120000000000001,"15min":1.2729999999999999},"min":{"1min":0.79200000000000004,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.8300000000000001},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.278,"15min":1.2629999999999999},"min":{"1min":0.90800000000000003,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.6359999999999999,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.272,"15min":1.2310000000000001},"min":{"1min":0.72699999999999998,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.7210000000000001,"5min":2.1499999999999999,"15min":3.286},"last":1.2050000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2789999999999999,"15min":1.29},"min":{"1min":0.745,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.986,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3160000000000001,"15min":1.288},"min":{"1min":0.80500000000000005,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.419}]},{"osd":7,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3460000000000001,"15min":1.327},"min":{"1min":0.83699999999999997,"5min":0.82299999999999995,"15min":0.59099999999999997},"max":{"1min":2.137,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.351},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":1.901,"5min":1.9930000000000001,"15min":3.653},"last":1.288}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42812,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329028,"statfs":{"total":96636764160,"available":96592924672,"internally_reserved":0,"allocated":3387392,"data_stored":1828886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.054,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.82599999999999996,"15min":0.82699999999999996},"min":{"1min":0.48299999999999998,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.607,"5min":1.607,"15min":2.3620000000000001},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.008,"15min":1.0109999999999999},"min":{"1min":0.54200000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4770000000000001,"5min":1.893,"15min":1.893},"last":1.236},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86299999999999999,"15min":0.86799999999999999},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.444,"5min":1.671,"15min":2.4990000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93799999999999994,"15min":0.96799999999999997},"min":{"1min":0.55900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5209999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85999999999999999,"15min":0.88100000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":1.361,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":1.0640000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2210000000000001,"15min":1.214},"min":{"1min":0.93400000000000005,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.8380000000000001,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.665},{"interface":"front","average":{"1min":1.071,"5min":1.083,"15min":1.107},"min":{"1min":0.75,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.496,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":1.036}]},{"osd":5,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1399999999999999,"15min":1.1499999999999999},"min":{"1min":0.71199999999999997,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.6100000000000001,"5min":1.6910000000000001,"15min":2.6850000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.149,"5min":1.1419999999999999,"15min":1.159},"min":{"1min":0.753,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.782,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.157}]},{"osd":6,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.248,"15min":1.254},"min":{"1min":0.76200000000000001,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.6990000000000001,"5min":1.8560000000000001,"15min":2.052},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.742,"5min":1.8440000000000001,"15min":2.823},"last":1.1919999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2989999999999999,"15min":1.3320000000000001},"min":{"1min":0.91500000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.921,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2769999999999999,"15min":1.298},"min":{"1min":0.86399999999999999,"5min":0.79600000000000004,"15min":0.747},"max":{"1min":1.837,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.96899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:22:56.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:22:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.467+0000 7f8976c00640 1 Processor -- start 2023-11-24T08:22:56.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.467+0000 7f8976c00640 1 -- start start 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.468+0000 7f8976c00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f89700ce6c0 0x7f89700cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.468+0000 7f8976c00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700cece0 con 0x7f89700cf450 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.468+0000 7f8976c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897006d040 con 0x7f897015c7f0 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.468+0000 7f8976c00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700cfe40 con 0x7f89700ce6c0 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f89700ce6c0 0x7f89700cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f89700ce6c0 0x7f89700cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46002/0 (socket says 172.21.15.107:46002) 2023-11-24T08:22:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 -- 172.21.15.107:0/2535595730 learned_addr learned my addr 172.21.15.107:0/2535595730 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 -- 172.21.15.107:0/2535595730 >> v1:172.21.15.120:6789/0 conn(0x7f897015c7f0 legacy=0x7f897006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 -- 172.21.15.107:0/2535595730 >> v1:172.21.15.107:6789/0 conn(0x7f89700cf450 legacy=0x7f89700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f896f7fe640 1 -- 172.21.15.107:0/2535595730 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835099886 0 0) 0x7f89700cece0 con 0x7f89700cf450 2023-11-24T08:22:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.469+0000 7f897519d640 1 -- 172.21.15.107:0/2535595730 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897006d320 con 0x7f89700ce6c0 2023-11-24T08:22:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f897519d640 1 --2- 172.21.15.107:0/2535595730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700ce6c0 0x7f89700cf900 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f8968009870 tx=0x7f896802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9456b8c541a0dd4b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f896f7fe640 1 -- 172.21.15.107:0/2535595730 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968030860 con 0x7f89700ce6c0 2023-11-24T08:22:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f896f7fe640 1 -- 172.21.15.107:0/2535595730 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89680309c0 con 0x7f89700ce6c0 2023-11-24T08:22:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f896f7fe640 1 -- 172.21.15.107:0/2535595730 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968030d50 con 0x7f89700ce6c0 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f8976c00640 1 -- 172.21.15.107:0/2535595730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700ce6c0 msgr2=0x7f89700cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.471+0000 7f8976c00640 1 --2- 172.21.15.107:0/2535595730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700ce6c0 0x7f89700cf900 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f8968009870 tx=0x7f896802b9a0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.472+0000 7f8976c00640 1 -- 172.21.15.107:0/2535595730 shutdown_connections 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.472+0000 7f8976c00640 1 --2- 172.21.15.107:0/2535595730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700ce6c0 0x7f89700cf900 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.472+0000 7f8976c00640 1 -- 172.21.15.107:0/2535595730 >> 172.21.15.107:0/2535595730 conn(0x7f897006a920 msgr2=0x7f897006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.472+0000 7f8976c00640 1 -- 172.21.15.107:0/2535595730 shutdown_connections 2023-11-24T08:22:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.472+0000 7f8976c00640 1 -- 172.21.15.107:0/2535595730 wait complete. 2023-11-24T08:22:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.473+0000 7f8976c00640 1 Processor -- start 2023-11-24T08:22:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.474+0000 7f8976c00640 1 -- start start 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.474+0000 7f8976c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 0x7f897014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f8976c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f8976c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f897015c7f0 0x7f8970154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f8976c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897015a850 con 0x7f89700cf450 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f8976c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970159c40 con 0x7f89700ce6c0 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f8976c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8970159f20 con 0x7f897015c7f0 2023-11-24T08:22:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46004/0 (socket says 172.21.15.107:46004) 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 -- 172.21.15.107:0/2028354084 learned_addr learned my addr 172.21.15.107:0/2028354084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 -- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f897015c7f0 msgr2=0x7f8970154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f897499c640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 0x7f897014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f897519d640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f897015c7f0 0x7f8970154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f897015c7f0 0x7f8970154540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 -- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 msgr2=0x7f897014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 0x7f897014b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f896ffff640 1 -- 172.21.15.107:0/2028354084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680094d0 con 0x7f89700cf450 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.475+0000 7f897499c640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 0x7f897014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f896ffff640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f895800c890 tx=0x7f895800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958003da0 con 0x7f89700cf450 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958005070 con 0x7f89700cf450 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958005420 con 0x7f89700cf450 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897015a200 con 0x7f89700cf450 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.476+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897014b940 con 0x7f89700cf450 2023-11-24T08:22:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.477+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970067470 con 0x7f89700cf450 2023-11-24T08:22:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.478+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f8958005580 con 0x7f89700cf450 2023-11-24T08:22:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.479+0000 7f896d7fa640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 0x7f895006f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.480+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8958088070 con 0x7f89700cf450 2023-11-24T08:22:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.480+0000 7f897499c640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 0x7f895006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.480+0000 7f897499c640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 0x7f895006f580 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f896000a830 tx=0x7f8960008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.484+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8958054c20 con 0x7f89700cf450 2023-11-24T08:22:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.670+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89700cd730 con 0x7f895006d0c0 2023-11-24T08:22:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.682+0000 7f896d7fa640 1 -- 172.21.15.107:0/2028354084 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287587 (secure 0 0 0) 0x7f89700cd730 con 0x7f895006d0c0 2023-11-24T08:22:56.685 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:56.689 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:22:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 msgr2=0x7f895006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 0x7f895006f580 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f896000a830 tx=0x7f8960008040 comp rx=0 tx=0).stop 2023-11-24T08:22:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 msgr2=0x7f897015b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f895800c890 tx=0x7f895800cd50 comp rx=0 tx=0).stop 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 shutdown_connections 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f895006d0c0 0x7f895006f580 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f897015c7f0 0x7f8970154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700cf450 0x7f897015b320 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 --2- 172.21.15.107:0/2028354084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89700ce6c0 0x7f897014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 >> 172.21.15.107:0/2028354084 conn(0x7f897006a920 msgr2=0x7f897015f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 shutdown_connections 2023-11-24T08:22:56.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.688+0000 7f8976c00640 1 -- 172.21.15.107:0/2028354084 wait complete. 2023-11-24T08:22:56.708 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2023-11-24T08:22:55.777425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":334812,"kb_used_data":22940,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754639908,"statfs":{"total":773094113280,"available":772751265792,"internally_reserved":0,"allocated":23490560,"data_stored":11074504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":319213118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001659"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925468+0000","last_change":"2023-11-24T08:22:51.777727+0000","last_active":"2023-11-24T08:22:52.925468+0000","last_peered":"2023-11-24T08:22:52.925468+0000","last_clean":"2023-11-24T08:22:52.925468+0000","last_became_active":"2023-11-24T08:22:51.777448+0000","last_became_peered":"2023-11-24T08:22:51.777448+0000","last_unstale":"2023-11-24T08:22:52.925468+0000","last_undegraded":"2023-11-24T08:22:52.925468+0000","last_fullsized":"2023-11-24T08:22:52.925468+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:47:59.480275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924301+0000","last_change":"2023-11-24T08:22:51.776411+0000","last_active":"2023-11-24T08:22:52.924301+0000","last_peered":"2023-11-24T08:22:52.924301+0000","last_clean":"2023-11-24T08:22:52.924301+0000","last_became_active":"2023-11-24T08:22:51.776107+0000","last_became_peered":"2023-11-24T08:22:51.776107+0000","last_unstale":"2023-11-24T08:22:52.924301+0000","last_undegraded":"2023-11-24T08:22:52.924301+0000","last_fullsized":"2023-11-24T08:22:52.924301+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:22.407095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922399+0000","last_change":"2023-11-24T08:22:51.794496+0000","last_active":"2023-11-24T08:22:52.922399+0000","last_peered":"2023-11-24T08:22:52.922399+0000","last_clean":"2023-11-24T08:22:52.922399+0000","last_became_active":"2023-11-24T08:22:51.794173+0000","last_became_peered":"2023-11-24T08:22:51.794173+0000","last_unstale":"2023-11-24T08:22:52.922399+0000","last_undegraded":"2023-11-24T08:22:52.922399+0000","last_fullsized":"2023-11-24T08:22:52.922399+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:59:26.274775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922986+0000","last_change":"2023-11-24T08:22:51.804163+0000","last_active":"2023-11-24T08:22:52.922986+0000","last_peered":"2023-11-24T08:22:52.922986+0000","last_clean":"2023-11-24T08:22:52.922986+0000","last_became_active":"2023-11-24T08:22:51.803902+0000","last_became_peered":"2023-11-24T08:22:51.803902+0000","last_unstale":"2023-11-24T08:22:52.922986+0000","last_undegraded":"2023-11-24T08:22:52.922986+0000","last_fullsized":"2023-11-24T08:22:52.922986+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:04:12.102399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925564+0000","last_change":"2023-11-24T08:22:51.797757+0000","last_active":"2023-11-24T08:22:52.925564+0000","last_peered":"2023-11-24T08:22:52.925564+0000","last_clean":"2023-11-24T08:22:52.925564+0000","last_became_active":"2023-11-24T08:22:51.793587+0000","last_became_peered":"2023-11-24T08:22:51.793587+0000","last_unstale":"2023-11-24T08:22:52.925564+0000","last_undegraded":"2023-11-24T08:22:52.925564+0000","last_fullsized":"2023-11-24T08:22:52.925564+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:38.507421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925027+0000","last_change":"2023-11-24T08:22:51.786858+0000","last_active":"2023-11-24T08:22:52.925027+0000","last_peered":"2023-11-24T08:22:52.925027+0000","last_clean":"2023-11-24T08:22:52.925027+0000","last_became_active":"2023-11-24T08:22:51.786553+0000","last_became_peered":"2023-11-24T08:22:51.786553+0000","last_unstale":"2023-11-24T08:22:52.925027+0000","last_undegraded":"2023-11-24T08:22:52.925027+0000","last_fullsized":"2023-11-24T08:22:52.925027+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:52.058301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923599+0000","last_change":"2023-11-24T08:22:51.802586+0000","last_active":"2023-11-24T08:22:52.923599+0000","last_peered":"2023-11-24T08:22:52.923599+0000","last_clean":"2023-11-24T08:22:52.923599+0000","last_became_active":"2023-11-24T08:22:51.802020+0000","last_became_peered":"2023-11-24T08:22:51.802020+0000","last_unstale":"2023-11-24T08:22:52.923599+0000","last_undegraded":"2023-11-24T08:22:52.923599+0000","last_fullsized":"2023-11-24T08:22:52.923599+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:30:18.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924681+0000","last_change":"2023-11-24T08:22:51.803602+0000","last_active":"2023-11-24T08:22:52.924681+0000","last_peered":"2023-11-24T08:22:52.924681+0000","last_clean":"2023-11-24T08:22:52.924681+0000","last_became_active":"2023-11-24T08:22:51.803301+0000","last_became_peered":"2023-11-24T08:22:51.803301+0000","last_unstale":"2023-11-24T08:22:52.924681+0000","last_undegraded":"2023-11-24T08:22:52.924681+0000","last_fullsized":"2023-11-24T08:22:52.924681+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:40.548660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925917+0000","last_change":"2023-11-24T08:22:51.793706+0000","last_active":"2023-11-24T08:22:52.925917+0000","last_peered":"2023-11-24T08:22:52.925917+0000","last_clean":"2023-11-24T08:22:52.925917+0000","last_became_active":"2023-11-24T08:22:51.792710+0000","last_became_peered":"2023-11-24T08:22:51.792710+0000","last_unstale":"2023-11-24T08:22:52.925917+0000","last_undegraded":"2023-11-24T08:22:52.925917+0000","last_fullsized":"2023-11-24T08:22:52.925917+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:06:59.522187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922228+0000","last_change":"2023-11-24T08:22:51.801479+0000","last_active":"2023-11-24T08:22:52.922228+0000","last_peered":"2023-11-24T08:22:52.922228+0000","last_clean":"2023-11-24T08:22:52.922228+0000","last_became_active":"2023-11-24T08:22:51.799551+0000","last_became_peered":"2023-11-24T08:22:51.799551+0000","last_unstale":"2023-11-24T08:22:52.922228+0000","last_undegraded":"2023-11-24T08:22:52.922228+0000","last_fullsized":"2023-11-24T08:22:52.922228+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:07.946995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922633+0000","last_change":"2023-11-24T08:22:51.800282+0000","last_active":"2023-11-24T08:22:52.922633+0000","last_peered":"2023-11-24T08:22:52.922633+0000","last_clean":"2023-11-24T08:22:52.922633+0000","last_became_active":"2023-11-24T08:22:51.799894+0000","last_became_peered":"2023-11-24T08:22:51.799894+0000","last_unstale":"2023-11-24T08:22:52.922633+0000","last_undegraded":"2023-11-24T08:22:52.922633+0000","last_fullsized":"2023-11-24T08:22:52.922633+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:06.821388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925262+0000","last_change":"2023-11-24T08:22:51.775736+0000","last_active":"2023-11-24T08:22:52.925262+0000","last_peered":"2023-11-24T08:22:52.925262+0000","last_clean":"2023-11-24T08:22:52.925262+0000","last_became_active":"2023-11-24T08:22:51.775470+0000","last_became_peered":"2023-11-24T08:22:51.775470+0000","last_unstale":"2023-11-24T08:22:52.925262+0000","last_undegraded":"2023-11-24T08:22:52.925262+0000","last_fullsized":"2023-11-24T08:22:52.925262+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:03.904749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788333+0000","last_change":"2023-11-24T08:22:51.781032+0000","last_active":"2023-11-24T08:22:52.788333+0000","last_peered":"2023-11-24T08:22:52.788333+0000","last_clean":"2023-11-24T08:22:52.788333+0000","last_became_active":"2023-11-24T08:22:51.780696+0000","last_became_peered":"2023-11-24T08:22:51.780696+0000","last_unstale":"2023-11-24T08:22:52.788333+0000","last_undegraded":"2023-11-24T08:22:52.788333+0000","last_fullsized":"2023-11-24T08:22:52.788333+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:07:10.613046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784870+0000","last_change":"2023-11-24T08:22:51.771728+0000","last_active":"2023-11-24T08:22:52.784870+0000","last_peered":"2023-11-24T08:22:52.784870+0000","last_clean":"2023-11-24T08:22:52.784870+0000","last_became_active":"2023-11-24T08:22:51.771420+0000","last_became_peered":"2023-11-24T08:22:51.771420+0000","last_unstale":"2023-11-24T08:22:52.784870+0000","last_undegraded":"2023-11-24T08:22:52.784870+0000","last_fullsized":"2023-11-24T08:22:52.784870+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:37:52.623774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925867+0000","last_change":"2023-11-24T08:22:51.789133+0000","last_active":"2023-11-24T08:22:52.925867+0000","last_peered":"2023-11-24T08:22:52.925867+0000","last_clean":"2023-11-24T08:22:52.925867+0000","last_became_active":"2023-11-24T08:22:51.788839+0000","last_became_peered":"2023-11-24T08:22:51.788839+0000","last_unstale":"2023-11-24T08:22:52.925867+0000","last_undegraded":"2023-11-24T08:22:52.925867+0000","last_fullsized":"2023-11-24T08:22:52.925867+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:46:16.777665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921682+0000","last_change":"2023-11-24T08:22:51.775096+0000","last_active":"2023-11-24T08:22:52.921682+0000","last_peered":"2023-11-24T08:22:52.921682+0000","last_clean":"2023-11-24T08:22:52.921682+0000","last_became_active":"2023-11-24T08:22:51.774737+0000","last_became_peered":"2023-11-24T08:22:51.774737+0000","last_unstale":"2023-11-24T08:22:52.921682+0000","last_undegraded":"2023-11-24T08:22:52.921682+0000","last_fullsized":"2023-11-24T08:22:52.921682+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:14:26.577628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789366+0000","last_change":"2023-11-24T08:22:51.794566+0000","last_active":"2023-11-24T08:22:52.789366+0000","last_peered":"2023-11-24T08:22:52.789366+0000","last_clean":"2023-11-24T08:22:52.789366+0000","last_became_active":"2023-11-24T08:22:51.794242+0000","last_became_peered":"2023-11-24T08:22:51.794242+0000","last_unstale":"2023-11-24T08:22:52.789366+0000","last_undegraded":"2023-11-24T08:22:52.789366+0000","last_fullsized":"2023-11-24T08:22:52.789366+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:29:54.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922364+0000","last_change":"2023-11-24T08:22:51.784286+0000","last_active":"2023-11-24T08:22:52.922364+0000","last_peered":"2023-11-24T08:22:52.922364+0000","last_clean":"2023-11-24T08:22:52.922364+0000","last_became_active":"2023-11-24T08:22:51.783974+0000","last_became_peered":"2023-11-24T08:22:51.783974+0000","last_unstale":"2023-11-24T08:22:52.922364+0000","last_undegraded":"2023-11-24T08:22:52.922364+0000","last_fullsized":"2023-11-24T08:22:52.922364+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:02:26.041628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926522+0000","last_change":"2023-11-24T08:22:51.799070+0000","last_active":"2023-11-24T08:22:52.926522+0000","last_peered":"2023-11-24T08:22:52.926522+0000","last_clean":"2023-11-24T08:22:52.926522+0000","last_became_active":"2023-11-24T08:22:51.798655+0000","last_became_peered":"2023-11-24T08:22:51.798655+0000","last_unstale":"2023-11-24T08:22:52.926522+0000","last_undegraded":"2023-11-24T08:22:52.926522+0000","last_fullsized":"2023-11-24T08:22:52.926522+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:37:52.623774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923554+0000","last_change":"2023-11-24T08:22:51.793936+0000","last_active":"2023-11-24T08:22:52.923554+0000","last_peered":"2023-11-24T08:22:52.923554+0000","last_clean":"2023-11-24T08:22:52.923554+0000","last_became_active":"2023-11-24T08:22:51.793685+0000","last_became_peered":"2023-11-24T08:22:51.793685+0000","last_unstale":"2023-11-24T08:22:52.923554+0000","last_undegraded":"2023-11-24T08:22:52.923554+0000","last_fullsized":"2023-11-24T08:22:52.923554+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:21:36.568123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923269+0000","last_change":"2023-11-24T08:22:51.792098+0000","last_active":"2023-11-24T08:22:52.923269+0000","last_peered":"2023-11-24T08:22:52.923269+0000","last_clean":"2023-11-24T08:22:52.923269+0000","last_became_active":"2023-11-24T08:22:51.791788+0000","last_became_peered":"2023-11-24T08:22:51.791788+0000","last_unstale":"2023-11-24T08:22:52.923269+0000","last_undegraded":"2023-11-24T08:22:52.923269+0000","last_fullsized":"2023-11-24T08:22:52.923269+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:36:54.595128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788099+0000","last_change":"2023-11-24T08:22:51.778589+0000","last_active":"2023-11-24T08:22:52.788099+0000","last_peered":"2023-11-24T08:22:52.788099+0000","last_clean":"2023-11-24T08:22:52.788099+0000","last_became_active":"2023-11-24T08:22:51.773828+0000","last_became_peered":"2023-11-24T08:22:51.773828+0000","last_unstale":"2023-11-24T08:22:52.788099+0000","last_undegraded":"2023-11-24T08:22:52.788099+0000","last_fullsized":"2023-11-24T08:22:52.788099+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:28:21.899461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926193+0000","last_change":"2023-11-24T08:22:51.793063+0000","last_active":"2023-11-24T08:22:52.926193+0000","last_peered":"2023-11-24T08:22:52.926193+0000","last_clean":"2023-11-24T08:22:52.926193+0000","last_became_active":"2023-11-24T08:22:51.792275+0000","last_became_peered":"2023-11-24T08:22:51.792275+0000","last_unstale":"2023-11-24T08:22:52.926193+0000","last_undegraded":"2023-11-24T08:22:52.926193+0000","last_fullsized":"2023-11-24T08:22:52.926193+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:31.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787304+0000","last_change":"2023-11-24T08:22:51.791377+0000","last_active":"2023-11-24T08:22:52.787304+0000","last_peered":"2023-11-24T08:22:52.787304+0000","last_clean":"2023-11-24T08:22:52.787304+0000","last_became_active":"2023-11-24T08:22:51.789372+0000","last_became_peered":"2023-11-24T08:22:51.789372+0000","last_unstale":"2023-11-24T08:22:52.787304+0000","last_undegraded":"2023-11-24T08:22:52.787304+0000","last_fullsized":"2023-11-24T08:22:52.787304+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:30.547159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924042+0000","last_change":"2023-11-24T08:22:51.770004+0000","last_active":"2023-11-24T08:22:52.924042+0000","last_peered":"2023-11-24T08:22:52.924042+0000","last_clean":"2023-11-24T08:22:52.924042+0000","last_became_active":"2023-11-24T08:22:51.769733+0000","last_became_peered":"2023-11-24T08:22:51.769733+0000","last_unstale":"2023-11-24T08:22:52.924042+0000","last_undegraded":"2023-11-24T08:22:52.924042+0000","last_fullsized":"2023-11-24T08:22:52.924042+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.396929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923893+0000","last_change":"2023-11-24T08:22:51.799243+0000","last_active":"2023-11-24T08:22:52.923893+0000","last_peered":"2023-11-24T08:22:52.923893+0000","last_clean":"2023-11-24T08:22:52.923893+0000","last_became_active":"2023-11-24T08:22:51.798774+0000","last_became_peered":"2023-11-24T08:22:51.798774+0000","last_unstale":"2023-11-24T08:22:52.923893+0000","last_undegraded":"2023-11-24T08:22:52.923893+0000","last_fullsized":"2023-11-24T08:22:52.923893+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:06:22.315999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925547+0000","last_change":"2023-11-24T08:22:51.797353+0000","last_active":"2023-11-24T08:22:52.925547+0000","last_peered":"2023-11-24T08:22:52.925547+0000","last_clean":"2023-11-24T08:22:52.925547+0000","last_became_active":"2023-11-24T08:22:51.797000+0000","last_became_peered":"2023-11-24T08:22:51.797000+0000","last_unstale":"2023-11-24T08:22:52.925547+0000","last_undegraded":"2023-11-24T08:22:52.925547+0000","last_fullsized":"2023-11-24T08:22:52.925547+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:44.412249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925514+0000","last_change":"2023-11-24T08:22:51.798827+0000","last_active":"2023-11-24T08:22:52.925514+0000","last_peered":"2023-11-24T08:22:52.925514+0000","last_clean":"2023-11-24T08:22:52.925514+0000","last_became_active":"2023-11-24T08:22:51.798565+0000","last_became_peered":"2023-11-24T08:22:51.798565+0000","last_unstale":"2023-11-24T08:22:52.925514+0000","last_undegraded":"2023-11-24T08:22:52.925514+0000","last_fullsized":"2023-11-24T08:22:52.925514+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:48.991722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.785175+0000","last_change":"2023-11-24T08:22:51.774282+0000","last_active":"2023-11-24T08:22:52.785175+0000","last_peered":"2023-11-24T08:22:52.785175+0000","last_clean":"2023-11-24T08:22:52.785175+0000","last_became_active":"2023-11-24T08:22:51.773842+0000","last_became_peered":"2023-11-24T08:22:51.773842+0000","last_unstale":"2023-11-24T08:22:52.785175+0000","last_undegraded":"2023-11-24T08:22:52.785175+0000","last_fullsized":"2023-11-24T08:22:52.785175+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:29:19.650715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784876+0000","last_change":"2023-11-24T08:22:51.793901+0000","last_active":"2023-11-24T08:22:52.784876+0000","last_peered":"2023-11-24T08:22:52.784876+0000","last_clean":"2023-11-24T08:22:52.784876+0000","last_became_active":"2023-11-24T08:22:51.793589+0000","last_became_peered":"2023-11-24T08:22:51.793589+0000","last_unstale":"2023-11-24T08:22:52.784876+0000","last_undegraded":"2023-11-24T08:22:52.784876+0000","last_fullsized":"2023-11-24T08:22:52.784876+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:03:54.450966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922825+0000","last_change":"2023-11-24T08:22:51.792690+0000","last_active":"2023-11-24T08:22:52.922825+0000","last_peered":"2023-11-24T08:22:52.922825+0000","last_clean":"2023-11-24T08:22:52.922825+0000","last_became_active":"2023-11-24T08:22:51.792440+0000","last_became_peered":"2023-11-24T08:22:51.792440+0000","last_unstale":"2023-11-24T08:22:52.922825+0000","last_undegraded":"2023-11-24T08:22:52.922825+0000","last_fullsized":"2023-11-24T08:22:52.922825+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:59:27.056912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924643+0000","last_change":"2023-11-24T08:22:51.799593+0000","last_active":"2023-11-24T08:22:52.924643+0000","last_peered":"2023-11-24T08:22:52.924643+0000","last_clean":"2023-11-24T08:22:52.924643+0000","last_became_active":"2023-11-24T08:22:51.799310+0000","last_became_peered":"2023-11-24T08:22:51.799310+0000","last_unstale":"2023-11-24T08:22:52.924643+0000","last_undegraded":"2023-11-24T08:22:52.924643+0000","last_fullsized":"2023-11-24T08:22:52.924643+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:22:45.012196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921984+0000","last_change":"2023-11-24T08:22:51.773425+0000","last_active":"2023-11-24T08:22:52.921984+0000","last_peered":"2023-11-24T08:22:52.921984+0000","last_clean":"2023-11-24T08:22:52.921984+0000","last_became_active":"2023-11-24T08:22:51.773104+0000","last_became_peered":"2023-11-24T08:22:51.773104+0000","last_unstale":"2023-11-24T08:22:52.921984+0000","last_undegraded":"2023-11-24T08:22:52.921984+0000","last_fullsized":"2023-11-24T08:22:52.921984+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:22:33.620021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"219'4","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925639+0000","last_change":"2023-11-24T08:22:50.781535+0000","last_active":"2023-11-24T08:22:52.925639+0000","last_peered":"2023-11-24T08:22:52.925639+0000","last_clean":"2023-11-24T08:22:52.925639+0000","last_became_active":"2023-11-24T08:22:50.781248+0000","last_became_peered":"2023-11-24T08:22:50.781248+0000","last_unstale":"2023-11-24T08:22:52.925639+0000","last_undegraded":"2023-11-24T08:22:52.925639+0000","last_fullsized":"2023-11-24T08:22:52.925639+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:33.342667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926371+0000","last_change":"2023-11-24T08:22:51.799930+0000","last_active":"2023-11-24T08:22:52.926371+0000","last_peered":"2023-11-24T08:22:52.926371+0000","last_clean":"2023-11-24T08:22:52.926371+0000","last_became_active":"2023-11-24T08:22:51.799619+0000","last_became_peered":"2023-11-24T08:22:51.799619+0000","last_unstale":"2023-11-24T08:22:52.926371+0000","last_undegraded":"2023-11-24T08:22:52.926371+0000","last_fullsized":"2023-11-24T08:22:52.926371+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788781+0000","last_change":"2023-11-24T08:22:50.778878+0000","last_active":"2023-11-24T08:22:52.788781+0000","last_peered":"2023-11-24T08:22:52.788781+0000","last_clean":"2023-11-24T08:22:52.788781+0000","last_became_active":"2023-11-24T08:22:50.776758+0000","last_became_peered":"2023-11-24T08:22:50.776758+0000","last_unstale":"2023-11-24T08:22:52.788781+0000","last_undegraded":"2023-11-24T08:22:52.788781+0000","last_fullsized":"2023-11-24T08:22:52.788781+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:51.659657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925713+0000","last_change":"2023-11-24T08:22:51.787267+0000","last_active":"2023-11-24T08:22:52.925713+0000","last_peered":"2023-11-24T08:22:52.925713+0000","last_clean":"2023-11-24T08:22:52.925713+0000","last_became_active":"2023-11-24T08:22:51.785985+0000","last_became_peered":"2023-11-24T08:22:51.785985+0000","last_unstale":"2023-11-24T08:22:52.925713+0000","last_undegraded":"2023-11-24T08:22:52.925713+0000","last_fullsized":"2023-11-24T08:22:52.925713+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:14:24.572297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784670+0000","last_change":"2023-11-24T08:22:50.779628+0000","last_active":"2023-11-24T08:22:52.784670+0000","last_peered":"2023-11-24T08:22:52.784670+0000","last_clean":"2023-11-24T08:22:52.784670+0000","last_became_active":"2023-11-24T08:22:50.779381+0000","last_became_peered":"2023-11-24T08:22:50.779381+0000","last_unstale":"2023-11-24T08:22:52.784670+0000","last_undegraded":"2023-11-24T08:22:52.784670+0000","last_fullsized":"2023-11-24T08:22:52.784670+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:46:15.759319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784583+0000","last_change":"2023-11-24T08:22:51.779434+0000","last_active":"2023-11-24T08:22:52.784583+0000","last_peered":"2023-11-24T08:22:52.784583+0000","last_clean":"2023-11-24T08:22:52.784583+0000","last_became_active":"2023-11-24T08:22:51.777412+0000","last_became_peered":"2023-11-24T08:22:51.777412+0000","last_unstale":"2023-11-24T08:22:52.784583+0000","last_undegraded":"2023-11-24T08:22:52.784583+0000","last_fullsized":"2023-11-24T08:22:52.784583+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924005+0000","last_change":"2023-11-24T08:22:50.778954+0000","last_active":"2023-11-24T08:22:52.924005+0000","last_peered":"2023-11-24T08:22:52.924005+0000","last_clean":"2023-11-24T08:22:52.924005+0000","last_became_active":"2023-11-24T08:22:50.778634+0000","last_became_peered":"2023-11-24T08:22:50.778634+0000","last_unstale":"2023-11-24T08:22:52.924005+0000","last_undegraded":"2023-11-24T08:22:52.924005+0000","last_fullsized":"2023-11-24T08:22:52.924005+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:43.393903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924200+0000","last_change":"2023-11-24T08:22:51.792720+0000","last_active":"2023-11-24T08:22:52.924200+0000","last_peered":"2023-11-24T08:22:52.924200+0000","last_clean":"2023-11-24T08:22:52.924200+0000","last_became_active":"2023-11-24T08:22:51.790380+0000","last_became_peered":"2023-11-24T08:22:51.790380+0000","last_unstale":"2023-11-24T08:22:52.924200+0000","last_undegraded":"2023-11-24T08:22:52.924200+0000","last_fullsized":"2023-11-24T08:22:52.924200+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:01.821084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784976+0000","last_change":"2023-11-24T08:22:50.774941+0000","last_active":"2023-11-24T08:22:52.784976+0000","last_peered":"2023-11-24T08:22:52.784976+0000","last_clean":"2023-11-24T08:22:52.784976+0000","last_became_active":"2023-11-24T08:22:50.774679+0000","last_became_peered":"2023-11-24T08:22:50.774679+0000","last_unstale":"2023-11-24T08:22:52.784976+0000","last_undegraded":"2023-11-24T08:22:52.784976+0000","last_fullsized":"2023-11-24T08:22:52.784976+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:14.978362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922087+0000","last_change":"2023-11-24T08:22:51.804145+0000","last_active":"2023-11-24T08:22:52.922087+0000","last_peered":"2023-11-24T08:22:52.922087+0000","last_clean":"2023-11-24T08:22:52.922087+0000","last_became_active":"2023-11-24T08:22:51.803860+0000","last_became_peered":"2023-11-24T08:22:51.803860+0000","last_unstale":"2023-11-24T08:22:52.922087+0000","last_undegraded":"2023-11-24T08:22:52.922087+0000","last_fullsized":"2023-11-24T08:22:52.922087+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:06.393248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921290+0000","last_change":"2023-11-24T08:22:50.775057+0000","last_active":"2023-11-24T08:22:52.921290+0000","last_peered":"2023-11-24T08:22:52.921290+0000","last_clean":"2023-11-24T08:22:52.921290+0000","last_became_active":"2023-11-24T08:22:50.774786+0000","last_became_peered":"2023-11-24T08:22:50.774786+0000","last_unstale":"2023-11-24T08:22:52.921290+0000","last_undegraded":"2023-11-24T08:22:52.921290+0000","last_fullsized":"2023-11-24T08:22:52.921290+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:16.893872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926232+0000","last_change":"2023-11-24T08:22:51.796536+0000","last_active":"2023-11-24T08:22:52.926232+0000","last_peered":"2023-11-24T08:22:52.926232+0000","last_clean":"2023-11-24T08:22:52.926232+0000","last_became_active":"2023-11-24T08:22:51.793407+0000","last_became_peered":"2023-11-24T08:22:51.793407+0000","last_unstale":"2023-11-24T08:22:52.926232+0000","last_undegraded":"2023-11-24T08:22:52.926232+0000","last_fullsized":"2023-11-24T08:22:52.926232+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:13:33.590373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924665+0000","last_change":"2023-11-24T08:22:50.773854+0000","last_active":"2023-11-24T08:22:52.924665+0000","last_peered":"2023-11-24T08:22:52.924665+0000","last_clean":"2023-11-24T08:22:52.924665+0000","last_became_active":"2023-11-24T08:22:50.773253+0000","last_became_peered":"2023-11-24T08:22:50.773253+0000","last_unstale":"2023-11-24T08:22:52.924665+0000","last_undegraded":"2023-11-24T08:22:52.924665+0000","last_fullsized":"2023-11-24T08:22:52.924665+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:31:49.360836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923617+0000","last_change":"2023-11-24T08:22:51.785163+0000","last_active":"2023-11-24T08:22:52.923617+0000","last_peered":"2023-11-24T08:22:52.923617+0000","last_clean":"2023-11-24T08:22:52.923617+0000","last_became_active":"2023-11-24T08:22:51.784863+0000","last_became_peered":"2023-11-24T08:22:51.784863+0000","last_unstale":"2023-11-24T08:22:52.923617+0000","last_undegraded":"2023-11-24T08:22:52.923617+0000","last_fullsized":"2023-11-24T08:22:52.923617+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:20.243410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784429+0000","last_change":"2023-11-24T08:22:50.781824+0000","last_active":"2023-11-24T08:22:52.784429+0000","last_peered":"2023-11-24T08:22:52.784429+0000","last_clean":"2023-11-24T08:22:52.784429+0000","last_became_active":"2023-11-24T08:22:50.781414+0000","last_became_peered":"2023-11-24T08:22:50.781414+0000","last_unstale":"2023-11-24T08:22:52.784429+0000","last_undegraded":"2023-11-24T08:22:52.784429+0000","last_fullsized":"2023-11-24T08:22:52.784429+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:48.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787394+0000","last_change":"2023-11-24T08:22:51.776040+0000","last_active":"2023-11-24T08:22:52.787394+0000","last_peered":"2023-11-24T08:22:52.787394+0000","last_clean":"2023-11-24T08:22:52.787394+0000","last_became_active":"2023-11-24T08:22:51.773527+0000","last_became_peered":"2023-11-24T08:22:51.773527+0000","last_unstale":"2023-11-24T08:22:52.787394+0000","last_undegraded":"2023-11-24T08:22:52.787394+0000","last_fullsized":"2023-11-24T08:22:52.787394+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:44.597552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922466+0000","last_change":"2023-11-24T08:22:50.781407+0000","last_active":"2023-11-24T08:22:52.922466+0000","last_peered":"2023-11-24T08:22:52.922466+0000","last_clean":"2023-11-24T08:22:52.922466+0000","last_became_active":"2023-11-24T08:22:50.781148+0000","last_became_peered":"2023-11-24T08:22:50.781148+0000","last_unstale":"2023-11-24T08:22:52.922466+0000","last_undegraded":"2023-11-24T08:22:52.922466+0000","last_fullsized":"2023-11-24T08:22:52.922466+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:13:55.550456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921681+0000","last_change":"2023-11-24T08:22:51.799971+0000","last_active":"2023-11-24T08:22:52.921681+0000","last_peered":"2023-11-24T08:22:52.921681+0000","last_clean":"2023-11-24T08:22:52.921681+0000","last_became_active":"2023-11-24T08:22:51.799676+0000","last_became_peered":"2023-11-24T08:22:51.799676+0000","last_unstale":"2023-11-24T08:22:52.921681+0000","last_undegraded":"2023-11-24T08:22:52.921681+0000","last_fullsized":"2023-11-24T08:22:52.921681+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:32:14.080962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"219'3","reported_seq":15,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925521+0000","last_change":"2023-11-24T08:22:50.783856+0000","last_active":"2023-11-24T08:22:52.925521+0000","last_peered":"2023-11-24T08:22:52.925521+0000","last_clean":"2023-11-24T08:22:52.925521+0000","last_became_active":"2023-11-24T08:22:50.783314+0000","last_became_peered":"2023-11-24T08:22:50.783314+0000","last_unstale":"2023-11-24T08:22:52.925521+0000","last_undegraded":"2023-11-24T08:22:52.925521+0000","last_fullsized":"2023-11-24T08:22:52.925521+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:23:48.657613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922946+0000","last_change":"2023-11-24T08:22:51.797312+0000","last_active":"2023-11-24T08:22:52.922946+0000","last_peered":"2023-11-24T08:22:52.922946+0000","last_clean":"2023-11-24T08:22:52.922946+0000","last_became_active":"2023-11-24T08:22:51.795064+0000","last_became_peered":"2023-11-24T08:22:51.795064+0000","last_unstale":"2023-11-24T08:22:52.922946+0000","last_undegraded":"2023-11-24T08:22:52.922946+0000","last_fullsized":"2023-11-24T08:22:52.922946+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:27:38.465084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"219'6","reported_seq":18,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925408+0000","last_change":"2023-11-24T08:22:50.781012+0000","last_active":"2023-11-24T08:22:52.925408+0000","last_peered":"2023-11-24T08:22:52.925408+0000","last_clean":"2023-11-24T08:22:52.925408+0000","last_became_active":"2023-11-24T08:22:50.780110+0000","last_became_peered":"2023-11-24T08:22:50.780110+0000","last_unstale":"2023-11-24T08:22:52.925408+0000","last_undegraded":"2023-11-24T08:22:52.925408+0000","last_fullsized":"2023-11-24T08:22:52.925408+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:38:50.896631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923426+0000","last_change":"2023-11-24T08:22:51.782867+0000","last_active":"2023-11-24T08:22:52.923426+0000","last_peered":"2023-11-24T08:22:52.923426+0000","last_clean":"2023-11-24T08:22:52.923426+0000","last_became_active":"2023-11-24T08:22:51.782516+0000","last_became_peered":"2023-11-24T08:22:51.782516+0000","last_unstale":"2023-11-24T08:22:52.923426+0000","last_undegraded":"2023-11-24T08:22:52.923426+0000","last_fullsized":"2023-11-24T08:22:52.923426+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:06.822674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925789+0000","last_change":"2023-11-24T08:22:50.776490+0000","last_active":"2023-11-24T08:22:52.925789+0000","last_peered":"2023-11-24T08:22:52.925789+0000","last_clean":"2023-11-24T08:22:52.925789+0000","last_became_active":"2023-11-24T08:22:50.776208+0000","last_became_peered":"2023-11-24T08:22:50.776208+0000","last_unstale":"2023-11-24T08:22:52.925789+0000","last_undegraded":"2023-11-24T08:22:52.925789+0000","last_fullsized":"2023-11-24T08:22:52.925789+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:02:37.489075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924304+0000","last_change":"2023-11-24T08:22:51.791619+0000","last_active":"2023-11-24T08:22:52.924304+0000","last_peered":"2023-11-24T08:22:52.924304+0000","last_clean":"2023-11-24T08:22:52.924304+0000","last_became_active":"2023-11-24T08:22:51.790781+0000","last_became_peered":"2023-11-24T08:22:51.790781+0000","last_unstale":"2023-11-24T08:22:52.924304+0000","last_undegraded":"2023-11-24T08:22:52.924304+0000","last_fullsized":"2023-11-24T08:22:52.924304+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:13:56.568802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925247+0000","last_change":"2023-11-24T08:22:50.786246+0000","last_active":"2023-11-24T08:22:52.925247+0000","last_peered":"2023-11-24T08:22:52.925247+0000","last_clean":"2023-11-24T08:22:52.925247+0000","last_became_active":"2023-11-24T08:22:50.785939+0000","last_became_peered":"2023-11-24T08:22:50.785939+0000","last_unstale":"2023-11-24T08:22:52.925247+0000","last_undegraded":"2023-11-24T08:22:52.925247+0000","last_fullsized":"2023-11-24T08:22:52.925247+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:44:03.310218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926642+0000","last_change":"2023-11-24T08:22:51.796637+0000","last_active":"2023-11-24T08:22:52.926642+0000","last_peered":"2023-11-24T08:22:52.926642+0000","last_clean":"2023-11-24T08:22:52.926642+0000","last_became_active":"2023-11-24T08:22:51.796304+0000","last_became_peered":"2023-11-24T08:22:51.796304+0000","last_unstale":"2023-11-24T08:22:52.926642+0000","last_undegraded":"2023-11-24T08:22:52.926642+0000","last_fullsized":"2023-11-24T08:22:52.926642+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:03:54.450966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924743+0000","last_change":"2023-11-24T08:22:50.779231+0000","last_active":"2023-11-24T08:22:52.924743+0000","last_peered":"2023-11-24T08:22:52.924743+0000","last_clean":"2023-11-24T08:22:52.924743+0000","last_became_active":"2023-11-24T08:22:50.778964+0000","last_became_peered":"2023-11-24T08:22:50.778964+0000","last_unstale":"2023-11-24T08:22:52.924743+0000","last_undegraded":"2023-11-24T08:22:52.924743+0000","last_fullsized":"2023-11-24T08:22:52.924743+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:24:00.337076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926313+0000","last_change":"2023-11-24T08:22:51.806877+0000","last_active":"2023-11-24T08:22:52.926313+0000","last_peered":"2023-11-24T08:22:52.926313+0000","last_clean":"2023-11-24T08:22:52.926313+0000","last_became_active":"2023-11-24T08:22:51.806471+0000","last_became_peered":"2023-11-24T08:22:51.806471+0000","last_unstale":"2023-11-24T08:22:52.926313+0000","last_undegraded":"2023-11-24T08:22:52.926313+0000","last_fullsized":"2023-11-24T08:22:52.926313+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:15.996708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"219'8","reported_seq":20,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922264+0000","last_change":"2023-11-24T08:22:50.775105+0000","last_active":"2023-11-24T08:22:52.922264+0000","last_peered":"2023-11-24T08:22:52.922264+0000","last_clean":"2023-11-24T08:22:52.922264+0000","last_became_active":"2023-11-24T08:22:50.774763+0000","last_became_peered":"2023-11-24T08:22:50.774763+0000","last_unstale":"2023-11-24T08:22:52.922264+0000","last_undegraded":"2023-11-24T08:22:52.922264+0000","last_fullsized":"2023-11-24T08:22:52.922264+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:24.996177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923293+0000","last_change":"2023-11-24T08:22:51.806834+0000","last_active":"2023-11-24T08:22:52.923293+0000","last_peered":"2023-11-24T08:22:52.923293+0000","last_clean":"2023-11-24T08:22:52.923293+0000","last_became_active":"2023-11-24T08:22:51.806537+0000","last_became_peered":"2023-11-24T08:22:51.806537+0000","last_unstale":"2023-11-24T08:22:52.923293+0000","last_undegraded":"2023-11-24T08:22:52.923293+0000","last_fullsized":"2023-11-24T08:22:52.923293+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:54:16.344938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925902+0000","last_change":"2023-11-24T08:22:50.785703+0000","last_active":"2023-11-24T08:22:52.925902+0000","last_peered":"2023-11-24T08:22:52.925902+0000","last_clean":"2023-11-24T08:22:52.925902+0000","last_became_active":"2023-11-24T08:22:50.785376+0000","last_became_peered":"2023-11-24T08:22:50.785376+0000","last_unstale":"2023-11-24T08:22:52.925902+0000","last_undegraded":"2023-11-24T08:22:52.925902+0000","last_fullsized":"2023-11-24T08:22:52.925902+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:12:53.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925406+0000","last_change":"2023-11-24T08:22:51.799712+0000","last_active":"2023-11-24T08:22:52.925406+0000","last_peered":"2023-11-24T08:22:52.925406+0000","last_clean":"2023-11-24T08:22:52.925406+0000","last_became_active":"2023-11-24T08:22:51.797643+0000","last_became_peered":"2023-11-24T08:22:51.797643+0000","last_unstale":"2023-11-24T08:22:52.925406+0000","last_undegraded":"2023-11-24T08:22:52.925406+0000","last_fullsized":"2023-11-24T08:22:52.925406+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:59:04.078503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924858+0000","last_change":"2023-11-24T08:22:50.780404+0000","last_active":"2023-11-24T08:22:52.924858+0000","last_peered":"2023-11-24T08:22:52.924858+0000","last_clean":"2023-11-24T08:22:52.924858+0000","last_became_active":"2023-11-24T08:22:50.780135+0000","last_became_peered":"2023-11-24T08:22:50.780135+0000","last_unstale":"2023-11-24T08:22:52.924858+0000","last_undegraded":"2023-11-24T08:22:52.924858+0000","last_fullsized":"2023-11-24T08:22:52.924858+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:32:31.491498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923228+0000","last_change":"2023-11-24T08:22:51.800842+0000","last_active":"2023-11-24T08:22:52.923228+0000","last_peered":"2023-11-24T08:22:52.923228+0000","last_clean":"2023-11-24T08:22:52.923228+0000","last_became_active":"2023-11-24T08:22:51.800557+0000","last_became_peered":"2023-11-24T08:22:51.800557+0000","last_unstale":"2023-11-24T08:22:52.923228+0000","last_undegraded":"2023-11-24T08:22:52.923228+0000","last_fullsized":"2023-11-24T08:22:52.923228+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:47.929727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789148+0000","last_change":"2023-11-24T08:22:50.778673+0000","last_active":"2023-11-24T08:22:52.789148+0000","last_peered":"2023-11-24T08:22:52.789148+0000","last_clean":"2023-11-24T08:22:52.789148+0000","last_became_active":"2023-11-24T08:22:50.776374+0000","last_became_peered":"2023-11-24T08:22:50.776374+0000","last_unstale":"2023-11-24T08:22:52.789148+0000","last_undegraded":"2023-11-24T08:22:52.789148+0000","last_fullsized":"2023-11-24T08:22:52.789148+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:58.378583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921938+0000","last_change":"2023-11-24T08:22:51.800647+0000","last_active":"2023-11-24T08:22:52.921938+0000","last_peered":"2023-11-24T08:22:52.921938+0000","last_clean":"2023-11-24T08:22:52.921938+0000","last_became_active":"2023-11-24T08:22:51.800376+0000","last_became_peered":"2023-11-24T08:22:51.800376+0000","last_unstale":"2023-11-24T08:22:52.921938+0000","last_undegraded":"2023-11-24T08:22:52.921938+0000","last_fullsized":"2023-11-24T08:22:52.921938+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:56:54.873686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922755+0000","last_change":"2023-11-24T08:22:50.779373+0000","last_active":"2023-11-24T08:22:52.922755+0000","last_peered":"2023-11-24T08:22:52.922755+0000","last_clean":"2023-11-24T08:22:52.922755+0000","last_became_active":"2023-11-24T08:22:50.779083+0000","last_became_peered":"2023-11-24T08:22:50.779083+0000","last_unstale":"2023-11-24T08:22:52.922755+0000","last_undegraded":"2023-11-24T08:22:52.922755+0000","last_fullsized":"2023-11-24T08:22:52.922755+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:02:14.978362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922647+0000","last_change":"2023-11-24T08:22:51.789865+0000","last_active":"2023-11-24T08:22:52.922647+0000","last_peered":"2023-11-24T08:22:52.922647+0000","last_clean":"2023-11-24T08:22:52.922647+0000","last_became_active":"2023-11-24T08:22:51.787758+0000","last_became_peered":"2023-11-24T08:22:51.787758+0000","last_unstale":"2023-11-24T08:22:52.922647+0000","last_undegraded":"2023-11-24T08:22:52.922647+0000","last_fullsized":"2023-11-24T08:22:52.922647+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:57:33.688597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921945+0000","last_change":"2023-11-24T08:22:50.784445+0000","last_active":"2023-11-24T08:22:52.921945+0000","last_peered":"2023-11-24T08:22:52.921945+0000","last_clean":"2023-11-24T08:22:52.921945+0000","last_became_active":"2023-11-24T08:22:50.784160+0000","last_became_peered":"2023-11-24T08:22:50.784160+0000","last_unstale":"2023-11-24T08:22:52.921945+0000","last_undegraded":"2023-11-24T08:22:52.921945+0000","last_fullsized":"2023-11-24T08:22:52.921945+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:27:37.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923915+0000","last_change":"2023-11-24T08:22:51.780327+0000","last_active":"2023-11-24T08:22:52.923915+0000","last_peered":"2023-11-24T08:22:52.923915+0000","last_clean":"2023-11-24T08:22:52.923915+0000","last_became_active":"2023-11-24T08:22:51.779991+0000","last_became_peered":"2023-11-24T08:22:51.779991+0000","last_unstale":"2023-11-24T08:22:52.923915+0000","last_undegraded":"2023-11-24T08:22:52.923915+0000","last_fullsized":"2023-11-24T08:22:52.923915+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:52.678003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787970+0000","last_change":"2023-11-24T08:22:50.777143+0000","last_active":"2023-11-24T08:22:52.787970+0000","last_peered":"2023-11-24T08:22:52.787970+0000","last_clean":"2023-11-24T08:22:52.787970+0000","last_became_active":"2023-11-24T08:22:50.776825+0000","last_became_peered":"2023-11-24T08:22:50.776825+0000","last_unstale":"2023-11-24T08:22:52.787970+0000","last_undegraded":"2023-11-24T08:22:52.787970+0000","last_fullsized":"2023-11-24T08:22:52.787970+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:59:26.038566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789041+0000","last_change":"2023-11-24T08:22:51.801047+0000","last_active":"2023-11-24T08:22:52.789041+0000","last_peered":"2023-11-24T08:22:52.789041+0000","last_clean":"2023-11-24T08:22:52.789041+0000","last_became_active":"2023-11-24T08:22:51.800767+0000","last_became_peered":"2023-11-24T08:22:51.800767+0000","last_unstale":"2023-11-24T08:22:52.789041+0000","last_undegraded":"2023-11-24T08:22:52.789041+0000","last_fullsized":"2023-11-24T08:22:52.789041+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:27:36.559163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787428+0000","last_change":"2023-11-24T08:22:50.778643+0000","last_active":"2023-11-24T08:22:52.787428+0000","last_peered":"2023-11-24T08:22:52.787428+0000","last_clean":"2023-11-24T08:22:52.787428+0000","last_became_active":"2023-11-24T08:22:50.776369+0000","last_became_peered":"2023-11-24T08:22:50.776369+0000","last_unstale":"2023-11-24T08:22:52.787428+0000","last_undegraded":"2023-11-24T08:22:52.787428+0000","last_fullsized":"2023-11-24T08:22:52.787428+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:42:21.388748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.789037+0000","last_change":"2023-11-24T08:22:51.789812+0000","last_active":"2023-11-24T08:22:52.789037+0000","last_peered":"2023-11-24T08:22:52.789037+0000","last_clean":"2023-11-24T08:22:52.789037+0000","last_became_active":"2023-11-24T08:22:51.789556+0000","last_became_peered":"2023-11-24T08:22:51.789556+0000","last_unstale":"2023-11-24T08:22:52.789037+0000","last_undegraded":"2023-11-24T08:22:52.789037+0000","last_fullsized":"2023-11-24T08:22:52.789037+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:58.378488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784151+0000","last_change":"2023-11-24T08:22:50.781615+0000","last_active":"2023-11-24T08:22:52.784151+0000","last_peered":"2023-11-24T08:22:52.784151+0000","last_clean":"2023-11-24T08:22:52.784151+0000","last_became_active":"2023-11-24T08:22:50.781317+0000","last_became_peered":"2023-11-24T08:22:50.781317+0000","last_unstale":"2023-11-24T08:22:52.784151+0000","last_undegraded":"2023-11-24T08:22:52.784151+0000","last_fullsized":"2023-11-24T08:22:52.784151+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:04:39.530314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.922552+0000","last_change":"2023-11-24T08:22:51.792983+0000","last_active":"2023-11-24T08:22:52.922552+0000","last_peered":"2023-11-24T08:22:52.922552+0000","last_clean":"2023-11-24T08:22:52.922552+0000","last_became_active":"2023-11-24T08:22:51.790323+0000","last_became_peered":"2023-11-24T08:22:51.790323+0000","last_unstale":"2023-11-24T08:22:52.922552+0000","last_undegraded":"2023-11-24T08:22:52.922552+0000","last_fullsized":"2023-11-24T08:22:52.922552+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:54:11.361938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"219'2","reported_seq":14,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923460+0000","last_change":"2023-11-24T08:22:50.778404+0000","last_active":"2023-11-24T08:22:52.923460+0000","last_peered":"2023-11-24T08:22:52.923460+0000","last_clean":"2023-11-24T08:22:52.923460+0000","last_became_active":"2023-11-24T08:22:50.778115+0000","last_became_peered":"2023-11-24T08:22:50.778115+0000","last_unstale":"2023-11-24T08:22:52.923460+0000","last_undegraded":"2023-11-24T08:22:52.923460+0000","last_fullsized":"2023-11-24T08:22:52.923460+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:40:18.495861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924050+0000","last_change":"2023-11-24T08:22:51.792274+0000","last_active":"2023-11-24T08:22:52.924050+0000","last_peered":"2023-11-24T08:22:52.924050+0000","last_clean":"2023-11-24T08:22:52.924050+0000","last_became_active":"2023-11-24T08:22:51.791963+0000","last_became_peered":"2023-11-24T08:22:51.791963+0000","last_unstale":"2023-11-24T08:22:52.924050+0000","last_undegraded":"2023-11-24T08:22:52.924050+0000","last_fullsized":"2023-11-24T08:22:52.924050+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:44.779376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.926004+0000","last_change":"2023-11-24T08:22:50.784469+0000","last_active":"2023-11-24T08:22:52.926004+0000","last_peered":"2023-11-24T08:22:52.926004+0000","last_clean":"2023-11-24T08:22:52.926004+0000","last_became_active":"2023-11-24T08:22:50.784101+0000","last_became_peered":"2023-11-24T08:22:50.784101+0000","last_unstale":"2023-11-24T08:22:52.926004+0000","last_undegraded":"2023-11-24T08:22:52.926004+0000","last_fullsized":"2023-11-24T08:22:52.926004+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:40.933218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921443+0000","last_change":"2023-11-24T08:22:51.800590+0000","last_active":"2023-11-24T08:22:52.921443+0000","last_peered":"2023-11-24T08:22:52.921443+0000","last_clean":"2023-11-24T08:22:52.921443+0000","last_became_active":"2023-11-24T08:22:51.799524+0000","last_became_peered":"2023-11-24T08:22:51.799524+0000","last_unstale":"2023-11-24T08:22:52.921443+0000","last_undegraded":"2023-11-24T08:22:52.921443+0000","last_fullsized":"2023-11-24T08:22:52.921443+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:58:21.460691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784827+0000","last_change":"2023-11-24T08:22:50.785737+0000","last_active":"2023-11-24T08:22:52.784827+0000","last_peered":"2023-11-24T08:22:52.784827+0000","last_clean":"2023-11-24T08:22:52.784827+0000","last_became_active":"2023-11-24T08:22:50.784848+0000","last_became_peered":"2023-11-24T08:22:50.784848+0000","last_unstale":"2023-11-24T08:22:52.784827+0000","last_undegraded":"2023-11-24T08:22:52.784827+0000","last_fullsized":"2023-11-24T08:22:52.784827+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:43.393903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925624+0000","last_change":"2023-11-24T08:22:51.806833+0000","last_active":"2023-11-24T08:22:52.925624+0000","last_peered":"2023-11-24T08:22:52.925624+0000","last_clean":"2023-11-24T08:22:52.925624+0000","last_became_active":"2023-11-24T08:22:51.806542+0000","last_became_peered":"2023-11-24T08:22:51.806542+0000","last_unstale":"2023-11-24T08:22:52.925624+0000","last_undegraded":"2023-11-24T08:22:52.925624+0000","last_fullsized":"2023-11-24T08:22:52.925624+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:55:10.306378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784685+0000","last_change":"2023-11-24T08:22:50.785247+0000","last_active":"2023-11-24T08:22:52.784685+0000","last_peered":"2023-11-24T08:22:52.784685+0000","last_clean":"2023-11-24T08:22:52.784685+0000","last_became_active":"2023-11-24T08:22:50.783812+0000","last_became_peered":"2023-11-24T08:22:50.783812+0000","last_unstale":"2023-11-24T08:22:52.784685+0000","last_undegraded":"2023-11-24T08:22:52.784685+0000","last_fullsized":"2023-11-24T08:22:52.784685+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:43.761030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.784537+0000","last_change":"2023-11-24T08:22:51.800739+0000","last_active":"2023-11-24T08:22:52.784537+0000","last_peered":"2023-11-24T08:22:52.784537+0000","last_clean":"2023-11-24T08:22:52.784537+0000","last_became_active":"2023-11-24T08:22:51.792824+0000","last_became_peered":"2023-11-24T08:22:51.792824+0000","last_unstale":"2023-11-24T08:22:52.784537+0000","last_undegraded":"2023-11-24T08:22:52.784537+0000","last_fullsized":"2023-11-24T08:22:52.784537+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:31.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788240+0000","last_change":"2023-11-24T08:22:50.778327+0000","last_active":"2023-11-24T08:22:52.788240+0000","last_peered":"2023-11-24T08:22:52.788240+0000","last_clean":"2023-11-24T08:22:52.788240+0000","last_became_active":"2023-11-24T08:22:50.778023+0000","last_became_peered":"2023-11-24T08:22:50.778023+0000","last_unstale":"2023-11-24T08:22:52.788240+0000","last_undegraded":"2023-11-24T08:22:52.788240+0000","last_fullsized":"2023-11-24T08:22:52.788240+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:52:05.804327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925577+0000","last_change":"2023-11-24T08:22:51.780715+0000","last_active":"2023-11-24T08:22:52.925577+0000","last_peered":"2023-11-24T08:22:52.925577+0000","last_clean":"2023-11-24T08:22:52.925577+0000","last_became_active":"2023-11-24T08:22:51.780439+0000","last_became_peered":"2023-11-24T08:22:51.780439+0000","last_unstale":"2023-11-24T08:22:52.925577+0000","last_undegraded":"2023-11-24T08:22:52.925577+0000","last_fullsized":"2023-11-24T08:22:52.925577+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:21.447010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.925507+0000","last_change":"2023-11-24T08:22:50.787361+0000","last_active":"2023-11-24T08:22:52.925507+0000","last_peered":"2023-11-24T08:22:52.925507+0000","last_clean":"2023-11-24T08:22:52.925507+0000","last_became_active":"2023-11-24T08:22:50.786928+0000","last_became_peered":"2023-11-24T08:22:50.786928+0000","last_unstale":"2023-11-24T08:22:52.925507+0000","last_undegraded":"2023-11-24T08:22:52.925507+0000","last_fullsized":"2023-11-24T08:22:52.925507+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:49.871592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924631+0000","last_change":"2023-11-24T08:22:51.777572+0000","last_active":"2023-11-24T08:22:52.924631+0000","last_peered":"2023-11-24T08:22:52.924631+0000","last_clean":"2023-11-24T08:22:52.924631+0000","last_became_active":"2023-11-24T08:22:51.777330+0000","last_became_peered":"2023-11-24T08:22:51.777330+0000","last_unstale":"2023-11-24T08:22:52.924631+0000","last_undegraded":"2023-11-24T08:22:52.924631+0000","last_fullsized":"2023-11-24T08:22:52.924631+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:24.144535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.921373+0000","last_change":"2023-11-24T08:22:50.776498+0000","last_active":"2023-11-24T08:22:52.921373+0000","last_peered":"2023-11-24T08:22:52.921373+0000","last_clean":"2023-11-24T08:22:52.921373+0000","last_became_active":"2023-11-24T08:22:50.775902+0000","last_became_peered":"2023-11-24T08:22:50.775902+0000","last_unstale":"2023-11-24T08:22:52.921373+0000","last_undegraded":"2023-11-24T08:22:52.921373+0000","last_fullsized":"2023-11-24T08:22:52.921373+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:12:41.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.788719+0000","last_change":"2023-11-24T08:22:51.774749+0000","last_active":"2023-11-24T08:22:52.788719+0000","last_peered":"2023-11-24T08:22:52.788719+0000","last_clean":"2023-11-24T08:22:52.788719+0000","last_became_active":"2023-11-24T08:22:51.773418+0000","last_became_peered":"2023-11-24T08:22:51.773418+0000","last_unstale":"2023-11-24T08:22:52.788719+0000","last_undegraded":"2023-11-24T08:22:52.788719+0000","last_fullsized":"2023-11-24T08:22:52.788719+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:49.605933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.787479+0000","last_change":"2023-11-24T08:22:50.779076+0000","last_active":"2023-11-24T08:22:52.787479+0000","last_peered":"2023-11-24T08:22:52.787479+0000","last_clean":"2023-11-24T08:22:52.787479+0000","last_became_active":"2023-11-24T08:22:50.777061+0000","last_became_peered":"2023-11-24T08:22:50.777061+0000","last_unstale":"2023-11-24T08:22:52.787479+0000","last_undegraded":"2023-11-24T08:22:52.787479+0000","last_fullsized":"2023-11-24T08:22:52.787479+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:21:23.126189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.924603+0000","last_change":"2023-11-24T08:22:51.772269+0000","last_active":"2023-11-24T08:22:52.924603+0000","last_peered":"2023-11-24T08:22:52.924603+0000","last_clean":"2023-11-24T08:22:52.924603+0000","last_became_active":"2023-11-24T08:22:51.771935+0000","last_became_peered":"2023-11-24T08:22:51.771935+0000","last_unstale":"2023-11-24T08:22:52.924603+0000","last_undegraded":"2023-11-24T08:22:52.924603+0000","last_fullsized":"2023-11-24T08:22:52.924603+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":218,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:50.731519+0000","last_clean_scrub_stamp":"2023-11-24T08:22:50.731519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:16.718366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"219'1","reported_seq":13,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-24T08:22:52.923896+0000","last_change":"2023-11-24T08:22:50.775624+0000","last_active":"2023-11-24T08:22:52.923896+0000","last_peered":"2023-11-24T08:22:52.923896+0000","last_clean":"2023-11-24T08:22:52.923896+0000","last_became_active":"2023-11-24T08:22:50.775271+0000","last_became_peered":"2023-11-24T08:22:50.775271+0000","last_unstale":"2023-11-24T08:22:52.923896+0000","last_undegraded":"2023-11-24T08:22:52.923896+0000","last_fullsized":"2023-11-24T08:22:52.923896+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:22:49.713173+0000","last_clean_scrub_stamp":"2023-11-24T08:22:49.713173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:36:24.106567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":342,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-11-24T08:22:51.778949+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:22:51.778949+0000","last_peered":"2023-11-24T08:22:51.778949+0000","last_clean":"2023-11-24T08:22:51.778949+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:22:51.778949+0000","last_undegraded":"2023-11-24T08:22:51.778949+0000","last_fullsized":"2023-11-24T08:22:51.778949+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42808,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329032,"statfs":{"total":96636764160,"available":96592928768,"internally_reserved":0,"allocated":3383296,"data_stored":1827398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0920000000000001,"15min":1.0620000000000001},"min":{"1min":0.61199999999999999,"5min":0.57199999999999995,"15min":0.42399999999999999},"max":{"1min":1.4790000000000001,"5min":2.048,"15min":2.7240000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1180000000000001},"min":{"1min":0.86199999999999999,"5min":0.67200000000000004,"15min":0.53700000000000003},"max":{"1min":1.5109999999999999,"5min":1.899,"15min":2.7109999999999999},"last":0.85299999999999998}]},{"osd":1,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.139,"15min":1.1519999999999999},"min":{"1min":0.748,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.4079999999999999,"5min":1.681,"15min":3.8519999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.628},"max":{"1min":1.6519999999999999,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":0.97199999999999998}]},{"osd":2,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0940000000000001,"15min":1.1220000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.478,"5min":1.762,"15min":2.4430000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.246,"15min":1.2889999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.60399999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":4.0650000000000004},"last":1.282}]},{"osd":3,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1910000000000001,"15min":1.2090000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.504},"max":{"1min":1.5760000000000001,"5min":1.6240000000000001,"15min":3.133},"last":1.363},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3160000000000001,"15min":1.359},"min":{"1min":0.89200000000000002,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.9710000000000001,"5min":2.173,"15min":3.9209999999999998},"last":1.4319999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3240000000000001,"15min":1.3480000000000001},"min":{"1min":0.94399999999999995,"5min":0.84099999999999997,"15min":0.54700000000000004},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":2.851},"last":1.319},{"interface":"front","average":{"1min":1.248,"5min":1.325,"15min":1.351},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.675,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.5129999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.423,"15min":1.464},"min":{"1min":0.89600000000000002,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":1.8180000000000001,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.4490000000000001,"15min":1.4910000000000001},"min":{"1min":1.0760000000000001,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.722}]},{"osd":6,"last update":"Fri Nov 24 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5569999999999999,"15min":1.573},"min":{"1min":1.1499999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.992},"last":1.627},{"interface":"front","average":{"1min":1.401,"5min":1.46,"15min":1.498},"min":{"1min":1.115,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.7030000000000001,"5min":2.4159999999999999,"15min":3.798},"last":1.6060000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2789376,"data_stored":1237018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.028,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.59099999999999997},"max":{"1min":1.552,"5min":1.7929999999999999,"15min":1.95},"last":1.3029999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.095,"15min":1.081},"min":{"1min":0.67600000000000005,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.379,"5min":2.008,"15min":2.008},"last":1.456}]},{"osd":1,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.61599999999999999},"max":{"1min":1.581,"5min":1.8100000000000001,"15min":2.0230000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.1779999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.65000000000000002},"max":{"1min":1.6699999999999999,"5min":2.206,"15min":2.206},"last":1.3460000000000001}]},{"osd":2,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.115,"15min":1.1080000000000001},"min":{"1min":0.78900000000000003,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2150000000000001,"15min":1.2430000000000001},"min":{"1min":0.72999999999999998,"5min":0.72599999999999998,"15min":0.60399999999999998},"max":{"1min":1.518,"5min":1.7,"15min":2.0369999999999999},"last":1.409}]},{"osd":3,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.343,"15min":1.333},"min":{"1min":0.874,"5min":0.77100000000000002,"15min":0.71499999999999997},"max":{"1min":1.7989999999999999,"5min":2.6850000000000001,"15min":4.7880000000000003},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2669999999999999,"15min":1.298},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.794,"5min":1.907,"15min":4.8319999999999999},"last":1.367}]},{"osd":4,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.381,"15min":1.4019999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0209999999999999,"5min":3.673,"15min":3.673},"last":1.117},{"interface":"front","average":{"1min":1.397,"5min":1.4590000000000001,"15min":1.4710000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9850000000000001,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":1.198}]},{"osd":5,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.345,"15min":1.3999999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.8180000000000001,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5389999999999999,"15min":1.546},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8080000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.54}]},{"osd":7,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5800000000000001,"15min":1.5740000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.2599999999999998,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.395},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5449999999999999,"15min":1.571},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1400000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4330000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38052,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333788,"statfs":{"total":96636764160,"available":96597798912,"internally_reserved":0,"allocated":2768896,"data_stored":1230606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.016},"min":{"1min":0.61599999999999999,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.696,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.135,"15min":1.1020000000000001},"min":{"1min":0.70099999999999996,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.6639999999999999,"5min":5.4299999999999997,"15min":5.4299999999999997},"last":1.4179999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.177,"15min":1.131},"min":{"1min":0.65000000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.665,"5min":7.258,"15min":7.258},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.159,"5min":1.2190000000000001,"15min":1.1899999999999999},"min":{"1min":0.80200000000000005,"5min":0.70699999999999996,"15min":0.48699999999999999},"max":{"1min":1.653,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2210000000000001,"15min":1.1830000000000001},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.51600000000000001},"max":{"1min":1.556,"5min":6.8929999999999998,"15min":6.8929999999999998},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2390000000000001,"15min":1.177},"min":{"1min":0.96099999999999997,"5min":0.68000000000000005,"15min":0.64200000000000002},"max":{"1min":1.806,"5min":8.0869999999999997,"15min":8.0869999999999997},"last":1.0449999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.252,"15min":1.181},"min":{"1min":0.76200000000000001,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.5069999999999999,"5min":9.0120000000000005,"15min":9.0120000000000005},"last":1.121},{"interface":"front","average":{"1min":1.282,"5min":1.3320000000000001,"15min":1.278},"min":{"1min":0.90800000000000003,"5min":0.82799999999999996,"15min":0.55200000000000005},"max":{"1min":1.694,"5min":7.2130000000000001,"15min":7.2130000000000001},"last":1.581}]},{"osd":4,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.4850000000000001,"15min":1.383},"min":{"1min":0.98199999999999998,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.8240000000000001,"5min":11.813000000000001,"15min":11.813000000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5640000000000001,"15min":1.48},"min":{"1min":1.258,"5min":0.65900000000000003,"15min":0.46700000000000003},"max":{"1min":1.98,"5min":12.327999999999999,"15min":12.327999999999999},"last":1.667}]},{"osd":6,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3859999999999999,"15min":1.3420000000000001},"min":{"1min":0.97999999999999998,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.917,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":1.202},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.5069999999999999,"15min":1.448},"min":{"1min":0.88300000000000001,"5min":0.70299999999999996,"15min":0.60999999999999999},"max":{"1min":1.7669999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.369}]},{"osd":7,"last update":"Fri Nov 24 08:22:23 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.546,"15min":1.4970000000000001},"min":{"1min":0.877,"5min":0.86199999999999999,"15min":0.52900000000000003},"max":{"1min":2.0379999999999998,"5min":8.1660000000000004,"15min":8.1660000000000004},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.7649999999999999,"15min":1.647},"min":{"1min":1.2390000000000001,"5min":0.88600000000000001,"15min":0.71199999999999997},"max":{"1min":2.008,"5min":19.663,"15min":19.663},"last":1.627}]}]},{"osd":4,"up_from":13,"seq":55834575071,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2785280,"data_stored":1236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0469999999999999,"15min":1.0549999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.26,"5min":1.5860000000000001,"15min":1.8779999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.0840000000000001,"15min":1.0800000000000001},"min":{"1min":0.78300000000000003,"5min":0.75900000000000001,"15min":0.46100000000000002},"max":{"1min":1.5009999999999999,"5min":1.7070000000000001,"15min":5.452},"last":0.95899999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.115,"15min":1.1419999999999999},"min":{"1min":0.69899999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6359999999999999,"5min":1.6399999999999999,"15min":1.8100000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.103,"15min":1.1439999999999999},"min":{"1min":0.70799999999999996,"5min":0.63800000000000001,"15min":0.51900000000000002},"max":{"1min":1.329,"5min":1.7150000000000001,"15min":1.988},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0549999999999999,"15min":1.0600000000000001},"min":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.5149999999999999,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.244,"15min":1.24},"min":{"1min":0.84199999999999997,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.494,"5min":1.827,"15min":2.0470000000000002},"last":1.3200000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.337,"15min":1.331},"min":{"1min":1.012,"5min":0.754,"15min":0.62},"max":{"1min":1.6699999999999999,"5min":1.925,"15min":2.121},"last":1.431},{"interface":"front","average":{"1min":1.276,"5min":1.272,"15min":1.274},"min":{"1min":0.88700000000000001,"5min":0.60499999999999998,"15min":0.55800000000000005},"max":{"1min":1.744,"5min":1.744,"15min":2.3340000000000001},"last":0.85499999999999998}]},{"osd":5,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4890000000000001,"15min":1.4910000000000001},"min":{"1min":1.02,"5min":0.78900000000000003,"15min":0.69299999999999995},"max":{"1min":1.7390000000000001,"5min":2.0179999999999998,"15min":2.2429999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.375,"15min":1.4139999999999999},"min":{"1min":0.98199999999999998,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8520000000000001,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.2210000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2509999999999999,"15min":1.2450000000000001},"min":{"1min":0.81399999999999995,"5min":0.629,"15min":0.55400000000000005},"max":{"1min":1.659,"5min":1.8540000000000001,"15min":1.907},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.538,"15min":1.5349999999999999},"min":{"1min":1.242,"5min":0.73899999999999999,"15min":0.58099999999999996},"max":{"1min":1.897,"5min":2.1190000000000002,"15min":2.2890000000000001},"last":1.631}]},{"osd":7,"last update":"Fri Nov 24 08:22:24 2023","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.458,"15min":1.478},"min":{"1min":1.1399999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8,"5min":1.948,"15min":3.1219999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.415,"15min":1.403},"min":{"1min":1.159,"5min":0.69299999999999995,"15min":0.61799999999999999},"max":{"1min":2.077,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.2769999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575070,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":2805760,"data_stored":1239156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.82199999999999995,"15min":0.872},"min":{"1min":0.45700000000000002,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":1.2490000000000001,"5min":1.5820000000000001,"15min":10.557},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92600000000000005,"15min":0.92300000000000004},"min":{"1min":0.60699999999999998,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.234,"5min":1.653,"15min":3.8260000000000001},"last":1.089}]},{"osd":1,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":0.79600000000000004},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.95699999999999996,"5min":1.407,"15min":2.101},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89400000000000002,"15min":0.97499999999999998},"min":{"1min":0.58199999999999996,"5min":0.5,"15min":0.379},"max":{"1min":1.667,"5min":1.667,"15min":13.090999999999999},"last":1.1519999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.871,"15min":0.89400000000000002},"min":{"1min":0.54000000000000004,"5min":0.433,"15min":0.39900000000000002},"max":{"1min":1.4650000000000001,"5min":1.976,"15min":3.9489999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.97899999999999998,"15min":1.073},"min":{"1min":0.64400000000000002,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":1.5369999999999999,"5min":1.665,"15min":13.460000000000001},"last":0.755}]},{"osd":4,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2649999999999999,"15min":1.2709999999999999},"min":{"1min":1.0069999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.621,"5min":1.845,"15min":4.367},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.242,"15min":1.222},"min":{"1min":0.96999999999999997,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.4159999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.248,"15min":1.306},"min":{"1min":0.86599999999999999,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.5649999999999999,"5min":2.1230000000000002,"15min":13.231},"last":1.282},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2070000000000001,"15min":1.236},"min":{"1min":0.80000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.617,"5min":1.8620000000000001,"15min":5.1180000000000003},"last":1.6319999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.304,"15min":1.3660000000000001},"min":{"1min":0.81399999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.621,"5min":2.2549999999999999,"15min":13.359999999999999},"last":1.232},{"interface":"front","average":{"1min":1.266,"5min":1.24,"15min":1.244},"min":{"1min":0.89300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.736,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":1.4630000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:22:12 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2949999999999999,"15min":1.3089999999999999},"min":{"1min":0.93000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":1.73,"5min":2.0720000000000001,"15min":4.266},"last":1.734},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2949999999999999,"15min":1.3009999999999999},"min":{"1min":0.78500000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.744,"5min":3.0030000000000001,"15min":3.331},"last":1.669}]}]},{"osd":2,"up_from":13,"seq":55834575068,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2789376,"data_stored":1237558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.38200000000000001,"5min":0.34200000000000003,"15min":0.29999999999999999},"max":{"1min":1.468,"5min":1.5429999999999999,"15min":1.601},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.85499999999999998},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.123,"5min":1.4810000000000001,"15min":1.782},"last":0.95799999999999996}]},{"osd":1,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.54300000000000004,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.526,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.97199999999999998,"15min":0.95199999999999996},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":1.4019999999999999,"5min":1.6699999999999999,"15min":1.8360000000000001},"last":1.1990000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.89100000000000001,"15min":0.85999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":1.397,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.008,"15min":1.022},"min":{"1min":0.59399999999999997,"5min":0.52500000000000002,"15min":0.41399999999999998},"max":{"1min":1.395,"5min":1.948,"15min":1.948},"last":1.093}]},{"osd":4,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.268,"15min":1.2689999999999999},"min":{"1min":0.747,"5min":0.70699999999999996,"15min":0.53700000000000003},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.248,"15min":1.244},"min":{"1min":0.93899999999999995,"5min":0.78700000000000003,"15min":0.745},"max":{"1min":1.7609999999999999,"5min":1.8200000000000001,"15min":1.929},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.2190000000000001,"15min":1.2350000000000001},"min":{"1min":0.83699999999999997,"5min":0.753,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2569999999999999,"15min":1.238},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1919999999999999,"15min":1.1879999999999999},"min":{"1min":0.77200000000000002,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.7709999999999999,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":1.458},{"interface":"front","average":{"1min":1.28,"5min":1.3080000000000001,"15min":1.3140000000000001},"min":{"1min":0.92300000000000004,"5min":0.78800000000000003,"15min":0.65600000000000003},"max":{"1min":1.625,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.355}]},{"osd":7,"last update":"Fri Nov 24 08:22:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.319,"15min":1.3100000000000001},"min":{"1min":0.79900000000000004,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2929999999999999,"15min":1.282},"min":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.877,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.6910000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42216,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329624,"statfs":{"total":96636764160,"available":96593534976,"internally_reserved":0,"allocated":2781184,"data_stored":1236906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85599999999999998,"15min":0.81999999999999995},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.28399999999999997},"max":{"1min":1.2969999999999999,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.94999999999999996,"15min":0.90900000000000003},"min":{"1min":0.55200000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":0.94999999999999996}]},{"osd":2,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.83599999999999997,"15min":0.81799999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.37},"max":{"1min":1.3069999999999999,"5min":1.823,"15min":1.861},"last":0.998},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.95199999999999996,"15min":0.93000000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.6950000000000001,"5min":1.7490000000000001,"15min":1.925},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.069,"15min":1.016},"min":{"1min":0.53400000000000003,"5min":0.436,"15min":0.40699999999999997},"max":{"1min":1.4470000000000001,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93799999999999994,"15min":0.93000000000000005},"min":{"1min":0.623,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.444,"5min":1.9330000000000001,"15min":2.3879999999999999},"last":0.84599999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3460000000000001,"15min":1.2929999999999999},"min":{"1min":0.97699999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.8200000000000001,"5min":1.984,"15min":2.7959999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3120000000000001,"15min":1.2729999999999999},"min":{"1min":0.79200000000000004,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.8300000000000001},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.278,"15min":1.2629999999999999},"min":{"1min":0.90800000000000003,"5min":0.748,"15min":0.69599999999999995},"max":{"1min":1.6359999999999999,"5min":2.1150000000000002,"15min":3.7639999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.272,"15min":1.2310000000000001},"min":{"1min":0.72699999999999998,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.7210000000000001,"5min":2.1499999999999999,"15min":3.286},"last":1.2050000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2789999999999999,"15min":1.29},"min":{"1min":0.745,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.986,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3160000000000001,"15min":1.288},"min":{"1min":0.80500000000000005,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.419}]},{"osd":7,"last update":"Fri Nov 24 08:22:17 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3460000000000001,"15min":1.327},"min":{"1min":0.83699999999999997,"5min":0.82299999999999995,"15min":0.59099999999999997},"max":{"1min":2.137,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.351},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":1.901,"5min":1.9930000000000001,"15min":3.653},"last":1.288}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42812,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329028,"statfs":{"total":96636764160,"available":96592924672,"internally_reserved":0,"allocated":3387392,"data_stored":1828886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":1.054,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.82599999999999996,"15min":0.82699999999999996},"min":{"1min":0.48299999999999998,"5min":0.36499999999999999,"15min":0.35899999999999999},"max":{"1min":1.607,"5min":1.607,"15min":2.3620000000000001},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.008,"15min":1.0109999999999999},"min":{"1min":0.54200000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4770000000000001,"5min":1.893,"15min":1.893},"last":1.236},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86299999999999999,"15min":0.86799999999999999},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.444,"5min":1.671,"15min":2.4990000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93799999999999994,"15min":0.96799999999999997},"min":{"1min":0.55900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5209999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85999999999999999,"15min":0.88100000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":1.361,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":1.0640000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2210000000000001,"15min":1.214},"min":{"1min":0.93400000000000005,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.8380000000000001,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.665},{"interface":"front","average":{"1min":1.071,"5min":1.083,"15min":1.107},"min":{"1min":0.75,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.496,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":1.036}]},{"osd":5,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1399999999999999,"15min":1.1499999999999999},"min":{"1min":0.71199999999999997,"5min":0.70299999999999996,"15min":0.69699999999999995},"max":{"1min":1.6100000000000001,"5min":1.6910000000000001,"15min":2.6850000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.149,"5min":1.1419999999999999,"15min":1.159},"min":{"1min":0.753,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.782,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.157}]},{"osd":6,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.248,"15min":1.254},"min":{"1min":0.76200000000000001,"5min":0.73999999999999999,"15min":0.66800000000000004},"max":{"1min":1.6990000000000001,"5min":1.8560000000000001,"15min":2.052},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.742,"5min":1.8440000000000001,"15min":2.823},"last":1.1919999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:22:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2989999999999999,"15min":1.3320000000000001},"min":{"1min":0.91500000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.921,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2769999999999999,"15min":1.298},"min":{"1min":0.86399999999999999,"5min":0.79600000000000004,"15min":0.747},"max":{"1min":1.837,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.96899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:22:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:22:56.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:22:56.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.960+0000 7f6306989640 1 Processor -- start 2023-11-24T08:22:56.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.960+0000 7f6306989640 1 -- start start 2023-11-24T08:22:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.961+0000 7f6306989640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f630015b440 0x7f63000cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.961+0000 7f6306989640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63000c59b0 0x7f630006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f6306989640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000c5dc0 con 0x7f63000cbf20 2023-11-24T08:22:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f6306989640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63000cc3d0 con 0x7f630015b440 2023-11-24T08:22:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f6306989640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630015b820 con 0x7f63000c59b0 2023-11-24T08:22:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f630015b440 0x7f63000cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f630015b440 0x7f63000cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46012/0 (socket says 172.21.15.107:46012) 2023-11-24T08:22:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 -- 172.21.15.107:0/2752037996 learned_addr learned my addr 172.21.15.107:0/2752037996 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62ff7fe640 1 --1- 172.21.15.107:0/2752037996 >> v1:172.21.15.107:6789/0 conn(0x7f63000cbf20 0x7f63000c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52570/0 (socket says 172.21.15.107:52570) 2023-11-24T08:22:56.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 -- 172.21.15.107:0/2752037996 >> v2:172.21.15.120:3300/0 conn(0x7f63000c59b0 msgr2=0x7f630006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:56.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 --2- 172.21.15.107:0/2752037996 >> v2:172.21.15.120:3300/0 conn(0x7f63000c59b0 0x7f630006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 -- 172.21.15.107:0/2752037996 >> v1:172.21.15.107:6789/0 conn(0x7f63000cbf20 legacy=0x7f63000c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62fffff640 1 -- 172.21.15.107:0/2752037996 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630015eea0 con 0x7f630015b440 2023-11-24T08:22:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.962+0000 7f62feffd640 1 -- 172.21.15.107:0/2752037996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158856818 0 0) 0x7f63000c5dc0 con 0x7f63000cbf20 2023-11-24T08:22:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f62fffff640 1 --2- 172.21.15.107:0/2752037996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000cb9e0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f63000cb910 tx=0x7f62f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3344cc46066b961f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f62feffd640 1 -- 172.21.15.107:0/2752037996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0031870 con 0x7f630015b440 2023-11-24T08:22:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f62feffd640 1 -- 172.21.15.107:0/2752037996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f00319d0 con 0x7f630015b440 2023-11-24T08:22:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f62feffd640 1 -- 172.21.15.107:0/2752037996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0031d60 con 0x7f630015b440 2023-11-24T08:22:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f6306989640 1 -- 172.21.15.107:0/2752037996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 msgr2=0x7f63000cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.964+0000 7f6306989640 1 --2- 172.21.15.107:0/2752037996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000cb9e0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f63000cb910 tx=0x7f62f002b910 comp rx=0 tx=0).stop 2023-11-24T08:22:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 -- 172.21.15.107:0/2752037996 shutdown_connections 2023-11-24T08:22:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 --2- 172.21.15.107:0/2752037996 >> v2:172.21.15.120:3300/0 conn(0x7f63000c59b0 0x7f630006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 --2- 172.21.15.107:0/2752037996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000cb9e0 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 -- 172.21.15.107:0/2752037996 >> 172.21.15.107:0/2752037996 conn(0x7f63000b0c50 msgr2=0x7f63000b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 -- 172.21.15.107:0/2752037996 shutdown_connections 2023-11-24T08:22:56.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.965+0000 7f6306989640 1 -- 172.21.15.107:0/2752037996 wait complete. 2023-11-24T08:22:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.966+0000 7f6306989640 1 Processor -- start 2023-11-24T08:22:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 -- start start 2023-11-24T08:22:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63000cbf20 0x7f6300156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000c7cb0 con 0x7f630015b440 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000c8fd0 con 0x7f63000c59b0 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.967+0000 7f6306989640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63001f6a40 con 0x7f63000cbf20 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.968+0000 7f62ff7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63000cbf20 0x7f6300156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.968+0000 7f62fffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.968+0000 7f6304f26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.968+0000 7f62fffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:47036/0 (socket says 172.21.15.107:47036) 2023-11-24T08:22:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.968+0000 7f62fffff640 1 -- 172.21.15.107:0/3517862283 learned_addr learned my addr 172.21.15.107:0/3517862283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.969+0000 7f62fffff640 1 -- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63000cbf20 msgr2=0x7f6300156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.969+0000 7f62fffff640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63000cbf20 0x7f6300156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.969+0000 7f62fffff640 1 -- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 msgr2=0x7f63000c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.969+0000 7f62fffff640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.969+0000 7f62fffff640 1 -- 172.21.15.107:0/3517862283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00094d0 con 0x7f63000c59b0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f62fffff640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f62f002b950 tx=0x7f62f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f003ad10 con 0x7f63000c59b0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f003ae70 con 0x7f63000c59b0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0044610 con 0x7f63000c59b0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630009a450 con 0x7f63000c59b0 2023-11-24T08:22:56.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.970+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63000c7410 con 0x7f63000c59b0 2023-11-24T08:22:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.971+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630004e8b0 con 0x7f63000c59b0 2023-11-24T08:22:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.973+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f62f0044770 con 0x7f63000c59b0 2023-11-24T08:22:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.974+0000 7f62fcff9640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 0x7f62ec06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.974+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62f002d040 con 0x7f63000c59b0 2023-11-24T08:22:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.974+0000 7f62ff7fe640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 0x7f62ec06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.975+0000 7f62ff7fe640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 0x7f62ec06f550 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f63000b0b90 tx=0x7f62f4007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:56.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:56.988+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62f0078260 con 0x7f63000c59b0 2023-11-24T08:22:57.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.245+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f63000ae170 con 0x7f63000c59b0 2023-11-24T08:22:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.256+0000 7f62fcff9640 1 -- 172.21.15.107:0/3517862283 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+147 (secure 0 0 0) 0x7f62f0052030 con 0x7f63000c59b0 2023-11-24T08:22:57.258 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:22:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:22:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:22:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:22:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:22:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:22:57.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 msgr2=0x7f62ec06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 0x7f62ec06f550 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f63000b0b90 tx=0x7f62f4007d50 comp rx=0 tx=0).stop 2023-11-24T08:22:57.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 msgr2=0x7f63000c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f62f002b950 tx=0x7f62f00046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 shutdown_connections 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f62ec06d090 0x7f62ec06f550 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f630015b440 0x7f63000c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63000cbf20 0x7f6300156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 --2- 172.21.15.107:0/3517862283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63000c59b0 0x7f63000c6a90 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 >> 172.21.15.107:0/3517862283 conn(0x7f63000b0c50 msgr2=0x7f630006c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 shutdown_connections 2023-11-24T08:22:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.259+0000 7f6306989640 1 -- 172.21.15.107:0/3517862283 wait complete. 2023-11-24T08:22:57.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:22:57.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.472+0000 7f5cd17ec640 1 Processor -- start 2023-11-24T08:22:57.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.472+0000 7f5cd17ec640 1 -- start start 2023-11-24T08:22:57.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5cd17ec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 0x7f5ccc0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5cd17ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ccc0b3890 0x7f5ccc167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5cd17ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc06d180 con 0x7f5ccc1461d0 2023-11-24T08:22:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5cd17ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b3d20 con 0x7f5ccc0b3890 2023-11-24T08:22:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5cd17ec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc06ba70 con 0x7f5ccc15eaf0 2023-11-24T08:22:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.474+0000 7f5ccb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ccc0b3890 0x7f5ccc167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5ccc1461d0 0x7f5ccc0b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52574/0 (socket says 172.21.15.107:52574) 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ccc0b3890 0x7f5ccc167110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46030/0 (socket says 172.21.15.107:46030) 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccaffd640 1 -- 172.21.15.107:0/3197810086 learned_addr learned my addr 172.21.15.107:0/3197810086 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccb7fe640 1 -- 172.21.15.107:0/3197810086 >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 msgr2=0x7f5ccc0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5cca7fc640 1 --2- 172.21.15.107:0/3197810086 >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 0x7f5ccc0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccb7fe640 1 --2- 172.21.15.107:0/3197810086 >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 0x7f5ccc0b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccb7fe640 1 -- 172.21.15.107:0/3197810086 >> v1:172.21.15.107:6789/0 conn(0x7f5ccc1461d0 legacy=0x7f5ccc0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5ccb7fe640 1 -- 172.21.15.107:0/3197810086 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc06bd50 con 0x7f5ccc0b3890 2023-11-24T08:22:57.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5cc9ffb640 1 -- 172.21.15.107:0/3197810086 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814041175 0 0) 0x7f5ccc06d180 con 0x7f5ccc1461d0 2023-11-24T08:22:57.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.475+0000 7f5cca7fc640 1 --2- 172.21.15.107:0/3197810086 >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 0x7f5ccc0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.476+0000 7f5ccb7fe640 1 --2- 172.21.15.107:0/3197810086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc167110 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0b6320 tx=0x7f5cc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35150cca1da1ca23 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cc9ffb640 1 -- 172.21.15.107:0/3197810086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc402cc80 con 0x7f5ccc0b3890 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cc9ffb640 1 -- 172.21.15.107:0/3197810086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc402cde0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cc9ffb640 1 -- 172.21.15.107:0/3197810086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4034660 con 0x7f5ccc0b3890 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 -- 172.21.15.107:0/3197810086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 msgr2=0x7f5ccc167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/3197810086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc167110 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0b6320 tx=0x7f5cc402bed0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 -- 172.21.15.107:0/3197810086 shutdown_connections 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/3197810086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc167110 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/3197810086 >> v2:172.21.15.120:3300/0 conn(0x7f5ccc15eaf0 0x7f5ccc0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 -- 172.21.15.107:0/3197810086 >> 172.21.15.107:0/3197810086 conn(0x7f5ccc061340 msgr2=0x7f5ccc061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 -- 172.21.15.107:0/3197810086 shutdown_connections 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.477+0000 7f5cd17ec640 1 -- 172.21.15.107:0/3197810086 wait complete. 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.479+0000 7f5cd17ec640 1 Processor -- start 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.479+0000 7f5cd17ec640 1 -- start start 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ccc1461d0 0x7f5ccc0ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ccc15eaf0 0x7f5ccc0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0cec50 con 0x7f5ccc0b3890 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0cca80 con 0x7f5ccc15eaf0 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5cd17ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0cccb0 con 0x7f5ccc1461d0 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5ccaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5ccaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46032/0 (socket says 172.21.15.107:46032) 2023-11-24T08:22:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.480+0000 7f5ccaffd640 1 -- 172.21.15.107:0/276152753 learned_addr learned my addr 172.21.15.107:0/276152753 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 -- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ccc1461d0 msgr2=0x7f5ccc0ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ccc1461d0 0x7f5ccc0ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 -- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ccc15eaf0 msgr2=0x7f5ccc0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ccc15eaf0 0x7f5ccc0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 -- 172.21.15.107:0/276152753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4009a70 con 0x7f5ccc0b3890 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5ccaffd640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c500 tx=0x7f5cbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc007560 con 0x7f5ccc0b3890 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc0047e0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc004b90 con 0x7f5ccc0b3890 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.481+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc0ccf90 con 0x7f5ccc0b3890 2023-11-24T08:22:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.482+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc144810 con 0x7f5ccc0b3890 2023-11-24T08:22:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.483+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc0cd4b0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.484+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f5cbc004cf0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.484+0000 7f5cb37fe640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 0x7f5cb806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.485+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cbc087900 con 0x7f5ccc0b3890 2023-11-24T08:22:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.485+0000 7f5cca7fc640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 0x7f5cb806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:57.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.486+0000 7f5cca7fc640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 0x7f5cb806f550 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:57.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.489+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ccc0cd4b0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.710 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-11-24T08:22:57.708+0000 7f7be95d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:22:57.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.713+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ccc0cd4b0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.715+0000 7f5cb37fe640 1 -- 172.21.15.107:0/276152753 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f5ccc0cd4b0 con 0x7f5ccc0b3890 2023-11-24T08:22:57.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:22:57.717 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:22:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 138 2023-11-24T08:22:57.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 msgr2=0x7f5cb806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 0x7f5cb806f550 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0008040 comp rx=0 tx=0).stop 2023-11-24T08:22:57.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 msgr2=0x7f5ccc0d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:57.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c500 tx=0x7f5cbc00c9c0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 shutdown_connections 2023-11-24T08:22:57.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cb806d090 0x7f5cb806f550 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ccc15eaf0 0x7f5ccc0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ccc1461d0 0x7f5ccc0ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 --2- 172.21.15.107:0/276152753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ccc0b3890 0x7f5ccc0d1000 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:57.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.717+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 >> 172.21.15.107:0/276152753 conn(0x7f5ccc061340 msgr2=0x7f5ccc0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:57.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.718+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 shutdown_connections 2023-11-24T08:22:57.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:57.718+0000 7f5cd17ec640 1 -- 172.21.15.107:0/276152753 wait complete. 2023-11-24T08:22:58.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:22:51.212515+0000', 'modified': '2023-11-24T08:22:54.271878+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12213}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12213': {'gid': 12213, 'name': 'a', 'rank': 0, 'incarnation': 134, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/1864313322', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1864313322}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1864313322}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:22:58.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:22:58.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-24T08:22:58.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.360+0000 7fbe44d21640 1 Processor -- start 2023-11-24T08:22:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.361+0000 7fbe44d21640 1 -- start start 2023-11-24T08:22:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe44d21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe4015d490 0x7fbe400cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe44d21640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe400b63d0 0x7fbe400d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe44d21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400cea30 con 0x7fbe400ce270 2023-11-24T08:22:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe44d21640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe400d0450 con 0x7fbe400b63d0 2023-11-24T08:22:58.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe44d21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400d0bf0 con 0x7fbe4015d490 2023-11-24T08:22:58.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe3effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe400b63d0 0x7fbe400d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe3ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbe400ce270 0x7fbe40160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33756/0 (socket says 172.21.15.107:33756) 2023-11-24T08:22:58.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.362+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2388600499 learned_addr learned my addr 172.21.15.107:0/2388600499 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:58.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe4015d490 0x7fbe400cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3effd640 1 -- 172.21.15.107:0/2388600499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe4015d490 msgr2=0x7fbe400cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe4015d490 0x7fbe400cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3effd640 1 -- 172.21.15.107:0/2388600499 >> v1:172.21.15.107:6789/0 conn(0x7fbe400ce270 legacy=0x7fbe40160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3effd640 1 -- 172.21.15.107:0/2388600499 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400d2260 con 0x7fbe400b63d0 2023-11-24T08:22:58.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.363+0000 7fbe3e7fc640 1 -- 172.21.15.107:0/2388600499 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260879479 0 0) 0x7fbe400cea30 con 0x7fbe400ce270 2023-11-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.364+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe400b63d0 0x7fbe400d3230 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fbe340097a0 tx=0x7fbe3402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7b08608edb7794 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.364+0000 7fbe3e7fc640 1 -- 172.21.15.107:0/2388600499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3402d040 con 0x7fbe400b63d0 2023-11-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.364+0000 7fbe3e7fc640 1 -- 172.21.15.107:0/2388600499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe34031810 con 0x7fbe400b63d0 2023-11-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.364+0000 7fbe3e7fc640 1 -- 172.21.15.107:0/2388600499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34031ba0 con 0x7fbe400b63d0 2023-11-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 -- 172.21.15.107:0/2388600499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe400b63d0 msgr2=0x7fbe400d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe400b63d0 0x7fbe400d3230 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fbe340097a0 tx=0x7fbe3402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 -- 172.21.15.107:0/2388600499 shutdown_connections 2023-11-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe400b63d0 0x7fbe400d3230 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2388600499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe4015d490 0x7fbe400cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 -- 172.21.15.107:0/2388600499 >> 172.21.15.107:0/2388600499 conn(0x7fbe40061340 msgr2=0x7fbe40061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 -- 172.21.15.107:0/2388600499 shutdown_connections 2023-11-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.365+0000 7fbe44d21640 1 -- 172.21.15.107:0/2388600499 wait complete. 2023-11-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.367+0000 7fbe44d21640 1 Processor -- start 2023-11-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.367+0000 7fbe44d21640 1 -- start start 2023-11-24T08:22:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe400ce270 0x7fbe4015f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe400ccfa0 0x7fbe40152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400c8470 con 0x7fbe4015d490 2023-11-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4015f570 con 0x7fbe400ccfa0 2023-11-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe44d21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4015e280 con 0x7fbe400ce270 2023-11-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46970/0 (socket says 172.21.15.107:46970) 2023-11-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.368+0000 7fbe3effd640 1 -- 172.21.15.107:0/2380443135 learned_addr learned my addr 172.21.15.107:0/2380443135 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe400ce270 0x7fbe4015f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 -- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe400ce270 msgr2=0x7fbe4015f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe400ce270 0x7fbe4015f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 -- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe400ccfa0 msgr2=0x7fbe40152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe400ccfa0 0x7fbe40152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe400ccfa0 0x7fbe40152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 -- 172.21.15.107:0/2380443135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34009400 con 0x7fbe4015d490 2023-11-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fbe3402bec0 tx=0x7fbe340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe27fff640 1 -- 172.21.15.107:0/2380443135 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3403ad60 con 0x7fbe4015d490 2023-11-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe27fff640 1 -- 172.21.15.107:0/2380443135 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe340423f0 con 0x7fbe4015d490 2023-11-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe400ca370 con 0x7fbe4015d490 2023-11-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.369+0000 7fbe27fff640 1 -- 172.21.15.107:0/2380443135 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe340427a0 con 0x7fbe4015d490 2023-11-24T08:22:58.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.370+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4015e7b0 con 0x7fbe4015d490 2023-11-24T08:22:58.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.372+0000 7fbe27fff640 1 -- 172.21.15.107:0/2380443135 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fbe3404b3b0 con 0x7fbe4015d490 2023-11-24T08:22:58.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.373+0000 7fbe27fff640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 0x7fbe2c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.373+0000 7fbe27fff640 1 -- 172.21.15.107:0/2380443135 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe340aa970 con 0x7fbe4015d490 2023-11-24T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.373+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 0x7fbe2c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.373+0000 7fbe48bf2740 1 Processor -- start 2023-11-24T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.373+0000 7fbe48bf2740 1 -- start start 2023-11-24T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 0x7fbe2c06f550 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbe30008e50 tx=0x7fbe300083a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17bd8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd2d70 0x55dd17bd7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17b36880 con 0x55dd17bd77b0 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17b45ae0 con 0x55dd17bd8070 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.374+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd17b431d0 con 0x55dd17bd2d70 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46972/0 (socket says 172.21.15.107:46972) 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2453710439 learned_addr learned my addr 172.21.15.107:0/2453710439 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17bd8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd2d70 msgr2=0x55dd17bd7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd2d70 0x55dd17bd7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd2d70 0x55dd17bd7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 msgr2=0x55dd17bd8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17bd8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2453710439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd17b06860 con 0x55dd17bd77b0 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17bd8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.375+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fbe380016b0 tx=0x7fbe38001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fb1467fe088e25 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe257fa640 1 -- 172.21.15.107:0/2453710439 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380074e0 con 0x55dd17bd77b0 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe257fa640 1 -- 172.21.15.107:0/2453710439 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe38007640 con 0x55dd17bd77b0 2023-11-24T08:22:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe257fa640 1 -- 172.21.15.107:0/2453710439 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38004a20 con 0x55dd17bd77b0 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2453710439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 msgr2=0x55dd17c1b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fbe380016b0 tx=0x7fbe38001b70 comp rx=0 tx=0).stop 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2453710439 shutdown_connections 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c1b730 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd2d70 0x55dd17bd7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2453710439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17bd8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.376+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2453710439 >> 172.21.15.107:0/2453710439 conn(0x55dd17b99c70 msgr2=0x55dd17c1db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.377+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2453710439 shutdown_connections 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.377+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2453710439 wait complete. 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.377+0000 7fbe48bf2740 1 -- start start 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.377+0000 7fbe48bf2740 1 Processor -- start 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 0x55dd17bfe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd8070 0x55dd17c00f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17b3eac0 con 0x55dd17bd77b0 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17adb2f0 con 0x55dd17bd2d70 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.378+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd17af8c70 con 0x55dd17bd8070 2023-11-24T08:22:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe48bf2740 10 client.0 _ll_register_callbacks cb 0x55dd17afd3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46980/0 (socket says 172.21.15.107:46980) 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 -- 172.21.15.107:0/644845304 learned_addr learned my addr 172.21.15.107:0/644845304 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 -- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd8070 msgr2=0x55dd17c00f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd8070 0x55dd17c00f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 0x55dd17bfe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 -- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 msgr2=0x55dd17bfe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 0x55dd17bfe4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 -- 172.21.15.107:0/644845304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe380012b0 con 0x55dd17bd77b0 2023-11-24T08:22:58.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 0x55dd17bfe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe3effd640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006f60 tx=0x7fbe3407b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c926f5c9151db61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.379+0000 7fbe1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34083440 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe340810b0 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd17b37aa0 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe34081290 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 10 client.12348 fetch_fsmap learned FSMap version 138 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55dd17b3d140 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dd17b37aa0 con 0x55dd17bd77b0 2023-11-24T08:22:58.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 138) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbe34076f30 con 0x55dd17bd77b0 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 10 client.12348 fetch_fsmap finished waiting for FSMap version 138 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 10 client.12348 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 20 client.12348 populate_metadata read hostname 'smithi107' 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c23550 0x55dd17c23930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 4 client.12348 mds_command: new command op to 12213 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.380+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 --> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd17b377c0 con 0x55dd17c23550 2023-11-24T08:22:58.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.381+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c23550 0x55dd17c23930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.381+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c23550 0x55dd17c23930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a25c74543e64231 server_cookie=e14a0c5083b879ff in_seq=0 out_seq=0 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.381+0000 7fbe1e7fc640 10 client.12348 ms_handle_connect on v2:172.21.15.107:6836/1864313322 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/644845304 <== mds.0 v2:172.21.15.107:6836/1864313322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dd17b377c0 con 0x55dd17c23550 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe1e7fc640 10 client.12348 handle_command_reply: tid=0 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 client.12348 shutdown 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 msgr2=0x55dd17c25530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006f60 tx=0x7fbe3407b980 comp rx=0 tx=0).stop 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 shutdown_connections 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c23550 0x55dd17c23930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd8070 0x55dd17c00f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd77b0 0x55dd17c25530 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/644845304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd2d70 0x55dd17bfe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe1e7fc640 0 client.12348 ms_handle_reset on v2:172.21.15.107:6836/1864313322 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 >> 172.21.15.107:0/644845304 conn(0x55dd17b99c70 msgr2=0x55dd17c1ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 shutdown_connections 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.382+0000 7fbe48bf2740 1 -- 172.21.15.107:0/644845304 wait complete. 2023-11-24T08:22:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.383+0000 7fbe48bf2740 20 client.12348 trim_cache size 0 max 16384 2023-11-24T08:22:58.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.399+0000 7fbe48bf2740 1 Processor -- start 2023-11-24T08:22:58.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.399+0000 7fbe48bf2740 1 -- start start 2023-11-24T08:22:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c1dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd178a1f20 con 0x55dd17bd2d70 2023-11-24T08:22:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17bd35e0 con 0x55dd17bd8070 2023-11-24T08:22:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.400+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd17c01350 con 0x55dd17bd77b0 2023-11-24T08:22:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46996/0 (socket says 172.21.15.107:46996) 2023-11-24T08:22:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 -- 172.21.15.107:0/1920765754 learned_addr learned my addr 172.21.15.107:0/1920765754 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 -- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 msgr2=0x55dd17bce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 -- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 msgr2=0x55dd17c1dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c1dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 -- 172.21.15.107:0/1920765754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd17adab80 con 0x55dd17bd2d70 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.401+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x55dd17c220c0 tx=0x7fbe30013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512a171f0000e5d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe25ffb640 1 -- 172.21.15.107:0/1920765754 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30023070 con 0x55dd17bd2d70 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe25ffb640 1 -- 172.21.15.107:0/1920765754 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30010c30 con 0x55dd17bd2d70 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe25ffb640 1 -- 172.21.15.107:0/1920765754 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30010dc0 con 0x55dd17bd2d70 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 -- 172.21.15.107:0/1920765754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 msgr2=0x55dd17c25510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x55dd17c220c0 tx=0x7fbe30013000 comp rx=0 tx=0).stop 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 -- 172.21.15.107:0/1920765754 shutdown_connections 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c1dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/1920765754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17bd2d70 0x55dd17c25510 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 -- 172.21.15.107:0/1920765754 >> 172.21.15.107:0/1920765754 conn(0x55dd17b99c70 msgr2=0x55dd17c027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 -- 172.21.15.107:0/1920765754 shutdown_connections 2023-11-24T08:22:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.402+0000 7fbe48bf2740 1 -- 172.21.15.107:0/1920765754 wait complete. 2023-11-24T08:22:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.403+0000 7fbe48bf2740 1 -- start start 2023-11-24T08:22:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.403+0000 7fbe48bf2740 1 Processor -- start 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bd2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c8a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17c90020 con 0x55dd17c3c8b0 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd17c3e440 con 0x55dd17bd8070 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd17c90220 con 0x55dd17bd77b0 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe48bf2740 10 client.0 _ll_register_callbacks cb 0x7fbe34077480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47002/0 (socket says 172.21.15.107:47002) 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.404+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2363729648 learned_addr learned my addr 172.21.15.107:0/2363729648 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3effd640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c8a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bd2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 msgr2=0x55dd17bd2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bd2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 msgr2=0x55dd17c8a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c8a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 -- 172.21.15.107:0/2363729648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe30008040 con 0x55dd17c3c8b0 2023-11-24T08:22:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe3ffff640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fbe38001650 tx=0x7fbe38004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d251d710acb9047e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38007a40 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe38004d10 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.405+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe38007640 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe3800d730 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 10 client.12354 fetch_fsmap learned FSMap version 138 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55dd17b00170 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe38007640 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 138) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbe3800d890 con 0x55dd17c3c8b0 2023-11-24T08:22:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 10 client.12354 fetch_fsmap finished waiting for FSMap version 138 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 10 client.12354 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 20 client.12354 populate_metadata read hostname 'smithi107' 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c21bd0 0x55dd17bfc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 4 client.12354 mds_command: new command op to 12213 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 --> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dd1783f6c0 con 0x55dd17c21bd0 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.406+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c21bd0 0x55dd17bfc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe3f7fe640 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c21bd0 0x55dd17bfc590 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba22d0139e8d2de7 server_cookie=f3c30e3746ac4eb8 in_seq=0 out_seq=0 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe1e7fc640 10 client.12354 ms_handle_connect on v2:172.21.15.107:6836/1864313322 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe1e7fc640 1 -- 172.21.15.107:0/2363729648 <== mds.0 v2:172.21.15.107:6836/1864313322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55dd1783f6c0 con 0x55dd17c21bd0 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe1e7fc640 10 client.12354 handle_command_reply: tid=0 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 client.12354 shutdown 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 msgr2=0x55dd17c1dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fbe38001650 tx=0x7fbe38004a10 comp rx=0 tx=0).stop 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 shutdown_connections 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:6836/1864313322,v1:172.21.15.107:6837/1864313322] conn(0x55dd17c21bd0 0x55dd17bfc590 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dd17c3c8b0 0x55dd17c1dd50 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe1e7fc640 0 client.12354 ms_handle_reset on v2:172.21.15.107:6836/1864313322 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd17bd8070 0x55dd17c8a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 --2- 172.21.15.107:0/2363729648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dd17bd77b0 0x55dd17bd2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.407+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 >> 172.21.15.107:0/2363729648 conn(0x55dd17b99c70 msgr2=0x55dd17bd8450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 shutdown_connections 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe48bf2740 1 -- 172.21.15.107:0/2363729648 wait complete. 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe48bf2740 20 client.12354 trim_cache size 0 max 16384 2023-11-24T08:22:58.416 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 12213, 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:22:58.417 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6531482740000003, 2023-11-24T08:22:58.418 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 138, 2023-11-24T08:22:58.418 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 220, 2023-11-24T08:22:58.418 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 220, 2023-11-24T08:22:58.418 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.057112844000001 2023-11-24T08:22:58.418 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 msgr2=0x7fbe2c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 0x7fbe2c06f550 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbe30008e50 tx=0x7fbe300083a0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 msgr2=0x7fbe4015fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fbe3402bec0 tx=0x7fbe340046c0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 shutdown_connections 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe2c06d090 0x7fbe2c06f550 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe400ccfa0 0x7fbe40152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe4015d490 0x7fbe4015fed0 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 --2- 172.21.15.107:0/2380443135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe400ce270 0x7fbe4015f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.408+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 >> 172.21.15.107:0/2380443135 conn(0x7fbe40061340 msgr2=0x7fbe40061720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.409+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 shutdown_connections 2023-11-24T08:22:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.409+0000 7fbe44d21640 1 -- 172.21.15.107:0/2380443135 wait complete. 2023-11-24T08:22:58.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:22:58.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.635+0000 7fb895a6b640 1 Processor -- start 2023-11-24T08:22:58.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.635+0000 7fb895a6b640 1 -- start start 2023-11-24T08:22:58.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb895a6b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb89015c7f0 0x7fb89006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb895a6b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8900cece0 con 0x7fb8900ce6c0 2023-11-24T08:22:58.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb895a6b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89006d040 con 0x7fb8900cf450 2023-11-24T08:22:58.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb895a6b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8900ceaf0 con 0x7fb89015c7f0 2023-11-24T08:22:58.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb88ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8900ce6c0 0x7fb8900714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33770/0 (socket says 172.21.15.107:33770) 2023-11-24T08:22:58.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.637+0000 7fb88ffff640 1 -- 172.21.15.107:0/229108877 learned_addr learned my addr 172.21.15.107:0/229108877 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:22:58.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/229108877 >> v2:172.21.15.120:3300/0 conn(0x7fb89015c7f0 0x7fb89006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343521274 0 0) 0x7fb8900cece0 con 0x7fb8900ce6c0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87c003520 con 0x7fb8900ce6c0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447046210 0 0) 0x7fb87c003520 con 0x7fb8900ce6c0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 >> v2:172.21.15.120:3300/0 conn(0x7fb89015c7f0 msgr2=0x7fb89006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 --2- 172.21.15.107:0/229108877 >> v2:172.21.15.120:3300/0 conn(0x7fb89015c7f0 0x7fb89006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 >> v1:172.21.15.120:6789/0 conn(0x7fb8900cf450 legacy=0x7fb8900cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89006d320 con 0x7fb8900ce6c0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fb8880019f0 con 0x7fb8900ce6c0 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 >> v1:172.21.15.107:6789/0 conn(0x7fb8900ce6c0 legacy=0x7fb8900714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 0x7fb87c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c0047c0 0x7fb87c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb87c005180 0x7fb87c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00bff0 con 0x7fb87c003970 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00c160 con 0x7fb87c005180 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.638+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00c320 con 0x7fb87c0047c0 2023-11-24T08:22:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 0x7fb87c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 -- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c0047c0 msgr2=0x7fb87c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88ffff640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb87c005180 0x7fb87c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c0047c0 0x7fb87c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 -- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb87c005180 msgr2=0x7fb87c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb87c005180 0x7fb87c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88ffff640 1 -- 172.21.15.107:0/229108877 reap_dead start 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.639+0000 7fb88f7fe640 1 -- 172.21.15.107:0/229108877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c003520 con 0x7fb87c003970 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.640+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 0x7fb87c003d90 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fb880009ac0 tx=0x7fb87c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba90f5001b6c856 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.640+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88002c6b0 con 0x7fb87c003970 2023-11-24T08:22:58.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.640+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88002c810 con 0x7fb87c003970 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.640+0000 7fb88effd640 1 -- 172.21.15.107:0/229108877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88002cba0 con 0x7fb87c003970 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 -- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 msgr2=0x7fb87c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 0x7fb87c003d90 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fb880009ac0 tx=0x7fb87c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 -- 172.21.15.107:0/229108877 shutdown_connections 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 --2- 172.21.15.107:0/229108877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb87c003970 0x7fb87c003d90 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 -- 172.21.15.107:0/229108877 >> 172.21.15.107:0/229108877 conn(0x7fb89006a920 msgr2=0x7fb89006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 -- 172.21.15.107:0/229108877 shutdown_connections 2023-11-24T08:22:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.641+0000 7fb895a6b640 1 -- 172.21.15.107:0/229108877 wait complete. 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.643+0000 7fb895a6b640 1 Processor -- start 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.643+0000 7fb895a6b640 1 -- start start 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c003970 0x7fb89014b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb890158040 0x7fb8901584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8901567a0 con 0x7fb890155de0 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8900ac160 con 0x7fb890158040 2023-11-24T08:22:58.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb895a6b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8900ac440 con 0x7fb87c003970 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb887fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb887fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47022/0 (socket says 172.21.15.107:47022) 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.644+0000 7fb887fff640 1 -- 172.21.15.107:0/145431711 learned_addr learned my addr 172.21.15.107:0/145431711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c003970 0x7fb89014b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 -- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c003970 msgr2=0x7fb89014b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c003970 0x7fb89014b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 -- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb890158040 msgr2=0x7fb8901584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb890158040 0x7fb8901584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 -- 172.21.15.107:0/145431711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8900ac720 con 0x7fb890155de0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb887fff640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c500 tx=0x7fb87800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8780052d0 con 0x7fb890155de0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878005430 con 0x7fb890155de0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878012870 con 0x7fb890155de0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.645+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8900aca00 con 0x7fb890155de0 2023-11-24T08:22:58.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.646+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89014b8a0 con 0x7fb890155de0 2023-11-24T08:22:58.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.647+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb890067470 con 0x7fb890155de0 2023-11-24T08:22:58.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.647+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fb8780129d0 con 0x7fb890155de0 2023-11-24T08:22:58.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.648+0000 7fb88cff9640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 0x7fb87006f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:22:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.649+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb878088370 con 0x7fb890155de0 2023-11-24T08:22:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.649+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 0x7fb87006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:22:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.649+0000 7fb88f7fe640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 0x7fb87006f580 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fb880036000 tx=0x7fb880030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:22:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.653+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8780589b0 con 0x7fb890155de0 2023-11-24T08:22:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.891+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb8900cd310 con 0x7fb890155de0 2023-11-24T08:22:58.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.892+0000 7fb88cff9640 1 -- 172.21.15.107:0/145431711 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v138) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8780011b0 con 0x7fb890155de0 2023-11-24T08:22:58.894 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.894+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 msgr2=0x7fb87006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.894+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 0x7fb87006f580 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fb880036000 tx=0x7fb880030000 comp rx=0 tx=0).stop 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 msgr2=0x7fb890156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c500 tx=0x7fb87800c9c0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 shutdown_connections 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb87006d0c0 0x7fb87006f580 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb890158040 0x7fb8901584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb890155de0 0x7fb890156260 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 --2- 172.21.15.107:0/145431711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87c003970 0x7fb89014b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:22:58.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 >> 172.21.15.107:0/145431711 conn(0x7fb89006a920 msgr2=0x7fb8900b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:22:58.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 shutdown_connections 2023-11-24T08:22:58.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:22:58.895+0000 7fb895a6b640 1 -- 172.21.15.107:0/145431711 wait complete. 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:22:58.916 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:22:58.917 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:22:58.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:58.917 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:22:58.969 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:22:58.969 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:22:58.969 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:22:58.969 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.969 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:22:58.970 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41291sec preferred_lft 41291sec 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:22:58.971 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:22:58.972 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:22:58.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:58.973 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:22:59.063 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:22:59.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.067 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:22:59.122 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:22:59.122 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:22:59.122 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:22:59.122 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:22:59.122 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:22:59.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:22:59.123 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:22:59.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.307 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:22:59.357 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:22:59.357 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:22:59.357 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:22:59.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:22:59.445 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:22:59.445 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:22:59.498 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:22:59.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:22:59.499 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:22:59.550 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:22:59.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:59.688 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:59.689 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:22:59.720 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:22:59.720 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:22:59.721 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:22:59.721 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:22:59.721 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:22:59.722 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:22:59.722 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:22:59.723 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:22:59.723 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:22:59.723 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:22:59.727 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:22:59.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:22:59.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:22:59.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:22:59.874 INFO:teuthology.orchestra.run.smithi107.stdout:client7900 2023-11-24T08:22:59.877 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:22:59.878 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:22:59.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:23:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.189+0000 7f4a02b68640 1 Processor -- start 2023-11-24T08:23:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.189+0000 7f4a02b68640 1 -- start start 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a02b68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49fc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a02b68640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f49fc0b2a70 0x7f49fc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a02b68640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49fc15ab80 con 0x7f49fc0b2630 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a02b68640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49fc15a990 con 0x7f49fc0b2a70 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a02b68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc15e680 con 0x7f49fc0b2250 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f49fc0b2630 0x7f49fc0ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33774/0 (socket says 172.21.15.107:33774) 2023-11-24T08:23:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fbfff640 1 -- 172.21.15.107:0/306860345 learned_addr learned my addr 172.21.15.107:0/306860345 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a00904640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49fc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> v2:172.21.15.120:3300/0 conn(0x7f49fc0b2a70 0x7f49fc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544407182 0 0) 0x7f49fc15ab80 con 0x7f49fc0b2630 2023-11-24T08:23:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec003520 con 0x7f49fc0b2630 2023-11-24T08:23:00.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845508169 0 0) 0x7f49ec003520 con 0x7f49fc0b2630 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 msgr2=0x7f49fc15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49fc15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 >> v2:172.21.15.120:3300/0 conn(0x7f49fc0b2a70 msgr2=0x7f49fc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 --2- 172.21.15.107:0/306860345 >> v2:172.21.15.120:3300/0 conn(0x7f49fc0b2a70 0x7f49fc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc15e960 con 0x7f49fc0b2630 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f49e8001670 con 0x7f49fc0b2630 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> v2:172.21.15.120:3300/0 conn(0x7f49fc0b2a70 0x7f49fc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 >> v1:172.21.15.107:6789/0 conn(0x7f49fc0b2630 legacy=0x7f49fc0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49ec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec003f40 0x7f49ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49ec004d00 0x7f49ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc15e680 con 0x7f49ec004d00 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec00b9b0 con 0x7f49ec003f40 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.191+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec00bb70 con 0x7f49fc0b2250 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49ec004d00 0x7f49ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f49fbfff640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec003f40 0x7f49ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a00904640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49ec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 -- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 msgr2=0x7f49ec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49fc0b2250 0x7f49ec003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 -- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec003f40 msgr2=0x7f49ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec003f40 0x7f49ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f4a01105640 1 -- 172.21.15.107:0/306860345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec003520 con 0x7f49ec004d00 2023-11-24T08:23:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.192+0000 7f49fbfff640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec003f40 0x7f49ec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.193+0000 7f4a01105640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49ec004d00 0x7f49ec005180 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f49f40073e0 tx=0x7f49ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca62b1f501afb3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.193+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f402cdc0 con 0x7f49ec004d00 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.193+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f4034440 con 0x7f49ec004d00 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.193+0000 7f49fb7fe640 1 -- 172.21.15.107:0/306860345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f40347b0 con 0x7f49ec004d00 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 -- 172.21.15.107:0/306860345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49ec004d00 msgr2=0x7f49ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 --2- 172.21.15.107:0/306860345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49ec004d00 0x7f49ec005180 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f49f40073e0 tx=0x7f49ec0057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a01105640 1 -- 172.21.15.107:0/306860345 reap_dead start 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 -- 172.21.15.107:0/306860345 shutdown_connections 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 -- 172.21.15.107:0/306860345 >> 172.21.15.107:0/306860345 conn(0x7f49fc06a920 msgr2=0x7f49fc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 -- 172.21.15.107:0/306860345 shutdown_connections 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.194+0000 7f4a02b68640 1 -- 172.21.15.107:0/306860345 wait complete. 2023-11-24T08:23:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.196+0000 7f4a02b68640 1 Processor -- start 2023-11-24T08:23:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.196+0000 7f4a02b68640 1 -- start start 2023-11-24T08:23:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49ec004d00 0x7f49fc1f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 0x7f49fc151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc159cc0 con 0x7f49fc1592d0 2023-11-24T08:23:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc1f4430 con 0x7f49fc0cbee0 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a02b68640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc1f7350 con 0x7f49ec004d00 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a01105640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a01105640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47046/0 (socket says 172.21.15.107:47046) 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a01105640 1 -- 172.21.15.107:0/160153847 learned_addr learned my addr 172.21.15.107:0/160153847 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a01105640 1 -- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49ec004d00 msgr2=0x7f49fc1f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.197+0000 7f4a01105640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49ec004d00 0x7f49fc1f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a01105640 1 -- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 msgr2=0x7f49fc151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f49fbfff640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 0x7f49fc151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a01105640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 0x7f49fc151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a01105640 1 -- 172.21.15.107:0/160153847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc0cd7c0 con 0x7f49fc1592d0 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f49fbfff640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 0x7f49fc151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a01105640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f49f4002230 tx=0x7f49f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f403e440 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f403e5e0 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f403e990 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc151c50 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.198+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc0cc3c0 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.200+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc067470 con 0x7f49fc1592d0 2023-11-24T08:23:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.201+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f49f403eaf0 con 0x7f49fc1592d0 2023-11-24T08:23:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.201+0000 7f49f97fa640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 0x7f49e006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.202+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49f40aa9d0 con 0x7f49fc1592d0 2023-11-24T08:23:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.202+0000 7f4a00904640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 0x7f49e006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:00.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.202+0000 7f4a00904640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 0x7f49e006f550 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009950 tx=0x7f49f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:00.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.206+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49f4077630 con 0x7f49fc1592d0 2023-11-24T08:23:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.436+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f49fc0cb3f0 con 0x7f49fc1592d0 2023-11-24T08:23:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.437+0000 7f49f97fa640 1 -- 172.21.15.107:0/160153847 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v138) v1 ==== 53+0+73 (secure 0 0 0) 0x7f49f4039020 con 0x7f49fc1592d0 2023-11-24T08:23:00.439 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:23:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 msgr2=0x7f49e006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 0x7f49e006f550 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009950 tx=0x7f49f0008040 comp rx=0 tx=0).stop 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 msgr2=0x7f49fc159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f49f4002230 tx=0x7f49f4004ab0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 shutdown_connections 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f49e006d090 0x7f49e006f550 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49fc1592d0 0x7f49fc159780 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49fc0cbee0 0x7f49fc151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 --2- 172.21.15.107:0/160153847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49ec004d00 0x7f49fc1f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:00.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.439+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 >> 172.21.15.107:0/160153847 conn(0x7f49fc06a920 msgr2=0x7f49fc0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:00.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.440+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 shutdown_connections 2023-11-24T08:23:00.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:00.440+0000 7f4a02b68640 1 -- 172.21.15.107:0/160153847 wait complete. 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:23:00.459 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:23:00.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.460 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:23:00.519 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:23:00.520 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41288sec preferred_lft 41288sec 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:23:00.521 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:23:00.522 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:23:00.523 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:00.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.524 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:23:00.616 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:23:00.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.621 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:23:00.675 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:23:00.675 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:23:00.675 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:23:00.675 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:23:00.675 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:23:00.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.676 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:23:00.676 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:23:00.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:23:00.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:23:00.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:23:00.677 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:23:00.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.864 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:23:00.914 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:23:00.914 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:23:00.915 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:23:00.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:00.915 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:23:01.008 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:23:01.008 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:23:01.062 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:23:01.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:23:01.063 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:23:01.113 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:23:01.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:01.113 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:01.253 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:01.254 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:23:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:23:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:23:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:23:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:23:01.291 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:23:01.291 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:23:01.291 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:23:01.291 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:23:01.291 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:23:01.295 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:23:01.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:01.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:23:01.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:01.383 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:23:01.440 INFO:teuthology.orchestra.run.smithi120.stdout:client7906 2023-11-24T08:23:01.440 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:01.440 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:01.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:23:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.721+0000 7f5bf9b0f640 1 Processor -- start 2023-11-24T08:23:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.722+0000 7f5bf9b0f640 1 -- start start 2023-11-24T08:23:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.723+0000 7f5bf9b0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5bf40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.723+0000 7f5bf9b0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40cece0 con 0x7f5bf415c7f0 2023-11-24T08:23:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.723+0000 7f5bf9b0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf406d040 con 0x7f5bf40cdfb0 2023-11-24T08:23:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.723+0000 7f5bf9b0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4156010 con 0x7f5bf40cf450 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5bf415c7f0 0x7f5bf406db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33778/0 (socket says 172.21.15.107:33778) 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf37fe640 1 -- 172.21.15.107:0/709448037 learned_addr learned my addr 172.21.15.107:0/709448037 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bebfff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5bf40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289373230 0 0) 0x7f5bf40cece0 con 0x7f5bf415c7f0 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be0003520 con 0x7f5bf415c7f0 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338975920 0 0) 0x7f5be0003520 con 0x7f5bf415c7f0 2023-11-24T08:23:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 msgr2=0x7f5bf40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5bf40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 >> v1:172.21.15.120:6789/0 conn(0x7f5bf40cdfb0 legacy=0x7f5bf40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf406d320 con 0x7f5bf415c7f0 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5be4001670 con 0x7f5bf415c7f0 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.724+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 >> v1:172.21.15.107:6789/0 conn(0x7f5bf415c7f0 legacy=0x7f5bf406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5be00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5be00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5be0004d00 0x7f5be0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4156010 con 0x7f5be0003f20 2023-11-24T08:23:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be000ba60 con 0x7f5be0004d00 2023-11-24T08:23:01.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be000bc20 con 0x7f5bf40cf450 2023-11-24T08:23:01.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5be00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf37fe640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5be00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 -- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 msgr2=0x7f5be00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf40cf450 0x7f5be00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 -- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5be0004d00 msgr2=0x7f5be0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf3fff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5be0004d00 0x7f5be0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5be0004d00 0x7f5be0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bebfff640 1 -- 172.21.15.107:0/709448037 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be0003520 con 0x7f5be0003f20 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.725+0000 7f5bf3fff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5be0004d00 0x7f5be0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bebfff640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5be00047c0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009ac0 tx=0x7f5be00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5ca20f97b57adc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc02c6b0 con 0x7f5be0003f20 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc02c810 con 0x7f5be0003f20 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bf2ffd640 1 -- 172.21.15.107:0/709448037 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc02cba0 con 0x7f5be0003f20 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/709448037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 msgr2=0x7f5be00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.727+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/709448037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5be00047c0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009ac0 tx=0x7f5be00057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.728+0000 7f5bf3fff640 1 -- 172.21.15.107:0/709448037 reap_dead start 2023-11-24T08:23:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.728+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/709448037 shutdown_connections 2023-11-24T08:23:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.728+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/709448037 >> 172.21.15.107:0/709448037 conn(0x7f5bf406a920 msgr2=0x7f5bf406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.728+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/709448037 shutdown_connections 2023-11-24T08:23:01.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.728+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/709448037 wait complete. 2023-11-24T08:23:01.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.730+0000 7f5bf9b0f640 1 Processor -- start 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.730+0000 7f5bf9b0f640 1 -- start start 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf4152880 0x7f5bf4152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bf414f140 0x7f5bf414f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40aec50 con 0x7f5be0003f20 2023-11-24T08:23:01.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40aef30 con 0x7f5bf414f140 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf9b0f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40af210 con 0x7f5bf4152880 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47054/0 (socket says 172.21.15.107:47054) 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 -- 172.21.15.107:0/2767637419 learned_addr learned my addr 172.21.15.107:0/2767637419 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bebfff640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf4152880 0x7f5bf4152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 -- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf4152880 msgr2=0x7f5bf4152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf4152880 0x7f5bf4152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 -- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bf414f140 msgr2=0x7f5bf414f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.731+0000 7f5bf37fe640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bf414f140 0x7f5bf414f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf37fe640 1 -- 172.21.15.107:0/2767637419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40af4f0 con 0x7f5be0003f20 2023-11-24T08:23:01.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf37fe640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f5be4000ff0 tx=0x7f5be400dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40074e0 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4007640 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40049e0 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf40af7d0 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.732+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf414fae0 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.733+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf40afcf0 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.734+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f5be4004b40 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.735+0000 7f5bf0ff9640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 0x7f5bd406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.735+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5be4088320 con 0x7f5be0003f20 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.736+0000 7f5bebfff640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 0x7f5bd406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.736+0000 7f5bebfff640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 0x7f5bd406f580 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc036000 tx=0x7f5bdc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:01.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.740+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bf40afcf0 con 0x7f5be0003f20 2023-11-24T08:23:01.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:01.975+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5bf40afcf0 con 0x7f5be0003f20 2023-11-24T08:23:02.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.892+0000 7f5bf0ff9640 1 -- 172.21.15.107:0/2767637419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v139) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5bf40afcf0 con 0x7f5be0003f20 2023-11-24T08:23:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 msgr2=0x7f5bd406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 0x7f5bd406f580 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc036000 tx=0x7f5bdc030000 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 msgr2=0x7f5bf4154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f5be4000ff0 tx=0x7f5be400dc70 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 shutdown_connections 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5bd406d0c0 0x7f5bd406f580 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bf414f140 0x7f5bf414f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bf4152880 0x7f5bf4152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 --2- 172.21.15.107:0/2767637419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0003f20 0x7f5bf4154640 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.896+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 >> 172.21.15.107:0/2767637419 conn(0x7f5bf406a920 msgr2=0x7f5bf406da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.897+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 shutdown_connections 2023-11-24T08:23:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:02.897+0000 7f5bf9b0f640 1 -- 172.21.15.107:0/2767637419 wait complete. 2023-11-24T08:23:02.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:23:03.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.135+0000 7fe86f0ad640 1 Processor -- start 2023-11-24T08:23:03.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.135+0000 7fe86f0ad640 1 -- start start 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe86815b1d0 0x7fe868071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe8680b1080 0x7fe86806c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8680b1510 con 0x7fe86815b1d0 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8680ca0e0 con 0x7fe8680b1080 2023-11-24T08:23:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.136+0000 7fe86f0ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868072800 con 0x7fe868070550 2023-11-24T08:23:03.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe86815b1d0 0x7fe868071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe86815b1d0 0x7fe868071f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47056/0 (socket says 172.21.15.107:47056) 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 -- 172.21.15.107:0/1295464985 learned_addr learned my addr 172.21.15.107:0/1295464985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e8ac640 1 --2- 172.21.15.107:0/1295464985 >> v2:172.21.15.120:3300/0 conn(0x7fe8680b1080 0x7fe86806c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 -- 172.21.15.107:0/1295464985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 msgr2=0x7fe8680b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86d8aa640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 -- 172.21.15.107:0/1295464985 >> v2:172.21.15.120:3300/0 conn(0x7fe8680b1080 msgr2=0x7fe86806c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1295464985 >> v2:172.21.15.120:3300/0 conn(0x7fe8680b1080 0x7fe86806c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86e0ab640 1 -- 172.21.15.107:0/1295464985 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8680c97b0 con 0x7fe86815b1d0 2023-11-24T08:23:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.137+0000 7fe86d8aa640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.138+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b1d0 0x7fe868071f40 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fe8680b51a0 tx=0x7fe86402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ab43b40928096d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.139+0000 7fe86d0a9640 1 -- 172.21.15.107:0/1295464985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864031870 con 0x7fe86815b1d0 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.139+0000 7fe86d0a9640 1 -- 172.21.15.107:0/1295464985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8640319d0 con 0x7fe86815b1d0 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.139+0000 7fe86d0a9640 1 -- 172.21.15.107:0/1295464985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864031d60 con 0x7fe86815b1d0 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.139+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1295464985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b1d0 msgr2=0x7fe868071f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.139+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b1d0 0x7fe868071f40 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fe8680b51a0 tx=0x7fe86402b910 comp rx=0 tx=0).stop 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1295464985 shutdown_connections 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1295464985 >> v2:172.21.15.120:3300/0 conn(0x7fe8680b1080 0x7fe86806c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1295464985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe86815b1d0 0x7fe868071f40 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1295464985 >> 172.21.15.107:0/1295464985 conn(0x7fe8680cbb40 msgr2=0x7fe8680cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1295464985 shutdown_connections 2023-11-24T08:23:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.140+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1295464985 wait complete. 2023-11-24T08:23:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.141+0000 7fe86f0ad640 1 Processor -- start 2023-11-24T08:23:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.142+0000 7fe86f0ad640 1 -- start start 2023-11-24T08:23:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.142+0000 7fe86f0ad640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.142+0000 7fe86f0ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.142+0000 7fe86f0ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 0x7fe86809a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86f0ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe868151350 con 0x7fe8680b1080 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86f0ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86809b260 con 0x7fe86815b1d0 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86f0ad640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86809b540 con 0x7fe868070550 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47064/0 (socket says 172.21.15.107:47064) 2023-11-24T08:23:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 -- 172.21.15.107:0/1776518027 learned_addr learned my addr 172.21.15.107:0/1776518027 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86e8ac640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 0x7fe86809a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 -- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 msgr2=0x7fe8680acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 -- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 msgr2=0x7fe86809a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 0x7fe86809a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86d8aa640 1 -- 172.21.15.107:0/1776518027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640094d0 con 0x7fe8680b1080 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.143+0000 7fe86e8ac640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 0x7fe86809a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe86d8aa640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c7c0 tx=0x7fe85800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858007a30 con 0x7fe8680b1080 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe858007b90 con 0x7fe8680b1080 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85800ce70 con 0x7fe8680b1080 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86809b820 con 0x7fe8680b1080 2023-11-24T08:23:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.144+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86809a8b0 con 0x7fe8680b1080 2023-11-24T08:23:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.145+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86809ba60 con 0x7fe8680b1080 2023-11-24T08:23:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.146+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fe8580113f0 con 0x7fe8680b1080 2023-11-24T08:23:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.147+0000 7fe85effd640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 0x7fe84c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.147+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe858088830 con 0x7fe8680b1080 2023-11-24T08:23:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.147+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 0x7fe84c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.148+0000 7fe86e0ab640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 0x7fe84c06f550 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe864036000 tx=0x7fe86403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.151+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8580553e0 con 0x7fe8680b1080 2023-11-24T08:23:03.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.367+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe86815b5b0 con 0x7fe84c06d090 2023-11-24T08:23:03.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.370+0000 7fe85effd640 1 -- 172.21.15.107:0/1776518027 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe864031bc0 con 0x7fe84c06d090 2023-11-24T08:23:03.371 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:23:03.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:03.371 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:23:03.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:03.371 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:23:03.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 msgr2=0x7fe84c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 0x7fe84c06f550 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe864036000 tx=0x7fe86403c040 comp rx=0 tx=0).stop 2023-11-24T08:23:03.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 msgr2=0x7fe8680ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c7c0 tx=0x7fe85800cc80 comp rx=0 tx=0).stop 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 shutdown_connections 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe84c06d090 0x7fe84c06f550 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe86815b1d0 0x7fe86809a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8680b1080 0x7fe8680ad510 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 --2- 172.21.15.107:0/1776518027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe868070550 0x7fe8680acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 >> 172.21.15.107:0/1776518027 conn(0x7fe8680cbb40 msgr2=0x7fe8680cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.372+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 shutdown_connections 2023-11-24T08:23:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.373+0000 7fe86f0ad640 1 -- 172.21.15.107:0/1776518027 wait complete. 2023-11-24T08:23:03.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:23:03.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.606+0000 7fbbc71f1640 1 Processor -- start 2023-11-24T08:23:03.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.607+0000 7fbbc71f1640 1 -- start start 2023-11-24T08:23:03.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc71f1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 0x7fbbc00c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc71f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbbc0161b10 0x7fbbc006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc71f1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc0161f60 con 0x7fbbc00c65f0 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc71f1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00c6a70 con 0x7fbbc0161b10 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc71f1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc015de00 con 0x7fbbc0140520 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbc578e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbbc0161b10 0x7fbbc006ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbbffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbbc00c65f0 0x7fbbc0161400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33786/0 (socket says 172.21.15.107:33786) 2023-11-24T08:23:03.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.608+0000 7fbbbffff640 1 -- 172.21.15.107:0/1963376852 learned_addr learned my addr 172.21.15.107:0/1963376852 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc4f8d640 1 --2- 172.21.15.107:0/1963376852 >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 0x7fbbc00c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc578e640 1 -- 172.21.15.107:0/1963376852 >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 msgr2=0x7fbbc00c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc578e640 1 --2- 172.21.15.107:0/1963376852 >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 0x7fbbc00c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc578e640 1 -- 172.21.15.107:0/1963376852 >> v1:172.21.15.107:6789/0 conn(0x7fbbc00c65f0 legacy=0x7fbbc0161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc578e640 1 -- 172.21.15.107:0/1963376852 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc015e0e0 con 0x7fbbc0161b10 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbc4f8d640 1 --2- 172.21.15.107:0/1963376852 >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 0x7fbbc00c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.609+0000 7fbbbf7fe640 1 -- 172.21.15.107:0/1963376852 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211991660 0 0) 0x7fbbc0161f60 con 0x7fbbc00c65f0 2023-11-24T08:23:03.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.610+0000 7fbbc578e640 1 --2- 172.21.15.107:0/1963376852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0161b10 0x7fbbc006ff70 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fbbc006e0b0 tx=0x7fbbb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ab92f8edcf4887 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.610+0000 7fbbbf7fe640 1 -- 172.21.15.107:0/1963376852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8031830 con 0x7fbbc0161b10 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.610+0000 7fbbbf7fe640 1 -- 172.21.15.107:0/1963376852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb8031990 con 0x7fbbc0161b10 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.610+0000 7fbbbf7fe640 1 -- 172.21.15.107:0/1963376852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8031d20 con 0x7fbbc0161b10 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1963376852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0161b10 msgr2=0x7fbbc006ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1963376852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0161b10 0x7fbbc006ff70 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fbbc006e0b0 tx=0x7fbbb802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1963376852 shutdown_connections 2023-11-24T08:23:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1963376852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0161b10 0x7fbbc006ff70 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1963376852 >> v2:172.21.15.120:3300/0 conn(0x7fbbc0140520 0x7fbbc00c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1963376852 >> 172.21.15.107:0/1963376852 conn(0x7fbbc00b2e40 msgr2=0x7fbbc00b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1963376852 shutdown_connections 2023-11-24T08:23:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.611+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1963376852 wait complete. 2023-11-24T08:23:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.613+0000 7fbbc71f1640 1 Processor -- start 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.613+0000 7fbbc71f1640 1 -- start start 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc00c65f0 0x7fbbc015c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc0161b10 0x7fbbc00b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00b19a0 con 0x7fbbc0140520 2023-11-24T08:23:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc014b6b0 con 0x7fbbc0161b10 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc71f1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00b0d90 con 0x7fbbc00c65f0 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47082/0 (socket says 172.21.15.107:47082) 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbbffff640 1 -- 172.21.15.107:0/1098760522 learned_addr learned my addr 172.21.15.107:0/1098760522 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.614+0000 7fbbc4f8d640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc00c65f0 0x7fbbc015c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 -- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc00c65f0 msgr2=0x7fbbc015c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc00c65f0 0x7fbbc015c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 -- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc0161b10 msgr2=0x7fbbc00b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc0161b10 0x7fbbc00b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 -- 172.21.15.107:0/1098760522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009400 con 0x7fbbc0140520 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbffff640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c4d0 tx=0x7fbba800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80074e0 con 0x7fbbc0140520 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba8007660 con 0x7fbbc0140520 2023-11-24T08:23:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8004a20 con 0x7fbbc0140520 2023-11-24T08:23:03.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc00b1070 con 0x7fbbc0140520 2023-11-24T08:23:03.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.615+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc00b1630 con 0x7fbbc0140520 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.617+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc00b1b10 con 0x7fbbc0140520 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.618+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fbba8004b80 con 0x7fbbc0140520 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.618+0000 7fbbbd7fa640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 0x7fbba006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.619+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbba8087930 con 0x7fbbc0140520 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.619+0000 7fbbc4f8d640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 0x7fbba006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.619+0000 7fbbc4f8d640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 0x7fbba006f550 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00075b0 tx=0x7fbbb0007540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.623+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbba80544e0 con 0x7fbbc0140520 2023-11-24T08:23:03.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.815+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbbc004e8b0 con 0x7fbbc0140520 2023-11-24T08:23:03.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.816+0000 7fbbbd7fa640 1 -- 172.21.15.107:0/1098760522 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v53)=0 v53) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbba8018070 con 0x7fbbc0140520 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 msgr2=0x7fbba006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 0x7fbba006f550 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00075b0 tx=0x7fbbb0007540 comp rx=0 tx=0).stop 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 msgr2=0x7fbbc00b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c4d0 tx=0x7fbba800c990 comp rx=0 tx=0).stop 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 shutdown_connections 2023-11-24T08:23:03.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbba006d090 0x7fbba006f550 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbc0161b10 0x7fbbc00b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0140520 0x7fbbc00b2260 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 --2- 172.21.15.107:0/1098760522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbc00c65f0 0x7fbbc015c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 >> 172.21.15.107:0/1098760522 conn(0x7fbbc00b2e40 msgr2=0x7fbbc00cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 shutdown_connections 2023-11-24T08:23:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:03.819+0000 7fbbc71f1640 1 -- 172.21.15.107:0/1098760522 wait complete. 2023-11-24T08:23:03.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol4247 --mode=777 2023-11-24T08:23:04.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.051+0000 7fbe77fff640 1 Processor -- start 2023-11-24T08:23:04.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.052+0000 7fbe77fff640 1 -- start start 2023-11-24T08:23:04.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.053+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 0x7fbe780d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.053+0000 7fbe77fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815ce20 0x7fbe7815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.053+0000 7fbe77fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7815ea30 con 0x7fbe78160b00 2023-11-24T08:23:04.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.053+0000 7fbe77fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe780d0560 con 0x7fbe7815ce20 2023-11-24T08:23:04.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.053+0000 7fbe77fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7815de20 con 0x7fbe780cfbe0 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815ce20 0x7fbe7815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7815ce20 0x7fbe7815d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47094/0 (socket says 172.21.15.107:47094) 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 -- 172.21.15.107:0/1787747531 learned_addr learned my addr 172.21.15.107:0/1787747531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe76ffd640 1 --1- 172.21.15.107:0/1787747531 >> v1:172.21.15.107:6789/0 conn(0x7fbe78160b00 0x7fbe780d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33800/0 (socket says 172.21.15.107:33800) 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe767fc640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 0x7fbe780d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 -- 172.21.15.107:0/1787747531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 msgr2=0x7fbe780d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:04.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 0x7fbe780d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 -- 172.21.15.107:0/1787747531 >> v1:172.21.15.107:6789/0 conn(0x7fbe78160b00 legacy=0x7fbe780d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe777fe640 1 -- 172.21.15.107:0/1787747531 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7815e100 con 0x7fbe7815ce20 2023-11-24T08:23:04.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.054+0000 7fbe767fc640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 0x7fbe780d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:04.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.055+0000 7fbe777fe640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815ce20 0x7fbe7815d2b0 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009870 tx=0x7fbe7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694f4a8f8414b03f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1787747531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70030840 con 0x7fbe7815ce20 2023-11-24T08:23:04.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1787747531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe700309a0 con 0x7fbe7815ce20 2023-11-24T08:23:04.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1787747531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70030d30 con 0x7fbe7815ce20 2023-11-24T08:23:04.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 -- 172.21.15.107:0/1787747531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815ce20 msgr2=0x7fbe7815d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815ce20 0x7fbe7815d2b0 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009870 tx=0x7fbe7002b980 comp rx=0 tx=0).stop 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 -- 172.21.15.107:0/1787747531 shutdown_connections 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7815ce20 0x7fbe7815d2b0 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1787747531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe780cfbe0 0x7fbe780d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 -- 172.21.15.107:0/1787747531 >> 172.21.15.107:0/1787747531 conn(0x7fbe7809c510 msgr2=0x7fbe780d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.056+0000 7fbe77fff640 1 -- 172.21.15.107:0/1787747531 shutdown_connections 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.057+0000 7fbe77fff640 1 -- 172.21.15.107:0/1787747531 wait complete. 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.058+0000 7fbe77fff640 1 Processor -- start 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.058+0000 7fbe77fff640 1 -- start start 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cfbe0 0x7fbe78157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815ce20 0x7fbe780c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780c76c0 con 0x7fbe78160b00 2023-11-24T08:23:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780c79a0 con 0x7fbe780cfbe0 2023-11-24T08:23:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe77fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780c7c80 con 0x7fbe7815ce20 2023-11-24T08:23:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.059+0000 7fbe777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47098/0 (socket says 172.21.15.107:47098) 2023-11-24T08:23:04.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 -- 172.21.15.107:0/3051266633 learned_addr learned my addr 172.21.15.107:0/3051266633 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 -- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815ce20 msgr2=0x7fbe780c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815ce20 0x7fbe780c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 -- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cfbe0 msgr2=0x7fbe78157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cfbe0 0x7fbe78157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 -- 172.21.15.107:0/3051266633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700094d0 con 0x7fbe78160b00 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.060+0000 7fbe777fe640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fbe7002b9c0 tx=0x7fbe70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.061+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70039c20 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.061+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70039da0 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.061+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70043630 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.061+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe780c7f60 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.061+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe780c7220 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.062+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe780b5e80 con 0x7fbe78160b00 2023-11-24T08:23:04.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.063+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fbe70042390 con 0x7fbe78160b00 2023-11-24T08:23:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.064+0000 7fbe6ffff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 0x7fbe5806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.064+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe700aaae0 con 0x7fbe78160b00 2023-11-24T08:23:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.064+0000 7fbe76ffd640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 0x7fbe5806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.065+0000 7fbe76ffd640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 0x7fbe5806f550 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fbe68007fd0 tx=0x7fbe68007d80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.068+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe780b5e80 con 0x7fbe78160b00 2023-11-24T08:23:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.283+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol4247", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7815ee40 con 0x7fbe5806d090 2023-11-24T08:23:04.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:04.296+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:04.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:04.296+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:04.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:04.296+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:04.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:04.296+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:04.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:04.296+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:04.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.336+0000 7fbe6ffff640 1 -- 172.21.15.107:0/3051266633 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe6800c730 con 0x7fbe5806d090 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 msgr2=0x7fbe5806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 0x7fbe5806f550 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fbe68007fd0 tx=0x7fbe68007d80 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 msgr2=0x7fbe780b8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fbe7002b9c0 tx=0x7fbe70004730 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 shutdown_connections 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe5806d090 0x7fbe5806f550 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe78160b00 0x7fbe780b8320 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815ce20 0x7fbe780c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 --2- 172.21.15.107:0/3051266633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe780cfbe0 0x7fbe78157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 >> 172.21.15.107:0/3051266633 conn(0x7fbe7809c510 msgr2=0x7fbe780d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 shutdown_connections 2023-11-24T08:23:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.339+0000 7fbe77fff640 1 -- 172.21.15.107:0/3051266633 wait complete. 2023-11-24T08:23:04.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4247 2023-11-24T08:23:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.551+0000 7f2ecce51640 1 Processor -- start 2023-11-24T08:23:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.551+0000 7f2ecce51640 1 -- start start 2023-11-24T08:23:04.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.552+0000 7f2ecce51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ec815b440 0x7f2ec80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ecce51640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2ec80cbf20 0x7f2ec80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ecce51640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec815eea0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ecce51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec80cc3d0 con 0x7f2ec815b440 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ecce51640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec815b820 con 0x7f2ec80cbf20 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ec815b440 0x7f2ec80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2ec815b440 0x7f2ec80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47132/0 (socket says 172.21.15.107:47132) 2023-11-24T08:23:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 -- 172.21.15.107:0/3276865238 learned_addr learned my addr 172.21.15.107:0/3276865238 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ebffff640 1 --2- 172.21.15.107:0/3276865238 >> v2:172.21.15.120:3300/0 conn(0x7f2ec80cbf20 0x7f2ec80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 -- 172.21.15.107:0/3276865238 >> v2:172.21.15.120:3300/0 conn(0x7f2ec80cbf20 msgr2=0x7f2ec80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 --2- 172.21.15.107:0/3276865238 >> v2:172.21.15.120:3300/0 conn(0x7f2ec80cbf20 0x7f2ec80c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 -- 172.21.15.107:0/3276865238 >> v1:172.21.15.120:6789/0 conn(0x7f2ec80c57e0 legacy=0x7f2ec80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.553+0000 7f2ec659c640 1 -- 172.21.15.107:0/3276865238 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec806d600 con 0x7f2ec815b440 2023-11-24T08:23:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.554+0000 7f2ec659c640 1 --2- 172.21.15.107:0/3276865238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec815b440 0x7f2ec80cb9e0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f2ec806e890 tx=0x7f2eb802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9054a3a90adde8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.555+0000 7f2ec5d9b640 1 -- 172.21.15.107:0/3276865238 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80317e0 con 0x7f2ec815b440 2023-11-24T08:23:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.555+0000 7f2ec5d9b640 1 -- 172.21.15.107:0/3276865238 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb8031940 con 0x7f2ec815b440 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.555+0000 7f2ec5d9b640 1 -- 172.21.15.107:0/3276865238 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8031cd0 con 0x7f2ec815b440 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.555+0000 7f2ecce51640 1 -- 172.21.15.107:0/3276865238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec815b440 msgr2=0x7f2ec80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.555+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3276865238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec815b440 0x7f2ec80cb9e0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f2ec806e890 tx=0x7f2eb802b880 comp rx=0 tx=0).stop 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 -- 172.21.15.107:0/3276865238 shutdown_connections 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3276865238 >> v2:172.21.15.120:3300/0 conn(0x7f2ec80cbf20 0x7f2ec80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3276865238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec815b440 0x7f2ec80cb9e0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 -- 172.21.15.107:0/3276865238 >> 172.21.15.107:0/3276865238 conn(0x7f2ec80b0c50 msgr2=0x7f2ec80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 -- 172.21.15.107:0/3276865238 shutdown_connections 2023-11-24T08:23:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.556+0000 7f2ecce51640 1 -- 172.21.15.107:0/3276865238 wait complete. 2023-11-24T08:23:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.557+0000 7f2ecce51640 1 Processor -- start 2023-11-24T08:23:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.558+0000 7f2ecce51640 1 -- start start 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.558+0000 7f2ecce51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ecce51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ec80cbf20 0x7f2ec80b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ecce51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ec815b440 0x7f2ec80704c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ecce51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec81433f0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ecce51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80999d0 con 0x7f2ec80cbf20 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ecce51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80b49a0 con 0x7f2ec815b440 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47138/0 (socket says 172.21.15.107:47138) 2023-11-24T08:23:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 -- 172.21.15.107:0/3515507390 learned_addr learned my addr 172.21.15.107:0/3515507390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 -- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ec815b440 msgr2=0x7f2ec80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ec815b440 0x7f2ec80704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 -- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ec80cbf20 msgr2=0x7f2ec80b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ec80cbf20 0x7f2ec80b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.559+0000 7f2ec659c640 1 -- 172.21.15.107:0/3515507390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80094d0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ec659c640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8000c00 tx=0x7f2eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8032980 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb8032b00 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80453f0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec8099c20 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.560+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec809a1e0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.561+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec80c93c0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.562+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7f2eb804e450 con 0x7f2ec80c57e0 2023-11-24T08:23:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.563+0000 7f2ebeffd640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 0x7f2ea806f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.563+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2eb80aab60 con 0x7f2ec80c57e0 2023-11-24T08:23:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.563+0000 7f2ebffff640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 0x7f2ea806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.564+0000 7f2ebffff640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 0x7f2ea806f550 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80710b0 tx=0x7f2eb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.567+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2eb80777c0 con 0x7f2ec80c57e0 2023-11-24T08:23:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.787+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4247", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ec80cbd50 con 0x7f2ea806d090 2023-11-24T08:23:04.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.796+0000 7f2ebeffd640 1 -- 172.21.15.107:0/3515507390 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f2ec80cbd50 con 0x7f2ea806d090 2023-11-24T08:23:04.798 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd 2023-11-24T08:23:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 msgr2=0x7f2ea806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 0x7f2ea806f550 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80710b0 tx=0x7f2eb0008040 comp rx=0 tx=0).stop 2023-11-24T08:23:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 msgr2=0x7f2ec8142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8000c00 tx=0x7f2eb80046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 shutdown_connections 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2ea806d090 0x7f2ea806f550 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ec815b440 0x7f2ec80704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ec80cbf20 0x7f2ec80b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 --2- 172.21.15.107:0/3515507390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec80c57e0 0x7f2ec8142eb0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 >> 172.21.15.107:0/3515507390 conn(0x7f2ec80b0c50 msgr2=0x7f2ec80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 shutdown_connections 2023-11-24T08:23:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:04.799+0000 7f2ecce51640 1 -- 172.21.15.107:0/3515507390 wait complete. 2023-11-24T08:23:04.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4247 2023-11-24T08:23:05.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.031+0000 7fdccf355640 1 Processor -- start 2023-11-24T08:23:05.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.032+0000 7fdccf355640 1 -- start start 2023-11-24T08:23:05.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.033+0000 7fdccf355640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdcc815d3f0 0x7fdcc80d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.033+0000 7fdccf355640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdcc815d7d0 0x7fdcc80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.033+0000 7fdccf355640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80d15e0 con 0x7fdcc80d0e40 2023-11-24T08:23:05.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.033+0000 7fdccf355640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc80d0280 con 0x7fdcc815d7d0 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.033+0000 7fdccf355640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc80d1300 con 0x7fdcc815d3f0 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdcceb54640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdcc80d0e40 0x7fdcc80afea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33812/0 (socket says 172.21.15.107:33812) 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdcceb54640 1 -- 172.21.15.107:0/3165463065 learned_addr learned my addr 172.21.15.107:0/3165463065 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccdb52640 1 --2- 172.21.15.107:0/3165463065 >> v2:172.21.15.107:3300/0 conn(0x7fdcc815d7d0 0x7fdcc80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdcce353640 1 --2- 172.21.15.107:0/3165463065 >> v2:172.21.15.120:3300/0 conn(0x7fdcc815d3f0 0x7fdcc80d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccd351640 1 -- 172.21.15.107:0/3165463065 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648024217 0 0) 0x7fdcc80d15e0 con 0x7fdcc80d0e40 2023-11-24T08:23:05.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccd351640 1 -- 172.21.15.107:0/3165463065 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb4003520 con 0x7fdcc80d0e40 2023-11-24T08:23:05.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccdb52640 1 -- 172.21.15.107:0/3165463065 >> v2:172.21.15.120:3300/0 conn(0x7fdcc815d3f0 msgr2=0x7fdcc80d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:05.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccdb52640 1 --2- 172.21.15.107:0/3165463065 >> v2:172.21.15.120:3300/0 conn(0x7fdcc815d3f0 0x7fdcc80d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccdb52640 1 -- 172.21.15.107:0/3165463065 >> v1:172.21.15.107:6789/0 conn(0x7fdcc80d0e40 legacy=0x7fdcc80afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.034+0000 7fdccdb52640 1 -- 172.21.15.107:0/3165463065 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80cfc30 con 0x7fdcc815d7d0 2023-11-24T08:23:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.035+0000 7fdccdb52640 1 --2- 172.21.15.107:0/3165463065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d7d0 0x7fdcc80d1d70 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8067c60 tx=0x7fdcbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0048d7895eb47c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccd351640 1 -- 172.21.15.107:0/3165463065 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc031830 con 0x7fdcc815d7d0 2023-11-24T08:23:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccd351640 1 -- 172.21.15.107:0/3165463065 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc031990 con 0x7fdcc815d7d0 2023-11-24T08:23:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccd351640 1 -- 172.21.15.107:0/3165463065 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc031d20 con 0x7fdcc815d7d0 2023-11-24T08:23:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 -- 172.21.15.107:0/3165463065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d7d0 msgr2=0x7fdcc80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 --2- 172.21.15.107:0/3165463065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d7d0 0x7fdcc80d1d70 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8067c60 tx=0x7fdcbc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 -- 172.21.15.107:0/3165463065 shutdown_connections 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 --2- 172.21.15.107:0/3165463065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d7d0 0x7fdcc80d1d70 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 --2- 172.21.15.107:0/3165463065 >> v2:172.21.15.120:3300/0 conn(0x7fdcc815d3f0 0x7fdcc80d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 -- 172.21.15.107:0/3165463065 >> 172.21.15.107:0/3165463065 conn(0x7fdcc809c5b0 msgr2=0x7fdcc80d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 -- 172.21.15.107:0/3165463065 shutdown_connections 2023-11-24T08:23:05.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.036+0000 7fdccf355640 1 -- 172.21.15.107:0/3165463065 wait complete. 2023-11-24T08:23:05.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.038+0000 7fdccf355640 1 Processor -- start 2023-11-24T08:23:05.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.038+0000 7fdccf355640 1 -- start start 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcc80d0e40 0x7fdcc814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcc815d7d0 0x7fdcc80c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc814cf80 con 0x7fdcc815d3f0 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80b5040 con 0x7fdcc80d0e40 2023-11-24T08:23:05.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccf355640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80b5320 con 0x7fdcc815d7d0 2023-11-24T08:23:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccdb52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccdb52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47150/0 (socket says 172.21.15.107:47150) 2023-11-24T08:23:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.039+0000 7fdccdb52640 1 -- 172.21.15.107:0/1844659344 learned_addr learned my addr 172.21.15.107:0/1844659344 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 -- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcc815d7d0 msgr2=0x7fdcc80c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcc815d7d0 0x7fdcc80c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 -- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcc80d0e40 msgr2=0x7fdcc814ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcc80d0e40 0x7fdcc814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 -- 172.21.15.107:0/1844659344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc009400 con 0x7fdcc815d3f0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccdb52640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc000c00 tx=0x7fdcbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc03a8f0 con 0x7fdcc815d3f0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc03aa70 con 0x7fdcc815d3f0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc03ae20 con 0x7fdcc815d3f0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.040+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc80b5600 con 0x7fdcc815d3f0 2023-11-24T08:23:05.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.041+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc80cd140 con 0x7fdcc815d3f0 2023-11-24T08:23:05.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.042+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc804e840 con 0x7fdcc815d3f0 2023-11-24T08:23:05.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.043+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 45) v1 ==== 92342+0+0 (secure 0 0 0) 0x7fdcbc04d450 con 0x7fdcc815d3f0 2023-11-24T08:23:05.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.043+0000 7fdcbaffd640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 0x7fdcac06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:05.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.043+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcbc0aa850 con 0x7fdcc815d3f0 2023-11-24T08:23:05.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.044+0000 7fdcce353640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 0x7fdcac06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:05.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.044+0000 7fdcce353640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 0x7fdcac06f550 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098f0 tx=0x7fdcc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:05.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.048+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdcbc077360 con 0x7fdcc815d3f0 2023-11-24T08:23:05.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.266+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4247", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcc80d0710 con 0x7fdcac06d090 2023-11-24T08:23:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.276+0000 7fdcbaffd640 1 -- 172.21.15.107:0/1844659344 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fdcc4009e70 con 0x7fdcac06d090 2023-11-24T08:23:05.278 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd 2023-11-24T08:23:05.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.278+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 msgr2=0x7fdcac06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.278+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 0x7fdcac06f550 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098f0 tx=0x7fdcc4008040 comp rx=0 tx=0).stop 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.278+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 msgr2=0x7fdcc8147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.278+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc000c00 tx=0x7fdcbc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 shutdown_connections 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdcac06d090 0x7fdcac06f550 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcc815d7d0 0x7fdcc80c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcc815d3f0 0x7fdcc8147090 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 --2- 172.21.15.107:0/1844659344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcc80d0e40 0x7fdcc814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 >> 172.21.15.107:0/1844659344 conn(0x7fdcc809c5b0 msgr2=0x7fdcc80d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 shutdown_connections 2023-11-24T08:23:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:05.279+0000 7fdccf355640 1 -- 172.21.15.107:0/1844659344 wait complete. 2023-11-24T08:23:05.297 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol4247 with 16 files each 1MB size under directory volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd 2023-11-24T08:23:05.297 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:05.297 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:05.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:05.345 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336643 s, 31.1 MB/s 2023-11-24T08:23:05.491 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:05.491 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:05.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:05.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:05.624 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:05.624 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:05.625 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259593 s, 40.4 MB/s 2023-11-24T08:23:05.626 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:05.626 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:05.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:05.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:05.765 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:05.765 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:05.765 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267626 s, 39.2 MB/s 2023-11-24T08:23:05.767 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:05.767 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:05.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:05.824 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208701 s, 50.2 MB/s 2023-11-24T08:23:05.902 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:05.902 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:05.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:05.958 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297839 s, 35.2 MB/s 2023-11-24T08:23:06.043 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.043 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.100 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.182 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.183 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.183 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294257 s, 35.6 MB/s 2023-11-24T08:23:06.184 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.184 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252357 s, 41.6 MB/s 2023-11-24T08:23:06.317 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.317 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.374 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.444 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.445 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.445 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215126 s, 48.7 MB/s 2023-11-24T08:23:06.446 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.446 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252494 s, 41.5 MB/s 2023-11-24T08:23:06.584 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.584 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.642 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.722 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275124 s, 38.1 MB/s 2023-11-24T08:23:06.724 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.724 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.858 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.858 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.858 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218582 s, 48.0 MB/s 2023-11-24T08:23:06.860 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.860 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:06.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:06.919 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253492 s, 41.4 MB/s 2023-11-24T08:23:06.998 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:06.998 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:07.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:07.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022865 s, 45.9 MB/s 2023-11-24T08:23:07.182 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:07.182 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:07.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:07.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201158 s, 52.1 MB/s 2023-11-24T08:23:07.316 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:07.316 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:07.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:07.369 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196373 s, 53.4 MB/s 2023-11-24T08:23:07.441 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:07.441 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:07.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:07.499 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol4247/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:23:07.570 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:23:07.570 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:23:07.571 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188504 s, 55.6 MB/s 2023-11-24T08:23:07.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol4247 subvol_snap4594 2023-11-24T08:23:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.783+0000 7f4a5dd01640 1 Processor -- start 2023-11-24T08:23:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.783+0000 7f4a5dd01640 1 -- start start 2023-11-24T08:23:07.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.784+0000 7f4a5dd01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 0x7f4a580acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a5dd01640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a5806ac10 0x7f4a580ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a5dd01640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a580cfed0 con 0x7f4a580b2400 2023-11-24T08:23:07.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a5dd01640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a580caae0 con 0x7f4a5806ac10 2023-11-24T08:23:07.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a5dd01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580cadc0 con 0x7f4a5815d920 2023-11-24T08:23:07.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a57fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a5806ac10 0x7f4a580ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:07.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a57fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a5806ac10 0x7f4a580ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:49404/0 (socket says 172.21.15.107:49404) 2023-11-24T08:23:07.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a57fff640 1 -- 172.21.15.107:0/3887274919 learned_addr learned my addr 172.21.15.107:0/3887274919 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:07.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.785+0000 7f4a577fe640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 0x7f4a580acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:07.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.787+0000 7f4a57fff640 1 -- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 msgr2=0x7f4a580acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:07.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.787+0000 7f4a57fff640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 0x7f4a580acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.787+0000 7f4a57fff640 1 -- 172.21.15.107:0/3887274919 >> v1:172.21.15.120:6789/0 conn(0x7f4a580b2400 legacy=0x7f4a5806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:07.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.787+0000 7f4a57fff640 1 -- 172.21.15.107:0/3887274919 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580cb0a0 con 0x7f4a5806ac10 2023-11-24T08:23:07.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.787+0000 7f4a577fe640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 0x7f4a580acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:23:07.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.788+0000 7f4a57fff640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a580ccb90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009e50 tx=0x7f4a5002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc2e7d97afb8eac0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:07.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.789+0000 7f4a567fc640 1 -- 172.21.15.107:0/3887274919 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5002cb90 con 0x7f4a5806ac10 2023-11-24T08:23:07.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.789+0000 7f4a567fc640 1 -- 172.21.15.107:0/3887274919 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5002ccf0 con 0x7f4a5806ac10 2023-11-24T08:23:07.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.789+0000 7f4a567fc640 1 -- 172.21.15.107:0/3887274919 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50034610 con 0x7f4a5806ac10 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 -- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 msgr2=0x7f4a580ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a580ccb90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009e50 tx=0x7f4a5002bff0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 -- 172.21.15.107:0/3887274919 shutdown_connections 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a580ccb90 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/3887274919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a5815d920 0x7f4a580acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 -- 172.21.15.107:0/3887274919 >> 172.21.15.107:0/3887274919 conn(0x7f4a580999d0 msgr2=0x7f4a580b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:07.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 -- 172.21.15.107:0/3887274919 shutdown_connections 2023-11-24T08:23:07.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.790+0000 7f4a5dd01640 1 -- 172.21.15.107:0/3887274919 wait complete. 2023-11-24T08:23:07.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.792+0000 7f4a5dd01640 1 Processor -- start 2023-11-24T08:23:07.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.792+0000 7f4a5dd01640 1 -- start start 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a58154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a580b2400 0x7f4a580b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580b38d0 con 0x7f4a5815d920 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580b2130 con 0x7f4a5806ac10 2023-11-24T08:23:07.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a5dd01640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58153940 con 0x7f4a580b2400 2023-11-24T08:23:07.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47156/0 (socket says 172.21.15.107:47156) 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.793+0000 7f4a57fff640 1 -- 172.21.15.107:0/735361618 learned_addr learned my addr 172.21.15.107:0/735361618 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a577fe640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a58154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 -- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a580b2400 msgr2=0x7f4a580b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a580b2400 0x7f4a580b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 -- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 msgr2=0x7f4a58154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a58154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 -- 172.21.15.107:0/735361618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48009400 con 0x7f4a5815d920 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a577fe640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a58154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.794+0000 7f4a57fff640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009e50 tx=0x7f4a50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.795+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50009510 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.795+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5003e450 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.795+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5003e800 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.795+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a50009b30 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.795+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a58153f00 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.796+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58067470 con 0x7f4a5815d920 2023-11-24T08:23:07.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.797+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 46) v1 ==== 92393+0+0 (secure 0 0 0) 0x7f4a5003e960 con 0x7f4a5815d920 2023-11-24T08:23:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.798+0000 7f4a3ffff640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 0x7f4a4406f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.798+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a500ab2c0 con 0x7f4a5815d920 2023-11-24T08:23:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.798+0000 7f4a577fe640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 0x7f4a4406f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.799+0000 7f4a577fe640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 0x7f4a4406f5e0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009bb0 tx=0x7f4a48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:07.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:07.803+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a50077e70 con 0x7f4a5815d920 2023-11-24T08:23:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.025+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol4247", "snap_name": "subvol_snap4594", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a580cfd20 con 0x7f4a4406d120 2023-11-24T08:23:08.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.053+0000 7f4a3ffff640 1 -- 172.21.15.107:0/735361618 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a580cfd20 con 0x7f4a4406d120 2023-11-24T08:23:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 msgr2=0x7f4a4406f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 0x7f4a4406f5e0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009bb0 tx=0x7f4a48008040 comp rx=0 tx=0).stop 2023-11-24T08:23:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 msgr2=0x7f4a580b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009e50 tx=0x7f4a50004a40 comp rx=0 tx=0).stop 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 shutdown_connections 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f4a4406d120 0x7f4a4406f5e0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a5815d920 0x7f4a580b3230 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a580b2400 0x7f4a580b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 --2- 172.21.15.107:0/735361618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a5806ac10 0x7f4a58154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 >> 172.21.15.107:0/735361618 conn(0x7f4a580999d0 msgr2=0x7f4a5806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.057+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 shutdown_connections 2023-11-24T08:23:08.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.058+0000 7f4a5dd01640 1 -- 172.21.15.107:0/735361618 wait complete. 2023-11-24T08:23:08.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol4247 --retain-snapshots 2023-11-24T08:23:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.279+0000 7fc0a1209640 1 Processor -- start 2023-11-24T08:23:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.280+0000 7fc0a1209640 1 -- start start 2023-11-24T08:23:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc0a1209640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc09c0cc470 0x7fc09c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc0a1209640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc0a1209640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0ce270 con 0x7fc09c0cf450 2023-11-24T08:23:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc0a1209640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c06c560 con 0x7fc09c0cc470 2023-11-24T08:23:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc0a1209640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c06d0a0 con 0x7fc09c0ce640 2023-11-24T08:23:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.281+0000 7fc09ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc09c0cc470 0x7fc09c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc09c0cc470 0x7fc09c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49988/0 (socket says 172.21.15.107:49988) 2023-11-24T08:23:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 -- 172.21.15.107:0/749857704 learned_addr learned my addr 172.21.15.107:0/749857704 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 -- 172.21.15.107:0/749857704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 msgr2=0x7fc09c0b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09a59c640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c0b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 -- 172.21.15.107:0/749857704 >> v1:172.21.15.120:6789/0 conn(0x7fc09c0cf450 legacy=0x7fc09c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:08.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.282+0000 7fc09ad9d640 1 -- 172.21.15.107:0/749857704 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0b4200 con 0x7fc09c0cc470 2023-11-24T08:23:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.283+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cc470 0x7fc09c0abfc0 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009e50 tx=0x7fc08c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6605e9690427e43f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc099d9b640 1 -- 172.21.15.107:0/749857704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c02ca70 con 0x7fc09c0cc470 2023-11-24T08:23:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc099d9b640 1 -- 172.21.15.107:0/749857704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c02cbd0 con 0x7fc09c0cc470 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc099d9b640 1 -- 172.21.15.107:0/749857704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0343f0 con 0x7fc09c0cc470 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 -- 172.21.15.107:0/749857704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cc470 msgr2=0x7fc09c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cc470 0x7fc09c0abfc0 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009e50 tx=0x7fc08c02bed0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 -- 172.21.15.107:0/749857704 shutdown_connections 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 --2- 172.21.15.107:0/749857704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cc470 0x7fc09c0abfc0 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 -- 172.21.15.107:0/749857704 >> 172.21.15.107:0/749857704 conn(0x7fc09c06a920 msgr2=0x7fc09c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:08.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 -- 172.21.15.107:0/749857704 shutdown_connections 2023-11-24T08:23:08.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.284+0000 7fc0a1209640 1 -- 172.21.15.107:0/749857704 wait complete. 2023-11-24T08:23:08.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.286+0000 7fc0a1209640 1 Processor -- start 2023-11-24T08:23:08.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.286+0000 7fc0a1209640 1 -- start start 2023-11-24T08:23:08.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 0x7fc09c1494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c149a20 con 0x7fc09c0cf450 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c14b750 con 0x7fc09c0cc470 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc0a1209640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c14ba30 con 0x7fc09c0ce640 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.287+0000 7fc09b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49994/0 (socket says 172.21.15.107:49994) 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 -- 172.21.15.107:0/2367442999 learned_addr learned my addr 172.21.15.107:0/2367442999 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:08.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 0x7fc09c1494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 -- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 msgr2=0x7fc09c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 -- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 msgr2=0x7fc09c1494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 0x7fc09c1494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 -- 172.21.15.107:0/2367442999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009b30 con 0x7fc09c0cf450 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 0x7fc09c1494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.288+0000 7fc09b59e640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fc09400b3e0 tx=0x7fc09400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.289+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094011830 con 0x7fc09c0cf450 2023-11-24T08:23:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.289+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0940119b0 con 0x7fc09c0cf450 2023-11-24T08:23:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.289+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094011d60 con 0x7fc09c0cf450 2023-11-24T08:23:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.289+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c14bd10 con 0x7fc09c0cf450 2023-11-24T08:23:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.289+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c15abd0 con 0x7fc09c0cf450 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.290+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c14c230 con 0x7fc09c0cf450 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.291+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 46) v1 ==== 92393+0+0 (secure 0 0 0) 0x7fc09401a400 con 0x7fc09c0cf450 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.292+0000 7fc0937fe640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 0x7fc07c06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.292+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0940888a0 con 0x7fc09c0cf450 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.292+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 0x7fc07c06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.293+0000 7fc09ad9d640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 0x7fc07c06f5e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc08c035000 tx=0x7fc08c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.296+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc09c14c230 con 0x7fc09c0cf450 2023-11-24T08:23:08.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.516+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol4247", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc09c14b8c0 con 0x7fc07c06d120 2023-11-24T08:23:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.584+0000 7fc0937fe640 1 -- 172.21.15.107:0/2367442999 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc09c14b8c0 con 0x7fc07c06d120 2023-11-24T08:23:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 msgr2=0x7fc07c06f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 0x7fc07c06f5e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc08c035000 tx=0x7fc08c030000 comp rx=0 tx=0).stop 2023-11-24T08:23:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 msgr2=0x7fc09c14c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fc09400b3e0 tx=0x7fc09400b8a0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 shutdown_connections 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc07c06d120 0x7fc07c06f5e0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc09c0cf450 0x7fc09c14c510 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0ce640 0x7fc09c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 --2- 172.21.15.107:0/2367442999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c0cc470 0x7fc09c1494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 >> 172.21.15.107:0/2367442999 conn(0x7fc09c06a920 msgr2=0x7fc09c0ac5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.587+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 shutdown_connections 2023-11-24T08:23:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.588+0000 7fc0a1209640 1 -- 172.21.15.107:0/2367442999 wait complete. 2023-11-24T08:23:08.595 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.594+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.595 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.594+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.595 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.594+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.595 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.594+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.595 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.594+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol4247 subvol_snap4594 subvol_clone7352 2023-11-24T08:23:08.617 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.614+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.617 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.614+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.617 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.614+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.617 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.614+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.618 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:08.614+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.812+0000 7f5c482b4640 1 Processor -- start 2023-11-24T08:23:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.812+0000 7f5c482b4640 1 -- start start 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c482b4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c40072160 0x7f5c4015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c482b4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c400c49f0 0x7f5c4006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c482b4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4015ec90 con 0x7f5c4006dc40 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c482b4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4006e090 con 0x7f5c40072160 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c482b4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c400cd530 con 0x7f5c400c49f0 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c46050640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c40072160 0x7f5c4015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c46851640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c4006dc40 0x7f5c400cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33386/0 (socket says 172.21.15.107:33386) 2023-11-24T08:23:08.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.814+0000 7f5c46851640 1 -- 172.21.15.107:0/3217240764 learned_addr learned my addr 172.21.15.107:0/3217240764 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.815+0000 7f5c4584f640 1 --2- 172.21.15.107:0/3217240764 >> v2:172.21.15.120:3300/0 conn(0x7f5c400c49f0 0x7f5c4006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.815+0000 7f5c46050640 1 -- 172.21.15.107:0/3217240764 >> v2:172.21.15.120:3300/0 conn(0x7f5c400c49f0 msgr2=0x7f5c4006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.815+0000 7f5c46050640 1 --2- 172.21.15.107:0/3217240764 >> v2:172.21.15.120:3300/0 conn(0x7f5c400c49f0 0x7f5c4006d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.815+0000 7f5c46050640 1 -- 172.21.15.107:0/3217240764 >> v1:172.21.15.107:6789/0 conn(0x7f5c4006dc40 legacy=0x7f5c400cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.815+0000 7f5c46050640 1 -- 172.21.15.107:0/3217240764 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c400c4ed0 con 0x7f5c40072160 2023-11-24T08:23:08.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.816+0000 7f5c46050640 1 --2- 172.21.15.107:0/3217240764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c4015e750 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f5c400cef60 tx=0x7f5c3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf92535be1284b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.816+0000 7f5c4504e640 1 -- 172.21.15.107:0/3217240764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031830 con 0x7f5c40072160 2023-11-24T08:23:08.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.816+0000 7f5c4504e640 1 -- 172.21.15.107:0/3217240764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c031990 con 0x7f5c40072160 2023-11-24T08:23:08.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.816+0000 7f5c4504e640 1 -- 172.21.15.107:0/3217240764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031d20 con 0x7f5c40072160 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 -- 172.21.15.107:0/3217240764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 msgr2=0x7f5c4015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3217240764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c4015e750 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f5c400cef60 tx=0x7f5c3c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 -- 172.21.15.107:0/3217240764 shutdown_connections 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3217240764 >> v2:172.21.15.120:3300/0 conn(0x7f5c400c49f0 0x7f5c4006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3217240764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c4015e750 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 -- 172.21.15.107:0/3217240764 >> 172.21.15.107:0/3217240764 conn(0x7f5c400b0c50 msgr2=0x7f5c400b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:08.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 -- 172.21.15.107:0/3217240764 shutdown_connections 2023-11-24T08:23:08.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.817+0000 7f5c482b4640 1 -- 172.21.15.107:0/3217240764 wait complete. 2023-11-24T08:23:08.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.819+0000 7f5c482b4640 1 Processor -- start 2023-11-24T08:23:08.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.819+0000 7f5c482b4640 1 -- start start 2023-11-24T08:23:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4006dc40 0x7f5c4015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c400c49f0 0x7f5c40158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40158470 con 0x7f5c40072160 2023-11-24T08:23:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c400b1e30 con 0x7f5c400c49f0 2023-11-24T08:23:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c482b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c400b2080 con 0x7f5c4006dc40 2023-11-24T08:23:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50032/0 (socket says 172.21.15.107:50032) 2023-11-24T08:23:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 -- 172.21.15.107:0/3708994223 learned_addr learned my addr 172.21.15.107:0/3708994223 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c46050640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4006dc40 0x7f5c4015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 -- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4006dc40 msgr2=0x7f5c4015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c46851640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c400c49f0 0x7f5c40158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4006dc40 0x7f5c4015a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 -- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c400c49f0 msgr2=0x7f5c40158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c400c49f0 0x7f5c40158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.820+0000 7f5c4584f640 1 -- 172.21.15.107:0/3708994223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c009400 con 0x7f5c40072160 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c4584f640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000d830 tx=0x7f5c3000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000b6c0 con 0x7f5c40072160 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3000b820 con 0x7f5c40072160 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c400b2360 con 0x7f5c40072160 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000bbb0 con 0x7f5c40072160 2023-11-24T08:23:08.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.821+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40158020 con 0x7f5c40072160 2023-11-24T08:23:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.822+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c401585b0 con 0x7f5c40072160 2023-11-24T08:23:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.824+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 46) v1 ==== 92393+0+0 (secure 0 0 0) 0x7f5c3001e460 con 0x7f5c40072160 2023-11-24T08:23:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.824+0000 7f5c36ffd640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 0x7f5c2406f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.825+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c30088f10 con 0x7f5c40072160 2023-11-24T08:23:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.825+0000 7f5c46050640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 0x7f5c2406f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.825+0000 7f5c46050640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 0x7f5c2406f5e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c036000 tx=0x7f5c3c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:08.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:08.829+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c401585b0 con 0x7f5c40072160 2023-11-24T08:23:09.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:09.054+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol4247", "snap_name": "subvol_snap4594", "target_sub_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c400c5770 con 0x7f5c2406d120 2023-11-24T08:23:10.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:10.212+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 47) v1 ==== 92495+0+0 (secure 0 0 0) 0x7f5c30055e70 con 0x7f5c40072160 2023-11-24T08:23:11.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.573+0000 7f5c36ffd640 1 -- 172.21.15.107:0/3708994223 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c3c031b00 con 0x7f5c2406d120 2023-11-24T08:23:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 msgr2=0x7f5c2406f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 0x7f5c2406f5e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c036000 tx=0x7f5c3c008030 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 msgr2=0x7f5c40143d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000d830 tx=0x7f5c3000dcf0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 shutdown_connections 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5c2406d120 0x7f5c2406f5e0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c400c49f0 0x7f5c40158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c40072160 0x7f5c40143d50 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 --2- 172.21.15.107:0/3708994223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c4006dc40 0x7f5c4015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 >> 172.21.15.107:0/3708994223 conn(0x7f5c400b0c50 msgr2=0x7f5c4006cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 shutdown_connections 2023-11-24T08:23:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.577+0000 7f5c482b4640 1 -- 172.21.15.107:0/3708994223 wait complete. 2023-11-24T08:23:11.586 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.585+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.586 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.585+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.586 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.585+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.586 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.585+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.586 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.585+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7352 2023-11-24T08:23:11.607 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.606+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.607 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.606+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.607 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.606+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.607 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.606+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.607 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:23:11.606+0000 7f3767fa7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:23:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.791+0000 7f9950a21640 1 Processor -- start 2023-11-24T08:23:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.791+0000 7f9950a21640 1 -- start start 2023-11-24T08:23:11.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.792+0000 7f9950a21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c15a1c0 0x7f994c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.792+0000 7f9950a21640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f994c0b0b40 0x7f994c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.792+0000 7f9950a21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c06d430 con 0x7f994c15ded0 2023-11-24T08:23:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.792+0000 7f9950a21640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c15ea70 con 0x7f994c0b0b40 2023-11-24T08:23:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.792+0000 7f9950a21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15ec50 con 0x7f994c15a1c0 2023-11-24T08:23:11.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f994c0b0b40 0x7f994c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f994c0b0b40 0x7f994c0b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50074/0 (socket says 172.21.15.107:50074) 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 -- 172.21.15.107:0/2188012717 learned_addr learned my addr 172.21.15.107:0/2188012717 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 -- 172.21.15.107:0/2188012717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c15a1c0 msgr2=0x7f994c06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 --2- 172.21.15.107:0/2188012717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c15a1c0 0x7f994c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 -- 172.21.15.107:0/2188012717 >> v1:172.21.15.107:6789/0 conn(0x7f994c15ded0 legacy=0x7f994c15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.793+0000 7f9943fff640 1 -- 172.21.15.107:0/2188012717 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0b14e0 con 0x7f994c0b0b40 2023-11-24T08:23:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.794+0000 7f9943fff640 1 --2- 172.21.15.107:0/2188012717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c0b0b40 0x7f994c0b0fa0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f9934009870 tx=0x7f993402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3194b29b6fb403 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f994affd640 1 -- 172.21.15.107:0/2188012717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934030840 con 0x7f994c0b0b40 2023-11-24T08:23:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f994affd640 1 -- 172.21.15.107:0/2188012717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99340309a0 con 0x7f994c0b0b40 2023-11-24T08:23:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f994affd640 1 -- 172.21.15.107:0/2188012717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934030d30 con 0x7f994c0b0b40 2023-11-24T08:23:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 -- 172.21.15.107:0/2188012717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c0b0b40 msgr2=0x7f994c0b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 --2- 172.21.15.107:0/2188012717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c0b0b40 0x7f994c0b0fa0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f9934009870 tx=0x7f993402b980 comp rx=0 tx=0).stop 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 -- 172.21.15.107:0/2188012717 shutdown_connections 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 --2- 172.21.15.107:0/2188012717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c0b0b40 0x7f994c0b0fa0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 --2- 172.21.15.107:0/2188012717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c15a1c0 0x7f994c06d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 -- 172.21.15.107:0/2188012717 >> 172.21.15.107:0/2188012717 conn(0x7f994c0cbb40 msgr2=0x7f994c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:11.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 -- 172.21.15.107:0/2188012717 shutdown_connections 2023-11-24T08:23:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.795+0000 7f9950a21640 1 -- 172.21.15.107:0/2188012717 wait complete. 2023-11-24T08:23:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.797+0000 7f9950a21640 1 Processor -- start 2023-11-24T08:23:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.797+0000 7f9950a21640 1 -- start start 2023-11-24T08:23:11.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 0x7f994c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c15ded0 0x7f994c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0c6500 con 0x7f994c15a1c0 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0c67e0 con 0x7f994c15ded0 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.798+0000 7f9950a21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0c6ac0 con 0x7f994c0b0b40 2023-11-24T08:23:11.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50086/0 (socket says 172.21.15.107:50086) 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 -- 172.21.15.107:0/3712567779 learned_addr learned my addr 172.21.15.107:0/3712567779 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f994b7fe640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 0x7f994c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 -- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 msgr2=0x7f994c159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 0x7f994c159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 -- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c15ded0 msgr2=0x7f994c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f994bfff640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c15ded0 0x7f994c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c15ded0 0x7f994c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 -- 172.21.15.107:0/3712567779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340094d0 con 0x7f994c15a1c0 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f994b7fe640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 0x7f994c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.799+0000 7f9943fff640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f9934004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.800+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934030810 con 0x7f994c15a1c0 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.800+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934043400 con 0x7f994c15a1c0 2023-11-24T08:23:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.800+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c0c6da0 con 0x7f994c15a1c0 2023-11-24T08:23:11.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.800+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99340437d0 con 0x7f994c15a1c0 2023-11-24T08:23:11.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.800+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c0c5790 con 0x7f994c15a1c0 2023-11-24T08:23:11.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.801+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0c6f30 con 0x7f994c15a1c0 2023-11-24T08:23:11.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.802+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 47) v1 ==== 92495+0+0 (secure 0 0 0) 0x7f9934043930 con 0x7f994c15a1c0 2023-11-24T08:23:11.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.803+0000 7f9948ff9640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 0x7f992c06f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:11.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.803+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f993403e070 con 0x7f994c15a1c0 2023-11-24T08:23:11.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.803+0000 7f994b7fe640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 0x7f992c06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:11.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.804+0000 7f994b7fe640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 0x7f992c06f700 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f993c006fd0 tx=0x7f993c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:11.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:11.807+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99340786f0 con 0x7f994c15a1c0 2023-11-24T08:23:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.036+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f994c06d0c0 con 0x7f992c06d240 2023-11-24T08:23:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.053+0000 7f9948ff9640 1 -- 172.21.15.107:0/3712567779 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f993c00ae20 con 0x7f992c06d240 2023-11-24T08:23:12.055 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:12.055 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4247", 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap4594" 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:12.056 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:12.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 msgr2=0x7f992c06f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:12.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 0x7f992c06f700 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f993c006fd0 tx=0x7f993c008040 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 msgr2=0x7f994c15c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f9934004730 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 shutdown_connections 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f992c06d240 0x7f992c06f700 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c15ded0 0x7f994c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f994c15a1c0 0x7f994c15c770 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 --2- 172.21.15.107:0/3712567779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c0b0b40 0x7f994c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 >> 172.21.15.107:0/3712567779 conn(0x7f994c0cbb40 msgr2=0x7f994c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 shutdown_connections 2023-11-24T08:23:12.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:12.056+0000 7f9950a21640 1 -- 172.21.15.107:0/3712567779 wait complete. 2023-11-24T08:23:13.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7352 2023-11-24T08:23:13.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.301+0000 7f9c8fa3d640 1 Processor -- start 2023-11-24T08:23:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.302+0000 7f9c8fa3d640 1 -- start start 2023-11-24T08:23:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.303+0000 7f9c8fa3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806b780 0x7f9c88161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.303+0000 7f9c8fa3d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c8806bb60 0x7f9c880adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.303+0000 7f9c8fa3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880ac840 con 0x7f9c8806e950 2023-11-24T08:23:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.303+0000 7f9c8fa3d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c88162100 con 0x7f9c8806bb60 2023-11-24T08:23:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.303+0000 7f9c8fa3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c880ad6a0 con 0x7f9c8806b780 2023-11-24T08:23:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c8806bb60 0x7f9c880adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c8806bb60 0x7f9c880adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50092/0 (socket says 172.21.15.107:50092) 2023-11-24T08:23:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 -- 172.21.15.107:0/4160532227 learned_addr learned my addr 172.21.15.107:0/4160532227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8d7d9640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806b780 0x7f9c88161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 -- 172.21.15.107:0/4160532227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806b780 msgr2=0x7f9c88161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806b780 0x7f9c88161bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 -- 172.21.15.107:0/4160532227 >> v1:172.21.15.120:6789/0 conn(0x7f9c8806e950 legacy=0x7f9c880b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.304+0000 7f9c8dfda640 1 -- 172.21.15.107:0/4160532227 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880aca90 con 0x7f9c8806bb60 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.305+0000 7f9c8dfda640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806bb60 0x7f9c880adb30 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009870 tx=0x7f9c8002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d3e63fc0145800 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.305+0000 7f9c87fff640 1 -- 172.21.15.107:0/4160532227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80031640 con 0x7f9c8806bb60 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c87fff640 1 -- 172.21.15.107:0/4160532227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c800317a0 con 0x7f9c8806bb60 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c87fff640 1 -- 172.21.15.107:0/4160532227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80031b30 con 0x7f9c8806bb60 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/4160532227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806bb60 msgr2=0x7f9c880adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806bb60 0x7f9c880adb30 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009870 tx=0x7f9c8002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/4160532227 shutdown_connections 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806bb60 0x7f9c880adb30 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/4160532227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806b780 0x7f9c88161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/4160532227 >> 172.21.15.107:0/4160532227 conn(0x7f9c88070fa0 msgr2=0x7f9c880713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/4160532227 shutdown_connections 2023-11-24T08:23:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.306+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/4160532227 wait complete. 2023-11-24T08:23:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.308+0000 7f9c8fa3d640 1 Processor -- start 2023-11-24T08:23:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.309+0000 7f9c8fa3d640 1 -- start start 2023-11-24T08:23:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806bb60 0x7f9c8815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 0x7f9c8815c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88159bd0 con 0x7f9c8806b780 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88159eb0 con 0x7f9c8806e950 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8fa3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8815a190 con 0x7f9c8806bb60 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8d7d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8d7d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50100/0 (socket says 172.21.15.107:50100) 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8d7d9640 1 -- 172.21.15.107:0/1852704663 learned_addr learned my addr 172.21.15.107:0/1852704663 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.310+0000 7f9c8cfd8640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806bb60 0x7f9c8815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8dfda640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 0x7f9c8815c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 -- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806bb60 msgr2=0x7f9c8815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806bb60 0x7f9c8815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 -- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 msgr2=0x7f9c8815c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 0x7f9c8815c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 -- 172.21.15.107:0/1852704663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c800094d0 con 0x7f9c8806b780 2023-11-24T08:23:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8dfda640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 0x7f9c8815c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8d7d9640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800ce10 tx=0x7f9c7800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78003b90 con 0x7f9c8806b780 2023-11-24T08:23:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c78003cf0 con 0x7f9c8806b780 2023-11-24T08:23:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78003ea0 con 0x7f9c8806b780 2023-11-24T08:23:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.311+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8815a470 con 0x7f9c8806b780 2023-11-24T08:23:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.312+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c88154a70 con 0x7f9c8806b780 2023-11-24T08:23:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.313+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8815a990 con 0x7f9c8806b780 2023-11-24T08:23:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.314+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f9c78004240 con 0x7f9c8806b780 2023-11-24T08:23:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.315+0000 7f9c85ffb640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 0x7f9c7406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.315+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c78012070 con 0x7f9c8806b780 2023-11-24T08:23:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.315+0000 7f9c8cfd8640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 0x7f9c7406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.316+0000 7f9c8cfd8640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 0x7f9c7406f820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c006fd0 tx=0x7f9c7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.319+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c78055e10 con 0x7f9c8806b780 2023-11-24T08:23:13.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.545+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c880ad330 con 0x7f9c7406d360 2023-11-24T08:23:13.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.555+0000 7f9c85ffb640 1 -- 172.21.15.107:0/1852704663 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+163 (secure 0 0 0) 0x7f9c7c00bd80 con 0x7f9c7406d360 2023-11-24T08:23:13.557 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:13.557 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol4247", 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap4594" 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:13.558 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.557+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 msgr2=0x7f9c7406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.557+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 0x7f9c7406f820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c006fd0 tx=0x7f9c7c008040 comp rx=0 tx=0).stop 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.557+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 msgr2=0x7f9c88154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.557+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800ce10 tx=0x7f9c7800a660 comp rx=0 tx=0).stop 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 shutdown_connections 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9c7406d360 0x7f9c7406f820 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c8806e950 0x7f9c8815c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c8806bb60 0x7f9c8815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 --2- 172.21.15.107:0/1852704663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c8806b780 0x7f9c88154530 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:13.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 >> 172.21.15.107:0/1852704663 conn(0x7f9c88070fa0 msgr2=0x7f9c880b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:13.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 shutdown_connections 2023-11-24T08:23:13.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:13.558+0000 7f9c8fa3d640 1 -- 172.21.15.107:0/1852704663 wait complete. 2023-11-24T08:23:14.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7352 2023-11-24T08:23:14.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.804+0000 7f20de7ec640 1 Processor -- start 2023-11-24T08:23:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.805+0000 7f20de7ec640 1 -- start start 2023-11-24T08:23:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.806+0000 7f20de7ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20d806de10 0x7f20d815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.806+0000 7f20de7ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d815ec90 con 0x7f20d80c49f0 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.806+0000 7f20de7ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d80cd530 con 0x7f20d8072160 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.806+0000 7f20de7ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d80c4ed0 con 0x7f20d806de10 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20d806de10 0x7f20d815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20d806de10 0x7f20d815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50104/0 (socket says 172.21.15.107:50104) 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 -- 172.21.15.107:0/3359228585 learned_addr learned my addr 172.21.15.107:0/3359228585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20dcd89640 1 --1- 172.21.15.107:0/3359228585 >> v1:172.21.15.107:6789/0 conn(0x7f20d80c49f0 0x7f20d80705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33410/0 (socket says 172.21.15.107:33410) 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 -- 172.21.15.107:0/3359228585 >> v1:172.21.15.120:6789/0 conn(0x7f20d8072160 legacy=0x7f20d806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 -- 172.21.15.107:0/3359228585 >> v1:172.21.15.107:6789/0 conn(0x7f20d80c49f0 legacy=0x7f20d80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:14.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.807+0000 7f20cffff640 1 -- 172.21.15.107:0/3359228585 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d80c51b0 con 0x7f20d806de10 2023-11-24T08:23:14.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.808+0000 7f20cffff640 1 --2- 172.21.15.107:0/3359228585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d806de10 0x7f20d815e750 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f20c000bb10 tx=0x7f20c002e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a987f27b928a3af server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:14.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.809+0000 7f20d77fe640 1 -- 172.21.15.107:0/3359228585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0036440 con 0x7f20d806de10 2023-11-24T08:23:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.809+0000 7f20d77fe640 1 -- 172.21.15.107:0/3359228585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c00365a0 con 0x7f20d806de10 2023-11-24T08:23:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.809+0000 7f20d77fe640 1 -- 172.21.15.107:0/3359228585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0036930 con 0x7f20d806de10 2023-11-24T08:23:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.809+0000 7f20de7ec640 1 -- 172.21.15.107:0/3359228585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d806de10 msgr2=0x7f20d815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.809+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3359228585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d806de10 0x7f20d815e750 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f20c000bb10 tx=0x7f20c002e160 comp rx=0 tx=0).stop 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.810+0000 7f20de7ec640 1 -- 172.21.15.107:0/3359228585 shutdown_connections 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.810+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3359228585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d806de10 0x7f20d815e750 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.810+0000 7f20de7ec640 1 -- 172.21.15.107:0/3359228585 >> 172.21.15.107:0/3359228585 conn(0x7f20d80b0c50 msgr2=0x7f20d80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.810+0000 7f20de7ec640 1 -- 172.21.15.107:0/3359228585 shutdown_connections 2023-11-24T08:23:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.810+0000 7f20de7ec640 1 -- 172.21.15.107:0/3359228585 wait complete. 2023-11-24T08:23:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.811+0000 7f20de7ec640 1 Processor -- start 2023-11-24T08:23:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.812+0000 7f20de7ec640 1 -- start start 2023-11-24T08:23:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.812+0000 7f20de7ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d806de10 0x7f20d8158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.812+0000 7f20de7ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 0x7f20d815b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20de7ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20de7ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8156890 con 0x7f20d80c49f0 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20de7ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d809bc90 con 0x7f20d8072160 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20de7ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d809bf70 con 0x7f20d806de10 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50116/0 (socket says 172.21.15.107:50116) 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 -- 172.21.15.107:0/3687693391 learned_addr learned my addr 172.21.15.107:0/3687693391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20d7fff640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d806de10 0x7f20d8158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20cffff640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 0x7f20d815b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 -- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d806de10 msgr2=0x7f20d8158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d806de10 0x7f20d8158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 -- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 msgr2=0x7f20d815b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 0x7f20d815b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20dcd89640 1 -- 172.21.15.107:0/3687693391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c000b770 con 0x7f20d80c49f0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.813+0000 7f20cffff640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 0x7f20d815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20dcd89640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f20d000c6c0 tx=0x7f20d000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0004d60 con 0x7f20d80c49f0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d000a640 con 0x7f20d80c49f0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d000a9b0 con 0x7f20d80c49f0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d809c250 con 0x7f20d80c49f0 2023-11-24T08:23:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.814+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d815caa0 con 0x7f20d80c49f0 2023-11-24T08:23:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.815+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d809c770 con 0x7f20d80c49f0 2023-11-24T08:23:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.816+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f20d000ab10 con 0x7f20d80c49f0 2023-11-24T08:23:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.817+0000 7f20d57fa640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 0x7f20b806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.817+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20d00879e0 con 0x7f20d80c49f0 2023-11-24T08:23:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.817+0000 7f20d7fff640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 0x7f20b806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.818+0000 7f20d7fff640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 0x7f20b806f820 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f20c8004ea0 tx=0x7f20c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:14.822+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20d809c770 con 0x7f20d80c49f0 2023-11-24T08:23:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.047+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d80c5a50 con 0x7f20b806d360 2023-11-24T08:23:15.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.531+0000 7f20d57fa640 1 -- 172.21.15.107:0/3687693391 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f20c8007db0 con 0x7f20b806d360 2023-11-24T08:23:15.533 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:15.533 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:23:15.534 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:23:15.534 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:15.534 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:15.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 msgr2=0x7f20b806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 0x7f20b806f820 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f20c8004ea0 tx=0x7f20c8009040 comp rx=0 tx=0).stop 2023-11-24T08:23:15.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 msgr2=0x7f20d815baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f20d000c6c0 tx=0x7f20d000cb80 comp rx=0 tx=0).stop 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 shutdown_connections 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20b806d360 0x7f20b806f820 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d80c49f0 0x7f20d815baf0 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d8072160 0x7f20d815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 --2- 172.21.15.107:0/3687693391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d806de10 0x7f20d8158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 >> 172.21.15.107:0/3687693391 conn(0x7f20d80b0c50 msgr2=0x7f20d80abfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:15.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 shutdown_connections 2023-11-24T08:23:15.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.534+0000 7f20de7ec640 1 -- 172.21.15.107:0/3687693391 wait complete. 2023-11-24T08:23:15.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone7352 2023-11-24T08:23:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.776+0000 7f1f6c6e7640 1 Processor -- start 2023-11-24T08:23:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.777+0000 7f1f6c6e7640 1 -- start start 2023-11-24T08:23:15.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f6c6e7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f6415d720 0x7f1f640d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f6c6e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f640d3870 0x7f1f640b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f6c6e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6415db00 con 0x7f1f640b1400 2023-11-24T08:23:15.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f6c6e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f640d3d20 con 0x7f1f640d3870 2023-11-24T08:23:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f6c6e7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f640d0ce0 con 0x7f1f6415d720 2023-11-24T08:23:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.778+0000 7f1f69c82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f640d3870 0x7f1f640b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f640d3870 0x7f1f640b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50128/0 (socket says 172.21.15.107:50128) 2023-11-24T08:23:15.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 -- 172.21.15.107:0/3601634992 learned_addr learned my addr 172.21.15.107:0/3601634992 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f6a483640 1 --2- 172.21.15.107:0/3601634992 >> v2:172.21.15.120:3300/0 conn(0x7f1f6415d720 0x7f1f640d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 -- 172.21.15.107:0/3601634992 >> v2:172.21.15.120:3300/0 conn(0x7f1f6415d720 msgr2=0x7f1f640d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 --2- 172.21.15.107:0/3601634992 >> v2:172.21.15.120:3300/0 conn(0x7f1f6415d720 0x7f1f640d3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69481640 1 -- 172.21.15.107:0/3601634992 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257664194 0 0) 0x7f1f6415db00 con 0x7f1f640b1400 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 -- 172.21.15.107:0/3601634992 >> v1:172.21.15.107:6789/0 conn(0x7f1f640b1400 legacy=0x7f1f640b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.779+0000 7f1f69c82640 1 -- 172.21.15.107:0/3601634992 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6406d400 con 0x7f1f640d3870 2023-11-24T08:23:15.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f69c82640 1 --2- 172.21.15.107:0/3601634992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640d3870 0x7f1f640b0ec0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f1f6408b890 tx=0x7f1f5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8e8b25fc6a118c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:15.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f69481640 1 -- 172.21.15.107:0/3601634992 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54031870 con 0x7f1f640d3870 2023-11-24T08:23:15.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f69481640 1 -- 172.21.15.107:0/3601634992 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f540319d0 con 0x7f1f640d3870 2023-11-24T08:23:15.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f69481640 1 -- 172.21.15.107:0/3601634992 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54031d60 con 0x7f1f640d3870 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/3601634992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640d3870 msgr2=0x7f1f640b0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.781+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/3601634992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640d3870 0x7f1f640b0ec0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f1f6408b890 tx=0x7f1f5402b910 comp rx=0 tx=0).stop 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/3601634992 shutdown_connections 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/3601634992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640d3870 0x7f1f640b0ec0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/3601634992 >> v2:172.21.15.120:3300/0 conn(0x7f1f6415d720 0x7f1f640d3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/3601634992 >> 172.21.15.107:0/3601634992 conn(0x7f1f6408af60 msgr2=0x7f1f6409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/3601634992 shutdown_connections 2023-11-24T08:23:15.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.782+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/3601634992 wait complete. 2023-11-24T08:23:15.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.784+0000 7f1f6c6e7640 1 Processor -- start 2023-11-24T08:23:15.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.784+0000 7f1f6c6e7640 1 -- start start 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f640d3870 0x7f1f640cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f6415d720 0x7f1f640c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640caf10 con 0x7f1f640b1400 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640cb1f0 con 0x7f1f640d3870 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6c6e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640cb4d0 con 0x7f1f6415d720 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50136/0 (socket says 172.21.15.107:50136) 2023-11-24T08:23:15.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 -- 172.21.15.107:0/1104286228 learned_addr learned my addr 172.21.15.107:0/1104286228 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:15.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 -- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f6415d720 msgr2=0x7f1f640c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f6415d720 0x7f1f640c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 -- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f640d3870 msgr2=0x7f1f640cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f640d3870 0x7f1f640cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.785+0000 7f1f6a483640 1 -- 172.21.15.107:0/1104286228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540094d0 con 0x7f1f640b1400 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f6a483640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000efc0 tx=0x7f1f6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60014480 con 0x7f1f640b1400 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6000f040 con 0x7f1f640b1400 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60014800 con 0x7f1f640b1400 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f640cb7b0 con 0x7f1f640b1400 2023-11-24T08:23:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.786+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f640cd940 con 0x7f1f640b1400 2023-11-24T08:23:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.787+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f640cc4f0 con 0x7f1f640b1400 2023-11-24T08:23:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.788+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f1f60014960 con 0x7f1f640b1400 2023-11-24T08:23:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.789+0000 7f1f5affd640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 0x7f1f4806f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.789+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f60089480 con 0x7f1f640b1400 2023-11-24T08:23:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.790+0000 7f1f69c82640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 0x7f1f4806f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:15.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.790+0000 7f1f69c82640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 0x7f1f4806f870 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:15.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:15.794+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f60056010 con 0x7f1f640b1400 2023-11-24T08:23:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.014+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f6415d2f0 con 0x7f1f4806d3b0 2023-11-24T08:23:16.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.023+0000 7f1f5affd640 1 -- 172.21.15.107:0/1104286228 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f1f6415d2f0 con 0x7f1f4806d3b0 2023-11-24T08:23:16.025 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06 2023-11-24T08:23:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 msgr2=0x7f1f4806f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 0x7f1f4806f870 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).stop 2023-11-24T08:23:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 msgr2=0x7f1f640cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000efc0 tx=0x7f1f6000c490 comp rx=0 tx=0).stop 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 shutdown_connections 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1f4806d3b0 0x7f1f4806f870 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f6415d720 0x7f1f640c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f640d3870 0x7f1f640cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 --2- 172.21.15.107:0/1104286228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f640b1400 0x7f1f640cbfb0 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 >> 172.21.15.107:0/1104286228 conn(0x7f1f6408af60 msgr2=0x7f1f640cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 shutdown_connections 2023-11-24T08:23:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.026+0000 7f1f6c6e7640 1 -- 172.21.15.107:0/1104286228 wait complete. 2023-11-24T08:23:16.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone7352 2023-11-24T08:23:16.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.265+0000 7feeeabcb640 1 Processor -- start 2023-11-24T08:23:16.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.266+0000 7feeeabcb640 1 -- start start 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeeabcb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee40b0b40 0x7feee40b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeeabcb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feee406c5b0 0x7feee406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeeabcb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee415ed20 con 0x7feee415b260 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeeabcb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee415b640 con 0x7feee406c5b0 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeeabcb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee415f8a0 con 0x7feee40b0b40 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feee9bc9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feee415b260 0x7feee406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33428/0 (socket says 172.21.15.107:33428) 2023-11-24T08:23:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feee9bc9640 1 -- 172.21.15.107:0/3923896999 learned_addr learned my addr 172.21.15.107:0/3923896999 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.267+0000 7feeea3ca640 1 --2- 172.21.15.107:0/3923896999 >> v2:172.21.15.107:3300/0 conn(0x7feee406c5b0 0x7feee406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.268+0000 7feee93c8640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee40b0b40 0x7feee40b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.268+0000 7feeea3ca640 1 -- 172.21.15.107:0/3923896999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee40b0b40 msgr2=0x7feee40b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.268+0000 7feeea3ca640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee40b0b40 0x7feee40b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.268+0000 7feeea3ca640 1 -- 172.21.15.107:0/3923896999 >> v1:172.21.15.107:6789/0 conn(0x7feee415b260 legacy=0x7feee406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.268+0000 7feeea3ca640 1 -- 172.21.15.107:0/3923896999 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40b14c0 con 0x7feee406c5b0 2023-11-24T08:23:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.269+0000 7feeea3ca640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee406c5b0 0x7feee406ca40 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7feedc009d90 tx=0x7feedc02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1fee0eee0d3d48 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.270+0000 7feee8bc7640 1 -- 172.21.15.107:0/3923896999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc02cb40 con 0x7feee406c5b0 2023-11-24T08:23:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.270+0000 7feee8bc7640 1 -- 172.21.15.107:0/3923896999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc02cca0 con 0x7feee406c5b0 2023-11-24T08:23:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.270+0000 7feee8bc7640 1 -- 172.21.15.107:0/3923896999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc034660 con 0x7feee406c5b0 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.270+0000 7feeeabcb640 1 -- 172.21.15.107:0/3923896999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee406c5b0 msgr2=0x7feee406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.270+0000 7feeeabcb640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee406c5b0 0x7feee406ca40 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7feedc009d90 tx=0x7feedc02bd90 comp rx=0 tx=0).stop 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 -- 172.21.15.107:0/3923896999 shutdown_connections 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee406c5b0 0x7feee406ca40 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 --2- 172.21.15.107:0/3923896999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee40b0b40 0x7feee40b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 -- 172.21.15.107:0/3923896999 >> 172.21.15.107:0/3923896999 conn(0x7feee40cbb40 msgr2=0x7feee40cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 -- 172.21.15.107:0/3923896999 shutdown_connections 2023-11-24T08:23:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.271+0000 7feeeabcb640 1 -- 172.21.15.107:0/3923896999 wait complete. 2023-11-24T08:23:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.273+0000 7feeeabcb640 1 Processor -- start 2023-11-24T08:23:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.273+0000 7feeeabcb640 1 -- start start 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feee406c5b0 0x7feee414f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee415b260 0x7feee40c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee415cee0 con 0x7feee40b0b40 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee40caa00 con 0x7feee406c5b0 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeeabcb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee40cace0 con 0x7feee415b260 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50164/0 (socket says 172.21.15.107:50164) 2023-11-24T08:23:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 -- 172.21.15.107:0/4080823929 learned_addr learned my addr 172.21.15.107:0/4080823929 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:16.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feeea3ca640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee415b260 0x7feee40c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 -- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee415b260 msgr2=0x7feee40c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee415b260 0x7feee40c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee9bc9640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feee406c5b0 0x7feee414f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 -- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feee406c5b0 msgr2=0x7feee414f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.275+0000 7feee93c8640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feee406c5b0 0x7feee414f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feee93c8640 1 -- 172.21.15.107:0/4080823929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc009a70 con 0x7feee40b0b40 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feee93c8640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7feed400d5f0 tx=0x7feed400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40125f0 con 0x7feee40b0b40 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4012750 con 0x7feee40b0b40 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4012b00 con 0x7feee40b0b40 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee40cafc0 con 0x7feee40b0b40 2023-11-24T08:23:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.276+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee415a800 con 0x7feee40b0b40 2023-11-24T08:23:16.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.278+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee404e840 con 0x7feee40b0b40 2023-11-24T08:23:16.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.279+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7feed4012c60 con 0x7feee40b0b40 2023-11-24T08:23:16.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.279+0000 7feeda7fc640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 0x7feec806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.280+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feed4088d50 con 0x7feee40b0b40 2023-11-24T08:23:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.280+0000 7feee9bc9640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 0x7feec806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.280+0000 7feee9bc9640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 0x7feec806f820 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7feee0005880 tx=0x7feee0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:16.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.284+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feed40558e0 con 0x7feee40b0b40 2023-11-24T08:23:16.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:23:16.324 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:23:16.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.508+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7feee415aa20 con 0x7feec806d360 2023-11-24T08:23:16.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.522+0000 7feeda7fc640 1 -- 172.21.15.107:0/4080823929 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7feee415aa20 con 0x7feec806d360 2023-11-24T08:23:16.524 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:16.524 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:23:04", 2023-11-24T08:23:16.524 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:23:16.524 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:23:16.525 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 16777216, 2023-11-24T08:23:16.525 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:23:11", 2023-11-24T08:23:16.525 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:23:14", 2023-11-24T08:23:16.525 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:23:16.526 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:23:16.526 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:23:16.526 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:23:16.526 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:23:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:23:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:23:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:23:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:23:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:23:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:23:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:23:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:23:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:23:07", 2023-11-24T08:23:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06", 2023-11-24T08:23:16.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:23:16.529 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:23:16.529 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:23:16.529 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:23:16.529 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 msgr2=0x7feec806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 0x7feec806f820 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7feee0005880 tx=0x7feee0009040 comp rx=0 tx=0).stop 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 msgr2=0x7feee415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7feed400d5f0 tx=0x7feed400dab0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 shutdown_connections 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feec806d360 0x7feec806f820 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feee415b260 0x7feee40c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee40b0b40 0x7feee415c500 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 --2- 172.21.15.107:0/4080823929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feee406c5b0 0x7feee414f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 >> 172.21.15.107:0/4080823929 conn(0x7feee40cbb40 msgr2=0x7feee40add10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 shutdown_connections 2023-11-24T08:23:16.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:16.525+0000 7feeeabcb640 1 -- 172.21.15.107:0/4080823929 wait complete. 2023-11-24T08:23:16.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd: ceph.quota.max_bytes: No such attribute 2023-11-24T08:23:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:23:16.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.652 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:23:16.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.703 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10 2023-11-24T08:23:16.757 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3 2023-11-24T08:23:16.758 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12 2023-11-24T08:23:16.759 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1 2023-11-24T08:23:16.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.760 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.813 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:23:16.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.814 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/) 2023-11-24T08:23:16.862 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:23:16.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:16.911 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:23:16.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/) 2023-11-24T08:23:16.960 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:23:16.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:16.961 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:17.009 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:23:17.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.010 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/) 2023-11-24T08:23:17.061 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:23:17.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd) 2023-11-24T08:23:17.116 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:17.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.118 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/) 2023-11-24T08:23:17.174 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:17.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.175 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2) 2023-11-24T08:23:17.227 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:17.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.2) 2023-11-24T08:23:17.288 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:17.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.289 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2) 2023-11-24T08:23:17.338 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:17.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.340 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.2) 2023-11-24T08:23:17.389 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:17.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2) 2023-11-24T08:23:17.443 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:17.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.2) 2023-11-24T08:23:17.494 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:17.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.496 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.2) 2023-11-24T08:23:17.547 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:17.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.2) 2023-11-24T08:23:17.601 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:17.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.602 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14) 2023-11-24T08:23:17.654 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:17.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.14) 2023-11-24T08:23:17.707 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:17.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14) 2023-11-24T08:23:17.754 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:17.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.755 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.14) 2023-11-24T08:23:17.855 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:17.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.856 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14) 2023-11-24T08:23:17.911 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:17.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.912 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.14) 2023-11-24T08:23:17.967 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:17.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:17.968 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.14) 2023-11-24T08:23:18.022 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:18.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.023 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.14) 2023-11-24T08:23:18.076 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:18.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.078 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13) 2023-11-24T08:23:18.134 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:18.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.135 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.13) 2023-11-24T08:23:18.192 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:18.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.194 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13) 2023-11-24T08:23:18.244 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:18.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.245 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.13) 2023-11-24T08:23:18.295 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:18.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.296 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13) 2023-11-24T08:23:18.348 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:18.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.349 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.13) 2023-11-24T08:23:18.401 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:18.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.402 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.13) 2023-11-24T08:23:18.454 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:18.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.455 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.13) 2023-11-24T08:23:18.508 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:18.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.509 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10) 2023-11-24T08:23:18.562 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:18.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.563 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.10) 2023-11-24T08:23:18.619 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:18.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.621 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10) 2023-11-24T08:23:18.674 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:18.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.676 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.10) 2023-11-24T08:23:18.729 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:18.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.731 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10) 2023-11-24T08:23:18.783 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:18.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.784 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.10) 2023-11-24T08:23:18.836 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:18.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.838 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.10) 2023-11-24T08:23:18.888 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:18.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.889 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.10) 2023-11-24T08:23:18.943 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:18.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:18.944 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5) 2023-11-24T08:23:18.999 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.5) 2023-11-24T08:23:19.057 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.059 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5) 2023-11-24T08:23:19.112 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:19.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.114 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.5) 2023-11-24T08:23:19.167 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:19.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5) 2023-11-24T08:23:19.224 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:19.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.226 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.5) 2023-11-24T08:23:19.278 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:19.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.280 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.5) 2023-11-24T08:23:19.331 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:19.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.332 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.5) 2023-11-24T08:23:19.384 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:19.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4) 2023-11-24T08:23:19.437 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.4) 2023-11-24T08:23:19.492 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.494 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4) 2023-11-24T08:23:19.546 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:19.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.548 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.4) 2023-11-24T08:23:19.599 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:19.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4) 2023-11-24T08:23:19.648 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:19.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.649 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.4) 2023-11-24T08:23:19.699 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:19.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.4) 2023-11-24T08:23:19.751 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:19.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.752 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.4) 2023-11-24T08:23:19.802 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:19.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.804 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7) 2023-11-24T08:23:19.854 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.855 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.7) 2023-11-24T08:23:19.910 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:19.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.911 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7) 2023-11-24T08:23:19.961 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:19.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:19.962 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.7) 2023-11-24T08:23:20.014 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:20.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.016 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7) 2023-11-24T08:23:20.067 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:20.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.068 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.7) 2023-11-24T08:23:20.122 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:20.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.124 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.7) 2023-11-24T08:23:20.176 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:20.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.177 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.7) 2023-11-24T08:23:20.229 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:20.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.230 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6) 2023-11-24T08:23:20.283 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:20.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.285 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.6) 2023-11-24T08:23:20.340 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:20.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6) 2023-11-24T08:23:20.392 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:20.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.6) 2023-11-24T08:23:20.441 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:20.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6) 2023-11-24T08:23:20.492 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:20.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.493 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.6) 2023-11-24T08:23:20.544 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:20.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.546 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.6) 2023-11-24T08:23:20.592 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:20.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.594 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.6) 2023-11-24T08:23:20.694 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:20.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.695 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0) 2023-11-24T08:23:20.749 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:20.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.751 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.0) 2023-11-24T08:23:20.808 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:20.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.810 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0) 2023-11-24T08:23:20.866 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:20.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.867 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.0) 2023-11-24T08:23:20.922 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:20.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0) 2023-11-24T08:23:20.975 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:20.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:20.976 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.0) 2023-11-24T08:23:21.030 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:21.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.031 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.0) 2023-11-24T08:23:21.084 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:21.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.085 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.0) 2023-11-24T08:23:21.136 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:21.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9) 2023-11-24T08:23:21.189 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:21.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.191 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.9) 2023-11-24T08:23:21.243 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:21.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.245 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9) 2023-11-24T08:23:21.292 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:21.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.293 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.9) 2023-11-24T08:23:21.345 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:21.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.346 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9) 2023-11-24T08:23:21.392 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:21.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.9) 2023-11-24T08:23:21.490 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:21.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.9) 2023-11-24T08:23:21.545 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:21.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.547 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.9) 2023-11-24T08:23:21.600 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:21.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15) 2023-11-24T08:23:21.654 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:21.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.656 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.15) 2023-11-24T08:23:21.714 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:21.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.714 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15) 2023-11-24T08:23:21.768 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:21.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.15) 2023-11-24T08:23:21.821 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:21.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.823 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15) 2023-11-24T08:23:21.877 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:21.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.879 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.15) 2023-11-24T08:23:21.930 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:21.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.931 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.15) 2023-11-24T08:23:21.982 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:21.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:21.983 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.15) 2023-11-24T08:23:22.035 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:22.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.036 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8) 2023-11-24T08:23:22.086 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:22.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.087 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.8) 2023-11-24T08:23:22.138 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:22.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.139 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8) 2023-11-24T08:23:22.191 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:22.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.192 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.8) 2023-11-24T08:23:22.242 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:22.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8) 2023-11-24T08:23:22.294 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:22.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.8) 2023-11-24T08:23:22.344 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:22.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.346 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.8) 2023-11-24T08:23:22.394 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:22.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.396 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.8) 2023-11-24T08:23:22.442 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:22.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11) 2023-11-24T08:23:22.542 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:22.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.543 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.11) 2023-11-24T08:23:22.591 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:22.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.593 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11) 2023-11-24T08:23:22.641 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:22.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.642 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.11) 2023-11-24T08:23:22.692 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:22.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.693 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11) 2023-11-24T08:23:22.743 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:22.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.744 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.11) 2023-11-24T08:23:22.796 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:22.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.797 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.11) 2023-11-24T08:23:22.845 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:22.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.846 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.11) 2023-11-24T08:23:22.891 INFO:teuthology.orchestra.run.smithi107.stdout:1700814186 2023-11-24T08:23:22.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.893 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3) 2023-11-24T08:23:22.993 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:22.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:22.995 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.3) 2023-11-24T08:23:23.052 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:23.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3) 2023-11-24T08:23:23.103 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:23.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.104 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.3) 2023-11-24T08:23:23.152 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:23.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.153 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3) 2023-11-24T08:23:23.253 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:23.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.254 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.3) 2023-11-24T08:23:23.307 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:23.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.308 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.3) 2023-11-24T08:23:23.359 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:23.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.360 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.3) 2023-11-24T08:23:23.412 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:23.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.413 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12) 2023-11-24T08:23:23.470 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:23.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.12) 2023-11-24T08:23:23.526 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:23.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12) 2023-11-24T08:23:23.582 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:23.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.584 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.12) 2023-11-24T08:23:23.637 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:23.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.638 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12) 2023-11-24T08:23:23.690 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:23.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.691 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.12) 2023-11-24T08:23:23.740 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:23.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.741 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.12) 2023-11-24T08:23:23.791 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:23.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.792 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.12) 2023-11-24T08:23:23.842 INFO:teuthology.orchestra.run.smithi107.stdout:1700814187 2023-11-24T08:23:23.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.844 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1) 2023-11-24T08:23:23.894 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:23.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.896 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.1) 2023-11-24T08:23:23.945 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:23:23.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.947 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1) 2023-11-24T08:23:23.993 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:23.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:23.994 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.1) 2023-11-24T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:23:24.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:24.043 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1) 2023-11-24T08:23:24.092 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:24.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:24.093 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.1) 2023-11-24T08:23:24.149 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:23:24.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:24.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol4247/.snap/subvol_snap4594/2049132d-fbcf-4852-82af-9fa0bbae89fd/subvolume_file.1) 2023-11-24T08:23:24.203 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:24.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:24.204 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone7352/24751e76-b7f9-4802-8f52-e2b2b9763a06/subvolume_file.1) 2023-11-24T08:23:24.251 INFO:teuthology.orchestra.run.smithi107.stdout:1700814185 2023-11-24T08:23:24.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol4247 subvol_snap4594 2023-11-24T08:23:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.457+0000 7fe1a5f35640 1 Processor -- start 2023-11-24T08:23:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.457+0000 7fe1a5f35640 1 -- start start 2023-11-24T08:23:24.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a5f35640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1a006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a5f35640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a01448f0 con 0x7fe1a0144510 2023-11-24T08:23:24.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a5f35640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a0144d80 con 0x7fe1a015edd0 2023-11-24T08:23:24.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a5f35640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015f2d0 con 0x7fe1a00b11f0 2023-11-24T08:23:24.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a4f33640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe1a0144510 0x7fe1a00b0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43434/0 (socket says 172.21.15.107:43434) 2023-11-24T08:23:24.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.459+0000 7fe1a4f33640 1 -- 172.21.15.107:0/3637767611 learned_addr learned my addr 172.21.15.107:0/3637767611 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737244322 0 0) 0x7fe1a01448f0 con 0x7fe1a0144510 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe188003520 con 0x7fe1a0144510 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1a006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367771749 0 0) 0x7fe188003520 con 0x7fe1a0144510 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 msgr2=0x7fe1a006d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1a006d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 >> v1:172.21.15.120:6789/0 conn(0x7fe1a015edd0 legacy=0x7fe1a00b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a015f5b0 con 0x7fe1a0144510 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe194001670 con 0x7fe1a0144510 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 >> v1:172.21.15.107:6789/0 conn(0x7fe1a0144510 legacy=0x7fe1a00b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1880039a0 0x7fe188003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe188005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015f2d0 con 0x7fe188004d00 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18800bb60 con 0x7fe1880039a0 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.460+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18800bd20 con 0x7fe1a00b11f0 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe188005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a4f33640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1880039a0 0x7fe188003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 -- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 msgr2=0x7fe1880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 -- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1880039a0 msgr2=0x7fe188003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1880039a0 0x7fe188003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe1a5734640 1 -- 172.21.15.107:0/3637767611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe188003520 con 0x7fe188004d00 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.461+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a00b11f0 0x7fe1880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.462+0000 7fe1a5734640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe188005180 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fe1a015f200 tx=0x7fe1880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5559f92436811033 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.462+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198003b70 con 0x7fe188004d00 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.462+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe198003cd0 con 0x7fe188004d00 2023-11-24T08:23:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe19f7fe640 1 -- 172.21.15.107:0/3637767611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198005310 con 0x7fe188004d00 2023-11-24T08:23:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3637767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 msgr2=0x7fe188005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3637767611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe188005180 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fe1a015f200 tx=0x7fe1880057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5734640 1 -- 172.21.15.107:0/3637767611 reap_dead start 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3637767611 shutdown_connections 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3637767611 >> 172.21.15.107:0/3637767611 conn(0x7fe1a00cb8a0 msgr2=0x7fe1a00cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3637767611 shutdown_connections 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.463+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3637767611 wait complete. 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.465+0000 7fe1a5f35640 1 Processor -- start 2023-11-24T08:23:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.465+0000 7fe1a5f35640 1 -- start start 2023-11-24T08:23:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a0132910 0x7fe1a00b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a013f410 0x7fe1a013f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00702f0 con 0x7fe188004d00 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a013fdd0 con 0x7fe1a013f410 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a5f35640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00704d0 con 0x7fe1a0132910 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a4f33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a4f33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42880/0 (socket says 172.21.15.107:42880) 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.466+0000 7fe1a4f33640 1 -- 172.21.15.107:0/3630592005 learned_addr learned my addr 172.21.15.107:0/3630592005 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 -- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a0132910 msgr2=0x7fe1a00b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a0132910 0x7fe1a00b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a0132910 0x7fe1a00b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 -- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a013f410 msgr2=0x7fe1a013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a013f410 0x7fe1a013f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 -- 172.21.15.107:0/3630592005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00707b0 con 0x7fe188004d00 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a4f33640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fe19400e7f0 tx=0x7fe19400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400c850 con 0x7fe188004d00 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19400c9b0 con 0x7fe188004d00 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400cd40 con 0x7fe188004d00 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.467+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a0070a90 con 0x7fe188004d00 2023-11-24T08:23:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.468+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a0071050 con 0x7fe188004d00 2023-11-24T08:23:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.469+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a004e840 con 0x7fe188004d00 2023-11-24T08:23:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.470+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fe194013420 con 0x7fe188004d00 2023-11-24T08:23:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.471+0000 7fe19d7fa640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 0x7fe18c06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.471+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe194088530 con 0x7fe188004d00 2023-11-24T08:23:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.471+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 0x7fe18c06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.472+0000 7fe19ffff640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 0x7fe18c06f850 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00cee50 tx=0x7fe190008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.476+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1940550e0 con 0x7fe188004d00 2023-11-24T08:23:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.702+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol4247", "snap_name": "subvol_snap4594", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1a015c8d0 con 0x7fe18c06d390 2023-11-24T08:23:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.772+0000 7fe19d7fa640 1 -- 172.21.15.107:0/3630592005 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe1a015c8d0 con 0x7fe18c06d390 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.775+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 msgr2=0x7fe18c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 0x7fe18c06f850 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00cee50 tx=0x7fe190008040 comp rx=0 tx=0).stop 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 msgr2=0x7fe1a00b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fe19400e7f0 tx=0x7fe19400ecb0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 shutdown_connections 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe18c06d390 0x7fe18c06f850 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a013f410 0x7fe1a013f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a0132910 0x7fe1a00b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 --2- 172.21.15.107:0/3630592005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe188004d00 0x7fe1a00b4080 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 >> 172.21.15.107:0/3630592005 conn(0x7fe1a00cb8a0 msgr2=0x7fe1a015ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 shutdown_connections 2023-11-24T08:23:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.776+0000 7fe1a5f35640 1 -- 172.21.15.107:0/3630592005 wait complete. 2023-11-24T08:23:24.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone7352 2023-11-24T08:23:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.997+0000 7f66bd8c9640 1 Processor -- start 2023-11-24T08:23:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:24.998+0000 7f66bd8c9640 1 -- start start 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66bd8c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 0x7f66b80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66bd8c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66b80cbf20 0x7f66b80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66bd8c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b815eea0 con 0x7f66b80c57e0 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66bd8c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b80cc3d0 con 0x7f66b80cbf20 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66bd8c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b815b820 con 0x7f66b815b440 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66b80cbf20 0x7f66b80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66b80cbf20 0x7f66b80c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42890/0 (socket says 172.21.15.107:42890) 2023-11-24T08:23:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 -- 172.21.15.107:0/1879070407 learned_addr learned my addr 172.21.15.107:0/1879070407 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66b6ffd640 1 --2- 172.21.15.107:0/1879070407 >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 0x7f66b80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 -- 172.21.15.107:0/1879070407 >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 msgr2=0x7f66b80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 --2- 172.21.15.107:0/1879070407 >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 0x7f66b80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.000+0000 7f66ae7fc640 1 -- 172.21.15.107:0/1879070407 >> v1:172.21.15.120:6789/0 conn(0x7f66b80c57e0 legacy=0x7f66b80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.001+0000 7f66ae7fc640 1 -- 172.21.15.107:0/1879070407 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b806d600 con 0x7f66b80cbf20 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.001+0000 7f66b6ffd640 1 --2- 172.21.15.107:0/1879070407 >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 0x7f66b80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.002+0000 7f66ae7fc640 1 --2- 172.21.15.107:0/1879070407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80cbf20 0x7f66b80c52a0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f66b806e770 tx=0x7f66a002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bee3014f57b6ca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.002+0000 7f66b67fc640 1 -- 172.21.15.107:0/1879070407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0031800 con 0x7f66b80cbf20 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.002+0000 7f66b67fc640 1 -- 172.21.15.107:0/1879070407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0031960 con 0x7f66b80cbf20 2023-11-24T08:23:25.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.002+0000 7f66b67fc640 1 -- 172.21.15.107:0/1879070407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0031cf0 con 0x7f66b80cbf20 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 -- 172.21.15.107:0/1879070407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80cbf20 msgr2=0x7f66b80c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/1879070407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80cbf20 0x7f66b80c52a0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f66b806e770 tx=0x7f66a002b8a0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 -- 172.21.15.107:0/1879070407 shutdown_connections 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/1879070407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80cbf20 0x7f66b80c52a0 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/1879070407 >> v2:172.21.15.120:3300/0 conn(0x7f66b815b440 0x7f66b80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 -- 172.21.15.107:0/1879070407 >> 172.21.15.107:0/1879070407 conn(0x7f66b80b0c50 msgr2=0x7f66b80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 -- 172.21.15.107:0/1879070407 shutdown_connections 2023-11-24T08:23:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.003+0000 7f66bd8c9640 1 -- 172.21.15.107:0/1879070407 wait complete. 2023-11-24T08:23:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.005+0000 7f66bd8c9640 1 Processor -- start 2023-11-24T08:23:25.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.005+0000 7f66bd8c9640 1 -- start start 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66b80cbf20 0x7f66b80b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66b815b440 0x7f66b8070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80b3a20 con 0x7f66b80c57e0 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80b2e10 con 0x7f66b80cbf20 2023-11-24T08:23:25.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66bd8c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80b30f0 con 0x7f66b815b440 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42894/0 (socket says 172.21.15.107:42894) 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 -- 172.21.15.107:0/3859407321 learned_addr learned my addr 172.21.15.107:0/3859407321 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 -- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66b815b440 msgr2=0x7f66b8070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66b815b440 0x7f66b8070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 -- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66b80cbf20 msgr2=0x7f66b80b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66b80cbf20 0x7f66b80b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.006+0000 7f66b6ffd640 1 -- 172.21.15.107:0/3859407321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00094d0 con 0x7f66b80c57e0 2023-11-24T08:23:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66b6ffd640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c7f0 tx=0x7f66a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8007820 con 0x7f66b80c57e0 2023-11-24T08:23:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8007980 con 0x7f66b80c57e0 2023-11-24T08:23:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8007cf0 con 0x7f66b80c57e0 2023-11-24T08:23:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b80b33d0 con 0x7f66b80c57e0 2023-11-24T08:23:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.007+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b8153ab0 con 0x7f66b80c57e0 2023-11-24T08:23:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.008+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b804e8b0 con 0x7f66b80c57e0 2023-11-24T08:23:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.009+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f66a80113f0 con 0x7f66b80c57e0 2023-11-24T08:23:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.010+0000 7f66b4ff9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 0x7f669806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.010+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66a8088b80 con 0x7f66b80c57e0 2023-11-24T08:23:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.011+0000 7f66ae7fc640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 0x7f669806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.011+0000 7f66ae7fc640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 0x7f669806f820 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f66a0036000 tx=0x7f66a0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.015+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66a8055730 con 0x7f66b80c57e0 2023-11-24T08:23:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.236+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone7352", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b80b3260 con 0x7f669806d360 2023-11-24T08:23:25.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.262+0000 7f66b4ff9640 1 -- 172.21.15.107:0/3859407321 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f66b80b3260 con 0x7f669806d360 2023-11-24T08:23:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.264+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 msgr2=0x7f669806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.264+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 0x7f669806f820 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f66a0036000 tx=0x7f66a0030000 comp rx=0 tx=0).stop 2023-11-24T08:23:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 msgr2=0x7f66b8153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c7f0 tx=0x7f66a800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 shutdown_connections 2023-11-24T08:23:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f669806d360 0x7f669806f820 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66b815b440 0x7f66b8070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66b80cbf20 0x7f66b80b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 --2- 172.21.15.107:0/3859407321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b80c57e0 0x7f66b8153570 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 >> 172.21.15.107:0/3859407321 conn(0x7f66b80b0c50 msgr2=0x7f66b8154a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 shutdown_connections 2023-11-24T08:23:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.265+0000 7f66bd8c9640 1 -- 172.21.15.107:0/3859407321 wait complete. 2023-11-24T08:23:25.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2023-11-24T08:23:25.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.483+0000 7fdacb0d8640 1 Processor -- start 2023-11-24T08:23:25.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.483+0000 7fdacb0d8640 1 -- start start 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdacb0d8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac41401e0 0x7fdac4142a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdacb0d8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdac415e460 0x7fdac40b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdacb0d8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac40b36e0 con 0x7fdac40b32b0 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdacb0d8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac40b4e50 con 0x7fdac415e460 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdacb0d8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac40b42f0 con 0x7fdac41401e0 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.485+0000 7fdac98d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdac415e460 0x7fdac40b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdac415e460 0x7fdac40b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42908/0 (socket says 172.21.15.107:42908) 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdaca8d7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdac40b32b0 0x7fdac40d13b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43440/0 (socket says 172.21.15.107:43440) 2023-11-24T08:23:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 -- 172.21.15.107:0/112532271 learned_addr learned my addr 172.21.15.107:0/112532271 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdaca0d6640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac41401e0 0x7fdac4142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 -- 172.21.15.107:0/112532271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac41401e0 msgr2=0x7fdac4142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac41401e0 0x7fdac4142a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 -- 172.21.15.107:0/112532271 >> v1:172.21.15.107:6789/0 conn(0x7fdac40b32b0 legacy=0x7fdac40d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.486+0000 7fdac98d5640 1 -- 172.21.15.107:0/112532271 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4143080 con 0x7fdac415e460 2023-11-24T08:23:25.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.487+0000 7fdac98d5640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac415e460 0x7fdac40b2d70 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009870 tx=0x7fdab402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f25dc8a60f70f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.487+0000 7fdac90d4640 1 -- 172.21.15.107:0/112532271 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4030840 con 0x7fdac415e460 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.487+0000 7fdac90d4640 1 -- 172.21.15.107:0/112532271 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab40309a0 con 0x7fdac415e460 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.487+0000 7fdac90d4640 1 -- 172.21.15.107:0/112532271 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4030d30 con 0x7fdac415e460 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 -- 172.21.15.107:0/112532271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac415e460 msgr2=0x7fdac40b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac415e460 0x7fdac40b2d70 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009870 tx=0x7fdab402b980 comp rx=0 tx=0).stop 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 -- 172.21.15.107:0/112532271 shutdown_connections 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac415e460 0x7fdac40b2d70 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/112532271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac41401e0 0x7fdac4142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 -- 172.21.15.107:0/112532271 >> 172.21.15.107:0/112532271 conn(0x7fdac4060f20 msgr2=0x7fdac4061300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:25.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 -- 172.21.15.107:0/112532271 shutdown_connections 2023-11-24T08:23:25.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.488+0000 7fdacb0d8640 1 -- 172.21.15.107:0/112532271 wait complete. 2023-11-24T08:23:25.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.490+0000 7fdacb0d8640 1 Processor -- start 2023-11-24T08:23:25.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.490+0000 7fdacb0d8640 1 -- start start 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 0x7fdac41536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdac415e460 0x7fdac406c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac4157d40 con 0x7fdac41401e0 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac4139dc0 con 0x7fdac415e460 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdacb0d8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac41570c0 con 0x7fdac40b32b0 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdac98d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdac98d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42914/0 (socket says 172.21.15.107:42914) 2023-11-24T08:23:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.491+0000 7fdac98d5640 1 -- 172.21.15.107:0/661036884 learned_addr learned my addr 172.21.15.107:0/661036884 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdaca0d6640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 0x7fdac41536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 -- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 msgr2=0x7fdac41536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 0x7fdac41536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 -- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdac415e460 msgr2=0x7fdac406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdac415e460 0x7fdac406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 -- 172.21.15.107:0/661036884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab40094d0 con 0x7fdac41401e0 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdaca0d6640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 0x7fdac41536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:25.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdac98d5640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4039c90 con 0x7fdac41401e0 2023-11-24T08:23:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab4039df0 con 0x7fdac41401e0 2023-11-24T08:23:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac41573a0 con 0x7fdac41401e0 2023-11-24T08:23:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4043640 con 0x7fdac41401e0 2023-11-24T08:23:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.492+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac4157960 con 0x7fdac41401e0 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.494+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac404e840 con 0x7fdac41401e0 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.495+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fdab40437a0 con 0x7fdac41401e0 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.496+0000 7fdabaffd640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 0x7fdaa806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.496+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdab403e070 con 0x7fdac41401e0 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.496+0000 7fdaca0d6640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 0x7fdaa806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:25.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.497+0000 7fdaca0d6640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 0x7fdaa806f820 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fdac0006fd0 tx=0x7fdac0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:25.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.501+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdab40b7080 con 0x7fdac41401e0 2023-11-24T08:23:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.717+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7fdac4143920 con 0x7fdaa806d360 2023-11-24T08:23:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.725+0000 7fdabaffd640 1 -- 172.21.15.107:0/661036884 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fdac00095d0 con 0x7fdaa806d360 2023-11-24T08:23:25.727 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.727+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 msgr2=0x7fdaa806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 0x7fdaa806f820 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fdac0006fd0 tx=0x7fdac0008040 comp rx=0 tx=0).stop 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 msgr2=0x7fdac41e7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4004730 comp rx=0 tx=0).stop 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 shutdown_connections 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdaa806d360 0x7fdaa806f820 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdac415e460 0x7fdac406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdac41401e0 0x7fdac41e7650 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 --2- 172.21.15.107:0/661036884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdac40b32b0 0x7fdac41536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 >> 172.21.15.107:0/661036884 conn(0x7fdac4060f20 msgr2=0x7fdac4061300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 shutdown_connections 2023-11-24T08:23:25.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:25.728+0000 7fdacb0d8640 1 -- 172.21.15.107:0/661036884 wait complete. 2023-11-24T08:23:25.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:25.748 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:23:25.807 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:23:25.808 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:23:25.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:23:25.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:23:26.053 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:23:26.053 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:26.053 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:26.115 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:26.115 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:26.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:26.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.385+0000 7f98e18b4640 1 Processor -- start 2023-11-24T08:23:26.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.385+0000 7f98e18b4640 1 -- start start 2023-11-24T08:23:26.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.386+0000 7f98e18b4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98dc15c7f0 0x7f98dc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.386+0000 7f98e18b4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0cece0 con 0x7f98dc0cf450 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.386+0000 7f98e18b4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc06d040 con 0x7f98dc0ce6c0 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.386+0000 7f98e18b4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc0ceaf0 con 0x7f98dc15c7f0 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98dc15c7f0 0x7f98dc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98dc15c7f0 0x7f98dc06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42924/0 (socket says 172.21.15.107:42924) 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98d27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98dc0cf450 0x7f98dc0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43450/0 (socket says 172.21.15.107:43450) 2023-11-24T08:23:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 -- 172.21.15.107:0/23300095 learned_addr learned my addr 172.21.15.107:0/23300095 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:26.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 -- 172.21.15.107:0/23300095 >> v1:172.21.15.120:6789/0 conn(0x7f98dc0ce6c0 legacy=0x7f98dc0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:26.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 -- 172.21.15.107:0/23300095 >> v1:172.21.15.107:6789/0 conn(0x7f98dc0cf450 legacy=0x7f98dc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.387+0000 7f98daffd640 1 -- 172.21.15.107:0/23300095 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc06d320 con 0x7f98dc15c7f0 2023-11-24T08:23:26.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.388+0000 7f98daffd640 1 --2- 172.21.15.107:0/23300095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc15c7f0 0x7f98dc06db20 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb65bf353c5938f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98da7fc640 1 -- 172.21.15.107:0/23300095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc030860 con 0x7f98dc15c7f0 2023-11-24T08:23:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98da7fc640 1 -- 172.21.15.107:0/23300095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc0309c0 con 0x7f98dc15c7f0 2023-11-24T08:23:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98da7fc640 1 -- 172.21.15.107:0/23300095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc030d50 con 0x7f98dc15c7f0 2023-11-24T08:23:26.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 -- 172.21.15.107:0/23300095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc15c7f0 msgr2=0x7f98dc06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 --2- 172.21.15.107:0/23300095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc15c7f0 0x7f98dc06db20 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 -- 172.21.15.107:0/23300095 shutdown_connections 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 --2- 172.21.15.107:0/23300095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc15c7f0 0x7f98dc06db20 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 -- 172.21.15.107:0/23300095 >> 172.21.15.107:0/23300095 conn(0x7f98dc06a920 msgr2=0x7f98dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 -- 172.21.15.107:0/23300095 shutdown_connections 2023-11-24T08:23:26.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.389+0000 7f98e18b4640 1 -- 172.21.15.107:0/23300095 wait complete. 2023-11-24T08:23:26.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.391+0000 7f98e18b4640 1 Processor -- start 2023-11-24T08:23:26.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.391+0000 7f98e18b4640 1 -- start start 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98dc0ce6c0 0x7f98dc071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98dc15c7f0 0x7f98dc1462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1469d0 con 0x7f98dc0cf450 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0b3cf0 con 0x7f98dc0ce6c0 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98e18b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0b3fd0 con 0x7f98dc15c7f0 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98d27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98d27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42936/0 (socket says 172.21.15.107:42936) 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.392+0000 7f98d27fc640 1 -- 172.21.15.107:0/4223629381 learned_addr learned my addr 172.21.15.107:0/4223629381 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98daffd640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98dc0ce6c0 0x7f98dc071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98db7fe640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98dc15c7f0 0x7f98dc1462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 -- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98dc15c7f0 msgr2=0x7f98dc1462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98dc15c7f0 0x7f98dc1462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 -- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98dc0ce6c0 msgr2=0x7f98dc071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98dc0ce6c0 0x7f98dc071740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 -- 172.21.15.107:0/4223629381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0094d0 con 0x7f98dc0cf450 2023-11-24T08:23:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d27fc640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c4d0 tx=0x7f98c400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4007480 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c40075e0 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40049e0 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.393+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc146c70 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.394+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc071c80 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.395+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc147190 con 0x7f98dc0cf450 2023-11-24T08:23:26.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.396+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f98c4004b40 con 0x7f98dc0cf450 2023-11-24T08:23:26.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.397+0000 7f98d8ff9640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 0x7f98c006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.397+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98c4087960 con 0x7f98dc0cf450 2023-11-24T08:23:26.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.397+0000 7f98daffd640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 0x7f98c006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.398+0000 7f98daffd640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 0x7f98c006f820 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.402+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98dc147190 con 0x7f98dc0cf450 2023-11-24T08:23:26.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.592+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98dc147190 con 0x7f98dc0cf450 2023-11-24T08:23:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.596+0000 7f98d8ff9640 1 -- 172.21.15.107:0/4223629381 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f98dc147190 con 0x7f98dc0cf450 2023-11-24T08:23:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:26.598 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:22:52.770317+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"220","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 msgr2=0x7f98c006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 0x7f98c006f820 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03b040 comp rx=0 tx=0).stop 2023-11-24T08:23:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 msgr2=0x7f98dc145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c4d0 tx=0x7f98c400c990 comp rx=0 tx=0).stop 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 shutdown_connections 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f98c006d360 0x7f98c006f820 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98dc15c7f0 0x7f98dc1462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98dc0cf450 0x7f98dc145d80 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 --2- 172.21.15.107:0/4223629381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98dc0ce6c0 0x7f98dc071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.598+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 >> 172.21.15.107:0/4223629381 conn(0x7f98dc06a920 msgr2=0x7f98dc0cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.599+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 shutdown_connections 2023-11-24T08:23:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.599+0000 7f98e18b4640 1 -- 172.21.15.107:0/4223629381 wait complete. 2023-11-24T08:23:26.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:22:49.030320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:22:50.221299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '220', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:23:26.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:23:26.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.836+0000 7f99936e7640 1 Processor -- start 2023-11-24T08:23:26.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.836+0000 7f99936e7640 1 -- start start 2023-11-24T08:23:26.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f99936e7640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 0x7f998c161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f99936e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 0x7f998c0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f99936e7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c1612f0 con 0x7f998c161fa0 2023-11-24T08:23:26.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f99936e7640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c0cff70 con 0x7f998c0cfb90 2023-11-24T08:23:26.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f99936e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0afb20 con 0x7f998c0b2400 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f9991483640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 0x7f998c161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f9991c84640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 0x7f998c0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f9991483640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 0x7f998c161a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:56672/0 (socket says 172.21.15.107:56672) 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.838+0000 7f9991483640 1 -- 172.21.15.107:0/432361654 learned_addr learned my addr 172.21.15.107:0/432361654 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.839+0000 7f9991c84640 1 -- 172.21.15.107:0/432361654 >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 msgr2=0x7f998c161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.839+0000 7f9991c84640 1 --2- 172.21.15.107:0/432361654 >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 0x7f998c161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.839+0000 7f9991c84640 1 -- 172.21.15.107:0/432361654 >> v1:172.21.15.120:6789/0 conn(0x7f998c161fa0 legacy=0x7f998c06ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.839+0000 7f9991c84640 1 -- 172.21.15.107:0/432361654 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c06a850 con 0x7f998c0b2400 2023-11-24T08:23:26.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.840+0000 7f9991c84640 1 --2- 172.21.15.107:0/432361654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 0x7f998c0cc940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d90 tx=0x7f998402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2365a64cbf7bdf38 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.841+0000 7f9983fff640 1 -- 172.21.15.107:0/432361654 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998402cce0 con 0x7f998c0b2400 2023-11-24T08:23:26.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.841+0000 7f9983fff640 1 -- 172.21.15.107:0/432361654 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998402ce40 con 0x7f998c0b2400 2023-11-24T08:23:26.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.841+0000 7f9983fff640 1 -- 172.21.15.107:0/432361654 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984034660 con 0x7f998c0b2400 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 -- 172.21.15.107:0/432361654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 msgr2=0x7f998c0cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 --2- 172.21.15.107:0/432361654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 0x7f998c0cc940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d90 tx=0x7f998402bf30 comp rx=0 tx=0).stop 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 -- 172.21.15.107:0/432361654 shutdown_connections 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 --2- 172.21.15.107:0/432361654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0b2400 0x7f998c0cc940 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 --2- 172.21.15.107:0/432361654 >> v2:172.21.15.120:3300/0 conn(0x7f998c0cfb90 0x7f998c161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 -- 172.21.15.107:0/432361654 >> 172.21.15.107:0/432361654 conn(0x7f998c0999d0 msgr2=0x7f998c0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 -- 172.21.15.107:0/432361654 shutdown_connections 2023-11-24T08:23:26.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.842+0000 7f99936e7640 1 -- 172.21.15.107:0/432361654 wait complete. 2023-11-24T08:23:26.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.844+0000 7f99936e7640 1 Processor -- start 2023-11-24T08:23:26.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.844+0000 7f99936e7640 1 -- start start 2023-11-24T08:23:26.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f998c0b2400 0x7f998c0b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0cfb90 0x7f998c1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0b4a30 con 0x7f998c1594c0 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0b4d10 con 0x7f998c0b2400 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f99936e7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0b4ff0 con 0x7f998c0cfb90 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f9991c84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f9991c84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42948/0 (socket says 172.21.15.107:42948) 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f9991c84640 1 -- 172.21.15.107:0/2260394672 learned_addr learned my addr 172.21.15.107:0/2260394672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f9991483640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f998c0b2400 0x7f998c0b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.845+0000 7f9990c82640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0cfb90 0x7f998c1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 -- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0cfb90 msgr2=0x7f998c1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0cfb90 0x7f998c1524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 -- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f998c0b2400 msgr2=0x7f998c0b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f998c0b2400 0x7f998c0b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 -- 172.21.15.107:0/2260394672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984009a70 con 0x7f998c1594c0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9991c84640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d90 tx=0x7f9984004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998403d600 con 0x7f998c1594c0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998403d780 con 0x7f998c1594c0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c0b52d0 con 0x7f998c1594c0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998403db30 con 0x7f998c1594c0 2023-11-24T08:23:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.846+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c1529f0 con 0x7f998c1594c0 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.847+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c0b57f0 con 0x7f998c1594c0 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.848+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f998403b040 con 0x7f998c1594c0 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.849+0000 7f9981ffb640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 0x7f997006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.849+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99840aae40 con 0x7f998c1594c0 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.850+0000 7f9991483640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 0x7f997006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.850+0000 7f9991483640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 0x7f997006f820 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9988009b60 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:26.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:26.854+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f998c0b57f0 con 0x7f998c1594c0 2023-11-24T08:23:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.043+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f998c0b57f0 con 0x7f998c1594c0 2023-11-24T08:23:27.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.044+0000 7f9981ffb640 1 -- 172.21.15.107:0/2260394672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v220) v1 ==== 93+0+10 (secure 0 0 0) 0x7f998c0b57f0 con 0x7f998c1594c0 2023-11-24T08:23:27.046 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 msgr2=0x7f997006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 0x7f997006f820 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9988009b60 tx=0x7f9988008040 comp rx=0 tx=0).stop 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 msgr2=0x7f998c0b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d90 tx=0x7f9984004a40 comp rx=0 tx=0).stop 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 shutdown_connections 2023-11-24T08:23:27.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f997006d360 0x7f997006f820 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c1594c0 0x7f998c0b40c0 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f998c0cfb90 0x7f998c1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 --2- 172.21.15.107:0/2260394672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f998c0b2400 0x7f998c0b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 >> 172.21.15.107:0/2260394672 conn(0x7f998c0999d0 msgr2=0x7f998c0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.046+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 shutdown_connections 2023-11-24T08:23:27.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.047+0000 7f99936e7640 1 -- 172.21.15.107:0/2260394672 wait complete. 2023-11-24T08:23:27.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:23:27.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.286+0000 7f651fc90640 1 Processor -- start 2023-11-24T08:23:27.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.286+0000 7f651fc90640 1 -- start start 2023-11-24T08:23:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651fc90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f65180c49f0 0x7f651815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651fc90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651815ec90 con 0x7f6518072160 2023-11-24T08:23:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651fc90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65180cd530 con 0x7f651806de10 2023-11-24T08:23:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651fc90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65180c4ed0 con 0x7f65180c49f0 2023-11-24T08:23:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651da2c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6518072160 0x7f651806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43458/0 (socket says 172.21.15.107:43458) 2023-11-24T08:23:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651da2c640 1 -- 172.21.15.107:0/1898498186 learned_addr learned my addr 172.21.15.107:0/1898498186 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889952922 0 0) 0x7f6514001110 con 0x7f6518072160 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6504005730 con 0x7f6518072160 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.288+0000 7f651e22d640 1 --2- 172.21.15.107:0/1898498186 >> v2:172.21.15.120:3300/0 conn(0x7f65180c49f0 0x7f651815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2597056969 0 0) 0x7f6504005730 con 0x7f6518072160 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 >> v2:172.21.15.120:3300/0 conn(0x7f65180c49f0 msgr2=0x7f651815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 --2- 172.21.15.107:0/1898498186 >> v2:172.21.15.120:3300/0 conn(0x7f65180c49f0 0x7f651815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 >> v1:172.21.15.120:6789/0 conn(0x7f651806de10 legacy=0x7f65180705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180c51b0 con 0x7f6518072160 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6514001800 con 0x7f6518072160 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 >> v1:172.21.15.107:6789/0 conn(0x7f6518072160 legacy=0x7f651806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504005b80 0x7f6504005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65040069d0 0x7f6504006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 0x7f650400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180c4ed0 con 0x7f6504007390 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650400e2c0 con 0x7f6504005b80 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650400e480 con 0x7f65040069d0 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651e22d640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 0x7f650400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651d22b640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65040069d0 0x7f6504006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651da2c640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504005b80 0x7f6504005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651e22d640 1 -- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65040069d0 msgr2=0x7f6504006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.289+0000 7f651e22d640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65040069d0 0x7f6504006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.290+0000 7f651e22d640 1 -- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504005b80 msgr2=0x7f6504005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.290+0000 7f651e22d640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504005b80 0x7f6504005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.290+0000 7f651e22d640 1 -- 172.21.15.107:0/1898498186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504005730 con 0x7f6504007390 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.290+0000 7f651da2c640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504005b80 0x7f6504005f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.290+0000 7f651e22d640 1 -- 172.21.15.107:0/1898498186 reap_dead start 2023-11-24T08:23:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651e22d640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 0x7f650400db90 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f65180c4e00 tx=0x7f6504007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d38c7eb00b977be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651002cca0 con 0x7f6504007390 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651002ce00 con 0x7f6504007390 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651ca2a640 1 -- 172.21.15.107:0/1898498186 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510003bd0 con 0x7f6504007390 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651fc90640 1 -- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 msgr2=0x7f650400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.291+0000 7f651fc90640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 0x7f650400db90 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f65180c4e00 tx=0x7f6504007a00 comp rx=0 tx=0).stop 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.292+0000 7f651fc90640 1 -- 172.21.15.107:0/1898498186 shutdown_connections 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.292+0000 7f651fc90640 1 --2- 172.21.15.107:0/1898498186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504007390 0x7f650400db90 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.292+0000 7f651fc90640 1 -- 172.21.15.107:0/1898498186 >> 172.21.15.107:0/1898498186 conn(0x7f65180b0c50 msgr2=0x7f65180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.292+0000 7f651fc90640 1 -- 172.21.15.107:0/1898498186 shutdown_connections 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.292+0000 7f651fc90640 1 -- 172.21.15.107:0/1898498186 wait complete. 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.293+0000 7f651fc90640 1 Processor -- start 2023-11-24T08:23:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.293+0000 7f651fc90640 1 -- start start 2023-11-24T08:23:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 0x7f65180cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65180ae2b0 0x7f651815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180cbf60 con 0x7f65180ade50 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180cb930 con 0x7f6504007390 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.294+0000 7f651fc90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6518147f50 con 0x7f65180ae2b0 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42958/0 (socket says 172.21.15.107:42958) 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 -- 172.21.15.107:0/4167710877 learned_addr learned my addr 172.21.15.107:0/4167710877 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651da2c640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 0x7f65180cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 -- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65180ae2b0 msgr2=0x7f651815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651e22d640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65180ae2b0 0x7f651815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65180ae2b0 0x7f651815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 -- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 msgr2=0x7f65180cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 0x7f65180cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 -- 172.21.15.107:0/4167710877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180cbbf0 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651da2c640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 0x7f65180cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.295+0000 7f651d22b640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f650800d4e0 tx=0x7f650800d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.296+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508012800 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.296+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6508012960 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.296+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508012d50 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.296+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651815aef0 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.296+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65180ca240 con 0x7f65180ade50 2023-11-24T08:23:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.297+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180cc0d0 con 0x7f65180ade50 2023-11-24T08:23:27.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.298+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f6508019360 con 0x7f65180ade50 2023-11-24T08:23:27.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.299+0000 7f650e7fc640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 0x7f64fc06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.299+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f65080889d0 con 0x7f65180ade50 2023-11-24T08:23:27.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.299+0000 7f651da2c640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 0x7f64fc06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.300+0000 7f651da2c640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 0x7f64fc06f850 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f65140078d0 tx=0x7f6514009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.303+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6508055450 con 0x7f65180ade50 2023-11-24T08:23:27.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.493+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65180cbd80 con 0x7f65180ade50 2023-11-24T08:23:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.494+0000 7f650e7fc640 1 -- 172.21.15.107:0/4167710877 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v221) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6508020070 con 0x7f65180ade50 2023-11-24T08:23:27.496 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:23:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.496+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 msgr2=0x7f64fc06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.496+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 0x7f64fc06f850 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f65140078d0 tx=0x7f6514009040 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.496+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 msgr2=0x7f651815a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.496+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f650800d4e0 tx=0x7f650800d9a0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 shutdown_connections 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f64fc06d390 0x7f64fc06f850 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65180ae2b0 0x7f651815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65180ade50 0x7f651815a470 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 --2- 172.21.15.107:0/4167710877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6504007390 0x7f65180cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 >> 172.21.15.107:0/4167710877 conn(0x7f65180b0c50 msgr2=0x7f65180710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 shutdown_connections 2023-11-24T08:23:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.497+0000 7f651fc90640 1 -- 172.21.15.107:0/4167710877 wait complete. 2023-11-24T08:23:27.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:23:27.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.725+0000 7f9f7e656640 1 Processor -- start 2023-11-24T08:23:27.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.726+0000 7f9f7e656640 1 -- start start 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.727+0000 7f9f7e656640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9f7806e640 0x7f9f780ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.727+0000 7f9f7e656640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9f780ad3c0 0x7f9f780708f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.727+0000 7f9f7e656640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7815b670 con 0x7f9f7815b290 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.727+0000 7f9f7e656640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780adbd0 con 0x7f9f7806e640 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.727+0000 7f9f7e656640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780adeb0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9f7806e640 0x7f9f780ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9f7806e640 0x7f9f780ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42966/0 (socket says 172.21.15.107:42966) 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 -- 172.21.15.107:0/3998275935 learned_addr learned my addr 172.21.15.107:0/3998275935 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f7cbf3640 1 --2- 172.21.15.107:0/3998275935 >> v2:172.21.15.120:3300/0 conn(0x7f9f780ad3c0 0x7f9f780708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 -- 172.21.15.107:0/3998275935 >> v2:172.21.15.120:3300/0 conn(0x7f9f780ad3c0 msgr2=0x7f9f780708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 --2- 172.21.15.107:0/3998275935 >> v2:172.21.15.120:3300/0 conn(0x7f9f780ad3c0 0x7f9f780708f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 -- 172.21.15.107:0/3998275935 >> v1:172.21.15.120:6789/0 conn(0x7f9f7815b290 legacy=0x7f9f7806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.728+0000 7f9f777fe640 1 -- 172.21.15.107:0/3998275935 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f780ae190 con 0x7f9f7806e640 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.729+0000 7f9f777fe640 1 --2- 172.21.15.107:0/3998275935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7806e640 0x7f9f780ace80 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f9f78072c50 tx=0x7f9f6c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe3d8944fb0faca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.729+0000 7f9f76ffd640 1 -- 172.21.15.107:0/3998275935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c02ec90 con 0x7f9f7806e640 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.729+0000 7f9f76ffd640 1 -- 172.21.15.107:0/3998275935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c02edf0 con 0x7f9f7806e640 2023-11-24T08:23:27.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.729+0000 7f9f76ffd640 1 -- 172.21.15.107:0/3998275935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c036610 con 0x7f9f7806e640 2023-11-24T08:23:27.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 -- 172.21.15.107:0/3998275935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7806e640 msgr2=0x7f9f780ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 --2- 172.21.15.107:0/3998275935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7806e640 0x7f9f780ace80 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f9f78072c50 tx=0x7f9f6c02e0f0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 -- 172.21.15.107:0/3998275935 shutdown_connections 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 --2- 172.21.15.107:0/3998275935 >> v2:172.21.15.120:3300/0 conn(0x7f9f780ad3c0 0x7f9f780708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 --2- 172.21.15.107:0/3998275935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7806e640 0x7f9f780ace80 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 -- 172.21.15.107:0/3998275935 >> 172.21.15.107:0/3998275935 conn(0x7f9f780b0b70 msgr2=0x7f9f780b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 -- 172.21.15.107:0/3998275935 shutdown_connections 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.730+0000 7f9f7e656640 1 -- 172.21.15.107:0/3998275935 wait complete. 2023-11-24T08:23:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.732+0000 7f9f7e656640 1 Processor -- start 2023-11-24T08:23:27.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.732+0000 7f9f7e656640 1 -- start start 2023-11-24T08:23:27.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 0x7f9f781582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f7815b290 0x7f9f780cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78151040 con 0x7f9f780ad3c0 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781588e0 con 0x7f9f7806e640 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f7e656640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78158bc0 con 0x7f9f7815b290 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42968/0 (socket says 172.21.15.107:42968) 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.733+0000 7f9f777fe640 1 -- 172.21.15.107:0/2293300882 learned_addr learned my addr 172.21.15.107:0/2293300882 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 -- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f7815b290 msgr2=0x7f9f780cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f7815b290 0x7f9f780cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 -- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 msgr2=0x7f9f781582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f77fff640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 0x7f9f781582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 0x7f9f781582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 -- 172.21.15.107:0/2293300882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c009b30 con 0x7f9f780ad3c0 2023-11-24T08:23:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f77fff640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 0x7f9f781582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f777fe640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000f80 tx=0x7f9f6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c03f670 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c03f7f0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c03fba0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f78158ea0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.734+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f78159460 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.735+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f78159650 con 0x7f9f780ad3c0 2023-11-24T08:23:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.736+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f9f6c03fd00 con 0x7f9f780ad3c0 2023-11-24T08:23:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.737+0000 7f9f74ff9640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 0x7f9f6406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.737+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f9f6c0ab190 con 0x7f9f780ad3c0 2023-11-24T08:23:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.738+0000 7f9f77fff640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 0x7f9f6406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:27.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.738+0000 7f9f77fff640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 0x7f9f6406f820 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f9f68007fd0 tx=0x7f9f68009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.742+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f6c077ca0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.936+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f78159650 con 0x7f9f780ad3c0 2023-11-24T08:23:27.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.937+0000 7f9f74ff9640 1 -- 172.21.15.107:0/2293300882 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v221) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9f6c07b2c0 con 0x7f9f780ad3c0 2023-11-24T08:23:27.939 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:23:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 msgr2=0x7f9f6406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 0x7f9f6406f820 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f9f68007fd0 tx=0x7f9f68009040 comp rx=0 tx=0).stop 2023-11-24T08:23:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 msgr2=0x7f9f78143880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000f80 tx=0x7f9f6c004a40 comp rx=0 tx=0).stop 2023-11-24T08:23:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 shutdown_connections 2023-11-24T08:23:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9f6406d360 0x7f9f6406f820 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f7815b290 0x7f9f780cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f780ad3c0 0x7f9f78143880 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 --2- 172.21.15.107:0/2293300882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f7806e640 0x7f9f781582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:27.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 >> 172.21.15.107:0/2293300882 conn(0x7f9f780b0b70 msgr2=0x7f9f7815bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:27.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 shutdown_connections 2023-11-24T08:23:27.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:27.939+0000 7f9f7e656640 1 -- 172.21.15.107:0/2293300882 wait complete. 2023-11-24T08:23:27.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:23:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.171+0000 7fae027bc640 1 Processor -- start 2023-11-24T08:23:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.171+0000 7fae027bc640 1 -- start start 2023-11-24T08:23:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.172+0000 7fae027bc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fadfc072570 0x7fadfc0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.172+0000 7fae027bc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc06d270 con 0x7fadfc159920 2023-11-24T08:23:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fae027bc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc06c660 con 0x7fadfc159d00 2023-11-24T08:23:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fae027bc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc06c940 con 0x7fadfc072570 2023-11-24T08:23:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fae017ba640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fadfc159920 0x7fadfc0ae310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40418/0 (socket says 172.21.15.107:40418) 2023-11-24T08:23:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fae017ba640 1 -- 172.21.15.107:0/3547607222 learned_addr learned my addr 172.21.15.107:0/3547607222 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462909037 0 0) 0x7fadfc06d270 con 0x7fadfc159920 2023-11-24T08:23:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8003520 con 0x7fadfc159920 2023-11-24T08:23:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fae01fbb640 1 --2- 172.21.15.107:0/3547607222 >> v2:172.21.15.120:3300/0 conn(0x7fadfc072570 0x7fadfc0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.173+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283668296 0 0) 0x7fade8003520 con 0x7fadfc159920 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 >> v2:172.21.15.120:3300/0 conn(0x7fadfc072570 msgr2=0x7fadfc0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 --2- 172.21.15.107:0/3547607222 >> v2:172.21.15.120:3300/0 conn(0x7fadfc072570 0x7fadfc0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 >> v1:172.21.15.120:6789/0 conn(0x7fadfc159d00 legacy=0x7fadfc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc06cc20 con 0x7fadfc159920 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fadf8001670 con 0x7fadfc159920 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 >> v1:172.21.15.107:6789/0 conn(0x7fadfc159920 legacy=0x7fadfc0ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 0x7fade8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade80047c0 0x7fade8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80052a0 0x7fade800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc06c940 con 0x7fade8003970 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade800c150 con 0x7fade80052a0 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade800e520 con 0x7fade80047c0 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 0x7fade8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae01fbb640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80052a0 0x7fade800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 -- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade80047c0 msgr2=0x7fade8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade80047c0 0x7fade8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 -- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80052a0 msgr2=0x7fade800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade80052a0 0x7fade800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae017ba640 1 -- 172.21.15.107:0/3547607222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8003520 con 0x7fade8003970 2023-11-24T08:23:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.174+0000 7fae01fbb640 1 -- 172.21.15.107:0/3547607222 reap_dead start 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.175+0000 7fae017ba640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 0x7fade8003d70 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fadf8001eb0 tx=0x7fade80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94cac326c4afd12 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8005ad0 con 0x7fade8003970 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf8005c50 con 0x7fade8003970 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fadf3fff640 1 -- 172.21.15.107:0/3547607222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf802c8b0 con 0x7fade8003970 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 -- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 msgr2=0x7fade8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 0x7fade8003d70 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fadf8001eb0 tx=0x7fade80057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 -- 172.21.15.107:0/3547607222 shutdown_connections 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 --2- 172.21.15.107:0/3547607222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fade8003970 0x7fade8003d70 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 -- 172.21.15.107:0/3547607222 >> 172.21.15.107:0/3547607222 conn(0x7fadfc0cbb40 msgr2=0x7fadfc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 -- 172.21.15.107:0/3547607222 shutdown_connections 2023-11-24T08:23:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.176+0000 7fae027bc640 1 -- 172.21.15.107:0/3547607222 wait complete. 2023-11-24T08:23:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.178+0000 7fae027bc640 1 Processor -- start 2023-11-24T08:23:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.178+0000 7fae027bc640 1 -- start start 2023-11-24T08:23:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 0x7fadfc15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadfc071380 0x7fadfc071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15b620 con 0x7fadfc1f3410 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc071d40 con 0x7fadfc071380 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.179+0000 7fae027bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc1ef520 con 0x7fade8003970 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40026/0 (socket says 172.21.15.107:40026) 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 -- 172.21.15.107:0/3627267874 learned_addr learned my addr 172.21.15.107:0/3627267874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae017ba640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 0x7fadfc15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 -- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 msgr2=0x7fadfc15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 0x7fadfc15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 -- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadfc071380 msgr2=0x7fadfc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadfc071380 0x7fadfc071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 -- 172.21.15.107:0/3627267874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc1ef800 con 0x7fadfc1f3410 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae017ba640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 0x7fadfc15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fae00fb9640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fadec00c3c0 tx=0x7fadec00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec011790 con 0x7fadfc1f3410 2023-11-24T08:23:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.180+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec0118f0 con 0x7fadfc1f3410 2023-11-24T08:23:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.181+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc1efae0 con 0x7fadfc1f3410 2023-11-24T08:23:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.181+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec011ca0 con 0x7fadfc1f3410 2023-11-24T08:23:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.181+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc1f3890 con 0x7fadfc1f3410 2023-11-24T08:23:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.182+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc1f02e0 con 0x7fadfc1f3410 2023-11-24T08:23:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.183+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fadec01c460 con 0x7fadfc1f3410 2023-11-24T08:23:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.184+0000 7fadf1ffb640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 0x7fade006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.184+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fadec088750 con 0x7fadfc1f3410 2023-11-24T08:23:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.184+0000 7fae017ba640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 0x7fade006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.185+0000 7fae017ba640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 0x7fade006f820 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fadf8034000 tx=0x7fadf8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.188+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadfc1f02e0 con 0x7fadfc1f3410 2023-11-24T08:23:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.392+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fadfc1f02e0 con 0x7fadfc1f3410 2023-11-24T08:23:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.393+0000 7fadf1ffb640 1 -- 172.21.15.107:0/3627267874 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v221) v1 ==== 80+0+0 (secure 0 0 0) 0x7fadfc1f02e0 con 0x7fadfc1f3410 2023-11-24T08:23:28.395 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:23:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 msgr2=0x7fade006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 0x7fade006f820 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fadf8034000 tx=0x7fadf8009030 comp rx=0 tx=0).stop 2023-11-24T08:23:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 msgr2=0x7fadfc14f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fadec00c3c0 tx=0x7fadec00c7c0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 shutdown_connections 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fade006d360 0x7fade006f820 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadfc071380 0x7fadfc071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadfc1f3410 0x7fadfc14f4d0 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 --2- 172.21.15.107:0/3627267874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fade8003970 0x7fadfc15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 >> 172.21.15.107:0/3627267874 conn(0x7fadfc0cbb40 msgr2=0x7fadfc0ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 shutdown_connections 2023-11-24T08:23:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.396+0000 7fae027bc640 1 -- 172.21.15.107:0/3627267874 wait complete. 2023-11-24T08:23:28.415 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:23:28.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:28.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:23:28.537 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:23:28.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:28.537 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:23:28.595 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:28.595 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:28.636 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:28.636 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:28.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.934+0000 7f5d9e116640 1 Processor -- start 2023-11-24T08:23:28.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.934+0000 7f5d9e116640 1 -- start start 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d9e116640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d9815c7f0 0x7f5d9806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d9e116640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d980cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d9e116640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d980cece0 con 0x7f5d980cf450 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d9e116640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d980cfe40 con 0x7f5d9815c7f0 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d9e116640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98156010 con 0x7f5d980ce6c0 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d96ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5d980cf450 0x7f5d980cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40420/0 (socket says 172.21.15.107:40420) 2023-11-24T08:23:28.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d96ffd640 1 -- 172.21.15.107:0/568785671 learned_addr learned my addr 172.21.15.107:0/568785671 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d97fff640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d980cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.936+0000 7f5d977fe640 1 --2- 172.21.15.107:0/568785671 >> v2:172.21.15.120:3300/0 conn(0x7f5d9815c7f0 0x7f5d9806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086627278 0 0) 0x7f5d980cece0 con 0x7f5d980cf450 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d80003520 con 0x7f5d980cf450 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120823382 0 0) 0x7f5d80003520 con 0x7f5d980cf450 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 msgr2=0x7f5d980cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d980cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 >> v2:172.21.15.120:3300/0 conn(0x7f5d9815c7f0 msgr2=0x7f5d9806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 --2- 172.21.15.107:0/568785671 >> v2:172.21.15.120:3300/0 conn(0x7f5d9815c7f0 0x7f5d9806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9806d040 con 0x7f5d980cf450 2023-11-24T08:23:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5d8c001670 con 0x7f5d980cf450 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 >> v1:172.21.15.107:6789/0 conn(0x7f5d980cf450 legacy=0x7f5d980cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d80003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d80003f40 0x7f5d800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d80004d00 0x7f5d80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980cfe40 con 0x7f5d80003f40 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98156010 con 0x7f5d80004d00 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.937+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8000b980 con 0x7f5d980ce6c0 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d80003f40 0x7f5d800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d977fe640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d80003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d97fff640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d80004d00 0x7f5d80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 -- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 msgr2=0x7f5d80003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d980ce6c0 0x7f5d80003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 -- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d80004d00 msgr2=0x7f5d80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:28.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d80004d00 0x7f5d80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d96ffd640 1 -- 172.21.15.107:0/568785671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80003520 con 0x7f5d80003f40 2023-11-24T08:23:28.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.938+0000 7f5d97fff640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d80004d00 0x7f5d80005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.939+0000 7f5d96ffd640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d80003f40 0x7f5d800047c0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c140 tx=0x7f5d800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbe755a9b1bebd4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c005ad0 con 0x7f5d80003f40 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c005c50 con 0x7f5d80003f40 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d967fc640 1 -- 172.21.15.107:0/568785671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c02c810 con 0x7f5d80003f40 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 -- 172.21.15.107:0/568785671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d80003f40 msgr2=0x7f5d800047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 --2- 172.21.15.107:0/568785671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d80003f40 0x7f5d800047c0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c140 tx=0x7f5d800057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d97fff640 1 -- 172.21.15.107:0/568785671 reap_dead start 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 -- 172.21.15.107:0/568785671 shutdown_connections 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 -- 172.21.15.107:0/568785671 >> 172.21.15.107:0/568785671 conn(0x7f5d9806a920 msgr2=0x7f5d9806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 -- 172.21.15.107:0/568785671 shutdown_connections 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.940+0000 7f5d9e116640 1 -- 172.21.15.107:0/568785671 wait complete. 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.942+0000 7f5d9e116640 1 Processor -- start 2023-11-24T08:23:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.942+0000 7f5d9e116640 1 -- start start 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d80003f40 0x7f5d9815acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d980adc80 0x7f5d9815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98157320 con 0x7f5d98151490 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98156280 con 0x7f5d980adc80 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.943+0000 7f5d9e116640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98156560 con 0x7f5d80003f40 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40040/0 (socket says 172.21.15.107:40040) 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 -- 172.21.15.107:0/1374002880 learned_addr learned my addr 172.21.15.107:0/1374002880 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d96ffd640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d980adc80 0x7f5d9815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d977fe640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d80003f40 0x7f5d9815acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 -- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d80003f40 msgr2=0x7f5d9815acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d80003f40 0x7f5d9815acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 -- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d980adc80 msgr2=0x7f5d9815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d980adc80 0x7f5d9815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.944+0000 7f5d97fff640 1 -- 172.21.15.107:0/1374002880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98156840 con 0x7f5d98151490 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d97fff640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f5d980b5270 tx=0x7f5d9000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9000fe90 con 0x7f5d98151490 2023-11-24T08:23:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d90011040 con 0x7f5d98151490 2023-11-24T08:23:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90017640 con 0x7f5d98151490 2023-11-24T08:23:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d98156b20 con 0x7f5d98151490 2023-11-24T08:23:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.945+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d980ae680 con 0x7f5d98151490 2023-11-24T08:23:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.946+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d98157040 con 0x7f5d98151490 2023-11-24T08:23:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.947+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f5d900203b0 con 0x7f5d98151490 2023-11-24T08:23:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.949+0000 7f5d7ffff640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 0x7f5d8406f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.949+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5d900889e0 con 0x7f5d98151490 2023-11-24T08:23:28.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.949+0000 7f5d977fe640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 0x7f5d8406f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:28.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.950+0000 7f5d977fe640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 0x7f5d8406f850 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009a90 tx=0x7f5d88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:28.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:28.953+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d98157040 con 0x7f5d98151490 2023-11-24T08:23:29.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.146+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d98157040 con 0x7f5d98151490 2023-11-24T08:23:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.148+0000 7f5d7ffff640 1 -- 172.21.15.107:0/1374002880 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f5d98157040 con 0x7f5d98151490 2023-11-24T08:23:29.151 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:29.151 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":221,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:27.170359+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":221,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 msgr2=0x7f5d8406f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 0x7f5d8406f850 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009a90 tx=0x7f5d88008040 comp rx=0 tx=0).stop 2023-11-24T08:23:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 msgr2=0x7f5d980abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f5d980b5270 tx=0x7f5d9000f4a0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 shutdown_connections 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5d8406d390 0x7f5d8406f850 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d98151490 0x7f5d980abfc0 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d980adc80 0x7f5d9815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 --2- 172.21.15.107:0/1374002880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d80003f40 0x7f5d9815acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 >> 172.21.15.107:0/1374002880 conn(0x7f5d9806a920 msgr2=0x7f5d980b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 shutdown_connections 2023-11-24T08:23:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.151+0000 7f5d9e116640 1 -- 172.21.15.107:0/1374002880 wait complete. 2023-11-24T08:23:29.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:22:49.030320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:22:50.221299+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 221, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:23:29.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:23:29.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.380+0000 7f3cdd49b640 1 Processor -- start 2023-11-24T08:23:29.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.380+0000 7f3cdd49b640 1 -- start start 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.382+0000 7f3cdd49b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3cd80c4220 0x7f3cd80c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.382+0000 7f3cdd49b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815d7f0 0x7f3cd80cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.382+0000 7f3cdd49b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd815e400 con 0x7f3cd815dc50 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.382+0000 7f3cdd49b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd80ce880 con 0x7f3cd80c4220 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.382+0000 7f3cdd49b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80c5ed0 con 0x7f3cd815d7f0 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3cd80c4220 0x7f3cd80c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3cd80c4220 0x7f3cd80c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40044/0 (socket says 172.21.15.107:40044) 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 -- 172.21.15.107:0/1609254051 learned_addr learned my addr 172.21.15.107:0/1609254051 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd77fe640 1 --1- 172.21.15.107:0/1609254051 >> v1:172.21.15.107:6789/0 conn(0x7f3cd815dc50 0x7f3cd80cb0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40426/0 (socket says 172.21.15.107:40426) 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 -- 172.21.15.107:0/1609254051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815d7f0 msgr2=0x7f3cd80cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 --2- 172.21.15.107:0/1609254051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815d7f0 0x7f3cd80cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 -- 172.21.15.107:0/1609254051 >> v1:172.21.15.107:6789/0 conn(0x7f3cd815dc50 legacy=0x7f3cd80cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd67fc640 1 -- 172.21.15.107:0/1609254051 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657594142 0 0) 0x7f3cd815e400 con 0x7f3cd815dc50 2023-11-24T08:23:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.383+0000 7f3cd6ffd640 1 -- 172.21.15.107:0/1609254051 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd80c46e0 con 0x7f3cd80c4220 2023-11-24T08:23:29.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.384+0000 7f3cd6ffd640 1 --2- 172.21.15.107:0/1609254051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd80c4220 0x7f3cd80c5990 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800bb10 tx=0x7f3cc802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea69ebf3f8769 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cd67fc640 1 -- 172.21.15.107:0/1609254051 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8036440 con 0x7f3cd80c4220 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cd67fc640 1 -- 172.21.15.107:0/1609254051 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc80365a0 con 0x7f3cd80c4220 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cd67fc640 1 -- 172.21.15.107:0/1609254051 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8036930 con 0x7f3cd80c4220 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 -- 172.21.15.107:0/1609254051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd80c4220 msgr2=0x7f3cd80c5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/1609254051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd80c4220 0x7f3cd80c5990 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800bb10 tx=0x7f3cc802e150 comp rx=0 tx=0).stop 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 -- 172.21.15.107:0/1609254051 shutdown_connections 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/1609254051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815d7f0 0x7f3cd80cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/1609254051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd80c4220 0x7f3cd80c5990 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 -- 172.21.15.107:0/1609254051 >> 172.21.15.107:0/1609254051 conn(0x7f3cd80b0c50 msgr2=0x7f3cd80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 -- 172.21.15.107:0/1609254051 shutdown_connections 2023-11-24T08:23:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.385+0000 7f3cdd49b640 1 -- 172.21.15.107:0/1609254051 wait complete. 2023-11-24T08:23:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.387+0000 7f3cdd49b640 1 Processor -- start 2023-11-24T08:23:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.387+0000 7f3cdd49b640 1 -- start start 2023-11-24T08:23:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80c4220 0x7f3cd8099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815dc50 0x7f3cd8156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd8156430 con 0x7f3cd815d7f0 2023-11-24T08:23:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd8154700 con 0x7f3cd80c4220 2023-11-24T08:23:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cdd49b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd8155820 con 0x7f3cd815dc50 2023-11-24T08:23:30.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cce7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cce7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40052/0 (socket says 172.21.15.107:40052) 2023-11-24T08:23:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.388+0000 7f3cce7fc640 1 -- 172.21.15.107:0/272328524 learned_addr learned my addr 172.21.15.107:0/272328524 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cd6ffd640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80c4220 0x7f3cd8099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cd77fe640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815dc50 0x7f3cd8156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 -- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815dc50 msgr2=0x7f3cd8156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815dc50 0x7f3cd8156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 -- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80c4220 msgr2=0x7f3cd8099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80c4220 0x7f3cd8099a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 -- 172.21.15.107:0/272328524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc800b770 con 0x7f3cd815d7f0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cce7fc640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000b4a0 tx=0x7f3cc000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0010610 con 0x7f3cd815d7f0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0010770 con 0x7f3cd815d7f0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd8155b00 con 0x7f3cd815d7f0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.389+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0010b20 con 0x7f3cd815d7f0 2023-11-24T08:23:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.390+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd81560c0 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.391+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd804e8b0 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.392+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f3cc0010c80 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.392+0000 7f3ccffff640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 0x7f3cb806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.393+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f3cc00884d0 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.393+0000 7f3cd6ffd640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 0x7f3cb806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.393+0000 7f3cd6ffd640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 0x7f3cb806f820 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8038000 tx=0x7f3cc8032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.397+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cc0054ff0 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.580+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3cd80c5c50 con 0x7f3cd815d7f0 2023-11-24T08:23:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.581+0000 7f3ccffff640 1 -- 172.21.15.107:0/272328524 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v221) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3cc001e070 con 0x7f3cd815d7f0 2023-11-24T08:23:30.908 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 msgr2=0x7f3cb806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 0x7f3cb806f820 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8038000 tx=0x7f3cc8032000 comp rx=0 tx=0).stop 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 msgr2=0x7f3cd81568c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000b4a0 tx=0x7f3cc000b960 comp rx=0 tx=0).stop 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 shutdown_connections 2023-11-24T08:23:30.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3cb806d360 0x7f3cb806f820 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd815dc50 0x7f3cd8156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd815d7f0 0x7f3cd81568c0 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 --2- 172.21.15.107:0/272328524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80c4220 0x7f3cd8099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 >> 172.21.15.107:0/272328524 conn(0x7f3cd80b0c50 msgr2=0x7f3cd80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 shutdown_connections 2023-11-24T08:23:30.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:29.584+0000 7f3cdd49b640 1 -- 172.21.15.107:0/272328524 wait complete. 2023-11-24T08:23:30.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:23:31.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.133+0000 7f1eb974c640 1 Processor -- start 2023-11-24T08:23:31.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.133+0000 7f1eb974c640 1 -- start start 2023-11-24T08:23:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb974c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb406de10 0x7f1eb415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb974c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1eb40c49f0 0x7f1eb40705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb974c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb415ec90 con 0x7f1eb4072160 2023-11-24T08:23:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb974c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40cd530 con 0x7f1eb40c49f0 2023-11-24T08:23:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb974c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40c4ed0 con 0x7f1eb406de10 2023-11-24T08:23:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1eb40c49f0 0x7f1eb40705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1eb40c49f0 0x7f1eb40705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40058/0 (socket says 172.21.15.107:40058) 2023-11-24T08:23:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb37fe640 1 -- 172.21.15.107:0/2551503546 learned_addr learned my addr 172.21.15.107:0/2551503546 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.135+0000 7f1eb27fc640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb406de10 0x7f1eb415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.136+0000 7f1eb37fe640 1 -- 172.21.15.107:0/2551503546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb406de10 msgr2=0x7f1eb415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.136+0000 7f1eb37fe640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb406de10 0x7f1eb415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.136+0000 7f1eb37fe640 1 -- 172.21.15.107:0/2551503546 >> v1:172.21.15.120:6789/0 conn(0x7f1eb4072160 legacy=0x7f1eb406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.136+0000 7f1eb37fe640 1 -- 172.21.15.107:0/2551503546 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb40c51b0 con 0x7f1eb40c49f0 2023-11-24T08:23:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.137+0000 7f1eb37fe640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40705c0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00d150 tx=0x7f1eac02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921a19c096bb548d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.137+0000 7f1eb1ffb640 1 -- 172.21.15.107:0/2551503546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00b0d0 con 0x7f1eb40c49f0 2023-11-24T08:23:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.137+0000 7f1eb1ffb640 1 -- 172.21.15.107:0/2551503546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00b230 con 0x7f1eb40c49f0 2023-11-24T08:23:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.137+0000 7f1eb1ffb640 1 -- 172.21.15.107:0/2551503546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac009600 con 0x7f1eb40c49f0 2023-11-24T08:23:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 -- 172.21.15.107:0/2551503546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb40c49f0 msgr2=0x7f1eb40705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40705c0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00d150 tx=0x7f1eac02e160 comp rx=0 tx=0).stop 2023-11-24T08:23:31.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 -- 172.21.15.107:0/2551503546 shutdown_connections 2023-11-24T08:23:31.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40705c0 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2551503546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb406de10 0x7f1eb415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 -- 172.21.15.107:0/2551503546 >> 172.21.15.107:0/2551503546 conn(0x7f1eb40b0c50 msgr2=0x7f1eb40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:31.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 -- 172.21.15.107:0/2551503546 shutdown_connections 2023-11-24T08:23:31.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.138+0000 7f1eb974c640 1 -- 172.21.15.107:0/2551503546 wait complete. 2023-11-24T08:23:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.140+0000 7f1eb974c640 1 Processor -- start 2023-11-24T08:23:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.140+0000 7f1eb974c640 1 -- start start 2023-11-24T08:23:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb4072160 0x7f1eb4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40ae960 con 0x7f1eb406de10 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4152430 con 0x7f1eb40c49f0 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb974c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40aff50 con 0x7f1eb4072160 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40072/0 (socket says 172.21.15.107:40072) 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.141+0000 7f1eb2ffd640 1 -- 172.21.15.107:0/2960575237 learned_addr learned my addr 172.21.15.107:0/2960575237 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb27fc640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb4072160 0x7f1eb4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb37fe640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 -- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb4072160 msgr2=0x7f1eb4151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb4072160 0x7f1eb4151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 -- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 msgr2=0x7f1eb40af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 -- 172.21.15.107:0/2960575237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac00bd40 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb37fe640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb2ffd640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c450 tx=0x7f1ea400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40052d0 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4005430 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4012870 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb40ad940 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.142+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb40adf00 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.143+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb404e8b0 con 0x7f1eb406de10 2023-11-24T08:23:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.145+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f1ea40129d0 con 0x7f1eb406de10 2023-11-24T08:23:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.145+0000 7f1e9b7fe640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 0x7f1ea006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.146+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f1ea4088240 con 0x7f1eb406de10 2023-11-24T08:23:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.146+0000 7f1eb27fc640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 0x7f1ea006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.146+0000 7f1eb27fc640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 0x7f1ea006f820 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40b0bb0 tx=0x7f1ea8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.150+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ea4054d50 con 0x7f1eb406de10 2023-11-24T08:23:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.342+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1eb415ec90 con 0x7f1eb406de10 2023-11-24T08:23:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.343+0000 7f1e9b7fe640 1 -- 172.21.15.107:0/2960575237 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v221) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1eb415ec90 con 0x7f1eb406de10 2023-11-24T08:23:31.345 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:23:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 msgr2=0x7f1ea006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 0x7f1ea006f820 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40b0bb0 tx=0x7f1ea8008040 comp rx=0 tx=0).stop 2023-11-24T08:23:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 msgr2=0x7f1eb40ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c450 tx=0x7f1ea400c910 comp rx=0 tx=0).stop 2023-11-24T08:23:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 shutdown_connections 2023-11-24T08:23:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1ea006d360 0x7f1ea006f820 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb40c49f0 0x7f1eb40af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb4072160 0x7f1eb4151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 --2- 172.21.15.107:0/2960575237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1eb406de10 0x7f1eb40ae420 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 >> 172.21.15.107:0/2960575237 conn(0x7f1eb40b0c50 msgr2=0x7f1eb406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 shutdown_connections 2023-11-24T08:23:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.346+0000 7f1eb974c640 1 -- 172.21.15.107:0/2960575237 wait complete. 2023-11-24T08:23:31.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:23:31.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.574+0000 7f36b288c640 1 Processor -- start 2023-11-24T08:23:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.574+0000 7f36b288c640 1 -- start start 2023-11-24T08:23:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.575+0000 7f36b288c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36ac15a1c0 0x7f36ac06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b288c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b288c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac06d430 con 0x7f36ac0b0b40 2023-11-24T08:23:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b288c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ac15ea70 con 0x7f36ac15a1c0 2023-11-24T08:23:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b288c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac15ec50 con 0x7f36ac15e5e0 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b188a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36ac15a1c0 0x7f36ac06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b208b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b208b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:50862/0 (socket says 172.21.15.107:50862) 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b188a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36ac15a1c0 0x7f36ac06d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42678/0 (socket says 172.21.15.107:42678) 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.576+0000 7f36b208b640 1 -- 172.21.15.107:0/434407379 learned_addr learned my addr 172.21.15.107:0/434407379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.577+0000 7f36b188a640 1 -- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 msgr2=0x7f36ac0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:31.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.577+0000 7f36b188a640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.577+0000 7f36b188a640 1 -- 172.21.15.107:0/434407379 >> v1:172.21.15.120:6789/0 conn(0x7f36ac0b0b40 legacy=0x7f36ac15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.577+0000 7f36b188a640 1 -- 172.21.15.107:0/434407379 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac0b14e0 con 0x7f36ac15a1c0 2023-11-24T08:23:31.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.577+0000 7f36b208b640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:23:31.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.578+0000 7f36b188a640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac06d8c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f36a80097a0 tx=0x7f36a802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28230e7f8a8e1bd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b0888640 1 -- 172.21.15.107:0/434407379 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a802d040 con 0x7f36ac15a1c0 2023-11-24T08:23:31.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b0888640 1 -- 172.21.15.107:0/434407379 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a8031810 con 0x7f36ac15a1c0 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b0888640 1 -- 172.21.15.107:0/434407379 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8031ba0 con 0x7f36ac15a1c0 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 -- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 msgr2=0x7f36ac06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac06d8c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f36a80097a0 tx=0x7f36a802b8b0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 -- 172.21.15.107:0/434407379 shutdown_connections 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 --2- 172.21.15.107:0/434407379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac06d8c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.579+0000 7f36b288c640 1 -- 172.21.15.107:0/434407379 >> 172.21.15.107:0/434407379 conn(0x7f36ac0cbb40 msgr2=0x7f36ac0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.580+0000 7f36b288c640 1 -- 172.21.15.107:0/434407379 shutdown_connections 2023-11-24T08:23:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.580+0000 7f36b288c640 1 -- 172.21.15.107:0/434407379 wait complete. 2023-11-24T08:23:31.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.581+0000 7f36b288c640 1 Processor -- start 2023-11-24T08:23:31.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.581+0000 7f36b288c640 1 -- start start 2023-11-24T08:23:31.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac1f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac1f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac155c00 con 0x7f36ac0b0b40 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac0c67e0 con 0x7f36ac15a1c0 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.582+0000 7f36b288c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac09b880 con 0x7f36ac15e5e0 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40088/0 (socket says 172.21.15.107:40088) 2023-11-24T08:23:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 -- 172.21.15.107:0/2136145803 learned_addr learned my addr 172.21.15.107:0/2136145803 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b1089640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac1f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b208b640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac1f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 -- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 msgr2=0x7f36ac1f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac1f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 -- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 msgr2=0x7f36ac1f6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac1f6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b188a640 1 -- 172.21.15.107:0/2136145803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a4009b30 con 0x7f36ac0b0b40 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b1089640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac1f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.583+0000 7f36b208b640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac1f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36b188a640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f36a802b8f0 tx=0x7f36a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8006e10 con 0x7f36ac0b0b40 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a803ad60 con 0x7f36ac0b0b40 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a8009400 con 0x7f36ac0b0b40 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8044630 con 0x7f36ac0b0b40 2023-11-24T08:23:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.584+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ac1f7460 con 0x7f36ac0b0b40 2023-11-24T08:23:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.585+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ac04e840 con 0x7f36ac0b0b40 2023-11-24T08:23:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.586+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f36a804d450 con 0x7f36ac0b0b40 2023-11-24T08:23:31.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.587+0000 7f36a27fc640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 0x7f369006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:31.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.587+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f36a80aab10 con 0x7f36ac0b0b40 2023-11-24T08:23:31.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.588+0000 7f36b1089640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 0x7f369006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:31.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.588+0000 7f36b1089640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 0x7f369006f820 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f36ac159b20 tx=0x7f369c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:31.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.591+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36a80b4070 con 0x7f36ac0b0b40 2023-11-24T08:23:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.785+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36ac0ca020 con 0x7f36ac0b0b40 2023-11-24T08:23:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.786+0000 7f36a27fc640 1 -- 172.21.15.107:0/2136145803 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v221) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36a8035020 con 0x7f36ac0b0b40 2023-11-24T08:23:31.788 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.788+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 msgr2=0x7f369006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.788+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 0x7f369006f820 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f36ac159b20 tx=0x7f369c009040 comp rx=0 tx=0).stop 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.788+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 msgr2=0x7f36ac1556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.788+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f36a802b8f0 tx=0x7f36a80046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 shutdown_connections 2023-11-24T08:23:31.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f369006d360 0x7f369006f820 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36ac15e5e0 0x7f36ac1f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ac15a1c0 0x7f36ac1f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 --2- 172.21.15.107:0/2136145803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36ac0b0b40 0x7f36ac1556c0 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 >> 172.21.15.107:0/2136145803 conn(0x7f36ac0cbb40 msgr2=0x7f36ac15cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 shutdown_connections 2023-11-24T08:23:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:31.789+0000 7f36b288c640 1 -- 172.21.15.107:0/2136145803 wait complete. 2023-11-24T08:23:31.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:23:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.013+0000 7fa3db851640 1 Processor -- start 2023-11-24T08:23:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.014+0000 7fa3db851640 1 -- start start 2023-11-24T08:23:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.015+0000 7fa3db851640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40ccbd0 0x7fa3d41660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.015+0000 7fa3db851640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d41655f0 con 0x7fa3d40c9650 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.015+0000 7fa3db851640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40cc380 con 0x7fa3d40ccfb0 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.015+0000 7fa3db851640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d4162b30 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.015+0000 7fa3d95ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40ccbd0 0x7fa3d41660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.016+0000 7fa3d95ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3d40ccbd0 0x7fa3d41660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40094/0 (socket says 172.21.15.107:40094) 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.016+0000 7fa3d95ed640 1 -- 172.21.15.107:0/64782913 learned_addr learned my addr 172.21.15.107:0/64782913 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.016+0000 7fa3d95ed640 1 -- 172.21.15.107:0/64782913 >> v1:172.21.15.120:6789/0 conn(0x7fa3d40ccfb0 legacy=0x7fa3d4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.016+0000 7fa3d95ed640 1 -- 172.21.15.107:0/64782913 >> v1:172.21.15.107:6789/0 conn(0x7fa3d40c9650 legacy=0x7fa3d40c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.016+0000 7fa3d95ed640 1 -- 172.21.15.107:0/64782913 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4162e10 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.017+0000 7fa3d95ed640 1 --2- 172.21.15.107:0/64782913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d41660b0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf99601fde88faf9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.017+0000 7fa3cbfff640 1 -- 172.21.15.107:0/64782913 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0030860 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.017+0000 7fa3cbfff640 1 -- 172.21.15.107:0/64782913 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d00309c0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.017+0000 7fa3cbfff640 1 -- 172.21.15.107:0/64782913 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0030d50 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 -- 172.21.15.107:0/64782913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 msgr2=0x7fa3d41660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 --2- 172.21.15.107:0/64782913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d41660b0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 -- 172.21.15.107:0/64782913 shutdown_connections 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 --2- 172.21.15.107:0/64782913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d41660b0 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 -- 172.21.15.107:0/64782913 >> 172.21.15.107:0/64782913 conn(0x7fa3d40b0c50 msgr2=0x7fa3d40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 -- 172.21.15.107:0/64782913 shutdown_connections 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.018+0000 7fa3db851640 1 -- 172.21.15.107:0/64782913 wait complete. 2023-11-24T08:23:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.020+0000 7fa3db851640 1 Processor -- start 2023-11-24T08:23:32.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.020+0000 7fa3db851640 1 -- start start 2023-11-24T08:23:32.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40c9650 0x7fa3d4164fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 0x7fa3d415c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4158d40 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d41604c0 con 0x7fa3d40ccfb0 2023-11-24T08:23:32.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3db851640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d41edba0 con 0x7fa3d40c9650 2023-11-24T08:23:32.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3d8dec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3d8dec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40096/0 (socket says 172.21.15.107:40096) 2023-11-24T08:23:32.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.021+0000 7fa3d8dec640 1 -- 172.21.15.107:0/2350404722 learned_addr learned my addr 172.21.15.107:0/2350404722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d9dee640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 0x7fa3d415c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d95ed640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40c9650 0x7fa3d4164fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 -- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40c9650 msgr2=0x7fa3d4164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40c9650 0x7fa3d4164fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 -- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 msgr2=0x7fa3d415c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 0x7fa3d415c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 -- 172.21.15.107:0/2350404722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00094d0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d9dee640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 0x7fa3d415c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3d8dec640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c410 tx=0x7fa3c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40035f0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4003750 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d41ede80 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4003b40 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.022+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d41636f0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.023+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d4160650 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.025+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fa3c4003ca0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.025+0000 7fa3c9ffb640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 0x7fa3b806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.026+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa3c4088000 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.026+0000 7fa3d95ed640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 0x7fa3b806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.026+0000 7fa3d95ed640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 0x7fa3b806f820 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0036000 tx=0x7fa3d003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.030+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3c4054b10 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.234+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa3d404e8b0 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.234+0000 7fa3c9ffb640 1 -- 172.21.15.107:0/2350404722 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v221) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa3c4017070 con 0x7fa3d40ccbd0 2023-11-24T08:23:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:23:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 msgr2=0x7fa3b806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 0x7fa3b806f820 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0036000 tx=0x7fa3d003b040 comp rx=0 tx=0).stop 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 msgr2=0x7fa3d4158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c410 tx=0x7fa3c400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 shutdown_connections 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa3b806d360 0x7fa3b806f820 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40ccfb0 0x7fa3d415c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d40ccbd0 0x7fa3d4158800 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 --2- 172.21.15.107:0/2350404722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40c9650 0x7fa3d4164fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 >> 172.21.15.107:0/2350404722 conn(0x7fa3d40b0c50 msgr2=0x7fa3d40c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 shutdown_connections 2023-11-24T08:23:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.237+0000 7fa3db851640 1 -- 172.21.15.107:0/2350404722 wait complete. 2023-11-24T08:23:32.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:23:32.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:32.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:23:32.373 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:23:32.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:32.373 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:23:32.427 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:32.428 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:32.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:23:32.487 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:32.487 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:32.547 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:23:32.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.766+0000 7f5590ecf640 1 Processor -- start 2023-11-24T08:23:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.766+0000 7f5590ecf640 1 -- start start 2023-11-24T08:23:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.767+0000 7f5590ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0cfb60 0x7f558c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.767+0000 7f5590ecf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f558c15dc80 0x7f558c15e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.767+0000 7f5590ecf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c0ce0b0 con 0x7f558c157c70 2023-11-24T08:23:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f5590ecf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c161960 con 0x7f558c15dc80 2023-11-24T08:23:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f5590ecf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c15e670 con 0x7f558c0cfb60 2023-11-24T08:23:32.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f558c15dc80 0x7f558c15e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f558c15dc80 0x7f558c15e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40108/0 (socket says 172.21.15.107:40108) 2023-11-24T08:23:32.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 -- 172.21.15.107:0/1947008521 learned_addr learned my addr 172.21.15.107:0/1947008521 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:32.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f5589d9b640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0cfb60 0x7f558c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 -- 172.21.15.107:0/1947008521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0cfb60 msgr2=0x7f558c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:32.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0cfb60 0x7f558c161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.768+0000 7f558ad9d640 1 -- 172.21.15.107:0/1947008521 >> v1:172.21.15.120:6789/0 conn(0x7f558c157c70 legacy=0x7f558c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:32.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.769+0000 7f558ad9d640 1 -- 172.21.15.107:0/1947008521 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c15e8d0 con 0x7f558c15dc80 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.770+0000 7f558ad9d640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c15dc80 0x7f558c15e130 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f5584009e50 tx=0x7f558402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f144ad546290218b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.770+0000 7f558959a640 1 -- 172.21.15.107:0/1947008521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558402c9e0 con 0x7f558c15dc80 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.770+0000 7f558959a640 1 -- 172.21.15.107:0/1947008521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558402cb40 con 0x7f558c15dc80 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.770+0000 7f558959a640 1 -- 172.21.15.107:0/1947008521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55840343f0 con 0x7f558c15dc80 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 -- 172.21.15.107:0/1947008521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c15dc80 msgr2=0x7f558c15e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c15dc80 0x7f558c15e130 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f5584009e50 tx=0x7f558402be40 comp rx=0 tx=0).stop 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 -- 172.21.15.107:0/1947008521 shutdown_connections 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c15dc80 0x7f558c15e130 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 --2- 172.21.15.107:0/1947008521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c0cfb60 0x7f558c161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 -- 172.21.15.107:0/1947008521 >> 172.21.15.107:0/1947008521 conn(0x7f558c06a920 msgr2=0x7f558c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 -- 172.21.15.107:0/1947008521 shutdown_connections 2023-11-24T08:23:32.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.771+0000 7f5590ecf640 1 -- 172.21.15.107:0/1947008521 wait complete. 2023-11-24T08:23:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.773+0000 7f5590ecf640 1 Processor -- start 2023-11-24T08:23:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.773+0000 7f5590ecf640 1 -- start start 2023-11-24T08:23:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 0x7f558c14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15dc80 0x7f558c14f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c151d80 con 0x7f558c0cfb60 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c152750 con 0x7f558c15dc80 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5590ecf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c1511b0 con 0x7f558c157c70 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40118/0 (socket says 172.21.15.107:40118) 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 -- 172.21.15.107:0/3349782770 learned_addr learned my addr 172.21.15.107:0/3349782770 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5589d9b640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 0x7f558c14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 -- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 msgr2=0x7f558c14bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 0x7f558c14bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 -- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15dc80 msgr2=0x7f558c14f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15dc80 0x7f558c14f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f558a59c640 1 -- 172.21.15.107:0/3349782770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584009b30 con 0x7f558c0cfb60 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.774+0000 7f5589d9b640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 0x7f558c14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f558a59c640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b3f0 tx=0x7f557c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c011830 con 0x7f558c0cfb60 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c011990 con 0x7f558c0cfb60 2023-11-24T08:23:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c011d40 con 0x7f558c0cfb60 2023-11-24T08:23:32.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c14c0f0 con 0x7f558c0cfb60 2023-11-24T08:23:32.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.775+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c151730 con 0x7f558c0cfb60 2023-11-24T08:23:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.776+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c067470 con 0x7f558c0cfb60 2023-11-24T08:23:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.777+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f557c01a400 con 0x7f558c0cfb60 2023-11-24T08:23:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.778+0000 7f557affd640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 0x7f556c06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.778+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f557c0886e0 con 0x7f558c0cfb60 2023-11-24T08:23:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.778+0000 7f5589d9b640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 0x7f556c06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.779+0000 7f5589d9b640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 0x7f556c06f820 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f558c0ced40 tx=0x7f5580008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:32.782+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f557c0551f0 con 0x7f558c0cfb60 2023-11-24T08:23:33.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.012+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f558c0cd910 con 0x7f558c0cfb60 2023-11-24T08:23:33.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.013+0000 7f557affd640 1 -- 172.21.15.107:0/3349782770 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f557c015090 con 0x7f558c0cfb60 2023-11-24T08:23:33.016 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:33.016 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-11-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 msgr2=0x7f556c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 0x7f556c06f820 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f558c0ced40 tx=0x7f5580008040 comp rx=0 tx=0).stop 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 msgr2=0x7f558c152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b3f0 tx=0x7f557c00b8b0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 shutdown_connections 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f556c06d360 0x7f556c06f820 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f558c15dc80 0x7f558c14f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f558c157c70 0x7f558c14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 --2- 172.21.15.107:0/3349782770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f558c0cfb60 0x7f558c152210 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 >> 172.21.15.107:0/3349782770 conn(0x7f558c06a920 msgr2=0x7f558c06c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 shutdown_connections 2023-11-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.016+0000 7f5590ecf640 1 -- 172.21.15.107:0/3349782770 wait complete. 2023-11-24T08:23:33.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:33.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.234+0000 7f6fde8bb640 1 Processor -- start 2023-11-24T08:23:33.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.235+0000 7f6fde8bb640 1 -- start start 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fde8bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd815d4e0 0x7f6fd80d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fde8bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fd80d38e0 0x7f6fd80cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fde8bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80ce990 con 0x7f6fd80d0070 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fde8bb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd80d4110 con 0x7f6fd815d4e0 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fde8bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd80d04e0 con 0x7f6fd80d38e0 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fdd8b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd815d4e0 0x7f6fd80d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fdd8b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6fd815d4e0 0x7f6fd80d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40126/0 (socket says 172.21.15.107:40126) 2023-11-24T08:23:33.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.236+0000 7f6fdd8b9640 1 -- 172.21.15.107:0/3673104262 learned_addr learned my addr 172.21.15.107:0/3673104262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:33.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.237+0000 7f6fdd8b9640 1 -- 172.21.15.107:0/3673104262 >> v2:172.21.15.120:3300/0 conn(0x7f6fd80d38e0 msgr2=0x7f6fd80cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:33.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.237+0000 7f6fdd8b9640 1 --2- 172.21.15.107:0/3673104262 >> v2:172.21.15.120:3300/0 conn(0x7f6fd80d38e0 0x7f6fd80cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.237+0000 7f6fdd8b9640 1 -- 172.21.15.107:0/3673104262 >> v1:172.21.15.120:6789/0 conn(0x7f6fd80d0070 legacy=0x7f6fd80d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:33.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.237+0000 7f6fdd8b9640 1 -- 172.21.15.107:0/3673104262 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80cec70 con 0x7f6fd815d4e0 2023-11-24T08:23:33.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.238+0000 7f6fdd8b9640 1 --2- 172.21.15.107:0/3673104262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd80d1e20 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80b9280 tx=0x7f6fcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31858b4026dcb698 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.238+0000 7f6fdc8b7640 1 -- 172.21.15.107:0/3673104262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc031870 con 0x7f6fd815d4e0 2023-11-24T08:23:33.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.238+0000 7f6fdc8b7640 1 -- 172.21.15.107:0/3673104262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0319d0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.238+0000 7f6fdc8b7640 1 -- 172.21.15.107:0/3673104262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc031d60 con 0x7f6fd815d4e0 2023-11-24T08:23:33.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 -- 172.21.15.107:0/3673104262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 msgr2=0x7f6fd80d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/3673104262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd80d1e20 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80b9280 tx=0x7f6fcc02b910 comp rx=0 tx=0).stop 2023-11-24T08:23:33.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 -- 172.21.15.107:0/3673104262 shutdown_connections 2023-11-24T08:23:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/3673104262 >> v2:172.21.15.120:3300/0 conn(0x7f6fd80d38e0 0x7f6fd80cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/3673104262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd80d1e20 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 -- 172.21.15.107:0/3673104262 >> 172.21.15.107:0/3673104262 conn(0x7f6fd809c510 msgr2=0x7f6fd80d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 -- 172.21.15.107:0/3673104262 shutdown_connections 2023-11-24T08:23:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.239+0000 7f6fde8bb640 1 -- 172.21.15.107:0/3673104262 wait complete. 2023-11-24T08:23:33.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.241+0000 7f6fde8bb640 1 Processor -- start 2023-11-24T08:23:33.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.241+0000 7f6fde8bb640 1 -- start start 2023-11-24T08:23:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd80d0070 0x7f6fd80c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 0x7f6fd8160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80b77f0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80b79f0 con 0x7f6fd80d38e0 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde8bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80b8770 con 0x7f6fd80d0070 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde0ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde0ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40130/0 (socket says 172.21.15.107:40130) 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde0ba640 1 -- 172.21.15.107:0/166738739 learned_addr learned my addr 172.21.15.107:0/166738739 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fdd8b9640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd80d0070 0x7f6fd80c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fdd0b8640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 0x7f6fd8160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.242+0000 7f6fde0ba640 1 -- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd80d0070 msgr2=0x7f6fd80c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde0ba640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd80d0070 0x7f6fd80c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde0ba640 1 -- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 msgr2=0x7f6fd8160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde0ba640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 0x7f6fd8160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde0ba640 1 -- 172.21.15.107:0/166738739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0094d0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fdd0b8640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 0x7f6fd8160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde0ba640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c780 tx=0x7f6fd000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0003d80 con 0x7f6fd815d4e0 2023-11-24T08:23:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000ce90 con 0x7f6fd815d4e0 2023-11-24T08:23:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00052b0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd80c72e0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.243+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd80b7f70 con 0x7f6fd815d4e0 2023-11-24T08:23:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.244+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd804e840 con 0x7f6fd815d4e0 2023-11-24T08:23:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.245+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f6fd0005410 con 0x7f6fd815d4e0 2023-11-24T08:23:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.246+0000 7f6fca7fc640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 0x7f6fbc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.246+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6fd0087db0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.247+0000 7f6fdd8b9640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 0x7f6fbc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.247+0000 7f6fdd8b9640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 0x7f6fbc06f820 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc036000 tx=0x7f6fcc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.251+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fd00548c0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.439+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd806cbf0 con 0x7f6fd815d4e0 2023-11-24T08:23:33.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.441+0000 7f6fca7fc640 1 -- 172.21.15.107:0/166738739 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f6fd0017070 con 0x7f6fd815d4e0 2023-11-24T08:23:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:33.444 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":221,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:27.170359+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":221,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:33.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 msgr2=0x7f6fbc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 0x7f6fbc06f820 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc036000 tx=0x7f6fcc03c040 comp rx=0 tx=0).stop 2023-11-24T08:23:33.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 msgr2=0x7f6fd81f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c780 tx=0x7f6fd000cc40 comp rx=0 tx=0).stop 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 shutdown_connections 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6fbc06d360 0x7f6fbc06f820 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fd815d4e0 0x7f6fd81f6a00 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fd80d38e0 0x7f6fd8160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 --2- 172.21.15.107:0/166738739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fd80d0070 0x7f6fd80c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 >> 172.21.15.107:0/166738739 conn(0x7f6fd809c510 msgr2=0x7f6fd80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 shutdown_connections 2023-11-24T08:23:33.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.444+0000 7f6fde8bb640 1 -- 172.21.15.107:0/166738739 wait complete. 2023-11-24T08:23:33.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:22:49.030320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:22:50.221299+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 221, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:23:33.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.666+0000 7f4937a90640 1 Processor -- start 2023-11-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.666+0000 7f4937a90640 1 -- start start 2023-11-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f4937a90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 0x7f49300ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f4937a90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4930140720 0x7f49300cb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f4937a90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4930161600 con 0x7f4930161080 2023-11-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f4937a90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49301618e0 con 0x7f4930140720 2023-11-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f4937a90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4930161bc0 con 0x7f4930140340 2023-11-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f493502b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4930140720 0x7f49300cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f493602d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4930161080 0x7f49300c60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:48578/0 (socket says 172.21.15.107:48578) 2023-11-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f493602d640 1 -- 172.21.15.107:0/3899884585 learned_addr learned my addr 172.21.15.107:0/3899884585 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.668+0000 7f493582c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 0x7f49300ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493482a640 1 -- 172.21.15.107:0/3899884585 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415637653 0 0) 0x7f4930161600 con 0x7f4930161080 2023-11-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493482a640 1 -- 172.21.15.107:0/3899884585 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f491c003520 con 0x7f4930161080 2023-11-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493582c640 1 -- 172.21.15.107:0/3899884585 >> v2:172.21.15.120:3300/0 conn(0x7f4930140720 msgr2=0x7f49300cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493582c640 1 --2- 172.21.15.107:0/3899884585 >> v2:172.21.15.120:3300/0 conn(0x7f4930140720 0x7f49300cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493582c640 1 -- 172.21.15.107:0/3899884585 >> v1:172.21.15.120:6789/0 conn(0x7f4930161080 legacy=0x7f49300c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.669+0000 7f493582c640 1 -- 172.21.15.107:0/3899884585 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930161ea0 con 0x7f4930140340 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.671+0000 7f493582c640 1 --2- 172.21.15.107:0/3899884585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 0x7f49300ce410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f492c0097a0 tx=0x7f492c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abc9eaff7ba169c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.671+0000 7f493482a640 1 -- 172.21.15.107:0/3899884585 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c031830 con 0x7f4930140340 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.671+0000 7f493482a640 1 -- 172.21.15.107:0/3899884585 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c031990 con 0x7f4930140340 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.671+0000 7f493482a640 1 -- 172.21.15.107:0/3899884585 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c031d00 con 0x7f4930140340 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 -- 172.21.15.107:0/3899884585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 msgr2=0x7f49300ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 --2- 172.21.15.107:0/3899884585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 0x7f49300ce410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f492c0097a0 tx=0x7f492c02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 -- 172.21.15.107:0/3899884585 shutdown_connections 2023-11-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 --2- 172.21.15.107:0/3899884585 >> v2:172.21.15.120:3300/0 conn(0x7f4930140720 0x7f49300cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 --2- 172.21.15.107:0/3899884585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930140340 0x7f49300ce410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 -- 172.21.15.107:0/3899884585 >> 172.21.15.107:0/3899884585 conn(0x7f49300b2e40 msgr2=0x7f49300b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 -- 172.21.15.107:0/3899884585 shutdown_connections 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.672+0000 7f4937a90640 1 -- 172.21.15.107:0/3899884585 wait complete. 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.674+0000 7f4937a90640 1 Processor -- start 2023-11-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.674+0000 7f4937a90640 1 -- start start 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4930140720 0x7f493013e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930161080 0x7f493013e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300ca060 con 0x7f4930140340 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300ca320 con 0x7f4930140720 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f4937a90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49300ca600 con 0x7f4930161080 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40142/0 (socket says 172.21.15.107:40142) 2023-11-24T08:23:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 -- 172.21.15.107:0/2754541402 learned_addr learned my addr 172.21.15.107:0/2754541402 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493602d640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930161080 0x7f493013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 -- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930161080 msgr2=0x7f493013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493502b640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4930140720 0x7f493013e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930161080 0x7f493013e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 -- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4930140720 msgr2=0x7f493013e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4930140720 0x7f493013e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.675+0000 7f493582c640 1 -- 172.21.15.107:0/2754541402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c009400 con 0x7f4930140340 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f493582c640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f492c02b910 tx=0x7f492c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c004250 con 0x7f4930140340 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c0043b0 con 0x7f4930140340 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c004520 con 0x7f4930140340 2023-11-24T08:23:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49300ca8e0 con 0x7f4930140340 2023-11-24T08:23:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.676+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49300c9730 con 0x7f4930140340 2023-11-24T08:23:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.677+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493004e8b0 con 0x7f4930140340 2023-11-24T08:23:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.678+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f492c003830 con 0x7f4930140340 2023-11-24T08:23:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.680+0000 7f49267fc640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 0x7f491406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.680+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f492c0aa1d0 con 0x7f4930140340 2023-11-24T08:23:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.680+0000 7f493502b640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 0x7f491406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.681+0000 7f493502b640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 0x7f491406f820 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f49300cbd70 tx=0x7f4920008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:33.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.685+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f492c076ce0 con 0x7f4930140340 2023-11-24T08:23:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.874+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49300c9db0 con 0x7f4930140340 2023-11-24T08:23:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.875+0000 7f49267fc640 1 -- 172.21.15.107:0/2754541402 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v221) v1 ==== 93+0+10 (secure 0 0 0) 0x7f492c048020 con 0x7f4930140340 2023-11-24T08:23:33.877 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.877+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 msgr2=0x7f491406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 0x7f491406f820 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f49300cbd70 tx=0x7f4920008040 comp rx=0 tx=0).stop 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 msgr2=0x7f49300cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f492c02b910 tx=0x7f492c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 shutdown_connections 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f491406d360 0x7f491406f820 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4930161080 0x7f493013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4930140720 0x7f493013e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 --2- 172.21.15.107:0/2754541402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4930140340 0x7f49300cae00 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 >> 172.21.15.107:0/2754541402 conn(0x7f49300b2e40 msgr2=0x7f49300c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 shutdown_connections 2023-11-24T08:23:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:33.878+0000 7f4937a90640 1 -- 172.21.15.107:0/2754541402 wait complete. 2023-11-24T08:23:33.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:23:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.100+0000 7f1772127640 1 Processor -- start 2023-11-24T08:23:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.101+0000 7f1772127640 1 -- start start 2023-11-24T08:23:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.102+0000 7f1772127640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.102+0000 7f1772127640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 0x7f176c0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.102+0000 7f1772127640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c160cd0 con 0x7f176c0ce260 2023-11-24T08:23:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.102+0000 7f1772127640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c0b8f60 con 0x7f176c0d3890 2023-11-24T08:23:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.102+0000 7f1772127640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0cdcb0 con 0x7f176c1608f0 2023-11-24T08:23:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.103+0000 7f1771926640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 0x7f176c0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.103+0000 7f1771125640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.103+0000 7f1771926640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 0x7f176c0b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42746/0 (socket says 172.21.15.107:42746) 2023-11-24T08:23:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.103+0000 7f1771926640 1 -- 172.21.15.107:0/2021133139 learned_addr learned my addr 172.21.15.107:0/2021133139 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.104+0000 7f1763fff640 1 -- 172.21.15.107:0/2021133139 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636767414 0 0) 0x7f176c160cd0 con 0x7f176c0ce260 2023-11-24T08:23:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.104+0000 7f1771125640 1 -- 172.21.15.107:0/2021133139 >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 msgr2=0x7f176c0b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.104+0000 7f1771125640 1 --2- 172.21.15.107:0/2021133139 >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 0x7f176c0b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.104+0000 7f1771125640 1 -- 172.21.15.107:0/2021133139 >> v1:172.21.15.120:6789/0 conn(0x7f176c0ce260 legacy=0x7f176c0d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.104+0000 7f1771125640 1 -- 172.21.15.107:0/2021133139 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0cdf90 con 0x7f176c1608f0 2023-11-24T08:23:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.105+0000 7f1771125640 1 --2- 172.21.15.107:0/2021133139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0d1ac0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17680097a0 tx=0x7f176802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28501f48272b8f76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.105+0000 7f1763fff640 1 -- 172.21.15.107:0/2021133139 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768031830 con 0x7f176c1608f0 2023-11-24T08:23:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1763fff640 1 -- 172.21.15.107:0/2021133139 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1768031990 con 0x7f176c1608f0 2023-11-24T08:23:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1763fff640 1 -- 172.21.15.107:0/2021133139 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768031d20 con 0x7f176c1608f0 2023-11-24T08:23:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 -- 172.21.15.107:0/2021133139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 msgr2=0x7f176c0d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 --2- 172.21.15.107:0/2021133139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0d1ac0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17680097a0 tx=0x7f176802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 -- 172.21.15.107:0/2021133139 shutdown_connections 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 --2- 172.21.15.107:0/2021133139 >> v2:172.21.15.120:3300/0 conn(0x7f176c0d3890 0x7f176c0b63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 --2- 172.21.15.107:0/2021133139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0d1ac0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 -- 172.21.15.107:0/2021133139 >> 172.21.15.107:0/2021133139 conn(0x7f176c061340 msgr2=0x7f176c061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 -- 172.21.15.107:0/2021133139 shutdown_connections 2023-11-24T08:23:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.106+0000 7f1772127640 1 -- 172.21.15.107:0/2021133139 wait complete. 2023-11-24T08:23:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.108+0000 7f1772127640 1 Processor -- start 2023-11-24T08:23:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.108+0000 7f1772127640 1 -- start start 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176c0d3890 0x7f176c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0cc1f0 con 0x7f176c0ce260 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0ca370 con 0x7f176c0d3890 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1772127640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0ca610 con 0x7f176c1608f0 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1771125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1771125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40146/0 (socket says 172.21.15.107:40146) 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.109+0000 7f1771125640 1 -- 172.21.15.107:0/2441905076 learned_addr learned my addr 172.21.15.107:0/2441905076 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1770924640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176c0d3890 0x7f176c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 -- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 msgr2=0x7f176c0b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 -- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176c0d3890 msgr2=0x7f176c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176c0d3890 0x7f176c0ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 -- 172.21.15.107:0/2441905076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1768009400 con 0x7f176c0ce260 2023-11-24T08:23:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.110+0000 7f1771125640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f1768000c00 tx=0x7f17680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.111+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176803a830 con 0x7f176c0ce260 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.111+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176803a9b0 con 0x7f176c0ce260 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.111+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176803ad60 con 0x7f176c0ce260 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.111+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c0ca8f0 con 0x7f176c0ce260 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.111+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c0caeb0 con 0x7f176c0ce260 2023-11-24T08:23:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.112+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c04e840 con 0x7f176c0ce260 2023-11-24T08:23:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.113+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f176804d450 con 0x7f176c0ce260 2023-11-24T08:23:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.114+0000 7f1761ffb640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 0x7f175006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.114+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f17680aa840 con 0x7f176c0ce260 2023-11-24T08:23:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.115+0000 7f1770924640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 0x7f175006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.115+0000 7f1770924640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 0x7f175006f820 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f176c15dc20 tx=0x7f175c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.119+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17680773f0 con 0x7f176c0ce260 2023-11-24T08:23:34.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.309+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f176c0b5ec0 con 0x7f176c0ce260 2023-11-24T08:23:34.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.309+0000 7f1761ffb640 1 -- 172.21.15.107:0/2441905076 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v221) v1 ==== 104+0+11 (secure 0 0 0) 0x7f17680350a0 con 0x7f176c0ce260 2023-11-24T08:23:34.312 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:23:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 msgr2=0x7f175006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 0x7f175006f820 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f176c15dc20 tx=0x7f175c009040 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 msgr2=0x7f176c0cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f1768000c00 tx=0x7f17680046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 shutdown_connections 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f175006d360 0x7f175006f820 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f176c1608f0 0x7f176c0b78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176c0d3890 0x7f176c0ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 --2- 172.21.15.107:0/2441905076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f176c0ce260 0x7f176c0cc850 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 >> 172.21.15.107:0/2441905076 conn(0x7f176c061340 msgr2=0x7f176c15e970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 shutdown_connections 2023-11-24T08:23:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.312+0000 7f1772127640 1 -- 172.21.15.107:0/2441905076 wait complete. 2023-11-24T08:23:34.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:23:34.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.543+0000 7fdb89367640 1 Processor -- start 2023-11-24T08:23:34.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.544+0000 7fdb89367640 1 -- start start 2023-11-24T08:23:34.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb89367640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb840ce7f0 0x7fdb840d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb89367640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb840d1d30 0x7fdb840b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb89367640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8415daa0 con 0x7fdb8415d6c0 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb89367640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb840d41c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb89367640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840d21c0 con 0x7fdb840d1d30 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.545+0000 7fdb837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb840ce7f0 0x7fdb840d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb840ce7f0 0x7fdb840d17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40156/0 (socket says 172.21.15.107:40156) 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 -- 172.21.15.107:0/2603404772 learned_addr learned my addr 172.21.15.107:0/2603404772 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 -- 172.21.15.107:0/2603404772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb840d1d30 msgr2=0x7fdb840b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 --2- 172.21.15.107:0/2603404772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb840d1d30 0x7fdb840b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 -- 172.21.15.107:0/2603404772 >> v1:172.21.15.120:6789/0 conn(0x7fdb8415d6c0 legacy=0x7fdb840d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.546+0000 7fdb837fe640 1 -- 172.21.15.107:0/2603404772 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb840ced40 con 0x7fdb840ce7f0 2023-11-24T08:23:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.547+0000 7fdb837fe640 1 --2- 172.21.15.107:0/2603404772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb840d17f0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009870 tx=0x7fdb7802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd837c9e510447f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.547+0000 7fdb82ffd640 1 -- 172.21.15.107:0/2603404772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb780317c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.547+0000 7fdb82ffd640 1 -- 172.21.15.107:0/2603404772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb78031920 con 0x7fdb840ce7f0 2023-11-24T08:23:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.547+0000 7fdb82ffd640 1 -- 172.21.15.107:0/2603404772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78031cb0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 -- 172.21.15.107:0/2603404772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 msgr2=0x7fdb840d17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 --2- 172.21.15.107:0/2603404772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb840d17f0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009870 tx=0x7fdb7802b860 comp rx=0 tx=0).stop 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 -- 172.21.15.107:0/2603404772 shutdown_connections 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 --2- 172.21.15.107:0/2603404772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb840d1d30 0x7fdb840b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 --2- 172.21.15.107:0/2603404772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb840d17f0 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 -- 172.21.15.107:0/2603404772 >> 172.21.15.107:0/2603404772 conn(0x7fdb8409c5b0 msgr2=0x7fdb840d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 -- 172.21.15.107:0/2603404772 shutdown_connections 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.548+0000 7fdb89367640 1 -- 172.21.15.107:0/2603404772 wait complete. 2023-11-24T08:23:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.550+0000 7fdb89367640 1 Processor -- start 2023-11-24T08:23:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.550+0000 7fdb89367640 1 -- start start 2023-11-24T08:23:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 0x7fdb8415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb8415d6c0 0x7fdb840b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8415bf70 con 0x7fdb840ce7f0 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8415bc90 con 0x7fdb840d1d30 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb89367640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840b6fc0 con 0x7fdb8415d6c0 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40168/0 (socket says 172.21.15.107:40168) 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.551+0000 7fdb83fff640 1 -- 172.21.15.107:0/1087280512 learned_addr learned my addr 172.21.15.107:0/1087280512 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb88b66640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb8415d6c0 0x7fdb840b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb837fe640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 0x7fdb8415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 -- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb8415d6c0 msgr2=0x7fdb840b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb8415d6c0 0x7fdb840b7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 -- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 msgr2=0x7fdb8415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 0x7fdb8415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 -- 172.21.15.107:0/1087280512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780094d0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb837fe640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 0x7fdb8415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb83fff640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c410 tx=0x7fdb7400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb740035f0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74003750 con 0x7fdb840ce7f0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb840b72a0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.552+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74003b40 con 0x7fdb840ce7f0 2023-11-24T08:23:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.553+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb840b7fe0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.554+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb840b77c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.554+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fdb74003ca0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.555+0000 7fdb80ff9640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 0x7fdb7006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.556+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fdb74087ed0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.556+0000 7fdb837fe640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 0x7fdb7006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.556+0000 7fdb837fe640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 0x7fdb7006f820 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fdb78036000 tx=0x7fdb7803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.560+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb840b77c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.750+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdb840b77c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.751+0000 7fdb80ff9640 1 -- 172.21.15.107:0/1087280512 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v221) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdb840b77c0 con 0x7fdb840ce7f0 2023-11-24T08:23:34.753 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:23:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 msgr2=0x7fdb7006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 0x7fdb7006f820 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fdb78036000 tx=0x7fdb7803d040 comp rx=0 tx=0).stop 2023-11-24T08:23:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 msgr2=0x7fdb841548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c410 tx=0x7fdb7400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 shutdown_connections 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fdb7006d360 0x7fdb7006f820 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb8415d6c0 0x7fdb840b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb840d1d30 0x7fdb8415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 --2- 172.21.15.107:0/1087280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb840ce7f0 0x7fdb841548e0 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 >> 172.21.15.107:0/1087280512 conn(0x7fdb8409c5b0 msgr2=0x7fdb840b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 shutdown_connections 2023-11-24T08:23:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.754+0000 7fdb89367640 1 -- 172.21.15.107:0/1087280512 wait complete. 2023-11-24T08:23:34.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:34.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.992+0000 7fafc4a31640 1 Processor -- start 2023-11-24T08:23:34.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.993+0000 7fafc4a31640 1 -- start start 2023-11-24T08:23:34.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.994+0000 7fafc4a31640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fafc015d4e0 0x7fafc00d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.994+0000 7fafc4a31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fafc00d0240 0x7fafc00b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.994+0000 7fafc4a31640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00ce990 con 0x7fafc00d38e0 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.994+0000 7fafc4a31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc00d4110 con 0x7fafc00d0240 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.994+0000 7fafc4a31640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc00cec70 con 0x7fafc015d4e0 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fafc00d0240 0x7fafc00b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fafc00d0240 0x7fafc00b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40180/0 (socket says 172.21.15.107:40180) 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 -- 172.21.15.107:0/1218505212 learned_addr learned my addr 172.21.15.107:0/1218505212 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbed9d640 1 --2- 172.21.15.107:0/1218505212 >> v2:172.21.15.120:3300/0 conn(0x7fafc015d4e0 0x7fafc00d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 -- 172.21.15.107:0/1218505212 >> v2:172.21.15.120:3300/0 conn(0x7fafc015d4e0 msgr2=0x7fafc00d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 --2- 172.21.15.107:0/1218505212 >> v2:172.21.15.120:3300/0 conn(0x7fafc015d4e0 0x7fafc00d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 -- 172.21.15.107:0/1218505212 >> v1:172.21.15.120:6789/0 conn(0x7fafc00d38e0 legacy=0x7fafc00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:34.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.995+0000 7fafbf59e640 1 -- 172.21.15.107:0/1218505212 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc00cef50 con 0x7fafc00d0240 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.996+0000 7fafbf59e640 1 --2- 172.21.15.107:0/1218505212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc00b2e50 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fafc008b800 tx=0x7fafb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b492e1975be882b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.996+0000 7fafbdd9b640 1 -- 172.21.15.107:0/1218505212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031870 con 0x7fafc00d0240 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.996+0000 7fafbdd9b640 1 -- 172.21.15.107:0/1218505212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb80319d0 con 0x7fafc00d0240 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.996+0000 7fafbdd9b640 1 -- 172.21.15.107:0/1218505212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031d60 con 0x7fafc00d0240 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 -- 172.21.15.107:0/1218505212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 msgr2=0x7fafc00b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 --2- 172.21.15.107:0/1218505212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc00b2e50 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fafc008b800 tx=0x7fafb802b910 comp rx=0 tx=0).stop 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 -- 172.21.15.107:0/1218505212 shutdown_connections 2023-11-24T08:23:34.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 --2- 172.21.15.107:0/1218505212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc00b2e50 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 --2- 172.21.15.107:0/1218505212 >> v2:172.21.15.120:3300/0 conn(0x7fafc015d4e0 0x7fafc00d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 -- 172.21.15.107:0/1218505212 >> 172.21.15.107:0/1218505212 conn(0x7fafc009c510 msgr2=0x7fafc00d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 -- 172.21.15.107:0/1218505212 shutdown_connections 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.997+0000 7fafc4a31640 1 -- 172.21.15.107:0/1218505212 wait complete. 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.999+0000 7fafc4a31640 1 Processor -- start 2023-11-24T08:23:35.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:34.999+0000 7fafc4a31640 1 -- start start 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 0x7fafc00cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 0x7fafc00b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00cc310 con 0x7fafc00d0240 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00cb700 con 0x7fafc00d38e0 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafc4a31640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00cb9e0 con 0x7fafc015d4e0 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafbed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafbed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40184/0 (socket says 172.21.15.107:40184) 2023-11-24T08:23:35.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafbed9d640 1 -- 172.21.15.107:0/3717536344 learned_addr learned my addr 172.21.15.107:0/3717536344 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.000+0000 7fafbe59c640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 0x7fafc00cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbf59e640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 0x7fafc00b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 -- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 msgr2=0x7fafc00b7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 0x7fafc00b7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 -- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 msgr2=0x7fafc00cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 0x7fafc00cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 -- 172.21.15.107:0/3717536344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80094d0 con 0x7fafc00d0240 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbe59c640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 0x7fafc00cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:35.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbf59e640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 0x7fafc00b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafbed9d640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c7f0 tx=0x7fafa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8004220 con 0x7fafc00d0240 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa8004380 con 0x7fafc00d0240 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8007840 con 0x7fafc00d0240 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc00cbcc0 con 0x7fafc00d0240 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.001+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc015bb40 con 0x7fafc00d0240 2023-11-24T08:23:35.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.002+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc004e840 con 0x7fafc00d0240 2023-11-24T08:23:35.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.003+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fafa8007bb0 con 0x7fafc00d0240 2023-11-24T08:23:35.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.004+0000 7fafb77fe640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 0x7fafa006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.004+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fafa8088730 con 0x7fafc00d0240 2023-11-24T08:23:35.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.005+0000 7fafbe59c640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 0x7fafa006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.005+0000 7fafbe59c640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 0x7fafa006f820 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fafc015def0 tx=0x7fafb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.008+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafa80552e0 con 0x7fafc00d0240 2023-11-24T08:23:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.240+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc00b4420 con 0x7fafc00d0240 2023-11-24T08:23:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.241+0000 7fafb77fe640 1 -- 172.21.15.107:0/3717536344 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fafa801e020 con 0x7fafc00d0240 2023-11-24T08:23:35.243 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:35.243 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:23:35.244 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 msgr2=0x7fafa006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 0x7fafa006f820 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fafc015def0 tx=0x7fafb0009040 comp rx=0 tx=0).stop 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 msgr2=0x7fafc015b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c7f0 tx=0x7fafa800ccb0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 shutdown_connections 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fafa006d360 0x7fafa006f820 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafc015d4e0 0x7fafc00b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafc00d38e0 0x7fafc00cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 --2- 172.21.15.107:0/3717536344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafc00d0240 0x7fafc015b0f0 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 >> 172.21.15.107:0/3717536344 conn(0x7fafc009c510 msgr2=0x7fafc015c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 shutdown_connections 2023-11-24T08:23:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.244+0000 7fafc4a31640 1 -- 172.21.15.107:0/3717536344 wait complete. 2023-11-24T08:23:35.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:35.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.467+0000 7f83556d3640 1 Processor -- start 2023-11-24T08:23:35.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.468+0000 7f83556d3640 1 -- start start 2023-11-24T08:23:35.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.469+0000 7f83556d3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f835015c7f0 0x7f835006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.469+0000 7f83556d3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83500ce6c0 0x7f83500cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.469+0000 7f83556d3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500cece0 con 0x7f83500cf450 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.469+0000 7f83556d3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83500cfe40 con 0x7f835015c7f0 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.469+0000 7f83556d3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83500ceaf0 con 0x7f83500ce6c0 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f835015c7f0 0x7f835006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f835015c7f0 0x7f835006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40198/0 (socket says 172.21.15.107:40198) 2023-11-24T08:23:35.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 -- 172.21.15.107:0/861347589 learned_addr learned my addr 172.21.15.107:0/861347589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 -- 172.21.15.107:0/861347589 >> v2:172.21.15.120:3300/0 conn(0x7f83500ce6c0 msgr2=0x7f83500cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 --2- 172.21.15.107:0/861347589 >> v2:172.21.15.120:3300/0 conn(0x7f83500ce6c0 0x7f83500cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 -- 172.21.15.107:0/861347589 >> v1:172.21.15.107:6789/0 conn(0x7f83500cf450 legacy=0x7f83500cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.470+0000 7f834effd640 1 -- 172.21.15.107:0/861347589 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835006d040 con 0x7f835015c7f0 2023-11-24T08:23:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.471+0000 7f834effd640 1 --2- 172.21.15.107:0/861347589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f835006db20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f83500d00e0 tx=0x7f834002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a9b35145d1425a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f834e7fc640 1 -- 172.21.15.107:0/861347589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340030870 con 0x7f835015c7f0 2023-11-24T08:23:35.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f834e7fc640 1 -- 172.21.15.107:0/861347589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83400309d0 con 0x7f835015c7f0 2023-11-24T08:23:35.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f834e7fc640 1 -- 172.21.15.107:0/861347589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340030d60 con 0x7f835015c7f0 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 -- 172.21.15.107:0/861347589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 msgr2=0x7f835006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 --2- 172.21.15.107:0/861347589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f835006db20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f83500d00e0 tx=0x7f834002b9b0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 -- 172.21.15.107:0/861347589 shutdown_connections 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 --2- 172.21.15.107:0/861347589 >> v2:172.21.15.120:3300/0 conn(0x7f83500ce6c0 0x7f83500cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 --2- 172.21.15.107:0/861347589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f835006db20 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 -- 172.21.15.107:0/861347589 >> 172.21.15.107:0/861347589 conn(0x7f835006a920 msgr2=0x7f835006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 -- 172.21.15.107:0/861347589 shutdown_connections 2023-11-24T08:23:35.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.472+0000 7f83556d3640 1 -- 172.21.15.107:0/861347589 wait complete. 2023-11-24T08:23:35.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.474+0000 7f83556d3640 1 Processor -- start 2023-11-24T08:23:35.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.474+0000 7f83556d3640 1 -- start start 2023-11-24T08:23:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 0x7f83501483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 0x7f83500c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83500ace10 con 0x7f835015c7f0 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83500ad0f0 con 0x7f83500ce6c0 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.475+0000 7f83556d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8350155de0 con 0x7f83500cf450 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40200/0 (socket says 172.21.15.107:40200) 2023-11-24T08:23:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 -- 172.21.15.107:0/3680340078 learned_addr learned my addr 172.21.15.107:0/3680340078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834effd640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 0x7f83501483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f83467fc640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 0x7f83500c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 -- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 msgr2=0x7f83500c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 0x7f83500c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 -- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 msgr2=0x7f83501483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 0x7f83501483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 -- 172.21.15.107:0/3680340078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400094d0 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f83467fc640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 0x7f83500c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834effd640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 0x7f83501483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.476+0000 7f834f7fe640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7f834800b9e0 tx=0x7f834800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.477+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834800caa0 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.477+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834800cc20 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.477+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348014440 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.477+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83501560c0 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.477+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83500c90d0 con 0x7f835015c7f0 2023-11-24T08:23:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.478+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83501565e0 con 0x7f835015c7f0 2023-11-24T08:23:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.479+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f834801d3b0 con 0x7f835015c7f0 2023-11-24T08:23:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.480+0000 7f8347fff640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 0x7f833006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.480+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f83480887f0 con 0x7f835015c7f0 2023-11-24T08:23:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.480+0000 7f834effd640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 0x7f833006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.481+0000 7f834effd640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 0x7f833006f820 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8340036000 tx=0x7f834003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.485+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83501565e0 con 0x7f835015c7f0 2023-11-24T08:23:35.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.677+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83501565e0 con 0x7f835015c7f0 2023-11-24T08:23:35.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.679+0000 7f8347fff640 1 -- 172.21.15.107:0/3680340078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f83501565e0 con 0x7f835015c7f0 2023-11-24T08:23:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:35.681 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":222,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:35.160197+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":221,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.681+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 msgr2=0x7f833006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.681+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 0x7f833006f820 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8340036000 tx=0x7f834003b040 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 msgr2=0x7f8350157700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7f834800b9e0 tx=0x7f834800bea0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 shutdown_connections 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f833006d360 0x7f833006f820 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f835015c7f0 0x7f8350157700 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83500cf450 0x7f83500c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 --2- 172.21.15.107:0/3680340078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83500ce6c0 0x7f83501483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 >> 172.21.15.107:0/3680340078 conn(0x7f835006a920 msgr2=0x7f83500cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 shutdown_connections 2023-11-24T08:23:35.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.682+0000 7f83556d3640 1 -- 172.21.15.107:0/3680340078 wait complete. 2023-11-24T08:23:35.702 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:23:35.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.915+0000 7fe09359e640 1 Processor -- start 2023-11-24T08:23:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.915+0000 7fe09359e640 1 -- start start 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.916+0000 7fe09359e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe094144b30 0x7fe09415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.916+0000 7fe09359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe0940b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.916+0000 7fe09359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09415bf80 con 0x7fe0940b1de0 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.916+0000 7fe09359e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940b1560 con 0x7fe094144b30 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.916+0000 7fe09359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b0fb0 con 0x7fe09415c410 2023-11-24T08:23:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe094144b30 0x7fe09415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe094144b30 0x7fe09415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40210/0 (socket says 172.21.15.107:40210) 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 -- 172.21.15.107:0/1085712105 learned_addr learned my addr 172.21.15.107:0/1085712105 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe08bfff640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe0940b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 -- 172.21.15.107:0/1085712105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 msgr2=0x7fe0940b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe0940b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 -- 172.21.15.107:0/1085712105 >> v1:172.21.15.107:6789/0 conn(0x7fe0940b1de0 legacy=0x7fe09406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe091d9b640 1 -- 172.21.15.107:0/1085712105 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123665365 0 0) 0x7fe09415bf80 con 0x7fe0940b1de0 2023-11-24T08:23:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.917+0000 7fe09259c640 1 -- 172.21.15.107:0/1085712105 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940b1280 con 0x7fe094144b30 2023-11-24T08:23:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.918+0000 7fe09259c640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe094144b30 0x7fe09415f920 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe08402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7dc47317e5568c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.919+0000 7fe091d9b640 1 -- 172.21.15.107:0/1085712105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084030840 con 0x7fe094144b30 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.919+0000 7fe091d9b640 1 -- 172.21.15.107:0/1085712105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0840309a0 con 0x7fe094144b30 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.919+0000 7fe091d9b640 1 -- 172.21.15.107:0/1085712105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084030d30 con 0x7fe094144b30 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.919+0000 7fe09359e640 1 -- 172.21.15.107:0/1085712105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe094144b30 msgr2=0x7fe09415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.919+0000 7fe09359e640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe094144b30 0x7fe09415f920 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe08402b980 comp rx=0 tx=0).stop 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 -- 172.21.15.107:0/1085712105 shutdown_connections 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe0940b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 --2- 172.21.15.107:0/1085712105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe094144b30 0x7fe09415f920 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 -- 172.21.15.107:0/1085712105 >> 172.21.15.107:0/1085712105 conn(0x7fe0940cb830 msgr2=0x7fe0940cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 -- 172.21.15.107:0/1085712105 shutdown_connections 2023-11-24T08:23:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.920+0000 7fe09359e640 1 -- 172.21.15.107:0/1085712105 wait complete. 2023-11-24T08:23:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.921+0000 7fe09359e640 1 Processor -- start 2023-11-24T08:23:35.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 -- start start 2023-11-24T08:23:35.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe094144b30 0x7fe0940c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe09413bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f6b80 con 0x7fe0940b1de0 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f6e20 con 0x7fe094144b30 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.922+0000 7fe09359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f7100 con 0x7fe09415c410 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40220/0 (socket says 172.21.15.107:40220) 2023-11-24T08:23:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 -- 172.21.15.107:0/2452350757 learned_addr learned my addr 172.21.15.107:0/2452350757 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe08bfff640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe094144b30 0x7fe0940c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 -- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 msgr2=0x7fe09413bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe09413bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 -- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe094144b30 msgr2=0x7fe0940c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe094144b30 0x7fe0940c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.923+0000 7fe09259c640 1 -- 172.21.15.107:0/2452350757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840094d0 con 0x7fe0940b1de0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe09259c640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe084004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084039c90 con 0x7fe0940b1de0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe084039df0 con 0x7fe0940b1de0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0941f73e0 con 0x7fe0940b1de0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084043620 con 0x7fe0940b1de0 2023-11-24T08:23:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.924+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0940c8f20 con 0x7fe0940b1de0 2023-11-24T08:23:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.925+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0941f7900 con 0x7fe0940b1de0 2023-11-24T08:23:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.926+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fe084043780 con 0x7fe0940b1de0 2023-11-24T08:23:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.927+0000 7fe08affd640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 0x7fe07406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.927+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe08403e070 con 0x7fe0940b1de0 2023-11-24T08:23:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.927+0000 7fe08bfff640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 0x7fe07406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.928+0000 7fe08bfff640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 0x7fe07406f820 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fe07c006fd0 tx=0x7fe07c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:35.931+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0941f7900 con 0x7fe0940b1de0 2023-11-24T08:23:36.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.163+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0941f7900 con 0x7fe0940b1de0 2023-11-24T08:23:36.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.164+0000 7fe08affd640 1 -- 172.21.15.107:0/2452350757 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fe0941f7900 con 0x7fe0940b1de0 2023-11-24T08:23:36.167 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:36.167 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:23:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-11-24T08:23:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 msgr2=0x7fe07406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 0x7fe07406f820 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fe07c006fd0 tx=0x7fe07c008040 comp rx=0 tx=0).stop 2023-11-24T08:23:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 msgr2=0x7fe09415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe084004730 comp rx=0 tx=0).stop 2023-11-24T08:23:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 shutdown_connections 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe07406d360 0x7fe07406f820 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415c410 0x7fe09413bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe094144b30 0x7fe0940c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 --2- 172.21.15.107:0/2452350757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0940b1de0 0x7fe09415a670 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 >> 172.21.15.107:0/2452350757 conn(0x7fe0940cb830 msgr2=0x7fe0941439f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 shutdown_connections 2023-11-24T08:23:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.167+0000 7fe09359e640 1 -- 172.21.15.107:0/2452350757 wait complete. 2023-11-24T08:23:36.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.389+0000 7ff5e9b73640 1 Processor -- start 2023-11-24T08:23:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.389+0000 7ff5e9b73640 1 -- start start 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e9b73640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e4161080 0x7ff5e40b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e9b73640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5e40b0fb0 0x7ff5e40cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e9b73640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e4161510 con 0x7ff5e4143870 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e9b73640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e4070460 con 0x7ff5e4161080 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e9b73640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e40b1420 con 0x7ff5e40b0fb0 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e4161080 0x7ff5e40b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5e4161080 0x7ff5e40b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40226/0 (socket says 172.21.15.107:40226) 2023-11-24T08:23:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 -- 172.21.15.107:0/28408632 learned_addr learned my addr 172.21.15.107:0/28408632 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 -- 172.21.15.107:0/28408632 >> v2:172.21.15.120:3300/0 conn(0x7ff5e40b0fb0 msgr2=0x7ff5e40cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 --2- 172.21.15.107:0/28408632 >> v2:172.21.15.120:3300/0 conn(0x7ff5e40b0fb0 0x7ff5e40cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 -- 172.21.15.107:0/28408632 >> v1:172.21.15.120:6789/0 conn(0x7ff5e4143870 legacy=0x7ff5e4146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.391+0000 7ff5e3fff640 1 -- 172.21.15.107:0/28408632 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e41617f0 con 0x7ff5e4161080 2023-11-24T08:23:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.392+0000 7ff5e3fff640 1 --2- 172.21.15.107:0/28408632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e4161080 0x7ff5e40b0a70 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40705a0 tx=0x7ff5d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e88a715212921c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.393+0000 7ff5e37fe640 1 -- 172.21.15.107:0/28408632 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0031870 con 0x7ff5e4161080 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.393+0000 7ff5e37fe640 1 -- 172.21.15.107:0/28408632 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d00319d0 con 0x7ff5e4161080 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.393+0000 7ff5e37fe640 1 -- 172.21.15.107:0/28408632 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0031d60 con 0x7ff5e4161080 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.393+0000 7ff5e9b73640 1 -- 172.21.15.107:0/28408632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e4161080 msgr2=0x7ff5e40b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.393+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/28408632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e4161080 0x7ff5e40b0a70 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40705a0 tx=0x7ff5d002b910 comp rx=0 tx=0).stop 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 -- 172.21.15.107:0/28408632 shutdown_connections 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/28408632 >> v2:172.21.15.120:3300/0 conn(0x7ff5e40b0fb0 0x7ff5e40cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/28408632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e4161080 0x7ff5e40b0a70 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 -- 172.21.15.107:0/28408632 >> 172.21.15.107:0/28408632 conn(0x7ff5e40cb830 msgr2=0x7ff5e40cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 -- 172.21.15.107:0/28408632 shutdown_connections 2023-11-24T08:23:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.394+0000 7ff5e9b73640 1 -- 172.21.15.107:0/28408632 wait complete. 2023-11-24T08:23:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.395+0000 7ff5e9b73640 1 Processor -- start 2023-11-24T08:23:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.395+0000 7ff5e9b73640 1 -- start start 2023-11-24T08:23:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e40b0fb0 0x7ff5e415f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 0x7ff5e4141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415fdc0 con 0x7ff5e415c860 2023-11-24T08:23:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b1b60 con 0x7ff5e4143870 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.396+0000 7ff5e9b73640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b1e40 con 0x7ff5e40b0fb0 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40230/0 (socket says 172.21.15.107:40230) 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 -- 172.21.15.107:0/1674768405 learned_addr learned my addr 172.21.15.107:0/1674768405 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e8b71640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e40b0fb0 0x7ff5e415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e3fff640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 0x7ff5e4141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 -- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e40b0fb0 msgr2=0x7ff5e415f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e40b0fb0 0x7ff5e415f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 -- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 msgr2=0x7ff5e4141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 0x7ff5e4141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e9372640 1 -- 172.21.15.107:0/1674768405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00094d0 con 0x7ff5e415c860 2023-11-24T08:23:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.397+0000 7ff5e3fff640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 0x7ff5e4141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:36.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e9372640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00b970 tx=0x7ff5dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00cb40 con 0x7ff5e415c860 2023-11-24T08:23:36.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc00ccc0 con 0x7ff5e415c860 2023-11-24T08:23:36.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc014680 con 0x7ff5e415c860 2023-11-24T08:23:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e40b2120 con 0x7ff5e415c860 2023-11-24T08:23:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.398+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e40b26e0 con 0x7ff5e415c860 2023-11-24T08:23:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.399+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4067990 con 0x7ff5e415c860 2023-11-24T08:23:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.400+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7ff5dc01d3b0 con 0x7ff5e415c860 2023-11-24T08:23:36.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.402+0000 7ff5e17fa640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 0x7ff5c806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.402+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff5dc0887e0 con 0x7ff5e415c860 2023-11-24T08:23:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.402+0000 7ff5e8b71640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 0x7ff5c806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.403+0000 7ff5e8b71640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 0x7ff5c806f820 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80015d0 tx=0x7ff5d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.406+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5dc055390 con 0x7ff5e415c860 2023-11-24T08:23:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.633+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e404e840 con 0x7ff5e415c860 2023-11-24T08:23:36.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.633+0000 7ff5e17fa640 1 -- 172.21.15.107:0/1674768405 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4936 (secure 0 0 0) 0x7ff5dc019020 con 0x7ff5e415c860 2023-11-24T08:23:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:36.636 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:22:51.212515+0000","modified":"2023-11-24T08:22:54.271878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12213":{"gid":12213,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/1864313322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1864313322},{"type":"v1","addr":"172.21.15.107:6837","nonce":1864313322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-11-24T08:23:36.636 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-11-24T08:23:36.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 msgr2=0x7ff5c806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 0x7ff5c806f820 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80015d0 tx=0x7ff5d8009040 comp rx=0 tx=0).stop 2023-11-24T08:23:36.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 msgr2=0x7ff5e4142220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00b970 tx=0x7ff5dc00be30 comp rx=0 tx=0).stop 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 shutdown_connections 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5c806d360 0x7ff5c806f820 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5e415c860 0x7ff5e4142220 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e4143870 0x7ff5e4141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 --2- 172.21.15.107:0/1674768405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e40b0fb0 0x7ff5e415f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 >> 172.21.15.107:0/1674768405 conn(0x7ff5e40cb830 msgr2=0x7ff5e4070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 shutdown_connections 2023-11-24T08:23:36.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.636+0000 7ff5e9b73640 1 -- 172.21.15.107:0/1674768405 wait complete. 2023-11-24T08:23:36.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:36.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.865+0000 7f754e4cf640 1 Processor -- start 2023-11-24T08:23:36.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.865+0000 7f754e4cf640 1 -- start start 2023-11-24T08:23:36.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.866+0000 7f754e4cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7548157c70 0x7f7548161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.866+0000 7f754e4cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815e390 0x7f75480cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.866+0000 7f754e4cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75480ce0b0 con 0x7f75480cf450 2023-11-24T08:23:36.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.866+0000 7f754e4cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75480cfe40 con 0x7f7548157c70 2023-11-24T08:23:36.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.866+0000 7f754e4cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7548161960 con 0x7f754815e390 2023-11-24T08:23:36.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.867+0000 7f7547fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7548157c70 0x7f7548161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.867+0000 7f7547fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7548157c70 0x7f7548161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:42818/0 (socket says 172.21.15.107:42818) 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.867+0000 7f7547fff640 1 -- 172.21.15.107:0/3505573192 learned_addr learned my addr 172.21.15.107:0/3505573192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.867+0000 7f754ca6c640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815e390 0x7f75480cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.868+0000 7f7547fff640 1 -- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815e390 msgr2=0x7f75480cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.868+0000 7f7547fff640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815e390 0x7f75480cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.868+0000 7f7547fff640 1 -- 172.21.15.107:0/3505573192 >> v1:172.21.15.120:6789/0 conn(0x7f75480cf450 legacy=0x7f754815dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.868+0000 7f7547fff640 1 -- 172.21.15.107:0/3505573192 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754815e8d0 con 0x7f7548157c70 2023-11-24T08:23:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.869+0000 7f7547fff640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f7548161df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7538009870 tx=0x7f753802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc1357dae3c635a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f75477fe640 1 -- 172.21.15.107:0/3505573192 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538031640 con 0x7f7548157c70 2023-11-24T08:23:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f75477fe640 1 -- 172.21.15.107:0/3505573192 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75380317a0 con 0x7f7548157c70 2023-11-24T08:23:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f75477fe640 1 -- 172.21.15.107:0/3505573192 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538031b30 con 0x7f7548157c70 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 -- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 msgr2=0x7f7548161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f7548161df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7538009870 tx=0x7f753802b8f0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 -- 172.21.15.107:0/3505573192 shutdown_connections 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815e390 0x7f75480cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3505573192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f7548161df0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 -- 172.21.15.107:0/3505573192 >> 172.21.15.107:0/3505573192 conn(0x7f754806a920 msgr2=0x7f754806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 -- 172.21.15.107:0/3505573192 shutdown_connections 2023-11-24T08:23:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.870+0000 7f754e4cf640 1 -- 172.21.15.107:0/3505573192 wait complete. 2023-11-24T08:23:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.872+0000 7f754e4cf640 1 Processor -- start 2023-11-24T08:23:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.872+0000 7f754e4cf640 1 -- start start 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75480cf450 0x7f7548099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f75480c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75480c9f20 con 0x7f754815e390 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7548153900 con 0x7f7548157c70 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754e4cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480c93a0 con 0x7f75480cf450 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754ca6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754ca6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40242/0 (socket says 172.21.15.107:40242) 2023-11-24T08:23:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.873+0000 7f754ca6c640 1 -- 172.21.15.107:0/3523927399 learned_addr learned my addr 172.21.15.107:0/3523927399 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f753ffff640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f75480c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f7547fff640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75480cf450 0x7f7548099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 -- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75480cf450 msgr2=0x7f7548099140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75480cf450 0x7f7548099140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 -- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 msgr2=0x7f75480c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f75480c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 -- 172.21.15.107:0/3523927399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380094d0 con 0x7f754815e390 2023-11-24T08:23:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f754ca6c640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f754000efc0 tx=0x7f754000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.874+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754000cd20 con 0x7f754815e390 2023-11-24T08:23:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.875+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754000f040 con 0x7f754815e390 2023-11-24T08:23:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.875+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75480c9680 con 0x7f754815e390 2023-11-24T08:23:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.875+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754000cea0 con 0x7f754815e390 2023-11-24T08:23:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.875+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75480c9c40 con 0x7f754815e390 2023-11-24T08:23:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.876+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548067470 con 0x7f754815e390 2023-11-24T08:23:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.877+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f75400148d0 con 0x7f754815e390 2023-11-24T08:23:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.878+0000 7f75457fa640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 0x7f752806f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.878+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7540089a30 con 0x7f754815e390 2023-11-24T08:23:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.878+0000 7f7547fff640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 0x7f752806f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.879+0000 7f7547fff640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 0x7f752806f850 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7538036000 tx=0x7f753803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:36.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:36.882+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75400565c0 con 0x7f754815e390 2023-11-24T08:23:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.074+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75480cd910 con 0x7f754815e390 2023-11-24T08:23:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.077+0000 7f75457fa640 1 -- 172.21.15.107:0/3523927399 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f7540018070 con 0x7f754815e390 2023-11-24T08:23:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:37.080 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":222,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:35.160197+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:22:49.030320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-11-24T08:22:50.221299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":221,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:37.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 msgr2=0x7f752806f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 0x7f752806f850 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7538036000 tx=0x7f753803c040 comp rx=0 tx=0).stop 2023-11-24T08:23:37.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 msgr2=0x7f7548099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f754000efc0 tx=0x7f754000c490 comp rx=0 tx=0).stop 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 shutdown_connections 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f752806d390 0x7f752806f850 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f754815e390 0x7f7548099f00 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7548157c70 0x7f75480c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 --2- 172.21.15.107:0/3523927399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75480cf450 0x7f7548099140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 >> 172.21.15.107:0/3523927399 conn(0x7f754806a920 msgr2=0x7f7548160010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 shutdown_connections 2023-11-24T08:23:37.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.080+0000 7f754e4cf640 1 -- 172.21.15.107:0/3523927399 wait complete. 2023-11-24T08:23:37.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:23:37.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.306+0000 7f5ac35e5640 1 Processor -- start 2023-11-24T08:23:37.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.306+0000 7f5ac35e5640 1 -- start start 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac35e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5abc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac35e5640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5abc0c57e0 0x7f5abc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac35e5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc15eea0 con 0x7f5abc15b440 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac35e5640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5abc15b820 con 0x7f5abc0c57e0 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac35e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0abf30 con 0x7f5abc0cc0f0 2023-11-24T08:23:37.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac1381640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5abc15b440 0x7f5abc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40460/0 (socket says 172.21.15.107:40460) 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.308+0000 7f5ac1381640 1 -- 172.21.15.107:0/3220780406 learned_addr learned my addr 172.21.15.107:0/3220780406 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ac0b80640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5abc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400121520 0 0) 0x7f5abc15eea0 con 0x7f5abc15b440 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8003520 con 0x7f5abc15b440 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ac1b82640 1 --2- 172.21.15.107:0/3220780406 >> v2:172.21.15.120:3300/0 conn(0x7f5abc0c57e0 0x7f5abc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3448006266 0 0) 0x7f5aa8003520 con 0x7f5abc15b440 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 msgr2=0x7f5abc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5abc0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 >> v2:172.21.15.120:3300/0 conn(0x7f5abc0c57e0 msgr2=0x7f5abc06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 --2- 172.21.15.107:0/3220780406 >> v2:172.21.15.120:3300/0 conn(0x7f5abc0c57e0 0x7f5abc06c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc15f0f0 con 0x7f5abc15b440 2023-11-24T08:23:37.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5ab8001670 con 0x7f5abc15b440 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 >> v1:172.21.15.107:6789/0 conn(0x7f5abc15b440 legacy=0x7f5abc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8003970 0x7f5aa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5aa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa8004d00 0x7f5aa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc15b820 con 0x7f5aa8004d00 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0abf30 con 0x7f5aa8003970 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.309+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa800bab0 con 0x7f5abc0cc0f0 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa8004d00 0x7f5aa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1381640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8003970 0x7f5aa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac0b80640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5aa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 -- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 msgr2=0x7f5aa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc0cc0f0 0x7f5aa80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 -- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8003970 msgr2=0x7f5aa8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8003970 0x7f5aa8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.310+0000 7f5ac1b82640 1 -- 172.21.15.107:0/3220780406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8003520 con 0x7f5aa8004d00 2023-11-24T08:23:37.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.311+0000 7f5ac1b82640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa8004d00 0x7f5aa8005180 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4007360 tx=0x7f5aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c02eec39908b5a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.311+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab402ccf0 con 0x7f5aa8004d00 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.311+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab402ce50 con 0x7f5aa8004d00 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.311+0000 7f5ab3fff640 1 -- 172.21.15.107:0/3220780406 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4034660 con 0x7f5aa8004d00 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 -- 172.21.15.107:0/3220780406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa8004d00 msgr2=0x7f5aa8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/3220780406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa8004d00 0x7f5aa8005180 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4007360 tx=0x7f5aa80057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac1b82640 1 -- 172.21.15.107:0/3220780406 reap_dead start 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 -- 172.21.15.107:0/3220780406 shutdown_connections 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 -- 172.21.15.107:0/3220780406 >> 172.21.15.107:0/3220780406 conn(0x7f5abc0b0c50 msgr2=0x7f5abc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 -- 172.21.15.107:0/3220780406 shutdown_connections 2023-11-24T08:23:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.312+0000 7f5ac35e5640 1 -- 172.21.15.107:0/3220780406 wait complete. 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.314+0000 7f5ac35e5640 1 Processor -- start 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.314+0000 7f5ac35e5640 1 -- start start 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8004d00 0x7f5abc0aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc15b9a0 0x7f5abc15be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc15c360 con 0x7f5abc09cc10 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0ce250 con 0x7f5aa8004d00 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac35e5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc1f2b40 con 0x7f5abc15b9a0 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40252/0 (socket says 172.21.15.107:40252) 2023-11-24T08:23:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 -- 172.21.15.107:0/130996721 learned_addr learned my addr 172.21.15.107:0/130996721 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 -- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc15b9a0 msgr2=0x7f5abc15be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc15b9a0 0x7f5abc15be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 -- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8004d00 msgr2=0x7f5abc0aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac1381640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8004d00 0x7f5abc0aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8004d00 0x7f5abc0aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.315+0000 7f5ac0b80640 1 -- 172.21.15.107:0/130996721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc0cdcb0 con 0x7f5abc09cc10 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ac0b80640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f5aac010840 tx=0x7f5aac010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac009610 con 0x7f5abc09cc10 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac00eb60 con 0x7f5abc09cc10 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac01c400 con 0x7f5abc09cc10 2023-11-24T08:23:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc1f70d0 con 0x7f5abc09cc10 2023-11-24T08:23:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.316+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc155bc0 con 0x7f5abc09cc10 2023-11-24T08:23:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.317+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc04e8b0 con 0x7f5abc09cc10 2023-11-24T08:23:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.318+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f5aac01c560 con 0x7f5abc09cc10 2023-11-24T08:23:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.319+0000 7f5ab1ffb640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 0x7f5aa006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.319+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5aac088ea0 con 0x7f5abc09cc10 2023-11-24T08:23:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.319+0000 7f5ac1381640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 0x7f5aa006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.320+0000 7f5ac1381640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 0x7f5aa006f820 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80057f0 tx=0x7f5ab8000f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.324+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5aac055a60 con 0x7f5abc09cc10 2023-11-24T08:23:37.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.553+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5abc0ce420 con 0x7f5abc09cc10 2023-11-24T08:23:37.556 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:23:37.554+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:23:37.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.557+0000 7f5ab1ffb640 1 -- 172.21.15.107:0/130996721 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v140) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5aac059080 con 0x7f5abc09cc10 2023-11-24T08:23:37.560 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 msgr2=0x7f5aa006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 0x7f5aa006f820 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80057f0 tx=0x7f5ab8000f90 comp rx=0 tx=0).stop 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 msgr2=0x7f5abc1f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f5aac010840 tx=0x7f5aac010d00 comp rx=0 tx=0).stop 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 shutdown_connections 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5aa006d360 0x7f5aa006f820 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5abc15b9a0 0x7f5abc15be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5abc09cc10 0x7f5abc1f2600 secure :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f5aac010840 tx=0x7f5aac010d00 comp rx=0 tx=0).stop 2023-11-24T08:23:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 --2- 172.21.15.107:0/130996721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa8004d00 0x7f5abc0aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 >> 172.21.15.107:0/130996721 conn(0x7f5abc0b0c50 msgr2=0x7f5abc0c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 shutdown_connections 2023-11-24T08:23:37.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.561+0000 7f5ac35e5640 1 -- 172.21.15.107:0/130996721 wait complete. 2023-11-24T08:23:37.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:23:37.677 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:23:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.760+0000 7fb7394d4640 1 Processor -- start 2023-11-24T08:23:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.760+0000 7fb7394d4640 1 -- start start 2023-11-24T08:23:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb7394d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7341660b0 0x7fb734162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb7394d4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb734166510 con 0x7fb734162540 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb7394d4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7341655f0 con 0x7fb7340ccbd0 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb7394d4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7340cc380 con 0x7fb7341660b0 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7341660b0 0x7fb734162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7341660b0 0x7fb734162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40280/0 (socket says 172.21.15.107:40280) 2023-11-24T08:23:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 -- 172.21.15.107:0/1794403765 learned_addr learned my addr 172.21.15.107:0/1794403765 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 -- 172.21.15.107:0/1794403765 >> v1:172.21.15.120:6789/0 conn(0x7fb7340ccbd0 legacy=0x7fb7340ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 -- 172.21.15.107:0/1794403765 >> v1:172.21.15.107:6789/0 conn(0x7fb734162540 legacy=0x7fb7340c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb72bfff640 1 -- 172.21.15.107:0/1794403765 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734162b30 con 0x7fb7341660b0 2023-11-24T08:23:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.762+0000 7fb7327fc640 1 -- 172.21.15.107:0/1794403765 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332542794 0 0) 0x7fb72c001490 con 0x7fb734162540 2023-11-24T08:23:37.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.763+0000 7fb72bfff640 1 --2- 172.21.15.107:0/1794403765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7341660b0 0x7fb734162000 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009870 tx=0x7fb71c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd57dcf2b1aac68 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7327fc640 1 -- 172.21.15.107:0/1794403765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c030850 con 0x7fb7341660b0 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7327fc640 1 -- 172.21.15.107:0/1794403765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c0309b0 con 0x7fb7341660b0 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7327fc640 1 -- 172.21.15.107:0/1794403765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c030d40 con 0x7fb7341660b0 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 -- 172.21.15.107:0/1794403765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7341660b0 msgr2=0x7fb734162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1794403765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7341660b0 0x7fb734162000 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009870 tx=0x7fb71c02b990 comp rx=0 tx=0).stop 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 -- 172.21.15.107:0/1794403765 shutdown_connections 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1794403765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7341660b0 0x7fb734162000 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 -- 172.21.15.107:0/1794403765 >> 172.21.15.107:0/1794403765 conn(0x7fb7340b0c50 msgr2=0x7fb7340b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 -- 172.21.15.107:0/1794403765 shutdown_connections 2023-11-24T08:23:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.764+0000 7fb7394d4640 1 -- 172.21.15.107:0/1794403765 wait complete. 2023-11-24T08:23:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.766+0000 7fb7394d4640 1 Processor -- start 2023-11-24T08:23:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.766+0000 7fb7394d4640 1 -- start start 2023-11-24T08:23:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb734162540 0x7fb734071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7341660b0 0x7fb7340acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340c4d20 con 0x7fb7340ccbd0 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734072860 con 0x7fb7341660b0 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb7394d4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340c3f60 con 0x7fb734162540 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb732ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb732ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40294/0 (socket says 172.21.15.107:40294) 2023-11-24T08:23:37.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.767+0000 7fb732ffd640 1 -- 172.21.15.107:0/1576230931 learned_addr learned my addr 172.21.15.107:0/1576230931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 -- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb734162540 msgr2=0x7fb734071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb734162540 0x7fb734071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 -- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7341660b0 msgr2=0x7fb7340acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7341660b0 0x7fb7340acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 -- 172.21.15.107:0/1576230931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c0094d0 con 0x7fb7340ccbd0 2023-11-24T08:23:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb732ffd640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7fb72400b3f0 tx=0x7fb72400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011830 con 0x7fb7340ccbd0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724011990 con 0x7fb7340ccbd0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011d40 con 0x7fb7340ccbd0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7340c4240 con 0x7fb7340ccbd0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.768+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7340c4800 con 0x7fb7340ccbd0 2023-11-24T08:23:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.769+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73404e8b0 con 0x7fb7340ccbd0 2023-11-24T08:23:37.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.770+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fb72401a400 con 0x7fb7340ccbd0 2023-11-24T08:23:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.772+0000 7fb72b7fe640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 0x7fb71406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.772+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb724088680 con 0x7fb7340ccbd0 2023-11-24T08:23:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.772+0000 7fb72bfff640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 0x7fb71406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.773+0000 7fb72bfff640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 0x7fb71406f820 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb71c036000 tx=0x7fb71c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.776+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb724055230 con 0x7fb7340ccbd0 2023-11-24T08:23:37.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:37.988+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb7340ae210 con 0x7fb7340ccbd0 2023-11-24T08:23:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.568+0000 7fb72b7fe640 1 -- 172.21.15.107:0/1576230931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v141) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb724058850 con 0x7fb7340ccbd0 2023-11-24T08:23:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 msgr2=0x7fb71406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 0x7fb71406f820 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb71c036000 tx=0x7fb71c03b040 comp rx=0 tx=0).stop 2023-11-24T08:23:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 msgr2=0x7fb734157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7fb72400b3f0 tx=0x7fb72400b8b0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 shutdown_connections 2023-11-24T08:23:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb71406d360 0x7fb71406f820 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.571+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7341660b0 0x7fb7340acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.572+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb734162540 0x7fb734071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.572+0000 7fb7394d4640 1 --2- 172.21.15.107:0/1576230931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340ccbd0 0x7fb734157e40 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.572+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 >> 172.21.15.107:0/1576230931 conn(0x7fb7340b0c50 msgr2=0x7fb7340af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.572+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 shutdown_connections 2023-11-24T08:23:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.572+0000 7fb7394d4640 1 -- 172.21.15.107:0/1576230931 wait complete. 2023-11-24T08:23:38.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:23:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.808+0000 7f21df7ec640 1 Processor -- start 2023-11-24T08:23:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.808+0000 7f21df7ec640 1 -- start start 2023-11-24T08:23:38.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21df7ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21d80ce6c0 0x7f21d80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21df7ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80cece0 con 0x7f21d815c7f0 2023-11-24T08:23:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21df7ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d806d040 con 0x7f21d80cf450 2023-11-24T08:23:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21df7ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80cfe40 con 0x7f21d80ce6c0 2023-11-24T08:23:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21dd588640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f21d815c7f0 0x7f21d806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53308/0 (socket says 172.21.15.107:53308) 2023-11-24T08:23:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21dd588640 1 -- 172.21.15.107:0/2334442286 learned_addr learned my addr 172.21.15.107:0/2334442286 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:38.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21ddd89640 1 --2- 172.21.15.107:0/2334442286 >> v2:172.21.15.107:3300/0 conn(0x7f21d80ce6c0 0x7f21d80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:38.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21cffff640 1 -- 172.21.15.107:0/2334442286 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370733601 0 0) 0x7f21d80cece0 con 0x7f21d815c7f0 2023-11-24T08:23:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21cffff640 1 -- 172.21.15.107:0/2334442286 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c4003520 con 0x7f21d815c7f0 2023-11-24T08:23:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21ddd89640 1 -- 172.21.15.107:0/2334442286 >> v1:172.21.15.120:6789/0 conn(0x7f21d80cf450 legacy=0x7f21d80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21ddd89640 1 -- 172.21.15.107:0/2334442286 >> v1:172.21.15.107:6789/0 conn(0x7f21d815c7f0 legacy=0x7f21d806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.810+0000 7f21ddd89640 1 -- 172.21.15.107:0/2334442286 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d806d320 con 0x7f21d80ce6c0 2023-11-24T08:23:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21ddd89640 1 --2- 172.21.15.107:0/2334442286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80ce6c0 0x7f21d80cf900 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d90 tx=0x7f21d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0c7731889a7dd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21cffff640 1 -- 172.21.15.107:0/2334442286 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d002cc60 con 0x7f21d80ce6c0 2023-11-24T08:23:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21cffff640 1 -- 172.21.15.107:0/2334442286 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d002cdc0 con 0x7f21d80ce6c0 2023-11-24T08:23:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21cffff640 1 -- 172.21.15.107:0/2334442286 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0034660 con 0x7f21d80ce6c0 2023-11-24T08:23:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 -- 172.21.15.107:0/2334442286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80ce6c0 msgr2=0x7f21d80cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2334442286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80ce6c0 0x7f21d80cf900 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d90 tx=0x7f21d002beb0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 -- 172.21.15.107:0/2334442286 shutdown_connections 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2334442286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80ce6c0 0x7f21d80cf900 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 -- 172.21.15.107:0/2334442286 >> 172.21.15.107:0/2334442286 conn(0x7f21d806a920 msgr2=0x7f21d806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 -- 172.21.15.107:0/2334442286 shutdown_connections 2023-11-24T08:23:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.812+0000 7f21df7ec640 1 -- 172.21.15.107:0/2334442286 wait complete. 2023-11-24T08:23:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.814+0000 7f21df7ec640 1 Processor -- start 2023-11-24T08:23:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.814+0000 7f21df7ec640 1 -- start start 2023-11-24T08:23:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80ce6c0 0x7f21d8151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d815c7f0 0x7f21d8156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8159d50 con 0x7f21d80cf450 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8151a80 con 0x7f21d80ce6c0 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.815+0000 7f21df7ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d815b5b0 con 0x7f21d815c7f0 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59680/0 (socket says 172.21.15.107:59680) 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 -- 172.21.15.107:0/2038587974 learned_addr learned my addr 172.21.15.107:0/2038587974 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dd588640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80ce6c0 0x7f21d8151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 -- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d815c7f0 msgr2=0x7f21d8156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d815c7f0 0x7f21d8156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 -- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80ce6c0 msgr2=0x7f21d8151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80ce6c0 0x7f21d8151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 -- 172.21.15.107:0/2038587974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009a70 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21dcd87640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f21c800c430 tx=0x7f21c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.816+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8011790 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.817+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c80118f0 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.817+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d815b890 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.817+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8011ce0 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.817+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80ac340 con 0x7f21d80cf450 2023-11-24T08:23:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.818+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d8067470 con 0x7f21d80cf450 2023-11-24T08:23:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.819+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f21c801c460 con 0x7f21d80cf450 2023-11-24T08:23:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.820+0000 7f21cdffb640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 0x7f21bc06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.820+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f21c8088870 con 0x7f21d80cf450 2023-11-24T08:23:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.820+0000 7f21dd588640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 0x7f21bc06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.821+0000 7f21dd588640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 0x7f21bc06f850 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f21d40011e0 tx=0x7f21d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:38.824+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21c8055380 con 0x7f21d80cf450 2023-11-24T08:23:39.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.018+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f21d80cd730 con 0x7f21d80cf450 2023-11-24T08:23:39.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.571+0000 7f21cdffb640 1 -- 172.21.15.107:0/2038587974 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v224) v1 ==== 181+0+0 (secure 0 0 0) 0x7f21c80589a0 con 0x7f21d80cf450 2023-11-24T08:23:39.573 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:23:39.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 msgr2=0x7f21bc06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 0x7f21bc06f850 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f21d40011e0 tx=0x7f21d4009040 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 msgr2=0x7f21d8155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f21c800c430 tx=0x7f21c800c8f0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 shutdown_connections 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f21bc06d390 0x7f21bc06f850 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d815c7f0 0x7f21d8156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d80cf450 0x7f21d8155de0 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 --2- 172.21.15.107:0/2038587974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80ce6c0 0x7f21d8151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 >> 172.21.15.107:0/2038587974 conn(0x7f21d806a920 msgr2=0x7f21d80b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:39.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.575+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 shutdown_connections 2023-11-24T08:23:39.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.576+0000 7f21df7ec640 1 -- 172.21.15.107:0/2038587974 wait complete. 2023-11-24T08:23:39.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:23:39.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.810+0000 7fba156dc640 1 Processor -- start 2023-11-24T08:23:39.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.811+0000 7fba156dc640 1 -- start start 2023-11-24T08:23:39.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.812+0000 7fba156dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 0x7fba100b3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.812+0000 7fba156dc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fba1015d510 0x7fba100aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.812+0000 7fba156dc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba10161e30 con 0x7fba100d05f0 2023-11-24T08:23:39.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.812+0000 7fba156dc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba100af360 con 0x7fba1015d510 2023-11-24T08:23:39.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.812+0000 7fba156dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba100722d0 con 0x7fba100d0210 2023-11-24T08:23:39.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba14edb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba100d05f0 0x7fba10161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53314/0 (socket says 172.21.15.107:53314) 2023-11-24T08:23:39.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba14edb640 1 -- 172.21.15.107:0/2939093673 learned_addr learned my addr 172.21.15.107:0/2939093673 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba07fff640 1 --2- 172.21.15.107:0/2939093673 >> v2:172.21.15.120:3300/0 conn(0x7fba1015d510 0x7fba100aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba0ffff640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 0x7fba100b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296212134 0 0) 0x7fba08001490 con 0x7fba100d05f0 2023-11-24T08:23:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.813+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc003520 con 0x7fba100d05f0 2023-11-24T08:23:39.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252062121 0 0) 0x7fb9fc003520 con 0x7fba100d05f0 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 msgr2=0x7fba100b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 0x7fba100b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 >> v2:172.21.15.120:3300/0 conn(0x7fba1015d510 msgr2=0x7fba100aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 --2- 172.21.15.107:0/2939093673 >> v2:172.21.15.120:3300/0 conn(0x7fba1015d510 0x7fba100aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100725b0 con 0x7fba100d05f0 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fba08001b80 con 0x7fba100d05f0 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 >> v1:172.21.15.107:6789/0 conn(0x7fba100d05f0 legacy=0x7fba10161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 0x7fb9fc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fb9fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9fc004d00 0x7fb9fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100af360 con 0x7fb9fc003f40 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100722d0 con 0x7fb9fc004d00 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc00b980 con 0x7fba100d0210 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba07fff640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fb9fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba07fff640 1 -- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 msgr2=0x7fb9fc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba07fff640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100d0210 0x7fb9fc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba07fff640 1 -- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9fc004d00 msgr2=0x7fb9fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba14edb640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9fc004d00 0x7fb9fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.814+0000 7fba07fff640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9fc004d00 0x7fb9fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.815+0000 7fba07fff640 1 -- 172.21.15.107:0/2939093673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc003520 con 0x7fb9fc003f40 2023-11-24T08:23:39.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.815+0000 7fba14edb640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9fc004d00 0x7fb9fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba07fff640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fb9fc0047c0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009ac0 tx=0x7fb9fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b022aa28a7aa05 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802c6b0 con 0x7fb9fc003f40 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f802c810 con 0x7fb9fc003f40 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba0f7fe640 1 -- 172.21.15.107:0/2939093673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802cba0 con 0x7fb9fc003f40 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba156dc640 1 -- 172.21.15.107:0/2939093673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 msgr2=0x7fb9fc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba156dc640 1 --2- 172.21.15.107:0/2939093673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fb9fc0047c0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009ac0 tx=0x7fb9fc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.816+0000 7fba14edb640 1 -- 172.21.15.107:0/2939093673 reap_dead start 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.817+0000 7fba156dc640 1 -- 172.21.15.107:0/2939093673 shutdown_connections 2023-11-24T08:23:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.817+0000 7fba156dc640 1 -- 172.21.15.107:0/2939093673 >> 172.21.15.107:0/2939093673 conn(0x7fba1006d860 msgr2=0x7fba1006dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:39.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.817+0000 7fba156dc640 1 -- 172.21.15.107:0/2939093673 shutdown_connections 2023-11-24T08:23:39.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.817+0000 7fba156dc640 1 -- 172.21.15.107:0/2939093673 wait complete. 2023-11-24T08:23:39.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.818+0000 7fba156dc640 1 Processor -- start 2023-11-24T08:23:39.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 -- start start 2023-11-24T08:23:39.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100cf110 0x7fba100cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba100cf570 0x7fba100ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10141f00 con 0x7fb9fc003f40 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba101e6df0 con 0x7fba100cf570 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.819+0000 7fba156dc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba101e70d0 con 0x7fba100cf110 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59692/0 (socket says 172.21.15.107:59692) 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 -- 172.21.15.107:0/2560864399 learned_addr learned my addr 172.21.15.107:0/2560864399 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 -- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100cf110 msgr2=0x7fba100cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100cf110 0x7fba100cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 -- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba100cf570 msgr2=0x7fba100ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:39.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba100cf570 0x7fba100ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 -- 172.21.15.107:0/2560864399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba101e73b0 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.820+0000 7fba0ffff640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e7c0 tx=0x7fba0000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.821+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0000eed0 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.821+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0000c940 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.821+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba101e7690 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.821+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0000ccd0 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.821+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1014eb10 con 0x7fb9fc003f40 2023-11-24T08:23:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.822+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1004e840 con 0x7fb9fc003f40 2023-11-24T08:23:39.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.823+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fba00013470 con 0x7fb9fc003f40 2023-11-24T08:23:39.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.824+0000 7fba0d7fa640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 0x7fb9f006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:39.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.825+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fba000888e0 con 0x7fb9fc003f40 2023-11-24T08:23:39.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.825+0000 7fba07fff640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 0x7fb9f006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:39.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.825+0000 7fba07fff640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 0x7fb9f006f820 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8036000 tx=0x7fb9f802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:39.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:39.829+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba000553f0 con 0x7fb9fc003f40 2023-11-24T08:23:40.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.025+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba100ae9f0 con 0x7fb9fc003f40 2023-11-24T08:23:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.574+0000 7fba0d7fa640 1 -- 172.21.15.107:0/2560864399 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v225) v1 ==== 169+0+0 (secure 0 0 0) 0x7fba000170d0 con 0x7fb9fc003f40 2023-11-24T08:23:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:23:40.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 msgr2=0x7fb9f006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 0x7fb9f006f820 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8036000 tx=0x7fb9f802e000 comp rx=0 tx=0).stop 2023-11-24T08:23:40.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 msgr2=0x7fba101419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e7c0 tx=0x7fba0000ec80 comp rx=0 tx=0).stop 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 shutdown_connections 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb9f006d360 0x7fb9f006f820 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba100cf570 0x7fba100ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba100cf110 0x7fba100cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 --2- 172.21.15.107:0/2560864399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9fc003f40 0x7fba101419c0 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.578+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 >> 172.21.15.107:0/2560864399 conn(0x7fba1006d860 msgr2=0x7fba10159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.579+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 shutdown_connections 2023-11-24T08:23:40.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.579+0000 7fba156dc640 1 -- 172.21.15.107:0/2560864399 wait complete. 2023-11-24T08:23:40.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:23:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.822+0000 7fb6ef14e640 1 Processor -- start 2023-11-24T08:23:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.822+0000 7fb6ef14e640 1 -- start start 2023-11-24T08:23:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.823+0000 7fb6ef14e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6e80d1a00 0x7fb6e80b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.823+0000 7fb6ef14e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80b41b0 con 0x7fb6e81609f0 2023-11-24T08:23:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.823+0000 7fb6ef14e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80b4490 con 0x7fb6e8160dd0 2023-11-24T08:23:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.823+0000 7fb6ef14e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e80b0090 con 0x7fb6e80d1a00 2023-11-24T08:23:40.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ed94b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6e80d1a00 0x7fb6e80b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:40.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ee14c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6e81609f0 0x7fb6e80d12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53320/0 (socket says 172.21.15.107:53320) 2023-11-24T08:23:40.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ee14c640 1 -- 172.21.15.107:0/3590669026 learned_addr learned my addr 172.21.15.107:0/3590669026 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ed94b640 1 -- 172.21.15.107:0/3590669026 >> v1:172.21.15.120:6789/0 conn(0x7fb6e8160dd0 legacy=0x7fb6e80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ed94b640 1 -- 172.21.15.107:0/3590669026 >> v1:172.21.15.107:6789/0 conn(0x7fb6e81609f0 legacy=0x7fb6e80d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ed94b640 1 -- 172.21.15.107:0/3590669026 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e80b0370 con 0x7fb6e80d1a00 2023-11-24T08:23:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.824+0000 7fb6ed14a640 1 -- 172.21.15.107:0/3590669026 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138568652 0 0) 0x7fb6e80b41b0 con 0x7fb6e81609f0 2023-11-24T08:23:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.825+0000 7fb6ed94b640 1 --2- 172.21.15.107:0/3590669026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e80d1a00 0x7fb6e80b2be0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097a0 tx=0x7fb6d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6d7d3848bceb54 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.825+0000 7fb6ed14a640 1 -- 172.21.15.107:0/3590669026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8031830 con 0x7fb6e80d1a00 2023-11-24T08:23:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.825+0000 7fb6ed14a640 1 -- 172.21.15.107:0/3590669026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8031990 con 0x7fb6e80d1a00 2023-11-24T08:23:40.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.825+0000 7fb6ed14a640 1 -- 172.21.15.107:0/3590669026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8031d20 con 0x7fb6e80d1a00 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 -- 172.21.15.107:0/3590669026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e80d1a00 msgr2=0x7fb6e80b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/3590669026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e80d1a00 0x7fb6e80b2be0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097a0 tx=0x7fb6d802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 -- 172.21.15.107:0/3590669026 shutdown_connections 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/3590669026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e80d1a00 0x7fb6e80b2be0 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 -- 172.21.15.107:0/3590669026 >> 172.21.15.107:0/3590669026 conn(0x7fb6e809c5b0 msgr2=0x7fb6e80d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 -- 172.21.15.107:0/3590669026 shutdown_connections 2023-11-24T08:23:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.826+0000 7fb6ef14e640 1 -- 172.21.15.107:0/3590669026 wait complete. 2023-11-24T08:23:40.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.828+0000 7fb6ef14e640 1 Processor -- start 2023-11-24T08:23:40.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.828+0000 7fb6ef14e640 1 -- start start 2023-11-24T08:23:40.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6e80d1a00 0x7fb6e80ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6e8160dd0 0x7fb6e80a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e809f860 con 0x7fb6e81609f0 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e81f3e20 con 0x7fb6e8160dd0 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ef14e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e81f3b40 con 0x7fb6e80d1a00 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ed94b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ed94b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59714/0 (socket says 172.21.15.107:59714) 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.829+0000 7fb6ed94b640 1 -- 172.21.15.107:0/42176960 learned_addr learned my addr 172.21.15.107:0/42176960 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ee14c640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6e80d1a00 0x7fb6e80ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 -- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6e80d1a00 msgr2=0x7fb6e80ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6e80d1a00 0x7fb6e80ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 -- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6e8160dd0 msgr2=0x7fb6e80a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6e8160dd0 0x7fb6e80a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:40.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 -- 172.21.15.107:0/42176960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8009400 con 0x7fb6e81609f0 2023-11-24T08:23:40.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ed94b640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098c0 tx=0x7fb6d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:40.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d803ba70 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d803bbd0 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80453f0 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.830+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e809fa90 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.831+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e80a0050 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.832+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e80a0240 con 0x7fb6e81609f0 2023-11-24T08:23:40.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.833+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fb6d804e450 con 0x7fb6e81609f0 2023-11-24T08:23:40.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.833+0000 7fb6deffd640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 0x7fb6cc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:40.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.834+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb6d80aab80 con 0x7fb6e81609f0 2023-11-24T08:23:40.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.834+0000 7fb6ee14c640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 0x7fb6cc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:40.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.834+0000 7fb6ee14c640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 0x7fb6cc06f820 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4000f60 tx=0x7fb6e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:40.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:40.838+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6d8077690 con 0x7fb6e81609f0 2023-11-24T08:23:41.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.029+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb6e804e840 con 0x7fb6e81609f0 2023-11-24T08:23:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.034+0000 7fb6deffd640 1 -- 172.21.15.107:0/42176960 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb6d807acb0 con 0x7fb6e81609f0 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.036+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 msgr2=0x7fb6cc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.036+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 0x7fb6cc06f820 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4000f60 tx=0x7fb6e4009040 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 msgr2=0x7fb6e80cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098c0 tx=0x7fb6d80046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 shutdown_connections 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb6cc06d360 0x7fb6cc06f820 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6e8160dd0 0x7fb6e80a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e81609f0 0x7fb6e80cbbe0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 --2- 172.21.15.107:0/42176960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6e80d1a00 0x7fb6e80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 >> 172.21.15.107:0/42176960 conn(0x7fb6e809c5b0 msgr2=0x7fb6e80cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:41.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 shutdown_connections 2023-11-24T08:23:41.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.037+0000 7fb6ef14e640 1 -- 172.21.15.107:0/42176960 wait complete. 2023-11-24T08:23:41.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone' 2023-11-24T08:23:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.263+0000 7f5f06ce7640 1 Processor -- start 2023-11-24T08:23:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.263+0000 7f5f06ce7640 1 -- start start 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f06ce7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f000ce6c0 0x7f5f000cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f06ce7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000cece0 con 0x7f5f0015c7f0 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f06ce7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0006d040 con 0x7f5f000cf450 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f06ce7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000cfe40 con 0x7f5f000ce6c0 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f04a83640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5f0015c7f0 0x7f5f0006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53324/0 (socket says 172.21.15.107:53324) 2023-11-24T08:23:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.265+0000 7f5f04a83640 1 -- 172.21.15.107:0/3827784891 learned_addr learned my addr 172.21.15.107:0/3827784891 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5f05284640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f000ce6c0 0x7f5f000cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729873392 0 0) 0x7f5f000cece0 con 0x7f5f0015c7f0 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef0003520 con 0x7f5f0015c7f0 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4250577418 0 0) 0x7f5ef0003520 con 0x7f5f0015c7f0 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f000ce6c0 msgr2=0x7f5f000cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f000ce6c0 0x7f5f000cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 >> v1:172.21.15.120:6789/0 conn(0x7f5f000cf450 legacy=0x7f5f000cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0006d320 con 0x7f5f0015c7f0 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f5ef4001670 con 0x7f5f0015c7f0 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 >> v1:172.21.15.107:6789/0 conn(0x7f5f0015c7f0 legacy=0x7f5f0006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5ef0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef00047c0 0x7f5ef0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef0005180 0x7f5ef000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000cfe40 con 0x7f5ef00039a0 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef000c050 con 0x7f5ef0005180 2023-11-24T08:23:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef000c210 con 0x7f5ef00047c0 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5f05284640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f000ce6c0 0x7f5f000cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.266+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5ef0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f04a83640 1 -- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef00047c0 msgr2=0x7f5ef0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5efffff640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef00047c0 0x7f5ef0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f05284640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef0005180 0x7f5ef000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ef00047c0 0x7f5ef0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f04a83640 1 -- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef0005180 msgr2=0x7f5ef000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ef0005180 0x7f5ef000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.267+0000 7f5f04a83640 1 -- 172.21.15.107:0/3827784891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef0003520 con 0x7f5ef00039a0 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.268+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5ef0003dc0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400c240 tx=0x7f5ef00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7feda1ea148d98c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.268+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4005ad0 con 0x7f5ef00039a0 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.268+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef4005c30 con 0x7f5ef00039a0 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.268+0000 7f5eff7fe640 1 -- 172.21.15.107:0/3827784891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef402c8b0 con 0x7f5ef00039a0 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3827784891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 msgr2=0x7f5ef0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3827784891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5ef0003dc0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400c240 tx=0x7f5ef00057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f05284640 1 -- 172.21.15.107:0/3827784891 reap_dead start 2023-11-24T08:23:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3827784891 shutdown_connections 2023-11-24T08:23:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3827784891 >> 172.21.15.107:0/3827784891 conn(0x7f5f0006a920 msgr2=0x7f5f0006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3827784891 shutdown_connections 2023-11-24T08:23:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.269+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3827784891 wait complete. 2023-11-24T08:23:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.271+0000 7f5f06ce7640 1 Processor -- start 2023-11-24T08:23:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.271+0000 7f5f06ce7640 1 -- start start 2023-11-24T08:23:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f00151180 0x7f5f000b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f001508e0 0x7f5f000b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000afef0 con 0x7f5ef00039a0 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001519d0 con 0x7f5f001508e0 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f06ce7640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000af2e0 con 0x7f5f00151180 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f04a83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f04a83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59730/0 (socket says 172.21.15.107:59730) 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.272+0000 7f5f04a83640 1 -- 172.21.15.107:0/3940447505 learned_addr learned my addr 172.21.15.107:0/3940447505 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5efffff640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f00151180 0x7f5f000b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f05284640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f001508e0 0x7f5f000b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 -- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f00151180 msgr2=0x7f5f000b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f00151180 0x7f5f000b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 -- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f001508e0 msgr2=0x7f5f000b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f001508e0 0x7f5f000b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 -- 172.21.15.107:0/3940447505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000af5c0 con 0x7f5ef00039a0 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f04a83640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f5ef402c450 tx=0x7f5ef4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4039070 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef402cce0 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef403c040 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f000af8a0 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.273+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f000b08c0 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.274+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00067470 con 0x7f5ef00039a0 2023-11-24T08:23:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.276+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f5ef4032070 con 0x7f5ef00039a0 2023-11-24T08:23:41.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.276+0000 7f5efd7fa640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 0x7f5ee406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.277+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ef40a9de0 con 0x7f5ef00039a0 2023-11-24T08:23:41.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.277+0000 7f5efffff640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 0x7f5ee406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.277+0000 7f5efffff640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 0x7f5ee406f820 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f5f00158e70 tx=0x7f5eec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.281+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ef40b5080 con 0x7f5ef00039a0 2023-11-24T08:23:41.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.540+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone"]} v 0) v1 -- 0x7f5f000cd730 con 0x7f5ef00039a0 2023-11-24T08:23:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.592+0000 7f5efd7fa640 1 -- 172.21.15.107:0/3940447505 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone"]}]=0 v953) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5ef4076860 con 0x7f5ef00039a0 2023-11-24T08:23:41.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.595+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 msgr2=0x7f5ee406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 0x7f5ee406f820 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f5f00158e70 tx=0x7f5eec009040 comp rx=0 tx=0).stop 2023-11-24T08:23:41.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 msgr2=0x7f5f000aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f5ef402c450 tx=0x7f5ef4001a00 comp rx=0 tx=0).stop 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 shutdown_connections 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5ee406d360 0x7f5ee406f820 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f001508e0 0x7f5f000b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f00151180 0x7f5f000b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 --2- 172.21.15.107:0/3940447505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ef00039a0 0x7f5f000aec50 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 >> 172.21.15.107:0/3940447505 conn(0x7f5f0006a920 msgr2=0x7f5f0006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 shutdown_connections 2023-11-24T08:23:41.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.596+0000 7f5f06ce7640 1 -- 172.21.15.107:0/3940447505 wait complete. 2023-11-24T08:23:41.616 INFO:tasks.cephfs_test_runner:clone a snapshot from a snapshot retained subvolume ... ok 2023-11-24T08:23:41.616 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_retain_snapshot_clone_from_newer_snapshot (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) clone a subvolume from recreated subvolume's latest snapshot 2023-11-24T08:23:41.616 INFO:tasks.cephfs_test_runner:test_subvolume_retain_snapshot_clone_from_newer_snapshot (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:23:41.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.822+0000 7fc0aad33640 1 Processor -- start 2023-11-24T08:23:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.823+0000 7fc0aad33640 1 -- start start 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 0x7fc0a406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 0x7fc0a40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0a40cdfb0 0x7fc0a40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a4156010 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a406ce30 con 0x7fc0a415c7f0 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0aad33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40cfe10 con 0x7fc0a40cf450 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a92d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0a40cdfb0 0x7fc0a40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a92d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0a40cdfb0 0x7fc0a40ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59734/0 (socket says 172.21.15.107:59734) 2023-11-24T08:23:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a92d0640 1 -- 172.21.15.107:0/2539285810 learned_addr learned my addr 172.21.15.107:0/2539285810 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a92d0640 1 -- 172.21.15.107:0/2539285810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 msgr2=0x7fc0a40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a3fff640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 0x7fc0a40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.824+0000 7fc0a8acf640 1 --2- 172.21.15.107:0/2539285810 >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 0x7fc0a406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 0x7fc0a40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a92d0640 1 -- 172.21.15.107:0/2539285810 >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 msgr2=0x7fc0a406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/2539285810 >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 0x7fc0a406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a92d0640 1 -- 172.21.15.107:0/2539285810 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40cece0 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a3fff640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 0x7fc0a40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.825+0000 7fc0a8acf640 1 --2- 172.21.15.107:0/2539285810 >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 0x7fc0a406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a40cdfb0 0x7fc0a40ce440 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40d0130 tx=0x7fc09c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ca8cdb7aa25c3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0a37fe640 1 -- 172.21.15.107:0/2539285810 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c031880 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0a37fe640 1 -- 172.21.15.107:0/2539285810 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c0319e0 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0a37fe640 1 -- 172.21.15.107:0/2539285810 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c031d70 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0aad33640 1 -- 172.21.15.107:0/2539285810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a40cdfb0 msgr2=0x7fc0a40ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.826+0000 7fc0aad33640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a40cdfb0 0x7fc0a40ce440 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40d0130 tx=0x7fc09c02b920 comp rx=0 tx=0).stop 2023-11-24T08:23:41.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 -- 172.21.15.107:0/2539285810 shutdown_connections 2023-11-24T08:23:41.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a40cdfb0 0x7fc0a40ce440 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 --2- 172.21.15.107:0/2539285810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cf450 0x7fc0a40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 --2- 172.21.15.107:0/2539285810 >> v2:172.21.15.120:3300/0 conn(0x7fc0a415c7f0 0x7fc0a406db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 -- 172.21.15.107:0/2539285810 >> 172.21.15.107:0/2539285810 conn(0x7fc0a406a920 msgr2=0x7fc0a406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 -- 172.21.15.107:0/2539285810 shutdown_connections 2023-11-24T08:23:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.827+0000 7fc0aad33640 1 -- 172.21.15.107:0/2539285810 wait complete. 2023-11-24T08:23:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.828+0000 7fc0aad33640 1 Processor -- start 2023-11-24T08:23:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.829+0000 7fc0aad33640 1 -- start start 2023-11-24T08:23:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.829+0000 7fc0aad33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cdfb0 0x7fc0a41ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.829+0000 7fc0aad33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a40cf450 0x7fc0a409c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.829+0000 7fc0aad33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0aad33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a414f830 con 0x7fc0a415c7f0 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0aad33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a41eec10 con 0x7fc0a40cf450 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0aad33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a415dcd0 con 0x7fc0a40cdfb0 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59748/0 (socket says 172.21.15.107:59748) 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 -- 172.21.15.107:0/68400408 learned_addr learned my addr 172.21.15.107:0/68400408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 -- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cdfb0 msgr2=0x7fc0a41ee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cdfb0 0x7fc0a41ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 -- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a40cf450 msgr2=0x7fc0a409c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a40cf450 0x7fc0a409c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 -- 172.21.15.107:0/68400408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0094d0 con 0x7fc0a415c7f0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.830+0000 7fc0a92d0640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0090b0 tx=0x7fc09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.831+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c03ae10 con 0x7fc0a415c7f0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.831+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c0443f0 con 0x7fc0a415c7f0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.831+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0447a0 con 0x7fc0a415c7f0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.831+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a409cc50 con 0x7fc0a415c7f0 2023-11-24T08:23:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.831+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a414fa80 con 0x7fc0a415c7f0 2023-11-24T08:23:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.832+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a4067470 con 0x7fc0a415c7f0 2023-11-24T08:23:41.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.833+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fc09c04d450 con 0x7fc0a415c7f0 2023-11-24T08:23:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.834+0000 7fc0a17fa640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 0x7fc08806f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.834+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc09c0aafe0 con 0x7fc0a415c7f0 2023-11-24T08:23:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.834+0000 7fc0a8acf640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 0x7fc08806f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.835+0000 7fc0a8acf640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 0x7fc08806f850 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc098008e00 tx=0x7fc098008330 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:41.838+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc09c0e2e90 con 0x7fc0a415c7f0 2023-11-24T08:23:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.030+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0a40cd730 con 0x7fc0a415c7f0 2023-11-24T08:23:42.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.032+0000 7fc0a17fa640 1 -- 172.21.15.107:0/68400408 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+12998 (secure 0 0 0) 0x7fc09c07b620 con 0x7fc0a415c7f0 2023-11-24T08:23:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:42.034 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":225,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:40.572332+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/1864313322":"2023-11-25T08:23:37.555578+0000","172.21.15.107:6836/1864313322":"2023-11-25T08:23:37.555578+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.034+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 msgr2=0x7fc08806f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.034+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 0x7fc08806f850 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc098008e00 tx=0x7fc098008330 comp rx=0 tx=0).stop 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.034+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 msgr2=0x7fc0a409c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.034+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0090b0 tx=0x7fc09c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 shutdown_connections 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc08806d390 0x7fc08806f850 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0a415c7f0 0x7fc0a409c5f0 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a40cf450 0x7fc0a409c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 --2- 172.21.15.107:0/68400408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0a40cdfb0 0x7fc0a41ee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 >> 172.21.15.107:0/68400408 conn(0x7fc0a406a920 msgr2=0x7fc0a406db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 shutdown_connections 2023-11-24T08:23:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.035+0000 7fc0aad33640 1 -- 172.21.15.107:0/68400408 wait complete. 2023-11-24T08:23:42.055 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:23:42.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:23:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.269+0000 7f5cea82d640 1 Processor -- start 2023-11-24T08:23:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.269+0000 7f5cea82d640 1 -- start start 2023-11-24T08:23:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 0x7f5ce40cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce406b820 0x7f5ce40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ce406ff70 0x7f5ce4070400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40cc740 con 0x7f5ce406ff70 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce406c8f0 con 0x7f5ce40c4680 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.270+0000 7f5cea82d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40cb850 con 0x7f5ce406b820 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ce406ff70 0x7f5ce4070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ce406ff70 0x7f5ce4070400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59764/0 (socket says 172.21.15.107:59764) 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 -- 172.21.15.107:0/3732002834 learned_addr learned my addr 172.21.15.107:0/3732002834 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce37fe640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce406b820 0x7f5ce40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce3fff640 1 --2- 172.21.15.107:0/3732002834 >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 0x7f5ce40cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 -- 172.21.15.107:0/3732002834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce406b820 msgr2=0x7f5ce40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce406b820 0x7f5ce40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 -- 172.21.15.107:0/3732002834 >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 msgr2=0x7f5ce40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 --2- 172.21.15.107:0/3732002834 >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 0x7f5ce40cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce8dca640 1 -- 172.21.15.107:0/3732002834 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40cbb30 con 0x7f5ce406ff70 2023-11-24T08:23:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.271+0000 7f5ce3fff640 1 --2- 172.21.15.107:0/3732002834 >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 0x7f5ce40cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.273+0000 7f5ce8dca640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406ff70 0x7f5ce4070400 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40ce980 tx=0x7f5cdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611ee30ce11eb2fa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.273+0000 7f5ce2ffd640 1 -- 172.21.15.107:0/3732002834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc031870 con 0x7f5ce406ff70 2023-11-24T08:23:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.273+0000 7f5ce2ffd640 1 -- 172.21.15.107:0/3732002834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc0319d0 con 0x7f5ce406ff70 2023-11-24T08:23:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.273+0000 7f5ce2ffd640 1 -- 172.21.15.107:0/3732002834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc031d60 con 0x7f5ce406ff70 2023-11-24T08:23:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 -- 172.21.15.107:0/3732002834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406ff70 msgr2=0x7f5ce4070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406ff70 0x7f5ce4070400 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40ce980 tx=0x7f5cdc02b910 comp rx=0 tx=0).stop 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 -- 172.21.15.107:0/3732002834 shutdown_connections 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406ff70 0x7f5ce4070400 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 --2- 172.21.15.107:0/3732002834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce406b820 0x7f5ce40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 --2- 172.21.15.107:0/3732002834 >> v2:172.21.15.120:3300/0 conn(0x7f5ce40c4680 0x7f5ce40cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 -- 172.21.15.107:0/3732002834 >> 172.21.15.107:0/3732002834 conn(0x7f5ce406e7d0 msgr2=0x7f5ce406ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 -- 172.21.15.107:0/3732002834 shutdown_connections 2023-11-24T08:23:42.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.274+0000 7f5cea82d640 1 -- 172.21.15.107:0/3732002834 wait complete. 2023-11-24T08:23:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.276+0000 7f5cea82d640 1 Processor -- start 2023-11-24T08:23:42.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.276+0000 7f5cea82d640 1 -- start start 2023-11-24T08:23:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ce406ff70 0x7f5ce4159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce40c4680 0x7f5ce40ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40cb0f0 con 0x7f5ce406b820 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40c8400 con 0x7f5ce406ff70 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5cea82d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40c86e0 con 0x7f5ce40c4680 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59780/0 (socket says 172.21.15.107:59780) 2023-11-24T08:23:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.277+0000 7f5ce3fff640 1 -- 172.21.15.107:0/1134107267 learned_addr learned my addr 172.21.15.107:0/1134107267 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce37fe640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ce406ff70 0x7f5ce4159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 -- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce40c4680 msgr2=0x7f5ce40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce40c4680 0x7f5ce40ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 -- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ce406ff70 msgr2=0x7f5ce4159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ce406ff70 0x7f5ce4159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 -- 172.21.15.107:0/1134107267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0094d0 con 0x7f5ce406b820 2023-11-24T08:23:42.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce3fff640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400ce00 tx=0x7f5cd400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd400f890 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd400f9f0 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce40c89c0 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd400fdc0 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.278+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce4159c40 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.280+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce404e8b0 con 0x7f5ce406b820 2023-11-24T08:23:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.280+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f5cd400b040 con 0x7f5ce406b820 2023-11-24T08:23:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.281+0000 7f5ce0ff9640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 0x7f5cc406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.281+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5cd4089180 con 0x7f5ce406b820 2023-11-24T08:23:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.282+0000 7f5ce37fe640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 0x7f5cc406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.282+0000 7f5ce37fe640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 0x7f5cc406f820 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40c9e80 tx=0x7f5ccc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.286+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cd4055d30 con 0x7f5ce406b820 2023-11-24T08:23:42.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.475+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ce40ccd00 con 0x7f5ce406b820 2023-11-24T08:23:42.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.476+0000 7f5ce0ff9640 1 -- 172.21.15.107:0/1134107267 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5cd401d030 con 0x7f5ce406b820 2023-11-24T08:23:42.478 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 msgr2=0x7f5cc406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 0x7f5cc406f820 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40c9e80 tx=0x7f5ccc008040 comp rx=0 tx=0).stop 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 msgr2=0x7f5ce40b5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400ce00 tx=0x7f5cd400a650 comp rx=0 tx=0).stop 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 shutdown_connections 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5cc406d360 0x7f5cc406f820 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ce40c4680 0x7f5ce40ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ce406ff70 0x7f5ce4159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 --2- 172.21.15.107:0/1134107267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ce406b820 0x7f5ce40b5d40 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 >> 172.21.15.107:0/1134107267 conn(0x7f5ce406e7d0 msgr2=0x7f5ce40b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 shutdown_connections 2023-11-24T08:23:42.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.479+0000 7f5cea82d640 1 -- 172.21.15.107:0/1134107267 wait complete. 2023-11-24T08:23:42.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot' 2023-11-24T08:23:42.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.707+0000 7fc85ade3640 1 Processor -- start 2023-11-24T08:23:42.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.708+0000 7fc85ade3640 1 -- start start 2023-11-24T08:23:42.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.709+0000 7fc85ade3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 0x7fc8540ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.709+0000 7fc85ade3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.709+0000 7fc85ade3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8540ce950 con 0x7fc85415a4f0 2023-11-24T08:23:42.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.709+0000 7fc85ade3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8540b3a40 con 0x7fc8540b20c0 2023-11-24T08:23:42.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.709+0000 7fc85ade3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540b2e30 con 0x7fc8540cdfb0 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.710+0000 7fc859380640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.710+0000 7fc853fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 0x7fc8540ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.710+0000 7fc859380640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:59946/0 (socket says 172.21.15.107:59946) 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.710+0000 7fc859380640 1 -- 172.21.15.107:0/1838998534 learned_addr learned my addr 172.21.15.107:0/1838998534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.711+0000 7fc853fff640 1 -- 172.21.15.107:0/1838998534 >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 msgr2=0x7fc8540b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.711+0000 7fc853fff640 1 --2- 172.21.15.107:0/1838998534 >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.711+0000 7fc853fff640 1 -- 172.21.15.107:0/1838998534 >> v1:172.21.15.120:6789/0 conn(0x7fc85415a4f0 legacy=0x7fc8540b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.711+0000 7fc853fff640 1 -- 172.21.15.107:0/1838998534 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540b3110 con 0x7fc8540cdfb0 2023-11-24T08:23:42.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.711+0000 7fc859380640 1 --2- 172.21.15.107:0/1838998534 >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:23:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.712+0000 7fc853fff640 1 --2- 172.21.15.107:0/1838998534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 0x7fc8540ce410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8400097a0 tx=0x7fc84002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=885b9798c90f32fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc8537fe640 1 -- 172.21.15.107:0/1838998534 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84002d040 con 0x7fc8540cdfb0 2023-11-24T08:23:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc8537fe640 1 -- 172.21.15.107:0/1838998534 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc840031810 con 0x7fc8540cdfb0 2023-11-24T08:23:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc8537fe640 1 -- 172.21.15.107:0/1838998534 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc840031ba0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 -- 172.21.15.107:0/1838998534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 msgr2=0x7fc8540ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 --2- 172.21.15.107:0/1838998534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 0x7fc8540ce410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8400097a0 tx=0x7fc84002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 -- 172.21.15.107:0/1838998534 shutdown_connections 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 --2- 172.21.15.107:0/1838998534 >> v2:172.21.15.120:3300/0 conn(0x7fc8540b20c0 0x7fc8540b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 --2- 172.21.15.107:0/1838998534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8540cdfb0 0x7fc8540ce410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.713+0000 7fc85ade3640 1 -- 172.21.15.107:0/1838998534 >> 172.21.15.107:0/1838998534 conn(0x7fc8540b0c70 msgr2=0x7fc8540b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.714+0000 7fc85ade3640 1 -- 172.21.15.107:0/1838998534 shutdown_connections 2023-11-24T08:23:42.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.714+0000 7fc85ade3640 1 -- 172.21.15.107:0/1838998534 wait complete. 2023-11-24T08:23:42.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.715+0000 7fc85ade3640 1 Processor -- start 2023-11-24T08:23:42.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.715+0000 7fc85ade3640 1 -- start start 2023-11-24T08:23:42.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 0x7fc8540c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc85415a4f0 0x7fc8540cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540c96a0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541f3500 con 0x7fc8540b20c0 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.716+0000 7fc85ade3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540c8b20 con 0x7fc85415a4f0 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59794/0 (socket says 172.21.15.107:59794) 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 -- 172.21.15.107:0/618589384 learned_addr learned my addr 172.21.15.107:0/618589384 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc858b7f640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 0x7fc8540c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc859380640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc85415a4f0 0x7fc8540cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 -- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc85415a4f0 msgr2=0x7fc8540cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc85415a4f0 0x7fc8540cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 -- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 msgr2=0x7fc8540c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:42.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 0x7fc8540c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:42.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 -- 172.21.15.107:0/618589384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c00bd40 con 0x7fc8540cdfb0 2023-11-24T08:23:42.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc858b7f640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 0x7fc8540c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:42.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.717+0000 7fc853fff640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098c0 tx=0x7fc8400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.718+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84003ace0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.718+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84003ae60 con 0x7fc8540cdfb0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.718+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc840044640 con 0x7fc8540cdfb0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.718+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc840009400 con 0x7fc8540cdfb0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.718+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8540c90e0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.719+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8541f36c0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.720+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fc84004d440 con 0x7fc8540cdfb0 2023-11-24T08:23:42.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.722+0000 7fc8517fa640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 0x7fc83806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:42.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.722+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8400aac00 con 0x7fc8540cdfb0 2023-11-24T08:23:42.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.722+0000 7fc858b7f640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 0x7fc83806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:42.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.723+0000 7fc858b7f640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 0x7fc83806f820 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc8480015e0 tx=0x7fc848009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:42.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.726+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8541f36c0 con 0x7fc8540cdfb0 2023-11-24T08:23:42.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:42.984+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot"]} v 0) v1 -- 0x7fc8541f36c0 con 0x7fc8540cdfb0 2023-11-24T08:23:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.610+0000 7fc8517fa640 1 -- 172.21.15.107:0/618589384 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot"]}]=0 v955) v1 ==== 192+0+0 (secure 0 0 0) 0x7fc8541f36c0 con 0x7fc8540cdfb0 2023-11-24T08:23:43.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 msgr2=0x7fc83806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 0x7fc83806f820 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc8480015e0 tx=0x7fc848009040 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 msgr2=0x7fc854149090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098c0 tx=0x7fc8400046c0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 shutdown_connections 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc83806d360 0x7fc83806f820 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc85415a4f0 0x7fc8540cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8540cdfb0 0x7fc854149090 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 --2- 172.21.15.107:0/618589384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8540b20c0 0x7fc8540c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.613+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 >> 172.21.15.107:0/618589384 conn(0x7fc8540b0c70 msgr2=0x7fc8540ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.614+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 shutdown_connections 2023-11-24T08:23:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.614+0000 7fc85ade3640 1 -- 172.21.15.107:0/618589384 wait complete. 2023-11-24T08:23:43.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:23:43.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.843+0000 7f2d9e832640 1 Processor -- start 2023-11-24T08:23:43.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.843+0000 7f2d9e832640 1 -- start start 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e832640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d9815d3c0 0x7f2d980d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e832640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 0x7f2d980b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e832640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b6090 con 0x7f2d980d0e10 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e832640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d980cf0b0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e832640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d980d1c80 con 0x7f2d980d0360 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9d830640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d9815d3c0 0x7f2d980d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9d830640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2d9815d3c0 0x7f2d980d15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59806/0 (socket says 172.21.15.107:59806) 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9d830640 1 -- 172.21.15.107:0/2041414875 learned_addr learned my addr 172.21.15.107:0/2041414875 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:43.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9d02f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2d980d0e10 0x7f2d980cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53328/0 (socket says 172.21.15.107:53328) 2023-11-24T08:23:43.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9e031640 1 --2- 172.21.15.107:0/2041414875 >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 0x7f2d980b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.845+0000 7f2d9d830640 1 -- 172.21.15.107:0/2041414875 >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 msgr2=0x7f2d980b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.846+0000 7f2d9d830640 1 --2- 172.21.15.107:0/2041414875 >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 0x7f2d980b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.846+0000 7f2d9d830640 1 -- 172.21.15.107:0/2041414875 >> v1:172.21.15.107:6789/0 conn(0x7f2d980d0e10 legacy=0x7f2d980cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.846+0000 7f2d9c82e640 1 -- 172.21.15.107:0/2041414875 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690934636 0 0) 0x7f2d980b6090 con 0x7f2d980d0e10 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.846+0000 7f2d9d830640 1 -- 172.21.15.107:0/2041414875 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980d1f60 con 0x7f2d9815d3c0 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.846+0000 7f2d9e031640 1 --2- 172.21.15.107:0/2041414875 >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 0x7f2d980b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9d830640 1 --2- 172.21.15.107:0/2041414875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980d15f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f2d9808cbc0 tx=0x7f2d9402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d060178167d862b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9c82e640 1 -- 172.21.15.107:0/2041414875 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d94030860 con 0x7f2d9815d3c0 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9c82e640 1 -- 172.21.15.107:0/2041414875 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d940309c0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9c82e640 1 -- 172.21.15.107:0/2041414875 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d94030d50 con 0x7f2d9815d3c0 2023-11-24T08:23:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9e832640 1 -- 172.21.15.107:0/2041414875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 msgr2=0x7f2d980d15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.847+0000 7f2d9e832640 1 --2- 172.21.15.107:0/2041414875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980d15f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f2d9808cbc0 tx=0x7f2d9402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 -- 172.21.15.107:0/2041414875 shutdown_connections 2023-11-24T08:23:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 --2- 172.21.15.107:0/2041414875 >> v2:172.21.15.120:3300/0 conn(0x7f2d980d0360 0x7f2d980b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 --2- 172.21.15.107:0/2041414875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980d15f0 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 -- 172.21.15.107:0/2041414875 >> 172.21.15.107:0/2041414875 conn(0x7f2d9809c570 msgr2=0x7f2d980d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 -- 172.21.15.107:0/2041414875 shutdown_connections 2023-11-24T08:23:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.848+0000 7f2d9e832640 1 -- 172.21.15.107:0/2041414875 wait complete. 2023-11-24T08:23:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.849+0000 7f2d9e832640 1 Processor -- start 2023-11-24T08:23:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.850+0000 7f2d9e832640 1 -- start start 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.850+0000 7f2d9e832640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 0x7f2d9809fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.850+0000 7f2d9e832640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 0x7f2d9814b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e832640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e832640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9814c5e0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e832640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9814c020 con 0x7f2d980d0360 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e832640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9814c300 con 0x7f2d980d0e10 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59818/0 (socket says 172.21.15.107:59818) 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 -- 172.21.15.107:0/1770549910 learned_addr learned my addr 172.21.15.107:0/1770549910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9d02f640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 0x7f2d9814b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9d830640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 0x7f2d9809fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 -- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 msgr2=0x7f2d9814b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 0x7f2d9814b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 -- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 msgr2=0x7f2d9809fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 0x7f2d9809fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9e031640 1 -- 172.21.15.107:0/1770549910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d940094d0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9d830640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 0x7f2d9809fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.851+0000 7f2d9d02f640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 0x7f2d9814b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d9e031640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000ce70 tx=0x7f2d9000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90003b90 con 0x7f2d9815d3c0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d90003d10 con 0x7f2d9815d3c0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d980b21b0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90004260 con 0x7f2d9815d3c0 2023-11-24T08:23:43.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.852+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d980a0330 con 0x7f2d9815d3c0 2023-11-24T08:23:43.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.853+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d980b26d0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.854+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f2d900043c0 con 0x7f2d9815d3c0 2023-11-24T08:23:43.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.855+0000 7f2d8e7fc640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 0x7f2d7c06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:43.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.855+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2d90089130 con 0x7f2d9815d3c0 2023-11-24T08:23:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.856+0000 7f2d9d830640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 0x7f2d7c06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.856+0000 7f2d9d830640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 0x7f2d7c06f820 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f2d94036000 tx=0x7f2d9403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:43.860+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d90055ce0 con 0x7f2d9815d3c0 2023-11-24T08:23:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.050+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2d9804e840 con 0x7f2d9815d3c0 2023-11-24T08:23:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.058+0000 7f2d8e7fc640 1 -- 172.21.15.107:0/1770549910 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v55)=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d90003e80 con 0x7f2d9815d3c0 2023-11-24T08:23:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 msgr2=0x7f2d7c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 0x7f2d7c06f820 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f2d94036000 tx=0x7f2d9403b040 comp rx=0 tx=0).stop 2023-11-24T08:23:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 msgr2=0x7f2d980b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000ce70 tx=0x7f2d9000a710 comp rx=0 tx=0).stop 2023-11-24T08:23:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 shutdown_connections 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2d7c06d360 0x7f2d7c06f820 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d9815d3c0 0x7f2d980b29b0 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d980d0e10 0x7f2d9814b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 --2- 172.21.15.107:0/1770549910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d980d0360 0x7f2d9809fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 >> 172.21.15.107:0/1770549910 conn(0x7f2d9809c570 msgr2=0x7f2d9806c640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 shutdown_connections 2023-11-24T08:23:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.061+0000 7f2d9e832640 1 -- 172.21.15.107:0/1770549910 wait complete. 2023-11-24T08:23:44.081 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:23:44.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:23:44.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:23:44.317 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:23:44.317 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:44.317 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:44.377 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:23:44.377 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:44.377 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:44.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:23:44.436 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:44.436 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:44.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:23:44.496 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:23:44.496 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:44.555 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:23:44.556 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:23:44.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:44.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.770+0000 7fa60859e640 1 Processor -- start 2023-11-24T08:23:44.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.770+0000 7fa60859e640 1 -- start start 2023-11-24T08:23:44.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60859e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa60015c7f0 0x7fa60006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60859e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6000cf450 0x7fa6000cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60859e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6000cece0 con 0x7fa6000cdfb0 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60859e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa600156010 con 0x7fa6000cf450 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60859e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60006ce30 con 0x7fa60015c7f0 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa605b39640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6000cf450 0x7fa6000cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa605b39640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6000cf450 0x7fa6000cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59826/0 (socket says 172.21.15.107:59826) 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa605b39640 1 -- 172.21.15.107:0/664678262 learned_addr learned my addr 172.21.15.107:0/664678262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa605b39640 1 -- 172.21.15.107:0/664678262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa60015c7f0 msgr2=0x7fa60006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa60633a640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa60015c7f0 0x7fa60006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.772+0000 7fa605b39640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa60015c7f0 0x7fa60006db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.773+0000 7fa605b39640 1 -- 172.21.15.107:0/664678262 >> v1:172.21.15.120:6789/0 conn(0x7fa6000cdfb0 legacy=0x7fa6000ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.773+0000 7fa605b39640 1 -- 172.21.15.107:0/664678262 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60006d040 con 0x7fa6000cf450 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.774+0000 7fa605b39640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6000cf450 0x7fa6000cf8d0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009870 tx=0x7fa5f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ff4cbb20118fec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.774+0000 7fa605338640 1 -- 172.21.15.107:0/664678262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0031640 con 0x7fa6000cf450 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.774+0000 7fa605338640 1 -- 172.21.15.107:0/664678262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f00317a0 con 0x7fa6000cf450 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.774+0000 7fa605338640 1 -- 172.21.15.107:0/664678262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0031b30 con 0x7fa6000cf450 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 -- 172.21.15.107:0/664678262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6000cf450 msgr2=0x7fa6000cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6000cf450 0x7fa6000cf8d0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009870 tx=0x7fa5f002b8f0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 -- 172.21.15.107:0/664678262 shutdown_connections 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6000cf450 0x7fa6000cf8d0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 --2- 172.21.15.107:0/664678262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa60015c7f0 0x7fa60006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 -- 172.21.15.107:0/664678262 >> 172.21.15.107:0/664678262 conn(0x7fa60006a920 msgr2=0x7fa60006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 -- 172.21.15.107:0/664678262 shutdown_connections 2023-11-24T08:23:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.775+0000 7fa60859e640 1 -- 172.21.15.107:0/664678262 wait complete. 2023-11-24T08:23:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.777+0000 7fa60859e640 1 Processor -- start 2023-11-24T08:23:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.777+0000 7fa60859e640 1 -- start start 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6000cdfb0 0x7fa600159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6000cf450 0x7fa60014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60015aea0 con 0x7fa60015c7f0 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa600159e00 con 0x7fa6000cf450 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60859e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60015a0e0 con 0x7fa6000cdfb0 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59840/0 (socket says 172.21.15.107:59840) 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 -- 172.21.15.107:0/1200313206 learned_addr learned my addr 172.21.15.107:0/1200313206 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa60633a640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6000cdfb0 0x7fa600159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa605b39640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6000cf450 0x7fa60014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 -- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6000cdfb0 msgr2=0x7fa600159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6000cdfb0 0x7fa600159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 -- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6000cf450 msgr2=0x7fa60014b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6000cf450 0x7fa60014b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.779+0000 7fa606b3b640 1 -- 172.21.15.107:0/1200313206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f00094d0 con 0x7fa60015c7f0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa606b3b640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c780 tx=0x7fa5f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8004b10 con 0x7fa60015c7f0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f8004c70 con 0x7fa60015c7f0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60015a3c0 con 0x7fa60015c7f0 2023-11-24T08:23:44.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f800aa90 con 0x7fa60015c7f0 2023-11-24T08:23:44.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.780+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60015a980 con 0x7fa60015c7f0 2023-11-24T08:23:44.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.781+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60015ab70 con 0x7fa60015c7f0 2023-11-24T08:23:44.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.783+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fa5f800abf0 con 0x7fa60015c7f0 2023-11-24T08:23:44.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.784+0000 7fa5f6ffd640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 0x7fa5e406f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:44.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.784+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa5f8088a00 con 0x7fa60015c7f0 2023-11-24T08:23:44.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.784+0000 7fa60633a640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 0x7fa5e406f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:44.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.785+0000 7fa60633a640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 0x7fa5e406f850 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00b7c0 tx=0x7fa5fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:44.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:44.788+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5f8055510 con 0x7fa60015c7f0 2023-11-24T08:23:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.015+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa600067470 con 0x7fa60015c7f0 2023-11-24T08:23:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.016+0000 7fa5f6ffd640 1 -- 172.21.15.107:0/1200313206 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+3715 (secure 0 0 0) 0x7fa5f8058b30 con 0x7fa60015c7f0 2023-11-24T08:23:45.018 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:45.018 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":141,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/863222921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":863222921},{"type":"v1","addr":"172.21.15.120:6834","nonce":863222921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":7843,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3684581645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3684581645},{"type":"v1","addr":"172.21.15.120:6835","nonce":3684581645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":12201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4117577322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4117577322},{"type":"v1","addr":"172.21.15.107:6835","nonce":4117577322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":12654,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1561916155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1561916155},{"type":"v1","addr":"172.21.15.107:6837","nonce":1561916155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[]} 2023-11-24T08:23:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 141 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.018+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 msgr2=0x7fa5e406f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.018+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 0x7fa5e406f850 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00b7c0 tx=0x7fa5fc008040 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 msgr2=0x7fa6001510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c780 tx=0x7fa5f800cc40 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 shutdown_connections 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5e406d390 0x7fa5e406f850 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa60015c7f0 0x7fa6001510c0 unknown :-1 s=CLOSED pgs=3145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6000cf450 0x7fa60014b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 --2- 172.21.15.107:0/1200313206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6000cdfb0 0x7fa600159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 >> 172.21.15.107:0/1200313206 conn(0x7fa60006a920 msgr2=0x7fa6000cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 shutdown_connections 2023-11-24T08:23:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.019+0000 7fa60859e640 1 -- 172.21.15.107:0/1200313206 wait complete. 2023-11-24T08:23:45.040 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:23:45.040 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:23:45.040 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:23:45.040 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:23:45.040 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:23:45.040 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:23:45.041 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:23:45.041 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:23:45.041 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:23:45.041 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:23:45.041 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:23:45.041 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:23:45.041 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:23:45.042 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:23:45.042 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:23:45.042 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:23:45.063 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:23:45.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:23:45.067 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:23:45.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:23:45.071 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:23:45.071 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:23:45.072 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:23:45.072 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:23:45.116 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:23:45.117 DEBUG:teuthology.parallel:result is None 2023-11-24T08:23:45.120 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:23:45.120 DEBUG:teuthology.parallel:result is None 2023-11-24T08:23:45.124 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:23:45.124 DEBUG:teuthology.parallel:result is None 2023-11-24T08:23:45.124 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:23:45.125 DEBUG:teuthology.parallel:result is None 2023-11-24T08:23:45.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:23:45.275 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:23:45.279 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:23:45.289 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:23:45.289 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:23:45.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.317+0000 7f31840e0640 1 Processor -- start 2023-11-24T08:23:45.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.318+0000 7f31840e0640 1 -- start start 2023-11-24T08:23:45.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f31840e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f31840e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c15ec90 con 0x7f317c072160 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f31840e0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c0cd530 con 0x7f317c0c49f0 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f31840e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0c4ed0 con 0x7f317c06de10 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f3181e7c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f317c072160 0x7f317c06d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53346/0 (socket says 172.21.15.107:53346) 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f3181e7c640 1 -- 172.21.15.107:0/2722052300 learned_addr learned my addr 172.21.15.107:0/2722052300 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f318167b640 1 --2- 172.21.15.107:0/2722052300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.319+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280037203 0 0) 0x7f3178001110 con 0x7f317c072160 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3168003520 con 0x7f317c072160 2023-11-24T08:23:45.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098096101 0 0) 0x7f317c0cd530 con 0x7f317c0c49f0 2023-11-24T08:23:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3178001110 con 0x7f317c0c49f0 2023-11-24T08:23:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f318167b640 1 -- 172.21.15.107:0/2722052300 >> v1:172.21.15.120:6789/0 conn(0x7f317c0c49f0 legacy=0x7f317c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f318167b640 1 -- 172.21.15.107:0/2722052300 >> v1:172.21.15.107:6789/0 conn(0x7f317c072160 legacy=0x7f317c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.320+0000 7f318167b640 1 -- 172.21.15.107:0/2722052300 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c0c51b0 con 0x7f317c06de10 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.321+0000 7f318167b640 1 --2- 172.21.15.107:0/2722052300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c15e750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f316c0097d0 tx=0x7f316c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ef0812003bec70 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.321+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c031890 con 0x7f317c06de10 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.321+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316c0319f0 con 0x7f317c06de10 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.321+0000 7f3180e7a640 1 -- 172.21.15.107:0/2722052300 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c031d80 con 0x7f317c06de10 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 -- 172.21.15.107:0/2722052300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 msgr2=0x7f317c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 --2- 172.21.15.107:0/2722052300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c15e750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f316c0097d0 tx=0x7f316c02b930 comp rx=0 tx=0).stop 2023-11-24T08:23:45.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 -- 172.21.15.107:0/2722052300 shutdown_connections 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 --2- 172.21.15.107:0/2722052300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c15e750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 -- 172.21.15.107:0/2722052300 >> 172.21.15.107:0/2722052300 conn(0x7f317c0b0c50 msgr2=0x7f317c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 -- 172.21.15.107:0/2722052300 shutdown_connections 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.322+0000 7f31840e0640 1 -- 172.21.15.107:0/2722052300 wait complete. 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.324+0000 7f31840e0640 1 Processor -- start 2023-11-24T08:23:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.324+0000 7f31840e0640 1 -- start start 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c09a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 0x7f317c09c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c09cb20 con 0x7f317c0c49f0 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0c7140 con 0x7f317c072160 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f31840e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0c79e0 con 0x7f317c06de10 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59888/0 (socket says 172.21.15.107:59888) 2023-11-24T08:23:45.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 -- 172.21.15.107:0/2471345563 learned_addr learned my addr 172.21.15.107:0/2471345563 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318167b640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 0x7f317c09c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f3181e7c640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c09a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 -- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 msgr2=0x7f317c09a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c09a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 -- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 msgr2=0x7f317c09c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 0x7f317c09c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318267d640 1 -- 172.21.15.107:0/2471345563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c009400 con 0x7f317c0c49f0 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f318167b640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 0x7f317c09c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.325+0000 7f3181e7c640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c09a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f318267d640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f317400c680 tx=0x7f317400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174003b70 con 0x7f317c0c49f0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3174003cd0 con 0x7f317c0c49f0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174005310 con 0x7f317c0c49f0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c159750 con 0x7f317c0c49f0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.326+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c0c7390 con 0x7f317c0c49f0 2023-11-24T08:23:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.327+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c0c7b50 con 0x7f317c0c49f0 2023-11-24T08:23:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.328+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f3174005470 con 0x7f317c0c49f0 2023-11-24T08:23:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.329+0000 7f31727fc640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 0x7f316006f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.329+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3174088640 con 0x7f317c0c49f0 2023-11-24T08:23:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.330+0000 7f3181e7c640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 0x7f316006f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.330+0000 7f3181e7c640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 0x7f316006f850 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3178001370 tx=0x7f3178009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.334+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3174055150 con 0x7f317c0c49f0 2023-11-24T08:23:45.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:45.533+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f317c04e8b0 con 0x7f317c0c49f0 2023-11-24T08:23:46.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:23:46.472 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:23:46.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.661+0000 7f31727fc640 1 -- 172.21.15.107:0/2471345563 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v230) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3174058770 con 0x7f317c0c49f0 2023-11-24T08:23:46.663 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:23:46.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 msgr2=0x7f316006f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 0x7f316006f850 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3178001370 tx=0x7f3178009040 comp rx=0 tx=0).stop 2023-11-24T08:23:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 msgr2=0x7f317c159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f317400c680 tx=0x7f317400cb40 comp rx=0 tx=0).stop 2023-11-24T08:23:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 shutdown_connections 2023-11-24T08:23:46.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f316006d390 0x7f316006f850 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317c0c49f0 0x7f317c159210 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f317c072160 0x7f317c09c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 --2- 172.21.15.107:0/2471345563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f317c06de10 0x7f317c09a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 >> 172.21.15.107:0/2471345563 conn(0x7f317c0b0c50 msgr2=0x7f317c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 shutdown_connections 2023-11-24T08:23:46.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.665+0000 7f31840e0640 1 -- 172.21.15.107:0/2471345563 wait complete. 2023-11-24T08:23:46.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:23:46.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.900+0000 7fa62b189640 1 Processor -- start 2023-11-24T08:23:46.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.901+0000 7fa62b189640 1 -- start start 2023-11-24T08:23:46.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.902+0000 7fa62b189640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa62415d520 0x7fa6240b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.902+0000 7fa62b189640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6240b6be0 0x7fa6240d2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.902+0000 7fa62b189640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6240d3120 con 0x7fa6240d27a0 2023-11-24T08:23:46.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.902+0000 7fa62b189640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6240b2930 con 0x7fa6240b6be0 2023-11-24T08:23:46.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.902+0000 7fa62b189640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6240d1420 con 0x7fa62415d520 2023-11-24T08:23:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6240b6be0 0x7fa6240d2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa6240b6be0 0x7fa6240d2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59894/0 (socket says 172.21.15.107:59894) 2023-11-24T08:23:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 -- 172.21.15.107:0/917173408 learned_addr learned my addr 172.21.15.107:0/917173408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa628f25640 1 --2- 172.21.15.107:0/917173408 >> v2:172.21.15.120:3300/0 conn(0x7fa62415d520 0x7fa6240b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 -- 172.21.15.107:0/917173408 >> v2:172.21.15.120:3300/0 conn(0x7fa62415d520 msgr2=0x7fa6240b8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:46.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 --2- 172.21.15.107:0/917173408 >> v2:172.21.15.120:3300/0 conn(0x7fa62415d520 0x7fa6240b8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 -- 172.21.15.107:0/917173408 >> v1:172.21.15.107:6789/0 conn(0x7fa6240d27a0 legacy=0x7fa6240b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.903+0000 7fa629726640 1 -- 172.21.15.107:0/917173408 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6240d1700 con 0x7fa6240b6be0 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.904+0000 7fa629726640 1 --2- 172.21.15.107:0/917173408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240b6be0 0x7fa6240d2be0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fa6240b5b10 tx=0x7fa61c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801bce76c2b2a0b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.904+0000 7fa61b7fe640 1 -- 172.21.15.107:0/917173408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c02f850 con 0x7fa6240b6be0 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa61b7fe640 1 -- 172.21.15.107:0/917173408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c02f9b0 con 0x7fa6240b6be0 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa61b7fe640 1 -- 172.21.15.107:0/917173408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c02fd40 con 0x7fa6240b6be0 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 -- 172.21.15.107:0/917173408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240b6be0 msgr2=0x7fa6240d2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 --2- 172.21.15.107:0/917173408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240b6be0 0x7fa6240d2be0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fa6240b5b10 tx=0x7fa61c02ba30 comp rx=0 tx=0).stop 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 -- 172.21.15.107:0/917173408 shutdown_connections 2023-11-24T08:23:46.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 --2- 172.21.15.107:0/917173408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240b6be0 0x7fa6240d2be0 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 --2- 172.21.15.107:0/917173408 >> v2:172.21.15.120:3300/0 conn(0x7fa62415d520 0x7fa6240b8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 -- 172.21.15.107:0/917173408 >> 172.21.15.107:0/917173408 conn(0x7fa624061340 msgr2=0x7fa624061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:46.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 -- 172.21.15.107:0/917173408 shutdown_connections 2023-11-24T08:23:46.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.905+0000 7fa62b189640 1 -- 172.21.15.107:0/917173408 wait complete. 2023-11-24T08:23:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.907+0000 7fa62b189640 1 Processor -- start 2023-11-24T08:23:46.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.907+0000 7fa62b189640 1 -- start start 2023-11-24T08:23:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6240b6be0 0x7fa6240cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa62415d520 0x7fa62415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240cb690 con 0x7fa6240d27a0 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240cb3c0 con 0x7fa62415d520 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa62b189640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62415daa0 con 0x7fa6240b6be0 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa61bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa61bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59908/0 (socket says 172.21.15.107:59908) 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.908+0000 7fa61bfff640 1 -- 172.21.15.107:0/4041740665 learned_addr learned my addr 172.21.15.107:0/4041740665 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa628f25640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6240b6be0 0x7fa6240cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa629726640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa62415d520 0x7fa62415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 -- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6240b6be0 msgr2=0x7fa6240cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6240b6be0 0x7fa6240cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 -- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa62415d520 msgr2=0x7fa62415b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa62415d520 0x7fa62415b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 -- 172.21.15.107:0/4041740665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c0094d0 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa61bfff640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c450 tx=0x7fa61400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6140052d0 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa614005430 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614012870 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62415dd80 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.909+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62415e340 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.910+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62415e560 con 0x7fa6240d27a0 2023-11-24T08:23:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.911+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fa6140129d0 con 0x7fa6240d27a0 2023-11-24T08:23:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.912+0000 7fa6197fa640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 0x7fa61006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.912+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa614088420 con 0x7fa6240d27a0 2023-11-24T08:23:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.912+0000 7fa628f25640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 0x7fa61006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.913+0000 7fa628f25640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 0x7fa61006f820 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa620009e90 tx=0x7fa62000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:46.916+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa614054f30 con 0x7fa6240d27a0 2023-11-24T08:23:47.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.154+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa62404e8b0 con 0x7fa6240d27a0 2023-11-24T08:23:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.155+0000 7fa6197fa640 1 -- 172.21.15.107:0/4041740665 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa614058550 con 0x7fa6240d27a0 2023-11-24T08:23:47.157 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:47.157 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:47.157 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:23:47.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.158 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:23:47.158 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:23:47.158 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:23:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:23:47.161 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:23:47.162 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:23:47.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:23:47.162 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.166 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:23:47.167 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.168 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:23:47.169 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:23:47.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:23:47.172 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.173 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:23:47.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:47.175 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:23:47.175 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:23:47.175 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:47.176 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:23:47.177 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 msgr2=0x7fa61006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 0x7fa61006f820 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa620009e90 tx=0x7fa62000b040 comp rx=0 tx=0).stop 2023-11-24T08:23:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 msgr2=0x7fa6241598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c450 tx=0x7fa61400c910 comp rx=0 tx=0).stop 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 shutdown_connections 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa61006d360 0x7fa61006f820 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa62415d520 0x7fa62415b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240d27a0 0x7fa6241598a0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 --2- 172.21.15.107:0/4041740665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6240b6be0 0x7fa6240cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 >> 172.21.15.107:0/4041740665 conn(0x7fa624061340 msgr2=0x7fa6240b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 shutdown_connections 2023-11-24T08:23:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.157+0000 7fa62b189640 1 -- 172.21.15.107:0/4041740665 wait complete. 2023-11-24T08:23:47.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:47.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.381+0000 7f87e031d640 1 Processor -- start 2023-11-24T08:23:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.381+0000 7f87e031d640 1 -- start start 2023-11-24T08:23:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87e031d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87d81461d0 0x7f87d80b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87e031d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 0x7f87d8167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87e031d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d806d180 con 0x7f87d80b4fe0 2023-11-24T08:23:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87e031d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d806ba70 con 0x7f87d81461d0 2023-11-24T08:23:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87e031d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d806bd50 con 0x7f87d80b3a60 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87de0b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87d81461d0 0x7f87d80b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87de0b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87d81461d0 0x7f87d80b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59918/0 (socket says 172.21.15.107:59918) 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.383+0000 7f87de0b9640 1 -- 172.21.15.107:0/3741848453 learned_addr learned my addr 172.21.15.107:0/3741848453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de8ba640 1 --2- 172.21.15.107:0/3741848453 >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 0x7f87d8167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de0b9640 1 -- 172.21.15.107:0/3741848453 >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 msgr2=0x7f87d8167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de0b9640 1 --2- 172.21.15.107:0/3741848453 >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 0x7f87d8167110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de0b9640 1 -- 172.21.15.107:0/3741848453 >> v1:172.21.15.107:6789/0 conn(0x7f87d80b4fe0 legacy=0x7f87d80b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de0b9640 1 -- 172.21.15.107:0/3741848453 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d806c030 con 0x7f87d81461d0 2023-11-24T08:23:47.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.384+0000 7f87de8ba640 1 --2- 172.21.15.107:0/3741848453 >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 0x7f87d8167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:23:47.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87de0b9640 1 --2- 172.21.15.107:0/3741848453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80b4aa0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009870 tx=0x7f87d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d0a3305bd15693 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87dd0b7640 1 -- 172.21.15.107:0/3741848453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4031870 con 0x7f87d81461d0 2023-11-24T08:23:47.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87dd0b7640 1 -- 172.21.15.107:0/3741848453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d40319d0 con 0x7f87d81461d0 2023-11-24T08:23:47.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87dd0b7640 1 -- 172.21.15.107:0/3741848453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4031d60 con 0x7f87d81461d0 2023-11-24T08:23:47.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87e031d640 1 -- 172.21.15.107:0/3741848453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 msgr2=0x7f87d80b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.386+0000 7f87e031d640 1 --2- 172.21.15.107:0/3741848453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80b4aa0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009870 tx=0x7f87d402b910 comp rx=0 tx=0).stop 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 -- 172.21.15.107:0/3741848453 shutdown_connections 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 --2- 172.21.15.107:0/3741848453 >> v2:172.21.15.120:3300/0 conn(0x7f87d80b3a60 0x7f87d8167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 --2- 172.21.15.107:0/3741848453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80b4aa0 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 -- 172.21.15.107:0/3741848453 >> 172.21.15.107:0/3741848453 conn(0x7f87d8061340 msgr2=0x7f87d8061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 -- 172.21.15.107:0/3741848453 shutdown_connections 2023-11-24T08:23:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.387+0000 7f87e031d640 1 -- 172.21.15.107:0/3741848453 wait complete. 2023-11-24T08:23:47.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.389+0000 7f87e031d640 1 Processor -- start 2023-11-24T08:23:47.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.389+0000 7f87e031d640 1 -- start start 2023-11-24T08:23:47.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87d80b3a60 0x7f87d814fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d80b4fe0 0x7f87d81f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d814d0c0 con 0x7f87d81461d0 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d81f2be0 con 0x7f87d80b4fe0 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87e031d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80cdde0 con 0x7f87d80b3a60 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59930/0 (socket says 172.21.15.107:59930) 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 -- 172.21.15.107:0/1957711286 learned_addr learned my addr 172.21.15.107:0/1957711286 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87dd8b8640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d80b4fe0 0x7f87d81f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de0b9640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87d80b3a60 0x7f87d814fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 -- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87d80b3a60 msgr2=0x7f87d814fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87d80b3a60 0x7f87d814fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 -- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d80b4fe0 msgr2=0x7f87d81f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d80b4fe0 0x7f87d81f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.390+0000 7f87de8ba640 1 -- 172.21.15.107:0/1957711286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d40094d0 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87de8ba640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007ef0 tx=0x7f87d000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d0003b90 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d000f040 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d000de80 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d8146730 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.391+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d80ce3a0 con 0x7f87d81461d0 2023-11-24T08:23:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.393+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d8067470 con 0x7f87d81461d0 2023-11-24T08:23:47.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.393+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f87d0004030 con 0x7f87d81461d0 2023-11-24T08:23:47.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.394+0000 7f87ceffd640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 0x7f87bc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.394+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f87d0088ac0 con 0x7f87d81461d0 2023-11-24T08:23:47.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.395+0000 7f87de0b9640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 0x7f87bc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.395+0000 7f87de0b9640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 0x7f87bc06f820 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f87d4036000 tx=0x7f87d403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.398+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87d008fe60 con 0x7f87d81461d0 2023-11-24T08:23:47.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.589+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d80b4a80 con 0x7f87d81461d0 2023-11-24T08:23:47.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.591+0000 7f87ceffd640 1 -- 172.21.15.107:0/1957711286 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f87d001a020 con 0x7f87d81461d0 2023-11-24T08:23:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:47.593 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:46.652800+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 msgr2=0x7f87bc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 0x7f87bc06f820 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f87d4036000 tx=0x7f87d403c040 comp rx=0 tx=0).stop 2023-11-24T08:23:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 msgr2=0x7f87d80cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007ef0 tx=0x7f87d000d4a0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 shutdown_connections 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f87bc06d360 0x7f87bc06f820 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87d81461d0 0x7f87d80cd590 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d80b4fe0 0x7f87d81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 --2- 172.21.15.107:0/1957711286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87d80b3a60 0x7f87d814fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 >> 172.21.15.107:0/1957711286 conn(0x7f87d8061340 msgr2=0x7f87d8142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 shutdown_connections 2023-11-24T08:23:47.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.594+0000 7f87e031d640 1 -- 172.21.15.107:0/1957711286 wait complete. 2023-11-24T08:23:47.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:23:47.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:23:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.825+0000 7efedc94c640 1 Processor -- start 2023-11-24T08:23:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.825+0000 7efedc94c640 1 -- start start 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efedc94c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efed80c49f0 0x7efed815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efedc94c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed815ec90 con 0x7efed8072160 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efedc94c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed80cd530 con 0x7efed806de10 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efedc94c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80c4ed0 con 0x7efed80c49f0 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efed659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efed8072160 0x7efed806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53358/0 (socket says 172.21.15.107:53358) 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efed659c640 1 -- 172.21.15.107:0/3970583021 learned_addr learned my addr 172.21.15.107:0/3970583021 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:47.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.827+0000 7efed6d9d640 1 --2- 172.21.15.107:0/3970583021 >> v2:172.21.15.107:3300/0 conn(0x7efed80c49f0 0x7efed815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.828+0000 7efed6d9d640 1 -- 172.21.15.107:0/3970583021 >> v1:172.21.15.120:6789/0 conn(0x7efed806de10 legacy=0x7efed80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.828+0000 7efed6d9d640 1 -- 172.21.15.107:0/3970583021 >> v1:172.21.15.107:6789/0 conn(0x7efed8072160 legacy=0x7efed806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.828+0000 7efed6d9d640 1 -- 172.21.15.107:0/3970583021 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80c51b0 con 0x7efed80c49f0 2023-11-24T08:23:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.829+0000 7efed6d9d640 1 --2- 172.21.15.107:0/3970583021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed80c49f0 0x7efed815e750 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7efed0009d90 tx=0x7efed002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce40d2f0cae914d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.829+0000 7efed559a640 1 -- 172.21.15.107:0/3970583021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed002cc60 con 0x7efed80c49f0 2023-11-24T08:23:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.829+0000 7efed559a640 1 -- 172.21.15.107:0/3970583021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed002cdc0 con 0x7efed80c49f0 2023-11-24T08:23:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.829+0000 7efed559a640 1 -- 172.21.15.107:0/3970583021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0034660 con 0x7efed80c49f0 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 -- 172.21.15.107:0/3970583021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed80c49f0 msgr2=0x7efed815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 --2- 172.21.15.107:0/3970583021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed80c49f0 0x7efed815e750 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7efed0009d90 tx=0x7efed002beb0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 -- 172.21.15.107:0/3970583021 shutdown_connections 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 --2- 172.21.15.107:0/3970583021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed80c49f0 0x7efed815e750 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 -- 172.21.15.107:0/3970583021 >> 172.21.15.107:0/3970583021 conn(0x7efed80b0c50 msgr2=0x7efed80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 -- 172.21.15.107:0/3970583021 shutdown_connections 2023-11-24T08:23:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.830+0000 7efedc94c640 1 -- 172.21.15.107:0/3970583021 wait complete. 2023-11-24T08:23:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.832+0000 7efedc94c640 1 Processor -- start 2023-11-24T08:23:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.832+0000 7efedc94c640 1 -- start start 2023-11-24T08:23:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 0x7efed80caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed80c49f0 0x7efed80aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed814fbd0 con 0x7efed806de10 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed80afc10 con 0x7efed8072160 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efedc94c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed814fe70 con 0x7efed80c49f0 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59940/0 (socket says 172.21.15.107:59940) 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 -- 172.21.15.107:0/3474665953 learned_addr learned my addr 172.21.15.107:0/3474665953 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed5d9b640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 0x7efed80caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 -- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed80c49f0 msgr2=0x7efed80aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed80c49f0 0x7efed80aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 -- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 msgr2=0x7efed80caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 0x7efed80caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed659c640 1 -- 172.21.15.107:0/3474665953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0009a70 con 0x7efed806de10 2023-11-24T08:23:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.833+0000 7efed5d9b640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 0x7efed80caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efed659c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7efec800c4a0 tx=0x7efec800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8005ce0 con 0x7efed806de10 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8005e40 con 0x7efed806de10 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed8150150 con 0x7efed806de10 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8007270 con 0x7efed806de10 2023-11-24T08:23:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.834+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed80af400 con 0x7efed806de10 2023-11-24T08:23:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.835+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed8150390 con 0x7efed806de10 2023-11-24T08:23:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.836+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7efec8007490 con 0x7efed806de10 2023-11-24T08:23:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.837+0000 7efec6ffd640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 0x7efeb806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.837+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efec8087bd0 con 0x7efed806de10 2023-11-24T08:23:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.837+0000 7efed5d9b640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 0x7efeb806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:47.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.838+0000 7efed5d9b640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 0x7efeb806f820 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efed80cf430 tx=0x7efecc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:47.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:47.841+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efed8150390 con 0x7efed806de10 2023-11-24T08:23:48.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.034+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efed815ec90 con 0x7efed806de10 2023-11-24T08:23:48.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.035+0000 7efec6ffd640 1 -- 172.21.15.107:0/3474665953 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v230) v1 ==== 93+0+10 (secure 0 0 0) 0x7efed815ec90 con 0x7efed806de10 2023-11-24T08:23:48.037 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:23:48.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 msgr2=0x7efeb806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 0x7efeb806f820 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efed80cf430 tx=0x7efecc009040 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 msgr2=0x7efed80caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7efec800c4a0 tx=0x7efec800c960 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 shutdown_connections 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efeb806d360 0x7efeb806f820 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed80c49f0 0x7efed80aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed8072160 0x7efed80caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.038+0000 7efedc94c640 1 --2- 172.21.15.107:0/3474665953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efed806de10 0x7efed80caa20 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.039+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 >> 172.21.15.107:0/3474665953 conn(0x7efed80b0c50 msgr2=0x7efed80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.039+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 shutdown_connections 2023-11-24T08:23:48.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.039+0000 7efedc94c640 1 -- 172.21.15.107:0/3474665953 wait complete. 2023-11-24T08:23:48.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:23:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.266+0000 7fdea9c56640 1 Processor -- start 2023-11-24T08:23:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.266+0000 7fdea9c56640 1 -- start start 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.267+0000 7fdea9c56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdea4159ce0 0x7fdea406de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.267+0000 7fdea9c56640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdea40b21d0 0x7fdea40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea9c56640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea40b3a40 con 0x7fdea40cbf80 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea9c56640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea40cc3d0 con 0x7fdea4159ce0 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea9c56640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea40cdfa0 con 0x7fdea40b21d0 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdea4159ce0 0x7fdea406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdea4159ce0 0x7fdea406de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56956/0 (socket says 172.21.15.107:56956) 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 -- 172.21.15.107:0/1015753848 learned_addr learned my addr 172.21.15.107:0/1015753848 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 -- 172.21.15.107:0/1015753848 >> v2:172.21.15.120:3300/0 conn(0x7fdea40b21d0 msgr2=0x7fdea40cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 --2- 172.21.15.107:0/1015753848 >> v2:172.21.15.120:3300/0 conn(0x7fdea40b21d0 0x7fdea40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 -- 172.21.15.107:0/1015753848 >> v1:172.21.15.120:6789/0 conn(0x7fdea40cbf80 legacy=0x7fdea415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.268+0000 7fdea37fe640 1 -- 172.21.15.107:0/1015753848 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea40cd9e0 con 0x7fdea4159ce0 2023-11-24T08:23:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.269+0000 7fdea37fe640 1 --2- 172.21.15.107:0/1015753848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4159ce0 0x7fdea406de80 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fdea40cb410 tx=0x7fde9402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ec91eb2d0f23d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.269+0000 7fdea27fc640 1 -- 172.21.15.107:0/1015753848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9402eea0 con 0x7fdea4159ce0 2023-11-24T08:23:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea27fc640 1 -- 172.21.15.107:0/1015753848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde94032070 con 0x7fdea4159ce0 2023-11-24T08:23:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea27fc640 1 -- 172.21.15.107:0/1015753848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94038620 con 0x7fdea4159ce0 2023-11-24T08:23:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 -- 172.21.15.107:0/1015753848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4159ce0 msgr2=0x7fdea406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 --2- 172.21.15.107:0/1015753848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4159ce0 0x7fdea406de80 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fdea40cb410 tx=0x7fde9402e0f0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 -- 172.21.15.107:0/1015753848 shutdown_connections 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 --2- 172.21.15.107:0/1015753848 >> v2:172.21.15.120:3300/0 conn(0x7fdea40b21d0 0x7fdea40cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 --2- 172.21.15.107:0/1015753848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4159ce0 0x7fdea406de80 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 -- 172.21.15.107:0/1015753848 >> 172.21.15.107:0/1015753848 conn(0x7fdea406a920 msgr2=0x7fdea406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 -- 172.21.15.107:0/1015753848 shutdown_connections 2023-11-24T08:23:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.270+0000 7fdea9c56640 1 -- 172.21.15.107:0/1015753848 wait complete. 2023-11-24T08:23:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.272+0000 7fdea9c56640 1 Processor -- start 2023-11-24T08:23:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.272+0000 7fdea9c56640 1 -- start start 2023-11-24T08:23:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea40b21d0 0x7fdea414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea4159ce0 0x7fdea409cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f4990 con 0x7fdea40cbf80 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f4be0 con 0x7fdea4159ce0 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea9c56640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f4ec0 con 0x7fdea40b21d0 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56960/0 (socket says 172.21.15.107:56960) 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.273+0000 7fdea2ffd640 1 -- 172.21.15.107:0/547189525 learned_addr learned my addr 172.21.15.107:0/547189525 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea37fe640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea40b21d0 0x7fdea414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 -- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea40b21d0 msgr2=0x7fdea414c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea40b21d0 0x7fdea414c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 -- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea4159ce0 msgr2=0x7fdea409cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea4159ce0 0x7fdea409cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 -- 172.21.15.107:0/547189525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9400bd40 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea2ffd640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00b410 tx=0x7fde8c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c011830 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c0119b0 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c011d60 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.274+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea41f51a0 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.275+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea414c970 con 0x7fdea40cbf80 2023-11-24T08:23:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.276+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea41f56c0 con 0x7fdea40cbf80 2023-11-24T08:23:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.277+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fde8c01a400 con 0x7fdea40cbf80 2023-11-24T08:23:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.278+0000 7fde9bfff640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 0x7fde8406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.278+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fde8c088a80 con 0x7fdea40cbf80 2023-11-24T08:23:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.278+0000 7fdea37fe640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 0x7fde8406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.279+0000 7fdea37fe640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 0x7fde8406f820 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fde94039000 tx=0x7fde94034000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.282+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdea41f56c0 con 0x7fdea40cbf80 2023-11-24T08:23:48.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.477+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdea41f56c0 con 0x7fdea40cbf80 2023-11-24T08:23:48.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.478+0000 7fde9bfff640 1 -- 172.21.15.107:0/547189525 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v230) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdea41f56c0 con 0x7fdea40cbf80 2023-11-24T08:23:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:48.480 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:23:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:23:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:23:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:23:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:23:48.481 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:23:48.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 msgr2=0x7fde8406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 0x7fde8406f820 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fde94039000 tx=0x7fde94034000 comp rx=0 tx=0).stop 2023-11-24T08:23:48.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 msgr2=0x7fdea409c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00b410 tx=0x7fde8c00b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 shutdown_connections 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8406d360 0x7fde8406f820 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea4159ce0 0x7fdea409cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea40cbf80 0x7fdea409c640 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 --2- 172.21.15.107:0/547189525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea40b21d0 0x7fdea414c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 >> 172.21.15.107:0/547189525 conn(0x7fdea406a920 msgr2=0x7fdea40ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:48.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 shutdown_connections 2023-11-24T08:23:48.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.481+0000 7fdea9c56640 1 -- 172.21.15.107:0/547189525 wait complete. 2023-11-24T08:23:48.502 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:23:48.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:23:48.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.721+0000 7f0d6ca1a640 1 Processor -- start 2023-11-24T08:23:48.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.721+0000 7f0d6ca1a640 1 -- start start 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d6ca1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d6815e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d6ca1a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d6806ddb0 0x7f0d68071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d6ca1a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6815ec20 con 0x7f0d680b2ad0 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d6ca1a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d680b4fd0 con 0x7f0d6806ddb0 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d6ca1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d68143c10 con 0x7f0d681431f0 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d5effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0d680b2ad0 0x7f0d6806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44070/0 (socket says 172.21.15.107:44070) 2023-11-24T08:23:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d5effd640 1 -- 172.21.15.107:0/3500606039 learned_addr learned my addr 172.21.15.107:0/3500606039 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.723+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412237055 0 0) 0x7f0d6815ec20 con 0x7f0d680b2ad0 2023-11-24T08:23:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d54005730 con 0x7f0d680b2ad0 2023-11-24T08:23:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d677fe640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d6815e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d67fff640 1 --2- 172.21.15.107:0/3500606039 >> v2:172.21.15.120:3300/0 conn(0x7f0d6806ddb0 0x7f0d68071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789830576 0 0) 0x7f0d54005730 con 0x7f0d680b2ad0 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 msgr2=0x7f0d6815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d6815e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 >> v2:172.21.15.120:3300/0 conn(0x7f0d6806ddb0 msgr2=0x7f0d68071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 --2- 172.21.15.107:0/3500606039 >> v2:172.21.15.120:3300/0 conn(0x7f0d6806ddb0 0x7f0d68071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d680705b0 con 0x7f0d680b2ad0 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0d500017f0 con 0x7f0d680b2ad0 2023-11-24T08:23:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 >> v1:172.21.15.107:6789/0 conn(0x7f0d680b2ad0 legacy=0x7f0d6806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d54005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d540061d0 0x7f0d540069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d54007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680b4fd0 con 0x7f0d54006f10 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d68143c10 con 0x7f0d540061d0 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5400dd50 con 0x7f0d681431f0 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.724+0000 7f0d67fff640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d54007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d677fe640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d54005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d5effd640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d540061d0 0x7f0d540069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d67fff640 1 -- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 msgr2=0x7f0d54005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d67fff640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d681431f0 0x7f0d54005ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d67fff640 1 -- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d540061d0 msgr2=0x7f0d540069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d67fff640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d540061d0 0x7f0d540069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.725+0000 7f0d67fff640 1 -- 172.21.15.107:0/3500606039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54005730 con 0x7f0d54006f10 2023-11-24T08:23:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.726+0000 7f0d67fff640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d54007390 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000ba30 tx=0x7f0d54007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4a8793aeee5bb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.726+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60032680 con 0x7f0d54006f10 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.726+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d600327e0 con 0x7f0d54006f10 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.726+0000 7f0d66ffd640 1 -- 172.21.15.107:0/3500606039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60032b70 con 0x7f0d54006f10 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3500606039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 msgr2=0x7f0d54007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3500606039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d54007390 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000ba30 tx=0x7f0d54007a00 comp rx=0 tx=0).stop 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d67fff640 1 -- 172.21.15.107:0/3500606039 reap_dead start 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3500606039 shutdown_connections 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3500606039 >> 172.21.15.107:0/3500606039 conn(0x7f0d680cb8a0 msgr2=0x7f0d680cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3500606039 shutdown_connections 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.727+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3500606039 wait complete. 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.728+0000 7f0d6ca1a640 1 Processor -- start 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.729+0000 7f0d6ca1a640 1 -- start start 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.729+0000 7f0d6ca1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d6ca1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6815c410 0x7f0d68139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d6ca1a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6814a490 0x7f0d6814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d6ca1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d68139fd0 con 0x7f0d54006f10 2023-11-24T08:23:48.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d6ca1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6815b6e0 con 0x7f0d6815c410 2023-11-24T08:23:48.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d6ca1a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814ae50 con 0x7f0d6814a490 2023-11-24T08:23:48.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56972/0 (socket says 172.21.15.107:56972) 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 -- 172.21.15.107:0/3002181162 learned_addr learned my addr 172.21.15.107:0/3002181162 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 -- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6814a490 msgr2=0x7f0d6814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6814a490 0x7f0d6814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 -- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6815c410 msgr2=0x7f0d68139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6815c410 0x7f0d68139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 -- 172.21.15.107:0/3002181162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6813acf0 con 0x7f0d54006f10 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.730+0000 7f0d677fe640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f0d58007e90 tx=0x7f0d5800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.731+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d580145f0 con 0x7f0d54006f10 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.731+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d58014750 con 0x7f0d54006f10 2023-11-24T08:23:48.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.731+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58014ae0 con 0x7f0d54006f10 2023-11-24T08:23:48.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.731+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6815b930 con 0x7f0d54006f10 2023-11-24T08:23:48.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.731+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6815bef0 con 0x7f0d54006f10 2023-11-24T08:23:48.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.732+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6815c0b0 con 0x7f0d54006f10 2023-11-24T08:23:48.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.733+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f0d58014c40 con 0x7f0d54006f10 2023-11-24T08:23:48.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.734+0000 7f0d64ff9640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 0x7f0d4806f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:48.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.734+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d580894f0 con 0x7f0d54006f10 2023-11-24T08:23:48.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.734+0000 7f0d5effd640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 0x7f0d4806f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:48.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.735+0000 7f0d5effd640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 0x7f0d4806f850 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0d50001a20 tx=0x7f0d50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:48.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.740+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d580560a0 con 0x7f0d54006f10 2023-11-24T08:23:48.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:48.929+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0d6804e840 con 0x7f0d54006f10 2023-11-24T08:23:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.692+0000 7f0d64ff9640 1 -- 172.21.15.107:0/3002181162 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v231) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0d580596c0 con 0x7f0d54006f10 2023-11-24T08:23:49.695 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:23:49.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 msgr2=0x7f0d4806f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 0x7f0d4806f850 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0d50001a20 tx=0x7f0d50009040 comp rx=0 tx=0).stop 2023-11-24T08:23:49.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 msgr2=0x7f0d6813a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f0d58007e90 tx=0x7f0d5800f520 comp rx=0 tx=0).stop 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 shutdown_connections 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0d4806d390 0x7f0d4806f850 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6814a490 0x7f0d6814a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6815c410 0x7f0d68139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 --2- 172.21.15.107:0/3002181162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d54006f10 0x7f0d6813a6d0 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.697+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 >> 172.21.15.107:0/3002181162 conn(0x7f0d680cb8a0 msgr2=0x7f0d680ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.698+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 shutdown_connections 2023-11-24T08:23:49.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.698+0000 7f0d6ca1a640 1 -- 172.21.15.107:0/3002181162 wait complete. 2023-11-24T08:23:49.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:23:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.934+0000 7f14fbf41640 1 Processor -- start 2023-11-24T08:23:49.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.934+0000 7f14fbf41640 1 -- start start 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14fbf41640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f14f415d7f0 0x7f14f40c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14fbf41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14f40c4220 0x7f14f40cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14fbf41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f415e400 con 0x7f14f415dbd0 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14fbf41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40ce880 con 0x7f14f40c4220 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14fbf41640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40c5ed0 con 0x7f14f415d7f0 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14f94dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14f40c4220 0x7f14f40cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14f94dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14f40c4220 0x7f14f40cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56986/0 (socket says 172.21.15.107:56986) 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.936+0000 7f14f94dc640 1 -- 172.21.15.107:0/546167127 learned_addr learned my addr 172.21.15.107:0/546167127 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.937+0000 7f14f9cdd640 1 --2- 172.21.15.107:0/546167127 >> v2:172.21.15.120:3300/0 conn(0x7f14f415d7f0 0x7f14f40c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.937+0000 7f14f94dc640 1 -- 172.21.15.107:0/546167127 >> v2:172.21.15.120:3300/0 conn(0x7f14f415d7f0 msgr2=0x7f14f40c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.937+0000 7f14f94dc640 1 --2- 172.21.15.107:0/546167127 >> v2:172.21.15.120:3300/0 conn(0x7f14f415d7f0 0x7f14f40c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.937+0000 7f14f94dc640 1 -- 172.21.15.107:0/546167127 >> v1:172.21.15.120:6789/0 conn(0x7f14f415dbd0 legacy=0x7f14f40cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.937+0000 7f14f94dc640 1 -- 172.21.15.107:0/546167127 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40c46e0 con 0x7f14f40c4220 2023-11-24T08:23:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.938+0000 7f14f94dc640 1 --2- 172.21.15.107:0/546167127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f40c4220 0x7f14f40cabb0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f14f406ea50 tx=0x7f14f002e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125eb341451aa9ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.938+0000 7f14f8cdb640 1 -- 172.21.15.107:0/546167127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f000b0d0 con 0x7f14f40c4220 2023-11-24T08:23:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.938+0000 7f14f8cdb640 1 -- 172.21.15.107:0/546167127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f000b230 con 0x7f14f40c4220 2023-11-24T08:23:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.938+0000 7f14f8cdb640 1 -- 172.21.15.107:0/546167127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0009600 con 0x7f14f40c4220 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 -- 172.21.15.107:0/546167127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f40c4220 msgr2=0x7f14f40cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 --2- 172.21.15.107:0/546167127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f40c4220 0x7f14f40cabb0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f14f406ea50 tx=0x7f14f002e180 comp rx=0 tx=0).stop 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 -- 172.21.15.107:0/546167127 shutdown_connections 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 --2- 172.21.15.107:0/546167127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f40c4220 0x7f14f40cabb0 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 --2- 172.21.15.107:0/546167127 >> v2:172.21.15.120:3300/0 conn(0x7f14f415d7f0 0x7f14f40c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 -- 172.21.15.107:0/546167127 >> 172.21.15.107:0/546167127 conn(0x7f14f40b0c50 msgr2=0x7f14f40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 -- 172.21.15.107:0/546167127 shutdown_connections 2023-11-24T08:23:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.939+0000 7f14fbf41640 1 -- 172.21.15.107:0/546167127 wait complete. 2023-11-24T08:23:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.941+0000 7f14fbf41640 1 Processor -- start 2023-11-24T08:23:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.941+0000 7f14fbf41640 1 -- start start 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14f40c4220 0x7f14f4151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14f415d7f0 0x7f14f406c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4152350 con 0x7f14f415dbd0 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40c6250 con 0x7f14f415d7f0 2023-11-24T08:23:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fbf41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40c72d0 con 0x7f14f40c4220 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fa4de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fa4de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56988/0 (socket says 172.21.15.107:56988) 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14fa4de640 1 -- 172.21.15.107:0/755048603 learned_addr learned my addr 172.21.15.107:0/755048603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14f94dc640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14f415d7f0 0x7f14f406c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.942+0000 7f14f9cdd640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14f40c4220 0x7f14f4151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 -- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14f40c4220 msgr2=0x7f14f4151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14f40c4220 0x7f14f4151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 -- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14f415d7f0 msgr2=0x7f14f406c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14f415d7f0 0x7f14f406c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 -- 172.21.15.107:0/755048603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f000bd40 con 0x7f14f415dbd0 2023-11-24T08:23:49.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14fa4de640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00c3c0 tx=0x7f14ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:49.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.943+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0052d0 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.944+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec005430 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.944+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec007260 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.944+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f40c6510 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.944+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f40c6ad0 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.945+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f404e8b0 con 0x7f14f415dbd0 2023-11-24T08:23:49.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.946+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f14ec0073c0 con 0x7f14f415dbd0 2023-11-24T08:23:49.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.947+0000 7f14ea7fc640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 0x7f14e006f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:49.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.948+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f14ec0882f0 con 0x7f14f415dbd0 2023-11-24T08:23:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.948+0000 7f14f9cdd640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 0x7f14e006f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:49.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.948+0000 7f14f9cdd640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 0x7f14e006f850 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f14e4006fd0 tx=0x7f14e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:49.953+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14ec054f30 con 0x7f14f415dbd0 2023-11-24T08:23:50.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.145+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f14f40c4000 con 0x7f14f415dbd0 2023-11-24T08:23:50.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.706+0000 7f14ea7fc640 1 -- 172.21.15.107:0/755048603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v232) v1 ==== 170+0+0 (secure 0 0 0) 0x7f14ec020020 con 0x7f14f415dbd0 2023-11-24T08:23:50.709 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:23:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.710+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 msgr2=0x7f14e006f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.710+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 0x7f14e006f850 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f14e4006fd0 tx=0x7f14e4008040 comp rx=0 tx=0).stop 2023-11-24T08:23:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.710+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 msgr2=0x7f14f406ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.710+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00c3c0 tx=0x7f14ec00c880 comp rx=0 tx=0).stop 2023-11-24T08:23:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 shutdown_connections 2023-11-24T08:23:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f14e006d390 0x7f14e006f850 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14f415dbd0 0x7f14f406ca10 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14f415d7f0 0x7f14f406c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 --2- 172.21.15.107:0/755048603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14f40c4220 0x7f14f4151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.711+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 >> 172.21.15.107:0/755048603 conn(0x7f14f40b0c50 msgr2=0x7f14f40afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.712+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 shutdown_connections 2023-11-24T08:23:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.712+0000 7f14fbf41640 1 -- 172.21.15.107:0/755048603 wait complete. 2023-11-24T08:23:50.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:23:50.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.929+0000 7fca22c09640 1 Processor -- start 2023-11-24T08:23:50.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.929+0000 7fca22c09640 1 -- start start 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca22c09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca1c0ccbd0 0x7fca1c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca22c09640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fca1c0ccfb0 0x7fca1c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca22c09640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1c1655f0 con 0x7fca1c162540 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca22c09640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c0cc380 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca22c09640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c162b30 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca209a5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca1c0ccbd0 0x7fca1c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca209a5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca1c0ccbd0 0x7fca1c1660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57000/0 (socket says 172.21.15.107:57000) 2023-11-24T08:23:50.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.931+0000 7fca209a5640 1 -- 172.21.15.107:0/1074545310 learned_addr learned my addr 172.21.15.107:0/1074545310 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.932+0000 7fca209a5640 1 -- 172.21.15.107:0/1074545310 >> v2:172.21.15.120:3300/0 conn(0x7fca1c0ccfb0 msgr2=0x7fca1c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.932+0000 7fca209a5640 1 --2- 172.21.15.107:0/1074545310 >> v2:172.21.15.120:3300/0 conn(0x7fca1c0ccfb0 0x7fca1c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.932+0000 7fca209a5640 1 -- 172.21.15.107:0/1074545310 >> v1:172.21.15.120:6789/0 conn(0x7fca1c162540 legacy=0x7fca1c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.932+0000 7fca209a5640 1 -- 172.21.15.107:0/1074545310 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c162e10 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.933+0000 7fca209a5640 1 --2- 172.21.15.107:0/1074545310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccbd0 0x7fca1c1660b0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fca1c06e700 tx=0x7fca0c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210dab630ca7d8d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.933+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1074545310 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c02ca80 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.933+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1074545310 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c02cbe0 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.933+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1074545310 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0343f0 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 -- 172.21.15.107:0/1074545310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccbd0 msgr2=0x7fca1c1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 --2- 172.21.15.107:0/1074545310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccbd0 0x7fca1c1660b0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fca1c06e700 tx=0x7fca0c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 -- 172.21.15.107:0/1074545310 shutdown_connections 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 --2- 172.21.15.107:0/1074545310 >> v2:172.21.15.120:3300/0 conn(0x7fca1c0ccfb0 0x7fca1c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 --2- 172.21.15.107:0/1074545310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccbd0 0x7fca1c1660b0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 -- 172.21.15.107:0/1074545310 >> 172.21.15.107:0/1074545310 conn(0x7fca1c0b0c50 msgr2=0x7fca1c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 -- 172.21.15.107:0/1074545310 shutdown_connections 2023-11-24T08:23:50.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.934+0000 7fca22c09640 1 -- 172.21.15.107:0/1074545310 wait complete. 2023-11-24T08:23:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.936+0000 7fca22c09640 1 Processor -- start 2023-11-24T08:23:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.936+0000 7fca22c09640 1 -- start start 2023-11-24T08:23:50.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca1c0ccbd0 0x7fca1c158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 0x7fca1c0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0727d0 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1601e0 con 0x7fca1c162540 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca22c09640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0b27b0 con 0x7fca1c0ccbd0 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57014/0 (socket says 172.21.15.107:57014) 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.937+0000 7fca1bfff640 1 -- 172.21.15.107:0/3001506227 learned_addr learned my addr 172.21.15.107:0/3001506227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca211a6640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 0x7fca1c0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 -- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca1c0ccbd0 msgr2=0x7fca1c158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:50.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca1c0ccbd0 0x7fca1c158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 -- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 msgr2=0x7fca1c0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 0x7fca1c0ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 -- 172.21.15.107:0/3001506227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c009b30 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca211a6640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 0x7fca1c0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca1bfff640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fca0400b410 tx=0x7fca0400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011830 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca040119b0 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011d60 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.938+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c0724f0 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.939+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c15fe00 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.940+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c15f980 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.941+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fca0401a400 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.942+0000 7fca197fa640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 0x7fc9fc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.942+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fca04088a00 con 0x7fca1c0ccfb0 2023-11-24T08:23:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.942+0000 7fca209a5640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 0x7fc9fc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.943+0000 7fca209a5640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 0x7fc9fc06f820 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fca0c035000 tx=0x7fca0c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:50.947+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca1c15f980 con 0x7fca1c0ccfb0 2023-11-24T08:23:51.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.170+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fca1c15f980 con 0x7fca1c0ccfb0 2023-11-24T08:23:51.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.711+0000 7fca197fa640 1 -- 172.21.15.107:0/3001506227 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 33 and data pool 34 v143) v1 ==== 174+0+0 (secure 0 0 0) 0x7fca1c15f980 con 0x7fca1c0ccfb0 2023-11-24T08:23:51.713 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 33 and data pool 34 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 msgr2=0x7fc9fc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 0x7fc9fc06f820 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fca0c035000 tx=0x7fca0c02e000 comp rx=0 tx=0).stop 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 msgr2=0x7fca1c071ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fca0400b410 tx=0x7fca0400b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 shutdown_connections 2023-11-24T08:23:51.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc9fc06d360 0x7fc9fc06f820 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca1c162540 0x7fca1c0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0ccfb0 0x7fca1c071ba0 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 --2- 172.21.15.107:0/3001506227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca1c0ccbd0 0x7fca1c158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.715+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 >> 172.21.15.107:0/3001506227 conn(0x7fca1c0b0c50 msgr2=0x7fca1c06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.716+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 shutdown_connections 2023-11-24T08:23:51.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.716+0000 7fca22c09640 1 -- 172.21.15.107:0/3001506227 wait complete. 2023-11-24T08:23:51.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.930+0000 7fd9478ec640 1 Processor -- start 2023-11-24T08:23:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.931+0000 7fd9478ec640 1 -- start start 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd9478ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd9400ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd9478ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9400cd470 0x7fd94015b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd9478ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94015ab80 con 0x7fd9400b2250 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd9478ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94015a990 con 0x7fd9400cd470 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd9478ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94015e3c0 con 0x7fd94015df80 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd945e89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9400cd470 0x7fd94015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd945e89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd9400cd470 0x7fd94015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57016/0 (socket says 172.21.15.107:57016) 2023-11-24T08:23:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.932+0000 7fd945e89640 1 -- 172.21.15.107:0/4061280828 learned_addr learned my addr 172.21.15.107:0/4061280828 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.933+0000 7fd945e89640 1 -- 172.21.15.107:0/4061280828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 msgr2=0x7fd9400ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.933+0000 7fd944e87640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd9400ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.933+0000 7fd945e89640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd9400ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.933+0000 7fd945e89640 1 -- 172.21.15.107:0/4061280828 >> v1:172.21.15.107:6789/0 conn(0x7fd9400b2250 legacy=0x7fd9400b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.933+0000 7fd945e89640 1 -- 172.21.15.107:0/4061280828 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9400b2d40 con 0x7fd9400cd470 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd945e89640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400cd470 0x7fd94015b080 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7fd93800d150 tx=0x7fd93802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4fdb46204ce3d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd937fff640 1 -- 172.21.15.107:0/4061280828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800b0d0 con 0x7fd9400cd470 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd937fff640 1 -- 172.21.15.107:0/4061280828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800b230 con 0x7fd9400cd470 2023-11-24T08:23:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd937fff640 1 -- 172.21.15.107:0/4061280828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938009600 con 0x7fd9400cd470 2023-11-24T08:23:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd9478ec640 1 -- 172.21.15.107:0/4061280828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400cd470 msgr2=0x7fd94015b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.934+0000 7fd9478ec640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400cd470 0x7fd94015b080 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7fd93800d150 tx=0x7fd93802e160 comp rx=0 tx=0).stop 2023-11-24T08:23:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 -- 172.21.15.107:0/4061280828 shutdown_connections 2023-11-24T08:23:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400cd470 0x7fd94015b080 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 --2- 172.21.15.107:0/4061280828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd9400ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 -- 172.21.15.107:0/4061280828 >> 172.21.15.107:0/4061280828 conn(0x7fd94006a920 msgr2=0x7fd94006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 -- 172.21.15.107:0/4061280828 shutdown_connections 2023-11-24T08:23:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.935+0000 7fd9478ec640 1 -- 172.21.15.107:0/4061280828 wait complete. 2023-11-24T08:23:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.936+0000 7fd9478ec640 1 Processor -- start 2023-11-24T08:23:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.936+0000 7fd9478ec640 1 -- start start 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9400cd470 0x7fd940150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd940152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9401537a0 con 0x7fd9400b2250 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940152b90 con 0x7fd9400cd470 2023-11-24T08:23:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.937+0000 7fd9478ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd940152e70 con 0x7fd94015df80 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57024/0 (socket says 172.21.15.107:57024) 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 -- 172.21.15.107:0/1069849418 learned_addr learned my addr 172.21.15.107:0/1069849418 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd944e87640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9400cd470 0x7fd940150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 -- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 msgr2=0x7fd940152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd940152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 -- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9400cd470 msgr2=0x7fd940150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9400cd470 0x7fd940150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 -- 172.21.15.107:0/1069849418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93800bd40 con 0x7fd9400b2250 2023-11-24T08:23:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.938+0000 7fd945688640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00c450 tx=0x7fd93c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.939+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0052d0 con 0x7fd9400b2250 2023-11-24T08:23:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.939+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c005430 con 0x7fd9400b2250 2023-11-24T08:23:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.939+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c012870 con 0x7fd9400b2250 2023-11-24T08:23:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.939+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd940153150 con 0x7fd9400b2250 2023-11-24T08:23:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.939+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd940150ab0 con 0x7fd9400b2250 2023-11-24T08:23:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.940+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9400b3c20 con 0x7fd9400b2250 2023-11-24T08:23:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.941+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fd93c0129d0 con 0x7fd9400b2250 2023-11-24T08:23:51.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.942+0000 7fd935ffb640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 0x7fd92406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:51.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.942+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd93c088370 con 0x7fd9400b2250 2023-11-24T08:23:51.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.942+0000 7fd944e87640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 0x7fd92406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:51.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.943+0000 7fd944e87640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 0x7fd92406f820 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd94009d310 tx=0x7fd930008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:51.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:51.947+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd93c054f20 con 0x7fd9400b2250 2023-11-24T08:23:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.128+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd940067470 con 0x7fd9400b2250 2023-11-24T08:23:52.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.131+0000 7fd935ffb640 1 -- 172.21.15.107:0/1069849418 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+16528 (secure 0 0 0) 0x7fd93c058540 con 0x7fd9400b2250 2023-11-24T08:23:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:52.134 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":233,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:51.172037+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 msgr2=0x7fd92406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 0x7fd92406f820 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd94009d310 tx=0x7fd930008040 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 msgr2=0x7fd940153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00c450 tx=0x7fd93c00c910 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 shutdown_connections 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd92406d360 0x7fd92406f820 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd94015df80 0x7fd940152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9400cd470 0x7fd940150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 --2- 172.21.15.107:0/1069849418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9400b2250 0x7fd940153c30 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 >> 172.21.15.107:0/1069849418 conn(0x7fd94006a920 msgr2=0x7fd9400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 shutdown_connections 2023-11-24T08:23:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.134+0000 7fd9478ec640 1 -- 172.21.15.107:0/1069849418 wait complete. 2023-11-24T08:23:52.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.366+0000 7fc46159e640 1 Processor -- start 2023-11-24T08:23:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.366+0000 7fc46159e640 1 -- start start 2023-11-24T08:23:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.367+0000 7fc46159e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc45c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.367+0000 7fc46159e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c06e060 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc46159e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c13f9e0 con 0x7fc45c13d120 2023-11-24T08:23:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc46159e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c13fcc0 con 0x7fc45c06d6c0 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc460d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc45c0cdbf0 0x7fc45c0ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44084/0 (socket says 172.21.15.107:44084) 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc460d9d640 1 -- 172.21.15.107:0/2366712614 learned_addr learned my addr 172.21.15.107:0/2366712614 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc453fff640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc45c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137890850 0 0) 0x7fc45c06e060 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.368+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc448003520 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345394403 0 0) 0x7fc448003520 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 msgr2=0x7fc45c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc45c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 >> v1:172.21.15.120:6789/0 conn(0x7fc45c13d120 legacy=0x7fc45c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c13ffa0 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc4540019f0 con 0x7fc45c0cdbf0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 >> v1:172.21.15.107:6789/0 conn(0x7fc45c0cdbf0 legacy=0x7fc45c0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc4480039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448003f20 0x7fc4480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc448004d00 0x7fc448005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c13fcc0 con 0x7fc448004d00 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44800bb10 con 0x7fc448003f20 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44800dee0 con 0x7fc45c06d6c0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.369+0000 7fc460d9d640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc448004d00 0x7fc448005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc460d9d640 1 -- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 msgr2=0x7fc4480039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc45bfff640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc4480039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc453fff640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448003f20 0x7fc4480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc460d9d640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc4480039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc460d9d640 1 -- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448003f20 msgr2=0x7fc4480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc460d9d640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448003f20 0x7fc4480047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc460d9d640 1 -- 172.21.15.107:0/2366712614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448003520 con 0x7fc448004d00 2023-11-24T08:23:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.370+0000 7fc45bfff640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c06d6c0 0x7fc4480039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc460d9d640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc448004d00 0x7fc448005180 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7fc45400a830 tx=0x7fc4480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a27f249e09cb76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4540072d0 con 0x7fc448004d00 2023-11-24T08:23:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45402f070 con 0x7fc448004d00 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc45b7fe640 1 -- 172.21.15.107:0/2366712614 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4540076b0 con 0x7fc448004d00 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc46159e640 1 -- 172.21.15.107:0/2366712614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc448004d00 msgr2=0x7fc448005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc46159e640 1 --2- 172.21.15.107:0/2366712614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc448004d00 0x7fc448005180 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7fc45400a830 tx=0x7fc4480057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.371+0000 7fc460d9d640 1 -- 172.21.15.107:0/2366712614 reap_dead start 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.372+0000 7fc46159e640 1 -- 172.21.15.107:0/2366712614 shutdown_connections 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.372+0000 7fc46159e640 1 -- 172.21.15.107:0/2366712614 >> 172.21.15.107:0/2366712614 conn(0x7fc45c0cb830 msgr2=0x7fc45c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.372+0000 7fc46159e640 1 -- 172.21.15.107:0/2366712614 shutdown_connections 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.372+0000 7fc46159e640 1 -- 172.21.15.107:0/2366712614 wait complete. 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.373+0000 7fc46159e640 1 Processor -- start 2023-11-24T08:23:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.373+0000 7fc46159e640 1 -- start start 2023-11-24T08:23:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448004d00 0x7fc45c138970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c15b8c0 0x7fc45c0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15cb00 con 0x7fc45c0af620 2023-11-24T08:23:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c138eb0 con 0x7fc448004d00 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.374+0000 7fc46159e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15be80 con 0x7fc45c15b8c0 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57048/0 (socket says 172.21.15.107:57048) 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 -- 172.21.15.107:0/3625953316 learned_addr learned my addr 172.21.15.107:0/3625953316 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 -- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c15b8c0 msgr2=0x7fc45c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c15b8c0 0x7fc45c0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 -- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448004d00 msgr2=0x7fc45c138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448004d00 0x7fc45c138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 -- 172.21.15.107:0/3625953316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c15a9d0 con 0x7fc45c0af620 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.375+0000 7fc460d9d640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7fc454001cb0 tx=0x7fc454001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.376+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454001250 con 0x7fc45c0af620 2023-11-24T08:23:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.376+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc454040480 con 0x7fc45c0af620 2023-11-24T08:23:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.376+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454040810 con 0x7fc45c0af620 2023-11-24T08:23:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.376+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c15c160 con 0x7fc45c0af620 2023-11-24T08:23:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.376+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c15c720 con 0x7fc45c0af620 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.377+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c04e840 con 0x7fc45c0af620 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.378+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fc454031030 con 0x7fc45c0af620 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.379+0000 7fc4597fa640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 0x7fc43c06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.379+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4540aa2a0 con 0x7fc45c0af620 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.379+0000 7fc45bfff640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 0x7fc43c06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.380+0000 7fc45bfff640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 0x7fc43c06f820 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0016d0 tx=0x7fc44c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.383+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc454076e50 con 0x7fc45c0af620 2023-11-24T08:23:52.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.558+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc45c0cd7d0 con 0x7fc45c0af620 2023-11-24T08:23:52.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.559+0000 7fc4597fa640 1 -- 172.21.15.107:0/3625953316 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+16528 (secure 0 0 0) 0x7fc45407a470 con 0x7fc45c0af620 2023-11-24T08:23:52.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:52.562 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":233,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:51.172037+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:52.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.561+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 msgr2=0x7fc43c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 0x7fc43c06f820 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0016d0 tx=0x7fc44c009040 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 msgr2=0x7fc45c0afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7fc454001cb0 tx=0x7fc454001d80 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 shutdown_connections 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43c06d360 0x7fc43c06f820 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45c0af620 0x7fc45c0afaa0 unknown :-1 s=CLOSED pgs=3179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc45c15b8c0 0x7fc45c0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 --2- 172.21.15.107:0/3625953316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc448004d00 0x7fc45c138970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 >> 172.21.15.107:0/3625953316 conn(0x7fc45c0cb830 msgr2=0x7fc45c0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 shutdown_connections 2023-11-24T08:23:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.562+0000 7fc46159e640 1 -- 172.21.15.107:0/3625953316 wait complete. 2023-11-24T08:23:52.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:23:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.795+0000 7fdea6248640 1 Processor -- start 2023-11-24T08:23:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.795+0000 7fdea6248640 1 -- start start 2023-11-24T08:23:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.797+0000 7fdea6248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.797+0000 7fdea6248640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdea00cd110 0x7fdea0070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.797+0000 7fdea6248640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea00cd510 con 0x7fdea0146eb0 2023-11-24T08:23:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.797+0000 7fdea6248640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea0161a10 con 0x7fdea00cd110 2023-11-24T08:23:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.797+0000 7fdea6248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea0161cf0 con 0x7fdea006e160 2023-11-24T08:23:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.798+0000 7fde9effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.798+0000 7fde9ffff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdea00cd110 0x7fdea0070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.798+0000 7fde9effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56060/0 (socket says 172.21.15.107:56060) 2023-11-24T08:23:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.798+0000 7fde9effd640 1 -- 172.21.15.107:0/935757958 learned_addr learned my addr 172.21.15.107:0/935757958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.798+0000 7fde9f7fe640 1 --1- 172.21.15.107:0/935757958 >> v1:172.21.15.120:6789/0 conn(0x7fdea0146eb0 0x7fdea0161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:38034/0 (socket says 172.21.15.107:38034) 2023-11-24T08:23:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.799+0000 7fde9effd640 1 -- 172.21.15.107:0/935757958 >> v2:172.21.15.120:3300/0 conn(0x7fdea00cd110 msgr2=0x7fdea0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.799+0000 7fde9effd640 1 --2- 172.21.15.107:0/935757958 >> v2:172.21.15.120:3300/0 conn(0x7fdea00cd110 0x7fdea0070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.799+0000 7fde9effd640 1 -- 172.21.15.107:0/935757958 >> v1:172.21.15.120:6789/0 conn(0x7fdea0146eb0 legacy=0x7fdea0161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.799+0000 7fde9effd640 1 -- 172.21.15.107:0/935757958 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0161fd0 con 0x7fdea006e160 2023-11-24T08:23:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.800+0000 7fde9effd640 1 --2- 172.21.15.107:0/935757958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde94009870 tx=0x7fde9402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2a5205783858d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.800+0000 7fde9e7fc640 1 -- 172.21.15.107:0/935757958 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94030840 con 0x7fdea006e160 2023-11-24T08:23:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.800+0000 7fde9e7fc640 1 -- 172.21.15.107:0/935757958 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde940309a0 con 0x7fdea006e160 2023-11-24T08:23:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.800+0000 7fde9e7fc640 1 -- 172.21.15.107:0/935757958 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94030d30 con 0x7fdea006e160 2023-11-24T08:23:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 -- 172.21.15.107:0/935757958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 msgr2=0x7fdea00ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 --2- 172.21.15.107:0/935757958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde94009870 tx=0x7fde9402b980 comp rx=0 tx=0).stop 2023-11-24T08:23:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 -- 172.21.15.107:0/935757958 shutdown_connections 2023-11-24T08:23:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 --2- 172.21.15.107:0/935757958 >> v2:172.21.15.120:3300/0 conn(0x7fdea00cd110 0x7fdea0070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 --2- 172.21.15.107:0/935757958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea006e160 0x7fdea00ccbd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 -- 172.21.15.107:0/935757958 >> 172.21.15.107:0/935757958 conn(0x7fdea00b0c50 msgr2=0x7fdea00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 -- 172.21.15.107:0/935757958 shutdown_connections 2023-11-24T08:23:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.801+0000 7fdea6248640 1 -- 172.21.15.107:0/935757958 wait complete. 2023-11-24T08:23:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.803+0000 7fdea6248640 1 Processor -- start 2023-11-24T08:23:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.803+0000 7fdea6248640 1 -- start start 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea006e160 0x7fdea00b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 0x7fdea00b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00b2ce0 con 0x7fdea0146eb0 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00b36e0 con 0x7fdea006e160 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.804+0000 7fdea6248640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00b39c0 con 0x7fdea00cd110 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57050/0 (socket says 172.21.15.107:57050) 2023-11-24T08:23:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 -- 172.21.15.107:0/1230786791 learned_addr learned my addr 172.21.15.107:0/1230786791 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9f7fe640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea006e160 0x7fdea00b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9effd640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 0x7fdea00b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 -- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 msgr2=0x7fdea00b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 0x7fdea00b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 -- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea006e160 msgr2=0x7fdea00b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea006e160 0x7fdea00b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 -- 172.21.15.107:0/1230786791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940094d0 con 0x7fdea0146eb0 2023-11-24T08:23:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9effd640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 0x7fdea00b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.805+0000 7fde9ffff640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7fde9800efc0 tx=0x7fde9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.806+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98014480 con 0x7fdea0146eb0 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.806+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9800f040 con 0x7fdea0146eb0 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.806+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98014820 con 0x7fdea0146eb0 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.806+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea00b3ca0 con 0x7fdea0146eb0 2023-11-24T08:23:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.806+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea00b0850 con 0x7fdea0146eb0 2023-11-24T08:23:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.807+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea00b0a40 con 0x7fdea0146eb0 2023-11-24T08:23:52.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.808+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fde98014980 con 0x7fdea0146eb0 2023-11-24T08:23:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.809+0000 7fde87fff640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 0x7fde8c06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.809+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde98089470 con 0x7fdea0146eb0 2023-11-24T08:23:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.809+0000 7fde9f7fe640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 0x7fde8c06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:52.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.810+0000 7fde9f7fe640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 0x7fde8c06f820 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fde90000f60 tx=0x7fde90007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:52.813+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdea00b0a40 con 0x7fdea0146eb0 2023-11-24T08:23:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.040+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdea00b41c0 con 0x7fdea0146eb0 2023-11-24T08:23:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.746+0000 7fde87fff640 1 -- 172.21.15.107:0/1230786791 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v146) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdea00b41c0 con 0x7fdea0146eb0 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 msgr2=0x7fde8c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 0x7fde8c06f820 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fde90000f60 tx=0x7fde90007ca0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 msgr2=0x7fdea015eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7fde9800efc0 tx=0x7fde9800c490 comp rx=0 tx=0).stop 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 shutdown_connections 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fde8c06d360 0x7fde8c06f820 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea0146eb0 0x7fdea015eec0 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdea00cd110 0x7fdea00b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 --2- 172.21.15.107:0/1230786791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdea006e160 0x7fdea00b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 >> 172.21.15.107:0/1230786791 conn(0x7fdea00b0c50 msgr2=0x7fdea0145080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:53.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 shutdown_connections 2023-11-24T08:23:53.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.750+0000 7fdea6248640 1 -- 172.21.15.107:0/1230786791 wait complete. 2023-11-24T08:23:53.774 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:23:53.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:23:53.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.990+0000 7f0ac32cf640 1 Processor -- start 2023-11-24T08:23:53.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.990+0000 7f0ac32cf640 1 -- start start 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac32cf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac32cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0abc0c57e0 0x7f0abc06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac32cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc15eea0 con 0x7f0abc15b440 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac32cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc15b820 con 0x7f0abc0c57e0 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac32cf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc0abf30 con 0x7f0abc0cc0f0 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac106b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0abc15b440 0x7f0abc0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44098/0 (socket says 172.21.15.107:44098) 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac106b640 1 -- 172.21.15.107:0/3923986805 learned_addr learned my addr 172.21.15.107:0/3923986805 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:53.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.992+0000 7f0ac186c640 1 --2- 172.21.15.107:0/3923986805 >> v2:172.21.15.107:3300/0 conn(0x7f0abc0c57e0 0x7f0abc06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:53.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.993+0000 7f0ac086a640 1 --2- 172.21.15.107:0/3923986805 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:53.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.993+0000 7f0ac186c640 1 -- 172.21.15.107:0/3923986805 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 msgr2=0x7f0abc0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:53.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.993+0000 7f0ac186c640 1 --2- 172.21.15.107:0/3923986805 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.993+0000 7f0ac186c640 1 -- 172.21.15.107:0/3923986805 >> v1:172.21.15.107:6789/0 conn(0x7f0abc15b440 legacy=0x7f0abc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:53.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.993+0000 7f0ac186c640 1 -- 172.21.15.107:0/3923986805 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc15f0f0 con 0x7f0abc0c57e0 2023-11-24T08:23:53.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.994+0000 7f0ac186c640 1 --2- 172.21.15.107:0/3923986805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc06c5f0 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f0abc06e720 tx=0x7f0ab402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bddf9c584fc9e9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:53.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.994+0000 7f0ab3fff640 1 -- 172.21.15.107:0/3923986805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab402cbe0 con 0x7f0abc0c57e0 2023-11-24T08:23:53.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.994+0000 7f0ab3fff640 1 -- 172.21.15.107:0/3923986805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab402cd40 con 0x7f0abc0c57e0 2023-11-24T08:23:53.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.994+0000 7f0ab3fff640 1 -- 172.21.15.107:0/3923986805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4034660 con 0x7f0abc0c57e0 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 -- 172.21.15.107:0/3923986805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 msgr2=0x7f0abc06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/3923986805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc06c5f0 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f0abc06e720 tx=0x7f0ab402be30 comp rx=0 tx=0).stop 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 -- 172.21.15.107:0/3923986805 shutdown_connections 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/3923986805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc06c5f0 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/3923986805 >> v2:172.21.15.120:3300/0 conn(0x7f0abc0cc0f0 0x7f0abc0c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:53.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 -- 172.21.15.107:0/3923986805 >> 172.21.15.107:0/3923986805 conn(0x7f0abc0b0c50 msgr2=0x7f0abc0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:53.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 -- 172.21.15.107:0/3923986805 shutdown_connections 2023-11-24T08:23:53.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.995+0000 7f0ac32cf640 1 -- 172.21.15.107:0/3923986805 wait complete. 2023-11-24T08:23:53.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.997+0000 7f0ac32cf640 1 Processor -- start 2023-11-24T08:23:53.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.997+0000 7f0ac32cf640 1 -- start start 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 0x7f0abc155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc15b440 0x7f0abc158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15ab80 con 0x7f0abc0c57e0 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc1595a0 con 0x7f0abc15b440 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac32cf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc159880 con 0x7f0abc0cc0f0 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57074/0 (socket says 172.21.15.107:57074) 2023-11-24T08:23:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 -- 172.21.15.107:0/1652328630 learned_addr learned my addr 172.21.15.107:0/1652328630 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac186c640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc15b440 0x7f0abc158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 -- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 msgr2=0x7f0abc155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac086a640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 0x7f0abc155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 0x7f0abc155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 -- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc15b440 msgr2=0x7f0abc158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc15b440 0x7f0abc158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.998+0000 7f0ac106b640 1 -- 172.21.15.107:0/1652328630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab4009a70 con 0x7f0abc0c57e0 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ac086a640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 0x7f0abc155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ac106b640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c480 tx=0x7f0ab800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8005ce0 con 0x7f0abc0c57e0 2023-11-24T08:23:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab8005e40 con 0x7f0abc0c57e0 2023-11-24T08:23:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80072f0 con 0x7f0abc0c57e0 2023-11-24T08:23:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc159b60 con 0x7f0abc0c57e0 2023-11-24T08:23:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:53.999+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc15a120 con 0x7f0abc0c57e0 2023-11-24T08:23:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.000+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc04e8b0 con 0x7f0abc0c57e0 2023-11-24T08:23:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.001+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f0ab8007510 con 0x7f0abc0c57e0 2023-11-24T08:23:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.002+0000 7f0ab1ffb640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 0x7f0aa006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.002+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ab8087d60 con 0x7f0abc0c57e0 2023-11-24T08:23:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.002+0000 7f0ac086a640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 0x7f0aa006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.003+0000 7f0ac086a640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 0x7f0aa006f820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0abc153680 tx=0x7f0aac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:54.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.006+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ab80549c0 con 0x7f0abc0c57e0 2023-11-24T08:23:54.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.234+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0abc15acc0 con 0x7f0abc0c57e0 2023-11-24T08:23:54.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.757+0000 7f0ab1ffb640 1 -- 172.21.15.107:0/1652328630 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v147) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ab8017070 con 0x7f0abc0c57e0 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.760+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 msgr2=0x7f0aa006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.760+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 0x7f0aa006f820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0abc153680 tx=0x7f0aac008040 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.760+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 msgr2=0x7f0abc15a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.760+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c480 tx=0x7f0ab800c940 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 shutdown_connections 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0aa006d360 0x7f0aa006f820 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0abc15b440 0x7f0abc158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0abc0cc0f0 0x7f0abc155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 --2- 172.21.15.107:0/1652328630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0abc0c57e0 0x7f0abc15a640 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:54.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 >> 172.21.15.107:0/1652328630 conn(0x7f0abc0b0c50 msgr2=0x7f0abc071390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:54.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 shutdown_connections 2023-11-24T08:23:54.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.761+0000 7f0ac32cf640 1 -- 172.21.15.107:0/1652328630 wait complete. 2023-11-24T08:23:54.784 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:23:54.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:54.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.996+0000 7fa7787c4640 1 Processor -- start 2023-11-24T08:23:54.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.996+0000 7fa7787c4640 1 -- start start 2023-11-24T08:23:54.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.997+0000 7fa7787c4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa77015c7f0 0x7fa77006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.997+0000 7fa7787c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:54.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa7787c4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700cece0 con 0x7fa7700cdfb0 2023-11-24T08:23:55.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa7787c4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa770156010 con 0x7fa77015c7f0 2023-11-24T08:23:55.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa7787c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77006ce30 con 0x7fa7700cf450 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa775d5f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa775d5f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56092/0 (socket says 172.21.15.107:56092) 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa775d5f640 1 -- 172.21.15.107:0/2329405492 learned_addr learned my addr 172.21.15.107:0/2329405492 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.998+0000 7fa776560640 1 --2- 172.21.15.107:0/2329405492 >> v2:172.21.15.120:3300/0 conn(0x7fa77015c7f0 0x7fa77006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.999+0000 7fa775d5f640 1 -- 172.21.15.107:0/2329405492 >> v2:172.21.15.120:3300/0 conn(0x7fa77015c7f0 msgr2=0x7fa77006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.999+0000 7fa775d5f640 1 --2- 172.21.15.107:0/2329405492 >> v2:172.21.15.120:3300/0 conn(0x7fa77015c7f0 0x7fa77006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.999+0000 7fa775d5f640 1 -- 172.21.15.107:0/2329405492 >> v1:172.21.15.120:6789/0 conn(0x7fa7700cdfb0 legacy=0x7fa7700ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:54.999+0000 7fa775d5f640 1 -- 172.21.15.107:0/2329405492 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77006d040 con 0x7fa7700cf450 2023-11-24T08:23:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.000+0000 7fa775d5f640 1 --2- 172.21.15.107:0/2329405492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa760009870 tx=0x7fa76002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31196ebeb506343a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa77555e640 1 -- 172.21.15.107:0/2329405492 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760030840 con 0x7fa7700cf450 2023-11-24T08:23:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa77555e640 1 -- 172.21.15.107:0/2329405492 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600309a0 con 0x7fa7700cf450 2023-11-24T08:23:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa77555e640 1 -- 172.21.15.107:0/2329405492 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760030d30 con 0x7fa7700cf450 2023-11-24T08:23:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 -- 172.21.15.107:0/2329405492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 msgr2=0x7fa7700cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 --2- 172.21.15.107:0/2329405492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa760009870 tx=0x7fa76002b980 comp rx=0 tx=0).stop 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 -- 172.21.15.107:0/2329405492 shutdown_connections 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 --2- 172.21.15.107:0/2329405492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7700cf450 0x7fa7700cf8d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 --2- 172.21.15.107:0/2329405492 >> v2:172.21.15.120:3300/0 conn(0x7fa77015c7f0 0x7fa77006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.001+0000 7fa7787c4640 1 -- 172.21.15.107:0/2329405492 >> 172.21.15.107:0/2329405492 conn(0x7fa77006a920 msgr2=0x7fa77006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.002+0000 7fa7787c4640 1 -- 172.21.15.107:0/2329405492 shutdown_connections 2023-11-24T08:23:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.002+0000 7fa7787c4640 1 -- 172.21.15.107:0/2329405492 wait complete. 2023-11-24T08:23:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.003+0000 7fa7787c4640 1 Processor -- start 2023-11-24T08:23:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.003+0000 7fa7787c4640 1 -- start start 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 0x7fa770147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 0x7fa77015f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701571a0 con 0x7fa7700cf450 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701f1560 con 0x7fa7700cdfb0 2023-11-24T08:23:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.004+0000 7fa7787c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701f1840 con 0x7fa77015c7f0 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57088/0 (socket says 172.21.15.107:57088) 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 -- 172.21.15.107:0/1986712810 learned_addr learned my addr 172.21.15.107:0/1986712810 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa776560640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 0x7fa770147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa776d61640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 0x7fa77015f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 -- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 msgr2=0x7fa77015f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 0x7fa77015f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 -- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 msgr2=0x7fa770147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 0x7fa770147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa775d5f640 1 -- 172.21.15.107:0/1986712810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600094d0 con 0x7fa7700cf450 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa776560640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 0x7fa770147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.005+0000 7fa776d61640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 0x7fa77015f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa775d5f640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fa760000c00 tx=0x7fa760004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760039c90 con 0x7fa7700cf450 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760039df0 con 0x7fa7700cf450 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7701f1b20 con 0x7fa7700cf450 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760043640 con 0x7fa7700cf450 2023-11-24T08:23:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.006+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa770147ac0 con 0x7fa7700cf450 2023-11-24T08:23:55.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.007+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa770147c50 con 0x7fa7700cf450 2023-11-24T08:23:55.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.008+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fa7600437a0 con 0x7fa7700cf450 2023-11-24T08:23:55.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.009+0000 7fa766ffd640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 0x7fa75406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.009+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa76003e070 con 0x7fa7700cf450 2023-11-24T08:23:55.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.009+0000 7fa776560640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 0x7fa75406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.010+0000 7fa776560640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 0x7fa75406f820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009a20 tx=0x7fa76c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.014+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa760078410 con 0x7fa7700cf450 2023-11-24T08:23:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.242+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77006e4a0 con 0x7fa7700cf450 2023-11-24T08:23:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.244+0000 7fa766ffd640 1 -- 172.21.15.107:0/1986712810 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fa76007ba30 con 0x7fa7700cf450 2023-11-24T08:23:55.246 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:55.246 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":147,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:23:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 147 2023-11-24T08:23:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 msgr2=0x7fa75406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 0x7fa75406f820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009a20 tx=0x7fa76c008040 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 msgr2=0x7fa77015f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fa760000c00 tx=0x7fa760004730 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 shutdown_connections 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa75406d360 0x7fa75406f820 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa77015c7f0 0x7fa77015f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7700cf450 0x7fa77015f3d0 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 --2- 172.21.15.107:0/1986712810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7700cdfb0 0x7fa770147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.246+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 >> 172.21.15.107:0/1986712810 conn(0x7fa77006a920 msgr2=0x7fa7700ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.247+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 shutdown_connections 2023-11-24T08:23:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.247+0000 7fa7787c4640 1 -- 172.21.15.107:0/1986712810 wait complete. 2023-11-24T08:23:55.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:23:55.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.474+0000 7fadc211d640 1 Processor -- start 2023-11-24T08:23:55.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.474+0000 7fadc211d640 1 -- start start 2023-11-24T08:23:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.475+0000 7fadc211d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 0x7fadbc0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.475+0000 7fadc211d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.475+0000 7fadc211d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc0cece0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.475+0000 7fadc211d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadbc0cfe40 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.475+0000 7fadc211d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadbc156010 con 0x7fadbc0cf450 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.476+0000 7fadbbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.476+0000 7fadbbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57100/0 (socket says 172.21.15.107:57100) 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.476+0000 7fadbbfff640 1 -- 172.21.15.107:0/996803110 learned_addr learned my addr 172.21.15.107:0/996803110 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.476+0000 7fadb3fff640 1 --2- 172.21.15.107:0/996803110 >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 0x7fadbc0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.477+0000 7fadbbfff640 1 -- 172.21.15.107:0/996803110 >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 msgr2=0x7fadbc0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.477+0000 7fadbbfff640 1 --2- 172.21.15.107:0/996803110 >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 0x7fadbc0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.477+0000 7fadbbfff640 1 -- 172.21.15.107:0/996803110 >> v1:172.21.15.120:6789/0 conn(0x7fadbc15c7f0 legacy=0x7fadbc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.477+0000 7fadbbfff640 1 -- 172.21.15.107:0/996803110 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc06d040 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.477+0000 7fadb3fff640 1 --2- 172.21.15.107:0/996803110 >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 0x7fadbc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:23:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadbbfff640 1 --2- 172.21.15.107:0/996803110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fadbc070410 tx=0x7fadb402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1755abd515b20428 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadbaffd640 1 -- 172.21.15.107:0/996803110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb402cb20 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadbaffd640 1 -- 172.21.15.107:0/996803110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb402cc80 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadbaffd640 1 -- 172.21.15.107:0/996803110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb40343f0 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadc211d640 1 -- 172.21.15.107:0/996803110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc0cdfb0 msgr2=0x7fadbc0ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.478+0000 7fadc211d640 1 --2- 172.21.15.107:0/996803110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fadbc070410 tx=0x7fadb402bf80 comp rx=0 tx=0).stop 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 -- 172.21.15.107:0/996803110 shutdown_connections 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 --2- 172.21.15.107:0/996803110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc0cdfb0 0x7fadbc0ce3e0 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 --2- 172.21.15.107:0/996803110 >> v2:172.21.15.120:3300/0 conn(0x7fadbc0cf450 0x7fadbc0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 -- 172.21.15.107:0/996803110 >> 172.21.15.107:0/996803110 conn(0x7fadbc06a920 msgr2=0x7fadbc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 -- 172.21.15.107:0/996803110 shutdown_connections 2023-11-24T08:23:55.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.479+0000 7fadc211d640 1 -- 172.21.15.107:0/996803110 wait complete. 2023-11-24T08:23:55.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.480+0000 7fadc211d640 1 Processor -- start 2023-11-24T08:23:55.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.481+0000 7fadc211d640 1 -- start start 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.481+0000 7fadc211d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadbc0cdfb0 0x7fadbc146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadc211d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadbc0cf450 0x7fadbc0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadc211d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadc211d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0ac360 con 0x7fadbc15c7f0 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadc211d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0ac640 con 0x7fadbc0cf450 2023-11-24T08:23:55.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadc211d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0ac920 con 0x7fadbc0cdfb0 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57114/0 (socket says 172.21.15.107:57114) 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 -- 172.21.15.107:0/1112582585 learned_addr learned my addr 172.21.15.107:0/1112582585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 -- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadbc0cdfb0 msgr2=0x7fadbc146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadbc0cdfb0 0x7fadbc146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 -- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadbc0cf450 msgr2=0x7fadbc0ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadb3fff640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadbc0cf450 0x7fadbc0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadbc0cf450 0x7fadbc0ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.482+0000 7fadbbfff640 1 -- 172.21.15.107:0/1112582585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4009b30 con 0x7fadbc15c7f0 2023-11-24T08:23:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadbbfff640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fadb4036040 tx=0x7fadb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4009510 con 0x7fadbc15c7f0 2023-11-24T08:23:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb402c920 con 0x7fadbc15c7f0 2023-11-24T08:23:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb403e690 con 0x7fadbc15c7f0 2023-11-24T08:23:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc0acc00 con 0x7fadbc15c7f0 2023-11-24T08:23:55.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.483+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc1467c0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.484+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadbc067470 con 0x7fadbc15c7f0 2023-11-24T08:23:55.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.485+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fadb403e7f0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.486+0000 7fadb8ff9640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 0x7fad9c06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.486+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb40ab3c0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.487+0000 7fadbb7fe640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 0x7fad9c06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.487+0000 7fadbb7fe640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 0x7fad9c06f850 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.491+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadb4077ed0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.682+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadbc0cd730 con 0x7fadbc15c7f0 2023-11-24T08:23:55.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.684+0000 7fadb8ff9640 1 -- 172.21.15.107:0/1112582585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+16515 (secure 0 0 0) 0x7fadb407b4f0 con 0x7fadbc15c7f0 2023-11-24T08:23:55.686 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:55.686 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":234,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:23:52.724493+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.686+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 msgr2=0x7fad9c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 0x7fad9c06f850 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac0079b0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 msgr2=0x7fadbc15eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fadb4036040 tx=0x7fadb4004a40 comp rx=0 tx=0).stop 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 shutdown_connections 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad9c06d390 0x7fad9c06f850 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadbc15c7f0 0x7fadbc15eec0 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fadbc0cf450 0x7fadbc0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 --2- 172.21.15.107:0/1112582585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fadbc0cdfb0 0x7fadbc146280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 >> 172.21.15.107:0/1112582585 conn(0x7fadbc06a920 msgr2=0x7fadbc1554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:55.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 shutdown_connections 2023-11-24T08:23:55.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.687+0000 7fadc211d640 1 -- 172.21.15.107:0/1112582585 wait complete. 2023-11-24T08:23:55.708 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:23:55.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:23:55.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.919+0000 7fe3d1fc9640 1 Processor -- start 2023-11-24T08:23:55.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.919+0000 7fe3d1fc9640 1 -- start start 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d1fc9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3cc15c410 0x7fe3cc15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d1fc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d1fc9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc15bf80 con 0x7fe3cc144b30 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d1fc9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0b27b0 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d1fc9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0b1560 con 0x7fe3cc15c410 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57130/0 (socket says 172.21.15.107:57130) 2023-11-24T08:23:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 -- 172.21.15.107:0/1591622634 learned_addr learned my addr 172.21.15.107:0/1591622634 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 -- 172.21.15.107:0/1591622634 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc15c410 msgr2=0x7fe3cc15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 --2- 172.21.15.107:0/1591622634 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc15c410 0x7fe3cc15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 -- 172.21.15.107:0/1591622634 >> v1:172.21.15.120:6789/0 conn(0x7fe3cc144b30 legacy=0x7fe3cc0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.921+0000 7fe3d17c8640 1 -- 172.21.15.107:0/1591622634 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc0b1280 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3d17c8640 1 --2- 172.21.15.107:0/1591622634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0cc7a0 tx=0x7fe3c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d41b6cc55b17b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3cb7fe640 1 -- 172.21.15.107:0/1591622634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c402ca80 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3cb7fe640 1 -- 172.21.15.107:0/1591622634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c402cbe0 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3cb7fe640 1 -- 172.21.15.107:0/1591622634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40343f0 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/1591622634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0b1de0 msgr2=0x7fe3cc0b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.923+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/1591622634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0cc7a0 tx=0x7fe3c402bee0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/1591622634 shutdown_connections 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/1591622634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc0b2270 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/1591622634 >> v2:172.21.15.120:3300/0 conn(0x7fe3cc15c410 0x7fe3cc15f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/1591622634 >> 172.21.15.107:0/1591622634 conn(0x7fe3cc0cb830 msgr2=0x7fe3cc0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/1591622634 shutdown_connections 2023-11-24T08:23:55.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.924+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/1591622634 wait complete. 2023-11-24T08:23:55.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.925+0000 7fe3d1fc9640 1 Processor -- start 2023-11-24T08:23:55.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.926+0000 7fe3d1fc9640 1 -- start start 2023-11-24T08:23:55.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc1f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 0x7fe3cc0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0a9350 con 0x7fe3cc144b30 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0a8430 con 0x7fe3cc0b1de0 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d1fc9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0a8680 con 0x7fe3cc15c410 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57142/0 (socket says 172.21.15.107:57142) 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 -- 172.21.15.107:0/258897209 learned_addr learned my addr 172.21.15.107:0/258897209 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d0fc7640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc1f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d17c8640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 0x7fe3cc0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 -- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 msgr2=0x7fe3cc0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 0x7fe3cc0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 -- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 msgr2=0x7fe3cc1f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:55.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc1f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3cbfff640 1 -- 172.21.15.107:0/258897209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4009b30 con 0x7fe3cc144b30 2023-11-24T08:23:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d0fc7640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc1f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.927+0000 7fe3d17c8640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 0x7fe3cc0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3cbfff640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b440 tx=0x7fe3b800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8011890 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80119f0 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8011da0 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc0a8960 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.928+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc1f3630 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.929+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc1f3850 con 0x7fe3cc144b30 2023-11-24T08:23:55.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.930+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fe3b801a400 con 0x7fe3cc144b30 2023-11-24T08:23:55.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.931+0000 7fe3c97fa640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 0x7fe3b006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:55.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.931+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b8088bd0 con 0x7fe3cc144b30 2023-11-24T08:23:55.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.932+0000 7fe3d0fc7640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 0x7fe3b006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.932+0000 7fe3d0fc7640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 0x7fe3b006f820 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0007fd0 tx=0x7fe3c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:55.935+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3b8055720 con 0x7fe3cc144b30 2023-11-24T08:23:56.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.122+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3cc15bc10 con 0x7fe3b006d360 2023-11-24T08:23:56.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.145+0000 7fe3c97fa640 1 -- 172.21.15.107:0/258897209 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287785 (secure 0 0 0) 0x7fe3c0001370 con 0x7fe3b006d360 2023-11-24T08:23:56.150 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:56.152 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:23:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 msgr2=0x7fe3b006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 0x7fe3b006f820 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0007fd0 tx=0x7fe3c0009040 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 msgr2=0x7fe3cc0a8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b440 tx=0x7fe3b800b900 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 shutdown_connections 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe3b006d360 0x7fe3b006f820 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3cc15c410 0x7fe3cc0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3cc144b30 0x7fe3cc0a8e10 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 --2- 172.21.15.107:0/258897209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3cc0b1de0 0x7fe3cc1f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 >> 172.21.15.107:0/258897209 conn(0x7fe3cc0cb830 msgr2=0x7fe3cc0cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 shutdown_connections 2023-11-24T08:23:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.150+0000 7fe3d1fc9640 1 -- 172.21.15.107:0/258897209 wait complete. 2023-11-24T08:23:56.170 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2023-11-24T08:23:55.794949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336168,"kb_used_data":24300,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754638552,"statfs":{"total":773094113280,"available":772749877248,"internally_reserved":0,"allocated":24883200,"data_stored":11696216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":319213118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001541"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743778+0000","last_change":"2023-11-24T08:23:51.760223+0000","last_active":"2023-11-24T08:23:52.743778+0000","last_peered":"2023-11-24T08:23:52.743778+0000","last_clean":"2023-11-24T08:23:52.743778+0000","last_became_active":"2023-11-24T08:23:51.759936+0000","last_became_peered":"2023-11-24T08:23:51.759936+0000","last_unstale":"2023-11-24T08:23:52.743778+0000","last_undegraded":"2023-11-24T08:23:52.743778+0000","last_fullsized":"2023-11-24T08:23:52.743778+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:15.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436533+0000","last_change":"2023-11-24T08:23:51.748473+0000","last_active":"2023-11-24T08:23:53.436533+0000","last_peered":"2023-11-24T08:23:53.436533+0000","last_clean":"2023-11-24T08:23:53.436533+0000","last_became_active":"2023-11-24T08:23:51.747482+0000","last_became_peered":"2023-11-24T08:23:51.747482+0000","last_unstale":"2023-11-24T08:23:53.436533+0000","last_undegraded":"2023-11-24T08:23:53.436533+0000","last_fullsized":"2023-11-24T08:23:53.436533+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:00:42.129811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740801+0000","last_change":"2023-11-24T08:23:51.741946+0000","last_active":"2023-11-24T08:23:52.740801+0000","last_peered":"2023-11-24T08:23:52.740801+0000","last_clean":"2023-11-24T08:23:52.740801+0000","last_became_active":"2023-11-24T08:23:51.741639+0000","last_became_peered":"2023-11-24T08:23:51.741639+0000","last_unstale":"2023-11-24T08:23:52.740801+0000","last_undegraded":"2023-11-24T08:23:52.740801+0000","last_fullsized":"2023-11-24T08:23:52.740801+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:08.236700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453254+0000","last_change":"2023-11-24T08:23:51.743159+0000","last_active":"2023-11-24T08:23:53.453254+0000","last_peered":"2023-11-24T08:23:53.453254+0000","last_clean":"2023-11-24T08:23:53.453254+0000","last_became_active":"2023-11-24T08:23:51.742877+0000","last_became_peered":"2023-11-24T08:23:51.742877+0000","last_unstale":"2023-11-24T08:23:53.453254+0000","last_undegraded":"2023-11-24T08:23:53.453254+0000","last_fullsized":"2023-11-24T08:23:53.453254+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:15:23.012137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450767+0000","last_change":"2023-11-24T08:23:51.746320+0000","last_active":"2023-11-24T08:23:53.450767+0000","last_peered":"2023-11-24T08:23:53.450767+0000","last_clean":"2023-11-24T08:23:53.450767+0000","last_became_active":"2023-11-24T08:23:51.745895+0000","last_became_peered":"2023-11-24T08:23:51.745895+0000","last_unstale":"2023-11-24T08:23:53.450767+0000","last_undegraded":"2023-11-24T08:23:53.450767+0000","last_fullsized":"2023-11-24T08:23:53.450767+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:06.385572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451925+0000","last_change":"2023-11-24T08:23:51.749477+0000","last_active":"2023-11-24T08:23:53.451925+0000","last_peered":"2023-11-24T08:23:53.451925+0000","last_clean":"2023-11-24T08:23:53.451925+0000","last_became_active":"2023-11-24T08:23:51.749187+0000","last_became_peered":"2023-11-24T08:23:51.749187+0000","last_unstale":"2023-11-24T08:23:53.451925+0000","last_undegraded":"2023-11-24T08:23:53.451925+0000","last_fullsized":"2023-11-24T08:23:53.451925+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:03.867689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.739516+0000","last_change":"2023-11-24T08:23:51.743897+0000","last_active":"2023-11-24T08:23:52.739516+0000","last_peered":"2023-11-24T08:23:52.739516+0000","last_clean":"2023-11-24T08:23:52.739516+0000","last_became_active":"2023-11-24T08:23:51.743623+0000","last_became_peered":"2023-11-24T08:23:51.743623+0000","last_unstale":"2023-11-24T08:23:52.739516+0000","last_undegraded":"2023-11-24T08:23:52.739516+0000","last_fullsized":"2023-11-24T08:23:52.739516+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:37.880842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452373+0000","last_change":"2023-11-24T08:23:51.734684+0000","last_active":"2023-11-24T08:23:53.452373+0000","last_peered":"2023-11-24T08:23:53.452373+0000","last_clean":"2023-11-24T08:23:53.452373+0000","last_became_active":"2023-11-24T08:23:51.734394+0000","last_became_peered":"2023-11-24T08:23:51.734394+0000","last_unstale":"2023-11-24T08:23:53.452373+0000","last_undegraded":"2023-11-24T08:23:53.452373+0000","last_fullsized":"2023-11-24T08:23:53.452373+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:14.968654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452595+0000","last_change":"2023-11-24T08:23:51.751526+0000","last_active":"2023-11-24T08:23:53.452595+0000","last_peered":"2023-11-24T08:23:53.452595+0000","last_clean":"2023-11-24T08:23:53.452595+0000","last_became_active":"2023-11-24T08:23:51.751240+0000","last_became_peered":"2023-11-24T08:23:51.751240+0000","last_unstale":"2023-11-24T08:23:53.452595+0000","last_undegraded":"2023-11-24T08:23:53.452595+0000","last_fullsized":"2023-11-24T08:23:53.452595+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:15:24.535237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950850+0000","last_change":"2023-11-24T08:23:51.739001+0000","last_active":"2023-11-24T08:23:52.950850+0000","last_peered":"2023-11-24T08:23:52.950850+0000","last_clean":"2023-11-24T08:23:52.950850+0000","last_became_active":"2023-11-24T08:23:51.738734+0000","last_became_peered":"2023-11-24T08:23:51.738734+0000","last_unstale":"2023-11-24T08:23:52.950850+0000","last_undegraded":"2023-11-24T08:23:52.950850+0000","last_fullsized":"2023-11-24T08:23:52.950850+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:44.975136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743558+0000","last_change":"2023-11-24T08:23:51.747955+0000","last_active":"2023-11-24T08:23:52.743558+0000","last_peered":"2023-11-24T08:23:52.743558+0000","last_clean":"2023-11-24T08:23:52.743558+0000","last_became_active":"2023-11-24T08:23:51.747685+0000","last_became_peered":"2023-11-24T08:23:51.747685+0000","last_unstale":"2023-11-24T08:23:52.743558+0000","last_undegraded":"2023-11-24T08:23:52.743558+0000","last_fullsized":"2023-11-24T08:23:52.743558+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:16.740605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452650+0000","last_change":"2023-11-24T08:23:51.752043+0000","last_active":"2023-11-24T08:23:53.452650+0000","last_peered":"2023-11-24T08:23:53.452650+0000","last_clean":"2023-11-24T08:23:53.452650+0000","last_became_active":"2023-11-24T08:23:51.750768+0000","last_became_peered":"2023-11-24T08:23:51.750768+0000","last_unstale":"2023-11-24T08:23:53.452650+0000","last_undegraded":"2023-11-24T08:23:53.452650+0000","last_fullsized":"2023-11-24T08:23:53.452650+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:07:59.485127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453276+0000","last_change":"2023-11-24T08:23:51.745029+0000","last_active":"2023-11-24T08:23:53.453276+0000","last_peered":"2023-11-24T08:23:53.453276+0000","last_clean":"2023-11-24T08:23:53.453276+0000","last_became_active":"2023-11-24T08:23:51.744034+0000","last_became_peered":"2023-11-24T08:23:51.744034+0000","last_unstale":"2023-11-24T08:23:53.453276+0000","last_undegraded":"2023-11-24T08:23:53.453276+0000","last_fullsized":"2023-11-24T08:23:53.453276+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:33.706450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435745+0000","last_change":"2023-11-24T08:23:51.752793+0000","last_active":"2023-11-24T08:23:53.435745+0000","last_peered":"2023-11-24T08:23:53.435745+0000","last_clean":"2023-11-24T08:23:53.435745+0000","last_became_active":"2023-11-24T08:23:51.752451+0000","last_became_peered":"2023-11-24T08:23:51.752451+0000","last_unstale":"2023-11-24T08:23:53.435745+0000","last_undegraded":"2023-11-24T08:23:53.435745+0000","last_fullsized":"2023-11-24T08:23:53.435745+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:38.009798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435311+0000","last_change":"2023-11-24T08:23:51.751343+0000","last_active":"2023-11-24T08:23:53.435311+0000","last_peered":"2023-11-24T08:23:53.435311+0000","last_clean":"2023-11-24T08:23:53.435311+0000","last_became_active":"2023-11-24T08:23:51.751028+0000","last_became_peered":"2023-11-24T08:23:51.751028+0000","last_unstale":"2023-11-24T08:23:53.435311+0000","last_undegraded":"2023-11-24T08:23:53.435311+0000","last_fullsized":"2023-11-24T08:23:53.435311+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:08.960033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950380+0000","last_change":"2023-11-24T08:23:51.738190+0000","last_active":"2023-11-24T08:23:52.950380+0000","last_peered":"2023-11-24T08:23:52.950380+0000","last_clean":"2023-11-24T08:23:52.950380+0000","last_became_active":"2023-11-24T08:23:51.737832+0000","last_became_peered":"2023-11-24T08:23:51.737832+0000","last_unstale":"2023-11-24T08:23:52.950380+0000","last_undegraded":"2023-11-24T08:23:52.950380+0000","last_fullsized":"2023-11-24T08:23:52.950380+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:57:16.681306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950988+0000","last_change":"2023-11-24T08:23:51.746317+0000","last_active":"2023-11-24T08:23:52.950988+0000","last_peered":"2023-11-24T08:23:52.950988+0000","last_clean":"2023-11-24T08:23:52.950988+0000","last_became_active":"2023-11-24T08:23:51.745703+0000","last_became_peered":"2023-11-24T08:23:51.745703+0000","last_unstale":"2023-11-24T08:23:52.950988+0000","last_undegraded":"2023-11-24T08:23:52.950988+0000","last_fullsized":"2023-11-24T08:23:52.950988+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:59.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169836+0000","last_change":"2023-11-24T08:23:51.740685+0000","last_active":"2023-11-24T08:23:53.169836+0000","last_peered":"2023-11-24T08:23:53.169836+0000","last_clean":"2023-11-24T08:23:53.169836+0000","last_became_active":"2023-11-24T08:23:51.740309+0000","last_became_peered":"2023-11-24T08:23:51.740309+0000","last_unstale":"2023-11-24T08:23:53.169836+0000","last_undegraded":"2023-11-24T08:23:53.169836+0000","last_fullsized":"2023-11-24T08:23:53.169836+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:44.375189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950995+0000","last_change":"2023-11-24T08:23:51.748036+0000","last_active":"2023-11-24T08:23:52.950995+0000","last_peered":"2023-11-24T08:23:52.950995+0000","last_clean":"2023-11-24T08:23:52.950995+0000","last_became_active":"2023-11-24T08:23:51.747544+0000","last_became_peered":"2023-11-24T08:23:51.747544+0000","last_unstale":"2023-11-24T08:23:52.950995+0000","last_undegraded":"2023-11-24T08:23:52.950995+0000","last_fullsized":"2023-11-24T08:23:52.950995+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:52.021241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452853+0000","last_change":"2023-11-24T08:23:51.749777+0000","last_active":"2023-11-24T08:23:53.452853+0000","last_peered":"2023-11-24T08:23:53.452853+0000","last_clean":"2023-11-24T08:23:53.452853+0000","last_became_active":"2023-11-24T08:23:51.749495+0000","last_became_peered":"2023-11-24T08:23:51.749495+0000","last_unstale":"2023-11-24T08:23:53.452853+0000","last_undegraded":"2023-11-24T08:23:53.452853+0000","last_fullsized":"2023-11-24T08:23:53.452853+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:49.785657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450841+0000","last_change":"2023-11-24T08:23:51.738083+0000","last_active":"2023-11-24T08:23:53.450841+0000","last_peered":"2023-11-24T08:23:53.450841+0000","last_clean":"2023-11-24T08:23:53.450841+0000","last_became_active":"2023-11-24T08:23:51.737807+0000","last_became_peered":"2023-11-24T08:23:51.737807+0000","last_unstale":"2023-11-24T08:23:53.450841+0000","last_undegraded":"2023-11-24T08:23:53.450841+0000","last_fullsized":"2023-11-24T08:23:53.450841+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:01:09.972139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743015+0000","last_change":"2023-11-24T08:23:51.759704+0000","last_active":"2023-11-24T08:23:52.743015+0000","last_peered":"2023-11-24T08:23:52.743015+0000","last_clean":"2023-11-24T08:23:52.743015+0000","last_became_active":"2023-11-24T08:23:51.759271+0000","last_became_peered":"2023-11-24T08:23:51.759271+0000","last_unstale":"2023-11-24T08:23:52.743015+0000","last_undegraded":"2023-11-24T08:23:52.743015+0000","last_fullsized":"2023-11-24T08:23:52.743015+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:05:39.390596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435525+0000","last_change":"2023-11-24T08:23:51.750543+0000","last_active":"2023-11-24T08:23:53.435525+0000","last_peered":"2023-11-24T08:23:53.435525+0000","last_clean":"2023-11-24T08:23:53.435525+0000","last_became_active":"2023-11-24T08:23:51.747525+0000","last_became_peered":"2023-11-24T08:23:51.747525+0000","last_unstale":"2023-11-24T08:23:53.435525+0000","last_undegraded":"2023-11-24T08:23:53.435525+0000","last_fullsized":"2023-11-24T08:23:53.435525+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:46:04.853199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.741955+0000","last_change":"2023-11-24T08:23:51.751055+0000","last_active":"2023-11-24T08:23:52.741955+0000","last_peered":"2023-11-24T08:23:52.741955+0000","last_clean":"2023-11-24T08:23:52.741955+0000","last_became_active":"2023-11-24T08:23:51.749957+0000","last_became_peered":"2023-11-24T08:23:51.749957+0000","last_unstale":"2023-11-24T08:23:52.741955+0000","last_undegraded":"2023-11-24T08:23:52.741955+0000","last_fullsized":"2023-11-24T08:23:52.741955+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:38.470361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740982+0000","last_change":"2023-11-24T08:23:51.747921+0000","last_active":"2023-11-24T08:23:52.740982+0000","last_peered":"2023-11-24T08:23:52.740982+0000","last_clean":"2023-11-24T08:23:52.740982+0000","last_became_active":"2023-11-24T08:23:51.747568+0000","last_became_peered":"2023-11-24T08:23:51.747568+0000","last_unstale":"2023-11-24T08:23:52.740982+0000","last_undegraded":"2023-11-24T08:23:52.740982+0000","last_fullsized":"2023-11-24T08:23:52.740982+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:15.786481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436579+0000","last_change":"2023-11-24T08:23:51.749672+0000","last_active":"2023-11-24T08:23:53.436579+0000","last_peered":"2023-11-24T08:23:53.436579+0000","last_clean":"2023-11-24T08:23:53.436579+0000","last_became_active":"2023-11-24T08:23:51.749404+0000","last_became_peered":"2023-11-24T08:23:51.749404+0000","last_unstale":"2023-11-24T08:23:53.436579+0000","last_undegraded":"2023-11-24T08:23:53.436579+0000","last_fullsized":"2023-11-24T08:23:53.436579+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:38:15.676270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743689+0000","last_change":"2023-11-24T08:23:51.754562+0000","last_active":"2023-11-24T08:23:52.743689+0000","last_peered":"2023-11-24T08:23:52.743689+0000","last_clean":"2023-11-24T08:23:52.743689+0000","last_became_active":"2023-11-24T08:23:51.753797+0000","last_became_peered":"2023-11-24T08:23:51.753797+0000","last_unstale":"2023-11-24T08:23:52.743689+0000","last_undegraded":"2023-11-24T08:23:52.743689+0000","last_fullsized":"2023-11-24T08:23:52.743689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:31.064463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742996+0000","last_change":"2023-11-24T08:23:51.760228+0000","last_active":"2023-11-24T08:23:52.742996+0000","last_peered":"2023-11-24T08:23:52.742996+0000","last_clean":"2023-11-24T08:23:52.742996+0000","last_became_active":"2023-11-24T08:23:51.759891+0000","last_became_peered":"2023-11-24T08:23:51.759891+0000","last_unstale":"2023-11-24T08:23:52.742996+0000","last_undegraded":"2023-11-24T08:23:52.742996+0000","last_fullsized":"2023-11-24T08:23:52.742996+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:10:49.559926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452016+0000","last_change":"2023-11-24T08:23:51.750778+0000","last_active":"2023-11-24T08:23:53.452016+0000","last_peered":"2023-11-24T08:23:53.452016+0000","last_clean":"2023-11-24T08:23:53.452016+0000","last_became_active":"2023-11-24T08:23:51.750497+0000","last_became_peered":"2023-11-24T08:23:51.750497+0000","last_unstale":"2023-11-24T08:23:53.452016+0000","last_undegraded":"2023-11-24T08:23:53.452016+0000","last_fullsized":"2023-11-24T08:23:53.452016+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:59.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435832+0000","last_change":"2023-11-24T08:23:51.750754+0000","last_active":"2023-11-24T08:23:53.435832+0000","last_peered":"2023-11-24T08:23:53.435832+0000","last_clean":"2023-11-24T08:23:53.435832+0000","last_became_active":"2023-11-24T08:23:51.747916+0000","last_became_peered":"2023-11-24T08:23:51.747916+0000","last_unstale":"2023-11-24T08:23:53.435832+0000","last_undegraded":"2023-11-24T08:23:53.435832+0000","last_fullsized":"2023-11-24T08:23:53.435832+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:13.601471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435246+0000","last_change":"2023-11-24T08:23:51.746923+0000","last_active":"2023-11-24T08:23:53.435246+0000","last_peered":"2023-11-24T08:23:53.435246+0000","last_clean":"2023-11-24T08:23:53.435246+0000","last_became_active":"2023-11-24T08:23:51.746326+0000","last_became_peered":"2023-11-24T08:23:51.746326+0000","last_unstale":"2023-11-24T08:23:53.435246+0000","last_undegraded":"2023-11-24T08:23:53.435246+0000","last_fullsized":"2023-11-24T08:23:53.435246+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:49.785657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950996+0000","last_change":"2023-11-24T08:23:51.744211+0000","last_active":"2023-11-24T08:23:52.950996+0000","last_peered":"2023-11-24T08:23:52.950996+0000","last_clean":"2023-11-24T08:23:52.950996+0000","last_became_active":"2023-11-24T08:23:51.743943+0000","last_became_peered":"2023-11-24T08:23:51.743943+0000","last_unstale":"2023-11-24T08:23:52.950996+0000","last_undegraded":"2023-11-24T08:23:52.950996+0000","last_fullsized":"2023-11-24T08:23:52.950996+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:37:48.954662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"233'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169994+0000","last_change":"2023-11-24T08:23:50.745414+0000","last_active":"2023-11-24T08:23:53.169994+0000","last_peered":"2023-11-24T08:23:53.169994+0000","last_clean":"2023-11-24T08:23:53.169994+0000","last_became_active":"2023-11-24T08:23:50.744421+0000","last_became_peered":"2023-11-24T08:23:50.744421+0000","last_unstale":"2023-11-24T08:23:53.169994+0000","last_undegraded":"2023-11-24T08:23:53.169994+0000","last_fullsized":"2023-11-24T08:23:53.169994+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:09.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169438+0000","last_change":"2023-11-24T08:23:51.756431+0000","last_active":"2023-11-24T08:23:53.169438+0000","last_peered":"2023-11-24T08:23:53.169438+0000","last_clean":"2023-11-24T08:23:53.169438+0000","last_became_active":"2023-11-24T08:23:51.750896+0000","last_became_peered":"2023-11-24T08:23:51.750896+0000","last_unstale":"2023-11-24T08:23:53.169438+0000","last_undegraded":"2023-11-24T08:23:53.169438+0000","last_fullsized":"2023-11-24T08:23:53.169438+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:10.269318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950817+0000","last_change":"2023-11-24T08:23:50.742646+0000","last_active":"2023-11-24T08:23:52.950817+0000","last_peered":"2023-11-24T08:23:52.950817+0000","last_clean":"2023-11-24T08:23:52.950817+0000","last_became_active":"2023-11-24T08:23:50.742321+0000","last_became_peered":"2023-11-24T08:23:50.742321+0000","last_unstale":"2023-11-24T08:23:52.950817+0000","last_undegraded":"2023-11-24T08:23:52.950817+0000","last_fullsized":"2023-11-24T08:23:52.950817+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:53.045264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169979+0000","last_change":"2023-11-24T08:23:51.734522+0000","last_active":"2023-11-24T08:23:53.169979+0000","last_peered":"2023-11-24T08:23:53.169979+0000","last_clean":"2023-11-24T08:23:53.169979+0000","last_became_active":"2023-11-24T08:23:51.732052+0000","last_became_peered":"2023-11-24T08:23:51.732052+0000","last_unstale":"2023-11-24T08:23:53.169979+0000","last_undegraded":"2023-11-24T08:23:53.169979+0000","last_fullsized":"2023-11-24T08:23:53.169979+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:38.470361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743926+0000","last_change":"2023-11-24T08:23:50.737165+0000","last_active":"2023-11-24T08:23:52.743926+0000","last_peered":"2023-11-24T08:23:52.743926+0000","last_clean":"2023-11-24T08:23:52.743926+0000","last_became_active":"2023-11-24T08:23:50.735760+0000","last_became_peered":"2023-11-24T08:23:50.735760+0000","last_unstale":"2023-11-24T08:23:52.743926+0000","last_undegraded":"2023-11-24T08:23:52.743926+0000","last_fullsized":"2023-11-24T08:23:52.743926+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:43.358885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436592+0000","last_change":"2023-11-24T08:23:51.752929+0000","last_active":"2023-11-24T08:23:53.436592+0000","last_peered":"2023-11-24T08:23:53.436592+0000","last_clean":"2023-11-24T08:23:53.436592+0000","last_became_active":"2023-11-24T08:23:51.752643+0000","last_became_peered":"2023-11-24T08:23:51.752643+0000","last_unstale":"2023-11-24T08:23:53.436592+0000","last_undegraded":"2023-11-24T08:23:53.436592+0000","last_fullsized":"2023-11-24T08:23:53.436592+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:52.263816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950863+0000","last_change":"2023-11-24T08:23:50.743094+0000","last_active":"2023-11-24T08:23:52.950863+0000","last_peered":"2023-11-24T08:23:52.950863+0000","last_clean":"2023-11-24T08:23:52.950863+0000","last_became_active":"2023-11-24T08:23:50.740403+0000","last_became_peered":"2023-11-24T08:23:50.740403+0000","last_unstale":"2023-11-24T08:23:52.950863+0000","last_undegraded":"2023-11-24T08:23:52.950863+0000","last_fullsized":"2023-11-24T08:23:52.950863+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:33.307649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435376+0000","last_change":"2023-11-24T08:23:51.747086+0000","last_active":"2023-11-24T08:23:53.435376+0000","last_peered":"2023-11-24T08:23:53.435376+0000","last_clean":"2023-11-24T08:23:53.435376+0000","last_became_active":"2023-11-24T08:23:51.746646+0000","last_became_peered":"2023-11-24T08:23:51.746646+0000","last_unstale":"2023-11-24T08:23:53.435376+0000","last_undegraded":"2023-11-24T08:23:53.435376+0000","last_fullsized":"2023-11-24T08:23:53.435376+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:04.547941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451822+0000","last_change":"2023-11-24T08:23:50.743259+0000","last_active":"2023-11-24T08:23:53.451822+0000","last_peered":"2023-11-24T08:23:53.451822+0000","last_clean":"2023-11-24T08:23:53.451822+0000","last_became_active":"2023-11-24T08:23:50.741163+0000","last_became_peered":"2023-11-24T08:23:50.741163+0000","last_unstale":"2023-11-24T08:23:53.451822+0000","last_undegraded":"2023-11-24T08:23:53.451822+0000","last_fullsized":"2023-11-24T08:23:53.451822+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:20.259677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435881+0000","last_change":"2023-11-24T08:23:51.757722+0000","last_active":"2023-11-24T08:23:53.435881+0000","last_peered":"2023-11-24T08:23:53.435881+0000","last_clean":"2023-11-24T08:23:53.435881+0000","last_became_active":"2023-11-24T08:23:51.756041+0000","last_became_peered":"2023-11-24T08:23:51.756041+0000","last_unstale":"2023-11-24T08:23:53.435881+0000","last_undegraded":"2023-11-24T08:23:53.435881+0000","last_fullsized":"2023-11-24T08:23:53.435881+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:29:47.614952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435997+0000","last_change":"2023-11-24T08:23:50.735773+0000","last_active":"2023-11-24T08:23:53.435997+0000","last_peered":"2023-11-24T08:23:53.435997+0000","last_clean":"2023-11-24T08:23:53.435997+0000","last_became_active":"2023-11-24T08:23:50.735532+0000","last_became_peered":"2023-11-24T08:23:50.735532+0000","last_unstale":"2023-11-24T08:23:53.435997+0000","last_undegraded":"2023-11-24T08:23:53.435997+0000","last_fullsized":"2023-11-24T08:23:53.435997+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:43.958832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740340+0000","last_change":"2023-11-24T08:23:51.755193+0000","last_active":"2023-11-24T08:23:52.740340+0000","last_peered":"2023-11-24T08:23:52.740340+0000","last_clean":"2023-11-24T08:23:52.740340+0000","last_became_active":"2023-11-24T08:23:51.754865+0000","last_became_peered":"2023-11-24T08:23:51.754865+0000","last_unstale":"2023-11-24T08:23:52.740340+0000","last_undegraded":"2023-11-24T08:23:52.740340+0000","last_fullsized":"2023-11-24T08:23:52.740340+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:05.162558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950354+0000","last_change":"2023-11-24T08:23:50.747033+0000","last_active":"2023-11-24T08:23:52.950354+0000","last_peered":"2023-11-24T08:23:52.950354+0000","last_clean":"2023-11-24T08:23:52.950354+0000","last_became_active":"2023-11-24T08:23:50.746726+0000","last_became_peered":"2023-11-24T08:23:50.746726+0000","last_unstale":"2023-11-24T08:23:52.950354+0000","last_undegraded":"2023-11-24T08:23:52.950354+0000","last_fullsized":"2023-11-24T08:23:52.950354+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:01:08.955834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453634+0000","last_change":"2023-11-24T08:23:51.745227+0000","last_active":"2023-11-24T08:23:53.453634+0000","last_peered":"2023-11-24T08:23:53.453634+0000","last_clean":"2023-11-24T08:23:53.453634+0000","last_became_active":"2023-11-24T08:23:51.744752+0000","last_became_peered":"2023-11-24T08:23:51.744752+0000","last_unstale":"2023-11-24T08:23:53.453634+0000","last_undegraded":"2023-11-24T08:23:53.453634+0000","last_fullsized":"2023-11-24T08:23:53.453634+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:23.159738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450698+0000","last_change":"2023-11-24T08:23:50.742451+0000","last_active":"2023-11-24T08:23:53.450698+0000","last_peered":"2023-11-24T08:23:53.450698+0000","last_clean":"2023-11-24T08:23:53.450698+0000","last_became_active":"2023-11-24T08:23:50.742075+0000","last_became_peered":"2023-11-24T08:23:50.742075+0000","last_unstale":"2023-11-24T08:23:53.450698+0000","last_undegraded":"2023-11-24T08:23:53.450698+0000","last_fullsized":"2023-11-24T08:23:53.450698+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:52:24.053860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453443+0000","last_change":"2023-11-24T08:23:51.749332+0000","last_active":"2023-11-24T08:23:53.453443+0000","last_peered":"2023-11-24T08:23:53.453443+0000","last_clean":"2023-11-24T08:23:53.453443+0000","last_became_active":"2023-11-24T08:23:51.749083+0000","last_became_peered":"2023-11-24T08:23:51.749083+0000","last_unstale":"2023-11-24T08:23:53.453443+0000","last_undegraded":"2023-11-24T08:23:53.453443+0000","last_fullsized":"2023-11-24T08:23:53.453443+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:41.351199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453721+0000","last_change":"2023-11-24T08:23:50.738749+0000","last_active":"2023-11-24T08:23:53.453721+0000","last_peered":"2023-11-24T08:23:53.453721+0000","last_clean":"2023-11-24T08:23:53.453721+0000","last_became_active":"2023-11-24T08:23:50.738203+0000","last_became_peered":"2023-11-24T08:23:50.738203+0000","last_unstale":"2023-11-24T08:23:53.453721+0000","last_undegraded":"2023-11-24T08:23:53.453721+0000","last_fullsized":"2023-11-24T08:23:53.453721+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:43.164878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169770+0000","last_change":"2023-11-24T08:23:51.746034+0000","last_active":"2023-11-24T08:23:53.169770+0000","last_peered":"2023-11-24T08:23:53.169770+0000","last_clean":"2023-11-24T08:23:53.169770+0000","last_became_active":"2023-11-24T08:23:51.745777+0000","last_became_peered":"2023-11-24T08:23:51.745777+0000","last_unstale":"2023-11-24T08:23:53.169770+0000","last_undegraded":"2023-11-24T08:23:53.169770+0000","last_fullsized":"2023-11-24T08:23:53.169770+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:00:04.041443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.951157+0000","last_change":"2023-11-24T08:23:50.745757+0000","last_active":"2023-11-24T08:23:52.951157+0000","last_peered":"2023-11-24T08:23:52.951157+0000","last_clean":"2023-11-24T08:23:52.951157+0000","last_became_active":"2023-11-24T08:23:50.745472+0000","last_became_peered":"2023-11-24T08:23:50.745472+0000","last_unstale":"2023-11-24T08:23:52.951157+0000","last_undegraded":"2023-11-24T08:23:52.951157+0000","last_fullsized":"2023-11-24T08:23:52.951157+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:11:11.435717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452502+0000","last_change":"2023-11-24T08:23:51.742673+0000","last_active":"2023-11-24T08:23:53.452502+0000","last_peered":"2023-11-24T08:23:53.452502+0000","last_clean":"2023-11-24T08:23:53.452502+0000","last_became_active":"2023-11-24T08:23:51.742313+0000","last_became_peered":"2023-11-24T08:23:51.742313+0000","last_unstale":"2023-11-24T08:23:53.452502+0000","last_undegraded":"2023-11-24T08:23:53.452502+0000","last_fullsized":"2023-11-24T08:23:53.452502+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:57:21.409950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"233'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453398+0000","last_change":"2023-11-24T08:23:50.740617+0000","last_active":"2023-11-24T08:23:53.453398+0000","last_peered":"2023-11-24T08:23:53.453398+0000","last_clean":"2023-11-24T08:23:53.453398+0000","last_became_active":"2023-11-24T08:23:50.740303+0000","last_became_peered":"2023-11-24T08:23:50.740303+0000","last_unstale":"2023-11-24T08:23:53.453398+0000","last_undegraded":"2023-11-24T08:23:53.453398+0000","last_fullsized":"2023-11-24T08:23:53.453398+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:57.631797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743948+0000","last_change":"2023-11-24T08:23:51.760226+0000","last_active":"2023-11-24T08:23:52.743948+0000","last_peered":"2023-11-24T08:23:52.743948+0000","last_clean":"2023-11-24T08:23:52.743948+0000","last_became_active":"2023-11-24T08:23:51.759752+0000","last_became_peered":"2023-11-24T08:23:51.759752+0000","last_unstale":"2023-11-24T08:23:52.743948+0000","last_undegraded":"2023-11-24T08:23:52.743948+0000","last_fullsized":"2023-11-24T08:23:52.743948+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:38:52.586714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"233'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169356+0000","last_change":"2023-11-24T08:23:50.734037+0000","last_active":"2023-11-24T08:23:53.169356+0000","last_peered":"2023-11-24T08:23:53.169356+0000","last_clean":"2023-11-24T08:23:53.169356+0000","last_became_active":"2023-11-24T08:23:50.733725+0000","last_became_peered":"2023-11-24T08:23:50.733725+0000","last_unstale":"2023-11-24T08:23:53.169356+0000","last_undegraded":"2023-11-24T08:23:53.169356+0000","last_fullsized":"2023-11-24T08:23:53.169356+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:32.635233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450732+0000","last_change":"2023-11-24T08:23:51.749969+0000","last_active":"2023-11-24T08:23:53.450732+0000","last_peered":"2023-11-24T08:23:53.450732+0000","last_clean":"2023-11-24T08:23:53.450732+0000","last_became_active":"2023-11-24T08:23:51.749501+0000","last_became_peered":"2023-11-24T08:23:51.749501+0000","last_unstale":"2023-11-24T08:23:53.450732+0000","last_undegraded":"2023-11-24T08:23:53.450732+0000","last_fullsized":"2023-11-24T08:23:53.450732+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:35:24.707920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451523+0000","last_change":"2023-11-24T08:23:50.743196+0000","last_active":"2023-11-24T08:23:53.451523+0000","last_peered":"2023-11-24T08:23:53.451523+0000","last_clean":"2023-11-24T08:23:53.451523+0000","last_became_active":"2023-11-24T08:23:50.742851+0000","last_became_peered":"2023-11-24T08:23:50.742851+0000","last_unstale":"2023-11-24T08:23:53.451523+0000","last_undegraded":"2023-11-24T08:23:53.451523+0000","last_fullsized":"2023-11-24T08:23:53.451523+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:58.426446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742202+0000","last_change":"2023-11-24T08:23:51.754214+0000","last_active":"2023-11-24T08:23:52.742202+0000","last_peered":"2023-11-24T08:23:52.742202+0000","last_clean":"2023-11-24T08:23:52.742202+0000","last_became_active":"2023-11-24T08:23:51.753116+0000","last_became_peered":"2023-11-24T08:23:51.753116+0000","last_unstale":"2023-11-24T08:23:52.742202+0000","last_undegraded":"2023-11-24T08:23:52.742202+0000","last_fullsized":"2023-11-24T08:23:52.742202+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:14:33.553313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453116+0000","last_change":"2023-11-24T08:23:50.745479+0000","last_active":"2023-11-24T08:23:53.453116+0000","last_peered":"2023-11-24T08:23:53.453116+0000","last_clean":"2023-11-24T08:23:53.453116+0000","last_became_active":"2023-11-24T08:23:50.745166+0000","last_became_peered":"2023-11-24T08:23:50.745166+0000","last_unstale":"2023-11-24T08:23:53.453116+0000","last_undegraded":"2023-11-24T08:23:53.453116+0000","last_fullsized":"2023-11-24T08:23:53.453116+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:49.087604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451019+0000","last_change":"2023-11-24T08:23:51.757099+0000","last_active":"2023-11-24T08:23:53.451019+0000","last_peered":"2023-11-24T08:23:53.451019+0000","last_clean":"2023-11-24T08:23:53.451019+0000","last_became_active":"2023-11-24T08:23:51.756846+0000","last_became_peered":"2023-11-24T08:23:51.756846+0000","last_unstale":"2023-11-24T08:23:53.451019+0000","last_undegraded":"2023-11-24T08:23:53.451019+0000","last_fullsized":"2023-11-24T08:23:53.451019+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:18.755797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452540+0000","last_change":"2023-11-24T08:23:50.739261+0000","last_active":"2023-11-24T08:23:53.452540+0000","last_peered":"2023-11-24T08:23:53.452540+0000","last_clean":"2023-11-24T08:23:53.452540+0000","last_became_active":"2023-11-24T08:23:50.738972+0000","last_became_peered":"2023-11-24T08:23:50.738972+0000","last_unstale":"2023-11-24T08:23:53.452540+0000","last_undegraded":"2023-11-24T08:23:53.452540+0000","last_fullsized":"2023-11-24T08:23:53.452540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:26.180592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451170+0000","last_change":"2023-11-24T08:23:51.746135+0000","last_active":"2023-11-24T08:23:53.451170+0000","last_peered":"2023-11-24T08:23:53.451170+0000","last_clean":"2023-11-24T08:23:53.451170+0000","last_became_active":"2023-11-24T08:23:51.745547+0000","last_became_peered":"2023-11-24T08:23:51.745547+0000","last_unstale":"2023-11-24T08:23:53.451170+0000","last_undegraded":"2023-11-24T08:23:53.451170+0000","last_fullsized":"2023-11-24T08:23:53.451170+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:32.431718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"233'8","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452128+0000","last_change":"2023-11-24T08:23:50.737570+0000","last_active":"2023-11-24T08:23:53.452128+0000","last_peered":"2023-11-24T08:23:53.452128+0000","last_clean":"2023-11-24T08:23:53.452128+0000","last_became_active":"2023-11-24T08:23:50.737287+0000","last_became_peered":"2023-11-24T08:23:50.737287+0000","last_unstale":"2023-11-24T08:23:53.452128+0000","last_undegraded":"2023-11-24T08:23:53.452128+0000","last_fullsized":"2023-11-24T08:23:53.452128+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:06.196465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743055+0000","last_change":"2023-11-24T08:23:51.749623+0000","last_active":"2023-11-24T08:23:52.743055+0000","last_peered":"2023-11-24T08:23:52.743055+0000","last_clean":"2023-11-24T08:23:52.743055+0000","last_became_active":"2023-11-24T08:23:51.749348+0000","last_became_peered":"2023-11-24T08:23:51.749348+0000","last_unstale":"2023-11-24T08:23:52.743055+0000","last_undegraded":"2023-11-24T08:23:52.743055+0000","last_fullsized":"2023-11-24T08:23:52.743055+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:44.742316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452931+0000","last_change":"2023-11-24T08:23:50.747731+0000","last_active":"2023-11-24T08:23:53.452931+0000","last_peered":"2023-11-24T08:23:53.452931+0000","last_clean":"2023-11-24T08:23:53.452931+0000","last_became_active":"2023-11-24T08:23:50.746950+0000","last_became_peered":"2023-11-24T08:23:50.746950+0000","last_unstale":"2023-11-24T08:23:53.452931+0000","last_undegraded":"2023-11-24T08:23:53.452931+0000","last_fullsized":"2023-11-24T08:23:53.452931+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:53.045264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743981+0000","last_change":"2023-11-24T08:23:51.753810+0000","last_active":"2023-11-24T08:23:52.743981+0000","last_peered":"2023-11-24T08:23:52.743981+0000","last_clean":"2023-11-24T08:23:52.743981+0000","last_became_active":"2023-11-24T08:23:51.753554+0000","last_became_peered":"2023-11-24T08:23:51.753554+0000","last_unstale":"2023-11-24T08:23:52.743981+0000","last_undegraded":"2023-11-24T08:23:52.743981+0000","last_fullsized":"2023-11-24T08:23:52.743981+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:54.413906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169980+0000","last_change":"2023-11-24T08:23:50.738359+0000","last_active":"2023-11-24T08:23:53.169980+0000","last_peered":"2023-11-24T08:23:53.169980+0000","last_clean":"2023-11-24T08:23:53.169980+0000","last_became_active":"2023-11-24T08:23:50.738060+0000","last_became_peered":"2023-11-24T08:23:50.738060+0000","last_unstale":"2023-11-24T08:23:53.169980+0000","last_undegraded":"2023-11-24T08:23:53.169980+0000","last_fullsized":"2023-11-24T08:23:53.169980+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:49.836574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453866+0000","last_change":"2023-11-24T08:23:51.747211+0000","last_active":"2023-11-24T08:23:53.453866+0000","last_peered":"2023-11-24T08:23:53.453866+0000","last_clean":"2023-11-24T08:23:53.453866+0000","last_became_active":"2023-11-24T08:23:51.746945+0000","last_became_peered":"2023-11-24T08:23:51.746945+0000","last_unstale":"2023-11-24T08:23:53.453866+0000","last_undegraded":"2023-11-24T08:23:53.453866+0000","last_fullsized":"2023-11-24T08:23:53.453866+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:13.855889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.741038+0000","last_change":"2023-11-24T08:23:50.740141+0000","last_active":"2023-11-24T08:23:52.741038+0000","last_peered":"2023-11-24T08:23:52.741038+0000","last_clean":"2023-11-24T08:23:52.741038+0000","last_became_active":"2023-11-24T08:23:50.739848+0000","last_became_peered":"2023-11-24T08:23:50.739848+0000","last_unstale":"2023-11-24T08:23:52.741038+0000","last_undegraded":"2023-11-24T08:23:52.741038+0000","last_fullsized":"2023-11-24T08:23:52.741038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:42.775374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.744035+0000","last_change":"2023-11-24T08:23:51.760036+0000","last_active":"2023-11-24T08:23:52.744035+0000","last_peered":"2023-11-24T08:23:52.744035+0000","last_clean":"2023-11-24T08:23:52.744035+0000","last_became_active":"2023-11-24T08:23:51.759528+0000","last_became_peered":"2023-11-24T08:23:51.759528+0000","last_unstale":"2023-11-24T08:23:52.744035+0000","last_undegraded":"2023-11-24T08:23:52.744035+0000","last_fullsized":"2023-11-24T08:23:52.744035+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:19.613655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169311+0000","last_change":"2023-11-24T08:23:50.744713+0000","last_active":"2023-11-24T08:23:53.169311+0000","last_peered":"2023-11-24T08:23:53.169311+0000","last_clean":"2023-11-24T08:23:53.169311+0000","last_became_active":"2023-11-24T08:23:50.744400+0000","last_became_peered":"2023-11-24T08:23:50.744400+0000","last_unstale":"2023-11-24T08:23:53.169311+0000","last_undegraded":"2023-11-24T08:23:53.169311+0000","last_fullsized":"2023-11-24T08:23:53.169311+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:11.049034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435807+0000","last_change":"2023-11-24T08:23:51.756290+0000","last_active":"2023-11-24T08:23:53.435807+0000","last_peered":"2023-11-24T08:23:53.435807+0000","last_clean":"2023-11-24T08:23:53.435807+0000","last_became_active":"2023-11-24T08:23:51.753588+0000","last_became_peered":"2023-11-24T08:23:51.753588+0000","last_unstale":"2023-11-24T08:23:53.435807+0000","last_undegraded":"2023-11-24T08:23:53.435807+0000","last_fullsized":"2023-11-24T08:23:53.435807+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:52.270610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451547+0000","last_change":"2023-11-24T08:23:50.741463+0000","last_active":"2023-11-24T08:23:53.451547+0000","last_peered":"2023-11-24T08:23:53.451547+0000","last_clean":"2023-11-24T08:23:53.451547+0000","last_became_active":"2023-11-24T08:23:50.740837+0000","last_became_peered":"2023-11-24T08:23:50.740837+0000","last_unstale":"2023-11-24T08:23:53.451547+0000","last_undegraded":"2023-11-24T08:23:53.451547+0000","last_fullsized":"2023-11-24T08:23:53.451547+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:06.503801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742387+0000","last_change":"2023-11-24T08:23:51.751098+0000","last_active":"2023-11-24T08:23:52.742387+0000","last_peered":"2023-11-24T08:23:52.742387+0000","last_clean":"2023-11-24T08:23:52.742387+0000","last_became_active":"2023-11-24T08:23:51.750773+0000","last_became_peered":"2023-11-24T08:23:51.750773+0000","last_unstale":"2023-11-24T08:23:52.742387+0000","last_undegraded":"2023-11-24T08:23:52.742387+0000","last_fullsized":"2023-11-24T08:23:52.742387+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:15.959648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450695+0000","last_change":"2023-11-24T08:23:50.739395+0000","last_active":"2023-11-24T08:23:53.450695+0000","last_peered":"2023-11-24T08:23:53.450695+0000","last_clean":"2023-11-24T08:23:53.450695+0000","last_became_active":"2023-11-24T08:23:50.739078+0000","last_became_peered":"2023-11-24T08:23:50.739078+0000","last_unstale":"2023-11-24T08:23:53.450695+0000","last_undegraded":"2023-11-24T08:23:53.450695+0000","last_fullsized":"2023-11-24T08:23:53.450695+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:37.400823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452849+0000","last_change":"2023-11-24T08:23:51.746079+0000","last_active":"2023-11-24T08:23:53.452849+0000","last_peered":"2023-11-24T08:23:53.452849+0000","last_clean":"2023-11-24T08:23:53.452849+0000","last_became_active":"2023-11-24T08:23:51.745802+0000","last_became_peered":"2023-11-24T08:23:51.745802+0000","last_unstale":"2023-11-24T08:23:53.452849+0000","last_undegraded":"2023-11-24T08:23:53.452849+0000","last_fullsized":"2023-11-24T08:23:53.452849+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:01.191648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436126+0000","last_change":"2023-11-24T08:23:50.734260+0000","last_active":"2023-11-24T08:23:53.436126+0000","last_peered":"2023-11-24T08:23:53.436126+0000","last_clean":"2023-11-24T08:23:53.436126+0000","last_became_active":"2023-11-24T08:23:50.732860+0000","last_became_peered":"2023-11-24T08:23:50.732860+0000","last_unstale":"2023-11-24T08:23:53.436126+0000","last_undegraded":"2023-11-24T08:23:53.436126+0000","last_fullsized":"2023-11-24T08:23:53.436126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:58.426911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435876+0000","last_change":"2023-11-24T08:23:51.748664+0000","last_active":"2023-11-24T08:23:53.435876+0000","last_peered":"2023-11-24T08:23:53.435876+0000","last_clean":"2023-11-24T08:23:53.435876+0000","last_became_active":"2023-11-24T08:23:51.742119+0000","last_became_peered":"2023-11-24T08:23:51.742119+0000","last_unstale":"2023-11-24T08:23:53.435876+0000","last_undegraded":"2023-11-24T08:23:53.435876+0000","last_fullsized":"2023-11-24T08:23:53.435876+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:22.852132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452389+0000","last_change":"2023-11-24T08:23:50.742621+0000","last_active":"2023-11-24T08:23:53.452389+0000","last_peered":"2023-11-24T08:23:53.452389+0000","last_clean":"2023-11-24T08:23:53.452389+0000","last_became_active":"2023-11-24T08:23:50.742332+0000","last_became_peered":"2023-11-24T08:23:50.742332+0000","last_unstale":"2023-11-24T08:23:53.452389+0000","last_undegraded":"2023-11-24T08:23:53.452389+0000","last_fullsized":"2023-11-24T08:23:53.452389+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:10.989887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452944+0000","last_change":"2023-11-24T08:23:51.751924+0000","last_active":"2023-11-24T08:23:53.452944+0000","last_peered":"2023-11-24T08:23:53.452944+0000","last_clean":"2023-11-24T08:23:53.452944+0000","last_became_active":"2023-11-24T08:23:51.751631+0000","last_became_peered":"2023-11-24T08:23:51.751631+0000","last_unstale":"2023-11-24T08:23:53.452944+0000","last_undegraded":"2023-11-24T08:23:53.452944+0000","last_fullsized":"2023-11-24T08:23:53.452944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:08.960033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453761+0000","last_change":"2023-11-24T08:23:50.745721+0000","last_active":"2023-11-24T08:23:53.453761+0000","last_peered":"2023-11-24T08:23:53.453761+0000","last_clean":"2023-11-24T08:23:53.453761+0000","last_became_active":"2023-11-24T08:23:50.743807+0000","last_became_peered":"2023-11-24T08:23:50.743807+0000","last_unstale":"2023-11-24T08:23:53.453761+0000","last_undegraded":"2023-11-24T08:23:53.453761+0000","last_fullsized":"2023-11-24T08:23:53.453761+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:08.387783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452312+0000","last_change":"2023-11-24T08:23:51.753149+0000","last_active":"2023-11-24T08:23:53.452312+0000","last_peered":"2023-11-24T08:23:53.452312+0000","last_clean":"2023-11-24T08:23:53.452312+0000","last_became_active":"2023-11-24T08:23:51.752892+0000","last_became_peered":"2023-11-24T08:23:51.752892+0000","last_unstale":"2023-11-24T08:23:53.452312+0000","last_undegraded":"2023-11-24T08:23:53.452312+0000","last_fullsized":"2023-11-24T08:23:53.452312+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:37:48.954662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.170133+0000","last_change":"2023-11-24T08:23:50.737842+0000","last_active":"2023-11-24T08:23:53.170133+0000","last_peered":"2023-11-24T08:23:53.170133+0000","last_clean":"2023-11-24T08:23:53.170133+0000","last_became_active":"2023-11-24T08:23:50.737598+0000","last_became_peered":"2023-11-24T08:23:50.737598+0000","last_unstale":"2023-11-24T08:23:53.170133+0000","last_undegraded":"2023-11-24T08:23:53.170133+0000","last_fullsized":"2023-11-24T08:23:53.170133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.291573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740797+0000","last_change":"2023-11-24T08:23:51.756579+0000","last_active":"2023-11-24T08:23:52.740797+0000","last_peered":"2023-11-24T08:23:52.740797+0000","last_clean":"2023-11-24T08:23:52.740797+0000","last_became_active":"2023-11-24T08:23:51.756331+0000","last_became_peered":"2023-11-24T08:23:51.756331+0000","last_unstale":"2023-11-24T08:23:52.740797+0000","last_undegraded":"2023-11-24T08:23:52.740797+0000","last_fullsized":"2023-11-24T08:23:52.740797+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:47.627900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740468+0000","last_change":"2023-11-24T08:23:50.738854+0000","last_active":"2023-11-24T08:23:52.740468+0000","last_peered":"2023-11-24T08:23:52.740468+0000","last_clean":"2023-11-24T08:23:52.740468+0000","last_became_active":"2023-11-24T08:23:50.738570+0000","last_became_peered":"2023-11-24T08:23:50.738570+0000","last_unstale":"2023-11-24T08:23:52.740468+0000","last_undegraded":"2023-11-24T08:23:52.740468+0000","last_fullsized":"2023-11-24T08:23:52.740468+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:09:53.907460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452743+0000","last_change":"2023-11-24T08:23:51.747288+0000","last_active":"2023-11-24T08:23:53.452743+0000","last_peered":"2023-11-24T08:23:53.452743+0000","last_clean":"2023-11-24T08:23:53.452743+0000","last_became_active":"2023-11-24T08:23:51.746998+0000","last_became_peered":"2023-11-24T08:23:51.746998+0000","last_unstale":"2023-11-24T08:23:53.452743+0000","last_undegraded":"2023-11-24T08:23:53.452743+0000","last_fullsized":"2023-11-24T08:23:53.452743+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:30.172216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740596+0000","last_change":"2023-11-24T08:23:50.744213+0000","last_active":"2023-11-24T08:23:52.740596+0000","last_peered":"2023-11-24T08:23:52.740596+0000","last_clean":"2023-11-24T08:23:52.740596+0000","last_became_active":"2023-11-24T08:23:50.743316+0000","last_became_peered":"2023-11-24T08:23:50.743316+0000","last_unstale":"2023-11-24T08:23:52.740596+0000","last_undegraded":"2023-11-24T08:23:52.740596+0000","last_fullsized":"2023-11-24T08:23:52.740596+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:26.702946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451788+0000","last_change":"2023-11-24T08:23:51.743942+0000","last_active":"2023-11-24T08:23:53.451788+0000","last_peered":"2023-11-24T08:23:53.451788+0000","last_clean":"2023-11-24T08:23:53.451788+0000","last_became_active":"2023-11-24T08:23:51.743659+0000","last_became_peered":"2023-11-24T08:23:51.743659+0000","last_unstale":"2023-11-24T08:23:53.451788+0000","last_undegraded":"2023-11-24T08:23:53.451788+0000","last_fullsized":"2023-11-24T08:23:53.451788+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:55:02.933279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169475+0000","last_change":"2023-11-24T08:23:50.737231+0000","last_active":"2023-11-24T08:23:53.169475+0000","last_peered":"2023-11-24T08:23:53.169475+0000","last_clean":"2023-11-24T08:23:53.169475+0000","last_became_active":"2023-11-24T08:23:50.736949+0000","last_became_peered":"2023-11-24T08:23:50.736949+0000","last_unstale":"2023-11-24T08:23:53.169475+0000","last_undegraded":"2023-11-24T08:23:53.169475+0000","last_fullsized":"2023-11-24T08:23:53.169475+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:05.768024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436269+0000","last_change":"2023-11-24T08:23:51.752405+0000","last_active":"2023-11-24T08:23:53.436269+0000","last_peered":"2023-11-24T08:23:53.436269+0000","last_clean":"2023-11-24T08:23:53.436269+0000","last_became_active":"2023-11-24T08:23:51.752076+0000","last_became_peered":"2023-11-24T08:23:51.752076+0000","last_unstale":"2023-11-24T08:23:53.436269+0000","last_undegraded":"2023-11-24T08:23:53.436269+0000","last_fullsized":"2023-11-24T08:23:53.436269+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:10.526577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169402+0000","last_change":"2023-11-24T08:23:50.736509+0000","last_active":"2023-11-24T08:23:53.169402+0000","last_peered":"2023-11-24T08:23:53.169402+0000","last_clean":"2023-11-24T08:23:53.169402+0000","last_became_active":"2023-11-24T08:23:50.734921+0000","last_became_peered":"2023-11-24T08:23:50.734921+0000","last_unstale":"2023-11-24T08:23:53.169402+0000","last_undegraded":"2023-11-24T08:23:53.169402+0000","last_fullsized":"2023-11-24T08:23:53.169402+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:00:25.221410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435848+0000","last_change":"2023-11-24T08:23:51.749798+0000","last_active":"2023-11-24T08:23:53.435848+0000","last_peered":"2023-11-24T08:23:53.435848+0000","last_clean":"2023-11-24T08:23:53.435848+0000","last_became_active":"2023-11-24T08:23:51.749531+0000","last_became_peered":"2023-11-24T08:23:51.749531+0000","last_unstale":"2023-11-24T08:23:53.435848+0000","last_undegraded":"2023-11-24T08:23:53.435848+0000","last_fullsized":"2023-11-24T08:23:53.435848+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:54.835444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.739864+0000","last_change":"2023-11-24T08:23:50.741024+0000","last_active":"2023-11-24T08:23:52.739864+0000","last_peered":"2023-11-24T08:23:52.739864+0000","last_clean":"2023-11-24T08:23:52.739864+0000","last_became_active":"2023-11-24T08:23:50.740717+0000","last_became_peered":"2023-11-24T08:23:50.740717+0000","last_unstale":"2023-11-24T08:23:52.739864+0000","last_undegraded":"2023-11-24T08:23:52.739864+0000","last_fullsized":"2023-11-24T08:23:52.739864+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:10.613056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950970+0000","last_change":"2023-11-24T08:23:51.739273+0000","last_active":"2023-11-24T08:23:52.950970+0000","last_peered":"2023-11-24T08:23:52.950970+0000","last_clean":"2023-11-24T08:23:52.950970+0000","last_became_active":"2023-11-24T08:23:51.738963+0000","last_became_peered":"2023-11-24T08:23:51.738963+0000","last_unstale":"2023-11-24T08:23:52.950970+0000","last_undegraded":"2023-11-24T08:23:52.950970+0000","last_fullsized":"2023-11-24T08:23:52.950970+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:03.867689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169313+0000","last_change":"2023-11-24T08:23:50.744587+0000","last_active":"2023-11-24T08:23:53.169313+0000","last_peered":"2023-11-24T08:23:53.169313+0000","last_clean":"2023-11-24T08:23:53.169313+0000","last_became_active":"2023-11-24T08:23:50.744007+0000","last_became_peered":"2023-11-24T08:23:50.744007+0000","last_unstale":"2023-11-24T08:23:53.169313+0000","last_undegraded":"2023-11-24T08:23:53.169313+0000","last_fullsized":"2023-11-24T08:23:53.169313+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:40.898200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453440+0000","last_change":"2023-11-24T08:23:51.749577+0000","last_active":"2023-11-24T08:23:53.453440+0000","last_peered":"2023-11-24T08:23:53.453440+0000","last_clean":"2023-11-24T08:23:53.453440+0000","last_became_active":"2023-11-24T08:23:51.748280+0000","last_became_peered":"2023-11-24T08:23:51.748280+0000","last_unstale":"2023-11-24T08:23:53.453440+0000","last_undegraded":"2023-11-24T08:23:53.453440+0000","last_fullsized":"2023-11-24T08:23:53.453440+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:50.603390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-11-24T08:23:51.728785+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:23:51.728785+0000","last_peered":"2023-11-24T08:23:51.728785+0000","last_clean":"2023-11-24T08:23:51.728785+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:23:51.728785+0000","last_undegraded":"2023-11-24T08:23:51.728785+0000","last_fullsized":"2023-11-24T08:23:51.728785+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42976,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328864,"statfs":{"total":96636764160,"available":96592756736,"internally_reserved":0,"allocated":3551232,"data_stored":1902442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0880000000000001,"15min":1.0629999999999999},"min":{"1min":0.64000000000000001,"5min":0.61199999999999999,"15min":0.42399999999999999},"max":{"1min":1.752,"5min":2.048,"15min":2.7240000000000002},"last":0.88},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.1299999999999999,"15min":1.103},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.53700000000000003},"max":{"1min":1.256,"5min":1.899,"15min":2.403},"last":0.77800000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.137,"15min":1.1479999999999999},"min":{"1min":0.77200000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.5860000000000001,"5min":1.681,"15min":3.8519999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1739999999999999,"15min":1.2050000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.66600000000000004},"max":{"1min":1.409,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.359}]},{"osd":2,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.101,"15min":1.113},"min":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.4910000000000001,"5min":1.762,"15min":2.4430000000000001},"last":1.554},{"interface":"front","average":{"1min":1.222,"5min":1.2370000000000001,"15min":1.2849999999999999},"min":{"1min":0.84499999999999997,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8,"5min":1.8,"15min":4.0650000000000004},"last":1.3320000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.2,"15min":1.208},"min":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":3.133},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.325,"15min":1.361},"min":{"1min":0.95099999999999996,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.7669999999999999,"5min":2.173,"15min":3.9209999999999998},"last":1.139}]},{"osd":4,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3009999999999999,"15min":1.335},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.54700000000000004},"max":{"1min":1.556,"5min":1.8799999999999999,"15min":2.665},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.3080000000000001,"15min":1.341},"min":{"1min":0.54500000000000004,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.617,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.6830000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.429,"15min":1.4650000000000001},"min":{"1min":0.80800000000000005,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":1.8560000000000001,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.242},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.448,"15min":1.484},"min":{"1min":0.69499999999999995,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":1.8680000000000001,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.5029999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.548,"15min":1.5629999999999999},"min":{"1min":0.77000000000000002,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.9970000000000001,"5min":2.2610000000000001,"15min":2.992},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4470000000000001,"15min":1.488},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.8240000000000001,"5min":2.4159999999999999,"15min":3.798},"last":1.871}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42396,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329444,"statfs":{"total":96636764160,"available":96593350656,"internally_reserved":0,"allocated":2965504,"data_stored":1317804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.0660000000000001,"15min":1.0720000000000001},"min":{"1min":0.69799999999999995,"5min":0.61199999999999999,"15min":0.59099999999999997},"max":{"1min":5.3040000000000003,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":1.492},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.2070000000000001,"15min":1.1240000000000001},"min":{"1min":0.86699999999999999,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":13.516999999999999,"5min":13.516999999999999,"15min":13.516999999999999},"last":0.80600000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.143,"15min":1.149},"min":{"1min":0.74399999999999999,"5min":0.67700000000000005,"15min":0.61599999999999999},"max":{"1min":5.7050000000000001,"5min":5.7050000000000001,"15min":5.7050000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.851,"5min":1.3109999999999999,"15min":1.2270000000000001},"min":{"1min":0.83499999999999996,"5min":0.72799999999999998,"15min":0.65000000000000002},"max":{"1min":13.57,"5min":13.57,"15min":13.57},"last":1.103}]},{"osd":2,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.115,"15min":1.105},"min":{"1min":0.80900000000000005,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.258,"15min":1.258},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.60399999999999998},"max":{"1min":5.3440000000000003,"5min":5.3440000000000003,"15min":5.3440000000000003},"last":1.22}]},{"osd":3,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.4550000000000001,"5min":1.5509999999999999,"15min":1.409},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":23.989000000000001,"5min":23.989000000000001,"15min":23.989000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":2.4220000000000002,"5min":1.494,"15min":1.3740000000000001},"min":{"1min":0.94299999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":1.4350000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.911,"5min":1.47,"15min":1.4379999999999999},"min":{"1min":0.70199999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":1.7969999999999999},{"interface":"front","average":{"1min":2.3559999999999999,"5min":1.6259999999999999,"15min":1.5289999999999999},"min":{"1min":0.67300000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":20.173999999999999,"5min":20.173999999999999,"15min":20.173999999999999},"last":1.5229999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.3879999999999999,"15min":1.4119999999999999},"min":{"1min":0.77100000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.599},{"interface":"front","average":{"1min":1.8839999999999999,"5min":1.595,"15min":1.571},"min":{"1min":1.0129999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":8.3350000000000009,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":1.887}]},{"osd":7,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.669,"15min":1.6140000000000001},"min":{"1min":0.97599999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":13.612,"5min":13.612,"15min":13.612},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.5269999999999999,"15min":1.5669999999999999},"min":{"1min":0.95699999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9239999999999999,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.6659999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575081,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38244,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333596,"statfs":{"total":96636764160,"available":96597602304,"internally_reserved":0,"allocated":2965504,"data_stored":1317840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1,"15min":1.0109999999999999},"min":{"1min":0.61199999999999999,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.4299999999999999,"5min":1.696,"15min":4.9850000000000003},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.107,"15min":1.097},"min":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.4970000000000001,"5min":1.6639999999999999,"15min":5.4299999999999997},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.127,"15min":1.133},"min":{"1min":0.70499999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6000000000000001,"5min":1.7350000000000001,"15min":7.258},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.177,"15min":1.1899999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6359999999999999,"5min":1.796,"15min":6.5199999999999996},"last":1.5149999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.155,"15min":1.1739999999999999},"min":{"1min":0.72299999999999998,"5min":0.68100000000000005,"15min":0.65100000000000002},"max":{"1min":1.419,"5min":1.7829999999999999,"15min":6.8929999999999998},"last":1.601},{"interface":"front","average":{"1min":1.095,"5min":1.163,"15min":1.1799999999999999},"min":{"1min":0.72899999999999998,"5min":0.72399999999999998,"15min":0.68000000000000005},"max":{"1min":1.7050000000000001,"5min":1.923,"15min":8.0869999999999997},"last":1.4199999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1759999999999999,"15min":1.1839999999999999},"min":{"1min":0.85099999999999998,"5min":0.70799999999999996,"15min":0.66600000000000004},"max":{"1min":1.7170000000000001,"5min":2.242,"15min":9.0120000000000005},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.266,"15min":1.282},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.66800000000000004},"max":{"1min":1.6020000000000001,"5min":1.956,"15min":7.2130000000000001},"last":1.4650000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93200000000000005,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.917,"5min":2.536,"15min":11.813000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4590000000000001,"15min":1.4770000000000001},"min":{"1min":0.88900000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.667,"5min":2.4550000000000001,"15min":12.327999999999999},"last":1.6830000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.319,"15min":1.345},"min":{"1min":0.95299999999999996,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.536,"5min":2.246,"15min":9.4770000000000003},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.98199999999999998,"5min":0.70299999999999996,"15min":0.69299999999999995},"max":{"1min":1.9670000000000001,"5min":2.452,"15min":9.9019999999999992},"last":1.006}]},{"osd":7,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.49,"15min":1.5},"min":{"1min":1.246,"5min":0.86199999999999999,"15min":0.54300000000000004},"max":{"1min":1.6519999999999999,"5min":2.802,"15min":8.1660000000000004},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.6499999999999999},"min":{"1min":1.272,"5min":0.88600000000000001,"15min":0.75600000000000001},"max":{"1min":2.0070000000000001,"5min":2.9489999999999998,"15min":19.663},"last":1.6379999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42404,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329436,"statfs":{"total":96636764160,"available":96593342464,"internally_reserved":0,"allocated":2973696,"data_stored":1319382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0720000000000001,"15min":1.054},"min":{"1min":0.70699999999999996,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.8779999999999999},"last":1.528},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.46100000000000002},"max":{"1min":1.3140000000000001,"5min":1.591,"15min":5.452},"last":0.83599999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.139,"15min":1.1399999999999999},"min":{"1min":0.73599999999999999,"5min":0.67600000000000005,"15min":0.46300000000000002},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1000000000000001,"15min":1.133},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.51900000000000002},"max":{"1min":1.3089999999999999,"5min":1.6839999999999999,"15min":1.988},"last":1.0669999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0669999999999999,"15min":1.0629999999999999},"min":{"1min":0.79200000000000004,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.444,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.248},"min":{"1min":0.92800000000000005,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0470000000000002},"last":1.3129999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3480000000000001,"15min":1.337},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.62},"max":{"1min":1.722,"5min":1.7869999999999999,"15min":2.121},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2929999999999999,"15min":1.274},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.55800000000000005},"max":{"1min":1.6579999999999999,"5min":1.744,"15min":2.3340000000000001},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.492,"15min":1.4930000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.69299999999999995},"max":{"1min":1.913,"5min":1.913,"15min":2.2429999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4159999999999999,"15min":1.419},"min":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.66200000000000003},"max":{"1min":1.996,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.151}]},{"osd":6,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.278,"15min":1.2549999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.55400000000000005},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.5629999999999999,"15min":1.5409999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.58099999999999996},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.2890000000000001},"last":1.6699999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.472,"15min":1.472},"min":{"1min":0.754,"5min":0.72899999999999998,"15min":0.60799999999999998},"max":{"1min":1.9450000000000001,"5min":1.948,"15min":3.1219999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.4299999999999999,"15min":1.413},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.61799999999999999},"max":{"1min":1.98,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.4359999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42384,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329456,"statfs":{"total":96636764160,"available":96593362944,"internally_reserved":0,"allocated":2953216,"data_stored":1311560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83499999999999996,"15min":0.83199999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":1.3759999999999999,"5min":1.5820000000000001,"15min":3.181},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.93700000000000006,"15min":0.91200000000000003},"min":{"1min":0.55400000000000005,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.288,"5min":1.5349999999999999,"15min":1.7310000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.79100000000000004},"min":{"1min":0.40699999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.452,"5min":1.452,"15min":2.101},"last":0.622},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92100000000000004,"15min":0.92600000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.496,"5min":1.667,"15min":3.3260000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.88400000000000001,"15min":0.88100000000000001},"min":{"1min":0.57999999999999996,"5min":0.443,"15min":0.39900000000000002},"max":{"1min":1.7789999999999999,"5min":1.976,"15min":2.3700000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45200000000000001},"max":{"1min":1.5669999999999999,"5min":1.665,"15min":3.2480000000000002},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2629999999999999,"15min":1.2490000000000001},"min":{"1min":0.81100000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5009999999999999,"5min":1.845,"15min":2.1840000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.2270000000000001,"15min":1.212},"min":{"1min":0.78400000000000003,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.835,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.105}]},{"osd":5,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2589999999999999,"15min":1.2549999999999999},"min":{"1min":0.83199999999999996,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.8759999999999999,"5min":2.1230000000000002,"15min":3.4399999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.77600000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.883,"5min":1.883,"15min":1.992},"last":1.077}]},{"osd":6,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.82099999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9790000000000001,"5min":2.2549999999999999,"15min":3.5030000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.234,"15min":1.238},"min":{"1min":0.82199999999999995,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6870000000000001,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":0.91200000000000003}]},{"osd":7,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3049999999999999,"15min":1.296},"min":{"1min":0.878,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1030000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3089999999999999,"15min":1.298},"min":{"1min":0.83499999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.9330000000000001,"5min":3.0030000000000001,"15min":3.331},"last":1.1839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575080,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42400,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329440,"statfs":{"total":96636764160,"available":96593346560,"internally_reserved":0,"allocated":2969600,"data_stored":1313686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.76900000000000002,"15min":0.76100000000000001},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":1.5429999999999999,"15min":1.601},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81399999999999995,"15min":0.84999999999999998},"min":{"1min":0.34000000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.474,"5min":1.4810000000000001,"15min":1.782},"last":1.0880000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.55000000000000004,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.3049999999999999,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":1.002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96299999999999997,"15min":0.94899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":1.625,"5min":1.6699999999999999,"15min":1.788},"last":1.343}]},{"osd":3,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.88700000000000001,"15min":0.85899999999999999},"min":{"1min":0.54400000000000004,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":1.2989999999999999,"5min":1.633,"15min":1.6850000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.02,"15min":1.0129999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.41399999999999998},"max":{"1min":1.5740000000000001,"5min":1.948,"15min":1.948},"last":1.137}]},{"osd":4,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.53700000000000003},"max":{"1min":1.6519999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.587},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.246,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.78700000000000003,"15min":0.745},"max":{"1min":1.619,"5min":1.8200000000000001,"15min":1.921},"last":1.431}]},{"osd":5,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.222,"15min":1.2290000000000001},"min":{"1min":0.82799999999999996,"5min":0.753,"15min":0.61499999999999999},"max":{"1min":1.538,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.296},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.252,"15min":1.2310000000000001},"min":{"1min":0.77400000000000002,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.5980000000000001,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":1.4750000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.202,"15min":1.1919999999999999},"min":{"1min":0.82099999999999995,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.6559999999999999,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":1.518},{"interface":"front","average":{"1min":1.244,"5min":1.3129999999999999,"15min":1.3049999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.65600000000000003},"max":{"1min":1.6779999999999999,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.298,"15min":1.3029999999999999},"min":{"1min":0.875,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.714,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3069999999999999,"15min":1.2829999999999999},"min":{"1min":0.91900000000000004,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.865,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1919999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42396,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329444,"statfs":{"total":96636764160,"available":96593350656,"internally_reserved":0,"allocated":2957312,"data_stored":1311594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.86699999999999999,"15min":0.82499999999999996},"min":{"1min":0.47999999999999998,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":1.4490000000000001,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.94699999999999995,"15min":0.91800000000000004},"min":{"1min":0.52500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.7130000000000001,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":1.1359999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81899999999999995,"15min":0.81499999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":1.3380000000000001,"5min":1.823,"15min":1.861},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.93100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.754,"5min":1.754,"15min":1.925},"last":0.76600000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0609999999999999,"15min":1.0149999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7569999999999999,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.91900000000000004,"15min":0.92300000000000004},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.3879999999999999},"last":0.81200000000000006}]},{"osd":4,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.351,"15min":1.2949999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.65300000000000002},"max":{"1min":1.9099999999999999,"5min":1.984,"15min":2.028},"last":1.222},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3029999999999999,"15min":1.27},"min":{"1min":0.96699999999999997,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.8180000000000001,"5min":2.0750000000000002,"15min":2.141},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.278,"15min":1.256},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8260000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2450000000000001,"15min":1.2170000000000001},"min":{"1min":0.77600000000000002,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.8939999999999999,"5min":2.1499999999999999,"15min":2.246},"last":1.1850000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2649999999999999,"15min":1.2829999999999999},"min":{"1min":0.77000000000000002,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.9710000000000001,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.056},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3049999999999999,"15min":1.2849999999999999},"min":{"1min":0.86099999999999999,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":2.0379999999999998,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.3560000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.3240000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.59099999999999997},"max":{"1min":1.8029999999999999,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.264},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.3700000000000001,"15min":1.343},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.653},"last":1.0920000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42968,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328872,"statfs":{"total":96636764160,"available":96592764928,"internally_reserved":0,"allocated":3547136,"data_stored":1901908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72999999999999998},"min":{"1min":0.45800000000000002,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.97799999999999998,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83399999999999996,"15min":0.82899999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35899999999999999},"max":{"1min":1.466,"5min":1.607,"15min":2.3620000000000001},"last":1.089}]},{"osd":2,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.01,"15min":1.0069999999999999},"min":{"1min":0.55800000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.635,"5min":1.663,"15min":1.893},"last":1.758},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.879,"15min":0.86599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39400000000000002},"max":{"1min":1.482,"5min":1.671,"15min":2.4990000000000001},"last":1.272}]},{"osd":3,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94499999999999995,"15min":0.95999999999999996},"min":{"1min":0.45900000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.3819999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.405},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85599999999999998,"15min":0.88100000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":1.333,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":1.6080000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.228,"15min":1.212},"min":{"1min":0.84599999999999997,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.7749999999999999,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1000000000000001,"15min":1.105},"min":{"1min":0.76100000000000001,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.5600000000000001,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":1.464}]},{"osd":5,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1359999999999999,"15min":1.1459999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.587,"5min":1.667,"15min":2.6850000000000001},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.135,"15min":1.159},"min":{"1min":0.71199999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6040000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.7290000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.252,"15min":1.2609999999999999},"min":{"1min":0.80500000000000005,"5min":0.73999999999999999,"15min":0.71099999999999997},"max":{"1min":1.7310000000000001,"5min":1.8560000000000001,"15min":2.052},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.244,"15min":1.252},"min":{"1min":0.72999999999999998,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.6919999999999999,"5min":1.742,"15min":2.823},"last":1.873}]},{"osd":7,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.304,"15min":1.3320000000000001},"min":{"1min":0.96699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7809999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.747},"max":{"1min":1.8520000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.9199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:23:56.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:23:56.210 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:23:56.185+0000 7f8915cf5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:23:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.426+0000 7feb96423640 1 Processor -- start 2023-11-24T08:23:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.426+0000 7feb96423640 1 -- start start 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.427+0000 7feb96423640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb90146fd0 0x7feb9006c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb96423640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90070c50 con 0x7feb900b43d0 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb96423640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90070f30 con 0x7feb9006cb20 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb96423640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9006c1c0 con 0x7feb90146fd0 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb8ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb90146fd0 0x7feb9006c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb8ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb90146fd0 0x7feb9006c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57150/0 (socket says 172.21.15.107:57150) 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb949c0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feb900b43d0 0x7feb900b4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44102/0 (socket says 172.21.15.107:44102) 2023-11-24T08:23:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.428+0000 7feb8ffff640 1 -- 172.21.15.107:0/2489351713 learned_addr learned my addr 172.21.15.107:0/2489351713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.429+0000 7feb8ffff640 1 -- 172.21.15.107:0/2489351713 >> v1:172.21.15.120:6789/0 conn(0x7feb9006cb20 legacy=0x7feb9015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.429+0000 7feb8ffff640 1 -- 172.21.15.107:0/2489351713 >> v1:172.21.15.107:6789/0 conn(0x7feb900b43d0 legacy=0x7feb900b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.429+0000 7feb8ffff640 1 -- 172.21.15.107:0/2489351713 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9006bb00 con 0x7feb90146fd0 2023-11-24T08:23:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.430+0000 7feb8ffff640 1 --2- 172.21.15.107:0/2489351713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb90146fd0 0x7feb9006c5e0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7feb80009e50 tx=0x7feb8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c8f4e72f6b5fd4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.430+0000 7feb8effd640 1 -- 172.21.15.107:0/2489351713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8002ca80 con 0x7feb90146fd0 2023-11-24T08:23:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.430+0000 7feb8effd640 1 -- 172.21.15.107:0/2489351713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8002cbe0 con 0x7feb90146fd0 2023-11-24T08:23:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.430+0000 7feb8effd640 1 -- 172.21.15.107:0/2489351713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800343f0 con 0x7feb90146fd0 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 -- 172.21.15.107:0/2489351713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb90146fd0 msgr2=0x7feb9006c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 --2- 172.21.15.107:0/2489351713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb90146fd0 0x7feb9006c5e0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7feb80009e50 tx=0x7feb8002bee0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 -- 172.21.15.107:0/2489351713 shutdown_connections 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 --2- 172.21.15.107:0/2489351713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb90146fd0 0x7feb9006c5e0 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 -- 172.21.15.107:0/2489351713 >> 172.21.15.107:0/2489351713 conn(0x7feb9006e8a0 msgr2=0x7feb9006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 -- 172.21.15.107:0/2489351713 shutdown_connections 2023-11-24T08:23:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.431+0000 7feb96423640 1 -- 172.21.15.107:0/2489351713 wait complete. 2023-11-24T08:23:56.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.433+0000 7feb96423640 1 Processor -- start 2023-11-24T08:23:56.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.433+0000 7feb96423640 1 -- start start 2023-11-24T08:23:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 0x7feb90137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb90146fd0 0x7feb9015cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90134660 con 0x7feb900b43d0 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90137010 con 0x7feb9006cb20 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb96423640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb900cb440 con 0x7feb90146fd0 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57156/0 (socket says 172.21.15.107:57156) 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.434+0000 7feb8f7fe640 1 -- 172.21.15.107:0/4049419103 learned_addr learned my addr 172.21.15.107:0/4049419103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8ffff640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 0x7feb90137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb949c0640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb90146fd0 0x7feb9015cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 -- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb90146fd0 msgr2=0x7feb9015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb90146fd0 0x7feb9015cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 -- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 msgr2=0x7feb90137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 0x7feb90137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 -- 172.21.15.107:0/4049419103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80009b30 con 0x7feb900b43d0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8ffff640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 0x7feb90137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8f7fe640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7feb7800c400 tx=0x7feb7800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb780052d0 con 0x7feb900b43d0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb78005430 con 0x7feb900b43d0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.435+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb900cb720 con 0x7feb900b43d0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.436+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78003830 con 0x7feb900b43d0 2023-11-24T08:23:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.436+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb900cbce0 con 0x7feb900b43d0 2023-11-24T08:23:56.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.437+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb901347d0 con 0x7feb900b43d0 2023-11-24T08:23:56.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.438+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7feb78003990 con 0x7feb900b43d0 2023-11-24T08:23:56.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.439+0000 7feb8cff9640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 0x7feb7006f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.439+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb78087e00 con 0x7feb900b43d0 2023-11-24T08:23:56.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.439+0000 7feb8ffff640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 0x7feb7006f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.440+0000 7feb8ffff640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 0x7feb7006f850 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7feb80035000 tx=0x7feb80030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.443+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb901347d0 con 0x7feb900b43d0 2023-11-24T08:23:56.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.614+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb9015df10 con 0x7feb7006d390 2023-11-24T08:23:56.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.623+0000 7feb8cff9640 1 -- 172.21.15.107:0/4049419103 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287785 (secure 0 0 0) 0x7feb800344f0 con 0x7feb7006d390 2023-11-24T08:23:56.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:56.630 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 msgr2=0x7feb7006f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 0x7feb7006f850 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7feb80035000 tx=0x7feb80030000 comp rx=0 tx=0).stop 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 msgr2=0x7feb900cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7feb7800c400 tx=0x7feb7800c8c0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 shutdown_connections 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feb7006d390 0x7feb7006f850 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb90146fd0 0x7feb9015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb900b43d0 0x7feb900cf070 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 --2- 172.21.15.107:0/4049419103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9006cb20 0x7feb90137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 >> 172.21.15.107:0/4049419103 conn(0x7feb9006e8a0 msgr2=0x7feb9015e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 shutdown_connections 2023-11-24T08:23:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.629+0000 7feb96423640 1 -- 172.21.15.107:0/4049419103 wait complete. 2023-11-24T08:23:56.648 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2023-11-24T08:23:55.794949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336168,"kb_used_data":24300,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754638552,"statfs":{"total":773094113280,"available":772749877248,"internally_reserved":0,"allocated":24883200,"data_stored":11696216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":319213118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001541"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743778+0000","last_change":"2023-11-24T08:23:51.760223+0000","last_active":"2023-11-24T08:23:52.743778+0000","last_peered":"2023-11-24T08:23:52.743778+0000","last_clean":"2023-11-24T08:23:52.743778+0000","last_became_active":"2023-11-24T08:23:51.759936+0000","last_became_peered":"2023-11-24T08:23:51.759936+0000","last_unstale":"2023-11-24T08:23:52.743778+0000","last_undegraded":"2023-11-24T08:23:52.743778+0000","last_fullsized":"2023-11-24T08:23:52.743778+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:15.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436533+0000","last_change":"2023-11-24T08:23:51.748473+0000","last_active":"2023-11-24T08:23:53.436533+0000","last_peered":"2023-11-24T08:23:53.436533+0000","last_clean":"2023-11-24T08:23:53.436533+0000","last_became_active":"2023-11-24T08:23:51.747482+0000","last_became_peered":"2023-11-24T08:23:51.747482+0000","last_unstale":"2023-11-24T08:23:53.436533+0000","last_undegraded":"2023-11-24T08:23:53.436533+0000","last_fullsized":"2023-11-24T08:23:53.436533+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:00:42.129811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740801+0000","last_change":"2023-11-24T08:23:51.741946+0000","last_active":"2023-11-24T08:23:52.740801+0000","last_peered":"2023-11-24T08:23:52.740801+0000","last_clean":"2023-11-24T08:23:52.740801+0000","last_became_active":"2023-11-24T08:23:51.741639+0000","last_became_peered":"2023-11-24T08:23:51.741639+0000","last_unstale":"2023-11-24T08:23:52.740801+0000","last_undegraded":"2023-11-24T08:23:52.740801+0000","last_fullsized":"2023-11-24T08:23:52.740801+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:08.236700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453254+0000","last_change":"2023-11-24T08:23:51.743159+0000","last_active":"2023-11-24T08:23:53.453254+0000","last_peered":"2023-11-24T08:23:53.453254+0000","last_clean":"2023-11-24T08:23:53.453254+0000","last_became_active":"2023-11-24T08:23:51.742877+0000","last_became_peered":"2023-11-24T08:23:51.742877+0000","last_unstale":"2023-11-24T08:23:53.453254+0000","last_undegraded":"2023-11-24T08:23:53.453254+0000","last_fullsized":"2023-11-24T08:23:53.453254+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:15:23.012137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450767+0000","last_change":"2023-11-24T08:23:51.746320+0000","last_active":"2023-11-24T08:23:53.450767+0000","last_peered":"2023-11-24T08:23:53.450767+0000","last_clean":"2023-11-24T08:23:53.450767+0000","last_became_active":"2023-11-24T08:23:51.745895+0000","last_became_peered":"2023-11-24T08:23:51.745895+0000","last_unstale":"2023-11-24T08:23:53.450767+0000","last_undegraded":"2023-11-24T08:23:53.450767+0000","last_fullsized":"2023-11-24T08:23:53.450767+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:06.385572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451925+0000","last_change":"2023-11-24T08:23:51.749477+0000","last_active":"2023-11-24T08:23:53.451925+0000","last_peered":"2023-11-24T08:23:53.451925+0000","last_clean":"2023-11-24T08:23:53.451925+0000","last_became_active":"2023-11-24T08:23:51.749187+0000","last_became_peered":"2023-11-24T08:23:51.749187+0000","last_unstale":"2023-11-24T08:23:53.451925+0000","last_undegraded":"2023-11-24T08:23:53.451925+0000","last_fullsized":"2023-11-24T08:23:53.451925+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:03.867689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.739516+0000","last_change":"2023-11-24T08:23:51.743897+0000","last_active":"2023-11-24T08:23:52.739516+0000","last_peered":"2023-11-24T08:23:52.739516+0000","last_clean":"2023-11-24T08:23:52.739516+0000","last_became_active":"2023-11-24T08:23:51.743623+0000","last_became_peered":"2023-11-24T08:23:51.743623+0000","last_unstale":"2023-11-24T08:23:52.739516+0000","last_undegraded":"2023-11-24T08:23:52.739516+0000","last_fullsized":"2023-11-24T08:23:52.739516+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:37.880842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452373+0000","last_change":"2023-11-24T08:23:51.734684+0000","last_active":"2023-11-24T08:23:53.452373+0000","last_peered":"2023-11-24T08:23:53.452373+0000","last_clean":"2023-11-24T08:23:53.452373+0000","last_became_active":"2023-11-24T08:23:51.734394+0000","last_became_peered":"2023-11-24T08:23:51.734394+0000","last_unstale":"2023-11-24T08:23:53.452373+0000","last_undegraded":"2023-11-24T08:23:53.452373+0000","last_fullsized":"2023-11-24T08:23:53.452373+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:14.968654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452595+0000","last_change":"2023-11-24T08:23:51.751526+0000","last_active":"2023-11-24T08:23:53.452595+0000","last_peered":"2023-11-24T08:23:53.452595+0000","last_clean":"2023-11-24T08:23:53.452595+0000","last_became_active":"2023-11-24T08:23:51.751240+0000","last_became_peered":"2023-11-24T08:23:51.751240+0000","last_unstale":"2023-11-24T08:23:53.452595+0000","last_undegraded":"2023-11-24T08:23:53.452595+0000","last_fullsized":"2023-11-24T08:23:53.452595+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:15:24.535237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950850+0000","last_change":"2023-11-24T08:23:51.739001+0000","last_active":"2023-11-24T08:23:52.950850+0000","last_peered":"2023-11-24T08:23:52.950850+0000","last_clean":"2023-11-24T08:23:52.950850+0000","last_became_active":"2023-11-24T08:23:51.738734+0000","last_became_peered":"2023-11-24T08:23:51.738734+0000","last_unstale":"2023-11-24T08:23:52.950850+0000","last_undegraded":"2023-11-24T08:23:52.950850+0000","last_fullsized":"2023-11-24T08:23:52.950850+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:44.975136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743558+0000","last_change":"2023-11-24T08:23:51.747955+0000","last_active":"2023-11-24T08:23:52.743558+0000","last_peered":"2023-11-24T08:23:52.743558+0000","last_clean":"2023-11-24T08:23:52.743558+0000","last_became_active":"2023-11-24T08:23:51.747685+0000","last_became_peered":"2023-11-24T08:23:51.747685+0000","last_unstale":"2023-11-24T08:23:52.743558+0000","last_undegraded":"2023-11-24T08:23:52.743558+0000","last_fullsized":"2023-11-24T08:23:52.743558+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:16.740605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452650+0000","last_change":"2023-11-24T08:23:51.752043+0000","last_active":"2023-11-24T08:23:53.452650+0000","last_peered":"2023-11-24T08:23:53.452650+0000","last_clean":"2023-11-24T08:23:53.452650+0000","last_became_active":"2023-11-24T08:23:51.750768+0000","last_became_peered":"2023-11-24T08:23:51.750768+0000","last_unstale":"2023-11-24T08:23:53.452650+0000","last_undegraded":"2023-11-24T08:23:53.452650+0000","last_fullsized":"2023-11-24T08:23:53.452650+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:07:59.485127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453276+0000","last_change":"2023-11-24T08:23:51.745029+0000","last_active":"2023-11-24T08:23:53.453276+0000","last_peered":"2023-11-24T08:23:53.453276+0000","last_clean":"2023-11-24T08:23:53.453276+0000","last_became_active":"2023-11-24T08:23:51.744034+0000","last_became_peered":"2023-11-24T08:23:51.744034+0000","last_unstale":"2023-11-24T08:23:53.453276+0000","last_undegraded":"2023-11-24T08:23:53.453276+0000","last_fullsized":"2023-11-24T08:23:53.453276+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:58:33.706450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435745+0000","last_change":"2023-11-24T08:23:51.752793+0000","last_active":"2023-11-24T08:23:53.435745+0000","last_peered":"2023-11-24T08:23:53.435745+0000","last_clean":"2023-11-24T08:23:53.435745+0000","last_became_active":"2023-11-24T08:23:51.752451+0000","last_became_peered":"2023-11-24T08:23:51.752451+0000","last_unstale":"2023-11-24T08:23:53.435745+0000","last_undegraded":"2023-11-24T08:23:53.435745+0000","last_fullsized":"2023-11-24T08:23:53.435745+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:42:38.009798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435311+0000","last_change":"2023-11-24T08:23:51.751343+0000","last_active":"2023-11-24T08:23:53.435311+0000","last_peered":"2023-11-24T08:23:53.435311+0000","last_clean":"2023-11-24T08:23:53.435311+0000","last_became_active":"2023-11-24T08:23:51.751028+0000","last_became_peered":"2023-11-24T08:23:51.751028+0000","last_unstale":"2023-11-24T08:23:53.435311+0000","last_undegraded":"2023-11-24T08:23:53.435311+0000","last_fullsized":"2023-11-24T08:23:53.435311+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:08.960033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950380+0000","last_change":"2023-11-24T08:23:51.738190+0000","last_active":"2023-11-24T08:23:52.950380+0000","last_peered":"2023-11-24T08:23:52.950380+0000","last_clean":"2023-11-24T08:23:52.950380+0000","last_became_active":"2023-11-24T08:23:51.737832+0000","last_became_peered":"2023-11-24T08:23:51.737832+0000","last_unstale":"2023-11-24T08:23:52.950380+0000","last_undegraded":"2023-11-24T08:23:52.950380+0000","last_fullsized":"2023-11-24T08:23:52.950380+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:57:16.681306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950988+0000","last_change":"2023-11-24T08:23:51.746317+0000","last_active":"2023-11-24T08:23:52.950988+0000","last_peered":"2023-11-24T08:23:52.950988+0000","last_clean":"2023-11-24T08:23:52.950988+0000","last_became_active":"2023-11-24T08:23:51.745703+0000","last_became_peered":"2023-11-24T08:23:51.745703+0000","last_unstale":"2023-11-24T08:23:52.950988+0000","last_undegraded":"2023-11-24T08:23:52.950988+0000","last_fullsized":"2023-11-24T08:23:52.950988+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:59.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169836+0000","last_change":"2023-11-24T08:23:51.740685+0000","last_active":"2023-11-24T08:23:53.169836+0000","last_peered":"2023-11-24T08:23:53.169836+0000","last_clean":"2023-11-24T08:23:53.169836+0000","last_became_active":"2023-11-24T08:23:51.740309+0000","last_became_peered":"2023-11-24T08:23:51.740309+0000","last_unstale":"2023-11-24T08:23:53.169836+0000","last_undegraded":"2023-11-24T08:23:53.169836+0000","last_fullsized":"2023-11-24T08:23:53.169836+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:44.375189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950995+0000","last_change":"2023-11-24T08:23:51.748036+0000","last_active":"2023-11-24T08:23:52.950995+0000","last_peered":"2023-11-24T08:23:52.950995+0000","last_clean":"2023-11-24T08:23:52.950995+0000","last_became_active":"2023-11-24T08:23:51.747544+0000","last_became_peered":"2023-11-24T08:23:51.747544+0000","last_unstale":"2023-11-24T08:23:52.950995+0000","last_undegraded":"2023-11-24T08:23:52.950995+0000","last_fullsized":"2023-11-24T08:23:52.950995+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:28:52.021241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452853+0000","last_change":"2023-11-24T08:23:51.749777+0000","last_active":"2023-11-24T08:23:53.452853+0000","last_peered":"2023-11-24T08:23:53.452853+0000","last_clean":"2023-11-24T08:23:53.452853+0000","last_became_active":"2023-11-24T08:23:51.749495+0000","last_became_peered":"2023-11-24T08:23:51.749495+0000","last_unstale":"2023-11-24T08:23:53.452853+0000","last_undegraded":"2023-11-24T08:23:53.452853+0000","last_fullsized":"2023-11-24T08:23:53.452853+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:49.785657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450841+0000","last_change":"2023-11-24T08:23:51.738083+0000","last_active":"2023-11-24T08:23:53.450841+0000","last_peered":"2023-11-24T08:23:53.450841+0000","last_clean":"2023-11-24T08:23:53.450841+0000","last_became_active":"2023-11-24T08:23:51.737807+0000","last_became_peered":"2023-11-24T08:23:51.737807+0000","last_unstale":"2023-11-24T08:23:53.450841+0000","last_undegraded":"2023-11-24T08:23:53.450841+0000","last_fullsized":"2023-11-24T08:23:53.450841+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:01:09.972139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743015+0000","last_change":"2023-11-24T08:23:51.759704+0000","last_active":"2023-11-24T08:23:52.743015+0000","last_peered":"2023-11-24T08:23:52.743015+0000","last_clean":"2023-11-24T08:23:52.743015+0000","last_became_active":"2023-11-24T08:23:51.759271+0000","last_became_peered":"2023-11-24T08:23:51.759271+0000","last_unstale":"2023-11-24T08:23:52.743015+0000","last_undegraded":"2023-11-24T08:23:52.743015+0000","last_fullsized":"2023-11-24T08:23:52.743015+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:05:39.390596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435525+0000","last_change":"2023-11-24T08:23:51.750543+0000","last_active":"2023-11-24T08:23:53.435525+0000","last_peered":"2023-11-24T08:23:53.435525+0000","last_clean":"2023-11-24T08:23:53.435525+0000","last_became_active":"2023-11-24T08:23:51.747525+0000","last_became_peered":"2023-11-24T08:23:51.747525+0000","last_unstale":"2023-11-24T08:23:53.435525+0000","last_undegraded":"2023-11-24T08:23:53.435525+0000","last_fullsized":"2023-11-24T08:23:53.435525+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:46:04.853199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.741955+0000","last_change":"2023-11-24T08:23:51.751055+0000","last_active":"2023-11-24T08:23:52.741955+0000","last_peered":"2023-11-24T08:23:52.741955+0000","last_clean":"2023-11-24T08:23:52.741955+0000","last_became_active":"2023-11-24T08:23:51.749957+0000","last_became_peered":"2023-11-24T08:23:51.749957+0000","last_unstale":"2023-11-24T08:23:52.741955+0000","last_undegraded":"2023-11-24T08:23:52.741955+0000","last_fullsized":"2023-11-24T08:23:52.741955+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:38.470361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740982+0000","last_change":"2023-11-24T08:23:51.747921+0000","last_active":"2023-11-24T08:23:52.740982+0000","last_peered":"2023-11-24T08:23:52.740982+0000","last_clean":"2023-11-24T08:23:52.740982+0000","last_became_active":"2023-11-24T08:23:51.747568+0000","last_became_peered":"2023-11-24T08:23:51.747568+0000","last_unstale":"2023-11-24T08:23:52.740982+0000","last_undegraded":"2023-11-24T08:23:52.740982+0000","last_fullsized":"2023-11-24T08:23:52.740982+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:15.786481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436579+0000","last_change":"2023-11-24T08:23:51.749672+0000","last_active":"2023-11-24T08:23:53.436579+0000","last_peered":"2023-11-24T08:23:53.436579+0000","last_clean":"2023-11-24T08:23:53.436579+0000","last_became_active":"2023-11-24T08:23:51.749404+0000","last_became_peered":"2023-11-24T08:23:51.749404+0000","last_unstale":"2023-11-24T08:23:53.436579+0000","last_undegraded":"2023-11-24T08:23:53.436579+0000","last_fullsized":"2023-11-24T08:23:53.436579+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:38:15.676270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743689+0000","last_change":"2023-11-24T08:23:51.754562+0000","last_active":"2023-11-24T08:23:52.743689+0000","last_peered":"2023-11-24T08:23:52.743689+0000","last_clean":"2023-11-24T08:23:52.743689+0000","last_became_active":"2023-11-24T08:23:51.753797+0000","last_became_peered":"2023-11-24T08:23:51.753797+0000","last_unstale":"2023-11-24T08:23:52.743689+0000","last_undegraded":"2023-11-24T08:23:52.743689+0000","last_fullsized":"2023-11-24T08:23:52.743689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:31.064463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742996+0000","last_change":"2023-11-24T08:23:51.760228+0000","last_active":"2023-11-24T08:23:52.742996+0000","last_peered":"2023-11-24T08:23:52.742996+0000","last_clean":"2023-11-24T08:23:52.742996+0000","last_became_active":"2023-11-24T08:23:51.759891+0000","last_became_peered":"2023-11-24T08:23:51.759891+0000","last_unstale":"2023-11-24T08:23:52.742996+0000","last_undegraded":"2023-11-24T08:23:52.742996+0000","last_fullsized":"2023-11-24T08:23:52.742996+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:10:49.559926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452016+0000","last_change":"2023-11-24T08:23:51.750778+0000","last_active":"2023-11-24T08:23:53.452016+0000","last_peered":"2023-11-24T08:23:53.452016+0000","last_clean":"2023-11-24T08:23:53.452016+0000","last_became_active":"2023-11-24T08:23:51.750497+0000","last_became_peered":"2023-11-24T08:23:51.750497+0000","last_unstale":"2023-11-24T08:23:53.452016+0000","last_undegraded":"2023-11-24T08:23:53.452016+0000","last_fullsized":"2023-11-24T08:23:53.452016+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:59.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435832+0000","last_change":"2023-11-24T08:23:51.750754+0000","last_active":"2023-11-24T08:23:53.435832+0000","last_peered":"2023-11-24T08:23:53.435832+0000","last_clean":"2023-11-24T08:23:53.435832+0000","last_became_active":"2023-11-24T08:23:51.747916+0000","last_became_peered":"2023-11-24T08:23:51.747916+0000","last_unstale":"2023-11-24T08:23:53.435832+0000","last_undegraded":"2023-11-24T08:23:53.435832+0000","last_fullsized":"2023-11-24T08:23:53.435832+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:18:13.601471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435246+0000","last_change":"2023-11-24T08:23:51.746923+0000","last_active":"2023-11-24T08:23:53.435246+0000","last_peered":"2023-11-24T08:23:53.435246+0000","last_clean":"2023-11-24T08:23:53.435246+0000","last_became_active":"2023-11-24T08:23:51.746326+0000","last_became_peered":"2023-11-24T08:23:51.746326+0000","last_unstale":"2023-11-24T08:23:53.435246+0000","last_undegraded":"2023-11-24T08:23:53.435246+0000","last_fullsized":"2023-11-24T08:23:53.435246+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:09:49.785657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950996+0000","last_change":"2023-11-24T08:23:51.744211+0000","last_active":"2023-11-24T08:23:52.950996+0000","last_peered":"2023-11-24T08:23:52.950996+0000","last_clean":"2023-11-24T08:23:52.950996+0000","last_became_active":"2023-11-24T08:23:51.743943+0000","last_became_peered":"2023-11-24T08:23:51.743943+0000","last_unstale":"2023-11-24T08:23:52.950996+0000","last_undegraded":"2023-11-24T08:23:52.950996+0000","last_fullsized":"2023-11-24T08:23:52.950996+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:37:48.954662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"233'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169994+0000","last_change":"2023-11-24T08:23:50.745414+0000","last_active":"2023-11-24T08:23:53.169994+0000","last_peered":"2023-11-24T08:23:53.169994+0000","last_clean":"2023-11-24T08:23:53.169994+0000","last_became_active":"2023-11-24T08:23:50.744421+0000","last_became_peered":"2023-11-24T08:23:50.744421+0000","last_unstale":"2023-11-24T08:23:53.169994+0000","last_undegraded":"2023-11-24T08:23:53.169994+0000","last_fullsized":"2023-11-24T08:23:53.169994+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:09.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169438+0000","last_change":"2023-11-24T08:23:51.756431+0000","last_active":"2023-11-24T08:23:53.169438+0000","last_peered":"2023-11-24T08:23:53.169438+0000","last_clean":"2023-11-24T08:23:53.169438+0000","last_became_active":"2023-11-24T08:23:51.750896+0000","last_became_peered":"2023-11-24T08:23:51.750896+0000","last_unstale":"2023-11-24T08:23:53.169438+0000","last_undegraded":"2023-11-24T08:23:53.169438+0000","last_fullsized":"2023-11-24T08:23:53.169438+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:56:10.269318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950817+0000","last_change":"2023-11-24T08:23:50.742646+0000","last_active":"2023-11-24T08:23:52.950817+0000","last_peered":"2023-11-24T08:23:52.950817+0000","last_clean":"2023-11-24T08:23:52.950817+0000","last_became_active":"2023-11-24T08:23:50.742321+0000","last_became_peered":"2023-11-24T08:23:50.742321+0000","last_unstale":"2023-11-24T08:23:52.950817+0000","last_undegraded":"2023-11-24T08:23:52.950817+0000","last_fullsized":"2023-11-24T08:23:52.950817+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:53.045264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169979+0000","last_change":"2023-11-24T08:23:51.734522+0000","last_active":"2023-11-24T08:23:53.169979+0000","last_peered":"2023-11-24T08:23:53.169979+0000","last_clean":"2023-11-24T08:23:53.169979+0000","last_became_active":"2023-11-24T08:23:51.732052+0000","last_became_peered":"2023-11-24T08:23:51.732052+0000","last_unstale":"2023-11-24T08:23:53.169979+0000","last_undegraded":"2023-11-24T08:23:53.169979+0000","last_fullsized":"2023-11-24T08:23:53.169979+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:03:38.470361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743926+0000","last_change":"2023-11-24T08:23:50.737165+0000","last_active":"2023-11-24T08:23:52.743926+0000","last_peered":"2023-11-24T08:23:52.743926+0000","last_clean":"2023-11-24T08:23:52.743926+0000","last_became_active":"2023-11-24T08:23:50.735760+0000","last_became_peered":"2023-11-24T08:23:50.735760+0000","last_unstale":"2023-11-24T08:23:52.743926+0000","last_undegraded":"2023-11-24T08:23:52.743926+0000","last_fullsized":"2023-11-24T08:23:52.743926+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:43.358885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436592+0000","last_change":"2023-11-24T08:23:51.752929+0000","last_active":"2023-11-24T08:23:53.436592+0000","last_peered":"2023-11-24T08:23:53.436592+0000","last_clean":"2023-11-24T08:23:53.436592+0000","last_became_active":"2023-11-24T08:23:51.752643+0000","last_became_peered":"2023-11-24T08:23:51.752643+0000","last_unstale":"2023-11-24T08:23:53.436592+0000","last_undegraded":"2023-11-24T08:23:53.436592+0000","last_fullsized":"2023-11-24T08:23:53.436592+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:52.263816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950863+0000","last_change":"2023-11-24T08:23:50.743094+0000","last_active":"2023-11-24T08:23:52.950863+0000","last_peered":"2023-11-24T08:23:52.950863+0000","last_clean":"2023-11-24T08:23:52.950863+0000","last_became_active":"2023-11-24T08:23:50.740403+0000","last_became_peered":"2023-11-24T08:23:50.740403+0000","last_unstale":"2023-11-24T08:23:52.950863+0000","last_undegraded":"2023-11-24T08:23:52.950863+0000","last_fullsized":"2023-11-24T08:23:52.950863+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:33.307649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435376+0000","last_change":"2023-11-24T08:23:51.747086+0000","last_active":"2023-11-24T08:23:53.435376+0000","last_peered":"2023-11-24T08:23:53.435376+0000","last_clean":"2023-11-24T08:23:53.435376+0000","last_became_active":"2023-11-24T08:23:51.746646+0000","last_became_peered":"2023-11-24T08:23:51.746646+0000","last_unstale":"2023-11-24T08:23:53.435376+0000","last_undegraded":"2023-11-24T08:23:53.435376+0000","last_fullsized":"2023-11-24T08:23:53.435376+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:23:04.547941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451822+0000","last_change":"2023-11-24T08:23:50.743259+0000","last_active":"2023-11-24T08:23:53.451822+0000","last_peered":"2023-11-24T08:23:53.451822+0000","last_clean":"2023-11-24T08:23:53.451822+0000","last_became_active":"2023-11-24T08:23:50.741163+0000","last_became_peered":"2023-11-24T08:23:50.741163+0000","last_unstale":"2023-11-24T08:23:53.451822+0000","last_undegraded":"2023-11-24T08:23:53.451822+0000","last_fullsized":"2023-11-24T08:23:53.451822+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:20.259677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435881+0000","last_change":"2023-11-24T08:23:51.757722+0000","last_active":"2023-11-24T08:23:53.435881+0000","last_peered":"2023-11-24T08:23:53.435881+0000","last_clean":"2023-11-24T08:23:53.435881+0000","last_became_active":"2023-11-24T08:23:51.756041+0000","last_became_peered":"2023-11-24T08:23:51.756041+0000","last_unstale":"2023-11-24T08:23:53.435881+0000","last_undegraded":"2023-11-24T08:23:53.435881+0000","last_fullsized":"2023-11-24T08:23:53.435881+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:29:47.614952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435997+0000","last_change":"2023-11-24T08:23:50.735773+0000","last_active":"2023-11-24T08:23:53.435997+0000","last_peered":"2023-11-24T08:23:53.435997+0000","last_clean":"2023-11-24T08:23:53.435997+0000","last_became_active":"2023-11-24T08:23:50.735532+0000","last_became_peered":"2023-11-24T08:23:50.735532+0000","last_unstale":"2023-11-24T08:23:53.435997+0000","last_undegraded":"2023-11-24T08:23:53.435997+0000","last_fullsized":"2023-11-24T08:23:53.435997+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:43.958832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740340+0000","last_change":"2023-11-24T08:23:51.755193+0000","last_active":"2023-11-24T08:23:52.740340+0000","last_peered":"2023-11-24T08:23:52.740340+0000","last_clean":"2023-11-24T08:23:52.740340+0000","last_became_active":"2023-11-24T08:23:51.754865+0000","last_became_peered":"2023-11-24T08:23:51.754865+0000","last_unstale":"2023-11-24T08:23:52.740340+0000","last_undegraded":"2023-11-24T08:23:52.740340+0000","last_fullsized":"2023-11-24T08:23:52.740340+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:49:05.162558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950354+0000","last_change":"2023-11-24T08:23:50.747033+0000","last_active":"2023-11-24T08:23:52.950354+0000","last_peered":"2023-11-24T08:23:52.950354+0000","last_clean":"2023-11-24T08:23:52.950354+0000","last_became_active":"2023-11-24T08:23:50.746726+0000","last_became_peered":"2023-11-24T08:23:50.746726+0000","last_unstale":"2023-11-24T08:23:52.950354+0000","last_undegraded":"2023-11-24T08:23:52.950354+0000","last_fullsized":"2023-11-24T08:23:52.950354+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:01:08.955834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453634+0000","last_change":"2023-11-24T08:23:51.745227+0000","last_active":"2023-11-24T08:23:53.453634+0000","last_peered":"2023-11-24T08:23:53.453634+0000","last_clean":"2023-11-24T08:23:53.453634+0000","last_became_active":"2023-11-24T08:23:51.744752+0000","last_became_peered":"2023-11-24T08:23:51.744752+0000","last_unstale":"2023-11-24T08:23:53.453634+0000","last_undegraded":"2023-11-24T08:23:53.453634+0000","last_fullsized":"2023-11-24T08:23:53.453634+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:42:23.159738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450698+0000","last_change":"2023-11-24T08:23:50.742451+0000","last_active":"2023-11-24T08:23:53.450698+0000","last_peered":"2023-11-24T08:23:53.450698+0000","last_clean":"2023-11-24T08:23:53.450698+0000","last_became_active":"2023-11-24T08:23:50.742075+0000","last_became_peered":"2023-11-24T08:23:50.742075+0000","last_unstale":"2023-11-24T08:23:53.450698+0000","last_undegraded":"2023-11-24T08:23:53.450698+0000","last_fullsized":"2023-11-24T08:23:53.450698+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:52:24.053860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453443+0000","last_change":"2023-11-24T08:23:51.749332+0000","last_active":"2023-11-24T08:23:53.453443+0000","last_peered":"2023-11-24T08:23:53.453443+0000","last_clean":"2023-11-24T08:23:53.453443+0000","last_became_active":"2023-11-24T08:23:51.749083+0000","last_became_peered":"2023-11-24T08:23:51.749083+0000","last_unstale":"2023-11-24T08:23:53.453443+0000","last_undegraded":"2023-11-24T08:23:53.453443+0000","last_fullsized":"2023-11-24T08:23:53.453443+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:49:41.351199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453721+0000","last_change":"2023-11-24T08:23:50.738749+0000","last_active":"2023-11-24T08:23:53.453721+0000","last_peered":"2023-11-24T08:23:53.453721+0000","last_clean":"2023-11-24T08:23:53.453721+0000","last_became_active":"2023-11-24T08:23:50.738203+0000","last_became_peered":"2023-11-24T08:23:50.738203+0000","last_unstale":"2023-11-24T08:23:53.453721+0000","last_undegraded":"2023-11-24T08:23:53.453721+0000","last_fullsized":"2023-11-24T08:23:53.453721+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:39:43.164878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169770+0000","last_change":"2023-11-24T08:23:51.746034+0000","last_active":"2023-11-24T08:23:53.169770+0000","last_peered":"2023-11-24T08:23:53.169770+0000","last_clean":"2023-11-24T08:23:53.169770+0000","last_became_active":"2023-11-24T08:23:51.745777+0000","last_became_peered":"2023-11-24T08:23:51.745777+0000","last_unstale":"2023-11-24T08:23:53.169770+0000","last_undegraded":"2023-11-24T08:23:53.169770+0000","last_fullsized":"2023-11-24T08:23:53.169770+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:00:04.041443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.951157+0000","last_change":"2023-11-24T08:23:50.745757+0000","last_active":"2023-11-24T08:23:52.951157+0000","last_peered":"2023-11-24T08:23:52.951157+0000","last_clean":"2023-11-24T08:23:52.951157+0000","last_became_active":"2023-11-24T08:23:50.745472+0000","last_became_peered":"2023-11-24T08:23:50.745472+0000","last_unstale":"2023-11-24T08:23:52.951157+0000","last_undegraded":"2023-11-24T08:23:52.951157+0000","last_fullsized":"2023-11-24T08:23:52.951157+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:11:11.435717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452502+0000","last_change":"2023-11-24T08:23:51.742673+0000","last_active":"2023-11-24T08:23:53.452502+0000","last_peered":"2023-11-24T08:23:53.452502+0000","last_clean":"2023-11-24T08:23:53.452502+0000","last_became_active":"2023-11-24T08:23:51.742313+0000","last_became_peered":"2023-11-24T08:23:51.742313+0000","last_unstale":"2023-11-24T08:23:53.452502+0000","last_undegraded":"2023-11-24T08:23:53.452502+0000","last_fullsized":"2023-11-24T08:23:53.452502+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:57:21.409950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"233'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453398+0000","last_change":"2023-11-24T08:23:50.740617+0000","last_active":"2023-11-24T08:23:53.453398+0000","last_peered":"2023-11-24T08:23:53.453398+0000","last_clean":"2023-11-24T08:23:53.453398+0000","last_became_active":"2023-11-24T08:23:50.740303+0000","last_became_peered":"2023-11-24T08:23:50.740303+0000","last_unstale":"2023-11-24T08:23:53.453398+0000","last_undegraded":"2023-11-24T08:23:53.453398+0000","last_fullsized":"2023-11-24T08:23:53.453398+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:17:57.631797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743948+0000","last_change":"2023-11-24T08:23:51.760226+0000","last_active":"2023-11-24T08:23:52.743948+0000","last_peered":"2023-11-24T08:23:52.743948+0000","last_clean":"2023-11-24T08:23:52.743948+0000","last_became_active":"2023-11-24T08:23:51.759752+0000","last_became_peered":"2023-11-24T08:23:51.759752+0000","last_unstale":"2023-11-24T08:23:52.743948+0000","last_undegraded":"2023-11-24T08:23:52.743948+0000","last_fullsized":"2023-11-24T08:23:52.743948+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:38:52.586714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"233'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169356+0000","last_change":"2023-11-24T08:23:50.734037+0000","last_active":"2023-11-24T08:23:53.169356+0000","last_peered":"2023-11-24T08:23:53.169356+0000","last_clean":"2023-11-24T08:23:53.169356+0000","last_became_active":"2023-11-24T08:23:50.733725+0000","last_became_peered":"2023-11-24T08:23:50.733725+0000","last_unstale":"2023-11-24T08:23:53.169356+0000","last_undegraded":"2023-11-24T08:23:53.169356+0000","last_fullsized":"2023-11-24T08:23:53.169356+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:58:32.635233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450732+0000","last_change":"2023-11-24T08:23:51.749969+0000","last_active":"2023-11-24T08:23:53.450732+0000","last_peered":"2023-11-24T08:23:53.450732+0000","last_clean":"2023-11-24T08:23:53.450732+0000","last_became_active":"2023-11-24T08:23:51.749501+0000","last_became_peered":"2023-11-24T08:23:51.749501+0000","last_unstale":"2023-11-24T08:23:53.450732+0000","last_undegraded":"2023-11-24T08:23:53.450732+0000","last_fullsized":"2023-11-24T08:23:53.450732+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:35:24.707920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451523+0000","last_change":"2023-11-24T08:23:50.743196+0000","last_active":"2023-11-24T08:23:53.451523+0000","last_peered":"2023-11-24T08:23:53.451523+0000","last_clean":"2023-11-24T08:23:53.451523+0000","last_became_active":"2023-11-24T08:23:50.742851+0000","last_became_peered":"2023-11-24T08:23:50.742851+0000","last_unstale":"2023-11-24T08:23:53.451523+0000","last_undegraded":"2023-11-24T08:23:53.451523+0000","last_fullsized":"2023-11-24T08:23:53.451523+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:55:58.426446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742202+0000","last_change":"2023-11-24T08:23:51.754214+0000","last_active":"2023-11-24T08:23:52.742202+0000","last_peered":"2023-11-24T08:23:52.742202+0000","last_clean":"2023-11-24T08:23:52.742202+0000","last_became_active":"2023-11-24T08:23:51.753116+0000","last_became_peered":"2023-11-24T08:23:51.753116+0000","last_unstale":"2023-11-24T08:23:52.742202+0000","last_undegraded":"2023-11-24T08:23:52.742202+0000","last_fullsized":"2023-11-24T08:23:52.742202+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:14:33.553313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453116+0000","last_change":"2023-11-24T08:23:50.745479+0000","last_active":"2023-11-24T08:23:53.453116+0000","last_peered":"2023-11-24T08:23:53.453116+0000","last_clean":"2023-11-24T08:23:53.453116+0000","last_became_active":"2023-11-24T08:23:50.745166+0000","last_became_peered":"2023-11-24T08:23:50.745166+0000","last_unstale":"2023-11-24T08:23:53.453116+0000","last_undegraded":"2023-11-24T08:23:53.453116+0000","last_fullsized":"2023-11-24T08:23:53.453116+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:49.087604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451019+0000","last_change":"2023-11-24T08:23:51.757099+0000","last_active":"2023-11-24T08:23:53.451019+0000","last_peered":"2023-11-24T08:23:53.451019+0000","last_clean":"2023-11-24T08:23:53.451019+0000","last_became_active":"2023-11-24T08:23:51.756846+0000","last_became_peered":"2023-11-24T08:23:51.756846+0000","last_unstale":"2023-11-24T08:23:53.451019+0000","last_undegraded":"2023-11-24T08:23:53.451019+0000","last_fullsized":"2023-11-24T08:23:53.451019+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:18.755797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452540+0000","last_change":"2023-11-24T08:23:50.739261+0000","last_active":"2023-11-24T08:23:53.452540+0000","last_peered":"2023-11-24T08:23:53.452540+0000","last_clean":"2023-11-24T08:23:53.452540+0000","last_became_active":"2023-11-24T08:23:50.738972+0000","last_became_peered":"2023-11-24T08:23:50.738972+0000","last_unstale":"2023-11-24T08:23:53.452540+0000","last_undegraded":"2023-11-24T08:23:53.452540+0000","last_fullsized":"2023-11-24T08:23:53.452540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:39:26.180592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451170+0000","last_change":"2023-11-24T08:23:51.746135+0000","last_active":"2023-11-24T08:23:53.451170+0000","last_peered":"2023-11-24T08:23:53.451170+0000","last_clean":"2023-11-24T08:23:53.451170+0000","last_became_active":"2023-11-24T08:23:51.745547+0000","last_became_peered":"2023-11-24T08:23:51.745547+0000","last_unstale":"2023-11-24T08:23:53.451170+0000","last_undegraded":"2023-11-24T08:23:53.451170+0000","last_fullsized":"2023-11-24T08:23:53.451170+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:32.431718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"233'8","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452128+0000","last_change":"2023-11-24T08:23:50.737570+0000","last_active":"2023-11-24T08:23:53.452128+0000","last_peered":"2023-11-24T08:23:53.452128+0000","last_clean":"2023-11-24T08:23:53.452128+0000","last_became_active":"2023-11-24T08:23:50.737287+0000","last_became_peered":"2023-11-24T08:23:50.737287+0000","last_unstale":"2023-11-24T08:23:53.452128+0000","last_undegraded":"2023-11-24T08:23:53.452128+0000","last_fullsized":"2023-11-24T08:23:53.452128+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:41:06.196465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743055+0000","last_change":"2023-11-24T08:23:51.749623+0000","last_active":"2023-11-24T08:23:52.743055+0000","last_peered":"2023-11-24T08:23:52.743055+0000","last_clean":"2023-11-24T08:23:52.743055+0000","last_became_active":"2023-11-24T08:23:51.749348+0000","last_became_peered":"2023-11-24T08:23:51.749348+0000","last_unstale":"2023-11-24T08:23:52.743055+0000","last_undegraded":"2023-11-24T08:23:52.743055+0000","last_fullsized":"2023-11-24T08:23:52.743055+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:44.742316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452931+0000","last_change":"2023-11-24T08:23:50.747731+0000","last_active":"2023-11-24T08:23:53.452931+0000","last_peered":"2023-11-24T08:23:53.452931+0000","last_clean":"2023-11-24T08:23:53.452931+0000","last_became_active":"2023-11-24T08:23:50.746950+0000","last_became_peered":"2023-11-24T08:23:50.746950+0000","last_unstale":"2023-11-24T08:23:53.452931+0000","last_undegraded":"2023-11-24T08:23:53.452931+0000","last_fullsized":"2023-11-24T08:23:53.452931+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:13:53.045264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.743981+0000","last_change":"2023-11-24T08:23:51.753810+0000","last_active":"2023-11-24T08:23:52.743981+0000","last_peered":"2023-11-24T08:23:52.743981+0000","last_clean":"2023-11-24T08:23:52.743981+0000","last_became_active":"2023-11-24T08:23:51.753554+0000","last_became_peered":"2023-11-24T08:23:51.753554+0000","last_unstale":"2023-11-24T08:23:52.743981+0000","last_undegraded":"2023-11-24T08:23:52.743981+0000","last_fullsized":"2023-11-24T08:23:52.743981+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:54.413906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169980+0000","last_change":"2023-11-24T08:23:50.738359+0000","last_active":"2023-11-24T08:23:53.169980+0000","last_peered":"2023-11-24T08:23:53.169980+0000","last_clean":"2023-11-24T08:23:53.169980+0000","last_became_active":"2023-11-24T08:23:50.738060+0000","last_became_peered":"2023-11-24T08:23:50.738060+0000","last_unstale":"2023-11-24T08:23:53.169980+0000","last_undegraded":"2023-11-24T08:23:53.169980+0000","last_fullsized":"2023-11-24T08:23:53.169980+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:49.836574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453866+0000","last_change":"2023-11-24T08:23:51.747211+0000","last_active":"2023-11-24T08:23:53.453866+0000","last_peered":"2023-11-24T08:23:53.453866+0000","last_clean":"2023-11-24T08:23:53.453866+0000","last_became_active":"2023-11-24T08:23:51.746945+0000","last_became_peered":"2023-11-24T08:23:51.746945+0000","last_unstale":"2023-11-24T08:23:53.453866+0000","last_undegraded":"2023-11-24T08:23:53.453866+0000","last_fullsized":"2023-11-24T08:23:53.453866+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:13.855889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"233'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.741038+0000","last_change":"2023-11-24T08:23:50.740141+0000","last_active":"2023-11-24T08:23:52.741038+0000","last_peered":"2023-11-24T08:23:52.741038+0000","last_clean":"2023-11-24T08:23:52.741038+0000","last_became_active":"2023-11-24T08:23:50.739848+0000","last_became_peered":"2023-11-24T08:23:50.739848+0000","last_unstale":"2023-11-24T08:23:52.741038+0000","last_undegraded":"2023-11-24T08:23:52.741038+0000","last_fullsized":"2023-11-24T08:23:52.741038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:42.775374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.744035+0000","last_change":"2023-11-24T08:23:51.760036+0000","last_active":"2023-11-24T08:23:52.744035+0000","last_peered":"2023-11-24T08:23:52.744035+0000","last_clean":"2023-11-24T08:23:52.744035+0000","last_became_active":"2023-11-24T08:23:51.759528+0000","last_became_peered":"2023-11-24T08:23:51.759528+0000","last_unstale":"2023-11-24T08:23:52.744035+0000","last_undegraded":"2023-11-24T08:23:52.744035+0000","last_fullsized":"2023-11-24T08:23:52.744035+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:19.613655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169311+0000","last_change":"2023-11-24T08:23:50.744713+0000","last_active":"2023-11-24T08:23:53.169311+0000","last_peered":"2023-11-24T08:23:53.169311+0000","last_clean":"2023-11-24T08:23:53.169311+0000","last_became_active":"2023-11-24T08:23:50.744400+0000","last_became_peered":"2023-11-24T08:23:50.744400+0000","last_unstale":"2023-11-24T08:23:53.169311+0000","last_undegraded":"2023-11-24T08:23:53.169311+0000","last_fullsized":"2023-11-24T08:23:53.169311+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:05:11.049034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435807+0000","last_change":"2023-11-24T08:23:51.756290+0000","last_active":"2023-11-24T08:23:53.435807+0000","last_peered":"2023-11-24T08:23:53.435807+0000","last_clean":"2023-11-24T08:23:53.435807+0000","last_became_active":"2023-11-24T08:23:51.753588+0000","last_became_peered":"2023-11-24T08:23:51.753588+0000","last_unstale":"2023-11-24T08:23:53.435807+0000","last_undegraded":"2023-11-24T08:23:53.435807+0000","last_fullsized":"2023-11-24T08:23:53.435807+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:27:52.270610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451547+0000","last_change":"2023-11-24T08:23:50.741463+0000","last_active":"2023-11-24T08:23:53.451547+0000","last_peered":"2023-11-24T08:23:53.451547+0000","last_clean":"2023-11-24T08:23:53.451547+0000","last_became_active":"2023-11-24T08:23:50.740837+0000","last_became_peered":"2023-11-24T08:23:50.740837+0000","last_unstale":"2023-11-24T08:23:53.451547+0000","last_undegraded":"2023-11-24T08:23:53.451547+0000","last_fullsized":"2023-11-24T08:23:53.451547+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:06.503801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.742387+0000","last_change":"2023-11-24T08:23:51.751098+0000","last_active":"2023-11-24T08:23:52.742387+0000","last_peered":"2023-11-24T08:23:52.742387+0000","last_clean":"2023-11-24T08:23:52.742387+0000","last_became_active":"2023-11-24T08:23:51.750773+0000","last_became_peered":"2023-11-24T08:23:51.750773+0000","last_unstale":"2023-11-24T08:23:52.742387+0000","last_undegraded":"2023-11-24T08:23:52.742387+0000","last_fullsized":"2023-11-24T08:23:52.742387+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:03:15.959648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.450695+0000","last_change":"2023-11-24T08:23:50.739395+0000","last_active":"2023-11-24T08:23:53.450695+0000","last_peered":"2023-11-24T08:23:53.450695+0000","last_clean":"2023-11-24T08:23:53.450695+0000","last_became_active":"2023-11-24T08:23:50.739078+0000","last_became_peered":"2023-11-24T08:23:50.739078+0000","last_unstale":"2023-11-24T08:23:53.450695+0000","last_undegraded":"2023-11-24T08:23:53.450695+0000","last_fullsized":"2023-11-24T08:23:53.450695+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:37.400823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452849+0000","last_change":"2023-11-24T08:23:51.746079+0000","last_active":"2023-11-24T08:23:53.452849+0000","last_peered":"2023-11-24T08:23:53.452849+0000","last_clean":"2023-11-24T08:23:53.452849+0000","last_became_active":"2023-11-24T08:23:51.745802+0000","last_became_peered":"2023-11-24T08:23:51.745802+0000","last_unstale":"2023-11-24T08:23:53.452849+0000","last_undegraded":"2023-11-24T08:23:53.452849+0000","last_fullsized":"2023-11-24T08:23:53.452849+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:33:01.191648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436126+0000","last_change":"2023-11-24T08:23:50.734260+0000","last_active":"2023-11-24T08:23:53.436126+0000","last_peered":"2023-11-24T08:23:53.436126+0000","last_clean":"2023-11-24T08:23:53.436126+0000","last_became_active":"2023-11-24T08:23:50.732860+0000","last_became_peered":"2023-11-24T08:23:50.732860+0000","last_unstale":"2023-11-24T08:23:53.436126+0000","last_undegraded":"2023-11-24T08:23:53.436126+0000","last_fullsized":"2023-11-24T08:23:53.436126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:48:58.426911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435876+0000","last_change":"2023-11-24T08:23:51.748664+0000","last_active":"2023-11-24T08:23:53.435876+0000","last_peered":"2023-11-24T08:23:53.435876+0000","last_clean":"2023-11-24T08:23:53.435876+0000","last_became_active":"2023-11-24T08:23:51.742119+0000","last_became_peered":"2023-11-24T08:23:51.742119+0000","last_unstale":"2023-11-24T08:23:53.435876+0000","last_undegraded":"2023-11-24T08:23:53.435876+0000","last_fullsized":"2023-11-24T08:23:53.435876+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:23:22.852132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452389+0000","last_change":"2023-11-24T08:23:50.742621+0000","last_active":"2023-11-24T08:23:53.452389+0000","last_peered":"2023-11-24T08:23:53.452389+0000","last_clean":"2023-11-24T08:23:53.452389+0000","last_became_active":"2023-11-24T08:23:50.742332+0000","last_became_peered":"2023-11-24T08:23:50.742332+0000","last_unstale":"2023-11-24T08:23:53.452389+0000","last_undegraded":"2023-11-24T08:23:53.452389+0000","last_fullsized":"2023-11-24T08:23:53.452389+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:10.989887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452944+0000","last_change":"2023-11-24T08:23:51.751924+0000","last_active":"2023-11-24T08:23:53.452944+0000","last_peered":"2023-11-24T08:23:53.452944+0000","last_clean":"2023-11-24T08:23:53.452944+0000","last_became_active":"2023-11-24T08:23:51.751631+0000","last_became_peered":"2023-11-24T08:23:51.751631+0000","last_unstale":"2023-11-24T08:23:53.452944+0000","last_undegraded":"2023-11-24T08:23:53.452944+0000","last_fullsized":"2023-11-24T08:23:53.452944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:39:08.960033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"233'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453761+0000","last_change":"2023-11-24T08:23:50.745721+0000","last_active":"2023-11-24T08:23:53.453761+0000","last_peered":"2023-11-24T08:23:53.453761+0000","last_clean":"2023-11-24T08:23:53.453761+0000","last_became_active":"2023-11-24T08:23:50.743807+0000","last_became_peered":"2023-11-24T08:23:50.743807+0000","last_unstale":"2023-11-24T08:23:53.453761+0000","last_undegraded":"2023-11-24T08:23:53.453761+0000","last_fullsized":"2023-11-24T08:23:53.453761+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:08.387783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452312+0000","last_change":"2023-11-24T08:23:51.753149+0000","last_active":"2023-11-24T08:23:53.452312+0000","last_peered":"2023-11-24T08:23:53.452312+0000","last_clean":"2023-11-24T08:23:53.452312+0000","last_became_active":"2023-11-24T08:23:51.752892+0000","last_became_peered":"2023-11-24T08:23:51.752892+0000","last_unstale":"2023-11-24T08:23:53.452312+0000","last_undegraded":"2023-11-24T08:23:53.452312+0000","last_fullsized":"2023-11-24T08:23:53.452312+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:37:48.954662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.170133+0000","last_change":"2023-11-24T08:23:50.737842+0000","last_active":"2023-11-24T08:23:53.170133+0000","last_peered":"2023-11-24T08:23:53.170133+0000","last_clean":"2023-11-24T08:23:53.170133+0000","last_became_active":"2023-11-24T08:23:50.737598+0000","last_became_peered":"2023-11-24T08:23:50.737598+0000","last_unstale":"2023-11-24T08:23:53.170133+0000","last_undegraded":"2023-11-24T08:23:53.170133+0000","last_fullsized":"2023-11-24T08:23:53.170133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:15.291573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740797+0000","last_change":"2023-11-24T08:23:51.756579+0000","last_active":"2023-11-24T08:23:52.740797+0000","last_peered":"2023-11-24T08:23:52.740797+0000","last_clean":"2023-11-24T08:23:52.740797+0000","last_became_active":"2023-11-24T08:23:51.756331+0000","last_became_peered":"2023-11-24T08:23:51.756331+0000","last_unstale":"2023-11-24T08:23:52.740797+0000","last_undegraded":"2023-11-24T08:23:52.740797+0000","last_fullsized":"2023-11-24T08:23:52.740797+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:32:47.627900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740468+0000","last_change":"2023-11-24T08:23:50.738854+0000","last_active":"2023-11-24T08:23:52.740468+0000","last_peered":"2023-11-24T08:23:52.740468+0000","last_clean":"2023-11-24T08:23:52.740468+0000","last_became_active":"2023-11-24T08:23:50.738570+0000","last_became_peered":"2023-11-24T08:23:50.738570+0000","last_unstale":"2023-11-24T08:23:52.740468+0000","last_undegraded":"2023-11-24T08:23:52.740468+0000","last_fullsized":"2023-11-24T08:23:52.740468+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:09:53.907460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.452743+0000","last_change":"2023-11-24T08:23:51.747288+0000","last_active":"2023-11-24T08:23:53.452743+0000","last_peered":"2023-11-24T08:23:53.452743+0000","last_clean":"2023-11-24T08:23:53.452743+0000","last_became_active":"2023-11-24T08:23:51.746998+0000","last_became_peered":"2023-11-24T08:23:51.746998+0000","last_unstale":"2023-11-24T08:23:53.452743+0000","last_undegraded":"2023-11-24T08:23:53.452743+0000","last_fullsized":"2023-11-24T08:23:53.452743+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:54:30.172216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.740596+0000","last_change":"2023-11-24T08:23:50.744213+0000","last_active":"2023-11-24T08:23:52.740596+0000","last_peered":"2023-11-24T08:23:52.740596+0000","last_clean":"2023-11-24T08:23:52.740596+0000","last_became_active":"2023-11-24T08:23:50.743316+0000","last_became_peered":"2023-11-24T08:23:50.743316+0000","last_unstale":"2023-11-24T08:23:52.740596+0000","last_undegraded":"2023-11-24T08:23:52.740596+0000","last_fullsized":"2023-11-24T08:23:52.740596+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:49:26.702946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.451788+0000","last_change":"2023-11-24T08:23:51.743942+0000","last_active":"2023-11-24T08:23:53.451788+0000","last_peered":"2023-11-24T08:23:53.451788+0000","last_clean":"2023-11-24T08:23:53.451788+0000","last_became_active":"2023-11-24T08:23:51.743659+0000","last_became_peered":"2023-11-24T08:23:51.743659+0000","last_unstale":"2023-11-24T08:23:53.451788+0000","last_undegraded":"2023-11-24T08:23:53.451788+0000","last_fullsized":"2023-11-24T08:23:53.451788+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:55:02.933279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169475+0000","last_change":"2023-11-24T08:23:50.737231+0000","last_active":"2023-11-24T08:23:53.169475+0000","last_peered":"2023-11-24T08:23:53.169475+0000","last_clean":"2023-11-24T08:23:53.169475+0000","last_became_active":"2023-11-24T08:23:50.736949+0000","last_became_peered":"2023-11-24T08:23:50.736949+0000","last_unstale":"2023-11-24T08:23:53.169475+0000","last_undegraded":"2023-11-24T08:23:53.169475+0000","last_fullsized":"2023-11-24T08:23:53.169475+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:05.768024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.436269+0000","last_change":"2023-11-24T08:23:51.752405+0000","last_active":"2023-11-24T08:23:53.436269+0000","last_peered":"2023-11-24T08:23:53.436269+0000","last_clean":"2023-11-24T08:23:53.436269+0000","last_became_active":"2023-11-24T08:23:51.752076+0000","last_became_peered":"2023-11-24T08:23:51.752076+0000","last_unstale":"2023-11-24T08:23:53.436269+0000","last_undegraded":"2023-11-24T08:23:53.436269+0000","last_fullsized":"2023-11-24T08:23:53.436269+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:56:10.526577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169402+0000","last_change":"2023-11-24T08:23:50.736509+0000","last_active":"2023-11-24T08:23:53.169402+0000","last_peered":"2023-11-24T08:23:53.169402+0000","last_clean":"2023-11-24T08:23:53.169402+0000","last_became_active":"2023-11-24T08:23:50.734921+0000","last_became_peered":"2023-11-24T08:23:50.734921+0000","last_unstale":"2023-11-24T08:23:53.169402+0000","last_undegraded":"2023-11-24T08:23:53.169402+0000","last_fullsized":"2023-11-24T08:23:53.169402+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:00:25.221410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.435848+0000","last_change":"2023-11-24T08:23:51.749798+0000","last_active":"2023-11-24T08:23:53.435848+0000","last_peered":"2023-11-24T08:23:53.435848+0000","last_clean":"2023-11-24T08:23:53.435848+0000","last_became_active":"2023-11-24T08:23:51.749531+0000","last_became_peered":"2023-11-24T08:23:51.749531+0000","last_unstale":"2023-11-24T08:23:53.435848+0000","last_undegraded":"2023-11-24T08:23:53.435848+0000","last_fullsized":"2023-11-24T08:23:53.435848+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:54.835444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.739864+0000","last_change":"2023-11-24T08:23:50.741024+0000","last_active":"2023-11-24T08:23:52.739864+0000","last_peered":"2023-11-24T08:23:52.739864+0000","last_clean":"2023-11-24T08:23:52.739864+0000","last_became_active":"2023-11-24T08:23:50.740717+0000","last_became_peered":"2023-11-24T08:23:50.740717+0000","last_unstale":"2023-11-24T08:23:52.739864+0000","last_undegraded":"2023-11-24T08:23:52.739864+0000","last_fullsized":"2023-11-24T08:23:52.739864+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:13:10.613056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:52.950970+0000","last_change":"2023-11-24T08:23:51.739273+0000","last_active":"2023-11-24T08:23:52.950970+0000","last_peered":"2023-11-24T08:23:52.950970+0000","last_clean":"2023-11-24T08:23:52.950970+0000","last_became_active":"2023-11-24T08:23:51.738963+0000","last_became_peered":"2023-11-24T08:23:51.738963+0000","last_unstale":"2023-11-24T08:23:52.950970+0000","last_undegraded":"2023-11-24T08:23:52.950970+0000","last_fullsized":"2023-11-24T08:23:52.950970+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:18:03.867689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.169313+0000","last_change":"2023-11-24T08:23:50.744587+0000","last_active":"2023-11-24T08:23:53.169313+0000","last_peered":"2023-11-24T08:23:53.169313+0000","last_clean":"2023-11-24T08:23:53.169313+0000","last_became_active":"2023-11-24T08:23:50.744007+0000","last_became_peered":"2023-11-24T08:23:50.744007+0000","last_unstale":"2023-11-24T08:23:53.169313+0000","last_undegraded":"2023-11-24T08:23:53.169313+0000","last_fullsized":"2023-11-24T08:23:53.169313+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:49.678155+0000","last_clean_scrub_stamp":"2023-11-24T08:23:49.678155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:40.898200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-11-24T08:23:53.453440+0000","last_change":"2023-11-24T08:23:51.749577+0000","last_active":"2023-11-24T08:23:53.453440+0000","last_peered":"2023-11-24T08:23:53.453440+0000","last_clean":"2023-11-24T08:23:53.453440+0000","last_became_active":"2023-11-24T08:23:51.748280+0000","last_became_peered":"2023-11-24T08:23:51.748280+0000","last_unstale":"2023-11-24T08:23:53.453440+0000","last_undegraded":"2023-11-24T08:23:53.453440+0000","last_fullsized":"2023-11-24T08:23:53.453440+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:23:50.694459+0000","last_clean_scrub_stamp":"2023-11-24T08:23:50.694459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:50.603390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-11-24T08:23:51.728785+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:23:51.728785+0000","last_peered":"2023-11-24T08:23:51.728785+0000","last_clean":"2023-11-24T08:23:51.728785+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:23:51.728785+0000","last_undegraded":"2023-11-24T08:23:51.728785+0000","last_fullsized":"2023-11-24T08:23:51.728785+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42976,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328864,"statfs":{"total":96636764160,"available":96592756736,"internally_reserved":0,"allocated":3551232,"data_stored":1902442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0880000000000001,"15min":1.0629999999999999},"min":{"1min":0.64000000000000001,"5min":0.61199999999999999,"15min":0.42399999999999999},"max":{"1min":1.752,"5min":2.048,"15min":2.7240000000000002},"last":0.88},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.1299999999999999,"15min":1.103},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.53700000000000003},"max":{"1min":1.256,"5min":1.899,"15min":2.403},"last":0.77800000000000002}]},{"osd":1,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.137,"15min":1.1479999999999999},"min":{"1min":0.77200000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.5860000000000001,"5min":1.681,"15min":3.8519999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1739999999999999,"15min":1.2050000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.66600000000000004},"max":{"1min":1.409,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.359}]},{"osd":2,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.101,"15min":1.113},"min":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.4910000000000001,"5min":1.762,"15min":2.4430000000000001},"last":1.554},{"interface":"front","average":{"1min":1.222,"5min":1.2370000000000001,"15min":1.2849999999999999},"min":{"1min":0.84499999999999997,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8,"5min":1.8,"15min":4.0650000000000004},"last":1.3320000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.2,"15min":1.208},"min":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":3.133},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.325,"15min":1.361},"min":{"1min":0.95099999999999996,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.7669999999999999,"5min":2.173,"15min":3.9209999999999998},"last":1.139}]},{"osd":4,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3009999999999999,"15min":1.335},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.54700000000000004},"max":{"1min":1.556,"5min":1.8799999999999999,"15min":2.665},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.3080000000000001,"15min":1.341},"min":{"1min":0.54500000000000004,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.617,"5min":2.0419999999999998,"15min":3.4870000000000001},"last":1.6830000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.429,"15min":1.4650000000000001},"min":{"1min":0.80800000000000005,"5min":0.55700000000000005,"15min":0.42699999999999999},"max":{"1min":1.8560000000000001,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.242},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.448,"15min":1.484},"min":{"1min":0.69499999999999995,"5min":0.65400000000000003,"15min":0.622},"max":{"1min":1.8680000000000001,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.5029999999999999}]},{"osd":6,"last update":"Fri Nov 24 08:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.548,"15min":1.5629999999999999},"min":{"1min":0.77000000000000002,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.9970000000000001,"5min":2.2610000000000001,"15min":2.992},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4470000000000001,"15min":1.488},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.8240000000000001,"5min":2.4159999999999999,"15min":3.798},"last":1.871}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42396,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329444,"statfs":{"total":96636764160,"available":96593350656,"internally_reserved":0,"allocated":2965504,"data_stored":1317804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.0660000000000001,"15min":1.0720000000000001},"min":{"1min":0.69799999999999995,"5min":0.61199999999999999,"15min":0.59099999999999997},"max":{"1min":5.3040000000000003,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":1.492},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.2070000000000001,"15min":1.1240000000000001},"min":{"1min":0.86699999999999999,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":13.516999999999999,"5min":13.516999999999999,"15min":13.516999999999999},"last":0.80600000000000005}]},{"osd":1,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.143,"15min":1.149},"min":{"1min":0.74399999999999999,"5min":0.67700000000000005,"15min":0.61599999999999999},"max":{"1min":5.7050000000000001,"5min":5.7050000000000001,"15min":5.7050000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.851,"5min":1.3109999999999999,"15min":1.2270000000000001},"min":{"1min":0.83499999999999996,"5min":0.72799999999999998,"15min":0.65000000000000002},"max":{"1min":13.57,"5min":13.57,"15min":13.57},"last":1.103}]},{"osd":2,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.115,"15min":1.105},"min":{"1min":0.80900000000000005,"5min":0.70399999999999996,"15min":0.61899999999999999},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.258,"15min":1.258},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.60399999999999998},"max":{"1min":5.3440000000000003,"5min":5.3440000000000003,"15min":5.3440000000000003},"last":1.22}]},{"osd":3,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.4550000000000001,"5min":1.5509999999999999,"15min":1.409},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":23.989000000000001,"5min":23.989000000000001,"15min":23.989000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":2.4220000000000002,"5min":1.494,"15min":1.3740000000000001},"min":{"1min":0.94299999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":1.4350000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.911,"5min":1.47,"15min":1.4379999999999999},"min":{"1min":0.70199999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":1.7969999999999999},{"interface":"front","average":{"1min":2.3559999999999999,"5min":1.6259999999999999,"15min":1.5289999999999999},"min":{"1min":0.67300000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":20.173999999999999,"5min":20.173999999999999,"15min":20.173999999999999},"last":1.5229999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.3879999999999999,"15min":1.4119999999999999},"min":{"1min":0.77100000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.599},{"interface":"front","average":{"1min":1.8839999999999999,"5min":1.595,"15min":1.571},"min":{"1min":1.0129999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":8.3350000000000009,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":1.887}]},{"osd":7,"last update":"Fri Nov 24 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.669,"15min":1.6140000000000001},"min":{"1min":0.97599999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":13.612,"5min":13.612,"15min":13.612},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.5269999999999999,"15min":1.5669999999999999},"min":{"1min":0.95699999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9239999999999999,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.6659999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575081,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38244,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333596,"statfs":{"total":96636764160,"available":96597602304,"internally_reserved":0,"allocated":2965504,"data_stored":1317840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1,"15min":1.0109999999999999},"min":{"1min":0.61199999999999999,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.4299999999999999,"5min":1.696,"15min":4.9850000000000003},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.107,"15min":1.097},"min":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.4970000000000001,"5min":1.6639999999999999,"15min":5.4299999999999997},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.127,"15min":1.133},"min":{"1min":0.70499999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6000000000000001,"5min":1.7350000000000001,"15min":7.258},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.177,"15min":1.1899999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6359999999999999,"5min":1.796,"15min":6.5199999999999996},"last":1.5149999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.155,"15min":1.1739999999999999},"min":{"1min":0.72299999999999998,"5min":0.68100000000000005,"15min":0.65100000000000002},"max":{"1min":1.419,"5min":1.7829999999999999,"15min":6.8929999999999998},"last":1.601},{"interface":"front","average":{"1min":1.095,"5min":1.163,"15min":1.1799999999999999},"min":{"1min":0.72899999999999998,"5min":0.72399999999999998,"15min":0.68000000000000005},"max":{"1min":1.7050000000000001,"5min":1.923,"15min":8.0869999999999997},"last":1.4199999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1759999999999999,"15min":1.1839999999999999},"min":{"1min":0.85099999999999998,"5min":0.70799999999999996,"15min":0.66600000000000004},"max":{"1min":1.7170000000000001,"5min":2.242,"15min":9.0120000000000005},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.266,"15min":1.282},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.66800000000000004},"max":{"1min":1.6020000000000001,"5min":1.956,"15min":7.2130000000000001},"last":1.4650000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93200000000000005,"5min":0.79800000000000004,"15min":0.60799999999999998},"max":{"1min":1.917,"5min":2.536,"15min":11.813000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4590000000000001,"15min":1.4770000000000001},"min":{"1min":0.88900000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.667,"5min":2.4550000000000001,"15min":12.327999999999999},"last":1.6830000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.319,"15min":1.345},"min":{"1min":0.95299999999999996,"5min":0.74299999999999999,"15min":0.48899999999999999},"max":{"1min":1.536,"5min":2.246,"15min":9.4770000000000003},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.98199999999999998,"5min":0.70299999999999996,"15min":0.69299999999999995},"max":{"1min":1.9670000000000001,"5min":2.452,"15min":9.9019999999999992},"last":1.006}]},{"osd":7,"last update":"Fri Nov 24 08:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.49,"15min":1.5},"min":{"1min":1.246,"5min":0.86199999999999999,"15min":0.54300000000000004},"max":{"1min":1.6519999999999999,"5min":2.802,"15min":8.1660000000000004},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.6499999999999999},"min":{"1min":1.272,"5min":0.88600000000000001,"15min":0.75600000000000001},"max":{"1min":2.0070000000000001,"5min":2.9489999999999998,"15min":19.663},"last":1.6379999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42404,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329436,"statfs":{"total":96636764160,"available":96593342464,"internally_reserved":0,"allocated":2973696,"data_stored":1319382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0720000000000001,"15min":1.054},"min":{"1min":0.70699999999999996,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.8779999999999999},"last":1.528},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.46100000000000002},"max":{"1min":1.3140000000000001,"5min":1.591,"15min":5.452},"last":0.83599999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.139,"15min":1.1399999999999999},"min":{"1min":0.73599999999999999,"5min":0.67600000000000005,"15min":0.46300000000000002},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1000000000000001,"15min":1.133},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.51900000000000002},"max":{"1min":1.3089999999999999,"5min":1.6839999999999999,"15min":1.988},"last":1.0669999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0669999999999999,"15min":1.0629999999999999},"min":{"1min":0.79200000000000004,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.444,"5min":1.6519999999999999,"15min":1.7929999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.248},"min":{"1min":0.92800000000000005,"5min":0.67900000000000005,"15min":0.505},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0470000000000002},"last":1.3129999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3480000000000001,"15min":1.337},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.62},"max":{"1min":1.722,"5min":1.7869999999999999,"15min":2.121},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2929999999999999,"15min":1.274},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.55800000000000005},"max":{"1min":1.6579999999999999,"5min":1.744,"15min":2.3340000000000001},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.492,"15min":1.4930000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.69299999999999995},"max":{"1min":1.913,"5min":1.913,"15min":2.2429999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4159999999999999,"15min":1.419},"min":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.66200000000000003},"max":{"1min":1.996,"5min":2.0419999999999998,"15min":2.8090000000000002},"last":1.151}]},{"osd":6,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.278,"15min":1.2549999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.55400000000000005},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.5629999999999999,"15min":1.5409999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.58099999999999996},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.2890000000000001},"last":1.6699999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:23:28 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.472,"15min":1.472},"min":{"1min":0.754,"5min":0.72899999999999998,"15min":0.60799999999999998},"max":{"1min":1.9450000000000001,"5min":1.948,"15min":3.1219999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.4299999999999999,"15min":1.413},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.61799999999999999},"max":{"1min":1.98,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.4359999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42384,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329456,"statfs":{"total":96636764160,"available":96593362944,"internally_reserved":0,"allocated":2953216,"data_stored":1311560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83499999999999996,"15min":0.83199999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":1.3759999999999999,"5min":1.5820000000000001,"15min":3.181},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.93700000000000006,"15min":0.91200000000000003},"min":{"1min":0.55400000000000005,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.288,"5min":1.5349999999999999,"15min":1.7310000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.79100000000000004},"min":{"1min":0.40699999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.452,"5min":1.452,"15min":2.101},"last":0.622},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92100000000000004,"15min":0.92600000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.496,"5min":1.667,"15min":3.3260000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.88400000000000001,"15min":0.88100000000000001},"min":{"1min":0.57999999999999996,"5min":0.443,"15min":0.39900000000000002},"max":{"1min":1.7789999999999999,"5min":1.976,"15min":2.3700000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45200000000000001},"max":{"1min":1.5669999999999999,"5min":1.665,"15min":3.2480000000000002},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2629999999999999,"15min":1.2490000000000001},"min":{"1min":0.81100000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5009999999999999,"5min":1.845,"15min":2.1840000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.2270000000000001,"15min":1.212},"min":{"1min":0.78400000000000003,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.835,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.105}]},{"osd":5,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2589999999999999,"15min":1.2549999999999999},"min":{"1min":0.83199999999999996,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":1.8759999999999999,"5min":2.1230000000000002,"15min":3.4399999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.77600000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.883,"5min":1.883,"15min":1.992},"last":1.077}]},{"osd":6,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.82099999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9790000000000001,"5min":2.2549999999999999,"15min":3.5030000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.234,"15min":1.238},"min":{"1min":0.82199999999999995,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6870000000000001,"5min":2.3980000000000001,"15min":2.4060000000000001},"last":0.91200000000000003}]},{"osd":7,"last update":"Fri Nov 24 08:23:12 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3049999999999999,"15min":1.296},"min":{"1min":0.878,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1030000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3089999999999999,"15min":1.298},"min":{"1min":0.83499999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.9330000000000001,"5min":3.0030000000000001,"15min":3.331},"last":1.1839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575080,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42400,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329440,"statfs":{"total":96636764160,"available":96593346560,"internally_reserved":0,"allocated":2969600,"data_stored":1313686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.76900000000000002,"15min":0.76100000000000001},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":1.5429999999999999,"15min":1.601},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81399999999999995,"15min":0.84999999999999998},"min":{"1min":0.34000000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.474,"5min":1.4810000000000001,"15min":1.782},"last":1.0880000000000001}]},{"osd":1,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.55000000000000004,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.3049999999999999,"5min":1.5660000000000001,"15min":1.7809999999999999},"last":1.002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96299999999999997,"15min":0.94899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":1.625,"5min":1.6699999999999999,"15min":1.788},"last":1.343}]},{"osd":3,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.88700000000000001,"15min":0.85899999999999999},"min":{"1min":0.54400000000000004,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":1.2989999999999999,"5min":1.633,"15min":1.6850000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.02,"15min":1.0129999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.41399999999999998},"max":{"1min":1.5740000000000001,"5min":1.948,"15min":1.948},"last":1.137}]},{"osd":4,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.53700000000000003},"max":{"1min":1.6519999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.587},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.246,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.78700000000000003,"15min":0.745},"max":{"1min":1.619,"5min":1.8200000000000001,"15min":1.921},"last":1.431}]},{"osd":5,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.222,"15min":1.2290000000000001},"min":{"1min":0.82799999999999996,"5min":0.753,"15min":0.61499999999999999},"max":{"1min":1.538,"5min":1.8360000000000001,"15min":2.3149999999999999},"last":1.296},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.252,"15min":1.2310000000000001},"min":{"1min":0.77400000000000002,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.5980000000000001,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":1.4750000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.202,"15min":1.1919999999999999},"min":{"1min":0.82099999999999995,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.6559999999999999,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":1.518},{"interface":"front","average":{"1min":1.244,"5min":1.3129999999999999,"15min":1.3049999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.65600000000000003},"max":{"1min":1.6779999999999999,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:23:25 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.298,"15min":1.3029999999999999},"min":{"1min":0.875,"5min":0.77400000000000002,"15min":0.56399999999999995},"max":{"1min":1.714,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3069999999999999,"15min":1.2829999999999999},"min":{"1min":0.91900000000000004,"5min":0.77700000000000002,"15min":0.746},"max":{"1min":1.865,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1919999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42396,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329444,"statfs":{"total":96636764160,"available":96593350656,"internally_reserved":0,"allocated":2957312,"data_stored":1311594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.86699999999999999,"15min":0.82499999999999996},"min":{"1min":0.47999999999999998,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":1.4490000000000001,"5min":1.6240000000000001,"15min":1.7749999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.94699999999999995,"15min":0.91800000000000004},"min":{"1min":0.52500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.7130000000000001,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":1.1359999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81899999999999995,"15min":0.81499999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":1.3380000000000001,"5min":1.823,"15min":1.861},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.93100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.754,"5min":1.754,"15min":1.925},"last":0.76600000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0609999999999999,"15min":1.0149999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7569999999999999,"5min":2.0779999999999998,"15min":2.4169999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.91900000000000004,"15min":0.92300000000000004},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.3879999999999999},"last":0.81200000000000006}]},{"osd":4,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.351,"15min":1.2949999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.65300000000000002},"max":{"1min":1.9099999999999999,"5min":1.984,"15min":2.028},"last":1.222},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3029999999999999,"15min":1.27},"min":{"1min":0.96699999999999997,"5min":0.67800000000000005,"15min":0.55900000000000005},"max":{"1min":1.8180000000000001,"5min":2.0750000000000002,"15min":2.141},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.278,"15min":1.256},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8260000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2450000000000001,"15min":1.2170000000000001},"min":{"1min":0.77600000000000002,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":1.8939999999999999,"5min":2.1499999999999999,"15min":2.246},"last":1.1850000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2649999999999999,"15min":1.2829999999999999},"min":{"1min":0.77000000000000002,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.9710000000000001,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":1.056},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3049999999999999,"15min":1.2849999999999999},"min":{"1min":0.86099999999999999,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":2.0379999999999998,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.3560000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:23:19 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.3240000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.59099999999999997},"max":{"1min":1.8029999999999999,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":1.264},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.3700000000000001,"15min":1.343},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.653},"last":1.0920000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42968,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328872,"statfs":{"total":96636764160,"available":96592764928,"internally_reserved":0,"allocated":3547136,"data_stored":1901908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72999999999999998},"min":{"1min":0.45800000000000002,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.97799999999999998,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83399999999999996,"15min":0.82899999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35899999999999999},"max":{"1min":1.466,"5min":1.607,"15min":2.3620000000000001},"last":1.089}]},{"osd":2,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.01,"15min":1.0069999999999999},"min":{"1min":0.55800000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.635,"5min":1.663,"15min":1.893},"last":1.758},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.879,"15min":0.86599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39400000000000002},"max":{"1min":1.482,"5min":1.671,"15min":2.4990000000000001},"last":1.272}]},{"osd":3,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94499999999999995,"15min":0.95999999999999996},"min":{"1min":0.45900000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.3819999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.405},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85599999999999998,"15min":0.88100000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":1.333,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":1.6080000000000001}]},{"osd":4,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.228,"15min":1.212},"min":{"1min":0.84599999999999997,"5min":0.77500000000000002,"15min":0.72399999999999998},"max":{"1min":1.7749999999999999,"5min":2.2210000000000001,"15min":3.7610000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1000000000000001,"15min":1.105},"min":{"1min":0.76100000000000001,"5min":0.69799999999999995,"15min":0.67800000000000005},"max":{"1min":1.5600000000000001,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":1.464}]},{"osd":5,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1359999999999999,"15min":1.1459999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.587,"5min":1.667,"15min":2.6850000000000001},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.135,"15min":1.159},"min":{"1min":0.71199999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6040000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.7290000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.252,"15min":1.2609999999999999},"min":{"1min":0.80500000000000005,"5min":0.73999999999999999,"15min":0.71099999999999997},"max":{"1min":1.7310000000000001,"5min":1.8560000000000001,"15min":2.052},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.244,"15min":1.252},"min":{"1min":0.72999999999999998,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.6919999999999999,"5min":1.742,"15min":2.823},"last":1.873}]},{"osd":7,"last update":"Fri Nov 24 08:23:08 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.304,"15min":1.3320000000000001},"min":{"1min":0.96699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7809999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.747},"max":{"1min":1.8520000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.9199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:23:56.653 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:23:56.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:23:56.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.908+0000 7f21a6431640 1 Processor -- start 2023-11-24T08:23:56.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.908+0000 7f21a6431640 1 -- start start 2023-11-24T08:23:56.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a6431640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f21a01660b0 0x7f21a0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a6431640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f21a00c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a6431640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a0166510 con 0x7f21a00ccbd0 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a6431640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a01655f0 con 0x7f21a01660b0 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a6431640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00cc380 con 0x7f21a0162540 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f219ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f21a00ccbd0 0x7f21a00ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44114/0 (socket says 172.21.15.107:44114) 2023-11-24T08:23:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f219ffff640 1 -- 172.21.15.107:0/4289149323 learned_addr learned my addr 172.21.15.107:0/4289149323 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306777319 0 0) 0x7f2190001110 con 0x7f21a00ccbd0 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003520 con 0x7f21a00ccbd0 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f21a49ce640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f21a00c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.910+0000 7f219f7fe640 1 --2- 172.21.15.107:0/4289149323 >> v2:172.21.15.120:3300/0 conn(0x7f21a01660b0 0x7f21a0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627786517 0 0) 0x7f2188003520 con 0x7f21a00ccbd0 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 msgr2=0x7f21a00c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f21a00c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 >> v2:172.21.15.120:3300/0 conn(0x7f21a01660b0 msgr2=0x7f21a0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 --2- 172.21.15.107:0/4289149323 >> v2:172.21.15.120:3300/0 conn(0x7f21a01660b0 0x7f21a0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0162b30 con 0x7f21a00ccbd0 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2190001800 con 0x7f21a00ccbd0 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 >> v1:172.21.15.107:6789/0 conn(0x7f21a00ccbd0 legacy=0x7f21a00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f2188003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2188003fa0 0x7f21880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188004d00 0x7f2188005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01655f0 con 0x7f2188003fa0 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00cc380 con 0x7f2188004d00 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218800ba80 con 0x7f21a0162540 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.911+0000 7f219f7fe640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2188003fa0 0x7f21880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219ffff640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f2188003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219f7fe640 1 -- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 msgr2=0x7f2188003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:56.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219f7fe640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0162540 0x7f2188003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219f7fe640 1 -- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188004d00 msgr2=0x7f2188005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:56.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219f7fe640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188004d00 0x7f2188005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.912+0000 7f219f7fe640 1 -- 172.21.15.107:0/4289149323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188003520 con 0x7f2188003fa0 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.913+0000 7f219f7fe640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2188003fa0 0x7f21880047c0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f219400ba30 tx=0x7f21880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0adedc49310bcc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.913+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194032680 con 0x7f2188003fa0 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.913+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21940327e0 con 0x7f2188003fa0 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.913+0000 7f219effd640 1 -- 172.21.15.107:0/4289149323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194032b70 con 0x7f2188003fa0 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 -- 172.21.15.107:0/4289149323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2188003fa0 msgr2=0x7f21880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 --2- 172.21.15.107:0/4289149323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2188003fa0 0x7f21880047c0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f219400ba30 tx=0x7f21880057f0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a49ce640 1 -- 172.21.15.107:0/4289149323 reap_dead start 2023-11-24T08:23:56.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 -- 172.21.15.107:0/4289149323 shutdown_connections 2023-11-24T08:23:56.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 -- 172.21.15.107:0/4289149323 >> 172.21.15.107:0/4289149323 conn(0x7f21a00b0c50 msgr2=0x7f21a00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:56.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 -- 172.21.15.107:0/4289149323 shutdown_connections 2023-11-24T08:23:56.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.914+0000 7f21a6431640 1 -- 172.21.15.107:0/4289149323 wait complete. 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.916+0000 7f21a6431640 1 Processor -- start 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.916+0000 7f21a6431640 1 -- start start 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 0x7f21a00ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 0x7f21a00704c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00acac0 con 0x7f21a00ac100 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00accd0 con 0x7f2188003fa0 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a6431640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0071050 con 0x7f21a0156040 2023-11-24T08:23:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a49ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a49ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57174/0 (socket says 172.21.15.107:57174) 2023-11-24T08:23:56.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.917+0000 7f21a49ce640 1 -- 172.21.15.107:0/173430303 learned_addr learned my addr 172.21.15.107:0/173430303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219f7fe640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 0x7f21a00704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219ffff640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 0x7f21a00ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 -- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 msgr2=0x7f21a00704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 0x7f21a00704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 -- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 msgr2=0x7f21a00ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 0x7f21a00ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 -- 172.21.15.107:0/173430303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0071330 con 0x7f21a00ac100 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219ffff640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 0x7f21a00ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219f7fe640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 0x7f21a00704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a49ce640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f21a00cc9d0 tx=0x7f219800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198018070 con 0x7f21a00ac100 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800eb90 con 0x7f21a00ac100 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198003990 con 0x7f21a00ac100 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.918+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a00acf70 con 0x7f21a00ac100 2023-11-24T08:23:56.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.919+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a0071610 con 0x7f21a00ac100 2023-11-24T08:23:56.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.920+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a004e8b0 con 0x7f21a00ac100 2023-11-24T08:23:56.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.921+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f2198014420 con 0x7f21a00ac100 2023-11-24T08:23:56.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.921+0000 7f219cff9640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 0x7f218c06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:56.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.922+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21980885c0 con 0x7f21a00ac100 2023-11-24T08:23:56.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.922+0000 7f219ffff640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 0x7f218c06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:56.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.922+0000 7f219ffff640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 0x7f218c06f850 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f21900019b0 tx=0x7f2190009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:56.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:56.926+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2198055170 con 0x7f21a00ac100 2023-11-24T08:23:57.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.165+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f21a012c1b0 con 0x7f21a00ac100 2023-11-24T08:23:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.169+0000 7f219cff9640 1 -- 172.21.15.107:0/173430303 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2198058790 con 0x7f21a00ac100 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:23:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 msgr2=0x7f218c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 0x7f218c06f850 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f21900019b0 tx=0x7f2190009040 comp rx=0 tx=0).stop 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 msgr2=0x7f21a00ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f21a00cc9d0 tx=0x7f219800e4b0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 shutdown_connections 2023-11-24T08:23:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f218c06d390 0x7f218c06f850 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a00ac100 0x7f21a00ac580 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21a0156040 0x7f21a00704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 --2- 172.21.15.107:0/173430303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188003fa0 0x7f21a00ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 >> 172.21.15.107:0/173430303 conn(0x7f21a00b0c50 msgr2=0x7f21a006ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 shutdown_connections 2023-11-24T08:23:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.173+0000 7f21a6431640 1 -- 172.21.15.107:0/173430303 wait complete. 2023-11-24T08:23:57.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:23:57.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.400+0000 7f67f142d640 1 Processor -- start 2023-11-24T08:23:57.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.400+0000 7f67f142d640 1 -- start start 2023-11-24T08:23:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67f142d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67ec0b2250 0x7f67ec15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67f142d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67f142d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ec15ab80 con 0x7f67ec0cd470 2023-11-24T08:23:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67f142d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ec15a990 con 0x7f67ec0b2250 2023-11-24T08:23:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67f142d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec15e680 con 0x7f67ec0b2630 2023-11-24T08:23:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.402+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:56154/0 (socket says 172.21.15.107:56154) 2023-11-24T08:23:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67ea7fc640 1 -- 172.21.15.107:0/372933872 learned_addr learned my addr 172.21.15.107:0/372933872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eb7fe640 1 --1- 172.21.15.107:0/372933872 >> v1:172.21.15.120:6789/0 conn(0x7f67ec0cd470 0x7f67ec15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:38084/0 (socket says 172.21.15.107:38084) 2023-11-24T08:23:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eaffd640 1 --2- 172.21.15.107:0/372933872 >> v2:172.21.15.120:3300/0 conn(0x7f67ec0b2250 0x7f67ec15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67e9ffb640 1 -- 172.21.15.107:0/372933872 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983891353 0 0) 0x7f67ec15ab80 con 0x7f67ec0cd470 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67e9ffb640 1 -- 172.21.15.107:0/372933872 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4003520 con 0x7f67ec0cd470 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eaffd640 1 -- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 msgr2=0x7f67ec0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eaffd640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eaffd640 1 -- 172.21.15.107:0/372933872 >> v1:172.21.15.120:6789/0 conn(0x7f67ec0cd470 legacy=0x7f67ec15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.403+0000 7f67eaffd640 1 -- 172.21.15.107:0/372933872 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec15e960 con 0x7f67ec0b2250 2023-11-24T08:23:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.404+0000 7f67ea7fc640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:23:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.405+0000 7f67eaffd640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec15df80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00bd90 tx=0x7f67dc02dad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5848b118b7d9310 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67e9ffb640 1 -- 172.21.15.107:0/372933872 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc02f890 con 0x7f67ec0b2250 2023-11-24T08:23:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67e9ffb640 1 -- 172.21.15.107:0/372933872 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc02f9f0 con 0x7f67ec0b2250 2023-11-24T08:23:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67e9ffb640 1 -- 172.21.15.107:0/372933872 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc02fd60 con 0x7f67ec0b2250 2023-11-24T08:23:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 -- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 msgr2=0x7f67ec15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec15df80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00bd90 tx=0x7f67dc02dad0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 -- 172.21.15.107:0/372933872 shutdown_connections 2023-11-24T08:23:57.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0b2630 0x7f67ec0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 --2- 172.21.15.107:0/372933872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec15df80 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.406+0000 7f67f142d640 1 -- 172.21.15.107:0/372933872 >> 172.21.15.107:0/372933872 conn(0x7f67ec06a920 msgr2=0x7f67ec06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:57.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.407+0000 7f67f142d640 1 -- 172.21.15.107:0/372933872 shutdown_connections 2023-11-24T08:23:57.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.407+0000 7f67f142d640 1 -- 172.21.15.107:0/372933872 wait complete. 2023-11-24T08:23:57.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.408+0000 7f67f142d640 1 Processor -- start 2023-11-24T08:23:57.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.409+0000 7f67f142d640 1 -- start start 2023-11-24T08:23:57.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.409+0000 7f67f142d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.409+0000 7f67f142d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67f142d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 0x7f67ec0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67f142d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec1f32e0 con 0x7f67ec0b2630 2023-11-24T08:23:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67f142d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0ac500 con 0x7f67ec0b2250 2023-11-24T08:23:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67f142d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0ac730 con 0x7f67ec0cd470 2023-11-24T08:23:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57180/0 (socket says 172.21.15.107:57180) 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 -- 172.21.15.107:0/3999524907 learned_addr learned my addr 172.21.15.107:0/3999524907 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67eaffd640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67eb7fe640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 0x7f67ec0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 -- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 msgr2=0x7f67ec0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 0x7f67ec0abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 -- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 msgr2=0x7f67ec151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67ea7fc640 1 -- 172.21.15.107:0/3999524907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00094d0 con 0x7f67ec0b2630 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67eaffd640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.410+0000 7f67eb7fe640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 0x7f67ec0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67ea7fc640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7f67e000e9d0 tx=0x7f67e000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:57.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e000ca60 con 0x7f67ec0b2630 2023-11-24T08:23:57.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e000cbc0 con 0x7f67ec0b2630 2023-11-24T08:23:57.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0013470 con 0x7f67ec0b2630 2023-11-24T08:23:57.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc00ba70 con 0x7f67ec0b2630 2023-11-24T08:23:57.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.411+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec0accf0 con 0x7f67ec0b2630 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.412+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec067470 con 0x7f67ec0b2630 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.413+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f67e00135d0 con 0x7f67ec0b2630 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.415+0000 7f67d37fe640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 0x7f67d806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.415+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67e0088af0 con 0x7f67ec0b2630 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.415+0000 7f67eaffd640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 0x7f67d806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:57.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.415+0000 7f67eaffd640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 0x7f67d806f820 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f67dc038000 tx=0x7f67dc033000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.419+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67e00556a0 con 0x7f67ec0b2630 2023-11-24T08:23:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.647+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67ec0cea00 con 0x7f67ec0b2630 2023-11-24T08:23:57.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.648+0000 7f67d37fe640 1 -- 172.21.15.107:0/3999524907 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f67e0021090 con 0x7f67ec0b2630 2023-11-24T08:23:57.651 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:23:57.651 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":147,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:23:57.651 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 147 2023-11-24T08:23:57.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 msgr2=0x7f67d806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 0x7f67d806f820 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f67dc038000 tx=0x7f67dc033000 comp rx=0 tx=0).stop 2023-11-24T08:23:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 msgr2=0x7f67ec1443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7f67e000e9d0 tx=0x7f67e000ee90 comp rx=0 tx=0).stop 2023-11-24T08:23:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 shutdown_connections 2023-11-24T08:23:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f67d806d360 0x7f67d806f820 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67ec0cd470 0x7f67ec0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec0b2630 0x7f67ec1443f0 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 --2- 172.21.15.107:0/3999524907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec0b2250 0x7f67ec151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.651+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 >> 172.21.15.107:0/3999524907 conn(0x7f67ec06a920 msgr2=0x7f67ec0cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.652+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 shutdown_connections 2023-11-24T08:23:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:57.652+0000 7f67f142d640 1 -- 172.21.15.107:0/3999524907 wait complete. 2023-11-24T08:23:58.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 147, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:23:51.172573+0000', 'modified': '2023-11-24T08:23:54.236400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12747}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12747': {'gid': 12747, 'name': 'c', 'rank': 0, 'incarnation': 143, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/438366562', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 438366562}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 438366562}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:23:58.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:23:58.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:23:58.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.300+0000 7f7cb460e640 1 Processor -- start 2023-11-24T08:23:58.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.300+0000 7f7cb460e640 1 -- start start 2023-11-24T08:23:58.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.302+0000 7f7cb460e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7cac15d580 0x7f7cac0d4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.302+0000 7f7cb460e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cac0d1e30 0x7f7cac0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.302+0000 7f7cb460e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac0cf300 con 0x7f7cac0cfc80 2023-11-24T08:23:58.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.302+0000 7f7cb460e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cac0d1090 con 0x7f7cac0d1e30 2023-11-24T08:23:58.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.302+0000 7f7cb460e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cac0d1370 con 0x7f7cac15d580 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cac0d1e30 0x7f7cac0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb23aa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7cac15d580 0x7f7cac0d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7cac0d1e30 0x7f7cac0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38484/0 (socket says 172.21.15.107:38484) 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/1730943318 learned_addr learned my addr 172.21.15.107:0/1730943318 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/1730943318 >> v2:172.21.15.120:3300/0 conn(0x7f7cac15d580 msgr2=0x7f7cac0d4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/1730943318 >> v2:172.21.15.120:3300/0 conn(0x7f7cac15d580 0x7f7cac0d4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/1730943318 >> v1:172.21.15.107:6789/0 conn(0x7f7cac0cfc80 legacy=0x7f7cac0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.303+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/1730943318 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0d1650 con 0x7f7cac0d1e30 2023-11-24T08:23:58.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.304+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/1730943318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0cead0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f7cac072b90 tx=0x7f7c9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cffd3804b69c28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb13a8640 1 -- 172.21.15.107:0/1730943318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c030860 con 0x7f7cac0d1e30 2023-11-24T08:23:58.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb13a8640 1 -- 172.21.15.107:0/1730943318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c0309c0 con 0x7f7cac0d1e30 2023-11-24T08:23:58.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb13a8640 1 -- 172.21.15.107:0/1730943318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c030d50 con 0x7f7cac0d1e30 2023-11-24T08:23:58.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 -- 172.21.15.107:0/1730943318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac0d1e30 msgr2=0x7f7cac0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 --2- 172.21.15.107:0/1730943318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0cead0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f7cac072b90 tx=0x7f7c9c02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 -- 172.21.15.107:0/1730943318 shutdown_connections 2023-11-24T08:23:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 --2- 172.21.15.107:0/1730943318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0cead0 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 --2- 172.21.15.107:0/1730943318 >> v2:172.21.15.120:3300/0 conn(0x7f7cac15d580 0x7f7cac0d4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 -- 172.21.15.107:0/1730943318 >> 172.21.15.107:0/1730943318 conn(0x7f7cac09c770 msgr2=0x7f7cac0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 -- 172.21.15.107:0/1730943318 shutdown_connections 2023-11-24T08:23:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.305+0000 7f7cb460e640 1 -- 172.21.15.107:0/1730943318 wait complete. 2023-11-24T08:23:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.307+0000 7f7cb460e640 1 Processor -- start 2023-11-24T08:23:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.307+0000 7f7cb460e640 1 -- start start 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cac0cfc80 0x7f7cac1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1f49d0 con 0x7f7cac15d580 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1f3d00 con 0x7f7cac0d1e30 2023-11-24T08:23:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb460e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1f4030 con 0x7f7cac0cfc80 2023-11-24T08:23:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.308+0000 7f7cb2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38486/0 (socket says 172.21.15.107:38486) 2023-11-24T08:23:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 -- 172.21.15.107:0/498031294 learned_addr learned my addr 172.21.15.107:0/498031294 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 -- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cac0cfc80 msgr2=0x7f7cac1589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cac0cfc80 0x7f7cac1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 -- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cac0d1e30 msgr2=0x7f7cac0b13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 -- 172.21.15.107:0/498031294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c0094d0 con 0x7f7cac15d580 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4001eb0 tx=0x7f7ca400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7ca2ffd640 1 -- 172.21.15.107:0/498031294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40052d0 con 0x7f7cac15d580 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7ca2ffd640 1 -- 172.21.15.107:0/498031294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca4005430 con 0x7f7cac15d580 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7ca2ffd640 1 -- 172.21.15.107:0/498031294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4003810 con 0x7f7cac15d580 2023-11-24T08:23:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.309+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac1f4310 con 0x7f7cac15d580 2023-11-24T08:23:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.310+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac1f2fd0 con 0x7f7cac15d580 2023-11-24T08:23:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.312+0000 7f7ca2ffd640 1 -- 172.21.15.107:0/498031294 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f7ca4003970 con 0x7f7cac15d580 2023-11-24T08:23:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.314+0000 7f7ca2ffd640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 0x7f7c9006f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.314+0000 7f7ca2ffd640 1 -- 172.21.15.107:0/498031294 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ca4087ea0 con 0x7f7cac15d580 2023-11-24T08:23:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.314+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 0x7f7c9006f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.314+0000 7f7cb71a3740 1 Processor -- start 2023-11-24T08:23:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.314+0000 7f7cb71a3740 1 -- start start 2023-11-24T08:23:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 0x7f7c9006f850 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fa0 tx=0x7f7ca800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e65833f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658382560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e65829d880 con 0x55e658339ba0 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6582acae0 con 0x55e65833eea0 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6582aa1d0 con 0x55e65833e5e0 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb1ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb1ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38500/0 (socket says 172.21.15.107:38500) 2023-11-24T08:23:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.315+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/176921729 learned_addr learned my addr 172.21.15.107:0/176921729 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 msgr2=0x55e658382560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658382560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 msgr2=0x55e65833f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e65833f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/176921729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e65826d860 con 0x55e658339ba0 2023-11-24T08:23:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009960 tx=0x7f7c9c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c020b04243be1c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7c8ffff640 1 -- 172.21.15.107:0/176921729 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c030b30 con 0x55e658339ba0 2023-11-24T08:23:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7c8ffff640 1 -- 172.21.15.107:0/176921729 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c030c90 con 0x55e658339ba0 2023-11-24T08:23:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.316+0000 7f7c8ffff640 1 -- 172.21.15.107:0/176921729 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c044480 con 0x55e658339ba0 2023-11-24T08:23:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 -- 172.21.15.107:0/176921729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 msgr2=0x55e65833e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009960 tx=0x7f7c9c030000 comp rx=0 tx=0).stop 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 -- 172.21.15.107:0/176921729 shutdown_connections 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658382560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e65833e0a0 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/176921729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e65833f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 -- 172.21.15.107:0/176921729 >> 172.21.15.107:0/176921729 conn(0x55e658300c70 msgr2=0x55e658384990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 -- 172.21.15.107:0/176921729 shutdown_connections 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.317+0000 7f7cb71a3740 1 -- 172.21.15.107:0/176921729 wait complete. 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.318+0000 7f7cb71a3740 1 -- start start 2023-11-24T08:23:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.318+0000 7f7cb71a3740 1 Processor -- start 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e65838c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833eea0 0x55e658383260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6582a5ac0 con 0x55e65833e5e0 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6582422f0 con 0x55e658339ba0 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e65825fc70 con 0x55e65833eea0 2023-11-24T08:23:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb71a3740 10 client.0 _ll_register_callbacks cb 0x55e6582643b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.319+0000 7f7cb1ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38506/0 (socket says 172.21.15.107:38506) 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/3147937974 learned_addr learned my addr 172.21.15.107:0/3147937974 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833eea0 msgr2=0x55e658383260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833eea0 0x55e658383260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 msgr2=0x55e65838c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e65838c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 -- 172.21.15.107:0/3147937974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac0d1650 con 0x55e65833e5e0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7cb1ba9640 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c02bfb0 tx=0x7f7c9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b0ad2d17b2ce2e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.320+0000 7f7c8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c044690 con 0x55e65833e5e0 2023-11-24T08:23:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c9c03e070 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e65829eaa0 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c9c0447f0 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 10 client.12879 fetch_fsmap learned FSMap version 147 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=2+}) v3 -- 0x55e6582a4140 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e65829eaa0 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 147) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7c9c044950 con 0x55e65833e5e0 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 10 client.12879 fetch_fsmap finished waiting for FSMap version 147 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 10 client.12879 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 20 client.12879 populate_metadata read hostname 'smithi107' 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e658368a90 0x55e658368e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 4 client.12879 mds_command: new command op to 12747 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:23:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.321+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 --> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e65829e7c0 con 0x55e658368a90 2023-11-24T08:23:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.322+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e658368a90 0x55e658368e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.322+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e658368a90 0x55e658368e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=865e25c8048e992f server_cookie=1a39220850ee8c82 in_seq=0 out_seq=0 2023-11-24T08:23:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.322+0000 7f7c8d7fa640 10 client.12879 ms_handle_connect on v2:172.21.15.107:6836/438366562 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/3147937974 <== mds.0 v2:172.21.15.107:6836/438366562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e65829e7c0 con 0x55e658368a90 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7c8d7fa640 10 client.12879 handle_command_reply: tid=0 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 client.12879 shutdown 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 msgr2=0x55e658382d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c02bfb0 tx=0x7f7c9c004730 comp rx=0 tx=0).stop 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 shutdown_connections 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e658368a90 0x55e658368e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833eea0 0x55e658383260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833e5e0 0x55e658382d20 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7c8d7fa640 0 client.12879 ms_handle_reset on v2:172.21.15.107:6836/438366562 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/3147937974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e65838c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 >> 172.21.15.107:0/3147937974 conn(0x55e658300c70 msgr2=0x55e658338b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 shutdown_connections 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.323+0000 7f7cb71a3740 1 -- 172.21.15.107:0/3147937974 wait complete. 2023-11-24T08:23:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.324+0000 7f7cb71a3740 20 client.12879 trim_cache size 0 max 16384 2023-11-24T08:23:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.341+0000 7f7cb71a3740 1 Processor -- start 2023-11-24T08:23:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.341+0000 7f7cb71a3740 1 -- start start 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658368e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e658338d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e658008f20 con 0x55e658339ba0 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6583977b0 con 0x55e65833eea0 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e658369c40 con 0x55e65833e5e0 2023-11-24T08:23:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38510/0 (socket says 172.21.15.107:38510) 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 -- 172.21.15.107:0/4211929309 learned_addr learned my addr 172.21.15.107:0/4211929309 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 -- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 msgr2=0x55e658368e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658368e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 -- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 msgr2=0x55e658338d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e658338d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.342+0000 7f7cb23aa640 1 -- 172.21.15.107:0/4211929309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e658241b80 con 0x55e658339ba0 2023-11-24T08:23:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009d80 tx=0x7f7ca8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fed7b5f80f361f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7ca0ff9640 1 -- 172.21.15.107:0/4211929309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8022270 con 0x55e658339ba0 2023-11-24T08:23:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7ca0ff9640 1 -- 172.21.15.107:0/4211929309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca801b040 con 0x55e658339ba0 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7ca0ff9640 1 -- 172.21.15.107:0/4211929309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8022610 con 0x55e658339ba0 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 -- 172.21.15.107:0/4211929309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 msgr2=0x55e658383f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009d80 tx=0x7f7ca8009750 comp rx=0 tx=0).stop 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 -- 172.21.15.107:0/4211929309 shutdown_connections 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e65833eea0 0x55e658338d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658368e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/4211929309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e658339ba0 0x55e658383f60 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.343+0000 7f7cb71a3740 1 -- 172.21.15.107:0/4211929309 >> 172.21.15.107:0/4211929309 conn(0x55e658300c70 msgr2=0x55e65835f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.344+0000 7f7cb71a3740 1 -- 172.21.15.107:0/4211929309 shutdown_connections 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.344+0000 7f7cb71a3740 1 -- 172.21.15.107:0/4211929309 wait complete. 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.344+0000 7f7cb71a3740 1 -- start start 2023-11-24T08:23:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.344+0000 7f7cb71a3740 1 Processor -- start 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e658362a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658362f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6583668e0 con 0x55e65833eea0 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e658364df0 con 0x55e658339ba0 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e658363490 con 0x55e65833e5e0 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.345+0000 7f7cb71a3740 10 client.0 _ll_register_callbacks cb 0x55e65833e530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38520/0 (socket says 172.21.15.107:38520) 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 -- 172.21.15.107:0/300150124 learned_addr learned my addr 172.21.15.107:0/300150124 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 -- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 msgr2=0x55e658362f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658362f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 -- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 msgr2=0x55e658362a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e658362a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 -- 172.21.15.107:0/300150124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca8009bf0 con 0x55e65833eea0 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7cb2bab640 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4000fb0 tx=0x7f7ca4058eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c3f26bedb4c60e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.346+0000 7f7c8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40604e0 con 0x55e65833eea0 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ca4060640 con 0x55e65833eea0 2023-11-24T08:23:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c9c030c90 con 0x55e65833eea0 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ca4054460 con 0x55e65833eea0 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 10 client.12885 fetch_fsmap learned FSMap version 147 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=2+}) v3 -- 0x55e658267170 con 0x55e65833eea0 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c030c90 con 0x55e65833eea0 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 147) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7ca40545c0 con 0x55e65833eea0 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 10 client.12885 fetch_fsmap finished waiting for FSMap version 147 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 10 client.12885 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 20 client.12885 populate_metadata read hostname 'smithi107' 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e6583ef4a0 0x55e65837a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 4 client.12885 mds_command: new command op to 12747 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:23:58.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.347+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 --> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e657fa66c0 con 0x55e6583ef4a0 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e6583ef4a0 0x55e65837a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7cb23aa640 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e6583ef4a0 0x55e65837a320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4dc07a39f864a97 server_cookie=e58b6038ac8a776c in_seq=0 out_seq=0 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7c8d7fa640 10 client.12885 ms_handle_connect on v2:172.21.15.107:6836/438366562 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7c8d7fa640 1 -- 172.21.15.107:0/300150124 <== mds.0 v2:172.21.15.107:6836/438366562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55e657fa66c0 con 0x55e6583ef4a0 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7c8d7fa640 10 client.12885 handle_command_reply: tid=0 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7cb71a3740 1 client.12885 shutdown 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.348+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 msgr2=0x55e6583eed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4000fb0 tx=0x7f7ca4058eb0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 shutdown_connections 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:6836/438366562,v1:172.21.15.107:6837/438366562] conn(0x55e6583ef4a0 0x55e65837a320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e65833eea0 0x55e6583eed50 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e65833e5e0 0x55e658362f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 --2- 172.21.15.107:0/300150124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e658339ba0 0x55e658362a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7c8d7fa640 0 client.12885 ms_handle_reset on v2:172.21.15.107:6836/438366562 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 >> 172.21.15.107:0/300150124 conn(0x55e658300c70 msgr2=0x55e658383550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 shutdown_connections 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 1 -- 172.21.15.107:0/300150124 wait complete. 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb71a3740 20 client.12885 trim_cache size 0 max 16384 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:23:58.355 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 12747, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.6359911220000001, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 147, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 234, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 234, 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.064807482000001 2023-11-24T08:23:58.356 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:23:58.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 msgr2=0x7f7c9006f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 0x7f7c9006f850 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fa0 tx=0x7f7ca800b040 comp rx=0 tx=0).stop 2023-11-24T08:23:58.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.349+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 msgr2=0x7f7cac0b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4001eb0 tx=0x7f7ca400c890 comp rx=0 tx=0).stop 2023-11-24T08:23:58.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 shutdown_connections 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7c9006d390 0x7f7c9006f850 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cac15d580 0x7f7cac0b1900 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cac0d1e30 0x7f7cac0b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 --2- 172.21.15.107:0/498031294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cac0cfc80 0x7f7cac1589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 >> 172.21.15.107:0/498031294 conn(0x7f7cac09c770 msgr2=0x7f7cac0b4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 shutdown_connections 2023-11-24T08:23:58.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.350+0000 7f7cb460e640 1 -- 172.21.15.107:0/498031294 wait complete. 2023-11-24T08:23:58.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:23:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.567+0000 7f48e3933640 1 Processor -- start 2023-11-24T08:23:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.567+0000 7f48e3933640 1 -- start start 2023-11-24T08:23:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.568+0000 7f48e3933640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 0x7f48dc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.568+0000 7f48e3933640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.568+0000 7f48e3933640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc15ab80 con 0x7f48dc0cd470 2023-11-24T08:23:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.568+0000 7f48e3933640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc15a990 con 0x7f48dc0b2250 2023-11-24T08:23:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.568+0000 7f48e3933640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc15e680 con 0x7f48dc0b2630 2023-11-24T08:23:58.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.569+0000 7f48e0ece640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.569+0000 7f48e1ed0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f48dc0cd470 0x7f48dc15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58048/0 (socket says 172.21.15.107:58048) 2023-11-24T08:23:58.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.569+0000 7f48e1ed0640 1 -- 172.21.15.107:0/1691218275 learned_addr learned my addr 172.21.15.107:0/1691218275 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:23:58.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.569+0000 7f48e16cf640 1 --2- 172.21.15.107:0/1691218275 >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 0x7f48dc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48d3fff640 1 -- 172.21.15.107:0/1691218275 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580244339 0 0) 0x7f48dc15ab80 con 0x7f48dc0cd470 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48d3fff640 1 -- 172.21.15.107:0/1691218275 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c8003520 con 0x7f48dc0cd470 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48e0ece640 1 -- 172.21.15.107:0/1691218275 >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 msgr2=0x7f48dc15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48e0ece640 1 --2- 172.21.15.107:0/1691218275 >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 0x7f48dc15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48e0ece640 1 -- 172.21.15.107:0/1691218275 >> v1:172.21.15.120:6789/0 conn(0x7f48dc0cd470 legacy=0x7f48dc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48e0ece640 1 -- 172.21.15.107:0/1691218275 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc15e960 con 0x7f48dc0b2630 2023-11-24T08:23:58.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.570+0000 7f48e16cf640 1 --2- 172.21.15.107:0/1691218275 >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 0x7f48dc15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:23:58.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.572+0000 7f48e0ece640 1 --2- 172.21.15.107:0/1691218275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0ccf30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0097a0 tx=0x7f48cc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c60b533b0af6263 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.572+0000 7f48d3fff640 1 -- 172.21.15.107:0/1691218275 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc031830 con 0x7f48dc0b2630 2023-11-24T08:23:58.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.572+0000 7f48d3fff640 1 -- 172.21.15.107:0/1691218275 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc031990 con 0x7f48dc0b2630 2023-11-24T08:23:58.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.572+0000 7f48d3fff640 1 -- 172.21.15.107:0/1691218275 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc031d20 con 0x7f48dc0b2630 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 -- 172.21.15.107:0/1691218275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 msgr2=0x7f48dc0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 --2- 172.21.15.107:0/1691218275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0ccf30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0097a0 tx=0x7f48cc02b8d0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 -- 172.21.15.107:0/1691218275 shutdown_connections 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 --2- 172.21.15.107:0/1691218275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0ccf30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 --2- 172.21.15.107:0/1691218275 >> v2:172.21.15.120:3300/0 conn(0x7f48dc0b2250 0x7f48dc15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 -- 172.21.15.107:0/1691218275 >> 172.21.15.107:0/1691218275 conn(0x7f48dc06a920 msgr2=0x7f48dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 -- 172.21.15.107:0/1691218275 shutdown_connections 2023-11-24T08:23:58.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.573+0000 7f48e3933640 1 -- 172.21.15.107:0/1691218275 wait complete. 2023-11-24T08:23:58.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.575+0000 7f48e3933640 1 Processor -- start 2023-11-24T08:23:58.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.575+0000 7f48e3933640 1 -- start start 2023-11-24T08:23:58.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0cd470 0x7f48dc071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b00d0 con 0x7f48dc0b2250 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0af450 con 0x7f48dc0cd470 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e3933640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0af730 con 0x7f48dc0b2630 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.576+0000 7f48e16cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38524/0 (socket says 172.21.15.107:38524) 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 -- 172.21.15.107:0/535632848 learned_addr learned my addr 172.21.15.107:0/535632848 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 -- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 msgr2=0x7f48dc0aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 -- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0cd470 msgr2=0x7f48dc071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0cd470 0x7f48dc071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 -- 172.21.15.107:0/535632848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8009400 con 0x7f48dc0b2250 2023-11-24T08:23:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e16cf640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f48d800ebe0 tx=0x7f48d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8009760 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d800cc30 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.577+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc009400 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.578+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d800eee0 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.578+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc0afcf0 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.579+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc067470 con 0x7f48dc0b2250 2023-11-24T08:23:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.580+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f48d8013470 con 0x7f48dc0b2250 2023-11-24T08:23:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.581+0000 7f48d1ffb640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 0x7f48c006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:23:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.582+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48d8088bd0 con 0x7f48dc0b2250 2023-11-24T08:23:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.582+0000 7f48e0ece640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 0x7f48c006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:23:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.582+0000 7f48e0ece640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 0x7f48c006f820 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f48cc036000 tx=0x7f48cc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:23:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.586+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48d80556e0 con 0x7f48dc0b2250 2023-11-24T08:23:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.821+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f48dc0cea00 con 0x7f48dc0b2250 2023-11-24T08:23:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.822+0000 7f48d1ffb640 1 -- 172.21.15.107:0/535632848 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v147) v1 ==== 53+0+73 (secure 0 0 0) 0x7f48d8021060 con 0x7f48dc0b2250 2023-11-24T08:23:58.824 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:23:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 msgr2=0x7f48c006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 0x7f48c006f820 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f48cc036000 tx=0x7f48cc030000 comp rx=0 tx=0).stop 2023-11-24T08:23:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 msgr2=0x7f48dc153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:23:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f48d800ebe0 tx=0x7f48d800c520 comp rx=0 tx=0).stop 2023-11-24T08:23:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 shutdown_connections 2023-11-24T08:23:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48c006d360 0x7f48c006f820 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0cd470 0x7f48dc071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48dc0b2630 0x7f48dc0aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 --2- 172.21.15.107:0/535632848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48dc0b2250 0x7f48dc153c40 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:23:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 >> 172.21.15.107:0/535632848 conn(0x7f48dc06a920 msgr2=0x7f48dc15ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:23:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 shutdown_connections 2023-11-24T08:23:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:23:58.825+0000 7f48e3933640 1 -- 172.21.15.107:0/535632848 wait complete. 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:23:58.846 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:23:58.847 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:23:58.847 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:23:58.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:58.847 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:23:58.900 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:23:58.900 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:23:58.900 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:23:58.900 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.900 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:58.901 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41231sec preferred_lft 41231sec 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:23:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:23:58.903 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.903 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:23:58.903 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:23:58.903 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:23:58.903 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:23:58.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:58.904 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:23:58.995 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:23:58.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:58.999 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:23:59.052 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:23:59.052 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:23:59.052 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:23:59.052 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:23:59.052 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:23:59.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.053 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:23:59.053 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:23:59.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:23:59.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:23:59.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:23:59.054 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:23:59.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.238 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:23:59.289 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:23:59.289 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:23:59.290 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:23:59.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:23:59.383 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:23:59.383 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:23:59.439 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:23:59.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:23:59.440 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:23:59.492 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:23:59.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:59.636 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:59.636 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:23:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:23:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:23:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:23:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:23:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:23:59.673 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:23:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:23:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:23:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:23:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:23:59.678 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:23:59.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:23:59.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:23:59.768 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:23:59.828 INFO:teuthology.orchestra.run.smithi107.stdout:client6902 2023-11-24T08:23:59.828 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:23:59.828 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:23:59.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:24:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.134+0000 7fcb93563640 1 Processor -- start 2023-11-24T08:24:00.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.134+0000 7fcb93563640 1 -- start start 2023-11-24T08:24:00.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.135+0000 7fcb93563640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb8c13d120 0x7fcb8c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.135+0000 7fcb93563640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb8c06db20 0x7fcb8c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.135+0000 7fcb93563640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c0cec00 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.135+0000 7fcb93563640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c13f9e0 con 0x7fcb8c13d120 2023-11-24T08:24:00.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.135+0000 7fcb93563640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c13fcc0 con 0x7fcb8c06db20 2023-11-24T08:24:00.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb8c13d120 0x7fcb8c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:00.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb8c13d120 0x7fcb8c15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38538/0 (socket says 172.21.15.107:38538) 2023-11-24T08:24:00.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 -- 172.21.15.107:0/1983550463 learned_addr learned my addr 172.21.15.107:0/1983550463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:00.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 -- 172.21.15.107:0/1983550463 >> v2:172.21.15.120:3300/0 conn(0x7fcb8c06db20 msgr2=0x7fcb8c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 --2- 172.21.15.107:0/1983550463 >> v2:172.21.15.120:3300/0 conn(0x7fcb8c06db20 0x7fcb8c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 -- 172.21.15.107:0/1983550463 >> v1:172.21.15.107:6789/0 conn(0x7fcb8c06d6c0 legacy=0x7fcb8c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.136+0000 7fcb92561640 1 -- 172.21.15.107:0/1983550463 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c13ffa0 con 0x7fcb8c13d120 2023-11-24T08:24:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.137+0000 7fcb92561640 1 --2- 172.21.15.107:0/1983550463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c13d120 0x7fcb8c15af40 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c158170 tx=0x7fcb8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc422b52c786f65 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:00.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb9155f640 1 -- 172.21.15.107:0/1983550463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031870 con 0x7fcb8c13d120 2023-11-24T08:24:00.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb9155f640 1 -- 172.21.15.107:0/1983550463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb880319d0 con 0x7fcb8c13d120 2023-11-24T08:24:00.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb9155f640 1 -- 172.21.15.107:0/1983550463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031d60 con 0x7fcb8c13d120 2023-11-24T08:24:00.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 -- 172.21.15.107:0/1983550463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c13d120 msgr2=0x7fcb8c15af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 --2- 172.21.15.107:0/1983550463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c13d120 0x7fcb8c15af40 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c158170 tx=0x7fcb8802b910 comp rx=0 tx=0).stop 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 -- 172.21.15.107:0/1983550463 shutdown_connections 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 --2- 172.21.15.107:0/1983550463 >> v2:172.21.15.120:3300/0 conn(0x7fcb8c06db20 0x7fcb8c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 --2- 172.21.15.107:0/1983550463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c13d120 0x7fcb8c15af40 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.138+0000 7fcb93563640 1 -- 172.21.15.107:0/1983550463 >> 172.21.15.107:0/1983550463 conn(0x7fcb8c0cb830 msgr2=0x7fcb8c0cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.139+0000 7fcb93563640 1 -- 172.21.15.107:0/1983550463 shutdown_connections 2023-11-24T08:24:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.139+0000 7fcb93563640 1 -- 172.21.15.107:0/1983550463 wait complete. 2023-11-24T08:24:00.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.140+0000 7fcb93563640 1 Processor -- start 2023-11-24T08:24:00.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.140+0000 7fcb93563640 1 -- start start 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb8c06db20 0x7fcb8c0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8c13d120 0x7fcb8c0b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0ad9a0 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0acd20 con 0x7fcb8c13d120 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.141+0000 7fcb93563640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0ad000 con 0x7fcb8c06db20 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38554/0 (socket says 172.21.15.107:38554) 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 -- 172.21.15.107:0/373687272 learned_addr learned my addr 172.21.15.107:0/373687272 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 -- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb8c06db20 msgr2=0x7fcb8c0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb8c06db20 0x7fcb8c0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 -- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8c13d120 msgr2=0x7fcb8c0b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8c13d120 0x7fcb8c0b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 -- 172.21.15.107:0/373687272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880094d0 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.142+0000 7fcb92561640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.143+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031840 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.143+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb880443f0 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.143+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880447c0 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.143+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c0ad2e0 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.143+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c0b3130 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.144+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c0b3320 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.145+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7fcb8804d450 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.146+0000 7fcb82ffd640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 0x7fcb7006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:00.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.146+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb8802d040 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.146+0000 7fcb91d60640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 0x7fcb7006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.146+0000 7fcb91d60640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 0x7fcb7006f820 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c000f60 tx=0x7fcb7c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:00.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.150+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb8c0b3320 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.383+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcb8c0b3320 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.384+0000 7fcb82ffd640 1 -- 172.21.15.107:0/373687272 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v147) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcb8c0b3320 con 0x7fcb8c06d6c0 2023-11-24T08:24:00.386 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.386+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 msgr2=0x7fcb7006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.386+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 0x7fcb7006f820 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c000f60 tx=0x7fcb7c0079b0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.386+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 msgr2=0x7fcb8c0addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.386+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb880046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 shutdown_connections 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcb7006d360 0x7fcb7006f820 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8c13d120 0x7fcb8c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb8c06db20 0x7fcb8c0b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 --2- 172.21.15.107:0/373687272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb8c06d6c0 0x7fcb8c0addc0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 >> 172.21.15.107:0/373687272 conn(0x7fcb8c0cb830 msgr2=0x7fcb8c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 shutdown_connections 2023-11-24T08:24:00.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:00.387+0000 7fcb93563640 1 -- 172.21.15.107:0/373687272 wait complete. 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:24:00.406 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:24:00.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.407 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:24:00.468 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:24:00.469 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41228sec preferred_lft 41228sec 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.470 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:24:00.471 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:24:00.472 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:24:00.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.473 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:24:00.564 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:24:00.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.570 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:24:00.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:24:00.628 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:24:00.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.818 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:24:00.866 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:24:00.866 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:24:00.867 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:24:00.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:00.868 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:24:00.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:24:00.961 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:24:01.013 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:24:01.014 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:24:01.014 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:24:01.062 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:24:01.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:01.062 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:24:01.204 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:24:01.204 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:24:01.236 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:24:01.236 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:24:01.237 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:24:01.237 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:24:01.237 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:24:01.238 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:24:01.238 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:24:01.238 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:24:01.239 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:24:01.239 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:24:01.244 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:24:01.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:01.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:24:01.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:01.322 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:24:01.380 INFO:teuthology.orchestra.run.smithi120.stdout:client6905 2023-11-24T08:24:01.380 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:01.380 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:24:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.661+0000 7f8148590640 1 Processor -- start 2023-11-24T08:24:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.661+0000 7f8148590640 1 -- start start 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8140157c70 0x7f8140161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f81400cf450 0x7f81400cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f814015dc80 0x7f814015e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400cfe10 con 0x7f814015dc80 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400ce0b0 con 0x7f81400cf450 2023-11-24T08:24:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8148590640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8140161960 con 0x7f8140157c70 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8146b2d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f814015dc80 0x7f814015e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8146b2d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f814015dc80 0x7f814015e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38570/0 (socket says 172.21.15.107:38570) 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.662+0000 7f8146b2d640 1 -- 172.21.15.107:0/43173377 learned_addr learned my addr 172.21.15.107:0/43173377 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8145b2b640 1 --2- 172.21.15.107:0/43173377 >> v2:172.21.15.120:3300/0 conn(0x7f81400cf450 0x7f81400cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f814632c640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8140157c70 0x7f8140161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8146b2d640 1 -- 172.21.15.107:0/43173377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8140157c70 msgr2=0x7f8140161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8146b2d640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8140157c70 0x7f8140161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8146b2d640 1 -- 172.21.15.107:0/43173377 >> v2:172.21.15.120:3300/0 conn(0x7f81400cf450 msgr2=0x7f81400cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8146b2d640 1 --2- 172.21.15.107:0/43173377 >> v2:172.21.15.120:3300/0 conn(0x7f81400cf450 0x7f81400cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.663+0000 7f8146b2d640 1 -- 172.21.15.107:0/43173377 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814015e650 con 0x7f814015dc80 2023-11-24T08:24:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.664+0000 7f8146b2d640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814015dc80 0x7f814015e110 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f814006b550 tx=0x7f813802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f9d60800c1bfa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.664+0000 7f814532a640 1 -- 172.21.15.107:0/43173377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813802c9e0 con 0x7f814015dc80 2023-11-24T08:24:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.664+0000 7f814532a640 1 -- 172.21.15.107:0/43173377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813802cb40 con 0x7f814015dc80 2023-11-24T08:24:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.664+0000 7f814532a640 1 -- 172.21.15.107:0/43173377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380343f0 con 0x7f814015dc80 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 -- 172.21.15.107:0/43173377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814015dc80 msgr2=0x7f814015e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814015dc80 0x7f814015e110 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f814006b550 tx=0x7f813802be40 comp rx=0 tx=0).stop 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 -- 172.21.15.107:0/43173377 shutdown_connections 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814015dc80 0x7f814015e110 unknown :-1 s=CLOSED pgs=3207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 --2- 172.21.15.107:0/43173377 >> v2:172.21.15.120:3300/0 conn(0x7f81400cf450 0x7f81400cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 --2- 172.21.15.107:0/43173377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8140157c70 0x7f8140161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 -- 172.21.15.107:0/43173377 >> 172.21.15.107:0/43173377 conn(0x7f814006a920 msgr2=0x7f814006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 -- 172.21.15.107:0/43173377 shutdown_connections 2023-11-24T08:24:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.665+0000 7f8148590640 1 -- 172.21.15.107:0/43173377 wait complete. 2023-11-24T08:24:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.667+0000 7f8148590640 1 Processor -- start 2023-11-24T08:24:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.667+0000 7f8148590640 1 -- start start 2023-11-24T08:24:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8140157c70 0x7f8140145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814015dc80 0x7f8140099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814009a140 con 0x7f81400cf450 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81401470e0 con 0x7f8140157c70 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81401f2eb0 con 0x7f814015dc80 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38572/0 (socket says 172.21.15.107:38572) 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 -- 172.21.15.107:0/4090313621 learned_addr learned my addr 172.21.15.107:0/4090313621 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8146b2d640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814015dc80 0x7f8140099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 -- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814015dc80 msgr2=0x7f8140099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814015dc80 0x7f8140099c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 -- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8140157c70 msgr2=0x7f8140145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8140157c70 0x7f8140145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 -- 172.21.15.107:0/4090313621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8138009b30 con 0x7f81400cf450 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f814632c640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f813c00d830 tx=0x7f813c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c00b6c0 con 0x7f81400cf450 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c00b820 con 0x7f81400cf450 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c00bbb0 con 0x7f81400cf450 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81401467b0 con 0x7f81400cf450 2023-11-24T08:24:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.669+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8140146d70 con 0x7f81400cf450 2023-11-24T08:24:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.670+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8140067470 con 0x7f81400cf450 2023-11-24T08:24:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.672+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f813c01e460 con 0x7f81400cf450 2023-11-24T08:24:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.673+0000 7f8136ffd640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 0x7f812406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.673+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f813c088f70 con 0x7f81400cf450 2023-11-24T08:24:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.673+0000 7f8145b2b640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 0x7f812406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.674+0000 7f8145b2b640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 0x7f812406f820 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f8130006fd0 tx=0x7f8130008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.679+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f813c055b00 con 0x7f81400cf450 2023-11-24T08:24:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:01.908+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f81400cd910 con 0x7f81400cf450 2023-11-24T08:24:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.903+0000 7f8136ffd640 1 -- 172.21.15.107:0/4090313621 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v148) v1 ==== 136+0+0 (secure 0 0 0) 0x7f813c059120 con 0x7f81400cf450 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.906+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 msgr2=0x7f812406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.906+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 0x7f812406f820 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f8130006fd0 tx=0x7f8130008040 comp rx=0 tx=0).stop 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 msgr2=0x7f814009a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f813c00d830 tx=0x7f813c00dcf0 comp rx=0 tx=0).stop 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 shutdown_connections 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f812406d360 0x7f812406f820 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f814015dc80 0x7f8140099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8140157c70 0x7f8140145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 --2- 172.21.15.107:0/4090313621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81400cf450 0x7f814009a2e0 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 >> 172.21.15.107:0/4090313621 conn(0x7f814006a920 msgr2=0x7f814006ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 shutdown_connections 2023-11-24T08:24:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:02.907+0000 7f8148590640 1 -- 172.21.15.107:0/4090313621 wait complete. 2023-11-24T08:24:02.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:24:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.144+0000 7f90c5475640 1 Processor -- start 2023-11-24T08:24:03.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.144+0000 7f90c5475640 1 -- start start 2023-11-24T08:24:03.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90c5475640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90c00cc0f0 0x7f90c00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90c5475640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90c5475640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c015eea0 con 0x7f90c015b440 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90c5475640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c015b820 con 0x7f90c00cc0f0 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90c5475640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00abf30 con 0x7f90c00c57e0 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.146+0000 7f90be7fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90c00cc0f0 0x7f90c00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90beffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f90c015b440 0x7f90c00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58064/0 (socket says 172.21.15.107:58064) 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90beffd640 1 -- 172.21.15.107:0/2540041982 learned_addr learned my addr 172.21.15.107:0/2540041982 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90be7fc640 1 -- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 msgr2=0x7f90c006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90be7fc640 1 --2- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90be7fc640 1 -- 172.21.15.107:0/2540041982 >> v1:172.21.15.120:6789/0 conn(0x7f90c015b440 legacy=0x7f90c00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.147+0000 7f90be7fc640 1 -- 172.21.15.107:0/2540041982 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c015f0f0 con 0x7f90c00cc0f0 2023-11-24T08:24:03.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.149+0000 7f90be7fc640 1 --2- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c00c52a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f90b40097a0 tx=0x7f90b402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de6fe94a90be4556 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.149+0000 7f90bdffb640 1 -- 172.21.15.107:0/2540041982 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b402e070 con 0x7f90c00cc0f0 2023-11-24T08:24:03.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.149+0000 7f90bdffb640 1 -- 172.21.15.107:0/2540041982 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b40315a0 con 0x7f90c00cc0f0 2023-11-24T08:24:03.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.149+0000 7f90bdffb640 1 -- 172.21.15.107:0/2540041982 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4031930 con 0x7f90c00cc0f0 2023-11-24T08:24:03.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 -- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 msgr2=0x7f90c00c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 --2- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c00c52a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f90b40097a0 tx=0x7f90b402b840 comp rx=0 tx=0).stop 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 -- 172.21.15.107:0/2540041982 shutdown_connections 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 --2- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 --2- 172.21.15.107:0/2540041982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c00c52a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.150+0000 7f90c5475640 1 -- 172.21.15.107:0/2540041982 >> 172.21.15.107:0/2540041982 conn(0x7f90c00b0c50 msgr2=0x7f90c00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.151+0000 7f90c5475640 1 -- 172.21.15.107:0/2540041982 shutdown_connections 2023-11-24T08:24:03.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.151+0000 7f90c5475640 1 -- 172.21.15.107:0/2540041982 wait complete. 2023-11-24T08:24:03.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.152+0000 7f90c5475640 1 Processor -- start 2023-11-24T08:24:03.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.152+0000 7f90c5475640 1 -- start start 2023-11-24T08:24:03.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c01566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c0159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c015ab90 con 0x7f90c015b440 2023-11-24T08:24:03.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00c7840 con 0x7f90c00cc0f0 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.153+0000 7f90c5475640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c0159980 con 0x7f90c00c57e0 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38584/0 (socket says 172.21.15.107:38584) 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 -- 172.21.15.107:0/1474913567 learned_addr learned my addr 172.21.15.107:0/1474913567 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90beffd640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c01566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90be7fc640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c0159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 -- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 msgr2=0x7f90c01566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c01566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 -- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 msgr2=0x7f90c0159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c0159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.154+0000 7f90bf7fe640 1 -- 172.21.15.107:0/1474913567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4009400 con 0x7f90c015b440 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90bf7fe640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f90b800efc0 tx=0x7f90b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8014480 con 0x7f90c015b440 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b800f040 con 0x7f90c015b440 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8014820 con 0x7f90c015b440 2023-11-24T08:24:03.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c015a4f0 con 0x7f90c015b440 2023-11-24T08:24:03.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.155+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c00c6720 con 0x7f90c015b440 2023-11-24T08:24:03.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.156+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c00c68b0 con 0x7f90c015b440 2023-11-24T08:24:03.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.157+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f90b8014980 con 0x7f90c015b440 2023-11-24T08:24:03.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.158+0000 7f90a77fe640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 0x7f90ac06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.158+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90b8089740 con 0x7f90c015b440 2023-11-24T08:24:03.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.158+0000 7f90beffd640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 0x7f90ac06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.159+0000 7f90beffd640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 0x7f90ac06f820 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005880 tx=0x7f90b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.162+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90b8056270 con 0x7f90c015b440 2023-11-24T08:24:03.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.376+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f90c0155320 con 0x7f90ac06d360 2023-11-24T08:24:03.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.378+0000 7f90a77fe640 1 -- 172.21.15.107:0/1474913567 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f90c0155320 con 0x7f90ac06d360 2023-11-24T08:24:03.379 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:24:03.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:03.379 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:24:03.379 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:03.379 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:24:03.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.380+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 msgr2=0x7f90ac06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.380+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 0x7f90ac06f820 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005880 tx=0x7f90b0009040 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.380+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 msgr2=0x7f90c00af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.380+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f90b800efc0 tx=0x7f90b800c490 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 shutdown_connections 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f90ac06d360 0x7f90ac06f820 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c015b440 0x7f90c00af9c0 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90c00cc0f0 0x7f90c0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 --2- 172.21.15.107:0/1474913567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90c00c57e0 0x7f90c01566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 >> 172.21.15.107:0/1474913567 conn(0x7f90c00b0c50 msgr2=0x7f90c0070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 shutdown_connections 2023-11-24T08:24:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.381+0000 7f90c5475640 1 -- 172.21.15.107:0/1474913567 wait complete. 2023-11-24T08:24:03.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:24:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.617+0000 7f805f0bb640 1 Processor -- start 2023-11-24T08:24:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.617+0000 7f805f0bb640 1 -- start start 2023-11-24T08:24:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.618+0000 7f805f0bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80580cc0f0 0x7f80580c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.618+0000 7f805f0bb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805815eea0 con 0x7f805815b440 2023-11-24T08:24:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.618+0000 7f805f0bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80580abf30 con 0x7f80580c57e0 2023-11-24T08:24:03.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.618+0000 7f805f0bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805815b820 con 0x7f80580cc0f0 2023-11-24T08:24:03.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.619+0000 7f805ce57640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f805815b440 0x7f80580cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52646/0 (socket says 172.21.15.107:52646) 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.619+0000 7f805ce57640 1 -- 172.21.15.107:0/4149668272 learned_addr learned my addr 172.21.15.107:0/4149668272 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.619+0000 7f8057fff640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80580cc0f0 0x7f80580c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.619+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844456119 0 0) 0x7f805815eea0 con 0x7f805815b440 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.619+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048003520 con 0x7f805815b440 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036349108 0 0) 0x7f8048003520 con 0x7f805815b440 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80580cc0f0 msgr2=0x7f80580c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80580cc0f0 0x7f80580c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 >> v1:172.21.15.120:6789/0 conn(0x7f80580c57e0 legacy=0x7f805806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805815f0f0 con 0x7f805815b440 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f804c001670 con 0x7f805815b440 2023-11-24T08:24:03.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 >> v1:172.21.15.107:6789/0 conn(0x7f805815b440 legacy=0x7f80580cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80480039a0 0x7f8048003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048004300 0x7f80480047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048005080 0x7f804800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805815b820 con 0x7f8048005080 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804800bfa0 con 0x7f8048004300 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804800c160 con 0x7f80480039a0 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f8057fff640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80580cc0f0 0x7f80580c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.620+0000 7f805d658640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048005080 0x7f804800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805ce57640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80480039a0 0x7f8048003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805d658640 1 -- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80480039a0 msgr2=0x7f8048003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805d658640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80480039a0 0x7f8048003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805d658640 1 -- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048004300 msgr2=0x7f80480047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:03.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805d658640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048004300 0x7f80480047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.621+0000 7f805d658640 1 -- 172.21.15.107:0/4149668272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048003520 con 0x7f8048005080 2023-11-24T08:24:03.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f805d658640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048005080 0x7f804800b980 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f8050001f60 tx=0x7f80480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e97a1e36fdb435 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805002d6f0 con 0x7f8048005080 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805002d870 con 0x7f8048005080 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f80577fe640 1 -- 172.21.15.107:0/4149668272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805002dc20 con 0x7f8048005080 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f805f0bb640 1 -- 172.21.15.107:0/4149668272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048005080 msgr2=0x7f804800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.622+0000 7f805f0bb640 1 --2- 172.21.15.107:0/4149668272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048005080 0x7f804800b980 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f8050001f60 tx=0x7f80480057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.623+0000 7f805d658640 1 -- 172.21.15.107:0/4149668272 reap_dead start 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.623+0000 7f805f0bb640 1 -- 172.21.15.107:0/4149668272 shutdown_connections 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.623+0000 7f805f0bb640 1 -- 172.21.15.107:0/4149668272 >> 172.21.15.107:0/4149668272 conn(0x7f80580b0c50 msgr2=0x7f80580b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.623+0000 7f805f0bb640 1 -- 172.21.15.107:0/4149668272 shutdown_connections 2023-11-24T08:24:03.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.623+0000 7f805f0bb640 1 -- 172.21.15.107:0/4149668272 wait complete. 2023-11-24T08:24:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.624+0000 7f805f0bb640 1 Processor -- start 2023-11-24T08:24:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.625+0000 7f805f0bb640 1 -- start start 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048005080 0x7f80580c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8058152e00 0x7f8058145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80580aed70 con 0x7f80580b1c80 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80580aef80 con 0x7f8048005080 2023-11-24T08:24:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805f0bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80580af260 con 0x7f8058152e00 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38606/0 (socket says 172.21.15.107:38606) 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 -- 172.21.15.107:0/715084032 learned_addr learned my addr 172.21.15.107:0/715084032 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f8057fff640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8058152e00 0x7f8058145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805ce57640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048005080 0x7f80580c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 -- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8058152e00 msgr2=0x7f8058145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8058152e00 0x7f8058145d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 -- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048005080 msgr2=0x7f80580c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048005080 0x7f80580c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.626+0000 7f805d658640 1 -- 172.21.15.107:0/715084032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580af540 con 0x7f80580b1c80 2023-11-24T08:24:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f805d658640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f8050002080 tx=0x7f80500015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050003b70 con 0x7f80580b1c80 2023-11-24T08:24:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050003cd0 con 0x7f80580b1c80 2023-11-24T08:24:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80580af820 con 0x7f80580b1c80 2023-11-24T08:24:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050005290 con 0x7f80580b1c80 2023-11-24T08:24:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.627+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8058146260 con 0x7f80580b1c80 2023-11-24T08:24:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.628+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80580afd40 con 0x7f80580b1c80 2023-11-24T08:24:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.629+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f80500053f0 con 0x7f80580b1c80 2023-11-24T08:24:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.631+0000 7f80557fa640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 0x7f803c06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.631+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8050031070 con 0x7f80580b1c80 2023-11-24T08:24:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.631+0000 7f805ce57640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 0x7f803c06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.632+0000 7f805ce57640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 0x7f803c06f820 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f804c001eb0 tx=0x7f804c001ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.635+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80580afd40 con 0x7f80580b1c80 2023-11-24T08:24:03.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.827+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f80580afd40 con 0x7f80580b1c80 2023-11-24T08:24:03.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.828+0000 7f80557fa640 1 -- 172.21.15.107:0/715084032 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v55)=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8050077410 con 0x7f80580b1c80 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 msgr2=0x7f803c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 0x7f803c06f820 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f804c001eb0 tx=0x7f804c001ce0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 msgr2=0x7f8058157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f8050002080 tx=0x7f80500015b0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 shutdown_connections 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f803c06d360 0x7f803c06f820 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80580b1c80 0x7f8058157890 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8058152e00 0x7f8058145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 --2- 172.21.15.107:0/715084032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8048005080 0x7f80580c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 >> 172.21.15.107:0/715084032 conn(0x7f80580b0c50 msgr2=0x7f80580cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 shutdown_connections 2023-11-24T08:24:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:03.831+0000 7f805f0bb640 1 -- 172.21.15.107:0/715084032 wait complete. 2023-11-24T08:24:03.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol1655 --mode=777 2023-11-24T08:24:04.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.078+0000 7f339a690640 1 Processor -- start 2023-11-24T08:24:04.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.078+0000 7f339a690640 1 -- start start 2023-11-24T08:24:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.079+0000 7f339a690640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 0x7f339406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.079+0000 7f339a690640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 0x7f33940cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.079+0000 7f339a690640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33940cece0 con 0x7f33940cf450 2023-11-24T08:24:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.079+0000 7f339a690640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3394156010 con 0x7f33940ce6c0 2023-11-24T08:24:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.079+0000 7f339a690640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339406ce30 con 0x7f339415c7f0 2023-11-24T08:24:04.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.080+0000 7f338bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f33940cf450 0x7f33940cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:58068/0 (socket says 172.21.15.107:58068) 2023-11-24T08:24:04.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.080+0000 7f3393fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 0x7f339406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.080+0000 7f338bfff640 1 -- 172.21.15.107:0/398655480 learned_addr learned my addr 172.21.15.107:0/398655480 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:04.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.080+0000 7f3398c2d640 1 --2- 172.21.15.107:0/398655480 >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 0x7f33940cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.081+0000 7f3393fff640 1 -- 172.21.15.107:0/398655480 >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 msgr2=0x7f33940cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.081+0000 7f3393fff640 1 --2- 172.21.15.107:0/398655480 >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 0x7f33940cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.081+0000 7f3393fff640 1 -- 172.21.15.107:0/398655480 >> v1:172.21.15.120:6789/0 conn(0x7f33940cf450 legacy=0x7f33940cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.081+0000 7f3393fff640 1 -- 172.21.15.107:0/398655480 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339406d040 con 0x7f339415c7f0 2023-11-24T08:24:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.081+0000 7f3398c2d640 1 --2- 172.21.15.107:0/398655480 >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 0x7f33940cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:24:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.082+0000 7f3393fff640 1 --2- 172.21.15.107:0/398655480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 0x7f339406db20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbfdb7016c001d14 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.083+0000 7f33937fe640 1 -- 172.21.15.107:0/398655480 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338402d040 con 0x7f339415c7f0 2023-11-24T08:24:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.083+0000 7f33937fe640 1 -- 172.21.15.107:0/398655480 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384031810 con 0x7f339415c7f0 2023-11-24T08:24:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.083+0000 7f33937fe640 1 -- 172.21.15.107:0/398655480 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384031ba0 con 0x7f339415c7f0 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 -- 172.21.15.107:0/398655480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 msgr2=0x7f339406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 --2- 172.21.15.107:0/398655480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 0x7f339406db20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 -- 172.21.15.107:0/398655480 shutdown_connections 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 --2- 172.21.15.107:0/398655480 >> v2:172.21.15.120:3300/0 conn(0x7f33940ce6c0 0x7f33940cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 --2- 172.21.15.107:0/398655480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f339415c7f0 0x7f339406db20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 -- 172.21.15.107:0/398655480 >> 172.21.15.107:0/398655480 conn(0x7f339406a920 msgr2=0x7f339406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 -- 172.21.15.107:0/398655480 shutdown_connections 2023-11-24T08:24:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.084+0000 7f339a690640 1 -- 172.21.15.107:0/398655480 wait complete. 2023-11-24T08:24:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.086+0000 7f339a690640 1 Processor -- start 2023-11-24T08:24:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.087+0000 7f339a690640 1 -- start start 2023-11-24T08:24:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 0x7f3394154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339415c7f0 0x7f3394154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b20e0 con 0x7f33940ce6c0 2023-11-24T08:24:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941583d0 con 0x7f339415c7f0 2023-11-24T08:24:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f339a690640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33941586b0 con 0x7f33940cf450 2023-11-24T08:24:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f3393fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f3393fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38616/0 (socket says 172.21.15.107:38616) 2023-11-24T08:24:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.088+0000 7f3393fff640 1 -- 172.21.15.107:0/1309466221 learned_addr learned my addr 172.21.15.107:0/1309466221 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f338bfff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 0x7f3394154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3398c2d640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339415c7f0 0x7f3394154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 -- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 msgr2=0x7f3394154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 0x7f3394154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 -- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339415c7f0 msgr2=0x7f3394154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339415c7f0 0x7f3394154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 -- 172.21.15.107:0/1309466221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c009a70 con 0x7f33940ce6c0 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f338bfff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 0x7f3394154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f3393fff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f3384000c00 tx=0x7f33840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384006e10 con 0x7f33940ce6c0 2023-11-24T08:24:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338403ad60 con 0x7f33940ce6c0 2023-11-24T08:24:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384044630 con 0x7f33940ce6c0 2023-11-24T08:24:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.089+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3384009400 con 0x7f33940ce6c0 2023-11-24T08:24:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.090+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394158c70 con 0x7f33940ce6c0 2023-11-24T08:24:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.091+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394158e60 con 0x7f33940ce6c0 2023-11-24T08:24:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.092+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f338404d450 con 0x7f33940ce6c0 2023-11-24T08:24:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.093+0000 7f33917fa640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 0x7f337406f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.093+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33840aab10 con 0x7f33940ce6c0 2023-11-24T08:24:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.094+0000 7f338bfff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 0x7f337406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.094+0000 7f338bfff640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 0x7f337406f820 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f339409d310 tx=0x7f337c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.098+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3384077620 con 0x7f33940ce6c0 2023-11-24T08:24:04.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.317+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol1655", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f339406e060 con 0x7f337406d360 2023-11-24T08:24:04.327 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:04.326+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:04.327 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:04.326+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:04.327 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:04.326+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:04.327 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:04.326+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:04.327 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:04.326+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.384+0000 7f33917fa640 1 -- 172.21.15.107:0/1309466221 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f337c0013d0 con 0x7f337406d360 2023-11-24T08:24:04.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 msgr2=0x7f337406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 0x7f337406f820 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f339409d310 tx=0x7f337c009040 comp rx=0 tx=0).stop 2023-11-24T08:24:04.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 msgr2=0x7f339414bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f3384000c00 tx=0x7f33840046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 shutdown_connections 2023-11-24T08:24:04.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f337406d360 0x7f337406f820 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339415c7f0 0x7f3394154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33940cf450 0x7f3394154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 --2- 172.21.15.107:0/1309466221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33940ce6c0 0x7f339414bbb0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 >> 172.21.15.107:0/1309466221 conn(0x7f339406a920 msgr2=0x7f33940cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.387+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 shutdown_connections 2023-11-24T08:24:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.388+0000 7f339a690640 1 -- 172.21.15.107:0/1309466221 wait complete. 2023-11-24T08:24:04.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1655 2023-11-24T08:24:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.616+0000 7f6ef2301640 1 Processor -- start 2023-11-24T08:24:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.616+0000 7f6ef2301640 1 -- start start 2023-11-24T08:24:04.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef2301640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6eec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef2301640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6eec0cbf20 0x7f6eec0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef2301640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0cc3a0 con 0x7f6eec0c57e0 2023-11-24T08:24:04.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef2301640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec15b820 con 0x7f6eec0cbf20 2023-11-24T08:24:04.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef2301640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec15eea0 con 0x7f6eec15b440 2023-11-24T08:24:04.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef089e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6eec0c57e0 0x7f6eec0c3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52652/0 (socket says 172.21.15.107:52652) 2023-11-24T08:24:04.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.618+0000 7f6ef089e640 1 -- 172.21.15.107:0/3455368808 learned_addr learned my addr 172.21.15.107:0/3455368808 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:04.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eebfff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6eec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/3455368808 >> v2:172.21.15.120:3300/0 conn(0x7f6eec0cbf20 0x7f6eec0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022233018 0 0) 0x7f6ee4001490 con 0x7f6eec0c57e0 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed8003520 con 0x7f6eec0c57e0 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516850982 0 0) 0x7f6ed8003520 con 0x7f6eec0c57e0 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 msgr2=0x7f6eec0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6eec0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 >> v2:172.21.15.120:3300/0 conn(0x7f6eec0cbf20 msgr2=0x7f6eec0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:04.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/3455368808 >> v2:172.21.15.120:3300/0 conn(0x7f6eec0cbf20 0x7f6eec0c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec06d600 con 0x7f6eec0c57e0 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f6ee4001b80 con 0x7f6eec0c57e0 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 >> v1:172.21.15.107:6789/0 conn(0x7f6eec0c57e0 legacy=0x7f6eec0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ed8003970 0x7f6ed8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed80047c0 0x7f6ed8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.619+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6ed8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec15b820 con 0x7f6ed8003970 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec15eea0 con 0x7f6ed80047c0 2023-11-24T08:24:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed800b980 con 0x7f6eec15b440 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ed8003970 0x7f6ed8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6ef089e640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6ed8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 -- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 msgr2=0x7f6ed8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed80047c0 0x7f6ed8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6ed8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 -- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed80047c0 msgr2=0x7f6ed8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed80047c0 0x7f6ed8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6eebfff640 1 -- 172.21.15.107:0/3455368808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8003520 con 0x7f6ed8003970 2023-11-24T08:24:04.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.620+0000 7f6ef089e640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec15b440 0x7f6ed8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.621+0000 7f6eebfff640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ed8003970 0x7f6ed8003d90 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009af0 tx=0x7f6ed80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde2acf1620968de server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc02c6b0 con 0x7f6ed8003970 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc02c810 con 0x7f6ed8003970 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/3455368808 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc02cba0 con 0x7f6ed8003970 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 -- 172.21.15.107:0/3455368808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ed8003970 msgr2=0x7f6ed8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 --2- 172.21.15.107:0/3455368808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ed8003970 0x7f6ed8003d90 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009af0 tx=0x7f6ed80057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef089e640 1 -- 172.21.15.107:0/3455368808 reap_dead start 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 -- 172.21.15.107:0/3455368808 shutdown_connections 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 -- 172.21.15.107:0/3455368808 >> 172.21.15.107:0/3455368808 conn(0x7f6eec0b0c50 msgr2=0x7f6eec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 -- 172.21.15.107:0/3455368808 shutdown_connections 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.622+0000 7f6ef2301640 1 -- 172.21.15.107:0/3455368808 wait complete. 2023-11-24T08:24:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.624+0000 7f6ef2301640 1 Processor -- start 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.624+0000 7f6ef2301640 1 -- start start 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed8003970 0x7f6eec1e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec0b0100 0x7f6eec15e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b24f0 con 0x7f6eec0b3db0 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b47a0 con 0x7f6ed8003970 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.625+0000 7f6ef2301640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b0580 con 0x7f6eec0b0100 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38666/0 (socket says 172.21.15.107:38666) 2023-11-24T08:24:04.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 -- 172.21.15.107:0/2442608189 learned_addr learned my addr 172.21.15.107:0/2442608189 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6eebfff640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed8003970 0x7f6eec1e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 -- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec0b0100 msgr2=0x7f6eec15e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec0b0100 0x7f6eec15e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 -- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed8003970 msgr2=0x7f6eec1e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed8003970 0x7f6eec1e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 -- 172.21.15.107:0/2442608189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec1e6f10 con 0x7f6eec0b3db0 2023-11-24T08:24:04.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.626+0000 7f6ef089e640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f6eec06e770 tx=0x7f6ee400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.627+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee400bb80 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.627+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee400bce0 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.627+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4007310 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.627+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec0b1e30 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.627+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec15e690 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.628+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec0b2630 con 0x7f6eec0b3db0 2023-11-24T08:24:04.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.629+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 48) v1 ==== 92597+0+0 (secure 0 0 0) 0x7f6ee4007470 con 0x7f6eec0b3db0 2023-11-24T08:24:04.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.630+0000 7f6ee97fa640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 0x7f6ecc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:04.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.630+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ee4088bc0 con 0x7f6eec0b3db0 2023-11-24T08:24:04.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.630+0000 7f6eebfff640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 0x7f6ecc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:04.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.631+0000 7f6eebfff640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 0x7f6ecc06f820 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f6edc036000 tx=0x7f6edc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.634+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6eec0b2630 con 0x7f6eec0b3db0 2023-11-24T08:24:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.848+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1655", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eec0b1cb0 con 0x7f6ecc06d360 2023-11-24T08:24:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.856+0000 7f6ee97fa640 1 -- 172.21.15.107:0/2442608189 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f6edc02c980 con 0x7f6ecc06d360 2023-11-24T08:24:04.857 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f 2023-11-24T08:24:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 msgr2=0x7f6ecc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 0x7f6ecc06f820 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f6edc036000 tx=0x7f6edc030000 comp rx=0 tx=0).stop 2023-11-24T08:24:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 msgr2=0x7f6eec0b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f6eec06e770 tx=0x7f6ee400b4a0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 shutdown_connections 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6ecc06d360 0x7f6ecc06f820 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0b3db0 0x7f6eec0b4260 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eec0b0100 0x7f6eec15e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 --2- 172.21.15.107:0/2442608189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ed8003970 0x7f6eec1e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 >> 172.21.15.107:0/2442608189 conn(0x7f6eec0b0c50 msgr2=0x7f6eec071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.858+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 shutdown_connections 2023-11-24T08:24:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:04.859+0000 7f6ef2301640 1 -- 172.21.15.107:0/2442608189 wait complete. 2023-11-24T08:24:04.876 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol1655 with 16 files each 1MB size under directory volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f 2023-11-24T08:24:04.876 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:04.876 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:04.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:04.929 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327572 s, 32.0 MB/s 2023-11-24T08:24:05.025 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.025 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.079 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.206 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270235 s, 38.8 MB/s 2023-11-24T08:24:05.210 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.210 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.268 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.349 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027318 s, 38.4 MB/s 2023-11-24T08:24:05.350 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.351 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.410 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208266 s, 50.3 MB/s 2023-11-24T08:24:05.484 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.484 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.541 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.618 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.618 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.619 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256249 s, 40.9 MB/s 2023-11-24T08:24:05.620 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.620 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.675 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.750 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.750 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.750 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221881 s, 47.3 MB/s 2023-11-24T08:24:05.751 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.751 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.811 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:05.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:05.886 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:05.886 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219444 s, 47.8 MB/s 2023-11-24T08:24:05.888 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:05.888 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:05.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:05.942 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265052 s, 39.6 MB/s 2023-11-24T08:24:06.025 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.025 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.083 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238806 s, 43.9 MB/s 2023-11-24T08:24:06.163 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.163 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.222 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.289 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161858 s, 64.8 MB/s 2023-11-24T08:24:06.290 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.291 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.342 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261191 s, 40.1 MB/s 2023-11-24T08:24:06.469 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.470 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017747 s, 59.1 MB/s 2023-11-24T08:24:06.601 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.601 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197935 s, 53.0 MB/s 2023-11-24T08:24:06.730 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.730 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240783 s, 43.5 MB/s 2023-11-24T08:24:06.857 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.858 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:06.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:06.915 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244588 s, 42.9 MB/s 2023-11-24T08:24:06.993 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:06.993 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:07.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:07.045 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/7139aedc-645a-4753-b52a-6102e72ab83f/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:07.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:07.122 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:07.122 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240745 s, 43.6 MB/s 2023-11-24T08:24:07.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol1655 subvol_snap4297 2023-11-24T08:24:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.350+0000 7faed8d33640 1 Processor -- start 2023-11-24T08:24:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.350+0000 7faed8d33640 1 -- start start 2023-11-24T08:24:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.351+0000 7faed8d33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40cfb60 0x7faed40abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.351+0000 7faed8d33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed40ce640 0x7faed40b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.351+0000 7faed8d33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed40ce270 con 0x7faed40cc470 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.351+0000 7faed8d33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed406c560 con 0x7faed40ce640 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.351+0000 7faed8d33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed406d0a0 con 0x7faed40cfb60 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed40ce640 0x7faed40b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed40ce640 0x7faed40b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38680/0 (socket says 172.21.15.107:38680) 2023-11-24T08:24:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 -- 172.21.15.107:0/337317052 learned_addr learned my addr 172.21.15.107:0/337317052 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 -- 172.21.15.107:0/337317052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40cfb60 msgr2=0x7faed40abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 --2- 172.21.15.107:0/337317052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40cfb60 0x7faed40abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 -- 172.21.15.107:0/337317052 >> v1:172.21.15.120:6789/0 conn(0x7faed40cc470 legacy=0x7faed40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.352+0000 7faed2d9d640 1 -- 172.21.15.107:0/337317052 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40b4200 con 0x7faed40ce640 2023-11-24T08:24:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.353+0000 7faed2d9d640 1 --2- 172.21.15.107:0/337317052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40ce640 0x7faed40b3cc0 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7faecc009e50 tx=0x7faecc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8070012272ead6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.353+0000 7faed1d9b640 1 -- 172.21.15.107:0/337317052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc02ca70 con 0x7faed40ce640 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.353+0000 7faed1d9b640 1 -- 172.21.15.107:0/337317052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc02cbd0 con 0x7faed40ce640 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.353+0000 7faed1d9b640 1 -- 172.21.15.107:0/337317052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc0343f0 con 0x7faed40ce640 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 -- 172.21.15.107:0/337317052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40ce640 msgr2=0x7faed40b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 --2- 172.21.15.107:0/337317052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40ce640 0x7faed40b3cc0 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7faecc009e50 tx=0x7faecc02bed0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 -- 172.21.15.107:0/337317052 shutdown_connections 2023-11-24T08:24:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 --2- 172.21.15.107:0/337317052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40ce640 0x7faed40b3cc0 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 --2- 172.21.15.107:0/337317052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40cfb60 0x7faed40abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 -- 172.21.15.107:0/337317052 >> 172.21.15.107:0/337317052 conn(0x7faed406a920 msgr2=0x7faed406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 -- 172.21.15.107:0/337317052 shutdown_connections 2023-11-24T08:24:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.354+0000 7faed8d33640 1 -- 172.21.15.107:0/337317052 wait complete. 2023-11-24T08:24:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.356+0000 7faed8d33640 1 Processor -- start 2023-11-24T08:24:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.356+0000 7faed8d33640 1 -- start start 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40ce640 0x7faed4159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40cfb60 0x7faed415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed415a660 con 0x7faed40cc470 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed415afb0 con 0x7faed40cfb60 2023-11-24T08:24:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed8d33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed415a380 con 0x7faed40ce640 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38692/0 (socket says 172.21.15.107:38692) 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.357+0000 7faed259c640 1 -- 172.21.15.107:0/2581888575 learned_addr learned my addr 172.21.15.107:0/2581888575 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faecbfff640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40ce640 0x7faed4159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 -- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40ce640 msgr2=0x7faed4159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40ce640 0x7faed4159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed2d9d640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40cfb60 0x7faed415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 -- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40cfb60 msgr2=0x7faed415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40cfb60 0x7faed415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 -- 172.21.15.107:0/2581888575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc009b30 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed259c640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7faec400c410 tx=0x7faec400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40035f0 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec4003750 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4003b40 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed415a960 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.358+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed40716a0 con 0x7faed40cc470 2023-11-24T08:24:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.360+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed415a7c0 con 0x7faed40cc470 2023-11-24T08:24:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.361+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 49) v1 ==== 92648+0+0 (secure 0 0 0) 0x7faec4003ca0 con 0x7faed40cc470 2023-11-24T08:24:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.362+0000 7faecaffd640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 0x7faeb406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.362+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faec4087ab0 con 0x7faed40cc470 2023-11-24T08:24:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.362+0000 7faecbfff640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 0x7faeb406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.362+0000 7faecbfff640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 0x7faeb406f8e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7faed415cef0 tx=0x7faebc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.366+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faed415a7c0 con 0x7faed40cc470 2023-11-24T08:24:07.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.587+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol1655", "snap_name": "subvol_snap4297", "target": ["mon-mgr", ""]}) v1 -- 0x7faed406cd30 con 0x7faeb406d420 2023-11-24T08:24:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.619+0000 7faecaffd640 1 -- 172.21.15.107:0/2581888575 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faebc00a380 con 0x7faeb406d420 2023-11-24T08:24:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 msgr2=0x7faeb406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 0x7faeb406f8e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7faed415cef0 tx=0x7faebc008040 comp rx=0 tx=0).stop 2023-11-24T08:24:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 msgr2=0x7faed4156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7faec400c410 tx=0x7faec400c8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 shutdown_connections 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7faeb406d420 0x7faeb406f8e0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40cfb60 0x7faed415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40ce640 0x7faed4159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 --2- 172.21.15.107:0/2581888575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed40cc470 0x7faed4156b20 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 >> 172.21.15.107:0/2581888575 conn(0x7faed406a920 msgr2=0x7faed4160190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 shutdown_connections 2023-11-24T08:24:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.623+0000 7faed8d33640 1 -- 172.21.15.107:0/2581888575 wait complete. 2023-11-24T08:24:07.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol1655 --retain-snapshots 2023-11-24T08:24:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.861+0000 7f1b7f431640 1 Processor -- start 2023-11-24T08:24:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.862+0000 7f1b7f431640 1 -- start start 2023-11-24T08:24:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 0x7f1b780acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780b2400 0x7f1b7806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b780af400 0x7f1b7806aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b780cfed0 con 0x7f1b780af400 2023-11-24T08:24:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b780caae0 con 0x7f1b7815d920 2023-11-24T08:24:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7f431640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b780cadc0 con 0x7f1b780b2400 2023-11-24T08:24:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7d9ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b780af400 0x7f1b7806aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7d9ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1b780af400 0x7f1b7806aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38700/0 (socket says 172.21.15.107:38700) 2023-11-24T08:24:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.863+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/3189963101 learned_addr learned my addr 172.21.15.107:0/3189963101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7c9cc640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780b2400 0x7f1b7806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/3189963101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780b2400 msgr2=0x7f1b7806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d1cd640 1 --2- 172.21.15.107:0/3189963101 >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 0x7f1b780acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780b2400 0x7f1b7806a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/3189963101 >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 msgr2=0x7f1b780acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/3189963101 >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 0x7f1b780acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/3189963101 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b780cb0a0 con 0x7f1b780af400 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.864+0000 7f1b7d1cd640 1 --2- 172.21.15.107:0/3189963101 >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 0x7f1b780acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.865+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b780af400 0x7f1b7806aa40 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f1b780d1200 tx=0x7f1b7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5b015ae202c6c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.865+0000 7f1b77fff640 1 -- 172.21.15.107:0/3189963101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70030860 con 0x7f1b780af400 2023-11-24T08:24:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.865+0000 7f1b77fff640 1 -- 172.21.15.107:0/3189963101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b700309c0 con 0x7f1b780af400 2023-11-24T08:24:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.865+0000 7f1b77fff640 1 -- 172.21.15.107:0/3189963101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70030d50 con 0x7f1b780af400 2023-11-24T08:24:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 -- 172.21.15.107:0/3189963101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b780af400 msgr2=0x7f1b7806aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b780af400 0x7f1b7806aa40 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f1b780d1200 tx=0x7f1b7002b9a0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 -- 172.21.15.107:0/3189963101 shutdown_connections 2023-11-24T08:24:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b780af400 0x7f1b7806aa40 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 --2- 172.21.15.107:0/3189963101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780b2400 0x7f1b7806a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 --2- 172.21.15.107:0/3189963101 >> v2:172.21.15.120:3300/0 conn(0x7f1b7815d920 0x7f1b780acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 -- 172.21.15.107:0/3189963101 >> 172.21.15.107:0/3189963101 conn(0x7f1b780999d0 msgr2=0x7f1b780b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 -- 172.21.15.107:0/3189963101 shutdown_connections 2023-11-24T08:24:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.866+0000 7f1b7f431640 1 -- 172.21.15.107:0/3189963101 wait complete. 2023-11-24T08:24:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.868+0000 7f1b7f431640 1 Processor -- start 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.868+0000 7f1b7f431640 1 -- start start 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780af400 0x7f1b780b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b780b2400 0x7f1b780c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781f3590 con 0x7f1b7815d920 2023-11-24T08:24:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7815ba20 con 0x7f1b780b2400 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7f431640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7815bd00 con 0x7f1b780af400 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38702/0 (socket says 172.21.15.107:38702) 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/2227275294 learned_addr learned my addr 172.21.15.107:0/2227275294 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d1cd640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780af400 0x7f1b780b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780af400 msgr2=0x7f1b780b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780af400 0x7f1b780b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b780b2400 msgr2=0x7f1b780c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7c9cc640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b780b2400 0x7f1b780c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b780b2400 0x7f1b780c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.869+0000 7f1b7d9ce640 1 -- 172.21.15.107:0/2227275294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700094d0 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b7d9ce640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f1b7002b9e0 tx=0x7f1b70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70039c20 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b70039da0 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70043630 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7815bfe0 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.870+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7814bc00 con 0x7f1b7815d920 2023-11-24T08:24:07.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.871+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b780ceb30 con 0x7f1b7815d920 2023-11-24T08:24:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.872+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 49) v1 ==== 92648+0+0 (secure 0 0 0) 0x7f1b7004c3b0 con 0x7f1b7815d920 2023-11-24T08:24:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.874+0000 7f1b75ffb640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 0x7f1b6406f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.874+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b700ab050 con 0x7f1b7815d920 2023-11-24T08:24:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.874+0000 7f1b7d1cd640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 0x7f1b6406f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.875+0000 7f1b7d1cd640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 0x7f1b6406f8b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1b780c3e70 tx=0x7f1b680084f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:07.878+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b70077b60 con 0x7f1b7815d920 2023-11-24T08:24:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.095+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol1655", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1b780ad510 con 0x7f1b6406d3f0 2023-11-24T08:24:08.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.164+0000 7f1b75ffb640 1 -- 172.21.15.107:0/2227275294 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1b6800b420 con 0x7f1b6406d3f0 2023-11-24T08:24:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.167+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 msgr2=0x7f1b6406f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.167+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 0x7f1b6406f8b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1b780c3e70 tx=0x7f1b680084f0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.167+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 msgr2=0x7f1b7814dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.167+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f1b7002b9e0 tx=0x7f1b70004730 comp rx=0 tx=0).stop 2023-11-24T08:24:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 shutdown_connections 2023-11-24T08:24:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f1b6406d3f0 0x7f1b6406f8b0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b7815d920 0x7f1b7814dc90 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b780b2400 0x7f1b780c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 --2- 172.21.15.107:0/2227275294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b780af400 0x7f1b780b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 >> 172.21.15.107:0/2227275294 conn(0x7f1b780999d0 msgr2=0x7f1b7806de30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 shutdown_connections 2023-11-24T08:24:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.168+0000 7f1b7f431640 1 -- 172.21.15.107:0/2227275294 wait complete. 2023-11-24T08:24:08.174 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.174+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.175 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.174+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.175 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.174+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.175 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.174+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.175 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.174+0000 7f37667a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol1655 --mode=777 2023-11-24T08:24:08.198 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.195+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.198 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.195+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.198 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.195+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.198 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.195+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.199 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:08.195+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.396+0000 7fb951728640 1 Processor -- start 2023-11-24T08:24:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.397+0000 7fb951728640 1 -- start start 2023-11-24T08:24:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb951728640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb94c1431f0 0x7fb94c15e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb951728640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb94c06ddb0 0x7fb94c071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb951728640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c15ec20 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb951728640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0b4fd0 con 0x7fb94c06ddb0 2023-11-24T08:24:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb951728640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c143c10 con 0x7fb94c1431f0 2023-11-24T08:24:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.398+0000 7fb950f27640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb94c06ddb0 0x7fb94c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb94c06ddb0 0x7fb94c071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37630/0 (socket says 172.21.15.107:37630) 2023-11-24T08:24:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 -- 172.21.15.107:0/2361862371 learned_addr learned my addr 172.21.15.107:0/2361862371 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 -- 172.21.15.107:0/2361862371 >> v2:172.21.15.120:3300/0 conn(0x7fb94c1431f0 msgr2=0x7fb94c15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 --2- 172.21.15.107:0/2361862371 >> v2:172.21.15.120:3300/0 conn(0x7fb94c1431f0 0x7fb94c15e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 -- 172.21.15.107:0/2361862371 >> v1:172.21.15.120:6789/0 conn(0x7fb94c0b2ad0 legacy=0x7fb94c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.399+0000 7fb950f27640 1 -- 172.21.15.107:0/2361862371 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0705b0 con 0x7fb94c06ddb0 2023-11-24T08:24:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.400+0000 7fb950f27640 1 --2- 172.21.15.107:0/2361862371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c06ddb0 0x7fb94c071650 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b5110 tx=0x7fb94402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713478ef930f4018 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.400+0000 7fb94b7fe640 1 -- 172.21.15.107:0/2361862371 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94402c9e0 con 0x7fb94c06ddb0 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.400+0000 7fb94b7fe640 1 -- 172.21.15.107:0/2361862371 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94402cb40 con 0x7fb94c06ddb0 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb94b7fe640 1 -- 172.21.15.107:0/2361862371 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440343f0 con 0x7fb94c06ddb0 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 -- 172.21.15.107:0/2361862371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c06ddb0 msgr2=0x7fb94c071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 --2- 172.21.15.107:0/2361862371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c06ddb0 0x7fb94c071650 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b5110 tx=0x7fb94402be40 comp rx=0 tx=0).stop 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 -- 172.21.15.107:0/2361862371 shutdown_connections 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 --2- 172.21.15.107:0/2361862371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c06ddb0 0x7fb94c071650 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 --2- 172.21.15.107:0/2361862371 >> v2:172.21.15.120:3300/0 conn(0x7fb94c1431f0 0x7fb94c15e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 -- 172.21.15.107:0/2361862371 >> 172.21.15.107:0/2361862371 conn(0x7fb94c0cb8a0 msgr2=0x7fb94c0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 -- 172.21.15.107:0/2361862371 shutdown_connections 2023-11-24T08:24:08.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.401+0000 7fb951728640 1 -- 172.21.15.107:0/2361862371 wait complete. 2023-11-24T08:24:08.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.403+0000 7fb951728640 1 Processor -- start 2023-11-24T08:24:08.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.403+0000 7fb951728640 1 -- start start 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb94c06ddb0 0x7fb94c137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb94c1431f0 0x7fb94c0c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f7550 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f2f20 con 0x7fb94c1431f0 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb951728640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f69d0 con 0x7fb94c06ddb0 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb943fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37642/0 (socket says 172.21.15.107:37642) 2023-11-24T08:24:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.404+0000 7fb943fff640 1 -- 172.21.15.107:0/1043033395 learned_addr learned my addr 172.21.15.107:0/1043033395 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb94bfff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb94c06ddb0 0x7fb94c137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb950f27640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb94c1431f0 0x7fb94c0c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 -- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb94c06ddb0 msgr2=0x7fb94c137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb94c06ddb0 0x7fb94c137b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 -- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb94c1431f0 msgr2=0x7fb94c0c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb94c1431f0 0x7fb94c0c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 -- 172.21.15.107:0/1043033395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944009b30 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb943fff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fb93400c400 tx=0x7fb93400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.405+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340054e0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.406+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb934003640 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.406+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340039f0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.406+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c1f6cb0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.406+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c1f7270 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.407+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c1f30e0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.408+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 49) v1 ==== 92648+0+0 (secure 0 0 0) 0x7fb934003b50 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.409+0000 7fb9497fa640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 0x7fb92c06f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.409+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb934087bc0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.409+0000 7fb94bfff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 0x7fb92c06f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.410+0000 7fb94bfff640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 0x7fb92c06f8b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b810 tx=0x7fb93c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.413+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb94c1f30e0 con 0x7fb94c0b2ad0 2023-11-24T08:24:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:08.631+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol1655", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fb94c070e50 con 0x7fb92c06d3f0 2023-11-24T08:24:10.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.265+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7fb934054aa0 con 0x7fb94c0b2ad0 2023-11-24T08:24:10.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.409+0000 7fb9497fa640 1 -- 172.21.15.107:0/1043033395 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb93c00b400 con 0x7fb92c06d3f0 2023-11-24T08:24:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 msgr2=0x7fb92c06f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 0x7fb92c06f8b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b810 tx=0x7fb93c008040 comp rx=0 tx=0).stop 2023-11-24T08:24:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 msgr2=0x7fb94c13e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fb93400c400 tx=0x7fb93400c8c0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 shutdown_connections 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb92c06d3f0 0x7fb92c06f8b0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb94c1431f0 0x7fb94c0c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb94c0b2ad0 0x7fb94c13e3c0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 --2- 172.21.15.107:0/1043033395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb94c06ddb0 0x7fb94c137b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 >> 172.21.15.107:0/1043033395 conn(0x7fb94c0cb8a0 msgr2=0x7fb94c06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 shutdown_connections 2023-11-24T08:24:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.413+0000 7fb951728640 1 -- 172.21.15.107:0/1043033395 wait complete. 2023-11-24T08:24:10.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1655 2023-11-24T08:24:10.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.644+0000 7fd0e7c90640 1 Processor -- start 2023-11-24T08:24:10.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.644+0000 7fd0e7c90640 1 -- start start 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e7c90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd0e00cb850 0x7fd0e00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e7c90640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 0x7fd0e00cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e7c90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e00cc170 con 0x7fd0e00d0a00 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e7c90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e0070680 con 0x7fd0e00cb850 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e7c90640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e0070870 con 0x7fd0e00b40d0 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e5a2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd0e00cb850 0x7fd0e00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e522b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd0e00d0a00 0x7fd0e006ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43572/0 (socket says 172.21.15.107:43572) 2023-11-24T08:24:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e522b640 1 -- 172.21.15.107:0/2135506667 learned_addr learned my addr 172.21.15.107:0/2135506667 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e622d640 1 --2- 172.21.15.107:0/2135506667 >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 0x7fd0e00cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/2135506667 >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 msgr2=0x7fd0e00cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e5a2c640 1 --2- 172.21.15.107:0/2135506667 >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 0x7fd0e00cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/2135506667 >> v1:172.21.15.107:6789/0 conn(0x7fd0e00d0a00 legacy=0x7fd0e006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e4a2a640 1 -- 172.21.15.107:0/2135506667 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120270269 0 0) 0x7fd0e00cc170 con 0x7fd0e00d0a00 2023-11-24T08:24:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/2135506667 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e0070b50 con 0x7fd0e00cb850 2023-11-24T08:24:10.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.646+0000 7fd0e622d640 1 --2- 172.21.15.107:0/2135506667 >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 0x7fd0e00cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:10.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e5a2c640 1 --2- 172.21.15.107:0/2135506667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00cb850 0x7fd0e00cbc30 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fd0e006df10 tx=0x7fd0dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d127ba5a49db94b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:10.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e4a2a640 1 -- 172.21.15.107:0/2135506667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc02e070 con 0x7fd0e00cb850 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e4a2a640 1 -- 172.21.15.107:0/2135506667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc0315a0 con 0x7fd0e00cb850 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e4a2a640 1 -- 172.21.15.107:0/2135506667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc031930 con 0x7fd0e00cb850 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e7c90640 1 -- 172.21.15.107:0/2135506667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00cb850 msgr2=0x7fd0e00cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.648+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/2135506667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00cb850 0x7fd0e00cbc30 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fd0e006df10 tx=0x7fd0dc02b840 comp rx=0 tx=0).stop 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 -- 172.21.15.107:0/2135506667 shutdown_connections 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/2135506667 >> v2:172.21.15.120:3300/0 conn(0x7fd0e00b40d0 0x7fd0e00cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/2135506667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00cb850 0x7fd0e00cbc30 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 -- 172.21.15.107:0/2135506667 >> 172.21.15.107:0/2135506667 conn(0x7fd0e006e7d0 msgr2=0x7fd0e006ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 -- 172.21.15.107:0/2135506667 shutdown_connections 2023-11-24T08:24:10.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.649+0000 7fd0e7c90640 1 -- 172.21.15.107:0/2135506667 wait complete. 2023-11-24T08:24:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.650+0000 7fd0e7c90640 1 Processor -- start 2023-11-24T08:24:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.650+0000 7fd0e7c90640 1 -- start start 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e00cb850 0x7fd0e014c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e00d0a00 0x7fd0e00c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0157a30 con 0x7fd0e00b40d0 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00c52a0 con 0x7fd0e00cb850 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.651+0000 7fd0e7c90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00c5580 con 0x7fd0e00d0a00 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37650/0 (socket says 172.21.15.107:37650) 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/4083140722 learned_addr learned my addr 172.21.15.107:0/4083140722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e00d0a00 msgr2=0x7fd0e00c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e00d0a00 0x7fd0e00c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e00cb850 msgr2=0x7fd0e014c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e00cb850 0x7fd0e014c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 -- 172.21.15.107:0/4083140722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc009400 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.652+0000 7fd0e5a2c640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009890 tx=0x7fd0dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.653+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc032900 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.653+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc032a80 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.653+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc032e30 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.653+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e00c5860 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.653+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e00c5e20 con 0x7fd0e00b40d0 2023-11-24T08:24:10.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.654+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e00c6010 con 0x7fd0e00b40d0 2023-11-24T08:24:10.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.655+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7fd0dc03b3f0 con 0x7fd0e00b40d0 2023-11-24T08:24:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.656+0000 7fd0d67fc640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 0x7fd0c406f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.656+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0dc0aaaf0 con 0x7fd0e00b40d0 2023-11-24T08:24:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.656+0000 7fd0e522b640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 0x7fd0c406f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.657+0000 7fd0e522b640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 0x7fd0c406f9d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0007fd0 tx=0x7fd0d0007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:10.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.661+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0dc077600 con 0x7fd0e00b40d0 2023-11-24T08:24:10.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.888+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1655", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0e00cf150 con 0x7fd0c406d510 2023-11-24T08:24:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.908+0000 7fd0d67fc640 1 -- 172.21.15.107:0/4083140722 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fd0d0001220 con 0x7fd0c406d510 2023-11-24T08:24:10.909 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 msgr2=0x7fd0c406f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 0x7fd0c406f9d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0007fd0 tx=0x7fd0d0007960 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 msgr2=0x7fd0e00c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009890 tx=0x7fd0dc0046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 shutdown_connections 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0c406d510 0x7fd0c406f9d0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e00d0a00 0x7fd0e00c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e00cb850 0x7fd0e014c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 --2- 172.21.15.107:0/4083140722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0e00b40d0 0x7fd0e00c6340 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 >> 172.21.15.107:0/4083140722 conn(0x7fd0e006e7d0 msgr2=0x7fd0e006cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:10.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 shutdown_connections 2023-11-24T08:24:10.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:10.911+0000 7fd0e7c90640 1 -- 172.21.15.107:0/4083140722 wait complete. 2023-11-24T08:24:10.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1655 2023-11-24T08:24:11.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.137+0000 7f00d4ba9640 1 Processor -- start 2023-11-24T08:24:11.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.138+0000 7f00d4ba9640 1 -- start start 2023-11-24T08:24:11.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 0x7f00d015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00d00c49f0 0x7f00d006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d00cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d006e0d0 con 0x7f00d00c49f0 2023-11-24T08:24:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d015ec90 con 0x7f00d0072160 2023-11-24T08:24:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.139+0000 7f00d4ba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d00cd530 con 0x7f00d006dc40 2023-11-24T08:24:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00d00c49f0 0x7f00d006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00d00c49f0 0x7f00d006d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37660/0 (socket says 172.21.15.107:37660) 2023-11-24T08:24:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 -- 172.21.15.107:0/485570227 learned_addr learned my addr 172.21.15.107:0/485570227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 -- 172.21.15.107:0/485570227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 msgr2=0x7f00d00cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 --2- 172.21.15.107:0/485570227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d00cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00ce59c640 1 --2- 172.21.15.107:0/485570227 >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 0x7f00d015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 -- 172.21.15.107:0/485570227 >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 msgr2=0x7f00d015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 --2- 172.21.15.107:0/485570227 >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 0x7f00d015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00c7fff640 1 -- 172.21.15.107:0/485570227 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d00c4ed0 con 0x7f00d00c49f0 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.140+0000 7f00ce59c640 1 --2- 172.21.15.107:0/485570227 >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 0x7f00d015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.141+0000 7f00c7fff640 1 --2- 172.21.15.107:0/485570227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d00c49f0 0x7f00d006d700 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f00d00ccae0 tx=0x7f00b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34445a5495fc12d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00cdd9b640 1 -- 172.21.15.107:0/485570227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8031870 con 0x7f00d00c49f0 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00cdd9b640 1 -- 172.21.15.107:0/485570227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b80319d0 con 0x7f00d00c49f0 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00cdd9b640 1 -- 172.21.15.107:0/485570227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8031d60 con 0x7f00d00c49f0 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 -- 172.21.15.107:0/485570227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d00c49f0 msgr2=0x7f00d006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/485570227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d00c49f0 0x7f00d006d700 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f00d00ccae0 tx=0x7f00b802b910 comp rx=0 tx=0).stop 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 -- 172.21.15.107:0/485570227 shutdown_connections 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/485570227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d00cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/485570227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d00c49f0 0x7f00d006d700 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/485570227 >> v2:172.21.15.120:3300/0 conn(0x7f00d0072160 0x7f00d015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.142+0000 7f00d4ba9640 1 -- 172.21.15.107:0/485570227 >> 172.21.15.107:0/485570227 conn(0x7f00d00b0c50 msgr2=0x7f00d00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.143+0000 7f00d4ba9640 1 -- 172.21.15.107:0/485570227 shutdown_connections 2023-11-24T08:24:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.143+0000 7f00d4ba9640 1 -- 172.21.15.107:0/485570227 wait complete. 2023-11-24T08:24:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.144+0000 7f00d4ba9640 1 Processor -- start 2023-11-24T08:24:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.144+0000 7f00d4ba9640 1 -- start start 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00d00c49f0 0x7f00d014e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d0156360 con 0x7f00d0072160 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d0156ee0 con 0x7f00d00c49f0 2023-11-24T08:24:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.145+0000 7f00d4ba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01565b0 con 0x7f00d006dc40 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37674/0 (socket says 172.21.15.107:37674) 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 -- 172.21.15.107:0/2715701815 learned_addr learned my addr 172.21.15.107:0/2715701815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 -- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 msgr2=0x7f00d015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 -- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00d00c49f0 msgr2=0x7f00d014e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00d00c49f0 0x7f00d014e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 -- 172.21.15.107:0/2715701815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b80094d0 con 0x7f00d0072160 2023-11-24T08:24:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.146+0000 7f00c7fff640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f00b802b950 tx=0x7f00b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.147+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8031820 con 0x7f00d0072160 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.147+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b80443f0 con 0x7f00d0072160 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.147+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b80447a0 con 0x7f00d0072160 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.147+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d0156890 con 0x7f00d0072160 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.147+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d014ed10 con 0x7f00d0072160 2023-11-24T08:24:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.148+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d004e8b0 con 0x7f00d0072160 2023-11-24T08:24:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.149+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7f00b804d450 con 0x7f00d0072160 2023-11-24T08:24:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.150+0000 7f00c6ffd640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 0x7f00b006fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.150+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00b802d040 con 0x7f00d0072160 2023-11-24T08:24:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.150+0000 7f00ce59c640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 0x7f00b006fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.151+0000 7f00ce59c640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 0x7f00b006fa00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f00c0006fd0 tx=0x7f00c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.155+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00b8077b30 con 0x7f00d0072160 2023-11-24T08:24:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.378+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1655", "target": ["mon-mgr", ""]}) v1 -- 0x7f00d0157050 con 0x7f00b006d540 2023-11-24T08:24:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.386+0000 7f00c6ffd640 1 -- 172.21.15.107:0/2715701815 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f00d0157050 con 0x7f00b006d540 2023-11-24T08:24:11.387 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0 2023-11-24T08:24:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.388+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 msgr2=0x7f00b006fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.388+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 0x7f00b006fa00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f00c0006fd0 tx=0x7f00c0008040 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.388+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 msgr2=0x7f00d01532c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.388+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f00b802b950 tx=0x7f00b80046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.388+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 shutdown_connections 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f00b006d540 0x7f00b006fa00 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00d00c49f0 0x7f00d014e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00d0072160 0x7f00d01532c0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 --2- 172.21.15.107:0/2715701815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00d006dc40 0x7f00d015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 >> 172.21.15.107:0/2715701815 conn(0x7f00d00b0c50 msgr2=0x7f00d015e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 shutdown_connections 2023-11-24T08:24:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:11.389+0000 7f00d4ba9640 1 -- 172.21.15.107:0/2715701815 wait complete. 2023-11-24T08:24:11.407 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol1655 with 16 files each 1MB size under directory volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0 2023-11-24T08:24:11.408 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:11.408 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:11.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:11.458 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246136 s, 42.6 MB/s 2023-11-24T08:24:11.586 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:11.586 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:11.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:11.646 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:11.720 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205155 s, 51.1 MB/s 2023-11-24T08:24:11.721 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:11.721 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:11.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:11.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:11.854 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:11.854 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:11.854 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270107 s, 38.8 MB/s 2023-11-24T08:24:11.855 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:11.856 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:11.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:11.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:11.990 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:11.990 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:11.990 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275509 s, 38.1 MB/s 2023-11-24T08:24:11.992 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:11.992 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.050 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.141 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338782 s, 31.0 MB/s 2023-11-24T08:24:12.143 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.143 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.202 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027587 s, 38.0 MB/s 2023-11-24T08:24:12.284 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.284 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.340 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.414 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.415 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.415 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212645 s, 49.3 MB/s 2023-11-24T08:24:12.416 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.416 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263535 s, 39.8 MB/s 2023-11-24T08:24:12.550 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.550 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.678 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.678 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.678 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221378 s, 47.4 MB/s 2023-11-24T08:24:12.679 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.679 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.737 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.819 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.819 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.819 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264281 s, 39.7 MB/s 2023-11-24T08:24:12.821 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:12.821 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:12.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:12.873 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:12.998 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:12.999 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:12.999 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023371 s, 44.9 MB/s 2023-11-24T08:24:13.000 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:13.000 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:13.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:13.056 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:13.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:13.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:13.130 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172171 s, 60.9 MB/s 2023-11-24T08:24:13.131 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:13.131 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:13.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:13.184 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252946 s, 41.5 MB/s 2023-11-24T08:24:13.265 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:13.265 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:13.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:13.320 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:13.397 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:13.397 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:13.397 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023945 s, 43.8 MB/s 2023-11-24T08:24:13.399 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:13.399 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:13.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:13.456 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:13.530 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:13.530 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:13.530 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205235 s, 51.1 MB/s 2023-11-24T08:24:13.531 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:13.532 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:13.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:13.589 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol1655/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:24:13.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:24:13.665 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:24:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212577 s, 49.3 MB/s 2023-11-24T08:24:13.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol1655 subvol_snap194 2023-11-24T08:24:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.884+0000 7fb131901640 1 Processor -- start 2023-11-24T08:24:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.884+0000 7fb131901640 1 -- start start 2023-11-24T08:24:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb131901640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb12c15c7f0 0x7fb12c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb131901640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12c0cece0 con 0x7fb12c0ce6c0 2023-11-24T08:24:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb131901640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12c06d040 con 0x7fb12c0cf450 2023-11-24T08:24:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb131901640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12c0ceaf0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb12affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb12c15c7f0 0x7fb12c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:13.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb12affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb12c15c7f0 0x7fb12c06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37690/0 (socket says 172.21.15.107:37690) 2023-11-24T08:24:13.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb12affd640 1 -- 172.21.15.107:0/784821170 learned_addr learned my addr 172.21.15.107:0/784821170 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:13.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.886+0000 7fb12b7fe640 1 --1- 172.21.15.107:0/784821170 >> v1:172.21.15.107:6789/0 conn(0x7fb12c0ce6c0 0x7fb12c0714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43578/0 (socket says 172.21.15.107:43578) 2023-11-24T08:24:13.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.887+0000 7fb12affd640 1 -- 172.21.15.107:0/784821170 >> v1:172.21.15.120:6789/0 conn(0x7fb12c0cf450 legacy=0x7fb12c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:13.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.887+0000 7fb12affd640 1 -- 172.21.15.107:0/784821170 >> v1:172.21.15.107:6789/0 conn(0x7fb12c0ce6c0 legacy=0x7fb12c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:13.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.887+0000 7fb12affd640 1 -- 172.21.15.107:0/784821170 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c06d320 con 0x7fb12c15c7f0 2023-11-24T08:24:13.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.887+0000 7fb12a7fc640 1 -- 172.21.15.107:0/784821170 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676466818 0 0) 0x7fb12c0cece0 con 0x7fb12c0ce6c0 2023-11-24T08:24:13.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.888+0000 7fb12affd640 1 --2- 172.21.15.107:0/784821170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c06db20 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009870 tx=0x7fb11c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9fc1fcd275a8d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.888+0000 7fb12a7fc640 1 -- 172.21.15.107:0/784821170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c030860 con 0x7fb12c15c7f0 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.888+0000 7fb12a7fc640 1 -- 172.21.15.107:0/784821170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c0309c0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.888+0000 7fb12a7fc640 1 -- 172.21.15.107:0/784821170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c030d50 con 0x7fb12c15c7f0 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 -- 172.21.15.107:0/784821170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 msgr2=0x7fb12c06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 --2- 172.21.15.107:0/784821170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c06db20 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009870 tx=0x7fb11c02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 -- 172.21.15.107:0/784821170 shutdown_connections 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 --2- 172.21.15.107:0/784821170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c06db20 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 -- 172.21.15.107:0/784821170 >> 172.21.15.107:0/784821170 conn(0x7fb12c06a920 msgr2=0x7fb12c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 -- 172.21.15.107:0/784821170 shutdown_connections 2023-11-24T08:24:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.889+0000 7fb131901640 1 -- 172.21.15.107:0/784821170 wait complete. 2023-11-24T08:24:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.891+0000 7fb131901640 1 Processor -- start 2023-11-24T08:24:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.891+0000 7fb131901640 1 -- start start 2023-11-24T08:24:13.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb12c0ce6c0 0x7fb12c1f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb12c0cf450 0x7fb12c154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c09cda0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c157330 con 0x7fb12c0cf450 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb131901640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c157610 con 0x7fb12c0ce6c0 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb12b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb12b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37696/0 (socket says 172.21.15.107:37696) 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb12b7fe640 1 -- 172.21.15.107:0/4024632958 learned_addr learned my addr 172.21.15.107:0/4024632958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb12affd640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb12c0ce6c0 0x7fb12c1f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:13.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.892+0000 7fb1227fc640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb12c0cf450 0x7fb12c154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 -- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb12c0ce6c0 msgr2=0x7fb12c1f1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb12c0ce6c0 0x7fb12c1f1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 -- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb12c0cf450 msgr2=0x7fb12c154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb12c0cf450 0x7fb12c154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 -- 172.21.15.107:0/4024632958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0094d0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.893+0000 7fb12b7fe640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7fb124002230 tx=0x7fb12400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:13.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.895+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400ce30 con 0x7fb12c15c7f0 2023-11-24T08:24:13.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.895+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12400a640 con 0x7fb12c15c7f0 2023-11-24T08:24:13.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.895+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c1578f0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.895+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400a9f0 con 0x7fb12c15c7f0 2023-11-24T08:24:13.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.895+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c0ad550 con 0x7fb12c15c7f0 2023-11-24T08:24:13.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.897+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c067470 con 0x7fb12c15c7f0 2023-11-24T08:24:13.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.898+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7fb12400ab50 con 0x7fb12c15c7f0 2023-11-24T08:24:13.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.898+0000 7fb128ff9640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 0x7fb10c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:13.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.899+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb124087d00 con 0x7fb12c15c7f0 2023-11-24T08:24:13.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.899+0000 7fb12affd640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 0x7fb10c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:13.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.900+0000 7fb12affd640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 0x7fb10c06fa00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb11c036000 tx=0x7fb11c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:13.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:13.903+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1240548b0 con 0x7fb12c15c7f0 2023-11-24T08:24:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.125+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol1655", "snap_name": "subvol_snap194", "target": ["mon-mgr", ""]}) v1 -- 0x7fb12c0cd730 con 0x7fb10c06d540 2023-11-24T08:24:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.175+0000 7fb128ff9640 1 -- 172.21.15.107:0/4024632958 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb12c0cd730 con 0x7fb10c06d540 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 msgr2=0x7fb10c06fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 0x7fb10c06fa00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb11c036000 tx=0x7fb11c03b040 comp rx=0 tx=0).stop 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 msgr2=0x7fb12c0ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7fb124002230 tx=0x7fb12400cbe0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 shutdown_connections 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb10c06d540 0x7fb10c06fa00 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c15c7f0 0x7fb12c0ad010 unknown :-1 s=CLOSED pgs=3236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb12c0cf450 0x7fb12c154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 --2- 172.21.15.107:0/4024632958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb12c0ce6c0 0x7fb12c1f1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 >> 172.21.15.107:0/4024632958 conn(0x7fb12c06a920 msgr2=0x7fb12c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 shutdown_connections 2023-11-24T08:24:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.179+0000 7fb131901640 1 -- 172.21.15.107:0/4024632958 wait complete. 2023-11-24T08:24:14.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol1655 --retain-snapshots 2023-11-24T08:24:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.409+0000 7f48d6fd5640 1 Processor -- start 2023-11-24T08:24:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.410+0000 7f48d6fd5640 1 -- start start 2023-11-24T08:24:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48d6fd5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48d00cf450 0x7f48d00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48d6fd5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d00cece0 con 0x7f48d015c7f0 2023-11-24T08:24:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48d6fd5640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d006d040 con 0x7f48d00cdfb0 2023-11-24T08:24:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48d6fd5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d00cfe40 con 0x7f48d00cf450 2023-11-24T08:24:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48d00cf450 0x7f48d00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48d00cf450 0x7f48d00cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37706/0 (socket says 172.21.15.107:37706) 2023-11-24T08:24:14.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.411+0000 7f48cffff640 1 -- 172.21.15.107:0/2927144828 learned_addr learned my addr 172.21.15.107:0/2927144828 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:14.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.412+0000 7f48cffff640 1 -- 172.21.15.107:0/2927144828 >> v1:172.21.15.120:6789/0 conn(0x7f48d00cdfb0 legacy=0x7f48d00ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:14.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.412+0000 7f48cffff640 1 -- 172.21.15.107:0/2927144828 >> v1:172.21.15.107:6789/0 conn(0x7f48d015c7f0 legacy=0x7f48d006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.412+0000 7f48cffff640 1 -- 172.21.15.107:0/2927144828 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d006d320 con 0x7f48d00cf450 2023-11-24T08:24:14.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.413+0000 7f48cffff640 1 --2- 172.21.15.107:0/2927144828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00cf900 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009870 tx=0x7f48bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb514aabdeb8fac3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48cf7fe640 1 -- 172.21.15.107:0/2927144828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc030860 con 0x7f48d00cf450 2023-11-24T08:24:14.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48cf7fe640 1 -- 172.21.15.107:0/2927144828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc0309c0 con 0x7f48d00cf450 2023-11-24T08:24:14.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48cf7fe640 1 -- 172.21.15.107:0/2927144828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc030d50 con 0x7f48d00cf450 2023-11-24T08:24:14.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2927144828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 msgr2=0x7f48d00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2927144828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00cf900 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009870 tx=0x7f48bc02b9a0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2927144828 shutdown_connections 2023-11-24T08:24:14.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2927144828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00cf900 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2927144828 >> 172.21.15.107:0/2927144828 conn(0x7f48d006a920 msgr2=0x7f48d006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2927144828 shutdown_connections 2023-11-24T08:24:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.414+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2927144828 wait complete. 2023-11-24T08:24:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.416+0000 7f48d6fd5640 1 Processor -- start 2023-11-24T08:24:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.416+0000 7f48d6fd5640 1 -- start start 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d00cdfb0 0x7f48d0154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d015c7f0 0x7f48d00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01589e0 con 0x7f48d00cf450 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0155e80 con 0x7f48d015c7f0 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48d6fd5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0156160 con 0x7f48d00cdfb0 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37714/0 (socket says 172.21.15.107:37714) 2023-11-24T08:24:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.417+0000 7f48cffff640 1 -- 172.21.15.107:0/2826973031 learned_addr learned my addr 172.21.15.107:0/2826973031 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 -- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d00cdfb0 msgr2=0x7f48d0154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d00cdfb0 0x7f48d0154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 -- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d015c7f0 msgr2=0x7f48d00b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d015c7f0 0x7f48d00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 -- 172.21.15.107:0/2826973031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0094d0 con 0x7f48d00cf450 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cffff640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009870 tx=0x7f48bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc030810 con 0x7f48d00cf450 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc043400 con 0x7f48d00cf450 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0437b0 con 0x7f48d00cf450 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.418+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d0156440 con 0x7f48d00cf450 2023-11-24T08:24:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.419+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d00cb170 con 0x7f48d00cf450 2023-11-24T08:24:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.420+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d00c94e0 con 0x7f48d00cf450 2023-11-24T08:24:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.421+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7f48bc043910 con 0x7f48d00cf450 2023-11-24T08:24:14.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.422+0000 7f48cd7fa640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 0x7f48b406f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.422+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48bc03e070 con 0x7f48d00cf450 2023-11-24T08:24:14.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.422+0000 7f48d4d71640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 0x7f48b406f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.423+0000 7f48d4d71640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 0x7f48b406f9d0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f48c4000f60 tx=0x7f48c40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.426+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48bc0788e0 con 0x7f48d00cf450 2023-11-24T08:24:14.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.644+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol1655", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f48d0155ff0 con 0x7f48b406d510 2023-11-24T08:24:14.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.714+0000 7f48cd7fa640 1 -- 172.21.15.107:0/2826973031 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f48d0155ff0 con 0x7f48b406d510 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.717+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 msgr2=0x7f48b406f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.717+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 0x7f48b406f9d0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f48c4000f60 tx=0x7f48c40079b0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.717+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 msgr2=0x7f48d00abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.717+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009870 tx=0x7f48bc004730 comp rx=0 tx=0).stop 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.717+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 shutdown_connections 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f48b406d510 0x7f48b406f9d0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d015c7f0 0x7f48d00b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48d00cf450 0x7f48d00abfc0 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 --2- 172.21.15.107:0/2826973031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48d00cdfb0 0x7f48d0154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 >> 172.21.15.107:0/2826973031 conn(0x7f48d006a920 msgr2=0x7f48d006db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 shutdown_connections 2023-11-24T08:24:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.718+0000 7f48d6fd5640 1 -- 172.21.15.107:0/2826973031 wait complete. 2023-11-24T08:24:14.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol1655 subvol_snap194 subvol_clone9816 2023-11-24T08:24:14.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.949+0000 7fa0f2b33640 1 Processor -- start 2023-11-24T08:24:14.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.949+0000 7fa0f2b33640 1 -- start start 2023-11-24T08:24:14.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.950+0000 7fa0f2b33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0ec15b440 0x7fa0ec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0f2b33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0c5dc0 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0f2b33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec15eea0 con 0x7fa0ec0c59b0 2023-11-24T08:24:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0f2b33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ec0cc3d0 con 0x7fa0ec15b440 2023-11-24T08:24:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0ebfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa0ec0cbf20 0x7fa0ec0c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43592/0 (socket says 172.21.15.107:43592) 2023-11-24T08:24:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0ebfff640 1 -- 172.21.15.107:0/3772766673 learned_addr learned my addr 172.21.15.107:0/3772766673 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487058330 0 0) 0x7fa0ec0c5dc0 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc003520 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.951+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/3772766673 >> v2:172.21.15.120:3300/0 conn(0x7fa0ec15b440 0x7fa0ec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589866697 0 0) 0x7fa0dc003520 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 >> v2:172.21.15.120:3300/0 conn(0x7fa0ec15b440 msgr2=0x7fa0ec0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 --2- 172.21.15.107:0/3772766673 >> v2:172.21.15.120:3300/0 conn(0x7fa0ec15b440 0x7fa0ec0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 >> v1:172.21.15.120:6789/0 conn(0x7fa0ec0c59b0 legacy=0x7fa0ec06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0abf30 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa0d8001670 con 0x7fa0ec0cbf20 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 >> v1:172.21.15.107:6789/0 conn(0x7fa0ec0cbf20 legacy=0x7fa0ec0c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0dc0039c0 0x7fa0dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0047c0 0x7fa0dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 0x7fa0dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0cc3d0 con 0x7fa0dc0052a0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc00c1c0 con 0x7fa0dc0039c0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc00c380 con 0x7fa0dc0047c0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0f10d0640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 0x7fa0dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0ebfff640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0047c0 0x7fa0dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.952+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0dc0039c0 0x7fa0dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f10d0640 1 -- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0047c0 msgr2=0x7fa0dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f10d0640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0047c0 0x7fa0dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f10d0640 1 -- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0dc0039c0 msgr2=0x7fa0dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f10d0640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0dc0039c0 0x7fa0dc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f10d0640 1 -- 172.21.15.107:0/3772766673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc003520 con 0x7fa0dc0052a0 2023-11-24T08:24:14.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.953+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0dc0039c0 0x7fa0dc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0f10d0640 1 -- 172.21.15.107:0/3772766673 reap_dead start 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0f10d0640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 0x7fa0dc00b980 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4001e50 tx=0x7fa0dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315a36056db92403 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4003b70 con 0x7fa0dc0052a0 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e4003d10 con 0x7fa0dc0052a0 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0eb7fe640 1 -- 172.21.15.107:0/3772766673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4005310 con 0x7fa0dc0052a0 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0f2b33640 1 -- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 msgr2=0x7fa0dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.954+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 0x7fa0dc00b980 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4001e50 tx=0x7fa0dc0057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.955+0000 7fa0f2b33640 1 -- 172.21.15.107:0/3772766673 shutdown_connections 2023-11-24T08:24:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.955+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/3772766673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0dc0052a0 0x7fa0dc00b980 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.955+0000 7fa0f2b33640 1 -- 172.21.15.107:0/3772766673 >> 172.21.15.107:0/3772766673 conn(0x7fa0ec0b0c50 msgr2=0x7fa0ec0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.955+0000 7fa0f2b33640 1 -- 172.21.15.107:0/3772766673 shutdown_connections 2023-11-24T08:24:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.955+0000 7fa0f2b33640 1 -- 172.21.15.107:0/3772766673 wait complete. 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.956+0000 7fa0f2b33640 1 Processor -- start 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.957+0000 7fa0f2b33640 1 -- start start 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.957+0000 7fa0f2b33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 0x7fa0ec155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.957+0000 7fa0f2b33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.957+0000 7fa0f2b33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0ec09a3e0 0x7fa0ec09a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f2b33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0addd0 con 0x7fa0ec0ad410 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f2b33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec09add0 con 0x7fa0ec09a3e0 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f2b33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec09bc00 con 0x7fa0dc0052a0 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37728/0 (socket says 172.21.15.107:37728) 2023-11-24T08:24:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 -- 172.21.15.107:0/969770768 learned_addr learned my addr 172.21.15.107:0/969770768 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 0x7fa0ec155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 -- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 msgr2=0x7fa0ec155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f10d0640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0ec09a3e0 0x7fa0ec09a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 0x7fa0ec155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 -- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0ec09a3e0 msgr2=0x7fa0ec09a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0ec09a3e0 0x7fa0ec09a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0ebfff640 1 -- 172.21.15.107:0/969770768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec09b080 con 0x7fa0ec0ad410 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.958+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 0x7fa0ec155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0ebfff640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800e730 tx=0x7fa0d800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800c6c0 con 0x7fa0ec0ad410 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d800c820 con 0x7fa0ec0ad410 2023-11-24T08:24:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec09b360 con 0x7fa0ec0ad410 2023-11-24T08:24:14.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800cbd0 con 0x7fa0ec0ad410 2023-11-24T08:24:14.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.959+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec09b920 con 0x7fa0ec0ad410 2023-11-24T08:24:14.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.960+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec067680 con 0x7fa0ec0ad410 2023-11-24T08:24:14.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.961+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7fa0d800cd30 con 0x7fa0ec0ad410 2023-11-24T08:24:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.962+0000 7fa0e97fa640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 0x7fa0d006f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.962+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0d8088870 con 0x7fa0ec0ad410 2023-11-24T08:24:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.962+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 0x7fa0d006f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.963+0000 7fa0f08cf640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 0x7fa0d006f9d0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec15ae50 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:14.967+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0d8055380 con 0x7fa0ec0ad410 2023-11-24T08:24:15.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.193+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol1655", "snap_name": "subvol_snap194", "target_sub_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0ec0ca780 con 0x7fa0d006d510 2023-11-24T08:24:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.507+0000 7fa0e97fa640 1 -- 172.21.15.107:0/969770768 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa0ec0ca780 con 0x7fa0d006d510 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 msgr2=0x7fa0d006f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 0x7fa0d006f9d0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec15ae50 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 msgr2=0x7fa0ec0ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800e730 tx=0x7fa0d800ebf0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 shutdown_connections 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa0d006d510 0x7fa0d006f9d0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0ec09a3e0 0x7fa0ec09a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0ec0ad410 0x7fa0ec0ad890 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 --2- 172.21.15.107:0/969770768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0dc0052a0 0x7fa0ec155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 >> 172.21.15.107:0/969770768 conn(0x7fa0ec0b0c50 msgr2=0x7fa0ec071940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 shutdown_connections 2023-11-24T08:24:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.511+0000 7fa0f2b33640 1 -- 172.21.15.107:0/969770768 wait complete. 2023-11-24T08:24:15.520 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.518+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.520 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.518+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.520 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.518+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.520 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.518+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.521 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.519+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9816 2023-11-24T08:24:15.548 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.547+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.549 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.547+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.549 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.547+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.549 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.547+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.549 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:24:15.547+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:24:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.730+0000 7f940de21640 1 Processor -- start 2023-11-24T08:24:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.730+0000 7f940de21640 1 -- start start 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.731+0000 7f940de21640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f940815b260 0x7f940806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.731+0000 7f940de21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f94080b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.731+0000 7f940de21640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940806d5c0 con 0x7f94080b1080 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f940de21640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94080b1530 con 0x7f940815b260 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f940de21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940815b640 con 0x7f940806e580 2023-11-24T08:24:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f940d620640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94080b1080 0x7f94080ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43608/0 (socket says 172.21.15.107:43608) 2023-11-24T08:24:15.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f940d620640 1 -- 172.21.15.107:0/3740635505 learned_addr learned my addr 172.21.15.107:0/3740635505 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:15.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f9407fff640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f94080b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f940ce1f640 1 --2- 172.21.15.107:0/3740635505 >> v2:172.21.15.120:3300/0 conn(0x7f940815b260 0x7f940806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767474156 0 0) 0x7f940806d5c0 con 0x7f94080b1080 2023-11-24T08:24:15.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.732+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f8003520 con 0x7f94080b1080 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196659451 0 0) 0x7f93f8003520 con 0x7f94080b1080 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 msgr2=0x7f94080b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f94080b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 >> v2:172.21.15.120:3300/0 conn(0x7f940815b260 msgr2=0x7f940806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 --2- 172.21.15.107:0/3740635505 >> v2:172.21.15.120:3300/0 conn(0x7f940815b260 0x7f940806e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940815ed20 con 0x7f94080b1080 2023-11-24T08:24:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f94000019f0 con 0x7f94080b1080 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 >> v1:172.21.15.107:6789/0 conn(0x7f94080b1080 legacy=0x7f94080ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f93f8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93f8003ef0 0x7f93f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93f8004d00 0x7f93f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080b1530 con 0x7f93f8004d00 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940815b640 con 0x7f93f8003ef0 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f800ba60 con 0x7f940806e580 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.733+0000 7f940d620640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93f8004d00 0x7f93f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940ce1f640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f93f8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f9407fff640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93f8003ef0 0x7f93f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940d620640 1 -- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 msgr2=0x7f93f8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940d620640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f940806e580 0x7f93f8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940d620640 1 -- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93f8003ef0 msgr2=0x7f93f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940d620640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93f8003ef0 0x7f93f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.734+0000 7f940d620640 1 -- 172.21.15.107:0/3740635505 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8003520 con 0x7f93f8004d00 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.735+0000 7f940d620640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93f8004d00 0x7f93f8005180 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f940000a8d0 tx=0x7f93f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19836eb1a1a37147 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.735+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400034630 con 0x7f93f8004d00 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.735+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940002f070 con 0x7f93f8004d00 2023-11-24T08:24:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.735+0000 7f94077fe640 1 -- 172.21.15.107:0/3740635505 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000349d0 con 0x7f93f8004d00 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 -- 172.21.15.107:0/3740635505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93f8004d00 msgr2=0x7f93f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 --2- 172.21.15.107:0/3740635505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93f8004d00 0x7f93f8005180 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f940000a8d0 tx=0x7f93f80057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940d620640 1 -- 172.21.15.107:0/3740635505 reap_dead start 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 -- 172.21.15.107:0/3740635505 shutdown_connections 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 -- 172.21.15.107:0/3740635505 >> 172.21.15.107:0/3740635505 conn(0x7f94080cbb40 msgr2=0x7f94080cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 -- 172.21.15.107:0/3740635505 shutdown_connections 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.736+0000 7f940de21640 1 -- 172.21.15.107:0/3740635505 wait complete. 2023-11-24T08:24:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.738+0000 7f940de21640 1 Processor -- start 2023-11-24T08:24:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.738+0000 7f940de21640 1 -- start start 2023-11-24T08:24:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 0x7f94081546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94080b05b0 0x7f94080b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081f3950 con 0x7f940809bba0 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081f2b00 con 0x7f94080b05b0 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940de21640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940809c560 con 0x7f93f8004d00 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.739+0000 7f940d620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37770/0 (socket says 172.21.15.107:37770) 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 -- 172.21.15.107:0/4181761699 learned_addr learned my addr 172.21.15.107:0/4181761699 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940ce1f640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 0x7f94081546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f9407fff640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94080b05b0 0x7f94080b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 -- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 msgr2=0x7f94081546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 0x7f94081546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 -- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94080b05b0 msgr2=0x7f94080b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94080b05b0 0x7f94080b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 -- 172.21.15.107:0/4181761699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94081f2d10 con 0x7f940809bba0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940ce1f640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 0x7f94081546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.740+0000 7f940d620640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f9400001cb0 tx=0x7f9400001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.741+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400034840 con 0x7f940809bba0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.741+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94000349a0 con 0x7f940809bba0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.741+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94081f2fb0 con 0x7f940809bba0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.741+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400034d50 con 0x7f940809bba0 2023-11-24T08:24:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.741+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94081f3570 con 0x7f940809bba0 2023-11-24T08:24:15.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.742+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940804e840 con 0x7f940809bba0 2023-11-24T08:24:15.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.744+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 50) v1 ==== 92750+0+0 (secure 0 0 0) 0x7f9400031030 con 0x7f940809bba0 2023-11-24T08:24:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.745+0000 7f94057fa640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 0x7f93ec06f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.745+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94000aacd0 con 0x7f940809bba0 2023-11-24T08:24:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.745+0000 7f940ce1f640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 0x7f93ec06f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.746+0000 7f940ce1f640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 0x7f93ec06f9d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009a80 tx=0x7f93fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.751+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94000779a0 con 0x7f940809bba0 2023-11-24T08:24:15.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.978+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f940815ef70 con 0x7f93ec06d510 2023-11-24T08:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.991+0000 7f94057fa640 1 -- 172.21.15.107:0/4181761699 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7f93fc012a10 con 0x7f93ec06d510 2023-11-24T08:24:15.993 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:24:15.993 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:24:15.993 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:24:15.993 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1655", 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap194" 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:15.994 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 msgr2=0x7f93ec06f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 0x7f93ec06f9d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009a80 tx=0x7f93fc008040 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 msgr2=0x7f940809c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f9400001cb0 tx=0x7f9400001360 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 shutdown_connections 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f93ec06d510 0x7f93ec06f9d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f940809bba0 0x7f940809c020 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94080b05b0 0x7f94080b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 --2- 172.21.15.107:0/4181761699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93f8004d00 0x7f94081546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 >> 172.21.15.107:0/4181761699 conn(0x7f94080cbb40 msgr2=0x7f94080ac7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 shutdown_connections 2023-11-24T08:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:15.994+0000 7f940de21640 1 -- 172.21.15.107:0/4181761699 wait complete. 2023-11-24T08:24:16.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:24:16.639 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:24:17.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9816 2023-11-24T08:24:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.242+0000 7f71a7fff640 1 Processor -- start 2023-11-24T08:24:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.242+0000 7f71a7fff640 1 -- start start 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.244+0000 7f71a7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71a80b0b40 0x7f71a80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.244+0000 7f71a7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f71a815ded0 0x7f71a815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.244+0000 7f71a7fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a806d430 con 0x7f71a815a1c0 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a7fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a815ec50 con 0x7f71a815ded0 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a7fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a806cb00 con 0x7f71a80b0b40 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71a815a1c0 0x7f71a806d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43614/0 (socket says 172.21.15.107:43614) 2023-11-24T08:24:17.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a6ffd640 1 -- 172.21.15.107:0/1552669033 learned_addr learned my addr 172.21.15.107:0/1552669033 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a77fe640 1 --2- 172.21.15.107:0/1552669033 >> v2:172.21.15.107:3300/0 conn(0x7f71a815ded0 0x7f71a815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f719ffff640 1 --2- 172.21.15.107:0/1552669033 >> v2:172.21.15.120:3300/0 conn(0x7f71a80b0b40 0x7f71a80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a67fc640 1 -- 172.21.15.107:0/1552669033 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767881323 0 0) 0x7f71a806d430 con 0x7f71a815a1c0 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a77fe640 1 -- 172.21.15.107:0/1552669033 >> v2:172.21.15.120:3300/0 conn(0x7f71a80b0b40 msgr2=0x7f71a80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a77fe640 1 --2- 172.21.15.107:0/1552669033 >> v2:172.21.15.120:3300/0 conn(0x7f71a80b0b40 0x7f71a80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a77fe640 1 -- 172.21.15.107:0/1552669033 >> v1:172.21.15.107:6789/0 conn(0x7f71a815a1c0 legacy=0x7f71a806d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.245+0000 7f71a77fe640 1 -- 172.21.15.107:0/1552669033 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a80b14e0 con 0x7f71a815ded0 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.246+0000 7f71a77fe640 1 --2- 172.21.15.107:0/1552669033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815ded0 0x7f71a815e360 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f71a80cee20 tx=0x7f71a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34186f339020fe4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a67fc640 1 -- 172.21.15.107:0/1552669033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a002cc80 con 0x7f71a815ded0 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a67fc640 1 -- 172.21.15.107:0/1552669033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a002cde0 con 0x7f71a815ded0 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a67fc640 1 -- 172.21.15.107:0/1552669033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0034660 con 0x7f71a815ded0 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 -- 172.21.15.107:0/1552669033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815ded0 msgr2=0x7f71a815e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 --2- 172.21.15.107:0/1552669033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815ded0 0x7f71a815e360 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f71a80cee20 tx=0x7f71a002bed0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 -- 172.21.15.107:0/1552669033 shutdown_connections 2023-11-24T08:24:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 --2- 172.21.15.107:0/1552669033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815ded0 0x7f71a815e360 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 --2- 172.21.15.107:0/1552669033 >> v2:172.21.15.120:3300/0 conn(0x7f71a80b0b40 0x7f71a80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 -- 172.21.15.107:0/1552669033 >> 172.21.15.107:0/1552669033 conn(0x7f71a80cbb40 msgr2=0x7f71a80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.247+0000 7f71a7fff640 1 -- 172.21.15.107:0/1552669033 shutdown_connections 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.248+0000 7f71a7fff640 1 -- 172.21.15.107:0/1552669033 wait complete. 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.249+0000 7f71a7fff640 1 Processor -- start 2023-11-24T08:24:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.249+0000 7f71a7fff640 1 -- start start 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71a80b0b40 0x7f71a81f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71a815ded0 0x7f71a809adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a80c8520 con 0x7f71a815a1c0 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a80c7980 con 0x7f71a80b0b40 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.250+0000 7f71a7fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a80c7bf0 con 0x7f71a815ded0 2023-11-24T08:24:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37790/0 (socket says 172.21.15.107:37790) 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 -- 172.21.15.107:0/925576263 learned_addr learned my addr 172.21.15.107:0/925576263 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 -- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71a815ded0 msgr2=0x7f71a809adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71a815ded0 0x7f71a809adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 -- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71a80b0b40 msgr2=0x7f71a81f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71a80b0b40 0x7f71a81f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 -- 172.21.15.107:0/925576263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0009a70 con 0x7f71a815a1c0 2023-11-24T08:24:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.251+0000 7f719ffff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f719000d830 tx=0x7f719000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.252+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719000b6c0 con 0x7f71a815a1c0 2023-11-24T08:24:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.252+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719000b820 con 0x7f71a815a1c0 2023-11-24T08:24:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.252+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719000bbb0 con 0x7f71a815a1c0 2023-11-24T08:24:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.252+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a815ea20 con 0x7f71a815a1c0 2023-11-24T08:24:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.252+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a80c81b0 con 0x7f71a815a1c0 2023-11-24T08:24:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.253+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a804e840 con 0x7f71a815a1c0 2023-11-24T08:24:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.254+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f719001e460 con 0x7f71a815a1c0 2023-11-24T08:24:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.255+0000 7f719f7fe640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 0x7f718806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.255+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71900890d0 con 0x7f71a815a1c0 2023-11-24T08:24:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.256+0000 7f71a6ffd640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 0x7f718806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:17.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.256+0000 7f71a6ffd640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 0x7f718806fb20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f7198007fd0 tx=0x7f7198007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:17.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.260+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7190055c80 con 0x7f71a815a1c0 2023-11-24T08:24:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.488+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f71a815e8a0 con 0x7f718806d660 2023-11-24T08:24:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.496+0000 7f719f7fe640 1 -- 172.21.15.107:0/925576263 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7f71980070d0 con 0x7f718806d660 2023-11-24T08:24:17.498 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:24:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:24:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:24:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol1655", 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap194" 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:17.499 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:24:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 msgr2=0x7f718806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 0x7f718806fb20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f7198007fd0 tx=0x7f7198007920 comp rx=0 tx=0).stop 2023-11-24T08:24:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 msgr2=0x7f71a809a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f719000d830 tx=0x7f719000dcf0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 shutdown_connections 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f718806d660 0x7f718806fb20 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71a815ded0 0x7f71a809adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a815a1c0 0x7f71a809a8b0 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 --2- 172.21.15.107:0/925576263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71a80b0b40 0x7f71a81f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 >> 172.21.15.107:0/925576263 conn(0x7f71a80cbb40 msgr2=0x7f71a8071190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 shutdown_connections 2023-11-24T08:24:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:17.499+0000 7f71a7fff640 1 -- 172.21.15.107:0/925576263 wait complete. 2023-11-24T08:24:18.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9816 2023-11-24T08:24:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.742+0000 7f7d30890640 1 Processor -- start 2023-11-24T08:24:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.742+0000 7f7d30890640 1 -- start start 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d30890640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d2c0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d30890640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c166510 con 0x7f7d2c1660b0 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d30890640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c1655f0 con 0x7f7d2c0ccbd0 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d30890640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0cc380 con 0x7f7d2c0c9650 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d29d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7d2c1660b0 0x7f7d2c162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45208/0 (socket says 172.21.15.107:45208) 2023-11-24T08:24:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.744+0000 7f7d29d9b640 1 -- 172.21.15.107:0/4108073730 learned_addr learned my addr 172.21.15.107:0/4108073730 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2ad9d640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d2c0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281218805 0 0) 0x7f7d20001110 con 0x7f7d2c1660b0 2023-11-24T08:24:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d14003520 con 0x7f7d2c1660b0 2023-11-24T08:24:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987761239 0 0) 0x7f7d14003520 con 0x7f7d2c1660b0 2023-11-24T08:24:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 msgr2=0x7f7d2c0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d2c0c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 >> v1:172.21.15.120:6789/0 conn(0x7f7d2c0ccbd0 legacy=0x7f7d2c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c162b30 con 0x7f7d2c1660b0 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7d20001800 con 0x7f7d2c1660b0 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 >> v1:172.21.15.107:6789/0 conn(0x7f7d2c1660b0 legacy=0x7f7d2c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d140039a0 0x7f7d14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d14004d00 0x7f7d14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0cc380 con 0x7f7d140039a0 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1400ba60 con 0x7f7d14004d00 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.745+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1400bc20 con 0x7f7d2c0c9650 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d140039a0 0x7f7d14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d29d9b640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2ad9d640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d14004d00 0x7f7d14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 -- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 msgr2=0x7f7d140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d2c0c9650 0x7f7d140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 -- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d14004d00 msgr2=0x7f7d14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d14004d00 0x7f7d14005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.746+0000 7f7d2a59c640 1 -- 172.21.15.107:0/4108073730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14003520 con 0x7f7d140039a0 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d140039a0 0x7f7d14003dc0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c001720 tx=0x7f7d140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5384096cb4ff602 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0052d0 con 0x7f7d140039a0 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c02e050 con 0x7f7d140039a0 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d2959a640 1 -- 172.21.15.107:0/4108073730 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0035f0 con 0x7f7d140039a0 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d30890640 1 -- 172.21.15.107:0/4108073730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d140039a0 msgr2=0x7f7d14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.748+0000 7f7d30890640 1 --2- 172.21.15.107:0/4108073730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d140039a0 0x7f7d14003dc0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c001720 tx=0x7f7d140057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.749+0000 7f7d2ad9d640 1 -- 172.21.15.107:0/4108073730 reap_dead start 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.749+0000 7f7d30890640 1 -- 172.21.15.107:0/4108073730 shutdown_connections 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.749+0000 7f7d30890640 1 -- 172.21.15.107:0/4108073730 >> 172.21.15.107:0/4108073730 conn(0x7f7d2c0b0c50 msgr2=0x7f7d2c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.749+0000 7f7d30890640 1 -- 172.21.15.107:0/4108073730 shutdown_connections 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.749+0000 7f7d30890640 1 -- 172.21.15.107:0/4108073730 wait complete. 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.751+0000 7f7d30890640 1 Processor -- start 2023-11-24T08:24:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.751+0000 7f7d30890640 1 -- start start 2023-11-24T08:24:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.751+0000 7f7d30890640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d140039a0 0x7f7d2c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d30890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d30890640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d2c0b4430 0x7f7d2c163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d30890640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0b2e80 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d30890640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0b29f0 con 0x7f7d2c0b4430 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d30890640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0b1de0 con 0x7f7d140039a0 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38634/0 (socket says 172.21.15.107:38634) 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 -- 172.21.15.107:0/146812042 learned_addr learned my addr 172.21.15.107:0/146812042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d140039a0 0x7f7d2c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 -- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d140039a0 msgr2=0x7f7d2c0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d140039a0 0x7f7d2c0ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d2ad9d640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d2c0b4430 0x7f7d2c163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 -- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d2c0b4430 msgr2=0x7f7d2c163050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d2c0b4430 0x7f7d2c163050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.752+0000 7f7d29d9b640 1 -- 172.21.15.107:0/146812042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0b20c0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d29d9b640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e730 tx=0x7f7d2000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000ee40 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000c8d0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000cc80 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c0b23a0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.753+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c0cab10 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.754+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c0b2ba0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.755+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f7d2001d460 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.756+0000 7f7d1affd640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 0x7f7d0c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.756+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d200886f0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.757+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 0x7f7d0c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.757+0000 7f7d2a59c640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 0x7f7d0c06faf0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c033000 tx=0x7f7d1c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.761+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d2c0b2ba0 con 0x7f7d2c0b3a70 2023-11-24T08:24:18.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:18.988+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d2c165b90 con 0x7f7d0c06d630 2023-11-24T08:24:19.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.559+0000 7f7d1affd640 1 -- 172.21.15.107:0/146812042 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f7d1c0053d0 con 0x7f7d0c06d630 2023-11-24T08:24:19.783 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:24:19.783 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:24:19.783 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:24:19.783 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:19.783 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:24:19.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 msgr2=0x7f7d0c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:19.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 0x7f7d0c06faf0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c033000 tx=0x7f7d1c009030 comp rx=0 tx=0).stop 2023-11-24T08:24:19.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 msgr2=0x7f7d2c0b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:19.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e730 tx=0x7f7d2000ebf0 comp rx=0 tx=0).stop 2023-11-24T08:24:19.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 shutdown_connections 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7d0c06d630 0x7f7d0c06faf0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d2c0b4430 0x7f7d2c163050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2c0b3a70 0x7f7d2c0b3ef0 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 --2- 172.21.15.107:0/146812042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d140039a0 0x7f7d2c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 >> 172.21.15.107:0/146812042 conn(0x7f7d2c0b0c50 msgr2=0x7f7d2c0af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:19.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 shutdown_connections 2023-11-24T08:24:19.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:19.562+0000 7f7d30890640 1 -- 172.21.15.107:0/146812042 wait complete. 2023-11-24T08:24:19.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone9816 2023-11-24T08:24:20.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.009+0000 7f46357ec640 1 Processor -- start 2023-11-24T08:24:20.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.009+0000 7f46357ec640 1 -- start start 2023-11-24T08:24:20.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f46357ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f4630166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f46357ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463015eea0 con 0x7f46300cc0f0 2023-11-24T08:24:20.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f46357ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463006d600 con 0x7f463015b440 2023-11-24T08:24:20.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f46357ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300c5e00 con 0x7f46300c59b0 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f462e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f46300cc0f0 0x7f46300c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45220/0 (socket says 172.21.15.107:45220) 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.011+0000 7f462e7fc640 1 -- 172.21.15.107:0/1241746689 learned_addr learned my addr 172.21.15.107:0/1241746689 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462f7fe640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f4630166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275212371 0 0) 0x7f463015eea0 con 0x7f46300cc0f0 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4618003520 con 0x7f46300cc0f0 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299989581 0 0) 0x7f4618003520 con 0x7f46300cc0f0 2023-11-24T08:24:20.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 msgr2=0x7f4630166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f4630166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 >> v1:172.21.15.120:6789/0 conn(0x7f463015b440 legacy=0x7f46300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300abf30 con 0x7f46300cc0f0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f4624001670 con 0x7f46300cc0f0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 >> v1:172.21.15.107:6789/0 conn(0x7f46300cc0f0 legacy=0x7f46300c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46180039a0 0x7f4618003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46180047c0 0x7f4618004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.012+0000 7f462dffb640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f46180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461800ba80 con 0x7f46180047c0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461800bbf0 con 0x7f46180039a0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461800bdb0 con 0x7f46300c59b0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46180047c0 0x7f4618004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462f7fe640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f46180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462effd640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46180039a0 0x7f4618003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 -- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 msgr2=0x7f46180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46300c59b0 0x7f46180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 -- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46180039a0 msgr2=0x7f4618003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46180039a0 0x7f4618003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.013+0000 7f462e7fc640 1 -- 172.21.15.107:0/1241746689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618003520 con 0x7f46180047c0 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.014+0000 7f462e7fc640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46180047c0 0x7f4618004c40 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f462400c240 tx=0x7f46180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1418334e315979 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.014+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624005ad0 con 0x7f46180047c0 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.014+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4624005c50 con 0x7f46180047c0 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f462dffb640 1 -- 172.21.15.107:0/1241746689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462402c8b0 con 0x7f46180047c0 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 -- 172.21.15.107:0/1241746689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46180047c0 msgr2=0x7f4618004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 --2- 172.21.15.107:0/1241746689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46180047c0 0x7f4618004c40 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f462400c240 tx=0x7f46180057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f462f7fe640 1 -- 172.21.15.107:0/1241746689 reap_dead start 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 -- 172.21.15.107:0/1241746689 shutdown_connections 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 -- 172.21.15.107:0/1241746689 >> 172.21.15.107:0/1241746689 conn(0x7f46300b0c50 msgr2=0x7f46300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 -- 172.21.15.107:0/1241746689 shutdown_connections 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.015+0000 7f46357ec640 1 -- 172.21.15.107:0/1241746689 wait complete. 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.017+0000 7f46357ec640 1 Processor -- start 2023-11-24T08:24:20.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.017+0000 7f46357ec640 1 -- start start 2023-11-24T08:24:20.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46180047c0 0x7f463015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4630154f80 0x7f463015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46301547e0 con 0x7f4630154b00 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463015e960 con 0x7f4630154f80 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f46357ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300ae970 con 0x7f46180047c0 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f462e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f462e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38658/0 (socket says 172.21.15.107:38658) 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.018+0000 7f462e7fc640 1 -- 172.21.15.107:0/3109526725 learned_addr learned my addr 172.21.15.107:0/3109526725 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462effd640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46180047c0 0x7f463015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462f7fe640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4630154f80 0x7f463015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 -- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46180047c0 msgr2=0x7f463015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46180047c0 0x7f463015dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 -- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4630154f80 msgr2=0x7f463015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4630154f80 0x7f463015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 -- 172.21.15.107:0/3109526725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463009a380 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f462e7fc640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f462400c360 tx=0x7f4624001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.019+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462402ccc0 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.020+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462402ce20 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.020+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46240072b0 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.020+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463015e0c0 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.020+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463015e680 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.021+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463009a510 con 0x7f4630154b00 2023-11-24T08:24:20.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.022+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f4624032070 con 0x7f4630154b00 2023-11-24T08:24:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.023+0000 7f46177fe640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 0x7f461c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.023+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f462403c040 con 0x7f4630154b00 2023-11-24T08:24:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.023+0000 7f462effd640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 0x7f461c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.024+0000 7f462effd640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 0x7f461c06faf0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4620001690 tx=0x7f4620009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.027+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46240765f0 con 0x7f4630154b00 2023-11-24T08:24:20.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.246+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f463015f3d0 con 0x7f461c06d630 2023-11-24T08:24:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.258+0000 7f46177fe640 1 -- 172.21.15.107:0/3109526725 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f462000d050 con 0x7f461c06d630 2023-11-24T08:24:20.260 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029 2023-11-24T08:24:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 msgr2=0x7f461c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 0x7f461c06faf0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4620001690 tx=0x7f4620009040 comp rx=0 tx=0).stop 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 msgr2=0x7f463015baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f462400c360 tx=0x7f4624001a00 comp rx=0 tx=0).stop 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 shutdown_connections 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f461c06d630 0x7f461c06faf0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4630154f80 0x7f463015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4630154b00 0x7f463015baf0 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 --2- 172.21.15.107:0/3109526725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46180047c0 0x7f463015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.260+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 >> 172.21.15.107:0/3109526725 conn(0x7f46300b0c50 msgr2=0x7f463006cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.261+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 shutdown_connections 2023-11-24T08:24:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.261+0000 7f46357ec640 1 -- 172.21.15.107:0/3109526725 wait complete. 2023-11-24T08:24:20.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone9816 2023-11-24T08:24:20.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.486+0000 7f9680de6640 1 Processor -- start 2023-11-24T08:24:20.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.486+0000 7f9680de6640 1 -- start start 2023-11-24T08:24:20.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.487+0000 7f9680de6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f967c15b260 0x7f967c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.487+0000 7f9680de6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f967c0b1250 0x7f967c06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f9680de6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c15ed20 con 0x7f967c06e580 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f9680de6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c15b640 con 0x7f967c15b260 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f9680de6640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c15f8a0 con 0x7f967c0b1250 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f967c15b260 0x7f967c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f967c15b260 0x7f967c06e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38668/0 (socket says 172.21.15.107:38668) 2023-11-24T08:24:20.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 -- 172.21.15.107:0/2852696210 learned_addr learned my addr 172.21.15.107:0/2852696210 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:20.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 -- 172.21.15.107:0/2852696210 >> v2:172.21.15.120:3300/0 conn(0x7f967c0b1250 msgr2=0x7f967c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:20.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 --2- 172.21.15.107:0/2852696210 >> v2:172.21.15.120:3300/0 conn(0x7f967c0b1250 0x7f967c06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 -- 172.21.15.107:0/2852696210 >> v1:172.21.15.120:6789/0 conn(0x7f967c06e580 legacy=0x7f967c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:20.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.488+0000 7f967b7fe640 1 -- 172.21.15.107:0/2852696210 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0abe50 con 0x7f967c15b260 2023-11-24T08:24:20.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.489+0000 7f967b7fe640 1 --2- 172.21.15.107:0/2852696210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c15b260 0x7f967c06e040 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f967c0b1a20 tx=0x7f966c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7459c94d95d9ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f967a7fc640 1 -- 172.21.15.107:0/2852696210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c031750 con 0x7f967c15b260 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f967a7fc640 1 -- 172.21.15.107:0/2852696210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c0318b0 con 0x7f967c15b260 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f967a7fc640 1 -- 172.21.15.107:0/2852696210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c031c40 con 0x7f967c15b260 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 -- 172.21.15.107:0/2852696210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c15b260 msgr2=0x7f967c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 --2- 172.21.15.107:0/2852696210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c15b260 0x7f967c06e040 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f967c0b1a20 tx=0x7f966c02b7f0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 -- 172.21.15.107:0/2852696210 shutdown_connections 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 --2- 172.21.15.107:0/2852696210 >> v2:172.21.15.120:3300/0 conn(0x7f967c0b1250 0x7f967c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 --2- 172.21.15.107:0/2852696210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c15b260 0x7f967c06e040 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 -- 172.21.15.107:0/2852696210 >> 172.21.15.107:0/2852696210 conn(0x7f967c0cbb40 msgr2=0x7f967c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 -- 172.21.15.107:0/2852696210 shutdown_connections 2023-11-24T08:24:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.490+0000 7f9680de6640 1 -- 172.21.15.107:0/2852696210 wait complete. 2023-11-24T08:24:20.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.492+0000 7f9680de6640 1 Processor -- start 2023-11-24T08:24:20.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.492+0000 7f9680de6640 1 -- start start 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f967c0b1250 0x7f967c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f967c15b260 0x7f967c1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c158c20 con 0x7f967c06e580 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0b2390 con 0x7f967c15b260 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f9680de6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0b1d80 con 0x7f967c0b1250 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f967b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f967b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38670/0 (socket says 172.21.15.107:38670) 2023-11-24T08:24:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.493+0000 7f967b7fe640 1 -- 172.21.15.107:0/1272089681 learned_addr learned my addr 172.21.15.107:0/1272089681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 -- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f967c0b1250 msgr2=0x7f967c1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f967c0b1250 0x7f967c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 -- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f967c15b260 msgr2=0x7f967c1586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f967c15b260 0x7f967c1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 -- 172.21.15.107:0/1272089681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0094d0 con 0x7f967c06e580 2023-11-24T08:24:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f967b7fe640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f966c000c00 tx=0x7f966c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c033c60 con 0x7f967c06e580 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c033de0 con 0x7f967c06e580 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c03d6b0 con 0x7f967c06e580 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.494+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c09bf00 con 0x7f967c06e580 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.495+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c09c1e0 con 0x7f967c06e580 2023-11-24T08:24:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.496+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c04e840 con 0x7f967c06e580 2023-11-24T08:24:20.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.497+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f966c03a040 con 0x7f967c06e580 2023-11-24T08:24:20.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.497+0000 7f9663fff640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 0x7f966806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.498+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f966c0aaa20 con 0x7f967c06e580 2023-11-24T08:24:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.498+0000 7f967affd640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 0x7f966806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.498+0000 7f967affd640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 0x7f966806faf0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f967c15a650 tx=0x7f9670006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:20.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.502+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f966c0775d0 con 0x7f967c06e580 2023-11-24T08:24:20.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.720+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f967c067de0 con 0x7f966806d630 2023-11-24T08:24:20.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.736+0000 7f9663fff640 1 -- 172.21.15.107:0/1272089681 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+708 (secure 0 0 0) 0x7f967c067de0 con 0x7f966806d630 2023-11-24T08:24:20.738 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:24:20.738 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:24:10", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 16777216, 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:24:15", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:24:18", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:24:20.739 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:24:13", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:24:20.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:24:20.741 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:24:20.741 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:24:20.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.738+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 msgr2=0x7f966806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.738+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 0x7f966806faf0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f967c15a650 tx=0x7f9670006ba0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.738+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 msgr2=0x7f967c0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.738+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f966c000c00 tx=0x7f966c0046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 shutdown_connections 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f966806d630 0x7f966806faf0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f967c15b260 0x7f967c1586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f967c0b1250 0x7f967c1581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 --2- 172.21.15.107:0/1272089681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f967c06e580 0x7f967c0b45d0 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 >> 172.21.15.107:0/1272089681 conn(0x7f967c0cbb40 msgr2=0x7f967c159460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:20.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 shutdown_connections 2023-11-24T08:24:20.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:20.739+0000 7f9680de6640 1 -- 172.21.15.107:0/1272089681 wait complete. 2023-11-24T08:24:20.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:20.760 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:20.814 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0: ceph.quota.max_bytes: No such attribute 2023-11-24T08:24:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:24:20.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:20.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:20.869 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:24:20.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:20.870 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:20.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:20.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:20.978 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0 2023-11-24T08:24:20.978 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2 2023-11-24T08:24:20.978 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14 2023-11-24T08:24:20.978 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13 2023-11-24T08:24:20.978 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15 2023-11-24T08:24:20.979 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8 2023-11-24T08:24:20.980 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11 2023-11-24T08:24:20.980 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3 2023-11-24T08:24:20.980 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12 2023-11-24T08:24:20.980 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1 2023-11-24T08:24:20.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:20.981 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:21.029 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:24:21.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.031 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/) 2023-11-24T08:24:21.083 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:24:21.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.084 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:21.136 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:24:21.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.138 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/) 2023-11-24T08:24:21.191 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:24:21.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.192 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:21.241 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:24:21.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.243 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/) 2023-11-24T08:24:21.292 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:24:21.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.294 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0) 2023-11-24T08:24:21.341 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:21.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.343 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/) 2023-11-24T08:24:21.392 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:21.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.393 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2) 2023-11-24T08:24:21.442 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:21.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.444 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.2) 2023-11-24T08:24:21.503 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:21.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.505 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2) 2023-11-24T08:24:21.560 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:21.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.561 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.2) 2023-11-24T08:24:21.614 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:21.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.615 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2) 2023-11-24T08:24:21.668 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:21.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.669 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.2) 2023-11-24T08:24:21.722 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:21.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.723 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.2) 2023-11-24T08:24:21.773 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:21.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.775 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.2) 2023-11-24T08:24:21.832 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:21.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.833 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14) 2023-11-24T08:24:21.885 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:21.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.887 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.14) 2023-11-24T08:24:21.944 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:21.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:21.946 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14) 2023-11-24T08:24:22.000 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.001 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.14) 2023-11-24T08:24:22.053 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.054 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14) 2023-11-24T08:24:22.103 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:22.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.105 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.14) 2023-11-24T08:24:22.156 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:22.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.158 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.14) 2023-11-24T08:24:22.211 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:22.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.212 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.14) 2023-11-24T08:24:22.267 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:22.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.269 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13) 2023-11-24T08:24:22.321 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:22.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.323 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.13) 2023-11-24T08:24:22.380 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:22.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.381 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13) 2023-11-24T08:24:22.435 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.436 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.13) 2023-11-24T08:24:22.489 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13) 2023-11-24T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:22.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.542 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.13) 2023-11-24T08:24:22.594 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:22.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.596 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.13) 2023-11-24T08:24:22.650 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:22.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.651 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.13) 2023-11-24T08:24:22.704 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:22.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.705 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10) 2023-11-24T08:24:22.756 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:22.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.757 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.10) 2023-11-24T08:24:22.812 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:22.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.814 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10) 2023-11-24T08:24:22.862 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.863 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.10) 2023-11-24T08:24:22.913 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:22.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.914 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10) 2023-11-24T08:24:22.962 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:22.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:22.964 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.10) 2023-11-24T08:24:23.013 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:23.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.014 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.10) 2023-11-24T08:24:23.062 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.10) 2023-11-24T08:24:23.113 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.114 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5) 2023-11-24T08:24:23.167 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:23.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.5) 2023-11-24T08:24:23.225 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:23.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.227 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5) 2023-11-24T08:24:23.280 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:23.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.281 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.5) 2023-11-24T08:24:23.331 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:23.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.332 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5) 2023-11-24T08:24:23.384 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:23.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.385 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.5) 2023-11-24T08:24:23.436 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:23.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.5) 2023-11-24T08:24:23.488 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.489 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.5) 2023-11-24T08:24:23.548 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.549 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4) 2023-11-24T08:24:23.601 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:23.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.602 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.4) 2023-11-24T08:24:23.660 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:23.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.662 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4) 2023-11-24T08:24:23.708 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:23.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.709 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.4) 2023-11-24T08:24:23.760 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:23.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.762 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4) 2023-11-24T08:24:23.814 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:23.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.4) 2023-11-24T08:24:23.865 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:23.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.866 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.4) 2023-11-24T08:24:23.917 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.919 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.4) 2023-11-24T08:24:23.969 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:23.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:23.970 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7) 2023-11-24T08:24:24.018 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.019 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.7) 2023-11-24T08:24:24.061 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7) 2023-11-24T08:24:24.161 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:24.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.163 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.7) 2023-11-24T08:24:24.217 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:24.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.218 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7) 2023-11-24T08:24:24.272 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:24.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.273 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.7) 2023-11-24T08:24:24.326 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:24.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.328 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.7) 2023-11-24T08:24:24.387 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:24.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.388 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.7) 2023-11-24T08:24:24.448 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:24.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.448 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6) 2023-11-24T08:24:24.502 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.504 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.6) 2023-11-24T08:24:24.557 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6) 2023-11-24T08:24:24.614 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:24.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.616 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.6) 2023-11-24T08:24:24.669 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:24.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.670 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6) 2023-11-24T08:24:24.725 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:24.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.727 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.6) 2023-11-24T08:24:24.775 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:24.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.6) 2023-11-24T08:24:24.826 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:24.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.828 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.6) 2023-11-24T08:24:24.877 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:24.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.879 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0) 2023-11-24T08:24:24.929 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.931 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.0) 2023-11-24T08:24:24.982 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:24.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:24.984 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0) 2023-11-24T08:24:25.042 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.044 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.0) 2023-11-24T08:24:25.099 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.107 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0) 2023-11-24T08:24:25.161 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:25.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.164 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.0) 2023-11-24T08:24:25.218 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:25.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.220 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.0) 2023-11-24T08:24:25.273 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:25.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.0) 2023-11-24T08:24:25.324 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:25.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.325 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9) 2023-11-24T08:24:25.376 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:25.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.377 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.9) 2023-11-24T08:24:25.432 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:25.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.434 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9) 2023-11-24T08:24:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.488 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.9) 2023-11-24T08:24:25.544 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.547 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9) 2023-11-24T08:24:25.599 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:25.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.600 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.9) 2023-11-24T08:24:25.653 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:25.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.9) 2023-11-24T08:24:25.707 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:25.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.9) 2023-11-24T08:24:25.760 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:25.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.761 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15) 2023-11-24T08:24:25.813 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:25.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.15) 2023-11-24T08:24:25.868 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:25.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.869 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15) 2023-11-24T08:24:25.921 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.923 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.15) 2023-11-24T08:24:25.974 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:25.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:25.976 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15) 2023-11-24T08:24:26.028 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.029 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.15) 2023-11-24T08:24:26.077 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.079 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.15) 2023-11-24T08:24:26.128 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:26.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.129 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.15) 2023-11-24T08:24:26.177 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:26.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.178 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8) 2023-11-24T08:24:26.228 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:26.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.229 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.8) 2023-11-24T08:24:26.282 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:26.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.284 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8) 2023-11-24T08:24:26.334 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:26.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.335 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.8) 2023-11-24T08:24:26.385 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:26.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8) 2023-11-24T08:24:26.437 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.8) 2023-11-24T08:24:26.489 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.8) 2023-11-24T08:24:26.544 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:26.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.545 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.8) 2023-11-24T08:24:26.596 INFO:teuthology.orchestra.run.smithi107.stdout:1700814252 2023-11-24T08:24:26.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11) 2023-11-24T08:24:26.651 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:26.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.652 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.11) 2023-11-24T08:24:26.706 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:26.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11) 2023-11-24T08:24:26.760 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:26.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.762 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.11) 2023-11-24T08:24:26.815 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:26.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.816 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11) 2023-11-24T08:24:26.867 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.869 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.11) 2023-11-24T08:24:26.924 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:26.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.11) 2023-11-24T08:24:26.974 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:26.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:26.975 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.11) 2023-11-24T08:24:27.021 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:27.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3) 2023-11-24T08:24:27.119 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:27.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.121 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.3) 2023-11-24T08:24:27.169 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:27.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.170 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3) 2023-11-24T08:24:27.218 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:27.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.220 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.3) 2023-11-24T08:24:27.269 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:27.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.271 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3) 2023-11-24T08:24:27.323 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:27.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.325 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.3) 2023-11-24T08:24:27.378 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:27.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.380 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.3) 2023-11-24T08:24:27.434 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:27.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.436 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.3) 2023-11-24T08:24:27.487 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:27.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.488 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12) 2023-11-24T08:24:27.540 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:27.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.541 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.12) 2023-11-24T08:24:27.594 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:27.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.595 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12) 2023-11-24T08:24:27.648 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:27.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.649 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.12) 2023-11-24T08:24:27.702 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:27.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.703 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12) 2023-11-24T08:24:27.757 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:27.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.758 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.12) 2023-11-24T08:24:27.808 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:27.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.810 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.12) 2023-11-24T08:24:27.867 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:27.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.869 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.12) 2023-11-24T08:24:27.921 INFO:teuthology.orchestra.run.smithi107.stdout:1700814253 2023-11-24T08:24:27.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.922 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1) 2023-11-24T08:24:27.973 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:27.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:27.974 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.1) 2023-11-24T08:24:28.029 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:24:28.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.030 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1) 2023-11-24T08:24:28.081 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:28.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.082 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.1) 2023-11-24T08:24:28.134 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:24:28.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.135 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1) 2023-11-24T08:24:28.187 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:28.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.188 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.1) 2023-11-24T08:24:28.241 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:24:28.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.242 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol1655/.snap/subvol_snap194/906ee78b-8a57-431c-848a-96134c3f8ec0/subvolume_file.1) 2023-11-24T08:24:28.294 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:28.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:28.295 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone9816/e64e5677-e062-466a-b34e-d5adcfb4f029/subvolume_file.1) 2023-11-24T08:24:28.347 INFO:teuthology.orchestra.run.smithi107.stdout:1700814251 2023-11-24T08:24:28.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol1655 subvol_snap4297 2023-11-24T08:24:28.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.563+0000 7fd0e093a640 1 Processor -- start 2023-11-24T08:24:28.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.564+0000 7fd0e093a640 1 -- start start 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.565+0000 7fd0e093a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 0x7fd0dc161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.565+0000 7fd0e093a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0ce0b0 con 0x7fd0dc157c70 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.565+0000 7fd0e093a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc161960 con 0x7fd0dc0cfb60 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.565+0000 7fd0e093a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc15e8d0 con 0x7fd0dc15e390 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.565+0000 7fd0da59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd0dc157c70 0x7fd0dc0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57324/0 (socket says 172.21.15.107:57324) 2023-11-24T08:24:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0da59c640 1 -- 172.21.15.107:0/3207083944 learned_addr learned my addr 172.21.15.107:0/3207083944 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0dad9d640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 0x7fd0dc161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712332912 0 0) 0x7fd0dc0ce0b0 con 0x7fd0dc157c70 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c8003520 con 0x7fd0dc157c70 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903708053 0 0) 0x7fd0c8003520 con 0x7fd0dc157c70 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 msgr2=0x7fd0dc161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 0x7fd0dc161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 >> v1:172.21.15.120:6789/0 conn(0x7fd0dc0cfb60 legacy=0x7fd0dc15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc15ebb0 con 0x7fd0dc157c70 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fd0cc0019f0 con 0x7fd0dc157c70 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.566+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 >> v1:172.21.15.107:6789/0 conn(0x7fd0dc157c70 legacy=0x7fd0dc0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c80039a0 0x7fd0c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 0x7fd0c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c8004d00 0x7fd0c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc15e8d0 con 0x7fd0c80039a0 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c800bb40 con 0x7fd0c8004d00 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c800bd00 con 0x7fd0dc15e390 2023-11-24T08:24:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c80039a0 0x7fd0c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 -- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 msgr2=0x7fd0c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:28.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc15e390 0x7fd0c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 -- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c8004d00 msgr2=0x7fd0c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:28.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c8004d00 0x7fd0c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.567+0000 7fd0da59c640 1 -- 172.21.15.107:0/3207083944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8003520 con 0x7fd0c80039a0 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.568+0000 7fd0da59c640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c80039a0 0x7fd0c8003dc0 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00a760 tx=0x7fd0c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2929bedfc2be38a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc034630 con 0x7fd0c80039a0 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0cc02f070 con 0x7fd0c80039a0 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0d959a640 1 -- 172.21.15.107:0/3207083944 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc034a10 con 0x7fd0c80039a0 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 -- 172.21.15.107:0/3207083944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c80039a0 msgr2=0x7fd0c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 --2- 172.21.15.107:0/3207083944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c80039a0 0x7fd0c8003dc0 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00a760 tx=0x7fd0c80057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0dad9d640 1 -- 172.21.15.107:0/3207083944 reap_dead start 2023-11-24T08:24:28.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 -- 172.21.15.107:0/3207083944 shutdown_connections 2023-11-24T08:24:28.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 -- 172.21.15.107:0/3207083944 >> 172.21.15.107:0/3207083944 conn(0x7fd0dc06a920 msgr2=0x7fd0dc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:28.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 -- 172.21.15.107:0/3207083944 shutdown_connections 2023-11-24T08:24:28.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.569+0000 7fd0e093a640 1 -- 172.21.15.107:0/3207083944 wait complete. 2023-11-24T08:24:28.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.571+0000 7fd0e093a640 1 Processor -- start 2023-11-24T08:24:28.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.571+0000 7fd0e093a640 1 -- start start 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 0x7fd0dc0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc0cbde0 0x7fd0dc1f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0cc260 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c8e00 con 0x7fd0c80039a0 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.572+0000 7fd0e093a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c90e0 con 0x7fd0dc0cbde0 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34938/0 (socket says 172.21.15.107:34938) 2023-11-24T08:24:28.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 -- 172.21.15.107:0/2698495053 learned_addr learned my addr 172.21.15.107:0/2698495053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:28.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0da59c640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 0x7fd0dc0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 -- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc0cbde0 msgr2=0x7fd0dc1f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0dad9d640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc0cbde0 0x7fd0dc1f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc0cbde0 0x7fd0dc1f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 -- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 msgr2=0x7fd0dc0c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 0x7fd0dc0c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0d9d9b640 1 -- 172.21.15.107:0/2698495053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0c93c0 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.573+0000 7fd0da59c640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 0x7fd0dc0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0d9d9b640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c500 tx=0x7fd0c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c40052d0 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c4005430 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0c96a0 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4012870 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.574+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc0cc640 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.575+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc067470 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.576+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fd0c40129d0 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.577+0000 7fd0d2ffd640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 0x7fd0bc06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:28.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.577+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c40885a0 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.578+0000 7fd0da59c640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 0x7fd0bc06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:28.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.578+0000 7fd0da59c640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 0x7fd0bc06faf0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc034000 tx=0x7fd0cc02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:28.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.582+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0c4055150 con 0x7fd0dc0c88a0 2023-11-24T08:24:28.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.805+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol1655", "snap_name": "subvol_snap4297", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0dc158050 con 0x7fd0bc06d630 2023-11-24T08:24:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.878+0000 7fd0d2ffd640 1 -- 172.21.15.107:0/2698495053 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd0cc040460 con 0x7fd0bc06d630 2023-11-24T08:24:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.881+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 msgr2=0x7fd0bc06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.881+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 0x7fd0bc06faf0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc034000 tx=0x7fd0cc02f000 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.881+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 msgr2=0x7fd0dc0cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.881+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c500 tx=0x7fd0c400c9c0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 shutdown_connections 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd0bc06d630 0x7fd0bc06faf0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0dc0cbde0 0x7fd0dc1f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0dc0c88a0 0x7fd0dc0cb8a0 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 --2- 172.21.15.107:0/2698495053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c80039a0 0x7fd0dc0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 >> 172.21.15.107:0/2698495053 conn(0x7fd0dc06a920 msgr2=0x7fd0dc070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 shutdown_connections 2023-11-24T08:24:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:28.882+0000 7fd0e093a640 1 -- 172.21.15.107:0/2698495053 wait complete. 2023-11-24T08:24:28.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol1655 subvol_snap194 2023-11-24T08:24:29.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.109+0000 7fd391b63640 1 Processor -- start 2023-11-24T08:24:29.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.110+0000 7fd391b63640 1 -- start start 2023-11-24T08:24:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.111+0000 7fd391b63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd38c15d260 0x7fd38c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.111+0000 7fd391b63640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd38c0d2750 0x7fd38c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.111+0000 7fd391b63640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0ce280 con 0x7fd38c0d17b0 2023-11-24T08:24:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.111+0000 7fd391b63640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd38c0ce4d0 con 0x7fd38c15d260 2023-11-24T08:24:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.111+0000 7fd391b63640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd38c0ce7b0 con 0x7fd38c0d2750 2023-11-24T08:24:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd38c15d260 0x7fd38c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd38c15d260 0x7fd38c0cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34940/0 (socket says 172.21.15.107:34940) 2023-11-24T08:24:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 -- 172.21.15.107:0/55867033 learned_addr learned my addr 172.21.15.107:0/55867033 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 -- 172.21.15.107:0/55867033 >> v2:172.21.15.120:3300/0 conn(0x7fd38c0d2750 msgr2=0x7fd38c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 --2- 172.21.15.107:0/55867033 >> v2:172.21.15.120:3300/0 conn(0x7fd38c0d2750 0x7fd38c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 -- 172.21.15.107:0/55867033 >> v1:172.21.15.120:6789/0 conn(0x7fd38c0d17b0 legacy=0x7fd38c0d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.112+0000 7fd390b61640 1 -- 172.21.15.107:0/55867033 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0b8b40 con 0x7fd38c15d260 2023-11-24T08:24:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.113+0000 7fd390b61640 1 --2- 172.21.15.107:0/55867033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c15d260 0x7fd38c0cdb50 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0b5750 tx=0x7fd38002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aee3430cd919d72 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd38b7fe640 1 -- 172.21.15.107:0/55867033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380031870 con 0x7fd38c15d260 2023-11-24T08:24:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd38b7fe640 1 -- 172.21.15.107:0/55867033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3800319d0 con 0x7fd38c15d260 2023-11-24T08:24:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd38b7fe640 1 -- 172.21.15.107:0/55867033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380031d60 con 0x7fd38c15d260 2023-11-24T08:24:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 -- 172.21.15.107:0/55867033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c15d260 msgr2=0x7fd38c0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 --2- 172.21.15.107:0/55867033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c15d260 0x7fd38c0cdb50 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0b5750 tx=0x7fd38002b910 comp rx=0 tx=0).stop 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 -- 172.21.15.107:0/55867033 shutdown_connections 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 --2- 172.21.15.107:0/55867033 >> v2:172.21.15.120:3300/0 conn(0x7fd38c0d2750 0x7fd38c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 --2- 172.21.15.107:0/55867033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c15d260 0x7fd38c0cdb50 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 -- 172.21.15.107:0/55867033 >> 172.21.15.107:0/55867033 conn(0x7fd38c061340 msgr2=0x7fd38c061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 -- 172.21.15.107:0/55867033 shutdown_connections 2023-11-24T08:24:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.114+0000 7fd391b63640 1 -- 172.21.15.107:0/55867033 wait complete. 2023-11-24T08:24:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.116+0000 7fd391b63640 1 Processor -- start 2023-11-24T08:24:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.116+0000 7fd391b63640 1 -- start start 2023-11-24T08:24:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd38c0d17b0 0x7fd38c1f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 0x7fd38c1f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c1f2f00 con 0x7fd38c0d2750 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c15f870 con 0x7fd38c15d260 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.117+0000 7fd391b63640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c1f3e30 con 0x7fd38c0d17b0 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34946/0 (socket says 172.21.15.107:34946) 2023-11-24T08:24:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 -- 172.21.15.107:0/4110267248 learned_addr learned my addr 172.21.15.107:0/4110267248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd390b61640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd38c0d17b0 0x7fd38c1f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd391362640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 0x7fd38c1f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 -- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd38c0d17b0 msgr2=0x7fd38c1f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd38c0d17b0 0x7fd38c1f0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 -- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 msgr2=0x7fd38c1f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 0x7fd38c1f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 -- 172.21.15.107:0/4110267248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800094d0 con 0x7fd38c0d2750 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd391362640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 0x7fd38c1f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.118+0000 7fd38bfff640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c410 tx=0x7fd37800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.119+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3780035f0 con 0x7fd38c0d2750 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.119+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd378003750 con 0x7fd38c0d2750 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.119+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378003b40 con 0x7fd38c0d2750 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.119+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c157e10 con 0x7fd38c0d2750 2023-11-24T08:24:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.119+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c1f30f0 con 0x7fd38c0d2750 2023-11-24T08:24:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.120+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c067990 con 0x7fd38c0d2750 2023-11-24T08:24:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.121+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fd378003ca0 con 0x7fd38c0d2750 2023-11-24T08:24:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.122+0000 7fd3897fa640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 0x7fd37006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.122+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd378087bf0 con 0x7fd38c0d2750 2023-11-24T08:24:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.122+0000 7fd390b61640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 0x7fd37006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.123+0000 7fd390b61640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 0x7fd37006faf0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd380036000 tx=0x7fd38003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.126+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd378054700 con 0x7fd38c0d2750 2023-11-24T08:24:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.354+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol1655", "snap_name": "subvol_snap194", "target": ["mon-mgr", ""]}) v1 -- 0x7fd38c0b9100 con 0x7fd37006d630 2023-11-24T08:24:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.431+0000 7fd3897fa640 1 -- 172.21.15.107:0/4110267248 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd380031d10 con 0x7fd37006d630 2023-11-24T08:24:29.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.434+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 msgr2=0x7fd37006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.434+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 0x7fd37006faf0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd380036000 tx=0x7fd38003c040 comp rx=0 tx=0).stop 2023-11-24T08:24:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 msgr2=0x7fd38c157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c410 tx=0x7fd37800c8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 shutdown_connections 2023-11-24T08:24:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd37006d630 0x7fd37006faf0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c15d260 0x7fd38c1f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd38c0d2750 0x7fd38c157780 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 --2- 172.21.15.107:0/4110267248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd38c0d17b0 0x7fd38c1f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 >> 172.21.15.107:0/4110267248 conn(0x7fd38c061340 msgr2=0x7fd38c15c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 shutdown_connections 2023-11-24T08:24:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.435+0000 7fd391b63640 1 -- 172.21.15.107:0/4110267248 wait complete. 2023-11-24T08:24:29.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone9816 2023-11-24T08:24:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.662+0000 7f932540e640 1 Processor -- start 2023-11-24T08:24:29.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.663+0000 7f932540e640 1 -- start start 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9320072160 0x7f932015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 0x7f932006d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f932006dc40 0x7f93200cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932006e0f0 con 0x7f932006dc40 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932015ec90 con 0x7f9320072160 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f932540e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93200cd530 con 0x7f93200c49f0 2023-11-24T08:24:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f931f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f932006dc40 0x7f93200cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f931f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f932006dc40 0x7f93200cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34956/0 (socket says 172.21.15.107:34956) 2023-11-24T08:24:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.665+0000 7f931f7fe640 1 -- 172.21.15.107:0/4248258316 learned_addr learned my addr 172.21.15.107:0/4248258316 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931effd640 1 --2- 172.21.15.107:0/4248258316 >> v2:172.21.15.120:3300/0 conn(0x7f9320072160 0x7f932015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f9317fff640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 0x7f932006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931f7fe640 1 -- 172.21.15.107:0/4248258316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 msgr2=0x7f932006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931f7fe640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 0x7f932006d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931f7fe640 1 -- 172.21.15.107:0/4248258316 >> v2:172.21.15.120:3300/0 conn(0x7f9320072160 msgr2=0x7f932015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931f7fe640 1 --2- 172.21.15.107:0/4248258316 >> v2:172.21.15.120:3300/0 conn(0x7f9320072160 0x7f932015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f931f7fe640 1 -- 172.21.15.107:0/4248258316 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200c4ed0 con 0x7f932006dc40 2023-11-24T08:24:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.666+0000 7f9317fff640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 0x7f932006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.667+0000 7f931f7fe640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f93200cd750 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f93200ccae0 tx=0x7f931802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e62371b0aa09ca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.667+0000 7f931e7fc640 1 -- 172.21.15.107:0/4248258316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931802ca70 con 0x7f932006dc40 2023-11-24T08:24:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.667+0000 7f931e7fc640 1 -- 172.21.15.107:0/4248258316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931802cbd0 con 0x7f932006dc40 2023-11-24T08:24:29.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.667+0000 7f931e7fc640 1 -- 172.21.15.107:0/4248258316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93180343f0 con 0x7f932006dc40 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 -- 172.21.15.107:0/4248258316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 msgr2=0x7f93200cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f93200cd750 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f93200ccae0 tx=0x7f931802bed0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 -- 172.21.15.107:0/4248258316 shutdown_connections 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f93200cd750 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 --2- 172.21.15.107:0/4248258316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93200c49f0 0x7f932006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 --2- 172.21.15.107:0/4248258316 >> v2:172.21.15.120:3300/0 conn(0x7f9320072160 0x7f932015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 -- 172.21.15.107:0/4248258316 >> 172.21.15.107:0/4248258316 conn(0x7f93200b0c50 msgr2=0x7f93200b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:29.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 -- 172.21.15.107:0/4248258316 shutdown_connections 2023-11-24T08:24:29.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.668+0000 7f932540e640 1 -- 172.21.15.107:0/4248258316 wait complete. 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.670+0000 7f932540e640 1 Processor -- start 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.670+0000 7f932540e640 1 -- start start 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9320072160 0x7f932014cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93200c49f0 0x7f93200cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932009c770 con 0x7f932006dc40 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932014d110 con 0x7f93200c49f0 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f932540e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93200cb860 con 0x7f9320072160 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34962/0 (socket says 172.21.15.107:34962) 2023-11-24T08:24:29.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 -- 172.21.15.107:0/3449906695 learned_addr learned my addr 172.21.15.107:0/3449906695 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f9317fff640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9320072160 0x7f932014cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931f7fe640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93200c49f0 0x7f93200cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 -- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9320072160 msgr2=0x7f932014cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9320072160 0x7f932014cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 -- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93200c49f0 msgr2=0x7f93200cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93200c49f0 0x7f93200cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.671+0000 7f931effd640 1 -- 172.21.15.107:0/3449906695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318009b30 con 0x7f932006dc40 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f931effd640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f931000d800 tx=0x7f931000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93100096a0 con 0x7f932006dc40 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9310009800 con 0x7f932006dc40 2023-11-24T08:24:29.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310009b90 con 0x7f932006dc40 2023-11-24T08:24:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93200cac50 con 0x7f932006dc40 2023-11-24T08:24:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.672+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93200cb210 con 0x7f932006dc40 2023-11-24T08:24:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.673+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932004e8b0 con 0x7f932006dc40 2023-11-24T08:24:29.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.674+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f9310009cf0 con 0x7f932006dc40 2023-11-24T08:24:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.675+0000 7f93177fe640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 0x7f930006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.675+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93100890f0 con 0x7f932006dc40 2023-11-24T08:24:29.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.675+0000 7f9317fff640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 0x7f930006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:29.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.676+0000 7f9317fff640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 0x7f930006faf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f93200719c0 tx=0x7f9308008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:29.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.680+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9310055ca0 con 0x7f932006dc40 2023-11-24T08:24:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.897+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone9816", "target": ["mon-mgr", ""]}) v1 -- 0x7f93200c5770 con 0x7f930006d630 2023-11-24T08:24:29.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.924+0000 7f93177fe640 1 -- 172.21.15.107:0/3449906695 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f930800c040 con 0x7f930006d630 2023-11-24T08:24:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 msgr2=0x7f930006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 0x7f930006faf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f93200719c0 tx=0x7f9308008040 comp rx=0 tx=0).stop 2023-11-24T08:24:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 msgr2=0x7f932014e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f931000d800 tx=0x7f931000dcc0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 shutdown_connections 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f930006d630 0x7f930006faf0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93200c49f0 0x7f93200cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9320072160 0x7f932014cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 --2- 172.21.15.107:0/3449906695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f932006dc40 0x7f932014e7e0 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 >> 172.21.15.107:0/3449906695 conn(0x7f93200b0c50 msgr2=0x7f93200ac030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 shutdown_connections 2023-11-24T08:24:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:29.927+0000 7f932540e640 1 -- 172.21.15.107:0/3449906695 wait complete. 2023-11-24T08:24:29.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2023-11-24T08:24:30.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.150+0000 7fe693790640 1 Processor -- start 2023-11-24T08:24:30.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.150+0000 7fe693790640 1 -- start start 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.151+0000 7fe693790640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe68c146eb0 0x7fe68c161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe693790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 0x7fe68c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe693790640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c161a10 con 0x7fe68c06e160 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe693790640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68c161cf0 con 0x7fe68c146eb0 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe693790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c161fd0 con 0x7fe68c0cd2e0 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe68c146eb0 0x7fe68c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe68c146eb0 0x7fe68c161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34966/0 (socket says 172.21.15.107:34966) 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 -- 172.21.15.107:0/2243812431 learned_addr learned my addr 172.21.15.107:0/2243812431 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe690d2b640 1 --1- 172.21.15.107:0/2243812431 >> v1:172.21.15.107:6789/0 conn(0x7fe68c06e160 0x7fe68c0ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57336/0 (socket says 172.21.15.107:57336) 2023-11-24T08:24:30.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe691d2d640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 0x7fe68c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:30.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 -- 172.21.15.107:0/2243812431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 msgr2=0x7fe68c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 0x7fe68c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 -- 172.21.15.107:0/2243812431 >> v1:172.21.15.107:6789/0 conn(0x7fe68c06e160 legacy=0x7fe68c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe69152c640 1 -- 172.21.15.107:0/2243812431 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c070660 con 0x7fe68c146eb0 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe683fff640 1 -- 172.21.15.107:0/2243812431 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825407243 0 0) 0x7fe68c161a10 con 0x7fe68c06e160 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.152+0000 7fe691d2d640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 0x7fe68c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe69152c640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c146eb0 0x7fe68c161150 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fe688009870 tx=0x7fe68802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f07a0099f1a24d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe683fff640 1 -- 172.21.15.107:0/2243812431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688030840 con 0x7fe68c146eb0 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe683fff640 1 -- 172.21.15.107:0/2243812431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6880309a0 con 0x7fe68c146eb0 2023-11-24T08:24:30.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe683fff640 1 -- 172.21.15.107:0/2243812431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688030d30 con 0x7fe68c146eb0 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe693790640 1 -- 172.21.15.107:0/2243812431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c146eb0 msgr2=0x7fe68c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe693790640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c146eb0 0x7fe68c161150 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fe688009870 tx=0x7fe68802b980 comp rx=0 tx=0).stop 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe693790640 1 -- 172.21.15.107:0/2243812431 shutdown_connections 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe693790640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c0cd2e0 0x7fe68c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.154+0000 7fe693790640 1 --2- 172.21.15.107:0/2243812431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c146eb0 0x7fe68c161150 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.155+0000 7fe693790640 1 -- 172.21.15.107:0/2243812431 >> 172.21.15.107:0/2243812431 conn(0x7fe68c0b0c50 msgr2=0x7fe68c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:30.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.155+0000 7fe693790640 1 -- 172.21.15.107:0/2243812431 shutdown_connections 2023-11-24T08:24:30.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.155+0000 7fe693790640 1 -- 172.21.15.107:0/2243812431 wait complete. 2023-11-24T08:24:30.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.156+0000 7fe693790640 1 Processor -- start 2023-11-24T08:24:30.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.156+0000 7fe693790640 1 -- start start 2023-11-24T08:24:30.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c0cd2e0 0x7fe68c1429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c146eb0 0x7fe68c0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0b1d10 con 0x7fe68c06e160 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c15f2b0 con 0x7fe68c0cd2e0 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.157+0000 7fe693790640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0b1f60 con 0x7fe68c146eb0 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34970/0 (socket says 172.21.15.107:34970) 2023-11-24T08:24:30.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 -- 172.21.15.107:0/2775625303 learned_addr learned my addr 172.21.15.107:0/2775625303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:30.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 -- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c146eb0 msgr2=0x7fe68c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c146eb0 0x7fe68c0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 -- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c0cd2e0 msgr2=0x7fe68c1429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c0cd2e0 0x7fe68c1429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 -- 172.21.15.107:0/2775625303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6880094d0 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe69152c640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fe688000c00 tx=0x7fe688004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.158+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688030810 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.159+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688039d90 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.159+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c0b2240 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.159+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688043670 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.159+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c15eef0 con 0x7fe68c06e160 2023-11-24T08:24:30.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.160+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c0b2760 con 0x7fe68c06e160 2023-11-24T08:24:30.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.161+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fe68804c3b0 con 0x7fe68c06e160 2023-11-24T08:24:30.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.162+0000 7fe681ffb640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 0x7fe67006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:30.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.162+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6880aaf50 con 0x7fe68c06e160 2023-11-24T08:24:30.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.162+0000 7fe690d2b640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 0x7fe67006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:30.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.163+0000 7fe690d2b640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 0x7fe67006faf0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe67c007fd0 tx=0x7fe67c007940 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:30.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.166+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe68c0b2760 con 0x7fe68c06e160 2023-11-24T08:24:30.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.389+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7fe68c147290 con 0x7fe67006d630 2023-11-24T08:24:30.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.396+0000 7fe681ffb640 1 -- 172.21.15.107:0/2775625303 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fe67c0070d0 con 0x7fe67006d630 2023-11-24T08:24:30.398 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.398+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 msgr2=0x7fe67006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.398+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 0x7fe67006faf0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe67c007fd0 tx=0x7fe67c007940 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.398+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 msgr2=0x7fe68c15e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.398+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fe688000c00 tx=0x7fe688004730 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 shutdown_connections 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe67006d630 0x7fe67006faf0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe68c146eb0 0x7fe68c0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c0cd2e0 0x7fe68c1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 --2- 172.21.15.107:0/2775625303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe68c06e160 0x7fe68c15e9b0 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 >> 172.21.15.107:0/2775625303 conn(0x7fe68c0b0c50 msgr2=0x7fe68c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:30.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 shutdown_connections 2023-11-24T08:24:30.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:30.399+0000 7fe693790640 1 -- 172.21.15.107:0/2775625303 wait complete. 2023-11-24T08:24:30.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:30.419 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:24:30.472 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:24:30.474 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:24:30.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:24:30.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:24:30.715 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:24:30.716 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:30.716 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:30.778 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:30.778 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:30.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:31.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.090+0000 7fc37b432640 1 Processor -- start 2023-11-24T08:24:31.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.091+0000 7fc37b432640 1 -- start start 2023-11-24T08:24:31.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.092+0000 7fc37b432640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc3740d0240 0x7fc374168080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.092+0000 7fc37b432640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740ce990 con 0x7fc3740d1e20 2023-11-24T08:24:31.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.092+0000 7fc37b432640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740cec70 con 0x7fc37415d4e0 2023-11-24T08:24:31.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.092+0000 7fc37b432640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740d4110 con 0x7fc3740d0240 2023-11-24T08:24:31.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc37ac31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc3740d0240 0x7fc374168080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc379c2f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc3740d1e20 0x7fc3740cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57344/0 (socket says 172.21.15.107:57344) 2023-11-24T08:24:31.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc379c2f640 1 -- 172.21.15.107:0/3355015024 learned_addr learned my addr 172.21.15.107:0/3355015024 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:31.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc37ac31640 1 -- 172.21.15.107:0/3355015024 >> v1:172.21.15.120:6789/0 conn(0x7fc37415d4e0 legacy=0x7fc3740d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:31.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc37ac31640 1 -- 172.21.15.107:0/3355015024 >> v1:172.21.15.107:6789/0 conn(0x7fc3740d1e20 legacy=0x7fc3740cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc37ac31640 1 -- 172.21.15.107:0/3355015024 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740cef50 con 0x7fc3740d0240 2023-11-24T08:24:31.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.093+0000 7fc37942e640 1 -- 172.21.15.107:0/3355015024 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197210901 0 0) 0x7fc3740ce990 con 0x7fc3740d1e20 2023-11-24T08:24:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.094+0000 7fc37ac31640 1 --2- 172.21.15.107:0/3355015024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d0240 0x7fc374168080 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d90 tx=0x7fc36c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17945838ba070551 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.094+0000 7fc37942e640 1 -- 172.21.15.107:0/3355015024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c02cd00 con 0x7fc3740d0240 2023-11-24T08:24:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37942e640 1 -- 172.21.15.107:0/3355015024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c02ce60 con 0x7fc3740d0240 2023-11-24T08:24:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37942e640 1 -- 172.21.15.107:0/3355015024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c034660 con 0x7fc3740d0240 2023-11-24T08:24:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 -- 172.21.15.107:0/3355015024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d0240 msgr2=0x7fc374168080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 --2- 172.21.15.107:0/3355015024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d0240 0x7fc374168080 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d90 tx=0x7fc36c02bf50 comp rx=0 tx=0).stop 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 -- 172.21.15.107:0/3355015024 shutdown_connections 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 --2- 172.21.15.107:0/3355015024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d0240 0x7fc374168080 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 -- 172.21.15.107:0/3355015024 >> 172.21.15.107:0/3355015024 conn(0x7fc37409c510 msgr2=0x7fc3740d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 -- 172.21.15.107:0/3355015024 shutdown_connections 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.095+0000 7fc37b432640 1 -- 172.21.15.107:0/3355015024 wait complete. 2023-11-24T08:24:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.097+0000 7fc37b432640 1 Processor -- start 2023-11-24T08:24:31.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.097+0000 7fc37b432640 1 -- start start 2023-11-24T08:24:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 0x7fc3740b88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc37415d4e0 0x7fc3740c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740b8e00 con 0x7fc3740d1e20 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740c7190 con 0x7fc3740d0240 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc37b432640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740c93c0 con 0x7fc37415d4e0 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.098+0000 7fc379c2f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34982/0 (socket says 172.21.15.107:34982) 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 -- 172.21.15.107:0/2338635429 learned_addr learned my addr 172.21.15.107:0/2338635429 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc37a430640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 0x7fc3740b88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 -- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc37415d4e0 msgr2=0x7fc3740c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc37415d4e0 0x7fc3740c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 -- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 msgr2=0x7fc3740b88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 0x7fc3740b88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 -- 172.21.15.107:0/2338635429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c009a70 con 0x7fc3740d1e20 2023-11-24T08:24:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc37a430640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 0x7fc3740b88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.099+0000 7fc379c2f640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fc36400c4a0 tx=0x7fc36400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.100+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3640074e0 con 0x7fc3740d1e20 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.100+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc364007640 con 0x7fc3740d1e20 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.100+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3740c96a0 con 0x7fc3740d1e20 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.100+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364004a00 con 0x7fc3740d1e20 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.100+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740c9c60 con 0x7fc3740d1e20 2023-11-24T08:24:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.101+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37404e840 con 0x7fc3740d1e20 2023-11-24T08:24:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.102+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fc364004b60 con 0x7fc3740d1e20 2023-11-24T08:24:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.103+0000 7fc36affd640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 0x7fc35806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.104+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc364087b90 con 0x7fc3740d1e20 2023-11-24T08:24:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.104+0000 7fc37a430640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 0x7fc35806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.104+0000 7fc37a430640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 0x7fc35806faf0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc370000f60 tx=0x7fc370009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.108+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3640546a0 con 0x7fc3740d1e20 2023-11-24T08:24:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.330+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc374072090 con 0x7fc3740d1e20 2023-11-24T08:24:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.332+0000 7fc36affd640 1 -- 172.21.15.107:0/2338635429 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+16626 (secure 0 0 0) 0x7fc364018070 con 0x7fc3740d1e20 2023-11-24T08:24:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:31.334 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:31.229251+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":235,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 msgr2=0x7fc35806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 0x7fc35806faf0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc370000f60 tx=0x7fc370009040 comp rx=0 tx=0).stop 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 msgr2=0x7fc3741f5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fc36400c4a0 tx=0x7fc36400c960 comp rx=0 tx=0).stop 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 shutdown_connections 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35806d630 0x7fc35806faf0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc37415d4e0 0x7fc3740c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3740d1e20 0x7fc3741f5de0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 --2- 172.21.15.107:0/2338635429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740d0240 0x7fc3740b88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 >> 172.21.15.107:0/2338635429 conn(0x7fc37409c510 msgr2=0x7fc37415c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 shutdown_connections 2023-11-24T08:24:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.334+0000 7fc37b432640 1 -- 172.21.15.107:0/2338635429 wait complete. 2023-11-24T08:24:31.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:23:48.986422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:23:50.200838+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 235, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:24:31.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:24:31.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.541+0000 7f66ecc51640 1 Processor -- start 2023-11-24T08:24:31.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.542+0000 7f66ecc51640 1 -- start start 2023-11-24T08:24:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66ecc51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e80ccbd0 0x7f66e81660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66ecc51640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66e80ccfb0 0x7f66e8162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66ecc51640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e81655f0 con 0x7f66e8162540 2023-11-24T08:24:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66ecc51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e80cc380 con 0x7f66e80ccbd0 2023-11-24T08:24:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66ecc51640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e8162b30 con 0x7f66e80ccfb0 2023-11-24T08:24:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.543+0000 7f66e659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e80ccbd0 0x7f66e81660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66e80ccbd0 0x7f66e81660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34988/0 (socket says 172.21.15.107:34988) 2023-11-24T08:24:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 -- 172.21.15.107:0/2348357075 learned_addr learned my addr 172.21.15.107:0/2348357075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 -- 172.21.15.107:0/2348357075 >> v2:172.21.15.120:3300/0 conn(0x7f66e80ccfb0 msgr2=0x7f66e8162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:31.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 --2- 172.21.15.107:0/2348357075 >> v2:172.21.15.120:3300/0 conn(0x7f66e80ccfb0 0x7f66e8162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 -- 172.21.15.107:0/2348357075 >> v1:172.21.15.120:6789/0 conn(0x7f66e8162540 legacy=0x7f66e80c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:31.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.544+0000 7f66e659c640 1 -- 172.21.15.107:0/2348357075 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8162e10 con 0x7f66e80ccbd0 2023-11-24T08:24:31.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.545+0000 7f66e659c640 1 --2- 172.21.15.107:0/2348357075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e80ccbd0 0x7f66e81660b0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f66e806e700 tx=0x7f66d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0e7d69466111ae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.545+0000 7f66e5d9b640 1 -- 172.21.15.107:0/2348357075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8031870 con 0x7f66e80ccbd0 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.545+0000 7f66e5d9b640 1 -- 172.21.15.107:0/2348357075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d80319d0 con 0x7f66e80ccbd0 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.545+0000 7f66e5d9b640 1 -- 172.21.15.107:0/2348357075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8031d60 con 0x7f66e80ccbd0 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 -- 172.21.15.107:0/2348357075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e80ccbd0 msgr2=0x7f66e81660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 --2- 172.21.15.107:0/2348357075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e80ccbd0 0x7f66e81660b0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f66e806e700 tx=0x7f66d802b910 comp rx=0 tx=0).stop 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 -- 172.21.15.107:0/2348357075 shutdown_connections 2023-11-24T08:24:31.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 --2- 172.21.15.107:0/2348357075 >> v2:172.21.15.120:3300/0 conn(0x7f66e80ccfb0 0x7f66e8162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 --2- 172.21.15.107:0/2348357075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e80ccbd0 0x7f66e81660b0 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 -- 172.21.15.107:0/2348357075 >> 172.21.15.107:0/2348357075 conn(0x7f66e80b0c50 msgr2=0x7f66e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:31.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 -- 172.21.15.107:0/2348357075 shutdown_connections 2023-11-24T08:24:31.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.546+0000 7f66ecc51640 1 -- 172.21.15.107:0/2348357075 wait complete. 2023-11-24T08:24:31.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.548+0000 7f66ecc51640 1 Processor -- start 2023-11-24T08:24:31.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.548+0000 7f66ecc51640 1 -- start start 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e80ccbd0 0x7f66e8164670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e80ccfb0 0x7f66e81574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8157f70 con 0x7f66e8162540 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80acc50 con 0x7f66e80ccbd0 2023-11-24T08:24:31.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.549+0000 7f66ecc51640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80ac970 con 0x7f66e80ccfb0 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35002/0 (socket says 172.21.15.107:35002) 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 -- 172.21.15.107:0/4284495212 learned_addr learned my addr 172.21.15.107:0/4284495212 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66dffff640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e80ccfb0 0x7f66e81574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e659c640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e80ccbd0 0x7f66e8164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 -- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e80ccfb0 msgr2=0x7f66e81574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e80ccfb0 0x7f66e81574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 -- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e80ccbd0 msgr2=0x7f66e8164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e80ccbd0 0x7f66e8164670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 -- 172.21.15.107:0/4284495212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d80094d0 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.550+0000 7f66e6d9d640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c780 tx=0x7f66e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.551+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0003d80 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.551+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e000ce90 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.551+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00052b0 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.551+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e8163b90 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.551+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e8164150 con 0x7f66e8162540 2023-11-24T08:24:31.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.552+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e8164370 con 0x7f66e8162540 2023-11-24T08:24:31.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.553+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f66e0005410 con 0x7f66e8162540 2023-11-24T08:24:31.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.554+0000 7f66deffd640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 0x7f66c806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.554+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f66e0087dd0 con 0x7f66e8162540 2023-11-24T08:24:31.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.554+0000 7f66e659c640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 0x7f66c806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.555+0000 7f66e659c640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 0x7f66c806faf0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f66d8036000 tx=0x7f66d803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.558+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66e0054980 con 0x7f66e8162540 2023-11-24T08:24:31.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.750+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66e804e8b0 con 0x7f66e8162540 2023-11-24T08:24:31.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.751+0000 7f66deffd640 1 -- 172.21.15.107:0/4284495212 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66e0017070 con 0x7f66e8162540 2023-11-24T08:24:31.753 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:24:31.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 msgr2=0x7f66c806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 0x7f66c806faf0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f66d8036000 tx=0x7f66d803c040 comp rx=0 tx=0).stop 2023-11-24T08:24:31.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 msgr2=0x7f66e8157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c780 tx=0x7f66e000cc40 comp rx=0 tx=0).stop 2023-11-24T08:24:31.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 shutdown_connections 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f66c806d630 0x7f66c806faf0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66e8162540 0x7f66e8157a30 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66e80ccfb0 0x7f66e81574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 --2- 172.21.15.107:0/4284495212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66e80ccbd0 0x7f66e8164670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.754+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 >> 172.21.15.107:0/4284495212 conn(0x7f66e80b0c50 msgr2=0x7f66e80afb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.755+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 shutdown_connections 2023-11-24T08:24:31.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.755+0000 7f66ecc51640 1 -- 172.21.15.107:0/4284495212 wait complete. 2023-11-24T08:24:31.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:24:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.972+0000 7fc9501de640 1 Processor -- start 2023-11-24T08:24:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.972+0000 7fc9501de640 1 -- start start 2023-11-24T08:24:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc9501de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc948072160 0x7fc94815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc9501de640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9480c49f0 0x7fc94806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc9501de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94815ec90 con 0x7fc94806dc40 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc9501de640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94806e090 con 0x7fc948072160 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc9501de640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480cd530 con 0x7fc9480c49f0 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc94df7a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc948072160 0x7fc94815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc948072160 0x7fc94815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35018/0 (socket says 172.21.15.107:35018) 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.974+0000 7fc94e77b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc94806dc40 0x7fc9480cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57346/0 (socket says 172.21.15.107:57346) 2023-11-24T08:24:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 -- 172.21.15.107:0/4248314634 learned_addr learned my addr 172.21.15.107:0/4248314634 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94d779640 1 --2- 172.21.15.107:0/4248314634 >> v2:172.21.15.120:3300/0 conn(0x7fc9480c49f0 0x7fc94806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 -- 172.21.15.107:0/4248314634 >> v2:172.21.15.120:3300/0 conn(0x7fc9480c49f0 msgr2=0x7fc94806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 --2- 172.21.15.107:0/4248314634 >> v2:172.21.15.120:3300/0 conn(0x7fc9480c49f0 0x7fc94806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 -- 172.21.15.107:0/4248314634 >> v1:172.21.15.107:6789/0 conn(0x7fc94806dc40 legacy=0x7fc9480cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.975+0000 7fc94df7a640 1 -- 172.21.15.107:0/4248314634 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480c4ed0 con 0x7fc948072160 2023-11-24T08:24:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.976+0000 7fc94df7a640 1 --2- 172.21.15.107:0/4248314634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc948072160 0x7fc94815e750 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7fc9480cef60 tx=0x7fc94402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5412349e1e316c81 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.976+0000 7fc94cf78640 1 -- 172.21.15.107:0/4248314634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944030860 con 0x7fc948072160 2023-11-24T08:24:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc94cf78640 1 -- 172.21.15.107:0/4248314634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9440309c0 con 0x7fc948072160 2023-11-24T08:24:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc94cf78640 1 -- 172.21.15.107:0/4248314634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944030d50 con 0x7fc948072160 2023-11-24T08:24:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 -- 172.21.15.107:0/4248314634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc948072160 msgr2=0x7fc94815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 --2- 172.21.15.107:0/4248314634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc948072160 0x7fc94815e750 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7fc9480cef60 tx=0x7fc94402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 -- 172.21.15.107:0/4248314634 shutdown_connections 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 --2- 172.21.15.107:0/4248314634 >> v2:172.21.15.120:3300/0 conn(0x7fc9480c49f0 0x7fc94806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 --2- 172.21.15.107:0/4248314634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc948072160 0x7fc94815e750 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 -- 172.21.15.107:0/4248314634 >> 172.21.15.107:0/4248314634 conn(0x7fc9480b0c50 msgr2=0x7fc9480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 -- 172.21.15.107:0/4248314634 shutdown_connections 2023-11-24T08:24:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.977+0000 7fc9501de640 1 -- 172.21.15.107:0/4248314634 wait complete. 2023-11-24T08:24:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.979+0000 7fc9501de640 1 Processor -- start 2023-11-24T08:24:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.979+0000 7fc9501de640 1 -- start start 2023-11-24T08:24:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc948072160 0x7fc948154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9480c49f0 0x7fc948154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948154ec0 con 0x7fc94806dc40 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480c87a0 con 0x7fc9480c49f0 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc9501de640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480c8a80 con 0x7fc948072160 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc94df7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc94df7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35020/0 (socket says 172.21.15.107:35020) 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.980+0000 7fc94df7a640 1 -- 172.21.15.107:0/3741304771 learned_addr learned my addr 172.21.15.107:0/3741304771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 -- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc948072160 msgr2=0x7fc948154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc948072160 0x7fc948154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94e77b640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9480c49f0 0x7fc948154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 -- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9480c49f0 msgr2=0x7fc948154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9480c49f0 0x7fc948154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 -- 172.21.15.107:0/3741304771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440094d0 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc94df7a640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc94402b9e0 tx=0x7fc944004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944030850 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.981+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc944043400 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.982+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9440437b0 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.982+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9480c8d60 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.982+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc948163520 con 0x7fc94806dc40 2023-11-24T08:24:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.983+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9481636b0 con 0x7fc94806dc40 2023-11-24T08:24:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.984+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fc944043910 con 0x7fc94806dc40 2023-11-24T08:24:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.986+0000 7fc93e7fc640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 0x7fc92c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:31.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.986+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fc94403e070 con 0x7fc94806dc40 2023-11-24T08:24:31.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.986+0000 7fc94d779640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 0x7fc92c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:31.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.987+0000 7fc94d779640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 0x7fc92c06faf0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc938006fd0 tx=0x7fc938008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:31.991+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc944078930 con 0x7fc94806dc40 2023-11-24T08:24:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.181+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc94804e8b0 con 0x7fc94806dc40 2023-11-24T08:24:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.182+0000 7fc93e7fc640 1 -- 172.21.15.107:0/3741304771 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v235) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9440340a0 con 0x7fc94806dc40 2023-11-24T08:24:32.184 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 msgr2=0x7fc92c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 0x7fc92c06faf0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc938006fd0 tx=0x7fc938008040 comp rx=0 tx=0).stop 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 msgr2=0x7fc9480ac1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc94402b9e0 tx=0x7fc944004730 comp rx=0 tx=0).stop 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 shutdown_connections 2023-11-24T08:24:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc92c06d630 0x7fc92c06faf0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9480c49f0 0x7fc948154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc948072160 0x7fc948154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 --2- 172.21.15.107:0/3741304771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc94806dc40 0x7fc9480ac1a0 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 >> 172.21.15.107:0/3741304771 conn(0x7fc9480b0c50 msgr2=0x7fc9480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 shutdown_connections 2023-11-24T08:24:32.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.185+0000 7fc9501de640 1 -- 172.21.15.107:0/3741304771 wait complete. 2023-11-24T08:24:32.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:24:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.414+0000 7fa660fb0640 1 Processor -- start 2023-11-24T08:24:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.414+0000 7fa660fb0640 1 -- start start 2023-11-24T08:24:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.415+0000 7fa660fb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa65c1660b0 0x7fa65c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.415+0000 7fa660fb0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c166510 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.415+0000 7fa660fb0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c1655f0 con 0x7fa65c162540 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.415+0000 7fa660fb0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c0cc380 con 0x7fa65c1660b0 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa65a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa65c0ccbd0 0x7fa65c0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57354/0 (socket says 172.21.15.107:57354) 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa65a59c640 1 -- 172.21.15.107:0/3397008139 learned_addr learned my addr 172.21.15.107:0/3397008139 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa659d9b640 1 --2- 172.21.15.107:0/3397008139 >> v2:172.21.15.107:3300/0 conn(0x7fa65c1660b0 0x7fa65c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa65959a640 1 -- 172.21.15.107:0/3397008139 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042567458 0 0) 0x7fa64c001110 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa65959a640 1 -- 172.21.15.107:0/3397008139 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa644003520 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa659d9b640 1 -- 172.21.15.107:0/3397008139 >> v1:172.21.15.120:6789/0 conn(0x7fa65c162540 legacy=0x7fa65c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa659d9b640 1 -- 172.21.15.107:0/3397008139 >> v1:172.21.15.107:6789/0 conn(0x7fa65c0ccbd0 legacy=0x7fa65c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.416+0000 7fa659d9b640 1 -- 172.21.15.107:0/3397008139 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c162b30 con 0x7fa65c1660b0 2023-11-24T08:24:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.418+0000 7fa659d9b640 1 --2- 172.21.15.107:0/3397008139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c1660b0 0x7fa65c162000 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fa6500097a0 tx=0x7fa65002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a5ea07fcb87ab1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.418+0000 7fa65959a640 1 -- 172.21.15.107:0/3397008139 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650031830 con 0x7fa65c1660b0 2023-11-24T08:24:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.418+0000 7fa65959a640 1 -- 172.21.15.107:0/3397008139 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650031990 con 0x7fa65c1660b0 2023-11-24T08:24:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.418+0000 7fa65959a640 1 -- 172.21.15.107:0/3397008139 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650031d20 con 0x7fa65c1660b0 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 -- 172.21.15.107:0/3397008139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c1660b0 msgr2=0x7fa65c162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 --2- 172.21.15.107:0/3397008139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c1660b0 0x7fa65c162000 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fa6500097a0 tx=0x7fa65002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 -- 172.21.15.107:0/3397008139 shutdown_connections 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 --2- 172.21.15.107:0/3397008139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c1660b0 0x7fa65c162000 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 -- 172.21.15.107:0/3397008139 >> 172.21.15.107:0/3397008139 conn(0x7fa65c0b0c50 msgr2=0x7fa65c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 -- 172.21.15.107:0/3397008139 shutdown_connections 2023-11-24T08:24:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.419+0000 7fa660fb0640 1 -- 172.21.15.107:0/3397008139 wait complete. 2023-11-24T08:24:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.420+0000 7fa660fb0640 1 Processor -- start 2023-11-24T08:24:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.421+0000 7fa660fb0640 1 -- start start 2023-11-24T08:24:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa65c162540 0x7fa65c157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa65c1660b0 0x7fa65c1642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0c8660 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0c9f20 con 0x7fa65c162540 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa660fb0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c158350 con 0x7fa65c1660b0 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35052/0 (socket says 172.21.15.107:35052) 2023-11-24T08:24:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 -- 172.21.15.107:0/2220853260 learned_addr learned my addr 172.21.15.107:0/2220853260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 -- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa65c1660b0 msgr2=0x7fa65c1642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa65c1660b0 0x7fa65c1642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 -- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa65c162540 msgr2=0x7fa65c157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa65c162540 0x7fa65c157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.422+0000 7fa65a59c640 1 -- 172.21.15.107:0/2220853260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650009400 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa65a59c640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c420 tx=0x7fa64c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c005ce0 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c005e40 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0072f0 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c0ca1c0 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.423+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c0c6640 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.424+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c0c8af0 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.425+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fa64c007510 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.426+0000 7fa64affd640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 0x7fa63c06fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.426+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa64c087d20 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.426+0000 7fa659d9b640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 0x7fa63c06fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.427+0000 7fa659d9b640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 0x7fa63c06fb40 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa650036000 tx=0x7fa650030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.430+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa64c0549b0 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.620+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa65c166510 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.621+0000 7fa64affd640 1 -- 172.21.15.107:0/2220853260 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v235) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa65c166510 con 0x7fa65c0ccbd0 2023-11-24T08:24:32.623 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:24:32.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.623+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 msgr2=0x7fa63c06fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.623+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 0x7fa63c06fb40 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa650036000 tx=0x7fa650030000 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.623+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 msgr2=0x7fa65c0adea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.623+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c420 tx=0x7fa64c00c8e0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 shutdown_connections 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa63c06d680 0x7fa63c06fb40 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa65c1660b0 0x7fa65c1642b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa65c162540 0x7fa65c157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 --2- 172.21.15.107:0/2220853260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa65c0ccbd0 0x7fa65c0adea0 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 >> 172.21.15.107:0/2220853260 conn(0x7fa65c0b0c50 msgr2=0x7fa65c0aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:32.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 shutdown_connections 2023-11-24T08:24:32.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.624+0000 7fa660fb0640 1 -- 172.21.15.107:0/2220853260 wait complete. 2023-11-24T08:24:32.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:24:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.852+0000 7f939bffc640 1 Processor -- start 2023-11-24T08:24:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.852+0000 7f939bffc640 1 -- start start 2023-11-24T08:24:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.853+0000 7f939bffc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9394072160 0x7f939415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f939bffc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939415ec90 con 0x7f939406de10 2023-11-24T08:24:32.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f939bffc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93940cd530 con 0x7f93940c49f0 2023-11-24T08:24:32.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f939bffc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940c4ed0 con 0x7f9394072160 2023-11-24T08:24:32.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f939a599640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f939406de10 0x7f93940705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57356/0 (socket says 172.21.15.107:57356) 2023-11-24T08:24:32.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f939a599640 1 -- 172.21.15.107:0/1442353295 learned_addr learned my addr 172.21.15.107:0/1442353295 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591110080 0 0) 0x7f938c001490 con 0x7f939406de10 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9380003520 con 0x7f939406de10 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.854+0000 7f9399d98640 1 --2- 172.21.15.107:0/1442353295 >> v2:172.21.15.120:3300/0 conn(0x7f9394072160 0x7f939415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742667462 0 0) 0x7f9380003520 con 0x7f939406de10 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 >> v2:172.21.15.120:3300/0 conn(0x7f9394072160 msgr2=0x7f939415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 --2- 172.21.15.107:0/1442353295 >> v2:172.21.15.120:3300/0 conn(0x7f9394072160 0x7f939415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 >> v1:172.21.15.120:6789/0 conn(0x7f93940c49f0 legacy=0x7f939406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940c51b0 con 0x7f939406de10 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f938c001b80 con 0x7f939406de10 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 >> v1:172.21.15.107:6789/0 conn(0x7f939406de10 legacy=0x7f93940705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93800039c0 0x7f9380003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800051d0 0x7f938000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940c4ed0 con 0x7f93800047c0 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938000c120 con 0x7f93800039c0 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938000e4f0 con 0x7f93800051d0 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9399597640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.855+0000 7f9399d98640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93800039c0 0x7f9380003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f9399597640 1 -- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800051d0 msgr2=0x7f938000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f9399597640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800051d0 0x7f938000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f9399597640 1 -- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93800039c0 msgr2=0x7f9380003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f9399597640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93800039c0 0x7f9380003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f9399597640 1 -- 172.21.15.107:0/1442353295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9380003520 con 0x7f93800047c0 2023-11-24T08:24:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.856+0000 7f939a599640 1 -- 172.21.15.107:0/1442353295 reap_dead start 2023-11-24T08:24:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f9399597640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 0x7f9380004c40 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f9384001750 tx=0x7f93800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fc4b78238eeb02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93840052d0 con 0x7f93800047c0 2023-11-24T08:24:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938402e050 con 0x7f93800047c0 2023-11-24T08:24:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f9398d96640 1 -- 172.21.15.107:0/1442353295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384003830 con 0x7f93800047c0 2023-11-24T08:24:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f939bffc640 1 -- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 msgr2=0x7f9380004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.857+0000 7f939bffc640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 0x7f9380004c40 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f9384001750 tx=0x7f93800057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.858+0000 7f939bffc640 1 -- 172.21.15.107:0/1442353295 shutdown_connections 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.858+0000 7f939bffc640 1 --2- 172.21.15.107:0/1442353295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.858+0000 7f939bffc640 1 -- 172.21.15.107:0/1442353295 >> 172.21.15.107:0/1442353295 conn(0x7f93940b0c50 msgr2=0x7f93940b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.858+0000 7f939bffc640 1 -- 172.21.15.107:0/1442353295 shutdown_connections 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.858+0000 7f939bffc640 1 -- 172.21.15.107:0/1442353295 wait complete. 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.859+0000 7f939bffc640 1 Processor -- start 2023-11-24T08:24:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 -- start start 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800047c0 0x7f939414a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93940b1d40 0x7f93940b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b2700 con 0x7f93940cb870 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939414a980 con 0x7f93940b1d40 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.860+0000 7f939bffc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93940cc260 con 0x7f93800047c0 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35076/0 (socket says 172.21.15.107:35076) 2023-11-24T08:24:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 -- 172.21.15.107:0/1307964218 learned_addr learned my addr 172.21.15.107:0/1307964218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f9399597640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93940b1d40 0x7f93940b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 -- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800047c0 msgr2=0x7f939414a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f9399d98640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800047c0 0x7f939414a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800047c0 0x7f939414a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 -- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93940b1d40 msgr2=0x7f93940b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93940b1d40 0x7f93940b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 -- 172.21.15.107:0/1307964218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940cc510 con 0x7f93940cb870 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.861+0000 7f939a599640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f93940cd1b0 tx=0x7f938c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.862+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00cec0 con 0x7f93940cb870 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.862+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c00a610 con 0x7f93940cb870 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.862+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00a9c0 con 0x7f93940cb870 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.862+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939414ac20 con 0x7f93940cb870 2023-11-24T08:24:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.862+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93940cc7f0 con 0x7f93940cb870 2023-11-24T08:24:32.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.863+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939404e8b0 con 0x7f93940cb870 2023-11-24T08:24:32.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.864+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f938c00ab20 con 0x7f93940cb870 2023-11-24T08:24:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.865+0000 7f938a7fc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 0x7f937806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.865+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f938c0882e0 con 0x7f93940cb870 2023-11-24T08:24:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.865+0000 7f9399d98640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 0x7f937806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.866+0000 7f9399d98640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 0x7f937806fb20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f9390006fd0 tx=0x7f9390008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:32.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:32.869+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f938c054e90 con 0x7f93940cb870 2023-11-24T08:24:33.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.075+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f939415ec90 con 0x7f93940cb870 2023-11-24T08:24:33.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.076+0000 7f938a7fc640 1 -- 172.21.15.107:0/1307964218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v235) v1 ==== 80+0+0 (secure 0 0 0) 0x7f939415ec90 con 0x7f93940cb870 2023-11-24T08:24:33.077 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:24:33.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 msgr2=0x7f937806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 0x7f937806fb20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f9390006fd0 tx=0x7f9390008040 comp rx=0 tx=0).stop 2023-11-24T08:24:33.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 msgr2=0x7f93940cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f93940cd1b0 tx=0x7f938c00ccd0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 shutdown_connections 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f937806d660 0x7f937806fb20 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93940cb870 0x7f93940cbd20 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93940b1d40 0x7f93940b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 --2- 172.21.15.107:0/1307964218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93800047c0 0x7f939414a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 >> 172.21.15.107:0/1307964218 conn(0x7f93940b0c50 msgr2=0x7f93940ac790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 shutdown_connections 2023-11-24T08:24:33.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.078+0000 7f939bffc640 1 -- 172.21.15.107:0/1307964218 wait complete. 2023-11-24T08:24:33.100 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T08:24:33.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:33.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T08:24:33.222 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:24:33.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:33.222 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:24:33.288 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:33.288 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:33.330 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:33.330 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:33.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.629+0000 7f2478451640 1 Processor -- start 2023-11-24T08:24:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.629+0000 7f2478451640 1 -- start start 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f2478451640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f247015c7f0 0x7f247006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f2478451640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24700cece0 con 0x7f24700cf450 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f2478451640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247006d040 con 0x7f24700ce6c0 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f2478451640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24700ceaf0 con 0x7f247015c7f0 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f24761ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f247015c7f0 0x7f247006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f24759ec640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24700cf450 0x7f24700cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57368/0 (socket says 172.21.15.107:57368) 2023-11-24T08:24:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.631+0000 7f24759ec640 1 -- 172.21.15.107:0/1605369909 learned_addr learned my addr 172.21.15.107:0/1605369909 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.632+0000 7f24751eb640 1 -- 172.21.15.107:0/1605369909 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118772835 0 0) 0x7f24700cece0 con 0x7f24700cf450 2023-11-24T08:24:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.632+0000 7f24751eb640 1 -- 172.21.15.107:0/1605369909 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c003520 con 0x7f24700cf450 2023-11-24T08:24:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.632+0000 7f24761ed640 1 -- 172.21.15.107:0/1605369909 >> v1:172.21.15.120:6789/0 conn(0x7f24700ce6c0 legacy=0x7f24700714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.632+0000 7f24761ed640 1 -- 172.21.15.107:0/1605369909 >> v1:172.21.15.107:6789/0 conn(0x7f24700cf450 legacy=0x7f24700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.632+0000 7f24761ed640 1 -- 172.21.15.107:0/1605369909 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247006d320 con 0x7f247015c7f0 2023-11-24T08:24:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.633+0000 7f24761ed640 1 --2- 172.21.15.107:0/1605369909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015c7f0 0x7f247006db20 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097a0 tx=0x7f246c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a4e508e2e0971d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.634+0000 7f24751eb640 1 -- 172.21.15.107:0/1605369909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c030610 con 0x7f247015c7f0 2023-11-24T08:24:33.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.634+0000 7f24751eb640 1 -- 172.21.15.107:0/1605369909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c030770 con 0x7f247015c7f0 2023-11-24T08:24:33.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.634+0000 7f24751eb640 1 -- 172.21.15.107:0/1605369909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c030b00 con 0x7f247015c7f0 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.634+0000 7f2478451640 1 -- 172.21.15.107:0/1605369909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015c7f0 msgr2=0x7f247006db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.634+0000 7f2478451640 1 --2- 172.21.15.107:0/1605369909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015c7f0 0x7f247006db20 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097a0 tx=0x7f246c02b960 comp rx=0 tx=0).stop 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.635+0000 7f2478451640 1 -- 172.21.15.107:0/1605369909 shutdown_connections 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.635+0000 7f2478451640 1 --2- 172.21.15.107:0/1605369909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f247015c7f0 0x7f247006db20 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.635+0000 7f2478451640 1 -- 172.21.15.107:0/1605369909 >> 172.21.15.107:0/1605369909 conn(0x7f247006a920 msgr2=0x7f247006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.635+0000 7f2478451640 1 -- 172.21.15.107:0/1605369909 shutdown_connections 2023-11-24T08:24:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.635+0000 7f2478451640 1 -- 172.21.15.107:0/1605369909 wait complete. 2023-11-24T08:24:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.637+0000 7f2478451640 1 Processor -- start 2023-11-24T08:24:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.637+0000 7f2478451640 1 -- start start 2023-11-24T08:24:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 0x7f24701442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247015c7f0 0x7f24701596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247015b940 con 0x7f24700cf450 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700b3b80 con 0x7f247015c7f0 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f2478451640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24700b3e60 con 0x7f24700ce6c0 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35080/0 (socket says 172.21.15.107:35080) 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 -- 172.21.15.107:0/1762967099 learned_addr learned my addr 172.21.15.107:0/1762967099 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24761ed640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 0x7f24701442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 -- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 msgr2=0x7f24701442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 0x7f24701442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 -- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247015c7f0 msgr2=0x7f24701596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24769ee640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247015c7f0 0x7f24701596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247015c7f0 0x7f24701596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24759ec640 1 -- 172.21.15.107:0/1762967099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c009400 con 0x7f24700cf450 2023-11-24T08:24:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.638+0000 7f24761ed640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 0x7f24701442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f24759ec640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f246000c500 tx=0x7f246000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460005ce0 con 0x7f24700cf450 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2460005e40 con 0x7f24700cf450 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460007330 con 0x7f24700cf450 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2470151220 con 0x7f24700cf450 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.639+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24701517e0 con 0x7f24700cf450 2023-11-24T08:24:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.640+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2470151a00 con 0x7f24700cf450 2023-11-24T08:24:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.641+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f2460007550 con 0x7f24700cf450 2023-11-24T08:24:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.642+0000 7f2466ffd640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 0x7f245406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.642+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f2460088040 con 0x7f24700cf450 2023-11-24T08:24:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.643+0000 7f24761ed640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 0x7f245406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.643+0000 7f24761ed640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 0x7f245406faf0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f246c036000 tx=0x7f246c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.646+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2460054c10 con 0x7f24700cf450 2023-11-24T08:24:33.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.841+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2470067470 con 0x7f24700cf450 2023-11-24T08:24:33.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.845+0000 7f2466ffd640 1 -- 172.21.15.107:0/1762967099 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f2460017070 con 0x7f24700cf450 2023-11-24T08:24:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:33.847 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:31.229251+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":235,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:33.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 msgr2=0x7f245406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 0x7f245406faf0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f246c036000 tx=0x7f246c03b040 comp rx=0 tx=0).stop 2023-11-24T08:24:33.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 msgr2=0x7f24701ef200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:33.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f246000c500 tx=0x7f246000c9c0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 shutdown_connections 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f245406d630 0x7f245406faf0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f247015c7f0 0x7f24701596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24700cf450 0x7f24701ef200 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 --2- 172.21.15.107:0/1762967099 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24700ce6c0 0x7f24701442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 >> 172.21.15.107:0/1762967099 conn(0x7f247006a920 msgr2=0x7f247006c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 shutdown_connections 2023-11-24T08:24:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:33.848+0000 7f2478451640 1 -- 172.21.15.107:0/1762967099 wait complete. 2023-11-24T08:24:33.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:23:48.986422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:23:50.200838+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 235, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:24:33.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:24:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.082+0000 7f2588051640 1 Processor -- start 2023-11-24T08:24:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.082+0000 7f2588051640 1 -- start start 2023-11-24T08:24:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f2588051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f25800c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f2588051640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25800c57e0 0x7f258006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f2588051640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258015eea0 con 0x7f258015b440 2023-11-24T08:24:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f2588051640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258015b820 con 0x7f25800c57e0 2023-11-24T08:24:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f2588051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25800abf30 con 0x7f25800cc0f0 2023-11-24T08:24:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.084+0000 7f25865ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25800c57e0 0x7f258006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25800c57e0 0x7f258006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35082/0 (socket says 172.21.15.107:35082) 2023-11-24T08:24:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 -- 172.21.15.107:0/2845353473 learned_addr learned my addr 172.21.15.107:0/2845353473 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25855ec640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f25800c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 -- 172.21.15.107:0/2845353473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 msgr2=0x7f25800c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f25800c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 -- 172.21.15.107:0/2845353473 >> v1:172.21.15.120:6789/0 conn(0x7f258015b440 legacy=0x7f25800cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.085+0000 7f25865ee640 1 -- 172.21.15.107:0/2845353473 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258015f0f0 con 0x7f25800c57e0 2023-11-24T08:24:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.086+0000 7f25865ee640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25800c57e0 0x7f258006c5f0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f2578009e50 tx=0x7f257802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2406ed87f74fb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2584deb640 1 -- 172.21.15.107:0/2845353473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257802c9e0 con 0x7f25800c57e0 2023-11-24T08:24:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2584deb640 1 -- 172.21.15.107:0/2845353473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257802cb40 con 0x7f25800c57e0 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2584deb640 1 -- 172.21.15.107:0/2845353473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780343f0 con 0x7f25800c57e0 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 -- 172.21.15.107:0/2845353473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25800c57e0 msgr2=0x7f258006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25800c57e0 0x7f258006c5f0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f2578009e50 tx=0x7f257802be40 comp rx=0 tx=0).stop 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 -- 172.21.15.107:0/2845353473 shutdown_connections 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25800c57e0 0x7f258006c5f0 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 --2- 172.21.15.107:0/2845353473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f25800c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 -- 172.21.15.107:0/2845353473 >> 172.21.15.107:0/2845353473 conn(0x7f25800b0c50 msgr2=0x7f25800b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:34.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.087+0000 7f2588051640 1 -- 172.21.15.107:0/2845353473 shutdown_connections 2023-11-24T08:24:34.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.088+0000 7f2588051640 1 -- 172.21.15.107:0/2845353473 wait complete. 2023-11-24T08:24:34.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.089+0000 7f2588051640 1 Processor -- start 2023-11-24T08:24:34.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.089+0000 7f2588051640 1 -- start start 2023-11-24T08:24:34.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 0x7f2580154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f2580159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b3410 con 0x7f258015b440 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b36f0 con 0x7f25800c57e0 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.090+0000 7f2588051640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b39d0 con 0x7f25800cc0f0 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35096/0 (socket says 172.21.15.107:35096) 2023-11-24T08:24:34.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 -- 172.21.15.107:0/839326697 learned_addr learned my addr 172.21.15.107:0/839326697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25855ec640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f2580159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f2585ded640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 0x7f2580154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 -- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 msgr2=0x7f2580159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f2580159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 -- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 msgr2=0x7f2580154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 0x7f2580154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25865ee640 1 -- 172.21.15.107:0/839326697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578009b30 con 0x7f258015b440 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f25855ec640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f2580159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.091+0000 7f2585ded640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 0x7f2580154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f25865ee640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f2578036040 tx=0x7f2578004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578009510 con 0x7f258015b440 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257803e450 con 0x7f258015b440 2023-11-24T08:24:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257803e7e0 con 0x7f258015b440 2023-11-24T08:24:34.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25800b3cb0 con 0x7f258015b440 2023-11-24T08:24:34.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.092+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2580154ad0 con 0x7f258015b440 2023-11-24T08:24:34.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.093+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2580154c60 con 0x7f258015b440 2023-11-24T08:24:34.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.094+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f257803e940 con 0x7f258015b440 2023-11-24T08:24:34.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.095+0000 7f25767fc640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 0x7f256406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.095+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f25780ab710 con 0x7f258015b440 2023-11-24T08:24:34.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.095+0000 7f2585ded640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 0x7f256406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.096+0000 7f2585ded640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 0x7f256406faf0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f257c00a830 tx=0x7f257c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.099+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25780783e0 con 0x7f258015b440 2023-11-24T08:24:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.290+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2580154c60 con 0x7f258015b440 2023-11-24T08:24:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.290+0000 7f25767fc640 1 -- 172.21.15.107:0/839326697 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2578043020 con 0x7f258015b440 2023-11-24T08:24:34.292 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:24:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 msgr2=0x7f256406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 0x7f256406faf0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f257c00a830 tx=0x7f257c008040 comp rx=0 tx=0).stop 2023-11-24T08:24:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 msgr2=0x7f2580159fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f2578036040 tx=0x7f2578004a40 comp rx=0 tx=0).stop 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 shutdown_connections 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f256406d630 0x7f256406faf0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258015b440 0x7f2580159fc0 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25800cc0f0 0x7f2580159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 --2- 172.21.15.107:0/839326697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25800c57e0 0x7f2580154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 >> 172.21.15.107:0/839326697 conn(0x7f25800b0c50 msgr2=0x7f2580070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 shutdown_connections 2023-11-24T08:24:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.293+0000 7f2588051640 1 -- 172.21.15.107:0/839326697 wait complete. 2023-11-24T08:24:34.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:24:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.522+0000 7f378cd6c640 1 Processor -- start 2023-11-24T08:24:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.523+0000 7f378cd6c640 1 -- start start 2023-11-24T08:24:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.524+0000 7f378cd6c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37880cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.524+0000 7f378cd6c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37880d1f40 0x7f37880ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.524+0000 7f378cd6c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37880d15e0 con 0x7f378815d3f0 2023-11-24T08:24:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.524+0000 7f378cd6c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37880d0280 con 0x7f37880d1f40 2023-11-24T08:24:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.524+0000 7f378cd6c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37880b0eb0 con 0x7f378815d7d0 2023-11-24T08:24:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.525+0000 7f3787fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37880d1f40 0x7f37880ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.525+0000 7f3786ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37880cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.525+0000 7f37877fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f378815d3f0 0x7f37880d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:40436/0 (socket says 172.21.15.107:40436) 2023-11-24T08:24:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.525+0000 7f37877fe640 1 -- 172.21.15.107:0/4153745866 learned_addr learned my addr 172.21.15.107:0/4153745866 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.526+0000 7f3787fff640 1 -- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 msgr2=0x7f37880cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.526+0000 7f3787fff640 1 --2- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37880cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.526+0000 7f3787fff640 1 -- 172.21.15.107:0/4153745866 >> v1:172.21.15.120:6789/0 conn(0x7f378815d3f0 legacy=0x7f37880d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.526+0000 7f3787fff640 1 -- 172.21.15.107:0/4153745866 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37880d1300 con 0x7f37880d1f40 2023-11-24T08:24:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.527+0000 7f3787fff640 1 --2- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880ce280 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5924617062621e0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.528+0000 7f37867fc640 1 -- 172.21.15.107:0/4153745866 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378002d040 con 0x7f37880d1f40 2023-11-24T08:24:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.528+0000 7f37867fc640 1 -- 172.21.15.107:0/4153745866 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780031810 con 0x7f37880d1f40 2023-11-24T08:24:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.528+0000 7f37867fc640 1 -- 172.21.15.107:0/4153745866 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780031ba0 con 0x7f37880d1f40 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.528+0000 7f378cd6c640 1 -- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 msgr2=0x7f37880ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.528+0000 7f378cd6c640 1 --2- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880ce280 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b8b0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 -- 172.21.15.107:0/4153745866 shutdown_connections 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 --2- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880ce280 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 --2- 172.21.15.107:0/4153745866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37880cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 -- 172.21.15.107:0/4153745866 >> 172.21.15.107:0/4153745866 conn(0x7f378809c5b0 msgr2=0x7f37880d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 -- 172.21.15.107:0/4153745866 shutdown_connections 2023-11-24T08:24:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.529+0000 7f378cd6c640 1 -- 172.21.15.107:0/4153745866 wait complete. 2023-11-24T08:24:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.530+0000 7f378cd6c640 1 Processor -- start 2023-11-24T08:24:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.531+0000 7f378cd6c640 1 -- start start 2023-11-24T08:24:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37881505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37880b2e40 con 0x7f378815d3f0 2023-11-24T08:24:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37880cc4c0 con 0x7f37880d1f40 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f378cd6c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37880b5440 con 0x7f378815d7d0 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35100/0 (socket says 172.21.15.107:35100) 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 -- 172.21.15.107:0/2899907676 learned_addr learned my addr 172.21.15.107:0/2899907676 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f37877fe640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3787fff640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37881505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 -- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 msgr2=0x7f37881505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37881505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 -- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 msgr2=0x7f37880b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.532+0000 7f3786ffd640 1 -- 172.21.15.107:0/2899907676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780009400 con 0x7f378815d3f0 2023-11-24T08:24:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f3787fff640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37881505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f3786ffd640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f377000e840 tx=0x7f377000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700094f0 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377000c940 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000ccd0 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37880b5720 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.533+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37880b3370 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.534+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378804e840 con 0x7f378815d3f0 2023-11-24T08:24:34.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.535+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f3770013470 con 0x7f378815d3f0 2023-11-24T08:24:34.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.536+0000 7f377ffff640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 0x7f376806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.536+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f3770088ae0 con 0x7f378815d3f0 2023-11-24T08:24:34.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.537+0000 7f37877fe640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 0x7f376806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.537+0000 7f37877fe640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 0x7f376806faf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f3778005c00 tx=0x7f3778009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.541+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37700555f0 con 0x7f378815d3f0 2023-11-24T08:24:34.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.733+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37880b27c0 con 0x7f378815d3f0 2023-11-24T08:24:34.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.734+0000 7f377ffff640 1 -- 172.21.15.107:0/2899907676 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v235) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37700210a0 con 0x7f378815d3f0 2023-11-24T08:24:34.736 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:24:34.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 msgr2=0x7f376806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 0x7f376806faf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f3778005c00 tx=0x7f3778009040 comp rx=0 tx=0).stop 2023-11-24T08:24:34.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 msgr2=0x7f37880cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f377000e840 tx=0x7f377000ed00 comp rx=0 tx=0).stop 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 shutdown_connections 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f376806d630 0x7f376806faf0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f378815d7d0 0x7f37881505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f378815d3f0 0x7f37880cbf80 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 --2- 172.21.15.107:0/2899907676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37880d1f40 0x7f37880b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.736+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 >> 172.21.15.107:0/2899907676 conn(0x7f378809c5b0 msgr2=0x7f378815c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.737+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 shutdown_connections 2023-11-24T08:24:34.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.737+0000 7f378cd6c640 1 -- 172.21.15.107:0/2899907676 wait complete. 2023-11-24T08:24:34.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:24:34.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.963+0000 7f7324683640 1 Processor -- start 2023-11-24T08:24:34.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.963+0000 7f7324683640 1 -- start start 2023-11-24T08:24:34.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.964+0000 7f7324683640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f731c06d600 0x7f731c0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.964+0000 7f7324683640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c15e400 con 0x7f731c15d7f0 2023-11-24T08:24:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.964+0000 7f7324683640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0c46e0 con 0x7f731c0c4220 2023-11-24T08:24:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.964+0000 7f7324683640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c0ce880 con 0x7f731c06d600 2023-11-24T08:24:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732241f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f731c15d7f0 0x7f731c0c5990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57376/0 (socket says 172.21.15.107:57376) 2023-11-24T08:24:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732241f640 1 -- 172.21.15.107:0/200464441 learned_addr learned my addr 172.21.15.107:0/200464441 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403995065 0 0) 0x7f731c15e400 con 0x7f731c15d7f0 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308003520 con 0x7f731c15d7f0 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f7322c20640 1 --2- 172.21.15.107:0/200464441 >> v2:172.21.15.120:3300/0 conn(0x7f731c06d600 0x7f731c0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395957451 0 0) 0x7f7308003520 con 0x7f731c15d7f0 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 >> v2:172.21.15.120:3300/0 conn(0x7f731c06d600 msgr2=0x7f731c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 --2- 172.21.15.107:0/200464441 >> v2:172.21.15.120:3300/0 conn(0x7f731c06d600 0x7f731c0cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 >> v1:172.21.15.120:6789/0 conn(0x7f731c0c4220 legacy=0x7f731c0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0c49c0 con 0x7f731c15d7f0 2023-11-24T08:24:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.965+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f73180019f0 con 0x7f731c15d7f0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 >> v1:172.21.15.107:6789/0 conn(0x7f731c15d7f0 legacy=0x7f731c0c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 0x7f7308003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73080047c0 0x7f7308004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080051d0 0x7f730800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730800e2e0 con 0x7f73080039c0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730800e450 con 0x7f73080047c0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730800e610 con 0x7f73080051d0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 0x7f7308003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f7321c1e640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73080047c0 0x7f7308004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 -- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080051d0 msgr2=0x7f730800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080051d0 0x7f730800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 -- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73080047c0 msgr2=0x7f7308004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73080047c0 0x7f7308004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f732241f640 1 -- 172.21.15.107:0/200464441 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308003520 con 0x7f73080039c0 2023-11-24T08:24:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f7322c20640 1 -- 172.21.15.107:0/200464441 reap_dead start 2023-11-24T08:24:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.966+0000 7f7321c1e640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73080047c0 0x7f7308004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.967+0000 7f732241f640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 0x7f7308003dc0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f731800a7e0 tx=0x7f73080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed01613c372acfb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.967+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318034630 con 0x7f73080039c0 2023-11-24T08:24:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731802f070 con 0x7f73080039c0 2023-11-24T08:24:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f732141d640 1 -- 172.21.15.107:0/200464441 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318034a10 con 0x7f73080039c0 2023-11-24T08:24:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 -- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 msgr2=0x7f7308003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 0x7f7308003dc0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f731800a7e0 tx=0x7f73080057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 -- 172.21.15.107:0/200464441 shutdown_connections 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 --2- 172.21.15.107:0/200464441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73080039c0 0x7f7308003dc0 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 -- 172.21.15.107:0/200464441 >> 172.21.15.107:0/200464441 conn(0x7f731c0b0c50 msgr2=0x7f731c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 -- 172.21.15.107:0/200464441 shutdown_connections 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.968+0000 7f7324683640 1 -- 172.21.15.107:0/200464441 wait complete. 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.970+0000 7f7324683640 1 Processor -- start 2023-11-24T08:24:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.970+0000 7f7324683640 1 -- start start 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080039c0 0x7f731c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 0x7f731c0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0ad4c0 con 0x7f731c0721a0 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c071a10 con 0x7f731c0b3540 2023-11-24T08:24:34.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7324683640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0ad720 con 0x7f73080039c0 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7322c20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7322c20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35120/0 (socket says 172.21.15.107:35120) 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.971+0000 7f7322c20640 1 -- 172.21.15.107:0/3715156021 learned_addr learned my addr 172.21.15.107:0/3715156021 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7321c1e640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 0x7f731c0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f732241f640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080039c0 0x7f731c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 -- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080039c0 msgr2=0x7f731c071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080039c0 0x7f731c071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 -- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 msgr2=0x7f731c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 0x7f731c0acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 -- 172.21.15.107:0/3715156021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0ada00 con 0x7f731c0721a0 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7321c1e640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 0x7f731c0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:34.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7322c20640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f731c09d170 tx=0x7f731400d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314017070 con 0x7f731c0721a0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314004020 con 0x7f731c0721a0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314004210 con 0x7f731c0721a0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.972+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c071c80 con 0x7f731c0721a0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.973+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c072650 con 0x7f731c0721a0 2023-11-24T08:24:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.973+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c072840 con 0x7f731c0721a0 2023-11-24T08:24:34.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.974+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f7314004370 con 0x7f731c0721a0 2023-11-24T08:24:34.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.975+0000 7f7312ffd640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 0x7f730006fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:34.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.975+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f73140881b0 con 0x7f731c0721a0 2023-11-24T08:24:34.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.976+0000 7f732241f640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 0x7f730006fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:34.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.976+0000 7f732241f640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 0x7f730006fb70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f7318034000 tx=0x7f731802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:34.980+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7314054d40 con 0x7f731c0721a0 2023-11-24T08:24:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.153+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f731c0b3920 con 0x7f731c0721a0 2023-11-24T08:24:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.154+0000 7f7312ffd640 1 -- 172.21.15.107:0/3715156021 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v235) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7314058360 con 0x7f731c0721a0 2023-11-24T08:24:35.156 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.156+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 msgr2=0x7f730006fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.156+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 0x7f730006fb70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f7318034000 tx=0x7f731802f000 comp rx=0 tx=0).stop 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.156+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 msgr2=0x7f731c0cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.156+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f731c09d170 tx=0x7f731400d8a0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 shutdown_connections 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f730006d6b0 0x7f730006fb70 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f731c0721a0 0x7f731c0cbc20 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f731c0b3540 0x7f731c0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 --2- 172.21.15.107:0/3715156021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73080039c0 0x7f731c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 >> 172.21.15.107:0/3715156021 conn(0x7f731c0b0c50 msgr2=0x7f731c158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:35.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 shutdown_connections 2023-11-24T08:24:35.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.157+0000 7f7324683640 1 -- 172.21.15.107:0/3715156021 wait complete. 2023-11-24T08:24:35.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T08:24:35.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.379+0000 7ff29759e640 1 Processor -- start 2023-11-24T08:24:35.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.379+0000 7ff29759e640 1 -- start start 2023-11-24T08:24:35.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff29815a1c0 0x7ff298070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2980b15e0 0x7ff29815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff29806d640 0x7ff29806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980cd920 con 0x7ff2980b15e0 2023-11-24T08:24:35.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980b1a50 con 0x7ff29815a1c0 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.380+0000 7ff29759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29806df90 con 0x7ff29806d640 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2980b15e0 0x7ff29815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2980b15e0 0x7ff29815dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35124/0 (socket says 172.21.15.107:35124) 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 -- 172.21.15.107:0/117262245 learned_addr learned my addr 172.21.15.107:0/117262245 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 -- 172.21.15.107:0/117262245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff29806d640 msgr2=0x7ff29806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 --2- 172.21.15.107:0/117262245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff29806d640 0x7ff29806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 -- 172.21.15.107:0/117262245 >> v2:172.21.15.120:3300/0 conn(0x7ff29815a1c0 msgr2=0x7ff298070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 --2- 172.21.15.107:0/117262245 >> v2:172.21.15.120:3300/0 conn(0x7ff29815a1c0 0x7ff298070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.381+0000 7ff28dd9b640 1 -- 172.21.15.107:0/117262245 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29815e2e0 con 0x7ff2980b15e0 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.382+0000 7ff28dd9b640 1 --2- 172.21.15.107:0/117262245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2980b15e0 0x7ff29815dda0 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7ff2980ccc20 tx=0x7ff28002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34117fda01b736e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.382+0000 7ff295d9b640 1 -- 172.21.15.107:0/117262245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280030850 con 0x7ff2980b15e0 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.382+0000 7ff295d9b640 1 -- 172.21.15.107:0/117262245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2800309b0 con 0x7ff2980b15e0 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff295d9b640 1 -- 172.21.15.107:0/117262245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280030d40 con 0x7ff2980b15e0 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 -- 172.21.15.107:0/117262245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2980b15e0 msgr2=0x7ff29815dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 --2- 172.21.15.107:0/117262245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2980b15e0 0x7ff29815dda0 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7ff2980ccc20 tx=0x7ff28002b990 comp rx=0 tx=0).stop 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 -- 172.21.15.107:0/117262245 shutdown_connections 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 --2- 172.21.15.107:0/117262245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff29806d640 0x7ff29806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 --2- 172.21.15.107:0/117262245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2980b15e0 0x7ff29815dda0 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 --2- 172.21.15.107:0/117262245 >> v2:172.21.15.120:3300/0 conn(0x7ff29815a1c0 0x7ff298070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 -- 172.21.15.107:0/117262245 >> 172.21.15.107:0/117262245 conn(0x7ff2980cbb40 msgr2=0x7ff2980cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 -- 172.21.15.107:0/117262245 shutdown_connections 2023-11-24T08:24:35.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.383+0000 7ff29759e640 1 -- 172.21.15.107:0/117262245 wait complete. 2023-11-24T08:24:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.385+0000 7ff29759e640 1 Processor -- start 2023-11-24T08:24:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.385+0000 7ff29759e640 1 -- start start 2023-11-24T08:24:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2980b15e0 0x7ff2980b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff29815a1c0 0x7ff2980b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29809bc90 con 0x7ff29806d640 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29809bf70 con 0x7ff29815a1c0 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29809c250 con 0x7ff2980b15e0 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35136/0 (socket says 172.21.15.107:35136) 2023-11-24T08:24:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 -- 172.21.15.107:0/2415480010 learned_addr learned my addr 172.21.15.107:0/2415480010 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 -- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2980b15e0 msgr2=0x7ff2980b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2980b15e0 0x7ff2980b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 -- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff29815a1c0 msgr2=0x7ff2980b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff29815a1c0 0x7ff2980b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.386+0000 7ff29659c640 1 -- 172.21.15.107:0/2415480010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800094d0 con 0x7ff29806d640 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff29659c640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7ff28800b410 tx=0x7ff28800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288011830 con 0x7ff29806d640 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2880119b0 con 0x7ff29806d640 2023-11-24T08:24:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29809c530 con 0x7ff29806d640 2023-11-24T08:24:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288011d60 con 0x7ff29806d640 2023-11-24T08:24:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.387+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2981f3590 con 0x7ff29806d640 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.388+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29804e840 con 0x7ff29806d640 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.389+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7ff28801a400 con 0x7ff29806d640 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.390+0000 7ff28f7fe640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 0x7ff27806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.390+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff288088ae0 con 0x7ff29806d640 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.390+0000 7ff28dd9b640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 0x7ff27806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.391+0000 7ff28dd9b640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 0x7ff27806faf0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff280036000 tx=0x7ff28003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.395+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff288055690 con 0x7ff29806d640 2023-11-24T08:24:35.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.588+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff2980cd4d0 con 0x7ff29806d640 2023-11-24T08:24:35.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.588+0000 7ff28f7fe640 1 -- 172.21.15.107:0/2415480010 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v236) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff288015070 con 0x7ff29806d640 2023-11-24T08:24:35.590 INFO:teuthology.orchestra.run.smithi107.stderr:listed 0 entries 2023-11-24T08:24:35.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 msgr2=0x7ff27806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 0x7ff27806faf0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff280036000 tx=0x7ff28003b040 comp rx=0 tx=0).stop 2023-11-24T08:24:35.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 msgr2=0x7ff29809b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:35.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7ff28800b410 tx=0x7ff28800b8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 shutdown_connections 2023-11-24T08:24:35.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff27806d630 0x7ff27806faf0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff29815a1c0 0x7ff2980b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2980b15e0 0x7ff2980b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 --2- 172.21.15.107:0/2415480010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff29806d640 0x7ff29809b5d0 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:35.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.591+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 >> 172.21.15.107:0/2415480010 conn(0x7ff2980cbb40 msgr2=0x7ff2980c19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:35.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.592+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 shutdown_connections 2023-11-24T08:24:35.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:35.592+0000 7ff29759e640 1 -- 172.21.15.107:0/2415480010 wait complete. 2023-11-24T08:24:35.610 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T08:24:35.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:35.610 DEBUG:teuthology.orchestra.run.smithi120:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T08:24:35.723 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:24:35.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:24:35.723 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T08:24:35.774 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:35.774 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:35.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:24:35.833 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:35.833 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:35.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:24:35.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:24:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.117+0000 7f683e70f640 1 Processor -- start 2023-11-24T08:24:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.117+0000 7f683e70f640 1 -- start start 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f683e70f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68380ce6c0 0x7f68380cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f683e70f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68380cece0 con 0x7f68380cf450 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f683e70f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683806d040 con 0x7f683815c7f0 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f683e70f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68380cfe40 con 0x7f68380ce6c0 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f68377fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f68380cf450 0x7f68380cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57388/0 (socket says 172.21.15.107:57388) 2023-11-24T08:24:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.119+0000 7f68377fe640 1 -- 172.21.15.107:0/488054554 learned_addr learned my addr 172.21.15.107:0/488054554 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210273593 0 0) 0x7f68380cece0 con 0x7f68380cf450 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6820003520 con 0x7f68380cf450 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f683ccac640 1 --2- 172.21.15.107:0/488054554 >> v2:172.21.15.120:3300/0 conn(0x7f68380ce6c0 0x7f68380cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095208405 0 0) 0x7f6820003520 con 0x7f68380cf450 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 >> v2:172.21.15.120:3300/0 conn(0x7f68380ce6c0 msgr2=0x7f68380cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 --2- 172.21.15.107:0/488054554 >> v2:172.21.15.120:3300/0 conn(0x7f68380ce6c0 0x7f68380cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 >> v1:172.21.15.120:6789/0 conn(0x7f683815c7f0 legacy=0x7f683806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683806d320 con 0x7f68380cf450 2023-11-24T08:24:36.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f682c001670 con 0x7f68380cf450 2023-11-24T08:24:36.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 >> v1:172.21.15.107:6789/0 conn(0x7f68380cf450 legacy=0x7f68380cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f6820003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68200047c0 0x7f6820004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6820005180 0x7f682000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380cfe40 con 0x7f6820003970 2023-11-24T08:24:36.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682000c0b0 con 0x7f6820005180 2023-11-24T08:24:36.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682000c270 con 0x7f68200047c0 2023-11-24T08:24:36.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.120+0000 7f6837fff640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f6820003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f6837fff640 1 -- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68200047c0 msgr2=0x7f6820004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:36.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f6837fff640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68200047c0 0x7f6820004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f6837fff640 1 -- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6820005180 msgr2=0x7f682000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:36.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f6837fff640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6820005180 0x7f682000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f6837fff640 1 -- 172.21.15.107:0/488054554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820003520 con 0x7f6820003970 2023-11-24T08:24:36.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.121+0000 7f683ccac640 1 -- 172.21.15.107:0/488054554 reap_dead start 2023-11-24T08:24:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.122+0000 7f6837fff640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f6820003d70 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7f6828003d80 tx=0x7f68200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec8e599b3935ff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.122+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828032600 con 0x7f6820003970 2023-11-24T08:24:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.122+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682802e050 con 0x7f6820003970 2023-11-24T08:24:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.122+0000 7f6836ffd640 1 -- 172.21.15.107:0/488054554 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280329e0 con 0x7f6820003970 2023-11-24T08:24:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 -- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 msgr2=0x7f6820003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f6820003d70 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7f6828003d80 tx=0x7f68200057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 -- 172.21.15.107:0/488054554 shutdown_connections 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 --2- 172.21.15.107:0/488054554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f6820003d70 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 -- 172.21.15.107:0/488054554 >> 172.21.15.107:0/488054554 conn(0x7f683806a920 msgr2=0x7f683806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 -- 172.21.15.107:0/488054554 shutdown_connections 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.123+0000 7f683e70f640 1 -- 172.21.15.107:0/488054554 wait complete. 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.125+0000 7f683e70f640 1 Processor -- start 2023-11-24T08:24:36.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.125+0000 7f683e70f640 1 -- start start 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68381533e0 0x7f68380b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68380b3ec0 0x7f683815ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b2180 con 0x7f6820003970 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b2460 con 0x7f68380b3ec0 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f683e70f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b2740 con 0x7f68381533e0 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35152/0 (socket says 172.21.15.107:35152) 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 -- 172.21.15.107:0/4225968705 learned_addr learned my addr 172.21.15.107:0/4225968705 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 -- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68381533e0 msgr2=0x7f68380b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68381533e0 0x7f68380b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 -- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68380b3ec0 msgr2=0x7f683815ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68380b3ec0 0x7f683815ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.126+0000 7f6837fff640 1 -- 172.21.15.107:0/4225968705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380b2a20 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f6837fff640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7f6828035040 tx=0x7f68280036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682803c410 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682803c570 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68380b2d00 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682803c8e0 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.127+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683815b170 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.128+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838067470 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.129+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f6828039050 con 0x7f6820003970 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.130+0000 7f6834ff9640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 0x7f682406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.130+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f68280a9d10 con 0x7f6820003970 2023-11-24T08:24:36.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.130+0000 7f68377fe640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 0x7f682406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.131+0000 7f68377fe640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 0x7f682406faf0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f682c000f60 tx=0x7f682c0015c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.134+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6828076970 con 0x7f6820003970 2023-11-24T08:24:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.364+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68380cd730 con 0x7f6820003970 2023-11-24T08:24:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.366+0000 7f6834ff9640 1 -- 172.21.15.107:0/4225968705 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f682804b070 con 0x7f6820003970 2023-11-24T08:24:36.368 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:36.368 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:24:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.368+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 msgr2=0x7f682406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.368+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 0x7f682406faf0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f682c000f60 tx=0x7f682c0015c0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.368+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 msgr2=0x7f683814f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.368+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7f6828035040 tx=0x7f68280036f0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 shutdown_connections 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f682406d630 0x7f682406faf0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68380b3ec0 0x7f683815ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68381533e0 0x7f68380b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 --2- 172.21.15.107:0/4225968705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6820003970 0x7f683814f4b0 unknown :-1 s=CLOSED pgs=3298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 >> 172.21.15.107:0/4225968705 conn(0x7f683806a920 msgr2=0x7f68380cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 shutdown_connections 2023-11-24T08:24:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.369+0000 7f683e70f640 1 -- 172.21.15.107:0/4225968705 wait complete. 2023-11-24T08:24:36.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.600+0000 7f25f2290640 1 Processor -- start 2023-11-24T08:24:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.600+0000 7f25f2290640 1 -- start start 2023-11-24T08:24:36.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.601+0000 7f25f2290640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0b2250 0x7f25ec15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.601+0000 7f25f2290640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25ec0b2630 0x7f25ec0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.601+0000 7f25f2290640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec15ab80 con 0x7f25ec0cd470 2023-11-24T08:24:36.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.601+0000 7f25f2290640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec15a990 con 0x7f25ec0b2630 2023-11-24T08:24:36.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.601+0000 7f25f2290640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15e680 con 0x7f25ec0b2250 2023-11-24T08:24:36.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.602+0000 7f25ebfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0b2250 0x7f25ec15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.602+0000 7f25f082d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f25ec0cd470 0x7f25ec15b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:40450/0 (socket says 172.21.15.107:40450) 2023-11-24T08:24:36.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.602+0000 7f25f082d640 1 -- 172.21.15.107:0/964660220 learned_addr learned my addr 172.21.15.107:0/964660220 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:36.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.602+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/964660220 >> v2:172.21.15.120:3300/0 conn(0x7f25ec0b2630 0x7f25ec0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.603+0000 7f25eb7fe640 1 -- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0b2250 msgr2=0x7f25ec15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.603+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0b2250 0x7f25ec15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.603+0000 7f25eb7fe640 1 -- 172.21.15.107:0/964660220 >> v1:172.21.15.120:6789/0 conn(0x7f25ec0cd470 legacy=0x7f25ec15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.603+0000 7f25eb7fe640 1 -- 172.21.15.107:0/964660220 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec15e960 con 0x7f25ec0b2630 2023-11-24T08:24:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.604+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec0ccf30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f25e00097a0 tx=0x7f25e002dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0c563f9fb62c93 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.604+0000 7f25eaffd640 1 -- 172.21.15.107:0/964660220 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e002f8a0 con 0x7f25ec0b2630 2023-11-24T08:24:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.604+0000 7f25eaffd640 1 -- 172.21.15.107:0/964660220 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e002fa00 con 0x7f25ec0b2630 2023-11-24T08:24:36.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.604+0000 7f25eaffd640 1 -- 172.21.15.107:0/964660220 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e002fd90 con 0x7f25ec0b2630 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 -- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 msgr2=0x7f25ec0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 --2- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec0ccf30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f25e00097a0 tx=0x7f25e002dae0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 -- 172.21.15.107:0/964660220 shutdown_connections 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 --2- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec0ccf30 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 --2- 172.21.15.107:0/964660220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0b2250 0x7f25ec15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 -- 172.21.15.107:0/964660220 >> 172.21.15.107:0/964660220 conn(0x7f25ec06a920 msgr2=0x7f25ec06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:36.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 -- 172.21.15.107:0/964660220 shutdown_connections 2023-11-24T08:24:36.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.605+0000 7f25f2290640 1 -- 172.21.15.107:0/964660220 wait complete. 2023-11-24T08:24:36.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.607+0000 7f25f2290640 1 Processor -- start 2023-11-24T08:24:36.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.607+0000 7f25f2290640 1 -- start start 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec098920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 0x7f25ec151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15db30 con 0x7f25ec0b2250 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec1510f0 con 0x7f25ec0b2630 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25f2290640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec098e60 con 0x7f25ec0cd470 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35156/0 (socket says 172.21.15.107:35156) 2023-11-24T08:24:36.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.608+0000 7f25ebfff640 1 -- 172.21.15.107:0/3261947026 learned_addr learned my addr 172.21.15.107:0/3261947026 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25f082d640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 0x7f25ec151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 -- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 msgr2=0x7f25ec151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 0x7f25ec151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 -- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 msgr2=0x7f25ec098920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec098920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 -- 172.21.15.107:0/3261947026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e0009400 con 0x7f25ec0b2250 2023-11-24T08:24:36.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25f082d640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 0x7f25ec151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25ebfff640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00e9e0 tx=0x7f25dc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc009670 con 0x7f25ec0b2250 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc00c9d0 con 0x7f25ec0b2250 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc00cd80 con 0x7f25ec0b2250 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.609+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec1507c0 con 0x7f25ec0b2250 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.610+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec150d80 con 0x7f25ec0b2250 2023-11-24T08:24:36.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.611+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec067470 con 0x7f25ec0b2250 2023-11-24T08:24:36.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.612+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f25dc013470 con 0x7f25ec0b2250 2023-11-24T08:24:36.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.613+0000 7f25e8ff9640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 0x7f25d806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:36.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.613+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f25dc088b20 con 0x7f25ec0b2250 2023-11-24T08:24:36.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.613+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 0x7f25d806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:36.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.614+0000 7f25eb7fe640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 0x7f25d806fb20 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f25e0038000 tx=0x7f25e0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:36.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.617+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25dc0556d0 con 0x7f25ec0b2250 2023-11-24T08:24:36.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.810+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25ec0cea00 con 0x7f25ec0b2250 2023-11-24T08:24:36.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.813+0000 7f25e8ff9640 1 -- 172.21.15.107:0/3261947026 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f25dc021090 con 0x7f25ec0b2250 2023-11-24T08:24:36.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:36.815 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":236,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:35.220224+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":235,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.815+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 msgr2=0x7f25d806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.815+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 0x7f25d806fb20 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f25e0038000 tx=0x7f25e0008030 comp rx=0 tx=0).stop 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.815+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 msgr2=0x7f25ec0723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.815+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00e9e0 tx=0x7f25dc00eea0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 shutdown_connections 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f25d806d660 0x7f25d806fb20 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25ec0cd470 0x7f25ec151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25ec0b2630 0x7f25ec098920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 --2- 172.21.15.107:0/3261947026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25ec0b2250 0x7f25ec0723c0 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:36.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 >> 172.21.15.107:0/3261947026 conn(0x7f25ec06a920 msgr2=0x7f25ec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:36.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 shutdown_connections 2023-11-24T08:24:36.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:36.816+0000 7f25f2290640 1 -- 172.21.15.107:0/3261947026 wait complete. 2023-11-24T08:24:36.836 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T08:23:48.986422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T08:23:50.200838+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 235, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:24:36.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:24:37.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.048+0000 7f55ec4ec640 1 Processor -- start 2023-11-24T08:24:37.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.049+0000 7f55ec4ec640 1 -- start start 2023-11-24T08:24:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55e415b440 0x7f55e40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55e40c57e0 0x7f55e40c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e40cc3d0 con 0x7f55e415b440 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e415b820 con 0x7f55e40c57e0 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ec4ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e415eea0 con 0x7f55e40cbf20 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.050+0000 7f55ea288640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55e415b440 0x7f55e40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55e415b440 0x7f55e40cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35168/0 (socket says 172.21.15.107:35168) 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 -- 172.21.15.107:0/1184848268 learned_addr learned my addr 172.21.15.107:0/1184848268 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 -- 172.21.15.107:0/1184848268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 msgr2=0x7f55e40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 --2- 172.21.15.107:0/1184848268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 -- 172.21.15.107:0/1184848268 >> v2:172.21.15.120:3300/0 conn(0x7f55e40c57e0 msgr2=0x7f55e40c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 --2- 172.21.15.107:0/1184848268 >> v2:172.21.15.120:3300/0 conn(0x7f55e40c57e0 0x7f55e40c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.051+0000 7f55ea288640 1 -- 172.21.15.107:0/1184848268 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e40abf30 con 0x7f55e415b440 2023-11-24T08:24:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.052+0000 7f55ea288640 1 --2- 172.21.15.107:0/1184848268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40cb9e0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7f55e40cb910 tx=0x7f55d402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab93daa9c1a4aeae server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.052+0000 7f55e9286640 1 -- 172.21.15.107:0/1184848268 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d40317d0 con 0x7f55e415b440 2023-11-24T08:24:37.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.052+0000 7f55e9286640 1 -- 172.21.15.107:0/1184848268 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4031930 con 0x7f55e415b440 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.052+0000 7f55e9286640 1 -- 172.21.15.107:0/1184848268 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4031cc0 con 0x7f55e415b440 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 -- 172.21.15.107:0/1184848268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 msgr2=0x7f55e40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/1184848268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40cb9e0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7f55e40cb910 tx=0x7f55d402b870 comp rx=0 tx=0).stop 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 -- 172.21.15.107:0/1184848268 shutdown_connections 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/1184848268 >> v2:172.21.15.120:3300/0 conn(0x7f55e40c57e0 0x7f55e40c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/1184848268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e40c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/1184848268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40cb9e0 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 -- 172.21.15.107:0/1184848268 >> 172.21.15.107:0/1184848268 conn(0x7f55e40b0c50 msgr2=0x7f55e40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 -- 172.21.15.107:0/1184848268 shutdown_connections 2023-11-24T08:24:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.053+0000 7f55ec4ec640 1 -- 172.21.15.107:0/1184848268 wait complete. 2023-11-24T08:24:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.055+0000 7f55ec4ec640 1 Processor -- start 2023-11-24T08:24:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.055+0000 7f55ec4ec640 1 -- start start 2023-11-24T08:24:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 0x7f55e40ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e4156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e41568b0 con 0x7f55e415b440 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e40afa70 con 0x7f55e40c57e0 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55ec4ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e40afd50 con 0x7f55e40cbf20 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55eaa89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55eaa89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35176/0 (socket says 172.21.15.107:35176) 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.056+0000 7f55eaa89640 1 -- 172.21.15.107:0/3160017665 learned_addr learned my addr 172.21.15.107:0/3160017665 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 -- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 msgr2=0x7f55e4156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55e9a87640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e4156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e4156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 -- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 msgr2=0x7f55e40ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55ea288640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 0x7f55e40ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 0x7f55e40ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 -- 172.21.15.107:0/3160017665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40094d0 con 0x7f55e415b440 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55e9a87640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e4156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55ea288640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 0x7f55e40ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55eaa89640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00b970 tx=0x7f55dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00cb40 con 0x7f55e415b440 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc00ccc0 con 0x7f55e415b440 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc014680 con 0x7f55e415b440 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.057+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e40b0030 con 0x7f55e415b440 2023-11-24T08:24:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.058+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e40cd960 con 0x7f55e415b440 2023-11-24T08:24:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.059+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e404e8b0 con 0x7f55e415b440 2023-11-24T08:24:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.060+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f55dc01d3b0 con 0x7f55e415b440 2023-11-24T08:24:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.061+0000 7f55e2ffd640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 0x7f55d006fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.061+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f55dc088880 con 0x7f55e415b440 2023-11-24T08:24:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.061+0000 7f55ea288640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 0x7f55d006fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.062+0000 7f55ea288640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 0x7f55d006fb20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036000 tx=0x7f55d403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.066+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55dc055430 con 0x7f55e415b440 2023-11-24T08:24:37.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.255+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55e406dc80 con 0x7f55e415b440 2023-11-24T08:24:37.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.256+0000 7f55e2ffd640 1 -- 172.21.15.107:0/3160017665 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v236) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55dc019020 con 0x7f55e415b440 2023-11-24T08:24:37.258 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 msgr2=0x7f55d006fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 0x7f55d006fb20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036000 tx=0x7f55d403d040 comp rx=0 tx=0).stop 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 msgr2=0x7f55e40c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00b970 tx=0x7f55dc00be30 comp rx=0 tx=0).stop 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 shutdown_connections 2023-11-24T08:24:37.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f55d006d660 0x7f55d006fb20 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55e415b440 0x7f55e40c70d0 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e40cbf20 0x7f55e4156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 --2- 172.21.15.107:0/3160017665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e40c57e0 0x7f55e40ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 >> 172.21.15.107:0/3160017665 conn(0x7f55e40b0c50 msgr2=0x7f55e40cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 shutdown_connections 2023-11-24T08:24:37.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.259+0000 7f55ec4ec640 1 -- 172.21.15.107:0/3160017665 wait complete. 2023-11-24T08:24:37.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T08:24:37.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.486+0000 7fbe77fff640 1 Processor -- start 2023-11-24T08:24:37.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.486+0000 7fbe77fff640 1 -- start start 2023-11-24T08:24:37.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe77fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe780b0b40 0x7fbe780b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe77fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7806c5b0 0x7fbe7806ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe77fff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7815ed20 con 0x7fbe7815b260 2023-11-24T08:24:37.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe77fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7815b640 con 0x7fbe7806c5b0 2023-11-24T08:24:37.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe77fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe780b14f0 con 0x7fbe780b0b40 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.488+0000 7fbe777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7806c5b0 0x7fbe7806ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe7806c5b0 0x7fbe7806ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35180/0 (socket says 172.21.15.107:35180) 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 -- 172.21.15.107:0/1949970376 learned_addr learned my addr 172.21.15.107:0/1949970376 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe767fc640 1 --2- 172.21.15.107:0/1949970376 >> v2:172.21.15.120:3300/0 conn(0x7fbe780b0b40 0x7fbe780b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 -- 172.21.15.107:0/1949970376 >> v2:172.21.15.120:3300/0 conn(0x7fbe780b0b40 msgr2=0x7fbe780b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 --2- 172.21.15.107:0/1949970376 >> v2:172.21.15.120:3300/0 conn(0x7fbe780b0b40 0x7fbe780b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 -- 172.21.15.107:0/1949970376 >> v1:172.21.15.107:6789/0 conn(0x7fbe7815b260 legacy=0x7fbe7806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.489+0000 7fbe777fe640 1 -- 172.21.15.107:0/1949970376 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7815f8a0 con 0x7fbe7806c5b0 2023-11-24T08:24:37.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.490+0000 7fbe777fe640 1 --2- 172.21.15.107:0/1949970376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7806c5b0 0x7fbe7806ca20 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fbe780ce370 tx=0x7fbe7002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0482767e13ba7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.490+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1949970376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7002c9f0 con 0x7fbe7806c5b0 2023-11-24T08:24:37.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.490+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1949970376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7002cb50 con 0x7fbe7806c5b0 2023-11-24T08:24:37.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.490+0000 7fbe75ffb640 1 -- 172.21.15.107:0/1949970376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700343f0 con 0x7fbe7806c5b0 2023-11-24T08:24:37.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 -- 172.21.15.107:0/1949970376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7806c5b0 msgr2=0x7fbe7806ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1949970376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7806c5b0 0x7fbe7806ca20 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fbe780ce370 tx=0x7fbe7002be50 comp rx=0 tx=0).stop 2023-11-24T08:24:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 -- 172.21.15.107:0/1949970376 shutdown_connections 2023-11-24T08:24:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1949970376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe7806c5b0 0x7fbe7806ca20 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 --2- 172.21.15.107:0/1949970376 >> v2:172.21.15.120:3300/0 conn(0x7fbe780b0b40 0x7fbe780b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 -- 172.21.15.107:0/1949970376 >> 172.21.15.107:0/1949970376 conn(0x7fbe780cbb40 msgr2=0x7fbe780cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 -- 172.21.15.107:0/1949970376 shutdown_connections 2023-11-24T08:24:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.491+0000 7fbe77fff640 1 -- 172.21.15.107:0/1949970376 wait complete. 2023-11-24T08:24:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.493+0000 7fbe77fff640 1 Processor -- start 2023-11-24T08:24:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.493+0000 7fbe77fff640 1 -- start start 2023-11-24T08:24:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe7806c5b0 0x7fbe7809c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815b260 0x7fbe780ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781503c0 con 0x7fbe780b0b40 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780afeb0 con 0x7fbe7806c5b0 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe77fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7814fd70 con 0x7fbe7815b260 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35190/0 (socket says 172.21.15.107:35190) 2023-11-24T08:24:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.494+0000 7fbe767fc640 1 -- 172.21.15.107:0/2016514703 learned_addr learned my addr 172.21.15.107:0/2016514703 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 -- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815b260 msgr2=0x7fbe780ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815b260 0x7fbe780ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 -- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe7806c5b0 msgr2=0x7fbe7809c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe7806c5b0 0x7fbe7809c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 -- 172.21.15.107:0/2016514703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009b30 con 0x7fbe780b0b40 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe767fc640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00d5f0 tx=0x7fbe6c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0125f0 con 0x7fbe780b0b40 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c012750 con 0x7fbe780b0b40 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.495+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe78150050 con 0x7fbe780b0b40 2023-11-24T08:24:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.496+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c012b00 con 0x7fbe780b0b40 2023-11-24T08:24:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.496+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7814eeb0 con 0x7fbe780b0b40 2023-11-24T08:24:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.497+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7814fbd0 con 0x7fbe780b0b40 2023-11-24T08:24:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.498+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fbe6c012c60 con 0x7fbe780b0b40 2023-11-24T08:24:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.499+0000 7fbe5f7fe640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 0x7fbe6406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.499+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbe6c088dd0 con 0x7fbe780b0b40 2023-11-24T08:24:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.499+0000 7fbe76ffd640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 0x7fbe6406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.500+0000 7fbe76ffd640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 0x7fbe6406faf0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fbe68000f60 tx=0x7fbe680079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.503+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe7814fbd0 con 0x7fbe780b0b40 2023-11-24T08:24:37.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.692+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe7814fbd0 con 0x7fbe780b0b40 2023-11-24T08:24:37.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.693+0000 7fbe5f7fe640 1 -- 172.21.15.107:0/2016514703 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v236) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe7814fbd0 con 0x7fbe780b0b40 2023-11-24T08:24:37.695 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 msgr2=0x7fbe6406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 0x7fbe6406faf0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fbe68000f60 tx=0x7fbe680079b0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 msgr2=0x7fbe7814e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00d5f0 tx=0x7fbe6c00dab0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 shutdown_connections 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbe6406d630 0x7fbe6406faf0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe7815b260 0x7fbe780ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe780b0b40 0x7fbe7814e970 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 --2- 172.21.15.107:0/2016514703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe7806c5b0 0x7fbe7809c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.695+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 >> 172.21.15.107:0/2016514703 conn(0x7fbe780cbb40 msgr2=0x7fbe78151480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:37.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.696+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 shutdown_connections 2023-11-24T08:24:37.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.696+0000 7fbe77fff640 1 -- 172.21.15.107:0/2016514703 wait complete. 2023-11-24T08:24:37.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T08:24:37.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.921+0000 7fa06cc9b640 1 Processor -- start 2023-11-24T08:24:37.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.921+0000 7fa06cc9b640 1 -- start start 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06cc9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa068161b10 0x7fa068168260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06cc9b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa068161f50 con 0x7fa068140520 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06cc9b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06815de00 con 0x7fa0680c67c0 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06cc9b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06815e0e0 con 0x7fa068161b10 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa068140520 0x7fa0680c60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42444/0 (socket says 172.21.15.107:42444) 2023-11-24T08:24:37.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.923+0000 7fa06659c640 1 -- 172.21.15.107:0/822508444 learned_addr learned my addr 172.21.15.107:0/822508444 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa066d9d640 1 --2- 172.21.15.107:0/822508444 >> v2:172.21.15.120:3300/0 conn(0x7fa068161b10 0x7fa068168260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728897360 0 0) 0x7fa068161f50 con 0x7fa068140520 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa054003520 con 0x7fa068140520 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456168155 0 0) 0x7fa054003520 con 0x7fa068140520 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 >> v2:172.21.15.120:3300/0 conn(0x7fa068161b10 msgr2=0x7fa068168260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 --2- 172.21.15.107:0/822508444 >> v2:172.21.15.120:3300/0 conn(0x7fa068161b10 0x7fa068168260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 >> v1:172.21.15.120:6789/0 conn(0x7fa0680c67c0 legacy=0x7fa068161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06815e3c0 con 0x7fa068140520 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa058001670 con 0x7fa068140520 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 >> v1:172.21.15.107:6789/0 conn(0x7fa068140520 legacy=0x7fa0680c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.924+0000 7fa065d9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0540039c0 0x7fa054003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa065d9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0540047c0 0x7fa054004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa065d9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa05400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06815e0e0 con 0x7fa054005180 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05400c080 con 0x7fa0540039c0 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05400c240 con 0x7fa0540047c0 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa05400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa06659c640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0540039c0 0x7fa054003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa05dd9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0540047c0 0x7fa054004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 -- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0540047c0 msgr2=0x7fa054004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0540047c0 0x7fa054004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 -- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0540039c0 msgr2=0x7fa054003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0540039c0 0x7fa054003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.925+0000 7fa066d9d640 1 -- 172.21.15.107:0/822508444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054003520 con 0x7fa054005180 2023-11-24T08:24:37.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.926+0000 7fa066d9d640 1 -- 172.21.15.107:0/822508444 reap_dead start 2023-11-24T08:24:37.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.926+0000 7fa066d9d640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa05400b980 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fc0 tx=0x7fa0540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33349c44b96496e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06002cc60 con 0x7fa054005180 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06000f040 con 0x7fa054005180 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa065d9b640 1 -- 172.21.15.107:0/822508444 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060003990 con 0x7fa054005180 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 -- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 msgr2=0x7fa05400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa05400b980 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fc0 tx=0x7fa0540057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 -- 172.21.15.107:0/822508444 shutdown_connections 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/822508444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa05400b980 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 -- 172.21.15.107:0/822508444 >> 172.21.15.107:0/822508444 conn(0x7fa0680b2e40 msgr2=0x7fa0680b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 -- 172.21.15.107:0/822508444 shutdown_connections 2023-11-24T08:24:37.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.927+0000 7fa06cc9b640 1 -- 172.21.15.107:0/822508444 wait complete. 2023-11-24T08:24:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.929+0000 7fa06cc9b640 1 Processor -- start 2023-11-24T08:24:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.929+0000 7fa06cc9b640 1 -- start start 2023-11-24T08:24:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa06815a320 0x7fa06813e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0680b3220 0x7fa0680b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680cfcb0 con 0x7fa054005180 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680b3be0 con 0x7fa0680b3220 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06cc9b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06813ea10 con 0x7fa06815a320 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48310/0 (socket says 172.21.15.107:48310) 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.930+0000 7fa06659c640 1 -- 172.21.15.107:0/4134538802 learned_addr learned my addr 172.21.15.107:0/4134538802 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa066d9d640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0680b3220 0x7fa0680b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa05dd9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa06815a320 0x7fa06813e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 -- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa06815a320 msgr2=0x7fa06813e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa06815a320 0x7fa06813e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 -- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0680b3220 msgr2=0x7fa0680b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0680b3220 0x7fa0680b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 -- 172.21.15.107:0/4134538802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06813ec90 con 0x7fa054005180 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06659c640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7fa05800e730 tx=0x7fa05800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800c6c0 con 0x7fa054005180 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05800c840 con 0x7fa054005180 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800cbf0 con 0x7fa054005180 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.931+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06813ef70 con 0x7fa054005180 2023-11-24T08:24:37.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.932+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06813f530 con 0x7fa054005180 2023-11-24T08:24:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.933+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06813fd60 con 0x7fa054005180 2023-11-24T08:24:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.934+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fa05800cd50 con 0x7fa054005180 2023-11-24T08:24:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.935+0000 7fa05f7fe640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 0x7fa04806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:37.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.935+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa058088510 con 0x7fa054005180 2023-11-24T08:24:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.935+0000 7fa05dd9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 0x7fa04806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:37.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.936+0000 7fa05dd9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 0x7fa04806fb20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa0681f67f0 tx=0x7fa050009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:37.939+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0580550c0 con 0x7fa054005180 2023-11-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.128+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa0680cb810 con 0x7fa054005180 2023-11-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.129+0000 7fa05f7fe640 1 -- 172.21.15.107:0/4134538802 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v236) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa0580586e0 con 0x7fa054005180 2023-11-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-11-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 msgr2=0x7fa04806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 0x7fa04806fb20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa0681f67f0 tx=0x7fa050009040 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 msgr2=0x7fa0680cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7fa05800e730 tx=0x7fa05800ebf0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 shutdown_connections 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa04806d660 0x7fa04806fb20 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0680b3220 0x7fa0680b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa06815a320 0x7fa06813e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 --2- 172.21.15.107:0/4134538802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa054005180 0x7fa0680cf770 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 >> 172.21.15.107:0/4134538802 conn(0x7fa0680b2e40 msgr2=0x7fa0680b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 shutdown_connections 2023-11-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.132+0000 7fa06cc9b640 1 -- 172.21.15.107:0/4134538802 wait complete. 2023-11-24T08:24:38.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.360+0000 7f4622d0a640 1 Processor -- start 2023-11-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.360+0000 7f4622d0a640 1 -- start start 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4622d0a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 0x7f461c0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4622d0a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f461c0b3890 0x7f461c167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4622d0a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461c06d180 con 0x7f461c1461d0 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4622d0a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0b3d20 con 0x7f461c0b3890 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4622d0a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c06ba70 con 0x7f461c15eaf0 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f46212a7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f461c0b3890 0x7f461c167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4620aa6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f461c1461d0 0x7f461c0b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42452/0 (socket says 172.21.15.107:42452) 2023-11-24T08:24:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f4620aa6640 1 -- 172.21.15.107:0/2873382718 learned_addr learned my addr 172.21.15.107:0/2873382718 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.362+0000 7f46212a7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f461c0b3890 0x7f461c167110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48322/0 (socket says 172.21.15.107:48322) 2023-11-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f4613fff640 1 --2- 172.21.15.107:0/2873382718 >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 0x7f461c0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f46212a7640 1 -- 172.21.15.107:0/2873382718 >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 msgr2=0x7f461c0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f46212a7640 1 --2- 172.21.15.107:0/2873382718 >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 0x7f461c0b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f46212a7640 1 -- 172.21.15.107:0/2873382718 >> v1:172.21.15.107:6789/0 conn(0x7f461c1461d0 legacy=0x7f461c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f46212a7640 1 -- 172.21.15.107:0/2873382718 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c06bd50 con 0x7f461c0b3890 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f46137fe640 1 -- 172.21.15.107:0/2873382718 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051322731 0 0) 0x7f461c06d180 con 0x7f461c1461d0 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.363+0000 7f4613fff640 1 --2- 172.21.15.107:0/2873382718 >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 0x7f461c0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.364+0000 7f46212a7640 1 --2- 172.21.15.107:0/2873382718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c0b3890 0x7f461c167110 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f461c0b6320 tx=0x7f461402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b40da8fd098a44c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.364+0000 7f46137fe640 1 -- 172.21.15.107:0/2873382718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461402cc80 con 0x7f461c0b3890 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.364+0000 7f46137fe640 1 -- 172.21.15.107:0/2873382718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461402cde0 con 0x7f461c0b3890 2023-11-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.364+0000 7f46137fe640 1 -- 172.21.15.107:0/2873382718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614034660 con 0x7f461c0b3890 2023-11-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 -- 172.21.15.107:0/2873382718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c0b3890 msgr2=0x7f461c167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2873382718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c0b3890 0x7f461c167110 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f461c0b6320 tx=0x7f461402bed0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 -- 172.21.15.107:0/2873382718 shutdown_connections 2023-11-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2873382718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c0b3890 0x7f461c167110 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2873382718 >> v2:172.21.15.120:3300/0 conn(0x7f461c15eaf0 0x7f461c0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 -- 172.21.15.107:0/2873382718 >> 172.21.15.107:0/2873382718 conn(0x7f461c061340 msgr2=0x7f461c061750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 -- 172.21.15.107:0/2873382718 shutdown_connections 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.365+0000 7f4622d0a640 1 -- 172.21.15.107:0/2873382718 wait complete. 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.367+0000 7f4622d0a640 1 Processor -- start 2023-11-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.367+0000 7f4622d0a640 1 -- start start 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f461c0b3890 0x7f461c0d1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 0x7f461c0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0cec50 con 0x7f461c1461d0 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0cca80 con 0x7f461c0b3890 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4622d0a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0cccb0 con 0x7f461c15eaf0 2023-11-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4613fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4613fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48336/0 (socket says 172.21.15.107:48336) 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.368+0000 7f4613fff640 1 -- 172.21.15.107:0/2913633736 learned_addr learned my addr 172.21.15.107:0/2913633736 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4620aa6640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f461c0b3890 0x7f461c0d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 -- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 msgr2=0x7f461c0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f46212a7640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 0x7f461c0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 0x7f461c0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 -- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f461c0b3890 msgr2=0x7f461c0d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f461c0b3890 0x7f461c0d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 -- 172.21.15.107:0/2913633736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614009a70 con 0x7f461c1461d0 2023-11-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f46212a7640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 0x7f461c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4613fff640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c820 tx=0x7f460c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c007a30 con 0x7f461c1461d0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c007b90 con 0x7f461c1461d0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c00ce70 con 0x7f461c1461d0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c0ccf90 con 0x7f461c1461d0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.369+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c144810 con 0x7f461c1461d0 2023-11-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.370+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c0cd4b0 con 0x7f461c1461d0 2023-11-24T08:24:38.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.372+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f460c0113f0 con 0x7f461c1461d0 2023-11-24T08:24:38.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.372+0000 7f46117fa640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 0x7f460806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.373+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f460c088b80 con 0x7f461c1461d0 2023-11-24T08:24:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.373+0000 7f4620aa6640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 0x7f460806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.373+0000 7f4620aa6640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 0x7f460806faf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f4618007fd0 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.377+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f461c0cd4b0 con 0x7f461c1461d0 2023-11-24T08:24:38.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.585+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f461c0ced90 con 0x7f461c1461d0 2023-11-24T08:24:38.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.585+0000 7f46117fa640 1 -- 172.21.15.107:0/2913633736 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f461c0ced90 con 0x7f461c1461d0 2023-11-24T08:24:38.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:38.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:24:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 msgr2=0x7f460806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 0x7f460806faf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f4618007fd0 tx=0x7f4618009040 comp rx=0 tx=0).stop 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 msgr2=0x7f461c0ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c820 tx=0x7f460c00cc20 comp rx=0 tx=0).stop 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 shutdown_connections 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f460806d630 0x7f460806faf0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461c15eaf0 0x7f461c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c1461d0 0x7f461c0ce170 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 --2- 172.21.15.107:0/2913633736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f461c0b3890 0x7f461c0d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 >> 172.21.15.107:0/2913633736 conn(0x7f461c061340 msgr2=0x7f461c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 shutdown_connections 2023-11-24T08:24:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.588+0000 7f4622d0a640 1 -- 172.21.15.107:0/2913633736 wait complete. 2023-11-24T08:24:38.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.823+0000 7f9d43901640 1 Processor -- start 2023-11-24T08:24:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.824+0000 7f9d43901640 1 -- start start 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d43901640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d3c0cf450 0x7f9d3c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d43901640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d3c0cdfb0 0x7f9d3c0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d43901640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0cece0 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d43901640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c0cfe40 con 0x7f9d3c0cf450 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d43901640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c156010 con 0x7f9d3c0cdfb0 2023-11-24T08:24:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d40e9c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d3c0cf450 0x7f9d3c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.825+0000 7f9d40e9c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d3c0cf450 0x7f9d3c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48350/0 (socket says 172.21.15.107:48350) 2023-11-24T08:24:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.826+0000 7f9d40e9c640 1 -- 172.21.15.107:0/631960509 learned_addr learned my addr 172.21.15.107:0/631960509 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.826+0000 7f9d40e9c640 1 -- 172.21.15.107:0/631960509 >> v2:172.21.15.120:3300/0 conn(0x7f9d3c0cdfb0 msgr2=0x7f9d3c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.826+0000 7f9d40e9c640 1 --2- 172.21.15.107:0/631960509 >> v2:172.21.15.120:3300/0 conn(0x7f9d3c0cdfb0 0x7f9d3c0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.826+0000 7f9d40e9c640 1 -- 172.21.15.107:0/631960509 >> v1:172.21.15.120:6789/0 conn(0x7f9d3c15c7f0 legacy=0x7f9d3c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.826+0000 7f9d40e9c640 1 -- 172.21.15.107:0/631960509 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c06d040 con 0x7f9d3c0cf450 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.827+0000 7f9d40e9c640 1 --2- 172.21.15.107:0/631960509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0cf900 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0d00e0 tx=0x7f9d2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d42d300074c5a71 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.827+0000 7f9d33fff640 1 -- 172.21.15.107:0/631960509 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c031870 con 0x7f9d3c0cf450 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.827+0000 7f9d33fff640 1 -- 172.21.15.107:0/631960509 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c0319d0 con 0x7f9d3c0cf450 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.827+0000 7f9d33fff640 1 -- 172.21.15.107:0/631960509 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c031d60 con 0x7f9d3c0cf450 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 -- 172.21.15.107:0/631960509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0cf450 msgr2=0x7f9d3c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 --2- 172.21.15.107:0/631960509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0cf900 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0d00e0 tx=0x7f9d2c02b910 comp rx=0 tx=0).stop 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 -- 172.21.15.107:0/631960509 shutdown_connections 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 --2- 172.21.15.107:0/631960509 >> v2:172.21.15.120:3300/0 conn(0x7f9d3c0cdfb0 0x7f9d3c0ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 --2- 172.21.15.107:0/631960509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0cf900 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 -- 172.21.15.107:0/631960509 >> 172.21.15.107:0/631960509 conn(0x7f9d3c06a920 msgr2=0x7f9d3c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 -- 172.21.15.107:0/631960509 shutdown_connections 2023-11-24T08:24:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.828+0000 7f9d43901640 1 -- 172.21.15.107:0/631960509 wait complete. 2023-11-24T08:24:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.830+0000 7f9d43901640 1 Processor -- start 2023-11-24T08:24:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.830+0000 7f9d43901640 1 -- start start 2023-11-24T08:24:38.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 0x7f9d3c0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0acc70 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0acf50 con 0x7f9d3c0cf450 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d43901640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0ad230 con 0x7f9d3c0cdfb0 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48354/0 (socket says 172.21.15.107:48354) 2023-11-24T08:24:38.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 -- 172.21.15.107:0/296718125 learned_addr learned my addr 172.21.15.107:0/296718125 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:38.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d40e9c640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d4169d640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 0x7f9d3c0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 -- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 msgr2=0x7f9d3c0cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 0x7f9d3c0cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 -- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d3c0cf450 msgr2=0x7f9d3c0ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d41e9e640 1 -- 172.21.15.107:0/296718125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c0094d0 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.832+0000 7f9d4169d640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 0x7f9d3c0cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d41e9e640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b9e0 tx=0x7f9d3400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400caa0 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3400cc20 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c0ad510 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34014440 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.833+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c0ac960 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.834+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c0ada30 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.835+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f9d3401d3b0 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.836+0000 7f9d31ffb640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 0x7f9d2006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.836+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9d34088860 con 0x7f9d3c15c7f0 2023-11-24T08:24:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.837+0000 7f9d4169d640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 0x7f9d2006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:38.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.837+0000 7f9d4169d640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 0x7f9d2006faf0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800a810 tx=0x7f9d38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:38.841+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d3c0ada30 con 0x7f9d3c15c7f0 2023-11-24T08:24:39.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.031+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d3c0ada30 con 0x7f9d3c15c7f0 2023-11-24T08:24:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.033+0000 7f9d31ffb640 1 -- 172.21.15.107:0/296718125 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f9d3c0ada30 con 0x7f9d3c15c7f0 2023-11-24T08:24:39.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:39.035 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":236,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:35.220224+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":235,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.035+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 msgr2=0x7f9d2006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 0x7f9d2006faf0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800a810 tx=0x7f9d38008040 comp rx=0 tx=0).stop 2023-11-24T08:24:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 msgr2=0x7f9d3c149340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b9e0 tx=0x7f9d3400bea0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 shutdown_connections 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f9d2006d630 0x7f9d2006faf0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c15c7f0 0x7f9d3c149340 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d3c0cf450 0x7f9d3c0ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 --2- 172.21.15.107:0/296718125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3c0cdfb0 0x7f9d3c0cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 >> 172.21.15.107:0/296718125 conn(0x7f9d3c06a920 msgr2=0x7f9d3c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 shutdown_connections 2023-11-24T08:24:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.036+0000 7f9d43901640 1 -- 172.21.15.107:0/296718125 wait complete. 2023-11-24T08:24:39.056 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T08:24:39.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:24:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.261+0000 7f5a4a9ef640 1 Processor -- start 2023-11-24T08:24:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.261+0000 7f5a4a9ef640 1 -- start start 2023-11-24T08:24:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 0x7f5a4415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 0x7f5a440b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a440b1de0 0x7f5a440b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a440b27b0 con 0x7f5a440b1de0 2023-11-24T08:24:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4415bf80 con 0x7f5a4415c410 2023-11-24T08:24:39.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a9ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440b1560 con 0x7f5a44144b30 2023-11-24T08:24:39.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.262+0000 7f5a4a1ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a440b1de0 0x7f5a440b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a440b1de0 0x7f5a440b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48366/0 (socket says 172.21.15.107:48366) 2023-11-24T08:24:39.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 -- 172.21.15.107:0/1907773379 learned_addr learned my addr 172.21.15.107:0/1907773379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:39.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a499ed640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 0x7f5a4415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 -- 172.21.15.107:0/1907773379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 msgr2=0x7f5a4415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:39.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 0x7f5a4415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 -- 172.21.15.107:0/1907773379 >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 msgr2=0x7f5a440b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a491ec640 1 --2- 172.21.15.107:0/1907773379 >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 0x7f5a440b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 --2- 172.21.15.107:0/1907773379 >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 0x7f5a440b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a4a1ee640 1 -- 172.21.15.107:0/1907773379 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440b0fb0 con 0x7f5a440b1de0 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a499ed640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 0x7f5a4415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.263+0000 7f5a491ec640 1 --2- 172.21.15.107:0/1907773379 >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 0x7f5a440b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.264+0000 7f5a4a1ee640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440b2270 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f5a4406e820 tx=0x7f5a3c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aeba75e5ec520ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a489eb640 1 -- 172.21.15.107:0/1907773379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c02ca80 con 0x7f5a440b1de0 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a489eb640 1 -- 172.21.15.107:0/1907773379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c02cbe0 con 0x7f5a440b1de0 2023-11-24T08:24:39.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a489eb640 1 -- 172.21.15.107:0/1907773379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0343f0 con 0x7f5a440b1de0 2023-11-24T08:24:39.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/1907773379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 msgr2=0x7f5a440b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440b2270 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f5a4406e820 tx=0x7f5a3c02bee0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/1907773379 shutdown_connections 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440b2270 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/1907773379 >> v2:172.21.15.120:3300/0 conn(0x7f5a4415c410 0x7f5a440b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/1907773379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a44144b30 0x7f5a4415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/1907773379 >> 172.21.15.107:0/1907773379 conn(0x7f5a440cb830 msgr2=0x7f5a440cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/1907773379 shutdown_connections 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.265+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/1907773379 wait complete. 2023-11-24T08:24:39.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.267+0000 7f5a4a9ef640 1 Processor -- start 2023-11-24T08:24:39.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.267+0000 7f5a4a9ef640 1 -- start start 2023-11-24T08:24:39.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a44144b30 0x7f5a44143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 0x7f5a44143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44130d90 con 0x7f5a440b1de0 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a441304c0 con 0x7f5a44144b30 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.268+0000 7f5a4a9ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440b04b0 con 0x7f5a4415c410 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48374/0 (socket says 172.21.15.107:48374) 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 -- 172.21.15.107:0/2583834276 learned_addr learned my addr 172.21.15.107:0/2583834276 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a491ec640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a44144b30 0x7f5a44143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a4a1ee640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 0x7f5a44143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 -- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 msgr2=0x7f5a44143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 0x7f5a44143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 -- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a44144b30 msgr2=0x7f5a44143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a44144b30 0x7f5a44143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 -- 172.21.15.107:0/2583834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009b30 con 0x7f5a440b1de0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a4a1ee640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 0x7f5a44143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.269+0000 7f5a499ed640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c7f0 tx=0x7f5a4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.270+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000cea0 con 0x7f5a440b1de0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.270+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a40007a30 con 0x7f5a440b1de0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.270+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40007dc0 con 0x7f5a440b1de0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.270+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44140d80 con 0x7f5a440b1de0 2023-11-24T08:24:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.270+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441306d0 con 0x7f5a440b1de0 2023-11-24T08:24:39.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.271+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a440b3610 con 0x7f5a440b1de0 2023-11-24T08:24:39.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.272+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f5a400113f0 con 0x7f5a440b1de0 2023-11-24T08:24:39.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.273+0000 7f5a3a7fc640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 0x7f5a2806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.273+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5a40088a30 con 0x7f5a440b1de0 2023-11-24T08:24:39.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.273+0000 7f5a491ec640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 0x7f5a2806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.274+0000 7f5a491ec640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 0x7f5a2806faf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5a44133610 tx=0x7f5a34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.277+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a440b3610 con 0x7f5a440b1de0 2023-11-24T08:24:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.504+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a440b3610 con 0x7f5a440b1de0 2023-11-24T08:24:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.506+0000 7f5a3a7fc640 1 -- 172.21.15.107:0/2583834276 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f5a440b3610 con 0x7f5a440b1de0 2023-11-24T08:24:39.508 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:39.508 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:24:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-11-24T08:24:39.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 msgr2=0x7f5a2806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 0x7f5a2806faf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5a44133610 tx=0x7f5a34008040 comp rx=0 tx=0).stop 2023-11-24T08:24:39.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 msgr2=0x7f5a440aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c7f0 tx=0x7f5a4000ccb0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 shutdown_connections 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f5a2806d630 0x7f5a2806faf0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a4415c410 0x7f5a44143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a44144b30 0x7f5a44143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 --2- 172.21.15.107:0/2583834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440b1de0 0x7f5a440aff70 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.508+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 >> 172.21.15.107:0/2583834276 conn(0x7f5a440cb830 msgr2=0x7f5a440722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.509+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 shutdown_connections 2023-11-24T08:24:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.509+0000 7f5a4a9ef640 1 -- 172.21.15.107:0/2583834276 wait complete. 2023-11-24T08:24:39.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:24:39.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.737+0000 7f63f34ee640 1 Processor -- start 2023-11-24T08:24:39.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.737+0000 7f63f34ee640 1 -- start start 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f34ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63ec0cf450 0x7f63ec0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f34ee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec0cece0 con 0x7f63ec0cdfb0 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f34ee640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec06d040 con 0x7f63ec15c7f0 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f34ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec0cfe40 con 0x7f63ec0cf450 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f0a89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63ec0cf450 0x7f63ec0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f1a8b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f63ec0cdfb0 0x7f63ec0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42464/0 (socket says 172.21.15.107:42464) 2023-11-24T08:24:39.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.738+0000 7f63f0a89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63ec0cf450 0x7f63ec0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48380/0 (socket says 172.21.15.107:48380) 2023-11-24T08:24:39.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.739+0000 7f63f1a8b640 1 -- 172.21.15.107:0/1639096160 learned_addr learned my addr 172.21.15.107:0/1639096160 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:39.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.739+0000 7f63f0a89640 1 -- 172.21.15.107:0/1639096160 >> v1:172.21.15.120:6789/0 conn(0x7f63ec15c7f0 legacy=0x7f63ec06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:39.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.739+0000 7f63f0a89640 1 -- 172.21.15.107:0/1639096160 >> v1:172.21.15.107:6789/0 conn(0x7f63ec0cdfb0 legacy=0x7f63ec0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.739+0000 7f63f0a89640 1 -- 172.21.15.107:0/1639096160 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec06d320 con 0x7f63ec0cf450 2023-11-24T08:24:39.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.740+0000 7f63f0a89640 1 --2- 172.21.15.107:0/1639096160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec0cf900 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92fc7c7bb0e8715 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.740+0000 7f63dffff640 1 -- 172.21.15.107:0/1639096160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0030710 con 0x7f63ec0cf450 2023-11-24T08:24:39.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.740+0000 7f63dffff640 1 -- 172.21.15.107:0/1639096160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e002e050 con 0x7f63ec0cf450 2023-11-24T08:24:39.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.740+0000 7f63dffff640 1 -- 172.21.15.107:0/1639096160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0030a90 con 0x7f63ec0cf450 2023-11-24T08:24:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 -- 172.21.15.107:0/1639096160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 msgr2=0x7f63ec0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 --2- 172.21.15.107:0/1639096160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec0cf900 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e002b710 comp rx=0 tx=0).stop 2023-11-24T08:24:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 -- 172.21.15.107:0/1639096160 shutdown_connections 2023-11-24T08:24:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 --2- 172.21.15.107:0/1639096160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec0cf900 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 -- 172.21.15.107:0/1639096160 >> 172.21.15.107:0/1639096160 conn(0x7f63ec06a920 msgr2=0x7f63ec06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 -- 172.21.15.107:0/1639096160 shutdown_connections 2023-11-24T08:24:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.741+0000 7f63f34ee640 1 -- 172.21.15.107:0/1639096160 wait complete. 2023-11-24T08:24:39.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.743+0000 7f63f34ee640 1 Processor -- start 2023-11-24T08:24:39.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.743+0000 7f63f34ee640 1 -- start start 2023-11-24T08:24:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63ec0cdfb0 0x7f63ec06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63ec15c7f0 0x7f63ec147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec155f80 con 0x7f63ec0cf450 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec15b5e0 con 0x7f63ec0cdfb0 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.744+0000 7f63f34ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec156220 con 0x7f63ec15c7f0 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48388/0 (socket says 172.21.15.107:48388) 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 -- 172.21.15.107:0/3236187484 learned_addr learned my addr 172.21.15.107:0/3236187484 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f128a640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63ec0cdfb0 0x7f63ec06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f1a8b640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63ec15c7f0 0x7f63ec147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 -- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63ec15c7f0 msgr2=0x7f63ec147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63ec15c7f0 0x7f63ec147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 -- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63ec0cdfb0 msgr2=0x7f63ec06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63ec0cdfb0 0x7f63ec06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.745+0000 7f63f0a89640 1 -- 172.21.15.107:0/3236187484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0009400 con 0x7f63ec0cf450 2023-11-24T08:24:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63f0a89640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0032910 con 0x7f63ec0cf450 2023-11-24T08:24:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e0032a70 con 0x7f63ec0cf450 2023-11-24T08:24:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec156500 con 0x7f63ec0cf450 2023-11-24T08:24:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0032e20 con 0x7f63ec0cf450 2023-11-24T08:24:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.746+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec156ac0 con 0x7f63ec0cf450 2023-11-24T08:24:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.748+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ec0ae610 con 0x7f63ec0cf450 2023-11-24T08:24:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.749+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f63e0039040 con 0x7f63ec0cf450 2023-11-24T08:24:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.750+0000 7f63ddffb640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 0x7f63d006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.750+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f63e00aa7f0 con 0x7f63ec0cf450 2023-11-24T08:24:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.750+0000 7f63f128a640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 0x7f63d006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:39.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.751+0000 7f63f128a640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 0x7f63d006faf0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f63e8006fd0 tx=0x7f63e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.755+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63e0077300 con 0x7f63ec0cf450 2023-11-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.981+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63ec151ec0 con 0x7f63ec0cf450 2023-11-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.982+0000 7f63ddffb640 1 -- 172.21.15.107:0/3236187484 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f63ec151ec0 con 0x7f63ec0cf450 2023-11-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:23:51.172573+0000","modified":"2023-11-24T08:23:54.236400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12747":{"gid":12747,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/438366562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":438366562},{"type":"v1","addr":"172.21.15.107:6837","nonce":438366562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-11-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.984+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 msgr2=0x7f63d006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.984+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 0x7f63d006faf0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f63e8006fd0 tx=0x7f63e8008040 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.984+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 msgr2=0x7f63ec147000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.984+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e00046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 shutdown_connections 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f63d006d630 0x7f63d006faf0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63ec15c7f0 0x7f63ec147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63ec0cf450 0x7f63ec147000 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 --2- 172.21.15.107:0/3236187484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63ec0cdfb0 0x7f63ec06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 >> 172.21.15.107:0/3236187484 conn(0x7f63ec06a920 msgr2=0x7f63ec154b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 shutdown_connections 2023-11-24T08:24:39.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:39.985+0000 7f63f34ee640 1 -- 172.21.15.107:0/3236187484 wait complete. 2023-11-24T08:24:40.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.208+0000 7efdbc978640 1 Processor -- start 2023-11-24T08:24:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.208+0000 7efdbc978640 1 -- start start 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdbc978640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 0x7efdb806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdbc978640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdb80cf450 0x7efdb80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdbc978640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb80cece0 con 0x7efdb80cdfb0 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdbc978640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80cfe40 con 0x7efdb80cf450 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdbc978640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80ceaf0 con 0x7efdb815c7f0 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdb5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdb80cf450 0x7efdb80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdb5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdb80cf450 0x7efdb80cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48396/0 (socket says 172.21.15.107:48396) 2023-11-24T08:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.210+0000 7efdb5d9b640 1 -- 172.21.15.107:0/1078647673 learned_addr learned my addr 172.21.15.107:0/1078647673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:40.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb659c640 1 --2- 172.21.15.107:0/1078647673 >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 0x7efdb806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb5d9b640 1 -- 172.21.15.107:0/1078647673 >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 msgr2=0x7efdb806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:40.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb5d9b640 1 --2- 172.21.15.107:0/1078647673 >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 0x7efdb806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb5d9b640 1 -- 172.21.15.107:0/1078647673 >> v1:172.21.15.120:6789/0 conn(0x7efdb80cdfb0 legacy=0x7efdb80ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:40.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb5d9b640 1 -- 172.21.15.107:0/1078647673 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb806d040 con 0x7efdb80cf450 2023-11-24T08:24:40.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.211+0000 7efdb659c640 1 --2- 172.21.15.107:0/1078647673 >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 0x7efdb806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:40.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.212+0000 7efdb5d9b640 1 --2- 172.21.15.107:0/1078647673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb80cf900 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7efdb80d0130 tx=0x7efdac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b50018fb5ee35d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.212+0000 7efdb559a640 1 -- 172.21.15.107:0/1078647673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac031870 con 0x7efdb80cf450 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.212+0000 7efdb559a640 1 -- 172.21.15.107:0/1078647673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac0319d0 con 0x7efdb80cf450 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.212+0000 7efdb559a640 1 -- 172.21.15.107:0/1078647673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac031d60 con 0x7efdb80cf450 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 -- 172.21.15.107:0/1078647673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 msgr2=0x7efdb80cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 --2- 172.21.15.107:0/1078647673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb80cf900 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7efdb80d0130 tx=0x7efdac02b910 comp rx=0 tx=0).stop 2023-11-24T08:24:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 -- 172.21.15.107:0/1078647673 shutdown_connections 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 --2- 172.21.15.107:0/1078647673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb80cf900 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 --2- 172.21.15.107:0/1078647673 >> v2:172.21.15.120:3300/0 conn(0x7efdb815c7f0 0x7efdb806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 -- 172.21.15.107:0/1078647673 >> 172.21.15.107:0/1078647673 conn(0x7efdb806a920 msgr2=0x7efdb806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 -- 172.21.15.107:0/1078647673 shutdown_connections 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.213+0000 7efdbc978640 1 -- 172.21.15.107:0/1078647673 wait complete. 2023-11-24T08:24:40.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.215+0000 7efdbc978640 1 Processor -- start 2023-11-24T08:24:40.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.215+0000 7efdbc978640 1 -- start start 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80cdfb0 0x7efdb814b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb815c7f0 0x7efdb815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f37a0 con 0x7efdb80cf450 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80b3c50 con 0x7efdb80cdfb0 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdbc978640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80b33b0 con 0x7efdb815c7f0 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdb5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdb5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48400/0 (socket says 172.21.15.107:48400) 2023-11-24T08:24:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.216+0000 7efdb5d9b640 1 -- 172.21.15.107:0/4157602695 learned_addr learned my addr 172.21.15.107:0/4157602695 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb6d9d640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb815c7f0 0x7efdb815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb659c640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80cdfb0 0x7efdb814b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 -- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb815c7f0 msgr2=0x7efdb815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb815c7f0 0x7efdb815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 -- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80cdfb0 msgr2=0x7efdb814b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80cdfb0 0x7efdb814b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 -- 172.21.15.107:0/4157602695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0094d0 con 0x7efdb80cf450 2023-11-24T08:24:40.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdb5d9b640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7efdac000c00 tx=0x7efdac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac031840 con 0x7efdb80cf450 2023-11-24T08:24:40.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac0443f0 con 0x7efdb80cf450 2023-11-24T08:24:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb80b3690 con 0x7efdb80cf450 2023-11-24T08:24:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac0447a0 con 0x7efdb80cf450 2023-11-24T08:24:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.217+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb80ac160 con 0x7efdb80cf450 2023-11-24T08:24:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.219+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb80abfc0 con 0x7efdb80cf450 2023-11-24T08:24:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.220+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7efdac04d450 con 0x7efdb80cf450 2023-11-24T08:24:40.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.221+0000 7efda6ffd640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 0x7efd9806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.221+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efdac02d040 con 0x7efdb80cf450 2023-11-24T08:24:40.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.221+0000 7efdb659c640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 0x7efd9806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.222+0000 7efdb659c640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 0x7efd9806fb20 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efda800b840 tx=0x7efda8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.228+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdb80abfc0 con 0x7efdb80cf450 2023-11-24T08:24:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.411+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdb80abfc0 con 0x7efdb80cf450 2023-11-24T08:24:40.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.413+0000 7efda6ffd640 1 -- 172.21.15.107:0/4157602695 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+16582 (secure 0 0 0) 0x7efdb80abfc0 con 0x7efdb80cf450 2023-11-24T08:24:40.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:40.415 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":236,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:35.220224+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:23:48.986422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2023-11-24T08:23:50.200838+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":235,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":34,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.415+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 msgr2=0x7efd9806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.415+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 0x7efd9806fb20 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efda800b840 tx=0x7efda8008040 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.415+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 msgr2=0x7efdb815e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.415+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7efdac000c00 tx=0x7efdac0046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 shutdown_connections 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7efd9806d660 0x7efd9806fb20 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efdb815c7f0 0x7efdb815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb80cf450 0x7efdb815e350 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 --2- 172.21.15.107:0/4157602695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb80cdfb0 0x7efdb814b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 >> 172.21.15.107:0/4157602695 conn(0x7efdb806a920 msgr2=0x7efdb80ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:40.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 shutdown_connections 2023-11-24T08:24:40.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.416+0000 7efdbc978640 1 -- 172.21.15.107:0/4157602695 wait complete. 2023-11-24T08:24:40.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T08:24:40.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.643+0000 7fab4e20e640 1 Processor -- start 2023-11-24T08:24:40.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.644+0000 7fab4e20e640 1 -- start start 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab4e20e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab4806de10 0x7fab4815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab4e20e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 0x7fab480705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab4e20e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815ec90 con 0x7fab48072160 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab4e20e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480cd530 con 0x7fab4806de10 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab4e20e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480c4ed0 con 0x7fab480c49f0 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab46ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab4806de10 0x7fab4815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab477fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fab48072160 0x7fab4806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42478/0 (socket says 172.21.15.107:42478) 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab46ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab4806de10 0x7fab4815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48416/0 (socket says 172.21.15.107:48416) 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.645+0000 7fab477fe640 1 -- 172.21.15.107:0/3739149393 learned_addr learned my addr 172.21.15.107:0/3739149393 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab47fff640 1 --2- 172.21.15.107:0/3739149393 >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 0x7fab480705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab46ffd640 1 -- 172.21.15.107:0/3739149393 >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 msgr2=0x7fab480705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab46ffd640 1 --2- 172.21.15.107:0/3739149393 >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 0x7fab480705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab46ffd640 1 -- 172.21.15.107:0/3739149393 >> v1:172.21.15.107:6789/0 conn(0x7fab48072160 legacy=0x7fab4806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab46ffd640 1 -- 172.21.15.107:0/3739149393 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480c51b0 con 0x7fab4806de10 2023-11-24T08:24:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.646+0000 7fab47fff640 1 --2- 172.21.15.107:0/3739149393 >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 0x7fab480705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.647+0000 7fab46ffd640 1 --2- 172.21.15.107:0/3739149393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4806de10 0x7fab4815e750 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7fab48072a30 tx=0x7fab3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f51cbe4c8fc7728 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.647+0000 7fab467fc640 1 -- 172.21.15.107:0/3739149393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30031830 con 0x7fab4806de10 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab467fc640 1 -- 172.21.15.107:0/3739149393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30031990 con 0x7fab4806de10 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab467fc640 1 -- 172.21.15.107:0/3739149393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30031d20 con 0x7fab4806de10 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 -- 172.21.15.107:0/3739149393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4806de10 msgr2=0x7fab4815e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3739149393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4806de10 0x7fab4815e750 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7fab48072a30 tx=0x7fab3002b8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 -- 172.21.15.107:0/3739149393 shutdown_connections 2023-11-24T08:24:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3739149393 >> v2:172.21.15.120:3300/0 conn(0x7fab480c49f0 0x7fab480705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3739149393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4806de10 0x7fab4815e750 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 -- 172.21.15.107:0/3739149393 >> 172.21.15.107:0/3739149393 conn(0x7fab480b0c50 msgr2=0x7fab480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 -- 172.21.15.107:0/3739149393 shutdown_connections 2023-11-24T08:24:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.648+0000 7fab4e20e640 1 -- 172.21.15.107:0/3739149393 wait complete. 2023-11-24T08:24:40.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.650+0000 7fab4e20e640 1 Processor -- start 2023-11-24T08:24:40.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.650+0000 7fab4e20e640 1 -- start start 2023-11-24T08:24:40.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 0x7fab480b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab480c49f0 0x7fab4806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480b1d10 con 0x7fab48072160 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480b1f60 con 0x7fab4806de10 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab4e20e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480b2240 con 0x7fab480c49f0 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab46ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab46ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48420/0 (socket says 172.21.15.107:48420) 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.651+0000 7fab46ffd640 1 -- 172.21.15.107:0/3219181258 learned_addr learned my addr 172.21.15.107:0/3219181258 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 -- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab480c49f0 msgr2=0x7fab4806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab480c49f0 0x7fab4806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 -- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 msgr2=0x7fab480b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab477fe640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 0x7fab480b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 0x7fab480b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 -- 172.21.15.107:0/3219181258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30009400 con 0x7fab48072160 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab477fe640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 0x7fab480b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab46ffd640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7fab30009890 tx=0x7fab300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3003ba70 con 0x7fab48072160 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3003bbf0 con 0x7fab48072160 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30045400 con 0x7fab48072160 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.652+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab480b2520 con 0x7fab48072160 2023-11-24T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.653+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab480b3260 con 0x7fab48072160 2023-11-24T08:24:40.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.654+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab480b2a40 con 0x7fab48072160 2023-11-24T08:24:40.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.654+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fab3004e440 con 0x7fab48072160 2023-11-24T08:24:40.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.656+0000 7fab3ffff640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 0x7fab2806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.656+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fab300aaca0 con 0x7fab48072160 2023-11-24T08:24:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.656+0000 7fab477fe640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 0x7fab2806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.657+0000 7fab477fe640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 0x7fab2806faf0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fab38007fd0 tx=0x7fab38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:40.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.660+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab480b2a40 con 0x7fab48072160 2023-11-24T08:24:40.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.889+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fab480b2a40 con 0x7fab48072160 2023-11-24T08:24:40.891 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-11-24T08:24:40.890+0000 7fcb3975d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T08:24:40.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.892+0000 7fab3ffff640 1 -- 172.21.15.107:0/3219181258 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v149) v1 ==== 161+0+0 (secure 0 0 0) 0x7fab480b2a40 con 0x7fab48072160 2023-11-24T08:24:40.895 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T08:24:40.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 msgr2=0x7fab2806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 0x7fab2806faf0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fab38007fd0 tx=0x7fab38009040 comp rx=0 tx=0).stop 2023-11-24T08:24:40.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 msgr2=0x7fab4806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:40.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7fab30009890 tx=0x7fab300046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 shutdown_connections 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fab2806d630 0x7fab2806faf0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab480c49f0 0x7fab4806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab48072160 0x7fab4806c5f0 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 --2- 172.21.15.107:0/3219181258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab4806de10 0x7fab480b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 >> 172.21.15.107:0/3219181258 conn(0x7fab480b0c50 msgr2=0x7fab480abf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 shutdown_connections 2023-11-24T08:24:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:40.896+0000 7fab4e20e640 1 -- 172.21.15.107:0/3219181258 wait complete. 2023-11-24T08:24:40.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T08:24:40.976 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:24:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.111+0000 7f481c8ed640 1 Processor -- start 2023-11-24T08:24:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.111+0000 7f481c8ed640 1 -- start start 2023-11-24T08:24:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.112+0000 7f481c8ed640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4818154980 0x7f48180ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.112+0000 7f481c8ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4818072710 0x7f481815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.112+0000 7f481c8ed640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180709c0 con 0x7f48180b3090 2023-11-24T08:24:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.112+0000 7f481c8ed640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48180b3510 con 0x7f4818072710 2023-11-24T08:24:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.112+0000 7f481c8ed640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180b0df0 con 0x7f4818154980 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4818072710 0x7f481815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4818072710 0x7f481815a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48444/0 (socket says 172.21.15.107:48444) 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 -- 172.21.15.107:0/2542909696 learned_addr learned my addr 172.21.15.107:0/2542909696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f4816d9d640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4818154980 0x7f48180ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 -- 172.21.15.107:0/2542909696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4818154980 msgr2=0x7f48180ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4818154980 0x7f48180ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 -- 172.21.15.107:0/2542909696 >> v1:172.21.15.120:6789/0 conn(0x7f48180b3090 legacy=0x7f4818072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.113+0000 7f481759e640 1 -- 172.21.15.107:0/2542909696 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481815af40 con 0x7f4818072710 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.114+0000 7f481759e640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481815a8a0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f4810009e50 tx=0x7f481002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d66b32a4d6052a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.114+0000 7f4815d9b640 1 -- 172.21.15.107:0/2542909696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481002ec80 con 0x7f4818072710 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f4815d9b640 1 -- 172.21.15.107:0/2542909696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481002ede0 con 0x7f4818072710 2023-11-24T08:24:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f4815d9b640 1 -- 172.21.15.107:0/2542909696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810036610 con 0x7f4818072710 2023-11-24T08:24:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 -- 172.21.15.107:0/2542909696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 msgr2=0x7f481815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481815a8a0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f4810009e50 tx=0x7f481002e0e0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 -- 172.21.15.107:0/2542909696 shutdown_connections 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481815a8a0 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 --2- 172.21.15.107:0/2542909696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4818154980 0x7f48180ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 -- 172.21.15.107:0/2542909696 >> 172.21.15.107:0/2542909696 conn(0x7f481806a500 msgr2=0x7f481806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 -- 172.21.15.107:0/2542909696 shutdown_connections 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.115+0000 7f481c8ed640 1 -- 172.21.15.107:0/2542909696 wait complete. 2023-11-24T08:24:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.117+0000 7f481c8ed640 1 Processor -- start 2023-11-24T08:24:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.117+0000 7f481c8ed640 1 -- start start 2023-11-24T08:24:41.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 0x7f4818156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4818154980 0x7f481815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481814c5c0 con 0x7f4818072710 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180cf680 con 0x7f4818154980 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f481c8ed640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4818157530 con 0x7f48180b3090 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f4816d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f4816d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48460/0 (socket says 172.21.15.107:48460) 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.118+0000 7f4816d9d640 1 -- 172.21.15.107:0/3490352560 learned_addr learned my addr 172.21.15.107:0/3490352560 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 -- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 msgr2=0x7f4818156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f481659c640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 0x7f4818156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f481759e640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4818154980 0x7f481815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 0x7f4818156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 -- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4818154980 msgr2=0x7f481815ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4818154980 0x7f481815ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 -- 172.21.15.107:0/3490352560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810009b30 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f481659c640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 0x7f4818156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f4816d9d640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f480800d800 tx=0x7f480800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480800b6c0 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480800b820 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480800bbb0 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.119+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48181577d0 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.120+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48180ae7d0 con 0x7f4818072710 2023-11-24T08:24:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.121+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4818067990 con 0x7f4818072710 2023-11-24T08:24:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.122+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f480800bd10 con 0x7f4818072710 2023-11-24T08:24:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.122+0000 7f47ff7fe640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 0x7f480406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.123+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5882+0+0 (secure 0 0 0) 0x7f4808088ea0 con 0x7f4818072710 2023-11-24T08:24:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.123+0000 7f481659c640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 0x7f480406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.123+0000 7f481659c640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 0x7f480406faf0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f4818150290 tx=0x7f480c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:41.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.127+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4808055a50 con 0x7f4818072710 2023-11-24T08:24:41.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.342+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f48180b29b0 con 0x7f4818072710 2023-11-24T08:24:41.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.896+0000 7f47ff7fe640 1 -- 172.21.15.107:0/3490352560 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v150) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4808059070 con 0x7f4818072710 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 msgr2=0x7f480406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 0x7f480406faf0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f4818150290 tx=0x7f480c008040 comp rx=0 tx=0).stop 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 msgr2=0x7f481814c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f480800d800 tx=0x7f480800dcc0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 shutdown_connections 2023-11-24T08:24:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f480406d630 0x7f480406faf0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4818154980 0x7f481815ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48180b3090 0x7f4818156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 --2- 172.21.15.107:0/3490352560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4818072710 0x7f481814c080 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 >> 172.21.15.107:0/3490352560 conn(0x7f481806a500 msgr2=0x7f4818070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 shutdown_connections 2023-11-24T08:24:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:41.900+0000 7f481c8ed640 1 -- 172.21.15.107:0/3490352560 wait complete. 2023-11-24T08:24:41.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T08:24:42.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.116+0000 7f2e09ecf640 1 Processor -- start 2023-11-24T08:24:42.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.117+0000 7f2e09ecf640 1 -- start start 2023-11-24T08:24:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.118+0000 7f2e09ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.118+0000 7f2e09ecf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e040d0ad0 0x7f2e040d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.118+0000 7f2e09ecf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e040ce2c0 con 0x7f2e0415d740 2023-11-24T08:24:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.118+0000 7f2e09ecf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e040cec60 con 0x7f2e040d0ad0 2023-11-24T08:24:42.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.118+0000 7f2e09ecf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040d1710 con 0x7f2e040cf080 2023-11-24T08:24:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.119+0000 7f2e02ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.119+0000 7f2e03fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e040d0ad0 0x7f2e040d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.119+0000 7f2e02ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:48556/0 (socket says 172.21.15.107:48556) 2023-11-24T08:24:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.119+0000 7f2e037fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2e0415d740 0x7f2e040d1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:37118/0 (socket says 172.21.15.107:37118) 2023-11-24T08:24:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.119+0000 7f2e02ffd640 1 -- 172.21.15.107:0/195786108 learned_addr learned my addr 172.21.15.107:0/195786108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:42.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.120+0000 7f2e02ffd640 1 -- 172.21.15.107:0/195786108 >> v2:172.21.15.120:3300/0 conn(0x7f2e040d0ad0 msgr2=0x7f2e040d1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:42.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.120+0000 7f2e02ffd640 1 --2- 172.21.15.107:0/195786108 >> v2:172.21.15.120:3300/0 conn(0x7f2e040d0ad0 0x7f2e040d1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:42.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.120+0000 7f2e02ffd640 1 -- 172.21.15.107:0/195786108 >> v1:172.21.15.120:6789/0 conn(0x7f2e0415d740 legacy=0x7f2e040d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:42.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.120+0000 7f2e02ffd640 1 -- 172.21.15.107:0/195786108 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e040ce5a0 con 0x7f2e040cf080 2023-11-24T08:24:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.121+0000 7f2e02ffd640 1 --2- 172.21.15.107:0/195786108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009870 tx=0x7f2dec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4c24747aabdd7b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:42.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e027fc640 1 -- 172.21.15.107:0/195786108 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec030840 con 0x7f2e040cf080 2023-11-24T08:24:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e027fc640 1 -- 172.21.15.107:0/195786108 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec0309a0 con 0x7f2e040cf080 2023-11-24T08:24:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e027fc640 1 -- 172.21.15.107:0/195786108 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec030d30 con 0x7f2e040cf080 2023-11-24T08:24:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e09ecf640 1 -- 172.21.15.107:0/195786108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 msgr2=0x7f2e040d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/195786108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009870 tx=0x7f2dec02b980 comp rx=0 tx=0).stop 2023-11-24T08:24:42.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e09ecf640 1 -- 172.21.15.107:0/195786108 shutdown_connections 2023-11-24T08:24:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.122+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/195786108 >> v2:172.21.15.120:3300/0 conn(0x7f2e040d0ad0 0x7f2e040d1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.123+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/195786108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040cf080 0x7f2e040d19f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.123+0000 7f2e09ecf640 1 -- 172.21.15.107:0/195786108 >> 172.21.15.107:0/195786108 conn(0x7f2e0408af60 msgr2=0x7f2e040d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.123+0000 7f2e09ecf640 1 -- 172.21.15.107:0/195786108 shutdown_connections 2023-11-24T08:24:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.123+0000 7f2e09ecf640 1 -- 172.21.15.107:0/195786108 wait complete. 2023-11-24T08:24:42.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.125+0000 7f2e09ecf640 1 Processor -- start 2023-11-24T08:24:42.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.125+0000 7f2e09ecf640 1 -- start start 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.126+0000 7f2e09ecf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e040cf080 0x7f2e040b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e09ecf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040d0ad0 0x7f2e040b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e09ecf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e09ecf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e04152ca0 con 0x7f2e0415d740 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e09ecf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040b84c0 con 0x7f2e040cf080 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e09ecf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040b87a0 con 0x7f2e040d0ad0 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48472/0 (socket says 172.21.15.107:48472) 2023-11-24T08:24:42.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 -- 172.21.15.107:0/893790591 learned_addr learned my addr 172.21.15.107:0/893790591 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:42.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e037fe640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e040cf080 0x7f2e040b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:42.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 -- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040d0ad0 msgr2=0x7f2e040b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040d0ad0 0x7f2e040b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 -- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e040cf080 msgr2=0x7f2e040b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.127+0000 7f2e03fff640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e040cf080 0x7f2e040b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2e03fff640 1 -- 172.21.15.107:0/893790591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0094d0 con 0x7f2e0415d740 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2e03fff640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e930 tx=0x7f2dfc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc009690 con 0x7f2e0415d740 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc00c980 con 0x7f2e0415d740 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc00cd30 con 0x7f2e0415d740 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e040b8a80 con 0x7f2e0415d740 2023-11-24T08:24:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.128+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e04153810 con 0x7f2e0415d740 2023-11-24T08:24:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.130+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e04153a00 con 0x7f2e0415d740 2023-11-24T08:24:42.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.131+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f2dfc013470 con 0x7f2e0415d740 2023-11-24T08:24:42.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.131+0000 7f2dfbfff640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 0x7f2de406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:42.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.132+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 5882+0+0 (secure 0 0 0) 0x7f2dfc088b50 con 0x7f2e0415d740 2023-11-24T08:24:42.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.132+0000 7f2e037fe640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 0x7f2de406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:42.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.132+0000 7f2e037fe640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 0x7f2de406faf0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f2df40012e0 tx=0x7f2df4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:42.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.136+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dfc0557c0 con 0x7f2e0415d740 2023-11-24T08:24:42.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:42.328+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e04067470 con 0x7f2e0415d740 2023-11-24T08:24:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.331+0000 7f2dfbfff640 1 -- 172.21.15.107:0/893790591 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v239) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2dfc0210d0 con 0x7f2e0415d740 2023-11-24T08:24:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2023-11-24T08:24:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.335+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 msgr2=0x7f2de406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.335+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 0x7f2de406faf0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f2df40012e0 tx=0x7f2df4009040 comp rx=0 tx=0).stop 2023-11-24T08:24:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.335+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 msgr2=0x7f2e04148f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.335+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e930 tx=0x7f2dfc00edf0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 shutdown_connections 2023-11-24T08:24:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2de406d630 0x7f2de406faf0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e0415d740 0x7f2e04148f40 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e040d0ad0 0x7f2e040b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 --2- 172.21.15.107:0/893790591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e040cf080 0x7f2e040b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.336+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 >> 172.21.15.107:0/893790591 conn(0x7f2e0408af60 msgr2=0x7f2e040d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.338+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 shutdown_connections 2023-11-24T08:24:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.338+0000 7f2e09ecf640 1 -- 172.21.15.107:0/893790591 wait complete. 2023-11-24T08:24:43.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T08:24:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.565+0000 7f6ebf6bb640 1 Processor -- start 2023-11-24T08:24:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.565+0000 7f6ebf6bb640 1 -- start start 2023-11-24T08:24:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.566+0000 7f6ebf6bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 0x7f6eb80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.566+0000 7f6ebf6bb640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6eb80cbf80 0x7f6eb806c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.566+0000 7f6ebf6bb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb80b3a40 con 0x7f6eb8159ce0 2023-11-24T08:24:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.566+0000 7f6ebf6bb640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb80cc410 con 0x7f6eb80cbf80 2023-11-24T08:24:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.566+0000 7f6ebf6bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb80cdfa0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.567+0000 7f6ebcc56640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 0x7f6eb80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.567+0000 7f6ebdc58640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6eb80cbf80 0x7f6eb806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.567+0000 7f6ebd457640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6eb8159ce0 0x7f6eb806de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:37124/0 (socket says 172.21.15.107:37124) 2023-11-24T08:24:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.567+0000 7f6ebd457640 1 -- 172.21.15.107:0/1013777578 learned_addr learned my addr 172.21.15.107:0/1013777578 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.568+0000 7f6ebdc58640 1 -- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 msgr2=0x7f6eb80cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.568+0000 7f6ebdc58640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 0x7f6eb80cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.568+0000 7f6ebdc58640 1 -- 172.21.15.107:0/1013777578 >> v1:172.21.15.120:6789/0 conn(0x7f6eb8159ce0 legacy=0x7f6eb806de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.568+0000 7f6ebdc58640 1 -- 172.21.15.107:0/1013777578 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb80cd9e0 con 0x7f6eb80cbf80 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.568+0000 7f6ebcc56640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 0x7f6eb80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:24:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.569+0000 7f6ebdc58640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb806c560 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009d90 tx=0x7f6eb002e0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c14a7016f6ecbdc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.569+0000 7f6eaffff640 1 -- 172.21.15.107:0/1013777578 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb002ee60 con 0x7f6eb80cbf80 2023-11-24T08:24:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.569+0000 7f6eaffff640 1 -- 172.21.15.107:0/1013777578 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb0036440 con 0x7f6eb80cbf80 2023-11-24T08:24:43.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.569+0000 7f6eaffff640 1 -- 172.21.15.107:0/1013777578 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00367d0 con 0x7f6eb80cbf80 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 msgr2=0x7f6eb806c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb806c560 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009d90 tx=0x7f6eb002e0b0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/1013777578 shutdown_connections 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb806c560 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/1013777578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb80b21d0 0x7f6eb80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/1013777578 >> 172.21.15.107:0/1013777578 conn(0x7f6eb806a920 msgr2=0x7f6eb806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:43.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/1013777578 shutdown_connections 2023-11-24T08:24:43.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.570+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/1013777578 wait complete. 2023-11-24T08:24:43.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.572+0000 7f6ebf6bb640 1 Processor -- start 2023-11-24T08:24:43.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.572+0000 7f6ebf6bb640 1 -- start start 2023-11-24T08:24:43.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb8156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb8159ce0 0x7f6eb8099260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb809b1e0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb8156a80 con 0x7f6eb80cbf80 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebf6bb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb809b430 con 0x7f6eb8159ce0 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebd457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebd457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48480/0 (socket says 172.21.15.107:48480) 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebd457640 1 -- 172.21.15.107:0/3609412817 learned_addr learned my addr 172.21.15.107:0/3609412817 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.573+0000 7f6ebcc56640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb8156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebdc58640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb8159ce0 0x7f6eb8099260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 -- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb8159ce0 msgr2=0x7f6eb8099260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb8159ce0 0x7f6eb8099260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 -- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 msgr2=0x7f6eb8156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:43.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb8156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:43.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 -- 172.21.15.107:0/3609412817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea800a5a0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebd457640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c3f0 tx=0x7f6eb400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:43.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4005ce0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4005e40 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb0009a70 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40072f0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.574+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb809bcd0 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.575+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb809b820 con 0x7f6eb80b21d0 2023-11-24T08:24:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.576+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f6eb4007450 con 0x7f6eb80b21d0 2023-11-24T08:24:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.578+0000 7f6eadffb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 0x7f6e9c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.578+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5444+0+0 (secure 0 0 0) 0x7f6eb4087e80 con 0x7f6eb80b21d0 2023-11-24T08:24:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.578+0000 7f6ebcc56640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 0x7f6e9c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.579+0000 7f6ebcc56640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 0x7f6e9c06faf0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8155900 tx=0x7f6ea8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:43.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.582+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6eb4054a30 con 0x7f6eb80b21d0 2023-11-24T08:24:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:43.773+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6eb8067470 con 0x7f6eb80b21d0 2023-11-24T08:24:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.341+0000 7f6eadffb640 1 -- 172.21.15.107:0/3609412817 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v240) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6eb4017070 con 0x7f6eb80b21d0 2023-11-24T08:24:44.343 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2023-11-24T08:24:44.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.345+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 msgr2=0x7f6e9c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.345+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 0x7f6e9c06faf0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8155900 tx=0x7f6ea8008040 comp rx=0 tx=0).stop 2023-11-24T08:24:44.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.345+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 msgr2=0x7f6eb809aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.345+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c3f0 tx=0x7f6eb400c8b0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 shutdown_connections 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f6e9c06d630 0x7f6e9c06faf0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6eb8159ce0 0x7f6eb8099260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6eb80cbf80 0x7f6eb8156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 --2- 172.21.15.107:0/3609412817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eb80b21d0 0x7f6eb809aca0 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.347+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 >> 172.21.15.107:0/3609412817 conn(0x7f6eb806a920 msgr2=0x7f6eb8153390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.348+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 shutdown_connections 2023-11-24T08:24:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.348+0000 7f6ebf6bb640 1 -- 172.21.15.107:0/3609412817 wait complete. 2023-11-24T08:24:44.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T08:24:44.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.586+0000 7fcad76c9640 1 Processor -- start 2023-11-24T08:24:44.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.586+0000 7fcad76c9640 1 -- start start 2023-11-24T08:24:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.588+0000 7fcad76c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad015e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.588+0000 7fcad76c9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcad00c49f0 0x7fcad00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.588+0000 7fcad76c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad015ec90 con 0x7fcad0072160 2023-11-24T08:24:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.588+0000 7fcad76c9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad00cd530 con 0x7fcad00c49f0 2023-11-24T08:24:44.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.588+0000 7fcad76c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00c4ed0 con 0x7fcad006de10 2023-11-24T08:24:44.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.589+0000 7fcad5c66640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcad00c49f0 0x7fcad00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.589+0000 7fcad4c64640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.589+0000 7fcad5c66640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcad00c49f0 0x7fcad00705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:44504/0 (socket says 172.21.15.107:44504) 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.589+0000 7fcad5c66640 1 -- 172.21.15.107:0/951439916 learned_addr learned my addr 172.21.15.107:0/951439916 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.589+0000 7fcad5465640 1 --1- 172.21.15.107:0/951439916 >> v1:172.21.15.120:6789/0 conn(0x7fcad0072160 0x7fcad006d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:37134/0 (socket says 172.21.15.107:37134) 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.590+0000 7fcad5c66640 1 -- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 msgr2=0x7fcad015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.590+0000 7fcad5c66640 1 --2- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.590+0000 7fcad5c66640 1 -- 172.21.15.107:0/951439916 >> v1:172.21.15.120:6789/0 conn(0x7fcad0072160 legacy=0x7fcad006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.590+0000 7fcad5c66640 1 -- 172.21.15.107:0/951439916 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00c51b0 con 0x7fcad00c49f0 2023-11-24T08:24:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.591+0000 7fcad5c66640 1 --2- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad00705c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac802e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23405b961f78e2d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.591+0000 7fcac7fff640 1 -- 172.21.15.107:0/951439916 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac800b0d0 con 0x7fcad00c49f0 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcac7fff640 1 -- 172.21.15.107:0/951439916 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac800b230 con 0x7fcad00c49f0 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcac7fff640 1 -- 172.21.15.107:0/951439916 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8009600 con 0x7fcad00c49f0 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 -- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 msgr2=0x7fcad00705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 --2- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad00705c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac802e160 comp rx=0 tx=0).stop 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 -- 172.21.15.107:0/951439916 shutdown_connections 2023-11-24T08:24:44.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 --2- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad00705c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 --2- 172.21.15.107:0/951439916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 -- 172.21.15.107:0/951439916 >> 172.21.15.107:0/951439916 conn(0x7fcad00b0c50 msgr2=0x7fcad00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 -- 172.21.15.107:0/951439916 shutdown_connections 2023-11-24T08:24:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.592+0000 7fcad76c9640 1 -- 172.21.15.107:0/951439916 wait complete. 2023-11-24T08:24:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.594+0000 7fcad76c9640 1 Processor -- start 2023-11-24T08:24:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.594+0000 7fcad76c9640 1 -- start start 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad015bc20 con 0x7fcad0072160 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad015bec0 con 0x7fcad00c49f0 2023-11-24T08:24:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad76c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00b4a10 con 0x7fcad006de10 2023-11-24T08:24:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad4c64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad4c64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48488/0 (socket says 172.21.15.107:48488) 2023-11-24T08:24:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.595+0000 7fcad4c64640 1 -- 172.21.15.107:0/3746850910 learned_addr learned my addr 172.21.15.107:0/3746850910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad5c66640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad5465640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 -- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 msgr2=0x7fcad006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad006c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 -- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 msgr2=0x7fcad015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 -- 172.21.15.107:0/3746850910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac800bd40 con 0x7fcad0072160 2023-11-24T08:24:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad4c64640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7fcac000e9b0 tx=0x7fcac000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0009670 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac000c9a0 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac000cd50 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.596+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad015b050 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.597+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad015b610 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.598+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad015b800 con 0x7fcad0072160 2023-11-24T08:24:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.599+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fcac0013470 con 0x7fcad0072160 2023-11-24T08:24:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.600+0000 7fcac5ffb640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 0x7fcab406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.600+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcac0088c30 con 0x7fcad0072160 2023-11-24T08:24:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.600+0000 7fcad5465640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 0x7fcab406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.601+0000 7fcad5465640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 0x7fcab406faf0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.604+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcac0055740 con 0x7fcad0072160 2023-11-24T08:24:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.795+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcad004e8b0 con 0x7fcad0072160 2023-11-24T08:24:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.802+0000 7fcac5ffb640 1 -- 172.21.15.107:0/3746850910 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcac00170a0 con 0x7fcad0072160 2023-11-24T08:24:44.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 msgr2=0x7fcab406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 0x7fcab406faf0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc009040 comp rx=0 tx=0).stop 2023-11-24T08:24:44.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 msgr2=0x7fcad015a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7fcac000e9b0 tx=0x7fcac000ee70 comp rx=0 tx=0).stop 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 shutdown_connections 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcab406d630 0x7fcab406faf0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcad00c49f0 0x7fcad015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcad0072160 0x7fcad015a470 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 --2- 172.21.15.107:0/3746850910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcad006de10 0x7fcad006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 >> 172.21.15.107:0/3746850910 conn(0x7fcad00b0c50 msgr2=0x7fcad00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 shutdown_connections 2023-11-24T08:24:44.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:44.805+0000 7fcad76c9640 1 -- 172.21.15.107:0/3746850910 wait complete. 2023-11-24T08:24:44.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot' 2023-11-24T08:24:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.048+0000 7fcd0d809640 1 Processor -- start 2023-11-24T08:24:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.048+0000 7fcd0d809640 1 -- start start 2023-11-24T08:24:45.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d809640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd08161080 0x7fcd080b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d809640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd08161510 con 0x7fcd080b0fb0 2023-11-24T08:24:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d809640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd081617f0 con 0x7fcd08143870 2023-11-24T08:24:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d809640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd080b1460 con 0x7fcd08161080 2023-11-24T08:24:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d008640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcd080b0fb0 0x7fcd080cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42484/0 (socket says 172.21.15.107:42484) 2023-11-24T08:24:45.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd0d008640 1 -- 172.21.15.107:0/2620252499 learned_addr learned my addr 172.21.15.107:0/2620252499 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:45.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd07fff640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd08161080 0x7fcd080b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.050+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709035457 0 0) 0x7fcd08161510 con 0x7fcd080b0fb0 2023-11-24T08:24:45.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf8003520 con 0x7fcd080b0fb0 2023-11-24T08:24:45.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244930230 0 0) 0x7fccf8003520 con 0x7fcd080b0fb0 2023-11-24T08:24:45.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd08161080 msgr2=0x7fcd080b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd08161080 0x7fcd080b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 >> v1:172.21.15.120:6789/0 conn(0x7fcd08143870 legacy=0x7fcd08146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08161ad0 con 0x7fcd080b0fb0 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fcd000019f0 con 0x7fcd080b0fb0 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 >> v1:172.21.15.107:6789/0 conn(0x7fcd080b0fb0 legacy=0x7fcd080cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf80039a0 0x7fccf8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf80047c0 0x7fccf8004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fccf800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd080b1460 con 0x7fccf8005140 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf800bec0 con 0x7fccf80039a0 2023-11-24T08:24:45.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf800c000 con 0x7fccf80047c0 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.051+0000 7fcd0d008640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fccf800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0d008640 1 -- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf80047c0 msgr2=0x7fccf8004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0d008640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf80047c0 0x7fccf8004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0d008640 1 -- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf80039a0 msgr2=0x7fccf8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0c807640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf80039a0 0x7fccf8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0d008640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf80039a0 0x7fccf8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.052+0000 7fcd0d008640 1 -- 172.21.15.107:0/2620252499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf8003520 con 0x7fccf8005140 2023-11-24T08:24:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.053+0000 7fcd0d008640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fccf800b980 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7fcd0000a750 tx=0x7fccf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bc6653bcd2f697 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.053+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd00034630 con 0x7fccf8005140 2023-11-24T08:24:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.053+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0002f070 con 0x7fccf8005140 2023-11-24T08:24:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.053+0000 7fcd077fe640 1 -- 172.21.15.107:0/2620252499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd000349f0 con 0x7fccf8005140 2023-11-24T08:24:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 -- 172.21.15.107:0/2620252499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 msgr2=0x7fccf800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 --2- 172.21.15.107:0/2620252499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fccf800b980 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7fcd0000a750 tx=0x7fccf80057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d008640 1 -- 172.21.15.107:0/2620252499 reap_dead start 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 -- 172.21.15.107:0/2620252499 shutdown_connections 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 -- 172.21.15.107:0/2620252499 >> 172.21.15.107:0/2620252499 conn(0x7fcd080cb830 msgr2=0x7fcd080cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 -- 172.21.15.107:0/2620252499 shutdown_connections 2023-11-24T08:24:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.054+0000 7fcd0d809640 1 -- 172.21.15.107:0/2620252499 wait complete. 2023-11-24T08:24:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.056+0000 7fcd0d809640 1 Processor -- start 2023-11-24T08:24:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.056+0000 7fcd0d809640 1 -- start start 2023-11-24T08:24:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd0815f650 0x7fcd08141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd08141940 0x7fcd080b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd081400b0 con 0x7fccf8005140 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0815e5b0 con 0x7fcd08141940 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0d809640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0815e890 con 0x7fcd0815f650 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0c807640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0c807640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48510/0 (socket says 172.21.15.107:48510) 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.057+0000 7fcd0c807640 1 -- 172.21.15.107:0/966364914 learned_addr learned my addr 172.21.15.107:0/966364914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 -- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd0815f650 msgr2=0x7fcd08141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd0815f650 0x7fcd08141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 -- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd08141940 msgr2=0x7fcd080b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd08141940 0x7fcd080b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 -- 172.21.15.107:0/966364914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0815eb70 con 0x7fccf8005140 2023-11-24T08:24:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0c807640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7fccfc001eb0 tx=0x7fccfc00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc005480 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc003610 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc003980 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0815ee50 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.058+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0815e1f0 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.059+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0804e840 con 0x7fccf8005140 2023-11-24T08:24:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.060+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fccfc003ae0 con 0x7fccf8005140 2023-11-24T08:24:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.061+0000 7fcd057fa640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 0x7fccec06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.061+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fccfc088780 con 0x7fccf8005140 2023-11-24T08:24:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.062+0000 7fcd07fff640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 0x7fccec06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.062+0000 7fcd07fff640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 0x7fccec06faf0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcd08160140 tx=0x7fccf4009500 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.065+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccfc055330 con 0x7fccf8005140 2023-11-24T08:24:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.333+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot"]} v 0) v1 -- 0x7fcd080ce8b0 con 0x7fccf8005140 2023-11-24T08:24:45.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.344+0000 7fcd057fa640 1 -- 172.21.15.107:0/966364914 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_clone_from_newer_snapshot"]}]=0 v1011) v1 ==== 189+0+0 (secure 0 0 0) 0x7fccfc058950 con 0x7fccf8005140 2023-11-24T08:24:45.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 msgr2=0x7fccec06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 0x7fccec06faf0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcd08160140 tx=0x7fccf4009500 comp rx=0 tx=0).stop 2023-11-24T08:24:45.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 msgr2=0x7fcd0813fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7fccfc001eb0 tx=0x7fccfc00d9d0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 shutdown_connections 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fccec06d630 0x7fccec06faf0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd08141940 0x7fcd080b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd0815f650 0x7fcd08141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 --2- 172.21.15.107:0/966364914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf8005140 0x7fcd0813fb70 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 >> 172.21.15.107:0/966364914 conn(0x7fcd080cb830 msgr2=0x7fcd080b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:45.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 shutdown_connections 2023-11-24T08:24:45.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.348+0000 7fcd0d809640 1 -- 172.21.15.107:0/966364914 wait complete. 2023-11-24T08:24:45.368 INFO:tasks.cephfs_test_runner:clone a subvolume from recreated subvolume's latest snapshot ... ok 2023-11-24T08:24:45.369 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) recreate a subvolume from one of its retained snapshots 2023-11-24T08:24:45.369 INFO:tasks.cephfs_test_runner:test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:24:45.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:45.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.581+0000 7f3b0be3d640 1 Processor -- start 2023-11-24T08:24:45.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.582+0000 7f3b0be3d640 1 -- start start 2023-11-24T08:24:45.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.583+0000 7f3b0be3d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b0415c7f0 0x7f3b0406db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.583+0000 7f3b0be3d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b040cece0 con 0x7f3b040ce6c0 2023-11-24T08:24:45.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.583+0000 7f3b0be3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0406d040 con 0x7f3b040cf450 2023-11-24T08:24:45.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.583+0000 7f3b0be3d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b040ceaf0 con 0x7f3b0415c7f0 2023-11-24T08:24:45.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.583+0000 7f3b0a3da640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3b040ce6c0 0x7f3b040714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42498/0 (socket says 172.21.15.107:42498) 2023-11-24T08:24:45.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b0a3da640 1 -- 172.21.15.107:0/2597634081 learned_addr learned my addr 172.21.15.107:0/2597634081 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:45.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/2597634081 >> v2:172.21.15.120:3300/0 conn(0x7f3b0415c7f0 0x7f3b0406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261785064 0 0) 0x7f3b040cece0 con 0x7f3b040ce6c0 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af0003520 con 0x7f3b040ce6c0 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204996389 0 0) 0x7f3af0003520 con 0x7f3b040ce6c0 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 >> v2:172.21.15.120:3300/0 conn(0x7f3b0415c7f0 msgr2=0x7f3b0406db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 --2- 172.21.15.107:0/2597634081 >> v2:172.21.15.120:3300/0 conn(0x7f3b0415c7f0 0x7f3b0406db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 >> v1:172.21.15.120:6789/0 conn(0x7f3b040cf450 legacy=0x7f3b040cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0406d320 con 0x7f3b040ce6c0 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f3afc0019f0 con 0x7f3b040ce6c0 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.584+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 >> v1:172.21.15.107:6789/0 conn(0x7f3b040ce6c0 legacy=0x7f3b040714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3af0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3af00047c0 0x7f3af0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3af0005180 0x7f3af000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af000c050 con 0x7f3af00039c0 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af000c1c0 con 0x7f3af00047c0 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af000c380 con 0x7f3af0005180 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3af0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 -- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3af0005180 msgr2=0x7f3af000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3af0005180 0x7f3af000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 -- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3af00047c0 msgr2=0x7f3af0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3af00047c0 0x7f3af0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b09bd9640 1 -- 172.21.15.107:0/2597634081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0003520 con 0x7f3af00039c0 2023-11-24T08:24:45.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.585+0000 7f3b0a3da640 1 -- 172.21.15.107:0/2597634081 reap_dead start 2023-11-24T08:24:45.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.586+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3af0003de0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009ac0 tx=0x7f3af00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5adcc0873a81b81 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0002c6d0 con 0x7f3af00039c0 2023-11-24T08:24:45.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0002c830 con 0x7f3af00039c0 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b08bd7640 1 -- 172.21.15.107:0/2597634081 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0002cbc0 con 0x7f3af00039c0 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 -- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 msgr2=0x7f3af0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3af0003de0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009ac0 tx=0x7f3af00057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 -- 172.21.15.107:0/2597634081 shutdown_connections 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/2597634081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3af0003de0 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 -- 172.21.15.107:0/2597634081 >> 172.21.15.107:0/2597634081 conn(0x7f3b0406a920 msgr2=0x7f3b0406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 -- 172.21.15.107:0/2597634081 shutdown_connections 2023-11-24T08:24:45.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.587+0000 7f3b0be3d640 1 -- 172.21.15.107:0/2597634081 wait complete. 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.589+0000 7f3b0be3d640 1 Processor -- start 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.589+0000 7f3b0be3d640 1 -- start start 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b040cb930 0x7f3b041470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b041475f0 0x7f3b040aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0414b360 con 0x7f3af00039c0 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04147c20 con 0x7f3b041475f0 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b0be3d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04147f00 con 0x7f3b040cb930 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b09bd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b09bd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48534/0 (socket says 172.21.15.107:48534) 2023-11-24T08:24:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.590+0000 7f3b09bd9640 1 -- 172.21.15.107:0/3315215044 learned_addr learned my addr 172.21.15.107:0/3315215044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 -- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b040cb930 msgr2=0x7f3b041470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b040cb930 0x7f3b041470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 -- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b041475f0 msgr2=0x7f3b040aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b041475f0 0x7f3b040aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 -- 172.21.15.107:0/3315215044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b041f6700 con 0x7f3af00039c0 2023-11-24T08:24:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b09bd9640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f3b0002c270 tx=0x7f3b00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003810 con 0x7f3af00039c0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00003970 con 0x7f3af00039c0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003ce0 con 0x7f3af00039c0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.591+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b041f69e0 con 0x7f3af00039c0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.592+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b040cbdb0 con 0x7f3af00039c0 2023-11-24T08:24:45.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.593+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b04067470 con 0x7f3af00039c0 2023-11-24T08:24:45.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.594+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f3b00040660 con 0x7f3af00039c0 2023-11-24T08:24:45.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.595+0000 7f3afa7fc640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 0x7f3ae806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:45.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.595+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3b000aa910 con 0x7f3af00039c0 2023-11-24T08:24:45.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.595+0000 7f3b093d8640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 0x7f3ae806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:45.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.596+0000 7f3b093d8640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 0x7f3ae806fb20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3b0414b730 tx=0x7f3af4007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:45.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.599+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b000b6080 con 0x7f3af00039c0 2023-11-24T08:24:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.793+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b040cd730 con 0x7f3af00039c0 2023-11-24T08:24:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.795+0000 7f3afa7fc640 1 -- 172.21.15.107:0/3315215044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+12996 (secure 0 0 0) 0x7f3b00039090 con 0x7f3af00039c0 2023-11-24T08:24:45.797 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:45.797 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":240,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:44.333102+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/438366562":"2023-11-25T08:24:40.890936+0000","172.21.15.107:6836/438366562":"2023-11-25T08:24:40.890936+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.798+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 msgr2=0x7f3ae806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.798+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 0x7f3ae806fb20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3b0414b730 tx=0x7f3af4007990 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.798+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 msgr2=0x7f3b041f71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.798+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f3b0002c270 tx=0x7f3b00004730 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 shutdown_connections 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f3ae806d660 0x7f3ae806fb20 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b041475f0 0x7f3b040aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b040cb930 0x7f3b041470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 --2- 172.21.15.107:0/3315215044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3af00039c0 0x7f3b041f71e0 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 >> 172.21.15.107:0/3315215044 conn(0x7f3b0406a920 msgr2=0x7f3b040afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 shutdown_connections 2023-11-24T08:24:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:45.799+0000 7f3b0be3d640 1 -- 172.21.15.107:0/3315215044 wait complete. 2023-11-24T08:24:45.820 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:24:45.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:24:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.037+0000 7f46ede87640 1 Processor -- start 2023-11-24T08:24:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.037+0000 7f46ede87640 1 -- start start 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46ede87640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e815b440 0x7f46e80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46ede87640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 0x7f46e80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46ede87640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e815eea0 con 0x7f46e80c57e0 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46ede87640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e80cc3d0 con 0x7f46e815b440 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46ede87640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e815b820 con 0x7f46e80cbf20 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46e77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e815b440 0x7f46e80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46e77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46e815b440 0x7f46e80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48546/0 (socket says 172.21.15.107:48546) 2023-11-24T08:24:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.039+0000 7f46e77fe640 1 -- 172.21.15.107:0/368988078 learned_addr learned my addr 172.21.15.107:0/368988078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e6ffd640 1 --2- 172.21.15.107:0/368988078 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 0x7f46e80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e77fe640 1 -- 172.21.15.107:0/368988078 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 msgr2=0x7f46e80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e77fe640 1 --2- 172.21.15.107:0/368988078 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 0x7f46e80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e77fe640 1 -- 172.21.15.107:0/368988078 >> v1:172.21.15.120:6789/0 conn(0x7f46e80c57e0 legacy=0x7f46e80c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e77fe640 1 -- 172.21.15.107:0/368988078 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e806d600 con 0x7f46e815b440 2023-11-24T08:24:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.040+0000 7f46e6ffd640 1 --2- 172.21.15.107:0/368988078 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 0x7f46e80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.041+0000 7f46e77fe640 1 --2- 172.21.15.107:0/368988078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80cb9e0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7f46e806e890 tx=0x7f46d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be9a7e64276f16b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46e67fc640 1 -- 172.21.15.107:0/368988078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d80317e0 con 0x7f46e815b440 2023-11-24T08:24:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46e67fc640 1 -- 172.21.15.107:0/368988078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d8031940 con 0x7f46e815b440 2023-11-24T08:24:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46e67fc640 1 -- 172.21.15.107:0/368988078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8031cd0 con 0x7f46e815b440 2023-11-24T08:24:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 -- 172.21.15.107:0/368988078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 msgr2=0x7f46e80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 --2- 172.21.15.107:0/368988078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80cb9e0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7f46e806e890 tx=0x7f46d802b880 comp rx=0 tx=0).stop 2023-11-24T08:24:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 -- 172.21.15.107:0/368988078 shutdown_connections 2023-11-24T08:24:46.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 --2- 172.21.15.107:0/368988078 >> v2:172.21.15.120:3300/0 conn(0x7f46e80cbf20 0x7f46e80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 --2- 172.21.15.107:0/368988078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80cb9e0 unknown :-1 s=CLOSED pgs=3336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.042+0000 7f46ede87640 1 -- 172.21.15.107:0/368988078 >> 172.21.15.107:0/368988078 conn(0x7f46e80b0c50 msgr2=0x7f46e80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:46.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.043+0000 7f46ede87640 1 -- 172.21.15.107:0/368988078 shutdown_connections 2023-11-24T08:24:46.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.043+0000 7f46ede87640 1 -- 172.21.15.107:0/368988078 wait complete. 2023-11-24T08:24:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.044+0000 7f46ede87640 1 Processor -- start 2023-11-24T08:24:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.044+0000 7f46ede87640 1 -- start start 2023-11-24T08:24:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80c57e0 0x7f46e81e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e80cbf20 0x7f46e815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e815df10 con 0x7f46e815b440 2023-11-24T08:24:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80c7390 con 0x7f46e80cbf20 2023-11-24T08:24:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.045+0000 7f46ede87640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80c7670 con 0x7f46e80c57e0 2023-11-24T08:24:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48552/0 (socket says 172.21.15.107:48552) 2023-11-24T08:24:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 -- 172.21.15.107:0/904775193 learned_addr learned my addr 172.21.15.107:0/904775193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e6ffd640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e80cbf20 0x7f46e815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 -- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80c57e0 msgr2=0x7f46e81e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80c57e0 0x7f46e81e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 -- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e80cbf20 msgr2=0x7f46e815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e80cbf20 0x7f46e815b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 -- 172.21.15.107:0/904775193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80094d0 con 0x7f46e815b440 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.046+0000 7f46e7fff640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c780 tx=0x7f46e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.047+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0004b60 con 0x7f46e815b440 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.047+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e000ce90 con 0x7f46e815b440 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.047+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e000a850 con 0x7f46e815b440 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.047+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e80c7950 con 0x7f46e815b440 2023-11-24T08:24:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.047+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e80c7f10 con 0x7f46e815b440 2023-11-24T08:24:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.048+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e804e8b0 con 0x7f46e815b440 2023-11-24T08:24:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.049+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f46e000a9b0 con 0x7f46e815b440 2023-11-24T08:24:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.050+0000 7f46cffff640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 0x7f46d406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.051+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f46e0088ba0 con 0x7f46e815b440 2023-11-24T08:24:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.051+0000 7f46e77fe640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 0x7f46d406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.051+0000 7f46e77fe640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 0x7f46d406faf0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.055+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46e00556b0 con 0x7f46e815b440 2023-11-24T08:24:46.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.245+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46e806dc80 con 0x7f46e815b440 2023-11-24T08:24:46.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.246+0000 7f46cffff640 1 -- 172.21.15.107:0/904775193 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v240) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46e0058cd0 con 0x7f46e815b440 2023-11-24T08:24:46.248 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.248+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 msgr2=0x7f46d406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.248+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 0x7f46d406faf0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d8030000 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 msgr2=0x7f46e80c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c780 tx=0x7f46e000cc40 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 shutdown_connections 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f46d406d630 0x7f46d406faf0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e815b440 0x7f46e80c6ab0 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46e80cbf20 0x7f46e815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 --2- 172.21.15.107:0/904775193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46e80c57e0 0x7f46e81e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 >> 172.21.15.107:0/904775193 conn(0x7f46e80b0c50 msgr2=0x7f46e806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:46.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 shutdown_connections 2023-11-24T08:24:46.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.249+0000 7f46ede87640 1 -- 172.21.15.107:0/904775193 wait complete. 2023-11-24T08:24:46.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_recreate' 2023-11-24T08:24:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.466+0000 7f3746d61640 1 Processor -- start 2023-11-24T08:24:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.467+0000 7f3746d61640 1 -- start start 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 0x7f37400b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37400abf50 0x7f37400b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37400ccea0 0x7f37400cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37400b4540 con 0x7f37400ccea0 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37400cd870 con 0x7f37400abf50 2023-11-24T08:24:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f3746d61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37400711e0 con 0x7f374015b5e0 2023-11-24T08:24:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.468+0000 7f37452fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37400ccea0 0x7f37400cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37400ccea0 0x7f37400cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48560/0 (socket says 172.21.15.107:48560) 2023-11-24T08:24:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 -- 172.21.15.107:0/2092577874 learned_addr learned my addr 172.21.15.107:0/2092577874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f3744afd640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 0x7f37400b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f373ffff640 1 --2- 172.21.15.107:0/2092577874 >> v2:172.21.15.120:3300/0 conn(0x7f37400abf50 0x7f37400b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 -- 172.21.15.107:0/2092577874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 msgr2=0x7f37400b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 0x7f37400b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 -- 172.21.15.107:0/2092577874 >> v2:172.21.15.120:3300/0 conn(0x7f37400abf50 msgr2=0x7f37400b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 --2- 172.21.15.107:0/2092577874 >> v2:172.21.15.120:3300/0 conn(0x7f37400abf50 0x7f37400b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f37452fe640 1 -- 172.21.15.107:0/2092577874 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37400cdc80 con 0x7f37400ccea0 2023-11-24T08:24:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.469+0000 7f3744afd640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 0x7f37400b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.470+0000 7f37452fe640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37400ccea0 0x7f37400cd330 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f374006f980 tx=0x7f373802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa365b10249122b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f373f7fe640 1 -- 172.21.15.107:0/2092577874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373802ca70 con 0x7f37400ccea0 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f373f7fe640 1 -- 172.21.15.107:0/2092577874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373802cbd0 con 0x7f37400ccea0 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f373f7fe640 1 -- 172.21.15.107:0/2092577874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380343f0 con 0x7f37400ccea0 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 -- 172.21.15.107:0/2092577874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37400ccea0 msgr2=0x7f37400cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37400ccea0 0x7f37400cd330 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f374006f980 tx=0x7f373802bed0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 -- 172.21.15.107:0/2092577874 shutdown_connections 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37400ccea0 0x7f37400cd330 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 --2- 172.21.15.107:0/2092577874 >> v2:172.21.15.120:3300/0 conn(0x7f37400abf50 0x7f37400b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 --2- 172.21.15.107:0/2092577874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f374015b5e0 0x7f37400b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 -- 172.21.15.107:0/2092577874 >> 172.21.15.107:0/2092577874 conn(0x7f37400b0c70 msgr2=0x7f37400b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.471+0000 7f3746d61640 1 -- 172.21.15.107:0/2092577874 shutdown_connections 2023-11-24T08:24:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.472+0000 7f3746d61640 1 -- 172.21.15.107:0/2092577874 wait complete. 2023-11-24T08:24:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.473+0000 7f3746d61640 1 Processor -- start 2023-11-24T08:24:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.473+0000 7f3746d61640 1 -- start start 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.474+0000 7f3746d61640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37400abf50 0x7f374014a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.474+0000 7f3746d61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37400ccea0 0x7f3740099930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f3746d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f3746d61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374009a660 con 0x7f374015b5e0 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f3746d61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374006c590 con 0x7f37400ccea0 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f3746d61640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374009a380 con 0x7f37400abf50 2023-11-24T08:24:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48574/0 (socket says 172.21.15.107:48574) 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 -- 172.21.15.107:0/838182424 learned_addr learned my addr 172.21.15.107:0/838182424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f3744afd640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37400abf50 0x7f374014a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 -- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37400abf50 msgr2=0x7f374014a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37400abf50 0x7f374014a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f373ffff640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37400ccea0 0x7f3740099930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 -- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37400ccea0 msgr2=0x7f3740099930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37400ccea0 0x7f3740099930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.475+0000 7f37452fe640 1 -- 172.21.15.107:0/838182424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738009b30 con 0x7f374015b5e0 2023-11-24T08:24:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f37452fe640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f3738036040 tx=0x7f3738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3738009510 con 0x7f374015b5e0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373803e450 con 0x7f374015b5e0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373803e7e0 con 0x7f374015b5e0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374006c7e0 con 0x7f374015b5e0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.476+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374006cda0 con 0x7f374015b5e0 2023-11-24T08:24:46.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.477+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374006cfc0 con 0x7f374015b5e0 2023-11-24T08:24:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.478+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f373803e940 con 0x7f374015b5e0 2023-11-24T08:24:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.479+0000 7f373d7fa640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 0x7f372406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.479+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f37380ab6f0 con 0x7f374015b5e0 2023-11-24T08:24:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.479+0000 7f3744afd640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 0x7f372406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.480+0000 7f3744afd640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 0x7f372406faf0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f37340096d0 tx=0x7f3734008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:46.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.483+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37380782a0 con 0x7f374015b5e0 2023-11-24T08:24:46.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:46.724+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_recreate"]} v 0) v1 -- 0x7f374004e8b0 con 0x7f374015b5e0 2023-11-24T08:24:46.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:24:46.783 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:24:47.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.356+0000 7f373d7fa640 1 -- 172.21.15.107:0/838182424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_retain_snapshot_recreate"]}]=0 v1013) v1 ==== 175+0+0 (secure 0 0 0) 0x7f37380430f0 con 0x7f374015b5e0 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.359+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 msgr2=0x7f372406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.359+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 0x7f372406faf0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f37340096d0 tx=0x7f3734008040 comp rx=0 tx=0).stop 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 msgr2=0x7f3740159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f3738036040 tx=0x7f3738004a40 comp rx=0 tx=0).stop 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 shutdown_connections 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f372406d630 0x7f372406faf0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f374015b5e0 0x7f3740159120 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37400ccea0 0x7f3740099930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 --2- 172.21.15.107:0/838182424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37400abf50 0x7f374014a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 >> 172.21.15.107:0/838182424 conn(0x7f37400b0c70 msgr2=0x7f37400afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:47.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 shutdown_connections 2023-11-24T08:24:47.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.360+0000 7f3746d61640 1 -- 172.21.15.107:0/838182424 wait complete. 2023-11-24T08:24:47.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:24:47.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.597+0000 7f47ce1fa640 1 Processor -- start 2023-11-24T08:24:47.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.597+0000 7f47ce1fa640 1 -- start start 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47ce1fa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47c815d810 0x7f47c8071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47ce1fa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47c8071ea0 0x7f47c8072330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47ce1fa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c815e9b0 con 0x7f47c815de90 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47ce1fa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c8072870 con 0x7f47c8071ea0 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47ce1fa640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c815ebe0 con 0x7f47c815d810 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47cc9f7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47c8071ea0 0x7f47c8072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47cc9f7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47c8071ea0 0x7f47c8072330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48582/0 (socket says 172.21.15.107:48582) 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47cc9f7640 1 -- 172.21.15.107:0/13388853 learned_addr learned my addr 172.21.15.107:0/13388853 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.599+0000 7f47cd9f9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47c815de90 0x7f47c815e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42502/0 (socket says 172.21.15.107:42502) 2023-11-24T08:24:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.600+0000 7f47cc9f7640 1 -- 172.21.15.107:0/13388853 >> v2:172.21.15.120:3300/0 conn(0x7f47c815d810 msgr2=0x7f47c8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.600+0000 7f47cc9f7640 1 --2- 172.21.15.107:0/13388853 >> v2:172.21.15.120:3300/0 conn(0x7f47c815d810 0x7f47c8071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.600+0000 7f47cc9f7640 1 -- 172.21.15.107:0/13388853 >> v1:172.21.15.107:6789/0 conn(0x7f47c815de90 legacy=0x7f47c815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.600+0000 7f47cc9f7640 1 -- 172.21.15.107:0/13388853 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8070860 con 0x7f47c8071ea0 2023-11-24T08:24:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.601+0000 7f47cc9f7640 1 --2- 172.21.15.107:0/13388853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8071ea0 0x7f47c8072330 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f47c806d3d0 tx=0x7f47b802dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac11de3461e1d65e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.601+0000 7f47bffff640 1 -- 172.21.15.107:0/13388853 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b802e970 con 0x7f47c8071ea0 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.601+0000 7f47bffff640 1 -- 172.21.15.107:0/13388853 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b802ead0 con 0x7f47c8071ea0 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.601+0000 7f47bffff640 1 -- 172.21.15.107:0/13388853 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b802ee60 con 0x7f47c8071ea0 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 -- 172.21.15.107:0/13388853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8071ea0 msgr2=0x7f47c8072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/13388853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8071ea0 0x7f47c8072330 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f47c806d3d0 tx=0x7f47b802dbc0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 -- 172.21.15.107:0/13388853 shutdown_connections 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/13388853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8071ea0 0x7f47c8072330 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/13388853 >> v2:172.21.15.120:3300/0 conn(0x7f47c815d810 0x7f47c8071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 -- 172.21.15.107:0/13388853 >> 172.21.15.107:0/13388853 conn(0x7f47c806a500 msgr2=0x7f47c806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:47.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 -- 172.21.15.107:0/13388853 shutdown_connections 2023-11-24T08:24:47.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.602+0000 7f47ce1fa640 1 -- 172.21.15.107:0/13388853 wait complete. 2023-11-24T08:24:47.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.603+0000 7f47ce1fa640 1 Processor -- start 2023-11-24T08:24:47.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 -- start start 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 0x7f47c8146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47c815de90 0x7f47c815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c815c580 con 0x7f47c815d810 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c8147150 con 0x7f47c8071ea0 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.604+0000 7f47ce1fa640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c815b970 con 0x7f47c815de90 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48594/0 (socket says 172.21.15.107:48594) 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 -- 172.21.15.107:0/1636420669 learned_addr learned my addr 172.21.15.107:0/1636420669 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 -- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47c815de90 msgr2=0x7f47c815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47c815de90 0x7f47c815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 -- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 msgr2=0x7f47c8146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cd1f8640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 0x7f47c8146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 0x7f47c8146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cc9f7640 1 -- 172.21.15.107:0/1636420669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b80094d0 con 0x7f47c815d810 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.605+0000 7f47cd1f8640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 0x7f47c8146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47cc9f7640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f47b8000c00 tx=0x7f47b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:47.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8041830 con 0x7f47c815d810 2023-11-24T08:24:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b8041990 con 0x7f47c815d810 2023-11-24T08:24:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c815bc50 con 0x7f47c815d810 2023-11-24T08:24:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8041d40 con 0x7f47c815d810 2023-11-24T08:24:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.606+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c815c210 con 0x7f47c815d810 2023-11-24T08:24:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.607+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c815c6f0 con 0x7f47c815d810 2023-11-24T08:24:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.608+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f47b803d040 con 0x7f47c815d810 2023-11-24T08:24:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.609+0000 7f47bdffb640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 0x7f47ac06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.609+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f47b80ad2c0 con 0x7f47c815d810 2023-11-24T08:24:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.609+0000 7f47cd1f8640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 0x7f47ac06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.610+0000 7f47cd1f8640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 0x7f47ac06faf0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f47c4006fd0 tx=0x7f47c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:47.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.613+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47b8079dd0 con 0x7f47c815d810 2023-11-24T08:24:47.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.805+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f47c804e840 con 0x7f47c815d810 2023-11-24T08:24:47.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.810+0000 7f47bdffb640 1 -- 172.21.15.107:0/1636420669 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47b803b0c0 con 0x7f47c815d810 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 msgr2=0x7f47ac06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 0x7f47ac06faf0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f47c4006fd0 tx=0x7f47c4008040 comp rx=0 tx=0).stop 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 msgr2=0x7f47c815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f47b8000c00 tx=0x7f47b80046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 shutdown_connections 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.813+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f47ac06d630 0x7f47ac06faf0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47c815de90 0x7f47c815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c815d810 0x7f47c815ae00 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 --2- 172.21.15.107:0/1636420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47c8071ea0 0x7f47c8146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 >> 172.21.15.107:0/1636420669 conn(0x7f47c806a500 msgr2=0x7f47c80ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 shutdown_connections 2023-11-24T08:24:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:47.814+0000 7f47ce1fa640 1 -- 172.21.15.107:0/1636420669 wait complete. 2023-11-24T08:24:47.835 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:24:47.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:24:47.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:24:48.072 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T08:24:48.072 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:48.072 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:48.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:24:48.133 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:48.133 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:48.192 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:24:48.192 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:24:48.192 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:48.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054258,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249376k,nr_inodes=812344,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498760k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-24T08:24:48.250 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:24:48.250 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:24:48.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-24T08:24:48.309 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T08:24:48.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:24:48.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.531+0000 7fc45abd4640 1 Processor -- start 2023-11-24T08:24:48.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.531+0000 7fc45abd4640 1 -- start start 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc45abd4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4540d1e20 0x7fc4540cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc45abd4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc4540b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc45abd4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4540d04b0 con 0x7fc45415d4e0 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc45abd4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540d4110 con 0x7fc4540d1e20 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc45abd4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540ce990 con 0x7fc4540d0070 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc4593d1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4540d1e20 0x7fc4540cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc4593d1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4540d1e20 0x7fc4540cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43740/0 (socket says 172.21.15.107:43740) 2023-11-24T08:24:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.533+0000 7fc4593d1640 1 -- 172.21.15.107:0/3572431660 learned_addr learned my addr 172.21.15.107:0/3572431660 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.534+0000 7fc4593d1640 1 -- 172.21.15.107:0/3572431660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 msgr2=0x7fc4540b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.534+0000 7fc4593d1640 1 --2- 172.21.15.107:0/3572431660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc4540b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.534+0000 7fc4593d1640 1 -- 172.21.15.107:0/3572431660 >> v1:172.21.15.120:6789/0 conn(0x7fc45415d4e0 legacy=0x7fc4540d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.534+0000 7fc4593d1640 1 -- 172.21.15.107:0/3572431660 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540cec70 con 0x7fc4540d1e20 2023-11-24T08:24:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.535+0000 7fc4593d1640 1 --2- 172.21.15.107:0/3572431660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4540d1e20 0x7fc4540cfb30 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fc444009870 tx=0x7fc44402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43a6b575dd44863 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.535+0000 7fc458bd0640 1 -- 172.21.15.107:0/3572431660 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444031640 con 0x7fc4540d1e20 2023-11-24T08:24:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.535+0000 7fc458bd0640 1 -- 172.21.15.107:0/3572431660 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4440317a0 con 0x7fc4540d1e20 2023-11-24T08:24:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.535+0000 7fc458bd0640 1 -- 172.21.15.107:0/3572431660 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444031b30 con 0x7fc4540d1e20 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 -- 172.21.15.107:0/3572431660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4540d1e20 msgr2=0x7fc4540cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3572431660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4540d1e20 0x7fc4540cfb30 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fc444009870 tx=0x7fc44402b8f0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 -- 172.21.15.107:0/3572431660 shutdown_connections 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3572431660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc4540b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3572431660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4540d1e20 0x7fc4540cfb30 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 -- 172.21.15.107:0/3572431660 >> 172.21.15.107:0/3572431660 conn(0x7fc45409c510 msgr2=0x7fc4540d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 -- 172.21.15.107:0/3572431660 shutdown_connections 2023-11-24T08:24:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.536+0000 7fc45abd4640 1 -- 172.21.15.107:0/3572431660 wait complete. 2023-11-24T08:24:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.538+0000 7fc45abd4640 1 Processor -- start 2023-11-24T08:24:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.538+0000 7fc45abd4640 1 -- start start 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc454152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4540d1e20 0x7fc4540b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b7ad0 con 0x7fc45415d4e0 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540ca7b0 con 0x7fc4540d1e20 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45abd4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b7510 con 0x7fc4540d0070 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43746/0 (socket says 172.21.15.107:43746) 2023-11-24T08:24:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 -- 172.21.15.107:0/3021023159 learned_addr learned my addr 172.21.15.107:0/3021023159 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 -- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 msgr2=0x7fc454152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc454152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 -- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4540d1e20 msgr2=0x7fc4540b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4540d1e20 0x7fc4540b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.539+0000 7fc45a3d3640 1 -- 172.21.15.107:0/3021023159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4440094d0 con 0x7fc45415d4e0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc45a3d3640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00b970 tx=0x7fc44c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00cb40 con 0x7fc45415d4e0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c00ccc0 con 0x7fc45415d4e0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c014680 con 0x7fc45415d4e0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4540b77f0 con 0x7fc45415d4e0 2023-11-24T08:24:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.540+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4540b67b0 con 0x7fc45415d4e0 2023-11-24T08:24:48.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.541+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4540ca2a0 con 0x7fc45415d4e0 2023-11-24T08:24:48.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.542+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fc44c01d3b0 con 0x7fc45415d4e0 2023-11-24T08:24:48.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.543+0000 7fc44a7fc640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 0x7fc43806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:48.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.543+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc44c0888f0 con 0x7fc45415d4e0 2023-11-24T08:24:48.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.544+0000 7fc459bd2640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 0x7fc43806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:48.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.544+0000 7fc459bd2640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 0x7fc43806faf0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc450009650 tx=0x7fc450006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.548+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4540ca2a0 con 0x7fc45415d4e0 2023-11-24T08:24:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.775+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4540ca2a0 con 0x7fc45415d4e0 2023-11-24T08:24:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.776+0000 7fc44a7fc640 1 -- 172.21.15.107:0/3021023159 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+3718 (secure 0 0 0) 0x7fc4540ca2a0 con 0x7fc45415d4e0 2023-11-24T08:24:48.778 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:48.779 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6834/1760000421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1760000421},{"type":"v1","addr":"172.21.15.120:6834","nonce":1760000421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":8029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3366979353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6833","nonce":3366979353},{"type":"v1","addr":"172.21.15.120:6835","nonce":3366979353}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2321049347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2321049347},{"type":"v1","addr":"172.21.15.107:6835","nonce":2321049347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142},{"gid":13218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4147764906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4147764906},{"type":"v1","addr":"172.21.15.107:6837","nonce":4147764906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[]} 2023-11-24T08:24:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-11-24T08:24:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 msgr2=0x7fc43806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 0x7fc43806faf0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc450009650 tx=0x7fc450006ba0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 msgr2=0x7fc4540b8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00b970 tx=0x7fc44c00be30 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 shutdown_connections 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc43806d630 0x7fc43806faf0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc45415d4e0 0x7fc4540b8890 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4540d1e20 0x7fc4540b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.779+0000 7fc45abd4640 1 --2- 172.21.15.107:0/3021023159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4540d0070 0x7fc454152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.780+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 >> 172.21.15.107:0/3021023159 conn(0x7fc45409c510 msgr2=0x7fc45406b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.780+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 shutdown_connections 2023-11-24T08:24:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:48.780+0000 7fc45abd4640 1 -- 172.21.15.107:0/3021023159 wait complete. 2023-11-24T08:24:48.802 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T08:24:48.802 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T08:24:48.803 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:24:48.803 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:24:48.803 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T08:24:48.803 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T08:24:48.803 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:24:48.803 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:24:48.804 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T08:24:48.804 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T08:24:48.804 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:24:48.804 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:24:48.804 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T08:24:48.805 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T08:24:48.805 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:24:48.805 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:24:48.821 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T08:24:48.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T08:24:48.827 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T08:24:48.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T08:24:48.828 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T08:24:48.829 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T08:24:48.870 INFO:tasks.ceph.mds.a:Started 2023-11-24T08:24:48.870 DEBUG:teuthology.parallel:result is None 2023-11-24T08:24:48.877 INFO:tasks.ceph.mds.b:Started 2023-11-24T08:24:48.878 DEBUG:teuthology.parallel:result is None 2023-11-24T08:24:48.885 INFO:tasks.ceph.mds.d:Started 2023-11-24T08:24:48.885 DEBUG:teuthology.parallel:result is None 2023-11-24T08:24:49.037 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2023-11-24T08:24:49.039 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2023-11-24T08:24:49.049 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-11-24T08:24:53.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-11-24T08:24:54.821 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T08:24:54.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T08:24:54.831 INFO:tasks.ceph.mds.c:Started 2023-11-24T08:24:54.832 DEBUG:teuthology.parallel:result is None 2023-11-24T08:24:54.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T08:24:54.997 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-11-24T08:24:55.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.029+0000 7fbdcb65d640 1 Processor -- start 2023-11-24T08:24:55.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.029+0000 7fbdcb65d640 1 -- start start 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdcb65d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 0x7fbdc40c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdcb65d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc415eea0 con 0x7fbdc415b440 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdcb65d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc40abf30 con 0x7fbdc40c57e0 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdcb65d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc415b820 con 0x7fbdc40cc0f0 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdc93f9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbdc415b440 0x7fbdc40cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58552/0 (socket says 172.21.15.107:58552) 2023-11-24T08:24:55.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.030+0000 7fbdc93f9640 1 -- 172.21.15.107:0/1354804048 learned_addr learned my addr 172.21.15.107:0/1354804048 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478089367 0 0) 0x7fbdc415eea0 con 0x7fbdc415b440 2023-11-24T08:24:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb0003520 con 0x7fbdc415b440 2023-11-24T08:24:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 0x7fbdc40c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257757829 0 0) 0x7fbdb0003520 con 0x7fbdc415b440 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 msgr2=0x7fbdc40c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 0x7fbdc40c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 >> v1:172.21.15.120:6789/0 conn(0x7fbdc40c57e0 legacy=0x7fbdc406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc415f0f0 con 0x7fbdc415b440 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fbdc0001670 con 0x7fbdc415b440 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 >> v1:172.21.15.107:6789/0 conn(0x7fbdc415b440 legacy=0x7fbdc40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 0x7fbdb00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdb0004020 0x7fbdb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdb0004d00 0x7fbdb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc415b820 con 0x7fbdb0004020 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb000bc60 con 0x7fbdb0004d00 2023-11-24T08:24:55.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.031+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb000be20 con 0x7fbdc40cc0f0 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdb0004020 0x7fbdb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 msgr2=0x7fbdb00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc40cc0f0 0x7fbdb00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc9bfa640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdb0004d00 0x7fbdb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdb0004d00 msgr2=0x7fbdb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdb0004d00 0x7fbdb0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1354804048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0003520 con 0x7fbdb0004020 2023-11-24T08:24:55.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.032+0000 7fbdc9bfa640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdb0004d00 0x7fbdb0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.033+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdb0004020 0x7fbdb00047c0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400ba30 tx=0x7fbdb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d075e1e494b4028 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.033+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4032680 con 0x7fbdb0004020 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.033+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb40327e0 con 0x7fbdb0004020 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.033+0000 7fbdbbfff640 1 -- 172.21.15.107:0/1354804048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4032b70 con 0x7fbdb0004020 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1354804048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdb0004020 msgr2=0x7fbdb00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1354804048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdb0004020 0x7fbdb00047c0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400ba30 tx=0x7fbdb00057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdc9bfa640 1 -- 172.21.15.107:0/1354804048 reap_dead start 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1354804048 shutdown_connections 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1354804048 >> 172.21.15.107:0/1354804048 conn(0x7fbdc40b0c50 msgr2=0x7fbdc40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1354804048 shutdown_connections 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.034+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1354804048 wait complete. 2023-11-24T08:24:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.036+0000 7fbdcb65d640 1 Processor -- start 2023-11-24T08:24:55.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.036+0000 7fbdcb65d640 1 -- start start 2023-11-24T08:24:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 0x7fbdc415a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc4155de0 0x7fbdc41540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc41509d0 con 0x7fbdc40c6370 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc41f76a0 con 0x7fbdc4155de0 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdcb65d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc41f70e0 con 0x7fbdb0004020 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdc8bf8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdc8bf8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43804/0 (socket says 172.21.15.107:43804) 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.037+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1335092205 learned_addr learned my addr 172.21.15.107:0/1335092205 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc9bfa640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc4155de0 0x7fbdc41540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc93f9640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 0x7fbdc415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 msgr2=0x7fbdc415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 0x7fbdc415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc4155de0 msgr2=0x7fbdc41540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:55.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc4155de0 0x7fbdc41540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 -- 172.21.15.107:0/1335092205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc41f73c0 con 0x7fbdc40c6370 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc93f9640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 0x7fbdc415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdc8bf8640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402bf00 tx=0x7fbdb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4030aa0 con 0x7fbdc40c6370 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb4030c20 con 0x7fbdc40c6370 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb403b420 con 0x7fbdc40c6370 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc41555e0 con 0x7fbdc40c6370 2023-11-24T08:24:55.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.038+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc415aac0 con 0x7fbdc40c6370 2023-11-24T08:24:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.040+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc404e8b0 con 0x7fbdc40c6370 2023-11-24T08:24:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.041+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fbdb403b580 con 0x7fbdc40c6370 2023-11-24T08:24:55.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.041+0000 7fbdb9ffb640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 0x7fbda806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.042+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbdb40ab300 con 0x7fbdc40c6370 2023-11-24T08:24:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.042+0000 7fbdc93f9640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 0x7fbda806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:55.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.042+0000 7fbdc93f9640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 0x7fbda806faf0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0007a10 tx=0x7fbdc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:55.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.046+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdb4077fd0 con 0x7fbdc40c6370 2023-11-24T08:24:55.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:55.251+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbdc406dc80 con 0x7fbdc40c6370 2023-11-24T08:24:56.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.236+0000 7fbdb9ffb640 1 -- 172.21.15.107:0/1335092205 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v245) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbdb4036070 con 0x7fbdc40c6370 2023-11-24T08:24:56.238 INFO:teuthology.orchestra.run.smithi107.stderr: removed all blocklist entries 2023-11-24T08:24:56.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 msgr2=0x7fbda806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 0x7fbda806faf0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0007a10 tx=0x7fbdc0009040 comp rx=0 tx=0).stop 2023-11-24T08:24:56.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 msgr2=0x7fbdc41f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402bf00 tx=0x7fbdb4004730 comp rx=0 tx=0).stop 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 shutdown_connections 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fbda806d630 0x7fbda806faf0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc4155de0 0x7fbdc41540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc40c6370 0x7fbdc41f6770 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 --2- 172.21.15.107:0/1335092205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdb0004020 0x7fbdc415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 >> 172.21.15.107:0/1335092205 conn(0x7fbdc40b0c50 msgr2=0x7fbdc4071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 shutdown_connections 2023-11-24T08:24:56.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.239+0000 7fbdcb65d640 1 -- 172.21.15.107:0/1335092205 wait complete. 2023-11-24T08:24:56.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T08:24:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.468+0000 7f31f1fff640 1 Processor -- start 2023-11-24T08:24:56.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.468+0000 7f31f1fff640 1 -- start start 2023-11-24T08:24:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f1fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31ec06d0b0 0x7f31ec070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f1fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31ec0708b0 0x7f31ec13e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f1fff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec15b7a0 con 0x7f31ec13e0e0 2023-11-24T08:24:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f1fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ec15ba60 con 0x7f31ec06d0b0 2023-11-24T08:24:56.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f1fff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ec15bd40 con 0x7f31ec0708b0 2023-11-24T08:24:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31ec06d0b0 0x7f31ec070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31ec06d0b0 0x7f31ec070370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43814/0 (socket says 172.21.15.107:43814) 2023-11-24T08:24:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31ebfff640 1 -- 172.21.15.107:0/3559609611 learned_addr learned my addr 172.21.15.107:0/3559609611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:56.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.470+0000 7f31f17fe640 1 --2- 172.21.15.107:0/3559609611 >> v2:172.21.15.120:3300/0 conn(0x7f31ec0708b0 0x7f31ec13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.471+0000 7f31ebfff640 1 -- 172.21.15.107:0/3559609611 >> v2:172.21.15.120:3300/0 conn(0x7f31ec0708b0 msgr2=0x7f31ec13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.471+0000 7f31ebfff640 1 --2- 172.21.15.107:0/3559609611 >> v2:172.21.15.120:3300/0 conn(0x7f31ec0708b0 0x7f31ec13e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.471+0000 7f31ebfff640 1 -- 172.21.15.107:0/3559609611 >> v1:172.21.15.120:6789/0 conn(0x7f31ec13e0e0 legacy=0x7f31ec06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.471+0000 7f31ebfff640 1 -- 172.21.15.107:0/3559609611 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec15c020 con 0x7f31ec06d0b0 2023-11-24T08:24:56.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.472+0000 7f31ebfff640 1 --2- 172.21.15.107:0/3559609611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec06d0b0 0x7f31ec070370 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f31ec071120 tx=0x7f31d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4dbe8e14fed21d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.472+0000 7f31eb7fe640 1 -- 172.21.15.107:0/3559609611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8031870 con 0x7f31ec06d0b0 2023-11-24T08:24:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.472+0000 7f31eb7fe640 1 -- 172.21.15.107:0/3559609611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d80319d0 con 0x7f31ec06d0b0 2023-11-24T08:24:56.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.472+0000 7f31eb7fe640 1 -- 172.21.15.107:0/3559609611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8031d60 con 0x7f31ec06d0b0 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 -- 172.21.15.107:0/3559609611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec06d0b0 msgr2=0x7f31ec070370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 --2- 172.21.15.107:0/3559609611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec06d0b0 0x7f31ec070370 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f31ec071120 tx=0x7f31d802b910 comp rx=0 tx=0).stop 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 -- 172.21.15.107:0/3559609611 shutdown_connections 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 --2- 172.21.15.107:0/3559609611 >> v2:172.21.15.120:3300/0 conn(0x7f31ec0708b0 0x7f31ec13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 --2- 172.21.15.107:0/3559609611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec06d0b0 0x7f31ec070370 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 -- 172.21.15.107:0/3559609611 >> 172.21.15.107:0/3559609611 conn(0x7f31ec0b4630 msgr2=0x7f31ec0b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:56.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 -- 172.21.15.107:0/3559609611 shutdown_connections 2023-11-24T08:24:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.473+0000 7f31f1fff640 1 -- 172.21.15.107:0/3559609611 wait complete. 2023-11-24T08:24:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.475+0000 7f31f1fff640 1 Processor -- start 2023-11-24T08:24:56.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.475+0000 7f31f1fff640 1 -- start start 2023-11-24T08:24:56.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.476+0000 7f31f1fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31ec06d0b0 0x7f31ec13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.476+0000 7f31f1fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f1fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 0x7f31ec0aac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f1fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec0a8430 con 0x7f31ec0708b0 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f1fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec0af990 con 0x7f31ec06d0b0 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f1fff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec0b0940 con 0x7f31ec13e0e0 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43826/0 (socket says 172.21.15.107:43826) 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 -- 172.21.15.107:0/1842190330 learned_addr learned my addr 172.21.15.107:0/1842190330 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f0ffd640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31ec06d0b0 0x7f31ec13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f17fe640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 0x7f31ec0aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 -- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 msgr2=0x7f31ec0aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 0x7f31ec0aac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 -- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31ec06d0b0 msgr2=0x7f31ec13bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31ec06d0b0 0x7f31ec13bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31ebfff640 1 -- 172.21.15.107:0/1842190330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d80094d0 con 0x7f31ec0708b0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.477+0000 7f31f17fe640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 0x7f31ec0aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31ebfff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f31d802b950 tx=0x7f31d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d803abe0 con 0x7f31ec0708b0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d803ad40 con 0x7f31ec0708b0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec0a8680 con 0x7f31ec0708b0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8044610 con 0x7f31ec0708b0 2023-11-24T08:24:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.478+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec0afee0 con 0x7f31ec0708b0 2023-11-24T08:24:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.480+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec0b00d0 con 0x7f31ec0708b0 2023-11-24T08:24:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.480+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f31d8044770 con 0x7f31ec0708b0 2023-11-24T08:24:56.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.481+0000 7f31e97fa640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 0x7f31d006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.481+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f31d802d040 con 0x7f31ec0708b0 2023-11-24T08:24:56.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.482+0000 7f31f0ffd640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 0x7f31d006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.482+0000 7f31f0ffd640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 0x7f31d006faf0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f31e0006fd0 tx=0x7f31e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.486+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31d8077b40 con 0x7f31ec0708b0 2023-11-24T08:24:56.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.741+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f31ec04e840 con 0x7f31ec0708b0 2023-11-24T08:24:56.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.742+0000 7f31e97fa640 1 -- 172.21.15.107:0/1842190330 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f31d8035090 con 0x7f31ec0708b0 2023-11-24T08:24:56.744 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:56.744 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:24:56.744 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_dump": [ 2023-11-24T08:24:56.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.a", 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlEtVpLBAAl6gwaBNK6rH/oaQsV5RWPQ==", 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.745 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.b", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlTHnvDhAALYzeqUY0TsSTjgTbeXJo3g==", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.c", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlnnVOOhAAbmFTjHtSqKpvB3kQ4QcbqQ==", 2023-11-24T08:24:56.746 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mds.d", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlDcubHxAAY8UT6alyMvempa5CsE0UgQ==", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:24:56.747 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.0", 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDkWGBlA+toFxAAUMZSLDxHDEw908Fptxl1Sg==", 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.1", 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDnWGBlj7o2MRAAOClP758BSF3G8owqmsIpBQ==", 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.2", 2023-11-24T08:24:56.749 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDrWGBlzgOWDhAAdceyNuKbWC9KHfQV9gwccg==", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.3", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDuWGBlYVchGRAASAcxnj4XM8cRvMQ/uDpOfA==", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.750 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.4", 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD1WGBlMYUdIxAAESIYOw0MqgLoBZokXh3Ilw==", 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.5", 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD5WGBl5lSlARAAHEmx2cOTIhKwnnuHmoTTLg==", 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.6", 2023-11-24T08:24:56.752 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD8WGBlU1MGGRAA1IXSSukrnyzrw/gGJhasUg==", 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "osd.7", 2023-11-24T08:24:56.753 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQD/WGBlIZYKMRAAjNigB6jVOI/mmhZySpYiQA==", 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow profile osd", 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile osd", 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.754 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.0", 2023-11-24T08:24:56.755 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg==", 2023-11-24T08:24:56.755 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.755 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.755 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow r", 2023-11-24T08:24:56.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.1", 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl+R2GBhAA1nlXMiGtGct1dpVYXN/W/g==", 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:24:56.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.2", 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDfWGBlzNsQNxAA3YFcU90QQCiDo9MVHb93bg==", 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.3", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDgWGBl5SK2ERAAVeKXqw70iVEDaI+6nPWKUw==", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow r", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow rw", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow rwx" 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "client.admin", 2023-11-24T08:24:56.758 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDdWGBlTawhHhAALk6nd4ZKZixlS6wPhopO5w==", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": "allow *", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow *", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.x", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlh2V/IRAAQMK1ATTN6qJcGRjHpj9k0A==", 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.759 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "entity": "mgr.y", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "AQDeWGBlebY9FRAAs4z7NfTOs0SdWegag9Ks/g==", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": { 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": "allow *", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": "allow profile mgr", 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": "allow *" 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:56.761 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-11-24T08:24:56.761 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 msgr2=0x7f31d006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 0x7f31d006faf0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f31e0006fd0 tx=0x7f31e0008040 comp rx=0 tx=0).stop 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 msgr2=0x7f31ec0b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f31d802b950 tx=0x7f31d80046c0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 shutdown_connections 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f31d006d630 0x7f31d006faf0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31ec13e0e0 0x7f31ec0aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31ec0708b0 0x7f31ec0b0400 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 --2- 172.21.15.107:0/1842190330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31ec06d0b0 0x7f31ec13bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 >> 172.21.15.107:0/1842190330 conn(0x7f31ec0b4630 msgr2=0x7f31ec06c970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 shutdown_connections 2023-11-24T08:24:56.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.745+0000 7f31f1fff640 1 -- 172.21.15.107:0/1842190330 wait complete. 2023-11-24T08:24:56.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:24:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.969+0000 7fe821d97640 1 Processor -- start 2023-11-24T08:24:56.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.969+0000 7fe821d97640 1 -- start start 2023-11-24T08:24:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe821d97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 0x7fe81c06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe821d97640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe81c0cf450 0x7fe81c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe821d97640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0cece0 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe821d97640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81c156010 con 0x7fe81c0cf450 2023-11-24T08:24:56.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe821d97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c06ce30 con 0x7fe81c15c7f0 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe81bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe81c0cdfb0 0x7fe81c0ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58562/0 (socket says 172.21.15.107:58562) 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.971+0000 7fe81bfff640 1 -- 172.21.15.107:0/1614386046 learned_addr learned my addr 172.21.15.107:0/1614386046 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 0x7fe81c06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81affd640 1 --2- 172.21.15.107:0/1614386046 >> v2:172.21.15.120:3300/0 conn(0x7fe81c0cf450 0x7fe81c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243148199 0 0) 0x7fe81c0cece0 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003520 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811990734 0 0) 0x7fe808003520 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 msgr2=0x7fe81c06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 0x7fe81c06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 >> v2:172.21.15.120:3300/0 conn(0x7fe81c0cf450 msgr2=0x7fe81c0cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 --2- 172.21.15.107:0/1614386046 >> v2:172.21.15.120:3300/0 conn(0x7fe81c0cf450 0x7fe81c0cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c06d040 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fe814001670 con 0x7fe81c0cdfb0 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 >> v1:172.21.15.107:6789/0 conn(0x7fe81c0cdfb0 legacy=0x7fe81c0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe808003970 0x7fe808003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.972+0000 7fe81a7fc640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81a7fc640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 0x7fe808005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c156010 con 0x7fe808003970 2023-11-24T08:24:56.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c06ce30 con 0x7fe8080047c0 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80800b980 con 0x7fe81c15c7f0 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe808003970 0x7fe808003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81affd640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 -- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 msgr2=0x7fe808005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe81c15c7f0 0x7fe808005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 -- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8080047c0 msgr2=0x7fe808004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81b7fe640 1 -- 172.21.15.107:0/1614386046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808003520 con 0x7fe808003970 2023-11-24T08:24:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.973+0000 7fe81affd640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.974+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe808003970 0x7fe808003d90 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009fc0 tx=0x7fe8080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419b5e4ff82d3b5f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c02cc60 con 0x7fe808003970 2023-11-24T08:24:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c00f040 con 0x7fe808003970 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe81a7fc640 1 -- 172.21.15.107:0/1614386046 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c034440 con 0x7fe808003970 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 -- 172.21.15.107:0/1614386046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe808003970 msgr2=0x7fe808003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 --2- 172.21.15.107:0/1614386046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe808003970 0x7fe808003d90 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009fc0 tx=0x7fe8080057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe81bfff640 1 -- 172.21.15.107:0/1614386046 reap_dead start 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 -- 172.21.15.107:0/1614386046 shutdown_connections 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 -- 172.21.15.107:0/1614386046 >> 172.21.15.107:0/1614386046 conn(0x7fe81c06a920 msgr2=0x7fe81c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 -- 172.21.15.107:0/1614386046 shutdown_connections 2023-11-24T08:24:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.975+0000 7fe821d97640 1 -- 172.21.15.107:0/1614386046 wait complete. 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.977+0000 7fe821d97640 1 Processor -- start 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.977+0000 7fe821d97640 1 -- start start 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe808003970 0x7fe81c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe81c154250 0x7fe81c1546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c154c10 con 0x7fe81c156150 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c15aa10 con 0x7fe81c154250 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe821d97640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c15a580 con 0x7fe808003970 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe81affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe81affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43842/0 (socket says 172.21.15.107:43842) 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.978+0000 7fe81affd640 1 -- 172.21.15.107:0/1887514683 learned_addr learned my addr 172.21.15.107:0/1887514683 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 -- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe808003970 msgr2=0x7fe81c14f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe808003970 0x7fe81c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe808003970 0x7fe81c14f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 -- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe81c154250 msgr2=0x7fe81c1546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81bfff640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe81c154250 0x7fe81c1546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe81c154250 0x7fe81c1546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 -- 172.21.15.107:0/1887514683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c159970 con 0x7fe81c156150 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe81affd640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fe804010a30 tx=0x7fe804010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.979+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400ec80 con 0x7fe81c156150 2023-11-24T08:24:56.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.980+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80400ede0 con 0x7fe81c156150 2023-11-24T08:24:56.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.980+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80401c620 con 0x7fe81c156150 2023-11-24T08:24:56.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.980+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c159c50 con 0x7fe81c156150 2023-11-24T08:24:56.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.980+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c15a210 con 0x7fe81c156150 2023-11-24T08:24:56.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.981+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c067470 con 0x7fe81c156150 2023-11-24T08:24:56.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.982+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fe80401c780 con 0x7fe81c156150 2023-11-24T08:24:56.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.983+0000 7fe813fff640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 0x7fe7fc06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:56.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.983+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe804088ef0 con 0x7fe81c156150 2023-11-24T08:24:56.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.983+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 0x7fe7fc06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:56.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.984+0000 7fe81b7fe640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 0x7fe7fc06fb20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe80c036000 tx=0x7fe80c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:56.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:56.987+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe804055a00 con 0x7fe81c156150 2023-11-24T08:24:57.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.179+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe81c15a6c0 con 0x7fe81c156150 2023-11-24T08:24:57.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.181+0000 7fe813fff640 1 -- 172.21.15.107:0/1887514683 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fe804059020 con 0x7fe81c156150 2023-11-24T08:24:57.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:57.183 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":245,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:24:56.233201+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 msgr2=0x7fe7fc06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 0x7fe7fc06fb20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe80c036000 tx=0x7fe80c02e000 comp rx=0 tx=0).stop 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 msgr2=0x7fe81c1565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fe804010a30 tx=0x7fe804010ef0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 shutdown_connections 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe7fc06d660 0x7fe7fc06fb20 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe81c154250 0x7fe81c1546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe81c156150 0x7fe81c1565d0 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 --2- 172.21.15.107:0/1887514683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe808003970 0x7fe81c14f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 >> 172.21.15.107:0/1887514683 conn(0x7fe81c06a920 msgr2=0x7fe81c0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:57.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 shutdown_connections 2023-11-24T08:24:57.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.184+0000 7fe821d97640 1 -- 172.21.15.107:0/1887514683 wait complete. 2023-11-24T08:24:57.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T08:04:43.509673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T08:24:57.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T08:24:57.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.416+0000 7fc07ad05640 1 Processor -- start 2023-11-24T08:24:57.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.416+0000 7fc07ad05640 1 -- start start 2023-11-24T08:24:57.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.418+0000 7fc07ad05640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc0740c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.418+0000 7fc07ad05640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07415eea0 con 0x7fc0740c57e0 2023-11-24T08:24:57.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.418+0000 7fc07ad05640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0740abf30 con 0x7fc07415b440 2023-11-24T08:24:57.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.418+0000 7fc07ad05640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07415b820 con 0x7fc0740cc0f0 2023-11-24T08:24:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0792a2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc0740c57e0 0x7fc07406c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58570/0 (socket says 172.21.15.107:58570) 2023-11-24T08:24:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0792a2640 1 -- 172.21.15.107:0/3031150529 learned_addr learned my addr 172.21.15.107:0/3031150529 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:24:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc073fff640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc0740c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003370726 0 0) 0x7fc07415eea0 con 0x7fc0740c57e0 2023-11-24T08:24:57.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c003520 con 0x7fc0740c57e0 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2571269044 0 0) 0x7fc05c003520 con 0x7fc0740c57e0 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 msgr2=0x7fc0740c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.419+0000 7fc0737fe640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc0740c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 >> v1:172.21.15.120:6789/0 conn(0x7fc07415b440 legacy=0x7fc0740cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07415f0f0 con 0x7fc0740c57e0 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fc06c0019f0 con 0x7fc0740c57e0 2023-11-24T08:24:57.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 >> v1:172.21.15.107:6789/0 conn(0x7fc0740c57e0 legacy=0x7fc07406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc05c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c0047c0 0x7fc05c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc05c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07415b820 con 0x7fc05c0039a0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c00bab0 con 0x7fc05c0047c0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c010080 con 0x7fc0740cc0f0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc05c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc073fff640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c0047c0 0x7fc05c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0792a2640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc05c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 -- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 msgr2=0x7fc05c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:57.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc05c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 -- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c0047c0 msgr2=0x7fc05c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:57.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c0047c0 0x7fc05c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc078aa1640 1 -- 172.21.15.107:0/3031150529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c003520 con 0x7fc05c0039a0 2023-11-24T08:24:57.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.420+0000 7fc0792a2640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0740cc0f0 0x7fc05c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:57.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc05c003d80 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fc068001860 tx=0x7fc05c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deff386a8fa88529 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068033630 con 0x7fc05c0039a0 2023-11-24T08:24:57.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0680337b0 con 0x7fc05c0039a0 2023-11-24T08:24:57.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc0737fe640 1 -- 172.21.15.107:0/3031150529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068033b60 con 0x7fc05c0039a0 2023-11-24T08:24:57.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc07ad05640 1 -- 172.21.15.107:0/3031150529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 msgr2=0x7fc05c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3031150529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc05c003d80 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fc068001860 tx=0x7fc05c0057f0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.422+0000 7fc0792a2640 1 -- 172.21.15.107:0/3031150529 reap_dead start 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.423+0000 7fc07ad05640 1 -- 172.21.15.107:0/3031150529 shutdown_connections 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.423+0000 7fc07ad05640 1 -- 172.21.15.107:0/3031150529 >> 172.21.15.107:0/3031150529 conn(0x7fc0740b0c50 msgr2=0x7fc0740b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.423+0000 7fc07ad05640 1 -- 172.21.15.107:0/3031150529 shutdown_connections 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.423+0000 7fc07ad05640 1 -- 172.21.15.107:0/3031150529 wait complete. 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.424+0000 7fc07ad05640 1 Processor -- start 2023-11-24T08:24:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.424+0000 7fc07ad05640 1 -- start start 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0741461a0 0x7fc0740af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740aff00 0x7fc0741f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740b0380 con 0x7fc05c0039a0 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074158e80 con 0x7fc0740aff00 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.425+0000 7fc07ad05640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0741f36c0 con 0x7fc0741461a0 2023-11-24T08:24:57.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43854/0 (socket says 172.21.15.107:43854) 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 -- 172.21.15.107:0/3811851904 learned_addr learned my addr 172.21.15.107:0/3811851904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc0792a2640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740aff00 0x7fc0741f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc073fff640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0741461a0 0x7fc0740af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 -- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0741461a0 msgr2=0x7fc0740af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0741461a0 0x7fc0740af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 -- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740aff00 msgr2=0x7fc0741f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740aff00 0x7fc0741f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 -- 172.21.15.107:0/3811851904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0741f33e0 con 0x7fc05c0039a0 2023-11-24T08:24:57.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.426+0000 7fc078aa1640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fc068008fd0 tx=0x7fc068001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.427+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0680035f0 con 0x7fc05c0039a0 2023-11-24T08:24:57.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.427+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc068003750 con 0x7fc05c0039a0 2023-11-24T08:24:57.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.427+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068003ac0 con 0x7fc05c0039a0 2023-11-24T08:24:57.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.427+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0740c6170 con 0x7fc05c0039a0 2023-11-24T08:24:57.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.427+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0740c6730 con 0x7fc05c0039a0 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.428+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0740c68f0 con 0x7fc05c0039a0 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.429+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fc068003c20 con 0x7fc05c0039a0 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.430+0000 7fc0717fa640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 0x7fc06006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.430+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0680aa720 con 0x7fc05c0039a0 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.430+0000 7fc073fff640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 0x7fc06006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.431+0000 7fc073fff640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 0x7fc06006faf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fc0740ad4b0 tx=0x7fc064008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.434+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc068077230 con 0x7fc05c0039a0 2023-11-24T08:24:57.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.625+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0740b02e0 con 0x7fc05c0039a0 2023-11-24T08:24:57.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.626+0000 7fc0717fa640 1 -- 172.21.15.107:0/3811851904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v245) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc06802f0a0 con 0x7fc05c0039a0 2023-11-24T08:24:57.628 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 msgr2=0x7fc06006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 0x7fc06006faf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fc0740ad4b0 tx=0x7fc064008040 comp rx=0 tx=0).stop 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 msgr2=0x7fc0740c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fc068008fd0 tx=0x7fc068001230 comp rx=0 tx=0).stop 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 shutdown_connections 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc06006d630 0x7fc06006faf0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740aff00 0x7fc0741f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0741461a0 0x7fc0740af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 --2- 172.21.15.107:0/3811851904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc05c0039a0 0x7fc0740c6c50 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 >> 172.21.15.107:0/3811851904 conn(0x7fc0740b0c50 msgr2=0x7fc0740ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:57.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 shutdown_connections 2023-11-24T08:24:57.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.629+0000 7fc07ad05640 1 -- 172.21.15.107:0/3811851904 wait complete. 2023-11-24T08:24:57.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T08:24:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.857+0000 7fcda2f41640 1 Processor -- start 2023-11-24T08:24:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.858+0000 7fcda2f41640 1 -- start start 2023-11-24T08:24:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcda2f41640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd9c15df80 0x7fcd9c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcda2f41640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd9c0cd470 0x7fcd9c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcda2f41640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c15e3f0 con 0x7fcd9c0b2250 2023-11-24T08:24:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcda2f41640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c15a990 con 0x7fcd9c15df80 2023-11-24T08:24:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcda2f41640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c15ab80 con 0x7fcd9c0cd470 2023-11-24T08:24:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcd9bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd9c15df80 0x7fcd9c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcd9bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd9c15df80 0x7fcd9c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43864/0 (socket says 172.21.15.107:43864) 2023-11-24T08:24:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.859+0000 7fcd9bfff640 1 -- 172.21.15.107:0/1095080322 learned_addr learned my addr 172.21.15.107:0/1095080322 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.860+0000 7fcda14de640 1 --2- 172.21.15.107:0/1095080322 >> v2:172.21.15.120:3300/0 conn(0x7fcd9c0cd470 0x7fcd9c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.860+0000 7fcd9bfff640 1 -- 172.21.15.107:0/1095080322 >> v2:172.21.15.120:3300/0 conn(0x7fcd9c0cd470 msgr2=0x7fcd9c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.860+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/1095080322 >> v2:172.21.15.120:3300/0 conn(0x7fcd9c0cd470 0x7fcd9c15b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.860+0000 7fcd9bfff640 1 -- 172.21.15.107:0/1095080322 >> v1:172.21.15.120:6789/0 conn(0x7fcd9c0b2250 legacy=0x7fcd9c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.860+0000 7fcd9bfff640 1 -- 172.21.15.107:0/1095080322 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0b2d40 con 0x7fcd9c15df80 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.861+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/1095080322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c15df80 0x7fcd9c0ccf30 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0b2fa0 tx=0x7fcd8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ea655d7ba2def5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.861+0000 7fcd9b7fe640 1 -- 172.21.15.107:0/1095080322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88031870 con 0x7fcd9c15df80 2023-11-24T08:24:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.861+0000 7fcd9b7fe640 1 -- 172.21.15.107:0/1095080322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd880319d0 con 0x7fcd9c15df80 2023-11-24T08:24:57.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.861+0000 7fcd9b7fe640 1 -- 172.21.15.107:0/1095080322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88031d60 con 0x7fcd9c15df80 2023-11-24T08:24:57.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 -- 172.21.15.107:0/1095080322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c15df80 msgr2=0x7fcd9c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 --2- 172.21.15.107:0/1095080322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c15df80 0x7fcd9c0ccf30 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0b2fa0 tx=0x7fcd8802b910 comp rx=0 tx=0).stop 2023-11-24T08:24:57.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 -- 172.21.15.107:0/1095080322 shutdown_connections 2023-11-24T08:24:57.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 --2- 172.21.15.107:0/1095080322 >> v2:172.21.15.120:3300/0 conn(0x7fcd9c0cd470 0x7fcd9c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 --2- 172.21.15.107:0/1095080322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c15df80 0x7fcd9c0ccf30 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 -- 172.21.15.107:0/1095080322 >> 172.21.15.107:0/1095080322 conn(0x7fcd9c06a920 msgr2=0x7fcd9c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 -- 172.21.15.107:0/1095080322 shutdown_connections 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.862+0000 7fcda2f41640 1 -- 172.21.15.107:0/1095080322 wait complete. 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.864+0000 7fcda2f41640 1 Processor -- start 2023-11-24T08:24:57.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.864+0000 7fcda2f41640 1 -- start start 2023-11-24T08:24:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 0x7fcd9c0ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd9c15df80 0x7fcd9c1f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ac060 con 0x7fcd9c0b2250 2023-11-24T08:24:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ac340 con 0x7fcd9c15df80 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda2f41640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ac620 con 0x7fcd9c0cd470 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda0cdd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda0cdd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43880/0 (socket says 172.21.15.107:43880) 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.865+0000 7fcda0cdd640 1 -- 172.21.15.107:0/939797683 learned_addr learned my addr 172.21.15.107:0/939797683 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda14de640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd9c15df80 0x7fcd9c1f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 0x7fcd9c0ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 -- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 msgr2=0x7fcd9c0ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 0x7fcd9c0ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 -- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd9c15df80 msgr2=0x7fcd9c1f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd9c15df80 0x7fcd9c1f7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 -- 172.21.15.107:0/939797683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd880094d0 con 0x7fcd9c0b2250 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 0x7fcd9c0ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda0cdd640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000c410 tx=0x7fcd9000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900035f0 con 0x7fcd9c0b2250 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd90003750 con 0x7fcd9c0b2250 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90003b40 con 0x7fcd9c0b2250 2023-11-24T08:24:57.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c0ac900 con 0x7fcd9c0b2250 2023-11-24T08:24:57.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.866+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c0ad640 con 0x7fcd9c0b2250 2023-11-24T08:24:57.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.867+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c0ace20 con 0x7fcd9c0b2250 2023-11-24T08:24:57.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.868+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fcd90003ca0 con 0x7fcd9c0b2250 2023-11-24T08:24:57.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.869+0000 7fcd997fa640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 0x7fcd8006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:57.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.869+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcd90087b50 con 0x7fcd9c0b2250 2023-11-24T08:24:57.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.870+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 0x7fcd8006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:57.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.870+0000 7fcd9bfff640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 0x7fcd8006faf0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcd88036000 tx=0x7fcd8803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:57.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:57.874+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd9c0ace20 con 0x7fcd9c0b2250 2023-11-24T08:24:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.067+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcd9c0ace20 con 0x7fcd9c0b2250 2023-11-24T08:24:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.067+0000 7fcd997fa640 1 -- 172.21.15.107:0/939797683 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v245) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcd9c0ace20 con 0x7fcd9c0b2250 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:24:58.070 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:24:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 msgr2=0x7fcd8006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 0x7fcd8006faf0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcd88036000 tx=0x7fcd8803c040 comp rx=0 tx=0).stop 2023-11-24T08:24:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 msgr2=0x7fcd9c14fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000c410 tx=0x7fcd9000c8d0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 shutdown_connections 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fcd8006d630 0x7fcd8006faf0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd9c15df80 0x7fcd9c1f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd9c0cd470 0x7fcd9c0ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 --2- 172.21.15.107:0/939797683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd9c0b2250 0x7fcd9c14fa20 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 >> 172.21.15.107:0/939797683 conn(0x7fcd9c06a920 msgr2=0x7fcd9c0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 shutdown_connections 2023-11-24T08:24:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.070+0000 7fcda2f41640 1 -- 172.21.15.107:0/939797683 wait complete. 2023-11-24T08:24:58.091 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T08:24:58.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T08:24:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.308+0000 7f73d361c640 1 Processor -- start 2023-11-24T08:24:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.309+0000 7f73d361c640 1 -- start start 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d361c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73cc15df80 0x7f73cc0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d361c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d361c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc15e3f0 con 0x7f73cc0b2250 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d361c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc15a990 con 0x7f73cc15df80 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d361c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc15ab80 con 0x7f73cc0cd470 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73cc15df80 0x7f73cc0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f73cc15df80 0x7f73cc0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60038/0 (socket says 172.21.15.107:60038) 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 -- 172.21.15.107:0/3888107653 learned_addr learned my addr 172.21.15.107:0/3888107653 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d13b8640 1 --1- 172.21.15.107:0/3888107653 >> v1:172.21.15.107:6789/0 conn(0x7f73cc0b2250 0x7f73cc0b2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41048/0 (socket says 172.21.15.107:41048) 2023-11-24T08:24:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 -- 172.21.15.107:0/3888107653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 msgr2=0x7f73cc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d1bb9640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc15b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 -- 172.21.15.107:0/3888107653 >> v1:172.21.15.107:6789/0 conn(0x7f73cc0b2250 legacy=0x7f73cc0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.310+0000 7f73d0bb7640 1 -- 172.21.15.107:0/3888107653 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0b2d40 con 0x7f73cc15df80 2023-11-24T08:24:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73d0bb7640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc15df80 0x7f73cc0ccf30 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708d65b6e3edc798 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73c3fff640 1 -- 172.21.15.107:0/3888107653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc030840 con 0x7f73cc15df80 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73c3fff640 1 -- 172.21.15.107:0/3888107653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc0309a0 con 0x7f73cc15df80 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73c3fff640 1 -- 172.21.15.107:0/3888107653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc030d30 con 0x7f73cc15df80 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73d361c640 1 -- 172.21.15.107:0/3888107653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc15df80 msgr2=0x7f73cc0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.312+0000 7f73d361c640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc15df80 0x7f73cc0ccf30 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc02b980 comp rx=0 tx=0).stop 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 -- 172.21.15.107:0/3888107653 shutdown_connections 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 --2- 172.21.15.107:0/3888107653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc15df80 0x7f73cc0ccf30 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 -- 172.21.15.107:0/3888107653 >> 172.21.15.107:0/3888107653 conn(0x7f73cc06a920 msgr2=0x7f73cc06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 -- 172.21.15.107:0/3888107653 shutdown_connections 2023-11-24T08:24:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.313+0000 7f73d361c640 1 -- 172.21.15.107:0/3888107653 wait complete. 2023-11-24T08:24:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.314+0000 7f73d361c640 1 Processor -- start 2023-11-24T08:24:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.314+0000 7f73d361c640 1 -- start start 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc0af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73cc15df80 0x7f73cc099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b1120 con 0x7f73cc0b2250 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b0080 con 0x7f73cc15df80 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.315+0000 7f73d361c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b0360 con 0x7f73cc0cd470 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60050/0 (socket says 172.21.15.107:60050) 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 -- 172.21.15.107:0/969849468 learned_addr learned my addr 172.21.15.107:0/969849468 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 -- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 msgr2=0x7f73cc0af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc0af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 -- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73cc15df80 msgr2=0x7f73cc099140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73cc15df80 0x7f73cc099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 -- 172.21.15.107:0/969849468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0094d0 con 0x7f73cc0b2250 2023-11-24T08:24:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73d13b8640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f73c800c4d0 tx=0x7f73c800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.316+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8007480 con 0x7f73cc0b2250 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.317+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c80075e0 con 0x7f73cc0b2250 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.317+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80049e0 con 0x7f73cc0b2250 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.317+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc0b0640 con 0x7f73cc0b2250 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.317+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc0b0c00 con 0x7f73cc0b2250 2023-11-24T08:24:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.318+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc0b0df0 con 0x7f73cc0b2250 2023-11-24T08:24:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.319+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f73c8004b40 con 0x7f73cc0b2250 2023-11-24T08:24:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.320+0000 7f73c1ffb640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 0x7f73b006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.320+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f73c8087a90 con 0x7f73cc0b2250 2023-11-24T08:24:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.320+0000 7f73d0bb7640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 0x7f73b006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.321+0000 7f73d0bb7640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 0x7f73b006faf0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f73bc036000 tx=0x7f73bc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.324+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73c8054650 con 0x7f73cc0b2250 2023-11-24T08:24:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:58.514+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f73cc067470 con 0x7f73cc0b2250 2023-11-24T08:24:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.270+0000 7f73c1ffb640 1 -- 172.21.15.107:0/969849468 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v246) v1 ==== 138+0+0 (secure 0 0 0) 0x7f73c8018070 con 0x7f73cc0b2250 2023-11-24T08:24:59.273 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-11-24T08:24:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.274+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 msgr2=0x7f73b006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.274+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 0x7f73b006faf0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f73bc036000 tx=0x7f73bc03b040 comp rx=0 tx=0).stop 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.274+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 msgr2=0x7f73cc153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.274+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f73c800c4d0 tx=0x7f73c800c990 comp rx=0 tx=0).stop 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 shutdown_connections 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f73b006d630 0x7f73b006faf0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73cc15df80 0x7f73cc099140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73cc0cd470 0x7f73cc0af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 --2- 172.21.15.107:0/969849468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73cc0b2250 0x7f73cc153fb0 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 >> 172.21.15.107:0/969849468 conn(0x7f73cc06a920 msgr2=0x7f73cc0adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 shutdown_connections 2023-11-24T08:24:59.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.275+0000 7f73d361c640 1 -- 172.21.15.107:0/969849468 wait complete. 2023-11-24T08:24:59.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T08:24:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.510+0000 7fc7abfff640 1 Processor -- start 2023-11-24T08:24:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.510+0000 7fc7abfff640 1 -- start start 2023-11-24T08:24:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7abfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7abfff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7abfff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0cec30 con 0x7fc7ac0ce480 2023-11-24T08:24:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7abfff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac0ce950 con 0x7fc7ac0c27a0 2023-11-24T08:24:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7abfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0ccdc0 con 0x7fc7ac161810 2023-11-24T08:24:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7aaffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.512+0000 7fc7ab7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7ab7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.107:49072/0 (socket says 172.21.15.107:49072) 2023-11-24T08:24:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7ab7fe640 1 -- 172.21.15.107:0/2190991330 learned_addr learned my addr 172.21.15.107:0/2190991330 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7aaffd640 1 -- 172.21.15.107:0/2190991330 >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 msgr2=0x7fc7ac165380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:24:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7aaffd640 1 --2- 172.21.15.107:0/2190991330 >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7aaffd640 1 -- 172.21.15.107:0/2190991330 >> v1:172.21.15.120:6789/0 conn(0x7fc7ac0ce480 legacy=0x7fc7ac0c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7aaffd640 1 -- 172.21.15.107:0/2190991330 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0cd0a0 con 0x7fc7ac161810 2023-11-24T08:24:59.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.513+0000 7fc7ab7fe640 1 --2- 172.21.15.107:0/2190991330 >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:24:59.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.515+0000 7fc7aaffd640 1 --2- 172.21.15.107:0/2190991330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0cdf40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=badf013aa21a58ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.515+0000 7fc7a9ffb640 1 -- 172.21.15.107:0/2190991330 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c031780 con 0x7fc7ac161810 2023-11-24T08:24:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.515+0000 7fc7a9ffb640 1 -- 172.21.15.107:0/2190991330 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c0318e0 con 0x7fc7ac161810 2023-11-24T08:24:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.515+0000 7fc7a9ffb640 1 -- 172.21.15.107:0/2190991330 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c031c70 con 0x7fc7ac161810 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 -- 172.21.15.107:0/2190991330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 msgr2=0x7fc7ac0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 --2- 172.21.15.107:0/2190991330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0cdf40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b820 comp rx=0 tx=0).stop 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 -- 172.21.15.107:0/2190991330 shutdown_connections 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 --2- 172.21.15.107:0/2190991330 >> v2:172.21.15.120:3300/0 conn(0x7fc7ac0c27a0 0x7fc7ac165380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 --2- 172.21.15.107:0/2190991330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0cdf40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 -- 172.21.15.107:0/2190991330 >> 172.21.15.107:0/2190991330 conn(0x7fc7ac0cbb40 msgr2=0x7fc7ac0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 -- 172.21.15.107:0/2190991330 shutdown_connections 2023-11-24T08:24:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.516+0000 7fc7abfff640 1 -- 172.21.15.107:0/2190991330 wait complete. 2023-11-24T08:24:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.518+0000 7fc7abfff640 1 Processor -- start 2023-11-24T08:24:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.518+0000 7fc7abfff640 1 -- start start 2023-11-24T08:24:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac0c27a0 0x7fc7ac0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0c7440 con 0x7fc7ac0ce480 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0c7ce0 con 0x7fc7ac0c27a0 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7abfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac1f8490 con 0x7fc7ac161810 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7aa7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7aa7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60054/0 (socket says 172.21.15.107:60054) 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.519+0000 7fc7aa7fc640 1 -- 172.21.15.107:0/1163801094 learned_addr learned my addr 172.21.15.107:0/1163801094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aaffd640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac0c27a0 0x7fc7ac0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7ab7fe640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 -- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 msgr2=0x7fc7ac0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 -- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac0c27a0 msgr2=0x7fc7ac0aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:24:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac0c27a0 0x7fc7ac0aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:24:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 -- 172.21.15.107:0/1163801094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a4009b30 con 0x7fc7ac0ce480 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7aa7fc640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c430 tx=0x7fc7a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a00052d0 con 0x7fc7ac0ce480 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a0005430 con 0x7fc7ac0ce480 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c009400 con 0x7fc7ac0ce480 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.520+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0003830 con 0x7fc7ac0ce480 2023-11-24T08:24:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.521+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0c7690 con 0x7fc7ac0ce480 2023-11-24T08:24:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.521+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac0c7e70 con 0x7fc7ac0ce480 2023-11-24T08:24:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.523+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fc7a0003990 con 0x7fc7ac0ce480 2023-11-24T08:24:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.524+0000 7fc793fff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 0x7fc79806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:24:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.525+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc7a00882f0 con 0x7fc7ac0ce480 2023-11-24T08:24:59.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.525+0000 7fc7aaffd640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 0x7fc79806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:24:59.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.525+0000 7fc7aaffd640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 0x7fc79806fb20 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fc79c036000 tx=0x7fc79c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:24:59.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.528+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7a0054f80 con 0x7fc7ac0ce480 2023-11-24T08:24:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:24:59.719+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc7ac04e840 con 0x7fc7ac0ce480 2023-11-24T08:24:59.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2023-11-24T08:25:00.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.283+0000 7fc793fff640 1 -- 172.21.15.107:0/1163801094 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v247) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc7a0017020 con 0x7fc7ac0ce480 2023-11-24T08:25:00.285 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-11-24T08:25:00.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.288+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 msgr2=0x7fc79806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.288+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 0x7fc79806fb20 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fc79c036000 tx=0x7fc79c03d040 comp rx=0 tx=0).stop 2023-11-24T08:25:00.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.288+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 msgr2=0x7fc7ac1f7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.288+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c430 tx=0x7fc7a000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 shutdown_connections 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc79806d660 0x7fc79806fb20 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac161810 0x7fc7ac0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac0ce480 0x7fc7ac1f7f50 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 --2- 172.21.15.107:0/1163801094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac0c27a0 0x7fc7ac0aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 >> 172.21.15.107:0/1163801094 conn(0x7fc7ac0cbb40 msgr2=0x7fc7ac071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:00.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 shutdown_connections 2023-11-24T08:25:00.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.289+0000 7fc7abfff640 1 -- 172.21.15.107:0/1163801094 wait complete. 2023-11-24T08:25:00.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T08:25:00.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.505+0000 7fe0b549b640 1 Processor -- start 2023-11-24T08:25:00.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.506+0000 7fe0b549b640 1 -- start start 2023-11-24T08:25:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.507+0000 7fe0b549b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 0x7fe0b00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.507+0000 7fe0b549b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0b00c57e0 0x7fe0b006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.507+0000 7fe0b549b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b015eea0 con 0x7fe0b015b440 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.507+0000 7fe0b549b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b015b820 con 0x7fe0b00c57e0 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.507+0000 7fe0b549b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00abf30 con 0x7fe0b00cc0f0 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0b00c57e0 0x7fe0b006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0b00c57e0 0x7fe0b006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60066/0 (socket says 172.21.15.107:60066) 2023-11-24T08:25:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 -- 172.21.15.107:0/3641997818 learned_addr learned my addr 172.21.15.107:0/3641997818 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0ae7fc640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 0x7fe0b00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 -- 172.21.15.107:0/3641997818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 msgr2=0x7fe0b00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 0x7fe0b00c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 -- 172.21.15.107:0/3641997818 >> v1:172.21.15.107:6789/0 conn(0x7fe0b015b440 legacy=0x7fe0b00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0af7fe640 1 -- 172.21.15.107:0/3641997818 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b015f0f0 con 0x7fe0b00c57e0 2023-11-24T08:25:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.508+0000 7fe0ae7fc640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 0x7fe0b00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:00.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.509+0000 7fe0af7fe640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b006c5f0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009e50 tx=0x7fe0a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b27b8526cd6b5c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:00.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0adffb640 1 -- 172.21.15.107:0/3641997818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a802ca70 con 0x7fe0b00c57e0 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0adffb640 1 -- 172.21.15.107:0/3641997818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a802cbd0 con 0x7fe0b00c57e0 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0adffb640 1 -- 172.21.15.107:0/3641997818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80343f0 con 0x7fe0b00c57e0 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 -- 172.21.15.107:0/3641997818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 msgr2=0x7fe0b006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b006c5f0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009e50 tx=0x7fe0a802bed0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 -- 172.21.15.107:0/3641997818 shutdown_connections 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b006c5f0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 --2- 172.21.15.107:0/3641997818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b00cc0f0 0x7fe0b00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.510+0000 7fe0b549b640 1 -- 172.21.15.107:0/3641997818 >> 172.21.15.107:0/3641997818 conn(0x7fe0b00b0c50 msgr2=0x7fe0b00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.511+0000 7fe0b549b640 1 -- 172.21.15.107:0/3641997818 shutdown_connections 2023-11-24T08:25:00.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.511+0000 7fe0b549b640 1 -- 172.21.15.107:0/3641997818 wait complete. 2023-11-24T08:25:00.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.512+0000 7fe0b549b640 1 Processor -- start 2023-11-24T08:25:00.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.512+0000 7fe0b549b640 1 -- start start 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b00cc0f0 0x7fe0b015d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b015b440 0x7fe0b0155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01f7b40 con 0x7fe0b00c57e0 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015c9a0 con 0x7fe0b00cc0f0 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.513+0000 7fe0b549b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015cc80 con 0x7fe0b015b440 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60076/0 (socket says 172.21.15.107:60076) 2023-11-24T08:25:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1318153026 learned_addr learned my addr 172.21.15.107:0/1318153026 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0ae7fc640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b00cc0f0 0x7fe0b015d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b015b440 msgr2=0x7fe0b0155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b015b440 0x7fe0b0155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b00cc0f0 msgr2=0x7fe0b015d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b00cc0f0 0x7fe0b015d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:00.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.514+0000 7fe0aeffd640 1 -- 172.21.15.107:0/1318153026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a8009b30 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0aeffd640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000c400 tx=0x7fe0a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a00052d0 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a0005430 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0003830 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b015cf60 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.515+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b00b4690 con 0x7fe0b00c57e0 2023-11-24T08:25:00.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.516+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b015d480 con 0x7fe0b00c57e0 2023-11-24T08:25:00.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.517+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fe0a0003990 con 0x7fe0b00c57e0 2023-11-24T08:25:00.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.518+0000 7fe0a77fe640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 0x7fe09006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:00.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.518+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe0a0087ec0 con 0x7fe0b00c57e0 2023-11-24T08:25:00.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.519+0000 7fe0ae7fc640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 0x7fe09006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:00.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.519+0000 7fe0ae7fc640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 0x7fe09006faf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00707a0 tx=0x7fe098008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.524+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0a0054a70 con 0x7fe0b00c57e0 2023-11-24T08:25:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:00.749+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe0b00b1560 con 0x7fe0b00c57e0 2023-11-24T08:25:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.286+0000 7fe0a77fe640 1 -- 172.21.15.107:0/1318153026 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v153) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe0a0017020 con 0x7fe0b00c57e0 2023-11-24T08:25:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 35 and data pool 36 2023-11-24T08:25:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 msgr2=0x7fe09006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 0x7fe09006faf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00707a0 tx=0x7fe098008040 comp rx=0 tx=0).stop 2023-11-24T08:25:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 msgr2=0x7fe0b00b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000c400 tx=0x7fe0a000c8c0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 shutdown_connections 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fe09006d630 0x7fe09006faf0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0b015b440 0x7fe0b0155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0b00cc0f0 0x7fe0b015d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 --2- 172.21.15.107:0/1318153026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0b00c57e0 0x7fe0b00b4150 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.291+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 >> 172.21.15.107:0/1318153026 conn(0x7fe0b00b0c50 msgr2=0x7fe0b0071ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.292+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 shutdown_connections 2023-11-24T08:25:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.292+0000 7fe0b549b640 1 -- 172.21.15.107:0/1318153026 wait complete. 2023-11-24T08:25:01.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:25:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 Processor -- start 2023-11-24T08:25:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 -- start start 2023-11-24T08:25:01.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f768015c410 0x7f768015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768015bf80 con 0x7f76800b1de0 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800b1560 con 0x7f7680144b30 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.512+0000 7f767f59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76800b1280 con 0x7f768015c410 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f768015c410 0x7f768015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f768015c410 0x7f768015f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60082/0 (socket says 172.21.15.107:60082) 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 -- 172.21.15.107:0/3484401662 learned_addr learned my addr 172.21.15.107:0/3484401662 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 -- 172.21.15.107:0/3484401662 >> v1:172.21.15.120:6789/0 conn(0x7f7680144b30 legacy=0x7f76800b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:01.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 -- 172.21.15.107:0/3484401662 >> v1:172.21.15.107:6789/0 conn(0x7f76800b1de0 legacy=0x7f76800b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.513+0000 7f7677fff640 1 -- 172.21.15.107:0/3484401662 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768015b930 con 0x7f768015c410 2023-11-24T08:25:01.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.514+0000 7f7677fff640 1 --2- 172.21.15.107:0/3484401662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f768015f920 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f7668009870 tx=0x7f766802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f191388123f42fb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767dd9b640 1 -- 172.21.15.107:0/3484401662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668030850 con 0x7f768015c410 2023-11-24T08:25:01.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767dd9b640 1 -- 172.21.15.107:0/3484401662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76680309b0 con 0x7f768015c410 2023-11-24T08:25:01.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767dd9b640 1 -- 172.21.15.107:0/3484401662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668030d40 con 0x7f768015c410 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767f59e640 1 -- 172.21.15.107:0/3484401662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 msgr2=0x7f768015f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767f59e640 1 --2- 172.21.15.107:0/3484401662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f768015f920 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f7668009870 tx=0x7f766802b990 comp rx=0 tx=0).stop 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767f59e640 1 -- 172.21.15.107:0/3484401662 shutdown_connections 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767f59e640 1 --2- 172.21.15.107:0/3484401662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f768015f920 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.515+0000 7f767f59e640 1 -- 172.21.15.107:0/3484401662 >> 172.21.15.107:0/3484401662 conn(0x7f76800cb830 msgr2=0x7f76800cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.516+0000 7f767f59e640 1 -- 172.21.15.107:0/3484401662 shutdown_connections 2023-11-24T08:25:01.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.516+0000 7f767f59e640 1 -- 172.21.15.107:0/3484401662 wait complete. 2023-11-24T08:25:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.517+0000 7f767f59e640 1 Processor -- start 2023-11-24T08:25:01.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.517+0000 7f767f59e640 1 -- start start 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 0x7f7680143cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7680144b30 0x7f76801f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7680142b30 con 0x7f768015c410 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801f6650 con 0x7f76800b1de0 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767f59e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7680144230 con 0x7f7680144b30 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.518+0000 7f767ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60084/0 (socket says 172.21.15.107:60084) 2023-11-24T08:25:01.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 -- 172.21.15.107:0/1087920478 learned_addr learned my addr 172.21.15.107:0/1087920478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 -- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7680144b30 msgr2=0x7f76801f3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767e59c640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 0x7f7680143cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f7677fff640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7680144b30 0x7f76801f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7680144b30 0x7f76801f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 -- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 msgr2=0x7f7680143cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 0x7f7680143cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 -- 172.21.15.107:0/1087920478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680094d0 con 0x7f768015c410 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767e59c640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 0x7f7680143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.519+0000 7f767ed9d640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f767800c6b0 tx=0x7f767800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.520+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678003b70 con 0x7f768015c410 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.520+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7678003cf0 con 0x7f768015c410 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.520+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678005310 con 0x7f768015c410 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.520+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7680141eb0 con 0x7f768015c410 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.520+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7680142470 con 0x7f768015c410 2023-11-24T08:25:01.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.521+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7680142c70 con 0x7f768015c410 2023-11-24T08:25:01.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.522+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f7678005470 con 0x7f768015c410 2023-11-24T08:25:01.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.523+0000 7f7676ffd640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 0x7f766006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.523+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7678088810 con 0x7f768015c410 2023-11-24T08:25:01.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.523+0000 7f767e59c640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 0x7f766006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.524+0000 7f767e59c640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 0x7f766006faf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f767000a830 tx=0x7f7670008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.527+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7680142c70 con 0x7f768015c410 2023-11-24T08:25:01.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.707+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7680142c70 con 0x7f768015c410 2023-11-24T08:25:01.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.710+0000 7f7676ffd640 1 -- 172.21.15.107:0/1087920478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16528 (secure 0 0 0) 0x7f7680142c70 con 0x7f768015c410 2023-11-24T08:25:01.712 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:01.712 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":248,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:25:00.751046+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:24:58.565087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-11-24T08:24:59.776879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:25:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 msgr2=0x7f766006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 0x7f766006faf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f767000a830 tx=0x7f7670008040 comp rx=0 tx=0).stop 2023-11-24T08:25:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 msgr2=0x7f7680142f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f767800c6b0 tx=0x7f767800cb70 comp rx=0 tx=0).stop 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 shutdown_connections 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f766006d630 0x7f766006faf0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f768015c410 0x7f7680142f50 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7680144b30 0x7f76801f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 --2- 172.21.15.107:0/1087920478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76800b1de0 0x7f7680143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 >> 172.21.15.107:0/1087920478 conn(0x7f76800cb830 msgr2=0x7f76800b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 shutdown_connections 2023-11-24T08:25:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.712+0000 7f767f59e640 1 -- 172.21.15.107:0/1087920478 wait complete. 2023-11-24T08:25:01.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:25:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.927+0000 7f56deba9640 1 Processor -- start 2023-11-24T08:25:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.927+0000 7f56deba9640 1 -- start start 2023-11-24T08:25:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56deba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2250 0x7f56d815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56deba9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f56d80b2630 0x7f56d80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56deba9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d815ab80 con 0x7f56d80cd470 2023-11-24T08:25:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56deba9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d815a990 con 0x7f56d80b2630 2023-11-24T08:25:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56deba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d815e680 con 0x7f56d80b2250 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f56d80b2630 0x7f56d80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f56d80b2630 0x7f56d80ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60100/0 (socket says 172.21.15.107:60100) 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 -- 172.21.15.107:0/74499224 learned_addr learned my addr 172.21.15.107:0/74499224 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56dc945640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2250 0x7f56d815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 -- 172.21.15.107:0/74499224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2250 msgr2=0x7f56d815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2250 0x7f56d815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 -- 172.21.15.107:0/74499224 >> v1:172.21.15.120:6789/0 conn(0x7f56d80cd470 legacy=0x7f56d815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.929+0000 7f56d7fff640 1 -- 172.21.15.107:0/74499224 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d815e960 con 0x7f56d80b2630 2023-11-24T08:25:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.931+0000 7f56d7fff640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2630 0x7f56d80ccf30 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f56c400bb10 tx=0x7f56c402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5b06e35cd7bc28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.931+0000 7f56d77fe640 1 -- 172.21.15.107:0/74499224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4036440 con 0x7f56d80b2630 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.931+0000 7f56d77fe640 1 -- 172.21.15.107:0/74499224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40365a0 con 0x7f56d80b2630 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.931+0000 7f56d77fe640 1 -- 172.21.15.107:0/74499224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4036930 con 0x7f56d80b2630 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 -- 172.21.15.107:0/74499224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2630 msgr2=0x7f56d80ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2630 0x7f56d80ccf30 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f56c400bb10 tx=0x7f56c402e150 comp rx=0 tx=0).stop 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 -- 172.21.15.107:0/74499224 shutdown_connections 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2630 0x7f56d80ccf30 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 --2- 172.21.15.107:0/74499224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2250 0x7f56d815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 -- 172.21.15.107:0/74499224 >> 172.21.15.107:0/74499224 conn(0x7f56d806a920 msgr2=0x7f56d806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 -- 172.21.15.107:0/74499224 shutdown_connections 2023-11-24T08:25:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.932+0000 7f56deba9640 1 -- 172.21.15.107:0/74499224 wait complete. 2023-11-24T08:25:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.934+0000 7f56deba9640 1 Processor -- start 2023-11-24T08:25:01.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.934+0000 7f56deba9640 1 -- start start 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2630 0x7f56d8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d80cd470 0x7f56d80ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d815cc60 con 0x7f56d80b2250 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d815c050 con 0x7f56d80cd470 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56deba9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d815c330 con 0x7f56d80b2630 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60114/0 (socket says 172.21.15.107:60114) 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 -- 172.21.15.107:0/710668408 learned_addr learned my addr 172.21.15.107:0/710668408 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 -- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2630 msgr2=0x7f56d8157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2630 0x7f56d8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 -- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d80cd470 msgr2=0x7f56d80ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d80cd470 0x7f56d80ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.935+0000 7f56dc945640 1 -- 172.21.15.107:0/710668408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c400b770 con 0x7f56d80b2250 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56dc945640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00d800 tx=0x7f56cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00b6c0 con 0x7f56d80b2250 2023-11-24T08:25:01.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc00b820 con 0x7f56d80b2250 2023-11-24T08:25:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00bbb0 con 0x7f56d80b2250 2023-11-24T08:25:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d815c610 con 0x7f56d80b2250 2023-11-24T08:25:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.936+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d80caeb0 con 0x7f56d80b2250 2023-11-24T08:25:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.937+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d8067470 con 0x7f56d80b2250 2023-11-24T08:25:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.938+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f56cc00bd10 con 0x7f56d80b2250 2023-11-24T08:25:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.939+0000 7f56d57fa640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 0x7f56bc06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.939+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56cc088ee0 con 0x7f56d80b2250 2023-11-24T08:25:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.939+0000 7f56d7fff640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 0x7f56bc06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.940+0000 7f56d7fff640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 0x7f56bc06faf0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f56c4038000 tx=0x7f56c4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:01.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:01.944+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56cc055a90 con 0x7f56d80b2250 2023-11-24T08:25:02.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.121+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56d80cea00 con 0x7f56d80b2250 2023-11-24T08:25:02.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.123+0000 7f56d57fa640 1 -- 172.21.15.107:0/710668408 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16528 (secure 0 0 0) 0x7f56cc0590b0 con 0x7f56d80b2250 2023-11-24T08:25:02.126 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:02.126 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":248,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:25:00.751046+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:24:58.565087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-11-24T08:24:59.776879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 msgr2=0x7f56bc06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 0x7f56bc06faf0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f56c4038000 tx=0x7f56c4030000 comp rx=0 tx=0).stop 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 msgr2=0x7f56d81f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00d800 tx=0x7f56cc00dcc0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 shutdown_connections 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f56bc06d630 0x7f56bc06faf0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d80cd470 0x7f56d80ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56d80b2630 0x7f56d8157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 --2- 172.21.15.107:0/710668408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56d80b2250 0x7f56d81f3100 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 >> 172.21.15.107:0/710668408 conn(0x7f56d806a920 msgr2=0x7f56d8158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 shutdown_connections 2023-11-24T08:25:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.126+0000 7f56deba9640 1 -- 172.21.15.107:0/710668408 wait complete. 2023-11-24T08:25:02.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T08:25:02.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.343+0000 7f00971db640 1 Processor -- start 2023-11-24T08:25:02.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.344+0000 7f00971db640 1 -- start start 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f009015d3b0 0x7f0090072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d790 0x7f009006fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00900aefb0 0x7f00900b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900b4020 con 0x7f009015d3b0 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900af440 con 0x7f00900aefb0 2023-11-24T08:25:02.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00971db640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009006e9c0 con 0x7f009015d790 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f009015d3b0 0x7f0090072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f009015d3b0 0x7f0090072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60128/0 (socket says 172.21.15.107:60128) 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 -- 172.21.15.107:0/1400147161 learned_addr learned my addr 172.21.15.107:0/1400147161 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 -- 172.21.15.107:0/1400147161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d790 msgr2=0x7f009006fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 --2- 172.21.15.107:0/1400147161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d790 0x7f009006fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 -- 172.21.15.107:0/1400147161 >> v2:172.21.15.120:3300/0 conn(0x7f00900aefb0 msgr2=0x7f00900b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 --2- 172.21.15.107:0/1400147161 >> v2:172.21.15.120:3300/0 conn(0x7f00900aefb0 0x7f00900b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.345+0000 7f00961d9640 1 -- 172.21.15.107:0/1400147161 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009006eca0 con 0x7f009015d3b0 2023-11-24T08:25:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.346+0000 7f00961d9640 1 --2- 172.21.15.107:0/1400147161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d3b0 0x7f0090072590 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f00900cc590 tx=0x7f008c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9f01c46ddf76ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00951d7640 1 -- 172.21.15.107:0/1400147161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c031870 con 0x7f009015d3b0 2023-11-24T08:25:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00951d7640 1 -- 172.21.15.107:0/1400147161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c0319d0 con 0x7f009015d3b0 2023-11-24T08:25:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00951d7640 1 -- 172.21.15.107:0/1400147161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c031d60 con 0x7f009015d3b0 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 -- 172.21.15.107:0/1400147161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d3b0 msgr2=0x7f0090072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 --2- 172.21.15.107:0/1400147161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d3b0 0x7f0090072590 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f00900cc590 tx=0x7f008c02b910 comp rx=0 tx=0).stop 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 -- 172.21.15.107:0/1400147161 shutdown_connections 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 --2- 172.21.15.107:0/1400147161 >> v2:172.21.15.120:3300/0 conn(0x7f00900aefb0 0x7f00900b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 --2- 172.21.15.107:0/1400147161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d790 0x7f009006fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 --2- 172.21.15.107:0/1400147161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d3b0 0x7f0090072590 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 -- 172.21.15.107:0/1400147161 >> 172.21.15.107:0/1400147161 conn(0x7f009006d6c0 msgr2=0x7f009006daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 -- 172.21.15.107:0/1400147161 shutdown_connections 2023-11-24T08:25:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.347+0000 7f00971db640 1 -- 172.21.15.107:0/1400147161 wait complete. 2023-11-24T08:25:02.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.349+0000 7f00971db640 1 Processor -- start 2023-11-24T08:25:02.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.349+0000 7f00971db640 1 -- start start 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00900aefb0 0x7f009015faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 0x7f0090156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0090160030 con 0x7f009015d790 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900cc5e0 con 0x7f00900aefb0 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00971db640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900cbe30 con 0x7f009015d3b0 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00969da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00969da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60132/0 (socket says 172.21.15.107:60132) 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.350+0000 7f00969da640 1 -- 172.21.15.107:0/1900516326 learned_addr learned my addr 172.21.15.107:0/1900516326 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 -- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 msgr2=0x7f0090156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00961d9640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00900aefb0 0x7f009015faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00959d8640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 0x7f0090156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 0x7f0090156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 -- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00900aefb0 msgr2=0x7f009015faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00900aefb0 0x7f009015faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 -- 172.21.15.107:0/1900516326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0094d0 con 0x7f009015d790 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00959d8640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 0x7f0090156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00969da640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f008800b9a0 tx=0x7f008800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008800cba0 con 0x7f009015d790 2023-11-24T08:25:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008800cd20 con 0x7f009015d790 2023-11-24T08:25:02.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088014680 con 0x7f009015d790 2023-11-24T08:25:02.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.351+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0090160220 con 0x7f009015d790 2023-11-24T08:25:02.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.352+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00901607e0 con 0x7f009015d790 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.353+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009004e840 con 0x7f009015d790 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.353+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f008801d3b0 con 0x7f009015d790 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.354+0000 7f0086ffd640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 0x7f007406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.354+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0088088980 con 0x7f009015d790 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.355+0000 7f00961d9640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 0x7f007406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:02.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.355+0000 7f00961d9640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 0x7f007406faf0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f008c036000 tx=0x7f008c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:02.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.359+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0088055490 con 0x7f009015d790 2023-11-24T08:25:02.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:02.573+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f009015c480 con 0x7f009015d790 2023-11-24T08:25:03.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.345+0000 7f0086ffd640 1 -- 172.21.15.107:0/1900516326 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v156) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0088019020 con 0x7f009015d790 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 msgr2=0x7f007406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 0x7f007406faf0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f008c036000 tx=0x7f008c03c040 comp rx=0 tx=0).stop 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 msgr2=0x7f009015acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f008800b9a0 tx=0x7f008800be60 comp rx=0 tx=0).stop 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 shutdown_connections 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f007406d630 0x7f007406faf0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f009015d790 0x7f009015acf0 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f009015d3b0 0x7f0090156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 --2- 172.21.15.107:0/1900516326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00900aefb0 0x7f009015faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 >> 172.21.15.107:0/1900516326 conn(0x7f009006d6c0 msgr2=0x7f00900ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:03.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 shutdown_connections 2023-11-24T08:25:03.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.349+0000 7f00971db640 1 -- 172.21.15.107:0/1900516326 wait complete. 2023-11-24T08:25:03.370 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T08:25:03.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T08:25:03.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.581+0000 7f554fe1e640 1 Processor -- start 2023-11-24T08:25:03.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.582+0000 7f554fe1e640 1 -- start start 2023-11-24T08:25:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554fe1e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55480cc0f0 0x7f55480c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554fe1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55480c57e0 0x7f554806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554fe1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554815eea0 con 0x7f554815b440 2023-11-24T08:25:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554fe1e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554815b820 con 0x7f55480c57e0 2023-11-24T08:25:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554fe1e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55480abf30 con 0x7f55480cc0f0 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.583+0000 7f554e3bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55480c57e0 0x7f554806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f55480c57e0 0x7f554806c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60146/0 (socket says 172.21.15.107:60146) 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 -- 172.21.15.107:0/2245941234 learned_addr learned my addr 172.21.15.107:0/2245941234 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554dbba640 1 --1- 172.21.15.107:0/2245941234 >> v1:172.21.15.107:6789/0 conn(0x7f554815b440 0x7f55480cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41068/0 (socket says 172.21.15.107:41068) 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 -- 172.21.15.107:0/2245941234 >> v2:172.21.15.120:3300/0 conn(0x7f55480cc0f0 msgr2=0x7f55480c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:03.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 --2- 172.21.15.107:0/2245941234 >> v2:172.21.15.120:3300/0 conn(0x7f55480cc0f0 0x7f55480c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 -- 172.21.15.107:0/2245941234 >> v1:172.21.15.107:6789/0 conn(0x7f554815b440 legacy=0x7f55480cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.584+0000 7f554e3bb640 1 -- 172.21.15.107:0/2245941234 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554815f0f0 con 0x7f55480c57e0 2023-11-24T08:25:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.585+0000 7f554e3bb640 1 --2- 172.21.15.107:0/2245941234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55480c57e0 0x7f554806c5f0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f554806e720 tx=0x7f554002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af351e52d3e2814 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.585+0000 7f554cbb8640 1 -- 172.21.15.107:0/2245941234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540031870 con 0x7f55480c57e0 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.585+0000 7f554cbb8640 1 -- 172.21.15.107:0/2245941234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55400319d0 con 0x7f55480c57e0 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.585+0000 7f554cbb8640 1 -- 172.21.15.107:0/2245941234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540031d60 con 0x7f55480c57e0 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 -- 172.21.15.107:0/2245941234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55480c57e0 msgr2=0x7f554806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 --2- 172.21.15.107:0/2245941234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55480c57e0 0x7f554806c5f0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f554806e720 tx=0x7f554002b910 comp rx=0 tx=0).stop 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 -- 172.21.15.107:0/2245941234 shutdown_connections 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 --2- 172.21.15.107:0/2245941234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55480c57e0 0x7f554806c5f0 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 --2- 172.21.15.107:0/2245941234 >> v2:172.21.15.120:3300/0 conn(0x7f55480cc0f0 0x7f55480c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 -- 172.21.15.107:0/2245941234 >> 172.21.15.107:0/2245941234 conn(0x7f55480b0c50 msgr2=0x7f55480b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 -- 172.21.15.107:0/2245941234 shutdown_connections 2023-11-24T08:25:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.586+0000 7f554fe1e640 1 -- 172.21.15.107:0/2245941234 wait complete. 2023-11-24T08:25:03.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.588+0000 7f554fe1e640 1 Processor -- start 2023-11-24T08:25:03.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.588+0000 7f554fe1e640 1 -- start start 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55480c57e0 0x7f55481547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55480cc0f0 0x7f554815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554809bce0 con 0x7f554815b440 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554809bfc0 con 0x7f55480c57e0 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554fe1e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554809c2a0 con 0x7f55480cc0f0 2023-11-24T08:25:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60154/0 (socket says 172.21.15.107:60154) 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 -- 172.21.15.107:0/4077830171 learned_addr learned my addr 172.21.15.107:0/4077830171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 -- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55480cc0f0 msgr2=0x7f554815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55480cc0f0 0x7f554815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 -- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55480c57e0 msgr2=0x7f55481547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55480c57e0 0x7f55481547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.590+0000 7f554e3bb640 1 -- 172.21.15.107:0/4077830171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55400094d0 con 0x7f554815b440 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f554e3bb640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7f554002b950 tx=0x7f55400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55400443f0 con 0x7f554815b440 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5540044570 con 0x7f554815b440 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554809c580 con 0x7f554815b440 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540044920 con 0x7f554815b440 2023-11-24T08:25:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.591+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5548154ce0 con 0x7f554815b440 2023-11-24T08:25:03.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.592+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554804e8b0 con 0x7f554815b440 2023-11-24T08:25:03.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.593+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f5540044a80 con 0x7f554815b440 2023-11-24T08:25:03.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.595+0000 7f55467fc640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 0x7f552c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:03.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.595+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f554002d040 con 0x7f554815b440 2023-11-24T08:25:03.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.595+0000 7f554dbba640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 0x7f552c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:03.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.596+0000 7f554dbba640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 0x7f552c06faf0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f553c007fd0 tx=0x7f553c007d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:03.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.599+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5540077e70 con 0x7f554815b440 2023-11-24T08:25:03.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:03.829+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f55480aee40 con 0x7f554815b440 2023-11-24T08:25:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.358+0000 7f55467fc640 1 -- 172.21.15.107:0/4077830171 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v157) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5540035070 con 0x7f554815b440 2023-11-24T08:25:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.362+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 msgr2=0x7f552c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.362+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 0x7f552c06faf0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f553c007fd0 tx=0x7f553c007d40 comp rx=0 tx=0).stop 2023-11-24T08:25:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.362+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 msgr2=0x7f55480cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.362+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7f554002b950 tx=0x7f55400046c0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.362+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 shutdown_connections 2023-11-24T08:25:04.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f552c06d630 0x7f552c06faf0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554815b440 0x7f55480cdc50 unknown :-1 s=CLOSED pgs=3379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55480cc0f0 0x7f554815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 --2- 172.21.15.107:0/4077830171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55480c57e0 0x7f55481547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 >> 172.21.15.107:0/4077830171 conn(0x7f55480b0c50 msgr2=0x7f5548071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 shutdown_connections 2023-11-24T08:25:04.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.363+0000 7f554fe1e640 1 -- 172.21.15.107:0/4077830171 wait complete. 2023-11-24T08:25:04.383 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T08:25:04.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:25:04.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.599+0000 7fad5cce1640 1 Processor -- start 2023-11-24T08:25:04.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.599+0000 7fad5cce1640 1 -- start start 2023-11-24T08:25:04.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad58144b30 0x7fad5815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5815c410 0x7fad580b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad580b1de0 0x7fad580b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad580b27b0 con 0x7fad580b1de0 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5815bf80 con 0x7fad58144b30 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.600+0000 7fad5cce1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad580b1560 con 0x7fad5815c410 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad580b1de0 0x7fad580b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad580b1de0 0x7fad580b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60162/0 (socket says 172.21.15.107:60162) 2023-11-24T08:25:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 -- 172.21.15.107:0/1835140919 learned_addr learned my addr 172.21.15.107:0/1835140919 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 -- 172.21.15.107:0/1835140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5815c410 msgr2=0x7fad580b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 --2- 172.21.15.107:0/1835140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5815c410 0x7fad580b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 -- 172.21.15.107:0/1835140919 >> v2:172.21.15.120:3300/0 conn(0x7fad58144b30 msgr2=0x7fad5815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 --2- 172.21.15.107:0/1835140919 >> v2:172.21.15.120:3300/0 conn(0x7fad58144b30 0x7fad5815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.601+0000 7fad57fff640 1 -- 172.21.15.107:0/1835140919 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580b0fb0 con 0x7fad580b1de0 2023-11-24T08:25:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.602+0000 7fad57fff640 1 --2- 172.21.15.107:0/1835140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad580b1de0 0x7fad580b2270 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7fad5806f690 tx=0x7fad5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2789b084e321c14d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad56ffd640 1 -- 172.21.15.107:0/1835140919 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5002ca80 con 0x7fad580b1de0 2023-11-24T08:25:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad56ffd640 1 -- 172.21.15.107:0/1835140919 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5002cbe0 con 0x7fad580b1de0 2023-11-24T08:25:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad56ffd640 1 -- 172.21.15.107:0/1835140919 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad500343f0 con 0x7fad580b1de0 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 -- 172.21.15.107:0/1835140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad580b1de0 msgr2=0x7fad580b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1835140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad580b1de0 0x7fad580b2270 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7fad5806f690 tx=0x7fad5002bee0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 -- 172.21.15.107:0/1835140919 shutdown_connections 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1835140919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad580b1de0 0x7fad580b2270 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1835140919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5815c410 0x7fad580b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1835140919 >> v2:172.21.15.120:3300/0 conn(0x7fad58144b30 0x7fad5815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 -- 172.21.15.107:0/1835140919 >> 172.21.15.107:0/1835140919 conn(0x7fad580cb830 msgr2=0x7fad580cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 -- 172.21.15.107:0/1835140919 shutdown_connections 2023-11-24T08:25:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.603+0000 7fad5cce1640 1 -- 172.21.15.107:0/1835140919 wait complete. 2023-11-24T08:25:04.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.605+0000 7fad5cce1640 1 Processor -- start 2023-11-24T08:25:04.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.605+0000 7fad5cce1640 1 -- start start 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad580b1de0 0x7fad5813b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad58144b30 0x7fad5815a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58139e30 con 0x7fad5815c410 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5813c0a0 con 0x7fad580b1de0 2023-11-24T08:25:04.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.606+0000 7fad5cce1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5813c380 con 0x7fad58144b30 2023-11-24T08:25:04.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60174/0 (socket says 172.21.15.107:60174) 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 -- 172.21.15.107:0/1235296474 learned_addr learned my addr 172.21.15.107:0/1235296474 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 -- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad58144b30 msgr2=0x7fad5815a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad58144b30 0x7fad5815a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 -- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad580b1de0 msgr2=0x7fad5813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad580b1de0 0x7fad5813b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 -- 172.21.15.107:0/1235296474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad50009b30 con 0x7fad5815c410 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.607+0000 7fad57fff640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fad50036040 tx=0x7fad50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.608+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5003e450 con 0x7fad5815c410 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.608+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5003e5d0 con 0x7fad5815c410 2023-11-24T08:25:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.608+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5003e980 con 0x7fad5815c410 2023-11-24T08:25:04.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.608+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad58139770 con 0x7fad5815c410 2023-11-24T08:25:04.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.608+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5813bce0 con 0x7fad5815c410 2023-11-24T08:25:04.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.609+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5804e840 con 0x7fad5815c410 2023-11-24T08:25:04.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.610+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fad5002cd50 con 0x7fad5815c410 2023-11-24T08:25:04.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.611+0000 7fad54ff9640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 0x7fad3806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:04.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.611+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad500ab610 con 0x7fad5815c410 2023-11-24T08:25:04.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.611+0000 7fad577fe640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 0x7fad3806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:04.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.612+0000 7fad577fe640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 0x7fad3806faf0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fad48009710 tx=0x7fad48006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.615+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad500782f0 con 0x7fad5815c410 2023-11-24T08:25:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.841+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5806d860 con 0x7fad5815c410 2023-11-24T08:25:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.843+0000 7fad54ff9640 1 -- 172.21.15.107:0/1235296474 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fad50043090 con 0x7fad5815c410 2023-11-24T08:25:04.844 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:04.844 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":157,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2856897517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2856897517},{"type":"v1","addr":"172.21.15.120:6833","nonce":2856897517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3120363367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3120363367},{"type":"v1","addr":"172.21.15.120:6835","nonce":3120363367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151},{"gid":13284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1553701545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1553701545},{"type":"v1","addr":"172.21.15.107:6835","nonce":1553701545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:25:00.751585+0000","modified":"2023-11-24T08:25:03.830324+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13293":{"gid":13293,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/548967818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":548967818},{"type":"v1","addr":"172.21.15.107:6837","nonce":548967818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-11-24T08:25:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 157 2023-11-24T08:25:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 msgr2=0x7fad3806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 0x7fad3806faf0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fad48009710 tx=0x7fad48006c90 comp rx=0 tx=0).stop 2023-11-24T08:25:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 msgr2=0x7fad580c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fad50036040 tx=0x7fad50004a40 comp rx=0 tx=0).stop 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 shutdown_connections 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fad3806d630 0x7fad3806faf0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad5815c410 0x7fad580c4b90 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad58144b30 0x7fad5815a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 --2- 172.21.15.107:0/1235296474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad580b1de0 0x7fad5813b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.845+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 >> 172.21.15.107:0/1235296474 conn(0x7fad580cb830 msgr2=0x7fad5815d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.846+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 shutdown_connections 2023-11-24T08:25:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:04.846+0000 7fad5cce1640 1 -- 172.21.15.107:0/1235296474 wait complete. 2023-11-24T08:25:04.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T08:25:05.067 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-11-24T08:25:05.067+0000 7f506fcdc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T08:25:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.074+0000 7fd07a127640 1 Processor -- start 2023-11-24T08:25:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.074+0000 7fd07a127640 1 -- start start 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd07a127640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd07415c410 0x7fd07415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd07a127640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0740b1de0 0x7fd0740b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd07a127640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07415bf80 con 0x7fd074144b30 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd07a127640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740b27b0 con 0x7fd07415c410 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd07a127640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740b1560 con 0x7fd0740b1de0 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd078924640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd07415c410 0x7fd07415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd078924640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd07415c410 0x7fd07415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60184/0 (socket says 172.21.15.107:60184) 2023-11-24T08:25:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd078924640 1 -- 172.21.15.107:0/3484432105 learned_addr learned my addr 172.21.15.107:0/3484432105 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.076+0000 7fd079926640 1 --2- 172.21.15.107:0/3484432105 >> v2:172.21.15.120:3300/0 conn(0x7fd0740b1de0 0x7fd0740b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.077+0000 7fd078924640 1 -- 172.21.15.107:0/3484432105 >> v2:172.21.15.120:3300/0 conn(0x7fd0740b1de0 msgr2=0x7fd0740b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.077+0000 7fd078924640 1 --2- 172.21.15.107:0/3484432105 >> v2:172.21.15.120:3300/0 conn(0x7fd0740b1de0 0x7fd0740b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.077+0000 7fd078924640 1 -- 172.21.15.107:0/3484432105 >> v1:172.21.15.120:6789/0 conn(0x7fd074144b30 legacy=0x7fd0740b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.077+0000 7fd078924640 1 -- 172.21.15.107:0/3484432105 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0740b1280 con 0x7fd07415c410 2023-11-24T08:25:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.078+0000 7fd078924640 1 --2- 172.21.15.107:0/3484432105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd07415c410 0x7fd07415f920 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7fd074072a50 tx=0x7fd06402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d0c1e817959251 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.078+0000 7fd06bfff640 1 -- 172.21.15.107:0/3484432105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064031870 con 0x7fd07415c410 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.078+0000 7fd06bfff640 1 -- 172.21.15.107:0/3484432105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0640319d0 con 0x7fd07415c410 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.078+0000 7fd06bfff640 1 -- 172.21.15.107:0/3484432105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064031d60 con 0x7fd07415c410 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 -- 172.21.15.107:0/3484432105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd07415c410 msgr2=0x7fd07415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 --2- 172.21.15.107:0/3484432105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd07415c410 0x7fd07415f920 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7fd074072a50 tx=0x7fd06402b910 comp rx=0 tx=0).stop 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 -- 172.21.15.107:0/3484432105 shutdown_connections 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 --2- 172.21.15.107:0/3484432105 >> v2:172.21.15.120:3300/0 conn(0x7fd0740b1de0 0x7fd0740b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 --2- 172.21.15.107:0/3484432105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd07415c410 0x7fd07415f920 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 -- 172.21.15.107:0/3484432105 >> 172.21.15.107:0/3484432105 conn(0x7fd0740cb830 msgr2=0x7fd0740cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 -- 172.21.15.107:0/3484432105 shutdown_connections 2023-11-24T08:25:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.079+0000 7fd07a127640 1 -- 172.21.15.107:0/3484432105 wait complete. 2023-11-24T08:25:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.081+0000 7fd07a127640 1 Processor -- start 2023-11-24T08:25:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.081+0000 7fd07a127640 1 -- start start 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd074144b30 0x7fd074130810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd07415c410 0x7fd0740ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd074131110 con 0x7fd0740b1de0 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0741313f0 con 0x7fd074144b30 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd07a127640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0741316d0 con 0x7fd07415c410 2023-11-24T08:25:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60186/0 (socket says 172.21.15.107:60186) 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 -- 172.21.15.107:0/3981503843 learned_addr learned my addr 172.21.15.107:0/3981503843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 -- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd07415c410 msgr2=0x7fd0740ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd07415c410 0x7fd0740ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 -- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd074144b30 msgr2=0x7fd074130810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd074144b30 0x7fd074130810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.082+0000 7fd079125640 1 -- 172.21.15.107:0/3981503843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0640094d0 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd079125640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c430 tx=0x7fd07000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011790 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0700118f0 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011ce0 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0741319b0 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.083+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd074130d50 con 0x7fd0740b1de0 2023-11-24T08:25:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.084+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd074131ed0 con 0x7fd0740b1de0 2023-11-24T08:25:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.085+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fd07001c460 con 0x7fd0740b1de0 2023-11-24T08:25:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.087+0000 7fd069ffb640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 0x7fd05806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.087+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0700889d0 con 0x7fd0740b1de0 2023-11-24T08:25:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.087+0000 7fd078924640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 0x7fd05806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.088+0000 7fd078924640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 0x7fd05806faf0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd064036000 tx=0x7fd064030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.091+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0700c0e90 con 0x7fd0740b1de0 2023-11-24T08:25:05.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.283+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd07404e840 con 0x7fd0740b1de0 2023-11-24T08:25:05.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.285+0000 7fd069ffb640 1 -- 172.21.15.107:0/3981503843 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+16515 (secure 0 0 0) 0x7fd074131ed0 con 0x7fd0740b1de0 2023-11-24T08:25:05.286 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:05.286 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":249,"fsid":"a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d","created":"2023-11-24T08:04:29.847487+0000","modified":"2023-11-24T08:25:02.305133+0000","last_up_change":"2023-11-24T08:04:42.719409+0000","last_in_change":"2023-11-24T08:04:33.323479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:04:43.509673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-11-24T08:24:58.565087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-11-24T08:24:59.776879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a391934b-b663-4d35-bf9a-98364789c9c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6809","nonce":1814653023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6811","nonce":1814653023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6815","nonce":1814653023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1814653023},{"type":"v1","addr":"172.21.15.107:6813","nonce":1814653023}]},"public_addr":"172.21.15.107:6809/1814653023","cluster_addr":"172.21.15.107:6811/1814653023","heartbeat_back_addr":"172.21.15.107:6815/1814653023","heartbeat_front_addr":"172.21.15.107:6813/1814653023","state":["exists","up"]},{"osd":1,"uuid":"428f478c-5a65-4cf2-9e5d-9ed1ece89179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6801","nonce":132161644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6803","nonce":132161644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6808","nonce":132161644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":132161644},{"type":"v1","addr":"172.21.15.107:6805","nonce":132161644}]},"public_addr":"172.21.15.107:6801/132161644","cluster_addr":"172.21.15.107:6803/132161644","heartbeat_back_addr":"172.21.15.107:6808/132161644","heartbeat_front_addr":"172.21.15.107:6805/132161644","state":["exists","up"]},{"osd":2,"uuid":"c404c5f0-390b-4849-8786-3712dec22fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6817","nonce":4260350828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6822","nonce":4260350828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6831","nonce":4260350828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6825","nonce":4260350828},{"type":"v1","addr":"172.21.15.107:6827","nonce":4260350828}]},"public_addr":"172.21.15.107:6817/4260350828","cluster_addr":"172.21.15.107:6822/4260350828","heartbeat_back_addr":"172.21.15.107:6831/4260350828","heartbeat_front_addr":"172.21.15.107:6827/4260350828","state":["exists","up"]},{"osd":3,"uuid":"2b920ea1-1b8c-476e-a7be-cd01c78a93b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6819","nonce":1213738659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6821","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6823","nonce":1213738659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6829","nonce":1213738659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1213738659},{"type":"v1","addr":"172.21.15.107:6826","nonce":1213738659}]},"public_addr":"172.21.15.107:6819/1213738659","cluster_addr":"172.21.15.107:6823/1213738659","heartbeat_back_addr":"172.21.15.107:6829/1213738659","heartbeat_front_addr":"172.21.15.107:6826/1213738659","state":["exists","up"]},{"osd":4,"uuid":"0a932040-69ef-4757-9e06-f3e1772c9426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6801","nonce":239423732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6807","nonce":239423732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6815","nonce":239423732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6809","nonce":239423732},{"type":"v1","addr":"172.21.15.120:6812","nonce":239423732}]},"public_addr":"172.21.15.120:6801/239423732","cluster_addr":"172.21.15.120:6807/239423732","heartbeat_back_addr":"172.21.15.120:6815/239423732","heartbeat_front_addr":"172.21.15.120:6812/239423732","state":["exists","up"]},{"osd":5,"uuid":"dba0f2ab-c900-4ee4-a501-ae62eb3b57ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6803","nonce":3047087772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6805","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6806","nonce":3047087772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6813","nonce":3047087772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3047087772},{"type":"v1","addr":"172.21.15.120:6810","nonce":3047087772}]},"public_addr":"172.21.15.120:6803/3047087772","cluster_addr":"172.21.15.120:6806/3047087772","heartbeat_back_addr":"172.21.15.120:6813/3047087772","heartbeat_front_addr":"172.21.15.120:6810/3047087772","state":["exists","up"]},{"osd":6,"uuid":"307fd4c8-bea5-4c0f-a3f0-786e9e8728e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6817","nonce":498313091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6821","nonce":498313091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6831","nonce":498313091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":498313091},{"type":"v1","addr":"172.21.15.120:6827","nonce":498313091}]},"public_addr":"172.21.15.120:6817/498313091","cluster_addr":"172.21.15.120:6821/498313091","heartbeat_back_addr":"172.21.15.120:6831/498313091","heartbeat_front_addr":"172.21.15.120:6827/498313091","state":["exists","up"]},{"osd":7,"uuid":"e0953d37-b98c-44e9-9f1e-cad4370876fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6819","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6820","nonce":46675067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6823","nonce":46675067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6829","nonce":46675067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6825","nonce":46675067},{"type":"v1","addr":"172.21.15.120:6826","nonce":46675067}]},"public_addr":"172.21.15.120:6820/46675067","cluster_addr":"172.21.15.120:6823/46675067","heartbeat_back_addr":"172.21.15.120:6829/46675067","heartbeat_front_addr":"172.21.15.120:6826/46675067","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.868808+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.614505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.904641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.725642+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.763851+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.766262+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.741401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:04:38.767704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:25:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.287+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 msgr2=0x7fd05806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.287+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 0x7fd05806faf0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd064036000 tx=0x7fd064030000 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.287+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 msgr2=0x7fd0741321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.287+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c430 tx=0x7fd07000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 shutdown_connections 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd05806d630 0x7fd05806faf0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd07415c410 0x7fd0740ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd074144b30 0x7fd074130810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 --2- 172.21.15.107:0/3981503843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0740b1de0 0x7fd0741321b0 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 >> 172.21.15.107:0/3981503843 conn(0x7fd0740cb830 msgr2=0x7fd0740b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 shutdown_connections 2023-11-24T08:25:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.288+0000 7fd07a127640 1 -- 172.21.15.107:0/3981503843 wait complete. 2023-11-24T08:25:05.307 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T08:25:05.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:25:05.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.513+0000 7f7b304ee640 1 Processor -- start 2023-11-24T08:25:05.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.514+0000 7f7b304ee640 1 -- start start 2023-11-24T08:25:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b304ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b280cf450 0x7f7b280cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b304ee640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b280cdfb0 0x7f7b280ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b304ee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b280cece0 con 0x7f7b2815c7f0 2023-11-24T08:25:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b304ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b280cfe40 con 0x7f7b280cf450 2023-11-24T08:25:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b304ee640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b28156010 con 0x7f7b280cdfb0 2023-11-24T08:25:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b2e28a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7b2815c7f0 0x7f7b2806db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41070/0 (socket says 172.21.15.107:41070) 2023-11-24T08:25:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b2e28a640 1 -- 172.21.15.107:0/2864812606 learned_addr learned my addr 172.21.15.107:0/2864812606 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.515+0000 7f7b2da89640 1 --2- 172.21.15.107:0/2864812606 >> v2:172.21.15.107:3300/0 conn(0x7f7b280cf450 0x7f7b280cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2ea8b640 1 --2- 172.21.15.107:0/2864812606 >> v2:172.21.15.120:3300/0 conn(0x7f7b280cdfb0 0x7f7b280ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2d288640 1 -- 172.21.15.107:0/2864812606 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219688216 0 0) 0x7f7b280cece0 con 0x7f7b2815c7f0 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2d288640 1 -- 172.21.15.107:0/2864812606 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14003520 con 0x7f7b2815c7f0 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2da89640 1 -- 172.21.15.107:0/2864812606 >> v2:172.21.15.120:3300/0 conn(0x7f7b280cdfb0 msgr2=0x7f7b280ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2da89640 1 --2- 172.21.15.107:0/2864812606 >> v2:172.21.15.120:3300/0 conn(0x7f7b280cdfb0 0x7f7b280ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2da89640 1 -- 172.21.15.107:0/2864812606 >> v1:172.21.15.107:6789/0 conn(0x7f7b2815c7f0 legacy=0x7f7b2806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.516+0000 7f7b2da89640 1 -- 172.21.15.107:0/2864812606 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2806d040 con 0x7f7b280cf450 2023-11-24T08:25:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.517+0000 7f7b2da89640 1 --2- 172.21.15.107:0/2864812606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b280cf450 0x7f7b280cf900 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f7b280d00e0 tx=0x7f7b1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381646f41cc01b5c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.517+0000 7f7b2d288640 1 -- 172.21.15.107:0/2864812606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18031830 con 0x7f7b280cf450 2023-11-24T08:25:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b2d288640 1 -- 172.21.15.107:0/2864812606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b18031990 con 0x7f7b280cf450 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b2d288640 1 -- 172.21.15.107:0/2864812606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18031d20 con 0x7f7b280cf450 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 -- 172.21.15.107:0/2864812606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b280cf450 msgr2=0x7f7b280cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 --2- 172.21.15.107:0/2864812606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b280cf450 0x7f7b280cf900 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f7b280d00e0 tx=0x7f7b1802b8d0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 -- 172.21.15.107:0/2864812606 shutdown_connections 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 --2- 172.21.15.107:0/2864812606 >> v2:172.21.15.120:3300/0 conn(0x7f7b280cdfb0 0x7f7b280ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 --2- 172.21.15.107:0/2864812606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b280cf450 0x7f7b280cf900 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 -- 172.21.15.107:0/2864812606 >> 172.21.15.107:0/2864812606 conn(0x7f7b2806a920 msgr2=0x7f7b2806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 -- 172.21.15.107:0/2864812606 shutdown_connections 2023-11-24T08:25:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.518+0000 7f7b304ee640 1 -- 172.21.15.107:0/2864812606 wait complete. 2023-11-24T08:25:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.520+0000 7f7b304ee640 1 Processor -- start 2023-11-24T08:25:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.520+0000 7f7b304ee640 1 -- start start 2023-11-24T08:25:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280cdfb0 0x7f7b2815f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b280cf450 0x7f7b2814b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815fd40 con 0x7f7b2815c7f0 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815fb30 con 0x7f7b280cdfb0 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.521+0000 7f7b304ee640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2814b960 con 0x7f7b280cf450 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60200/0 (socket says 172.21.15.107:60200) 2023-11-24T08:25:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 -- 172.21.15.107:0/4119317771 learned_addr learned my addr 172.21.15.107:0/4119317771 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 -- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b280cf450 msgr2=0x7f7b2814b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b280cf450 0x7f7b2814b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 -- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280cdfb0 msgr2=0x7f7b2815f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280cdfb0 0x7f7b2815f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 -- 172.21.15.107:0/4119317771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b18009400 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b2ea8b640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007d90 tx=0x7f7b2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.522+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20014420 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.523+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b200145a0 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.523+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20014950 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.523+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b28158fc0 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.523+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b280b2560 con 0x7f7b2815c7f0 2023-11-24T08:25:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.524+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28067470 con 0x7f7b2815c7f0 2023-11-24T08:25:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.525+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f7b20014ab0 con 0x7f7b2815c7f0 2023-11-24T08:25:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.526+0000 7f7b1effd640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 0x7f7b0c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.526+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b20089330 con 0x7f7b2815c7f0 2023-11-24T08:25:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.526+0000 7f7b2e28a640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 0x7f7b0c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.527+0000 7f7b2e28a640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 0x7f7b0c06faf0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7b24001570 tx=0x7f7b24007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.530+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b20055e40 con 0x7f7b2815c7f0 2023-11-24T08:25:05.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.718+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b2806d600 con 0x7f7b0c06d630 2023-11-24T08:25:05.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.740+0000 7f7b1effd640 1 -- 172.21.15.107:0/4119317771 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287713 (secure 0 0 0) 0x7f7b240058b0 con 0x7f7b0c06d630 2023-11-24T08:25:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:05.746 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:25:05.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 msgr2=0x7f7b0c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 0x7f7b0c06faf0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7b24001570 tx=0x7f7b24007930 comp rx=0 tx=0).stop 2023-11-24T08:25:05.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 msgr2=0x7f7b28156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:05.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007d90 tx=0x7f7b2000e4b0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 shutdown_connections 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f7b0c06d630 0x7f7b0c06faf0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b2815c7f0 0x7f7b28156150 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.745+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b280cf450 0x7f7b2814b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.746+0000 7f7b304ee640 1 --2- 172.21.15.107:0/4119317771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280cdfb0 0x7f7b2815f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.746+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 >> 172.21.15.107:0/4119317771 conn(0x7f7b2806a920 msgr2=0x7f7b28155440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.746+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 shutdown_connections 2023-11-24T08:25:05.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:05.746+0000 7f7b304ee640 1 -- 172.21.15.107:0/4119317771 wait complete. 2023-11-24T08:25:05.766 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":852,"stamp":"2023-11-24T08:25:03.817951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346056,"kb_used_data":25864,"kb_used_omap":12,"kb_used_meta":320051,"kb_avail":754628664,"statfs":{"total":773094113280,"available":772739751936,"internally_reserved":0,"allocated":26484736,"data_stored":12425908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":327732798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.544956"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327329+0000","last_change":"2023-11-24T08:25:01.330095+0000","last_active":"2023-11-24T08:25:02.327329+0000","last_peered":"2023-11-24T08:25:02.327329+0000","last_clean":"2023-11-24T08:25:02.327329+0000","last_became_active":"2023-11-24T08:25:01.329468+0000","last_became_peered":"2023-11-24T08:25:01.329468+0000","last_unstale":"2023-11-24T08:25:02.327329+0000","last_undegraded":"2023-11-24T08:25:02.327329+0000","last_fullsized":"2023-11-24T08:25:02.327329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:12.511268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327354+0000","last_change":"2023-11-24T08:25:01.332177+0000","last_active":"2023-11-24T08:25:02.327354+0000","last_peered":"2023-11-24T08:25:02.327354+0000","last_clean":"2023-11-24T08:25:02.327354+0000","last_became_active":"2023-11-24T08:25:01.328911+0000","last_became_peered":"2023-11-24T08:25:01.328911+0000","last_unstale":"2023-11-24T08:25:02.327354+0000","last_undegraded":"2023-11-24T08:25:02.327354+0000","last_fullsized":"2023-11-24T08:25:02.327354+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:24.546643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379811+0000","last_change":"2023-11-24T08:25:01.319246+0000","last_active":"2023-11-24T08:25:02.379811+0000","last_peered":"2023-11-24T08:25:02.379811+0000","last_clean":"2023-11-24T08:25:02.379811+0000","last_became_active":"2023-11-24T08:25:01.318547+0000","last_became_peered":"2023-11-24T08:25:01.318547+0000","last_unstale":"2023-11-24T08:25:02.379811+0000","last_undegraded":"2023-11-24T08:25:02.379811+0000","last_fullsized":"2023-11-24T08:25:02.379811+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.104566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379393+0000","last_change":"2023-11-24T08:25:01.319435+0000","last_active":"2023-11-24T08:25:02.379393+0000","last_peered":"2023-11-24T08:25:02.379393+0000","last_clean":"2023-11-24T08:25:02.379393+0000","last_became_active":"2023-11-24T08:25:01.319129+0000","last_became_peered":"2023-11-24T08:25:01.319129+0000","last_unstale":"2023-11-24T08:25:02.379393+0000","last_undegraded":"2023-11-24T08:25:02.379393+0000","last_fullsized":"2023-11-24T08:25:02.379393+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:18.538022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362430+0000","last_change":"2023-11-24T08:25:01.323515+0000","last_active":"2023-11-24T08:25:02.362430+0000","last_peered":"2023-11-24T08:25:02.362430+0000","last_clean":"2023-11-24T08:25:02.362430+0000","last_became_active":"2023-11-24T08:25:01.323236+0000","last_became_peered":"2023-11-24T08:25:01.323236+0000","last_unstale":"2023-11-24T08:25:02.362430+0000","last_undegraded":"2023-11-24T08:25:02.362430+0000","last_fullsized":"2023-11-24T08:25:02.362430+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:18:22.616146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362102+0000","last_change":"2023-11-24T08:25:01.318160+0000","last_active":"2023-11-24T08:25:02.362102+0000","last_peered":"2023-11-24T08:25:02.362102+0000","last_clean":"2023-11-24T08:25:02.362102+0000","last_became_active":"2023-11-24T08:25:01.317847+0000","last_became_peered":"2023-11-24T08:25:01.317847+0000","last_unstale":"2023-11-24T08:25:02.362102+0000","last_undegraded":"2023-11-24T08:25:02.362102+0000","last_fullsized":"2023-11-24T08:25:02.362102+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:23.524948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459081+0000","last_change":"2023-11-24T08:25:01.325577+0000","last_active":"2023-11-24T08:25:02.459081+0000","last_peered":"2023-11-24T08:25:02.459081+0000","last_clean":"2023-11-24T08:25:02.459081+0000","last_became_active":"2023-11-24T08:25:01.324919+0000","last_became_peered":"2023-11-24T08:25:01.324919+0000","last_unstale":"2023-11-24T08:25:02.459081+0000","last_undegraded":"2023-11-24T08:25:02.459081+0000","last_fullsized":"2023-11-24T08:25:02.459081+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:10.769637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379261+0000","last_change":"2023-11-24T08:25:01.327945+0000","last_active":"2023-11-24T08:25:02.379261+0000","last_peered":"2023-11-24T08:25:02.379261+0000","last_clean":"2023-11-24T08:25:02.379261+0000","last_became_active":"2023-11-24T08:25:01.327673+0000","last_became_peered":"2023-11-24T08:25:01.327673+0000","last_unstale":"2023-11-24T08:25:02.379261+0000","last_undegraded":"2023-11-24T08:25:02.379261+0000","last_fullsized":"2023-11-24T08:25:02.379261+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:58:30.987939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460293+0000","last_change":"2023-11-24T08:25:01.329042+0000","last_active":"2023-11-24T08:25:02.460293+0000","last_peered":"2023-11-24T08:25:02.460293+0000","last_clean":"2023-11-24T08:25:02.460293+0000","last_became_active":"2023-11-24T08:25:01.328768+0000","last_became_peered":"2023-11-24T08:25:01.328768+0000","last_unstale":"2023-11-24T08:25:02.460293+0000","last_undegraded":"2023-11-24T08:25:02.460293+0000","last_fullsized":"2023-11-24T08:25:02.460293+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:50.929188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460715+0000","last_change":"2023-11-24T08:25:01.334656+0000","last_active":"2023-11-24T08:25:02.460715+0000","last_peered":"2023-11-24T08:25:02.460715+0000","last_clean":"2023-11-24T08:25:02.460715+0000","last_became_active":"2023-11-24T08:25:01.334340+0000","last_became_peered":"2023-11-24T08:25:01.334340+0000","last_unstale":"2023-11-24T08:25:02.460715+0000","last_undegraded":"2023-11-24T08:25:02.460715+0000","last_fullsized":"2023-11-24T08:25:02.460715+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:18.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457769+0000","last_change":"2023-11-24T08:25:01.331176+0000","last_active":"2023-11-24T08:25:02.457769+0000","last_peered":"2023-11-24T08:25:02.457769+0000","last_clean":"2023-11-24T08:25:02.457769+0000","last_became_active":"2023-11-24T08:25:01.330893+0000","last_became_peered":"2023-11-24T08:25:01.330893+0000","last_unstale":"2023-11-24T08:25:02.457769+0000","last_undegraded":"2023-11-24T08:25:02.457769+0000","last_fullsized":"2023-11-24T08:25:02.457769+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:43:30.627163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326949+0000","last_change":"2023-11-24T08:25:01.332821+0000","last_active":"2023-11-24T08:25:02.326949+0000","last_peered":"2023-11-24T08:25:02.326949+0000","last_clean":"2023-11-24T08:25:02.326949+0000","last_became_active":"2023-11-24T08:25:01.331469+0000","last_became_peered":"2023-11-24T08:25:01.331469+0000","last_unstale":"2023-11-24T08:25:02.326949+0000","last_undegraded":"2023-11-24T08:25:02.326949+0000","last_fullsized":"2023-11-24T08:25:02.326949+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:33:18.982077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379618+0000","last_change":"2023-11-24T08:25:01.335307+0000","last_active":"2023-11-24T08:25:02.379618+0000","last_peered":"2023-11-24T08:25:02.379618+0000","last_clean":"2023-11-24T08:25:02.379618+0000","last_became_active":"2023-11-24T08:25:01.334719+0000","last_became_peered":"2023-11-24T08:25:01.334719+0000","last_unstale":"2023-11-24T08:25:02.379618+0000","last_undegraded":"2023-11-24T08:25:02.379618+0000","last_fullsized":"2023-11-24T08:25:02.379618+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:23.179460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460642+0000","last_change":"2023-11-24T08:25:01.322748+0000","last_active":"2023-11-24T08:25:02.460642+0000","last_peered":"2023-11-24T08:25:02.460642+0000","last_clean":"2023-11-24T08:25:02.460642+0000","last_became_active":"2023-11-24T08:25:01.322329+0000","last_became_peered":"2023-11-24T08:25:01.322329+0000","last_unstale":"2023-11-24T08:25:02.460642+0000","last_undegraded":"2023-11-24T08:25:02.460642+0000","last_fullsized":"2023-11-24T08:25:02.460642+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:23.433878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379306+0000","last_change":"2023-11-24T08:25:01.328269+0000","last_active":"2023-11-24T08:25:02.379306+0000","last_peered":"2023-11-24T08:25:02.379306+0000","last_clean":"2023-11-24T08:25:02.379306+0000","last_became_active":"2023-11-24T08:25:01.327370+0000","last_became_peered":"2023-11-24T08:25:01.327370+0000","last_unstale":"2023-11-24T08:25:02.379306+0000","last_undegraded":"2023-11-24T08:25:02.379306+0000","last_fullsized":"2023-11-24T08:25:02.379306+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:34.113226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458619+0000","last_change":"2023-11-24T08:25:01.325800+0000","last_active":"2023-11-24T08:25:02.458619+0000","last_peered":"2023-11-24T08:25:02.458619+0000","last_clean":"2023-11-24T08:25:02.458619+0000","last_became_active":"2023-11-24T08:25:01.325137+0000","last_became_peered":"2023-11-24T08:25:01.325137+0000","last_unstale":"2023-11-24T08:25:02.458619+0000","last_undegraded":"2023-11-24T08:25:02.458619+0000","last_fullsized":"2023-11-24T08:25:02.458619+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:04.413433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458965+0000","last_change":"2023-11-24T08:25:01.312666+0000","last_active":"2023-11-24T08:25:02.458965+0000","last_peered":"2023-11-24T08:25:02.458965+0000","last_clean":"2023-11-24T08:25:02.458965+0000","last_became_active":"2023-11-24T08:25:01.311919+0000","last_became_peered":"2023-11-24T08:25:01.311919+0000","last_unstale":"2023-11-24T08:25:02.458965+0000","last_undegraded":"2023-11-24T08:25:02.458965+0000","last_fullsized":"2023-11-24T08:25:02.458965+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:57.192941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379591+0000","last_change":"2023-11-24T08:25:01.316892+0000","last_active":"2023-11-24T08:25:02.379591+0000","last_peered":"2023-11-24T08:25:02.379591+0000","last_clean":"2023-11-24T08:25:02.379591+0000","last_became_active":"2023-11-24T08:25:01.316608+0000","last_became_peered":"2023-11-24T08:25:01.316608+0000","last_unstale":"2023-11-24T08:25:02.379591+0000","last_undegraded":"2023-11-24T08:25:02.379591+0000","last_fullsized":"2023-11-24T08:25:02.379591+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:47.587787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459131+0000","last_change":"2023-11-24T08:25:01.319730+0000","last_active":"2023-11-24T08:25:02.459131+0000","last_peered":"2023-11-24T08:25:02.459131+0000","last_clean":"2023-11-24T08:25:02.459131+0000","last_became_active":"2023-11-24T08:25:01.313366+0000","last_became_peered":"2023-11-24T08:25:01.313366+0000","last_unstale":"2023-11-24T08:25:02.459131+0000","last_undegraded":"2023-11-24T08:25:02.459131+0000","last_fullsized":"2023-11-24T08:25:02.459131+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.104566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459399+0000","last_change":"2023-11-24T08:25:01.327234+0000","last_active":"2023-11-24T08:25:02.459399+0000","last_peered":"2023-11-24T08:25:02.459399+0000","last_clean":"2023-11-24T08:25:02.459399+0000","last_became_active":"2023-11-24T08:25:01.326914+0000","last_became_peered":"2023-11-24T08:25:01.326914+0000","last_unstale":"2023-11-24T08:25:02.459399+0000","last_undegraded":"2023-11-24T08:25:02.459399+0000","last_fullsized":"2023-11-24T08:25:02.459399+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:25.254259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457739+0000","last_change":"2023-11-24T08:25:01.318731+0000","last_active":"2023-11-24T08:25:02.457739+0000","last_peered":"2023-11-24T08:25:02.457739+0000","last_clean":"2023-11-24T08:25:02.457739+0000","last_became_active":"2023-11-24T08:25:01.318486+0000","last_became_peered":"2023-11-24T08:25:01.318486+0000","last_unstale":"2023-11-24T08:25:02.457739+0000","last_undegraded":"2023-11-24T08:25:02.457739+0000","last_fullsized":"2023-11-24T08:25:02.457739+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:16.102357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459272+0000","last_change":"2023-11-24T08:25:01.319521+0000","last_active":"2023-11-24T08:25:02.459272+0000","last_peered":"2023-11-24T08:25:02.459272+0000","last_clean":"2023-11-24T08:25:02.459272+0000","last_became_active":"2023-11-24T08:25:01.319263+0000","last_became_peered":"2023-11-24T08:25:01.319263+0000","last_unstale":"2023-11-24T08:25:02.459272+0000","last_undegraded":"2023-11-24T08:25:02.459272+0000","last_fullsized":"2023-11-24T08:25:02.459272+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:46:45.332206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459430+0000","last_change":"2023-11-24T08:25:01.312201+0000","last_active":"2023-11-24T08:25:02.459430+0000","last_peered":"2023-11-24T08:25:02.459430+0000","last_clean":"2023-11-24T08:25:02.459430+0000","last_became_active":"2023-11-24T08:25:01.311831+0000","last_became_peered":"2023-11-24T08:25:01.311831+0000","last_unstale":"2023-11-24T08:25:02.459430+0000","last_undegraded":"2023-11-24T08:25:02.459430+0000","last_fullsized":"2023-11-24T08:25:02.459430+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:16:32.590126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459147+0000","last_change":"2023-11-24T08:25:01.328795+0000","last_active":"2023-11-24T08:25:02.459147+0000","last_peered":"2023-11-24T08:25:02.459147+0000","last_clean":"2023-11-24T08:25:02.459147+0000","last_became_active":"2023-11-24T08:25:01.328509+0000","last_became_peered":"2023-11-24T08:25:01.328509+0000","last_unstale":"2023-11-24T08:25:02.459147+0000","last_undegraded":"2023-11-24T08:25:02.459147+0000","last_fullsized":"2023-11-24T08:25:02.459147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:13.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327745+0000","last_change":"2023-11-24T08:25:01.330266+0000","last_active":"2023-11-24T08:25:02.327745+0000","last_peered":"2023-11-24T08:25:02.327745+0000","last_clean":"2023-11-24T08:25:02.327745+0000","last_became_active":"2023-11-24T08:25:01.329823+0000","last_became_peered":"2023-11-24T08:25:01.329823+0000","last_unstale":"2023-11-24T08:25:02.327745+0000","last_undegraded":"2023-11-24T08:25:02.327745+0000","last_fullsized":"2023-11-24T08:25:02.327745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:10.769637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362473+0000","last_change":"2023-11-24T08:25:01.331571+0000","last_active":"2023-11-24T08:25:02.362473+0000","last_peered":"2023-11-24T08:25:02.362473+0000","last_clean":"2023-11-24T08:25:02.362473+0000","last_became_active":"2023-11-24T08:25:01.331288+0000","last_became_peered":"2023-11-24T08:25:01.331288+0000","last_unstale":"2023-11-24T08:25:02.362473+0000","last_undegraded":"2023-11-24T08:25:02.362473+0000","last_fullsized":"2023-11-24T08:25:02.362473+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:58:45.667771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362425+0000","last_change":"2023-11-24T08:25:01.327647+0000","last_active":"2023-11-24T08:25:02.362425+0000","last_peered":"2023-11-24T08:25:02.362425+0000","last_clean":"2023-11-24T08:25:02.362425+0000","last_became_active":"2023-11-24T08:25:01.327127+0000","last_became_peered":"2023-11-24T08:25:01.327127+0000","last_unstale":"2023-11-24T08:25:02.362425+0000","last_undegraded":"2023-11-24T08:25:02.362425+0000","last_fullsized":"2023-11-24T08:25:02.362425+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:29.203493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459745+0000","last_change":"2023-11-24T08:25:01.325447+0000","last_active":"2023-11-24T08:25:02.459745+0000","last_peered":"2023-11-24T08:25:02.459745+0000","last_clean":"2023-11-24T08:25:02.459745+0000","last_became_active":"2023-11-24T08:25:01.325008+0000","last_became_peered":"2023-11-24T08:25:01.325008+0000","last_unstale":"2023-11-24T08:25:02.459745+0000","last_undegraded":"2023-11-24T08:25:02.459745+0000","last_fullsized":"2023-11-24T08:25:02.459745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:18.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325993+0000","last_change":"2023-11-24T08:25:01.315692+0000","last_active":"2023-11-24T08:25:02.325993+0000","last_peered":"2023-11-24T08:25:02.325993+0000","last_clean":"2023-11-24T08:25:02.325993+0000","last_became_active":"2023-11-24T08:25:01.315367+0000","last_became_peered":"2023-11-24T08:25:01.315367+0000","last_unstale":"2023-11-24T08:25:02.325993+0000","last_undegraded":"2023-11-24T08:25:02.325993+0000","last_fullsized":"2023-11-24T08:25:02.325993+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:18.092625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458257+0000","last_change":"2023-11-24T08:25:01.325282+0000","last_active":"2023-11-24T08:25:02.458257+0000","last_peered":"2023-11-24T08:25:02.458257+0000","last_clean":"2023-11-24T08:25:02.458257+0000","last_became_active":"2023-11-24T08:25:01.322853+0000","last_became_peered":"2023-11-24T08:25:01.322853+0000","last_unstale":"2023-11-24T08:25:02.458257+0000","last_undegraded":"2023-11-24T08:25:02.458257+0000","last_fullsized":"2023-11-24T08:25:02.458257+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:03.454240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379312+0000","last_change":"2023-11-24T08:25:01.316555+0000","last_active":"2023-11-24T08:25:02.379312+0000","last_peered":"2023-11-24T08:25:02.379312+0000","last_clean":"2023-11-24T08:25:02.379312+0000","last_became_active":"2023-11-24T08:25:01.316149+0000","last_became_peered":"2023-11-24T08:25:01.316149+0000","last_unstale":"2023-11-24T08:25:02.379312+0000","last_undegraded":"2023-11-24T08:25:02.379312+0000","last_fullsized":"2023-11-24T08:25:02.379312+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:09.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458510+0000","last_change":"2023-11-24T08:25:01.327987+0000","last_active":"2023-11-24T08:25:02.458510+0000","last_peered":"2023-11-24T08:25:02.458510+0000","last_clean":"2023-11-24T08:25:02.458510+0000","last_became_active":"2023-11-24T08:25:01.327723+0000","last_became_peered":"2023-11-24T08:25:01.327723+0000","last_unstale":"2023-11-24T08:25:02.458510+0000","last_undegraded":"2023-11-24T08:25:02.458510+0000","last_fullsized":"2023-11-24T08:25:02.458510+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:06:22.279973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"248'4","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460626+0000","last_change":"2023-11-24T08:25:00.317048+0000","last_active":"2023-11-24T08:25:02.460626+0000","last_peered":"2023-11-24T08:25:02.460626+0000","last_clean":"2023-11-24T08:25:02.460626+0000","last_became_active":"2023-11-24T08:25:00.316751+0000","last_became_peered":"2023-11-24T08:25:00.316751+0000","last_unstale":"2023-11-24T08:25:02.460626+0000","last_undegraded":"2023-11-24T08:25:02.460626+0000","last_fullsized":"2023-11-24T08:25:02.460626+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:29.088703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459723+0000","last_change":"2023-11-24T08:25:01.334541+0000","last_active":"2023-11-24T08:25:02.459723+0000","last_peered":"2023-11-24T08:25:02.459723+0000","last_clean":"2023-11-24T08:25:02.459723+0000","last_became_active":"2023-11-24T08:25:01.334224+0000","last_became_peered":"2023-11-24T08:25:01.334224+0000","last_unstale":"2023-11-24T08:25:02.459723+0000","last_undegraded":"2023-11-24T08:25:02.459723+0000","last_fullsized":"2023-11-24T08:25:02.459723+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:43.284439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459922+0000","last_change":"2023-11-24T08:25:00.320585+0000","last_active":"2023-11-24T08:25:02.459922+0000","last_peered":"2023-11-24T08:25:02.459922+0000","last_clean":"2023-11-24T08:25:02.459922+0000","last_became_active":"2023-11-24T08:25:00.320273+0000","last_became_peered":"2023-11-24T08:25:00.320273+0000","last_unstale":"2023-11-24T08:25:02.459922+0000","last_undegraded":"2023-11-24T08:25:02.459922+0000","last_fullsized":"2023-11-24T08:25:02.459922+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:31.719759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325999+0000","last_change":"2023-11-24T08:25:01.330769+0000","last_active":"2023-11-24T08:25:02.325999+0000","last_peered":"2023-11-24T08:25:02.325999+0000","last_clean":"2023-11-24T08:25:02.325999+0000","last_became_active":"2023-11-24T08:25:01.330421+0000","last_became_peered":"2023-11-24T08:25:01.330421+0000","last_unstale":"2023-11-24T08:25:02.325999+0000","last_undegraded":"2023-11-24T08:25:02.325999+0000","last_fullsized":"2023-11-24T08:25:02.325999+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:53.759171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457699+0000","last_change":"2023-11-24T08:25:00.317565+0000","last_active":"2023-11-24T08:25:02.457699+0000","last_peered":"2023-11-24T08:25:02.457699+0000","last_clean":"2023-11-24T08:25:02.457699+0000","last_became_active":"2023-11-24T08:25:00.315560+0000","last_became_peered":"2023-11-24T08:25:00.315560+0000","last_unstale":"2023-11-24T08:25:02.457699+0000","last_undegraded":"2023-11-24T08:25:02.457699+0000","last_fullsized":"2023-11-24T08:25:02.457699+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:01.301706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460019+0000","last_change":"2023-11-24T08:25:01.321154+0000","last_active":"2023-11-24T08:25:02.460019+0000","last_peered":"2023-11-24T08:25:02.460019+0000","last_clean":"2023-11-24T08:25:02.460019+0000","last_became_active":"2023-11-24T08:25:01.320868+0000","last_became_peered":"2023-11-24T08:25:01.320868+0000","last_unstale":"2023-11-24T08:25:02.460019+0000","last_undegraded":"2023-11-24T08:25:02.460019+0000","last_fullsized":"2023-11-24T08:25:02.460019+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:50:00.181379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457554+0000","last_change":"2023-11-24T08:25:00.317762+0000","last_active":"2023-11-24T08:25:02.457554+0000","last_peered":"2023-11-24T08:25:02.457554+0000","last_clean":"2023-11-24T08:25:02.457554+0000","last_became_active":"2023-11-24T08:25:00.316357+0000","last_became_peered":"2023-11-24T08:25:00.316357+0000","last_unstale":"2023-11-24T08:25:02.457554+0000","last_undegraded":"2023-11-24T08:25:02.457554+0000","last_fullsized":"2023-11-24T08:25:02.457554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:08:39.294181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379803+0000","last_change":"2023-11-24T08:25:01.310919+0000","last_active":"2023-11-24T08:25:02.379803+0000","last_peered":"2023-11-24T08:25:02.379803+0000","last_clean":"2023-11-24T08:25:02.379803+0000","last_became_active":"2023-11-24T08:25:01.310647+0000","last_became_peered":"2023-11-24T08:25:01.310647+0000","last_unstale":"2023-11-24T08:25:02.379803+0000","last_undegraded":"2023-11-24T08:25:02.379803+0000","last_fullsized":"2023-11-24T08:25:02.379803+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:57.192941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326600+0000","last_change":"2023-11-24T08:25:00.321262+0000","last_active":"2023-11-24T08:25:02.326600+0000","last_peered":"2023-11-24T08:25:02.326600+0000","last_clean":"2023-11-24T08:25:02.326600+0000","last_became_active":"2023-11-24T08:25:00.320945+0000","last_became_peered":"2023-11-24T08:25:00.320945+0000","last_unstale":"2023-11-24T08:25:02.326600+0000","last_undegraded":"2023-11-24T08:25:02.326600+0000","last_fullsized":"2023-11-24T08:25:02.326600+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:58.291118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458888+0000","last_change":"2023-11-24T08:25:01.334235+0000","last_active":"2023-11-24T08:25:02.458888+0000","last_peered":"2023-11-24T08:25:02.458888+0000","last_clean":"2023-11-24T08:25:02.458888+0000","last_became_active":"2023-11-24T08:25:01.333914+0000","last_became_peered":"2023-11-24T08:25:01.333914+0000","last_unstale":"2023-11-24T08:25:02.458888+0000","last_undegraded":"2023-11-24T08:25:02.458888+0000","last_fullsized":"2023-11-24T08:25:02.458888+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:35:36.164882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362223+0000","last_change":"2023-11-24T08:25:00.310204+0000","last_active":"2023-11-24T08:25:02.362223+0000","last_peered":"2023-11-24T08:25:02.362223+0000","last_clean":"2023-11-24T08:25:02.362223+0000","last_became_active":"2023-11-24T08:25:00.309880+0000","last_became_peered":"2023-11-24T08:25:00.309880+0000","last_unstale":"2023-11-24T08:25:02.362223+0000","last_undegraded":"2023-11-24T08:25:02.362223+0000","last_fullsized":"2023-11-24T08:25:02.362223+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:19.333339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459951+0000","last_change":"2023-11-24T08:25:01.314336+0000","last_active":"2023-11-24T08:25:02.459951+0000","last_peered":"2023-11-24T08:25:02.459951+0000","last_clean":"2023-11-24T08:25:02.459951+0000","last_became_active":"2023-11-24T08:25:01.314084+0000","last_became_peered":"2023-11-24T08:25:01.314084+0000","last_unstale":"2023-11-24T08:25:02.459951+0000","last_undegraded":"2023-11-24T08:25:02.459951+0000","last_fullsized":"2023-11-24T08:25:02.459951+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:09.700254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325044+0000","last_change":"2023-11-24T08:25:00.313527+0000","last_active":"2023-11-24T08:25:02.325044+0000","last_peered":"2023-11-24T08:25:02.325044+0000","last_clean":"2023-11-24T08:25:02.325044+0000","last_became_active":"2023-11-24T08:25:00.313201+0000","last_became_peered":"2023-11-24T08:25:00.313201+0000","last_unstale":"2023-11-24T08:25:02.325044+0000","last_undegraded":"2023-11-24T08:25:02.325044+0000","last_fullsized":"2023-11-24T08:25:02.325044+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:31.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459293+0000","last_change":"2023-11-24T08:25:01.309578+0000","last_active":"2023-11-24T08:25:02.459293+0000","last_peered":"2023-11-24T08:25:02.459293+0000","last_clean":"2023-11-24T08:25:02.459293+0000","last_became_active":"2023-11-24T08:25:01.306969+0000","last_became_peered":"2023-11-24T08:25:01.306969+0000","last_unstale":"2023-11-24T08:25:02.459293+0000","last_undegraded":"2023-11-24T08:25:02.459293+0000","last_fullsized":"2023-11-24T08:25:02.459293+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:51.707800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379570+0000","last_change":"2023-11-24T08:25:00.316813+0000","last_active":"2023-11-24T08:25:02.379570+0000","last_peered":"2023-11-24T08:25:02.379570+0000","last_clean":"2023-11-24T08:25:02.379570+0000","last_became_active":"2023-11-24T08:25:00.316266+0000","last_became_peered":"2023-11-24T08:25:00.316266+0000","last_unstale":"2023-11-24T08:25:02.379570+0000","last_undegraded":"2023-11-24T08:25:02.379570+0000","last_fullsized":"2023-11-24T08:25:02.379570+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:05:52.092847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379497+0000","last_change":"2023-11-24T08:25:01.333485+0000","last_active":"2023-11-24T08:25:02.379497+0000","last_peered":"2023-11-24T08:25:02.379497+0000","last_clean":"2023-11-24T08:25:02.379497+0000","last_became_active":"2023-11-24T08:25:01.333082+0000","last_became_peered":"2023-11-24T08:25:01.333082+0000","last_unstale":"2023-11-24T08:25:02.379497+0000","last_undegraded":"2023-11-24T08:25:02.379497+0000","last_fullsized":"2023-11-24T08:25:02.379497+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:14.431188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459865+0000","last_change":"2023-11-24T08:25:00.319785+0000","last_active":"2023-11-24T08:25:02.459865+0000","last_peered":"2023-11-24T08:25:02.459865+0000","last_clean":"2023-11-24T08:25:02.459865+0000","last_became_active":"2023-11-24T08:25:00.319524+0000","last_became_peered":"2023-11-24T08:25:00.319524+0000","last_unstale":"2023-11-24T08:25:02.459865+0000","last_undegraded":"2023-11-24T08:25:02.459865+0000","last_fullsized":"2023-11-24T08:25:02.459865+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:00.830631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325519+0000","last_change":"2023-11-24T08:25:01.326899+0000","last_active":"2023-11-24T08:25:02.325519+0000","last_peered":"2023-11-24T08:25:02.325519+0000","last_clean":"2023-11-24T08:25:02.325519+0000","last_became_active":"2023-11-24T08:25:01.326619+0000","last_became_peered":"2023-11-24T08:25:01.326619+0000","last_unstale":"2023-11-24T08:25:02.325519+0000","last_undegraded":"2023-11-24T08:25:02.325519+0000","last_fullsized":"2023-11-24T08:25:02.325519+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:43.724381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326651+0000","last_change":"2023-11-24T08:25:00.318702+0000","last_active":"2023-11-24T08:25:02.326651+0000","last_peered":"2023-11-24T08:25:02.326651+0000","last_clean":"2023-11-24T08:25:02.326651+0000","last_became_active":"2023-11-24T08:25:00.318180+0000","last_became_peered":"2023-11-24T08:25:00.318180+0000","last_unstale":"2023-11-24T08:25:02.326651+0000","last_undegraded":"2023-11-24T08:25:02.326651+0000","last_fullsized":"2023-11-24T08:25:02.326651+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:50.337476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379372+0000","last_change":"2023-11-24T08:25:01.319328+0000","last_active":"2023-11-24T08:25:02.379372+0000","last_peered":"2023-11-24T08:25:02.379372+0000","last_clean":"2023-11-24T08:25:02.379372+0000","last_became_active":"2023-11-24T08:25:01.319051+0000","last_became_peered":"2023-11-24T08:25:01.319051+0000","last_unstale":"2023-11-24T08:25:02.379372+0000","last_undegraded":"2023-11-24T08:25:02.379372+0000","last_fullsized":"2023-11-24T08:25:02.379372+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:10:59.363646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"248'3","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459854+0000","last_change":"2023-11-24T08:25:00.305695+0000","last_active":"2023-11-24T08:25:02.459854+0000","last_peered":"2023-11-24T08:25:02.459854+0000","last_clean":"2023-11-24T08:25:02.459854+0000","last_became_active":"2023-11-24T08:25:00.305329+0000","last_became_peered":"2023-11-24T08:25:00.305329+0000","last_unstale":"2023-11-24T08:25:02.459854+0000","last_undegraded":"2023-11-24T08:25:02.459854+0000","last_fullsized":"2023-11-24T08:25:02.459854+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:22.161492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362551+0000","last_change":"2023-11-24T08:25:01.316785+0000","last_active":"2023-11-24T08:25:02.362551+0000","last_peered":"2023-11-24T08:25:02.362551+0000","last_clean":"2023-11-24T08:25:02.362551+0000","last_became_active":"2023-11-24T08:25:01.316491+0000","last_became_peered":"2023-11-24T08:25:01.316491+0000","last_unstale":"2023-11-24T08:25:02.362551+0000","last_undegraded":"2023-11-24T08:25:02.362551+0000","last_fullsized":"2023-11-24T08:25:02.362551+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:43.056955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"248'6","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460819+0000","last_change":"2023-11-24T08:25:00.315811+0000","last_active":"2023-11-24T08:25:02.460819+0000","last_peered":"2023-11-24T08:25:02.460819+0000","last_clean":"2023-11-24T08:25:02.460819+0000","last_became_active":"2023-11-24T08:25:00.315497+0000","last_became_peered":"2023-11-24T08:25:00.315497+0000","last_unstale":"2023-11-24T08:25:02.460819+0000","last_undegraded":"2023-11-24T08:25:02.460819+0000","last_fullsized":"2023-11-24T08:25:02.460819+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:42.706413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327294+0000","last_change":"2023-11-24T08:25:01.331805+0000","last_active":"2023-11-24T08:25:02.327294+0000","last_peered":"2023-11-24T08:25:02.327294+0000","last_clean":"2023-11-24T08:25:02.327294+0000","last_became_active":"2023-11-24T08:25:01.330983+0000","last_became_peered":"2023-11-24T08:25:01.330983+0000","last_unstale":"2023-11-24T08:25:02.327294+0000","last_undegraded":"2023-11-24T08:25:02.327294+0000","last_fullsized":"2023-11-24T08:25:02.327294+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:30.106671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326172+0000","last_change":"2023-11-24T08:25:00.317270+0000","last_active":"2023-11-24T08:25:02.326172+0000","last_peered":"2023-11-24T08:25:02.326172+0000","last_clean":"2023-11-24T08:25:02.326172+0000","last_became_active":"2023-11-24T08:25:00.315767+0000","last_became_peered":"2023-11-24T08:25:00.315767+0000","last_unstale":"2023-11-24T08:25:02.326172+0000","last_undegraded":"2023-11-24T08:25:02.326172+0000","last_fullsized":"2023-11-24T08:25:02.326172+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:50.689833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379786+0000","last_change":"2023-11-24T08:25:01.324126+0000","last_active":"2023-11-24T08:25:02.379786+0000","last_peered":"2023-11-24T08:25:02.379786+0000","last_clean":"2023-11-24T08:25:02.379786+0000","last_became_active":"2023-11-24T08:25:01.323851+0000","last_became_peered":"2023-11-24T08:25:01.323851+0000","last_unstale":"2023-11-24T08:25:02.379786+0000","last_undegraded":"2023-11-24T08:25:02.379786+0000","last_fullsized":"2023-11-24T08:25:02.379786+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:32.430121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326226+0000","last_change":"2023-11-24T08:25:00.319921+0000","last_active":"2023-11-24T08:25:02.326226+0000","last_peered":"2023-11-24T08:25:02.326226+0000","last_clean":"2023-11-24T08:25:02.326226+0000","last_became_active":"2023-11-24T08:25:00.319611+0000","last_became_peered":"2023-11-24T08:25:00.319611+0000","last_unstale":"2023-11-24T08:25:02.326226+0000","last_undegraded":"2023-11-24T08:25:02.326226+0000","last_fullsized":"2023-11-24T08:25:02.326226+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:57.507582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460294+0000","last_change":"2023-11-24T08:25:01.315600+0000","last_active":"2023-11-24T08:25:02.460294+0000","last_peered":"2023-11-24T08:25:02.460294+0000","last_clean":"2023-11-24T08:25:02.460294+0000","last_became_active":"2023-11-24T08:25:01.314832+0000","last_became_peered":"2023-11-24T08:25:01.314832+0000","last_unstale":"2023-11-24T08:25:02.460294+0000","last_undegraded":"2023-11-24T08:25:02.460294+0000","last_fullsized":"2023-11-24T08:25:02.460294+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:51.355444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325768+0000","last_change":"2023-11-24T08:25:00.318665+0000","last_active":"2023-11-24T08:25:02.325768+0000","last_peered":"2023-11-24T08:25:02.325768+0000","last_clean":"2023-11-24T08:25:02.325768+0000","last_became_active":"2023-11-24T08:25:00.318365+0000","last_became_peered":"2023-11-24T08:25:00.318365+0000","last_unstale":"2023-11-24T08:25:02.325768+0000","last_undegraded":"2023-11-24T08:25:02.325768+0000","last_fullsized":"2023-11-24T08:25:02.325768+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:34:58.663930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459658+0000","last_change":"2023-11-24T08:25:01.313649+0000","last_active":"2023-11-24T08:25:02.459658+0000","last_peered":"2023-11-24T08:25:02.459658+0000","last_clean":"2023-11-24T08:25:02.459658+0000","last_became_active":"2023-11-24T08:25:01.313184+0000","last_became_peered":"2023-11-24T08:25:01.313184+0000","last_unstale":"2023-11-24T08:25:02.459658+0000","last_undegraded":"2023-11-24T08:25:02.459658+0000","last_fullsized":"2023-11-24T08:25:02.459658+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:01.841805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"248'8","reported_seq":20,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459025+0000","last_change":"2023-11-24T08:25:00.317907+0000","last_active":"2023-11-24T08:25:02.459025+0000","last_peered":"2023-11-24T08:25:02.459025+0000","last_clean":"2023-11-24T08:25:02.459025+0000","last_became_active":"2023-11-24T08:25:00.317636+0000","last_became_peered":"2023-11-24T08:25:00.317636+0000","last_unstale":"2023-11-24T08:25:02.459025+0000","last_undegraded":"2023-11-24T08:25:02.459025+0000","last_fullsized":"2023-11-24T08:25:02.459025+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:55.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362039+0000","last_change":"2023-11-24T08:25:01.320958+0000","last_active":"2023-11-24T08:25:02.362039+0000","last_peered":"2023-11-24T08:25:02.362039+0000","last_clean":"2023-11-24T08:25:02.362039+0000","last_became_active":"2023-11-24T08:25:01.320677+0000","last_became_peered":"2023-11-24T08:25:01.320677+0000","last_unstale":"2023-11-24T08:25:02.362039+0000","last_undegraded":"2023-11-24T08:25:02.362039+0000","last_fullsized":"2023-11-24T08:25:02.362039+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:49.725715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458406+0000","last_change":"2023-11-24T08:25:00.310337+0000","last_active":"2023-11-24T08:25:02.458406+0000","last_peered":"2023-11-24T08:25:02.458406+0000","last_clean":"2023-11-24T08:25:02.458406+0000","last_became_active":"2023-11-24T08:25:00.310041+0000","last_became_peered":"2023-11-24T08:25:00.310041+0000","last_unstale":"2023-11-24T08:25:02.458406+0000","last_undegraded":"2023-11-24T08:25:02.458406+0000","last_fullsized":"2023-11-24T08:25:02.458406+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:41:22.718681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458391+0000","last_change":"2023-11-24T08:25:01.329443+0000","last_active":"2023-11-24T08:25:02.458391+0000","last_peered":"2023-11-24T08:25:02.458391+0000","last_clean":"2023-11-24T08:25:02.458391+0000","last_became_active":"2023-11-24T08:25:01.329072+0000","last_became_peered":"2023-11-24T08:25:01.329072+0000","last_unstale":"2023-11-24T08:25:02.458391+0000","last_undegraded":"2023-11-24T08:25:02.458391+0000","last_fullsized":"2023-11-24T08:25:02.458391+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:11:44.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362582+0000","last_change":"2023-11-24T08:25:00.302549+0000","last_active":"2023-11-24T08:25:02.362582+0000","last_peered":"2023-11-24T08:25:02.362582+0000","last_clean":"2023-11-24T08:25:02.362582+0000","last_became_active":"2023-11-24T08:25:00.302158+0000","last_became_peered":"2023-11-24T08:25:00.302158+0000","last_unstale":"2023-11-24T08:25:02.362582+0000","last_undegraded":"2023-11-24T08:25:02.362582+0000","last_fullsized":"2023-11-24T08:25:02.362582+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:48.666202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362247+0000","last_change":"2023-11-24T08:25:01.321841+0000","last_active":"2023-11-24T08:25:02.362247+0000","last_peered":"2023-11-24T08:25:02.362247+0000","last_clean":"2023-11-24T08:25:02.362247+0000","last_became_active":"2023-11-24T08:25:01.321562+0000","last_became_peered":"2023-11-24T08:25:01.321562+0000","last_unstale":"2023-11-24T08:25:02.362247+0000","last_undegraded":"2023-11-24T08:25:02.362247+0000","last_fullsized":"2023-11-24T08:25:02.362247+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:27.402198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457977+0000","last_change":"2023-11-24T08:25:00.313073+0000","last_active":"2023-11-24T08:25:02.457977+0000","last_peered":"2023-11-24T08:25:02.457977+0000","last_clean":"2023-11-24T08:25:02.457977+0000","last_became_active":"2023-11-24T08:25:00.312731+0000","last_became_peered":"2023-11-24T08:25:00.312731+0000","last_unstale":"2023-11-24T08:25:02.457977+0000","last_undegraded":"2023-11-24T08:25:02.457977+0000","last_fullsized":"2023-11-24T08:25:02.457977+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:16.865134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379752+0000","last_change":"2023-11-24T08:25:01.321460+0000","last_active":"2023-11-24T08:25:02.379752+0000","last_peered":"2023-11-24T08:25:02.379752+0000","last_clean":"2023-11-24T08:25:02.379752+0000","last_became_active":"2023-11-24T08:25:01.320125+0000","last_became_peered":"2023-11-24T08:25:01.320125+0000","last_unstale":"2023-11-24T08:25:02.379752+0000","last_undegraded":"2023-11-24T08:25:02.379752+0000","last_fullsized":"2023-11-24T08:25:02.379752+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:04.413433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326194+0000","last_change":"2023-11-24T08:25:00.314601+0000","last_active":"2023-11-24T08:25:02.326194+0000","last_peered":"2023-11-24T08:25:02.326194+0000","last_clean":"2023-11-24T08:25:02.326194+0000","last_became_active":"2023-11-24T08:25:00.314286+0000","last_became_peered":"2023-11-24T08:25:00.314286+0000","last_unstale":"2023-11-24T08:25:02.326194+0000","last_undegraded":"2023-11-24T08:25:02.326194+0000","last_fullsized":"2023-11-24T08:25:02.326194+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:56.174973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458939+0000","last_change":"2023-11-24T08:25:01.334173+0000","last_active":"2023-11-24T08:25:02.458939+0000","last_peered":"2023-11-24T08:25:02.458939+0000","last_clean":"2023-11-24T08:25:02.458939+0000","last_became_active":"2023-11-24T08:25:01.333892+0000","last_became_peered":"2023-11-24T08:25:01.333892+0000","last_unstale":"2023-11-24T08:25:02.458939+0000","last_undegraded":"2023-11-24T08:25:02.458939+0000","last_fullsized":"2023-11-24T08:25:02.458939+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:29:28.813896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326174+0000","last_change":"2023-11-24T08:25:00.320831+0000","last_active":"2023-11-24T08:25:02.326174+0000","last_peered":"2023-11-24T08:25:02.326174+0000","last_clean":"2023-11-24T08:25:02.326174+0000","last_became_active":"2023-11-24T08:25:00.320455+0000","last_became_peered":"2023-11-24T08:25:00.320455+0000","last_unstale":"2023-11-24T08:25:02.326174+0000","last_undegraded":"2023-11-24T08:25:02.326174+0000","last_fullsized":"2023-11-24T08:25:02.326174+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:24.236291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459061+0000","last_change":"2023-11-24T08:25:01.321225+0000","last_active":"2023-11-24T08:25:02.459061+0000","last_peered":"2023-11-24T08:25:02.459061+0000","last_clean":"2023-11-24T08:25:02.459061+0000","last_became_active":"2023-11-24T08:25:01.320895+0000","last_became_peered":"2023-11-24T08:25:01.320895+0000","last_unstale":"2023-11-24T08:25:02.459061+0000","last_undegraded":"2023-11-24T08:25:02.459061+0000","last_fullsized":"2023-11-24T08:25:02.459061+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:21.689443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379436+0000","last_change":"2023-11-24T08:25:00.317132+0000","last_active":"2023-11-24T08:25:02.379436+0000","last_peered":"2023-11-24T08:25:02.379436+0000","last_clean":"2023-11-24T08:25:02.379436+0000","last_became_active":"2023-11-24T08:25:00.316463+0000","last_became_peered":"2023-11-24T08:25:00.316463+0000","last_unstale":"2023-11-24T08:25:02.379436+0000","last_undegraded":"2023-11-24T08:25:02.379436+0000","last_fullsized":"2023-11-24T08:25:02.379436+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:12.427710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460682+0000","last_change":"2023-11-24T08:25:01.309890+0000","last_active":"2023-11-24T08:25:02.460682+0000","last_peered":"2023-11-24T08:25:02.460682+0000","last_clean":"2023-11-24T08:25:02.460682+0000","last_became_active":"2023-11-24T08:25:01.307813+0000","last_became_peered":"2023-11-24T08:25:01.307813+0000","last_unstale":"2023-11-24T08:25:02.460682+0000","last_undegraded":"2023-11-24T08:25:02.460682+0000","last_fullsized":"2023-11-24T08:25:02.460682+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:18.092625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458475+0000","last_change":"2023-11-24T08:25:00.313095+0000","last_active":"2023-11-24T08:25:02.458475+0000","last_peered":"2023-11-24T08:25:02.458475+0000","last_clean":"2023-11-24T08:25:02.458475+0000","last_became_active":"2023-11-24T08:25:00.312561+0000","last_became_peered":"2023-11-24T08:25:00.312561+0000","last_unstale":"2023-11-24T08:25:02.458475+0000","last_undegraded":"2023-11-24T08:25:02.458475+0000","last_fullsized":"2023-11-24T08:25:02.458475+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:23.528675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362649+0000","last_change":"2023-11-24T08:25:01.318829+0000","last_active":"2023-11-24T08:25:02.362649+0000","last_peered":"2023-11-24T08:25:02.362649+0000","last_clean":"2023-11-24T08:25:02.362649+0000","last_became_active":"2023-11-24T08:25:01.318034+0000","last_became_peered":"2023-11-24T08:25:01.318034+0000","last_unstale":"2023-11-24T08:25:02.362649+0000","last_undegraded":"2023-11-24T08:25:02.362649+0000","last_fullsized":"2023-11-24T08:25:02.362649+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:32.983090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325288+0000","last_change":"2023-11-24T08:25:00.315474+0000","last_active":"2023-11-24T08:25:02.325288+0000","last_peered":"2023-11-24T08:25:02.325288+0000","last_clean":"2023-11-24T08:25:02.325288+0000","last_became_active":"2023-11-24T08:25:00.315155+0000","last_became_peered":"2023-11-24T08:25:00.315155+0000","last_unstale":"2023-11-24T08:25:02.325288+0000","last_undegraded":"2023-11-24T08:25:02.325288+0000","last_fullsized":"2023-11-24T08:25:02.325288+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:22.161492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379478+0000","last_change":"2023-11-24T08:25:01.332535+0000","last_active":"2023-11-24T08:25:02.379478+0000","last_peered":"2023-11-24T08:25:02.379478+0000","last_clean":"2023-11-24T08:25:02.379478+0000","last_became_active":"2023-11-24T08:25:01.332223+0000","last_became_peered":"2023-11-24T08:25:01.332223+0000","last_unstale":"2023-11-24T08:25:02.379478+0000","last_undegraded":"2023-11-24T08:25:02.379478+0000","last_fullsized":"2023-11-24T08:25:02.379478+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:24:14.125930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327189+0000","last_change":"2023-11-24T08:25:00.320354+0000","last_active":"2023-11-24T08:25:02.327189+0000","last_peered":"2023-11-24T08:25:02.327189+0000","last_clean":"2023-11-24T08:25:02.327189+0000","last_became_active":"2023-11-24T08:25:00.319020+0000","last_became_peered":"2023-11-24T08:25:00.319020+0000","last_unstale":"2023-11-24T08:25:02.327189+0000","last_undegraded":"2023-11-24T08:25:02.327189+0000","last_fullsized":"2023-11-24T08:25:02.327189+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:19.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458680+0000","last_change":"2023-11-24T08:25:01.317528+0000","last_active":"2023-11-24T08:25:02.458680+0000","last_peered":"2023-11-24T08:25:02.458680+0000","last_clean":"2023-11-24T08:25:02.458680+0000","last_became_active":"2023-11-24T08:25:01.317198+0000","last_became_peered":"2023-11-24T08:25:01.317198+0000","last_unstale":"2023-11-24T08:25:02.458680+0000","last_undegraded":"2023-11-24T08:25:02.458680+0000","last_fullsized":"2023-11-24T08:25:02.458680+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:32.430121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327401+0000","last_change":"2023-11-24T08:25:00.319830+0000","last_active":"2023-11-24T08:25:02.327401+0000","last_peered":"2023-11-24T08:25:02.327401+0000","last_clean":"2023-11-24T08:25:02.327401+0000","last_became_active":"2023-11-24T08:25:00.319542+0000","last_became_peered":"2023-11-24T08:25:00.319542+0000","last_unstale":"2023-11-24T08:25:02.327401+0000","last_undegraded":"2023-11-24T08:25:02.327401+0000","last_fullsized":"2023-11-24T08:25:02.327401+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:42:43.615118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379315+0000","last_change":"2023-11-24T08:25:01.332499+0000","last_active":"2023-11-24T08:25:02.379315+0000","last_peered":"2023-11-24T08:25:02.379315+0000","last_clean":"2023-11-24T08:25:02.379315+0000","last_became_active":"2023-11-24T08:25:01.332200+0000","last_became_peered":"2023-11-24T08:25:01.332200+0000","last_unstale":"2023-11-24T08:25:02.379315+0000","last_undegraded":"2023-11-24T08:25:02.379315+0000","last_fullsized":"2023-11-24T08:25:02.379315+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:39.750205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457878+0000","last_change":"2023-11-24T08:25:00.314997+0000","last_active":"2023-11-24T08:25:02.457878+0000","last_peered":"2023-11-24T08:25:02.457878+0000","last_clean":"2023-11-24T08:25:02.457878+0000","last_became_active":"2023-11-24T08:25:00.314683+0000","last_became_peered":"2023-11-24T08:25:00.314683+0000","last_unstale":"2023-11-24T08:25:02.457878+0000","last_undegraded":"2023-11-24T08:25:02.457878+0000","last_fullsized":"2023-11-24T08:25:02.457878+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:18:30.024631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460418+0000","last_change":"2023-11-24T08:25:01.319241+0000","last_active":"2023-11-24T08:25:02.460418+0000","last_peered":"2023-11-24T08:25:02.460418+0000","last_clean":"2023-11-24T08:25:02.460418+0000","last_became_active":"2023-11-24T08:25:01.318938+0000","last_became_peered":"2023-11-24T08:25:01.318938+0000","last_unstale":"2023-11-24T08:25:02.460418+0000","last_undegraded":"2023-11-24T08:25:02.460418+0000","last_fullsized":"2023-11-24T08:25:02.460418+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:32.737727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327319+0000","last_change":"2023-11-24T08:25:00.321938+0000","last_active":"2023-11-24T08:25:02.327319+0000","last_peered":"2023-11-24T08:25:02.327319+0000","last_clean":"2023-11-24T08:25:02.327319+0000","last_became_active":"2023-11-24T08:25:00.320417+0000","last_became_peered":"2023-11-24T08:25:00.320417+0000","last_unstale":"2023-11-24T08:25:02.327319+0000","last_undegraded":"2023-11-24T08:25:02.327319+0000","last_fullsized":"2023-11-24T08:25:02.327319+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:55.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460726+0000","last_change":"2023-11-24T08:25:01.331162+0000","last_active":"2023-11-24T08:25:02.460726+0000","last_peered":"2023-11-24T08:25:02.460726+0000","last_clean":"2023-11-24T08:25:02.460726+0000","last_became_active":"2023-11-24T08:25:01.329796+0000","last_became_peered":"2023-11-24T08:25:01.329796+0000","last_unstale":"2023-11-24T08:25:02.460726+0000","last_undegraded":"2023-11-24T08:25:02.460726+0000","last_fullsized":"2023-11-24T08:25:02.460726+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:01.668751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326865+0000","last_change":"2023-11-24T08:25:00.317352+0000","last_active":"2023-11-24T08:25:02.326865+0000","last_peered":"2023-11-24T08:25:02.326865+0000","last_clean":"2023-11-24T08:25:02.326865+0000","last_became_active":"2023-11-24T08:25:00.317091+0000","last_became_peered":"2023-11-24T08:25:00.317091+0000","last_unstale":"2023-11-24T08:25:02.326865+0000","last_undegraded":"2023-11-24T08:25:02.326865+0000","last_fullsized":"2023-11-24T08:25:02.326865+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:00.823837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379859+0000","last_change":"2023-11-24T08:25:01.319950+0000","last_active":"2023-11-24T08:25:02.379859+0000","last_peered":"2023-11-24T08:25:02.379859+0000","last_clean":"2023-11-24T08:25:02.379859+0000","last_became_active":"2023-11-24T08:25:01.319517+0000","last_became_peered":"2023-11-24T08:25:01.319517+0000","last_unstale":"2023-11-24T08:25:02.379859+0000","last_undegraded":"2023-11-24T08:25:02.379859+0000","last_fullsized":"2023-11-24T08:25:02.379859+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:51.707800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379481+0000","last_change":"2023-11-24T08:25:00.317308+0000","last_active":"2023-11-24T08:25:02.379481+0000","last_peered":"2023-11-24T08:25:02.379481+0000","last_clean":"2023-11-24T08:25:02.379481+0000","last_became_active":"2023-11-24T08:25:00.316866+0000","last_became_peered":"2023-11-24T08:25:00.316866+0000","last_unstale":"2023-11-24T08:25:02.379481+0000","last_undegraded":"2023-11-24T08:25:02.379481+0000","last_fullsized":"2023-11-24T08:25:02.379481+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:50:08.003236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362635+0000","last_change":"2023-11-24T08:25:01.323588+0000","last_active":"2023-11-24T08:25:02.362635+0000","last_peered":"2023-11-24T08:25:02.362635+0000","last_clean":"2023-11-24T08:25:02.362635+0000","last_became_active":"2023-11-24T08:25:01.323314+0000","last_became_peered":"2023-11-24T08:25:01.323314+0000","last_unstale":"2023-11-24T08:25:02.362635+0000","last_undegraded":"2023-11-24T08:25:02.362635+0000","last_fullsized":"2023-11-24T08:25:02.362635+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:36.774886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458859+0000","last_change":"2023-11-24T08:25:00.318842+0000","last_active":"2023-11-24T08:25:02.458859+0000","last_peered":"2023-11-24T08:25:02.458859+0000","last_clean":"2023-11-24T08:25:02.458859+0000","last_became_active":"2023-11-24T08:25:00.318568+0000","last_became_peered":"2023-11-24T08:25:00.318568+0000","last_unstale":"2023-11-24T08:25:02.458859+0000","last_undegraded":"2023-11-24T08:25:02.458859+0000","last_fullsized":"2023-11-24T08:25:02.458859+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:46.569819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379611+0000","last_change":"2023-11-24T08:25:01.316023+0000","last_active":"2023-11-24T08:25:02.379611+0000","last_peered":"2023-11-24T08:25:02.379611+0000","last_clean":"2023-11-24T08:25:02.379611+0000","last_became_active":"2023-11-24T08:25:01.315713+0000","last_became_peered":"2023-11-24T08:25:01.315713+0000","last_unstale":"2023-11-24T08:25:02.379611+0000","last_undegraded":"2023-11-24T08:25:02.379611+0000","last_fullsized":"2023-11-24T08:25:02.379611+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:01.848599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325026+0000","last_change":"2023-11-24T08:25:00.315354+0000","last_active":"2023-11-24T08:25:02.325026+0000","last_peered":"2023-11-24T08:25:02.325026+0000","last_clean":"2023-11-24T08:25:02.325026+0000","last_became_active":"2023-11-24T08:25:00.315072+0000","last_became_peered":"2023-11-24T08:25:00.315072+0000","last_unstale":"2023-11-24T08:25:02.325026+0000","last_undegraded":"2023-11-24T08:25:02.325026+0000","last_fullsized":"2023-11-24T08:25:02.325026+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:03.395465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326637+0000","last_change":"2023-11-24T08:25:01.321323+0000","last_active":"2023-11-24T08:25:02.326637+0000","last_peered":"2023-11-24T08:25:02.326637+0000","last_clean":"2023-11-24T08:25:02.326637+0000","last_became_active":"2023-11-24T08:25:01.319285+0000","last_became_peered":"2023-11-24T08:25:01.319285+0000","last_unstale":"2023-11-24T08:25:02.326637+0000","last_undegraded":"2023-11-24T08:25:02.326637+0000","last_fullsized":"2023-11-24T08:25:02.326637+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:45.471299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":368,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-11-24T08:25:01.299233+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:25:01.299233+0000","last_peered":"2023-11-24T08:25:01.299233+0000","last_clean":"2023-11-24T08:25:01.299233+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:25:01.299233+0000","last_undegraded":"2023-11-24T08:25:01.299233+0000","last_fullsized":"2023-11-24T08:25:01.299233+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43184,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328656,"statfs":{"total":96636764160,"available":96592543744,"internally_reserved":0,"allocated":3768320,"data_stored":1997676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.42399999999999999},"max":{"1min":1.6830000000000001,"5min":2.048,"15min":2.7240000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.085,"15min":1.087},"min":{"1min":0.65200000000000002,"5min":0.61899999999999999,"15min":0.53700000000000003},"max":{"1min":1.464,"5min":1.899,"15min":2.3340000000000001},"last":1.117}]},{"osd":1,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.115,"15min":1.1479999999999999},"min":{"1min":0.60299999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4750000000000001,"5min":1.681,"15min":3.8519999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.175,"15min":1.204},"min":{"1min":0.752,"5min":0.68700000000000006,"15min":0.66600000000000004},"max":{"1min":1.6140000000000001,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.004}]},{"osd":2,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1020000000000001,"15min":1.1120000000000001},"min":{"1min":0.68899999999999995,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.7150000000000001,"5min":1.762,"15min":2.4430000000000001},"last":1.528},{"interface":"front","average":{"1min":1.256,"5min":1.232,"15min":1.2849999999999999},"min":{"1min":0.70599999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.617,"5min":1.8,"15min":4.0650000000000004},"last":1.238}]},{"osd":3,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1910000000000001,"15min":1.2130000000000001},"min":{"1min":0.747,"5min":0.64300000000000002,"15min":0.60199999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":3.133},"last":1.302},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.296,"15min":1.3540000000000001},"min":{"1min":0.78700000000000003,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.6080000000000001,"5min":2.173,"15min":3.9209999999999998},"last":1.337}]},{"osd":4,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.282,"15min":1.325},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.712,"5min":1.8799999999999999,"15min":2.665},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.327,"5min":1.302,"15min":1.3400000000000001},"min":{"1min":0.54500000000000004,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.6830000000000001,"5min":1.7130000000000001,"15min":3.4870000000000001},"last":1.2010000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4059999999999999,"15min":1.454},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.42699999999999999},"max":{"1min":1.861,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4330000000000001,"15min":1.468},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.264}]},{"osd":6,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.60199999999999998},"max":{"1min":1.8069999999999999,"5min":2.2610000000000001,"15min":2.992},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.411,"15min":1.4790000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.871,"5min":1.871,"15min":3.798},"last":1.446}]}]},{"osd":6,"up_from":13,"seq":55834575095,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42580,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329260,"statfs":{"total":96636764160,"available":96593162240,"internally_reserved":0,"allocated":3153920,"data_stored":1405044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.492,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.026,"5min":1.2030000000000001,"15min":1.123},"min":{"1min":0.66800000000000004,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.5609999999999999,"5min":13.516999999999999,"15min":13.516999999999999},"last":0.74299999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.131,"15min":1.1459999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.61599999999999999},"max":{"1min":1.423,"5min":5.7050000000000001,"15min":5.7050000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.2809999999999999,"15min":1.2210000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.65000000000000002},"max":{"1min":1.679,"5min":13.57,"15min":13.57},"last":1.363}]},{"osd":2,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.125,"15min":1.1020000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.61899999999999999},"max":{"1min":1.679,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2370000000000001,"15min":1.2549999999999999},"min":{"1min":0.73699999999999999,"5min":0.72199999999999998,"15min":0.60399999999999998},"max":{"1min":1.603,"5min":5.3440000000000003,"15min":5.3440000000000003},"last":1.3089999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.544,"15min":1.409},"min":{"1min":0.93700000000000006,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7729999999999999,"5min":23.989000000000001,"15min":23.989000000000001},"last":1.54},{"interface":"front","average":{"1min":1.28,"5min":1.496,"15min":1.3720000000000001},"min":{"1min":0.93799999999999994,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.99399999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.4530000000000001,"15min":1.4390000000000001},"min":{"1min":0.76800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.7969999999999999,"5min":11.73,"15min":11.73},"last":1.415},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.617,"15min":1.526},"min":{"1min":0.76500000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.1800000000000002,"5min":20.173999999999999,"15min":20.173999999999999},"last":1.6180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3819999999999999,"15min":1.4079999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.835,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5700000000000001,"15min":1.5649999999999999},"min":{"1min":0.76500000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.97,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":1.75}]},{"osd":7,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.653,"15min":1.611},"min":{"1min":0.83699999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.0659999999999998,"5min":13.612,"15min":13.612},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.5029999999999999,"15min":1.5600000000000001},"min":{"1min":0.94199999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.948,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.6000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3162112,"data_stored":1405100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.008,"15min":1.018},"min":{"1min":0.65900000000000003,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.72,"5min":1.72,"15min":4.9850000000000003},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1040000000000001,"15min":1.1000000000000001},"min":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.5449999999999999,"5min":1.6639999999999999,"15min":5.4299999999999997},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.123,"15min":1.1339999999999999},"min":{"1min":0.755,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5660000000000001,"5min":1.665,"15min":7.258},"last":1.145},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1819999999999999,"15min":1.198},"min":{"1min":0.75700000000000001,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.631,"5min":1.659,"15min":6.5199999999999996},"last":1.5189999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.181,"15min":1.1819999999999999},"min":{"1min":0.85899999999999999,"5min":0.68100000000000005,"15min":0.65100000000000002},"max":{"1min":1.6240000000000001,"5min":1.7829999999999999,"15min":6.8929999999999998},"last":1.294},{"interface":"front","average":{"1min":1.147,"5min":1.1599999999999999,"15min":1.1719999999999999},"min":{"1min":0.746,"5min":0.72399999999999998,"15min":0.68000000000000005},"max":{"1min":1.536,"5min":1.923,"15min":8.0869999999999997},"last":0.99299999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.151,"15min":1.1739999999999999},"min":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.66600000000000004},"max":{"1min":1.4199999999999999,"5min":1.9670000000000001,"15min":9.0120000000000005},"last":1.087},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.98999999999999999,"5min":0.80100000000000005,"15min":0.66800000000000004},"max":{"1min":1.7709999999999999,"5min":1.956,"15min":7.2130000000000001},"last":1.028}]},{"osd":4,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.397,"15min":1.3919999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.60799999999999998},"max":{"1min":1.7130000000000001,"5min":2.536,"15min":11.813000000000001},"last":1.204},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.4710000000000001,"15min":1.484},"min":{"1min":1.284,"5min":0.88100000000000001,"15min":0.65900000000000003},"max":{"1min":1.855,"5min":2.4550000000000001,"15min":12.327999999999999},"last":1.446}]},{"osd":6,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2989999999999999,"15min":1.337},"min":{"1min":0.94199999999999995,"5min":0.81899999999999995,"15min":0.48899999999999999},"max":{"1min":1.6890000000000001,"5min":2.246,"15min":9.4770000000000003},"last":1.486},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4159999999999999,"15min":1.4510000000000001},"min":{"1min":1.006,"5min":0.88300000000000001,"15min":0.69299999999999995},"max":{"1min":1.8200000000000001,"5min":2.3519999999999999,"15min":9.9019999999999992},"last":1.5980000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4770000000000001,"15min":1.494},"min":{"1min":1.0129999999999999,"5min":0.877,"15min":0.54300000000000004},"max":{"1min":1.782,"5min":2.802,"15min":8.1660000000000004},"last":1.355},{"interface":"front","average":{"1min":1.593,"5min":1.6040000000000001,"15min":1.6499999999999999},"min":{"1min":1.2430000000000001,"5min":1.099,"15min":0.75600000000000001},"max":{"1min":1.9510000000000001,"5min":2.9489999999999998,"15min":19.663},"last":1.651}]}]},{"osd":4,"up_from":13,"seq":55834575097,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":3170304,"data_stored":1406746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0720000000000001,"15min":1.054},"min":{"1min":0.72799999999999998,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.5649999999999999,"5min":1.6160000000000001,"15min":1.8779999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.123,"5min":1.0960000000000001,"15min":1.0820000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.46100000000000002},"max":{"1min":1.526,"5min":1.544,"15min":5.452},"last":1.0489999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1739999999999999,"15min":1.137},"min":{"1min":0.79200000000000004,"5min":0.69899999999999995,"15min":0.46300000000000002},"max":{"1min":1.827,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.194},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.137,"15min":1.1359999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.51900000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.988},"last":1.1659999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.083,"15min":1.069},"min":{"1min":0.90500000000000003,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.4830000000000001,"5min":1.5209999999999999,"15min":1.7929999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.2969999999999999,"15min":1.2509999999999999},"min":{"1min":0.80500000000000005,"5min":0.754,"15min":0.505},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.4430000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.357,"15min":1.337},"min":{"1min":0.91200000000000003,"5min":0.67900000000000005,"15min":0.62},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.306,"15min":1.282},"min":{"1min":0.83199999999999996,"5min":0.69399999999999995,"15min":0.55800000000000005},"max":{"1min":1.968,"5min":1.968,"15min":2.3340000000000001},"last":1.286}]},{"osd":5,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.69299999999999995},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":2.2429999999999999},"last":2.2269999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.4350000000000001,"15min":1.4219999999999999},"min":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.66200000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.8090000000000002},"last":1.782}]},{"osd":6,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3120000000000001,"15min":1.264},"min":{"1min":0.89500000000000002,"5min":0.66400000000000003,"15min":0.55400000000000005},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.736},{"interface":"front","average":{"1min":1.627,"5min":1.5720000000000001,"15min":1.546},"min":{"1min":0.98299999999999998,"5min":0.69499999999999995,"15min":0.58099999999999996},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":2.3809999999999998}]},{"osd":7,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.4890000000000001,"15min":1.48},"min":{"1min":1.079,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":3.1219999999999999},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.462,"15min":1.425},"min":{"1min":1.101,"5min":0.72999999999999998,"15min":0.61799999999999999},"max":{"1min":2.0249999999999999,"5min":2.1190000000000002,"15min":2.3100000000000001},"last":1.849}]}]},{"osd":3,"up_from":13,"seq":55834575096,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42592,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329248,"statfs":{"total":96636764160,"available":96593149952,"internally_reserved":0,"allocated":3162112,"data_stored":1405136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.85799999999999998,"15min":0.83499999999999996},"min":{"1min":0.496,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":1.5620000000000001,"5min":1.5820000000000001,"15min":3.181},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.47299999999999998,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.248,"5min":1.5349999999999999,"15min":1.7310000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79300000000000004,"15min":0.80100000000000005},"min":{"1min":0.38400000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":2.101},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.95999999999999996,"15min":0.93400000000000005},"min":{"1min":0.52600000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":3.3260000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91700000000000004,"15min":0.89100000000000001},"min":{"1min":0.53000000000000003,"5min":0.46400000000000002,"15min":0.39900000000000002},"max":{"1min":1.3600000000000001,"5min":1.976,"15min":2.3700000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0409999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.45600000000000002,"15min":0.45200000000000001},"max":{"1min":1.851,"5min":1.851,"15min":3.2480000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2769999999999999,"15min":1.256},"min":{"1min":0.81999999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.786,"5min":1.845,"15min":2.1840000000000002},"last":1.04},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.242,"15min":1.2170000000000001},"min":{"1min":0.93200000000000005,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.5369999999999999,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.2969999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2909999999999999,"15min":1.2689999999999999},"min":{"1min":0.69899999999999995,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":3.4399999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.262,"15min":1.2270000000000001},"min":{"1min":0.81999999999999995,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.992},"last":1.3660000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.3540000000000001,"15min":1.3300000000000001},"min":{"1min":0.82799999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":3.5030000000000001},"last":1.165},{"interface":"front","average":{"1min":1.268,"5min":1.266,"15min":1.246},"min":{"1min":0.75900000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8260000000000001,"5min":1.8420000000000001,"15min":2.4060000000000001},"last":1.4139999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3280000000000001,"15min":1.3049999999999999},"min":{"1min":0.78700000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":2.012,"5min":2.0369999999999999,"15min":2.1030000000000002},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.355,"15min":1.3140000000000001},"min":{"1min":0.73299999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":3.331},"last":1.6160000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575094,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42584,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329256,"statfs":{"total":96636764160,"available":96593158144,"internally_reserved":0,"allocated":3153920,"data_stored":1405044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81000000000000005,"15min":0.76100000000000001},"min":{"1min":0.41099999999999998,"5min":0.38200000000000001,"15min":0.29999999999999999},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.84899999999999998,"15min":0.85399999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.601,"5min":1.601,"15min":1.782},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.91900000000000004,"15min":0.88800000000000001},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.7809999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0089999999999999,"15min":0.95099999999999996},"min":{"1min":0.52800000000000002,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88500000000000001,"15min":0.85799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.4370000000000001,"5min":1.4530000000000001,"15min":1.6850000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0569999999999999,"15min":1.014},"min":{"1min":0.53500000000000003,"5min":0.52100000000000002,"15min":0.41399999999999998},"max":{"1min":1.6839999999999999,"5min":1.948,"15min":1.948},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.288,"15min":1.268},"min":{"1min":0.625,"5min":0.625,"15min":0.53700000000000003},"max":{"1min":1.6919999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2769999999999999,"15min":1.24},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7050000000000001,"5min":1.8200000000000001,"15min":1.921},"last":1.026}]},{"osd":5,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.238,"15min":1.224},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.61499999999999999},"max":{"1min":1.641,"5min":1.821,"15min":2.3149999999999999},"last":1},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2629999999999999,"15min":1.226},"min":{"1min":0.79800000000000004,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.5,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":0.82199999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.226,"15min":1.1910000000000001},"min":{"1min":0.81899999999999995,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.5760000000000001,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.335,"15min":1.3},"min":{"1min":0.82299999999999995,"5min":0.82199999999999995,"15min":0.65600000000000003},"max":{"1min":1.9950000000000001,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.085}]},{"osd":7,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.321,"15min":1.3009999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.56399999999999995},"max":{"1min":1.839,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.3200000000000001,"15min":1.2729999999999999},"min":{"1min":0.83999999999999997,"5min":0.78300000000000003,"15min":0.746},"max":{"1min":2.1419999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.095}]}]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3158016,"data_stored":1405134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.82699999999999996},"min":{"1min":0.48899999999999999,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":1.1779999999999999,"5min":1.4490000000000001,"15min":1.7749999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.97099999999999997,"15min":0.92100000000000004},"min":{"1min":0.51400000000000001,"5min":0.46800000000000003,"15min":0.35499999999999998},"max":{"1min":1.373,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.82299999999999995,"15min":0.81399999999999995},"min":{"1min":0.435,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":1.2989999999999999,"5min":1.823,"15min":1.861},"last":0.432},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.95999999999999996,"15min":0.93300000000000005},"min":{"1min":0.49399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3140000000000001,"5min":1.754,"15min":1.925},"last":0.379}]},{"osd":3,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.0389999999999999,"15min":1.0149999999999999},"min":{"1min":0.53900000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6559999999999999,"5min":1.7569999999999999,"15min":2.4169999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93500000000000005,"15min":0.91900000000000004},"min":{"1min":0.52200000000000002,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.407,"5min":1.8620000000000001,"15min":2.3879999999999999},"last":0.497}]},{"osd":4,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3560000000000001,"15min":1.2989999999999999},"min":{"1min":0.90900000000000003,"5min":0.66700000000000004,"15min":0.65300000000000002},"max":{"1min":1.8779999999999999,"5min":1.9099999999999999,"15min":2.028},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.238,"5min":1.3220000000000001,"15min":1.2729999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.55900000000000005},"max":{"1min":1.736,"5min":2.0750000000000002,"15min":2.141},"last":1.036}]},{"osd":5,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2789999999999999,"15min":1.258},"min":{"1min":0.78100000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8460000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.004},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.79200000000000004,"5min":0.72699999999999998,"15min":0.57299999999999995},"max":{"1min":1.806,"5min":2.1499999999999999,"15min":2.246},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.2709999999999999,"15min":1.28},"min":{"1min":0.81999999999999995,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.72,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3100000000000001,"15min":1.2789999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.7090000000000001,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.246}]},{"osd":7,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.321,"15min":1.321},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.59099999999999997},"max":{"1min":1.7849999999999999,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.375,"15min":1.3360000000000001},"min":{"1min":0.79000000000000004,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":1.871,"5min":2.1000000000000001,"15min":3.653},"last":1.0189999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575096,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43176,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328664,"statfs":{"total":96636764160,"available":96592551936,"internally_reserved":0,"allocated":3756032,"data_stored":1996028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73799999999999999,"15min":0.73299999999999998},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":1.3260000000000001,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":0.38},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.84899999999999998,"15min":0.82499999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.173,"5min":1.607,"15min":2.3620000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0289999999999999,"15min":1.0109999999999999},"min":{"1min":0.622,"5min":0.54200000000000004,"15min":0.41399999999999998},"max":{"1min":2.7480000000000002,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.90900000000000003,"15min":0.876},"min":{"1min":0.443,"5min":0.443,"15min":0.39400000000000002},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.86399999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97899999999999998,"15min":0.95899999999999996},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.88400000000000001,"15min":0.88900000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.625,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.244,"15min":1.1990000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.716,"5min":2.2210000000000001,"15min":2.3010000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.121,"15min":1.113},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.67800000000000005},"max":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.153,"15min":1.1539999999999999},"min":{"1min":0.80400000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":2.6850000000000001},"last":0.998},{"interface":"front","average":{"1min":1.25,"5min":1.1679999999999999,"15min":1.1619999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.1930000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.286,"15min":1.266},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.02},{"interface":"front","average":{"1min":1.28,"5min":1.2729999999999999,"15min":1.2549999999999999},"min":{"1min":0.77600000000000002,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.873,"5min":1.873,"15min":2.823},"last":0.753}]},{"osd":7,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.3149999999999999,"15min":1.327},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9119999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.145},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.304,"15min":1.3029999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9670000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.1639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:25:05.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T08:25:06.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.029+0000 7f444d6b4640 1 Processor -- start 2023-11-24T08:25:06.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.030+0000 7f444d6b4640 1 -- start start 2023-11-24T08:25:06.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.031+0000 7f444d6b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.031+0000 7f444d6b4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 0x7f44480ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.031+0000 7f444d6b4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444815ab80 con 0x7f44480cd470 2023-11-24T08:25:06.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f444d6b4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444815a990 con 0x7f44480b2630 2023-11-24T08:25:06.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f444d6b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444815e680 con 0x7f44480b2250 2023-11-24T08:25:06.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f443e7fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 0x7f44480ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:57990/0 (socket says 172.21.15.107:57990) 2023-11-24T08:25:06.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.032+0000 7f4446ffd640 1 -- 172.21.15.107:0/431146719 learned_addr learned my addr 172.21.15.107:0/431146719 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:06.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.033+0000 7f4446ffd640 1 -- 172.21.15.107:0/431146719 >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 msgr2=0x7f44480ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:06.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.033+0000 7f4446ffd640 1 --2- 172.21.15.107:0/431146719 >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 0x7f44480ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.033+0000 7f4446ffd640 1 -- 172.21.15.107:0/431146719 >> v1:172.21.15.120:6789/0 conn(0x7f44480cd470 legacy=0x7f444815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.033+0000 7f4446ffd640 1 -- 172.21.15.107:0/431146719 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444815e960 con 0x7f44480b2250 2023-11-24T08:25:06.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.033+0000 7f443e7fc640 1 --2- 172.21.15.107:0/431146719 >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 0x7f44480ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T08:25:06.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.034+0000 7f4446ffd640 1 --2- 172.21.15.107:0/431146719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4438009870 tx=0x7f443802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23ed0f64b1c316 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.035+0000 7f44467fc640 1 -- 172.21.15.107:0/431146719 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438030840 con 0x7f44480b2250 2023-11-24T08:25:06.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.035+0000 7f44467fc640 1 -- 172.21.15.107:0/431146719 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44380309a0 con 0x7f44480b2250 2023-11-24T08:25:06.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.035+0000 7f44467fc640 1 -- 172.21.15.107:0/431146719 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438030d30 con 0x7f44480b2250 2023-11-24T08:25:06.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 -- 172.21.15.107:0/431146719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 msgr2=0x7f444815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 --2- 172.21.15.107:0/431146719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4438009870 tx=0x7f443802b980 comp rx=0 tx=0).stop 2023-11-24T08:25:06.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 -- 172.21.15.107:0/431146719 shutdown_connections 2023-11-24T08:25:06.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 --2- 172.21.15.107:0/431146719 >> v2:172.21.15.120:3300/0 conn(0x7f44480b2630 0x7f44480ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 --2- 172.21.15.107:0/431146719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f444815df80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 -- 172.21.15.107:0/431146719 >> 172.21.15.107:0/431146719 conn(0x7f444806a920 msgr2=0x7f444806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:06.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 -- 172.21.15.107:0/431146719 shutdown_connections 2023-11-24T08:25:06.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.036+0000 7f444d6b4640 1 -- 172.21.15.107:0/431146719 wait complete. 2023-11-24T08:25:06.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.038+0000 7f444d6b4640 1 Processor -- start 2023-11-24T08:25:06.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.038+0000 7f444d6b4640 1 -- start start 2023-11-24T08:25:06.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.039+0000 7f444d6b4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f44480b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.039+0000 7f444d6b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44480b2630 0x7f44480afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.039+0000 7f444d6b4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.039+0000 7f444d6b4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44480b0290 con 0x7f44480cd470 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.039+0000 7f444d6b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44480b0510 con 0x7f44480b2630 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f444d6b4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44480b07f0 con 0x7f44480b2250 2023-11-24T08:25:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60204/0 (socket says 172.21.15.107:60204) 2023-11-24T08:25:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 -- 172.21.15.107:0/1648668589 learned_addr learned my addr 172.21.15.107:0/1648668589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f443e7fc640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44480b2630 0x7f44480afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f4446ffd640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f44480b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 -- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 msgr2=0x7f44480b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f44480b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 -- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44480b2630 msgr2=0x7f44480afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44480b2630 0x7f44480afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.040+0000 7f44477fe640 1 -- 172.21.15.107:0/1648668589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443000b6a0 con 0x7f44480cd470 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f44477fe640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f444000c730 tx=0x7f444000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440003b70 con 0x7f44480cd470 2023-11-24T08:25:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44380094d0 con 0x7f44480cd470 2023-11-24T08:25:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440003cf0 con 0x7f44480cd470 2023-11-24T08:25:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440003ea0 con 0x7f44480cd470 2023-11-24T08:25:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.041+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44480ac780 con 0x7f44480cd470 2023-11-24T08:25:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.042+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44480ac9a0 con 0x7f44480cd470 2023-11-24T08:25:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.044+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f4440005310 con 0x7f44480cd470 2023-11-24T08:25:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.044+0000 7f443ffff640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 0x7f442806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.045+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4440088820 con 0x7f44480cd470 2023-11-24T08:25:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.045+0000 7f4446ffd640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 0x7f442806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.046+0000 7f4446ffd640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 0x7f442806fb20 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4438036000 tx=0x7f4438008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.051+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44400552b0 con 0x7f44480cd470 2023-11-24T08:25:06.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.236+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4448158720 con 0x7f442806d660 2023-11-24T08:25:06.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.258+0000 7f443ffff640 1 -- 172.21.15.107:0/1648668589 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287713 (secure 0 0 0) 0x7f4448158720 con 0x7f442806d660 2023-11-24T08:25:06.263 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-11-24T08:25:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 msgr2=0x7f442806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 0x7f442806fb20 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4438036000 tx=0x7f4438008030 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 msgr2=0x7f4448152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f444000c730 tx=0x7f444000cbf0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 shutdown_connections 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f442806d660 0x7f442806fb20 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44480cd470 0x7f4448152aa0 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44480b2630 0x7f44480afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 --2- 172.21.15.107:0/1648668589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44480b2250 0x7f44480b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 >> 172.21.15.107:0/1648668589 conn(0x7f444806a920 msgr2=0x7f44481596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 shutdown_connections 2023-11-24T08:25:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.265+0000 7f444d6b4640 1 -- 172.21.15.107:0/1648668589 wait complete. 2023-11-24T08:25:06.286 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2023-11-24T08:25:05.818798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346056,"kb_used_data":25864,"kb_used_omap":12,"kb_used_meta":320051,"kb_avail":754628664,"statfs":{"total":773094113280,"available":772739751936,"internally_reserved":0,"allocated":26484736,"data_stored":12425908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12738,"internal_metadata":327732798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001819"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327329+0000","last_change":"2023-11-24T08:25:01.330095+0000","last_active":"2023-11-24T08:25:02.327329+0000","last_peered":"2023-11-24T08:25:02.327329+0000","last_clean":"2023-11-24T08:25:02.327329+0000","last_became_active":"2023-11-24T08:25:01.329468+0000","last_became_peered":"2023-11-24T08:25:01.329468+0000","last_unstale":"2023-11-24T08:25:02.327329+0000","last_undegraded":"2023-11-24T08:25:02.327329+0000","last_fullsized":"2023-11-24T08:25:02.327329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:12.511268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327354+0000","last_change":"2023-11-24T08:25:01.332177+0000","last_active":"2023-11-24T08:25:02.327354+0000","last_peered":"2023-11-24T08:25:02.327354+0000","last_clean":"2023-11-24T08:25:02.327354+0000","last_became_active":"2023-11-24T08:25:01.328911+0000","last_became_peered":"2023-11-24T08:25:01.328911+0000","last_unstale":"2023-11-24T08:25:02.327354+0000","last_undegraded":"2023-11-24T08:25:02.327354+0000","last_fullsized":"2023-11-24T08:25:02.327354+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:24.546643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379811+0000","last_change":"2023-11-24T08:25:01.319246+0000","last_active":"2023-11-24T08:25:02.379811+0000","last_peered":"2023-11-24T08:25:02.379811+0000","last_clean":"2023-11-24T08:25:02.379811+0000","last_became_active":"2023-11-24T08:25:01.318547+0000","last_became_peered":"2023-11-24T08:25:01.318547+0000","last_unstale":"2023-11-24T08:25:02.379811+0000","last_undegraded":"2023-11-24T08:25:02.379811+0000","last_fullsized":"2023-11-24T08:25:02.379811+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.104566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379393+0000","last_change":"2023-11-24T08:25:01.319435+0000","last_active":"2023-11-24T08:25:02.379393+0000","last_peered":"2023-11-24T08:25:02.379393+0000","last_clean":"2023-11-24T08:25:02.379393+0000","last_became_active":"2023-11-24T08:25:01.319129+0000","last_became_peered":"2023-11-24T08:25:01.319129+0000","last_unstale":"2023-11-24T08:25:02.379393+0000","last_undegraded":"2023-11-24T08:25:02.379393+0000","last_fullsized":"2023-11-24T08:25:02.379393+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:18.538022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362430+0000","last_change":"2023-11-24T08:25:01.323515+0000","last_active":"2023-11-24T08:25:02.362430+0000","last_peered":"2023-11-24T08:25:02.362430+0000","last_clean":"2023-11-24T08:25:02.362430+0000","last_became_active":"2023-11-24T08:25:01.323236+0000","last_became_peered":"2023-11-24T08:25:01.323236+0000","last_unstale":"2023-11-24T08:25:02.362430+0000","last_undegraded":"2023-11-24T08:25:02.362430+0000","last_fullsized":"2023-11-24T08:25:02.362430+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:18:22.616146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362102+0000","last_change":"2023-11-24T08:25:01.318160+0000","last_active":"2023-11-24T08:25:02.362102+0000","last_peered":"2023-11-24T08:25:02.362102+0000","last_clean":"2023-11-24T08:25:02.362102+0000","last_became_active":"2023-11-24T08:25:01.317847+0000","last_became_peered":"2023-11-24T08:25:01.317847+0000","last_unstale":"2023-11-24T08:25:02.362102+0000","last_undegraded":"2023-11-24T08:25:02.362102+0000","last_fullsized":"2023-11-24T08:25:02.362102+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:23.524948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459081+0000","last_change":"2023-11-24T08:25:01.325577+0000","last_active":"2023-11-24T08:25:02.459081+0000","last_peered":"2023-11-24T08:25:02.459081+0000","last_clean":"2023-11-24T08:25:02.459081+0000","last_became_active":"2023-11-24T08:25:01.324919+0000","last_became_peered":"2023-11-24T08:25:01.324919+0000","last_unstale":"2023-11-24T08:25:02.459081+0000","last_undegraded":"2023-11-24T08:25:02.459081+0000","last_fullsized":"2023-11-24T08:25:02.459081+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:10.769637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379261+0000","last_change":"2023-11-24T08:25:01.327945+0000","last_active":"2023-11-24T08:25:02.379261+0000","last_peered":"2023-11-24T08:25:02.379261+0000","last_clean":"2023-11-24T08:25:02.379261+0000","last_became_active":"2023-11-24T08:25:01.327673+0000","last_became_peered":"2023-11-24T08:25:01.327673+0000","last_unstale":"2023-11-24T08:25:02.379261+0000","last_undegraded":"2023-11-24T08:25:02.379261+0000","last_fullsized":"2023-11-24T08:25:02.379261+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:58:30.987939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460293+0000","last_change":"2023-11-24T08:25:01.329042+0000","last_active":"2023-11-24T08:25:02.460293+0000","last_peered":"2023-11-24T08:25:02.460293+0000","last_clean":"2023-11-24T08:25:02.460293+0000","last_became_active":"2023-11-24T08:25:01.328768+0000","last_became_peered":"2023-11-24T08:25:01.328768+0000","last_unstale":"2023-11-24T08:25:02.460293+0000","last_undegraded":"2023-11-24T08:25:02.460293+0000","last_fullsized":"2023-11-24T08:25:02.460293+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:50.929188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460715+0000","last_change":"2023-11-24T08:25:01.334656+0000","last_active":"2023-11-24T08:25:02.460715+0000","last_peered":"2023-11-24T08:25:02.460715+0000","last_clean":"2023-11-24T08:25:02.460715+0000","last_became_active":"2023-11-24T08:25:01.334340+0000","last_became_peered":"2023-11-24T08:25:01.334340+0000","last_unstale":"2023-11-24T08:25:02.460715+0000","last_undegraded":"2023-11-24T08:25:02.460715+0000","last_fullsized":"2023-11-24T08:25:02.460715+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:18.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457769+0000","last_change":"2023-11-24T08:25:01.331176+0000","last_active":"2023-11-24T08:25:02.457769+0000","last_peered":"2023-11-24T08:25:02.457769+0000","last_clean":"2023-11-24T08:25:02.457769+0000","last_became_active":"2023-11-24T08:25:01.330893+0000","last_became_peered":"2023-11-24T08:25:01.330893+0000","last_unstale":"2023-11-24T08:25:02.457769+0000","last_undegraded":"2023-11-24T08:25:02.457769+0000","last_fullsized":"2023-11-24T08:25:02.457769+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:43:30.627163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326949+0000","last_change":"2023-11-24T08:25:01.332821+0000","last_active":"2023-11-24T08:25:02.326949+0000","last_peered":"2023-11-24T08:25:02.326949+0000","last_clean":"2023-11-24T08:25:02.326949+0000","last_became_active":"2023-11-24T08:25:01.331469+0000","last_became_peered":"2023-11-24T08:25:01.331469+0000","last_unstale":"2023-11-24T08:25:02.326949+0000","last_undegraded":"2023-11-24T08:25:02.326949+0000","last_fullsized":"2023-11-24T08:25:02.326949+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:33:18.982077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379618+0000","last_change":"2023-11-24T08:25:01.335307+0000","last_active":"2023-11-24T08:25:02.379618+0000","last_peered":"2023-11-24T08:25:02.379618+0000","last_clean":"2023-11-24T08:25:02.379618+0000","last_became_active":"2023-11-24T08:25:01.334719+0000","last_became_peered":"2023-11-24T08:25:01.334719+0000","last_unstale":"2023-11-24T08:25:02.379618+0000","last_undegraded":"2023-11-24T08:25:02.379618+0000","last_fullsized":"2023-11-24T08:25:02.379618+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:23.179460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460642+0000","last_change":"2023-11-24T08:25:01.322748+0000","last_active":"2023-11-24T08:25:02.460642+0000","last_peered":"2023-11-24T08:25:02.460642+0000","last_clean":"2023-11-24T08:25:02.460642+0000","last_became_active":"2023-11-24T08:25:01.322329+0000","last_became_peered":"2023-11-24T08:25:01.322329+0000","last_unstale":"2023-11-24T08:25:02.460642+0000","last_undegraded":"2023-11-24T08:25:02.460642+0000","last_fullsized":"2023-11-24T08:25:02.460642+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:23.433878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379306+0000","last_change":"2023-11-24T08:25:01.328269+0000","last_active":"2023-11-24T08:25:02.379306+0000","last_peered":"2023-11-24T08:25:02.379306+0000","last_clean":"2023-11-24T08:25:02.379306+0000","last_became_active":"2023-11-24T08:25:01.327370+0000","last_became_peered":"2023-11-24T08:25:01.327370+0000","last_unstale":"2023-11-24T08:25:02.379306+0000","last_undegraded":"2023-11-24T08:25:02.379306+0000","last_fullsized":"2023-11-24T08:25:02.379306+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:16:34.113226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458619+0000","last_change":"2023-11-24T08:25:01.325800+0000","last_active":"2023-11-24T08:25:02.458619+0000","last_peered":"2023-11-24T08:25:02.458619+0000","last_clean":"2023-11-24T08:25:02.458619+0000","last_became_active":"2023-11-24T08:25:01.325137+0000","last_became_peered":"2023-11-24T08:25:01.325137+0000","last_unstale":"2023-11-24T08:25:02.458619+0000","last_undegraded":"2023-11-24T08:25:02.458619+0000","last_fullsized":"2023-11-24T08:25:02.458619+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:04.413433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458965+0000","last_change":"2023-11-24T08:25:01.312666+0000","last_active":"2023-11-24T08:25:02.458965+0000","last_peered":"2023-11-24T08:25:02.458965+0000","last_clean":"2023-11-24T08:25:02.458965+0000","last_became_active":"2023-11-24T08:25:01.311919+0000","last_became_peered":"2023-11-24T08:25:01.311919+0000","last_unstale":"2023-11-24T08:25:02.458965+0000","last_undegraded":"2023-11-24T08:25:02.458965+0000","last_fullsized":"2023-11-24T08:25:02.458965+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:57.192941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379591+0000","last_change":"2023-11-24T08:25:01.316892+0000","last_active":"2023-11-24T08:25:02.379591+0000","last_peered":"2023-11-24T08:25:02.379591+0000","last_clean":"2023-11-24T08:25:02.379591+0000","last_became_active":"2023-11-24T08:25:01.316608+0000","last_became_peered":"2023-11-24T08:25:01.316608+0000","last_unstale":"2023-11-24T08:25:02.379591+0000","last_undegraded":"2023-11-24T08:25:02.379591+0000","last_fullsized":"2023-11-24T08:25:02.379591+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:47.587787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459131+0000","last_change":"2023-11-24T08:25:01.319730+0000","last_active":"2023-11-24T08:25:02.459131+0000","last_peered":"2023-11-24T08:25:02.459131+0000","last_clean":"2023-11-24T08:25:02.459131+0000","last_became_active":"2023-11-24T08:25:01.313366+0000","last_became_peered":"2023-11-24T08:25:01.313366+0000","last_unstale":"2023-11-24T08:25:02.459131+0000","last_undegraded":"2023-11-24T08:25:02.459131+0000","last_fullsized":"2023-11-24T08:25:02.459131+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:20.104566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459399+0000","last_change":"2023-11-24T08:25:01.327234+0000","last_active":"2023-11-24T08:25:02.459399+0000","last_peered":"2023-11-24T08:25:02.459399+0000","last_clean":"2023-11-24T08:25:02.459399+0000","last_became_active":"2023-11-24T08:25:01.326914+0000","last_became_peered":"2023-11-24T08:25:01.326914+0000","last_unstale":"2023-11-24T08:25:02.459399+0000","last_undegraded":"2023-11-24T08:25:02.459399+0000","last_fullsized":"2023-11-24T08:25:02.459399+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:25.254259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457739+0000","last_change":"2023-11-24T08:25:01.318731+0000","last_active":"2023-11-24T08:25:02.457739+0000","last_peered":"2023-11-24T08:25:02.457739+0000","last_clean":"2023-11-24T08:25:02.457739+0000","last_became_active":"2023-11-24T08:25:01.318486+0000","last_became_peered":"2023-11-24T08:25:01.318486+0000","last_unstale":"2023-11-24T08:25:02.457739+0000","last_undegraded":"2023-11-24T08:25:02.457739+0000","last_fullsized":"2023-11-24T08:25:02.457739+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:16.102357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459272+0000","last_change":"2023-11-24T08:25:01.319521+0000","last_active":"2023-11-24T08:25:02.459272+0000","last_peered":"2023-11-24T08:25:02.459272+0000","last_clean":"2023-11-24T08:25:02.459272+0000","last_became_active":"2023-11-24T08:25:01.319263+0000","last_became_peered":"2023-11-24T08:25:01.319263+0000","last_unstale":"2023-11-24T08:25:02.459272+0000","last_undegraded":"2023-11-24T08:25:02.459272+0000","last_fullsized":"2023-11-24T08:25:02.459272+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:46:45.332206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459430+0000","last_change":"2023-11-24T08:25:01.312201+0000","last_active":"2023-11-24T08:25:02.459430+0000","last_peered":"2023-11-24T08:25:02.459430+0000","last_clean":"2023-11-24T08:25:02.459430+0000","last_became_active":"2023-11-24T08:25:01.311831+0000","last_became_peered":"2023-11-24T08:25:01.311831+0000","last_unstale":"2023-11-24T08:25:02.459430+0000","last_undegraded":"2023-11-24T08:25:02.459430+0000","last_fullsized":"2023-11-24T08:25:02.459430+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:16:32.590126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459147+0000","last_change":"2023-11-24T08:25:01.328795+0000","last_active":"2023-11-24T08:25:02.459147+0000","last_peered":"2023-11-24T08:25:02.459147+0000","last_clean":"2023-11-24T08:25:02.459147+0000","last_became_active":"2023-11-24T08:25:01.328509+0000","last_became_peered":"2023-11-24T08:25:01.328509+0000","last_unstale":"2023-11-24T08:25:02.459147+0000","last_undegraded":"2023-11-24T08:25:02.459147+0000","last_fullsized":"2023-11-24T08:25:02.459147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:30:13.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327745+0000","last_change":"2023-11-24T08:25:01.330266+0000","last_active":"2023-11-24T08:25:02.327745+0000","last_peered":"2023-11-24T08:25:02.327745+0000","last_clean":"2023-11-24T08:25:02.327745+0000","last_became_active":"2023-11-24T08:25:01.329823+0000","last_became_peered":"2023-11-24T08:25:01.329823+0000","last_unstale":"2023-11-24T08:25:02.327745+0000","last_undegraded":"2023-11-24T08:25:02.327745+0000","last_fullsized":"2023-11-24T08:25:02.327745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:34:10.769637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362473+0000","last_change":"2023-11-24T08:25:01.331571+0000","last_active":"2023-11-24T08:25:02.362473+0000","last_peered":"2023-11-24T08:25:02.362473+0000","last_clean":"2023-11-24T08:25:02.362473+0000","last_became_active":"2023-11-24T08:25:01.331288+0000","last_became_peered":"2023-11-24T08:25:01.331288+0000","last_unstale":"2023-11-24T08:25:02.362473+0000","last_undegraded":"2023-11-24T08:25:02.362473+0000","last_fullsized":"2023-11-24T08:25:02.362473+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:58:45.667771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362425+0000","last_change":"2023-11-24T08:25:01.327647+0000","last_active":"2023-11-24T08:25:02.362425+0000","last_peered":"2023-11-24T08:25:02.362425+0000","last_clean":"2023-11-24T08:25:02.362425+0000","last_became_active":"2023-11-24T08:25:01.327127+0000","last_became_peered":"2023-11-24T08:25:01.327127+0000","last_unstale":"2023-11-24T08:25:02.362425+0000","last_undegraded":"2023-11-24T08:25:02.362425+0000","last_fullsized":"2023-11-24T08:25:02.362425+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:29.203493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459745+0000","last_change":"2023-11-24T08:25:01.325447+0000","last_active":"2023-11-24T08:25:02.459745+0000","last_peered":"2023-11-24T08:25:02.459745+0000","last_clean":"2023-11-24T08:25:02.459745+0000","last_became_active":"2023-11-24T08:25:01.325008+0000","last_became_peered":"2023-11-24T08:25:01.325008+0000","last_unstale":"2023-11-24T08:25:02.459745+0000","last_undegraded":"2023-11-24T08:25:02.459745+0000","last_fullsized":"2023-11-24T08:25:02.459745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:18.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325993+0000","last_change":"2023-11-24T08:25:01.315692+0000","last_active":"2023-11-24T08:25:02.325993+0000","last_peered":"2023-11-24T08:25:02.325993+0000","last_clean":"2023-11-24T08:25:02.325993+0000","last_became_active":"2023-11-24T08:25:01.315367+0000","last_became_peered":"2023-11-24T08:25:01.315367+0000","last_unstale":"2023-11-24T08:25:02.325993+0000","last_undegraded":"2023-11-24T08:25:02.325993+0000","last_fullsized":"2023-11-24T08:25:02.325993+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:18.092625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458257+0000","last_change":"2023-11-24T08:25:01.325282+0000","last_active":"2023-11-24T08:25:02.458257+0000","last_peered":"2023-11-24T08:25:02.458257+0000","last_clean":"2023-11-24T08:25:02.458257+0000","last_became_active":"2023-11-24T08:25:01.322853+0000","last_became_peered":"2023-11-24T08:25:01.322853+0000","last_unstale":"2023-11-24T08:25:02.458257+0000","last_undegraded":"2023-11-24T08:25:02.458257+0000","last_fullsized":"2023-11-24T08:25:02.458257+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:03.454240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379312+0000","last_change":"2023-11-24T08:25:01.316555+0000","last_active":"2023-11-24T08:25:02.379312+0000","last_peered":"2023-11-24T08:25:02.379312+0000","last_clean":"2023-11-24T08:25:02.379312+0000","last_became_active":"2023-11-24T08:25:01.316149+0000","last_became_peered":"2023-11-24T08:25:01.316149+0000","last_unstale":"2023-11-24T08:25:02.379312+0000","last_undegraded":"2023-11-24T08:25:02.379312+0000","last_fullsized":"2023-11-24T08:25:02.379312+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:09:09.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458510+0000","last_change":"2023-11-24T08:25:01.327987+0000","last_active":"2023-11-24T08:25:02.458510+0000","last_peered":"2023-11-24T08:25:02.458510+0000","last_clean":"2023-11-24T08:25:02.458510+0000","last_became_active":"2023-11-24T08:25:01.327723+0000","last_became_peered":"2023-11-24T08:25:01.327723+0000","last_unstale":"2023-11-24T08:25:02.458510+0000","last_undegraded":"2023-11-24T08:25:02.458510+0000","last_fullsized":"2023-11-24T08:25:02.458510+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:06:22.279973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"248'4","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460626+0000","last_change":"2023-11-24T08:25:00.317048+0000","last_active":"2023-11-24T08:25:02.460626+0000","last_peered":"2023-11-24T08:25:02.460626+0000","last_clean":"2023-11-24T08:25:02.460626+0000","last_became_active":"2023-11-24T08:25:00.316751+0000","last_became_peered":"2023-11-24T08:25:00.316751+0000","last_unstale":"2023-11-24T08:25:02.460626+0000","last_undegraded":"2023-11-24T08:25:02.460626+0000","last_fullsized":"2023-11-24T08:25:02.460626+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:29.088703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459723+0000","last_change":"2023-11-24T08:25:01.334541+0000","last_active":"2023-11-24T08:25:02.459723+0000","last_peered":"2023-11-24T08:25:02.459723+0000","last_clean":"2023-11-24T08:25:02.459723+0000","last_became_active":"2023-11-24T08:25:01.334224+0000","last_became_peered":"2023-11-24T08:25:01.334224+0000","last_unstale":"2023-11-24T08:25:02.459723+0000","last_undegraded":"2023-11-24T08:25:02.459723+0000","last_fullsized":"2023-11-24T08:25:02.459723+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:59:43.284439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459922+0000","last_change":"2023-11-24T08:25:00.320585+0000","last_active":"2023-11-24T08:25:02.459922+0000","last_peered":"2023-11-24T08:25:02.459922+0000","last_clean":"2023-11-24T08:25:02.459922+0000","last_became_active":"2023-11-24T08:25:00.320273+0000","last_became_peered":"2023-11-24T08:25:00.320273+0000","last_unstale":"2023-11-24T08:25:02.459922+0000","last_undegraded":"2023-11-24T08:25:02.459922+0000","last_fullsized":"2023-11-24T08:25:02.459922+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:31.719759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325999+0000","last_change":"2023-11-24T08:25:01.330769+0000","last_active":"2023-11-24T08:25:02.325999+0000","last_peered":"2023-11-24T08:25:02.325999+0000","last_clean":"2023-11-24T08:25:02.325999+0000","last_became_active":"2023-11-24T08:25:01.330421+0000","last_became_peered":"2023-11-24T08:25:01.330421+0000","last_unstale":"2023-11-24T08:25:02.325999+0000","last_undegraded":"2023-11-24T08:25:02.325999+0000","last_fullsized":"2023-11-24T08:25:02.325999+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:53.759171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457699+0000","last_change":"2023-11-24T08:25:00.317565+0000","last_active":"2023-11-24T08:25:02.457699+0000","last_peered":"2023-11-24T08:25:02.457699+0000","last_clean":"2023-11-24T08:25:02.457699+0000","last_became_active":"2023-11-24T08:25:00.315560+0000","last_became_peered":"2023-11-24T08:25:00.315560+0000","last_unstale":"2023-11-24T08:25:02.457699+0000","last_undegraded":"2023-11-24T08:25:02.457699+0000","last_fullsized":"2023-11-24T08:25:02.457699+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:50:01.301706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460019+0000","last_change":"2023-11-24T08:25:01.321154+0000","last_active":"2023-11-24T08:25:02.460019+0000","last_peered":"2023-11-24T08:25:02.460019+0000","last_clean":"2023-11-24T08:25:02.460019+0000","last_became_active":"2023-11-24T08:25:01.320868+0000","last_became_peered":"2023-11-24T08:25:01.320868+0000","last_unstale":"2023-11-24T08:25:02.460019+0000","last_undegraded":"2023-11-24T08:25:02.460019+0000","last_fullsized":"2023-11-24T08:25:02.460019+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:50:00.181379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457554+0000","last_change":"2023-11-24T08:25:00.317762+0000","last_active":"2023-11-24T08:25:02.457554+0000","last_peered":"2023-11-24T08:25:02.457554+0000","last_clean":"2023-11-24T08:25:02.457554+0000","last_became_active":"2023-11-24T08:25:00.316357+0000","last_became_peered":"2023-11-24T08:25:00.316357+0000","last_unstale":"2023-11-24T08:25:02.457554+0000","last_undegraded":"2023-11-24T08:25:02.457554+0000","last_fullsized":"2023-11-24T08:25:02.457554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:08:39.294181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379803+0000","last_change":"2023-11-24T08:25:01.310919+0000","last_active":"2023-11-24T08:25:02.379803+0000","last_peered":"2023-11-24T08:25:02.379803+0000","last_clean":"2023-11-24T08:25:02.379803+0000","last_became_active":"2023-11-24T08:25:01.310647+0000","last_became_peered":"2023-11-24T08:25:01.310647+0000","last_unstale":"2023-11-24T08:25:02.379803+0000","last_undegraded":"2023-11-24T08:25:02.379803+0000","last_fullsized":"2023-11-24T08:25:02.379803+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:57.192941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326600+0000","last_change":"2023-11-24T08:25:00.321262+0000","last_active":"2023-11-24T08:25:02.326600+0000","last_peered":"2023-11-24T08:25:02.326600+0000","last_clean":"2023-11-24T08:25:02.326600+0000","last_became_active":"2023-11-24T08:25:00.320945+0000","last_became_peered":"2023-11-24T08:25:00.320945+0000","last_unstale":"2023-11-24T08:25:02.326600+0000","last_undegraded":"2023-11-24T08:25:02.326600+0000","last_fullsized":"2023-11-24T08:25:02.326600+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:58.291118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458888+0000","last_change":"2023-11-24T08:25:01.334235+0000","last_active":"2023-11-24T08:25:02.458888+0000","last_peered":"2023-11-24T08:25:02.458888+0000","last_clean":"2023-11-24T08:25:02.458888+0000","last_became_active":"2023-11-24T08:25:01.333914+0000","last_became_peered":"2023-11-24T08:25:01.333914+0000","last_unstale":"2023-11-24T08:25:02.458888+0000","last_undegraded":"2023-11-24T08:25:02.458888+0000","last_fullsized":"2023-11-24T08:25:02.458888+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:35:36.164882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362223+0000","last_change":"2023-11-24T08:25:00.310204+0000","last_active":"2023-11-24T08:25:02.362223+0000","last_peered":"2023-11-24T08:25:02.362223+0000","last_clean":"2023-11-24T08:25:02.362223+0000","last_became_active":"2023-11-24T08:25:00.309880+0000","last_became_peered":"2023-11-24T08:25:00.309880+0000","last_unstale":"2023-11-24T08:25:02.362223+0000","last_undegraded":"2023-11-24T08:25:02.362223+0000","last_fullsized":"2023-11-24T08:25:02.362223+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:19.333339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459951+0000","last_change":"2023-11-24T08:25:01.314336+0000","last_active":"2023-11-24T08:25:02.459951+0000","last_peered":"2023-11-24T08:25:02.459951+0000","last_clean":"2023-11-24T08:25:02.459951+0000","last_became_active":"2023-11-24T08:25:01.314084+0000","last_became_peered":"2023-11-24T08:25:01.314084+0000","last_unstale":"2023-11-24T08:25:02.459951+0000","last_undegraded":"2023-11-24T08:25:02.459951+0000","last_fullsized":"2023-11-24T08:25:02.459951+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:09.700254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325044+0000","last_change":"2023-11-24T08:25:00.313527+0000","last_active":"2023-11-24T08:25:02.325044+0000","last_peered":"2023-11-24T08:25:02.325044+0000","last_clean":"2023-11-24T08:25:02.325044+0000","last_became_active":"2023-11-24T08:25:00.313201+0000","last_became_peered":"2023-11-24T08:25:00.313201+0000","last_unstale":"2023-11-24T08:25:02.325044+0000","last_undegraded":"2023-11-24T08:25:02.325044+0000","last_fullsized":"2023-11-24T08:25:02.325044+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:31.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459293+0000","last_change":"2023-11-24T08:25:01.309578+0000","last_active":"2023-11-24T08:25:02.459293+0000","last_peered":"2023-11-24T08:25:02.459293+0000","last_clean":"2023-11-24T08:25:02.459293+0000","last_became_active":"2023-11-24T08:25:01.306969+0000","last_became_peered":"2023-11-24T08:25:01.306969+0000","last_unstale":"2023-11-24T08:25:02.459293+0000","last_undegraded":"2023-11-24T08:25:02.459293+0000","last_fullsized":"2023-11-24T08:25:02.459293+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:51.707800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379570+0000","last_change":"2023-11-24T08:25:00.316813+0000","last_active":"2023-11-24T08:25:02.379570+0000","last_peered":"2023-11-24T08:25:02.379570+0000","last_clean":"2023-11-24T08:25:02.379570+0000","last_became_active":"2023-11-24T08:25:00.316266+0000","last_became_peered":"2023-11-24T08:25:00.316266+0000","last_unstale":"2023-11-24T08:25:02.379570+0000","last_undegraded":"2023-11-24T08:25:02.379570+0000","last_fullsized":"2023-11-24T08:25:02.379570+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:05:52.092847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379497+0000","last_change":"2023-11-24T08:25:01.333485+0000","last_active":"2023-11-24T08:25:02.379497+0000","last_peered":"2023-11-24T08:25:02.379497+0000","last_clean":"2023-11-24T08:25:02.379497+0000","last_became_active":"2023-11-24T08:25:01.333082+0000","last_became_peered":"2023-11-24T08:25:01.333082+0000","last_unstale":"2023-11-24T08:25:02.379497+0000","last_undegraded":"2023-11-24T08:25:02.379497+0000","last_fullsized":"2023-11-24T08:25:02.379497+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:47:14.431188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459865+0000","last_change":"2023-11-24T08:25:00.319785+0000","last_active":"2023-11-24T08:25:02.459865+0000","last_peered":"2023-11-24T08:25:02.459865+0000","last_clean":"2023-11-24T08:25:02.459865+0000","last_became_active":"2023-11-24T08:25:00.319524+0000","last_became_peered":"2023-11-24T08:25:00.319524+0000","last_unstale":"2023-11-24T08:25:02.459865+0000","last_undegraded":"2023-11-24T08:25:02.459865+0000","last_fullsized":"2023-11-24T08:25:02.459865+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:00.830631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325519+0000","last_change":"2023-11-24T08:25:01.326899+0000","last_active":"2023-11-24T08:25:02.325519+0000","last_peered":"2023-11-24T08:25:02.325519+0000","last_clean":"2023-11-24T08:25:02.325519+0000","last_became_active":"2023-11-24T08:25:01.326619+0000","last_became_peered":"2023-11-24T08:25:01.326619+0000","last_unstale":"2023-11-24T08:25:02.325519+0000","last_undegraded":"2023-11-24T08:25:02.325519+0000","last_fullsized":"2023-11-24T08:25:02.325519+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:43.724381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326651+0000","last_change":"2023-11-24T08:25:00.318702+0000","last_active":"2023-11-24T08:25:02.326651+0000","last_peered":"2023-11-24T08:25:02.326651+0000","last_clean":"2023-11-24T08:25:02.326651+0000","last_became_active":"2023-11-24T08:25:00.318180+0000","last_became_peered":"2023-11-24T08:25:00.318180+0000","last_unstale":"2023-11-24T08:25:02.326651+0000","last_undegraded":"2023-11-24T08:25:02.326651+0000","last_fullsized":"2023-11-24T08:25:02.326651+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:50.337476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379372+0000","last_change":"2023-11-24T08:25:01.319328+0000","last_active":"2023-11-24T08:25:02.379372+0000","last_peered":"2023-11-24T08:25:02.379372+0000","last_clean":"2023-11-24T08:25:02.379372+0000","last_became_active":"2023-11-24T08:25:01.319051+0000","last_became_peered":"2023-11-24T08:25:01.319051+0000","last_unstale":"2023-11-24T08:25:02.379372+0000","last_undegraded":"2023-11-24T08:25:02.379372+0000","last_fullsized":"2023-11-24T08:25:02.379372+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:10:59.363646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"248'3","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459854+0000","last_change":"2023-11-24T08:25:00.305695+0000","last_active":"2023-11-24T08:25:02.459854+0000","last_peered":"2023-11-24T08:25:02.459854+0000","last_clean":"2023-11-24T08:25:02.459854+0000","last_became_active":"2023-11-24T08:25:00.305329+0000","last_became_peered":"2023-11-24T08:25:00.305329+0000","last_unstale":"2023-11-24T08:25:02.459854+0000","last_undegraded":"2023-11-24T08:25:02.459854+0000","last_fullsized":"2023-11-24T08:25:02.459854+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:22.161492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362551+0000","last_change":"2023-11-24T08:25:01.316785+0000","last_active":"2023-11-24T08:25:02.362551+0000","last_peered":"2023-11-24T08:25:02.362551+0000","last_clean":"2023-11-24T08:25:02.362551+0000","last_became_active":"2023-11-24T08:25:01.316491+0000","last_became_peered":"2023-11-24T08:25:01.316491+0000","last_unstale":"2023-11-24T08:25:02.362551+0000","last_undegraded":"2023-11-24T08:25:02.362551+0000","last_fullsized":"2023-11-24T08:25:02.362551+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:43.056955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"248'6","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460819+0000","last_change":"2023-11-24T08:25:00.315811+0000","last_active":"2023-11-24T08:25:02.460819+0000","last_peered":"2023-11-24T08:25:02.460819+0000","last_clean":"2023-11-24T08:25:02.460819+0000","last_became_active":"2023-11-24T08:25:00.315497+0000","last_became_peered":"2023-11-24T08:25:00.315497+0000","last_unstale":"2023-11-24T08:25:02.460819+0000","last_undegraded":"2023-11-24T08:25:02.460819+0000","last_fullsized":"2023-11-24T08:25:02.460819+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:42.706413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327294+0000","last_change":"2023-11-24T08:25:01.331805+0000","last_active":"2023-11-24T08:25:02.327294+0000","last_peered":"2023-11-24T08:25:02.327294+0000","last_clean":"2023-11-24T08:25:02.327294+0000","last_became_active":"2023-11-24T08:25:01.330983+0000","last_became_peered":"2023-11-24T08:25:01.330983+0000","last_unstale":"2023-11-24T08:25:02.327294+0000","last_undegraded":"2023-11-24T08:25:02.327294+0000","last_fullsized":"2023-11-24T08:25:02.327294+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:30.106671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326172+0000","last_change":"2023-11-24T08:25:00.317270+0000","last_active":"2023-11-24T08:25:02.326172+0000","last_peered":"2023-11-24T08:25:02.326172+0000","last_clean":"2023-11-24T08:25:02.326172+0000","last_became_active":"2023-11-24T08:25:00.315767+0000","last_became_peered":"2023-11-24T08:25:00.315767+0000","last_unstale":"2023-11-24T08:25:02.326172+0000","last_undegraded":"2023-11-24T08:25:02.326172+0000","last_fullsized":"2023-11-24T08:25:02.326172+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:50.689833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379786+0000","last_change":"2023-11-24T08:25:01.324126+0000","last_active":"2023-11-24T08:25:02.379786+0000","last_peered":"2023-11-24T08:25:02.379786+0000","last_clean":"2023-11-24T08:25:02.379786+0000","last_became_active":"2023-11-24T08:25:01.323851+0000","last_became_peered":"2023-11-24T08:25:01.323851+0000","last_unstale":"2023-11-24T08:25:02.379786+0000","last_undegraded":"2023-11-24T08:25:02.379786+0000","last_fullsized":"2023-11-24T08:25:02.379786+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:32.430121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326226+0000","last_change":"2023-11-24T08:25:00.319921+0000","last_active":"2023-11-24T08:25:02.326226+0000","last_peered":"2023-11-24T08:25:02.326226+0000","last_clean":"2023-11-24T08:25:02.326226+0000","last_became_active":"2023-11-24T08:25:00.319611+0000","last_became_peered":"2023-11-24T08:25:00.319611+0000","last_unstale":"2023-11-24T08:25:02.326226+0000","last_undegraded":"2023-11-24T08:25:02.326226+0000","last_fullsized":"2023-11-24T08:25:02.326226+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:17:57.507582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460294+0000","last_change":"2023-11-24T08:25:01.315600+0000","last_active":"2023-11-24T08:25:02.460294+0000","last_peered":"2023-11-24T08:25:02.460294+0000","last_clean":"2023-11-24T08:25:02.460294+0000","last_became_active":"2023-11-24T08:25:01.314832+0000","last_became_peered":"2023-11-24T08:25:01.314832+0000","last_unstale":"2023-11-24T08:25:02.460294+0000","last_undegraded":"2023-11-24T08:25:02.460294+0000","last_fullsized":"2023-11-24T08:25:02.460294+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:10:51.355444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325768+0000","last_change":"2023-11-24T08:25:00.318665+0000","last_active":"2023-11-24T08:25:02.325768+0000","last_peered":"2023-11-24T08:25:02.325768+0000","last_clean":"2023-11-24T08:25:02.325768+0000","last_became_active":"2023-11-24T08:25:00.318365+0000","last_became_peered":"2023-11-24T08:25:00.318365+0000","last_unstale":"2023-11-24T08:25:02.325768+0000","last_undegraded":"2023-11-24T08:25:02.325768+0000","last_fullsized":"2023-11-24T08:25:02.325768+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:34:58.663930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459658+0000","last_change":"2023-11-24T08:25:01.313649+0000","last_active":"2023-11-24T08:25:02.459658+0000","last_peered":"2023-11-24T08:25:02.459658+0000","last_clean":"2023-11-24T08:25:02.459658+0000","last_became_active":"2023-11-24T08:25:01.313184+0000","last_became_peered":"2023-11-24T08:25:01.313184+0000","last_unstale":"2023-11-24T08:25:02.459658+0000","last_undegraded":"2023-11-24T08:25:02.459658+0000","last_fullsized":"2023-11-24T08:25:02.459658+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:01.841805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"248'8","reported_seq":20,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459025+0000","last_change":"2023-11-24T08:25:00.317907+0000","last_active":"2023-11-24T08:25:02.459025+0000","last_peered":"2023-11-24T08:25:02.459025+0000","last_clean":"2023-11-24T08:25:02.459025+0000","last_became_active":"2023-11-24T08:25:00.317636+0000","last_became_peered":"2023-11-24T08:25:00.317636+0000","last_unstale":"2023-11-24T08:25:02.459025+0000","last_undegraded":"2023-11-24T08:25:02.459025+0000","last_fullsized":"2023-11-24T08:25:02.459025+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:55.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362039+0000","last_change":"2023-11-24T08:25:01.320958+0000","last_active":"2023-11-24T08:25:02.362039+0000","last_peered":"2023-11-24T08:25:02.362039+0000","last_clean":"2023-11-24T08:25:02.362039+0000","last_became_active":"2023-11-24T08:25:01.320677+0000","last_became_peered":"2023-11-24T08:25:01.320677+0000","last_unstale":"2023-11-24T08:25:02.362039+0000","last_undegraded":"2023-11-24T08:25:02.362039+0000","last_fullsized":"2023-11-24T08:25:02.362039+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:08:49.725715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458406+0000","last_change":"2023-11-24T08:25:00.310337+0000","last_active":"2023-11-24T08:25:02.458406+0000","last_peered":"2023-11-24T08:25:02.458406+0000","last_clean":"2023-11-24T08:25:02.458406+0000","last_became_active":"2023-11-24T08:25:00.310041+0000","last_became_peered":"2023-11-24T08:25:00.310041+0000","last_unstale":"2023-11-24T08:25:02.458406+0000","last_undegraded":"2023-11-24T08:25:02.458406+0000","last_fullsized":"2023-11-24T08:25:02.458406+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:41:22.718681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458391+0000","last_change":"2023-11-24T08:25:01.329443+0000","last_active":"2023-11-24T08:25:02.458391+0000","last_peered":"2023-11-24T08:25:02.458391+0000","last_clean":"2023-11-24T08:25:02.458391+0000","last_became_active":"2023-11-24T08:25:01.329072+0000","last_became_peered":"2023-11-24T08:25:01.329072+0000","last_unstale":"2023-11-24T08:25:02.458391+0000","last_undegraded":"2023-11-24T08:25:02.458391+0000","last_fullsized":"2023-11-24T08:25:02.458391+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:11:44.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362582+0000","last_change":"2023-11-24T08:25:00.302549+0000","last_active":"2023-11-24T08:25:02.362582+0000","last_peered":"2023-11-24T08:25:02.362582+0000","last_clean":"2023-11-24T08:25:02.362582+0000","last_became_active":"2023-11-24T08:25:00.302158+0000","last_became_peered":"2023-11-24T08:25:00.302158+0000","last_unstale":"2023-11-24T08:25:02.362582+0000","last_undegraded":"2023-11-24T08:25:02.362582+0000","last_fullsized":"2023-11-24T08:25:02.362582+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:48.666202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362247+0000","last_change":"2023-11-24T08:25:01.321841+0000","last_active":"2023-11-24T08:25:02.362247+0000","last_peered":"2023-11-24T08:25:02.362247+0000","last_clean":"2023-11-24T08:25:02.362247+0000","last_became_active":"2023-11-24T08:25:01.321562+0000","last_became_peered":"2023-11-24T08:25:01.321562+0000","last_unstale":"2023-11-24T08:25:02.362247+0000","last_undegraded":"2023-11-24T08:25:02.362247+0000","last_fullsized":"2023-11-24T08:25:02.362247+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:27.402198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"248'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457977+0000","last_change":"2023-11-24T08:25:00.313073+0000","last_active":"2023-11-24T08:25:02.457977+0000","last_peered":"2023-11-24T08:25:02.457977+0000","last_clean":"2023-11-24T08:25:02.457977+0000","last_became_active":"2023-11-24T08:25:00.312731+0000","last_became_peered":"2023-11-24T08:25:00.312731+0000","last_unstale":"2023-11-24T08:25:02.457977+0000","last_undegraded":"2023-11-24T08:25:02.457977+0000","last_fullsized":"2023-11-24T08:25:02.457977+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:16.865134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379752+0000","last_change":"2023-11-24T08:25:01.321460+0000","last_active":"2023-11-24T08:25:02.379752+0000","last_peered":"2023-11-24T08:25:02.379752+0000","last_clean":"2023-11-24T08:25:02.379752+0000","last_became_active":"2023-11-24T08:25:01.320125+0000","last_became_peered":"2023-11-24T08:25:01.320125+0000","last_unstale":"2023-11-24T08:25:02.379752+0000","last_undegraded":"2023-11-24T08:25:02.379752+0000","last_fullsized":"2023-11-24T08:25:02.379752+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:04.413433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326194+0000","last_change":"2023-11-24T08:25:00.314601+0000","last_active":"2023-11-24T08:25:02.326194+0000","last_peered":"2023-11-24T08:25:02.326194+0000","last_clean":"2023-11-24T08:25:02.326194+0000","last_became_active":"2023-11-24T08:25:00.314286+0000","last_became_peered":"2023-11-24T08:25:00.314286+0000","last_unstale":"2023-11-24T08:25:02.326194+0000","last_undegraded":"2023-11-24T08:25:02.326194+0000","last_fullsized":"2023-11-24T08:25:02.326194+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:30:56.174973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458939+0000","last_change":"2023-11-24T08:25:01.334173+0000","last_active":"2023-11-24T08:25:02.458939+0000","last_peered":"2023-11-24T08:25:02.458939+0000","last_clean":"2023-11-24T08:25:02.458939+0000","last_became_active":"2023-11-24T08:25:01.333892+0000","last_became_peered":"2023-11-24T08:25:01.333892+0000","last_unstale":"2023-11-24T08:25:02.458939+0000","last_undegraded":"2023-11-24T08:25:02.458939+0000","last_fullsized":"2023-11-24T08:25:02.458939+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:29:28.813896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326174+0000","last_change":"2023-11-24T08:25:00.320831+0000","last_active":"2023-11-24T08:25:02.326174+0000","last_peered":"2023-11-24T08:25:02.326174+0000","last_clean":"2023-11-24T08:25:02.326174+0000","last_became_active":"2023-11-24T08:25:00.320455+0000","last_became_peered":"2023-11-24T08:25:00.320455+0000","last_unstale":"2023-11-24T08:25:02.326174+0000","last_undegraded":"2023-11-24T08:25:02.326174+0000","last_fullsized":"2023-11-24T08:25:02.326174+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:24.236291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.459061+0000","last_change":"2023-11-24T08:25:01.321225+0000","last_active":"2023-11-24T08:25:02.459061+0000","last_peered":"2023-11-24T08:25:02.459061+0000","last_clean":"2023-11-24T08:25:02.459061+0000","last_became_active":"2023-11-24T08:25:01.320895+0000","last_became_peered":"2023-11-24T08:25:01.320895+0000","last_unstale":"2023-11-24T08:25:02.459061+0000","last_undegraded":"2023-11-24T08:25:02.459061+0000","last_fullsized":"2023-11-24T08:25:02.459061+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:16:21.689443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379436+0000","last_change":"2023-11-24T08:25:00.317132+0000","last_active":"2023-11-24T08:25:02.379436+0000","last_peered":"2023-11-24T08:25:02.379436+0000","last_clean":"2023-11-24T08:25:02.379436+0000","last_became_active":"2023-11-24T08:25:00.316463+0000","last_became_peered":"2023-11-24T08:25:00.316463+0000","last_unstale":"2023-11-24T08:25:02.379436+0000","last_undegraded":"2023-11-24T08:25:02.379436+0000","last_fullsized":"2023-11-24T08:25:02.379436+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:19:12.427710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460682+0000","last_change":"2023-11-24T08:25:01.309890+0000","last_active":"2023-11-24T08:25:02.460682+0000","last_peered":"2023-11-24T08:25:02.460682+0000","last_clean":"2023-11-24T08:25:02.460682+0000","last_became_active":"2023-11-24T08:25:01.307813+0000","last_became_peered":"2023-11-24T08:25:01.307813+0000","last_unstale":"2023-11-24T08:25:02.460682+0000","last_undegraded":"2023-11-24T08:25:02.460682+0000","last_fullsized":"2023-11-24T08:25:02.460682+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:20:18.092625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458475+0000","last_change":"2023-11-24T08:25:00.313095+0000","last_active":"2023-11-24T08:25:02.458475+0000","last_peered":"2023-11-24T08:25:02.458475+0000","last_clean":"2023-11-24T08:25:02.458475+0000","last_became_active":"2023-11-24T08:25:00.312561+0000","last_became_peered":"2023-11-24T08:25:00.312561+0000","last_unstale":"2023-11-24T08:25:02.458475+0000","last_undegraded":"2023-11-24T08:25:02.458475+0000","last_fullsized":"2023-11-24T08:25:02.458475+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:23.528675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362649+0000","last_change":"2023-11-24T08:25:01.318829+0000","last_active":"2023-11-24T08:25:02.362649+0000","last_peered":"2023-11-24T08:25:02.362649+0000","last_clean":"2023-11-24T08:25:02.362649+0000","last_became_active":"2023-11-24T08:25:01.318034+0000","last_became_peered":"2023-11-24T08:25:01.318034+0000","last_unstale":"2023-11-24T08:25:02.362649+0000","last_undegraded":"2023-11-24T08:25:02.362649+0000","last_fullsized":"2023-11-24T08:25:02.362649+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:17:32.983090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325288+0000","last_change":"2023-11-24T08:25:00.315474+0000","last_active":"2023-11-24T08:25:02.325288+0000","last_peered":"2023-11-24T08:25:02.325288+0000","last_clean":"2023-11-24T08:25:02.325288+0000","last_became_active":"2023-11-24T08:25:00.315155+0000","last_became_peered":"2023-11-24T08:25:00.315155+0000","last_unstale":"2023-11-24T08:25:02.325288+0000","last_undegraded":"2023-11-24T08:25:02.325288+0000","last_fullsized":"2023-11-24T08:25:02.325288+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:19:22.161492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379478+0000","last_change":"2023-11-24T08:25:01.332535+0000","last_active":"2023-11-24T08:25:02.379478+0000","last_peered":"2023-11-24T08:25:02.379478+0000","last_clean":"2023-11-24T08:25:02.379478+0000","last_became_active":"2023-11-24T08:25:01.332223+0000","last_became_peered":"2023-11-24T08:25:01.332223+0000","last_unstale":"2023-11-24T08:25:02.379478+0000","last_undegraded":"2023-11-24T08:25:02.379478+0000","last_fullsized":"2023-11-24T08:25:02.379478+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:24:14.125930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"248'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327189+0000","last_change":"2023-11-24T08:25:00.320354+0000","last_active":"2023-11-24T08:25:02.327189+0000","last_peered":"2023-11-24T08:25:02.327189+0000","last_clean":"2023-11-24T08:25:02.327189+0000","last_became_active":"2023-11-24T08:25:00.319020+0000","last_became_peered":"2023-11-24T08:25:00.319020+0000","last_unstale":"2023-11-24T08:25:02.327189+0000","last_undegraded":"2023-11-24T08:25:02.327189+0000","last_fullsized":"2023-11-24T08:25:02.327189+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:19.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458680+0000","last_change":"2023-11-24T08:25:01.317528+0000","last_active":"2023-11-24T08:25:02.458680+0000","last_peered":"2023-11-24T08:25:02.458680+0000","last_clean":"2023-11-24T08:25:02.458680+0000","last_became_active":"2023-11-24T08:25:01.317198+0000","last_became_peered":"2023-11-24T08:25:01.317198+0000","last_unstale":"2023-11-24T08:25:02.458680+0000","last_undegraded":"2023-11-24T08:25:02.458680+0000","last_fullsized":"2023-11-24T08:25:02.458680+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:24:32.430121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327401+0000","last_change":"2023-11-24T08:25:00.319830+0000","last_active":"2023-11-24T08:25:02.327401+0000","last_peered":"2023-11-24T08:25:02.327401+0000","last_clean":"2023-11-24T08:25:02.327401+0000","last_became_active":"2023-11-24T08:25:00.319542+0000","last_became_peered":"2023-11-24T08:25:00.319542+0000","last_unstale":"2023-11-24T08:25:02.327401+0000","last_undegraded":"2023-11-24T08:25:02.327401+0000","last_fullsized":"2023-11-24T08:25:02.327401+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:42:43.615118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379315+0000","last_change":"2023-11-24T08:25:01.332499+0000","last_active":"2023-11-24T08:25:02.379315+0000","last_peered":"2023-11-24T08:25:02.379315+0000","last_clean":"2023-11-24T08:25:02.379315+0000","last_became_active":"2023-11-24T08:25:01.332200+0000","last_became_peered":"2023-11-24T08:25:01.332200+0000","last_unstale":"2023-11-24T08:25:02.379315+0000","last_undegraded":"2023-11-24T08:25:02.379315+0000","last_fullsized":"2023-11-24T08:25:02.379315+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:55:39.750205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.457878+0000","last_change":"2023-11-24T08:25:00.314997+0000","last_active":"2023-11-24T08:25:02.457878+0000","last_peered":"2023-11-24T08:25:02.457878+0000","last_clean":"2023-11-24T08:25:02.457878+0000","last_became_active":"2023-11-24T08:25:00.314683+0000","last_became_peered":"2023-11-24T08:25:00.314683+0000","last_unstale":"2023-11-24T08:25:02.457878+0000","last_undegraded":"2023-11-24T08:25:02.457878+0000","last_fullsized":"2023-11-24T08:25:02.457878+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:18:30.024631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460418+0000","last_change":"2023-11-24T08:25:01.319241+0000","last_active":"2023-11-24T08:25:02.460418+0000","last_peered":"2023-11-24T08:25:02.460418+0000","last_clean":"2023-11-24T08:25:02.460418+0000","last_became_active":"2023-11-24T08:25:01.318938+0000","last_became_peered":"2023-11-24T08:25:01.318938+0000","last_unstale":"2023-11-24T08:25:02.460418+0000","last_undegraded":"2023-11-24T08:25:02.460418+0000","last_fullsized":"2023-11-24T08:25:02.460418+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:43:32.737727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.327319+0000","last_change":"2023-11-24T08:25:00.321938+0000","last_active":"2023-11-24T08:25:02.327319+0000","last_peered":"2023-11-24T08:25:02.327319+0000","last_clean":"2023-11-24T08:25:02.327319+0000","last_became_active":"2023-11-24T08:25:00.320417+0000","last_became_peered":"2023-11-24T08:25:00.320417+0000","last_unstale":"2023-11-24T08:25:02.327319+0000","last_undegraded":"2023-11-24T08:25:02.327319+0000","last_fullsized":"2023-11-24T08:25:02.327319+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:55.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.460726+0000","last_change":"2023-11-24T08:25:01.331162+0000","last_active":"2023-11-24T08:25:02.460726+0000","last_peered":"2023-11-24T08:25:02.460726+0000","last_clean":"2023-11-24T08:25:02.460726+0000","last_became_active":"2023-11-24T08:25:01.329796+0000","last_became_peered":"2023-11-24T08:25:01.329796+0000","last_unstale":"2023-11-24T08:25:02.460726+0000","last_undegraded":"2023-11-24T08:25:02.460726+0000","last_fullsized":"2023-11-24T08:25:02.460726+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:58:01.668751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326865+0000","last_change":"2023-11-24T08:25:00.317352+0000","last_active":"2023-11-24T08:25:02.326865+0000","last_peered":"2023-11-24T08:25:02.326865+0000","last_clean":"2023-11-24T08:25:02.326865+0000","last_became_active":"2023-11-24T08:25:00.317091+0000","last_became_peered":"2023-11-24T08:25:00.317091+0000","last_unstale":"2023-11-24T08:25:02.326865+0000","last_undegraded":"2023-11-24T08:25:02.326865+0000","last_fullsized":"2023-11-24T08:25:02.326865+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:51:00.823837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379859+0000","last_change":"2023-11-24T08:25:01.319950+0000","last_active":"2023-11-24T08:25:02.379859+0000","last_peered":"2023-11-24T08:25:02.379859+0000","last_clean":"2023-11-24T08:25:02.379859+0000","last_became_active":"2023-11-24T08:25:01.319517+0000","last_became_peered":"2023-11-24T08:25:01.319517+0000","last_unstale":"2023-11-24T08:25:02.379859+0000","last_undegraded":"2023-11-24T08:25:02.379859+0000","last_fullsized":"2023-11-24T08:25:02.379859+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:01:51.707800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379481+0000","last_change":"2023-11-24T08:25:00.317308+0000","last_active":"2023-11-24T08:25:02.379481+0000","last_peered":"2023-11-24T08:25:02.379481+0000","last_clean":"2023-11-24T08:25:02.379481+0000","last_became_active":"2023-11-24T08:25:00.316866+0000","last_became_peered":"2023-11-24T08:25:00.316866+0000","last_unstale":"2023-11-24T08:25:02.379481+0000","last_undegraded":"2023-11-24T08:25:02.379481+0000","last_fullsized":"2023-11-24T08:25:02.379481+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:50:08.003236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.362635+0000","last_change":"2023-11-24T08:25:01.323588+0000","last_active":"2023-11-24T08:25:02.362635+0000","last_peered":"2023-11-24T08:25:02.362635+0000","last_clean":"2023-11-24T08:25:02.362635+0000","last_became_active":"2023-11-24T08:25:01.323314+0000","last_became_peered":"2023-11-24T08:25:01.323314+0000","last_unstale":"2023-11-24T08:25:02.362635+0000","last_undegraded":"2023-11-24T08:25:02.362635+0000","last_fullsized":"2023-11-24T08:25:02.362635+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:40:36.774886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.458859+0000","last_change":"2023-11-24T08:25:00.318842+0000","last_active":"2023-11-24T08:25:02.458859+0000","last_peered":"2023-11-24T08:25:02.458859+0000","last_clean":"2023-11-24T08:25:02.458859+0000","last_became_active":"2023-11-24T08:25:00.318568+0000","last_became_peered":"2023-11-24T08:25:00.318568+0000","last_unstale":"2023-11-24T08:25:02.458859+0000","last_undegraded":"2023-11-24T08:25:02.458859+0000","last_fullsized":"2023-11-24T08:25:02.458859+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:43:46.569819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.379611+0000","last_change":"2023-11-24T08:25:01.316023+0000","last_active":"2023-11-24T08:25:02.379611+0000","last_peered":"2023-11-24T08:25:02.379611+0000","last_clean":"2023-11-24T08:25:02.379611+0000","last_became_active":"2023-11-24T08:25:01.315713+0000","last_became_peered":"2023-11-24T08:25:01.315713+0000","last_unstale":"2023-11-24T08:25:02.379611+0000","last_undegraded":"2023-11-24T08:25:02.379611+0000","last_fullsized":"2023-11-24T08:25:02.379611+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:29:01.848599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.325026+0000","last_change":"2023-11-24T08:25:00.315354+0000","last_active":"2023-11-24T08:25:02.325026+0000","last_peered":"2023-11-24T08:25:02.325026+0000","last_clean":"2023-11-24T08:25:02.325026+0000","last_became_active":"2023-11-24T08:25:00.315072+0000","last_became_peered":"2023-11-24T08:25:00.315072+0000","last_unstale":"2023-11-24T08:25:02.325026+0000","last_undegraded":"2023-11-24T08:25:02.325026+0000","last_fullsized":"2023-11-24T08:25:02.325026+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:24:59.254480+0000","last_clean_scrub_stamp":"2023-11-24T08:24:59.254480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:27:03.395465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-11-24T08:25:02.326637+0000","last_change":"2023-11-24T08:25:01.321323+0000","last_active":"2023-11-24T08:25:02.326637+0000","last_peered":"2023-11-24T08:25:02.326637+0000","last_clean":"2023-11-24T08:25:02.326637+0000","last_became_active":"2023-11-24T08:25:01.319285+0000","last_became_peered":"2023-11-24T08:25:01.319285+0000","last_unstale":"2023-11-24T08:25:02.326637+0000","last_undegraded":"2023-11-24T08:25:02.326637+0000","last_fullsized":"2023-11-24T08:25:02.326637+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:25:00.272448+0000","last_clean_scrub_stamp":"2023-11-24T08:25:00.272448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:45.471299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":368,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-11-24T08:25:01.299233+0000","last_change":"2023-11-24T08:04:44.814222+0000","last_active":"2023-11-24T08:25:01.299233+0000","last_peered":"2023-11-24T08:25:01.299233+0000","last_clean":"2023-11-24T08:25:01.299233+0000","last_became_active":"2023-11-24T08:04:44.812059+0000","last_became_peered":"2023-11-24T08:04:44.812059+0000","last_unstale":"2023-11-24T08:25:01.299233+0000","last_undegraded":"2023-11-24T08:25:01.299233+0000","last_fullsized":"2023-11-24T08:25:01.299233+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:04:43.751333+0000","last_clean_scrub_stamp":"2023-11-24T08:04:43.751333+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:46:25.730649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43184,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328656,"statfs":{"total":96636764160,"available":96592543744,"internally_reserved":0,"allocated":3768320,"data_stored":1997676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":40434124},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.42399999999999999},"max":{"1min":1.6830000000000001,"5min":2.048,"15min":2.7240000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.085,"15min":1.087},"min":{"1min":0.65200000000000002,"5min":0.61899999999999999,"15min":0.53700000000000003},"max":{"1min":1.464,"5min":1.899,"15min":2.3340000000000001},"last":1.117}]},{"osd":1,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.115,"15min":1.1479999999999999},"min":{"1min":0.60299999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4750000000000001,"5min":1.681,"15min":3.8519999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.175,"15min":1.204},"min":{"1min":0.752,"5min":0.68700000000000006,"15min":0.66600000000000004},"max":{"1min":1.6140000000000001,"5min":3.6640000000000001,"15min":3.6640000000000001},"last":1.004}]},{"osd":2,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1020000000000001,"15min":1.1120000000000001},"min":{"1min":0.68899999999999995,"5min":0.65500000000000003,"15min":0.55700000000000005},"max":{"1min":1.7150000000000001,"5min":1.762,"15min":2.4430000000000001},"last":1.528},{"interface":"front","average":{"1min":1.256,"5min":1.232,"15min":1.2849999999999999},"min":{"1min":0.70599999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.617,"5min":1.8,"15min":4.0650000000000004},"last":1.238}]},{"osd":3,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1910000000000001,"15min":1.2130000000000001},"min":{"1min":0.747,"5min":0.64300000000000002,"15min":0.60199999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":3.133},"last":1.302},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.296,"15min":1.3540000000000001},"min":{"1min":0.78700000000000003,"5min":0.76600000000000001,"15min":0.622},"max":{"1min":1.6080000000000001,"5min":2.173,"15min":3.9209999999999998},"last":1.337}]},{"osd":4,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.282,"15min":1.325},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.712,"5min":1.8799999999999999,"15min":2.665},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.327,"5min":1.302,"15min":1.3400000000000001},"min":{"1min":0.54500000000000004,"5min":0.51800000000000002,"15min":0.45400000000000001},"max":{"1min":1.6830000000000001,"5min":1.7130000000000001,"15min":3.4870000000000001},"last":1.2010000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4059999999999999,"15min":1.454},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.42699999999999999},"max":{"1min":1.861,"5min":2.1840000000000002,"15min":4.0010000000000003},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4330000000000001,"15min":1.468},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.0720000000000001,"15min":3.4239999999999999},"last":1.264}]},{"osd":6,"last update":"Fri Nov 24 08:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.60199999999999998},"max":{"1min":1.8069999999999999,"5min":2.2610000000000001,"15min":2.992},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.411,"15min":1.4790000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.871,"5min":1.871,"15min":3.798},"last":1.446}]}]},{"osd":6,"up_from":13,"seq":55834575095,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42580,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329260,"statfs":{"total":96636764160,"available":96593162240,"internally_reserved":0,"allocated":3153920,"data_stored":1405044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.492,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.026,"5min":1.2030000000000001,"15min":1.123},"min":{"1min":0.66800000000000004,"5min":0.624,"15min":0.58099999999999996},"max":{"1min":1.5609999999999999,"5min":13.516999999999999,"15min":13.516999999999999},"last":0.74299999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.131,"15min":1.1459999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.61599999999999999},"max":{"1min":1.423,"5min":5.7050000000000001,"15min":5.7050000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.2809999999999999,"15min":1.2210000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.65000000000000002},"max":{"1min":1.679,"5min":13.57,"15min":13.57},"last":1.363}]},{"osd":2,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.125,"15min":1.1020000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.61899999999999999},"max":{"1min":1.679,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2370000000000001,"15min":1.2549999999999999},"min":{"1min":0.73699999999999999,"5min":0.72199999999999998,"15min":0.60399999999999998},"max":{"1min":1.603,"5min":5.3440000000000003,"15min":5.3440000000000003},"last":1.3089999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.544,"15min":1.409},"min":{"1min":0.93700000000000006,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7729999999999999,"5min":23.989000000000001,"15min":23.989000000000001},"last":1.54},{"interface":"front","average":{"1min":1.28,"5min":1.496,"15min":1.3720000000000001},"min":{"1min":0.93799999999999994,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.99399999999999999}]},{"osd":4,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.4530000000000001,"15min":1.4390000000000001},"min":{"1min":0.76800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.7969999999999999,"5min":11.73,"15min":11.73},"last":1.415},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.617,"15min":1.526},"min":{"1min":0.76500000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.1800000000000002,"5min":20.173999999999999,"15min":20.173999999999999},"last":1.6180000000000001}]},{"osd":5,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3819999999999999,"15min":1.4079999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.835,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5700000000000001,"15min":1.5649999999999999},"min":{"1min":0.76500000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.97,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":1.75}]},{"osd":7,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.653,"15min":1.611},"min":{"1min":0.83699999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.0659999999999998,"5min":13.612,"15min":13.612},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.5029999999999999,"15min":1.5600000000000001},"min":{"1min":0.94199999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.948,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.6000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3162112,"data_stored":1405100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.008,"15min":1.018},"min":{"1min":0.65900000000000003,"5min":0.55700000000000005,"15min":0.53200000000000003},"max":{"1min":1.72,"5min":1.72,"15min":4.9850000000000003},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1040000000000001,"15min":1.1000000000000001},"min":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.60599999999999998},"max":{"1min":1.5449999999999999,"5min":1.6639999999999999,"15min":5.4299999999999997},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.123,"15min":1.1339999999999999},"min":{"1min":0.755,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5660000000000001,"5min":1.665,"15min":7.258},"last":1.145},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1819999999999999,"15min":1.198},"min":{"1min":0.75700000000000001,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.631,"5min":1.659,"15min":6.5199999999999996},"last":1.5189999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.181,"15min":1.1819999999999999},"min":{"1min":0.85899999999999999,"5min":0.68100000000000005,"15min":0.65100000000000002},"max":{"1min":1.6240000000000001,"5min":1.7829999999999999,"15min":6.8929999999999998},"last":1.294},{"interface":"front","average":{"1min":1.147,"5min":1.1599999999999999,"15min":1.1719999999999999},"min":{"1min":0.746,"5min":0.72399999999999998,"15min":0.68000000000000005},"max":{"1min":1.536,"5min":1.923,"15min":8.0869999999999997},"last":0.99299999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.151,"15min":1.1739999999999999},"min":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.66600000000000004},"max":{"1min":1.4199999999999999,"5min":1.9670000000000001,"15min":9.0120000000000005},"last":1.087},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.98999999999999999,"5min":0.80100000000000005,"15min":0.66800000000000004},"max":{"1min":1.7709999999999999,"5min":1.956,"15min":7.2130000000000001},"last":1.028}]},{"osd":4,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.397,"15min":1.3919999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.60799999999999998},"max":{"1min":1.7130000000000001,"5min":2.536,"15min":11.813000000000001},"last":1.204},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.4710000000000001,"15min":1.484},"min":{"1min":1.284,"5min":0.88100000000000001,"15min":0.65900000000000003},"max":{"1min":1.855,"5min":2.4550000000000001,"15min":12.327999999999999},"last":1.446}]},{"osd":6,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2989999999999999,"15min":1.337},"min":{"1min":0.94199999999999995,"5min":0.81899999999999995,"15min":0.48899999999999999},"max":{"1min":1.6890000000000001,"5min":2.246,"15min":9.4770000000000003},"last":1.486},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4159999999999999,"15min":1.4510000000000001},"min":{"1min":1.006,"5min":0.88300000000000001,"15min":0.69299999999999995},"max":{"1min":1.8200000000000001,"5min":2.3519999999999999,"15min":9.9019999999999992},"last":1.5980000000000001}]},{"osd":7,"last update":"Fri Nov 24 08:24:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4770000000000001,"15min":1.494},"min":{"1min":1.0129999999999999,"5min":0.877,"15min":0.54300000000000004},"max":{"1min":1.782,"5min":2.802,"15min":8.1660000000000004},"last":1.355},{"interface":"front","average":{"1min":1.593,"5min":1.6040000000000001,"15min":1.6499999999999999},"min":{"1min":1.2430000000000001,"5min":1.099,"15min":0.75600000000000001},"max":{"1min":1.9510000000000001,"5min":2.9489999999999998,"15min":19.663},"last":1.651}]}]},{"osd":4,"up_from":13,"seq":55834575097,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":3170304,"data_stored":1406746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0720000000000001,"15min":1.054},"min":{"1min":0.72799999999999998,"5min":0.621,"15min":0.42699999999999999},"max":{"1min":1.5649999999999999,"5min":1.6160000000000001,"15min":1.8779999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.123,"5min":1.0960000000000001,"15min":1.0820000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.46100000000000002},"max":{"1min":1.526,"5min":1.544,"15min":5.452},"last":1.0489999999999999}]},{"osd":1,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1739999999999999,"15min":1.137},"min":{"1min":0.79200000000000004,"5min":0.69899999999999995,"15min":0.46300000000000002},"max":{"1min":1.827,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.194},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.137,"15min":1.1359999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.51900000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.988},"last":1.1659999999999999}]},{"osd":2,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.083,"15min":1.069},"min":{"1min":0.90500000000000003,"5min":0.66100000000000003,"15min":0.64600000000000002},"max":{"1min":1.4830000000000001,"5min":1.5209999999999999,"15min":1.7929999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.2969999999999999,"15min":1.2509999999999999},"min":{"1min":0.80500000000000005,"5min":0.754,"15min":0.505},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.4430000000000001}]},{"osd":3,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.357,"15min":1.337},"min":{"1min":0.91200000000000003,"5min":0.67900000000000005,"15min":0.62},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.306,"15min":1.282},"min":{"1min":0.83199999999999996,"5min":0.69399999999999995,"15min":0.55800000000000005},"max":{"1min":1.968,"5min":1.968,"15min":2.3340000000000001},"last":1.286}]},{"osd":5,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.69299999999999995},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":2.2429999999999999},"last":2.2269999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.4350000000000001,"15min":1.4219999999999999},"min":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.66200000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.8090000000000002},"last":1.782}]},{"osd":6,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3120000000000001,"15min":1.264},"min":{"1min":0.89500000000000002,"5min":0.66400000000000003,"15min":0.55400000000000005},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.736},{"interface":"front","average":{"1min":1.627,"5min":1.5720000000000001,"15min":1.546},"min":{"1min":0.98299999999999998,"5min":0.69499999999999995,"15min":0.58099999999999996},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":2.3809999999999998}]},{"osd":7,"last update":"Fri Nov 24 08:24:31 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.4890000000000001,"15min":1.48},"min":{"1min":1.079,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":3.1219999999999999},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.462,"15min":1.425},"min":{"1min":1.101,"5min":0.72999999999999998,"15min":0.61799999999999999},"max":{"1min":2.0249999999999999,"5min":2.1190000000000002,"15min":2.3100000000000001},"last":1.849}]}]},{"osd":3,"up_from":13,"seq":55834575096,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42592,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329248,"statfs":{"total":96636764160,"available":96593149952,"internally_reserved":0,"allocated":3162112,"data_stored":1405136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.85799999999999998,"15min":0.83499999999999996},"min":{"1min":0.496,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":1.5620000000000001,"5min":1.5820000000000001,"15min":3.181},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.47299999999999998,"5min":0.378,"15min":0.31900000000000001},"max":{"1min":1.248,"5min":1.5349999999999999,"15min":1.7310000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79300000000000004,"15min":0.80100000000000005},"min":{"1min":0.38400000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":2.101},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.95999999999999996,"15min":0.93400000000000005},"min":{"1min":0.52600000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":3.3260000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91700000000000004,"15min":0.89100000000000001},"min":{"1min":0.53000000000000003,"5min":0.46400000000000002,"15min":0.39900000000000002},"max":{"1min":1.3600000000000001,"5min":1.976,"15min":2.3700000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0409999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.45600000000000002,"15min":0.45200000000000001},"max":{"1min":1.851,"5min":1.851,"15min":3.2480000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2769999999999999,"15min":1.256},"min":{"1min":0.81999999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.786,"5min":1.845,"15min":2.1840000000000002},"last":1.04},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.242,"15min":1.2170000000000001},"min":{"1min":0.93200000000000005,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.5369999999999999,"5min":2.0099999999999998,"15min":3.0459999999999998},"last":1.2969999999999999}]},{"osd":5,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2909999999999999,"15min":1.2689999999999999},"min":{"1min":0.69899999999999995,"5min":0.626,"15min":0.60499999999999998},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":3.4399999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.262,"15min":1.2270000000000001},"min":{"1min":0.81999999999999995,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.992},"last":1.3660000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.3540000000000001,"15min":1.3300000000000001},"min":{"1min":0.82799999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":3.5030000000000001},"last":1.165},{"interface":"front","average":{"1min":1.268,"5min":1.266,"15min":1.246},"min":{"1min":0.75900000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8260000000000001,"5min":1.8420000000000001,"15min":2.4060000000000001},"last":1.4139999999999999}]},{"osd":7,"last update":"Fri Nov 24 08:24:14 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3280000000000001,"15min":1.3049999999999999},"min":{"1min":0.78700000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"max":{"1min":2.012,"5min":2.0369999999999999,"15min":2.1030000000000002},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.355,"15min":1.3140000000000001},"min":{"1min":0.73299999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":3.331},"last":1.6160000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575094,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42584,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329256,"statfs":{"total":96636764160,"available":96593158144,"internally_reserved":0,"allocated":3153920,"data_stored":1405044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81000000000000005,"15min":0.76100000000000001},"min":{"1min":0.41099999999999998,"5min":0.38200000000000001,"15min":0.29999999999999999},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.84899999999999998,"15min":0.85399999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.29699999999999999},"max":{"1min":1.601,"5min":1.601,"15min":1.782},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.91900000000000004,"15min":0.88800000000000001},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.378},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.7809999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0089999999999999,"15min":0.95099999999999996},"min":{"1min":0.52800000000000002,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88500000000000001,"15min":0.85799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.4370000000000001,"5min":1.4530000000000001,"15min":1.6850000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0569999999999999,"15min":1.014},"min":{"1min":0.53500000000000003,"5min":0.52100000000000002,"15min":0.41399999999999998},"max":{"1min":1.6839999999999999,"5min":1.948,"15min":1.948},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.288,"15min":1.268},"min":{"1min":0.625,"5min":0.625,"15min":0.53700000000000003},"max":{"1min":1.6919999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2769999999999999,"15min":1.24},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7050000000000001,"5min":1.8200000000000001,"15min":1.921},"last":1.026}]},{"osd":5,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.238,"15min":1.224},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.61499999999999999},"max":{"1min":1.641,"5min":1.821,"15min":2.3149999999999999},"last":1},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2629999999999999,"15min":1.226},"min":{"1min":0.79800000000000004,"5min":0.71099999999999997,"15min":0.499},"max":{"1min":1.5,"5min":1.9399999999999999,"15min":2.0910000000000002},"last":0.82199999999999995}]},{"osd":6,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.226,"15min":1.1910000000000001},"min":{"1min":0.81899999999999995,"5min":0.66500000000000004,"15min":0.65000000000000002},"max":{"1min":1.5760000000000001,"5min":1.8560000000000001,"15min":2.2879999999999998},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.335,"15min":1.3},"min":{"1min":0.82299999999999995,"5min":0.82199999999999995,"15min":0.65600000000000003},"max":{"1min":1.9950000000000001,"5min":2.0379999999999998,"15min":2.4060000000000001},"last":1.085}]},{"osd":7,"last update":"Fri Nov 24 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.321,"15min":1.3009999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.56399999999999995},"max":{"1min":1.839,"5min":1.9790000000000001,"15min":2.3999999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.3200000000000001,"15min":1.2729999999999999},"min":{"1min":0.83999999999999997,"5min":0.78300000000000003,"15min":0.746},"max":{"1min":2.1419999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.095}]}]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3158016,"data_stored":1405134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.82699999999999996},"min":{"1min":0.48899999999999999,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":1.1779999999999999,"5min":1.4490000000000001,"15min":1.7749999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.97099999999999997,"15min":0.92100000000000004},"min":{"1min":0.51400000000000001,"5min":0.46800000000000003,"15min":0.35499999999999998},"max":{"1min":1.373,"5min":1.7350000000000001,"15min":2.1619999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.82299999999999995,"15min":0.81399999999999995},"min":{"1min":0.435,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":1.2989999999999999,"5min":1.823,"15min":1.861},"last":0.432},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.95999999999999996,"15min":0.93300000000000005},"min":{"1min":0.49399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3140000000000001,"5min":1.754,"15min":1.925},"last":0.379}]},{"osd":3,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.0389999999999999,"15min":1.0149999999999999},"min":{"1min":0.53900000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6559999999999999,"5min":1.7569999999999999,"15min":2.4169999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93500000000000005,"15min":0.91900000000000004},"min":{"1min":0.52200000000000002,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":1.407,"5min":1.8620000000000001,"15min":2.3879999999999999},"last":0.497}]},{"osd":4,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3560000000000001,"15min":1.2989999999999999},"min":{"1min":0.90900000000000003,"5min":0.66700000000000004,"15min":0.65300000000000002},"max":{"1min":1.8779999999999999,"5min":1.9099999999999999,"15min":2.028},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.238,"5min":1.3220000000000001,"15min":1.2729999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.55900000000000005},"max":{"1min":1.736,"5min":2.0750000000000002,"15min":2.141},"last":1.036}]},{"osd":5,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2789999999999999,"15min":1.258},"min":{"1min":0.78100000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8460000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.004},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2549999999999999,"15min":1.2130000000000001},"min":{"1min":0.79200000000000004,"5min":0.72699999999999998,"15min":0.57299999999999995},"max":{"1min":1.806,"5min":2.1499999999999999,"15min":2.246},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.2709999999999999,"15min":1.28},"min":{"1min":0.81999999999999995,"5min":0.745,"15min":0.52400000000000002},"max":{"1min":1.72,"5min":2.0720000000000001,"15min":4.2750000000000004},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3100000000000001,"15min":1.2789999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.63100000000000001},"max":{"1min":1.7090000000000001,"5min":2.0979999999999999,"15min":3.7040000000000002},"last":1.246}]},{"osd":7,"last update":"Fri Nov 24 08:24:19 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.321,"15min":1.321},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.59099999999999997},"max":{"1min":1.7849999999999999,"5min":2.1669999999999998,"15min":2.7280000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.375,"15min":1.3360000000000001},"min":{"1min":0.79000000000000004,"5min":0.78800000000000003,"15min":0.624},"max":{"1min":1.871,"5min":2.1000000000000001,"15min":3.653},"last":1.0189999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575096,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43176,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328664,"statfs":{"total":96636764160,"available":96592551936,"internally_reserved":0,"allocated":3756032,"data_stored":1996028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73799999999999999,"15min":0.73299999999999998},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":1.3260000000000001,"5min":1.5189999999999999,"15min":1.9850000000000001},"last":0.38},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.84899999999999998,"15min":0.82499999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.173,"5min":1.607,"15min":2.3620000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0289999999999999,"15min":1.0109999999999999},"min":{"1min":0.622,"5min":0.54200000000000004,"15min":0.41399999999999998},"max":{"1min":2.7480000000000002,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.90900000000000003,"15min":0.876},"min":{"1min":0.443,"5min":0.443,"15min":0.39400000000000002},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.86399999999999999}]},{"osd":3,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97899999999999998,"15min":0.95899999999999996},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.88400000000000001,"15min":0.88900000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.625,"5min":1.7190000000000001,"15min":1.9710000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.244,"15min":1.1990000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.716,"5min":2.2210000000000001,"15min":2.3010000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.121,"15min":1.113},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.67800000000000005},"max":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":2.5960000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.153,"15min":1.1539999999999999},"min":{"1min":0.80400000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":2.6850000000000001},"last":0.998},{"interface":"front","average":{"1min":1.25,"5min":1.1679999999999999,"15min":1.1619999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.1930000000000001}]},{"osd":6,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.286,"15min":1.266},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.02},{"interface":"front","average":{"1min":1.28,"5min":1.2729999999999999,"15min":1.2549999999999999},"min":{"1min":0.77600000000000002,"5min":0.72199999999999998,"15min":0.70899999999999996},"max":{"1min":1.873,"5min":1.873,"15min":2.823},"last":0.753}]},{"osd":7,"last update":"Fri Nov 24 08:24:09 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.3149999999999999,"15min":1.327},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9119999999999999,"5min":2.1680000000000001,"15min":2.6989999999999998},"last":1.145},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.304,"15min":1.3029999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9670000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.1639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:25:06.294 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T08:25:06.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T08:25:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.554+0000 7feaecb90640 1 Processor -- start 2023-11-24T08:25:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.554+0000 7feaecb90640 1 -- start start 2023-11-24T08:25:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feaecb90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7feae8166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feaecb90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae815eea0 con 0x7feae815b440 2023-11-24T08:25:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feaecb90640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae806d600 con 0x7feae80cc0f0 2023-11-24T08:25:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feaecb90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80c5e00 con 0x7feae80c59b0 2023-11-24T08:25:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feae659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feae815b440 0x7feae80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41086/0 (socket says 172.21.15.107:41086) 2023-11-24T08:25:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feae659c640 1 -- 172.21.15.107:0/209363971 learned_addr learned my addr 172.21.15.107:0/209363971 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929819195 0 0) 0x7feae815eea0 con 0x7feae815b440 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feae6d9d640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7feae8166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.556+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead4003520 con 0x7feae815b440 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605188324 0 0) 0x7fead4003520 con 0x7feae815b440 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 msgr2=0x7feae8166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7feae8166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 >> v1:172.21.15.120:6789/0 conn(0x7feae80cc0f0 legacy=0x7feae80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80abf30 con 0x7feae815b440 2023-11-24T08:25:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fead8001670 con 0x7feae815b440 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 >> v1:172.21.15.107:6789/0 conn(0x7feae815b440 legacy=0x7feae80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7fead40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7fead40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fead4004d00 0x7fead4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bb90 con 0x7fead4003fd0 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bd00 con 0x7fead4004d00 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bec0 con 0x7feae80c59b0 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feadffff640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7fead40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae659c640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7fead40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.557+0000 7feae6d9d640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fead4004d00 0x7fead4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feadffff640 1 -- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 msgr2=0x7fead40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feadffff640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80c59b0 0x7fead40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feadffff640 1 -- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fead4004d00 msgr2=0x7fead4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feadffff640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fead4004d00 0x7fead4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feadffff640 1 -- 172.21.15.107:0/209363971 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4003520 con 0x7fead4003fd0 2023-11-24T08:25:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.558+0000 7feae6d9d640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fead4004d00 0x7fead4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.559+0000 7feadffff640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7fead40047c0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fead0001750 tx=0x7fead40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bc6069dbcd93a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.559+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead00052d0 con 0x7fead4003fd0 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.559+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead002e050 con 0x7fead4003fd0 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.559+0000 7feae5d9b640 1 -- 172.21.15.107:0/209363971 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0003830 con 0x7fead4003fd0 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 -- 172.21.15.107:0/209363971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 msgr2=0x7fead40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 --2- 172.21.15.107:0/209363971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7fead40047c0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fead0001750 tx=0x7fead40057f0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feae6d9d640 1 -- 172.21.15.107:0/209363971 reap_dead start 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 -- 172.21.15.107:0/209363971 shutdown_connections 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 -- 172.21.15.107:0/209363971 >> 172.21.15.107:0/209363971 conn(0x7feae80b0c50 msgr2=0x7feae80b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 -- 172.21.15.107:0/209363971 shutdown_connections 2023-11-24T08:25:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.560+0000 7feaecb90640 1 -- 172.21.15.107:0/209363971 wait complete. 2023-11-24T08:25:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.561+0000 7feaecb90640 1 Processor -- start 2023-11-24T08:25:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.562+0000 7feaecb90640 1 -- start start 2023-11-24T08:25:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae80b1cb0 0x7feae80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80b3d30 0x7feae80c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80b41b0 con 0x7fead4003fd0 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8158470 con 0x7feae80b1cb0 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feaecb90640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80b3010 con 0x7feae80b3d30 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60212/0 (socket says 172.21.15.107:60212) 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 -- 172.21.15.107:0/4215593496 learned_addr learned my addr 172.21.15.107:0/4215593496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 -- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80b3d30 msgr2=0x7feae80c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80b3d30 0x7feae80c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 -- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae80b1cb0 msgr2=0x7feae80b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae80b1cb0 0x7feae80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.563+0000 7feae659c640 1 -- 172.21.15.107:0/4215593496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80b2490 con 0x7fead4003fd0 2023-11-24T08:25:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feae659c640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fead800e7b0 tx=0x7fead800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead800c8d0 con 0x7fead4003fd0 2023-11-24T08:25:06.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead800ca50 con 0x7fead4003fd0 2023-11-24T08:25:06.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead800ce00 con 0x7fead4003fd0 2023-11-24T08:25:06.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae80b2770 con 0x7fead4003fd0 2023-11-24T08:25:06.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.564+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae80b2d30 con 0x7fead4003fd0 2023-11-24T08:25:06.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.565+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae80c60b0 con 0x7fead4003fd0 2023-11-24T08:25:06.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.566+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fead801d460 con 0x7fead4003fd0 2023-11-24T08:25:06.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.567+0000 7feadeffd640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 0x7feac806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:06.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.567+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead8088940 con 0x7fead4003fd0 2023-11-24T08:25:06.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.567+0000 7feadffff640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 0x7feac806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:06.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.568+0000 7feadffff640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 0x7feac806fb20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fead0033000 tx=0x7fead002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:06.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.571+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fead80554f0 con 0x7fead4003fd0 2023-11-24T08:25:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.830+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feae804e8b0 con 0x7fead4003fd0 2023-11-24T08:25:06.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.834+0000 7feadeffd640 1 -- 172.21.15.107:0/4215593496 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v29) v1 ==== 190+0+147 (secure 0 0 0) 0x7fead8058b10 con 0x7fead4003fd0 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDfWGBlmAKILBAAo5WOCjL8K/80AT0uyYeklg== 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stdout: caps mds = "allow" 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stdout: caps mon = "allow r" 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T08:25:06.836 INFO:teuthology.orchestra.run.smithi107.stderr:updated caps for client.0 2023-11-24T08:25:06.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.837+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 msgr2=0x7feac806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.837+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 0x7feac806fb20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fead0033000 tx=0x7fead002f000 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.837+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 msgr2=0x7feae8157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.837+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fead800e7b0 tx=0x7fead800ec70 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 shutdown_connections 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7feac806d660 0x7feac806fb20 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feae80b3d30 0x7feae80c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae80b1cb0 0x7feae80b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 --2- 172.21.15.107:0/4215593496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fead4003fd0 0x7feae8157f30 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 >> 172.21.15.107:0/4215593496 conn(0x7feae80b0c50 msgr2=0x7feae80c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 shutdown_connections 2023-11-24T08:25:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:06.838+0000 7feaecb90640 1 -- 172.21.15.107:0/4215593496 wait complete. 2023-11-24T08:25:06.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T08:25:07.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.064+0000 7f315bfff640 1 Processor -- start 2023-11-24T08:25:07.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.064+0000 7f315bfff640 1 -- start start 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c0d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f315c0d0240 0x7f315c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315bfff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0ce990 con 0x7f315c0d38e0 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315bfff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0d4110 con 0x7f315c0d0240 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c0cec70 con 0x7f315c15d4e0 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f315c0d0240 0x7f315c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f315c0d0240 0x7f315c0b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60228/0 (socket says 172.21.15.107:60228) 2023-11-24T08:25:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f315c0d38e0 0x7f315c0cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41096/0 (socket says 172.21.15.107:41096) 2023-11-24T08:25:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.066+0000 7f315b7fe640 1 -- 172.21.15.107:0/1722535431 learned_addr learned my addr 172.21.15.107:0/1722535431 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315affd640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c0d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315b7fe640 1 -- 172.21.15.107:0/1722535431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 msgr2=0x7f315c0d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315b7fe640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c0d1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315b7fe640 1 -- 172.21.15.107:0/1722535431 >> v1:172.21.15.107:6789/0 conn(0x7f315c0d38e0 legacy=0x7f315c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315b7fe640 1 -- 172.21.15.107:0/1722535431 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0cef50 con 0x7f315c0d0240 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.067+0000 7f315affd640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c0d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f315b7fe640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c0b2e50 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8c0771a84a3e5c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f3159ffb640 1 -- 172.21.15.107:0/1722535431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154030840 con 0x7f315c0d0240 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f3159ffb640 1 -- 172.21.15.107:0/1722535431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31540309a0 con 0x7f315c0d0240 2023-11-24T08:25:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f3159ffb640 1 -- 172.21.15.107:0/1722535431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154030d30 con 0x7f315c0d0240 2023-11-24T08:25:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f315bfff640 1 -- 172.21.15.107:0/1722535431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 msgr2=0x7f315c0b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.068+0000 7f315bfff640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c0b2e50 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402b980 comp rx=0 tx=0).stop 2023-11-24T08:25:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 -- 172.21.15.107:0/1722535431 shutdown_connections 2023-11-24T08:25:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c0b2e50 unknown :-1 s=CLOSED pgs=3392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 --2- 172.21.15.107:0/1722535431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c0d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 -- 172.21.15.107:0/1722535431 >> 172.21.15.107:0/1722535431 conn(0x7f315c09c510 msgr2=0x7f315c0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 -- 172.21.15.107:0/1722535431 shutdown_connections 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.069+0000 7f315bfff640 1 -- 172.21.15.107:0/1722535431 wait complete. 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.070+0000 7f315bfff640 1 Processor -- start 2023-11-24T08:25:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.070+0000 7f315bfff640 1 -- start start 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f315c0d38e0 0x7f315c0cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c1f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1f60d0 con 0x7f315c0d0240 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1f4af0 con 0x7f315c0d38e0 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.071+0000 7f315bfff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1f4dd0 con 0x7f315c15d4e0 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60238/0 (socket says 172.21.15.107:60238) 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 -- 172.21.15.107:0/2834997363 learned_addr learned my addr 172.21.15.107:0/2834997363 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 -- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 msgr2=0x7f315c1f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c1f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 -- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f315c0d38e0 msgr2=0x7f315c0cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f315c0d38e0 0x7f315c0cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 -- 172.21.15.107:0/2834997363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540094d0 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f315affd640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f314c00ce00 tx=0x7f314c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.072+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00f890 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.073+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c00f9f0 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.073+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00fdc0 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.073+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c1f50b0 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.073+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c1f5670 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.074+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c04e840 con 0x7f315c0d0240 2023-11-24T08:25:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.075+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f314c00b040 con 0x7f315c0d0240 2023-11-24T08:25:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.076+0000 7f31537fe640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 0x7f313c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.076+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f314c089480 con 0x7f315c0d0240 2023-11-24T08:25:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.076+0000 7f315a7fc640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 0x7f313c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.076+0000 7f315a7fc640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 0x7f313c06faf0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f3144007fd0 tx=0x7f3144007980 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.080+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f314c055f90 con 0x7f315c0d0240 2023-11-24T08:25:07.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.291+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f315c0d2000 con 0x7f315c0d0240 2023-11-24T08:25:07.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.292+0000 7f31537fe640 1 -- 172.21.15.107:0/2834997363 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f314c01d070 con 0x7f315c0d0240 2023-11-24T08:25:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-11-24T08:25:07.293 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":157,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2856897517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2856897517},{"type":"v1","addr":"172.21.15.120:6833","nonce":2856897517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3120363367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3120363367},{"type":"v1","addr":"172.21.15.120:6835","nonce":3120363367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151},{"gid":13284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1553701545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1553701545},{"type":"v1","addr":"172.21.15.107:6835","nonce":1553701545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T08:25:00.751585+0000","modified":"2023-11-24T08:25:03.830324+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13293":{"gid":13293,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6837/548967818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":548967818},{"type":"v1","addr":"172.21.15.107:6837","nonce":548967818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-11-24T08:25:07.294 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 157 2023-11-24T08:25:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 msgr2=0x7f313c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 0x7f313c06faf0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f3144007fd0 tx=0x7f3144007980 comp rx=0 tx=0).stop 2023-11-24T08:25:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 msgr2=0x7f315c157960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f314c00ce00 tx=0x7f314c00a650 comp rx=0 tx=0).stop 2023-11-24T08:25:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 shutdown_connections 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f313c06d630 0x7f313c06faf0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f315c15d4e0 0x7f315c1f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f315c0d38e0 0x7f315c0cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 --2- 172.21.15.107:0/2834997363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f315c0d0240 0x7f315c157960 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 >> 172.21.15.107:0/2834997363 conn(0x7f315c09c510 msgr2=0x7f315c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 shutdown_connections 2023-11-24T08:25:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.295+0000 7f315bfff640 1 -- 172.21.15.107:0/2834997363 wait complete. 2023-11-24T08:25:07.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T08:25:00.751585+0000', 'modified': '2023-11-24T08:25:03.830324+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13293': {'gid': 13293, 'name': 'c', 'rank': 0, 'incarnation': 153, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6837/548967818', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 548967818}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 548967818}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T08:25:07.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T08:25:07.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T08:25:07.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.940+0000 7fb558f1d640 1 Processor -- start 2023-11-24T08:25:07.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.941+0000 7fb558f1d640 1 -- start start 2023-11-24T08:25:07.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.942+0000 7fb558f1d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb55415b440 0x7fb5540cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.942+0000 7fb558f1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5540c59b0 0x7fb55406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb558f1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540c5dc0 con 0x7fb5540cbf20 2023-11-24T08:25:07.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb558f1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5540cc3d0 con 0x7fb5540c59b0 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb558f1d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55415b820 con 0x7fb55415b440 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5540c59b0 0x7fb55406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5540c59b0 0x7fb55406c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38722/0 (socket says 172.21.15.107:38722) 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 -- 172.21.15.107:0/66183937 learned_addr learned my addr 172.21.15.107:0/66183937 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb55259c640 1 --2- 172.21.15.107:0/66183937 >> v2:172.21.15.120:3300/0 conn(0x7fb55415b440 0x7fb5540cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 -- 172.21.15.107:0/66183937 >> v2:172.21.15.120:3300/0 conn(0x7fb55415b440 msgr2=0x7fb5540cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 --2- 172.21.15.107:0/66183937 >> v2:172.21.15.120:3300/0 conn(0x7fb55415b440 0x7fb5540cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.943+0000 7fb552d9d640 1 -- 172.21.15.107:0/66183937 >> v1:172.21.15.120:6789/0 conn(0x7fb5540cbf20 legacy=0x7fb5540c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.944+0000 7fb552d9d640 1 -- 172.21.15.107:0/66183937 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55415eea0 con 0x7fb5540c59b0 2023-11-24T08:25:07.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.945+0000 7fb552d9d640 1 --2- 172.21.15.107:0/66183937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540c59b0 0x7fb55406c5f0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fb55406e770 tx=0x7fb54c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a52c23eb55428ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.945+0000 7fb55159a640 1 -- 172.21.15.107:0/66183937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0317e0 con 0x7fb5540c59b0 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.945+0000 7fb55159a640 1 -- 172.21.15.107:0/66183937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c031940 con 0x7fb5540c59b0 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.945+0000 7fb55159a640 1 -- 172.21.15.107:0/66183937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031cd0 con 0x7fb5540c59b0 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 -- 172.21.15.107:0/66183937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540c59b0 msgr2=0x7fb55406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 --2- 172.21.15.107:0/66183937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540c59b0 0x7fb55406c5f0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fb55406e770 tx=0x7fb54c02b880 comp rx=0 tx=0).stop 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 -- 172.21.15.107:0/66183937 shutdown_connections 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 --2- 172.21.15.107:0/66183937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540c59b0 0x7fb55406c5f0 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 --2- 172.21.15.107:0/66183937 >> v2:172.21.15.120:3300/0 conn(0x7fb55415b440 0x7fb5540cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 -- 172.21.15.107:0/66183937 >> 172.21.15.107:0/66183937 conn(0x7fb5540b0c50 msgr2=0x7fb5540b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 -- 172.21.15.107:0/66183937 shutdown_connections 2023-11-24T08:25:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.946+0000 7fb558f1d640 1 -- 172.21.15.107:0/66183937 wait complete. 2023-11-24T08:25:07.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.948+0000 7fb558f1d640 1 Processor -- start 2023-11-24T08:25:07.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.948+0000 7fb558f1d640 1 -- start start 2023-11-24T08:25:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.949+0000 7fb558f1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5540c59b0 0x7fb5541548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.949+0000 7fb558f1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb558f1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb55415b440 0x7fb5540afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb558f1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540b3c20 con 0x7fb5540cbf20 2023-11-24T08:25:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb558f1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540b4c10 con 0x7fb5540c59b0 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb558f1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5541576f0 con 0x7fb55415b440 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38728/0 (socket says 172.21.15.107:38728) 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 -- 172.21.15.107:0/3285242257 learned_addr learned my addr 172.21.15.107:0/3285242257 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 -- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb55415b440 msgr2=0x7fb5540afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb552d9d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb55415b440 0x7fb5540afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb55259c640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5540c59b0 0x7fb5541548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb55415b440 0x7fb5540afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 -- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5540c59b0 msgr2=0x7fb5541548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5540c59b0 0x7fb5541548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.950+0000 7fb551d9b640 1 -- 172.21.15.107:0/3285242257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0094d0 con 0x7fb5540cbf20 2023-11-24T08:25:07.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb551d9b640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c430 tx=0x7fb53c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb54affd640 1 -- 172.21.15.107:0/3285242257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c011790 con 0x7fb5540cbf20 2023-11-24T08:25:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb54affd640 1 -- 172.21.15.107:0/3285242257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c0118f0 con 0x7fb5540cbf20 2023-11-24T08:25:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5540b3e50 con 0x7fb5540cbf20 2023-11-24T08:25:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb54affd640 1 -- 172.21.15.107:0/3285242257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c011ce0 con 0x7fb5540cbf20 2023-11-24T08:25:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.951+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5540b4410 con 0x7fb5540cbf20 2023-11-24T08:25:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.953+0000 7fb54affd640 1 -- 172.21.15.107:0/3285242257 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fb53c01c460 con 0x7fb5540cbf20 2023-11-24T08:25:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.954+0000 7fb54affd640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 0x7fb53406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.955+0000 7fb54affd640 1 -- 172.21.15.107:0/3285242257 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb53c088970 con 0x7fb5540cbf20 2023-11-24T08:25:07.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.955+0000 7fb55259c640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 0x7fb53406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.955+0000 7fb55bb91740 1 Processor -- start 2023-11-24T08:25:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.955+0000 7fb55bb91740 1 -- start start 2023-11-24T08:25:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.955+0000 7fb55259c640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 0x7fb53406fb20 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb544009e90 tx=0x7fb54400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256abf90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac3d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256ab58880 con 0x56256abf9ea0 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256ab67ae0 con 0x56256abf4ba0 2023-11-24T08:25:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56256ab651d0 con 0x56256abf95e0 2023-11-24T08:25:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38734/0 (socket says 172.21.15.107:38734) 2023-11-24T08:25:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.956+0000 7fb55259c640 1 -- 172.21.15.107:0/3918491564 learned_addr learned my addr 172.21.15.107:0/3918491564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 -- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 msgr2=0x56256ac3d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac3d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 -- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 msgr2=0x56256abf90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256abf90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 -- 172.21.15.107:0/3918491564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56256ab28860 con 0x56256abf9ea0 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb55259c640 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fb544016710 tx=0x7fb544012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9177425e139bf9f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb533fff640 1 -- 172.21.15.107:0/3918491564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440149a0 con 0x56256abf9ea0 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb533fff640 1 -- 172.21.15.107:0/3918491564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb544014b00 con 0x56256abf9ea0 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.957+0000 7fb533fff640 1 -- 172.21.15.107:0/3918491564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544014e70 con 0x56256abf9ea0 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 -- 172.21.15.107:0/3918491564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 msgr2=0x56256abfa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fb544016710 tx=0x7fb544012000 comp rx=0 tx=0).stop 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 -- 172.21.15.107:0/3918491564 shutdown_connections 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac3d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256abf90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3918491564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256abfa280 unknown :-1 s=CLOSED pgs=3396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 -- 172.21.15.107:0/3918491564 >> 172.21.15.107:0/3918491564 conn(0x56256abbbc70 msgr2=0x56256ac3f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 -- 172.21.15.107:0/3918491564 shutdown_connections 2023-11-24T08:25:07.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.958+0000 7fb55bb91740 1 -- 172.21.15.107:0/3918491564 wait complete. 2023-11-24T08:25:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.959+0000 7fb55bb91740 1 -- start start 2023-11-24T08:25:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.959+0000 7fb55bb91740 1 Processor -- start 2023-11-24T08:25:07.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256ac1d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac222a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256ab60ac0 con 0x56256abf9ea0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256aafd2f0 con 0x56256abf4ba0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56256ab1ac70 con 0x56256abf95e0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb55bb91740 10 client.0 _ll_register_callbacks cb 0x56256ab1f3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38748/0 (socket says 172.21.15.107:38748) 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.960+0000 7fb552d9d640 1 -- 172.21.15.107:0/3677175719 learned_addr learned my addr 172.21.15.107:0/3677175719 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb551d9b640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac222a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb55259c640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256ac1d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 -- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 msgr2=0x56256ac222a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac222a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 -- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 msgr2=0x56256ac1d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256ac1d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 -- 172.21.15.107:0/3677175719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb544016420 con 0x56256abf9ea0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb552d9d640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0090b0 tx=0x7fb54c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c65319afda513c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.961+0000 7fb5317fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031ea0 con 0x56256abf9ea0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb54c032680 con 0x56256abf9ea0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56256ab59aa0 con 0x56256abf9ea0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb54c032840 con 0x56256abf9ea0 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 10 client.13425 fetch_fsmap learned FSMap version 157 2023-11-24T08:25:07.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) v3 -- 0x56256ab5f140 con 0x56256abf9ea0 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56256ab59aa0 con 0x56256abf9ea0 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 157) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb54c0329a0 con 0x56256abf9ea0 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 10 client.13425 fetch_fsmap finished waiting for FSMap version 157 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 10 client.13425 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 20 client.13425 populate_metadata read hostname 'smithi107' 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac467d0 0x56256ac44040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 4 client.13425 mds_command: new command op to 13293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.962+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 --> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56256ab597c0 con 0x56256ac467d0 2023-11-24T08:25:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.963+0000 7fb55259c640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac467d0 0x56256ac44040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.963+0000 7fb55259c640 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac467d0 0x56256ac44040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b7ea959235fb2b server_cookie=e67758b330e6fb47 in_seq=0 out_seq=0 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.963+0000 7fb5317fa640 10 client.13425 ms_handle_connect on v2:172.21.15.107:6836/548967818 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb5317fa640 1 -- 172.21.15.107:0/3677175719 <== mds.0 v2:172.21.15.107:6836/548967818 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56256ab597c0 con 0x56256ac467d0 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb5317fa640 10 client.13425 handle_command_reply: tid=0 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 client.13425 shutdown 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 msgr2=0x56256ac20ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0090b0 tx=0x7fb54c004730 comp rx=0 tx=0).stop 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 shutdown_connections 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac467d0 0x56256ac44040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac20ec0 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf95e0 0x56256ac222a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 --2- 172.21.15.107:0/3677175719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf4ba0 0x56256ac1d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb5317fa640 0 client.13425 ms_handle_reset on v2:172.21.15.107:6836/548967818 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 >> 172.21.15.107:0/3677175719 conn(0x56256abbbc70 msgr2=0x56256ac3f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.964+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 shutdown_connections 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.965+0000 7fb55bb91740 1 -- 172.21.15.107:0/3677175719 wait complete. 2023-11-24T08:25:07.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.965+0000 7fb55bb91740 20 client.13425 trim_cache size 0 max 16384 2023-11-24T08:25:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 Processor -- start 2023-11-24T08:25:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 -- start start 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac1c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 0x56256ac1c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.982+0000 7fb55bb91740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256a8c3f20 con 0x56256abf4ba0 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256abf7a20 con 0x56256abf95e0 2023-11-24T08:25:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56256ab14e30 con 0x56256abf9ea0 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38764/0 (socket says 172.21.15.107:38764) 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 -- 172.21.15.107:0/1312304546 learned_addr learned my addr 172.21.15.107:0/1312304546 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb552d9d640 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 0x56256ac1c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 -- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 msgr2=0x56256ac1c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 0x56256ac1c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 -- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 msgr2=0x56256ac1c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac1c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb55259c640 1 -- 172.21.15.107:0/1312304546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56256aafcb80 con 0x56256abf4ba0 2023-11-24T08:25:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.983+0000 7fb552d9d640 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 0x56256ac1c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb55259c640 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fb544016fd0 tx=0x7fb544014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62321ffd88dda4bf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb548ff9640 1 -- 172.21.15.107:0/1312304546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544014e70 con 0x56256abf4ba0 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb548ff9640 1 -- 172.21.15.107:0/1312304546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54402b440 con 0x56256abf4ba0 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb548ff9640 1 -- 172.21.15.107:0/1312304546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54402b7b0 con 0x56256abf4ba0 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb55bb91740 1 -- 172.21.15.107:0/1312304546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 msgr2=0x56256abf3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.984+0000 7fb55bb91740 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fb544016fd0 tx=0x7fb544014000 comp rx=0 tx=0).stop 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 -- 172.21.15.107:0/1312304546 shutdown_connections 2023-11-24T08:25:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf9ea0 0x56256ac1c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac1c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 --2- 172.21.15.107:0/1312304546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf4ba0 0x56256abf3d70 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 -- 172.21.15.107:0/1312304546 >> 172.21.15.107:0/1312304546 conn(0x56256abbbc70 msgr2=0x56256ac219f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 -- 172.21.15.107:0/1312304546 shutdown_connections 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 -- 172.21.15.107:0/1312304546 wait complete. 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.985+0000 7fb55bb91740 1 -- start start 2023-11-24T08:25:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.986+0000 7fb55bb91740 1 Processor -- start 2023-11-24T08:25:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf4ba0 0x56256ac3f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac3fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256ab62a40 con 0x56256abf9ea0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56256ac5f760 con 0x56256abf95e0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56256abf40d0 con 0x56256abf4ba0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb55bb91740 10 client.0 _ll_register_callbacks cb 0x7fb54c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38774/0 (socket says 172.21.15.107:38774) 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 -- 172.21.15.107:0/2167024783 learned_addr learned my addr 172.21.15.107:0/2167024783 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 -- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf4ba0 msgr2=0x56256ac3f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf4ba0 0x56256ac3f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 -- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 msgr2=0x56256ac3fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb551d9b640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac3fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac3fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.987+0000 7fb552d9d640 1 -- 172.21.15.107:0/2167024783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56256ab28860 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb552d9d640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fb54c004770 tx=0x7fb54c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd453fb625a2a42f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c004450 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb54c000e40 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb544014b00 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb54c02bf00 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 10 client.13431 fetch_fsmap learned FSMap version 157 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) v3 -- 0x56256ab22170 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb544014b00 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 157) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb54c005990 con 0x56256abf9ea0 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 10 client.13431 fetch_fsmap finished waiting for FSMap version 157 2023-11-24T08:25:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 10 client.13431 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 20 client.13431 populate_metadata read hostname 'smithi107' 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac5ef80 0x56256ac59380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 4 client.13431 mds_command: new command op to 13293 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.988+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 --> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56256a8616c0 con 0x56256ac5ef80 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb55259c640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac5ef80 0x56256ac59380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb55259c640 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac5ef80 0x56256ac59380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=542fa04134c73463 server_cookie=54cde2c55e8ffb2d in_seq=0 out_seq=0 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb5317fa640 10 client.13431 ms_handle_connect on v2:172.21.15.107:6836/548967818 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb5317fa640 1 -- 172.21.15.107:0/2167024783 <== mds.0 v2:172.21.15.107:6836/548967818 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x56256a8616c0 con 0x56256ac5ef80 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb5317fa640 10 client.13431 handle_command_reply: tid=0 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.989+0000 7fb55bb91740 1 client.13431 shutdown 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 msgr2=0x56256ac58e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fb54c004770 tx=0x7fb54c009240 comp rx=0 tx=0).stop 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 shutdown_connections 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:6836/548967818,v1:172.21.15.107:6837/548967818] conn(0x56256ac5ef80 0x56256ac59380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56256abf9ea0 0x56256ac58e40 unknown :-1 s=CLOSED pgs=3399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56256abf95e0 0x56256ac3fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 --2- 172.21.15.107:0/2167024783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56256abf4ba0 0x56256ac3f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb5317fa640 0 client.13431 ms_handle_reset on v2:172.21.15.107:6836/548967818 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 >> 172.21.15.107:0/2167024783 conn(0x56256abbbc70 msgr2=0x56256ac219a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 shutdown_connections 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 1 -- 172.21.15.107:0/2167024783 wait complete. 2023-11-24T08:25:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.990+0000 7fb55bb91740 20 client.13431 trim_cache size 0 max 16384 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d", 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 13293, 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.700403305, 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 157, 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 249, 2023-11-24T08:25:07.996 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 249, 2023-11-24T08:25:07.997 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 12.987254378999999 2023-11-24T08:25:07.997 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:25:07.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 msgr2=0x7fb53406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 0x7fb53406fb20 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb544009e90 tx=0x7fb54400b040 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 msgr2=0x7fb554156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c430 tx=0x7fb53c00c8f0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 shutdown_connections 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fb53406d660 0x7fb53406fb20 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb55415b440 0x7fb5540afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5540cbf20 0x7fb554156c70 unknown :-1 s=CLOSED pgs=3395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 --2- 172.21.15.107:0/3285242257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5540c59b0 0x7fb5541548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 >> 172.21.15.107:0/3285242257 conn(0x7fb5540b0c50 msgr2=0x7fb55406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 shutdown_connections 2023-11-24T08:25:07.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:07.991+0000 7fb558f1d640 1 -- 172.21.15.107:0/3285242257 wait complete. 2023-11-24T08:25:08.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:25:08.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.212+0000 7f13d9d24640 1 Processor -- start 2023-11-24T08:25:08.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.213+0000 7f13d9d24640 1 -- start start 2023-11-24T08:25:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.214+0000 7f13d9d24640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13d40cc470 0x7f13d40abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.214+0000 7f13d9d24640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 0x7f13d40b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.214+0000 7f13d9d24640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d40ce270 con 0x7f13d40ce640 2023-11-24T08:25:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.214+0000 7f13d9d24640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d406c560 con 0x7f13d40cc470 2023-11-24T08:25:08.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.214+0000 7f13d9d24640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d406d0a0 con 0x7f13d40cfb60 2023-11-24T08:25:08.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.215+0000 7f13d37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13d40cc470 0x7f13d40abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.215+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 0x7f13d40b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.215+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 0x7f13d40b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:60480/0 (socket says 172.21.15.107:60480) 2023-11-24T08:25:08.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.215+0000 7f13d3fff640 1 -- 172.21.15.107:0/2329538983 learned_addr learned my addr 172.21.15.107:0/2329538983 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d2ffd640 1 -- 172.21.15.107:0/2329538983 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925248460 0 0) 0x7f13d40ce270 con 0x7f13d40ce640 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d2ffd640 1 -- 172.21.15.107:0/2329538983 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0003520 con 0x7f13d40ce640 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d37fe640 1 -- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 msgr2=0x7f13d40b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d37fe640 1 --2- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 0x7f13d40b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d37fe640 1 -- 172.21.15.107:0/2329538983 >> v1:172.21.15.120:6789/0 conn(0x7f13d40ce640 legacy=0x7f13d40cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.216+0000 7f13d37fe640 1 -- 172.21.15.107:0/2329538983 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40b4200 con 0x7f13d40cc470 2023-11-24T08:25:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.218+0000 7f13d37fe640 1 --2- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40cc470 0x7f13d40abfc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd2e82effc989a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.219+0000 7f13d2ffd640 1 -- 172.21.15.107:0/2329538983 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4031830 con 0x7f13d40cc470 2023-11-24T08:25:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.219+0000 7f13d2ffd640 1 -- 172.21.15.107:0/2329538983 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4031990 con 0x7f13d40cc470 2023-11-24T08:25:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.219+0000 7f13d2ffd640 1 -- 172.21.15.107:0/2329538983 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4031d20 con 0x7f13d40cc470 2023-11-24T08:25:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.219+0000 7f13d9d24640 1 -- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40cc470 msgr2=0x7f13d40abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.219+0000 7f13d9d24640 1 --2- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40cc470 0x7f13d40abfc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c402b8d0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 -- 172.21.15.107:0/2329538983 shutdown_connections 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 --2- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cfb60 0x7f13d40b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 --2- 172.21.15.107:0/2329538983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40cc470 0x7f13d40abfc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 -- 172.21.15.107:0/2329538983 >> 172.21.15.107:0/2329538983 conn(0x7f13d406a920 msgr2=0x7f13d406ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 -- 172.21.15.107:0/2329538983 shutdown_connections 2023-11-24T08:25:08.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.220+0000 7f13d9d24640 1 -- 172.21.15.107:0/2329538983 wait complete. 2023-11-24T08:25:08.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.222+0000 7f13d9d24640 1 Processor -- start 2023-11-24T08:25:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.222+0000 7f13d9d24640 1 -- start start 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cc470 0x7f13d414e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40ce640 0x7f13d414ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d4151f60 con 0x7f13d40cfb60 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d414f610 con 0x7f13d40ce640 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.223+0000 7f13d9d24640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d414f8f0 con 0x7f13d40cc470 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38780/0 (socket says 172.21.15.107:38780) 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 -- 172.21.15.107:0/653681982 learned_addr learned my addr 172.21.15.107:0/653681982 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13cbfff640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40ce640 0x7f13d414ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d37fe640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cc470 0x7f13d414e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 -- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cc470 msgr2=0x7f13d414e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cc470 0x7f13d414e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 -- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40ce640 msgr2=0x7f13d414ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40ce640 0x7f13d414ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.224+0000 7f13d3fff640 1 -- 172.21.15.107:0/653681982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c4009400 con 0x7f13d40cfb60 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d3fff640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00ec10 tx=0x7f13cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc00cb20 con 0x7f13d40cfb60 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc00f040 con 0x7f13d40cfb60 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003990 con 0x7f13d40cfb60 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d414fbd0 con 0x7f13d40cfb60 2023-11-24T08:25:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.225+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d4150190 con 0x7f13d40cfb60 2023-11-24T08:25:08.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.226+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d41503b0 con 0x7f13d40cfb60 2023-11-24T08:25:08.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.227+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f13cc003d00 con 0x7f13d40cfb60 2023-11-24T08:25:08.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.228+0000 7f13d0ff9640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 0x7f13b406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:08.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.228+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13cc089100 con 0x7f13d40cfb60 2023-11-24T08:25:08.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.228+0000 7f13d37fe640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 0x7f13b406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:08.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.229+0000 7f13d37fe640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 0x7f13b406fb20 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:08.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.233+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13cc055b90 con 0x7f13d40cfb60 2023-11-24T08:25:08.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.468+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f13d4067470 con 0x7f13d40cfb60 2023-11-24T08:25:08.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.469+0000 7f13d0ff9640 1 -- 172.21.15.107:0/653681982 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v157) v1 ==== 53+0+73 (secure 0 0 0) 0x7f13cc0591b0 con 0x7f13d40cfb60 2023-11-24T08:25:08.469 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:25:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 msgr2=0x7f13b406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 0x7f13b406fb20 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c403c040 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 msgr2=0x7f13d4151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00ec10 tx=0x7f13cc00c490 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 shutdown_connections 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f13b406d660 0x7f13b406fb20 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13d40cfb60 0x7f13d4151a20 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d40ce640 0x7f13d414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 --2- 172.21.15.107:0/653681982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d40cc470 0x7f13d414e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 >> 172.21.15.107:0/653681982 conn(0x7f13d406a920 msgr2=0x7f13d40b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 shutdown_connections 2023-11-24T08:25:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:08.471+0000 7f13d9d24640 1 -- 172.21.15.107:0/653681982 wait complete. 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:25:08.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:25:08.492 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:25:08.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.492 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41162sec preferred_lft 41162sec 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::10a4:9aff:fedc:647b/64 scope link 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 7a:5e:6a:23:66:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::785e:6aff:fe23:6621/64 scope link 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 62:cb:39:d6:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::60cb:39ff:fed6:dd6c/64 scope link 2023-11-24T08:25:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:08.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.551 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:25:08.641 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-11-24T08:25:08.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.645 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-11-24T08:25:08.699 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-11-24T08:25:08.699 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:25:08.699 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:25:08.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:25:08.700 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:25:08.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:25:08.700 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-11-24T08:25:08.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.882 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-11-24T08:25:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T08:25:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T08:25:08.936 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:25:08.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:08.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-11-24T08:25:09.028 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:25:09.029 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T08:25:09.085 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T08:25:09.085 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:25:09.085 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:25:09.138 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T08:25:09.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:09.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:09.278 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:09.278 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:25:09.312 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:09.312 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:25:09.312 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:25:09.313 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:25:09.313 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:25:09.315 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-24T08:25:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:25:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:25:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:25:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:25:09.321 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:25:09.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:09.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T08:25:09.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:09.404 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:25:09.461 INFO:teuthology.orchestra.run.smithi107.stdout:client13437 2023-11-24T08:25:09.461 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:09.461 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:09.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T08:25:09.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.768+0000 7fd73759e640 1 Processor -- start 2023-11-24T08:25:09.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.769+0000 7fd73759e640 1 -- start start 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd73815d4e0 0x7fd7380d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d38e0 0x7fd7380cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7380d0070 0x7fd7380d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7380d4110 con 0x7fd73815d4e0 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7380d04e0 con 0x7fd7380d0070 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380ce990 con 0x7fd7380d38e0 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd73815d4e0 0x7fd7380d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd73815d4e0 0x7fd7380d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38788/0 (socket says 172.21.15.107:38788) 2023-11-24T08:25:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 -- 172.21.15.107:0/888774972 learned_addr learned my addr 172.21.15.107:0/888774972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:09.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 -- 172.21.15.107:0/888774972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d38e0 msgr2=0x7fd7380cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:09.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 --2- 172.21.15.107:0/888774972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d38e0 0x7fd7380cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 -- 172.21.15.107:0/888774972 >> v2:172.21.15.120:3300/0 conn(0x7fd7380d0070 msgr2=0x7fd7380d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:09.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 --2- 172.21.15.107:0/888774972 >> v2:172.21.15.120:3300/0 conn(0x7fd7380d0070 0x7fd7380d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.770+0000 7fd73659c640 1 -- 172.21.15.107:0/888774972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380cec70 con 0x7fd73815d4e0 2023-11-24T08:25:09.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd73659c640 1 --2- 172.21.15.107:0/888774972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd73815d4e0 0x7fd7380d1e20 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7fd7380b2a00 tx=0x7fd72802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b6770d9dae0c10 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd735d9b640 1 -- 172.21.15.107:0/888774972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728031870 con 0x7fd73815d4e0 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd735d9b640 1 -- 172.21.15.107:0/888774972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7280319d0 con 0x7fd73815d4e0 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd735d9b640 1 -- 172.21.15.107:0/888774972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728031d60 con 0x7fd73815d4e0 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd73759e640 1 -- 172.21.15.107:0/888774972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd73815d4e0 msgr2=0x7fd7380d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.772+0000 7fd73759e640 1 --2- 172.21.15.107:0/888774972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd73815d4e0 0x7fd7380d1e20 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7fd7380b2a00 tx=0x7fd72802b910 comp rx=0 tx=0).stop 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 -- 172.21.15.107:0/888774972 shutdown_connections 2023-11-24T08:25:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 --2- 172.21.15.107:0/888774972 >> v2:172.21.15.120:3300/0 conn(0x7fd7380d0070 0x7fd7380d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 --2- 172.21.15.107:0/888774972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d38e0 0x7fd7380cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 --2- 172.21.15.107:0/888774972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd73815d4e0 0x7fd7380d1e20 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 -- 172.21.15.107:0/888774972 >> 172.21.15.107:0/888774972 conn(0x7fd73809c510 msgr2=0x7fd7380d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 -- 172.21.15.107:0/888774972 shutdown_connections 2023-11-24T08:25:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.773+0000 7fd73759e640 1 -- 172.21.15.107:0/888774972 wait complete. 2023-11-24T08:25:09.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.774+0000 7fd73759e640 1 Processor -- start 2023-11-24T08:25:09.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.774+0000 7fd73759e640 1 -- start start 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d0070 0x7fd73815da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd73815d4e0 0x7fd7381495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380c6eb0 con 0x7fd7380d38e0 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7381602e0 con 0x7fd73815d4e0 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.775+0000 7fd73759e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73815ffd0 con 0x7fd7380d0070 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38802/0 (socket says 172.21.15.107:38802) 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 -- 172.21.15.107:0/1543209489 learned_addr learned my addr 172.21.15.107:0/1543209489 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 -- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d0070 msgr2=0x7fd73815da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d0070 0x7fd73815da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 -- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd73815d4e0 msgr2=0x7fd7381495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd73815d4e0 0x7fd7381495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 -- 172.21.15.107:0/1543209489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280094d0 con 0x7fd7380d38e0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.776+0000 7fd72ffff640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fd72000b4d0 tx=0x7fd72000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.777+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720010870 con 0x7fd7380d38e0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.777+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7200109d0 con 0x7fd7380d38e0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.777+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720010d80 con 0x7fd7380d38e0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.777+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd738149b10 con 0x7fd7380d38e0 2023-11-24T08:25:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.777+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7380c73e0 con 0x7fd7380d38e0 2023-11-24T08:25:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.778+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73804e840 con 0x7fd7380d38e0 2023-11-24T08:25:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.779+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7fd72000f3c0 con 0x7fd7380d38e0 2023-11-24T08:25:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.780+0000 7fd72effd640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 0x7fd71806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.780+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7200888a0 con 0x7fd7380d38e0 2023-11-24T08:25:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.780+0000 7fd73659c640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 0x7fd71806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.781+0000 7fd73659c640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 0x7fd71806faf0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd728036000 tx=0x7fd728030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:09.784+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd720055450 con 0x7fd7380d38e0 2023-11-24T08:25:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.015+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd738160420 con 0x7fd7380d38e0 2023-11-24T08:25:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.016+0000 7fd72effd640 1 -- 172.21.15.107:0/1543209489 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v157) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd72001e060 con 0x7fd7380d38e0 2023-11-24T08:25:10.017 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T08:25:10.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 msgr2=0x7fd71806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:10.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 0x7fd71806faf0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd728036000 tx=0x7fd728030000 comp rx=0 tx=0).stop 2023-11-24T08:25:10.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 msgr2=0x7fd7380c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fd72000b4d0 tx=0x7fd72000b990 comp rx=0 tx=0).stop 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 shutdown_connections 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fd71806d630 0x7fd71806faf0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd73815d4e0 0x7fd7381495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7380d38e0 0x7fd7380c96d0 unknown :-1 s=CLOSED pgs=3403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 --2- 172.21.15.107:0/1543209489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380d0070 0x7fd73815da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.019+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 >> 172.21.15.107:0/1543209489 conn(0x7fd73809c510 msgr2=0x7fd7380b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.020+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 shutdown_connections 2023-11-24T08:25:10.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:10.020+0000 7fd73759e640 1 -- 172.21.15.107:0/1543209489 wait complete. 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T08:25:10.039 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T08:25:10.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T08:25:10.040 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:25:10.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.040 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2023-11-24T08:25:10.100 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T08:25:10.100 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T08:25:10.100 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:10.101 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 41159sec preferred_lft 41159sec 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::fc10:1ff:fe89:27cb/64 scope link 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.102 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether f6:21:c6:bb:d9:74 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::f421:c6ff:febb:d974/64 scope link 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether b2:e0:7e:f4:77:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::b0e0:7eff:fef4:770c/64 scope link 2023-11-24T08:25:10.103 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2023-11-24T08:25:10.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.104 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T08:25:10.200 INFO:teuthology.orchestra.run.smithi120.stdout:1 2023-11-24T08:25:10.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.204 DEBUG:teuthology.orchestra.run.smithi120:> route 2023-11-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2023-11-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T08:25:10.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T08:25:10.261 DEBUG:teuthology.orchestra.run.smithi120:> ') 2023-11-24T08:25:10.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.443 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2023-11-24T08:25:10.494 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T08:25:10.494 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T08:25:10.495 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:25:10.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.495 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2023-11-24T08:25:10.585 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:25:10.585 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T08:25:10.638 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T08:25:10.638 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T08:25:10.638 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:25:10.689 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T08:25:10.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.689 DEBUG:teuthology.orchestra.run.smithi120:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:10.827 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:10.827 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:25:10.862 INFO:teuthology.orchestra.run.smithi120.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T08:25:10.862 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T08:25:10.862 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.120:3300,172.21.15.120:3301" 2023-11-24T08:25:10.862 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: trying mount with new device syntax: 1@a3b42e6b-9b23-45ce-b34c-94ff3ef37e5d.cephfs=/ 2023-11-24T08:25:10.862 INFO:teuthology.orchestra.run.smithi120.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.107:3300/172.21.15.120:3300/172.21.15.120:3301" will pass to kernel 2023-11-24T08:25:10.864 INFO:teuthology.orchestra.run.smithi120.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-24T08:25:10.864 INFO:teuthology.orchestra.run.smithi120.stdout: You just mounted a file system that supports labels which does not 2023-11-24T08:25:10.864 INFO:teuthology.orchestra.run.smithi120.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-24T08:25:10.865 INFO:teuthology.orchestra.run.smithi120.stdout: applications will generate AVC messages and not be allowed access to 2023-11-24T08:25:10.865 INFO:teuthology.orchestra.run.smithi120.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-24T08:25:10.869 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T08:25:10.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.869 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T08:25:10.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:10.957 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi120.stdout:client13446 2023-11-24T08:25:11.016 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:25:11.016 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:11.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T08:25:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.293+0000 7f94b08ca640 1 Processor -- start 2023-11-24T08:25:11.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.293+0000 7f94b08ca640 1 -- start start 2023-11-24T08:25:11.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.295+0000 7f94b08ca640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac15e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.295+0000 7f94b08ca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94ac0ac840 0x7f94ac0accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.295+0000 7f94b08ca640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac15de90 con 0x7f94ac146660 2023-11-24T08:25:11.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.295+0000 7f94b08ca640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac0ad210 con 0x7f94ac0ac840 2023-11-24T08:25:11.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.295+0000 7f94b08ca640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac146a40 con 0x7f94ac0b1560 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94ac0ac840 0x7f94ac0accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94ac0ac840 0x7f94ac0accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38818/0 (socket says 172.21.15.107:38818) 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 -- 172.21.15.107:0/4270300725 learned_addr learned my addr 172.21.15.107:0/4270300725 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94a3fff640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac15e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 -- 172.21.15.107:0/4270300725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 msgr2=0x7f94ac15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac15e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 -- 172.21.15.107:0/4270300725 >> v1:172.21.15.120:6789/0 conn(0x7f94ac146660 legacy=0x7f94ac160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.296+0000 7f94abfff640 1 -- 172.21.15.107:0/4270300725 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac161830 con 0x7f94ac0ac840 2023-11-24T08:25:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.297+0000 7f94abfff640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0accd0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009e50 tx=0x7f94a402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bae933cb6073d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.298+0000 7f94aaffd640 1 -- 172.21.15.107:0/4270300725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a402cb90 con 0x7f94ac0ac840 2023-11-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.298+0000 7f94aaffd640 1 -- 172.21.15.107:0/4270300725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a402ccf0 con 0x7f94ac0ac840 2023-11-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.298+0000 7f94aaffd640 1 -- 172.21.15.107:0/4270300725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4034610 con 0x7f94ac0ac840 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.298+0000 7f94b08ca640 1 -- 172.21.15.107:0/4270300725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 msgr2=0x7f94ac0accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.298+0000 7f94b08ca640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0accd0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009e50 tx=0x7f94a402bff0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 -- 172.21.15.107:0/4270300725 shutdown_connections 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0accd0 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 --2- 172.21.15.107:0/4270300725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac15e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 -- 172.21.15.107:0/4270300725 >> 172.21.15.107:0/4270300725 conn(0x7f94ac0cb8a0 msgr2=0x7f94ac0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 -- 172.21.15.107:0/4270300725 shutdown_connections 2023-11-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.299+0000 7f94b08ca640 1 -- 172.21.15.107:0/4270300725 wait complete. 2023-11-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.300+0000 7f94b08ca640 1 Processor -- start 2023-11-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.301+0000 7f94b08ca640 1 -- start start 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.301+0000 7f94b08ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.301+0000 7f94b08ca640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac1f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.301+0000 7f94b08ca640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac146660 0x7f94ac1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94b08ca640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac072620 con 0x7f94ac0ac840 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94b08ca640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac072340 con 0x7f94ac146660 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94b08ca640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac138610 con 0x7f94ac0b1560 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38824/0 (socket says 172.21.15.107:38824) 2023-11-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 -- 172.21.15.107:0/2492987344 learned_addr learned my addr 172.21.15.107:0/2492987344 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 -- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 msgr2=0x7f94ac1f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac1f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 -- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac146660 msgr2=0x7f94ac1f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac146660 0x7f94ac1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 -- 172.21.15.107:0/2492987344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4009b30 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.302+0000 7f94ab7fe640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c450 tx=0x7f949c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.303+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0052d0 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.303+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c005430 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.303+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac1388f0 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.303+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c012870 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.303+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac142dd0 con 0x7f94ac0ac840 2023-11-24T08:25:11.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.304+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac138e10 con 0x7f94ac0ac840 2023-11-24T08:25:11.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.305+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f949c0129d0 con 0x7f94ac0ac840 2023-11-24T08:25:11.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.306+0000 7f94a8ff9640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 0x7f948c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.306+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f949c088470 con 0x7f94ac0ac840 2023-11-24T08:25:11.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.306+0000 7f94a3fff640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 0x7f948c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.307+0000 7f94a3fff640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 0x7f948c06faf0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9494006e20 tx=0x7f9494008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.311+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f949c055020 con 0x7f94ac0ac840 2023-11-24T08:25:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.545+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f94ac04e840 con 0x7f94ac0ac840 2023-11-24T08:25:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.599+0000 7f94a8ff9640 1 -- 172.21.15.107:0/2492987344 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v158) v1 ==== 136+0+0 (secure 0 0 0) 0x7f949c058640 con 0x7f94ac0ac840 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 msgr2=0x7f948c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 0x7f948c06faf0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9494006e20 tx=0x7f9494008040 comp rx=0 tx=0).stop 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 msgr2=0x7f94ac0c4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c450 tx=0x7f949c00c910 comp rx=0 tx=0).stop 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 shutdown_connections 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f948c06d630 0x7f948c06faf0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac146660 0x7f94ac1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac0b1560 0x7f94ac1f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 --2- 172.21.15.107:0/2492987344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac0ac840 0x7f94ac0c4740 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 >> 172.21.15.107:0/2492987344 conn(0x7f94ac0cb8a0 msgr2=0x7f94ac15fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 shutdown_connections 2023-11-24T08:25:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.602+0000 7f94b08ca640 1 -- 172.21.15.107:0/2492987344 wait complete. 2023-11-24T08:25:11.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T08:25:11.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.836+0000 7f717d7ef640 1 Processor -- start 2023-11-24T08:25:11.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.836+0000 7f717d7ef640 1 -- start start 2023-11-24T08:25:11.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.837+0000 7f717d7ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f717815b1d0 0x7f7178071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.837+0000 7f717d7ef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f71780b1250 0x7f717815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.837+0000 7f717d7ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780ca0e0 con 0x7f7178070550 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.837+0000 7f717d7ef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7178072800 con 0x7f71780b1250 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.837+0000 7f717d7ef640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780c97b0 con 0x7f717815b1d0 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f716ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7178070550 0x7f71780b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52412/0 (socket says 172.21.15.107:52412) 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f716ffff640 1 -- 172.21.15.107:0/2294748911 learned_addr learned my addr 172.21.15.107:0/2294748911 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:11.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f717cfee640 1 --2- 172.21.15.107:0/2294748911 >> v2:172.21.15.107:3300/0 conn(0x7f71780b1250 0x7f717815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f7177fff640 1 --2- 172.21.15.107:0/2294748911 >> v2:172.21.15.120:3300/0 conn(0x7f717815b1d0 0x7f7178071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f71777fe640 1 -- 172.21.15.107:0/2294748911 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690008280 0 0) 0x7f71780ca0e0 con 0x7f7178070550 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f71777fe640 1 -- 172.21.15.107:0/2294748911 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7164003520 con 0x7f7178070550 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f717cfee640 1 -- 172.21.15.107:0/2294748911 >> v2:172.21.15.120:3300/0 conn(0x7f717815b1d0 msgr2=0x7f7178071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f717cfee640 1 --2- 172.21.15.107:0/2294748911 >> v2:172.21.15.120:3300/0 conn(0x7f717815b1d0 0x7f7178071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f717cfee640 1 -- 172.21.15.107:0/2294748911 >> v1:172.21.15.107:6789/0 conn(0x7f7178070550 legacy=0x7f71780b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.838+0000 7f717cfee640 1 -- 172.21.15.107:0/2294748911 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780c9a90 con 0x7f71780b1250 2023-11-24T08:25:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.839+0000 7f717cfee640 1 --2- 172.21.15.107:0/2294748911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71780b1250 0x7f717815edd0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f717806e7a0 tx=0x7f717002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9015ec805e5e7dfc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f71777fe640 1 -- 172.21.15.107:0/2294748911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717002cc80 con 0x7f71780b1250 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f71777fe640 1 -- 172.21.15.107:0/2294748911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717002cde0 con 0x7f71780b1250 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f71777fe640 1 -- 172.21.15.107:0/2294748911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170034660 con 0x7f71780b1250 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 -- 172.21.15.107:0/2294748911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71780b1250 msgr2=0x7f717815edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 --2- 172.21.15.107:0/2294748911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71780b1250 0x7f717815edd0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f717806e7a0 tx=0x7f717002bed0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 -- 172.21.15.107:0/2294748911 shutdown_connections 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 --2- 172.21.15.107:0/2294748911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71780b1250 0x7f717815edd0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 --2- 172.21.15.107:0/2294748911 >> v2:172.21.15.120:3300/0 conn(0x7f717815b1d0 0x7f7178071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 -- 172.21.15.107:0/2294748911 >> 172.21.15.107:0/2294748911 conn(0x7f71780cbb40 msgr2=0x7f71780cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 -- 172.21.15.107:0/2294748911 shutdown_connections 2023-11-24T08:25:11.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.840+0000 7f717d7ef640 1 -- 172.21.15.107:0/2294748911 wait complete. 2023-11-24T08:25:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.842+0000 7f717d7ef640 1 Processor -- start 2023-11-24T08:25:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.842+0000 7f717d7ef640 1 -- start start 2023-11-24T08:25:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 0x7f7178156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71780b1250 0x7f71780cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780c6930 con 0x7f717815b1d0 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780c2a20 con 0x7f7178070550 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717d7ef640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7178157bd0 con 0x7f71780b1250 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.843+0000 7f717cfee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38840/0 (socket says 172.21.15.107:38840) 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 -- 172.21.15.107:0/1592672635 learned_addr learned my addr 172.21.15.107:0/1592672635 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f7177fff640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 0x7f7178156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f716ffff640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71780b1250 0x7f71780cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 -- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71780b1250 msgr2=0x7f71780cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71780b1250 0x7f71780cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 -- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 msgr2=0x7f7178156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 0x7f7178156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 -- 172.21.15.107:0/1592672635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7170009a70 con 0x7f717815b1d0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f7177fff640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 0x7f7178156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.844+0000 7f717cfee640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f717002c4e0 tx=0x7f7170004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.845+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170040650 con 0x7f717815b1d0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.845+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71700407b0 con 0x7f717815b1d0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.845+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170040b60 con 0x7f717815b1d0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.845+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71780c2c70 con 0x7f717815b1d0 2023-11-24T08:25:11.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.845+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71780c3230 con 0x7f717815b1d0 2023-11-24T08:25:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.846+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717804e840 con 0x7f717815b1d0 2023-11-24T08:25:11.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.847+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f7170040cc0 con 0x7f717815b1d0 2023-11-24T08:25:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.848+0000 7f71757fa640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 0x7f715806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:11.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.849+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71700ab6b0 con 0x7f717815b1d0 2023-11-24T08:25:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.849+0000 7f7177fff640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 0x7f715806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:11.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.849+0000 7f7177fff640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 0x7f715806faf0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:11.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:11.853+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71700781d0 con 0x7f717815b1d0 2023-11-24T08:25:12.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.068+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f717815b5b0 con 0x7f715806d630 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.071+0000 7f71757fa640 1 -- 172.21.15.107:0/1592672635 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7168009530 con 0x7f715806d630 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephfs" 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:12.072 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-11-24T08:25:12.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 msgr2=0x7f715806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 0x7f715806faf0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168008040 comp rx=0 tx=0).stop 2023-11-24T08:25:12.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 msgr2=0x7f71780cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f717002c4e0 tx=0x7f7170004a40 comp rx=0 tx=0).stop 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 shutdown_connections 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f715806d630 0x7f715806faf0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f717815b1d0 0x7f71780cf4d0 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71780b1250 0x7f71780cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 --2- 172.21.15.107:0/1592672635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7178070550 0x7f7178156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 >> 172.21.15.107:0/1592672635 conn(0x7f71780cbb40 msgr2=0x7f71780b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 shutdown_connections 2023-11-24T08:25:12.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.074+0000 7f717d7ef640 1 -- 172.21.15.107:0/1592672635 wait complete. 2023-11-24T08:25:12.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T08:25:12.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.294+0000 7f173543d640 1 Processor -- start 2023-11-24T08:25:12.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.294+0000 7f173543d640 1 -- start start 2023-11-24T08:25:12.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f173543d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17300cc0f0 0x7f17300c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f173543d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300c57e0 0x7f173006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f173543d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173015eea0 con 0x7f173015b440 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f173543d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173015b820 con 0x7f17300cc0f0 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f173543d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17300abf30 con 0x7f17300c57e0 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f172effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f173015b440 0x7f17300cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52426/0 (socket says 172.21.15.107:52426) 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f172effd640 1 -- 172.21.15.107:0/401695275 learned_addr learned my addr 172.21.15.107:0/401695275 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.296+0000 7f1727fff640 1 --2- 172.21.15.107:0/401695275 >> v2:172.21.15.107:3300/0 conn(0x7f17300cc0f0 0x7f17300c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.297+0000 7f1727fff640 1 -- 172.21.15.107:0/401695275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300c57e0 msgr2=0x7f173006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.297+0000 7f1727fff640 1 --2- 172.21.15.107:0/401695275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300c57e0 0x7f173006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.297+0000 7f1727fff640 1 -- 172.21.15.107:0/401695275 >> v1:172.21.15.107:6789/0 conn(0x7f173015b440 legacy=0x7f17300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.297+0000 7f1727fff640 1 -- 172.21.15.107:0/401695275 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173015f0f0 con 0x7f17300cc0f0 2023-11-24T08:25:12.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.298+0000 7f1727fff640 1 --2- 172.21.15.107:0/401695275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17300cc0f0 0x7f17300c52a0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f17180097a0 tx=0x7f171802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be99a0c5b5cb476 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.298+0000 7f172e7fc640 1 -- 172.21.15.107:0/401695275 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718031780 con 0x7f17300cc0f0 2023-11-24T08:25:12.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.298+0000 7f172e7fc640 1 -- 172.21.15.107:0/401695275 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17180318e0 con 0x7f17300cc0f0 2023-11-24T08:25:12.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.298+0000 7f172e7fc640 1 -- 172.21.15.107:0/401695275 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718031c70 con 0x7f17300cc0f0 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 -- 172.21.15.107:0/401695275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17300cc0f0 msgr2=0x7f17300c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 --2- 172.21.15.107:0/401695275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17300cc0f0 0x7f17300c52a0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f17180097a0 tx=0x7f171802b820 comp rx=0 tx=0).stop 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 -- 172.21.15.107:0/401695275 shutdown_connections 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 --2- 172.21.15.107:0/401695275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300c57e0 0x7f173006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 --2- 172.21.15.107:0/401695275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17300cc0f0 0x7f17300c52a0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 -- 172.21.15.107:0/401695275 >> 172.21.15.107:0/401695275 conn(0x7f17300b0c50 msgr2=0x7f17300b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 -- 172.21.15.107:0/401695275 shutdown_connections 2023-11-24T08:25:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.299+0000 7f173543d640 1 -- 172.21.15.107:0/401695275 wait complete. 2023-11-24T08:25:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.301+0000 7f173543d640 1 Processor -- start 2023-11-24T08:25:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.301+0000 7f173543d640 1 -- start start 2023-11-24T08:25:12.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17300c57e0 0x7f173015a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300cc0f0 0x7f173015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730158a30 con 0x7f173015b440 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730158c80 con 0x7f17300c57e0 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f173543d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730158f60 con 0x7f17300cc0f0 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f172f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f172f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38860/0 (socket says 172.21.15.107:38860) 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f172f7fe640 1 -- 172.21.15.107:0/2007880363 learned_addr learned my addr 172.21.15.107:0/2007880363 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:12.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f172effd640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17300c57e0 0x7f173015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.303+0000 7f1727fff640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300cc0f0 0x7f173015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 -- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300cc0f0 msgr2=0x7f173015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300cc0f0 0x7f173015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 -- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17300c57e0 msgr2=0x7f173015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17300c57e0 0x7f173015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 -- 172.21.15.107:0/2007880363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718009400 con 0x7f173015b440 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f172f7fe640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f172800b970 tx=0x7f172800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172800cb40 con 0x7f173015b440 2023-11-24T08:25:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172800ccc0 con 0x7f173015b440 2023-11-24T08:25:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728014680 con 0x7f173015b440 2023-11-24T08:25:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1730159240 con 0x7f173015b440 2023-11-24T08:25:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.304+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1730159800 con 0x7f173015b440 2023-11-24T08:25:12.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.306+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1730061a50 con 0x7f173015b440 2023-11-24T08:25:12.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.307+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f172801d3b0 con 0x7f173015b440 2023-11-24T08:25:12.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.307+0000 7f17277fe640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 0x7f171006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.308+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1728088980 con 0x7f173015b440 2023-11-24T08:25:12.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.308+0000 7f172effd640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 0x7f171006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.308+0000 7f172effd640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 0x7f171006faf0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f1720005880 tx=0x7f1720009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.312+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1728055490 con 0x7f173015b440 2023-11-24T08:25:12.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.488+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f17301599f0 con 0x7f173015b440 2023-11-24T08:25:12.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.489+0000 7f17277fe640 1 -- 172.21.15.107:0/2007880363 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1728019020 con 0x7f173015b440 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 msgr2=0x7f171006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 0x7f171006faf0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f1720005880 tx=0x7f1720009040 comp rx=0 tx=0).stop 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 msgr2=0x7f1730156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f172800b970 tx=0x7f172800be30 comp rx=0 tx=0).stop 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 shutdown_connections 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f171006d630 0x7f171006faf0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f173015b440 0x7f1730156500 unknown :-1 s=CLOSED pgs=3412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17300cc0f0 0x7f173015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 --2- 172.21.15.107:0/2007880363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17300c57e0 0x7f173015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.491+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 >> 172.21.15.107:0/2007880363 conn(0x7f17300b0c50 msgr2=0x7f1730071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:12.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.492+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 shutdown_connections 2023-11-24T08:25:12.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.492+0000 7f173543d640 1 -- 172.21.15.107:0/2007880363 wait complete. 2023-11-24T08:25:12.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol724 --mode=777 2023-11-24T08:25:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.719+0000 7f2537f0c640 1 Processor -- start 2023-11-24T08:25:12.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.719+0000 7f2537f0c640 1 -- start start 2023-11-24T08:25:12.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25300cc470 0x7f25300abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f25300b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25300cf450 0x7f25300cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25300cfe20 con 0x7f25300cf450 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253006c560 con 0x7f25300cc470 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.720+0000 7f2537f0c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300ce270 con 0x7f25300ce640 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25300cf450 0x7f25300cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25300cf450 0x7f25300cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38876/0 (socket says 172.21.15.107:38876) 2023-11-24T08:25:12.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 -- 172.21.15.107:0/4023326527 learned_addr learned my addr 172.21.15.107:0/4023326527 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f2535ca8640 1 --2- 172.21.15.107:0/4023326527 >> v2:172.21.15.120:3300/0 conn(0x7f25300cc470 0x7f25300abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25354a7640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f25300b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 -- 172.21.15.107:0/4023326527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 msgr2=0x7f25300b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f25300b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 -- 172.21.15.107:0/4023326527 >> v2:172.21.15.120:3300/0 conn(0x7f25300cc470 msgr2=0x7f25300abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 --2- 172.21.15.107:0/4023326527 >> v2:172.21.15.120:3300/0 conn(0x7f25300cc470 0x7f25300abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.721+0000 7f25364a9640 1 -- 172.21.15.107:0/4023326527 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300b4200 con 0x7f25300cf450 2023-11-24T08:25:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.722+0000 7f25364a9640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f25300cf8e0 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f25300b51d0 tx=0x7f252802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed8760ecefd8748 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2534ca6640 1 -- 172.21.15.107:0/4023326527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252802ca80 con 0x7f25300cf450 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2534ca6640 1 -- 172.21.15.107:0/4023326527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252802cbe0 con 0x7f25300cf450 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2534ca6640 1 -- 172.21.15.107:0/4023326527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280343f0 con 0x7f25300cf450 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 -- 172.21.15.107:0/4023326527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 msgr2=0x7f25300cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f25300cf8e0 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f25300b51d0 tx=0x7f252802bee0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 -- 172.21.15.107:0/4023326527 shutdown_connections 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f25300cf8e0 unknown :-1 s=CLOSED pgs=3413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 --2- 172.21.15.107:0/4023326527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f25300b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 --2- 172.21.15.107:0/4023326527 >> v2:172.21.15.120:3300/0 conn(0x7f25300cc470 0x7f25300abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 -- 172.21.15.107:0/4023326527 >> 172.21.15.107:0/4023326527 conn(0x7f253006a920 msgr2=0x7f253006ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 -- 172.21.15.107:0/4023326527 shutdown_connections 2023-11-24T08:25:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.723+0000 7f2537f0c640 1 -- 172.21.15.107:0/4023326527 wait complete. 2023-11-24T08:25:12.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.725+0000 7f2537f0c640 1 Processor -- start 2023-11-24T08:25:12.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.726+0000 7f2537f0c640 1 -- start start 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 0x7f253014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f2530163650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253009b420 con 0x7f25300cf450 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253009b700 con 0x7f25300cc470 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2537f0c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253009b9e0 con 0x7f25300ce640 2023-11-24T08:25:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f25364a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f25364a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38878/0 (socket says 172.21.15.107:38878) 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f25364a9640 1 -- 172.21.15.107:0/3141431613 learned_addr learned my addr 172.21.15.107:0/3141431613 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f25354a7640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f2530163650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.727+0000 7f2535ca8640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 0x7f253014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 -- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 msgr2=0x7f2530163650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f2530163650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 -- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 msgr2=0x7f253014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 0x7f253014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 -- 172.21.15.107:0/3141431613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528009b30 con 0x7f25300cf450 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f2535ca8640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 0x7f253014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25364a9640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f2528009e50 tx=0x7f2528004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528009510 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252803e450 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252803e800 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253009bcc0 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.728+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253014f8e0 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.729+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253009bf00 con 0x7f25300cf450 2023-11-24T08:25:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.730+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f252802cd50 con 0x7f25300cf450 2023-11-24T08:25:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.732+0000 7f25227fc640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 0x7f251406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.732+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25280ab4b0 con 0x7f25300cf450 2023-11-24T08:25:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.732+0000 7f2535ca8640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 0x7f251406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.733+0000 7f2535ca8640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 0x7f251406faf0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f252c006fd0 tx=0x7f252c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:12.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.736+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f253009bf00 con 0x7f25300cf450 2023-11-24T08:25:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:12.936+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol724", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f253009b870 con 0x7f251406d630 2023-11-24T08:25:12.946 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:12.945+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:12.946 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:12.945+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:12.946 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:12.945+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:12.946 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:12.945+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:12.947 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:12.945+0000 7f376379e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:13.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.004+0000 7f25227fc640 1 -- 172.21.15.107:0/3141431613 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f253009b870 con 0x7f251406d630 2023-11-24T08:25:13.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 msgr2=0x7f251406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 0x7f251406faf0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f252c006fd0 tx=0x7f252c008040 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 msgr2=0x7f2530071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f2528009e50 tx=0x7f2528004a40 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 shutdown_connections 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f251406d630 0x7f251406faf0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25300cf450 0x7f2530071610 unknown :-1 s=CLOSED pgs=3414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25300ce640 0x7f2530163650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.006+0000 7f2537f0c640 1 --2- 172.21.15.107:0/3141431613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25300cc470 0x7f253014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.007+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 >> 172.21.15.107:0/3141431613 conn(0x7f253006a920 msgr2=0x7f25300accb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.007+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 shutdown_connections 2023-11-24T08:25:13.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.007+0000 7f2537f0c640 1 -- 172.21.15.107:0/3141431613 wait complete. 2023-11-24T08:25:13.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol724 2023-11-24T08:25:13.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.222+0000 7f94c9d1d640 1 Processor -- start 2023-11-24T08:25:13.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.222+0000 7f94c9d1d640 1 -- start start 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c9d1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94c415e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c9d1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c415ec90 con 0x7f94c40c49f0 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c9d1d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40cd530 con 0x7f94c406de10 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c9d1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40c4ed0 con 0x7f94c4072160 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94c40c49f0 0x7f94c406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52438/0 (socket says 172.21.15.107:52438) 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c2ffd640 1 -- 172.21.15.107:0/1209512555 learned_addr learned my addr 172.21.15.107:0/1209512555 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.224+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94c415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569606760 0 0) 0x7f94b8001110 con 0x7f94c40c49f0 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac005730 con 0x7f94c40c49f0 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676729058 0 0) 0x7f94ac005730 con 0x7f94c40c49f0 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 msgr2=0x7f94c415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94c415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 >> v1:172.21.15.120:6789/0 conn(0x7f94c406de10 legacy=0x7f94c40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40c51b0 con 0x7f94c40c49f0 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f94b8001800 con 0x7f94c40c49f0 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 >> v1:172.21.15.107:6789/0 conn(0x7f94c40c49f0 legacy=0x7f94c406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac005bb0 0x7f94ac005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac006560 0x7f94ac006a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94ac007290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40c4ed0 con 0x7f94ac005bb0 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0077d0 con 0x7f94ac006560 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.225+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac00ff90 con 0x7f94c4072160 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac005bb0 0x7f94ac005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c3fff640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94ac007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c2ffd640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac006560 0x7f94ac006a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 -- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 msgr2=0x7f94ac007290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94c4072160 0x7f94ac007290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 -- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac006560 msgr2=0x7f94ac006a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac006560 0x7f94ac006a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c37fe640 1 -- 172.21.15.107:0/1209512555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac005730 con 0x7f94ac005bb0 2023-11-24T08:25:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.226+0000 7f94c2ffd640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac006560 0x7f94ac006a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.227+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac005bb0 0x7f94ac005f90 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009ac0 tx=0x7f94ac007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ee44436dd1a28e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b402c6b0 con 0x7f94ac005bb0 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b402c810 con 0x7f94ac005bb0 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c27fc640 1 -- 172.21.15.107:0/1209512555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b402cba0 con 0x7f94ac005bb0 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1209512555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac005bb0 msgr2=0x7f94ac005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1209512555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94ac005bb0 0x7f94ac005f90 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009ac0 tx=0x7f94ac007a00 comp rx=0 tx=0).stop 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c3fff640 1 -- 172.21.15.107:0/1209512555 reap_dead start 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1209512555 shutdown_connections 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1209512555 >> 172.21.15.107:0/1209512555 conn(0x7f94c40b0c50 msgr2=0x7f94c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1209512555 shutdown_connections 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.228+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1209512555 wait complete. 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.230+0000 7f94c9d1d640 1 Processor -- start 2023-11-24T08:25:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.230+0000 7f94c9d1d640 1 -- start start 2023-11-24T08:25:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac005bb0 0x7f94c40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c40cc4a0 0x7f94c409b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c409bf40 con 0x7f94c40cc020 2023-11-24T08:25:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c409c220 con 0x7f94c40cc4a0 2023-11-24T08:25:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c9d1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40ca320 con 0x7f94ac005bb0 2023-11-24T08:25:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.231+0000 7f94c2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38902/0 (socket says 172.21.15.107:38902) 2023-11-24T08:25:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 -- 172.21.15.107:0/1187337015 learned_addr learned my addr 172.21.15.107:0/1187337015 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 -- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac005bb0 msgr2=0x7f94c40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac005bb0 0x7f94c40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 -- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c40cc4a0 msgr2=0x7f94c409b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c40cc4a0 0x7f94c409b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 -- 172.21.15.107:0/1187337015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40ca600 con 0x7f94c40cc020 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94c2ffd640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7f94b800dae0 tx=0x7f94b8007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b80074e0 con 0x7f94c40cc020 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.232+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b8007640 con 0x7f94c40cc020 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.233+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b80049e0 con 0x7f94c40cc020 2023-11-24T08:25:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.233+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c40ca8e0 con 0x7f94c40cc020 2023-11-24T08:25:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.233+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c41579a0 con 0x7f94c40cc020 2023-11-24T08:25:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.234+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c404e8b0 con 0x7f94c40cc020 2023-11-24T08:25:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.235+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f94b8004b40 con 0x7f94c40cc020 2023-11-24T08:25:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.236+0000 7f94abfff640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 0x7f94b006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.236+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94b80882a0 con 0x7f94c40cc020 2023-11-24T08:25:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.236+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 0x7f94b006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.237+0000 7f94c37fe640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 0x7f94b006faf0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f94b4036000 tx=0x7f94b402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.240+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94b8054e50 con 0x7f94c40cc020 2023-11-24T08:25:13.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.458+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f94c40b0810 con 0x7f94b006d630 2023-11-24T08:25:13.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.469+0000 7f94abfff640 1 -- 172.21.15.107:0/1187337015 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+65 (secure 0 0 0) 0x7f94c40b0810 con 0x7f94b006d630 2023-11-24T08:25:13.470 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6 2023-11-24T08:25:13.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 msgr2=0x7f94b006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 0x7f94b006faf0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f94b4036000 tx=0x7f94b402e000 comp rx=0 tx=0).stop 2023-11-24T08:25:13.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 msgr2=0x7f94c409b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7f94b800dae0 tx=0x7f94b8007e50 comp rx=0 tx=0).stop 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 shutdown_connections 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f94b006d630 0x7f94b006faf0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94c40cc4a0 0x7f94c409b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94c40cc020 0x7f94c409b3d0 unknown :-1 s=CLOSED pgs=3419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 --2- 172.21.15.107:0/1187337015 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94ac005bb0 0x7f94c40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.471+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 >> 172.21.15.107:0/1187337015 conn(0x7f94c40b0c50 msgr2=0x7f94c40b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.472+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 shutdown_connections 2023-11-24T08:25:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.472+0000 7f94c9d1d640 1 -- 172.21.15.107:0/1187337015 wait complete. 2023-11-24T08:25:13.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol724 2023-11-24T08:25:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.702+0000 7f0f7e615640 1 Processor -- start 2023-11-24T08:25:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.702+0000 7f0f7e615640 1 -- start start 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f7e615640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f780cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f7e615640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f780c59b0 0x7f0f7806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f7e615640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780cc3a0 con 0x7f0f780cbf20 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f7e615640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7815b820 con 0x7f0f780c59b0 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f7e615640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7815eea0 con 0x7f0f7815b440 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f6ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0f780cbf20 0x7f0f780c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52440/0 (socket says 172.21.15.107:52440) 2023-11-24T08:25:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.704+0000 7f0f6ffff640 1 -- 172.21.15.107:0/659013860 learned_addr learned my addr 172.21.15.107:0/659013860 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f77fff640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f780cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606836417 0 0) 0x7f0f60001110 con 0x7f0f780cbf20 2023-11-24T08:25:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f64003520 con 0x7f0f780cbf20 2023-11-24T08:25:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> v2:172.21.15.120:3300/0 conn(0x7f0f780c59b0 0x7f0f7806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636205087 0 0) 0x7f0f64003520 con 0x7f0f780cbf20 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 msgr2=0x7f0f780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f780cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 >> v2:172.21.15.120:3300/0 conn(0x7f0f780c59b0 msgr2=0x7f0f7806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 --2- 172.21.15.107:0/659013860 >> v2:172.21.15.120:3300/0 conn(0x7f0f780c59b0 0x7f0f7806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780abf30 con 0x7f0f780cbf20 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f0f60001800 con 0x7f0f780cbf20 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 >> v1:172.21.15.107:6789/0 conn(0x7f0f780cbf20 legacy=0x7f0f780c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f64003970 0x7f0f64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64004d00 0x7f0f64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7815b820 con 0x7f0f64004d00 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7815eea0 con 0x7f0f64003970 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.705+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6400ba30 con 0x7f0f7815b440 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> v2:172.21.15.120:3300/0 conn(0x7f0f780c59b0 0x7f0f7806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64004d00 0x7f0f64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f77fff640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f64003970 0x7f0f64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f6ffff640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 msgr2=0x7f0f640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f7815b440 0x7f0f640047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f64003970 msgr2=0x7f0f64003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f64003970 0x7f0f64003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.706+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/659013860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64003520 con 0x7f0f64004d00 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.707+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64004d00 0x7f0f64005180 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f0f700073e0 tx=0x7f0f640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4babf5d35e819ddb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7002cdc0 con 0x7f0f64004d00 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70034440 con 0x7f0f64004d00 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f777fe640 1 -- 172.21.15.107:0/659013860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700347b0 con 0x7f0f64004d00 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f7e615640 1 -- 172.21.15.107:0/659013860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64004d00 msgr2=0x7f0f64005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f7e615640 1 --2- 172.21.15.107:0/659013860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64004d00 0x7f0f64005180 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f0f700073e0 tx=0x7f0f640057f0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/659013860 reap_dead start 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f7e615640 1 -- 172.21.15.107:0/659013860 shutdown_connections 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.708+0000 7f0f7e615640 1 -- 172.21.15.107:0/659013860 >> 172.21.15.107:0/659013860 conn(0x7f0f780b0c50 msgr2=0x7f0f780b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.709+0000 7f0f7e615640 1 -- 172.21.15.107:0/659013860 shutdown_connections 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.709+0000 7f0f7e615640 1 -- 172.21.15.107:0/659013860 wait complete. 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.710+0000 7f0f7e615640 1 Processor -- start 2023-11-24T08:25:13.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.710+0000 7f0f7e615640 1 -- start start 2023-11-24T08:25:13.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f64004d00 0x7f0f780c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f780ce690 0x7f0f78155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780c8690 con 0x7f0f7814fb50 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78150510 con 0x7f0f780ce690 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.711+0000 7f0f7e615640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780c8ab0 con 0x7f0f64004d00 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38918/0 (socket says 172.21.15.107:38918) 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/3135840024 learned_addr learned my addr 172.21.15.107:0/3135840024 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f64004d00 msgr2=0x7f0f780c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f64004d00 0x7f0f780c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f780ce690 msgr2=0x7f0f78155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f780ce690 0x7f0f78155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 -- 172.21.15.107:0/3135840024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780c7a10 con 0x7f0f7814fb50 2023-11-24T08:25:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.712+0000 7f0f7cbb2640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f0f70037040 tx=0x7f0f70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.713+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7003e440 con 0x7f0f7814fb50 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.713+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7003e5a0 con 0x7f0f7814fb50 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.713+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780c7cf0 con 0x7f0f7814fb50 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.713+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7003e950 con 0x7f0f7814fb50 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.713+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f780c82b0 con 0x7f0f7814fb50 2023-11-24T08:25:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.714+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f780c87d0 con 0x7f0f7814fb50 2023-11-24T08:25:13.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.715+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 51) v1 ==== 92852+0+0 (secure 0 0 0) 0x7f0f7003eab0 con 0x7f0f7814fb50 2023-11-24T08:25:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.716+0000 7f0f757fa640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 0x7f0f5806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.717+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f700aad50 con 0x7f0f7814fb50 2023-11-24T08:25:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.717+0000 7f0f77fff640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 0x7f0f5806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.717+0000 7f0f77fff640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 0x7f0f5806faf0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009890 tx=0x7f0f68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:13.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.721+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f780c87d0 con 0x7f0f7814fb50 2023-11-24T08:25:13.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.940+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f7815f6b0 con 0x7f0f5806d630 2023-11-24T08:25:13.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.945+0000 7f0f757fa640 1 -- 172.21.15.107:0/3135840024 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+65 (secure 0 0 0) 0x7f0f68009e70 con 0x7f0f5806d630 2023-11-24T08:25:13.947 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.947+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 msgr2=0x7f0f5806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.947+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 0x7f0f5806faf0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009890 tx=0x7f0f68008040 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.947+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 msgr2=0x7f0f7814ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.947+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f0f70037040 tx=0x7f0f70004ab0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 shutdown_connections 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f0f5806d630 0x7f0f5806faf0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f7814fb50 0x7f0f7814ffd0 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f780ce690 0x7f0f78155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 --2- 172.21.15.107:0/3135840024 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f64004d00 0x7f0f780c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 >> 172.21.15.107:0/3135840024 conn(0x7f0f780b0c50 msgr2=0x7f0f780acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 shutdown_connections 2023-11-24T08:25:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:13.948+0000 7f0f7e615640 1 -- 172.21.15.107:0/3135840024 wait complete. 2023-11-24T08:25:13.969 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol724 with 16 files each 1MB size under directory volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6 2023-11-24T08:25:13.969 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:13.969 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.022 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423987 s, 24.7 MB/s 2023-11-24T08:25:14.132 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.132 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.188 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.271 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.272 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.272 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286418 s, 36.6 MB/s 2023-11-24T08:25:14.273 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.274 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.332 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303965 s, 34.5 MB/s 2023-11-24T08:25:14.418 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.418 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.470 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.596 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.596 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.596 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237613 s, 44.1 MB/s 2023-11-24T08:25:14.599 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.599 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.723 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.723 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.723 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234031 s, 44.8 MB/s 2023-11-24T08:25:14.725 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.725 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.776 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320006 s, 32.8 MB/s 2023-11-24T08:25:14.909 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:14.909 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:14.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:14.968 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270781 s, 38.7 MB/s 2023-11-24T08:25:15.050 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.050 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.106 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172544 s, 60.8 MB/s 2023-11-24T08:25:15.179 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.179 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.231 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.304 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.305 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223758 s, 46.9 MB/s 2023-11-24T08:25:15.306 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.306 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.364 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.434 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.434 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186587 s, 56.2 MB/s 2023-11-24T08:25:15.435 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.435 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.484 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230576 s, 45.5 MB/s 2023-11-24T08:25:15.610 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.610 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.661 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.791 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308382 s, 34.0 MB/s 2023-11-24T08:25:15.793 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.793 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.847 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:15.918 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197642 s, 53.1 MB/s 2023-11-24T08:25:15.920 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:15.920 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:15.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:15.975 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:16.048 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211658 s, 49.5 MB/s 2023-11-24T08:25:16.049 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:16.049 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:16.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:16.106 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:16.179 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:16.179 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:16.180 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197749 s, 53.0 MB/s 2023-11-24T08:25:16.181 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:25:16.181 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:25:16.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:16.239 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol724/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0) 2023-11-24T08:25:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-11-24T08:25:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-11-24T08:25:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176856 s, 59.3 MB/s 2023-11-24T08:25:16.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol724 subvol_snap5619 2023-11-24T08:25:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.528+0000 7fc3758c9640 1 Processor -- start 2023-11-24T08:25:16.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.528+0000 7fc3758c9640 1 -- start start 2023-11-24T08:25:16.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.530+0000 7fc3758c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc3700b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.530+0000 7fc3758c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc37015de30 0x7fc37015e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.530+0000 7fc3758c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700ad010 con 0x7fc37015e7b0 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.530+0000 7fc3758c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3700b39c0 con 0x7fc37015de30 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.530+0000 7fc3758c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700acd20 con 0x7fc3701549f0 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc37015de30 0x7fc37015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc37015de30 0x7fc37015e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38928/0 (socket says 172.21.15.107:38928) 2023-11-24T08:25:16.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 -- 172.21.15.107:0/1528462471 learned_addr learned my addr 172.21.15.107:0/1528462471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 -- 172.21.15.107:0/1528462471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 msgr2=0x7fc3700b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:16.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc36effd640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc3700b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc3700b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 -- 172.21.15.107:0/1528462471 >> v1:172.21.15.120:6789/0 conn(0x7fc37015e7b0 legacy=0x7fc37015a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc3667fc640 1 -- 172.21.15.107:0/1528462471 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37015ec20 con 0x7fc37015de30 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.531+0000 7fc36effd640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc3700b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.532+0000 7fc3667fc640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37015e270 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fc358009870 tx=0x7fc35802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74433af5829c6af7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.533+0000 7fc36e7fc640 1 -- 172.21.15.107:0/1528462471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580317c0 con 0x7fc37015de30 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.533+0000 7fc36e7fc640 1 -- 172.21.15.107:0/1528462471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358031920 con 0x7fc37015de30 2023-11-24T08:25:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.533+0000 7fc36e7fc640 1 -- 172.21.15.107:0/1528462471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358031cb0 con 0x7fc37015de30 2023-11-24T08:25:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.533+0000 7fc3758c9640 1 -- 172.21.15.107:0/1528462471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 msgr2=0x7fc37015e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.533+0000 7fc3758c9640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37015e270 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fc358009870 tx=0x7fc35802b860 comp rx=0 tx=0).stop 2023-11-24T08:25:16.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 -- 172.21.15.107:0/1528462471 shutdown_connections 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37015e270 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 --2- 172.21.15.107:0/1528462471 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc3700b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 -- 172.21.15.107:0/1528462471 >> 172.21.15.107:0/1528462471 conn(0x7fc3700b0be0 msgr2=0x7fc3700b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 -- 172.21.15.107:0/1528462471 shutdown_connections 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.534+0000 7fc3758c9640 1 -- 172.21.15.107:0/1528462471 wait complete. 2023-11-24T08:25:16.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.536+0000 7fc3758c9640 1 Processor -- start 2023-11-24T08:25:16.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.536+0000 7fc3758c9640 1 -- start start 2023-11-24T08:25:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc37013ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015e7b0 0x7fc37013c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37014b2a0 con 0x7fc37015de30 2023-11-24T08:25:16.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37009b9d0 con 0x7fc37015e7b0 2023-11-24T08:25:16.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3758c9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc370140460 con 0x7fc3701549f0 2023-11-24T08:25:16.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3667fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3667fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38940/0 (socket says 172.21.15.107:38940) 2023-11-24T08:25:16.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.537+0000 7fc3667fc640 1 -- 172.21.15.107:0/2902884618 learned_addr learned my addr 172.21.15.107:0/2902884618 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:16.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc36effd640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc37013ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc36f7fe640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015e7b0 0x7fc37013c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 -- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 msgr2=0x7fc37013ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc37013ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 -- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015e7b0 msgr2=0x7fc37013c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015e7b0 0x7fc37013c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 -- 172.21.15.107:0/2902884618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3580094d0 con 0x7fc37015de30 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc36effd640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc37013ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.538+0000 7fc3667fc640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7fc35802b8a0 tx=0x7fc3580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.539+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580329a0 con 0x7fc37015de30 2023-11-24T08:25:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.539+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358032b40 con 0x7fc37015de30 2023-11-24T08:25:16.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.539+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580453f0 con 0x7fc37015de30 2023-11-24T08:25:16.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.539+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37009c920 con 0x7fc37015de30 2023-11-24T08:25:16.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.539+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37014ace0 con 0x7fc37015de30 2023-11-24T08:25:16.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.540+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37004e840 con 0x7fc37015de30 2023-11-24T08:25:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.541+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 52) v1 ==== 92903+0+0 (secure 0 0 0) 0x7fc35804e450 con 0x7fc37015de30 2023-11-24T08:25:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.542+0000 7fc36cff9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 0x7fc35006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.542+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3580aae20 con 0x7fc37015de30 2023-11-24T08:25:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.542+0000 7fc36effd640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 0x7fc35006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:16.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.543+0000 7fc36effd640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 0x7fc35006fb80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc360006fd0 tx=0x7fc360008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:16.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.546+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc358077a60 con 0x7fc37015de30 2023-11-24T08:25:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.771+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol724", "snap_name": "subvol_snap5619", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3700b36e0 con 0x7fc35006d6c0 2023-11-24T08:25:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.807+0000 7fc36cff9640 1 -- 172.21.15.107:0/2902884618 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc36000c040 con 0x7fc35006d6c0 2023-11-24T08:25:16.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 msgr2=0x7fc35006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:16.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 0x7fc35006fb80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc360006fd0 tx=0x7fc360008040 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 msgr2=0x7fc37009c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7fc35802b8a0 tx=0x7fc3580046c0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 shutdown_connections 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fc35006d6c0 0x7fc35006fb80 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015e7b0 0x7fc37013c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc37015de30 0x7fc37009c3e0 unknown :-1 s=CLOSED pgs=3424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 --2- 172.21.15.107:0/2902884618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3701549f0 0x7fc37013ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.809+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 >> 172.21.15.107:0/2902884618 conn(0x7fc3700b0be0 msgr2=0x7fc370071970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.810+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 shutdown_connections 2023-11-24T08:25:16.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:16.810+0000 7fc3758c9640 1 -- 172.21.15.107:0/2902884618 wait complete. 2023-11-24T08:25:16.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol724 --retain-snapshots 2023-11-24T08:25:16.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:25:16.934 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T08:25:17.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.034+0000 7f944ffff640 1 Processor -- start 2023-11-24T08:25:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.034+0000 7f944ffff640 1 -- start start 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94500b2ad0 0x7f945006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94500b2f60 0x7f945006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944ffff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9450143c10 con 0x7f94501431f0 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944ffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94500b4fd0 con 0x7f94500b2ad0 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94500705b0 con 0x7f94500b2f60 2023-11-24T08:25:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f94467fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94500b2ad0 0x7f945006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f94467fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f94500b2ad0 0x7f945006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38952/0 (socket says 172.21.15.107:38952) 2023-11-24T08:25:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f94467fc640 1 -- 172.21.15.107:0/312201044 learned_addr learned my addr 172.21.15.107:0/312201044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944effd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f94501431f0 0x7f945015e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.107:52374/0 (socket says 172.21.15.107:52374) 2023-11-24T08:25:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.036+0000 7f944f7fe640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94500b2f60 0x7f945006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.037+0000 7f94467fc640 1 -- 172.21.15.107:0/312201044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94500b2f60 msgr2=0x7f945006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:17.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.037+0000 7f94467fc640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94500b2f60 0x7f945006dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.037+0000 7f944e7fc640 1 -- 172.21.15.107:0/312201044 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477968445 0 0) 0x7f9450143c10 con 0x7f94501431f0 2023-11-24T08:25:17.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.037+0000 7f94467fc640 1 -- 172.21.15.107:0/312201044 >> v1:172.21.15.120:6789/0 conn(0x7f94501431f0 legacy=0x7f945015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.037+0000 7f94467fc640 1 -- 172.21.15.107:0/312201044 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450070890 con 0x7f94500b2ad0 2023-11-24T08:25:17.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.038+0000 7f94467fc640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2ad0 0x7f945006d6a0 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f943800baf0 tx=0x7f943802e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524fd1c1a36a0176 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.038+0000 7f944e7fc640 1 -- 172.21.15.107:0/312201044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943802ee50 con 0x7f94500b2ad0 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.038+0000 7f944e7fc640 1 -- 172.21.15.107:0/312201044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438004030 con 0x7f94500b2ad0 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.038+0000 7f944e7fc640 1 -- 172.21.15.107:0/312201044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94380043e0 con 0x7f94500b2ad0 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 -- 172.21.15.107:0/312201044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2ad0 msgr2=0x7f945006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2ad0 0x7f945006d6a0 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f943800baf0 tx=0x7f943802e0a0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 -- 172.21.15.107:0/312201044 shutdown_connections 2023-11-24T08:25:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94500b2f60 0x7f945006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 --2- 172.21.15.107:0/312201044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2ad0 0x7f945006d6a0 unknown :-1 s=CLOSED pgs=3425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 -- 172.21.15.107:0/312201044 >> 172.21.15.107:0/312201044 conn(0x7f94500cb8a0 msgr2=0x7f94500cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 -- 172.21.15.107:0/312201044 shutdown_connections 2023-11-24T08:25:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.039+0000 7f944ffff640 1 -- 172.21.15.107:0/312201044 wait complete. 2023-11-24T08:25:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.041+0000 7f944ffff640 1 Processor -- start 2023-11-24T08:25:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.041+0000 7f944ffff640 1 -- start start 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94500b2ad0 0x7f94501e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 0x7f945015bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94500c81d0 con 0x7f94500b2f60 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94500c8d50 con 0x7f94500b2ad0 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944ffff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94501e6140 con 0x7f94501431f0 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f94467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f94467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38968/0 (socket says 172.21.15.107:38968) 2023-11-24T08:25:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f94467fc640 1 -- 172.21.15.107:0/4096569477 learned_addr learned my addr 172.21.15.107:0/4096569477 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.042+0000 7f944effd640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94500b2ad0 0x7f94501e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f944f7fe640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 0x7f945015bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 -- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 msgr2=0x7f945015bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 0x7f945015bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 -- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94500b2ad0 msgr2=0x7f94501e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94500b2ad0 0x7f94501e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 -- 172.21.15.107:0/4096569477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943800b700 con 0x7f94500b2f60 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f944f7fe640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 0x7f945015bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f94467fc640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f943802e0e0 tx=0x7f94380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438044070 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438003610 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94380039c0 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94500c8420 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.043+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94500c89e0 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.044+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94500c8ec0 con 0x7f94500b2f60 2023-11-24T08:25:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.045+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 52) v1 ==== 92903+0+0 (secure 0 0 0) 0x7f9438003b20 con 0x7f94500b2f60 2023-11-24T08:25:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.046+0000 7f9447fff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 0x7f943006fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.046+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94380aac00 con 0x7f94500b2f60 2023-11-24T08:25:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.047+0000 7f944effd640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 0x7f943006fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.047+0000 7f944effd640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 0x7f943006fbb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9440007ed0 tx=0x7f9440007e60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.051+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94380777b0 con 0x7f94500b2f60 2023-11-24T08:25:17.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.271+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol724", "retain_snapshots": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9450067680 con 0x7f943006d6f0 2023-11-24T08:25:17.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.342+0000 7f9447fff640 1 -- 172.21.15.107:0/4096569477 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9450067680 con 0x7f943006d6f0 2023-11-24T08:25:17.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 msgr2=0x7f943006fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 0x7f943006fbb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9440007ed0 tx=0x7f9440007e60 comp rx=0 tx=0).stop 2023-11-24T08:25:17.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 msgr2=0x7f945012e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f943802e0e0 tx=0x7f94380046c0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 shutdown_connections 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f943006d6f0 0x7f943006fbb0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94501431f0 0x7f945015bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500b2f60 0x7f945012e260 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 --2- 172.21.15.107:0/4096569477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94500b2ad0 0x7f94501e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 >> 172.21.15.107:0/4096569477 conn(0x7f94500cb8a0 msgr2=0x7f94500b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.345+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 shutdown_connections 2023-11-24T08:25:17.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.346+0000 7f944ffff640 1 -- 172.21.15.107:0/4096569477 wait complete. 2023-11-24T08:25:17.353 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.351+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.353 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.351+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.353 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.351+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.353 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.351+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.353 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.351+0000 7f37657a2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol724 subvol_snap5619 subvol724 2023-11-24T08:25:17.374 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.373+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.375 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.373+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.375 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.373+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.375 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.373+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.375 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:17.373+0000 7f3764fa1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:17.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.575+0000 7f764b085640 1 Processor -- start 2023-11-24T08:25:17.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.575+0000 7f764b085640 1 -- start start 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f764b085640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7644166160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f764b085640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440ca0e0 con 0x7f7644070550 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f764b085640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644072800 con 0x7f764415b1d0 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f764b085640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764415fde0 con 0x7f76440b1250 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f7649882640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7644070550 0x7f76440b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52454/0 (socket says 172.21.15.107:52454) 2023-11-24T08:25:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.577+0000 7f7649882640 1 -- 172.21.15.107:0/3554704405 learned_addr learned my addr 172.21.15.107:0/3554704405 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212972698 0 0) 0x7f76440ca0e0 con 0x7f7644070550 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f764a884640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7644166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003520 con 0x7f7644070550 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984633207 0 0) 0x7f7630003520 con 0x7f7644070550 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 msgr2=0x7f7644166160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7644166160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 >> v1:172.21.15.120:6789/0 conn(0x7f764415b1d0 legacy=0x7f7644071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440c97b0 con 0x7f7644070550 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f7634001670 con 0x7f7644070550 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 >> v1:172.21.15.107:6789/0 conn(0x7f7644070550 legacy=0x7f76440b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76300039a0 0x7f7630003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7630005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764415fde0 con 0x7f76300039a0 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763000bb40 con 0x7f76300047c0 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763000bd00 con 0x7f76440b1250 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.578+0000 7f764a083640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76300039a0 0x7f7630003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f7649882640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a884640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7630005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a083640 1 -- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 msgr2=0x7f7630005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:17.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a083640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76440b1250 0x7f7630005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a083640 1 -- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300047c0 msgr2=0x7f7630004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:17.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a083640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.579+0000 7f764a083640 1 -- 172.21.15.107:0/3554704405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630003520 con 0x7f76300039a0 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.580+0000 7f764a083640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76300039a0 0x7f7630003dc0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f7640001750 tx=0x7f76300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3a448c39ec7731 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.580+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400052d0 con 0x7f76300039a0 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.580+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764002e050 con 0x7f76300039a0 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.580+0000 7f7649081640 1 -- 172.21.15.107:0/3554704405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400035f0 con 0x7f76300039a0 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 -- 172.21.15.107:0/3554704405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76300039a0 msgr2=0x7f7630003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 --2- 172.21.15.107:0/3554704405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76300039a0 0x7f7630003dc0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f7640001750 tx=0x7f76300057f0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764a884640 1 -- 172.21.15.107:0/3554704405 reap_dead start 2023-11-24T08:25:17.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 -- 172.21.15.107:0/3554704405 shutdown_connections 2023-11-24T08:25:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 -- 172.21.15.107:0/3554704405 >> 172.21.15.107:0/3554704405 conn(0x7f76440cbb40 msgr2=0x7f76440cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 -- 172.21.15.107:0/3554704405 shutdown_connections 2023-11-24T08:25:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.581+0000 7f764b085640 1 -- 172.21.15.107:0/3554704405 wait complete. 2023-11-24T08:25:17.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.583+0000 7f764b085640 1 Processor -- start 2023-11-24T08:25:17.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.583+0000 7f764b085640 1 -- start start 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300039a0 0x7f76440c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 0x7f76440b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76440c50c0 con 0x7f764409b230 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76440b2200 con 0x7f76440c5550 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764b085640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76440c44b0 con 0x7f76300039a0 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764a884640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764a884640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39022/0 (socket says 172.21.15.107:39022) 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764a884640 1 -- 172.21.15.107:0/1962905395 learned_addr learned my addr 172.21.15.107:0/1962905395 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f764a083640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300039a0 0x7f76440c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.584+0000 7f7649882640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 0x7f76440b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 -- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300039a0 msgr2=0x7f76440c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300039a0 0x7f76440c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 -- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 msgr2=0x7f76440b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 0x7f76440b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 -- 172.21.15.107:0/1962905395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440c4790 con 0x7f764409b230 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f7649882640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 0x7f76440b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:17.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764a884640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f76440c96a0 tx=0x7f763c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c007db0 con 0x7f764409b230 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c00f050 con 0x7f764409b230 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76440c4a70 con 0x7f764409b230 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c014650 con 0x7f764409b230 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.585+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764409b6b0 con 0x7f764409b230 2023-11-24T08:25:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.587+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764404e840 con 0x7f764409b230 2023-11-24T08:25:17.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.588+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 52) v1 ==== 92903+0+0 (secure 0 0 0) 0x7f763c0147b0 con 0x7f764409b230 2023-11-24T08:25:17.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.589+0000 7f763affd640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 0x7f762806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:17.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.589+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f763c088f60 con 0x7f764409b230 2023-11-24T08:25:17.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.589+0000 7f764a083640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 0x7f762806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:17.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.590+0000 7f764a083640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 0x7f762806fb80 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7640033000 tx=0x7f7640009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:17.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.593+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f763c055a70 con 0x7f764409b230 2023-11-24T08:25:17.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:17.820+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol724", "snap_name": "subvol_snap5619", "target_sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f76440ae670 con 0x7f762806d6c0 2023-11-24T08:25:18.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:18.679+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 53) v1 ==== 93005+0+0 (secure 0 0 0) 0x7f763c055e40 con 0x7f764409b230 2023-11-24T08:25:20.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.234+0000 7f763affd640 1 -- 172.21.15.107:0/1962905395 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76440ae670 con 0x7f762806d6c0 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 msgr2=0x7f762806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 0x7f762806fb80 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7640033000 tx=0x7f7640009030 comp rx=0 tx=0).stop 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 msgr2=0x7f764414a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f76440c96a0 tx=0x7f763c009da0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 shutdown_connections 2023-11-24T08:25:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f762806d6c0 0x7f762806fb80 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f764409b230 0x7f764414a070 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76440c5550 0x7f76440b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 --2- 172.21.15.107:0/1962905395 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300039a0 0x7f76440c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 >> 172.21.15.107:0/1962905395 conn(0x7f76440cbb40 msgr2=0x7f76440ac030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 shutdown_connections 2023-11-24T08:25:20.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.239+0000 7f764b085640 1 -- 172.21.15.107:0/1962905395 wait complete. 2023-11-24T08:25:20.250 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.248+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.250 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.248+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.250 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.248+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.250 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.248+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.250 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.248+0000 7f37697aa640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol724 2023-11-24T08:25:20.274 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.272+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.274 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.272+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.274 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.272+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.274 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.272+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.274 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-11-24T08:25:20.272+0000 7f3768fa9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T08:25:20.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.460+0000 7f2c167ec640 1 Processor -- start 2023-11-24T08:25:20.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.460+0000 7f2c167ec640 1 -- start start 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c167ec640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2c100c59b0 0x7f2c10166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c167ec640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c100c5dc0 con 0x7f2c1015b440 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c167ec640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1015eea0 con 0x7f2c100cc0f0 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c167ec640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1015b820 con 0x7f2c100c59b0 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c0ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2c1015b440 0x7f2c100cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39344/0 (socket says 172.21.15.107:39344) 2023-11-24T08:25:20.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c0ffff640 1 -- 172.21.15.107:0/716934379 learned_addr learned my addr 172.21.15.107:0/716934379 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.462+0000 7f2c14d89640 1 --2- 172.21.15.107:0/716934379 >> v2:172.21.15.120:3300/0 conn(0x7f2c100c59b0 0x7f2c10166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044272163 0 0) 0x7f2c100c5dc0 con 0x7f2c1015b440 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf8003520 con 0x7f2c1015b440 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254090031 0 0) 0x7f2bf8003520 con 0x7f2c1015b440 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 >> v2:172.21.15.120:3300/0 conn(0x7f2c100c59b0 msgr2=0x7f2c10166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 --2- 172.21.15.107:0/716934379 >> v2:172.21.15.120:3300/0 conn(0x7f2c100c59b0 0x7f2c10166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 >> v1:172.21.15.120:6789/0 conn(0x7f2c100cc0f0 legacy=0x7f2c100c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1006d600 con 0x7f2c1015b440 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7f2c00001670 con 0x7f2c1015b440 2023-11-24T08:25:20.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 >> v1:172.21.15.107:6789/0 conn(0x7f2c1015b440 legacy=0x7f2c100cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 0x7f2bf8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf80047c0 0x7f2bf8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8005180 0x7f2bf800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1015b820 con 0x7f2bf8003970 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf800c120 con 0x7f2bf8005180 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.463+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf800c2e0 con 0x7f2bf80047c0 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 0x7f2bf8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c14d89640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8005180 0x7f2bf800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 -- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf80047c0 msgr2=0x7f2bf8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf80047c0 0x7f2bf8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 -- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8005180 msgr2=0x7f2bf800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:20.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8005180 0x7f2bf800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c0ffff640 1 -- 172.21.15.107:0/716934379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8003520 con 0x7f2bf8003970 2023-11-24T08:25:20.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.464+0000 7f2c14d89640 1 -- 172.21.15.107:0/716934379 reap_dead start 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.465+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 0x7f2bf8003d90 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000c190 tx=0x7f2bf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bc9bf92a582ad6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.465+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c00005ad0 con 0x7f2bf8003970 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.465+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c00005c30 con 0x7f2bf8003970 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.465+0000 7f2c0effd640 1 -- 172.21.15.107:0/716934379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0002ca70 con 0x7f2bf8003970 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 -- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 msgr2=0x7f2bf8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 0x7f2bf8003d90 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000c190 tx=0x7f2bf80057f0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 -- 172.21.15.107:0/716934379 shutdown_connections 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 --2- 172.21.15.107:0/716934379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bf8003970 0x7f2bf8003d90 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 -- 172.21.15.107:0/716934379 >> 172.21.15.107:0/716934379 conn(0x7f2c100b0c50 msgr2=0x7f2c100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 -- 172.21.15.107:0/716934379 shutdown_connections 2023-11-24T08:25:20.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.466+0000 7f2c167ec640 1 -- 172.21.15.107:0/716934379 wait complete. 2023-11-24T08:25:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.468+0000 7f2c167ec640 1 Processor -- start 2023-11-24T08:25:20.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.468+0000 7f2c167ec640 1 -- start start 2023-11-24T08:25:20.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 0x7f2c1015e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c10156740 0x7f2c101598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1015dba0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c100c9080 con 0x7f2bf8003970 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c167ec640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c10156bc0 con 0x7f2c10156740 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60768/0 (socket says 172.21.15.107:60768) 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.469+0000 7f2c0f7fe640 1 -- 172.21.15.107:0/269149547 learned_addr learned my addr 172.21.15.107:0/269149547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 0x7f2c1015e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 -- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c10156740 msgr2=0x7f2c101598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c10156740 0x7f2c101598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 -- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 msgr2=0x7f2c1015e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 0x7f2c1015e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 -- 172.21.15.107:0/269149547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1015cf90 con 0x7f2c1015c9f0 2023-11-24T08:25:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 0x7f2c1015e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0f7fe640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c390 tx=0x7f2c0400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c040052d0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c04005450 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1015d270 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c04007260 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.470+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1015d830 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.472+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1015d9f0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.473+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 53) v1 ==== 93005+0+0 (secure 0 0 0) 0x7f2c040073c0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.473+0000 7f2c0cff9640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 0x7f2bfc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.474+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c04087da0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.474+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 0x7f2bfc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:20.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.474+0000 7f2c0ffff640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 0x7f2bfc06fca0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f2c00034000 tx=0x7f2c00009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:20.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.478+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c040548b0 con 0x7f2c1015c9f0 2023-11-24T08:25:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.707+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c1015f3d0 con 0x7f2bfc06d7e0 2023-11-24T08:25:20.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.722+0000 7f2c0cff9640 1 -- 172.21.15.107:0/269149547 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7f2c00001470 con 0x7f2bfc06d7e0 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol724", 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5619" 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:20.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:20.725 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:25:20.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.724+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 msgr2=0x7f2bfc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.724+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 0x7f2bfc06fca0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f2c00034000 tx=0x7f2c00009030 comp rx=0 tx=0).stop 2023-11-24T08:25:20.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.724+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 msgr2=0x7f2c10157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:20.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.724+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c390 tx=0x7f2c0400c850 comp rx=0 tx=0).stop 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 shutdown_connections 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f2bfc06d7e0 0x7f2bfc06fca0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c10156740 0x7f2c101598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c1015c9f0 0x7f2c10157fe0 unknown :-1 s=CLOSED pgs=3440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 --2- 172.21.15.107:0/269149547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf8003970 0x7f2c1015e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:20.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 >> 172.21.15.107:0/269149547 conn(0x7f2c100b0c50 msgr2=0x7f2c100b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:20.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 shutdown_connections 2023-11-24T08:25:20.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:20.725+0000 7f2c167ec640 1 -- 172.21.15.107:0/269149547 wait complete. 2023-11-24T08:25:21.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol724 2023-11-24T08:25:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.969+0000 7ff18ed7a640 1 Processor -- start 2023-11-24T08:25:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.969+0000 7ff18ed7a640 1 -- start start 2023-11-24T08:25:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18ed7a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1880b0b40 0x7ff1880b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18ed7a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18ed7a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18815ed20 con 0x7ff18815b260 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18ed7a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18815b640 con 0x7ff1880b0b40 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18ed7a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880b14f0 con 0x7ff18806c5b0 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18e579640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18d577640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1880b0b40 0x7ff1880b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18e579640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.107:38910/0 (socket says 172.21.15.107:38910) 2023-11-24T08:25:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.971+0000 7ff18e579640 1 -- 172.21.15.107:0/2676312913 learned_addr learned my addr 172.21.15.107:0/2676312913 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.972+0000 7ff18e579640 1 -- 172.21.15.107:0/2676312913 >> v2:172.21.15.120:3300/0 conn(0x7ff1880b0b40 msgr2=0x7ff1880b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.972+0000 7ff18e579640 1 --2- 172.21.15.107:0/2676312913 >> v2:172.21.15.120:3300/0 conn(0x7ff1880b0b40 0x7ff1880b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.972+0000 7ff18e579640 1 -- 172.21.15.107:0/2676312913 >> v1:172.21.15.120:6789/0 conn(0x7ff18815b260 legacy=0x7ff18806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.972+0000 7ff18e579640 1 -- 172.21.15.107:0/2676312913 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18815f8a0 con 0x7ff18806c5b0 2023-11-24T08:25:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.973+0000 7ff18e579640 1 --2- 172.21.15.107:0/2676312913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff180009e50 tx=0x7ff18002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72d85f37eb46ec3c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.974+0000 7ff18cd76640 1 -- 172.21.15.107:0/2676312913 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18002c9e0 con 0x7ff18806c5b0 2023-11-24T08:25:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.974+0000 7ff18cd76640 1 -- 172.21.15.107:0/2676312913 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18002cb40 con 0x7ff18806c5b0 2023-11-24T08:25:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.974+0000 7ff18cd76640 1 -- 172.21.15.107:0/2676312913 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800343f0 con 0x7ff18806c5b0 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 -- 172.21.15.107:0/2676312913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 msgr2=0x7ff18806ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/2676312913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff180009e50 tx=0x7ff18002be40 comp rx=0 tx=0).stop 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 -- 172.21.15.107:0/2676312913 shutdown_connections 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/2676312913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18806c5b0 0x7ff18806ca40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/2676312913 >> v2:172.21.15.120:3300/0 conn(0x7ff1880b0b40 0x7ff1880b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 -- 172.21.15.107:0/2676312913 >> 172.21.15.107:0/2676312913 conn(0x7ff1880cbb40 msgr2=0x7ff1880cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 -- 172.21.15.107:0/2676312913 shutdown_connections 2023-11-24T08:25:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.975+0000 7ff18ed7a640 1 -- 172.21.15.107:0/2676312913 wait complete. 2023-11-24T08:25:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.977+0000 7ff18ed7a640 1 Processor -- start 2023-11-24T08:25:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.977+0000 7ff18ed7a640 1 -- start start 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.978+0000 7ff18ed7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.978+0000 7ff18ed7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1880b0b40 0x7ff1881f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18ed7a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18815b260 0x7ff1881514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18ed7a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff188159950 con 0x7ff18806c5b0 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18ed7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880c29a0 con 0x7ff1880b0b40 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18ed7a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1881f6d20 con 0x7ff18815b260 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60774/0 (socket says 172.21.15.107:60774) 2023-11-24T08:25:21.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 -- 172.21.15.107:0/1359047222 learned_addr learned my addr 172.21.15.107:0/1359047222 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 -- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18815b260 msgr2=0x7ff1881514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18815b260 0x7ff1881514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 -- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1880b0b40 msgr2=0x7ff1881f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1880b0b40 0x7ff1881f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 -- 172.21.15.107:0/1359047222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180009b30 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.979+0000 7ff18dd78640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c400 tx=0x7ff18400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.980+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840052d0 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.980+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff184005430 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.980+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184003830 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.980+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1881f7000 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.980+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff188151a30 con 0x7ff18806c5b0 2023-11-24T08:25:21.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.981+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1881f7520 con 0x7ff18806c5b0 2023-11-24T08:25:21.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.982+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 53) v1 ==== 93005+0+0 (secure 0 0 0) 0x7ff184003990 con 0x7ff18806c5b0 2023-11-24T08:25:21.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.983+0000 7ff17e7fc640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 0x7ff16c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:21.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.983+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff184087f50 con 0x7ff18806c5b0 2023-11-24T08:25:21.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.983+0000 7ff18d577640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 0x7ff16c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:21.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.984+0000 7ff18d577640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 0x7ff16c06fca0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff178006e80 tx=0x7ff178006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:21.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:21.988+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff184054b00 con 0x7ff18806c5b0 2023-11-24T08:25:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.199+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1880ce3e0 con 0x7ff16c06d7e0 2023-11-24T08:25:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.210+0000 7ff17e7fc640 1 -- 172.21.15.107:0/1359047222 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+162 (secure 0 0 0) 0x7ff1880ce3e0 con 0x7ff16c06d7e0 2023-11-24T08:25:22.212 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:25:22.212 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:25:22.212 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "pending", 2023-11-24T08:25:22.212 INFO:teuthology.orchestra.run.smithi107.stdout: "source": { 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout: "volume": "cephfs", 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout: "subvolume": "subvol724", 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot": "subvol_snap5619" 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:22.213 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 msgr2=0x7ff16c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 0x7ff16c06fca0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff178006e80 tx=0x7ff178006dd0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 msgr2=0x7ff1880c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c400 tx=0x7ff18400c8c0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 shutdown_connections 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff16c06d7e0 0x7ff16c06fca0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.213+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18815b260 0x7ff1881514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.214+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1880b0b40 0x7ff1881f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.214+0000 7ff18ed7a640 1 --2- 172.21.15.107:0/1359047222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff18806c5b0 0x7ff1880c5420 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.214+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 >> 172.21.15.107:0/1359047222 conn(0x7ff1880cbb40 msgr2=0x7ff1880b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.214+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 shutdown_connections 2023-11-24T08:25:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:22.214+0000 7ff18ed7a640 1 -- 172.21.15.107:0/1359047222 wait complete. 2023-11-24T08:25:23.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol724 2023-11-24T08:25:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.460+0000 7f20d5e2e640 1 Processor -- start 2023-11-24T08:25:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.460+0000 7f20d5e2e640 1 -- start start 2023-11-24T08:25:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d5e2e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20d015d3f0 0x7f20d00d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d5e2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d00d0e40 0x7f20d00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d5e2e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d00d15e0 con 0x7f20d015d7d0 2023-11-24T08:25:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d5e2e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d00d0280 con 0x7f20d015d3f0 2023-11-24T08:25:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d5e2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00d1300 con 0x7f20d00d0e40 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20cffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f20d015d7d0 0x7f20d00afea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39346/0 (socket says 172.21.15.107:39346) 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20cffff640 1 -- 172.21.15.107:0/1588896090 learned_addr learned my addr 172.21.15.107:0/1588896090 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.462+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/1588896090 >> v2:172.21.15.107:3300/0 conn(0x7f20d015d3f0 0x7f20d00d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.463+0000 7f20d562d640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d00d0e40 0x7f20d00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.463+0000 7f20d4e2c640 1 -- 172.21.15.107:0/1588896090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d00d0e40 msgr2=0x7f20d00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.463+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d00d0e40 0x7f20d00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.463+0000 7f20d4e2c640 1 -- 172.21.15.107:0/1588896090 >> v1:172.21.15.107:6789/0 conn(0x7f20d015d7d0 legacy=0x7f20d00afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:23.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.463+0000 7f20d4e2c640 1 -- 172.21.15.107:0/1588896090 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00cfc30 con 0x7f20d015d3f0 2023-11-24T08:25:23.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.465+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d015d3f0 0x7f20d00d1830 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097a0 tx=0x7f20c402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e52e04e6cbdc56b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:23.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.465+0000 7f20cf7fe640 1 -- 172.21.15.107:0/1588896090 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c402d040 con 0x7f20d015d3f0 2023-11-24T08:25:23.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.465+0000 7f20cf7fe640 1 -- 172.21.15.107:0/1588896090 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4031810 con 0x7f20d015d3f0 2023-11-24T08:25:23.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.465+0000 7f20cf7fe640 1 -- 172.21.15.107:0/1588896090 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4031ba0 con 0x7f20d015d3f0 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 -- 172.21.15.107:0/1588896090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d015d3f0 msgr2=0x7f20d00d1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d015d3f0 0x7f20d00d1830 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097a0 tx=0x7f20c402b8b0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 -- 172.21.15.107:0/1588896090 shutdown_connections 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d00d0e40 0x7f20d00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/1588896090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d015d3f0 0x7f20d00d1830 unknown :-1 s=CLOSED pgs=3443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 -- 172.21.15.107:0/1588896090 >> 172.21.15.107:0/1588896090 conn(0x7f20d009c5b0 msgr2=0x7f20d00d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 -- 172.21.15.107:0/1588896090 shutdown_connections 2023-11-24T08:25:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.466+0000 7f20d5e2e640 1 -- 172.21.15.107:0/1588896090 wait complete. 2023-11-24T08:25:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.468+0000 7f20d5e2e640 1 Processor -- start 2023-11-24T08:25:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.469+0000 7f20d5e2e640 1 -- start start 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d015d3f0 0x7f20d0159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 0x7f20d0159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00b5940 con 0x7f20d00d0e40 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00c9650 con 0x7f20d015d7d0 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d5e2e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00b52f0 con 0x7f20d015d3f0 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60794/0 (socket says 172.21.15.107:60794) 2023-11-24T08:25:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 -- 172.21.15.107:0/3174675129 learned_addr learned my addr 172.21.15.107:0/3174675129 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 -- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d015d3f0 msgr2=0x7f20d0159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d015d3f0 0x7f20d0159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d562d640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 0x7f20d0159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 -- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 msgr2=0x7f20d0159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 0x7f20d0159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d4e2c640 1 -- 172.21.15.107:0/3174675129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4009400 con 0x7f20d00d0e40 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.470+0000 7f20d562d640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 0x7f20d0159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20d4e2c640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f20c40098c0 tx=0x7f20c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c403ad60 con 0x7f20d00d0e40 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4044400 con 0x7f20d00d0e40 2023-11-24T08:25:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d00b55d0 con 0x7f20d00d0e40 2023-11-24T08:25:23.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c40447b0 con 0x7f20d00d0e40 2023-11-24T08:25:23.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.471+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d00b4f10 con 0x7f20d00d0e40 2023-11-24T08:25:23.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.473+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d00b5760 con 0x7f20d00d0e40 2023-11-24T08:25:23.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.473+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7f20c404d440 con 0x7f20d00d0e40 2023-11-24T08:25:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.474+0000 7f20cd7fa640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 0x7f20bc06fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.474+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20c40aaf30 con 0x7f20d00d0e40 2023-11-24T08:25:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.475+0000 7f20cffff640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 0x7f20bc06fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.475+0000 7f20cffff640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 0x7f20bc06fdc0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f20d00c6700 tx=0x7f20c0007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.479+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20c4077a40 con 0x7f20d00d0e40 2023-11-24T08:25:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:23.706+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d00d0710 con 0x7f20bc06d900 2023-11-24T08:25:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.215+0000 7f20cd7fa640 1 -- 172.21.15.107:0/3174675129 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f20c0001210 con 0x7f20bc06d900 2023-11-24T08:25:24.217 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:25:24.217 INFO:teuthology.orchestra.run.smithi107.stdout: "status": { 2023-11-24T08:25:24.218 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete" 2023-11-24T08:25:24.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-11-24T08:25:24.218 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:25:24.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 msgr2=0x7f20bc06fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 0x7f20bc06fdc0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f20d00c6700 tx=0x7f20c0007a40 comp rx=0 tx=0).stop 2023-11-24T08:25:24.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 msgr2=0x7f20d0155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f20c40098c0 tx=0x7f20c40046c0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 shutdown_connections 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f20bc06d900 0x7f20bc06fdc0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d015d7d0 0x7f20d0159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.218+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d015d3f0 0x7f20d0159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.219+0000 7f20d5e2e640 1 --2- 172.21.15.107:0/3174675129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d00d0e40 0x7f20d0155b50 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.219+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 >> 172.21.15.107:0/3174675129 conn(0x7f20d009c5b0 msgr2=0x7f20d0160450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.219+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 shutdown_connections 2023-11-24T08:25:24.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.219+0000 7f20d5e2e640 1 -- 172.21.15.107:0/3174675129 wait complete. 2023-11-24T08:25:24.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol724 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.462+0000 7ff61ea44640 1 Processor -- start 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.463+0000 7ff61ea44640 1 -- start start 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff618072160 0x7ff61815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6180c49f0 0x7ff61806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff61806dc40 0x7ff6180cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61806e0f0 con 0x7ff61806dc40 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61815ec90 con 0x7ff618072160 2023-11-24T08:25:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.464+0000 7ff61ea44640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180cd530 con 0x7ff6180c49f0 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff61806dc40 0x7ff6180cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff61806dc40 0x7ff6180cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60804/0 (socket says 172.21.15.107:60804) 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 -- 172.21.15.107:0/523428210 learned_addr learned my addr 172.21.15.107:0/523428210 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 -- 172.21.15.107:0/523428210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6180c49f0 msgr2=0x7ff61806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 --2- 172.21.15.107:0/523428210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6180c49f0 0x7ff61806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 -- 172.21.15.107:0/523428210 >> v2:172.21.15.120:3300/0 conn(0x7ff618072160 msgr2=0x7ff61815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 --2- 172.21.15.107:0/523428210 >> v2:172.21.15.120:3300/0 conn(0x7ff618072160 0x7ff61815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.465+0000 7ff61cfe1640 1 -- 172.21.15.107:0/523428210 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180c4ed0 con 0x7ff61806dc40 2023-11-24T08:25:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.466+0000 7ff61cfe1640 1 --2- 172.21.15.107:0/523428210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff61806dc40 0x7ff6180cd750 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7ff6180ccae0 tx=0x7ff61002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19222c99c0ef4ac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff6177fe640 1 -- 172.21.15.107:0/523428210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61002ca80 con 0x7ff61806dc40 2023-11-24T08:25:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff6177fe640 1 -- 172.21.15.107:0/523428210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61002cbe0 con 0x7ff61806dc40 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff6177fe640 1 -- 172.21.15.107:0/523428210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6100343f0 con 0x7ff61806dc40 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 -- 172.21.15.107:0/523428210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff61806dc40 msgr2=0x7ff6180cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 --2- 172.21.15.107:0/523428210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff61806dc40 0x7ff6180cd750 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7ff6180ccae0 tx=0x7ff61002bee0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 -- 172.21.15.107:0/523428210 shutdown_connections 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 --2- 172.21.15.107:0/523428210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff61806dc40 0x7ff6180cd750 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 --2- 172.21.15.107:0/523428210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6180c49f0 0x7ff61806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 --2- 172.21.15.107:0/523428210 >> v2:172.21.15.120:3300/0 conn(0x7ff618072160 0x7ff61815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 -- 172.21.15.107:0/523428210 >> 172.21.15.107:0/523428210 conn(0x7ff6180b0c50 msgr2=0x7ff6180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.467+0000 7ff61ea44640 1 -- 172.21.15.107:0/523428210 shutdown_connections 2023-11-24T08:25:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.468+0000 7ff61ea44640 1 -- 172.21.15.107:0/523428210 wait complete. 2023-11-24T08:25:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.469+0000 7ff61ea44640 1 Processor -- start 2023-11-24T08:25:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.469+0000 7ff61ea44640 1 -- start start 2023-11-24T08:25:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff61806dc40 0x7ff6181f0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6180c49f0 0x7ff6180afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180c9340 con 0x7ff618072160 2023-11-24T08:25:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180c8aa0 con 0x7ff6180c49f0 2023-11-24T08:25:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.470+0000 7ff61ea44640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180c8d80 con 0x7ff61806dc40 2023-11-24T08:25:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60810/0 (socket says 172.21.15.107:60810) 2023-11-24T08:25:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 -- 172.21.15.107:0/1614497266 learned_addr learned my addr 172.21.15.107:0/1614497266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 -- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff61806dc40 msgr2=0x7ff6181f0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff61806dc40 0x7ff6181f0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 -- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6180c49f0 msgr2=0x7ff6180afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6180c49f0 0x7ff6180afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 -- 172.21.15.107:0/1614497266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610009b30 con 0x7ff618072160 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.471+0000 7ff60ffff640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7ff60000b410 tx=0x7ff60000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.472+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600011830 con 0x7ff618072160 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.472+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000119b0 con 0x7ff618072160 2023-11-24T08:25:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.472+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600011d60 con 0x7ff618072160 2023-11-24T08:25:24.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.472+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6180c9060 con 0x7ff618072160 2023-11-24T08:25:24.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.472+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6181f0e90 con 0x7ff618072160 2023-11-24T08:25:24.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.473+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff618157770 con 0x7ff618072160 2023-11-24T08:25:24.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.474+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7ff60001a400 con 0x7ff618072160 2023-11-24T08:25:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.475+0000 7ff6157fa640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 0x7ff5f806fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.475+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff600088c70 con 0x7ff618072160 2023-11-24T08:25:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.475+0000 7ff617fff640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 0x7ff5f806fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.476+0000 7ff617fff640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 0x7ff5f806fdc0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff608009710 tx=0x7ff608006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.480+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff618157770 con 0x7ff618072160 2023-11-24T08:25:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.698+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6180ada20 con 0x7ff5f806d900 2023-11-24T08:25:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.709+0000 7ff6157fa640 1 -- 172.21.15.107:0/1614497266 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+65 (secure 0 0 0) 0x7ff6180ada20 con 0x7ff5f806d900 2023-11-24T08:25:24.711 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad 2023-11-24T08:25:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 msgr2=0x7ff5f806fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 0x7ff5f806fdc0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff608009710 tx=0x7ff608006c90 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 msgr2=0x7ff6180c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7ff60000b410 tx=0x7ff60000b8d0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 shutdown_connections 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7ff5f806d900 0x7ff5f806fdc0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6180c49f0 0x7ff6180afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff618072160 0x7ff6180c7fe0 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 --2- 172.21.15.107:0/1614497266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff61806dc40 0x7ff6181f0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.712+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 >> 172.21.15.107:0/1614497266 conn(0x7ff6180b0c50 msgr2=0x7ff61815e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.713+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 shutdown_connections 2023-11-24T08:25:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.713+0000 7ff61ea44640 1 -- 172.21.15.107:0/1614497266 wait complete. 2023-11-24T08:25:24.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol724 2023-11-24T08:25:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.948+0000 7fa5ee67c640 1 Processor -- start 2023-11-24T08:25:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.948+0000 7fa5ee67c640 1 -- start start 2023-11-24T08:25:24.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5ee67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5e80cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5ee67c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80b23f0 con 0x7fa5e8160a70 2023-11-24T08:25:24.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5ee67c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80d09c0 con 0x7fa5e8160e50 2023-11-24T08:25:24.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5ee67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b25c0 con 0x7fa5e80d0370 2023-11-24T08:25:24.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5e7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa5e8160a70 0x7fa5e80cfc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39356/0 (socket says 172.21.15.107:39356) 2023-11-24T08:25:24.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5e7fff640 1 -- 172.21.15.107:0/2129779686 learned_addr learned my addr 172.21.15.107:0/2129779686 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.950+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5e80cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560752247 0 0) 0x7fa5e80b23f0 con 0x7fa5e8160a70 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d4003520 con 0x7fa5e8160a70 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158058235 0 0) 0x7fa5d4003520 con 0x7fa5e8160a70 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 msgr2=0x7fa5e80cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5e80cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 >> v1:172.21.15.120:6789/0 conn(0x7fa5e8160e50 legacy=0x7fa5e80ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80b2110 con 0x7fa5e8160a70 2023-11-24T08:25:24.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 302854337 0 0) 0x7fa5d80019f0 con 0x7fa5e8160a70 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 >> v1:172.21.15.107:6789/0 conn(0x7fa5e8160a70 legacy=0x7fa5e80cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d40039a0 0x7fa5d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5d40051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b25c0 con 0x7fa5d40047c0 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d400bb10 con 0x7fa5d40039a0 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.951+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d400bcd0 con 0x7fa5e80d0370 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e7fff640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d40039a0 0x7fa5d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5ecc19640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5d40051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 -- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 msgr2=0x7fa5d40051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5d40051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 -- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d40039a0 msgr2=0x7fa5d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d40039a0 0x7fa5d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e77fe640 1 -- 172.21.15.107:0/2129779686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4003520 con 0x7fa5d40047c0 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5ecc19640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e80d0370 0x7fa5d40051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:24.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.952+0000 7fa5e7fff640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5d40039a0 0x7fa5d4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.953+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5d4004c40 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ac0 tx=0x7fa5d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2480a243e0381ac0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.953+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d002c6b0 con 0x7fa5d40047c0 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.953+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d002c810 con 0x7fa5d40047c0 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.953+0000 7fa5e6ffd640 1 -- 172.21.15.107:0/2129779686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d002cba0 con 0x7fa5d40047c0 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 -- 172.21.15.107:0/2129779686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 msgr2=0x7fa5d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/2129779686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5d4004c40 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ac0 tx=0x7fa5d40057f0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ecc19640 1 -- 172.21.15.107:0/2129779686 reap_dead start 2023-11-24T08:25:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 -- 172.21.15.107:0/2129779686 shutdown_connections 2023-11-24T08:25:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 -- 172.21.15.107:0/2129779686 >> 172.21.15.107:0/2129779686 conn(0x7fa5e809c6d0 msgr2=0x7fa5e80d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 -- 172.21.15.107:0/2129779686 shutdown_connections 2023-11-24T08:25:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.954+0000 7fa5ee67c640 1 -- 172.21.15.107:0/2129779686 wait complete. 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.956+0000 7fa5ee67c640 1 Processor -- start 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.956+0000 7fa5ee67c640 1 -- start start 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e8151b20 0x7fa5e80b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5e8154300 0x7fa5e80c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b68e0 con 0x7fa5d40047c0 2023-11-24T08:25:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b8bc0 con 0x7fa5e8154300 2023-11-24T08:25:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5ee67c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b6b30 con 0x7fa5e8151b20 2023-11-24T08:25:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60830/0 (socket says 172.21.15.107:60830) 2023-11-24T08:25:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.957+0000 7fa5e7fff640 1 -- 172.21.15.107:0/1240495511 learned_addr learned my addr 172.21.15.107:0/1240495511 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 -- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e8151b20 msgr2=0x7fa5e80b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e8151b20 0x7fa5e80b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5ecc19640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5e8154300 0x7fa5e80c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e8151b20 0x7fa5e80b5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 -- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5e8154300 msgr2=0x7fa5e80c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5e8154300 0x7fa5e80c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 -- 172.21.15.107:0/1240495511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80b6e10 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e7fff640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800dfc0 tx=0x7fa5d800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8004d60 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d80070d0 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.958+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e80b70f0 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.959+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8007440 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.959+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e8151f80 con 0x7fa5d40047c0 2023-11-24T08:25:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.960+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e8153f60 con 0x7fa5d40047c0 2023-11-24T08:25:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.961+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7fa5d80075a0 con 0x7fa5d40047c0 2023-11-24T08:25:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.961+0000 7fa5e4ff9640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 0x7fa5c806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.962+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d8088f50 con 0x7fa5d40047c0 2023-11-24T08:25:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.962+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 0x7fa5c806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.963+0000 7fa5e77fe640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 0x7fa5c806fdf0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ac0 tx=0x7fa5d0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:24.966+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5d8055b00 con 0x7fa5d40047c0 2023-11-24T08:25:25.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.185+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5e80b2860 con 0x7fa5c806d930 2023-11-24T08:25:25.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.199+0000 7fa5e4ff9640 1 -- 172.21.15.107:0/1240495511 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+701 (secure 0 0 0) 0x7fa5d0004140 con 0x7fa5c806d930 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "atime": "2023-11-24 08:25:12", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_pcent": "undefined", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_quota": "infinite", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "bytes_used": 16777216, 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "created_at": "2023-11-24 08:25:20", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "ctime": "2023-11-24 08:25:23", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "data_pool": "cephfs_data", 2023-11-24T08:25:25.201 INFO:teuthology.orchestra.run.smithi107.stdout: "features": [ 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-clone", 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-autoprotect", 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot-retention" 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 0, 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": 16895, 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_addrs": [ 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6789", 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6789", 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.120:6790" 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "mtime": "2023-11-24 08:25:16", 2023-11-24T08:25:25.202 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad", 2023-11-24T08:25:25.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_namespace": "", 2023-11-24T08:25:25.203 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "complete", 2023-11-24T08:25:25.203 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "clone", 2023-11-24T08:25:25.203 INFO:teuthology.orchestra.run.smithi107.stdout: "uid": 0 2023-11-24T08:25:25.203 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 msgr2=0x7fa5c806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 0x7fa5c806fdf0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ac0 tx=0x7fa5d0030000 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 msgr2=0x7fa5e80b85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800dfc0 tx=0x7fa5d800b560 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 shutdown_connections 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7fa5c806d930 0x7fa5c806fdf0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5e8154300 0x7fa5e80c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5e8151b20 0x7fa5e80b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 --2- 172.21.15.107:0/1240495511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d40047c0 0x7fa5e80b85e0 unknown :-1 s=CLOSED pgs=3449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 >> 172.21.15.107:0/1240495511 conn(0x7fa5e809c6d0 msgr2=0x7fa5e80b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:25.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.201+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 shutdown_connections 2023-11-24T08:25:25.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:25.202+0000 7fa5ee67c640 1 -- 172.21.15.107:0/1240495511 wait complete. 2023-11-24T08:25:25.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.221 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6: ceph.quota.max_bytes: No such attribute 2023-11-24T08:25:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:25:25.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.276 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.326 INFO:teuthology.orchestra.run.smithi107.stdout:cephfs_data 2023-11-24T08:25:25.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.327 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.373 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.472 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6 2023-11-24T08:25:25.472 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9 2023-11-24T08:25:25.473 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15 2023-11-24T08:25:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8 2023-11-24T08:25:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11 2023-11-24T08:25:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3 2023-11-24T08:25:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12 2023-11-24T08:25:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1 2023-11-24T08:25:25.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.475 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.528 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:25:25.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.529 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/) 2023-11-24T08:25:25.583 INFO:teuthology.orchestra.run.smithi107.stdout:41ff 2023-11-24T08:25:25.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.585 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.640 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:25:25.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.641 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/) 2023-11-24T08:25:25.694 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:25:25.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.695 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.748 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:25:25.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.750 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/) 2023-11-24T08:25:25.802 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-11-24T08:25:25.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.803 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6) 2023-11-24T08:25:25.857 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:25.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.859 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/) 2023-11-24T08:25:25.912 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:25.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.914 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2) 2023-11-24T08:25:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:25.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:25.967 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.2) 2023-11-24T08:25:26.031 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:26.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.032 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2) 2023-11-24T08:25:26.088 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:26.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.090 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.2) 2023-11-24T08:25:26.141 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:26.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.177 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2) 2023-11-24T08:25:26.238 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:26.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.240 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.2) 2023-11-24T08:25:26.296 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:26.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.298 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.2) 2023-11-24T08:25:26.354 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:26.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.356 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.2) 2023-11-24T08:25:26.411 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:26.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.412 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14) 2023-11-24T08:25:26.465 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:26.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.466 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.14) 2023-11-24T08:25:26.524 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:26.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.526 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14) 2023-11-24T08:25:26.574 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:26.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.576 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.14) 2023-11-24T08:25:26.624 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:26.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.625 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14) 2023-11-24T08:25:26.679 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:26.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.680 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.14) 2023-11-24T08:25:26.731 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:26.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.733 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.14) 2023-11-24T08:25:26.783 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:26.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.785 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.14) 2023-11-24T08:25:26.835 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:26.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.837 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13) 2023-11-24T08:25:26.888 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:26.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.890 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.13) 2023-11-24T08:25:26.948 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:26.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:26.951 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13) 2023-11-24T08:25:27.001 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.002 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.13) 2023-11-24T08:25:27.051 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.052 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13) 2023-11-24T08:25:27.100 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:27.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.101 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.13) 2023-11-24T08:25:27.152 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:27.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.154 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.13) 2023-11-24T08:25:27.203 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:27.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.205 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.13) 2023-11-24T08:25:27.256 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:27.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.257 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10) 2023-11-24T08:25:27.306 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:27.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.10) 2023-11-24T08:25:27.365 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:27.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.367 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10) 2023-11-24T08:25:27.422 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.423 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.10) 2023-11-24T08:25:27.474 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.475 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10) 2023-11-24T08:25:27.527 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:27.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.528 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.10) 2023-11-24T08:25:27.584 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:27.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.586 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.10) 2023-11-24T08:25:27.639 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:27.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.641 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.10) 2023-11-24T08:25:27.695 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:27.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.697 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5) 2023-11-24T08:25:27.746 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:27.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.748 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.5) 2023-11-24T08:25:27.806 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:27.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.808 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5) 2023-11-24T08:25:27.863 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.864 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.5) 2023-11-24T08:25:27.915 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:27.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.916 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5) 2023-11-24T08:25:27.965 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:27.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:27.966 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.5) 2023-11-24T08:25:28.016 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:28.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.018 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.5) 2023-11-24T08:25:28.066 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:28.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.067 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.5) 2023-11-24T08:25:28.118 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:28.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.119 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4) 2023-11-24T08:25:28.172 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:28.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.174 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.4) 2023-11-24T08:25:28.232 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:28.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.234 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4) 2023-11-24T08:25:28.287 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:28.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.288 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.4) 2023-11-24T08:25:28.339 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:28.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.341 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4) 2023-11-24T08:25:28.390 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:28.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.391 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.4) 2023-11-24T08:25:28.439 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:28.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.440 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.4) 2023-11-24T08:25:28.540 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:28.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.541 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.4) 2023-11-24T08:25:28.595 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:28.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.597 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7) 2023-11-24T08:25:28.652 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:28.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.653 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.7) 2023-11-24T08:25:28.713 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:28.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.715 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7) 2023-11-24T08:25:28.768 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:28.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.7) 2023-11-24T08:25:28.821 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:28.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.822 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7) 2023-11-24T08:25:28.872 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:28.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.874 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.7) 2023-11-24T08:25:28.925 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:28.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.926 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.7) 2023-11-24T08:25:28.979 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:28.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:28.980 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.7) 2023-11-24T08:25:29.033 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:29.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.035 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6) 2023-11-24T08:25:29.086 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:29.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.087 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.6) 2023-11-24T08:25:29.139 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:29.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.141 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6) 2023-11-24T08:25:29.192 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:29.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.194 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.6) 2023-11-24T08:25:29.250 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:29.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.251 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6) 2023-11-24T08:25:29.304 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:29.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.306 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.6) 2023-11-24T08:25:29.360 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:29.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.362 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.6) 2023-11-24T08:25:29.415 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:29.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.417 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.6) 2023-11-24T08:25:29.470 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:29.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.471 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0) 2023-11-24T08:25:29.524 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:29.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.525 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.0) 2023-11-24T08:25:29.579 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:29.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0) 2023-11-24T08:25:29.633 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:29.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.634 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.0) 2023-11-24T08:25:29.685 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:29.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.687 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0) 2023-11-24T08:25:29.739 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:29.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.740 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.0) 2023-11-24T08:25:29.793 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:29.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.794 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.0) 2023-11-24T08:25:29.847 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:29.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.849 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.0) 2023-11-24T08:25:29.909 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:29.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.910 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9) 2023-11-24T08:25:29.963 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:29.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:29.965 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.9) 2023-11-24T08:25:30.020 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:30.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.021 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9) 2023-11-24T08:25:30.072 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.074 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.9) 2023-11-24T08:25:30.126 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.127 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9) 2023-11-24T08:25:30.181 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:30.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.182 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.9) 2023-11-24T08:25:30.234 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:30.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.236 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.9) 2023-11-24T08:25:30.290 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:30.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.291 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.9) 2023-11-24T08:25:30.345 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:30.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.347 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15) 2023-11-24T08:25:30.397 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:30.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.399 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.15) 2023-11-24T08:25:30.448 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:30.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.449 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15) 2023-11-24T08:25:30.497 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.498 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.15) 2023-11-24T08:25:30.551 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.552 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15) 2023-11-24T08:25:30.608 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:30.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.609 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.15) 2023-11-24T08:25:30.662 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:30.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.664 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.15) 2023-11-24T08:25:30.716 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:30.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.718 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.15) 2023-11-24T08:25:30.772 INFO:teuthology.orchestra.run.smithi107.stdout:1700814316 2023-11-24T08:25:30.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.774 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8) 2023-11-24T08:25:30.825 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:30.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.826 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.8) 2023-11-24T08:25:30.877 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:30.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.879 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8) 2023-11-24T08:25:30.930 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.932 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.8) 2023-11-24T08:25:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:30.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:30.985 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8) 2023-11-24T08:25:31.038 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.040 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.8) 2023-11-24T08:25:31.092 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.093 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.8) 2023-11-24T08:25:31.145 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:31.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.146 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.8) 2023-11-24T08:25:31.198 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:31.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.199 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11) 2023-11-24T08:25:31.251 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:31.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.252 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.11) 2023-11-24T08:25:31.306 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:31.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.307 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11) 2023-11-24T08:25:31.358 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:31.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.359 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.11) 2023-11-24T08:25:31.417 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:31.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.418 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11) 2023-11-24T08:25:31.473 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.475 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.11) 2023-11-24T08:25:31.525 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.527 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.11) 2023-11-24T08:25:31.578 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:31.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.580 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.11) 2023-11-24T08:25:31.631 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:31.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.632 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3) 2023-11-24T08:25:31.684 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:31.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.685 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.3) 2023-11-24T08:25:31.737 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:31.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.739 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3) 2023-11-24T08:25:31.795 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:31.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.796 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.3) 2023-11-24T08:25:31.849 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:31.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3) 2023-11-24T08:25:31.904 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.3) 2023-11-24T08:25:31.958 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:31.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:31.959 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.3) 2023-11-24T08:25:32.011 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:32.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.012 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.3) 2023-11-24T08:25:32.065 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:32.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.066 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12) 2023-11-24T08:25:32.116 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:32.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.118 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.12) 2023-11-24T08:25:32.167 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:32.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.168 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12) 2023-11-24T08:25:32.219 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:32.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.221 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.12) 2023-11-24T08:25:32.273 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:32.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.275 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12) 2023-11-24T08:25:32.331 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:32.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.333 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.12) 2023-11-24T08:25:32.387 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:32.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.388 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.12) 2023-11-24T08:25:32.440 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:32.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.442 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.12) 2023-11-24T08:25:32.494 INFO:teuthology.orchestra.run.smithi107.stdout:1700814315 2023-11-24T08:25:32.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.496 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1) 2023-11-24T08:25:32.546 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:32.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.548 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.1) 2023-11-24T08:25:32.600 INFO:teuthology.orchestra.run.smithi107.stdout:81a4 2023-11-24T08:25:32.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.601 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1) 2023-11-24T08:25:32.653 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:32.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.654 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.1) 2023-11-24T08:25:32.706 INFO:teuthology.orchestra.run.smithi107.stdout:1000 2023-11-24T08:25:32.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.708 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1) 2023-11-24T08:25:32.760 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:32.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.761 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.1) 2023-11-24T08:25:32.813 INFO:teuthology.orchestra.run.smithi107.stdout:1301 2023-11-24T08:25:32.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.815 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/.snap/subvol_snap5619/daf1222a-ce60-46b0-8ef8-d806aa336bc6/subvolume_file.1) 2023-11-24T08:25:32.865 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:32.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:32.866 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol724/3fccd0c2-f415-4753-b201-7089670b33ad/subvolume_file.1) 2023-11-24T08:25:32.917 INFO:teuthology.orchestra.run.smithi107.stdout:1700814314 2023-11-24T08:25:32.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol724 subvol_snap5619 2023-11-24T08:25:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.137+0000 7f67469a0640 1 Processor -- start 2023-11-24T08:25:33.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.138+0000 7f67469a0640 1 -- start start 2023-11-24T08:25:33.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.139+0000 7f67469a0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6740160b00 0x7f67400d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.139+0000 7f67469a0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67400d02f0 0x7f674015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.139+0000 7f67469a0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674015ea30 con 0x7f67400d12e0 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.139+0000 7f67469a0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674015de20 con 0x7f6740160b00 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.139+0000 7f67469a0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674015e100 con 0x7f67400d02f0 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674599e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6740160b00 0x7f67400d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674519d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f67400d12e0 0x7f67400cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35860/0 (socket says 172.21.15.107:35860) 2023-11-24T08:25:33.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674519d640 1 -- 172.21.15.107:0/3355178114 learned_addr learned my addr 172.21.15.107:0/3355178114 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674599e640 1 -- 172.21.15.107:0/3355178114 >> v2:172.21.15.120:3300/0 conn(0x7f67400d02f0 msgr2=0x7f674015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674599e640 1 --2- 172.21.15.107:0/3355178114 >> v2:172.21.15.120:3300/0 conn(0x7f67400d02f0 0x7f674015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674599e640 1 -- 172.21.15.107:0/3355178114 >> v1:172.21.15.107:6789/0 conn(0x7f67400d12e0 legacy=0x7f67400cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674599e640 1 -- 172.21.15.107:0/3355178114 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674015e3e0 con 0x7f6740160b00 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.140+0000 7f674499c640 1 -- 172.21.15.107:0/3355178114 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128139747 0 0) 0x7f674015ea30 con 0x7f67400d12e0 2023-11-24T08:25:33.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.141+0000 7f674599e640 1 --2- 172.21.15.107:0/3355178114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6740160b00 0x7f67400d0da0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f674008c500 tx=0x7f673002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646a225fd6467d59 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f674499c640 1 -- 172.21.15.107:0/3355178114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673002cc90 con 0x7f6740160b00 2023-11-24T08:25:33.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f674499c640 1 -- 172.21.15.107:0/3355178114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673002cdf0 con 0x7f6740160b00 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f674499c640 1 -- 172.21.15.107:0/3355178114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6730034660 con 0x7f6740160b00 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 -- 172.21.15.107:0/3355178114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6740160b00 msgr2=0x7f67400d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 --2- 172.21.15.107:0/3355178114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6740160b00 0x7f67400d0da0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f674008c500 tx=0x7f673002bee0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 -- 172.21.15.107:0/3355178114 shutdown_connections 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 --2- 172.21.15.107:0/3355178114 >> v2:172.21.15.120:3300/0 conn(0x7f67400d02f0 0x7f674015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 --2- 172.21.15.107:0/3355178114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6740160b00 0x7f67400d0da0 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 -- 172.21.15.107:0/3355178114 >> 172.21.15.107:0/3355178114 conn(0x7f674009c510 msgr2=0x7f67400d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 -- 172.21.15.107:0/3355178114 shutdown_connections 2023-11-24T08:25:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.142+0000 7f67469a0640 1 -- 172.21.15.107:0/3355178114 wait complete. 2023-11-24T08:25:33.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.144+0000 7f67469a0640 1 Processor -- start 2023-11-24T08:25:33.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.144+0000 7f67469a0640 1 -- start start 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67400d02f0 0x7f67401507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6740160b00 0x7f674015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6740150d10 con 0x7f67400d12e0 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b68e0 con 0x7f67400d02f0 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f67469a0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b6bc0 con 0x7f6740160b00 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.145+0000 7f674519d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54236/0 (socket says 172.21.15.107:54236) 2023-11-24T08:25:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 -- 172.21.15.107:0/680885453 learned_addr learned my addr 172.21.15.107:0/680885453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 -- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6740160b00 msgr2=0x7f674015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6740160b00 0x7f674015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 -- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67400d02f0 msgr2=0x7f67401507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67400d02f0 0x7f67401507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 -- 172.21.15.107:0/680885453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730009a70 con 0x7f67400d12e0 2023-11-24T08:25:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f674519d640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f673400c4d0 tx=0x7f673400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340074e0 con 0x7f67400d12e0 2023-11-24T08:25:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6734007660 con 0x7f67400d12e0 2023-11-24T08:25:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734004a20 con 0x7f67400d12e0 2023-11-24T08:25:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.146+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740152050 con 0x7f67400d12e0 2023-11-24T08:25:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.147+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67400b6ea0 con 0x7f67400d12e0 2023-11-24T08:25:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.148+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6740152570 con 0x7f67400d12e0 2023-11-24T08:25:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.149+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7f6734004b80 con 0x7f67400d12e0 2023-11-24T08:25:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.150+0000 7f673e7fc640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 0x7f672c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.150+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6734087bf0 con 0x7f67400d12e0 2023-11-24T08:25:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.150+0000 7f674599e640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 0x7f672c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.151+0000 7f674599e640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 0x7f672c06fdf0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6730036000 tx=0x7f673002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.155+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6740152570 con 0x7f67400d12e0 2023-11-24T08:25:33.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.385+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol724", "snap_name": "subvol_snap5619", "target": ["mon-mgr", ""]}) v1 -- 0x7f674015ee40 con 0x7f672c06d930 2023-11-24T08:25:33.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.448+0000 7f673e7fc640 1 -- 172.21.15.107:0/680885453 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f67300344c0 con 0x7f672c06d930 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.450+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 msgr2=0x7f672c06fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.450+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 0x7f672c06fdf0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6730036000 tx=0x7f673002e000 comp rx=0 tx=0).stop 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.450+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 msgr2=0x7f67400c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.450+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f673400c4d0 tx=0x7f673400c990 comp rx=0 tx=0).stop 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 shutdown_connections 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f672c06d930 0x7f672c06fdf0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6740160b00 0x7f674015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67400d12e0 0x7f67400c6d00 unknown :-1 s=CLOSED pgs=3452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 --2- 172.21.15.107:0/680885453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67400d02f0 0x7f67401507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 >> 172.21.15.107:0/680885453 conn(0x7f674009c510 msgr2=0x7f67400d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 shutdown_connections 2023-11-24T08:25:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.451+0000 7f67469a0640 1 -- 172.21.15.107:0/680885453 wait complete. 2023-11-24T08:25:33.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol724 2023-11-24T08:25:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.679+0000 7f46bc333640 1 Processor -- start 2023-11-24T08:25:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.680+0000 7f46bc333640 1 -- start start 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46bc333640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46b415d720 0x7f46b40d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46bc333640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 0x7f46b40b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46bc333640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b415db00 con 0x7f46b40d3870 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46bc333640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40d3d20 con 0x7f46b415d720 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46bc333640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40d0ce0 con 0x7f46b406ca90 2023-11-24T08:25:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46ba0cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46b415d720 0x7f46b40d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.681+0000 7f46ba0cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46b415d720 0x7f46b40d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54252/0 (socket says 172.21.15.107:54252) 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba0cf640 1 -- 172.21.15.107:0/2165617180 learned_addr learned my addr 172.21.15.107:0/2165617180 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba0cf640 1 -- 172.21.15.107:0/2165617180 >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 msgr2=0x7f46b40b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba8d0640 1 --2- 172.21.15.107:0/2165617180 >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 0x7f46b40b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/2165617180 >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 0x7f46b40b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba0cf640 1 -- 172.21.15.107:0/2165617180 >> v1:172.21.15.120:6789/0 conn(0x7f46b40d3870 legacy=0x7f46b40b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba0cf640 1 -- 172.21.15.107:0/2165617180 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b406d400 con 0x7f46b415d720 2023-11-24T08:25:33.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.682+0000 7f46ba8d0640 1 --2- 172.21.15.107:0/2165617180 >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 0x7f46b40b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.683+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/2165617180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b415d720 0x7f46b40d3330 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f46b40d4140 tx=0x7f46a802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65905f0e3c924f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.683+0000 7f46b90cd640 1 -- 172.21.15.107:0/2165617180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a802c9f0 con 0x7f46b415d720 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.683+0000 7f46b90cd640 1 -- 172.21.15.107:0/2165617180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a802cb50 con 0x7f46b415d720 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.683+0000 7f46b90cd640 1 -- 172.21.15.107:0/2165617180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a80343f0 con 0x7f46b415d720 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 -- 172.21.15.107:0/2165617180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b415d720 msgr2=0x7f46b40d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 --2- 172.21.15.107:0/2165617180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b415d720 0x7f46b40d3330 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f46b40d4140 tx=0x7f46a802be50 comp rx=0 tx=0).stop 2023-11-24T08:25:33.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 -- 172.21.15.107:0/2165617180 shutdown_connections 2023-11-24T08:25:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 --2- 172.21.15.107:0/2165617180 >> v2:172.21.15.120:3300/0 conn(0x7f46b406ca90 0x7f46b40b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 --2- 172.21.15.107:0/2165617180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b415d720 0x7f46b40d3330 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 -- 172.21.15.107:0/2165617180 >> 172.21.15.107:0/2165617180 conn(0x7f46b408af60 msgr2=0x7f46b409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 -- 172.21.15.107:0/2165617180 shutdown_connections 2023-11-24T08:25:33.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.684+0000 7f46bc333640 1 -- 172.21.15.107:0/2165617180 wait complete. 2023-11-24T08:25:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.686+0000 7f46bc333640 1 Processor -- start 2023-11-24T08:25:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.686+0000 7f46bc333640 1 -- start start 2023-11-24T08:25:33.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 0x7f46b40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46b415d720 0x7f46b415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b41f38a0 con 0x7f46b40d3870 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b41f3b80 con 0x7f46b406ca90 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46bc333640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b41f4160 con 0x7f46b415d720 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.687+0000 7f46b98ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54266/0 (socket says 172.21.15.107:54266) 2023-11-24T08:25:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 -- 172.21.15.107:0/3613421909 learned_addr learned my addr 172.21.15.107:0/3613421909 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:33.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 0x7f46b40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 -- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46b415d720 msgr2=0x7f46b415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46b415d720 0x7f46b415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 -- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 msgr2=0x7f46b40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 0x7f46b40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 -- 172.21.15.107:0/3613421909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8009b30 con 0x7f46b40d3870 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 0x7f46b40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b98ce640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c430 tx=0x7f46a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.688+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0011790 con 0x7f46b40d3870 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.689+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a00118f0 con 0x7f46b40d3870 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.689+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0011ce0 con 0x7f46b40d3870 2023-11-24T08:25:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.689+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b41f4440 con 0x7f46b40d3870 2023-11-24T08:25:33.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.689+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b40cd240 con 0x7f46b40d3870 2023-11-24T08:25:33.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.690+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b404e8b0 con 0x7f46b40d3870 2023-11-24T08:25:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.691+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7f46a001c460 con 0x7f46b40d3870 2023-11-24T08:25:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.692+0000 7f46b2ffd640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 0x7f469806fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.692+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46a0088aa0 con 0x7f46b40d3870 2023-11-24T08:25:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.692+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 0x7f469806fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.693+0000 7f46ba0cf640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 0x7f469806fdc0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f46a8035000 tx=0x7f46a8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.697+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46a00555b0 con 0x7f46b40d3870 2023-11-24T08:25:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.916+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol724", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b40b06c0 con 0x7f469806d900 2023-11-24T08:25:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.937+0000 7f46b2ffd640 1 -- 172.21.15.107:0/3613421909 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46a8040520 con 0x7f469806d900 2023-11-24T08:25:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.940+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 msgr2=0x7f469806fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.940+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 0x7f469806fdc0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f46a8035000 tx=0x7f46a8030000 comp rx=0 tx=0).stop 2023-11-24T08:25:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.940+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 msgr2=0x7f46b41f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.940+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c430 tx=0x7f46a000c8f0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 shutdown_connections 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f469806d900 0x7f469806fdc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46b415d720 0x7f46b415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46b40d3870 0x7f46b41f2e90 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 --2- 172.21.15.107:0/3613421909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b406ca90 0x7f46b40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 >> 172.21.15.107:0/3613421909 conn(0x7f46b408af60 msgr2=0x7f46b40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 shutdown_connections 2023-11-24T08:25:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:33.941+0000 7f46bc333640 1 -- 172.21.15.107:0/3613421909 wait complete. 2023-11-24T08:25:33.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2023-11-24T08:25:34.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.168+0000 7f38f5b1d640 1 Processor -- start 2023-11-24T08:25:34.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.168+0000 7f38f5b1d640 1 -- start start 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38f5b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38f00c67c0 0x7f38f0161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38f5b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38f5b1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f015de00 con 0x7f38f0140520 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38f5b1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f015e0e0 con 0x7f38f00c67c0 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38f5b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f015e3c0 con 0x7f38f0161940 2023-11-24T08:25:34.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38effff640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 msgr2=0x7f38f006ff70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2023-11-24T08:25:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38eeffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38f00c67c0 0x7f38f0161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38effff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:25:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38eeffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f38f00c67c0 0x7f38f0161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54274/0 (socket says 172.21.15.107:54274) 2023-11-24T08:25:34.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.170+0000 7f38eeffd640 1 -- 172.21.15.107:0/4062073260 learned_addr learned my addr 172.21.15.107:0/4062073260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.171+0000 7f38eeffd640 1 -- 172.21.15.107:0/4062073260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 msgr2=0x7f38f006ff70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.171+0000 7f38eeffd640 1 --2- 172.21.15.107:0/4062073260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.171+0000 7f38eeffd640 1 -- 172.21.15.107:0/4062073260 >> v1:172.21.15.107:6789/0 conn(0x7f38f0140520 legacy=0x7f38f00c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.171+0000 7f38eeffd640 1 -- 172.21.15.107:0/4062073260 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f015e6a0 con 0x7f38f00c67c0 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.172+0000 7f38eeffd640 1 --2- 172.21.15.107:0/4062073260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f0161400 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009870 tx=0x7f38e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c429fc7fb05b950f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.172+0000 7f38ee7fc640 1 -- 172.21.15.107:0/4062073260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4030860 con 0x7f38f00c67c0 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.172+0000 7f38ee7fc640 1 -- 172.21.15.107:0/4062073260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e40309c0 con 0x7f38f00c67c0 2023-11-24T08:25:34.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.172+0000 7f38ee7fc640 1 -- 172.21.15.107:0/4062073260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4030d50 con 0x7f38f00c67c0 2023-11-24T08:25:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4062073260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 msgr2=0x7f38f0161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4062073260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f0161400 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009870 tx=0x7f38e402b9a0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4062073260 shutdown_connections 2023-11-24T08:25:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4062073260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f006ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4062073260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f0161400 unknown :-1 s=CLOSED pgs=3460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4062073260 >> 172.21.15.107:0/4062073260 conn(0x7f38f00b2e40 msgr2=0x7f38f00b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4062073260 shutdown_connections 2023-11-24T08:25:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.173+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4062073260 wait complete. 2023-11-24T08:25:34.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.175+0000 7f38f5b1d640 1 Processor -- start 2023-11-24T08:25:34.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.175+0000 7f38f5b1d640 1 -- start start 2023-11-24T08:25:34.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 0x7f38f0157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f0157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f013f320 con 0x7f38f00c67c0 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0159180 con 0x7f38f0140520 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38f5b1d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f013e4d0 con 0x7f38f0161940 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38eeffd640 1 -- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 msgr2=0x7f38f0157920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38effff640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 msgr2=0x7f38f0157e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38eeffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 0x7f38f0157920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38effff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f0157e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54290/0 (socket says 172.21.15.107:54290) 2023-11-24T08:25:34.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.176+0000 7f38ef7fe640 1 -- 172.21.15.107:0/4012566849 learned_addr learned my addr 172.21.15.107:0/4012566849 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 -- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 msgr2=0x7f38f0157e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f0157e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 -- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 msgr2=0x7f38f0157920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 0x7f38f0157920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 -- 172.21.15.107:0/4012566849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e40094d0 con 0x7f38f00c67c0 2023-11-24T08:25:34.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38ef7fe640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f38e000c7c0 tx=0x7f38e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0003b90 con 0x7f38f00c67c0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e0003cf0 con 0x7f38f00c67c0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f013e6e0 con 0x7f38f00c67c0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0003ea0 con 0x7f38f00c67c0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.177+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f013ec60 con 0x7f38f00c67c0 2023-11-24T08:25:34.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.179+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f004e8b0 con 0x7f38f00c67c0 2023-11-24T08:25:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.180+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 54) v1 ==== 93107+0+0 (secure 0 0 0) 0x7f38e000aa90 con 0x7f38f00c67c0 2023-11-24T08:25:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.180+0000 7f38d7fff640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 0x7f38dc06fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T08:25:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.181+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38e00889f0 con 0x7f38f00c67c0 2023-11-24T08:25:34.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.181+0000 7f38eeffd640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 0x7f38dc06fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T08:25:34.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.182+0000 7f38eeffd640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 0x7f38dc06fd70 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f38e4036000 tx=0x7f38e403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T08:25:34.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.185+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38e0055470 con 0x7f38f00c67c0 2023-11-24T08:25:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.401+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 --> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7f38f0140900 con 0x7f38dc06d8b0 2023-11-24T08:25:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2023-11-24T08:25:34.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.408+0000 7f38d7fff640 1 -- 172.21.15.107:0/4012566849 <== mgr.4107 v2:172.21.15.107:6832/1315562075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f38e4030bb0 con 0x7f38dc06d8b0 2023-11-24T08:25:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 msgr2=0x7f38dc06fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 0x7f38dc06fd70 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f38e4036000 tx=0x7f38e403b040 comp rx=0 tx=0).stop 2023-11-24T08:25:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 msgr2=0x7f38f01573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T08:25:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f38e000c7c0 tx=0x7f38e000cc80 comp rx=0 tx=0).stop 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 shutdown_connections 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:6832/1315562075,v1:172.21.15.107:6833/1315562075] conn(0x7f38dc06d8b0 0x7f38dc06fd70 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38f0161940 0x7f38f0157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38f0140520 0x7f38f0157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 --2- 172.21.15.107:0/4012566849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38f00c67c0 0x7f38f01573e0 unknown :-1 s=CLOSED pgs=3461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 >> 172.21.15.107:0/4012566849 conn(0x7f38f00b2e40 msgr2=0x7f38f00d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 shutdown_connections 2023-11-24T08:25:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-11-24T08:25:34.411+0000 7f38f5b1d640 1 -- 172.21.15.107:0/4012566849 wait complete. 2023-11-24T08:25:34.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:25:34.431 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-11-24T08:25:34.495 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-11-24T08:25:34.496 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T08:25:34.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T08:25:34.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:25:34.606 INFO:tasks.cephfs_test_runner:recreate a subvolume from one of its retained snapshots ... ERROR 2023-11-24T08:25:34.606 INFO:tasks.cephfs_test_runner: 2023-11-24T08:25:34.606 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T08:25:34.606 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner:recreate a subvolume from one of its retained snapshots 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/test_volumes.py", line 409, in tearDown 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/cephfs_test_case.py", line 203, in tearDown 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: self.mds_cluster.clear_firewall() 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/filesystem.py", line 476, in clear_firewall 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: clear_firewall(self._ctx) 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/nuke/actions.py", line 19, in clear_firewall 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: ctx.cluster.run( 2023-11-24T08:25:34.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-11-24T08:25:34.608 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner: raise ConnectionLostError(command=self.command, 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner: 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:Ran 16 tests in 1202.266s 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner: 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner: 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:recreate a subvolume from one of its retained snapshots 2023-11-24T08:25:34.609 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/test_volumes.py", line 409, in tearDown 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/cephfs_test_case.py", line 203, in tearDown 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: self.mds_cluster.clear_firewall() 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/filesystem.py", line 476, in clear_firewall 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: clear_firewall(self._ctx) 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/nuke/actions.py", line 19, in clear_firewall 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: ctx.cluster.run( 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-11-24T08:25:34.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner: raise ConnectionLostError(command=self.command, 2023-11-24T08:25:34.611 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 2023-11-24T08:25:34.612 INFO:tasks.cephfs_test_runner: 2023-11-24T08:25:34.612 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:25:34.677 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=84f8c90c340444debc26c3baea5d2d7f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T08:25:34.679 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-24T08:25:34.688 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-24T08:25:34.727 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-24T08:25:34.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:25:34.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:25:34.733 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:25:38.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-24T08:25:38.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-11-24T08:25:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2023-11-24T08:25:38.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-11-24T08:25:43.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-11-24T08:25:43.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~5s 2023-11-24T08:25:43.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~5s 2023-11-24T08:25:43.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2023-11-24T08:25:43.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:25:43.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:25:47.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:25:47.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:25:48.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~10s 2023-11-24T08:25:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~10s 2023-11-24T08:25:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~10s 2023-11-24T08:25:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~10s 2023-11-24T08:25:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~10s 2023-11-24T08:25:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~10s 2023-11-24T08:25:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~10s 2023-11-24T08:25:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~10s 2023-11-24T08:25:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~10s 2023-11-24T08:25:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~10s 2023-11-24T08:25:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~10s 2023-11-24T08:25:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~10s 2023-11-24T08:25:53.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~15s 2023-11-24T08:25:53.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~15s 2023-11-24T08:25:53.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~15s 2023-11-24T08:25:53.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~15s 2023-11-24T08:25:53.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~15s 2023-11-24T08:25:53.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~15s 2023-11-24T08:25:53.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~15s 2023-11-24T08:25:53.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2023-11-24T08:25:53.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~15s 2023-11-24T08:25:53.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~15s 2023-11-24T08:25:53.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~15s 2023-11-24T08:25:53.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~15s 2023-11-24T08:25:53.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~15s 2023-11-24T08:25:53.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~15s 2023-11-24T08:25:53.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~15s 2023-11-24T08:25:53.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~15s 2023-11-24T08:25:53.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~15s 2023-11-24T08:25:58.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~20s 2023-11-24T08:25:58.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~20s 2023-11-24T08:25:58.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~20s 2023-11-24T08:25:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~20s 2023-11-24T08:25:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~20s 2023-11-24T08:25:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~20s 2023-11-24T08:25:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~20s 2023-11-24T08:25:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~20s 2023-11-24T08:25:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~20s 2023-11-24T08:25:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~20s 2023-11-24T08:25:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~20s 2023-11-24T08:25:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2023-11-24T08:25:58.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2023-11-24T08:25:58.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2023-11-24T08:25:58.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2023-11-24T08:25:58.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~20s 2023-11-24T08:25:58.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~20s 2023-11-24T08:26:03.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~25s 2023-11-24T08:26:03.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~25s 2023-11-24T08:26:03.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~25s 2023-11-24T08:26:03.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~25s 2023-11-24T08:26:03.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~25s 2023-11-24T08:26:08.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2023-11-24T08:26:08.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~30s 2023-11-24T08:26:08.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~30s 2023-11-24T08:26:08.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~30s 2023-11-24T08:26:13.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~35s 2023-11-24T08:26:13.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~35s 2023-11-24T08:26:13.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~35s 2023-11-24T08:26:13.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~35s 2023-11-24T08:26:13.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2023-11-24T08:26:13.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2023-11-24T08:26:13.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2023-11-24T08:26:13.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2023-11-24T08:26:13.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~35s 2023-11-24T08:26:13.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2023-11-24T08:26:13.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2023-11-24T08:26:13.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~35s 2023-11-24T08:26:13.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~35s 2023-11-24T08:26:13.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~35s 2023-11-24T08:26:13.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~35s 2023-11-24T08:26:13.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~35s 2023-11-24T08:26:13.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~35s 2023-11-24T08:26:18.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:18.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~40s 2023-11-24T08:26:18.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~40s 2023-11-24T08:26:18.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~40s 2023-11-24T08:26:18.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~40s 2023-11-24T08:26:18.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~40s 2023-11-24T08:26:18.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~40s 2023-11-24T08:26:21.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:23.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~45s 2023-11-24T08:26:23.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~45s 2023-11-24T08:26:23.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~45s 2023-11-24T08:26:23.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~45s 2023-11-24T08:26:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~45s 2023-11-24T08:26:23.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~45s 2023-11-24T08:26:28.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2023-11-24T08:26:28.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~50s 2023-11-24T08:26:28.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~50s 2023-11-24T08:26:28.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~50s 2023-11-24T08:26:28.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~50s 2023-11-24T08:26:28.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~50s 2023-11-24T08:26:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~50s 2023-11-24T08:26:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~50s 2023-11-24T08:26:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~50s 2023-11-24T08:26:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~50s 2023-11-24T08:26:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~50s 2023-11-24T08:26:30.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:30.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:30.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:30.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:33.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:33.411 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~55s 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~55s 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~55s 2023-11-24T08:26:33.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~55s 2023-11-24T08:26:33.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~55s 2023-11-24T08:26:33.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~55s 2023-11-24T08:26:33.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~55s 2023-11-24T08:26:33.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~55s 2023-11-24T08:26:36.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:36.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:36.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:36.486 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:26:36.487 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:26:36.493 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-24T08:26:36.503 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T08:26:36.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:36.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:38.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~60s 2023-11-24T08:26:38.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2023-11-24T08:26:38.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~60s 2023-11-24T08:26:38.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~60s 2023-11-24T08:26:38.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2023-11-24T08:26:38.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~60s 2023-11-24T08:26:38.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~60s 2023-11-24T08:26:38.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~60s 2023-11-24T08:26:38.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~60s 2023-11-24T08:26:38.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~60s 2023-11-24T08:26:39.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:43.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2023-11-24T08:26:43.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~65s 2023-11-24T08:26:43.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~65s 2023-11-24T08:26:45.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:45.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:48.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:48.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:48.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:48.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2023-11-24T08:26:48.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2023-11-24T08:26:48.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~70s 2023-11-24T08:26:48.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~70s 2023-11-24T08:26:51.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:26:51.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:51.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:51.554 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:26:51.559 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 183, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:26:51.563 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:26:53.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~75s 2023-11-24T08:26:53.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~75s 2023-11-24T08:26:53.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~75s 2023-11-24T08:26:53.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~75s 2023-11-24T08:26:58.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2023-11-24T08:26:58.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2023-11-24T08:26:58.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2023-11-24T08:26:58.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2023-11-24T08:26:58.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~80s 2023-11-24T08:26:58.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~80s 2023-11-24T08:26:58.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~80s 2023-11-24T08:26:58.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~80s 2023-11-24T08:26:58.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~80s 2023-11-24T08:26:58.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~80s 2023-11-24T08:26:58.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~80s 2023-11-24T08:26:58.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~80s 2023-11-24T08:27:03.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:03.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:03.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2023-11-24T08:27:03.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2023-11-24T08:27:03.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2023-11-24T08:27:03.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~85s 2023-11-24T08:27:03.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~85s 2023-11-24T08:27:03.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~85s 2023-11-24T08:27:03.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~85s 2023-11-24T08:27:03.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~85s 2023-11-24T08:27:03.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~85s 2023-11-24T08:27:03.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~85s 2023-11-24T08:27:06.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:27:08.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~90s 2023-11-24T08:27:08.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~90s 2023-11-24T08:27:08.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~90s 2023-11-24T08:27:08.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~90s 2023-11-24T08:27:09.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:09.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:09.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:27:09.702 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:27:09.703 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:27:09.722 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-24T08:27:09.723 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T08:27:09.723 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.728 ERROR:teuthology.misc:Saw exception from ceph.mds.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2023-11-24T08:27:09.755 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T08:27:09.755 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.756 ERROR:teuthology.misc:Saw exception from ceph.mds.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2023-11-24T08:27:09.769 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T08:27:09.769 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.769 ERROR:teuthology.misc:Saw exception from ceph.mds.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2023-11-24T08:27:09.782 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T08:27:09.783 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.783 ERROR:teuthology.misc:Saw exception from ceph.mds.d Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2023-11-24T08:27:09.796 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-24T08:27:09.796 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-24T08:27:09.797 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.797 ERROR:teuthology.misc:Saw exception from ceph.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2023-11-24T08:27:09.815 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-24T08:27:09.815 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.816 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.816 ERROR:teuthology.misc:Saw exception from ceph.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' 2023-11-24T08:27:09.833 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-24T08:27:09.833 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.833 ERROR:teuthology.misc:Saw exception from ceph.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2023-11-24T08:27:09.850 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-24T08:27:09.850 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.851 ERROR:teuthology.misc:Saw exception from ceph.osd.3 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3' 2023-11-24T08:27:09.867 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-24T08:27:09.867 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.867 ERROR:teuthology.misc:Saw exception from ceph.osd.4 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4' 2023-11-24T08:27:09.884 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-24T08:27:09.884 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.885 ERROR:teuthology.misc:Saw exception from ceph.osd.5 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5' 2023-11-24T08:27:09.901 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-24T08:27:09.901 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.902 ERROR:teuthology.misc:Saw exception from ceph.osd.6 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6' 2023-11-24T08:27:09.918 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-24T08:27:09.919 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.919 ERROR:teuthology.misc:Saw exception from ceph.osd.7 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2023-11-24T08:27:09.936 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-24T08:27:09.936 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-24T08:27:09.936 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.936 ERROR:teuthology.misc:Saw exception from ceph.mgr.y Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-11-24T08:27:09.957 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-24T08:27:09.958 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.958 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.958 ERROR:teuthology.misc:Saw exception from ceph.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-11-24T08:27:09.981 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-24T08:27:09.981 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-24T08:27:09.981 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:09.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:09.981 ERROR:teuthology.misc:Saw exception from ceph.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a' 2023-11-24T08:27:10.007 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-24T08:27:10.007 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:10.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:10.008 ERROR:teuthology.misc:Saw exception from ceph.mon.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b' 2023-11-24T08:27:10.034 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-24T08:27:10.035 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T08:27:10.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-24T08:27:10.035 ERROR:teuthology.misc:Saw exception from ceph.mon.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi120 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c' 2023-11-24T08:27:10.061 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-24T08:27:10.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:10.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:13.122 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:27:13.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2023-11-24T08:27:13.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2023-11-24T08:27:13.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2023-11-24T08:27:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~95s 2023-11-24T08:27:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~95s 2023-11-24T08:27:18.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~100s 2023-11-24T08:27:18.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~100s 2023-11-24T08:27:18.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~100s 2023-11-24T08:27:18.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~100s 2023-11-24T08:27:22.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:22.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:23.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~105s 2023-11-24T08:27:23.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~105s 2023-11-24T08:27:23.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~105s 2023-11-24T08:27:23.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~105s 2023-11-24T08:27:28.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2023-11-24T08:27:28.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~110s 2023-11-24T08:27:28.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~110s 2023-11-24T08:27:28.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~110s 2023-11-24T08:27:28.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~110s 2023-11-24T08:27:33.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2023-11-24T08:27:33.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~115s 2023-11-24T08:27:33.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~115s 2023-11-24T08:27:33.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~115s 2023-11-24T08:27:33.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~115s 2023-11-24T08:27:33.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~115s 2023-11-24T08:27:33.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2023-11-24T08:27:33.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~115s 2023-11-24T08:27:33.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~115s 2023-11-24T08:27:33.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2023-11-24T08:27:33.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~115s 2023-11-24T08:27:33.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~115s 2023-11-24T08:27:38.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~120s 2023-11-24T08:27:38.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2023-11-24T08:27:38.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2023-11-24T08:27:38.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2023-11-24T08:27:38.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~120s 2023-11-24T08:27:38.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2023-11-24T08:27:38.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~120s 2023-11-24T08:27:38.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~120s 2023-11-24T08:27:38.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~120s 2023-11-24T08:27:38.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~120s 2023-11-24T08:27:38.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~120s 2023-11-24T08:27:38.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~120s 2023-11-24T08:27:38.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2023-11-24T08:27:38.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~120s 2023-11-24T08:27:38.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~120s 2023-11-24T08:27:38.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~120s 2023-11-24T08:27:38.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~120s 2023-11-24T08:27:40.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:27:43.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~125s 2023-11-24T08:27:43.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2023-11-24T08:27:43.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~125s 2023-11-24T08:27:43.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2023-11-24T08:27:43.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~125s 2023-11-24T08:27:43.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~125s 2023-11-24T08:27:43.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~125s 2023-11-24T08:27:43.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~125s 2023-11-24T08:27:43.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~125s 2023-11-24T08:27:43.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~125s 2023-11-24T08:27:43.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~125s 2023-11-24T08:27:43.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~125s 2023-11-24T08:27:48.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2023-11-24T08:27:48.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2023-11-24T08:27:48.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~130s 2023-11-24T08:27:48.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~130s 2023-11-24T08:27:52.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:52.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2023-11-24T08:27:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2023-11-24T08:27:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2023-11-24T08:27:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2023-11-24T08:27:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~135s 2023-11-24T08:27:53.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~135s 2023-11-24T08:27:53.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~135s 2023-11-24T08:27:53.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~135s 2023-11-24T08:27:53.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~135s 2023-11-24T08:27:53.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~135s 2023-11-24T08:27:53.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~135s 2023-11-24T08:27:54.139 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:27:57.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:27:57.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:27:57.267 ERROR:teuthology.orchestra.connection:Error authenticating with smithi107.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2023-11-24T08:27:58.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2023-11-24T08:27:58.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2023-11-24T08:27:58.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~140s 2023-11-24T08:27:58.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~140s 2023-11-24T08:27:58.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~140s 2023-11-24T08:28:01.397 ERROR:teuthology.orchestra.connection:Error authenticating with smithi107.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2023-11-24T08:28:03.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2023-11-24T08:28:03.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~145s 2023-11-24T08:28:03.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~145s 2023-11-24T08:28:03.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~145s 2023-11-24T08:28:08.486 ERROR:teuthology.orchestra.connection:Error authenticating with smithi107.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2023-11-24T08:28:08.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~150s 2023-11-24T08:28:08.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~150s 2023-11-24T08:28:08.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~150s 2023-11-24T08:28:08.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~150s 2023-11-24T08:28:13.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~155s 2023-11-24T08:28:13.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~155s 2023-11-24T08:28:13.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~155s 2023-11-24T08:28:13.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~155s 2023-11-24T08:28:18.579 ERROR:teuthology.orchestra.connection:Error authenticating with smithi107.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2023-11-24T08:28:18.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2023-11-24T08:28:18.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2023-11-24T08:28:18.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2023-11-24T08:28:18.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2023-11-24T08:28:18.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~160s 2023-11-24T08:28:18.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~160s 2023-11-24T08:28:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~160s 2023-11-24T08:28:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~160s 2023-11-24T08:28:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~160s 2023-11-24T08:28:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~160s 2023-11-24T08:28:23.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2023-11-24T08:28:23.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2023-11-24T08:28:23.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2023-11-24T08:28:23.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2023-11-24T08:28:23.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~165s 2023-11-24T08:28:23.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~165s 2023-11-24T08:28:23.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~165s 2023-11-24T08:28:23.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~165s 2023-11-24T08:28:23.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~165s 2023-11-24T08:28:23.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~165s 2023-11-24T08:28:23.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~165s 2023-11-24T08:28:23.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2023-11-24T08:28:23.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~165s 2023-11-24T08:28:23.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~165s 2023-11-24T08:28:23.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~165s 2023-11-24T08:28:23.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~165s 2023-11-24T08:28:23.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~165s 2023-11-24T08:28:28.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2023-11-24T08:28:28.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2023-11-24T08:28:28.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2023-11-24T08:28:28.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2023-11-24T08:28:28.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~170s 2023-11-24T08:28:28.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~170s 2023-11-24T08:28:28.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~170s 2023-11-24T08:28:28.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~170s 2023-11-24T08:28:28.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~170s 2023-11-24T08:28:28.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~170s 2023-11-24T08:28:28.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~170s 2023-11-24T08:28:28.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~170s 2023-11-24T08:28:28.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~170s 2023-11-24T08:28:28.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2023-11-24T08:28:28.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~170s 2023-11-24T08:28:28.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~170s 2023-11-24T08:28:28.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~170s 2023-11-24T08:28:31.710 ERROR:teuthology.orchestra.connection:Error authenticating with smithi107.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2023-11-24T08:28:33.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2023-11-24T08:28:33.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2023-11-24T08:28:33.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2023-11-24T08:28:33.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~175s 2023-11-24T08:28:33.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~175s 2023-11-24T08:28:33.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~175s 2023-11-24T08:28:33.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~175s 2023-11-24T08:28:33.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~175s 2023-11-24T08:28:33.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~175s 2023-11-24T08:28:38.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2023-11-24T08:28:38.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2023-11-24T08:28:38.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~180s 2023-11-24T08:28:38.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~180s 2023-11-24T08:28:38.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~180s 2023-11-24T08:28:43.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~185s 2023-11-24T08:28:43.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~185s 2023-11-24T08:28:43.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~185s 2023-11-24T08:28:43.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~185s 2023-11-24T08:28:43.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~185s 2023-11-24T08:28:43.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~185s 2023-11-24T08:28:43.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~185s 2023-11-24T08:28:43.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~185s 2023-11-24T08:28:43.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~185s 2023-11-24T08:28:43.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~185s 2023-11-24T08:28:43.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~185s 2023-11-24T08:28:43.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~185s 2023-11-24T08:28:48.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~190s 2023-11-24T08:28:48.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~190s 2023-11-24T08:28:48.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~190s 2023-11-24T08:28:48.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~190s 2023-11-24T08:28:48.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~190s 2023-11-24T08:28:48.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~190s 2023-11-24T08:28:53.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~195s 2023-11-24T08:28:53.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2023-11-24T08:28:53.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2023-11-24T08:28:53.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~195s 2023-11-24T08:28:53.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2023-11-24T08:28:53.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~195s 2023-11-24T08:28:53.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~195s 2023-11-24T08:28:53.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2023-11-24T08:28:53.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~195s 2023-11-24T08:28:53.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~195s 2023-11-24T08:28:58.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2023-11-24T08:28:58.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2023-11-24T08:28:58.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~200s 2023-11-24T08:28:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~200s 2023-11-24T08:28:58.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~200s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~205s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~205s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2023-11-24T08:29:03.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~205s 2023-11-24T08:29:03.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~205s 2023-11-24T08:29:03.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~205s 2023-11-24T08:29:03.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~205s 2023-11-24T08:29:03.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~205s 2023-11-24T08:29:06.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:06.275 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:29:06.275 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-24T08:29:06.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:06.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~210s 2023-11-24T08:29:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~210s 2023-11-24T08:29:08.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~210s 2023-11-24T08:29:08.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~210s 2023-11-24T08:29:09.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:13.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~215s 2023-11-24T08:29:13.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2023-11-24T08:29:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~215s 2023-11-24T08:29:13.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~215s 2023-11-24T08:29:18.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:18.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:18.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2023-11-24T08:29:18.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2023-11-24T08:29:18.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2023-11-24T08:29:18.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~220s 2023-11-24T08:29:18.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~220s 2023-11-24T08:29:18.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~220s 2023-11-24T08:29:18.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~220s 2023-11-24T08:29:18.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~220s 2023-11-24T08:29:18.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~220s 2023-11-24T08:29:18.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~220s 2023-11-24T08:29:18.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~220s 2023-11-24T08:29:18.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~220s 2023-11-24T08:29:21.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:23.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~225s 2023-11-24T08:29:23.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~225s 2023-11-24T08:29:23.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~225s 2023-11-24T08:29:23.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2023-11-24T08:29:23.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~225s 2023-11-24T08:29:23.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~225s 2023-11-24T08:29:23.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~225s 2023-11-24T08:29:23.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~225s 2023-11-24T08:29:23.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~225s 2023-11-24T08:29:28.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~230s 2023-11-24T08:29:28.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~230s 2023-11-24T08:29:28.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~230s 2023-11-24T08:29:28.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~230s 2023-11-24T08:29:28.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~230s 2023-11-24T08:29:33.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:33.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:33.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:33.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~235s 2023-11-24T08:29:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~235s 2023-11-24T08:29:33.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~235s 2023-11-24T08:29:36.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:36.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:38.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~240s 2023-11-24T08:29:38.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~240s 2023-11-24T08:29:38.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~240s 2023-11-24T08:29:38.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~240s 2023-11-24T08:29:38.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~240s 2023-11-24T08:29:39.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:39.554 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:29:39.555 INFO:tasks.ceph:Archiving crash dumps... 2023-11-24T08:29:39.556 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582/remote/smithi107/crash 2023-11-24T08:29:39.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:39.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:42.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:43.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2023-11-24T08:29:43.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~245s 2023-11-24T08:29:43.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~245s 2023-11-24T08:29:43.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~245s 2023-11-24T08:29:48.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2023-11-24T08:29:48.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~250s 2023-11-24T08:29:48.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~250s 2023-11-24T08:29:48.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2023-11-24T08:29:48.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~250s 2023-11-24T08:29:48.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~250s 2023-11-24T08:29:48.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~250s 2023-11-24T08:29:48.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~250s 2023-11-24T08:29:48.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~250s 2023-11-24T08:29:48.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~250s 2023-11-24T08:29:48.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~250s 2023-11-24T08:29:48.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~250s 2023-11-24T08:29:48.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~250s 2023-11-24T08:29:48.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~250s 2023-11-24T08:29:48.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~250s 2023-11-24T08:29:48.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~250s 2023-11-24T08:29:48.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~250s 2023-11-24T08:29:51.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:29:51.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:29:53.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~255s 2023-11-24T08:29:53.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~255s 2023-11-24T08:29:53.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~255s 2023-11-24T08:29:54.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-11-24T08:29:58.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2023-11-24T08:29:58.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2023-11-24T08:29:58.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2023-11-24T08:29:58.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~260s 2023-11-24T08:29:58.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~260s 2023-11-24T08:29:58.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~260s 2023-11-24T08:29:58.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~260s 2023-11-24T08:29:58.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~260s 2023-11-24T08:29:58.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~260s 2023-11-24T08:29:58.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~260s 2023-11-24T08:30:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2023-11-24T08:30:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2023-11-24T08:30:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2023-11-24T08:30:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~265s 2023-11-24T08:30:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~265s 2023-11-24T08:30:03.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~265s 2023-11-24T08:30:03.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~265s 2023-11-24T08:30:03.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~265s 2023-11-24T08:30:03.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~265s 2023-11-24T08:30:03.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~265s 2023-11-24T08:30:03.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~265s 2023-11-24T08:30:06.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:30:06.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:30:08.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2023-11-24T08:30:08.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~270s 2023-11-24T08:30:08.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~270s 2023-11-24T08:30:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~270s 2023-11-24T08:30:08.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~270s 2023-11-24T08:30:13.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~275s 2023-11-24T08:30:13.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2023-11-24T08:30:13.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2023-11-24T08:30:13.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~275s 2023-11-24T08:30:13.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~275s 2023-11-24T08:30:13.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~275s 2023-11-24T08:30:13.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~275s 2023-11-24T08:30:13.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~275s 2023-11-24T08:30:13.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~275s 2023-11-24T08:30:13.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~275s 2023-11-24T08:30:13.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~275s 2023-11-24T08:30:13.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~275s 2023-11-24T08:30:18.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~280s 2023-11-24T08:30:18.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~280s 2023-11-24T08:30:18.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~280s 2023-11-24T08:30:18.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~280s 2023-11-24T08:30:23.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~285s 2023-11-24T08:30:23.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~285s 2023-11-24T08:30:23.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~285s 2023-11-24T08:30:23.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~285s 2023-11-24T08:30:23.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~285s 2023-11-24T08:30:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~285s 2023-11-24T08:30:23.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~285s 2023-11-24T08:30:23.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~285s 2023-11-24T08:30:23.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~285s 2023-11-24T08:30:23.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~285s 2023-11-24T08:30:28.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2023-11-24T08:30:28.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~290s 2023-11-24T08:30:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~290s 2023-11-24T08:30:28.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~290s 2023-11-24T08:30:33.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~295s 2023-11-24T08:30:33.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~295s 2023-11-24T08:30:33.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~295s 2023-11-24T08:30:38.446 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-11-24T08:30:38.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-11-24T08:30:38.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T08:30:38.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~300s 2023-11-24T08:30:38.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~300s 2023-11-24T08:30:38.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~300s 2023-11-24T08:30:38.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~300s 2023-11-24T08:30:38.814 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-24T08:30:38.814 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:30:38.814 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:30:38.867 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-11-24T08:30:38.867 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2023-11-24T08:30:38.872 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054577,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495224k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18663 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247608k,nr_inodes=811902,mode=700,uid=1000,gid=1301,inode64 0 0 2023-11-24T08:30:38.872 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:30:38.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:30:38.873 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T08:30:38.900 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582/remote/smithi120/crash 2023-11-24T08:30:38.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:30:38.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:30:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:30:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: cd: /home/ubuntu/cephtest: No such file or directory 2023-11-24T08:30:38.954 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-11-24T08:30:38.954 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:30:39.008 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054577,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495224k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18663 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247608k,nr_inodes=811902,mode=700,uid=1000,gid=1301,inode64 0 0 2023-11-24T08:30:39.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:30:39.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:30:39.009 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-24T08:30:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:30:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: cd: /home/ubuntu/cephtest: No such file or directory 2023-11-24T08:30:39.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:30:39.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:30:54.823 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:30:54.824 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:03.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:03.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:03.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:03.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:03.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:03.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:15.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:15.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:15.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:15.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:15.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:15.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:18.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:18.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:18.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:18.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:18.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:18.840 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:31:18.840 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi120 2023-11-24T08:31:18.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:18.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:18.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:18.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:18.852 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-11-24T08:31:18.852 INFO:tasks.ceph:Shutting down logrotate 2023-11-24T08:31:18.853 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 256, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 221, in end self.thread.get() File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/ceph.py", line 183, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:31:18.875 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-24T08:31:18.887 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:31:18.897 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-24T08:31:18.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T08:31:18.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:18.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:18.912 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2023-11-24T08:31:27.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:27.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:27.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:27.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:31:27.927 DEBUG:teuthology.orchestra.run.smithi120:> true 2023-11-24T08:31:27.990 DEBUG:teuthology.orchestra.run.smithi120:> true 2023-11-24T08:31:28.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:28.038 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2023-11-24T08:31:28.038 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T08:31:28.042 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2023-11-24T08:31:28.043 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T08:31:28.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342644k,nr_inodes=4085661,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275920k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15983 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275916k,nr_inodes=818979,mode=700,uid=1000,gid=1271,inode64 0 0 2023-11-24T08:31:28.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:31:28.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T08:31:28.055 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-24T08:31:28.101 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: cd: /home/ubuntu/cephtest: No such file or directory 2023-11-24T08:31:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:28.154 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-24T08:31:28.169 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-24T08:31:28.169 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:31:28.172 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:31:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2023-11-24T08:31:28.187 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T08:31:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-11-24T08:31:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 17 17 +2105us[+2132us] +/- 80ms 2023-11-24T08:31:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 17 17 -821us[ -794us] +/- 56ms 2023-11-24T08:31:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 17 17 -1050us[-1050us] +/- 88ms 2023-11-24T08:31:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T08:31:28.201 INFO:teuthology.orchestra.run.smithi120.stdout: remote refid st t when poll reach delay offset jitter 2023-11-24T08:31:28.201 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================== 2023-11-24T08:31:28.202 INFO:teuthology.orchestra.run.smithi120.stdout: hv01.front.sepi 44.4.53.2 2 u 31 64 1 0.163 -82.249 0.000 2023-11-24T08:31:28.202 INFO:teuthology.orchestra.run.smithi120.stdout: hv02.front.sepi 63.231.80.2 3 u 28 64 1 0.121 -79.379 0.000 2023-11-24T08:31:28.202 INFO:teuthology.orchestra.run.smithi120.stdout: hv03.front.sepi 74.6.168.72 3 u 28 64 1 0.225 -79.118 0.000 2023-11-24T08:31:28.202 INFO:teuthology.orchestra.run.smithi120.stdout: hv04.front.sepi .XFAC. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:31:28.203 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-24T08:31:28.212 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-24T08:31:28.213 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-24T08:31:28.246 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-24T08:31:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/archive/audit’: No such file or directory 2023-11-24T08:31:28.261 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2023-11-24T08:31:28.269 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-24T08:31:28.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-24T08:31:28.313 INFO:teuthology.task.internal:Duration was 2380.347324 seconds 2023-11-24T08:31:28.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-24T08:31:28.348 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-24T08:31:28.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T08:31:28.354 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T08:31:28.397 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T08:31:28.820 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-24T08:31:28.820 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-11-24T08:31:28.821 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T08:31:28.842 INFO:teuthology.orchestra.run.smithi107.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2023-11-24T08:31:28.843 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2023-11-24T08:31:28.843 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T08:31:28.852 INFO:teuthology.orchestra.run.smithi120.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2023-11-24T08:31:28.854 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-24T08:31:28.855 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T08:31:28.885 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T08:31:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2023-11-24T08:31:28.902 INFO:teuthology.orchestra.run.smithi120.stderr:find: '/home/ubuntu/cephtest/archive/syslog': No such file or directory 2023-11-24T08:31:28.925 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-24T08:31:28.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T08:31:28.953 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T08:31:28.964 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2023-11-24T08:31:28.974 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2023-11-24T08:31:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:28.982 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/syslog.py", line 185, in syslog run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz' 2023-11-24T08:31:28.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-24T08:31:29.005 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-24T08:31:29.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T08:31:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:mv: cannot stat '/etc/sudoers.orig.teuthology': No such file or directory 2023-11-24T08:31:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:29.049 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2023-11-24T08:31:29.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-24T08:31:29.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T08:31:29.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T08:31:29.110 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2023-11-24T08:31:29.119 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-11-24T08:31:29.122 INFO:teuthology.orchestra.run.smithi120.stderr:find: '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2023-11-24T08:31:29.128 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2023-11-24T08:31:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2023-11-24T08:31:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:29.141 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2023-11-24T08:31:29.141 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 491, in coredump run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: "sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump" 2023-11-24T08:31:29.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-24T08:31:29.161 INFO:teuthology.task.internal:Transferring archived files... 2023-11-24T08:31:29.163 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465582/remote/smithi107 2023-11-24T08:31:29.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T08:31:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-11-24T08:31:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2023-11-24T08:31:29.208 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/tarfile.py", line 2318, in next tarinfo = self.tarinfo.fromtarfile(self) File "/usr/lib/python3.8/tarfile.py", line 1105, in fromtarfile obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors) File "/usr/lib/python3.8/tarfile.py", line 1041, in frombuf raise EmptyHeaderError("empty header") tarfile.EmptyHeaderError: empty header During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/misc.py", line 749, in pull_directory tar = tarfile.open(mode='r|', fileobj=r.stdout) File "/usr/lib/python3.8/tarfile.py", line 1633, in open t = cls(name, filemode, stream, **kwargs) File "/usr/lib/python3.8/tarfile.py", line 1514, in __init__ self.firstmember = self.next() File "/usr/lib/python3.8/tarfile.py", line 2333, in next raise ReadError("empty file") tarfile.ReadError: empty file 2023-11-24T08:31:29.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-24T08:31:29.232 INFO:teuthology.task.internal:Not uploading archives. 2023-11-24T08:31:29.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-24T08:31:29.265 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-24T08:31:29.266 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T08:31:29.271 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T08:31:29.277 INFO:teuthology.orchestra.run.smithi120.stderr:find: '/home/ubuntu/cephtest': No such file or directory 2023-11-24T08:31:29.279 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': No such file or directory 2023-11-24T08:31:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest’: No such file or directory 2023-11-24T08:31:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:31:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': No such file or directory 2023-11-24T08:31:29.285 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73843baf45f8a92d8e3fcbea3f024c97e73036da/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 253, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 521, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi107 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-24T08:31:29.295 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-24T08:31:29.304 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-24T08:31:29.332 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionError: Failed to reconnect to smithi107 2023-11-24T08:31:29.333 INFO:teuthology.nuke:Checking targets against current locks 2023-11-24T08:31:29.361 INFO:teuthology.nuke:Not nuking smithi107.front.sepia.ceph.com because description doesn't match: rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi != /home/teuthworker/archive/rishabh-2023-11-24_08:24:50-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465592 2023-11-24T08:31:29.384 INFO:teuthology.nuke:Not nuking smithi120.front.sepia.ceph.com because description doesn't match: rishabh-2023-11-24_07:31:48-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi != /home/teuthworker/archive/rishabh-2023-11-24_08:24:50-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465589 2023-11-24T08:31:29.384 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUWWHRQesnb0foUsW5bdD9/0e/M5daF7En+5E/TguR9kZ+kwLCXni0H9oxKVzMyoRSWtO4uh+NIzSB/tqpFRv8= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCd+WpaFe5MZUrwVrtno2fV8ntxSVfm4mTvahUZm6VEP8Jfebp4KLxUXw2rc9pNjqB960xGtkr+TIUiqqjxEqd0= 2023-11-24T08:31:29.395 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 client.1-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} duration: 2380.347323656082 failure_reason: 'Test failure: test_subvolume_retain_snapshot_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=84f8c90c340444debc26c3baea5d2d7f status: fail success: false 2023-11-24T08:31:29.396 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T08:31:29.469 INFO:teuthology.run:FAIL